2024-06-29T14:11:41.633 INFO:root:teuthology version: 0.0.1.dev329+g544fecb 2024-06-29T14:11:41.633 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} --name mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi --verbose --owner scheduled_mchangir@teuthology -- /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926/orig.config.yaml 2024-06-29T14:11:41.685 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-29T14:11:41.770 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926 branch: wip-mchangir-qa-handle-correct-CephManager-instantiation-debug description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} email: mchangir@redhat.com first_in_suite: false job_id: '7778926' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-qa-handle-correct-CephManager-instantiation-debug ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: client: pid file: '' global: ms die on skipped message: false mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: cf146c2f9dac0bd5aed66008255af2113db872f7 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: cf146c2f9dac0bd5aed66008255af2113db872f7 kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: qa-handle-correct-CephManager-instantiation sha1: 019bb938d4977149d100ab4c3d2c706147295470 owner: scheduled_mchangir@teuthology priority: 25 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1683 sha1: cf146c2f9dac0bd5aed66008255af2113db872f7 sleep_before_teardown: 0 suite: fs suite_branch: qa-handle-correct-CephManager-instantiation suite_path: /home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa suite_relpath: qa suite_repo: https://github.com/mchangir/ceph.git suite_sha1: 019bb938d4977149d100ab4c3d2c706147295470 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHAK94OvNRrl6Hj3oVfsn5HM5Yxko9bHDQ11F3N3QYtWe3AN0O1HBUIhOTMbVt6J0N98/F53Io9l/Tbbul/QF1c= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdOwx6NJOHHxMeaoXCSq7R8JV4iSVti8gQ/L1RjNqGpgsM+Z4UzrIV5IzWZTegiQpYwljsAKeE9TY36xsCKDu0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 544fecbcd55f3d2b6f478823823ce40cbefef1d4 timestamp: 2024-06-29_14:01:10 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698093 2024-06-29T14:11:41.771 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa; will attempt to use it 2024-06-29T14:11:41.771 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks 2024-06-29T14:11:41.772 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-29T14:11:41.773 INFO:teuthology.task.internal:Checking packages... 2024-06-29T14:11:41.794 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'cf146c2f9dac0bd5aed66008255af2113db872f7' 2024-06-29T14:11:41.794 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-29T14:11:41.794 INFO:teuthology.packaging:ref: None 2024-06-29T14:11:41.794 INFO:teuthology.packaging:tag: None 2024-06-29T14:11:41.794 INFO:teuthology.packaging:branch: wip-mchangir-qa-handle-correct-CephManager-instantiation-debug 2024-06-29T14:11:41.794 INFO:teuthology.packaging:sha1: cf146c2f9dac0bd5aed66008255af2113db872f7 2024-06-29T14:11:41.794 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-qa-handle-correct-CephManager-instantiation-debug 2024-06-29T14:11:42.039 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-4577.gcf146c2f 2024-06-29T14:11:42.041 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-29T14:11:42.572 INFO:teuthology.task.internal:no buildpackages task found 2024-06-29T14:11:42.572 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-29T14:11:42.699 INFO:teuthology.task.internal:Saving configuration 2024-06-29T14:11:42.717 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-29T14:11:42.723 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-29T14:11:42.747 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-29 14:06:07.313698', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHAK94OvNRrl6Hj3oVfsn5HM5Yxko9bHDQ11F3N3QYtWe3AN0O1HBUIhOTMbVt6J0N98/F53Io9l/Tbbul/QF1c='} 2024-06-29T14:11:42.769 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-29 14:06:07.312261', 'locked_by': 'scheduled_mchangir@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdOwx6NJOHHxMeaoXCSq7R8JV4iSVti8gQ/L1RjNqGpgsM+Z4UzrIV5IzWZTegiQpYwljsAKeE9TY36xsCKDu0='} 2024-06-29T14:11:42.769 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-29T14:11:42.775 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-06-29T14:11:42.775 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-06-29T14:11:42.775 INFO:teuthology.run_tasks:Running task console_log... 2024-06-29T14:11:42.844 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4e70033ac0>, signals=[15]) 2024-06-29T14:11:42.845 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-29T14:11:42.851 INFO:teuthology.task.internal:Opening connections... 2024-06-29T14:11:42.851 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:11:42.852 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:11:42.934 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T14:11:42.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:11:43.013 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-29T14:11:43.021 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-06-29T14:11:43.039 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-06-29T14:11:43.040 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-06-29T14:11:43.096 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2024-06-29T14:11:43.096 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="9" 2024-06-29T14:11:43.096 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2024-06-29T14:11:43.096 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2024-06-29T14:11:43.096 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="9" 2024-06-29T14:11:43.096 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el9" 2024-06-29T14:11:43.096 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-29T14:11:43.096 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2024-06-29T14:11:43.097 INFO:teuthology.orchestra.run.smithi081.stdout:LOGO="fedora-logo-icon" 2024-06-29T14:11:43.097 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-29T14:11:43.097 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2024-06-29T14:11:43.097 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-29T14:11:43.097 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-29T14:11:43.097 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-29T14:11:43.098 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-06-29T14:11:43.131 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2024-06-29T14:11:43.149 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2024-06-29T14:11:43.150 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2024-06-29T14:11:43.206 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="CentOS Stream" 2024-06-29T14:11:43.206 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="9" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:ID="centos" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="rhel fedora" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="9" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el9" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:LOGO="fedora-logo-icon" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://centos.org/" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-29T14:11:43.207 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-29T14:11:43.208 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2024-06-29T14:11:43.230 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-29T14:11:43.240 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-29T14:11:43.245 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-29T14:11:43.245 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-06-29T14:11:43.248 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2024-06-29T14:11:43.264 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-29T14:11:43.269 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-29T14:11:43.270 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-06-29T14:11:43.304 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2024-06-29T14:11:43.329 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-29T14:11:43.383 INFO:teuthology.run_tasks:Running task kernel... 2024-06-29T14:11:43.401 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-06-29T14:11:43.402 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-06-29T14:11:43.402 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-06-29T14:11:43.402 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-29T14:11:43.403 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-29T14:11:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:11:43.419 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-06-29T14:11:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:11:43.420 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-06-29T14:11:43.475 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-437.el9.x86_64 2024-06-29T14:11:43.475 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.14.0-437.el9.x86_64 2024-06-29T14:11:43.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum install -y kernel 2024-06-29T14:11:43.477 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-437.el9.x86_64 2024-06-29T14:11:43.477 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.14.0-437.el9.x86_64 2024-06-29T14:11:43.477 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2024-06-29T14:11:46.799 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.1 MB 00:01 2024-06-29T14:11:46.993 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.1 MB 00:01 2024-06-29T14:11:51.469 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 20 MB 00:03 2024-06-29T14:11:55.480 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-06-29T14:11:55.595 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.4 MB 00:00 2024-06-29T14:11:57.496 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 17 kB 00:00 2024-06-29T14:11:58.374 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-06-29T14:12:00.588 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 6.4 MB 00:01 2024-06-29T14:12:02.239 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 17 kB 00:00 2024-06-29T14:12:03.418 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-06-29T14:12:03.639 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-06-29T14:12:05.711 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-29T14:12:05.712 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-29T14:12:05.804 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 5.14.0-467.el9 baseos 1.2 M 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core x86_64 5.14.0-467.el9 baseos 16 M 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules x86_64 5.14.0-467.el9 baseos 35 M 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core x86_64 5.14.0-467.el9 baseos 29 M 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:12:05.814 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-06-29T14:12:05.815 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-06-29T14:12:05.815 INFO:teuthology.orchestra.run.smithi081.stdout:Install 4 Packages 2024-06-29T14:12:05.815 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:12:05.815 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 82 M 2024-06-29T14:12:05.815 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 125 M 2024-06-29T14:12:05.815 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-06-29T14:12:06.665 INFO:teuthology.orchestra.run.smithi081.stdout:(1/4): kernel-5.14.0-467.el9.x86_64.rpm 1.5 MB/s | 1.2 MB 00:00 2024-06-29T14:12:08.682 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-06-29T14:12:10.772 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-29T14:12:10.772 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-29T14:12:10.861 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-06-29T14:12:10.870 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-06-29T14:12:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-06-29T14:12:10.870 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-06-29T14:12:10.870 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-06-29T14:12:10.870 INFO:teuthology.orchestra.run.smithi167.stdout: kernel x86_64 5.14.0-467.el9 baseos 1.2 M 2024-06-29T14:12:10.870 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-06-29T14:12:10.871 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core x86_64 5.14.0-467.el9 baseos 16 M 2024-06-29T14:12:10.871 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules x86_64 5.14.0-467.el9 baseos 35 M 2024-06-29T14:12:10.871 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core x86_64 5.14.0-467.el9 baseos 29 M 2024-06-29T14:12:10.871 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:12:10.871 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-06-29T14:12:10.871 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-06-29T14:12:10.871 INFO:teuthology.orchestra.run.smithi167.stdout:Install 4 Packages 2024-06-29T14:12:10.871 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:12:10.872 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 82 M 2024-06-29T14:12:10.872 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 125 M 2024-06-29T14:12:10.872 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-06-29T14:12:11.092 INFO:teuthology.orchestra.run.smithi167.stdout:(1/4): kernel-5.14.0-467.el9.x86_64.rpm 6.3 MB/s | 1.2 MB 00:00 2024-06-29T14:12:11.426 INFO:teuthology.orchestra.run.smithi167.stdout:(2/4): kernel-core-5.14.0-467.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-06-29T14:12:11.884 INFO:teuthology.orchestra.run.smithi167.stdout:(3/4): kernel-modules-5.14.0-467.el9.x86_64.rpm 36 MB/s | 35 MB 00:00 2024-06-29T14:12:12.285 INFO:teuthology.orchestra.run.smithi167.stdout:(4/4): kernel-modules-core-5.14.0-467.el9.x86_6 25 MB/s | 29 MB 00:01 2024-06-29T14:12:12.285 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-06-29T14:12:12.285 INFO:teuthology.orchestra.run.smithi167.stdout:Total 58 MB/s | 82 MB 00:01 2024-06-29T14:12:12.998 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-06-29T14:12:13.333 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-06-29T14:12:13.333 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-06-29T14:12:14.294 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-06-29T14:12:14.294 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-06-29T14:12:16.731 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-06-29T14:12:17.423 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-core-5.14.0-467.el9.x86_64 1/4 2024-06-29T14:12:17.511 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:12:20.191 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:12:20.342 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:12:21.289 INFO:teuthology.orchestra.run.smithi081.stdout:(2/4): kernel-core-5.14.0-467.el9.x86_64.rpm 1.1 MB/s | 16 MB 00:15 2024-06-29T14:12:25.108 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:12:25.232 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:30.110 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:12:41.329 INFO:teuthology.orchestra.run.smithi081.stdout:(3/4): kernel-modules-core-5.14.0-467.el9.x86_6 865 kB/s | 29 MB 00:34 2024-06-29T14:12:47.362 INFO:teuthology.orchestra.run.smithi081.stdout:(4/4): kernel-modules-5.14.0-467.el9.x86_64.rpm 872 kB/s | 35 MB 00:41 2024-06-29T14:12:47.362 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-06-29T14:12:47.362 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.0 MB/s | 82 MB 00:41 2024-06-29T14:12:48.075 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-06-29T14:12:48.425 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-06-29T14:12:48.425 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-06-29T14:12:49.402 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-06-29T14:12:49.403 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-06-29T14:12:51.839 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-06-29T14:12:52.555 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-core-5.14.0-467.el9.x86_64 1/4 2024-06-29T14:12:52.655 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:12:55.335 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:12:55.477 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:13:00.068 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:13:00.108 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:13:00.271 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:13:00.334 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:13:00.505 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:13:00.506 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-5.14.0-467.el9.x86_64 1/4 2024-06-29T14:13:00.506 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:13:00.506 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:13:01.330 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:13:01.330 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:13:01.330 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-06-29T14:13:01.330 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-5.14.0-467.el9.x86_64 2024-06-29T14:13:01.331 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core-5.14.0-467.el9.x86_64 2024-06-29T14:13:01.331 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-5.14.0-467.el9.x86_64 2024-06-29T14:13:01.331 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core-5.14.0-467.el9.x86_64 2024-06-29T14:13:01.331 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:13:01.331 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-06-29T14:13:01.671 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-06-29T14:13:01.750 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-29T14:13:01.750 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-29T14:13:01.751 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-467.el9.x86_64 2024-06-29T14:13:01.751 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-29T14:13:01.751 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-29T14:13:01.751 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2024-06-29T14:13:01.751 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2024-06-29T14:13:01.751 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-06-29T14:13:01.805 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-29T14:13:01.805 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-29T14:13:01.805 DEBUG:teuthology.orchestra.run.smithi167:> sudo rpm -qi grub2-tools 2024-06-29T14:13:01.836 INFO:teuthology.orchestra.run.smithi167.stdout:Name : grub2-tools 2024-06-29T14:13:01.836 INFO:teuthology.orchestra.run.smithi167.stdout:Epoch : 1 2024-06-29T14:13:01.836 INFO:teuthology.orchestra.run.smithi167.stdout:Version : 2.06 2024-06-29T14:13:01.836 INFO:teuthology.orchestra.run.smithi167.stdout:Release : 68.el9 2024-06-29T14:13:01.836 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: x86_64 2024-06-29T14:13:01.836 INFO:teuthology.orchestra.run.smithi167.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-29T14:13:01.836 INFO:teuthology.orchestra.run.smithi167.stdout:Group : Unspecified 2024-06-29T14:13:01.836 INFO:teuthology.orchestra.run.smithi167.stdout:Size : 8264254 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:License : GPLv3+ 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:Packager : builder@centos.org 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:Vendor : CentOS 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:Summary : Support tools for GRUB. 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:Description : 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-29T14:13:01.837 INFO:teuthology.orchestra.run.smithi167.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-29T14:13:01.838 INFO:teuthology.orchestra.run.smithi167.stdout:hardware devices. 2024-06-29T14:13:01.838 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:13:01.838 INFO:teuthology.orchestra.run.smithi167.stdout:This subpackage provides tools for support of all platforms. 2024-06-29T14:13:01.838 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-29T14:13:01.839 INFO:teuthology.task.kernel:Updating grub on smithi167 to boot 5.14.0-467.el9.x86_64 2024-06-29T14:13:01.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-29T14:13:02.457 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2024-06-29T14:13:03.817 INFO:teuthology.orchestra.run.smithi167.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-29T14:13:03.845 INFO:teuthology.orchestra.run.smithi167.stderr:done 2024-06-29T14:13:03.854 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-06-29T14:13:03.869 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.AjB5udkvXs 2024-06-29T14:13:03.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AjB5udkvXs 2024-06-29T14:13:03.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.AjB5udkvXs 2024-06-29T14:13:04.077 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.AjB5udkvXs is 6KB 2024-06-29T14:13:04.126 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.AjB5udkvXs 2024-06-29T14:13:04.141 DEBUG:teuthology.orchestra.run.smithi167:> sudo /bin/ls /boot/loader/entries || true 2024-06-29T14:13:04.203 INFO:teuthology.orchestra.run.smithi167.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-29T14:13:04.203 INFO:teuthology.orchestra.run.smithi167.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-29T14:13:04.204 INFO:teuthology.orchestra.run.smithi167.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-29T14:13:04.204 INFO:teuthology.orchestra.run.smithi167.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64.conf 2024-06-29T14:13:04.205 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64 2024-06-29T14:13:04.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2024-06-29T14:13:05.135 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:13:34.353 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-06-29T14:13:34.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-06-29T14:13:34.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:13:34.575 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:13:34.615 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:13:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:13:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-5.14.0-467.el9.x86_64 1/4 2024-06-29T14:13:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-29T14:13:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-29T14:13:35.889 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-29T14:13:35.889 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:13:35.889 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-06-29T14:13:35.890 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-5.14.0-467.el9.x86_64 2024-06-29T14:13:35.890 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core-5.14.0-467.el9.x86_64 2024-06-29T14:13:35.890 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-5.14.0-467.el9.x86_64 2024-06-29T14:13:35.890 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core-5.14.0-467.el9.x86_64 2024-06-29T14:13:35.890 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:13:35.890 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-06-29T14:13:36.289 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-06-29T14:13:36.374 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-29T14:13:36.374 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-29T14:13:36.374 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-467.el9.x86_64 2024-06-29T14:13:36.374 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-29T14:13:36.374 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-29T14:13:36.375 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-06-29T14:13:36.375 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2024-06-29T14:13:36.375 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-06-29T14:13:36.428 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-29T14:13:36.428 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-29T14:13:36.428 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2024-06-29T14:13:36.467 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2024-06-29T14:13:36.467 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2024-06-29T14:13:36.467 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.06 2024-06-29T14:13:36.467 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 68.el9 2024-06-29T14:13:36.467 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:Group : Unspecified 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 8264254 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : builder@centos.org 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-29T14:13:36.468 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2024-06-29T14:13:36.469 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2024-06-29T14:13:36.469 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:13:36.469 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-29T14:13:36.469 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-29T14:13:36.469 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-29T14:13:36.469 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2024-06-29T14:13:36.469 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:13:36.469 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2024-06-29T14:13:36.470 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-29T14:13:36.470 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 5.14.0-467.el9.x86_64 2024-06-29T14:13:36.470 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-29T14:13:37.083 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-06-29T14:13:38.456 INFO:teuthology.orchestra.run.smithi081.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-29T14:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-06-29T14:13:38.489 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-06-29T14:13:38.504 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.x5JK12WMtr 2024-06-29T14:13:38.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.x5JK12WMtr 2024-06-29T14:13:38.569 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.x5JK12WMtr 2024-06-29T14:13:38.711 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.x5JK12WMtr is 6KB 2024-06-29T14:13:38.761 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.x5JK12WMtr 2024-06-29T14:13:38.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2024-06-29T14:13:38.846 INFO:teuthology.orchestra.run.smithi081.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-29T14:13:38.847 INFO:teuthology.orchestra.run.smithi081.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-29T14:13:38.847 INFO:teuthology.orchestra.run.smithi081.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-29T14:13:38.847 INFO:teuthology.orchestra.run.smithi081.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64.conf 2024-06-29T14:13:38.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64 2024-06-29T14:13:39.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-06-29T14:14:08.931 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-06-29T14:14:09.013 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-06-29T14:14:09.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-06-29T14:14:09.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:14:17.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-06-29T14:14:17.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:14:20.994 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-06-29T14:14:27.363 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-06-29T14:14:32.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-06-29T14:14:32.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:14:36.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-06-29T14:14:36.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-06-29T14:14:36.365 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:14:39.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-06-29T14:14:51.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-06-29T14:14:51.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:14:51.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-06-29T14:14:51.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:14:51.464 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-06-29T14:14:51.893 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-06-29T14:14:51.893 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-467.el9.x86_64"... 2024-06-29T14:14:51.893 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-06-29T14:14:51.908 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-467.el9.x86_64 2024-06-29T14:14:51.908 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-467.el9.x86_64 vs 5.14.0-467.el9.x86_64 2024-06-29T14:14:51.908 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-29T14:14:51.908 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-29T14:14:52.909 DEBUG:teuthology.parallel:result is None 2024-06-29T14:15:23.198 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-06-29T14:15:23.610 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-06-29T14:15:23.611 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-467.el9.x86_64"... 2024-06-29T14:15:23.611 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-06-29T14:15:23.625 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-467.el9.x86_64 2024-06-29T14:15:23.626 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-467.el9.x86_64 vs 5.14.0-467.el9.x86_64 2024-06-29T14:15:23.626 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-29T14:15:23.626 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-29T14:15:24.627 DEBUG:teuthology.parallel:result is None 2024-06-29T14:15:24.628 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-29T14:15:24.635 INFO:teuthology.task.internal:Creating test directory... 2024-06-29T14:15:24.635 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-29T14:15:24.638 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-29T14:15:24.658 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-29T14:15:24.677 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-29T14:15:24.687 INFO:teuthology.task.internal:Creating archive directory... 2024-06-29T14:15:24.687 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-29T14:15:24.690 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-29T14:15:24.796 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-29T14:15:24.802 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-29T14:15:24.803 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-29T14:15:24.805 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-29T14:15:24.878 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-29T14:15:24.885 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-29T14:15:24.899 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-29T14:15:24.939 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-29T14:15:24.940 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-29T14:15:24.948 INFO:teuthology.task.internal:Configuring sudo... 2024-06-29T14:15:24.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-29T14:15:24.950 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-29T14:15:24.979 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-29T14:15:24.991 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-29T14:15:24.992 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-29T14:15:25.011 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-29T14:15:25.036 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-29T14:15:25.104 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-29T14:15:25.144 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-29T14:15:25.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-29T14:15:25.261 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:15:25.262 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-29T14:15:25.323 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-29T14:15:25.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-29T14:15:25.422 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-29T14:15:25.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-29T14:15:25.549 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:15:25.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-29T14:15:25.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-06-29T14:15:25.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2024-06-29T14:15:25.667 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-29T14:15:25.704 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-29T14:15:26.191 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-29T14:15:26.200 INFO:teuthology.task.internal:Starting timer... 2024-06-29T14:15:26.200 INFO:teuthology.run_tasks:Running task pcp... 2024-06-29T14:15:26.211 INFO:teuthology.run_tasks:Running task selinux... 2024-06-29T14:15:26.220 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-06-29T14:15:26.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2024-06-29T14:15:26.289 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2024-06-29T14:15:26.290 DEBUG:teuthology.orchestra.run.smithi167:> sudo service auditd rotate 2024-06-29T14:15:26.388 INFO:teuthology.orchestra.run.smithi167.stdout:Rotating logs: 2024-06-29T14:15:26.391 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-29T14:15:26.391 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2024-06-29T14:15:26.422 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2024-06-29T14:15:26.422 DEBUG:teuthology.orchestra.run.smithi167:> /usr/sbin/getenforce 2024-06-29T14:15:26.456 INFO:teuthology.orchestra.run.smithi167.stdout:Permissive 2024-06-29T14:15:26.457 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi081.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi167.front.sepia.ceph.com': 'permissive'} 2024-06-29T14:15:26.457 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-29T14:15:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:15:26.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-29T14:15:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:15:26.525 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-29T14:15:26.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2024-06-29T14:15:26.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo /usr/sbin/setenforce permissive 2024-06-29T14:15:26.593 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-29T14:15:26.603 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-29T14:15:26.606 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-29T14:15:26.782 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-29T14:15:26.793 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-29T14:15:26.794 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-29T14:19:45.232 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi167.front.sepia.ceph.com')] 2024-06-29T14:19:45.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-06-29T14:19:45.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:19:45.321 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-06-29T14:19:45.394 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-06-29T14:19:45.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-06-29T14:19:45.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-29T14:19:45.472 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-06-29T14:19:45.549 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-06-29T14:19:45.550 INFO:teuthology.run_tasks:Running task clock... 2024-06-29T14:19:45.561 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-29T14:19:45.562 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-29T14:19:45.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-29T14:19:45.565 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-29T14:19:45.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-29T14:19:45.592 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-29T14:19:45.606 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-29T14:19:45.618 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-29T14:19:45.631 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2024-06-29T14:19:45.632 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-29T14:19:45.641 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2024-06-29T14:19:45.655 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-29T14:19:45.658 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: ntpd: command not found 2024-06-29T14:19:45.667 INFO:teuthology.orchestra.run.smithi167.stdout:506 Cannot talk to daemon 2024-06-29T14:19:45.669 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-29T14:19:45.682 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-29T14:19:45.696 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-29T14:19:45.723 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-06-29T14:19:45.726 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-29T14:19:45.726 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-06-29T14:19:45.726 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:19:45.726 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:19:45.726 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:19:45.726 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:19:45.750 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-06-29T14:19:45.754 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-29T14:19:45.754 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-06-29T14:19:45.754 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:19:45.754 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:19:45.754 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:19:45.755 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T14:19:45.755 INFO:teuthology.run_tasks:Running task install... 2024-06-29T14:19:45.764 DEBUG:teuthology.task.install:project ceph 2024-06-29T14:19:45.764 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7'}} 2024-06-29T14:19:45.764 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7'} 2024-06-29T14:19:45.764 INFO:teuthology.task.install:Using flavor: default 2024-06-29T14:19:45.772 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-29T14:19:45.772 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-06-29T14:19:45.772 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7', 'tag': None, 'wait_for_package': False} 2024-06-29T14:19:45.773 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cf146c2f9dac0bd5aed66008255af2113db872f7 2024-06-29T14:19:45.775 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7', 'tag': None, 'wait_for_package': False} 2024-06-29T14:19:45.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cf146c2f9dac0bd5aed66008255af2113db872f7 2024-06-29T14:19:45.918 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/ 2024-06-29T14:19:45.918 INFO:teuthology.task.install.rpm:Package version is 19.0.0-4577.gcf146c2f 2024-06-29T14:19:45.923 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/ 2024-06-29T14:19:45.924 INFO:teuthology.task.install.rpm:Package version is 19.0.0-4577.gcf146c2f 2024-06-29T14:19:46.029 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-29T14:19:46.029 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:19:46.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-29T14:19:46.037 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-mchangir-qa-handle-correct-CephManager-instantiation-debug/cf146c2f9dac0bd5aed66008255af2113db872f7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-29T14:19:46.038 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:19:46.038 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-29T14:19:46.055 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-29T14:19:46.055 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cf146c2f9dac0bd5aed66008255af2113db872f7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-29T14:19:46.064 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-29T14:19:46.064 DEBUG:teuthology.orchestra.run.smithi167:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cf146c2f9dac0bd5aed66008255af2113db872f7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-29T14:19:46.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-29T14:19:46.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-29T14:19:46.195 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-29T14:19:46.205 DEBUG:teuthology.orchestra.run.smithi167:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-29T14:19:46.259 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2024-06-29T14:19:46.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2024-06-29T14:19:46.269 INFO:teuthology.orchestra.run.smithi167.stdout:check_obsoletes = 1 2024-06-29T14:19:46.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum clean all 2024-06-29T14:19:46.496 INFO:teuthology.orchestra.run.smithi081.stdout:45 files removed 2024-06-29T14:19:46.500 INFO:teuthology.orchestra.run.smithi167.stdout:45 files removed 2024-06-29T14:19:46.526 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-29T14:19:46.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-29T14:19:47.158 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 234 kB/s | 84 kB 00:00 2024-06-29T14:19:47.217 INFO:teuthology.orchestra.run.smithi167.stdout:ceph packages for x86_64 204 kB/s | 84 kB 00:00 2024-06-29T14:19:47.618 INFO:teuthology.orchestra.run.smithi167.stdout:ceph noarch packages 68 kB/s | 24 kB 00:00 2024-06-29T14:19:47.633 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 56 kB/s | 24 kB 00:00 2024-06-29T14:19:47.927 INFO:teuthology.orchestra.run.smithi167.stdout:ceph source packages 7.3 kB/s | 1.9 kB 00:00 2024-06-29T14:19:48.001 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-06-29T14:19:49.885 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.1 MB 00:01 2024-06-29T14:19:51.876 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.1 MB 00:03 2024-06-29T14:19:53.395 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 8.3 MB/s | 20 MB 00:02 2024-06-29T14:19:53.769 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-06-29T14:19:57.595 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.4 MB 00:00 2024-06-29T14:19:58.956 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.4 MB 00:01 2024-06-29T14:19:59.196 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 17 kB 00:00 2024-06-29T14:20:00.092 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-06-29T14:20:00.437 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 17 kB 00:00 2024-06-29T14:20:01.949 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-06-29T14:20:05.340 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-06-29T14:20:07.172 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-06-29T14:20:07.792 INFO:teuthology.orchestra.run.smithi167.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-29T14:20:07.792 INFO:teuthology.orchestra.run.smithi167.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-29T14:20:07.803 INFO:teuthology.orchestra.run.smithi167.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-29T14:20:07.804 INFO:teuthology.orchestra.run.smithi167.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-29T14:20:07.805 INFO:teuthology.orchestra.run.smithi167.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-29T14:20:07.806 INFO:teuthology.orchestra.run.smithi167.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-29T14:20:07.808 INFO:teuthology.orchestra.run.smithi167.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-29T14:20:07.810 INFO:teuthology.orchestra.run.smithi167.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-29T14:20:07.811 INFO:teuthology.orchestra.run.smithi167.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-29T14:20:07.876 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-06-29T14:20:07.895 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: ceph x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 6.6 k 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 5.4 M 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 866 k 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 147 k 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 1.5 M 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 143 k 2024-06-29T14:20:07.896 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 14 M 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 7.4 M 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 49 k 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 10 M 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 43 M 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 274 k 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 764 k 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: cephfs-mirror x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 227 k 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: cephfs-shell x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 27 k 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: cephfs-top noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 18 k 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-29T14:20:07.897 INFO:teuthology.orchestra.run.smithi167.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 32 k 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 759 k 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 127 k 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 161 k 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 321 k 2024-06-29T14:20:07.898 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 301 k 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 99 k 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 86 k 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.0 M 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 169 k 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: xfsprogs-devel x86_64 6.4.0-1.el9 appstream 95 k 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrading: 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: NetworkManager x86_64 1:1.48.0-1.el9 baseos 2.3 M 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: NetworkManager-libnm x86_64 1:1.48.0-1.el9 baseos 1.8 M 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: NetworkManager-team x86_64 1:1.48.0-1.el9 baseos 41 k 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: NetworkManager-tui x86_64 1:1.48.0-1.el9 baseos 248 k 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-29T14:20:07.899 INFO:teuthology.orchestra.run.smithi167.stdout: librados2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.3 M 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.1 M 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: xfsprogs x86_64 6.4.0-1.el9 baseos 1.1 M 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 20 M 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 34 k 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 2.3 M 2024-06-29T14:20:07.900 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 277 k 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 4.9 M 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 18 M 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 16 k 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 25 k 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-29T14:20:07.901 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 158 k 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 500 k 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 5.2 M 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-29T14:20:07.902 INFO:teuthology.orchestra.run.smithi167.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-29T14:20:07.903 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 45 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 136 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth noarch 1:2.30.0-1.el9 epel 213 k 2024-06-29T14:20:07.904 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-29T14:20:07.905 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-29T14:20:07.906 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-29T14:20:07.907 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-29T14:20:07.908 INFO:teuthology.orchestra.run.smithi167.stdout:Installing weak dependencies: 2024-06-29T14:20:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-29T14:20:07.909 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:20:07.909 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-06-29T14:20:07.909 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================= 2024-06-29T14:20:07.910 INFO:teuthology.orchestra.run.smithi167.stdout:Install 148 Packages 2024-06-29T14:20:07.910 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrade 12 Packages 2024-06-29T14:20:07.910 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:20:07.910 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 231 M 2024-06-29T14:20:07.910 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-06-29T14:20:08.625 INFO:teuthology.orchestra.run.smithi167.stdout:(1/160): ceph-19.0.0-4577.gcf146c2f.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-06-29T14:20:08.816 INFO:teuthology.orchestra.run.smithi167.stdout:(2/160): ceph-fuse-19.0.0-4577.gcf146c2f.el9.x8 4.4 MB/s | 866 kB 00:00 2024-06-29T14:20:08.892 INFO:teuthology.orchestra.run.smithi167.stdout:(3/160): ceph-immutable-object-cache-19.0.0-457 1.9 MB/s | 147 kB 00:00 2024-06-29T14:20:09.117 INFO:teuthology.orchestra.run.smithi167.stdout:(4/160): ceph-mds-19.0.0-4577.gcf146c2f.el9.x86 10 MB/s | 2.3 MB 00:00 2024-06-29T14:20:09.392 INFO:teuthology.orchestra.run.smithi167.stdout:(5/160): ceph-common-19.0.0-4577.gcf146c2f.el9. 21 MB/s | 20 MB 00:00 2024-06-29T14:20:09.434 INFO:teuthology.orchestra.run.smithi167.stdout:(6/160): ceph-mgr-19.0.0-4577.gcf146c2f.el9.x86 4.8 MB/s | 1.5 MB 00:00 2024-06-29T14:20:09.576 INFO:teuthology.orchestra.run.smithi167.stdout:(7/160): ceph-mon-19.0.0-4577.gcf146c2f.el9.x86 27 MB/s | 4.9 MB 00:00 2024-06-29T14:20:09.633 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-29T14:20:09.634 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-29T14:20:09.645 INFO:teuthology.orchestra.run.smithi081.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-29T14:20:09.645 INFO:teuthology.orchestra.run.smithi081.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-29T14:20:09.646 INFO:teuthology.orchestra.run.smithi081.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-29T14:20:09.647 INFO:teuthology.orchestra.run.smithi081.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-29T14:20:09.649 INFO:teuthology.orchestra.run.smithi081.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-29T14:20:09.651 INFO:teuthology.orchestra.run.smithi081.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-29T14:20:09.652 INFO:teuthology.orchestra.run.smithi081.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-29T14:20:09.716 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-06-29T14:20:09.737 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================= 2024-06-29T14:20:09.737 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-06-29T14:20:09.737 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================= 2024-06-29T14:20:09.738 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-06-29T14:20:09.738 INFO:teuthology.orchestra.run.smithi081.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-29T14:20:09.738 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 6.6 k 2024-06-29T14:20:09.738 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 5.4 M 2024-06-29T14:20:09.738 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 866 k 2024-06-29T14:20:09.739 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 147 k 2024-06-29T14:20:09.739 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 1.5 M 2024-06-29T14:20:09.739 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 143 k 2024-06-29T14:20:09.739 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 14 M 2024-06-29T14:20:09.739 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 7.4 M 2024-06-29T14:20:09.740 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 49 k 2024-06-29T14:20:09.740 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 10 M 2024-06-29T14:20:09.740 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 43 M 2024-06-29T14:20:09.740 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 274 k 2024-06-29T14:20:09.740 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 764 k 2024-06-29T14:20:09.741 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 227 k 2024-06-29T14:20:09.741 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-shell x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 27 k 2024-06-29T14:20:09.741 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-top noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 18 k 2024-06-29T14:20:09.741 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-29T14:20:09.741 INFO:teuthology.orchestra.run.smithi081.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-29T14:20:09.742 INFO:teuthology.orchestra.run.smithi081.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-29T14:20:09.742 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 32 k 2024-06-29T14:20:09.742 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 759 k 2024-06-29T14:20:09.742 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 127 k 2024-06-29T14:20:09.742 INFO:teuthology.orchestra.run.smithi081.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-06-29T14:20:09.742 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-29T14:20:09.743 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-06-29T14:20:09.743 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-29T14:20:09.743 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-29T14:20:09.743 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-29T14:20:09.743 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 161 k 2024-06-29T14:20:09.744 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 321 k 2024-06-29T14:20:09.744 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 301 k 2024-06-29T14:20:09.744 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 99 k 2024-06-29T14:20:09.744 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 86 k 2024-06-29T14:20:09.744 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.0 M 2024-06-29T14:20:09.745 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 169 k 2024-06-29T14:20:09.745 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-devel x86_64 6.4.0-1.el9 appstream 95 k 2024-06-29T14:20:09.745 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-06-29T14:20:09.745 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager x86_64 1:1.48.0-1.el9 baseos 2.3 M 2024-06-29T14:20:09.745 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-libnm x86_64 1:1.48.0-1.el9 baseos 1.8 M 2024-06-29T14:20:09.745 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-team x86_64 1:1.48.0-1.el9 baseos 41 k 2024-06-29T14:20:09.746 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-tui x86_64 1:1.48.0-1.el9 baseos 248 k 2024-06-29T14:20:09.746 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-29T14:20:09.746 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-29T14:20:09.746 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-29T14:20:09.746 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.3 M 2024-06-29T14:20:09.747 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 3.1 M 2024-06-29T14:20:09.747 INFO:teuthology.orchestra.run.smithi081.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-06-29T14:20:09.747 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-06-29T14:20:09.747 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs x86_64 6.4.0-1.el9 baseos 1.1 M 2024-06-29T14:20:09.747 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-06-29T14:20:09.747 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-29T14:20:09.748 INFO:teuthology.orchestra.run.smithi081.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-29T14:20:09.748 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-29T14:20:09.748 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 20 M 2024-06-29T14:20:09.748 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 34 k 2024-06-29T14:20:09.749 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 2.3 M 2024-06-29T14:20:09.749 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 277 k 2024-06-29T14:20:09.749 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 4.9 M 2024-06-29T14:20:09.749 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 18 M 2024-06-29T14:20:09.749 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:19.0.0-4577.gcf146c2f.el9 ceph-noarch 16 k 2024-06-29T14:20:09.749 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 25 k 2024-06-29T14:20:09.749 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-29T14:20:09.749 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-29T14:20:09.749 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-29T14:20:09.749 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 158 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 500 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-29T14:20:09.750 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 5.2 M 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-29T14:20:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 45 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:19.0.0-4577.gcf146c2f.el9 ceph 136 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-29T14:20:09.752 INFO:teuthology.orchestra.run.smithi081.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:2.30.0-1.el9 epel 213 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-29T14:20:09.753 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-29T14:20:09.754 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-29T14:20:09.755 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-29T14:20:09.756 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout:Installing weak dependencies: 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================= 2024-06-29T14:20:09.757 INFO:teuthology.orchestra.run.smithi081.stdout:Install 148 Packages 2024-06-29T14:20:09.758 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 12 Packages 2024-06-29T14:20:09.758 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:20:09.758 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 231 M 2024-06-29T14:20:09.758 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-06-29T14:20:09.968 INFO:teuthology.orchestra.run.smithi167.stdout:(8/160): ceph-radosgw-19.0.0-4577.gcf146c2f.el9 26 MB/s | 10 MB 00:00 2024-06-29T14:20:10.044 INFO:teuthology.orchestra.run.smithi167.stdout:(9/160): ceph-base-19.0.0-4577.gcf146c2f.el9.x8 3.4 MB/s | 5.4 MB 00:01 2024-06-29T14:20:10.069 INFO:teuthology.orchestra.run.smithi167.stdout:(10/160): ceph-selinux-19.0.0-4577.gcf146c2f.el 250 kB/s | 25 kB 00:00 2024-06-29T14:20:10.119 INFO:teuthology.orchestra.run.smithi167.stdout:(11/160): cephfs-mirror-19.0.0-4577.gcf146c2f.e 4.4 MB/s | 227 kB 00:00 2024-06-29T14:20:10.178 INFO:teuthology.orchestra.run.smithi167.stdout:(12/160): cephfs-shell-19.0.0-4577.gcf146c2f.el 465 kB/s | 27 kB 00:00 2024-06-29T14:20:10.237 INFO:teuthology.orchestra.run.smithi167.stdout:(13/160): libcephfs-devel-19.0.0-4577.gcf146c2f 545 kB/s | 32 kB 00:00 2024-06-29T14:20:10.424 INFO:teuthology.orchestra.run.smithi081.stdout:(1/160): ceph-19.0.0-4577.gcf146c2f.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-06-29T14:20:10.437 INFO:teuthology.orchestra.run.smithi167.stdout:(14/160): ceph-osd-19.0.0-4577.gcf146c2f.el9.x8 18 MB/s | 18 MB 00:01 2024-06-29T14:20:10.471 INFO:teuthology.orchestra.run.smithi167.stdout:(15/160): libcephfs2-19.0.0-4577.gcf146c2f.el9. 3.2 MB/s | 759 kB 00:00 2024-06-29T14:20:10.512 INFO:teuthology.orchestra.run.smithi167.stdout:(16/160): libcephsqlite-19.0.0-4577.gcf146c2f.e 2.1 MB/s | 158 kB 00:00 2024-06-29T14:20:10.538 INFO:teuthology.orchestra.run.smithi167.stdout:(17/160): librados-devel-19.0.0-4577.gcf146c2f. 1.9 MB/s | 127 kB 00:00 2024-06-29T14:20:10.596 INFO:teuthology.orchestra.run.smithi167.stdout:(18/160): libradosstriper1-19.0.0-4577.gcf146c2 5.8 MB/s | 500 kB 00:00 2024-06-29T14:20:10.663 INFO:teuthology.orchestra.run.smithi167.stdout:(19/160): python3-ceph-argparse-19.0.0-4577.gcf 679 kB/s | 45 kB 00:00 2024-06-29T14:20:10.666 INFO:teuthology.orchestra.run.smithi081.stdout:(2/160): ceph-base-19.0.0-4577.gcf146c2f.el9.x8 11 MB/s | 5.4 MB 00:00 2024-06-29T14:20:10.725 INFO:teuthology.orchestra.run.smithi081.stdout:(3/160): ceph-immutable-object-cache-19.0.0-457 2.5 MB/s | 147 kB 00:00 2024-06-29T14:20:10.755 INFO:teuthology.orchestra.run.smithi167.stdout:(20/160): librgw2-19.0.0-4577.gcf146c2f.el9.x86 24 MB/s | 5.2 MB 00:00 2024-06-29T14:20:10.781 INFO:teuthology.orchestra.run.smithi167.stdout:(21/160): python3-ceph-common-19.0.0-4577.gcf14 1.1 MB/s | 136 kB 00:00 2024-06-29T14:20:10.814 INFO:teuthology.orchestra.run.smithi167.stdout:(22/160): python3-cephfs-19.0.0-4577.gcf146c2f. 2.7 MB/s | 161 kB 00:00 2024-06-29T14:20:10.816 INFO:teuthology.orchestra.run.smithi081.stdout:(4/160): ceph-mds-19.0.0-4577.gcf146c2f.el9.x86 25 MB/s | 2.3 MB 00:00 2024-06-29T14:20:10.840 INFO:teuthology.orchestra.run.smithi167.stdout:(23/160): python3-rados-19.0.0-4577.gcf146c2f.e 5.3 MB/s | 321 kB 00:00 2024-06-29T14:20:10.865 INFO:teuthology.orchestra.run.smithi167.stdout:(24/160): python3-rbd-19.0.0-4577.gcf146c2f.el9 5.9 MB/s | 301 kB 00:00 2024-06-29T14:20:10.899 INFO:teuthology.orchestra.run.smithi167.stdout:(25/160): python3-rgw-19.0.0-4577.gcf146c2f.el9 1.7 MB/s | 99 kB 00:00 2024-06-29T14:20:10.908 INFO:teuthology.orchestra.run.smithi081.stdout:(5/160): ceph-mgr-19.0.0-4577.gcf146c2f.el9.x86 16 MB/s | 1.5 MB 00:00 2024-06-29T14:20:10.924 INFO:teuthology.orchestra.run.smithi167.stdout:(26/160): rbd-fuse-19.0.0-4577.gcf146c2f.el9.x8 1.4 MB/s | 86 kB 00:00 2024-06-29T14:20:11.009 INFO:teuthology.orchestra.run.smithi081.stdout:(6/160): ceph-fuse-19.0.0-4577.gcf146c2f.el9.x8 1.4 MB/s | 866 kB 00:00 2024-06-29T14:20:11.041 INFO:teuthology.orchestra.run.smithi167.stdout:(27/160): rbd-nbd-19.0.0-4577.gcf146c2f.el9.x86 1.4 MB/s | 169 kB 00:00 2024-06-29T14:20:11.116 INFO:teuthology.orchestra.run.smithi167.stdout:(28/160): ceph-grafana-dashboards-19.0.0-4577.g 457 kB/s | 34 kB 00:00 2024-06-29T14:20:11.159 INFO:teuthology.orchestra.run.smithi081.stdout:(7/160): ceph-mon-19.0.0-4577.gcf146c2f.el9.x86 20 MB/s | 4.9 MB 00:00 2024-06-29T14:20:11.175 INFO:teuthology.orchestra.run.smithi167.stdout:(29/160): rbd-mirror-19.0.0-4577.gcf146c2f.el9. 11 MB/s | 3.0 MB 00:00 2024-06-29T14:20:11.200 INFO:teuthology.orchestra.run.smithi167.stdout:(30/160): ceph-mgr-cephadm-19.0.0-4577.gcf146c2 1.7 MB/s | 143 kB 00:00 2024-06-29T14:20:11.642 INFO:teuthology.orchestra.run.smithi167.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 17 MB/s | 7.4 MB 00:00 2024-06-29T14:20:11.726 INFO:teuthology.orchestra.run.smithi167.stdout:(32/160): ceph-mgr-modules-core-19.0.0-4577.gcf 3.3 MB/s | 277 kB 00:00 2024-06-29T14:20:11.776 INFO:teuthology.orchestra.run.smithi167.stdout:(33/160): ceph-mgr-rook-19.0.0-4577.gcf146c2f.e 987 kB/s | 49 kB 00:00 2024-06-29T14:20:11.834 INFO:teuthology.orchestra.run.smithi167.stdout:(34/160): ceph-prometheus-alerts-19.0.0-4577.gc 283 kB/s | 16 kB 00:00 2024-06-29T14:20:11.859 INFO:teuthology.orchestra.run.smithi081.stdout:(8/160): ceph-radosgw-19.0.0-4577.gcf146c2f.el9 15 MB/s | 10 MB 00:00 2024-06-29T14:20:11.901 INFO:teuthology.orchestra.run.smithi081.stdout:(9/160): ceph-selinux-19.0.0-4577.gcf146c2f.el9 609 kB/s | 25 kB 00:00 2024-06-29T14:20:11.902 INFO:teuthology.orchestra.run.smithi167.stdout:(35/160): ceph-volume-19.0.0-4577.gcf146c2f.el9 4.0 MB/s | 274 kB 00:00 2024-06-29T14:20:11.994 INFO:teuthology.orchestra.run.smithi167.stdout:(36/160): cephadm-19.0.0-4577.gcf146c2f.el9.noa 8.1 MB/s | 764 kB 00:00 2024-06-29T14:20:12.069 INFO:teuthology.orchestra.run.smithi167.stdout:(37/160): cephfs-top-19.0.0-4577.gcf146c2f.el9. 243 kB/s | 18 kB 00:00 2024-06-29T14:20:12.336 INFO:teuthology.orchestra.run.smithi167.stdout:(38/160): ceph-mgr-dashboard-19.0.0-4577.gcf146 12 MB/s | 14 MB 00:01 2024-06-29T14:20:12.353 INFO:teuthology.orchestra.run.smithi167.stdout:(39/160): fuse-2.9.9-15.el9.x86_64.rpm 283 kB/s | 80 kB 00:00 2024-06-29T14:20:12.554 INFO:teuthology.orchestra.run.smithi167.stdout:(40/160): libquadmath-11.4.1-3.el9.x86_64.rpm 976 kB/s | 195 kB 00:00 2024-06-29T14:20:12.587 INFO:teuthology.orchestra.run.smithi167.stdout:(41/160): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-06-29T14:20:12.754 INFO:teuthology.orchestra.run.smithi167.stdout:(42/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-06-29T14:20:12.888 INFO:teuthology.orchestra.run.smithi167.stdout:(43/160): python3-cryptography-36.0.1-4.el9.x86 4.1 MB/s | 1.2 MB 00:00 2024-06-29T14:20:12.946 INFO:teuthology.orchestra.run.smithi167.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-06-29T14:20:13.046 INFO:teuthology.orchestra.run.smithi167.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 853 kB/s | 135 kB 00:00 2024-06-29T14:20:13.114 INFO:teuthology.orchestra.run.smithi167.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 748 kB/s | 125 kB 00:00 2024-06-29T14:20:13.568 INFO:teuthology.orchestra.run.smithi081.stdout:(10/160): ceph-osd-19.0.0-4577.gcf146c2f.el9.x8 6.9 MB/s | 18 MB 00:02 2024-06-29T14:20:13.572 INFO:teuthology.orchestra.run.smithi167.stdout:(47/160): ceph-test-19.0.0-4577.gcf146c2f.el9.x 12 MB/s | 43 MB 00:03 2024-06-29T14:20:13.606 INFO:teuthology.orchestra.run.smithi167.stdout:(48/160): python3-urllib3-1.26.5-5.el9.noarch.r 384 kB/s | 215 kB 00:00 2024-06-29T14:20:13.635 INFO:teuthology.orchestra.run.smithi081.stdout:(11/160): cephfs-mirror-19.0.0-4577.gcf146c2f.e 3.3 MB/s | 227 kB 00:00 2024-06-29T14:20:14.035 INFO:teuthology.orchestra.run.smithi081.stdout:(12/160): ceph-test-19.0.0-4577.gcf146c2f.el9.x 20 MB/s | 43 MB 00:02 2024-06-29T14:20:14.052 INFO:teuthology.orchestra.run.smithi081.stdout:(13/160): cephfs-shell-19.0.0-4577.gcf146c2f.el 65 kB/s | 27 kB 00:00 2024-06-29T14:20:14.077 INFO:teuthology.orchestra.run.smithi081.stdout:(14/160): libcephfs-devel-19.0.0-4577.gcf146c2f 763 kB/s | 32 kB 00:00 2024-06-29T14:20:14.128 INFO:teuthology.orchestra.run.smithi081.stdout:(15/160): libcephsqlite-19.0.0-4577.gcf146c2f.e 3.1 MB/s | 158 kB 00:00 2024-06-29T14:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout:(16/160): libcephfs2-19.0.0-4577.gcf146c2f.el9. 6.3 MB/s | 759 kB 00:00 2024-06-29T14:20:14.195 INFO:teuthology.orchestra.run.smithi081.stdout:(17/160): librados-devel-19.0.0-4577.gcf146c2f. 1.8 MB/s | 127 kB 00:00 2024-06-29T14:20:14.245 INFO:teuthology.orchestra.run.smithi081.stdout:(18/160): libradosstriper1-19.0.0-4577.gcf146c2 6.5 MB/s | 500 kB 00:00 2024-06-29T14:20:14.315 INFO:teuthology.orchestra.run.smithi167.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 151 kB/s | 107 kB 00:00 2024-06-29T14:20:14.329 INFO:teuthology.orchestra.run.smithi081.stdout:(19/160): python3-ceph-argparse-19.0.0-4577.gcf 542 kB/s | 45 kB 00:00 2024-06-29T14:20:14.396 INFO:teuthology.orchestra.run.smithi081.stdout:(20/160): python3-ceph-common-19.0.0-4577.gcf14 2.0 MB/s | 136 kB 00:00 2024-06-29T14:20:14.431 INFO:teuthology.orchestra.run.smithi167.stdout:(50/160): automake-1.16.2-8.el9.noarch.rpm 523 kB/s | 689 kB 00:01 2024-06-29T14:20:14.480 INFO:teuthology.orchestra.run.smithi081.stdout:(21/160): librgw2-19.0.0-4577.gcf146c2f.el9.x86 18 MB/s | 5.2 MB 00:00 2024-06-29T14:20:14.505 INFO:teuthology.orchestra.run.smithi081.stdout:(22/160): python3-cephfs-19.0.0-4577.gcf146c2f. 1.5 MB/s | 161 kB 00:00 2024-06-29T14:20:14.530 INFO:teuthology.orchestra.run.smithi081.stdout:(23/160): python3-rados-19.0.0-4577.gcf146c2f.e 6.3 MB/s | 321 kB 00:00 2024-06-29T14:20:14.540 INFO:teuthology.orchestra.run.smithi167.stdout:(51/160): bison-3.7.4-5.el9.x86_64.rpm 975 kB/s | 943 kB 00:00 2024-06-29T14:20:14.572 INFO:teuthology.orchestra.run.smithi081.stdout:(24/160): python3-rbd-19.0.0-4577.gcf146c2f.el9 4.4 MB/s | 301 kB 00:00 2024-06-29T14:20:14.597 INFO:teuthology.orchestra.run.smithi081.stdout:(25/160): python3-rgw-19.0.0-4577.gcf146c2f.el9 1.4 MB/s | 99 kB 00:00 2024-06-29T14:20:14.639 INFO:teuthology.orchestra.run.smithi081.stdout:(26/160): rbd-fuse-19.0.0-4577.gcf146c2f.el9.x8 1.2 MB/s | 86 kB 00:00 2024-06-29T14:20:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:(27/160): rbd-nbd-19.0.0-4577.gcf146c2f.el9.x86 2.5 MB/s | 169 kB 00:00 2024-06-29T14:20:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:(28/160): rbd-mirror-19.0.0-4577.gcf146c2f.el9. 19 MB/s | 3.0 MB 00:00 2024-06-29T14:20:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:(29/160): ceph-grafana-dashboards-19.0.0-4577.g 454 kB/s | 34 kB 00:00 2024-06-29T14:20:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:(30/160): ceph-mgr-cephadm-19.0.0-4577.gcf146c2 2.8 MB/s | 143 kB 00:00 2024-06-29T14:20:14.899 INFO:teuthology.orchestra.run.smithi167.stdout:(52/160): elfutils-libelf-devel-0.191-4.el9.x86 37 kB/s | 22 kB 00:00 2024-06-29T14:20:15.125 INFO:teuthology.orchestra.run.smithi081.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 23 MB/s | 7.4 MB 00:00 2024-06-29T14:20:15.174 INFO:teuthology.orchestra.run.smithi167.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-06-29T14:20:15.183 INFO:teuthology.orchestra.run.smithi081.stdout:(32/160): ceph-mgr-modules-core-19.0.0-4577.gcf 4.7 MB/s | 277 kB 00:00 2024-06-29T14:20:15.233 INFO:teuthology.orchestra.run.smithi081.stdout:(33/160): ceph-mgr-rook-19.0.0-4577.gcf146c2f.e 990 kB/s | 49 kB 00:00 2024-06-29T14:20:15.275 INFO:teuthology.orchestra.run.smithi081.stdout:(34/160): ceph-prometheus-alerts-19.0.0-4577.gc 397 kB/s | 16 kB 00:00 2024-06-29T14:20:15.282 INFO:teuthology.orchestra.run.smithi167.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 369 kB/s | 313 kB 00:00 2024-06-29T14:20:15.326 INFO:teuthology.orchestra.run.smithi081.stdout:(35/160): ceph-volume-19.0.0-4577.gcf146c2f.el9 5.4 MB/s | 274 kB 00:00 2024-06-29T14:20:15.401 INFO:teuthology.orchestra.run.smithi081.stdout:(36/160): cephadm-19.0.0-4577.gcf146c2f.el9.noa 10 MB/s | 764 kB 00:00 2024-06-29T14:20:15.443 INFO:teuthology.orchestra.run.smithi081.stdout:(37/160): cephfs-top-19.0.0-4577.gcf146c2f.el9. 437 kB/s | 18 kB 00:00 2024-06-29T14:20:15.585 INFO:teuthology.orchestra.run.smithi081.stdout:(38/160): ceph-mgr-dashboard-19.0.0-4577.gcf146 17 MB/s | 14 MB 00:00 2024-06-29T14:20:15.749 INFO:teuthology.orchestra.run.smithi167.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 32 kB/s | 18 kB 00:00 2024-06-29T14:20:15.791 INFO:teuthology.orchestra.run.smithi167.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 22 kB/s | 11 kB 00:00 2024-06-29T14:20:16.019 INFO:teuthology.orchestra.run.smithi081.stdout:(39/160): fuse-2.9.9-15.el9.x86_64.rpm 140 kB/s | 80 kB 00:00 2024-06-29T14:20:16.058 INFO:teuthology.orchestra.run.smithi167.stdout:(57/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.6 MB/s | 3.0 MB 00:01 2024-06-29T14:20:16.252 INFO:teuthology.orchestra.run.smithi081.stdout:(40/160): libquadmath-11.4.1-3.el9.x86_64.rpm 837 kB/s | 195 kB 00:00 2024-06-29T14:20:16.369 INFO:teuthology.orchestra.run.smithi081.stdout:(41/160): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-06-29T14:20:16.411 INFO:teuthology.orchestra.run.smithi081.stdout:(42/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-06-29T14:20:16.484 INFO:teuthology.orchestra.run.smithi167.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 65 kB/s | 45 kB 00:00 2024-06-29T14:20:16.534 INFO:teuthology.orchestra.run.smithi167.stdout:(59/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 204 kB/s | 160 kB 00:00 2024-06-29T14:20:16.653 INFO:teuthology.orchestra.run.smithi081.stdout:(43/160): python3-cryptography-36.0.1-4.el9.x86 4.3 MB/s | 1.2 MB 00:00 2024-06-29T14:20:16.678 INFO:teuthology.orchestra.run.smithi081.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 399 kB/s | 106 kB 00:00 2024-06-29T14:20:16.762 INFO:teuthology.orchestra.run.smithi081.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-06-29T14:20:16.787 INFO:teuthology.orchestra.run.smithi081.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-06-29T14:20:16.879 INFO:teuthology.orchestra.run.smithi081.stdout:(47/160): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-06-29T14:20:17.009 INFO:teuthology.orchestra.run.smithi167.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 717 kB/s | 662 kB 00:00 2024-06-29T14:20:17.038 INFO:teuthology.orchestra.run.smithi081.stdout:(48/160): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 689 kB 00:00 2024-06-29T14:20:17.089 INFO:teuthology.orchestra.run.smithi081.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-06-29T14:20:17.110 INFO:teuthology.orchestra.run.smithi167.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 44 kB/s | 25 kB 00:00 2024-06-29T14:20:17.147 INFO:teuthology.orchestra.run.smithi081.stdout:(50/160): elfutils-libelf-devel-0.191-4.el9.x86 372 kB/s | 22 kB 00:00 2024-06-29T14:20:17.172 INFO:teuthology.orchestra.run.smithi081.stdout:(51/160): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-06-29T14:20:17.214 INFO:teuthology.orchestra.run.smithi081.stdout:(52/160): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-06-29T14:20:17.240 INFO:teuthology.orchestra.run.smithi081.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 489 kB/s | 33 kB 00:00 2024-06-29T14:20:17.310 INFO:teuthology.orchestra.run.smithi081.stdout:(54/160): flexiblas-openblas-openmp-3.0.4-8.el9 272 kB/s | 18 kB 00:00 2024-06-29T14:20:17.366 INFO:teuthology.orchestra.run.smithi081.stdout:(55/160): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-06-29T14:20:17.432 INFO:teuthology.orchestra.run.smithi081.stdout:(56/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-06-29T14:20:17.435 INFO:teuthology.orchestra.run.smithi167.stdout:(62/160): libtool-2.4.6-46.el9.x86_64.rpm 611 kB/s | 581 kB 00:00 2024-06-29T14:20:17.491 INFO:teuthology.orchestra.run.smithi081.stdout:(57/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-06-29T14:20:17.508 INFO:teuthology.orchestra.run.smithi081.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 601 kB/s | 45 kB 00:00 2024-06-29T14:20:17.575 INFO:teuthology.orchestra.run.smithi081.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 662 kB 00:00 2024-06-29T14:20:17.600 INFO:teuthology.orchestra.run.smithi081.stdout:(60/160): libtool-2.4.6-46.el9.x86_64.rpm 6.2 MB/s | 581 kB 00:00 2024-06-29T14:20:17.634 INFO:teuthology.orchestra.run.smithi081.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-06-29T14:20:17.659 INFO:teuthology.orchestra.run.smithi081.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 807 kB/s | 47 kB 00:00 2024-06-29T14:20:17.701 INFO:teuthology.orchestra.run.smithi081.stdout:(63/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-06-29T14:20:17.726 INFO:teuthology.orchestra.run.smithi081.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-06-29T14:20:17.735 INFO:teuthology.orchestra.run.smithi167.stdout:(63/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 65 kB/s | 47 kB 00:00 2024-06-29T14:20:17.760 INFO:teuthology.orchestra.run.smithi081.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 35 kB 00:00 2024-06-29T14:20:18.102 INFO:teuthology.orchestra.run.smithi081.stdout:(66/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-06-29T14:20:18.111 INFO:teuthology.orchestra.run.smithi167.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 292 kB/s | 292 kB 00:01 2024-06-29T14:20:18.178 INFO:teuthology.orchestra.run.smithi081.stdout:(67/160): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2024-06-29T14:20:18.278 INFO:teuthology.orchestra.run.smithi081.stdout:(68/160): postgresql-13.14-1.el9.x86_64.rpm 9.1 MB/s | 1.6 MB 00:00 2024-06-29T14:20:18.303 INFO:teuthology.orchestra.run.smithi081.stdout:(69/160): postgresql-contrib-13.14-1.el9.x86_64 7.1 MB/s | 911 kB 00:00 2024-06-29T14:20:18.353 INFO:teuthology.orchestra.run.smithi081.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-06-29T14:20:18.411 INFO:teuthology.orchestra.run.smithi167.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 193 kB/s | 188 kB 00:00 2024-06-29T14:20:18.445 INFO:teuthology.orchestra.run.smithi081.stdout:(71/160): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-06-29T14:20:18.521 INFO:teuthology.orchestra.run.smithi081.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-06-29T14:20:18.544 INFO:teuthology.orchestra.run.smithi167.stdout:(66/160): openblas-0.3.21-2.el9.x86_64.rpm 44 kB/s | 35 kB 00:00 2024-06-29T14:20:18.704 INFO:teuthology.orchestra.run.smithi081.stdout:(73/160): ceph-common-19.0.0-4577.gcf146c2f.el9 2.3 MB/s | 20 MB 00:08 2024-06-29T14:20:18.788 INFO:teuthology.orchestra.run.smithi081.stdout:(74/160): postgresql-server-13.14-1.el9.x86_64. 12 MB/s | 5.8 MB 00:00 2024-06-29T14:20:18.846 INFO:teuthology.orchestra.run.smithi081.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-06-29T14:20:18.955 INFO:teuthology.orchestra.run.smithi081.stdout:(76/160): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-06-29T14:20:18.981 INFO:teuthology.orchestra.run.smithi081.stdout:(77/160): python3-colorama-0.4.6-3.el9.noarch.r 222 kB/s | 61 kB 00:00 2024-06-29T14:20:19.006 INFO:teuthology.orchestra.run.smithi081.stdout:(78/160): python3-jinja2-2.11.3-6.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-06-29T14:20:19.031 INFO:teuthology.orchestra.run.smithi081.stdout:(79/160): python3-jsonpatch-1.21-16.el9.noarch. 340 kB/s | 26 kB 00:00 2024-06-29T14:20:19.056 INFO:teuthology.orchestra.run.smithi081.stdout:(80/160): python3-jsonpointer-2.0-4.el9.noarch. 254 kB/s | 19 kB 00:00 2024-06-29T14:20:19.082 INFO:teuthology.orchestra.run.smithi081.stdout:(81/160): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-06-29T14:20:19.107 INFO:teuthology.orchestra.run.smithi081.stdout:(82/160): python3-markupsafe-1.1.1-12.el9.x86_6 457 kB/s | 35 kB 00:00 2024-06-29T14:20:19.141 INFO:teuthology.orchestra.run.smithi081.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-06-29T14:20:19.166 INFO:teuthology.orchestra.run.smithi081.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-06-29T14:20:19.200 INFO:teuthology.orchestra.run.smithi081.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-06-29T14:20:19.225 INFO:teuthology.orchestra.run.smithi081.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 719 kB/s | 42 kB 00:00 2024-06-29T14:20:19.258 INFO:teuthology.orchestra.run.smithi081.stdout:(87/160): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-06-29T14:20:19.275 INFO:teuthology.orchestra.run.smithi081.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-06-29T14:20:19.318 INFO:teuthology.orchestra.run.smithi081.stdout:(89/160): python3-pyasn1-modules-0.4.8-6.el9.no 4.6 MB/s | 279 kB 00:00 2024-06-29T14:20:19.343 INFO:teuthology.orchestra.run.smithi081.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 760 kB/s | 51 kB 00:00 2024-06-29T14:20:19.377 INFO:teuthology.orchestra.run.smithi081.stdout:(91/160): python3-requests-oauthlib-1.3.0-12.el 919 kB/s | 54 kB 00:00 2024-06-29T14:20:19.443 INFO:teuthology.orchestra.run.smithi081.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 628 kB/s | 42 kB 00:00 2024-06-29T14:20:19.445 INFO:teuthology.orchestra.run.smithi167.stdout:(67/160): openblas-openmp-0.3.21-2.el9.x86_64.r 3.6 MB/s | 4.7 MB 00:01 2024-06-29T14:20:19.527 INFO:teuthology.orchestra.run.smithi081.stdout:(93/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-06-29T14:20:19.552 INFO:teuthology.orchestra.run.smithi081.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 406 kB/s | 44 kB 00:00 2024-06-29T14:20:19.594 INFO:teuthology.orchestra.run.smithi081.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-06-29T14:20:19.619 INFO:teuthology.orchestra.run.smithi081.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 535 kB/s | 36 kB 00:00 2024-06-29T14:20:19.661 INFO:teuthology.orchestra.run.smithi081.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 707 kB/s | 47 kB 00:00 2024-06-29T14:20:19.695 INFO:teuthology.orchestra.run.smithi081.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 3.9 MB/s | 304 kB 00:00 2024-06-29T14:20:19.706 INFO:teuthology.orchestra.run.smithi167.stdout:(68/160): postgresql-13.14-1.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-06-29T14:20:19.737 INFO:teuthology.orchestra.run.smithi081.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 770 kB/s | 58 kB 00:00 2024-06-29T14:20:19.779 INFO:teuthology.orchestra.run.smithi081.stdout:(100/160): xfsprogs-devel-6.4.0-1.el9.x86_64.rp 1.1 MB/s | 95 kB 00:00 2024-06-29T14:20:19.804 INFO:teuthology.orchestra.run.smithi081.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 949 kB/s | 64 kB 00:00 2024-06-29T14:20:19.812 INFO:teuthology.orchestra.run.smithi167.stdout:(69/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 3.2 MB/s | 4.4 MB 00:01 2024-06-29T14:20:19.855 INFO:teuthology.orchestra.run.smithi081.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 596 kB/s | 45 kB 00:00 2024-06-29T14:20:20.080 INFO:teuthology.orchestra.run.smithi081.stdout:(103/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-06-29T14:20:20.105 INFO:teuthology.orchestra.run.smithi081.stdout:(104/160): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-06-29T14:20:20.197 INFO:teuthology.orchestra.run.smithi081.stdout:(105/160): abseil-cpp-20211102.0-3.el9.x86_64.r 4.6 MB/s | 548 kB 00:00 2024-06-29T14:20:20.214 INFO:teuthology.orchestra.run.smithi081.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-06-29T14:20:20.239 INFO:teuthology.orchestra.run.smithi081.stdout:(107/160): gperftools-libs-2.9.1-2.el9.x86_64.r 2.3 MB/s | 309 kB 00:00 2024-06-29T14:20:20.256 INFO:teuthology.orchestra.run.smithi081.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.5 MB/s | 26 kB 00:00 2024-06-29T14:20:20.282 INFO:teuthology.orchestra.run.smithi081.stdout:(109/160): protobuf-compiler-3.14.0-13.el9.x86_ 2.0 MB/s | 863 kB 00:00 2024-06-29T14:20:20.307 INFO:teuthology.orchestra.run.smithi081.stdout:(110/160): liboath-2.6.7-2.el9.x86_64.rpm 960 kB/s | 49 kB 00:00 2024-06-29T14:20:20.357 INFO:teuthology.orchestra.run.smithi081.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 5.8 MB/s | 151 kB 00:00 2024-06-29T14:20:20.391 INFO:teuthology.orchestra.run.smithi081.stdout:(112/160): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-06-29T14:20:20.433 INFO:teuthology.orchestra.run.smithi081.stdout:(113/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-06-29T14:20:20.458 INFO:teuthology.orchestra.run.smithi081.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-06-29T14:20:20.484 INFO:teuthology.orchestra.run.smithi081.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-06-29T14:20:20.504 INFO:teuthology.orchestra.run.smithi167.stdout:(70/160): postgresql-contrib-13.14-1.el9.x86_64 861 kB/s | 911 kB 00:01 2024-06-29T14:20:20.509 INFO:teuthology.orchestra.run.smithi081.stdout:(116/160): python3-certifi-2021.10.8-2.el9.noar 602 kB/s | 15 kB 00:00 2024-06-29T14:20:20.529 INFO:teuthology.orchestra.run.smithi167.stdout:(71/160): postgresql-private-libs-13.14-1.el9.x 168 kB/s | 133 kB 00:00 2024-06-29T14:20:20.542 INFO:teuthology.orchestra.run.smithi081.stdout:(117/160): python3-asyncssh-2.13.2-5.el9.noarch 3.5 MB/s | 548 kB 00:00 2024-06-29T14:20:20.568 INFO:teuthology.orchestra.run.smithi081.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-06-29T14:20:20.601 INFO:teuthology.orchestra.run.smithi081.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-06-29T14:20:20.627 INFO:teuthology.orchestra.run.smithi081.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.0 MB/s | 303 kB 00:00 2024-06-29T14:20:20.652 INFO:teuthology.orchestra.run.smithi081.stdout:(121/160): python3-google-auth-2.30.0-1.el9.noa 4.1 MB/s | 213 kB 00:00 2024-06-29T14:20:20.686 INFO:teuthology.orchestra.run.smithi081.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-06-29T14:20:20.711 INFO:teuthology.orchestra.run.smithi081.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 428 kB/s | 11 kB 00:00 2024-06-29T14:20:20.736 INFO:teuthology.orchestra.run.smithi081.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 704 kB/s | 18 kB 00:00 2024-06-29T14:20:20.762 INFO:teuthology.orchestra.run.smithi081.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 924 kB/s | 23 kB 00:00 2024-06-29T14:20:20.829 INFO:teuthology.orchestra.run.smithi081.stdout:(126/160): libarrow-9.0.0-11.el9.x86_64.rpm 7.2 MB/s | 4.4 MB 00:00 2024-06-29T14:20:20.854 INFO:teuthology.orchestra.run.smithi081.stdout:(127/160): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-06-29T14:20:20.879 INFO:teuthology.orchestra.run.smithi081.stdout:(128/160): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-06-29T14:20:20.905 INFO:teuthology.orchestra.run.smithi081.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-06-29T14:20:20.930 INFO:teuthology.orchestra.run.smithi081.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-06-29T14:20:20.955 INFO:teuthology.orchestra.run.smithi081.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-06-29T14:20:20.997 INFO:teuthology.orchestra.run.smithi081.stdout:(132/160): python3-grpcio-1.46.7-10.el9.x86_64. 5.5 MB/s | 2.0 MB 00:00 2024-06-29T14:20:21.023 INFO:teuthology.orchestra.run.smithi081.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 653 kB/s | 16 kB 00:00 2024-06-29T14:20:21.046 INFO:teuthology.orchestra.run.smithi167.stdout:(72/160): postgresql-server-13.14-1.el9.x86_64. 4.7 MB/s | 5.8 MB 00:01 2024-06-29T14:20:21.048 INFO:teuthology.orchestra.run.smithi081.stdout:(134/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-06-29T14:20:21.073 INFO:teuthology.orchestra.run.smithi081.stdout:(135/160): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-06-29T14:20:21.107 INFO:teuthology.orchestra.run.smithi081.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 480 kB/s | 28 kB 00:00 2024-06-29T14:20:21.132 INFO:teuthology.orchestra.run.smithi081.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 522 kB/s | 31 kB 00:00 2024-06-29T14:20:21.166 INFO:teuthology.orchestra.run.smithi081.stdout:(138/160): python3-kubernetes-26.1.0-2.el9.noar 3.3 MB/s | 1.0 MB 00:00 2024-06-29T14:20:21.192 INFO:teuthology.orchestra.run.smithi081.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-06-29T14:20:21.217 INFO:teuthology.orchestra.run.smithi081.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-06-29T14:20:21.242 INFO:teuthology.orchestra.run.smithi081.stdout:(141/160): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-06-29T14:20:21.246 INFO:teuthology.orchestra.run.smithi167.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 122 kB/s | 87 kB 00:00 2024-06-29T14:20:21.267 INFO:teuthology.orchestra.run.smithi081.stdout:(142/160): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-06-29T14:20:21.292 INFO:teuthology.orchestra.run.smithi081.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-06-29T14:20:21.318 INFO:teuthology.orchestra.run.smithi081.stdout:(144/160): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-06-29T14:20:21.335 INFO:teuthology.orchestra.run.smithi081.stdout:(145/160): python3-zc-lockfile-2.0-10.el9.noarc 475 kB/s | 20 kB 00:00 2024-06-29T14:20:21.360 INFO:teuthology.orchestra.run.smithi081.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-06-29T14:20:21.385 INFO:teuthology.orchestra.run.smithi081.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-06-29T14:20:21.427 INFO:teuthology.orchestra.run.smithi081.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-06-29T14:20:21.471 INFO:teuthology.orchestra.run.smithi167.stdout:(74/160): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:00 2024-06-29T14:20:21.636 INFO:teuthology.orchestra.run.smithi081.stdout:(149/160): NetworkManager-1.48.0-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-06-29T14:20:21.778 INFO:teuthology.orchestra.run.smithi081.stdout:(150/160): NetworkManager-libnm-1.48.0-1.el9.x8 13 MB/s | 1.8 MB 00:00 2024-06-29T14:20:21.836 INFO:teuthology.orchestra.run.smithi081.stdout:(151/160): librados2-19.0.0-4577.gcf146c2f.el9. 7.0 MB/s | 3.3 MB 00:00 2024-06-29T14:20:21.878 INFO:teuthology.orchestra.run.smithi081.stdout:(152/160): NetworkManager-team-1.48.0-1.el9.x86 412 kB/s | 41 kB 00:00 2024-06-29T14:20:21.930 INFO:teuthology.orchestra.run.smithi167.stdout:(75/160): python3-colorama-0.4.6-3.el9.noarch.r 90 kB/s | 61 kB 00:00 2024-06-29T14:20:21.945 INFO:teuthology.orchestra.run.smithi081.stdout:(153/160): NetworkManager-tui-1.48.0-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-06-29T14:20:21.979 INFO:teuthology.orchestra.run.smithi081.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 366 kB/s | 37 kB 00:00 2024-06-29T14:20:22.046 INFO:teuthology.orchestra.run.smithi081.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 2.0 MB/s | 207 kB 00:00 2024-06-29T14:20:22.088 INFO:teuthology.orchestra.run.smithi081.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 2.3 MB/s | 260 kB 00:00 2024-06-29T14:20:22.197 INFO:teuthology.orchestra.run.smithi081.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 9.0 MB/s | 1.4 MB 00:00 2024-06-29T14:20:22.255 INFO:teuthology.orchestra.run.smithi081.stdout:(158/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-06-29T14:20:22.330 INFO:teuthology.orchestra.run.smithi081.stdout:(159/160): xfsprogs-6.4.0-1.el9.x86_64.rpm 8.2 MB/s | 1.1 MB 00:00 2024-06-29T14:20:22.372 INFO:teuthology.orchestra.run.smithi167.stdout:(76/160): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 6.0 MB 00:01 2024-06-29T14:20:22.380 INFO:teuthology.orchestra.run.smithi081.stdout:(160/160): librbd1-19.0.0-4577.gcf146c2f.el9.x8 3.1 MB/s | 3.1 MB 00:00 2024-06-29T14:20:22.387 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-06-29T14:20:22.387 INFO:teuthology.orchestra.run.smithi081.stdout:Total 18 MB/s | 231 MB 00:12 2024-06-29T14:20:22.397 INFO:teuthology.orchestra.run.smithi167.stdout:(77/160): python3-devel-3.9.18-3.el9.x86_64.rpm 271 kB/s | 244 kB 00:00 2024-06-29T14:20:22.889 INFO:teuthology.orchestra.run.smithi167.stdout:(78/160): python3-jinja2-2.11.3-6.el9.noarch.rp 259 kB/s | 248 kB 00:00 2024-06-29T14:20:22.931 INFO:teuthology.orchestra.run.smithi167.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 36 kB/s | 19 kB 00:00 2024-06-29T14:20:22.956 INFO:teuthology.orchestra.run.smithi167.stdout:(80/160): python3-jsonpatch-1.21-16.el9.noarch. 44 kB/s | 26 kB 00:00 2024-06-29T14:20:23.540 INFO:teuthology.orchestra.run.smithi167.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 57 kB/s | 35 kB 00:00 2024-06-29T14:20:23.647 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-06-29T14:20:23.673 INFO:teuthology.orchestra.run.smithi167.stdout:(82/160): python3-mako-1.1.4-6.el9.noarch.rpm 220 kB/s | 172 kB 00:00 2024-06-29T14:20:23.850 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-06-29T14:20:23.850 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-06-29T14:20:24.174 INFO:teuthology.orchestra.run.smithi167.stdout:(83/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-06-29T14:20:24.349 INFO:teuthology.orchestra.run.smithi167.stdout:(84/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 310 kB/s | 250 kB 00:00 2024-06-29T14:20:24.483 INFO:teuthology.orchestra.run.smithi167.stdout:(85/160): python3-oauthlib-3.1.1-5.el9.noarch.r 275 kB/s | 222 kB 00:00 2024-06-29T14:20:24.858 INFO:teuthology.orchestra.run.smithi167.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 113 kB/s | 77 kB 00:00 2024-06-29T14:20:24.966 INFO:teuthology.orchestra.run.smithi167.stdout:(87/160): python3-prettytable-0.7.2-27.el9.noar 68 kB/s | 42 kB 00:00 2024-06-29T14:20:25.300 INFO:teuthology.orchestra.run.smithi167.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 330 kB/s | 269 kB 00:00 2024-06-29T14:20:25.618 INFO:teuthology.orchestra.run.smithi167.stdout:(89/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 209 kB/s | 159 kB 00:00 2024-06-29T14:20:25.801 INFO:teuthology.orchestra.run.smithi167.stdout:(90/160): python3-pyasn1-modules-0.4.8-6.el9.no 334 kB/s | 279 kB 00:00 2024-06-29T14:20:25.943 INFO:teuthology.orchestra.run.smithi167.stdout:(91/160): python3-pytz-2021.1-5.el9.noarch.rpm 79 kB/s | 51 kB 00:00 2024-06-29T14:20:26.044 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-06-29T14:20:26.045 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-06-29T14:20:26.276 INFO:teuthology.orchestra.run.smithi167.stdout:(92/160): python3-requests-oauthlib-1.3.0-12.el 81 kB/s | 54 kB 00:00 2024-06-29T14:20:26.568 INFO:teuthology.orchestra.run.smithi167.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 67 kB/s | 42 kB 00:00 2024-06-29T14:20:26.894 INFO:teuthology.orchestra.run.smithi167.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 72 kB/s | 44 kB 00:00 2024-06-29T14:20:27.319 INFO:teuthology.orchestra.run.smithi167.stdout:(95/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-06-29T14:20:27.465 INFO:teuthology.orchestra.run.smithi167.stdout:(96/160): qatlib-24.02.0-1.el9.x86_64.rpm 246 kB/s | 221 kB 00:00 2024-06-29T14:20:27.678 INFO:teuthology.orchestra.run.smithi167.stdout:(97/160): qatlib-service-24.02.0-1.el9.x86_64.r 46 kB/s | 36 kB 00:00 2024-06-29T14:20:28.045 INFO:teuthology.orchestra.run.smithi167.stdout:(98/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 65 kB/s | 47 kB 00:00 2024-06-29T14:20:28.320 INFO:teuthology.orchestra.run.smithi167.stdout:(99/160): socat-1.7.4.1-6.el9.x86_64.rpm 356 kB/s | 304 kB 00:00 2024-06-29T14:20:28.345 INFO:teuthology.orchestra.run.smithi167.stdout:(100/160): uuid-1.6.2-55.el9.x86_64.rpm 87 kB/s | 58 kB 00:00 2024-06-29T14:20:28.746 INFO:teuthology.orchestra.run.smithi167.stdout:(101/160): xfsprogs-devel-6.4.0-1.el9.x86_64.rp 135 kB/s | 95 kB 00:00 2024-06-29T14:20:28.829 INFO:teuthology.orchestra.run.smithi167.stdout:(102/160): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-06-29T14:20:28.971 INFO:teuthology.orchestra.run.smithi167.stdout:(103/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-06-29T14:20:28.996 INFO:teuthology.orchestra.run.smithi167.stdout:(104/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 94 kB/s | 64 kB 00:00 2024-06-29T14:20:29.088 INFO:teuthology.orchestra.run.smithi167.stdout:(105/160): protobuf-compiler-3.14.0-13.el9.x86_ 3.3 MB/s | 863 kB 00:00 2024-06-29T14:20:29.147 INFO:teuthology.orchestra.run.smithi167.stdout:(106/160): gperftools-libs-2.9.1-2.el9.x86_64.r 2.0 MB/s | 309 kB 00:00 2024-06-29T14:20:29.181 INFO:teuthology.orchestra.run.smithi167.stdout:(107/160): abseil-cpp-20211102.0-3.el9.x86_64.r 2.6 MB/s | 548 kB 00:00 2024-06-29T14:20:29.197 INFO:teuthology.orchestra.run.smithi167.stdout:(108/160): grpc-data-1.46.7-10.el9.noarch.rpm 183 kB/s | 19 kB 00:00 2024-06-29T14:20:29.214 INFO:teuthology.orchestra.run.smithi167.stdout:(109/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 775 kB/s | 26 kB 00:00 2024-06-29T14:20:29.231 INFO:teuthology.orchestra.run.smithi167.stdout:(110/160): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-06-29T14:20:29.248 INFO:teuthology.orchestra.run.smithi167.stdout:(111/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-06-29T14:20:29.274 INFO:teuthology.orchestra.run.smithi167.stdout:(112/160): luarocks-3.9.2-5.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-06-29T14:20:29.316 INFO:teuthology.orchestra.run.smithi167.stdout:(113/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-06-29T14:20:29.341 INFO:teuthology.orchestra.run.smithi167.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-06-29T14:20:29.378 INFO:teuthology.orchestra.run.smithi167.stdout:(115/160): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-06-29T14:20:29.408 INFO:teuthology.orchestra.run.smithi167.stdout:(116/160): python3-cachetools-4.2.4-1.el9.noarc 479 kB/s | 32 kB 00:00 2024-06-29T14:20:29.434 INFO:teuthology.orchestra.run.smithi167.stdout:(117/160): python3-certifi-2021.10.8-2.el9.noar 255 kB/s | 15 kB 00:00 2024-06-29T14:20:29.459 INFO:teuthology.orchestra.run.smithi167.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-06-29T14:20:29.484 INFO:teuthology.orchestra.run.smithi167.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-06-29T14:20:29.518 INFO:teuthology.orchestra.run.smithi167.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.0 MB/s | 303 kB 00:00 2024-06-29T14:20:29.543 INFO:teuthology.orchestra.run.smithi167.stdout:(121/160): python3-google-auth-2.30.0-1.el9.noa 3.6 MB/s | 213 kB 00:00 2024-06-29T14:20:29.586 INFO:teuthology.orchestra.run.smithi167.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-06-29T14:20:29.611 INFO:teuthology.orchestra.run.smithi167.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 428 kB/s | 11 kB 00:00 2024-06-29T14:20:29.636 INFO:teuthology.orchestra.run.smithi167.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 713 kB/s | 18 kB 00:00 2024-06-29T14:20:29.661 INFO:teuthology.orchestra.run.smithi167.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 931 kB/s | 23 kB 00:00 2024-06-29T14:20:29.686 INFO:teuthology.orchestra.run.smithi167.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 782 kB/s | 19 kB 00:00 2024-06-29T14:20:29.712 INFO:teuthology.orchestra.run.smithi167.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 788 kB/s | 20 kB 00:00 2024-06-29T14:20:29.779 INFO:teuthology.orchestra.run.smithi167.stdout:(128/160): python3-grpcio-1.46.7-10.el9.x86_64. 7.8 MB/s | 2.0 MB 00:00 2024-06-29T14:20:29.804 INFO:teuthology.orchestra.run.smithi167.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-06-29T14:20:29.821 INFO:teuthology.orchestra.run.smithi167.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 4.7 MB/s | 79 kB 00:00 2024-06-29T14:20:29.855 INFO:teuthology.orchestra.run.smithi167.stdout:(131/160): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-06-29T14:20:29.880 INFO:teuthology.orchestra.run.smithi167.stdout:(132/160): python3-natsort-7.1.1-5.el9.noarch.r 985 kB/s | 58 kB 00:00 2024-06-29T14:20:29.955 INFO:teuthology.orchestra.run.smithi167.stdout:(133/160): libarrow-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-06-29T14:20:30.109 INFO:teuthology.orchestra.run.smithi167.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 72 kB/s | 16 kB 00:00 2024-06-29T14:20:30.156 INFO:teuthology.orchestra.run.smithi167.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 601 kB/s | 28 kB 00:00 2024-06-29T14:20:30.181 INFO:teuthology.orchestra.run.smithi167.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 399 kB/s | 90 kB 00:00 2024-06-29T14:20:30.207 INFO:teuthology.orchestra.run.smithi167.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-06-29T14:20:30.232 INFO:teuthology.orchestra.run.smithi167.stdout:(138/160): python3-pecan-1.4.2-3.el9.noarch.rpm 723 kB/s | 272 kB 00:00 2024-06-29T14:20:30.257 INFO:teuthology.orchestra.run.smithi167.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-06-29T14:20:30.282 INFO:teuthology.orchestra.run.smithi167.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-06-29T14:20:30.308 INFO:teuthology.orchestra.run.smithi167.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-06-29T14:20:30.333 INFO:teuthology.orchestra.run.smithi167.stdout:(142/160): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-06-29T14:20:30.358 INFO:teuthology.orchestra.run.smithi167.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-06-29T14:20:30.392 INFO:teuthology.orchestra.run.smithi167.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 599 kB/s | 20 kB 00:00 2024-06-29T14:20:30.418 INFO:teuthology.orchestra.run.smithi167.stdout:(145/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-06-29T14:20:30.443 INFO:teuthology.orchestra.run.smithi167.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-06-29T14:20:30.468 INFO:teuthology.orchestra.run.smithi167.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-06-29T14:20:30.518 INFO:teuthology.orchestra.run.smithi167.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-06-29T14:20:30.702 INFO:teuthology.orchestra.run.smithi167.stdout:(149/160): librados2-19.0.0-4577.gcf146c2f.el9. 13 MB/s | 3.3 MB 00:00 2024-06-29T14:20:30.869 INFO:teuthology.orchestra.run.smithi167.stdout:(150/160): librbd1-19.0.0-4577.gcf146c2f.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-06-29T14:20:30.928 INFO:teuthology.orchestra.run.smithi167.stdout:(151/160): NetworkManager-1.48.0-1.el9.x86_64.r 5.7 MB/s | 2.3 MB 00:00 2024-06-29T14:20:31.036 INFO:teuthology.orchestra.run.smithi167.stdout:(152/160): NetworkManager-team-1.48.0-1.el9.x86 247 kB/s | 41 kB 00:00 2024-06-29T14:20:31.078 INFO:teuthology.orchestra.run.smithi167.stdout:(153/160): NetworkManager-libnm-1.48.0-1.el9.x8 4.8 MB/s | 1.8 MB 00:00 2024-06-29T14:20:31.128 INFO:teuthology.orchestra.run.smithi167.stdout:(154/160): NetworkManager-tui-1.48.0-1.el9.x86_ 1.2 MB/s | 248 kB 00:00 2024-06-29T14:20:31.162 INFO:teuthology.orchestra.run.smithi167.stdout:(155/160): elfutils-debuginfod-client-0.191-4.e 293 kB/s | 37 kB 00:00 2024-06-29T14:20:31.296 INFO:teuthology.orchestra.run.smithi167.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 953 kB/s | 207 kB 00:00 2024-06-29T14:20:31.346 INFO:teuthology.orchestra.run.smithi167.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 1.2 MB/s | 260 kB 00:00 2024-06-29T14:20:31.447 INFO:teuthology.orchestra.run.smithi167.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 4.8 MB/s | 1.4 MB 00:00 2024-06-29T14:20:31.630 INFO:teuthology.orchestra.run.smithi167.stdout:(159/160): xfsprogs-6.4.0-1.el9.x86_64.rpm 3.8 MB/s | 1.1 MB 00:00 2024-06-29T14:20:31.688 INFO:teuthology.orchestra.run.smithi167.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 6.1 MB/s | 2.4 MB 00:00 2024-06-29T14:20:31.696 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-06-29T14:20:31.696 INFO:teuthology.orchestra.run.smithi167.stdout:Total 9.7 MB/s | 231 MB 00:23 2024-06-29T14:20:32.682 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-06-29T14:20:32.794 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-06-29T14:20:32.834 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-06-29T14:20:32.912 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-06-29T14:20:32.932 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-06-29T14:20:32.953 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-06-29T14:20:32.979 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-06-29T14:20:33.028 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-06-29T14:20:33.134 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-06-29T14:20:33.134 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-06-29T14:20:33.425 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-06-29T14:20:33.496 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:20:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:20:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:20:33.730 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:20:33.760 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 10/172 2024-06-29T14:20:33.807 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-06-29T14:20:34.003 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-06-29T14:20:34.058 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-06-29T14:20:34.095 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:20:34.191 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:20:34.221 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 15/172 2024-06-29T14:20:34.267 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 15/172 2024-06-29T14:20:34.288 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-06-29T14:20:34.312 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-06-29T14:20:34.474 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-06-29T14:20:34.579 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-06-29T14:20:34.854 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : xfsprogs-6.4.0-1.el9.x86_64 18/172 2024-06-29T14:20:34.997 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-libnm-1:1.48.0-1.el9.x86_64 19/172 2024-06-29T14:20:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.0-1.el9.x86_64 19/172 2024-06-29T14:20:35.143 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:20:35.217 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:20:35.285 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-06-29T14:20:35.285 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-06-29T14:20:35.415 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:20:35.522 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-06-29T14:20:35.546 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-06-29T14:20:35.581 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-06-29T14:20:35.663 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-06-29T14:20:35.707 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-06-29T14:20:35.748 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-06-29T14:20:35.791 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-06-29T14:20:35.823 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-06-29T14:20:35.852 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-06-29T14:20:35.922 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-06-29T14:20:35.974 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/172 2024-06-29T14:20:36.005 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-06-29T14:20:36.048 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-06-29T14:20:36.124 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-06-29T14:20:36.167 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9. 35/172 2024-06-29T14:20:36.202 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el 36/172 2024-06-29T14:20:36.231 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_6 37/172 2024-06-29T14:20:36.300 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-06-29T14:20:36.346 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-06-29T14:20:36.420 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-06-29T14:20:36.551 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-06-29T14:20:36.629 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-06-29T14:20:37.149 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-06-29T14:20:37.227 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-06-29T14:20:37.256 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-06-29T14:20:37.285 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-06-29T14:20:37.325 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-06-29T14:20:37.363 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_6 48/172 2024-06-29T14:20:37.405 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-06-29T14:20:37.551 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-06-29T14:20:37.618 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-06-29T14:20:37.709 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-06-29T14:20:37.774 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-06-29T14:20:37.846 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-06-29T14:20:37.878 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-06-29T14:20:37.914 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-06-29T14:20:37.961 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-06-29T14:20:37.997 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-06-29T14:20:38.040 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-06-29T14:20:38.082 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-06-29T14:20:38.217 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-06-29T14:20:38.448 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-06-29T14:20:38.503 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 63/172 2024-06-29T14:20:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-06-29T14:20:38.635 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:2.30.0-1.el9.noarch 65/172 2024-06-29T14:20:38.668 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-06-29T14:20:38.700 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-06-29T14:20:38.725 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-06-29T14:20:38.830 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-06-29T14:20:38.968 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-06-29T14:20:39.016 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-06-29T14:20:39.044 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-06-29T14:20:39.083 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-06-29T14:20:39.127 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-06-29T14:20:39.166 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-06-29T14:20:39.196 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-06-29T14:20:39.230 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-06-29T14:20:39.263 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-06-29T14:20:39.305 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-06-29T14:20:39.340 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-06-29T14:20:39.426 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-06-29T14:20:39.478 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-06-29T14:20:39.509 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-06-29T14:20:39.612 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-06-29T14:20:39.660 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-06-29T14:20:40.378 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-06-29T14:20:40.425 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-06-29T14:20:40.458 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-06-29T14:20:40.558 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-06-29T14:20:40.613 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-06-29T14:20:40.659 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-06-29T14:20:40.745 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-06-29T14:20:40.786 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-06-29T14:20:40.963 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-06-29T14:20:40.993 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-06-29T14:20:41.070 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-06-29T14:20:41.722 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-06-29T14:20:41.809 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-06-29T14:20:41.841 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-06-29T14:20:41.893 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-06-29T14:20:41.938 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-06-29T14:20:42.005 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-06-29T14:20:42.045 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-06-29T14:20:42.071 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-06-29T14:20:42.118 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-06-29T14:20:42.518 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-06-29T14:20:42.589 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:20:42.706 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:20:42.732 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:20:42.830 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:20:42.861 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 10/172 2024-06-29T14:20:42.904 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-06-29T14:20:43.075 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-06-29T14:20:43.102 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-06-29T14:20:43.160 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-06-29T14:20:43.163 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-06-29T14:20:43.187 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:20:43.191 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-06-29T14:20:43.225 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-06-29T14:20:43.291 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:20:43.321 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 15/172 2024-06-29T14:20:43.384 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 15/172 2024-06-29T14:20:43.406 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-06-29T14:20:43.426 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-06-29T14:20:43.542 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-06-29T14:20:43.587 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 104/172 2024-06-29T14:20:43.616 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-06-29T14:20:43.677 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 104/172 2024-06-29T14:20:43.709 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 105/172 2024-06-29T14:20:43.721 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-06-29T14:20:43.745 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-06-29T14:20:43.966 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : xfsprogs-6.4.0-1.el9.x86_64 18/172 2024-06-29T14:20:44.106 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : NetworkManager-libnm-1:1.48.0-1.el9.x86_64 19/172 2024-06-29T14:20:44.167 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.0-1.el9.x86_64 19/172 2024-06-29T14:20:44.220 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-06-29T14:20:44.249 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 108/172 2024-06-29T14:20:44.271 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:20:44.321 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 108/172 2024-06-29T14:20:44.334 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:20:44.351 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 109/172 2024-06-29T14:20:44.544 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: NetworkManager-1:1.48.0-1.el9.x86_64 20/172 2024-06-29T14:20:44.649 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-06-29T14:20:44.680 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-06-29T14:20:44.714 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-06-29T14:20:44.779 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-06-29T14:20:44.824 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-06-29T14:20:44.873 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-06-29T14:20:44.925 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-06-29T14:20:44.959 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-06-29T14:20:44.986 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-06-29T14:20:45.054 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-06-29T14:20:45.115 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/172 2024-06-29T14:20:45.147 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-06-29T14:20:45.195 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-06-29T14:20:45.274 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-06-29T14:20:45.309 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9. 35/172 2024-06-29T14:20:45.353 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el 36/172 2024-06-29T14:20:45.382 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_6 37/172 2024-06-29T14:20:45.450 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-06-29T14:20:45.496 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-06-29T14:20:45.570 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-06-29T14:20:45.711 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-06-29T14:20:45.765 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-06-29T14:20:46.286 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-06-29T14:20:46.369 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-06-29T14:20:46.398 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-06-29T14:20:46.436 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-06-29T14:20:46.484 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-06-29T14:20:46.522 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_6 48/172 2024-06-29T14:20:46.564 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-06-29T14:20:46.565 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:46.598 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:46.713 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-06-29T14:20:46.767 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-06-29T14:20:46.857 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-06-29T14:20:46.922 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-06-29T14:20:46.995 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-06-29T14:20:47.034 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-06-29T14:20:47.071 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-06-29T14:20:47.117 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-06-29T14:20:47.154 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-06-29T14:20:47.158 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:47.193 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 111/172 2024-06-29T14:20:47.196 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-06-29T14:20:47.247 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-06-29T14:20:47.289 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 111/172 2024-06-29T14:20:47.290 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-29T14:20:47.290 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-29T14:20:47.290 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:20:47.328 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 112/172 2024-06-29T14:20:47.382 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-06-29T14:20:47.424 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-06-29T14:20:47.465 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 63/172 2024-06-29T14:20:47.562 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-06-29T14:20:47.600 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-google-auth-1:2.30.0-1.el9.noarch 65/172 2024-06-29T14:20:47.640 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-06-29T14:20:47.673 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-06-29T14:20:47.698 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-06-29T14:20:47.782 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-06-29T14:20:47.916 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-06-29T14:20:47.947 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-06-29T14:20:47.987 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-06-29T14:20:48.018 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-06-29T14:20:48.056 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-06-29T14:20:48.088 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-06-29T14:20:48.118 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-06-29T14:20:48.152 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-06-29T14:20:48.186 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-06-29T14:20:48.227 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-06-29T14:20:48.270 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-06-29T14:20:48.357 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-06-29T14:20:48.401 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-06-29T14:20:48.432 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-06-29T14:20:48.534 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-06-29T14:20:48.583 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-06-29T14:20:49.302 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-06-29T14:20:49.346 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-06-29T14:20:49.381 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-06-29T14:20:49.479 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-06-29T14:20:49.535 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-06-29T14:20:49.580 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-06-29T14:20:49.677 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-06-29T14:20:49.708 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-06-29T14:20:49.886 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-06-29T14:20:49.906 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-06-29T14:20:49.985 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-06-29T14:20:50.634 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-06-29T14:20:50.745 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-06-29T14:20:51.990 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-06-29T14:20:52.086 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-06-29T14:20:52.113 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-06-29T14:20:52.146 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-06-29T14:20:52.464 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-06-29T14:20:52.501 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 104/172 2024-06-29T14:20:52.583 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 104/172 2024-06-29T14:20:52.615 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 105/172 2024-06-29T14:20:52.650 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-06-29T14:20:53.135 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-06-29T14:20:53.162 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 108/172 2024-06-29T14:20:53.242 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 108/172 2024-06-29T14:20:53.272 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 109/172 2024-06-29T14:20:55.471 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:55.511 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:56.072 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 110/172 2024-06-29T14:20:56.114 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 111/172 2024-06-29T14:20:56.216 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 111/172 2024-06-29T14:20:56.216 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-29T14:20:56.216 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-29T14:20:56.216 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:20:56.249 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 112/172 2024-06-29T14:20:58.716 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 112/172 2024-06-29T14:20:58.716 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2024-06-29T14:20:58.716 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2024-06-29T14:20:58.716 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2024-06-29T14:20:58.716 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2024-06-29T14:20:58.717 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2024-06-29T14:20:58.717 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /root 2024-06-29T14:20:58.717 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2024-06-29T14:20:58.717 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:20:58.911 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 113/172 2024-06-29T14:20:58.926 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 113/172 2024-06-29T14:20:58.926 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:20:58.926 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-29T14:20:58.926 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:00.505 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 114/172 2024-06-29T14:21:00.531 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 114/172 2024-06-29T14:21:00.532 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:00.532 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-29T14:21:00.532 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-29T14:21:00.532 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-29T14:21:00.532 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:00.756 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 115/172 2024-06-29T14:21:00.779 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 115/172 2024-06-29T14:21:00.779 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:00.780 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-29T14:21:00.780 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-29T14:21:00.780 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-29T14:21:00.780 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:01.240 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 116/172 2024-06-29T14:21:01.263 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 116/172 2024-06-29T14:21:01.263 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:01.263 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-29T14:21:01.263 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-29T14:21:01.263 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-29T14:21:01.264 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:01.362 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-06-29T14:21:01.410 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-06-29T14:21:01.465 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-06-29T14:21:01.652 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-06-29T14:21:01.699 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-06-29T14:21:01.777 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-06-29T14:21:01.874 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-06-29T14:21:02.012 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-06-29T14:21:02.049 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-06-29T14:21:02.083 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-06-29T14:21:02.109 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.9-15.el9.x86_64 127/172 2024-06-29T14:21:02.484 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 128/172 2024-06-29T14:21:02.503 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 128/172 2024-06-29T14:21:02.535 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.e 129/172 2024-06-29T14:21:02.600 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f. 130/172 2024-06-29T14:21:02.629 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 131/172 2024-06-29T14:21:05.741 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 131/172 2024-06-29T14:21:05.794 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 132/172 2024-06-29T14:21:06.750 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 132/172 2024-06-29T14:21:06.786 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 133/172 2024-06-29T14:21:06.897 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 133/172 2024-06-29T14:21:07.060 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el 134/172 2024-06-29T14:21:07.088 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 135/172 2024-06-29T14:21:07.112 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 135/172 2024-06-29T14:21:07.112 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:07.112 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-29T14:21:07.112 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-29T14:21:07.112 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-29T14:21:07.112 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:07.164 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 136/172 2024-06-29T14:21:07.175 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 136/172 2024-06-29T14:21:07.273 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 137/172 2024-06-29T14:21:07.333 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 138/172 2024-06-29T14:21:07.857 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 112/172 2024-06-29T14:21:07.857 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /sys 2024-06-29T14:21:07.857 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /proc 2024-06-29T14:21:07.857 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /mnt 2024-06-29T14:21:07.857 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /var/tmp 2024-06-29T14:21:07.857 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /home 2024-06-29T14:21:07.857 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /root 2024-06-29T14:21:07.857 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /tmp 2024-06-29T14:21:07.858 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:08.030 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 113/172 2024-06-29T14:21:08.046 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 113/172 2024-06-29T14:21:08.046 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:08.046 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-29T14:21:08.046 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:08.260 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-06-29T14:21:08.291 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 140/172 2024-06-29T14:21:08.314 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 140/172 2024-06-29T14:21:08.314 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:08.314 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-29T14:21:08.314 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-29T14:21:08.315 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-29T14:21:08.315 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:08.358 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-4577.gcf146 141/172 2024-06-29T14:21:08.381 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-4577.gcf146 141/172 2024-06-29T14:21:08.381 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:08.381 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-29T14:21:08.381 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:08.426 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 142/172 2024-06-29T14:21:08.448 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 142/172 2024-06-29T14:21:08.449 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:08.449 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-29T14:21:08.449 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-29T14:21:08.449 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-29T14:21:08.449 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:08.760 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 143/172 2024-06-29T14:21:08.783 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 143/172 2024-06-29T14:21:08.783 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:08.783 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-29T14:21:08.783 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-29T14:21:08.784 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-29T14:21:08.784 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:09.608 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 114/172 2024-06-29T14:21:09.635 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 114/172 2024-06-29T14:21:09.635 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:09.635 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-29T14:21:09.635 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-29T14:21:09.635 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-29T14:21:09.636 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:09.859 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 115/172 2024-06-29T14:21:09.882 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 115/172 2024-06-29T14:21:09.883 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:09.883 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-29T14:21:09.883 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-29T14:21:09.883 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-29T14:21:09.883 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:10.351 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 116/172 2024-06-29T14:21:10.374 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 116/172 2024-06-29T14:21:10.375 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:10.375 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-29T14:21:10.375 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-29T14:21:10.375 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-29T14:21:10.375 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:10.482 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-06-29T14:21:10.538 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-06-29T14:21:10.593 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-06-29T14:21:10.779 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-06-29T14:21:10.836 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-06-29T14:21:10.913 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-06-29T14:21:11.002 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-06-29T14:21:11.141 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-06-29T14:21:11.186 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-06-29T14:21:11.218 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-06-29T14:21:11.245 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : fuse-2.9.9-15.el9.x86_64 127/172 2024-06-29T14:21:11.674 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 128/172 2024-06-29T14:21:11.702 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 128/172 2024-06-29T14:21:11.734 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.e 129/172 2024-06-29T14:21:11.787 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f. 130/172 2024-06-29T14:21:11.818 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 131/172 2024-06-29T14:21:13.064 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 144/172 2024-06-29T14:21:13.105 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 145/172 2024-06-29T14:21:13.145 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 146/172 2024-06-29T14:21:13.178 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xfsprogs-devel-6.4.0-1.el9.x86_64 147/172 2024-06-29T14:21:13.205 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-06-29T14:21:13.389 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 149/172 2024-06-29T14:21:13.435 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-06-29T14:21:13.467 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_ 151/172 2024-06-29T14:21:13.508 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-06-29T14:21:13.537 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-team-1:1.48.0-1.el9.x86_64 153/172 2024-06-29T14:21:13.584 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-tui-1:1.48.0-1.el9.x86_64 154/172 2024-06-29T14:21:14.190 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:21:14.223 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:21:14.243 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:21:14.920 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 131/172 2024-06-29T14:21:14.979 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 132/172 2024-06-29T14:21:15.069 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 156/172 2024-06-29T14:21:15.153 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-06-29T14:21:15.195 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-06-29T14:21:15.277 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-06-29T14:21:15.306 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-06-29T14:21:15.332 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-06-29T14:21:15.357 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-06-29T14:21:15.383 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-06-29T14:21:15.383 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-06-29T14:21:15.459 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-06-29T14:21:15.484 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-06-29T14:21:15.484 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-06-29T14:21:15.552 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-06-29T14:21:15.577 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-06-29T14:21:15.583 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:21:15.583 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:21:15.771 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:21:15.771 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-06-29T14:21:15.839 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-06-29T14:21:15.863 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-06-29T14:21:15.897 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-06-29T14:21:15.936 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 132/172 2024-06-29T14:21:15.963 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 133/172 2024-06-29T14:21:15.970 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:21:16.073 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 133/172 2024-06-29T14:21:16.246 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el 134/172 2024-06-29T14:21:16.274 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 135/172 2024-06-29T14:21:16.298 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 135/172 2024-06-29T14:21:16.298 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:16.298 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-29T14:21:16.298 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-29T14:21:16.298 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-29T14:21:16.298 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:16.349 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 136/172 2024-06-29T14:21:16.361 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 136/172 2024-06-29T14:21:16.462 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 137/172 2024-06-29T14:21:16.522 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 138/172 2024-06-29T14:21:17.460 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-06-29T14:21:17.502 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 140/172 2024-06-29T14:21:17.526 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 140/172 2024-06-29T14:21:17.526 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:17.526 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-29T14:21:17.526 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-29T14:21:17.526 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-29T14:21:17.526 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:17.570 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-4577.gcf146 141/172 2024-06-29T14:21:17.592 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-4577.gcf146 141/172 2024-06-29T14:21:17.592 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:17.593 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-29T14:21:17.593 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:17.637 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 142/172 2024-06-29T14:21:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 142/172 2024-06-29T14:21:17.660 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:17.660 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-29T14:21:17.660 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-29T14:21:17.660 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-29T14:21:17.660 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:17.971 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 143/172 2024-06-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 143/172 2024-06-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-29T14:21:17.994 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:22.259 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 144/172 2024-06-29T14:21:22.300 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 145/172 2024-06-29T14:21:22.340 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 146/172 2024-06-29T14:21:22.382 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : xfsprogs-devel-6.4.0-1.el9.x86_64 147/172 2024-06-29T14:21:22.409 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 1/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 2/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 3/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 4/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-4577.gcf146 5/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 6/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 7/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 10/172 2024-06-29T14:21:22.447 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 11/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 12/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 13/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_ 15/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 16/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 17/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_6 18/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 19/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 20/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el 21/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9. 22/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_6 23/172 2024-06-29T14:21:22.448 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 24/172 2024-06-29T14:21:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 25/172 2024-06-29T14:21:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 26/172 2024-06-29T14:21:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 27/172 2024-06-29T14:21:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 28/172 2024-06-29T14:21:22.449 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 29/172 2024-06-29T14:21:22.461 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f. 30/172 2024-06-29T14:21:22.461 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 31/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 32/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 33/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el 34/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 35/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.e 36/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 37/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 38/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 39/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-06-29T14:21:22.462 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-06-29T14:21:22.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-06-29T14:21:22.464 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-06-29T14:21:22.465 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-06-29T14:21:22.466 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-devel-6.4.0-1.el9.x86_64 101/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-06-29T14:21:22.467 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-06-29T14:21:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:2.30.0-1.el9.noarch 122/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-06-29T14:21:22.469 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-06-29T14:21:22.470 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 149/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 151/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-1:1.48.0-1.el9.x86_64 153/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-libnm-1:1.48.0-1.el9.x86_64 155/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-team-1:1.48.0-1.el9.x86_64 157/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-tui-1:1.48.0-1.el9.x86_64 159/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-06-29T14:21:22.471 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-06-29T14:21:22.472 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-06-29T14:21:22.472 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-06-29T14:21:22.472 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-06-29T14:21:22.472 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-06-29T14:21:22.472 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-06-29T14:21:22.472 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-06-29T14:21:22.472 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-06-29T14:21:22.472 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-6.4.0-1.el9.x86_64 171/172 2024-06-29T14:21:22.595 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 149/172 2024-06-29T14:21:22.647 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-06-29T14:21:22.679 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_ 151/172 2024-06-29T14:21:22.711 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-06-29T14:21:22.741 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : NetworkManager-team-1:1.48.0-1.el9.x86_64 153/172 2024-06-29T14:21:22.779 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : NetworkManager-tui-1:1.48.0-1.el9.x86_64 154/172 2024-06-29T14:21:23.383 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:21:23.428 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:21:23.448 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-06-29T14:21:24.282 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 156/172 2024-06-29T14:21:24.375 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-06-29T14:21:24.417 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-06-29T14:21:24.515 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-06-29T14:21:24.546 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-06-29T14:21:24.572 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-06-29T14:21:24.596 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-06-29T14:21:24.623 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-06-29T14:21:24.623 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-06-29T14:21:24.708 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-06-29T14:21:24.740 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-06-29T14:21:24.740 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-06-29T14:21:24.825 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-06-29T14:21:24.850 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-06-29T14:21:24.856 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:21:24.856 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:21:25.044 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-06-29T14:21:25.044 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-06-29T14:21:25.128 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-06-29T14:21:25.153 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-06-29T14:21:25.186 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-06-29T14:21:25.261 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:21:25.326 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:21:25.326 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:25.326 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-06-29T14:21:25.326 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-1:1.48.0-1.el9.x86_64 2024-06-29T14:21:25.326 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-libnm-1:1.48.0-1.el9.x86_64 2024-06-29T14:21:25.326 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-team-1:1.48.0-1.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-tui-1:1.48.0-1.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-6.4.0-1.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: automake-1.16.2-8.el9.noarch 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: bison-3.7.4-5.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-29T14:21:25.327 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:25.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: flex-2.6.4-9.el9.x86_64 2024-06-29T14:21:25.329 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.330 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: libtool-2.4.6-46.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-29T14:21:25.331 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: lua-5.4.4-4.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-13.14-1.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-29T14:21:25.332 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-29T14:21:25.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-29T14:21:25.334 INFO:teuthology.orchestra.run.smithi081.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-29T14:21:25.334 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-29T14:21:25.334 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-29T14:21:25.334 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:2.30.0-1.el9.noarch 2024-06-29T14:21:25.334 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-29T14:21:25.334 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-29T14:21:25.334 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-29T14:21:25.334 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-29T14:21:25.334 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-29T14:21:25.334 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-29T14:21:25.335 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply-3.11-14.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-29T14:21:25.336 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-29T14:21:25.337 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: re2-1:20211101-3.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-devel-6.4.0-1.el9.x86_64 2024-06-29T14:21:25.338 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-29T14:21:25.339 INFO:teuthology.orchestra.run.smithi081.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-29T14:21:25.339 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:25.339 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-06-29T14:21:25.793 DEBUG:teuthology.parallel:result is None 2024-06-29T14:21:31.667 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:21:31.668 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 1/172 2024-06-29T14:21:31.668 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 2/172 2024-06-29T14:21:31.668 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 3/172 2024-06-29T14:21:31.668 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 4/172 2024-06-29T14:21:31.668 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-4577.gcf146 5/172 2024-06-29T14:21:31.668 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 6/172 2024-06-29T14:21:31.668 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 7/172 2024-06-29T14:21:31.668 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 8/172 2024-06-29T14:21:31.668 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 9/172 2024-06-29T14:21:31.668 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 10/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 11/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 12/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 13/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 14/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_ 15/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 16/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 17/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_6 18/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86 19/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 20/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el 21/172 2024-06-29T14:21:31.669 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9. 22/172 2024-06-29T14:21:31.670 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_6 23/172 2024-06-29T14:21:31.670 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 24/172 2024-06-29T14:21:31.670 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 25/172 2024-06-29T14:21:31.670 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 26/172 2024-06-29T14:21:31.670 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 27/172 2024-06-29T14:21:31.670 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 28/172 2024-06-29T14:21:31.670 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 29/172 2024-06-29T14:21:31.670 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f. 30/172 2024-06-29T14:21:31.670 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noa 31/172 2024-06-29T14:21:31.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.n 32/172 2024-06-29T14:21:31.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf1 33/172 2024-06-29T14:21:31.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el 34/172 2024-06-29T14:21:31.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 35/172 2024-06-29T14:21:31.683 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.e 36/172 2024-06-29T14:21:31.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 37/172 2024-06-29T14:21:31.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 38/172 2024-06-29T14:21:31.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 39/172 2024-06-29T14:21:31.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/172 2024-06-29T14:21:31.684 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-06-29T14:21:31.685 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-06-29T14:21:31.685 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-06-29T14:21:31.685 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-06-29T14:21:31.685 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-06-29T14:21:31.685 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-06-29T14:21:31.686 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-06-29T14:21:31.686 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/172 2024-06-29T14:21:31.686 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-06-29T14:21:31.686 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-06-29T14:21:31.686 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-06-29T14:21:31.686 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-06-29T14:21:31.687 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-06-29T14:21:31.687 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-06-29T14:21:31.687 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-06-29T14:21:31.687 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-06-29T14:21:31.687 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-06-29T14:21:31.688 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-06-29T14:21:31.688 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-06-29T14:21:31.688 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-06-29T14:21:31.688 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-06-29T14:21:31.688 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-06-29T14:21:31.688 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-06-29T14:21:31.688 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-06-29T14:21:31.689 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-06-29T14:21:31.689 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-06-29T14:21:31.689 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-06-29T14:21:31.689 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-06-29T14:21:31.689 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-06-29T14:21:31.689 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-06-29T14:21:31.689 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-06-29T14:21:31.689 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-06-29T14:21:31.690 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-06-29T14:21:31.690 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-06-29T14:21:31.690 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-06-29T14:21:31.690 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-06-29T14:21:31.690 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-06-29T14:21:31.690 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-06-29T14:21:31.690 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-06-29T14:21:31.691 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-06-29T14:21:31.691 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-06-29T14:21:31.691 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-06-29T14:21:31.691 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-06-29T14:21:31.691 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-06-29T14:21:31.691 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-06-29T14:21:31.691 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-06-29T14:21:31.691 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-06-29T14:21:31.692 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-06-29T14:21:31.692 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-06-29T14:21:31.692 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-06-29T14:21:31.692 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-06-29T14:21:31.692 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-06-29T14:21:31.693 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-06-29T14:21:31.693 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-06-29T14:21:31.693 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-06-29T14:21:31.693 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-06-29T14:21:31.693 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-06-29T14:21:31.693 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-06-29T14:21:31.694 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-06-29T14:21:31.694 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-06-29T14:21:31.694 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : xfsprogs-devel-6.4.0-1.el9.x86_64 101/172 2024-06-29T14:21:31.694 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-06-29T14:21:31.694 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-06-29T14:21:31.695 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-06-29T14:21:31.695 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-06-29T14:21:31.695 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-06-29T14:21:31.695 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-06-29T14:21:31.695 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-06-29T14:21:31.695 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-06-29T14:21:31.696 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-06-29T14:21:31.696 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-06-29T14:21:31.696 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-06-29T14:21:31.696 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-06-29T14:21:31.696 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-06-29T14:21:31.696 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-06-29T14:21:31.696 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-06-29T14:21:31.696 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-06-29T14:21:31.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/172 2024-06-29T14:21:31.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-06-29T14:21:31.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-06-29T14:21:31.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-06-29T14:21:31.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-google-auth-1:2.30.0-1.el9.noarch 122/172 2024-06-29T14:21:31.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-06-29T14:21:31.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-06-29T14:21:31.697 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-06-29T14:21:31.698 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-06-29T14:21:31.699 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-06-29T14:21:31.700 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 149/172 2024-06-29T14:21:31.700 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-06-29T14:21:31.700 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 151/172 2024-06-29T14:21:31.700 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-06-29T14:21:31.701 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : NetworkManager-1:1.48.0-1.el9.x86_64 153/172 2024-06-29T14:21:31.701 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-06-29T14:21:31.701 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : NetworkManager-libnm-1:1.48.0-1.el9.x86_64 155/172 2024-06-29T14:21:31.701 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-06-29T14:21:31.701 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : NetworkManager-team-1:1.48.0-1.el9.x86_64 157/172 2024-06-29T14:21:31.701 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : NetworkManager-tui-1:1.48.0-1.el9.x86_64 159/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-06-29T14:21:31.702 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : xfsprogs-6.4.0-1.el9.x86_64 171/172 2024-06-29T14:21:34.548 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-06-29T14:21:34.548 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:34.548 INFO:teuthology.orchestra.run.smithi167.stdout:Upgraded: 2024-06-29T14:21:34.548 INFO:teuthology.orchestra.run.smithi167.stdout: NetworkManager-1:1.48.0-1.el9.x86_64 2024-06-29T14:21:34.548 INFO:teuthology.orchestra.run.smithi167.stdout: NetworkManager-libnm-1:1.48.0-1.el9.x86_64 2024-06-29T14:21:34.548 INFO:teuthology.orchestra.run.smithi167.stdout: NetworkManager-team-1:1.48.0-1.el9.x86_64 2024-06-29T14:21:34.548 INFO:teuthology.orchestra.run.smithi167.stdout: NetworkManager-tui-1:1.48.0-1.el9.x86_64 2024-06-29T14:21:34.548 INFO:teuthology.orchestra.run.smithi167.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-29T14:21:34.548 INFO:teuthology.orchestra.run.smithi167.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-29T14:21:34.548 INFO:teuthology.orchestra.run.smithi167.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: xfsprogs-6.4.0-1.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: automake-1.16.2-8.el9.noarch 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: bison-3.7.4-5.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.549 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.550 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: cephfs-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: cephfs-shell-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: cephfs-top-2:19.0.0-4577.gcf146c2f.el9.noarch 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: flex-2.6.4-9.el9.x86_64 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-29T14:21:34.551 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-29T14:21:34.552 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: libtool-2.4.6-46.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: lua-5.4.4-4.el9.x86_64 2024-06-29T14:21:34.553 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-13.14-1.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-29T14:21:34.554 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-29T14:21:34.555 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth-1:2.30.0-1.el9.noarch 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-29T14:21:34.556 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply-3.11-14.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-29T14:21:34.557 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-29T14:21:34.558 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-29T14:21:34.559 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-2:19.0.0-4577.gcf146c2f.el9.x86_64 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout: re2-1:20211101-3.el9.x86_64 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout: xfsprogs-devel-6.4.0-1.el9.x86_64 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:34.560 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-06-29T14:21:34.979 DEBUG:teuthology.parallel:result is None 2024-06-29T14:21:34.980 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cf146c2f9dac0bd5aed66008255af2113db872f7 2024-06-29T14:21:35.121 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-29T14:21:35.148 INFO:teuthology.orchestra.run.smithi081.stdout:19.0.0-4577.gcf146c2f.el9 2024-06-29T14:21:35.148 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-4577.gcf146c2f.el9 2024-06-29T14:21:35.149 INFO:teuthology.task.install:The correct ceph version 19.0.0-4577.gcf146c2f is installed. 2024-06-29T14:21:35.151 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=cf146c2f9dac0bd5aed66008255af2113db872f7 2024-06-29T14:21:35.368 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-29T14:21:35.391 INFO:teuthology.orchestra.run.smithi167.stdout:19.0.0-4577.gcf146c2f.el9 2024-06-29T14:21:35.391 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-4577.gcf146c2f.el9 2024-06-29T14:21:35.391 INFO:teuthology.task.install:The correct ceph version 19.0.0-4577.gcf146c2f is installed. 2024-06-29T14:21:35.393 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-29T14:21:35.393 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:35.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-29T14:21:35.426 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:21:35.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-29T14:21:35.459 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-29T14:21:35.459 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:35.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-06-29T14:21:35.496 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-29T14:21:35.565 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:21:35.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2024-06-29T14:21:35.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-29T14:21:35.656 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-29T14:21:35.657 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:35.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-29T14:21:35.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-29T14:21:35.753 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:21:35.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-29T14:21:35.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-29T14:21:35.844 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-29T14:21:35.844 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:35.844 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-06-29T14:21:35.869 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-29T14:21:35.933 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:21:35.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/stdin-killer 2024-06-29T14:21:35.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-29T14:21:36.023 INFO:teuthology.run_tasks:Running task ceph... 2024-06-29T14:21:36.102 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-06-29T14:21:36.102 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /var/log/ceph 2024-06-29T14:21:36.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /var/log/ceph 2024-06-29T14:21:36.137 INFO:tasks.ceph:Disabling ceph logrotate... 2024-06-29T14:21:36.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-29T14:21:36.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-29T14:21:36.204 INFO:tasks.ceph:Creating extra log directories... 2024-06-29T14:21:36.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-29T14:21:36.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-29T14:21:36.275 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-06-29T14:21:36.275 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-06-29T14:21:36.275 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-06-29T14:21:36.275 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:36.275 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-06-29T14:21:36.322 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-06-29T14:21:36.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-06-29T14:21:36.459 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:21:36.460 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-06-29T14:21:36.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-06-29T14:21:36.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-06-29T14:21:36.616 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-06-29T14:21:36.617 INFO:tasks.ceph:config {'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-06-29T14:21:36.617 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926', 'branch': 'wip-mchangir-qa-handle-correct-CephManager-instantiation-debug', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap-schedule}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7778926', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-qa-handle-correct-CephManager-instantiation-debug'}, 'ceph': {'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'qa-handle-correct-CephManager-instantiation', 'sha1': '019bb938d4977149d100ab4c3d2c706147295470'}}, 'owner': 'scheduled_mchangir@teuthology', 'priority': 25, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1683, 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'qa-handle-correct-CephManager-instantiation', 'suite_path': '/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/mchangir/ceph.git', 'suite_sha1': '019bb938d4977149d100ab4c3d2c706147295470', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi081.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHAK94OvNRrl6Hj3oVfsn5HM5Yxko9bHDQ11F3N3QYtWe3AN0O1HBUIhOTMbVt6J0N98/F53Io9l/Tbbul/QF1c=', 'smithi167.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKdOwx6NJOHHxMeaoXCSq7R8JV4iSVti8gQ/L1RjNqGpgsM+Z4UzrIV5IzWZTegiQpYwljsAKeE9TY36xsCKDu0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7'}}, {'ceph': {'create_rbd_pool': False, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'cf146c2f9dac0bd5aed66008255af2113db872f7', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '544fecbcd55f3d2b6f478823823ce40cbefef1d4', 'timestamp': '2024-06-29_14:01:10', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698093'} 2024-06-29T14:21:36.617 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-29T14:21:36.620 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-29T14:21:36.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/run/ceph 2024-06-29T14:21:36.675 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/run/ceph 2024-06-29T14:21:36.732 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:36.732 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-06-29T14:21:36.752 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-29T14:21:36.752 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-06-29T14:21:36.807 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-29T14:21:36.807 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:21:36.807 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-06-29T14:21:36.807 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:21:36.807 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:21:36.807 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-06-29 14:21:16.312470571 +0000 2024-06-29T14:21:36.807 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-06-29 14:19:06.188279847 +0000 2024-06-29T14:21:36.807 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-06-29 14:19:06.188279847 +0000 2024-06-29T14:21:36.807 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-06-29 14:19:06.188279847 +0000 2024-06-29T14:21:36.807 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-29T14:21:36.871 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-06-29T14:21:36.872 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-06-29T14:21:36.872 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000180775 s, 2.8 MB/s 2024-06-29T14:21:36.872 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-29T14:21:36.929 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-06-29T14:21:36.984 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-29T14:21:36.984 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:21:36.984 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-06-29T14:21:36.984 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:21:36.984 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:21:36.985 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-06-29 14:21:16.313470552 +0000 2024-06-29T14:21:36.985 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-06-29 14:19:06.517273489 +0000 2024-06-29T14:21:36.985 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-06-29 14:19:06.517273489 +0000 2024-06-29T14:21:36.985 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-06-29 14:19:06.517273489 +0000 2024-06-29T14:21:36.985 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-29T14:21:37.047 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-06-29T14:21:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-06-29T14:21:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000195661 s, 2.6 MB/s 2024-06-29T14:21:37.049 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-29T14:21:37.104 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-06-29T14:21:37.160 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-29T14:21:37.160 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:21:37.160 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-29T14:21:37.160 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:21:37.160 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:21:37.160 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-06-29 14:21:16.313470552 +0000 2024-06-29T14:21:37.160 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-06-29 14:19:06.842267209 +0000 2024-06-29T14:21:37.161 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-06-29 14:19:06.842267209 +0000 2024-06-29T14:21:37.161 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-06-29 14:19:06.842267209 +0000 2024-06-29T14:21:37.161 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-29T14:21:37.222 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-06-29T14:21:37.223 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-06-29T14:21:37.223 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000166287 s, 3.1 MB/s 2024-06-29T14:21:37.224 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-29T14:21:37.280 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-06-29T14:21:37.336 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-29T14:21:37.336 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:21:37.336 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-06-29T14:21:37.336 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:21:37.336 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:21:37.336 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-06-29 14:21:16.313470552 +0000 2024-06-29T14:21:37.336 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-06-29 14:19:07.163261006 +0000 2024-06-29T14:21:37.336 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-06-29 14:19:07.163261006 +0000 2024-06-29T14:21:37.336 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-06-29 14:19:07.163261006 +0000 2024-06-29T14:21:37.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-29T14:21:37.399 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-06-29T14:21:37.399 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-06-29T14:21:37.399 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000189614 s, 2.7 MB/s 2024-06-29T14:21:37.400 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-29T14:21:37.457 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:37.457 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:21:37.457 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2024-06-29T14:21:37.474 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-29T14:21:37.474 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 2024-06-29T14:21:37.531 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-29T14:21:37.531 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:21:37.531 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 673 Links: 1 2024-06-29T14:21:37.531 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:21:37.531 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:21:37.531 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-06-29 14:21:25.629925018 +0000 2024-06-29T14:21:37.531 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-06-29 14:19:05.616124130 +0000 2024-06-29T14:21:37.531 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-06-29 14:19:05.616124130 +0000 2024-06-29T14:21:37.532 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-06-29 14:19:05.616124130 +0000 2024-06-29T14:21:37.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-29T14:21:37.596 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-06-29T14:21:37.596 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-06-29T14:21:37.596 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000209073 s, 2.4 MB/s 2024-06-29T14:21:37.597 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-29T14:21:37.654 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 2024-06-29T14:21:37.711 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-29T14:21:37.711 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:21:37.711 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 693 Links: 1 2024-06-29T14:21:37.711 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:21:37.711 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:21:37.712 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-06-29 14:21:25.629925018 +0000 2024-06-29T14:21:37.712 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-06-29 14:19:05.964118316 +0000 2024-06-29T14:21:37.712 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-06-29 14:19:05.964118316 +0000 2024-06-29T14:21:37.712 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-06-29 14:19:05.964118316 +0000 2024-06-29T14:21:37.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-29T14:21:37.775 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-06-29T14:21:37.775 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-06-29T14:21:37.775 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000196719 s, 2.6 MB/s 2024-06-29T14:21:37.776 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-29T14:21:37.833 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 2024-06-29T14:21:37.890 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-29T14:21:37.891 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:21:37.891 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 711 Links: 1 2024-06-29T14:21:37.891 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:21:37.891 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:21:37.891 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-06-29 14:21:25.630925002 +0000 2024-06-29T14:21:37.891 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-06-29 14:19:06.303112653 +0000 2024-06-29T14:21:37.891 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-06-29 14:19:06.303112653 +0000 2024-06-29T14:21:37.892 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-06-29 14:19:06.303112653 +0000 2024-06-29T14:21:37.892 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-29T14:21:37.955 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-06-29T14:21:37.955 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-06-29T14:21:37.955 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000164007 s, 3.1 MB/s 2024-06-29T14:21:37.957 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-29T14:21:38.013 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 2024-06-29T14:21:38.070 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-29T14:21:38.070 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-29T14:21:38.070 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 730 Links: 1 2024-06-29T14:21:38.070 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-29T14:21:38.070 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-06-29T14:21:38.070 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-06-29 14:21:25.630925002 +0000 2024-06-29T14:21:38.070 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-06-29 14:19:06.623107307 +0000 2024-06-29T14:21:38.070 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-06-29 14:19:06.623107307 +0000 2024-06-29T14:21:38.070 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-06-29 14:19:06.623107307 +0000 2024-06-29T14:21:38.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-29T14:21:38.133 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-06-29T14:21:38.133 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-06-29T14:21:38.133 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.00020969 s, 2.4 MB/s 2024-06-29T14:21:38.135 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-29T14:21:38.191 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:38.191 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-06-29T14:21:38.191 INFO:tasks.ceph:Generating config... 2024-06-29T14:21:38.193 INFO:tasks.ceph:[client] pid file = 2024-06-29T14:21:38.193 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-06-29T14:21:38.193 INFO:tasks.ceph:[mgr] debug client = 20 2024-06-29T14:21:38.193 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-06-29T14:21:38.193 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-06-29T14:21:38.193 INFO:tasks.ceph:[mgr] debug ms = 1 2024-06-29T14:21:38.193 INFO:tasks.ceph:[mon] debug mon = 20 2024-06-29T14:21:38.193 INFO:tasks.ceph:[mon] debug ms = 1 2024-06-29T14:21:38.193 INFO:tasks.ceph:[mon] debug paxos = 20 2024-06-29T14:21:38.193 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-06-29T14:21:38.193 INFO:tasks.ceph:[osd] bdev async discard = True 2024-06-29T14:21:38.193 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] debug ms = 1 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] debug osd = 20 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-06-29T14:21:38.194 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-06-29T14:21:38.195 INFO:tasks.ceph:Setting up mon.a... 2024-06-29T14:21:38.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-06-29T14:21:38.264 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.keyring 2024-06-29T14:21:38.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-06-29T14:21:38.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-29T14:21:38.346 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.81'), ('mon.b', '172.21.15.167'), ('mon.c', '[v2:172.21.15.167:3301,v1:172.21.15.167:6790]')] 2024-06-29T14:21:38.347 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.81,172.21.15.167,[v2:172.21.15.167:3301,v1:172.21.15.167:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'pid file': ''}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}} 2024-06-29T14:21:38.348 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:38.348 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-06-29T14:21:38.401 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.81 --add b 172.21.15.167 --addv c '[v2:172.21.15.167:3301,v1:172.21.15.167:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:38.497 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:38.497 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: generated fsid e6805bd5-4748-48b4-ab28-fbf8a8d96ae4 2024-06-29T14:21:38.497 INFO:teuthology.orchestra.run.smithi081.stdout:setting min_mon_release = quincy 2024-06-29T14:21:38.498 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 0 2024-06-29T14:21:38.498 INFO:teuthology.orchestra.run.smithi081.stdout:fsid e6805bd5-4748-48b4-ab28-fbf8a8d96ae4 2024-06-29T14:21:38.498 INFO:teuthology.orchestra.run.smithi081.stdout:last_changed 2024-06-29T14:21:38.499864+0000 2024-06-29T14:21:38.498 INFO:teuthology.orchestra.run.smithi081.stdout:created 2024-06-29T14:21:38.499864+0000 2024-06-29T14:21:38.498 INFO:teuthology.orchestra.run.smithi081.stdout:min_mon_release 17 (quincy) 2024-06-29T14:21:38.498 INFO:teuthology.orchestra.run.smithi081.stdout:election_strategy: 1 2024-06-29T14:21:38.499 INFO:teuthology.orchestra.run.smithi081.stdout:0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-06-29T14:21:38.499 INFO:teuthology.orchestra.run.smithi081.stdout:1: [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] mon.b 2024-06-29T14:21:38.499 INFO:teuthology.orchestra.run.smithi081.stdout:2: [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] mon.c 2024-06-29T14:21:38.499 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-06-29T14:21:38.501 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-06-29T14:21:38.517 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e6805bd5-4748-48b4-ab28-fbf8a8d96ae4... 2024-06-29T14:21:38.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-29T14:21:38.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: chdir = "" 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: auth supported = cephx 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: filestore xattr use omap = true 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: mon clock drift allowed = 1.000 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: osd crush chooseleaf type = 0 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: auth debug = true 2024-06-29T14:21:38.600 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on old message = true 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on bug = true 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: mon pg warn max object skew = 0 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: # disable pg_autoscaler by default for new pools 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default size = 2 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow primary affinity = true 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow pg remap = true 2024-06-29T14:21:38.601 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on legacy crush tunables = false 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on crush straw calc version zero = false 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on no sortbitwise = false 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on osd down out interval zero = false 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on too few osds = false 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: mon_allow_pool_size_one = true 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: osd default data pool replay window = 5 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: mon allow pool delete = true 2024-06-29T14:21:38.602 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: mon cluster log file level = debug 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: debug asserts on shutdown = true 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: mon health detail to clog = false 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: mon host = "172.21.15.81,172.21.15.167,[v2:172.21.15.167:3301,v1:172.21.15.167:6790]" 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on skipped message = False 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = e6805bd5-4748-48b4-ab28-fbf8a8d96ae4 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout:[osd] 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: osd journal size = 100 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub load threshold = 5.0 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub max interval = 600 2024-06-29T14:21:38.603 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock profile = high_recovery_ops 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: osd recover clone overlap = true 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: osd recovery max chunk = 1048576 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug shutdown = true 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug op order = true 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify stray on activate = true 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug trim objects = true 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: osd open classes on start = true 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug pg log writeout = true 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: osd deep scrub update digest min age = 30 2024-06-29T14:21:38.604 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: osd map max advance = 10 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: journal zero on create = true 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: filestore ondisk finisher threads = 3 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: filestore apply finisher threads = 3 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: bdev debug aio = true 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug misdirected ops = true 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: bdev async discard = True 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: bdev enable discard = True 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore allocator = bitmap 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore block size = 96636764160 2024-06-29T14:21:38.605 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore fsck on mount = True 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluefs = 1/20 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluestore = 1/20 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: debug osd = 20 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: debug rocksdb = 4/10 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd backfillfull_ratio = 0.85 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd full ratio = 0.9 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd nearfull ratio = 0.8 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: osd failsafe full ratio = 0.95 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: osd objectstore = bluestore 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr] 2024-06-29T14:21:38.606 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: debug mgr = 20 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: mgr/telemetry/nag = false 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: debug client = 20 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: debug finisher = 20 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout:[mon] 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: debug paxos = 20 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-06-29T14:21:38.607 INFO:teuthology.orchestra.run.smithi081.stdout: mon data avail warn = 5 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: mon mgr mkfs grace = 240 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd reporter subtree level = osd 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd prime pg temp = true 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: auth mon ticket ttl = 660 # 11m 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: auth service ticket ttl = 240 # 4m 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: # don't complain about insecure global_id in the test suite 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-29T14:21:38.608 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-29T14:21:38.609 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.609 INFO:teuthology.orchestra.run.smithi081.stdout: # 1m isn't quite enough 2024-06-29T14:21:38.609 INFO:teuthology.orchestra.run.smithi081.stdout: mon_down_mkfs_grace = 2m 2024-06-29T14:21:38.609 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.609 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_filestore_osds = false 2024-06-29T14:21:38.609 INFO:teuthology.orchestra.run.smithi081.stdout: mon down mkfs grace = 300 2024-06-29T14:21:38.609 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:21:38.609 INFO:teuthology.orchestra.run.smithi081.stdout:[client] 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout: rgw cache enabled = true 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable ops log = true 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable usage log = true 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout: pid file = "" 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.a] 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.b] 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.c] 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.a] 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.c] 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.b] 2024-06-29T14:21:38.610 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.d] 2024-06-29T14:21:38.611 INFO:teuthology.orchestra.run.smithi167.stdout:[global] 2024-06-29T14:21:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: chdir = "" 2024-06-29T14:21:38.611 INFO:teuthology.orchestra.run.smithi167.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: auth supported = cephx 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: filestore xattr use omap = true 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: mon clock drift allowed = 1.000 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: osd crush chooseleaf type = 0 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: auth debug = true 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on old message = true 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on bug = true 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-29T14:21:38.612 INFO:teuthology.orchestra.run.smithi167.stdout: mon pg warn max object skew = 0 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: # disable pg_autoscaler by default for new pools 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default size = 2 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow primary affinity = true 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow pg remap = true 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on legacy crush tunables = false 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on crush straw calc version zero = false 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on no sortbitwise = false 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on osd down out interval zero = false 2024-06-29T14:21:38.613 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on too few osds = false 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: mon_allow_pool_size_one = true 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: osd default data pool replay window = 5 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: mon allow pool delete = true 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: mon cluster log file level = debug 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: debug asserts on shutdown = true 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: mon health detail to clog = false 2024-06-29T14:21:38.614 INFO:teuthology.orchestra.run.smithi167.stdout: mon host = "172.21.15.81,172.21.15.167,[v2:172.21.15.167:3301,v1:172.21.15.167:6790]" 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on skipped message = False 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: fsid = e6805bd5-4748-48b4-ab28-fbf8a8d96ae4 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout:[osd] 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: osd journal size = 100 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub load threshold = 5.0 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub max interval = 600 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: osd mclock profile = high_recovery_ops 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: osd recover clone overlap = true 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: osd recovery max chunk = 1048576 2024-06-29T14:21:38.615 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug shutdown = true 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug op order = true 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify stray on activate = true 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug trim objects = true 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: osd open classes on start = true 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug pg log writeout = true 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: osd deep scrub update digest min age = 30 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: osd map max advance = 10 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: journal zero on create = true 2024-06-29T14:21:38.616 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: filestore ondisk finisher threads = 3 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: filestore apply finisher threads = 3 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: bdev debug aio = true 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug misdirected ops = true 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: bdev async discard = True 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: bdev enable discard = True 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore allocator = bitmap 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore block size = 96636764160 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore fsck on mount = True 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluefs = 1/20 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluestore = 1/20 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-06-29T14:21:38.617 INFO:teuthology.orchestra.run.smithi167.stdout: debug osd = 20 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: debug rocksdb = 4/10 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd backfillfull_ratio = 0.85 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd full ratio = 0.9 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd nearfull ratio = 0.8 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: osd failsafe full ratio = 0.95 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: osd objectstore = bluestore 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr] 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: debug mgr = 20 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-06-29T14:21:38.618 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: mgr/telemetry/nag = false 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: debug client = 20 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: debug finisher = 20 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout:[mon] 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: debug paxos = 20 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: mon data avail warn = 5 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: mon mgr mkfs grace = 240 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd reporter subtree level = osd 2024-06-29T14:21:38.619 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd prime pg temp = true 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: auth mon ticket ttl = 660 # 11m 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: auth service ticket ttl = 240 # 4m 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: # don't complain about insecure global_id in the test suite 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: # 1m isn't quite enough 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: mon_down_mkfs_grace = 2m 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.620 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_filestore_osds = false 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: mon down mkfs grace = 300 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout:[client] 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: rgw cache enabled = true 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable ops log = true 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable usage log = true 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout: pid file = "" 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.a] 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.b] 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.c] 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.a] 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.c] 2024-06-29T14:21:38.621 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.b] 2024-06-29T14:21:38.622 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.d] 2024-06-29T14:21:38.623 INFO:tasks.ceph:Creating admin key on mon.a... 2024-06-29T14:21:38.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-06-29T14:21:38.728 INFO:tasks.ceph:Copying monmap to all nodes... 2024-06-29T14:21:38.728 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:38.728 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-06-29T14:21:38.744 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:38.744 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-06-29T14:21:38.799 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:21:38.799 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:38.799 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-29T14:21:38.799 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-29T14:21:38.873 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:21:38.873 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:38.928 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T14:21:38.928 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:21:38.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-29T14:21:38.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-29T14:21:38.964 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:21:38.964 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:39.020 INFO:tasks.ceph:Setting up mon nodes... 2024-06-29T14:21:39.020 INFO:tasks.ceph:Setting up mgr nodes... 2024-06-29T14:21:39.020 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-06-29T14:21:39.080 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-06-29T14:21:39.083 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-06-29T14:21:39.148 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-06-29T14:21:39.151 INFO:tasks.ceph:Setting up mds nodes... 2024-06-29T14:21:39.151 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-06-29T14:21:39.211 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-06-29T14:21:39.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-06-29T14:21:39.239 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-06-29T14:21:39.339 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-06-29T14:21:39.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-06-29T14:21:39.366 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-06-29T14:21:39.427 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-06-29T14:21:39.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-06-29T14:21:39.455 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-06-29T14:21:39.556 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-06-29T14:21:39.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-06-29T14:21:39.583 INFO:tasks.ceph_client:Setting up client nodes... 2024-06-29T14:21:39.584 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-29T14:21:39.634 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-06-29T14:21:39.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-06-29T14:21:39.697 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-06-29T14:21:39.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-29T14:21:39.760 INFO:teuthology.orchestra.run.smithi167.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-06-29T14:21:39.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-06-29T14:21:39.824 INFO:teuthology.orchestra.run.smithi167.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-06-29T14:21:39.837 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-06-29T14:21:39.838 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-06-29T14:21:39.838 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-06-29T14:21:39.863 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:39.863 INFO:tasks.ceph:role: osd.0 2024-06-29T14:21:39.863 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:21:39.864 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-29T14:21:39.935 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:39.935 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:39.935 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:39.935 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:39.935 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:39.935 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:39.935 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:39.936 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:39.936 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:39.936 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:40.445 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-06-29T14:21:40.447 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-06-29T14:21:40.447 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-06-29T14:21:40.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-06-29T14:21:40.595 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-06-29T14:21:40.662 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:40.662 INFO:tasks.ceph:role: osd.1 2024-06-29T14:21:40.662 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:21:40.662 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-29T14:21:40.736 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:40.736 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:40.736 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:40.736 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:40.736 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:40.736 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:40.736 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:40.737 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:40.737 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:40.737 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:41.197 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-06-29T14:21:41.199 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-06-29T14:21:41.200 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-06-29T14:21:41.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-06-29T14:21:41.327 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-06-29T14:21:41.393 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:41.393 INFO:tasks.ceph:role: osd.2 2024-06-29T14:21:41.394 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:21:41.394 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-29T14:21:41.467 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:41.468 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:41.468 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:41.468 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:41.468 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:41.468 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:41.468 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:41.468 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:41.468 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:41.468 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:41.967 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-06-29T14:21:41.969 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-06-29T14:21:41.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-06-29T14:21:42.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-06-29T14:21:42.152 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-06-29T14:21:42.218 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:42.218 INFO:tasks.ceph:role: osd.3 2024-06-29T14:21:42.218 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:21:42.218 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-06-29T14:21:42.290 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:42.291 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:42.291 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:42.291 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:42.291 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:42.291 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:42.291 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:42.291 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:42.291 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:42.291 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:42.796 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-06-29T14:21:42.799 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-06-29T14:21:42.799 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-06-29T14:21:42.944 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-06-29T14:21:42.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:43.089+0000 7f16d9753640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-06-29T14:21:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:43.090+0000 7f16d9753640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-06-29T14:21:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:43.090+0000 7f16d9753640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-29T14:21:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:43.090+0000 7f16d9753640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-29T14:21:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:43.090+0000 7f16d9753640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-29T14:21:43.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:43.090+0000 7f16d9753640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-06-29T14:21:45.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-29T14:21:45.978 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:46.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:46.081+0000 7f97b0823640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-06-29T14:21:46.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:46.082+0000 7f97b0823640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-06-29T14:21:46.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:46.082+0000 7f97b0823640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-29T14:21:46.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:46.082+0000 7f97b0823640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-29T14:21:46.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:46.082+0000 7f97b0823640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-29T14:21:46.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:46.082+0000 7f97b0823640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-06-29T14:21:48.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-29T14:21:48.959 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:49.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:49.060+0000 7f303dc0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-06-29T14:21:49.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:49.061+0000 7f303dc0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-06-29T14:21:49.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:49.061+0000 7f303dc0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-29T14:21:49.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:49.061+0000 7f303dc0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-29T14:21:49.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:49.061+0000 7f303dc0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-29T14:21:49.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:49.061+0000 7f303dc0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-06-29T14:21:51.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-29T14:21:51.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:52.050+0000 7f167582c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-06-29T14:21:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:52.050+0000 7f167582c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-06-29T14:21:52.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:52.050+0000 7f167582c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-29T14:21:52.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:52.050+0000 7f167582c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-29T14:21:52.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:52.050+0000 7f167582c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-29T14:21:52.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:21:52.050+0000 7f167582c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-06-29T14:21:54.908 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-29T14:21:54.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-06-29T14:21:54.968 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:54.968 INFO:tasks.ceph:role: osd.4 2024-06-29T14:21:54.968 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T14:21:54.969 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-29T14:21:55.044 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:55.045 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:55.045 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:55.045 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:55.045 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:55.045 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:55.569 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-06-29T14:21:55.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-06-29T14:21:55.571 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-06-29T14:21:55.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-06-29T14:21:55.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-06-29T14:21:55.823 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:55.824 INFO:tasks.ceph:role: osd.5 2024-06-29T14:21:55.824 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T14:21:55.824 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-29T14:21:55.897 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:55.898 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:55.898 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:55.898 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:55.898 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:55.898 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:55.898 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:55.898 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:55.899 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:55.899 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:56.342 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-06-29T14:21:56.344 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-06-29T14:21:56.344 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-06-29T14:21:56.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-06-29T14:21:56.462 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-06-29T14:21:56.527 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:56.527 INFO:tasks.ceph:role: osd.6 2024-06-29T14:21:56.527 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T14:21:56.527 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-29T14:21:56.599 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:56.599 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:56.599 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:56.599 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:56.600 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:56.600 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:56.600 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:57.099 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-06-29T14:21:57.102 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-06-29T14:21:57.102 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-06-29T14:21:57.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-06-29T14:21:57.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-06-29T14:21:57.348 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-29T14:21:57.348 INFO:tasks.ceph:role: osd.7 2024-06-29T14:21:57.348 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T14:21:57.348 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-06-29T14:21:57.420 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-06-29T14:21:57.421 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-29T14:21:57.421 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-29T14:21:57.421 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-29T14:21:57.421 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-29T14:21:57.421 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2024-06-29T14:21:57.421 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-29T14:21:57.421 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-29T14:21:57.421 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-29T14:21:57.421 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-29T14:21:57.931 INFO:teuthology.orchestra.run.smithi167.stdout:Discarding blocks...Done. 2024-06-29T14:21:57.933 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2024-06-29T14:21:57.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-06-29T14:21:58.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-06-29T14:21:58.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:21:58.208 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:21:58.209+0000 7f8eae40d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-06-29T14:21:58.208 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:21:58.209+0000 7f8eae40d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-06-29T14:21:58.208 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:21:58.209+0000 7f8eae40d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-29T14:21:58.208 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:21:58.209+0000 7f8eae40d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-29T14:21:58.208 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:21:58.209+0000 7f8eae40d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-29T14:21:58.209 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:21:58.210+0000 7f8eae40d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-06-29T14:22:01.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-29T14:22:01.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:22:01.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:01.185+0000 7fe24920e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-06-29T14:22:01.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:01.186+0000 7fe24920e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-06-29T14:22:01.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:01.186+0000 7fe24920e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-29T14:22:01.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:01.186+0000 7fe24920e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-29T14:22:01.185 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:01.186+0000 7fe24920e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-29T14:22:01.185 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:01.186+0000 7fe24920e640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-06-29T14:22:04.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-29T14:22:04.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:22:04.168 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:04.169+0000 7fc2dccd5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-06-29T14:22:04.168 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:04.170+0000 7fc2dccd5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-06-29T14:22:04.168 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:04.170+0000 7fc2dccd5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-29T14:22:04.168 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:04.170+0000 7fc2dccd5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-29T14:22:04.169 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:04.170+0000 7fc2dccd5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-29T14:22:04.169 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:04.170+0000 7fc2dccd5640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-06-29T14:22:06.619 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:22:06.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:22:07.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-29T14:22:07.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:22:07.144 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:07.145+0000 7fac34216640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-06-29T14:22:07.144 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:07.145+0000 7fac34216640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-06-29T14:22:07.144 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:07.145+0000 7fac34216640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-29T14:22:07.144 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:07.145+0000 7fac34216640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-29T14:22:07.144 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:07.145+0000 7fac34216640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-29T14:22:07.144 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:07.146+0000 7fac34216640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-06-29T14:22:10.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-29T14:22:10.040 INFO:tasks.ceph:Reading keys from all nodes... 2024-06-29T14:22:10.041 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:10.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-06-29T14:22:10.071 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:10.071 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-06-29T14:22:10.139 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:10.139 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-06-29T14:22:10.206 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:10.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-06-29T14:22:10.273 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:10.273 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-06-29T14:22:10.339 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:10.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-06-29T14:22:10.405 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:10.406 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-06-29T14:22:10.474 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:10.474 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-06-29T14:22:10.504 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:10.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-06-29T14:22:10.574 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:10.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-06-29T14:22:10.642 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:10.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-06-29T14:22:10.709 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:10.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-06-29T14:22:10.775 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:10.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-06-29T14:22:10.841 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:10.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-06-29T14:22:10.907 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:10.907 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-06-29T14:22:10.923 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:10.923 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-06-29T14:22:10.980 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:10.980 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-06-29T14:22:10.996 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:10.996 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-06-29T14:22:11.052 INFO:tasks.ceph:Adding keys to all mons... 2024-06-29T14:22:11.052 DEBUG:teuthology.orchestra.run.smithi081:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-29T14:22:11.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-29T14:22:11.095 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.y] 2024-06-29T14:22:11.096 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ== 2024-06-29T14:22:11.096 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.a] 2024-06-29T14:22:11.096 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw== 2024-06-29T14:22:11.096 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.c] 2024-06-29T14:22:11.096 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw== 2024-06-29T14:22:11.096 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2024-06-29T14:22:11.096 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ== 2024-06-29T14:22:11.096 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2024-06-29T14:22:11.096 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ== 2024-06-29T14:22:11.096 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2024-06-29T14:22:11.097 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw== 2024-06-29T14:22:11.097 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.3] 2024-06-29T14:22:11.097 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g== 2024-06-29T14:22:11.097 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.x] 2024-06-29T14:22:11.097 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw== 2024-06-29T14:22:11.097 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.b] 2024-06-29T14:22:11.097 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g== 2024-06-29T14:22:11.097 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.d] 2024-06-29T14:22:11.097 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA== 2024-06-29T14:22:11.097 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.4] 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA== 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.5] 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA== 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.6] 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ== 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.7] 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g== 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout:[client.2] 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ== 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout:[client.1] 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA== 2024-06-29T14:22:11.098 INFO:teuthology.orchestra.run.smithi081.stdout:[client.3] 2024-06-29T14:22:11.099 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw== 2024-06-29T14:22:11.116 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.y] 2024-06-29T14:22:11.116 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ== 2024-06-29T14:22:11.116 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.a] 2024-06-29T14:22:11.116 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw== 2024-06-29T14:22:11.116 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.c] 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw== 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.0] 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ== 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.1] 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ== 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.2] 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw== 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.3] 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g== 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr.x] 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw== 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.b] 2024-06-29T14:22:11.117 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g== 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout:[mds.d] 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA== 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.4] 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA== 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.5] 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA== 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.6] 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ== 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout:[osd.7] 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g== 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout:[client.0] 2024-06-29T14:22:11.118 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T14:22:11.119 INFO:teuthology.orchestra.run.smithi167.stdout:[client.2] 2024-06-29T14:22:11.119 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ== 2024-06-29T14:22:11.119 INFO:teuthology.orchestra.run.smithi167.stdout:[client.1] 2024-06-29T14:22:11.119 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA== 2024-06-29T14:22:11.119 INFO:teuthology.orchestra.run.smithi167.stdout:[client.3] 2024-06-29T14:22:11.119 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw== 2024-06-29T14:22:11.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-29T14:22:11.142 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-29T14:22:11.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:22:11.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:22:11.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:22:11.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:22:11.358 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.489 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.614 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-29T14:22:11.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-29T14:22:11.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:22:11.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:22:11.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:22:11.749 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-29T14:22:11.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.940 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:11.942 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:12.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:12.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-29T14:22:12.068 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:22:12.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:22:12.134 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:22:12.137 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:22:12.201 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:22:12.203 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:22:12.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:22:12.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-29T14:22:12.333 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-06-29T14:22:12.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-06-29T14:22:12.358 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-29T14:22:12.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-06-29T14:22:12.489 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-06-29T14:22:12.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-29T14:22:12.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-06-29T14:22:12.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-06-29T14:22:12.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-29T14:22:12.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-06-29T14:22:12.853 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:22:12.856 DEBUG:teuthology.orchestra.run.smithi167:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-29T14:22:12.912 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-06-29T14:22:12.913 INFO:tasks.ceph.mon.a:Restarting daemon 2024-06-29T14:22:12.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-06-29T14:22:12.916 INFO:tasks.ceph.mon.a:Started 2024-06-29T14:22:12.917 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-29T14:22:12.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-06-29T14:22:12.959 INFO:tasks.ceph.mon.b:Started 2024-06-29T14:22:12.960 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-29T14:22:12.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-06-29T14:22:12.965 INFO:tasks.ceph.mon.c:Started 2024-06-29T14:22:12.965 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-06-29T14:22:12.965 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-06-29T14:22:12.965 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-06-29T14:22:12.970 INFO:tasks.ceph.mgr.y:Started 2024-06-29T14:22:12.970 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-06-29T14:22:12.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-06-29T14:22:12.977 INFO:tasks.ceph.mgr.x:Started 2024-06-29T14:22:12.977 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-06-29T14:22:12.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-06-29T14:22:12.982 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-06-29T14:22:12.982 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set client debug_client 20 2024-06-29T14:22:12.986 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-06-29T14:22:12.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set client debug_ms 1 2024-06-29T14:22:13.032 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-06-29T14:22:13.033 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-06-29T14:22:13.078 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-06-29T14:22:13.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-06-29T14:22:13.126 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-06-29T14:22:13.126 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-06-29T14:22:13.173 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-06-29T14:22:13.174 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-06-29T14:22:13.221 INFO:tasks.ceph:setting config [mds] debug_ms = 1 2024-06-29T14:22:13.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mds debug_ms 1 2024-06-29T14:22:13.267 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-06-29T14:22:13.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-06-29T14:22:13.313 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-06-29T14:22:13.313 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-06-29T14:22:13.358 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-06-29T14:22:13.358 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-06-29T14:22:13.403 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-06-29T14:22:13.403 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-06-29T14:22:13.447 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-06-29T14:22:13.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-06-29T14:22:13.491 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-06-29T14:22:13.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-06-29T14:22:13.538 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-06-29T14:22:13.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-06-29T14:22:13.581 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-06-29T14:22:13.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-06-29T14:22:13.624 INFO:tasks.ceph:setting config [mon] mon_warn_on_pool_no_app = False 2024-06-29T14:22:13.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set mon mon_warn_on_pool_no_app False 2024-06-29T14:22:13.667 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-06-29T14:22:13.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-06-29T14:22:13.711 DEBUG:tasks.ceph:set 18 configs 2024-06-29T14:22:13.711 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi081') 2024-06-29T14:22:18.460 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi081') 2024-06-29T14:22:18.622 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.233+0000 7fdbcb568640 1 -- 172.21.15.81:0/3505253720 shutdown_connections 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.233+0000 7fdbcb568640 1 --2- 172.21.15.81:0/3505253720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbc40b2fe0 0x7fdbc40b33f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.233+0000 7fdbcb568640 1 --2- 172.21.15.81:0/3505253720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbc409b660 0x7fdbc409ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.233+0000 7fdbcb568640 1 --2- 172.21.15.81:0/3505253720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbc414d660 0x7fdbc409b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.233+0000 7fdbcb568640 1 -- 172.21.15.81:0/3505253720 >> 172.21.15.81:0/3505253720 conn(0x7fdbc40b1ec0 msgr2=0x7fdbc40b0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.233+0000 7fdbcb568640 1 -- 172.21.15.81:0/3505253720 shutdown_connections 2024-06-29T14:22:19.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.233+0000 7fdbcb568640 1 -- 172.21.15.81:0/3505253720 wait complete. 2024-06-29T14:22:19.242 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.282+0000 7f3e01943640 1 -- 172.21.15.81:0/3331635213 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f3dfc0ccc90 con 0x7f3de8006b40 2024-06-29T14:22:19.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.288+0000 7f3de3fff640 1 -- 172.21.15.81:0/3331635213 <== mon.2 v2:172.21.15.167:3301/0 8 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3df0026c50 con 0x7f3de8006b40 2024-06-29T14:22:19.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.288+0000 7f3de3fff640 1 -- 172.21.15.81:0/3331635213 <== mon.2 v2:172.21.15.167:3301/0 9 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v18)=0 v18) ==== 123+0+0 (secure 0 0 0) 0x7f3df002f3f0 con 0x7f3de8006b40 2024-06-29T14:22:19.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7f3e01943640 1 -- 172.21.15.81:0/3331635213 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3de8006b40 msgr2=0x7f3dfc14fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7f3e01943640 1 --2- 172.21.15.81:0/3331635213 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3de8006b40 0x7f3dfc14fbd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3de80060a0 tx=0x7f3df0014830 comp rx=0 tx=0).stop 2024-06-29T14:22:19.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7f3e01943640 1 -- 172.21.15.81:0/3331635213 shutdown_connections 2024-06-29T14:22:19.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7f3e01943640 1 --2- 172.21.15.81:0/3331635213 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3de8008820 0x7f3dfc072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7f3e01943640 1 --2- 172.21.15.81:0/3331635213 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3de8006b40 0x7f3dfc14fbd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7f3e01943640 1 --2- 172.21.15.81:0/3331635213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de8003af0 0x7f3dfc15f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7f3e01943640 1 -- 172.21.15.81:0/3331635213 >> 172.21.15.81:0/3331635213 conn(0x7f3dfc0b1ec0 msgr2=0x7f3dfc15c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7fe60a1f0640 1 -- 172.21.15.81:0/2429823056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5f0004060 msgr2=0x7fe5f0004490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7fe60a1f0640 1 --2- 172.21.15.81:0/2429823056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5f0004060 0x7fe5f0004490 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe604075820 tx=0x7fe5f802b7e0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7fe60a1f0640 1 -- 172.21.15.81:0/2429823056 shutdown_connections 2024-06-29T14:22:19.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7fe60a1f0640 1 --2- 172.21.15.81:0/2429823056 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5f0004af0 0x7fe5f0004ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7f3e01943640 1 -- 172.21.15.81:0/3331635213 shutdown_connections 2024-06-29T14:22:19.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7fe60a1f0640 1 --2- 172.21.15.81:0/2429823056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5f0004060 0x7fe5f0004490 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7fe60a1f0640 1 --2- 172.21.15.81:0/2429823056 >> v2:172.21.15.167:3300/0 conn(0x7fe5f0006690 0x7fe5f0003b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7fe60a1f0640 1 -- 172.21.15.81:0/2429823056 >> 172.21.15.81:0/2429823056 conn(0x7fe6040b1ec0 msgr2=0x7fe6040b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7f3e01943640 1 -- 172.21.15.81:0/3331635213 wait complete. 2024-06-29T14:22:19.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7fe60a1f0640 1 -- 172.21.15.81:0/2429823056 shutdown_connections 2024-06-29T14:22:19.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.289+0000 7fe60a1f0640 1 -- 172.21.15.81:0/2429823056 wait complete. 2024-06-29T14:22:19.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.292+0000 7fe60a1f0640 1 Processor -- start 2024-06-29T14:22:19.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.292+0000 7fe60a1f0640 1 -- start start 2024-06-29T14:22:19.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.294+0000 7fe60a1f0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5f0004060 0x7fe604158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.294+0000 7fe60a1f0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5f0004af0 0x7fe6040cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.294+0000 7fe60a1f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5f0006690 0x7fe6040b3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.294+0000 7fe60a1f0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6040cc8c0 con 0x7fe5f0006690 2024-06-29T14:22:19.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.294+0000 7fe60a1f0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe60406c770 con 0x7fe5f0004060 2024-06-29T14:22:19.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.294+0000 7fe60a1f0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe6040cca40 con 0x7fe5f0004af0 2024-06-29T14:22:19.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe603fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5f0006690 0x7fe6040b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe6037fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5f0004060 0x7fe604158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe602ffd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5f0004af0 0x7fe6040cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe603fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5f0006690 0x7fe6040b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38994/0 (socket says 172.21.15.81:38994) 2024-06-29T14:22:19.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe603fff640 1 -- 172.21.15.81:0/3279907083 learned_addr learned my addr 172.21.15.81:0/3279907083 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:19.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe602ffd640 1 -- 172.21.15.81:0/3279907083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5f0004060 msgr2=0x7fe604158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe602ffd640 1 --2- 172.21.15.81:0/3279907083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5f0004060 0x7fe604158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe602ffd640 1 -- 172.21.15.81:0/3279907083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5f0006690 msgr2=0x7fe6040b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe602ffd640 1 --2- 172.21.15.81:0/3279907083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5f0006690 0x7fe6040b3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe602ffd640 1 -- 172.21.15.81:0/3279907083 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60415e780 con 0x7fe5f0004af0 2024-06-29T14:22:19.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe6037fe640 1 --2- 172.21.15.81:0/3279907083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5f0004060 0x7fe604158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:19.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe603fff640 1 --2- 172.21.15.81:0/3279907083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5f0006690 0x7fe6040b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:19.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe602ffd640 1 --2- 172.21.15.81:0/3279907083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5f0004af0 0x7fe6040cbdc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8033000 tx=0x7fe5f8000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:19.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.295+0000 7fe5ebfff640 1 -- 172.21.15.81:0/3279907083 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8043070 con 0x7fe5f0004af0 2024-06-29T14:22:19.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.296+0000 7fe5ebfff640 1 -- 172.21.15.81:0/3279907083 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5f803e760 con 0x7fe5f0004af0 2024-06-29T14:22:19.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.296+0000 7fe5ebfff640 1 -- 172.21.15.81:0/3279907083 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8039040 con 0x7fe5f0004af0 2024-06-29T14:22:19.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.296+0000 7fe60a1f0640 1 -- 172.21.15.81:0/3279907083 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6041f4f10 con 0x7fe5f0004af0 2024-06-29T14:22:19.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.296+0000 7fe60a1f0640 1 -- 172.21.15.81:0/3279907083 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6041f54d0 con 0x7fe5f0004af0 2024-06-29T14:22:19.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.296+0000 7fe5ebfff640 1 -- 172.21.15.81:0/3279907083 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe5f803c070 con 0x7fe5f0004af0 2024-06-29T14:22:19.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.296+0000 7fe5ebfff640 1 -- 172.21.15.81:0/3279907083 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe5f8047900 con 0x7fe5f0004af0 2024-06-29T14:22:19.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.297+0000 7fe60a1f0640 1 -- 172.21.15.81:0/3279907083 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6041f4c80 con 0x7fe5f0004af0 2024-06-29T14:22:19.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.299+0000 7fe5ebfff640 1 -- 172.21.15.81:0/3279907083 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fe5f8047400 con 0x7fe5f0004af0 2024-06-29T14:22:19.300 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.300 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.361+0000 7fe60a1f0640 1 -- 172.21.15.81:0/3279907083 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fe60415c8c0 con 0x7fe5f0004af0 2024-06-29T14:22:19.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.368+0000 7fe5ebfff640 1 -- 172.21.15.81:0/3279907083 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v19)=0 v19) ==== 108+0+0 (secure 0 0 0) 0x7fe60415c8c0 con 0x7fe5f0004af0 2024-06-29T14:22:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.368+0000 7fe60a1f0640 1 -- 172.21.15.81:0/3279907083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5f0004af0 msgr2=0x7fe6040cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.368+0000 7fe60a1f0640 1 --2- 172.21.15.81:0/3279907083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5f0004af0 0x7fe6040cbdc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8033000 tx=0x7fe5f8000ed0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.369+0000 7fe60a1f0640 1 -- 172.21.15.81:0/3279907083 shutdown_connections 2024-06-29T14:22:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.369+0000 7fe60a1f0640 1 --2- 172.21.15.81:0/3279907083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5f0006690 0x7fe6040b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.369+0000 7fe60a1f0640 1 --2- 172.21.15.81:0/3279907083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5f0004af0 0x7fe6040cbdc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.369+0000 7fe60a1f0640 1 --2- 172.21.15.81:0/3279907083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5f0004060 0x7fe604158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.369+0000 7fe60a1f0640 1 -- 172.21.15.81:0/3279907083 >> 172.21.15.81:0/3279907083 conn(0x7fe6040b1ec0 msgr2=0x7fe60415d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.369+0000 7fe60a1f0640 1 -- 172.21.15.81:0/3279907083 shutdown_connections 2024-06-29T14:22:19.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.369+0000 7fe60a1f0640 1 -- 172.21.15.81:0/3279907083 wait complete. 2024-06-29T14:22:19.377 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.378 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '1'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.378 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.378 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.378 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.378 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.378 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.378 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.378 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.379 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.379 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.379 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi081') 2024-06-29T14:22:19.379 INFO:tasks.ceph:Setting crush tunables to default 2024-06-29T14:22:19.379 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd crush tunables default 2024-06-29T14:22:19.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.495+0000 7f638ab7a640 1 -- 172.21.15.81:0/3736751153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63840c5160 msgr2=0x7f63840ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.495+0000 7f638ab7a640 1 --2- 172.21.15.81:0/3736751153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63840c5160 0x7f63840ce0d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6378009830 tx=0x7f637802a510 comp rx=0 tx=0).stop 2024-06-29T14:22:19.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.495+0000 7f638ab7a640 1 -- 172.21.15.81:0/3736751153 shutdown_connections 2024-06-29T14:22:19.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.495+0000 7f638ab7a640 1 --2- 172.21.15.81:0/3736751153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f638406ce70 0x7f638406d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.495+0000 7f638ab7a640 1 --2- 172.21.15.81:0/3736751153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63840c5160 0x7f63840ce0d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.495+0000 7f638ab7a640 1 -- 172.21.15.81:0/3736751153 >> 172.21.15.81:0/3736751153 conn(0x7f63840d00b0 msgr2=0x7f63840d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.495+0000 7f638ab7a640 1 -- 172.21.15.81:0/3736751153 shutdown_connections 2024-06-29T14:22:19.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.495+0000 7f638ab7a640 1 -- 172.21.15.81:0/3736751153 wait complete. 2024-06-29T14:22:19.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.497+0000 7f638ab7a640 1 Processor -- start 2024-06-29T14:22:19.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.497+0000 7f638ab7a640 1 -- start start 2024-06-29T14:22:19.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.498+0000 7f638ab7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f638406ce70 0x7f6384152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.498+0000 7f638ab7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63840c5160 0x7f638409d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f638ab7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63840ca700 0x7f63840c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f638ab7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f63840cfa90 con 0x7f63840c5160 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f638ab7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f63840cf910 con 0x7f63840ca700 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f638ab7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f638406d7c0 con 0x7f638406ce70 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6383fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63840c5160 0x7f638409d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6383fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63840c5160 0x7f638409d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39006/0 (socket says 172.21.15.81:39006) 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6383fff640 1 -- 172.21.15.81:0/1890610944 learned_addr learned my addr 172.21.15.81:0/1890610944 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6388916640 1 --2- 172.21.15.81:0/1890610944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f638406ce70 0x7f6384152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6389117640 1 --2- 172.21.15.81:0/1890610944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63840ca700 0x7f63840c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6383fff640 1 -- 172.21.15.81:0/1890610944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f638406ce70 msgr2=0x7f6384152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6383fff640 1 --2- 172.21.15.81:0/1890610944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f638406ce70 0x7f6384152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6383fff640 1 -- 172.21.15.81:0/1890610944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63840ca700 msgr2=0x7f63840c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6383fff640 1 --2- 172.21.15.81:0/1890610944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63840ca700 0x7f63840c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6383fff640 1 -- 172.21.15.81:0/1890610944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638409c7b0 con 0x7f63840c5160 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6389117640 1 --2- 172.21.15.81:0/1890610944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63840ca700 0x7f63840c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f6383fff640 1 --2- 172.21.15.81:0/1890610944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63840c5160 0x7f638409d300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f637000c3a0 tx=0x7f637000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f63817fa640 1 -- 172.21.15.81:0/1890610944 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6370011790 con 0x7f63840c5160 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f63817fa640 1 -- 172.21.15.81:0/1890610944 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6370017070 con 0x7f63840c5160 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f638ab7a640 1 -- 172.21.15.81:0/1890610944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63840c9800 con 0x7f63840c5160 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f63817fa640 1 -- 172.21.15.81:0/1890610944 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6370011cc0 con 0x7f63840c5160 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.499+0000 7f638ab7a640 1 -- 172.21.15.81:0/1890610944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63840c9dc0 con 0x7f63840c5160 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.500+0000 7f63817fa640 1 -- 172.21.15.81:0/1890610944 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f637001c460 con 0x7f63840c5160 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.500+0000 7f63817fa640 1 -- 172.21.15.81:0/1890610944 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6370012810 con 0x7f63840c5160 2024-06-29T14:22:19.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.500+0000 7f638ab7a640 1 -- 172.21.15.81:0/1890610944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f638409ad70 con 0x7f63840c5160 2024-06-29T14:22:19.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.502+0000 7f63817fa640 1 -- 172.21.15.81:0/1890610944 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f638409ad70 con 0x7f63840c5160 2024-06-29T14:22:19.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.568+0000 7f638ab7a640 1 -- 172.21.15.81:0/1890610944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f63840cf650 con 0x7f63840c5160 2024-06-29T14:22:19.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.572+0000 7f63817fa640 1 -- 172.21.15.81:0/1890610944 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f63840cf650 con 0x7f63840c5160 2024-06-29T14:22:19.570 INFO:teuthology.orchestra.run.smithi081.stderr:adjusted tunables profile to default 2024-06-29T14:22:19.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.573+0000 7f638ab7a640 1 -- 172.21.15.81:0/1890610944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63840c5160 msgr2=0x7f638409d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:19.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.573+0000 7f638ab7a640 1 --2- 172.21.15.81:0/1890610944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63840c5160 0x7f638409d300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f637000c3a0 tx=0x7f637000c860 comp rx=0 tx=0).stop 2024-06-29T14:22:19.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.573+0000 7f638ab7a640 1 -- 172.21.15.81:0/1890610944 shutdown_connections 2024-06-29T14:22:19.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.573+0000 7f638ab7a640 1 --2- 172.21.15.81:0/1890610944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63840ca700 0x7f63840c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.573+0000 7f638ab7a640 1 --2- 172.21.15.81:0/1890610944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63840c5160 0x7f638409d300 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.573+0000 7f638ab7a640 1 --2- 172.21.15.81:0/1890610944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f638406ce70 0x7f6384152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:19.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.573+0000 7f638ab7a640 1 -- 172.21.15.81:0/1890610944 >> 172.21.15.81:0/1890610944 conn(0x7f63840d00b0 msgr2=0x7f6384073a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:19.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.573+0000 7f638ab7a640 1 -- 172.21.15.81:0/1890610944 shutdown_connections 2024-06-29T14:22:19.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:19.573+0000 7f638ab7a640 1 -- 172.21.15.81:0/1890610944 wait complete. 2024-06-29T14:22:19.582 INFO:tasks.ceph:check_enable_crimson: False 2024-06-29T14:22:19.582 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-06-29T14:22:19.582 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:19.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-06-29T14:22:19.607 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:19.607 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-06-29T14:22:19.672 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:19.672 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-06-29T14:22:19.738 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:22:19.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-06-29T14:22:19.803 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:19.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-06-29T14:22:19.830 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:19.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-06-29T14:22:19.895 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:19.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-06-29T14:22:19.961 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:22:19.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-06-29T14:22:20.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 41a27439-5583-43cf-8abe-2131cda2eb06 0 2024-06-29T14:22:20.182 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.183+0000 7f0be4865640 1 -- 172.21.15.167:0/1107716345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bc8004800 msgr2=0x7f0bc8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.182 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.183+0000 7f0be4865640 1 --2- 172.21.15.167:0/1107716345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bc8004800 0x7f0bc8004c10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4006fc0 tx=0x7f0bc8005830 comp rx=0 tx=0).stop 2024-06-29T14:22:20.183 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.183+0000 7f0be4865640 1 -- 172.21.15.167:0/1107716345 shutdown_connections 2024-06-29T14:22:20.183 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.183+0000 7f0be4865640 1 --2- 172.21.15.167:0/1107716345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0bc8004800 0x7f0bc8004c10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.183 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.183+0000 7f0be4865640 1 -- 172.21.15.167:0/1107716345 >> 172.21.15.167:0/1107716345 conn(0x7f0be00d0b70 msgr2=0x7f0be00d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.183 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.183+0000 7f0be4865640 1 -- 172.21.15.167:0/1107716345 shutdown_connections 2024-06-29T14:22:20.183 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.184+0000 7f0be4865640 1 -- 172.21.15.167:0/1107716345 wait complete. 2024-06-29T14:22:20.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.185+0000 7f0be4865640 1 Processor -- start 2024-06-29T14:22:20.184 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.185+0000 7f0be4865640 1 -- start start 2024-06-29T14:22:20.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0be4865640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bc8004800 0x7f0be0137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0be4865640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0be015a130 0x7f0be0141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0be4865640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be015a650 0x7f0be01422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0be4865640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0be00732d0 con 0x7f0be015a650 2024-06-29T14:22:20.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0be4865640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0be0073150 con 0x7f0be015a130 2024-06-29T14:22:20.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0be4865640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0be01384a0 con 0x7f0bc8004800 2024-06-29T14:22:20.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bde59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bc8004800 0x7f0be0137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bddd9b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0be015a130 0x7f0be0141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bde59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bc8004800 0x7f0be0137c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:48142/0 (socket says 172.21.15.167:48142) 2024-06-29T14:22:20.187 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bde59c640 1 -- 172.21.15.167:0/2021985553 learned_addr learned my addr 172.21.15.167:0/2021985553 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bded9d640 1 --2- 172.21.15.167:0/2021985553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be015a650 0x7f0be01422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bde59c640 1 -- 172.21.15.167:0/2021985553 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0be015a130 msgr2=0x7f0be0141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bde59c640 1 --2- 172.21.15.167:0/2021985553 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0be015a130 0x7f0be0141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bde59c640 1 -- 172.21.15.167:0/2021985553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be015a650 msgr2=0x7f0be01422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bde59c640 1 --2- 172.21.15.167:0/2021985553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be015a650 0x7f0be01422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bde59c640 1 -- 172.21.15.167:0/2021985553 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be013d100 con 0x7f0bc8004800 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bddd9b640 1 --2- 172.21.15.167:0/2021985553 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0be015a130 0x7f0be0141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bde59c640 1 --2- 172.21.15.167:0/2021985553 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bc8004800 0x7f0be0137c10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0011ab0 tx=0x7f0bd0011f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bceffd640 1 -- 172.21.15.167:0/2021985553 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd001e070 con 0x7f0bc8004800 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.187+0000 7f0bceffd640 1 -- 172.21.15.167:0/2021985553 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0bd000fc70 con 0x7f0bc8004800 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.188+0000 7f0be4865640 1 -- 172.21.15.167:0/2021985553 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be0144030 con 0x7f0bc8004800 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.188+0000 7f0bceffd640 1 -- 172.21.15.167:0/2021985553 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0019670 con 0x7f0bc8004800 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.188+0000 7f0be4865640 1 -- 172.21.15.167:0/2021985553 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be0143d50 con 0x7f0bc8004800 2024-06-29T14:22:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.188+0000 7f0bceffd640 1 -- 172.21.15.167:0/2021985553 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0bd0021070 con 0x7f0bc8004800 2024-06-29T14:22:20.189 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.188+0000 7f0bceffd640 1 -- 172.21.15.167:0/2021985553 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f0bd00238c0 con 0x7f0bc8004800 2024-06-29T14:22:20.189 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.188+0000 7f0be4865640 1 -- 172.21.15.167:0/2021985553 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be015b5c0 con 0x7f0bc8004800 2024-06-29T14:22:20.189 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.191+0000 7f0bceffd640 1 -- 172.21.15.167:0/2021985553 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f0be013d100 con 0x7f0bc8004800 2024-06-29T14:22:20.262 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.264+0000 7f0be4865640 1 -- 172.21.15.167:0/2021985553 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd new", "uuid": "41a27439-5583-43cf-8abe-2131cda2eb06", "id": 0} v 0) -- 0x7f0be015cc60 con 0x7f0bc8004800 2024-06-29T14:22:20.266 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.267+0000 7f0bceffd640 1 -- 172.21.15.167:0/2021985553 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "41a27439-5583-43cf-8abe-2131cda2eb06", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f0bd0023b10 con 0x7f0bc8004800 2024-06-29T14:22:20.266 INFO:teuthology.orchestra.run.smithi167.stdout:0 2024-06-29T14:22:20.268 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.268+0000 7f0be4865640 1 -- 172.21.15.167:0/2021985553 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bc8004800 msgr2=0x7f0be0137c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.268 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.268+0000 7f0be4865640 1 --2- 172.21.15.167:0/2021985553 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bc8004800 0x7f0be0137c10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0011ab0 tx=0x7f0bd0011f70 comp rx=0 tx=0).stop 2024-06-29T14:22:20.268 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.268+0000 7f0be4865640 1 -- 172.21.15.167:0/2021985553 shutdown_connections 2024-06-29T14:22:20.268 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.268+0000 7f0be4865640 1 --2- 172.21.15.167:0/2021985553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be015a650 0x7f0be01422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.268 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.268+0000 7f0be4865640 1 --2- 172.21.15.167:0/2021985553 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0be015a130 0x7f0be0141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.268 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.268+0000 7f0be4865640 1 --2- 172.21.15.167:0/2021985553 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bc8004800 0x7f0be0137c10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.269 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.268+0000 7f0be4865640 1 -- 172.21.15.167:0/2021985553 >> 172.21.15.167:0/2021985553 conn(0x7f0be00d0b70 msgr2=0x7f0be00cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.269 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.268+0000 7f0be4865640 1 -- 172.21.15.167:0/2021985553 shutdown_connections 2024-06-29T14:22:20.269 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.268+0000 7f0be4865640 1 -- 172.21.15.167:0/2021985553 wait complete. 2024-06-29T14:22:20.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new a4372bb3-c9cd-484e-9651-43ad2cce450a 1 2024-06-29T14:22:20.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.394+0000 7fdd31731640 1 -- 172.21.15.167:0/178160931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd2c073480 msgr2=0x7fdd2c15b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.394+0000 7fdd31731640 1 --2- 172.21.15.167:0/178160931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd2c073480 0x7fdd2c15b1e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009900 tx=0x7fdd1c02a610 comp rx=0 tx=0).stop 2024-06-29T14:22:20.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.394+0000 7fdd31731640 1 -- 172.21.15.167:0/178160931 shutdown_connections 2024-06-29T14:22:20.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.394+0000 7fdd31731640 1 --2- 172.21.15.167:0/178160931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd2c073480 0x7fdd2c15b1e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.394+0000 7fdd31731640 1 --2- 172.21.15.167:0/178160931 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd2c0b46a0 0x7fdd2c072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.393 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.394+0000 7fdd31731640 1 -- 172.21.15.167:0/178160931 >> 172.21.15.167:0/178160931 conn(0x7fdd2c0d0b70 msgr2=0x7fdd2c0d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.394+0000 7fdd31731640 1 -- 172.21.15.167:0/178160931 shutdown_connections 2024-06-29T14:22:20.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.394+0000 7fdd31731640 1 -- 172.21.15.167:0/178160931 wait complete. 2024-06-29T14:22:20.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.396+0000 7fdd31731640 1 Processor -- start 2024-06-29T14:22:20.394 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.396+0000 7fdd31731640 1 -- start start 2024-06-29T14:22:20.397 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.397+0000 7fdd31731640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd2c073480 0x7fdd2c1e75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.397 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.397+0000 7fdd31731640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd2c0b46a0 0x7fdd2c1e7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.397 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.397+0000 7fdd31731640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd2c15b720 0x7fdd2c06c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.397+0000 7fdd31731640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c06f850 con 0x7fdd2c15b720 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.397+0000 7fdd31731640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c06f6d0 con 0x7fdd2c073480 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.397+0000 7fdd31731640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdd2c06f9d0 con 0x7fdd2c0b46a0 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.397+0000 7fdd2a7fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd2c0b46a0 0x7fdd2c1e7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.397+0000 7fdd2affd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd2c073480 0x7fdd2c1e75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.397+0000 7fdd2a7fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd2c0b46a0 0x7fdd2c1e7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:48162/0 (socket says 172.21.15.167:48162) 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.397+0000 7fdd2a7fc640 1 -- 172.21.15.167:0/1869114336 learned_addr learned my addr 172.21.15.167:0/1869114336 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd2b7fe640 1 --2- 172.21.15.167:0/1869114336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd2c15b720 0x7fdd2c06c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd2affd640 1 -- 172.21.15.167:0/1869114336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd2c0b46a0 msgr2=0x7fdd2c1e7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd2affd640 1 --2- 172.21.15.167:0/1869114336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd2c0b46a0 0x7fdd2c1e7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd2affd640 1 -- 172.21.15.167:0/1869114336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd2c15b720 msgr2=0x7fdd2c06c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd2affd640 1 --2- 172.21.15.167:0/1869114336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd2c15b720 0x7fdd2c06c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.398 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd2affd640 1 -- 172.21.15.167:0/1869114336 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2c13c140 con 0x7fdd2c073480 2024-06-29T14:22:20.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd2affd640 1 --2- 172.21.15.167:0/1869114336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd2c073480 0x7fdd2c1e75e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400e6d0 tx=0x7fdd1400eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:20.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd237fe640 1 -- 172.21.15.167:0/1869114336 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1400cab0 con 0x7fdd2c073480 2024-06-29T14:22:20.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd237fe640 1 -- 172.21.15.167:0/1869114336 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd1400cd40 con 0x7fdd2c073480 2024-06-29T14:22:20.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd31731640 1 -- 172.21.15.167:0/1869114336 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd2c13c420 con 0x7fdd2c073480 2024-06-29T14:22:20.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd237fe640 1 -- 172.21.15.167:0/1869114336 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14010690 con 0x7fdd2c073480 2024-06-29T14:22:20.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd31731640 1 -- 172.21.15.167:0/1869114336 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd2c13c9e0 con 0x7fdd2c073480 2024-06-29T14:22:20.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd237fe640 1 -- 172.21.15.167:0/1869114336 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdd14010830 con 0x7fdd2c073480 2024-06-29T14:22:20.399 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.398+0000 7fdd237fe640 1 -- 172.21.15.167:0/1869114336 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fdd140217a0 con 0x7fdd2c073480 2024-06-29T14:22:20.400 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.399+0000 7fdd31731640 1 -- 172.21.15.167:0/1869114336 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd2c15b1e0 con 0x7fdd2c073480 2024-06-29T14:22:20.400 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.401+0000 7fdd237fe640 1 -- 172.21.15.167:0/1869114336 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fdd14018c20 con 0x7fdd2c073480 2024-06-29T14:22:20.474 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.476+0000 7fdd31731640 1 -- 172.21.15.167:0/1869114336 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a4372bb3-c9cd-484e-9651-43ad2cce450a", "id": 1} v 0) -- 0x7fdd2c13bbd0 con 0x7fdd2c073480 2024-06-29T14:22:20.477 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.479+0000 7fdd237fe640 1 -- 172.21.15.167:0/1869114336 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a4372bb3-c9cd-484e-9651-43ad2cce450a", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fdd14018e00 con 0x7fdd2c073480 2024-06-29T14:22:20.478 INFO:teuthology.orchestra.run.smithi167.stdout:1 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.480+0000 7fdd31731640 1 -- 172.21.15.167:0/1869114336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd2c073480 msgr2=0x7fdd2c1e75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.480+0000 7fdd31731640 1 --2- 172.21.15.167:0/1869114336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd2c073480 0x7fdd2c1e75e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400e6d0 tx=0x7fdd1400eb90 comp rx=0 tx=0).stop 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.480+0000 7fdd31731640 1 -- 172.21.15.167:0/1869114336 shutdown_connections 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.480+0000 7fdd31731640 1 --2- 172.21.15.167:0/1869114336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd2c15b720 0x7fdd2c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.480+0000 7fdd31731640 1 --2- 172.21.15.167:0/1869114336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd2c0b46a0 0x7fdd2c1e7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.480+0000 7fdd31731640 1 --2- 172.21.15.167:0/1869114336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd2c073480 0x7fdd2c1e75e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.480+0000 7fdd31731640 1 -- 172.21.15.167:0/1869114336 >> 172.21.15.167:0/1869114336 conn(0x7fdd2c0d0b70 msgr2=0x7fdd2c0b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.479 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.480+0000 7fdd31731640 1 -- 172.21.15.167:0/1869114336 shutdown_connections 2024-06-29T14:22:20.480 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.480+0000 7fdd31731640 1 -- 172.21.15.167:0/1869114336 wait complete. 2024-06-29T14:22:20.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 4e572bd2-c163-436a-b90b-9c2e48641954 2 2024-06-29T14:22:20.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.606+0000 7f9d8d427640 1 -- 172.21.15.167:0/1406034316 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d880ae040 msgr2=0x7f9d880744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.606+0000 7f9d8d427640 1 --2- 172.21.15.167:0/1406034316 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d880ae040 0x7f9d880744f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009860 tx=0x7f9d7002a510 comp rx=0 tx=0).stop 2024-06-29T14:22:20.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.606+0000 7f9d8d427640 1 -- 172.21.15.167:0/1406034316 shutdown_connections 2024-06-29T14:22:20.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.606+0000 7f9d8d427640 1 --2- 172.21.15.167:0/1406034316 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d880ae040 0x7f9d880744f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.606+0000 7f9d8d427640 1 -- 172.21.15.167:0/1406034316 >> 172.21.15.167:0/1406034316 conn(0x7f9d880b1ec0 msgr2=0x7f9d880b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.606+0000 7f9d8d427640 1 -- 172.21.15.167:0/1406034316 shutdown_connections 2024-06-29T14:22:20.605 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.606+0000 7f9d8d427640 1 -- 172.21.15.167:0/1406034316 wait complete. 2024-06-29T14:22:20.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.608+0000 7f9d8d427640 1 Processor -- start 2024-06-29T14:22:20.606 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.608+0000 7f9d8d427640 1 -- start start 2024-06-29T14:22:20.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.609+0000 7f9d8d427640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d880acfc0 0x7f9d88150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.609+0000 7f9d8d427640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d880ae040 0x7f9d880b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.609+0000 7f9d8d427640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d8815afa0 0x7f9d8814f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.609+0000 7f9d8d427640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9d88073a50 con 0x7f9d880acfc0 2024-06-29T14:22:20.609 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.609+0000 7f9d8d427640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9d880738d0 con 0x7f9d8815afa0 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.609+0000 7f9d8d427640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9d88073bd0 con 0x7f9d880ae040 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d867fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d880ae040 0x7f9d880b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d877fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d8815afa0 0x7f9d8814f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d867fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d880ae040 0x7f9d880b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:48174/0 (socket says 172.21.15.167:48174) 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d86ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d880acfc0 0x7f9d88150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d867fc640 1 -- 172.21.15.167:0/2199824980 learned_addr learned my addr 172.21.15.167:0/2199824980 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d877fe640 1 -- 172.21.15.167:0/2199824980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d880ae040 msgr2=0x7f9d880b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d877fe640 1 --2- 172.21.15.167:0/2199824980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d880ae040 0x7f9d880b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d877fe640 1 -- 172.21.15.167:0/2199824980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d880acfc0 msgr2=0x7f9d88150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d877fe640 1 --2- 172.21.15.167:0/2199824980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d880acfc0 0x7f9d88150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d877fe640 1 -- 172.21.15.167:0/2199824980 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d880c9360 con 0x7f9d8815afa0 2024-06-29T14:22:20.610 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d867fc640 1 --2- 172.21.15.167:0/2199824980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d880ae040 0x7f9d880b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:20.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d877fe640 1 --2- 172.21.15.167:0/2199824980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d8815afa0 0x7f9d8814f370 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000c700 tx=0x7f9d8000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:20.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d7f7fe640 1 -- 172.21.15.167:0/2199824980 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80003b70 con 0x7f9d8815afa0 2024-06-29T14:22:20.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d8d427640 1 -- 172.21.15.167:0/2199824980 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d88151440 con 0x7f9d8815afa0 2024-06-29T14:22:20.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d7f7fe640 1 -- 172.21.15.167:0/2199824980 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d80003d10 con 0x7f9d8815afa0 2024-06-29T14:22:20.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d7f7fe640 1 -- 172.21.15.167:0/2199824980 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80005330 con 0x7f9d8815afa0 2024-06-29T14:22:20.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.610+0000 7f9d8d427640 1 -- 172.21.15.167:0/2199824980 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d880b4e60 con 0x7f9d8815afa0 2024-06-29T14:22:20.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.611+0000 7f9d7f7fe640 1 -- 172.21.15.167:0/2199824980 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9d800054d0 con 0x7f9d8815afa0 2024-06-29T14:22:20.611 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.611+0000 7f9d7f7fe640 1 -- 172.21.15.167:0/2199824980 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f9d800128e0 con 0x7f9d8815afa0 2024-06-29T14:22:20.612 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.612+0000 7f9d8d427640 1 -- 172.21.15.167:0/2199824980 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d880c9530 con 0x7f9d8815afa0 2024-06-29T14:22:20.612 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.614+0000 7f9d7f7fe640 1 -- 172.21.15.167:0/2199824980 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f9d800124c0 con 0x7f9d8815afa0 2024-06-29T14:22:20.684 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.686+0000 7f9d8d427640 1 -- 172.21.15.167:0/2199824980 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4e572bd2-c163-436a-b90b-9c2e48641954", "id": 2} v 0) -- 0x7f9d880744f0 con 0x7f9d8815afa0 2024-06-29T14:22:20.690 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.690+0000 7f9d7f7fe640 1 -- 172.21.15.167:0/2199824980 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4e572bd2-c163-436a-b90b-9c2e48641954", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f9d80012bc0 con 0x7f9d8815afa0 2024-06-29T14:22:20.691 INFO:teuthology.orchestra.run.smithi167.stdout:2 2024-06-29T14:22:20.692 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.691+0000 7f9d8d427640 1 -- 172.21.15.167:0/2199824980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d8815afa0 msgr2=0x7f9d8814f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.692 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.691+0000 7f9d8d427640 1 --2- 172.21.15.167:0/2199824980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d8815afa0 0x7f9d8814f370 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000c700 tx=0x7f9d8000cbc0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.692 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.691+0000 7f9d8d427640 1 -- 172.21.15.167:0/2199824980 shutdown_connections 2024-06-29T14:22:20.693 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.691+0000 7f9d8d427640 1 --2- 172.21.15.167:0/2199824980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d8815afa0 0x7f9d8814f370 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.693 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.691+0000 7f9d8d427640 1 --2- 172.21.15.167:0/2199824980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d880ae040 0x7f9d880b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.693 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.691+0000 7f9d8d427640 1 --2- 172.21.15.167:0/2199824980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d880acfc0 0x7f9d88150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.693 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.691+0000 7f9d8d427640 1 -- 172.21.15.167:0/2199824980 >> 172.21.15.167:0/2199824980 conn(0x7f9d880b1ec0 msgr2=0x7f9d881517c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.693 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.691+0000 7f9d8d427640 1 -- 172.21.15.167:0/2199824980 shutdown_connections 2024-06-29T14:22:20.693 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.691+0000 7f9d8d427640 1 -- 172.21.15.167:0/2199824980 wait complete. 2024-06-29T14:22:20.704 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new cc3ff76f-fd5a-4d48-bce5-930fc8c6595c 3 2024-06-29T14:22:20.821 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.822+0000 7f7ca724e640 1 -- 172.21.15.167:0/3367211496 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ca00744f0 msgr2=0x7f7ca006e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.821 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.822+0000 7f7ca724e640 1 --2- 172.21.15.167:0/3367211496 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ca00744f0 0x7f7ca006e710 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7ca009dc40 tx=0x7f7c8c02a4d0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.821 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.822+0000 7f7ca724e640 1 -- 172.21.15.167:0/3367211496 shutdown_connections 2024-06-29T14:22:20.821 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.822+0000 7f7ca724e640 1 --2- 172.21.15.167:0/3367211496 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ca006ec50 0x7f7ca00acfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.821 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.822+0000 7f7ca724e640 1 --2- 172.21.15.167:0/3367211496 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ca00744f0 0x7f7ca006e710 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.822+0000 7f7ca724e640 1 --2- 172.21.15.167:0/3367211496 >> v2:172.21.15.81:3300/0 conn(0x7f7ca015afa0 0x7f7ca00ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.822+0000 7f7ca724e640 1 -- 172.21.15.167:0/3367211496 >> 172.21.15.167:0/3367211496 conn(0x7f7ca00b1ec0 msgr2=0x7f7ca00b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.822+0000 7f7ca724e640 1 -- 172.21.15.167:0/3367211496 shutdown_connections 2024-06-29T14:22:20.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.822+0000 7f7ca724e640 1 -- 172.21.15.167:0/3367211496 wait complete. 2024-06-29T14:22:20.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.824+0000 7f7ca724e640 1 Processor -- start 2024-06-29T14:22:20.822 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.824+0000 7f7ca724e640 1 -- start start 2024-06-29T14:22:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.825+0000 7f7ca724e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ca006ec50 0x7f7ca01554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.825+0000 7f7ca724e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ca00744f0 0x7f7ca0159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.825+0000 7f7ca724e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca015afa0 0x7f7ca015a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.825+0000 7f7ca724e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca0073790 con 0x7f7ca015afa0 2024-06-29T14:22:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.825+0000 7f7ca724e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca0073610 con 0x7f7ca006ec50 2024-06-29T14:22:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.825+0000 7f7ca724e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca0073910 con 0x7f7ca00744f0 2024-06-29T14:22:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.825+0000 7f7ca4fea640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ca006ec50 0x7f7ca01554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.825+0000 7f7c9ffff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ca00744f0 0x7f7ca0159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.826 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.825+0000 7f7ca4fea640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ca006ec50 0x7f7ca01554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:35018/0 (socket says 172.21.15.167:35018) 2024-06-29T14:22:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7c9ffff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ca00744f0 0x7f7ca0159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:48192/0 (socket says 172.21.15.167:48192) 2024-06-29T14:22:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7ca4fea640 1 -- 172.21.15.167:0/1670412892 learned_addr learned my addr 172.21.15.167:0/1670412892 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-06-29T14:22:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7ca57eb640 1 --2- 172.21.15.167:0/1670412892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca015afa0 0x7f7ca015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7c9ffff640 1 -- 172.21.15.167:0/1670412892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ca006ec50 msgr2=0x7f7ca01554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7c9ffff640 1 --2- 172.21.15.167:0/1670412892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ca006ec50 0x7f7ca01554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.827 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7c9ffff640 1 -- 172.21.15.167:0/1670412892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca015afa0 msgr2=0x7f7ca015a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7c9ffff640 1 --2- 172.21.15.167:0/1670412892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca015afa0 0x7f7ca015a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7c9ffff640 1 -- 172.21.15.167:0/1670412892 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca00cc200 con 0x7f7ca00744f0 2024-06-29T14:22:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7ca4fea640 1 --2- 172.21.15.167:0/1670412892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ca006ec50 0x7f7ca01554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7c9ffff640 1 --2- 172.21.15.167:0/1670412892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ca00744f0 0x7f7ca0159d00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ca00732e0 tx=0x7f7c8c0038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:20.828 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7c9d7fa640 1 -- 172.21.15.167:0/1670412892 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c03e030 con 0x7f7ca00744f0 2024-06-29T14:22:20.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7c9d7fa640 1 -- 172.21.15.167:0/1670412892 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c8c003d90 con 0x7f7ca00744f0 2024-06-29T14:22:20.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7ca724e640 1 -- 172.21.15.167:0/1670412892 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca015b920 con 0x7f7ca00744f0 2024-06-29T14:22:20.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7c9d7fa640 1 -- 172.21.15.167:0/1670412892 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c035040 con 0x7f7ca00744f0 2024-06-29T14:22:20.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.826+0000 7f7ca724e640 1 -- 172.21.15.167:0/1670412892 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca00ca3e0 con 0x7f7ca00744f0 2024-06-29T14:22:20.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.827+0000 7f7c9d7fa640 1 -- 172.21.15.167:0/1670412892 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7c8c041030 con 0x7f7ca00744f0 2024-06-29T14:22:20.829 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.827+0000 7f7c9d7fa640 1 -- 172.21.15.167:0/1670412892 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f7c8c043b70 con 0x7f7ca00744f0 2024-06-29T14:22:20.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.827+0000 7f7ca724e640 1 -- 172.21.15.167:0/1670412892 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca015b640 con 0x7f7ca00744f0 2024-06-29T14:22:20.830 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.829+0000 7f7c9d7fa640 1 -- 172.21.15.167:0/1670412892 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f7c8c004f20 con 0x7f7ca00744f0 2024-06-29T14:22:20.900 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.902+0000 7f7ca724e640 1 -- 172.21.15.167:0/1670412892 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cc3ff76f-fd5a-4d48-bce5-930fc8c6595c", "id": 3} v 0) -- 0x7f7ca00701a0 con 0x7f7ca00744f0 2024-06-29T14:22:20.904 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.905+0000 7f7c9d7fa640 1 -- 172.21.15.167:0/1670412892 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cc3ff76f-fd5a-4d48-bce5-930fc8c6595c", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f7c8c0436e0 con 0x7f7ca00744f0 2024-06-29T14:22:20.904 INFO:teuthology.orchestra.run.smithi167.stdout:3 2024-06-29T14:22:20.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.906+0000 7f7ca724e640 1 -- 172.21.15.167:0/1670412892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ca00744f0 msgr2=0x7f7ca0159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:20.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.906+0000 7f7ca724e640 1 --2- 172.21.15.167:0/1670412892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ca00744f0 0x7f7ca0159d00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ca00732e0 tx=0x7f7c8c0038a0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.906+0000 7f7ca724e640 1 -- 172.21.15.167:0/1670412892 shutdown_connections 2024-06-29T14:22:20.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.906+0000 7f7ca724e640 1 --2- 172.21.15.167:0/1670412892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca015afa0 0x7f7ca015a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.906+0000 7f7ca724e640 1 --2- 172.21.15.167:0/1670412892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ca00744f0 0x7f7ca0159d00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.905 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.906+0000 7f7ca724e640 1 --2- 172.21.15.167:0/1670412892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ca006ec50 0x7f7ca01554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:20.906 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.906+0000 7f7ca724e640 1 -- 172.21.15.167:0/1670412892 >> 172.21.15.167:0/1670412892 conn(0x7f7ca00b1ec0 msgr2=0x7f7ca0068960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:20.906 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.906+0000 7f7ca724e640 1 -- 172.21.15.167:0/1670412892 shutdown_connections 2024-06-29T14:22:20.906 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:20.906+0000 7f7ca724e640 1 -- 172.21.15.167:0/1670412892 wait complete. 2024-06-29T14:22:20.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364 4 2024-06-29T14:22:21.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.032+0000 7f4c455fb640 1 -- 172.21.15.167:0/3159305669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c400d0230 msgr2=0x7f4c400d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.032+0000 7f4c455fb640 1 --2- 172.21.15.167:0/3159305669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c400d0230 0x7f4c400d0640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4c28009830 tx=0x7f4c2802a530 comp rx=0 tx=0).stop 2024-06-29T14:22:21.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.032+0000 7f4c455fb640 1 -- 172.21.15.167:0/3159305669 shutdown_connections 2024-06-29T14:22:21.031 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.032+0000 7f4c455fb640 1 --2- 172.21.15.167:0/3159305669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c400d0230 0x7f4c400d0640 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.032 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.032+0000 7f4c455fb640 1 -- 172.21.15.167:0/3159305669 >> 172.21.15.167:0/3159305669 conn(0x7f4c400b3680 msgr2=0x7f4c400b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.032 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.033+0000 7f4c455fb640 1 -- 172.21.15.167:0/3159305669 shutdown_connections 2024-06-29T14:22:21.032 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.033+0000 7f4c455fb640 1 -- 172.21.15.167:0/3159305669 wait complete. 2024-06-29T14:22:21.033 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.034+0000 7f4c455fb640 1 Processor -- start 2024-06-29T14:22:21.033 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.034+0000 7f4c455fb640 1 -- start start 2024-06-29T14:22:21.036 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.035+0000 7f4c455fb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c400b3200 0x7f4c4009f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.036 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c455fb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c400b4f40 0x7f4c401571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.036 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c455fb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c400d0230 0x7f4c401f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.036 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c455fb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4c400b5e50 con 0x7f4c400b4f40 2024-06-29T14:22:21.036 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c455fb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4c400b5cd0 con 0x7f4c400d0230 2024-06-29T14:22:21.036 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c455fb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4c400b5fd0 con 0x7f4c400b3200 2024-06-29T14:22:21.036 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c3effd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c400b3200 0x7f4c4009f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.036 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c3effd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c400b3200 0x7f4c4009f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.167:48206/0 (socket says 172.21.15.167:48206) 2024-06-29T14:22:21.036 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c3effd640 1 -- 172.21.15.167:0/3539273696 learned_addr learned my addr 172.21.15.167:0/3539273696 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c3f7fe640 1 --2- 172.21.15.167:0/3539273696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c400d0230 0x7f4c401f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c37fff640 1 --2- 172.21.15.167:0/3539273696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c400b4f40 0x7f4c401571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c3effd640 1 -- 172.21.15.167:0/3539273696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c400d0230 msgr2=0x7f4c401f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c3effd640 1 --2- 172.21.15.167:0/3539273696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c400d0230 0x7f4c401f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c3effd640 1 -- 172.21.15.167:0/3539273696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c400b4f40 msgr2=0x7f4c401571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c3effd640 1 --2- 172.21.15.167:0/3539273696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c400b4f40 0x7f4c401571e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c3effd640 1 -- 172.21.15.167:0/3539273696 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c40157720 con 0x7f4c400b3200 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c3effd640 1 --2- 172.21.15.167:0/3539273696 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c400b3200 0x7f4c4009f350 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000c500 tx=0x7f4c3000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c377fe640 1 -- 172.21.15.167:0/3539273696 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c300052d0 con 0x7f4c400b3200 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c377fe640 1 -- 172.21.15.167:0/3539273696 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c30005470 con 0x7f4c400b3200 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c377fe640 1 -- 172.21.15.167:0/3539273696 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30012870 con 0x7f4c400b3200 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c455fb640 1 -- 172.21.15.167:0/3539273696 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c40157ba0 con 0x7f4c400b3200 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.036+0000 7f4c455fb640 1 -- 172.21.15.167:0/3539273696 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c40158160 con 0x7f4c400b3200 2024-06-29T14:22:21.037 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.037+0000 7f4c377fe640 1 -- 172.21.15.167:0/3539273696 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4c30012a10 con 0x7f4c400b3200 2024-06-29T14:22:21.038 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.037+0000 7f4c377fe640 1 -- 172.21.15.167:0/3539273696 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f4c3001ba00 con 0x7f4c400b3200 2024-06-29T14:22:21.038 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.037+0000 7f4c455fb640 1 -- 172.21.15.167:0/3539273696 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c400d0640 con 0x7f4c400b3200 2024-06-29T14:22:21.038 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.040+0000 7f4c377fe640 1 -- 172.21.15.167:0/3539273696 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f4c400d0b80 con 0x7f4c400b3200 2024-06-29T14:22:21.111 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.112+0000 7f4c455fb640 1 -- 172.21.15.167:0/3539273696 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364", "id": 4} v 0) -- 0x7f4c4015c570 con 0x7f4c400b3200 2024-06-29T14:22:21.114 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.116+0000 7f4c377fe640 1 -- 172.21.15.167:0/3539273696 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f4c3001b5c0 con 0x7f4c400b3200 2024-06-29T14:22:21.115 INFO:teuthology.orchestra.run.smithi167.stdout:4 2024-06-29T14:22:21.117 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.117+0000 7f4c455fb640 1 -- 172.21.15.167:0/3539273696 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c400b3200 msgr2=0x7f4c4009f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.117 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.117+0000 7f4c455fb640 1 --2- 172.21.15.167:0/3539273696 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c400b3200 0x7f4c4009f350 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000c500 tx=0x7f4c3000c9c0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.117 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.117+0000 7f4c455fb640 1 -- 172.21.15.167:0/3539273696 shutdown_connections 2024-06-29T14:22:21.117 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.117+0000 7f4c455fb640 1 --2- 172.21.15.167:0/3539273696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c400d0230 0x7f4c401f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.117 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.117+0000 7f4c455fb640 1 --2- 172.21.15.167:0/3539273696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c400b4f40 0x7f4c401571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.117 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.117+0000 7f4c455fb640 1 --2- 172.21.15.167:0/3539273696 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c400b3200 0x7f4c4009f350 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.118 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.117+0000 7f4c455fb640 1 -- 172.21.15.167:0/3539273696 >> 172.21.15.167:0/3539273696 conn(0x7f4c400b3680 msgr2=0x7f4c400cb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.118 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.117+0000 7f4c455fb640 1 -- 172.21.15.167:0/3539273696 shutdown_connections 2024-06-29T14:22:21.118 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.117+0000 7f4c455fb640 1 -- 172.21.15.167:0/3539273696 wait complete. 2024-06-29T14:22:21.127 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 788dfe4a-cb5c-478b-9a7c-0861025a6f45 5 2024-06-29T14:22:21.242 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.243+0000 7f5ee1afd640 1 -- 172.21.15.167:0/4113530408 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc143860 msgr2=0x7f5edc0b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.242 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.243+0000 7f5ee1afd640 1 --2- 172.21.15.167:0/4113530408 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc143860 0x7f5edc0b52b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5edc06ff40 tx=0x7f5ec402a560 comp rx=0 tx=0).stop 2024-06-29T14:22:21.242 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.243+0000 7f5ee1afd640 1 -- 172.21.15.167:0/4113530408 shutdown_connections 2024-06-29T14:22:21.242 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.243+0000 7f5ee1afd640 1 --2- 172.21.15.167:0/4113530408 >> v2:172.21.15.81:3300/0 conn(0x7f5edc0b57f0 0x7f5edc072f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.242 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.243+0000 7f5ee1afd640 1 --2- 172.21.15.167:0/4113530408 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc143860 0x7f5edc0b52b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.242 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.243+0000 7f5ee1afd640 1 -- 172.21.15.167:0/4113530408 >> 172.21.15.167:0/4113530408 conn(0x7f5edc0d0b70 msgr2=0x7f5edc0d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.242 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.243+0000 7f5ee1afd640 1 -- 172.21.15.167:0/4113530408 shutdown_connections 2024-06-29T14:22:21.242 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.243+0000 7f5ee1afd640 1 -- 172.21.15.167:0/4113530408 wait complete. 2024-06-29T14:22:21.243 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.245+0000 7f5ee1afd640 1 Processor -- start 2024-06-29T14:22:21.243 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.245+0000 7f5ee1afd640 1 -- start start 2024-06-29T14:22:21.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5ee1afd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc073450 0x7f5edc13f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.246 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5ee1afd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc0b57f0 0x7f5edc154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5ee1afd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc143860 0x7f5edc0b2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5ee1afd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc15eef0 con 0x7f5edc073450 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5ee1afd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc0b48a0 con 0x7f5edc0b57f0 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5ee1afd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5edc15f070 con 0x7f5edc143860 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5edaffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc0b57f0 0x7f5edc154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5edbfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc143860 0x7f5edc0b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5edaffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc0b57f0 0x7f5edc154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:35048/0 (socket says 172.21.15.167:35048) 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5edaffd640 1 -- 172.21.15.167:0/2734788648 learned_addr learned my addr 172.21.15.167:0/2734788648 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.246+0000 7f5edb7fe640 1 --2- 172.21.15.167:0/2734788648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc073450 0x7f5edc13f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5edaffd640 1 -- 172.21.15.167:0/2734788648 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc143860 msgr2=0x7f5edc0b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5edaffd640 1 --2- 172.21.15.167:0/2734788648 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc143860 0x7f5edc0b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.247 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5edaffd640 1 -- 172.21.15.167:0/2734788648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc073450 msgr2=0x7f5edc13f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5edaffd640 1 --2- 172.21.15.167:0/2734788648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc073450 0x7f5edc13f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5edaffd640 1 -- 172.21.15.167:0/2734788648 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5edc15e1c0 con 0x7f5edc0b57f0 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5edbfff640 1 --2- 172.21.15.167:0/2734788648 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc143860 0x7f5edc0b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5edaffd640 1 --2- 172.21.15.167:0/2734788648 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc0b57f0 0x7f5edc154120 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009fd0 tx=0x7f5ecc00ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5ed3fff640 1 -- 172.21.15.167:0/2734788648 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc00cc60 con 0x7f5edc0b57f0 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5ed3fff640 1 -- 172.21.15.167:0/2734788648 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ecc00ce00 con 0x7f5edc0b57f0 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5ed3fff640 1 -- 172.21.15.167:0/2734788648 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc005280 con 0x7f5edc0b57f0 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5ee1afd640 1 -- 172.21.15.167:0/2734788648 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5edc152380 con 0x7f5edc0b57f0 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5ee1afd640 1 -- 172.21.15.167:0/2734788648 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5edc152940 con 0x7f5edc0b57f0 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5ed3fff640 1 -- 172.21.15.167:0/2734788648 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5ecc0047a0 con 0x7f5edc0b57f0 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.247+0000 7f5ed3fff640 1 -- 172.21.15.167:0/2734788648 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f5ecc0189b0 con 0x7f5edc0b57f0 2024-06-29T14:22:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.248+0000 7f5ee1afd640 1 -- 172.21.15.167:0/2734788648 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5edc15f5a0 con 0x7f5edc0b57f0 2024-06-29T14:22:21.249 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.250+0000 7f5ed3fff640 1 -- 172.21.15.167:0/2734788648 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f5ecc038030 con 0x7f5edc0b57f0 2024-06-29T14:22:21.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.324+0000 7f5ee1afd640 1 -- 172.21.15.167:0/2734788648 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "788dfe4a-cb5c-478b-9a7c-0861025a6f45", "id": 5} v 0) -- 0x7f5edc0d0200 con 0x7f5edc0b57f0 2024-06-29T14:22:21.326 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.328+0000 7f5ed3fff640 1 -- 172.21.15.167:0/2734788648 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "788dfe4a-cb5c-478b-9a7c-0861025a6f45", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f5ecc026020 con 0x7f5edc0b57f0 2024-06-29T14:22:21.326 INFO:teuthology.orchestra.run.smithi167.stdout:5 2024-06-29T14:22:21.327 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.329+0000 7f5ee1afd640 1 -- 172.21.15.167:0/2734788648 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc0b57f0 msgr2=0x7f5edc154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.327 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.329+0000 7f5ee1afd640 1 --2- 172.21.15.167:0/2734788648 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc0b57f0 0x7f5edc154120 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc009fd0 tx=0x7f5ecc00ec20 comp rx=0 tx=0).stop 2024-06-29T14:22:21.327 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.329+0000 7f5ee1afd640 1 -- 172.21.15.167:0/2734788648 shutdown_connections 2024-06-29T14:22:21.327 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.329+0000 7f5ee1afd640 1 --2- 172.21.15.167:0/2734788648 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc143860 0x7f5edc0b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.327 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.329+0000 7f5ee1afd640 1 --2- 172.21.15.167:0/2734788648 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc0b57f0 0x7f5edc154120 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.329+0000 7f5ee1afd640 1 --2- 172.21.15.167:0/2734788648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc073450 0x7f5edc13f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.329+0000 7f5ee1afd640 1 -- 172.21.15.167:0/2734788648 >> 172.21.15.167:0/2734788648 conn(0x7f5edc0d0b70 msgr2=0x7f5edc140ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.329+0000 7f5ee1afd640 1 -- 172.21.15.167:0/2734788648 shutdown_connections 2024-06-29T14:22:21.328 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.329+0000 7f5ee1afd640 1 -- 172.21.15.167:0/2734788648 wait complete. 2024-06-29T14:22:21.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 2c771d70-f7cf-49be-b6c6-066a411a0374 6 2024-06-29T14:22:21.453 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.454+0000 7f3f56ece640 1 -- 172.21.15.167:0/3567706063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f5015bdf0 msgr2=0x7f3f5006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.454 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.454+0000 7f3f56ece640 1 --2- 172.21.15.167:0/3567706063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f5015bdf0 0x7f3f5006ed90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3f5015c380 tx=0x7f3f4802a9f0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.454 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.454+0000 7f3f56ece640 1 -- 172.21.15.167:0/3567706063 shutdown_connections 2024-06-29T14:22:21.454 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.454+0000 7f3f56ece640 1 --2- 172.21.15.167:0/3567706063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f5015bdf0 0x7f3f5006ed90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.454 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.454+0000 7f3f56ece640 1 --2- 172.21.15.167:0/3567706063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f500cd150 0x7f3f50072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.455 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.454+0000 7f3f56ece640 1 -- 172.21.15.167:0/3567706063 >> 172.21.15.167:0/3567706063 conn(0x7f3f500b2920 msgr2=0x7f3f500b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.455 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.454+0000 7f3f56ece640 1 -- 172.21.15.167:0/3567706063 shutdown_connections 2024-06-29T14:22:21.455 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.454+0000 7f3f56ece640 1 -- 172.21.15.167:0/3567706063 wait complete. 2024-06-29T14:22:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.456+0000 7f3f56ece640 1 Processor -- start 2024-06-29T14:22:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.456+0000 7f3f56ece640 1 -- start start 2024-06-29T14:22:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.457+0000 7f3f56ece640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f50073480 0x7f3f5014b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.457+0000 7f3f56ece640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f500cd150 0x7f3f5014a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.457+0000 7f3f56ece640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f5015bdf0 0x7f3f500cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.457+0000 7f3f56ece640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5006f660 con 0x7f3f500cd150 2024-06-29T14:22:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.457+0000 7f3f56ece640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5006f4e0 con 0x7f3f50073480 2024-06-29T14:22:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.457+0000 7f3f56ece640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3f5006f7e0 con 0x7f3f5015bdf0 2024-06-29T14:22:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f54c6a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f50073480 0x7f3f5014b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f54c6a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f50073480 0x7f3f5014b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:35074/0 (socket says 172.21.15.167:35074) 2024-06-29T14:22:21.461 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f5546b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f5015bdf0 0x7f3f500cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.461 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f54c6a640 1 -- 172.21.15.167:0/3595392924 learned_addr learned my addr 172.21.15.167:0/3595392924 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-06-29T14:22:21.461 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f4ffff640 1 --2- 172.21.15.167:0/3595392924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f500cd150 0x7f3f5014a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.461 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f54c6a640 1 -- 172.21.15.167:0/3595392924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f5015bdf0 msgr2=0x7f3f500cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.461 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f54c6a640 1 --2- 172.21.15.167:0/3595392924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f5015bdf0 0x7f3f500cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.461 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f54c6a640 1 -- 172.21.15.167:0/3595392924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f500cd150 msgr2=0x7f3f5014a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f54c6a640 1 --2- 172.21.15.167:0/3595392924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f500cd150 0x7f3f5014a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f54c6a640 1 -- 172.21.15.167:0/3595392924 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5014b950 con 0x7f3f50073480 2024-06-29T14:22:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f54c6a640 1 --2- 172.21.15.167:0/3595392924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f50073480 0x7f3f5014b410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400e9f0 tx=0x7f3f4400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f4d7fa640 1 -- 172.21.15.167:0/3595392924 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4400cd80 con 0x7f3f50073480 2024-06-29T14:22:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f4d7fa640 1 -- 172.21.15.167:0/3595392924 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f44004020 con 0x7f3f50073480 2024-06-29T14:22:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f56ece640 1 -- 172.21.15.167:0/3595392924 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f5014acc0 con 0x7f3f50073480 2024-06-29T14:22:21.463 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f4d7fa640 1 -- 172.21.15.167:0/3595392924 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f44004470 con 0x7f3f50073480 2024-06-29T14:22:21.463 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f56ece640 1 -- 172.21.15.167:0/3595392924 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f5014e450 con 0x7f3f50073480 2024-06-29T14:22:21.463 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.458+0000 7f3f4d7fa640 1 -- 172.21.15.167:0/3595392924 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3f44004b50 con 0x7f3f50073480 2024-06-29T14:22:21.463 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.459+0000 7f3f4d7fa640 1 -- 172.21.15.167:0/3595392924 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f3f44003cd0 con 0x7f3f50073480 2024-06-29T14:22:21.463 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.459+0000 7f3f56ece640 1 -- 172.21.15.167:0/3595392924 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f5006fd70 con 0x7f3f50073480 2024-06-29T14:22:21.464 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.461+0000 7f3f4d7fa640 1 -- 172.21.15.167:0/3595392924 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f3f44009490 con 0x7f3f50073480 2024-06-29T14:22:21.531 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.533+0000 7f3f56ece640 1 -- 172.21.15.167:0/3595392924 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2c771d70-f7cf-49be-b6c6-066a411a0374", "id": 6} v 0) -- 0x7f3f5015d700 con 0x7f3f50073480 2024-06-29T14:22:21.535 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.536+0000 7f3f4d7fa640 1 -- 172.21.15.167:0/3595392924 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2c771d70-f7cf-49be-b6c6-066a411a0374", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f3f440037b0 con 0x7f3f50073480 2024-06-29T14:22:21.536 INFO:teuthology.orchestra.run.smithi167.stdout:6 2024-06-29T14:22:21.538 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.537+0000 7f3f56ece640 1 -- 172.21.15.167:0/3595392924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f50073480 msgr2=0x7f3f5014b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.539 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.537+0000 7f3f56ece640 1 --2- 172.21.15.167:0/3595392924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f50073480 0x7f3f5014b410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400e9f0 tx=0x7f3f4400eeb0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.539 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.537+0000 7f3f56ece640 1 -- 172.21.15.167:0/3595392924 shutdown_connections 2024-06-29T14:22:21.539 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.537+0000 7f3f56ece640 1 --2- 172.21.15.167:0/3595392924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f5015bdf0 0x7f3f500cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.539 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.537+0000 7f3f56ece640 1 --2- 172.21.15.167:0/3595392924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f500cd150 0x7f3f5014a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.540 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.537+0000 7f3f56ece640 1 --2- 172.21.15.167:0/3595392924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f50073480 0x7f3f5014b410 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.540 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.537+0000 7f3f56ece640 1 -- 172.21.15.167:0/3595392924 >> 172.21.15.167:0/3595392924 conn(0x7f3f500b2920 msgr2=0x7f3f500709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.540 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.537+0000 7f3f56ece640 1 -- 172.21.15.167:0/3595392924 shutdown_connections 2024-06-29T14:22:21.540 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.537+0000 7f3f56ece640 1 -- 172.21.15.167:0/3595392924 wait complete. 2024-06-29T14:22:21.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new e293ba03-0801-463c-bbe7-ed71eb33b0f8 7 2024-06-29T14:22:21.662 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.663+0000 7fe95c409640 1 -- 172.21.15.167:0/1750143168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95406f030 msgr2=0x7fe95415cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.662 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.663+0000 7fe95c409640 1 --2- 172.21.15.167:0/1750143168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95406f030 0x7fe95415cdc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe954074a00 tx=0x7fe95002a4d0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.662 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.663+0000 7fe95c409640 1 -- 172.21.15.167:0/1750143168 shutdown_connections 2024-06-29T14:22:21.662 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.663+0000 7fe95c409640 1 --2- 172.21.15.167:0/1750143168 >> v2:172.21.15.81:3300/0 conn(0x7fe9540ad030 0x7fe9540ad440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.663 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.663+0000 7fe95c409640 1 --2- 172.21.15.167:0/1750143168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95406f030 0x7fe95415cdc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.663 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.663+0000 7fe95c409640 1 -- 172.21.15.167:0/1750143168 >> 172.21.15.167:0/1750143168 conn(0x7fe9540b1ec0 msgr2=0x7fe9540b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.663 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.663+0000 7fe95c409640 1 -- 172.21.15.167:0/1750143168 shutdown_connections 2024-06-29T14:22:21.663 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.663+0000 7fe95c409640 1 -- 172.21.15.167:0/1750143168 wait complete. 2024-06-29T14:22:21.663 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.665+0000 7fe95c409640 1 Processor -- start 2024-06-29T14:22:21.663 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.665+0000 7fe95c409640 1 -- start start 2024-06-29T14:22:21.666 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.666+0000 7fe95c409640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95406c2f0 0x7fe95414ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.666 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.666+0000 7fe95c409640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9540ad030 0x7fe95409cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.666+0000 7fe95c409640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95415a690 0x7fe95415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.666+0000 7fe95c409640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe9540ceff0 con 0x7fe9540ad030 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.666+0000 7fe95c409640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe9540cee70 con 0x7fe95415a690 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.666+0000 7fe95c409640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe9540cf170 con 0x7fe95406c2f0 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a9a6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95415a690 0x7fe95415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a9a6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95415a690 0x7fe95415aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.167:35088/0 (socket says 172.21.15.167:35088) 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a1a5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95406c2f0 0x7fe95414ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a9a6640 1 -- 172.21.15.167:0/69616318 learned_addr learned my addr 172.21.15.167:0/69616318 (peer_addr_for_me v2:172.21.15.167:0/0) 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe9599a4640 1 --2- 172.21.15.167:0/69616318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9540ad030 0x7fe95409cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a1a5640 1 -- 172.21.15.167:0/69616318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95415a690 msgr2=0x7fe95415aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a1a5640 1 --2- 172.21.15.167:0/69616318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95415a690 0x7fe95415aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a1a5640 1 -- 172.21.15.167:0/69616318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9540ad030 msgr2=0x7fe95409cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a1a5640 1 --2- 172.21.15.167:0/69616318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9540ad030 0x7fe95409cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a1a5640 1 -- 172.21.15.167:0/69616318 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95415afe0 con 0x7fe95406c2f0 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a9a6640 1 --2- 172.21.15.167:0/69616318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95415a690 0x7fe95415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95a1a5640 1 --2- 172.21.15.167:0/69616318 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95406c2f0 0x7fe95414ec40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe94800c3c0 tx=0x7fe94800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe946ffd640 1 -- 172.21.15.167:0/69616318 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe948017070 con 0x7fe95406c2f0 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe946ffd640 1 -- 172.21.15.167:0/69616318 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9480053e0 con 0x7fe95406c2f0 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95c409640 1 -- 172.21.15.167:0/69616318 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe954074610 con 0x7fe95406c2f0 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe946ffd640 1 -- 172.21.15.167:0/69616318 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe948003850 con 0x7fe95406c2f0 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.667+0000 7fe95c409640 1 -- 172.21.15.167:0/69616318 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe95415d5b0 con 0x7fe95406c2f0 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.668+0000 7fe946ffd640 1 -- 172.21.15.167:0/69616318 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe94801a070 con 0x7fe95406c2f0 2024-06-29T14:22:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.668+0000 7fe946ffd640 1 -- 172.21.15.167:0/69616318 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fe948011d00 con 0x7fe95406c2f0 2024-06-29T14:22:21.669 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.669+0000 7fe95c409640 1 -- 172.21.15.167:0/69616318 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9540cf5d0 con 0x7fe95406c2f0 2024-06-29T14:22:21.669 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.671+0000 7fe946ffd640 1 -- 172.21.15.167:0/69616318 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7fe948021020 con 0x7fe95406c2f0 2024-06-29T14:22:21.742 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.743+0000 7fe95c409640 1 -- 172.21.15.167:0/69616318 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e293ba03-0801-463c-bbe7-ed71eb33b0f8", "id": 7} v 0) -- 0x7fe9541e7820 con 0x7fe95406c2f0 2024-06-29T14:22:21.746 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.747+0000 7fe946ffd640 1 -- 172.21.15.167:0/69616318 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e293ba03-0801-463c-bbe7-ed71eb33b0f8", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fe95415afe0 con 0x7fe95406c2f0 2024-06-29T14:22:21.746 INFO:teuthology.orchestra.run.smithi167.stdout:7 2024-06-29T14:22:21.748 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.748+0000 7fe95c409640 1 -- 172.21.15.167:0/69616318 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95406c2f0 msgr2=0x7fe95414ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.748 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.748+0000 7fe95c409640 1 --2- 172.21.15.167:0/69616318 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95406c2f0 0x7fe95414ec40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe94800c3c0 tx=0x7fe94800c880 comp rx=0 tx=0).stop 2024-06-29T14:22:21.748 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.748+0000 7fe95c409640 1 -- 172.21.15.167:0/69616318 shutdown_connections 2024-06-29T14:22:21.748 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.748+0000 7fe95c409640 1 --2- 172.21.15.167:0/69616318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95415a690 0x7fe95415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.749 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.748+0000 7fe95c409640 1 --2- 172.21.15.167:0/69616318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9540ad030 0x7fe95409cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.749 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.748+0000 7fe95c409640 1 --2- 172.21.15.167:0/69616318 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95406c2f0 0x7fe95414ec40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.749 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.748+0000 7fe95c409640 1 -- 172.21.15.167:0/69616318 >> 172.21.15.167:0/69616318 conn(0x7fe9540b1ec0 msgr2=0x7fe9540cf900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.749 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.748+0000 7fe95c409640 1 -- 172.21.15.167:0/69616318 shutdown_connections 2024-06-29T14:22:21.749 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:22:21.748+0000 7fe95c409640 1 -- 172.21.15.167:0/69616318 wait complete. 2024-06-29T14:22:21.757 INFO:tasks.ceph.osd.0:Restarting daemon 2024-06-29T14:22:21.758 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-06-29T14:22:21.761 INFO:tasks.ceph.osd.0:Started 2024-06-29T14:22:21.761 INFO:tasks.ceph.osd.1:Restarting daemon 2024-06-29T14:22:21.761 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-06-29T14:22:21.764 INFO:tasks.ceph.osd.1:Started 2024-06-29T14:22:21.764 INFO:tasks.ceph.osd.2:Restarting daemon 2024-06-29T14:22:21.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-06-29T14:22:21.766 INFO:tasks.ceph.osd.2:Started 2024-06-29T14:22:21.767 INFO:tasks.ceph.osd.3:Restarting daemon 2024-06-29T14:22:21.767 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-06-29T14:22:21.769 INFO:tasks.ceph.osd.3:Started 2024-06-29T14:22:21.769 INFO:tasks.ceph.osd.4:Restarting daemon 2024-06-29T14:22:21.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-06-29T14:22:21.771 INFO:tasks.ceph.osd.4:Started 2024-06-29T14:22:21.771 INFO:tasks.ceph.osd.5:Restarting daemon 2024-06-29T14:22:21.771 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-06-29T14:22:21.773 INFO:tasks.ceph.osd.5:Started 2024-06-29T14:22:21.773 INFO:tasks.ceph.osd.6:Restarting daemon 2024-06-29T14:22:21.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-06-29T14:22:21.776 INFO:tasks.ceph.osd.6:Started 2024-06-29T14:22:21.777 INFO:tasks.ceph.osd.7:Restarting daemon 2024-06-29T14:22:21.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-06-29T14:22:21.779 INFO:tasks.ceph.osd.7:Started 2024-06-29T14:22:21.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.921+0000 7f7c2b09c640 1 -- 172.21.15.81:0/3789305692 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c240b9040 msgr2=0x7f7c240d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.921+0000 7f7c2b09c640 1 --2- 172.21.15.81:0/3789305692 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c240b9040 0x7f7c240d0480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009830 tx=0x7f7c1802a650 comp rx=0 tx=0).stop 2024-06-29T14:22:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.921+0000 7f7c2b09c640 1 -- 172.21.15.81:0/3789305692 shutdown_connections 2024-06-29T14:22:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.921+0000 7f7c2b09c640 1 --2- 172.21.15.81:0/3789305692 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c240b9040 0x7f7c240d0480 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.921+0000 7f7c2b09c640 1 -- 172.21.15.81:0/3789305692 >> 172.21.15.81:0/3789305692 conn(0x7f7c2409cf30 msgr2=0x7f7c240d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.921+0000 7f7c2b09c640 1 -- 172.21.15.81:0/3789305692 shutdown_connections 2024-06-29T14:22:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.921+0000 7f7c2b09c640 1 -- 172.21.15.81:0/3789305692 wait complete. 2024-06-29T14:22:21.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.923+0000 7f7c2b09c640 1 Processor -- start 2024-06-29T14:22:21.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.923+0000 7f7c2b09c640 1 -- start start 2024-06-29T14:22:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c2b09c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c240b5250 0x7f7c2414c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c2b09c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c240b9040 0x7f7c241f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c2b09c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c240d2a90 0x7f7c2415cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c2b09c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7c240d0d50 con 0x7f7c240b5250 2024-06-29T14:22:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c2b09c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7c240d0bd0 con 0x7f7c240d2a90 2024-06-29T14:22:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c2b09c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7c240b80b0 con 0x7f7c240b9040 2024-06-29T14:22:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c28e38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c240b5250 0x7f7c2414c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c28e38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c240b5250 0x7f7c2414c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42578/0 (socket says 172.21.15.81:42578) 2024-06-29T14:22:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c28e38640 1 -- 172.21.15.81:0/4219920280 learned_addr learned my addr 172.21.15.81:0/4219920280 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c23fff640 1 --2- 172.21.15.81:0/4219920280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c240b9040 0x7f7c241f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c29639640 1 --2- 172.21.15.81:0/4219920280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c240d2a90 0x7f7c2415cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c28e38640 1 -- 172.21.15.81:0/4219920280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c240b9040 msgr2=0x7f7c241f5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c28e38640 1 --2- 172.21.15.81:0/4219920280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c240b9040 0x7f7c241f5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c28e38640 1 -- 172.21.15.81:0/4219920280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c240d2a90 msgr2=0x7f7c2415cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c28e38640 1 --2- 172.21.15.81:0/4219920280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c240d2a90 0x7f7c2415cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c28e38640 1 -- 172.21.15.81:0/4219920280 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2414e880 con 0x7f7c240b5250 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c28e38640 1 --2- 172.21.15.81:0/4219920280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c240b5250 0x7f7c2414c0a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009950 tx=0x7f7c18033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c217fa640 1 -- 172.21.15.81:0/4219920280 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18031d60 con 0x7f7c240b5250 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c217fa640 1 -- 172.21.15.81:0/4219920280 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c18043070 con 0x7f7c240b5250 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c2b09c640 1 -- 172.21.15.81:0/4219920280 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2414eb60 con 0x7f7c240b5250 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c217fa640 1 -- 172.21.15.81:0/4219920280 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18033290 con 0x7f7c240b5250 2024-06-29T14:22:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.925+0000 7f7c2b09c640 1 -- 172.21.15.81:0/4219920280 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2414c670 con 0x7f7c240b5250 2024-06-29T14:22:21.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.926+0000 7f7c217fa640 1 -- 172.21.15.81:0/4219920280 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7c18033430 con 0x7f7c240b5250 2024-06-29T14:22:21.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.926+0000 7f7c217fa640 1 -- 172.21.15.81:0/4219920280 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f7c18033720 con 0x7f7c240b5250 2024-06-29T14:22:21.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.927+0000 7f7c2b09c640 1 -- 172.21.15.81:0/4219920280 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c2414c8d0 con 0x7f7c240b5250 2024-06-29T14:22:21.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.929+0000 7f7c217fa640 1 -- 172.21.15.81:0/4219920280 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f7c180035f0 con 0x7f7c240b5250 2024-06-29T14:22:21.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.990+0000 7f7c2b09c640 1 -- 172.21.15.81:0/4219920280 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c240d0480 con 0x7f7c240b5250 2024-06-29T14:22:21.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.991+0000 7f7c217fa640 1 -- 172.21.15.81:0/4219920280 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f7c180037d0 con 0x7f7c240b5250 2024-06-29T14:22:21.989 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:21.990 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":10,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:21.745630+0000","last_up_change":"0.000000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:21.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.992+0000 7f7c2b09c640 1 -- 172.21.15.81:0/4219920280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c240b5250 msgr2=0x7f7c2414c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:21.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.992+0000 7f7c2b09c640 1 --2- 172.21.15.81:0/4219920280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c240b5250 0x7f7c2414c0a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009950 tx=0x7f7c18033c20 comp rx=0 tx=0).stop 2024-06-29T14:22:21.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.992+0000 7f7c2b09c640 1 -- 172.21.15.81:0/4219920280 shutdown_connections 2024-06-29T14:22:21.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.992+0000 7f7c2b09c640 1 --2- 172.21.15.81:0/4219920280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c240d2a90 0x7f7c2415cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.992+0000 7f7c2b09c640 1 --2- 172.21.15.81:0/4219920280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c240b9040 0x7f7c241f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.992+0000 7f7c2b09c640 1 --2- 172.21.15.81:0/4219920280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c240b5250 0x7f7c2414c0a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:21.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.992+0000 7f7c2b09c640 1 -- 172.21.15.81:0/4219920280 >> 172.21.15.81:0/4219920280 conn(0x7f7c2409cf30 msgr2=0x7f7c240d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:21.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.992+0000 7f7c2b09c640 1 -- 172.21.15.81:0/4219920280 shutdown_connections 2024-06-29T14:22:21.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:21.992+0000 7f7c2b09c640 1 -- 172.21.15.81:0/4219920280 wait complete. 2024-06-29T14:22:22.006 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-06-29T14:22:22.006 INFO:tasks.ceph:Waiting for OSDs to come up 2024-06-29T14:22:22.387 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-06-29T14:22:22.388+0000 7f3221b25640 -1 Falling back to public interface 2024-06-29T14:22:22.393 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-06-29T14:22:22.395+0000 7fac35a24640 -1 Falling back to public interface 2024-06-29T14:22:22.395 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-06-29T14:22:22.396+0000 7f4b94544640 -1 Falling back to public interface 2024-06-29T14:22:22.396 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-06-29T14:22:22.398+0000 7ff53d01d640 -1 Falling back to public interface 2024-06-29T14:22:22.397 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-06-29T14:22:22.398+0000 7fd52609a640 -1 Falling back to public interface 2024-06-29T14:22:22.402 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-06-29T14:22:22.404+0000 7f290de43640 -1 Falling back to public interface 2024-06-29T14:22:22.403 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-06-29T14:22:22.405+0000 7f7d743d8640 -1 Falling back to public interface 2024-06-29T14:22:22.417 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-06-29T14:22:22.417+0000 7f4b678b1640 -1 Falling back to public interface 2024-06-29T14:22:22.813 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:22.918 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.920+0000 7f866752b640 1 -- 172.21.15.81:0/446530000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86600709d0 msgr2=0x7f8660070de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.918 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.920+0000 7f866752b640 1 --2- 172.21.15.81:0/446530000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86600709d0 0x7f8660070de0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f865c009830 tx=0x7f865c02a3f0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.918 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.920+0000 7f866752b640 1 -- 172.21.15.81:0/446530000 shutdown_connections 2024-06-29T14:22:22.919 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.920+0000 7f866752b640 1 --2- 172.21.15.81:0/446530000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86600709d0 0x7f8660070de0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.919 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.920+0000 7f866752b640 1 --2- 172.21.15.81:0/446530000 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f866006ffa0 0x7f866015ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.919 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.920+0000 7f866752b640 1 -- 172.21.15.81:0/446530000 >> 172.21.15.81:0/446530000 conn(0x7f86600d0970 msgr2=0x7f86600d0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:22.919 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.920+0000 7f866752b640 1 -- 172.21.15.81:0/446530000 shutdown_connections 2024-06-29T14:22:22.919 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.920+0000 7f866752b640 1 -- 172.21.15.81:0/446530000 wait complete. 2024-06-29T14:22:22.920 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.922+0000 7f866752b640 1 Processor -- start 2024-06-29T14:22:22.920 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.922+0000 7f866752b640 1 -- start start 2024-06-29T14:22:22.922 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f866752b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f866006ffa0 0x7f8660145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.922 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f866752b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86600709d0 0x7f86600cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.922 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f866752b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86600af830 0x7f86600cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:22.922 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f866752b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f86600b3f90 con 0x7f86600af830 2024-06-29T14:22:22.922 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f866752b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f86600b3e10 con 0x7f866006ffa0 2024-06-29T14:22:22.922 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f866752b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f86600b4110 con 0x7f86600709d0 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f8664ac6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86600709d0 0x7f86600cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f86652c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f866006ffa0 0x7f8660145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f8665ac8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86600af830 0x7f86600cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f8664ac6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86600709d0 0x7f86600cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:39386/0 (socket says 172.21.15.81:39386) 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f86652c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f866006ffa0 0x7f8660145f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46094/0 (socket says 172.21.15.81:46094) 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f8664ac6640 1 -- 172.21.15.81:0/2546203404 learned_addr learned my addr 172.21.15.81:0/2546203404 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f8664ac6640 1 -- 172.21.15.81:0/2546203404 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f866006ffa0 msgr2=0x7f8660145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f8664ac6640 1 --2- 172.21.15.81:0/2546203404 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f866006ffa0 0x7f8660145f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f8664ac6640 1 -- 172.21.15.81:0/2546203404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86600af830 msgr2=0x7f86600cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f8664ac6640 1 --2- 172.21.15.81:0/2546203404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86600af830 0x7f86600cfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.923+0000 7f8664ac6640 1 -- 172.21.15.81:0/2546203404 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86600aefc0 con 0x7f86600709d0 2024-06-29T14:22:22.923 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.924+0000 7f8665ac8640 1 --2- 172.21.15.81:0/2546203404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86600af830 0x7f86600cfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:22.924 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.924+0000 7f86652c7640 1 --2- 172.21.15.81:0/2546203404 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f866006ffa0 0x7f8660145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:22.924 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.924+0000 7f8664ac6640 1 --2- 172.21.15.81:0/2546203404 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86600709d0 0x7f86600cf7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f865c02a9e0 tx=0x7f865c033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:22.924 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.924+0000 7f8655ffb640 1 -- 172.21.15.81:0/2546203404 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c005c90 con 0x7f86600709d0 2024-06-29T14:22:22.924 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.924+0000 7f8655ffb640 1 -- 172.21.15.81:0/2546203404 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f865c031e50 con 0x7f86600709d0 2024-06-29T14:22:22.924 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.924+0000 7f8655ffb640 1 -- 172.21.15.81:0/2546203404 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c005260 con 0x7f86600709d0 2024-06-29T14:22:22.924 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.924+0000 7f866752b640 1 -- 172.21.15.81:0/2546203404 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86601466d0 con 0x7f86600709d0 2024-06-29T14:22:22.924 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.924+0000 7f866752b640 1 -- 172.21.15.81:0/2546203404 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8660148230 con 0x7f86600709d0 2024-06-29T14:22:22.924 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.925+0000 7f8655ffb640 1 -- 172.21.15.81:0/2546203404 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 2) ==== 49122+0+0 (secure 0 0 0) 0x7f865c005400 con 0x7f86600709d0 2024-06-29T14:22:22.925 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.925+0000 7f866752b640 1 -- 172.21.15.81:0/2546203404 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86600b4630 con 0x7f86600709d0 2024-06-29T14:22:22.925 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.925+0000 7f8655ffb640 1 -- 172.21.15.81:0/2546203404 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f865c0719b0 con 0x7f86600709d0 2024-06-29T14:22:22.925 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.928+0000 7f8655ffb640 1 -- 172.21.15.81:0/2546203404 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73745 (secure 0 0 0) 0x7f865c03f030 con 0x7f86600709d0 2024-06-29T14:22:22.988 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.989+0000 7f866752b640 1 -- 172.21.15.81:0/2546203404 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8660154bd0 con 0x7f86600709d0 2024-06-29T14:22:22.988 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.989+0000 7f8655ffb640 1 -- 172.21.15.81:0/2546203404 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f865c03fe40 con 0x7f86600709d0 2024-06-29T14:22:22.988 INFO:teuthology.misc.health.smithi081.stdout: 2024-06-29T14:22:22.988 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:21.745630+0000","last_up_change":"0.000000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:22.989 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.990+0000 7f866752b640 1 -- 172.21.15.81:0/2546203404 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86600709d0 msgr2=0x7f86600cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:22.989 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.990+0000 7f866752b640 1 --2- 172.21.15.81:0/2546203404 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86600709d0 0x7f86600cf7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f865c02a9e0 tx=0x7f865c033c20 comp rx=0 tx=0).stop 2024-06-29T14:22:22.989 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.990+0000 7f866752b640 1 -- 172.21.15.81:0/2546203404 shutdown_connections 2024-06-29T14:22:22.989 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.990+0000 7f866752b640 1 --2- 172.21.15.81:0/2546203404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86600af830 0x7f86600cfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.989 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.990+0000 7f866752b640 1 --2- 172.21.15.81:0/2546203404 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86600709d0 0x7f86600cf7e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.989 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.990+0000 7f866752b640 1 --2- 172.21.15.81:0/2546203404 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f866006ffa0 0x7f8660145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:22.990 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.990+0000 7f866752b640 1 -- 172.21.15.81:0/2546203404 >> 172.21.15.81:0/2546203404 conn(0x7f86600d0970 msgr2=0x7f86600afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:22.990 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.990+0000 7f866752b640 1 -- 172.21.15.81:0/2546203404 shutdown_connections 2024-06-29T14:22:22.990 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:22.990+0000 7f866752b640 1 -- 172.21.15.81:0/2546203404 wait complete. 2024-06-29T14:22:22.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-29T14:22:23.705 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:22:23.706+0000 7f125bc69640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-06-29T14:22:23.706 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:22:23.706+0000 7f125bc69640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-06-29T14:22:25.814 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-06-29T14:22:25.814+0000 7f3221b25640 -1 osd.5 0 log_to_monitors true 2024-06-29T14:22:25.864 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-06-29T14:22:25.866+0000 7fd52609a640 -1 osd.3 0 log_to_monitors true 2024-06-29T14:22:25.876 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-06-29T14:22:25.877+0000 7f4b94544640 -1 osd.7 0 log_to_monitors true 2024-06-29T14:22:25.889 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-06-29T14:22:25.889+0000 7f7d743d8640 -1 osd.6 0 log_to_monitors true 2024-06-29T14:22:25.906 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-06-29T14:22:25.907+0000 7ff53d01d640 -1 osd.1 0 log_to_monitors true 2024-06-29T14:22:25.917 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-06-29T14:22:25.919+0000 7f290de43640 -1 osd.2 0 log_to_monitors true 2024-06-29T14:22:25.921 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-06-29T14:22:25.924+0000 7fac35a24640 -1 osd.0 0 log_to_monitors true 2024-06-29T14:22:25.925 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-06-29T14:22:25.926+0000 7f4b678b1640 -1 osd.4 0 log_to_monitors true 2024-06-29T14:22:29.287 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-06-29T14:22:29.287+0000 7f4b90cee640 -1 osd.7 0 waiting for initial osdmap 2024-06-29T14:22:29.307 INFO:tasks.ceph.osd.7.smithi167.stderr:2024-06-29T14:22:29.308+0000 7f4b882d5640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:22:29.369 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-06-29T14:22:29.370+0000 7f321dabb640 -1 osd.5 0 waiting for initial osdmap 2024-06-29T14:22:29.385 INFO:tasks.ceph.osd.5.smithi167.stderr:2024-06-29T14:22:29.386+0000 7f32158b6640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:22:29.424 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-06-29T14:22:29.425+0000 7f4b63847640 -1 osd.4 0 waiting for initial osdmap 2024-06-29T14:22:29.435 INFO:tasks.ceph.osd.4.smithi167.stderr:2024-06-29T14:22:29.436+0000 7f4b5b642640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:22:29.461 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-06-29T14:22:29.463+0000 7fd522030640 -1 osd.3 0 waiting for initial osdmap 2024-06-29T14:22:29.465 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-06-29T14:22:29.467+0000 7f2909dd9640 -1 osd.2 0 waiting for initial osdmap 2024-06-29T14:22:29.468 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-06-29T14:22:29.470+0000 7ff5389e7640 -1 osd.1 0 waiting for initial osdmap 2024-06-29T14:22:29.477 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-06-29T14:22:29.479+0000 7fd519e2b640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:22:29.481 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-06-29T14:22:29.483+0000 7f2901bd4640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:22:29.486 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-06-29T14:22:29.488+0000 7ff5307f3640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:22:29.509 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-06-29T14:22:29.510+0000 7f7d7036e640 -1 osd.6 0 waiting for initial osdmap 2024-06-29T14:22:29.516 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-06-29T14:22:29.518+0000 7fac31415640 -1 osd.0 0 waiting for initial osdmap 2024-06-29T14:22:29.521 INFO:tasks.ceph.osd.6.smithi167.stderr:2024-06-29T14:22:29.522+0000 7f7d68169640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:22:29.539 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-06-29T14:22:29.541+0000 7fac29210640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-29T14:22:29.804 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:29.910 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.912+0000 7f0176362640 1 -- 172.21.15.81:0/4213147885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01700b40c0 msgr2=0x7f01700ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:29.910 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.912+0000 7f0176362640 1 --2- 172.21.15.81:0/4213147885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01700b40c0 0x7f01700ce390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0170073920 tx=0x7f016002a540 comp rx=0 tx=0).stop 2024-06-29T14:22:29.911 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.912+0000 7f0176362640 1 -- 172.21.15.81:0/4213147885 shutdown_connections 2024-06-29T14:22:29.911 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.912+0000 7f0176362640 1 --2- 172.21.15.81:0/4213147885 >> v2:172.21.15.167:3300/0 conn(0x7f01700ce8d0 0x7f0170160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.911 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.912+0000 7f0176362640 1 --2- 172.21.15.81:0/4213147885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01700b40c0 0x7f01700ce390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.911 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.912+0000 7f0176362640 1 -- 172.21.15.81:0/4213147885 >> 172.21.15.81:0/4213147885 conn(0x7f01700b35a0 msgr2=0x7f01700b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:29.911 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.912+0000 7f0176362640 1 -- 172.21.15.81:0/4213147885 shutdown_connections 2024-06-29T14:22:29.911 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.912+0000 7f0176362640 1 -- 172.21.15.81:0/4213147885 wait complete. 2024-06-29T14:22:29.911 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.913+0000 7f0176362640 1 Processor -- start 2024-06-29T14:22:29.912 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.914+0000 7f0176362640 1 -- start start 2024-06-29T14:22:29.914 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f0176362640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01700b40c0 0x7f01701f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:29.914 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f0176362640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01700ce8d0 0x7f017009ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f0176362640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0170160650 0x7f017015e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f0176362640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0170160df0 con 0x7f01700ce8d0 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f0176362640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0170160c70 con 0x7f0170160650 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f0176362640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0170160f70 con 0x7f01700b40c0 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f016f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01700ce8d0 0x7f017009ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f016f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01700ce8d0 0x7f017009ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42640/0 (socket says 172.21.15.81:42640) 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f016f7fe640 1 -- 172.21.15.81:0/168204901 learned_addr learned my addr 172.21.15.81:0/168204901 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f01748ff640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0170160650 0x7f017015e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f016ffff640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01700b40c0 0x7f01701f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f016f7fe640 1 -- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01700b40c0 msgr2=0x7f01701f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f016f7fe640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01700b40c0 0x7f01701f2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f016f7fe640 1 -- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0170160650 msgr2=0x7f017015e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f016f7fe640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0170160650 0x7f017015e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.915+0000 7f016f7fe640 1 -- 172.21.15.81:0/168204901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01701f5520 con 0x7f01700ce8d0 2024-06-29T14:22:29.916 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.916+0000 7f016f7fe640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01700ce8d0 0x7f017009ba70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f015800c970 tx=0x7f015800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:29.916 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.916+0000 7f016cff9640 1 -- 172.21.15.81:0/168204901 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158004220 con 0x7f01700ce8d0 2024-06-29T14:22:29.916 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.916+0000 7f016cff9640 1 -- 172.21.15.81:0/168204901 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01580043e0 con 0x7f01700ce8d0 2024-06-29T14:22:29.916 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.916+0000 7f0176362640 1 -- 172.21.15.81:0/168204901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01701f5800 con 0x7f01700ce8d0 2024-06-29T14:22:29.916 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.916+0000 7f016cff9640 1 -- 172.21.15.81:0/168204901 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0158007c10 con 0x7f01700ce8d0 2024-06-29T14:22:29.916 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.916+0000 7f0176362640 1 -- 172.21.15.81:0/168204901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01701f5dc0 con 0x7f01700ce8d0 2024-06-29T14:22:29.917 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.917+0000 7f0176362640 1 -- 172.21.15.81:0/168204901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01700ce390 con 0x7f01700ce8d0 2024-06-29T14:22:29.917 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.917+0000 7f016cff9640 1 -- 172.21.15.81:0/168204901 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 4) ==== 98145+0+0 (secure 0 0 0) 0x7f0158003610 con 0x7f01700ce8d0 2024-06-29T14:22:29.917 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.918+0000 7f016cff9640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0150076270 0x7f0150078780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:29.917 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.918+0000 7f016cff9640 1 -- 172.21.15.81:0/168204901 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f0158096940 con 0x7f01700ce8d0 2024-06-29T14:22:29.917 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.918+0000 7f016ffff640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0150076270 0x7f0150078780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:29.917 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.918+0000 7f016ffff640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0150076270 0x7f0150078780 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0160033000 tx=0x7f0160004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:29.920 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:29.922+0000 7f016cff9640 1 -- 172.21.15.81:0/168204901 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0158014030 con 0x7f01700ce8d0 2024-06-29T14:22:30.103 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.105+0000 7f0176362640 1 -- 172.21.15.81:0/168204901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01701f5f90 con 0x7f01700ce8d0 2024-06-29T14:22:30.104 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.106+0000 7f016cff9640 1 -- 172.21.15.81:0/168204901 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f01701f5f90 con 0x7f01700ce8d0 2024-06-29T14:22:30.104 INFO:teuthology.misc.health.smithi081.stdout: 2024-06-29T14:22:30.104 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":12,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:27.300502+0000","last_up_change":"0.000000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:30.106 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 -- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0150076270 msgr2=0x7f0150078780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:30.106 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0150076270 0x7f0150078780 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0160033000 tx=0x7f0160004730 comp rx=0 tx=0).stop 2024-06-29T14:22:30.106 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 -- 172.21.15.81:0/168204901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01700ce8d0 msgr2=0x7f017009ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:30.107 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01700ce8d0 0x7f017009ba70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f015800c970 tx=0x7f015800ce30 comp rx=0 tx=0).stop 2024-06-29T14:22:30.107 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 -- 172.21.15.81:0/168204901 shutdown_connections 2024-06-29T14:22:30.107 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0150076270 0x7f0150078780 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:30.107 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0170160650 0x7f017015e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:30.107 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01700ce8d0 0x7f017009ba70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:30.107 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 --2- 172.21.15.81:0/168204901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01700b40c0 0x7f01701f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:30.107 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 -- 172.21.15.81:0/168204901 >> 172.21.15.81:0/168204901 conn(0x7f01700b35a0 msgr2=0x7f01700716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:30.107 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 -- 172.21.15.81:0/168204901 shutdown_connections 2024-06-29T14:22:30.107 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:30.108+0000 7f0176362640 1 -- 172.21.15.81:0/168204901 wait complete. 2024-06-29T14:22:30.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-29T14:22:36.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:22:36.707 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:22:36.924 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:37.041 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.042+0000 7fdd09b94640 1 -- 172.21.15.81:0/1354601902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd040cfeb0 msgr2=0x7fdd040d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.041 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.042+0000 7fdd09b94640 1 --2- 172.21.15.81:0/1354601902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd040cfeb0 0x7fdd040d02c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4005fd0 tx=0x7fdcf402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.041 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.042+0000 7fdd09b94640 1 -- 172.21.15.81:0/1354601902 shutdown_connections 2024-06-29T14:22:37.041 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.042+0000 7fdd09b94640 1 --2- 172.21.15.81:0/1354601902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd040ceb90 0x7fdd040ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.042 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.042+0000 7fdd09b94640 1 --2- 172.21.15.81:0/1354601902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd040cfeb0 0x7fdd040d02c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.042 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.042+0000 7fdd09b94640 1 -- 172.21.15.81:0/1354601902 >> 172.21.15.81:0/1354601902 conn(0x7fdd040d1a60 msgr2=0x7fdd040d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.042 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.042+0000 7fdd09b94640 1 -- 172.21.15.81:0/1354601902 shutdown_connections 2024-06-29T14:22:37.042 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.042+0000 7fdd09b94640 1 -- 172.21.15.81:0/1354601902 wait complete. 2024-06-29T14:22:37.042 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.043+0000 7fdd09b94640 1 Processor -- start 2024-06-29T14:22:37.042 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.044+0000 7fdd09b94640 1 -- start start 2024-06-29T14:22:37.045 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd09b94640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd040cc3d0 0x7fdd041f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.045 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd09b94640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd040cfeb0 0x7fdd04156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd09b94640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd04157610 0x7fdd04154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd09b94640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdd040b60b0 con 0x7fdd040cc3d0 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd09b94640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdd040b5f30 con 0x7fdd04157610 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd09b94640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdd040b6230 con 0x7fdd040cfeb0 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd040cc3d0 0x7fdd041f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd040cc3d0 0x7fdd041f69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39944/0 (socket says 172.21.15.81:39944) 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd037fe640 1 -- 172.21.15.81:0/156944321 learned_addr learned my addr 172.21.15.81:0/156944321 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd02ffd640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd040cfeb0 0x7fdd04156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd03fff640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd04157610 0x7fdd04154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd037fe640 1 -- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd040cfeb0 msgr2=0x7fdd04156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd037fe640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd040cfeb0 0x7fdd04156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.046 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd037fe640 1 -- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd04157610 msgr2=0x7fdd04154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.047 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd037fe640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd04157610 0x7fdd04154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.047 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd037fe640 1 -- 172.21.15.81:0/156944321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd04157db0 con 0x7fdd040cc3d0 2024-06-29T14:22:37.047 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.045+0000 7fdd02ffd640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd040cfeb0 0x7fdd04156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:37.047 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.046+0000 7fdd037fe640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd040cc3d0 0x7fdd041f69b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4034040 tx=0x7fdcf40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.047 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.046+0000 7fdcebfff640 1 -- 172.21.15.81:0/156944321 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf403e030 con 0x7fdd040cc3d0 2024-06-29T14:22:37.047 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.046+0000 7fdcebfff640 1 -- 172.21.15.81:0/156944321 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdcf4003ca0 con 0x7fdd040cc3d0 2024-06-29T14:22:37.047 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.046+0000 7fdd09b94640 1 -- 172.21.15.81:0/156944321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd0409b990 con 0x7fdd040cc3d0 2024-06-29T14:22:37.047 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.046+0000 7fdcebfff640 1 -- 172.21.15.81:0/156944321 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4035040 con 0x7fdd040cc3d0 2024-06-29T14:22:37.047 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.046+0000 7fdd09b94640 1 -- 172.21.15.81:0/156944321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd0409bf50 con 0x7fdd040cc3d0 2024-06-29T14:22:37.048 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.047+0000 7fdd09b94640 1 -- 172.21.15.81:0/156944321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd04075710 con 0x7fdd040cc3d0 2024-06-29T14:22:37.048 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.047+0000 7fdcebfff640 1 -- 172.21.15.81:0/156944321 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdcf4041030 con 0x7fdd040cc3d0 2024-06-29T14:22:37.048 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.048+0000 7fdcebfff640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdcf00762d0 0x7fdcf00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.048 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.048+0000 7fdcebfff640 1 -- 172.21.15.81:0/156944321 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fdcf40b7910 con 0x7fdd040cc3d0 2024-06-29T14:22:37.048 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.048+0000 7fdd02ffd640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdcf00762d0 0x7fdcf00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.048 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.049+0000 7fdd02ffd640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdcf00762d0 0x7fdcf00787e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdd041574d0 tx=0x7fdcf8009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.051 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.053+0000 7fdcebfff640 1 -- 172.21.15.81:0/156944321 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdd04075710 con 0x7fdd040cc3d0 2024-06-29T14:22:37.233 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.234+0000 7fdd09b94640 1 -- 172.21.15.81:0/156944321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd040d02c0 con 0x7fdd040cc3d0 2024-06-29T14:22:37.234 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.235+0000 7fdcebfff640 1 -- 172.21.15.81:0/156944321 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12872 (secure 0 0 0) 0x7fdcf40815d0 con 0x7fdd040cc3d0 2024-06-29T14:22:37.234 INFO:teuthology.misc.health.smithi081.stdout: 2024-06-29T14:22:37.234 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":16,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:33.559740+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:37.236 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.236+0000 7fdd09b94640 1 -- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdcf00762d0 msgr2=0x7fdcf00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.236 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.236+0000 7fdd09b94640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdcf00762d0 0x7fdcf00787e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdd041574d0 tx=0x7fdcf8009040 comp rx=0 tx=0).stop 2024-06-29T14:22:37.236 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.236+0000 7fdd09b94640 1 -- 172.21.15.81:0/156944321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd040cc3d0 msgr2=0x7fdd041f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.237 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.236+0000 7fdd09b94640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd040cc3d0 0x7fdd041f69b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4034040 tx=0x7fdcf40038a0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.237 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.237+0000 7fdd09b94640 1 -- 172.21.15.81:0/156944321 shutdown_connections 2024-06-29T14:22:37.237 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.237+0000 7fdd09b94640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdcf00762d0 0x7fdcf00787e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.237 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.237+0000 7fdd09b94640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd04157610 0x7fdd04154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.237 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.237+0000 7fdd09b94640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd040cfeb0 0x7fdd04156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.237 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.237+0000 7fdd09b94640 1 --2- 172.21.15.81:0/156944321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd040cc3d0 0x7fdd041f69b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.237 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.237+0000 7fdd09b94640 1 -- 172.21.15.81:0/156944321 >> 172.21.15.81:0/156944321 conn(0x7fdd040d1a60 msgr2=0x7fdd040b3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.237 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.237+0000 7fdd09b94640 1 -- 172.21.15.81:0/156944321 shutdown_connections 2024-06-29T14:22:37.237 INFO:teuthology.misc.health.smithi081.stderr:2024-06-29T14:22:37.237+0000 7fdd09b94640 1 -- 172.21.15.81:0/156944321 wait complete. 2024-06-29T14:22:37.246 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-06-29T14:22:37.246 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-06-29T14:22:37.246 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:22:37.246 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:22:37.249 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:22:37.249 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:22:37.249 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:22:37.251 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:22:37.251 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:22:37.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:22:37.254 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:22:37.254 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:22:37.254 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:22:37.256 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:22:37.256 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-06-29T14:22:37.256 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:37.335 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:22:37.335 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T14:22:37.352 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T14:22:37.352 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:22:37.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.379+0000 7f5ee47ac640 1 -- 172.21.15.81:0/2469388268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ec8003a30 msgr2=0x7f5ec8003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.379+0000 7f5ee47ac640 1 --2- 172.21.15.81:0/2469388268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ec8003a30 0x7f5ec8003ea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8006fc0 tx=0x7f5ec8005830 comp rx=0 tx=0).stop 2024-06-29T14:22:37.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.379+0000 7f5ee2d49640 1 -- 172.21.15.81:0/2469388268 reap_dead start 2024-06-29T14:22:37.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.379+0000 7f5ee47ac640 1 -- 172.21.15.81:0/2469388268 shutdown_connections 2024-06-29T14:22:37.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.379+0000 7f5ee47ac640 1 -- 172.21.15.81:0/2469388268 >> 172.21.15.81:0/2469388268 conn(0x7f5edc0b3680 msgr2=0x7f5edc0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.379+0000 7f5ee47ac640 1 -- 172.21.15.81:0/2469388268 shutdown_connections 2024-06-29T14:22:37.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.379+0000 7f5ee47ac640 1 -- 172.21.15.81:0/2469388268 wait complete. 2024-06-29T14:22:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.381+0000 7f5ee47ac640 1 Processor -- start 2024-06-29T14:22:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.381+0000 7f5ee47ac640 1 -- start start 2024-06-29T14:22:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.382+0000 7f5ee47ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5ec8003a30 0x7f5edc1f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.382+0000 7f5ee47ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc150f40 0x7f5edc14e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.382+0000 7f5ee47ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc155f40 0x7f5edc1f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.382+0000 7f5ee47ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc164d90 con 0x7f5edc155f40 2024-06-29T14:22:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.382+0000 7f5ee47ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc164c10 con 0x7f5edc150f40 2024-06-29T14:22:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.382+0000 7f5ee47ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5edc164f10 con 0x7f5ec8003a30 2024-06-29T14:22:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.382+0000 7f5ee2d49640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc155f40 0x7f5edc1f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.382+0000 7f5ee2d49640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc155f40 0x7f5edc1f3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40016/0 (socket says 172.21.15.81:40016) 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee2d49640 1 -- 172.21.15.81:0/3435886003 learned_addr learned my addr 172.21.15.81:0/3435886003 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee1d47640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc150f40 0x7f5edc14e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee2548640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5ec8003a30 0x7f5edc1f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee2d49640 1 -- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5ec8003a30 msgr2=0x7f5edc1f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee2d49640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5ec8003a30 0x7f5edc1f41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee2d49640 1 -- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc150f40 msgr2=0x7f5edc14e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee2d49640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc150f40 0x7f5edc14e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee2d49640 1 -- 172.21.15.81:0/3435886003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5edc1f4730 con 0x7f5edc155f40 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee2d49640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc155f40 0x7f5edc1f3030 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4005fd0 tx=0x7f5ed4011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ed2ffd640 1 -- 172.21.15.81:0/3435886003 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed400fd10 con 0x7f5edc155f40 2024-06-29T14:22:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ed2ffd640 1 -- 172.21.15.81:0/3435886003 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ed4003970 con 0x7f5edc155f40 2024-06-29T14:22:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee47ac640 1 -- 172.21.15.81:0/3435886003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5edc154ea0 con 0x7f5edc155f40 2024-06-29T14:22:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ed2ffd640 1 -- 172.21.15.81:0/3435886003 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4003dc0 con 0x7f5edc155f40 2024-06-29T14:22:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.383+0000 7f5ee47ac640 1 -- 172.21.15.81:0/3435886003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5edc155460 con 0x7f5edc155f40 2024-06-29T14:22:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.384+0000 7f5ee47ac640 1 -- 172.21.15.81:0/3435886003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5edc0cc7b0 con 0x7f5edc155f40 2024-06-29T14:22:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.385+0000 7f5ed2ffd640 1 -- 172.21.15.81:0/3435886003 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5ed4003f60 con 0x7f5edc155f40 2024-06-29T14:22:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.385+0000 7f5ed2ffd640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ec00762d0 0x7f5ec00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.385+0000 7f5ed2ffd640 1 -- 172.21.15.81:0/3435886003 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f5ed4097d40 con 0x7f5edc155f40 2024-06-29T14:22:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.385+0000 7f5ee2548640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ec00762d0 0x7f5ec00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.386+0000 7f5ee2548640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ec00762d0 0x7f5ec00787e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8033000 tx=0x7f5ed8000fe0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.389+0000 7f5ed2ffd640 1 -- 172.21.15.81:0/3435886003 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5ed40619f0 con 0x7f5edc155f40 2024-06-29T14:22:37.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.568+0000 7f5ee47ac640 1 -- 172.21.15.81:0/3435886003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5edc071650 con 0x7f5edc155f40 2024-06-29T14:22:37.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.568+0000 7f5ed2ffd640 1 -- 172.21.15.81:0/3435886003 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12872 (secure 0 0 0) 0x7f5ed40612f0 con 0x7f5edc155f40 2024-06-29T14:22:37.568 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:37.568 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:33.559740+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:37.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.570+0000 7f5ee47ac640 1 -- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ec00762d0 msgr2=0x7f5ec00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.570+0000 7f5ee47ac640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ec00762d0 0x7f5ec00787e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8033000 tx=0x7f5ed8000fe0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.570+0000 7f5ee47ac640 1 -- 172.21.15.81:0/3435886003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc155f40 msgr2=0x7f5edc1f3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.570+0000 7f5ee47ac640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc155f40 0x7f5edc1f3030 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4005fd0 tx=0x7f5ed4011f90 comp rx=0 tx=0).stop 2024-06-29T14:22:37.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.570+0000 7f5ee47ac640 1 -- 172.21.15.81:0/3435886003 shutdown_connections 2024-06-29T14:22:37.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.570+0000 7f5ee47ac640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ec00762d0 0x7f5ec00787e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.570+0000 7f5ee47ac640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc155f40 0x7f5edc1f3030 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.570+0000 7f5ee47ac640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc150f40 0x7f5edc14e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.570+0000 7f5ee47ac640 1 --2- 172.21.15.81:0/3435886003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5ec8003a30 0x7f5edc1f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.571+0000 7f5ee47ac640 1 -- 172.21.15.81:0/3435886003 >> 172.21.15.81:0/3435886003 conn(0x7f5edc0b3680 msgr2=0x7f5edc157660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.571+0000 7f5ee47ac640 1 -- 172.21.15.81:0/3435886003 shutdown_connections 2024-06-29T14:22:37.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.571+0000 7f5ee47ac640 1 -- 172.21.15.81:0/3435886003 wait complete. 2024-06-29T14:22:37.582 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:22:37.583 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:37.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.699+0000 7fd07d2ba640 1 -- 172.21.15.81:0/518344994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd060005150 msgr2=0x7fd06000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.699+0000 7fd07d2ba640 1 --2- 172.21.15.81:0/518344994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd060005150 0x7fd06000b9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd07808cf40 tx=0x7fd060005830 comp rx=0 tx=0).stop 2024-06-29T14:22:37.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.699+0000 7fd0777fe640 1 -- 172.21.15.81:0/518344994 reap_dead start 2024-06-29T14:22:37.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.699+0000 7fd07d2ba640 1 -- 172.21.15.81:0/518344994 shutdown_connections 2024-06-29T14:22:37.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.699+0000 7fd07d2ba640 1 -- 172.21.15.81:0/518344994 >> 172.21.15.81:0/518344994 conn(0x7fd07809cf30 msgr2=0x7fd0780d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.699+0000 7fd07d2ba640 1 -- 172.21.15.81:0/518344994 shutdown_connections 2024-06-29T14:22:37.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.699+0000 7fd07d2ba640 1 -- 172.21.15.81:0/518344994 wait complete. 2024-06-29T14:22:37.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.701+0000 7fd07d2ba640 1 Processor -- start 2024-06-29T14:22:37.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.701+0000 7fd07d2ba640 1 -- start start 2024-06-29T14:22:37.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.702+0000 7fd07d2ba640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd060005150 0x7fd0781509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.702+0000 7fd07d2ba640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd07815bda0 0x7fd07806ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.702+0000 7fd07d2ba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd07806d390 0x7fd07809d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.702+0000 7fd07d2ba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0780d0810 con 0x7fd07806d390 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.702+0000 7fd07d2ba640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd0780d0690 con 0x7fd060005150 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.702+0000 7fd07d2ba640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd0780b79a0 con 0x7fd07815bda0 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.702+0000 7fd0777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd07806d390 0x7fd07809d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.702+0000 7fd0777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd07806d390 0x7fd07809d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40028/0 (socket says 172.21.15.81:40028) 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.702+0000 7fd0777fe640 1 -- 172.21.15.81:0/2956405048 learned_addr learned my addr 172.21.15.81:0/2956405048 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd076ffd640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd060005150 0x7fd0781509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd0767fc640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd07815bda0 0x7fd07806ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd0777fe640 1 -- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd07815bda0 msgr2=0x7fd07806ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd0777fe640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd07815bda0 0x7fd07806ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd0777fe640 1 -- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd060005150 msgr2=0x7fd0781509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd0777fe640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd060005150 0x7fd0781509d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd0777fe640 1 -- 172.21.15.81:0/2956405048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07806d7a0 con 0x7fd07806d390 2024-06-29T14:22:37.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd0777fe640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd07806d390 0x7fd07809d9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd07002b650 tx=0x7fd070003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd05f7fe640 1 -- 172.21.15.81:0/2956405048 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd070038040 con 0x7fd07806d390 2024-06-29T14:22:37.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd05f7fe640 1 -- 172.21.15.81:0/2956405048 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd070003910 con 0x7fd07806d390 2024-06-29T14:22:37.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd07d2ba640 1 -- 172.21.15.81:0/2956405048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd07814e010 con 0x7fd07806d390 2024-06-29T14:22:37.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd05f7fe640 1 -- 172.21.15.81:0/2956405048 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd070005c90 con 0x7fd07806d390 2024-06-29T14:22:37.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.703+0000 7fd07d2ba640 1 -- 172.21.15.81:0/2956405048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd07814e5d0 con 0x7fd07806d390 2024-06-29T14:22:37.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.704+0000 7fd07d2ba640 1 -- 172.21.15.81:0/2956405048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd07806a2b0 con 0x7fd07806d390 2024-06-29T14:22:37.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.705+0000 7fd05f7fe640 1 -- 172.21.15.81:0/2956405048 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd070003ce0 con 0x7fd07806d390 2024-06-29T14:22:37.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.705+0000 7fd05f7fe640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd064076300 0x7fd064078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:37.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.705+0000 7fd05f7fe640 1 -- 172.21.15.81:0/2956405048 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fd0700b76e0 con 0x7fd07806d390 2024-06-29T14:22:37.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.705+0000 7fd076ffd640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd064076300 0x7fd064078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:37.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.706+0000 7fd076ffd640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd064076300 0x7fd064078810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd07808e230 tx=0x7fd068009500 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:37.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.710+0000 7fd05f7fe640 1 -- 172.21.15.81:0/2956405048 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd07806a2b0 con 0x7fd07806d390 2024-06-29T14:22:37.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.888+0000 7fd07d2ba640 1 -- 172.21.15.81:0/2956405048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd0780d09c0 con 0x7fd07806d390 2024-06-29T14:22:37.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.889+0000 7fd05f7fe640 1 -- 172.21.15.81:0/2956405048 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fd070081300 con 0x7fd07806d390 2024-06-29T14:22:37.888 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:22:37.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 -- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd064076300 msgr2=0x7fd064078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd064076300 0x7fd064078810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd07808e230 tx=0x7fd068009500 comp rx=0 tx=0).stop 2024-06-29T14:22:37.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 -- 172.21.15.81:0/2956405048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd07806d390 msgr2=0x7fd07809d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:37.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd07806d390 0x7fd07809d9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd07002b650 tx=0x7fd070003820 comp rx=0 tx=0).stop 2024-06-29T14:22:37.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 -- 172.21.15.81:0/2956405048 shutdown_connections 2024-06-29T14:22:37.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd064076300 0x7fd064078810 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd07806d390 0x7fd07809d9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd07815bda0 0x7fd07806ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 --2- 172.21.15.81:0/2956405048 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd060005150 0x7fd0781509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:37.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 -- 172.21.15.81:0/2956405048 >> 172.21.15.81:0/2956405048 conn(0x7fd07809cf30 msgr2=0x7fd0780d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:37.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 -- 172.21.15.81:0/2956405048 shutdown_connections 2024-06-29T14:22:37.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:37.891+0000 7fd07d2ba640 1 -- 172.21.15.81:0/2956405048 wait complete. 2024-06-29T14:22:37.901 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:38.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.015+0000 7f6f9b018640 1 -- 172.21.15.81:0/2792305618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f940cd540 msgr2=0x7f6f940b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.015+0000 7f6f9b018640 1 --2- 172.21.15.81:0/2792305618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f940cd540 0x7f6f940b47b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6f940b2c10 tx=0x7f6f8402a530 comp rx=0 tx=0).stop 2024-06-29T14:22:38.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.015+0000 7f6f9b018640 1 -- 172.21.15.81:0/2792305618 shutdown_connections 2024-06-29T14:22:38.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.015+0000 7f6f9b018640 1 --2- 172.21.15.81:0/2792305618 >> v2:172.21.15.167:3300/0 conn(0x7f6f940b4cf0 0x7f6f940d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.015+0000 7f6f9b018640 1 --2- 172.21.15.81:0/2792305618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f940cd540 0x7f6f940b47b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.015+0000 7f6f9b018640 1 -- 172.21.15.81:0/2792305618 >> 172.21.15.81:0/2792305618 conn(0x7f6f940d1960 msgr2=0x7f6f940d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:38.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.015+0000 7f6f9b018640 1 -- 172.21.15.81:0/2792305618 shutdown_connections 2024-06-29T14:22:38.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.015+0000 7f6f9b018640 1 -- 172.21.15.81:0/2792305618 wait complete. 2024-06-29T14:22:38.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.017+0000 7f6f9b018640 1 Processor -- start 2024-06-29T14:22:38.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.017+0000 7f6f9b018640 1 -- start start 2024-06-29T14:22:38.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f9b018640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f940b4cf0 0x7f6f941f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f9b018640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f940b6d60 0x7f6f9415a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f9b018640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f940cd540 0x7f6f941573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f9b018640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f940d0630 con 0x7f6f940b4cf0 2024-06-29T14:22:38.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f9b018640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6f940b2790 con 0x7f6f940cd540 2024-06-29T14:22:38.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f9b018640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6f940d07b0 con 0x7f6f940b6d60 2024-06-29T14:22:38.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f98db4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f940b4cf0 0x7f6f941f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f98db4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f940b4cf0 0x7f6f941f39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40048/0 (socket says 172.21.15.81:40048) 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f98db4640 1 -- 172.21.15.81:0/3531924975 learned_addr learned my addr 172.21.15.81:0/3531924975 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f995b5640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f940cd540 0x7f6f941573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f8bfff640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f940b6d60 0x7f6f9415a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f98db4640 1 -- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f940b6d60 msgr2=0x7f6f9415a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f98db4640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f940b6d60 0x7f6f9415a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f98db4640 1 -- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f940cd540 msgr2=0x7f6f941573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f98db4640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f940cd540 0x7f6f941573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f98db4640 1 -- 172.21.15.81:0/3531924975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f940ca130 con 0x7f6f940b4cf0 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f98db4640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f940b4cf0 0x7f6f941f39f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000c390 tx=0x7f6f9000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f897fa640 1 -- 172.21.15.81:0/3531924975 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90017070 con 0x7f6f940b4cf0 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f897fa640 1 -- 172.21.15.81:0/3531924975 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f900035f0 con 0x7f6f940b4cf0 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f897fa640 1 -- 172.21.15.81:0/3531924975 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90003a20 con 0x7f6f940b4cf0 2024-06-29T14:22:38.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.019+0000 7f6f9b018640 1 -- 172.21.15.81:0/3531924975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f940c9e50 con 0x7f6f940b4cf0 2024-06-29T14:22:38.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.020+0000 7f6f9b018640 1 -- 172.21.15.81:0/3531924975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f941f31f0 con 0x7f6f940b4cf0 2024-06-29T14:22:38.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.021+0000 7f6f9b018640 1 -- 172.21.15.81:0/3531924975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f940d00f0 con 0x7f6f940b4cf0 2024-06-29T14:22:38.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.021+0000 7f6f897fa640 1 -- 172.21.15.81:0/3531924975 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6f9001a070 con 0x7f6f940b4cf0 2024-06-29T14:22:38.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.022+0000 7f6f897fa640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f800762d0 0x7f6f800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:38.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.022+0000 7f6f8bfff640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f800762d0 0x7f6f800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:38.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.022+0000 7f6f897fa640 1 -- 172.21.15.81:0/3531924975 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f6f90096790 con 0x7f6f940b4cf0 2024-06-29T14:22:38.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.022+0000 7f6f8bfff640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f800762d0 0x7f6f800787e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6f84034000 tx=0x7f6f84004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:38.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.026+0000 7f6f897fa640 1 -- 172.21.15.81:0/3531924975 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6f940ca130 con 0x7f6f940b4cf0 2024-06-29T14:22:38.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.243+0000 7f6f9b018640 1 -- 172.21.15.81:0/3531924975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f940cdec0 con 0x7f6f940b4cf0 2024-06-29T14:22:38.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.243+0000 7f6f897fa640 1 -- 172.21.15.81:0/3531924975 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1434 (secure 0 0 0) 0x7f6f90068430 con 0x7f6f940b4cf0 2024-06-29T14:22:38.243 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:38.243 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":2,"btime":"2024-06-29T14:22:37:342079+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3681460169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3681460169},{"type":"v1","addr":"172.21.15.81:6834","nonce":3681460169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-06-29T14:22:38.243 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 2 2024-06-29T14:22:38.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 -- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f800762d0 msgr2=0x7f6f800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f800762d0 0x7f6f800787e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6f84034000 tx=0x7f6f84004730 comp rx=0 tx=0).stop 2024-06-29T14:22:38.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 -- 172.21.15.81:0/3531924975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f940b4cf0 msgr2=0x7f6f941f39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:38.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f940b4cf0 0x7f6f941f39f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000c390 tx=0x7f6f9000c850 comp rx=0 tx=0).stop 2024-06-29T14:22:38.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 -- 172.21.15.81:0/3531924975 shutdown_connections 2024-06-29T14:22:38.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f800762d0 0x7f6f800787e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f940cd540 0x7f6f941573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f940b6d60 0x7f6f9415a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 --2- 172.21.15.81:0/3531924975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f940b4cf0 0x7f6f941f39f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:38.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 -- 172.21.15.81:0/3531924975 >> 172.21.15.81:0/3531924975 conn(0x7f6f940d1960 msgr2=0x7f6f940b5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:38.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 -- 172.21.15.81:0/3531924975 shutdown_connections 2024-06-29T14:22:38.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:38.245+0000 7f6f9b018640 1 -- 172.21.15.81:0/3531924975 wait complete. 2024-06-29T14:22:40.256 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:40.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.383+0000 7fababd31640 1 -- 172.21.15.81:0/1738113402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab8c004800 msgr2=0x7fab8c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.383+0000 7fababd31640 1 --2- 172.21.15.81:0/1738113402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab8c004800 0x7fab8c004c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab98001690 tx=0x7fab8c005830 comp rx=0 tx=0).stop 2024-06-29T14:22:40.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.384+0000 7fababd31640 1 -- 172.21.15.81:0/1738113402 shutdown_connections 2024-06-29T14:22:40.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.384+0000 7fababd31640 1 --2- 172.21.15.81:0/1738113402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab8c004800 0x7fab8c004c10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.384+0000 7fababd31640 1 -- 172.21.15.81:0/1738113402 >> 172.21.15.81:0/1738113402 conn(0x7faba406ccc0 msgr2=0x7faba40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:40.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.384+0000 7fababd31640 1 -- 172.21.15.81:0/1738113402 shutdown_connections 2024-06-29T14:22:40.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.384+0000 7fababd31640 1 -- 172.21.15.81:0/1738113402 wait complete. 2024-06-29T14:22:40.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.385+0000 7fababd31640 1 Processor -- start 2024-06-29T14:22:40.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.385+0000 7fababd31640 1 -- start start 2024-06-29T14:22:40.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7fababd31640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab8c004800 0x7faba41f04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7fababd31640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faba40ac660 0x7faba40aca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7fababd31640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faba40ae500 0x7faba40ae910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7fababd31640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faba406fbf0 con 0x7fab8c004800 2024-06-29T14:22:40.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7fababd31640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faba406fa70 con 0x7faba40ae500 2024-06-29T14:22:40.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7fababd31640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faba406fd70 con 0x7faba40ac660 2024-06-29T14:22:40.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba9acd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab8c004800 0x7faba41f04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba9acd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab8c004800 0x7faba41f04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47944/0 (socket says 172.21.15.81:47944) 2024-06-29T14:22:40.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba9acd640 1 -- 172.21.15.81:0/2739588307 learned_addr learned my addr 172.21.15.81:0/2739588307 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:40.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba9acd640 1 -- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faba40ac660 msgr2=0x7faba40aca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba9acd640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faba40ac660 0x7faba40aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba9acd640 1 -- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faba40ae500 msgr2=0x7faba40ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba9acd640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faba40ae500 0x7faba40ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba9acd640 1 -- 172.21.15.81:0/2739588307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba40acfb0 con 0x7fab8c004800 2024-06-29T14:22:40.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba9acd640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab8c004800 0x7faba41f04e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fab94012790 tx=0x7fab94012c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:40.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba27fc640 1 -- 172.21.15.81:0/2739588307 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94010d00 con 0x7fab8c004800 2024-06-29T14:22:40.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba27fc640 1 -- 172.21.15.81:0/2739588307 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab94004030 con 0x7fab8c004800 2024-06-29T14:22:40.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7fababd31640 1 -- 172.21.15.81:0/2739588307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba40aee50 con 0x7fab8c004800 2024-06-29T14:22:40.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7faba27fc640 1 -- 172.21.15.81:0/2739588307 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94010ea0 con 0x7fab8c004800 2024-06-29T14:22:40.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.387+0000 7fababd31640 1 -- 172.21.15.81:0/2739588307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba414dc10 con 0x7fab8c004800 2024-06-29T14:22:40.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.389+0000 7fababd31640 1 -- 172.21.15.81:0/2739588307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faba40cf2a0 con 0x7fab8c004800 2024-06-29T14:22:40.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.389+0000 7faba27fc640 1 -- 172.21.15.81:0/2739588307 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fab940035f0 con 0x7fab8c004800 2024-06-29T14:22:40.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.390+0000 7faba27fc640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fab90076300 0x7fab90078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.390+0000 7faba27fc640 1 -- 172.21.15.81:0/2739588307 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fab94099270 con 0x7fab8c004800 2024-06-29T14:22:40.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.390+0000 7faba92cc640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fab90076300 0x7fab90078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.391+0000 7faba92cc640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fab90076300 0x7fab90078810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fab98032000 tx=0x7fab98001120 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:40.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.394+0000 7faba27fc640 1 -- 172.21.15.81:0/2739588307 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faba40aee50 con 0x7fab8c004800 2024-06-29T14:22:40.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.611+0000 7fababd31640 1 -- 172.21.15.81:0/2739588307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faba40b0100 con 0x7fab8c004800 2024-06-29T14:22:40.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.611+0000 7faba27fc640 1 -- 172.21.15.81:0/2739588307 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3906 (secure 0 0 0) 0x7fab94017070 con 0x7fab8c004800 2024-06-29T14:22:40.611 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:40.611 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":3,"btime":"2024-06-29T14:22:38:343082+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4295,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/96042442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":96042442},{"type":"v1","addr":"172.21.15.167:6837","nonce":96042442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3681460169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3681460169},{"type":"v1","addr":"172.21.15.81:6834","nonce":3681460169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3175849002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3175849002},{"type":"v1","addr":"172.21.15.81:6835","nonce":3175849002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4360,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3579540962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3579540962},{"type":"v1","addr":"172.21.15.167:6836","nonce":3579540962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-06-29T14:22:40.611 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 3 2024-06-29T14:22:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.613+0000 7fababd31640 1 -- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fab90076300 msgr2=0x7fab90078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.613+0000 7fababd31640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fab90076300 0x7fab90078810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fab98032000 tx=0x7fab98001120 comp rx=0 tx=0).stop 2024-06-29T14:22:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.613+0000 7fababd31640 1 -- 172.21.15.81:0/2739588307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab8c004800 msgr2=0x7faba41f04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.613+0000 7fababd31640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab8c004800 0x7faba41f04e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fab94012790 tx=0x7fab94012c50 comp rx=0 tx=0).stop 2024-06-29T14:22:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.614+0000 7fababd31640 1 -- 172.21.15.81:0/2739588307 shutdown_connections 2024-06-29T14:22:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.614+0000 7fababd31640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fab90076300 0x7fab90078810 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.614+0000 7fababd31640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faba40ae500 0x7faba40ae910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.614+0000 7fababd31640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faba40ac660 0x7faba40aca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.614+0000 7fababd31640 1 --2- 172.21.15.81:0/2739588307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab8c004800 0x7faba41f04e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.614+0000 7fababd31640 1 -- 172.21.15.81:0/2739588307 >> 172.21.15.81:0/2739588307 conn(0x7faba406ccc0 msgr2=0x7faba4068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:40.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.614+0000 7fababd31640 1 -- 172.21.15.81:0/2739588307 shutdown_connections 2024-06-29T14:22:40.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.614+0000 7fababd31640 1 -- 172.21.15.81:0/2739588307 wait complete. 2024-06-29T14:22:40.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.738+0000 7f59a6dbd640 1 -- 172.21.15.81:0/2469272503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c003a30 msgr2=0x7f598c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.738+0000 7f59a6dbd640 1 --2- 172.21.15.81:0/2469272503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c003a30 0x7f598c003ea0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f59900016c0 tx=0x7f598c005830 comp rx=0 tx=0).stop 2024-06-29T14:22:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.739+0000 7f59a535a640 1 -- 172.21.15.81:0/2469272503 reap_dead start 2024-06-29T14:22:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.739+0000 7f59a6dbd640 1 -- 172.21.15.81:0/2469272503 shutdown_connections 2024-06-29T14:22:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.739+0000 7f59a6dbd640 1 -- 172.21.15.81:0/2469272503 >> 172.21.15.81:0/2469272503 conn(0x7f59a00b3510 msgr2=0x7f59a00b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.739+0000 7f59a6dbd640 1 -- 172.21.15.81:0/2469272503 shutdown_connections 2024-06-29T14:22:40.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.739+0000 7f59a6dbd640 1 -- 172.21.15.81:0/2469272503 wait complete. 2024-06-29T14:22:40.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.740+0000 7f59a6dbd640 1 Processor -- start 2024-06-29T14:22:40.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.740+0000 7f59a6dbd640 1 -- start start 2024-06-29T14:22:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a6dbd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c003a30 0x7f59a01e06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a6dbd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59a006e080 0x7f59a009f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a6dbd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59a00ae7f0 0x7f59a00aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a6dbd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f59a00ce8a0 con 0x7f598c003a30 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a6dbd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f59a00ce720 con 0x7f59a006e080 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a6dbd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f59a00b4f20 con 0x7f59a00ae7f0 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a4b59640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c003a30 0x7f59a01e06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a4b59640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c003a30 0x7f59a01e06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47968/0 (socket says 172.21.15.81:47968) 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a4b59640 1 -- 172.21.15.81:0/3614256251 learned_addr learned my addr 172.21.15.81:0/3614256251 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a535a640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59a00ae7f0 0x7f59a00aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f599ffff640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59a006e080 0x7f59a009f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a4b59640 1 -- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59a00ae7f0 msgr2=0x7f59a00aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a4b59640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59a00ae7f0 0x7f59a00aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a4b59640 1 -- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59a006e080 msgr2=0x7f59a009f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a4b59640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59a006e080 0x7f59a009f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a4b59640 1 -- 172.21.15.81:0/3614256251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a006e490 con 0x7f598c003a30 2024-06-29T14:22:40.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a535a640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59a00ae7f0 0x7f59a00aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:40.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f59a4b59640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c003a30 0x7f59a01e06f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f599002a930 tx=0x7f5990006360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:40.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.742+0000 7f599d7fa640 1 -- 172.21.15.81:0/3614256251 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5990038070 con 0x7f598c003a30 2024-06-29T14:22:40.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.743+0000 7f599d7fa640 1 -- 172.21.15.81:0/3614256251 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5990003af0 con 0x7f598c003a30 2024-06-29T14:22:40.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.743+0000 7f59a6dbd640 1 -- 172.21.15.81:0/3614256251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a006f1a0 con 0x7f598c003a30 2024-06-29T14:22:40.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.743+0000 7f599d7fa640 1 -- 172.21.15.81:0/3614256251 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599003d460 con 0x7f598c003a30 2024-06-29T14:22:40.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.743+0000 7f59a6dbd640 1 -- 172.21.15.81:0/3614256251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a006d880 con 0x7f598c003a30 2024-06-29T14:22:40.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.744+0000 7f59a6dbd640 1 -- 172.21.15.81:0/3614256251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59a00d02d0 con 0x7f598c003a30 2024-06-29T14:22:40.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.744+0000 7f599d7fa640 1 -- 172.21.15.81:0/3614256251 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5990030070 con 0x7f598c003a30 2024-06-29T14:22:40.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.745+0000 7f599d7fa640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f59800762d0 0x7f59800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:40.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.745+0000 7f599d7fa640 1 -- 172.21.15.81:0/3614256251 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f59900b7220 con 0x7f598c003a30 2024-06-29T14:22:40.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.745+0000 7f599ffff640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f59800762d0 0x7f59800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:40.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.745+0000 7f599ffff640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f59800762d0 0x7f59800787e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f59a01f7ec0 tx=0x7f5988005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:40.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.749+0000 7f599d7fa640 1 -- 172.21.15.81:0/3614256251 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5990042060 con 0x7f598c003a30 2024-06-29T14:22:40.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.930+0000 7f59a6dbd640 1 -- 172.21.15.81:0/3614256251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59a00cdf00 con 0x7f598c003a30 2024-06-29T14:22:40.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.930+0000 7f599d7fa640 1 -- 172.21.15.81:0/3614256251 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12872 (secure 0 0 0) 0x7f5990080e40 con 0x7f598c003a30 2024-06-29T14:22:40.930 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:40.930 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:33.559740+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:40.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.932+0000 7f59a6dbd640 1 -- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f59800762d0 msgr2=0x7f59800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.932+0000 7f59a6dbd640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f59800762d0 0x7f59800787e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f59a01f7ec0 tx=0x7f5988005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:40.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.932+0000 7f59a6dbd640 1 -- 172.21.15.81:0/3614256251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c003a30 msgr2=0x7f59a01e06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:40.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.932+0000 7f59a6dbd640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c003a30 0x7f59a01e06f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f599002a930 tx=0x7f5990006360 comp rx=0 tx=0).stop 2024-06-29T14:22:40.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.933+0000 7f59a6dbd640 1 -- 172.21.15.81:0/3614256251 shutdown_connections 2024-06-29T14:22:40.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.933+0000 7f59a6dbd640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f59800762d0 0x7f59800787e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.933+0000 7f59a6dbd640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59a00ae7f0 0x7f59a00aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.933+0000 7f59a6dbd640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59a006e080 0x7f59a009f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.933+0000 7f59a6dbd640 1 --2- 172.21.15.81:0/3614256251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c003a30 0x7f59a01e06f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:40.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.933+0000 7f59a6dbd640 1 -- 172.21.15.81:0/3614256251 >> 172.21.15.81:0/3614256251 conn(0x7f59a00b3510 msgr2=0x7f59a0071940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:40.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.933+0000 7f59a6dbd640 1 -- 172.21.15.81:0/3614256251 shutdown_connections 2024-06-29T14:22:40.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:40.933+0000 7f59a6dbd640 1 -- 172.21.15.81:0/3614256251 wait complete. 2024-06-29T14:22:40.943 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:22:40.943 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:41.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.059+0000 7f15353c7640 1 -- 172.21.15.81:0/3031208030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15300b9040 msgr2=0x7f15300d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.059+0000 7f15353c7640 1 --2- 172.21.15.81:0/3031208030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15300b9040 0x7f15300d0480 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f15300a1b70 tx=0x7f152002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.059+0000 7f15353c7640 1 -- 172.21.15.81:0/3031208030 shutdown_connections 2024-06-29T14:22:41.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.059+0000 7f15353c7640 1 --2- 172.21.15.81:0/3031208030 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15300b7290 0x7f15300b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.059+0000 7f15353c7640 1 --2- 172.21.15.81:0/3031208030 >> v2:172.21.15.167:3300/0 conn(0x7f15300b5250 0x7f15300d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.059+0000 7f15353c7640 1 --2- 172.21.15.81:0/3031208030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15300b9040 0x7f15300d0480 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.059+0000 7f15353c7640 1 -- 172.21.15.81:0/3031208030 >> 172.21.15.81:0/3031208030 conn(0x7f153009cf30 msgr2=0x7f15300d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:41.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.060+0000 7f15353c7640 1 -- 172.21.15.81:0/3031208030 shutdown_connections 2024-06-29T14:22:41.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.060+0000 7f15353c7640 1 -- 172.21.15.81:0/3031208030 wait complete. 2024-06-29T14:22:41.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.061+0000 7f15353c7640 1 Processor -- start 2024-06-29T14:22:41.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.061+0000 7f15353c7640 1 -- start start 2024-06-29T14:22:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f15353c7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15300b5250 0x7f15301e6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f15353c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15300b7290 0x7f15300b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f15353c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15300b9040 0x7f153015e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f15353c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f15300b7f70 con 0x7f15300b9040 2024-06-29T14:22:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f15353c7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f15300b7df0 con 0x7f15300b7290 2024-06-29T14:22:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f15353c7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f15300b80f0 con 0x7f15300b5250 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f152f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15300b9040 0x7f153015e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f152f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15300b9040 0x7f153015e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47986/0 (socket says 172.21.15.81:47986) 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f152f7fe640 1 -- 172.21.15.81:0/3944204627 learned_addr learned my addr 172.21.15.81:0/3944204627 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f152effd640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15300b5250 0x7f15301e6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f15267fc640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15300b7290 0x7f15300b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f152f7fe640 1 -- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15300b5250 msgr2=0x7f15301e6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f152f7fe640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15300b5250 0x7f15301e6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f152f7fe640 1 -- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15300b7290 msgr2=0x7f15300b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f152f7fe640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15300b7290 0x7f15300b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f152f7fe640 1 -- 172.21.15.81:0/3944204627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15300b08c0 con 0x7f15300b9040 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.063+0000 7f152f7fe640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15300b9040 0x7f153015e0a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f152800d9e0 tx=0x7f152800dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.064+0000 7f1527fff640 1 -- 172.21.15.81:0/3944204627 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1528017070 con 0x7f15300b9040 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.064+0000 7f1527fff640 1 -- 172.21.15.81:0/3944204627 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f152800be50 con 0x7f15300b9040 2024-06-29T14:22:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.064+0000 7f15353c7640 1 -- 172.21.15.81:0/3944204627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15300b0ba0 con 0x7f15300b9040 2024-06-29T14:22:41.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.064+0000 7f1527fff640 1 -- 172.21.15.81:0/3944204627 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15280106c0 con 0x7f15300b9040 2024-06-29T14:22:41.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.064+0000 7f15353c7640 1 -- 172.21.15.81:0/3944204627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15300b1160 con 0x7f15300b9040 2024-06-29T14:22:41.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.065+0000 7f15353c7640 1 -- 172.21.15.81:0/3944204627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15300b76a0 con 0x7f15300b9040 2024-06-29T14:22:41.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.065+0000 7f1527fff640 1 -- 172.21.15.81:0/3944204627 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f152801a070 con 0x7f15300b9040 2024-06-29T14:22:41.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.066+0000 7f1527fff640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f15100762d0 0x7f15100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.066+0000 7f1527fff640 1 -- 172.21.15.81:0/3944204627 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f15280976e0 con 0x7f15300b9040 2024-06-29T14:22:41.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.066+0000 7f152effd640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f15100762d0 0x7f15100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.067+0000 7f152effd640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f15100762d0 0x7f15100787e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1520033000 tx=0x7f1520004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.070+0000 7f1527fff640 1 -- 172.21.15.81:0/3944204627 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1528060d40 con 0x7f15300b9040 2024-06-29T14:22:41.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.248+0000 7f15353c7640 1 -- 172.21.15.81:0/3944204627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15300b4730 con 0x7f15300b9040 2024-06-29T14:22:41.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.249+0000 7f1527fff640 1 -- 172.21.15.81:0/3944204627 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f1528065040 con 0x7f15300b9040 2024-06-29T14:22:41.248 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:22:41.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 -- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f15100762d0 msgr2=0x7f15100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f15100762d0 0x7f15100787e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1520033000 tx=0x7f1520004730 comp rx=0 tx=0).stop 2024-06-29T14:22:41.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 -- 172.21.15.81:0/3944204627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15300b9040 msgr2=0x7f153015e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15300b9040 0x7f153015e0a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f152800d9e0 tx=0x7f152800dea0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 -- 172.21.15.81:0/3944204627 shutdown_connections 2024-06-29T14:22:41.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f15100762d0 0x7f15100787e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15300b9040 0x7f153015e0a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15300b7290 0x7f15300b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 --2- 172.21.15.81:0/3944204627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15300b5250 0x7f15301e6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 -- 172.21.15.81:0/3944204627 >> 172.21.15.81:0/3944204627 conn(0x7f153009cf30 msgr2=0x7f153006d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:41.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 -- 172.21.15.81:0/3944204627 shutdown_connections 2024-06-29T14:22:41.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.251+0000 7f15353c7640 1 -- 172.21.15.81:0/3944204627 wait complete. 2024-06-29T14:22:41.262 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:22:41.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.377+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2180075524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e78003a30 msgr2=0x7f9e78003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.377+0000 7f9e90b0a640 1 --2- 172.21.15.81:0/2180075524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e78003a30 0x7f9e78003ea0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c0016c0 tx=0x7f9e78005830 comp rx=0 tx=0).stop 2024-06-29T14:22:41.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.377+0000 7f9e8ad9d640 1 -- 172.21.15.81:0/2180075524 reap_dead start 2024-06-29T14:22:41.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.377+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2180075524 shutdown_connections 2024-06-29T14:22:41.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.377+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2180075524 >> 172.21.15.81:0/2180075524 conn(0x7f9e8c0b56d0 msgr2=0x7f9e8c0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:41.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.377+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2180075524 shutdown_connections 2024-06-29T14:22:41.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.377+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2180075524 wait complete. 2024-06-29T14:22:41.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.379+0000 7f9e90b0a640 1 Processor -- start 2024-06-29T14:22:41.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.379+0000 7f9e90b0a640 1 -- start start 2024-06-29T14:22:41.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.380+0000 7f9e90b0a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9e78003a30 0x7f9e8c068210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e90b0a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e8c1d3250 0x7f9e8c068750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e90b0a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e8c09e360 0x7f9e8c1d5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e90b0a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e8c0b3ea0 con 0x7f9e8c1d3250 2024-06-29T14:22:41.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e90b0a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9e8c0b3d20 con 0x7f9e78003a30 2024-06-29T14:22:41.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e90b0a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9e8c0b4020 con 0x7f9e8c09e360 2024-06-29T14:22:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e81d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e8c1d3250 0x7f9e8c068750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e81d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e8c1d3250 0x7f9e8c068750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47994/0 (socket says 172.21.15.81:47994) 2024-06-29T14:22:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e81d9b640 1 -- 172.21.15.81:0/2550982859 learned_addr learned my addr 172.21.15.81:0/2550982859 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e8a59c640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9e78003a30 0x7f9e8c068210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e8ad9d640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e8c09e360 0x7f9e8c1d5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e81d9b640 1 -- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e8c09e360 msgr2=0x7f9e8c1d5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e81d9b640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e8c09e360 0x7f9e8c1d5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e81d9b640 1 -- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9e78003a30 msgr2=0x7f9e8c068210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e81d9b640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9e78003a30 0x7f9e8c068210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e81d9b640 1 -- 172.21.15.81:0/2550982859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8c09eaa0 con 0x7f9e8c1d3250 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e8ad9d640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e8c09e360 0x7f9e8c1d5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e81d9b640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e8c1d3250 0x7f9e8c068750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9e740066b0 tx=0x7f9e740039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e837fe640 1 -- 172.21.15.81:0/2550982859 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e74011700 con 0x7f9e8c1d3250 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e837fe640 1 -- 172.21.15.81:0/2550982859 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e74016020 con 0x7f9e8c1d3250 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e837fe640 1 -- 172.21.15.81:0/2550982859 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e74011b60 con 0x7f9e8c1d3250 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2550982859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e8c09ec90 con 0x7f9e8c1d3250 2024-06-29T14:22:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.381+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2550982859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e8c1d3b70 con 0x7f9e8c1d3250 2024-06-29T14:22:41.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.382+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2550982859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e8c0b45b0 con 0x7f9e8c1d3250 2024-06-29T14:22:41.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.383+0000 7f9e837fe640 1 -- 172.21.15.81:0/2550982859 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9e74011d00 con 0x7f9e8c1d3250 2024-06-29T14:22:41.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.383+0000 7f9e837fe640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9e6c0762d0 0x7f9e6c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.383+0000 7f9e837fe640 1 -- 172.21.15.81:0/2550982859 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f9e74097ef0 con 0x7f9e8c1d3250 2024-06-29T14:22:41.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.384+0000 7f9e8a59c640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9e6c0762d0 0x7f9e6c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.384+0000 7f9e8a59c640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9e6c0762d0 0x7f9e6c0787e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c032000 tx=0x7f9e7c001190 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.388+0000 7f9e837fe640 1 -- 172.21.15.81:0/2550982859 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9e74020020 con 0x7f9e8c1d3250 2024-06-29T14:22:41.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.569+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2550982859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9e8c0b2280 con 0x7f9e8c1d3250 2024-06-29T14:22:41.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.570+0000 7f9e837fe640 1 -- 172.21.15.81:0/2550982859 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f9e74061b20 con 0x7f9e8c1d3250 2024-06-29T14:22:41.569 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:22:41.570 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:22:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9e6c0762d0 msgr2=0x7f9e6c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9e6c0762d0 0x7f9e6c0787e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c032000 tx=0x7f9e7c001190 comp rx=0 tx=0).stop 2024-06-29T14:22:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2550982859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e8c1d3250 msgr2=0x7f9e8c068750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e8c1d3250 0x7f9e8c068750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9e740066b0 tx=0x7f9e740039c0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2550982859 shutdown_connections 2024-06-29T14:22:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9e6c0762d0 0x7f9e6c0787e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9e8c09e360 0x7f9e8c1d5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e8c1d3250 0x7f9e8c068750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 --2- 172.21.15.81:0/2550982859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9e78003a30 0x7f9e8c068210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2550982859 >> 172.21.15.81:0/2550982859 conn(0x7f9e8c0b56d0 msgr2=0x7f9e8c0b8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2550982859 shutdown_connections 2024-06-29T14:22:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.572+0000 7f9e90b0a640 1 -- 172.21.15.81:0/2550982859 wait complete. 2024-06-29T14:22:41.582 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:22:41.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.702+0000 7f0a5d368640 1 -- 172.21.15.81:0/1522786286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a400051c0 msgr2=0x7f0a4000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.702+0000 7f0a5d368640 1 --2- 172.21.15.81:0/1522786286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a400051c0 0x7f0a4000b9c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009b50 tx=0x7f0a40005830 comp rx=0 tx=0).stop 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.702+0000 7f0a577fe640 1 -- 172.21.15.81:0/1522786286 reap_dead start 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.702+0000 7f0a5d368640 1 -- 172.21.15.81:0/1522786286 shutdown_connections 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.702+0000 7f0a5d368640 1 -- 172.21.15.81:0/1522786286 >> 172.21.15.81:0/1522786286 conn(0x7f0a580d3c50 msgr2=0x7f0a580d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.702+0000 7f0a5d368640 1 -- 172.21.15.81:0/1522786286 shutdown_connections 2024-06-29T14:22:41.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.702+0000 7f0a5d368640 1 -- 172.21.15.81:0/1522786286 wait complete. 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.704+0000 7f0a5d368640 1 Processor -- start 2024-06-29T14:22:41.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.704+0000 7f0a5d368640 1 -- start start 2024-06-29T14:22:41.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.705+0000 7f0a5d368640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a400051c0 0x7f0a5815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a5d368640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a5815d4b0 0x7f0a580b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a5d368640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a5815cef0 0x7f0a580b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a5d368640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0a580b2ea0 con 0x7f0a5815cef0 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a5d368640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0a580d0cd0 con 0x7f0a400051c0 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a5d368640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0a580b3020 con 0x7f0a5815d4b0 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a5815cef0 0x7f0a580b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a5815cef0 0x7f0a580b5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48002/0 (socket says 172.21.15.81:48002) 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a577fe640 1 -- 172.21.15.81:0/3800721490 learned_addr learned my addr 172.21.15.81:0/3800721490 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a567fc640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a5815d4b0 0x7f0a580b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a56ffd640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a400051c0 0x7f0a5815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a577fe640 1 -- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a5815d4b0 msgr2=0x7f0a580b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a577fe640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a5815d4b0 0x7f0a580b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a577fe640 1 -- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a400051c0 msgr2=0x7f0a5815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a577fe640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a400051c0 0x7f0a5815c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:41.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a577fe640 1 -- 172.21.15.81:0/3800721490 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5815cd00 con 0x7f0a5815cef0 2024-06-29T14:22:41.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a577fe640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a5815cef0 0x7f0a580b5bb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009b50 tx=0x7f0a50033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a3f7fe640 1 -- 172.21.15.81:0/3800721490 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50038040 con 0x7f0a5815cef0 2024-06-29T14:22:41.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a3f7fe640 1 -- 172.21.15.81:0/3800721490 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a50004020 con 0x7f0a5815cef0 2024-06-29T14:22:41.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a5d368640 1 -- 172.21.15.81:0/3800721490 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5815ed30 con 0x7f0a5815cef0 2024-06-29T14:22:41.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a3f7fe640 1 -- 172.21.15.81:0/3800721490 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50004450 con 0x7f0a5815cef0 2024-06-29T14:22:41.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.706+0000 7f0a5d368640 1 -- 172.21.15.81:0/3800721490 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5815f2f0 con 0x7f0a5815cef0 2024-06-29T14:22:41.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.707+0000 7f0a5d368640 1 -- 172.21.15.81:0/3800721490 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a580cf510 con 0x7f0a5815cef0 2024-06-29T14:22:41.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.708+0000 7f0a3f7fe640 1 -- 172.21.15.81:0/3800721490 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0a50044020 con 0x7f0a5815cef0 2024-06-29T14:22:41.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.709+0000 7f0a3f7fe640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0a440762d0 0x7f0a440787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:41.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.709+0000 7f0a3f7fe640 1 -- 172.21.15.81:0/3800721490 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f0a500b7500 con 0x7f0a5815cef0 2024-06-29T14:22:41.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.709+0000 7f0a56ffd640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0a440762d0 0x7f0a440787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:41.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.709+0000 7f0a56ffd640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0a440762d0 0x7f0a440787e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0a48001990 tx=0x7f0a48006e90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:41.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.713+0000 7f0a3f7fe640 1 -- 172.21.15.81:0/3800721490 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0a500811b0 con 0x7f0a5815cef0 2024-06-29T14:22:41.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:41.893+0000 7f0a5d368640 1 -- 172.21.15.81:0/3800721490 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0a580c7470 con 0x7f0a5815cef0 2024-06-29T14:22:42.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.673+0000 7f0a3f7fe640 1 -- 172.21.15.81:0/3800721490 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7f0a500891a0 con 0x7f0a5815cef0 2024-06-29T14:22:42.672 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T14:22:42.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 -- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0a440762d0 msgr2=0x7f0a440787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:42.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0a440762d0 0x7f0a440787e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0a48001990 tx=0x7f0a48006e90 comp rx=0 tx=0).stop 2024-06-29T14:22:42.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 -- 172.21.15.81:0/3800721490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a5815cef0 msgr2=0x7f0a580b5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:42.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a5815cef0 0x7f0a580b5bb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009b50 tx=0x7f0a50033670 comp rx=0 tx=0).stop 2024-06-29T14:22:42.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 -- 172.21.15.81:0/3800721490 shutdown_connections 2024-06-29T14:22:42.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0a440762d0 0x7f0a440787e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a5815cef0 0x7f0a580b5bb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a5815d4b0 0x7f0a580b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 --2- 172.21.15.81:0/3800721490 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a400051c0 0x7f0a5815c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 -- 172.21.15.81:0/3800721490 >> 172.21.15.81:0/3800721490 conn(0x7f0a580d3c50 msgr2=0x7f0a580b8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:42.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 -- 172.21.15.81:0/3800721490 shutdown_connections 2024-06-29T14:22:42.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.676+0000 7f0a5d368640 1 -- 172.21.15.81:0/3800721490 wait complete. 2024-06-29T14:22:42.691 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:22:42.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.816+0000 7f1db14d6640 1 -- 172.21.15.81:0/4165485729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dac070020 msgr2=0x7f1dac0cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:42.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.816+0000 7f1db14d6640 1 --2- 172.21.15.81:0/4165485729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dac070020 0x7f1dac0cff30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1dac072ce0 tx=0x7f1d9402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.816+0000 7f1db14d6640 1 -- 172.21.15.81:0/4165485729 shutdown_connections 2024-06-29T14:22:42.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.816+0000 7f1db14d6640 1 --2- 172.21.15.81:0/4165485729 >> v2:172.21.15.167:3300/0 conn(0x7f1dac0d0470 0x7f1dac06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.816+0000 7f1db14d6640 1 --2- 172.21.15.81:0/4165485729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dac070020 0x7f1dac0cff30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.816+0000 7f1db14d6640 1 --2- 172.21.15.81:0/4165485729 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dac070d60 0x7f1dac06fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.816+0000 7f1db14d6640 1 -- 172.21.15.81:0/4165485729 >> 172.21.15.81:0/4165485729 conn(0x7f1dac0b2920 msgr2=0x7f1dac0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:42.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.816+0000 7f1db14d6640 1 -- 172.21.15.81:0/4165485729 shutdown_connections 2024-06-29T14:22:42.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.816+0000 7f1db14d6640 1 -- 172.21.15.81:0/4165485729 wait complete. 2024-06-29T14:22:42.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.818+0000 7f1db14d6640 1 Processor -- start 2024-06-29T14:22:42.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.818+0000 7f1db14d6640 1 -- start start 2024-06-29T14:22:42.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.819+0000 7f1db14d6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dac070020 0x7f1dac1368c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:42.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.819+0000 7f1db14d6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dac070d60 0x7f1dac158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:42.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.819+0000 7f1db14d6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dac0d0470 0x7f1dac142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:42.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.819+0000 7f1db14d6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac06cbc0 con 0x7f1dac070d60 2024-06-29T14:22:42.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.819+0000 7f1db14d6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1dac06ca40 con 0x7f1dac070020 2024-06-29T14:22:42.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.819+0000 7f1db14d6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1dac15fb30 con 0x7f1dac0d0470 2024-06-29T14:22:42.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.819+0000 7f1da27fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dac070d60 0x7f1dac158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:42.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da27fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dac070d60 0x7f1dac158a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48008/0 (socket says 172.21.15.81:48008) 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da27fc640 1 -- 172.21.15.81:0/985064003 learned_addr learned my addr 172.21.15.81:0/985064003 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1daaffd640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dac070020 0x7f1dac1368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1dab7fe640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dac0d0470 0x7f1dac142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da27fc640 1 -- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dac0d0470 msgr2=0x7f1dac142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da27fc640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dac0d0470 0x7f1dac142430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da27fc640 1 -- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dac070020 msgr2=0x7f1dac1368c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da27fc640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dac070020 0x7f1dac1368c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da27fc640 1 -- 172.21.15.81:0/985064003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dac15ac50 con 0x7f1dac070d60 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1daaffd640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dac070020 0x7f1dac1368c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1dab7fe640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dac0d0470 0x7f1dac142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da27fc640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dac070d60 0x7f1dac158a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d94034040 tx=0x7f1d94003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da3fff640 1 -- 172.21.15.81:0/985064003 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9403e030 con 0x7f1dac070d60 2024-06-29T14:22:42.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da3fff640 1 -- 172.21.15.81:0/985064003 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d94003e00 con 0x7f1dac070d60 2024-06-29T14:22:42.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1db14d6640 1 -- 172.21.15.81:0/985064003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dac1595f0 con 0x7f1dac070d60 2024-06-29T14:22:42.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1da3fff640 1 -- 172.21.15.81:0/985064003 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94035040 con 0x7f1dac070d60 2024-06-29T14:22:42.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.820+0000 7f1db14d6640 1 -- 172.21.15.81:0/985064003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dac159bb0 con 0x7f1dac070d60 2024-06-29T14:22:42.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.821+0000 7f1db14d6640 1 -- 172.21.15.81:0/985064003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dac159310 con 0x7f1dac070d60 2024-06-29T14:22:42.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.822+0000 7f1da3fff640 1 -- 172.21.15.81:0/985064003 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1d94041030 con 0x7f1dac070d60 2024-06-29T14:22:42.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.823+0000 7f1da3fff640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d8c076300 0x7f1d8c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:42.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.823+0000 7f1da3fff640 1 -- 172.21.15.81:0/985064003 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f1d940b7da0 con 0x7f1dac070d60 2024-06-29T14:22:42.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.823+0000 7f1daaffd640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d8c076300 0x7f1d8c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:42.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.823+0000 7f1daaffd640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d8c076300 0x7f1d8c078810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009a20 tx=0x7f1d9c005d30 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:42.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:42.827+0000 7f1da3fff640 1 -- 172.21.15.81:0/985064003 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1d94081850 con 0x7f1dac070d60 2024-06-29T14:22:43.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.010+0000 7f1db14d6640 1 -- 172.21.15.81:0/985064003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1dac073b20 con 0x7f1dac070d60 2024-06-29T14:22:43.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.680+0000 7f1da3fff640 1 -- 172.21.15.81:0/985064003 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) ==== 170+0+0 (secure 0 0 0) 0x7f1d94081150 con 0x7f1dac070d60 2024-06-29T14:22:43.680 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T14:22:43.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 -- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d8c076300 msgr2=0x7f1d8c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d8c076300 0x7f1d8c078810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009a20 tx=0x7f1d9c005d30 comp rx=0 tx=0).stop 2024-06-29T14:22:43.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 -- 172.21.15.81:0/985064003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dac070d60 msgr2=0x7f1dac158a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dac070d60 0x7f1dac158a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d94034040 tx=0x7f1d94003910 comp rx=0 tx=0).stop 2024-06-29T14:22:43.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 -- 172.21.15.81:0/985064003 shutdown_connections 2024-06-29T14:22:43.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d8c076300 0x7f1d8c078810 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dac0d0470 0x7f1dac142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dac070d60 0x7f1dac158a50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 --2- 172.21.15.81:0/985064003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dac070020 0x7f1dac1368c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 -- 172.21.15.81:0/985064003 >> 172.21.15.81:0/985064003 conn(0x7f1dac0b2920 msgr2=0x7f1dac15b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:43.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 -- 172.21.15.81:0/985064003 shutdown_connections 2024-06-29T14:22:43.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.683+0000 7f1db14d6640 1 -- 172.21.15.81:0/985064003 wait complete. 2024-06-29T14:22:43.707 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:22:43.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.828+0000 7fc0a74fe640 1 -- 172.21.15.81:0/4116327226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0a006ef90 msgr2=0x7fc0a006f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.828+0000 7fc0a74fe640 1 --2- 172.21.15.81:0/4116327226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0a006ef90 0x7fc0a006f3a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc090005fd0 tx=0x7fc09002a530 comp rx=0 tx=0).stop 2024-06-29T14:22:43.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.828+0000 7fc0a74fe640 1 -- 172.21.15.81:0/4116327226 shutdown_connections 2024-06-29T14:22:43.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.828+0000 7fc0a74fe640 1 --2- 172.21.15.81:0/4116327226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0a006ef90 0x7fc0a006f3a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.828+0000 7fc0a74fe640 1 --2- 172.21.15.81:0/4116327226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0a00b57b0 0x7fc0a015aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.828+0000 7fc0a74fe640 1 -- 172.21.15.81:0/4116327226 >> 172.21.15.81:0/4116327226 conn(0x7fc0a00d0930 msgr2=0x7fc0a00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:43.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.828+0000 7fc0a74fe640 1 -- 172.21.15.81:0/4116327226 shutdown_connections 2024-06-29T14:22:43.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.828+0000 7fc0a74fe640 1 -- 172.21.15.81:0/4116327226 wait complete. 2024-06-29T14:22:43.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.830+0000 7fc0a74fe640 1 Processor -- start 2024-06-29T14:22:43.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.830+0000 7fc0a74fe640 1 -- start start 2024-06-29T14:22:43.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a74fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0a006ef90 0x7fc0a01e7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a74fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0a00b57b0 0x7fc0a0151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a74fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0a00ce1d0 0x7fc0a00ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a74fe640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a00cecf0 con 0x7fc0a00b57b0 2024-06-29T14:22:43.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a74fe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a00b4b80 con 0x7fc0a006ef90 2024-06-29T14:22:43.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a74fe640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc0a00cee70 con 0x7fc0a00ce1d0 2024-06-29T14:22:43.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a4a99640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0a00b57b0 0x7fc0a0151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a4a99640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0a00b57b0 0x7fc0a0151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48020/0 (socket says 172.21.15.81:48020) 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a4a99640 1 -- 172.21.15.81:0/1802804076 learned_addr learned my addr 172.21.15.81:0/1802804076 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a529a640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0a006ef90 0x7fc0a01e7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.831+0000 7fc0a5a9b640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0a00ce1d0 0x7fc0a00ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc0a4a99640 1 -- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0a00ce1d0 msgr2=0x7fc0a00ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc0a4a99640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0a00ce1d0 0x7fc0a00ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc0a4a99640 1 -- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0a006ef90 msgr2=0x7fc0a01e7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc0a4a99640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0a006ef90 0x7fc0a01e7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc0a4a99640 1 -- 172.21.15.81:0/1802804076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a01e71e0 con 0x7fc0a00b57b0 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc0a529a640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0a006ef90 0x7fc0a01e7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc0a5a9b640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0a00ce1d0 0x7fc0a00ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc0a4a99640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0a00b57b0 0x7fc0a0151d40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc090034040 tx=0x7fc0900039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:43.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc095ffb640 1 -- 172.21.15.81:0/1802804076 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09003e030 con 0x7fc0a00b57b0 2024-06-29T14:22:43.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc095ffb640 1 -- 172.21.15.81:0/1802804076 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0900047a0 con 0x7fc0a00b57b0 2024-06-29T14:22:43.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc0a74fe640 1 -- 172.21.15.81:0/1802804076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0a01e95a0 con 0x7fc0a00b57b0 2024-06-29T14:22:43.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc095ffb640 1 -- 172.21.15.81:0/1802804076 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc090035040 con 0x7fc0a00b57b0 2024-06-29T14:22:43.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.832+0000 7fc0a74fe640 1 -- 172.21.15.81:0/1802804076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0a01e6d00 con 0x7fc0a00b57b0 2024-06-29T14:22:43.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.833+0000 7fc0a74fe640 1 -- 172.21.15.81:0/1802804076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0a006e2a0 con 0x7fc0a00b57b0 2024-06-29T14:22:43.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.834+0000 7fc095ffb640 1 -- 172.21.15.81:0/1802804076 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc090041030 con 0x7fc0a00b57b0 2024-06-29T14:22:43.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.834+0000 7fc095ffb640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc0840762d0 0x7fc0840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:43.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.835+0000 7fc095ffb640 1 -- 172.21.15.81:0/1802804076 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7fc0900b8380 con 0x7fc0a00b57b0 2024-06-29T14:22:43.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.835+0000 7fc0a529a640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc0840762d0 0x7fc0840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:43.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.835+0000 7fc0a529a640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc0840762d0 0x7fc0840787e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0058c0 tx=0x7fc09c005c90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:43.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:43.838+0000 7fc095ffb640 1 -- 172.21.15.81:0/1802804076 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0a006e2a0 con 0x7fc0a00b57b0 2024-06-29T14:22:44.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.055+0000 7fc0a74fe640 1 -- 172.21.15.81:0/1802804076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc0a006f3a0 con 0x7fc0a00b57b0 2024-06-29T14:22:44.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.682+0000 7fc095ffb640 1 -- 172.21.15.81:0/1802804076 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fc090081d90 con 0x7fc0a00b57b0 2024-06-29T14:22:44.681 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 2 and data pool 3 2024-06-29T14:22:44.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.685+0000 7fc0a74fe640 1 -- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc0840762d0 msgr2=0x7fc0840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.685+0000 7fc0a74fe640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc0840762d0 0x7fc0840787e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0058c0 tx=0x7fc09c005c90 comp rx=0 tx=0).stop 2024-06-29T14:22:44.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.685+0000 7fc0a74fe640 1 -- 172.21.15.81:0/1802804076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0a00b57b0 msgr2=0x7fc0a0151d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.685+0000 7fc0a74fe640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0a00b57b0 0x7fc0a0151d40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc090034040 tx=0x7fc0900039e0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.686+0000 7fc0a74fe640 1 -- 172.21.15.81:0/1802804076 shutdown_connections 2024-06-29T14:22:44.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.686+0000 7fc0a74fe640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc0840762d0 0x7fc0840787e0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0058c0 tx=0x7fc09c005c90 comp rx=0 tx=0).stop 2024-06-29T14:22:44.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.686+0000 7fc0a74fe640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0a00ce1d0 0x7fc0a00ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.686+0000 7fc0a74fe640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0a00b57b0 0x7fc0a0151d40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.686+0000 7fc0a74fe640 1 --2- 172.21.15.81:0/1802804076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0a006ef90 0x7fc0a01e7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.686+0000 7fc0a74fe640 1 -- 172.21.15.81:0/1802804076 >> 172.21.15.81:0/1802804076 conn(0x7fc0a00d0930 msgr2=0x7fc0a0159a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:44.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.687+0000 7fc0a74fe640 1 -- 172.21.15.81:0/1802804076 shutdown_connections 2024-06-29T14:22:44.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.687+0000 7fc0a74fe640 1 -- 172.21.15.81:0/1802804076 wait complete. 2024-06-29T14:22:44.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:44.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.835+0000 7fb546e66640 1 -- 172.21.15.81:0/4015194655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b3960 msgr2=0x7fb5400b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.835+0000 7fb546e66640 1 --2- 172.21.15.81:0/4015194655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b3960 0x7fb5400b3d70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb538005fd0 tx=0x7fb53802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.835+0000 7fb546e66640 1 -- 172.21.15.81:0/4015194655 shutdown_connections 2024-06-29T14:22:44.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.835+0000 7fb546e66640 1 --2- 172.21.15.81:0/4015194655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b3960 0x7fb5400b3d70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.835+0000 7fb546e66640 1 --2- 172.21.15.81:0/4015194655 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b57b0 0x7fb54015aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.835+0000 7fb546e66640 1 -- 172.21.15.81:0/4015194655 >> 172.21.15.81:0/4015194655 conn(0x7fb5400d0930 msgr2=0x7fb5400d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:44.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.835+0000 7fb546e66640 1 -- 172.21.15.81:0/4015194655 shutdown_connections 2024-06-29T14:22:44.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.835+0000 7fb546e66640 1 -- 172.21.15.81:0/4015194655 wait complete. 2024-06-29T14:22:44.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.837+0000 7fb546e66640 1 Processor -- start 2024-06-29T14:22:44.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.837+0000 7fb546e66640 1 -- start start 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.838+0000 7fb546e66640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54006ef90 0x7fb540156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.838+0000 7fb546e66640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b3960 0x7fb5401533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.838+0000 7fb546e66640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b57b0 0x7fb5401e4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.838+0000 7fb546e66640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400b4640 con 0x7fb54006ef90 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.838+0000 7fb546e66640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400b44c0 con 0x7fb5400b57b0 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.838+0000 7fb546e66640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb5400b47c0 con 0x7fb5400b3960 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.838+0000 7fb544c02640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54006ef90 0x7fb540156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.838+0000 7fb544c02640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54006ef90 0x7fb540156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48042/0 (socket says 172.21.15.81:48042) 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.838+0000 7fb544c02640 1 -- 172.21.15.81:0/3554926689 learned_addr learned my addr 172.21.15.81:0/3554926689 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb544c02640 1 -- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b3960 msgr2=0x7fb5401533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb53ffff640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b3960 0x7fb5401533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb544c02640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b3960 0x7fb5401533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb544c02640 1 -- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b57b0 msgr2=0x7fb5401e4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb545403640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b57b0 0x7fb5401e4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb544c02640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b57b0 0x7fb5401e4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb544c02640 1 -- 172.21.15.81:0/3554926689 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb540153930 con 0x7fb54006ef90 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb545403640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b57b0 0x7fb5401e4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb544c02640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54006ef90 0x7fb540156c70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb53000e790 tx=0x7fb53000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb53d7fa640 1 -- 172.21.15.81:0/3554926689 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53000eea0 con 0x7fb54006ef90 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb53d7fa640 1 -- 172.21.15.81:0/3554926689 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb53000cdb0 con 0x7fb54006ef90 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb546e66640 1 -- 172.21.15.81:0/3554926689 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5401e5870 con 0x7fb54006ef90 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb53d7fa640 1 -- 172.21.15.81:0/3554926689 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb530010690 con 0x7fb54006ef90 2024-06-29T14:22:44.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.839+0000 7fb546e66640 1 -- 172.21.15.81:0/3554926689 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb540156470 con 0x7fb54006ef90 2024-06-29T14:22:44.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.840+0000 7fb546e66640 1 -- 172.21.15.81:0/3554926689 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54006e2a0 con 0x7fb54006ef90 2024-06-29T14:22:44.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.841+0000 7fb53d7fa640 1 -- 172.21.15.81:0/3554926689 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb530010830 con 0x7fb54006ef90 2024-06-29T14:22:44.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.841+0000 7fb53d7fa640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb52c076300 0x7fb52c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:44.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.842+0000 7fb53d7fa640 1 -- 172.21.15.81:0/3554926689 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fb5300980a0 con 0x7fb54006ef90 2024-06-29T14:22:44.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.842+0000 7fb53ffff640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb52c076300 0x7fb52c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:44.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.842+0000 7fb53ffff640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb52c076300 0x7fb52c078810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb534007fd0 tx=0x7fb534009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:44.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:44.846+0000 7fb53d7fa640 1 -- 172.21.15.81:0/3554926689 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb54006e2a0 con 0x7fb54006ef90 2024-06-29T14:22:45.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.028+0000 7fb546e66640 1 -- 172.21.15.81:0/3554926689 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb540155f80 con 0x7fb54006ef90 2024-06-29T14:22:45.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.029+0000 7fb53d7fa640 1 -- 172.21.15.81:0/3554926689 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16606 (secure 0 0 0) 0x7fb530061a00 con 0x7fb54006ef90 2024-06-29T14:22:45.029 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:45.029 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":19,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:44.057673+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:45.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.031+0000 7fb546e66640 1 -- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb52c076300 msgr2=0x7fb52c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.031+0000 7fb546e66640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb52c076300 0x7fb52c078810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb534007fd0 tx=0x7fb534009040 comp rx=0 tx=0).stop 2024-06-29T14:22:45.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.031+0000 7fb546e66640 1 -- 172.21.15.81:0/3554926689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54006ef90 msgr2=0x7fb540156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.031+0000 7fb546e66640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54006ef90 0x7fb540156c70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb53000e790 tx=0x7fb53000ec50 comp rx=0 tx=0).stop 2024-06-29T14:22:45.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.032+0000 7fb546e66640 1 -- 172.21.15.81:0/3554926689 shutdown_connections 2024-06-29T14:22:45.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.032+0000 7fb546e66640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb52c076300 0x7fb52c078810 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.032+0000 7fb546e66640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b57b0 0x7fb5401e4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.032+0000 7fb546e66640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b3960 0x7fb5401533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.032+0000 7fb546e66640 1 --2- 172.21.15.81:0/3554926689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54006ef90 0x7fb540156c70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.032+0000 7fb546e66640 1 -- 172.21.15.81:0/3554926689 >> 172.21.15.81:0/3554926689 conn(0x7fb5400d0930 msgr2=0x7fb5400ce190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:45.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.032+0000 7fb546e66640 1 -- 172.21.15.81:0/3554926689 shutdown_connections 2024-06-29T14:22:45.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.032+0000 7fb546e66640 1 -- 172.21.15.81:0/3554926689 wait complete. 2024-06-29T14:22:45.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.155+0000 7faf1d04d640 1 -- 172.21.15.81:0/4002950603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf180cd480 msgr2=0x7faf180ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.155+0000 7faf1d04d640 1 --2- 172.21.15.81:0/4002950603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf180cd480 0x7faf180ce510 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7faf00009830 tx=0x7faf0002a480 comp rx=0 tx=0).stop 2024-06-29T14:22:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.155+0000 7faf1d04d640 1 -- 172.21.15.81:0/4002950603 shutdown_connections 2024-06-29T14:22:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.155+0000 7faf1d04d640 1 --2- 172.21.15.81:0/4002950603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf180cd480 0x7faf180ce510 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.155+0000 7faf1d04d640 1 -- 172.21.15.81:0/4002950603 >> 172.21.15.81:0/4002950603 conn(0x7faf180b3510 msgr2=0x7faf180b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.155+0000 7faf1d04d640 1 -- 172.21.15.81:0/4002950603 shutdown_connections 2024-06-29T14:22:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.155+0000 7faf1d04d640 1 -- 172.21.15.81:0/4002950603 wait complete. 2024-06-29T14:22:45.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.157+0000 7faf1d04d640 1 Processor -- start 2024-06-29T14:22:45.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.157+0000 7faf1d04d640 1 -- start start 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.158+0000 7faf1d04d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf180b6eb0 0x7faf1815c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.158+0000 7faf1d04d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf180cd480 0x7faf1815ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf1d04d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf180cea50 0x7faf18161900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf1d04d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf180b52b0 con 0x7faf180b6eb0 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf1d04d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faf180b5130 con 0x7faf180cd480 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf1d04d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faf18167660 con 0x7faf180cea50 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf16d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf180b6eb0 0x7faf1815c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf16d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf180b6eb0 0x7faf1815c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48058/0 (socket says 172.21.15.81:48058) 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf16d9d640 1 -- 172.21.15.81:0/1763900743 learned_addr learned my addr 172.21.15.81:0/1763900743 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf0e59c640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf180cd480 0x7faf1815ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf1759e640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf180cea50 0x7faf18161900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf16d9d640 1 -- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf180cea50 msgr2=0x7faf18161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf16d9d640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf180cea50 0x7faf18161900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf16d9d640 1 -- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf180cd480 msgr2=0x7faf1815ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf16d9d640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf180cd480 0x7faf1815ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf16d9d640 1 -- 172.21.15.81:0/1763900743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1815ca10 con 0x7faf180b6eb0 2024-06-29T14:22:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf16d9d640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf180b6eb0 0x7faf1815c4d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faf0800c3c0 tx=0x7faf0800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf0ffff640 1 -- 172.21.15.81:0/1763900743 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf08017070 con 0x7faf180b6eb0 2024-06-29T14:22:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf0ffff640 1 -- 172.21.15.81:0/1763900743 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf080035f0 con 0x7faf180b6eb0 2024-06-29T14:22:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf0ffff640 1 -- 172.21.15.81:0/1763900743 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf08003a40 con 0x7faf180b6eb0 2024-06-29T14:22:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf1d04d640 1 -- 172.21.15.81:0/1763900743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf18164b50 con 0x7faf180b6eb0 2024-06-29T14:22:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.159+0000 7faf1d04d640 1 -- 172.21.15.81:0/1763900743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf18165110 con 0x7faf180b6eb0 2024-06-29T14:22:45.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.160+0000 7faf1d04d640 1 -- 172.21.15.81:0/1763900743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf180ce510 con 0x7faf180b6eb0 2024-06-29T14:22:45.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.161+0000 7faf0ffff640 1 -- 172.21.15.81:0/1763900743 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faf0801a070 con 0x7faf180b6eb0 2024-06-29T14:22:45.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.161+0000 7faf0ffff640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faef8076300 0x7faef8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.162+0000 7faf0ffff640 1 -- 172.21.15.81:0/1763900743 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7faf08096ae0 con 0x7faf180b6eb0 2024-06-29T14:22:45.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.162+0000 7faf0e59c640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faef8076300 0x7faef8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.162+0000 7faf0e59c640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faef8076300 0x7faef8078810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faf00034000 tx=0x7faf00004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:45.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.166+0000 7faf0ffff640 1 -- 172.21.15.81:0/1763900743 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faf080603b0 con 0x7faf180b6eb0 2024-06-29T14:22:45.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.347+0000 7faf1d04d640 1 -- 172.21.15.81:0/1763900743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf180cdf00 con 0x7faf180b6eb0 2024-06-29T14:22:45.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.348+0000 7faf0ffff640 1 -- 172.21.15.81:0/1763900743 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16606 (secure 0 0 0) 0x7faf080930f0 con 0x7faf180b6eb0 2024-06-29T14:22:45.348 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:45.348 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":19,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:44.057673+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 -- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faef8076300 msgr2=0x7faef8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faef8076300 0x7faef8078810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faf00034000 tx=0x7faf00004730 comp rx=0 tx=0).stop 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 -- 172.21.15.81:0/1763900743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf180b6eb0 msgr2=0x7faf1815c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf180b6eb0 0x7faf1815c4d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faf0800c3c0 tx=0x7faf0800c880 comp rx=0 tx=0).stop 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 -- 172.21.15.81:0/1763900743 shutdown_connections 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faef8076300 0x7faef8078810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf180cea50 0x7faf18161900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf180cd480 0x7faf1815ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 --2- 172.21.15.81:0/1763900743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf180b6eb0 0x7faf1815c4d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 -- 172.21.15.81:0/1763900743 >> 172.21.15.81:0/1763900743 conn(0x7faf180b3510 msgr2=0x7faf18070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.350+0000 7faf1d04d640 1 -- 172.21.15.81:0/1763900743 shutdown_connections 2024-06-29T14:22:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.351+0000 7faf1d04d640 1 -- 172.21.15.81:0/1763900743 wait complete. 2024-06-29T14:22:45.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:22:45.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.479+0000 7f71b2e64640 1 -- 172.21.15.81:0/4006495048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7194004800 msgr2=0x7f7194004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.479+0000 7f71b2e64640 1 --2- 172.21.15.81:0/4006495048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7194004800 0x7f7194004c10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f719c009d00 tx=0x7f7194005830 comp rx=0 tx=0).stop 2024-06-29T14:22:45.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.479+0000 7f71b2e64640 1 -- 172.21.15.81:0/4006495048 shutdown_connections 2024-06-29T14:22:45.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.479+0000 7f71b2e64640 1 --2- 172.21.15.81:0/4006495048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7194004800 0x7f7194004c10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.479+0000 7f71b2e64640 1 -- 172.21.15.81:0/4006495048 >> 172.21.15.81:0/4006495048 conn(0x7f71ac0d0930 msgr2=0x7f71ac0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:45.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.479+0000 7f71b2e64640 1 -- 172.21.15.81:0/4006495048 shutdown_connections 2024-06-29T14:22:45.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.479+0000 7f71b2e64640 1 -- 172.21.15.81:0/4006495048 wait complete. 2024-06-29T14:22:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.481+0000 7f71b2e64640 1 Processor -- start 2024-06-29T14:22:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.481+0000 7f71b2e64640 1 -- start start 2024-06-29T14:22:45.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71b2e64640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7194004800 0x7f71ac1f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71b2e64640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71ac0ab090 0x7f71ac0ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71b2e64640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71ac15daa0 0x7f71ac15deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71b2e64640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac15b370 con 0x7f71ac0ab090 2024-06-29T14:22:45.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71b2e64640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac15b1f0 con 0x7f71ac15daa0 2024-06-29T14:22:45.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71b2e64640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f71ac06ef90 con 0x7f7194004800 2024-06-29T14:22:45.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71a3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71ac0ab090 0x7f71ac0ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71a3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71ac0ab090 0x7f71ac0ab4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48066/0 (socket says 172.21.15.81:48066) 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71a3fff640 1 -- 172.21.15.81:0/3327700401 learned_addr learned my addr 172.21.15.81:0/3327700401 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71b1401640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71ac15daa0 0x7f71ac15deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.482+0000 7f71b0c00640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7194004800 0x7f71ac1f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71a3fff640 1 -- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7194004800 msgr2=0x7f71ac1f2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71a3fff640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7194004800 0x7f71ac1f2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71a3fff640 1 -- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71ac15daa0 msgr2=0x7f71ac15deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71a3fff640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71ac15daa0 0x7f71ac15deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71a3fff640 1 -- 172.21.15.81:0/3327700401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71ac15e3f0 con 0x7f71ac0ab090 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71b0c00640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7194004800 0x7f71ac1f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71b1401640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71ac15daa0 0x7f71ac15deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71a3fff640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71ac0ab090 0x7f71ac0ab4a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f719c009e20 tx=0x7f719c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71a17fa640 1 -- 172.21.15.81:0/3327700401 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c038040 con 0x7f71ac0ab090 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71a17fa640 1 -- 172.21.15.81:0/3327700401 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f719c02b9f0 con 0x7f71ac0ab090 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71a17fa640 1 -- 172.21.15.81:0/3327700401 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c003c70 con 0x7f71ac0ab090 2024-06-29T14:22:45.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71b2e64640 1 -- 172.21.15.81:0/3327700401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71ac06c490 con 0x7f71ac0ab090 2024-06-29T14:22:45.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.483+0000 7f71b2e64640 1 -- 172.21.15.81:0/3327700401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71ac06ca50 con 0x7f71ac0ab090 2024-06-29T14:22:45.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.484+0000 7f71b2e64640 1 -- 172.21.15.81:0/3327700401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71ac06e2a0 con 0x7f71ac0ab090 2024-06-29T14:22:45.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.485+0000 7f71a17fa640 1 -- 172.21.15.81:0/3327700401 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f719c043020 con 0x7f71ac0ab090 2024-06-29T14:22:45.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.485+0000 7f71a17fa640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7198076300 0x7f7198078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:45.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.485+0000 7f71a17fa640 1 -- 172.21.15.81:0/3327700401 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f719c0b7d80 con 0x7f71ac0ab090 2024-06-29T14:22:45.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.486+0000 7f71b0c00640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7198076300 0x7f7198078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:45.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.486+0000 7f71b0c00640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7198076300 0x7f7198078810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f71a8001960 tx=0x7f71a8006ef0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:45.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.490+0000 7f71a17fa640 1 -- 172.21.15.81:0/3327700401 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f71ac06e2a0 con 0x7f71ac0ab090 2024-06-29T14:22:45.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:45.711+0000 7f71b2e64640 1 -- 172.21.15.81:0/3327700401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f71ac156130 con 0x7f71ac0ab090 2024-06-29T14:22:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.705+0000 7f71a17fa640 1 -- 172.21.15.81:0/3327700401 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f71ac156130 con 0x7f71ac0ab090 2024-06-29T14:22:46.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.708+0000 7f71b2e64640 1 -- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7198076300 msgr2=0x7f7198078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.708+0000 7f71b2e64640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7198076300 0x7f7198078810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f71a8001960 tx=0x7f71a8006ef0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.708+0000 7f71b2e64640 1 -- 172.21.15.81:0/3327700401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71ac0ab090 msgr2=0x7f71ac0ab4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.708+0000 7f71b2e64640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71ac0ab090 0x7f71ac0ab4a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f719c009e20 tx=0x7f719c003820 comp rx=0 tx=0).stop 2024-06-29T14:22:46.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.709+0000 7f71b2e64640 1 -- 172.21.15.81:0/3327700401 shutdown_connections 2024-06-29T14:22:46.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.709+0000 7f71b2e64640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7198076300 0x7f7198078810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.709+0000 7f71b2e64640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71ac15daa0 0x7f71ac15deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.709+0000 7f71b2e64640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71ac0ab090 0x7f71ac0ab4a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.709+0000 7f71b2e64640 1 --2- 172.21.15.81:0/3327700401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7194004800 0x7f71ac1f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.709+0000 7f71b2e64640 1 -- 172.21.15.81:0/3327700401 >> 172.21.15.81:0/3327700401 conn(0x7f71ac0d0930 msgr2=0x7f71ac0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:46.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.709+0000 7f71b2e64640 1 -- 172.21.15.81:0/3327700401 shutdown_connections 2024-06-29T14:22:46.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.709+0000 7f71b2e64640 1 -- 172.21.15.81:0/3327700401 wait complete. 2024-06-29T14:22:46.721 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:22:46.722 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:22:46.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.845+0000 7feccb3aa640 1 -- 172.21.15.81:0/3667850596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc40b4cf0 msgr2=0x7fecc40d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.845+0000 7feccb3aa640 1 --2- 172.21.15.81:0/3667850596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc40b4cf0 0x7fecc40d00f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fecc40b4550 tx=0x7fecbc02aa30 comp rx=0 tx=0).stop 2024-06-29T14:22:46.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.845+0000 7feccb3aa640 1 -- 172.21.15.81:0/3667850596 shutdown_connections 2024-06-29T14:22:46.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.845+0000 7feccb3aa640 1 --2- 172.21.15.81:0/3667850596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc40b4cf0 0x7fecc40d00f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.845+0000 7feccb3aa640 1 --2- 172.21.15.81:0/3667850596 >> v2:172.21.15.167:3300/0 conn(0x7fecc40cea10 0x7fecc40b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.845+0000 7feccb3aa640 1 --2- 172.21.15.81:0/3667850596 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fecc40b6d60 0x7fecc40ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.845+0000 7feccb3aa640 1 -- 172.21.15.81:0/3667850596 >> 172.21.15.81:0/3667850596 conn(0x7fecc40d1960 msgr2=0x7fecc40d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:46.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.845+0000 7feccb3aa640 1 -- 172.21.15.81:0/3667850596 shutdown_connections 2024-06-29T14:22:46.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.845+0000 7feccb3aa640 1 -- 172.21.15.81:0/3667850596 wait complete. 2024-06-29T14:22:46.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.847+0000 7feccb3aa640 1 Processor -- start 2024-06-29T14:22:46.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.847+0000 7feccb3aa640 1 -- start start 2024-06-29T14:22:46.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7feccb3aa640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fecc40b4cf0 0x7fecc41e3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7feccb3aa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc40b6d60 0x7fecc40c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7feccb3aa640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fecc40cea10 0x7fecc40c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7feccb3aa640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fecc40d09c0 con 0x7fecc40b6d60 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7feccb3aa640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fecc40d0840 con 0x7fecc40b4cf0 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7feccb3aa640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fecc40cc530 con 0x7fecc40cea10 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7fecc8945640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc40b6d60 0x7fecc40c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7fecc8945640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc40b6d60 0x7fecc40c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48072/0 (socket says 172.21.15.81:48072) 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7fecc8945640 1 -- 172.21.15.81:0/3908834103 learned_addr learned my addr 172.21.15.81:0/3908834103 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7fecc9947640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fecc40cea10 0x7fecc40c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7fecc9146640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fecc40b4cf0 0x7fecc41e3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7fecc8945640 1 -- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fecc40cea10 msgr2=0x7fecc40c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7fecc8945640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fecc40cea10 0x7fecc40c8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7fecc8945640 1 -- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fecc40b4cf0 msgr2=0x7fecc41e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:46.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7fecc8945640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fecc40b4cf0 0x7fecc41e3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:46.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.848+0000 7fecc8945640 1 -- 172.21.15.81:0/3908834103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc40c54a0 con 0x7fecc40b6d60 2024-06-29T14:22:46.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.849+0000 7fecc8945640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc40b6d60 0x7fecc40c8a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fecb400d700 tx=0x7fecb400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:46.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.849+0000 7fecb9ffb640 1 -- 172.21.15.81:0/3908834103 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4017070 con 0x7fecc40b6d60 2024-06-29T14:22:46.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.849+0000 7fecb9ffb640 1 -- 172.21.15.81:0/3908834103 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecb4009b10 con 0x7fecc40b6d60 2024-06-29T14:22:46.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.849+0000 7feccb3aa640 1 -- 172.21.15.81:0/3908834103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc40c5780 con 0x7fecc40b6d60 2024-06-29T14:22:46.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.849+0000 7fecb9ffb640 1 -- 172.21.15.81:0/3908834103 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4010460 con 0x7fecc40b6d60 2024-06-29T14:22:46.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.849+0000 7feccb3aa640 1 -- 172.21.15.81:0/3908834103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc40c5d40 con 0x7fecc40b6d60 2024-06-29T14:22:46.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.850+0000 7feccb3aa640 1 -- 172.21.15.81:0/3908834103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecc40ce4d0 con 0x7fecc40b6d60 2024-06-29T14:22:46.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.851+0000 7fecb9ffb640 1 -- 172.21.15.81:0/3908834103 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fecb401a070 con 0x7fecc40b6d60 2024-06-29T14:22:46.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.851+0000 7fecb9ffb640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7feca80762d0 0x7feca80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:46.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.851+0000 7fecb9ffb640 1 -- 172.21.15.81:0/3908834103 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fecb4097740 con 0x7fecc40b6d60 2024-06-29T14:22:46.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.851+0000 7fecc9146640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7feca80762d0 0x7feca80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:46.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.852+0000 7fecc9146640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7feca80762d0 0x7feca80787e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fecc40c5220 tx=0x7fecc000b260 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:46.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:46.855+0000 7fecb9ffb640 1 -- 172.21.15.81:0/3908834103 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fecb4060a50 con 0x7fecc40b6d60 2024-06-29T14:22:47.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.077+0000 7feccb3aa640 1 -- 172.21.15.81:0/3908834103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fecc40cdec0 con 0x7fecc40b6d60 2024-06-29T14:22:47.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.707+0000 7fecb9ffb640 1 -- 172.21.15.81:0/3908834103 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7fecb4064d50 con 0x7fecc40b6d60 2024-06-29T14:22:47.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 -- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7feca80762d0 msgr2=0x7feca80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7feca80762d0 0x7feca80787e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fecc40c5220 tx=0x7fecc000b260 comp rx=0 tx=0).stop 2024-06-29T14:22:47.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 -- 172.21.15.81:0/3908834103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc40b6d60 msgr2=0x7fecc40c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc40b6d60 0x7fecc40c8a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fecb400d700 tx=0x7fecb400dbc0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 -- 172.21.15.81:0/3908834103 shutdown_connections 2024-06-29T14:22:47.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7feca80762d0 0x7feca80787e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fecc40cea10 0x7fecc40c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc40b6d60 0x7fecc40c8a10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 --2- 172.21.15.81:0/3908834103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fecc40b4cf0 0x7fecc41e3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 -- 172.21.15.81:0/3908834103 >> 172.21.15.81:0/3908834103 conn(0x7fecc40d1960 msgr2=0x7fecc40b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:47.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 -- 172.21.15.81:0/3908834103 shutdown_connections 2024-06-29T14:22:47.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.710+0000 7feccb3aa640 1 -- 172.21.15.81:0/3908834103 wait complete. 2024-06-29T14:22:47.723 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:22:47.723 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:47.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.843+0000 7f3c44bf0640 1 -- 172.21.15.81:0/1270982471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c2c005300 msgr2=0x7f3c2c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.843+0000 7f3c44bf0640 1 --2- 172.21.15.81:0/1270982471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c2c005300 0x7f3c2c00b9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3c3800c640 tx=0x7f3c2c005830 comp rx=0 tx=0).stop 2024-06-29T14:22:47.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.843+0000 7f3c44bf0640 1 -- 172.21.15.81:0/1270982471 shutdown_connections 2024-06-29T14:22:47.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.843+0000 7f3c44bf0640 1 --2- 172.21.15.81:0/1270982471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c2c005300 0x7f3c2c00b9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.843+0000 7f3c44bf0640 1 -- 172.21.15.81:0/1270982471 >> 172.21.15.81:0/1270982471 conn(0x7f3c400d1960 msgr2=0x7f3c400d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:47.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.843+0000 7f3c44bf0640 1 -- 172.21.15.81:0/1270982471 shutdown_connections 2024-06-29T14:22:47.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.843+0000 7f3c44bf0640 1 -- 172.21.15.81:0/1270982471 wait complete. 2024-06-29T14:22:47.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.845+0000 7f3c44bf0640 1 Processor -- start 2024-06-29T14:22:47.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.845+0000 7f3c44bf0640 1 -- start start 2024-06-29T14:22:47.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c44bf0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c2c005300 0x7f3c400af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c44bf0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c400b1080 0x7f3c401d8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c44bf0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c401d8650 0x7f3c401e3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c44bf0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3c400ce860 con 0x7f3c2c005300 2024-06-29T14:22:47.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c44bf0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3c400ce6e0 con 0x7f3c401d8650 2024-06-29T14:22:47.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c44bf0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3c400b47b0 con 0x7f3c400b1080 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c3e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c2c005300 0x7f3c400af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c3e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c2c005300 0x7f3c400af120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48078/0 (socket says 172.21.15.81:48078) 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c3e59c640 1 -- 172.21.15.81:0/2132416544 learned_addr learned my addr 172.21.15.81:0/2132416544 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c3ed9d640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c401d8650 0x7f3c401e3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c35d9b640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c400b1080 0x7f3c401d8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c3e59c640 1 -- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c400b1080 msgr2=0x7f3c401d8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c3e59c640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c400b1080 0x7f3c401d8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c3e59c640 1 -- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c401d8650 msgr2=0x7f3c401e3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c3e59c640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c401d8650 0x7f3c401e3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c3e59c640 1 -- 172.21.15.81:0/2132416544 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c401dbb80 con 0x7f3c2c005300 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c3e59c640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c2c005300 0x7f3c400af120 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3c30003ec0 tx=0x7f3c3000b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c377fe640 1 -- 172.21.15.81:0/2132416544 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30010770 con 0x7f3c2c005300 2024-06-29T14:22:47.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c377fe640 1 -- 172.21.15.81:0/2132416544 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c30010910 con 0x7f3c2c005300 2024-06-29T14:22:47.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c44bf0640 1 -- 172.21.15.81:0/2132416544 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c401dbe60 con 0x7f3c2c005300 2024-06-29T14:22:47.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c377fe640 1 -- 172.21.15.81:0/2132416544 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30010d40 con 0x7f3c2c005300 2024-06-29T14:22:47.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.847+0000 7f3c44bf0640 1 -- 172.21.15.81:0/2132416544 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c401dc420 con 0x7f3c2c005300 2024-06-29T14:22:47.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.848+0000 7f3c44bf0640 1 -- 172.21.15.81:0/2132416544 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c400b4c80 con 0x7f3c2c005300 2024-06-29T14:22:47.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.849+0000 7f3c377fe640 1 -- 172.21.15.81:0/2132416544 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3c30023440 con 0x7f3c2c005300 2024-06-29T14:22:47.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.849+0000 7f3c377fe640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3c200762d0 0x7f3c200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:47.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.850+0000 7f3c377fe640 1 -- 172.21.15.81:0/2132416544 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3c30098300 con 0x7f3c2c005300 2024-06-29T14:22:47.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.850+0000 7f3c35d9b640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3c200762d0 0x7f3c200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:47.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.850+0000 7f3c35d9b640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3c200762d0 0x7f3c200787e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3c401e5970 tx=0x7f3c28006b50 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:47.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:47.854+0000 7f3c377fe640 1 -- 172.21.15.81:0/2132416544 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3c30061580 con 0x7f3c2c005300 2024-06-29T14:22:48.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.073+0000 7f3c44bf0640 1 -- 172.21.15.81:0/2132416544 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c400cdec0 con 0x7f3c2c005300 2024-06-29T14:22:48.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.073+0000 7f3c377fe640 1 -- 172.21.15.81:0/2132416544 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5217 (secure 0 0 0) 0x7f3c30065880 con 0x7f3c2c005300 2024-06-29T14:22:48.072 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:48.072 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"btime":"2024-06-29T14:22:47:707315+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4295,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/96042442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":96042442},{"type":"v1","addr":"172.21.15.167:6837","nonce":96042442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3681460169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3681460169},{"type":"v1","addr":"172.21.15.81:6834","nonce":3681460169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3175849002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3175849002},{"type":"v1","addr":"172.21.15.81:6835","nonce":3175849002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:22:44.057632+0000","modified":"2024-06-29T14:22:47.079866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4360":{"gid":4360,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/3579540962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3579540962},{"type":"v1","addr":"172.21.15.167:6836","nonce":3579540962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4360,"qdb_cluster":[4360]},"id":1}]} 2024-06-29T14:22:48.073 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 -- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3c200762d0 msgr2=0x7f3c200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3c200762d0 0x7f3c200787e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3c401e5970 tx=0x7f3c28006b50 comp rx=0 tx=0).stop 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 -- 172.21.15.81:0/2132416544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c2c005300 msgr2=0x7f3c400af120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c2c005300 0x7f3c400af120 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3c30003ec0 tx=0x7f3c3000b9a0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 -- 172.21.15.81:0/2132416544 shutdown_connections 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3c200762d0 0x7f3c200787e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c401d8650 0x7f3c401e3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c400b1080 0x7f3c401d8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 --2- 172.21.15.81:0/2132416544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c2c005300 0x7f3c400af120 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 -- 172.21.15.81:0/2132416544 >> 172.21.15.81:0/2132416544 conn(0x7f3c400d1960 msgr2=0x7f3c401e4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 -- 172.21.15.81:0/2132416544 shutdown_connections 2024-06-29T14:22:48.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.075+0000 7f3c44bf0640 1 -- 172.21.15.81:0/2132416544 wait complete. 2024-06-29T14:22:48.085 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:48.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.199+0000 7f4a4c16f640 1 -- 172.21.15.81:0/4131801238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4415aed0 msgr2=0x7f4a440b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.199+0000 7f4a4c16f640 1 --2- 172.21.15.81:0/4131801238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4415aed0 0x7f4a440b4070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c005fd0 tx=0x7f4a3c02a940 comp rx=0 tx=0).stop 2024-06-29T14:22:48.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.199+0000 7f4a4c16f640 1 -- 172.21.15.81:0/4131801238 shutdown_connections 2024-06-29T14:22:48.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.199+0000 7f4a4c16f640 1 --2- 172.21.15.81:0/4131801238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4415aed0 0x7f4a440b4070 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.199+0000 7f4a4c16f640 1 -- 172.21.15.81:0/4131801238 >> 172.21.15.81:0/4131801238 conn(0x7f4a440d0930 msgr2=0x7f4a440d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:48.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.199+0000 7f4a4c16f640 1 -- 172.21.15.81:0/4131801238 shutdown_connections 2024-06-29T14:22:48.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.199+0000 7f4a4c16f640 1 -- 172.21.15.81:0/4131801238 wait complete. 2024-06-29T14:22:48.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.201+0000 7f4a4c16f640 1 Processor -- start 2024-06-29T14:22:48.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.201+0000 7f4a4c16f640 1 -- start start 2024-06-29T14:22:48.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.202+0000 7f4a4c16f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a4406f6a0 0x7f4a4414b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.202+0000 7f4a4c16f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a440b57b0 0x7f4a440aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.202+0000 7f4a4c16f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4415aed0 0x7f4a44152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.202+0000 7f4a4c16f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4a440b4730 con 0x7f4a4406f6a0 2024-06-29T14:22:48.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.202+0000 7f4a4c16f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4a440b45b0 con 0x7f4a4415aed0 2024-06-29T14:22:48.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.202+0000 7f4a4c16f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4a440b48b0 con 0x7f4a440b57b0 2024-06-29T14:22:48.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.202+0000 7f4a49f0b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a4406f6a0 0x7f4a4414b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.202+0000 7f4a49f0b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a4406f6a0 0x7f4a4414b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48092/0 (socket says 172.21.15.81:48092) 2024-06-29T14:22:48.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.202+0000 7f4a49f0b640 1 -- 172.21.15.81:0/897672216 learned_addr learned my addr 172.21.15.81:0/897672216 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a4a70c640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4415aed0 0x7f4a44152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a4970a640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a440b57b0 0x7f4a440aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a49f0b640 1 -- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a440b57b0 msgr2=0x7f4a440aae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a49f0b640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a440b57b0 0x7f4a440aae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a49f0b640 1 -- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4415aed0 msgr2=0x7f4a44152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a49f0b640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4415aed0 0x7f4a44152c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a49f0b640 1 -- 172.21.15.81:0/897672216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a440a96d0 con 0x7f4a4406f6a0 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a4970a640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a440b57b0 0x7f4a440aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a4a70c640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4415aed0 0x7f4a44152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a49f0b640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a4406f6a0 0x7f4a4414b300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400e690 tx=0x7f4a3400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a427fc640 1 -- 172.21.15.81:0/897672216 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a340052d0 con 0x7f4a4406f6a0 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a427fc640 1 -- 172.21.15.81:0/897672216 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a3400c850 con 0x7f4a4406f6a0 2024-06-29T14:22:48.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a427fc640 1 -- 172.21.15.81:0/897672216 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3400cca0 con 0x7f4a4406f6a0 2024-06-29T14:22:48.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a4c16f640 1 -- 172.21.15.81:0/897672216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a440a99b0 con 0x7f4a4406f6a0 2024-06-29T14:22:48.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.203+0000 7f4a4c16f640 1 -- 172.21.15.81:0/897672216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a440a9f70 con 0x7f4a4406f6a0 2024-06-29T14:22:48.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.204+0000 7f4a4c16f640 1 -- 172.21.15.81:0/897672216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a4406e2a0 con 0x7f4a4406f6a0 2024-06-29T14:22:48.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.205+0000 7f4a427fc640 1 -- 172.21.15.81:0/897672216 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4a340035f0 con 0x7f4a4406f6a0 2024-06-29T14:22:48.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.205+0000 7f4a427fc640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4a300762d0 0x7f4a300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.205+0000 7f4a427fc640 1 -- 172.21.15.81:0/897672216 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4a34099dc0 con 0x7f4a4406f6a0 2024-06-29T14:22:48.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.205+0000 7f4a4970a640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4a300762d0 0x7f4a300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.206+0000 7f4a4970a640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4a300762d0 0x7f4a300787e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4a441524a0 tx=0x7f4a38009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:48.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.209+0000 7f4a427fc640 1 -- 172.21.15.81:0/897672216 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4a4406e2a0 con 0x7f4a4406f6a0 2024-06-29T14:22:48.350 INFO:tasks.ceph.mds.d.smithi167.stderr:2024-06-29T14:22:48.352+0000 7f2ec47fc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:22:48.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.391+0000 7f4a4c16f640 1 -- 172.21.15.81:0/897672216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a440cfcd0 con 0x7f4a4406f6a0 2024-06-29T14:22:48.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.392+0000 7f4a427fc640 1 -- 172.21.15.81:0/897672216 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f4a34063720 con 0x7f4a4406f6a0 2024-06-29T14:22:48.392 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:48.392 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:48.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 -- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4a300762d0 msgr2=0x7f4a300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4a300762d0 0x7f4a300787e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4a441524a0 tx=0x7f4a38009040 comp rx=0 tx=0).stop 2024-06-29T14:22:48.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 -- 172.21.15.81:0/897672216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a4406f6a0 msgr2=0x7f4a4414b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a4406f6a0 0x7f4a4414b300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400e690 tx=0x7f4a3400eb50 comp rx=0 tx=0).stop 2024-06-29T14:22:48.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 -- 172.21.15.81:0/897672216 shutdown_connections 2024-06-29T14:22:48.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4a300762d0 0x7f4a300787e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a4415aed0 0x7f4a44152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a440b57b0 0x7f4a440aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 --2- 172.21.15.81:0/897672216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a4406f6a0 0x7f4a4414b300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 -- 172.21.15.81:0/897672216 >> 172.21.15.81:0/897672216 conn(0x7f4a440d0930 msgr2=0x7f4a440b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:48.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 -- 172.21.15.81:0/897672216 shutdown_connections 2024-06-29T14:22:48.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.395+0000 7f4a4c16f640 1 -- 172.21.15.81:0/897672216 wait complete. 2024-06-29T14:22:48.405 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:22:48.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:22:48.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.518+0000 7f36306b3640 1 -- 172.21.15.81:0/209288572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f362815cd00 msgr2=0x7f362815d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.518+0000 7f36306b3640 1 --2- 172.21.15.81:0/209288572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f362815cd00 0x7f362815d510 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f36280b4f10 tx=0x7f362002a820 comp rx=0 tx=0).stop 2024-06-29T14:22:48.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.518+0000 7f36306b3640 1 -- 172.21.15.81:0/209288572 shutdown_connections 2024-06-29T14:22:48.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.518+0000 7f36306b3640 1 --2- 172.21.15.81:0/209288572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f362815cd00 0x7f362815d510 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.518+0000 7f36306b3640 1 --2- 172.21.15.81:0/209288572 >> v2:172.21.15.167:3300/0 conn(0x7f3628145530 0x7f3628073280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.518+0000 7f36306b3640 1 -- 172.21.15.81:0/209288572 >> 172.21.15.81:0/209288572 conn(0x7f362806ccc0 msgr2=0x7f36280b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:48.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.518+0000 7f36306b3640 1 -- 172.21.15.81:0/209288572 shutdown_connections 2024-06-29T14:22:48.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.518+0000 7f36306b3640 1 -- 172.21.15.81:0/209288572 wait complete. 2024-06-29T14:22:48.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.520+0000 7f36306b3640 1 Processor -- start 2024-06-29T14:22:48.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.520+0000 7f36306b3640 1 -- start start 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f36306b3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36280737c0 0x7f36281410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f36306b3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3628145530 0x7f36280cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f36306b3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f362815cd00 0x7f3628158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f36306b3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f362815dbd0 con 0x7f362815cd00 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f36306b3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f362815da50 con 0x7f36280737c0 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f36306b3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f362815dd50 con 0x7f3628145530 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362ec50640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f362815cd00 0x7f3628158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362ec50640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f362815cd00 0x7f3628158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48104/0 (socket says 172.21.15.81:48104) 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362ec50640 1 -- 172.21.15.81:0/3570281087 learned_addr learned my addr 172.21.15.81:0/3570281087 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362e44f640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36280737c0 0x7f36281410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362dc4e640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3628145530 0x7f36280cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362ec50640 1 -- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3628145530 msgr2=0x7f36280cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362ec50640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3628145530 0x7f36280cd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362ec50640 1 -- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36280737c0 msgr2=0x7f36281410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362ec50640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36280737c0 0x7f36281410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362ec50640 1 -- 172.21.15.81:0/3570281087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36280b4390 con 0x7f362815cd00 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362e44f640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36280737c0 0x7f36281410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.521+0000 7f362dc4e640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3628145530 0x7f36280cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.522+0000 7f362ec50640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f362815cd00 0x7f3628158110 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f362002ae10 tx=0x7f3620032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.522+0000 7f361effd640 1 -- 172.21.15.81:0/3570281087 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362003f020 con 0x7f362815cd00 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.522+0000 7f361effd640 1 -- 172.21.15.81:0/3570281087 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3620003f70 con 0x7f362815cd00 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.522+0000 7f36306b3640 1 -- 172.21.15.81:0/3570281087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36280b40b0 con 0x7f362815cd00 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.522+0000 7f361effd640 1 -- 172.21.15.81:0/3570281087 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620037040 con 0x7f362815cd00 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.522+0000 7f36306b3640 1 -- 172.21.15.81:0/3570281087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36280aed60 con 0x7f362815cd00 2024-06-29T14:22:48.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.523+0000 7f36306b3640 1 -- 172.21.15.81:0/3570281087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36280b3e20 con 0x7f362815cd00 2024-06-29T14:22:48.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.524+0000 7f361effd640 1 -- 172.21.15.81:0/3570281087 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3620042020 con 0x7f362815cd00 2024-06-29T14:22:48.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.524+0000 7f361effd640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f360c076300 0x7f360c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.524+0000 7f361effd640 1 -- 172.21.15.81:0/3570281087 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f36200b8180 con 0x7f362815cd00 2024-06-29T14:22:48.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.524+0000 7f362e44f640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f360c076300 0x7f360c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.525+0000 7f362e44f640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f360c076300 0x7f360c078810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f362400ba60 tx=0x7f3624005cf0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:48.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.528+0000 7f361effd640 1 -- 172.21.15.81:0/3570281087 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3620049020 con 0x7f362815cd00 2024-06-29T14:22:48.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.704+0000 7f36306b3640 1 -- 172.21.15.81:0/3570281087 --> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f362815e2c0 con 0x7f360c076300 2024-06-29T14:22:48.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.709+0000 7f361effd640 1 -- 172.21.15.81:0/3570281087 <== mgr.4126 v2:172.21.15.167:6832/3101542440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251865 (secure 0 0 0) 0x7f362815e2c0 con 0x7f360c076300 2024-06-29T14:22:48.713 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:48.715 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:22:48.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.712+0000 7f36306b3640 1 -- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f360c076300 msgr2=0x7f360c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.712+0000 7f36306b3640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f360c076300 0x7f360c078810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f362400ba60 tx=0x7f3624005cf0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.712+0000 7f36306b3640 1 -- 172.21.15.81:0/3570281087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f362815cd00 msgr2=0x7f3628158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.712+0000 7f36306b3640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f362815cd00 0x7f3628158110 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f362002ae10 tx=0x7f3620032c90 comp rx=0 tx=0).stop 2024-06-29T14:22:48.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.712+0000 7f36306b3640 1 -- 172.21.15.81:0/3570281087 shutdown_connections 2024-06-29T14:22:48.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.712+0000 7f36306b3640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f360c076300 0x7f360c078810 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.712+0000 7f36306b3640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f362815cd00 0x7f3628158110 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.712+0000 7f36306b3640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3628145530 0x7f36280cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.712+0000 7f36306b3640 1 --2- 172.21.15.81:0/3570281087 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f36280737c0 0x7f36281410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.712+0000 7f36306b3640 1 -- 172.21.15.81:0/3570281087 >> 172.21.15.81:0/3570281087 conn(0x7f362806ccc0 msgr2=0x7f362815b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:48.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.713+0000 7f36306b3640 1 -- 172.21.15.81:0/3570281087 shutdown_connections 2024-06-29T14:22:48.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.713+0000 7f36306b3640 1 -- 172.21.15.81:0/3570281087 wait complete. 2024-06-29T14:22:48.723 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:22:48.705188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218892,"kb_used_data":2708,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755828,"statfs":{"total":773094113280,"available":772869967872,"internally_reserved":0,"allocated":2772992,"data_stored":1695693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001510"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.726473+0000","last_change":"2024-06-29T14:22:44.722050+0000","last_active":"2024-06-29T14:22:44.726473+0000","last_peered":"2024-06-29T14:22:44.726473+0000","last_clean":"2024-06-29T14:22:44.726473+0000","last_became_active":"2024-06-29T14:22:44.721719+0000","last_became_peered":"2024-06-29T14:22:44.721719+0000","last_unstale":"2024-06-29T14:22:44.726473+0000","last_undegraded":"2024-06-29T14:22:44.726473+0000","last_fullsized":"2024-06-29T14:22:44.726473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:29.350494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.715538+0000","last_change":"2024-06-29T14:22:44.709906+0000","last_active":"2024-06-29T14:22:44.715538+0000","last_peered":"2024-06-29T14:22:44.715538+0000","last_clean":"2024-06-29T14:22:44.715538+0000","last_became_active":"2024-06-29T14:22:44.709406+0000","last_became_peered":"2024-06-29T14:22:44.709406+0000","last_unstale":"2024-06-29T14:22:44.715538+0000","last_undegraded":"2024-06-29T14:22:44.715538+0000","last_fullsized":"2024-06-29T14:22:44.715538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.708302+0000","last_change":"2024-06-29T14:22:44.705357+0000","last_active":"2024-06-29T14:22:44.708302+0000","last_peered":"2024-06-29T14:22:44.708302+0000","last_clean":"2024-06-29T14:22:44.708302+0000","last_became_active":"2024-06-29T14:22:44.704402+0000","last_became_peered":"2024-06-29T14:22:44.704402+0000","last_unstale":"2024-06-29T14:22:44.708302+0000","last_undegraded":"2024-06-29T14:22:44.708302+0000","last_fullsized":"2024-06-29T14:22:44.708302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:27.065414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711532+0000","last_change":"2024-06-29T14:22:44.708801+0000","last_active":"2024-06-29T14:22:44.711532+0000","last_peered":"2024-06-29T14:22:44.711532+0000","last_clean":"2024-06-29T14:22:44.711532+0000","last_became_active":"2024-06-29T14:22:44.708456+0000","last_became_peered":"2024-06-29T14:22:44.708456+0000","last_unstale":"2024-06-29T14:22:44.711532+0000","last_undegraded":"2024-06-29T14:22:44.711532+0000","last_fullsized":"2024-06-29T14:22:44.711532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723935+0000","last_change":"2024-06-29T14:22:44.719917+0000","last_active":"2024-06-29T14:22:44.723935+0000","last_peered":"2024-06-29T14:22:44.723935+0000","last_clean":"2024-06-29T14:22:44.723935+0000","last_became_active":"2024-06-29T14:22:44.719617+0000","last_became_peered":"2024-06-29T14:22:44.719617+0000","last_unstale":"2024-06-29T14:22:44.723935+0000","last_undegraded":"2024-06-29T14:22:44.723935+0000","last_fullsized":"2024-06-29T14:22:44.723935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.707418+0000","last_change":"2024-06-29T14:22:44.701697+0000","last_active":"2024-06-29T14:22:44.707418+0000","last_peered":"2024-06-29T14:22:44.707418+0000","last_clean":"2024-06-29T14:22:44.707418+0000","last_became_active":"2024-06-29T14:22:44.701325+0000","last_became_peered":"2024-06-29T14:22:44.701325+0000","last_unstale":"2024-06-29T14:22:44.707418+0000","last_undegraded":"2024-06-29T14:22:44.707418+0000","last_fullsized":"2024-06-29T14:22:44.707418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723978+0000","last_change":"2024-06-29T14:22:44.720635+0000","last_active":"2024-06-29T14:22:44.723978+0000","last_peered":"2024-06-29T14:22:44.723978+0000","last_clean":"2024-06-29T14:22:44.723978+0000","last_became_active":"2024-06-29T14:22:44.720316+0000","last_became_peered":"2024-06-29T14:22:44.720316+0000","last_unstale":"2024-06-29T14:22:44.723978+0000","last_undegraded":"2024-06-29T14:22:44.723978+0000","last_fullsized":"2024-06-29T14:22:44.723978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.716831+0000","last_change":"2024-06-29T14:22:44.706868+0000","last_active":"2024-06-29T14:22:44.716831+0000","last_peered":"2024-06-29T14:22:44.716831+0000","last_clean":"2024-06-29T14:22:44.716831+0000","last_became_active":"2024-06-29T14:22:44.706298+0000","last_became_peered":"2024-06-29T14:22:44.706298+0000","last_unstale":"2024-06-29T14:22:44.716831+0000","last_undegraded":"2024-06-29T14:22:44.716831+0000","last_fullsized":"2024-06-29T14:22:44.716831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.726541+0000","last_change":"2024-06-29T14:22:44.722834+0000","last_active":"2024-06-29T14:22:44.726541+0000","last_peered":"2024-06-29T14:22:44.726541+0000","last_clean":"2024-06-29T14:22:44.726541+0000","last_became_active":"2024-06-29T14:22:44.722345+0000","last_became_peered":"2024-06-29T14:22:44.722345+0000","last_unstale":"2024-06-29T14:22:44.726541+0000","last_undegraded":"2024-06-29T14:22:44.726541+0000","last_fullsized":"2024-06-29T14:22:44.726541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:25.905572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.706519+0000","last_change":"2024-06-29T14:22:44.702528+0000","last_active":"2024-06-29T14:22:44.706519+0000","last_peered":"2024-06-29T14:22:44.706519+0000","last_clean":"2024-06-29T14:22:44.706519+0000","last_became_active":"2024-06-29T14:22:44.702246+0000","last_became_peered":"2024-06-29T14:22:44.702246+0000","last_unstale":"2024-06-29T14:22:44.706519+0000","last_undegraded":"2024-06-29T14:22:44.706519+0000","last_fullsized":"2024-06-29T14:22:44.706519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.708795+0000","last_change":"2024-06-29T14:22:44.703308+0000","last_active":"2024-06-29T14:22:44.708795+0000","last_peered":"2024-06-29T14:22:44.708795+0000","last_clean":"2024-06-29T14:22:44.708795+0000","last_became_active":"2024-06-29T14:22:44.703018+0000","last_became_peered":"2024-06-29T14:22:44.703018+0000","last_unstale":"2024-06-29T14:22:44.708795+0000","last_undegraded":"2024-06-29T14:22:44.708795+0000","last_fullsized":"2024-06-29T14:22:44.708795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:44.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.705106+0000","last_change":"2024-06-29T14:22:44.702902+0000","last_active":"2024-06-29T14:22:44.705106+0000","last_peered":"2024-06-29T14:22:44.705106+0000","last_clean":"2024-06-29T14:22:44.705106+0000","last_became_active":"2024-06-29T14:22:44.702595+0000","last_became_peered":"2024-06-29T14:22:44.702595+0000","last_unstale":"2024-06-29T14:22:44.705106+0000","last_undegraded":"2024-06-29T14:22:44.705106+0000","last_fullsized":"2024-06-29T14:22:44.705106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:27.065414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713319+0000","last_change":"2024-06-29T14:22:44.703127+0000","last_active":"2024-06-29T14:22:44.713319+0000","last_peered":"2024-06-29T14:22:44.713319+0000","last_clean":"2024-06-29T14:22:44.713319+0000","last_became_active":"2024-06-29T14:22:44.702813+0000","last_became_peered":"2024-06-29T14:22:44.702813+0000","last_unstale":"2024-06-29T14:22:44.713319+0000","last_undegraded":"2024-06-29T14:22:44.713319+0000","last_fullsized":"2024-06-29T14:22:44.713319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:40.770948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.718503+0000","last_change":"2024-06-29T14:22:44.714140+0000","last_active":"2024-06-29T14:22:44.718503+0000","last_peered":"2024-06-29T14:22:44.718503+0000","last_clean":"2024-06-29T14:22:44.718503+0000","last_became_active":"2024-06-29T14:22:44.713863+0000","last_became_peered":"2024-06-29T14:22:44.713863+0000","last_unstale":"2024-06-29T14:22:44.718503+0000","last_undegraded":"2024-06-29T14:22:44.718503+0000","last_fullsized":"2024-06-29T14:22:44.718503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.702367+0000","last_change":"2024-06-29T14:22:44.699851+0000","last_active":"2024-06-29T14:22:44.702367+0000","last_peered":"2024-06-29T14:22:44.702367+0000","last_clean":"2024-06-29T14:22:44.702367+0000","last_became_active":"2024-06-29T14:22:44.697960+0000","last_became_peered":"2024-06-29T14:22:44.697960+0000","last_unstale":"2024-06-29T14:22:44.702367+0000","last_undegraded":"2024-06-29T14:22:44.702367+0000","last_fullsized":"2024-06-29T14:22:44.702367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711405+0000","last_change":"2024-06-29T14:22:44.708069+0000","last_active":"2024-06-29T14:22:44.711405+0000","last_peered":"2024-06-29T14:22:44.711405+0000","last_clean":"2024-06-29T14:22:44.711405+0000","last_became_active":"2024-06-29T14:22:44.707765+0000","last_became_peered":"2024-06-29T14:22:44.707765+0000","last_unstale":"2024-06-29T14:22:44.711405+0000","last_undegraded":"2024-06-29T14:22:44.711405+0000","last_fullsized":"2024-06-29T14:22:44.711405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723894+0000","last_change":"2024-06-29T14:22:44.721271+0000","last_active":"2024-06-29T14:22:44.723894+0000","last_peered":"2024-06-29T14:22:44.723894+0000","last_clean":"2024-06-29T14:22:44.723894+0000","last_became_active":"2024-06-29T14:22:44.720978+0000","last_became_peered":"2024-06-29T14:22:44.720978+0000","last_unstale":"2024-06-29T14:22:44.723894+0000","last_undegraded":"2024-06-29T14:22:44.723894+0000","last_fullsized":"2024-06-29T14:22:44.723894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.705796+0000","last_change":"2024-06-29T14:22:44.702764+0000","last_active":"2024-06-29T14:22:44.705796+0000","last_peered":"2024-06-29T14:22:44.705796+0000","last_clean":"2024-06-29T14:22:44.705796+0000","last_became_active":"2024-06-29T14:22:44.702408+0000","last_became_peered":"2024-06-29T14:22:44.702408+0000","last_unstale":"2024-06-29T14:22:44.705796+0000","last_undegraded":"2024-06-29T14:22:44.705796+0000","last_fullsized":"2024-06-29T14:22:44.705796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:40.770948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.722007+0000","last_change":"2024-06-29T14:22:44.718193+0000","last_active":"2024-06-29T14:22:44.722007+0000","last_peered":"2024-06-29T14:22:44.722007+0000","last_clean":"2024-06-29T14:22:44.722007+0000","last_became_active":"2024-06-29T14:22:44.717900+0000","last_became_peered":"2024-06-29T14:22:44.717900+0000","last_unstale":"2024-06-29T14:22:44.722007+0000","last_undegraded":"2024-06-29T14:22:44.722007+0000","last_fullsized":"2024-06-29T14:22:44.722007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.705676+0000","last_change":"2024-06-29T14:22:44.701520+0000","last_active":"2024-06-29T14:22:44.705676+0000","last_peered":"2024-06-29T14:22:44.705676+0000","last_clean":"2024-06-29T14:22:44.705676+0000","last_became_active":"2024-06-29T14:22:44.701106+0000","last_became_peered":"2024-06-29T14:22:44.701106+0000","last_unstale":"2024-06-29T14:22:44.705676+0000","last_undegraded":"2024-06-29T14:22:44.705676+0000","last_fullsized":"2024-06-29T14:22:44.705676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:40.008975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.720248+0000","last_change":"2024-06-29T14:22:44.717603+0000","last_active":"2024-06-29T14:22:44.720248+0000","last_peered":"2024-06-29T14:22:44.720248+0000","last_clean":"2024-06-29T14:22:44.720248+0000","last_became_active":"2024-06-29T14:22:44.717305+0000","last_became_peered":"2024-06-29T14:22:44.717305+0000","last_unstale":"2024-06-29T14:22:44.720248+0000","last_undegraded":"2024-06-29T14:22:44.720248+0000","last_fullsized":"2024-06-29T14:22:44.720248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:44.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.721712+0000","last_change":"2024-06-29T14:22:44.717305+0000","last_active":"2024-06-29T14:22:44.721712+0000","last_peered":"2024-06-29T14:22:44.721712+0000","last_clean":"2024-06-29T14:22:44.721712+0000","last_became_active":"2024-06-29T14:22:44.716953+0000","last_became_peered":"2024-06-29T14:22:44.716953+0000","last_unstale":"2024-06-29T14:22:44.721712+0000","last_undegraded":"2024-06-29T14:22:44.721712+0000","last_fullsized":"2024-06-29T14:22:44.721712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713365+0000","last_change":"2024-06-29T14:22:44.701947+0000","last_active":"2024-06-29T14:22:44.713365+0000","last_peered":"2024-06-29T14:22:44.713365+0000","last_clean":"2024-06-29T14:22:44.713365+0000","last_became_active":"2024-06-29T14:22:44.701624+0000","last_became_peered":"2024-06-29T14:22:44.701624+0000","last_unstale":"2024-06-29T14:22:44.713365+0000","last_undegraded":"2024-06-29T14:22:44.713365+0000","last_fullsized":"2024-06-29T14:22:44.713365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:10.828903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711433+0000","last_change":"2024-06-29T14:22:44.709275+0000","last_active":"2024-06-29T14:22:44.711433+0000","last_peered":"2024-06-29T14:22:44.711433+0000","last_clean":"2024-06-29T14:22:44.711433+0000","last_became_active":"2024-06-29T14:22:44.708702+0000","last_became_peered":"2024-06-29T14:22:44.708702+0000","last_unstale":"2024-06-29T14:22:44.711433+0000","last_undegraded":"2024-06-29T14:22:44.711433+0000","last_fullsized":"2024-06-29T14:22:44.711433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713308+0000","last_change":"2024-06-29T14:22:44.702814+0000","last_active":"2024-06-29T14:22:44.713308+0000","last_peered":"2024-06-29T14:22:44.713308+0000","last_clean":"2024-06-29T14:22:44.713308+0000","last_became_active":"2024-06-29T14:22:44.702479+0000","last_became_peered":"2024-06-29T14:22:44.702479+0000","last_unstale":"2024-06-29T14:22:44.713308+0000","last_undegraded":"2024-06-29T14:22:44.713308+0000","last_fullsized":"2024-06-29T14:22:44.713308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:46.021153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723587+0000","last_change":"2024-06-29T14:22:44.718581+0000","last_active":"2024-06-29T14:22:44.723587+0000","last_peered":"2024-06-29T14:22:44.723587+0000","last_clean":"2024-06-29T14:22:44.723587+0000","last_became_active":"2024-06-29T14:22:44.718277+0000","last_became_peered":"2024-06-29T14:22:44.718277+0000","last_unstale":"2024-06-29T14:22:44.723587+0000","last_undegraded":"2024-06-29T14:22:44.723587+0000","last_fullsized":"2024-06-29T14:22:44.723587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.710075+0000","last_change":"2024-06-29T14:22:44.704193+0000","last_active":"2024-06-29T14:22:44.710075+0000","last_peered":"2024-06-29T14:22:44.710075+0000","last_clean":"2024-06-29T14:22:44.710075+0000","last_became_active":"2024-06-29T14:22:44.703892+0000","last_became_peered":"2024-06-29T14:22:44.703892+0000","last_unstale":"2024-06-29T14:22:44.710075+0000","last_undegraded":"2024-06-29T14:22:44.710075+0000","last_fullsized":"2024-06-29T14:22:44.710075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713271+0000","last_change":"2024-06-29T14:22:44.703952+0000","last_active":"2024-06-29T14:22:44.713271+0000","last_peered":"2024-06-29T14:22:44.713271+0000","last_clean":"2024-06-29T14:22:44.713271+0000","last_became_active":"2024-06-29T14:22:44.703431+0000","last_became_peered":"2024-06-29T14:22:44.703431+0000","last_unstale":"2024-06-29T14:22:44.713271+0000","last_undegraded":"2024-06-29T14:22:44.713271+0000","last_fullsized":"2024-06-29T14:22:44.713271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:40.008975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.718495+0000","last_change":"2024-06-29T14:22:44.714045+0000","last_active":"2024-06-29T14:22:44.718495+0000","last_peered":"2024-06-29T14:22:44.718495+0000","last_clean":"2024-06-29T14:22:44.718495+0000","last_became_active":"2024-06-29T14:22:44.713749+0000","last_became_peered":"2024-06-29T14:22:44.713749+0000","last_unstale":"2024-06-29T14:22:44.718495+0000","last_undegraded":"2024-06-29T14:22:44.718495+0000","last_fullsized":"2024-06-29T14:22:44.718495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:40.008975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711566+0000","last_change":"2024-06-29T14:22:44.709553+0000","last_active":"2024-06-29T14:22:44.711566+0000","last_peered":"2024-06-29T14:22:44.711566+0000","last_clean":"2024-06-29T14:22:44.711566+0000","last_became_active":"2024-06-29T14:22:44.709153+0000","last_became_peered":"2024-06-29T14:22:44.709153+0000","last_unstale":"2024-06-29T14:22:44.711566+0000","last_undegraded":"2024-06-29T14:22:44.711566+0000","last_fullsized":"2024-06-29T14:22:44.711566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.702909+0000","last_change":"2024-06-29T14:22:44.699167+0000","last_active":"2024-06-29T14:22:44.702909+0000","last_peered":"2024-06-29T14:22:44.702909+0000","last_clean":"2024-06-29T14:22:44.702909+0000","last_became_active":"2024-06-29T14:22:44.698265+0000","last_became_peered":"2024-06-29T14:22:44.698265+0000","last_unstale":"2024-06-29T14:22:44.702909+0000","last_undegraded":"2024-06-29T14:22:44.702909+0000","last_fullsized":"2024-06-29T14:22:44.702909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:46.021153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713809+0000","last_change":"2024-06-29T14:22:44.710341+0000","last_active":"2024-06-29T14:22:44.713809+0000","last_peered":"2024-06-29T14:22:44.713809+0000","last_clean":"2024-06-29T14:22:44.713809+0000","last_became_active":"2024-06-29T14:22:44.709992+0000","last_became_peered":"2024-06-29T14:22:44.709992+0000","last_unstale":"2024-06-29T14:22:44.713809+0000","last_undegraded":"2024-06-29T14:22:44.713809+0000","last_fullsized":"2024-06-29T14:22:44.713809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717603+0000","last_change":"2024-06-29T14:22:44.713512+0000","last_active":"2024-06-29T14:22:44.717603+0000","last_peered":"2024-06-29T14:22:44.717603+0000","last_clean":"2024-06-29T14:22:44.717603+0000","last_became_active":"2024-06-29T14:22:44.713208+0000","last_became_peered":"2024-06-29T14:22:44.713208+0000","last_unstale":"2024-06-29T14:22:44.717603+0000","last_undegraded":"2024-06-29T14:22:44.717603+0000","last_fullsized":"2024-06-29T14:22:44.717603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:16.078185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.719080+0000","last_change":"2024-06-29T14:22:43.711234+0000","last_active":"2024-06-29T14:22:44.719080+0000","last_peered":"2024-06-29T14:22:44.719080+0000","last_clean":"2024-06-29T14:22:44.719080+0000","last_became_active":"2024-06-29T14:22:43.710904+0000","last_became_peered":"2024-06-29T14:22:43.710904+0000","last_unstale":"2024-06-29T14:22:44.719080+0000","last_undegraded":"2024-06-29T14:22:44.719080+0000","last_fullsized":"2024-06-29T14:22:44.719080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.708355+0000","last_change":"2024-06-29T14:22:44.706122+0000","last_active":"2024-06-29T14:22:44.708355+0000","last_peered":"2024-06-29T14:22:44.708355+0000","last_clean":"2024-06-29T14:22:44.708355+0000","last_became_active":"2024-06-29T14:22:44.705746+0000","last_became_peered":"2024-06-29T14:22:44.705746+0000","last_unstale":"2024-06-29T14:22:44.708355+0000","last_undegraded":"2024-06-29T14:22:44.708355+0000","last_fullsized":"2024-06-29T14:22:44.708355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:16.078185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.704301+0000","last_change":"2024-06-29T14:22:43.700941+0000","last_active":"2024-06-29T14:22:44.704301+0000","last_peered":"2024-06-29T14:22:44.704301+0000","last_clean":"2024-06-29T14:22:44.704301+0000","last_became_active":"2024-06-29T14:22:43.700234+0000","last_became_peered":"2024-06-29T14:22:43.700234+0000","last_unstale":"2024-06-29T14:22:44.704301+0000","last_undegraded":"2024-06-29T14:22:44.704301+0000","last_fullsized":"2024-06-29T14:22:44.704301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.701067+0000","last_change":"2024-06-29T14:22:43.707763+0000","last_active":"2024-06-29T14:22:44.701067+0000","last_peered":"2024-06-29T14:22:44.701067+0000","last_clean":"2024-06-29T14:22:44.701067+0000","last_became_active":"2024-06-29T14:22:43.707378+0000","last_became_peered":"2024-06-29T14:22:43.707378+0000","last_unstale":"2024-06-29T14:22:44.701067+0000","last_undegraded":"2024-06-29T14:22:44.701067+0000","last_fullsized":"2024-06-29T14:22:44.701067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:26.060090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713428+0000","last_change":"2024-06-29T14:22:44.705946+0000","last_active":"2024-06-29T14:22:44.713428+0000","last_peered":"2024-06-29T14:22:44.713428+0000","last_clean":"2024-06-29T14:22:44.713428+0000","last_became_active":"2024-06-29T14:22:44.705469+0000","last_became_peered":"2024-06-29T14:22:44.705469+0000","last_unstale":"2024-06-29T14:22:44.713428+0000","last_undegraded":"2024-06-29T14:22:44.713428+0000","last_fullsized":"2024-06-29T14:22:44.713428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.691119+0000","last_change":"2024-06-29T14:22:43.698273+0000","last_active":"2024-06-29T14:22:44.691119+0000","last_peered":"2024-06-29T14:22:44.691119+0000","last_clean":"2024-06-29T14:22:44.691119+0000","last_became_active":"2024-06-29T14:22:43.697950+0000","last_became_peered":"2024-06-29T14:22:43.697950+0000","last_unstale":"2024-06-29T14:22:44.691119+0000","last_undegraded":"2024-06-29T14:22:44.691119+0000","last_fullsized":"2024-06-29T14:22:44.691119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.712749+0000","last_change":"2024-06-29T14:22:44.708264+0000","last_active":"2024-06-29T14:22:44.712749+0000","last_peered":"2024-06-29T14:22:44.712749+0000","last_clean":"2024-06-29T14:22:44.712749+0000","last_became_active":"2024-06-29T14:22:44.707924+0000","last_became_peered":"2024-06-29T14:22:44.707924+0000","last_unstale":"2024-06-29T14:22:44.712749+0000","last_undegraded":"2024-06-29T14:22:44.712749+0000","last_fullsized":"2024-06-29T14:22:44.712749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.692397+0000","last_change":"2024-06-29T14:22:43.706343+0000","last_active":"2024-06-29T14:22:44.692397+0000","last_peered":"2024-06-29T14:22:44.692397+0000","last_clean":"2024-06-29T14:22:44.692397+0000","last_became_active":"2024-06-29T14:22:43.706009+0000","last_became_peered":"2024-06-29T14:22:43.706009+0000","last_unstale":"2024-06-29T14:22:44.692397+0000","last_undegraded":"2024-06-29T14:22:44.692397+0000","last_fullsized":"2024-06-29T14:22:44.692397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.721748+0000","last_change":"2024-06-29T14:22:44.717406+0000","last_active":"2024-06-29T14:22:44.721748+0000","last_peered":"2024-06-29T14:22:44.721748+0000","last_clean":"2024-06-29T14:22:44.721748+0000","last_became_active":"2024-06-29T14:22:44.717095+0000","last_became_peered":"2024-06-29T14:22:44.717095+0000","last_unstale":"2024-06-29T14:22:44.721748+0000","last_undegraded":"2024-06-29T14:22:44.721748+0000","last_fullsized":"2024-06-29T14:22:44.721748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.690530+0000","last_change":"2024-06-29T14:22:43.701234+0000","last_active":"2024-06-29T14:22:44.690530+0000","last_peered":"2024-06-29T14:22:44.690530+0000","last_clean":"2024-06-29T14:22:44.690530+0000","last_became_active":"2024-06-29T14:22:43.700923+0000","last_became_peered":"2024-06-29T14:22:43.700923+0000","last_unstale":"2024-06-29T14:22:44.690530+0000","last_undegraded":"2024-06-29T14:22:44.690530+0000","last_fullsized":"2024-06-29T14:22:44.690530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711635+0000","last_change":"2024-06-29T14:22:44.708529+0000","last_active":"2024-06-29T14:22:44.711635+0000","last_peered":"2024-06-29T14:22:44.711635+0000","last_clean":"2024-06-29T14:22:44.711635+0000","last_became_active":"2024-06-29T14:22:44.708019+0000","last_became_peered":"2024-06-29T14:22:44.708019+0000","last_unstale":"2024-06-29T14:22:44.711635+0000","last_undegraded":"2024-06-29T14:22:44.711635+0000","last_fullsized":"2024-06-29T14:22:44.711635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:10.828903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717881+0000","last_change":"2024-06-29T14:22:43.705026+0000","last_active":"2024-06-29T14:22:44.717881+0000","last_peered":"2024-06-29T14:22:44.717881+0000","last_clean":"2024-06-29T14:22:44.717881+0000","last_became_active":"2024-06-29T14:22:43.704704+0000","last_became_peered":"2024-06-29T14:22:43.704704+0000","last_unstale":"2024-06-29T14:22:44.717881+0000","last_undegraded":"2024-06-29T14:22:44.717881+0000","last_fullsized":"2024-06-29T14:22:44.717881+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.718863+0000","last_change":"2024-06-29T14:22:44.716149+0000","last_active":"2024-06-29T14:22:44.718863+0000","last_peered":"2024-06-29T14:22:44.718863+0000","last_clean":"2024-06-29T14:22:44.718863+0000","last_became_active":"2024-06-29T14:22:44.715832+0000","last_became_peered":"2024-06-29T14:22:44.715832+0000","last_unstale":"2024-06-29T14:22:44.718863+0000","last_undegraded":"2024-06-29T14:22:44.718863+0000","last_fullsized":"2024-06-29T14:22:44.718863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:10.828903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.726574+0000","last_change":"2024-06-29T14:22:43.707443+0000","last_active":"2024-06-29T14:22:44.726574+0000","last_peered":"2024-06-29T14:22:44.726574+0000","last_clean":"2024-06-29T14:22:44.726574+0000","last_became_active":"2024-06-29T14:22:43.707109+0000","last_became_peered":"2024-06-29T14:22:43.707109+0000","last_unstale":"2024-06-29T14:22:44.726574+0000","last_undegraded":"2024-06-29T14:22:44.726574+0000","last_fullsized":"2024-06-29T14:22:44.726574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723833+0000","last_change":"2024-06-29T14:22:44.719076+0000","last_active":"2024-06-29T14:22:44.723833+0000","last_peered":"2024-06-29T14:22:44.723833+0000","last_clean":"2024-06-29T14:22:44.723833+0000","last_became_active":"2024-06-29T14:22:44.718736+0000","last_became_peered":"2024-06-29T14:22:44.718736+0000","last_unstale":"2024-06-29T14:22:44.723833+0000","last_undegraded":"2024-06-29T14:22:44.723833+0000","last_fullsized":"2024-06-29T14:22:44.723833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.709281+0000","last_change":"2024-06-29T14:22:43.700129+0000","last_active":"2024-06-29T14:22:44.709281+0000","last_peered":"2024-06-29T14:22:44.709281+0000","last_clean":"2024-06-29T14:22:44.709281+0000","last_became_active":"2024-06-29T14:22:43.699771+0000","last_became_peered":"2024-06-29T14:22:43.699771+0000","last_unstale":"2024-06-29T14:22:44.709281+0000","last_undegraded":"2024-06-29T14:22:44.709281+0000","last_fullsized":"2024-06-29T14:22:44.709281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:53.109228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.718527+0000","last_change":"2024-06-29T14:22:44.716496+0000","last_active":"2024-06-29T14:22:44.718527+0000","last_peered":"2024-06-29T14:22:44.718527+0000","last_clean":"2024-06-29T14:22:44.718527+0000","last_became_active":"2024-06-29T14:22:44.716244+0000","last_became_peered":"2024-06-29T14:22:44.716244+0000","last_unstale":"2024-06-29T14:22:44.718527+0000","last_undegraded":"2024-06-29T14:22:44.718527+0000","last_fullsized":"2024-06-29T14:22:44.718527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.689538+0000","last_change":"2024-06-29T14:22:43.706155+0000","last_active":"2024-06-29T14:22:44.689538+0000","last_peered":"2024-06-29T14:22:44.689538+0000","last_clean":"2024-06-29T14:22:44.689538+0000","last_became_active":"2024-06-29T14:22:43.705468+0000","last_became_peered":"2024-06-29T14:22:43.705468+0000","last_unstale":"2024-06-29T14:22:44.689538+0000","last_undegraded":"2024-06-29T14:22:44.689538+0000","last_fullsized":"2024-06-29T14:22:44.689538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723909+0000","last_change":"2024-06-29T14:22:44.719377+0000","last_active":"2024-06-29T14:22:44.723909+0000","last_peered":"2024-06-29T14:22:44.723909+0000","last_clean":"2024-06-29T14:22:44.723909+0000","last_became_active":"2024-06-29T14:22:44.719065+0000","last_became_peered":"2024-06-29T14:22:44.719065+0000","last_unstale":"2024-06-29T14:22:44.723909+0000","last_undegraded":"2024-06-29T14:22:44.723909+0000","last_fullsized":"2024-06-29T14:22:44.723909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.692197+0000","last_change":"2024-06-29T14:22:43.698317+0000","last_active":"2024-06-29T14:22:44.692197+0000","last_peered":"2024-06-29T14:22:44.692197+0000","last_clean":"2024-06-29T14:22:44.692197+0000","last_became_active":"2024-06-29T14:22:43.697987+0000","last_became_peered":"2024-06-29T14:22:43.697987+0000","last_unstale":"2024-06-29T14:22:44.692197+0000","last_undegraded":"2024-06-29T14:22:44.692197+0000","last_fullsized":"2024-06-29T14:22:44.692197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713785+0000","last_change":"2024-06-29T14:22:44.710262+0000","last_active":"2024-06-29T14:22:44.713785+0000","last_peered":"2024-06-29T14:22:44.713785+0000","last_clean":"2024-06-29T14:22:44.713785+0000","last_became_active":"2024-06-29T14:22:44.709948+0000","last_became_peered":"2024-06-29T14:22:44.709948+0000","last_unstale":"2024-06-29T14:22:44.713785+0000","last_undegraded":"2024-06-29T14:22:44.713785+0000","last_fullsized":"2024-06-29T14:22:44.713785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.712977+0000","last_change":"2024-06-29T14:22:43.703719+0000","last_active":"2024-06-29T14:22:44.712977+0000","last_peered":"2024-06-29T14:22:44.712977+0000","last_clean":"2024-06-29T14:22:44.712977+0000","last_became_active":"2024-06-29T14:22:43.703240+0000","last_became_peered":"2024-06-29T14:22:43.703240+0000","last_unstale":"2024-06-29T14:22:44.712977+0000","last_undegraded":"2024-06-29T14:22:44.712977+0000","last_fullsized":"2024-06-29T14:22:44.712977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:26.060090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.721860+0000","last_change":"2024-06-29T14:22:44.715784+0000","last_active":"2024-06-29T14:22:44.721860+0000","last_peered":"2024-06-29T14:22:44.721860+0000","last_clean":"2024-06-29T14:22:44.721860+0000","last_became_active":"2024-06-29T14:22:44.715472+0000","last_became_peered":"2024-06-29T14:22:44.715472+0000","last_unstale":"2024-06-29T14:22:44.721860+0000","last_undegraded":"2024-06-29T14:22:44.721860+0000","last_fullsized":"2024-06-29T14:22:44.721860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.704925+0000","last_change":"2024-06-29T14:22:43.700262+0000","last_active":"2024-06-29T14:22:44.704925+0000","last_peered":"2024-06-29T14:22:44.704925+0000","last_clean":"2024-06-29T14:22:44.704925+0000","last_became_active":"2024-06-29T14:22:43.699933+0000","last_became_peered":"2024-06-29T14:22:43.699933+0000","last_unstale":"2024-06-29T14:22:44.704925+0000","last_undegraded":"2024-06-29T14:22:44.704925+0000","last_fullsized":"2024-06-29T14:22:44.704925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:49.306537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.722004+0000","last_change":"2024-06-29T14:22:44.717647+0000","last_active":"2024-06-29T14:22:44.722004+0000","last_peered":"2024-06-29T14:22:44.722004+0000","last_clean":"2024-06-29T14:22:44.722004+0000","last_became_active":"2024-06-29T14:22:44.717352+0000","last_became_peered":"2024-06-29T14:22:44.717352+0000","last_unstale":"2024-06-29T14:22:44.722004+0000","last_undegraded":"2024-06-29T14:22:44.722004+0000","last_fullsized":"2024-06-29T14:22:44.722004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713891+0000","last_change":"2024-06-29T14:22:43.706725+0000","last_active":"2024-06-29T14:22:44.713891+0000","last_peered":"2024-06-29T14:22:44.713891+0000","last_clean":"2024-06-29T14:22:44.713891+0000","last_became_active":"2024-06-29T14:22:43.706166+0000","last_became_peered":"2024-06-29T14:22:43.706166+0000","last_unstale":"2024-06-29T14:22:44.713891+0000","last_undegraded":"2024-06-29T14:22:44.713891+0000","last_fullsized":"2024-06-29T14:22:44.713891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.726499+0000","last_change":"2024-06-29T14:22:44.722510+0000","last_active":"2024-06-29T14:22:44.726499+0000","last_peered":"2024-06-29T14:22:44.726499+0000","last_clean":"2024-06-29T14:22:44.726499+0000","last_became_active":"2024-06-29T14:22:44.722129+0000","last_became_peered":"2024-06-29T14:22:44.722129+0000","last_unstale":"2024-06-29T14:22:44.726499+0000","last_undegraded":"2024-06-29T14:22:44.726499+0000","last_fullsized":"2024-06-29T14:22:44.726499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717864+0000","last_change":"2024-06-29T14:22:43.699816+0000","last_active":"2024-06-29T14:22:44.717864+0000","last_peered":"2024-06-29T14:22:44.717864+0000","last_clean":"2024-06-29T14:22:44.717864+0000","last_became_active":"2024-06-29T14:22:43.699427+0000","last_became_peered":"2024-06-29T14:22:43.699427+0000","last_unstale":"2024-06-29T14:22:44.717864+0000","last_undegraded":"2024-06-29T14:22:44.717864+0000","last_fullsized":"2024-06-29T14:22:44.717864+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711508+0000","last_change":"2024-06-29T14:22:44.707863+0000","last_active":"2024-06-29T14:22:44.711508+0000","last_peered":"2024-06-29T14:22:44.711508+0000","last_clean":"2024-06-29T14:22:44.711508+0000","last_became_active":"2024-06-29T14:22:44.707498+0000","last_became_peered":"2024-06-29T14:22:44.707498+0000","last_unstale":"2024-06-29T14:22:44.711508+0000","last_undegraded":"2024-06-29T14:22:44.711508+0000","last_fullsized":"2024-06-29T14:22:44.711508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:44.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.691671+0000","last_change":"2024-06-29T14:22:43.695216+0000","last_active":"2024-06-29T14:22:44.691671+0000","last_peered":"2024-06-29T14:22:44.691671+0000","last_clean":"2024-06-29T14:22:44.691671+0000","last_became_active":"2024-06-29T14:22:43.694485+0000","last_became_peered":"2024-06-29T14:22:43.694485+0000","last_unstale":"2024-06-29T14:22:44.691671+0000","last_undegraded":"2024-06-29T14:22:44.691671+0000","last_fullsized":"2024-06-29T14:22:44.691671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723964+0000","last_change":"2024-06-29T14:22:44.720432+0000","last_active":"2024-06-29T14:22:44.723964+0000","last_peered":"2024-06-29T14:22:44.723964+0000","last_clean":"2024-06-29T14:22:44.723964+0000","last_became_active":"2024-06-29T14:22:44.720099+0000","last_became_peered":"2024-06-29T14:22:44.720099+0000","last_unstale":"2024-06-29T14:22:44.723964+0000","last_undegraded":"2024-06-29T14:22:44.723964+0000","last_fullsized":"2024-06-29T14:22:44.723964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":108,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.708020+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:22:44.708020+0000","last_peered":"2024-06-29T14:22:44.708020+0000","last_clean":"2024-06-29T14:22:44.708020+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:22:44.708020+0000","last_undegraded":"2024-06-29T14:22:44.708020+0000","last_fullsized":"2024-06-29T14:22:44.708020+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.688749+0000","last_change":"2024-06-29T14:22:43.694439+0000","last_active":"2024-06-29T14:22:44.688749+0000","last_peered":"2024-06-29T14:22:44.688749+0000","last_clean":"2024-06-29T14:22:44.688749+0000","last_became_active":"2024-06-29T14:22:43.694115+0000","last_became_peered":"2024-06-29T14:22:43.694115+0000","last_unstale":"2024-06-29T14:22:44.688749+0000","last_undegraded":"2024-06-29T14:22:44.688749+0000","last_fullsized":"2024-06-29T14:22:44.688749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.716978+0000","last_change":"2024-06-29T14:22:44.707439+0000","last_active":"2024-06-29T14:22:44.716978+0000","last_peered":"2024-06-29T14:22:44.716978+0000","last_clean":"2024-06-29T14:22:44.716978+0000","last_became_active":"2024-06-29T14:22:44.707120+0000","last_became_peered":"2024-06-29T14:22:44.707120+0000","last_unstale":"2024-06-29T14:22:44.716978+0000","last_undegraded":"2024-06-29T14:22:44.716978+0000","last_fullsized":"2024-06-29T14:22:44.716978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:44.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.705756+0000","last_change":"2024-06-29T14:22:43.712228+0000","last_active":"2024-06-29T14:22:44.705756+0000","last_peered":"2024-06-29T14:22:44.705756+0000","last_clean":"2024-06-29T14:22:44.705756+0000","last_became_active":"2024-06-29T14:22:43.711907+0000","last_became_peered":"2024-06-29T14:22:43.711907+0000","last_unstale":"2024-06-29T14:22:44.705756+0000","last_undegraded":"2024-06-29T14:22:44.705756+0000","last_fullsized":"2024-06-29T14:22:44.705756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.708270+0000","last_change":"2024-06-29T14:22:44.705791+0000","last_active":"2024-06-29T14:22:44.708270+0000","last_peered":"2024-06-29T14:22:44.708270+0000","last_clean":"2024-06-29T14:22:44.708270+0000","last_became_active":"2024-06-29T14:22:44.705029+0000","last_became_peered":"2024-06-29T14:22:44.705029+0000","last_unstale":"2024-06-29T14:22:44.708270+0000","last_undegraded":"2024-06-29T14:22:44.708270+0000","last_fullsized":"2024-06-29T14:22:44.708270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:02.776890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.691424+0000","last_change":"2024-06-29T14:22:43.706346+0000","last_active":"2024-06-29T14:22:44.691424+0000","last_peered":"2024-06-29T14:22:44.691424+0000","last_clean":"2024-06-29T14:22:44.691424+0000","last_became_active":"2024-06-29T14:22:43.706016+0000","last_became_peered":"2024-06-29T14:22:43.706016+0000","last_unstale":"2024-06-29T14:22:44.691424+0000","last_undegraded":"2024-06-29T14:22:44.691424+0000","last_fullsized":"2024-06-29T14:22:44.691424+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.694035+0000","last_change":"2024-06-29T14:22:44.692833+0000","last_active":"2024-06-29T14:22:44.694035+0000","last_peered":"2024-06-29T14:22:44.694035+0000","last_clean":"2024-06-29T14:22:44.694035+0000","last_became_active":"2024-06-29T14:22:44.692138+0000","last_became_peered":"2024-06-29T14:22:44.692138+0000","last_unstale":"2024-06-29T14:22:44.694035+0000","last_undegraded":"2024-06-29T14:22:44.694035+0000","last_fullsized":"2024-06-29T14:22:44.694035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.689572+0000","last_change":"2024-06-29T14:22:43.703727+0000","last_active":"2024-06-29T14:22:44.689572+0000","last_peered":"2024-06-29T14:22:44.689572+0000","last_clean":"2024-06-29T14:22:44.689572+0000","last_became_active":"2024-06-29T14:22:43.703413+0000","last_became_peered":"2024-06-29T14:22:43.703413+0000","last_unstale":"2024-06-29T14:22:44.689572+0000","last_undegraded":"2024-06-29T14:22:44.689572+0000","last_fullsized":"2024-06-29T14:22:44.689572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717215+0000","last_change":"2024-06-29T14:22:44.712628+0000","last_active":"2024-06-29T14:22:44.717215+0000","last_peered":"2024-06-29T14:22:44.717215+0000","last_clean":"2024-06-29T14:22:44.717215+0000","last_became_active":"2024-06-29T14:22:44.712293+0000","last_became_peered":"2024-06-29T14:22:44.712293+0000","last_unstale":"2024-06-29T14:22:44.717215+0000","last_undegraded":"2024-06-29T14:22:44.717215+0000","last_fullsized":"2024-06-29T14:22:44.717215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:02.776890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.686643+0000","last_change":"2024-06-29T14:22:43.702999+0000","last_active":"2024-06-29T14:22:44.686643+0000","last_peered":"2024-06-29T14:22:44.686643+0000","last_clean":"2024-06-29T14:22:44.686643+0000","last_became_active":"2024-06-29T14:22:43.702670+0000","last_became_peered":"2024-06-29T14:22:43.702670+0000","last_unstale":"2024-06-29T14:22:44.686643+0000","last_undegraded":"2024-06-29T14:22:44.686643+0000","last_fullsized":"2024-06-29T14:22:44.686643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711471+0000","last_change":"2024-06-29T14:22:44.708301+0000","last_active":"2024-06-29T14:22:44.711471+0000","last_peered":"2024-06-29T14:22:44.711471+0000","last_clean":"2024-06-29T14:22:44.711471+0000","last_became_active":"2024-06-29T14:22:44.707732+0000","last_became_peered":"2024-06-29T14:22:44.707732+0000","last_unstale":"2024-06-29T14:22:44.711471+0000","last_undegraded":"2024-06-29T14:22:44.711471+0000","last_fullsized":"2024-06-29T14:22:44.711471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.712649+0000","last_change":"2024-06-29T14:22:43.711647+0000","last_active":"2024-06-29T14:22:44.712649+0000","last_peered":"2024-06-29T14:22:44.712649+0000","last_clean":"2024-06-29T14:22:44.712649+0000","last_became_active":"2024-06-29T14:22:43.710944+0000","last_became_peered":"2024-06-29T14:22:43.710944+0000","last_unstale":"2024-06-29T14:22:44.712649+0000","last_undegraded":"2024-06-29T14:22:44.712649+0000","last_fullsized":"2024-06-29T14:22:44.712649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:30.671807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.697623+0000","last_change":"2024-06-29T14:22:44.696383+0000","last_active":"2024-06-29T14:22:44.697623+0000","last_peered":"2024-06-29T14:22:44.697623+0000","last_clean":"2024-06-29T14:22:44.697623+0000","last_became_active":"2024-06-29T14:22:44.696071+0000","last_became_peered":"2024-06-29T14:22:44.696071+0000","last_unstale":"2024-06-29T14:22:44.697623+0000","last_undegraded":"2024-06-29T14:22:44.697623+0000","last_fullsized":"2024-06-29T14:22:44.697623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:44.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.687497+0000","last_change":"2024-06-29T14:22:43.703248+0000","last_active":"2024-06-29T14:22:44.687497+0000","last_peered":"2024-06-29T14:22:44.687497+0000","last_clean":"2024-06-29T14:22:44.687497+0000","last_became_active":"2024-06-29T14:22:43.702925+0000","last_became_peered":"2024-06-29T14:22:43.702925+0000","last_unstale":"2024-06-29T14:22:44.687497+0000","last_undegraded":"2024-06-29T14:22:44.687497+0000","last_fullsized":"2024-06-29T14:22:44.687497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:39.003651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.726353+0000","last_change":"2024-06-29T14:22:44.722533+0000","last_active":"2024-06-29T14:22:44.726353+0000","last_peered":"2024-06-29T14:22:44.726353+0000","last_clean":"2024-06-29T14:22:44.726353+0000","last_became_active":"2024-06-29T14:22:44.722296+0000","last_became_peered":"2024-06-29T14:22:44.722296+0000","last_unstale":"2024-06-29T14:22:44.726353+0000","last_undegraded":"2024-06-29T14:22:44.726353+0000","last_fullsized":"2024-06-29T14:22:44.726353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.706124+0000","last_change":"2024-06-29T14:22:43.705703+0000","last_active":"2024-06-29T14:22:44.706124+0000","last_peered":"2024-06-29T14:22:44.706124+0000","last_clean":"2024-06-29T14:22:44.706124+0000","last_became_active":"2024-06-29T14:22:43.705306+0000","last_became_peered":"2024-06-29T14:22:43.705306+0000","last_unstale":"2024-06-29T14:22:44.706124+0000","last_undegraded":"2024-06-29T14:22:44.706124+0000","last_fullsized":"2024-06-29T14:22:44.706124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717337+0000","last_change":"2024-06-29T14:22:44.712109+0000","last_active":"2024-06-29T14:22:44.717337+0000","last_peered":"2024-06-29T14:22:44.717337+0000","last_clean":"2024-06-29T14:22:44.717337+0000","last_became_active":"2024-06-29T14:22:44.711374+0000","last_became_peered":"2024-06-29T14:22:44.711374+0000","last_unstale":"2024-06-29T14:22:44.717337+0000","last_undegraded":"2024-06-29T14:22:44.717337+0000","last_fullsized":"2024-06-29T14:22:44.717337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:40.770948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.712731+0000","last_change":"2024-06-29T14:22:43.711036+0000","last_active":"2024-06-29T14:22:44.712731+0000","last_peered":"2024-06-29T14:22:44.712731+0000","last_clean":"2024-06-29T14:22:44.712731+0000","last_became_active":"2024-06-29T14:22:43.710662+0000","last_became_peered":"2024-06-29T14:22:43.710662+0000","last_unstale":"2024-06-29T14:22:44.712731+0000","last_undegraded":"2024-06-29T14:22:44.712731+0000","last_fullsized":"2024-06-29T14:22:44.712731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:01.984922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.703605+0000","last_change":"2024-06-29T14:22:44.699330+0000","last_active":"2024-06-29T14:22:44.703605+0000","last_peered":"2024-06-29T14:22:44.703605+0000","last_clean":"2024-06-29T14:22:44.703605+0000","last_became_active":"2024-06-29T14:22:44.698611+0000","last_became_peered":"2024-06-29T14:22:44.698611+0000","last_unstale":"2024-06-29T14:22:44.703605+0000","last_undegraded":"2024-06-29T14:22:44.703605+0000","last_fullsized":"2024-06-29T14:22:44.703605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:54.114553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.687489+0000","last_change":"2024-06-29T14:22:43.711354+0000","last_active":"2024-06-29T14:22:44.687489+0000","last_peered":"2024-06-29T14:22:44.687489+0000","last_clean":"2024-06-29T14:22:44.687489+0000","last_became_active":"2024-06-29T14:22:43.710734+0000","last_became_peered":"2024-06-29T14:22:43.710734+0000","last_unstale":"2024-06-29T14:22:44.687489+0000","last_undegraded":"2024-06-29T14:22:44.687489+0000","last_fullsized":"2024-06-29T14:22:44.687489+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713063+0000","last_change":"2024-06-29T14:22:44.694835+0000","last_active":"2024-06-29T14:22:44.713063+0000","last_peered":"2024-06-29T14:22:44.713063+0000","last_clean":"2024-06-29T14:22:44.713063+0000","last_became_active":"2024-06-29T14:22:44.694463+0000","last_became_peered":"2024-06-29T14:22:44.694463+0000","last_unstale":"2024-06-29T14:22:44.713063+0000","last_undegraded":"2024-06-29T14:22:44.713063+0000","last_fullsized":"2024-06-29T14:22:44.713063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:54.114553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.687595+0000","last_change":"2024-06-29T14:22:43.705353+0000","last_active":"2024-06-29T14:22:44.687595+0000","last_peered":"2024-06-29T14:22:44.687595+0000","last_clean":"2024-06-29T14:22:44.687595+0000","last_became_active":"2024-06-29T14:22:43.705002+0000","last_became_peered":"2024-06-29T14:22:43.705002+0000","last_unstale":"2024-06-29T14:22:44.687595+0000","last_undegraded":"2024-06-29T14:22:44.687595+0000","last_fullsized":"2024-06-29T14:22:44.687595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:30.671807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717066+0000","last_change":"2024-06-29T14:22:44.710202+0000","last_active":"2024-06-29T14:22:44.717066+0000","last_peered":"2024-06-29T14:22:44.717066+0000","last_clean":"2024-06-29T14:22:44.717066+0000","last_became_active":"2024-06-29T14:22:44.709900+0000","last_became_peered":"2024-06-29T14:22:44.709900+0000","last_unstale":"2024-06-29T14:22:44.717066+0000","last_undegraded":"2024-06-29T14:22:44.717066+0000","last_fullsized":"2024-06-29T14:22:44.717066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.721905+0000","last_change":"2024-06-29T14:22:43.712161+0000","last_active":"2024-06-29T14:22:44.721905+0000","last_peered":"2024-06-29T14:22:44.721905+0000","last_clean":"2024-06-29T14:22:44.721905+0000","last_became_active":"2024-06-29T14:22:43.710970+0000","last_became_peered":"2024-06-29T14:22:43.710970+0000","last_unstale":"2024-06-29T14:22:44.721905+0000","last_undegraded":"2024-06-29T14:22:44.721905+0000","last_fullsized":"2024-06-29T14:22:44.721905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:43.918058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.706536+0000","last_change":"2024-06-29T14:22:44.702756+0000","last_active":"2024-06-29T14:22:44.706536+0000","last_peered":"2024-06-29T14:22:44.706536+0000","last_clean":"2024-06-29T14:22:44.706536+0000","last_became_active":"2024-06-29T14:22:44.702438+0000","last_became_peered":"2024-06-29T14:22:44.702438+0000","last_unstale":"2024-06-29T14:22:44.706536+0000","last_undegraded":"2024-06-29T14:22:44.706536+0000","last_fullsized":"2024-06-29T14:22:44.706536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.701740+0000","last_change":"2024-06-29T14:22:44.697060+0000","last_active":"2024-06-29T14:22:44.701740+0000","last_peered":"2024-06-29T14:22:44.701740+0000","last_clean":"2024-06-29T14:22:44.701740+0000","last_became_active":"2024-06-29T14:22:44.696660+0000","last_became_peered":"2024-06-29T14:22:44.696660+0000","last_unstale":"2024-06-29T14:22:44.701740+0000","last_undegraded":"2024-06-29T14:22:44.701740+0000","last_fullsized":"2024-06-29T14:22:44.701740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:40.008975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.720423+0000","last_change":"2024-06-29T14:22:43.711693+0000","last_active":"2024-06-29T14:22:44.720423+0000","last_peered":"2024-06-29T14:22:44.720423+0000","last_clean":"2024-06-29T14:22:44.720423+0000","last_became_active":"2024-06-29T14:22:43.710688+0000","last_became_peered":"2024-06-29T14:22:43.710688+0000","last_unstale":"2024-06-29T14:22:44.720423+0000","last_undegraded":"2024-06-29T14:22:44.720423+0000","last_fullsized":"2024-06-29T14:22:44.720423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.701690+0000","last_change":"2024-06-29T14:22:44.697174+0000","last_active":"2024-06-29T14:22:44.701690+0000","last_peered":"2024-06-29T14:22:44.701690+0000","last_clean":"2024-06-29T14:22:44.701690+0000","last_became_active":"2024-06-29T14:22:44.696735+0000","last_became_peered":"2024-06-29T14:22:44.696735+0000","last_unstale":"2024-06-29T14:22:44.701690+0000","last_undegraded":"2024-06-29T14:22:44.701690+0000","last_fullsized":"2024-06-29T14:22:44.701690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.721621+0000","last_change":"2024-06-29T14:22:43.711172+0000","last_active":"2024-06-29T14:22:44.721621+0000","last_peered":"2024-06-29T14:22:44.721621+0000","last_clean":"2024-06-29T14:22:44.721621+0000","last_became_active":"2024-06-29T14:22:43.710779+0000","last_became_peered":"2024-06-29T14:22:43.710779+0000","last_unstale":"2024-06-29T14:22:44.721621+0000","last_undegraded":"2024-06-29T14:22:44.721621+0000","last_fullsized":"2024-06-29T14:22:44.721621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:39.003651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711661+0000","last_change":"2024-06-29T14:22:44.708519+0000","last_active":"2024-06-29T14:22:44.711661+0000","last_peered":"2024-06-29T14:22:44.711661+0000","last_clean":"2024-06-29T14:22:44.711661+0000","last_became_active":"2024-06-29T14:22:44.708212+0000","last_became_peered":"2024-06-29T14:22:44.708212+0000","last_unstale":"2024-06-29T14:22:44.711661+0000","last_undegraded":"2024-06-29T14:22:44.711661+0000","last_fullsized":"2024-06-29T14:22:44.711661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:46.021153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.716918+0000","last_change":"2024-06-29T14:22:43.706244+0000","last_active":"2024-06-29T14:22:44.716918+0000","last_peered":"2024-06-29T14:22:44.716918+0000","last_clean":"2024-06-29T14:22:44.716918+0000","last_became_active":"2024-06-29T14:22:43.705937+0000","last_became_peered":"2024-06-29T14:22:43.705937+0000","last_unstale":"2024-06-29T14:22:44.716918+0000","last_undegraded":"2024-06-29T14:22:44.716918+0000","last_fullsized":"2024-06-29T14:22:44.716918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717024+0000","last_change":"2024-06-29T14:22:44.707925+0000","last_active":"2024-06-29T14:22:44.717024+0000","last_peered":"2024-06-29T14:22:44.717024+0000","last_clean":"2024-06-29T14:22:44.717024+0000","last_became_active":"2024-06-29T14:22:44.707640+0000","last_became_peered":"2024-06-29T14:22:44.707640+0000","last_unstale":"2024-06-29T14:22:44.717024+0000","last_undegraded":"2024-06-29T14:22:44.717024+0000","last_fullsized":"2024-06-29T14:22:44.717024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.716659+0000","last_change":"2024-06-29T14:22:43.711121+0000","last_active":"2024-06-29T14:22:44.716659+0000","last_peered":"2024-06-29T14:22:44.716659+0000","last_clean":"2024-06-29T14:22:44.716659+0000","last_became_active":"2024-06-29T14:22:43.710708+0000","last_became_peered":"2024-06-29T14:22:43.710708+0000","last_unstale":"2024-06-29T14:22:44.716659+0000","last_undegraded":"2024-06-29T14:22:44.716659+0000","last_fullsized":"2024-06-29T14:22:44.716659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:49.306537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":655584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":196608,"data_stored":64724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":64760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":655640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:22:48.727 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:22:48.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.877+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/1291375677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e780d0770 msgr2=0x7f5e780ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.877+0000 7f5e7e0ac640 1 --2- 172.21.15.81:0/1291375677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e780d0770 0x7f5e780ce520 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5e78069610 tx=0x7f5e7002a560 comp rx=0 tx=0).stop 2024-06-29T14:22:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.877+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/1291375677 shutdown_connections 2024-06-29T14:22:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.877+0000 7f5e7e0ac640 1 --2- 172.21.15.81:0/1291375677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e780d0770 0x7f5e780ce520 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.877+0000 7f5e7e0ac640 1 --2- 172.21.15.81:0/1291375677 >> v2:172.21.15.167:3300/0 conn(0x7f5e780cccf0 0x7f5e780d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.877+0000 7f5e7e0ac640 1 --2- 172.21.15.81:0/1291375677 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e780b3200 0x7f5e780cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.877+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/1291375677 >> 172.21.15.81:0/1291375677 conn(0x7f5e780b3680 msgr2=0x7f5e780b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.877+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/1291375677 shutdown_connections 2024-06-29T14:22:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.877+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/1291375677 wait complete. 2024-06-29T14:22:48.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.879+0000 7f5e7e0ac640 1 Processor -- start 2024-06-29T14:22:48.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.879+0000 7f5e7e0ac640 1 -- start start 2024-06-29T14:22:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e7e0ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e780b3200 0x7f5e780cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e7e0ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e780cccf0 0x7f5e780c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e7e0ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e780d0770 0x7f5e780c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e7e0ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5e780cefe0 con 0x7f5e780cccf0 2024-06-29T14:22:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e7e0ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5e780cee60 con 0x7f5e780d0770 2024-06-29T14:22:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e7e0ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5e780cf160 con 0x7f5e780b3200 2024-06-29T14:22:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e76ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e780cccf0 0x7f5e780c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e76ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e780cccf0 0x7f5e780c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48112/0 (socket says 172.21.15.81:48112) 2024-06-29T14:22:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e76ffd640 1 -- 172.21.15.81:0/2588097701 learned_addr learned my addr 172.21.15.81:0/2588097701 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e77fff640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e780d0770 0x7f5e780c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e777fe640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e780b3200 0x7f5e780cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e76ffd640 1 -- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e780b3200 msgr2=0x7f5e780cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e76ffd640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e780b3200 0x7f5e780cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e76ffd640 1 -- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e780d0770 msgr2=0x7f5e780c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:48.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e76ffd640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e780d0770 0x7f5e780c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:48.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e76ffd640 1 -- 172.21.15.81:0/2588097701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e7806e670 con 0x7f5e780cccf0 2024-06-29T14:22:48.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e777fe640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e780b3200 0x7f5e780cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:48.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e76ffd640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e780cccf0 0x7f5e780c98a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00b410 tx=0x7f5e6c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:48.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e5ffff640 1 -- 172.21.15.81:0/2588097701 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c015020 con 0x7f5e780cccf0 2024-06-29T14:22:48.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e5ffff640 1 -- 172.21.15.81:0/2588097701 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e6c011940 con 0x7f5e780cccf0 2024-06-29T14:22:48.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/2588097701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e7806e0f0 con 0x7f5e780cccf0 2024-06-29T14:22:48.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e5ffff640 1 -- 172.21.15.81:0/2588097701 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c011d90 con 0x7f5e780cccf0 2024-06-29T14:22:48.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.881+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/2588097701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e7806ee20 con 0x7f5e780cccf0 2024-06-29T14:22:48.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.882+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/2588097701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e780d0230 con 0x7f5e780cccf0 2024-06-29T14:22:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.883+0000 7f5e5ffff640 1 -- 172.21.15.81:0/2588097701 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5e6c018020 con 0x7f5e780cccf0 2024-06-29T14:22:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.884+0000 7f5e5ffff640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e64076300 0x7f5e64078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.884+0000 7f5e5ffff640 1 -- 172.21.15.81:0/2588097701 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5e6c097630 con 0x7f5e780cccf0 2024-06-29T14:22:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.884+0000 7f5e777fe640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e64076300 0x7f5e64078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.885+0000 7f5e777fe640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e64076300 0x7f5e64078810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5e68005c40 tx=0x7f5e6800a410 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:48.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:48.888+0000 7f5e5ffff640 1 -- 172.21.15.81:0/2588097701 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5e6c01f020 con 0x7f5e780cccf0 2024-06-29T14:22:49.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.065+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/2588097701 --> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5e7815af90 con 0x7f5e64076300 2024-06-29T14:22:49.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.069+0000 7f5e5ffff640 1 -- 172.21.15.81:0/2588097701 <== mgr.4126 v2:172.21.15.167:6832/3101542440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251865 (secure 0 0 0) 0x7f5e7815af90 con 0x7f5e64076300 2024-06-29T14:22:49.073 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:49.075 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:22:49.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e64076300 msgr2=0x7f5e64078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e64076300 0x7f5e64078810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5e68005c40 tx=0x7f5e6800a410 comp rx=0 tx=0).stop 2024-06-29T14:22:49.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/2588097701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e780cccf0 msgr2=0x7f5e780c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e780cccf0 0x7f5e780c98a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00b410 tx=0x7f5e6c00b8d0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/2588097701 shutdown_connections 2024-06-29T14:22:49.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e64076300 0x7f5e64078810 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e780d0770 0x7f5e780c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e780cccf0 0x7f5e780c98a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 --2- 172.21.15.81:0/2588097701 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e780b3200 0x7f5e780cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/2588097701 >> 172.21.15.81:0/2588097701 conn(0x7f5e780b3680 msgr2=0x7f5e780d21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.072+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/2588097701 shutdown_connections 2024-06-29T14:22:49.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.073+0000 7f5e7e0ac640 1 -- 172.21.15.81:0/2588097701 wait complete. 2024-06-29T14:22:49.082 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:22:48.705188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218892,"kb_used_data":2708,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755828,"statfs":{"total":773094113280,"available":772869967872,"internally_reserved":0,"allocated":2772992,"data_stored":1695693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001510"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.726473+0000","last_change":"2024-06-29T14:22:44.722050+0000","last_active":"2024-06-29T14:22:44.726473+0000","last_peered":"2024-06-29T14:22:44.726473+0000","last_clean":"2024-06-29T14:22:44.726473+0000","last_became_active":"2024-06-29T14:22:44.721719+0000","last_became_peered":"2024-06-29T14:22:44.721719+0000","last_unstale":"2024-06-29T14:22:44.726473+0000","last_undegraded":"2024-06-29T14:22:44.726473+0000","last_fullsized":"2024-06-29T14:22:44.726473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:29.350494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.715538+0000","last_change":"2024-06-29T14:22:44.709906+0000","last_active":"2024-06-29T14:22:44.715538+0000","last_peered":"2024-06-29T14:22:44.715538+0000","last_clean":"2024-06-29T14:22:44.715538+0000","last_became_active":"2024-06-29T14:22:44.709406+0000","last_became_peered":"2024-06-29T14:22:44.709406+0000","last_unstale":"2024-06-29T14:22:44.715538+0000","last_undegraded":"2024-06-29T14:22:44.715538+0000","last_fullsized":"2024-06-29T14:22:44.715538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.708302+0000","last_change":"2024-06-29T14:22:44.705357+0000","last_active":"2024-06-29T14:22:44.708302+0000","last_peered":"2024-06-29T14:22:44.708302+0000","last_clean":"2024-06-29T14:22:44.708302+0000","last_became_active":"2024-06-29T14:22:44.704402+0000","last_became_peered":"2024-06-29T14:22:44.704402+0000","last_unstale":"2024-06-29T14:22:44.708302+0000","last_undegraded":"2024-06-29T14:22:44.708302+0000","last_fullsized":"2024-06-29T14:22:44.708302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:27.065414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711532+0000","last_change":"2024-06-29T14:22:44.708801+0000","last_active":"2024-06-29T14:22:44.711532+0000","last_peered":"2024-06-29T14:22:44.711532+0000","last_clean":"2024-06-29T14:22:44.711532+0000","last_became_active":"2024-06-29T14:22:44.708456+0000","last_became_peered":"2024-06-29T14:22:44.708456+0000","last_unstale":"2024-06-29T14:22:44.711532+0000","last_undegraded":"2024-06-29T14:22:44.711532+0000","last_fullsized":"2024-06-29T14:22:44.711532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723935+0000","last_change":"2024-06-29T14:22:44.719917+0000","last_active":"2024-06-29T14:22:44.723935+0000","last_peered":"2024-06-29T14:22:44.723935+0000","last_clean":"2024-06-29T14:22:44.723935+0000","last_became_active":"2024-06-29T14:22:44.719617+0000","last_became_peered":"2024-06-29T14:22:44.719617+0000","last_unstale":"2024-06-29T14:22:44.723935+0000","last_undegraded":"2024-06-29T14:22:44.723935+0000","last_fullsized":"2024-06-29T14:22:44.723935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.707418+0000","last_change":"2024-06-29T14:22:44.701697+0000","last_active":"2024-06-29T14:22:44.707418+0000","last_peered":"2024-06-29T14:22:44.707418+0000","last_clean":"2024-06-29T14:22:44.707418+0000","last_became_active":"2024-06-29T14:22:44.701325+0000","last_became_peered":"2024-06-29T14:22:44.701325+0000","last_unstale":"2024-06-29T14:22:44.707418+0000","last_undegraded":"2024-06-29T14:22:44.707418+0000","last_fullsized":"2024-06-29T14:22:44.707418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723978+0000","last_change":"2024-06-29T14:22:44.720635+0000","last_active":"2024-06-29T14:22:44.723978+0000","last_peered":"2024-06-29T14:22:44.723978+0000","last_clean":"2024-06-29T14:22:44.723978+0000","last_became_active":"2024-06-29T14:22:44.720316+0000","last_became_peered":"2024-06-29T14:22:44.720316+0000","last_unstale":"2024-06-29T14:22:44.723978+0000","last_undegraded":"2024-06-29T14:22:44.723978+0000","last_fullsized":"2024-06-29T14:22:44.723978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.716831+0000","last_change":"2024-06-29T14:22:44.706868+0000","last_active":"2024-06-29T14:22:44.716831+0000","last_peered":"2024-06-29T14:22:44.716831+0000","last_clean":"2024-06-29T14:22:44.716831+0000","last_became_active":"2024-06-29T14:22:44.706298+0000","last_became_peered":"2024-06-29T14:22:44.706298+0000","last_unstale":"2024-06-29T14:22:44.716831+0000","last_undegraded":"2024-06-29T14:22:44.716831+0000","last_fullsized":"2024-06-29T14:22:44.716831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.726541+0000","last_change":"2024-06-29T14:22:44.722834+0000","last_active":"2024-06-29T14:22:44.726541+0000","last_peered":"2024-06-29T14:22:44.726541+0000","last_clean":"2024-06-29T14:22:44.726541+0000","last_became_active":"2024-06-29T14:22:44.722345+0000","last_became_peered":"2024-06-29T14:22:44.722345+0000","last_unstale":"2024-06-29T14:22:44.726541+0000","last_undegraded":"2024-06-29T14:22:44.726541+0000","last_fullsized":"2024-06-29T14:22:44.726541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:25.905572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.706519+0000","last_change":"2024-06-29T14:22:44.702528+0000","last_active":"2024-06-29T14:22:44.706519+0000","last_peered":"2024-06-29T14:22:44.706519+0000","last_clean":"2024-06-29T14:22:44.706519+0000","last_became_active":"2024-06-29T14:22:44.702246+0000","last_became_peered":"2024-06-29T14:22:44.702246+0000","last_unstale":"2024-06-29T14:22:44.706519+0000","last_undegraded":"2024-06-29T14:22:44.706519+0000","last_fullsized":"2024-06-29T14:22:44.706519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.708795+0000","last_change":"2024-06-29T14:22:44.703308+0000","last_active":"2024-06-29T14:22:44.708795+0000","last_peered":"2024-06-29T14:22:44.708795+0000","last_clean":"2024-06-29T14:22:44.708795+0000","last_became_active":"2024-06-29T14:22:44.703018+0000","last_became_peered":"2024-06-29T14:22:44.703018+0000","last_unstale":"2024-06-29T14:22:44.708795+0000","last_undegraded":"2024-06-29T14:22:44.708795+0000","last_fullsized":"2024-06-29T14:22:44.708795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:44.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.705106+0000","last_change":"2024-06-29T14:22:44.702902+0000","last_active":"2024-06-29T14:22:44.705106+0000","last_peered":"2024-06-29T14:22:44.705106+0000","last_clean":"2024-06-29T14:22:44.705106+0000","last_became_active":"2024-06-29T14:22:44.702595+0000","last_became_peered":"2024-06-29T14:22:44.702595+0000","last_unstale":"2024-06-29T14:22:44.705106+0000","last_undegraded":"2024-06-29T14:22:44.705106+0000","last_fullsized":"2024-06-29T14:22:44.705106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:27.065414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713319+0000","last_change":"2024-06-29T14:22:44.703127+0000","last_active":"2024-06-29T14:22:44.713319+0000","last_peered":"2024-06-29T14:22:44.713319+0000","last_clean":"2024-06-29T14:22:44.713319+0000","last_became_active":"2024-06-29T14:22:44.702813+0000","last_became_peered":"2024-06-29T14:22:44.702813+0000","last_unstale":"2024-06-29T14:22:44.713319+0000","last_undegraded":"2024-06-29T14:22:44.713319+0000","last_fullsized":"2024-06-29T14:22:44.713319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:40.770948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.718503+0000","last_change":"2024-06-29T14:22:44.714140+0000","last_active":"2024-06-29T14:22:44.718503+0000","last_peered":"2024-06-29T14:22:44.718503+0000","last_clean":"2024-06-29T14:22:44.718503+0000","last_became_active":"2024-06-29T14:22:44.713863+0000","last_became_peered":"2024-06-29T14:22:44.713863+0000","last_unstale":"2024-06-29T14:22:44.718503+0000","last_undegraded":"2024-06-29T14:22:44.718503+0000","last_fullsized":"2024-06-29T14:22:44.718503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.702367+0000","last_change":"2024-06-29T14:22:44.699851+0000","last_active":"2024-06-29T14:22:44.702367+0000","last_peered":"2024-06-29T14:22:44.702367+0000","last_clean":"2024-06-29T14:22:44.702367+0000","last_became_active":"2024-06-29T14:22:44.697960+0000","last_became_peered":"2024-06-29T14:22:44.697960+0000","last_unstale":"2024-06-29T14:22:44.702367+0000","last_undegraded":"2024-06-29T14:22:44.702367+0000","last_fullsized":"2024-06-29T14:22:44.702367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711405+0000","last_change":"2024-06-29T14:22:44.708069+0000","last_active":"2024-06-29T14:22:44.711405+0000","last_peered":"2024-06-29T14:22:44.711405+0000","last_clean":"2024-06-29T14:22:44.711405+0000","last_became_active":"2024-06-29T14:22:44.707765+0000","last_became_peered":"2024-06-29T14:22:44.707765+0000","last_unstale":"2024-06-29T14:22:44.711405+0000","last_undegraded":"2024-06-29T14:22:44.711405+0000","last_fullsized":"2024-06-29T14:22:44.711405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723894+0000","last_change":"2024-06-29T14:22:44.721271+0000","last_active":"2024-06-29T14:22:44.723894+0000","last_peered":"2024-06-29T14:22:44.723894+0000","last_clean":"2024-06-29T14:22:44.723894+0000","last_became_active":"2024-06-29T14:22:44.720978+0000","last_became_peered":"2024-06-29T14:22:44.720978+0000","last_unstale":"2024-06-29T14:22:44.723894+0000","last_undegraded":"2024-06-29T14:22:44.723894+0000","last_fullsized":"2024-06-29T14:22:44.723894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.705796+0000","last_change":"2024-06-29T14:22:44.702764+0000","last_active":"2024-06-29T14:22:44.705796+0000","last_peered":"2024-06-29T14:22:44.705796+0000","last_clean":"2024-06-29T14:22:44.705796+0000","last_became_active":"2024-06-29T14:22:44.702408+0000","last_became_peered":"2024-06-29T14:22:44.702408+0000","last_unstale":"2024-06-29T14:22:44.705796+0000","last_undegraded":"2024-06-29T14:22:44.705796+0000","last_fullsized":"2024-06-29T14:22:44.705796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:40.770948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.722007+0000","last_change":"2024-06-29T14:22:44.718193+0000","last_active":"2024-06-29T14:22:44.722007+0000","last_peered":"2024-06-29T14:22:44.722007+0000","last_clean":"2024-06-29T14:22:44.722007+0000","last_became_active":"2024-06-29T14:22:44.717900+0000","last_became_peered":"2024-06-29T14:22:44.717900+0000","last_unstale":"2024-06-29T14:22:44.722007+0000","last_undegraded":"2024-06-29T14:22:44.722007+0000","last_fullsized":"2024-06-29T14:22:44.722007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.705676+0000","last_change":"2024-06-29T14:22:44.701520+0000","last_active":"2024-06-29T14:22:44.705676+0000","last_peered":"2024-06-29T14:22:44.705676+0000","last_clean":"2024-06-29T14:22:44.705676+0000","last_became_active":"2024-06-29T14:22:44.701106+0000","last_became_peered":"2024-06-29T14:22:44.701106+0000","last_unstale":"2024-06-29T14:22:44.705676+0000","last_undegraded":"2024-06-29T14:22:44.705676+0000","last_fullsized":"2024-06-29T14:22:44.705676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:40.008975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.720248+0000","last_change":"2024-06-29T14:22:44.717603+0000","last_active":"2024-06-29T14:22:44.720248+0000","last_peered":"2024-06-29T14:22:44.720248+0000","last_clean":"2024-06-29T14:22:44.720248+0000","last_became_active":"2024-06-29T14:22:44.717305+0000","last_became_peered":"2024-06-29T14:22:44.717305+0000","last_unstale":"2024-06-29T14:22:44.720248+0000","last_undegraded":"2024-06-29T14:22:44.720248+0000","last_fullsized":"2024-06-29T14:22:44.720248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:44.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.721712+0000","last_change":"2024-06-29T14:22:44.717305+0000","last_active":"2024-06-29T14:22:44.721712+0000","last_peered":"2024-06-29T14:22:44.721712+0000","last_clean":"2024-06-29T14:22:44.721712+0000","last_became_active":"2024-06-29T14:22:44.716953+0000","last_became_peered":"2024-06-29T14:22:44.716953+0000","last_unstale":"2024-06-29T14:22:44.721712+0000","last_undegraded":"2024-06-29T14:22:44.721712+0000","last_fullsized":"2024-06-29T14:22:44.721712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713365+0000","last_change":"2024-06-29T14:22:44.701947+0000","last_active":"2024-06-29T14:22:44.713365+0000","last_peered":"2024-06-29T14:22:44.713365+0000","last_clean":"2024-06-29T14:22:44.713365+0000","last_became_active":"2024-06-29T14:22:44.701624+0000","last_became_peered":"2024-06-29T14:22:44.701624+0000","last_unstale":"2024-06-29T14:22:44.713365+0000","last_undegraded":"2024-06-29T14:22:44.713365+0000","last_fullsized":"2024-06-29T14:22:44.713365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:10.828903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711433+0000","last_change":"2024-06-29T14:22:44.709275+0000","last_active":"2024-06-29T14:22:44.711433+0000","last_peered":"2024-06-29T14:22:44.711433+0000","last_clean":"2024-06-29T14:22:44.711433+0000","last_became_active":"2024-06-29T14:22:44.708702+0000","last_became_peered":"2024-06-29T14:22:44.708702+0000","last_unstale":"2024-06-29T14:22:44.711433+0000","last_undegraded":"2024-06-29T14:22:44.711433+0000","last_fullsized":"2024-06-29T14:22:44.711433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713308+0000","last_change":"2024-06-29T14:22:44.702814+0000","last_active":"2024-06-29T14:22:44.713308+0000","last_peered":"2024-06-29T14:22:44.713308+0000","last_clean":"2024-06-29T14:22:44.713308+0000","last_became_active":"2024-06-29T14:22:44.702479+0000","last_became_peered":"2024-06-29T14:22:44.702479+0000","last_unstale":"2024-06-29T14:22:44.713308+0000","last_undegraded":"2024-06-29T14:22:44.713308+0000","last_fullsized":"2024-06-29T14:22:44.713308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:46.021153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723587+0000","last_change":"2024-06-29T14:22:44.718581+0000","last_active":"2024-06-29T14:22:44.723587+0000","last_peered":"2024-06-29T14:22:44.723587+0000","last_clean":"2024-06-29T14:22:44.723587+0000","last_became_active":"2024-06-29T14:22:44.718277+0000","last_became_peered":"2024-06-29T14:22:44.718277+0000","last_unstale":"2024-06-29T14:22:44.723587+0000","last_undegraded":"2024-06-29T14:22:44.723587+0000","last_fullsized":"2024-06-29T14:22:44.723587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.710075+0000","last_change":"2024-06-29T14:22:44.704193+0000","last_active":"2024-06-29T14:22:44.710075+0000","last_peered":"2024-06-29T14:22:44.710075+0000","last_clean":"2024-06-29T14:22:44.710075+0000","last_became_active":"2024-06-29T14:22:44.703892+0000","last_became_peered":"2024-06-29T14:22:44.703892+0000","last_unstale":"2024-06-29T14:22:44.710075+0000","last_undegraded":"2024-06-29T14:22:44.710075+0000","last_fullsized":"2024-06-29T14:22:44.710075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713271+0000","last_change":"2024-06-29T14:22:44.703952+0000","last_active":"2024-06-29T14:22:44.713271+0000","last_peered":"2024-06-29T14:22:44.713271+0000","last_clean":"2024-06-29T14:22:44.713271+0000","last_became_active":"2024-06-29T14:22:44.703431+0000","last_became_peered":"2024-06-29T14:22:44.703431+0000","last_unstale":"2024-06-29T14:22:44.713271+0000","last_undegraded":"2024-06-29T14:22:44.713271+0000","last_fullsized":"2024-06-29T14:22:44.713271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:40.008975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.718495+0000","last_change":"2024-06-29T14:22:44.714045+0000","last_active":"2024-06-29T14:22:44.718495+0000","last_peered":"2024-06-29T14:22:44.718495+0000","last_clean":"2024-06-29T14:22:44.718495+0000","last_became_active":"2024-06-29T14:22:44.713749+0000","last_became_peered":"2024-06-29T14:22:44.713749+0000","last_unstale":"2024-06-29T14:22:44.718495+0000","last_undegraded":"2024-06-29T14:22:44.718495+0000","last_fullsized":"2024-06-29T14:22:44.718495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:40.008975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711566+0000","last_change":"2024-06-29T14:22:44.709553+0000","last_active":"2024-06-29T14:22:44.711566+0000","last_peered":"2024-06-29T14:22:44.711566+0000","last_clean":"2024-06-29T14:22:44.711566+0000","last_became_active":"2024-06-29T14:22:44.709153+0000","last_became_peered":"2024-06-29T14:22:44.709153+0000","last_unstale":"2024-06-29T14:22:44.711566+0000","last_undegraded":"2024-06-29T14:22:44.711566+0000","last_fullsized":"2024-06-29T14:22:44.711566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.702909+0000","last_change":"2024-06-29T14:22:44.699167+0000","last_active":"2024-06-29T14:22:44.702909+0000","last_peered":"2024-06-29T14:22:44.702909+0000","last_clean":"2024-06-29T14:22:44.702909+0000","last_became_active":"2024-06-29T14:22:44.698265+0000","last_became_peered":"2024-06-29T14:22:44.698265+0000","last_unstale":"2024-06-29T14:22:44.702909+0000","last_undegraded":"2024-06-29T14:22:44.702909+0000","last_fullsized":"2024-06-29T14:22:44.702909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:46.021153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713809+0000","last_change":"2024-06-29T14:22:44.710341+0000","last_active":"2024-06-29T14:22:44.713809+0000","last_peered":"2024-06-29T14:22:44.713809+0000","last_clean":"2024-06-29T14:22:44.713809+0000","last_became_active":"2024-06-29T14:22:44.709992+0000","last_became_peered":"2024-06-29T14:22:44.709992+0000","last_unstale":"2024-06-29T14:22:44.713809+0000","last_undegraded":"2024-06-29T14:22:44.713809+0000","last_fullsized":"2024-06-29T14:22:44.713809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717603+0000","last_change":"2024-06-29T14:22:44.713512+0000","last_active":"2024-06-29T14:22:44.717603+0000","last_peered":"2024-06-29T14:22:44.717603+0000","last_clean":"2024-06-29T14:22:44.717603+0000","last_became_active":"2024-06-29T14:22:44.713208+0000","last_became_peered":"2024-06-29T14:22:44.713208+0000","last_unstale":"2024-06-29T14:22:44.717603+0000","last_undegraded":"2024-06-29T14:22:44.717603+0000","last_fullsized":"2024-06-29T14:22:44.717603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:16.078185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.719080+0000","last_change":"2024-06-29T14:22:43.711234+0000","last_active":"2024-06-29T14:22:44.719080+0000","last_peered":"2024-06-29T14:22:44.719080+0000","last_clean":"2024-06-29T14:22:44.719080+0000","last_became_active":"2024-06-29T14:22:43.710904+0000","last_became_peered":"2024-06-29T14:22:43.710904+0000","last_unstale":"2024-06-29T14:22:44.719080+0000","last_undegraded":"2024-06-29T14:22:44.719080+0000","last_fullsized":"2024-06-29T14:22:44.719080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.708355+0000","last_change":"2024-06-29T14:22:44.706122+0000","last_active":"2024-06-29T14:22:44.708355+0000","last_peered":"2024-06-29T14:22:44.708355+0000","last_clean":"2024-06-29T14:22:44.708355+0000","last_became_active":"2024-06-29T14:22:44.705746+0000","last_became_peered":"2024-06-29T14:22:44.705746+0000","last_unstale":"2024-06-29T14:22:44.708355+0000","last_undegraded":"2024-06-29T14:22:44.708355+0000","last_fullsized":"2024-06-29T14:22:44.708355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:16.078185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.704301+0000","last_change":"2024-06-29T14:22:43.700941+0000","last_active":"2024-06-29T14:22:44.704301+0000","last_peered":"2024-06-29T14:22:44.704301+0000","last_clean":"2024-06-29T14:22:44.704301+0000","last_became_active":"2024-06-29T14:22:43.700234+0000","last_became_peered":"2024-06-29T14:22:43.700234+0000","last_unstale":"2024-06-29T14:22:44.704301+0000","last_undegraded":"2024-06-29T14:22:44.704301+0000","last_fullsized":"2024-06-29T14:22:44.704301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.701067+0000","last_change":"2024-06-29T14:22:43.707763+0000","last_active":"2024-06-29T14:22:44.701067+0000","last_peered":"2024-06-29T14:22:44.701067+0000","last_clean":"2024-06-29T14:22:44.701067+0000","last_became_active":"2024-06-29T14:22:43.707378+0000","last_became_peered":"2024-06-29T14:22:43.707378+0000","last_unstale":"2024-06-29T14:22:44.701067+0000","last_undegraded":"2024-06-29T14:22:44.701067+0000","last_fullsized":"2024-06-29T14:22:44.701067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:26.060090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713428+0000","last_change":"2024-06-29T14:22:44.705946+0000","last_active":"2024-06-29T14:22:44.713428+0000","last_peered":"2024-06-29T14:22:44.713428+0000","last_clean":"2024-06-29T14:22:44.713428+0000","last_became_active":"2024-06-29T14:22:44.705469+0000","last_became_peered":"2024-06-29T14:22:44.705469+0000","last_unstale":"2024-06-29T14:22:44.713428+0000","last_undegraded":"2024-06-29T14:22:44.713428+0000","last_fullsized":"2024-06-29T14:22:44.713428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.691119+0000","last_change":"2024-06-29T14:22:43.698273+0000","last_active":"2024-06-29T14:22:44.691119+0000","last_peered":"2024-06-29T14:22:44.691119+0000","last_clean":"2024-06-29T14:22:44.691119+0000","last_became_active":"2024-06-29T14:22:43.697950+0000","last_became_peered":"2024-06-29T14:22:43.697950+0000","last_unstale":"2024-06-29T14:22:44.691119+0000","last_undegraded":"2024-06-29T14:22:44.691119+0000","last_fullsized":"2024-06-29T14:22:44.691119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.712749+0000","last_change":"2024-06-29T14:22:44.708264+0000","last_active":"2024-06-29T14:22:44.712749+0000","last_peered":"2024-06-29T14:22:44.712749+0000","last_clean":"2024-06-29T14:22:44.712749+0000","last_became_active":"2024-06-29T14:22:44.707924+0000","last_became_peered":"2024-06-29T14:22:44.707924+0000","last_unstale":"2024-06-29T14:22:44.712749+0000","last_undegraded":"2024-06-29T14:22:44.712749+0000","last_fullsized":"2024-06-29T14:22:44.712749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.692397+0000","last_change":"2024-06-29T14:22:43.706343+0000","last_active":"2024-06-29T14:22:44.692397+0000","last_peered":"2024-06-29T14:22:44.692397+0000","last_clean":"2024-06-29T14:22:44.692397+0000","last_became_active":"2024-06-29T14:22:43.706009+0000","last_became_peered":"2024-06-29T14:22:43.706009+0000","last_unstale":"2024-06-29T14:22:44.692397+0000","last_undegraded":"2024-06-29T14:22:44.692397+0000","last_fullsized":"2024-06-29T14:22:44.692397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.721748+0000","last_change":"2024-06-29T14:22:44.717406+0000","last_active":"2024-06-29T14:22:44.721748+0000","last_peered":"2024-06-29T14:22:44.721748+0000","last_clean":"2024-06-29T14:22:44.721748+0000","last_became_active":"2024-06-29T14:22:44.717095+0000","last_became_peered":"2024-06-29T14:22:44.717095+0000","last_unstale":"2024-06-29T14:22:44.721748+0000","last_undegraded":"2024-06-29T14:22:44.721748+0000","last_fullsized":"2024-06-29T14:22:44.721748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.690530+0000","last_change":"2024-06-29T14:22:43.701234+0000","last_active":"2024-06-29T14:22:44.690530+0000","last_peered":"2024-06-29T14:22:44.690530+0000","last_clean":"2024-06-29T14:22:44.690530+0000","last_became_active":"2024-06-29T14:22:43.700923+0000","last_became_peered":"2024-06-29T14:22:43.700923+0000","last_unstale":"2024-06-29T14:22:44.690530+0000","last_undegraded":"2024-06-29T14:22:44.690530+0000","last_fullsized":"2024-06-29T14:22:44.690530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711635+0000","last_change":"2024-06-29T14:22:44.708529+0000","last_active":"2024-06-29T14:22:44.711635+0000","last_peered":"2024-06-29T14:22:44.711635+0000","last_clean":"2024-06-29T14:22:44.711635+0000","last_became_active":"2024-06-29T14:22:44.708019+0000","last_became_peered":"2024-06-29T14:22:44.708019+0000","last_unstale":"2024-06-29T14:22:44.711635+0000","last_undegraded":"2024-06-29T14:22:44.711635+0000","last_fullsized":"2024-06-29T14:22:44.711635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:10.828903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717881+0000","last_change":"2024-06-29T14:22:43.705026+0000","last_active":"2024-06-29T14:22:44.717881+0000","last_peered":"2024-06-29T14:22:44.717881+0000","last_clean":"2024-06-29T14:22:44.717881+0000","last_became_active":"2024-06-29T14:22:43.704704+0000","last_became_peered":"2024-06-29T14:22:43.704704+0000","last_unstale":"2024-06-29T14:22:44.717881+0000","last_undegraded":"2024-06-29T14:22:44.717881+0000","last_fullsized":"2024-06-29T14:22:44.717881+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.718863+0000","last_change":"2024-06-29T14:22:44.716149+0000","last_active":"2024-06-29T14:22:44.718863+0000","last_peered":"2024-06-29T14:22:44.718863+0000","last_clean":"2024-06-29T14:22:44.718863+0000","last_became_active":"2024-06-29T14:22:44.715832+0000","last_became_peered":"2024-06-29T14:22:44.715832+0000","last_unstale":"2024-06-29T14:22:44.718863+0000","last_undegraded":"2024-06-29T14:22:44.718863+0000","last_fullsized":"2024-06-29T14:22:44.718863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:10.828903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.726574+0000","last_change":"2024-06-29T14:22:43.707443+0000","last_active":"2024-06-29T14:22:44.726574+0000","last_peered":"2024-06-29T14:22:44.726574+0000","last_clean":"2024-06-29T14:22:44.726574+0000","last_became_active":"2024-06-29T14:22:43.707109+0000","last_became_peered":"2024-06-29T14:22:43.707109+0000","last_unstale":"2024-06-29T14:22:44.726574+0000","last_undegraded":"2024-06-29T14:22:44.726574+0000","last_fullsized":"2024-06-29T14:22:44.726574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723833+0000","last_change":"2024-06-29T14:22:44.719076+0000","last_active":"2024-06-29T14:22:44.723833+0000","last_peered":"2024-06-29T14:22:44.723833+0000","last_clean":"2024-06-29T14:22:44.723833+0000","last_became_active":"2024-06-29T14:22:44.718736+0000","last_became_peered":"2024-06-29T14:22:44.718736+0000","last_unstale":"2024-06-29T14:22:44.723833+0000","last_undegraded":"2024-06-29T14:22:44.723833+0000","last_fullsized":"2024-06-29T14:22:44.723833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.709281+0000","last_change":"2024-06-29T14:22:43.700129+0000","last_active":"2024-06-29T14:22:44.709281+0000","last_peered":"2024-06-29T14:22:44.709281+0000","last_clean":"2024-06-29T14:22:44.709281+0000","last_became_active":"2024-06-29T14:22:43.699771+0000","last_became_peered":"2024-06-29T14:22:43.699771+0000","last_unstale":"2024-06-29T14:22:44.709281+0000","last_undegraded":"2024-06-29T14:22:44.709281+0000","last_fullsized":"2024-06-29T14:22:44.709281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:53.109228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.718527+0000","last_change":"2024-06-29T14:22:44.716496+0000","last_active":"2024-06-29T14:22:44.718527+0000","last_peered":"2024-06-29T14:22:44.718527+0000","last_clean":"2024-06-29T14:22:44.718527+0000","last_became_active":"2024-06-29T14:22:44.716244+0000","last_became_peered":"2024-06-29T14:22:44.716244+0000","last_unstale":"2024-06-29T14:22:44.718527+0000","last_undegraded":"2024-06-29T14:22:44.718527+0000","last_fullsized":"2024-06-29T14:22:44.718527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.689538+0000","last_change":"2024-06-29T14:22:43.706155+0000","last_active":"2024-06-29T14:22:44.689538+0000","last_peered":"2024-06-29T14:22:44.689538+0000","last_clean":"2024-06-29T14:22:44.689538+0000","last_became_active":"2024-06-29T14:22:43.705468+0000","last_became_peered":"2024-06-29T14:22:43.705468+0000","last_unstale":"2024-06-29T14:22:44.689538+0000","last_undegraded":"2024-06-29T14:22:44.689538+0000","last_fullsized":"2024-06-29T14:22:44.689538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723909+0000","last_change":"2024-06-29T14:22:44.719377+0000","last_active":"2024-06-29T14:22:44.723909+0000","last_peered":"2024-06-29T14:22:44.723909+0000","last_clean":"2024-06-29T14:22:44.723909+0000","last_became_active":"2024-06-29T14:22:44.719065+0000","last_became_peered":"2024-06-29T14:22:44.719065+0000","last_unstale":"2024-06-29T14:22:44.723909+0000","last_undegraded":"2024-06-29T14:22:44.723909+0000","last_fullsized":"2024-06-29T14:22:44.723909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.692197+0000","last_change":"2024-06-29T14:22:43.698317+0000","last_active":"2024-06-29T14:22:44.692197+0000","last_peered":"2024-06-29T14:22:44.692197+0000","last_clean":"2024-06-29T14:22:44.692197+0000","last_became_active":"2024-06-29T14:22:43.697987+0000","last_became_peered":"2024-06-29T14:22:43.697987+0000","last_unstale":"2024-06-29T14:22:44.692197+0000","last_undegraded":"2024-06-29T14:22:44.692197+0000","last_fullsized":"2024-06-29T14:22:44.692197+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713785+0000","last_change":"2024-06-29T14:22:44.710262+0000","last_active":"2024-06-29T14:22:44.713785+0000","last_peered":"2024-06-29T14:22:44.713785+0000","last_clean":"2024-06-29T14:22:44.713785+0000","last_became_active":"2024-06-29T14:22:44.709948+0000","last_became_peered":"2024-06-29T14:22:44.709948+0000","last_unstale":"2024-06-29T14:22:44.713785+0000","last_undegraded":"2024-06-29T14:22:44.713785+0000","last_fullsized":"2024-06-29T14:22:44.713785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.712977+0000","last_change":"2024-06-29T14:22:43.703719+0000","last_active":"2024-06-29T14:22:44.712977+0000","last_peered":"2024-06-29T14:22:44.712977+0000","last_clean":"2024-06-29T14:22:44.712977+0000","last_became_active":"2024-06-29T14:22:43.703240+0000","last_became_peered":"2024-06-29T14:22:43.703240+0000","last_unstale":"2024-06-29T14:22:44.712977+0000","last_undegraded":"2024-06-29T14:22:44.712977+0000","last_fullsized":"2024-06-29T14:22:44.712977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:26.060090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.721860+0000","last_change":"2024-06-29T14:22:44.715784+0000","last_active":"2024-06-29T14:22:44.721860+0000","last_peered":"2024-06-29T14:22:44.721860+0000","last_clean":"2024-06-29T14:22:44.721860+0000","last_became_active":"2024-06-29T14:22:44.715472+0000","last_became_peered":"2024-06-29T14:22:44.715472+0000","last_unstale":"2024-06-29T14:22:44.721860+0000","last_undegraded":"2024-06-29T14:22:44.721860+0000","last_fullsized":"2024-06-29T14:22:44.721860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.704925+0000","last_change":"2024-06-29T14:22:43.700262+0000","last_active":"2024-06-29T14:22:44.704925+0000","last_peered":"2024-06-29T14:22:44.704925+0000","last_clean":"2024-06-29T14:22:44.704925+0000","last_became_active":"2024-06-29T14:22:43.699933+0000","last_became_peered":"2024-06-29T14:22:43.699933+0000","last_unstale":"2024-06-29T14:22:44.704925+0000","last_undegraded":"2024-06-29T14:22:44.704925+0000","last_fullsized":"2024-06-29T14:22:44.704925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:49.306537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.722004+0000","last_change":"2024-06-29T14:22:44.717647+0000","last_active":"2024-06-29T14:22:44.722004+0000","last_peered":"2024-06-29T14:22:44.722004+0000","last_clean":"2024-06-29T14:22:44.722004+0000","last_became_active":"2024-06-29T14:22:44.717352+0000","last_became_peered":"2024-06-29T14:22:44.717352+0000","last_unstale":"2024-06-29T14:22:44.722004+0000","last_undegraded":"2024-06-29T14:22:44.722004+0000","last_fullsized":"2024-06-29T14:22:44.722004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713891+0000","last_change":"2024-06-29T14:22:43.706725+0000","last_active":"2024-06-29T14:22:44.713891+0000","last_peered":"2024-06-29T14:22:44.713891+0000","last_clean":"2024-06-29T14:22:44.713891+0000","last_became_active":"2024-06-29T14:22:43.706166+0000","last_became_peered":"2024-06-29T14:22:43.706166+0000","last_unstale":"2024-06-29T14:22:44.713891+0000","last_undegraded":"2024-06-29T14:22:44.713891+0000","last_fullsized":"2024-06-29T14:22:44.713891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.726499+0000","last_change":"2024-06-29T14:22:44.722510+0000","last_active":"2024-06-29T14:22:44.726499+0000","last_peered":"2024-06-29T14:22:44.726499+0000","last_clean":"2024-06-29T14:22:44.726499+0000","last_became_active":"2024-06-29T14:22:44.722129+0000","last_became_peered":"2024-06-29T14:22:44.722129+0000","last_unstale":"2024-06-29T14:22:44.726499+0000","last_undegraded":"2024-06-29T14:22:44.726499+0000","last_fullsized":"2024-06-29T14:22:44.726499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717864+0000","last_change":"2024-06-29T14:22:43.699816+0000","last_active":"2024-06-29T14:22:44.717864+0000","last_peered":"2024-06-29T14:22:44.717864+0000","last_clean":"2024-06-29T14:22:44.717864+0000","last_became_active":"2024-06-29T14:22:43.699427+0000","last_became_peered":"2024-06-29T14:22:43.699427+0000","last_unstale":"2024-06-29T14:22:44.717864+0000","last_undegraded":"2024-06-29T14:22:44.717864+0000","last_fullsized":"2024-06-29T14:22:44.717864+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711508+0000","last_change":"2024-06-29T14:22:44.707863+0000","last_active":"2024-06-29T14:22:44.711508+0000","last_peered":"2024-06-29T14:22:44.711508+0000","last_clean":"2024-06-29T14:22:44.711508+0000","last_became_active":"2024-06-29T14:22:44.707498+0000","last_became_peered":"2024-06-29T14:22:44.707498+0000","last_unstale":"2024-06-29T14:22:44.711508+0000","last_undegraded":"2024-06-29T14:22:44.711508+0000","last_fullsized":"2024-06-29T14:22:44.711508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:44.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.691671+0000","last_change":"2024-06-29T14:22:43.695216+0000","last_active":"2024-06-29T14:22:44.691671+0000","last_peered":"2024-06-29T14:22:44.691671+0000","last_clean":"2024-06-29T14:22:44.691671+0000","last_became_active":"2024-06-29T14:22:43.694485+0000","last_became_peered":"2024-06-29T14:22:43.694485+0000","last_unstale":"2024-06-29T14:22:44.691671+0000","last_undegraded":"2024-06-29T14:22:44.691671+0000","last_fullsized":"2024-06-29T14:22:44.691671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.723964+0000","last_change":"2024-06-29T14:22:44.720432+0000","last_active":"2024-06-29T14:22:44.723964+0000","last_peered":"2024-06-29T14:22:44.723964+0000","last_clean":"2024-06-29T14:22:44.723964+0000","last_became_active":"2024-06-29T14:22:44.720099+0000","last_became_peered":"2024-06-29T14:22:44.720099+0000","last_unstale":"2024-06-29T14:22:44.723964+0000","last_undegraded":"2024-06-29T14:22:44.723964+0000","last_fullsized":"2024-06-29T14:22:44.723964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":108,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.708020+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:22:44.708020+0000","last_peered":"2024-06-29T14:22:44.708020+0000","last_clean":"2024-06-29T14:22:44.708020+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:22:44.708020+0000","last_undegraded":"2024-06-29T14:22:44.708020+0000","last_fullsized":"2024-06-29T14:22:44.708020+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.688749+0000","last_change":"2024-06-29T14:22:43.694439+0000","last_active":"2024-06-29T14:22:44.688749+0000","last_peered":"2024-06-29T14:22:44.688749+0000","last_clean":"2024-06-29T14:22:44.688749+0000","last_became_active":"2024-06-29T14:22:43.694115+0000","last_became_peered":"2024-06-29T14:22:43.694115+0000","last_unstale":"2024-06-29T14:22:44.688749+0000","last_undegraded":"2024-06-29T14:22:44.688749+0000","last_fullsized":"2024-06-29T14:22:44.688749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.716978+0000","last_change":"2024-06-29T14:22:44.707439+0000","last_active":"2024-06-29T14:22:44.716978+0000","last_peered":"2024-06-29T14:22:44.716978+0000","last_clean":"2024-06-29T14:22:44.716978+0000","last_became_active":"2024-06-29T14:22:44.707120+0000","last_became_peered":"2024-06-29T14:22:44.707120+0000","last_unstale":"2024-06-29T14:22:44.716978+0000","last_undegraded":"2024-06-29T14:22:44.716978+0000","last_fullsized":"2024-06-29T14:22:44.716978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:44.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.705756+0000","last_change":"2024-06-29T14:22:43.712228+0000","last_active":"2024-06-29T14:22:44.705756+0000","last_peered":"2024-06-29T14:22:44.705756+0000","last_clean":"2024-06-29T14:22:44.705756+0000","last_became_active":"2024-06-29T14:22:43.711907+0000","last_became_peered":"2024-06-29T14:22:43.711907+0000","last_unstale":"2024-06-29T14:22:44.705756+0000","last_undegraded":"2024-06-29T14:22:44.705756+0000","last_fullsized":"2024-06-29T14:22:44.705756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.708270+0000","last_change":"2024-06-29T14:22:44.705791+0000","last_active":"2024-06-29T14:22:44.708270+0000","last_peered":"2024-06-29T14:22:44.708270+0000","last_clean":"2024-06-29T14:22:44.708270+0000","last_became_active":"2024-06-29T14:22:44.705029+0000","last_became_peered":"2024-06-29T14:22:44.705029+0000","last_unstale":"2024-06-29T14:22:44.708270+0000","last_undegraded":"2024-06-29T14:22:44.708270+0000","last_fullsized":"2024-06-29T14:22:44.708270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:02.776890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.691424+0000","last_change":"2024-06-29T14:22:43.706346+0000","last_active":"2024-06-29T14:22:44.691424+0000","last_peered":"2024-06-29T14:22:44.691424+0000","last_clean":"2024-06-29T14:22:44.691424+0000","last_became_active":"2024-06-29T14:22:43.706016+0000","last_became_peered":"2024-06-29T14:22:43.706016+0000","last_unstale":"2024-06-29T14:22:44.691424+0000","last_undegraded":"2024-06-29T14:22:44.691424+0000","last_fullsized":"2024-06-29T14:22:44.691424+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.694035+0000","last_change":"2024-06-29T14:22:44.692833+0000","last_active":"2024-06-29T14:22:44.694035+0000","last_peered":"2024-06-29T14:22:44.694035+0000","last_clean":"2024-06-29T14:22:44.694035+0000","last_became_active":"2024-06-29T14:22:44.692138+0000","last_became_peered":"2024-06-29T14:22:44.692138+0000","last_unstale":"2024-06-29T14:22:44.694035+0000","last_undegraded":"2024-06-29T14:22:44.694035+0000","last_fullsized":"2024-06-29T14:22:44.694035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.689572+0000","last_change":"2024-06-29T14:22:43.703727+0000","last_active":"2024-06-29T14:22:44.689572+0000","last_peered":"2024-06-29T14:22:44.689572+0000","last_clean":"2024-06-29T14:22:44.689572+0000","last_became_active":"2024-06-29T14:22:43.703413+0000","last_became_peered":"2024-06-29T14:22:43.703413+0000","last_unstale":"2024-06-29T14:22:44.689572+0000","last_undegraded":"2024-06-29T14:22:44.689572+0000","last_fullsized":"2024-06-29T14:22:44.689572+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717215+0000","last_change":"2024-06-29T14:22:44.712628+0000","last_active":"2024-06-29T14:22:44.717215+0000","last_peered":"2024-06-29T14:22:44.717215+0000","last_clean":"2024-06-29T14:22:44.717215+0000","last_became_active":"2024-06-29T14:22:44.712293+0000","last_became_peered":"2024-06-29T14:22:44.712293+0000","last_unstale":"2024-06-29T14:22:44.717215+0000","last_undegraded":"2024-06-29T14:22:44.717215+0000","last_fullsized":"2024-06-29T14:22:44.717215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:02.776890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.686643+0000","last_change":"2024-06-29T14:22:43.702999+0000","last_active":"2024-06-29T14:22:44.686643+0000","last_peered":"2024-06-29T14:22:44.686643+0000","last_clean":"2024-06-29T14:22:44.686643+0000","last_became_active":"2024-06-29T14:22:43.702670+0000","last_became_peered":"2024-06-29T14:22:43.702670+0000","last_unstale":"2024-06-29T14:22:44.686643+0000","last_undegraded":"2024-06-29T14:22:44.686643+0000","last_fullsized":"2024-06-29T14:22:44.686643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711471+0000","last_change":"2024-06-29T14:22:44.708301+0000","last_active":"2024-06-29T14:22:44.711471+0000","last_peered":"2024-06-29T14:22:44.711471+0000","last_clean":"2024-06-29T14:22:44.711471+0000","last_became_active":"2024-06-29T14:22:44.707732+0000","last_became_peered":"2024-06-29T14:22:44.707732+0000","last_unstale":"2024-06-29T14:22:44.711471+0000","last_undegraded":"2024-06-29T14:22:44.711471+0000","last_fullsized":"2024-06-29T14:22:44.711471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.712649+0000","last_change":"2024-06-29T14:22:43.711647+0000","last_active":"2024-06-29T14:22:44.712649+0000","last_peered":"2024-06-29T14:22:44.712649+0000","last_clean":"2024-06-29T14:22:44.712649+0000","last_became_active":"2024-06-29T14:22:43.710944+0000","last_became_peered":"2024-06-29T14:22:43.710944+0000","last_unstale":"2024-06-29T14:22:44.712649+0000","last_undegraded":"2024-06-29T14:22:44.712649+0000","last_fullsized":"2024-06-29T14:22:44.712649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:30.671807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.697623+0000","last_change":"2024-06-29T14:22:44.696383+0000","last_active":"2024-06-29T14:22:44.697623+0000","last_peered":"2024-06-29T14:22:44.697623+0000","last_clean":"2024-06-29T14:22:44.697623+0000","last_became_active":"2024-06-29T14:22:44.696071+0000","last_became_peered":"2024-06-29T14:22:44.696071+0000","last_unstale":"2024-06-29T14:22:44.697623+0000","last_undegraded":"2024-06-29T14:22:44.697623+0000","last_fullsized":"2024-06-29T14:22:44.697623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:44.923382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.687497+0000","last_change":"2024-06-29T14:22:43.703248+0000","last_active":"2024-06-29T14:22:44.687497+0000","last_peered":"2024-06-29T14:22:44.687497+0000","last_clean":"2024-06-29T14:22:44.687497+0000","last_became_active":"2024-06-29T14:22:43.702925+0000","last_became_peered":"2024-06-29T14:22:43.702925+0000","last_unstale":"2024-06-29T14:22:44.687497+0000","last_undegraded":"2024-06-29T14:22:44.687497+0000","last_fullsized":"2024-06-29T14:22:44.687497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:39.003651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.726353+0000","last_change":"2024-06-29T14:22:44.722533+0000","last_active":"2024-06-29T14:22:44.726353+0000","last_peered":"2024-06-29T14:22:44.726353+0000","last_clean":"2024-06-29T14:22:44.726353+0000","last_became_active":"2024-06-29T14:22:44.722296+0000","last_became_peered":"2024-06-29T14:22:44.722296+0000","last_unstale":"2024-06-29T14:22:44.726353+0000","last_undegraded":"2024-06-29T14:22:44.726353+0000","last_fullsized":"2024-06-29T14:22:44.726353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.706124+0000","last_change":"2024-06-29T14:22:43.705703+0000","last_active":"2024-06-29T14:22:44.706124+0000","last_peered":"2024-06-29T14:22:44.706124+0000","last_clean":"2024-06-29T14:22:44.706124+0000","last_became_active":"2024-06-29T14:22:43.705306+0000","last_became_peered":"2024-06-29T14:22:43.705306+0000","last_unstale":"2024-06-29T14:22:44.706124+0000","last_undegraded":"2024-06-29T14:22:44.706124+0000","last_fullsized":"2024-06-29T14:22:44.706124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717337+0000","last_change":"2024-06-29T14:22:44.712109+0000","last_active":"2024-06-29T14:22:44.717337+0000","last_peered":"2024-06-29T14:22:44.717337+0000","last_clean":"2024-06-29T14:22:44.717337+0000","last_became_active":"2024-06-29T14:22:44.711374+0000","last_became_peered":"2024-06-29T14:22:44.711374+0000","last_unstale":"2024-06-29T14:22:44.717337+0000","last_undegraded":"2024-06-29T14:22:44.717337+0000","last_fullsized":"2024-06-29T14:22:44.717337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:40.770948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.712731+0000","last_change":"2024-06-29T14:22:43.711036+0000","last_active":"2024-06-29T14:22:44.712731+0000","last_peered":"2024-06-29T14:22:44.712731+0000","last_clean":"2024-06-29T14:22:44.712731+0000","last_became_active":"2024-06-29T14:22:43.710662+0000","last_became_peered":"2024-06-29T14:22:43.710662+0000","last_unstale":"2024-06-29T14:22:44.712731+0000","last_undegraded":"2024-06-29T14:22:44.712731+0000","last_fullsized":"2024-06-29T14:22:44.712731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:01.984922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.703605+0000","last_change":"2024-06-29T14:22:44.699330+0000","last_active":"2024-06-29T14:22:44.703605+0000","last_peered":"2024-06-29T14:22:44.703605+0000","last_clean":"2024-06-29T14:22:44.703605+0000","last_became_active":"2024-06-29T14:22:44.698611+0000","last_became_peered":"2024-06-29T14:22:44.698611+0000","last_unstale":"2024-06-29T14:22:44.703605+0000","last_undegraded":"2024-06-29T14:22:44.703605+0000","last_fullsized":"2024-06-29T14:22:44.703605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:54.114553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.687489+0000","last_change":"2024-06-29T14:22:43.711354+0000","last_active":"2024-06-29T14:22:44.687489+0000","last_peered":"2024-06-29T14:22:44.687489+0000","last_clean":"2024-06-29T14:22:44.687489+0000","last_became_active":"2024-06-29T14:22:43.710734+0000","last_became_peered":"2024-06-29T14:22:43.710734+0000","last_unstale":"2024-06-29T14:22:44.687489+0000","last_undegraded":"2024-06-29T14:22:44.687489+0000","last_fullsized":"2024-06-29T14:22:44.687489+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.713063+0000","last_change":"2024-06-29T14:22:44.694835+0000","last_active":"2024-06-29T14:22:44.713063+0000","last_peered":"2024-06-29T14:22:44.713063+0000","last_clean":"2024-06-29T14:22:44.713063+0000","last_became_active":"2024-06-29T14:22:44.694463+0000","last_became_peered":"2024-06-29T14:22:44.694463+0000","last_unstale":"2024-06-29T14:22:44.713063+0000","last_undegraded":"2024-06-29T14:22:44.713063+0000","last_fullsized":"2024-06-29T14:22:44.713063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:54.114553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.687595+0000","last_change":"2024-06-29T14:22:43.705353+0000","last_active":"2024-06-29T14:22:44.687595+0000","last_peered":"2024-06-29T14:22:44.687595+0000","last_clean":"2024-06-29T14:22:44.687595+0000","last_became_active":"2024-06-29T14:22:43.705002+0000","last_became_peered":"2024-06-29T14:22:43.705002+0000","last_unstale":"2024-06-29T14:22:44.687595+0000","last_undegraded":"2024-06-29T14:22:44.687595+0000","last_fullsized":"2024-06-29T14:22:44.687595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:30.671807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717066+0000","last_change":"2024-06-29T14:22:44.710202+0000","last_active":"2024-06-29T14:22:44.717066+0000","last_peered":"2024-06-29T14:22:44.717066+0000","last_clean":"2024-06-29T14:22:44.717066+0000","last_became_active":"2024-06-29T14:22:44.709900+0000","last_became_peered":"2024-06-29T14:22:44.709900+0000","last_unstale":"2024-06-29T14:22:44.717066+0000","last_undegraded":"2024-06-29T14:22:44.717066+0000","last_fullsized":"2024-06-29T14:22:44.717066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.721905+0000","last_change":"2024-06-29T14:22:43.712161+0000","last_active":"2024-06-29T14:22:44.721905+0000","last_peered":"2024-06-29T14:22:44.721905+0000","last_clean":"2024-06-29T14:22:44.721905+0000","last_became_active":"2024-06-29T14:22:43.710970+0000","last_became_peered":"2024-06-29T14:22:43.710970+0000","last_unstale":"2024-06-29T14:22:44.721905+0000","last_undegraded":"2024-06-29T14:22:44.721905+0000","last_fullsized":"2024-06-29T14:22:44.721905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:43.918058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.706536+0000","last_change":"2024-06-29T14:22:44.702756+0000","last_active":"2024-06-29T14:22:44.706536+0000","last_peered":"2024-06-29T14:22:44.706536+0000","last_clean":"2024-06-29T14:22:44.706536+0000","last_became_active":"2024-06-29T14:22:44.702438+0000","last_became_peered":"2024-06-29T14:22:44.702438+0000","last_unstale":"2024-06-29T14:22:44.706536+0000","last_undegraded":"2024-06-29T14:22:44.706536+0000","last_fullsized":"2024-06-29T14:22:44.706536+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.701740+0000","last_change":"2024-06-29T14:22:44.697060+0000","last_active":"2024-06-29T14:22:44.701740+0000","last_peered":"2024-06-29T14:22:44.701740+0000","last_clean":"2024-06-29T14:22:44.701740+0000","last_became_active":"2024-06-29T14:22:44.696660+0000","last_became_peered":"2024-06-29T14:22:44.696660+0000","last_unstale":"2024-06-29T14:22:44.701740+0000","last_undegraded":"2024-06-29T14:22:44.701740+0000","last_fullsized":"2024-06-29T14:22:44.701740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:40.008975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.720423+0000","last_change":"2024-06-29T14:22:43.711693+0000","last_active":"2024-06-29T14:22:44.720423+0000","last_peered":"2024-06-29T14:22:44.720423+0000","last_clean":"2024-06-29T14:22:44.720423+0000","last_became_active":"2024-06-29T14:22:43.710688+0000","last_became_peered":"2024-06-29T14:22:43.710688+0000","last_unstale":"2024-06-29T14:22:44.720423+0000","last_undegraded":"2024-06-29T14:22:44.720423+0000","last_fullsized":"2024-06-29T14:22:44.720423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.701690+0000","last_change":"2024-06-29T14:22:44.697174+0000","last_active":"2024-06-29T14:22:44.701690+0000","last_peered":"2024-06-29T14:22:44.701690+0000","last_clean":"2024-06-29T14:22:44.701690+0000","last_became_active":"2024-06-29T14:22:44.696735+0000","last_became_peered":"2024-06-29T14:22:44.696735+0000","last_unstale":"2024-06-29T14:22:44.701690+0000","last_undegraded":"2024-06-29T14:22:44.701690+0000","last_fullsized":"2024-06-29T14:22:44.701690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.721621+0000","last_change":"2024-06-29T14:22:43.711172+0000","last_active":"2024-06-29T14:22:44.721621+0000","last_peered":"2024-06-29T14:22:44.721621+0000","last_clean":"2024-06-29T14:22:44.721621+0000","last_became_active":"2024-06-29T14:22:43.710779+0000","last_became_peered":"2024-06-29T14:22:43.710779+0000","last_unstale":"2024-06-29T14:22:44.721621+0000","last_undegraded":"2024-06-29T14:22:44.721621+0000","last_fullsized":"2024-06-29T14:22:44.721621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:39.003651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.711661+0000","last_change":"2024-06-29T14:22:44.708519+0000","last_active":"2024-06-29T14:22:44.711661+0000","last_peered":"2024-06-29T14:22:44.711661+0000","last_clean":"2024-06-29T14:22:44.711661+0000","last_became_active":"2024-06-29T14:22:44.708212+0000","last_became_peered":"2024-06-29T14:22:44.708212+0000","last_unstale":"2024-06-29T14:22:44.711661+0000","last_undegraded":"2024-06-29T14:22:44.711661+0000","last_fullsized":"2024-06-29T14:22:44.711661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:46.021153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.716918+0000","last_change":"2024-06-29T14:22:43.706244+0000","last_active":"2024-06-29T14:22:44.716918+0000","last_peered":"2024-06-29T14:22:44.716918+0000","last_clean":"2024-06-29T14:22:44.716918+0000","last_became_active":"2024-06-29T14:22:43.705937+0000","last_became_peered":"2024-06-29T14:22:43.705937+0000","last_unstale":"2024-06-29T14:22:44.716918+0000","last_undegraded":"2024-06-29T14:22:44.716918+0000","last_fullsized":"2024-06-29T14:22:44.716918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.717024+0000","last_change":"2024-06-29T14:22:44.707925+0000","last_active":"2024-06-29T14:22:44.717024+0000","last_peered":"2024-06-29T14:22:44.717024+0000","last_clean":"2024-06-29T14:22:44.717024+0000","last_became_active":"2024-06-29T14:22:44.707640+0000","last_became_peered":"2024-06-29T14:22:44.707640+0000","last_unstale":"2024-06-29T14:22:44.717024+0000","last_undegraded":"2024-06-29T14:22:44.717024+0000","last_fullsized":"2024-06-29T14:22:44.717024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-06-29T14:22:44.716659+0000","last_change":"2024-06-29T14:22:43.711121+0000","last_active":"2024-06-29T14:22:44.716659+0000","last_peered":"2024-06-29T14:22:44.716659+0000","last_clean":"2024-06-29T14:22:44.716659+0000","last_became_active":"2024-06-29T14:22:43.710708+0000","last_became_peered":"2024-06-29T14:22:43.710708+0000","last_unstale":"2024-06-29T14:22:44.716659+0000","last_undegraded":"2024-06-29T14:22:44.716659+0000","last_fullsized":"2024-06-29T14:22:44.716659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:49.306537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":655584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":196608,"data_stored":64724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":64760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":655640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:22:49.086 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:22:49.087 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-06-29T14:22:49.087 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-06-29T14:22:49.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.238+0000 7faecb83d640 1 -- 172.21.15.81:0/4103954547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeac003a30 msgr2=0x7faeac003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.238+0000 7faecb83d640 1 --2- 172.21.15.81:0/4103954547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeac003a30 0x7faeac003ea0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7faebc001630 tx=0x7faeac005830 comp rx=0 tx=0).stop 2024-06-29T14:22:49.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.238+0000 7faec9dda640 1 -- 172.21.15.81:0/4103954547 reap_dead start 2024-06-29T14:22:49.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.239+0000 7faecb83d640 1 -- 172.21.15.81:0/4103954547 shutdown_connections 2024-06-29T14:22:49.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.239+0000 7faecb83d640 1 -- 172.21.15.81:0/4103954547 >> 172.21.15.81:0/4103954547 conn(0x7faec40d1960 msgr2=0x7faec40d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.239+0000 7faecb83d640 1 -- 172.21.15.81:0/4103954547 shutdown_connections 2024-06-29T14:22:49.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.239+0000 7faecb83d640 1 -- 172.21.15.81:0/4103954547 wait complete. 2024-06-29T14:22:49.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.240+0000 7faecb83d640 1 Processor -- start 2024-06-29T14:22:49.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.240+0000 7faecb83d640 1 -- start start 2024-06-29T14:22:49.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faecb83d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeac003a30 0x7faec40ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faecb83d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec40ae760 0x7faec40aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faecb83d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec40af0b0 0x7faec41d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faecb83d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faec40d0500 con 0x7faec40af0b0 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faecb83d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faec40b2790 con 0x7faeac003a30 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faecb83d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faec40d0680 con 0x7faec40ae760 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec9dda640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec40af0b0 0x7faec41d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec9dda640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec40af0b0 0x7faec41d3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48120/0 (socket says 172.21.15.81:48120) 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec9dda640 1 -- 172.21.15.81:0/3586343826 learned_addr learned my addr 172.21.15.81:0/3586343826 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec95d9640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeac003a30 0x7faec40ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec8dd8640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec40ae760 0x7faec40aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec9dda640 1 -- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec40ae760 msgr2=0x7faec40aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec9dda640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec40ae760 0x7faec40aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec9dda640 1 -- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeac003a30 msgr2=0x7faec40ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec9dda640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeac003a30 0x7faec40ca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec9dda640 1 -- 172.21.15.81:0/3586343826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec40cb1c0 con 0x7faec40af0b0 2024-06-29T14:22:49.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec9dda640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec40af0b0 0x7faec41d3d30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7faeb4005fd0 tx=0x7faeb4010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec1ffb640 1 -- 172.21.15.81:0/3586343826 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4003970 con 0x7faec40af0b0 2024-06-29T14:22:49.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec1ffb640 1 -- 172.21.15.81:0/3586343826 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faeb4015070 con 0x7faec40af0b0 2024-06-29T14:22:49.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faec1ffb640 1 -- 172.21.15.81:0/3586343826 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4004030 con 0x7faec40af0b0 2024-06-29T14:22:49.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.242+0000 7faecb83d640 1 -- 172.21.15.81:0/3586343826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faec40cb4a0 con 0x7faec40af0b0 2024-06-29T14:22:49.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.243+0000 7faecb83d640 1 -- 172.21.15.81:0/3586343826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faec40cba60 con 0x7faec40af0b0 2024-06-29T14:22:49.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.244+0000 7faecb83d640 1 -- 172.21.15.81:0/3586343826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faec40caf30 con 0x7faec40af0b0 2024-06-29T14:22:49.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.244+0000 7faec1ffb640 1 -- 172.21.15.81:0/3586343826 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faeb40041d0 con 0x7faec40af0b0 2024-06-29T14:22:49.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.245+0000 7faec1ffb640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faeb00762d0 0x7faeb00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.245+0000 7faec1ffb640 1 -- 172.21.15.81:0/3586343826 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7faeb4098210 con 0x7faec40af0b0 2024-06-29T14:22:49.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.245+0000 7faec95d9640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faeb00762d0 0x7faeb00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.245+0000 7faec95d9640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faeb00762d0 0x7faeb00787e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faebc032000 tx=0x7faebc02c040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.249+0000 7faec1ffb640 1 -- 172.21.15.81:0/3586343826 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faeb4061520 con 0x7faec40af0b0 2024-06-29T14:22:49.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.431+0000 7faecb83d640 1 -- 172.21.15.81:0/3586343826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7faec40cbc60 con 0x7faec40af0b0 2024-06-29T14:22:49.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.431+0000 7faec1ffb640 1 -- 172.21.15.81:0/3586343826 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v27) ==== 86+0+160 (secure 0 0 0) 0x7faec40cbc60 con 0x7faec40af0b0 2024-06-29T14:22:49.431 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:49.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.433+0000 7faecb83d640 1 -- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faeb00762d0 msgr2=0x7faeb00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.433+0000 7faecb83d640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faeb00762d0 0x7faeb00787e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faebc032000 tx=0x7faebc02c040 comp rx=0 tx=0).stop 2024-06-29T14:22:49.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.433+0000 7faecb83d640 1 -- 172.21.15.81:0/3586343826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec40af0b0 msgr2=0x7faec41d3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.433+0000 7faecb83d640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec40af0b0 0x7faec41d3d30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7faeb4005fd0 tx=0x7faeb4010f40 comp rx=0 tx=0).stop 2024-06-29T14:22:49.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.433+0000 7faecb83d640 1 -- 172.21.15.81:0/3586343826 shutdown_connections 2024-06-29T14:22:49.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.433+0000 7faecb83d640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7faeb00762d0 0x7faeb00787e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.433+0000 7faecb83d640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec40af0b0 0x7faec41d3d30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.433+0000 7faecb83d640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec40ae760 0x7faec40aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.433+0000 7faecb83d640 1 --2- 172.21.15.81:0/3586343826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeac003a30 0x7faec40ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.433+0000 7faecb83d640 1 -- 172.21.15.81:0/3586343826 >> 172.21.15.81:0/3586343826 conn(0x7faec40d1960 msgr2=0x7faec41e4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.434+0000 7faecb83d640 1 -- 172.21.15.81:0/3586343826 shutdown_connections 2024-06-29T14:22:49.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.434+0000 7faecb83d640 1 -- 172.21.15.81:0/3586343826 wait complete. 2024-06-29T14:22:49.444 INFO:teuthology.orchestra.run.smithi081.stdout:[{"version":27,"timestamp":"2024-06-29T14:22:29.520059+0000","name":"","changes":[{"name":"osd.0/osd_mclock_max_capacity_iops_ssd","new_value":"4995.347675"}]}] 2024-06-29T14:22:49.444 INFO:tasks.ceph_manager:config epoch is 27 2024-06-29T14:22:49.444 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-29T14:22:49.444 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-06-29T14:22:49.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-06-29T14:22:49.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.558+0000 7fdd046a4640 1 -- 172.21.15.81:0/3318629084 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdcfc0cd540 msgr2=0x7fdcfc0b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.558+0000 7fdd046a4640 1 --2- 172.21.15.81:0/3318629084 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdcfc0cd540 0x7fdcfc0b47b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdcec009860 tx=0x7fdcec02a530 comp rx=0 tx=0).stop 2024-06-29T14:22:49.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.558+0000 7fdd046a4640 1 -- 172.21.15.81:0/3318629084 shutdown_connections 2024-06-29T14:22:49.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.558+0000 7fdd046a4640 1 --2- 172.21.15.81:0/3318629084 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdcfc0cd540 0x7fdcfc0b47b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.558+0000 7fdd046a4640 1 -- 172.21.15.81:0/3318629084 >> 172.21.15.81:0/3318629084 conn(0x7fdcfc0d1960 msgr2=0x7fdcfc0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.558+0000 7fdd046a4640 1 -- 172.21.15.81:0/3318629084 shutdown_connections 2024-06-29T14:22:49.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.558+0000 7fdd046a4640 1 -- 172.21.15.81:0/3318629084 wait complete. 2024-06-29T14:22:49.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.560+0000 7fdd046a4640 1 Processor -- start 2024-06-29T14:22:49.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.560+0000 7fdd046a4640 1 -- start start 2024-06-29T14:22:49.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd046a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcfc0b4cf0 0x7fdcfc159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd046a4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdcfc0b6d60 0x7fdcfc1576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd046a4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcfc0cd540 0x7fdcfc1554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd046a4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc0d0980 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd046a4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc0d0800 con 0x7fdcfc0cd540 2024-06-29T14:22:49.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd046a4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdcfc167640 con 0x7fdcfc0b6d60 2024-06-29T14:22:49.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd02440640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcfc0b4cf0 0x7fdcfc159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd02440640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcfc0b4cf0 0x7fdcfc159190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48126/0 (socket says 172.21.15.81:48126) 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd02440640 1 -- 172.21.15.81:0/1563359634 learned_addr learned my addr 172.21.15.81:0/1563359634 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd01c3f640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdcfc0b6d60 0x7fdcfc1576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd02c41640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcfc0cd540 0x7fdcfc1554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd02440640 1 -- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdcfc0b6d60 msgr2=0x7fdcfc1576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd02440640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdcfc0b6d60 0x7fdcfc1576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd02440640 1 -- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcfc0cd540 msgr2=0x7fdcfc1554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd02440640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcfc0cd540 0x7fdcfc1554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd02440640 1 -- 172.21.15.81:0/1563359634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcfc1589d0 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.561+0000 7fdd01c3f640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdcfc0b6d60 0x7fdcfc1576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.562+0000 7fdd02440640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcfc0b4cf0 0x7fdcfc159190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800c3f0 tx=0x7fdcf8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.562+0000 7fdcf2ffd640 1 -- 172.21.15.81:0/1563359634 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8017070 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.562+0000 7fdcf2ffd640 1 -- 172.21.15.81:0/1563359634 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdcf80053e0 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.562+0000 7fdcf2ffd640 1 -- 172.21.15.81:0/1563359634 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8003850 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.562+0000 7fdd046a4640 1 -- 172.21.15.81:0/1563359634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcfc159fb0 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.562+0000 7fdd046a4640 1 -- 172.21.15.81:0/1563359634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcfc1596d0 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.563+0000 7fdd046a4640 1 -- 172.21.15.81:0/1563359634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcfc0b47b0 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.564+0000 7fdcf2ffd640 1 -- 172.21.15.81:0/1563359634 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdcf801a070 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.564+0000 7fdcf2ffd640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdce0076300 0x7fdce0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.564+0000 7fdcf2ffd640 1 -- 172.21.15.81:0/1563359634 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fdcf8096b70 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.564+0000 7fdd01c3f640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdce0076300 0x7fdce0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.565+0000 7fdd01c3f640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdce0076300 0x7fdce0078810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdcec034000 tx=0x7fdcec004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.568+0000 7fdcf2ffd640 1 -- 172.21.15.81:0/1563359634 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdcf8060590 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.797+0000 7fdd046a4640 1 -- 172.21.15.81:0/1563359634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fdcfc15a4d0 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.800+0000 7fdcf2ffd640 1 -- 172.21.15.81:0/1563359634 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+189528 (secure 0 0 0) 0x7fdcf805fe90 con 0x7fdcfc0b4cf0 2024-06-29T14:22:49.801 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.802+0000 7fdd046a4640 1 -- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdce0076300 msgr2=0x7fdce0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.802+0000 7fdd046a4640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdce0076300 0x7fdce0078810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdcec034000 tx=0x7fdcec004730 comp rx=0 tx=0).stop 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.802+0000 7fdd046a4640 1 -- 172.21.15.81:0/1563359634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcfc0b4cf0 msgr2=0x7fdcfc159190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.802+0000 7fdd046a4640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcfc0b4cf0 0x7fdcfc159190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800c3f0 tx=0x7fdcf8001e70 comp rx=0 tx=0).stop 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.802+0000 7fdd046a4640 1 -- 172.21.15.81:0/1563359634 shutdown_connections 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.802+0000 7fdd046a4640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdce0076300 0x7fdce0078810 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.802+0000 7fdd046a4640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdcfc0cd540 0x7fdcfc1554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.802+0000 7fdd046a4640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdcfc0b6d60 0x7fdcfc1576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.802+0000 7fdd046a4640 1 --2- 172.21.15.81:0/1563359634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcfc0b4cf0 0x7fdcfc159190 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.803+0000 7fdd046a4640 1 -- 172.21.15.81:0/1563359634 >> 172.21.15.81:0/1563359634 conn(0x7fdcfc0d1960 msgr2=0x7fdcfc0700f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.803+0000 7fdd046a4640 1 -- 172.21.15.81:0/1563359634 shutdown_connections 2024-06-29T14:22:49.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.803+0000 7fdd046a4640 1 -- 172.21.15.81:0/1563359634 wait complete. 2024-06-29T14:22:49.813 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":5,"flags":0,"active_gid":4126,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":3101542440},{"type":"v1","addr":"172.21.15.167:6833","nonce":3101542440}]},"active_addr":"172.21.15.167:6833/3101542440","active_change":"2024-06-29T14:22:22.692029+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4155,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3134616648}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":2999883788}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3793734276}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3819768847}]}]} 2024-06-29T14:22:49.816 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-06-29T14:22:49.817 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-06-29T14:22:49.817 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:49.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.932+0000 7f84e6612640 1 -- 172.21.15.81:0/4010656969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84cc004d40 msgr2=0x7f84cc005170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.932+0000 7f84e6612640 1 --2- 172.21.15.81:0/4010656969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84cc004d40 0x7f84cc005170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f84d8001e40 tx=0x7f84cc005830 comp rx=0 tx=0).stop 2024-06-29T14:22:49.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.932+0000 7f84e4baf640 1 -- 172.21.15.81:0/4010656969 reap_dead start 2024-06-29T14:22:49.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.932+0000 7f84e6612640 1 -- 172.21.15.81:0/4010656969 shutdown_connections 2024-06-29T14:22:49.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.932+0000 7f84e6612640 1 -- 172.21.15.81:0/4010656969 >> 172.21.15.81:0/4010656969 conn(0x7f84e00d0930 msgr2=0x7f84e00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:49.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.932+0000 7f84e6612640 1 -- 172.21.15.81:0/4010656969 shutdown_connections 2024-06-29T14:22:49.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.932+0000 7f84e6612640 1 -- 172.21.15.81:0/4010656969 wait complete. 2024-06-29T14:22:49.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.933+0000 7f84e6612640 1 Processor -- start 2024-06-29T14:22:49.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.933+0000 7f84e6612640 1 -- start start 2024-06-29T14:22:49.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84e6612640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84cc004d40 0x7f84e014cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84e6612640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84e014f5d0 0x7f84e00c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84e6612640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84e00cb8a0 0x7f84e006c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84e6612640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f84e015b260 con 0x7f84e014f5d0 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84e6612640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f84e015b0e0 con 0x7f84cc004d40 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84e6612640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f84e015b3e0 con 0x7f84e00cb8a0 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84d7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84e014f5d0 0x7f84e00c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84d7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84e014f5d0 0x7f84e00c61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48142/0 (socket says 172.21.15.81:48142) 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84d7fff640 1 -- 172.21.15.81:0/2641743796 learned_addr learned my addr 172.21.15.81:0/2641743796 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84dffff640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84cc004d40 0x7f84e014cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84d7fff640 1 -- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84e00cb8a0 msgr2=0x7f84e006c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84d7fff640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84e00cb8a0 0x7f84e006c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84d7fff640 1 -- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84cc004d40 msgr2=0x7f84e014cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84d7fff640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84cc004d40 0x7f84e014cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84d7fff640 1 -- 172.21.15.81:0/2641743796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e014d250 con 0x7f84e014f5d0 2024-06-29T14:22:49.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84d7fff640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84e014f5d0 0x7f84e00c61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f84c8001b40 tx=0x7f84c800ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84dd7fa640 1 -- 172.21.15.81:0/2641743796 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c8015070 con 0x7f84e014f5d0 2024-06-29T14:22:49.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84dd7fa640 1 -- 172.21.15.81:0/2641743796 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84c8010980 con 0x7f84e014f5d0 2024-06-29T14:22:49.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84e6612640 1 -- 172.21.15.81:0/2641743796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e014ff90 con 0x7f84e014f5d0 2024-06-29T14:22:49.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84dd7fa640 1 -- 172.21.15.81:0/2641743796 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c8010d30 con 0x7f84e014f5d0 2024-06-29T14:22:49.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.935+0000 7f84e6612640 1 -- 172.21.15.81:0/2641743796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e00c5a00 con 0x7f84e014f5d0 2024-06-29T14:22:49.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.936+0000 7f84e6612640 1 -- 172.21.15.81:0/2641743796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84e006e2a0 con 0x7f84e014f5d0 2024-06-29T14:22:49.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.937+0000 7f84dd7fa640 1 -- 172.21.15.81:0/2641743796 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f84c8018070 con 0x7f84e014f5d0 2024-06-29T14:22:49.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.938+0000 7f84dd7fa640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f84c0076300 0x7f84c0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:49.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.938+0000 7f84dd7fa640 1 -- 172.21.15.81:0/2641743796 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f84c8097ad0 con 0x7f84e014f5d0 2024-06-29T14:22:49.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.938+0000 7f84dffff640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f84c0076300 0x7f84c0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:49.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.938+0000 7f84dffff640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f84c0076300 0x7f84c0078810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009a10 tx=0x7f84d0005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:49.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:49.942+0000 7f84dd7fa640 1 -- 172.21.15.81:0/2641743796 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f84e006e2a0 con 0x7f84e014f5d0 2024-06-29T14:22:50.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.124+0000 7f84e6612640 1 -- 172.21.15.81:0/2641743796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84e00b4890 con 0x7f84e014f5d0 2024-06-29T14:22:50.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.125+0000 7f84dd7fa640 1 -- 172.21.15.81:0/2641743796 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f84c8061430 con 0x7f84e014f5d0 2024-06-29T14:22:50.125 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:50.125 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 -- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f84c0076300 msgr2=0x7f84c0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f84c0076300 0x7f84c0078810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009a10 tx=0x7f84d0005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 -- 172.21.15.81:0/2641743796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84e014f5d0 msgr2=0x7f84e00c61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84e014f5d0 0x7f84e00c61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f84c8001b40 tx=0x7f84c800ea10 comp rx=0 tx=0).stop 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 -- 172.21.15.81:0/2641743796 shutdown_connections 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f84c0076300 0x7f84c0078810 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84e00cb8a0 0x7f84e006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84e014f5d0 0x7f84e00c61e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 --2- 172.21.15.81:0/2641743796 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84cc004d40 0x7f84e014cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 -- 172.21.15.81:0/2641743796 >> 172.21.15.81:0/2641743796 conn(0x7f84e00d0930 msgr2=0x7f84e00cf580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 -- 172.21.15.81:0/2641743796 shutdown_connections 2024-06-29T14:22:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.127+0000 7f84e6612640 1 -- 172.21.15.81:0/2641743796 wait complete. 2024-06-29T14:22:50.138 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-06-29T14:22:50.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:50.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.251+0000 7f107a9aa640 1 -- 172.21.15.81:0/976836045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f107415ae90 msgr2=0x7f10740c5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.251+0000 7f107a9aa640 1 --2- 172.21.15.81:0/976836045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f107415ae90 0x7f10740c5160 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f107415b2a0 tx=0x7f106402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.251+0000 7f107a9aa640 1 -- 172.21.15.81:0/976836045 shutdown_connections 2024-06-29T14:22:50.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.251+0000 7f107a9aa640 1 --2- 172.21.15.81:0/976836045 >> v2:172.21.15.167:3300/0 conn(0x7f10740cf860 0x7f10740c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.251+0000 7f107a9aa640 1 --2- 172.21.15.81:0/976836045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f107415ae90 0x7f10740c5160 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.251+0000 7f107a9aa640 1 -- 172.21.15.81:0/976836045 >> 172.21.15.81:0/976836045 conn(0x7f10740d00b0 msgr2=0x7f10740d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.251+0000 7f107a9aa640 1 -- 172.21.15.81:0/976836045 shutdown_connections 2024-06-29T14:22:50.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.251+0000 7f107a9aa640 1 -- 172.21.15.81:0/976836045 wait complete. 2024-06-29T14:22:50.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.253+0000 7f107a9aa640 1 Processor -- start 2024-06-29T14:22:50.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.253+0000 7f107a9aa640 1 -- start start 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.254+0000 7f107a9aa640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10740ca700 0x7f10741499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.254+0000 7f107a9aa640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10740cf860 0x7f107414c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.254+0000 7f107a9aa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f107415ae90 0x7f10741f6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.254+0000 7f107a9aa640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f107406d300 con 0x7f107415ae90 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.254+0000 7f107a9aa640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f10740cb5b0 con 0x7f10740ca700 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.254+0000 7f107a9aa640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f107406d480 con 0x7f10740cf860 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1078f47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f107415ae90 0x7f10741f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1078f47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f107415ae90 0x7f10741f6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35920/0 (socket says 172.21.15.81:35920) 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1078f47640 1 -- 172.21.15.81:0/2105210535 learned_addr learned my addr 172.21.15.81:0/2105210535 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:50.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f10737fe640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10740cf860 0x7f107414c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1073fff640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10740ca700 0x7f10741499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1078f47640 1 -- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10740cf860 msgr2=0x7f107414c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1078f47640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10740cf860 0x7f107414c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1078f47640 1 -- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10740ca700 msgr2=0x7f10741499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1078f47640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10740ca700 0x7f10741499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1078f47640 1 -- 172.21.15.81:0/2105210535 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107414c750 con 0x7f107415ae90 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1078f47640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f107415ae90 0x7f10741f6810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f106c00c730 tx=0x7f106c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1070ff9640 1 -- 172.21.15.81:0/2105210535 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106c017070 con 0x7f107415ae90 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1070ff9640 1 -- 172.21.15.81:0/2105210535 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f106c003c60 con 0x7f107415ae90 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f107a9aa640 1 -- 172.21.15.81:0/2105210535 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f107414cbd0 con 0x7f107415ae90 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f1070ff9640 1 -- 172.21.15.81:0/2105210535 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f106c0052d0 con 0x7f107415ae90 2024-06-29T14:22:50.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.255+0000 7f107a9aa640 1 -- 172.21.15.81:0/2105210535 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10741f6010 con 0x7f107415ae90 2024-06-29T14:22:50.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.256+0000 7f107a9aa640 1 -- 172.21.15.81:0/2105210535 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f107415e490 con 0x7f107415ae90 2024-06-29T14:22:50.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.257+0000 7f1070ff9640 1 -- 172.21.15.81:0/2105210535 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f106c01a070 con 0x7f107415ae90 2024-06-29T14:22:50.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.257+0000 7f1070ff9640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f10600762d0 0x7f10600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.257+0000 7f1070ff9640 1 -- 172.21.15.81:0/2105210535 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f106c096fa0 con 0x7f107415ae90 2024-06-29T14:22:50.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.258+0000 7f1073fff640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f10600762d0 0x7f10600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.258+0000 7f1073fff640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f10600762d0 0x7f10600787e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1064034000 tx=0x7f1064004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.262+0000 7f1070ff9640 1 -- 172.21.15.81:0/2105210535 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f107415e490 con 0x7f107415ae90 2024-06-29T14:22:50.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.443+0000 7f107a9aa640 1 -- 172.21.15.81:0/2105210535 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f107406da10 con 0x7f107415ae90 2024-06-29T14:22:50.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.444+0000 7f1070ff9640 1 -- 172.21.15.81:0/2105210535 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f106c060900 con 0x7f107415ae90 2024-06-29T14:22:50.443 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:50.443 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:50.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 -- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f10600762d0 msgr2=0x7f10600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f10600762d0 0x7f10600787e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1064034000 tx=0x7f1064004730 comp rx=0 tx=0).stop 2024-06-29T14:22:50.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 -- 172.21.15.81:0/2105210535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f107415ae90 msgr2=0x7f10741f6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f107415ae90 0x7f10741f6810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f106c00c730 tx=0x7f106c00cbf0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 -- 172.21.15.81:0/2105210535 shutdown_connections 2024-06-29T14:22:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f10600762d0 0x7f10600787e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f107415ae90 0x7f10741f6810 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10740cf860 0x7f107414c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 --2- 172.21.15.81:0/2105210535 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10740ca700 0x7f10741499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 -- 172.21.15.81:0/2105210535 >> 172.21.15.81:0/2105210535 conn(0x7f10740d00b0 msgr2=0x7f1074074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 -- 172.21.15.81:0/2105210535 shutdown_connections 2024-06-29T14:22:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.446+0000 7f107a9aa640 1 -- 172.21.15.81:0/2105210535 wait complete. 2024-06-29T14:22:50.457 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-06-29T14:22:50.457 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-06-29T14:22:50.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-06-29T14:22:50.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-06-29T14:22:50.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-06-29T14:22:50.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-06-29T14:22:50.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-06-29T14:22:50.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-06-29T14:22:50.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.668+0000 7f3bea438640 1 -- 172.21.15.81:0/184241197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be415e660 msgr2=0x7f3be406e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.668+0000 7f3bea438640 1 --2- 172.21.15.81:0/184241197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be415e660 0x7f3be406e0f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3be4070e50 tx=0x7f3bdc02aae0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.668+0000 7f3bea438640 1 -- 172.21.15.81:0/184241197 shutdown_connections 2024-06-29T14:22:50.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.668+0000 7f3bea438640 1 --2- 172.21.15.81:0/184241197 >> v2:172.21.15.167:3300/0 conn(0x7f3be4073160 0x7f3be4073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.668+0000 7f3bea438640 1 --2- 172.21.15.81:0/184241197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be415e660 0x7f3be406e0f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.668+0000 7f3bea438640 1 -- 172.21.15.81:0/184241197 >> 172.21.15.81:0/184241197 conn(0x7f3be40b1910 msgr2=0x7f3be40b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.670+0000 7fbd01a3c640 1 -- 172.21.15.81:0/3212212853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcfc073330 msgr2=0x7fbcfc15f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.670+0000 7fbd01a3c640 1 --2- 172.21.15.81:0/3212212853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcfc073330 0x7fbcfc15f7e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbcec005fd0 tx=0x7fbcec02a420 comp rx=0 tx=0).stop 2024-06-29T14:22:50.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.670+0000 7fbd01a3c640 1 -- 172.21.15.81:0/3212212853 shutdown_connections 2024-06-29T14:22:50.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.670+0000 7fbd01a3c640 1 --2- 172.21.15.81:0/3212212853 >> v2:172.21.15.167:3300/0 conn(0x7fbcfc070260 0x7fbcfc15b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.670+0000 7fbd01a3c640 1 --2- 172.21.15.81:0/3212212853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcfc073330 0x7fbcfc15f7e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.670+0000 7fbd01a3c640 1 -- 172.21.15.81:0/3212212853 >> 172.21.15.81:0/3212212853 conn(0x7fbcfc06c4d0 msgr2=0x7fbcfc06c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.670+0000 7fbd01a3c640 1 -- 172.21.15.81:0/3212212853 shutdown_connections 2024-06-29T14:22:50.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.671+0000 7fbd01a3c640 1 -- 172.21.15.81:0/3212212853 wait complete. 2024-06-29T14:22:50.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.669+0000 7f3bea438640 1 -- 172.21.15.81:0/184241197 shutdown_connections 2024-06-29T14:22:50.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.671+0000 7f420159a640 1 -- 172.21.15.81:0/2598326426 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420406f6f0 con 0x7f41fc00c0f0 2024-06-29T14:22:50.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.669+0000 7f3bea438640 1 -- 172.21.15.81:0/184241197 wait complete. 2024-06-29T14:22:50.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.672+0000 7f420359e640 1 -- 172.21.15.81:0/2598326426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41fc00c0f0 msgr2=0x7f41fc0516b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.672+0000 7f420359e640 1 --2- 172.21.15.81:0/2598326426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41fc00c0f0 0x7f41fc0516b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4204051d70 tx=0x7f42040a5490 comp rx=0 tx=0).stop 2024-06-29T14:22:50.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.672+0000 7f420359e640 1 -- 172.21.15.81:0/2598326426 shutdown_connections 2024-06-29T14:22:50.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.672+0000 7f420359e640 1 --2- 172.21.15.81:0/2598326426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41fc00c0f0 0x7f41fc0516b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.672+0000 7f420359e640 1 -- 172.21.15.81:0/2598326426 >> 172.21.15.81:0/2598326426 conn(0x7f41fc0569e0 msgr2=0x7f41fc056df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.672+0000 7f3bea438640 1 Processor -- start 2024-06-29T14:22:50.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.672+0000 7f420359e640 1 -- 172.21.15.81:0/2598326426 shutdown_connections 2024-06-29T14:22:50.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.672+0000 7f757dee4640 1 -- 172.21.15.81:0/3199176816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560003b90 msgr2=0x7f7560003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.673+0000 7f757dee4640 1 --2- 172.21.15.81:0/3199176816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560003b90 0x7f7560003fc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7568001810 tx=0x7f7560005830 comp rx=0 tx=0).stop 2024-06-29T14:22:50.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.673+0000 7f757dee4640 1 -- 172.21.15.81:0/3199176816 shutdown_connections 2024-06-29T14:22:50.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.673+0000 7f757dee4640 1 --2- 172.21.15.81:0/3199176816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560003b90 0x7f7560003fc0 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7568001810 tx=0x7f7560005830 comp rx=0 tx=0).stop 2024-06-29T14:22:50.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.673+0000 7f757dee4640 1 -- 172.21.15.81:0/3199176816 >> 172.21.15.81:0/3199176816 conn(0x7f757806c2f0 msgr2=0x7f757806c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.672+0000 7f420359e640 1 -- 172.21.15.81:0/2598326426 wait complete. 2024-06-29T14:22:50.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.672+0000 7f3bea438640 1 -- start start 2024-06-29T14:22:50.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.673+0000 7f757dee4640 1 -- 172.21.15.81:0/3199176816 shutdown_connections 2024-06-29T14:22:50.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.674+0000 7fbd01a3c640 1 Processor -- start 2024-06-29T14:22:50.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.674+0000 7f757dee4640 1 -- 172.21.15.81:0/3199176816 wait complete. 2024-06-29T14:22:50.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.674+0000 7fbd01a3c640 1 -- start start 2024-06-29T14:22:50.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.676+0000 7f420359e640 1 Processor -- start 2024-06-29T14:22:50.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.676+0000 7f420359e640 1 -- start start 2024-06-29T14:22:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.676+0000 7fbd01a3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcfc06dab0 0x7fbcfc153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.676+0000 7fbd01a3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcfc070260 0x7fbcfc156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.676+0000 7fbd01a3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcfc0b07c0 0x7fbcfc0b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7fbcfbfff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcfc070260 0x7fbcfc156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7fbcfbfff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcfc070260 0x7fbcfc156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46344/0 (socket says 172.21.15.81:46344) 2024-06-29T14:22:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7fbcfbfff640 1 -- 172.21.15.81:0/1562660751 learned_addr learned my addr 172.21.15.81:0/1562660751 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7f757dee4640 1 Processor -- start 2024-06-29T14:22:50.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7f3bea438640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3be4073160 0x7f3be414f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7f3bea438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be415e660 0x7f3be414f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7f3be3fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3be4073160 0x7f3be414f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7f757dee4640 1 -- start start 2024-06-29T14:22:50.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7f3bea438640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3be4161c50 0x7f3be414fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7f3be3fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3be4073160 0x7f3be414f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46360/0 (socket says 172.21.15.81:46360) 2024-06-29T14:22:50.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7f3be3fff640 1 -- 172.21.15.81:0/949851514 learned_addr learned my addr 172.21.15.81:0/949851514 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:50.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc15eb10 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc15e990 con 0x7fbcfc070260 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbcfc15ec90 con 0x7fbcfc06dab0 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7fbd0123b640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcfc0b07c0 0x7fbcfc0b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7fbd0123b640 1 -- 172.21.15.81:0/1562660751 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcfc06dab0 msgr2=0x7fbcfc153d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7fbd0123b640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcfc06dab0 0x7fbcfc153d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7fbd0123b640 1 -- 172.21.15.81:0/1562660751 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcfc070260 msgr2=0x7fbcfc156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7fbd0123b640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcfc070260 0x7fbcfc156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7fbd0123b640 1 -- 172.21.15.81:0/1562660751 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcfc154280 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7fbd0123b640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcfc0b07c0 0x7fbcfc0b0bd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400e9e0 tx=0x7fbcf400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7fbcf97fa640 1 -- 172.21.15.81:0/1562660751 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4004d20 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcfc0b1110 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.677+0000 7f3bea438640 1 -- 172.21.15.81:0/949851514 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3be40b36d0 con 0x7f3be415e660 2024-06-29T14:22:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f3bea438640 1 -- 172.21.15.81:0/949851514 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3be40b3550 con 0x7f3be4073160 2024-06-29T14:22:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f3bea438640 1 -- 172.21.15.81:0/949851514 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3be40b3850 con 0x7f3be4161c50 2024-06-29T14:22:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f3be37fe640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be415e660 0x7f3be414f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f3be89d5640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3be4161c50 0x7f3be414fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f3be3fff640 1 -- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3be4161c50 msgr2=0x7f3be414fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f3be3fff640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3be4161c50 0x7f3be414fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f3be3fff640 1 -- 172.21.15.81:0/949851514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be415e660 msgr2=0x7f3be414f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f3be3fff640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be415e660 0x7f3be414f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f3be3fff640 1 -- 172.21.15.81:0/949851514 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be4159410 con 0x7f3be4073160 2024-06-29T14:22:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f420359e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41fc00c0f0 0x7f41fc0da370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f3be3fff640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3be4073160 0x7f3be414f470 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009d60 tx=0x7f3bdc003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7f420359e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41fc00c6b0 0x7f41fc0e4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41fc0dfa60 0x7f41fc0dfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420259c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41fc00c0f0 0x7f41fc0da370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420259c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41fc00c0f0 0x7f41fc0da370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:58114/0 (socket says 172.21.15.81:58114) 2024-06-29T14:22:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420259c640 1 -- 172.21.15.81:0/3666022271 learned_addr learned my addr 172.21.15.81:0/3666022271 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc051d70 con 0x7f41fc0dfa60 2024-06-29T14:22:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f41fc051bf0 con 0x7f41fc00c6b0 2024-06-29T14:22:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f41fc051ef0 con 0x7f41fc00c0f0 2024-06-29T14:22:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f7112144640 1 -- 172.21.15.81:0/2168829034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70f8005150 msgr2=0x7f70f800ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f7112144640 1 --2- 172.21.15.81:0/2168829034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70f8005150 0x7f70f800ffd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7104001e70 tx=0x7f70f8005830 comp rx=0 tx=0).stop 2024-06-29T14:22:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f710bfff640 1 -- 172.21.15.81:0/2168829034 reap_dead start 2024-06-29T14:22:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f7112144640 1 -- 172.21.15.81:0/2168829034 shutdown_connections 2024-06-29T14:22:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f7112144640 1 -- 172.21.15.81:0/2168829034 >> 172.21.15.81:0/2168829034 conn(0x7f710c0d0b70 msgr2=0x7f710c0d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f3be0ff9640 1 -- 172.21.15.81:0/949851514 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc044070 con 0x7f3be4073160 2024-06-29T14:22:50.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f7112144640 1 -- 172.21.15.81:0/2168829034 shutdown_connections 2024-06-29T14:22:50.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f7112144640 1 -- 172.21.15.81:0/2168829034 wait complete. 2024-06-29T14:22:50.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f3be0ff9640 1 -- 172.21.15.81:0/949851514 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bdc004450 con 0x7f3be4073160 2024-06-29T14:22:50.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420259c640 1 -- 172.21.15.81:0/3666022271 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41fc00c6b0 msgr2=0x7f41fc0e4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420259c640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41fc00c6b0 0x7f41fc0e4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420259c640 1 -- 172.21.15.81:0/3666022271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41fc0dfa60 msgr2=0x7f41fc0dfe70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:22:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f3be0ff9640 1 -- 172.21.15.81:0/949851514 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc02d410 con 0x7f3be4073160 2024-06-29T14:22:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f4202d9d640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41fc0dfa60 0x7f41fc0dfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420259c640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41fc0dfa60 0x7f41fc0dfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f420259c640 1 -- 172.21.15.81:0/3666022271 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41fc0e03b0 con 0x7f41fc00c0f0 2024-06-29T14:22:50.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.679+0000 7f4202d9d640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41fc0dfa60 0x7f41fc0dfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:50.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f757dee4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f757815d680 0x7f75780cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f420259c640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41fc00c0f0 0x7f41fc0da370 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f41f400c3f0 tx=0x7f41f400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f757dee4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75780b0760 0x7f75780b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f75777fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f757815d680 0x7f75780cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f75777fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f757815d680 0x7f75780cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46384/0 (socket says 172.21.15.81:46384) 2024-06-29T14:22:50.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f75777fe640 1 -- 172.21.15.81:0/3747996057 learned_addr learned my addr 172.21.15.81:0/3747996057 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:50.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7fe69b506640 1 -- 172.21.15.81:0/443552932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6940737c0 msgr2=0x7fe694154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7fe69b506640 1 --2- 172.21.15.81:0/443552932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6940737c0 0x7fe694154300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe690005fd0 tx=0x7fe69002b250 comp rx=0 tx=0).stop 2024-06-29T14:22:50.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f757dee4640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f757815e3c0 0x7f757815e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f7576ffd640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75780b0760 0x7f75780b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f757dee4640 1 -- 172.21.15.81:0/3747996057 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f75781445a0 con 0x7f757815e3c0 2024-06-29T14:22:50.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7fe69b506640 1 -- 172.21.15.81:0/443552932 shutdown_connections 2024-06-29T14:22:50.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7fe69b506640 1 --2- 172.21.15.81:0/443552932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6940737c0 0x7fe694154300 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7fe69b506640 1 -- 172.21.15.81:0/443552932 >> 172.21.15.81:0/443552932 conn(0x7fe69406c920 msgr2=0x7fe6940b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f7577fff640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f757815e3c0 0x7f757815e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f41f2ffd640 1 -- 172.21.15.81:0/3666022271 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4017070 con 0x7f41fc00c0f0 2024-06-29T14:22:50.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7fe69b506640 1 -- 172.21.15.81:0/443552932 shutdown_connections 2024-06-29T14:22:50.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f757dee4640 1 -- 172.21.15.81:0/3747996057 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7578144420 con 0x7f757815d680 2024-06-29T14:22:50.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f41f2ffd640 1 -- 172.21.15.81:0/3666022271 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41f40035f0 con 0x7f41fc00c0f0 2024-06-29T14:22:50.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41fc0390c0 con 0x7f41fc00c0f0 2024-06-29T14:22:50.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7f757dee4640 1 -- 172.21.15.81:0/3747996057 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f75781624c0 con 0x7f75780b0760 2024-06-29T14:22:50.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7fb1b9968640 1 -- 172.21.15.81:0/2935599685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1a00039e0 msgr2=0x7fb1a0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7fb1b9968640 1 --2- 172.21.15.81:0/2935599685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1a00039e0 0x7fb1a0003e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac055830 tx=0x7fb1a0005830 comp rx=0 tx=0).stop 2024-06-29T14:22:50.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fb1b37fe640 1 -- 172.21.15.81:0/2935599685 reap_dead start 2024-06-29T14:22:50.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41fc17aba0 con 0x7f41fc00c0f0 2024-06-29T14:22:50.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fb1b9968640 1 -- 172.21.15.81:0/2935599685 shutdown_connections 2024-06-29T14:22:50.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fb1b9968640 1 -- 172.21.15.81:0/2935599685 >> 172.21.15.81:0/2935599685 conn(0x7fb1ac0362f0 msgr2=0x7fb1ac036700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fe837ac5640 1 -- 172.21.15.81:0/1826493165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe83008ccd0 msgr2=0x7fe8300cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fe837ac5640 1 --2- 172.21.15.81:0/1826493165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe83008ccd0 0x7fe8300cee70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe8300d4ce0 tx=0x7fe82002a390 comp rx=0 tx=0).stop 2024-06-29T14:22:50.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fb1b9968640 1 -- 172.21.15.81:0/2935599685 shutdown_connections 2024-06-29T14:22:50.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fe837ac5640 1 -- 172.21.15.81:0/1826493165 shutdown_connections 2024-06-29T14:22:50.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fe837ac5640 1 --2- 172.21.15.81:0/1826493165 >> v2:172.21.15.81:3300/0 conn(0x7fe8300d17b0 0x7fe8300d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fe837ac5640 1 --2- 172.21.15.81:0/1826493165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe83008ccd0 0x7fe8300cee70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fe837ac5640 1 --2- 172.21.15.81:0/1826493165 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe83008c270 0x7fe8300ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fe837ac5640 1 -- 172.21.15.81:0/1826493165 >> 172.21.15.81:0/1826493165 conn(0x7fe8300b56d0 msgr2=0x7fe8300b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fe837ac5640 1 -- 172.21.15.81:0/1826493165 shutdown_connections 2024-06-29T14:22:50.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fe837ac5640 1 -- 172.21.15.81:0/1826493165 wait complete. 2024-06-29T14:22:50.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fb1b9968640 1 -- 172.21.15.81:0/2935599685 wait complete. 2024-06-29T14:22:50.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.682+0000 7f3bea438640 1 -- 172.21.15.81:0/949851514 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be41596f0 con 0x7f3be4073160 2024-06-29T14:22:50.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.682+0000 7f3bea438640 1 -- 172.21.15.81:0/949851514 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be415fa30 con 0x7f3be4073160 2024-06-29T14:22:50.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7f41f2ffd640 1 -- 172.21.15.81:0/3666022271 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4003a40 con 0x7f41fc00c0f0 2024-06-29T14:22:50.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7fbcf97fa640 1 -- 172.21.15.81:0/1562660751 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbcf400cb50 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.678+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcfc1fcb60 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbcfc15b870 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7fbcf97fa640 1 -- 172.21.15.81:0/1562660751 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4005020 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.682+0000 7f7112144640 1 Processor -- start 2024-06-29T14:22:50.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.682+0000 7f7112144640 1 -- start start 2024-06-29T14:22:50.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7f7576ffd640 1 -- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f757815d680 msgr2=0x7f75780cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.680+0000 7fe69b506640 1 -- 172.21.15.81:0/443552932 wait complete. 2024-06-29T14:22:50.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.683+0000 7f3bea438640 1 -- 172.21.15.81:0/949851514 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3be4073570 con 0x7f3be4073160 2024-06-29T14:22:50.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.683+0000 7fe69b506640 1 Processor -- start 2024-06-29T14:22:50.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.683+0000 7fe69b506640 1 -- start start 2024-06-29T14:22:50.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7f7576ffd640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f757815d680 0x7f75780cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.681+0000 7f7576ffd640 1 -- 172.21.15.81:0/3747996057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f757815e3c0 msgr2=0x7f757815e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.682+0000 7f7576ffd640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f757815e3c0 0x7f757815e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.682+0000 7f7576ffd640 1 -- 172.21.15.81:0/3747996057 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75781f0ea0 con 0x7f75780b0760 2024-06-29T14:22:50.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.682+0000 7f7577fff640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f757815e3c0 0x7f757815e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:50.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.683+0000 7f7576ffd640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75780b0760 0x7f75780b0b70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7570005fd0 tx=0x7f757000edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.683+0000 7f755ffff640 1 -- 172.21.15.81:0/3747996057 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570004d20 con 0x7f75780b0760 2024-06-29T14:22:50.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.683+0000 7f755ffff640 1 -- 172.21.15.81:0/3747996057 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f757000cb20 con 0x7f75780b0760 2024-06-29T14:22:50.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.683+0000 7f755ffff640 1 -- 172.21.15.81:0/3747996057 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570005020 con 0x7f75780b0760 2024-06-29T14:22:50.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.683+0000 7f757dee4640 1 -- 172.21.15.81:0/3747996057 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75780b10b0 con 0x7f75780b0760 2024-06-29T14:22:50.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.683+0000 7f757dee4640 1 -- 172.21.15.81:0/3747996057 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75780aa190 con 0x7f75780b0760 2024-06-29T14:22:50.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.683+0000 7f41f2ffd640 1 -- 172.21.15.81:0/3666022271 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f41f401a070 con 0x7f41fc00c0f0 2024-06-29T14:22:50.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.684+0000 7f41f0ff9640 1 -- 172.21.15.81:0/3666022271 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41fc0e4bf0 con 0x7f41fc00c0f0 2024-06-29T14:22:50.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.684+0000 7fb1b9968640 1 Processor -- start 2024-06-29T14:22:50.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.684+0000 7f41f2ffd640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f41e40763a0 0x7f41e40788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.684+0000 7fe837ac5640 1 Processor -- start 2024-06-29T14:22:50.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.684+0000 7fbcf97fa640 1 -- 172.21.15.81:0/1562660751 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbcf40051c0 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f7112144640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70f8005150 0x7f710c138250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f4201d9b640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f41e40763a0 0x7f41e40788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f7112144640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f710c15a550 0x7f710c13b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7fe837ac5640 1 -- start start 2024-06-29T14:22:50.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f7112144640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f710c13b8e0 0x7f710c1371b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f41f2ffd640 1 -- 172.21.15.81:0/3666022271 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f41f4096ad0 con 0x7f41fc00c0f0 2024-06-29T14:22:50.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f7112144640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f710c0732d0 con 0x7f710c13b8e0 2024-06-29T14:22:50.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f7112144640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f710c073150 con 0x7f710c15a550 2024-06-29T14:22:50.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f7112144640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f710c06f4c0 con 0x7f70f8005150 2024-06-29T14:22:50.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f4201d9b640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f41e40763a0 0x7f41e40788b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41fc008ef0 tx=0x7f41f8006ba0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f41f2ffd640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] conn(0x7f41e407e670 0x7f41e4080ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f41f2ffd640 1 -- 172.21.15.81:0/3666022271 --> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41e4081190 con 0x7f41e407e670 2024-06-29T14:22:50.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f41f2ffd640 1 -- 172.21.15.81:0/3666022271 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f41f409d050 con 0x7f41fc00c0f0 2024-06-29T14:22:50.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7fb1b9968640 1 -- start start 2024-06-29T14:22:50.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f4202d9d640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] conn(0x7f41e407e670 0x7f41e4080ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f757dee4640 1 -- 172.21.15.81:0/3747996057 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7578072c20 con 0x7f75780b0760 2024-06-29T14:22:50.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f710b7fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70f8005150 0x7f710c138250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7f7103fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f710c15a550 0x7f710c13b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe69b506640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6940728d0 0x7fe694151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe69b506640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6940737c0 0x7fe6941fb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe69b506640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe694201bb0 0x7fe6940b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.685+0000 7fbcf97fa640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbce80762d0 0x7fbce80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fbcf97fa640 1 -- 172.21.15.81:0/1562660751 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbcf4098d20 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fbcf97fa640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] conn(0x7fbce807e4c0 0x7fbce8080910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fbcf97fa640 1 -- 172.21.15.81:0/1562660751 --> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbcf401fe80 con 0x7fbce807e4c0 2024-06-29T14:22:50.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fbcf97fa640 1 -- 172.21.15.81:0/1562660751 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fbcf40991b0 con 0x7fbcfc0b07c0 2024-06-29T14:22:50.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe69b506640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe69415afd0 con 0x7fe694201bb0 2024-06-29T14:22:50.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f3be0ff9640 1 -- 172.21.15.81:0/949851514 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3bdc02d5b0 con 0x7f3be4073160 2024-06-29T14:22:50.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe69a504640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6940728d0 0x7fe694151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe69a504640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6940728d0 0x7fe694151d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:58150/0 (socket says 172.21.15.81:58150) 2024-06-29T14:22:50.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f710bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f710c13b8e0 0x7f710c1371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f710bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f710c13b8e0 0x7f710c1371b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36020/0 (socket says 172.21.15.81:36020) 2024-06-29T14:22:50.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f710bfff640 1 -- 172.21.15.81:0/288005421 learned_addr learned my addr 172.21.15.81:0/288005421 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:50.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fbd00a3a640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbce80762d0 0x7fbce80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe69ad05640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe694201bb0 0x7fe6940b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe69a504640 1 -- 172.21.15.81:0/2730616679 learned_addr learned my addr 172.21.15.81:0/2730616679 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:50.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe699d03640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6940737c0 0x7fe6941fb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe69b506640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe694070010 con 0x7fe6940737c0 2024-06-29T14:22:50.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7fe69b506640 1 -- 172.21.15.81:0/2730616679 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe69415b150 con 0x7fe6940728d0 2024-06-29T14:22:50.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f4202d9d640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] conn(0x7f41e407e670 0x7f41e4080ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f710bfff640 1 -- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70f8005150 msgr2=0x7f710c138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f710bfff640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70f8005150 0x7f710c138250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f710bfff640 1 -- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f710c15a550 msgr2=0x7f710c13b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f710bfff640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f710c15a550 0x7f710c13b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f710bfff640 1 -- 172.21.15.81:0/288005421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710c141870 con 0x7f710c13b8e0 2024-06-29T14:22:50.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7fbcfbfff640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] conn(0x7fbce807e4c0 0x7fbce8080910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f3be0ff9640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3bd00763a0 0x7f3bd00788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f3be0ff9640 1 -- 172.21.15.81:0/949851514 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3bdc0b8640 con 0x7f3be4073160 2024-06-29T14:22:50.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f710bfff640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f710c13b8e0 0x7f710c1371b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f710c0b6130 tx=0x7f7104001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.686+0000 7f755ffff640 1 -- 172.21.15.81:0/3747996057 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f75700051c0 con 0x7f75780b0760 2024-06-29T14:22:50.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7fbd00a3a640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbce80762d0 0x7fbce80787e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbcf00095d0 tx=0x7fbcf0006b30 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f3be0ff9640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] conn(0x7f3bd007e610 0x7f3bd0080a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f3be0ff9640 1 -- 172.21.15.81:0/949851514 --> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3bdc02cad0 con 0x7f3bd007e610 2024-06-29T14:22:50.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f3be0ff9640 1 -- 172.21.15.81:0/949851514 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3bdc0b8a30 con 0x7f3be4073160 2024-06-29T14:22:50.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f3be37fe640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3bd00763a0 0x7f3bd00788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7fe69ad05640 1 -- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6940728d0 msgr2=0x7fe694151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f755ffff640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75640763a0 0x7f75640788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7fe69ad05640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6940728d0 0x7fe694151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7fe69ad05640 1 -- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6940737c0 msgr2=0x7fe6941fb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7fe69ad05640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6940737c0 0x7fe6941fb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7fe69ad05640 1 -- 172.21.15.81:0/2730616679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69414de50 con 0x7fe694201bb0 2024-06-29T14:22:50.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f7108ff9640 1 -- 172.21.15.81:0/288005421 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710400c380 con 0x7f710c13b8e0 2024-06-29T14:22:50.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f7108ff9640 1 -- 172.21.15.81:0/288005421 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7104001c00 con 0x7f710c13b8e0 2024-06-29T14:22:50.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f7108ff9640 1 -- 172.21.15.81:0/288005421 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710402bb10 con 0x7f710c13b8e0 2024-06-29T14:22:50.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7fe69ad05640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe694201bb0 0x7fe6940b09f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00c700 tx=0x7fe68c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7fb1b9968640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1a00039e0 0x7fb1ac176220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7fb1b9968640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1ac0d4ca0 0x7fb1ac0dd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f755ffff640 1 -- 172.21.15.81:0/3747996057 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7570097f10 con 0x7f75780b0760 2024-06-29T14:22:50.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f755ffff640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] conn(0x7f756407e590 0x7f75640809e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.687+0000 7f755ffff640 1 -- 172.21.15.81:0/3747996057 --> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7570018ec0 con 0x7f756407e590 2024-06-29T14:22:50.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fb1b9968640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1ac0ddf00 0x7fb1ac0502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7f755ffff640 1 -- 172.21.15.81:0/3747996057 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f757005d150 con 0x7f75780b0760 2024-06-29T14:22:50.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7f7577fff640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] conn(0x7f756407e590 0x7f75640809e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fb1b2ffd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1a00039e0 0x7fb1ac176220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7f3be37fe640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3bd00763a0 0x7f3bd00788b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3be414f090 tx=0x7f3bd8009060 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7f3be89d5640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] conn(0x7f3bd007e610 0x7f3bd0080a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f710c141b50 con 0x7f710c13b8e0 2024-06-29T14:22:50.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fe68affd640 1 -- 172.21.15.81:0/2730616679 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c003b70 con 0x7fe694201bb0 2024-06-29T14:22:50.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fb1b27fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1ac0d4ca0 0x7fb1ac0dd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fb1b27fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1ac0d4ca0 0x7fb1ac0dd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46412/0 (socket says 172.21.15.81:46412) 2024-06-29T14:22:50.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fb1b2ffd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1a00039e0 0x7fb1ac176220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:58160/0 (socket says 172.21.15.81:58160) 2024-06-29T14:22:50.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fe68affd640 1 -- 172.21.15.81:0/2730616679 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe68c003d10 con 0x7fe694201bb0 2024-06-29T14:22:50.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fe69b506640 1 -- 172.21.15.81:0/2730616679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6941525e0 con 0x7fe694201bb0 2024-06-29T14:22:50.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7f3be89d5640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] conn(0x7f3bd007e610 0x7f3bd0080a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7f7577fff640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] conn(0x7f756407e590 0x7f75640809e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f710c142630 con 0x7f710c13b8e0 2024-06-29T14:22:50.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fb1b9968640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac0e03f0 con 0x7fb1ac0ddf00 2024-06-29T14:22:50.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fb1b9968640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac0e0270 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fb1b9968640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb1ac00be20 con 0x7fb1a00039e0 2024-06-29T14:22:50.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fb1b27fc640 1 -- 172.21.15.81:0/717207184 learned_addr learned my addr 172.21.15.81:0/717207184 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:50.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7fbcfbfff640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] conn(0x7fbce807e4c0 0x7fbce8080910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.689+0000 7fe837ac5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe83008ccd0 0x7fe83009f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.689+0000 7f755ffff640 1 -- 172.21.15.81:0/3747996057 <== osd.3 v2:172.21.15.81:6809/3633523901 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7570018ec0 con 0x7f756407e590 2024-06-29T14:22:50.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.689+0000 7f3be0ff9640 1 -- 172.21.15.81:0/949851514 <== osd.1 v2:172.21.15.81:6808/2523783345 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3bdc02cad0 con 0x7f3bd007e610 2024-06-29T14:22:50.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.689+0000 7f75777fe640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75640763a0 0x7f75640788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.689+0000 7fe68affd640 1 -- 172.21.15.81:0/2730616679 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c005330 con 0x7fe694201bb0 2024-06-29T14:22:50.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.689+0000 7fe69b506640 1 -- 172.21.15.81:0/2730616679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe694152ba0 con 0x7fe694201bb0 2024-06-29T14:22:50.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.690+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f710c06f130 con 0x7f710c13b8e0 2024-06-29T14:22:50.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.690+0000 7fb1b27fc640 1 -- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1a00039e0 msgr2=0x7fb1ac176220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.690+0000 7fb1b27fc640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1a00039e0 0x7fb1ac176220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.690+0000 7fb1b27fc640 1 -- 172.21.15.81:0/717207184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1ac0ddf00 msgr2=0x7fb1ac0502c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:22:50.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.690+0000 7fb1b27fc640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1ac0ddf00 0x7fb1ac0502c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.690+0000 7fb1b27fc640 1 -- 172.21.15.81:0/717207184 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ac176760 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.688+0000 7f41f2ffd640 1 -- 172.21.15.81:0/3666022271 <== osd.2 v2:172.21.15.81:6824/4255314520 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f41e4081190 con 0x7f41e407e670 2024-06-29T14:22:50.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fe69b506640 1 -- 172.21.15.81:0/2730616679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe694154300 con 0x7fe694201bb0 2024-06-29T14:22:50.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fb1b27fc640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1ac0d4ca0 0x7fb1ac0dd9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00d500 tx=0x7fb19c00d9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fe837ac5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300d17b0 0x7fe8300b7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fe837ac5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8300b8180 0x7fe8301db990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fe837ac5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe8300d2790 con 0x7fe8300d17b0 2024-06-29T14:22:50.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fe837ac5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe8300d2610 con 0x7fe83008ccd0 2024-06-29T14:22:50.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fe837ac5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe8300b3240 con 0x7fe8300b8180 2024-06-29T14:22:50.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fe835060640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300d17b0 0x7fe8300b7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fe835060640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300d17b0 0x7fe8300b7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36038/0 (socket says 172.21.15.81:36038) 2024-06-29T14:22:50.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fe835060640 1 -- 172.21.15.81:0/1918173320 learned_addr learned my addr 172.21.15.81:0/1918173320 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:50.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fb1abfff640 1 -- 172.21.15.81:0/717207184 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c016020 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fb1abfff640 1 -- 172.21.15.81:0/717207184 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb19c012630 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.691+0000 7fb1abfff640 1 -- 172.21.15.81:0/717207184 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c012a80 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.692+0000 7fe68affd640 1 -- 172.21.15.81:0/2730616679 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe68c0054d0 con 0x7fe694201bb0 2024-06-29T14:22:50.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.689+0000 7fbcf97fa640 1 -- 172.21.15.81:0/1562660751 <== osd.0 v2:172.21.15.81:6800/1533488904 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcf401fe80 con 0x7fbce807e4c0 2024-06-29T14:22:50.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7fe68affd640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6780763a0 0x7fe6780788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.692+0000 7fe835060640 1 -- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8300b8180 msgr2=0x7fe8301db990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:22:50.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.692+0000 7fe835060640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8300b8180 0x7fe8301db990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.692+0000 7fe835060640 1 -- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe83008ccd0 msgr2=0x7fe83009f9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:22:50.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.692+0000 7fe835060640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe83008ccd0 0x7fe83009f9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.692+0000 7fe835060640 1 -- 172.21.15.81:0/1918173320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8300cb8b0 con 0x7fe8300d17b0 2024-06-29T14:22:50.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.692+0000 7fe835060640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300d17b0 0x7fe8300b7c40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe820009780 tx=0x7fe820033830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7fe8267fc640 1 -- 172.21.15.81:0/1918173320 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe820040030 con 0x7fe8300d17b0 2024-06-29T14:22:50.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7fe8267fc640 1 -- 172.21.15.81:0/1918173320 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe820033c30 con 0x7fe8300d17b0 2024-06-29T14:22:50.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ac0de310 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7fe69a504640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6780763a0 0x7fe6780788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7fe8267fc640 1 -- 172.21.15.81:0/1918173320 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe820037040 con 0x7fe8300d17b0 2024-06-29T14:22:50.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ac0e0f50 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7fe68affd640 1 -- 172.21.15.81:0/2730616679 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe68c0972f0 con 0x7fe694201bb0 2024-06-29T14:22:50.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8300a05a0 con 0x7fe8300d17b0 2024-06-29T14:22:50.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8300c7a10 con 0x7fe8300d17b0 2024-06-29T14:22:50.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7f7108ff9640 1 -- 172.21.15.81:0/288005421 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f710402bcb0 con 0x7f710c13b8e0 2024-06-29T14:22:50.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7f7108ff9640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70ec076300 0x7f70ec078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.693+0000 7f7108ff9640 1 -- 172.21.15.81:0/288005421 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f71040b70f0 con 0x7f710c13b8e0 2024-06-29T14:22:50.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7f7108ff9640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] conn(0x7f70ec07e4f0 0x7f70ec080940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7f7108ff9640 1 -- 172.21.15.81:0/288005421 --> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f710403ce50 con 0x7f70ec07e4f0 2024-06-29T14:22:50.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7fe69a504640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6780763a0 0x7fe6780788b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe690000c00 tx=0x7fe69002d040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7fe68affd640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] conn(0x7fe67807e610 0x7fe678080a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7fe68affd640 1 -- 172.21.15.81:0/2730616679 --> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe68c008e50 con 0x7fe67807e610 2024-06-29T14:22:50.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7fe68affd640 1 -- 172.21.15.81:0/2730616679 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe68c09e050 con 0x7fe694201bb0 2024-06-29T14:22:50.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7f7103fff640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] conn(0x7f70ec07e4f0 0x7f70ec080940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe83008c270 con 0x7fe8300d17b0 2024-06-29T14:22:50.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7f7108ff9640 1 -- 172.21.15.81:0/288005421 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f71040be050 con 0x7f710c13b8e0 2024-06-29T14:22:50.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7fe699d03640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] conn(0x7fe67807e610 0x7fe678080a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.694+0000 7f710b7fe640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70ec076300 0x7f70ec078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.695+0000 7f7103fff640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] conn(0x7f70ec07e4f0 0x7f70ec080940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.695+0000 7f75777fe640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75640763a0 0x7f75640788b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75780b2d50 tx=0x7f7568001360 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.695+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb1ac053860 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.695+0000 7fe699d03640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] conn(0x7fe67807e610 0x7fe678080a60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.695+0000 7f710b7fe640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70ec076300 0x7f70ec078810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f710c073bf0 tx=0x7f70fc0092a0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.696+0000 7fe8267fc640 1 -- 172.21.15.81:0/1918173320 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe820043070 con 0x7fe8300d17b0 2024-06-29T14:22:50.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.696+0000 7f7108ff9640 1 -- 172.21.15.81:0/288005421 <== osd.4 v2:172.21.15.167:6824/220767630 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f710403ce50 con 0x7f70ec07e4f0 2024-06-29T14:22:50.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fe68affd640 1 -- 172.21.15.81:0/2730616679 <== osd.5 v2:172.21.15.167:6800/4186590320 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe68c008e50 con 0x7fe67807e610 2024-06-29T14:22:50.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fe8267fc640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe8140763a0 0x7fe8140788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fe8267fc640 1 -- 172.21.15.81:0/1918173320 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe8200b7c90 con 0x7fe8300d17b0 2024-06-29T14:22:50.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fe8267fc640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] conn(0x7fe81407e670 0x7fe814080ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fe8267fc640 1 -- 172.21.15.81:0/1918173320 --> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe814081190 con 0x7fe81407e670 2024-06-29T14:22:50.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fe8267fc640 1 -- 172.21.15.81:0/1918173320 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe8200b8080 con 0x7fe8300d17b0 2024-06-29T14:22:50.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.696+0000 7fb1abfff640 1 -- 172.21.15.81:0/717207184 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb19c019020 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fb1abfff640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb194076320 0x7fb194078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fb1abfff640 1 -- 172.21.15.81:0/717207184 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb19c097600 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fb1abfff640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] conn(0x7fb19407e590 0x7fb1940809e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:50.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fb1abfff640 1 -- 172.21.15.81:0/717207184 --> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1940810b0 con 0x7fb19407e590 2024-06-29T14:22:50.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.697+0000 7fb1abfff640 1 -- 172.21.15.81:0/717207184 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb19c09e050 con 0x7fb1ac0d4ca0 2024-06-29T14:22:50.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.698+0000 7fe836062640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] conn(0x7fe81407e670 0x7fe814080ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.698+0000 7fb1b37fe640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] conn(0x7fb19407e590 0x7fb1940809e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.698+0000 7fb1b2ffd640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb194076320 0x7fb194078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.698+0000 7fe835861640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe8140763a0 0x7fe8140788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:50.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.698+0000 7fe836062640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] conn(0x7fe81407e670 0x7fe814080ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.699+0000 7fb1b2ffd640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb194076320 0x7fb194078830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4034000 tx=0x7fb1a40046c0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.699+0000 7fb1b37fe640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] conn(0x7fb19407e590 0x7fb1940809e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.699+0000 7fe8267fc640 1 -- 172.21.15.81:0/1918173320 <== osd.6 v2:172.21.15.167:6816/2006194601 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe814081190 con 0x7fe81407e670 2024-06-29T14:22:50.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.700+0000 7fb1abfff640 1 -- 172.21.15.81:0/717207184 <== osd.7 v2:172.21.15.167:6808/607598008 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb1940810b0 con 0x7fb19407e590 2024-06-29T14:22:50.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.700+0000 7fe835861640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe8140763a0 0x7fe8140788b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe828005fd0 tx=0x7fe8280095c0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:50.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.718+0000 7f3bea438640 1 -- 172.21.15.81:0/949851514 --> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3be40b2d70 con 0x7f3bd007e610 2024-06-29T14:22:50.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.719+0000 7f3be0ff9640 1 -- 172.21.15.81:0/949851514 <== osd.1 v2:172.21.15.81:6808/2523783345 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3be40b2d70 con 0x7f3bd007e610 2024-06-29T14:22:50.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.720+0000 7f3bca7fc640 1 -- 172.21.15.81:0/949851514 >> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] conn(0x7f3bd007e610 msgr2=0x7f3bd0080a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.720+0000 7f3bca7fc640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] conn(0x7f3bd007e610 0x7f3bd0080a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.720+0000 7f3bca7fc640 1 -- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3bd00763a0 msgr2=0x7f3bd00788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.720+0000 7f3bca7fc640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3bd00763a0 0x7f3bd00788b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3be414f090 tx=0x7f3bd8009060 comp rx=0 tx=0).stop 2024-06-29T14:22:50.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.721+0000 7f3bca7fc640 1 -- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3be4073160 msgr2=0x7f3be414f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.721+0000 7f3bca7fc640 1 --2- 172.21.15.81:0/949851514 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3be4073160 0x7f3be414f470 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009d60 tx=0x7f3bdc003f10 comp rx=0 tx=0).stop 2024-06-29T14:22:50.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.721+0000 7f3be89d5640 1 -- 172.21.15.81:0/949851514 reap_dead start 2024-06-29T14:22:50.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.721+0000 7f41f0ff9640 1 -- 172.21.15.81:0/3666022271 --> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f41fc00cb90 con 0x7f41e407e670 2024-06-29T14:22:50.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.721+0000 7f757dee4640 1 -- 172.21.15.81:0/3747996057 --> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7578145b10 con 0x7f756407e590 2024-06-29T14:22:50.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.721+0000 7f3bca7fc640 1 -- 172.21.15.81:0/949851514 shutdown_connections 2024-06-29T14:22:50.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.721+0000 7f3bca7fc640 1 -- 172.21.15.81:0/949851514 >> 172.21.15.81:0/949851514 conn(0x7f3be40b1910 msgr2=0x7f3be415b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.721+0000 7f41f2ffd640 1 -- 172.21.15.81:0/3666022271 <== osd.2 v2:172.21.15.81:6824/4255314520 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f41fc00cb90 con 0x7f41e407e670 2024-06-29T14:22:50.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 --> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbcfc1fc5f0 con 0x7fbce807e4c0 2024-06-29T14:22:50.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7f755ffff640 1 -- 172.21.15.81:0/3747996057 <== osd.3 v2:172.21.15.81:6809/3633523901 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7578145b10 con 0x7f756407e590 2024-06-29T14:22:50.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7fbcf97fa640 1 -- 172.21.15.81:0/1562660751 <== osd.0 v2:172.21.15.81:6800/1533488904 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbcfc1fc5f0 con 0x7fbce807e4c0 2024-06-29T14:22:50.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 >> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] conn(0x7f41e407e670 msgr2=0x7f41e4080ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7f420359e640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] conn(0x7f41e407e670 0x7f41e4080ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f41e40763a0 msgr2=0x7f41e40788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7f420359e640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f41e40763a0 0x7f41e40788b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41fc008ef0 tx=0x7f41f8006ba0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41fc00c0f0 msgr2=0x7f41fc0da370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7f420359e640 1 --2- 172.21.15.81:0/3666022271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41fc00c0f0 0x7f41fc0da370 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f41f400c3f0 tx=0x7f41f400c8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7f4202d9d640 1 -- 172.21.15.81:0/3666022271 reap_dead start 2024-06-29T14:22:50.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f3bca7fc640 1 -- 172.21.15.81:0/949851514 shutdown_connections 2024-06-29T14:22:50.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f3bca7fc640 1 -- 172.21.15.81:0/949851514 wait complete. 2024-06-29T14:22:50.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 shutdown_connections 2024-06-29T14:22:50.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 >> 172.21.15.81:0/3666022271 conn(0x7f41fc0569e0 msgr2=0x7f41fc0093b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.722+0000 7f755dffb640 1 -- 172.21.15.81:0/3747996057 >> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] conn(0x7f756407e590 msgr2=0x7f75640809e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f755dffb640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] conn(0x7f756407e590 0x7f75640809e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f755dffb640 1 -- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75640763a0 msgr2=0x7f75640788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f755dffb640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75640763a0 0x7f75640788b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75780b2d50 tx=0x7f7568001360 comp rx=0 tx=0).stop 2024-06-29T14:22:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 >> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] conn(0x7fbce807e4c0 msgr2=0x7fbce8080910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f755dffb640 1 -- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75780b0760 msgr2=0x7f75780b0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f755dffb640 1 --2- 172.21.15.81:0/3747996057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75780b0760 0x7f75780b0b70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7570005fd0 tx=0x7f757000edd0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f7577fff640 1 -- 172.21.15.81:0/3747996057 reap_dead start 2024-06-29T14:22:50.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7fbd01a3c640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] conn(0x7fbce807e4c0 0x7fbce8080910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f755dffb640 1 -- 172.21.15.81:0/3747996057 shutdown_connections 2024-06-29T14:22:50.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f755dffb640 1 -- 172.21.15.81:0/3747996057 >> 172.21.15.81:0/3747996057 conn(0x7f757806c2f0 msgr2=0x7f7578146310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 shutdown_connections 2024-06-29T14:22:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f420359e640 1 -- 172.21.15.81:0/3666022271 wait complete. 2024-06-29T14:22:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f755dffb640 1 -- 172.21.15.81:0/3747996057 shutdown_connections 2024-06-29T14:22:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7f755dffb640 1 -- 172.21.15.81:0/3747996057 wait complete. 2024-06-29T14:22:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbce80762d0 msgr2=0x7fbce80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7fbd01a3c640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbce80762d0 0x7fbce80787e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbcf00095d0 tx=0x7fbcf0006b30 comp rx=0 tx=0).stop 2024-06-29T14:22:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcfc0b07c0 msgr2=0x7fbcfc0b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.723+0000 7fbd01a3c640 1 --2- 172.21.15.81:0/1562660751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcfc0b07c0 0x7fbcfc0b0bd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400e9e0 tx=0x7fbcf400eea0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.724+0000 7fbd0123b640 1 -- 172.21.15.81:0/1562660751 reap_dead start 2024-06-29T14:22:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.724+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 shutdown_connections 2024-06-29T14:22:50.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.724+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 >> 172.21.15.81:0/1562660751 conn(0x7fbcfc06c4d0 msgr2=0x7fbcfc15a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.724+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 shutdown_connections 2024-06-29T14:22:50.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.724+0000 7fbd01a3c640 1 -- 172.21.15.81:0/1562660751 wait complete. 2024-06-29T14:22:50.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.729+0000 7fe69b506640 1 -- 172.21.15.81:0/2730616679 --> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe694152390 con 0x7fe67807e610 2024-06-29T14:22:50.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.730+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 --> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe8300a0310 con 0x7fe81407e670 2024-06-29T14:22:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.730+0000 7fe68affd640 1 -- 172.21.15.81:0/2730616679 <== osd.5 v2:172.21.15.167:6800/4186590320 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe694152390 con 0x7fe67807e610 2024-06-29T14:22:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.731+0000 7fe8267fc640 1 -- 172.21.15.81:0/1918173320 <== osd.6 v2:172.21.15.167:6816/2006194601 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe8300a0310 con 0x7fe81407e670 2024-06-29T14:22:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.731+0000 7fe688ff9640 1 -- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] conn(0x7fe67807e610 msgr2=0x7fe678080a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.731+0000 7fe688ff9640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] conn(0x7fe67807e610 0x7fe678080a60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.731+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] conn(0x7fe81407e670 msgr2=0x7fe814080ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.731+0000 7fe837ac5640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] conn(0x7fe81407e670 0x7fe814080ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.731+0000 7fe688ff9640 1 -- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6780763a0 msgr2=0x7fe6780788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.731+0000 7fe688ff9640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6780763a0 0x7fe6780788b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe690000c00 tx=0x7fe69002d040 comp rx=0 tx=0).stop 2024-06-29T14:22:50.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.731+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe8140763a0 msgr2=0x7fe8140788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe837ac5640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe8140763a0 0x7fe8140788b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe828005fd0 tx=0x7fe8280095c0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300d17b0 msgr2=0x7fe8300b7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe837ac5640 1 --2- 172.21.15.81:0/1918173320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300d17b0 0x7fe8300b7c40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe820009780 tx=0x7fe820033830 comp rx=0 tx=0).stop 2024-06-29T14:22:50.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe836062640 1 -- 172.21.15.81:0/1918173320 reap_dead start 2024-06-29T14:22:50.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe688ff9640 1 -- 172.21.15.81:0/2730616679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe694201bb0 msgr2=0x7fe6940b09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 shutdown_connections 2024-06-29T14:22:50.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 >> 172.21.15.81:0/1918173320 conn(0x7fe8300b56d0 msgr2=0x7fe8300b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe688ff9640 1 --2- 172.21.15.81:0/2730616679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe694201bb0 0x7fe6940b09f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00c700 tx=0x7fe68c00cbc0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 shutdown_connections 2024-06-29T14:22:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe837ac5640 1 -- 172.21.15.81:0/1918173320 wait complete. 2024-06-29T14:22:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe69ad05640 1 -- 172.21.15.81:0/2730616679 reap_dead start 2024-06-29T14:22:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe688ff9640 1 -- 172.21.15.81:0/2730616679 shutdown_connections 2024-06-29T14:22:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe688ff9640 1 -- 172.21.15.81:0/2730616679 >> 172.21.15.81:0/2730616679 conn(0x7fe69406c920 msgr2=0x7fe69406dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 --> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb1ac032b90 con 0x7fb19407e590 2024-06-29T14:22:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7fe688ff9640 1 -- 172.21.15.81:0/2730616679 shutdown_connections 2024-06-29T14:22:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.733+0000 7fe688ff9640 1 -- 172.21.15.81:0/2730616679 wait complete. 2024-06-29T14:22:50.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.732+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 --> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f710c139b30 con 0x7f70ec07e4f0 2024-06-29T14:22:50.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.733+0000 7fb1abfff640 1 -- 172.21.15.81:0/717207184 <== osd.7 v2:172.21.15.167:6808/607598008 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb1ac032b90 con 0x7fb19407e590 2024-06-29T14:22:50.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.733+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] conn(0x7fb19407e590 msgr2=0x7fb1940809e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.733+0000 7fb1b9968640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] conn(0x7fb19407e590 0x7fb1940809e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.734+0000 7f7108ff9640 1 -- 172.21.15.81:0/288005421 <== osd.4 v2:172.21.15.167:6824/220767630 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f710c139b30 con 0x7f70ec07e4f0 2024-06-29T14:22:50.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.734+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb194076320 msgr2=0x7fb194078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.734+0000 7fb1b9968640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb194076320 0x7fb194078830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4034000 tx=0x7fb1a40046c0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.734+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1ac0d4ca0 msgr2=0x7fb1ac0dd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.734+0000 7fb1b9968640 1 --2- 172.21.15.81:0/717207184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1ac0d4ca0 0x7fb1ac0dd9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00d500 tx=0x7fb19c00d9c0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.734+0000 7fb1b37fe640 1 -- 172.21.15.81:0/717207184 reap_dead start 2024-06-29T14:22:50.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.734+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 shutdown_connections 2024-06-29T14:22:50.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.734+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 >> 172.21.15.81:0/717207184 conn(0x7fb1ac0362f0 msgr2=0x7fb1ac0db520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.735+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 shutdown_connections 2024-06-29T14:22:50.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.735+0000 7fb1b9968640 1 -- 172.21.15.81:0/717207184 wait complete. 2024-06-29T14:22:50.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] conn(0x7f70ec07e4f0 msgr2=0x7f70ec080940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f7112144640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] conn(0x7f70ec07e4f0 0x7f70ec080940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70ec076300 msgr2=0x7f70ec078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f7112144640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70ec076300 0x7f70ec078810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f710c073bf0 tx=0x7f70fc0092a0 comp rx=0 tx=0).stop 2024-06-29T14:22:50.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f710c13b8e0 msgr2=0x7f710c1371b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:50.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f7112144640 1 --2- 172.21.15.81:0/288005421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f710c13b8e0 0x7f710c1371b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f710c0b6130 tx=0x7f7104001250 comp rx=0 tx=0).stop 2024-06-29T14:22:50.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f710bfff640 1 -- 172.21.15.81:0/288005421 reap_dead start 2024-06-29T14:22:50.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 shutdown_connections 2024-06-29T14:22:50.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 >> 172.21.15.81:0/288005421 conn(0x7f710c0d0b70 msgr2=0x7f710c0b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:50.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 shutdown_connections 2024-06-29T14:22:50.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:50.736+0000 7f7112144640 1 -- 172.21.15.81:0/288005421 wait complete. 2024-06-29T14:22:50.768 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:50.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-29T14:22:50.769 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:50.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-29T14:22:50.771 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-06-29T14:22:50.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-29T14:22:50.772 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:50.772 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-29T14:22:50.773 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-06-29T14:22:50.773 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-06-29T14:22:50.774 INFO:teuthology.orchestra.run.smithi081.stdout:55834574857 2024-06-29T14:22:50.774 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-06-29T14:22:50.775 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:50.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-06-29T14:22:50.776 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:50.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-29T14:22:51.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.017+0000 7fd1013f0640 1 -- 172.21.15.81:0/322121022 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0fc0acfc0 msgr2=0x7fd0fc0ad3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.017+0000 7fd1013f0640 1 --2- 172.21.15.81:0/322121022 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0fc0acfc0 0x7fd0fc0ad3d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4005fd0 tx=0x7fd0f402aa10 comp rx=0 tx=0).stop 2024-06-29T14:22:51.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.017+0000 7fd1013f0640 1 -- 172.21.15.81:0/322121022 shutdown_connections 2024-06-29T14:22:51.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.017+0000 7fd1013f0640 1 --2- 172.21.15.81:0/322121022 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0fc0acfc0 0x7fd0fc0ad3d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.017+0000 7fd1013f0640 1 --2- 172.21.15.81:0/322121022 >> v2:172.21.15.167:3300/0 conn(0x7fd0fc15afa0 0x7fd0fc0ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.017+0000 7fd1013f0640 1 -- 172.21.15.81:0/322121022 >> 172.21.15.81:0/322121022 conn(0x7fd0fc0b1ec0 msgr2=0x7fd0fc0b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.018+0000 7fd1013f0640 1 -- 172.21.15.81:0/322121022 shutdown_connections 2024-06-29T14:22:51.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.018+0000 7fd1013f0640 1 -- 172.21.15.81:0/322121022 wait complete. 2024-06-29T14:22:51.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.019+0000 7f2aa3874640 1 -- 172.21.15.81:0/2380816959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a9401dd90 msgr2=0x7f2a94053830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.019+0000 7f2aa3874640 1 --2- 172.21.15.81:0/2380816959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a9401dd90 0x7f2a94053830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2a9401e290 tx=0x7f2a8c02a460 comp rx=0 tx=0).stop 2024-06-29T14:22:51.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.019+0000 7f079a4b9640 1 -- 172.21.15.81:0/3465990465 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07940d0340 msgr2=0x7f07940b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.019+0000 7f079a4b9640 1 --2- 172.21.15.81:0/3465990465 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07940d0340 0x7f07940b4980 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f078c005fd0 tx=0x7f078c033130 comp rx=0 tx=0).stop 2024-06-29T14:22:51.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.020+0000 7f079a4b9640 1 -- 172.21.15.81:0/3465990465 shutdown_connections 2024-06-29T14:22:51.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.020+0000 7f079a4b9640 1 --2- 172.21.15.81:0/3465990465 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07940d0340 0x7f07940b4980 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.020+0000 7f2aa3874640 1 -- 172.21.15.81:0/2380816959 shutdown_connections 2024-06-29T14:22:51.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.020+0000 7f2aa3874640 1 --2- 172.21.15.81:0/2380816959 >> v2:172.21.15.167:3300/0 conn(0x7f2a94006010 0x7f2a94006420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.020+0000 7f2aa3874640 1 --2- 172.21.15.81:0/2380816959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a9401dd90 0x7f2a94053830 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.020+0000 7f079a4b9640 1 --2- 172.21.15.81:0/3465990465 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07940b0350 0x7f07940cfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.020+0000 7f079a4b9640 1 -- 172.21.15.81:0/3465990465 >> 172.21.15.81:0/3465990465 conn(0x7f07940b3680 msgr2=0x7f07940b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.020+0000 7f079a4b9640 1 -- 172.21.15.81:0/3465990465 shutdown_connections 2024-06-29T14:22:51.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.020+0000 7f2aa3874640 1 -- 172.21.15.81:0/2380816959 >> 172.21.15.81:0/2380816959 conn(0x7f2a940362c0 msgr2=0x7f2a940366d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.020+0000 7f079a4b9640 1 -- 172.21.15.81:0/3465990465 wait complete. 2024-06-29T14:22:51.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.021+0000 7fd1013f0640 1 Processor -- start 2024-06-29T14:22:51.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.021+0000 7fd1013f0640 1 -- start start 2024-06-29T14:22:51.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.021+0000 7f2aa3874640 1 -- 172.21.15.81:0/2380816959 shutdown_connections 2024-06-29T14:22:51.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.021+0000 7f2aa3874640 1 -- 172.21.15.81:0/2380816959 wait complete. 2024-06-29T14:22:51.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.022+0000 7f7721e46640 1 -- 172.21.15.81:0/2576233396 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7714006750 msgr2=0x7f771401dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.022+0000 7f7721e46640 1 --2- 172.21.15.81:0/2576233396 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7714006750 0x7f771401dbe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f771c051d70 tx=0x7f771c0a5420 comp rx=0 tx=0).stop 2024-06-29T14:22:51.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.022+0000 7f7721e46640 1 -- 172.21.15.81:0/2576233396 shutdown_connections 2024-06-29T14:22:51.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.022+0000 7f7721e46640 1 --2- 172.21.15.81:0/2576233396 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7714006750 0x7f771401dbe0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.022+0000 7f7721e46640 1 -- 172.21.15.81:0/2576233396 >> 172.21.15.81:0/2576233396 conn(0x7f77140362f0 msgr2=0x7f7714036700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.022+0000 7f7721e46640 1 -- 172.21.15.81:0/2576233396 shutdown_connections 2024-06-29T14:22:51.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.022+0000 7f7721e46640 1 -- 172.21.15.81:0/2576233396 wait complete. 2024-06-29T14:22:51.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.023+0000 7fd1013f0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0fc0744f0 0x7fd0fc1606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.023+0000 7fd1013f0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0fc0acfc0 0x7fd0fc0b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.023+0000 7f079a4b9640 1 Processor -- start 2024-06-29T14:22:51.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.023+0000 7fd0faffd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0fc0744f0 0x7fd0fc1606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7fd0faffd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0fc0744f0 0x7fd0fc1606f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:58204/0 (socket says 172.21.15.81:58204) 2024-06-29T14:22:51.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7fd0faffd640 1 -- 172.21.15.81:0/3846646159 learned_addr learned my addr 172.21.15.81:0/3846646159 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:51.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7f079a4b9640 1 -- start start 2024-06-29T14:22:51.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7fd0fa7fc640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0fc0acfc0 0x7fd0fc0b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7f2aa3874640 1 Processor -- start 2024-06-29T14:22:51.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7fd1013f0640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc15afa0 0x7fd0fc1556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7fd1013f0640 1 -- 172.21.15.81:0/3846646159 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc0739a0 con 0x7fd0fc15afa0 2024-06-29T14:22:51.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7fd1013f0640 1 -- 172.21.15.81:0/3846646159 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc073820 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7fd1013f0640 1 -- 172.21.15.81:0/3846646159 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd0fc073b20 con 0x7fd0fc0744f0 2024-06-29T14:22:51.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7fd0fb7fe640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc15afa0 0x7fd0fc1556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.023+0000 7fa896519640 1 -- 172.21.15.81:0/3625889511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa89006fa60 msgr2=0x7fa89015a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.023+0000 7fa896519640 1 --2- 172.21.15.81:0/3625889511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa89006fa60 0x7fa89015a400 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa878005fd0 tx=0x7fa87802b9c0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.023+0000 7fa896519640 1 -- 172.21.15.81:0/3625889511 shutdown_connections 2024-06-29T14:22:51.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.023+0000 7fa896519640 1 --2- 172.21.15.81:0/3625889511 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8900aec10 0x7fa8900cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.023+0000 7fa896519640 1 --2- 172.21.15.81:0/3625889511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa89006fa60 0x7fa89015a400 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa878005fd0 tx=0x7fa87802b9c0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.023+0000 7fa896519640 1 -- 172.21.15.81:0/3625889511 >> 172.21.15.81:0/3625889511 conn(0x7fa8900b1e40 msgr2=0x7fa89006c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7fa896519640 1 -- 172.21.15.81:0/3625889511 shutdown_connections 2024-06-29T14:22:51.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7fa896519640 1 -- 172.21.15.81:0/3625889511 wait complete. 2024-06-29T14:22:51.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.024+0000 7f2aa3874640 1 -- start start 2024-06-29T14:22:51.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.025+0000 7fd0fa7fc640 1 -- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0fc0744f0 msgr2=0x7fd0fc1606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.025+0000 7fd0fa7fc640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0fc0744f0 0x7fd0fc1606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.025+0000 7fd0fa7fc640 1 -- 172.21.15.81:0/3846646159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc15afa0 msgr2=0x7fd0fc1556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.025+0000 7fd0fa7fc640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc15afa0 0x7fd0fc1556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.025+0000 7fd0fa7fc640 1 -- 172.21.15.81:0/3846646159 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0fc0b4d40 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.025+0000 7fd0fb7fe640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc15afa0 0x7fd0fc1556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:51.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.025+0000 7fd0fa7fc640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0fc0acfc0 0x7fd0fc0b3ce0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc15b3e0 tx=0x7fd0f000c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.025+0000 7f7721e46640 1 Processor -- start 2024-06-29T14:22:51.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f7721e46640 1 -- start start 2024-06-29T14:22:51.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f079a4b9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f079406e840 0x7f07940cc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f079a4b9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07940b0350 0x7f0794151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f079a4b9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07940d0340 0x7f0794155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f079a4b9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f07940b5040 con 0x7f07940d0340 2024-06-29T14:22:51.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f079a4b9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f07940b4ec0 con 0x7f07940b0350 2024-06-29T14:22:51.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f079a4b9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f07940b51c0 con 0x7f079406e840 2024-06-29T14:22:51.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f0793fff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f079406e840 0x7f07940cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f07937fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07940b0350 0x7f0794151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f0798a56640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07940d0340 0x7f0794155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f0798a56640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07940d0340 0x7f0794155f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36140/0 (socket says 172.21.15.81:36140) 2024-06-29T14:22:51.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f0798a56640 1 -- 172.21.15.81:0/18933833 learned_addr learned my addr 172.21.15.81:0/18933833 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:51.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f2aa3874640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a94006010 0x7f2a94182310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.025+0000 7fd0e37fe640 1 -- 172.21.15.81:0/3846646159 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0017070 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7fd0e37fe640 1 -- 172.21.15.81:0/3846646159 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0f00053e0 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7fd0e37fe640 1 -- 172.21.15.81:0/3846646159 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0003870 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7fd1013f0640 1 -- 172.21.15.81:0/3846646159 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0fc0b44a0 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7fd1013f0640 1 -- 172.21.15.81:0/3846646159 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0fc0b4a60 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7ff199047640 1 -- 172.21.15.81:0/316591996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff194161a30 msgr2=0x7ff1940b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7ff199047640 1 --2- 172.21.15.81:0/316591996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff194161a30 0x7ff1940b4ec0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff188009830 tx=0x7ff18802a510 comp rx=0 tx=0).stop 2024-06-29T14:22:51.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7fa896519640 1 Processor -- start 2024-06-29T14:22:51.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7fa896519640 1 -- start start 2024-06-29T14:22:51.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f324a77e640 1 -- 172.21.15.81:0/3023846442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f323c00af70 msgr2=0x7f323c0374f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f324a77e640 1 --2- 172.21.15.81:0/3023846442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f323c00af70 0x7f323c0374f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f323c037f60 tx=0x7f323802a8c0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f324a77e640 1 -- 172.21.15.81:0/3023846442 shutdown_connections 2024-06-29T14:22:51.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f324a77e640 1 --2- 172.21.15.81:0/3023846442 >> v2:172.21.15.167:3300/0 conn(0x7f323c0e13d0 0x7f323c0e17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f324a77e640 1 --2- 172.21.15.81:0/3023846442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f323c00af70 0x7f323c0374f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f324a77e640 1 -- 172.21.15.81:0/3023846442 >> 172.21.15.81:0/3023846442 conn(0x7f323c01e3f0 msgr2=0x7f323c039e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f324a77e640 1 -- 172.21.15.81:0/3023846442 shutdown_connections 2024-06-29T14:22:51.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f324a77e640 1 -- 172.21.15.81:0/3023846442 wait complete. 2024-06-29T14:22:51.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.026+0000 7f6126ffd640 1 -- 172.21.15.81:0/2061111844 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6118004a00 con 0x7f612806ecc0 2024-06-29T14:22:51.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f612d52a640 1 -- 172.21.15.81:0/2061111844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f612806ecc0 msgr2=0x7f612806f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f612d52a640 1 --2- 172.21.15.81:0/2061111844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f612806ecc0 0x7f612806f0d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f612806f7c0 tx=0x7f611802ae20 comp rx=0 tx=0).stop 2024-06-29T14:22:51.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f612d52a640 1 -- 172.21.15.81:0/2061111844 shutdown_connections 2024-06-29T14:22:51.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f612d52a640 1 --2- 172.21.15.81:0/2061111844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f612806ecc0 0x7f612806f0d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f612d52a640 1 --2- 172.21.15.81:0/2061111844 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61280728d0 0x7f612815adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f612d52a640 1 --2- 172.21.15.81:0/2061111844 >> v2:172.21.15.167:3300/0 conn(0x7f61280737c0 0x7f6128154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f612d52a640 1 -- 172.21.15.81:0/2061111844 >> 172.21.15.81:0/2061111844 conn(0x7f612806c920 msgr2=0x7f61280b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f612d52a640 1 -- 172.21.15.81:0/2061111844 shutdown_connections 2024-06-29T14:22:51.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f612d52a640 1 -- 172.21.15.81:0/2061111844 wait complete. 2024-06-29T14:22:51.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f0798a56640 1 -- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f079406e840 msgr2=0x7f07940cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f0798a56640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f079406e840 0x7f07940cc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f0798a56640 1 -- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07940b0350 msgr2=0x7f0794151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f0798a56640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07940b0350 0x7f0794151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f0798a56640 1 -- 172.21.15.81:0/18933833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0794160310 con 0x7f07940d0340 2024-06-29T14:22:51.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f0798a56640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07940d0340 0x7f0794155f00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f078c009cb0 tx=0x7f078c013bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7f0790ff9640 1 -- 172.21.15.81:0/18933833 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c049020 con 0x7f07940d0340 2024-06-29T14:22:51.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7f0790ff9640 1 -- 172.21.15.81:0/18933833 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f078c004660 con 0x7f07940d0340 2024-06-29T14:22:51.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7f0790ff9640 1 -- 172.21.15.81:0/18933833 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c040040 con 0x7f07940d0340 2024-06-29T14:22:51.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7f079a4b9640 1 -- 172.21.15.81:0/18933833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07941605b0 con 0x7f07940d0340 2024-06-29T14:22:51.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7f079a4b9640 1 -- 172.21.15.81:0/18933833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0794160b70 con 0x7f07940d0340 2024-06-29T14:22:51.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f2aa3874640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a9401dd90 0x7f2a94048bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f2aa3874640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a940dc400 0x7f2a940dae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f2aa3874640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2a940e03c0 con 0x7f2a94006010 2024-06-29T14:22:51.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f2aa0e0f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a9401dd90 0x7f2a94048bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f2aa0e0f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a9401dd90 0x7f2a94048bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:58222/0 (socket says 172.21.15.81:58222) 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f2aa0e0f640 1 -- 172.21.15.81:0/3938202259 learned_addr learned my addr 172.21.15.81:0/3938202259 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f2aa1610640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a94006010 0x7f2a94182310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f2aa1e11640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a940dc400 0x7f2a940dae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2a940e0240 con 0x7f2a940dc400 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.027+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2a94006960 con 0x7f2a9401dd90 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7f2aa1610640 1 -- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a9401dd90 msgr2=0x7f2a94048bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7f2aa1610640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a9401dd90 0x7f2a94048bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7f2aa1610640 1 -- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a940dc400 msgr2=0x7f2a940dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7f2aa1610640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a940dc400 0x7f2a940dae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7f2aa1610640 1 -- 172.21.15.81:0/3938202259 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a94182850 con 0x7f2a94006010 2024-06-29T14:22:51.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7f2aa1610640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a94006010 0x7f2a94182310 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c031000 tx=0x7f2a8c005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7fd1013f0640 1 -- 172.21.15.81:0/3846646159 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0fc070170 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7ff199047640 1 -- 172.21.15.81:0/316591996 shutdown_connections 2024-06-29T14:22:51.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7ff199047640 1 --2- 172.21.15.81:0/316591996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff194161a30 0x7ff1940b4ec0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7ff199047640 1 -- 172.21.15.81:0/316591996 >> 172.21.15.81:0/316591996 conn(0x7ff1940cc920 msgr2=0x7ff1940ccd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7ff199047640 1 -- 172.21.15.81:0/316591996 shutdown_connections 2024-06-29T14:22:51.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.028+0000 7ff199047640 1 -- 172.21.15.81:0/316591996 wait complete. 2024-06-29T14:22:51.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7fa896519640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa890073410 0x7fa890152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7fa896519640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8900aec10 0x7fa8901f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7fa896519640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa890155190 0x7fa8901f93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7fa88ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa890073410 0x7fa890152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7fa896519640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa89006eff0 con 0x7fa890073410 2024-06-29T14:22:51.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7fa896519640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa89006ee70 con 0x7fa8900aec10 2024-06-29T14:22:51.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7fa88ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa890073410 0x7fa890152b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36154/0 (socket says 172.21.15.81:36154) 2024-06-29T14:22:51.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7fa896519640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa89006f170 con 0x7fa890155190 2024-06-29T14:22:51.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7fa88ffff640 1 -- 172.21.15.81:0/1961746120 learned_addr learned my addr 172.21.15.81:0/1961746120 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:51.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7f7721e46640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7714006750 0x7f77140d65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7f7721e46640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f771400bdf0 0x7f77140dd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7f7721e46640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7714053860 0x7f77140e2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7f7721e46640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f77140e0270 con 0x7f7714053860 2024-06-29T14:22:51.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7f7721e46640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f77140e03f0 con 0x7f7714006750 2024-06-29T14:22:51.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7f7721e46640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f77140dcb40 con 0x7f771400bdf0 2024-06-29T14:22:51.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7fa88f7fe640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8900aec10 0x7fa8901f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7fa894ab6640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa890155190 0x7fa8901f93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7f612d52a640 1 Processor -- start 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7f771b7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7714006750 0x7f77140d65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7f771b7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7714006750 0x7f77140d65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46484/0 (socket says 172.21.15.81:46484) 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7f771b7fe640 1 -- 172.21.15.81:0/1259542761 learned_addr learned my addr 172.21.15.81:0/1259542761 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7f771affd640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f771400bdf0 0x7f77140dd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7f324a77e640 1 Processor -- start 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.029+0000 7f324a77e640 1 -- start start 2024-06-29T14:22:51.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7f612d52a640 1 -- start start 2024-06-29T14:22:51.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.030+0000 7fd0e37fe640 1 -- 172.21.15.81:0/3846646159 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd0f001a070 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f771b7fe640 1 -- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f771400bdf0 msgr2=0x7f77140dd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f771b7fe640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f771400bdf0 0x7f77140dd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f771b7fe640 1 -- 172.21.15.81:0/1259542761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7714053860 msgr2=0x7f77140e2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f771b7fe640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7714053860 0x7f77140e2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7fa88f7fe640 1 -- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa890155190 msgr2=0x7fa8901f93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7fa88f7fe640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa890155190 0x7fa8901f93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7fa88f7fe640 1 -- 172.21.15.81:0/1961746120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa890073410 msgr2=0x7fa890152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7fa88f7fe640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa890073410 0x7fa890152b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7fa88f7fe640 1 -- 172.21.15.81:0/1961746120 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8901530c0 con 0x7fa8900aec10 2024-06-29T14:22:51.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f771b7fe640 1 -- 172.21.15.81:0/1259542761 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77140d6af0 con 0x7f7714006750 2024-06-29T14:22:51.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f771b7fe640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7714006750 0x7f77140d65b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f771000c430 tx=0x7f771000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7ff199047640 1 Processor -- start 2024-06-29T14:22:51.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7fd0e37fe640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd0e80763d0 0x7fd0e80788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7fa88f7fe640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8900aec10 0x7fa8901f8ea0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa878009000 tx=0x7fa878039590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7fd0e37fe640 1 -- 172.21.15.81:0/3846646159 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd0f00969c0 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f2a91ffb640 1 -- 172.21.15.81:0/3938202259 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c004250 con 0x7f2a94006010 2024-06-29T14:22:51.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f2a91ffb640 1 -- 172.21.15.81:0/3938202259 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a8c0043f0 con 0x7f2a94006010 2024-06-29T14:22:51.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a94048410 con 0x7f2a94006010 2024-06-29T14:22:51.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7fd0faffd640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd0e80763d0 0x7fd0e80788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7ff199047640 1 -- start start 2024-06-29T14:22:51.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f324a77e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f323c0dfd30 0x7f323c052030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f324a77e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f323c0e13d0 0x7f323c04b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.031+0000 7f324a77e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f323c049840 0x7f323c0e1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f324a77e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f323c006170 con 0x7f323c0dfd30 2024-06-29T14:22:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f324a77e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f323c005ff0 con 0x7f323c0e13d0 2024-06-29T14:22:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f324a77e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f323c0062f0 con 0x7f323c049840 2024-06-29T14:22:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f3248f7b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f323c0e13d0 0x7f323c04b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f3248f7b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f323c0e13d0 0x7f323c04b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46492/0 (socket says 172.21.15.81:46492) 2024-06-29T14:22:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f3248f7b640 1 -- 172.21.15.81:0/1115964871 learned_addr learned my addr 172.21.15.81:0/1115964871 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f3249f7d640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f323c049840 0x7f323c0e1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f324977c640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f323c0dfd30 0x7f323c052030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f0790ff9640 1 -- 172.21.15.81:0/18933833 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f078c04c020 con 0x7f07940d0340 2024-06-29T14:22:51.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f2a91ffb640 1 -- 172.21.15.81:0/3938202259 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c031bb0 con 0x7f2a94006010 2024-06-29T14:22:51.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a94038950 con 0x7f2a94006010 2024-06-29T14:22:51.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7fd0faffd640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd0e80763d0 0x7fd0e80788e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009bf0 tx=0x7fd0ec005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f079a4b9640 1 -- 172.21.15.81:0/18933833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0774005290 con 0x7f07940d0340 2024-06-29T14:22:51.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f3248f7b640 1 -- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f323c049840 msgr2=0x7f323c0e1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f3248f7b640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f323c049840 0x7f323c0e1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f3248f7b640 1 -- 172.21.15.81:0/1115964871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f323c0dfd30 msgr2=0x7f323c052030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f3248f7b640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f323c0dfd30 0x7f323c052030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f3248f7b640 1 -- 172.21.15.81:0/1115964871 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323c04b9d0 con 0x7f323c0e13d0 2024-06-29T14:22:51.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.032+0000 7f3249f7d640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f323c049840 0x7f323c0e1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:51.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f3248f7b640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f323c0e13d0 0x7f323c04b0f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f323800ac90 tx=0x7f3238034f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f0790ff9640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0780076320 0x7f0780078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f0790ff9640 1 -- 172.21.15.81:0/18933833 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f078c0c0600 con 0x7f07940d0340 2024-06-29T14:22:51.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f612d52a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f612806ecc0 0x7f6128149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f612d52a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61280728d0 0x7f6128148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f612d52a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61280737c0 0x7f6128148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f612d52a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61280b2660 con 0x7f612806ecc0 2024-06-29T14:22:51.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f612d52a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f612806ff50 con 0x7f61280737c0 2024-06-29T14:22:51.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f612d52a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f61280b27e0 con 0x7f61280728d0 2024-06-29T14:22:51.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f6127fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f612806ecc0 0x7f6128149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f6127fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f612806ecc0 0x7f6128149ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36174/0 (socket says 172.21.15.81:36174) 2024-06-29T14:22:51.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f6127fff640 1 -- 172.21.15.81:0/831386604 learned_addr learned my addr 172.21.15.81:0/831386604 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:51.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f61277fe640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61280728d0 0x7f6128148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f612cd29640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61280737c0 0x7f6128148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f61277fe640 1 -- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61280737c0 msgr2=0x7f6128148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f61277fe640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61280737c0 0x7f6128148a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f61277fe640 1 -- 172.21.15.81:0/831386604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f612806ecc0 msgr2=0x7f6128149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f61277fe640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f612806ecc0 0x7f6128149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.033+0000 7f61277fe640 1 -- 172.21.15.81:0/831386604 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6128152d60 con 0x7f61280728d0 2024-06-29T14:22:51.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7f0793fff640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0780076320 0x7f0780078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff199047640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff19406d300 0x7ff1940c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff199047640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1940cdfe0 0x7ff19415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff199047640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff194161a30 0x7ff1941fcb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a9400bdf0 con 0x7f2a94006010 2024-06-29T14:22:51.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff199047640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff1940cec80 con 0x7ff194161a30 2024-06-29T14:22:51.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff199047640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff1940ceb00 con 0x7ff1940cdfe0 2024-06-29T14:22:51.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff199047640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff1940af710 con 0x7ff19406d300 2024-06-29T14:22:51.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7f61277fe640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61280728d0 0x7f6128148510 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f61280cc9a0 tx=0x7f611c00b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff192d9d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff19406d300 0x7ff1940c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff19359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff194161a30 0x7ff1941fcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff192d9d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff19406d300 0x7ff1940c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:58266/0 (socket says 172.21.15.81:58266) 2024-06-29T14:22:51.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff192d9d640 1 -- 172.21.15.81:0/2929000322 learned_addr learned my addr 172.21.15.81:0/2929000322 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:51.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff19259c640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1940cdfe0 0x7ff19415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7f6124ff9640 1 -- 172.21.15.81:0/831386604 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c015020 con 0x7f61280728d0 2024-06-29T14:22:51.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7f6124ff9640 1 -- 172.21.15.81:0/831386604 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f611c011920 con 0x7f61280728d0 2024-06-29T14:22:51.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7f612d52a640 1 -- 172.21.15.81:0/831386604 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6128153040 con 0x7f61280728d0 2024-06-29T14:22:51.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7f0793fff640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0780076320 0x7f0780078830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f078400a730 tx=0x7f0784008040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff19359e640 1 -- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff19406d300 msgr2=0x7ff1940c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff19359e640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff19406d300 0x7ff1940c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff19359e640 1 -- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1940cdfe0 msgr2=0x7ff19415c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff19359e640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1940cdfe0 0x7ff19415c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff19359e640 1 -- 172.21.15.81:0/2929000322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1940c50a0 con 0x7ff194161a30 2024-06-29T14:22:51.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7f612d52a640 1 -- 172.21.15.81:0/831386604 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6128153600 con 0x7f61280728d0 2024-06-29T14:22:51.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7f6124ff9640 1 -- 172.21.15.81:0/831386604 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c011d70 con 0x7f61280728d0 2024-06-29T14:22:51.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.034+0000 7ff19359e640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff194161a30 0x7ff1941fcb30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff18400b3e0 tx=0x7ff18400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7f612d52a640 1 -- 172.21.15.81:0/831386604 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6108005290 con 0x7f61280728d0 2024-06-29T14:22:51.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7f7707fff640 1 -- 172.21.15.81:0/1259542761 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710017070 con 0x7f7714006750 2024-06-29T14:22:51.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7f3235ffb640 1 -- 172.21.15.81:0/1115964871 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3238032d50 con 0x7f323c0e13d0 2024-06-29T14:22:51.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7f7707fff640 1 -- 172.21.15.81:0/1259542761 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7710005320 con 0x7f7714006750 2024-06-29T14:22:51.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7f7721e46640 1 -- 172.21.15.81:0/1259542761 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771404bd90 con 0x7f7714006750 2024-06-29T14:22:51.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7f3235ffb640 1 -- 172.21.15.81:0/1115964871 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3238005020 con 0x7f323c0e13d0 2024-06-29T14:22:51.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7f3235ffb640 1 -- 172.21.15.81:0/1115964871 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3238005450 con 0x7f323c0e13d0 2024-06-29T14:22:51.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7f324a77e640 1 -- 172.21.15.81:0/1115964871 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f323c0d4240 con 0x7f323c0e13d0 2024-06-29T14:22:51.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7f7721e46640 1 -- 172.21.15.81:0/1259542761 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f771404c350 con 0x7f7714006750 2024-06-29T14:22:51.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7f6124ff9640 1 -- 172.21.15.81:0/831386604 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f611c018020 con 0x7f61280728d0 2024-06-29T14:22:51.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7fa88cff9640 1 -- 172.21.15.81:0/1961746120 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878039b90 con 0x7fa8900aec10 2024-06-29T14:22:51.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7fa88cff9640 1 -- 172.21.15.81:0/1961746120 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa878039e40 con 0x7fa8900aec10 2024-06-29T14:22:51.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7fa88cff9640 1 -- 172.21.15.81:0/1961746120 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87803b680 con 0x7fa8900aec10 2024-06-29T14:22:51.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7fa896519640 1 -- 172.21.15.81:0/1961746120 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8901f9c20 con 0x7fa8900aec10 2024-06-29T14:22:51.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.037+0000 7f324a77e640 1 -- 172.21.15.81:0/1115964871 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f323c0d4800 con 0x7f323c0e13d0 2024-06-29T14:22:51.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.037+0000 7fa896519640 1 -- 172.21.15.81:0/1961746120 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa890161f70 con 0x7fa8900aec10 2024-06-29T14:22:51.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.037+0000 7f6124ff9640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6114076300 0x7f6114078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.037+0000 7f6127fff640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6114076300 0x7f6114078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.037+0000 7f6124ff9640 1 -- 172.21.15.81:0/831386604 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f611c0973d0 con 0x7f61280728d0 2024-06-29T14:22:51.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.038+0000 7f6127fff640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6114076300 0x7f6114078810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6120009fd0 tx=0x7f612000b040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.038+0000 7f7707fff640 1 -- 172.21.15.81:0/1259542761 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710004840 con 0x7f7714006750 2024-06-29T14:22:51.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.038+0000 7f324a77e640 1 -- 172.21.15.81:0/1115964871 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f323c031df0 con 0x7f323c0e13d0 2024-06-29T14:22:51.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.038+0000 7fa8867fc640 1 -- 172.21.15.81:0/1961746120 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa86c005290 con 0x7fa8900aec10 2024-06-29T14:22:51.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.036+0000 7fd0e37fe640 1 -- 172.21.15.81:0/3846646159 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd0fc070170 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.039+0000 7f7721e46640 1 -- 172.21.15.81:0/1259542761 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76fc005290 con 0x7f7714006750 2024-06-29T14:22:51.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.039+0000 7f3235ffb640 1 -- 172.21.15.81:0/1115964871 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3238032630 con 0x7f323c0e13d0 2024-06-29T14:22:51.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.039+0000 7f3235ffb640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3228076300 0x7f3228078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.039+0000 7f3235ffb640 1 -- 172.21.15.81:0/1115964871 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f32380b87a0 con 0x7f323c0e13d0 2024-06-29T14:22:51.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.040+0000 7f324977c640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3228076300 0x7f3228078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.040+0000 7f7707fff640 1 -- 172.21.15.81:0/1259542761 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f771001a070 con 0x7f7714006750 2024-06-29T14:22:51.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.040+0000 7fa88cff9640 1 -- 172.21.15.81:0/1961746120 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa878043020 con 0x7fa8900aec10 2024-06-29T14:22:51.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.040+0000 7f7707fff640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f77000762d0 0x7f77000787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.040+0000 7f7707fff640 1 -- 172.21.15.81:0/1259542761 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7710096bd0 con 0x7f7714006750 2024-06-29T14:22:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.040+0000 7f771affd640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f77000762d0 0x7f77000787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.041+0000 7f0790ff9640 1 -- 172.21.15.81:0/18933833 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f078c053030 con 0x7f07940d0340 2024-06-29T14:22:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.040+0000 7fa88cff9640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa8740763a0 0x7fa8740788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.041+0000 7fa88cff9640 1 -- 172.21.15.81:0/1961746120 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa8780b8cd0 con 0x7fa8900aec10 2024-06-29T14:22:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.041+0000 7f324977c640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3228076300 0x7f3228078810 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f323c00e040 tx=0x7f3240009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.041+0000 7fa88ffff640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa8740763a0 0x7fa8740788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.041+0000 7ff17b7fe640 1 -- 172.21.15.81:0/2929000322 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff184015020 con 0x7ff194161a30 2024-06-29T14:22:51.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.041+0000 7ff199047640 1 -- 172.21.15.81:0/2929000322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1940c5380 con 0x7ff194161a30 2024-06-29T14:22:51.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.041+0000 7ff199047640 1 -- 172.21.15.81:0/2929000322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19409b940 con 0x7ff194161a30 2024-06-29T14:22:51.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.041+0000 7ff17b7fe640 1 -- 172.21.15.81:0/2929000322 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff184011830 con 0x7ff194161a30 2024-06-29T14:22:51.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.041+0000 7ff17b7fe640 1 -- 172.21.15.81:0/2929000322 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff184011c40 con 0x7ff194161a30 2024-06-29T14:22:51.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.042+0000 7fa88ffff640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa8740763a0 0x7fa8740788b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa8880019b0 tx=0x7fa888000bc0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.042+0000 7f771affd640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f77000762d0 0x7f77000787e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7714006d50 tx=0x7f771c0a7040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.043+0000 7f6124ff9640 1 -- 172.21.15.81:0/831386604 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f611c01f020 con 0x7f61280728d0 2024-06-29T14:22:51.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.043+0000 7ff199047640 1 -- 172.21.15.81:0/2929000322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff174005290 con 0x7ff194161a30 2024-06-29T14:22:51.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.043+0000 7f2a91ffb640 1 -- 172.21.15.81:0/3938202259 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2a8c0047a0 con 0x7f2a94006010 2024-06-29T14:22:51.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.044+0000 7f2a91ffb640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a800763a0 0x7f2a800788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.044+0000 7f2aa0e0f640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a800763a0 0x7f2a800788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.044+0000 7f2a91ffb640 1 -- 172.21.15.81:0/3938202259 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2a8c0c9c10 con 0x7f2a94006010 2024-06-29T14:22:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.044+0000 7f2aa0e0f640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a800763a0 0x7f2a800788b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2a94181860 tx=0x7f2a9800c040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.045+0000 7f3235ffb640 1 -- 172.21.15.81:0/1115964871 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3238040030 con 0x7f323c0e13d0 2024-06-29T14:22:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.047+0000 7f2a91ffb640 1 -- 172.21.15.81:0/3938202259 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2a8c063080 con 0x7f2a94006010 2024-06-29T14:22:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.050+0000 7ff17b7fe640 1 -- 172.21.15.81:0/2929000322 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff184018020 con 0x7ff194161a30 2024-06-29T14:22:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.051+0000 7ff17b7fe640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ff1800762d0 0x7ff1800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.051+0000 7ff17b7fe640 1 -- 172.21.15.81:0/2929000322 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff184097b20 con 0x7ff194161a30 2024-06-29T14:22:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.051+0000 7ff192d9d640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ff1800762d0 0x7ff1800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.052+0000 7fa88cff9640 1 -- 172.21.15.81:0/1961746120 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa8780be020 con 0x7fa8900aec10 2024-06-29T14:22:51.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.053+0000 7ff17b7fe640 1 -- 172.21.15.81:0/2929000322 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff18401f020 con 0x7ff194161a30 2024-06-29T14:22:51.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.055+0000 7f7707fff640 1 -- 172.21.15.81:0/1259542761 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f77100604a0 con 0x7f7714006750 2024-06-29T14:22:51.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.055+0000 7ff192d9d640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ff1800762d0 0x7ff1800787e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff1880095f0 tx=0x7ff188004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:51.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.473+0000 7fd1013f0640 1 -- 172.21.15.81:0/3846646159 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd0fc0b4220 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.474+0000 7fd0e37fe640 1 -- 172.21.15.81:0/3846646159 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd0fc0b4220 con 0x7fd0fc0acfc0 2024-06-29T14:22:51.473 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-06-29T14:22:51.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.477+0000 7fd0e17fa640 1 -- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd0e80763d0 msgr2=0x7fd0e80788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.477+0000 7f324a77e640 1 -- 172.21.15.81:0/1115964871 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f323c0d8f80 con 0x7f323c0e13d0 2024-06-29T14:22:51.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.477+0000 7fd0e17fa640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd0e80763d0 0x7fd0e80788e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009bf0 tx=0x7fd0ec005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:51.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7f079a4b9640 1 -- 172.21.15.81:0/18933833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f0774005850 con 0x7f07940d0340 2024-06-29T14:22:51.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7fd0e17fa640 1 -- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0fc0acfc0 msgr2=0x7fd0fc0b3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7fd0e17fa640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0fc0acfc0 0x7fd0fc0b3ce0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc15b3e0 tx=0x7fd0f000c820 comp rx=0 tx=0).stop 2024-06-29T14:22:51.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7fd0e17fa640 1 -- 172.21.15.81:0/3846646159 shutdown_connections 2024-06-29T14:22:51.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7fd0e17fa640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd0e80763d0 0x7fd0e80788e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7fd0e17fa640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc15afa0 0x7fd0fc1556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7fd0e17fa640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0fc0acfc0 0x7fd0fc0b3ce0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7fd0e17fa640 1 --2- 172.21.15.81:0/3846646159 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0fc0744f0 0x7fd0fc1606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7fd0e17fa640 1 -- 172.21.15.81:0/3846646159 >> 172.21.15.81:0/3846646159 conn(0x7fd0fc0b1ec0 msgr2=0x7fd0fc157c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7fd0e17fa640 1 -- 172.21.15.81:0/3846646159 shutdown_connections 2024-06-29T14:22:51.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7fd0e17fa640 1 -- 172.21.15.81:0/3846646159 wait complete. 2024-06-29T14:22:51.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7f0790ff9640 1 -- 172.21.15.81:0/18933833 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f078c08a040 con 0x7f07940d0340 2024-06-29T14:22:51.479 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-06-29T14:22:51.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.478+0000 7f3235ffb640 1 -- 172.21.15.81:0/1115964871 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f323802f070 con 0x7f323c0e13d0 2024-06-29T14:22:51.480 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-06-29T14:22:51.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.480+0000 7f612d52a640 1 -- 172.21.15.81:0/831386604 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f6108005580 con 0x7f61280728d0 2024-06-29T14:22:51.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.480+0000 7f6124ff9640 1 -- 172.21.15.81:0/831386604 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f611c060640 con 0x7f61280728d0 2024-06-29T14:22:51.481 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-06-29T14:22:51.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.481+0000 7f079a4b9640 1 -- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0780076320 msgr2=0x7f0780078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.481+0000 7f079a4b9640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0780076320 0x7f0780078830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f078400a730 tx=0x7f0784008040 comp rx=0 tx=0).stop 2024-06-29T14:22:51.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.482+0000 7f079a4b9640 1 -- 172.21.15.81:0/18933833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07940d0340 msgr2=0x7f0794155f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.482+0000 7f079a4b9640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07940d0340 0x7f0794155f00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f078c009cb0 tx=0x7f078c013bc0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.482+0000 7f079a4b9640 1 -- 172.21.15.81:0/18933833 shutdown_connections 2024-06-29T14:22:51.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.482+0000 7f079a4b9640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0780076320 0x7f0780078830 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.482+0000 7f079a4b9640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07940d0340 0x7f0794155f00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.482+0000 7f079a4b9640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07940b0350 0x7f0794151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.482+0000 7f079a4b9640 1 --2- 172.21.15.81:0/18933833 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f079406e840 0x7f07940cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.482+0000 7f079a4b9640 1 -- 172.21.15.81:0/18933833 >> 172.21.15.81:0/18933833 conn(0x7f07940b3680 msgr2=0x7f07940b61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.482+0000 7f079a4b9640 1 -- 172.21.15.81:0/18933833 shutdown_connections 2024-06-29T14:22:51.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.482+0000 7f079a4b9640 1 -- 172.21.15.81:0/18933833 wait complete. 2024-06-29T14:22:51.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.483+0000 7f324a77e640 1 -- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3228076300 msgr2=0x7f3228078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.483+0000 7f324a77e640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3228076300 0x7f3228078810 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f323c00e040 tx=0x7f3240009040 comp rx=0 tx=0).stop 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.483+0000 7f324a77e640 1 -- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f323c0e13d0 msgr2=0x7f323c04b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.483+0000 7f324a77e640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f323c0e13d0 0x7f323c04b0f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f323800ac90 tx=0x7f3238034f00 comp rx=0 tx=0).stop 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.483+0000 7f324a77e640 1 -- 172.21.15.81:0/1115964871 shutdown_connections 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.483+0000 7f324a77e640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3228076300 0x7f3228078810 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.483+0000 7f324a77e640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f323c049840 0x7f323c0e1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.483+0000 7f324a77e640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f323c0e13d0 0x7f323c04b0f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.483+0000 7f324a77e640 1 --2- 172.21.15.81:0/1115964871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f323c0dfd30 0x7f323c052030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.483+0000 7f324a77e640 1 -- 172.21.15.81:0/1115964871 >> 172.21.15.81:0/1115964871 conn(0x7f323c01e3f0 msgr2=0x7f323c0d8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f324a77e640 1 -- 172.21.15.81:0/1115964871 shutdown_connections 2024-06-29T14:22:51.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f324a77e640 1 -- 172.21.15.81:0/1115964871 wait complete. 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f612d52a640 1 -- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6114076300 msgr2=0x7f6114078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f612d52a640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6114076300 0x7f6114078810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6120009fd0 tx=0x7f612000b040 comp rx=0 tx=0).stop 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f612d52a640 1 -- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61280728d0 msgr2=0x7f6128148510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f612d52a640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61280728d0 0x7f6128148510 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f61280cc9a0 tx=0x7f611c00b8f0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f612d52a640 1 -- 172.21.15.81:0/831386604 shutdown_connections 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f612d52a640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6114076300 0x7f6114078810 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f612d52a640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61280737c0 0x7f6128148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f612d52a640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61280728d0 0x7f6128148510 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.484+0000 7f612d52a640 1 --2- 172.21.15.81:0/831386604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f612806ecc0 0x7f6128149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.485+0000 7f612d52a640 1 -- 172.21.15.81:0/831386604 >> 172.21.15.81:0/831386604 conn(0x7f612806c920 msgr2=0x7f61280cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.485+0000 7f612d52a640 1 -- 172.21.15.81:0/831386604 shutdown_connections 2024-06-29T14:22:51.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.485+0000 7f612d52a640 1 -- 172.21.15.81:0/831386604 wait complete. 2024-06-29T14:22:51.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.487+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2a940e26b0 con 0x7f2a94006010 2024-06-29T14:22:51.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.488+0000 7f2a91ffb640 1 -- 172.21.15.81:0/3938202259 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2a94182850 con 0x7f2a94006010 2024-06-29T14:22:51.488 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-06-29T14:22:51.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.490+0000 7ff199047640 1 -- 172.21.15.81:0/2929000322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff174005850 con 0x7ff194161a30 2024-06-29T14:22:51.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.490+0000 7ff17b7fe640 1 -- 172.21.15.81:0/2929000322 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff184060e10 con 0x7ff194161a30 2024-06-29T14:22:51.489 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-06-29T14:22:51.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a800763a0 msgr2=0x7f2a800788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a800763a0 0x7f2a800788b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2a94181860 tx=0x7f2a9800c040 comp rx=0 tx=0).stop 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a94006010 msgr2=0x7f2a94182310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a94006010 0x7f2a94182310 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c031000 tx=0x7f2a8c005480 comp rx=0 tx=0).stop 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 shutdown_connections 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a800763a0 0x7f2a800788b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a940dc400 0x7f2a940dae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a9401dd90 0x7f2a94048bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 --2- 172.21.15.81:0/3938202259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a94006010 0x7f2a94182310 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 >> 172.21.15.81:0/3938202259 conn(0x7f2a940362c0 msgr2=0x7f2a94034230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 shutdown_connections 2024-06-29T14:22:51.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.492+0000 7f2aa3874640 1 -- 172.21.15.81:0/3938202259 wait complete. 2024-06-29T14:22:51.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.494+0000 7fa8867fc640 1 -- 172.21.15.81:0/1961746120 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa86c005580 con 0x7fa8900aec10 2024-06-29T14:22:51.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.494+0000 7fa88cff9640 1 -- 172.21.15.81:0/1961746120 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa878082050 con 0x7fa8900aec10 2024-06-29T14:22:51.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 -- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ff1800762d0 msgr2=0x7ff1800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ff1800762d0 0x7ff1800787e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff1880095f0 tx=0x7ff188004730 comp rx=0 tx=0).stop 2024-06-29T14:22:51.494 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-06-29T14:22:51.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 -- 172.21.15.81:0/2929000322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff194161a30 msgr2=0x7ff1941fcb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff194161a30 0x7ff1941fcb30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff18400b3e0 tx=0x7ff18400b8a0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7f7721e46640 1 -- 172.21.15.81:0/1259542761 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f76fc005850 con 0x7f7714006750 2024-06-29T14:22:51.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 -- 172.21.15.81:0/2929000322 shutdown_connections 2024-06-29T14:22:51.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ff1800762d0 0x7ff1800787e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff194161a30 0x7ff1941fcb30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1940cdfe0 0x7ff19415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 --2- 172.21.15.81:0/2929000322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff19406d300 0x7ff1940c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 -- 172.21.15.81:0/2929000322 >> 172.21.15.81:0/2929000322 conn(0x7ff1940cc920 msgr2=0x7ff19406d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 -- 172.21.15.81:0/2929000322 shutdown_connections 2024-06-29T14:22:51.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.495+0000 7ff199047640 1 -- 172.21.15.81:0/2929000322 wait complete. 2024-06-29T14:22:51.497 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:51.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.496+0000 7f7707fff640 1 -- 172.21.15.81:0/1259542761 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f771005fda0 con 0x7f7714006750 2024-06-29T14:22:51.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 -- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f77000762d0 msgr2=0x7f77000787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f77000762d0 0x7f77000787e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7714006d50 tx=0x7f771c0a7040 comp rx=0 tx=0).stop 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 -- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7714006750 msgr2=0x7f77140d65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7714006750 0x7f77140d65b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f771000c430 tx=0x7f771000c8f0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 -- 172.21.15.81:0/1259542761 shutdown_connections 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f77000762d0 0x7f77000787e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7714053860 0x7f77140e2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f771400bdf0 0x7f77140dd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 --2- 172.21.15.81:0/1259542761 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7714006750 0x7f77140d65b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 -- 172.21.15.81:0/1259542761 >> 172.21.15.81:0/1259542761 conn(0x7f77140362f0 msgr2=0x7f77140049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 -- 172.21.15.81:0/1259542761 shutdown_connections 2024-06-29T14:22:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.499+0000 7f7721e46640 1 -- 172.21.15.81:0/1259542761 wait complete. 2024-06-29T14:22:51.500 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-06-29T14:22:51.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 -- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa8740763a0 msgr2=0x7fa8740788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa8740763a0 0x7fa8740788b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa8880019b0 tx=0x7fa888000bc0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 -- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8900aec10 msgr2=0x7fa8901f8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8900aec10 0x7fa8901f8ea0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa878009000 tx=0x7fa878039590 comp rx=0 tx=0).stop 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 -- 172.21.15.81:0/1961746120 shutdown_connections 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa8740763a0 0x7fa8740788b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa890155190 0x7fa8901f93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8900aec10 0x7fa8901f8ea0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 --2- 172.21.15.81:0/1961746120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa890073410 0x7fa890152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 -- 172.21.15.81:0/1961746120 >> 172.21.15.81:0/1961746120 conn(0x7fa8900b1e40 msgr2=0x7fa89006dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 -- 172.21.15.81:0/1961746120 shutdown_connections 2024-06-29T14:22:51.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:51.500+0000 7fa8867fc640 1 -- 172.21.15.81:0/1961746120 wait complete. 2024-06-29T14:22:51.504 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2024-06-29T14:22:51.506 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2024-06-29T14:22:51.507 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2024-06-29T14:22:51.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.3 2024-06-29T14:22:51.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2024-06-29T14:22:51.520 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2024-06-29T14:22:51.520 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2024-06-29T14:22:52.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-29T14:22:52.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-29T14:22:52.507 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-06-29T14:22:52.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-29T14:22:52.514 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-29T14:22:52.517 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-29T14:22:52.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-06-29T14:22:52.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-06-29T14:22:52.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.686+0000 7f5fee8ac640 1 -- 172.21.15.81:0/2498807643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe8161c50 msgr2=0x7f5fe8162060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.686+0000 7f5fee8ac640 1 --2- 172.21.15.81:0/2498807643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe8161c50 0x7f5fe8162060 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8074000 tx=0x7f5fe003ba80 comp rx=0 tx=0).stop 2024-06-29T14:22:52.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.687+0000 7f5fee8ac640 1 -- 172.21.15.81:0/2498807643 shutdown_connections 2024-06-29T14:22:52.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.687+0000 7f5fee8ac640 1 --2- 172.21.15.81:0/2498807643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe8161c50 0x7f5fe8162060 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.687+0000 7f5fee8ac640 1 --2- 172.21.15.81:0/2498807643 >> v2:172.21.15.167:3300/0 conn(0x7f5fe8073160 0x7f5fe8073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.687+0000 7f5fee8ac640 1 --2- 172.21.15.81:0/2498807643 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe815e660 0x7f5fe806e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.687+0000 7f5fee8ac640 1 -- 172.21.15.81:0/2498807643 >> 172.21.15.81:0/2498807643 conn(0x7f5fe80b1910 msgr2=0x7f5fe80b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.688+0000 7f5fee8ac640 1 -- 172.21.15.81:0/2498807643 shutdown_connections 2024-06-29T14:22:52.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.688+0000 7f5fee8ac640 1 -- 172.21.15.81:0/2498807643 wait complete. 2024-06-29T14:22:52.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.693+0000 7f5fee8ac640 1 Processor -- start 2024-06-29T14:22:52.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.693+0000 7f5fee8ac640 1 -- start start 2024-06-29T14:22:52.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.692+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/3660018549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5acc0b4560 msgr2=0x7f5acc140220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.692+0000 7f5ad2b2b640 1 --2- 172.21.15.81:0/3660018549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5acc0b4560 0x7f5acc140220 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009880 tx=0x7f5ac402a530 comp rx=0 tx=0).stop 2024-06-29T14:22:52.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.694+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/3660018549 shutdown_connections 2024-06-29T14:22:52.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.694+0000 7f5ad2b2b640 1 --2- 172.21.15.81:0/3660018549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5acc0b4560 0x7f5acc140220 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.694+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/3660018549 >> 172.21.15.81:0/3660018549 conn(0x7f5acc0b1910 msgr2=0x7f5acc0b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.696+0000 7f5fee8ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe8073160 0x7f5fe8131db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.697+0000 7f5fe7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe8073160 0x7f5fe8131db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.697+0000 7f5fe7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe8073160 0x7f5fe8131db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36220/0 (socket says 172.21.15.81:36220) 2024-06-29T14:22:52.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.697+0000 7f5fe7fff640 1 -- 172.21.15.81:0/268784861 learned_addr learned my addr 172.21.15.81:0/268784861 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:52.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.697+0000 7f5fee8ac640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe815e660 0x7f5fe8156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.697+0000 7f5fee8ac640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fe8161c50 0x7f5fe8156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.697+0000 7f5fee8ac640 1 -- 172.21.15.81:0/268784861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe813fb10 con 0x7f5fe8073160 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.697+0000 7f5fee8ac640 1 -- 172.21.15.81:0/268784861 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe8162970 con 0x7f5fe8161c50 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.697+0000 7f5fece49640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fe8161c50 0x7f5fe8156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.697+0000 7f5fee8ac640 1 -- 172.21.15.81:0/268784861 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5fe813fc90 con 0x7f5fe815e660 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.697+0000 7f5fe77fe640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe815e660 0x7f5fe8156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.698+0000 7f5fe7fff640 1 -- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe815e660 msgr2=0x7f5fe8156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.698+0000 7f5fe7fff640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe815e660 0x7f5fe8156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.698+0000 7f5fe7fff640 1 -- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fe8161c50 msgr2=0x7f5fe8156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.698+0000 7f5fe7fff640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fe8161c50 0x7f5fe8156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.698+0000 7f5fe7fff640 1 -- 172.21.15.81:0/268784861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe81587d0 con 0x7f5fe8073160 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.699+0000 7f5fe7fff640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe8073160 0x7f5fe8131db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800ea20 tx=0x7f5fd800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.699+0000 7f5fe77fe640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe815e660 0x7f5fe8156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:52.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.696+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/3660018549 shutdown_connections 2024-06-29T14:22:52.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.696+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/3660018549 wait complete. 2024-06-29T14:22:52.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.699+0000 7f5ad2b2b640 1 Processor -- start 2024-06-29T14:22:52.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.702+0000 7f5fe4ff9640 1 -- 172.21.15.81:0/268784861 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd800cc30 con 0x7f5fe8073160 2024-06-29T14:22:52.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.702+0000 7f5fe4ff9640 1 -- 172.21.15.81:0/268784861 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5fd8010430 con 0x7f5fe8073160 2024-06-29T14:22:52.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.702+0000 7f5fe4ff9640 1 -- 172.21.15.81:0/268784861 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd8010860 con 0x7f5fe8073160 2024-06-29T14:22:52.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.702+0000 7f5fee8ac640 1 -- 172.21.15.81:0/268784861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe813f7e0 con 0x7f5fe8073160 2024-06-29T14:22:52.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.702+0000 7f5fee8ac640 1 -- 172.21.15.81:0/268784861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe80aaa40 con 0x7f5fe8073160 2024-06-29T14:22:52.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.700+0000 7f5ad2b2b640 1 -- start start 2024-06-29T14:22:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.702+0000 7f5ad2b2b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5acc06e0f0 0x7f5acc13f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.703+0000 7f5ad2b2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5acc0b4560 0x7f5acc13e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.703+0000 7f5ad2b2b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5acc15e660 0x7f5acc0ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.703+0000 7f5ad08c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5acc06e0f0 0x7f5acc13f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.703+0000 7f5ad08c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5acc06e0f0 0x7f5acc13f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46550/0 (socket says 172.21.15.81:46550) 2024-06-29T14:22:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.703+0000 7f5ad08c7640 1 -- 172.21.15.81:0/1482036536 learned_addr learned my addr 172.21.15.81:0/1482036536 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.703+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/1482036536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc140970 con 0x7f5acc0b4560 2024-06-29T14:22:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.703+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/1482036536 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5acc142df0 con 0x7f5acc06e0f0 2024-06-29T14:22:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.703+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/1482036536 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5acc0ab930 con 0x7f5acc15e660 2024-06-29T14:22:52.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.704+0000 7f5acbfff640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5acc0b4560 0x7f5acc13e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.705+0000 7f5acbfff640 1 -- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5acc15e660 msgr2=0x7f5acc0ad6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:22:52.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.705+0000 7f5acbfff640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5acc15e660 0x7f5acc0ad6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.706+0000 7f5acbfff640 1 -- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5acc06e0f0 msgr2=0x7f5acc13f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.706+0000 7f5acbfff640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5acc06e0f0 0x7f5acc13f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.709+0000 7f5fe4ff9640 1 -- 172.21.15.81:0/268784861 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5fd8010a00 con 0x7f5fe8073160 2024-06-29T14:22:52.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.706+0000 7f5acbfff640 1 -- 172.21.15.81:0/1482036536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5acc13f690 con 0x7f5acc0b4560 2024-06-29T14:22:52.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.707+0000 7f5ad08c7640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5acc06e0f0 0x7f5acc13f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:52.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.709+0000 7f5acbfff640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5acc0b4560 0x7f5acc13e9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00c700 tx=0x7f5abc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.710+0000 7f5ac97fa640 1 -- 172.21.15.81:0/1482036536 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc003b70 con 0x7f5acc0b4560 2024-06-29T14:22:52.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.710+0000 7f5ac97fa640 1 -- 172.21.15.81:0/1482036536 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5abc003d30 con 0x7f5acc0b4560 2024-06-29T14:22:52.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.710+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/1482036536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5acc15a100 con 0x7f5acc0b4560 2024-06-29T14:22:52.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.710+0000 7f5fe4ff9640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5fd4076320 0x7f5fd4078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.711+0000 7f5fe4ff9640 1 -- 172.21.15.81:0/268784861 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5fd8096730 con 0x7f5fe8073160 2024-06-29T14:22:52.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.710+0000 7f5fee8ac640 1 -- 172.21.15.81:0/268784861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fe806e0f0 con 0x7f5fe8073160 2024-06-29T14:22:52.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.711+0000 7f5fe77fe640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5fd4076320 0x7f5fd4078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.711+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/1482036536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5acc159b40 con 0x7f5acc0b4560 2024-06-29T14:22:52.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.712+0000 7f5ac97fa640 1 -- 172.21.15.81:0/1482036536 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc0052d0 con 0x7f5acc0b4560 2024-06-29T14:22:52.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.714+0000 7f5ac97fa640 1 -- 172.21.15.81:0/1482036536 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5abc005470 con 0x7f5acc0b4560 2024-06-29T14:22:52.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.715+0000 7f5ac97fa640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ab8076350 0x7f5ab8078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.717+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/1482036536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aac005290 con 0x7f5acc0b4560 2024-06-29T14:22:52.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.718+0000 7f5ad08c7640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ab8076350 0x7f5ab8078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.719+0000 7f5ad08c7640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ab8076350 0x7f5ab8078860 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00018f0 tx=0x7f5ac0009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.720+0000 7f5fe77fe640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5fd4076320 0x7f5fd4078830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8074790 tx=0x7f5fdc008040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.723+0000 7f5fe4ff9640 1 -- 172.21.15.81:0/268784861 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5fd8061400 con 0x7f5fe8073160 2024-06-29T14:22:52.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.723+0000 7f5ac97fa640 1 -- 172.21.15.81:0/1482036536 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5abc0016a0 con 0x7f5acc0b4560 2024-06-29T14:22:52.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.732+0000 7f5ac97fa640 1 -- 172.21.15.81:0/1482036536 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5abc020020 con 0x7f5acc0b4560 2024-06-29T14:22:52.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.758+0000 7fbb52573640 1 -- 172.21.15.81:0/954577722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb4c073450 msgr2=0x7fbb4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.758+0000 7fbb52573640 1 --2- 172.21.15.81:0/954577722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb4c073450 0x7fbb4c06edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009830 tx=0x7fbb4002a3b0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.758+0000 7fbb52573640 1 -- 172.21.15.81:0/954577722 shutdown_connections 2024-06-29T14:22:52.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.758+0000 7fbb52573640 1 --2- 172.21.15.81:0/954577722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb4c073450 0x7fbb4c06edb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.758+0000 7fbb52573640 1 --2- 172.21.15.81:0/954577722 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb4c0b59c0 0x7fbb4c072f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.758+0000 7fbb52573640 1 -- 172.21.15.81:0/954577722 >> 172.21.15.81:0/954577722 conn(0x7fbb4c0d0b70 msgr2=0x7fbb4c0d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.758+0000 7fbb52573640 1 -- 172.21.15.81:0/954577722 shutdown_connections 2024-06-29T14:22:52.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.759+0000 7fbb52573640 1 -- 172.21.15.81:0/954577722 wait complete. 2024-06-29T14:22:52.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.761+0000 7fc81759e640 1 -- 172.21.15.81:0/896154857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc81003a960 msgr2=0x7fc8100e6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.761+0000 7fc81759e640 1 --2- 172.21.15.81:0/896154857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc81003a960 0x7fc8100e6810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc808009820 tx=0x7fc80802a480 comp rx=0 tx=0).stop 2024-06-29T14:22:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.762+0000 7fc81759e640 1 -- 172.21.15.81:0/896154857 shutdown_connections 2024-06-29T14:22:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.762+0000 7fc81759e640 1 --2- 172.21.15.81:0/896154857 >> v2:172.21.15.167:3300/0 conn(0x7fc810052260 0x7fc810033e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.762+0000 7fc81759e640 1 --2- 172.21.15.81:0/896154857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc81003a960 0x7fc8100e6810 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.762+0000 7fc81759e640 1 -- 172.21.15.81:0/896154857 >> 172.21.15.81:0/896154857 conn(0x7fc81001e3f0 msgr2=0x7fc810039e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.762+0000 7fc81759e640 1 -- 172.21.15.81:0/896154857 shutdown_connections 2024-06-29T14:22:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.762+0000 7fc81759e640 1 -- 172.21.15.81:0/896154857 wait complete. 2024-06-29T14:22:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.762+0000 7fbb52573640 1 Processor -- start 2024-06-29T14:22:52.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.763+0000 7fbb52573640 1 -- start start 2024-06-29T14:22:52.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7fde2922d640 1 -- 172.21.15.81:0/1192429813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde240ced60 msgr2=0x7fde240cf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7fde2922d640 1 --2- 172.21.15.81:0/1192429813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde240ced60 0x7fde240cf170 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fde240d0eb0 tx=0x7fde1802a4d0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7fc81759e640 1 Processor -- start 2024-06-29T14:22:52.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7fc81759e640 1 -- start start 2024-06-29T14:22:52.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.764+0000 7fef3b7fe640 1 -- 172.21.15.81:0/266355538 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef280047a0 con 0x7fef34054f30 2024-06-29T14:22:52.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7f2127d63640 1 -- 172.21.15.81:0/3655297142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2120073620 msgr2=0x7f212006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7f2127d63640 1 --2- 172.21.15.81:0/3655297142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2120073620 0x7f212006edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2120073bf0 tx=0x7f211802a980 comp rx=0 tx=0).stop 2024-06-29T14:22:52.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7fde2922d640 1 -- 172.21.15.81:0/1192429813 shutdown_connections 2024-06-29T14:22:52.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7fde2922d640 1 --2- 172.21.15.81:0/1192429813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde240ced60 0x7fde240cf170 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7fde2922d640 1 --2- 172.21.15.81:0/1192429813 >> v2:172.21.15.81:3300/0 conn(0x7fde24074130 0x7fde240ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7fde2922d640 1 -- 172.21.15.81:0/1192429813 >> 172.21.15.81:0/1192429813 conn(0x7fde2406c2f0 msgr2=0x7fde2406c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.765+0000 7fde2922d640 1 -- 172.21.15.81:0/1192429813 shutdown_connections 2024-06-29T14:22:52.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fde2922d640 1 -- 172.21.15.81:0/1192429813 wait complete. 2024-06-29T14:22:52.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fef41f48640 1 -- 172.21.15.81:0/266355538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef34054f30 msgr2=0x7fef34055340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fef41f48640 1 --2- 172.21.15.81:0/266355538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef34054f30 0x7fef34055340 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fef340352e0 tx=0x7fef2802a560 comp rx=0 tx=0).stop 2024-06-29T14:22:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7f2127d63640 1 -- 172.21.15.81:0/3655297142 shutdown_connections 2024-06-29T14:22:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7f2127d63640 1 --2- 172.21.15.81:0/3655297142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2120073620 0x7f212006edb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7f2127d63640 1 --2- 172.21.15.81:0/3655297142 >> v2:172.21.15.167:3300/0 conn(0x7f2120143860 0x7f21200b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7f2127d63640 1 -- 172.21.15.81:0/3655297142 >> 172.21.15.81:0/3655297142 conn(0x7f21200d0b70 msgr2=0x7f21200d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7f2127d63640 1 -- 172.21.15.81:0/3655297142 shutdown_connections 2024-06-29T14:22:52.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fef41f48640 1 -- 172.21.15.81:0/266355538 shutdown_connections 2024-06-29T14:22:52.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fef41f48640 1 --2- 172.21.15.81:0/266355538 >> v2:172.21.15.167:3300/0 conn(0x7fef34034ce0 0x7fef340e31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fef41f48640 1 --2- 172.21.15.81:0/266355538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef34054f30 0x7fef34055340 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fef41f48640 1 -- 172.21.15.81:0/266355538 >> 172.21.15.81:0/266355538 conn(0x7fef3401ecf0 msgr2=0x7fef3403a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fbb52573640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb4c073450 0x7fbb4c1f8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fbb52573640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb4c0b59c0 0x7fbb4c1fca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fbb52573640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb4c143860 0x7fbb4c1fcf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fbb52573640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4c06f6c0 con 0x7fbb4c0b59c0 2024-06-29T14:22:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fbb52573640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4c06f540 con 0x7fbb4c073450 2024-06-29T14:22:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fbb52573640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbb4c06f840 con 0x7fbb4c143860 2024-06-29T14:22:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fef41f48640 1 -- 172.21.15.81:0/266355538 shutdown_connections 2024-06-29T14:22:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7fef41f48640 1 -- 172.21.15.81:0/266355538 wait complete. 2024-06-29T14:22:52.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.766+0000 7f2127d63640 1 -- 172.21.15.81:0/3655297142 wait complete. 2024-06-29T14:22:52.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.767+0000 7fbb50b10640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb4c143860 0x7fbb4c1fcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.767+0000 7fbb50b10640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb4c143860 0x7fbb4c1fcf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:58348/0 (socket says 172.21.15.81:58348) 2024-06-29T14:22:52.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.767+0000 7fbb50b10640 1 -- 172.21.15.81:0/840875052 learned_addr learned my addr 172.21.15.81:0/840875052 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:52.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.767+0000 7fc81759e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc81003a960 0x7fc810048d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.767+0000 7fbb4bfff640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb4c073450 0x7fbb4c1f8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fbb50b10640 1 -- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb4c073450 msgr2=0x7fbb4c1f8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fbb50b10640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb4c073450 0x7fbb4c1f8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.767+0000 7fc81759e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc810052260 0x7fc8100e50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.767+0000 7fc81659c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc81003a960 0x7fc810048d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.767+0000 7fc81659c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc81003a960 0x7fc810048d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46602/0 (socket says 172.21.15.81:46602) 2024-06-29T14:22:52.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fbb50b10640 1 -- 172.21.15.81:0/840875052 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb4c0b59c0 msgr2=0x7fbb4c1fca00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:22:52.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fbb50b10640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb4c0b59c0 0x7fbb4c1fca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fbb50b10640 1 -- 172.21.15.81:0/840875052 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4c159050 con 0x7fbb4c143860 2024-06-29T14:22:52.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fc81759e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc810052670 0x7fc8100e5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fc81759e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc8100371f0 con 0x7fc810052670 2024-06-29T14:22:52.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fc81759e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc810034730 con 0x7fc81003a960 2024-06-29T14:22:52.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fc81759e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc810037370 con 0x7fc810052260 2024-06-29T14:22:52.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fc81659c640 1 -- 172.21.15.81:0/4088971453 learned_addr learned my addr 172.21.15.81:0/4088971453 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:52.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fde2922d640 1 Processor -- start 2024-06-29T14:22:52.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fbb50b10640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb4c143860 0x7fbb4c1fcf40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009950 tx=0x7fbb40033ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.769+0000 7fde2922d640 1 -- start start 2024-06-29T14:22:52.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fc81659c640 1 -- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc810052260 msgr2=0x7fc8100e50e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:22:52.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fc81659c640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc810052260 0x7fc8100e50e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fc81659c640 1 -- 172.21.15.81:0/4088971453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc810052670 msgr2=0x7fc8100e5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fc81659c640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc810052670 0x7fc8100e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.768+0000 7fc81659c640 1 -- 172.21.15.81:0/4088971453 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81004acd0 con 0x7fc81003a960 2024-06-29T14:22:52.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.769+0000 7fc81659c640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc81003a960 0x7fc810048d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc808009770 tx=0x7fc808033f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.769+0000 7fc806ffd640 1 -- 172.21.15.81:0/4088971453 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808004d70 con 0x7fc81003a960 2024-06-29T14:22:52.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.769+0000 7f2127d63640 1 Processor -- start 2024-06-29T14:22:52.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.769+0000 7f2127d63640 1 -- start start 2024-06-29T14:22:52.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.770+0000 7fef41f48640 1 Processor -- start 2024-06-29T14:22:52.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.770+0000 7fef41f48640 1 -- start start 2024-06-29T14:22:52.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.769+0000 7fbb48ff9640 1 -- 172.21.15.81:0/840875052 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb40037050 con 0x7fbb4c143860 2024-06-29T14:22:52.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.769+0000 7fbb48ff9640 1 -- 172.21.15.81:0/840875052 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb40038040 con 0x7fbb4c143860 2024-06-29T14:22:52.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.770+0000 7f268f282640 1 -- 172.21.15.81:0/732680986 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2674003a50 msgr2=0x7f2674003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.770+0000 7f268f282640 1 --2- 172.21.15.81:0/732680986 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2674003a50 0x7f2674003ec0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2688073a70 tx=0x7f2674005830 comp rx=0 tx=0).stop 2024-06-29T14:22:52.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.769+0000 7fc806ffd640 1 -- 172.21.15.81:0/4088971453 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc808004020 con 0x7fc81003a960 2024-06-29T14:22:52.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.770+0000 7fc806ffd640 1 -- 172.21.15.81:0/4088971453 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808004d70 con 0x7fc81003a960 2024-06-29T14:22:52.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.770+0000 7fc81759e640 1 -- 172.21.15.81:0/4088971453 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8100e4040 con 0x7fc81003a960 2024-06-29T14:22:52.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.770+0000 7fc81759e640 1 -- 172.21.15.81:0/4088971453 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8100e4600 con 0x7fc81003a960 2024-06-29T14:22:52.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7f268f282640 1 -- 172.21.15.81:0/732680986 shutdown_connections 2024-06-29T14:22:52.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7f268f282640 1 --2- 172.21.15.81:0/732680986 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2674003a50 0x7f2674003ec0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7f268f282640 1 -- 172.21.15.81:0/732680986 >> 172.21.15.81:0/732680986 conn(0x7f26880b20c0 msgr2=0x7f268806c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:52.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.770+0000 7fbb48ff9640 1 -- 172.21.15.81:0/840875052 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb40005240 con 0x7fbb4c143860 2024-06-29T14:22:52.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.770+0000 7fbb52573640 1 -- 172.21.15.81:0/840875052 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb4c159330 con 0x7fbb4c143860 2024-06-29T14:22:52.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.770+0000 7fbb52573640 1 -- 172.21.15.81:0/840875052 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb4c15a4c0 con 0x7fbb4c143860 2024-06-29T14:22:52.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7fde2922d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde24074130 0x7fde2406f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7fde2922d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde240ced60 0x7fde2406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7fde2922d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2415f0b0 0x7fde241504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7fde2922d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fde24075990 con 0x7fde24074130 2024-06-29T14:22:52.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7fde2922d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fde24075810 con 0x7fde240ced60 2024-06-29T14:22:52.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7fde2922d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fde240cf6b0 con 0x7fde2415f0b0 2024-06-29T14:22:52.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7f2127d63640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21200b57f0 0x7f2120153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7fbb327fc640 1 -- 172.21.15.81:0/840875052 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb4c1fd7a0 con 0x7fbb4c143860 2024-06-29T14:22:52.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7f2127d63640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2120143860 0x7f2120200e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde237fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde240ced60 0x7fde2406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde237fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde240ced60 0x7fde2406fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46610/0 (socket says 172.21.15.81:46610) 2024-06-29T14:22:52.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7f268f282640 1 -- 172.21.15.81:0/732680986 shutdown_connections 2024-06-29T14:22:52.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.771+0000 7f268f282640 1 -- 172.21.15.81:0/732680986 wait complete. 2024-06-29T14:22:52.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde237fe640 1 -- 172.21.15.81:0/3678099169 learned_addr learned my addr 172.21.15.81:0/3678099169 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:52.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde28a2c640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2415f0b0 0x7fde241504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde23fff640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde24074130 0x7fde2406f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fc81759e640 1 -- 172.21.15.81:0/4088971453 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc81000b0a0 con 0x7fc81003a960 2024-06-29T14:22:52.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde237fe640 1 -- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2415f0b0 msgr2=0x7fde241504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde237fe640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2415f0b0 0x7fde241504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7f21252fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2120143860 0x7f2120200e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7f21252fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2120143860 0x7f2120200e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36288/0 (socket says 172.21.15.81:36288) 2024-06-29T14:22:52.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde237fe640 1 -- 172.21.15.81:0/3678099169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde24074130 msgr2=0x7fde2406f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde237fe640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde24074130 0x7fde2406f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde237fe640 1 -- 172.21.15.81:0/3678099169 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2409cd40 con 0x7fde240ced60 2024-06-29T14:22:52.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde28a2c640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2415f0b0 0x7fde241504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:52.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.772+0000 7fde23fff640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde24074130 0x7fde2406f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:52.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7f21252fe640 1 -- 172.21.15.81:0/3505187798 learned_addr learned my addr 172.21.15.81:0/3505187798 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:52.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7f2127d63640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f212015dde0 0x7f21200ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7fde237fe640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde240ced60 0x7fde2406fc00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00cc90 tx=0x7fde1c00a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f212006f470 con 0x7f2120143860 2024-06-29T14:22:52.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7fde20ff9640 1 -- 172.21.15.81:0/3678099169 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c003b70 con 0x7fde240ced60 2024-06-29T14:22:52.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7f2126300640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f212015dde0 0x7f21200ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7fde20ff9640 1 -- 172.21.15.81:0/3678099169 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde1c003e00 con 0x7fde240ced60 2024-06-29T14:22:52.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f212006f2f0 con 0x7f21200b57f0 2024-06-29T14:22:52.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7fde2922d640 1 -- 172.21.15.81:0/3678099169 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2409d020 con 0x7fde240ced60 2024-06-29T14:22:52.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7fef41f48640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef34034800 0x7fef340dbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f212006f5f0 con 0x7f212015dde0 2024-06-29T14:22:52.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7fde2922d640 1 -- 172.21.15.81:0/3678099169 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2409d5e0 con 0x7fde240ced60 2024-06-29T14:22:52.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7fef41f48640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef34034ce0 0x7fef3404e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7fef40f46640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef34034800 0x7fef340dbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7fef3bfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef34034ce0 0x7fef3404e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fef3bfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef34034ce0 0x7fef3404e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:58382/0 (socket says 172.21.15.81:58382) 2024-06-29T14:22:52.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fef40f46640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef34034800 0x7fef340dbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:46624/0 (socket says 172.21.15.81:46624) 2024-06-29T14:22:52.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fef3bfff640 1 -- 172.21.15.81:0/2906224191 learned_addr learned my addr 172.21.15.81:0/2906224191 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:52.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fef41f48640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef34054f30 0x7fef3404e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.773+0000 7fbb48ff9640 1 -- 172.21.15.81:0/840875052 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbb40045020 con 0x7fbb4c143860 2024-06-29T14:22:52.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fbb48ff9640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbb38076320 0x7fbb38078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7f268f282640 1 Processor -- start 2024-06-29T14:22:52.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fef34052460 con 0x7fef34054f30 2024-06-29T14:22:52.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fbb4bfff640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbb38076320 0x7fbb38078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fef41747640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef34054f30 0x7fef3404e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fef340522e0 con 0x7fef34034800 2024-06-29T14:22:52.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fc806ffd640 1 -- 172.21.15.81:0/4088971453 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc8080041d0 con 0x7fc81003a960 2024-06-29T14:22:52.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7f21252fe640 1 -- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f212015dde0 msgr2=0x7f21200ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7f21252fe640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f212015dde0 0x7f21200ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7f21252fe640 1 -- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21200b57f0 msgr2=0x7f2120153520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7f21252fe640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21200b57f0 0x7f2120153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7f21252fe640 1 -- 172.21.15.81:0/3505187798 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2120154910 con 0x7f2120143860 2024-06-29T14:22:52.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fef340525e0 con 0x7fef34034ce0 2024-06-29T14:22:52.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7fbb48ff9640 1 -- 172.21.15.81:0/840875052 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbb4007e0c0 con 0x7fbb4c143860 2024-06-29T14:22:52.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7f21252fe640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2120143860 0x7f2120200e40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f211c00b780 tx=0x7f211c00bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7fbb4bfff640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbb38076320 0x7fbb38078830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c001a80 tx=0x7fbb3c009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7f2126300640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f212015dde0 0x7f21200ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:52.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7fc806ffd640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc7f80764d0 0x7fc7f80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7fef3bfff640 1 -- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef34034800 msgr2=0x7fef340dbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7fc806ffd640 1 -- 172.21.15.81:0/4088971453 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc80807dc80 con 0x7fc81003a960 2024-06-29T14:22:52.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.774+0000 7f268f282640 1 -- start start 2024-06-29T14:22:52.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7f21167fc640 1 -- 172.21.15.81:0/3505187798 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c019070 con 0x7f2120143860 2024-06-29T14:22:52.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7fef3bfff640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef34034800 0x7fef340dbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7fef3bfff640 1 -- 172.21.15.81:0/2906224191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef34054f30 msgr2=0x7fef3404e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2120154bf0 con 0x7f2120143860 2024-06-29T14:22:52.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7f21167fc640 1 -- 172.21.15.81:0/3505187798 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f211c00c830 con 0x7f2120143860 2024-06-29T14:22:52.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.776+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21201551b0 con 0x7f2120143860 2024-06-29T14:22:52.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7fef3bfff640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef34054f30 0x7fef3404e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.776+0000 7f21167fc640 1 -- 172.21.15.81:0/3505187798 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c00cc80 con 0x7f2120143860 2024-06-29T14:22:52.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7fef3bfff640 1 -- 172.21.15.81:0/2906224191 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef340e5030 con 0x7fef34034ce0 2024-06-29T14:22:52.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.775+0000 7fde20ff9640 1 -- 172.21.15.81:0/3678099169 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c004260 con 0x7fde240ced60 2024-06-29T14:22:52.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.776+0000 7fef3bfff640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef34034ce0 0x7fef3404e2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fef3c06bf70 tx=0x7fef3c069600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.776+0000 7f268f282640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2674003a50 0x7f2688135920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.776+0000 7fde2922d640 1 -- 172.21.15.81:0/3678099169 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde2406dac0 con 0x7fde240ced60 2024-06-29T14:22:52.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.776+0000 7f268e280640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2674003a50 0x7f2688135920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.776+0000 7fc815d9b640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc7f80764d0 0x7fc7f80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.776+0000 7f268e280640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2674003a50 0x7f2688135920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36324/0 (socket says 172.21.15.81:36324) 2024-06-29T14:22:52.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7f268e280640 1 -- 172.21.15.81:0/2059592941 learned_addr learned my addr 172.21.15.81:0/2059592941 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:52.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7f268f282640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268815d7a0 0x7f2688141c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7f268f282640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f268813d0a0 0x7f268813d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7f268da7f640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268815d7a0 0x7f2688141c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7fc815d9b640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc7f80764d0 0x7fc7f80789e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc818051e20 tx=0x7fc818073040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f268815ebe0 con 0x7f2674003a50 2024-06-29T14:22:52.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f268815ea60 con 0x7f268813d0a0 2024-06-29T14:22:52.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7f268ea81640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f268813d0a0 0x7f268813d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7fef397fa640 1 -- 172.21.15.81:0/2906224191 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c06ebd0 con 0x7fef34034ce0 2024-06-29T14:22:52.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7fef397fa640 1 -- 172.21.15.81:0/2906224191 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef3c06ed70 con 0x7fef34034ce0 2024-06-29T14:22:52.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7fef397fa640 1 -- 172.21.15.81:0/2906224191 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c094650 con 0x7fef34034ce0 2024-06-29T14:22:52.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f268815ed60 con 0x7f268815d7a0 2024-06-29T14:22:52.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.777+0000 7fbb48ff9640 1 -- 172.21.15.81:0/840875052 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb400821d0 con 0x7fbb4c143860 2024-06-29T14:22:52.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.778+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef340e5310 con 0x7fef34034ce0 2024-06-29T14:22:52.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.778+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fef340e58d0 con 0x7fef34034ce0 2024-06-29T14:22:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.778+0000 7fde20ff9640 1 -- 172.21.15.81:0/3678099169 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fde1c004400 con 0x7fde240ced60 2024-06-29T14:22:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.779+0000 7fde20ff9640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde100762a0 0x7fde100787b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.778+0000 7f268da7f640 1 -- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f268813d0a0 msgr2=0x7f268813d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.778+0000 7f268da7f640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f268813d0a0 0x7f268813d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.778+0000 7f268da7f640 1 -- 172.21.15.81:0/2059592941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2674003a50 msgr2=0x7f2688135920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.778+0000 7f268da7f640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2674003a50 0x7f2688135920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.778+0000 7f268da7f640 1 -- 172.21.15.81:0/2059592941 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268813d9f0 con 0x7f268815d7a0 2024-06-29T14:22:52.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.779+0000 7fde20ff9640 1 -- 172.21.15.81:0/3678099169 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fde1c0984b0 con 0x7fde240ced60 2024-06-29T14:22:52.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.779+0000 7f268da7f640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268815d7a0 0x7f2688141c40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2680005fd0 tx=0x7f2680010f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.779+0000 7f21167fc640 1 -- 172.21.15.81:0/3505187798 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f211c01c070 con 0x7f2120143860 2024-06-29T14:22:52.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.779+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef34052ab0 con 0x7fef34034ce0 2024-06-29T14:22:52.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.779+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2108005290 con 0x7f2120143860 2024-06-29T14:22:52.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.779+0000 7f21167fc640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2104076320 0x7f2104078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.780+0000 7fde23fff640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde100762a0 0x7fde100787b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.780+0000 7f21167fc640 1 -- 172.21.15.81:0/3505187798 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f211c096b40 con 0x7f2120143860 2024-06-29T14:22:52.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.780+0000 7f2125aff640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2104076320 0x7f2104078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.779+0000 7f267effd640 1 -- 172.21.15.81:0/2059592941 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680003970 con 0x7f268815d7a0 2024-06-29T14:22:52.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.780+0000 7f267effd640 1 -- 172.21.15.81:0/2059592941 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2680003c00 con 0x7f268815d7a0 2024-06-29T14:22:52.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.780+0000 7f267effd640 1 -- 172.21.15.81:0/2059592941 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2680004050 con 0x7f268815d7a0 2024-06-29T14:22:52.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.780+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2688136300 con 0x7f268815d7a0 2024-06-29T14:22:52.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.780+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2688136db0 con 0x7f268815d7a0 2024-06-29T14:22:52.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.780+0000 7fef397fa640 1 -- 172.21.15.81:0/2906224191 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fef3c0947f0 con 0x7fef34034ce0 2024-06-29T14:22:52.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.780+0000 7f2125aff640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2104076320 0x7f2104078830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f211000a860 tx=0x7f2110005b50 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.781+0000 7fef397fa640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fef200763a0 0x7fef200788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.781+0000 7fef397fa640 1 -- 172.21.15.81:0/2906224191 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fef3c112040 con 0x7fef34034ce0 2024-06-29T14:22:52.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.781+0000 7fef40f46640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fef200763a0 0x7fef200788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.781+0000 7fc806ffd640 1 -- 172.21.15.81:0/4088971453 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc81000b0a0 con 0x7fc81003a960 2024-06-29T14:22:52.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.782+0000 7fde23fff640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde100762a0 0x7fde100787b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fde14006fd0 tx=0x7fde14008040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.781+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2688136050 con 0x7f268815d7a0 2024-06-29T14:22:52.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.782+0000 7fef40f46640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fef200763a0 0x7fef200788b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fef30004eb0 tx=0x7fef30007e30 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.782+0000 7f267effd640 1 -- 172.21.15.81:0/2059592941 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f26800041f0 con 0x7f268815d7a0 2024-06-29T14:22:52.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.783+0000 7f267effd640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f266c0762d0 0x7f266c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:52.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.783+0000 7f268e280640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f266c0762d0 0x7f266c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:52.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.783+0000 7f267effd640 1 -- 172.21.15.81:0/2059592941 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2680098600 con 0x7f268815d7a0 2024-06-29T14:22:52.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.783+0000 7f268e280640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f266c0762d0 0x7f266c0787e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2684000c00 tx=0x7f2684001bc0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:52.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.785+0000 7fef397fa640 1 -- 172.21.15.81:0/2906224191 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fef3c0698b0 con 0x7fef34034ce0 2024-06-29T14:22:52.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.786+0000 7fde20ff9640 1 -- 172.21.15.81:0/3678099169 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fde1c014030 con 0x7fde240ced60 2024-06-29T14:22:52.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.790+0000 7f21167fc640 1 -- 172.21.15.81:0/3505187798 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f211c00ce20 con 0x7f2120143860 2024-06-29T14:22:52.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:52.791+0000 7f267effd640 1 -- 172.21.15.81:0/2059592941 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f268813d9f0 con 0x7f268815d7a0 2024-06-29T14:22:53.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.141+0000 7f5fee8ac640 1 -- 172.21.15.81:0/268784861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5fe80b2d70 con 0x7f5fe8073160 2024-06-29T14:22:53.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.142+0000 7f5fe4ff9640 1 -- 172.21.15.81:0/268784861 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5fd8065700 con 0x7f5fe8073160 2024-06-29T14:22:53.142 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-06-29T14:22:53.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.145+0000 7f5fce7fc640 1 -- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5fd4076320 msgr2=0x7f5fd4078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.145+0000 7f5fce7fc640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5fd4076320 0x7f5fd4078830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8074790 tx=0x7f5fdc008040 comp rx=0 tx=0).stop 2024-06-29T14:22:53.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.145+0000 7f5fce7fc640 1 -- 172.21.15.81:0/268784861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe8073160 msgr2=0x7f5fe8131db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.145+0000 7f5fce7fc640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe8073160 0x7f5fe8131db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800ea20 tx=0x7f5fd800eee0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.146+0000 7f5fce7fc640 1 -- 172.21.15.81:0/268784861 shutdown_connections 2024-06-29T14:22:53.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.146+0000 7f5fce7fc640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5fd4076320 0x7f5fd4078830 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.146+0000 7f5fce7fc640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fe8161c50 0x7f5fe8156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.146+0000 7f5fce7fc640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fe815e660 0x7f5fe8156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.146+0000 7f5fce7fc640 1 --2- 172.21.15.81:0/268784861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe8073160 0x7f5fe8131db0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.146+0000 7f5fce7fc640 1 -- 172.21.15.81:0/268784861 >> 172.21.15.81:0/268784861 conn(0x7f5fe80b1910 msgr2=0x7f5fe80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.146+0000 7f5fce7fc640 1 -- 172.21.15.81:0/268784861 shutdown_connections 2024-06-29T14:22:53.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.146+0000 7f5fce7fc640 1 -- 172.21.15.81:0/268784861 wait complete. 2024-06-29T14:22:53.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.157+0000 7f5ad2b2b640 1 -- 172.21.15.81:0/1482036536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5aac005580 con 0x7f5acc0b4560 2024-06-29T14:22:53.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.157+0000 7f5ac97fa640 1 -- 172.21.15.81:0/1482036536 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5abc060530 con 0x7f5acc0b4560 2024-06-29T14:22:53.157 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:53.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.161+0000 7f5ab2ffd640 1 -- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ab8076350 msgr2=0x7f5ab8078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.161+0000 7f5ab2ffd640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ab8076350 0x7f5ab8078860 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00018f0 tx=0x7f5ac0009040 comp rx=0 tx=0).stop 2024-06-29T14:22:53.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.161+0000 7f5ab2ffd640 1 -- 172.21.15.81:0/1482036536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5acc0b4560 msgr2=0x7f5acc13e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.161+0000 7f5ab2ffd640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5acc0b4560 0x7f5acc13e9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00c700 tx=0x7f5abc00cbc0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.161+0000 7f5ab2ffd640 1 -- 172.21.15.81:0/1482036536 shutdown_connections 2024-06-29T14:22:53.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.161+0000 7f5ab2ffd640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5ab8076350 0x7f5ab8078860 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.162+0000 7f5ab2ffd640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5acc15e660 0x7f5acc0ad6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.162+0000 7f5ab2ffd640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5acc0b4560 0x7f5acc13e9c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.162+0000 7f5ab2ffd640 1 --2- 172.21.15.81:0/1482036536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5acc06e0f0 0x7f5acc13f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.162+0000 7f5ab2ffd640 1 -- 172.21.15.81:0/1482036536 >> 172.21.15.81:0/1482036536 conn(0x7f5acc0b1910 msgr2=0x7f5acc0af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.162+0000 7f5ab2ffd640 1 -- 172.21.15.81:0/1482036536 shutdown_connections 2024-06-29T14:22:53.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.162+0000 7f5ab2ffd640 1 -- 172.21.15.81:0/1482036536 wait complete. 2024-06-29T14:22:53.162 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-06-29T14:22:53.163 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:53.178 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2024-06-29T14:22:53.178 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:53.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.191+0000 7fc81759e640 1 -- 172.21.15.81:0/4088971453 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc810037830 con 0x7fc81003a960 2024-06-29T14:22:53.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.191+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fef3404ee40 con 0x7fef34034ce0 2024-06-29T14:22:53.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.192+0000 7fc806ffd640 1 -- 172.21.15.81:0/4088971453 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc808067e20 con 0x7fc81003a960 2024-06-29T14:22:53.191 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-06-29T14:22:53.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.192+0000 7fef397fa640 1 -- 172.21.15.81:0/2906224191 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fef3c0db330 con 0x7fef34034ce0 2024-06-29T14:22:53.192 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 -- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc7f80764d0 msgr2=0x7fc7f80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc7f80764d0 0x7fc7f80789e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc818051e20 tx=0x7fc818073040 comp rx=0 tx=0).stop 2024-06-29T14:22:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 -- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc81003a960 msgr2=0x7fc810048d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc81003a960 0x7fc810048d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc808009770 tx=0x7fc808033f90 comp rx=0 tx=0).stop 2024-06-29T14:22:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 -- 172.21.15.81:0/4088971453 shutdown_connections 2024-06-29T14:22:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc7f80764d0 0x7fc7f80789e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc810052670 0x7fc8100e5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc810052260 0x7fc8100e50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 --2- 172.21.15.81:0/4088971453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc81003a960 0x7fc810048d80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 -- 172.21.15.81:0/4088971453 >> 172.21.15.81:0/4088971453 conn(0x7fc81001e3f0 msgr2=0x7fc81000b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 -- 172.21.15.81:0/4088971453 shutdown_connections 2024-06-29T14:22:53.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fc81759e640 1 -- 172.21.15.81:0/4088971453 wait complete. 2024-06-29T14:22:53.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fef200763a0 msgr2=0x7fef200788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fef41f48640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fef200763a0 0x7fef200788b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fef30004eb0 tx=0x7fef30007e30 comp rx=0 tx=0).stop 2024-06-29T14:22:53.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef34034ce0 msgr2=0x7fef3404e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fef41f48640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef34034ce0 0x7fef3404e2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fef3c06bf70 tx=0x7fef3c069600 comp rx=0 tx=0).stop 2024-06-29T14:22:53.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.194+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 shutdown_connections 2024-06-29T14:22:53.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.195+0000 7fef41f48640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fef200763a0 0x7fef200788b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.195+0000 7fef41f48640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef34054f30 0x7fef3404e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.195+0000 7fef41f48640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef34034ce0 0x7fef3404e2a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.195+0000 7fef41f48640 1 --2- 172.21.15.81:0/2906224191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef34034800 0x7fef340dbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.195+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 >> 172.21.15.81:0/2906224191 conn(0x7fef3401ecf0 msgr2=0x7fef3400c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.195+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 shutdown_connections 2024-06-29T14:22:53.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.195+0000 7fef41f48640 1 -- 172.21.15.81:0/2906224191 wait complete. 2024-06-29T14:22:53.206 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-06-29T14:22:53.206 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:53.207 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2024-06-29T14:22:53.207 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:53.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.209+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f26881368c0 con 0x7f268815d7a0 2024-06-29T14:22:53.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.209+0000 7f267effd640 1 -- 172.21.15.81:0/2059592941 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2680061880 con 0x7f268815d7a0 2024-06-29T14:22:53.208 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:53.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.212+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f266c0762d0 msgr2=0x7f266c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.212+0000 7f268f282640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f266c0762d0 0x7f266c0787e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2684000c00 tx=0x7f2684001bc0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.212+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268815d7a0 msgr2=0x7f2688141c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.212+0000 7f268f282640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268815d7a0 0x7f2688141c40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2680005fd0 tx=0x7f2680010f60 comp rx=0 tx=0).stop 2024-06-29T14:22:53.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.212+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 shutdown_connections 2024-06-29T14:22:53.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.212+0000 7f268f282640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f266c0762d0 0x7f266c0787e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.212+0000 7f268f282640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f268813d0a0 0x7f268813d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.212+0000 7f268f282640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f268815d7a0 0x7f2688141c40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.212+0000 7f268f282640 1 --2- 172.21.15.81:0/2059592941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2674003a50 0x7f2688135920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.212+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 >> 172.21.15.81:0/2059592941 conn(0x7f26880b20c0 msgr2=0x7f26880aef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.213+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 shutdown_connections 2024-06-29T14:22:53.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.213+0000 7f268f282640 1 -- 172.21.15.81:0/2059592941 wait complete. 2024-06-29T14:22:53.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.213+0000 7fbb327fc640 1 -- 172.21.15.81:0/840875052 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fbb4c06fd30 con 0x7fbb4c143860 2024-06-29T14:22:53.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.213+0000 7fbb48ff9640 1 -- 172.21.15.81:0/840875052 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbb400821d0 con 0x7fbb4c143860 2024-06-29T14:22:53.214 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 -- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbb38076320 msgr2=0x7fbb38078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbb38076320 0x7fbb38078830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c001a80 tx=0x7fbb3c009040 comp rx=0 tx=0).stop 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 -- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb4c143860 msgr2=0x7fbb4c1fcf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb4c143860 0x7fbb4c1fcf40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009950 tx=0x7fbb40033ba0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 -- 172.21.15.81:0/840875052 shutdown_connections 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fbb38076320 0x7fbb38078830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb4c143860 0x7fbb4c1fcf40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb4c0b59c0 0x7fbb4c1fca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 --2- 172.21.15.81:0/840875052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb4c073450 0x7fbb4c1f8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 -- 172.21.15.81:0/840875052 >> 172.21.15.81:0/840875052 conn(0x7fbb4c0d0b70 msgr2=0x7fbb4c0ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.216+0000 7fbb327fc640 1 -- 172.21.15.81:0/840875052 shutdown_connections 2024-06-29T14:22:53.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.217+0000 7fbb327fc640 1 -- 172.21.15.81:0/840875052 wait complete. 2024-06-29T14:22:53.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.219+0000 7fde2922d640 1 -- 172.21.15.81:0/3678099169 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fde2409d9c0 con 0x7fde240ced60 2024-06-29T14:22:53.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.219+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f2108005580 con 0x7f2120143860 2024-06-29T14:22:53.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.219+0000 7fde20ff9640 1 -- 172.21.15.81:0/3678099169 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fde1c061720 con 0x7fde240ced60 2024-06-29T14:22:53.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.219+0000 7f21167fc640 1 -- 172.21.15.81:0/3505187798 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f211c00ce20 con 0x7f2120143860 2024-06-29T14:22:53.219 INFO:teuthology.orchestra.run.smithi081.stdout:55834574857 2024-06-29T14:22:53.220 INFO:teuthology.orchestra.run.smithi081.stdout:55834574856 2024-06-29T14:22:53.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2104076320 msgr2=0x7f2104078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2104076320 0x7f2104078830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f211000a860 tx=0x7f2110005b50 comp rx=0 tx=0).stop 2024-06-29T14:22:53.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2120143860 msgr2=0x7f2120200e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2120143860 0x7f2120200e40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f211c00b780 tx=0x7f211c00bc40 comp rx=0 tx=0).stop 2024-06-29T14:22:53.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 shutdown_connections 2024-06-29T14:22:53.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2104076320 0x7f2104078830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f212015dde0 0x7f21200ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2120143860 0x7f2120200e40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 --2- 172.21.15.81:0/3505187798 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21200b57f0 0x7f2120153520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 >> 172.21.15.81:0/3505187798 conn(0x7f21200d0b70 msgr2=0x7f212006cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 shutdown_connections 2024-06-29T14:22:53.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.222+0000 7f2127d63640 1 -- 172.21.15.81:0/3505187798 wait complete. 2024-06-29T14:22:53.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 -- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde100762a0 msgr2=0x7fde100787b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde100762a0 0x7fde100787b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fde14006fd0 tx=0x7fde14008040 comp rx=0 tx=0).stop 2024-06-29T14:22:53.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 -- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde240ced60 msgr2=0x7fde2406fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde240ced60 0x7fde2406fc00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00cc90 tx=0x7fde1c00a620 comp rx=0 tx=0).stop 2024-06-29T14:22:53.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 -- 172.21.15.81:0/3678099169 shutdown_connections 2024-06-29T14:22:53.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde100762a0 0x7fde100787b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2415f0b0 0x7fde241504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde240ced60 0x7fde2406fc00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 --2- 172.21.15.81:0/3678099169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde24074130 0x7fde2406f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 -- 172.21.15.81:0/3678099169 >> 172.21.15.81:0/3678099169 conn(0x7fde2406c2f0 msgr2=0x7fde241607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 -- 172.21.15.81:0/3678099169 shutdown_connections 2024-06-29T14:22:53.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.223+0000 7fde2922d640 1 -- 172.21.15.81:0/3678099169 wait complete. 2024-06-29T14:22:53.226 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2024-06-29T14:22:53.226 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:53.233 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2024-06-29T14:22:53.233 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:53.235 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-06-29T14:22:53.235 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:53.237 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2024-06-29T14:22:53.237 DEBUG:teuthology.parallel:result is None 2024-06-29T14:22:53.238 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-29T14:22:53.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:22:53.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.394+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/875227531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa288004d40 msgr2=0x7fa288005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.394+0000 7fa2a3bfc640 1 --2- 172.21.15.81:0/875227531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa288004d40 0x7fa288005150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa29c08dbf0 tx=0x7fa288005830 comp rx=0 tx=0).stop 2024-06-29T14:22:53.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.394+0000 7fa2a2199640 1 -- 172.21.15.81:0/875227531 reap_dead start 2024-06-29T14:22:53.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.394+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/875227531 shutdown_connections 2024-06-29T14:22:53.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.394+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/875227531 >> 172.21.15.81:0/875227531 conn(0x7fa29c0b56d0 msgr2=0x7fa29c0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.394+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/875227531 shutdown_connections 2024-06-29T14:22:53.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.394+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/875227531 wait complete. 2024-06-29T14:22:53.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.395+0000 7fa2a3bfc640 1 Processor -- start 2024-06-29T14:22:53.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.396+0000 7fa2a3bfc640 1 -- start start 2024-06-29T14:22:53.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a3bfc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa288004d40 0x7fa29c0d0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a3bfc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa29c0d0cc0 0x7fa29c09fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a3bfc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa29c0a02f0 0x7fa29c1f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a3bfc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa29c0cf200 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a3bfc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa29c0cf080 con 0x7fa29c0a02f0 2024-06-29T14:22:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a3bfc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa29c0d1ec0 con 0x7fa288004d40 2024-06-29T14:22:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a1197640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa29c0d0cc0 0x7fa29c09fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a1197640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa29c0d0cc0 0x7fa29c09fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36334/0 (socket says 172.21.15.81:36334) 2024-06-29T14:22:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a1197640 1 -- 172.21.15.81:0/3864639727 learned_addr learned my addr 172.21.15.81:0/3864639727 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a1998640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa288004d40 0x7fa29c0d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a2199640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa29c0a02f0 0x7fa29c1f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a1197640 1 -- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa288004d40 msgr2=0x7fa29c0d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a1197640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa288004d40 0x7fa29c0d0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a1197640 1 -- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa29c0a02f0 msgr2=0x7fa29c1f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a1197640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa29c0a02f0 0x7fa29c1f39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a1197640 1 -- 172.21.15.81:0/3864639727 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa29c1d68d0 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2a1197640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa29c0d0cc0 0x7fa29c09fdb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00e5c0 tx=0x7fa28c00ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.397+0000 7fa2927fc640 1 -- 172.21.15.81:0/3864639727 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c015070 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.398+0000 7fa2927fc640 1 -- 172.21.15.81:0/3864639727 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa28c00ca60 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.398+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/3864639727 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa29c0a0b00 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.398+0000 7fa2927fc640 1 -- 172.21.15.81:0/3864639727 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c011460 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.398+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/3864639727 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa29c0cbcb0 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.399+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/3864639727 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa29c08c270 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.399+0000 7fa2927fc640 1 -- 172.21.15.81:0/3864639727 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa28c018070 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.400+0000 7fa2927fc640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa2800762d0 0x7fa2800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.400+0000 7fa2927fc640 1 -- 172.21.15.81:0/3864639727 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa28c097af0 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.400+0000 7fa2a1998640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa2800762d0 0x7fa2800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.400+0000 7fa2a1998640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa2800762d0 0x7fa2800787e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa298009b10 tx=0x7fa298005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:53.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.404+0000 7fa2927fc640 1 -- 172.21.15.81:0/3864639727 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa28c0614e0 con 0x7fa29c0d0cc0 2024-06-29T14:22:53.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.582+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/3864639727 --> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa29c0b2010 con 0x7fa2800762d0 2024-06-29T14:22:53.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.587+0000 7fa2927fc640 1 -- 172.21.15.81:0/3864639727 <== mgr.4126 v2:172.21.15.167:6832/3101542440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7fa29c0b2010 con 0x7fa2800762d0 2024-06-29T14:22:53.588 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:53.589 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:22:53.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa2800762d0 msgr2=0x7fa2800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa2800762d0 0x7fa2800787e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa298009b10 tx=0x7fa298005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:53.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/3864639727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa29c0d0cc0 msgr2=0x7fa29c09fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa29c0d0cc0 0x7fa29c09fdb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00e5c0 tx=0x7fa28c00ea80 comp rx=0 tx=0).stop 2024-06-29T14:22:53.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/3864639727 shutdown_connections 2024-06-29T14:22:53.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa2800762d0 0x7fa2800787e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa29c0a02f0 0x7fa29c1f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa29c0d0cc0 0x7fa29c09fdb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 --2- 172.21.15.81:0/3864639727 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa288004d40 0x7fa29c0d0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/3864639727 >> 172.21.15.81:0/3864639727 conn(0x7fa29c0b56d0 msgr2=0x7fa29c0b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/3864639727 shutdown_connections 2024-06-29T14:22:53.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.590+0000 7fa2a3bfc640 1 -- 172.21.15.81:0/3864639727 wait complete. 2024-06-29T14:22:53.599 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-06-29T14:22:52.706240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219020,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755700,"statfs":{"total":773094113280,"available":772869836800,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.020662"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316337+0000","last_change":"2024-06-29T14:22:44.722050+0000","last_active":"2024-06-29T14:22:46.316337+0000","last_peered":"2024-06-29T14:22:46.316337+0000","last_clean":"2024-06-29T14:22:46.316337+0000","last_became_active":"2024-06-29T14:22:44.721719+0000","last_became_peered":"2024-06-29T14:22:44.721719+0000","last_unstale":"2024-06-29T14:22:46.316337+0000","last_undegraded":"2024-06-29T14:22:46.316337+0000","last_fullsized":"2024-06-29T14:22:46.316337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:55:43.420129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701534+0000","last_change":"2024-06-29T14:22:44.709906+0000","last_active":"2024-06-29T14:22:45.701534+0000","last_peered":"2024-06-29T14:22:45.701534+0000","last_clean":"2024-06-29T14:22:45.701534+0000","last_became_active":"2024-06-29T14:22:44.709406+0000","last_became_peered":"2024-06-29T14:22:44.709406+0000","last_unstale":"2024-06-29T14:22:45.701534+0000","last_undegraded":"2024-06-29T14:22:45.701534+0000","last_fullsized":"2024-06-29T14:22:45.701534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:29.350494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315145+0000","last_change":"2024-06-29T14:22:44.705357+0000","last_active":"2024-06-29T14:22:46.315145+0000","last_peered":"2024-06-29T14:22:46.315145+0000","last_clean":"2024-06-29T14:22:46.315145+0000","last_became_active":"2024-06-29T14:22:44.704402+0000","last_became_peered":"2024-06-29T14:22:44.704402+0000","last_unstale":"2024-06-29T14:22:46.315145+0000","last_undegraded":"2024-06-29T14:22:46.315145+0000","last_fullsized":"2024-06-29T14:22:46.315145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704481+0000","last_change":"2024-06-29T14:22:44.708801+0000","last_active":"2024-06-29T14:22:45.704481+0000","last_peered":"2024-06-29T14:22:45.704481+0000","last_clean":"2024-06-29T14:22:45.704481+0000","last_became_active":"2024-06-29T14:22:44.708456+0000","last_became_peered":"2024-06-29T14:22:44.708456+0000","last_unstale":"2024-06-29T14:22:45.704481+0000","last_undegraded":"2024-06-29T14:22:45.704481+0000","last_fullsized":"2024-06-29T14:22:45.704481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:40.172805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315866+0000","last_change":"2024-06-29T14:22:44.719917+0000","last_active":"2024-06-29T14:22:46.315866+0000","last_peered":"2024-06-29T14:22:46.315866+0000","last_clean":"2024-06-29T14:22:46.315866+0000","last_became_active":"2024-06-29T14:22:44.719617+0000","last_became_peered":"2024-06-29T14:22:44.719617+0000","last_unstale":"2024-06-29T14:22:46.315866+0000","last_undegraded":"2024-06-29T14:22:46.315866+0000","last_fullsized":"2024-06-29T14:22:46.315866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:40.172805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704263+0000","last_change":"2024-06-29T14:22:44.701697+0000","last_active":"2024-06-29T14:22:45.704263+0000","last_peered":"2024-06-29T14:22:45.704263+0000","last_clean":"2024-06-29T14:22:45.704263+0000","last_became_active":"2024-06-29T14:22:44.701325+0000","last_became_peered":"2024-06-29T14:22:44.701325+0000","last_unstale":"2024-06-29T14:22:45.704263+0000","last_undegraded":"2024-06-29T14:22:45.704263+0000","last_fullsized":"2024-06-29T14:22:45.704263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:01.618482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315725+0000","last_change":"2024-06-29T14:22:44.720635+0000","last_active":"2024-06-29T14:22:46.315725+0000","last_peered":"2024-06-29T14:22:46.315725+0000","last_clean":"2024-06-29T14:22:46.315725+0000","last_became_active":"2024-06-29T14:22:44.720316+0000","last_became_peered":"2024-06-29T14:22:44.720316+0000","last_unstale":"2024-06-29T14:22:46.315725+0000","last_undegraded":"2024-06-29T14:22:46.315725+0000","last_fullsized":"2024-06-29T14:22:46.315725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:52.918063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314686+0000","last_change":"2024-06-29T14:22:44.706868+0000","last_active":"2024-06-29T14:22:46.314686+0000","last_peered":"2024-06-29T14:22:46.314686+0000","last_clean":"2024-06-29T14:22:46.314686+0000","last_became_active":"2024-06-29T14:22:44.706298+0000","last_became_peered":"2024-06-29T14:22:44.706298+0000","last_unstale":"2024-06-29T14:22:46.314686+0000","last_undegraded":"2024-06-29T14:22:46.314686+0000","last_fullsized":"2024-06-29T14:22:46.314686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316405+0000","last_change":"2024-06-29T14:22:44.722834+0000","last_active":"2024-06-29T14:22:46.316405+0000","last_peered":"2024-06-29T14:22:46.316405+0000","last_clean":"2024-06-29T14:22:46.316405+0000","last_became_active":"2024-06-29T14:22:44.722345+0000","last_became_peered":"2024-06-29T14:22:44.722345+0000","last_unstale":"2024-06-29T14:22:46.316405+0000","last_undegraded":"2024-06-29T14:22:46.316405+0000","last_fullsized":"2024-06-29T14:22:46.316405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:37:57.057668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315569+0000","last_change":"2024-06-29T14:22:44.702528+0000","last_active":"2024-06-29T14:22:46.315569+0000","last_peered":"2024-06-29T14:22:46.315569+0000","last_clean":"2024-06-29T14:22:46.315569+0000","last_became_active":"2024-06-29T14:22:44.702246+0000","last_became_peered":"2024-06-29T14:22:44.702246+0000","last_unstale":"2024-06-29T14:22:46.315569+0000","last_undegraded":"2024-06-29T14:22:46.315569+0000","last_fullsized":"2024-06-29T14:22:46.315569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:29.350494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315346+0000","last_change":"2024-06-29T14:22:44.703308+0000","last_active":"2024-06-29T14:22:46.315346+0000","last_peered":"2024-06-29T14:22:46.315346+0000","last_clean":"2024-06-29T14:22:46.315346+0000","last_became_active":"2024-06-29T14:22:44.703018+0000","last_became_peered":"2024-06-29T14:22:44.703018+0000","last_unstale":"2024-06-29T14:22:46.315346+0000","last_undegraded":"2024-06-29T14:22:46.315346+0000","last_fullsized":"2024-06-29T14:22:46.315346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703510+0000","last_change":"2024-06-29T14:22:44.702902+0000","last_active":"2024-06-29T14:22:45.703510+0000","last_peered":"2024-06-29T14:22:45.703510+0000","last_clean":"2024-06-29T14:22:45.703510+0000","last_became_active":"2024-06-29T14:22:44.702595+0000","last_became_peered":"2024-06-29T14:22:44.702595+0000","last_unstale":"2024-06-29T14:22:45.703510+0000","last_undegraded":"2024-06-29T14:22:45.703510+0000","last_fullsized":"2024-06-29T14:22:45.703510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:46.021153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314835+0000","last_change":"2024-06-29T14:22:44.703127+0000","last_active":"2024-06-29T14:22:46.314835+0000","last_peered":"2024-06-29T14:22:46.314835+0000","last_clean":"2024-06-29T14:22:46.314835+0000","last_became_active":"2024-06-29T14:22:44.702813+0000","last_became_peered":"2024-06-29T14:22:44.702813+0000","last_unstale":"2024-06-29T14:22:46.314835+0000","last_undegraded":"2024-06-29T14:22:46.314835+0000","last_fullsized":"2024-06-29T14:22:46.314835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314658+0000","last_change":"2024-06-29T14:22:44.714140+0000","last_active":"2024-06-29T14:22:46.314658+0000","last_peered":"2024-06-29T14:22:46.314658+0000","last_clean":"2024-06-29T14:22:46.314658+0000","last_became_active":"2024-06-29T14:22:44.713863+0000","last_became_peered":"2024-06-29T14:22:44.713863+0000","last_unstale":"2024-06-29T14:22:46.314658+0000","last_undegraded":"2024-06-29T14:22:46.314658+0000","last_fullsized":"2024-06-29T14:22:46.314658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701353+0000","last_change":"2024-06-29T14:22:44.699851+0000","last_active":"2024-06-29T14:22:45.701353+0000","last_peered":"2024-06-29T14:22:45.701353+0000","last_clean":"2024-06-29T14:22:45.701353+0000","last_became_active":"2024-06-29T14:22:44.697960+0000","last_became_peered":"2024-06-29T14:22:44.697960+0000","last_unstale":"2024-06-29T14:22:45.701353+0000","last_undegraded":"2024-06-29T14:22:45.701353+0000","last_fullsized":"2024-06-29T14:22:45.701353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703910+0000","last_change":"2024-06-29T14:22:44.708069+0000","last_active":"2024-06-29T14:22:45.703910+0000","last_peered":"2024-06-29T14:22:45.703910+0000","last_clean":"2024-06-29T14:22:45.703910+0000","last_became_active":"2024-06-29T14:22:44.707765+0000","last_became_peered":"2024-06-29T14:22:44.707765+0000","last_unstale":"2024-06-29T14:22:45.703910+0000","last_undegraded":"2024-06-29T14:22:45.703910+0000","last_fullsized":"2024-06-29T14:22:45.703910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704975+0000","last_change":"2024-06-29T14:22:44.721271+0000","last_active":"2024-06-29T14:22:45.704975+0000","last_peered":"2024-06-29T14:22:45.704975+0000","last_clean":"2024-06-29T14:22:45.704975+0000","last_became_active":"2024-06-29T14:22:44.720978+0000","last_became_peered":"2024-06-29T14:22:44.720978+0000","last_unstale":"2024-06-29T14:22:45.704975+0000","last_undegraded":"2024-06-29T14:22:45.704975+0000","last_fullsized":"2024-06-29T14:22:45.704975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:37:57.057668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725545+0000","last_change":"2024-06-29T14:22:44.702764+0000","last_active":"2024-06-29T14:22:45.725545+0000","last_peered":"2024-06-29T14:22:45.725545+0000","last_clean":"2024-06-29T14:22:45.725545+0000","last_became_active":"2024-06-29T14:22:44.702408+0000","last_became_peered":"2024-06-29T14:22:44.702408+0000","last_unstale":"2024-06-29T14:22:45.725545+0000","last_undegraded":"2024-06-29T14:22:45.725545+0000","last_fullsized":"2024-06-29T14:22:45.725545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316657+0000","last_change":"2024-06-29T14:22:44.718193+0000","last_active":"2024-06-29T14:22:46.316657+0000","last_peered":"2024-06-29T14:22:46.316657+0000","last_clean":"2024-06-29T14:22:46.316657+0000","last_became_active":"2024-06-29T14:22:44.717900+0000","last_became_peered":"2024-06-29T14:22:44.717900+0000","last_unstale":"2024-06-29T14:22:46.316657+0000","last_undegraded":"2024-06-29T14:22:46.316657+0000","last_fullsized":"2024-06-29T14:22:46.316657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.810207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315331+0000","last_change":"2024-06-29T14:22:44.701520+0000","last_active":"2024-06-29T14:22:46.315331+0000","last_peered":"2024-06-29T14:22:46.315331+0000","last_clean":"2024-06-29T14:22:46.315331+0000","last_became_active":"2024-06-29T14:22:44.701106+0000","last_became_peered":"2024-06-29T14:22:44.701106+0000","last_unstale":"2024-06-29T14:22:46.315331+0000","last_undegraded":"2024-06-29T14:22:46.315331+0000","last_fullsized":"2024-06-29T14:22:46.315331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315186+0000","last_change":"2024-06-29T14:22:44.717603+0000","last_active":"2024-06-29T14:22:46.315186+0000","last_peered":"2024-06-29T14:22:46.315186+0000","last_clean":"2024-06-29T14:22:46.315186+0000","last_became_active":"2024-06-29T14:22:44.717305+0000","last_became_peered":"2024-06-29T14:22:44.717305+0000","last_unstale":"2024-06-29T14:22:46.315186+0000","last_undegraded":"2024-06-29T14:22:46.315186+0000","last_fullsized":"2024-06-29T14:22:46.315186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704911+0000","last_change":"2024-06-29T14:22:44.717305+0000","last_active":"2024-06-29T14:22:45.704911+0000","last_peered":"2024-06-29T14:22:45.704911+0000","last_clean":"2024-06-29T14:22:45.704911+0000","last_became_active":"2024-06-29T14:22:44.716953+0000","last_became_peered":"2024-06-29T14:22:44.716953+0000","last_unstale":"2024-06-29T14:22:45.704911+0000","last_undegraded":"2024-06-29T14:22:45.704911+0000","last_fullsized":"2024-06-29T14:22:45.704911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:31:30.258889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315247+0000","last_change":"2024-06-29T14:22:44.701947+0000","last_active":"2024-06-29T14:22:46.315247+0000","last_peered":"2024-06-29T14:22:46.315247+0000","last_clean":"2024-06-29T14:22:46.315247+0000","last_became_active":"2024-06-29T14:22:44.701624+0000","last_became_peered":"2024-06-29T14:22:44.701624+0000","last_unstale":"2024-06-29T14:22:46.315247+0000","last_undegraded":"2024-06-29T14:22:46.315247+0000","last_fullsized":"2024-06-29T14:22:46.315247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:52.918063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703990+0000","last_change":"2024-06-29T14:22:44.709275+0000","last_active":"2024-06-29T14:22:45.703990+0000","last_peered":"2024-06-29T14:22:45.703990+0000","last_clean":"2024-06-29T14:22:45.703990+0000","last_became_active":"2024-06-29T14:22:44.708702+0000","last_became_peered":"2024-06-29T14:22:44.708702+0000","last_unstale":"2024-06-29T14:22:45.703990+0000","last_undegraded":"2024-06-29T14:22:45.703990+0000","last_fullsized":"2024-06-29T14:22:45.703990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:25.905572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314785+0000","last_change":"2024-06-29T14:22:44.702814+0000","last_active":"2024-06-29T14:22:46.314785+0000","last_peered":"2024-06-29T14:22:46.314785+0000","last_clean":"2024-06-29T14:22:46.314785+0000","last_became_active":"2024-06-29T14:22:44.702479+0000","last_became_peered":"2024-06-29T14:22:44.702479+0000","last_unstale":"2024-06-29T14:22:46.314785+0000","last_undegraded":"2024-06-29T14:22:46.314785+0000","last_fullsized":"2024-06-29T14:22:46.314785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315590+0000","last_change":"2024-06-29T14:22:44.718581+0000","last_active":"2024-06-29T14:22:46.315590+0000","last_peered":"2024-06-29T14:22:46.315590+0000","last_clean":"2024-06-29T14:22:46.315590+0000","last_became_active":"2024-06-29T14:22:44.718277+0000","last_became_peered":"2024-06-29T14:22:44.718277+0000","last_unstale":"2024-06-29T14:22:46.315590+0000","last_undegraded":"2024-06-29T14:22:46.315590+0000","last_fullsized":"2024-06-29T14:22:46.315590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701423+0000","last_change":"2024-06-29T14:22:44.704193+0000","last_active":"2024-06-29T14:22:45.701423+0000","last_peered":"2024-06-29T14:22:45.701423+0000","last_clean":"2024-06-29T14:22:45.701423+0000","last_became_active":"2024-06-29T14:22:44.703892+0000","last_became_peered":"2024-06-29T14:22:44.703892+0000","last_unstale":"2024-06-29T14:22:45.701423+0000","last_undegraded":"2024-06-29T14:22:45.701423+0000","last_fullsized":"2024-06-29T14:22:45.701423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315350+0000","last_change":"2024-06-29T14:22:44.703952+0000","last_active":"2024-06-29T14:22:46.315350+0000","last_peered":"2024-06-29T14:22:46.315350+0000","last_clean":"2024-06-29T14:22:46.315350+0000","last_became_active":"2024-06-29T14:22:44.703431+0000","last_became_peered":"2024-06-29T14:22:44.703431+0000","last_unstale":"2024-06-29T14:22:46.315350+0000","last_undegraded":"2024-06-29T14:22:46.315350+0000","last_fullsized":"2024-06-29T14:22:46.315350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:40.172805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315348+0000","last_change":"2024-06-29T14:22:44.714045+0000","last_active":"2024-06-29T14:22:46.315348+0000","last_peered":"2024-06-29T14:22:46.315348+0000","last_clean":"2024-06-29T14:22:46.315348+0000","last_became_active":"2024-06-29T14:22:44.713749+0000","last_became_peered":"2024-06-29T14:22:44.713749+0000","last_unstale":"2024-06-29T14:22:46.315348+0000","last_undegraded":"2024-06-29T14:22:46.315348+0000","last_fullsized":"2024-06-29T14:22:46.315348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704721+0000","last_change":"2024-06-29T14:22:44.709553+0000","last_active":"2024-06-29T14:22:45.704721+0000","last_peered":"2024-06-29T14:22:45.704721+0000","last_clean":"2024-06-29T14:22:45.704721+0000","last_became_active":"2024-06-29T14:22:44.709153+0000","last_became_peered":"2024-06-29T14:22:44.709153+0000","last_unstale":"2024-06-29T14:22:45.704721+0000","last_undegraded":"2024-06-29T14:22:45.704721+0000","last_fullsized":"2024-06-29T14:22:45.704721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:19.819693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725353+0000","last_change":"2024-06-29T14:22:44.699167+0000","last_active":"2024-06-29T14:22:45.725353+0000","last_peered":"2024-06-29T14:22:45.725353+0000","last_clean":"2024-06-29T14:22:45.725353+0000","last_became_active":"2024-06-29T14:22:44.698265+0000","last_became_peered":"2024-06-29T14:22:44.698265+0000","last_unstale":"2024-06-29T14:22:45.725353+0000","last_undegraded":"2024-06-29T14:22:45.725353+0000","last_fullsized":"2024-06-29T14:22:45.725353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315125+0000","last_change":"2024-06-29T14:22:44.710341+0000","last_active":"2024-06-29T14:22:46.315125+0000","last_peered":"2024-06-29T14:22:46.315125+0000","last_clean":"2024-06-29T14:22:46.315125+0000","last_became_active":"2024-06-29T14:22:44.709992+0000","last_became_peered":"2024-06-29T14:22:44.709992+0000","last_unstale":"2024-06-29T14:22:46.315125+0000","last_undegraded":"2024-06-29T14:22:46.315125+0000","last_fullsized":"2024-06-29T14:22:46.315125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703805+0000","last_change":"2024-06-29T14:22:44.713512+0000","last_active":"2024-06-29T14:22:45.703805+0000","last_peered":"2024-06-29T14:22:45.703805+0000","last_clean":"2024-06-29T14:22:45.703805+0000","last_became_active":"2024-06-29T14:22:44.713208+0000","last_became_peered":"2024-06-29T14:22:44.713208+0000","last_unstale":"2024-06-29T14:22:45.703805+0000","last_undegraded":"2024-06-29T14:22:45.703805+0000","last_fullsized":"2024-06-29T14:22:45.703805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315120+0000","last_change":"2024-06-29T14:22:43.711234+0000","last_active":"2024-06-29T14:22:46.315120+0000","last_peered":"2024-06-29T14:22:46.315120+0000","last_clean":"2024-06-29T14:22:46.315120+0000","last_became_active":"2024-06-29T14:22:43.710904+0000","last_became_peered":"2024-06-29T14:22:43.710904+0000","last_unstale":"2024-06-29T14:22:46.315120+0000","last_undegraded":"2024-06-29T14:22:46.315120+0000","last_fullsized":"2024-06-29T14:22:46.315120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315427+0000","last_change":"2024-06-29T14:22:44.706122+0000","last_active":"2024-06-29T14:22:46.315427+0000","last_peered":"2024-06-29T14:22:46.315427+0000","last_clean":"2024-06-29T14:22:46.315427+0000","last_became_active":"2024-06-29T14:22:44.705746+0000","last_became_peered":"2024-06-29T14:22:44.705746+0000","last_unstale":"2024-06-29T14:22:46.315427+0000","last_undegraded":"2024-06-29T14:22:46.315427+0000","last_fullsized":"2024-06-29T14:22:46.315427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:29.350494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315012+0000","last_change":"2024-06-29T14:22:43.700941+0000","last_active":"2024-06-29T14:22:46.315012+0000","last_peered":"2024-06-29T14:22:46.315012+0000","last_clean":"2024-06-29T14:22:46.315012+0000","last_became_active":"2024-06-29T14:22:43.700234+0000","last_became_peered":"2024-06-29T14:22:43.700234+0000","last_unstale":"2024-06-29T14:22:46.315012+0000","last_undegraded":"2024-06-29T14:22:46.315012+0000","last_fullsized":"2024-06-29T14:22:46.315012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315413+0000","last_change":"2024-06-29T14:22:43.707763+0000","last_active":"2024-06-29T14:22:46.315413+0000","last_peered":"2024-06-29T14:22:46.315413+0000","last_clean":"2024-06-29T14:22:46.315413+0000","last_became_active":"2024-06-29T14:22:43.707378+0000","last_became_peered":"2024-06-29T14:22:43.707378+0000","last_unstale":"2024-06-29T14:22:46.315413+0000","last_undegraded":"2024-06-29T14:22:46.315413+0000","last_fullsized":"2024-06-29T14:22:46.315413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:26.060090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314961+0000","last_change":"2024-06-29T14:22:44.705946+0000","last_active":"2024-06-29T14:22:46.314961+0000","last_peered":"2024-06-29T14:22:46.314961+0000","last_clean":"2024-06-29T14:22:46.314961+0000","last_became_active":"2024-06-29T14:22:44.705469+0000","last_became_peered":"2024-06-29T14:22:44.705469+0000","last_unstale":"2024-06-29T14:22:46.314961+0000","last_undegraded":"2024-06-29T14:22:46.314961+0000","last_fullsized":"2024-06-29T14:22:46.314961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704025+0000","last_change":"2024-06-29T14:22:43.698273+0000","last_active":"2024-06-29T14:22:45.704025+0000","last_peered":"2024-06-29T14:22:45.704025+0000","last_clean":"2024-06-29T14:22:45.704025+0000","last_became_active":"2024-06-29T14:22:43.697950+0000","last_became_peered":"2024-06-29T14:22:43.697950+0000","last_unstale":"2024-06-29T14:22:45.704025+0000","last_undegraded":"2024-06-29T14:22:45.704025+0000","last_fullsized":"2024-06-29T14:22:45.704025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701302+0000","last_change":"2024-06-29T14:22:44.708264+0000","last_active":"2024-06-29T14:22:45.701302+0000","last_peered":"2024-06-29T14:22:45.701302+0000","last_clean":"2024-06-29T14:22:45.701302+0000","last_became_active":"2024-06-29T14:22:44.707924+0000","last_became_peered":"2024-06-29T14:22:44.707924+0000","last_unstale":"2024-06-29T14:22:45.701302+0000","last_undegraded":"2024-06-29T14:22:45.701302+0000","last_fullsized":"2024-06-29T14:22:45.701302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315270+0000","last_change":"2024-06-29T14:22:43.706343+0000","last_active":"2024-06-29T14:22:46.315270+0000","last_peered":"2024-06-29T14:22:46.315270+0000","last_clean":"2024-06-29T14:22:46.315270+0000","last_became_active":"2024-06-29T14:22:43.706009+0000","last_became_peered":"2024-06-29T14:22:43.706009+0000","last_unstale":"2024-06-29T14:22:46.315270+0000","last_undegraded":"2024-06-29T14:22:46.315270+0000","last_fullsized":"2024-06-29T14:22:46.315270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704221+0000","last_change":"2024-06-29T14:22:44.717406+0000","last_active":"2024-06-29T14:22:45.704221+0000","last_peered":"2024-06-29T14:22:45.704221+0000","last_clean":"2024-06-29T14:22:45.704221+0000","last_became_active":"2024-06-29T14:22:44.717095+0000","last_became_peered":"2024-06-29T14:22:44.717095+0000","last_unstale":"2024-06-29T14:22:45.704221+0000","last_undegraded":"2024-06-29T14:22:45.704221+0000","last_fullsized":"2024-06-29T14:22:45.704221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:52.918063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704373+0000","last_change":"2024-06-29T14:22:43.701234+0000","last_active":"2024-06-29T14:22:45.704373+0000","last_peered":"2024-06-29T14:22:45.704373+0000","last_clean":"2024-06-29T14:22:45.704373+0000","last_became_active":"2024-06-29T14:22:43.700923+0000","last_became_peered":"2024-06-29T14:22:43.700923+0000","last_unstale":"2024-06-29T14:22:45.704373+0000","last_undegraded":"2024-06-29T14:22:45.704373+0000","last_fullsized":"2024-06-29T14:22:45.704373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315568+0000","last_change":"2024-06-29T14:22:44.708529+0000","last_active":"2024-06-29T14:22:46.315568+0000","last_peered":"2024-06-29T14:22:46.315568+0000","last_clean":"2024-06-29T14:22:46.315568+0000","last_became_active":"2024-06-29T14:22:44.708019+0000","last_became_peered":"2024-06-29T14:22:44.708019+0000","last_unstale":"2024-06-29T14:22:46.315568+0000","last_undegraded":"2024-06-29T14:22:46.315568+0000","last_fullsized":"2024-06-29T14:22:46.315568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701544+0000","last_change":"2024-06-29T14:22:43.705026+0000","last_active":"2024-06-29T14:22:45.701544+0000","last_peered":"2024-06-29T14:22:45.701544+0000","last_clean":"2024-06-29T14:22:45.701544+0000","last_became_active":"2024-06-29T14:22:43.704704+0000","last_became_peered":"2024-06-29T14:22:43.704704+0000","last_unstale":"2024-06-29T14:22:45.701544+0000","last_undegraded":"2024-06-29T14:22:45.701544+0000","last_fullsized":"2024-06-29T14:22:45.701544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703710+0000","last_change":"2024-06-29T14:22:44.716149+0000","last_active":"2024-06-29T14:22:45.703710+0000","last_peered":"2024-06-29T14:22:45.703710+0000","last_clean":"2024-06-29T14:22:45.703710+0000","last_became_active":"2024-06-29T14:22:44.715832+0000","last_became_peered":"2024-06-29T14:22:44.715832+0000","last_unstale":"2024-06-29T14:22:45.703710+0000","last_undegraded":"2024-06-29T14:22:45.703710+0000","last_fullsized":"2024-06-29T14:22:45.703710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:40.008975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315151+0000","last_change":"2024-06-29T14:22:43.707443+0000","last_active":"2024-06-29T14:22:46.315151+0000","last_peered":"2024-06-29T14:22:46.315151+0000","last_clean":"2024-06-29T14:22:46.315151+0000","last_became_active":"2024-06-29T14:22:43.707109+0000","last_became_peered":"2024-06-29T14:22:43.707109+0000","last_unstale":"2024-06-29T14:22:46.315151+0000","last_undegraded":"2024-06-29T14:22:46.315151+0000","last_fullsized":"2024-06-29T14:22:46.315151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316025+0000","last_change":"2024-06-29T14:22:44.719076+0000","last_active":"2024-06-29T14:22:46.316025+0000","last_peered":"2024-06-29T14:22:46.316025+0000","last_clean":"2024-06-29T14:22:46.316025+0000","last_became_active":"2024-06-29T14:22:44.718736+0000","last_became_peered":"2024-06-29T14:22:44.718736+0000","last_unstale":"2024-06-29T14:22:46.316025+0000","last_undegraded":"2024-06-29T14:22:46.316025+0000","last_fullsized":"2024-06-29T14:22:46.316025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:11.627629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315458+0000","last_change":"2024-06-29T14:22:43.700129+0000","last_active":"2024-06-29T14:22:46.315458+0000","last_peered":"2024-06-29T14:22:46.315458+0000","last_clean":"2024-06-29T14:22:46.315458+0000","last_became_active":"2024-06-29T14:22:43.699771+0000","last_became_peered":"2024-06-29T14:22:43.699771+0000","last_unstale":"2024-06-29T14:22:46.315458+0000","last_undegraded":"2024-06-29T14:22:46.315458+0000","last_fullsized":"2024-06-29T14:22:46.315458+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:53.109228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314834+0000","last_change":"2024-06-29T14:22:44.716496+0000","last_active":"2024-06-29T14:22:46.314834+0000","last_peered":"2024-06-29T14:22:46.314834+0000","last_clean":"2024-06-29T14:22:46.314834+0000","last_became_active":"2024-06-29T14:22:44.716244+0000","last_became_peered":"2024-06-29T14:22:44.716244+0000","last_unstale":"2024-06-29T14:22:46.314834+0000","last_undegraded":"2024-06-29T14:22:46.314834+0000","last_fullsized":"2024-06-29T14:22:46.314834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704937+0000","last_change":"2024-06-29T14:22:43.706155+0000","last_active":"2024-06-29T14:22:45.704937+0000","last_peered":"2024-06-29T14:22:45.704937+0000","last_clean":"2024-06-29T14:22:45.704937+0000","last_became_active":"2024-06-29T14:22:43.705468+0000","last_became_peered":"2024-06-29T14:22:43.705468+0000","last_unstale":"2024-06-29T14:22:45.704937+0000","last_undegraded":"2024-06-29T14:22:45.704937+0000","last_fullsized":"2024-06-29T14:22:45.704937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316005+0000","last_change":"2024-06-29T14:22:44.719377+0000","last_active":"2024-06-29T14:22:46.316005+0000","last_peered":"2024-06-29T14:22:46.316005+0000","last_clean":"2024-06-29T14:22:46.316005+0000","last_became_active":"2024-06-29T14:22:44.719065+0000","last_became_peered":"2024-06-29T14:22:44.719065+0000","last_unstale":"2024-06-29T14:22:46.316005+0000","last_undegraded":"2024-06-29T14:22:46.316005+0000","last_fullsized":"2024-06-29T14:22:46.316005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:51:55.034132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314958+0000","last_change":"2024-06-29T14:22:43.698317+0000","last_active":"2024-06-29T14:22:46.314958+0000","last_peered":"2024-06-29T14:22:46.314958+0000","last_clean":"2024-06-29T14:22:46.314958+0000","last_became_active":"2024-06-29T14:22:43.697987+0000","last_became_peered":"2024-06-29T14:22:43.697987+0000","last_unstale":"2024-06-29T14:22:46.314958+0000","last_undegraded":"2024-06-29T14:22:46.314958+0000","last_fullsized":"2024-06-29T14:22:46.314958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315030+0000","last_change":"2024-06-29T14:22:44.710262+0000","last_active":"2024-06-29T14:22:46.315030+0000","last_peered":"2024-06-29T14:22:46.315030+0000","last_clean":"2024-06-29T14:22:46.315030+0000","last_became_active":"2024-06-29T14:22:44.709948+0000","last_became_peered":"2024-06-29T14:22:44.709948+0000","last_unstale":"2024-06-29T14:22:46.315030+0000","last_undegraded":"2024-06-29T14:22:46.315030+0000","last_fullsized":"2024-06-29T14:22:46.315030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725271+0000","last_change":"2024-06-29T14:22:43.703719+0000","last_active":"2024-06-29T14:22:45.725271+0000","last_peered":"2024-06-29T14:22:45.725271+0000","last_clean":"2024-06-29T14:22:45.725271+0000","last_became_active":"2024-06-29T14:22:43.703240+0000","last_became_peered":"2024-06-29T14:22:43.703240+0000","last_unstale":"2024-06-29T14:22:45.725271+0000","last_undegraded":"2024-06-29T14:22:45.725271+0000","last_fullsized":"2024-06-29T14:22:45.725271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:26.060090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315785+0000","last_change":"2024-06-29T14:22:44.715784+0000","last_active":"2024-06-29T14:22:46.315785+0000","last_peered":"2024-06-29T14:22:46.315785+0000","last_clean":"2024-06-29T14:22:46.315785+0000","last_became_active":"2024-06-29T14:22:44.715472+0000","last_became_peered":"2024-06-29T14:22:44.715472+0000","last_unstale":"2024-06-29T14:22:46.315785+0000","last_undegraded":"2024-06-29T14:22:46.315785+0000","last_fullsized":"2024-06-29T14:22:46.315785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:01.618482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703393+0000","last_change":"2024-06-29T14:22:43.700262+0000","last_active":"2024-06-29T14:22:45.703393+0000","last_peered":"2024-06-29T14:22:45.703393+0000","last_clean":"2024-06-29T14:22:45.703393+0000","last_became_active":"2024-06-29T14:22:43.699933+0000","last_became_peered":"2024-06-29T14:22:43.699933+0000","last_unstale":"2024-06-29T14:22:45.703393+0000","last_undegraded":"2024-06-29T14:22:45.703393+0000","last_fullsized":"2024-06-29T14:22:45.703393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:49.306537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316305+0000","last_change":"2024-06-29T14:22:44.717647+0000","last_active":"2024-06-29T14:22:46.316305+0000","last_peered":"2024-06-29T14:22:46.316305+0000","last_clean":"2024-06-29T14:22:46.316305+0000","last_became_active":"2024-06-29T14:22:44.717352+0000","last_became_peered":"2024-06-29T14:22:44.717352+0000","last_unstale":"2024-06-29T14:22:46.316305+0000","last_undegraded":"2024-06-29T14:22:46.316305+0000","last_fullsized":"2024-06-29T14:22:46.316305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:19.819693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725473+0000","last_change":"2024-06-29T14:22:43.706725+0000","last_active":"2024-06-29T14:22:45.725473+0000","last_peered":"2024-06-29T14:22:45.725473+0000","last_clean":"2024-06-29T14:22:45.725473+0000","last_became_active":"2024-06-29T14:22:43.706166+0000","last_became_peered":"2024-06-29T14:22:43.706166+0000","last_unstale":"2024-06-29T14:22:45.725473+0000","last_undegraded":"2024-06-29T14:22:45.725473+0000","last_fullsized":"2024-06-29T14:22:45.725473+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316526+0000","last_change":"2024-06-29T14:22:44.722510+0000","last_active":"2024-06-29T14:22:46.316526+0000","last_peered":"2024-06-29T14:22:46.316526+0000","last_clean":"2024-06-29T14:22:46.316526+0000","last_became_active":"2024-06-29T14:22:44.722129+0000","last_became_peered":"2024-06-29T14:22:44.722129+0000","last_unstale":"2024-06-29T14:22:46.316526+0000","last_undegraded":"2024-06-29T14:22:46.316526+0000","last_fullsized":"2024-06-29T14:22:46.316526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:31:30.258889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701676+0000","last_change":"2024-06-29T14:22:43.699816+0000","last_active":"2024-06-29T14:22:45.701676+0000","last_peered":"2024-06-29T14:22:45.701676+0000","last_clean":"2024-06-29T14:22:45.701676+0000","last_became_active":"2024-06-29T14:22:43.699427+0000","last_became_peered":"2024-06-29T14:22:43.699427+0000","last_unstale":"2024-06-29T14:22:45.701676+0000","last_undegraded":"2024-06-29T14:22:45.701676+0000","last_fullsized":"2024-06-29T14:22:45.701676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704777+0000","last_change":"2024-06-29T14:22:44.707863+0000","last_active":"2024-06-29T14:22:45.704777+0000","last_peered":"2024-06-29T14:22:45.704777+0000","last_clean":"2024-06-29T14:22:45.704777+0000","last_became_active":"2024-06-29T14:22:44.707498+0000","last_became_peered":"2024-06-29T14:22:44.707498+0000","last_unstale":"2024-06-29T14:22:45.704777+0000","last_undegraded":"2024-06-29T14:22:45.704777+0000","last_fullsized":"2024-06-29T14:22:45.704777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:50.421143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725446+0000","last_change":"2024-06-29T14:22:43.695216+0000","last_active":"2024-06-29T14:22:45.725446+0000","last_peered":"2024-06-29T14:22:45.725446+0000","last_clean":"2024-06-29T14:22:45.725446+0000","last_became_active":"2024-06-29T14:22:43.694485+0000","last_became_peered":"2024-06-29T14:22:43.694485+0000","last_unstale":"2024-06-29T14:22:45.725446+0000","last_undegraded":"2024-06-29T14:22:45.725446+0000","last_fullsized":"2024-06-29T14:22:45.725446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316151+0000","last_change":"2024-06-29T14:22:44.720432+0000","last_active":"2024-06-29T14:22:46.316151+0000","last_peered":"2024-06-29T14:22:46.316151+0000","last_clean":"2024-06-29T14:22:46.316151+0000","last_became_active":"2024-06-29T14:22:44.720099+0000","last_became_peered":"2024-06-29T14:22:44.720099+0000","last_unstale":"2024-06-29T14:22:46.316151+0000","last_undegraded":"2024-06-29T14:22:46.316151+0000","last_fullsized":"2024-06-29T14:22:46.316151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:50.421143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315581+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:22:46.315581+0000","last_peered":"2024-06-29T14:22:46.315581+0000","last_clean":"2024-06-29T14:22:46.315581+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:22:46.315581+0000","last_undegraded":"2024-06-29T14:22:46.315581+0000","last_fullsized":"2024-06-29T14:22:46.315581+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703988+0000","last_change":"2024-06-29T14:22:43.694439+0000","last_active":"2024-06-29T14:22:45.703988+0000","last_peered":"2024-06-29T14:22:45.703988+0000","last_clean":"2024-06-29T14:22:45.703988+0000","last_became_active":"2024-06-29T14:22:43.694115+0000","last_became_peered":"2024-06-29T14:22:43.694115+0000","last_unstale":"2024-06-29T14:22:45.703988+0000","last_undegraded":"2024-06-29T14:22:45.703988+0000","last_fullsized":"2024-06-29T14:22:45.703988+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314689+0000","last_change":"2024-06-29T14:22:44.707439+0000","last_active":"2024-06-29T14:22:46.314689+0000","last_peered":"2024-06-29T14:22:46.314689+0000","last_clean":"2024-06-29T14:22:46.314689+0000","last_became_active":"2024-06-29T14:22:44.707120+0000","last_became_peered":"2024-06-29T14:22:44.707120+0000","last_unstale":"2024-06-29T14:22:46.314689+0000","last_undegraded":"2024-06-29T14:22:46.314689+0000","last_fullsized":"2024-06-29T14:22:46.314689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315705+0000","last_change":"2024-06-29T14:22:43.712228+0000","last_active":"2024-06-29T14:22:46.315705+0000","last_peered":"2024-06-29T14:22:46.315705+0000","last_clean":"2024-06-29T14:22:46.315705+0000","last_became_active":"2024-06-29T14:22:43.711907+0000","last_became_peered":"2024-06-29T14:22:43.711907+0000","last_unstale":"2024-06-29T14:22:46.315705+0000","last_undegraded":"2024-06-29T14:22:46.315705+0000","last_fullsized":"2024-06-29T14:22:46.315705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315247+0000","last_change":"2024-06-29T14:22:44.705791+0000","last_active":"2024-06-29T14:22:46.315247+0000","last_peered":"2024-06-29T14:22:46.315247+0000","last_clean":"2024-06-29T14:22:46.315247+0000","last_became_active":"2024-06-29T14:22:44.705029+0000","last_became_peered":"2024-06-29T14:22:44.705029+0000","last_unstale":"2024-06-29T14:22:46.315247+0000","last_undegraded":"2024-06-29T14:22:46.315247+0000","last_fullsized":"2024-06-29T14:22:46.315247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725377+0000","last_change":"2024-06-29T14:22:43.706346+0000","last_active":"2024-06-29T14:22:45.725377+0000","last_peered":"2024-06-29T14:22:45.725377+0000","last_clean":"2024-06-29T14:22:45.725377+0000","last_became_active":"2024-06-29T14:22:43.706016+0000","last_became_peered":"2024-06-29T14:22:43.706016+0000","last_unstale":"2024-06-29T14:22:45.725377+0000","last_undegraded":"2024-06-29T14:22:45.725377+0000","last_fullsized":"2024-06-29T14:22:45.725377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701343+0000","last_change":"2024-06-29T14:22:44.692833+0000","last_active":"2024-06-29T14:22:45.701343+0000","last_peered":"2024-06-29T14:22:45.701343+0000","last_clean":"2024-06-29T14:22:45.701343+0000","last_became_active":"2024-06-29T14:22:44.692138+0000","last_became_peered":"2024-06-29T14:22:44.692138+0000","last_unstale":"2024-06-29T14:22:45.701343+0000","last_undegraded":"2024-06-29T14:22:45.701343+0000","last_fullsized":"2024-06-29T14:22:45.701343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.705065+0000","last_change":"2024-06-29T14:22:43.703727+0000","last_active":"2024-06-29T14:22:45.705065+0000","last_peered":"2024-06-29T14:22:45.705065+0000","last_clean":"2024-06-29T14:22:45.705065+0000","last_became_active":"2024-06-29T14:22:43.703413+0000","last_became_peered":"2024-06-29T14:22:43.703413+0000","last_unstale":"2024-06-29T14:22:45.705065+0000","last_undegraded":"2024-06-29T14:22:45.705065+0000","last_fullsized":"2024-06-29T14:22:45.705065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703554+0000","last_change":"2024-06-29T14:22:44.712628+0000","last_active":"2024-06-29T14:22:45.703554+0000","last_peered":"2024-06-29T14:22:45.703554+0000","last_clean":"2024-06-29T14:22:45.703554+0000","last_became_active":"2024-06-29T14:22:44.712293+0000","last_became_peered":"2024-06-29T14:22:44.712293+0000","last_unstale":"2024-06-29T14:22:45.703554+0000","last_undegraded":"2024-06-29T14:22:45.703554+0000","last_fullsized":"2024-06-29T14:22:45.703554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:40.770948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701488+0000","last_change":"2024-06-29T14:22:43.702999+0000","last_active":"2024-06-29T14:22:45.701488+0000","last_peered":"2024-06-29T14:22:45.701488+0000","last_clean":"2024-06-29T14:22:45.701488+0000","last_became_active":"2024-06-29T14:22:43.702670+0000","last_became_peered":"2024-06-29T14:22:43.702670+0000","last_unstale":"2024-06-29T14:22:45.701488+0000","last_undegraded":"2024-06-29T14:22:45.701488+0000","last_fullsized":"2024-06-29T14:22:45.701488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704843+0000","last_change":"2024-06-29T14:22:44.708301+0000","last_active":"2024-06-29T14:22:45.704843+0000","last_peered":"2024-06-29T14:22:45.704843+0000","last_clean":"2024-06-29T14:22:45.704843+0000","last_became_active":"2024-06-29T14:22:44.707732+0000","last_became_peered":"2024-06-29T14:22:44.707732+0000","last_unstale":"2024-06-29T14:22:45.704843+0000","last_undegraded":"2024-06-29T14:22:45.704843+0000","last_fullsized":"2024-06-29T14:22:45.704843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:55:43.420129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316302+0000","last_change":"2024-06-29T14:22:43.711647+0000","last_active":"2024-06-29T14:22:46.316302+0000","last_peered":"2024-06-29T14:22:46.316302+0000","last_clean":"2024-06-29T14:22:46.316302+0000","last_became_active":"2024-06-29T14:22:43.710944+0000","last_became_peered":"2024-06-29T14:22:43.710944+0000","last_unstale":"2024-06-29T14:22:46.316302+0000","last_undegraded":"2024-06-29T14:22:46.316302+0000","last_fullsized":"2024-06-29T14:22:46.316302+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:30.671807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701389+0000","last_change":"2024-06-29T14:22:44.696383+0000","last_active":"2024-06-29T14:22:45.701389+0000","last_peered":"2024-06-29T14:22:45.701389+0000","last_clean":"2024-06-29T14:22:45.701389+0000","last_became_active":"2024-06-29T14:22:44.696071+0000","last_became_peered":"2024-06-29T14:22:44.696071+0000","last_unstale":"2024-06-29T14:22:45.701389+0000","last_undegraded":"2024-06-29T14:22:45.701389+0000","last_fullsized":"2024-06-29T14:22:45.701389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701505+0000","last_change":"2024-06-29T14:22:43.703248+0000","last_active":"2024-06-29T14:22:45.701505+0000","last_peered":"2024-06-29T14:22:45.701505+0000","last_clean":"2024-06-29T14:22:45.701505+0000","last_became_active":"2024-06-29T14:22:43.702925+0000","last_became_peered":"2024-06-29T14:22:43.702925+0000","last_unstale":"2024-06-29T14:22:45.701505+0000","last_undegraded":"2024-06-29T14:22:45.701505+0000","last_fullsized":"2024-06-29T14:22:45.701505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:39.003651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314975+0000","last_change":"2024-06-29T14:22:44.722533+0000","last_active":"2024-06-29T14:22:46.314975+0000","last_peered":"2024-06-29T14:22:46.314975+0000","last_clean":"2024-06-29T14:22:46.314975+0000","last_became_active":"2024-06-29T14:22:44.722296+0000","last_became_peered":"2024-06-29T14:22:44.722296+0000","last_unstale":"2024-06-29T14:22:46.314975+0000","last_undegraded":"2024-06-29T14:22:46.314975+0000","last_fullsized":"2024-06-29T14:22:46.314975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:29.350494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315500+0000","last_change":"2024-06-29T14:22:43.705703+0000","last_active":"2024-06-29T14:22:46.315500+0000","last_peered":"2024-06-29T14:22:46.315500+0000","last_clean":"2024-06-29T14:22:46.315500+0000","last_became_active":"2024-06-29T14:22:43.705306+0000","last_became_peered":"2024-06-29T14:22:43.705306+0000","last_unstale":"2024-06-29T14:22:46.315500+0000","last_undegraded":"2024-06-29T14:22:46.315500+0000","last_fullsized":"2024-06-29T14:22:46.315500+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314958+0000","last_change":"2024-06-29T14:22:44.712109+0000","last_active":"2024-06-29T14:22:46.314958+0000","last_peered":"2024-06-29T14:22:46.314958+0000","last_clean":"2024-06-29T14:22:46.314958+0000","last_became_active":"2024-06-29T14:22:44.711374+0000","last_became_peered":"2024-06-29T14:22:44.711374+0000","last_unstale":"2024-06-29T14:22:46.314958+0000","last_undegraded":"2024-06-29T14:22:46.314958+0000","last_fullsized":"2024-06-29T14:22:46.314958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315289+0000","last_change":"2024-06-29T14:22:43.711036+0000","last_active":"2024-06-29T14:22:46.315289+0000","last_peered":"2024-06-29T14:22:46.315289+0000","last_clean":"2024-06-29T14:22:46.315289+0000","last_became_active":"2024-06-29T14:22:43.710662+0000","last_became_peered":"2024-06-29T14:22:43.710662+0000","last_unstale":"2024-06-29T14:22:46.315289+0000","last_undegraded":"2024-06-29T14:22:46.315289+0000","last_fullsized":"2024-06-29T14:22:46.315289+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:01.984922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704030+0000","last_change":"2024-06-29T14:22:44.699330+0000","last_active":"2024-06-29T14:22:45.704030+0000","last_peered":"2024-06-29T14:22:45.704030+0000","last_clean":"2024-06-29T14:22:45.704030+0000","last_became_active":"2024-06-29T14:22:44.698611+0000","last_became_peered":"2024-06-29T14:22:44.698611+0000","last_unstale":"2024-06-29T14:22:45.704030+0000","last_undegraded":"2024-06-29T14:22:45.704030+0000","last_fullsized":"2024-06-29T14:22:45.704030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701414+0000","last_change":"2024-06-29T14:22:43.711354+0000","last_active":"2024-06-29T14:22:45.701414+0000","last_peered":"2024-06-29T14:22:45.701414+0000","last_clean":"2024-06-29T14:22:45.701414+0000","last_became_active":"2024-06-29T14:22:43.710734+0000","last_became_peered":"2024-06-29T14:22:43.710734+0000","last_unstale":"2024-06-29T14:22:45.701414+0000","last_undegraded":"2024-06-29T14:22:45.701414+0000","last_fullsized":"2024-06-29T14:22:45.701414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315307+0000","last_change":"2024-06-29T14:22:44.694835+0000","last_active":"2024-06-29T14:22:46.315307+0000","last_peered":"2024-06-29T14:22:46.315307+0000","last_clean":"2024-06-29T14:22:46.315307+0000","last_became_active":"2024-06-29T14:22:44.694463+0000","last_became_peered":"2024-06-29T14:22:44.694463+0000","last_unstale":"2024-06-29T14:22:46.315307+0000","last_undegraded":"2024-06-29T14:22:46.315307+0000","last_fullsized":"2024-06-29T14:22:46.315307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:01.618482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701665+0000","last_change":"2024-06-29T14:22:43.705353+0000","last_active":"2024-06-29T14:22:45.701665+0000","last_peered":"2024-06-29T14:22:45.701665+0000","last_clean":"2024-06-29T14:22:45.701665+0000","last_became_active":"2024-06-29T14:22:43.705002+0000","last_became_peered":"2024-06-29T14:22:43.705002+0000","last_unstale":"2024-06-29T14:22:45.701665+0000","last_undegraded":"2024-06-29T14:22:45.701665+0000","last_fullsized":"2024-06-29T14:22:45.701665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:30.671807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315007+0000","last_change":"2024-06-29T14:22:44.710202+0000","last_active":"2024-06-29T14:22:46.315007+0000","last_peered":"2024-06-29T14:22:46.315007+0000","last_clean":"2024-06-29T14:22:46.315007+0000","last_became_active":"2024-06-29T14:22:44.709900+0000","last_became_peered":"2024-06-29T14:22:44.709900+0000","last_unstale":"2024-06-29T14:22:46.315007+0000","last_undegraded":"2024-06-29T14:22:46.315007+0000","last_fullsized":"2024-06-29T14:22:46.315007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316125+0000","last_change":"2024-06-29T14:22:43.712161+0000","last_active":"2024-06-29T14:22:46.316125+0000","last_peered":"2024-06-29T14:22:46.316125+0000","last_clean":"2024-06-29T14:22:46.316125+0000","last_became_active":"2024-06-29T14:22:43.710970+0000","last_became_peered":"2024-06-29T14:22:43.710970+0000","last_unstale":"2024-06-29T14:22:46.316125+0000","last_undegraded":"2024-06-29T14:22:46.316125+0000","last_fullsized":"2024-06-29T14:22:46.316125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:43.918058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315157+0000","last_change":"2024-06-29T14:22:44.702756+0000","last_active":"2024-06-29T14:22:46.315157+0000","last_peered":"2024-06-29T14:22:46.315157+0000","last_clean":"2024-06-29T14:22:46.315157+0000","last_became_active":"2024-06-29T14:22:44.702438+0000","last_became_peered":"2024-06-29T14:22:44.702438+0000","last_unstale":"2024-06-29T14:22:46.315157+0000","last_undegraded":"2024-06-29T14:22:46.315157+0000","last_fullsized":"2024-06-29T14:22:46.315157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704611+0000","last_change":"2024-06-29T14:22:44.697060+0000","last_active":"2024-06-29T14:22:45.704611+0000","last_peered":"2024-06-29T14:22:45.704611+0000","last_clean":"2024-06-29T14:22:45.704611+0000","last_became_active":"2024-06-29T14:22:44.696660+0000","last_became_peered":"2024-06-29T14:22:44.696660+0000","last_unstale":"2024-06-29T14:22:45.704611+0000","last_undegraded":"2024-06-29T14:22:45.704611+0000","last_fullsized":"2024-06-29T14:22:45.704611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:51:55.034132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316026+0000","last_change":"2024-06-29T14:22:43.711693+0000","last_active":"2024-06-29T14:22:46.316026+0000","last_peered":"2024-06-29T14:22:46.316026+0000","last_clean":"2024-06-29T14:22:46.316026+0000","last_became_active":"2024-06-29T14:22:43.710688+0000","last_became_peered":"2024-06-29T14:22:43.710688+0000","last_unstale":"2024-06-29T14:22:46.316026+0000","last_undegraded":"2024-06-29T14:22:46.316026+0000","last_fullsized":"2024-06-29T14:22:46.316026+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704601+0000","last_change":"2024-06-29T14:22:44.697174+0000","last_active":"2024-06-29T14:22:45.704601+0000","last_peered":"2024-06-29T14:22:45.704601+0000","last_clean":"2024-06-29T14:22:45.704601+0000","last_became_active":"2024-06-29T14:22:44.696735+0000","last_became_peered":"2024-06-29T14:22:44.696735+0000","last_unstale":"2024-06-29T14:22:45.704601+0000","last_undegraded":"2024-06-29T14:22:45.704601+0000","last_fullsized":"2024-06-29T14:22:45.704601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:11.627629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316149+0000","last_change":"2024-06-29T14:22:43.711172+0000","last_active":"2024-06-29T14:22:46.316149+0000","last_peered":"2024-06-29T14:22:46.316149+0000","last_clean":"2024-06-29T14:22:46.316149+0000","last_became_active":"2024-06-29T14:22:43.710779+0000","last_became_peered":"2024-06-29T14:22:43.710779+0000","last_unstale":"2024-06-29T14:22:46.316149+0000","last_undegraded":"2024-06-29T14:22:46.316149+0000","last_fullsized":"2024-06-29T14:22:46.316149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:39.003651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315403+0000","last_change":"2024-06-29T14:22:44.708519+0000","last_active":"2024-06-29T14:22:46.315403+0000","last_peered":"2024-06-29T14:22:46.315403+0000","last_clean":"2024-06-29T14:22:46.315403+0000","last_became_active":"2024-06-29T14:22:44.708212+0000","last_became_peered":"2024-06-29T14:22:44.708212+0000","last_unstale":"2024-06-29T14:22:46.315403+0000","last_undegraded":"2024-06-29T14:22:46.315403+0000","last_fullsized":"2024-06-29T14:22:46.315403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315117+0000","last_change":"2024-06-29T14:22:43.706244+0000","last_active":"2024-06-29T14:22:46.315117+0000","last_peered":"2024-06-29T14:22:46.315117+0000","last_clean":"2024-06-29T14:22:46.315117+0000","last_became_active":"2024-06-29T14:22:43.705937+0000","last_became_peered":"2024-06-29T14:22:43.705937+0000","last_unstale":"2024-06-29T14:22:46.315117+0000","last_undegraded":"2024-06-29T14:22:46.315117+0000","last_fullsized":"2024-06-29T14:22:46.315117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315107+0000","last_change":"2024-06-29T14:22:44.707925+0000","last_active":"2024-06-29T14:22:46.315107+0000","last_peered":"2024-06-29T14:22:46.315107+0000","last_clean":"2024-06-29T14:22:46.315107+0000","last_became_active":"2024-06-29T14:22:44.707640+0000","last_became_peered":"2024-06-29T14:22:44.707640+0000","last_unstale":"2024-06-29T14:22:46.315107+0000","last_undegraded":"2024-06-29T14:22:46.315107+0000","last_fullsized":"2024-06-29T14:22:46.315107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:25.905572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315669+0000","last_change":"2024-06-29T14:22:43.711121+0000","last_active":"2024-06-29T14:22:46.315669+0000","last_peered":"2024-06-29T14:22:46.315669+0000","last_clean":"2024-06-29T14:22:46.315669+0000","last_became_active":"2024-06-29T14:22:43.710708+0000","last_became_peered":"2024-06-29T14:22:43.710708+0000","last_unstale":"2024-06-29T14:22:46.315669+0000","last_undegraded":"2024-06-29T14:22:46.315669+0000","last_fullsized":"2024-06-29T14:22:46.315669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:49.306537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:22:53.604 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:22:53.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.757+0000 7f8c29481640 1 -- 172.21.15.81:0/3045880069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c240d0940 msgr2=0x7f8c240b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.757+0000 7f8c29481640 1 --2- 172.21.15.81:0/3045880069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c240d0940 0x7f8c240b4f40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8c24073550 tx=0x7f8c1c02a980 comp rx=0 tx=0).stop 2024-06-29T14:22:53.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.757+0000 7f8c29481640 1 -- 172.21.15.81:0/3045880069 shutdown_connections 2024-06-29T14:22:53.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.757+0000 7f8c29481640 1 --2- 172.21.15.81:0/3045880069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c240d0940 0x7f8c240b4f40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.757+0000 7f8c29481640 1 --2- 172.21.15.81:0/3045880069 >> v2:172.21.15.167:3300/0 conn(0x7f8c240b3200 0x7f8c240cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.757+0000 7f8c29481640 1 -- 172.21.15.81:0/3045880069 >> 172.21.15.81:0/3045880069 conn(0x7f8c240b3680 msgr2=0x7f8c240b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.757+0000 7f8c29481640 1 -- 172.21.15.81:0/3045880069 shutdown_connections 2024-06-29T14:22:53.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.757+0000 7f8c29481640 1 -- 172.21.15.81:0/3045880069 wait complete. 2024-06-29T14:22:53.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.758+0000 7f8c29481640 1 Processor -- start 2024-06-29T14:22:53.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.759+0000 7f8c29481640 1 -- start start 2024-06-29T14:22:53.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c29481640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c240b3200 0x7f8c241dea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c29481640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c240cccf0 0x7f8c241dda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c29481640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c241d8340 0x7f8c240c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c29481640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c240b5bb0 con 0x7f8c241d8340 2024-06-29T14:22:53.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c29481640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8c240b5a30 con 0x7f8c240b3200 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c29481640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8c240b5d30 con 0x7f8c240cccf0 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c237fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c241d8340 0x7f8c240c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c237fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c241d8340 0x7f8c240c6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36350/0 (socket says 172.21.15.81:36350) 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c237fe640 1 -- 172.21.15.81:0/951728416 learned_addr learned my addr 172.21.15.81:0/951728416 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c22ffd640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c240b3200 0x7f8c241dea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c1a7fc640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c240cccf0 0x7f8c241dda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c237fe640 1 -- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c240cccf0 msgr2=0x7f8c241dda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c237fe640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c240cccf0 0x7f8c241dda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c237fe640 1 -- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c240b3200 msgr2=0x7f8c241dea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c237fe640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c240b3200 0x7f8c241dea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c237fe640 1 -- 172.21.15.81:0/951728416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c240c5120 con 0x7f8c241d8340 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.760+0000 7f8c237fe640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c241d8340 0x7f8c240c6b90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c02af70 tx=0x7f8c1c003ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:53.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.761+0000 7f8c1bfff640 1 -- 172.21.15.81:0/951728416 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c034040 con 0x7f8c241d8340 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.761+0000 7f8c1bfff640 1 -- 172.21.15.81:0/951728416 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c1c004640 con 0x7f8c241d8340 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.761+0000 7f8c29481640 1 -- 172.21.15.81:0/951728416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c241dcc80 con 0x7f8c241d8340 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.761+0000 7f8c1bfff640 1 -- 172.21.15.81:0/951728416 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c0456e0 con 0x7f8c241d8340 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.761+0000 7f8c29481640 1 -- 172.21.15.81:0/951728416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c241dd240 con 0x7f8c241d8340 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.762+0000 7f8c29481640 1 -- 172.21.15.81:0/951728416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c240d03b0 con 0x7f8c241d8340 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.762+0000 7f8c1bfff640 1 -- 172.21.15.81:0/951728416 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8c1c037020 con 0x7f8c241d8340 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.763+0000 7f8c1bfff640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c04076300 0x7f8c04078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.763+0000 7f8c1bfff640 1 -- 172.21.15.81:0/951728416 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8c1c0b90d0 con 0x7f8c241d8340 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.763+0000 7f8c22ffd640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c04076300 0x7f8c04078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.764+0000 7f8c22ffd640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c04076300 0x7f8c04078810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8c14005fd0 tx=0x7f8c1400a250 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:53.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.767+0000 7f8c1bfff640 1 -- 172.21.15.81:0/951728416 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8c1c0823e0 con 0x7f8c241d8340 2024-06-29T14:22:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.946+0000 7f8c29481640 1 -- 172.21.15.81:0/951728416 --> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8c240c5590 con 0x7f8c04076300 2024-06-29T14:22:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.951+0000 7f8c1bfff640 1 -- 172.21.15.81:0/951728416 <== mgr.4126 v2:172.21.15.167:6832/3101542440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7f8c240c5590 con 0x7f8c04076300 2024-06-29T14:22:53.952 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:53.953 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:22:53.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 -- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c04076300 msgr2=0x7f8c04078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c04076300 0x7f8c04078810 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8c14005fd0 tx=0x7f8c1400a250 comp rx=0 tx=0).stop 2024-06-29T14:22:53.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 -- 172.21.15.81:0/951728416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c241d8340 msgr2=0x7f8c240c6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:53.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c241d8340 0x7f8c240c6b90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c02af70 tx=0x7f8c1c003ef0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 -- 172.21.15.81:0/951728416 shutdown_connections 2024-06-29T14:22:53.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c04076300 0x7f8c04078810 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c241d8340 0x7f8c240c6b90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c240cccf0 0x7f8c241dda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 --2- 172.21.15.81:0/951728416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c240b3200 0x7f8c241dea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:53.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 -- 172.21.15.81:0/951728416 >> 172.21.15.81:0/951728416 conn(0x7f8c240b3680 msgr2=0x7f8c24070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:53.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 -- 172.21.15.81:0/951728416 shutdown_connections 2024-06-29T14:22:53.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:53.954+0000 7f8c29481640 1 -- 172.21.15.81:0/951728416 wait complete. 2024-06-29T14:22:53.963 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-06-29T14:22:52.706240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219020,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755700,"statfs":{"total":773094113280,"available":772869836800,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.020662"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316337+0000","last_change":"2024-06-29T14:22:44.722050+0000","last_active":"2024-06-29T14:22:46.316337+0000","last_peered":"2024-06-29T14:22:46.316337+0000","last_clean":"2024-06-29T14:22:46.316337+0000","last_became_active":"2024-06-29T14:22:44.721719+0000","last_became_peered":"2024-06-29T14:22:44.721719+0000","last_unstale":"2024-06-29T14:22:46.316337+0000","last_undegraded":"2024-06-29T14:22:46.316337+0000","last_fullsized":"2024-06-29T14:22:46.316337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:55:43.420129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701534+0000","last_change":"2024-06-29T14:22:44.709906+0000","last_active":"2024-06-29T14:22:45.701534+0000","last_peered":"2024-06-29T14:22:45.701534+0000","last_clean":"2024-06-29T14:22:45.701534+0000","last_became_active":"2024-06-29T14:22:44.709406+0000","last_became_peered":"2024-06-29T14:22:44.709406+0000","last_unstale":"2024-06-29T14:22:45.701534+0000","last_undegraded":"2024-06-29T14:22:45.701534+0000","last_fullsized":"2024-06-29T14:22:45.701534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:29.350494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315145+0000","last_change":"2024-06-29T14:22:44.705357+0000","last_active":"2024-06-29T14:22:46.315145+0000","last_peered":"2024-06-29T14:22:46.315145+0000","last_clean":"2024-06-29T14:22:46.315145+0000","last_became_active":"2024-06-29T14:22:44.704402+0000","last_became_peered":"2024-06-29T14:22:44.704402+0000","last_unstale":"2024-06-29T14:22:46.315145+0000","last_undegraded":"2024-06-29T14:22:46.315145+0000","last_fullsized":"2024-06-29T14:22:46.315145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704481+0000","last_change":"2024-06-29T14:22:44.708801+0000","last_active":"2024-06-29T14:22:45.704481+0000","last_peered":"2024-06-29T14:22:45.704481+0000","last_clean":"2024-06-29T14:22:45.704481+0000","last_became_active":"2024-06-29T14:22:44.708456+0000","last_became_peered":"2024-06-29T14:22:44.708456+0000","last_unstale":"2024-06-29T14:22:45.704481+0000","last_undegraded":"2024-06-29T14:22:45.704481+0000","last_fullsized":"2024-06-29T14:22:45.704481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:40.172805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315866+0000","last_change":"2024-06-29T14:22:44.719917+0000","last_active":"2024-06-29T14:22:46.315866+0000","last_peered":"2024-06-29T14:22:46.315866+0000","last_clean":"2024-06-29T14:22:46.315866+0000","last_became_active":"2024-06-29T14:22:44.719617+0000","last_became_peered":"2024-06-29T14:22:44.719617+0000","last_unstale":"2024-06-29T14:22:46.315866+0000","last_undegraded":"2024-06-29T14:22:46.315866+0000","last_fullsized":"2024-06-29T14:22:46.315866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:40.172805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704263+0000","last_change":"2024-06-29T14:22:44.701697+0000","last_active":"2024-06-29T14:22:45.704263+0000","last_peered":"2024-06-29T14:22:45.704263+0000","last_clean":"2024-06-29T14:22:45.704263+0000","last_became_active":"2024-06-29T14:22:44.701325+0000","last_became_peered":"2024-06-29T14:22:44.701325+0000","last_unstale":"2024-06-29T14:22:45.704263+0000","last_undegraded":"2024-06-29T14:22:45.704263+0000","last_fullsized":"2024-06-29T14:22:45.704263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:01.618482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315725+0000","last_change":"2024-06-29T14:22:44.720635+0000","last_active":"2024-06-29T14:22:46.315725+0000","last_peered":"2024-06-29T14:22:46.315725+0000","last_clean":"2024-06-29T14:22:46.315725+0000","last_became_active":"2024-06-29T14:22:44.720316+0000","last_became_peered":"2024-06-29T14:22:44.720316+0000","last_unstale":"2024-06-29T14:22:46.315725+0000","last_undegraded":"2024-06-29T14:22:46.315725+0000","last_fullsized":"2024-06-29T14:22:46.315725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:52.918063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314686+0000","last_change":"2024-06-29T14:22:44.706868+0000","last_active":"2024-06-29T14:22:46.314686+0000","last_peered":"2024-06-29T14:22:46.314686+0000","last_clean":"2024-06-29T14:22:46.314686+0000","last_became_active":"2024-06-29T14:22:44.706298+0000","last_became_peered":"2024-06-29T14:22:44.706298+0000","last_unstale":"2024-06-29T14:22:46.314686+0000","last_undegraded":"2024-06-29T14:22:46.314686+0000","last_fullsized":"2024-06-29T14:22:46.314686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316405+0000","last_change":"2024-06-29T14:22:44.722834+0000","last_active":"2024-06-29T14:22:46.316405+0000","last_peered":"2024-06-29T14:22:46.316405+0000","last_clean":"2024-06-29T14:22:46.316405+0000","last_became_active":"2024-06-29T14:22:44.722345+0000","last_became_peered":"2024-06-29T14:22:44.722345+0000","last_unstale":"2024-06-29T14:22:46.316405+0000","last_undegraded":"2024-06-29T14:22:46.316405+0000","last_fullsized":"2024-06-29T14:22:46.316405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:37:57.057668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315569+0000","last_change":"2024-06-29T14:22:44.702528+0000","last_active":"2024-06-29T14:22:46.315569+0000","last_peered":"2024-06-29T14:22:46.315569+0000","last_clean":"2024-06-29T14:22:46.315569+0000","last_became_active":"2024-06-29T14:22:44.702246+0000","last_became_peered":"2024-06-29T14:22:44.702246+0000","last_unstale":"2024-06-29T14:22:46.315569+0000","last_undegraded":"2024-06-29T14:22:46.315569+0000","last_fullsized":"2024-06-29T14:22:46.315569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:29.350494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315346+0000","last_change":"2024-06-29T14:22:44.703308+0000","last_active":"2024-06-29T14:22:46.315346+0000","last_peered":"2024-06-29T14:22:46.315346+0000","last_clean":"2024-06-29T14:22:46.315346+0000","last_became_active":"2024-06-29T14:22:44.703018+0000","last_became_peered":"2024-06-29T14:22:44.703018+0000","last_unstale":"2024-06-29T14:22:46.315346+0000","last_undegraded":"2024-06-29T14:22:46.315346+0000","last_fullsized":"2024-06-29T14:22:46.315346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703510+0000","last_change":"2024-06-29T14:22:44.702902+0000","last_active":"2024-06-29T14:22:45.703510+0000","last_peered":"2024-06-29T14:22:45.703510+0000","last_clean":"2024-06-29T14:22:45.703510+0000","last_became_active":"2024-06-29T14:22:44.702595+0000","last_became_peered":"2024-06-29T14:22:44.702595+0000","last_unstale":"2024-06-29T14:22:45.703510+0000","last_undegraded":"2024-06-29T14:22:45.703510+0000","last_fullsized":"2024-06-29T14:22:45.703510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:46.021153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314835+0000","last_change":"2024-06-29T14:22:44.703127+0000","last_active":"2024-06-29T14:22:46.314835+0000","last_peered":"2024-06-29T14:22:46.314835+0000","last_clean":"2024-06-29T14:22:46.314835+0000","last_became_active":"2024-06-29T14:22:44.702813+0000","last_became_peered":"2024-06-29T14:22:44.702813+0000","last_unstale":"2024-06-29T14:22:46.314835+0000","last_undegraded":"2024-06-29T14:22:46.314835+0000","last_fullsized":"2024-06-29T14:22:46.314835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314658+0000","last_change":"2024-06-29T14:22:44.714140+0000","last_active":"2024-06-29T14:22:46.314658+0000","last_peered":"2024-06-29T14:22:46.314658+0000","last_clean":"2024-06-29T14:22:46.314658+0000","last_became_active":"2024-06-29T14:22:44.713863+0000","last_became_peered":"2024-06-29T14:22:44.713863+0000","last_unstale":"2024-06-29T14:22:46.314658+0000","last_undegraded":"2024-06-29T14:22:46.314658+0000","last_fullsized":"2024-06-29T14:22:46.314658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:16.336127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701353+0000","last_change":"2024-06-29T14:22:44.699851+0000","last_active":"2024-06-29T14:22:45.701353+0000","last_peered":"2024-06-29T14:22:45.701353+0000","last_clean":"2024-06-29T14:22:45.701353+0000","last_became_active":"2024-06-29T14:22:44.697960+0000","last_became_peered":"2024-06-29T14:22:44.697960+0000","last_unstale":"2024-06-29T14:22:45.701353+0000","last_undegraded":"2024-06-29T14:22:45.701353+0000","last_fullsized":"2024-06-29T14:22:45.701353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703910+0000","last_change":"2024-06-29T14:22:44.708069+0000","last_active":"2024-06-29T14:22:45.703910+0000","last_peered":"2024-06-29T14:22:45.703910+0000","last_clean":"2024-06-29T14:22:45.703910+0000","last_became_active":"2024-06-29T14:22:44.707765+0000","last_became_peered":"2024-06-29T14:22:44.707765+0000","last_unstale":"2024-06-29T14:22:45.703910+0000","last_undegraded":"2024-06-29T14:22:45.703910+0000","last_fullsized":"2024-06-29T14:22:45.703910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704975+0000","last_change":"2024-06-29T14:22:44.721271+0000","last_active":"2024-06-29T14:22:45.704975+0000","last_peered":"2024-06-29T14:22:45.704975+0000","last_clean":"2024-06-29T14:22:45.704975+0000","last_became_active":"2024-06-29T14:22:44.720978+0000","last_became_peered":"2024-06-29T14:22:44.720978+0000","last_unstale":"2024-06-29T14:22:45.704975+0000","last_undegraded":"2024-06-29T14:22:45.704975+0000","last_fullsized":"2024-06-29T14:22:45.704975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:37:57.057668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725545+0000","last_change":"2024-06-29T14:22:44.702764+0000","last_active":"2024-06-29T14:22:45.725545+0000","last_peered":"2024-06-29T14:22:45.725545+0000","last_clean":"2024-06-29T14:22:45.725545+0000","last_became_active":"2024-06-29T14:22:44.702408+0000","last_became_peered":"2024-06-29T14:22:44.702408+0000","last_unstale":"2024-06-29T14:22:45.725545+0000","last_undegraded":"2024-06-29T14:22:45.725545+0000","last_fullsized":"2024-06-29T14:22:45.725545+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316657+0000","last_change":"2024-06-29T14:22:44.718193+0000","last_active":"2024-06-29T14:22:46.316657+0000","last_peered":"2024-06-29T14:22:46.316657+0000","last_clean":"2024-06-29T14:22:46.316657+0000","last_became_active":"2024-06-29T14:22:44.717900+0000","last_became_peered":"2024-06-29T14:22:44.717900+0000","last_unstale":"2024-06-29T14:22:46.316657+0000","last_undegraded":"2024-06-29T14:22:46.316657+0000","last_fullsized":"2024-06-29T14:22:46.316657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.810207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315331+0000","last_change":"2024-06-29T14:22:44.701520+0000","last_active":"2024-06-29T14:22:46.315331+0000","last_peered":"2024-06-29T14:22:46.315331+0000","last_clean":"2024-06-29T14:22:46.315331+0000","last_became_active":"2024-06-29T14:22:44.701106+0000","last_became_peered":"2024-06-29T14:22:44.701106+0000","last_unstale":"2024-06-29T14:22:46.315331+0000","last_undegraded":"2024-06-29T14:22:46.315331+0000","last_fullsized":"2024-06-29T14:22:46.315331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315186+0000","last_change":"2024-06-29T14:22:44.717603+0000","last_active":"2024-06-29T14:22:46.315186+0000","last_peered":"2024-06-29T14:22:46.315186+0000","last_clean":"2024-06-29T14:22:46.315186+0000","last_became_active":"2024-06-29T14:22:44.717305+0000","last_became_peered":"2024-06-29T14:22:44.717305+0000","last_unstale":"2024-06-29T14:22:46.315186+0000","last_undegraded":"2024-06-29T14:22:46.315186+0000","last_fullsized":"2024-06-29T14:22:46.315186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704911+0000","last_change":"2024-06-29T14:22:44.717305+0000","last_active":"2024-06-29T14:22:45.704911+0000","last_peered":"2024-06-29T14:22:45.704911+0000","last_clean":"2024-06-29T14:22:45.704911+0000","last_became_active":"2024-06-29T14:22:44.716953+0000","last_became_peered":"2024-06-29T14:22:44.716953+0000","last_unstale":"2024-06-29T14:22:45.704911+0000","last_undegraded":"2024-06-29T14:22:45.704911+0000","last_fullsized":"2024-06-29T14:22:45.704911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:31:30.258889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315247+0000","last_change":"2024-06-29T14:22:44.701947+0000","last_active":"2024-06-29T14:22:46.315247+0000","last_peered":"2024-06-29T14:22:46.315247+0000","last_clean":"2024-06-29T14:22:46.315247+0000","last_became_active":"2024-06-29T14:22:44.701624+0000","last_became_peered":"2024-06-29T14:22:44.701624+0000","last_unstale":"2024-06-29T14:22:46.315247+0000","last_undegraded":"2024-06-29T14:22:46.315247+0000","last_fullsized":"2024-06-29T14:22:46.315247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:52.918063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703990+0000","last_change":"2024-06-29T14:22:44.709275+0000","last_active":"2024-06-29T14:22:45.703990+0000","last_peered":"2024-06-29T14:22:45.703990+0000","last_clean":"2024-06-29T14:22:45.703990+0000","last_became_active":"2024-06-29T14:22:44.708702+0000","last_became_peered":"2024-06-29T14:22:44.708702+0000","last_unstale":"2024-06-29T14:22:45.703990+0000","last_undegraded":"2024-06-29T14:22:45.703990+0000","last_fullsized":"2024-06-29T14:22:45.703990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:25.905572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314785+0000","last_change":"2024-06-29T14:22:44.702814+0000","last_active":"2024-06-29T14:22:46.314785+0000","last_peered":"2024-06-29T14:22:46.314785+0000","last_clean":"2024-06-29T14:22:46.314785+0000","last_became_active":"2024-06-29T14:22:44.702479+0000","last_became_peered":"2024-06-29T14:22:44.702479+0000","last_unstale":"2024-06-29T14:22:46.314785+0000","last_undegraded":"2024-06-29T14:22:46.314785+0000","last_fullsized":"2024-06-29T14:22:46.314785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315590+0000","last_change":"2024-06-29T14:22:44.718581+0000","last_active":"2024-06-29T14:22:46.315590+0000","last_peered":"2024-06-29T14:22:46.315590+0000","last_clean":"2024-06-29T14:22:46.315590+0000","last_became_active":"2024-06-29T14:22:44.718277+0000","last_became_peered":"2024-06-29T14:22:44.718277+0000","last_unstale":"2024-06-29T14:22:46.315590+0000","last_undegraded":"2024-06-29T14:22:46.315590+0000","last_fullsized":"2024-06-29T14:22:46.315590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701423+0000","last_change":"2024-06-29T14:22:44.704193+0000","last_active":"2024-06-29T14:22:45.701423+0000","last_peered":"2024-06-29T14:22:45.701423+0000","last_clean":"2024-06-29T14:22:45.701423+0000","last_became_active":"2024-06-29T14:22:44.703892+0000","last_became_peered":"2024-06-29T14:22:44.703892+0000","last_unstale":"2024-06-29T14:22:45.701423+0000","last_undegraded":"2024-06-29T14:22:45.701423+0000","last_fullsized":"2024-06-29T14:22:45.701423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315350+0000","last_change":"2024-06-29T14:22:44.703952+0000","last_active":"2024-06-29T14:22:46.315350+0000","last_peered":"2024-06-29T14:22:46.315350+0000","last_clean":"2024-06-29T14:22:46.315350+0000","last_became_active":"2024-06-29T14:22:44.703431+0000","last_became_peered":"2024-06-29T14:22:44.703431+0000","last_unstale":"2024-06-29T14:22:46.315350+0000","last_undegraded":"2024-06-29T14:22:46.315350+0000","last_fullsized":"2024-06-29T14:22:46.315350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:40.172805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315348+0000","last_change":"2024-06-29T14:22:44.714045+0000","last_active":"2024-06-29T14:22:46.315348+0000","last_peered":"2024-06-29T14:22:46.315348+0000","last_clean":"2024-06-29T14:22:46.315348+0000","last_became_active":"2024-06-29T14:22:44.713749+0000","last_became_peered":"2024-06-29T14:22:44.713749+0000","last_unstale":"2024-06-29T14:22:46.315348+0000","last_undegraded":"2024-06-29T14:22:46.315348+0000","last_fullsized":"2024-06-29T14:22:46.315348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704721+0000","last_change":"2024-06-29T14:22:44.709553+0000","last_active":"2024-06-29T14:22:45.704721+0000","last_peered":"2024-06-29T14:22:45.704721+0000","last_clean":"2024-06-29T14:22:45.704721+0000","last_became_active":"2024-06-29T14:22:44.709153+0000","last_became_peered":"2024-06-29T14:22:44.709153+0000","last_unstale":"2024-06-29T14:22:45.704721+0000","last_undegraded":"2024-06-29T14:22:45.704721+0000","last_fullsized":"2024-06-29T14:22:45.704721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:19.819693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725353+0000","last_change":"2024-06-29T14:22:44.699167+0000","last_active":"2024-06-29T14:22:45.725353+0000","last_peered":"2024-06-29T14:22:45.725353+0000","last_clean":"2024-06-29T14:22:45.725353+0000","last_became_active":"2024-06-29T14:22:44.698265+0000","last_became_peered":"2024-06-29T14:22:44.698265+0000","last_unstale":"2024-06-29T14:22:45.725353+0000","last_undegraded":"2024-06-29T14:22:45.725353+0000","last_fullsized":"2024-06-29T14:22:45.725353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315125+0000","last_change":"2024-06-29T14:22:44.710341+0000","last_active":"2024-06-29T14:22:46.315125+0000","last_peered":"2024-06-29T14:22:46.315125+0000","last_clean":"2024-06-29T14:22:46.315125+0000","last_became_active":"2024-06-29T14:22:44.709992+0000","last_became_peered":"2024-06-29T14:22:44.709992+0000","last_unstale":"2024-06-29T14:22:46.315125+0000","last_undegraded":"2024-06-29T14:22:46.315125+0000","last_fullsized":"2024-06-29T14:22:46.315125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703805+0000","last_change":"2024-06-29T14:22:44.713512+0000","last_active":"2024-06-29T14:22:45.703805+0000","last_peered":"2024-06-29T14:22:45.703805+0000","last_clean":"2024-06-29T14:22:45.703805+0000","last_became_active":"2024-06-29T14:22:44.713208+0000","last_became_peered":"2024-06-29T14:22:44.713208+0000","last_unstale":"2024-06-29T14:22:45.703805+0000","last_undegraded":"2024-06-29T14:22:45.703805+0000","last_fullsized":"2024-06-29T14:22:45.703805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:31.677131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315120+0000","last_change":"2024-06-29T14:22:43.711234+0000","last_active":"2024-06-29T14:22:46.315120+0000","last_peered":"2024-06-29T14:22:46.315120+0000","last_clean":"2024-06-29T14:22:46.315120+0000","last_became_active":"2024-06-29T14:22:43.710904+0000","last_became_peered":"2024-06-29T14:22:43.710904+0000","last_unstale":"2024-06-29T14:22:46.315120+0000","last_undegraded":"2024-06-29T14:22:46.315120+0000","last_fullsized":"2024-06-29T14:22:46.315120+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315427+0000","last_change":"2024-06-29T14:22:44.706122+0000","last_active":"2024-06-29T14:22:46.315427+0000","last_peered":"2024-06-29T14:22:46.315427+0000","last_clean":"2024-06-29T14:22:46.315427+0000","last_became_active":"2024-06-29T14:22:44.705746+0000","last_became_peered":"2024-06-29T14:22:44.705746+0000","last_unstale":"2024-06-29T14:22:46.315427+0000","last_undegraded":"2024-06-29T14:22:46.315427+0000","last_fullsized":"2024-06-29T14:22:46.315427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:29.350494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315012+0000","last_change":"2024-06-29T14:22:43.700941+0000","last_active":"2024-06-29T14:22:46.315012+0000","last_peered":"2024-06-29T14:22:46.315012+0000","last_clean":"2024-06-29T14:22:46.315012+0000","last_became_active":"2024-06-29T14:22:43.700234+0000","last_became_peered":"2024-06-29T14:22:43.700234+0000","last_unstale":"2024-06-29T14:22:46.315012+0000","last_undegraded":"2024-06-29T14:22:46.315012+0000","last_fullsized":"2024-06-29T14:22:46.315012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315413+0000","last_change":"2024-06-29T14:22:43.707763+0000","last_active":"2024-06-29T14:22:46.315413+0000","last_peered":"2024-06-29T14:22:46.315413+0000","last_clean":"2024-06-29T14:22:46.315413+0000","last_became_active":"2024-06-29T14:22:43.707378+0000","last_became_peered":"2024-06-29T14:22:43.707378+0000","last_unstale":"2024-06-29T14:22:46.315413+0000","last_undegraded":"2024-06-29T14:22:46.315413+0000","last_fullsized":"2024-06-29T14:22:46.315413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:26.060090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314961+0000","last_change":"2024-06-29T14:22:44.705946+0000","last_active":"2024-06-29T14:22:46.314961+0000","last_peered":"2024-06-29T14:22:46.314961+0000","last_clean":"2024-06-29T14:22:46.314961+0000","last_became_active":"2024-06-29T14:22:44.705469+0000","last_became_peered":"2024-06-29T14:22:44.705469+0000","last_unstale":"2024-06-29T14:22:46.314961+0000","last_undegraded":"2024-06-29T14:22:46.314961+0000","last_fullsized":"2024-06-29T14:22:46.314961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704025+0000","last_change":"2024-06-29T14:22:43.698273+0000","last_active":"2024-06-29T14:22:45.704025+0000","last_peered":"2024-06-29T14:22:45.704025+0000","last_clean":"2024-06-29T14:22:45.704025+0000","last_became_active":"2024-06-29T14:22:43.697950+0000","last_became_peered":"2024-06-29T14:22:43.697950+0000","last_unstale":"2024-06-29T14:22:45.704025+0000","last_undegraded":"2024-06-29T14:22:45.704025+0000","last_fullsized":"2024-06-29T14:22:45.704025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701302+0000","last_change":"2024-06-29T14:22:44.708264+0000","last_active":"2024-06-29T14:22:45.701302+0000","last_peered":"2024-06-29T14:22:45.701302+0000","last_clean":"2024-06-29T14:22:45.701302+0000","last_became_active":"2024-06-29T14:22:44.707924+0000","last_became_peered":"2024-06-29T14:22:44.707924+0000","last_unstale":"2024-06-29T14:22:45.701302+0000","last_undegraded":"2024-06-29T14:22:45.701302+0000","last_fullsized":"2024-06-29T14:22:45.701302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315270+0000","last_change":"2024-06-29T14:22:43.706343+0000","last_active":"2024-06-29T14:22:46.315270+0000","last_peered":"2024-06-29T14:22:46.315270+0000","last_clean":"2024-06-29T14:22:46.315270+0000","last_became_active":"2024-06-29T14:22:43.706009+0000","last_became_peered":"2024-06-29T14:22:43.706009+0000","last_unstale":"2024-06-29T14:22:46.315270+0000","last_undegraded":"2024-06-29T14:22:46.315270+0000","last_fullsized":"2024-06-29T14:22:46.315270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704221+0000","last_change":"2024-06-29T14:22:44.717406+0000","last_active":"2024-06-29T14:22:45.704221+0000","last_peered":"2024-06-29T14:22:45.704221+0000","last_clean":"2024-06-29T14:22:45.704221+0000","last_became_active":"2024-06-29T14:22:44.717095+0000","last_became_peered":"2024-06-29T14:22:44.717095+0000","last_unstale":"2024-06-29T14:22:45.704221+0000","last_undegraded":"2024-06-29T14:22:45.704221+0000","last_fullsized":"2024-06-29T14:22:45.704221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:52.918063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704373+0000","last_change":"2024-06-29T14:22:43.701234+0000","last_active":"2024-06-29T14:22:45.704373+0000","last_peered":"2024-06-29T14:22:45.704373+0000","last_clean":"2024-06-29T14:22:45.704373+0000","last_became_active":"2024-06-29T14:22:43.700923+0000","last_became_peered":"2024-06-29T14:22:43.700923+0000","last_unstale":"2024-06-29T14:22:45.704373+0000","last_undegraded":"2024-06-29T14:22:45.704373+0000","last_fullsized":"2024-06-29T14:22:45.704373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315568+0000","last_change":"2024-06-29T14:22:44.708529+0000","last_active":"2024-06-29T14:22:46.315568+0000","last_peered":"2024-06-29T14:22:46.315568+0000","last_clean":"2024-06-29T14:22:46.315568+0000","last_became_active":"2024-06-29T14:22:44.708019+0000","last_became_peered":"2024-06-29T14:22:44.708019+0000","last_unstale":"2024-06-29T14:22:46.315568+0000","last_undegraded":"2024-06-29T14:22:46.315568+0000","last_fullsized":"2024-06-29T14:22:46.315568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701544+0000","last_change":"2024-06-29T14:22:43.705026+0000","last_active":"2024-06-29T14:22:45.701544+0000","last_peered":"2024-06-29T14:22:45.701544+0000","last_clean":"2024-06-29T14:22:45.701544+0000","last_became_active":"2024-06-29T14:22:43.704704+0000","last_became_peered":"2024-06-29T14:22:43.704704+0000","last_unstale":"2024-06-29T14:22:45.701544+0000","last_undegraded":"2024-06-29T14:22:45.701544+0000","last_fullsized":"2024-06-29T14:22:45.701544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703710+0000","last_change":"2024-06-29T14:22:44.716149+0000","last_active":"2024-06-29T14:22:45.703710+0000","last_peered":"2024-06-29T14:22:45.703710+0000","last_clean":"2024-06-29T14:22:45.703710+0000","last_became_active":"2024-06-29T14:22:44.715832+0000","last_became_peered":"2024-06-29T14:22:44.715832+0000","last_unstale":"2024-06-29T14:22:45.703710+0000","last_undegraded":"2024-06-29T14:22:45.703710+0000","last_fullsized":"2024-06-29T14:22:45.703710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:40.008975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315151+0000","last_change":"2024-06-29T14:22:43.707443+0000","last_active":"2024-06-29T14:22:46.315151+0000","last_peered":"2024-06-29T14:22:46.315151+0000","last_clean":"2024-06-29T14:22:46.315151+0000","last_became_active":"2024-06-29T14:22:43.707109+0000","last_became_peered":"2024-06-29T14:22:43.707109+0000","last_unstale":"2024-06-29T14:22:46.315151+0000","last_undegraded":"2024-06-29T14:22:46.315151+0000","last_fullsized":"2024-06-29T14:22:46.315151+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316025+0000","last_change":"2024-06-29T14:22:44.719076+0000","last_active":"2024-06-29T14:22:46.316025+0000","last_peered":"2024-06-29T14:22:46.316025+0000","last_clean":"2024-06-29T14:22:46.316025+0000","last_became_active":"2024-06-29T14:22:44.718736+0000","last_became_peered":"2024-06-29T14:22:44.718736+0000","last_unstale":"2024-06-29T14:22:46.316025+0000","last_undegraded":"2024-06-29T14:22:46.316025+0000","last_fullsized":"2024-06-29T14:22:46.316025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:11.627629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315458+0000","last_change":"2024-06-29T14:22:43.700129+0000","last_active":"2024-06-29T14:22:46.315458+0000","last_peered":"2024-06-29T14:22:46.315458+0000","last_clean":"2024-06-29T14:22:46.315458+0000","last_became_active":"2024-06-29T14:22:43.699771+0000","last_became_peered":"2024-06-29T14:22:43.699771+0000","last_unstale":"2024-06-29T14:22:46.315458+0000","last_undegraded":"2024-06-29T14:22:46.315458+0000","last_fullsized":"2024-06-29T14:22:46.315458+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:53.109228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314834+0000","last_change":"2024-06-29T14:22:44.716496+0000","last_active":"2024-06-29T14:22:46.314834+0000","last_peered":"2024-06-29T14:22:46.314834+0000","last_clean":"2024-06-29T14:22:46.314834+0000","last_became_active":"2024-06-29T14:22:44.716244+0000","last_became_peered":"2024-06-29T14:22:44.716244+0000","last_unstale":"2024-06-29T14:22:46.314834+0000","last_undegraded":"2024-06-29T14:22:46.314834+0000","last_fullsized":"2024-06-29T14:22:46.314834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:26.239540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704937+0000","last_change":"2024-06-29T14:22:43.706155+0000","last_active":"2024-06-29T14:22:45.704937+0000","last_peered":"2024-06-29T14:22:45.704937+0000","last_clean":"2024-06-29T14:22:45.704937+0000","last_became_active":"2024-06-29T14:22:43.705468+0000","last_became_peered":"2024-06-29T14:22:43.705468+0000","last_unstale":"2024-06-29T14:22:45.704937+0000","last_undegraded":"2024-06-29T14:22:45.704937+0000","last_fullsized":"2024-06-29T14:22:45.704937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316005+0000","last_change":"2024-06-29T14:22:44.719377+0000","last_active":"2024-06-29T14:22:46.316005+0000","last_peered":"2024-06-29T14:22:46.316005+0000","last_clean":"2024-06-29T14:22:46.316005+0000","last_became_active":"2024-06-29T14:22:44.719065+0000","last_became_peered":"2024-06-29T14:22:44.719065+0000","last_unstale":"2024-06-29T14:22:46.316005+0000","last_undegraded":"2024-06-29T14:22:46.316005+0000","last_fullsized":"2024-06-29T14:22:46.316005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:51:55.034132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314958+0000","last_change":"2024-06-29T14:22:43.698317+0000","last_active":"2024-06-29T14:22:46.314958+0000","last_peered":"2024-06-29T14:22:46.314958+0000","last_clean":"2024-06-29T14:22:46.314958+0000","last_became_active":"2024-06-29T14:22:43.697987+0000","last_became_peered":"2024-06-29T14:22:43.697987+0000","last_unstale":"2024-06-29T14:22:46.314958+0000","last_undegraded":"2024-06-29T14:22:46.314958+0000","last_fullsized":"2024-06-29T14:22:46.314958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315030+0000","last_change":"2024-06-29T14:22:44.710262+0000","last_active":"2024-06-29T14:22:46.315030+0000","last_peered":"2024-06-29T14:22:46.315030+0000","last_clean":"2024-06-29T14:22:46.315030+0000","last_became_active":"2024-06-29T14:22:44.709948+0000","last_became_peered":"2024-06-29T14:22:44.709948+0000","last_unstale":"2024-06-29T14:22:46.315030+0000","last_undegraded":"2024-06-29T14:22:46.315030+0000","last_fullsized":"2024-06-29T14:22:46.315030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:41.931607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725271+0000","last_change":"2024-06-29T14:22:43.703719+0000","last_active":"2024-06-29T14:22:45.725271+0000","last_peered":"2024-06-29T14:22:45.725271+0000","last_clean":"2024-06-29T14:22:45.725271+0000","last_became_active":"2024-06-29T14:22:43.703240+0000","last_became_peered":"2024-06-29T14:22:43.703240+0000","last_unstale":"2024-06-29T14:22:45.725271+0000","last_undegraded":"2024-06-29T14:22:45.725271+0000","last_fullsized":"2024-06-29T14:22:45.725271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:59:26.060090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315785+0000","last_change":"2024-06-29T14:22:44.715784+0000","last_active":"2024-06-29T14:22:46.315785+0000","last_peered":"2024-06-29T14:22:46.315785+0000","last_clean":"2024-06-29T14:22:46.315785+0000","last_became_active":"2024-06-29T14:22:44.715472+0000","last_became_peered":"2024-06-29T14:22:44.715472+0000","last_unstale":"2024-06-29T14:22:46.315785+0000","last_undegraded":"2024-06-29T14:22:46.315785+0000","last_fullsized":"2024-06-29T14:22:46.315785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:01.618482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703393+0000","last_change":"2024-06-29T14:22:43.700262+0000","last_active":"2024-06-29T14:22:45.703393+0000","last_peered":"2024-06-29T14:22:45.703393+0000","last_clean":"2024-06-29T14:22:45.703393+0000","last_became_active":"2024-06-29T14:22:43.699933+0000","last_became_peered":"2024-06-29T14:22:43.699933+0000","last_unstale":"2024-06-29T14:22:45.703393+0000","last_undegraded":"2024-06-29T14:22:45.703393+0000","last_fullsized":"2024-06-29T14:22:45.703393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:49.306537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316305+0000","last_change":"2024-06-29T14:22:44.717647+0000","last_active":"2024-06-29T14:22:46.316305+0000","last_peered":"2024-06-29T14:22:46.316305+0000","last_clean":"2024-06-29T14:22:46.316305+0000","last_became_active":"2024-06-29T14:22:44.717352+0000","last_became_peered":"2024-06-29T14:22:44.717352+0000","last_unstale":"2024-06-29T14:22:46.316305+0000","last_undegraded":"2024-06-29T14:22:46.316305+0000","last_fullsized":"2024-06-29T14:22:46.316305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:19.819693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725473+0000","last_change":"2024-06-29T14:22:43.706725+0000","last_active":"2024-06-29T14:22:45.725473+0000","last_peered":"2024-06-29T14:22:45.725473+0000","last_clean":"2024-06-29T14:22:45.725473+0000","last_became_active":"2024-06-29T14:22:43.706166+0000","last_became_peered":"2024-06-29T14:22:43.706166+0000","last_unstale":"2024-06-29T14:22:45.725473+0000","last_undegraded":"2024-06-29T14:22:45.725473+0000","last_fullsized":"2024-06-29T14:22:45.725473+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316526+0000","last_change":"2024-06-29T14:22:44.722510+0000","last_active":"2024-06-29T14:22:46.316526+0000","last_peered":"2024-06-29T14:22:46.316526+0000","last_clean":"2024-06-29T14:22:46.316526+0000","last_became_active":"2024-06-29T14:22:44.722129+0000","last_became_peered":"2024-06-29T14:22:44.722129+0000","last_unstale":"2024-06-29T14:22:46.316526+0000","last_undegraded":"2024-06-29T14:22:46.316526+0000","last_fullsized":"2024-06-29T14:22:46.316526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:31:30.258889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701676+0000","last_change":"2024-06-29T14:22:43.699816+0000","last_active":"2024-06-29T14:22:45.701676+0000","last_peered":"2024-06-29T14:22:45.701676+0000","last_clean":"2024-06-29T14:22:45.701676+0000","last_became_active":"2024-06-29T14:22:43.699427+0000","last_became_peered":"2024-06-29T14:22:43.699427+0000","last_unstale":"2024-06-29T14:22:45.701676+0000","last_undegraded":"2024-06-29T14:22:45.701676+0000","last_fullsized":"2024-06-29T14:22:45.701676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704777+0000","last_change":"2024-06-29T14:22:44.707863+0000","last_active":"2024-06-29T14:22:45.704777+0000","last_peered":"2024-06-29T14:22:45.704777+0000","last_clean":"2024-06-29T14:22:45.704777+0000","last_became_active":"2024-06-29T14:22:44.707498+0000","last_became_peered":"2024-06-29T14:22:44.707498+0000","last_unstale":"2024-06-29T14:22:45.704777+0000","last_undegraded":"2024-06-29T14:22:45.704777+0000","last_fullsized":"2024-06-29T14:22:45.704777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:50.421143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725446+0000","last_change":"2024-06-29T14:22:43.695216+0000","last_active":"2024-06-29T14:22:45.725446+0000","last_peered":"2024-06-29T14:22:45.725446+0000","last_clean":"2024-06-29T14:22:45.725446+0000","last_became_active":"2024-06-29T14:22:43.694485+0000","last_became_peered":"2024-06-29T14:22:43.694485+0000","last_unstale":"2024-06-29T14:22:45.725446+0000","last_undegraded":"2024-06-29T14:22:45.725446+0000","last_fullsized":"2024-06-29T14:22:45.725446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316151+0000","last_change":"2024-06-29T14:22:44.720432+0000","last_active":"2024-06-29T14:22:46.316151+0000","last_peered":"2024-06-29T14:22:46.316151+0000","last_clean":"2024-06-29T14:22:46.316151+0000","last_became_active":"2024-06-29T14:22:44.720099+0000","last_became_peered":"2024-06-29T14:22:44.720099+0000","last_unstale":"2024-06-29T14:22:46.316151+0000","last_undegraded":"2024-06-29T14:22:46.316151+0000","last_fullsized":"2024-06-29T14:22:46.316151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:50.421143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315581+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:22:46.315581+0000","last_peered":"2024-06-29T14:22:46.315581+0000","last_clean":"2024-06-29T14:22:46.315581+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:22:46.315581+0000","last_undegraded":"2024-06-29T14:22:46.315581+0000","last_fullsized":"2024-06-29T14:22:46.315581+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703988+0000","last_change":"2024-06-29T14:22:43.694439+0000","last_active":"2024-06-29T14:22:45.703988+0000","last_peered":"2024-06-29T14:22:45.703988+0000","last_clean":"2024-06-29T14:22:45.703988+0000","last_became_active":"2024-06-29T14:22:43.694115+0000","last_became_peered":"2024-06-29T14:22:43.694115+0000","last_unstale":"2024-06-29T14:22:45.703988+0000","last_undegraded":"2024-06-29T14:22:45.703988+0000","last_fullsized":"2024-06-29T14:22:45.703988+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:01.771566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314689+0000","last_change":"2024-06-29T14:22:44.707439+0000","last_active":"2024-06-29T14:22:46.314689+0000","last_peered":"2024-06-29T14:22:46.314689+0000","last_clean":"2024-06-29T14:22:46.314689+0000","last_became_active":"2024-06-29T14:22:44.707120+0000","last_became_peered":"2024-06-29T14:22:44.707120+0000","last_unstale":"2024-06-29T14:22:46.314689+0000","last_undegraded":"2024-06-29T14:22:46.314689+0000","last_fullsized":"2024-06-29T14:22:46.314689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:45.007248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315705+0000","last_change":"2024-06-29T14:22:43.712228+0000","last_active":"2024-06-29T14:22:46.315705+0000","last_peered":"2024-06-29T14:22:46.315705+0000","last_clean":"2024-06-29T14:22:46.315705+0000","last_became_active":"2024-06-29T14:22:43.711907+0000","last_became_peered":"2024-06-29T14:22:43.711907+0000","last_unstale":"2024-06-29T14:22:46.315705+0000","last_undegraded":"2024-06-29T14:22:46.315705+0000","last_fullsized":"2024-06-29T14:22:46.315705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315247+0000","last_change":"2024-06-29T14:22:44.705791+0000","last_active":"2024-06-29T14:22:46.315247+0000","last_peered":"2024-06-29T14:22:46.315247+0000","last_clean":"2024-06-29T14:22:46.315247+0000","last_became_active":"2024-06-29T14:22:44.705029+0000","last_became_peered":"2024-06-29T14:22:44.705029+0000","last_unstale":"2024-06-29T14:22:46.315247+0000","last_undegraded":"2024-06-29T14:22:46.315247+0000","last_fullsized":"2024-06-29T14:22:46.315247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.725377+0000","last_change":"2024-06-29T14:22:43.706346+0000","last_active":"2024-06-29T14:22:45.725377+0000","last_peered":"2024-06-29T14:22:45.725377+0000","last_clean":"2024-06-29T14:22:45.725377+0000","last_became_active":"2024-06-29T14:22:43.706016+0000","last_became_peered":"2024-06-29T14:22:43.706016+0000","last_unstale":"2024-06-29T14:22:45.725377+0000","last_undegraded":"2024-06-29T14:22:45.725377+0000","last_fullsized":"2024-06-29T14:22:45.725377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701343+0000","last_change":"2024-06-29T14:22:44.692833+0000","last_active":"2024-06-29T14:22:45.701343+0000","last_peered":"2024-06-29T14:22:45.701343+0000","last_clean":"2024-06-29T14:22:45.701343+0000","last_became_active":"2024-06-29T14:22:44.692138+0000","last_became_peered":"2024-06-29T14:22:44.692138+0000","last_unstale":"2024-06-29T14:22:45.701343+0000","last_undegraded":"2024-06-29T14:22:45.701343+0000","last_fullsized":"2024-06-29T14:22:45.701343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.705065+0000","last_change":"2024-06-29T14:22:43.703727+0000","last_active":"2024-06-29T14:22:45.705065+0000","last_peered":"2024-06-29T14:22:45.705065+0000","last_clean":"2024-06-29T14:22:45.705065+0000","last_became_active":"2024-06-29T14:22:43.703413+0000","last_became_peered":"2024-06-29T14:22:43.703413+0000","last_unstale":"2024-06-29T14:22:45.705065+0000","last_undegraded":"2024-06-29T14:22:45.705065+0000","last_fullsized":"2024-06-29T14:22:45.705065+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.703554+0000","last_change":"2024-06-29T14:22:44.712628+0000","last_active":"2024-06-29T14:22:45.703554+0000","last_peered":"2024-06-29T14:22:45.703554+0000","last_clean":"2024-06-29T14:22:45.703554+0000","last_became_active":"2024-06-29T14:22:44.712293+0000","last_became_peered":"2024-06-29T14:22:44.712293+0000","last_unstale":"2024-06-29T14:22:45.703554+0000","last_undegraded":"2024-06-29T14:22:45.703554+0000","last_fullsized":"2024-06-29T14:22:45.703554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:40.770948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701488+0000","last_change":"2024-06-29T14:22:43.702999+0000","last_active":"2024-06-29T14:22:45.701488+0000","last_peered":"2024-06-29T14:22:45.701488+0000","last_clean":"2024-06-29T14:22:45.701488+0000","last_became_active":"2024-06-29T14:22:43.702670+0000","last_became_peered":"2024-06-29T14:22:43.702670+0000","last_unstale":"2024-06-29T14:22:45.701488+0000","last_undegraded":"2024-06-29T14:22:45.701488+0000","last_fullsized":"2024-06-29T14:22:45.701488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:36:09.823578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704843+0000","last_change":"2024-06-29T14:22:44.708301+0000","last_active":"2024-06-29T14:22:45.704843+0000","last_peered":"2024-06-29T14:22:45.704843+0000","last_clean":"2024-06-29T14:22:45.704843+0000","last_became_active":"2024-06-29T14:22:44.707732+0000","last_became_peered":"2024-06-29T14:22:44.707732+0000","last_unstale":"2024-06-29T14:22:45.704843+0000","last_undegraded":"2024-06-29T14:22:45.704843+0000","last_fullsized":"2024-06-29T14:22:45.704843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:55:43.420129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316302+0000","last_change":"2024-06-29T14:22:43.711647+0000","last_active":"2024-06-29T14:22:46.316302+0000","last_peered":"2024-06-29T14:22:46.316302+0000","last_clean":"2024-06-29T14:22:46.316302+0000","last_became_active":"2024-06-29T14:22:43.710944+0000","last_became_peered":"2024-06-29T14:22:43.710944+0000","last_unstale":"2024-06-29T14:22:46.316302+0000","last_undegraded":"2024-06-29T14:22:46.316302+0000","last_fullsized":"2024-06-29T14:22:46.316302+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:30.671807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701389+0000","last_change":"2024-06-29T14:22:44.696383+0000","last_active":"2024-06-29T14:22:45.701389+0000","last_peered":"2024-06-29T14:22:45.701389+0000","last_clean":"2024-06-29T14:22:45.701389+0000","last_became_active":"2024-06-29T14:22:44.696071+0000","last_became_peered":"2024-06-29T14:22:44.696071+0000","last_unstale":"2024-06-29T14:22:45.701389+0000","last_undegraded":"2024-06-29T14:22:45.701389+0000","last_fullsized":"2024-06-29T14:22:45.701389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:11.894013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701505+0000","last_change":"2024-06-29T14:22:43.703248+0000","last_active":"2024-06-29T14:22:45.701505+0000","last_peered":"2024-06-29T14:22:45.701505+0000","last_clean":"2024-06-29T14:22:45.701505+0000","last_became_active":"2024-06-29T14:22:43.702925+0000","last_became_peered":"2024-06-29T14:22:43.702925+0000","last_unstale":"2024-06-29T14:22:45.701505+0000","last_undegraded":"2024-06-29T14:22:45.701505+0000","last_fullsized":"2024-06-29T14:22:45.701505+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:39.003651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314975+0000","last_change":"2024-06-29T14:22:44.722533+0000","last_active":"2024-06-29T14:22:46.314975+0000","last_peered":"2024-06-29T14:22:46.314975+0000","last_clean":"2024-06-29T14:22:46.314975+0000","last_became_active":"2024-06-29T14:22:44.722296+0000","last_became_peered":"2024-06-29T14:22:44.722296+0000","last_unstale":"2024-06-29T14:22:46.314975+0000","last_undegraded":"2024-06-29T14:22:46.314975+0000","last_fullsized":"2024-06-29T14:22:46.314975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:29.350494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315500+0000","last_change":"2024-06-29T14:22:43.705703+0000","last_active":"2024-06-29T14:22:46.315500+0000","last_peered":"2024-06-29T14:22:46.315500+0000","last_clean":"2024-06-29T14:22:46.315500+0000","last_became_active":"2024-06-29T14:22:43.705306+0000","last_became_peered":"2024-06-29T14:22:43.705306+0000","last_unstale":"2024-06-29T14:22:46.315500+0000","last_undegraded":"2024-06-29T14:22:46.315500+0000","last_fullsized":"2024-06-29T14:22:46.315500+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:45.015829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.314958+0000","last_change":"2024-06-29T14:22:44.712109+0000","last_active":"2024-06-29T14:22:46.314958+0000","last_peered":"2024-06-29T14:22:46.314958+0000","last_clean":"2024-06-29T14:22:46.314958+0000","last_became_active":"2024-06-29T14:22:44.711374+0000","last_became_peered":"2024-06-29T14:22:44.711374+0000","last_unstale":"2024-06-29T14:22:46.314958+0000","last_undegraded":"2024-06-29T14:22:46.314958+0000","last_fullsized":"2024-06-29T14:22:46.314958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:47.631230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315289+0000","last_change":"2024-06-29T14:22:43.711036+0000","last_active":"2024-06-29T14:22:46.315289+0000","last_peered":"2024-06-29T14:22:46.315289+0000","last_clean":"2024-06-29T14:22:46.315289+0000","last_became_active":"2024-06-29T14:22:43.710662+0000","last_became_peered":"2024-06-29T14:22:43.710662+0000","last_unstale":"2024-06-29T14:22:46.315289+0000","last_undegraded":"2024-06-29T14:22:46.315289+0000","last_fullsized":"2024-06-29T14:22:46.315289+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:01.984922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704030+0000","last_change":"2024-06-29T14:22:44.699330+0000","last_active":"2024-06-29T14:22:45.704030+0000","last_peered":"2024-06-29T14:22:45.704030+0000","last_clean":"2024-06-29T14:22:45.704030+0000","last_became_active":"2024-06-29T14:22:44.698611+0000","last_became_peered":"2024-06-29T14:22:44.698611+0000","last_unstale":"2024-06-29T14:22:45.704030+0000","last_undegraded":"2024-06-29T14:22:45.704030+0000","last_fullsized":"2024-06-29T14:22:45.704030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:02.990246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701414+0000","last_change":"2024-06-29T14:22:43.711354+0000","last_active":"2024-06-29T14:22:45.701414+0000","last_peered":"2024-06-29T14:22:45.701414+0000","last_clean":"2024-06-29T14:22:45.701414+0000","last_became_active":"2024-06-29T14:22:43.710734+0000","last_became_peered":"2024-06-29T14:22:43.710734+0000","last_unstale":"2024-06-29T14:22:45.701414+0000","last_undegraded":"2024-06-29T14:22:45.701414+0000","last_fullsized":"2024-06-29T14:22:45.701414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315307+0000","last_change":"2024-06-29T14:22:44.694835+0000","last_active":"2024-06-29T14:22:46.315307+0000","last_peered":"2024-06-29T14:22:46.315307+0000","last_clean":"2024-06-29T14:22:46.315307+0000","last_became_active":"2024-06-29T14:22:44.694463+0000","last_became_peered":"2024-06-29T14:22:44.694463+0000","last_unstale":"2024-06-29T14:22:46.315307+0000","last_undegraded":"2024-06-29T14:22:46.315307+0000","last_fullsized":"2024-06-29T14:22:46.315307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:36:01.618482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.701665+0000","last_change":"2024-06-29T14:22:43.705353+0000","last_active":"2024-06-29T14:22:45.701665+0000","last_peered":"2024-06-29T14:22:45.701665+0000","last_clean":"2024-06-29T14:22:45.701665+0000","last_became_active":"2024-06-29T14:22:43.705002+0000","last_became_peered":"2024-06-29T14:22:43.705002+0000","last_unstale":"2024-06-29T14:22:45.701665+0000","last_undegraded":"2024-06-29T14:22:45.701665+0000","last_fullsized":"2024-06-29T14:22:45.701665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:57:30.671807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315007+0000","last_change":"2024-06-29T14:22:44.710202+0000","last_active":"2024-06-29T14:22:46.315007+0000","last_peered":"2024-06-29T14:22:46.315007+0000","last_clean":"2024-06-29T14:22:46.315007+0000","last_became_active":"2024-06-29T14:22:44.709900+0000","last_became_peered":"2024-06-29T14:22:44.709900+0000","last_unstale":"2024-06-29T14:22:46.315007+0000","last_undegraded":"2024-06-29T14:22:46.315007+0000","last_fullsized":"2024-06-29T14:22:46.315007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:08.776628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316125+0000","last_change":"2024-06-29T14:22:43.712161+0000","last_active":"2024-06-29T14:22:46.316125+0000","last_peered":"2024-06-29T14:22:46.316125+0000","last_clean":"2024-06-29T14:22:46.316125+0000","last_became_active":"2024-06-29T14:22:43.710970+0000","last_became_peered":"2024-06-29T14:22:43.710970+0000","last_unstale":"2024-06-29T14:22:46.316125+0000","last_undegraded":"2024-06-29T14:22:46.316125+0000","last_fullsized":"2024-06-29T14:22:46.316125+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:43.918058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315157+0000","last_change":"2024-06-29T14:22:44.702756+0000","last_active":"2024-06-29T14:22:46.315157+0000","last_peered":"2024-06-29T14:22:46.315157+0000","last_clean":"2024-06-29T14:22:46.315157+0000","last_became_active":"2024-06-29T14:22:44.702438+0000","last_became_peered":"2024-06-29T14:22:44.702438+0000","last_unstale":"2024-06-29T14:22:46.315157+0000","last_undegraded":"2024-06-29T14:22:46.315157+0000","last_fullsized":"2024-06-29T14:22:46.315157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:16:12.066692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704611+0000","last_change":"2024-06-29T14:22:44.697060+0000","last_active":"2024-06-29T14:22:45.704611+0000","last_peered":"2024-06-29T14:22:45.704611+0000","last_clean":"2024-06-29T14:22:45.704611+0000","last_became_active":"2024-06-29T14:22:44.696660+0000","last_became_peered":"2024-06-29T14:22:44.696660+0000","last_unstale":"2024-06-29T14:22:45.704611+0000","last_undegraded":"2024-06-29T14:22:45.704611+0000","last_fullsized":"2024-06-29T14:22:45.704611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:51:55.034132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316026+0000","last_change":"2024-06-29T14:22:43.711693+0000","last_active":"2024-06-29T14:22:46.316026+0000","last_peered":"2024-06-29T14:22:46.316026+0000","last_clean":"2024-06-29T14:22:46.316026+0000","last_became_active":"2024-06-29T14:22:43.710688+0000","last_became_peered":"2024-06-29T14:22:43.710688+0000","last_unstale":"2024-06-29T14:22:46.316026+0000","last_undegraded":"2024-06-29T14:22:46.316026+0000","last_fullsized":"2024-06-29T14:22:46.316026+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:03:39.765624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:45.704601+0000","last_change":"2024-06-29T14:22:44.697174+0000","last_active":"2024-06-29T14:22:45.704601+0000","last_peered":"2024-06-29T14:22:45.704601+0000","last_clean":"2024-06-29T14:22:45.704601+0000","last_became_active":"2024-06-29T14:22:44.696735+0000","last_became_peered":"2024-06-29T14:22:44.696735+0000","last_unstale":"2024-06-29T14:22:45.704601+0000","last_undegraded":"2024-06-29T14:22:45.704601+0000","last_fullsized":"2024-06-29T14:22:45.704601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:11.627629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.316149+0000","last_change":"2024-06-29T14:22:43.711172+0000","last_active":"2024-06-29T14:22:46.316149+0000","last_peered":"2024-06-29T14:22:46.316149+0000","last_clean":"2024-06-29T14:22:46.316149+0000","last_became_active":"2024-06-29T14:22:43.710779+0000","last_became_peered":"2024-06-29T14:22:43.710779+0000","last_unstale":"2024-06-29T14:22:46.316149+0000","last_undegraded":"2024-06-29T14:22:46.316149+0000","last_fullsized":"2024-06-29T14:22:46.316149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:49:39.003651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315403+0000","last_change":"2024-06-29T14:22:44.708519+0000","last_active":"2024-06-29T14:22:46.315403+0000","last_peered":"2024-06-29T14:22:46.315403+0000","last_clean":"2024-06-29T14:22:46.315403+0000","last_became_active":"2024-06-29T14:22:44.708212+0000","last_became_peered":"2024-06-29T14:22:44.708212+0000","last_unstale":"2024-06-29T14:22:46.315403+0000","last_undegraded":"2024-06-29T14:22:46.315403+0000","last_fullsized":"2024-06-29T14:22:46.315403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:38.794003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315117+0000","last_change":"2024-06-29T14:22:43.706244+0000","last_active":"2024-06-29T14:22:46.315117+0000","last_peered":"2024-06-29T14:22:46.315117+0000","last_clean":"2024-06-29T14:22:46.315117+0000","last_became_active":"2024-06-29T14:22:43.705937+0000","last_became_peered":"2024-06-29T14:22:43.705937+0000","last_unstale":"2024-06-29T14:22:46.315117+0000","last_undegraded":"2024-06-29T14:22:46.315117+0000","last_fullsized":"2024-06-29T14:22:46.315117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:15.072861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315107+0000","last_change":"2024-06-29T14:22:44.707925+0000","last_active":"2024-06-29T14:22:46.315107+0000","last_peered":"2024-06-29T14:22:46.315107+0000","last_clean":"2024-06-29T14:22:46.315107+0000","last_became_active":"2024-06-29T14:22:44.707640+0000","last_became_peered":"2024-06-29T14:22:44.707640+0000","last_unstale":"2024-06-29T14:22:46.315107+0000","last_undegraded":"2024-06-29T14:22:46.315107+0000","last_fullsized":"2024-06-29T14:22:46.315107+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:43.675336+0000","last_clean_scrub_stamp":"2024-06-29T14:22:43.675336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:25.905572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-06-29T14:22:46.315669+0000","last_change":"2024-06-29T14:22:43.711121+0000","last_active":"2024-06-29T14:22:46.315669+0000","last_peered":"2024-06-29T14:22:46.315669+0000","last_clean":"2024-06-29T14:22:46.315669+0000","last_became_active":"2024-06-29T14:22:43.710708+0000","last_became_peered":"2024-06-29T14:22:43.710708+0000","last_unstale":"2024-06-29T14:22:46.315669+0000","last_undegraded":"2024-06-29T14:22:46.315669+0000","last_fullsized":"2024-06-29T14:22:46.315669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:42.670012+0000","last_clean_scrub_stamp":"2024-06-29T14:22:42.670012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:49.306537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:22:53.968 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-29T14:22:53.968 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-29T14:22:53.968 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-06-29T14:22:53.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-06-29T14:22:54.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.117+0000 7f6153168640 1 -- 172.21.15.81:0/2065163331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f614c073740 msgr2=0x7f614c15e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.117+0000 7f6153168640 1 --2- 172.21.15.81:0/2065163331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f614c073740 0x7f614c15e960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6144005fd0 tx=0x7f614402a530 comp rx=0 tx=0).stop 2024-06-29T14:22:54.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.118+0000 7f6153168640 1 -- 172.21.15.81:0/2065163331 shutdown_connections 2024-06-29T14:22:54.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.118+0000 7f6153168640 1 --2- 172.21.15.81:0/2065163331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f614c073740 0x7f614c15e960 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.118+0000 7f6153168640 1 --2- 172.21.15.81:0/2065163331 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f614c1412f0 0x7f614c0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.118+0000 7f6153168640 1 -- 172.21.15.81:0/2065163331 >> 172.21.15.81:0/2065163331 conn(0x7f614c06ccc0 msgr2=0x7f614c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.118+0000 7f6153168640 1 -- 172.21.15.81:0/2065163331 shutdown_connections 2024-06-29T14:22:54.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.118+0000 7f6153168640 1 -- 172.21.15.81:0/2065163331 wait complete. 2024-06-29T14:22:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.119+0000 7f6153168640 1 Processor -- start 2024-06-29T14:22:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.119+0000 7f6153168640 1 -- start start 2024-06-29T14:22:54.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6153168640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f614c0cf4d0 0x7f614c0c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6153168640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f614c1412f0 0x7f614c159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6153168640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f614c071f20 0x7f614c072330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6153168640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f614c0b3210 con 0x7f614c071f20 2024-06-29T14:22:54.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6153168640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f614c15f270 con 0x7f614c0cf4d0 2024-06-29T14:22:54.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6153168640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f614c0b3390 con 0x7f614c1412f0 2024-06-29T14:22:54.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6151705640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f614c071f20 0x7f614c072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6151705640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f614c071f20 0x7f614c072330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36370/0 (socket says 172.21.15.81:36370) 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6151705640 1 -- 172.21.15.81:0/3804360102 learned_addr learned my addr 172.21.15.81:0/3804360102 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6150f04640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f614c0cf4d0 0x7f614c0c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f614bfff640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f614c1412f0 0x7f614c159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6151705640 1 -- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f614c1412f0 msgr2=0x7f614c159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6151705640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f614c1412f0 0x7f614c159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6151705640 1 -- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f614c0cf4d0 msgr2=0x7f614c0c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6151705640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f614c0cf4d0 0x7f614c0c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6151705640 1 -- 172.21.15.81:0/3804360102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614c072870 con 0x7f614c071f20 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.121+0000 7f6151705640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f614c071f20 0x7f614c072330 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6144006010 tx=0x7f6144003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.122+0000 7f61497fa640 1 -- 172.21.15.81:0/3804360102 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614403e030 con 0x7f614c071f20 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.122+0000 7f61497fa640 1 -- 172.21.15.81:0/3804360102 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6144003d70 con 0x7f614c071f20 2024-06-29T14:22:54.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.122+0000 7f6153168640 1 -- 172.21.15.81:0/3804360102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f614c1f83c0 con 0x7f614c071f20 2024-06-29T14:22:54.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.122+0000 7f61497fa640 1 -- 172.21.15.81:0/3804360102 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6144035040 con 0x7f614c071f20 2024-06-29T14:22:54.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.122+0000 7f6153168640 1 -- 172.21.15.81:0/3804360102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f614c1f8980 con 0x7f614c071f20 2024-06-29T14:22:54.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.123+0000 7f6153168640 1 -- 172.21.15.81:0/3804360102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f614c0b38b0 con 0x7f614c071f20 2024-06-29T14:22:54.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.123+0000 7f61497fa640 1 -- 172.21.15.81:0/3804360102 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6144041030 con 0x7f614c071f20 2024-06-29T14:22:54.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.124+0000 7f61497fa640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f61380762d0 0x7f61380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.124+0000 7f61497fa640 1 -- 172.21.15.81:0/3804360102 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f61440b7df0 con 0x7f614c071f20 2024-06-29T14:22:54.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.124+0000 7f6150f04640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f61380762d0 0x7f61380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.125+0000 7f6150f04640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f61380762d0 0x7f61380787e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f613c004630 tx=0x7f613c00b2e0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.128+0000 7f61497fa640 1 -- 172.21.15.81:0/3804360102 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f614c072870 con 0x7f614c071f20 2024-06-29T14:22:54.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.348+0000 7f6153168640 1 -- 172.21.15.81:0/3804360102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f614c15ce30 con 0x7f614c071f20 2024-06-29T14:22:54.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.348+0000 7f61497fa640 1 -- 172.21.15.81:0/3804360102 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f6144081100 con 0x7f614c071f20 2024-06-29T14:22:54.347 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:54.348 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-29T14:22:54.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.350+0000 7f6153168640 1 -- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f61380762d0 msgr2=0x7f61380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.350+0000 7f6153168640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f61380762d0 0x7f61380787e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f613c004630 tx=0x7f613c00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.350+0000 7f6153168640 1 -- 172.21.15.81:0/3804360102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f614c071f20 msgr2=0x7f614c072330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.350+0000 7f6153168640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f614c071f20 0x7f614c072330 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6144006010 tx=0x7f6144003880 comp rx=0 tx=0).stop 2024-06-29T14:22:54.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.351+0000 7f6153168640 1 -- 172.21.15.81:0/3804360102 shutdown_connections 2024-06-29T14:22:54.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.351+0000 7f6153168640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f61380762d0 0x7f61380787e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.351+0000 7f6153168640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f614c071f20 0x7f614c072330 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.351+0000 7f6153168640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f614c1412f0 0x7f614c159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.351+0000 7f6153168640 1 --2- 172.21.15.81:0/3804360102 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f614c0cf4d0 0x7f614c0c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.351+0000 7f6153168640 1 -- 172.21.15.81:0/3804360102 >> 172.21.15.81:0/3804360102 conn(0x7f614c06ccc0 msgr2=0x7f614c13ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.351+0000 7f6153168640 1 -- 172.21.15.81:0/3804360102 shutdown_connections 2024-06-29T14:22:54.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.351+0000 7f6153168640 1 -- 172.21.15.81:0/3804360102 wait complete. 2024-06-29T14:22:54.360 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-06-29T14:22:54.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:54.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.476+0000 7f877788f640 1 -- 172.21.15.81:0/2027083064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87700cf040 msgr2=0x7f87700b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.476+0000 7f877788f640 1 --2- 172.21.15.81:0/2027083064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87700cf040 0x7f87700b3780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f87700b55d0 tx=0x7f876802a8f0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.476+0000 7f877788f640 1 -- 172.21.15.81:0/2027083064 shutdown_connections 2024-06-29T14:22:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.476+0000 7f877788f640 1 --2- 172.21.15.81:0/2027083064 >> v2:172.21.15.167:3300/0 conn(0x7f877008ccd0 0x7f87700b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.476+0000 7f877788f640 1 --2- 172.21.15.81:0/2027083064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87700cf040 0x7f87700b3780 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.476+0000 7f877788f640 1 -- 172.21.15.81:0/2027083064 >> 172.21.15.81:0/2027083064 conn(0x7f87700b56d0 msgr2=0x7f87700b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.476+0000 7f877788f640 1 -- 172.21.15.81:0/2027083064 shutdown_connections 2024-06-29T14:22:54.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.476+0000 7f877788f640 1 -- 172.21.15.81:0/2027083064 wait complete. 2024-06-29T14:22:54.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.478+0000 7f877788f640 1 Processor -- start 2024-06-29T14:22:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.478+0000 7f877788f640 1 -- start start 2024-06-29T14:22:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f877788f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f877008ccd0 0x7f87701e38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f877788f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87700cf040 0x7f87700a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f877788f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87700a1750 0x7f87700b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f877788f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87700b41a0 con 0x7f87700cf040 2024-06-29T14:22:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f877788f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f87700b4020 con 0x7f877008ccd0 2024-06-29T14:22:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f877788f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f87700b4320 con 0x7f87700a1750 2024-06-29T14:22:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f8774e2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87700cf040 0x7f87700a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f8774e2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87700cf040 0x7f87700a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36400/0 (socket says 172.21.15.81:36400) 2024-06-29T14:22:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f8774e2a640 1 -- 172.21.15.81:0/2755016983 learned_addr learned my addr 172.21.15.81:0/2755016983 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f877562b640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f877008ccd0 0x7f87701e38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f8775e2c640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87700a1750 0x7f87700b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f8774e2a640 1 -- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87700a1750 msgr2=0x7f87700b7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f8774e2a640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87700a1750 0x7f87700b7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f8774e2a640 1 -- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f877008ccd0 msgr2=0x7f87701e38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f8774e2a640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f877008ccd0 0x7f87701e38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f8774e2a640 1 -- 172.21.15.81:0/2755016983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87700c8870 con 0x7f87700cf040 2024-06-29T14:22:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f877562b640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f877008ccd0 0x7f87701e38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.479+0000 7f8775e2c640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87700a1750 0x7f87700b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.480+0000 7f8774e2a640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87700cf040 0x7f87700a1210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f876000b410 tx=0x7f876000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.480+0000 7f8765ffb640 1 -- 172.21.15.81:0/2755016983 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760015020 con 0x7f87700cf040 2024-06-29T14:22:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.480+0000 7f8765ffb640 1 -- 172.21.15.81:0/2755016983 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8760011940 con 0x7f87700cf040 2024-06-29T14:22:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.480+0000 7f8765ffb640 1 -- 172.21.15.81:0/2755016983 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760011d90 con 0x7f87700cf040 2024-06-29T14:22:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.480+0000 7f877788f640 1 -- 172.21.15.81:0/2755016983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87700c77d0 con 0x7f87700cf040 2024-06-29T14:22:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.480+0000 7f877788f640 1 -- 172.21.15.81:0/2755016983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87700c7d90 con 0x7f87700cf040 2024-06-29T14:22:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.481+0000 7f877788f640 1 -- 172.21.15.81:0/2755016983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f877008c270 con 0x7f87700cf040 2024-06-29T14:22:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.481+0000 7f8765ffb640 1 -- 172.21.15.81:0/2755016983 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8760018020 con 0x7f87700cf040 2024-06-29T14:22:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.482+0000 7f8765ffb640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87540762d0 0x7f87540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.482+0000 7f8765ffb640 1 -- 172.21.15.81:0/2755016983 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8760097770 con 0x7f87700cf040 2024-06-29T14:22:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.482+0000 7f877562b640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87540762d0 0x7f87540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.483+0000 7f877562b640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87540762d0 0x7f87540787e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f876c00a810 tx=0x7f876c008040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.486+0000 7f8765ffb640 1 -- 172.21.15.81:0/2755016983 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f876001f020 con 0x7f87700cf040 2024-06-29T14:22:54.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.668+0000 7f877788f640 1 -- 172.21.15.81:0/2755016983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87700a1be0 con 0x7f87700cf040 2024-06-29T14:22:54.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.669+0000 7f8765ffb640 1 -- 172.21.15.81:0/2755016983 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f87700a1be0 con 0x7f87700cf040 2024-06-29T14:22:54.668 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:54.668 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:54.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 -- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87540762d0 msgr2=0x7f87540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87540762d0 0x7f87540787e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f876c00a810 tx=0x7f876c008040 comp rx=0 tx=0).stop 2024-06-29T14:22:54.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 -- 172.21.15.81:0/2755016983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87700cf040 msgr2=0x7f87700a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87700cf040 0x7f87700a1210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f876000b410 tx=0x7f876000b8d0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 -- 172.21.15.81:0/2755016983 shutdown_connections 2024-06-29T14:22:54.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87540762d0 0x7f87540787e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87700a1750 0x7f87700b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87700cf040 0x7f87700a1210 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 --2- 172.21.15.81:0/2755016983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f877008ccd0 0x7f87701e38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 -- 172.21.15.81:0/2755016983 >> 172.21.15.81:0/2755016983 conn(0x7f87700b56d0 msgr2=0x7f87700b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 -- 172.21.15.81:0/2755016983 shutdown_connections 2024-06-29T14:22:54.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.671+0000 7f877788f640 1 -- 172.21.15.81:0/2755016983 wait complete. 2024-06-29T14:22:54.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:22:54.682 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.802+0000 7f7541409640 1 -- 172.21.15.81:0/3707068642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75240039e0 msgr2=0x7f7524003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.802+0000 7f7541409640 1 --2- 172.21.15.81:0/3707068642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75240039e0 0x7f7524003e10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f752c001e40 tx=0x7f7524005830 comp rx=0 tx=0).stop 2024-06-29T14:22:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.802+0000 7f753b7fe640 1 -- 172.21.15.81:0/3707068642 reap_dead start 2024-06-29T14:22:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.802+0000 7f7541409640 1 -- 172.21.15.81:0/3707068642 shutdown_connections 2024-06-29T14:22:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.802+0000 7f7541409640 1 -- 172.21.15.81:0/3707068642 >> 172.21.15.81:0/3707068642 conn(0x7f753c09cf30 msgr2=0x7f753c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.802+0000 7f7541409640 1 -- 172.21.15.81:0/3707068642 shutdown_connections 2024-06-29T14:22:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.802+0000 7f7541409640 1 -- 172.21.15.81:0/3707068642 wait complete. 2024-06-29T14:22:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.804+0000 7f7541409640 1 Processor -- start 2024-06-29T14:22:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.804+0000 7f7541409640 1 -- start start 2024-06-29T14:22:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.805+0000 7f7541409640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75240039e0 0x7f753c1e9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.805+0000 7f7541409640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f753c1f8490 0x7f753c0cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.805+0000 7f7541409640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f753c0cd5b0 0x7f753c0c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.805+0000 7f7541409640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f753c0d2310 con 0x7f75240039e0 2024-06-29T14:22:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.805+0000 7f7541409640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f753c0d2190 con 0x7f753c0cd5b0 2024-06-29T14:22:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.805+0000 7f7541409640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f753c0d2490 con 0x7f753c1f8490 2024-06-29T14:22:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75240039e0 0x7f753c1e9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75240039e0 0x7f753c1e9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36422/0 (socket says 172.21.15.81:36422) 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753affd640 1 -- 172.21.15.81:0/747919238 learned_addr learned my addr 172.21.15.81:0/747919238 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753a7fc640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f753c1f8490 0x7f753c0cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753b7fe640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f753c0cd5b0 0x7f753c0c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753affd640 1 -- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f753c1f8490 msgr2=0x7f753c0cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753affd640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f753c1f8490 0x7f753c0cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753affd640 1 -- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f753c0cd5b0 msgr2=0x7f753c0c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753affd640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f753c0cd5b0 0x7f753c0c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753affd640 1 -- 172.21.15.81:0/747919238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753c1516e0 con 0x7f75240039e0 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753a7fc640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f753c1f8490 0x7f753c0cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753b7fe640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f753c0cd5b0 0x7f753c0c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f753affd640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75240039e0 0x7f753c1e9000 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f752c001f60 tx=0x7f752c0049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f75237fe640 1 -- 172.21.15.81:0/747919238 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c037040 con 0x7f75240039e0 2024-06-29T14:22:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f75237fe640 1 -- 172.21.15.81:0/747919238 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f752c041070 con 0x7f75240039e0 2024-06-29T14:22:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f7541409640 1 -- 172.21.15.81:0/747919238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753c0ca4e0 con 0x7f75240039e0 2024-06-29T14:22:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f75237fe640 1 -- 172.21.15.81:0/747919238 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c006300 con 0x7f75240039e0 2024-06-29T14:22:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.806+0000 7f7541409640 1 -- 172.21.15.81:0/747919238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f753c1e8520 con 0x7f75240039e0 2024-06-29T14:22:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.807+0000 7f7541409640 1 -- 172.21.15.81:0/747919238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f753c0d2a20 con 0x7f75240039e0 2024-06-29T14:22:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.808+0000 7f75237fe640 1 -- 172.21.15.81:0/747919238 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f752c0064a0 con 0x7f75240039e0 2024-06-29T14:22:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.808+0000 7f75237fe640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75280762d0 0x7f75280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.809+0000 7f75237fe640 1 -- 172.21.15.81:0/747919238 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f752c0b7bf0 con 0x7f75240039e0 2024-06-29T14:22:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.809+0000 7f753a7fc640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75280762d0 0x7f75280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.809+0000 7f753a7fc640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75280762d0 0x7f75280787e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7530001240 tx=0x7f7530001150 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.813+0000 7f75237fe640 1 -- 172.21.15.81:0/747919238 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f752c0814c0 con 0x7f75240039e0 2024-06-29T14:22:54.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.993+0000 7f7541409640 1 -- 172.21.15.81:0/747919238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f753c068690 con 0x7f75240039e0 2024-06-29T14:22:54.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.994+0000 7f75237fe640 1 -- 172.21.15.81:0/747919238 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f752c080dc0 con 0x7f75240039e0 2024-06-29T14:22:54.992 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:22:54.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 -- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75280762d0 msgr2=0x7f75280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75280762d0 0x7f75280787e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7530001240 tx=0x7f7530001150 comp rx=0 tx=0).stop 2024-06-29T14:22:54.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 -- 172.21.15.81:0/747919238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75240039e0 msgr2=0x7f753c1e9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:54.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75240039e0 0x7f753c1e9000 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f752c001f60 tx=0x7f752c0049f0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 -- 172.21.15.81:0/747919238 shutdown_connections 2024-06-29T14:22:54.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f75280762d0 0x7f75280787e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f753c0cd5b0 0x7f753c0c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f753c1f8490 0x7f753c0cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 --2- 172.21.15.81:0/747919238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75240039e0 0x7f753c1e9000 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:54.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 -- 172.21.15.81:0/747919238 >> 172.21.15.81:0/747919238 conn(0x7f753c09cf30 msgr2=0x7f753c15de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:54.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 -- 172.21.15.81:0/747919238 shutdown_connections 2024-06-29T14:22:54.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:54.996+0000 7f7541409640 1 -- 172.21.15.81:0/747919238 wait complete. 2024-06-29T14:22:55.005 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:55.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.122+0000 7fa81cee4640 1 -- 172.21.15.81:0/3308226545 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8180b4f40 msgr2=0x7fa8180b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.122+0000 7fa81cee4640 1 --2- 172.21.15.81:0/3308226545 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8180b4f40 0x7fa8180b5350 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa810005fd0 tx=0x7fa81002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.122+0000 7fa81cee4640 1 -- 172.21.15.81:0/3308226545 shutdown_connections 2024-06-29T14:22:55.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.122+0000 7fa81cee4640 1 --2- 172.21.15.81:0/3308226545 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8180b4f40 0x7fa8180b5350 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.122+0000 7fa81cee4640 1 --2- 172.21.15.81:0/3308226545 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8180d0230 0x7fa8180d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.122+0000 7fa81cee4640 1 -- 172.21.15.81:0/3308226545 >> 172.21.15.81:0/3308226545 conn(0x7fa8180b3680 msgr2=0x7fa8180b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.122+0000 7fa81cee4640 1 -- 172.21.15.81:0/3308226545 shutdown_connections 2024-06-29T14:22:55.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.122+0000 7fa81cee4640 1 -- 172.21.15.81:0/3308226545 wait complete. 2024-06-29T14:22:55.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.125+0000 7fa81cee4640 1 Processor -- start 2024-06-29T14:22:55.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.125+0000 7fa81cee4640 1 -- start start 2024-06-29T14:22:55.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa81cee4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8180b3200 0x7fa8181df830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa81cee4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8180b4f40 0x7fa8181f8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa81cee4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8180d0230 0x7fa8181f8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa81cee4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa8180b5c20 con 0x7fa8180d0230 2024-06-29T14:22:55.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa81cee4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa8180b5aa0 con 0x7fa8180b3200 2024-06-29T14:22:55.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa81cee4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa8180b5da0 con 0x7fa8180b4f40 2024-06-29T14:22:55.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa816d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8180d0230 0x7fa8181f8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa816d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8180d0230 0x7fa8181f8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36434/0 (socket says 172.21.15.81:36434) 2024-06-29T14:22:55.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa80dd9b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8180b4f40 0x7fa8181f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa816d9d640 1 -- 172.21.15.81:0/2399228968 learned_addr learned my addr 172.21.15.81:0/2399228968 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:55.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa81659c640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8180b3200 0x7fa8181df830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa816d9d640 1 -- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8180b4f40 msgr2=0x7fa8181f8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa816d9d640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8180b4f40 0x7fa8181f8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa816d9d640 1 -- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8180b3200 msgr2=0x7fa8181df830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa816d9d640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8180b3200 0x7fa8181df830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.126+0000 7fa816d9d640 1 -- 172.21.15.81:0/2399228968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8181e20f0 con 0x7fa8180d0230 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.127+0000 7fa816d9d640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8180d0230 0x7fa8181f8ab0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa810009cd0 tx=0x7fa8100316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.127+0000 7fa80f7fe640 1 -- 172.21.15.81:0/2399228968 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810031990 con 0x7fa8180d0230 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.127+0000 7fa80f7fe640 1 -- 172.21.15.81:0/2399228968 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa810031c20 con 0x7fa8180d0230 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.127+0000 7fa80f7fe640 1 -- 172.21.15.81:0/2399228968 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8100330d0 con 0x7fa8180d0230 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.127+0000 7fa81cee4640 1 -- 172.21.15.81:0/2399228968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81806d4a0 con 0x7fa8180d0230 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.127+0000 7fa81cee4640 1 -- 172.21.15.81:0/2399228968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81806da60 con 0x7fa8180d0230 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.128+0000 7fa81cee4640 1 -- 172.21.15.81:0/2399228968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8180b5350 con 0x7fa8180d0230 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.129+0000 7fa80f7fe640 1 -- 172.21.15.81:0/2399228968 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa810033270 con 0x7fa8180d0230 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.129+0000 7fa80f7fe640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa7f80762f0 0x7fa7f8078800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.129+0000 7fa80f7fe640 1 -- 172.21.15.81:0/2399228968 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa8100b89b0 con 0x7fa8180d0230 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.129+0000 7fa81659c640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa7f80762f0 0x7fa7f8078800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.130+0000 7fa81659c640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa7f80762f0 0x7fa7f8078800 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa808000f60 tx=0x7fa808009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.133+0000 7fa80f7fe640 1 -- 172.21.15.81:0/2399228968 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa81003f030 con 0x7fa8180d0230 2024-06-29T14:22:55.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.315+0000 7fa81cee4640 1 -- 172.21.15.81:0/2399228968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa8180cdf30 con 0x7fa8180d0230 2024-06-29T14:22:55.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.315+0000 7fa80f7fe640 1 -- 172.21.15.81:0/2399228968 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7fa810081cc0 con 0x7fa8180d0230 2024-06-29T14:22:55.314 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:22:55.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.317+0000 7fa81cee4640 1 -- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa7f80762f0 msgr2=0x7fa7f8078800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.317+0000 7fa81cee4640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa7f80762f0 0x7fa7f8078800 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa808000f60 tx=0x7fa808009040 comp rx=0 tx=0).stop 2024-06-29T14:22:55.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.318+0000 7fa81cee4640 1 -- 172.21.15.81:0/2399228968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8180d0230 msgr2=0x7fa8181f8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.318+0000 7fa81cee4640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8180d0230 0x7fa8181f8ab0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa810009cd0 tx=0x7fa8100316d0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.318+0000 7fa81cee4640 1 -- 172.21.15.81:0/2399228968 shutdown_connections 2024-06-29T14:22:55.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.318+0000 7fa81cee4640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa7f80762f0 0x7fa7f8078800 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.318+0000 7fa81cee4640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8180d0230 0x7fa8181f8ab0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.318+0000 7fa81cee4640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8180b4f40 0x7fa8181f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.318+0000 7fa81cee4640 1 --2- 172.21.15.81:0/2399228968 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8180b3200 0x7fa8181df830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.318+0000 7fa81cee4640 1 -- 172.21.15.81:0/2399228968 >> 172.21.15.81:0/2399228968 conn(0x7fa8180b3680 msgr2=0x7fa8180700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.318+0000 7fa81cee4640 1 -- 172.21.15.81:0/2399228968 shutdown_connections 2024-06-29T14:22:55.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.318+0000 7fa81cee4640 1 -- 172.21.15.81:0/2399228968 wait complete. 2024-06-29T14:22:55.327 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.443+0000 7f333023c640 1 -- 172.21.15.81:0/1009566071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33280d0d00 msgr2=0x7f33280cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.443+0000 7f333023c640 1 --2- 172.21.15.81:0/1009566071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33280d0d00 0x7f33280cbfd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3318005fd0 tx=0x7f331802a530 comp rx=0 tx=0).stop 2024-06-29T14:22:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.444+0000 7f333023c640 1 -- 172.21.15.81:0/1009566071 shutdown_connections 2024-06-29T14:22:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.444+0000 7f333023c640 1 --2- 172.21.15.81:0/1009566071 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33280cc510 0x7f33280cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.444+0000 7f333023c640 1 --2- 172.21.15.81:0/1009566071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33280d0d00 0x7f33280cbfd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.444+0000 7f333023c640 1 -- 172.21.15.81:0/1009566071 >> 172.21.15.81:0/1009566071 conn(0x7f33280b3510 msgr2=0x7f33280b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.444+0000 7f333023c640 1 -- 172.21.15.81:0/1009566071 shutdown_connections 2024-06-29T14:22:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.444+0000 7f333023c640 1 -- 172.21.15.81:0/1009566071 wait complete. 2024-06-29T14:22:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.445+0000 7f333023c640 1 Processor -- start 2024-06-29T14:22:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.445+0000 7f333023c640 1 -- start start 2024-06-29T14:22:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f333023c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33280b7040 0x7f3328161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f333023c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33280cc510 0x7f33281f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f333023c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33280d0d00 0x7f33281e7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f333023c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f33280ced90 con 0x7f33280d0d00 2024-06-29T14:22:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f333023c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f33280cec10 con 0x7f33280b7040 2024-06-29T14:22:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f333023c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f33280b4030 con 0x7f33280cc510 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332e7d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33280d0d00 0x7f33281e7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332e7d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33280d0d00 0x7f33281e7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36454/0 (socket says 172.21.15.81:36454) 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332e7d9640 1 -- 172.21.15.81:0/3863485379 learned_addr learned my addr 172.21.15.81:0/3863485379 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332dfd8640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33280b7040 0x7f3328161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332d7d7640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33280cc510 0x7f33281f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332e7d9640 1 -- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33280cc510 msgr2=0x7f33281f59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332e7d9640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33280cc510 0x7f33281f59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332e7d9640 1 -- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33280b7040 msgr2=0x7f3328161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332e7d9640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33280b7040 0x7f3328161bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332e7d9640 1 -- 172.21.15.81:0/3863485379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33281e9350 con 0x7f33280d0d00 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f332e7d9640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33280d0d00 0x7f33281e7470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f332000e990 tx=0x7f332000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f331e7fc640 1 -- 172.21.15.81:0/3863485379 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3320004d20 con 0x7f33280d0d00 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f331e7fc640 1 -- 172.21.15.81:0/3863485379 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f332000caa0 con 0x7f33280d0d00 2024-06-29T14:22:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f331e7fc640 1 -- 172.21.15.81:0/3863485379 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3320005020 con 0x7f33280d0d00 2024-06-29T14:22:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.447+0000 7f333023c640 1 -- 172.21.15.81:0/3863485379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332809bbe0 con 0x7f33280d0d00 2024-06-29T14:22:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.448+0000 7f333023c640 1 -- 172.21.15.81:0/3863485379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f332809c1a0 con 0x7f33280d0d00 2024-06-29T14:22:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.449+0000 7f333023c640 1 -- 172.21.15.81:0/3863485379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33281e90c0 con 0x7f33280d0d00 2024-06-29T14:22:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.449+0000 7f331e7fc640 1 -- 172.21.15.81:0/3863485379 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f33200051c0 con 0x7f33280d0d00 2024-06-29T14:22:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.450+0000 7f331e7fc640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f330c076300 0x7f330c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.450+0000 7f331e7fc640 1 -- 172.21.15.81:0/3863485379 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3320097f40 con 0x7f33280d0d00 2024-06-29T14:22:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.450+0000 7f332dfd8640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f330c076300 0x7f330c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.450+0000 7f332dfd8640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f330c076300 0x7f330c078810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3324006fd0 tx=0x7f3324009060 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.454+0000 7f331e7fc640 1 -- 172.21.15.81:0/3863485379 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3320016030 con 0x7f33280d0d00 2024-06-29T14:22:55.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.634+0000 7f333023c640 1 -- 172.21.15.81:0/3863485379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f33280cc920 con 0x7f33280d0d00 2024-06-29T14:22:55.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.634+0000 7f331e7fc640 1 -- 172.21.15.81:0/3863485379 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f33200611c0 con 0x7f33280d0d00 2024-06-29T14:22:55.632 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 -- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f330c076300 msgr2=0x7f330c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f330c076300 0x7f330c078810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3324006fd0 tx=0x7f3324009060 comp rx=0 tx=0).stop 2024-06-29T14:22:55.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 -- 172.21.15.81:0/3863485379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33280d0d00 msgr2=0x7f33281e7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33280d0d00 0x7f33281e7470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f332000e990 tx=0x7f332000ee50 comp rx=0 tx=0).stop 2024-06-29T14:22:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 -- 172.21.15.81:0/3863485379 shutdown_connections 2024-06-29T14:22:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f330c076300 0x7f330c078810 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33280d0d00 0x7f33281e7470 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33280cc510 0x7f33281f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 --2- 172.21.15.81:0/3863485379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33280b7040 0x7f3328161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 -- 172.21.15.81:0/3863485379 >> 172.21.15.81:0/3863485379 conn(0x7f33280b3510 msgr2=0x7f33280d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 -- 172.21.15.81:0/3863485379 shutdown_connections 2024-06-29T14:22:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.636+0000 7f333023c640 1 -- 172.21.15.81:0/3863485379 wait complete. 2024-06-29T14:22:55.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:55.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.761+0000 7f839949e640 1 -- 172.21.15.81:0/2288835377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8380003a30 msgr2=0x7f8380003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.761+0000 7f839949e640 1 --2- 172.21.15.81:0/2288835377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8380003a30 0x7f8380003ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f838c0013d0 tx=0x7f8380005830 comp rx=0 tx=0).stop 2024-06-29T14:22:55.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.761+0000 7f83937fe640 1 -- 172.21.15.81:0/2288835377 reap_dead start 2024-06-29T14:22:55.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.761+0000 7f839949e640 1 -- 172.21.15.81:0/2288835377 shutdown_connections 2024-06-29T14:22:55.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.761+0000 7f839949e640 1 -- 172.21.15.81:0/2288835377 >> 172.21.15.81:0/2288835377 conn(0x7f839409cf30 msgr2=0x7f83940d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.761+0000 7f839949e640 1 -- 172.21.15.81:0/2288835377 shutdown_connections 2024-06-29T14:22:55.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.761+0000 7f839949e640 1 -- 172.21.15.81:0/2288835377 wait complete. 2024-06-29T14:22:55.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.763+0000 7f839949e640 1 Processor -- start 2024-06-29T14:22:55.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.763+0000 7f839949e640 1 -- start start 2024-06-29T14:22:55.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f839949e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8380003a30 0x7f8394150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f839949e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394151590 0x7f83941e8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f839949e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83940cccd0 0x7f83940cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f839949e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f83940d2310 con 0x7f8394151590 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f839949e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f83940d2190 con 0x7f8380003a30 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f839949e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f83940d2490 con 0x7f83940cccd0 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f83927fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394151590 0x7f83941e8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f83927fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394151590 0x7f83941e8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36472/0 (socket says 172.21.15.81:36472) 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f83927fc640 1 -- 172.21.15.81:0/3054175899 learned_addr learned my addr 172.21.15.81:0/3054175899 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f8392ffd640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8380003a30 0x7f8394150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f83937fe640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83940cccd0 0x7f83940cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f83927fc640 1 -- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83940cccd0 msgr2=0x7f83940cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f83927fc640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83940cccd0 0x7f83940cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f83927fc640 1 -- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8380003a30 msgr2=0x7f8394150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f83927fc640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8380003a30 0x7f8394150ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.764+0000 7f83927fc640 1 -- 172.21.15.81:0/3054175899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83940cd620 con 0x7f8394151590 2024-06-29T14:22:55.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.765+0000 7f83927fc640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394151590 0x7f83941e8460 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f837c004ed0 tx=0x7f837c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.765+0000 7f838b7fe640 1 -- 172.21.15.81:0/3054175899 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c015020 con 0x7f8394151590 2024-06-29T14:22:55.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.765+0000 7f838b7fe640 1 -- 172.21.15.81:0/3054175899 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f837c011960 con 0x7f8394151590 2024-06-29T14:22:55.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.765+0000 7f839949e640 1 -- 172.21.15.81:0/3054175899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83941e69d0 con 0x7f8394151590 2024-06-29T14:22:55.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.765+0000 7f838b7fe640 1 -- 172.21.15.81:0/3054175899 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c011d90 con 0x7f8394151590 2024-06-29T14:22:55.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.765+0000 7f839949e640 1 -- 172.21.15.81:0/3054175899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83941e6f50 con 0x7f8394151590 2024-06-29T14:22:55.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.766+0000 7f839949e640 1 -- 172.21.15.81:0/3054175899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83940d2a20 con 0x7f8394151590 2024-06-29T14:22:55.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.767+0000 7f838b7fe640 1 -- 172.21.15.81:0/3054175899 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f837c018020 con 0x7f8394151590 2024-06-29T14:22:55.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.767+0000 7f838b7fe640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f83740762d0 0x7f83740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:55.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.767+0000 7f838b7fe640 1 -- 172.21.15.81:0/3054175899 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f837c098080 con 0x7f8394151590 2024-06-29T14:22:55.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.767+0000 7f8392ffd640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f83740762d0 0x7f83740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:55.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.768+0000 7f8392ffd640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f83740762d0 0x7f83740787e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f838c033000 tx=0x7f838c001d70 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:55.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.772+0000 7f838b7fe640 1 -- 172.21.15.81:0/3054175899 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f837c01f020 con 0x7f8394151590 2024-06-29T14:22:55.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.988+0000 7f839949e640 1 -- 172.21.15.81:0/3054175899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83940682b0 con 0x7f8394151590 2024-06-29T14:22:55.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.989+0000 7f838b7fe640 1 -- 172.21.15.81:0/3054175899 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5217 (secure 0 0 0) 0x7f83940cd620 con 0x7f8394151590 2024-06-29T14:22:55.987 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:55.987 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"btime":"2024-06-29T14:22:47:707315+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4295,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/96042442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":96042442},{"type":"v1","addr":"172.21.15.167:6837","nonce":96042442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3681460169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3681460169},{"type":"v1","addr":"172.21.15.81:6834","nonce":3681460169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3175849002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3175849002},{"type":"v1","addr":"172.21.15.81:6835","nonce":3175849002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:22:44.057632+0000","modified":"2024-06-29T14:22:47.079866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4360":{"gid":4360,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/3579540962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3579540962},{"type":"v1","addr":"172.21.15.167:6836","nonce":3579540962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4360,"qdb_cluster":[4360]},"id":1}]} 2024-06-29T14:22:55.988 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 -- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f83740762d0 msgr2=0x7f83740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f83740762d0 0x7f83740787e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f838c033000 tx=0x7f838c001d70 comp rx=0 tx=0).stop 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 -- 172.21.15.81:0/3054175899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394151590 msgr2=0x7f83941e8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394151590 0x7f83941e8460 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f837c004ed0 tx=0x7f837c001630 comp rx=0 tx=0).stop 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 -- 172.21.15.81:0/3054175899 shutdown_connections 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f83740762d0 0x7f83740787e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83940cccd0 0x7f83940cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394151590 0x7f83941e8460 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 --2- 172.21.15.81:0/3054175899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8380003a30 0x7f8394150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 -- 172.21.15.81:0/3054175899 >> 172.21.15.81:0/3054175899 conn(0x7f839409cf30 msgr2=0x7f83940b85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 -- 172.21.15.81:0/3054175899 shutdown_connections 2024-06-29T14:22:55.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:55.991+0000 7f839949e640 1 -- 172.21.15.81:0/3054175899 wait complete. 2024-06-29T14:22:56.001 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.115+0000 7f8199826640 1 -- 172.21.15.81:0/3276836253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81940b5250 msgr2=0x7f81940d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.115+0000 7f8199826640 1 --2- 172.21.15.81:0/3276836253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81940b5250 0x7f81940d09c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f819408e290 tx=0x7f817c02a610 comp rx=0 tx=0).stop 2024-06-29T14:22:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.115+0000 7f8199826640 1 -- 172.21.15.81:0/3276836253 shutdown_connections 2024-06-29T14:22:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.115+0000 7f8199826640 1 --2- 172.21.15.81:0/3276836253 >> v2:172.21.15.167:3300/0 conn(0x7f81940b7290 0x7f81940b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.115+0000 7f8199826640 1 --2- 172.21.15.81:0/3276836253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81940b5250 0x7f81940d09c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.115+0000 7f8199826640 1 --2- 172.21.15.81:0/3276836253 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81940b9040 0x7f81940d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.115+0000 7f8199826640 1 -- 172.21.15.81:0/3276836253 >> 172.21.15.81:0/3276836253 conn(0x7f819409cf30 msgr2=0x7f81940d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.115+0000 7f8199826640 1 -- 172.21.15.81:0/3276836253 shutdown_connections 2024-06-29T14:22:56.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.115+0000 7f8199826640 1 -- 172.21.15.81:0/3276836253 wait complete. 2024-06-29T14:22:56.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.116+0000 7f8199826640 1 Processor -- start 2024-06-29T14:22:56.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.117+0000 7f8199826640 1 -- start start 2024-06-29T14:22:56.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f8199826640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81940b5250 0x7f8194156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f8199826640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81940b7290 0x7f81940b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f8199826640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81940b9040 0x7f81941538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f8199826640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f81940b7ff0 con 0x7f81940b9040 2024-06-29T14:22:56.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f8199826640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f81940b7e70 con 0x7f81940b5250 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f8199826640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f81940b8170 con 0x7f81940b7290 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f81937fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81940b9040 0x7f81941538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f81937fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81940b9040 0x7f81941538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36494/0 (socket says 172.21.15.81:36494) 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f81937fe640 1 -- 172.21.15.81:0/3519178660 learned_addr learned my addr 172.21.15.81:0/3519178660 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f818a7fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81940b7290 0x7f81940b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f8192ffd640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81940b5250 0x7f8194156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f81937fe640 1 -- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81940b7290 msgr2=0x7f81940b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f81937fe640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81940b7290 0x7f81940b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f81937fe640 1 -- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81940b5250 msgr2=0x7f8194156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f81937fe640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81940b5250 0x7f8194156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f81937fe640 1 -- 172.21.15.81:0/3519178660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81940b38f0 con 0x7f81940b9040 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.118+0000 7f81937fe640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81940b9040 0x7f81941538c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f818c00b9e0 tx=0x7f818c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.119+0000 7f8190ff9640 1 -- 172.21.15.81:0/3519178660 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c019070 con 0x7f81940b9040 2024-06-29T14:22:56.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.119+0000 7f8190ff9640 1 -- 172.21.15.81:0/3519178660 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f818c00cc30 con 0x7f81940b9040 2024-06-29T14:22:56.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.119+0000 7f8190ff9640 1 -- 172.21.15.81:0/3519178660 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c0146a0 con 0x7f81940b9040 2024-06-29T14:22:56.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.119+0000 7f8199826640 1 -- 172.21.15.81:0/3519178660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81940b3bd0 con 0x7f81940b9040 2024-06-29T14:22:56.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.119+0000 7f8199826640 1 -- 172.21.15.81:0/3519178660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81940b4190 con 0x7f81940b9040 2024-06-29T14:22:56.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.120+0000 7f8199826640 1 -- 172.21.15.81:0/3519178660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81940b3640 con 0x7f81940b9040 2024-06-29T14:22:56.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.120+0000 7f8190ff9640 1 -- 172.21.15.81:0/3519178660 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f818c01c070 con 0x7f81940b9040 2024-06-29T14:22:56.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.121+0000 7f8190ff9640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81740762d0 0x7f81740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.121+0000 7f8190ff9640 1 -- 172.21.15.81:0/3519178660 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f818c097390 con 0x7f81940b9040 2024-06-29T14:22:56.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.121+0000 7f8192ffd640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81740762d0 0x7f81740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.122+0000 7f8192ffd640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81740762d0 0x7f81740787e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f818400a490 tx=0x7f818400a420 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.125+0000 7f8190ff9640 1 -- 172.21.15.81:0/3519178660 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f818c00ce20 con 0x7f81940b9040 2024-06-29T14:22:56.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.307+0000 7f8199826640 1 -- 172.21.15.81:0/3519178660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81940b76a0 con 0x7f81940b9040 2024-06-29T14:22:56.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.308+0000 7f8190ff9640 1 -- 172.21.15.81:0/3519178660 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f818c060730 con 0x7f81940b9040 2024-06-29T14:22:56.307 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:56.307 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:56.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 -- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81740762d0 msgr2=0x7f81740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81740762d0 0x7f81740787e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f818400a490 tx=0x7f818400a420 comp rx=0 tx=0).stop 2024-06-29T14:22:56.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 -- 172.21.15.81:0/3519178660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81940b9040 msgr2=0x7f81941538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81940b9040 0x7f81941538c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f818c00b9e0 tx=0x7f818c00bea0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 -- 172.21.15.81:0/3519178660 shutdown_connections 2024-06-29T14:22:56.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81740762d0 0x7f81740787e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81940b9040 0x7f81941538c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81940b7290 0x7f81940b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 --2- 172.21.15.81:0/3519178660 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81940b5250 0x7f8194156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 -- 172.21.15.81:0/3519178660 >> 172.21.15.81:0/3519178660 conn(0x7f819409cf30 msgr2=0x7f81940683c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 -- 172.21.15.81:0/3519178660 shutdown_connections 2024-06-29T14:22:56.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.310+0000 7f8199826640 1 -- 172.21.15.81:0/3519178660 wait complete. 2024-06-29T14:22:56.319 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:22:56.320 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:22:56.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.437+0000 7f7b29c47640 1 -- 172.21.15.81:0/3307428440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b240ce910 msgr2=0x7f7b240cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.437+0000 7f7b29c47640 1 --2- 172.21.15.81:0/3307428440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b240ce910 0x7f7b240cc6e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c005fd0 tx=0x7f7b1c02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.437+0000 7f7b29c47640 1 -- 172.21.15.81:0/3307428440 shutdown_connections 2024-06-29T14:22:56.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.437+0000 7f7b29c47640 1 --2- 172.21.15.81:0/3307428440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b240ce910 0x7f7b240cc6e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.437+0000 7f7b29c47640 1 --2- 172.21.15.81:0/3307428440 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b240b7040 0x7f7b240d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.437+0000 7f7b29c47640 1 -- 172.21.15.81:0/3307428440 >> 172.21.15.81:0/3307428440 conn(0x7f7b240b3510 msgr2=0x7f7b240b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.437+0000 7f7b29c47640 1 -- 172.21.15.81:0/3307428440 shutdown_connections 2024-06-29T14:22:56.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.437+0000 7f7b29c47640 1 -- 172.21.15.81:0/3307428440 wait complete. 2024-06-29T14:22:56.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.439+0000 7f7b29c47640 1 Processor -- start 2024-06-29T14:22:56.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.439+0000 7f7b29c47640 1 -- start start 2024-06-29T14:22:56.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b29c47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b240b7040 0x7f7b241e7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b29c47640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7b240ce500 0x7f7b24148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b29c47640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b240ce910 0x7f7b240b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b29c47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7b240ccda0 con 0x7f7b240b7040 2024-06-29T14:22:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b29c47640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7b240ccc20 con 0x7f7b240ce500 2024-06-29T14:22:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b29c47640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7b240b4030 con 0x7f7b240ce910 2024-06-29T14:22:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b237fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b240b7040 0x7f7b241e7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b237fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b240b7040 0x7f7b241e7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36508/0 (socket says 172.21.15.81:36508) 2024-06-29T14:22:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b237fe640 1 -- 172.21.15.81:0/3241807374 learned_addr learned my addr 172.21.15.81:0/3241807374 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b237fe640 1 -- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b240ce910 msgr2=0x7f7b240b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b237fe640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b240ce910 0x7f7b240b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b237fe640 1 -- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7b240ce500 msgr2=0x7f7b24148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b237fe640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7b240ce500 0x7f7b24148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.440+0000 7f7b237fe640 1 -- 172.21.15.81:0/3241807374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2409cff0 con 0x7f7b240b7040 2024-06-29T14:22:56.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.441+0000 7f7b237fe640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b240b7040 0x7f7b241e7470 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7b1400d700 tx=0x7f7b1400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.441+0000 7f7b0bfff640 1 -- 172.21.15.81:0/3241807374 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b14017070 con 0x7f7b240b7040 2024-06-29T14:22:56.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.441+0000 7f7b0bfff640 1 -- 172.21.15.81:0/3241807374 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b14004310 con 0x7f7b240b7040 2024-06-29T14:22:56.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.441+0000 7f7b0bfff640 1 -- 172.21.15.81:0/3241807374 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b14009a80 con 0x7f7b240b7040 2024-06-29T14:22:56.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.441+0000 7f7b29c47640 1 -- 172.21.15.81:0/3241807374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b2409d2d0 con 0x7f7b240b7040 2024-06-29T14:22:56.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.441+0000 7f7b29c47640 1 -- 172.21.15.81:0/3241807374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b2409d890 con 0x7f7b240b7040 2024-06-29T14:22:56.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.442+0000 7f7b29c47640 1 -- 172.21.15.81:0/3241807374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b240d0d00 con 0x7f7b240b7040 2024-06-29T14:22:56.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.443+0000 7f7b0bfff640 1 -- 172.21.15.81:0/3241807374 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7b14009c20 con 0x7f7b240b7040 2024-06-29T14:22:56.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.443+0000 7f7b0bfff640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7b100762d0 0x7f7b100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.443+0000 7f7b0bfff640 1 -- 172.21.15.81:0/3241807374 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7b140977f0 con 0x7f7b240b7040 2024-06-29T14:22:56.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.443+0000 7f7b22ffd640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7b100762d0 0x7f7b100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.444+0000 7f7b22ffd640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7b100762d0 0x7f7b100787e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7b24158a30 tx=0x7f7b18007a30 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.448+0000 7f7b0bfff640 1 -- 172.21.15.81:0/3241807374 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7b140610c0 con 0x7f7b240b7040 2024-06-29T14:22:56.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.627+0000 7f7b29c47640 1 -- 172.21.15.81:0/3241807374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7b240b3ca0 con 0x7f7b240b7040 2024-06-29T14:22:56.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.627+0000 7f7b0bfff640 1 -- 172.21.15.81:0/3241807374 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f7b140609c0 con 0x7f7b240b7040 2024-06-29T14:22:56.626 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:22:56.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.629+0000 7f7b29c47640 1 -- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7b100762d0 msgr2=0x7f7b100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.629+0000 7f7b29c47640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7b100762d0 0x7f7b100787e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7b24158a30 tx=0x7f7b18007a30 comp rx=0 tx=0).stop 2024-06-29T14:22:56.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.629+0000 7f7b29c47640 1 -- 172.21.15.81:0/3241807374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b240b7040 msgr2=0x7f7b241e7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.629+0000 7f7b29c47640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b240b7040 0x7f7b241e7470 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7b1400d700 tx=0x7f7b1400dbc0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.630+0000 7f7b29c47640 1 -- 172.21.15.81:0/3241807374 shutdown_connections 2024-06-29T14:22:56.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.630+0000 7f7b29c47640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7b100762d0 0x7f7b100787e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.630+0000 7f7b29c47640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7b240ce910 0x7f7b240b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.630+0000 7f7b29c47640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7b240ce500 0x7f7b24148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.630+0000 7f7b29c47640 1 --2- 172.21.15.81:0/3241807374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b240b7040 0x7f7b241e7470 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.630+0000 7f7b29c47640 1 -- 172.21.15.81:0/3241807374 >> 172.21.15.81:0/3241807374 conn(0x7f7b240b3510 msgr2=0x7f7b240705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.630+0000 7f7b29c47640 1 -- 172.21.15.81:0/3241807374 shutdown_connections 2024-06-29T14:22:56.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.630+0000 7f7b29c47640 1 -- 172.21.15.81:0/3241807374 wait complete. 2024-06-29T14:22:56.639 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:22:56.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.758+0000 7f0c709c7640 1 -- 172.21.15.81:0/2194484891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c6c15bc20 msgr2=0x7f0c6c06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.758+0000 7f0c709c7640 1 --2- 172.21.15.81:0/2194484891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c6c15bc20 0x7f0c6c06ed90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c64005fd0 tx=0x7f0c6402a820 comp rx=0 tx=0).stop 2024-06-29T14:22:56.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.758+0000 7f0c709c7640 1 -- 172.21.15.81:0/2194484891 shutdown_connections 2024-06-29T14:22:56.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.758+0000 7f0c709c7640 1 --2- 172.21.15.81:0/2194484891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c6c15bc20 0x7f0c6c06ed90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.758+0000 7f0c709c7640 1 --2- 172.21.15.81:0/2194484891 >> v2:172.21.15.167:3300/0 conn(0x7f0c6c073650 0x7f0c6c15b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.758+0000 7f0c709c7640 1 -- 172.21.15.81:0/2194484891 >> 172.21.15.81:0/2194484891 conn(0x7f0c6c0b2920 msgr2=0x7f0c6c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.758+0000 7f0c709c7640 1 -- 172.21.15.81:0/2194484891 shutdown_connections 2024-06-29T14:22:56.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.758+0000 7f0c709c7640 1 -- 172.21.15.81:0/2194484891 wait complete. 2024-06-29T14:22:56.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.760+0000 7f0c709c7640 1 Processor -- start 2024-06-29T14:22:56.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.760+0000 7f0c709c7640 1 -- start start 2024-06-29T14:22:56.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c709c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c6c0cd150 0x7f0c6c0d0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c709c7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c6c15bc20 0x7f0c6c0afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c709c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c6c0d0dc0 0x7f0c6c155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c709c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6c06f660 con 0x7f0c6c0cd150 2024-06-29T14:22:56.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c709c7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6c06f4e0 con 0x7f0c6c0d0dc0 2024-06-29T14:22:56.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c709c7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0c6c06f7e0 con 0x7f0c6c15bc20 2024-06-29T14:22:56.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c6a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c6c0cd150 0x7f0c6c0d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c6a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c6c0cd150 0x7f0c6c0d0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36522/0 (socket says 172.21.15.81:36522) 2024-06-29T14:22:56.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c6a59c640 1 -- 172.21.15.81:0/3350362586 learned_addr learned my addr 172.21.15.81:0/3350362586 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c69d9b640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c6c15bc20 0x7f0c6c0afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.761+0000 7f0c6ad9d640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c6c0d0dc0 0x7f0c6c155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c6a59c640 1 -- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c6c15bc20 msgr2=0x7f0c6c0afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c6a59c640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c6c15bc20 0x7f0c6c0afbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c6a59c640 1 -- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c6c0d0dc0 msgr2=0x7f0c6c155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c6a59c640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c6c0d0dc0 0x7f0c6c155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c6a59c640 1 -- 172.21.15.81:0/3350362586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6c1f1800 con 0x7f0c6c0cd150 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c6a59c640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c6c0cd150 0x7f0c6c0d0460 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00c3d0 tx=0x7f0c5c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c5affd640 1 -- 172.21.15.81:0/3350362586 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c017070 con 0x7f0c6c0cd150 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c5affd640 1 -- 172.21.15.81:0/3350362586 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c5c0053c0 con 0x7f0c6c0cd150 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c709c7640 1 -- 172.21.15.81:0/3350362586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c6c1f1ac0 con 0x7f0c6c0cd150 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c5affd640 1 -- 172.21.15.81:0/3350362586 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c003850 con 0x7f0c6c0cd150 2024-06-29T14:22:56.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.762+0000 7f0c709c7640 1 -- 172.21.15.81:0/3350362586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c6c1f2080 con 0x7f0c6c0cd150 2024-06-29T14:22:56.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.763+0000 7f0c709c7640 1 -- 172.21.15.81:0/3350362586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c6c06fd00 con 0x7f0c6c0cd150 2024-06-29T14:22:56.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.764+0000 7f0c5affd640 1 -- 172.21.15.81:0/3350362586 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0c5c01a070 con 0x7f0c6c0cd150 2024-06-29T14:22:56.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.764+0000 7f0c5affd640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0c4c0762d0 0x7f0c4c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:56.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.764+0000 7f0c5affd640 1 -- 172.21.15.81:0/3350362586 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0c5c096ab0 con 0x7f0c6c0cd150 2024-06-29T14:22:56.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.764+0000 7f0c69d9b640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0c4c0762d0 0x7f0c4c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:56.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.765+0000 7f0c69d9b640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0c4c0762d0 0x7f0c4c0787e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c600096c0 tx=0x7f0c60009260 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:56.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.769+0000 7f0c5affd640 1 -- 172.21.15.81:0/3350362586 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0c5c05fdc0 con 0x7f0c6c0cd150 2024-06-29T14:22:56.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.948+0000 7f0c709c7640 1 -- 172.21.15.81:0/3350362586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0c6c070ea0 con 0x7f0c6c0cd150 2024-06-29T14:22:56.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.948+0000 7f0c5affd640 1 -- 172.21.15.81:0/3350362586 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f0c5c0640c0 con 0x7f0c6c0cd150 2024-06-29T14:22:56.947 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.950+0000 7f0c709c7640 1 -- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0c4c0762d0 msgr2=0x7f0c4c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.950+0000 7f0c709c7640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0c4c0762d0 0x7f0c4c0787e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c600096c0 tx=0x7f0c60009260 comp rx=0 tx=0).stop 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.950+0000 7f0c709c7640 1 -- 172.21.15.81:0/3350362586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c6c0cd150 msgr2=0x7f0c6c0d0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.950+0000 7f0c709c7640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c6c0cd150 0x7f0c6c0d0460 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00c3d0 tx=0x7f0c5c00c890 comp rx=0 tx=0).stop 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.950+0000 7f0c709c7640 1 -- 172.21.15.81:0/3350362586 shutdown_connections 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.950+0000 7f0c709c7640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0c4c0762d0 0x7f0c4c0787e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.950+0000 7f0c709c7640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c6c0d0dc0 0x7f0c6c155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.950+0000 7f0c709c7640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c6c15bc20 0x7f0c6c0afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.950+0000 7f0c709c7640 1 --2- 172.21.15.81:0/3350362586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c6c0cd150 0x7f0c6c0d0460 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.950+0000 7f0c709c7640 1 -- 172.21.15.81:0/3350362586 >> 172.21.15.81:0/3350362586 conn(0x7f0c6c0b2920 msgr2=0x7f0c6c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:56.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.951+0000 7f0c709c7640 1 -- 172.21.15.81:0/3350362586 shutdown_connections 2024-06-29T14:22:56.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:56.951+0000 7f0c709c7640 1 -- 172.21.15.81:0/3350362586 wait complete. 2024-06-29T14:22:56.960 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:22:57.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.078+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/2371323733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6e40ce920 msgr2=0x7fa6e40cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.078+0000 7fa6ea1eb640 1 --2- 172.21.15.81:0/2371323733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6e40ce920 0x7fa6e40cd480 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc005fd0 tx=0x7fa6dc02aa10 comp rx=0 tx=0).stop 2024-06-29T14:22:57.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.078+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/2371323733 shutdown_connections 2024-06-29T14:22:57.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.078+0000 7fa6ea1eb640 1 --2- 172.21.15.81:0/2371323733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6e40ce920 0x7fa6e40cd480 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.078+0000 7fa6ea1eb640 1 --2- 172.21.15.81:0/2371323733 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6e40b6eb0 0x7fa6e40d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.078+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/2371323733 >> 172.21.15.81:0/2371323733 conn(0x7fa6e40b3510 msgr2=0x7fa6e40b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:57.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.078+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/2371323733 shutdown_connections 2024-06-29T14:22:57.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.078+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/2371323733 wait complete. 2024-06-29T14:22:57.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.080+0000 7fa6ea1eb640 1 Processor -- start 2024-06-29T14:22:57.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.080+0000 7fa6ea1eb640 1 -- start start 2024-06-29T14:22:57.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6ea1eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6e40b6eb0 0x7fa6e4158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6ea1eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6e40ce510 0x7fa6e415f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6ea1eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6e40ce920 0x7fa6e4162eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6ea1eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e40b50a0 con 0x7fa6e40ce510 2024-06-29T14:22:57.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6ea1eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e40b4f20 con 0x7fa6e40ce920 2024-06-29T14:22:57.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6ea1eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa6e40b5220 con 0x7fa6e40b6eb0 2024-06-29T14:22:57.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6e40ce510 0x7fa6e415f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6e40ce510 0x7fa6e415f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36534/0 (socket says 172.21.15.81:36534) 2024-06-29T14:22:57.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e2ffd640 1 -- 172.21.15.81:0/3797958012 learned_addr learned my addr 172.21.15.81:0/3797958012 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:57.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e37fe640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6e40b6eb0 0x7fa6e4158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e3fff640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6e40ce920 0x7fa6e4162eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e2ffd640 1 -- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6e40b6eb0 msgr2=0x7fa6e4158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e2ffd640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6e40b6eb0 0x7fa6e4158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e2ffd640 1 -- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6e40ce920 msgr2=0x7fa6e4162eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e2ffd640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6e40ce920 0x7fa6e4162eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e2ffd640 1 -- 172.21.15.81:0/3797958012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e40aee80 con 0x7fa6e40ce510 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.081+0000 7fa6e3fff640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6e40ce920 0x7fa6e4162eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.082+0000 7fa6e2ffd640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6e40ce510 0x7fa6e415f160 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800c3d0 tx=0x7fa6d800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.082+0000 7fa6cbfff640 1 -- 172.21.15.81:0/3797958012 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8017070 con 0x7fa6e40ce510 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.082+0000 7fa6cbfff640 1 -- 172.21.15.81:0/3797958012 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa6d80053c0 con 0x7fa6e40ce510 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.082+0000 7fa6cbfff640 1 -- 172.21.15.81:0/3797958012 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8003850 con 0x7fa6e40ce510 2024-06-29T14:22:57.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.082+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/3797958012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e40af160 con 0x7fa6e40ce510 2024-06-29T14:22:57.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.082+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/3797958012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e40af440 con 0x7fa6e40ce510 2024-06-29T14:22:57.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.083+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/3797958012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e40d02d0 con 0x7fa6e40ce510 2024-06-29T14:22:57.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.083+0000 7fa6cbfff640 1 -- 172.21.15.81:0/3797958012 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa6d801a070 con 0x7fa6e40ce510 2024-06-29T14:22:57.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.084+0000 7fa6cbfff640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa6d0076300 0x7fa6d0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.084+0000 7fa6cbfff640 1 -- 172.21.15.81:0/3797958012 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa6d8096a50 con 0x7fa6e40ce510 2024-06-29T14:22:57.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.084+0000 7fa6e37fe640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa6d0076300 0x7fa6d0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.085+0000 7fa6e37fe640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa6d0076300 0x7fa6d0078810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400bb50 tx=0x7fa6d400b2e0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:57.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.088+0000 7fa6cbfff640 1 -- 172.21.15.81:0/3797958012 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa6d805fd60 con 0x7fa6e40ce510 2024-06-29T14:22:57.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.268+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/3797958012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa6e40cdf00 con 0x7fa6e40ce510 2024-06-29T14:22:57.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.268+0000 7fa6cbfff640 1 -- 172.21.15.81:0/3797958012 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fa6d80683a0 con 0x7fa6e40ce510 2024-06-29T14:22:57.267 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:22:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa6d0076300 msgr2=0x7fa6d0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa6d0076300 0x7fa6d0078810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400bb50 tx=0x7fa6d400b2e0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/3797958012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6e40ce510 msgr2=0x7fa6e415f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6e40ce510 0x7fa6e415f160 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800c3d0 tx=0x7fa6d800c890 comp rx=0 tx=0).stop 2024-06-29T14:22:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/3797958012 shutdown_connections 2024-06-29T14:22:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fa6d0076300 0x7fa6d0078810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6e40ce920 0x7fa6e4162eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6e40ce510 0x7fa6e415f160 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 --2- 172.21.15.81:0/3797958012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6e40b6eb0 0x7fa6e4158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/3797958012 >> 172.21.15.81:0/3797958012 conn(0x7fa6e40b3510 msgr2=0x7fa6e415cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:57.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.270+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/3797958012 shutdown_connections 2024-06-29T14:22:57.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.271+0000 7fa6ea1eb640 1 -- 172.21.15.81:0/3797958012 wait complete. 2024-06-29T14:22:57.280 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:57.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.395+0000 7f2ad712b640 1 -- 172.21.15.81:0/2344187645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ad00d0b90 msgr2=0x7f2ad00d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.395+0000 7f2ad712b640 1 --2- 172.21.15.81:0/2344187645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ad00d0b90 0x7f2ad00d1a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2ad008e330 tx=0x7f2ac002a610 comp rx=0 tx=0).stop 2024-06-29T14:22:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.395+0000 7f2ad712b640 1 -- 172.21.15.81:0/2344187645 shutdown_connections 2024-06-29T14:22:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.395+0000 7f2ad712b640 1 --2- 172.21.15.81:0/2344187645 >> v2:172.21.15.167:3300/0 conn(0x7f2ad00b5250 0x7f2ad00d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.395+0000 7f2ad712b640 1 --2- 172.21.15.81:0/2344187645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ad00d0b90 0x7f2ad00d1a80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.395+0000 7f2ad712b640 1 -- 172.21.15.81:0/2344187645 >> 172.21.15.81:0/2344187645 conn(0x7f2ad009cf30 msgr2=0x7f2ad00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.395+0000 7f2ad712b640 1 -- 172.21.15.81:0/2344187645 shutdown_connections 2024-06-29T14:22:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.395+0000 7f2ad712b640 1 -- 172.21.15.81:0/2344187645 wait complete. 2024-06-29T14:22:57.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.397+0000 7f2ad712b640 1 Processor -- start 2024-06-29T14:22:57.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.397+0000 7f2ad712b640 1 -- start start 2024-06-29T14:22:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.398+0000 7f2ad712b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ad00b5250 0x7f2ad01f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.398+0000 7f2ad712b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ad00b9040 0x7f2ad00c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.398+0000 7f2ad712b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ad00a0770 0x7f2ad00a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.398+0000 7f2ad712b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad00d2500 con 0x7f2ad00b5250 2024-06-29T14:22:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.398+0000 7f2ad712b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad00b84e0 con 0x7f2ad00a0770 2024-06-29T14:22:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.398+0000 7f2ad712b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad00d2680 con 0x7f2ad00b9040 2024-06-29T14:22:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.398+0000 7f2ad4ec7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ad00b5250 0x7f2ad01f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.398+0000 7f2ad4ec7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ad00b5250 0x7f2ad01f58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36548/0 (socket says 172.21.15.81:36548) 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.398+0000 7f2ad4ec7640 1 -- 172.21.15.81:0/589764571 learned_addr learned my addr 172.21.15.81:0/589764571 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ac7fff640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ad00b9040 0x7f2ad00c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ad56c8640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ad00a0770 0x7f2ad00a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ad4ec7640 1 -- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ad00b9040 msgr2=0x7f2ad00c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ad4ec7640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ad00b9040 0x7f2ad00c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ad4ec7640 1 -- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ad00a0770 msgr2=0x7f2ad00a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ad4ec7640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ad00a0770 0x7f2ad00a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ad4ec7640 1 -- 172.21.15.81:0/589764571 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad00a10c0 con 0x7f2ad00b5250 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ad4ec7640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ad00b5250 0x7f2ad01f58a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00ba50 tx=0x7f2acc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ac57fa640 1 -- 172.21.15.81:0/589764571 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc019070 con 0x7f2ad00b5250 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ac57fa640 1 -- 172.21.15.81:0/589764571 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2acc00cd00 con 0x7f2ad00b5250 2024-06-29T14:22:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ac57fa640 1 -- 172.21.15.81:0/589764571 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0146a0 con 0x7f2ad00b5250 2024-06-29T14:22:57.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ad712b640 1 -- 172.21.15.81:0/589764571 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad01f62c0 con 0x7f2ad00b5250 2024-06-29T14:22:57.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.399+0000 7f2ad712b640 1 -- 172.21.15.81:0/589764571 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad01f6880 con 0x7f2ad00b5250 2024-06-29T14:22:57.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.400+0000 7f2ad712b640 1 -- 172.21.15.81:0/589764571 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad00d2c40 con 0x7f2ad00b5250 2024-06-29T14:22:57.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.401+0000 7f2ac57fa640 1 -- 172.21.15.81:0/589764571 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2acc01c070 con 0x7f2ad00b5250 2024-06-29T14:22:57.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.401+0000 7f2ac57fa640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2abc0762d0 0x7f2abc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.401+0000 7f2ac57fa640 1 -- 172.21.15.81:0/589764571 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2acc097710 con 0x7f2ad00b5250 2024-06-29T14:22:57.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.402+0000 7f2ac7fff640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2abc0762d0 0x7f2abc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.402+0000 7f2ac7fff640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2abc0762d0 0x7f2abc0787e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0033000 tx=0x7f2ac0004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:57.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.406+0000 7f2ac57fa640 1 -- 172.21.15.81:0/589764571 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2ad00a10c0 con 0x7f2ad00b5250 2024-06-29T14:22:57.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.624+0000 7f2ad712b640 1 -- 172.21.15.81:0/589764571 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ad015e290 con 0x7f2ad00b5250 2024-06-29T14:22:57.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.624+0000 7f2ac57fa640 1 -- 172.21.15.81:0/589764571 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5217 (secure 0 0 0) 0x7f2acc060f60 con 0x7f2ad00b5250 2024-06-29T14:22:57.623 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:57.623 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"btime":"2024-06-29T14:22:47:707315+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4295,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/96042442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":96042442},{"type":"v1","addr":"172.21.15.167:6837","nonce":96042442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3681460169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3681460169},{"type":"v1","addr":"172.21.15.81:6834","nonce":3681460169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3175849002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3175849002},{"type":"v1","addr":"172.21.15.81:6835","nonce":3175849002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:22:44.057632+0000","modified":"2024-06-29T14:22:47.079866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4360":{"gid":4360,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/3579540962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3579540962},{"type":"v1","addr":"172.21.15.167:6836","nonce":3579540962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4360,"qdb_cluster":[4360]},"id":1}]} 2024-06-29T14:22:57.623 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2024-06-29T14:22:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.626+0000 7f2ad712b640 1 -- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2abc0762d0 msgr2=0x7f2abc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.626+0000 7f2ad712b640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2abc0762d0 0x7f2abc0787e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0033000 tx=0x7f2ac0004730 comp rx=0 tx=0).stop 2024-06-29T14:22:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.627+0000 7f2ad712b640 1 -- 172.21.15.81:0/589764571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ad00b5250 msgr2=0x7f2ad01f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.627+0000 7f2ad712b640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ad00b5250 0x7f2ad01f58a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00ba50 tx=0x7f2acc00bf10 comp rx=0 tx=0).stop 2024-06-29T14:22:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.627+0000 7f2ad712b640 1 -- 172.21.15.81:0/589764571 shutdown_connections 2024-06-29T14:22:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.627+0000 7f2ad712b640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2abc0762d0 0x7f2abc0787e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.627+0000 7f2ad712b640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ad00a0770 0x7f2ad00a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.627+0000 7f2ad712b640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ad00b9040 0x7f2ad00c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.627+0000 7f2ad712b640 1 --2- 172.21.15.81:0/589764571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ad00b5250 0x7f2ad01f58a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.627+0000 7f2ad712b640 1 -- 172.21.15.81:0/589764571 >> 172.21.15.81:0/589764571 conn(0x7f2ad009cf30 msgr2=0x7f2ad00b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.627+0000 7f2ad712b640 1 -- 172.21.15.81:0/589764571 shutdown_connections 2024-06-29T14:22:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.627+0000 7f2ad712b640 1 -- 172.21.15.81:0/589764571 wait complete. 2024-06-29T14:22:57.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:22:57.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.750+0000 7f675d038640 1 -- 172.21.15.81:0/1213609589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67580b3200 msgr2=0x7f67580cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.750+0000 7f675d038640 1 --2- 172.21.15.81:0/1213609589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67580b3200 0x7f67580cc7b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6748005fd0 tx=0x7f674802aa90 comp rx=0 tx=0).stop 2024-06-29T14:22:57.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.750+0000 7f675d038640 1 -- 172.21.15.81:0/1213609589 shutdown_connections 2024-06-29T14:22:57.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.750+0000 7f675d038640 1 --2- 172.21.15.81:0/1213609589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67580d0940 0x7f67580b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.750+0000 7f675d038640 1 --2- 172.21.15.81:0/1213609589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67580b3200 0x7f67580cc7b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.750+0000 7f675d038640 1 -- 172.21.15.81:0/1213609589 >> 172.21.15.81:0/1213609589 conn(0x7f67580b3680 msgr2=0x7f67580b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:57.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.750+0000 7f675d038640 1 -- 172.21.15.81:0/1213609589 shutdown_connections 2024-06-29T14:22:57.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.750+0000 7f675d038640 1 -- 172.21.15.81:0/1213609589 wait complete. 2024-06-29T14:22:57.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.752+0000 7f675d038640 1 Processor -- start 2024-06-29T14:22:57.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.752+0000 7f675d038640 1 -- start start 2024-06-29T14:22:57.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.753+0000 7f675d038640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67580b3200 0x7f67580c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.753+0000 7f675d038640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67580cccf0 0x7f67580c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.753+0000 7f675d038640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67580c7600 0x7f6758156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f675d038640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f67580b5c00 con 0x7f67580b3200 2024-06-29T14:22:57.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f675d038640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f67580b5a80 con 0x7f67580c7600 2024-06-29T14:22:57.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f675d038640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f67580b5d80 con 0x7f67580cccf0 2024-06-29T14:22:57.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f6756d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67580b3200 0x7f67580c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f6756d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67580b3200 0x7f67580c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36576/0 (socket says 172.21.15.81:36576) 2024-06-29T14:22:57.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f6756d9d640 1 -- 172.21.15.81:0/3933856515 learned_addr learned my addr 172.21.15.81:0/3933856515 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f675759e640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67580c7600 0x7f6758156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f675659c640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67580cccf0 0x7f67580c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f6756d9d640 1 -- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67580cccf0 msgr2=0x7f67580c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f6756d9d640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67580cccf0 0x7f67580c7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f6756d9d640 1 -- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67580c7600 msgr2=0x7f6758156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f6756d9d640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67580c7600 0x7f6758156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f6756d9d640 1 -- 172.21.15.81:0/3933856515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67580c6090 con 0x7f67580b3200 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f6756d9d640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67580b3200 0x7f67580c5840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6748006010 tx=0x7f6748003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f673f7fe640 1 -- 172.21.15.81:0/3933856515 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748044070 con 0x7f67580b3200 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f673f7fe640 1 -- 172.21.15.81:0/3933856515 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6748004250 con 0x7f67580b3200 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f673f7fe640 1 -- 172.21.15.81:0/3933856515 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748004660 con 0x7f67580b3200 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f675d038640 1 -- 172.21.15.81:0/3933856515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67580c6370 con 0x7f67580b3200 2024-06-29T14:22:57.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.754+0000 7f675d038640 1 -- 172.21.15.81:0/3933856515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67581f34b0 con 0x7f67580b3200 2024-06-29T14:22:57.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.755+0000 7f675d038640 1 -- 172.21.15.81:0/3933856515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67581f32a0 con 0x7f67580b3200 2024-06-29T14:22:57.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.756+0000 7f673f7fe640 1 -- 172.21.15.81:0/3933856515 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6748047070 con 0x7f67580b3200 2024-06-29T14:22:57.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.756+0000 7f673f7fe640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6744076300 0x7f6744078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:57.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.757+0000 7f673f7fe640 1 -- 172.21.15.81:0/3933856515 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f67480b8530 con 0x7f67580b3200 2024-06-29T14:22:57.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.757+0000 7f675659c640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6744076300 0x7f6744078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:57.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.757+0000 7f675659c640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6744076300 0x7f6744078810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f67581569c0 tx=0x7f674c009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:57.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.761+0000 7f673f7fe640 1 -- 172.21.15.81:0/3933856515 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6748032030 con 0x7f67580b3200 2024-06-29T14:22:57.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.943+0000 7f675d038640 1 -- 172.21.15.81:0/3933856515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67580cc7b0 con 0x7f67580b3200 2024-06-29T14:22:57.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.944+0000 7f673f7fe640 1 -- 172.21.15.81:0/3933856515 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f67480817b0 con 0x7f67580b3200 2024-06-29T14:22:57.942 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:57.942 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:22:57.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 -- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6744076300 msgr2=0x7f6744078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6744076300 0x7f6744078810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f67581569c0 tx=0x7f674c009040 comp rx=0 tx=0).stop 2024-06-29T14:22:57.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 -- 172.21.15.81:0/3933856515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67580b3200 msgr2=0x7f67580c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:57.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67580b3200 0x7f67580c5840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6748006010 tx=0x7f6748003ce0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 -- 172.21.15.81:0/3933856515 shutdown_connections 2024-06-29T14:22:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6744076300 0x7f6744078810 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67580c7600 0x7f6758156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67580cccf0 0x7f67580c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 --2- 172.21.15.81:0/3933856515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67580b3200 0x7f67580c5840 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 -- 172.21.15.81:0/3933856515 >> 172.21.15.81:0/3933856515 conn(0x7f67580b3680 msgr2=0x7f67580cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 -- 172.21.15.81:0/3933856515 shutdown_connections 2024-06-29T14:22:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:57.946+0000 7f675d038640 1 -- 172.21.15.81:0/3933856515 wait complete. 2024-06-29T14:22:57.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:22:58.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.071+0000 7fc5b1b70640 1 -- 172.21.15.81:0/13910493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc598004d40 msgr2=0x7fc598005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.071+0000 7fc5b1b70640 1 --2- 172.21.15.81:0/13910493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc598004d40 0x7fc598005150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009ed0 tx=0x7fc598005830 comp rx=0 tx=0).stop 2024-06-29T14:22:58.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.071+0000 7fc5abfff640 1 -- 172.21.15.81:0/13910493 reap_dead start 2024-06-29T14:22:58.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.071+0000 7fc5b1b70640 1 -- 172.21.15.81:0/13910493 shutdown_connections 2024-06-29T14:22:58.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.071+0000 7fc5b1b70640 1 -- 172.21.15.81:0/13910493 >> 172.21.15.81:0/13910493 conn(0x7fc5ac0b3510 msgr2=0x7fc5ac0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:58.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.071+0000 7fc5b1b70640 1 -- 172.21.15.81:0/13910493 shutdown_connections 2024-06-29T14:22:58.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.071+0000 7fc5b1b70640 1 -- 172.21.15.81:0/13910493 wait complete. 2024-06-29T14:22:58.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.073+0000 7fc5b1b70640 1 Processor -- start 2024-06-29T14:22:58.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.073+0000 7fc5b1b70640 1 -- start start 2024-06-29T14:22:58.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.074+0000 7fc5b1b70640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc598004d40 0x7fc5ac1dfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.074+0000 7fc5b1b70640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5ac09e2c0 0x7fc5ac0cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.074+0000 7fc5b1b70640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac09e6d0 0x7fc5ac0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.074+0000 7fc5b1b70640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac0b52b0 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.074+0000 7fc5b1b70640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac0b5130 con 0x7fc598004d40 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.074+0000 7fc5b1b70640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ac0d02d0 con 0x7fc5ac09e2c0 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.074+0000 7fc5abfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac09e6d0 0x7fc5ac0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.074+0000 7fc5abfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac09e6d0 0x7fc5ac0cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36592/0 (socket says 172.21.15.81:36592) 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.074+0000 7fc5abfff640 1 -- 172.21.15.81:0/72898725 learned_addr learned my addr 172.21.15.81:0/72898725 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5a3fff640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5ac09e2c0 0x7fc5ac0cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5ab7fe640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc598004d40 0x7fc5ac1dfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5abfff640 1 -- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5ac09e2c0 msgr2=0x7fc5ac0cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5abfff640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5ac09e2c0 0x7fc5ac0cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5abfff640 1 -- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc598004d40 msgr2=0x7fc5ac1dfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5abfff640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc598004d40 0x7fc5ac1dfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5abfff640 1 -- 172.21.15.81:0/72898725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ac1f2a10 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5abfff640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac09e6d0 0x7fc5ac0cca20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4002230 tx=0x7fc5a4034e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:58.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5a8ff9640 1 -- 172.21.15.81:0/72898725 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4043070 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5a8ff9640 1 -- 172.21.15.81:0/72898725 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5a4003c60 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5b1b70640 1 -- 172.21.15.81:0/72898725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ac1f3830 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5a8ff9640 1 -- 172.21.15.81:0/72898725 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4038040 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.075+0000 7fc5b1b70640 1 -- 172.21.15.81:0/72898725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ac09f180 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.076+0000 7fc5b1b70640 1 -- 172.21.15.81:0/72898725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ac0d0860 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.077+0000 7fc5a8ff9640 1 -- 172.21.15.81:0/72898725 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc5a4046070 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.077+0000 7fc5a8ff9640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc58c0762d0 0x7fc58c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.077+0000 7fc5a8ff9640 1 -- 172.21.15.81:0/72898725 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc5a40b7da0 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.077+0000 7fc5ab7fe640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc58c0762d0 0x7fc58c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.078+0000 7fc5ab7fe640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc58c0762d0 0x7fc58c0787e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc59c009b90 tx=0x7fc59c005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:58.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.082+0000 7fc5a8ff9640 1 -- 172.21.15.81:0/72898725 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc5ac1f2a10 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.299+0000 7fc5b1b70640 1 -- 172.21.15.81:0/72898725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5ac09ec30 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.299+0000 7fc5a8ff9640 1 -- 172.21.15.81:0/72898725 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5217 (secure 0 0 0) 0x7fc5a4081020 con 0x7fc5ac09e6d0 2024-06-29T14:22:58.298 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:22:58.298 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"btime":"2024-06-29T14:22:47:707315+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4295,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/96042442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":96042442},{"type":"v1","addr":"172.21.15.167:6837","nonce":96042442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3681460169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3681460169},{"type":"v1","addr":"172.21.15.81:6834","nonce":3681460169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3175849002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3175849002},{"type":"v1","addr":"172.21.15.81:6835","nonce":3175849002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:22:44.057632+0000","modified":"2024-06-29T14:22:47.079866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4360":{"gid":4360,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/3579540962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3579540962},{"type":"v1","addr":"172.21.15.167:6836","nonce":3579540962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4360,"qdb_cluster":[4360]},"id":1}]} 2024-06-29T14:22:58.298 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2024-06-29T14:22:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 -- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc58c0762d0 msgr2=0x7fc58c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc58c0762d0 0x7fc58c0787e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc59c009b90 tx=0x7fc59c005f90 comp rx=0 tx=0).stop 2024-06-29T14:22:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 -- 172.21.15.81:0/72898725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac09e6d0 msgr2=0x7fc5ac0cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac09e6d0 0x7fc5ac0cca20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4002230 tx=0x7fc5a4034e10 comp rx=0 tx=0).stop 2024-06-29T14:22:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 -- 172.21.15.81:0/72898725 shutdown_connections 2024-06-29T14:22:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc58c0762d0 0x7fc58c0787e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac09e6d0 0x7fc5ac0cca20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5ac09e2c0 0x7fc5ac0cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 --2- 172.21.15.81:0/72898725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc598004d40 0x7fc5ac1dfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 -- 172.21.15.81:0/72898725 >> 172.21.15.81:0/72898725 conn(0x7fc5ac0b3510 msgr2=0x7fc5ac0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 -- 172.21.15.81:0/72898725 shutdown_connections 2024-06-29T14:22:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.301+0000 7fc5b1b70640 1 -- 172.21.15.81:0/72898725 wait complete. 2024-06-29T14:22:58.713 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:22:44.057632+0000', 'modified': '2024-06-29T14:22:47.079866+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4360}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4360': {'gid': 4360, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.167:6836/3579540962', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6834', 'nonce': 3579540962}, {'type': 'v1', 'addr': '172.21.15.167:6836', 'nonce': 3579540962}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4360, 'qdb_cluster': [4360]} 2024-06-29T14:22:58.713 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:22:58.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-06-29T14:22:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.833+0000 7fb08d864640 1 -- 172.21.15.81:0/3899288055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 msgr2=0x7fb0880d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.833+0000 7fb08d864640 1 --2- 172.21.15.81:0/3899288055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb0880d0230 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb070005fd0 tx=0x7fb07002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.833+0000 7fb08d864640 1 -- 172.21.15.81:0/3899288055 shutdown_connections 2024-06-29T14:22:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.833+0000 7fb08d864640 1 --2- 172.21.15.81:0/3899288055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb0880d0230 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.833+0000 7fb08d864640 1 --2- 172.21.15.81:0/3899288055 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880b3200 0x7fb0880cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.833+0000 7fb08d864640 1 -- 172.21.15.81:0/3899288055 >> 172.21.15.81:0/3899288055 conn(0x7fb0880b3680 msgr2=0x7fb0880b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.833+0000 7fb08d864640 1 -- 172.21.15.81:0/3899288055 shutdown_connections 2024-06-29T14:22:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.833+0000 7fb08d864640 1 -- 172.21.15.81:0/3899288055 wait complete. 2024-06-29T14:22:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.835+0000 7fb08d864640 1 Processor -- start 2024-06-29T14:22:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.835+0000 7fb08d864640 1 -- start start 2024-06-29T14:22:58.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.836+0000 7fb08d864640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880b3200 0x7fb088155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.836+0000 7fb08d864640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb08809c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.836+0000 7fb08d864640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880d0770 0x7fb0880afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.836+0000 7fb08d864640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0880cedb0 con 0x7fb0880cccf0 2024-06-29T14:22:58.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.836+0000 7fb08d864640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb0880cec30 con 0x7fb0880b3200 2024-06-29T14:22:58.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.836+0000 7fb08d864640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb0880cef30 con 0x7fb0880d0770 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.836+0000 7fb07ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb08809c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb07ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb08809c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36614/0 (socket says 172.21.15.81:36614) 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb07ffff640 1 -- 172.21.15.81:0/2233027136 learned_addr learned my addr 172.21.15.81:0/2233027136 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb086ffd640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880b3200 0x7fb088155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb0877fe640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880d0770 0x7fb0880afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb07ffff640 1 -- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880d0770 msgr2=0x7fb0880afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb07ffff640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880d0770 0x7fb0880afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb07ffff640 1 -- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880b3200 msgr2=0x7fb088155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb07ffff640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880b3200 0x7fb088155a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb07ffff640 1 -- 172.21.15.81:0/2233027136 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb088156610 con 0x7fb0880cccf0 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb07ffff640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb08809c520 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb070000c00 tx=0x7fb0700038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:58.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb084ff9640 1 -- 172.21.15.81:0/2233027136 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07003e030 con 0x7fb0880cccf0 2024-06-29T14:22:58.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb084ff9640 1 -- 172.21.15.81:0/2233027136 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb070003ca0 con 0x7fb0880cccf0 2024-06-29T14:22:58.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb08d864640 1 -- 172.21.15.81:0/2233027136 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb08809e1a0 con 0x7fb0880cccf0 2024-06-29T14:22:58.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb084ff9640 1 -- 172.21.15.81:0/2233027136 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070035040 con 0x7fb0880cccf0 2024-06-29T14:22:58.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.837+0000 7fb08d864640 1 -- 172.21.15.81:0/2233027136 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb08809d3e0 con 0x7fb0880cccf0 2024-06-29T14:22:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.839+0000 7fb084ff9640 1 -- 172.21.15.81:0/2233027136 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb070041030 con 0x7fb0880cccf0 2024-06-29T14:22:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.839+0000 7fb084ff9640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb0680762d0 0x7fb0680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.839+0000 7fb084ff9640 1 -- 172.21.15.81:0/2233027136 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb0700b7ba0 con 0x7fb0880cccf0 2024-06-29T14:22:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.840+0000 7fb086ffd640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb0680762d0 0x7fb0680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.840+0000 7fb086ffd640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb0680762d0 0x7fb0680787e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb078005fd0 tx=0x7fb07800b2e0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.840+0000 7fb094804740 1 Processor -- start 2024-06-29T14:22:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.841+0000 7fb094804740 1 -- start start 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb094804740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad9b270 0x55b3db1876e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb094804740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad723b0 0x55b3db16b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb094804740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db16b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb094804740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b3db103bc0 con 0x55b3dad723b0 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb094804740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55b3db0b2630 con 0x55b3dad9b270 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb094804740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55b3db0a1bd0 con 0x55b3dad7a590 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb07ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad723b0 0x55b3db16b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb07ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad723b0 0x55b3db16b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36616/0 (socket says 172.21.15.81:36616) 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb07ffff640 1 -- 172.21.15.81:0/1539776195 learned_addr learned my addr 172.21.15.81:0/1539776195 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb086ffd640 1 --2- 172.21.15.81:0/1539776195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad9b270 0x55b3db1876e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb0877fe640 1 --2- 172.21.15.81:0/1539776195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db16b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb07ffff640 1 -- 172.21.15.81:0/1539776195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 msgr2=0x55b3db16b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb07ffff640 1 --2- 172.21.15.81:0/1539776195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db16b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb07ffff640 1 -- 172.21.15.81:0/1539776195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad9b270 msgr2=0x55b3db1876e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb07ffff640 1 --2- 172.21.15.81:0/1539776195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad9b270 0x55b3db1876e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb07ffff640 1 -- 172.21.15.81:0/1539776195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3db09eca0 con 0x55b3dad723b0 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb086ffd640 1 --2- 172.21.15.81:0/1539776195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad9b270 0x55b3db1876e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.842+0000 7fb07ffff640 1 --2- 172.21.15.81:0/1539776195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad723b0 0x55b3db16b3a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb0700b6070 tx=0x7fb070080d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a19d44a03448bf server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.843+0000 7fb07d7fa640 1 -- 172.21.15.81:0/1539776195 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070048040 con 0x55b3dad723b0 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.843+0000 7fb07d7fa640 1 -- 172.21.15.81:0/1539776195 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0700809c0 con 0x55b3dad723b0 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.843+0000 7fb07d7fa640 1 -- 172.21.15.81:0/1539776195 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070080eb0 con 0x55b3dad723b0 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.843+0000 7fb094804740 1 -- 172.21.15.81:0/1539776195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad723b0 msgr2=0x55b3db16b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.843+0000 7fb094804740 1 --2- 172.21.15.81:0/1539776195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad723b0 0x55b3db16b3a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb0700b6070 tx=0x7fb070080d40 comp rx=0 tx=0).stop 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.843+0000 7fb094804740 1 -- 172.21.15.81:0/1539776195 shutdown_connections 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.843+0000 7fb094804740 1 --2- 172.21.15.81:0/1539776195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db16b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.843+0000 7fb094804740 1 --2- 172.21.15.81:0/1539776195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad723b0 0x55b3db16b3a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.843+0000 7fb094804740 1 --2- 172.21.15.81:0/1539776195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad9b270 0x55b3db1876e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.843+0000 7fb094804740 1 -- 172.21.15.81:0/1539776195 >> 172.21.15.81:0/1539776195 conn(0x55b3dad8f3c0 msgr2=0x55b3db13ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:58.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.844+0000 7fb094804740 1 -- 172.21.15.81:0/1539776195 shutdown_connections 2024-06-29T14:22:58.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.844+0000 7fb094804740 1 -- 172.21.15.81:0/1539776195 wait complete. 2024-06-29T14:22:58.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.844+0000 7fb094804740 1 -- start start 2024-06-29T14:22:58.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.844+0000 7fb094804740 1 Processor -- start 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb094804740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db195e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb094804740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb094804740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad9b270 0x55b3db162f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb094804740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b3db03fae0 con 0x55b3dad9b270 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb094804740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55b3db0a5c10 con 0x55b3dad723b0 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb094804740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55b3dae15620 con 0x55b3dad7a590 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb094804740 10 client.0 _ll_register_callbacks cb 0x55b3db04c480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0877fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad9b270 0x55b3db162f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0877fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad9b270 0x55b3db162f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36620/0 (socket says 172.21.15.81:36620) 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0877fe640 1 -- 172.21.15.81:0/2524607531 learned_addr learned my addr 172.21.15.81:0/2524607531 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb086ffd640 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb07ffff640 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0877fe640 1 -- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 msgr2=0x55b3db183f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0877fe640 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0877fe640 1 -- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 msgr2=0x55b3db195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0877fe640 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0877fe640 1 -- 172.21.15.81:0/2524607531 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3dae06850 con 0x55b3dad9b270 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0877fe640 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad9b270 0x55b3db162f60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb0800125c0 tx=0x7fb0800129c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48a931b195923c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0667fc640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0667fc640 1 -- 172.21.15.81:0/2524607531 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb080019020 con 0x55b3dad9b270 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb0667fc640 1 -- 172.21.15.81:0/2524607531 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb080003b80 con 0x55b3dad9b270 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.846+0000 7fb094804740 1 -- 172.21.15.81:0/2524607531 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0700809c0 con 0x55b3dad9b270 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb0667fc640 1 -- 172.21.15.81:0/2524607531 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fb080023050 con 0x55b3dad9b270 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb094804740 10 client.4737 fetch_fsmap learned FSMap version 8 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb094804740 1 -- 172.21.15.81:0/2524607531 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55b3dae7d6b0 con 0x55b3dad9b270 2024-06-29T14:22:58.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb0667fc640 1 -- 172.21.15.81:0/2524607531 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0700809c0 con 0x55b3dad9b270 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb0667fc640 1 -- 172.21.15.81:0/2524607531 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7fb080026020 con 0x55b3dad9b270 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb094804740 10 client.4737 fetch_fsmap finished waiting for FSMap version 8 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb094804740 10 client.4737 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb094804740 20 client.4737 populate_metadata read hostname 'smithi081' 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb094804740 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:6834/3579540962,v1:172.21.15.167:6836/3579540962] conn(0x55b3dad59e40 0x55b3db198a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb094804740 4 client.4737 mds_command: new command op to 4360 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb094804740 1 -- 172.21.15.81:0/2524607531 --> [v2:172.21.15.167:6834/3579540962,v1:172.21.15.167:6836/3579540962] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3db0b4580 con 0x55b3dad59e40 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.847+0000 7fb086ffd640 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:6834/3579540962,v1:172.21.15.167:6836/3579540962] conn(0x55b3dad59e40 0x55b3db198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.848+0000 7fb0667fc640 10 client.4737 ms_handle_connect on v2:172.21.15.167:6834/3579540962 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.848+0000 7fb086ffd640 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:6834/3579540962,v1:172.21.15.167:6836/3579540962] conn(0x55b3dad59e40 0x55b3db198a00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fffbc7b3d602116 server_cookie=19411bde9e158705 in_seq=0 out_seq=0 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.848+0000 7fb0667fc640 1 -- 172.21.15.81:0/2524607531 <== mds.0 v2:172.21.15.167:6834/3579540962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b3db0b4580 con 0x55b3dad59e40 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.848+0000 7fb0667fc640 10 client.4737 handle_command_reply: tid=0 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.848+0000 7fb094804740 1 client.4737 shutdown 2024-06-29T14:22:58.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.848+0000 7fb094804740 1 -- 172.21.15.81:0/2524607531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad9b270 msgr2=0x55b3db162f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.848+0000 7fb094804740 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad9b270 0x55b3db162f60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb0800125c0 tx=0x7fb0800129c0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.848+0000 7fb094804740 1 -- 172.21.15.81:0/2524607531 shutdown_connections 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.848+0000 7fb094804740 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:6834/3579540962,v1:172.21.15.167:6836/3579540962] conn(0x55b3dad59e40 0x55b3db198a00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.848+0000 7fb094804740 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad9b270 0x55b3db162f60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.849+0000 7fb0667fc640 0 client.4737 ms_handle_reset on v2:172.21.15.167:6834/3579540962 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.849+0000 7fb094804740 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.849+0000 7fb094804740 1 --2- 172.21.15.81:0/2524607531 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.849+0000 7fb094804740 1 -- 172.21.15.81:0/2524607531 >> 172.21.15.81:0/2524607531 conn(0x55b3dad8f3c0 msgr2=0x55b3db13ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.849+0000 7fb094804740 1 -- 172.21.15.81:0/2524607531 shutdown_connections 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.849+0000 7fb094804740 1 -- 172.21.15.81:0/2524607531 wait complete. 2024-06-29T14:22:58.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.849+0000 7fb094804740 20 client.4737 trim_cache size 0 max 16384 2024-06-29T14:22:58.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.867+0000 7fb094804740 1 Processor -- start 2024-06-29T14:22:58.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.868+0000 7fb094804740 1 -- start start 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb094804740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db162ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb094804740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db1839d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb094804740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb094804740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0881fe790 con 0x55b3dad59e40 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb094804740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb0881e66d0 con 0x55b3dad723b0 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb094804740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb0881e6d20 con 0x55b3dad7a590 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb086ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db162ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb086ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db162ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36626/0 (socket says 172.21.15.81:36626) 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb086ffd640 1 -- 172.21.15.81:0/3362108548 learned_addr learned my addr 172.21.15.81:0/3362108548 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb0877fe640 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb07ffff640 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db1839d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb086ffd640 1 -- 172.21.15.81:0/3362108548 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 msgr2=0x55b3db183f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb086ffd640 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb086ffd640 1 -- 172.21.15.81:0/3362108548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 msgr2=0x55b3db1839d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb086ffd640 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db1839d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb086ffd640 1 -- 172.21.15.81:0/3362108548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3db059620 con 0x55b3dad59e40 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb0877fe640 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb07ffff640 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db1839d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.869+0000 7fb086ffd640 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db162ab0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb078011670 tx=0x7fb078013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0df14ebe833e4e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb07cff9640 1 -- 172.21.15.81:0/3362108548 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07801e7e0 con 0x55b3dad59e40 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb07cff9640 1 -- 172.21.15.81:0/3362108548 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb07801e980 con 0x55b3dad59e40 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb07cff9640 1 -- 172.21.15.81:0/3362108548 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07801ed30 con 0x55b3dad59e40 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb094804740 1 -- 172.21.15.81:0/3362108548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 msgr2=0x55b3db162ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb094804740 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db162ab0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb078011670 tx=0x7fb078013000 comp rx=0 tx=0).stop 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb094804740 1 -- 172.21.15.81:0/3362108548 shutdown_connections 2024-06-29T14:22:58.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb094804740 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb094804740 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db1839d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb094804740 1 --2- 172.21.15.81:0/3362108548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db162ab0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb094804740 1 -- 172.21.15.81:0/3362108548 >> 172.21.15.81:0/3362108548 conn(0x55b3dad8f3c0 msgr2=0x55b3db194ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:58.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb094804740 1 -- 172.21.15.81:0/3362108548 shutdown_connections 2024-06-29T14:22:58.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.870+0000 7fb094804740 1 -- 172.21.15.81:0/3362108548 wait complete. 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.871+0000 7fb094804740 1 -- start start 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.871+0000 7fb094804740 1 Processor -- start 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb094804740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db202600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb094804740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db183680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb094804740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb094804740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b3db169700 con 0x55b3dad59e40 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb094804740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55b3db1634a0 con 0x55b3dad723b0 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb094804740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55b3db16b030 con 0x55b3dad7a590 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb094804740 10 client.0 _ll_register_callbacks cb 0x7fb080001950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb086ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db202600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb086ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db202600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36634/0 (socket says 172.21.15.81:36634) 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb086ffd640 1 -- 172.21.15.81:0/809554277 learned_addr learned my addr 172.21.15.81:0/809554277 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb0877fe640 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb07ffff640 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db183680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb086ffd640 1 -- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 msgr2=0x55b3db183bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb086ffd640 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb086ffd640 1 -- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 msgr2=0x55b3db183680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb086ffd640 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db183680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb086ffd640 1 -- 172.21.15.81:0/809554277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3db182cc0 con 0x55b3dad59e40 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb086ffd640 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db202600 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb078011fd0 tx=0x7fb07800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94598180af4f9c2 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb0667fc640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb0667fc640 1 -- 172.21.15.81:0/809554277 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07801ed30 con 0x55b3dad59e40 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb0667fc640 1 -- 172.21.15.81:0/809554277 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb07803eaf0 con 0x55b3dad59e40 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb094804740 1 -- 172.21.15.81:0/809554277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb07801e980 con 0x55b3dad59e40 2024-06-29T14:22:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb0667fc640 1 -- 172.21.15.81:0/809554277 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fb078045050 con 0x55b3dad59e40 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb094804740 10 client.4743 fetch_fsmap learned FSMap version 8 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.873+0000 7fb094804740 1 -- 172.21.15.81:0/809554277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55b3db181a40 con 0x55b3dad59e40 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb0667fc640 1 -- 172.21.15.81:0/809554277 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb07801e980 con 0x55b3dad59e40 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb0667fc640 1 -- 172.21.15.81:0/809554277 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7fb078048020 con 0x55b3dad59e40 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb094804740 10 client.4743 fetch_fsmap finished waiting for FSMap version 8 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb094804740 10 client.4743 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb094804740 20 client.4743 populate_metadata read hostname 'smithi081' 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb094804740 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:6834/3579540962,v1:172.21.15.167:6836/3579540962] conn(0x55b3dad2f280 0x55b3db166910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb094804740 4 client.4743 mds_command: new command op to 4360 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb094804740 1 -- 172.21.15.81:0/809554277 --> [v2:172.21.15.167:6834/3579540962,v1:172.21.15.167:6836/3579540962] -- command(tid 0: {"prefix": "status"}) -- 0x55b3db0b0020 con 0x55b3dad2f280 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb07ffff640 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:6834/3579540962,v1:172.21.15.167:6836/3579540962] conn(0x55b3dad2f280 0x55b3db166910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb0667fc640 10 client.4743 ms_handle_connect on v2:172.21.15.167:6834/3579540962 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb07ffff640 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:6834/3579540962,v1:172.21.15.167:6836/3579540962] conn(0x55b3dad2f280 0x55b3db166910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6f70327ea07be24 server_cookie=76c45c87d01e1962 in_seq=0 out_seq=0 2024-06-29T14:22:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb0667fc640 1 -- 172.21.15.81:0/809554277 <== mds.0 v2:172.21.15.167:6834/3579540962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55b3db0b0020 con 0x55b3dad2f280 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb0667fc640 10 client.4743 handle_command_reply: tid=0 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.874+0000 7fb094804740 1 client.4743 shutdown 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 1 -- 172.21.15.81:0/809554277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 msgr2=0x55b3db202600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db202600 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb078011fd0 tx=0x7fb07800b900 comp rx=0 tx=0).stop 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 1 -- 172.21.15.81:0/809554277 shutdown_connections 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:6834/3579540962,v1:172.21.15.167:6836/3579540962] conn(0x55b3dad2f280 0x55b3db166910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55b3dad7a590 0x55b3db183bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55b3dad723b0 0x55b3db183680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb0667fc640 0 client.4743 ms_handle_reset on v2:172.21.15.167:6834/3579540962 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 1 --2- 172.21.15.81:0/809554277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3dad59e40 0x55b3db202600 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 1 -- 172.21.15.81:0/809554277 >> 172.21.15.81:0/809554277 conn(0x55b3dad8f3c0 msgr2=0x55b3db1878f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 1 -- 172.21.15.81:0/809554277 shutdown_connections 2024-06-29T14:22:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 1 -- 172.21.15.81:0/809554277 wait complete. 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb094804740 20 client.4743 trim_cache size 0 max 16384 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 4360, 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 14.190103647000001, 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 8, 2024-06-29T14:22:58.879 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 20, 2024-06-29T14:22:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 20, 2024-06-29T14:22:58.880 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 21.519035488 2024-06-29T14:22:58.880 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:22:58.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 -- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb0680762d0 msgr2=0x7fb0680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb0680762d0 0x7fb0680787e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb078005fd0 tx=0x7fb07800b2e0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 -- 172.21.15.81:0/2233027136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 msgr2=0x7fb08809c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:58.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb08809c520 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb070000c00 tx=0x7fb0700038a0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 -- 172.21.15.81:0/2233027136 shutdown_connections 2024-06-29T14:22:58.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb0680762d0 0x7fb0680787e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880d0770 0x7fb0880afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb08809c520 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 --2- 172.21.15.81:0/2233027136 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880b3200 0x7fb088155a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:58.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 -- 172.21.15.81:0/2233027136 >> 172.21.15.81:0/2233027136 conn(0x7fb0880b3680 msgr2=0x7fb08815be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:58.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 -- 172.21.15.81:0/2233027136 shutdown_connections 2024-06-29T14:22:58.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:58.875+0000 7fb08d864640 1 -- 172.21.15.81:0/2233027136 wait complete. 2024-06-29T14:22:58.885 INFO:teuthology.run_tasks:Running task sequential... 2024-06-29T14:22:59.020 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-06-29T14:22:59.021 INFO:teuthology.task.sequential:In sequential, running task print... 2024-06-29T14:22:59.021 INFO:teuthology.task.print:Enabling mgr modules 2024-06-29T14:22:59.021 INFO:teuthology.run_tasks:Running task kclient... 2024-06-29T14:22:59.051 INFO:tasks.kclient:Mounting kernel clients... 2024-06-29T14:22:59.051 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-06-29T14:22:59.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:59.052 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:22:59.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:59.068 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link delete ceph-brx 2024-06-29T14:22:59.135 INFO:teuthology.orchestra.run.smithi081.stderr:Cannot find device "ceph-brx" 2024-06-29T14:22:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:22:59.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:59.137 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:22:59.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:59.194 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link delete ceph-brx 2024-06-29T14:22:59.260 INFO:teuthology.orchestra.run.smithi081.stderr:Cannot find device "ceph-brx" 2024-06-29T14:22:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:22:59.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:59.262 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list 2024-06-29T14:22:59.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:59.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link delete ceph-brx 2024-06-29T14:22:59.354 INFO:teuthology.orchestra.run.smithi167.stderr:Cannot find device "ceph-brx" 2024-06-29T14:22:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:22:59.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:59.356 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list 2024-06-29T14:22:59.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:59.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link delete ceph-brx 2024-06-29T14:22:59.483 INFO:teuthology.orchestra.run.smithi167.stderr:Cannot find device "ceph-brx" 2024-06-29T14:22:59.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T14:22:59.485 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-06-29T14:22:59.485 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-29T14:22:59.485 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:22:59.486 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:22:59.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.601+0000 7fd81090b640 1 -- 172.21.15.81:0/438214706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80c0b6eb0 msgr2=0x7fd80c0d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:59.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.601+0000 7fd81090b640 1 --2- 172.21.15.81:0/438214706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80c0b6eb0 0x7fd80c0d02d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd80c072aa0 tx=0x7fd7fc02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.601+0000 7fd81090b640 1 -- 172.21.15.81:0/438214706 shutdown_connections 2024-06-29T14:22:59.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.601+0000 7fd81090b640 1 --2- 172.21.15.81:0/438214706 >> v2:172.21.15.167:3300/0 conn(0x7fd80c0ce920 0x7fd80c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.601+0000 7fd81090b640 1 --2- 172.21.15.81:0/438214706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80c0b6eb0 0x7fd80c0d02d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.601+0000 7fd81090b640 1 -- 172.21.15.81:0/438214706 >> 172.21.15.81:0/438214706 conn(0x7fd80c0b3510 msgr2=0x7fd80c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:59.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.601+0000 7fd81090b640 1 -- 172.21.15.81:0/438214706 shutdown_connections 2024-06-29T14:22:59.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.601+0000 7fd81090b640 1 -- 172.21.15.81:0/438214706 wait complete. 2024-06-29T14:22:59.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.603+0000 7fd81090b640 1 Processor -- start 2024-06-29T14:22:59.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.603+0000 7fd81090b640 1 -- start start 2024-06-29T14:22:59.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd81090b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80c0b6eb0 0x7fd80c1f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:59.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd81090b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd80c0ce510 0x7fd80c1db2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:59.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd81090b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd80c0ce920 0x7fd80c1e6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:59.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd81090b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd80c0b4f20 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd81090b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd80c0b29c0 con 0x7fd80c0ce920 2024-06-29T14:22:59.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd81090b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd80c0b50a0 con 0x7fd80c0ce510 2024-06-29T14:22:59.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd80a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80c0b6eb0 0x7fd80c1f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:59.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd80a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80c0b6eb0 0x7fd80c1f6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36650/0 (socket says 172.21.15.81:36650) 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd80a59c640 1 -- 172.21.15.81:0/2876994903 learned_addr learned my addr 172.21.15.81:0/2876994903 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd80ad9d640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd80c0ce920 0x7fd80c1e6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd801d9b640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd80c0ce510 0x7fd80c1db2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd80a59c640 1 -- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd80c0ce510 msgr2=0x7fd80c1db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd80a59c640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd80c0ce510 0x7fd80c1db2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd80a59c640 1 -- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd80c0ce920 msgr2=0x7fd80c1e6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd80a59c640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd80c0ce920 0x7fd80c1e6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd80a59c640 1 -- 172.21.15.81:0/2876994903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80c1f6b30 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd801d9b640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd80c0ce510 0x7fd80c1db2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd80a59c640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80c0b6eb0 0x7fd80c1f6230 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc02aa90 tx=0x7fd7fc033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.605+0000 7fd8037fe640 1 -- 172.21.15.81:0/2876994903 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc031d60 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.606+0000 7fd8037fe640 1 -- 172.21.15.81:0/2876994903 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7fc005020 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.606+0000 7fd8037fe640 1 -- 172.21.15.81:0/2876994903 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc005470 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.606+0000 7fd81090b640 1 -- 172.21.15.81:0/2876994903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd80c1f6e10 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.606+0000 7fd81090b640 1 -- 172.21.15.81:0/2876994903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd80c1df120 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.606+0000 7fd81090b640 1 -- 172.21.15.81:0/2876994903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd80c0d02d0 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.607+0000 7fd8037fe640 1 -- 172.21.15.81:0/2876994903 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd7fc0047a0 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.608+0000 7fd8037fe640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd7ec0762d0 0x7fd7ec0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:22:59.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.608+0000 7fd8037fe640 1 -- 172.21.15.81:0/2876994903 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd7fc0b8850 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.608+0000 7fd801d9b640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd7ec0762d0 0x7fd7ec0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:22:59.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.608+0000 7fd801d9b640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd7ec0762d0 0x7fd7ec0787e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd7f40046b0 tx=0x7fd7f4009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:22:59.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.612+0000 7fd8037fe640 1 -- 172.21.15.81:0/2876994903 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd7fc03f030 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.833+0000 7fd81090b640 1 -- 172.21.15.81:0/2876994903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd80c0cdf00 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.833+0000 7fd8037fe640 1 -- 172.21.15.81:0/2876994903 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7fd7fc08a1a0 con 0x7fd80c0b6eb0 2024-06-29T14:22:59.831 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:22:59.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 -- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd7ec0762d0 msgr2=0x7fd7ec0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:59.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd7ec0762d0 0x7fd7ec0787e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd7f40046b0 tx=0x7fd7f4009040 comp rx=0 tx=0).stop 2024-06-29T14:22:59.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 -- 172.21.15.81:0/2876994903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80c0b6eb0 msgr2=0x7fd80c1f6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:22:59.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80c0b6eb0 0x7fd80c1f6230 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc02aa90 tx=0x7fd7fc033c90 comp rx=0 tx=0).stop 2024-06-29T14:22:59.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 -- 172.21.15.81:0/2876994903 shutdown_connections 2024-06-29T14:22:59.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd7ec0762d0 0x7fd7ec0787e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd80c0ce920 0x7fd80c1e6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd80c0ce510 0x7fd80c1db2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 --2- 172.21.15.81:0/2876994903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80c0b6eb0 0x7fd80c1f6230 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:22:59.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 -- 172.21.15.81:0/2876994903 >> 172.21.15.81:0/2876994903 conn(0x7fd80c0b3510 msgr2=0x7fd80c0b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:22:59.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 -- 172.21.15.81:0/2876994903 shutdown_connections 2024-06-29T14:22:59.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59.835+0000 7fd81090b640 1 -- 172.21.15.81:0/2876994903 wait complete. 2024-06-29T14:22:59.844 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-29T14:22:59.845 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:22:59.845 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T14:22:59.845 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:22:59.845 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:22:59.845 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-29T14:22:59.845 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-29T14:22:59.845 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:22:59.845 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:22:59.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:22:59.845 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T14:22:59.861 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:22:59.861 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:22:59.861 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:22:59.861 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:59.861 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:22:59.861 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:59.861 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:22:59.862 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 42740sec preferred_lft 42740sec 2024-06-29T14:22:59.863 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T14:22:59.863 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:22:59.863 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-06-29T14:22:59.863 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:22:59.863 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:22:59.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add name ceph-brx type bridge 2024-06-29T14:22:59.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr flush dev ceph-brx 2024-06-29T14:22:59.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set ceph-brx up 2024-06-29T14:22:59.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-06-29T14:22:59.864 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:22:59.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:22:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:00.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:00.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:00.066 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:23:00.091 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T14:23:00.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:00.093 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T14:23:00.174 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T14:23:00.174 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:23:00.174 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:23:00.174 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:23:00.174 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:23:00.175 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:00.175 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:23:00.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:23:00.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:23:00.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:23:00.175 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:23:00.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:00.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:00.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:00.483 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:23:00.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:00.499 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list-id 2024-06-29T14:23:00.554 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:00.555 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:23:00.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:23:00.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-06-29T14:23:00.555 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:23:00.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:00.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:00.675 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-06-29T14:23:00.675 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:00.675 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:23:00.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-06-29T14:23:00.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-06-29T14:23:00.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-06-29T14:23:00.676 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-06-29T14:23:00.676 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-06-29T14:23:00.676 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:23:00.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:00.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:00.812 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:00.813 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:23:00.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T14:23:00.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set dev brx.0 master ceph-brx 2024-06-29T14:23:00.813 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:23:00.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:00.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:00.889 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:00.890 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:00.905 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:23:00.905 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:00.905 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:00.960 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T14:23:00.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:00.960 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:01.062 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:01.062 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:23:01.192 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:01.192 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:23:01.192 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:23:01.192 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:23:01.192 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:23:01.193 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:23:01.193 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:23:01.193 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:23:01.193 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:23:01.193 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:23:01.194 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:23:01.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:01.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:01.220 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:01.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:01.326 INFO:teuthology.orchestra.run.smithi081.stdout:client4535 2024-06-29T14:23:01.327 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:23:01.327 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:01.343 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:01.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:01.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:01.448 INFO:teuthology.orchestra.run.smithi081.stdout:client4535 2024-06-29T14:23:01.448 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:01.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:01.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:01.514 INFO:teuthology.orchestra.run.smithi081.stdout:client4535 2024-06-29T14:23:01.515 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.client4535/status') 2024-06-29T14:23:01.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:01.580 INFO:teuthology.orchestra.run.smithi081.stdout:instance: client.4535 (3)172.21.15.81:0/3476758425 2024-06-29T14:23:01.581 INFO:teuthology.orchestra.run.smithi081.stdout:blocklisted: false 2024-06-29T14:23:01.581 INFO:teuthology.orchestra.run.smithi081.stderr:0+1 records in 2024-06-29T14:23:01.581 INFO:teuthology.orchestra.run.smithi081.stderr:0+1 records out 2024-06-29T14:23:01.581 INFO:teuthology.orchestra.run.smithi081.stderr:70 bytes copied, 2.9478e-05 s, 2.4 MB/s 2024-06-29T14:23:01.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:01.588 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-06-29T14:23:01.588 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-29T14:23:01.588 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-06-29T14:23:01.588 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:23:01.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.703+0000 7f57e51e4640 1 -- 172.21.15.81:0/1360222726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c8003fd0 msgr2=0x7f57c8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:01.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.703+0000 7f57e51e4640 1 --2- 172.21.15.81:0/1360222726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c8003fd0 0x7f57c8004800 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f57d400c290 tx=0x7f57c8005830 comp rx=0 tx=0).stop 2024-06-29T14:23:01.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.703+0000 7f57df59e640 1 -- 172.21.15.81:0/1360222726 reap_dead start 2024-06-29T14:23:01.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.703+0000 7f57e51e4640 1 -- 172.21.15.81:0/1360222726 shutdown_connections 2024-06-29T14:23:01.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.703+0000 7f57e51e4640 1 -- 172.21.15.81:0/1360222726 >> 172.21.15.81:0/1360222726 conn(0x7f57e006ccc0 msgr2=0x7f57e00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:01.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.703+0000 7f57e51e4640 1 -- 172.21.15.81:0/1360222726 shutdown_connections 2024-06-29T14:23:01.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.703+0000 7f57e51e4640 1 -- 172.21.15.81:0/1360222726 wait complete. 2024-06-29T14:23:01.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.705+0000 7f57e51e4640 1 Processor -- start 2024-06-29T14:23:01.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.705+0000 7f57e51e4640 1 -- start start 2024-06-29T14:23:01.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.706+0000 7f57e51e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c8003fd0 0x7f57e00ab080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:01.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.706+0000 7f57e51e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57e015cfa0 0x7f57e015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:01.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.706+0000 7f57e51e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57e0071f20 0x7f57e0072330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:01.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.706+0000 7f57e51e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f57e015eea0 con 0x7f57c8003fd0 2024-06-29T14:23:01.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.706+0000 7f57e51e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f57e00cf300 con 0x7f57e0071f20 2024-06-29T14:23:01.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.706+0000 7f57e51e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f57e015f020 con 0x7f57e015cfa0 2024-06-29T14:23:01.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57ded9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c8003fd0 0x7f57e00ab080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:01.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57ded9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c8003fd0 0x7f57e00ab080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37888/0 (socket says 172.21.15.81:37888) 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57ded9d640 1 -- 172.21.15.81:0/1818916826 learned_addr learned my addr 172.21.15.81:0/1818916826 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57de59c640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57e015cfa0 0x7f57e015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57df59e640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57e0071f20 0x7f57e0072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57ded9d640 1 -- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57e015cfa0 msgr2=0x7f57e015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57ded9d640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57e015cfa0 0x7f57e015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57ded9d640 1 -- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57e0071f20 msgr2=0x7f57e0072330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57ded9d640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57e0071f20 0x7f57e0072330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57ded9d640 1 -- 172.21.15.81:0/1818916826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e013bc10 con 0x7f57c8003fd0 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57df59e640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57e0071f20 0x7f57e0072330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57ded9d640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c8003fd0 0x7f57e00ab080 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f57d0011a50 tx=0x7f57d0011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57c77fe640 1 -- 172.21.15.81:0/1818916826 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d001f070 con 0x7f57c8003fd0 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57c77fe640 1 -- 172.21.15.81:0/1818916826 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57d000fdc0 con 0x7f57c8003fd0 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57c77fe640 1 -- 172.21.15.81:0/1818916826 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d00126b0 con 0x7f57c8003fd0 2024-06-29T14:23:01.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57e51e4640 1 -- 172.21.15.81:0/1818916826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e0072870 con 0x7f57c8003fd0 2024-06-29T14:23:01.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.707+0000 7f57e51e4640 1 -- 172.21.15.81:0/1818916826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e013b090 con 0x7f57c8003fd0 2024-06-29T14:23:01.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.708+0000 7f57e51e4640 1 -- 172.21.15.81:0/1818916826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57e015e960 con 0x7f57c8003fd0 2024-06-29T14:23:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.709+0000 7f57c77fe640 1 -- 172.21.15.81:0/1818916826 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f57d0022070 con 0x7f57c8003fd0 2024-06-29T14:23:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.710+0000 7f57c77fe640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57cc076300 0x7f57cc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.710+0000 7f57c77fe640 1 -- 172.21.15.81:0/1818916826 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f57d00982d0 con 0x7f57c8003fd0 2024-06-29T14:23:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.710+0000 7f57de59c640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57cc076300 0x7f57cc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.711+0000 7f57de59c640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57cc076300 0x7f57cc078810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f57d4033000 tx=0x7f57d402d000 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:01.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.714+0000 7f57c77fe640 1 -- 172.21.15.81:0/1818916826 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f57e0072870 con 0x7f57c8003fd0 2024-06-29T14:23:01.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.933+0000 7f57e51e4640 1 -- 172.21.15.81:0/1818916826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f57e01f24d0 con 0x7f57c8003fd0 2024-06-29T14:23:01.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.933+0000 7f57c77fe640 1 -- 172.21.15.81:0/1818916826 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f57d00614d0 con 0x7f57c8003fd0 2024-06-29T14:23:01.932 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:23:01.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.935+0000 7f57e51e4640 1 -- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57cc076300 msgr2=0x7f57cc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:01.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.935+0000 7f57e51e4640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57cc076300 0x7f57cc078810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f57d4033000 tx=0x7f57d402d000 comp rx=0 tx=0).stop 2024-06-29T14:23:01.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.935+0000 7f57e51e4640 1 -- 172.21.15.81:0/1818916826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c8003fd0 msgr2=0x7f57e00ab080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:01.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.935+0000 7f57e51e4640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c8003fd0 0x7f57e00ab080 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f57d0011a50 tx=0x7f57d0011f10 comp rx=0 tx=0).stop 2024-06-29T14:23:01.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.936+0000 7f57e51e4640 1 -- 172.21.15.81:0/1818916826 shutdown_connections 2024-06-29T14:23:01.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.936+0000 7f57e51e4640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57cc076300 0x7f57cc078810 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:01.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.936+0000 7f57e51e4640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57e0071f20 0x7f57e0072330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:01.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.936+0000 7f57e51e4640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57e015cfa0 0x7f57e015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:01.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.936+0000 7f57e51e4640 1 --2- 172.21.15.81:0/1818916826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c8003fd0 0x7f57e00ab080 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:01.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.936+0000 7f57e51e4640 1 -- 172.21.15.81:0/1818916826 >> 172.21.15.81:0/1818916826 conn(0x7f57e006ccc0 msgr2=0x7f57e00afa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:01.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.936+0000 7f57e51e4640 1 -- 172.21.15.81:0/1818916826 shutdown_connections 2024-06-29T14:23:01.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:01.936+0000 7f57e51e4640 1 -- 172.21.15.81:0/1818916826 wait complete. 2024-06-29T14:23:01.945 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-29T14:23:01.945 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:23:01.945 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T14:23:01.945 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-06-29T14:23:01.945 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-06-29T14:23:01.945 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-29T14:23:01.945 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-29T14:23:01.945 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:23:01.945 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-06-29T14:23:01.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:01.945 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T14:23:01.961 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T14:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 42738sec preferred_lft 42738sec 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link tentative 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:01.963 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:23:01.964 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:23:01.964 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link tentative 2024-06-29T14:23:01.964 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:01.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:01.964 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:23:02.025 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T14:23:02.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:02.030 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T14:23:02.083 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T14:23:02.083 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:23:02.083 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:23:02.083 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:23:02.083 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:23:02.083 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:02.083 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:23:02.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:23:02.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:23:02.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:23:02.084 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:23:02.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:02.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:02.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:02.209 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:23:02.224 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:23:02.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:02.225 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list-id 2024-06-29T14:23:02.280 INFO:teuthology.orchestra.run.smithi081.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-06-29T14:23:02.280 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:02.280 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:23:02.281 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:23:02.281 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-06-29T14:23:02.281 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:23:02.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:02.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:02.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:02.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-06-29T14:23:02.421 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:23:02.421 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:23:02.421 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:23:02.421 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:02.421 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:23:02.421 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:02.422 INFO:teuthology.orchestra.run.smithi081.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:23:02.422 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 42:bb:ce:97:08:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-06-29T14:23:02.422 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-06-29T14:23:02.422 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:02.422 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::40bb:ceff:fe97:810/64 scope link tentative 2024-06-29T14:23:02.422 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:02.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:02.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-06-29T14:23:02.487 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 42:bb:ce:97:08:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::40bb:ceff:fe97:810/64 scope link tentative 2024-06-29T14:23:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:02.489 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-06-29T14:23:02.490 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:02.490 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:23:02.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-06-29T14:23:02.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-06-29T14:23:02.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-06-29T14:23:02.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-06-29T14:23:02.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-06-29T14:23:02.490 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:23:02.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:02.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:02.642 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:02.642 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:23:02.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.1 up 2024-06-29T14:23:02.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set dev brx.1 master ceph-brx 2024-06-29T14:23:02.642 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:23:02.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:02.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:02.719 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:02.719 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-06-29T14:23:02.733 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-06-29T14:23:02.734 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:02.734 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-06-29T14:23:02.789 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-06-29T14:23:02.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:02.789 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:02.883 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:02.883 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:23:02.891 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:02.892 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:23:02.892 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:23:02.892 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 2@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:23:02.892 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:23:02.892 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-06-29T14:23:02.892 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:23:02.892 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:23:02.892 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:23:02.893 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:23:02.894 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:23:02.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:02.895 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-06-29T14:23:02.919 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:03.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:03.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:03.015 INFO:teuthology.orchestra.run.smithi081.stdout:client4761 2024-06-29T14:23:03.016 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:23:03.016 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:03.031 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:03.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:03.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:03.128 INFO:teuthology.orchestra.run.smithi081.stdout:client4761 2024-06-29T14:23:03.128 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:03.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:03.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:03.184 INFO:teuthology.orchestra.run.smithi081.stdout:client4761 2024-06-29T14:23:03.184 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.client4761/status') 2024-06-29T14:23:03.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:03.243 INFO:teuthology.orchestra.run.smithi081.stdout:instance: client.4761 (3)192.168.0.2:0/2961600479 2024-06-29T14:23:03.243 INFO:teuthology.orchestra.run.smithi081.stdout:blocklisted: false 2024-06-29T14:23:03.243 INFO:teuthology.orchestra.run.smithi081.stderr:0+1 records in 2024-06-29T14:23:03.243 INFO:teuthology.orchestra.run.smithi081.stderr:0+1 records out 2024-06-29T14:23:03.243 INFO:teuthology.orchestra.run.smithi081.stderr:69 bytes copied, 2.3963e-05 s, 2.9 MB/s 2024-06-29T14:23:03.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:03.250 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-06-29T14:23:03.250 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-29T14:23:03.250 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-06-29T14:23:03.250 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:23:03.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.366+0000 7f4551b3c640 1 -- 172.21.15.81:0/4005762973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f454c0737c0 msgr2=0x7f454c15c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:03.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.366+0000 7f4551b3c640 1 --2- 172.21.15.81:0/4005762973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f454c0737c0 0x7f454c15c5f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f454c0b50c0 tx=0x7f454002a560 comp rx=0 tx=0).stop 2024-06-29T14:23:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.366+0000 7f4551b3c640 1 -- 172.21.15.81:0/4005762973 shutdown_connections 2024-06-29T14:23:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.366+0000 7f4551b3c640 1 --2- 172.21.15.81:0/4005762973 >> v2:172.21.15.167:3300/0 conn(0x7f454c15cb30 0x7f454c15f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.366+0000 7f4551b3c640 1 --2- 172.21.15.81:0/4005762973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f454c0737c0 0x7f454c15c5f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.366+0000 7f4551b3c640 1 --2- 172.21.15.81:0/4005762973 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f454c145530 0x7f454c073280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.366+0000 7f4551b3c640 1 -- 172.21.15.81:0/4005762973 >> 172.21.15.81:0/4005762973 conn(0x7f454c06ccc0 msgr2=0x7f454c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.366+0000 7f4551b3c640 1 -- 172.21.15.81:0/4005762973 shutdown_connections 2024-06-29T14:23:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.366+0000 7f4551b3c640 1 -- 172.21.15.81:0/4005762973 wait complete. 2024-06-29T14:23:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.367+0000 7f4551b3c640 1 Processor -- start 2024-06-29T14:23:03.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.368+0000 7f4551b3c640 1 -- start start 2024-06-29T14:23:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f4551b3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f454c0737c0 0x7f454c13f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f4551b3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f454c145530 0x7f454c0aa620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f4551b3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f454c15cb30 0x7f454c15ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f4551b3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f454c160000 con 0x7f454c0737c0 2024-06-29T14:23:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f4551b3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f454c15fe80 con 0x7f454c15cb30 2024-06-29T14:23:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f4551b3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f454c160180 con 0x7f454c145530 2024-06-29T14:23:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f454c0737c0 0x7f454c13f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f454c0737c0 0x7f454c13f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37914/0 (socket says 172.21.15.81:37914) 2024-06-29T14:23:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454b7fe640 1 -- 172.21.15.81:0/2780016295 learned_addr learned my addr 172.21.15.81:0/2780016295 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454affd640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f454c145530 0x7f454c0aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454bfff640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f454c15cb30 0x7f454c15ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454b7fe640 1 -- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f454c145530 msgr2=0x7f454c0aa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454b7fe640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f454c145530 0x7f454c0aa620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454b7fe640 1 -- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f454c15cb30 msgr2=0x7f454c15ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454b7fe640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f454c15cb30 0x7f454c15ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454b7fe640 1 -- 172.21.15.81:0/2780016295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f454c143a90 con 0x7f454c0737c0 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.369+0000 7f454affd640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f454c145530 0x7f454c0aa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.370+0000 7f454b7fe640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f454c0737c0 0x7f454c13f260 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f453c00d730 tx=0x7f453c00dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.370+0000 7f4533fff640 1 -- 172.21.15.81:0/2780016295 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c017070 con 0x7f454c0737c0 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.370+0000 7f4533fff640 1 -- 172.21.15.81:0/2780016295 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f453c009b40 con 0x7f454c0737c0 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.370+0000 7f4533fff640 1 -- 172.21.15.81:0/2780016295 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c005020 con 0x7f454c0737c0 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.370+0000 7f4551b3c640 1 -- 172.21.15.81:0/2780016295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f454c143d70 con 0x7f454c0737c0 2024-06-29T14:23:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.370+0000 7f4551b3c640 1 -- 172.21.15.81:0/2780016295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f454c144330 con 0x7f454c0737c0 2024-06-29T14:23:03.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.371+0000 7f4551b3c640 1 -- 172.21.15.81:0/2780016295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f454c1606a0 con 0x7f454c0737c0 2024-06-29T14:23:03.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.372+0000 7f4533fff640 1 -- 172.21.15.81:0/2780016295 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f453c0051c0 con 0x7f454c0737c0 2024-06-29T14:23:03.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.372+0000 7f4533fff640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4538076300 0x7f4538078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:03.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.372+0000 7f4533fff640 1 -- 172.21.15.81:0/2780016295 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f453c0977b0 con 0x7f454c0737c0 2024-06-29T14:23:03.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.372+0000 7f454affd640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4538076300 0x7f4538078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:03.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.373+0000 7f454affd640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4538076300 0x7f4538078810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4540033000 tx=0x7f4540004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:03.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.376+0000 7f4533fff640 1 -- 172.21.15.81:0/2780016295 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f453c061080 con 0x7f454c0737c0 2024-06-29T14:23:03.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.591+0000 7f4551b3c640 1 -- 172.21.15.81:0/2780016295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f454c0735a0 con 0x7f454c0737c0 2024-06-29T14:23:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.592+0000 7f4533fff640 1 -- 172.21.15.81:0/2780016295 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f453c069100 con 0x7f454c0737c0 2024-06-29T14:23:03.590 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:23:03.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 -- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4538076300 msgr2=0x7f4538078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:03.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4538076300 0x7f4538078810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4540033000 tx=0x7f4540004730 comp rx=0 tx=0).stop 2024-06-29T14:23:03.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 -- 172.21.15.81:0/2780016295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f454c0737c0 msgr2=0x7f454c13f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:03.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f454c0737c0 0x7f454c13f260 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f453c00d730 tx=0x7f453c00dbf0 comp rx=0 tx=0).stop 2024-06-29T14:23:03.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 -- 172.21.15.81:0/2780016295 shutdown_connections 2024-06-29T14:23:03.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4538076300 0x7f4538078810 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:03.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f454c15cb30 0x7f454c15ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:03.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f454c145530 0x7f454c0aa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:03.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 --2- 172.21.15.81:0/2780016295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f454c0737c0 0x7f454c13f260 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:03.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 -- 172.21.15.81:0/2780016295 >> 172.21.15.81:0/2780016295 conn(0x7f454c06ccc0 msgr2=0x7f454c0b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:03.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 -- 172.21.15.81:0/2780016295 shutdown_connections 2024-06-29T14:23:03.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:03.594+0000 7f4551b3c640 1 -- 172.21.15.81:0/2780016295 wait complete. 2024-06-29T14:23:03.603 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-29T14:23:03.603 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:23:03.603 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi167.front.sepia.ceph.com 2024-06-29T14:23:03.603 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-06-29T14:23:03.603 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-06-29T14:23:03.603 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-29T14:23:03.603 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-29T14:23:03.603 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:23:03.604 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-06-29T14:23:03.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:03.604 DEBUG:teuthology.orchestra.run.smithi167:> ip addr 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 ::1/128 scope host 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:d9:35:e6 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout: altname enp7s0f0 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:d9:35:e7 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:03.624 INFO:teuthology.orchestra.run.smithi167.stdout: altname enp7s0f1 2024-06-29T14:23:03.625 INFO:teuthology.orchestra.run.smithi167.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:03.625 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:88:73:08 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:03.625 INFO:teuthology.orchestra.run.smithi167.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:23:03.625 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:88:73:09 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:03.625 INFO:teuthology.orchestra.run.smithi167.stdout: inet 172.21.15.167/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:23:03.625 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft 42697sec preferred_lft 42697sec 2024-06-29T14:23:03.625 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 fe80::ec4:7aff:fe88:7309/64 scope link 2024-06-29T14:23:03.625 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:03.625 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-06-29T14:23:03.625 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:03.626 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2024-06-29T14:23:03.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link add name ceph-brx type bridge 2024-06-29T14:23:03.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip addr flush dev ceph-brx 2024-06-29T14:23:03.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link set ceph-brx up 2024-06-29T14:23:03.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-06-29T14:23:03.626 DEBUG:teuthology.orchestra.run.smithi167:> ') 2024-06-29T14:23:03.719 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:03.906 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:03.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:03.913 DEBUG:teuthology.orchestra.run.smithi167:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:23:03.937 INFO:teuthology.orchestra.run.smithi167.stdout:1 2024-06-29T14:23:03.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:03.939 DEBUG:teuthology.orchestra.run.smithi167:> route 2024-06-29T14:23:04.006 INFO:teuthology.orchestra.run.smithi167.stdout:Kernel IP routing table 2024-06-29T14:23:04.007 INFO:teuthology.orchestra.run.smithi167.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:23:04.007 INFO:teuthology.orchestra.run.smithi167.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:23:04.007 INFO:teuthology.orchestra.run.smithi167.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:23:04.007 INFO:teuthology.orchestra.run.smithi167.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:23:04.007 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:04.007 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2024-06-29T14:23:04.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:23:04.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:23:04.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:23:04.007 DEBUG:teuthology.orchestra.run.smithi167:> ') 2024-06-29T14:23:04.095 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:04.323 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:04.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:04.329 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list 2024-06-29T14:23:04.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:04.345 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list-id 2024-06-29T14:23:04.402 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:04.402 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2024-06-29T14:23:04.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-06-29T14:23:04.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-06-29T14:23:04.402 DEBUG:teuthology.orchestra.run.smithi167:> ') 2024-06-29T14:23:04.495 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:04.520 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:04.526 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-06-29T14:23:04.527 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:04.527 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2024-06-29T14:23:04.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-06-29T14:23:04.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-06-29T14:23:04.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-06-29T14:23:04.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-06-29T14:23:04.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-06-29T14:23:04.527 DEBUG:teuthology.orchestra.run.smithi167:> ') 2024-06-29T14:23:04.576 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:04.651 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:04.657 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:04.657 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2024-06-29T14:23:04.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link set brx.0 up 2024-06-29T14:23:04.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link set dev brx.0 master ceph-brx 2024-06-29T14:23:04.657 DEBUG:teuthology.orchestra.run.smithi167:> ') 2024-06-29T14:23:04.706 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:04.729 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:04.735 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:04.735 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-06-29T14:23:04.750 INFO:teuthology.orchestra.run.smithi167.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-06-29T14:23:04.751 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:04.751 DEBUG:teuthology.orchestra.run.smithi167:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-06-29T14:23:04.807 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-06-29T14:23:04.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:04.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:04.915 INFO:teuthology.orchestra.run.smithi167.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:04.915 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:23:05.052 INFO:teuthology.orchestra.run.smithi167.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:05.052 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:23:05.053 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:23:05.053 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: trying mount with new device syntax: 1@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:23:05.053 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:23:05.053 INFO:teuthology.orchestra.run.smithi167.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-06-29T14:23:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:23:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:23:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:23:05.054 INFO:teuthology.orchestra.run.smithi167.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:23:05.055 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:23:05.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:05.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-06-29T14:23:05.080 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:05.171 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:05.173 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:05.179 INFO:teuthology.orchestra.run.smithi167.stdout:client4550 2024-06-29T14:23:05.179 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:23:05.179 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:05.195 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:05.284 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:05.285 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:05.291 INFO:teuthology.orchestra.run.smithi167.stdout:client4550 2024-06-29T14:23:05.291 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:05.341 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:05.343 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:05.349 INFO:teuthology.orchestra.run.smithi167.stdout:client4550 2024-06-29T14:23:05.349 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.client4550/status') 2024-06-29T14:23:05.401 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:05.412 INFO:teuthology.orchestra.run.smithi167.stdout:instance: client.4550 (3)192.168.0.1:0/4260309181 2024-06-29T14:23:05.412 INFO:teuthology.orchestra.run.smithi167.stdout:blocklisted: false 2024-06-29T14:23:05.412 INFO:teuthology.orchestra.run.smithi167.stderr:0+1 records in 2024-06-29T14:23:05.412 INFO:teuthology.orchestra.run.smithi167.stderr:0+1 records out 2024-06-29T14:23:05.412 INFO:teuthology.orchestra.run.smithi167.stderr:69 bytes copied, 2.2189e-05 s, 3.1 MB/s 2024-06-29T14:23:05.413 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:05.419 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-06-29T14:23:05.419 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-29T14:23:05.420 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-06-29T14:23:05.420 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:23:05.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.547+0000 7fddbeae4640 1 -- 172.21.15.81:0/2262474455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb80b57b0 msgr2=0x7fddb815aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:05.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.547+0000 7fddbeae4640 1 --2- 172.21.15.81:0/2262474455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb80b57b0 0x7fddb815aed0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fddac009830 tx=0x7fddac02a510 comp rx=0 tx=0).stop 2024-06-29T14:23:05.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.547+0000 7fddbeae4640 1 -- 172.21.15.81:0/2262474455 shutdown_connections 2024-06-29T14:23:05.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.547+0000 7fddbeae4640 1 --2- 172.21.15.81:0/2262474455 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb806ef90 0x7fddb806f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:05.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.547+0000 7fddbeae4640 1 --2- 172.21.15.81:0/2262474455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb80b57b0 0x7fddb815aed0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:05.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.547+0000 7fddbeae4640 1 -- 172.21.15.81:0/2262474455 >> 172.21.15.81:0/2262474455 conn(0x7fddb80d0930 msgr2=0x7fddb80d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:05.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.547+0000 7fddbeae4640 1 -- 172.21.15.81:0/2262474455 shutdown_connections 2024-06-29T14:23:05.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.547+0000 7fddbeae4640 1 -- 172.21.15.81:0/2262474455 wait complete. 2024-06-29T14:23:05.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.549+0000 7fddbeae4640 1 Processor -- start 2024-06-29T14:23:05.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.549+0000 7fddbeae4640 1 -- start start 2024-06-29T14:23:05.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.550+0000 7fddbeae4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fddb806ef90 0x7fddb814a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:05.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.550+0000 7fddbeae4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb80b57b0 0x7fddb806c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.550+0000 7fddbeae4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb80ce1d0 0x7fddb806c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.550+0000 7fddbeae4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fddb80cecf0 con 0x7fddb80b57b0 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.550+0000 7fddbeae4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fddb80b4b80 con 0x7fddb806ef90 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.550+0000 7fddbeae4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fddb80cee70 con 0x7fddb80ce1d0 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.550+0000 7fddb7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb80b57b0 0x7fddb806c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb80b57b0 0x7fddb806c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37926/0 (socket says 172.21.15.81:37926) 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb7fff640 1 -- 172.21.15.81:0/36714592 learned_addr learned my addr 172.21.15.81:0/36714592 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddbd081640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb80ce1d0 0x7fddb806c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddbc880640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fddb806ef90 0x7fddb814a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb7fff640 1 -- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb80ce1d0 msgr2=0x7fddb806c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb7fff640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb80ce1d0 0x7fddb806c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:05.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb7fff640 1 -- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fddb806ef90 msgr2=0x7fddb814a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:05.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb7fff640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fddb806ef90 0x7fddb814a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:05.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb7fff640 1 -- 172.21.15.81:0/36714592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb81e2fd0 con 0x7fddb80b57b0 2024-06-29T14:23:05.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb7fff640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb80b57b0 0x7fddb806c2f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdda400e7c0 tx=0x7fdda400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:05.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb57fa640 1 -- 172.21.15.81:0/36714592 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda400ccc0 con 0x7fddb80b57b0 2024-06-29T14:23:05.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb57fa640 1 -- 172.21.15.81:0/36714592 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdda400ee70 con 0x7fddb80b57b0 2024-06-29T14:23:05.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddbeae4640 1 -- 172.21.15.81:0/36714592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb80ada60 con 0x7fddb80b57b0 2024-06-29T14:23:05.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddb57fa640 1 -- 172.21.15.81:0/36714592 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4010690 con 0x7fddb80b57b0 2024-06-29T14:23:05.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.551+0000 7fddbeae4640 1 -- 172.21.15.81:0/36714592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb80ae020 con 0x7fddb80b57b0 2024-06-29T14:23:05.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.552+0000 7fddbeae4640 1 -- 172.21.15.81:0/36714592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddb806e2a0 con 0x7fddb80b57b0 2024-06-29T14:23:05.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.553+0000 7fddb57fa640 1 -- 172.21.15.81:0/36714592 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdda4010830 con 0x7fddb80b57b0 2024-06-29T14:23:05.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.553+0000 7fddb57fa640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdd9c076300 0x7fdd9c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:05.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.554+0000 7fddb57fa640 1 -- 172.21.15.81:0/36714592 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fdda4098020 con 0x7fddb80b57b0 2024-06-29T14:23:05.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.554+0000 7fddbc880640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdd9c076300 0x7fdd9c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:05.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.554+0000 7fddbc880640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdd9c076300 0x7fdd9c078810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fddac034000 tx=0x7fddac004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:05.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.558+0000 7fddb57fa640 1 -- 172.21.15.81:0/36714592 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fddb806e2a0 con 0x7fddb80b57b0 2024-06-29T14:23:05.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.774+0000 7fddbeae4640 1 -- 172.21.15.81:0/36714592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fddb8159880 con 0x7fddb80b57b0 2024-06-29T14:23:05.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.774+0000 7fddb57fa640 1 -- 172.21.15.81:0/36714592 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7fdda4061980 con 0x7fddb80b57b0 2024-06-29T14:23:05.773 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 -- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdd9c076300 msgr2=0x7fdd9c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdd9c076300 0x7fdd9c078810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fddac034000 tx=0x7fddac004730 comp rx=0 tx=0).stop 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 -- 172.21.15.81:0/36714592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb80b57b0 msgr2=0x7fddb806c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb80b57b0 0x7fddb806c2f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdda400e7c0 tx=0x7fdda400ec80 comp rx=0 tx=0).stop 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 -- 172.21.15.81:0/36714592 shutdown_connections 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdd9c076300 0x7fdd9c078810 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb80ce1d0 0x7fddb806c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb80b57b0 0x7fddb806c2f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 --2- 172.21.15.81:0/36714592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fddb806ef90 0x7fddb814a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 -- 172.21.15.81:0/36714592 >> 172.21.15.81:0/36714592 conn(0x7fddb80d0930 msgr2=0x7fddb8155800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 -- 172.21.15.81:0/36714592 shutdown_connections 2024-06-29T14:23:05.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:05.776+0000 7fddbeae4640 1 -- 172.21.15.81:0/36714592 wait complete. 2024-06-29T14:23:05.786 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-29T14:23:05.786 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:23:05.786 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi167.front.sepia.ceph.com 2024-06-29T14:23:05.786 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-06-29T14:23:05.786 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-06-29T14:23:05.787 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-29T14:23:05.787 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-29T14:23:05.787 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:23:05.787 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-06-29T14:23:05.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:05.787 DEBUG:teuthology.orchestra.run.smithi167:> ip addr 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 ::1/128 scope host 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:d9:35:e6 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout: altname enp7s0f0 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:05.803 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:d9:35:e7 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout: altname enp7s0f1 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:88:73:08 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 0c:c4:7a:88:73:09 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout: inet 172.21.15.167/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft 42695sec preferred_lft 42695sec 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 fe80::ec4:7aff:fe88:7309/64 scope link 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 12:bb:31:bb:f9:c2 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:23:05.804 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:05.805 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 fe80::810:95ff:fe30:29ad/64 scope link 2024-06-29T14:23:05.805 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:05.805 INFO:teuthology.orchestra.run.smithi167.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:23:05.805 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether 12:bb:31:bb:f9:c2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-06-29T14:23:05.805 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 fe80::10bb:31ff:febb:f9c2/64 scope link tentative 2024-06-29T14:23:05.805 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:05.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:05.806 DEBUG:teuthology.orchestra.run.smithi167:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:23:05.867 INFO:teuthology.orchestra.run.smithi167.stdout:1 2024-06-29T14:23:05.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:05.868 DEBUG:teuthology.orchestra.run.smithi167:> route 2024-06-29T14:23:05.925 INFO:teuthology.orchestra.run.smithi167.stdout:Kernel IP routing table 2024-06-29T14:23:05.925 INFO:teuthology.orchestra.run.smithi167.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:23:05.925 INFO:teuthology.orchestra.run.smithi167.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:23:05.925 INFO:teuthology.orchestra.run.smithi167.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:23:05.925 INFO:teuthology.orchestra.run.smithi167.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:23:05.925 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:05.925 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2024-06-29T14:23:05.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:23:05.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:23:05.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:23:05.926 DEBUG:teuthology.orchestra.run.smithi167:> ') 2024-06-29T14:23:06.014 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:06.046 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:06.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:06.053 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list 2024-06-29T14:23:06.068 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-06-29T14:23:06.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:06.068 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list-id 2024-06-29T14:23:06.123 INFO:teuthology.orchestra.run.smithi167.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-06-29T14:23:06.124 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:06.124 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2024-06-29T14:23:06.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-06-29T14:23:06.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-06-29T14:23:06.124 DEBUG:teuthology.orchestra.run.smithi167:> ') 2024-06-29T14:23:06.213 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:06.235 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:06.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:06.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-06-29T14:23:06.266 INFO:teuthology.orchestra.run.smithi167.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:23:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:23:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:23:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 ::1/128 scope host 2024-06-29T14:23:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:06.267 INFO:teuthology.orchestra.run.smithi167.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:23:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether e6:55:e1:72:dd:07 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-06-29T14:23:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-06-29T14:23:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 fe80::e455:e1ff:fe72:dd07/64 scope link 2024-06-29T14:23:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:06.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:06.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-06-29T14:23:06.333 INFO:teuthology.orchestra.run.smithi167.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:23:06.333 INFO:teuthology.orchestra.run.smithi167.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:23:06.333 INFO:teuthology.orchestra.run.smithi167.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:23:06.333 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:06.333 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 ::1/128 scope host 2024-06-29T14:23:06.333 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:06.333 INFO:teuthology.orchestra.run.smithi167.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:23:06.333 INFO:teuthology.orchestra.run.smithi167.stdout: link/ether e6:55:e1:72:dd:07 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-06-29T14:23:06.333 INFO:teuthology.orchestra.run.smithi167.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-06-29T14:23:06.334 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:06.334 INFO:teuthology.orchestra.run.smithi167.stdout: inet6 fe80::e455:e1ff:fe72:dd07/64 scope link 2024-06-29T14:23:06.334 INFO:teuthology.orchestra.run.smithi167.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:06.335 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-06-29T14:23:06.335 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:06.335 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2024-06-29T14:23:06.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-06-29T14:23:06.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-06-29T14:23:06.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-06-29T14:23:06.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-06-29T14:23:06.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-06-29T14:23:06.335 DEBUG:teuthology.orchestra.run.smithi167:> ') 2024-06-29T14:23:06.424 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:06.482 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:06.488 DEBUG:teuthology.orchestra.run.smithi167:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:06.488 DEBUG:teuthology.orchestra.run.smithi167:> set -e 2024-06-29T14:23:06.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link set brx.1 up 2024-06-29T14:23:06.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link set dev brx.1 master ceph-brx 2024-06-29T14:23:06.488 DEBUG:teuthology.orchestra.run.smithi167:> ') 2024-06-29T14:23:06.537 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:06.560 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:06.566 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:06.566 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-06-29T14:23:06.582 INFO:teuthology.orchestra.run.smithi167.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-06-29T14:23:06.583 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:06.583 DEBUG:teuthology.orchestra.run.smithi167:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-06-29T14:23:06.638 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-06-29T14:23:06.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:06.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:06.734 INFO:teuthology.orchestra.run.smithi167.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:06.734 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:23:06.740 INFO:teuthology.orchestra.run.smithi167.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:06.740 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:23:06.741 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:23:06.741 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: trying mount with new device syntax: 3@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:23:06.741 INFO:teuthology.orchestra.run.smithi167.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:23:06.741 INFO:teuthology.orchestra.run.smithi167.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-06-29T14:23:06.741 INFO:teuthology.orchestra.run.smithi167.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:23:06.741 INFO:teuthology.orchestra.run.smithi167.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:23:06.741 INFO:teuthology.orchestra.run.smithi167.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:23:06.742 INFO:teuthology.orchestra.run.smithi167.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:23:06.743 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:23:06.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:06.743 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-06-29T14:23:06.768 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:06.858 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:06 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:06.860 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:06.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:23:06.866 INFO:teuthology.orchestra.run.smithi167.stdout:client4779 2024-06-29T14:23:06.866 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:23:06.866 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:06.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:23:06.883 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:07.003 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:07.004 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:07.010 INFO:teuthology.orchestra.run.smithi167.stdout:client4779 2024-06-29T14:23:07.010 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:07.059 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:07.060 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:07.066 INFO:teuthology.orchestra.run.smithi167.stdout:client4779 2024-06-29T14:23:07.066 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.client4779/status') 2024-06-29T14:23:07.116 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:07.126 INFO:teuthology.orchestra.run.smithi167.stdout:instance: client.4779 (3)172.21.15.167:0/2937257121 2024-06-29T14:23:07.126 INFO:teuthology.orchestra.run.smithi167.stdout:blocklisted: false 2024-06-29T14:23:07.126 INFO:teuthology.orchestra.run.smithi167.stderr:0+1 records in 2024-06-29T14:23:07.126 INFO:teuthology.orchestra.run.smithi167.stderr:0+1 records out 2024-06-29T14:23:07.126 INFO:teuthology.orchestra.run.smithi167.stderr:71 bytes copied, 2.3852e-05 s, 3.0 MB/s 2024-06-29T14:23:07.127 INFO:teuthology.orchestra.run.smithi167.stderr:2024-06-29T14:23:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:07.133 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-06-29T14:23:07.151 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:07.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.282+0000 7f1e0092a640 1 -- 172.21.15.81:0/943707448 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dfc073480 msgr2=0x7f1dfc15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.282+0000 7f1e0092a640 1 --2- 172.21.15.81:0/943707448 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dfc073480 0x7f1dfc15b6e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009830 tx=0x7f1df002a480 comp rx=0 tx=0).stop 2024-06-29T14:23:07.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.282+0000 7f1e0092a640 1 -- 172.21.15.81:0/943707448 shutdown_connections 2024-06-29T14:23:07.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.282+0000 7f1e0092a640 1 --2- 172.21.15.81:0/943707448 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dfc073480 0x7f1dfc15b6e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.282+0000 7f1e0092a640 1 --2- 172.21.15.81:0/943707448 >> v2:172.21.15.167:3300/0 conn(0x7f1dfc0cd150 0x7f1dfc072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.282+0000 7f1e0092a640 1 -- 172.21.15.81:0/943707448 >> 172.21.15.81:0/943707448 conn(0x7f1dfc0b2920 msgr2=0x7f1dfc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:07.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.282+0000 7f1e0092a640 1 -- 172.21.15.81:0/943707448 shutdown_connections 2024-06-29T14:23:07.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.282+0000 7f1e0092a640 1 -- 172.21.15.81:0/943707448 wait complete. 2024-06-29T14:23:07.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.284+0000 7f1e0092a640 1 Processor -- start 2024-06-29T14:23:07.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.284+0000 7f1e0092a640 1 -- start start 2024-06-29T14:23:07.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.285+0000 7f1e0092a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dfc073480 0x7f1dfc14be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.285+0000 7f1e0092a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dfc0cd150 0x7f1dfc156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.285+0000 7f1e0092a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dfc15bc20 0x7f1dfc1423e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.285+0000 7f1e0092a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc15c1b0 con 0x7f1dfc15bc20 2024-06-29T14:23:07.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.285+0000 7f1e0092a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc15c030 con 0x7f1dfc0cd150 2024-06-29T14:23:07.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.285+0000 7f1e0092a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1dfc15f060 con 0x7f1dfc073480 2024-06-29T14:23:07.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.285+0000 7f1dfad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dfc15bc20 0x7f1dfc1423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1dfad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dfc15bc20 0x7f1dfc1423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37940/0 (socket says 172.21.15.81:37940) 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1dfad9d640 1 -- 172.21.15.81:0/1702271464 learned_addr learned my addr 172.21.15.81:0/1702271464 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1df9d9b640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dfc0cd150 0x7f1dfc156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1dfa59c640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dfc073480 0x7f1dfc14be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1dfad9d640 1 -- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dfc073480 msgr2=0x7f1dfc14be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1dfad9d640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dfc073480 0x7f1dfc14be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1dfad9d640 1 -- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dfc0cd150 msgr2=0x7f1dfc156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1dfad9d640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dfc0cd150 0x7f1dfc156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1dfad9d640 1 -- 172.21.15.81:0/1702271464 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dfc1f6e60 con 0x7f1dfc15bc20 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1df9d9b640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dfc0cd150 0x7f1dfc156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1dfa59c640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dfc073480 0x7f1dfc14be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1dfad9d640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dfc15bc20 0x7f1dfc1423e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1df400c700 tx=0x7f1df400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:07.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1deaffd640 1 -- 172.21.15.81:0/1702271464 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4003b70 con 0x7f1dfc15bc20 2024-06-29T14:23:07.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1deaffd640 1 -- 172.21.15.81:0/1702271464 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1df4003d10 con 0x7f1dfc15bc20 2024-06-29T14:23:07.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1e0092a640 1 -- 172.21.15.81:0/1702271464 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dfc1f7140 con 0x7f1dfc15bc20 2024-06-29T14:23:07.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1deaffd640 1 -- 172.21.15.81:0/1702271464 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df400aab0 con 0x7f1dfc15bc20 2024-06-29T14:23:07.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.286+0000 7f1e0092a640 1 -- 172.21.15.81:0/1702271464 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dfc142920 con 0x7f1dfc15bc20 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.287+0000 7f1e0092a640 1 -- 172.21.15.81:0/1702271464 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dfc15f580 con 0x7f1dfc15bc20 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.288+0000 7f1deaffd640 1 -- 172.21.15.81:0/1702271464 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1df400ac50 con 0x7f1dfc15bc20 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.289+0000 7f1deaffd640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1ddc076300 0x7f1ddc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.289+0000 7f1deaffd640 1 -- 172.21.15.81:0/1702271464 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1df4097630 con 0x7f1dfc15bc20 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.289+0000 7f1dfa59c640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1ddc076300 0x7f1ddc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.289+0000 7f1dfa59c640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1ddc076300 0x7f1ddc078810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1dec005fd0 tx=0x7f1dec009260 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:07.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.293+0000 7f1deaffd640 1 -- 172.21.15.81:0/1702271464 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1dfc073890 con 0x7f1dfc15bc20 2024-06-29T14:23:07.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.481+0000 7f1e0092a640 1 -- 172.21.15.81:0/1702271464 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1dfc15b0e0 con 0x7f1dfc15bc20 2024-06-29T14:23:07.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.483+0000 7f1deaffd640 1 -- 172.21.15.81:0/1702271464 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f1df4060f00 con 0x7f1dfc15bc20 2024-06-29T14:23:07.481 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:07.481 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:07.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 -- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1ddc076300 msgr2=0x7f1ddc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1ddc076300 0x7f1ddc078810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1dec005fd0 tx=0x7f1dec009260 comp rx=0 tx=0).stop 2024-06-29T14:23:07.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 -- 172.21.15.81:0/1702271464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dfc15bc20 msgr2=0x7f1dfc1423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dfc15bc20 0x7f1dfc1423e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1df400c700 tx=0x7f1df400cbc0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 -- 172.21.15.81:0/1702271464 shutdown_connections 2024-06-29T14:23:07.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1ddc076300 0x7f1ddc078810 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dfc15bc20 0x7f1dfc1423e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1dfc0cd150 0x7f1dfc156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 --2- 172.21.15.81:0/1702271464 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1dfc073480 0x7f1dfc14be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 -- 172.21.15.81:0/1702271464 >> 172.21.15.81:0/1702271464 conn(0x7f1dfc0b2920 msgr2=0x7f1dfc0b0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:07.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 -- 172.21.15.81:0/1702271464 shutdown_connections 2024-06-29T14:23:07.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.485+0000 7f1e0092a640 1 -- 172.21.15.81:0/1702271464 wait complete. 2024-06-29T14:23:07.495 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:07.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:07.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.618+0000 7f769a3a0640 1 -- 172.21.15.81:0/2080783741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76940b2db0 msgr2=0x7f76940b31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.618+0000 7f769a3a0640 1 --2- 172.21.15.81:0/2080783741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76940b2db0 0x7f76940b31c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f76940753d0 tx=0x7f768c02aa30 comp rx=0 tx=0).stop 2024-06-29T14:23:07.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.618+0000 7f769a3a0640 1 -- 172.21.15.81:0/2080783741 shutdown_connections 2024-06-29T14:23:07.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.618+0000 7f769a3a0640 1 --2- 172.21.15.81:0/2080783741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76940b2db0 0x7f76940b31c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.618+0000 7f769a3a0640 1 --2- 172.21.15.81:0/2080783741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f769406d300 0x7f769406d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.618+0000 7f769a3a0640 1 --2- 172.21.15.81:0/2080783741 >> v2:172.21.15.167:3300/0 conn(0x7f769415ae10 0x7f7694072c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.618+0000 7f769a3a0640 1 -- 172.21.15.81:0/2080783741 >> 172.21.15.81:0/2080783741 conn(0x7f76940cfec0 msgr2=0x7f76940d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:07.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.618+0000 7f769a3a0640 1 -- 172.21.15.81:0/2080783741 shutdown_connections 2024-06-29T14:23:07.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.618+0000 7f769a3a0640 1 -- 172.21.15.81:0/2080783741 wait complete. 2024-06-29T14:23:07.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.620+0000 7f769a3a0640 1 Processor -- start 2024-06-29T14:23:07.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.621+0000 7f769a3a0640 1 -- start start 2024-06-29T14:23:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f769a3a0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f769406d300 0x7f7694152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f769a3a0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76940b2db0 0x7f76941546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f769a3a0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f769415ae10 0x7f7694154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f769a3a0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76940b3700 con 0x7f769415ae10 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f769a3a0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7694072680 con 0x7f76940b2db0 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f769a3a0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f76940b3880 con 0x7f769406d300 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f7699b9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f769415ae10 0x7f7694154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f7699b9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f769415ae10 0x7f7694154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37952/0 (socket says 172.21.15.81:37952) 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f7699b9f640 1 -- 172.21.15.81:0/3179217154 learned_addr learned my addr 172.21.15.81:0/3179217154 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f769939e640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f769406d300 0x7f7694152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f7698b9d640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76940b2db0 0x7f76941546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f7699b9f640 1 -- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f769406d300 msgr2=0x7f7694152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f7699b9f640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f769406d300 0x7f7694152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f7699b9f640 1 -- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76940b2db0 msgr2=0x7f76941546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f7699b9f640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76940b2db0 0x7f76941546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.622+0000 7f7699b9f640 1 -- 172.21.15.81:0/3179217154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f769409bfa0 con 0x7f769415ae10 2024-06-29T14:23:07.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.623+0000 7f7699b9f640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f769415ae10 0x7f7694154c30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f768c005fd0 tx=0x7f768c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:07.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.623+0000 7f7689ffb640 1 -- 172.21.15.81:0/3179217154 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c033050 con 0x7f769415ae10 2024-06-29T14:23:07.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.623+0000 7f7689ffb640 1 -- 172.21.15.81:0/3179217154 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f768c004450 con 0x7f769415ae10 2024-06-29T14:23:07.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.623+0000 7f769a3a0640 1 -- 172.21.15.81:0/3179217154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76941539a0 con 0x7f769415ae10 2024-06-29T14:23:07.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.623+0000 7f7689ffb640 1 -- 172.21.15.81:0/3179217154 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c038450 con 0x7f769415ae10 2024-06-29T14:23:07.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.623+0000 7f769a3a0640 1 -- 172.21.15.81:0/3179217154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76940ccf10 con 0x7f769415ae10 2024-06-29T14:23:07.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.624+0000 7f769a3a0640 1 -- 172.21.15.81:0/3179217154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f769409c170 con 0x7f769415ae10 2024-06-29T14:23:07.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.625+0000 7f7689ffb640 1 -- 172.21.15.81:0/3179217154 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f768c036070 con 0x7f769415ae10 2024-06-29T14:23:07.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.625+0000 7f7689ffb640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f76780762d0 0x7f76780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.625+0000 7f7689ffb640 1 -- 172.21.15.81:0/3179217154 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f768c0b8960 con 0x7f769415ae10 2024-06-29T14:23:07.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.626+0000 7f769939e640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f76780762d0 0x7f76780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.626+0000 7f769939e640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f76780762d0 0x7f76780787e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7690009ab0 tx=0x7f7690005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:07.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.630+0000 7f7689ffb640 1 -- 172.21.15.81:0/3179217154 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f768c081c70 con 0x7f769415ae10 2024-06-29T14:23:07.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.840+0000 7f769a3a0640 1 -- 172.21.15.81:0/3179217154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f76940b31c0 con 0x7f769415ae10 2024-06-29T14:23:07.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.840+0000 7f7689ffb640 1 -- 172.21.15.81:0/3179217154 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f768c085f70 con 0x7f769415ae10 2024-06-29T14:23:07.839 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.842+0000 7f769a3a0640 1 -- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f76780762d0 msgr2=0x7f76780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.842+0000 7f769a3a0640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f76780762d0 0x7f76780787e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7690009ab0 tx=0x7f7690005f90 comp rx=0 tx=0).stop 2024-06-29T14:23:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.842+0000 7f769a3a0640 1 -- 172.21.15.81:0/3179217154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f769415ae10 msgr2=0x7f7694154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.842+0000 7f769a3a0640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f769415ae10 0x7f7694154c30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f768c005fd0 tx=0x7f768c003f90 comp rx=0 tx=0).stop 2024-06-29T14:23:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.842+0000 7f769a3a0640 1 -- 172.21.15.81:0/3179217154 shutdown_connections 2024-06-29T14:23:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.842+0000 7f769a3a0640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f76780762d0 0x7f76780787e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.842+0000 7f769a3a0640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f769415ae10 0x7f7694154c30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.842+0000 7f769a3a0640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76940b2db0 0x7f76941546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.842+0000 7f769a3a0640 1 --2- 172.21.15.81:0/3179217154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f769406d300 0x7f7694152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.843+0000 7f769a3a0640 1 -- 172.21.15.81:0/3179217154 >> 172.21.15.81:0/3179217154 conn(0x7f76940cfec0 msgr2=0x7f769406ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.843+0000 7f769a3a0640 1 -- 172.21.15.81:0/3179217154 shutdown_connections 2024-06-29T14:23:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.843+0000 7f769a3a0640 1 -- 172.21.15.81:0/3179217154 wait complete. 2024-06-29T14:23:07.852 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:23:07.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.971+0000 7f1ff3a09640 1 -- 172.21.15.81:0/2015164199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fec159bd0 msgr2=0x7f1fec0d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.971+0000 7f1ff3a09640 1 --2- 172.21.15.81:0/2015164199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fec159bd0 0x7f1fec0d0010 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1fec072100 tx=0x7f1fdc02a5f0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.971+0000 7f1ff3a09640 1 -- 172.21.15.81:0/2015164199 shutdown_connections 2024-06-29T14:23:07.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.971+0000 7f1ff3a09640 1 --2- 172.21.15.81:0/2015164199 >> v2:172.21.15.167:3300/0 conn(0x7f1fec0d0550 0x7f1fec06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.971+0000 7f1ff3a09640 1 --2- 172.21.15.81:0/2015164199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fec159bd0 0x7f1fec0d0010 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.971+0000 7f1ff3a09640 1 -- 172.21.15.81:0/2015164199 >> 172.21.15.81:0/2015164199 conn(0x7f1fec0b2920 msgr2=0x7f1fec0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:07.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.971+0000 7f1ff3a09640 1 -- 172.21.15.81:0/2015164199 shutdown_connections 2024-06-29T14:23:07.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.971+0000 7f1ff3a09640 1 -- 172.21.15.81:0/2015164199 wait complete. 2024-06-29T14:23:07.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.974+0000 7f1ff3a09640 1 Processor -- start 2024-06-29T14:23:07.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.974+0000 7f1ff3a09640 1 -- start start 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.976+0000 7f1ff3a09640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fec06c830 0x7f1fec1f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.976+0000 7f1ff3a09640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fec0d0550 0x7f1fec1f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.976+0000 7f1ff3a09640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fec159bd0 0x7f1fec0ab620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.976+0000 7f1ff3a09640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec15fa90 con 0x7f1fec06c830 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.976+0000 7f1ff3a09640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec0703a0 con 0x7f1fec0d0550 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.976+0000 7f1ff3a09640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1fec15fc10 con 0x7f1fec159bd0 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.976+0000 7f1ff17a5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fec06c830 0x7f1fec1f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.976+0000 7f1ff17a5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fec06c830 0x7f1fec1f4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37964/0 (socket says 172.21.15.81:37964) 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.976+0000 7f1ff17a5640 1 -- 172.21.15.81:0/3510781668 learned_addr learned my addr 172.21.15.81:0/3510781668 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.976+0000 7f1ff0fa4640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fec0d0550 0x7f1fec1f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1ff1fa6640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fec159bd0 0x7f1fec0ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1ff17a5640 1 -- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fec159bd0 msgr2=0x7f1fec0ab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1ff17a5640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fec159bd0 0x7f1fec0ab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1ff17a5640 1 -- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fec0d0550 msgr2=0x7f1fec1f6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1ff17a5640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fec0d0550 0x7f1fec1f6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1ff17a5640 1 -- 172.21.15.81:0/3510781668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fec0aa9a0 con 0x7f1fec06c830 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1ff17a5640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fec06c830 0x7f1fec1f4740 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc034040 tx=0x7f1fdc003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1fe1ffb640 1 -- 172.21.15.81:0/3510781668 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc03e030 con 0x7f1fec06c830 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1fe1ffb640 1 -- 172.21.15.81:0/3510781668 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fdc003d10 con 0x7f1fec06c830 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1ff3a09640 1 -- 172.21.15.81:0/3510781668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fec141070 con 0x7f1fec06c830 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1fe1ffb640 1 -- 172.21.15.81:0/3510781668 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc035040 con 0x7f1fec06c830 2024-06-29T14:23:07.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.977+0000 7f1ff3a09640 1 -- 172.21.15.81:0/3510781668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fec141630 con 0x7f1fec06c830 2024-06-29T14:23:07.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.979+0000 7f1ff3a09640 1 -- 172.21.15.81:0/3510781668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fec140e60 con 0x7f1fec06c830 2024-06-29T14:23:07.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.979+0000 7f1fe1ffb640 1 -- 172.21.15.81:0/3510781668 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1fdc041030 con 0x7f1fec06c830 2024-06-29T14:23:07.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.980+0000 7f1fe1ffb640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1fd0076300 0x7f1fd0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:07.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.980+0000 7f1fe1ffb640 1 -- 172.21.15.81:0/3510781668 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1fdc0b7df0 con 0x7f1fec06c830 2024-06-29T14:23:07.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.980+0000 7f1ff0fa4640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1fd0076300 0x7f1fd0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:07.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.980+0000 7f1ff0fa4640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1fd0076300 0x7f1fd0078810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0b12a0 tx=0x7f1fe8005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:07.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:07.984+0000 7f1fe1ffb640 1 -- 172.21.15.81:0/3510781668 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1fdc081070 con 0x7f1fec06c830 2024-06-29T14:23:08.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.167+0000 7f1ff3a09640 1 -- 172.21.15.81:0/3510781668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1fec06c2f0 con 0x7f1fec06c830 2024-06-29T14:23:08.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.168+0000 7f1fe1ffb640 1 -- 172.21.15.81:0/3510781668 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f1fdc085370 con 0x7f1fec06c830 2024-06-29T14:23:08.166 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:23:08.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 -- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1fd0076300 msgr2=0x7f1fd0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1fd0076300 0x7f1fd0078810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0b12a0 tx=0x7f1fe8005f90 comp rx=0 tx=0).stop 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 -- 172.21.15.81:0/3510781668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fec06c830 msgr2=0x7f1fec1f4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fec06c830 0x7f1fec1f4740 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc034040 tx=0x7f1fdc003820 comp rx=0 tx=0).stop 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 -- 172.21.15.81:0/3510781668 shutdown_connections 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1fd0076300 0x7f1fd0078810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fec159bd0 0x7f1fec0ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fec0d0550 0x7f1fec1f6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 --2- 172.21.15.81:0/3510781668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fec06c830 0x7f1fec1f4740 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 -- 172.21.15.81:0/3510781668 >> 172.21.15.81:0/3510781668 conn(0x7f1fec0b2920 msgr2=0x7f1fec0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 -- 172.21.15.81:0/3510781668 shutdown_connections 2024-06-29T14:23:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.170+0000 7f1ff3a09640 1 -- 172.21.15.81:0/3510781668 wait complete. 2024-06-29T14:23:08.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:23:08.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.299+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2529251148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1de0003a00 msgr2=0x7f1de0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.299+0000 7f1dfba5e640 1 --2- 172.21.15.81:0/2529251148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1de0003a00 0x7f1de0003e70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009b20 tx=0x7f1de0005830 comp rx=0 tx=0).stop 2024-06-29T14:23:08.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.299+0000 7f1df9ffb640 1 -- 172.21.15.81:0/2529251148 reap_dead start 2024-06-29T14:23:08.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.299+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2529251148 shutdown_connections 2024-06-29T14:23:08.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.299+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2529251148 >> 172.21.15.81:0/2529251148 conn(0x7f1df40d3c60 msgr2=0x7f1df40d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:08.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.299+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2529251148 shutdown_connections 2024-06-29T14:23:08.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.299+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2529251148 wait complete. 2024-06-29T14:23:08.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.301+0000 7f1dfba5e640 1 Processor -- start 2024-06-29T14:23:08.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.301+0000 7f1dfba5e640 1 -- start start 2024-06-29T14:23:08.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.302+0000 7f1dfba5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1de0003a00 0x7f1df40a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.302+0000 7f1dfba5e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1df41e6070 0x7f1df41e9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.302+0000 7f1dfba5e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1df40c8030 0x7f1df41ea290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.302+0000 7f1dfba5e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1df40d15a0 con 0x7f1de0003a00 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.302+0000 7f1dfba5e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1df40d1420 con 0x7f1df40c8030 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.302+0000 7f1dfba5e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1df40d4990 con 0x7f1df41e6070 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.302+0000 7f1df97fa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1de0003a00 0x7f1df40a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.302+0000 7f1df97fa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1de0003a00 0x7f1df40a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37980/0 (socket says 172.21.15.81:37980) 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.302+0000 7f1df97fa640 1 -- 172.21.15.81:0/2759600518 learned_addr learned my addr 172.21.15.81:0/2759600518 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1df97fa640 1 -- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1df41e6070 msgr2=0x7f1df41e9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1df97fa640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1df41e6070 0x7f1df41e9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1df97fa640 1 -- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1df40c8030 msgr2=0x7f1df41ea290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1df97fa640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1df40c8030 0x7f1df41ea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1df97fa640 1 -- 172.21.15.81:0/2759600518 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df40cce80 con 0x7f1de0003a00 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1df97fa640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1de0003a00 0x7f1df40a1300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1df0036040 tx=0x7f1df0033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:08.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1de9ffb640 1 -- 172.21.15.81:0/2759600518 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0038040 con 0x7f1de0003a00 2024-06-29T14:23:08.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1de9ffb640 1 -- 172.21.15.81:0/2759600518 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1df0005020 con 0x7f1de0003a00 2024-06-29T14:23:08.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2759600518 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df40cd160 con 0x7f1de0003a00 2024-06-29T14:23:08.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1de9ffb640 1 -- 172.21.15.81:0/2759600518 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0005430 con 0x7f1de0003a00 2024-06-29T14:23:08.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.303+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2759600518 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df40cd720 con 0x7f1de0003a00 2024-06-29T14:23:08.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.304+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2759600518 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df40d2650 con 0x7f1de0003a00 2024-06-29T14:23:08.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.305+0000 7f1de9ffb640 1 -- 172.21.15.81:0/2759600518 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1df0045020 con 0x7f1de0003a00 2024-06-29T14:23:08.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.306+0000 7f1de9ffb640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1dd80762d0 0x7f1dd80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.306+0000 7f1de9ffb640 1 -- 172.21.15.81:0/2759600518 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1df00b8d30 con 0x7f1de0003a00 2024-06-29T14:23:08.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.306+0000 7f1df8ff9640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1dd80762d0 0x7f1dd80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:08.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.307+0000 7f1df8ff9640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1dd80762d0 0x7f1dd80787e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1df40c7f80 tx=0x7f1de4009530 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:08.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.313+0000 7f1de9ffb640 1 -- 172.21.15.81:0/2759600518 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1df0082890 con 0x7f1de0003a00 2024-06-29T14:23:08.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.499+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2759600518 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1df41e7490 con 0x7f1de0003a00 2024-06-29T14:23:08.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.500+0000 7f1de9ffb640 1 -- 172.21.15.81:0/2759600518 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f1df0082190 con 0x7f1de0003a00 2024-06-29T14:23:08.498 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:23:08.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1dd80762d0 msgr2=0x7f1dd80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1dd80762d0 0x7f1dd80787e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1df40c7f80 tx=0x7f1de4009530 comp rx=0 tx=0).stop 2024-06-29T14:23:08.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2759600518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1de0003a00 msgr2=0x7f1df40a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1de0003a00 0x7f1df40a1300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1df0036040 tx=0x7f1df0033670 comp rx=0 tx=0).stop 2024-06-29T14:23:08.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2759600518 shutdown_connections 2024-06-29T14:23:08.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1dd80762d0 0x7f1dd80787e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1df40c8030 0x7f1df41ea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1df41e6070 0x7f1df41e9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 --2- 172.21.15.81:0/2759600518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1de0003a00 0x7f1df40a1300 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2759600518 >> 172.21.15.81:0/2759600518 conn(0x7f1df40d3c60 msgr2=0x7f1df40b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:08.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2759600518 shutdown_connections 2024-06-29T14:23:08.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.502+0000 7f1dfba5e640 1 -- 172.21.15.81:0/2759600518 wait complete. 2024-06-29T14:23:08.511 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:08.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.626+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/2452958416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b7806f660 msgr2=0x7f3b7806fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.626+0000 7f3b7d5c7640 1 --2- 172.21.15.81:0/2452958416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b7806f660 0x7f3b7806fa70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3b70005fd0 tx=0x7f3b7002a4a0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.627+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/2452958416 shutdown_connections 2024-06-29T14:23:08.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.627+0000 7f3b7d5c7640 1 --2- 172.21.15.81:0/2452958416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b7815b5b0 0x7f3b780b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.627+0000 7f3b7d5c7640 1 --2- 172.21.15.81:0/2452958416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b7806f660 0x7f3b7806fa70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.627+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/2452958416 >> 172.21.15.81:0/2452958416 conn(0x7f3b7806ccc0 msgr2=0x7f3b780b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:08.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.627+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/2452958416 shutdown_connections 2024-06-29T14:23:08.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.627+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/2452958416 wait complete. 2024-06-29T14:23:08.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.628+0000 7f3b7d5c7640 1 Processor -- start 2024-06-29T14:23:08.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.629+0000 7f3b7d5c7640 1 -- start start 2024-06-29T14:23:08.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b7d5c7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b7806f660 0x7f3b781f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b7d5c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b780b35b0 0x7f3b780cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b7d5c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b7815b5b0 0x7f3b7814edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b7d5c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3b780722b0 con 0x7f3b7815b5b0 2024-06-29T14:23:08.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b7d5c7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3b78072130 con 0x7f3b780b35b0 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b7d5c7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3b78072430 con 0x7f3b7806f660 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b7815b5b0 0x7f3b7814edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b7815b5b0 0x7f3b7814edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38002/0 (socket says 172.21.15.81:38002) 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b777fe640 1 -- 172.21.15.81:0/1496034027 learned_addr learned my addr 172.21.15.81:0/1496034027 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b76ffd640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b7806f660 0x7f3b781f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b767fc640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b780b35b0 0x7f3b780cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b777fe640 1 -- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b7806f660 msgr2=0x7f3b781f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b777fe640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b7806f660 0x7f3b781f6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b777fe640 1 -- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b780b35b0 msgr2=0x7f3b780cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b777fe640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b780b35b0 0x7f3b780cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b777fe640 1 -- 172.21.15.81:0/1496034027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b781e2df0 con 0x7f3b7815b5b0 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.630+0000 7f3b777fe640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b7815b5b0 0x7f3b7814edd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3b70006010 tx=0x7f3b700039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:08.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.631+0000 7f3b5f7fe640 1 -- 172.21.15.81:0/1496034027 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7003e030 con 0x7f3b7815b5b0 2024-06-29T14:23:08.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.631+0000 7f3b5f7fe640 1 -- 172.21.15.81:0/1496034027 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b700047a0 con 0x7f3b7815b5b0 2024-06-29T14:23:08.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.631+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/1496034027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b781e30d0 con 0x7f3b7815b5b0 2024-06-29T14:23:08.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.631+0000 7f3b5f7fe640 1 -- 172.21.15.81:0/1496034027 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70035040 con 0x7f3b7815b5b0 2024-06-29T14:23:08.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.631+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/1496034027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b781e3690 con 0x7f3b7815b5b0 2024-06-29T14:23:08.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.632+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/1496034027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b780b4430 con 0x7f3b7815b5b0 2024-06-29T14:23:08.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.632+0000 7f3b5f7fe640 1 -- 172.21.15.81:0/1496034027 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3b70041030 con 0x7f3b7815b5b0 2024-06-29T14:23:08.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.633+0000 7f3b5f7fe640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3b640762d0 0x7f3b640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.633+0000 7f3b5f7fe640 1 -- 172.21.15.81:0/1496034027 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3b700b7ee0 con 0x7f3b7815b5b0 2024-06-29T14:23:08.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.633+0000 7f3b76ffd640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3b640762d0 0x7f3b640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:08.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.634+0000 7f3b76ffd640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3b640762d0 0x7f3b640787e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800b450 tx=0x7f3b6800d040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:08.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.637+0000 7f3b5f7fe640 1 -- 172.21.15.81:0/1496034027 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3b7806ffb0 con 0x7f3b7815b5b0 2024-06-29T14:23:08.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.821+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/1496034027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b780cf620 con 0x7f3b7815b5b0 2024-06-29T14:23:08.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.822+0000 7f3b5f7fe640 1 -- 172.21.15.81:0/1496034027 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f3b70081160 con 0x7f3b7815b5b0 2024-06-29T14:23:08.821 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:08.821 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.824+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3b640762d0 msgr2=0x7f3b640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.824+0000 7f3b7d5c7640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3b640762d0 0x7f3b640787e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800b450 tx=0x7f3b6800d040 comp rx=0 tx=0).stop 2024-06-29T14:23:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.824+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/1496034027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b7815b5b0 msgr2=0x7f3b7814edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.825+0000 7f3b7d5c7640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b7815b5b0 0x7f3b7814edd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3b70006010 tx=0x7f3b700039e0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.825+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/1496034027 shutdown_connections 2024-06-29T14:23:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.825+0000 7f3b7d5c7640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3b640762d0 0x7f3b640787e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.825+0000 7f3b7d5c7640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b7815b5b0 0x7f3b7814edd0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.825+0000 7f3b7d5c7640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b780b35b0 0x7f3b780cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.825+0000 7f3b7d5c7640 1 --2- 172.21.15.81:0/1496034027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b7806f660 0x7f3b781f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.825+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/1496034027 >> 172.21.15.81:0/1496034027 conn(0x7f3b7806ccc0 msgr2=0x7f3b78068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:08.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.825+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/1496034027 shutdown_connections 2024-06-29T14:23:08.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.825+0000 7f3b7d5c7640 1 -- 172.21.15.81:0/1496034027 wait complete. 2024-06-29T14:23:08.834 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:08.835 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.954+0000 7f476fa81640 1 -- 172.21.15.81:0/2246817825 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47680ce510 msgr2=0x7f47680ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.954+0000 7f476fa81640 1 --2- 172.21.15.81:0/2246817825 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47680ce510 0x7f47680ce920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f47680b41b0 tx=0x7f475c02a530 comp rx=0 tx=0).stop 2024-06-29T14:23:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.954+0000 7f476fa81640 1 -- 172.21.15.81:0/2246817825 shutdown_connections 2024-06-29T14:23:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.954+0000 7f476fa81640 1 --2- 172.21.15.81:0/2246817825 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47680ce510 0x7f47680ce920 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.954+0000 7f476fa81640 1 --2- 172.21.15.81:0/2246817825 >> v2:172.21.15.167:3300/0 conn(0x7f47680b6eb0 0x7f47680d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.954+0000 7f476fa81640 1 -- 172.21.15.81:0/2246817825 >> 172.21.15.81:0/2246817825 conn(0x7f47680b3510 msgr2=0x7f47680b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.954+0000 7f476fa81640 1 -- 172.21.15.81:0/2246817825 shutdown_connections 2024-06-29T14:23:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.954+0000 7f476fa81640 1 -- 172.21.15.81:0/2246817825 wait complete. 2024-06-29T14:23:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.956+0000 7f476fa81640 1 Processor -- start 2024-06-29T14:23:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.956+0000 7f476fa81640 1 -- start start 2024-06-29T14:23:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476fa81640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47680b4810 0x7f476815cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476fa81640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47680b6eb0 0x7f47680c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476fa81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47680ce510 0x7f47680c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476fa81640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f47680cd480 con 0x7f47680ce510 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476fa81640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f47680b29c0 con 0x7f47680b6eb0 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476fa81640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f47680cd600 con 0x7f47680b4810 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476e01e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47680ce510 0x7f47680c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476e01e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47680ce510 0x7f47680c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38010/0 (socket says 172.21.15.81:38010) 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476e01e640 1 -- 172.21.15.81:0/2707998879 learned_addr learned my addr 172.21.15.81:0/2707998879 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476d01c640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47680b6eb0 0x7f47680c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.957+0000 7f476d81d640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47680b4810 0x7f476815cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f476e01e640 1 -- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47680b4810 msgr2=0x7f476815cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f476e01e640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47680b4810 0x7f476815cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f476e01e640 1 -- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47680b6eb0 msgr2=0x7f47680c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f476e01e640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47680b6eb0 0x7f47680c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:08.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f476e01e640 1 -- 172.21.15.81:0/2707998879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476815d480 con 0x7f47680ce510 2024-06-29T14:23:08.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f476e01e640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47680ce510 0x7f47680c9ea0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f476000c730 tx=0x7f476000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:08.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f47667fc640 1 -- 172.21.15.81:0/2707998879 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4760003b70 con 0x7f47680ce510 2024-06-29T14:23:08.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f476fa81640 1 -- 172.21.15.81:0/2707998879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47680c9c90 con 0x7f47680ce510 2024-06-29T14:23:08.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f47667fc640 1 -- 172.21.15.81:0/2707998879 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4760003d10 con 0x7f47680ce510 2024-06-29T14:23:08.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f47667fc640 1 -- 172.21.15.81:0/2707998879 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476000aab0 con 0x7f47680ce510 2024-06-29T14:23:08.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.958+0000 7f476fa81640 1 -- 172.21.15.81:0/2707998879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47680c96d0 con 0x7f47680ce510 2024-06-29T14:23:08.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.959+0000 7f476fa81640 1 -- 172.21.15.81:0/2707998879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47680d02d0 con 0x7f47680ce510 2024-06-29T14:23:08.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.960+0000 7f47667fc640 1 -- 172.21.15.81:0/2707998879 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f476000ac50 con 0x7f47680ce510 2024-06-29T14:23:08.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.960+0000 7f47667fc640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f47540762d0 0x7f47540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:08.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.960+0000 7f47667fc640 1 -- 172.21.15.81:0/2707998879 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4760097600 con 0x7f47680ce510 2024-06-29T14:23:08.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.960+0000 7f476d81d640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f47540762d0 0x7f47540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:08.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.961+0000 7f476d81d640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f47540762d0 0x7f47540787e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f47580095f0 tx=0x7f4758005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:08.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:08.964+0000 7f47667fc640 1 -- 172.21.15.81:0/2707998879 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f476815d480 con 0x7f47680ce510 2024-06-29T14:23:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.146+0000 7f476fa81640 1 -- 172.21.15.81:0/2707998879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47680cdf00 con 0x7f47680ce510 2024-06-29T14:23:09.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.146+0000 7f47667fc640 1 -- 172.21.15.81:0/2707998879 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f4760060800 con 0x7f47680ce510 2024-06-29T14:23:09.145 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.148+0000 7f476fa81640 1 -- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f47540762d0 msgr2=0x7f47540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.148+0000 7f476fa81640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f47540762d0 0x7f47540787e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f47580095f0 tx=0x7f4758005f90 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.148+0000 7f476fa81640 1 -- 172.21.15.81:0/2707998879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47680ce510 msgr2=0x7f47680c9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.148+0000 7f476fa81640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47680ce510 0x7f47680c9ea0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f476000c730 tx=0x7f476000cbf0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.148+0000 7f476fa81640 1 -- 172.21.15.81:0/2707998879 shutdown_connections 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.148+0000 7f476fa81640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f47540762d0 0x7f47540787e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.148+0000 7f476fa81640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47680ce510 0x7f47680c9ea0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.148+0000 7f476fa81640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47680b6eb0 0x7f47680c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.148+0000 7f476fa81640 1 --2- 172.21.15.81:0/2707998879 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47680b4810 0x7f476815cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.148+0000 7f476fa81640 1 -- 172.21.15.81:0/2707998879 >> 172.21.15.81:0/2707998879 conn(0x7f47680b3510 msgr2=0x7f47680b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.149+0000 7f476fa81640 1 -- 172.21.15.81:0/2707998879 shutdown_connections 2024-06-29T14:23:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.149+0000 7f476fa81640 1 -- 172.21.15.81:0/2707998879 wait complete. 2024-06-29T14:23:09.158 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:23:09.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.275+0000 7f4356212640 1 -- 172.21.15.81:0/291196672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43500b8ba0 msgr2=0x7f43500d46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.275+0000 7f4356212640 1 --2- 172.21.15.81:0/291196672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43500b8ba0 0x7f43500d46f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4340005fd0 tx=0x7f434002a5b0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.275+0000 7f4356212640 1 -- 172.21.15.81:0/291196672 shutdown_connections 2024-06-29T14:23:09.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.275+0000 7f4356212640 1 --2- 172.21.15.81:0/291196672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43500b6a20 0x7f43500d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.275+0000 7f4356212640 1 --2- 172.21.15.81:0/291196672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43500b8ba0 0x7f43500d46f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.275+0000 7f4356212640 1 -- 172.21.15.81:0/291196672 >> 172.21.15.81:0/291196672 conn(0x7f43500d3bd0 msgr2=0x7f43500d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.275+0000 7f4356212640 1 -- 172.21.15.81:0/291196672 shutdown_connections 2024-06-29T14:23:09.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.275+0000 7f4356212640 1 -- 172.21.15.81:0/291196672 wait complete. 2024-06-29T14:23:09.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.276+0000 7f4356212640 1 Processor -- start 2024-06-29T14:23:09.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.277+0000 7f4356212640 1 -- start start 2024-06-29T14:23:09.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f4356212640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43500b6a20 0x7f43500c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f4356212640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43500b8ba0 0x7f43500cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f4356212640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43500d0950 0x7f4350151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f4356212640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f43500b9290 con 0x7f43500b8ba0 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f4356212640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f43500b9410 con 0x7f43500d0950 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f4356212640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f43500cfcd0 con 0x7f43500b6a20 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43500b8ba0 0x7f43500cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43500b8ba0 0x7f43500cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38040/0 (socket says 172.21.15.81:38040) 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434effd640 1 -- 172.21.15.81:0/3559391368 learned_addr learned my addr 172.21.15.81:0/3559391368 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434f7fe640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43500b6a20 0x7f43500c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434ffff640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43500d0950 0x7f4350151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434effd640 1 -- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43500b6a20 msgr2=0x7f43500c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434effd640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43500b6a20 0x7f43500c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434effd640 1 -- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43500d0950 msgr2=0x7f4350151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434effd640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43500d0950 0x7f4350151f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434effd640 1 -- 172.21.15.81:0/3559391368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43500cb8a0 con 0x7f43500b8ba0 2024-06-29T14:23:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434ffff640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43500d0950 0x7f4350151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:09.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.278+0000 7f434f7fe640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43500b6a20 0x7f43500c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:09.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.279+0000 7f434effd640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43500b8ba0 0x7f43500cd8c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f434400c9a0 tx=0x7f434400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:09.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.279+0000 7f4337fff640 1 -- 172.21.15.81:0/3559391368 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344007bb0 con 0x7f43500b8ba0 2024-06-29T14:23:09.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.279+0000 7f4337fff640 1 -- 172.21.15.81:0/3559391368 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4344007e40 con 0x7f43500b8ba0 2024-06-29T14:23:09.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.279+0000 7f4356212640 1 -- 172.21.15.81:0/3559391368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43500cbb80 con 0x7f43500b8ba0 2024-06-29T14:23:09.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.279+0000 7f4337fff640 1 -- 172.21.15.81:0/3559391368 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344005280 con 0x7f43500b8ba0 2024-06-29T14:23:09.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.279+0000 7f4356212640 1 -- 172.21.15.81:0/3559391368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43500cc140 con 0x7f43500b8ba0 2024-06-29T14:23:09.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.280+0000 7f4356212640 1 -- 172.21.15.81:0/3559391368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43500d0410 con 0x7f43500b8ba0 2024-06-29T14:23:09.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.280+0000 7f4337fff640 1 -- 172.21.15.81:0/3559391368 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4344005420 con 0x7f43500b8ba0 2024-06-29T14:23:09.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.281+0000 7f4337fff640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f433c0762d0 0x7f433c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.281+0000 7f4337fff640 1 -- 172.21.15.81:0/3559391368 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4344097b80 con 0x7f43500b8ba0 2024-06-29T14:23:09.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.281+0000 7f434f7fe640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f433c0762d0 0x7f433c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.282+0000 7f434f7fe640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f433c0762d0 0x7f433c0787e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4340033000 tx=0x7f4340004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:09.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.285+0000 7f4337fff640 1 -- 172.21.15.81:0/3559391368 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4344014030 con 0x7f43500b8ba0 2024-06-29T14:23:09.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.466+0000 7f4356212640 1 -- 172.21.15.81:0/3559391368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f43500cdea0 con 0x7f43500b8ba0 2024-06-29T14:23:09.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.467+0000 7f4337fff640 1 -- 172.21.15.81:0/3559391368 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f4344061460 con 0x7f43500b8ba0 2024-06-29T14:23:09.465 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:23:09.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 -- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f433c0762d0 msgr2=0x7f433c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f433c0762d0 0x7f433c0787e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4340033000 tx=0x7f4340004730 comp rx=0 tx=0).stop 2024-06-29T14:23:09.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 -- 172.21.15.81:0/3559391368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43500b8ba0 msgr2=0x7f43500cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43500b8ba0 0x7f43500cd8c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f434400c9a0 tx=0x7f434400ce60 comp rx=0 tx=0).stop 2024-06-29T14:23:09.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 -- 172.21.15.81:0/3559391368 shutdown_connections 2024-06-29T14:23:09.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f433c0762d0 0x7f433c0787e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43500d0950 0x7f4350151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43500b8ba0 0x7f43500cd8c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 --2- 172.21.15.81:0/3559391368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43500b6a20 0x7f43500c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 -- 172.21.15.81:0/3559391368 >> 172.21.15.81:0/3559391368 conn(0x7f43500d3bd0 msgr2=0x7f43500b4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 -- 172.21.15.81:0/3559391368 shutdown_connections 2024-06-29T14:23:09.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.469+0000 7f4356212640 1 -- 172.21.15.81:0/3559391368 wait complete. 2024-06-29T14:23:09.478 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:23:09.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.597+0000 7f971dcb9640 1 -- 172.21.15.81:0/26059786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9704003ff0 msgr2=0x7f9704004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.597+0000 7f971dcb9640 1 --2- 172.21.15.81:0/26059786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9704003ff0 0x7f9704004800 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9700009b50 tx=0x7f9704005830 comp rx=0 tx=0).stop 2024-06-29T14:23:09.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.597+0000 7f9717fff640 1 -- 172.21.15.81:0/26059786 reap_dead start 2024-06-29T14:23:09.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.597+0000 7f971dcb9640 1 -- 172.21.15.81:0/26059786 shutdown_connections 2024-06-29T14:23:09.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.597+0000 7f971dcb9640 1 -- 172.21.15.81:0/26059786 >> 172.21.15.81:0/26059786 conn(0x7f97180b3510 msgr2=0x7f97180b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.597+0000 7f971dcb9640 1 -- 172.21.15.81:0/26059786 shutdown_connections 2024-06-29T14:23:09.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.597+0000 7f971dcb9640 1 -- 172.21.15.81:0/26059786 wait complete. 2024-06-29T14:23:09.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.599+0000 7f971dcb9640 1 Processor -- start 2024-06-29T14:23:09.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.599+0000 7f971dcb9640 1 -- start start 2024-06-29T14:23:09.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f971dcb9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9704003ff0 0x7f97180aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f971dcb9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97180b10d0 0x7f97180b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f971dcb9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97180d1960 0x7f97180d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f971dcb9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f97180ce690 con 0x7f97180b10d0 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f971dcb9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f97180ce510 con 0x7f9704003ff0 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f971dcb9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f97180ce810 con 0x7f97180d1960 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f970ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97180b10d0 0x7f97180b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f970ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97180b10d0 0x7f97180b14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38058/0 (socket says 172.21.15.81:38058) 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f970ffff640 1 -- 172.21.15.81:0/3190415473 learned_addr learned my addr 172.21.15.81:0/3190415473 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f97177fe640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9704003ff0 0x7f97180aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.600+0000 7f9717fff640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97180d1960 0x7f97180d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f970ffff640 1 -- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97180d1960 msgr2=0x7f97180d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f970ffff640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97180d1960 0x7f97180d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f970ffff640 1 -- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9704003ff0 msgr2=0x7f97180aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f970ffff640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9704003ff0 0x7f97180aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f970ffff640 1 -- 172.21.15.81:0/3190415473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97180d22b0 con 0x7f97180b10d0 2024-06-29T14:23:09.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f970ffff640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97180b10d0 0x7f97180b14e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9700009a70 tx=0x7f9700033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:09.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f9714ff9640 1 -- 172.21.15.81:0/3190415473 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700038040 con 0x7f97180b10d0 2024-06-29T14:23:09.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f9714ff9640 1 -- 172.21.15.81:0/3190415473 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97000338f0 con 0x7f97180b10d0 2024-06-29T14:23:09.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f971dcb9640 1 -- 172.21.15.81:0/3190415473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97181f5010 con 0x7f97180b10d0 2024-06-29T14:23:09.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f9714ff9640 1 -- 172.21.15.81:0/3190415473 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700033d40 con 0x7f97180b10d0 2024-06-29T14:23:09.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.601+0000 7f971dcb9640 1 -- 172.21.15.81:0/3190415473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97181f55d0 con 0x7f97180b10d0 2024-06-29T14:23:09.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.602+0000 7f971dcb9640 1 -- 172.21.15.81:0/3190415473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f971806fd20 con 0x7f97180b10d0 2024-06-29T14:23:09.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.603+0000 7f9714ff9640 1 -- 172.21.15.81:0/3190415473 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9700043020 con 0x7f97180b10d0 2024-06-29T14:23:09.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.603+0000 7f9714ff9640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f96f8076300 0x7f96f8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.603+0000 7f9714ff9640 1 -- 172.21.15.81:0/3190415473 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f97000b7870 con 0x7f97180b10d0 2024-06-29T14:23:09.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.604+0000 7f97177fe640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f96f8076300 0x7f96f8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.604+0000 7f97177fe640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f96f8076300 0x7f96f8078810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9708000ff0 tx=0x7f9708006040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:09.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.608+0000 7f9714ff9640 1 -- 172.21.15.81:0/3190415473 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f971806fd20 con 0x7f97180b10d0 2024-06-29T14:23:09.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.789+0000 7f971dcb9640 1 -- 172.21.15.81:0/3190415473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f97180b5890 con 0x7f97180b10d0 2024-06-29T14:23:09.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.790+0000 7f9714ff9640 1 -- 172.21.15.81:0/3190415473 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f970004a020 con 0x7f97180b10d0 2024-06-29T14:23:09.788 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:23:09.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 -- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f96f8076300 msgr2=0x7f96f8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f96f8076300 0x7f96f8078810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9708000ff0 tx=0x7f9708006040 comp rx=0 tx=0).stop 2024-06-29T14:23:09.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 -- 172.21.15.81:0/3190415473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97180b10d0 msgr2=0x7f97180b14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97180b10d0 0x7f97180b14e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9700009a70 tx=0x7f9700033670 comp rx=0 tx=0).stop 2024-06-29T14:23:09.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 -- 172.21.15.81:0/3190415473 shutdown_connections 2024-06-29T14:23:09.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f96f8076300 0x7f96f8078810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97180d1960 0x7f97180d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97180b10d0 0x7f97180b14e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 --2- 172.21.15.81:0/3190415473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9704003ff0 0x7f97180aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 -- 172.21.15.81:0/3190415473 >> 172.21.15.81:0/3190415473 conn(0x7f97180b3510 msgr2=0x7f9718070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 -- 172.21.15.81:0/3190415473 shutdown_connections 2024-06-29T14:23:09.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.792+0000 7f971dcb9640 1 -- 172.21.15.81:0/3190415473 wait complete. 2024-06-29T14:23:09.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.918+0000 7f1d75b5b640 1 -- 172.21.15.81:0/2418287304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d58004800 msgr2=0x7f1d58004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.918+0000 7f1d75b5b640 1 --2- 172.21.15.81:0/2418287304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d58004800 0x7f1d58004c10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1d700cac90 tx=0x7f1d58005830 comp rx=0 tx=0).stop 2024-06-29T14:23:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.918+0000 7f1d6ffff640 1 -- 172.21.15.81:0/2418287304 reap_dead start 2024-06-29T14:23:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.918+0000 7f1d75b5b640 1 -- 172.21.15.81:0/2418287304 shutdown_connections 2024-06-29T14:23:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.918+0000 7f1d75b5b640 1 -- 172.21.15.81:0/2418287304 >> 172.21.15.81:0/2418287304 conn(0x7f1d7009cf30 msgr2=0x7f1d700d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.918+0000 7f1d75b5b640 1 -- 172.21.15.81:0/2418287304 shutdown_connections 2024-06-29T14:23:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.918+0000 7f1d75b5b640 1 -- 172.21.15.81:0/2418287304 wait complete. 2024-06-29T14:23:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.919+0000 7f1d75b5b640 1 Processor -- start 2024-06-29T14:23:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.920+0000 7f1d75b5b640 1 -- start start 2024-06-29T14:23:09.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d75b5b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d58004800 0x7f1d701f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d75b5b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d701584d0 0x7f1d701588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d75b5b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7015c990 0x7f1d7015cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d75b5b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1d700b8420 con 0x7f1d7015c990 2024-06-29T14:23:09.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d75b5b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1d700b82a0 con 0x7f1d58004800 2024-06-29T14:23:09.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d75b5b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1d700b85a0 con 0x7f1d701584d0 2024-06-29T14:23:09.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7015c990 0x7f1d7015cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7015c990 0x7f1d7015cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38072/0 (socket says 172.21.15.81:38072) 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6ffff640 1 -- 172.21.15.81:0/354331441 learned_addr learned my addr 172.21.15.81:0/354331441 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6effd640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d701584d0 0x7f1d701588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6f7fe640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d58004800 0x7f1d701f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6ffff640 1 -- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d701584d0 msgr2=0x7f1d701588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6ffff640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d701584d0 0x7f1d701588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6ffff640 1 -- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d58004800 msgr2=0x7f1d701f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6ffff640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d58004800 0x7f1d701f79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6ffff640 1 -- 172.21.15.81:0/354331441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d701e7670 con 0x7f1d7015c990 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d6ffff640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7015c990 0x7f1d7015cda0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d700d4460 tx=0x7f1d68001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.921+0000 7f1d57fff640 1 -- 172.21.15.81:0/354331441 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d68015020 con 0x7f1d7015c990 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.922+0000 7f1d57fff640 1 -- 172.21.15.81:0/354331441 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d680109a0 con 0x7f1d7015c990 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.922+0000 7f1d75b5b640 1 -- 172.21.15.81:0/354331441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d70158e20 con 0x7f1d7015c990 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.922+0000 7f1d57fff640 1 -- 172.21.15.81:0/354331441 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d68010df0 con 0x7f1d7015c990 2024-06-29T14:23:09.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.922+0000 7f1d75b5b640 1 -- 172.21.15.81:0/354331441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d701f4bb0 con 0x7f1d7015c990 2024-06-29T14:23:09.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.923+0000 7f1d75b5b640 1 -- 172.21.15.81:0/354331441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d700d3570 con 0x7f1d7015c990 2024-06-29T14:23:09.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.923+0000 7f1d57fff640 1 -- 172.21.15.81:0/354331441 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1d68018020 con 0x7f1d7015c990 2024-06-29T14:23:09.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.924+0000 7f1d57fff640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d5c0762d0 0x7f1d5c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:09.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.924+0000 7f1d57fff640 1 -- 172.21.15.81:0/354331441 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1d680980e0 con 0x7f1d7015c990 2024-06-29T14:23:09.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.924+0000 7f1d6f7fe640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d5c0762d0 0x7f1d5c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:09.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.924+0000 7f1d6f7fe640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d5c0762d0 0x7f1d5c0787e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d600049e0 tx=0x7f1d60006040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:09.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:09.928+0000 7f1d57fff640 1 -- 172.21.15.81:0/354331441 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1d700d3570 con 0x7f1d7015c990 2024-06-29T14:23:10.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.110+0000 7f1d75b5b640 1 -- 172.21.15.81:0/354331441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d700d0480 con 0x7f1d7015c990 2024-06-29T14:23:10.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.111+0000 7f1d57fff640 1 -- 172.21.15.81:0/354331441 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f1d6801f020 con 0x7f1d7015c990 2024-06-29T14:23:10.110 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:10.110 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:10.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.113+0000 7f1d75b5b640 1 -- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d5c0762d0 msgr2=0x7f1d5c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d5c0762d0 0x7f1d5c0787e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d600049e0 tx=0x7f1d60006040 comp rx=0 tx=0).stop 2024-06-29T14:23:10.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 -- 172.21.15.81:0/354331441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7015c990 msgr2=0x7f1d7015cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7015c990 0x7f1d7015cda0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d700d4460 tx=0x7f1d68001ab0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 -- 172.21.15.81:0/354331441 shutdown_connections 2024-06-29T14:23:10.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d5c0762d0 0x7f1d5c0787e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7015c990 0x7f1d7015cda0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d701584d0 0x7f1d701588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 --2- 172.21.15.81:0/354331441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d58004800 0x7f1d701f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 -- 172.21.15.81:0/354331441 >> 172.21.15.81:0/354331441 conn(0x7f1d7009cf30 msgr2=0x7f1d700b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 -- 172.21.15.81:0/354331441 shutdown_connections 2024-06-29T14:23:10.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.114+0000 7f1d75b5b640 1 -- 172.21.15.81:0/354331441 wait complete. 2024-06-29T14:23:10.123 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:10.124 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:10.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.242+0000 7fed0983c640 1 -- 172.21.15.81:0/1777034591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed040b57b0 msgr2=0x7fed0415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.242+0000 7fed0983c640 1 --2- 172.21.15.81:0/1777034591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed040b57b0 0x7fed0415aed0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009830 tx=0x7fecf402a5c0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.242+0000 7fed0983c640 1 -- 172.21.15.81:0/1777034591 shutdown_connections 2024-06-29T14:23:10.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.242+0000 7fed0983c640 1 --2- 172.21.15.81:0/1777034591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed040b57b0 0x7fed0415aed0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.242+0000 7fed0983c640 1 -- 172.21.15.81:0/1777034591 >> 172.21.15.81:0/1777034591 conn(0x7fed040d0930 msgr2=0x7fed040d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.242+0000 7fed0983c640 1 -- 172.21.15.81:0/1777034591 shutdown_connections 2024-06-29T14:23:10.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.242+0000 7fed0983c640 1 -- 172.21.15.81:0/1777034591 wait complete. 2024-06-29T14:23:10.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.244+0000 7fed0983c640 1 Processor -- start 2024-06-29T14:23:10.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.244+0000 7fed0983c640 1 -- start start 2024-06-29T14:23:10.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.245+0000 7fed0983c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed0406ef90 0x7fed041558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.245+0000 7fed0983c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed040b3960 0x7fed040af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.245+0000 7fed0983c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed040b57b0 0x7fed0406c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.245+0000 7fed0983c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fed040b4600 con 0x7fed0406ef90 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.245+0000 7fed0983c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fed040b4480 con 0x7fed040b3960 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.245+0000 7fed0983c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fed040b4780 con 0x7fed040b57b0 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.245+0000 7fed02ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed0406ef90 0x7fed041558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.245+0000 7fed02ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed0406ef90 0x7fed041558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49516/0 (socket says 172.21.15.81:49516) 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.245+0000 7fed02ffd640 1 -- 172.21.15.81:0/275352238 learned_addr learned my addr 172.21.15.81:0/275352238 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed037fe640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed040b57b0 0x7fed0406c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fecfa7fc640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed040b3960 0x7fed040af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed02ffd640 1 -- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed040b57b0 msgr2=0x7fed0406c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed02ffd640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed040b57b0 0x7fed0406c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed02ffd640 1 -- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed040b3960 msgr2=0x7fed040af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed02ffd640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed040b3960 0x7fed040af030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed02ffd640 1 -- 172.21.15.81:0/275352238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed04155df0 con 0x7fed0406ef90 2024-06-29T14:23:10.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fecfa7fc640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed040b3960 0x7fed040af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:10.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed02ffd640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed0406ef90 0x7fed041558b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fecf402abb0 tx=0x7fecf4033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed00ff9640 1 -- 172.21.15.81:0/275352238 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4031b50 con 0x7fed0406ef90 2024-06-29T14:23:10.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed00ff9640 1 -- 172.21.15.81:0/275352238 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecf4031de0 con 0x7fed0406ef90 2024-06-29T14:23:10.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed0983c640 1 -- 172.21.15.81:0/275352238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed04151f50 con 0x7fed0406ef90 2024-06-29T14:23:10.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed00ff9640 1 -- 172.21.15.81:0/275352238 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4005280 con 0x7fed0406ef90 2024-06-29T14:23:10.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.246+0000 7fed0983c640 1 -- 172.21.15.81:0/275352238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed041f6310 con 0x7fed0406ef90 2024-06-29T14:23:10.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.247+0000 7fed0983c640 1 -- 172.21.15.81:0/275352238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed0406e2a0 con 0x7fed0406ef90 2024-06-29T14:23:10.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.248+0000 7fed00ff9640 1 -- 172.21.15.81:0/275352238 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fecf4005420 con 0x7fed0406ef90 2024-06-29T14:23:10.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.248+0000 7fed00ff9640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fece40762d0 0x7fece40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.248+0000 7fed00ff9640 1 -- 172.21.15.81:0/275352238 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fecf40b8870 con 0x7fed0406ef90 2024-06-29T14:23:10.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.248+0000 7fecfa7fc640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fece40762d0 0x7fece40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.249+0000 7fecfa7fc640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fece40762d0 0x7fece40787e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fed04156f60 tx=0x7fecec009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.253+0000 7fed00ff9640 1 -- 172.21.15.81:0/275352238 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fed0406e2a0 con 0x7fed0406ef90 2024-06-29T14:23:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.431+0000 7fed0983c640 1 -- 172.21.15.81:0/275352238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fed040b4bd0 con 0x7fed0406ef90 2024-06-29T14:23:10.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.431+0000 7fed00ff9640 1 -- 172.21.15.81:0/275352238 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fecf403f030 con 0x7fed0406ef90 2024-06-29T14:23:10.430 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.433+0000 7fed0983c640 1 -- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fece40762d0 msgr2=0x7fece40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.433+0000 7fed0983c640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fece40762d0 0x7fece40787e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fed04156f60 tx=0x7fecec009040 comp rx=0 tx=0).stop 2024-06-29T14:23:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.433+0000 7fed0983c640 1 -- 172.21.15.81:0/275352238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed0406ef90 msgr2=0x7fed041558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.433+0000 7fed0983c640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed0406ef90 0x7fed041558b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fecf402abb0 tx=0x7fecf4033ba0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.433+0000 7fed0983c640 1 -- 172.21.15.81:0/275352238 shutdown_connections 2024-06-29T14:23:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.433+0000 7fed0983c640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fece40762d0 0x7fece40787e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.433+0000 7fed0983c640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed040b57b0 0x7fed0406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.433+0000 7fed0983c640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed040b3960 0x7fed040af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.433+0000 7fed0983c640 1 --2- 172.21.15.81:0/275352238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed0406ef90 0x7fed041558b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.433+0000 7fed0983c640 1 -- 172.21.15.81:0/275352238 >> 172.21.15.81:0/275352238 conn(0x7fed040d0930 msgr2=0x7fed040ce1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.434+0000 7fed0983c640 1 -- 172.21.15.81:0/275352238 shutdown_connections 2024-06-29T14:23:10.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.434+0000 7fed0983c640 1 -- 172.21.15.81:0/275352238 wait complete. 2024-06-29T14:23:10.443 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:23:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.560+0000 7fb33e39e640 1 -- 172.21.15.81:0/2711073789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb324003a00 msgr2=0x7fb324003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.560+0000 7fb33e39e640 1 --2- 172.21.15.81:0/2711073789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb324003a00 0x7fb324003e70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb328009b50 tx=0x7fb324005830 comp rx=0 tx=0).stop 2024-06-29T14:23:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.560+0000 7fb33c93b640 1 -- 172.21.15.81:0/2711073789 reap_dead start 2024-06-29T14:23:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.560+0000 7fb33e39e640 1 -- 172.21.15.81:0/2711073789 shutdown_connections 2024-06-29T14:23:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.560+0000 7fb33e39e640 1 -- 172.21.15.81:0/2711073789 >> 172.21.15.81:0/2711073789 conn(0x7fb33806ccc0 msgr2=0x7fb3380b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.560+0000 7fb33e39e640 1 -- 172.21.15.81:0/2711073789 shutdown_connections 2024-06-29T14:23:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.560+0000 7fb33e39e640 1 -- 172.21.15.81:0/2711073789 wait complete. 2024-06-29T14:23:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.561+0000 7fb33e39e640 1 Processor -- start 2024-06-29T14:23:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.562+0000 7fb33e39e640 1 -- start start 2024-06-29T14:23:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb33e39e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb324003a00 0x7fb3380b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb33e39e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb33813fe20 0x7fb3380aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb33e39e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb338140230 0x7fb3381e9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb33e39e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb338073400 con 0x7fb33813fe20 2024-06-29T14:23:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb33e39e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb338073280 con 0x7fb324003a00 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb33e39e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb338073580 con 0x7fb338140230 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb32ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb33813fe20 0x7fb3380aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb32ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb33813fe20 0x7fb3380aa5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49536/0 (socket says 172.21.15.81:49536) 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb32ffff640 1 -- 172.21.15.81:0/1413902796 learned_addr learned my addr 172.21.15.81:0/1413902796 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb33c93b640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb338140230 0x7fb3381e9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb337fff640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb324003a00 0x7fb3380b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb32ffff640 1 -- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb338140230 msgr2=0x7fb3381e9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb32ffff640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb338140230 0x7fb3381e9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb32ffff640 1 -- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb324003a00 msgr2=0x7fb3380b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb32ffff640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb324003a00 0x7fb3380b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb32ffff640 1 -- 172.21.15.81:0/1413902796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3380b36e0 con 0x7fb33813fe20 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb32ffff640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb33813fe20 0x7fb3380aa5f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb320006e40 tx=0x7fb3200049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb3357fa640 1 -- 172.21.15.81:0/1413902796 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb320015070 con 0x7fb33813fe20 2024-06-29T14:23:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb3357fa640 1 -- 172.21.15.81:0/1413902796 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb320010980 con 0x7fb33813fe20 2024-06-29T14:23:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.563+0000 7fb33e39e640 1 -- 172.21.15.81:0/1413902796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3380b39c0 con 0x7fb33813fe20 2024-06-29T14:23:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.564+0000 7fb3357fa640 1 -- 172.21.15.81:0/1413902796 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb320010dd0 con 0x7fb33813fe20 2024-06-29T14:23:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.564+0000 7fb33e39e640 1 -- 172.21.15.81:0/1413902796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3380b3f80 con 0x7fb33813fe20 2024-06-29T14:23:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.564+0000 7fb33e39e640 1 -- 172.21.15.81:0/1413902796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb33815d510 con 0x7fb33813fe20 2024-06-29T14:23:10.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.565+0000 7fb3357fa640 1 -- 172.21.15.81:0/1413902796 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb320018070 con 0x7fb33813fe20 2024-06-29T14:23:10.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.566+0000 7fb3357fa640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb3180762d0 0x7fb3180787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.566+0000 7fb3357fa640 1 -- 172.21.15.81:0/1413902796 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb320097f30 con 0x7fb33813fe20 2024-06-29T14:23:10.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.566+0000 7fb337fff640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb3180762d0 0x7fb3180787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.566+0000 7fb337fff640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb3180762d0 0x7fb3180787e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb328009b50 tx=0x7fb32802d040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.570+0000 7fb3357fa640 1 -- 172.21.15.81:0/1413902796 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb338073bd0 con 0x7fb33813fe20 2024-06-29T14:23:10.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.751+0000 7fb33e39e640 1 -- 172.21.15.81:0/1413902796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb3380b3470 con 0x7fb33813fe20 2024-06-29T14:23:10.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.751+0000 7fb3357fa640 1 -- 172.21.15.81:0/1413902796 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7fb320061890 con 0x7fb33813fe20 2024-06-29T14:23:10.750 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 -- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb3180762d0 msgr2=0x7fb3180787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb3180762d0 0x7fb3180787e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb328009b50 tx=0x7fb32802d040 comp rx=0 tx=0).stop 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 -- 172.21.15.81:0/1413902796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb33813fe20 msgr2=0x7fb3380aa5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb33813fe20 0x7fb3380aa5f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb320006e40 tx=0x7fb3200049d0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 -- 172.21.15.81:0/1413902796 shutdown_connections 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb3180762d0 0x7fb3180787e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb338140230 0x7fb3381e9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb33813fe20 0x7fb3380aa5f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 --2- 172.21.15.81:0/1413902796 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb324003a00 0x7fb3380b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 -- 172.21.15.81:0/1413902796 >> 172.21.15.81:0/1413902796 conn(0x7fb33806ccc0 msgr2=0x7fb33815a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 -- 172.21.15.81:0/1413902796 shutdown_connections 2024-06-29T14:23:10.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.753+0000 7fb33e39e640 1 -- 172.21.15.81:0/1413902796 wait complete. 2024-06-29T14:23:10.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:23:10.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.881+0000 7fc42f1ac640 1 -- 172.21.15.81:0/4103308745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4280cea50 msgr2=0x7fc4280b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.882+0000 7fc42f1ac640 1 --2- 172.21.15.81:0/4103308745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4280cea50 0x7fc4280b4810 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc420005fd0 tx=0x7fc42002a820 comp rx=0 tx=0).stop 2024-06-29T14:23:10.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.882+0000 7fc42f1ac640 1 -- 172.21.15.81:0/4103308745 shutdown_connections 2024-06-29T14:23:10.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.882+0000 7fc42f1ac640 1 --2- 172.21.15.81:0/4103308745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4280cea50 0x7fc4280b4810 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.882+0000 7fc42f1ac640 1 --2- 172.21.15.81:0/4103308745 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc4280cd480 0x7fc4280ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.882+0000 7fc42f1ac640 1 -- 172.21.15.81:0/4103308745 >> 172.21.15.81:0/4103308745 conn(0x7fc4280b3510 msgr2=0x7fc4280b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:10.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.882+0000 7fc42f1ac640 1 -- 172.21.15.81:0/4103308745 shutdown_connections 2024-06-29T14:23:10.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.882+0000 7fc42f1ac640 1 -- 172.21.15.81:0/4103308745 wait complete. 2024-06-29T14:23:10.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.883+0000 7fc42f1ac640 1 Processor -- start 2024-06-29T14:23:10.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.883+0000 7fc42f1ac640 1 -- start start 2024-06-29T14:23:10.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42f1ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc4280b6eb0 0x7fc42815f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42f1ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc4280cd480 0x7fc428155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42f1ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4280cea50 0x7fc4280b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42f1ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc4280b50e0 con 0x7fc4280cea50 2024-06-29T14:23:10.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42f1ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc4280b4f60 con 0x7fc4280cd480 2024-06-29T14:23:10.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42f1ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc4280b5260 con 0x7fc4280b6eb0 2024-06-29T14:23:10.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42d749640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4280cea50 0x7fc4280b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42d749640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4280cea50 0x7fc4280b0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49552/0 (socket says 172.21.15.81:49552) 2024-06-29T14:23:10.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42d749640 1 -- 172.21.15.81:0/519126873 learned_addr learned my addr 172.21.15.81:0/519126873 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:10.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42cf48640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc4280b6eb0 0x7fc42815f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc427fff640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc4280cd480 0x7fc428155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42d749640 1 -- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc4280b6eb0 msgr2=0x7fc42815f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42d749640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc4280b6eb0 0x7fc42815f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42d749640 1 -- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc4280cd480 msgr2=0x7fc428155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42d749640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc4280cd480 0x7fc428155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42d749640 1 -- 172.21.15.81:0/519126873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4280c8ad0 con 0x7fc4280cea50 2024-06-29T14:23:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.885+0000 7fc42d749640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4280cea50 0x7fc4280b0110 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc420009cd0 tx=0x7fc420032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.886+0000 7fc4257fa640 1 -- 172.21.15.81:0/519126873 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42003f020 con 0x7fc4280cea50 2024-06-29T14:23:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.886+0000 7fc4257fa640 1 -- 172.21.15.81:0/519126873 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc420004040 con 0x7fc4280cea50 2024-06-29T14:23:10.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.886+0000 7fc42f1ac640 1 -- 172.21.15.81:0/519126873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4280b0e90 con 0x7fc4280cea50 2024-06-29T14:23:10.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.886+0000 7fc4257fa640 1 -- 172.21.15.81:0/519126873 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420037040 con 0x7fc4280cea50 2024-06-29T14:23:10.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.886+0000 7fc42f1ac640 1 -- 172.21.15.81:0/519126873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4280b1450 con 0x7fc4280cea50 2024-06-29T14:23:10.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.887+0000 7fc42f1ac640 1 -- 172.21.15.81:0/519126873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4280cdf00 con 0x7fc4280cea50 2024-06-29T14:23:10.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.887+0000 7fc4257fa640 1 -- 172.21.15.81:0/519126873 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc420042020 con 0x7fc4280cea50 2024-06-29T14:23:10.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.888+0000 7fc4257fa640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc414076300 0x7fc414078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:10.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.888+0000 7fc4257fa640 1 -- 172.21.15.81:0/519126873 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc4200b7c30 con 0x7fc4280cea50 2024-06-29T14:23:10.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.888+0000 7fc42cf48640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc414076300 0x7fc414078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:10.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.889+0000 7fc42cf48640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc414076300 0x7fc414078810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc4280cdb60 tx=0x7fc418007eb0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:10.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:10.892+0000 7fc4257fa640 1 -- 172.21.15.81:0/519126873 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc420049030 con 0x7fc4280cea50 2024-06-29T14:23:11.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.072+0000 7fc42f1ac640 1 -- 172.21.15.81:0/519126873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc4280cbaf0 con 0x7fc4280cea50 2024-06-29T14:23:11.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.072+0000 7fc4257fa640 1 -- 172.21.15.81:0/519126873 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fc420080fd0 con 0x7fc4280cea50 2024-06-29T14:23:11.070 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:23:11.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 -- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc414076300 msgr2=0x7fc414078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc414076300 0x7fc414078810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc4280cdb60 tx=0x7fc418007eb0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 -- 172.21.15.81:0/519126873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4280cea50 msgr2=0x7fc4280b0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4280cea50 0x7fc4280b0110 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc420009cd0 tx=0x7fc420032d10 comp rx=0 tx=0).stop 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 -- 172.21.15.81:0/519126873 shutdown_connections 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc414076300 0x7fc414078810 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4280cea50 0x7fc4280b0110 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc4280cd480 0x7fc428155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 --2- 172.21.15.81:0/519126873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc4280b6eb0 0x7fc42815f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 -- 172.21.15.81:0/519126873 >> 172.21.15.81:0/519126873 conn(0x7fc4280b3510 msgr2=0x7fc428070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 -- 172.21.15.81:0/519126873 shutdown_connections 2024-06-29T14:23:11.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.074+0000 7fc42f1ac640 1 -- 172.21.15.81:0/519126873 wait complete. 2024-06-29T14:23:11.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:11.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.199+0000 7f63cfce4640 1 -- 172.21.15.81:0/1242094743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63b4005350 msgr2=0x7f63b400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.199+0000 7f63cfce4640 1 --2- 172.21.15.81:0/1242094743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63b4005350 0x7f63b400b9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f63c000c630 tx=0x7f63b4005830 comp rx=0 tx=0).stop 2024-06-29T14:23:11.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.199+0000 7f63cfce4640 1 -- 172.21.15.81:0/1242094743 shutdown_connections 2024-06-29T14:23:11.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.199+0000 7f63cfce4640 1 --2- 172.21.15.81:0/1242094743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63b4005350 0x7f63b400b9c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.199+0000 7f63cfce4640 1 -- 172.21.15.81:0/1242094743 >> 172.21.15.81:0/1242094743 conn(0x7f63c80b3510 msgr2=0x7f63c80b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:11.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.199+0000 7f63cfce4640 1 -- 172.21.15.81:0/1242094743 shutdown_connections 2024-06-29T14:23:11.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.199+0000 7f63cfce4640 1 -- 172.21.15.81:0/1242094743 wait complete. 2024-06-29T14:23:11.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.201+0000 7f63cfce4640 1 Processor -- start 2024-06-29T14:23:11.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.201+0000 7f63cfce4640 1 -- start start 2024-06-29T14:23:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.202+0000 7f63cfce4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63b4005350 0x7f63c81f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.202+0000 7f63cfce4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c806e530 0x7f63c81dfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.202+0000 7f63cfce4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80c7400 0x7f63c80c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.202+0000 7f63cfce4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f63c80ce8a0 con 0x7f63c80c7400 2024-06-29T14:23:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.202+0000 7f63cfce4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f63c80ce720 con 0x7f63b4005350 2024-06-29T14:23:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.202+0000 7f63cfce4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f63c80d02d0 con 0x7f63c806e530 2024-06-29T14:23:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63ce281640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80c7400 0x7f63c80c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63ce281640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80c7400 0x7f63c80c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49558/0 (socket says 172.21.15.81:49558) 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63ce281640 1 -- 172.21.15.81:0/1986892190 learned_addr learned my addr 172.21.15.81:0/1986892190 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63cda80640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63b4005350 0x7f63c81f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63cd27f640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c806e530 0x7f63c81dfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63ce281640 1 -- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c806e530 msgr2=0x7f63c81dfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63ce281640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c806e530 0x7f63c81dfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63ce281640 1 -- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63b4005350 msgr2=0x7f63c81f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63ce281640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63b4005350 0x7f63c81f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63ce281640 1 -- 172.21.15.81:0/1986892190 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c80c7d50 con 0x7f63c80c7400 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63cd27f640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c806e530 0x7f63c81dfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63ce281640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80c7400 0x7f63c80c7810 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f63c0000f80 tx=0x7f63c0003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63be7fc640 1 -- 172.21.15.81:0/1986892190 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c002b7f0 con 0x7f63c80c7400 2024-06-29T14:23:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63be7fc640 1 -- 172.21.15.81:0/1986892190 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63c002b990 con 0x7f63c80c7400 2024-06-29T14:23:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63cfce4640 1 -- 172.21.15.81:0/1986892190 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c81f57f0 con 0x7f63c80c7400 2024-06-29T14:23:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63be7fc640 1 -- 172.21.15.81:0/1986892190 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c002bde0 con 0x7f63c80c7400 2024-06-29T14:23:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.203+0000 7f63cfce4640 1 -- 172.21.15.81:0/1986892190 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c81f5db0 con 0x7f63c80c7400 2024-06-29T14:23:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.204+0000 7f63cfce4640 1 -- 172.21.15.81:0/1986892190 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c81f5560 con 0x7f63c80c7400 2024-06-29T14:23:11.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.205+0000 7f63be7fc640 1 -- 172.21.15.81:0/1986892190 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f63c003c030 con 0x7f63c80c7400 2024-06-29T14:23:11.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.205+0000 7f63be7fc640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f63ac076300 0x7f63ac078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.205+0000 7f63be7fc640 1 -- 172.21.15.81:0/1986892190 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f63c00b6ea0 con 0x7f63c80c7400 2024-06-29T14:23:11.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.206+0000 7f63cda80640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f63ac076300 0x7f63ac078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.206+0000 7f63cda80640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f63ac076300 0x7f63ac078810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f63c4006680 tx=0x7f63c4006b70 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:11.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.210+0000 7f63be7fc640 1 -- 172.21.15.81:0/1986892190 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f63c80c7d50 con 0x7f63c80c7400 2024-06-29T14:23:11.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.393+0000 7f63cfce4640 1 -- 172.21.15.81:0/1986892190 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63c80d0830 con 0x7f63c80c7400 2024-06-29T14:23:11.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.394+0000 7f63be7fc640 1 -- 172.21.15.81:0/1986892190 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f63c0080800 con 0x7f63c80c7400 2024-06-29T14:23:11.393 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:11.393 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 -- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f63ac076300 msgr2=0x7f63ac078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f63ac076300 0x7f63ac078810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f63c4006680 tx=0x7f63c4006b70 comp rx=0 tx=0).stop 2024-06-29T14:23:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 -- 172.21.15.81:0/1986892190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80c7400 msgr2=0x7f63c80c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80c7400 0x7f63c80c7810 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f63c0000f80 tx=0x7f63c0003ba0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 -- 172.21.15.81:0/1986892190 shutdown_connections 2024-06-29T14:23:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f63ac076300 0x7f63ac078810 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80c7400 0x7f63c80c7810 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c806e530 0x7f63c81dfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 --2- 172.21.15.81:0/1986892190 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63b4005350 0x7f63c81f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 -- 172.21.15.81:0/1986892190 >> 172.21.15.81:0/1986892190 conn(0x7f63c80b3510 msgr2=0x7f63c81e3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 -- 172.21.15.81:0/1986892190 shutdown_connections 2024-06-29T14:23:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.396+0000 7f63cfce4640 1 -- 172.21.15.81:0/1986892190 wait complete. 2024-06-29T14:23:11.406 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:11.406 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:11.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.524+0000 7f309d168640 1 -- 172.21.15.81:0/530563663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f309815cd00 msgr2=0x7f3098166ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.524+0000 7f309d168640 1 --2- 172.21.15.81:0/530563663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f309815cd00 0x7f3098166ac0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3090005fd0 tx=0x7f309002a820 comp rx=0 tx=0).stop 2024-06-29T14:23:11.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.524+0000 7f309d168640 1 -- 172.21.15.81:0/530563663 shutdown_connections 2024-06-29T14:23:11.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.524+0000 7f309d168640 1 --2- 172.21.15.81:0/530563663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f309815cd00 0x7f3098166ac0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.524+0000 7f309d168640 1 -- 172.21.15.81:0/530563663 >> 172.21.15.81:0/530563663 conn(0x7f309806ccc0 msgr2=0x7f30980b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:11.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.524+0000 7f309d168640 1 -- 172.21.15.81:0/530563663 shutdown_connections 2024-06-29T14:23:11.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.524+0000 7f309d168640 1 -- 172.21.15.81:0/530563663 wait complete. 2024-06-29T14:23:11.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.526+0000 7f309d168640 1 Processor -- start 2024-06-29T14:23:11.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.526+0000 7f309d168640 1 -- start start 2024-06-29T14:23:11.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f309d168640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3098145530 0x7f3098159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f309d168640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f309815cd00 0x7f3098137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f309d168640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f309813f4a0 0x7f309813f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f309d168640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3098167180 con 0x7f309815cd00 2024-06-29T14:23:11.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f309d168640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3098167000 con 0x7f3098145530 2024-06-29T14:23:11.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f309d168640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3098167300 con 0x7f309813f4a0 2024-06-29T14:23:11.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f309815cd00 0x7f3098137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f309815cd00 0x7f3098137bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49574/0 (socket says 172.21.15.81:49574) 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308e59c640 1 -- 172.21.15.81:0/2131291838 learned_addr learned my addr 172.21.15.81:0/2131291838 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f309759e640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f309813f4a0 0x7f309813f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f3096d9d640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3098145530 0x7f3098159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308e59c640 1 -- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f309813f4a0 msgr2=0x7f309813f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308e59c640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f309813f4a0 0x7f309813f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308e59c640 1 -- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3098145530 msgr2=0x7f3098159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308e59c640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3098145530 0x7f3098159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308e59c640 1 -- 172.21.15.81:0/2131291838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3098156d80 con 0x7f309815cd00 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f3096d9d640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3098145530 0x7f3098159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f309759e640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f309813f4a0 0x7f309813f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308e59c640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f309815cd00 0x7f3098137bf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f308000c3c0 tx=0x7f308000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:11.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308ffff640 1 -- 172.21.15.81:0/2131291838 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3080017070 con 0x7f309815cd00 2024-06-29T14:23:11.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308ffff640 1 -- 172.21.15.81:0/2131291838 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30800035f0 con 0x7f309815cd00 2024-06-29T14:23:11.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f309d168640 1 -- 172.21.15.81:0/2131291838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3098140660 con 0x7f309815cd00 2024-06-29T14:23:11.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f308ffff640 1 -- 172.21.15.81:0/2131291838 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3080003a80 con 0x7f309815cd00 2024-06-29T14:23:11.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.528+0000 7f309d168640 1 -- 172.21.15.81:0/2131291838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30981433d0 con 0x7f309815cd00 2024-06-29T14:23:11.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.530+0000 7f309d168640 1 -- 172.21.15.81:0/2131291838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30981403b0 con 0x7f309815cd00 2024-06-29T14:23:11.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.530+0000 7f308ffff640 1 -- 172.21.15.81:0/2131291838 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f308001a070 con 0x7f309815cd00 2024-06-29T14:23:11.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.531+0000 7f308ffff640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f30780762d0 0x7f30780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.531+0000 7f308ffff640 1 -- 172.21.15.81:0/2131291838 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3080096ac0 con 0x7f309815cd00 2024-06-29T14:23:11.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.531+0000 7f3096d9d640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f30780762d0 0x7f30780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.531+0000 7f3096d9d640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f30780762d0 0x7f30780787e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f308800a830 tx=0x7f3088008040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:11.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.535+0000 7f308ffff640 1 -- 172.21.15.81:0/2131291838 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f30800604e0 con 0x7f309815cd00 2024-06-29T14:23:11.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.713+0000 7f309d168640 1 -- 172.21.15.81:0/2131291838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f30980b1960 con 0x7f309815cd00 2024-06-29T14:23:11.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.714+0000 7f308ffff640 1 -- 172.21.15.81:0/2131291838 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f308005fde0 con 0x7f309815cd00 2024-06-29T14:23:11.712 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:11.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.715+0000 7f309d168640 1 -- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f30780762d0 msgr2=0x7f30780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.715+0000 7f309d168640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f30780762d0 0x7f30780787e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f308800a830 tx=0x7f3088008040 comp rx=0 tx=0).stop 2024-06-29T14:23:11.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.715+0000 7f309d168640 1 -- 172.21.15.81:0/2131291838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f309815cd00 msgr2=0x7f3098137bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.715+0000 7f309d168640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f309815cd00 0x7f3098137bf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f308000c3c0 tx=0x7f308000c880 comp rx=0 tx=0).stop 2024-06-29T14:23:11.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.716+0000 7f309d168640 1 -- 172.21.15.81:0/2131291838 shutdown_connections 2024-06-29T14:23:11.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.716+0000 7f309d168640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f30780762d0 0x7f30780787e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.716+0000 7f309d168640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f309813f4a0 0x7f309813f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.716+0000 7f309d168640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f309815cd00 0x7f3098137bf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.716+0000 7f309d168640 1 --2- 172.21.15.81:0/2131291838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3098145530 0x7f3098159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.716+0000 7f309d168640 1 -- 172.21.15.81:0/2131291838 >> 172.21.15.81:0/2131291838 conn(0x7f309806ccc0 msgr2=0x7f309815b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:11.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.716+0000 7f309d168640 1 -- 172.21.15.81:0/2131291838 shutdown_connections 2024-06-29T14:23:11.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.716+0000 7f309d168640 1 -- 172.21.15.81:0/2131291838 wait complete. 2024-06-29T14:23:11.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:23:11.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.845+0000 7f875c37b640 1 -- 172.21.15.81:0/1669891105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8740003a50 msgr2=0x7f8740003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.845+0000 7f875c37b640 1 --2- 172.21.15.81:0/1669891105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8740003a50 0x7f8740003ec0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8750009b50 tx=0x7f8740005830 comp rx=0 tx=0).stop 2024-06-29T14:23:11.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.845+0000 7f875c37b640 1 -- 172.21.15.81:0/1669891105 shutdown_connections 2024-06-29T14:23:11.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.845+0000 7f875c37b640 1 --2- 172.21.15.81:0/1669891105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8740003a50 0x7f8740003ec0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.845+0000 7f875c37b640 1 -- 172.21.15.81:0/1669891105 >> 172.21.15.81:0/1669891105 conn(0x7f87540b56d0 msgr2=0x7f87540b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:11.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.845+0000 7f875c37b640 1 -- 172.21.15.81:0/1669891105 shutdown_connections 2024-06-29T14:23:11.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.845+0000 7f875c37b640 1 -- 172.21.15.81:0/1669891105 wait complete. 2024-06-29T14:23:11.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.847+0000 7f875c37b640 1 Processor -- start 2024-06-29T14:23:11.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.847+0000 7f875c37b640 1 -- start start 2024-06-29T14:23:11.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.848+0000 7f875c37b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8740003a50 0x7f87541e4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.848+0000 7f875c37b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f875409d9b0 0x7f87540d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.848+0000 7f875c37b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87541e77b0 0x7f87540d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.848+0000 7f875c37b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87540ceab0 con 0x7f8740003a50 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.848+0000 7f875c37b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f87540ce930 con 0x7f875409d9b0 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.848+0000 7f875c37b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f87540cec30 con 0x7f87541e77b0 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.848+0000 7f875a117640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8740003a50 0x7f87541e4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.848+0000 7f875a117640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8740003a50 0x7f87541e4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49584/0 (socket says 172.21.15.81:49584) 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.848+0000 7f875a117640 1 -- 172.21.15.81:0/4011696691 learned_addr learned my addr 172.21.15.81:0/4011696691 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f875a117640 1 -- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87541e77b0 msgr2=0x7f87540d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f875a117640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87541e77b0 0x7f87540d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f875a117640 1 -- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f875409d9b0 msgr2=0x7f87540d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f875a117640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f875409d9b0 0x7f87540d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f875a117640 1 -- 172.21.15.81:0/4011696691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87540a1b80 con 0x7f8740003a50 2024-06-29T14:23:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f875a117640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8740003a50 0x7f87541e4140 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8750036040 tx=0x7f8750033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:11.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f874affd640 1 -- 172.21.15.81:0/4011696691 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8750038040 con 0x7f8740003a50 2024-06-29T14:23:11.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f874affd640 1 -- 172.21.15.81:0/4011696691 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8750041070 con 0x7f8740003a50 2024-06-29T14:23:11.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f875c37b640 1 -- 172.21.15.81:0/4011696691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f875409e090 con 0x7f8740003a50 2024-06-29T14:23:11.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f874affd640 1 -- 172.21.15.81:0/4011696691 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8750033e10 con 0x7f8740003a50 2024-06-29T14:23:11.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.849+0000 7f875c37b640 1 -- 172.21.15.81:0/4011696691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f875409e650 con 0x7f8740003a50 2024-06-29T14:23:11.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.850+0000 7f875c37b640 1 -- 172.21.15.81:0/4011696691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87540cf190 con 0x7f8740003a50 2024-06-29T14:23:11.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.851+0000 7f874affd640 1 -- 172.21.15.81:0/4011696691 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8750046460 con 0x7f8740003a50 2024-06-29T14:23:11.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.851+0000 7f874affd640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87380762d0 0x7f87380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:11.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.852+0000 7f874affd640 1 -- 172.21.15.81:0/4011696691 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f87500b81e0 con 0x7f8740003a50 2024-06-29T14:23:11.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.852+0000 7f8759916640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87380762d0 0x7f87380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:11.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.852+0000 7f8759916640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87380762d0 0x7f87380787e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8744001240 tx=0x7f87440011d0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:11.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:11.856+0000 7f874affd640 1 -- 172.21.15.81:0/4011696691 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8750081b40 con 0x7f8740003a50 2024-06-29T14:23:12.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.034+0000 7f875c37b640 1 -- 172.21.15.81:0/4011696691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f87540d1390 con 0x7f8740003a50 2024-06-29T14:23:12.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.035+0000 7f874affd640 1 -- 172.21.15.81:0/4011696691 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f8750081440 con 0x7f8740003a50 2024-06-29T14:23:12.033 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:23:12.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.036+0000 7f875c37b640 1 -- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87380762d0 msgr2=0x7f87380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.036+0000 7f875c37b640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87380762d0 0x7f87380787e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8744001240 tx=0x7f87440011d0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.036+0000 7f875c37b640 1 -- 172.21.15.81:0/4011696691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8740003a50 msgr2=0x7f87541e4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.036+0000 7f875c37b640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8740003a50 0x7f87541e4140 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8750036040 tx=0x7f8750033670 comp rx=0 tx=0).stop 2024-06-29T14:23:12.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.037+0000 7f875c37b640 1 -- 172.21.15.81:0/4011696691 shutdown_connections 2024-06-29T14:23:12.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.037+0000 7f875c37b640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f87380762d0 0x7f87380787e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.037+0000 7f875c37b640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87541e77b0 0x7f87540d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.037+0000 7f875c37b640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f875409d9b0 0x7f87540d05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.037+0000 7f875c37b640 1 --2- 172.21.15.81:0/4011696691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8740003a50 0x7f87541e4140 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.037+0000 7f875c37b640 1 -- 172.21.15.81:0/4011696691 >> 172.21.15.81:0/4011696691 conn(0x7f87540b56d0 msgr2=0x7f87540d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.037+0000 7f875c37b640 1 -- 172.21.15.81:0/4011696691 shutdown_connections 2024-06-29T14:23:12.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.037+0000 7f875c37b640 1 -- 172.21.15.81:0/4011696691 wait complete. 2024-06-29T14:23:12.046 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:23:12.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.164+0000 7f0981822640 1 -- 172.21.15.81:0/1848233341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f097c0b5250 msgr2=0x7f097c0d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.164+0000 7f0981822640 1 --2- 172.21.15.81:0/1848233341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f097c0b5250 0x7f097c0d1fc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0974005fd0 tx=0x7f097402a940 comp rx=0 tx=0).stop 2024-06-29T14:23:12.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.164+0000 7f0981822640 1 -- 172.21.15.81:0/1848233341 shutdown_connections 2024-06-29T14:23:12.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.164+0000 7f0981822640 1 --2- 172.21.15.81:0/1848233341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f097c0b5250 0x7f097c0d1fc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.164+0000 7f0981822640 1 --2- 172.21.15.81:0/1848233341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f097c0d0b90 0x7f097c0d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.164+0000 7f0981822640 1 -- 172.21.15.81:0/1848233341 >> 172.21.15.81:0/1848233341 conn(0x7f097c09cf30 msgr2=0x7f097c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.164+0000 7f0981822640 1 -- 172.21.15.81:0/1848233341 shutdown_connections 2024-06-29T14:23:12.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.164+0000 7f0981822640 1 -- 172.21.15.81:0/1848233341 wait complete. 2024-06-29T14:23:12.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.166+0000 7f0981822640 1 Processor -- start 2024-06-29T14:23:12.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.166+0000 7f0981822640 1 -- start start 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f0981822640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f097c0b5250 0x7f097c09e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f0981822640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f097c0b9040 0x7f097c0c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f0981822640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f097c09d9a0 0x7f097c0ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f0981822640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f097c0d2680 con 0x7f097c09d9a0 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f0981822640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f097c0d2500 con 0x7f097c0b5250 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f0981822640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f097c0d2800 con 0x7f097c0b9040 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f097c09d9a0 0x7f097c0ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f097c09d9a0 0x7f097c0ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49596/0 (socket says 172.21.15.81:49596) 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097b7fe640 1 -- 172.21.15.81:0/1779556275 learned_addr learned my addr 172.21.15.81:0/1779556275 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097a7fc640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f097c0b9040 0x7f097c0c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097affd640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f097c0b5250 0x7f097c09e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097b7fe640 1 -- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f097c0b9040 msgr2=0x7f097c0c9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097b7fe640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f097c0b9040 0x7f097c0c9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097b7fe640 1 -- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f097c0b5250 msgr2=0x7f097c09e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097b7fe640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f097c0b5250 0x7f097c09e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097b7fe640 1 -- 172.21.15.81:0/1779556275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c0ca440 con 0x7f097c09d9a0 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f097b7fe640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f097c09d9a0 0x7f097c0ccfd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0974009ca0 tx=0x7f0974031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.168+0000 7f0963fff640 1 -- 172.21.15.81:0/1779556275 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097403e070 con 0x7f097c09d9a0 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.169+0000 7f0963fff640 1 -- 172.21.15.81:0/1779556275 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0974004540 con 0x7f097c09d9a0 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.169+0000 7f0981822640 1 -- 172.21.15.81:0/1779556275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097c09f3c0 con 0x7f097c09d9a0 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.169+0000 7f0963fff640 1 -- 172.21.15.81:0/1779556275 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974030430 con 0x7f097c09d9a0 2024-06-29T14:23:12.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.169+0000 7f0981822640 1 -- 172.21.15.81:0/1779556275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f097c09f660 con 0x7f097c09d9a0 2024-06-29T14:23:12.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.169+0000 7f0981822640 1 -- 172.21.15.81:0/1779556275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f097c0d2d90 con 0x7f097c09d9a0 2024-06-29T14:23:12.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.170+0000 7f0963fff640 1 -- 172.21.15.81:0/1779556275 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f097403a440 con 0x7f097c09d9a0 2024-06-29T14:23:12.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.171+0000 7f0963fff640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f09680762d0 0x7f09680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.171+0000 7f0963fff640 1 -- 172.21.15.81:0/1779556275 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f09740b8fe0 con 0x7f097c09d9a0 2024-06-29T14:23:12.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.171+0000 7f097affd640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f09680762d0 0x7f09680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.172+0000 7f097affd640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f09680762d0 0x7f09680787e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f096c007fd0 tx=0x7f096c001370 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.175+0000 7f0963fff640 1 -- 172.21.15.81:0/1779556275 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0974082380 con 0x7f097c09d9a0 2024-06-29T14:23:12.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.353+0000 7f0981822640 1 -- 172.21.15.81:0/1779556275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f097c1f7b30 con 0x7f097c09d9a0 2024-06-29T14:23:12.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.354+0000 7f0963fff640 1 -- 172.21.15.81:0/1779556275 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f0974086680 con 0x7f097c09d9a0 2024-06-29T14:23:12.352 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:23:12.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 -- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f09680762d0 msgr2=0x7f09680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f09680762d0 0x7f09680787e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f096c007fd0 tx=0x7f096c001370 comp rx=0 tx=0).stop 2024-06-29T14:23:12.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 -- 172.21.15.81:0/1779556275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f097c09d9a0 msgr2=0x7f097c0ccfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f097c09d9a0 0x7f097c0ccfd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0974009ca0 tx=0x7f0974031ea0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 -- 172.21.15.81:0/1779556275 shutdown_connections 2024-06-29T14:23:12.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f09680762d0 0x7f09680787e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f097c09d9a0 0x7f097c0ccfd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f097c0b9040 0x7f097c0c9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 --2- 172.21.15.81:0/1779556275 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f097c0b5250 0x7f097c09e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 -- 172.21.15.81:0/1779556275 >> 172.21.15.81:0/1779556275 conn(0x7f097c09cf30 msgr2=0x7f097c15de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 -- 172.21.15.81:0/1779556275 shutdown_connections 2024-06-29T14:23:12.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.356+0000 7f0981822640 1 -- 172.21.15.81:0/1779556275 wait complete. 2024-06-29T14:23:12.382 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2024-06-29T14:23:12.382 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:23:12.383 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:12.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.497+0000 7f6622289640 1 -- 172.21.15.81:0/1773579887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f661c0ce920 msgr2=0x7f661c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.497+0000 7f6622289640 1 --2- 172.21.15.81:0/1773579887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f661c0ce920 0x7f661c0cd480 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6614005fd0 tx=0x7f661402a810 comp rx=0 tx=0).stop 2024-06-29T14:23:12.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.497+0000 7f6622289640 1 -- 172.21.15.81:0/1773579887 shutdown_connections 2024-06-29T14:23:12.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.497+0000 7f6622289640 1 --2- 172.21.15.81:0/1773579887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f661c0ce920 0x7f661c0cd480 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.497+0000 7f6622289640 1 --2- 172.21.15.81:0/1773579887 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f661c0b6eb0 0x7f661c0d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.497+0000 7f6622289640 1 -- 172.21.15.81:0/1773579887 >> 172.21.15.81:0/1773579887 conn(0x7f661c0b3510 msgr2=0x7f661c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.498+0000 7f6622289640 1 -- 172.21.15.81:0/1773579887 shutdown_connections 2024-06-29T14:23:12.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.498+0000 7f6622289640 1 -- 172.21.15.81:0/1773579887 wait complete. 2024-06-29T14:23:12.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.499+0000 7f6622289640 1 Processor -- start 2024-06-29T14:23:12.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.499+0000 7f6622289640 1 -- start start 2024-06-29T14:23:12.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6622289640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f661c0b6eb0 0x7f661c160e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6622289640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f661c0ce510 0x7f661c14e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6622289640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f661c0ce920 0x7f661c0af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6622289640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f661c0b4f20 con 0x7f661c0ce510 2024-06-29T14:23:12.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6622289640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f661c0b29c0 con 0x7f661c0ce920 2024-06-29T14:23:12.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6622289640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f661c0b50a0 con 0x7f661c0b6eb0 2024-06-29T14:23:12.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f661c0ce510 0x7f661c14e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f661c0ce510 0x7f661c14e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49606/0 (socket says 172.21.15.81:49606) 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661b7fe640 1 -- 172.21.15.81:0/905451460 learned_addr learned my addr 172.21.15.81:0/905451460 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661bfff640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f661c0b6eb0 0x7f661c160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661b7fe640 1 -- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f661c0b6eb0 msgr2=0x7f661c160e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661b7fe640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f661c0b6eb0 0x7f661c160e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661b7fe640 1 -- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f661c0ce920 msgr2=0x7f661c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6620826640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f661c0ce920 0x7f661c0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661b7fe640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f661c0ce920 0x7f661c0af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661b7fe640 1 -- 172.21.15.81:0/905451460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661c161790 con 0x7f661c0ce510 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661bfff640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f661c0b6eb0 0x7f661c160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6620826640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f661c0ce920 0x7f661c0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f661b7fe640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f661c0ce510 0x7f661c14e160 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f661000c3f0 tx=0x7f661000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6618ff9640 1 -- 172.21.15.81:0/905451460 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610017070 con 0x7f661c0ce510 2024-06-29T14:23:12.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6618ff9640 1 -- 172.21.15.81:0/905451460 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66100035f0 con 0x7f661c0ce510 2024-06-29T14:23:12.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.501+0000 7f6618ff9640 1 -- 172.21.15.81:0/905451460 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610003a40 con 0x7f661c0ce510 2024-06-29T14:23:12.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.502+0000 7f6622289640 1 -- 172.21.15.81:0/905451460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661c161a70 con 0x7f661c0ce510 2024-06-29T14:23:12.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.502+0000 7f6622289640 1 -- 172.21.15.81:0/905451460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f661c162030 con 0x7f661c0ce510 2024-06-29T14:23:12.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.502+0000 7f6622289640 1 -- 172.21.15.81:0/905451460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f661c0d02d0 con 0x7f661c0ce510 2024-06-29T14:23:12.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.503+0000 7f6618ff9640 1 -- 172.21.15.81:0/905451460 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f661001a070 con 0x7f661c0ce510 2024-06-29T14:23:12.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.504+0000 7f6618ff9640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66080762d0 0x7f66080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.504+0000 7f6618ff9640 1 -- 172.21.15.81:0/905451460 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6610096ba0 con 0x7f661c0ce510 2024-06-29T14:23:12.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.504+0000 7f661bfff640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66080762d0 0x7f66080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.504+0000 7f661bfff640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66080762d0 0x7f66080787e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f660c009aa0 tx=0x7f660c005db0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.508+0000 7f6618ff9640 1 -- 172.21.15.81:0/905451460 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6610060470 con 0x7f661c0ce510 2024-06-29T14:23:12.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.687+0000 7f6622289640 1 -- 172.21.15.81:0/905451460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f661c0b19e0 con 0x7f661c0ce510 2024-06-29T14:23:12.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.688+0000 7f6618ff9640 1 -- 172.21.15.81:0/905451460 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f661005fd70 con 0x7f661c0ce510 2024-06-29T14:23:12.686 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:12.687 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 -- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66080762d0 msgr2=0x7f66080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66080762d0 0x7f66080787e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f660c009aa0 tx=0x7f660c005db0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 -- 172.21.15.81:0/905451460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f661c0ce510 msgr2=0x7f661c14e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f661c0ce510 0x7f661c14e160 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f661000c3f0 tx=0x7f661000c8b0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 -- 172.21.15.81:0/905451460 shutdown_connections 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66080762d0 0x7f66080787e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f661c0ce920 0x7f661c0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f661c0ce510 0x7f661c14e160 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 --2- 172.21.15.81:0/905451460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f661c0b6eb0 0x7f661c160e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 -- 172.21.15.81:0/905451460 >> 172.21.15.81:0/905451460 conn(0x7f661c0b3510 msgr2=0x7f661c0cced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 -- 172.21.15.81:0/905451460 shutdown_connections 2024-06-29T14:23:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.690+0000 7f6622289640 1 -- 172.21.15.81:0/905451460 wait complete. 2024-06-29T14:23:12.700 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:12.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:12.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.818+0000 7f5b86638640 1 -- 172.21.15.81:0/1273104041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b68004d40 msgr2=0x7f5b68005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.818+0000 7f5b86638640 1 --2- 172.21.15.81:0/1273104041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b68004d40 0x7f5b68005150 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5b78005fd0 tx=0x7f5b68005830 comp rx=0 tx=0).stop 2024-06-29T14:23:12.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.818+0000 7f5b84bd5640 1 -- 172.21.15.81:0/1273104041 reap_dead start 2024-06-29T14:23:12.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.818+0000 7f5b86638640 1 -- 172.21.15.81:0/1273104041 shutdown_connections 2024-06-29T14:23:12.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.818+0000 7f5b86638640 1 -- 172.21.15.81:0/1273104041 >> 172.21.15.81:0/1273104041 conn(0x7f5b800d1960 msgr2=0x7f5b800d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:12.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.818+0000 7f5b86638640 1 -- 172.21.15.81:0/1273104041 shutdown_connections 2024-06-29T14:23:12.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.818+0000 7f5b86638640 1 -- 172.21.15.81:0/1273104041 wait complete. 2024-06-29T14:23:12.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.820+0000 7f5b86638640 1 Processor -- start 2024-06-29T14:23:12.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.820+0000 7f5b86638640 1 -- start start 2024-06-29T14:23:12.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.821+0000 7f5b86638640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b68004d40 0x7f5b801f5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.821+0000 7f5b86638640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800cb0c0 0x7f5b800c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.821+0000 7f5b86638640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b800c5970 0x7f5b801e02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.821+0000 7f5b86638640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5b800b47b0 con 0x7f5b800cb0c0 2024-06-29T14:23:12.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.821+0000 7f5b86638640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5b800b2790 con 0x7f5b800c5970 2024-06-29T14:23:12.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b86638640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5b800b4930 con 0x7f5b68004d40 2024-06-29T14:23:12.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800cb0c0 0x7f5b800c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800cb0c0 0x7f5b800c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49622/0 (socket says 172.21.15.81:49622) 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7f7fe640 1 -- 172.21.15.81:0/3861962735 learned_addr learned my addr 172.21.15.81:0/3861962735 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7ffff640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b68004d40 0x7f5b801f5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b84bd5640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b800c5970 0x7f5b801e02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7f7fe640 1 -- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b68004d40 msgr2=0x7f5b801f5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7f7fe640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b68004d40 0x7f5b801f5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7f7fe640 1 -- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b800c5970 msgr2=0x7f5b801e02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7f7fe640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b800c5970 0x7f5b801e02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7f7fe640 1 -- 172.21.15.81:0/3861962735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b800c5d80 con 0x7f5b800cb0c0 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b84bd5640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b800c5970 0x7f5b801e02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7f7fe640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800cb0c0 0x7f5b800c5430 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5b74011ae0 tx=0x7f5b74011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7cff9640 1 -- 172.21.15.81:0/3861962735 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74004460 con 0x7f5b800cb0c0 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7cff9640 1 -- 172.21.15.81:0/3861962735 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b7400f940 con 0x7f5b800cb0c0 2024-06-29T14:23:12.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b86638640 1 -- 172.21.15.81:0/3861962735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b800c9420 con 0x7f5b800cb0c0 2024-06-29T14:23:12.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b7cff9640 1 -- 172.21.15.81:0/3861962735 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7400fd70 con 0x7f5b800cb0c0 2024-06-29T14:23:12.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.822+0000 7f5b86638640 1 -- 172.21.15.81:0/3861962735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b800c9c00 con 0x7f5b800cb0c0 2024-06-29T14:23:12.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.823+0000 7f5b86638640 1 -- 172.21.15.81:0/3861962735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b800d07f0 con 0x7f5b800cb0c0 2024-06-29T14:23:12.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.824+0000 7f5b7cff9640 1 -- 172.21.15.81:0/3861962735 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5b740035f0 con 0x7f5b800cb0c0 2024-06-29T14:23:12.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.825+0000 7f5b7cff9640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5b6c0762d0 0x7f5b6c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.825+0000 7f5b7cff9640 1 -- 172.21.15.81:0/3861962735 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5b74098700 con 0x7f5b800cb0c0 2024-06-29T14:23:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.825+0000 7f5b7ffff640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5b6c0762d0 0x7f5b6c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.825+0000 7f5b7ffff640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5b6c0762d0 0x7f5b6c0787e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5b800c5220 tx=0x7f5b70000ed0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:12.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:12.829+0000 7f5b7cff9640 1 -- 172.21.15.81:0/3861962735 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5b800c5d80 con 0x7f5b800cb0c0 2024-06-29T14:23:13.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.008+0000 7f5b86638640 1 -- 172.21.15.81:0/3861962735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5b8006a5d0 con 0x7f5b800cb0c0 2024-06-29T14:23:13.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.008+0000 7f5b7cff9640 1 -- 172.21.15.81:0/3861962735 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f5b74062060 con 0x7f5b800cb0c0 2024-06-29T14:23:13.007 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:13.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.010+0000 7f5b86638640 1 -- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5b6c0762d0 msgr2=0x7f5b6c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.010+0000 7f5b86638640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5b6c0762d0 0x7f5b6c0787e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5b800c5220 tx=0x7f5b70000ed0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.010+0000 7f5b86638640 1 -- 172.21.15.81:0/3861962735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800cb0c0 msgr2=0x7f5b800c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.010+0000 7f5b86638640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800cb0c0 0x7f5b800c5430 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5b74011ae0 tx=0x7f5b74011ee0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.010+0000 7f5b86638640 1 -- 172.21.15.81:0/3861962735 shutdown_connections 2024-06-29T14:23:13.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.010+0000 7f5b86638640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5b6c0762d0 0x7f5b6c0787e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.010+0000 7f5b86638640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b800c5970 0x7f5b801e02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.010+0000 7f5b86638640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800cb0c0 0x7f5b800c5430 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.010+0000 7f5b86638640 1 --2- 172.21.15.81:0/3861962735 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b68004d40 0x7f5b801f5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.010+0000 7f5b86638640 1 -- 172.21.15.81:0/3861962735 >> 172.21.15.81:0/3861962735 conn(0x7f5b800d1960 msgr2=0x7f5b800b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:13.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.011+0000 7f5b86638640 1 -- 172.21.15.81:0/3861962735 shutdown_connections 2024-06-29T14:23:13.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.011+0000 7f5b86638640 1 -- 172.21.15.81:0/3861962735 wait complete. 2024-06-29T14:23:13.020 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:23:13.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.138+0000 7f38ea29d640 1 -- 172.21.15.81:0/4181157402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38cc004800 msgr2=0x7f38cc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.138+0000 7f38ea29d640 1 --2- 172.21.15.81:0/4181157402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38cc004800 0x7f38cc004c10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f38d80016c0 tx=0x7f38cc005830 comp rx=0 tx=0).stop 2024-06-29T14:23:13.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.138+0000 7f38ea29d640 1 -- 172.21.15.81:0/4181157402 shutdown_connections 2024-06-29T14:23:13.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.138+0000 7f38ea29d640 1 --2- 172.21.15.81:0/4181157402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38cc004800 0x7f38cc004c10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.138+0000 7f38ea29d640 1 -- 172.21.15.81:0/4181157402 >> 172.21.15.81:0/4181157402 conn(0x7f38e40b56d0 msgr2=0x7f38e40b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:13.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.138+0000 7f38ea29d640 1 -- 172.21.15.81:0/4181157402 shutdown_connections 2024-06-29T14:23:13.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.138+0000 7f38ea29d640 1 -- 172.21.15.81:0/4181157402 wait complete. 2024-06-29T14:23:13.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.140+0000 7f38ea29d640 1 Processor -- start 2024-06-29T14:23:13.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.140+0000 7f38ea29d640 1 -- start start 2024-06-29T14:23:13.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38ea29d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38cc004800 0x7f38e40a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38ea29d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38e409f800 0x7f38e41dc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38ea29d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38e41ddaa0 0x7f38e41ddeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38ea29d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f38e40cecc0 con 0x7f38cc004800 2024-06-29T14:23:13.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38ea29d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f38e40ceb40 con 0x7f38e409f800 2024-06-29T14:23:13.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38ea29d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f38e40cee40 con 0x7f38e41ddaa0 2024-06-29T14:23:13.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38cc004800 0x7f38e40a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38cc004800 0x7f38e40a1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49640/0 (socket says 172.21.15.81:49640) 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e3fff640 1 -- 172.21.15.81:0/3858325708 learned_addr learned my addr 172.21.15.81:0/3858325708 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e883a640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38e41ddaa0 0x7f38e41ddeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e37fe640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38e409f800 0x7f38e41dc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e3fff640 1 -- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38e41ddaa0 msgr2=0x7f38e41ddeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e3fff640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38e41ddaa0 0x7f38e41ddeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e3fff640 1 -- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38e409f800 msgr2=0x7f38e41dc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e3fff640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38e409f800 0x7f38e41dc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e3fff640 1 -- 172.21.15.81:0/3858325708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e40a02d0 con 0x7f38cc004800 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e37fe640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38e409f800 0x7f38e41dc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e3fff640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38cc004800 0x7f38e40a1370 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f38d4001eb0 tx=0x7f38d400ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e0ff9640 1 -- 172.21.15.81:0/3858325708 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4015070 con 0x7f38cc004800 2024-06-29T14:23:13.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.142+0000 7f38e0ff9640 1 -- 172.21.15.81:0/3858325708 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38d400ca00 con 0x7f38cc004800 2024-06-29T14:23:13.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.143+0000 7f38ea29d640 1 -- 172.21.15.81:0/3858325708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e40a05b0 con 0x7f38cc004800 2024-06-29T14:23:13.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.143+0000 7f38e0ff9640 1 -- 172.21.15.81:0/3858325708 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d400ce30 con 0x7f38cc004800 2024-06-29T14:23:13.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.143+0000 7f38ea29d640 1 -- 172.21.15.81:0/3858325708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e40a0b70 con 0x7f38cc004800 2024-06-29T14:23:13.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.143+0000 7f38ea29d640 1 -- 172.21.15.81:0/3858325708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38e40d19e0 con 0x7f38cc004800 2024-06-29T14:23:13.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.144+0000 7f38e0ff9640 1 -- 172.21.15.81:0/3858325708 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f38d4018070 con 0x7f38cc004800 2024-06-29T14:23:13.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.145+0000 7f38e0ff9640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f38d00762d0 0x7f38d00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.145+0000 7f38e0ff9640 1 -- 172.21.15.81:0/3858325708 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f38d4097a70 con 0x7f38cc004800 2024-06-29T14:23:13.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.145+0000 7f38e37fe640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f38d00762d0 0x7f38d00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.145+0000 7f38e37fe640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f38d00762d0 0x7f38d00787e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38d8032000 tx=0x7f38d8001190 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:13.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.149+0000 7f38e0ff9640 1 -- 172.21.15.81:0/3858325708 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f38d4060cf0 con 0x7f38cc004800 2024-06-29T14:23:13.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.328+0000 7f38ea29d640 1 -- 172.21.15.81:0/3858325708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f38e40b54b0 con 0x7f38cc004800 2024-06-29T14:23:13.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.328+0000 7f38e0ff9640 1 -- 172.21.15.81:0/3858325708 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f38d4064ff0 con 0x7f38cc004800 2024-06-29T14:23:13.327 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:23:13.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 -- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f38d00762d0 msgr2=0x7f38d00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f38d00762d0 0x7f38d00787e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38d8032000 tx=0x7f38d8001190 comp rx=0 tx=0).stop 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 -- 172.21.15.81:0/3858325708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38cc004800 msgr2=0x7f38e40a1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38cc004800 0x7f38e40a1370 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f38d4001eb0 tx=0x7f38d400ea70 comp rx=0 tx=0).stop 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 -- 172.21.15.81:0/3858325708 shutdown_connections 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f38d00762d0 0x7f38d00787e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38e41ddaa0 0x7f38e41ddeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38e409f800 0x7f38e41dc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 --2- 172.21.15.81:0/3858325708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38cc004800 0x7f38e40a1370 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 -- 172.21.15.81:0/3858325708 >> 172.21.15.81:0/3858325708 conn(0x7f38e40b56d0 msgr2=0x7f38e40b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 -- 172.21.15.81:0/3858325708 shutdown_connections 2024-06-29T14:23:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.330+0000 7f38ea29d640 1 -- 172.21.15.81:0/3858325708 wait complete. 2024-06-29T14:23:13.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:23:13.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.459+0000 7fed808c3640 1 -- 172.21.15.81:0/3061835165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed680040d0 msgr2=0x7fed68004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.459+0000 7fed808c3640 1 --2- 172.21.15.81:0/3061835165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed680040d0 0x7fed68004800 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fed6400baf0 tx=0x7fed68005830 comp rx=0 tx=0).stop 2024-06-29T14:23:13.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.459+0000 7fed7ad9d640 1 -- 172.21.15.81:0/3061835165 reap_dead start 2024-06-29T14:23:13.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.459+0000 7fed808c3640 1 -- 172.21.15.81:0/3061835165 shutdown_connections 2024-06-29T14:23:13.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.459+0000 7fed808c3640 1 -- 172.21.15.81:0/3061835165 >> 172.21.15.81:0/3061835165 conn(0x7fed7c0d1960 msgr2=0x7fed7c0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:13.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.459+0000 7fed808c3640 1 -- 172.21.15.81:0/3061835165 shutdown_connections 2024-06-29T14:23:13.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.459+0000 7fed808c3640 1 -- 172.21.15.81:0/3061835165 wait complete. 2024-06-29T14:23:13.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.462+0000 7fed808c3640 1 Processor -- start 2024-06-29T14:23:13.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.462+0000 7fed808c3640 1 -- start start 2024-06-29T14:23:13.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed808c3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed680040d0 0x7fed7c1e1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed808c3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed7c1e0c20 0x7fed7c1e7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed808c3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed7c0b5220 0x7fed7c0b5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed808c3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c0d0980 con 0x7fed7c0b5220 2024-06-29T14:23:13.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed808c3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c0d0800 con 0x7fed680040d0 2024-06-29T14:23:13.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed808c3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fed7c0b47b0 con 0x7fed7c1e0c20 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed7ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed7c0b5220 0x7fed7c0b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed7ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed7c0b5220 0x7fed7c0b5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49658/0 (socket says 172.21.15.81:49658) 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed7ad9d640 1 -- 172.21.15.81:0/1738907698 learned_addr learned my addr 172.21.15.81:0/1738907698 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed71d9b640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed7c1e0c20 0x7fed7c1e7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed7a59c640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed680040d0 0x7fed7c1e1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed7ad9d640 1 -- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed7c1e0c20 msgr2=0x7fed7c1e7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed7ad9d640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed7c1e0c20 0x7fed7c1e7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed7ad9d640 1 -- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed680040d0 msgr2=0x7fed7c1e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed7ad9d640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed680040d0 0x7fed7c1e1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed7ad9d640 1 -- 172.21.15.81:0/1738907698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7c0b5b70 con 0x7fed7c0b5220 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed7ad9d640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed7c0b5220 0x7fed7c0b5630 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fed74001330 tx=0x7fed7400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.464+0000 7fed737fe640 1 -- 172.21.15.81:0/1738907698 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7400ca10 con 0x7fed7c0b5220 2024-06-29T14:23:13.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.465+0000 7fed737fe640 1 -- 172.21.15.81:0/1738907698 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed74017070 con 0x7fed7c0b5220 2024-06-29T14:23:13.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.465+0000 7fed808c3640 1 -- 172.21.15.81:0/1738907698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed7c0cbc80 con 0x7fed7c0b5220 2024-06-29T14:23:13.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.465+0000 7fed737fe640 1 -- 172.21.15.81:0/1738907698 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed74011460 con 0x7fed7c0b5220 2024-06-29T14:23:13.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.465+0000 7fed808c3640 1 -- 172.21.15.81:0/1738907698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed7c1e26c0 con 0x7fed7c0b5220 2024-06-29T14:23:13.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.466+0000 7fed808c3640 1 -- 172.21.15.81:0/1738907698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed7c0d00f0 con 0x7fed7c0b5220 2024-06-29T14:23:13.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.466+0000 7fed737fe640 1 -- 172.21.15.81:0/1738907698 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fed74011600 con 0x7fed7c0b5220 2024-06-29T14:23:13.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.467+0000 7fed737fe640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fed5c076300 0x7fed5c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.467+0000 7fed737fe640 1 -- 172.21.15.81:0/1738907698 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fed74097aa0 con 0x7fed7c0b5220 2024-06-29T14:23:13.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.467+0000 7fed7a59c640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fed5c076300 0x7fed5c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.467+0000 7fed7a59c640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fed5c076300 0x7fed5c078810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fed6c000c00 tx=0x7fed6c006b70 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:13.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.471+0000 7fed737fe640 1 -- 172.21.15.81:0/1738907698 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fed7c0b5b70 con 0x7fed7c0b5220 2024-06-29T14:23:13.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.652+0000 7fed808c3640 1 -- 172.21.15.81:0/1738907698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fed7c1e2170 con 0x7fed7c0b5220 2024-06-29T14:23:13.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.652+0000 7fed737fe640 1 -- 172.21.15.81:0/1738907698 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fed74061380 con 0x7fed7c0b5220 2024-06-29T14:23:13.651 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:23:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 -- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fed5c076300 msgr2=0x7fed5c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fed5c076300 0x7fed5c078810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fed6c000c00 tx=0x7fed6c006b70 comp rx=0 tx=0).stop 2024-06-29T14:23:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 -- 172.21.15.81:0/1738907698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed7c0b5220 msgr2=0x7fed7c0b5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed7c0b5220 0x7fed7c0b5630 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fed74001330 tx=0x7fed7400eb20 comp rx=0 tx=0).stop 2024-06-29T14:23:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 -- 172.21.15.81:0/1738907698 shutdown_connections 2024-06-29T14:23:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fed5c076300 0x7fed5c078810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed7c0b5220 0x7fed7c0b5630 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed7c1e0c20 0x7fed7c1e7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 --2- 172.21.15.81:0/1738907698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed680040d0 0x7fed7c1e1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 -- 172.21.15.81:0/1738907698 >> 172.21.15.81:0/1738907698 conn(0x7fed7c0d1960 msgr2=0x7fed7c071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:13.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 -- 172.21.15.81:0/1738907698 shutdown_connections 2024-06-29T14:23:13.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.654+0000 7fed808c3640 1 -- 172.21.15.81:0/1738907698 wait complete. 2024-06-29T14:23:13.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-06-29T14:23:13.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.778+0000 7fb9139a1640 1 -- 172.21.15.81:0/2911436390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb90c0b6eb0 msgr2=0x7fb90c0d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.778+0000 7fb9139a1640 1 --2- 172.21.15.81:0/2911436390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb90c0b6eb0 0x7fb90c0d02d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb90c073550 tx=0x7fb8fc02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.778+0000 7fb9139a1640 1 -- 172.21.15.81:0/2911436390 shutdown_connections 2024-06-29T14:23:13.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.778+0000 7fb9139a1640 1 --2- 172.21.15.81:0/2911436390 >> v2:172.21.15.167:3300/0 conn(0x7fb90c0ce510 0x7fb90c0ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.778+0000 7fb9139a1640 1 --2- 172.21.15.81:0/2911436390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb90c0b6eb0 0x7fb90c0d02d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.778+0000 7fb9139a1640 1 -- 172.21.15.81:0/2911436390 >> 172.21.15.81:0/2911436390 conn(0x7fb90c0b3510 msgr2=0x7fb90c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:13.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.778+0000 7fb9139a1640 1 -- 172.21.15.81:0/2911436390 shutdown_connections 2024-06-29T14:23:13.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.778+0000 7fb9139a1640 1 -- 172.21.15.81:0/2911436390 wait complete. 2024-06-29T14:23:13.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.780+0000 7fb9139a1640 1 Processor -- start 2024-06-29T14:23:13.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.780+0000 7fb9139a1640 1 -- start start 2024-06-29T14:23:13.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.781+0000 7fb9139a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb90c0b4810 0x7fb90c1e1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.781+0000 7fb9139a1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb90c0b6eb0 0x7fb90c1e7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.781+0000 7fb9139a1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb90c0ce510 0x7fb90c0b5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.781+0000 7fb9139a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb90c0cd480 con 0x7fb90c0b4810 2024-06-29T14:23:13.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.781+0000 7fb9139a1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb90c0b29c0 con 0x7fb90c0ce510 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.781+0000 7fb9139a1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb90c0cd600 con 0x7fb90c0b6eb0 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.781+0000 7fb91173d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb90c0b4810 0x7fb90c1e1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.781+0000 7fb91173d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb90c0b4810 0x7fb90c1e1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49664/0 (socket says 172.21.15.81:49664) 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.781+0000 7fb91173d640 1 -- 172.21.15.81:0/2586935844 learned_addr learned my addr 172.21.15.81:0/2586935844 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb910f3c640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb90c0b6eb0 0x7fb90c1e7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb911f3e640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb90c0ce510 0x7fb90c0b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb91173d640 1 -- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb90c0b6eb0 msgr2=0x7fb90c1e7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb91173d640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb90c0b6eb0 0x7fb90c1e7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb91173d640 1 -- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb90c0ce510 msgr2=0x7fb90c0b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb91173d640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb90c0ce510 0x7fb90c0b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb91173d640 1 -- 172.21.15.81:0/2586935844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90c1e8060 con 0x7fb90c0b4810 2024-06-29T14:23:13.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb91173d640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb90c0b4810 0x7fb90c1e1990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0098f0 tx=0x7fb8fc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:13.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb909ffb640 1 -- 172.21.15.81:0/2586935844 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc031d60 con 0x7fb90c0b4810 2024-06-29T14:23:13.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb909ffb640 1 -- 172.21.15.81:0/2586935844 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8fc033e10 con 0x7fb90c0b4810 2024-06-29T14:23:13.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb9139a1640 1 -- 172.21.15.81:0/2586935844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb90c1e58a0 con 0x7fb90c0b4810 2024-06-29T14:23:13.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb909ffb640 1 -- 172.21.15.81:0/2586935844 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc005260 con 0x7fb90c0b4810 2024-06-29T14:23:13.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.782+0000 7fb9139a1640 1 -- 172.21.15.81:0/2586935844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb90c1e5e60 con 0x7fb90c0b4810 2024-06-29T14:23:13.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.783+0000 7fb9139a1640 1 -- 172.21.15.81:0/2586935844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb90c0d02d0 con 0x7fb90c0b4810 2024-06-29T14:23:13.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.784+0000 7fb909ffb640 1 -- 172.21.15.81:0/2586935844 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb8fc005400 con 0x7fb90c0b4810 2024-06-29T14:23:13.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.785+0000 7fb909ffb640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8f8076300 0x7fb8f8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:13.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.785+0000 7fb909ffb640 1 -- 172.21.15.81:0/2586935844 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb8fc0b89d0 con 0x7fb90c0b4810 2024-06-29T14:23:13.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.785+0000 7fb910f3c640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8f8076300 0x7fb8f8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:13.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.786+0000 7fb910f3c640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8f8076300 0x7fb8f8078810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb900005d70 tx=0x7fb900005d00 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:13.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:13.789+0000 7fb909ffb640 1 -- 172.21.15.81:0/2586935844 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb8fc03f030 con 0x7fb90c0b4810 2024-06-29T14:23:14.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.044+0000 7fb9139a1640 1 -- 172.21.15.81:0/2586935844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) -- 0x7fb90c0cee10 con 0x7fb90c0b4810 2024-06-29T14:23:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.813+0000 7fb909ffb640 1 -- 172.21.15.81:0/2586935844 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v54) ==== 161+0+0 (secure 0 0 0) 0x7fb8fc0822a0 con 0x7fb90c0b4810 2024-06-29T14:23:14.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 -- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8f8076300 msgr2=0x7fb8f8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:14.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8f8076300 0x7fb8f8078810 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb900005d70 tx=0x7fb900005d00 comp rx=0 tx=0).stop 2024-06-29T14:23:14.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 -- 172.21.15.81:0/2586935844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb90c0b4810 msgr2=0x7fb90c1e1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:14.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb90c0b4810 0x7fb90c1e1990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0098f0 tx=0x7fb8fc033c20 comp rx=0 tx=0).stop 2024-06-29T14:23:14.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 -- 172.21.15.81:0/2586935844 shutdown_connections 2024-06-29T14:23:14.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8f8076300 0x7fb8f8078810 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb90c0ce510 0x7fb90c0b5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb90c0b6eb0 0x7fb90c1e7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 --2- 172.21.15.81:0/2586935844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb90c0b4810 0x7fb90c1e1990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 -- 172.21.15.81:0/2586935844 >> 172.21.15.81:0/2586935844 conn(0x7fb90c0b3510 msgr2=0x7fb90c1e5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:14.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 -- 172.21.15.81:0/2586935844 shutdown_connections 2024-06-29T14:23:14.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.817+0000 7fb9139a1640 1 -- 172.21.15.81:0/2586935844 wait complete. 2024-06-29T14:23:14.828 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:23:14.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.949+0000 7f7e0dc26640 1 -- 172.21.15.81:0/1611098416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e08073030 msgr2=0x7f7e08073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:14.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.949+0000 7f7e0dc26640 1 --2- 172.21.15.81:0/1611098416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e08073030 0x7f7e08073440 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7e08071d20 tx=0x7f7dfc02a560 comp rx=0 tx=0).stop 2024-06-29T14:23:14.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.949+0000 7f7e0dc26640 1 -- 172.21.15.81:0/1611098416 shutdown_connections 2024-06-29T14:23:14.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.949+0000 7f7e0dc26640 1 --2- 172.21.15.81:0/1611098416 >> v2:172.21.15.167:3300/0 conn(0x7f7e0815e960 0x7f7e0815ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.949+0000 7f7e0dc26640 1 --2- 172.21.15.81:0/1611098416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e08073030 0x7f7e08073440 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.949+0000 7f7e0dc26640 1 -- 172.21.15.81:0/1611098416 >> 172.21.15.81:0/1611098416 conn(0x7f7e0806ccc0 msgr2=0x7f7e080b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:14.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.949+0000 7f7e0dc26640 1 -- 172.21.15.81:0/1611098416 shutdown_connections 2024-06-29T14:23:14.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.949+0000 7f7e0dc26640 1 -- 172.21.15.81:0/1611098416 wait complete. 2024-06-29T14:23:14.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.950+0000 7f7e0dc26640 1 Processor -- start 2024-06-29T14:23:14.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.951+0000 7f7e0dc26640 1 -- start start 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e0dc26640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e08073030 0x7f7e080aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e0dc26640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e081412f0 0x7f7e08153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e0dc26640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e0815e960 0x7f7e08071f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e0dc26640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7e080b3210 con 0x7f7e081412f0 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e0dc26640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7e08141740 con 0x7f7e0815e960 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e0dc26640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7e080b3390 con 0x7f7e08073030 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e06ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e081412f0 0x7f7e08153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e06ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e081412f0 0x7f7e08153b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49690/0 (socket says 172.21.15.81:49690) 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e06ffd640 1 -- 172.21.15.81:0/377862093 learned_addr learned my addr 172.21.15.81:0/377862093 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e077fe640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e08073030 0x7f7e080aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e07fff640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e0815e960 0x7f7e08071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:14.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e06ffd640 1 -- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e08073030 msgr2=0x7f7e080aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e06ffd640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e08073030 0x7f7e080aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e06ffd640 1 -- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e0815e960 msgr2=0x7f7e08071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e06ffd640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e0815e960 0x7f7e08071f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.952+0000 7f7e06ffd640 1 -- 172.21.15.81:0/377862093 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e080acb70 con 0x7f7e081412f0 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.953+0000 7f7e06ffd640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e081412f0 0x7f7e08153b30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc000c00 tx=0x7f7dfc0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.953+0000 7f7deffff640 1 -- 172.21.15.81:0/377862093 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc03e030 con 0x7f7e081412f0 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.953+0000 7f7deffff640 1 -- 172.21.15.81:0/377862093 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7dfc0047a0 con 0x7f7e081412f0 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.953+0000 7f7e0dc26640 1 -- 172.21.15.81:0/377862093 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e081572c0 con 0x7f7e081412f0 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.953+0000 7f7deffff640 1 -- 172.21.15.81:0/377862093 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc035040 con 0x7f7e081412f0 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.953+0000 7f7e0dc26640 1 -- 172.21.15.81:0/377862093 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e08157880 con 0x7f7e081412f0 2024-06-29T14:23:14.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.954+0000 7f7e0dc26640 1 -- 172.21.15.81:0/377862093 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e080b3970 con 0x7f7e081412f0 2024-06-29T14:23:14.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.955+0000 7f7deffff640 1 -- 172.21.15.81:0/377862093 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7dfc041030 con 0x7f7e081412f0 2024-06-29T14:23:14.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.955+0000 7f7deffff640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7df4076300 0x7f7df4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:14.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.955+0000 7f7deffff640 1 -- 172.21.15.81:0/377862093 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7dfc0b7f40 con 0x7f7e081412f0 2024-06-29T14:23:14.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.955+0000 7f7e077fe640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7df4076300 0x7f7df4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:14.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.956+0000 7f7e077fe640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7df4076300 0x7f7df4078810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7df8006fd0 tx=0x7f7df8009060 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:14.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:14.959+0000 7f7deffff640 1 -- 172.21.15.81:0/377862093 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7e08073980 con 0x7f7e081412f0 2024-06-29T14:23:15.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.141+0000 7f7e0dc26640 1 -- 172.21.15.81:0/377862093 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f7e0815a1f0 con 0x7f7e081412f0 2024-06-29T14:23:15.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.142+0000 7f7deffff640 1 -- 172.21.15.81:0/377862093 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v27) ==== 59+0+1998 (secure 0 0 0) 0x7f7dfc0811c0 con 0x7f7e081412f0 2024-06-29T14:23:15.140 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:23:15.140 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:23:15.140 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:23:15.140 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:23:15.140 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T14:23:15.141 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:23:15.142 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:23:15.142 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T14:23:15.142 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:23:15.142 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:23:15.142 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:23:15.142 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T14:23:15.142 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T14:23:15.142 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T14:23:15.142 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:23:15.142 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:23:15.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 -- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7df4076300 msgr2=0x7f7df4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7df4076300 0x7f7df4078810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7df8006fd0 tx=0x7f7df8009060 comp rx=0 tx=0).stop 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 -- 172.21.15.81:0/377862093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e081412f0 msgr2=0x7f7e08153b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e081412f0 0x7f7e08153b30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc000c00 tx=0x7f7dfc0039e0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 -- 172.21.15.81:0/377862093 shutdown_connections 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7df4076300 0x7f7df4078810 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e0815e960 0x7f7e08071f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e081412f0 0x7f7e08153b30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 --2- 172.21.15.81:0/377862093 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e08073030 0x7f7e080aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 -- 172.21.15.81:0/377862093 >> 172.21.15.81:0/377862093 conn(0x7f7e0806ccc0 msgr2=0x7f7e080af770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 -- 172.21.15.81:0/377862093 shutdown_connections 2024-06-29T14:23:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.144+0000 7f7e0dc26640 1 -- 172.21.15.81:0/377862093 wait complete. 2024-06-29T14:23:15.153 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:23:15.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.271+0000 7fd31d723640 1 -- 172.21.15.81:0/3186109272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3180d2a90 msgr2=0x7fd3180d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.271+0000 7fd31d723640 1 --2- 172.21.15.81:0/3186109272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3180d2a90 0x7fd3180d09c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd308005fd0 tx=0x7fd30802a940 comp rx=0 tx=0).stop 2024-06-29T14:23:15.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.271+0000 7fd31d723640 1 -- 172.21.15.81:0/3186109272 shutdown_connections 2024-06-29T14:23:15.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.271+0000 7fd31d723640 1 --2- 172.21.15.81:0/3186109272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3180d2a90 0x7fd3180d09c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.271+0000 7fd31d723640 1 --2- 172.21.15.81:0/3186109272 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3180b9040 0x7fd3180d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.271+0000 7fd31d723640 1 -- 172.21.15.81:0/3186109272 >> 172.21.15.81:0/3186109272 conn(0x7fd31809cf30 msgr2=0x7fd3180d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:15.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.271+0000 7fd31d723640 1 -- 172.21.15.81:0/3186109272 shutdown_connections 2024-06-29T14:23:15.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.271+0000 7fd31d723640 1 -- 172.21.15.81:0/3186109272 wait complete. 2024-06-29T14:23:15.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.273+0000 7fd31d723640 1 Processor -- start 2024-06-29T14:23:15.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.273+0000 7fd31d723640 1 -- start start 2024-06-29T14:23:15.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.274+0000 7fd31d723640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd3180b5250 0x7fd318155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.274+0000 7fd31d723640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3180b9040 0x7fd3180a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.274+0000 7fd31d723640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3180d2a90 0x7fd3180a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.274+0000 7fd31d723640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd3180b7d30 con 0x7fd3180d2a90 2024-06-29T14:23:15.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.274+0000 7fd31d723640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd3180b7bb0 con 0x7fd3180b5250 2024-06-29T14:23:15.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.274+0000 7fd31d723640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd3180b7eb0 con 0x7fd3180b9040 2024-06-29T14:23:15.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd3177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3180d2a90 0x7fd3180a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd3177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3180d2a90 0x7fd3180a0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49718/0 (socket says 172.21.15.81:49718) 2024-06-29T14:23:15.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd3177fe640 1 -- 172.21.15.81:0/992329735 learned_addr learned my addr 172.21.15.81:0/992329735 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd3167fc640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3180b9040 0x7fd3180a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd3177fe640 1 -- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3180b9040 msgr2=0x7fd3180a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd316ffd640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd3180b5250 0x7fd318155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd3177fe640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3180b9040 0x7fd3180a09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd3177fe640 1 -- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd3180b5250 msgr2=0x7fd318155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd3177fe640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd3180b5250 0x7fd318155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd3177fe640 1 -- 172.21.15.81:0/992329735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3181501c0 con 0x7fd3180d2a90 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd3177fe640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3180d2a90 0x7fd3180a0f10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd30802af30 tx=0x7fd308031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd30f7fe640 1 -- 172.21.15.81:0/992329735 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30803f030 con 0x7fd3180d2a90 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd30f7fe640 1 -- 172.21.15.81:0/992329735 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd308004250 con 0x7fd3180d2a90 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd31d723640 1 -- 172.21.15.81:0/992329735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd318150430 con 0x7fd3180d2a90 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd30f7fe640 1 -- 172.21.15.81:0/992329735 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd308004660 con 0x7fd3180d2a90 2024-06-29T14:23:15.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.275+0000 7fd31d723640 1 -- 172.21.15.81:0/992329735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3181509f0 con 0x7fd3180d2a90 2024-06-29T14:23:15.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.276+0000 7fd31d723640 1 -- 172.21.15.81:0/992329735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3180b83b0 con 0x7fd3180d2a90 2024-06-29T14:23:15.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.277+0000 7fd30f7fe640 1 -- 172.21.15.81:0/992329735 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd308042030 con 0x7fd3180d2a90 2024-06-29T14:23:15.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.278+0000 7fd30f7fe640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd2f8076300 0x7fd2f8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.278+0000 7fd30f7fe640 1 -- 172.21.15.81:0/992329735 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd3080b8970 con 0x7fd3180d2a90 2024-06-29T14:23:15.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.278+0000 7fd316ffd640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd2f8076300 0x7fd2f8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.278+0000 7fd316ffd640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd2f8076300 0x7fd2f8078810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd3100058c0 tx=0x7fd310005850 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:15.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.282+0000 7fd30f7fe640 1 -- 172.21.15.81:0/992329735 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd308082240 con 0x7fd3180d2a90 2024-06-29T14:23:15.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.465+0000 7fd31d723640 1 -- 172.21.15.81:0/992329735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd31815f5e0 con 0x7fd3180d2a90 2024-06-29T14:23:15.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.469+0000 7fd30f7fe640 1 -- 172.21.15.81:0/992329735 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7fd308081b40 con 0x7fd3180d2a90 2024-06-29T14:23:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 -- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd2f8076300 msgr2=0x7fd2f8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd2f8076300 0x7fd2f8078810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd3100058c0 tx=0x7fd310005850 comp rx=0 tx=0).stop 2024-06-29T14:23:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 -- 172.21.15.81:0/992329735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3180d2a90 msgr2=0x7fd3180a0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3180d2a90 0x7fd3180a0f10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd30802af30 tx=0x7fd308031e20 comp rx=0 tx=0).stop 2024-06-29T14:23:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 -- 172.21.15.81:0/992329735 shutdown_connections 2024-06-29T14:23:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fd2f8076300 0x7fd2f8078810 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3180d2a90 0x7fd3180a0f10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3180b9040 0x7fd3180a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 --2- 172.21.15.81:0/992329735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd3180b5250 0x7fd318155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 -- 172.21.15.81:0/992329735 >> 172.21.15.81:0/992329735 conn(0x7fd31809cf30 msgr2=0x7fd3180d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 -- 172.21.15.81:0/992329735 shutdown_connections 2024-06-29T14:23:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.471+0000 7fd31d723640 1 -- 172.21.15.81:0/992329735 wait complete. 2024-06-29T14:23:15.480 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:23:15.480 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:23:15.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:23:15.569 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:23:15.569 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:23:15.569 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:15.585 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:23:15.585 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:15.641 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:23:15.641 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:15.696 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.852+0000 7f1bbcf43640 1 -- 172.21.15.81:0/600751462 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1bb80b3200 msgr2=0x7f1bb80cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.852+0000 7f1bbcf43640 1 --2- 172.21.15.81:0/600751462 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1bb80b3200 0x7f1bb80cc7b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009830 tx=0x7f1ba802a510 comp rx=0 tx=0).stop 2024-06-29T14:23:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.853+0000 7f1bbcf43640 1 -- 172.21.15.81:0/600751462 shutdown_connections 2024-06-29T14:23:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.853+0000 7f1bbcf43640 1 --2- 172.21.15.81:0/600751462 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1bb80d0940 0x7f1bb80b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.853+0000 7f1bbcf43640 1 --2- 172.21.15.81:0/600751462 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1bb80b3200 0x7f1bb80cc7b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.853+0000 7f1bbcf43640 1 -- 172.21.15.81:0/600751462 >> 172.21.15.81:0/600751462 conn(0x7f1bb80b3680 msgr2=0x7f1bb80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:15.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.853+0000 7f1bbcf43640 1 -- 172.21.15.81:0/600751462 shutdown_connections 2024-06-29T14:23:15.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.853+0000 7f1bbcf43640 1 -- 172.21.15.81:0/600751462 wait complete. 2024-06-29T14:23:15.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.854+0000 7f1bbcf43640 1 Processor -- start 2024-06-29T14:23:15.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.855+0000 7f1bbcf43640 1 -- start start 2024-06-29T14:23:15.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bbcf43640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb80b3200 0x7f1bb80af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bbcf43640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1bb80cccf0 0x7f1bb815b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bbcf43640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1bb809d750 0x7f1bb80d1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bbcf43640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb80b5c50 con 0x7f1bb80b3200 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bbcf43640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb80b5ad0 con 0x7f1bb809d750 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bbcf43640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb80b5dd0 con 0x7f1bb80cccf0 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb80b3200 0x7f1bb80af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb80b3200 0x7f1bb80af370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49728/0 (socket says 172.21.15.81:49728) 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb659c640 1 -- 172.21.15.81:0/388217966 learned_addr learned my addr 172.21.15.81:0/388217966 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb5d9b640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1bb80cccf0 0x7f1bb815b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb6d9d640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1bb809d750 0x7f1bb80d1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb659c640 1 -- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1bb80cccf0 msgr2=0x7f1bb815b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb659c640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1bb80cccf0 0x7f1bb815b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb659c640 1 -- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1bb809d750 msgr2=0x7f1bb80d1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:15.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb659c640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1bb809d750 0x7f1bb80d1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:15.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb659c640 1 -- 172.21.15.81:0/388217966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb809ddf0 con 0x7f1bb80b3200 2024-06-29T14:23:15.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.856+0000 7f1bb659c640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb80b3200 0x7f1bb80af370 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009950 tx=0x7f1ba8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:15.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.857+0000 7f1baeffd640 1 -- 172.21.15.81:0/388217966 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8005c90 con 0x7f1bb80b3200 2024-06-29T14:23:15.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.857+0000 7f1bbcf43640 1 -- 172.21.15.81:0/388217966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb809e0d0 con 0x7f1bb80b3200 2024-06-29T14:23:15.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.857+0000 7f1baeffd640 1 -- 172.21.15.81:0/388217966 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ba8031e50 con 0x7f1bb80b3200 2024-06-29T14:23:15.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.857+0000 7f1baeffd640 1 -- 172.21.15.81:0/388217966 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8005260 con 0x7f1bb80b3200 2024-06-29T14:23:15.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.857+0000 7f1bbcf43640 1 -- 172.21.15.81:0/388217966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb809e690 con 0x7f1bb80b3200 2024-06-29T14:23:15.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.858+0000 7f1bbcf43640 1 -- 172.21.15.81:0/388217966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb80cc7b0 con 0x7f1bb80b3200 2024-06-29T14:23:15.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.859+0000 7f1baeffd640 1 -- 172.21.15.81:0/388217966 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1ba8005400 con 0x7f1bb80b3200 2024-06-29T14:23:15.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.859+0000 7f1baeffd640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1b980762d0 0x7f1b980787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:15.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.859+0000 7f1baeffd640 1 -- 172.21.15.81:0/388217966 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1ba80b8b80 con 0x7f1bb80b3200 2024-06-29T14:23:15.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.859+0000 7f1bb5d9b640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1b980762d0 0x7f1b980787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:15.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.860+0000 7f1bb5d9b640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1b980762d0 0x7f1b980787e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0007fd0 tx=0x7f1ba0009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:15.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:15.863+0000 7f1baeffd640 1 -- 172.21.15.81:0/388217966 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1ba803f030 con 0x7f1bb80b3200 2024-06-29T14:23:16.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.045+0000 7f1bbcf43640 1 -- 172.21.15.81:0/388217966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1bb80c6260 con 0x7f1bb80b3200 2024-06-29T14:23:16.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.046+0000 7f1baeffd640 1 -- 172.21.15.81:0/388217966 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f1ba8081e00 con 0x7f1bb80b3200 2024-06-29T14:23:16.045 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:16.045 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 -- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1b980762d0 msgr2=0x7f1b980787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1b980762d0 0x7f1b980787e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0007fd0 tx=0x7f1ba0009040 comp rx=0 tx=0).stop 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 -- 172.21.15.81:0/388217966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb80b3200 msgr2=0x7f1bb80af370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb80b3200 0x7f1bb80af370 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009950 tx=0x7f1ba8033c20 comp rx=0 tx=0).stop 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 -- 172.21.15.81:0/388217966 shutdown_connections 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1b980762d0 0x7f1b980787e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1bb809d750 0x7f1bb80d1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1bb80cccf0 0x7f1bb815b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 --2- 172.21.15.81:0/388217966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb80b3200 0x7f1bb80af370 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 -- 172.21.15.81:0/388217966 >> 172.21.15.81:0/388217966 conn(0x7f1bb80b3680 msgr2=0x7f1bb8070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 -- 172.21.15.81:0/388217966 shutdown_connections 2024-06-29T14:23:16.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.048+0000 7f1bbcf43640 1 -- 172.21.15.81:0/388217966 wait complete. 2024-06-29T14:23:16.058 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:16.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:16.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.177+0000 7f9138311640 1 -- 172.21.15.81:0/70011870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9118003a00 msgr2=0x7f9118003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.177+0000 7f9138311640 1 --2- 172.21.15.81:0/70011870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9118003a00 0x7f9118003e70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f912000c2e0 tx=0x7f9118005830 comp rx=0 tx=0).stop 2024-06-29T14:23:16.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.177+0000 7f91368ae640 1 -- 172.21.15.81:0/70011870 reap_dead start 2024-06-29T14:23:16.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.177+0000 7f9138311640 1 -- 172.21.15.81:0/70011870 shutdown_connections 2024-06-29T14:23:16.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.177+0000 7f9138311640 1 -- 172.21.15.81:0/70011870 >> 172.21.15.81:0/70011870 conn(0x7f91300d0930 msgr2=0x7f91300d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.177+0000 7f9138311640 1 -- 172.21.15.81:0/70011870 shutdown_connections 2024-06-29T14:23:16.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.177+0000 7f9138311640 1 -- 172.21.15.81:0/70011870 wait complete. 2024-06-29T14:23:16.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.179+0000 7f9138311640 1 Processor -- start 2024-06-29T14:23:16.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.179+0000 7f9138311640 1 -- start start 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.180+0000 7f9138311640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9118003a00 0x7f91301504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.180+0000 7f9138311640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91301563c0 0x7f91300c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f9138311640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91300c6940 0x7f91301f8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f9138311640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f91300cf920 con 0x7f91301563c0 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f9138311640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f91300cf7a0 con 0x7f9118003a00 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f9138311640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f91300cfaa0 con 0x7f91300c6940 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91358ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91301563c0 0x7f91300c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91358ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91301563c0 0x7f91300c6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49752/0 (socket says 172.21.15.81:49752) 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91358ac640 1 -- 172.21.15.81:0/2659202286 learned_addr learned my addr 172.21.15.81:0/2659202286 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91360ad640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9118003a00 0x7f91301504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91368ae640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91300c6940 0x7f91301f8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91358ac640 1 -- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91300c6940 msgr2=0x7f91301f8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91358ac640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91300c6940 0x7f91301f8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91358ac640 1 -- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9118003a00 msgr2=0x7f91301504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91358ac640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9118003a00 0x7f91301504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91358ac640 1 -- 172.21.15.81:0/2659202286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9130156c40 con 0x7f91301563c0 2024-06-29T14:23:16.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f91358ac640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91301563c0 0x7f91300c6400 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9124011ae0 tx=0x7f9124009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:16.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f912effd640 1 -- 172.21.15.81:0/2659202286 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912401e070 con 0x7f91301563c0 2024-06-29T14:23:16.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f912effd640 1 -- 172.21.15.81:0/2659202286 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9124004460 con 0x7f91301563c0 2024-06-29T14:23:16.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f912effd640 1 -- 172.21.15.81:0/2659202286 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912400fc60 con 0x7f91301563c0 2024-06-29T14:23:16.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f9138311640 1 -- 172.21.15.81:0/2659202286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91300c6d50 con 0x7f91301563c0 2024-06-29T14:23:16.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.181+0000 7f9138311640 1 -- 172.21.15.81:0/2659202286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91300ae760 con 0x7f91301563c0 2024-06-29T14:23:16.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.182+0000 7f9138311640 1 -- 172.21.15.81:0/2659202286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f913006e2a0 con 0x7f91301563c0 2024-06-29T14:23:16.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.183+0000 7f912effd640 1 -- 172.21.15.81:0/2659202286 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9124021070 con 0x7f91301563c0 2024-06-29T14:23:16.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.183+0000 7f912effd640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f911c0762d0 0x7f911c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.184+0000 7f912effd640 1 -- 172.21.15.81:0/2659202286 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9124098250 con 0x7f91301563c0 2024-06-29T14:23:16.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.184+0000 7f91360ad640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f911c0762d0 0x7f911c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.184+0000 7f91360ad640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f911c0762d0 0x7f911c0787e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9120033000 tx=0x7f9120038040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:16.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.188+0000 7f912effd640 1 -- 172.21.15.81:0/2659202286 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f913006e2a0 con 0x7f91301563c0 2024-06-29T14:23:16.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.366+0000 7f9138311640 1 -- 172.21.15.81:0/2659202286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f913006c6f0 con 0x7f91301563c0 2024-06-29T14:23:16.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.366+0000 7f912effd640 1 -- 172.21.15.81:0/2659202286 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f91300c6d50 con 0x7f91301563c0 2024-06-29T14:23:16.364 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 -- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f911c0762d0 msgr2=0x7f911c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f911c0762d0 0x7f911c0787e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9120033000 tx=0x7f9120038040 comp rx=0 tx=0).stop 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 -- 172.21.15.81:0/2659202286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91301563c0 msgr2=0x7f91300c6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91301563c0 0x7f91300c6400 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9124011ae0 tx=0x7f9124009e60 comp rx=0 tx=0).stop 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 -- 172.21.15.81:0/2659202286 shutdown_connections 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f911c0762d0 0x7f911c0787e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91300c6940 0x7f91301f8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91301563c0 0x7f91300c6400 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 --2- 172.21.15.81:0/2659202286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9118003a00 0x7f91301504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 -- 172.21.15.81:0/2659202286 >> 172.21.15.81:0/2659202286 conn(0x7f91300d0930 msgr2=0x7f913006ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 -- 172.21.15.81:0/2659202286 shutdown_connections 2024-06-29T14:23:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.368+0000 7f9138311640 1 -- 172.21.15.81:0/2659202286 wait complete. 2024-06-29T14:23:16.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:23:16.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.497+0000 7f4f96773640 1 -- 172.21.15.81:0/1467001889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f900b4250 msgr2=0x7f4f900cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.497+0000 7f4f96773640 1 --2- 172.21.15.81:0/1467001889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f900b4250 0x7f4f900cee70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4f900d4c50 tx=0x7f4f8802a9a0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.497+0000 7f4f96773640 1 -- 172.21.15.81:0/1467001889 shutdown_connections 2024-06-29T14:23:16.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.497+0000 7f4f96773640 1 --2- 172.21.15.81:0/1467001889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f900b4250 0x7f4f900cee70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.497+0000 7f4f96773640 1 --2- 172.21.15.81:0/1467001889 >> v2:172.21.15.167:3300/0 conn(0x7f4f9008c270 0x7f4f900ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.497+0000 7f4f96773640 1 -- 172.21.15.81:0/1467001889 >> 172.21.15.81:0/1467001889 conn(0x7f4f900b56d0 msgr2=0x7f4f900b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.497+0000 7f4f96773640 1 -- 172.21.15.81:0/1467001889 shutdown_connections 2024-06-29T14:23:16.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.497+0000 7f4f96773640 1 -- 172.21.15.81:0/1467001889 wait complete. 2024-06-29T14:23:16.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.499+0000 7f4f96773640 1 Processor -- start 2024-06-29T14:23:16.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.499+0000 7f4f96773640 1 -- start start 2024-06-29T14:23:16.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f96773640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f9008ccd0 0x7f4f901dbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f96773640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f900b4250 0x7f4f900c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f96773640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900c9f10 0x7f4f900b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f96773640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4f900d2250 con 0x7f4f9008ccd0 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f96773640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4f900d20d0 con 0x7f4f900c9f10 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f96773640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4f900d23d0 con 0x7f4f900b4250 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f8ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f9008ccd0 0x7f4f901dbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f8ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f9008ccd0 0x7f4f901dbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49760/0 (socket says 172.21.15.81:49760) 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f8ffff640 1 -- 172.21.15.81:0/1040155824 learned_addr learned my addr 172.21.15.81:0/1040155824 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f94d10640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900c9f10 0x7f4f900b79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f8f7fe640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f900b4250 0x7f4f900c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f8ffff640 1 -- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f900b4250 msgr2=0x7f4f900c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.500+0000 7f4f8ffff640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f900b4250 0x7f4f900c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.501+0000 7f4f8ffff640 1 -- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900c9f10 msgr2=0x7f4f900b79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.501+0000 7f4f8ffff640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900c9f10 0x7f4f900b79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.501+0000 7f4f8ffff640 1 -- 172.21.15.81:0/1040155824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f900c8070 con 0x7f4f9008ccd0 2024-06-29T14:23:16.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.501+0000 7f4f8ffff640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f9008ccd0 0x7f4f901dbcc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000e7c0 tx=0x7f4f8000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:16.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.501+0000 7f4f8cff9640 1 -- 172.21.15.81:0/1040155824 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8000cc60 con 0x7f4f9008ccd0 2024-06-29T14:23:16.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.501+0000 7f4f8cff9640 1 -- 172.21.15.81:0/1040155824 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f80010430 con 0x7f4f9008ccd0 2024-06-29T14:23:16.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.501+0000 7f4f96773640 1 -- 172.21.15.81:0/1040155824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f901dc200 con 0x7f4f9008ccd0 2024-06-29T14:23:16.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.501+0000 7f4f8cff9640 1 -- 172.21.15.81:0/1040155824 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80010860 con 0x7f4f9008ccd0 2024-06-29T14:23:16.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.501+0000 7f4f96773640 1 -- 172.21.15.81:0/1040155824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f900c8350 con 0x7f4f9008ccd0 2024-06-29T14:23:16.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.502+0000 7f4f96773640 1 -- 172.21.15.81:0/1040155824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f9008c270 con 0x7f4f9008ccd0 2024-06-29T14:23:16.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.503+0000 7f4f8cff9640 1 -- 172.21.15.81:0/1040155824 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4f80010a00 con 0x7f4f9008ccd0 2024-06-29T14:23:16.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.503+0000 7f4f8cff9640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4f7c0762d0 0x7f4f7c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.503+0000 7f4f8cff9640 1 -- 172.21.15.81:0/1040155824 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4f80097fb0 con 0x7f4f9008ccd0 2024-06-29T14:23:16.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.504+0000 7f4f8f7fe640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4f7c0762d0 0x7f4f7c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.504+0000 7f4f8f7fe640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4f7c0762d0 0x7f4f7c0787e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4f901e4370 tx=0x7f4f84009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:16.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.508+0000 7f4f8cff9640 1 -- 172.21.15.81:0/1040155824 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f800612c0 con 0x7f4f9008ccd0 2024-06-29T14:23:16.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.688+0000 7f4f96773640 1 -- 172.21.15.81:0/1040155824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4f900d1990 con 0x7f4f9008ccd0 2024-06-29T14:23:16.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.688+0000 7f4f8cff9640 1 -- 172.21.15.81:0/1040155824 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f4f800655c0 con 0x7f4f9008ccd0 2024-06-29T14:23:16.686 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 -- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4f7c0762d0 msgr2=0x7f4f7c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4f7c0762d0 0x7f4f7c0787e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4f901e4370 tx=0x7f4f84009040 comp rx=0 tx=0).stop 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 -- 172.21.15.81:0/1040155824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f9008ccd0 msgr2=0x7f4f901dbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f9008ccd0 0x7f4f901dbcc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000e7c0 tx=0x7f4f8000ec80 comp rx=0 tx=0).stop 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 -- 172.21.15.81:0/1040155824 shutdown_connections 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4f7c0762d0 0x7f4f7c0787e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f900c9f10 0x7f4f900b79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f900b4250 0x7f4f900c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 --2- 172.21.15.81:0/1040155824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f9008ccd0 0x7f4f901dbcc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 -- 172.21.15.81:0/1040155824 >> 172.21.15.81:0/1040155824 conn(0x7f4f900b56d0 msgr2=0x7f4f900b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 -- 172.21.15.81:0/1040155824 shutdown_connections 2024-06-29T14:23:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.690+0000 7f4f96773640 1 -- 172.21.15.81:0/1040155824 wait complete. 2024-06-29T14:23:16.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:23:16.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.819+0000 7f1a98bfb640 1 -- 172.21.15.81:0/1267966075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a941412f0 msgr2=0x7f1a940cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.819+0000 7f1a98bfb640 1 --2- 172.21.15.81:0/1267966075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a941412f0 0x7f1a940cef90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1a84009860 tx=0x7f1a8402a5a0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.819+0000 7f1a98bfb640 1 -- 172.21.15.81:0/1267966075 shutdown_connections 2024-06-29T14:23:16.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.819+0000 7f1a98bfb640 1 --2- 172.21.15.81:0/1267966075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a941412f0 0x7f1a940cef90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.819+0000 7f1a98bfb640 1 -- 172.21.15.81:0/1267966075 >> 172.21.15.81:0/1267966075 conn(0x7f1a9406ccc0 msgr2=0x7f1a940b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:16.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.819+0000 7f1a98bfb640 1 -- 172.21.15.81:0/1267966075 shutdown_connections 2024-06-29T14:23:16.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.819+0000 7f1a98bfb640 1 -- 172.21.15.81:0/1267966075 wait complete. 2024-06-29T14:23:16.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.821+0000 7f1a98bfb640 1 Processor -- start 2024-06-29T14:23:16.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.821+0000 7f1a98bfb640 1 -- start start 2024-06-29T14:23:16.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a98bfb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a940cf4d0 0x7f1a9413f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a98bfb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a941412f0 0x7f1a940aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a98bfb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a940af0a0 0x7f1a940c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a98bfb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a940b35a0 con 0x7f1a940cf4d0 2024-06-29T14:23:16.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a98bfb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a940b3420 con 0x7f1a940af0a0 2024-06-29T14:23:16.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a98bfb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a940b3720 con 0x7f1a941412f0 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a9259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a940cf4d0 0x7f1a9413f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a9259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a940cf4d0 0x7f1a9413f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49768/0 (socket says 172.21.15.81:49768) 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a9259c640 1 -- 172.21.15.81:0/737680843 learned_addr learned my addr 172.21.15.81:0/737680843 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a89d9b640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a941412f0 0x7f1a940aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a92d9d640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a940af0a0 0x7f1a940c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a9259c640 1 -- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a941412f0 msgr2=0x7f1a940aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a9259c640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a941412f0 0x7f1a940aeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a9259c640 1 -- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a940af0a0 msgr2=0x7f1a940c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a9259c640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a940af0a0 0x7f1a940c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a9259c640 1 -- 172.21.15.81:0/737680843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9413fe80 con 0x7f1a940cf4d0 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.823+0000 7f1a9259c640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a940cf4d0 0x7f1a9413f4d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1a8402ab90 tx=0x7f1a84033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.824+0000 7f1a8b7fe640 1 -- 172.21.15.81:0/737680843 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84031b50 con 0x7f1a940cf4d0 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.824+0000 7f1a8b7fe640 1 -- 172.21.15.81:0/737680843 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a84031de0 con 0x7f1a940cf4d0 2024-06-29T14:23:16.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.824+0000 7f1a98bfb640 1 -- 172.21.15.81:0/737680843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a94140160 con 0x7f1a940cf4d0 2024-06-29T14:23:16.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.824+0000 7f1a8b7fe640 1 -- 172.21.15.81:0/737680843 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84005260 con 0x7f1a940cf4d0 2024-06-29T14:23:16.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.824+0000 7f1a98bfb640 1 -- 172.21.15.81:0/737680843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a940c4e20 con 0x7f1a940cf4d0 2024-06-29T14:23:16.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.825+0000 7f1a98bfb640 1 -- 172.21.15.81:0/737680843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a941403a0 con 0x7f1a940cf4d0 2024-06-29T14:23:16.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.825+0000 7f1a8b7fe640 1 -- 172.21.15.81:0/737680843 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1a84005400 con 0x7f1a940cf4d0 2024-06-29T14:23:16.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.826+0000 7f1a8b7fe640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1a740762d0 0x7f1a740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:16.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.826+0000 7f1a8b7fe640 1 -- 172.21.15.81:0/737680843 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1a840b8740 con 0x7f1a940cf4d0 2024-06-29T14:23:16.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.826+0000 7f1a89d9b640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1a740762d0 0x7f1a740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:16.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.826+0000 7f1a89d9b640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1a740762d0 0x7f1a740787e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c000f60 tx=0x7f1a7c009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:16.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:16.830+0000 7f1a8b7fe640 1 -- 172.21.15.81:0/737680843 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a8403f030 con 0x7f1a940cf4d0 2024-06-29T14:23:17.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.011+0000 7f1a98bfb640 1 -- 172.21.15.81:0/737680843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1a940cef90 con 0x7f1a940cf4d0 2024-06-29T14:23:17.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.012+0000 7f1a8b7fe640 1 -- 172.21.15.81:0/737680843 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f1a84031770 con 0x7f1a940cf4d0 2024-06-29T14:23:17.010 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:23:17.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 -- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1a740762d0 msgr2=0x7f1a740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1a740762d0 0x7f1a740787e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c000f60 tx=0x7f1a7c009040 comp rx=0 tx=0).stop 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 -- 172.21.15.81:0/737680843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a940cf4d0 msgr2=0x7f1a9413f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a940cf4d0 0x7f1a9413f4d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1a8402ab90 tx=0x7f1a84033ba0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 -- 172.21.15.81:0/737680843 shutdown_connections 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1a740762d0 0x7f1a740787e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a940af0a0 0x7f1a940c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a941412f0 0x7f1a940aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 --2- 172.21.15.81:0/737680843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a940cf4d0 0x7f1a9413f4d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 -- 172.21.15.81:0/737680843 >> 172.21.15.81:0/737680843 conn(0x7f1a9406ccc0 msgr2=0x7f1a9415a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 -- 172.21.15.81:0/737680843 shutdown_connections 2024-06-29T14:23:17.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.014+0000 7f1a98bfb640 1 -- 172.21.15.81:0/737680843 wait complete. 2024-06-29T14:23:17.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:23:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.140+0000 7f3f07b9a640 1 -- 172.21.15.81:0/586458936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f0015d750 msgr2=0x7f3f000afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.140+0000 7f3f07b9a640 1 --2- 172.21.15.81:0/586458936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f0015d750 0x7f3f000afb60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3f000cf140 tx=0x7f3efc02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.140+0000 7f3f07b9a640 1 -- 172.21.15.81:0/586458936 shutdown_connections 2024-06-29T14:23:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.140+0000 7f3f07b9a640 1 --2- 172.21.15.81:0/586458936 >> v2:172.21.15.167:3300/0 conn(0x7f3f0015c030 0x7f3f0006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.140+0000 7f3f07b9a640 1 --2- 172.21.15.81:0/586458936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f0015d750 0x7f3f000afb60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.140+0000 7f3f07b9a640 1 -- 172.21.15.81:0/586458936 >> 172.21.15.81:0/586458936 conn(0x7f3f000d0930 msgr2=0x7f3f000d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.140+0000 7f3f07b9a640 1 -- 172.21.15.81:0/586458936 shutdown_connections 2024-06-29T14:23:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.140+0000 7f3f07b9a640 1 -- 172.21.15.81:0/586458936 wait complete. 2024-06-29T14:23:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.141+0000 7f3f07b9a640 1 Processor -- start 2024-06-29T14:23:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.142+0000 7f3f07b9a640 1 -- start start 2024-06-29T14:23:17.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f07b9a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f0015c030 0x7f3f001560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f07b9a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f0015c440 0x7f3f001467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f07b9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f0015d750 0x7f3f001571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f07b9a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0006cbc0 con 0x7f3f0015d750 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f07b9a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0006ca40 con 0x7f3f0015c440 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f07b9a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3f00160290 con 0x7f3f0015c030 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f06137640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f0015d750 0x7f3f001571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f06137640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f0015d750 0x7f3f001571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49794/0 (socket says 172.21.15.81:49794) 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f06137640 1 -- 172.21.15.81:0/1673374516 learned_addr learned my addr 172.21.15.81:0/1673374516 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f05135640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f0015c440 0x7f3f001467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f05936640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f0015c030 0x7f3f001560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f06137640 1 -- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f0015c030 msgr2=0x7f3f001560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f06137640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f0015c030 0x7f3f001560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f06137640 1 -- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f0015c440 msgr2=0x7f3f001467a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f06137640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f0015c440 0x7f3f001467a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f06137640 1 -- 172.21.15.81:0/1673374516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f001569d0 con 0x7f3f0015d750 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f05936640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f0015c030 0x7f3f001560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:17.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.143+0000 7f3f06137640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f0015d750 0x7f3f001571b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800cb10 tx=0x7f3ef800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:17.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.144+0000 7f3ef67fc640 1 -- 172.21.15.81:0/1673374516 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8016020 con 0x7f3f0015d750 2024-06-29T14:23:17.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.144+0000 7f3ef67fc640 1 -- 172.21.15.81:0/1673374516 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ef800ad60 con 0x7f3f0015d750 2024-06-29T14:23:17.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.144+0000 7f3f07b9a640 1 -- 172.21.15.81:0/1673374516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f00156cb0 con 0x7f3f0015d750 2024-06-29T14:23:17.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.144+0000 7f3ef67fc640 1 -- 172.21.15.81:0/1673374516 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8012680 con 0x7f3f0015d750 2024-06-29T14:23:17.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.144+0000 7f3f07b9a640 1 -- 172.21.15.81:0/1673374516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f00152650 con 0x7f3f0015d750 2024-06-29T14:23:17.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.144+0000 7f3f07b9a640 1 -- 172.21.15.81:0/1673374516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f0006e2a0 con 0x7f3f0015d750 2024-06-29T14:23:17.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.145+0000 7f3ef67fc640 1 -- 172.21.15.81:0/1673374516 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3ef8019020 con 0x7f3f0015d750 2024-06-29T14:23:17.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.146+0000 7f3ef67fc640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3ee40762d0 0x7f3ee40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.146+0000 7f3ef67fc640 1 -- 172.21.15.81:0/1673374516 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3ef8097a40 con 0x7f3f0015d750 2024-06-29T14:23:17.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.146+0000 7f3f05936640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3ee40762d0 0x7f3ee40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.147+0000 7f3f05936640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3ee40762d0 0x7f3ee40787e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3efc034000 tx=0x7f3efc004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:17.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.151+0000 7f3ef67fc640 1 -- 172.21.15.81:0/1673374516 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3f0006e2a0 con 0x7f3f0015d750 2024-06-29T14:23:17.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.342+0000 7f3f07b9a640 1 -- 172.21.15.81:0/1673374516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3f00152160 con 0x7f3f0015d750 2024-06-29T14:23:17.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.342+0000 7f3ef67fc640 1 -- 172.21.15.81:0/1673374516 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f3ef8097e20 con 0x7f3f0015d750 2024-06-29T14:23:17.340 INFO:teuthology.orchestra.run.smithi081.stderr:listed 0 entries 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 -- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3ee40762d0 msgr2=0x7f3ee40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3ee40762d0 0x7f3ee40787e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3efc034000 tx=0x7f3efc004730 comp rx=0 tx=0).stop 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 -- 172.21.15.81:0/1673374516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f0015d750 msgr2=0x7f3f001571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f0015d750 0x7f3f001571b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800cb10 tx=0x7f3ef800a060 comp rx=0 tx=0).stop 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 -- 172.21.15.81:0/1673374516 shutdown_connections 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3ee40762d0 0x7f3ee40787e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f0015d750 0x7f3f001571b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f0015c440 0x7f3f001467a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 --2- 172.21.15.81:0/1673374516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f0015c030 0x7f3f001560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 -- 172.21.15.81:0/1673374516 >> 172.21.15.81:0/1673374516 conn(0x7f3f000d0930 msgr2=0x7f3f0015b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 -- 172.21.15.81:0/1673374516 shutdown_connections 2024-06-29T14:23:17.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.344+0000 7f3f07b9a640 1 -- 172.21.15.81:0/1673374516 wait complete. 2024-06-29T14:23:17.354 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:23:17.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:17.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-06-29T14:23:17.380 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:23:17.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:17.380 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:23:17.444 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:23:17.444 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:17.463 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:23:17.463 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:17.520 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:23:17.520 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:17.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:17.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.693+0000 7f9cd9303640 1 -- 172.21.15.81:0/3944409121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc003a50 msgr2=0x7f9cbc003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.693+0000 7f9cd9303640 1 --2- 172.21.15.81:0/3944409121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc003a50 0x7f9cbc003ec0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4001dc0 tx=0x7f9cbc005830 comp rx=0 tx=0).stop 2024-06-29T14:23:17.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.693+0000 7f9cd9303640 1 -- 172.21.15.81:0/3944409121 shutdown_connections 2024-06-29T14:23:17.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.693+0000 7f9cd9303640 1 --2- 172.21.15.81:0/3944409121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc003a50 0x7f9cbc003ec0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.693+0000 7f9cd9303640 1 -- 172.21.15.81:0/3944409121 >> 172.21.15.81:0/3944409121 conn(0x7f9cd406ccc0 msgr2=0x7f9cd40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:17.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.693+0000 7f9cd9303640 1 -- 172.21.15.81:0/3944409121 shutdown_connections 2024-06-29T14:23:17.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.693+0000 7f9cd9303640 1 -- 172.21.15.81:0/3944409121 wait complete. 2024-06-29T14:23:17.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.695+0000 7f9cd9303640 1 Processor -- start 2024-06-29T14:23:17.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.695+0000 7f9cd9303640 1 -- start start 2024-06-29T14:23:17.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.696+0000 7f9cd9303640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc003a50 0x7f9cd40ac800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.696+0000 7f9cd9303640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9cd41433d0 0x7f9cd40ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.696+0000 7f9cd9303640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cd40ae7a0 0x7f9cd40aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.696+0000 7f9cd9303640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd40737d0 con 0x7f9cbc003a50 2024-06-29T14:23:17.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.696+0000 7f9cd9303640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd4073650 con 0x7f9cd40ae7a0 2024-06-29T14:23:17.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.696+0000 7f9cd9303640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd415ed70 con 0x7f9cd41433d0 2024-06-29T14:23:17.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.696+0000 7f9cd2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc003a50 0x7f9cd40ac800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.696+0000 7f9cd2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc003a50 0x7f9cd40ac800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49812/0 (socket says 172.21.15.81:49812) 2024-06-29T14:23:17.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.696+0000 7f9cd2ffd640 1 -- 172.21.15.81:0/2912835677 learned_addr learned my addr 172.21.15.81:0/2912835677 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:17.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd27fc640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9cd41433d0 0x7f9cd40ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd37fe640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cd40ae7a0 0x7f9cd40aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd2ffd640 1 -- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9cd41433d0 msgr2=0x7f9cd40ae260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd2ffd640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9cd41433d0 0x7f9cd40ae260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd2ffd640 1 -- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cd40ae7a0 msgr2=0x7f9cd40aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd2ffd640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cd40ae7a0 0x7f9cd40aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd2ffd640 1 -- 172.21.15.81:0/2912835677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd4143b40 con 0x7f9cbc003a50 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd37fe640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cd40ae7a0 0x7f9cd40aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd2ffd640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc003a50 0x7f9cd40ac800 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4000f80 tx=0x7f9cc40039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/2912835677 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4006230 con 0x7f9cbc003a50 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/2912835677 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9cc40063f0 con 0x7f9cbc003a50 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd9303640 1 -- 172.21.15.81:0/2912835677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd4143e20 con 0x7f9cbc003a50 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/2912835677 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4006840 con 0x7f9cbc003a50 2024-06-29T14:23:17.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.697+0000 7f9cd9303640 1 -- 172.21.15.81:0/2912835677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd40ab160 con 0x7f9cbc003a50 2024-06-29T14:23:17.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.698+0000 7f9cd9303640 1 -- 172.21.15.81:0/2912835677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cd40cdba0 con 0x7f9cbc003a50 2024-06-29T14:23:17.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.699+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/2912835677 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9cc4033480 con 0x7f9cbc003a50 2024-06-29T14:23:17.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.699+0000 7f9cbb7fe640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9cc0076300 0x7f9cc0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:17.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.699+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/2912835677 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9cc40b79a0 con 0x7f9cbc003a50 2024-06-29T14:23:17.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.699+0000 7f9cd27fc640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9cc0076300 0x7f9cc0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:17.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.700+0000 7f9cd27fc640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9cc0076300 0x7f9cc0078810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9cd415e5c0 tx=0x7f9cc800b260 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:17.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.704+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/2912835677 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9cd40cdba0 con 0x7f9cbc003a50 2024-06-29T14:23:17.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.883+0000 7f9cd9303640 1 -- 172.21.15.81:0/2912835677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9cd40a9a40 con 0x7f9cbc003a50 2024-06-29T14:23:17.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.885+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/2912835677 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f9cc4003b60 con 0x7f9cbc003a50 2024-06-29T14:23:17.883 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:17.884 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:17.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 -- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9cc0076300 msgr2=0x7f9cc0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9cc0076300 0x7f9cc0078810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9cd415e5c0 tx=0x7f9cc800b260 comp rx=0 tx=0).stop 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 -- 172.21.15.81:0/2912835677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc003a50 msgr2=0x7f9cd40ac800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc003a50 0x7f9cd40ac800 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4000f80 tx=0x7f9cc40039f0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 -- 172.21.15.81:0/2912835677 shutdown_connections 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f9cc0076300 0x7f9cc0078810 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cd40ae7a0 0x7f9cd40aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9cd41433d0 0x7f9cd40ae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 --2- 172.21.15.81:0/2912835677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc003a50 0x7f9cd40ac800 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 -- 172.21.15.81:0/2912835677 >> 172.21.15.81:0/2912835677 conn(0x7f9cd406ccc0 msgr2=0x7f9cd40b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 -- 172.21.15.81:0/2912835677 shutdown_connections 2024-06-29T14:23:17.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:17.887+0000 7f9cd9303640 1 -- 172.21.15.81:0/2912835677 wait complete. 2024-06-29T14:23:17.898 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:17.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:18.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.016+0000 7f6688410640 1 -- 172.21.15.81:0/1483417357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66800cf040 msgr2=0x7f66800b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.016+0000 7f6688410640 1 --2- 172.21.15.81:0/1483417357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66800cf040 0x7f66800b3780 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6678005fd0 tx=0x7f667802a4b0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.016+0000 7f6688410640 1 -- 172.21.15.81:0/1483417357 shutdown_connections 2024-06-29T14:23:18.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.016+0000 7f6688410640 1 --2- 172.21.15.81:0/1483417357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66800cf040 0x7f66800b3780 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.016+0000 7f6688410640 1 --2- 172.21.15.81:0/1483417357 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f668008ccd0 0x7f66800b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.016+0000 7f6688410640 1 -- 172.21.15.81:0/1483417357 >> 172.21.15.81:0/1483417357 conn(0x7f66800b56d0 msgr2=0x7f66800b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.016+0000 7f6688410640 1 -- 172.21.15.81:0/1483417357 shutdown_connections 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.016+0000 7f6688410640 1 -- 172.21.15.81:0/1483417357 wait complete. 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.018+0000 7f6688410640 1 Processor -- start 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.018+0000 7f6688410640 1 -- start start 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.019+0000 7f6688410640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668008ccd0 0x7f66800a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.019+0000 7f6688410640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800cf040 0x7f66801f4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.019+0000 7f6688410640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800b6a10 0x7f66800b6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.019+0000 7f6688410640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f66800b4240 con 0x7f668008ccd0 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.019+0000 7f6688410640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f66800b40c0 con 0x7f66800b6a10 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.019+0000 7f6688410640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f66800b43c0 con 0x7f66800cf040 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.019+0000 7f66861ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668008ccd0 0x7f66800a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.019+0000 7f66861ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668008ccd0 0x7f66800a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49834/0 (socket says 172.21.15.81:49834) 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.019+0000 7f66861ac640 1 -- 172.21.15.81:0/874487820 learned_addr learned my addr 172.21.15.81:0/874487820 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.019+0000 7f66859ab640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800cf040 0x7f66801f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f66869ad640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800b6a10 0x7f66800b6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f66861ac640 1 -- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800cf040 msgr2=0x7f66801f4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f66861ac640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800cf040 0x7f66801f4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f66861ac640 1 -- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800b6a10 msgr2=0x7f66800b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f66861ac640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800b6a10 0x7f66800b6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f66861ac640 1 -- 172.21.15.81:0/874487820 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66800b7360 con 0x7f668008ccd0 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f66859ab640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800cf040 0x7f66801f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f66869ad640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800b6a10 0x7f66800b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f66861ac640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668008ccd0 0x7f66800a1520 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f667c00c7a0 tx=0x7f667c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f6676ffd640 1 -- 172.21.15.81:0/874487820 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c017070 con 0x7f668008ccd0 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f6676ffd640 1 -- 172.21.15.81:0/874487820 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f667c005070 con 0x7f668008ccd0 2024-06-29T14:23:18.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f6676ffd640 1 -- 172.21.15.81:0/874487820 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c0054a0 con 0x7f668008ccd0 2024-06-29T14:23:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f6688410640 1 -- 172.21.15.81:0/874487820 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66800a1a60 con 0x7f668008ccd0 2024-06-29T14:23:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.020+0000 7f6688410640 1 -- 172.21.15.81:0/874487820 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668009fd60 con 0x7f668008ccd0 2024-06-29T14:23:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.021+0000 7f6688410640 1 -- 172.21.15.81:0/874487820 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f668008c270 con 0x7f668008ccd0 2024-06-29T14:23:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.022+0000 7f6676ffd640 1 -- 172.21.15.81:0/874487820 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f667c01a070 con 0x7f668008ccd0 2024-06-29T14:23:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.023+0000 7f6676ffd640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66640762d0 0x7f66640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.023+0000 7f6676ffd640 1 -- 172.21.15.81:0/874487820 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f667c096ed0 con 0x7f668008ccd0 2024-06-29T14:23:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.023+0000 7f66859ab640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66640762d0 0x7f66640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.024+0000 7f66859ab640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66640762d0 0x7f66640787e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f668006e5f0 tx=0x7f6670005cb0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:18.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.027+0000 7f6676ffd640 1 -- 172.21.15.81:0/874487820 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f66800b7360 con 0x7f668008ccd0 2024-06-29T14:23:18.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.205+0000 7f6688410640 1 -- 172.21.15.81:0/874487820 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f66801f4850 con 0x7f668008ccd0 2024-06-29T14:23:18.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.205+0000 7f6676ffd640 1 -- 172.21.15.81:0/874487820 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f667c060830 con 0x7f668008ccd0 2024-06-29T14:23:18.204 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:18.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.207+0000 7f6688410640 1 -- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66640762d0 msgr2=0x7f66640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.207+0000 7f6688410640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66640762d0 0x7f66640787e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f668006e5f0 tx=0x7f6670005cb0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.207+0000 7f6688410640 1 -- 172.21.15.81:0/874487820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668008ccd0 msgr2=0x7f66800a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.207+0000 7f6688410640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668008ccd0 0x7f66800a1520 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f667c00c7a0 tx=0x7f667c00cc60 comp rx=0 tx=0).stop 2024-06-29T14:23:18.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.207+0000 7f6688410640 1 -- 172.21.15.81:0/874487820 shutdown_connections 2024-06-29T14:23:18.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.207+0000 7f6688410640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f66640762d0 0x7f66640787e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.207+0000 7f6688410640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800b6a10 0x7f66800b6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.207+0000 7f6688410640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800cf040 0x7f66801f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.207+0000 7f6688410640 1 --2- 172.21.15.81:0/874487820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668008ccd0 0x7f66800a1520 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.207+0000 7f6688410640 1 -- 172.21.15.81:0/874487820 >> 172.21.15.81:0/874487820 conn(0x7f66800b56d0 msgr2=0x7f668006cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:18.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.208+0000 7f6688410640 1 -- 172.21.15.81:0/874487820 shutdown_connections 2024-06-29T14:23:18.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.208+0000 7f6688410640 1 -- 172.21.15.81:0/874487820 wait complete. 2024-06-29T14:23:18.218 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:23:18.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.336+0000 7f3956e65640 1 -- 172.21.15.81:0/3373487877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39500b6d60 msgr2=0x7f39500ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.336+0000 7f3956e65640 1 --2- 172.21.15.81:0/3373487877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39500b6d60 0x7f39500ce4d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f39500d2600 tx=0x7f394002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.336+0000 7f3956e65640 1 -- 172.21.15.81:0/3373487877 shutdown_connections 2024-06-29T14:23:18.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.336+0000 7f3956e65640 1 --2- 172.21.15.81:0/3373487877 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39500cd540 0x7f39500d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.336+0000 7f3956e65640 1 --2- 172.21.15.81:0/3373487877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39500b6d60 0x7f39500ce4d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.336+0000 7f3956e65640 1 -- 172.21.15.81:0/3373487877 >> 172.21.15.81:0/3373487877 conn(0x7f39500d1960 msgr2=0x7f39500d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:18.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.336+0000 7f3956e65640 1 -- 172.21.15.81:0/3373487877 shutdown_connections 2024-06-29T14:23:18.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.336+0000 7f3956e65640 1 -- 172.21.15.81:0/3373487877 wait complete. 2024-06-29T14:23:18.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.338+0000 7f3956e65640 1 Processor -- start 2024-06-29T14:23:18.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.338+0000 7f3956e65640 1 -- start start 2024-06-29T14:23:18.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.339+0000 7f3956e65640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39500b6d60 0x7f395015edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f3956e65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39500cd540 0x7f39500cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f3956e65640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39500cea10 0x7f39500cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f3956e65640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39500d0630 con 0x7f39500cd540 2024-06-29T14:23:18.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f3956e65640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f39500b2790 con 0x7f39500b6d60 2024-06-29T14:23:18.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f3956e65640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f39500d07b0 con 0x7f39500cea10 2024-06-29T14:23:18.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39500cd540 0x7f39500cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39500cd540 0x7f39500cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49856/0 (socket says 172.21.15.81:49856) 2024-06-29T14:23:18.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394ffff640 1 -- 172.21.15.81:0/2527118051 learned_addr learned my addr 172.21.15.81:0/2527118051 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:18.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f3954c01640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39500b6d60 0x7f395015edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f3955402640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39500cea10 0x7f39500cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394ffff640 1 -- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39500cea10 msgr2=0x7f39500cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394ffff640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39500cea10 0x7f39500cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394ffff640 1 -- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39500b6d60 msgr2=0x7f395015edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394ffff640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39500b6d60 0x7f395015edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394ffff640 1 -- 172.21.15.81:0/2527118051 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395015fb80 con 0x7f39500cd540 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f3955402640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39500cea10 0x7f39500cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394ffff640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39500cd540 0x7f39500cabe0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f394400c730 tx=0x7f394400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394d7fa640 1 -- 172.21.15.81:0/2527118051 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3944003da0 con 0x7f39500cd540 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394d7fa640 1 -- 172.21.15.81:0/2527118051 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f394400ce40 con 0x7f39500cd540 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f394d7fa640 1 -- 172.21.15.81:0/2527118051 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39440052d0 con 0x7f39500cd540 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.340+0000 7f3956e65640 1 -- 172.21.15.81:0/2527118051 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3950165d90 con 0x7f39500cd540 2024-06-29T14:23:18.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.341+0000 7f3956e65640 1 -- 172.21.15.81:0/2527118051 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3950166350 con 0x7f39500cd540 2024-06-29T14:23:18.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.342+0000 7f3956e65640 1 -- 172.21.15.81:0/2527118051 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39500c8710 con 0x7f39500cd540 2024-06-29T14:23:18.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.342+0000 7f394d7fa640 1 -- 172.21.15.81:0/2527118051 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3944005470 con 0x7f39500cd540 2024-06-29T14:23:18.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.343+0000 7f394d7fa640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f393c076300 0x7f393c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.343+0000 7f394d7fa640 1 -- 172.21.15.81:0/2527118051 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f39440974c0 con 0x7f39500cd540 2024-06-29T14:23:18.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.343+0000 7f3954c01640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f393c076300 0x7f393c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.344+0000 7f3954c01640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f393c076300 0x7f393c078810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3940033000 tx=0x7f3940004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:18.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.347+0000 7f394d7fa640 1 -- 172.21.15.81:0/2527118051 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3944060e20 con 0x7f39500cd540 2024-06-29T14:23:18.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.533+0000 7f3956e65640 1 -- 172.21.15.81:0/2527118051 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f39500ce4d0 con 0x7f39500cd540 2024-06-29T14:23:18.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.533+0000 7f394d7fa640 1 -- 172.21.15.81:0/2527118051 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f3944060720 con 0x7f39500cd540 2024-06-29T14:23:18.531 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:23:18.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 -- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f393c076300 msgr2=0x7f393c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f393c076300 0x7f393c078810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3940033000 tx=0x7f3940004730 comp rx=0 tx=0).stop 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 -- 172.21.15.81:0/2527118051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39500cd540 msgr2=0x7f39500cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39500cd540 0x7f39500cabe0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f394400c730 tx=0x7f394400cbf0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 -- 172.21.15.81:0/2527118051 shutdown_connections 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f393c076300 0x7f393c078810 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39500cea10 0x7f39500cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39500cd540 0x7f39500cabe0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 --2- 172.21.15.81:0/2527118051 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39500b6d60 0x7f395015edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 -- 172.21.15.81:0/2527118051 >> 172.21.15.81:0/2527118051 conn(0x7f39500d1960 msgr2=0x7f39500b5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 -- 172.21.15.81:0/2527118051 shutdown_connections 2024-06-29T14:23:18.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.535+0000 7f3956e65640 1 -- 172.21.15.81:0/2527118051 wait complete. 2024-06-29T14:23:18.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:23:18.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.663+0000 7f2219c66640 1 -- 172.21.15.81:0/1328057625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22140b6eb0 msgr2=0x7f22140d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.663+0000 7f2219c66640 1 --2- 172.21.15.81:0/1328057625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22140b6eb0 0x7f22140d02d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2214073550 tx=0x7f220402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.663+0000 7f2219c66640 1 -- 172.21.15.81:0/1328057625 shutdown_connections 2024-06-29T14:23:18.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.663+0000 7f2219c66640 1 --2- 172.21.15.81:0/1328057625 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22140b4810 0x7f22140b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.663+0000 7f2219c66640 1 --2- 172.21.15.81:0/1328057625 >> v2:172.21.15.167:3300/0 conn(0x7f22140ce510 0x7f22140ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.663+0000 7f2219c66640 1 --2- 172.21.15.81:0/1328057625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22140b6eb0 0x7f22140d02d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.663+0000 7f2219c66640 1 -- 172.21.15.81:0/1328057625 >> 172.21.15.81:0/1328057625 conn(0x7f22140b3510 msgr2=0x7f22140b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:18.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.663+0000 7f2219c66640 1 -- 172.21.15.81:0/1328057625 shutdown_connections 2024-06-29T14:23:18.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.663+0000 7f2219c66640 1 -- 172.21.15.81:0/1328057625 wait complete. 2024-06-29T14:23:18.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.665+0000 7f2219c66640 1 Processor -- start 2024-06-29T14:23:18.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.665+0000 7f2219c66640 1 -- start start 2024-06-29T14:23:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f2219c66640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22140b4810 0x7f221406eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f2219c66640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22140b6eb0 0x7f22140ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f2219c66640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22140ce510 0x7f22140ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f2219c66640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f22140b52e0 con 0x7f22140b6eb0 2024-06-29T14:23:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f2219c66640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f22140b5160 con 0x7f22140ce510 2024-06-29T14:23:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f2219c66640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f22140cc470 con 0x7f22140b4810 2024-06-29T14:23:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f2212ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22140b6eb0 0x7f22140ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f2212ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22140b6eb0 0x7f22140ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49872/0 (socket says 172.21.15.81:49872) 2024-06-29T14:23:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f2212ffd640 1 -- 172.21.15.81:0/2358073810 learned_addr learned my addr 172.21.15.81:0/2358073810 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:18.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f2213fff640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22140ce510 0x7f22140ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.666+0000 7f22137fe640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22140b4810 0x7f221406eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f2212ffd640 1 -- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22140b4810 msgr2=0x7f221406eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f2212ffd640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22140b4810 0x7f221406eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f2212ffd640 1 -- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22140ce510 msgr2=0x7f22140ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f2212ffd640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22140ce510 0x7f22140ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f2212ffd640 1 -- 172.21.15.81:0/2358073810 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22141e5d80 con 0x7f22140b6eb0 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f2212ffd640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22140b6eb0 0x7f22140ca0b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f220800d740 tx=0x7f220800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f21fbfff640 1 -- 172.21.15.81:0/2358073810 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2208017070 con 0x7f22140b6eb0 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f21fbfff640 1 -- 172.21.15.81:0/2358073810 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2208009cc0 con 0x7f22140b6eb0 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f2219c66640 1 -- 172.21.15.81:0/2358073810 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f221406f3e0 con 0x7f22140b6eb0 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f21fbfff640 1 -- 172.21.15.81:0/2358073810 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22080106c0 con 0x7f22140b6eb0 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.667+0000 7f2219c66640 1 -- 172.21.15.81:0/2358073810 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22141e54e0 con 0x7f22140b6eb0 2024-06-29T14:23:18.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.668+0000 7f2219c66640 1 -- 172.21.15.81:0/2358073810 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22141e4f70 con 0x7f22140b6eb0 2024-06-29T14:23:18.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.669+0000 7f21fbfff640 1 -- 172.21.15.81:0/2358073810 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f220801a070 con 0x7f22140b6eb0 2024-06-29T14:23:18.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.669+0000 7f21fbfff640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f22000762d0 0x7f22000787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.669+0000 7f21fbfff640 1 -- 172.21.15.81:0/2358073810 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f22080977d0 con 0x7f22140b6eb0 2024-06-29T14:23:18.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.669+0000 7f22137fe640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f22000762d0 0x7f22000787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.670+0000 7f22137fe640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f22000762d0 0x7f22000787e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2204033000 tx=0x7f2204004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:18.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.673+0000 7f21fbfff640 1 -- 172.21.15.81:0/2358073810 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2208060ae0 con 0x7f22140b6eb0 2024-06-29T14:23:18.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.855+0000 7f2219c66640 1 -- 172.21.15.81:0/2358073810 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f22140d02d0 con 0x7f22140b6eb0 2024-06-29T14:23:18.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.855+0000 7f21fbfff640 1 -- 172.21.15.81:0/2358073810 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f2208064de0 con 0x7f22140b6eb0 2024-06-29T14:23:18.855 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:23:18.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.857+0000 7f2219c66640 1 -- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f22000762d0 msgr2=0x7f22000787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.857+0000 7f2219c66640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f22000762d0 0x7f22000787e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2204033000 tx=0x7f2204004730 comp rx=0 tx=0).stop 2024-06-29T14:23:18.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.857+0000 7f2219c66640 1 -- 172.21.15.81:0/2358073810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22140b6eb0 msgr2=0x7f22140ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.857+0000 7f2219c66640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22140b6eb0 0x7f22140ca0b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f220800d740 tx=0x7f220800dc00 comp rx=0 tx=0).stop 2024-06-29T14:23:18.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.857+0000 7f2219c66640 1 -- 172.21.15.81:0/2358073810 shutdown_connections 2024-06-29T14:23:18.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.857+0000 7f2219c66640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f22000762d0 0x7f22000787e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.857+0000 7f2219c66640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22140ce510 0x7f22140ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.857+0000 7f2219c66640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22140b6eb0 0x7f22140ca0b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.857+0000 7f2219c66640 1 --2- 172.21.15.81:0/2358073810 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22140b4810 0x7f221406eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.857+0000 7f2219c66640 1 -- 172.21.15.81:0/2358073810 >> 172.21.15.81:0/2358073810 conn(0x7f22140b3510 msgr2=0x7f2214071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:18.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.858+0000 7f2219c66640 1 -- 172.21.15.81:0/2358073810 shutdown_connections 2024-06-29T14:23:18.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.858+0000 7f2219c66640 1 -- 172.21.15.81:0/2358073810 wait complete. 2024-06-29T14:23:18.867 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:23:18.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.983+0000 7f7696e9e640 1 -- 172.21.15.81:0/3615978946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76900ae840 msgr2=0x7f76900aec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.983+0000 7f7696e9e640 1 --2- 172.21.15.81:0/3615978946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76900ae840 0x7f76900aec50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f769006e210 tx=0x7f768802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.983+0000 7f7696e9e640 1 -- 172.21.15.81:0/3615978946 shutdown_connections 2024-06-29T14:23:18.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.983+0000 7f7696e9e640 1 --2- 172.21.15.81:0/3615978946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76900ae840 0x7f76900aec50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.983+0000 7f7696e9e640 1 --2- 172.21.15.81:0/3615978946 >> v2:172.21.15.167:3300/0 conn(0x7f769006f850 0x7f76900ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.983+0000 7f7696e9e640 1 -- 172.21.15.81:0/3615978946 >> 172.21.15.81:0/3615978946 conn(0x7f7690072950 msgr2=0x7f76900b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:18.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.983+0000 7f7696e9e640 1 -- 172.21.15.81:0/3615978946 shutdown_connections 2024-06-29T14:23:18.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.983+0000 7f7696e9e640 1 -- 172.21.15.81:0/3615978946 wait complete. 2024-06-29T14:23:18.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.985+0000 7f7696e9e640 1 Processor -- start 2024-06-29T14:23:18.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.985+0000 7f7696e9e640 1 -- start start 2024-06-29T14:23:18.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7696e9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f769006f850 0x7f7690151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7696e9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7690074520 0x7f76900cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7696e9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76900ae840 0x7f76900cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7696e9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f769015b190 con 0x7f7690074520 2024-06-29T14:23:18.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7696e9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f769015b010 con 0x7f769006f850 2024-06-29T14:23:18.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7696e9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f76900af190 con 0x7f76900ae840 2024-06-29T14:23:18.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7687fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7690074520 0x7f76900cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7687fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7690074520 0x7f76900cdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49890/0 (socket says 172.21.15.81:49890) 2024-06-29T14:23:18.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7687fff640 1 -- 172.21.15.81:0/544493195 learned_addr learned my addr 172.21.15.81:0/544493195 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7694c3a640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f769006f850 0x7f7690151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f769543b640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76900ae840 0x7f76900cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7687fff640 1 -- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76900ae840 msgr2=0x7f76900cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7687fff640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76900ae840 0x7f76900cbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7687fff640 1 -- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f769006f850 msgr2=0x7f7690151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7687fff640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f769006f850 0x7f7690151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7687fff640 1 -- 172.21.15.81:0/544493195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76900c9a20 con 0x7f7690074520 2024-06-29T14:23:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7687fff640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7690074520 0x7f76900cdb10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f768000d700 tx=0x7f768000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f76857fa640 1 -- 172.21.15.81:0/544493195 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680017070 con 0x7f7690074520 2024-06-29T14:23:18.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f76857fa640 1 -- 172.21.15.81:0/544493195 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7680005310 con 0x7f7690074520 2024-06-29T14:23:18.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.987+0000 7f7696e9e640 1 -- 172.21.15.81:0/544493195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7690149990 con 0x7f7690074520 2024-06-29T14:23:18.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.988+0000 7f76857fa640 1 -- 172.21.15.81:0/544493195 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76800097c0 con 0x7f7690074520 2024-06-29T14:23:18.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.988+0000 7f7696e9e640 1 -- 172.21.15.81:0/544493195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76900c9460 con 0x7f7690074520 2024-06-29T14:23:18.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.988+0000 7f7696e9e640 1 -- 172.21.15.81:0/544493195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76900ad590 con 0x7f7690074520 2024-06-29T14:23:18.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.989+0000 7f76857fa640 1 -- 172.21.15.81:0/544493195 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7680009960 con 0x7f7690074520 2024-06-29T14:23:18.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.990+0000 7f76857fa640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f767c0762d0 0x7f767c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:18.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.990+0000 7f76857fa640 1 -- 172.21.15.81:0/544493195 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7680097810 con 0x7f7690074520 2024-06-29T14:23:18.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.990+0000 7f7694c3a640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f767c0762d0 0x7f767c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:18.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.991+0000 7f7694c3a640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f767c0762d0 0x7f767c0787e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f768c007fd0 tx=0x7f768c009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:18.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:18.994+0000 7f76857fa640 1 -- 172.21.15.81:0/544493195 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76900c9a20 con 0x7f7690074520 2024-06-29T14:23:19.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.186+0000 7f7696e9e640 1 -- 172.21.15.81:0/544493195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f76900ae300 con 0x7f7690074520 2024-06-29T14:23:19.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.186+0000 7f76857fa640 1 -- 172.21.15.81:0/544493195 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f7680061170 con 0x7f7690074520 2024-06-29T14:23:19.184 INFO:teuthology.orchestra.run.smithi081.stderr:listed 0 entries 2024-06-29T14:23:19.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 -- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f767c0762d0 msgr2=0x7f767c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f767c0762d0 0x7f767c0787e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f768c007fd0 tx=0x7f768c009040 comp rx=0 tx=0).stop 2024-06-29T14:23:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 -- 172.21.15.81:0/544493195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7690074520 msgr2=0x7f76900cdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7690074520 0x7f76900cdb10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f768000d700 tx=0x7f768000dbc0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 -- 172.21.15.81:0/544493195 shutdown_connections 2024-06-29T14:23:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f767c0762d0 0x7f767c0787e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76900ae840 0x7f76900cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7690074520 0x7f76900cdb10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 --2- 172.21.15.81:0/544493195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f769006f850 0x7f7690151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 -- 172.21.15.81:0/544493195 >> 172.21.15.81:0/544493195 conn(0x7f7690072950 msgr2=0x7f76900afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 -- 172.21.15.81:0/544493195 shutdown_connections 2024-06-29T14:23:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.188+0000 7f7696e9e640 1 -- 172.21.15.81:0/544493195 wait complete. 2024-06-29T14:23:19.197 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-06-29T14:23:19.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:19.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-06-29T14:23:19.230 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T14:23:19.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:19.230 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-06-29T14:23:19.296 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:23:19.296 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:19.313 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:23:19.313 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:19.368 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:23:19.368 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:19.424 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:19.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.579+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3969918772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c68073b10 msgr2=0x7f8c68073f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.579+0000 7f8c6eae4640 1 --2- 172.21.15.81:0/3969918772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c68073b10 0x7f8c68073f20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8c58005fd0 tx=0x7f8c5802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.579+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3969918772 shutdown_connections 2024-06-29T14:23:19.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.579+0000 7f8c6eae4640 1 --2- 172.21.15.81:0/3969918772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c68073b10 0x7f8c68073f20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.579+0000 7f8c6eae4640 1 --2- 172.21.15.81:0/3969918772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c680ad0c0 0x7f8c6815bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.579+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3969918772 >> 172.21.15.81:0/3969918772 conn(0x7f8c68072950 msgr2=0x7f8c680b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:19.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.579+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3969918772 shutdown_connections 2024-06-29T14:23:19.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.579+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3969918772 wait complete. 2024-06-29T14:23:19.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.581+0000 7f8c6eae4640 1 Processor -- start 2024-06-29T14:23:19.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.581+0000 7f8c6eae4640 1 -- start start 2024-06-29T14:23:19.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6eae4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c68073b10 0x7f8c6814e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6eae4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c68074460 0x7f8c680b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6eae4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c680ad0c0 0x7f8c680b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6eae4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c680af300 con 0x7f8c68073b10 2024-06-29T14:23:19.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6eae4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8c680af180 con 0x7f8c680ad0c0 2024-06-29T14:23:19.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6eae4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8c680af480 con 0x7f8c68074460 2024-06-29T14:23:19.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6c880640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c68073b10 0x7f8c6814e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6c880640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c68073b10 0x7f8c6814e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49894/0 (socket says 172.21.15.81:49894) 2024-06-29T14:23:19.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6c880640 1 -- 172.21.15.81:0/3536934330 learned_addr learned my addr 172.21.15.81:0/3536934330 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c5ffff640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c68074460 0x7f8c680b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6c880640 1 -- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c68074460 msgr2=0x7f8c680b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6c880640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c68074460 0x7f8c680b1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.583+0000 7f8c6c880640 1 -- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c680ad0c0 msgr2=0x7f8c680b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.582+0000 7f8c6d081640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c680ad0c0 0x7f8c680b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.583+0000 7f8c6c880640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c680ad0c0 0x7f8c680b2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.583+0000 7f8c6c880640 1 -- 172.21.15.81:0/3536934330 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c681f9490 con 0x7f8c68073b10 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.583+0000 7f8c6c880640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c68073b10 0x7f8c6814e3b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400d700 tx=0x7f8c6400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.583+0000 7f8c5d7fa640 1 -- 172.21.15.81:0/3536934330 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64017070 con 0x7f8c68073b10 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.583+0000 7f8c5d7fa640 1 -- 172.21.15.81:0/3536934330 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c64005310 con 0x7f8c68073b10 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.583+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3536934330 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c681f86d0 con 0x7f8c68073b10 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.583+0000 7f8c5d7fa640 1 -- 172.21.15.81:0/3536934330 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64009820 con 0x7f8c68073b10 2024-06-29T14:23:19.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.583+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3536934330 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c681f8c90 con 0x7f8c68073b10 2024-06-29T14:23:19.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.584+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3536934330 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c680af930 con 0x7f8c68073b10 2024-06-29T14:23:19.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.585+0000 7f8c5d7fa640 1 -- 172.21.15.81:0/3536934330 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8c640099c0 con 0x7f8c68073b10 2024-06-29T14:23:19.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.585+0000 7f8c5d7fa640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c540762d0 0x7f8c540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.585+0000 7f8c5d7fa640 1 -- 172.21.15.81:0/3536934330 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8c64097880 con 0x7f8c68073b10 2024-06-29T14:23:19.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.585+0000 7f8c5ffff640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c540762d0 0x7f8c540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.586+0000 7f8c5ffff640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c540762d0 0x7f8c540787e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8c58033000 tx=0x7f8c58004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:19.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.589+0000 7f8c5d7fa640 1 -- 172.21.15.81:0/3536934330 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8c640611e0 con 0x7f8c68073b10 2024-06-29T14:23:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.769+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3536934330 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c680b11c0 con 0x7f8c68073b10 2024-06-29T14:23:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.771+0000 7f8c5d7fa640 1 -- 172.21.15.81:0/3536934330 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7f8c640691d0 con 0x7f8c68073b10 2024-06-29T14:23:19.769 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:19.769 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:19.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c540762d0 msgr2=0x7f8c540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c540762d0 0x7f8c540787e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8c58033000 tx=0x7f8c58004730 comp rx=0 tx=0).stop 2024-06-29T14:23:19.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3536934330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c68073b10 msgr2=0x7f8c6814e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c68073b10 0x7f8c6814e3b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400d700 tx=0x7f8c6400dbc0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3536934330 shutdown_connections 2024-06-29T14:23:19.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8c540762d0 0x7f8c540787e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c680ad0c0 0x7f8c680b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c68074460 0x7f8c680b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 --2- 172.21.15.81:0/3536934330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c68073b10 0x7f8c6814e3b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3536934330 >> 172.21.15.81:0/3536934330 conn(0x7f8c68072950 msgr2=0x7f8c6815c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:19.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3536934330 shutdown_connections 2024-06-29T14:23:19.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.773+0000 7f8c6eae4640 1 -- 172.21.15.81:0/3536934330 wait complete. 2024-06-29T14:23:19.782 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:19.783 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:19.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.902+0000 7fe63a466640 1 -- 172.21.15.81:0/3706902293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6341412f0 msgr2=0x7fe6340cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.902+0000 7fe63a466640 1 --2- 172.21.15.81:0/3706902293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6341412f0 0x7fe6340cef90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe634071d20 tx=0x7fe61c02a530 comp rx=0 tx=0).stop 2024-06-29T14:23:19.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.903+0000 7fe63a466640 1 -- 172.21.15.81:0/3706902293 shutdown_connections 2024-06-29T14:23:19.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.903+0000 7fe63a466640 1 --2- 172.21.15.81:0/3706902293 >> v2:172.21.15.167:3300/0 conn(0x7fe634073740 0x7fe63415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.903+0000 7fe63a466640 1 --2- 172.21.15.81:0/3706902293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6341412f0 0x7fe6340cef90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.903+0000 7fe63a466640 1 -- 172.21.15.81:0/3706902293 >> 172.21.15.81:0/3706902293 conn(0x7fe63406ccc0 msgr2=0x7fe6340b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:19.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.903+0000 7fe63a466640 1 -- 172.21.15.81:0/3706902293 shutdown_connections 2024-06-29T14:23:19.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.903+0000 7fe63a466640 1 -- 172.21.15.81:0/3706902293 wait complete. 2024-06-29T14:23:19.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.904+0000 7fe63a466640 1 Processor -- start 2024-06-29T14:23:19.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.905+0000 7fe63a466640 1 -- start start 2024-06-29T14:23:19.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe63a466640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6340cf4d0 0x7fe634140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe63a466640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6341412f0 0x7fe6341563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe63a466640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe634156930 0x7fe6341590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe63a466640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6340b3210 con 0x7fe6340cf4d0 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe63a466640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe63415f1f0 con 0x7fe634156930 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe63a466640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe6340b3390 con 0x7fe6341412f0 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe633fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6340cf4d0 0x7fe634140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe633fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6340cf4d0 0x7fe634140020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49914/0 (socket says 172.21.15.81:49914) 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe633fff640 1 -- 172.21.15.81:0/1963101850 learned_addr learned my addr 172.21.15.81:0/1963101850 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe638a03640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe634156930 0x7fe6341590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe6337fe640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6341412f0 0x7fe6341563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe633fff640 1 -- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6341412f0 msgr2=0x7fe6341563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe633fff640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6341412f0 0x7fe6341563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe633fff640 1 -- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe634156930 msgr2=0x7fe6341590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:19.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe633fff640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe634156930 0x7fe6341590f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:19.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe633fff640 1 -- 172.21.15.81:0/1963101850 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6341f7300 con 0x7fe6340cf4d0 2024-06-29T14:23:19.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.906+0000 7fe633fff640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6340cf4d0 0x7fe634140020 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009920 tx=0x7fe61c033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:19.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.907+0000 7fe630ff9640 1 -- 172.21.15.81:0/1963101850 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c031d60 con 0x7fe6340cf4d0 2024-06-29T14:23:19.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.907+0000 7fe630ff9640 1 -- 172.21.15.81:0/1963101850 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe61c005020 con 0x7fe6340cf4d0 2024-06-29T14:23:19.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.907+0000 7fe630ff9640 1 -- 172.21.15.81:0/1963101850 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c005470 con 0x7fe6340cf4d0 2024-06-29T14:23:19.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.907+0000 7fe63a466640 1 -- 172.21.15.81:0/1963101850 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6341f6780 con 0x7fe6340cf4d0 2024-06-29T14:23:19.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.907+0000 7fe63a466640 1 -- 172.21.15.81:0/1963101850 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6341f6d40 con 0x7fe6340cf4d0 2024-06-29T14:23:19.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.908+0000 7fe63a466640 1 -- 172.21.15.81:0/1963101850 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6340b38b0 con 0x7fe6340cf4d0 2024-06-29T14:23:19.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.908+0000 7fe630ff9640 1 -- 172.21.15.81:0/1963101850 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe61c0047a0 con 0x7fe6340cf4d0 2024-06-29T14:23:19.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.909+0000 7fe630ff9640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6140762d0 0x7fe6140787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:19.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.909+0000 7fe630ff9640 1 -- 172.21.15.81:0/1963101850 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe61c0b8870 con 0x7fe6340cf4d0 2024-06-29T14:23:19.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.909+0000 7fe6337fe640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6140762d0 0x7fe6140787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:19.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.910+0000 7fe6337fe640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6140762d0 0x7fe6140787e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe634159f70 tx=0x7fe624009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:19.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:19.913+0000 7fe630ff9640 1 -- 172.21.15.81:0/1963101850 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe61c03f030 con 0x7fe6340cf4d0 2024-06-29T14:23:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.092+0000 7fe63a466640 1 -- 172.21.15.81:0/1963101850 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe63406a610 con 0x7fe6340cf4d0 2024-06-29T14:23:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.092+0000 7fe630ff9640 1 -- 172.21.15.81:0/1963101850 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fe61c031770 con 0x7fe6340cf4d0 2024-06-29T14:23:20.090 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 -- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6140762d0 msgr2=0x7fe6140787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6140762d0 0x7fe6140787e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe634159f70 tx=0x7fe624009040 comp rx=0 tx=0).stop 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 -- 172.21.15.81:0/1963101850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6340cf4d0 msgr2=0x7fe634140020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6340cf4d0 0x7fe634140020 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009920 tx=0x7fe61c033c90 comp rx=0 tx=0).stop 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 -- 172.21.15.81:0/1963101850 shutdown_connections 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe6140762d0 0x7fe6140787e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe634156930 0x7fe6341590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6341412f0 0x7fe6341563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 --2- 172.21.15.81:0/1963101850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6340cf4d0 0x7fe634140020 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 -- 172.21.15.81:0/1963101850 >> 172.21.15.81:0/1963101850 conn(0x7fe63406ccc0 msgr2=0x7fe634073030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 -- 172.21.15.81:0/1963101850 shutdown_connections 2024-06-29T14:23:20.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.094+0000 7fe63a466640 1 -- 172.21.15.81:0/1963101850 wait complete. 2024-06-29T14:23:20.104 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:23:20.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.223+0000 7fe35428f640 1 -- 172.21.15.81:0/1128022456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe338004800 msgr2=0x7fe338004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.223+0000 7fe35428f640 1 --2- 172.21.15.81:0/1128022456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe338004800 0x7fe338004c10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009b50 tx=0x7fe338005830 comp rx=0 tx=0).stop 2024-06-29T14:23:20.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.223+0000 7fe35282c640 1 -- 172.21.15.81:0/1128022456 reap_dead start 2024-06-29T14:23:20.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.223+0000 7fe35428f640 1 -- 172.21.15.81:0/1128022456 shutdown_connections 2024-06-29T14:23:20.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.223+0000 7fe35428f640 1 -- 172.21.15.81:0/1128022456 >> 172.21.15.81:0/1128022456 conn(0x7fe34c06ccc0 msgr2=0x7fe34c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.223+0000 7fe35428f640 1 -- 172.21.15.81:0/1128022456 shutdown_connections 2024-06-29T14:23:20.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.223+0000 7fe35428f640 1 -- 172.21.15.81:0/1128022456 wait complete. 2024-06-29T14:23:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.224+0000 7fe35428f640 1 Processor -- start 2024-06-29T14:23:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.225+0000 7fe35428f640 1 -- start start 2024-06-29T14:23:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35428f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe338004800 0x7fe34c139b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35428f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c152920 0x7fe34c1e5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35428f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe34c13b220 0x7fe34c1596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35428f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c15b6f0 con 0x7fe34c152920 2024-06-29T14:23:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35428f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c15b570 con 0x7fe338004800 2024-06-29T14:23:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35428f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe34c15b870 con 0x7fe34c13b220 2024-06-29T14:23:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35182a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c152920 0x7fe34c1e5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35182a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c152920 0x7fe34c1e5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59720/0 (socket says 172.21.15.81:59720) 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35182a640 1 -- 172.21.15.81:0/1713481610 learned_addr learned my addr 172.21.15.81:0/1713481610 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35202b640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe338004800 0x7fe34c139b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35282c640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe34c13b220 0x7fe34c1596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35182a640 1 -- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe34c13b220 msgr2=0x7fe34c1596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35182a640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe34c13b220 0x7fe34c1596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35182a640 1 -- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe338004800 msgr2=0x7fe34c139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35182a640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe338004800 0x7fe34c139b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35182a640 1 -- 172.21.15.81:0/1713481610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34c1e5da0 con 0x7fe34c152920 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35202b640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe338004800 0x7fe34c139b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.226+0000 7fe35182a640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c152920 0x7fe34c1e5860 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009a70 tx=0x7fe33c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.227+0000 7fe342ffd640 1 -- 172.21.15.81:0/1713481610 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c043070 con 0x7fe34c152920 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.227+0000 7fe35428f640 1 -- 172.21.15.81:0/1713481610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe34c1e6220 con 0x7fe34c152920 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.227+0000 7fe342ffd640 1 -- 172.21.15.81:0/1713481610 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe33c004320 con 0x7fe34c152920 2024-06-29T14:23:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.227+0000 7fe342ffd640 1 -- 172.21.15.81:0/1713481610 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c038040 con 0x7fe34c152920 2024-06-29T14:23:20.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.227+0000 7fe35428f640 1 -- 172.21.15.81:0/1713481610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe34c13aa20 con 0x7fe34c152920 2024-06-29T14:23:20.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.228+0000 7fe35428f640 1 -- 172.21.15.81:0/1713481610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe34c1e5f70 con 0x7fe34c152920 2024-06-29T14:23:20.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.228+0000 7fe342ffd640 1 -- 172.21.15.81:0/1713481610 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe33c02b8a0 con 0x7fe34c152920 2024-06-29T14:23:20.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.229+0000 7fe342ffd640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe330076300 0x7fe330078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.229+0000 7fe342ffd640 1 -- 172.21.15.81:0/1713481610 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe33c0b7da0 con 0x7fe34c152920 2024-06-29T14:23:20.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.229+0000 7fe35202b640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe330076300 0x7fe330078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.229+0000 7fe35202b640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe330076300 0x7fe330078810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe348001be0 tx=0x7fe348002100 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.233+0000 7fe342ffd640 1 -- 172.21.15.81:0/1713481610 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe33c081030 con 0x7fe34c152920 2024-06-29T14:23:20.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.413+0000 7fe35428f640 1 -- 172.21.15.81:0/1713481610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe34c0ce1b0 con 0x7fe34c152920 2024-06-29T14:23:20.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.413+0000 7fe342ffd640 1 -- 172.21.15.81:0/1713481610 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7fe33c085330 con 0x7fe34c152920 2024-06-29T14:23:20.412 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:23:20.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 -- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe330076300 msgr2=0x7fe330078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe330076300 0x7fe330078810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe348001be0 tx=0x7fe348002100 comp rx=0 tx=0).stop 2024-06-29T14:23:20.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 -- 172.21.15.81:0/1713481610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c152920 msgr2=0x7fe34c1e5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c152920 0x7fe34c1e5860 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009a70 tx=0x7fe33c033670 comp rx=0 tx=0).stop 2024-06-29T14:23:20.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 -- 172.21.15.81:0/1713481610 shutdown_connections 2024-06-29T14:23:20.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe330076300 0x7fe330078810 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe34c13b220 0x7fe34c1596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c152920 0x7fe34c1e5860 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 --2- 172.21.15.81:0/1713481610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe338004800 0x7fe34c139b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 -- 172.21.15.81:0/1713481610 >> 172.21.15.81:0/1713481610 conn(0x7fe34c06ccc0 msgr2=0x7fe34c13d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 -- 172.21.15.81:0/1713481610 shutdown_connections 2024-06-29T14:23:20.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.416+0000 7fe35428f640 1 -- 172.21.15.81:0/1713481610 wait complete. 2024-06-29T14:23:20.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:23:20.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.543+0000 7fdc49447640 1 -- 172.21.15.81:0/2818973087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc440b9040 msgr2=0x7fdc440d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.543+0000 7fdc49447640 1 --2- 172.21.15.81:0/2818973087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc440b9040 0x7fdc440d0480 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdc34005fd0 tx=0x7fdc3402a670 comp rx=0 tx=0).stop 2024-06-29T14:23:20.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.543+0000 7fdc49447640 1 -- 172.21.15.81:0/2818973087 shutdown_connections 2024-06-29T14:23:20.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.543+0000 7fdc49447640 1 --2- 172.21.15.81:0/2818973087 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc440d2a90 0x7fdc440d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.543+0000 7fdc49447640 1 --2- 172.21.15.81:0/2818973087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc440b9040 0x7fdc440d0480 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.543+0000 7fdc49447640 1 -- 172.21.15.81:0/2818973087 >> 172.21.15.81:0/2818973087 conn(0x7fdc4409cf30 msgr2=0x7fdc440d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.543+0000 7fdc49447640 1 -- 172.21.15.81:0/2818973087 shutdown_connections 2024-06-29T14:23:20.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.543+0000 7fdc49447640 1 -- 172.21.15.81:0/2818973087 wait complete. 2024-06-29T14:23:20.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.545+0000 7fdc49447640 1 Processor -- start 2024-06-29T14:23:20.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.545+0000 7fdc49447640 1 -- start start 2024-06-29T14:23:20.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.546+0000 7fdc49447640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc440b5250 0x7fdc440b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.546+0000 7fdc49447640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc440b9040 0x7fdc441f7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.546+0000 7fdc49447640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc440d2a90 0x7fdc440a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.546+0000 7fdc49447640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc440b7d30 con 0x7fdc440b9040 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.546+0000 7fdc49447640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdc440b7bb0 con 0x7fdc440b5250 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.546+0000 7fdc49447640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdc440b7eb0 con 0x7fdc440d2a90 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.546+0000 7fdc3a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc440b9040 0x7fdc441f7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.546+0000 7fdc3a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc440b9040 0x7fdc441f7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59748/0 (socket says 172.21.15.81:59748) 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.546+0000 7fdc3a7fc640 1 -- 172.21.15.81:0/2514432561 learned_addr learned my addr 172.21.15.81:0/2514432561 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc3a7fc640 1 -- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc440d2a90 msgr2=0x7fdc440a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc3a7fc640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc440d2a90 0x7fdc440a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc3a7fc640 1 -- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc440b5250 msgr2=0x7fdc440b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc3a7fc640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc440b5250 0x7fdc440b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc3a7fc640 1 -- 172.21.15.81:0/2514432561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc441f82e0 con 0x7fdc440b9040 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc3a7fc640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc440b9040 0x7fdc441f7da0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00c3c0 tx=0x7fdc2c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc3bfff640 1 -- 172.21.15.81:0/2514432561 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c017070 con 0x7fdc440b9040 2024-06-29T14:23:20.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc3bfff640 1 -- 172.21.15.81:0/2514432561 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc2c0035f0 con 0x7fdc440b9040 2024-06-29T14:23:20.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc49447640 1 -- 172.21.15.81:0/2514432561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc440b3020 con 0x7fdc440b9040 2024-06-29T14:23:20.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc3bfff640 1 -- 172.21.15.81:0/2514432561 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c003a40 con 0x7fdc440b9040 2024-06-29T14:23:20.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.547+0000 7fdc49447640 1 -- 172.21.15.81:0/2514432561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc440b35e0 con 0x7fdc440b9040 2024-06-29T14:23:20.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.548+0000 7fdc49447640 1 -- 172.21.15.81:0/2514432561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc440b8440 con 0x7fdc440b9040 2024-06-29T14:23:20.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.549+0000 7fdc3bfff640 1 -- 172.21.15.81:0/2514432561 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdc2c01a070 con 0x7fdc440b9040 2024-06-29T14:23:20.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.549+0000 7fdc3bfff640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdc240762d0 0x7fdc240787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.549+0000 7fdc3bfff640 1 -- 172.21.15.81:0/2514432561 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fdc2c0969e0 con 0x7fdc440b9040 2024-06-29T14:23:20.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.550+0000 7fdc42ffd640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdc240762d0 0x7fdc240787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.550+0000 7fdc42ffd640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdc240762d0 0x7fdc240787e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdc34033000 tx=0x7fdc340046c0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.554+0000 7fdc3bfff640 1 -- 172.21.15.81:0/2514432561 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc2c0603d0 con 0x7fdc440b9040 2024-06-29T14:23:20.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.733+0000 7fdc49447640 1 -- 172.21.15.81:0/2514432561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdc440b20c0 con 0x7fdc440b9040 2024-06-29T14:23:20.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.733+0000 7fdc3bfff640 1 -- 172.21.15.81:0/2514432561 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fdc2c05fcd0 con 0x7fdc440b9040 2024-06-29T14:23:20.731 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:23:20.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 -- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdc240762d0 msgr2=0x7fdc240787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdc240762d0 0x7fdc240787e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdc34033000 tx=0x7fdc340046c0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 -- 172.21.15.81:0/2514432561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc440b9040 msgr2=0x7fdc441f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc440b9040 0x7fdc441f7da0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00c3c0 tx=0x7fdc2c00c880 comp rx=0 tx=0).stop 2024-06-29T14:23:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 -- 172.21.15.81:0/2514432561 shutdown_connections 2024-06-29T14:23:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fdc240762d0 0x7fdc240787e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc440d2a90 0x7fdc440a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc440b9040 0x7fdc441f7da0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 --2- 172.21.15.81:0/2514432561 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc440b5250 0x7fdc440b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 -- 172.21.15.81:0/2514432561 >> 172.21.15.81:0/2514432561 conn(0x7fdc4409cf30 msgr2=0x7fdc4406d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 -- 172.21.15.81:0/2514432561 shutdown_connections 2024-06-29T14:23:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.735+0000 7fdc49447640 1 -- 172.21.15.81:0/2514432561 wait complete. 2024-06-29T14:23:20.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:23:20.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.861+0000 7f428d0ce640 1 -- 172.21.15.81:0/2939424403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4270004e60 msgr2=0x7f42700052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.861+0000 7f428d0ce640 1 --2- 172.21.15.81:0/2939424403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4270004e60 0x7f42700052b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f428000c3f0 tx=0x7f4270005830 comp rx=0 tx=0).stop 2024-06-29T14:23:20.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.861+0000 7f428759e640 1 -- 172.21.15.81:0/2939424403 reap_dead start 2024-06-29T14:23:20.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.861+0000 7f428d0ce640 1 -- 172.21.15.81:0/2939424403 shutdown_connections 2024-06-29T14:23:20.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.861+0000 7f428d0ce640 1 -- 172.21.15.81:0/2939424403 >> 172.21.15.81:0/2939424403 conn(0x7f42880b3680 msgr2=0x7f42880b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:20.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.861+0000 7f428d0ce640 1 -- 172.21.15.81:0/2939424403 shutdown_connections 2024-06-29T14:23:20.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.861+0000 7f428d0ce640 1 -- 172.21.15.81:0/2939424403 wait complete. 2024-06-29T14:23:20.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.864+0000 7f428d0ce640 1 Processor -- start 2024-06-29T14:23:20.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.864+0000 7f428d0ce640 1 -- start start 2024-06-29T14:23:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f428d0ce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4270004e60 0x7f42881d8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f428d0ce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42880cb330 0x7f42881d67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f428d0ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f428806d5c0 0x7f428806d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f428d0ce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f42880d09d0 con 0x7f4270004e60 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f428d0ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f42880d0850 con 0x7f428806d5c0 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f428d0ce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f42880b5a60 con 0x7f42880cb330 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f4286d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4270004e60 0x7f42881d8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f4286d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4270004e60 0x7f42881d8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59778/0 (socket says 172.21.15.81:59778) 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f4286d9d640 1 -- 172.21.15.81:0/3404937443 learned_addr learned my addr 172.21.15.81:0/3404937443 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f428659c640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42880cb330 0x7f42881d67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f428759e640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f428806d5c0 0x7f428806d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f4286d9d640 1 -- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42880cb330 msgr2=0x7f42881d67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f4286d9d640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42880cb330 0x7f42881d67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f4286d9d640 1 -- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f428806d5c0 msgr2=0x7f428806d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f4286d9d640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f428806d5c0 0x7f428806d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f4286d9d640 1 -- 172.21.15.81:0/3404937443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42880ca290 con 0x7f4270004e60 2024-06-29T14:23:20.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.865+0000 7f4286d9d640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4270004e60 0x7f42881d8bc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4278001830 tx=0x7f4278001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.866+0000 7f426f7fe640 1 -- 172.21.15.81:0/3404937443 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278015070 con 0x7f4270004e60 2024-06-29T14:23:20.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.866+0000 7f426f7fe640 1 -- 172.21.15.81:0/3404937443 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f427800e9e0 con 0x7f4270004e60 2024-06-29T14:23:20.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.866+0000 7f428d0ce640 1 -- 172.21.15.81:0/3404937443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42880ca570 con 0x7f4270004e60 2024-06-29T14:23:20.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.866+0000 7f426f7fe640 1 -- 172.21.15.81:0/3404937443 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427800edf0 con 0x7f4270004e60 2024-06-29T14:23:20.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.866+0000 7f428d0ce640 1 -- 172.21.15.81:0/3404937443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42880cab30 con 0x7f4270004e60 2024-06-29T14:23:20.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.867+0000 7f428d0ce640 1 -- 172.21.15.81:0/3404937443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4288071d00 con 0x7f4270004e60 2024-06-29T14:23:20.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.867+0000 7f426f7fe640 1 -- 172.21.15.81:0/3404937443 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4278018070 con 0x7f4270004e60 2024-06-29T14:23:20.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.868+0000 7f426f7fe640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f42740762d0 0x7f42740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:20.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.868+0000 7f426f7fe640 1 -- 172.21.15.81:0/3404937443 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4278098060 con 0x7f4270004e60 2024-06-29T14:23:20.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.868+0000 7f428659c640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f42740762d0 0x7f42740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:20.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.869+0000 7f428659c640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f42740762d0 0x7f42740787e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f428806f5e0 tx=0x7f427c00b260 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:20.873+0000 7f426f7fe640 1 -- 172.21.15.81:0/3404937443 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4278061930 con 0x7f4270004e60 2024-06-29T14:23:21.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.064+0000 7f428d0ce640 1 -- 172.21.15.81:0/3404937443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f42880b5fc0 con 0x7f4270004e60 2024-06-29T14:23:21.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.065+0000 7f426f7fe640 1 -- 172.21.15.81:0/3404937443 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f4278061230 con 0x7f4270004e60 2024-06-29T14:23:21.063 INFO:teuthology.orchestra.run.smithi081.stderr:listed 0 entries 2024-06-29T14:23:21.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 -- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f42740762d0 msgr2=0x7f42740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f42740762d0 0x7f42740787e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f428806f5e0 tx=0x7f427c00b260 comp rx=0 tx=0).stop 2024-06-29T14:23:21.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 -- 172.21.15.81:0/3404937443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4270004e60 msgr2=0x7f42881d8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4270004e60 0x7f42881d8bc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4278001830 tx=0x7f4278001cf0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 -- 172.21.15.81:0/3404937443 shutdown_connections 2024-06-29T14:23:21.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f42740762d0 0x7f42740787e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f428806d5c0 0x7f428806d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42880cb330 0x7f42881d67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 --2- 172.21.15.81:0/3404937443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4270004e60 0x7f42881d8bc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 -- 172.21.15.81:0/3404937443 >> 172.21.15.81:0/3404937443 conn(0x7f42880b3680 msgr2=0x7f42880cf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:21.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 -- 172.21.15.81:0/3404937443 shutdown_connections 2024-06-29T14:23:21.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.067+0000 7f428d0ce640 1 -- 172.21.15.81:0/3404937443 wait complete. 2024-06-29T14:23:21.077 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-06-29T14:23:21.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:21.077 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-06-29T14:23:21.103 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:23:21.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:21.103 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-06-29T14:23:21.159 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:23:21.159 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:21.180 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:23:21.180 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:21.237 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:23:21.238 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:21.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:21.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.410+0000 7fb829703640 1 -- 172.21.15.81:0/1170458360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8240b6d60 msgr2=0x7fb8240ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.410+0000 7fb829703640 1 --2- 172.21.15.81:0/1170458360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8240b6d60 0x7fb8240ce4d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb8240cf0e0 tx=0x7fb81402aa30 comp rx=0 tx=0).stop 2024-06-29T14:23:21.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.411+0000 7fb829703640 1 -- 172.21.15.81:0/1170458360 shutdown_connections 2024-06-29T14:23:21.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.411+0000 7fb829703640 1 --2- 172.21.15.81:0/1170458360 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8240b4cf0 0x7fb8240d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.411+0000 7fb829703640 1 --2- 172.21.15.81:0/1170458360 >> v2:172.21.15.167:3300/0 conn(0x7fb8240cea10 0x7fb8240b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.411+0000 7fb829703640 1 --2- 172.21.15.81:0/1170458360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8240b6d60 0x7fb8240ce4d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.411+0000 7fb829703640 1 -- 172.21.15.81:0/1170458360 >> 172.21.15.81:0/1170458360 conn(0x7fb8240d1960 msgr2=0x7fb8240d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:21.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.411+0000 7fb829703640 1 -- 172.21.15.81:0/1170458360 shutdown_connections 2024-06-29T14:23:21.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.411+0000 7fb829703640 1 -- 172.21.15.81:0/1170458360 wait complete. 2024-06-29T14:23:21.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.412+0000 7fb829703640 1 Processor -- start 2024-06-29T14:23:21.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.413+0000 7fb829703640 1 -- start start 2024-06-29T14:23:21.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb829703640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8240b4cf0 0x7fb8241dc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb829703640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8240b6d60 0x7fb8240b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb829703640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8240cea10 0x7fb8240c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb829703640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8240d09c0 con 0x7fb8240cea10 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb829703640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb8240d0840 con 0x7fb8240b4cf0 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb829703640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb8240cc530 con 0x7fb8240b6d60 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb8237fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8240cea10 0x7fb8240c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb8237fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8240cea10 0x7fb8240c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59798/0 (socket says 172.21.15.81:59798) 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb822ffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8240b4cf0 0x7fb8241dc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb8237fe640 1 -- 172.21.15.81:0/3581180178 learned_addr learned my addr 172.21.15.81:0/3581180178 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb8227fc640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8240b6d60 0x7fb8240b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb8237fe640 1 -- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8240b6d60 msgr2=0x7fb8240b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb8237fe640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8240b6d60 0x7fb8240b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb8237fe640 1 -- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8240b4cf0 msgr2=0x7fb8241dc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb8237fe640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8240b4cf0 0x7fb8241dc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.414+0000 7fb8237fe640 1 -- 172.21.15.81:0/3581180178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8240c7ad0 con 0x7fb8240cea10 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.415+0000 7fb8237fe640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8240cea10 0x7fb8240c6d50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00b440 tx=0x7fb81c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.415+0000 7fb80b7fe640 1 -- 172.21.15.81:0/3581180178 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c015020 con 0x7fb8240cea10 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.415+0000 7fb80b7fe640 1 -- 172.21.15.81:0/3581180178 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb81c011920 con 0x7fb8240cea10 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.415+0000 7fb80b7fe640 1 -- 172.21.15.81:0/3581180178 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81c011d70 con 0x7fb8240cea10 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.415+0000 7fb829703640 1 -- 172.21.15.81:0/3581180178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8240c7db0 con 0x7fb8240cea10 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.415+0000 7fb829703640 1 -- 172.21.15.81:0/3581180178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8240c8370 con 0x7fb8240cea10 2024-06-29T14:23:21.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.416+0000 7fb829703640 1 -- 172.21.15.81:0/3581180178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8240ce4d0 con 0x7fb8240cea10 2024-06-29T14:23:21.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.416+0000 7fb80b7fe640 1 -- 172.21.15.81:0/3581180178 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb81c018020 con 0x7fb8240cea10 2024-06-29T14:23:21.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.417+0000 7fb80b7fe640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8100762d0 0x7fb8100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.417+0000 7fb80b7fe640 1 -- 172.21.15.81:0/3581180178 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb81c097550 con 0x7fb8240cea10 2024-06-29T14:23:21.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.417+0000 7fb822ffd640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8100762d0 0x7fb8100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.418+0000 7fb822ffd640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8100762d0 0x7fb8100787e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb814035000 tx=0x7fb81403b040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:21.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.421+0000 7fb80b7fe640 1 -- 172.21.15.81:0/3581180178 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb81c01f020 con 0x7fb8240cea10 2024-06-29T14:23:21.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.601+0000 7fb829703640 1 -- 172.21.15.81:0/3581180178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb82406a5d0 con 0x7fb8240cea10 2024-06-29T14:23:21.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.602+0000 7fb80b7fe640 1 -- 172.21.15.81:0/3581180178 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16593 (secure 0 0 0) 0x7fb81c0607d0 con 0x7fb8240cea10 2024-06-29T14:23:21.601 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:21.601 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:22:45.692955+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:21.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 -- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8100762d0 msgr2=0x7fb8100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8100762d0 0x7fb8100787e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb814035000 tx=0x7fb81403b040 comp rx=0 tx=0).stop 2024-06-29T14:23:21.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 -- 172.21.15.81:0/3581180178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8240cea10 msgr2=0x7fb8240c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8240cea10 0x7fb8240c6d50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00b440 tx=0x7fb81c00b840 comp rx=0 tx=0).stop 2024-06-29T14:23:21.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 -- 172.21.15.81:0/3581180178 shutdown_connections 2024-06-29T14:23:21.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb8100762d0 0x7fb8100787e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8240cea10 0x7fb8240c6d50 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8240b6d60 0x7fb8240b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 --2- 172.21.15.81:0/3581180178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8240b4cf0 0x7fb8241dc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 -- 172.21.15.81:0/3581180178 >> 172.21.15.81:0/3581180178 conn(0x7fb8240d1960 msgr2=0x7fb8240d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:21.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 -- 172.21.15.81:0/3581180178 shutdown_connections 2024-06-29T14:23:21.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.605+0000 7fb829703640 1 -- 172.21.15.81:0/3581180178 wait complete. 2024-06-29T14:23:21.614 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:21.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:21.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.733+0000 7f205e412640 1 -- 172.21.15.81:0/3131718049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20400099b0 msgr2=0x7f204000bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.733+0000 7f205e412640 1 --2- 172.21.15.81:0/3131718049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20400099b0 0x7f204000bdd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2050001dc0 tx=0x7f2040005830 comp rx=0 tx=0).stop 2024-06-29T14:23:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.733+0000 7f205c9af640 1 -- 172.21.15.81:0/3131718049 reap_dead start 2024-06-29T14:23:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.733+0000 7f205e412640 1 -- 172.21.15.81:0/3131718049 shutdown_connections 2024-06-29T14:23:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.733+0000 7f205e412640 1 -- 172.21.15.81:0/3131718049 >> 172.21.15.81:0/3131718049 conn(0x7f20580b3510 msgr2=0x7f20580b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.733+0000 7f205e412640 1 -- 172.21.15.81:0/3131718049 shutdown_connections 2024-06-29T14:23:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.733+0000 7f205e412640 1 -- 172.21.15.81:0/3131718049 wait complete. 2024-06-29T14:23:21.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.735+0000 7f205e412640 1 Processor -- start 2024-06-29T14:23:21.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.735+0000 7f205e412640 1 -- start start 2024-06-29T14:23:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.736+0000 7f205e412640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20580afc60 0x7f20580b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.736+0000 7f205e412640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20581dbcb0 0x7f20581dc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.736+0000 7f205e412640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20581dc600 0x7f20581de070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.736+0000 7f205e412640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f20580ce8a0 con 0x7f20581dbcb0 2024-06-29T14:23:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.736+0000 7f205e412640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f20580ce720 con 0x7f20581dc600 2024-06-29T14:23:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.736+0000 7f205e412640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f20580b4f20 con 0x7f20580afc60 2024-06-29T14:23:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f20577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20581dbcb0 0x7f20581dc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f20577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20581dbcb0 0x7f20581dc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59816/0 (socket says 172.21.15.81:59816) 2024-06-29T14:23:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f20577fe640 1 -- 172.21.15.81:0/704481672 learned_addr learned my addr 172.21.15.81:0/704481672 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f20577fe640 1 -- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20580afc60 msgr2=0x7f20580b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f20577fe640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20580afc60 0x7f20580b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f20577fe640 1 -- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20581dc600 msgr2=0x7f20581de070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f20577fe640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20581dc600 0x7f20581de070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f20577fe640 1 -- 172.21.15.81:0/704481672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20580b06b0 con 0x7f20581dbcb0 2024-06-29T14:23:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f20577fe640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20581dbcb0 0x7f20581dc0c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f204c00ea20 tx=0x7f204c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f2054ff9640 1 -- 172.21.15.81:0/704481672 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c00ac20 con 0x7f20581dbcb0 2024-06-29T14:23:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f2054ff9640 1 -- 172.21.15.81:0/704481672 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f204c005020 con 0x7f20581dbcb0 2024-06-29T14:23:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f205e412640 1 -- 172.21.15.81:0/704481672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20580b0990 con 0x7f20581dbcb0 2024-06-29T14:23:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f2054ff9640 1 -- 172.21.15.81:0/704481672 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c005450 con 0x7f20581dbcb0 2024-06-29T14:23:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.737+0000 7f205e412640 1 -- 172.21.15.81:0/704481672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20580b0f50 con 0x7f20581dbcb0 2024-06-29T14:23:21.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.738+0000 7f205e412640 1 -- 172.21.15.81:0/704481672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20580d02d0 con 0x7f20581dbcb0 2024-06-29T14:23:21.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.739+0000 7f2054ff9640 1 -- 172.21.15.81:0/704481672 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f204c0047a0 con 0x7f20581dbcb0 2024-06-29T14:23:21.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.740+0000 7f2054ff9640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f20440762d0 0x7f20440787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:21.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.740+0000 7f2054ff9640 1 -- 172.21.15.81:0/704481672 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f204c097f40 con 0x7f20581dbcb0 2024-06-29T14:23:21.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.740+0000 7f2057fff640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f20440762d0 0x7f20440787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:21.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.740+0000 7f2057fff640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f20440762d0 0x7f20440787e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2048001d80 tx=0x7f2048001d10 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:21.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.744+0000 7f2054ff9640 1 -- 172.21.15.81:0/704481672 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f204c0618a0 con 0x7f20581dbcb0 2024-06-29T14:23:21.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.923+0000 7f205e412640 1 -- 172.21.15.81:0/704481672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f20580cbbc0 con 0x7f20581dbcb0 2024-06-29T14:23:21.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.924+0000 7f2054ff9640 1 -- 172.21.15.81:0/704481672 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f204c0611a0 con 0x7f20581dbcb0 2024-06-29T14:23:21.923 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 -- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f20440762d0 msgr2=0x7f20440787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f20440762d0 0x7f20440787e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2048001d80 tx=0x7f2048001d10 comp rx=0 tx=0).stop 2024-06-29T14:23:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 -- 172.21.15.81:0/704481672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20581dbcb0 msgr2=0x7f20581dc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20581dbcb0 0x7f20581dc0c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f204c00ea20 tx=0x7f204c00eee0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 -- 172.21.15.81:0/704481672 shutdown_connections 2024-06-29T14:23:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f20440762d0 0x7f20440787e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20581dc600 0x7f20581de070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20581dbcb0 0x7f20581dc0c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 --2- 172.21.15.81:0/704481672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20580afc60 0x7f20580b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 -- 172.21.15.81:0/704481672 >> 172.21.15.81:0/704481672 conn(0x7f20580b3510 msgr2=0x7f20580b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 -- 172.21.15.81:0/704481672 shutdown_connections 2024-06-29T14:23:21.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:21.926+0000 7f205e412640 1 -- 172.21.15.81:0/704481672 wait complete. 2024-06-29T14:23:21.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:23:22.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.053+0000 7f5d5ce38640 1 -- 172.21.15.81:0/441388333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d580d17b0 msgr2=0x7f5d580d1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.053+0000 7f5d5ce38640 1 --2- 172.21.15.81:0/441388333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d580d17b0 0x7f5d580d1bc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5d580d4ce0 tx=0x7f5d5002a8f0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.054+0000 7f5d5ce38640 1 -- 172.21.15.81:0/441388333 shutdown_connections 2024-06-29T14:23:22.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.054+0000 7f5d5ce38640 1 --2- 172.21.15.81:0/441388333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d580d17b0 0x7f5d580d1bc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.054+0000 7f5d5ce38640 1 --2- 172.21.15.81:0/441388333 >> v2:172.21.15.167:3300/0 conn(0x7f5d5808ccd0 0x7f5d580cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.054+0000 7f5d5ce38640 1 --2- 172.21.15.81:0/441388333 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d5808c270 0x7f5d580ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.054+0000 7f5d5ce38640 1 -- 172.21.15.81:0/441388333 >> 172.21.15.81:0/441388333 conn(0x7f5d580b56d0 msgr2=0x7f5d580b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:22.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.054+0000 7f5d5ce38640 1 -- 172.21.15.81:0/441388333 shutdown_connections 2024-06-29T14:23:22.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.054+0000 7f5d5ce38640 1 -- 172.21.15.81:0/441388333 wait complete. 2024-06-29T14:23:22.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.056+0000 7f5d5ce38640 1 Processor -- start 2024-06-29T14:23:22.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.056+0000 7f5d5ce38640 1 -- start start 2024-06-29T14:23:22.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d5ce38640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d5808ccd0 0x7f5d581e3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d5ce38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d580d17b0 0x7f5d581f7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d5ce38640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d580a0480 0x7f5d5806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d5ce38640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5d580d2700 con 0x7f5d580d17b0 2024-06-29T14:23:22.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d5ce38640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5d580d2580 con 0x7f5d580a0480 2024-06-29T14:23:22.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d5ce38640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5d580b3240 con 0x7f5d5808ccd0 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d55d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d580d17b0 0x7f5d581f7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d55d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d580d17b0 0x7f5d581f7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59834/0 (socket says 172.21.15.81:59834) 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d55d9b640 1 -- 172.21.15.81:0/2128412088 learned_addr learned my addr 172.21.15.81:0/2128412088 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d56d9d640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d580a0480 0x7f5d5806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d5659c640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d5808ccd0 0x7f5d581e3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d55d9b640 1 -- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d5808ccd0 msgr2=0x7f5d581e3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d55d9b640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d5808ccd0 0x7f5d581e3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d55d9b640 1 -- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d580a0480 msgr2=0x7f5d5806d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d55d9b640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d580a0480 0x7f5d5806d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d55d9b640 1 -- 172.21.15.81:0/2128412088 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d581f8940 con 0x7f5d580d17b0 2024-06-29T14:23:22.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d5659c640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d5808ccd0 0x7f5d581e3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:22.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.057+0000 7f5d55d9b640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d580d17b0 0x7f5d581f7d60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00d700 tx=0x7f5d4c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:22.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.058+0000 7f5d46ffd640 1 -- 172.21.15.81:0/2128412088 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c017070 con 0x7f5d580d17b0 2024-06-29T14:23:22.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.058+0000 7f5d46ffd640 1 -- 172.21.15.81:0/2128412088 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d4c009b10 con 0x7f5d580d17b0 2024-06-29T14:23:22.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.058+0000 7f5d46ffd640 1 -- 172.21.15.81:0/2128412088 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c010460 con 0x7f5d580d17b0 2024-06-29T14:23:22.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.058+0000 7f5d5ce38640 1 -- 172.21.15.81:0/2128412088 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5809f9a0 con 0x7f5d580d17b0 2024-06-29T14:23:22.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.058+0000 7f5d5ce38640 1 -- 172.21.15.81:0/2128412088 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d581e4260 con 0x7f5d580d17b0 2024-06-29T14:23:22.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.059+0000 7f5d5ce38640 1 -- 172.21.15.81:0/2128412088 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d5808c270 con 0x7f5d580d17b0 2024-06-29T14:23:22.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.059+0000 7f5d46ffd640 1 -- 172.21.15.81:0/2128412088 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5d4c01a070 con 0x7f5d580d17b0 2024-06-29T14:23:22.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.060+0000 7f5d46ffd640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5d380762d0 0x7f5d380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.060+0000 7f5d46ffd640 1 -- 172.21.15.81:0/2128412088 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5d4c097700 con 0x7f5d580d17b0 2024-06-29T14:23:22.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.060+0000 7f5d5659c640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5d380762d0 0x7f5d380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.061+0000 7f5d5659c640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5d380762d0 0x7f5d380787e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009aa0 tx=0x7f5d48005db0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:22.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.064+0000 7f5d46ffd640 1 -- 172.21.15.81:0/2128412088 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d4c0610f0 con 0x7f5d580d17b0 2024-06-29T14:23:22.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.245+0000 7f5d5ce38640 1 -- 172.21.15.81:0/2128412088 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5d5806a350 con 0x7f5d580d17b0 2024-06-29T14:23:22.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.246+0000 7f5d46ffd640 1 -- 172.21.15.81:0/2128412088 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f5d4c0609f0 con 0x7f5d580d17b0 2024-06-29T14:23:22.244 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:23:22.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.247+0000 7f5d5ce38640 1 -- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5d380762d0 msgr2=0x7f5d380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.247+0000 7f5d5ce38640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5d380762d0 0x7f5d380787e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009aa0 tx=0x7f5d48005db0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.247+0000 7f5d5ce38640 1 -- 172.21.15.81:0/2128412088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d580d17b0 msgr2=0x7f5d581f7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.247+0000 7f5d5ce38640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d580d17b0 0x7f5d581f7d60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00d700 tx=0x7f5d4c00dbc0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.248+0000 7f5d5ce38640 1 -- 172.21.15.81:0/2128412088 shutdown_connections 2024-06-29T14:23:22.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.248+0000 7f5d5ce38640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5d380762d0 0x7f5d380787e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.248+0000 7f5d5ce38640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d580a0480 0x7f5d5806d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.248+0000 7f5d5ce38640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d580d17b0 0x7f5d581f7d60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.248+0000 7f5d5ce38640 1 --2- 172.21.15.81:0/2128412088 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d5808ccd0 0x7f5d581e3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.248+0000 7f5d5ce38640 1 -- 172.21.15.81:0/2128412088 >> 172.21.15.81:0/2128412088 conn(0x7f5d580b56d0 msgr2=0x7f5d581e5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:22.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.248+0000 7f5d5ce38640 1 -- 172.21.15.81:0/2128412088 shutdown_connections 2024-06-29T14:23:22.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.248+0000 7f5d5ce38640 1 -- 172.21.15.81:0/2128412088 wait complete. 2024-06-29T14:23:22.257 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:23:22.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.375+0000 7f658f1b2640 1 -- 172.21.15.81:0/1429475609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65880b57b0 msgr2=0x7f658815aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.375+0000 7f658f1b2640 1 --2- 172.21.15.81:0/1429475609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65880b57b0 0x7f658815aed0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6584009830 tx=0x7f658402a510 comp rx=0 tx=0).stop 2024-06-29T14:23:22.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.376+0000 7f658f1b2640 1 -- 172.21.15.81:0/1429475609 shutdown_connections 2024-06-29T14:23:22.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.376+0000 7f658f1b2640 1 --2- 172.21.15.81:0/1429475609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65880b3960 0x7f65880b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.376+0000 7f658f1b2640 1 --2- 172.21.15.81:0/1429475609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65880b57b0 0x7f658815aed0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.376+0000 7f658f1b2640 1 -- 172.21.15.81:0/1429475609 >> 172.21.15.81:0/1429475609 conn(0x7f65880d0930 msgr2=0x7f65880d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:22.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.376+0000 7f658f1b2640 1 -- 172.21.15.81:0/1429475609 shutdown_connections 2024-06-29T14:23:22.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.376+0000 7f658f1b2640 1 -- 172.21.15.81:0/1429475609 wait complete. 2024-06-29T14:23:22.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.377+0000 7f658f1b2640 1 Processor -- start 2024-06-29T14:23:22.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.377+0000 7f658f1b2640 1 -- start start 2024-06-29T14:23:22.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658f1b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658806ef90 0x7f65880af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658f1b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65880b3960 0x7f65881537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658f1b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65880b57b0 0x7f658806c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658f1b2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f65880b4430 con 0x7f658806ef90 2024-06-29T14:23:22.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658f1b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f65880b42b0 con 0x7f65880b57b0 2024-06-29T14:23:22.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658f1b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f65880b45b0 con 0x7f65880b3960 2024-06-29T14:23:22.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658cf4e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658806ef90 0x7f65880af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658cf4e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658806ef90 0x7f65880af720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59856/0 (socket says 172.21.15.81:59856) 2024-06-29T14:23:22.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658cf4e640 1 -- 172.21.15.81:0/709442115 learned_addr learned my addr 172.21.15.81:0/709442115 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658d74f640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65880b57b0 0x7f658806c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f657ffff640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65880b3960 0x7f65881537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658cf4e640 1 -- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65880b3960 msgr2=0x7f65881537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658cf4e640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65880b3960 0x7f65881537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658cf4e640 1 -- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65880b57b0 msgr2=0x7f658806c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658cf4e640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65880b57b0 0x7f658806c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658cf4e640 1 -- 172.21.15.81:0/709442115 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65881f6b10 con 0x7f658806ef90 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.379+0000 7f658cf4e640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658806ef90 0x7f65880af720 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6584009950 tx=0x7f6584033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.380+0000 7f657d7fa640 1 -- 172.21.15.81:0/709442115 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6584031d60 con 0x7f658806ef90 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.380+0000 7f657d7fa640 1 -- 172.21.15.81:0/709442115 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6584033e10 con 0x7f658806ef90 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.380+0000 7f658f1b2640 1 -- 172.21.15.81:0/709442115 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65881f6df0 con 0x7f658806ef90 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.380+0000 7f657d7fa640 1 -- 172.21.15.81:0/709442115 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6584005260 con 0x7f658806ef90 2024-06-29T14:23:22.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.380+0000 7f658f1b2640 1 -- 172.21.15.81:0/709442115 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65880ad270 con 0x7f658806ef90 2024-06-29T14:23:22.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.381+0000 7f658f1b2640 1 -- 172.21.15.81:0/709442115 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f658806e2a0 con 0x7f658806ef90 2024-06-29T14:23:22.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.381+0000 7f657d7fa640 1 -- 172.21.15.81:0/709442115 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6584005400 con 0x7f658806ef90 2024-06-29T14:23:22.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.382+0000 7f657d7fa640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6574076300 0x7f6574078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.382+0000 7f657d7fa640 1 -- 172.21.15.81:0/709442115 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f65840b8790 con 0x7f658806ef90 2024-06-29T14:23:22.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.382+0000 7f657ffff640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6574076300 0x7f6574078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.383+0000 7f657ffff640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6574076300 0x7f6574078810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f658806d040 tx=0x7f6578009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:22.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.386+0000 7f657d7fa640 1 -- 172.21.15.81:0/709442115 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f658806e2a0 con 0x7f658806ef90 2024-06-29T14:23:22.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.566+0000 7f658f1b2640 1 -- 172.21.15.81:0/709442115 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f65880b4a90 con 0x7f658806ef90 2024-06-29T14:23:22.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.566+0000 7f657d7fa640 1 -- 172.21.15.81:0/709442115 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f658403f030 con 0x7f658806ef90 2024-06-29T14:23:22.565 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 -- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6574076300 msgr2=0x7f6574078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6574076300 0x7f6574078810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f658806d040 tx=0x7f6578009040 comp rx=0 tx=0).stop 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 -- 172.21.15.81:0/709442115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658806ef90 msgr2=0x7f65880af720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658806ef90 0x7f65880af720 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6584009950 tx=0x7f6584033c20 comp rx=0 tx=0).stop 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 -- 172.21.15.81:0/709442115 shutdown_connections 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6574076300 0x7f6574078810 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65880b57b0 0x7f658806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65880b3960 0x7f65881537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 --2- 172.21.15.81:0/709442115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658806ef90 0x7f65880af720 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 -- 172.21.15.81:0/709442115 >> 172.21.15.81:0/709442115 conn(0x7f65880d0930 msgr2=0x7f65880cf540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 -- 172.21.15.81:0/709442115 shutdown_connections 2024-06-29T14:23:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.568+0000 7f658f1b2640 1 -- 172.21.15.81:0/709442115 wait complete. 2024-06-29T14:23:22.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:23:22.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.694+0000 7fe053e26640 1 -- 172.21.15.81:0/2894504868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe04c0b9040 msgr2=0x7fe04c0d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.694+0000 7fe053e26640 1 --2- 172.21.15.81:0/2894504868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe04c0b9040 0x7fe04c0d0480 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe04c08e330 tx=0x7fe04802a5f0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.694+0000 7fe053e26640 1 -- 172.21.15.81:0/2894504868 shutdown_connections 2024-06-29T14:23:22.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.694+0000 7fe053e26640 1 --2- 172.21.15.81:0/2894504868 >> v2:172.21.15.167:3300/0 conn(0x7fe04c0d2a90 0x7fe04c0d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.694+0000 7fe053e26640 1 --2- 172.21.15.81:0/2894504868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe04c0b9040 0x7fe04c0d0480 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.694+0000 7fe053e26640 1 -- 172.21.15.81:0/2894504868 >> 172.21.15.81:0/2894504868 conn(0x7fe04c09cf30 msgr2=0x7fe04c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:22.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.694+0000 7fe053e26640 1 -- 172.21.15.81:0/2894504868 shutdown_connections 2024-06-29T14:23:22.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.694+0000 7fe053e26640 1 -- 172.21.15.81:0/2894504868 wait complete. 2024-06-29T14:23:22.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.696+0000 7fe053e26640 1 Processor -- start 2024-06-29T14:23:22.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.696+0000 7fe053e26640 1 -- start start 2024-06-29T14:23:22.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe053e26640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe04c0b5250 0x7fe04c1f5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe053e26640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe04c0b9040 0x7fe04c1f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe053e26640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe04c0d2a90 0x7fe04c15fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe053e26640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe04c0b7d30 con 0x7fe04c0d2a90 2024-06-29T14:23:22.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe053e26640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe04c0b7bb0 con 0x7fe04c0b5250 2024-06-29T14:23:22.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe053e26640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe04c0b7eb0 con 0x7fe04c0b9040 2024-06-29T14:23:22.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0523c3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe04c0d2a90 0x7fe04c15fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0523c3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe04c0d2a90 0x7fe04c15fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59874/0 (socket says 172.21.15.81:59874) 2024-06-29T14:23:22.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0523c3640 1 -- 172.21.15.81:0/2342893649 learned_addr learned my addr 172.21.15.81:0/2342893649 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:22.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0513c1640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe04c0b9040 0x7fe04c1f4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe051bc2640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe04c0b5250 0x7fe04c1f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0523c3640 1 -- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe04c0b9040 msgr2=0x7fe04c1f4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0523c3640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe04c0b9040 0x7fe04c1f4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0523c3640 1 -- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe04c0b5250 msgr2=0x7fe04c1f5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0523c3640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe04c0b5250 0x7fe04c1f5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0523c3640 1 -- 172.21.15.81:0/2342893649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04c1f5e50 con 0x7fe04c0d2a90 2024-06-29T14:23:22.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe051bc2640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe04c0b5250 0x7fe04c1f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:22.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0513c1640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe04c0b9040 0x7fe04c1f4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:22.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0523c3640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe04c0d2a90 0x7fe04c15fcb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe04400cae0 tx=0x7fe0440071c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:22.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0427fc640 1 -- 172.21.15.81:0/2342893649 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe044007d40 con 0x7fe04c0d2a90 2024-06-29T14:23:22.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0427fc640 1 -- 172.21.15.81:0/2342893649 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe044003990 con 0x7fe04c0d2a90 2024-06-29T14:23:22.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe053e26640 1 -- 172.21.15.81:0/2342893649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe04c1e7450 con 0x7fe04c0d2a90 2024-06-29T14:23:22.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe0427fc640 1 -- 172.21.15.81:0/2342893649 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe044003e00 con 0x7fe04c0d2a90 2024-06-29T14:23:22.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.698+0000 7fe053e26640 1 -- 172.21.15.81:0/2342893649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe04c1f6080 con 0x7fe04c0d2a90 2024-06-29T14:23:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.699+0000 7fe053e26640 1 -- 172.21.15.81:0/2342893649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe04c0b8440 con 0x7fe04c0d2a90 2024-06-29T14:23:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.700+0000 7fe0427fc640 1 -- 172.21.15.81:0/2342893649 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe044004b20 con 0x7fe04c0d2a90 2024-06-29T14:23:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.701+0000 7fe0427fc640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0300762d0 0x7fe0300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.701+0000 7fe0427fc640 1 -- 172.21.15.81:0/2342893649 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe044098110 con 0x7fe04c0d2a90 2024-06-29T14:23:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.701+0000 7fe051bc2640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0300762d0 0x7fe0300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.702+0000 7fe051bc2640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0300762d0 0x7fe0300787e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe048033000 tx=0x7fe048004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:22.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.704+0000 7fe0427fc640 1 -- 172.21.15.81:0/2342893649 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe044014070 con 0x7fe04c0d2a90 2024-06-29T14:23:22.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.901+0000 7fe053e26640 1 -- 172.21.15.81:0/2342893649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe04c069230 con 0x7fe04c0d2a90 2024-06-29T14:23:22.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.901+0000 7fe0427fc640 1 -- 172.21.15.81:0/2342893649 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7fe044061b50 con 0x7fe04c0d2a90 2024-06-29T14:23:22.900 INFO:teuthology.orchestra.run.smithi081.stderr:listed 0 entries 2024-06-29T14:23:22.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 -- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0300762d0 msgr2=0x7fe0300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0300762d0 0x7fe0300787e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe048033000 tx=0x7fe048004730 comp rx=0 tx=0).stop 2024-06-29T14:23:22.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 -- 172.21.15.81:0/2342893649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe04c0d2a90 msgr2=0x7fe04c15fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:22.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe04c0d2a90 0x7fe04c15fcb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe04400cae0 tx=0x7fe0440071c0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 -- 172.21.15.81:0/2342893649 shutdown_connections 2024-06-29T14:23:22.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0300762d0 0x7fe0300787e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe04c0d2a90 0x7fe04c15fcb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe04c0b9040 0x7fe04c1f4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 --2- 172.21.15.81:0/2342893649 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe04c0b5250 0x7fe04c1f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:22.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 -- 172.21.15.81:0/2342893649 >> 172.21.15.81:0/2342893649 conn(0x7fe04c09cf30 msgr2=0x7fe04c0d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:22.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 -- 172.21.15.81:0/2342893649 shutdown_connections 2024-06-29T14:23:22.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:22.903+0000 7fe053e26640 1 -- 172.21.15.81:0/2342893649 wait complete. 2024-06-29T14:23:22.913 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-06-29T14:23:22.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:22.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-06-29T14:23:22.944 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T14:23:22.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:22.944 DEBUG:teuthology.orchestra.run.smithi167:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-06-29T14:23:23.002 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:23:23.002 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:23:23.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.117+0000 7fe24d103640 1 -- 172.21.15.81:0/2066007235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2480cd540 msgr2=0x7fe2480d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.117+0000 7fe24d103640 1 --2- 172.21.15.81:0/2066007235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2480cd540 0x7fe2480d00f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe2480b2c10 tx=0x7fe24002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.117+0000 7fe24d103640 1 -- 172.21.15.81:0/2066007235 shutdown_connections 2024-06-29T14:23:23.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.117+0000 7fe24d103640 1 --2- 172.21.15.81:0/2066007235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2480cd540 0x7fe2480d00f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.117+0000 7fe24d103640 1 --2- 172.21.15.81:0/2066007235 >> v2:172.21.15.167:3300/0 conn(0x7fe2480b6d60 0x7fe2480ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.117+0000 7fe24d103640 1 -- 172.21.15.81:0/2066007235 >> 172.21.15.81:0/2066007235 conn(0x7fe2480d1960 msgr2=0x7fe2480d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:23.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.117+0000 7fe24d103640 1 -- 172.21.15.81:0/2066007235 shutdown_connections 2024-06-29T14:23:23.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.117+0000 7fe24d103640 1 -- 172.21.15.81:0/2066007235 wait complete. 2024-06-29T14:23:23.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.119+0000 7fe24d103640 1 Processor -- start 2024-06-29T14:23:23.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.119+0000 7fe24d103640 1 -- start start 2024-06-29T14:23:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.120+0000 7fe24d103640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2480b6d60 0x7fe2481de030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.120+0000 7fe24d103640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2480cd540 0x7fe2480c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.120+0000 7fe24d103640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2480cea10 0x7fe2480b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.120+0000 7fe24d103640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe2480d0630 con 0x7fe2480cd540 2024-06-29T14:23:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.120+0000 7fe24d103640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe2480b2790 con 0x7fe2480cea10 2024-06-29T14:23:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.120+0000 7fe24d103640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe2480d07b0 con 0x7fe2480b6d60 2024-06-29T14:23:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.120+0000 7fe23e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2480cd540 0x7fe2480c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2480cd540 0x7fe2480c8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59890/0 (socket says 172.21.15.81:59890) 2024-06-29T14:23:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23e59c640 1 -- 172.21.15.81:0/2416702712 learned_addr learned my addr 172.21.15.81:0/2416702712 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23e59c640 1 -- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2480b6d60 msgr2=0x7fe2481de030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23e59c640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2480b6d60 0x7fe2481de030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe24759e640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2480cea10 0x7fe2480b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23e59c640 1 -- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2480cea10 msgr2=0x7fe2480b08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23e59c640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2480cea10 0x7fe2480b08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23e59c640 1 -- 172.21.15.81:0/2416702712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2481e7220 con 0x7fe2480cd540 2024-06-29T14:23:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23e59c640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2480cd540 0x7fe2480c8ae0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe23000c400 tx=0x7fe23000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23ffff640 1 -- 172.21.15.81:0/2416702712 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe230011790 con 0x7fe2480cd540 2024-06-29T14:23:23.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23ffff640 1 -- 172.21.15.81:0/2416702712 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe230017070 con 0x7fe2480cd540 2024-06-29T14:23:23.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe24d103640 1 -- 172.21.15.81:0/2416702712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2481dd550 con 0x7fe2480cd540 2024-06-29T14:23:23.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe23ffff640 1 -- 172.21.15.81:0/2416702712 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe230011ce0 con 0x7fe2480cd540 2024-06-29T14:23:23.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.121+0000 7fe24d103640 1 -- 172.21.15.81:0/2416702712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2480c9540 con 0x7fe2480cd540 2024-06-29T14:23:23.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.122+0000 7fe24d103640 1 -- 172.21.15.81:0/2416702712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2480cbe40 con 0x7fe2480cd540 2024-06-29T14:23:23.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.123+0000 7fe23ffff640 1 -- 172.21.15.81:0/2416702712 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe23001c460 con 0x7fe2480cd540 2024-06-29T14:23:23.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.123+0000 7fe23ffff640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe2280762d0 0x7fe2280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.123+0000 7fe23ffff640 1 -- 172.21.15.81:0/2416702712 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5804+0+0 (secure 0 0 0) 0x7fe230097660 con 0x7fe2480cd540 2024-06-29T14:23:23.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.123+0000 7fe246d9d640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe2280762d0 0x7fe2280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.124+0000 7fe246d9d640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe2280762d0 0x7fe2280787e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe2480c5220 tx=0x7fe238005b70 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:23.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.128+0000 7fe23ffff640 1 -- 172.21.15.81:0/2416702712 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe230060f60 con 0x7fe2480cd540 2024-06-29T14:23:23.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.345+0000 7fe24d103640 1 -- 172.21.15.81:0/2416702712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2481dd170 con 0x7fe2480cd540 2024-06-29T14:23:23.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.345+0000 7fe23ffff640 1 -- 172.21.15.81:0/2416702712 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5217 (secure 0 0 0) 0x7fe230060860 con 0x7fe2480cd540 2024-06-29T14:23:23.343 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:23.344 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"btime":"2024-06-29T14:22:47:707315+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4295,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/96042442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":96042442},{"type":"v1","addr":"172.21.15.167:6837","nonce":96042442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3681460169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3681460169},{"type":"v1","addr":"172.21.15.81:6834","nonce":3681460169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3175849002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3175849002},{"type":"v1","addr":"172.21.15.81:6835","nonce":3175849002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:22:44.057632+0000","modified":"2024-06-29T14:22:47.079866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4360":{"gid":4360,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/3579540962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3579540962},{"type":"v1","addr":"172.21.15.167:6836","nonce":3579540962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4360,"qdb_cluster":[4360]},"id":1}]} 2024-06-29T14:23:23.344 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2024-06-29T14:23:23.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.347+0000 7fe24d103640 1 -- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe2280762d0 msgr2=0x7fe2280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.347+0000 7fe24d103640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe2280762d0 0x7fe2280787e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe2480c5220 tx=0x7fe238005b70 comp rx=0 tx=0).stop 2024-06-29T14:23:23.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.347+0000 7fe24d103640 1 -- 172.21.15.81:0/2416702712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2480cd540 msgr2=0x7fe2480c8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.347+0000 7fe24d103640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2480cd540 0x7fe2480c8ae0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe23000c400 tx=0x7fe23000c8c0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.347+0000 7fe24d103640 1 -- 172.21.15.81:0/2416702712 shutdown_connections 2024-06-29T14:23:23.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.347+0000 7fe24d103640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe2280762d0 0x7fe2280787e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.348+0000 7fe24d103640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2480cea10 0x7fe2480b08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.348+0000 7fe24d103640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2480cd540 0x7fe2480c8ae0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.348+0000 7fe24d103640 1 --2- 172.21.15.81:0/2416702712 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2480b6d60 0x7fe2481de030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.348+0000 7fe24d103640 1 -- 172.21.15.81:0/2416702712 >> 172.21.15.81:0/2416702712 conn(0x7fe2480d1960 msgr2=0x7fe248070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:23.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.348+0000 7fe24d103640 1 -- 172.21.15.81:0/2416702712 shutdown_connections 2024-06-29T14:23:23.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.348+0000 7fe24d103640 1 -- 172.21.15.81:0/2416702712 wait complete. 2024-06-29T14:23:23.358 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:23.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.475+0000 7f31cbc47640 1 -- 172.21.15.81:0/3655598951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31b0004800 msgr2=0x7f31b0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.475+0000 7f31cbc47640 1 --2- 172.21.15.81:0/3655598951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31b0004800 0x7f31b0004c10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f31b40016f0 tx=0x7f31b0005830 comp rx=0 tx=0).stop 2024-06-29T14:23:23.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.475+0000 7f31ca1e4640 1 -- 172.21.15.81:0/3655598951 reap_dead start 2024-06-29T14:23:23.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.475+0000 7f31cbc47640 1 -- 172.21.15.81:0/3655598951 shutdown_connections 2024-06-29T14:23:23.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.475+0000 7f31cbc47640 1 -- 172.21.15.81:0/3655598951 >> 172.21.15.81:0/3655598951 conn(0x7f31c406ccc0 msgr2=0x7f31c40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:23.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.475+0000 7f31cbc47640 1 -- 172.21.15.81:0/3655598951 shutdown_connections 2024-06-29T14:23:23.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.475+0000 7f31cbc47640 1 -- 172.21.15.81:0/3655598951 wait complete. 2024-06-29T14:23:23.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.477+0000 7f31cbc47640 1 Processor -- start 2024-06-29T14:23:23.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.477+0000 7f31cbc47640 1 -- start start 2024-06-29T14:23:23.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.478+0000 7f31cbc47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31b0004800 0x7f31c4140600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.478+0000 7f31cbc47640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31c41527b0 0x7f31c41f0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31cbc47640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31c41f13a0 0x7f31c41f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31cbc47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f31c415eef0 con 0x7f31b0004800 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31cbc47640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f31c415ed70 con 0x7f31c41f13a0 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31cbc47640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f31c415f070 con 0x7f31c41527b0 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31c99e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31b0004800 0x7f31c4140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31c99e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31b0004800 0x7f31c4140600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59910/0 (socket says 172.21.15.81:59910) 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31c99e3640 1 -- 172.21.15.81:0/1756398771 learned_addr learned my addr 172.21.15.81:0/1756398771 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31c91e2640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31c41527b0 0x7f31c41f0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31ca1e4640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31c41f13a0 0x7f31c41f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31c99e3640 1 -- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31c41527b0 msgr2=0x7f31c41f0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31c99e3640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31c41527b0 0x7f31c41f0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31c99e3640 1 -- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31c41f13a0 msgr2=0x7f31c41f17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31c99e3640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31c41f13a0 0x7f31c41f17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31c99e3640 1 -- 172.21.15.81:0/1756398771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c4140b40 con 0x7f31b0004800 2024-06-29T14:23:23.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31c99e3640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31b0004800 0x7f31c4140600 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f31c000e610 tx=0x7f31c000ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:23.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31ba7fc640 1 -- 172.21.15.81:0/1756398771 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c0015070 con 0x7f31b0004800 2024-06-29T14:23:23.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31ba7fc640 1 -- 172.21.15.81:0/1756398771 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31c000ca60 con 0x7f31b0004800 2024-06-29T14:23:23.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31cbc47640 1 -- 172.21.15.81:0/1756398771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c4153350 con 0x7f31b0004800 2024-06-29T14:23:23.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31ba7fc640 1 -- 172.21.15.81:0/1756398771 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c0011460 con 0x7f31b0004800 2024-06-29T14:23:23.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.479+0000 7f31cbc47640 1 -- 172.21.15.81:0/1756398771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c4154b80 con 0x7f31b0004800 2024-06-29T14:23:23.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.480+0000 7f31cbc47640 1 -- 172.21.15.81:0/1756398771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31c415f250 con 0x7f31b0004800 2024-06-29T14:23:23.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.481+0000 7f31ba7fc640 1 -- 172.21.15.81:0/1756398771 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f31c0018070 con 0x7f31b0004800 2024-06-29T14:23:23.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.482+0000 7f31ba7fc640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f31a8076300 0x7f31a8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.482+0000 7f31ba7fc640 1 -- 172.21.15.81:0/1756398771 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5804+0+0 (secure 0 0 0) 0x7f31c0097b60 con 0x7f31b0004800 2024-06-29T14:23:23.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.482+0000 7f31c91e2640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f31a8076300 0x7f31a8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.482+0000 7f31c91e2640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f31a8076300 0x7f31a8078810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f31b4032000 tx=0x7f31b4001190 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:23.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.486+0000 7f31ba7fc640 1 -- 172.21.15.81:0/1756398771 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f31c0060d80 con 0x7f31b0004800 2024-06-29T14:23:23.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.671+0000 7f31cbc47640 1 -- 172.21.15.81:0/1756398771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31c40b4800 con 0x7f31b0004800 2024-06-29T14:23:23.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.672+0000 7f31ba7fc640 1 -- 172.21.15.81:0/1756398771 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16814 (secure 0 0 0) 0x7f31c0065080 con 0x7f31b0004800 2024-06-29T14:23:23.671 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:23.671 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":21,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:23:22.838977+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1},{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:23.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 -- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f31a8076300 msgr2=0x7f31a8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f31a8076300 0x7f31a8078810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f31b4032000 tx=0x7f31b4001190 comp rx=0 tx=0).stop 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 -- 172.21.15.81:0/1756398771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31b0004800 msgr2=0x7f31c4140600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31b0004800 0x7f31c4140600 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f31c000e610 tx=0x7f31c000ead0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 -- 172.21.15.81:0/1756398771 shutdown_connections 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f31a8076300 0x7f31a8078810 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31c41f13a0 0x7f31c41f17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31c41527b0 0x7f31c41f0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 --2- 172.21.15.81:0/1756398771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31b0004800 0x7f31c4140600 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 -- 172.21.15.81:0/1756398771 >> 172.21.15.81:0/1756398771 conn(0x7f31c406ccc0 msgr2=0x7f31c406e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 -- 172.21.15.81:0/1756398771 shutdown_connections 2024-06-29T14:23:23.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.674+0000 7f31cbc47640 1 -- 172.21.15.81:0/1756398771 wait complete. 2024-06-29T14:23:23.683 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:22:41.898363+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:22:43.015911+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:23.684 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:23.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.802+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1628994049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bf4003ff0 msgr2=0x7f1bf4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.802+0000 7f1c0fbfc640 1 --2- 172.21.15.81:0/1628994049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bf4003ff0 0x7f1bf4004800 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1c080514c0 tx=0x7f1bf4005830 comp rx=0 tx=0).stop 2024-06-29T14:23:23.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.802+0000 7f1c0e199640 1 -- 172.21.15.81:0/1628994049 reap_dead start 2024-06-29T14:23:23.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.802+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1628994049 shutdown_connections 2024-06-29T14:23:23.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.802+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1628994049 >> 172.21.15.81:0/1628994049 conn(0x7f1c080d0930 msgr2=0x7f1c080d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:23.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.803+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1628994049 shutdown_connections 2024-06-29T14:23:23.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.803+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1628994049 wait complete. 2024-06-29T14:23:23.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.804+0000 7f1c0fbfc640 1 Processor -- start 2024-06-29T14:23:23.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.804+0000 7f1c0fbfc640 1 -- start start 2024-06-29T14:23:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0fbfc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bf4003ff0 0x7f1c080c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0fbfc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c0806c4d0 0x7f1c0806c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0fbfc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c081f1b50 0x7f1c081f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0fbfc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0815b260 con 0x7f1bf4003ff0 2024-06-29T14:23:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0fbfc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0815b0e0 con 0x7f1c0806c4d0 2024-06-29T14:23:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0fbfc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0815b3e0 con 0x7f1c081f1b50 2024-06-29T14:23:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0d998640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bf4003ff0 0x7f1c080c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0d998640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bf4003ff0 0x7f1c080c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59922/0 (socket says 172.21.15.81:59922) 2024-06-29T14:23:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0d998640 1 -- 172.21.15.81:0/1486882298 learned_addr learned my addr 172.21.15.81:0/1486882298 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0d197640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c0806c4d0 0x7f1c0806c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0e199640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c081f1b50 0x7f1c081f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0d998640 1 -- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c081f1b50 msgr2=0x7f1c081f1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0d998640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c081f1b50 0x7f1c081f1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0d998640 1 -- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c0806c4d0 msgr2=0x7f1c0806c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0d998640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c0806c4d0 0x7f1c0806c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0d998640 1 -- 172.21.15.81:0/1486882298 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c081f4dd0 con 0x7f1bf4003ff0 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0e199640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c081f1b50 0x7f1c081f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0d998640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bf4003ff0 0x7f1c080c5f50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400e600 tx=0x7f1c0400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1bfe7fc640 1 -- 172.21.15.81:0/1486882298 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04015070 con 0x7f1bf4003ff0 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1bfe7fc640 1 -- 172.21.15.81:0/1486882298 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c0400ca30 con 0x7f1bf4003ff0 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1bfe7fc640 1 -- 172.21.15.81:0/1486882298 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0400ce80 con 0x7f1bf4003ff0 2024-06-29T14:23:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1486882298 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c080c6780 con 0x7f1bf4003ff0 2024-06-29T14:23:23.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.806+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1486882298 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c081f5350 con 0x7f1bf4003ff0 2024-06-29T14:23:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.807+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1486882298 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c0806e2a0 con 0x7f1bf4003ff0 2024-06-29T14:23:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.808+0000 7f1bfe7fc640 1 -- 172.21.15.81:0/1486882298 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1c04018070 con 0x7f1bf4003ff0 2024-06-29T14:23:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.809+0000 7f1bfe7fc640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1bec0762d0 0x7f1bec0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.809+0000 7f1bfe7fc640 1 -- 172.21.15.81:0/1486882298 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5804+0+0 (secure 0 0 0) 0x7f1c04097a80 con 0x7f1bf4003ff0 2024-06-29T14:23:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.809+0000 7f1c0d197640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1bec0762d0 0x7f1bec0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.810+0000 7f1c0d197640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1bec0762d0 0x7f1bec0787e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8035000 tx=0x7f1bf80046c0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:23.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:23.813+0000 7f1bfe7fc640 1 -- 172.21.15.81:0/1486882298 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1c0806e2a0 con 0x7f1bf4003ff0 2024-06-29T14:23:23.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.001+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1486882298 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1c080b3d70 con 0x7f1bf4003ff0 2024-06-29T14:23:23.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.001+0000 7f1bfe7fc640 1 -- 172.21.15.81:0/1486882298 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f1c04069400 con 0x7f1bf4003ff0 2024-06-29T14:23:23.999 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:24.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1bec0762d0 msgr2=0x7f1bec0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1bec0762d0 0x7f1bec0787e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8035000 tx=0x7f1bf80046c0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1486882298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bf4003ff0 msgr2=0x7f1c080c5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bf4003ff0 0x7f1c080c5f50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400e600 tx=0x7f1c0400eac0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1486882298 shutdown_connections 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1bec0762d0 0x7f1bec0787e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c081f1b50 0x7f1c081f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c0806c4d0 0x7f1c0806c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 --2- 172.21.15.81:0/1486882298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bf4003ff0 0x7f1c080c5f50 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1486882298 >> 172.21.15.81:0/1486882298 conn(0x7f1c080d0930 msgr2=0x7f1c080cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1486882298 shutdown_connections 2024-06-29T14:23:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.003+0000 7f1c0fbfc640 1 -- 172.21.15.81:0/1486882298 wait complete. 2024-06-29T14:23:24.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:23:24.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.131+0000 7f3589aef640 1 -- 172.21.15.81:0/1660908115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358406f850 msgr2=0x7f35840ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.131+0000 7f3589aef640 1 --2- 172.21.15.81:0/1660908115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358406f850 0x7f35840ae300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3578005fd0 tx=0x7f357802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.131+0000 7f3589aef640 1 -- 172.21.15.81:0/1660908115 shutdown_connections 2024-06-29T14:23:24.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.131+0000 7f3589aef640 1 --2- 172.21.15.81:0/1660908115 >> v2:172.21.15.167:3300/0 conn(0x7f35840ae840 0x7f35840aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.131+0000 7f3589aef640 1 --2- 172.21.15.81:0/1660908115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358406f850 0x7f35840ae300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.131+0000 7f3589aef640 1 -- 172.21.15.81:0/1660908115 >> 172.21.15.81:0/1660908115 conn(0x7f3584072950 msgr2=0x7f35840b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:24.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.131+0000 7f3589aef640 1 -- 172.21.15.81:0/1660908115 shutdown_connections 2024-06-29T14:23:24.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.131+0000 7f3589aef640 1 -- 172.21.15.81:0/1660908115 wait complete. 2024-06-29T14:23:24.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.133+0000 7f3589aef640 1 Processor -- start 2024-06-29T14:23:24.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.133+0000 7f3589aef640 1 -- start start 2024-06-29T14:23:24.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3589aef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358406f850 0x7f3584145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3589aef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3584074520 0x7f35840ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3589aef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840ae840 0x7f3584148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3589aef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f358415b190 con 0x7f3584074520 2024-06-29T14:23:24.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3589aef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f358415b010 con 0x7f35840ae840 2024-06-29T14:23:24.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3589aef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f35840af190 con 0x7f358406f850 2024-06-29T14:23:24.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3582ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3584074520 0x7f35840ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3582ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3584074520 0x7f35840ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59938/0 (socket says 172.21.15.81:59938) 2024-06-29T14:23:24.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3582ffd640 1 -- 172.21.15.81:0/175413003 learned_addr learned my addr 172.21.15.81:0/175413003 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3583fff640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840ae840 0x7f3584148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f35837fe640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358406f850 0x7f3584145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3582ffd640 1 -- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358406f850 msgr2=0x7f3584145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3582ffd640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358406f850 0x7f3584145d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3582ffd640 1 -- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840ae840 msgr2=0x7f3584148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3582ffd640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840ae840 0x7f3584148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3582ffd640 1 -- 172.21.15.81:0/175413003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35840ceee0 con 0x7f3584074520 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3582ffd640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3584074520 0x7f35840ce4f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3578034040 tx=0x7f35780035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f356bfff640 1 -- 172.21.15.81:0/175413003 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357803e030 con 0x7f3584074520 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f356bfff640 1 -- 172.21.15.81:0/175413003 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3578003b00 con 0x7f3584074520 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.135+0000 7f3589aef640 1 -- 172.21.15.81:0/175413003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35840cf1c0 con 0x7f3584074520 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.136+0000 7f356bfff640 1 -- 172.21.15.81:0/175413003 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3578035040 con 0x7f3584074520 2024-06-29T14:23:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.136+0000 7f3589aef640 1 -- 172.21.15.81:0/175413003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35840cea30 con 0x7f3584074520 2024-06-29T14:23:24.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.137+0000 7f3589aef640 1 -- 172.21.15.81:0/175413003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35840ae300 con 0x7f3584074520 2024-06-29T14:23:24.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.137+0000 7f356bfff640 1 -- 172.21.15.81:0/175413003 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3578041030 con 0x7f3584074520 2024-06-29T14:23:24.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.138+0000 7f356bfff640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f35700762d0 0x7f35700787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.138+0000 7f356bfff640 1 -- 172.21.15.81:0/175413003 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5804+0+0 (secure 0 0 0) 0x7f35780b7b60 con 0x7f3584074520 2024-06-29T14:23:24.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.138+0000 7f35837fe640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f35700762d0 0x7f35700787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.139+0000 7f35837fe640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f35700762d0 0x7f35700787e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3574001e20 tx=0x7f3574009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:24.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.142+0000 7f356bfff640 1 -- 172.21.15.81:0/175413003 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3578080d80 con 0x7f3584074520 2024-06-29T14:23:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.321+0000 7f3589aef640 1 -- 172.21.15.81:0/175413003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f35841e3550 con 0x7f3584074520 2024-06-29T14:23:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.321+0000 7f356bfff640 1 -- 172.21.15.81:0/175413003 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f3578085080 con 0x7f3584074520 2024-06-29T14:23:24.320 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.323+0000 7f3589aef640 1 -- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f35700762d0 msgr2=0x7f35700787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.323+0000 7f3589aef640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f35700762d0 0x7f35700787e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3574001e20 tx=0x7f3574009040 comp rx=0 tx=0).stop 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.323+0000 7f3589aef640 1 -- 172.21.15.81:0/175413003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3584074520 msgr2=0x7f35840ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.323+0000 7f3589aef640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3584074520 0x7f35840ce4f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3578034040 tx=0x7f35780035f0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.323+0000 7f3589aef640 1 -- 172.21.15.81:0/175413003 shutdown_connections 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.323+0000 7f3589aef640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f35700762d0 0x7f35700787e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.323+0000 7f3589aef640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35840ae840 0x7f3584148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.323+0000 7f3589aef640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3584074520 0x7f35840ce4f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.323+0000 7f3589aef640 1 --2- 172.21.15.81:0/175413003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358406f850 0x7f3584145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.323+0000 7f3589aef640 1 -- 172.21.15.81:0/175413003 >> 172.21.15.81:0/175413003 conn(0x7f3584072950 msgr2=0x7f35840afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.324+0000 7f3589aef640 1 -- 172.21.15.81:0/175413003 shutdown_connections 2024-06-29T14:23:24.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.324+0000 7f3589aef640 1 -- 172.21.15.81:0/175413003 wait complete. 2024-06-29T14:23:24.333 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:23:24.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.451+0000 7f24fe6ac640 1 -- 172.21.15.81:0/1826614286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80d2a90 msgr2=0x7f24f80d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.451+0000 7f24fe6ac640 1 --2- 172.21.15.81:0/1826614286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80d2a90 0x7f24f80d09c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f24f0005fd0 tx=0x7f24f002aa90 comp rx=0 tx=0).stop 2024-06-29T14:23:24.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.451+0000 7f24fe6ac640 1 -- 172.21.15.81:0/1826614286 shutdown_connections 2024-06-29T14:23:24.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.451+0000 7f24fe6ac640 1 --2- 172.21.15.81:0/1826614286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80d2a90 0x7f24f80d09c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.451+0000 7f24fe6ac640 1 --2- 172.21.15.81:0/1826614286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f80b9040 0x7f24f80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.451+0000 7f24fe6ac640 1 -- 172.21.15.81:0/1826614286 >> 172.21.15.81:0/1826614286 conn(0x7f24f809cf30 msgr2=0x7f24f80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:24.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.451+0000 7f24fe6ac640 1 -- 172.21.15.81:0/1826614286 shutdown_connections 2024-06-29T14:23:24.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.451+0000 7f24fe6ac640 1 -- 172.21.15.81:0/1826614286 wait complete. 2024-06-29T14:23:24.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.453+0000 7f24fe6ac640 1 Processor -- start 2024-06-29T14:23:24.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.453+0000 7f24fe6ac640 1 -- start start 2024-06-29T14:23:24.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24fe6ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80b5250 0x7f24f809dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24fe6ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24f80b9040 0x7f24f8151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24fe6ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f80d2a90 0x7f24f815d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24fe6ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f24f80b7d30 con 0x7f24f80b5250 2024-06-29T14:23:24.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24fe6ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f24f80b7bb0 con 0x7f24f80b9040 2024-06-29T14:23:24.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24fe6ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f24f80b7eb0 con 0x7f24f80d2a90 2024-06-29T14:23:24.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80b5250 0x7f24f809dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80b5250 0x7f24f809dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59968/0 (socket says 172.21.15.81:59968) 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f7fff640 1 -- 172.21.15.81:0/3790310234 learned_addr learned my addr 172.21.15.81:0/3790310234 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24fcc49640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f80d2a90 0x7f24f815d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f7fff640 1 -- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f80d2a90 msgr2=0x7f24f815d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f7fff640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f80d2a90 0x7f24f815d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f77fe640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24f80b9040 0x7f24f8151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f7fff640 1 -- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24f80b9040 msgr2=0x7f24f8151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f7fff640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24f80b9040 0x7f24f8151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f7fff640 1 -- 172.21.15.81:0/3790310234 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f81515a0 con 0x7f24f80b5250 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24fcc49640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f80d2a90 0x7f24f815d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f77fe640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24f80b9040 0x7f24f8151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f7fff640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80b5250 0x7f24f809dd40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f24e800b410 tx=0x7f24e800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:24.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f4ff9640 1 -- 172.21.15.81:0/3790310234 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8015020 con 0x7f24f80b5250 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f4ff9640 1 -- 172.21.15.81:0/3790310234 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24e8011940 con 0x7f24f80b5250 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24fe6ac640 1 -- 172.21.15.81:0/3790310234 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f80a0d30 con 0x7f24f80b5250 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24f4ff9640 1 -- 172.21.15.81:0/3790310234 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8011d90 con 0x7f24f80b5250 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.455+0000 7f24fe6ac640 1 -- 172.21.15.81:0/3790310234 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f80a12f0 con 0x7f24f80b5250 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.456+0000 7f24fe6ac640 1 -- 172.21.15.81:0/3790310234 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f80b8440 con 0x7f24f80b5250 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.457+0000 7f24f4ff9640 1 -- 172.21.15.81:0/3790310234 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f24e8018020 con 0x7f24f80b5250 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.457+0000 7f24f4ff9640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f24e40762d0 0x7f24e40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.457+0000 7f24f4ff9640 1 -- 172.21.15.81:0/3790310234 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5804+0+0 (secure 0 0 0) 0x7f24e80977f0 con 0x7f24f80b5250 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.458+0000 7f24f77fe640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f24e40762d0 0x7f24e40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.458+0000 7f24f77fe640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f24e40762d0 0x7f24e40787e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f24f815dea0 tx=0x7f24ec008040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:24.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.462+0000 7f24f4ff9640 1 -- 172.21.15.81:0/3790310234 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f24e801f020 con 0x7f24f80b5250 2024-06-29T14:23:24.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.641+0000 7f24fe6ac640 1 -- 172.21.15.81:0/3790310234 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f24f80d1e40 con 0x7f24f80b5250 2024-06-29T14:23:24.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.642+0000 7f24f4ff9640 1 -- 172.21.15.81:0/3790310234 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f24e8061070 con 0x7f24f80b5250 2024-06-29T14:23:24.640 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:23:24.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 -- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f24e40762d0 msgr2=0x7f24e40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f24e40762d0 0x7f24e40787e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f24f815dea0 tx=0x7f24ec008040 comp rx=0 tx=0).stop 2024-06-29T14:23:24.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 -- 172.21.15.81:0/3790310234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80b5250 msgr2=0x7f24f809dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80b5250 0x7f24f809dd40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f24e800b410 tx=0x7f24e800b8d0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 -- 172.21.15.81:0/3790310234 shutdown_connections 2024-06-29T14:23:24.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f24e40762d0 0x7f24e40787e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f80d2a90 0x7f24f815d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24f80b9040 0x7f24f8151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 --2- 172.21.15.81:0/3790310234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80b5250 0x7f24f809dd40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 -- 172.21.15.81:0/3790310234 >> 172.21.15.81:0/3790310234 conn(0x7f24f809cf30 msgr2=0x7f24f80d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:24.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 -- 172.21.15.81:0/3790310234 shutdown_connections 2024-06-29T14:23:24.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.644+0000 7f24fe6ac640 1 -- 172.21.15.81:0/3790310234 wait complete. 2024-06-29T14:23:24.654 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:23:24.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.771+0000 7f66852b2640 1 -- 172.21.15.81:0/1697079894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6668003a50 msgr2=0x7f6668003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.771+0000 7f66852b2640 1 --2- 172.21.15.81:0/1697079894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6668003a50 0x7f6668003ec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6670006fa0 tx=0x7f6668005830 comp rx=0 tx=0).stop 2024-06-29T14:23:24.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.771+0000 7f66852b2640 1 -- 172.21.15.81:0/1697079894 shutdown_connections 2024-06-29T14:23:24.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.771+0000 7f66852b2640 1 --2- 172.21.15.81:0/1697079894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6668003a50 0x7f6668003ec0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.771+0000 7f66852b2640 1 -- 172.21.15.81:0/1697079894 >> 172.21.15.81:0/1697079894 conn(0x7f66800d1960 msgr2=0x7f66800d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:24.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.771+0000 7f66852b2640 1 -- 172.21.15.81:0/1697079894 shutdown_connections 2024-06-29T14:23:24.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.771+0000 7f66852b2640 1 -- 172.21.15.81:0/1697079894 wait complete. 2024-06-29T14:23:24.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.773+0000 7f66852b2640 1 Processor -- start 2024-06-29T14:23:24.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.773+0000 7f66852b2640 1 -- start start 2024-06-29T14:23:24.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66852b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6668003a50 0x7f66801ded20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66852b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800c9f50 0x7f66800b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66852b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800b3640 0x7f66800b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66852b2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f66800d0890 con 0x7f6668003a50 2024-06-29T14:23:24.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66852b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f66800d0710 con 0x7f66800c9f50 2024-06-29T14:23:24.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66852b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f66800d0a10 con 0x7f66800b3640 2024-06-29T14:23:24.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6668003a50 0x7f66801ded20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6668003a50 0x7f66801ded20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59988/0 (socket says 172.21.15.81:59988) 2024-06-29T14:23:24.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667effd640 1 -- 172.21.15.81:0/2708727949 learned_addr learned my addr 172.21.15.81:0/2708727949 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667e7fc640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800c9f50 0x7f66800b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667f7fe640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800b3640 0x7f66800b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667effd640 1 -- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800b3640 msgr2=0x7f66800b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667effd640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800b3640 0x7f66800b3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667effd640 1 -- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800c9f50 msgr2=0x7f66800b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667effd640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800c9f50 0x7f66800b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667effd640 1 -- 172.21.15.81:0/2708727949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66800b3f90 con 0x7f6668003a50 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667e7fc640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800c9f50 0x7f66800b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f667effd640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6668003a50 0x7f66801ded20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f667000c320 tx=0x7f6670001d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66677fe640 1 -- 172.21.15.81:0/2708727949 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670032030 con 0x7f6668003a50 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66677fe640 1 -- 172.21.15.81:0/2708727949 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6670003b10 con 0x7f6668003a50 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66852b2640 1 -- 172.21.15.81:0/2708727949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66800b13a0 con 0x7f6668003a50 2024-06-29T14:23:24.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66677fe640 1 -- 172.21.15.81:0/2708727949 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667002e480 con 0x7f6668003a50 2024-06-29T14:23:24.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.775+0000 7f66852b2640 1 -- 172.21.15.81:0/2708727949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66800ae710 con 0x7f6668003a50 2024-06-29T14:23:24.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.776+0000 7f66852b2640 1 -- 172.21.15.81:0/2708727949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66800ce7c0 con 0x7f6668003a50 2024-06-29T14:23:24.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.777+0000 7f66677fe640 1 -- 172.21.15.81:0/2708727949 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6670035020 con 0x7f6668003a50 2024-06-29T14:23:24.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.778+0000 7f66677fe640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f666c0762d0 0x7f666c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:24.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.778+0000 7f66677fe640 1 -- 172.21.15.81:0/2708727949 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5804+0+0 (secure 0 0 0) 0x7f66700b7d20 con 0x7f6668003a50 2024-06-29T14:23:24.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.778+0000 7f667e7fc640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f666c0762d0 0x7f666c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:24.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.778+0000 7f667e7fc640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f666c0762d0 0x7f666c0787e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f66800c8610 tx=0x7f6674006be0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:24.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:24.782+0000 7f66677fe640 1 -- 172.21.15.81:0/2708727949 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6670047030 con 0x7f6668003a50 2024-06-29T14:23:24.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.000+0000 7f66852b2640 1 -- 172.21.15.81:0/2708727949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66800cdec0 con 0x7f6668003a50 2024-06-29T14:23:24.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.001+0000 7f66677fe640 1 -- 172.21.15.81:0/2708727949 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5217 (secure 0 0 0) 0x7f66800b3f90 con 0x7f6668003a50 2024-06-29T14:23:24.999 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:24.999 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"btime":"2024-06-29T14:22:47:707315+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4295,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/96042442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":96042442},{"type":"v1","addr":"172.21.15.167:6837","nonce":96042442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3681460169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3681460169},{"type":"v1","addr":"172.21.15.81:6834","nonce":3681460169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3175849002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3175849002},{"type":"v1","addr":"172.21.15.81:6835","nonce":3175849002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:22:44.057632+0000","modified":"2024-06-29T14:22:47.079866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4360":{"gid":4360,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/3579540962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3579540962},{"type":"v1","addr":"172.21.15.167:6836","nonce":3579540962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4360,"qdb_cluster":[4360]},"id":1}]} 2024-06-29T14:23:25.000 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2024-06-29T14:23:25.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 -- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f666c0762d0 msgr2=0x7f666c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f666c0762d0 0x7f666c0787e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f66800c8610 tx=0x7f6674006be0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 -- 172.21.15.81:0/2708727949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6668003a50 msgr2=0x7f66801ded20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6668003a50 0x7f66801ded20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f667000c320 tx=0x7f6670001d60 comp rx=0 tx=0).stop 2024-06-29T14:23:25.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 -- 172.21.15.81:0/2708727949 shutdown_connections 2024-06-29T14:23:25.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f666c0762d0 0x7f666c0787e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66800b3640 0x7f66800b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66800c9f50 0x7f66800b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 --2- 172.21.15.81:0/2708727949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6668003a50 0x7f66801ded20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 -- 172.21.15.81:0/2708727949 >> 172.21.15.81:0/2708727949 conn(0x7f66800d1960 msgr2=0x7f66800b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 -- 172.21.15.81:0/2708727949 shutdown_connections 2024-06-29T14:23:25.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.003+0000 7f66852b2640 1 -- 172.21.15.81:0/2708727949 wait complete. 2024-06-29T14:23:25.012 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:25.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.128+0000 7f6069b63640 1 -- 172.21.15.81:0/775355255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60640d0940 msgr2=0x7f60640b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.128+0000 7f6069b63640 1 --2- 172.21.15.81:0/775355255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60640d0940 0x7f60640b4f40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f605c005fd0 tx=0x7f605c02aa00 comp rx=0 tx=0).stop 2024-06-29T14:23:25.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.128+0000 7f6069b63640 1 -- 172.21.15.81:0/775355255 shutdown_connections 2024-06-29T14:23:25.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.128+0000 7f6069b63640 1 --2- 172.21.15.81:0/775355255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60640d0940 0x7f60640b4f40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.128+0000 7f6069b63640 1 --2- 172.21.15.81:0/775355255 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60640b3200 0x7f60640cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.128+0000 7f6069b63640 1 -- 172.21.15.81:0/775355255 >> 172.21.15.81:0/775355255 conn(0x7f60640b3680 msgr2=0x7f60640b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.128+0000 7f6069b63640 1 -- 172.21.15.81:0/775355255 shutdown_connections 2024-06-29T14:23:25.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.128+0000 7f6069b63640 1 -- 172.21.15.81:0/775355255 wait complete. 2024-06-29T14:23:25.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.130+0000 7f6069b63640 1 Processor -- start 2024-06-29T14:23:25.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.130+0000 7f6069b63640 1 -- start start 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f6069b63640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f60640b3200 0x7f6064155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f6069b63640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60640cccf0 0x7f60641e1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f6069b63640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60641562a0 0x7f6064158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f6069b63640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f60640b5c50 con 0x7f60640cccf0 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f6069b63640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f60640b5ad0 con 0x7f60640b3200 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f6069b63640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f60640b5dd0 con 0x7f60641562a0 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f6062ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60640cccf0 0x7f60641e1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f6062ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60640cccf0 0x7f60641e1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60010/0 (socket says 172.21.15.81:60010) 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f6062ffd640 1 -- 172.21.15.81:0/959963760 learned_addr learned my addr 172.21.15.81:0/959963760 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f60637fe640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f60640b3200 0x7f6064155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.131+0000 7f6063fff640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60641562a0 0x7f6064158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f6062ffd640 1 -- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60641562a0 msgr2=0x7f6064158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f6062ffd640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60641562a0 0x7f6064158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f6062ffd640 1 -- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f60640b3200 msgr2=0x7f6064155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f6062ffd640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f60640b3200 0x7f6064155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f6062ffd640 1 -- 172.21.15.81:0/959963760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60641e1e80 con 0x7f60640cccf0 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f6063fff640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60641562a0 0x7f6064158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f6062ffd640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60640cccf0 0x7f60641e1510 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f605800b3c0 tx=0x7f605800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f604bfff640 1 -- 172.21.15.81:0/959963760 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058011830 con 0x7f60640cccf0 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f604bfff640 1 -- 172.21.15.81:0/959963760 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6058016070 con 0x7f60640cccf0 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f6069b63640 1 -- 172.21.15.81:0/959963760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60641e2160 con 0x7f60640cccf0 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f604bfff640 1 -- 172.21.15.81:0/959963760 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058011d60 con 0x7f60640cccf0 2024-06-29T14:23:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.132+0000 7f6069b63640 1 -- 172.21.15.81:0/959963760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60641e2720 con 0x7f60640cccf0 2024-06-29T14:23:25.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.133+0000 7f6069b63640 1 -- 172.21.15.81:0/959963760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60640cc7b0 con 0x7f60640cccf0 2024-06-29T14:23:25.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.134+0000 7f604bfff640 1 -- 172.21.15.81:0/959963760 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f605801b460 con 0x7f60640cccf0 2024-06-29T14:23:25.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.134+0000 7f604bfff640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6050076300 0x7f6050078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.134+0000 7f604bfff640 1 -- 172.21.15.81:0/959963760 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5804+0+0 (secure 0 0 0) 0x7f60580973f0 con 0x7f60640cccf0 2024-06-29T14:23:25.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.135+0000 7f60637fe640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6050076300 0x7f6050078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.135+0000 7f60637fe640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6050076300 0x7f6050078810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6054004630 tx=0x7f605400b2e0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:25.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.139+0000 7f604bfff640 1 -- 172.21.15.81:0/959963760 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6058060d80 con 0x7f60640cccf0 2024-06-29T14:23:25.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.320+0000 7f6069b63640 1 -- 172.21.15.81:0/959963760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f606415a0e0 con 0x7f60640cccf0 2024-06-29T14:23:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.321+0000 7f604bfff640 1 -- 172.21.15.81:0/959963760 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16814 (secure 0 0 0) 0x7f6058060680 con 0x7f60640cccf0 2024-06-29T14:23:25.320 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:25.320 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":22,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:23:23.844041+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1},{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:25.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 -- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6050076300 msgr2=0x7f6050078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6050076300 0x7f6050078810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6054004630 tx=0x7f605400b2e0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 -- 172.21.15.81:0/959963760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60640cccf0 msgr2=0x7f60641e1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60640cccf0 0x7f60641e1510 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f605800b3c0 tx=0x7f605800b880 comp rx=0 tx=0).stop 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 -- 172.21.15.81:0/959963760 shutdown_connections 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6050076300 0x7f6050078810 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60641562a0 0x7f6064158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60640cccf0 0x7f60641e1510 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 --2- 172.21.15.81:0/959963760 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f60640b3200 0x7f6064155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 -- 172.21.15.81:0/959963760 >> 172.21.15.81:0/959963760 conn(0x7f60640b3680 msgr2=0x7f60640cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 -- 172.21.15.81:0/959963760 shutdown_connections 2024-06-29T14:23:25.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.323+0000 7f6069b63640 1 -- 172.21.15.81:0/959963760 wait complete. 2024-06-29T14:23:25.334 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:23:25.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:23:25.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.448+0000 7f576c363640 1 -- 172.21.15.81:0/3965825547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f576406fa90 msgr2=0x7f576406c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.448+0000 7f576c363640 1 --2- 172.21.15.81:0/3965825547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f576406fa90 0x7f576406c830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5764072d60 tx=0x7f575402aa30 comp rx=0 tx=0).stop 2024-06-29T14:23:25.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.449+0000 7f576c363640 1 -- 172.21.15.81:0/3965825547 shutdown_connections 2024-06-29T14:23:25.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.449+0000 7f576c363640 1 --2- 172.21.15.81:0/3965825547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f576406fa90 0x7f576406c830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.449+0000 7f576c363640 1 --2- 172.21.15.81:0/3965825547 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f576406f680 0x7f576406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.449+0000 7f576c363640 1 --2- 172.21.15.81:0/3965825547 >> v2:172.21.15.167:3300/0 conn(0x7f5764074520 0x7f576406f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.449+0000 7f576c363640 1 -- 172.21.15.81:0/3965825547 >> 172.21.15.81:0/3965825547 conn(0x7f5764072950 msgr2=0x7f57640b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.449+0000 7f576c363640 1 -- 172.21.15.81:0/3965825547 shutdown_connections 2024-06-29T14:23:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.449+0000 7f576c363640 1 -- 172.21.15.81:0/3965825547 wait complete. 2024-06-29T14:23:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.450+0000 7f576c363640 1 Processor -- start 2024-06-29T14:23:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.450+0000 7f576c363640 1 -- start start 2024-06-29T14:23:25.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576c363640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f576406f680 0x7f57641505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576c363640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f576406fa90 0x7f576414dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576c363640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5764074520 0x7f57640ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576c363640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f576406cd70 con 0x7f576406f680 2024-06-29T14:23:25.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576c363640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f576415b220 con 0x7f5764074520 2024-06-29T14:23:25.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576c363640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f576406cef0 con 0x7f576406fa90 2024-06-29T14:23:25.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576a0ff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f576406f680 0x7f57641505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576a0ff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f576406f680 0x7f57641505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60030/0 (socket says 172.21.15.81:60030) 2024-06-29T14:23:25.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576a0ff640 1 -- 172.21.15.81:0/307015057 learned_addr learned my addr 172.21.15.81:0/307015057 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576a900640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5764074520 0x7f57640ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f57698fe640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f576406fa90 0x7f576414dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576a0ff640 1 -- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f576406fa90 msgr2=0x7f576414dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576a0ff640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f576406fa90 0x7f576414dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576a0ff640 1 -- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5764074520 msgr2=0x7f57640ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576a0ff640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5764074520 0x7f57640ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576a0ff640 1 -- 172.21.15.81:0/307015057 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576414ebd0 con 0x7f576406f680 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576a0ff640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f576406f680 0x7f57641505c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f575c009830 tx=0x7f575c009cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f5762ffd640 1 -- 172.21.15.81:0/307015057 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c01d070 con 0x7f576406f680 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f5762ffd640 1 -- 172.21.15.81:0/307015057 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f575c0077a0 con 0x7f576406f680 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f5762ffd640 1 -- 172.21.15.81:0/307015057 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c007bf0 con 0x7f576406f680 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.452+0000 7f576c363640 1 -- 172.21.15.81:0/307015057 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f576414f800 con 0x7f576406f680 2024-06-29T14:23:25.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.453+0000 7f576c363640 1 -- 172.21.15.81:0/307015057 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f576414fdc0 con 0x7f576406f680 2024-06-29T14:23:25.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.453+0000 7f576c363640 1 -- 172.21.15.81:0/307015057 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f576406f140 con 0x7f576406f680 2024-06-29T14:23:25.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.454+0000 7f5762ffd640 1 -- 172.21.15.81:0/307015057 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f575c020070 con 0x7f576406f680 2024-06-29T14:23:25.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.455+0000 7f5762ffd640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57500762d0 0x7f57500787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.455+0000 7f5762ffd640 1 -- 172.21.15.81:0/307015057 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5804+0+0 (secure 0 0 0) 0x7f575c098fd0 con 0x7f576406f680 2024-06-29T14:23:25.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.455+0000 7f57698fe640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57500762d0 0x7f57500787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.455+0000 7f57698fe640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57500762d0 0x7f57500787e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f57640cfef0 tx=0x7f575800b260 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:25.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.459+0000 7f5762ffd640 1 -- 172.21.15.81:0/307015057 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f575c0622b0 con 0x7f576406f680 2024-06-29T14:23:25.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.679+0000 7f576c363640 1 -- 172.21.15.81:0/307015057 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f576406a520 con 0x7f576406f680 2024-06-29T14:23:25.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.679+0000 7f5762ffd640 1 -- 172.21.15.81:0/307015057 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5217 (secure 0 0 0) 0x7f575c0665b0 con 0x7f576406f680 2024-06-29T14:23:25.678 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:25.679 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"btime":"2024-06-29T14:22:47:707315+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4295,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/96042442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":96042442},{"type":"v1","addr":"172.21.15.167:6837","nonce":96042442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3681460169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3681460169},{"type":"v1","addr":"172.21.15.81:6834","nonce":3681460169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3175849002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3175849002},{"type":"v1","addr":"172.21.15.81:6835","nonce":3175849002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:22:44.057632+0000","modified":"2024-06-29T14:22:47.079866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4360":{"gid":4360,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/3579540962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3579540962},{"type":"v1","addr":"172.21.15.167:6836","nonce":3579540962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4360,"qdb_cluster":[4360]},"id":1}]} 2024-06-29T14:23:25.679 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2024-06-29T14:23:25.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 -- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57500762d0 msgr2=0x7f57500787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57500762d0 0x7f57500787e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f57640cfef0 tx=0x7f575800b260 comp rx=0 tx=0).stop 2024-06-29T14:23:25.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 -- 172.21.15.81:0/307015057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f576406f680 msgr2=0x7f57641505c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f576406f680 0x7f57641505c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f575c009830 tx=0x7f575c009cf0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 -- 172.21.15.81:0/307015057 shutdown_connections 2024-06-29T14:23:25.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f57500762d0 0x7f57500787e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5764074520 0x7f57640ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f576406fa90 0x7f576414dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 --2- 172.21.15.81:0/307015057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f576406f680 0x7f57641505c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 -- 172.21.15.81:0/307015057 >> 172.21.15.81:0/307015057 conn(0x7f5764072950 msgr2=0x7f5764157db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.681+0000 7f576c363640 1 -- 172.21.15.81:0/307015057 shutdown_connections 2024-06-29T14:23:25.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.682+0000 7f576c363640 1 -- 172.21.15.81:0/307015057 wait complete. 2024-06-29T14:23:25.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:23:25.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.805+0000 7fc6a303c640 1 -- 172.21.15.81:0/2550352181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc69c0b6eb0 msgr2=0x7fc69c0d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.805+0000 7fc6a303c640 1 --2- 172.21.15.81:0/2550352181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc69c0b6eb0 0x7fc69c0d02d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0b41b0 tx=0x7fc68c02a490 comp rx=0 tx=0).stop 2024-06-29T14:23:25.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.805+0000 7fc6a303c640 1 -- 172.21.15.81:0/2550352181 shutdown_connections 2024-06-29T14:23:25.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.805+0000 7fc6a303c640 1 --2- 172.21.15.81:0/2550352181 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc69c0ce510 0x7fc69c0ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.805+0000 7fc6a303c640 1 --2- 172.21.15.81:0/2550352181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc69c0b6eb0 0x7fc69c0d02d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.805+0000 7fc6a303c640 1 -- 172.21.15.81:0/2550352181 >> 172.21.15.81:0/2550352181 conn(0x7fc69c0b3510 msgr2=0x7fc69c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:25.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.805+0000 7fc6a303c640 1 -- 172.21.15.81:0/2550352181 shutdown_connections 2024-06-29T14:23:25.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.805+0000 7fc6a303c640 1 -- 172.21.15.81:0/2550352181 wait complete. 2024-06-29T14:23:25.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.807+0000 7fc6a303c640 1 Processor -- start 2024-06-29T14:23:25.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.807+0000 7fc6a303c640 1 -- start start 2024-06-29T14:23:25.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a303c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc69c0b4810 0x7fc69c15f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a303c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc69c0b6eb0 0x7fc69c14dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a303c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc69c0ce510 0x7fc69c160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a303c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c0cd480 con 0x7fc69c0b4810 2024-06-29T14:23:25.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a303c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c0b29c0 con 0x7fc69c0ce510 2024-06-29T14:23:25.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a303c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc69c0cd600 con 0x7fc69c0b6eb0 2024-06-29T14:23:25.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a0dd8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc69c0b4810 0x7fc69c15f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a0dd8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc69c0b4810 0x7fc69c15f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60038/0 (socket says 172.21.15.81:60038) 2024-06-29T14:23:25.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a0dd8640 1 -- 172.21.15.81:0/510264324 learned_addr learned my addr 172.21.15.81:0/510264324 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:25.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc69bfff640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc69c0b6eb0 0x7fc69c14dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a15d9640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc69c0ce510 0x7fc69c160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a0dd8640 1 -- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc69c0b6eb0 msgr2=0x7fc69c14dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a0dd8640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc69c0b6eb0 0x7fc69c14dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a0dd8640 1 -- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc69c0ce510 msgr2=0x7fc69c160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:25.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a0dd8640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc69c0ce510 0x7fc69c160190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:25.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.808+0000 7fc6a0dd8640 1 -- 172.21.15.81:0/510264324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c165f80 con 0x7fc69c0b4810 2024-06-29T14:23:25.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.809+0000 7fc6a0dd8640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc69c0b4810 0x7fc69c15f620 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc68c034040 tx=0x7fc68c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:25.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.809+0000 7fc6997fa640 1 -- 172.21.15.81:0/510264324 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68c03e030 con 0x7fc69c0b4810 2024-06-29T14:23:25.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.809+0000 7fc6997fa640 1 -- 172.21.15.81:0/510264324 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc68c0047a0 con 0x7fc69c0b4810 2024-06-29T14:23:25.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.809+0000 7fc6a303c640 1 -- 172.21.15.81:0/510264324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c0c9c20 con 0x7fc69c0b4810 2024-06-29T14:23:25.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.809+0000 7fc6997fa640 1 -- 172.21.15.81:0/510264324 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68c035040 con 0x7fc69c0b4810 2024-06-29T14:23:25.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.809+0000 7fc6a303c640 1 -- 172.21.15.81:0/510264324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c0ca1e0 con 0x7fc69c0b4810 2024-06-29T14:23:25.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.810+0000 7fc6a303c640 1 -- 172.21.15.81:0/510264324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc69c0d02d0 con 0x7fc69c0b4810 2024-06-29T14:23:25.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.811+0000 7fc6997fa640 1 -- 172.21.15.81:0/510264324 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc68c041030 con 0x7fc69c0b4810 2024-06-29T14:23:25.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.811+0000 7fc6997fa640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc688076300 0x7fc688078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:25.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.811+0000 7fc6997fa640 1 -- 172.21.15.81:0/510264324 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5804+0+0 (secure 0 0 0) 0x7fc68c0b8100 con 0x7fc69c0b4810 2024-06-29T14:23:25.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.812+0000 7fc69bfff640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc688076300 0x7fc688078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:25.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.812+0000 7fc69bfff640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc688076300 0x7fc688078810 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc69000a2e0 tx=0x7fc69000a270 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:25.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:25.816+0000 7fc6997fa640 1 -- 172.21.15.81:0/510264324 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc68c081970 con 0x7fc69c0b4810 2024-06-29T14:23:26.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.032+0000 7fc6a303c640 1 -- 172.21.15.81:0/510264324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc69c0cdf00 con 0x7fc69c0b4810 2024-06-29T14:23:26.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.033+0000 7fc6997fa640 1 -- 172.21.15.81:0/510264324 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5217 (secure 0 0 0) 0x7fc68c081270 con 0x7fc69c0b4810 2024-06-29T14:23:26.032 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:26.032 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"btime":"2024-06-29T14:22:47:707315+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4295,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/96042442","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":96042442},{"type":"v1","addr":"172.21.15.167:6837","nonce":96042442}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4344,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3681460169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3681460169},{"type":"v1","addr":"172.21.15.81:6834","nonce":3681460169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3175849002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3175849002},{"type":"v1","addr":"172.21.15.81:6835","nonce":3175849002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:22:44.057632+0000","modified":"2024-06-29T14:22:47.079866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4360},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4360":{"gid":4360,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/3579540962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3579540962},{"type":"v1","addr":"172.21.15.167:6836","nonce":3579540962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4360,"qdb_cluster":[4360]},"id":1}]} 2024-06-29T14:23:26.032 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2024-06-29T14:23:26.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 -- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc688076300 msgr2=0x7fc688078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc688076300 0x7fc688078810 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc69000a2e0 tx=0x7fc69000a270 comp rx=0 tx=0).stop 2024-06-29T14:23:26.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 -- 172.21.15.81:0/510264324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc69c0b4810 msgr2=0x7fc69c15f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc69c0b4810 0x7fc69c15f620 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc68c034040 tx=0x7fc68c0039e0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 -- 172.21.15.81:0/510264324 shutdown_connections 2024-06-29T14:23:26.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fc688076300 0x7fc688078810 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc69c0ce510 0x7fc69c160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc69c0b6eb0 0x7fc69c14dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 --2- 172.21.15.81:0/510264324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc69c0b4810 0x7fc69c15f620 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 -- 172.21.15.81:0/510264324 >> 172.21.15.81:0/510264324 conn(0x7fc69c0b3510 msgr2=0x7fc69c070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 -- 172.21.15.81:0/510264324 shutdown_connections 2024-06-29T14:23:26.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.035+0000 7fc6a303c640 1 -- 172.21.15.81:0/510264324 wait complete. 2024-06-29T14:23:26.046 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:26.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.161+0000 7f1c39fd4640 1 -- 172.21.15.81:0/3639400542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c1c005350 msgr2=0x7f1c1c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.161+0000 7f1c39fd4640 1 --2- 172.21.15.81:0/3639400542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c1c005350 0x7f1c1c00b9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1c340d4460 tx=0x7f1c1c005830 comp rx=0 tx=0).stop 2024-06-29T14:23:26.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.161+0000 7f1c39fd4640 1 -- 172.21.15.81:0/3639400542 shutdown_connections 2024-06-29T14:23:26.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.161+0000 7f1c39fd4640 1 --2- 172.21.15.81:0/3639400542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c1c005350 0x7f1c1c00b9c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.161+0000 7f1c39fd4640 1 -- 172.21.15.81:0/3639400542 >> 172.21.15.81:0/3639400542 conn(0x7f1c3409cf30 msgr2=0x7f1c340d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.161+0000 7f1c39fd4640 1 -- 172.21.15.81:0/3639400542 shutdown_connections 2024-06-29T14:23:26.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.161+0000 7f1c39fd4640 1 -- 172.21.15.81:0/3639400542 wait complete. 2024-06-29T14:23:26.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.163+0000 7f1c39fd4640 1 Processor -- start 2024-06-29T14:23:26.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.163+0000 7f1c39fd4640 1 -- start start 2024-06-29T14:23:26.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.164+0000 7f1c39fd4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c1c005350 0x7f1c3409fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.164+0000 7f1c39fd4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c3415d590 0x7f1c340a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.164+0000 7f1c39fd4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c341f7fa0 0x7f1c341f83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.164+0000 7f1c39fd4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1c340d0810 con 0x7f1c1c005350 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.164+0000 7f1c39fd4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1c340d0690 con 0x7f1c3415d590 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.164+0000 7f1c39fd4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1c340d0990 con 0x7f1c341f7fa0 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.164+0000 7f1c337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c1c005350 0x7f1c3409fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.164+0000 7f1c337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c1c005350 0x7f1c3409fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60068/0 (socket says 172.21.15.81:60068) 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.164+0000 7f1c337fe640 1 -- 172.21.15.81:0/2948121112 learned_addr learned my addr 172.21.15.81:0/2948121112 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c32ffd640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c3415d590 0x7f1c340a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c33fff640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c341f7fa0 0x7f1c341f83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c337fe640 1 -- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c341f7fa0 msgr2=0x7f1c341f83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c337fe640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c341f7fa0 0x7f1c341f83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c337fe640 1 -- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c3415d590 msgr2=0x7f1c340a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c337fe640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c3415d590 0x7f1c340a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c337fe640 1 -- 172.21.15.81:0/2948121112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c341f88f0 con 0x7f1c1c005350 2024-06-29T14:23:26.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c32ffd640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c3415d590 0x7f1c340a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:26.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c33fff640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c341f7fa0 0x7f1c341f83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:26.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c337fe640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c1c005350 0x7f1c3409fdc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1c24010a70 tx=0x7f1c24010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c1bfff640 1 -- 172.21.15.81:0/2948121112 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400ee00 con 0x7f1c1c005350 2024-06-29T14:23:26.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c1bfff640 1 -- 172.21.15.81:0/2948121112 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c24004b20 con 0x7f1c1c005350 2024-06-29T14:23:26.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c39fd4640 1 -- 172.21.15.81:0/2948121112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c340a0b10 con 0x7f1c1c005350 2024-06-29T14:23:26.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c1bfff640 1 -- 172.21.15.81:0/2948121112 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24005020 con 0x7f1c1c005350 2024-06-29T14:23:26.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.165+0000 7f1c39fd4640 1 -- 172.21.15.81:0/2948121112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c340a10d0 con 0x7f1c1c005350 2024-06-29T14:23:26.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.166+0000 7f1c39fd4640 1 -- 172.21.15.81:0/2948121112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c340d1a80 con 0x7f1c1c005350 2024-06-29T14:23:26.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.167+0000 7f1c1bfff640 1 -- 172.21.15.81:0/2948121112 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1c2401a030 con 0x7f1c1c005350 2024-06-29T14:23:26.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.167+0000 7f1c1bfff640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1c20076300 0x7f1c20078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.168+0000 7f1c1bfff640 1 -- 172.21.15.81:0/2948121112 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5804+0+0 (secure 0 0 0) 0x7f1c240986b0 con 0x7f1c1c005350 2024-06-29T14:23:26.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.168+0000 7f1c32ffd640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1c20076300 0x7f1c20078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.168+0000 7f1c32ffd640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1c20076300 0x7f1c20078810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1c28001010 tx=0x7f1c28001d00 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.171+0000 7f1c1bfff640 1 -- 172.21.15.81:0/2948121112 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1c341f88f0 con 0x7f1c1c005350 2024-06-29T14:23:26.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.352+0000 7f1c39fd4640 1 -- 172.21.15.81:0/2948121112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c340b10f0 con 0x7f1c1c005350 2024-06-29T14:23:26.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.354+0000 7f1c1bfff640 1 -- 172.21.15.81:0/2948121112 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16814 (secure 0 0 0) 0x7f1c240618d0 con 0x7f1c1c005350 2024-06-29T14:23:26.353 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:26.353 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":22,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:23:23.844041+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:22:41.898363+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-29T14:22:43.015911+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":5,"to":1},{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:26.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 -- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1c20076300 msgr2=0x7f1c20078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1c20076300 0x7f1c20078810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1c28001010 tx=0x7f1c28001d00 comp rx=0 tx=0).stop 2024-06-29T14:23:26.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 -- 172.21.15.81:0/2948121112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c1c005350 msgr2=0x7f1c3409fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c1c005350 0x7f1c3409fdc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1c24010a70 tx=0x7f1c24010f30 comp rx=0 tx=0).stop 2024-06-29T14:23:26.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 -- 172.21.15.81:0/2948121112 shutdown_connections 2024-06-29T14:23:26.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1c20076300 0x7f1c20078810 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c341f7fa0 0x7f1c341f83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c3415d590 0x7f1c340a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 --2- 172.21.15.81:0/2948121112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c1c005350 0x7f1c3409fdc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 -- 172.21.15.81:0/2948121112 >> 172.21.15.81:0/2948121112 conn(0x7f1c3409cf30 msgr2=0x7f1c3415e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 -- 172.21.15.81:0/2948121112 shutdown_connections 2024-06-29T14:23:26.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.356+0000 7f1c39fd4640 1 -- 172.21.15.81:0/2948121112 wait complete. 2024-06-29T14:23:26.366 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:23:26.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.482+0000 7f82168e4640 1 -- 172.21.15.81:0/3244625815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81f8005150 msgr2=0x7f81f800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.482+0000 7f82168e4640 1 --2- 172.21.15.81:0/3244625815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81f8005150 0x7f81f800b9c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8208009ed0 tx=0x7f81f8005830 comp rx=0 tx=0).stop 2024-06-29T14:23:26.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.482+0000 7f8214e81640 1 -- 172.21.15.81:0/3244625815 reap_dead start 2024-06-29T14:23:26.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.482+0000 7f82168e4640 1 -- 172.21.15.81:0/3244625815 shutdown_connections 2024-06-29T14:23:26.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.482+0000 7f82168e4640 1 -- 172.21.15.81:0/3244625815 >> 172.21.15.81:0/3244625815 conn(0x7f821009cf30 msgr2=0x7f82100d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.482+0000 7f82168e4640 1 -- 172.21.15.81:0/3244625815 shutdown_connections 2024-06-29T14:23:26.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.482+0000 7f82168e4640 1 -- 172.21.15.81:0/3244625815 wait complete. 2024-06-29T14:23:26.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.484+0000 7f82168e4640 1 Processor -- start 2024-06-29T14:23:26.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.484+0000 7f82168e4640 1 -- start start 2024-06-29T14:23:26.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.485+0000 7f82168e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f8005150 0x7f8210150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.485+0000 7f82168e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82100b2670 0x7f821015bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.485+0000 7f82168e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8210160840 0x7f8210160c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.485+0000 7f82168e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82100b8420 con 0x7f8210160840 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.485+0000 7f82168e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f82100b82a0 con 0x7f81f8005150 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.485+0000 7f82168e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f82100b85a0 con 0x7f82100b2670 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.485+0000 7f8214e81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8210160840 0x7f8210160c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.485+0000 7f8214e81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8210160840 0x7f8210160c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60078/0 (socket says 172.21.15.81:60078) 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.485+0000 7f8214e81640 1 -- 172.21.15.81:0/2696621242 learned_addr learned my addr 172.21.15.81:0/2696621242 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.485+0000 7f820f7fe640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82100b2670 0x7f821015bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f8214e81640 1 -- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82100b2670 msgr2=0x7f821015bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f820ffff640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f8005150 0x7f8210150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f8214e81640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82100b2670 0x7f821015bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f8214e81640 1 -- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f8005150 msgr2=0x7f8210150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f8214e81640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f8005150 0x7f8210150be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f8214e81640 1 -- 172.21.15.81:0/2696621242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8210161190 con 0x7f8210160840 2024-06-29T14:23:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f820f7fe640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82100b2670 0x7f821015bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f820ffff640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f8005150 0x7f8210150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f8214e81640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8210160840 0x7f8210160c50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8208009ed0 tx=0x7f8208033d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f820cff9640 1 -- 172.21.15.81:0/2696621242 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208044070 con 0x7f8210160840 2024-06-29T14:23:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f820cff9640 1 -- 172.21.15.81:0/2696621242 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8208039040 con 0x7f8210160840 2024-06-29T14:23:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f82168e4640 1 -- 172.21.15.81:0/2696621242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8210151120 con 0x7f8210160840 2024-06-29T14:23:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f820cff9640 1 -- 172.21.15.81:0/2696621242 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208038070 con 0x7f8210160840 2024-06-29T14:23:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.486+0000 7f82168e4640 1 -- 172.21.15.81:0/2696621242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f821014e7e0 con 0x7f8210160840 2024-06-29T14:23:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.487+0000 7f82168e4640 1 -- 172.21.15.81:0/2696621242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f821014e270 con 0x7f8210160840 2024-06-29T14:23:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.488+0000 7f820cff9640 1 -- 172.21.15.81:0/2696621242 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8208004030 con 0x7f8210160840 2024-06-29T14:23:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.488+0000 7f820cff9640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81fc076300 0x7f81fc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.488+0000 7f820cff9640 1 -- 172.21.15.81:0/2696621242 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5804+0+0 (secure 0 0 0) 0x7f82080b7c60 con 0x7f8210160840 2024-06-29T14:23:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.488+0000 7f820ffff640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81fc076300 0x7f81fc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.489+0000 7f820ffff640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81fc076300 0x7f81fc078810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8200001860 tx=0x7f8200001d00 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.492+0000 7f820cff9640 1 -- 172.21.15.81:0/2696621242 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f820807baf0 con 0x7f8210160840 2024-06-29T14:23:26.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.710+0000 7f82168e4640 1 -- 172.21.15.81:0/2696621242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f821015e8c0 con 0x7f8210160840 2024-06-29T14:23:26.710 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T14:23:26.712+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:23:26.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.713+0000 7f820cff9640 1 -- 172.21.15.81:0/2696621242 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f821015e8c0 con 0x7f8210160840 2024-06-29T14:23:26.712 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:23:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.715+0000 7f82168e4640 1 -- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81fc076300 msgr2=0x7f81fc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.715+0000 7f82168e4640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81fc076300 0x7f81fc078810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8200001860 tx=0x7f8200001d00 comp rx=0 tx=0).stop 2024-06-29T14:23:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.715+0000 7f82168e4640 1 -- 172.21.15.81:0/2696621242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8210160840 msgr2=0x7f8210160c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.715+0000 7f82168e4640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8210160840 0x7f8210160c50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8208009ed0 tx=0x7f8208033d70 comp rx=0 tx=0).stop 2024-06-29T14:23:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.715+0000 7f82168e4640 1 -- 172.21.15.81:0/2696621242 shutdown_connections 2024-06-29T14:23:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.715+0000 7f82168e4640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f81fc076300 0x7f81fc078810 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.715+0000 7f82168e4640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8210160840 0x7f8210160c50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.715+0000 7f82168e4640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82100b2670 0x7f821015bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.715+0000 7f82168e4640 1 --2- 172.21.15.81:0/2696621242 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81f8005150 0x7f8210150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.715+0000 7f82168e4640 1 -- 172.21.15.81:0/2696621242 >> 172.21.15.81:0/2696621242 conn(0x7f821009cf30 msgr2=0x7f82100cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.716+0000 7f82168e4640 1 -- 172.21.15.81:0/2696621242 shutdown_connections 2024-06-29T14:23:26.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.716+0000 7f82168e4640 1 -- 172.21.15.81:0/2696621242 wait complete. 2024-06-29T14:23:26.720 INFO:tasks.ceph.mds.d.smithi167.stderr: -7583> 2024-06-29T14:22:48.352+0000 7f2ec47fc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:23:26.726 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:23:26.779 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T14:23:26.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.842+0000 7f3dd60ef640 1 -- 172.21.15.81:0/2161282162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dd0072bd0 msgr2=0x7f3dd0072fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.842+0000 7f3dd60ef640 1 --2- 172.21.15.81:0/2161282162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dd0072bd0 0x7f3dd0072fe0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0009830 tx=0x7f3dc002a530 comp rx=0 tx=0).stop 2024-06-29T14:23:26.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.842+0000 7f3dd60ef640 1 -- 172.21.15.81:0/2161282162 shutdown_connections 2024-06-29T14:23:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.842+0000 7f3dd60ef640 1 --2- 172.21.15.81:0/2161282162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dd0072bd0 0x7f3dd0072fe0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.842+0000 7f3dd60ef640 1 -- 172.21.15.81:0/2161282162 >> 172.21.15.81:0/2161282162 conn(0x7f3dd00b1730 msgr2=0x7f3dd00b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.842+0000 7f3dd60ef640 1 -- 172.21.15.81:0/2161282162 shutdown_connections 2024-06-29T14:23:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.842+0000 7f3dd60ef640 1 -- 172.21.15.81:0/2161282162 wait complete. 2024-06-29T14:23:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.843+0000 7f3dd60ef640 1 Processor -- start 2024-06-29T14:23:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.844+0000 7f3dd60ef640 1 -- start start 2024-06-29T14:23:26.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dd60ef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dd0072bd0 0x7f3dd0158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dd60ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dd00b3b70 0x7f3dd01585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dd60ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dd00b4690 0x7f3dd0158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dd60ef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd0165a80 con 0x7f3dd00b3b70 2024-06-29T14:23:26.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dd60ef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd0165900 con 0x7f3dd0072bd0 2024-06-29T14:23:26.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dd60ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3dd0165c00 con 0x7f3dd00b4690 2024-06-29T14:23:26.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dceffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dd00b3b70 0x7f3dd01585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dceffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dd00b3b70 0x7f3dd01585a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60082/0 (socket says 172.21.15.81:60082) 2024-06-29T14:23:26.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dceffd640 1 -- 172.21.15.81:0/3986805039 learned_addr learned my addr 172.21.15.81:0/3986805039 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:26.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dcffff640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dd00b4690 0x7f3dd0158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dcf7fe640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dd0072bd0 0x7f3dd0158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dceffd640 1 -- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dd00b4690 msgr2=0x7f3dd0158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dceffd640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dd00b4690 0x7f3dd0158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dceffd640 1 -- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dd0072bd0 msgr2=0x7f3dd0158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dceffd640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dd0072bd0 0x7f3dd0158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dceffd640 1 -- 172.21.15.81:0/3986805039 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd0153be0 con 0x7f3dd00b3b70 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.845+0000 7f3dcf7fe640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dd0072bd0 0x7f3dd0158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.846+0000 7f3dceffd640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dd00b3b70 0x7f3dd01585a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400c370 tx=0x7f3dc400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.846+0000 7f3db7fff640 1 -- 172.21.15.81:0/3986805039 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4017070 con 0x7f3dd00b3b70 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.846+0000 7f3db7fff640 1 -- 172.21.15.81:0/3986805039 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dc40053c0 con 0x7f3dd00b3b70 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.846+0000 7f3db7fff640 1 -- 172.21.15.81:0/3986805039 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4003850 con 0x7f3dd00b3b70 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.846+0000 7f3dd60ef640 1 -- 172.21.15.81:0/3986805039 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd01540e0 con 0x7f3dd00b3b70 2024-06-29T14:23:26.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.846+0000 7f3dd60ef640 1 -- 172.21.15.81:0/3986805039 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd01546a0 con 0x7f3dd00b3b70 2024-06-29T14:23:26.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.847+0000 7f3dd60ef640 1 -- 172.21.15.81:0/3986805039 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dd01593a0 con 0x7f3dd00b3b70 2024-06-29T14:23:26.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.847+0000 7f3db7fff640 1 -- 172.21.15.81:0/3986805039 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3dc401a070 con 0x7f3dd00b3b70 2024-06-29T14:23:26.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.848+0000 7f3db7fff640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3dbc0762d0 0x7f3dbc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:26.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.848+0000 7f3db7fff640 1 -- 172.21.15.81:0/3986805039 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5890+0+0 (secure 0 0 0) 0x7f3dc4096c60 con 0x7f3dd00b3b70 2024-06-29T14:23:26.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.848+0000 7f3dcf7fe640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3dbc0762d0 0x7f3dbc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:26.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.849+0000 7f3dcf7fe640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3dbc0762d0 0x7f3dbc0787e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0034000 tx=0x7f3dc0004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:26.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:26.853+0000 7f3db7fff640 1 -- 172.21.15.81:0/3986805039 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3dd01593a0 con 0x7f3dd00b3b70 2024-06-29T14:23:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.073+0000 7f3dd60ef640 1 -- 172.21.15.81:0/3986805039 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f3dd0072fe0 con 0x7f3dd00b3b70 2024-06-29T14:23:27.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.720+0000 7f3db7fff640 1 -- 172.21.15.81:0/3986805039 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7f3dc4060480 con 0x7f3dd00b3b70 2024-06-29T14:23:27.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 -- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3dbc0762d0 msgr2=0x7f3dbc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:27.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3dbc0762d0 0x7f3dbc0787e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0034000 tx=0x7f3dc0004730 comp rx=0 tx=0).stop 2024-06-29T14:23:27.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 -- 172.21.15.81:0/3986805039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dd00b3b70 msgr2=0x7f3dd01585a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:27.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dd00b3b70 0x7f3dd01585a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400c370 tx=0x7f3dc400c830 comp rx=0 tx=0).stop 2024-06-29T14:23:27.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 -- 172.21.15.81:0/3986805039 shutdown_connections 2024-06-29T14:23:27.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3dbc0762d0 0x7f3dbc0787e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dd00b4690 0x7f3dd0158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dd00b3b70 0x7f3dd01585a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 --2- 172.21.15.81:0/3986805039 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dd0072bd0 0x7f3dd0158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 -- 172.21.15.81:0/3986805039 >> 172.21.15.81:0/3986805039 conn(0x7f3dd00b1730 msgr2=0x7f3dd015d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:27.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 -- 172.21.15.81:0/3986805039 shutdown_connections 2024-06-29T14:23:27.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.723+0000 7f3dd60ef640 1 -- 172.21.15.81:0/3986805039 wait complete. 2024-06-29T14:23:27.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:23:27.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.864+0000 7f25aff93640 1 -- 172.21.15.81:0/4094016818 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f25a8151090 msgr2=0x7f25a8073ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:27.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.864+0000 7f25aff93640 1 --2- 172.21.15.81:0/4094016818 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f25a8151090 0x7f25a8073ad0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2598009830 tx=0x7f259802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.865+0000 7f25aff93640 1 -- 172.21.15.81:0/4094016818 shutdown_connections 2024-06-29T14:23:27.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.865+0000 7f25aff93640 1 --2- 172.21.15.81:0/4094016818 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f25a8151090 0x7f25a8073ad0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.865+0000 7f25aff93640 1 --2- 172.21.15.81:0/4094016818 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f25a80cf100 0x7f25a80b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.865+0000 7f25aff93640 1 -- 172.21.15.81:0/4094016818 >> 172.21.15.81:0/4094016818 conn(0x7f25a806cd60 msgr2=0x7f25a80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:27.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.865+0000 7f25aff93640 1 -- 172.21.15.81:0/4094016818 shutdown_connections 2024-06-29T14:23:27.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.865+0000 7f25aff93640 1 -- 172.21.15.81:0/4094016818 wait complete. 2024-06-29T14:23:27.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.866+0000 7f25aff93640 1 Processor -- start 2024-06-29T14:23:27.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.866+0000 7f25aff93640 1 -- start start 2024-06-29T14:23:27.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25aff93640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f25a80cf100 0x7f25a814fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:27.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25aff93640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a8151090 0x7f25a8146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:27.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25aff93640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f25a815dde0 0x7f25a8166180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:27.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25aff93640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f25a8070010 con 0x7f25a8151090 2024-06-29T14:23:27.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25aff93640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f25a8074220 con 0x7f25a815dde0 2024-06-29T14:23:27.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25aff93640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f25a8070190 con 0x7f25a80cf100 2024-06-29T14:23:27.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25ad52e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a8151090 0x7f25a8146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:27.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25ad52e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a8151090 0x7f25a8146980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60092/0 (socket says 172.21.15.81:60092) 2024-06-29T14:23:27.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25ad52e640 1 -- 172.21.15.81:0/1127678377 learned_addr learned my addr 172.21.15.81:0/1127678377 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:27.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25ae530640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f25a815dde0 0x7f25a8166180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:27.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25add2f640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f25a80cf100 0x7f25a814fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:27.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25ad52e640 1 -- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f25a80cf100 msgr2=0x7f25a814fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:27.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25ad52e640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f25a80cf100 0x7f25a814fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25ad52e640 1 -- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f25a815dde0 msgr2=0x7f25a8166180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:27.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25ad52e640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f25a815dde0 0x7f25a8166180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:27.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25ad52e640 1 -- 172.21.15.81:0/1127678377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a8146ec0 con 0x7f25a8151090 2024-06-29T14:23:27.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.868+0000 7f25ad52e640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a8151090 0x7f25a8146980 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f259802aa90 tx=0x7f2598030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:27.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.869+0000 7f259e7fc640 1 -- 172.21.15.81:0/1127678377 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259803f070 con 0x7f25a8151090 2024-06-29T14:23:27.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.869+0000 7f259e7fc640 1 -- 172.21.15.81:0/1127678377 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2598030d40 con 0x7f25a8151090 2024-06-29T14:23:27.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.869+0000 7f25aff93640 1 -- 172.21.15.81:0/1127678377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a81473d0 con 0x7f25a8151090 2024-06-29T14:23:27.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.869+0000 7f259e7fc640 1 -- 172.21.15.81:0/1127678377 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2598004280 con 0x7f25a8151090 2024-06-29T14:23:27.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.869+0000 7f25aff93640 1 -- 172.21.15.81:0/1127678377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a8147990 con 0x7f25a8151090 2024-06-29T14:23:27.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.870+0000 7f25aff93640 1 -- 172.21.15.81:0/1127678377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a8148470 con 0x7f25a8151090 2024-06-29T14:23:27.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.870+0000 7f259e7fc640 1 -- 172.21.15.81:0/1127678377 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f25980035f0 con 0x7f25a8151090 2024-06-29T14:23:27.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.871+0000 7f259e7fc640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f258c076300 0x7f258c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:27.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.871+0000 7f259e7fc640 1 -- 172.21.15.81:0/1127678377 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5890+0+0 (secure 0 0 0) 0x7f25980b8140 con 0x7f25a8151090 2024-06-29T14:23:27.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.871+0000 7f25add2f640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f258c076300 0x7f258c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:27.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.872+0000 7f25add2f640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f258c076300 0x7f258c078810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f25a40099a0 tx=0x7f25a4009500 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:27.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:27.875+0000 7f259e7fc640 1 -- 172.21.15.81:0/1127678377 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f25a8148470 con 0x7f25a8151090 2024-06-29T14:23:28.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.058+0000 7f25aff93640 1 -- 172.21.15.81:0/1127678377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f25a80b2a30 con 0x7f25a8151090 2024-06-29T14:23:28.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.864+0000 7f259e7fc640 1 -- 172.21.15.81:0/1127678377 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) ==== 181+0+0 (secure 0 0 0) 0x7f2598081280 con 0x7f25a8151090 2024-06-29T14:23:28.863 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:23:28.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.867+0000 7f25aff93640 1 -- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f258c076300 msgr2=0x7f258c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:28.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.867+0000 7f25aff93640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f258c076300 0x7f258c078810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f25a40099a0 tx=0x7f25a4009500 comp rx=0 tx=0).stop 2024-06-29T14:23:28.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.867+0000 7f25aff93640 1 -- 172.21.15.81:0/1127678377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a8151090 msgr2=0x7f25a8146980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:28.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.867+0000 7f25aff93640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a8151090 0x7f25a8146980 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f259802aa90 tx=0x7f2598030650 comp rx=0 tx=0).stop 2024-06-29T14:23:28.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.868+0000 7f25aff93640 1 -- 172.21.15.81:0/1127678377 shutdown_connections 2024-06-29T14:23:28.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.868+0000 7f25aff93640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f258c076300 0x7f258c078810 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.868+0000 7f25aff93640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f25a815dde0 0x7f25a8166180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.868+0000 7f25aff93640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a8151090 0x7f25a8146980 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.868+0000 7f25aff93640 1 --2- 172.21.15.81:0/1127678377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f25a80cf100 0x7f25a814fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:28.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.868+0000 7f25aff93640 1 -- 172.21.15.81:0/1127678377 >> 172.21.15.81:0/1127678377 conn(0x7f25a806cd60 msgr2=0x7f25a80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:28.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.868+0000 7f25aff93640 1 -- 172.21.15.81:0/1127678377 shutdown_connections 2024-06-29T14:23:28.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:28.868+0000 7f25aff93640 1 -- 172.21.15.81:0/1127678377 wait complete. 2024-06-29T14:23:28.881 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:23:29.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.006+0000 7f4cf2cce640 1 -- 172.21.15.81:0/3624560881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cd4003a50 msgr2=0x7f4cd4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.006+0000 7f4cf2cce640 1 --2- 172.21.15.81:0/3624560881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cd4003a50 0x7f4cd4003ec0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80016c0 tx=0x7f4cd4005830 comp rx=0 tx=0).stop 2024-06-29T14:23:29.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.006+0000 7f4cf2cce640 1 -- 172.21.15.81:0/3624560881 shutdown_connections 2024-06-29T14:23:29.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.006+0000 7f4cf2cce640 1 --2- 172.21.15.81:0/3624560881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cd4003a50 0x7f4cd4003ec0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.006+0000 7f4cf2cce640 1 -- 172.21.15.81:0/3624560881 >> 172.21.15.81:0/3624560881 conn(0x7f4cec0b35a0 msgr2=0x7f4cec0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:29.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.006+0000 7f4cf2cce640 1 -- 172.21.15.81:0/3624560881 shutdown_connections 2024-06-29T14:23:29.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.006+0000 7f4cf2cce640 1 -- 172.21.15.81:0/3624560881 wait complete. 2024-06-29T14:23:29.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.008+0000 7f4cf2cce640 1 Processor -- start 2024-06-29T14:23:29.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.008+0000 7f4cf2cce640 1 -- start start 2024-06-29T14:23:29.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf2cce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cd4003a50 0x7f4cec1e17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf2cce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4cec0ca780 0x7f4cec0c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf2cce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cec1e3300 0x7f4cec1e3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf2cce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4cec0cc9b0 con 0x7f4cd4003a50 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf2cce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4cec0cc830 con 0x7f4cec1e3300 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf2cce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4cec0cff60 con 0x7f4cec0ca780 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf0a6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cd4003a50 0x7f4cec1e17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf0a6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cd4003a50 0x7f4cec1e17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60102/0 (socket says 172.21.15.81:60102) 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf0a6a640 1 -- 172.21.15.81:0/1605169436 learned_addr learned my addr 172.21.15.81:0/1605169436 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf126b640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cec1e3300 0x7f4cec1e3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4ce3fff640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4cec0ca780 0x7f4cec0c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf0a6a640 1 -- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4cec0ca780 msgr2=0x7f4cec0c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf0a6a640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4cec0ca780 0x7f4cec0c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf0a6a640 1 -- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cec1e3300 msgr2=0x7f4cec1e3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.009+0000 7f4cf0a6a640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cec1e3300 0x7f4cec1e3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.010+0000 7f4cf0a6a640 1 -- 172.21.15.81:0/1605169436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cec1e70e0 con 0x7f4cd4003a50 2024-06-29T14:23:29.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.010+0000 7f4cf0a6a640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cd4003a50 0x7f4cec1e17d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4ce802a910 tx=0x7f4ce80063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:29.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.010+0000 7f4ce17fa640 1 -- 172.21.15.81:0/1605169436 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8003a00 con 0x7f4cd4003a50 2024-06-29T14:23:29.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.010+0000 7f4ce17fa640 1 -- 172.21.15.81:0/1605169436 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ce8039070 con 0x7f4cd4003a50 2024-06-29T14:23:29.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.010+0000 7f4ce17fa640 1 -- 172.21.15.81:0/1605169436 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8033420 con 0x7f4cd4003a50 2024-06-29T14:23:29.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.010+0000 7f4cf2cce640 1 -- 172.21.15.81:0/1605169436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cec1e3c50 con 0x7f4cd4003a50 2024-06-29T14:23:29.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.010+0000 7f4cf2cce640 1 -- 172.21.15.81:0/1605169436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cec0c96e0 con 0x7f4cd4003a50 2024-06-29T14:23:29.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.011+0000 7f4cf2cce640 1 -- 172.21.15.81:0/1605169436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cec075770 con 0x7f4cd4003a50 2024-06-29T14:23:29.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.012+0000 7f4ce17fa640 1 -- 172.21.15.81:0/1605169436 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4ce8044020 con 0x7f4cd4003a50 2024-06-29T14:23:29.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.012+0000 7f4ce17fa640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4cd80762d0 0x7f4cd80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:29.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.012+0000 7f4ce17fa640 1 -- 172.21.15.81:0/1605169436 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5452+0+0 (secure 0 0 0) 0x7f4ce80b7040 con 0x7f4cd4003a50 2024-06-29T14:23:29.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.012+0000 7f4ce3fff640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4cd80762d0 0x7f4cd80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:29.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.013+0000 7f4ce3fff640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4cd80762d0 0x7f4cd80787e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc005eb0 tx=0x7f4cdc005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:29.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.017+0000 7f4ce17fa640 1 -- 172.21.15.81:0/1605169436 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4cec075770 con 0x7f4cd4003a50 2024-06-29T14:23:29.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.199+0000 7f4cf2cce640 1 -- 172.21.15.81:0/1605169436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4cec0d12a0 con 0x7f4cd4003a50 2024-06-29T14:23:29.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.866+0000 7f4ce17fa640 1 -- 172.21.15.81:0/1605169436 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) ==== 169+0+0 (secure 0 0 0) 0x7f4cec0d12a0 con 0x7f4cd4003a50 2024-06-29T14:23:29.865 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T14:23:29.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 -- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4cd80762d0 msgr2=0x7f4cd80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4cd80762d0 0x7f4cd80787e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc005eb0 tx=0x7f4cdc005f90 comp rx=0 tx=0).stop 2024-06-29T14:23:29.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 -- 172.21.15.81:0/1605169436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cd4003a50 msgr2=0x7f4cec1e17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:29.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cd4003a50 0x7f4cec1e17d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4ce802a910 tx=0x7f4ce80063a0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 -- 172.21.15.81:0/1605169436 shutdown_connections 2024-06-29T14:23:29.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4cd80762d0 0x7f4cd80787e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4cec1e3300 0x7f4cec1e3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4cec0ca780 0x7f4cec0c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 --2- 172.21.15.81:0/1605169436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cd4003a50 0x7f4cec1e17d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:29.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 -- 172.21.15.81:0/1605169436 >> 172.21.15.81:0/1605169436 conn(0x7f4cec0b35a0 msgr2=0x7f4cec0b4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:29.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 -- 172.21.15.81:0/1605169436 shutdown_connections 2024-06-29T14:23:29.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:29.870+0000 7f4cf2cce640 1 -- 172.21.15.81:0/1605169436 wait complete. 2024-06-29T14:23:29.882 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:23:29.882 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:23:29.882 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:23:29.882 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:23:29.885 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:23:29.885 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:23:29.885 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:23:29.885 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:23:29.886 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:23:29.886 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:23:29.886 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:23:29.886 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:23:29.886 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:23:29.886 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:23:29.886 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:23:29.887 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:23:29.892 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:23:29.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:23:29.895 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:23:29.895 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:23:29.898 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:23:29.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:23:29.902 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:23:29.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:23:29.937 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:23:29.937 DEBUG:teuthology.parallel:result is None 2024-06-29T14:23:29.938 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:23:29.938 DEBUG:teuthology.parallel:result is None 2024-06-29T14:23:29.944 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:23:29.944 DEBUG:teuthology.parallel:result is None 2024-06-29T14:23:29.945 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:23:29.945 DEBUG:teuthology.parallel:result is None 2024-06-29T14:23:29.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:23:30.024 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:23:30.024 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T14:23:30.035 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:23:30.035 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T14:23:30.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.067+0000 7f7cc0e38640 1 -- 172.21.15.81:0/3745125613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cbc08ccd0 msgr2=0x7f7cbc0b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:30.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.067+0000 7f7cc0e38640 1 --2- 172.21.15.81:0/3745125613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cbc08ccd0 0x7f7cbc0b3240 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009830 tx=0x7f7ca402a3f0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.067+0000 7f7cc0e38640 1 -- 172.21.15.81:0/3745125613 shutdown_connections 2024-06-29T14:23:30.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.067+0000 7f7cc0e38640 1 --2- 172.21.15.81:0/3745125613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cbc08ccd0 0x7f7cbc0b3240 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.067+0000 7f7cc0e38640 1 -- 172.21.15.81:0/3745125613 >> 172.21.15.81:0/3745125613 conn(0x7f7cbc0b56d0 msgr2=0x7f7cbc0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.067+0000 7f7cc0e38640 1 -- 172.21.15.81:0/3745125613 shutdown_connections 2024-06-29T14:23:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.067+0000 7f7cc0e38640 1 -- 172.21.15.81:0/3745125613 wait complete. 2024-06-29T14:23:30.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.069+0000 7f7cc0e38640 1 Processor -- start 2024-06-29T14:23:30.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.069+0000 7f7cc0e38640 1 -- start start 2024-06-29T14:23:30.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cc0e38640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7cbc08ccd0 0x7f7cbc15f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cc0e38640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7cbc0cf040 0x7f7cbc14c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cc0e38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cbc169210 0x7f7cbc14c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cc0e38640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc0b4410 con 0x7f7cbc169210 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cc0e38640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc0b4290 con 0x7f7cbc08ccd0 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cc0e38640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7cbc0b4590 con 0x7f7cbc0cf040 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cbad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cbc169210 0x7f7cbc14c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cbad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cbc169210 0x7f7cbc14c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60554/0 (socket says 172.21.15.81:60554) 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cbad9d640 1 -- 172.21.15.81:0/2443098710 learned_addr learned my addr 172.21.15.81:0/2443098710 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cb1d9b640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7cbc0cf040 0x7f7cbc14c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.070+0000 7f7cba59c640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7cbc08ccd0 0x7f7cbc15f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cbad9d640 1 -- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7cbc0cf040 msgr2=0x7f7cbc14c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cbad9d640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7cbc0cf040 0x7f7cbc14c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cbad9d640 1 -- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7cbc08ccd0 msgr2=0x7f7cbc15f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cbad9d640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7cbc08ccd0 0x7f7cbc15f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cbad9d640 1 -- 172.21.15.81:0/2443098710 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cbc14b240 con 0x7f7cbc169210 2024-06-29T14:23:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cbad9d640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cbc169210 0x7f7cbc14c820 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400ba10 tx=0x7f7cb400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cb37fe640 1 -- 172.21.15.81:0/2443098710 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4019070 con 0x7f7cbc169210 2024-06-29T14:23:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cb37fe640 1 -- 172.21.15.81:0/2443098710 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cb400cbd0 con 0x7f7cbc169210 2024-06-29T14:23:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cc0e38640 1 -- 172.21.15.81:0/2443098710 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cbc14b520 con 0x7f7cbc169210 2024-06-29T14:23:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cb37fe640 1 -- 172.21.15.81:0/2443098710 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4014440 con 0x7f7cbc169210 2024-06-29T14:23:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.071+0000 7f7cc0e38640 1 -- 172.21.15.81:0/2443098710 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cbc14bae0 con 0x7f7cbc169210 2024-06-29T14:23:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.072+0000 7f7cc0e38640 1 -- 172.21.15.81:0/2443098710 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cbc08c270 con 0x7f7cbc169210 2024-06-29T14:23:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.073+0000 7f7cb37fe640 1 -- 172.21.15.81:0/2443098710 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7cb401c070 con 0x7f7cbc169210 2024-06-29T14:23:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.073+0000 7f7cb37fe640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7c9c0762d0 0x7f7c9c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.073+0000 7f7cb37fe640 1 -- 172.21.15.81:0/2443098710 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 4943+0+0 (secure 0 0 0) 0x7f7cb4097100 con 0x7f7cbc169210 2024-06-29T14:23:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.073+0000 7f7cba59c640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7c9c0762d0 0x7f7c9c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:30.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.074+0000 7f7cba59c640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7c9c0762d0 0x7f7c9c0787e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7cac001720 tx=0x7f7cac007e90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:30.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.077+0000 7f7cb37fe640 1 -- 172.21.15.81:0/2443098710 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7cb4060690 con 0x7f7cbc169210 2024-06-29T14:23:30.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.271+0000 7f7cc0e38640 1 -- 172.21.15.81:0/2443098710 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f7cbc15f680 con 0x7f7cbc169210 2024-06-29T14:23:30.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.890+0000 7f7cb37fe640 1 -- 172.21.15.81:0/2443098710 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) ==== 97+0+0 (secure 0 0 0) 0x7f7cb4064990 con 0x7f7cbc169210 2024-06-29T14:23:30.889 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T14:23:30.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 -- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7c9c0762d0 msgr2=0x7f7c9c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:30.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7c9c0762d0 0x7f7c9c0787e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7cac001720 tx=0x7f7cac007e90 comp rx=0 tx=0).stop 2024-06-29T14:23:30.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 -- 172.21.15.81:0/2443098710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cbc169210 msgr2=0x7f7cbc14c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:30.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cbc169210 0x7f7cbc14c820 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7cb400ba10 tx=0x7f7cb400bed0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 -- 172.21.15.81:0/2443098710 shutdown_connections 2024-06-29T14:23:30.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f7c9c0762d0 0x7f7c9c0787e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cbc169210 0x7f7cbc14c820 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7cbc0cf040 0x7f7cbc14c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 --2- 172.21.15.81:0/2443098710 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7cbc08ccd0 0x7f7cbc15f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:30.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 -- 172.21.15.81:0/2443098710 >> 172.21.15.81:0/2443098710 conn(0x7f7cbc0b56d0 msgr2=0x7f7cbc06d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:30.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 -- 172.21.15.81:0/2443098710 shutdown_connections 2024-06-29T14:23:30.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:30.893+0000 7f7cc0e38640 1 -- 172.21.15.81:0/2443098710 wait complete. 2024-06-29T14:23:30.909 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:23:31.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.028+0000 7f5e5a743640 1 -- 172.21.15.81:0/2401517561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e3c003fd0 msgr2=0x7f5e3c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.028+0000 7f5e5a743640 1 --2- 172.21.15.81:0/2401517561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e3c003fd0 0x7f5e3c004800 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009b30 tx=0x7f5e3c005830 comp rx=0 tx=0).stop 2024-06-29T14:23:31.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.028+0000 7f5e58ce0640 1 -- 172.21.15.81:0/2401517561 reap_dead start 2024-06-29T14:23:31.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.028+0000 7f5e5a743640 1 -- 172.21.15.81:0/2401517561 shutdown_connections 2024-06-29T14:23:31.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.028+0000 7f5e5a743640 1 -- 172.21.15.81:0/2401517561 >> 172.21.15.81:0/2401517561 conn(0x7f5e5406ccc0 msgr2=0x7f5e540b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:31.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.028+0000 7f5e5a743640 1 -- 172.21.15.81:0/2401517561 shutdown_connections 2024-06-29T14:23:31.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.028+0000 7f5e5a743640 1 -- 172.21.15.81:0/2401517561 wait complete. 2024-06-29T14:23:31.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.029+0000 7f5e5a743640 1 Processor -- start 2024-06-29T14:23:31.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.030+0000 7f5e5a743640 1 -- start start 2024-06-29T14:23:31.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e5a743640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e3c003fd0 0x7f5e54140150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e5a743640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e540a9780 0x7f5e54157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e5a743640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e540a9b90 0x7f5e541f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e5a743640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5415ed70 con 0x7f5e3c003fd0 2024-06-29T14:23:31.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e5a743640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5e54141700 con 0x7f5e540a9b90 2024-06-29T14:23:31.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e5a743640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5e5415eef0 con 0x7f5e540a9780 2024-06-29T14:23:31.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e53fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e3c003fd0 0x7f5e54140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e53fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e3c003fd0 0x7f5e54140150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60580/0 (socket says 172.21.15.81:60580) 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e53fff640 1 -- 172.21.15.81:0/2367290487 learned_addr learned my addr 172.21.15.81:0/2367290487 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e58ce0640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e540a9b90 0x7f5e541f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e537fe640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e540a9780 0x7f5e54157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e53fff640 1 -- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e540a9780 msgr2=0x7f5e54157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e53fff640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e540a9780 0x7f5e54157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e53fff640 1 -- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e540a9b90 msgr2=0x7f5e541f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e53fff640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e540a9b90 0x7f5e541f1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e53fff640 1 -- 172.21.15.81:0/2367290487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e541564f0 con 0x7f5e3c003fd0 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e537fe640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e540a9780 0x7f5e54157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.031+0000 7f5e53fff640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e3c003fd0 0x7f5e54140150 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5e44006e20 tx=0x7f5e440049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.032+0000 7f5e50ff9640 1 -- 172.21.15.81:0/2367290487 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400d730 con 0x7f5e3c003fd0 2024-06-29T14:23:31.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.032+0000 7f5e50ff9640 1 -- 172.21.15.81:0/2367290487 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e4400d8f0 con 0x7f5e3c003fd0 2024-06-29T14:23:31.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.032+0000 7f5e5a743640 1 -- 172.21.15.81:0/2367290487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e541567d0 con 0x7f5e3c003fd0 2024-06-29T14:23:31.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.032+0000 7f5e50ff9640 1 -- 172.21.15.81:0/2367290487 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400dd40 con 0x7f5e3c003fd0 2024-06-29T14:23:31.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.032+0000 7f5e5a743640 1 -- 172.21.15.81:0/2367290487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e54156d90 con 0x7f5e3c003fd0 2024-06-29T14:23:31.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.033+0000 7f5e5a743640 1 -- 172.21.15.81:0/2367290487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e54073440 con 0x7f5e3c003fd0 2024-06-29T14:23:31.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.033+0000 7f5e50ff9640 1 -- 172.21.15.81:0/2367290487 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5e44016030 con 0x7f5e3c003fd0 2024-06-29T14:23:31.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.034+0000 7f5e50ff9640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e40076300 0x7f5e40078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.034+0000 7f5e50ff9640 1 -- 172.21.15.81:0/2367290487 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f5e44098b30 con 0x7f5e3c003fd0 2024-06-29T14:23:31.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.034+0000 7f5e537fe640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e40076300 0x7f5e40078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.034+0000 7f5e537fe640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e40076300 0x7f5e40078810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e48034000 tx=0x7f5e480046c0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:31.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.038+0000 7f5e50ff9640 1 -- 172.21.15.81:0/2367290487 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5e54073980 con 0x7f5e3c003fd0 2024-06-29T14:23:31.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.286+0000 7f5e5a743640 1 -- 172.21.15.81:0/2367290487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f5e54140690 con 0x7f5e3c003fd0 2024-06-29T14:23:31.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.286+0000 7f5e50ff9640 1 -- 172.21.15.81:0/2367290487 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) ==== 80+0+6645 (secure 0 0 0) 0x7f5e44062220 con 0x7f5e3c003fd0 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:23:31.286 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.287 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.288 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T14:23:31.290 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.291 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.292 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.293 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:23:31.294 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:23:31.295 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.296 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:23:31.297 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.bootstrap-mds", 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCaGIBmK+djDhAAdWp9qsUPqw/+1AB8dOzZ+A==", 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile bootstrap-mds" 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.bootstrap-mgr", 2024-06-29T14:23:31.298 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCaGIBmJPBjDhAACKBb/ZlbFhJmvmawfXxFnA==", 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile bootstrap-mgr" 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.bootstrap-osd", 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCaGIBmVPljDhAAT10fU+vPFnBjj+k5Em5aMg==", 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile bootstrap-osd" 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.299 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.bootstrap-rbd", 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCaGIBmXgFkDhAAZ+sneeWjeRSyM/R5+IxqUQ==", 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile bootstrap-rbd" 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCaGIBm3ghkDhAAbFk7yBs3gEbu5CUm7MfKbA==", 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.300 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.bootstrap-rgw", 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCaGIBmBhBkDhAAVM0r4rKme+WMAdpQnXSwTA==", 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile bootstrap-rgw" 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.301 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T14:23:31.302 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.288+0000 7f5e5a743640 1 -- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e40076300 msgr2=0x7f5e40078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.288+0000 7f5e5a743640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e40076300 0x7f5e40078810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e48034000 tx=0x7f5e480046c0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.288+0000 7f5e5a743640 1 -- 172.21.15.81:0/2367290487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e3c003fd0 msgr2=0x7f5e54140150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.288+0000 7f5e5a743640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e3c003fd0 0x7f5e54140150 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5e44006e20 tx=0x7f5e440049d0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.288+0000 7f5e5a743640 1 -- 172.21.15.81:0/2367290487 shutdown_connections 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.288+0000 7f5e5a743640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f5e40076300 0x7f5e40078810 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.288+0000 7f5e5a743640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e540a9b90 0x7f5e541f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.288+0000 7f5e5a743640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e540a9780 0x7f5e54157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.288+0000 7f5e5a743640 1 --2- 172.21.15.81:0/2367290487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e3c003fd0 0x7f5e54140150 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.288+0000 7f5e5a743640 1 -- 172.21.15.81:0/2367290487 >> 172.21.15.81:0/2367290487 conn(0x7f5e5406ccc0 msgr2=0x7f5e5415a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.289+0000 7f5e5a743640 1 -- 172.21.15.81:0/2367290487 shutdown_connections 2024-06-29T14:23:31.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.289+0000 7f5e5a743640 1 -- 172.21.15.81:0/2367290487 wait complete. 2024-06-29T14:23:31.305 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-06-29T14:23:31.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.420+0000 7fe0ba43f640 1 -- 172.21.15.81:0/2468831259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0b406ef90 msgr2=0x7fe0b406f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.420+0000 7fe0ba43f640 1 --2- 172.21.15.81:0/2468831259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0b406ef90 0x7fe0b406f3a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe0b40b53d0 tx=0x7fe0a802a540 comp rx=0 tx=0).stop 2024-06-29T14:23:31.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.421+0000 7fe0ba43f640 1 -- 172.21.15.81:0/2468831259 shutdown_connections 2024-06-29T14:23:31.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.421+0000 7fe0ba43f640 1 --2- 172.21.15.81:0/2468831259 >> v2:172.21.15.167:3300/0 conn(0x7fe0b40ce1d0 0x7fe0b40ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.421+0000 7fe0ba43f640 1 --2- 172.21.15.81:0/2468831259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0b406ef90 0x7fe0b406f3a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.421+0000 7fe0ba43f640 1 --2- 172.21.15.81:0/2468831259 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0b40b57b0 0x7fe0b415aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.421+0000 7fe0ba43f640 1 -- 172.21.15.81:0/2468831259 >> 172.21.15.81:0/2468831259 conn(0x7fe0b40d0930 msgr2=0x7fe0b40d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:31.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.421+0000 7fe0ba43f640 1 -- 172.21.15.81:0/2468831259 shutdown_connections 2024-06-29T14:23:31.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.421+0000 7fe0ba43f640 1 -- 172.21.15.81:0/2468831259 wait complete. 2024-06-29T14:23:31.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.422+0000 7fe0ba43f640 1 Processor -- start 2024-06-29T14:23:31.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.423+0000 7fe0ba43f640 1 -- start start 2024-06-29T14:23:31.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0ba43f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0b406ef90 0x7fe0b40ab410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0ba43f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0b40b57b0 0x7fe0b40ab950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0ba43f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0b40ce1d0 0x7fe0b4150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0ba43f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b40ceeb0 con 0x7fe0b40b57b0 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0ba43f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b40ced30 con 0x7fe0b406ef90 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0ba43f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b40cf030 con 0x7fe0b40ce1d0 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0b37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0b40b57b0 0x7fe0b40ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0b37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0b40b57b0 0x7fe0b40ab950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60608/0 (socket says 172.21.15.81:60608) 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0b37fe640 1 -- 172.21.15.81:0/790773481 learned_addr learned my addr 172.21.15.81:0/790773481 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0b3fff640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0b406ef90 0x7fe0b40ab410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0b89dc640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0b40ce1d0 0x7fe0b4150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0b37fe640 1 -- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0b40ce1d0 msgr2=0x7fe0b4150890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0b37fe640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0b40ce1d0 0x7fe0b4150890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0b37fe640 1 -- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0b406ef90 msgr2=0x7fe0b40ab410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0b37fe640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0b406ef90 0x7fe0b40ab410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.424+0000 7fe0b37fe640 1 -- 172.21.15.81:0/790773481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b40a9a60 con 0x7fe0b40b57b0 2024-06-29T14:23:31.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.425+0000 7fe0b37fe640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0b40b57b0 0x7fe0b40ab950 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8034040 tx=0x7fe0a8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:31.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.425+0000 7fe0b0ff9640 1 -- 172.21.15.81:0/790773481 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a803e030 con 0x7fe0b40b57b0 2024-06-29T14:23:31.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.425+0000 7fe0b0ff9640 1 -- 172.21.15.81:0/790773481 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0a8003e00 con 0x7fe0b40b57b0 2024-06-29T14:23:31.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.425+0000 7fe0ba43f640 1 -- 172.21.15.81:0/790773481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b40a9d40 con 0x7fe0b40b57b0 2024-06-29T14:23:31.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.425+0000 7fe0b0ff9640 1 -- 172.21.15.81:0/790773481 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8035040 con 0x7fe0b40b57b0 2024-06-29T14:23:31.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.425+0000 7fe0ba43f640 1 -- 172.21.15.81:0/790773481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b40aa300 con 0x7fe0b40b57b0 2024-06-29T14:23:31.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.426+0000 7fe0ba43f640 1 -- 172.21.15.81:0/790773481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0b406e2a0 con 0x7fe0b40b57b0 2024-06-29T14:23:31.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.426+0000 7fe0b0ff9640 1 -- 172.21.15.81:0/790773481 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe0a8041030 con 0x7fe0b40b57b0 2024-06-29T14:23:31.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.427+0000 7fe0b0ff9640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0a00762d0 0x7fe0a00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.427+0000 7fe0b0ff9640 1 -- 172.21.15.81:0/790773481 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fe0a80b7c10 con 0x7fe0b40b57b0 2024-06-29T14:23:31.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.427+0000 7fe0b3fff640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0a00762d0 0x7fe0a00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.428+0000 7fe0b3fff640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0a00762d0 0x7fe0a00787e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40058c0 tx=0x7fe0a4005850 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:31.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.431+0000 7fe0b0ff9640 1 -- 172.21.15.81:0/790773481 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe0b406e2a0 con 0x7fe0b40b57b0 2024-06-29T14:23:31.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.678+0000 7fe0ba43f640 1 -- 172.21.15.81:0/790773481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7fe0b40a97f0 con 0x7fe0b40b57b0 2024-06-29T14:23:31.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.681+0000 7fe0b0ff9640 1 -- 172.21.15.81:0/790773481 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v21) ==== 90+0+0 (secure 0 0 0) 0x7fe0a8081830 con 0x7fe0b40b57b0 2024-06-29T14:23:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 -- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0a00762d0 msgr2=0x7fe0a00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0a00762d0 0x7fe0a00787e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40058c0 tx=0x7fe0a4005850 comp rx=0 tx=0).stop 2024-06-29T14:23:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 -- 172.21.15.81:0/790773481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0b40b57b0 msgr2=0x7fe0b40ab950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0b40b57b0 0x7fe0b40ab950 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8034040 tx=0x7fe0a8003910 comp rx=0 tx=0).stop 2024-06-29T14:23:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 -- 172.21.15.81:0/790773481 shutdown_connections 2024-06-29T14:23:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fe0a00762d0 0x7fe0a00787e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0b40ce1d0 0x7fe0b4150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0b40b57b0 0x7fe0b40ab950 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 --2- 172.21.15.81:0/790773481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0b406ef90 0x7fe0b40ab410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 -- 172.21.15.81:0/790773481 >> 172.21.15.81:0/790773481 conn(0x7fe0b40d0930 msgr2=0x7fe0b40b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:31.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 -- 172.21.15.81:0/790773481 shutdown_connections 2024-06-29T14:23:31.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.683+0000 7fe0ba43f640 1 -- 172.21.15.81:0/790773481 wait complete. 2024-06-29T14:23:31.693 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-06-29T14:23:31.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.808+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/2684217127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e4808ccd0 msgr2=0x7f0e480cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.808+0000 7f0e4d5d4640 1 --2- 172.21.15.81:0/2684217127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e4808ccd0 0x7f0e480cee70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0e480d4c50 tx=0x7f0e3002a440 comp rx=0 tx=0).stop 2024-06-29T14:23:31.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.808+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/2684217127 shutdown_connections 2024-06-29T14:23:31.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.808+0000 7f0e4d5d4640 1 --2- 172.21.15.81:0/2684217127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e4808ccd0 0x7f0e480cee70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.808+0000 7f0e4d5d4640 1 --2- 172.21.15.81:0/2684217127 >> v2:172.21.15.167:3300/0 conn(0x7f0e4808c270 0x7f0e480ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.808+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/2684217127 >> 172.21.15.81:0/2684217127 conn(0x7f0e480b56d0 msgr2=0x7f0e480b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:31.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.808+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/2684217127 shutdown_connections 2024-06-29T14:23:31.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.808+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/2684217127 wait complete. 2024-06-29T14:23:31.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.810+0000 7f0e4d5d4640 1 Processor -- start 2024-06-29T14:23:31.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.810+0000 7f0e4d5d4640 1 -- start start 2024-06-29T14:23:31.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e4d5d4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e4808ccd0 0x7f0e481e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e4d5d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e480d17b0 0x7f0e480c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e4d5d4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e481e5c30 0x7f0e481e6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e4d5d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0e480d22d0 con 0x7f0e480d17b0 2024-06-29T14:23:31.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e4d5d4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0e480b46b0 con 0x7f0e4808ccd0 2024-06-29T14:23:31.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e4d5d4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0e480d2450 con 0x7f0e481e5c30 2024-06-29T14:23:31.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e467fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e480d17b0 0x7f0e480c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e467fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e480d17b0 0x7f0e480c8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60632/0 (socket says 172.21.15.81:60632) 2024-06-29T14:23:31.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e467fc640 1 -- 172.21.15.81:0/141390613 learned_addr learned my addr 172.21.15.81:0/141390613 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:31.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e46ffd640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e4808ccd0 0x7f0e481e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e477fe640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e481e5c30 0x7f0e481e6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e467fc640 1 -- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e481e5c30 msgr2=0x7f0e481e6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e467fc640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e481e5c30 0x7f0e481e6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e467fc640 1 -- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e4808ccd0 msgr2=0x7f0e481e3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:31.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.811+0000 7f0e467fc640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e4808ccd0 0x7f0e481e3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:31.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.812+0000 7f0e467fc640 1 -- 172.21.15.81:0/141390613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e481e6580 con 0x7f0e480d17b0 2024-06-29T14:23:31.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.812+0000 7f0e467fc640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e480d17b0 0x7f0e480c8c60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0e30000c00 tx=0x7f0e30003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:31.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.812+0000 7f0e3f7fe640 1 -- 172.21.15.81:0/141390613 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3003e030 con 0x7f0e480d17b0 2024-06-29T14:23:31.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.812+0000 7f0e3f7fe640 1 -- 172.21.15.81:0/141390613 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e30003e00 con 0x7f0e480d17b0 2024-06-29T14:23:31.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.812+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/141390613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e480c91a0 con 0x7f0e480d17b0 2024-06-29T14:23:31.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.812+0000 7f0e3f7fe640 1 -- 172.21.15.81:0/141390613 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30035040 con 0x7f0e480d17b0 2024-06-29T14:23:31.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.812+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/141390613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e480a0c20 con 0x7f0e480d17b0 2024-06-29T14:23:31.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.813+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/141390613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e4808c270 con 0x7f0e480d17b0 2024-06-29T14:23:31.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.813+0000 7f0e3f7fe640 1 -- 172.21.15.81:0/141390613 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0e30041030 con 0x7f0e480d17b0 2024-06-29T14:23:31.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.814+0000 7f0e3f7fe640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0e28076300 0x7f0e28078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:31.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.814+0000 7f0e3f7fe640 1 -- 172.21.15.81:0/141390613 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f0e300b7ad0 con 0x7f0e480d17b0 2024-06-29T14:23:31.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.814+0000 7f0e46ffd640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0e28076300 0x7f0e28078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:31.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.815+0000 7f0e46ffd640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0e28076300 0x7f0e28078810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0e38005fd0 tx=0x7f0e380095e0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:31.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:31.818+0000 7f0e3f7fe640 1 -- 172.21.15.81:0/141390613 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0e481e6580 con 0x7f0e480d17b0 2024-06-29T14:23:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.069+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/141390613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f0e4806a350 con 0x7f0e480d17b0 2024-06-29T14:23:32.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.072+0000 7f0e3f7fe640 1 -- 172.21.15.81:0/141390613 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v22) ==== 90+0+0 (secure 0 0 0) 0x7f0e30081130 con 0x7f0e480d17b0 2024-06-29T14:23:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0e28076300 msgr2=0x7f0e28078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0e28076300 0x7f0e28078810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0e38005fd0 tx=0x7f0e380095e0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/141390613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e480d17b0 msgr2=0x7f0e480c8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e480d17b0 0x7f0e480c8c60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0e30000c00 tx=0x7f0e30003910 comp rx=0 tx=0).stop 2024-06-29T14:23:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/141390613 shutdown_connections 2024-06-29T14:23:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0e28076300 0x7f0e28078810 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e481e5c30 0x7f0e481e6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e480d17b0 0x7f0e480c8c60 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 --2- 172.21.15.81:0/141390613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e4808ccd0 0x7f0e481e3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/141390613 >> 172.21.15.81:0/141390613 conn(0x7f0e480b56d0 msgr2=0x7f0e480d1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/141390613 shutdown_connections 2024-06-29T14:23:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.074+0000 7f0e4d5d4640 1 -- 172.21.15.81:0/141390613 wait complete. 2024-06-29T14:23:32.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-06-29T14:23:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.199+0000 7f8a7043c640 1 -- 172.21.15.81:0/3510089278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a680ce770 msgr2=0x7f8a680b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.199+0000 7f8a7043c640 1 --2- 172.21.15.81:0/3510089278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a680ce770 0x7f8a680b35c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8a680c7400 tx=0x7f8a6402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.199+0000 7f8a7043c640 1 -- 172.21.15.81:0/3510089278 shutdown_connections 2024-06-29T14:23:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.199+0000 7f8a7043c640 1 --2- 172.21.15.81:0/3510089278 >> v2:172.21.15.167:3300/0 conn(0x7f8a680ceb80 0x7f8a680b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.199+0000 7f8a7043c640 1 --2- 172.21.15.81:0/3510089278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a680ce770 0x7f8a680b35c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.199+0000 7f8a7043c640 1 -- 172.21.15.81:0/3510089278 >> 172.21.15.81:0/3510089278 conn(0x7f8a680d3c60 msgr2=0x7f8a680d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.199+0000 7f8a7043c640 1 -- 172.21.15.81:0/3510089278 shutdown_connections 2024-06-29T14:23:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.199+0000 7f8a7043c640 1 -- 172.21.15.81:0/3510089278 wait complete. 2024-06-29T14:23:32.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.201+0000 7f8a7043c640 1 Processor -- start 2024-06-29T14:23:32.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.201+0000 7f8a7043c640 1 -- start start 2024-06-29T14:23:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a7043c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a680ce770 0x7f8a681d6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a7043c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a680ceb80 0x7f8a680cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a7043c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a680d2240 0x7f8a681e3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a7043c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8a680d15a0 con 0x7f8a680d2240 2024-06-29T14:23:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a7043c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8a680d1420 con 0x7f8a680ce770 2024-06-29T14:23:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a7043c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8a680d4990 con 0x7f8a680ceb80 2024-06-29T14:23:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6e9d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a680d2240 0x7f8a681e3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6e9d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a680d2240 0x7f8a681e3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60652/0 (socket says 172.21.15.81:60652) 2024-06-29T14:23:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6e9d9640 1 -- 172.21.15.81:0/3274618384 learned_addr learned my addr 172.21.15.81:0/3274618384 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6d9d7640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a680ceb80 0x7f8a680cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6e1d8640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a680ce770 0x7f8a681d6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6e9d9640 1 -- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a680ceb80 msgr2=0x7f8a680cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6e9d9640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a680ceb80 0x7f8a680cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6e9d9640 1 -- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a680ce770 msgr2=0x7f8a681d6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6e9d9640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a680ce770 0x7f8a681d6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6e9d9640 1 -- 172.21.15.81:0/3274618384 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6809da00 con 0x7f8a680d2240 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a6e9d9640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a680d2240 0x7f8a681e3b00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000ba10 tx=0x7f8a6000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a5effd640 1 -- 172.21.15.81:0/3274618384 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60019070 con 0x7f8a680d2240 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a5effd640 1 -- 172.21.15.81:0/3274618384 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a6000cc30 con 0x7f8a680d2240 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a5effd640 1 -- 172.21.15.81:0/3274618384 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6000ce60 con 0x7f8a680d2240 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.203+0000 7f8a7043c640 1 -- 172.21.15.81:0/3274618384 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a680cb7d0 con 0x7f8a680d2240 2024-06-29T14:23:32.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.204+0000 7f8a7043c640 1 -- 172.21.15.81:0/3274618384 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a680cbd90 con 0x7f8a680d2240 2024-06-29T14:23:32.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.204+0000 7f8a7043c640 1 -- 172.21.15.81:0/3274618384 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a680a1d50 con 0x7f8a680d2240 2024-06-29T14:23:32.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.205+0000 7f8a5effd640 1 -- 172.21.15.81:0/3274618384 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8a6001c070 con 0x7f8a680d2240 2024-06-29T14:23:32.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.206+0000 7f8a5effd640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8a4c0762d0 0x7f8a4c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.206+0000 7f8a5effd640 1 -- 172.21.15.81:0/3274618384 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f8a60097040 con 0x7f8a680d2240 2024-06-29T14:23:32.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.206+0000 7f8a6e1d8640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8a4c0762d0 0x7f8a4c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.206+0000 7f8a6e1d8640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8a4c0762d0 0x7f8a4c0787e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a64033000 tx=0x7f8a64004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:32.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.210+0000 7f8a5effd640 1 -- 172.21.15.81:0/3274618384 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8a680a1d50 con 0x7f8a680d2240 2024-06-29T14:23:32.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.460+0000 7f8a7043c640 1 -- 172.21.15.81:0/3274618384 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f8a680b2300 con 0x7f8a680d2240 2024-06-29T14:23:32.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.462+0000 7f8a5effd640 1 -- 172.21.15.81:0/3274618384 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v23) ==== 90+0+0 (secure 0 0 0) 0x7f8a60060cf0 con 0x7f8a680d2240 2024-06-29T14:23:32.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 -- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8a4c0762d0 msgr2=0x7f8a4c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8a4c0762d0 0x7f8a4c0787e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a64033000 tx=0x7f8a64004730 comp rx=0 tx=0).stop 2024-06-29T14:23:32.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 -- 172.21.15.81:0/3274618384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a680d2240 msgr2=0x7f8a681e3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a680d2240 0x7f8a681e3b00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000ba10 tx=0x7f8a6000bed0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 -- 172.21.15.81:0/3274618384 shutdown_connections 2024-06-29T14:23:32.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f8a4c0762d0 0x7f8a4c0787e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a680d2240 0x7f8a681e3b00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8a680ceb80 0x7f8a680cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 --2- 172.21.15.81:0/3274618384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8a680ce770 0x7f8a681d6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 -- 172.21.15.81:0/3274618384 >> 172.21.15.81:0/3274618384 conn(0x7f8a680d3c60 msgr2=0x7f8a681e6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:32.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 -- 172.21.15.81:0/3274618384 shutdown_connections 2024-06-29T14:23:32.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.465+0000 7f8a7043c640 1 -- 172.21.15.81:0/3274618384 wait complete. 2024-06-29T14:23:32.475 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-06-29T14:23:32.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.591+0000 7f551992b640 1 -- 172.21.15.81:0/2213731011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55140b5250 msgr2=0x7f55140d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.591+0000 7f551992b640 1 --2- 172.21.15.81:0/2213731011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55140b5250 0x7f55140d09c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5508009830 tx=0x7f550802a5a0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.591+0000 7f551992b640 1 -- 172.21.15.81:0/2213731011 shutdown_connections 2024-06-29T14:23:32.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.591+0000 7f551992b640 1 --2- 172.21.15.81:0/2213731011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55140b5250 0x7f55140d09c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.591+0000 7f551992b640 1 --2- 172.21.15.81:0/2213731011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55140b9040 0x7f55140d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.591+0000 7f551992b640 1 -- 172.21.15.81:0/2213731011 >> 172.21.15.81:0/2213731011 conn(0x7f551409cf30 msgr2=0x7f55140d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:32.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.591+0000 7f551992b640 1 -- 172.21.15.81:0/2213731011 shutdown_connections 2024-06-29T14:23:32.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.591+0000 7f551992b640 1 -- 172.21.15.81:0/2213731011 wait complete. 2024-06-29T14:23:32.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.592+0000 7f551992b640 1 Processor -- start 2024-06-29T14:23:32.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.593+0000 7f551992b640 1 -- start start 2024-06-29T14:23:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f551992b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55140b5250 0x7f55141f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f551992b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55140b7290 0x7f551415d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f551992b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55140b9040 0x7f55140a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f551992b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f55140b7db0 con 0x7f55140b9040 2024-06-29T14:23:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f551992b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f55140d2d50 con 0x7f55140b7290 2024-06-29T14:23:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f551992b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f55140b7f30 con 0x7f55140b5250 2024-06-29T14:23:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f55137fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55140b9040 0x7f55140a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f55137fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55140b9040 0x7f55140a1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60672/0 (socket says 172.21.15.81:60672) 2024-06-29T14:23:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f55137fe640 1 -- 172.21.15.81:0/1043058188 learned_addr learned my addr 172.21.15.81:0/1043058188 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f5512ffd640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55140b5250 0x7f55141f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f55127fc640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55140b7290 0x7f551415d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f55137fe640 1 -- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55140b5250 msgr2=0x7f55141f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f55137fe640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55140b5250 0x7f55141f87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f55137fe640 1 -- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55140b7290 msgr2=0x7f551415d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f55137fe640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55140b7290 0x7f551415d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f55137fe640 1 -- 172.21.15.81:0/1043058188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55141f90c0 con 0x7f55140b9040 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.594+0000 7f55127fc640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55140b7290 0x7f551415d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.595+0000 7f55137fe640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55140b9040 0x7f55140a1030 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f550c00cab0 tx=0x7f550c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.595+0000 7f54fbfff640 1 -- 172.21.15.81:0/1043058188 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c016020 con 0x7f55140b9040 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.595+0000 7f54fbfff640 1 -- 172.21.15.81:0/1043058188 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f550c00ad20 con 0x7f55140b9040 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.595+0000 7f54fbfff640 1 -- 172.21.15.81:0/1043058188 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c012680 con 0x7f55140b9040 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.595+0000 7f551992b640 1 -- 172.21.15.81:0/1043058188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55141f93a0 con 0x7f55140b9040 2024-06-29T14:23:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.595+0000 7f551992b640 1 -- 172.21.15.81:0/1043058188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55140b2a90 con 0x7f55140b9040 2024-06-29T14:23:32.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.596+0000 7f551992b640 1 -- 172.21.15.81:0/1043058188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5514068210 con 0x7f55140b9040 2024-06-29T14:23:32.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.597+0000 7f54fbfff640 1 -- 172.21.15.81:0/1043058188 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f550c019020 con 0x7f55140b9040 2024-06-29T14:23:32.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.597+0000 7f54fbfff640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f55000762d0 0x7f55000787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.597+0000 7f54fbfff640 1 -- 172.21.15.81:0/1043058188 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f550c097a60 con 0x7f55140b9040 2024-06-29T14:23:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.597+0000 7f5512ffd640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f55000762d0 0x7f55000787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.598+0000 7f5512ffd640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f55000762d0 0x7f55000787e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5504005fd0 tx=0x7f5504005850 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.601+0000 7f54fbfff640 1 -- 172.21.15.81:0/1043058188 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5514068210 con 0x7f55140b9040 2024-06-29T14:23:32.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.854+0000 7f551992b640 1 -- 172.21.15.81:0/1043058188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f55140b84c0 con 0x7f55140b9040 2024-06-29T14:23:32.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.857+0000 7f54fbfff640 1 -- 172.21.15.81:0/1043058188 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v24) ==== 90+0+0 (secure 0 0 0) 0x7f550c020020 con 0x7f55140b9040 2024-06-29T14:23:32.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 -- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f55000762d0 msgr2=0x7f55000787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f55000762d0 0x7f55000787e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5504005fd0 tx=0x7f5504005850 comp rx=0 tx=0).stop 2024-06-29T14:23:32.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 -- 172.21.15.81:0/1043058188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55140b9040 msgr2=0x7f55140a1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55140b9040 0x7f55140a1030 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f550c00cab0 tx=0x7f550c00cf70 comp rx=0 tx=0).stop 2024-06-29T14:23:32.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 -- 172.21.15.81:0/1043058188 shutdown_connections 2024-06-29T14:23:32.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f55000762d0 0x7f55000787e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55140b9040 0x7f55140a1030 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55140b7290 0x7f551415d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 --2- 172.21.15.81:0/1043058188 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55140b5250 0x7f55141f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 -- 172.21.15.81:0/1043058188 >> 172.21.15.81:0/1043058188 conn(0x7f551409cf30 msgr2=0x7f55140d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:32.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 -- 172.21.15.81:0/1043058188 shutdown_connections 2024-06-29T14:23:32.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.859+0000 7f551992b640 1 -- 172.21.15.81:0/1043058188 wait complete. 2024-06-29T14:23:32.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-06-29T14:23:32.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.989+0000 7f182fdc1640 1 -- 172.21.15.81:0/3571176317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18280d0230 msgr2=0x7f18280d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.989+0000 7f182fdc1640 1 --2- 172.21.15.81:0/3571176317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18280d0230 0x7f18280d0640 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1818005fd0 tx=0x7f181802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.989+0000 7f182fdc1640 1 -- 172.21.15.81:0/3571176317 shutdown_connections 2024-06-29T14:23:32.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.989+0000 7f182fdc1640 1 --2- 172.21.15.81:0/3571176317 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18280b4f40 0x7f18280b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.989+0000 7f182fdc1640 1 --2- 172.21.15.81:0/3571176317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18280d0230 0x7f18280d0640 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.989+0000 7f182fdc1640 1 -- 172.21.15.81:0/3571176317 >> 172.21.15.81:0/3571176317 conn(0x7f18280b3680 msgr2=0x7f18280b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:32.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.989+0000 7f182fdc1640 1 -- 172.21.15.81:0/3571176317 shutdown_connections 2024-06-29T14:23:32.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.989+0000 7f182fdc1640 1 -- 172.21.15.81:0/3571176317 wait complete. 2024-06-29T14:23:32.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.991+0000 7f182fdc1640 1 Processor -- start 2024-06-29T14:23:32.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.991+0000 7f182fdc1640 1 -- start start 2024-06-29T14:23:32.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.992+0000 7f182fdc1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18280b3200 0x7f182814dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.992+0000 7f182fdc1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18280b4f40 0x7f1828158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.992+0000 7f182fdc1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18280d0230 0x7f182809e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182fdc1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f18280b5c20 con 0x7f18280d0230 2024-06-29T14:23:32.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182fdc1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f18280b5aa0 con 0x7f18280b4f40 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182fdc1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f18280b5da0 con 0x7f18280b3200 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182e35e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18280d0230 0x7f182809e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182e35e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18280d0230 0x7f182809e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60688/0 (socket says 172.21.15.81:60688) 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182e35e640 1 -- 172.21.15.81:0/4274532770 learned_addr learned my addr 172.21.15.81:0/4274532770 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182db5d640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18280b3200 0x7f182814dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182d35c640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18280b4f40 0x7f1828158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182e35e640 1 -- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18280b3200 msgr2=0x7f182814dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182e35e640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18280b3200 0x7f182814dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182e35e640 1 -- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18280b4f40 msgr2=0x7f1828158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182e35e640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18280b4f40 0x7f1828158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182e35e640 1 -- 172.21.15.81:0/4274532770 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182809cc70 con 0x7f18280d0230 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182e35e640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18280d0230 0x7f182809e3b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f182000d9d0 tx=0x7f182000de90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:32.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f181e7fc640 1 -- 172.21.15.81:0/4274532770 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1820017070 con 0x7f18280d0230 2024-06-29T14:23:32.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f181e7fc640 1 -- 172.21.15.81:0/4274532770 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1820010460 con 0x7f18280d0230 2024-06-29T14:23:32.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182fdc1640 1 -- 172.21.15.81:0/4274532770 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f182809cf50 con 0x7f18280d0230 2024-06-29T14:23:32.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f181e7fc640 1 -- 172.21.15.81:0/4274532770 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182000be60 con 0x7f18280d0230 2024-06-29T14:23:32.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.993+0000 7f182fdc1640 1 -- 172.21.15.81:0/4274532770 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f182809d510 con 0x7f18280d0230 2024-06-29T14:23:32.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.994+0000 7f182fdc1640 1 -- 172.21.15.81:0/4274532770 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18280b5350 con 0x7f18280d0230 2024-06-29T14:23:32.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.995+0000 7f181e7fc640 1 -- 172.21.15.81:0/4274532770 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f18200108b0 con 0x7f18280d0230 2024-06-29T14:23:32.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.996+0000 7f181e7fc640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f180c0762d0 0x7f180c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:32.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.996+0000 7f181e7fc640 1 -- 172.21.15.81:0/4274532770 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f18200979c0 con 0x7f18280d0230 2024-06-29T14:23:32.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.996+0000 7f182db5d640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f180c0762d0 0x7f180c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:32.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:32.996+0000 7f182db5d640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f180c0762d0 0x7f180c0787e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1824001720 tx=0x7f1824009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:32.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.000+0000 7f181e7fc640 1 -- 172.21.15.81:0/4274532770 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1820099e20 con 0x7f18280d0230 2024-06-29T14:23:33.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.251+0000 7f182fdc1640 1 -- 172.21.15.81:0/4274532770 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f182815b480 con 0x7f18280d0230 2024-06-29T14:23:33.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.256+0000 7f181e7fc640 1 -- 172.21.15.81:0/4274532770 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v25) ==== 97+0+0 (secure 0 0 0) 0x7f1820060f90 con 0x7f18280d0230 2024-06-29T14:23:33.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 -- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f180c0762d0 msgr2=0x7f180c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f180c0762d0 0x7f180c0787e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1824001720 tx=0x7f1824009040 comp rx=0 tx=0).stop 2024-06-29T14:23:33.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 -- 172.21.15.81:0/4274532770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18280d0230 msgr2=0x7f182809e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18280d0230 0x7f182809e3b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f182000d9d0 tx=0x7f182000de90 comp rx=0 tx=0).stop 2024-06-29T14:23:33.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 -- 172.21.15.81:0/4274532770 shutdown_connections 2024-06-29T14:23:33.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f180c0762d0 0x7f180c0787e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18280d0230 0x7f182809e3b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18280b4f40 0x7f1828158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 --2- 172.21.15.81:0/4274532770 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18280b3200 0x7f182814dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 -- 172.21.15.81:0/4274532770 >> 172.21.15.81:0/4274532770 conn(0x7f18280b3680 msgr2=0x7f1828070a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:33.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 -- 172.21.15.81:0/4274532770 shutdown_connections 2024-06-29T14:23:33.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.258+0000 7f182fdc1640 1 -- 172.21.15.81:0/4274532770 wait complete. 2024-06-29T14:23:33.268 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-06-29T14:23:33.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.384+0000 7f710b593640 1 -- 172.21.15.81:0/3687448450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70f0004000 msgr2=0x7f70f0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.384+0000 7f710b593640 1 --2- 172.21.15.81:0/3687448450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70f0004000 0x7f70f0004800 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f70f4006fc0 tx=0x7f70f0005830 comp rx=0 tx=0).stop 2024-06-29T14:23:33.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.384+0000 7f7109b30640 1 -- 172.21.15.81:0/3687448450 reap_dead start 2024-06-29T14:23:33.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.384+0000 7f710b593640 1 -- 172.21.15.81:0/3687448450 shutdown_connections 2024-06-29T14:23:33.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.384+0000 7f710b593640 1 -- 172.21.15.81:0/3687448450 >> 172.21.15.81:0/3687448450 conn(0x7f7104072950 msgr2=0x7f71040b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:33.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.384+0000 7f710b593640 1 -- 172.21.15.81:0/3687448450 shutdown_connections 2024-06-29T14:23:33.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.384+0000 7f710b593640 1 -- 172.21.15.81:0/3687448450 wait complete. 2024-06-29T14:23:33.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.385+0000 7f710b593640 1 Processor -- start 2024-06-29T14:23:33.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.386+0000 7f710b593640 1 -- start start 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f710b593640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70f0004000 0x7f7104151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f710b593640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f710415e240 0x7f71041f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f710b593640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71040d0030 0x7f71040d0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f710b593640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f710415b190 con 0x7f71040d0030 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f710b593640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f710415b010 con 0x7f70f0004000 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f710b593640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f71040aee20 con 0x7f710415e240 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f7109b30640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71040d0030 0x7f71040d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f7109b30640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71040d0030 0x7f71040d0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60698/0 (socket says 172.21.15.81:60698) 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f7109b30640 1 -- 172.21.15.81:0/3877810103 learned_addr learned my addr 172.21.15.81:0/3877810103 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f710932f640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70f0004000 0x7f7104151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f7108b2e640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f710415e240 0x7f71041f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f7109b30640 1 -- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f710415e240 msgr2=0x7f71041f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f7109b30640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f710415e240 0x7f71041f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f7109b30640 1 -- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70f0004000 msgr2=0x7f7104151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f7109b30640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70f0004000 0x7f7104151750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f7109b30640 1 -- 172.21.15.81:0/3877810103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710415e650 con 0x7f71040d0030 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.387+0000 7f7109b30640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71040d0030 0x7f71040d0440 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f71040737a0 tx=0x7f70fc010e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.388+0000 7f70f9ffb640 1 -- 172.21.15.81:0/3877810103 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc00edf0 con 0x7f71040d0030 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.388+0000 7f70f9ffb640 1 -- 172.21.15.81:0/3877810103 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70fc015070 con 0x7f71040d0030 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.388+0000 7f70f9ffb640 1 -- 172.21.15.81:0/3877810103 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc003d00 con 0x7f71040d0030 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.388+0000 7f710b593640 1 -- 172.21.15.81:0/3877810103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71040d0980 con 0x7f71040d0030 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.388+0000 7f710b593640 1 -- 172.21.15.81:0/3877810103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7104152280 con 0x7f71040d0030 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.389+0000 7f710b593640 1 -- 172.21.15.81:0/3877810103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7104151d10 con 0x7f71040d0030 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.390+0000 7f70f9ffb640 1 -- 172.21.15.81:0/3877810103 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f70fc004030 con 0x7f71040d0030 2024-06-29T14:23:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.390+0000 7f70f9ffb640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70e80762d0 0x7f70e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:33.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.390+0000 7f70f9ffb640 1 -- 172.21.15.81:0/3877810103 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f70fc097b80 con 0x7f71040d0030 2024-06-29T14:23:33.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.390+0000 7f710932f640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70e80762d0 0x7f70e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:33.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.391+0000 7f710932f640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70e80762d0 0x7f70e80787e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7100006fd0 tx=0x7f7100006d70 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:33.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.394+0000 7f70f9ffb640 1 -- 172.21.15.81:0/3877810103 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f71040d0980 con 0x7f71040d0030 2024-06-29T14:23:33.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.644+0000 7f710b593640 1 -- 172.21.15.81:0/3877810103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f71040cfd20 con 0x7f71040d0030 2024-06-29T14:23:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.648+0000 7f70f9ffb640 1 -- 172.21.15.81:0/3877810103 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v26) ==== 90+0+0 (secure 0 0 0) 0x7f70fc0611e0 con 0x7f71040d0030 2024-06-29T14:23:33.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 -- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70e80762d0 msgr2=0x7f70e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70e80762d0 0x7f70e80787e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7100006fd0 tx=0x7f7100006d70 comp rx=0 tx=0).stop 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 -- 172.21.15.81:0/3877810103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71040d0030 msgr2=0x7f71040d0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71040d0030 0x7f71040d0440 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f71040737a0 tx=0x7f70fc010e50 comp rx=0 tx=0).stop 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 -- 172.21.15.81:0/3877810103 shutdown_connections 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f70e80762d0 0x7f70e80787e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71040d0030 0x7f71040d0440 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f710415e240 0x7f71041f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 --2- 172.21.15.81:0/3877810103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70f0004000 0x7f7104151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 -- 172.21.15.81:0/3877810103 >> 172.21.15.81:0/3877810103 conn(0x7f7104072950 msgr2=0x7f7104068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 -- 172.21.15.81:0/3877810103 shutdown_connections 2024-06-29T14:23:33.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.650+0000 7f710b593640 1 -- 172.21.15.81:0/3877810103 wait complete. 2024-06-29T14:23:33.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:33.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.775+0000 7f45e9973640 1 -- 172.21.15.81:0/963661332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d0004800 msgr2=0x7f45d0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.775+0000 7f45e9973640 1 --2- 172.21.15.81:0/963661332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d0004800 0x7f45d0004c10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00c290 tx=0x7f45d0005830 comp rx=0 tx=0).stop 2024-06-29T14:23:33.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.775+0000 7f45e37fe640 1 -- 172.21.15.81:0/963661332 reap_dead start 2024-06-29T14:23:33.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.775+0000 7f45e9973640 1 -- 172.21.15.81:0/963661332 shutdown_connections 2024-06-29T14:23:33.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.775+0000 7f45e9973640 1 -- 172.21.15.81:0/963661332 >> 172.21.15.81:0/963661332 conn(0x7f45e40b2920 msgr2=0x7f45e40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:33.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.775+0000 7f45e9973640 1 -- 172.21.15.81:0/963661332 shutdown_connections 2024-06-29T14:23:33.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.775+0000 7f45e9973640 1 -- 172.21.15.81:0/963661332 wait complete. 2024-06-29T14:23:33.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.777+0000 7f45e9973640 1 Processor -- start 2024-06-29T14:23:33.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.777+0000 7f45e9973640 1 -- start start 2024-06-29T14:23:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e9973640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d0004800 0x7f45e415bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e9973640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45e413b270 0x7f45e4156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e9973640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f45e40b1830 0x7f45e40b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e9973640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f45e406fd30 con 0x7f45d0004800 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e9973640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f45e406fbb0 con 0x7f45e413b270 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e9973640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f45e406feb0 con 0x7f45e40b1830 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d0004800 0x7f45e415bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d0004800 0x7f45e415bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60712/0 (socket says 172.21.15.81:60712) 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e2ffd640 1 -- 172.21.15.81:0/1972839655 learned_addr learned my addr 172.21.15.81:0/1972839655 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e2ffd640 1 -- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f45e40b1830 msgr2=0x7f45e40b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45dbfff640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45e413b270 0x7f45e4156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e2ffd640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f45e40b1830 0x7f45e40b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e2ffd640 1 -- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45e413b270 msgr2=0x7f45e4156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e2ffd640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45e413b270 0x7f45e4156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.778+0000 7f45e2ffd640 1 -- 172.21.15.81:0/1972839655 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e40b2180 con 0x7f45d0004800 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.779+0000 7f45e2ffd640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d0004800 0x7f45e415bc50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f45d4011a60 tx=0x7f45d4011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.779+0000 7f45e0ff9640 1 -- 172.21.15.81:0/1972839655 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d401e070 con 0x7f45d0004800 2024-06-29T14:23:33.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.779+0000 7f45e0ff9640 1 -- 172.21.15.81:0/1972839655 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45d400fc70 con 0x7f45d0004800 2024-06-29T14:23:33.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.779+0000 7f45e9973640 1 -- 172.21.15.81:0/1972839655 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e413e130 con 0x7f45d0004800 2024-06-29T14:23:33.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.779+0000 7f45e0ff9640 1 -- 172.21.15.81:0/1972839655 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d4019410 con 0x7f45d0004800 2024-06-29T14:23:33.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.779+0000 7f45e9973640 1 -- 172.21.15.81:0/1972839655 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e413e6f0 con 0x7f45d0004800 2024-06-29T14:23:33.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.780+0000 7f45e9973640 1 -- 172.21.15.81:0/1972839655 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e40d0010 con 0x7f45d0004800 2024-06-29T14:23:33.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.781+0000 7f45e0ff9640 1 -- 172.21.15.81:0/1972839655 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f45d4021070 con 0x7f45d0004800 2024-06-29T14:23:33.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.781+0000 7f45e0ff9640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f45c4076300 0x7f45c4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:33.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.781+0000 7f45e0ff9640 1 -- 172.21.15.81:0/1972839655 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f45d4097b60 con 0x7f45d0004800 2024-06-29T14:23:33.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.782+0000 7f45dbfff640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f45c4076300 0x7f45c4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:33.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.782+0000 7f45dbfff640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f45c4076300 0x7f45c4078810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f45cc033000 tx=0x7f45cc02d000 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:33.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.786+0000 7f45e0ff9640 1 -- 172.21.15.81:0/1972839655 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f45e40b2180 con 0x7f45d0004800 2024-06-29T14:23:33.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.969+0000 7f45e9973640 1 -- 172.21.15.81:0/1972839655 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45e406f330 con 0x7f45d0004800 2024-06-29T14:23:33.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.970+0000 7f45e0ff9640 1 -- 172.21.15.81:0/1972839655 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+12879 (secure 0 0 0) 0x7f45d40611c0 con 0x7f45d0004800 2024-06-29T14:23:33.969 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:33.969 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":29,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:23:30.900904+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.971+0000 7f45e9973640 1 -- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f45c4076300 msgr2=0x7f45c4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.971+0000 7f45e9973640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f45c4076300 0x7f45c4078810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f45cc033000 tx=0x7f45cc02d000 comp rx=0 tx=0).stop 2024-06-29T14:23:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.971+0000 7f45e9973640 1 -- 172.21.15.81:0/1972839655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d0004800 msgr2=0x7f45e415bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.971+0000 7f45e9973640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d0004800 0x7f45e415bc50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f45d4011a60 tx=0x7f45d4011f20 comp rx=0 tx=0).stop 2024-06-29T14:23:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.972+0000 7f45e9973640 1 -- 172.21.15.81:0/1972839655 shutdown_connections 2024-06-29T14:23:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.972+0000 7f45e9973640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f45c4076300 0x7f45c4078810 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.972+0000 7f45e9973640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f45e40b1830 0x7f45e40b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.972+0000 7f45e9973640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f45e413b270 0x7f45e4156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.972+0000 7f45e9973640 1 --2- 172.21.15.81:0/1972839655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d0004800 0x7f45e415bc50 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.972+0000 7f45e9973640 1 -- 172.21.15.81:0/1972839655 >> 172.21.15.81:0/1972839655 conn(0x7f45e40b2920 msgr2=0x7f45e40689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:33.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.972+0000 7f45e9973640 1 -- 172.21.15.81:0/1972839655 shutdown_connections 2024-06-29T14:23:33.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:33.972+0000 7f45e9973640 1 -- 172.21.15.81:0/1972839655 wait complete. 2024-06-29T14:23:33.982 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:23:33.982 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:23:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.100+0000 7f6f3b626640 1 -- 172.21.15.81:0/2759277339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f3415e710 msgr2=0x7f6f3406f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.100+0000 7f6f3b626640 1 --2- 172.21.15.81:0/2759277339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f3415e710 0x7f6f3406f230 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6f340d1870 tx=0x7f6f3002a4d0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.100+0000 7f6f3b626640 1 -- 172.21.15.81:0/2759277339 shutdown_connections 2024-06-29T14:23:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.100+0000 7f6f3b626640 1 --2- 172.21.15.81:0/2759277339 >> v2:172.21.15.167:3300/0 conn(0x7f6f3406ca00 0x7f6f340af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.100+0000 7f6f3b626640 1 --2- 172.21.15.81:0/2759277339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f3415e710 0x7f6f3406f230 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.100+0000 7f6f3b626640 1 -- 172.21.15.81:0/2759277339 >> 172.21.15.81:0/2759277339 conn(0x7f6f340d0930 msgr2=0x7f6f340d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.100+0000 7f6f3b626640 1 -- 172.21.15.81:0/2759277339 shutdown_connections 2024-06-29T14:23:34.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.100+0000 7f6f3b626640 1 -- 172.21.15.81:0/2759277339 wait complete. 2024-06-29T14:23:34.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.101+0000 7f6f3b626640 1 Processor -- start 2024-06-29T14:23:34.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.101+0000 7f6f3b626640 1 -- start start 2024-06-29T14:23:34.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f3b626640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f34154fc0 0x7f6f34148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f3b626640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f3415e710 0x7f6f34158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f3b626640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f34157010 0x7f6f34158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f3b626640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f340b01c0 con 0x7f6f34154fc0 2024-06-29T14:23:34.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f3b626640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6f340b0040 con 0x7f6f34157010 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f3b626640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6f340b0340 con 0x7f6f3415e710 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f393c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f34154fc0 0x7f6f34148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f393c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f34154fc0 0x7f6f34148460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60730/0 (socket says 172.21.15.81:60730) 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f393c2640 1 -- 172.21.15.81:0/3919928951 learned_addr learned my addr 172.21.15.81:0/3919928951 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f393c2640 1 -- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f3415e710 msgr2=0x7f6f34158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f393c2640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f3415e710 0x7f6f34158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f393c2640 1 -- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f34157010 msgr2=0x7f6f34158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f39bc3640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f34157010 0x7f6f34158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f393c2640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f34157010 0x7f6f34158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f393c2640 1 -- 172.21.15.81:0/3919928951 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f34159440 con 0x7f6f34154fc0 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f393c2640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f34154fc0 0x7f6f34148460 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6f300099c0 tx=0x7f6f300039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.103+0000 7f6f29ffb640 1 -- 172.21.15.81:0/3919928951 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3003e030 con 0x7f6f34154fc0 2024-06-29T14:23:34.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.104+0000 7f6f29ffb640 1 -- 172.21.15.81:0/3919928951 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f300047a0 con 0x7f6f34154fc0 2024-06-29T14:23:34.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.104+0000 7f6f3b626640 1 -- 172.21.15.81:0/3919928951 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f34157590 con 0x7f6f34154fc0 2024-06-29T14:23:34.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.104+0000 7f6f29ffb640 1 -- 172.21.15.81:0/3919928951 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30035040 con 0x7f6f34154fc0 2024-06-29T14:23:34.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.104+0000 7f6f3b626640 1 -- 172.21.15.81:0/3919928951 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f34157b50 con 0x7f6f34154fc0 2024-06-29T14:23:34.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.105+0000 7f6f3b626640 1 -- 172.21.15.81:0/3919928951 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f34071530 con 0x7f6f34154fc0 2024-06-29T14:23:34.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.105+0000 7f6f29ffb640 1 -- 172.21.15.81:0/3919928951 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6f30041030 con 0x7f6f34154fc0 2024-06-29T14:23:34.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.106+0000 7f6f29ffb640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f18076300 0x7f6f18078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.106+0000 7f6f29ffb640 1 -- 172.21.15.81:0/3919928951 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f6f300b7b40 con 0x7f6f34154fc0 2024-06-29T14:23:34.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.106+0000 7f6f38bc1640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f18076300 0x7f6f18078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.107+0000 7f6f38bc1640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f18076300 0x7f6f18078810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6f24006fd0 tx=0x7f6f24006bc0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:34.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.110+0000 7f6f29ffb640 1 -- 172.21.15.81:0/3919928951 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6f34071530 con 0x7f6f34154fc0 2024-06-29T14:23:34.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.289+0000 7f6f3b626640 1 -- 172.21.15.81:0/3919928951 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6f3406e2a0 con 0x7f6f34154fc0 2024-06-29T14:23:34.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.290+0000 7f6f29ffb640 1 -- 172.21.15.81:0/3919928951 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f6f3406e2a0 con 0x7f6f34154fc0 2024-06-29T14:23:34.289 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:23:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 -- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f18076300 msgr2=0x7f6f18078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f18076300 0x7f6f18078810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6f24006fd0 tx=0x7f6f24006bc0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 -- 172.21.15.81:0/3919928951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f34154fc0 msgr2=0x7f6f34148460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f34154fc0 0x7f6f34148460 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6f300099c0 tx=0x7f6f300039e0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 -- 172.21.15.81:0/3919928951 shutdown_connections 2024-06-29T14:23:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6f18076300 0x7f6f18078810 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f34157010 0x7f6f34158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f3415e710 0x7f6f34158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 --2- 172.21.15.81:0/3919928951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f34154fc0 0x7f6f34148460 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 -- 172.21.15.81:0/3919928951 >> 172.21.15.81:0/3919928951 conn(0x7f6f340d0930 msgr2=0x7f6f34068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:34.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 -- 172.21.15.81:0/3919928951 shutdown_connections 2024-06-29T14:23:34.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.292+0000 7f6f3b626640 1 -- 172.21.15.81:0/3919928951 wait complete. 2024-06-29T14:23:34.302 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:23:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.415+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/262528644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d740b4430 msgr2=0x7f0d740cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.415+0000 7f0d7aa3c640 1 --2- 172.21.15.81:0/262528644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d740b4430 0x7f0d740cf130 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0d74070220 tx=0x7f0d6802a4d0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.415+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/262528644 shutdown_connections 2024-06-29T14:23:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.415+0000 7f0d7aa3c640 1 --2- 172.21.15.81:0/262528644 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d7406f660 0x7f0d7406fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.415+0000 7f0d7aa3c640 1 --2- 172.21.15.81:0/262528644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d740b4430 0x7f0d740cf130 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.415+0000 7f0d7aa3c640 1 --2- 172.21.15.81:0/262528644 >> v2:172.21.15.167:3300/0 conn(0x7f0d740b35b0 0x7f0d7415b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.415+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/262528644 >> 172.21.15.81:0/262528644 conn(0x7f0d7406ccc0 msgr2=0x7f0d740b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.416+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/262528644 shutdown_connections 2024-06-29T14:23:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.416+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/262528644 wait complete. 2024-06-29T14:23:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.417+0000 7f0d7aa3c640 1 Processor -- start 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.417+0000 7f0d7aa3c640 1 -- start start 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d7aa3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d7406f660 0x7f0d7414a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d7aa3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d740b35b0 0x7f0d74158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d7aa3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d740b4430 0x7f0d741f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d7aa3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7406f160 con 0x7f0d740b4430 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d7aa3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7406efe0 con 0x7f0d7406f660 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d7aa3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7406f2e0 con 0x7f0d740b35b0 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d78fd9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d740b4430 0x7f0d741f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d78fd9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d740b4430 0x7f0d741f3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60738/0 (socket says 172.21.15.81:60738) 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d78fd9640 1 -- 172.21.15.81:0/4036066365 learned_addr learned my addr 172.21.15.81:0/4036066365 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d73fff640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d7406f660 0x7f0d7414a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d737fe640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d740b35b0 0x7f0d74158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d78fd9640 1 -- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d740b35b0 msgr2=0x7f0d74158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d78fd9640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d740b35b0 0x7f0d74158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d78fd9640 1 -- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d7406f660 msgr2=0x7f0d7414a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d78fd9640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d7406f660 0x7f0d7414a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d78fd9640 1 -- 172.21.15.81:0/4036066365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7414b0d0 con 0x7f0d740b4430 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d73fff640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d7406f660 0x7f0d7414a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d737fe640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d740b35b0 0x7f0d74158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d78fd9640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d740b4430 0x7f0d741f3800 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00ea80 tx=0x7f0d6c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d70ff9640 1 -- 172.21.15.81:0/4036066365 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c00cde0 con 0x7f0d740b4430 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d70ff9640 1 -- 172.21.15.81:0/4036066365 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d6c004c10 con 0x7f0d740b4430 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.419+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/4036066365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d7414b3b0 con 0x7f0d740b4430 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.420+0000 7f0d70ff9640 1 -- 172.21.15.81:0/4036066365 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c005020 con 0x7f0d740b4430 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.420+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/4036066365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d7414b970 con 0x7f0d740b4430 2024-06-29T14:23:34.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.420+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/4036066365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d7415d4f0 con 0x7f0d740b4430 2024-06-29T14:23:34.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.421+0000 7f0d70ff9640 1 -- 172.21.15.81:0/4036066365 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0d6c0051c0 con 0x7f0d740b4430 2024-06-29T14:23:34.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.422+0000 7f0d70ff9640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0d600762d0 0x7f0d600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.422+0000 7f0d70ff9640 1 -- 172.21.15.81:0/4036066365 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f0d6c0980a0 con 0x7f0d740b4430 2024-06-29T14:23:34.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.422+0000 7f0d73fff640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0d600762d0 0x7f0d600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.422+0000 7f0d73fff640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0d600762d0 0x7f0d600787e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009b80 tx=0x7f0d64005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:34.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.426+0000 7f0d70ff9640 1 -- 172.21.15.81:0/4036066365 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0d7415d4f0 con 0x7f0d740b4430 2024-06-29T14:23:34.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.610+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/4036066365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f0d740cec70 con 0x7f0d740b4430 2024-06-29T14:23:34.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.611+0000 7f0d70ff9640 1 -- 172.21.15.81:0/4036066365 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) ==== 84+0+65 (secure 0 0 0) 0x7f0d740cec70 con 0x7f0d740b4430 2024-06-29T14:23:34.611 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:34.611 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:23:34.611 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:34.611 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T14:23:34.611 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T14:23:34.611 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:34.611 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:23:34.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0d600762d0 msgr2=0x7f0d600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0d600762d0 0x7f0d600787e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009b80 tx=0x7f0d64005f90 comp rx=0 tx=0).stop 2024-06-29T14:23:34.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/4036066365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d740b4430 msgr2=0x7f0d741f3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d740b4430 0x7f0d741f3800 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00ea80 tx=0x7f0d6c00ef40 comp rx=0 tx=0).stop 2024-06-29T14:23:34.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/4036066365 shutdown_connections 2024-06-29T14:23:34.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f0d600762d0 0x7f0d600787e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d740b4430 0x7f0d741f3800 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d740b35b0 0x7f0d74158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 --2- 172.21.15.81:0/4036066365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d7406f660 0x7f0d7414a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/4036066365 >> 172.21.15.81:0/4036066365 conn(0x7f0d7406ccc0 msgr2=0x7f0d74072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:34.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/4036066365 shutdown_connections 2024-06-29T14:23:34.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.613+0000 7f0d7aa3c640 1 -- 172.21.15.81:0/4036066365 wait complete. 2024-06-29T14:23:34.623 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:23:34.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:23:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.741+0000 7f1d38111640 1 -- 172.21.15.81:0/1981955481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d300b2cb0 msgr2=0x7f1d300d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.741+0000 7f1d38111640 1 --2- 172.21.15.81:0/1981955481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d300b2cb0 0x7f1d300d16b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1d24005fd0 tx=0x7f1d2402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.741+0000 7f1d38111640 1 -- 172.21.15.81:0/1981955481 shutdown_connections 2024-06-29T14:23:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.741+0000 7f1d38111640 1 --2- 172.21.15.81:0/1981955481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d300b2cb0 0x7f1d300d16b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.741+0000 7f1d38111640 1 --2- 172.21.15.81:0/1981955481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d300ae8f0 0x7f1d3015c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.741+0000 7f1d38111640 1 -- 172.21.15.81:0/1981955481 >> 172.21.15.81:0/1981955481 conn(0x7f1d3009aa60 msgr2=0x7f1d3009ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.741+0000 7f1d38111640 1 -- 172.21.15.81:0/1981955481 shutdown_connections 2024-06-29T14:23:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.741+0000 7f1d38111640 1 -- 172.21.15.81:0/1981955481 wait complete. 2024-06-29T14:23:34.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.743+0000 7f1d38111640 1 Processor -- start 2024-06-29T14:23:34.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.743+0000 7f1d38111640 1 -- start start 2024-06-29T14:23:34.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d38111640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d300ae8f0 0x7f1d301532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d38111640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d300b1a90 0x7f1d30148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d38111640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d300b2cb0 0x7f1d30154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d38111640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1d30160050 con 0x7f1d300b1a90 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d38111640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3015fed0 con 0x7f1d300ae8f0 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d38111640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1d301601d0 con 0x7f1d300b2cb0 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d356ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d300b1a90 0x7f1d30148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d356ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d300b1a90 0x7f1d30148460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60760/0 (socket says 172.21.15.81:60760) 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d356ac640 1 -- 172.21.15.81:0/994579614 learned_addr learned my addr 172.21.15.81:0/994579614 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d366ae640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d300b2cb0 0x7f1d30154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.744+0000 7f1d35ead640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d300ae8f0 0x7f1d301532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d356ac640 1 -- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d300b2cb0 msgr2=0x7f1d30154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d356ac640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d300b2cb0 0x7f1d30154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d356ac640 1 -- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d300ae8f0 msgr2=0x7f1d301532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d356ac640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d300ae8f0 0x7f1d301532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d356ac640 1 -- 172.21.15.81:0/994579614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d301489a0 con 0x7f1d300b1a90 2024-06-29T14:23:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d366ae640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d300b2cb0 0x7f1d30154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d35ead640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d300ae8f0 0x7f1d301532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d356ac640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d300b1a90 0x7f1d30148460 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1d24034040 tx=0x7f1d240038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d2e7fc640 1 -- 172.21.15.81:0/994579614 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2403e030 con 0x7f1d300b1a90 2024-06-29T14:23:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d2e7fc640 1 -- 172.21.15.81:0/994579614 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d24003ca0 con 0x7f1d300b1a90 2024-06-29T14:23:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d38111640 1 -- 172.21.15.81:0/994579614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d30152220 con 0x7f1d300b1a90 2024-06-29T14:23:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d2e7fc640 1 -- 172.21.15.81:0/994579614 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24035040 con 0x7f1d300b1a90 2024-06-29T14:23:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.745+0000 7f1d38111640 1 -- 172.21.15.81:0/994579614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d301527e0 con 0x7f1d300b1a90 2024-06-29T14:23:34.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.746+0000 7f1d38111640 1 -- 172.21.15.81:0/994579614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d30160680 con 0x7f1d300b1a90 2024-06-29T14:23:34.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.747+0000 7f1d2e7fc640 1 -- 172.21.15.81:0/994579614 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1d24041030 con 0x7f1d300b1a90 2024-06-29T14:23:34.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.747+0000 7f1d2e7fc640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d1c0762d0 0x7f1d1c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:34.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.748+0000 7f1d2e7fc640 1 -- 172.21.15.81:0/994579614 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f1d240b79a0 con 0x7f1d300b1a90 2024-06-29T14:23:34.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.748+0000 7f1d35ead640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d1c0762d0 0x7f1d1c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:34.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.748+0000 7f1d35ead640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d1c0762d0 0x7f1d1c0787e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1d20005fd0 tx=0x7f1d2000f360 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:34.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.752+0000 7f1d2e7fc640 1 -- 172.21.15.81:0/994579614 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1d24081650 con 0x7f1d300b1a90 2024-06-29T14:23:34.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:34.932+0000 7f1d38111640 1 -- 172.21.15.81:0/994579614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1d30071510 con 0x7f1d300b1a90 2024-06-29T14:23:35.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.913+0000 7f1d2e7fc640 1 -- 172.21.15.81:0/994579614 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) ==== 138+0+0 (secure 0 0 0) 0x7f1d24080f50 con 0x7f1d300b1a90 2024-06-29T14:23:35.912 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T14:23:35.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 -- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d1c0762d0 msgr2=0x7f1d1c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d1c0762d0 0x7f1d1c0787e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1d20005fd0 tx=0x7f1d2000f360 comp rx=0 tx=0).stop 2024-06-29T14:23:35.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 -- 172.21.15.81:0/994579614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d300b1a90 msgr2=0x7f1d30148460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:35.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d300b1a90 0x7f1d30148460 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1d24034040 tx=0x7f1d240038a0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 -- 172.21.15.81:0/994579614 shutdown_connections 2024-06-29T14:23:35.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f1d1c0762d0 0x7f1d1c0787e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d300b2cb0 0x7f1d30154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d300b1a90 0x7f1d30148460 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 --2- 172.21.15.81:0/994579614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d300ae8f0 0x7f1d301532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:35.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 -- 172.21.15.81:0/994579614 >> 172.21.15.81:0/994579614 conn(0x7f1d3009aa60 msgr2=0x7f1d30070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:35.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 -- 172.21.15.81:0/994579614 shutdown_connections 2024-06-29T14:23:35.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:35.916+0000 7f1d38111640 1 -- 172.21.15.81:0/994579614 wait complete. 2024-06-29T14:23:35.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:23:36.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.060+0000 7ffa5650a640 1 -- 172.21.15.81:0/612384978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa3c004d40 msgr2=0x7ffa3c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.060+0000 7ffa5650a640 1 --2- 172.21.15.81:0/612384978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa3c004d40 0x7ffa3c005150 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800c640 tx=0x7ffa3c005830 comp rx=0 tx=0).stop 2024-06-29T14:23:36.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.060+0000 7ffa54aa7640 1 -- 172.21.15.81:0/612384978 reap_dead start 2024-06-29T14:23:36.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.060+0000 7ffa5650a640 1 -- 172.21.15.81:0/612384978 shutdown_connections 2024-06-29T14:23:36.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.060+0000 7ffa5650a640 1 -- 172.21.15.81:0/612384978 >> 172.21.15.81:0/612384978 conn(0x7ffa500d0930 msgr2=0x7ffa500d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:36.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.060+0000 7ffa5650a640 1 -- 172.21.15.81:0/612384978 shutdown_connections 2024-06-29T14:23:36.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.060+0000 7ffa5650a640 1 -- 172.21.15.81:0/612384978 wait complete. 2024-06-29T14:23:36.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.061+0000 7ffa5650a640 1 Processor -- start 2024-06-29T14:23:36.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.062+0000 7ffa5650a640 1 -- start start 2024-06-29T14:23:36.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa5650a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa3c004d40 0x7ffa5014e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa5650a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa500aed30 0x7ffa500ac820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa5650a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa500af140 0x7ffa500acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa5650a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffa500cee70 con 0x7ffa500aed30 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa5650a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ffa500cecf0 con 0x7ffa3c004d40 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa5650a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ffa500ceff0 con 0x7ffa500af140 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa47fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa500aed30 0x7ffa500ac820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa47fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa500aed30 0x7ffa500ac820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60770/0 (socket says 172.21.15.81:60770) 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa47fff640 1 -- 172.21.15.81:0/1736542120 learned_addr learned my addr 172.21.15.81:0/1736542120 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa4ffff640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa3c004d40 0x7ffa5014e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa54aa7640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa500af140 0x7ffa500acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa47fff640 1 -- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa500af140 msgr2=0x7ffa500acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa47fff640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa500af140 0x7ffa500acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa47fff640 1 -- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa3c004d40 msgr2=0x7ffa5014e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa47fff640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa3c004d40 0x7ffa5014e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa47fff640 1 -- 172.21.15.81:0/1736542120 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa500aaea0 con 0x7ffa500aed30 2024-06-29T14:23:36.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa47fff640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa500aed30 0x7ffa500ac820 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ffa38011850 tx=0x7ffa38011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:36.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.063+0000 7ffa4d7fa640 1 -- 172.21.15.81:0/1736542120 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3800fd50 con 0x7ffa500aed30 2024-06-29T14:23:36.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.064+0000 7ffa4d7fa640 1 -- 172.21.15.81:0/1736542120 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa38004030 con 0x7ffa500aed30 2024-06-29T14:23:36.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.064+0000 7ffa5650a640 1 -- 172.21.15.81:0/1736542120 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa500aa600 con 0x7ffa500aed30 2024-06-29T14:23:36.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.064+0000 7ffa4d7fa640 1 -- 172.21.15.81:0/1736542120 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa38004480 con 0x7ffa500aed30 2024-06-29T14:23:36.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.064+0000 7ffa5650a640 1 -- 172.21.15.81:0/1736542120 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa500aabc0 con 0x7ffa500aed30 2024-06-29T14:23:36.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.064+0000 7ffa5650a640 1 -- 172.21.15.81:0/1736542120 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa5006e2a0 con 0x7ffa500aed30 2024-06-29T14:23:36.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.065+0000 7ffa4d7fa640 1 -- 172.21.15.81:0/1736542120 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffa380035f0 con 0x7ffa500aed30 2024-06-29T14:23:36.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.066+0000 7ffa4d7fa640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ffa30076300 0x7ffa30078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:36.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.066+0000 7ffa4d7fa640 1 -- 172.21.15.81:0/1736542120 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5227+0+0 (secure 0 0 0) 0x7ffa38098b10 con 0x7ffa500aed30 2024-06-29T14:23:36.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.066+0000 7ffa4ffff640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ffa30076300 0x7ffa30078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:36.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.067+0000 7ffa4ffff640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ffa30076300 0x7ffa30078810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffa40005e00 tx=0x7ffa40005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:36.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.070+0000 7ffa4d7fa640 1 -- 172.21.15.81:0/1736542120 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa5006e2a0 con 0x7ffa500aed30 2024-06-29T14:23:36.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.252+0000 7ffa5650a640 1 -- 172.21.15.81:0/1736542120 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ffa5015aed0 con 0x7ffa500aed30 2024-06-29T14:23:36.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.918+0000 7ffa4d7fa640 1 -- 172.21.15.81:0/1736542120 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) ==== 170+0+0 (secure 0 0 0) 0x7ffa38061f70 con 0x7ffa500aed30 2024-06-29T14:23:36.917 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T14:23:36.919 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:23:36.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 -- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ffa30076300 msgr2=0x7ffa30078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ffa30076300 0x7ffa30078810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffa40005e00 tx=0x7ffa40005f90 comp rx=0 tx=0).stop 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 -- 172.21.15.81:0/1736542120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa500aed30 msgr2=0x7ffa500ac820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa500aed30 0x7ffa500ac820 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ffa38011850 tx=0x7ffa38011d10 comp rx=0 tx=0).stop 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 -- 172.21.15.81:0/1736542120 shutdown_connections 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7ffa30076300 0x7ffa30078810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa500af140 0x7ffa500acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa500aed30 0x7ffa500ac820 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 --2- 172.21.15.81:0/1736542120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa3c004d40 0x7ffa5014e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 -- 172.21.15.81:0/1736542120 >> 172.21.15.81:0/1736542120 conn(0x7ffa500d0930 msgr2=0x7ffa500b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 -- 172.21.15.81:0/1736542120 shutdown_connections 2024-06-29T14:23:36.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:36.921+0000 7ffa5650a640 1 -- 172.21.15.81:0/1736542120 wait complete. 2024-06-29T14:23:36.922 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:23:36.950 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:23:37.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.111+0000 7f2199081640 1 -- 172.21.15.81:0/4185118957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f219406ccb0 msgr2=0x7f219406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.111+0000 7f2199081640 1 --2- 172.21.15.81:0/4185118957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f219406ccb0 0x7f219406d0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f21940b6210 tx=0x7f217c02a560 comp rx=0 tx=0).stop 2024-06-29T14:23:37.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.111+0000 7f2199081640 1 -- 172.21.15.81:0/4185118957 shutdown_connections 2024-06-29T14:23:37.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.111+0000 7f2199081640 1 --2- 172.21.15.81:0/4185118957 >> v2:172.21.15.167:3300/0 conn(0x7f21940cf2e0 0x7f21940cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.111+0000 7f2199081640 1 --2- 172.21.15.81:0/4185118957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f219406ccb0 0x7f219406d0c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.111+0000 7f2199081640 1 -- 172.21.15.81:0/4185118957 >> 172.21.15.81:0/4185118957 conn(0x7f2194072950 msgr2=0x7f21940b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:37.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.111+0000 7f2199081640 1 -- 172.21.15.81:0/4185118957 shutdown_connections 2024-06-29T14:23:37.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.111+0000 7f2199081640 1 -- 172.21.15.81:0/4185118957 wait complete. 2024-06-29T14:23:37.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.113+0000 7f2199081640 1 Processor -- start 2024-06-29T14:23:37.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.113+0000 7f2199081640 1 -- start start 2024-06-29T14:23:37.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2199081640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f219406ccb0 0x7f219415ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2199081640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21940ad0c0 0x7f219414d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2199081640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21940cf2e0 0x7f21941f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2199081640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f21940aef90 con 0x7f219406ccb0 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2199081640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f21940aee10 con 0x7f21940cf2e0 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2199081640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f21940af110 con 0x7f21940ad0c0 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2192d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f219406ccb0 0x7f219415ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2192d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f219406ccb0 0x7f219415ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60780/0 (socket says 172.21.15.81:60780) 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2192d9d640 1 -- 172.21.15.81:0/3997030891 learned_addr learned my addr 172.21.15.81:0/3997030891 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f218a59c640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21940ad0c0 0x7f219414d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f219359e640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21940cf2e0 0x7f21941f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2192d9d640 1 -- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21940ad0c0 msgr2=0x7f219414d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2192d9d640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21940ad0c0 0x7f219414d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2192d9d640 1 -- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21940cf2e0 msgr2=0x7f21941f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2192d9d640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21940cf2e0 0x7f21941f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2192d9d640 1 -- 172.21.15.81:0/3997030891 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2194151d00 con 0x7f219406ccb0 2024-06-29T14:23:37.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.115+0000 7f2192d9d640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f219406ccb0 0x7f219415ee50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f218400c3c0 tx=0x7f218400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:37.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.116+0000 7f218bfff640 1 -- 172.21.15.81:0/3997030891 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2184017070 con 0x7f219406ccb0 2024-06-29T14:23:37.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.116+0000 7f218bfff640 1 -- 172.21.15.81:0/3997030891 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21840035f0 con 0x7f219406ccb0 2024-06-29T14:23:37.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.116+0000 7f218bfff640 1 -- 172.21.15.81:0/3997030891 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2184003a40 con 0x7f219406ccb0 2024-06-29T14:23:37.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.116+0000 7f2199081640 1 -- 172.21.15.81:0/3997030891 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21941544b0 con 0x7f219406ccb0 2024-06-29T14:23:37.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.116+0000 7f2199081640 1 -- 172.21.15.81:0/3997030891 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f219414d9c0 con 0x7f219406ccb0 2024-06-29T14:23:37.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.117+0000 7f2199081640 1 -- 172.21.15.81:0/3997030891 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2194151ed0 con 0x7f219406ccb0 2024-06-29T14:23:37.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.117+0000 7f218bfff640 1 -- 172.21.15.81:0/3997030891 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f218401a070 con 0x7f219406ccb0 2024-06-29T14:23:37.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.118+0000 7f218bfff640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f21740762d0 0x7f21740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:37.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.118+0000 7f218bfff640 1 -- 172.21.15.81:0/3997030891 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5609+0+0 (secure 0 0 0) 0x7f21840969f0 con 0x7f219406ccb0 2024-06-29T14:23:37.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.118+0000 7f218a59c640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f21740762d0 0x7f21740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:37.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.119+0000 7f218a59c640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f21740762d0 0x7f21740787e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f217c033000 tx=0x7f217c004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:37.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.123+0000 7f218bfff640 1 -- 172.21.15.81:0/3997030891 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f21940cfc30 con 0x7f219406ccb0 2024-06-29T14:23:37.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.341+0000 7f2199081640 1 -- 172.21.15.81:0/3997030891 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f219406d0c0 con 0x7f219406ccb0 2024-06-29T14:23:37.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.920+0000 7f218bfff640 1 -- 172.21.15.81:0/3997030891 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7f218405fd60 con 0x7f219406ccb0 2024-06-29T14:23:37.920 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 4 and data pool 5 2024-06-29T14:23:37.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.926+0000 7f2199081640 1 -- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f21740762d0 msgr2=0x7f21740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.926+0000 7f2199081640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f21740762d0 0x7f21740787e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f217c033000 tx=0x7f217c004730 comp rx=0 tx=0).stop 2024-06-29T14:23:37.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.926+0000 7f2199081640 1 -- 172.21.15.81:0/3997030891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f219406ccb0 msgr2=0x7f219415ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:37.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.926+0000 7f2199081640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f219406ccb0 0x7f219415ee50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f218400c3c0 tx=0x7f218400c880 comp rx=0 tx=0).stop 2024-06-29T14:23:37.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.926+0000 7f2199081640 1 -- 172.21.15.81:0/3997030891 shutdown_connections 2024-06-29T14:23:37.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.926+0000 7f2199081640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f21740762d0 0x7f21740787e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.926+0000 7f2199081640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21940cf2e0 0x7f21941f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.926+0000 7f2199081640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21940ad0c0 0x7f219414d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.926+0000 7f2199081640 1 --2- 172.21.15.81:0/3997030891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f219406ccb0 0x7f219415ee50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:37.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.926+0000 7f2199081640 1 -- 172.21.15.81:0/3997030891 >> 172.21.15.81:0/3997030891 conn(0x7f2194072950 msgr2=0x7f2194157350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:37.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.927+0000 7f2199081640 1 -- 172.21.15.81:0/3997030891 shutdown_connections 2024-06-29T14:23:37.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:37.927+0000 7f2199081640 1 -- 172.21.15.81:0/3997030891 wait complete. 2024-06-29T14:23:37.957 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:38.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.073+0000 7f6b788a4640 1 -- 172.21.15.81:0/1692821269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740d0b90 msgr2=0x7f6b740d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.073+0000 7f6b788a4640 1 --2- 172.21.15.81:0/1692821269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740d0b90 0x7f6b740d1a80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009830 tx=0x7f6b6802a640 comp rx=0 tx=0).stop 2024-06-29T14:23:38.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.074+0000 7f6b788a4640 1 -- 172.21.15.81:0/1692821269 shutdown_connections 2024-06-29T14:23:38.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.074+0000 7f6b788a4640 1 --2- 172.21.15.81:0/1692821269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740d0b90 0x7f6b740d1a80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.074+0000 7f6b788a4640 1 -- 172.21.15.81:0/1692821269 >> 172.21.15.81:0/1692821269 conn(0x7f6b7409cf30 msgr2=0x7f6b740d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:38.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.074+0000 7f6b788a4640 1 -- 172.21.15.81:0/1692821269 shutdown_connections 2024-06-29T14:23:38.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.074+0000 7f6b788a4640 1 -- 172.21.15.81:0/1692821269 wait complete. 2024-06-29T14:23:38.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.075+0000 7f6b788a4640 1 Processor -- start 2024-06-29T14:23:38.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.076+0000 7f6b788a4640 1 -- start start 2024-06-29T14:23:38.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b788a4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740b5250 0x7f6b74158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b788a4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740b9040 0x7f6b74158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b788a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b74156c70 0x7f6b7415b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b788a4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6b740d26d0 con 0x7f6b74156c70 2024-06-29T14:23:38.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b788a4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6b740b85c0 con 0x7f6b740b9040 2024-06-29T14:23:38.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b788a4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6b740d2850 con 0x7f6b740b5250 2024-06-29T14:23:38.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b72d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b74156c70 0x7f6b7415b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b72d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b74156c70 0x7f6b7415b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60794/0 (socket says 172.21.15.81:60794) 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b72d9d640 1 -- 172.21.15.81:0/1199328444 learned_addr learned my addr 172.21.15.81:0/1199328444 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b7259c640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740b5250 0x7f6b74158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b71d9b640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740b9040 0x7f6b74158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b72d9d640 1 -- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740b5250 msgr2=0x7f6b74158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b72d9d640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740b5250 0x7f6b74158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b72d9d640 1 -- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740b9040 msgr2=0x7f6b74158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b72d9d640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740b9040 0x7f6b74158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b72d9d640 1 -- 172.21.15.81:0/1199328444 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b74157410 con 0x7f6b74156c70 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.077+0000 7f6b72d9d640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b74156c70 0x7f6b7415b350 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00caa0 tx=0x7f6b6c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.078+0000 7f6b62ffd640 1 -- 172.21.15.81:0/1199328444 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c016020 con 0x7f6b74156c70 2024-06-29T14:23:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.078+0000 7f6b62ffd640 1 -- 172.21.15.81:0/1199328444 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b6c00acf0 con 0x7f6b74156c70 2024-06-29T14:23:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.078+0000 7f6b788a4640 1 -- 172.21.15.81:0/1199328444 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7415a870 con 0x7f6b74156c70 2024-06-29T14:23:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.078+0000 7f6b62ffd640 1 -- 172.21.15.81:0/1199328444 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c012680 con 0x7f6b74156c70 2024-06-29T14:23:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.078+0000 7f6b788a4640 1 -- 172.21.15.81:0/1199328444 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b7415ae30 con 0x7f6b74156c70 2024-06-29T14:23:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.079+0000 7f6b788a4640 1 -- 172.21.15.81:0/1199328444 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b74157180 con 0x7f6b74156c70 2024-06-29T14:23:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.079+0000 7f6b62ffd640 1 -- 172.21.15.81:0/1199328444 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6b6c019020 con 0x7f6b74156c70 2024-06-29T14:23:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.080+0000 7f6b62ffd640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6b540762d0 0x7f6b540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.080+0000 7f6b62ffd640 1 -- 172.21.15.81:0/1199328444 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f6b6c097c10 con 0x7f6b74156c70 2024-06-29T14:23:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.080+0000 7f6b7259c640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6b540762d0 0x7f6b540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.080+0000 7f6b7259c640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6b540762d0 0x7f6b540787e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6b64000c00 tx=0x7f6b64009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.084+0000 7f6b62ffd640 1 -- 172.21.15.81:0/1199328444 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6b6c020020 con 0x7f6b74156c70 2024-06-29T14:23:38.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.267+0000 7f6b788a4640 1 -- 172.21.15.81:0/1199328444 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b740d2de0 con 0x7f6b74156c70 2024-06-29T14:23:38.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.268+0000 7f6b62ffd640 1 -- 172.21.15.81:0/1199328444 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16670 (secure 0 0 0) 0x7f6b6c060fb0 con 0x7f6b74156c70 2024-06-29T14:23:38.267 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:38.268 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":32,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:23:37.343239+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:34.937915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:36.258024+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:38.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 -- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6b540762d0 msgr2=0x7f6b540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6b540762d0 0x7f6b540787e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6b64000c00 tx=0x7f6b64009040 comp rx=0 tx=0).stop 2024-06-29T14:23:38.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 -- 172.21.15.81:0/1199328444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b74156c70 msgr2=0x7f6b7415b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b74156c70 0x7f6b7415b350 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00caa0 tx=0x7f6b6c00cf60 comp rx=0 tx=0).stop 2024-06-29T14:23:38.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 -- 172.21.15.81:0/1199328444 shutdown_connections 2024-06-29T14:23:38.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6b540762d0 0x7f6b540787e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b74156c70 0x7f6b7415b350 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740b9040 0x7f6b74158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 --2- 172.21.15.81:0/1199328444 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740b5250 0x7f6b74158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 -- 172.21.15.81:0/1199328444 >> 172.21.15.81:0/1199328444 conn(0x7f6b7409cf30 msgr2=0x7f6b740b7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:38.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 -- 172.21.15.81:0/1199328444 shutdown_connections 2024-06-29T14:23:38.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.270+0000 7f6b788a4640 1 -- 172.21.15.81:0/1199328444 wait complete. 2024-06-29T14:23:38.281 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:38.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.394+0000 7f2beeec0640 1 -- 172.21.15.81:0/1805915781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2be80ce770 msgr2=0x7f2be80b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.394+0000 7f2beeec0640 1 --2- 172.21.15.81:0/1805915781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2be80ce770 0x7f2be80b35c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2be4005fd0 tx=0x7f2be402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.394+0000 7f2beeec0640 1 -- 172.21.15.81:0/1805915781 shutdown_connections 2024-06-29T14:23:38.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.394+0000 7f2beeec0640 1 --2- 172.21.15.81:0/1805915781 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2be80d2950 0x7f2be80b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.394+0000 7f2beeec0640 1 --2- 172.21.15.81:0/1805915781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2be80ce770 0x7f2be80b35c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.394+0000 7f2beeec0640 1 -- 172.21.15.81:0/1805915781 >> 172.21.15.81:0/1805915781 conn(0x7f2be80d3c60 msgr2=0x7f2be80d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:38.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.394+0000 7f2beeec0640 1 -- 172.21.15.81:0/1805915781 shutdown_connections 2024-06-29T14:23:38.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.394+0000 7f2beeec0640 1 -- 172.21.15.81:0/1805915781 wait complete. 2024-06-29T14:23:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.396+0000 7f2beeec0640 1 Processor -- start 2024-06-29T14:23:38.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.396+0000 7f2beeec0640 1 -- start start 2024-06-29T14:23:38.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2beeec0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2be80ce770 0x7f2be80c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2beeec0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2be80ceb80 0x7f2be80c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2beeec0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2be80d2950 0x7f2be815f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2beeec0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2be80d15a0 con 0x7f2be80d2950 2024-06-29T14:23:38.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2beeec0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2be80d1420 con 0x7f2be80ceb80 2024-06-29T14:23:38.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2beeec0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2be80d4990 con 0x7f2be80ce770 2024-06-29T14:23:38.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bed45d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2be80d2950 0x7f2be815f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bed45d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2be80d2950 0x7f2be815f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60814/0 (socket says 172.21.15.81:60814) 2024-06-29T14:23:38.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bed45d640 1 -- 172.21.15.81:0/7616788 learned_addr learned my addr 172.21.15.81:0/7616788 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bdffff640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2be80ceb80 0x7f2be80c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2becc5c640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2be80ce770 0x7f2be80c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bed45d640 1 -- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2be80ce770 msgr2=0x7f2be80c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bed45d640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2be80ce770 0x7f2be80c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bed45d640 1 -- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2be80ceb80 msgr2=0x7f2be80c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bed45d640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2be80ceb80 0x7f2be80c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bed45d640 1 -- 172.21.15.81:0/7616788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be80b7060 con 0x7f2be80d2950 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bed45d640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2be80d2950 0x7f2be815f960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2be000da10 tx=0x7f2be000ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bdd7fa640 1 -- 172.21.15.81:0/7616788 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0017070 con 0x7f2be80d2950 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bdd7fa640 1 -- 172.21.15.81:0/7616788 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2be0003970 con 0x7f2be80d2950 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2beeec0640 1 -- 172.21.15.81:0/7616788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be80ca090 con 0x7f2be80d2950 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2bdd7fa640 1 -- 172.21.15.81:0/7616788 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be000be60 con 0x7f2be80d2950 2024-06-29T14:23:38.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.398+0000 7f2beeec0640 1 -- 172.21.15.81:0/7616788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be80b75e0 con 0x7f2be80d2950 2024-06-29T14:23:38.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.400+0000 7f2beeec0640 1 -- 172.21.15.81:0/7616788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be80d2240 con 0x7f2be80d2950 2024-06-29T14:23:38.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.400+0000 7f2bdd7fa640 1 -- 172.21.15.81:0/7616788 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2be0010460 con 0x7f2be80d2950 2024-06-29T14:23:38.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.401+0000 7f2bdd7fa640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2bd4076300 0x7f2bd4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.401+0000 7f2bdd7fa640 1 -- 172.21.15.81:0/7616788 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f2be0097a30 con 0x7f2be80d2950 2024-06-29T14:23:38.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.401+0000 7f2becc5c640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2bd4076300 0x7f2bd4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.401+0000 7f2becc5c640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2bd4076300 0x7f2bd4078810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2be4033000 tx=0x7f2be4004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.405+0000 7f2bdd7fa640 1 -- 172.21.15.81:0/7616788 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2be0060d40 con 0x7f2be80d2950 2024-06-29T14:23:38.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.587+0000 7f2beeec0640 1 -- 172.21.15.81:0/7616788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2be80c9e20 con 0x7f2be80d2950 2024-06-29T14:23:38.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.588+0000 7f2bdd7fa640 1 -- 172.21.15.81:0/7616788 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16670 (secure 0 0 0) 0x7f2be0065040 con 0x7f2be80d2950 2024-06-29T14:23:38.587 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:38.588 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":32,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:23:37.343239+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:34.937915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:36.258024+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.590+0000 7f2beeec0640 1 -- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2bd4076300 msgr2=0x7f2bd4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.590+0000 7f2beeec0640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2bd4076300 0x7f2bd4078810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2be4033000 tx=0x7f2be4004730 comp rx=0 tx=0).stop 2024-06-29T14:23:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.590+0000 7f2beeec0640 1 -- 172.21.15.81:0/7616788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2be80d2950 msgr2=0x7f2be815f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.590+0000 7f2beeec0640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2be80d2950 0x7f2be815f960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2be000da10 tx=0x7f2be000ded0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.591+0000 7f2beeec0640 1 -- 172.21.15.81:0/7616788 shutdown_connections 2024-06-29T14:23:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.591+0000 7f2beeec0640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2bd4076300 0x7f2bd4078810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.591+0000 7f2beeec0640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2be80d2950 0x7f2be815f960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.591+0000 7f2beeec0640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2be80ceb80 0x7f2be80c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.591+0000 7f2beeec0640 1 --2- 172.21.15.81:0/7616788 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2be80ce770 0x7f2be80c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.591+0000 7f2beeec0640 1 -- 172.21.15.81:0/7616788 >> 172.21.15.81:0/7616788 conn(0x7f2be80d3c60 msgr2=0x7f2be815eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.591+0000 7f2beeec0640 1 -- 172.21.15.81:0/7616788 shutdown_connections 2024-06-29T14:23:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.591+0000 7f2beeec0640 1 -- 172.21.15.81:0/7616788 wait complete. 2024-06-29T14:23:38.601 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:23:38.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.721+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/3807849031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e90003a00 msgr2=0x7f6e90003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.721+0000 7f6eaeaa4640 1 --2- 172.21.15.81:0/3807849031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e90003a00 0x7f6e90003e70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400c2e0 tx=0x7f6e90005830 comp rx=0 tx=0).stop 2024-06-29T14:23:38.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.721+0000 7f6ead041640 1 -- 172.21.15.81:0/3807849031 reap_dead start 2024-06-29T14:23:38.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.721+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/3807849031 shutdown_connections 2024-06-29T14:23:38.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.721+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/3807849031 >> 172.21.15.81:0/3807849031 conn(0x7f6ea809cf30 msgr2=0x7f6ea80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:38.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.721+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/3807849031 shutdown_connections 2024-06-29T14:23:38.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.721+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/3807849031 wait complete. 2024-06-29T14:23:38.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.723+0000 7f6eaeaa4640 1 Processor -- start 2024-06-29T14:23:38.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.723+0000 7f6eaeaa4640 1 -- start start 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6eaeaa4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e90003a00 0x7f6ea80a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6eaeaa4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ea809e300 0x7f6ea815c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6eaeaa4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ea80cc1c0 0x7f6ea80cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6eaeaa4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea80d1fc0 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6eaeaa4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea80b8500 con 0x7f6e90003a00 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6eaeaa4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea80d2140 con 0x7f6ea809e300 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6ead041640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ea80cc1c0 0x7f6ea80cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6ead041640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ea80cc1c0 0x7f6ea80cc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60836/0 (socket says 172.21.15.81:60836) 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6ead041640 1 -- 172.21.15.81:0/1919759101 learned_addr learned my addr 172.21.15.81:0/1919759101 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6eac840640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e90003a00 0x7f6ea80a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.724+0000 7f6e9ffff640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ea809e300 0x7f6ea815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6ead041640 1 -- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ea809e300 msgr2=0x7f6ea815c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6ead041640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ea809e300 0x7f6ea815c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6ead041640 1 -- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e90003a00 msgr2=0x7f6ea80a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:38.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6ead041640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e90003a00 0x7f6ea80a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:38.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6ead041640 1 -- 172.21.15.81:0/1919759101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea80c87c0 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6e9ffff640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ea809e300 0x7f6ea815c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:38.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6eac840640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e90003a00 0x7f6ea80a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:38.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6ead041640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ea80cc1c0 0x7f6ea80cc5d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0005fd0 tx=0x7f6ea0011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6e9d7fa640 1 -- 172.21.15.81:0/1919759101 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea000fd70 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6e9d7fa640 1 -- 172.21.15.81:0/1919759101 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ea0003970 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/1919759101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea80ccb10 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6e9d7fa640 1 -- 172.21.15.81:0/1919759101 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0003dc0 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.725+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/1919759101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea80a1180 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.726+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/1919759101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea80ce540 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.727+0000 7f6e9d7fa640 1 -- 172.21.15.81:0/1919759101 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6ea0003f60 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.727+0000 7f6e9d7fa640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6e940762d0 0x7f6e940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:38.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.727+0000 7f6e9d7fa640 1 -- 172.21.15.81:0/1919759101 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f6ea00982c0 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.727+0000 7f6eac840640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6e940762d0 0x7f6e940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:38.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.728+0000 7f6eac840640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6e940762d0 0x7f6e940787e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4033000 tx=0x7f6ea402d070 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:38.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.731+0000 7f6e9d7fa640 1 -- 172.21.15.81:0/1919759101 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6ea80ce540 con 0x7f6ea80cc1c0 2024-06-29T14:23:38.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:38.949+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/1919759101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6ea80c8b50 con 0x7f6ea80cc1c0 2024-06-29T14:23:39.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.950+0000 7f6e9d7fa640 1 -- 172.21.15.81:0/1919759101 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f6ea80ccb10 con 0x7f6ea80cc1c0 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.953+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6e940762d0 msgr2=0x7f6e940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.953+0000 7f6eaeaa4640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6e940762d0 0x7f6e940787e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4033000 tx=0x7f6ea402d070 comp rx=0 tx=0).stop 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.953+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/1919759101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ea80cc1c0 msgr2=0x7f6ea80cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.953+0000 7f6eaeaa4640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ea80cc1c0 0x7f6ea80cc5d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0005fd0 tx=0x7f6ea0011f90 comp rx=0 tx=0).stop 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.954+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/1919759101 shutdown_connections 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.954+0000 7f6eaeaa4640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f6e940762d0 0x7f6e940787e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.954+0000 7f6eaeaa4640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ea80cc1c0 0x7f6ea80cc5d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.954+0000 7f6eaeaa4640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ea809e300 0x7f6ea815c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.954+0000 7f6eaeaa4640 1 --2- 172.21.15.81:0/1919759101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e90003a00 0x7f6ea80a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.954+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/1919759101 >> 172.21.15.81:0/1919759101 conn(0x7f6ea809cf30 msgr2=0x7f6ea81613c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.954+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/1919759101 shutdown_connections 2024-06-29T14:23:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:39.954+0000 7f6eaeaa4640 1 -- 172.21.15.81:0/1919759101 wait complete. 2024-06-29T14:23:39.967 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:23:39.967 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:23:40.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.091+0000 7f3262e17640 1 -- 172.21.15.81:0/1572549486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c145530 msgr2=0x7f325c073280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.091+0000 7f3262e17640 1 --2- 172.21.15.81:0/1572549486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c145530 0x7f325c073280 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3258009830 tx=0x7f325802a510 comp rx=0 tx=0).stop 2024-06-29T14:23:40.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.091+0000 7f3262e17640 1 -- 172.21.15.81:0/1572549486 shutdown_connections 2024-06-29T14:23:40.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.091+0000 7f3262e17640 1 --2- 172.21.15.81:0/1572549486 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f325c15cd00 0x7f325c15d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.091+0000 7f3262e17640 1 --2- 172.21.15.81:0/1572549486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c145530 0x7f325c073280 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.091+0000 7f3262e17640 1 -- 172.21.15.81:0/1572549486 >> 172.21.15.81:0/1572549486 conn(0x7f325c06ccc0 msgr2=0x7f325c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:40.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.091+0000 7f3262e17640 1 -- 172.21.15.81:0/1572549486 shutdown_connections 2024-06-29T14:23:40.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.091+0000 7f3262e17640 1 -- 172.21.15.81:0/1572549486 wait complete. 2024-06-29T14:23:40.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.093+0000 7f3262e17640 1 Processor -- start 2024-06-29T14:23:40.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.093+0000 7f3262e17640 1 -- start start 2024-06-29T14:23:40.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.094+0000 7f3262e17640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f325c0737c0 0x7f325c1f3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.094+0000 7f3262e17640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c145530 0x7f325c13c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.094+0000 7f3262e17640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f325c15cd00 0x7f325c158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.094+0000 7f3262e17640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f325c15dde0 con 0x7f325c145530 2024-06-29T14:23:40.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.094+0000 7f3262e17640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f325c15dc60 con 0x7f325c15cd00 2024-06-29T14:23:40.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.094+0000 7f3262e17640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f325c15df60 con 0x7f325c0737c0 2024-06-29T14:23:40.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.094+0000 7f3253fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c145530 0x7f325c13c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.094+0000 7f3253fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c145530 0x7f325c13c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42042/0 (socket says 172.21.15.81:42042) 2024-06-29T14:23:40.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.094+0000 7f3253fff640 1 -- 172.21.15.81:0/3131437859 learned_addr learned my addr 172.21.15.81:0/3131437859 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f32613b4640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f325c15cd00 0x7f325c158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f3260bb3640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f325c0737c0 0x7f325c1f3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f3253fff640 1 -- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f325c0737c0 msgr2=0x7f325c1f3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f3253fff640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f325c0737c0 0x7f325c1f3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f3253fff640 1 -- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f325c15cd00 msgr2=0x7f325c158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f3253fff640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f325c15cd00 0x7f325c158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f3253fff640 1 -- 172.21.15.81:0/3131437859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325c13c9e0 con 0x7f325c145530 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f3260bb3640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f325c0737c0 0x7f325c1f3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f3253fff640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c145530 0x7f325c13c2c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f324c00c3f0 tx=0x7f324c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f32517fa640 1 -- 172.21.15.81:0/3131437859 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c017070 con 0x7f325c145530 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f32517fa640 1 -- 172.21.15.81:0/3131437859 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f324c0035f0 con 0x7f325c145530 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f3262e17640 1 -- 172.21.15.81:0/3131437859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f325c1f3b20 con 0x7f325c145530 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f32517fa640 1 -- 172.21.15.81:0/3131437859 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c003a40 con 0x7f325c145530 2024-06-29T14:23:40.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.095+0000 7f3262e17640 1 -- 172.21.15.81:0/3131437859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f325c1f40e0 con 0x7f325c145530 2024-06-29T14:23:40.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.096+0000 7f3262e17640 1 -- 172.21.15.81:0/3131437859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f325c15e4f0 con 0x7f325c145530 2024-06-29T14:23:40.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.097+0000 7f32517fa640 1 -- 172.21.15.81:0/3131437859 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f324c01a070 con 0x7f325c145530 2024-06-29T14:23:40.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.097+0000 7f32517fa640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f32480762d0 0x7f32480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:40.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.097+0000 7f32517fa640 1 -- 172.21.15.81:0/3131437859 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f324c096b00 con 0x7f325c145530 2024-06-29T14:23:40.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.097+0000 7f3260bb3640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f32480762d0 0x7f32480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:40.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.098+0000 7f3260bb3640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f32480762d0 0x7f32480787e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3258034000 tx=0x7f3258004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:40.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.101+0000 7f32517fa640 1 -- 172.21.15.81:0/3131437859 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f325c073bd0 con 0x7f325c145530 2024-06-29T14:23:40.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.323+0000 7f3262e17640 1 -- 172.21.15.81:0/3131437859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f325c155b60 con 0x7f325c145530 2024-06-29T14:23:40.891 INFO:tasks.ceph.mds.a.smithi081.stderr:2024-06-29T14:23:40.892+0000 7f25720bd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:23:40.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.956+0000 7f32517fa640 1 -- 172.21.15.81:0/3131437859 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f324c0603d0 con 0x7f325c145530 2024-06-29T14:23:40.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 -- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f32480762d0 msgr2=0x7f32480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f32480762d0 0x7f32480787e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3258034000 tx=0x7f3258004730 comp rx=0 tx=0).stop 2024-06-29T14:23:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 -- 172.21.15.81:0/3131437859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c145530 msgr2=0x7f325c13c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c145530 0x7f325c13c2c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f324c00c3f0 tx=0x7f324c00c8b0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 -- 172.21.15.81:0/3131437859 shutdown_connections 2024-06-29T14:23:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f32480762d0 0x7f32480787e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f325c15cd00 0x7f325c158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c145530 0x7f325c13c2c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 --2- 172.21.15.81:0/3131437859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f325c0737c0 0x7f325c1f3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 -- 172.21.15.81:0/3131437859 >> 172.21.15.81:0/3131437859 conn(0x7f325c06ccc0 msgr2=0x7f325c15f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 -- 172.21.15.81:0/3131437859 shutdown_connections 2024-06-29T14:23:40.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:40.959+0000 7f3262e17640 1 -- 172.21.15.81:0/3131437859 wait complete. 2024-06-29T14:23:40.971 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:23:40.971 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:23:41.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.090+0000 7f39d5d03640 1 -- 172.21.15.81:0/1123743123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39bc004800 msgr2=0x7f39bc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.090+0000 7f39d5d03640 1 --2- 172.21.15.81:0/1123743123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39bc004800 0x7f39bc004c10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f39b80016c0 tx=0x7f39bc005830 comp rx=0 tx=0).stop 2024-06-29T14:23:41.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.090+0000 7f39d5d03640 1 -- 172.21.15.81:0/1123743123 shutdown_connections 2024-06-29T14:23:41.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.090+0000 7f39d5d03640 1 --2- 172.21.15.81:0/1123743123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39bc004800 0x7f39bc004c10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.090+0000 7f39d5d03640 1 -- 172.21.15.81:0/1123743123 >> 172.21.15.81:0/1123743123 conn(0x7f39d00d0930 msgr2=0x7f39d00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.090+0000 7f39d5d03640 1 -- 172.21.15.81:0/1123743123 shutdown_connections 2024-06-29T14:23:41.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.090+0000 7f39d5d03640 1 -- 172.21.15.81:0/1123743123 wait complete. 2024-06-29T14:23:41.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.092+0000 7f39d5d03640 1 Processor -- start 2024-06-29T14:23:41.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.092+0000 7f39d5d03640 1 -- start start 2024-06-29T14:23:41.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39d5d03640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39bc004800 0x7f39d015ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39d5d03640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39d0150730 0x7f39d0150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39d5d03640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39d00ce5f0 0x7f39d00cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39d5d03640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39d006f1a0 con 0x7f39d0150730 2024-06-29T14:23:41.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39d5d03640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f39d00afe10 con 0x7f39d00ce5f0 2024-06-29T14:23:41.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39d5d03640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f39d006f320 con 0x7f39bc004800 2024-06-29T14:23:41.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39c7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39d0150730 0x7f39d0150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39c7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39d0150730 0x7f39d0150b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42054/0 (socket says 172.21.15.81:42054) 2024-06-29T14:23:41.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39c7fff640 1 -- 172.21.15.81:0/4263261678 learned_addr learned my addr 172.21.15.81:0/4263261678 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:41.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39cf7fe640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39bc004800 0x7f39d015ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39cffff640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39d00ce5f0 0x7f39d00cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39c7fff640 1 -- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39bc004800 msgr2=0x7f39d015ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39c7fff640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39bc004800 0x7f39d015ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39c7fff640 1 -- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39d00ce5f0 msgr2=0x7f39d00cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39c7fff640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39d00ce5f0 0x7f39d00cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.093+0000 7f39c7fff640 1 -- 172.21.15.81:0/4263261678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d00af310 con 0x7f39d0150730 2024-06-29T14:23:41.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.094+0000 7f39c7fff640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39d0150730 0x7f39d0150b40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f39b802a8e0 tx=0x7f39b80063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.094+0000 7f39ccff9640 1 -- 172.21.15.81:0/4263261678 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8003a00 con 0x7f39d0150730 2024-06-29T14:23:41.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.094+0000 7f39ccff9640 1 -- 172.21.15.81:0/4263261678 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39b8039070 con 0x7f39d0150730 2024-06-29T14:23:41.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.094+0000 7f39d5d03640 1 -- 172.21.15.81:0/4263261678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d014f690 con 0x7f39d0150730 2024-06-29T14:23:41.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.094+0000 7f39ccff9640 1 -- 172.21.15.81:0/4263261678 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8033420 con 0x7f39d0150730 2024-06-29T14:23:41.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.094+0000 7f39d5d03640 1 -- 172.21.15.81:0/4263261678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d014fc50 con 0x7f39d0150730 2024-06-29T14:23:41.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.095+0000 7f39d5d03640 1 -- 172.21.15.81:0/4263261678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d006e2a0 con 0x7f39d0150730 2024-06-29T14:23:41.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.096+0000 7f39ccff9640 1 -- 172.21.15.81:0/4263261678 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f39b8044020 con 0x7f39d0150730 2024-06-29T14:23:41.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.096+0000 7f39ccff9640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39b00762d0 0x7f39b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.096+0000 7f39ccff9640 1 -- 172.21.15.81:0/4263261678 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f39b80b70b0 con 0x7f39d0150730 2024-06-29T14:23:41.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.096+0000 7f39cf7fe640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39b00762d0 0x7f39b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.097+0000 7f39cf7fe640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39b00762d0 0x7f39b00787e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f39c0006cb0 tx=0x7f39c00067a0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.100+0000 7f39ccff9640 1 -- 172.21.15.81:0/4263261678 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f39d006e2a0 con 0x7f39d0150730 2024-06-29T14:23:41.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.317+0000 7f39d5d03640 1 -- 172.21.15.81:0/4263261678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39d00ce3e0 con 0x7f39d0150730 2024-06-29T14:23:41.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.317+0000 7f39ccff9640 1 -- 172.21.15.81:0/4263261678 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5220 (secure 0 0 0) 0x7f39b8080a40 con 0x7f39d0150730 2024-06-29T14:23:41.317 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:41.317 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"btime":"2024-06-29T14:23:40:952859+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3513859413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3513859413},{"type":"v1","addr":"172.21.15.81:6835","nonce":3513859413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4775,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/763942647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":763942647},{"type":"v1","addr":"172.21.15.167:6836","nonce":763942647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1148290480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1148290480},{"type":"v1","addr":"172.21.15.167:6837","nonce":1148290480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:37.343197+0000","modified":"2024-06-29T14:23:40.325922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5172":{"gid":5172,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/58595811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":58595811},{"type":"v1","addr":"172.21.15.81:6834","nonce":58595811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5172,"qdb_cluster":[5172]},"id":2}]} 2024-06-29T14:23:41.317 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 16 2024-06-29T14:23:41.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.319+0000 7f39d5d03640 1 -- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39b00762d0 msgr2=0x7f39b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.319+0000 7f39d5d03640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39b00762d0 0x7f39b00787e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f39c0006cb0 tx=0x7f39c00067a0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.319+0000 7f39d5d03640 1 -- 172.21.15.81:0/4263261678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39d0150730 msgr2=0x7f39d0150b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.319+0000 7f39d5d03640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39d0150730 0x7f39d0150b40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f39b802a8e0 tx=0x7f39b80063a0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.319+0000 7f39d5d03640 1 -- 172.21.15.81:0/4263261678 shutdown_connections 2024-06-29T14:23:41.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.319+0000 7f39d5d03640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39b00762d0 0x7f39b00787e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.319+0000 7f39d5d03640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39d00ce5f0 0x7f39d00cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.319+0000 7f39d5d03640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39d0150730 0x7f39d0150b40 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.319+0000 7f39d5d03640 1 --2- 172.21.15.81:0/4263261678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39bc004800 0x7f39d015ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.319+0000 7f39d5d03640 1 -- 172.21.15.81:0/4263261678 >> 172.21.15.81:0/4263261678 conn(0x7f39d00d0930 msgr2=0x7f39d015a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.320+0000 7f39d5d03640 1 -- 172.21.15.81:0/4263261678 shutdown_connections 2024-06-29T14:23:41.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.320+0000 7f39d5d03640 1 -- 172.21.15.81:0/4263261678 wait complete. 2024-06-29T14:23:41.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:23:41.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.444+0000 7f2a3db7a640 1 -- 172.21.15.81:0/2179548089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a20004800 msgr2=0x7f2a20004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.444+0000 7f2a3db7a640 1 --2- 172.21.15.81:0/2179548089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a20004800 0x7f2a20004c10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00c310 tx=0x7f2a20005830 comp rx=0 tx=0).stop 2024-06-29T14:23:41.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.444+0000 7f2a3db7a640 1 -- 172.21.15.81:0/2179548089 shutdown_connections 2024-06-29T14:23:41.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.444+0000 7f2a3db7a640 1 --2- 172.21.15.81:0/2179548089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a20004800 0x7f2a20004c10 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.444+0000 7f2a3db7a640 1 -- 172.21.15.81:0/2179548089 >> 172.21.15.81:0/2179548089 conn(0x7f2a3806ccc0 msgr2=0x7f2a380b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.444+0000 7f2a3db7a640 1 -- 172.21.15.81:0/2179548089 shutdown_connections 2024-06-29T14:23:41.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.444+0000 7f2a3db7a640 1 -- 172.21.15.81:0/2179548089 wait complete. 2024-06-29T14:23:41.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.446+0000 7f2a3db7a640 1 Processor -- start 2024-06-29T14:23:41.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.446+0000 7f2a3db7a640 1 -- start start 2024-06-29T14:23:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a3db7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a20004800 0x7f2a3813ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a3db7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a3813c3e0 0x7f2a3813f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a3db7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a38071f20 0x7f2a38072330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a3db7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2a380731b0 con 0x7f2a3813c3e0 2024-06-29T14:23:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a3db7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2a38073030 con 0x7f2a38071f20 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a3db7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2a38073330 con 0x7f2a20004800 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a36ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a3813c3e0 0x7f2a3813f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a36ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a3813c3e0 0x7f2a3813f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42064/0 (socket says 172.21.15.81:42064) 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a36ffd640 1 -- 172.21.15.81:0/253334902 learned_addr learned my addr 172.21.15.81:0/253334902 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a377fe640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a20004800 0x7f2a3813ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a37fff640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a38071f20 0x7f2a38072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a36ffd640 1 -- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a20004800 msgr2=0x7f2a3813ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a36ffd640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a20004800 0x7f2a3813ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a36ffd640 1 -- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a38071f20 msgr2=0x7f2a38072330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a36ffd640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a38071f20 0x7f2a38072330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a36ffd640 1 -- 172.21.15.81:0/253334902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a38138b80 con 0x7f2a3813c3e0 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a37fff640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a38071f20 0x7f2a38072330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a36ffd640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a3813c3e0 0x7f2a3813f120 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c02b180 tx=0x7f2a2c001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a1ffff640 1 -- 172.21.15.81:0/253334902 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c031040 con 0x7f2a3813c3e0 2024-06-29T14:23:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a1ffff640 1 -- 172.21.15.81:0/253334902 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a2c0035f0 con 0x7f2a3813c3e0 2024-06-29T14:23:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a3db7a640 1 -- 172.21.15.81:0/253334902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a38138e60 con 0x7f2a3813c3e0 2024-06-29T14:23:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.448+0000 7f2a1ffff640 1 -- 172.21.15.81:0/253334902 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c003a20 con 0x7f2a3813c3e0 2024-06-29T14:23:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.449+0000 7f2a3db7a640 1 -- 172.21.15.81:0/253334902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a38139420 con 0x7f2a3813c3e0 2024-06-29T14:23:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.449+0000 7f2a3db7a640 1 -- 172.21.15.81:0/253334902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a38073890 con 0x7f2a3813c3e0 2024-06-29T14:23:41.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.450+0000 7f2a1ffff640 1 -- 172.21.15.81:0/253334902 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2a2c03e030 con 0x7f2a3813c3e0 2024-06-29T14:23:41.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.451+0000 7f2a1ffff640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a24076300 0x7f2a24078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.451+0000 7f2a1ffff640 1 -- 172.21.15.81:0/253334902 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f2a2c0b7350 con 0x7f2a3813c3e0 2024-06-29T14:23:41.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.451+0000 7f2a377fe640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a24076300 0x7f2a24078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.452+0000 7f2a377fe640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a24076300 0x7f2a24078810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009cd0 tx=0x7f2a28005f90 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.455+0000 7f2a1ffff640 1 -- 172.21.15.81:0/253334902 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2a2c080cb0 con 0x7f2a3813c3e0 2024-06-29T14:23:41.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.637+0000 7f2a3db7a640 1 -- 172.21.15.81:0/253334902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a38138910 con 0x7f2a3813c3e0 2024-06-29T14:23:41.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.638+0000 7f2a1ffff640 1 -- 172.21.15.81:0/253334902 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16657 (secure 0 0 0) 0x7f2a2c0805b0 con 0x7f2a3813c3e0 2024-06-29T14:23:41.637 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:41.637 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":33,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:23:38.922056+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:34.937915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:36.258024+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:23:41.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 -- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a24076300 msgr2=0x7f2a24078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a24076300 0x7f2a24078810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009cd0 tx=0x7f2a28005f90 comp rx=0 tx=0).stop 2024-06-29T14:23:41.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 -- 172.21.15.81:0/253334902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a3813c3e0 msgr2=0x7f2a3813f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a3813c3e0 0x7f2a3813f120 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c02b180 tx=0x7f2a2c001d90 comp rx=0 tx=0).stop 2024-06-29T14:23:41.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 -- 172.21.15.81:0/253334902 shutdown_connections 2024-06-29T14:23:41.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f2a24076300 0x7f2a24078810 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2a38071f20 0x7f2a38072330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a3813c3e0 0x7f2a3813f120 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 --2- 172.21.15.81:0/253334902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2a20004800 0x7f2a3813ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 -- 172.21.15.81:0/253334902 >> 172.21.15.81:0/253334902 conn(0x7f2a3806ccc0 msgr2=0x7f2a3815df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 -- 172.21.15.81:0/253334902 shutdown_connections 2024-06-29T14:23:41.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.640+0000 7f2a3db7a640 1 -- 172.21.15.81:0/253334902 wait complete. 2024-06-29T14:23:41.651 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:23:41.651 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:23:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.765+0000 7f813067f640 1 -- 172.21.15.81:0/3834661458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8114004d90 msgr2=0x7f8114005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.765+0000 7f813067f640 1 --2- 172.21.15.81:0/3834661458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8114004d90 0x7f8114005220 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8120005fd0 tx=0x7f8114005830 comp rx=0 tx=0).stop 2024-06-29T14:23:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.765+0000 7f812ec1c640 1 -- 172.21.15.81:0/3834661458 reap_dead start 2024-06-29T14:23:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.765+0000 7f813067f640 1 -- 172.21.15.81:0/3834661458 shutdown_connections 2024-06-29T14:23:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.765+0000 7f813067f640 1 -- 172.21.15.81:0/3834661458 >> 172.21.15.81:0/3834661458 conn(0x7f8128072950 msgr2=0x7f81280b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.765+0000 7f813067f640 1 -- 172.21.15.81:0/3834661458 shutdown_connections 2024-06-29T14:23:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.765+0000 7f813067f640 1 -- 172.21.15.81:0/3834661458 wait complete. 2024-06-29T14:23:41.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.767+0000 7f813067f640 1 Processor -- start 2024-06-29T14:23:41.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.767+0000 7f813067f640 1 -- start start 2024-06-29T14:23:41.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.768+0000 7f813067f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8114004d90 0x7f812809db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.768+0000 7f813067f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81281f5ab0 0x7f81280afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.768+0000 7f813067f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81280b1000 0x7f81280b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.768+0000 7f813067f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f812815b190 con 0x7f81281f5ab0 2024-06-29T14:23:41.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.768+0000 7f813067f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f812815b010 con 0x7f8114004d90 2024-06-29T14:23:41.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.768+0000 7f813067f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f81280aee20 con 0x7f81280b1000 2024-06-29T14:23:41.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812dc1a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81281f5ab0 0x7f81280afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812dc1a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81281f5ab0 0x7f81280afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42082/0 (socket says 172.21.15.81:42082) 2024-06-29T14:23:41.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812dc1a640 1 -- 172.21.15.81:0/3035901953 learned_addr learned my addr 172.21.15.81:0/3035901953 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812e41b640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8114004d90 0x7f812809db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812ec1c640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81280b1000 0x7f81280b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812dc1a640 1 -- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81280b1000 msgr2=0x7f81280b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812dc1a640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81280b1000 0x7f81280b0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812dc1a640 1 -- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8114004d90 msgr2=0x7f812809db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812dc1a640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8114004d90 0x7f812809db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812dc1a640 1 -- 172.21.15.81:0/3035901953 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81280ccb60 con 0x7f81281f5ab0 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812e41b640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8114004d90 0x7f812809db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f812dc1a640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81281f5ab0 0x7f81280afd50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8118004d90 tx=0x7f8118001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f811effd640 1 -- 172.21.15.81:0/3035901953 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81180107d0 con 0x7f81281f5ab0 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f811effd640 1 -- 172.21.15.81:0/3035901953 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8118010a80 con 0x7f81281f5ab0 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f813067f640 1 -- 172.21.15.81:0/3035901953 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81280cce40 con 0x7f81281f5ab0 2024-06-29T14:23:41.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f811effd640 1 -- 172.21.15.81:0/3035901953 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8118011460 con 0x7f81281f5ab0 2024-06-29T14:23:41.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.769+0000 7f813067f640 1 -- 172.21.15.81:0/3035901953 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81280cd400 con 0x7f81281f5ab0 2024-06-29T14:23:41.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.770+0000 7f813067f640 1 -- 172.21.15.81:0/3035901953 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8128068ef0 con 0x7f81281f5ab0 2024-06-29T14:23:41.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.771+0000 7f811effd640 1 -- 172.21.15.81:0/3035901953 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8118011600 con 0x7f81281f5ab0 2024-06-29T14:23:41.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.771+0000 7f811effd640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f810c0762d0 0x7f810c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:41.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.771+0000 7f811effd640 1 -- 172.21.15.81:0/3035901953 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f81180982c0 con 0x7f81281f5ab0 2024-06-29T14:23:41.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.772+0000 7f812e41b640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f810c0762d0 0x7f810c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:41.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.772+0000 7f812e41b640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f810c0762d0 0x7f810c0787e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8124000f60 tx=0x7f8124000ed0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:41.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.776+0000 7f811effd640 1 -- 172.21.15.81:0/3035901953 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8128068ef0 con 0x7f81281f5ab0 2024-06-29T14:23:41.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.953+0000 7f813067f640 1 -- 172.21.15.81:0/3035901953 --> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f81281f6020 con 0x7f810c0762d0 2024-06-29T14:23:41.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.958+0000 7f811effd640 1 -- 172.21.15.81:0/3035901953 <== mgr.4126 v2:172.21.15.167:6832/3101542440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251929 (secure 0 0 0) 0x7f81281f6020 con 0x7f810c0762d0 2024-06-29T14:23:41.959 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:41.961 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:23:41.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 -- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f810c0762d0 msgr2=0x7f810c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f810c0762d0 0x7f810c0787e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8124000f60 tx=0x7f8124000ed0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 -- 172.21.15.81:0/3035901953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81281f5ab0 msgr2=0x7f81280afd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81281f5ab0 0x7f81280afd50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8118004d90 tx=0x7f8118001800 comp rx=0 tx=0).stop 2024-06-29T14:23:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 -- 172.21.15.81:0/3035901953 shutdown_connections 2024-06-29T14:23:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f810c0762d0 0x7f810c0787e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81280b1000 0x7f81280b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81281f5ab0 0x7f81280afd50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 --2- 172.21.15.81:0/3035901953 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8114004d90 0x7f812809db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 -- 172.21.15.81:0/3035901953 >> 172.21.15.81:0/3035901953 conn(0x7f8128072950 msgr2=0x7f81280cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 -- 172.21.15.81:0/3035901953 shutdown_connections 2024-06-29T14:23:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:41.961+0000 7f813067f640 1 -- 172.21.15.81:0/3035901953 wait complete. 2024-06-29T14:23:41.971 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-06-29T14:23:40.717420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220284,"kb_used_data":4100,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754436,"statfs":{"total":773094113280,"available":772868542464,"internally_reserved":0,"allocated":4198400,"data_stored":2338480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.799275"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321872+0000","last_change":"2024-06-29T14:23:37.939170+0000","last_active":"2024-06-29T14:23:40.321872+0000","last_peered":"2024-06-29T14:23:40.321872+0000","last_clean":"2024-06-29T14:23:40.321872+0000","last_became_active":"2024-06-29T14:23:37.938877+0000","last_became_peered":"2024-06-29T14:23:37.938877+0000","last_unstale":"2024-06-29T14:23:40.321872+0000","last_undegraded":"2024-06-29T14:23:40.321872+0000","last_fullsized":"2024-06-29T14:23:40.321872+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:23.983579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180776+0000","last_change":"2024-06-29T14:23:37.945549+0000","last_active":"2024-06-29T14:23:40.180776+0000","last_peered":"2024-06-29T14:23:40.180776+0000","last_clean":"2024-06-29T14:23:40.180776+0000","last_became_active":"2024-06-29T14:23:37.945244+0000","last_became_peered":"2024-06-29T14:23:37.945244+0000","last_unstale":"2024-06-29T14:23:40.180776+0000","last_undegraded":"2024-06-29T14:23:40.180776+0000","last_fullsized":"2024-06-29T14:23:40.180776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:29.365340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925985+0000","last_change":"2024-06-29T14:23:37.951081+0000","last_active":"2024-06-29T14:23:38.925985+0000","last_peered":"2024-06-29T14:23:38.925985+0000","last_clean":"2024-06-29T14:23:38.925985+0000","last_became_active":"2024-06-29T14:23:37.950719+0000","last_became_peered":"2024-06-29T14:23:37.950719+0000","last_unstale":"2024-06-29T14:23:38.925985+0000","last_undegraded":"2024-06-29T14:23:38.925985+0000","last_fullsized":"2024-06-29T14:23:38.925985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:22.666511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926215+0000","last_change":"2024-06-29T14:23:37.952847+0000","last_active":"2024-06-29T14:23:38.926215+0000","last_peered":"2024-06-29T14:23:38.926215+0000","last_clean":"2024-06-29T14:23:38.926215+0000","last_became_active":"2024-06-29T14:23:37.947188+0000","last_became_peered":"2024-06-29T14:23:37.947188+0000","last_unstale":"2024-06-29T14:23:38.926215+0000","last_undegraded":"2024-06-29T14:23:38.926215+0000","last_fullsized":"2024-06-29T14:23:38.926215+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:50.182625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321144+0000","last_change":"2024-06-29T14:23:37.935785+0000","last_active":"2024-06-29T14:23:40.321144+0000","last_peered":"2024-06-29T14:23:40.321144+0000","last_clean":"2024-06-29T14:23:40.321144+0000","last_became_active":"2024-06-29T14:23:37.935450+0000","last_became_peered":"2024-06-29T14:23:37.935450+0000","last_unstale":"2024-06-29T14:23:40.321144+0000","last_undegraded":"2024-06-29T14:23:40.321144+0000","last_fullsized":"2024-06-29T14:23:40.321144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:56.208133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180354+0000","last_change":"2024-06-29T14:23:37.930376+0000","last_active":"2024-06-29T14:23:40.180354+0000","last_peered":"2024-06-29T14:23:40.180354+0000","last_clean":"2024-06-29T14:23:40.180354+0000","last_became_active":"2024-06-29T14:23:37.930071+0000","last_became_peered":"2024-06-29T14:23:37.930071+0000","last_unstale":"2024-06-29T14:23:40.180354+0000","last_undegraded":"2024-06-29T14:23:40.180354+0000","last_fullsized":"2024-06-29T14:23:40.180354+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:20.327712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392471+0000","last_change":"2024-06-29T14:23:37.945999+0000","last_active":"2024-06-29T14:23:39.392471+0000","last_peered":"2024-06-29T14:23:39.392471+0000","last_clean":"2024-06-29T14:23:39.392471+0000","last_became_active":"2024-06-29T14:23:37.945649+0000","last_became_peered":"2024-06-29T14:23:37.945649+0000","last_unstale":"2024-06-29T14:23:39.392471+0000","last_undegraded":"2024-06-29T14:23:39.392471+0000","last_fullsized":"2024-06-29T14:23:39.392471+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:57.496214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392174+0000","last_change":"2024-06-29T14:23:37.949553+0000","last_active":"2024-06-29T14:23:39.392174+0000","last_peered":"2024-06-29T14:23:39.392174+0000","last_clean":"2024-06-29T14:23:39.392174+0000","last_became_active":"2024-06-29T14:23:37.949285+0000","last_became_peered":"2024-06-29T14:23:37.949285+0000","last_unstale":"2024-06-29T14:23:39.392174+0000","last_undegraded":"2024-06-29T14:23:39.392174+0000","last_fullsized":"2024-06-29T14:23:39.392174+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:09.623791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.948729+0000","last_change":"2024-06-29T14:23:37.944195+0000","last_active":"2024-06-29T14:23:37.948729+0000","last_peered":"2024-06-29T14:23:37.948729+0000","last_clean":"2024-06-29T14:23:37.948729+0000","last_became_active":"2024-06-29T14:23:37.943886+0000","last_became_peered":"2024-06-29T14:23:37.943886+0000","last_unstale":"2024-06-29T14:23:37.948729+0000","last_undegraded":"2024-06-29T14:23:37.948729+0000","last_fullsized":"2024-06-29T14:23:37.948729+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:02.807604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925935+0000","last_change":"2024-06-29T14:23:37.952875+0000","last_active":"2024-06-29T14:23:38.925935+0000","last_peered":"2024-06-29T14:23:38.925935+0000","last_clean":"2024-06-29T14:23:38.925935+0000","last_became_active":"2024-06-29T14:23:37.952578+0000","last_became_peered":"2024-06-29T14:23:37.952578+0000","last_unstale":"2024-06-29T14:23:38.925935+0000","last_undegraded":"2024-06-29T14:23:38.925935+0000","last_fullsized":"2024-06-29T14:23:38.925935+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:36.800293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392299+0000","last_change":"2024-06-29T14:23:37.952052+0000","last_active":"2024-06-29T14:23:39.392299+0000","last_peered":"2024-06-29T14:23:39.392299+0000","last_clean":"2024-06-29T14:23:39.392299+0000","last_became_active":"2024-06-29T14:23:37.951818+0000","last_became_peered":"2024-06-29T14:23:37.951818+0000","last_unstale":"2024-06-29T14:23:39.392299+0000","last_undegraded":"2024-06-29T14:23:39.392299+0000","last_fullsized":"2024-06-29T14:23:39.392299+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:46.137017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926121+0000","last_change":"2024-06-29T14:23:37.943825+0000","last_active":"2024-06-29T14:23:38.926121+0000","last_peered":"2024-06-29T14:23:38.926121+0000","last_clean":"2024-06-29T14:23:38.926121+0000","last_became_active":"2024-06-29T14:23:37.943535+0000","last_became_peered":"2024-06-29T14:23:37.943535+0000","last_unstale":"2024-06-29T14:23:38.926121+0000","last_undegraded":"2024-06-29T14:23:38.926121+0000","last_fullsized":"2024-06-29T14:23:38.926121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:03.047121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321033+0000","last_change":"2024-06-29T14:23:37.940287+0000","last_active":"2024-06-29T14:23:40.321033+0000","last_peered":"2024-06-29T14:23:40.321033+0000","last_clean":"2024-06-29T14:23:40.321033+0000","last_became_active":"2024-06-29T14:23:37.939963+0000","last_became_peered":"2024-06-29T14:23:37.939963+0000","last_unstale":"2024-06-29T14:23:40.321033+0000","last_undegraded":"2024-06-29T14:23:40.321033+0000","last_fullsized":"2024-06-29T14:23:40.321033+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:34:32.407360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180491+0000","last_change":"2024-06-29T14:23:37.941372+0000","last_active":"2024-06-29T14:23:40.180491+0000","last_peered":"2024-06-29T14:23:40.180491+0000","last_clean":"2024-06-29T14:23:40.180491+0000","last_became_active":"2024-06-29T14:23:37.941042+0000","last_became_peered":"2024-06-29T14:23:37.941042+0000","last_unstale":"2024-06-29T14:23:40.180491+0000","last_undegraded":"2024-06-29T14:23:40.180491+0000","last_fullsized":"2024-06-29T14:23:40.180491+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:56.602359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925863+0000","last_change":"2024-06-29T14:23:37.937216+0000","last_active":"2024-06-29T14:23:38.925863+0000","last_peered":"2024-06-29T14:23:38.925863+0000","last_clean":"2024-06-29T14:23:38.925863+0000","last_became_active":"2024-06-29T14:23:37.936935+0000","last_became_peered":"2024-06-29T14:23:37.936935+0000","last_unstale":"2024-06-29T14:23:38.925863+0000","last_undegraded":"2024-06-29T14:23:38.925863+0000","last_fullsized":"2024-06-29T14:23:38.925863+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:50.182625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392695+0000","last_change":"2024-06-29T14:23:37.952896+0000","last_active":"2024-06-29T14:23:39.392695+0000","last_peered":"2024-06-29T14:23:39.392695+0000","last_clean":"2024-06-29T14:23:39.392695+0000","last_became_active":"2024-06-29T14:23:37.952598+0000","last_became_peered":"2024-06-29T14:23:37.952598+0000","last_unstale":"2024-06-29T14:23:39.392695+0000","last_undegraded":"2024-06-29T14:23:39.392695+0000","last_fullsized":"2024-06-29T14:23:39.392695+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:20.327712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392632+0000","last_change":"2024-06-29T14:23:37.946407+0000","last_active":"2024-06-29T14:23:39.392632+0000","last_peered":"2024-06-29T14:23:39.392632+0000","last_clean":"2024-06-29T14:23:39.392632+0000","last_became_active":"2024-06-29T14:23:37.946127+0000","last_became_peered":"2024-06-29T14:23:37.946127+0000","last_unstale":"2024-06-29T14:23:39.392632+0000","last_undegraded":"2024-06-29T14:23:39.392632+0000","last_fullsized":"2024-06-29T14:23:39.392632+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:09.669198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180776+0000","last_change":"2024-06-29T14:23:37.951230+0000","last_active":"2024-06-29T14:23:40.180776+0000","last_peered":"2024-06-29T14:23:40.180776+0000","last_clean":"2024-06-29T14:23:40.180776+0000","last_became_active":"2024-06-29T14:23:37.950896+0000","last_became_peered":"2024-06-29T14:23:37.950896+0000","last_unstale":"2024-06-29T14:23:40.180776+0000","last_undegraded":"2024-06-29T14:23:40.180776+0000","last_fullsized":"2024-06-29T14:23:40.180776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:16:09.568501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180530+0000","last_change":"2024-06-29T14:23:37.951396+0000","last_active":"2024-06-29T14:23:40.180530+0000","last_peered":"2024-06-29T14:23:40.180530+0000","last_clean":"2024-06-29T14:23:40.180530+0000","last_became_active":"2024-06-29T14:23:37.951075+0000","last_became_peered":"2024-06-29T14:23:37.951075+0000","last_unstale":"2024-06-29T14:23:40.180530+0000","last_undegraded":"2024-06-29T14:23:40.180530+0000","last_fullsized":"2024-06-29T14:23:40.180530+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:15.819961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321071+0000","last_change":"2024-06-29T14:23:37.937200+0000","last_active":"2024-06-29T14:23:40.321071+0000","last_peered":"2024-06-29T14:23:40.321071+0000","last_clean":"2024-06-29T14:23:40.321071+0000","last_became_active":"2024-06-29T14:23:37.936889+0000","last_became_peered":"2024-06-29T14:23:37.936889+0000","last_unstale":"2024-06-29T14:23:40.321071+0000","last_undegraded":"2024-06-29T14:23:40.321071+0000","last_fullsized":"2024-06-29T14:23:40.321071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:30.663631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.948999+0000","last_change":"2024-06-29T14:23:37.942469+0000","last_active":"2024-06-29T14:23:37.948999+0000","last_peered":"2024-06-29T14:23:37.948999+0000","last_clean":"2024-06-29T14:23:37.948999+0000","last_became_active":"2024-06-29T14:23:37.942098+0000","last_became_peered":"2024-06-29T14:23:37.942098+0000","last_unstale":"2024-06-29T14:23:37.948999+0000","last_undegraded":"2024-06-29T14:23:37.948999+0000","last_fullsized":"2024-06-29T14:23:37.948999+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:46:51.326780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.940374+0000","last_change":"2024-06-29T14:23:37.936123+0000","last_active":"2024-06-29T14:23:37.940374+0000","last_peered":"2024-06-29T14:23:37.940374+0000","last_clean":"2024-06-29T14:23:37.940374+0000","last_became_active":"2024-06-29T14:23:37.935779+0000","last_became_peered":"2024-06-29T14:23:37.935779+0000","last_unstale":"2024-06-29T14:23:37.940374+0000","last_undegraded":"2024-06-29T14:23:37.940374+0000","last_fullsized":"2024-06-29T14:23:37.940374+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:56.208133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925814+0000","last_change":"2024-06-29T14:23:37.946192+0000","last_active":"2024-06-29T14:23:38.925814+0000","last_peered":"2024-06-29T14:23:38.925814+0000","last_clean":"2024-06-29T14:23:38.925814+0000","last_became_active":"2024-06-29T14:23:37.945845+0000","last_became_peered":"2024-06-29T14:23:37.945845+0000","last_unstale":"2024-06-29T14:23:38.925814+0000","last_undegraded":"2024-06-29T14:23:38.925814+0000","last_fullsized":"2024-06-29T14:23:38.925814+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:34:12.050331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925906+0000","last_change":"2024-06-29T14:23:37.935755+0000","last_active":"2024-06-29T14:23:38.925906+0000","last_peered":"2024-06-29T14:23:38.925906+0000","last_clean":"2024-06-29T14:23:38.925906+0000","last_became_active":"2024-06-29T14:23:37.935459+0000","last_became_peered":"2024-06-29T14:23:37.935459+0000","last_unstale":"2024-06-29T14:23:38.925906+0000","last_undegraded":"2024-06-29T14:23:38.925906+0000","last_fullsized":"2024-06-29T14:23:38.925906+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:46.137017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926824+0000","last_change":"2024-06-29T14:23:37.948974+0000","last_active":"2024-06-29T14:23:38.926824+0000","last_peered":"2024-06-29T14:23:38.926824+0000","last_clean":"2024-06-29T14:23:38.926824+0000","last_became_active":"2024-06-29T14:23:37.948222+0000","last_became_peered":"2024-06-29T14:23:37.948222+0000","last_unstale":"2024-06-29T14:23:38.926824+0000","last_undegraded":"2024-06-29T14:23:38.926824+0000","last_fullsized":"2024-06-29T14:23:38.926824+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:22.666511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180867+0000","last_change":"2024-06-29T14:23:37.944560+0000","last_active":"2024-06-29T14:23:40.180867+0000","last_peered":"2024-06-29T14:23:40.180867+0000","last_clean":"2024-06-29T14:23:40.180867+0000","last_became_active":"2024-06-29T14:23:37.944265+0000","last_became_peered":"2024-06-29T14:23:37.944265+0000","last_unstale":"2024-06-29T14:23:40.180867+0000","last_undegraded":"2024-06-29T14:23:40.180867+0000","last_fullsized":"2024-06-29T14:23:40.180867+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:08:09.582695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926082+0000","last_change":"2024-06-29T14:23:37.948749+0000","last_active":"2024-06-29T14:23:38.926082+0000","last_peered":"2024-06-29T14:23:38.926082+0000","last_clean":"2024-06-29T14:23:38.926082+0000","last_became_active":"2024-06-29T14:23:37.948288+0000","last_became_peered":"2024-06-29T14:23:37.948288+0000","last_unstale":"2024-06-29T14:23:38.926082+0000","last_undegraded":"2024-06-29T14:23:38.926082+0000","last_fullsized":"2024-06-29T14:23:38.926082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:09.623791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926273+0000","last_change":"2024-06-29T14:23:37.939162+0000","last_active":"2024-06-29T14:23:38.926273+0000","last_peered":"2024-06-29T14:23:38.926273+0000","last_clean":"2024-06-29T14:23:38.926273+0000","last_became_active":"2024-06-29T14:23:37.938886+0000","last_became_peered":"2024-06-29T14:23:37.938886+0000","last_unstale":"2024-06-29T14:23:38.926273+0000","last_undegraded":"2024-06-29T14:23:38.926273+0000","last_fullsized":"2024-06-29T14:23:38.926273+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:09.669198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.942322+0000","last_change":"2024-06-29T14:23:37.936801+0000","last_active":"2024-06-29T14:23:37.942322+0000","last_peered":"2024-06-29T14:23:37.942322+0000","last_clean":"2024-06-29T14:23:37.942322+0000","last_became_active":"2024-06-29T14:23:37.936512+0000","last_became_peered":"2024-06-29T14:23:37.936512+0000","last_unstale":"2024-06-29T14:23:37.942322+0000","last_undegraded":"2024-06-29T14:23:37.942322+0000","last_fullsized":"2024-06-29T14:23:37.942322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:17.132446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392383+0000","last_change":"2024-06-29T14:23:37.946974+0000","last_active":"2024-06-29T14:23:39.392383+0000","last_peered":"2024-06-29T14:23:39.392383+0000","last_clean":"2024-06-29T14:23:39.392383+0000","last_became_active":"2024-06-29T14:23:37.946635+0000","last_became_peered":"2024-06-29T14:23:37.946635+0000","last_unstale":"2024-06-29T14:23:39.392383+0000","last_undegraded":"2024-06-29T14:23:39.392383+0000","last_fullsized":"2024-06-29T14:23:39.392383+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:03.047121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.322024+0000","last_change":"2024-06-29T14:23:37.949738+0000","last_active":"2024-06-29T14:23:40.322024+0000","last_peered":"2024-06-29T14:23:40.322024+0000","last_clean":"2024-06-29T14:23:40.322024+0000","last_became_active":"2024-06-29T14:23:37.949439+0000","last_became_peered":"2024-06-29T14:23:37.949439+0000","last_unstale":"2024-06-29T14:23:40.322024+0000","last_undegraded":"2024-06-29T14:23:40.322024+0000","last_fullsized":"2024-06-29T14:23:40.322024+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:34:12.050331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392349+0000","last_change":"2024-06-29T14:23:37.944346+0000","last_active":"2024-06-29T14:23:39.392349+0000","last_peered":"2024-06-29T14:23:39.392349+0000","last_clean":"2024-06-29T14:23:39.392349+0000","last_became_active":"2024-06-29T14:23:37.943733+0000","last_became_peered":"2024-06-29T14:23:37.943733+0000","last_unstale":"2024-06-29T14:23:39.392349+0000","last_undegraded":"2024-06-29T14:23:39.392349+0000","last_fullsized":"2024-06-29T14:23:39.392349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:05.123440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926144+0000","last_change":"2024-06-29T14:23:37.939593+0000","last_active":"2024-06-29T14:23:38.926144+0000","last_peered":"2024-06-29T14:23:38.926144+0000","last_clean":"2024-06-29T14:23:38.926144+0000","last_became_active":"2024-06-29T14:23:37.939247+0000","last_became_peered":"2024-06-29T14:23:37.939247+0000","last_unstale":"2024-06-29T14:23:38.926144+0000","last_undegraded":"2024-06-29T14:23:38.926144+0000","last_fullsized":"2024-06-29T14:23:38.926144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:41.926725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.322107+0000","last_change":"2024-06-29T14:23:36.939194+0000","last_active":"2024-06-29T14:23:40.322107+0000","last_peered":"2024-06-29T14:23:40.322107+0000","last_clean":"2024-06-29T14:23:40.322107+0000","last_became_active":"2024-06-29T14:23:36.938887+0000","last_became_peered":"2024-06-29T14:23:36.938887+0000","last_unstale":"2024-06-29T14:23:40.322107+0000","last_undegraded":"2024-06-29T14:23:40.322107+0000","last_fullsized":"2024-06-29T14:23:40.322107+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:47.733035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.322079+0000","last_change":"2024-06-29T14:23:37.941082+0000","last_active":"2024-06-29T14:23:40.322079+0000","last_peered":"2024-06-29T14:23:40.322079+0000","last_clean":"2024-06-29T14:23:40.322079+0000","last_became_active":"2024-06-29T14:23:37.940795+0000","last_became_peered":"2024-06-29T14:23:37.940795+0000","last_unstale":"2024-06-29T14:23:40.322079+0000","last_undegraded":"2024-06-29T14:23:40.322079+0000","last_fullsized":"2024-06-29T14:23:40.322079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:41.926725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926255+0000","last_change":"2024-06-29T14:23:36.938779+0000","last_active":"2024-06-29T14:23:38.926255+0000","last_peered":"2024-06-29T14:23:38.926255+0000","last_clean":"2024-06-29T14:23:38.926255+0000","last_became_active":"2024-06-29T14:23:36.938479+0000","last_became_peered":"2024-06-29T14:23:36.938479+0000","last_unstale":"2024-06-29T14:23:38.926255+0000","last_undegraded":"2024-06-29T14:23:38.926255+0000","last_fullsized":"2024-06-29T14:23:38.926255+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:55.200059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.956745+0000","last_change":"2024-06-29T14:23:37.945859+0000","last_active":"2024-06-29T14:23:37.956745+0000","last_peered":"2024-06-29T14:23:37.956745+0000","last_clean":"2024-06-29T14:23:37.956745+0000","last_became_active":"2024-06-29T14:23:37.944779+0000","last_became_peered":"2024-06-29T14:23:37.944779+0000","last_unstale":"2024-06-29T14:23:37.956745+0000","last_undegraded":"2024-06-29T14:23:37.956745+0000","last_fullsized":"2024-06-29T14:23:37.956745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:28:33.557016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180572+0000","last_change":"2024-06-29T14:23:36.946478+0000","last_active":"2024-06-29T14:23:40.180572+0000","last_peered":"2024-06-29T14:23:40.180572+0000","last_clean":"2024-06-29T14:23:40.180572+0000","last_became_active":"2024-06-29T14:23:36.946090+0000","last_became_peered":"2024-06-29T14:23:36.946090+0000","last_unstale":"2024-06-29T14:23:40.180572+0000","last_undegraded":"2024-06-29T14:23:40.180572+0000","last_fullsized":"2024-06-29T14:23:40.180572+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:29.046411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926239+0000","last_change":"2024-06-29T14:23:37.945392+0000","last_active":"2024-06-29T14:23:38.926239+0000","last_peered":"2024-06-29T14:23:38.926239+0000","last_clean":"2024-06-29T14:23:38.926239+0000","last_became_active":"2024-06-29T14:23:37.944984+0000","last_became_peered":"2024-06-29T14:23:37.944984+0000","last_unstale":"2024-06-29T14:23:38.926239+0000","last_undegraded":"2024-06-29T14:23:38.926239+0000","last_fullsized":"2024-06-29T14:23:38.926239+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:46.137017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321117+0000","last_change":"2024-06-29T14:23:36.938376+0000","last_active":"2024-06-29T14:23:40.321117+0000","last_peered":"2024-06-29T14:23:40.321117+0000","last_clean":"2024-06-29T14:23:40.321117+0000","last_became_active":"2024-06-29T14:23:36.937882+0000","last_became_peered":"2024-06-29T14:23:36.937882+0000","last_unstale":"2024-06-29T14:23:40.321117+0000","last_undegraded":"2024-06-29T14:23:40.321117+0000","last_fullsized":"2024-06-29T14:23:40.321117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:32:22.490571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392343+0000","last_change":"2024-06-29T14:23:37.942821+0000","last_active":"2024-06-29T14:23:39.392343+0000","last_peered":"2024-06-29T14:23:39.392343+0000","last_clean":"2024-06-29T14:23:39.392343+0000","last_became_active":"2024-06-29T14:23:37.942496+0000","last_became_peered":"2024-06-29T14:23:37.942496+0000","last_unstale":"2024-06-29T14:23:39.392343+0000","last_undegraded":"2024-06-29T14:23:39.392343+0000","last_fullsized":"2024-06-29T14:23:39.392343+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:22.666511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.960028+0000","last_change":"2024-06-29T14:23:36.947122+0000","last_active":"2024-06-29T14:23:37.960028+0000","last_peered":"2024-06-29T14:23:37.960028+0000","last_clean":"2024-06-29T14:23:37.960028+0000","last_became_active":"2024-06-29T14:23:36.946730+0000","last_became_peered":"2024-06-29T14:23:36.946730+0000","last_unstale":"2024-06-29T14:23:37.960028+0000","last_undegraded":"2024-06-29T14:23:37.960028+0000","last_fullsized":"2024-06-29T14:23:37.960028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:53.868147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926257+0000","last_change":"2024-06-29T14:23:37.944325+0000","last_active":"2024-06-29T14:23:38.926257+0000","last_peered":"2024-06-29T14:23:38.926257+0000","last_clean":"2024-06-29T14:23:38.926257+0000","last_became_active":"2024-06-29T14:23:37.943973+0000","last_became_peered":"2024-06-29T14:23:37.943973+0000","last_unstale":"2024-06-29T14:23:38.926257+0000","last_undegraded":"2024-06-29T14:23:38.926257+0000","last_fullsized":"2024-06-29T14:23:38.926257+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:52.758662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.959951+0000","last_change":"2024-06-29T14:23:36.947050+0000","last_active":"2024-06-29T14:23:37.959951+0000","last_peered":"2024-06-29T14:23:37.959951+0000","last_clean":"2024-06-29T14:23:37.959951+0000","last_became_active":"2024-06-29T14:23:36.946695+0000","last_became_peered":"2024-06-29T14:23:36.946695+0000","last_unstale":"2024-06-29T14:23:37.959951+0000","last_undegraded":"2024-06-29T14:23:37.959951+0000","last_fullsized":"2024-06-29T14:23:37.959951+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:13:42.529115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.942022+0000","last_change":"2024-06-29T14:23:37.937537+0000","last_active":"2024-06-29T14:23:37.942022+0000","last_peered":"2024-06-29T14:23:37.942022+0000","last_clean":"2024-06-29T14:23:37.942022+0000","last_became_active":"2024-06-29T14:23:37.936920+0000","last_became_peered":"2024-06-29T14:23:37.936920+0000","last_unstale":"2024-06-29T14:23:37.942022+0000","last_undegraded":"2024-06-29T14:23:37.942022+0000","last_fullsized":"2024-06-29T14:23:37.942022+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:58.882288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180215+0000","last_change":"2024-06-29T14:23:36.939287+0000","last_active":"2024-06-29T14:23:40.180215+0000","last_peered":"2024-06-29T14:23:40.180215+0000","last_clean":"2024-06-29T14:23:40.180215+0000","last_became_active":"2024-06-29T14:23:36.938934+0000","last_became_peered":"2024-06-29T14:23:36.938934+0000","last_unstale":"2024-06-29T14:23:40.180215+0000","last_undegraded":"2024-06-29T14:23:40.180215+0000","last_fullsized":"2024-06-29T14:23:40.180215+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:34:11.042257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925788+0000","last_change":"2024-06-29T14:23:37.938497+0000","last_active":"2024-06-29T14:23:38.925788+0000","last_peered":"2024-06-29T14:23:38.925788+0000","last_clean":"2024-06-29T14:23:38.925788+0000","last_became_active":"2024-06-29T14:23:37.938217+0000","last_became_peered":"2024-06-29T14:23:37.938217+0000","last_unstale":"2024-06-29T14:23:38.925788+0000","last_undegraded":"2024-06-29T14:23:38.925788+0000","last_fullsized":"2024-06-29T14:23:38.925788+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:54:09.878998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.926031+0000","last_change":"2024-06-29T14:23:36.945009+0000","last_active":"2024-06-29T14:23:37.926031+0000","last_peered":"2024-06-29T14:23:37.926031+0000","last_clean":"2024-06-29T14:23:37.926031+0000","last_became_active":"2024-06-29T14:23:36.944318+0000","last_became_peered":"2024-06-29T14:23:36.944318+0000","last_unstale":"2024-06-29T14:23:37.926031+0000","last_undegraded":"2024-06-29T14:23:37.926031+0000","last_fullsized":"2024-06-29T14:23:37.926031+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:11.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926008+0000","last_change":"2024-06-29T14:23:37.941701+0000","last_active":"2024-06-29T14:23:38.926008+0000","last_peered":"2024-06-29T14:23:38.926008+0000","last_clean":"2024-06-29T14:23:38.926008+0000","last_became_active":"2024-06-29T14:23:37.941410+0000","last_became_peered":"2024-06-29T14:23:37.941410+0000","last_unstale":"2024-06-29T14:23:38.926008+0000","last_undegraded":"2024-06-29T14:23:38.926008+0000","last_fullsized":"2024-06-29T14:23:38.926008+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:05.123440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180881+0000","last_change":"2024-06-29T14:23:36.937096+0000","last_active":"2024-06-29T14:23:40.180881+0000","last_peered":"2024-06-29T14:23:40.180881+0000","last_clean":"2024-06-29T14:23:40.180881+0000","last_became_active":"2024-06-29T14:23:36.936836+0000","last_became_peered":"2024-06-29T14:23:36.936836+0000","last_unstale":"2024-06-29T14:23:40.180881+0000","last_undegraded":"2024-06-29T14:23:40.180881+0000","last_fullsized":"2024-06-29T14:23:40.180881+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:22.401053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180823+0000","last_change":"2024-06-29T14:23:37.950645+0000","last_active":"2024-06-29T14:23:40.180823+0000","last_peered":"2024-06-29T14:23:40.180823+0000","last_clean":"2024-06-29T14:23:40.180823+0000","last_became_active":"2024-06-29T14:23:37.950312+0000","last_became_peered":"2024-06-29T14:23:37.950312+0000","last_unstale":"2024-06-29T14:23:40.180823+0000","last_undegraded":"2024-06-29T14:23:40.180823+0000","last_fullsized":"2024-06-29T14:23:40.180823+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:33.856511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926412+0000","last_change":"2024-06-29T14:23:36.941120+0000","last_active":"2024-06-29T14:23:38.926412+0000","last_peered":"2024-06-29T14:23:38.926412+0000","last_clean":"2024-06-29T14:23:38.926412+0000","last_became_active":"2024-06-29T14:23:36.940820+0000","last_became_peered":"2024-06-29T14:23:36.940820+0000","last_unstale":"2024-06-29T14:23:38.926412+0000","last_undegraded":"2024-06-29T14:23:38.926412+0000","last_fullsized":"2024-06-29T14:23:38.926412+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:57.874214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180564+0000","last_change":"2024-06-29T14:23:37.942391+0000","last_active":"2024-06-29T14:23:40.180564+0000","last_peered":"2024-06-29T14:23:40.180564+0000","last_clean":"2024-06-29T14:23:40.180564+0000","last_became_active":"2024-06-29T14:23:37.942007+0000","last_became_peered":"2024-06-29T14:23:37.942007+0000","last_unstale":"2024-06-29T14:23:40.180564+0000","last_undegraded":"2024-06-29T14:23:40.180564+0000","last_fullsized":"2024-06-29T14:23:40.180564+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:32:14.563469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926090+0000","last_change":"2024-06-29T14:23:36.938132+0000","last_active":"2024-06-29T14:23:38.926090+0000","last_peered":"2024-06-29T14:23:38.926090+0000","last_clean":"2024-06-29T14:23:38.926090+0000","last_became_active":"2024-06-29T14:23:36.937786+0000","last_became_peered":"2024-06-29T14:23:36.937786+0000","last_unstale":"2024-06-29T14:23:38.926090+0000","last_undegraded":"2024-06-29T14:23:38.926090+0000","last_fullsized":"2024-06-29T14:23:38.926090+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:46:50.318706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926212+0000","last_change":"2024-06-29T14:23:37.950108+0000","last_active":"2024-06-29T14:23:38.926212+0000","last_peered":"2024-06-29T14:23:38.926212+0000","last_clean":"2024-06-29T14:23:38.926212+0000","last_became_active":"2024-06-29T14:23:37.949817+0000","last_became_peered":"2024-06-29T14:23:37.949817+0000","last_unstale":"2024-06-29T14:23:38.926212+0000","last_undegraded":"2024-06-29T14:23:38.926212+0000","last_fullsized":"2024-06-29T14:23:38.926212+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:21.205455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180554+0000","last_change":"2024-06-29T14:23:36.936852+0000","last_active":"2024-06-29T14:23:40.180554+0000","last_peered":"2024-06-29T14:23:40.180554+0000","last_clean":"2024-06-29T14:23:40.180554+0000","last_became_active":"2024-06-29T14:23:36.936550+0000","last_became_peered":"2024-06-29T14:23:36.936550+0000","last_unstale":"2024-06-29T14:23:40.180554+0000","last_undegraded":"2024-06-29T14:23:40.180554+0000","last_fullsized":"2024-06-29T14:23:40.180554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:40.918651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926035+0000","last_change":"2024-06-29T14:23:37.937924+0000","last_active":"2024-06-29T14:23:38.926035+0000","last_peered":"2024-06-29T14:23:38.926035+0000","last_clean":"2024-06-29T14:23:38.926035+0000","last_became_active":"2024-06-29T14:23:37.937618+0000","last_became_peered":"2024-06-29T14:23:37.937618+0000","last_unstale":"2024-06-29T14:23:38.926035+0000","last_undegraded":"2024-06-29T14:23:38.926035+0000","last_fullsized":"2024-06-29T14:23:38.926035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:48:59.553463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.960132+0000","last_change":"2024-06-29T14:23:36.944778+0000","last_active":"2024-06-29T14:23:37.960132+0000","last_peered":"2024-06-29T14:23:37.960132+0000","last_clean":"2024-06-29T14:23:37.960132+0000","last_became_active":"2024-06-29T14:23:36.944438+0000","last_became_peered":"2024-06-29T14:23:36.944438+0000","last_unstale":"2024-06-29T14:23:37.960132+0000","last_undegraded":"2024-06-29T14:23:37.960132+0000","last_fullsized":"2024-06-29T14:23:37.960132+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:47.733035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180504+0000","last_change":"2024-06-29T14:23:37.944354+0000","last_active":"2024-06-29T14:23:40.180504+0000","last_peered":"2024-06-29T14:23:40.180504+0000","last_clean":"2024-06-29T14:23:40.180504+0000","last_became_active":"2024-06-29T14:23:37.944020+0000","last_became_peered":"2024-06-29T14:23:37.944020+0000","last_unstale":"2024-06-29T14:23:40.180504+0000","last_undegraded":"2024-06-29T14:23:40.180504+0000","last_fullsized":"2024-06-29T14:23:40.180504+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:52:17.539322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926155+0000","last_change":"2024-06-29T14:23:36.938434+0000","last_active":"2024-06-29T14:23:38.926155+0000","last_peered":"2024-06-29T14:23:38.926155+0000","last_clean":"2024-06-29T14:23:38.926155+0000","last_became_active":"2024-06-29T14:23:36.937921+0000","last_became_peered":"2024-06-29T14:23:36.937921+0000","last_unstale":"2024-06-29T14:23:38.926155+0000","last_undegraded":"2024-06-29T14:23:38.926155+0000","last_fullsized":"2024-06-29T14:23:38.926155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:55.200059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392561+0000","last_change":"2024-06-29T14:23:37.944500+0000","last_active":"2024-06-29T14:23:39.392561+0000","last_peered":"2024-06-29T14:23:39.392561+0000","last_clean":"2024-06-29T14:23:39.392561+0000","last_became_active":"2024-06-29T14:23:37.944081+0000","last_became_peered":"2024-06-29T14:23:37.944081+0000","last_unstale":"2024-06-29T14:23:39.392561+0000","last_undegraded":"2024-06-29T14:23:39.392561+0000","last_fullsized":"2024-06-29T14:23:39.392561+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:21.205455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180155+0000","last_change":"2024-06-29T14:23:36.940884+0000","last_active":"2024-06-29T14:23:40.180155+0000","last_peered":"2024-06-29T14:23:40.180155+0000","last_clean":"2024-06-29T14:23:40.180155+0000","last_became_active":"2024-06-29T14:23:36.940548+0000","last_became_peered":"2024-06-29T14:23:36.940548+0000","last_unstale":"2024-06-29T14:23:40.180155+0000","last_undegraded":"2024-06-29T14:23:40.180155+0000","last_fullsized":"2024-06-29T14:23:40.180155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:01.799530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.322009+0000","last_change":"2024-06-29T14:23:37.941136+0000","last_active":"2024-06-29T14:23:40.322009+0000","last_peered":"2024-06-29T14:23:40.322009+0000","last_clean":"2024-06-29T14:23:40.322009+0000","last_became_active":"2024-06-29T14:23:37.940870+0000","last_became_peered":"2024-06-29T14:23:37.940870+0000","last_unstale":"2024-06-29T14:23:40.322009+0000","last_undegraded":"2024-06-29T14:23:40.322009+0000","last_fullsized":"2024-06-29T14:23:40.322009+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:28:33.557016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180188+0000","last_change":"2024-06-29T14:23:36.946906+0000","last_active":"2024-06-29T14:23:40.180188+0000","last_peered":"2024-06-29T14:23:40.180188+0000","last_clean":"2024-06-29T14:23:40.180188+0000","last_became_active":"2024-06-29T14:23:36.946599+0000","last_became_peered":"2024-06-29T14:23:36.946599+0000","last_unstale":"2024-06-29T14:23:40.180188+0000","last_undegraded":"2024-06-29T14:23:40.180188+0000","last_fullsized":"2024-06-29T14:23:40.180188+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:22.975505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.940245+0000","last_change":"2024-06-29T14:23:37.934667+0000","last_active":"2024-06-29T14:23:37.940245+0000","last_peered":"2024-06-29T14:23:37.940245+0000","last_clean":"2024-06-29T14:23:37.940245+0000","last_became_active":"2024-06-29T14:23:37.934352+0000","last_became_peered":"2024-06-29T14:23:37.934352+0000","last_unstale":"2024-06-29T14:23:37.940245+0000","last_undegraded":"2024-06-29T14:23:37.940245+0000","last_fullsized":"2024-06-29T14:23:37.940245+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:30.663631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926957+0000","last_change":"2024-06-29T14:23:36.941653+0000","last_active":"2024-06-29T14:23:38.926957+0000","last_peered":"2024-06-29T14:23:38.926957+0000","last_clean":"2024-06-29T14:23:38.926957+0000","last_became_active":"2024-06-29T14:23:36.941327+0000","last_became_peered":"2024-06-29T14:23:36.941327+0000","last_unstale":"2024-06-29T14:23:38.926957+0000","last_undegraded":"2024-06-29T14:23:38.926957+0000","last_fullsized":"2024-06-29T14:23:38.926957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:22.975505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926199+0000","last_change":"2024-06-29T14:23:37.934697+0000","last_active":"2024-06-29T14:23:38.926199+0000","last_peered":"2024-06-29T14:23:38.926199+0000","last_clean":"2024-06-29T14:23:38.926199+0000","last_became_active":"2024-06-29T14:23:37.934409+0000","last_became_peered":"2024-06-29T14:23:37.934409+0000","last_unstale":"2024-06-29T14:23:38.926199+0000","last_undegraded":"2024-06-29T14:23:38.926199+0000","last_fullsized":"2024-06-29T14:23:38.926199+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:57.496214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.960031+0000","last_change":"2024-06-29T14:23:36.945192+0000","last_active":"2024-06-29T14:23:37.960031+0000","last_peered":"2024-06-29T14:23:37.960031+0000","last_clean":"2024-06-29T14:23:37.960031+0000","last_became_active":"2024-06-29T14:23:36.944784+0000","last_became_peered":"2024-06-29T14:23:36.944784+0000","last_unstale":"2024-06-29T14:23:37.960031+0000","last_undegraded":"2024-06-29T14:23:37.960031+0000","last_fullsized":"2024-06-29T14:23:37.960031+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:01:51.989032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926458+0000","last_change":"2024-06-29T14:23:37.948937+0000","last_active":"2024-06-29T14:23:38.926458+0000","last_peered":"2024-06-29T14:23:38.926458+0000","last_clean":"2024-06-29T14:23:38.926458+0000","last_became_active":"2024-06-29T14:23:37.943575+0000","last_became_peered":"2024-06-29T14:23:37.943575+0000","last_unstale":"2024-06-29T14:23:38.926458+0000","last_undegraded":"2024-06-29T14:23:38.926458+0000","last_fullsized":"2024-06-29T14:23:38.926458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:52.758662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.960142+0000","last_change":"2024-06-29T14:23:36.945440+0000","last_active":"2024-06-29T14:23:37.960142+0000","last_peered":"2024-06-29T14:23:37.960142+0000","last_clean":"2024-06-29T14:23:37.960142+0000","last_became_active":"2024-06-29T14:23:36.944489+0000","last_became_peered":"2024-06-29T14:23:36.944489+0000","last_unstale":"2024-06-29T14:23:37.960142+0000","last_undegraded":"2024-06-29T14:23:37.960142+0000","last_fullsized":"2024-06-29T14:23:37.960142+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:34:31.399286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321055+0000","last_change":"2024-06-29T14:23:37.942181+0000","last_active":"2024-06-29T14:23:40.321055+0000","last_peered":"2024-06-29T14:23:40.321055+0000","last_clean":"2024-06-29T14:23:40.321055+0000","last_became_active":"2024-06-29T14:23:37.941910+0000","last_became_peered":"2024-06-29T14:23:37.941910+0000","last_unstale":"2024-06-29T14:23:40.321055+0000","last_undegraded":"2024-06-29T14:23:40.321055+0000","last_fullsized":"2024-06-29T14:23:40.321055+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:48.741109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.926262+0000","last_change":"2024-06-29T14:23:36.944581+0000","last_active":"2024-06-29T14:23:37.926262+0000","last_peered":"2024-06-29T14:23:37.926262+0000","last_clean":"2024-06-29T14:23:37.926262+0000","last_became_active":"2024-06-29T14:23:36.944286+0000","last_became_peered":"2024-06-29T14:23:36.944286+0000","last_unstale":"2024-06-29T14:23:37.926262+0000","last_undegraded":"2024-06-29T14:23:37.926262+0000","last_fullsized":"2024-06-29T14:23:37.926262+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:43.777161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321025+0000","last_change":"2024-06-29T14:23:37.933659+0000","last_active":"2024-06-29T14:23:40.321025+0000","last_peered":"2024-06-29T14:23:40.321025+0000","last_clean":"2024-06-29T14:23:40.321025+0000","last_became_active":"2024-06-29T14:23:37.933328+0000","last_became_peered":"2024-06-29T14:23:37.933328+0000","last_unstale":"2024-06-29T14:23:40.321025+0000","last_undegraded":"2024-06-29T14:23:40.321025+0000","last_fullsized":"2024-06-29T14:23:40.321025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:44.785235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926759+0000","last_change":"2024-06-29T14:23:36.942749+0000","last_active":"2024-06-29T14:23:38.926759+0000","last_peered":"2024-06-29T14:23:38.926759+0000","last_clean":"2024-06-29T14:23:38.926759+0000","last_became_active":"2024-06-29T14:23:36.942353+0000","last_became_peered":"2024-06-29T14:23:36.942353+0000","last_unstale":"2024-06-29T14:23:38.926759+0000","last_undegraded":"2024-06-29T14:23:38.926759+0000","last_fullsized":"2024-06-29T14:23:38.926759+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:47.733035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321166+0000","last_change":"2024-06-29T14:23:37.947132+0000","last_active":"2024-06-29T14:23:40.321166+0000","last_peered":"2024-06-29T14:23:40.321166+0000","last_clean":"2024-06-29T14:23:40.321166+0000","last_became_active":"2024-06-29T14:23:37.943954+0000","last_became_peered":"2024-06-29T14:23:37.943954+0000","last_unstale":"2024-06-29T14:23:40.321166+0000","last_undegraded":"2024-06-29T14:23:40.321166+0000","last_fullsized":"2024-06-29T14:23:40.321166+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:17.132446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321819+0000","last_change":"2024-06-29T14:23:36.945816+0000","last_active":"2024-06-29T14:23:40.321819+0000","last_peered":"2024-06-29T14:23:40.321819+0000","last_clean":"2024-06-29T14:23:40.321819+0000","last_became_active":"2024-06-29T14:23:36.945518+0000","last_became_peered":"2024-06-29T14:23:36.945518+0000","last_unstale":"2024-06-29T14:23:40.321819+0000","last_undegraded":"2024-06-29T14:23:40.321819+0000","last_fullsized":"2024-06-29T14:23:40.321819+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:53.868147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.320977+0000","last_change":"2024-06-29T14:23:37.947172+0000","last_active":"2024-06-29T14:23:40.320977+0000","last_peered":"2024-06-29T14:23:40.320977+0000","last_clean":"2024-06-29T14:23:40.320977+0000","last_became_active":"2024-06-29T14:23:37.940451+0000","last_became_peered":"2024-06-29T14:23:37.940451+0000","last_unstale":"2024-06-29T14:23:40.320977+0000","last_undegraded":"2024-06-29T14:23:40.320977+0000","last_fullsized":"2024-06-29T14:23:40.320977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:12.312016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.925471+0000","last_change":"2024-06-29T14:23:36.946365+0000","last_active":"2024-06-29T14:23:37.925471+0000","last_peered":"2024-06-29T14:23:37.925471+0000","last_clean":"2024-06-29T14:23:37.925471+0000","last_became_active":"2024-06-29T14:23:36.946079+0000","last_became_peered":"2024-06-29T14:23:36.946079+0000","last_unstale":"2024-06-29T14:23:37.925471+0000","last_undegraded":"2024-06-29T14:23:37.925471+0000","last_fullsized":"2024-06-29T14:23:37.925471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:10.736740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392258+0000","last_change":"2024-06-29T14:23:37.945266+0000","last_active":"2024-06-29T14:23:39.392258+0000","last_peered":"2024-06-29T14:23:39.392258+0000","last_clean":"2024-06-29T14:23:39.392258+0000","last_became_active":"2024-06-29T14:23:37.944936+0000","last_became_peered":"2024-06-29T14:23:37.944936+0000","last_unstale":"2024-06-29T14:23:39.392258+0000","last_undegraded":"2024-06-29T14:23:39.392258+0000","last_fullsized":"2024-06-29T14:23:39.392258+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:50.182625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925722+0000","last_change":"2024-06-29T14:23:36.938180+0000","last_active":"2024-06-29T14:23:38.925722+0000","last_peered":"2024-06-29T14:23:38.925722+0000","last_clean":"2024-06-29T14:23:38.925722+0000","last_became_active":"2024-06-29T14:23:36.937807+0000","last_became_peered":"2024-06-29T14:23:36.937807+0000","last_unstale":"2024-06-29T14:23:38.925722+0000","last_undegraded":"2024-06-29T14:23:38.925722+0000","last_fullsized":"2024-06-29T14:23:38.925722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:57.874214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321035+0000","last_change":"2024-06-29T14:23:37.938990+0000","last_active":"2024-06-29T14:23:40.321035+0000","last_peered":"2024-06-29T14:23:40.321035+0000","last_clean":"2024-06-29T14:23:40.321035+0000","last_became_active":"2024-06-29T14:23:37.938698+0000","last_became_peered":"2024-06-29T14:23:37.938698+0000","last_unstale":"2024-06-29T14:23:40.321035+0000","last_undegraded":"2024-06-29T14:23:40.321035+0000","last_fullsized":"2024-06-29T14:23:40.321035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:54.876220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.927033+0000","last_change":"2024-06-29T14:23:36.941461+0000","last_active":"2024-06-29T14:23:38.927033+0000","last_peered":"2024-06-29T14:23:38.927033+0000","last_clean":"2024-06-29T14:23:38.927033+0000","last_became_active":"2024-06-29T14:23:36.941167+0000","last_became_peered":"2024-06-29T14:23:36.941167+0000","last_unstale":"2024-06-29T14:23:38.927033+0000","last_undegraded":"2024-06-29T14:23:38.927033+0000","last_fullsized":"2024-06-29T14:23:38.927033+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:01.799530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926334+0000","last_change":"2024-06-29T14:23:37.944718+0000","last_active":"2024-06-29T14:23:38.926334+0000","last_peered":"2024-06-29T14:23:38.926334+0000","last_clean":"2024-06-29T14:23:38.926334+0000","last_became_active":"2024-06-29T14:23:37.943224+0000","last_became_peered":"2024-06-29T14:23:37.943224+0000","last_unstale":"2024-06-29T14:23:38.926334+0000","last_undegraded":"2024-06-29T14:23:38.926334+0000","last_fullsized":"2024-06-29T14:23:38.926334+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:20.327712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.928505+0000","last_change":"2024-06-29T14:23:36.937915+0000","last_active":"2024-06-29T14:23:38.928505+0000","last_peered":"2024-06-29T14:23:38.928505+0000","last_clean":"2024-06-29T14:23:38.928505+0000","last_became_active":"2024-06-29T14:23:36.937596+0000","last_became_peered":"2024-06-29T14:23:36.937596+0000","last_unstale":"2024-06-29T14:23:38.928505+0000","last_undegraded":"2024-06-29T14:23:38.928505+0000","last_fullsized":"2024-06-29T14:23:38.928505+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:16.124372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.927036+0000","last_change":"2024-06-29T14:23:37.955845+0000","last_active":"2024-06-29T14:23:38.927036+0000","last_peered":"2024-06-29T14:23:38.927036+0000","last_clean":"2024-06-29T14:23:38.927036+0000","last_became_active":"2024-06-29T14:23:37.955464+0000","last_became_peered":"2024-06-29T14:23:37.955464+0000","last_unstale":"2024-06-29T14:23:38.927036+0000","last_undegraded":"2024-06-29T14:23:38.927036+0000","last_fullsized":"2024-06-29T14:23:38.927036+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:23.409126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926500+0000","last_change":"2024-06-29T14:23:36.936705+0000","last_active":"2024-06-29T14:23:38.926500+0000","last_peered":"2024-06-29T14:23:38.926500+0000","last_clean":"2024-06-29T14:23:38.926500+0000","last_became_active":"2024-06-29T14:23:36.936394+0000","last_became_peered":"2024-06-29T14:23:36.936394+0000","last_unstale":"2024-06-29T14:23:38.926500+0000","last_undegraded":"2024-06-29T14:23:38.926500+0000","last_fullsized":"2024-06-29T14:23:38.926500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:57.874214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321598+0000","last_change":"2024-06-29T14:23:37.936616+0000","last_active":"2024-06-29T14:23:40.321598+0000","last_peered":"2024-06-29T14:23:40.321598+0000","last_clean":"2024-06-29T14:23:40.321598+0000","last_became_active":"2024-06-29T14:23:37.936318+0000","last_became_peered":"2024-06-29T14:23:37.936318+0000","last_unstale":"2024-06-29T14:23:40.321598+0000","last_undegraded":"2024-06-29T14:23:40.321598+0000","last_fullsized":"2024-06-29T14:23:40.321598+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:02.807604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926981+0000","last_change":"2024-06-29T14:23:36.939546+0000","last_active":"2024-06-29T14:23:38.926981+0000","last_peered":"2024-06-29T14:23:38.926981+0000","last_clean":"2024-06-29T14:23:38.926981+0000","last_became_active":"2024-06-29T14:23:36.939256+0000","last_became_peered":"2024-06-29T14:23:36.939256+0000","last_unstale":"2024-06-29T14:23:38.926981+0000","last_undegraded":"2024-06-29T14:23:38.926981+0000","last_fullsized":"2024-06-29T14:23:38.926981+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:29.655557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.927187+0000","last_change":"2024-06-29T14:23:37.947953+0000","last_active":"2024-06-29T14:23:38.927187+0000","last_peered":"2024-06-29T14:23:38.927187+0000","last_clean":"2024-06-29T14:23:38.927187+0000","last_became_active":"2024-06-29T14:23:37.947655+0000","last_became_peered":"2024-06-29T14:23:37.947655+0000","last_unstale":"2024-06-29T14:23:38.927187+0000","last_undegraded":"2024-06-29T14:23:38.927187+0000","last_fullsized":"2024-06-29T14:23:38.927187+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:30.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926258+0000","last_change":"2024-06-29T14:23:36.930826+0000","last_active":"2024-06-29T14:23:38.926258+0000","last_peered":"2024-06-29T14:23:38.926258+0000","last_clean":"2024-06-29T14:23:38.926258+0000","last_became_active":"2024-06-29T14:23:36.930509+0000","last_became_peered":"2024-06-29T14:23:36.930509+0000","last_unstale":"2024-06-29T14:23:38.926258+0000","last_undegraded":"2024-06-29T14:23:38.926258+0000","last_fullsized":"2024-06-29T14:23:38.926258+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:16.124372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180217+0000","last_change":"2024-06-29T14:23:37.950636+0000","last_active":"2024-06-29T14:23:40.180217+0000","last_peered":"2024-06-29T14:23:40.180217+0000","last_clean":"2024-06-29T14:23:40.180217+0000","last_became_active":"2024-06-29T14:23:37.950266+0000","last_became_peered":"2024-06-29T14:23:37.950266+0000","last_unstale":"2024-06-29T14:23:40.180217+0000","last_undegraded":"2024-06-29T14:23:40.180217+0000","last_fullsized":"2024-06-29T14:23:40.180217+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:09.669198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.925788+0000","last_change":"2024-06-29T14:23:36.942425+0000","last_active":"2024-06-29T14:23:37.925788+0000","last_peered":"2024-06-29T14:23:37.925788+0000","last_clean":"2024-06-29T14:23:37.925788+0000","last_became_active":"2024-06-29T14:23:36.942072+0000","last_became_peered":"2024-06-29T14:23:36.942072+0000","last_unstale":"2024-06-29T14:23:37.925788+0000","last_undegraded":"2024-06-29T14:23:37.925788+0000","last_fullsized":"2024-06-29T14:23:37.925788+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:05:58.738515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.948778+0000","last_change":"2024-06-29T14:23:37.944378+0000","last_active":"2024-06-29T14:23:37.948778+0000","last_peered":"2024-06-29T14:23:37.948778+0000","last_clean":"2024-06-29T14:23:37.948778+0000","last_became_active":"2024-06-29T14:23:37.944086+0000","last_became_peered":"2024-06-29T14:23:37.944086+0000","last_unstale":"2024-06-29T14:23:37.948778+0000","last_undegraded":"2024-06-29T14:23:37.948778+0000","last_fullsized":"2024-06-29T14:23:37.948778+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:23.983579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180379+0000","last_change":"2024-06-29T14:23:36.938840+0000","last_active":"2024-06-29T14:23:40.180379+0000","last_peered":"2024-06-29T14:23:40.180379+0000","last_clean":"2024-06-29T14:23:40.180379+0000","last_became_active":"2024-06-29T14:23:36.938473+0000","last_became_peered":"2024-06-29T14:23:36.938473+0000","last_unstale":"2024-06-29T14:23:40.180379+0000","last_undegraded":"2024-06-29T14:23:40.180379+0000","last_fullsized":"2024-06-29T14:23:40.180379+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:28:32.548942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392331+0000","last_change":"2024-06-29T14:23:37.952183+0000","last_active":"2024-06-29T14:23:39.392331+0000","last_peered":"2024-06-29T14:23:39.392331+0000","last_clean":"2024-06-29T14:23:39.392331+0000","last_became_active":"2024-06-29T14:23:37.951905+0000","last_became_peered":"2024-06-29T14:23:37.951905+0000","last_unstale":"2024-06-29T14:23:39.392331+0000","last_undegraded":"2024-06-29T14:23:39.392331+0000","last_fullsized":"2024-06-29T14:23:39.392331+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:52.758662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321820+0000","last_change":"2024-06-29T14:23:36.948071+0000","last_active":"2024-06-29T14:23:40.321820+0000","last_peered":"2024-06-29T14:23:40.321820+0000","last_clean":"2024-06-29T14:23:40.321820+0000","last_became_active":"2024-06-29T14:23:36.947743+0000","last_became_peered":"2024-06-29T14:23:36.947743+0000","last_unstale":"2024-06-29T14:23:40.321820+0000","last_undegraded":"2024-06-29T14:23:40.321820+0000","last_fullsized":"2024-06-29T14:23:40.321820+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:34:31.399286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":134,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392604+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:23:39.392604+0000","last_peered":"2024-06-29T14:23:39.392604+0000","last_clean":"2024-06-29T14:23:39.392604+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:23:39.392604+0000","last_undegraded":"2024-06-29T14:23:39.392604+0000","last_fullsized":"2024-06-29T14:23:39.392604+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":736936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":140794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":140282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":966656,"data_stored":736846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:23:41.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:23:42.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.127+0000 7f73d8324640 1 -- 172.21.15.81:0/752180255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73d00d0940 msgr2=0x7f73d00b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.127+0000 7f73d8324640 1 --2- 172.21.15.81:0/752180255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73d00d0940 0x7f73d00b4f40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f73c8005fd0 tx=0x7f73c802aa00 comp rx=0 tx=0).stop 2024-06-29T14:23:42.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.127+0000 7f73d8324640 1 -- 172.21.15.81:0/752180255 shutdown_connections 2024-06-29T14:23:42.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.127+0000 7f73d8324640 1 --2- 172.21.15.81:0/752180255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73d00d0940 0x7f73d00b4f40 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.127+0000 7f73d8324640 1 --2- 172.21.15.81:0/752180255 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73d00b3200 0x7f73d00cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.127+0000 7f73d8324640 1 -- 172.21.15.81:0/752180255 >> 172.21.15.81:0/752180255 conn(0x7f73d00b3680 msgr2=0x7f73d00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:42.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.127+0000 7f73d8324640 1 -- 172.21.15.81:0/752180255 shutdown_connections 2024-06-29T14:23:42.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.127+0000 7f73d8324640 1 -- 172.21.15.81:0/752180255 wait complete. 2024-06-29T14:23:42.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.129+0000 7f73d8324640 1 Processor -- start 2024-06-29T14:23:42.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.129+0000 7f73d8324640 1 -- start start 2024-06-29T14:23:42.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.130+0000 7f73d8324640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73d00b3200 0x7f73d01e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.130+0000 7f73d8324640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73d00cccf0 0x7f73d014af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.130+0000 7f73d8324640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73d015aa10 0x7f73d015ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d8324640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f73d00b5c50 con 0x7f73d00cccf0 2024-06-29T14:23:42.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d8324640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f73d00b5ad0 con 0x7f73d015aa10 2024-06-29T14:23:42.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d8324640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f73d00b5dd0 con 0x7f73d00b3200 2024-06-29T14:23:42.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d58bf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73d00cccf0 0x7f73d014af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d58bf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73d00cccf0 0x7f73d014af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42090/0 (socket says 172.21.15.81:42090) 2024-06-29T14:23:42.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d58bf640 1 -- 172.21.15.81:0/895085388 learned_addr learned my addr 172.21.15.81:0/895085388 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:42.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d60c0640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73d00b3200 0x7f73d01e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d68c1640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73d015aa10 0x7f73d015ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d58bf640 1 -- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73d00b3200 msgr2=0x7f73d01e67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d58bf640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73d00b3200 0x7f73d01e67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d58bf640 1 -- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73d015aa10 msgr2=0x7f73d015ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d58bf640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73d015aa10 0x7f73d015ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d58bf640 1 -- 172.21.15.81:0/895085388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d015b360 con 0x7f73d00cccf0 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d58bf640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73d00cccf0 0x7f73d014af40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f73c000b3c0 tx=0x7f73c000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73c6ffd640 1 -- 172.21.15.81:0/895085388 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0011830 con 0x7f73d00cccf0 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73c6ffd640 1 -- 172.21.15.81:0/895085388 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73c0016070 con 0x7f73d00cccf0 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d8324640 1 -- 172.21.15.81:0/895085388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d01e8260 con 0x7f73d00cccf0 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73c6ffd640 1 -- 172.21.15.81:0/895085388 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0011da0 con 0x7f73d00cccf0 2024-06-29T14:23:42.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.131+0000 7f73d8324640 1 -- 172.21.15.81:0/895085388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d01e74a0 con 0x7f73d00cccf0 2024-06-29T14:23:42.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.132+0000 7f73d8324640 1 -- 172.21.15.81:0/895085388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d01e6f10 con 0x7f73d00cccf0 2024-06-29T14:23:42.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.133+0000 7f73c6ffd640 1 -- 172.21.15.81:0/895085388 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f73c001b460 con 0x7f73d00cccf0 2024-06-29T14:23:42.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.133+0000 7f73c6ffd640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f73b4076300 0x7f73b4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.134+0000 7f73c6ffd640 1 -- 172.21.15.81:0/895085388 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f73c00973c0 con 0x7f73d00cccf0 2024-06-29T14:23:42.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.134+0000 7f73d60c0640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f73b4076300 0x7f73b4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.134+0000 7f73d60c0640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f73b4076300 0x7f73b4078810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0058c0 tx=0x7f73cc00c440 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:42.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.138+0000 7f73c6ffd640 1 -- 172.21.15.81:0/895085388 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f73c0020020 con 0x7f73d00cccf0 2024-06-29T14:23:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.315+0000 7f73d8324640 1 -- 172.21.15.81:0/895085388 --> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f73d014b480 con 0x7f73b4076300 2024-06-29T14:23:42.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.320+0000 7f73c6ffd640 1 -- 172.21.15.81:0/895085388 <== mgr.4126 v2:172.21.15.167:6832/3101542440 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251929 (secure 0 0 0) 0x7f73d014b480 con 0x7f73b4076300 2024-06-29T14:23:42.322 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:42.323 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:23:42.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.322+0000 7f73d8324640 1 -- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f73b4076300 msgr2=0x7f73b4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f73b4076300 0x7f73b4078810 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0058c0 tx=0x7f73cc00c440 comp rx=0 tx=0).stop 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 -- 172.21.15.81:0/895085388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73d00cccf0 msgr2=0x7f73d014af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73d00cccf0 0x7f73d014af40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f73c000b3c0 tx=0x7f73c000b880 comp rx=0 tx=0).stop 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 -- 172.21.15.81:0/895085388 shutdown_connections 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f73b4076300 0x7f73b4078810 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73d015aa10 0x7f73d015ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73d00cccf0 0x7f73d014af40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 --2- 172.21.15.81:0/895085388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73d00b3200 0x7f73d01e67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 -- 172.21.15.81:0/895085388 >> 172.21.15.81:0/895085388 conn(0x7f73d00b3680 msgr2=0x7f73d0158270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 -- 172.21.15.81:0/895085388 shutdown_connections 2024-06-29T14:23:42.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.323+0000 7f73d8324640 1 -- 172.21.15.81:0/895085388 wait complete. 2024-06-29T14:23:42.333 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-06-29T14:23:40.717420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220284,"kb_used_data":4100,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754436,"statfs":{"total":773094113280,"available":772868542464,"internally_reserved":0,"allocated":4198400,"data_stored":2338480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.799275"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321872+0000","last_change":"2024-06-29T14:23:37.939170+0000","last_active":"2024-06-29T14:23:40.321872+0000","last_peered":"2024-06-29T14:23:40.321872+0000","last_clean":"2024-06-29T14:23:40.321872+0000","last_became_active":"2024-06-29T14:23:37.938877+0000","last_became_peered":"2024-06-29T14:23:37.938877+0000","last_unstale":"2024-06-29T14:23:40.321872+0000","last_undegraded":"2024-06-29T14:23:40.321872+0000","last_fullsized":"2024-06-29T14:23:40.321872+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:23.983579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180776+0000","last_change":"2024-06-29T14:23:37.945549+0000","last_active":"2024-06-29T14:23:40.180776+0000","last_peered":"2024-06-29T14:23:40.180776+0000","last_clean":"2024-06-29T14:23:40.180776+0000","last_became_active":"2024-06-29T14:23:37.945244+0000","last_became_peered":"2024-06-29T14:23:37.945244+0000","last_unstale":"2024-06-29T14:23:40.180776+0000","last_undegraded":"2024-06-29T14:23:40.180776+0000","last_fullsized":"2024-06-29T14:23:40.180776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:29.365340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925985+0000","last_change":"2024-06-29T14:23:37.951081+0000","last_active":"2024-06-29T14:23:38.925985+0000","last_peered":"2024-06-29T14:23:38.925985+0000","last_clean":"2024-06-29T14:23:38.925985+0000","last_became_active":"2024-06-29T14:23:37.950719+0000","last_became_peered":"2024-06-29T14:23:37.950719+0000","last_unstale":"2024-06-29T14:23:38.925985+0000","last_undegraded":"2024-06-29T14:23:38.925985+0000","last_fullsized":"2024-06-29T14:23:38.925985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:22.666511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926215+0000","last_change":"2024-06-29T14:23:37.952847+0000","last_active":"2024-06-29T14:23:38.926215+0000","last_peered":"2024-06-29T14:23:38.926215+0000","last_clean":"2024-06-29T14:23:38.926215+0000","last_became_active":"2024-06-29T14:23:37.947188+0000","last_became_peered":"2024-06-29T14:23:37.947188+0000","last_unstale":"2024-06-29T14:23:38.926215+0000","last_undegraded":"2024-06-29T14:23:38.926215+0000","last_fullsized":"2024-06-29T14:23:38.926215+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:50.182625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321144+0000","last_change":"2024-06-29T14:23:37.935785+0000","last_active":"2024-06-29T14:23:40.321144+0000","last_peered":"2024-06-29T14:23:40.321144+0000","last_clean":"2024-06-29T14:23:40.321144+0000","last_became_active":"2024-06-29T14:23:37.935450+0000","last_became_peered":"2024-06-29T14:23:37.935450+0000","last_unstale":"2024-06-29T14:23:40.321144+0000","last_undegraded":"2024-06-29T14:23:40.321144+0000","last_fullsized":"2024-06-29T14:23:40.321144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:56.208133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180354+0000","last_change":"2024-06-29T14:23:37.930376+0000","last_active":"2024-06-29T14:23:40.180354+0000","last_peered":"2024-06-29T14:23:40.180354+0000","last_clean":"2024-06-29T14:23:40.180354+0000","last_became_active":"2024-06-29T14:23:37.930071+0000","last_became_peered":"2024-06-29T14:23:37.930071+0000","last_unstale":"2024-06-29T14:23:40.180354+0000","last_undegraded":"2024-06-29T14:23:40.180354+0000","last_fullsized":"2024-06-29T14:23:40.180354+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:20.327712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392471+0000","last_change":"2024-06-29T14:23:37.945999+0000","last_active":"2024-06-29T14:23:39.392471+0000","last_peered":"2024-06-29T14:23:39.392471+0000","last_clean":"2024-06-29T14:23:39.392471+0000","last_became_active":"2024-06-29T14:23:37.945649+0000","last_became_peered":"2024-06-29T14:23:37.945649+0000","last_unstale":"2024-06-29T14:23:39.392471+0000","last_undegraded":"2024-06-29T14:23:39.392471+0000","last_fullsized":"2024-06-29T14:23:39.392471+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:57.496214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392174+0000","last_change":"2024-06-29T14:23:37.949553+0000","last_active":"2024-06-29T14:23:39.392174+0000","last_peered":"2024-06-29T14:23:39.392174+0000","last_clean":"2024-06-29T14:23:39.392174+0000","last_became_active":"2024-06-29T14:23:37.949285+0000","last_became_peered":"2024-06-29T14:23:37.949285+0000","last_unstale":"2024-06-29T14:23:39.392174+0000","last_undegraded":"2024-06-29T14:23:39.392174+0000","last_fullsized":"2024-06-29T14:23:39.392174+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:09.623791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.948729+0000","last_change":"2024-06-29T14:23:37.944195+0000","last_active":"2024-06-29T14:23:37.948729+0000","last_peered":"2024-06-29T14:23:37.948729+0000","last_clean":"2024-06-29T14:23:37.948729+0000","last_became_active":"2024-06-29T14:23:37.943886+0000","last_became_peered":"2024-06-29T14:23:37.943886+0000","last_unstale":"2024-06-29T14:23:37.948729+0000","last_undegraded":"2024-06-29T14:23:37.948729+0000","last_fullsized":"2024-06-29T14:23:37.948729+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:02.807604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925935+0000","last_change":"2024-06-29T14:23:37.952875+0000","last_active":"2024-06-29T14:23:38.925935+0000","last_peered":"2024-06-29T14:23:38.925935+0000","last_clean":"2024-06-29T14:23:38.925935+0000","last_became_active":"2024-06-29T14:23:37.952578+0000","last_became_peered":"2024-06-29T14:23:37.952578+0000","last_unstale":"2024-06-29T14:23:38.925935+0000","last_undegraded":"2024-06-29T14:23:38.925935+0000","last_fullsized":"2024-06-29T14:23:38.925935+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:36.800293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392299+0000","last_change":"2024-06-29T14:23:37.952052+0000","last_active":"2024-06-29T14:23:39.392299+0000","last_peered":"2024-06-29T14:23:39.392299+0000","last_clean":"2024-06-29T14:23:39.392299+0000","last_became_active":"2024-06-29T14:23:37.951818+0000","last_became_peered":"2024-06-29T14:23:37.951818+0000","last_unstale":"2024-06-29T14:23:39.392299+0000","last_undegraded":"2024-06-29T14:23:39.392299+0000","last_fullsized":"2024-06-29T14:23:39.392299+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:46.137017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926121+0000","last_change":"2024-06-29T14:23:37.943825+0000","last_active":"2024-06-29T14:23:38.926121+0000","last_peered":"2024-06-29T14:23:38.926121+0000","last_clean":"2024-06-29T14:23:38.926121+0000","last_became_active":"2024-06-29T14:23:37.943535+0000","last_became_peered":"2024-06-29T14:23:37.943535+0000","last_unstale":"2024-06-29T14:23:38.926121+0000","last_undegraded":"2024-06-29T14:23:38.926121+0000","last_fullsized":"2024-06-29T14:23:38.926121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:03.047121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321033+0000","last_change":"2024-06-29T14:23:37.940287+0000","last_active":"2024-06-29T14:23:40.321033+0000","last_peered":"2024-06-29T14:23:40.321033+0000","last_clean":"2024-06-29T14:23:40.321033+0000","last_became_active":"2024-06-29T14:23:37.939963+0000","last_became_peered":"2024-06-29T14:23:37.939963+0000","last_unstale":"2024-06-29T14:23:40.321033+0000","last_undegraded":"2024-06-29T14:23:40.321033+0000","last_fullsized":"2024-06-29T14:23:40.321033+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:34:32.407360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180491+0000","last_change":"2024-06-29T14:23:37.941372+0000","last_active":"2024-06-29T14:23:40.180491+0000","last_peered":"2024-06-29T14:23:40.180491+0000","last_clean":"2024-06-29T14:23:40.180491+0000","last_became_active":"2024-06-29T14:23:37.941042+0000","last_became_peered":"2024-06-29T14:23:37.941042+0000","last_unstale":"2024-06-29T14:23:40.180491+0000","last_undegraded":"2024-06-29T14:23:40.180491+0000","last_fullsized":"2024-06-29T14:23:40.180491+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:56.602359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925863+0000","last_change":"2024-06-29T14:23:37.937216+0000","last_active":"2024-06-29T14:23:38.925863+0000","last_peered":"2024-06-29T14:23:38.925863+0000","last_clean":"2024-06-29T14:23:38.925863+0000","last_became_active":"2024-06-29T14:23:37.936935+0000","last_became_peered":"2024-06-29T14:23:37.936935+0000","last_unstale":"2024-06-29T14:23:38.925863+0000","last_undegraded":"2024-06-29T14:23:38.925863+0000","last_fullsized":"2024-06-29T14:23:38.925863+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:50.182625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392695+0000","last_change":"2024-06-29T14:23:37.952896+0000","last_active":"2024-06-29T14:23:39.392695+0000","last_peered":"2024-06-29T14:23:39.392695+0000","last_clean":"2024-06-29T14:23:39.392695+0000","last_became_active":"2024-06-29T14:23:37.952598+0000","last_became_peered":"2024-06-29T14:23:37.952598+0000","last_unstale":"2024-06-29T14:23:39.392695+0000","last_undegraded":"2024-06-29T14:23:39.392695+0000","last_fullsized":"2024-06-29T14:23:39.392695+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:20.327712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392632+0000","last_change":"2024-06-29T14:23:37.946407+0000","last_active":"2024-06-29T14:23:39.392632+0000","last_peered":"2024-06-29T14:23:39.392632+0000","last_clean":"2024-06-29T14:23:39.392632+0000","last_became_active":"2024-06-29T14:23:37.946127+0000","last_became_peered":"2024-06-29T14:23:37.946127+0000","last_unstale":"2024-06-29T14:23:39.392632+0000","last_undegraded":"2024-06-29T14:23:39.392632+0000","last_fullsized":"2024-06-29T14:23:39.392632+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:09.669198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180776+0000","last_change":"2024-06-29T14:23:37.951230+0000","last_active":"2024-06-29T14:23:40.180776+0000","last_peered":"2024-06-29T14:23:40.180776+0000","last_clean":"2024-06-29T14:23:40.180776+0000","last_became_active":"2024-06-29T14:23:37.950896+0000","last_became_peered":"2024-06-29T14:23:37.950896+0000","last_unstale":"2024-06-29T14:23:40.180776+0000","last_undegraded":"2024-06-29T14:23:40.180776+0000","last_fullsized":"2024-06-29T14:23:40.180776+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:16:09.568501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180530+0000","last_change":"2024-06-29T14:23:37.951396+0000","last_active":"2024-06-29T14:23:40.180530+0000","last_peered":"2024-06-29T14:23:40.180530+0000","last_clean":"2024-06-29T14:23:40.180530+0000","last_became_active":"2024-06-29T14:23:37.951075+0000","last_became_peered":"2024-06-29T14:23:37.951075+0000","last_unstale":"2024-06-29T14:23:40.180530+0000","last_undegraded":"2024-06-29T14:23:40.180530+0000","last_fullsized":"2024-06-29T14:23:40.180530+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:15.819961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321071+0000","last_change":"2024-06-29T14:23:37.937200+0000","last_active":"2024-06-29T14:23:40.321071+0000","last_peered":"2024-06-29T14:23:40.321071+0000","last_clean":"2024-06-29T14:23:40.321071+0000","last_became_active":"2024-06-29T14:23:37.936889+0000","last_became_peered":"2024-06-29T14:23:37.936889+0000","last_unstale":"2024-06-29T14:23:40.321071+0000","last_undegraded":"2024-06-29T14:23:40.321071+0000","last_fullsized":"2024-06-29T14:23:40.321071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:30.663631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.948999+0000","last_change":"2024-06-29T14:23:37.942469+0000","last_active":"2024-06-29T14:23:37.948999+0000","last_peered":"2024-06-29T14:23:37.948999+0000","last_clean":"2024-06-29T14:23:37.948999+0000","last_became_active":"2024-06-29T14:23:37.942098+0000","last_became_peered":"2024-06-29T14:23:37.942098+0000","last_unstale":"2024-06-29T14:23:37.948999+0000","last_undegraded":"2024-06-29T14:23:37.948999+0000","last_fullsized":"2024-06-29T14:23:37.948999+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:46:51.326780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.940374+0000","last_change":"2024-06-29T14:23:37.936123+0000","last_active":"2024-06-29T14:23:37.940374+0000","last_peered":"2024-06-29T14:23:37.940374+0000","last_clean":"2024-06-29T14:23:37.940374+0000","last_became_active":"2024-06-29T14:23:37.935779+0000","last_became_peered":"2024-06-29T14:23:37.935779+0000","last_unstale":"2024-06-29T14:23:37.940374+0000","last_undegraded":"2024-06-29T14:23:37.940374+0000","last_fullsized":"2024-06-29T14:23:37.940374+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:56.208133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925814+0000","last_change":"2024-06-29T14:23:37.946192+0000","last_active":"2024-06-29T14:23:38.925814+0000","last_peered":"2024-06-29T14:23:38.925814+0000","last_clean":"2024-06-29T14:23:38.925814+0000","last_became_active":"2024-06-29T14:23:37.945845+0000","last_became_peered":"2024-06-29T14:23:37.945845+0000","last_unstale":"2024-06-29T14:23:38.925814+0000","last_undegraded":"2024-06-29T14:23:38.925814+0000","last_fullsized":"2024-06-29T14:23:38.925814+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:34:12.050331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925906+0000","last_change":"2024-06-29T14:23:37.935755+0000","last_active":"2024-06-29T14:23:38.925906+0000","last_peered":"2024-06-29T14:23:38.925906+0000","last_clean":"2024-06-29T14:23:38.925906+0000","last_became_active":"2024-06-29T14:23:37.935459+0000","last_became_peered":"2024-06-29T14:23:37.935459+0000","last_unstale":"2024-06-29T14:23:38.925906+0000","last_undegraded":"2024-06-29T14:23:38.925906+0000","last_fullsized":"2024-06-29T14:23:38.925906+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:46.137017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926824+0000","last_change":"2024-06-29T14:23:37.948974+0000","last_active":"2024-06-29T14:23:38.926824+0000","last_peered":"2024-06-29T14:23:38.926824+0000","last_clean":"2024-06-29T14:23:38.926824+0000","last_became_active":"2024-06-29T14:23:37.948222+0000","last_became_peered":"2024-06-29T14:23:37.948222+0000","last_unstale":"2024-06-29T14:23:38.926824+0000","last_undegraded":"2024-06-29T14:23:38.926824+0000","last_fullsized":"2024-06-29T14:23:38.926824+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:22.666511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180867+0000","last_change":"2024-06-29T14:23:37.944560+0000","last_active":"2024-06-29T14:23:40.180867+0000","last_peered":"2024-06-29T14:23:40.180867+0000","last_clean":"2024-06-29T14:23:40.180867+0000","last_became_active":"2024-06-29T14:23:37.944265+0000","last_became_peered":"2024-06-29T14:23:37.944265+0000","last_unstale":"2024-06-29T14:23:40.180867+0000","last_undegraded":"2024-06-29T14:23:40.180867+0000","last_fullsized":"2024-06-29T14:23:40.180867+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:08:09.582695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926082+0000","last_change":"2024-06-29T14:23:37.948749+0000","last_active":"2024-06-29T14:23:38.926082+0000","last_peered":"2024-06-29T14:23:38.926082+0000","last_clean":"2024-06-29T14:23:38.926082+0000","last_became_active":"2024-06-29T14:23:37.948288+0000","last_became_peered":"2024-06-29T14:23:37.948288+0000","last_unstale":"2024-06-29T14:23:38.926082+0000","last_undegraded":"2024-06-29T14:23:38.926082+0000","last_fullsized":"2024-06-29T14:23:38.926082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:27:09.623791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926273+0000","last_change":"2024-06-29T14:23:37.939162+0000","last_active":"2024-06-29T14:23:38.926273+0000","last_peered":"2024-06-29T14:23:38.926273+0000","last_clean":"2024-06-29T14:23:38.926273+0000","last_became_active":"2024-06-29T14:23:37.938886+0000","last_became_peered":"2024-06-29T14:23:37.938886+0000","last_unstale":"2024-06-29T14:23:38.926273+0000","last_undegraded":"2024-06-29T14:23:38.926273+0000","last_fullsized":"2024-06-29T14:23:38.926273+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:09.669198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.942322+0000","last_change":"2024-06-29T14:23:37.936801+0000","last_active":"2024-06-29T14:23:37.942322+0000","last_peered":"2024-06-29T14:23:37.942322+0000","last_clean":"2024-06-29T14:23:37.942322+0000","last_became_active":"2024-06-29T14:23:37.936512+0000","last_became_peered":"2024-06-29T14:23:37.936512+0000","last_unstale":"2024-06-29T14:23:37.942322+0000","last_undegraded":"2024-06-29T14:23:37.942322+0000","last_fullsized":"2024-06-29T14:23:37.942322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:17.132446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392383+0000","last_change":"2024-06-29T14:23:37.946974+0000","last_active":"2024-06-29T14:23:39.392383+0000","last_peered":"2024-06-29T14:23:39.392383+0000","last_clean":"2024-06-29T14:23:39.392383+0000","last_became_active":"2024-06-29T14:23:37.946635+0000","last_became_peered":"2024-06-29T14:23:37.946635+0000","last_unstale":"2024-06-29T14:23:39.392383+0000","last_undegraded":"2024-06-29T14:23:39.392383+0000","last_fullsized":"2024-06-29T14:23:39.392383+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:03.047121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.322024+0000","last_change":"2024-06-29T14:23:37.949738+0000","last_active":"2024-06-29T14:23:40.322024+0000","last_peered":"2024-06-29T14:23:40.322024+0000","last_clean":"2024-06-29T14:23:40.322024+0000","last_became_active":"2024-06-29T14:23:37.949439+0000","last_became_peered":"2024-06-29T14:23:37.949439+0000","last_unstale":"2024-06-29T14:23:40.322024+0000","last_undegraded":"2024-06-29T14:23:40.322024+0000","last_fullsized":"2024-06-29T14:23:40.322024+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:34:12.050331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392349+0000","last_change":"2024-06-29T14:23:37.944346+0000","last_active":"2024-06-29T14:23:39.392349+0000","last_peered":"2024-06-29T14:23:39.392349+0000","last_clean":"2024-06-29T14:23:39.392349+0000","last_became_active":"2024-06-29T14:23:37.943733+0000","last_became_peered":"2024-06-29T14:23:37.943733+0000","last_unstale":"2024-06-29T14:23:39.392349+0000","last_undegraded":"2024-06-29T14:23:39.392349+0000","last_fullsized":"2024-06-29T14:23:39.392349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:05.123440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926144+0000","last_change":"2024-06-29T14:23:37.939593+0000","last_active":"2024-06-29T14:23:38.926144+0000","last_peered":"2024-06-29T14:23:38.926144+0000","last_clean":"2024-06-29T14:23:38.926144+0000","last_became_active":"2024-06-29T14:23:37.939247+0000","last_became_peered":"2024-06-29T14:23:37.939247+0000","last_unstale":"2024-06-29T14:23:38.926144+0000","last_undegraded":"2024-06-29T14:23:38.926144+0000","last_fullsized":"2024-06-29T14:23:38.926144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:41.926725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.322107+0000","last_change":"2024-06-29T14:23:36.939194+0000","last_active":"2024-06-29T14:23:40.322107+0000","last_peered":"2024-06-29T14:23:40.322107+0000","last_clean":"2024-06-29T14:23:40.322107+0000","last_became_active":"2024-06-29T14:23:36.938887+0000","last_became_peered":"2024-06-29T14:23:36.938887+0000","last_unstale":"2024-06-29T14:23:40.322107+0000","last_undegraded":"2024-06-29T14:23:40.322107+0000","last_fullsized":"2024-06-29T14:23:40.322107+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:47.733035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.322079+0000","last_change":"2024-06-29T14:23:37.941082+0000","last_active":"2024-06-29T14:23:40.322079+0000","last_peered":"2024-06-29T14:23:40.322079+0000","last_clean":"2024-06-29T14:23:40.322079+0000","last_became_active":"2024-06-29T14:23:37.940795+0000","last_became_peered":"2024-06-29T14:23:37.940795+0000","last_unstale":"2024-06-29T14:23:40.322079+0000","last_undegraded":"2024-06-29T14:23:40.322079+0000","last_fullsized":"2024-06-29T14:23:40.322079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:41.926725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926255+0000","last_change":"2024-06-29T14:23:36.938779+0000","last_active":"2024-06-29T14:23:38.926255+0000","last_peered":"2024-06-29T14:23:38.926255+0000","last_clean":"2024-06-29T14:23:38.926255+0000","last_became_active":"2024-06-29T14:23:36.938479+0000","last_became_peered":"2024-06-29T14:23:36.938479+0000","last_unstale":"2024-06-29T14:23:38.926255+0000","last_undegraded":"2024-06-29T14:23:38.926255+0000","last_fullsized":"2024-06-29T14:23:38.926255+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:55.200059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.956745+0000","last_change":"2024-06-29T14:23:37.945859+0000","last_active":"2024-06-29T14:23:37.956745+0000","last_peered":"2024-06-29T14:23:37.956745+0000","last_clean":"2024-06-29T14:23:37.956745+0000","last_became_active":"2024-06-29T14:23:37.944779+0000","last_became_peered":"2024-06-29T14:23:37.944779+0000","last_unstale":"2024-06-29T14:23:37.956745+0000","last_undegraded":"2024-06-29T14:23:37.956745+0000","last_fullsized":"2024-06-29T14:23:37.956745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:28:33.557016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180572+0000","last_change":"2024-06-29T14:23:36.946478+0000","last_active":"2024-06-29T14:23:40.180572+0000","last_peered":"2024-06-29T14:23:40.180572+0000","last_clean":"2024-06-29T14:23:40.180572+0000","last_became_active":"2024-06-29T14:23:36.946090+0000","last_became_peered":"2024-06-29T14:23:36.946090+0000","last_unstale":"2024-06-29T14:23:40.180572+0000","last_undegraded":"2024-06-29T14:23:40.180572+0000","last_fullsized":"2024-06-29T14:23:40.180572+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:29.046411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926239+0000","last_change":"2024-06-29T14:23:37.945392+0000","last_active":"2024-06-29T14:23:38.926239+0000","last_peered":"2024-06-29T14:23:38.926239+0000","last_clean":"2024-06-29T14:23:38.926239+0000","last_became_active":"2024-06-29T14:23:37.944984+0000","last_became_peered":"2024-06-29T14:23:37.944984+0000","last_unstale":"2024-06-29T14:23:38.926239+0000","last_undegraded":"2024-06-29T14:23:38.926239+0000","last_fullsized":"2024-06-29T14:23:38.926239+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:46.137017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321117+0000","last_change":"2024-06-29T14:23:36.938376+0000","last_active":"2024-06-29T14:23:40.321117+0000","last_peered":"2024-06-29T14:23:40.321117+0000","last_clean":"2024-06-29T14:23:40.321117+0000","last_became_active":"2024-06-29T14:23:36.937882+0000","last_became_peered":"2024-06-29T14:23:36.937882+0000","last_unstale":"2024-06-29T14:23:40.321117+0000","last_undegraded":"2024-06-29T14:23:40.321117+0000","last_fullsized":"2024-06-29T14:23:40.321117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:32:22.490571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392343+0000","last_change":"2024-06-29T14:23:37.942821+0000","last_active":"2024-06-29T14:23:39.392343+0000","last_peered":"2024-06-29T14:23:39.392343+0000","last_clean":"2024-06-29T14:23:39.392343+0000","last_became_active":"2024-06-29T14:23:37.942496+0000","last_became_peered":"2024-06-29T14:23:37.942496+0000","last_unstale":"2024-06-29T14:23:39.392343+0000","last_undegraded":"2024-06-29T14:23:39.392343+0000","last_fullsized":"2024-06-29T14:23:39.392343+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:22.666511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.960028+0000","last_change":"2024-06-29T14:23:36.947122+0000","last_active":"2024-06-29T14:23:37.960028+0000","last_peered":"2024-06-29T14:23:37.960028+0000","last_clean":"2024-06-29T14:23:37.960028+0000","last_became_active":"2024-06-29T14:23:36.946730+0000","last_became_peered":"2024-06-29T14:23:36.946730+0000","last_unstale":"2024-06-29T14:23:37.960028+0000","last_undegraded":"2024-06-29T14:23:37.960028+0000","last_fullsized":"2024-06-29T14:23:37.960028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:53.868147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926257+0000","last_change":"2024-06-29T14:23:37.944325+0000","last_active":"2024-06-29T14:23:38.926257+0000","last_peered":"2024-06-29T14:23:38.926257+0000","last_clean":"2024-06-29T14:23:38.926257+0000","last_became_active":"2024-06-29T14:23:37.943973+0000","last_became_peered":"2024-06-29T14:23:37.943973+0000","last_unstale":"2024-06-29T14:23:38.926257+0000","last_undegraded":"2024-06-29T14:23:38.926257+0000","last_fullsized":"2024-06-29T14:23:38.926257+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:52.758662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.959951+0000","last_change":"2024-06-29T14:23:36.947050+0000","last_active":"2024-06-29T14:23:37.959951+0000","last_peered":"2024-06-29T14:23:37.959951+0000","last_clean":"2024-06-29T14:23:37.959951+0000","last_became_active":"2024-06-29T14:23:36.946695+0000","last_became_peered":"2024-06-29T14:23:36.946695+0000","last_unstale":"2024-06-29T14:23:37.959951+0000","last_undegraded":"2024-06-29T14:23:37.959951+0000","last_fullsized":"2024-06-29T14:23:37.959951+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:13:42.529115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.942022+0000","last_change":"2024-06-29T14:23:37.937537+0000","last_active":"2024-06-29T14:23:37.942022+0000","last_peered":"2024-06-29T14:23:37.942022+0000","last_clean":"2024-06-29T14:23:37.942022+0000","last_became_active":"2024-06-29T14:23:37.936920+0000","last_became_peered":"2024-06-29T14:23:37.936920+0000","last_unstale":"2024-06-29T14:23:37.942022+0000","last_undegraded":"2024-06-29T14:23:37.942022+0000","last_fullsized":"2024-06-29T14:23:37.942022+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:58.882288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180215+0000","last_change":"2024-06-29T14:23:36.939287+0000","last_active":"2024-06-29T14:23:40.180215+0000","last_peered":"2024-06-29T14:23:40.180215+0000","last_clean":"2024-06-29T14:23:40.180215+0000","last_became_active":"2024-06-29T14:23:36.938934+0000","last_became_peered":"2024-06-29T14:23:36.938934+0000","last_unstale":"2024-06-29T14:23:40.180215+0000","last_undegraded":"2024-06-29T14:23:40.180215+0000","last_fullsized":"2024-06-29T14:23:40.180215+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:34:11.042257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925788+0000","last_change":"2024-06-29T14:23:37.938497+0000","last_active":"2024-06-29T14:23:38.925788+0000","last_peered":"2024-06-29T14:23:38.925788+0000","last_clean":"2024-06-29T14:23:38.925788+0000","last_became_active":"2024-06-29T14:23:37.938217+0000","last_became_peered":"2024-06-29T14:23:37.938217+0000","last_unstale":"2024-06-29T14:23:38.925788+0000","last_undegraded":"2024-06-29T14:23:38.925788+0000","last_fullsized":"2024-06-29T14:23:38.925788+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:54:09.878998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.926031+0000","last_change":"2024-06-29T14:23:36.945009+0000","last_active":"2024-06-29T14:23:37.926031+0000","last_peered":"2024-06-29T14:23:37.926031+0000","last_clean":"2024-06-29T14:23:37.926031+0000","last_became_active":"2024-06-29T14:23:36.944318+0000","last_became_peered":"2024-06-29T14:23:36.944318+0000","last_unstale":"2024-06-29T14:23:37.926031+0000","last_undegraded":"2024-06-29T14:23:37.926031+0000","last_fullsized":"2024-06-29T14:23:37.926031+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:11.303942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926008+0000","last_change":"2024-06-29T14:23:37.941701+0000","last_active":"2024-06-29T14:23:38.926008+0000","last_peered":"2024-06-29T14:23:38.926008+0000","last_clean":"2024-06-29T14:23:38.926008+0000","last_became_active":"2024-06-29T14:23:37.941410+0000","last_became_peered":"2024-06-29T14:23:37.941410+0000","last_unstale":"2024-06-29T14:23:38.926008+0000","last_undegraded":"2024-06-29T14:23:38.926008+0000","last_fullsized":"2024-06-29T14:23:38.926008+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:05.123440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180881+0000","last_change":"2024-06-29T14:23:36.937096+0000","last_active":"2024-06-29T14:23:40.180881+0000","last_peered":"2024-06-29T14:23:40.180881+0000","last_clean":"2024-06-29T14:23:40.180881+0000","last_became_active":"2024-06-29T14:23:36.936836+0000","last_became_peered":"2024-06-29T14:23:36.936836+0000","last_unstale":"2024-06-29T14:23:40.180881+0000","last_undegraded":"2024-06-29T14:23:40.180881+0000","last_fullsized":"2024-06-29T14:23:40.180881+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:22.401053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180823+0000","last_change":"2024-06-29T14:23:37.950645+0000","last_active":"2024-06-29T14:23:40.180823+0000","last_peered":"2024-06-29T14:23:40.180823+0000","last_clean":"2024-06-29T14:23:40.180823+0000","last_became_active":"2024-06-29T14:23:37.950312+0000","last_became_peered":"2024-06-29T14:23:37.950312+0000","last_unstale":"2024-06-29T14:23:40.180823+0000","last_undegraded":"2024-06-29T14:23:40.180823+0000","last_fullsized":"2024-06-29T14:23:40.180823+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:33.856511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926412+0000","last_change":"2024-06-29T14:23:36.941120+0000","last_active":"2024-06-29T14:23:38.926412+0000","last_peered":"2024-06-29T14:23:38.926412+0000","last_clean":"2024-06-29T14:23:38.926412+0000","last_became_active":"2024-06-29T14:23:36.940820+0000","last_became_peered":"2024-06-29T14:23:36.940820+0000","last_unstale":"2024-06-29T14:23:38.926412+0000","last_undegraded":"2024-06-29T14:23:38.926412+0000","last_fullsized":"2024-06-29T14:23:38.926412+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:57.874214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180564+0000","last_change":"2024-06-29T14:23:37.942391+0000","last_active":"2024-06-29T14:23:40.180564+0000","last_peered":"2024-06-29T14:23:40.180564+0000","last_clean":"2024-06-29T14:23:40.180564+0000","last_became_active":"2024-06-29T14:23:37.942007+0000","last_became_peered":"2024-06-29T14:23:37.942007+0000","last_unstale":"2024-06-29T14:23:40.180564+0000","last_undegraded":"2024-06-29T14:23:40.180564+0000","last_fullsized":"2024-06-29T14:23:40.180564+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:32:14.563469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926090+0000","last_change":"2024-06-29T14:23:36.938132+0000","last_active":"2024-06-29T14:23:38.926090+0000","last_peered":"2024-06-29T14:23:38.926090+0000","last_clean":"2024-06-29T14:23:38.926090+0000","last_became_active":"2024-06-29T14:23:36.937786+0000","last_became_peered":"2024-06-29T14:23:36.937786+0000","last_unstale":"2024-06-29T14:23:38.926090+0000","last_undegraded":"2024-06-29T14:23:38.926090+0000","last_fullsized":"2024-06-29T14:23:38.926090+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:46:50.318706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926212+0000","last_change":"2024-06-29T14:23:37.950108+0000","last_active":"2024-06-29T14:23:38.926212+0000","last_peered":"2024-06-29T14:23:38.926212+0000","last_clean":"2024-06-29T14:23:38.926212+0000","last_became_active":"2024-06-29T14:23:37.949817+0000","last_became_peered":"2024-06-29T14:23:37.949817+0000","last_unstale":"2024-06-29T14:23:38.926212+0000","last_undegraded":"2024-06-29T14:23:38.926212+0000","last_fullsized":"2024-06-29T14:23:38.926212+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:21.205455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180554+0000","last_change":"2024-06-29T14:23:36.936852+0000","last_active":"2024-06-29T14:23:40.180554+0000","last_peered":"2024-06-29T14:23:40.180554+0000","last_clean":"2024-06-29T14:23:40.180554+0000","last_became_active":"2024-06-29T14:23:36.936550+0000","last_became_peered":"2024-06-29T14:23:36.936550+0000","last_unstale":"2024-06-29T14:23:40.180554+0000","last_undegraded":"2024-06-29T14:23:40.180554+0000","last_fullsized":"2024-06-29T14:23:40.180554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:40.918651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926035+0000","last_change":"2024-06-29T14:23:37.937924+0000","last_active":"2024-06-29T14:23:38.926035+0000","last_peered":"2024-06-29T14:23:38.926035+0000","last_clean":"2024-06-29T14:23:38.926035+0000","last_became_active":"2024-06-29T14:23:37.937618+0000","last_became_peered":"2024-06-29T14:23:37.937618+0000","last_unstale":"2024-06-29T14:23:38.926035+0000","last_undegraded":"2024-06-29T14:23:38.926035+0000","last_fullsized":"2024-06-29T14:23:38.926035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:48:59.553463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.960132+0000","last_change":"2024-06-29T14:23:36.944778+0000","last_active":"2024-06-29T14:23:37.960132+0000","last_peered":"2024-06-29T14:23:37.960132+0000","last_clean":"2024-06-29T14:23:37.960132+0000","last_became_active":"2024-06-29T14:23:36.944438+0000","last_became_peered":"2024-06-29T14:23:36.944438+0000","last_unstale":"2024-06-29T14:23:37.960132+0000","last_undegraded":"2024-06-29T14:23:37.960132+0000","last_fullsized":"2024-06-29T14:23:37.960132+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:47.733035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180504+0000","last_change":"2024-06-29T14:23:37.944354+0000","last_active":"2024-06-29T14:23:40.180504+0000","last_peered":"2024-06-29T14:23:40.180504+0000","last_clean":"2024-06-29T14:23:40.180504+0000","last_became_active":"2024-06-29T14:23:37.944020+0000","last_became_peered":"2024-06-29T14:23:37.944020+0000","last_unstale":"2024-06-29T14:23:40.180504+0000","last_undegraded":"2024-06-29T14:23:40.180504+0000","last_fullsized":"2024-06-29T14:23:40.180504+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:52:17.539322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926155+0000","last_change":"2024-06-29T14:23:36.938434+0000","last_active":"2024-06-29T14:23:38.926155+0000","last_peered":"2024-06-29T14:23:38.926155+0000","last_clean":"2024-06-29T14:23:38.926155+0000","last_became_active":"2024-06-29T14:23:36.937921+0000","last_became_peered":"2024-06-29T14:23:36.937921+0000","last_unstale":"2024-06-29T14:23:38.926155+0000","last_undegraded":"2024-06-29T14:23:38.926155+0000","last_fullsized":"2024-06-29T14:23:38.926155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:33:55.200059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392561+0000","last_change":"2024-06-29T14:23:37.944500+0000","last_active":"2024-06-29T14:23:39.392561+0000","last_peered":"2024-06-29T14:23:39.392561+0000","last_clean":"2024-06-29T14:23:39.392561+0000","last_became_active":"2024-06-29T14:23:37.944081+0000","last_became_peered":"2024-06-29T14:23:37.944081+0000","last_unstale":"2024-06-29T14:23:39.392561+0000","last_undegraded":"2024-06-29T14:23:39.392561+0000","last_fullsized":"2024-06-29T14:23:39.392561+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:21.205455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180155+0000","last_change":"2024-06-29T14:23:36.940884+0000","last_active":"2024-06-29T14:23:40.180155+0000","last_peered":"2024-06-29T14:23:40.180155+0000","last_clean":"2024-06-29T14:23:40.180155+0000","last_became_active":"2024-06-29T14:23:36.940548+0000","last_became_peered":"2024-06-29T14:23:36.940548+0000","last_unstale":"2024-06-29T14:23:40.180155+0000","last_undegraded":"2024-06-29T14:23:40.180155+0000","last_fullsized":"2024-06-29T14:23:40.180155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:01.799530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.322009+0000","last_change":"2024-06-29T14:23:37.941136+0000","last_active":"2024-06-29T14:23:40.322009+0000","last_peered":"2024-06-29T14:23:40.322009+0000","last_clean":"2024-06-29T14:23:40.322009+0000","last_became_active":"2024-06-29T14:23:37.940870+0000","last_became_peered":"2024-06-29T14:23:37.940870+0000","last_unstale":"2024-06-29T14:23:40.322009+0000","last_undegraded":"2024-06-29T14:23:40.322009+0000","last_fullsized":"2024-06-29T14:23:40.322009+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:28:33.557016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180188+0000","last_change":"2024-06-29T14:23:36.946906+0000","last_active":"2024-06-29T14:23:40.180188+0000","last_peered":"2024-06-29T14:23:40.180188+0000","last_clean":"2024-06-29T14:23:40.180188+0000","last_became_active":"2024-06-29T14:23:36.946599+0000","last_became_peered":"2024-06-29T14:23:36.946599+0000","last_unstale":"2024-06-29T14:23:40.180188+0000","last_undegraded":"2024-06-29T14:23:40.180188+0000","last_fullsized":"2024-06-29T14:23:40.180188+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:22.975505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.940245+0000","last_change":"2024-06-29T14:23:37.934667+0000","last_active":"2024-06-29T14:23:37.940245+0000","last_peered":"2024-06-29T14:23:37.940245+0000","last_clean":"2024-06-29T14:23:37.940245+0000","last_became_active":"2024-06-29T14:23:37.934352+0000","last_became_peered":"2024-06-29T14:23:37.934352+0000","last_unstale":"2024-06-29T14:23:37.940245+0000","last_undegraded":"2024-06-29T14:23:37.940245+0000","last_fullsized":"2024-06-29T14:23:37.940245+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:30.663631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926957+0000","last_change":"2024-06-29T14:23:36.941653+0000","last_active":"2024-06-29T14:23:38.926957+0000","last_peered":"2024-06-29T14:23:38.926957+0000","last_clean":"2024-06-29T14:23:38.926957+0000","last_became_active":"2024-06-29T14:23:36.941327+0000","last_became_peered":"2024-06-29T14:23:36.941327+0000","last_unstale":"2024-06-29T14:23:38.926957+0000","last_undegraded":"2024-06-29T14:23:38.926957+0000","last_fullsized":"2024-06-29T14:23:38.926957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:22.975505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926199+0000","last_change":"2024-06-29T14:23:37.934697+0000","last_active":"2024-06-29T14:23:38.926199+0000","last_peered":"2024-06-29T14:23:38.926199+0000","last_clean":"2024-06-29T14:23:38.926199+0000","last_became_active":"2024-06-29T14:23:37.934409+0000","last_became_peered":"2024-06-29T14:23:37.934409+0000","last_unstale":"2024-06-29T14:23:38.926199+0000","last_undegraded":"2024-06-29T14:23:38.926199+0000","last_fullsized":"2024-06-29T14:23:38.926199+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:57.496214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.960031+0000","last_change":"2024-06-29T14:23:36.945192+0000","last_active":"2024-06-29T14:23:37.960031+0000","last_peered":"2024-06-29T14:23:37.960031+0000","last_clean":"2024-06-29T14:23:37.960031+0000","last_became_active":"2024-06-29T14:23:36.944784+0000","last_became_peered":"2024-06-29T14:23:36.944784+0000","last_unstale":"2024-06-29T14:23:37.960031+0000","last_undegraded":"2024-06-29T14:23:37.960031+0000","last_fullsized":"2024-06-29T14:23:37.960031+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:01:51.989032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926458+0000","last_change":"2024-06-29T14:23:37.948937+0000","last_active":"2024-06-29T14:23:38.926458+0000","last_peered":"2024-06-29T14:23:38.926458+0000","last_clean":"2024-06-29T14:23:38.926458+0000","last_became_active":"2024-06-29T14:23:37.943575+0000","last_became_peered":"2024-06-29T14:23:37.943575+0000","last_unstale":"2024-06-29T14:23:38.926458+0000","last_undegraded":"2024-06-29T14:23:38.926458+0000","last_fullsized":"2024-06-29T14:23:38.926458+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:52.758662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.960142+0000","last_change":"2024-06-29T14:23:36.945440+0000","last_active":"2024-06-29T14:23:37.960142+0000","last_peered":"2024-06-29T14:23:37.960142+0000","last_clean":"2024-06-29T14:23:37.960142+0000","last_became_active":"2024-06-29T14:23:36.944489+0000","last_became_peered":"2024-06-29T14:23:36.944489+0000","last_unstale":"2024-06-29T14:23:37.960142+0000","last_undegraded":"2024-06-29T14:23:37.960142+0000","last_fullsized":"2024-06-29T14:23:37.960142+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:34:31.399286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321055+0000","last_change":"2024-06-29T14:23:37.942181+0000","last_active":"2024-06-29T14:23:40.321055+0000","last_peered":"2024-06-29T14:23:40.321055+0000","last_clean":"2024-06-29T14:23:40.321055+0000","last_became_active":"2024-06-29T14:23:37.941910+0000","last_became_peered":"2024-06-29T14:23:37.941910+0000","last_unstale":"2024-06-29T14:23:40.321055+0000","last_undegraded":"2024-06-29T14:23:40.321055+0000","last_fullsized":"2024-06-29T14:23:40.321055+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:48.741109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.926262+0000","last_change":"2024-06-29T14:23:36.944581+0000","last_active":"2024-06-29T14:23:37.926262+0000","last_peered":"2024-06-29T14:23:37.926262+0000","last_clean":"2024-06-29T14:23:37.926262+0000","last_became_active":"2024-06-29T14:23:36.944286+0000","last_became_peered":"2024-06-29T14:23:36.944286+0000","last_unstale":"2024-06-29T14:23:37.926262+0000","last_undegraded":"2024-06-29T14:23:37.926262+0000","last_fullsized":"2024-06-29T14:23:37.926262+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:43.777161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321025+0000","last_change":"2024-06-29T14:23:37.933659+0000","last_active":"2024-06-29T14:23:40.321025+0000","last_peered":"2024-06-29T14:23:40.321025+0000","last_clean":"2024-06-29T14:23:40.321025+0000","last_became_active":"2024-06-29T14:23:37.933328+0000","last_became_peered":"2024-06-29T14:23:37.933328+0000","last_unstale":"2024-06-29T14:23:40.321025+0000","last_undegraded":"2024-06-29T14:23:40.321025+0000","last_fullsized":"2024-06-29T14:23:40.321025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:44.785235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926759+0000","last_change":"2024-06-29T14:23:36.942749+0000","last_active":"2024-06-29T14:23:38.926759+0000","last_peered":"2024-06-29T14:23:38.926759+0000","last_clean":"2024-06-29T14:23:38.926759+0000","last_became_active":"2024-06-29T14:23:36.942353+0000","last_became_peered":"2024-06-29T14:23:36.942353+0000","last_unstale":"2024-06-29T14:23:38.926759+0000","last_undegraded":"2024-06-29T14:23:38.926759+0000","last_fullsized":"2024-06-29T14:23:38.926759+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:47.733035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321166+0000","last_change":"2024-06-29T14:23:37.947132+0000","last_active":"2024-06-29T14:23:40.321166+0000","last_peered":"2024-06-29T14:23:40.321166+0000","last_clean":"2024-06-29T14:23:40.321166+0000","last_became_active":"2024-06-29T14:23:37.943954+0000","last_became_peered":"2024-06-29T14:23:37.943954+0000","last_unstale":"2024-06-29T14:23:40.321166+0000","last_undegraded":"2024-06-29T14:23:40.321166+0000","last_fullsized":"2024-06-29T14:23:40.321166+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:17.132446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321819+0000","last_change":"2024-06-29T14:23:36.945816+0000","last_active":"2024-06-29T14:23:40.321819+0000","last_peered":"2024-06-29T14:23:40.321819+0000","last_clean":"2024-06-29T14:23:40.321819+0000","last_became_active":"2024-06-29T14:23:36.945518+0000","last_became_peered":"2024-06-29T14:23:36.945518+0000","last_unstale":"2024-06-29T14:23:40.321819+0000","last_undegraded":"2024-06-29T14:23:40.321819+0000","last_fullsized":"2024-06-29T14:23:40.321819+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:53.868147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.320977+0000","last_change":"2024-06-29T14:23:37.947172+0000","last_active":"2024-06-29T14:23:40.320977+0000","last_peered":"2024-06-29T14:23:40.320977+0000","last_clean":"2024-06-29T14:23:40.320977+0000","last_became_active":"2024-06-29T14:23:37.940451+0000","last_became_peered":"2024-06-29T14:23:37.940451+0000","last_unstale":"2024-06-29T14:23:40.320977+0000","last_undegraded":"2024-06-29T14:23:40.320977+0000","last_fullsized":"2024-06-29T14:23:40.320977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:12.312016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.925471+0000","last_change":"2024-06-29T14:23:36.946365+0000","last_active":"2024-06-29T14:23:37.925471+0000","last_peered":"2024-06-29T14:23:37.925471+0000","last_clean":"2024-06-29T14:23:37.925471+0000","last_became_active":"2024-06-29T14:23:36.946079+0000","last_became_peered":"2024-06-29T14:23:36.946079+0000","last_unstale":"2024-06-29T14:23:37.925471+0000","last_undegraded":"2024-06-29T14:23:37.925471+0000","last_fullsized":"2024-06-29T14:23:37.925471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:10.736740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392258+0000","last_change":"2024-06-29T14:23:37.945266+0000","last_active":"2024-06-29T14:23:39.392258+0000","last_peered":"2024-06-29T14:23:39.392258+0000","last_clean":"2024-06-29T14:23:39.392258+0000","last_became_active":"2024-06-29T14:23:37.944936+0000","last_became_peered":"2024-06-29T14:23:37.944936+0000","last_unstale":"2024-06-29T14:23:39.392258+0000","last_undegraded":"2024-06-29T14:23:39.392258+0000","last_fullsized":"2024-06-29T14:23:39.392258+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:46:50.182625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.925722+0000","last_change":"2024-06-29T14:23:36.938180+0000","last_active":"2024-06-29T14:23:38.925722+0000","last_peered":"2024-06-29T14:23:38.925722+0000","last_clean":"2024-06-29T14:23:38.925722+0000","last_became_active":"2024-06-29T14:23:36.937807+0000","last_became_peered":"2024-06-29T14:23:36.937807+0000","last_unstale":"2024-06-29T14:23:38.925722+0000","last_undegraded":"2024-06-29T14:23:38.925722+0000","last_fullsized":"2024-06-29T14:23:38.925722+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:57.874214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321035+0000","last_change":"2024-06-29T14:23:37.938990+0000","last_active":"2024-06-29T14:23:40.321035+0000","last_peered":"2024-06-29T14:23:40.321035+0000","last_clean":"2024-06-29T14:23:40.321035+0000","last_became_active":"2024-06-29T14:23:37.938698+0000","last_became_peered":"2024-06-29T14:23:37.938698+0000","last_unstale":"2024-06-29T14:23:40.321035+0000","last_undegraded":"2024-06-29T14:23:40.321035+0000","last_fullsized":"2024-06-29T14:23:40.321035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:54.876220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.927033+0000","last_change":"2024-06-29T14:23:36.941461+0000","last_active":"2024-06-29T14:23:38.927033+0000","last_peered":"2024-06-29T14:23:38.927033+0000","last_clean":"2024-06-29T14:23:38.927033+0000","last_became_active":"2024-06-29T14:23:36.941167+0000","last_became_peered":"2024-06-29T14:23:36.941167+0000","last_unstale":"2024-06-29T14:23:38.927033+0000","last_undegraded":"2024-06-29T14:23:38.927033+0000","last_fullsized":"2024-06-29T14:23:38.927033+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:01.799530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926334+0000","last_change":"2024-06-29T14:23:37.944718+0000","last_active":"2024-06-29T14:23:38.926334+0000","last_peered":"2024-06-29T14:23:38.926334+0000","last_clean":"2024-06-29T14:23:38.926334+0000","last_became_active":"2024-06-29T14:23:37.943224+0000","last_became_peered":"2024-06-29T14:23:37.943224+0000","last_unstale":"2024-06-29T14:23:38.926334+0000","last_undegraded":"2024-06-29T14:23:38.926334+0000","last_fullsized":"2024-06-29T14:23:38.926334+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:20.327712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.928505+0000","last_change":"2024-06-29T14:23:36.937915+0000","last_active":"2024-06-29T14:23:38.928505+0000","last_peered":"2024-06-29T14:23:38.928505+0000","last_clean":"2024-06-29T14:23:38.928505+0000","last_became_active":"2024-06-29T14:23:36.937596+0000","last_became_peered":"2024-06-29T14:23:36.937596+0000","last_unstale":"2024-06-29T14:23:38.928505+0000","last_undegraded":"2024-06-29T14:23:38.928505+0000","last_fullsized":"2024-06-29T14:23:38.928505+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:16.124372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.927036+0000","last_change":"2024-06-29T14:23:37.955845+0000","last_active":"2024-06-29T14:23:38.927036+0000","last_peered":"2024-06-29T14:23:38.927036+0000","last_clean":"2024-06-29T14:23:38.927036+0000","last_became_active":"2024-06-29T14:23:37.955464+0000","last_became_peered":"2024-06-29T14:23:37.955464+0000","last_unstale":"2024-06-29T14:23:38.927036+0000","last_undegraded":"2024-06-29T14:23:38.927036+0000","last_fullsized":"2024-06-29T14:23:38.927036+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:23.409126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926500+0000","last_change":"2024-06-29T14:23:36.936705+0000","last_active":"2024-06-29T14:23:38.926500+0000","last_peered":"2024-06-29T14:23:38.926500+0000","last_clean":"2024-06-29T14:23:38.926500+0000","last_became_active":"2024-06-29T14:23:36.936394+0000","last_became_peered":"2024-06-29T14:23:36.936394+0000","last_unstale":"2024-06-29T14:23:38.926500+0000","last_undegraded":"2024-06-29T14:23:38.926500+0000","last_fullsized":"2024-06-29T14:23:38.926500+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:57.874214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321598+0000","last_change":"2024-06-29T14:23:37.936616+0000","last_active":"2024-06-29T14:23:40.321598+0000","last_peered":"2024-06-29T14:23:40.321598+0000","last_clean":"2024-06-29T14:23:40.321598+0000","last_became_active":"2024-06-29T14:23:37.936318+0000","last_became_peered":"2024-06-29T14:23:37.936318+0000","last_unstale":"2024-06-29T14:23:40.321598+0000","last_undegraded":"2024-06-29T14:23:40.321598+0000","last_fullsized":"2024-06-29T14:23:40.321598+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:55:02.807604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926981+0000","last_change":"2024-06-29T14:23:36.939546+0000","last_active":"2024-06-29T14:23:38.926981+0000","last_peered":"2024-06-29T14:23:38.926981+0000","last_clean":"2024-06-29T14:23:38.926981+0000","last_became_active":"2024-06-29T14:23:36.939256+0000","last_became_peered":"2024-06-29T14:23:36.939256+0000","last_unstale":"2024-06-29T14:23:38.926981+0000","last_undegraded":"2024-06-29T14:23:38.926981+0000","last_fullsized":"2024-06-29T14:23:38.926981+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:29.655557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.927187+0000","last_change":"2024-06-29T14:23:37.947953+0000","last_active":"2024-06-29T14:23:38.927187+0000","last_peered":"2024-06-29T14:23:38.927187+0000","last_clean":"2024-06-29T14:23:38.927187+0000","last_became_active":"2024-06-29T14:23:37.947655+0000","last_became_peered":"2024-06-29T14:23:37.947655+0000","last_unstale":"2024-06-29T14:23:38.927187+0000","last_undegraded":"2024-06-29T14:23:38.927187+0000","last_fullsized":"2024-06-29T14:23:38.927187+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:04:30.054485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:38.926258+0000","last_change":"2024-06-29T14:23:36.930826+0000","last_active":"2024-06-29T14:23:38.926258+0000","last_peered":"2024-06-29T14:23:38.926258+0000","last_clean":"2024-06-29T14:23:38.926258+0000","last_became_active":"2024-06-29T14:23:36.930509+0000","last_became_peered":"2024-06-29T14:23:36.930509+0000","last_unstale":"2024-06-29T14:23:38.926258+0000","last_undegraded":"2024-06-29T14:23:38.926258+0000","last_fullsized":"2024-06-29T14:23:38.926258+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:16.124372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180217+0000","last_change":"2024-06-29T14:23:37.950636+0000","last_active":"2024-06-29T14:23:40.180217+0000","last_peered":"2024-06-29T14:23:40.180217+0000","last_clean":"2024-06-29T14:23:40.180217+0000","last_became_active":"2024-06-29T14:23:37.950266+0000","last_became_peered":"2024-06-29T14:23:37.950266+0000","last_unstale":"2024-06-29T14:23:40.180217+0000","last_undegraded":"2024-06-29T14:23:40.180217+0000","last_fullsized":"2024-06-29T14:23:40.180217+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:09.669198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.925788+0000","last_change":"2024-06-29T14:23:36.942425+0000","last_active":"2024-06-29T14:23:37.925788+0000","last_peered":"2024-06-29T14:23:37.925788+0000","last_clean":"2024-06-29T14:23:37.925788+0000","last_became_active":"2024-06-29T14:23:36.942072+0000","last_became_peered":"2024-06-29T14:23:36.942072+0000","last_unstale":"2024-06-29T14:23:37.925788+0000","last_undegraded":"2024-06-29T14:23:37.925788+0000","last_fullsized":"2024-06-29T14:23:37.925788+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:05:58.738515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-06-29T14:23:37.948778+0000","last_change":"2024-06-29T14:23:37.944378+0000","last_active":"2024-06-29T14:23:37.948778+0000","last_peered":"2024-06-29T14:23:37.948778+0000","last_clean":"2024-06-29T14:23:37.948778+0000","last_became_active":"2024-06-29T14:23:37.944086+0000","last_became_peered":"2024-06-29T14:23:37.944086+0000","last_unstale":"2024-06-29T14:23:37.948778+0000","last_undegraded":"2024-06-29T14:23:37.948778+0000","last_fullsized":"2024-06-29T14:23:37.948778+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:23.983579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.180379+0000","last_change":"2024-06-29T14:23:36.938840+0000","last_active":"2024-06-29T14:23:40.180379+0000","last_peered":"2024-06-29T14:23:40.180379+0000","last_clean":"2024-06-29T14:23:40.180379+0000","last_became_active":"2024-06-29T14:23:36.938473+0000","last_became_peered":"2024-06-29T14:23:36.938473+0000","last_unstale":"2024-06-29T14:23:40.180379+0000","last_undegraded":"2024-06-29T14:23:40.180379+0000","last_fullsized":"2024-06-29T14:23:40.180379+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:28:32.548942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392331+0000","last_change":"2024-06-29T14:23:37.952183+0000","last_active":"2024-06-29T14:23:39.392331+0000","last_peered":"2024-06-29T14:23:39.392331+0000","last_clean":"2024-06-29T14:23:39.392331+0000","last_became_active":"2024-06-29T14:23:37.951905+0000","last_became_peered":"2024-06-29T14:23:37.951905+0000","last_unstale":"2024-06-29T14:23:39.392331+0000","last_undegraded":"2024-06-29T14:23:39.392331+0000","last_fullsized":"2024-06-29T14:23:39.392331+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:36.915091+0000","last_clean_scrub_stamp":"2024-06-29T14:23:36.915091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:52.758662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:40.321820+0000","last_change":"2024-06-29T14:23:36.948071+0000","last_active":"2024-06-29T14:23:40.321820+0000","last_peered":"2024-06-29T14:23:40.321820+0000","last_clean":"2024-06-29T14:23:40.321820+0000","last_became_active":"2024-06-29T14:23:36.947743+0000","last_became_peered":"2024-06-29T14:23:36.947743+0000","last_unstale":"2024-06-29T14:23:40.321820+0000","last_undegraded":"2024-06-29T14:23:40.321820+0000","last_fullsized":"2024-06-29T14:23:40.321820+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:23:35.907017+0000","last_clean_scrub_stamp":"2024-06-29T14:23:35.907017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:34:31.399286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":134,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-06-29T14:23:39.392604+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:23:39.392604+0000","last_peered":"2024-06-29T14:23:39.392604+0000","last_clean":"2024-06-29T14:23:39.392604+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:23:39.392604+0000","last_undegraded":"2024-06-29T14:23:39.392604+0000","last_fullsized":"2024-06-29T14:23:39.392604+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":736936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":368640,"data_stored":140794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":140282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":148112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":966656,"data_stored":736846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:23:42.336 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:23:42.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:23:42.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.494+0000 7f3998df5640 1 -- 172.21.15.81:0/347946047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39940b6eb0 msgr2=0x7f39940d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.494+0000 7f3998df5640 1 --2- 172.21.15.81:0/347946047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39940b6eb0 0x7f39940d02d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3994073340 tx=0x7f398402a950 comp rx=0 tx=0).stop 2024-06-29T14:23:42.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.494+0000 7f3998df5640 1 -- 172.21.15.81:0/347946047 shutdown_connections 2024-06-29T14:23:42.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.494+0000 7f3998df5640 1 --2- 172.21.15.81:0/347946047 >> v2:172.21.15.167:3300/0 conn(0x7f39940ce920 0x7f39940cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.494+0000 7f3998df5640 1 --2- 172.21.15.81:0/347946047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39940b6eb0 0x7f39940d02d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.494+0000 7f3998df5640 1 -- 172.21.15.81:0/347946047 >> 172.21.15.81:0/347946047 conn(0x7f39940b3510 msgr2=0x7f39940b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:42.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.494+0000 7f3998df5640 1 -- 172.21.15.81:0/347946047 shutdown_connections 2024-06-29T14:23:42.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.494+0000 7f3998df5640 1 -- 172.21.15.81:0/347946047 wait complete. 2024-06-29T14:23:42.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.496+0000 7f3998df5640 1 Processor -- start 2024-06-29T14:23:42.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.496+0000 7f3998df5640 1 -- start start 2024-06-29T14:23:42.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.497+0000 7f3998df5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39940b6eb0 0x7f39940af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.497+0000 7f3998df5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39940ce510 0x7f39940c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.497+0000 7f3998df5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39940ce920 0x7f399406de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.497+0000 7f3998df5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39940b4f20 con 0x7f39940b6eb0 2024-06-29T14:23:42.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.497+0000 7f3998df5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f39940b29c0 con 0x7f39940ce920 2024-06-29T14:23:42.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.497+0000 7f3998df5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f39940b50a0 con 0x7f39940ce510 2024-06-29T14:23:42.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.497+0000 7f399259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39940b6eb0 0x7f39940af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.497+0000 7f399259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39940b6eb0 0x7f39940af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42108/0 (socket says 172.21.15.81:42108) 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f399259c640 1 -- 172.21.15.81:0/3566295861 learned_addr learned my addr 172.21.15.81:0/3566295861 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f3992d9d640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39940ce920 0x7f399406de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f3991d9b640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39940ce510 0x7f39940c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f399259c640 1 -- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39940ce510 msgr2=0x7f39940c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f399259c640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39940ce510 0x7f39940c78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f399259c640 1 -- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39940ce920 msgr2=0x7f399406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f399259c640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39940ce920 0x7f399406de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f399259c640 1 -- 172.21.15.81:0/3566295861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39940b0190 con 0x7f39940b6eb0 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f3992d9d640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39940ce920 0x7f399406de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f399259c640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39940b6eb0 0x7f39940af840 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3984000f80 tx=0x7f3984030f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:42.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f3982ffd640 1 -- 172.21.15.81:0/3566295861 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984034050 con 0x7f39940b6eb0 2024-06-29T14:23:42.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f3982ffd640 1 -- 172.21.15.81:0/3566295861 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3984004340 con 0x7f39940b6eb0 2024-06-29T14:23:42.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f3998df5640 1 -- 172.21.15.81:0/3566295861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39940b0470 con 0x7f39940b6eb0 2024-06-29T14:23:42.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f3982ffd640 1 -- 172.21.15.81:0/3566295861 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984004770 con 0x7f39940b6eb0 2024-06-29T14:23:42.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.498+0000 7f3998df5640 1 -- 172.21.15.81:0/3566295861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39940b0a30 con 0x7f39940b6eb0 2024-06-29T14:23:42.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.499+0000 7f3998df5640 1 -- 172.21.15.81:0/3566295861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39940d02d0 con 0x7f39940b6eb0 2024-06-29T14:23:42.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.500+0000 7f3982ffd640 1 -- 172.21.15.81:0/3566295861 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3984037020 con 0x7f39940b6eb0 2024-06-29T14:23:42.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.500+0000 7f3982ffd640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39740762d0 0x7f39740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.501+0000 7f3982ffd640 1 -- 172.21.15.81:0/3566295861 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f39840b88f0 con 0x7f39940b6eb0 2024-06-29T14:23:42.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.501+0000 7f3991d9b640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39740762d0 0x7f39740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.501+0000 7f3991d9b640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39740762d0 0x7f39740787e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3988006fd0 tx=0x7f3988008040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:42.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.505+0000 7f3982ffd640 1 -- 172.21.15.81:0/3566295861 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3984082260 con 0x7f39940b6eb0 2024-06-29T14:23:42.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.758+0000 7f3998df5640 1 -- 172.21.15.81:0/3566295861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f39940cdf00 con 0x7f39940b6eb0 2024-06-29T14:23:42.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.761+0000 7f3982ffd640 1 -- 172.21.15.81:0/3566295861 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v27) ==== 190+0+147 (secure 0 0 0) 0x7f3984081b60 con 0x7f39940b6eb0 2024-06-29T14:23:42.760 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T14:23:42.760 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T14:23:42.760 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T14:23:42.760 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T14:23:42.760 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:23:42.760 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T14:23:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 -- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39740762d0 msgr2=0x7f39740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39740762d0 0x7f39740787e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3988006fd0 tx=0x7f3988008040 comp rx=0 tx=0).stop 2024-06-29T14:23:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 -- 172.21.15.81:0/3566295861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39940b6eb0 msgr2=0x7f39940af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39940b6eb0 0x7f39940af840 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3984000f80 tx=0x7f3984030f00 comp rx=0 tx=0).stop 2024-06-29T14:23:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 -- 172.21.15.81:0/3566295861 shutdown_connections 2024-06-29T14:23:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f39740762d0 0x7f39740787e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39940ce920 0x7f399406de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39940ce510 0x7f39940c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 --2- 172.21.15.81:0/3566295861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39940b6eb0 0x7f39940af840 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 -- 172.21.15.81:0/3566295861 >> 172.21.15.81:0/3566295861 conn(0x7f39940b3510 msgr2=0x7f3994070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 -- 172.21.15.81:0/3566295861 shutdown_connections 2024-06-29T14:23:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.764+0000 7f3998df5640 1 -- 172.21.15.81:0/3566295861 wait complete. 2024-06-29T14:23:42.774 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:23:42.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.889+0000 7f523dcb3640 1 -- 172.21.15.81:0/2178400956 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f523815d750 msgr2=0x7f52380afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.889+0000 7f523dcb3640 1 --2- 172.21.15.81:0/2178400956 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f523815d750 0x7f52380afb60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5228009830 tx=0x7f522802a510 comp rx=0 tx=0).stop 2024-06-29T14:23:42.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.889+0000 7f523dcb3640 1 -- 172.21.15.81:0/2178400956 shutdown_connections 2024-06-29T14:23:42.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.889+0000 7f523dcb3640 1 --2- 172.21.15.81:0/2178400956 >> v2:172.21.15.167:3300/0 conn(0x7f523806ca00 0x7f52380cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.889+0000 7f523dcb3640 1 --2- 172.21.15.81:0/2178400956 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f523815d750 0x7f52380afb60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.889+0000 7f523dcb3640 1 -- 172.21.15.81:0/2178400956 >> 172.21.15.81:0/2178400956 conn(0x7f52380d0930 msgr2=0x7f52380d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:42.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.889+0000 7f523dcb3640 1 -- 172.21.15.81:0/2178400956 shutdown_connections 2024-06-29T14:23:42.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.889+0000 7f523dcb3640 1 -- 172.21.15.81:0/2178400956 wait complete. 2024-06-29T14:23:42.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.890+0000 7f523dcb3640 1 Processor -- start 2024-06-29T14:23:42.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.891+0000 7f523dcb3640 1 -- start start 2024-06-29T14:23:42.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f523dcb3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f523815c030 0x7f5238155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f523dcb3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f523815d750 0x7f52381451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f523dcb3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5238153350 0x7f5238153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f523dcb3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f52380cfe60 con 0x7f523815d750 2024-06-29T14:23:42.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f523dcb3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f52380cfce0 con 0x7f5238153350 2024-06-29T14:23:42.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f523dcb3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f52380cffe0 con 0x7f523815c030 2024-06-29T14:23:42.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f5236ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f523815d750 0x7f52381451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f5236ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f523815d750 0x7f52381451e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42118/0 (socket says 172.21.15.81:42118) 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f5236ffd640 1 -- 172.21.15.81:0/1540041618 learned_addr learned my addr 172.21.15.81:0/1540041618 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f52377fe640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f523815c030 0x7f5238155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f5237fff640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5238153350 0x7f5238153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f5236ffd640 1 -- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f523815c030 msgr2=0x7f5238155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f5236ffd640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f523815c030 0x7f5238155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f5236ffd640 1 -- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5238153350 msgr2=0x7f5238153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f5236ffd640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5238153350 0x7f5238153760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.892+0000 7f5236ffd640 1 -- 172.21.15.81:0/1540041618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5238153ca0 con 0x7f523815d750 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.893+0000 7f5236ffd640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f523815d750 0x7f52381451e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f522c00c3a0 tx=0x7f522c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.893+0000 7f521ffff640 1 -- 172.21.15.81:0/1540041618 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c006890 con 0x7f523815d750 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.893+0000 7f521ffff640 1 -- 172.21.15.81:0/1540041618 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f522c006b20 con 0x7f523815d750 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.893+0000 7f523dcb3640 1 -- 172.21.15.81:0/1540041618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52381561a0 con 0x7f523815d750 2024-06-29T14:23:42.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.893+0000 7f521ffff640 1 -- 172.21.15.81:0/1540041618 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c007020 con 0x7f523815d750 2024-06-29T14:23:42.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.893+0000 7f523dcb3640 1 -- 172.21.15.81:0/1540041618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5238155ec0 con 0x7f523815d750 2024-06-29T14:23:42.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.894+0000 7f523dcb3640 1 -- 172.21.15.81:0/1540041618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f523806e2a0 con 0x7f523815d750 2024-06-29T14:23:42.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.894+0000 7f521ffff640 1 -- 172.21.15.81:0/1540041618 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f522c019020 con 0x7f523815d750 2024-06-29T14:23:42.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.895+0000 7f521ffff640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f52240762d0 0x7f52240787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:42.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.895+0000 7f521ffff640 1 -- 172.21.15.81:0/1540041618 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f522c096e70 con 0x7f523815d750 2024-06-29T14:23:42.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.895+0000 7f52377fe640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f52240762d0 0x7f52240787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:42.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.896+0000 7f52377fe640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f52240762d0 0x7f52240787e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5228034000 tx=0x7f5228004730 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:42.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:42.899+0000 7f521ffff640 1 -- 172.21.15.81:0/1540041618 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f523806e2a0 con 0x7f523815d750 2024-06-29T14:23:43.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.119+0000 7f523dcb3640 1 -- 172.21.15.81:0/1540041618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52380d0440 con 0x7f523815d750 2024-06-29T14:23:43.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.120+0000 7f521ffff640 1 -- 172.21.15.81:0/1540041618 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5220 (secure 0 0 0) 0x7f5238153ca0 con 0x7f523815d750 2024-06-29T14:23:43.119 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:23:43.119 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"btime":"2024-06-29T14:23:40:952859+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3513859413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3513859413},{"type":"v1","addr":"172.21.15.81:6835","nonce":3513859413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4775,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/763942647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":763942647},{"type":"v1","addr":"172.21.15.167:6836","nonce":763942647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1148290480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1148290480},{"type":"v1","addr":"172.21.15.167:6837","nonce":1148290480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:37.343197+0000","modified":"2024-06-29T14:23:40.325922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5172":{"gid":5172,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/58595811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":58595811},{"type":"v1","addr":"172.21.15.81:6834","nonce":58595811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5172,"qdb_cluster":[5172]},"id":2}]} 2024-06-29T14:23:43.120 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 16 2024-06-29T14:23:43.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 -- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f52240762d0 msgr2=0x7f52240787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f52240762d0 0x7f52240787e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5228034000 tx=0x7f5228004730 comp rx=0 tx=0).stop 2024-06-29T14:23:43.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 -- 172.21.15.81:0/1540041618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f523815d750 msgr2=0x7f52381451e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f523815d750 0x7f52381451e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f522c00c3a0 tx=0x7f522c00c860 comp rx=0 tx=0).stop 2024-06-29T14:23:43.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 -- 172.21.15.81:0/1540041618 shutdown_connections 2024-06-29T14:23:43.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f52240762d0 0x7f52240787e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5238153350 0x7f5238153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f523815d750 0x7f52381451e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 --2- 172.21.15.81:0/1540041618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f523815c030 0x7f5238155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 -- 172.21.15.81:0/1540041618 >> 172.21.15.81:0/1540041618 conn(0x7f52380d0930 msgr2=0x7f52380d0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:43.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 -- 172.21.15.81:0/1540041618 shutdown_connections 2024-06-29T14:23:43.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.122+0000 7f523dcb3640 1 -- 172.21.15.81:0/1540041618 wait complete. 2024-06-29T14:23:43.536 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:23:37.343197+0000', 'modified': '2024-06-29T14:23:40.325922+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5172}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5172': {'gid': 5172, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6834/58595811', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6832', 'nonce': 58595811}, {'type': 'v1', 'addr': '172.21.15.81:6834', 'nonce': 58595811}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5172, 'qdb_cluster': [5172]} 2024-06-29T14:23:43.536 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:23:43.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T14:23:43.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.653+0000 7f046f54a640 1 -- 172.21.15.81:0/280542645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04680cf4d0 msgr2=0x7f0468073030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.653+0000 7f046f54a640 1 --2- 172.21.15.81:0/280542645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04680cf4d0 0x7f0468073030 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0458005fd0 tx=0x7f045802a530 comp rx=0 tx=0).stop 2024-06-29T14:23:43.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.653+0000 7f046f54a640 1 -- 172.21.15.81:0/280542645 shutdown_connections 2024-06-29T14:23:43.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.653+0000 7f046f54a640 1 --2- 172.21.15.81:0/280542645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04680cf4d0 0x7f0468073030 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.653+0000 7f046f54a640 1 --2- 172.21.15.81:0/280542645 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f04681412f0 0x7f04680cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.653+0000 7f046f54a640 1 -- 172.21.15.81:0/280542645 >> 172.21.15.81:0/280542645 conn(0x7f046806ccc0 msgr2=0x7f04680b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:43.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.654+0000 7f046f54a640 1 -- 172.21.15.81:0/280542645 shutdown_connections 2024-06-29T14:23:43.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.654+0000 7f046f54a640 1 -- 172.21.15.81:0/280542645 wait complete. 2024-06-29T14:23:43.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.655+0000 7f046f54a640 1 Processor -- start 2024-06-29T14:23:43.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.655+0000 7f046f54a640 1 -- start start 2024-06-29T14:23:43.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046f54a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0468073570 0x7f0468157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046f54a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04680cf4d0 0x7f04680aa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046f54a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04681412f0 0x7f0468153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046f54a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f04680b3210 con 0x7f04681412f0 2024-06-29T14:23:43.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046f54a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f046815f280 con 0x7f04680cf4d0 2024-06-29T14:23:43.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046f54a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f04680b3390 con 0x7f0468073570 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046dae7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04681412f0 0x7f0468153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046dae7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04681412f0 0x7f0468153460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42148/0 (socket says 172.21.15.81:42148) 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046dae7640 1 -- 172.21.15.81:0/4051539761 learned_addr learned my addr 172.21.15.81:0/4051539761 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046d2e6640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0468073570 0x7f0468157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046cae5640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04680cf4d0 0x7f04680aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046dae7640 1 -- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0468073570 msgr2=0x7f0468157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046dae7640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0468073570 0x7f0468157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046dae7640 1 -- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04680cf4d0 msgr2=0x7f04680aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046dae7640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04680cf4d0 0x7f04680aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046dae7640 1 -- 172.21.15.81:0/4051539761 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046813efd0 con 0x7f04681412f0 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046dae7640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04681412f0 0x7f0468153460 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f046000b970 tx=0x7f046000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:43.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f045dffb640 1 -- 172.21.15.81:0/4051539761 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460019070 con 0x7f04681412f0 2024-06-29T14:23:43.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f045dffb640 1 -- 172.21.15.81:0/4051539761 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f046000cc30 con 0x7f04681412f0 2024-06-29T14:23:43.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f046f54a640 1 -- 172.21.15.81:0/4051539761 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046813f2b0 con 0x7f04681412f0 2024-06-29T14:23:43.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.657+0000 7f045dffb640 1 -- 172.21.15.81:0/4051539761 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04600146a0 con 0x7f04681412f0 2024-06-29T14:23:43.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.658+0000 7f046f54a640 1 -- 172.21.15.81:0/4051539761 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f046813f870 con 0x7f04681412f0 2024-06-29T14:23:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.660+0000 7f045dffb640 1 -- 172.21.15.81:0/4051539761 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f046001c070 con 0x7f04681412f0 2024-06-29T14:23:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.660+0000 7f045dffb640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f044c0762d0 0x7f044c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.661+0000 7f045dffb640 1 -- 172.21.15.81:0/4051539761 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f04600973c0 con 0x7f04681412f0 2024-06-29T14:23:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.661+0000 7f046d2e6640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f044c0762d0 0x7f044c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.661+0000 7f0476407740 1 Processor -- start 2024-06-29T14:23:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.661+0000 7f0476407740 1 -- start start 2024-06-29T14:23:43.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.661+0000 7f046d2e6640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f044c0762d0 0x7f044c0787e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f046400bb50 tx=0x7f046400b2e0 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.662+0000 7f0476407740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af1960270 0x555af1d4c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.662+0000 7f0476407740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af19373b0 0x555af1d303a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.662+0000 7f0476407740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1d308e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.662+0000 7f0476407740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555af1cc8bc0 con 0x555af193f590 2024-06-29T14:23:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.662+0000 7f0476407740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x555af1c77630 con 0x555af19373b0 2024-06-29T14:23:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.662+0000 7f0476407740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x555af1c66bd0 con 0x555af1960270 2024-06-29T14:23:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.662+0000 7f046dae7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1d308e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046dae7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1d308e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42152/0 (socket says 172.21.15.81:42152) 2024-06-29T14:23:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046dae7640 1 -- 172.21.15.81:0/1772891943 learned_addr learned my addr 172.21.15.81:0/1772891943 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046d2e6640 1 --2- 172.21.15.81:0/1772891943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af1960270 0x555af1d4c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046cae5640 1 --2- 172.21.15.81:0/1772891943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af19373b0 0x555af1d303a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046dae7640 1 -- 172.21.15.81:0/1772891943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af1960270 msgr2=0x555af1d4c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046dae7640 1 --2- 172.21.15.81:0/1772891943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af1960270 0x555af1d4c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046dae7640 1 -- 172.21.15.81:0/1772891943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af19373b0 msgr2=0x555af1d303a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046dae7640 1 --2- 172.21.15.81:0/1772891943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af19373b0 0x555af1d303a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046dae7640 1 -- 172.21.15.81:0/1772891943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555af1c63ca0 con 0x555af193f590 2024-06-29T14:23:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046cae5640 1 --2- 172.21.15.81:0/1772891943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af19373b0 0x555af1d303a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f046dae7640 1 --2- 172.21.15.81:0/1772891943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1d308e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0460095890 tx=0x7f0460060750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5276ae7c11993f07 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f0452ffd640 1 -- 172.21.15.81:0/1772891943 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04600970e0 con 0x555af193f590 2024-06-29T14:23:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f0452ffd640 1 -- 172.21.15.81:0/1772891943 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f046005c9e0 con 0x555af193f590 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.663+0000 7f0452ffd640 1 -- 172.21.15.81:0/1772891943 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04600691c0 con 0x555af193f590 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.664+0000 7f0476407740 1 -- 172.21.15.81:0/1772891943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 msgr2=0x555af1d308e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.664+0000 7f0476407740 1 --2- 172.21.15.81:0/1772891943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1d308e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0460095890 tx=0x7f0460060750 comp rx=0 tx=0).stop 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.664+0000 7f0476407740 1 -- 172.21.15.81:0/1772891943 shutdown_connections 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.664+0000 7f0476407740 1 --2- 172.21.15.81:0/1772891943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1d308e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.664+0000 7f0476407740 1 --2- 172.21.15.81:0/1772891943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af19373b0 0x555af1d303a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.664+0000 7f0476407740 1 --2- 172.21.15.81:0/1772891943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af1960270 0x555af1d4c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.664+0000 7f0476407740 1 -- 172.21.15.81:0/1772891943 >> 172.21.15.81:0/1772891943 conn(0x555af19543c0 msgr2=0x555af1d03f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.664+0000 7f0476407740 1 -- 172.21.15.81:0/1772891943 shutdown_connections 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.664+0000 7f0476407740 1 -- 172.21.15.81:0/1772891943 wait complete. 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.665+0000 7f0476407740 1 -- start start 2024-06-29T14:23:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.665+0000 7f0476407740 1 Processor -- start 2024-06-29T14:23:43.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f0476407740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af19373b0 0x555af1d4be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f0476407740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af193f590 0x555af1d2a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f0476407740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1d2a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f0476407740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555af1c04ae0 con 0x555af19373b0 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f0476407740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x555af1c6ac10 con 0x555af1960270 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f0476407740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x555af19da620 con 0x555af193f590 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f0476407740 10 client.0 _ll_register_callbacks cb 0x555af1c11480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f046d2e6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af19373b0 0x555af1d4be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f046d2e6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af19373b0 0x555af1d4be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42160/0 (socket says 172.21.15.81:42160) 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f046d2e6640 1 -- 172.21.15.81:0/588113214 learned_addr learned my addr 172.21.15.81:0/588113214 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f046dae7640 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1d2a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.666+0000 7f046cae5640 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af193f590 0x555af1d2a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f046d2e6640 1 -- 172.21.15.81:0/588113214 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af193f590 msgr2=0x555af1d2a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f046d2e6640 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af193f590 0x555af1d2a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f046d2e6640 1 -- 172.21.15.81:0/588113214 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 msgr2=0x555af1d2a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f046d2e6640 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1d2a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f046d2e6640 1 -- 172.21.15.81:0/588113214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555af19cb850 con 0x555af19373b0 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f046d2e6640 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af19373b0 0x555af1d4be70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f04640116c0 tx=0x7f0464004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f5110cac91afd4 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0447fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0447fff640 1 -- 172.21.15.81:0/588113214 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046401fa00 con 0x555af19373b0 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0447fff640 1 -- 172.21.15.81:0/588113214 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f046401fba0 con 0x555af19373b0 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0476407740 1 -- 172.21.15.81:0/588113214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f046005c9e0 con 0x555af19373b0 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0447fff640 1 -- 172.21.15.81:0/588113214 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f046401fde0 con 0x555af19373b0 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0476407740 10 client.5355 fetch_fsmap learned FSMap version 16 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0476407740 1 -- 172.21.15.81:0/588113214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x555af1a426b0 con 0x555af19373b0 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0447fff640 1 -- 172.21.15.81:0/588113214 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f046005c9e0 con 0x555af19373b0 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0447fff640 1 -- 172.21.15.81:0/588113214 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 16) ==== 3502+0+0 (secure 0 0 0) 0x7f0464030050 con 0x555af19373b0 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0476407740 10 client.5355 fetch_fsmap finished waiting for FSMap version 16 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0476407740 10 client.5355 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:23:43.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.667+0000 7f0476407740 20 client.5355 populate_metadata read hostname 'smithi081' 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.668+0000 7f0476407740 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.81:6832/58595811,v1:172.21.15.81:6834/58595811] conn(0x555af191ee40 0x555af1d68870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.668+0000 7f0476407740 4 client.5355 mds_command: new command op to 5172 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.668+0000 7f0476407740 1 -- 172.21.15.81:0/588113214 --> [v2:172.21.15.81:6832/58595811,v1:172.21.15.81:6834/58595811] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555af1c79580 con 0x555af191ee40 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.668+0000 7f046cae5640 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.81:6832/58595811,v1:172.21.15.81:6834/58595811] conn(0x555af191ee40 0x555af1d68870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.668+0000 7f0447fff640 10 client.5355 ms_handle_connect on v2:172.21.15.81:6832/58595811 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.668+0000 7f046cae5640 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.81:6832/58595811,v1:172.21.15.81:6834/58595811] conn(0x555af191ee40 0x555af1d68870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f70cb84c6c003b29 server_cookie=aa4b888a60b71368 in_seq=0 out_seq=0 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0447fff640 1 -- 172.21.15.81:0/588113214 <== mds.0 v2:172.21.15.81:6832/58595811 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x555af1c79580 con 0x555af191ee40 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0447fff640 10 client.5355 handle_command_reply: tid=0 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 client.5355 shutdown 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 -- 172.21.15.81:0/588113214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af19373b0 msgr2=0x555af1d4be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af19373b0 0x555af1d4be70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f04640116c0 tx=0x7f0464004730 comp rx=0 tx=0).stop 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 -- 172.21.15.81:0/588113214 shutdown_connections 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.81:6832/58595811,v1:172.21.15.81:6834/58595811] conn(0x555af191ee40 0x555af1d68870 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1d2a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0447fff640 0 client.5355 ms_handle_reset on v2:172.21.15.81:6832/58595811 2024-06-29T14:23:43.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af193f590 0x555af1d2a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 --2- 172.21.15.81:0/588113214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af19373b0 0x555af1d4be70 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 -- 172.21.15.81:0/588113214 >> 172.21.15.81:0/588113214 conn(0x555af19543c0 msgr2=0x555af1d03e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:43.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 -- 172.21.15.81:0/588113214 shutdown_connections 2024-06-29T14:23:43.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 1 -- 172.21.15.81:0/588113214 wait complete. 2024-06-29T14:23:43.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.669+0000 7f0476407740 20 client.5355 trim_cache size 0 max 16384 2024-06-29T14:23:43.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.688+0000 7f0476407740 1 Processor -- start 2024-06-29T14:23:43.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.688+0000 7f0476407740 1 -- start start 2024-06-29T14:23:43.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.689+0000 7f0476407740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af19373b0 0x555af1d07840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.689+0000 7f0476407740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1cffb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.689+0000 7f0476407740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1d00080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.689+0000 7f0476407740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0468072ce0 con 0x555af193f590 2024-06-29T14:23:43.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.689+0000 7f0476407740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0468072040 con 0x555af1960270 2024-06-29T14:23:43.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.689+0000 7f0476407740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0468072690 con 0x555af19373b0 2024-06-29T14:23:43.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f046cae5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1cffb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f046cae5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1cffb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42164/0 (socket says 172.21.15.81:42164) 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f046cae5640 1 -- 172.21.15.81:0/2089828042 learned_addr learned my addr 172.21.15.81:0/2089828042 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f046cae5640 1 -- 172.21.15.81:0/2089828042 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af19373b0 msgr2=0x555af1d07840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f046cae5640 1 --2- 172.21.15.81:0/2089828042 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af19373b0 0x555af1d07840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f046cae5640 1 -- 172.21.15.81:0/2089828042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 msgr2=0x555af1d00080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f046cae5640 1 --2- 172.21.15.81:0/2089828042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1d00080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f046cae5640 1 -- 172.21.15.81:0/2089828042 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555af1d4cf90 con 0x555af193f590 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f046cae5640 1 --2- 172.21.15.81:0/2089828042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1cffb40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f04580099f0 tx=0x7f0458004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e9d739c8bd4b1c server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f04527fc640 1 -- 172.21.15.81:0/2089828042 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045804a070 con 0x555af193f590 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f04527fc640 1 -- 172.21.15.81:0/2089828042 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04580043f0 con 0x555af193f590 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.690+0000 7f04527fc640 1 -- 172.21.15.81:0/2089828042 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458035040 con 0x555af193f590 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.691+0000 7f0476407740 1 -- 172.21.15.81:0/2089828042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 msgr2=0x555af1cffb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.691+0000 7f0476407740 1 --2- 172.21.15.81:0/2089828042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1cffb40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f04580099f0 tx=0x7f0458004730 comp rx=0 tx=0).stop 2024-06-29T14:23:43.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.691+0000 7f0476407740 1 -- 172.21.15.81:0/2089828042 shutdown_connections 2024-06-29T14:23:43.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.691+0000 7f0476407740 1 --2- 172.21.15.81:0/2089828042 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1d00080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.691+0000 7f0476407740 1 --2- 172.21.15.81:0/2089828042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1cffb40 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.691+0000 7f0476407740 1 --2- 172.21.15.81:0/2089828042 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af19373b0 0x555af1d07840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.691+0000 7f0476407740 1 -- 172.21.15.81:0/2089828042 >> 172.21.15.81:0/2089828042 conn(0x555af19543c0 msgr2=0x555af1d2b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:43.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.691+0000 7f0476407740 1 -- 172.21.15.81:0/2089828042 shutdown_connections 2024-06-29T14:23:43.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.691+0000 7f0476407740 1 -- 172.21.15.81:0/2089828042 wait complete. 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.692+0000 7f0476407740 1 -- start start 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.692+0000 7f0476407740 1 Processor -- start 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f0476407740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af19373b0 0x555af1d5d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f0476407740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1db2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f0476407740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1daf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f0476407740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555af1d285b0 con 0x555af193f590 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f0476407740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x555af1d441b0 con 0x555af1960270 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f0476407740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x555af1d2ae80 con 0x555af19373b0 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f0476407740 10 client.0 _ll_register_callbacks cb 0x7f0464010bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f046cae5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1db2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f046cae5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1db2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42180/0 (socket says 172.21.15.81:42180) 2024-06-29T14:23:43.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f046cae5640 1 -- 172.21.15.81:0/3738359862 learned_addr learned my addr 172.21.15.81:0/3738359862 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f046d2e6640 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af19373b0 0x555af1d5d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.693+0000 7f046dae7640 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1daf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f046cae5640 1 -- 172.21.15.81:0/3738359862 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af19373b0 msgr2=0x555af1d5d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f046cae5640 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af19373b0 0x555af1d5d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f046cae5640 1 -- 172.21.15.81:0/3738359862 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 msgr2=0x555af1daf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f046cae5640 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1daf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f046cae5640 1 -- 172.21.15.81:0/3738359862 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555af1db3230 con 0x555af193f590 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f046dae7640 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1daf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f046cae5640 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1db2cf0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0458048040 tx=0x7f0458005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae76c304b4226cf2 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0447fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0447fff640 1 -- 172.21.15.81:0/3738359862 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045804a040 con 0x555af193f590 2024-06-29T14:23:43.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0447fff640 1 -- 172.21.15.81:0/3738359862 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f04580037f0 con 0x555af193f590 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0476407740 1 -- 172.21.15.81:0/3738359862 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04580043f0 con 0x555af193f590 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0447fff640 1 -- 172.21.15.81:0/3738359862 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f0458035040 con 0x555af193f590 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0476407740 10 client.5361 fetch_fsmap learned FSMap version 16 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0476407740 1 -- 172.21.15.81:0/3738359862 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x555af1d5cb00 con 0x555af193f590 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0447fff640 1 -- 172.21.15.81:0/3738359862 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04580043f0 con 0x555af193f590 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0447fff640 1 -- 172.21.15.81:0/3738359862 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 16) ==== 3502+0+0 (secure 0 0 0) 0x7f0458058030 con 0x555af193f590 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0476407740 10 client.5361 fetch_fsmap finished waiting for FSMap version 16 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0476407740 10 client.5361 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0476407740 20 client.5361 populate_metadata read hostname 'smithi081' 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0476407740 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.81:6832/58595811,v1:172.21.15.81:6834/58595811] conn(0x555af1d2fe90 0x555af1d04d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0476407740 4 client.5361 mds_command: new command op to 5172 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.694+0000 7f0476407740 1 -- 172.21.15.81:0/3738359862 --> [v2:172.21.15.81:6832/58595811,v1:172.21.15.81:6834/58595811] -- command(tid 0: {"prefix": "status"}) -- 0x555af1c75020 con 0x555af1d2fe90 2024-06-29T14:23:43.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f046d2e6640 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.81:6832/58595811,v1:172.21.15.81:6834/58595811] conn(0x555af1d2fe90 0x555af1d04d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0447fff640 10 client.5361 ms_handle_connect on v2:172.21.15.81:6832/58595811 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f046d2e6640 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.81:6832/58595811,v1:172.21.15.81:6834/58595811] conn(0x555af1d2fe90 0x555af1d04d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96b4c708121d0671 server_cookie=685103269c094fe0 in_seq=0 out_seq=0 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0447fff640 1 -- 172.21.15.81:0/3738359862 <== mds.0 v2:172.21.15.81:6832/58595811 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x555af1c75020 con 0x555af1d2fe90 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0447fff640 10 client.5361 handle_command_reply: tid=0 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0476407740 1 client.5361 shutdown 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0476407740 1 -- 172.21.15.81:0/3738359862 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 msgr2=0x555af1db2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0476407740 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1db2cf0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0458048040 tx=0x7f0458005c90 comp rx=0 tx=0).stop 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0476407740 1 -- 172.21.15.81:0/3738359862 shutdown_connections 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0476407740 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.81:6832/58595811,v1:172.21.15.81:6834/58595811] conn(0x555af1d2fe90 0x555af1d04d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0476407740 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555af1960270 0x555af1daf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0476407740 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555af193f590 0x555af1db2cf0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0447fff640 0 client.5361 ms_handle_reset on v2:172.21.15.81:6832/58595811 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0476407740 1 --2- 172.21.15.81:0/3738359862 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555af19373b0 0x555af1d5d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.695+0000 7f0476407740 1 -- 172.21.15.81:0/3738359862 >> 172.21.15.81:0/3738359862 conn(0x555af19543c0 msgr2=0x555af1d5dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:43.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f0476407740 1 -- 172.21.15.81:0/3738359862 shutdown_connections 2024-06-29T14:23:43.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f0476407740 1 -- 172.21.15.81:0/3738359862 wait complete. 2024-06-29T14:23:43.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f0476407740 20 client.5361 trim_cache size 0 max 16384 2024-06-29T14:23:43.701 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:23:43.701 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T14:23:43.701 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T14:23:43.701 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 5172, 2024-06-29T14:23:43.701 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T14:23:43.701 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T14:23:43.701 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T14:23:43.702 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7746188690000002, 2024-06-29T14:23:43.702 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 16, 2024-06-29T14:23:43.702 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 33, 2024-06-29T14:23:43.702 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 33, 2024-06-29T14:23:43.702 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 13.669203401000001 2024-06-29T14:23:43.702 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:23:43.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 -- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f044c0762d0 msgr2=0x7f044c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f044c0762d0 0x7f044c0787e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f046400bb50 tx=0x7f046400b2e0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 -- 172.21.15.81:0/4051539761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04681412f0 msgr2=0x7f0468153460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04681412f0 0x7f0468153460 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f046000b970 tx=0x7f046000be30 comp rx=0 tx=0).stop 2024-06-29T14:23:43.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 -- 172.21.15.81:0/4051539761 shutdown_connections 2024-06-29T14:23:43.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f044c0762d0 0x7f044c0787e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04681412f0 0x7f0468153460 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04680cf4d0 0x7f04680aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 --2- 172.21.15.81:0/4051539761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0468073570 0x7f0468157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 -- 172.21.15.81:0/4051539761 >> 172.21.15.81:0/4051539761 conn(0x7f046806ccc0 msgr2=0x7f046806f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:43.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 -- 172.21.15.81:0/4051539761 shutdown_connections 2024-06-29T14:23:43.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.696+0000 7f046f54a640 1 -- 172.21.15.81:0/4051539761 wait complete. 2024-06-29T14:23:43.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:23:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.820+0000 7f44303bd640 1 -- 172.21.15.81:0/1423079615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4410004b60 msgr2=0x7f4410004fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.820+0000 7f44303bd640 1 --2- 172.21.15.81:0/1423079615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4410004b60 0x7f4410004fd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f441800c610 tx=0x7f4410006940 comp rx=0 tx=0).stop 2024-06-29T14:23:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.820+0000 7f44303bd640 1 -- 172.21.15.81:0/1423079615 shutdown_connections 2024-06-29T14:23:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.820+0000 7f44303bd640 1 --2- 172.21.15.81:0/1423079615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4410004b60 0x7f4410004fd0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.820+0000 7f44303bd640 1 -- 172.21.15.81:0/1423079615 >> 172.21.15.81:0/1423079615 conn(0x7f442806ccc0 msgr2=0x7f44280b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.820+0000 7f44303bd640 1 -- 172.21.15.81:0/1423079615 shutdown_connections 2024-06-29T14:23:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.820+0000 7f44303bd640 1 -- 172.21.15.81:0/1423079615 wait complete. 2024-06-29T14:23:43.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.822+0000 7f44303bd640 1 Processor -- start 2024-06-29T14:23:43.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.822+0000 7f44303bd640 1 -- start start 2024-06-29T14:23:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.823+0000 7f44303bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4410004b60 0x7f44281f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.823+0000 7f44303bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44280ada50 0x7f44280ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f44303bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44281f84a0 0x7f44281f88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f44303bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f442806fe00 con 0x7f44280ada50 2024-06-29T14:23:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f44303bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f442806fc80 con 0x7f4410004b60 2024-06-29T14:23:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f44303bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f442806ff80 con 0x7f44281f84a0 2024-06-29T14:23:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442d958640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44280ada50 0x7f44280ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442d958640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44280ada50 0x7f44280ade60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42198/0 (socket says 172.21.15.81:42198) 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442d958640 1 -- 172.21.15.81:0/3199854139 learned_addr learned my addr 172.21.15.81:0/3199854139 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442e159640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4410004b60 0x7f44281f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442e95a640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44281f84a0 0x7f44281f88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442d958640 1 -- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44281f84a0 msgr2=0x7f44281f88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442d958640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44281f84a0 0x7f44281f88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442d958640 1 -- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4410004b60 msgr2=0x7f44281f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442d958640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4410004b60 0x7f44281f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442d958640 1 -- 172.21.15.81:0/3199854139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44281f8df0 con 0x7f44280ada50 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442e159640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4410004b60 0x7f44281f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f442d958640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44280ada50 0x7f44280ade60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f441c0117b0 tx=0x7f441c011c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f4426ffd640 1 -- 172.21.15.81:0/3199854139 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c004460 con 0x7f44280ada50 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f4426ffd640 1 -- 172.21.15.81:0/3199854139 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f441c016070 con 0x7f44280ada50 2024-06-29T14:23:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f44303bd640 1 -- 172.21.15.81:0/3199854139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44280af7f0 con 0x7f44280ada50 2024-06-29T14:23:43.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f4426ffd640 1 -- 172.21.15.81:0/3199854139 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c00fba0 con 0x7f44280ada50 2024-06-29T14:23:43.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.824+0000 7f44303bd640 1 -- 172.21.15.81:0/3199854139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44280afdb0 con 0x7f44280ada50 2024-06-29T14:23:43.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.825+0000 7f44303bd640 1 -- 172.21.15.81:0/3199854139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44280ae4f0 con 0x7f44280ada50 2024-06-29T14:23:43.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.826+0000 7f4426ffd640 1 -- 172.21.15.81:0/3199854139 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f441c018030 con 0x7f44280ada50 2024-06-29T14:23:43.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.826+0000 7f4426ffd640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4414076300 0x7f4414078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:43.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.827+0000 7f4426ffd640 1 -- 172.21.15.81:0/3199854139 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f441c097fd0 con 0x7f44280ada50 2024-06-29T14:23:43.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.827+0000 7f442e159640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4414076300 0x7f4414078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:43.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.828+0000 7f442e159640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4414076300 0x7f4414078810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4418033000 tx=0x7f4418001360 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:43.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:43.831+0000 7f4426ffd640 1 -- 172.21.15.81:0/3199854139 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f44281f8df0 con 0x7f44280ada50 2024-06-29T14:23:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.054+0000 7f44303bd640 1 -- 172.21.15.81:0/3199854139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f44280cf2f0 con 0x7f44280ada50 2024-06-29T14:23:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.054+0000 7f4426ffd640 1 -- 172.21.15.81:0/3199854139 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f441c0618a0 con 0x7f44280ada50 2024-06-29T14:23:44.053 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:23:44.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 -- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4414076300 msgr2=0x7f4414078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:44.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4414076300 0x7f4414078810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4418033000 tx=0x7f4418001360 comp rx=0 tx=0).stop 2024-06-29T14:23:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 -- 172.21.15.81:0/3199854139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44280ada50 msgr2=0x7f44280ade60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44280ada50 0x7f44280ade60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f441c0117b0 tx=0x7f441c011c70 comp rx=0 tx=0).stop 2024-06-29T14:23:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 -- 172.21.15.81:0/3199854139 shutdown_connections 2024-06-29T14:23:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f4414076300 0x7f4414078810 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44281f84a0 0x7f44281f88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44280ada50 0x7f44280ade60 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 --2- 172.21.15.81:0/3199854139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4410004b60 0x7f44281f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 -- 172.21.15.81:0/3199854139 >> 172.21.15.81:0/3199854139 conn(0x7f442806ccc0 msgr2=0x7f4428071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 -- 172.21.15.81:0/3199854139 shutdown_connections 2024-06-29T14:23:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.056+0000 7f44303bd640 1 -- 172.21.15.81:0/3199854139 wait complete. 2024-06-29T14:23:44.066 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:23:44.066 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:23:44.066 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T14:23:44.066 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:23:44.067 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:44.067 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:23:44.067 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:23:44.067 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:23:44.067 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:23:44.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:44.067 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T14:23:44.083 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:23:44.083 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:23:44.083 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:23:44.083 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:44.084 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 42695sec preferred_lft 42695sec 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T14:23:44.085 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:44.086 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:23:44.086 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:23:44.086 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T14:23:44.086 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:23:44.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:44.086 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:23:44.147 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T14:23:44.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:44.149 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T14:23:44.205 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T14:23:44.205 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:23:44.205 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:23:44.205 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:23:44.205 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:23:44.205 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:23:44.206 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:23:44.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:23:44.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:23:44.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:23:44.206 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:23:44.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:23:44.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:44.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:44.331 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:23:44.346 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:23:44.346 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:23:44.347 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:23:44.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:44.347 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T14:23:44.412 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:44.412 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:44.467 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:23:44.468 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:23:44.468 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:44.523 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T14:23:44.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:44.523 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:44.617 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:44.617 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:23:44.625 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:23:44.625 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:23:44.625 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:23:44.625 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:23:44.625 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:23:44.626 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:23:44.626 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:23:44.626 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:23:44.626 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:23:44.626 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:23:44.627 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:23:44.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:44.628 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:23:44.652 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:23:44.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:23:44.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:23:44.754 INFO:teuthology.orchestra.run.smithi081.stdout:client5107 2024-06-29T14:23:44.754 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:23:44.754 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:23:44.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:23:44.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.926+0000 7f73111bd640 1 -- 172.21.15.81:0/2883545692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c0cf600 msgr2=0x7f730c0cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:44.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.926+0000 7f73111bd640 1 --2- 172.21.15.81:0/2883545692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c0cf600 0x7f730c0cfa10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7304009830 tx=0x7f730402a510 comp rx=0 tx=0).stop 2024-06-29T14:23:44.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.926+0000 7f73111bd640 1 -- 172.21.15.81:0/2883545692 shutdown_connections 2024-06-29T14:23:44.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.926+0000 7f73111bd640 1 --2- 172.21.15.81:0/2883545692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c0cf600 0x7f730c0cfa10 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:44.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.926+0000 7f73111bd640 1 --2- 172.21.15.81:0/2883545692 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c0ce3a0 0x7f730c0ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:44.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.926+0000 7f73111bd640 1 -- 172.21.15.81:0/2883545692 >> 172.21.15.81:0/2883545692 conn(0x7f730c0b35a0 msgr2=0x7f730c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:44.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.926+0000 7f73111bd640 1 -- 172.21.15.81:0/2883545692 shutdown_connections 2024-06-29T14:23:44.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.926+0000 7f73111bd640 1 -- 172.21.15.81:0/2883545692 wait complete. 2024-06-29T14:23:44.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.928+0000 7f73111bd640 1 Processor -- start 2024-06-29T14:23:44.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.928+0000 7f73111bd640 1 -- start start 2024-06-29T14:23:44.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.929+0000 7f73111bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c0ce3a0 0x7f730c0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:44.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.929+0000 7f73111bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c0cf600 0x7f730c0b5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:44.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f73111bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c09f3c0 0x7f730c1d12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:44.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f73111bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f730c0b2830 con 0x7f730c09f3c0 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f73111bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f730c0b26b0 con 0x7f730c0ce3a0 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f73111bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f730c0b6df0 con 0x7f730c0cf600 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730b59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c09f3c0 0x7f730c1d12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730b59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c09f3c0 0x7f730c1d12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42206/0 (socket says 172.21.15.81:42206) 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730b59e640 1 -- 172.21.15.81:0/2339222885 learned_addr learned my addr 172.21.15.81:0/2339222885 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730a59c640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c0cf600 0x7f730c0b5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730ad9d640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c0ce3a0 0x7f730c0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730b59e640 1 -- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c0cf600 msgr2=0x7f730c0b5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730b59e640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c0cf600 0x7f730c0b5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730b59e640 1 -- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c0ce3a0 msgr2=0x7f730c0b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730b59e640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c0ce3a0 0x7f730c0b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730b59e640 1 -- 172.21.15.81:0/2339222885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730c09c5f0 con 0x7f730c09f3c0 2024-06-29T14:23:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730ad9d640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c0ce3a0 0x7f730c0b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f730b59e640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c09f3c0 0x7f730c1d12a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f730402ab00 tx=0x7f7304033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f72f37fe640 1 -- 172.21.15.81:0/2339222885 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7304031d60 con 0x7f730c09f3c0 2024-06-29T14:23:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f72f37fe640 1 -- 172.21.15.81:0/2339222885 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7304033e10 con 0x7f730c09f3c0 2024-06-29T14:23:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f73111bd640 1 -- 172.21.15.81:0/2339222885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f730c09c8d0 con 0x7f730c09f3c0 2024-06-29T14:23:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f72f37fe640 1 -- 172.21.15.81:0/2339222885 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7304005220 con 0x7f730c09f3c0 2024-06-29T14:23:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.930+0000 7f73111bd640 1 -- 172.21.15.81:0/2339222885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f730c1d21d0 con 0x7f730c09f3c0 2024-06-29T14:23:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.931+0000 7f73111bd640 1 -- 172.21.15.81:0/2339222885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f730c0d1480 con 0x7f730c09f3c0 2024-06-29T14:23:44.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.932+0000 7f72f37fe640 1 -- 172.21.15.81:0/2339222885 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f73040053c0 con 0x7f730c09f3c0 2024-06-29T14:23:44.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.933+0000 7f72f37fe640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f72f80762d0 0x7f72f80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:44.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.933+0000 7f72f37fe640 1 -- 172.21.15.81:0/2339222885 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f73040b8830 con 0x7f730c09f3c0 2024-06-29T14:23:44.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.933+0000 7f730ad9d640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f72f80762d0 0x7f72f80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:44.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.934+0000 7f730ad9d640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f72f80762d0 0x7f72f80787e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0012e0 tx=0x7f72fc009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:44.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:44.937+0000 7f72f37fe640 1 -- 172.21.15.81:0/2339222885 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f730c0d1480 con 0x7f730c09f3c0 2024-06-29T14:23:45.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.145+0000 7f73111bd640 1 -- 172.21.15.81:0/2339222885 --> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f730c0b7360 con 0x7f72f80762d0 2024-06-29T14:23:45.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.148+0000 7f72f37fe640 1 -- 172.21.15.81:0/2339222885 <== mgr.4126 v2:172.21.15.167:6832/3101542440 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f730c0b7360 con 0x7f72f80762d0 2024-06-29T14:23:45.147 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:23:45.147 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:23:45.147 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T14:23:45.147 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:23:45.147 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:23:45.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 -- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f72f80762d0 msgr2=0x7f72f80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:45.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f72f80762d0 0x7f72f80787e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0012e0 tx=0x7f72fc009040 comp rx=0 tx=0).stop 2024-06-29T14:23:45.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 -- 172.21.15.81:0/2339222885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c09f3c0 msgr2=0x7f730c1d12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:45.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c09f3c0 0x7f730c1d12a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f730402ab00 tx=0x7f7304033c20 comp rx=0 tx=0).stop 2024-06-29T14:23:45.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 -- 172.21.15.81:0/2339222885 shutdown_connections 2024-06-29T14:23:45.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f72f80762d0 0x7f72f80787e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c09f3c0 0x7f730c1d12a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c0cf600 0x7f730c0b5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 --2- 172.21.15.81:0/2339222885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c0ce3a0 0x7f730c0b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 -- 172.21.15.81:0/2339222885 >> 172.21.15.81:0/2339222885 conn(0x7f730c0b35a0 msgr2=0x7f730c071660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:45.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 -- 172.21.15.81:0/2339222885 shutdown_connections 2024-06-29T14:23:45.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.150+0000 7f73111bd640 1 -- 172.21.15.81:0/2339222885 wait complete. 2024-06-29T14:23:45.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:23:45.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.275+0000 7fb482eb3640 1 -- 172.21.15.81:0/310687749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb47c0ce920 msgr2=0x7fb47c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:45.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.275+0000 7fb482eb3640 1 --2- 172.21.15.81:0/310687749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb47c0ce920 0x7fb47c0cd480 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fb474005fd0 tx=0x7fb47402aa10 comp rx=0 tx=0).stop 2024-06-29T14:23:45.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.275+0000 7fb482eb3640 1 -- 172.21.15.81:0/310687749 shutdown_connections 2024-06-29T14:23:45.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.275+0000 7fb482eb3640 1 --2- 172.21.15.81:0/310687749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb47c0ce920 0x7fb47c0cd480 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.275+0000 7fb482eb3640 1 --2- 172.21.15.81:0/310687749 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb47c0b6eb0 0x7fb47c0d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.275+0000 7fb482eb3640 1 -- 172.21.15.81:0/310687749 >> 172.21.15.81:0/310687749 conn(0x7fb47c0b3510 msgr2=0x7fb47c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:45.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.275+0000 7fb482eb3640 1 -- 172.21.15.81:0/310687749 shutdown_connections 2024-06-29T14:23:45.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.275+0000 7fb482eb3640 1 -- 172.21.15.81:0/310687749 wait complete. 2024-06-29T14:23:45.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.277+0000 7fb482eb3640 1 Processor -- start 2024-06-29T14:23:45.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.277+0000 7fb482eb3640 1 -- start start 2024-06-29T14:23:45.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb482eb3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb47c0b6eb0 0x7fb47c06ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:45.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb482eb3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb47c0ce510 0x7fb47c15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb482eb3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb47c0ce920 0x7fb47c164410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb482eb3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c0b50a0 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb482eb3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb47c0b4f20 con 0x7fb47c0ce510 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb482eb3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb47c0b5220 con 0x7fb47c0ce920 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb480c4f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb47c0b6eb0 0x7fb47c06ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb480c4f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb47c0b6eb0 0x7fb47c06ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42210/0 (socket says 172.21.15.81:42210) 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb480c4f640 1 -- 172.21.15.81:0/2013545758 learned_addr learned my addr 172.21.15.81:0/2013545758 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb481450640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb47c0ce920 0x7fb47c164410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.278+0000 7fb473fff640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb47c0ce510 0x7fb47c15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb480c4f640 1 -- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb47c0ce920 msgr2=0x7fb47c164410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb480c4f640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb47c0ce920 0x7fb47c164410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb480c4f640 1 -- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb47c0ce510 msgr2=0x7fb47c15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb480c4f640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb47c0ce510 0x7fb47c15bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb480c4f640 1 -- 172.21.15.81:0/2013545758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb47c0b6980 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb480c4f640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb47c0b6eb0 0x7fb47c06ee50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb47800d700 tx=0x7fb47800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb4717fa640 1 -- 172.21.15.81:0/2013545758 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb478017070 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb4717fa640 1 -- 172.21.15.81:0/2013545758 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb478005310 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb482eb3640 1 -- 172.21.15.81:0/2013545758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb47c0b6c60 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb4717fa640 1 -- 172.21.15.81:0/2013545758 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb478009820 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.279+0000 7fb482eb3640 1 -- 172.21.15.81:0/2013545758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb47c161e20 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.280+0000 7fb482eb3640 1 -- 172.21.15.81:0/2013545758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb47c0d02d0 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.281+0000 7fb4717fa640 1 -- 172.21.15.81:0/2013545758 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb4780099c0 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.281+0000 7fb4717fa640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb4680762d0 0x7fb4680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:45.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.281+0000 7fb4717fa640 1 -- 172.21.15.81:0/2013545758 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fb478097840 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.282+0000 7fb473fff640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb4680762d0 0x7fb4680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:45.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.282+0000 7fb473fff640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb4680762d0 0x7fb4680787e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0b1900 tx=0x7fb46c009040 comp rx=0 tx=0).ready entity=mgr.4126 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:45.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.287+0000 7fb4717fa640 1 -- 172.21.15.81:0/2013545758 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb4780611a0 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.511+0000 7fb482eb3640 1 -- 172.21.15.81:0/2013545758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fb47c0cdf00 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.981+0000 7fb4717fa640 1 -- 172.21.15.81:0/2013545758 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) ==== 92+0+0 (secure 0 0 0) 0x7fb478069190 con 0x7fb47c0b6eb0 2024-06-29T14:23:45.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 -- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb4680762d0 msgr2=0x7fb4680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:45.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb4680762d0 0x7fb4680787e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0b1900 tx=0x7fb46c009040 comp rx=0 tx=0).stop 2024-06-29T14:23:45.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 -- 172.21.15.81:0/2013545758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb47c0b6eb0 msgr2=0x7fb47c06ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:45.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb47c0b6eb0 0x7fb47c06ee50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb47800d700 tx=0x7fb47800dbc0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 -- 172.21.15.81:0/2013545758 shutdown_connections 2024-06-29T14:23:45.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fb4680762d0 0x7fb4680787e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb47c0ce920 0x7fb47c164410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb47c0ce510 0x7fb47c15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 --2- 172.21.15.81:0/2013545758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb47c0b6eb0 0x7fb47c06ee50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 -- 172.21.15.81:0/2013545758 >> 172.21.15.81:0/2013545758 conn(0x7fb47c0b3510 msgr2=0x7fb47c0ccca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.983+0000 7fb482eb3640 1 -- 172.21.15.81:0/2013545758 shutdown_connections 2024-06-29T14:23:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:45.984+0000 7fb482eb3640 1 -- 172.21.15.81:0/2013545758 wait complete. 2024-06-29T14:23:45.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:23:46.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.120+0000 7fde56c5b640 1 -- 172.21.15.81:0/540988082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde500d2950 msgr2=0x7fde50169940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.120+0000 7fde56c5b640 1 --2- 172.21.15.81:0/540988082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde500d2950 0x7fde50169940 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fde48005fd0 tx=0x7fde4802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.120+0000 7fde56c5b640 1 -- 172.21.15.81:0/540988082 shutdown_connections 2024-06-29T14:23:46.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.120+0000 7fde56c5b640 1 --2- 172.21.15.81:0/540988082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde500d2950 0x7fde50169940 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.120+0000 7fde56c5b640 1 -- 172.21.15.81:0/540988082 >> 172.21.15.81:0/540988082 conn(0x7fde500d3c60 msgr2=0x7fde500d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:46.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.120+0000 7fde56c5b640 1 -- 172.21.15.81:0/540988082 shutdown_connections 2024-06-29T14:23:46.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.120+0000 7fde56c5b640 1 -- 172.21.15.81:0/540988082 wait complete. 2024-06-29T14:23:46.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.122+0000 7fde56c5b640 1 Processor -- start 2024-06-29T14:23:46.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.122+0000 7fde56c5b640 1 -- start start 2024-06-29T14:23:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.123+0000 7fde56c5b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde500ce770 0x7fde501d2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.123+0000 7fde56c5b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde500d2950 0x7fde501e14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde56c5b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde501e1a00 0x7fde501e1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde56c5b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fde500d4990 con 0x7fde500ce770 2024-06-29T14:23:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde56c5b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fde500d1630 con 0x7fde500d2950 2024-06-29T14:23:46.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde56c5b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fde500d4b10 con 0x7fde501e1a00 2024-06-29T14:23:46.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde549f7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde500ce770 0x7fde501d2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:46.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde549f7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde500ce770 0x7fde501d2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42236/0 (socket says 172.21.15.81:42236) 2024-06-29T14:23:46.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde549f7640 1 -- 172.21.15.81:0/1008568106 learned_addr learned my addr 172.21.15.81:0/1008568106 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:46.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde47fff640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde500d2950 0x7fde501e14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:46.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde551f8640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde501e1a00 0x7fde501e1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:46.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde549f7640 1 -- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde501e1a00 msgr2=0x7fde501e1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde549f7640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde501e1a00 0x7fde501e1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde549f7640 1 -- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde500d2950 msgr2=0x7fde501e14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde549f7640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde500d2950 0x7fde501e14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde549f7640 1 -- 172.21.15.81:0/1008568106 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde500cbde0 con 0x7fde500ce770 2024-06-29T14:23:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde549f7640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde500ce770 0x7fde501d2e70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00c390 tx=0x7fde4c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde457fa640 1 -- 172.21.15.81:0/1008568106 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4c011790 con 0x7fde500ce770 2024-06-29T14:23:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde457fa640 1 -- 172.21.15.81:0/1008568106 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde4c017070 con 0x7fde500ce770 2024-06-29T14:23:46.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde56c5b640 1 -- 172.21.15.81:0/1008568106 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde500cc0c0 con 0x7fde500ce770 2024-06-29T14:23:46.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde457fa640 1 -- 172.21.15.81:0/1008568106 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4c011cc0 con 0x7fde500ce770 2024-06-29T14:23:46.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.124+0000 7fde56c5b640 1 -- 172.21.15.81:0/1008568106 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde500cc680 con 0x7fde500ce770 2024-06-29T14:23:46.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.126+0000 7fde56c5b640 1 -- 172.21.15.81:0/1008568106 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde500b9500 con 0x7fde500ce770 2024-06-29T14:23:46.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.126+0000 7fde457fa640 1 -- 172.21.15.81:0/1008568106 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 6) ==== 98218+0+0 (secure 0 0 0) 0x7fde4c01c460 con 0x7fde500ce770 2024-06-29T14:23:46.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.126+0000 7fde457fa640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde3c076320 0x7fde3c078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.127+0000 7fde47fff640 1 -- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde3c076320 msgr2=0x7fde3c078830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3101542440 2024-06-29T14:23:46.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.127+0000 7fde457fa640 1 -- 172.21.15.81:0/1008568106 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fde4c097540 con 0x7fde500ce770 2024-06-29T14:23:46.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.127+0000 7fde47fff640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde3c076320 0x7fde3c078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:23:46.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.131+0000 7fde457fa640 1 -- 172.21.15.81:0/1008568106 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fde500b9500 con 0x7fde500ce770 2024-06-29T14:23:46.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.322+0000 7fde56c5b640 1 -- 172.21.15.81:0/1008568106 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fde500b2aa0 con 0x7fde500ce770 2024-06-29T14:23:46.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.327+0000 7fde457fa640 1 -- 172.21.15.81:0/1008568106 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7fde500b2aa0 con 0x7fde500ce770 2024-06-29T14:23:46.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.328+0000 7fde47fff640 1 -- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde3c076320 msgr2=0x7fde3c078830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3101542440 2024-06-29T14:23:46.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.328+0000 7fde47fff640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde3c076320 0x7fde3c078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:23:46.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.329+0000 7fde56c5b640 1 -- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde3c076320 msgr2=0x7fde3c078830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:23:46.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.329+0000 7fde56c5b640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde3c076320 0x7fde3c078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.330+0000 7fde56c5b640 1 -- 172.21.15.81:0/1008568106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde500ce770 msgr2=0x7fde501d2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.330+0000 7fde56c5b640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde500ce770 0x7fde501d2e70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00c390 tx=0x7fde4c00c850 comp rx=0 tx=0).stop 2024-06-29T14:23:46.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.330+0000 7fde56c5b640 1 -- 172.21.15.81:0/1008568106 shutdown_connections 2024-06-29T14:23:46.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.330+0000 7fde56c5b640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7fde3c076320 0x7fde3c078830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.330+0000 7fde56c5b640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde501e1a00 0x7fde501e1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.330+0000 7fde56c5b640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde500d2950 0x7fde501e14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.330+0000 7fde56c5b640 1 --2- 172.21.15.81:0/1008568106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde500ce770 0x7fde501d2e70 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.330+0000 7fde56c5b640 1 -- 172.21.15.81:0/1008568106 >> 172.21.15.81:0/1008568106 conn(0x7fde500d3c60 msgr2=0x7fde501e5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:46.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.330+0000 7fde56c5b640 1 -- 172.21.15.81:0/1008568106 shutdown_connections 2024-06-29T14:23:46.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.330+0000 7fde56c5b640 1 -- 172.21.15.81:0/1008568106 wait complete. 2024-06-29T14:23:46.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:23:46.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.460+0000 7f29709a1640 1 -- 172.21.15.81:0/2239473546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f296c0cccf0 msgr2=0x7f296c0d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.460+0000 7f29709a1640 1 --2- 172.21.15.81:0/2239473546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f296c0cccf0 0x7f296c0d0230 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2954005fd0 tx=0x7f295402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.460+0000 7f29709a1640 1 -- 172.21.15.81:0/2239473546 shutdown_connections 2024-06-29T14:23:46.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.460+0000 7f29709a1640 1 --2- 172.21.15.81:0/2239473546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f296c0cccf0 0x7f296c0d0230 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.460+0000 7f29709a1640 1 --2- 172.21.15.81:0/2239473546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f296c0b3200 0x7f296c0cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.460+0000 7f29709a1640 1 -- 172.21.15.81:0/2239473546 >> 172.21.15.81:0/2239473546 conn(0x7f296c0b3680 msgr2=0x7f296c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.460+0000 7f29709a1640 1 -- 172.21.15.81:0/2239473546 shutdown_connections 2024-06-29T14:23:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.460+0000 7f29709a1640 1 -- 172.21.15.81:0/2239473546 wait complete. 2024-06-29T14:23:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.462+0000 7f29709a1640 1 Processor -- start 2024-06-29T14:23:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.462+0000 7f29709a1640 1 -- start start 2024-06-29T14:23:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.463+0000 7f29709a1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f296c0b3200 0x7f296c0ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.463+0000 7f29709a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f296c0cccf0 0x7f296c15a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.463+0000 7f29709a1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f296c0d0770 0x7f296c0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f29709a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f296c0cedb0 con 0x7f296c0cccf0 2024-06-29T14:23:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f29709a1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f296c0cec30 con 0x7f296c0d0770 2024-06-29T14:23:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f29709a1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f296c0cef30 con 0x7f296c0b3200 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2969d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f296c0cccf0 0x7f296c15a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2969d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f296c0cccf0 0x7f296c15a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42264/0 (socket says 172.21.15.81:42264) 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2969d9b640 1 -- 172.21.15.81:0/2498492105 learned_addr learned my addr 172.21.15.81:0/2498492105 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f296ad9d640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f296c0d0770 0x7f296c0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f296a59c640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f296c0b3200 0x7f296c0ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2969d9b640 1 -- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f296c0b3200 msgr2=0x7f296c0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2969d9b640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f296c0b3200 0x7f296c0ae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2969d9b640 1 -- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f296c0d0770 msgr2=0x7f296c0b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2969d9b640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f296c0d0770 0x7f296c0b0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2969d9b640 1 -- 172.21.15.81:0/2498492105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296c1e5f90 con 0x7f296c0cccf0 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2969d9b640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f296c0cccf0 0x7f296c15a9f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2954000c00 tx=0x7f29540038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2962ffd640 1 -- 172.21.15.81:0/2498492105 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295403e030 con 0x7f296c0cccf0 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2962ffd640 1 -- 172.21.15.81:0/2498492105 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2954003ca0 con 0x7f296c0cccf0 2024-06-29T14:23:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f29709a1640 1 -- 172.21.15.81:0/2498492105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f296c1e6270 con 0x7f296c0cccf0 2024-06-29T14:23:46.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.464+0000 7f2962ffd640 1 -- 172.21.15.81:0/2498492105 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2954035040 con 0x7f296c0cccf0 2024-06-29T14:23:46.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.465+0000 7f29709a1640 1 -- 172.21.15.81:0/2498492105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f296c1e6830 con 0x7f296c0cccf0 2024-06-29T14:23:46.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.466+0000 7f29709a1640 1 -- 172.21.15.81:0/2498492105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f296c0cc7b0 con 0x7f296c0cccf0 2024-06-29T14:23:46.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.466+0000 7f2962ffd640 1 -- 172.21.15.81:0/2498492105 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 6) ==== 98218+0+0 (secure 0 0 0) 0x7f2954041030 con 0x7f296c0cccf0 2024-06-29T14:23:46.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.467+0000 7f2962ffd640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f294c076320 0x7f294c078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.467+0000 7f296a59c640 1 -- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f294c076320 msgr2=0x7f294c078830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3101542440 2024-06-29T14:23:46.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.467+0000 7f2962ffd640 1 -- 172.21.15.81:0/2498492105 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f29540b7ca0 con 0x7f296c0cccf0 2024-06-29T14:23:46.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.467+0000 7f296a59c640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f294c076320 0x7f294c078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:23:46.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.475+0000 7f2962ffd640 1 -- 172.21.15.81:0/2498492105 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2954080f30 con 0x7f296c0cccf0 2024-06-29T14:23:46.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.663+0000 7f29709a1640 1 -- 172.21.15.81:0/2498492105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f296c0d1a10 con 0x7f296c0cccf0 2024-06-29T14:23:46.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.668+0000 7f296a59c640 1 -- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f294c076320 msgr2=0x7f294c078830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3101542440 2024-06-29T14:23:46.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.668+0000 7f296a59c640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f294c076320 0x7f294c078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:23:46.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.669+0000 7f2962ffd640 1 -- 172.21.15.81:0/2498492105 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f2954085230 con 0x7f296c0cccf0 2024-06-29T14:23:46.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 -- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f294c076320 msgr2=0x7f294c078830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:23:46.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f294c076320 0x7f294c078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 -- 172.21.15.81:0/2498492105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f296c0cccf0 msgr2=0x7f296c15a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f296c0cccf0 0x7f296c15a9f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2954000c00 tx=0x7f29540038a0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 -- 172.21.15.81:0/2498492105 shutdown_connections 2024-06-29T14:23:46.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f294c076320 0x7f294c078830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f296c0d0770 0x7f296c0b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f296c0cccf0 0x7f296c15a9f0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 --2- 172.21.15.81:0/2498492105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f296c0b3200 0x7f296c0ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 -- 172.21.15.81:0/2498492105 >> 172.21.15.81:0/2498492105 conn(0x7f296c0b3680 msgr2=0x7f296c0715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:46.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 -- 172.21.15.81:0/2498492105 shutdown_connections 2024-06-29T14:23:46.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.671+0000 7f29709a1640 1 -- 172.21.15.81:0/2498492105 wait complete. 2024-06-29T14:23:46.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:46.681 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T14:23:46.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:46.701 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2024-06-29T14:23:46.759 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2024-06-29T14:23:46.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.920+0000 7f3cd9622640 1 -- 172.21.15.81:0/1685918044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cd4143860 msgr2=0x7f3cd40b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.920+0000 7f3cd9622640 1 --2- 172.21.15.81:0/1685918044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cd4143860 0x7f3cd40b52b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009830 tx=0x7f3cc402a530 comp rx=0 tx=0).stop 2024-06-29T14:23:46.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.920+0000 7f3cd9622640 1 -- 172.21.15.81:0/1685918044 shutdown_connections 2024-06-29T14:23:46.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.920+0000 7f3cd9622640 1 --2- 172.21.15.81:0/1685918044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cd4143860 0x7f3cd40b52b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.920+0000 7f3cd9622640 1 -- 172.21.15.81:0/1685918044 >> 172.21.15.81:0/1685918044 conn(0x7f3cd40d0b70 msgr2=0x7f3cd40d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:46.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.920+0000 7f3cd9622640 1 -- 172.21.15.81:0/1685918044 shutdown_connections 2024-06-29T14:23:46.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.920+0000 7f3cd9622640 1 -- 172.21.15.81:0/1685918044 wait complete. 2024-06-29T14:23:46.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.922+0000 7f3cd9622640 1 Processor -- start 2024-06-29T14:23:46.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.922+0000 7f3cd9622640 1 -- start start 2024-06-29T14:23:46.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3cd9622640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3cd40b57f0 0x7f3cd415ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3cd9622640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cd4143860 0x7f3cd4136170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3cd9622640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cd413f140 0x7f3cd41f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3cd9622640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd406f850 con 0x7f3cd413f140 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3cd9622640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd406f6d0 con 0x7f3cd4143860 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3cd9622640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3cd406f9d0 con 0x7f3cd40b57f0 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3cd37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cd413f140 0x7f3cd41f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3cd37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cd413f140 0x7f3cd41f1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42274/0 (socket says 172.21.15.81:42274) 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3cd37fe640 1 -- 172.21.15.81:0/2786163053 learned_addr learned my addr 172.21.15.81:0/2786163053 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3cd2ffd640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3cd40b57f0 0x7f3cd415ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.923+0000 7f3ccbfff640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cd4143860 0x7f3cd4136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3cd37fe640 1 -- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3cd40b57f0 msgr2=0x7f3cd415ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3cd37fe640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3cd40b57f0 0x7f3cd415ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3cd37fe640 1 -- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cd4143860 msgr2=0x7f3cd4136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:46.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3cd37fe640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cd4143860 0x7f3cd4136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:46.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3cd37fe640 1 -- 172.21.15.81:0/2786163053 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd413f8c0 con 0x7f3cd413f140 2024-06-29T14:23:46.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3cd37fe640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cd413f140 0x7f3cd41f1b10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00cb10 tx=0x7f3ccc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:46.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc016020 con 0x7f3cd413f140 2024-06-29T14:23:46.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ccc00ad00 con 0x7f3cd413f140 2024-06-29T14:23:46.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3cd9622640 1 -- 172.21.15.81:0/2786163053 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd415e100 con 0x7f3cd413f140 2024-06-29T14:23:46.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc012680 con 0x7f3cd413f140 2024-06-29T14:23:46.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.924+0000 7f3cd9622640 1 -- 172.21.15.81:0/2786163053 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd415bfb0 con 0x7f3cd413f140 2024-06-29T14:23:46.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.925+0000 7f3cd9622640 1 -- 172.21.15.81:0/2786163053 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd415de70 con 0x7f3cd413f140 2024-06-29T14:23:46.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.926+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 6) ==== 98218+0+0 (secure 0 0 0) 0x7f3ccc019020 con 0x7f3cd413f140 2024-06-29T14:23:46.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.926+0000 7f3ccb7fe640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 0x7f3cb4078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:46.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.926+0000 7f3cd2ffd640 1 -- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 msgr2=0x7f3cb4078830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3101542440 2024-06-29T14:23:46.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.926+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f3ccc097da0 con 0x7f3cd413f140 2024-06-29T14:23:46.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.927+0000 7f3cd2ffd640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 0x7f3cb4078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:23:46.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:46.931+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3ccc020020 con 0x7f3cd413f140 2024-06-29T14:23:47.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:47.128+0000 7f3cd2ffd640 1 -- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 msgr2=0x7f3cb4078830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3101542440 2024-06-29T14:23:47.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:47.128+0000 7f3cd2ffd640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 0x7f3cb4078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:23:47.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:47.141+0000 7f3cd9622640 1 -- 172.21.15.81:0/2786163053 --> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f3cd415f090 con 0x7f3cb4076320 2024-06-29T14:23:47.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:47.530+0000 7f3cd2ffd640 1 -- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 msgr2=0x7f3cb4078830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3101542440 2024-06-29T14:23:47.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:47.530+0000 7f3cd2ffd640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 0x7f3cb4078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:23:48.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:48.331+0000 7f3cd2ffd640 1 -- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 msgr2=0x7f3cb4078830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3101542440 2024-06-29T14:23:48.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:48.331+0000 7f3cd2ffd640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 0x7f3cb4078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:23:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:49.604+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 7) ==== 97929+0+0 (secure 0 0 0) 0x7f3ccc05d2a0 con 0x7f3cd413f140 2024-06-29T14:23:49.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:49.605+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 msgr2=0x7f3cb4078830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:23:49.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:49.605+0000 7f3ccb7fe640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/3101542440,v1:172.21.15.167:6833/3101542440] conn(0x7f3cb4076320 0x7f3cb4078830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:49.774 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:23:49.775+0000 7f1d57384640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:49.774 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:23:49.775+0000 7f1d57384640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:49.774 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:23:49.775+0000 7f1d57384640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:49.774 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:23:49.775+0000 7f1d57384640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:49.774 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:23:49.775+0000 7f1d57384640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:49.774 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:23:49.775+0000 7f1d5a38a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:49.774 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:23:49.775+0000 7f1d5a38a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:49.774 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:23:49.775+0000 7f1d5a38a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:49.775 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:23:49.775+0000 7f1d5a38a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:49.775 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:23:49.775+0000 7f1d5a38a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:23:50.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.608+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 8) ==== 98056+0+0 (secure 0 0 0) 0x7f3ccc018cb0 con 0x7f3cd413f140 2024-06-29T14:23:50.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.609+0000 7f3ccb7fe640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f3cb407e220 0x7f3cb4080640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:50.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.609+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f3ccc067860 con 0x7f3cb407e220 2024-06-29T14:23:50.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.611+0000 7f3cd2ffd640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f3cb407e220 0x7f3cb4080640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:50.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.615+0000 7f3cd2ffd640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f3cb407e220 0x7f3cb4080640 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4034000 tx=0x7f3cc4004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.760+0000 7f3ccb7fe640 1 -- 172.21.15.81:0/2786163053 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f3ccc067860 con 0x7f3cb407e220 2024-06-29T14:23:50.759 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/dir0 2024-06-29T14:23:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.763+0000 7f3cd9622640 1 -- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f3cb407e220 msgr2=0x7f3cb4080640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.763+0000 7f3cd9622640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f3cb407e220 0x7f3cb4080640 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4034000 tx=0x7f3cc4004730 comp rx=0 tx=0).stop 2024-06-29T14:23:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.763+0000 7f3cd9622640 1 -- 172.21.15.81:0/2786163053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cd413f140 msgr2=0x7f3cd41f1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.763+0000 7f3cd9622640 1 --2- 172.21.15.81:0/2786163053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cd413f140 0x7f3cd41f1b10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00cb10 tx=0x7f3ccc00a060 comp rx=0 tx=0).stop 2024-06-29T14:23:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.763+0000 7f3cd37fe640 1 -- 172.21.15.81:0/2786163053 reap_dead start 2024-06-29T14:23:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.763+0000 7f3cd9622640 1 -- 172.21.15.81:0/2786163053 shutdown_connections 2024-06-29T14:23:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.763+0000 7f3cd9622640 1 -- 172.21.15.81:0/2786163053 >> 172.21.15.81:0/2786163053 conn(0x7f3cd40d0b70 msgr2=0x7f3cd415a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.763+0000 7f3cd9622640 1 -- 172.21.15.81:0/2786163053 shutdown_connections 2024-06-29T14:23:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.763+0000 7f3cd9622640 1 -- 172.21.15.81:0/2786163053 wait complete. 2024-06-29T14:23:50.773 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2024-06-29T14:23:50.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:50.773 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2024-06-29T14:23:50.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2024-06-29T14:23:50.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.949+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2891365936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d40b5250 msgr2=0x7fc2d40d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:50.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.949+0000 7fc2d93f5640 1 --2- 172.21.15.81:0/2891365936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d40b5250 0x7fc2d40d09c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009830 tx=0x7fc2c802a5a0 comp rx=0 tx=0).stop 2024-06-29T14:23:50.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.949+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2891365936 shutdown_connections 2024-06-29T14:23:50.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.949+0000 7fc2d93f5640 1 --2- 172.21.15.81:0/2891365936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d40b5250 0x7fc2d40d09c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:50.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.949+0000 7fc2d93f5640 1 --2- 172.21.15.81:0/2891365936 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2d40b9040 0x7fc2d40d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:50.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.949+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2891365936 >> 172.21.15.81:0/2891365936 conn(0x7fc2d409cf30 msgr2=0x7fc2d40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:50.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.949+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2891365936 shutdown_connections 2024-06-29T14:23:50.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.949+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2891365936 wait complete. 2024-06-29T14:23:50.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.951+0000 7fc2d93f5640 1 Processor -- start 2024-06-29T14:23:50.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.951+0000 7fc2d93f5640 1 -- start start 2024-06-29T14:23:50.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.952+0000 7fc2d93f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2d40b5250 0x7fc2d41f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.952+0000 7fc2d93f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2d40b7290 0x7fc2d41580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.952+0000 7fc2d93f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d40b9040 0x7fc2d40b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.952+0000 7fc2d93f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d40b7db0 con 0x7fc2d40b9040 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.952+0000 7fc2d93f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d40d2d50 con 0x7fc2d40b5250 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.952+0000 7fc2d93f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc2d40b7f30 con 0x7fc2d40b7290 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.952+0000 7fc2d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d40b9040 0x7fc2d40b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.952+0000 7fc2d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d40b9040 0x7fc2d40b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35870/0 (socket says 172.21.15.81:35870) 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.952+0000 7fc2d37fe640 1 -- 172.21.15.81:0/2879776045 learned_addr learned my addr 172.21.15.81:0/2879776045 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d27fc640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2d40b7290 0x7fc2d41580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d2ffd640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2d40b5250 0x7fc2d41f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d37fe640 1 -- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2d40b7290 msgr2=0x7fc2d41580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d37fe640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2d40b7290 0x7fc2d41580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:50.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d37fe640 1 -- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2d40b5250 msgr2=0x7fc2d41f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:50.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d37fe640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2d40b5250 0x7fc2d41f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:50.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d37fe640 1 -- 172.21.15.81:0/2879776045 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d40b0a70 con 0x7fc2d40b9040 2024-06-29T14:23:50.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d2ffd640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2d40b5250 0x7fc2d41f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:50.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d37fe640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d40b9040 0x7fc2d40b2c50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00c760 tx=0x7fc2cc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:50.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2bb7fe640 1 -- 172.21.15.81:0/2879776045 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc003b70 con 0x7fc2d40b9040 2024-06-29T14:23:50.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2bb7fe640 1 -- 172.21.15.81:0/2879776045 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2cc003d30 con 0x7fc2d40b9040 2024-06-29T14:23:50.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2879776045 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d40b0d50 con 0x7fc2d40b9040 2024-06-29T14:23:50.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2bb7fe640 1 -- 172.21.15.81:0/2879776045 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc0052d0 con 0x7fc2d40b9040 2024-06-29T14:23:50.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.953+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2879776045 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d41f8de0 con 0x7fc2d40b9040 2024-06-29T14:23:50.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.954+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2879776045 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d40b84c0 con 0x7fc2d40b9040 2024-06-29T14:23:50.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.955+0000 7fc2bb7fe640 1 -- 172.21.15.81:0/2879776045 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 98056+0+0 (secure 0 0 0) 0x7fc2cc005470 con 0x7fc2d40b9040 2024-06-29T14:23:50.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.955+0000 7fc2bb7fe640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fc2c00761a0 0x7fc2c00786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:50.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.955+0000 7fc2bb7fe640 1 -- 172.21.15.81:0/2879776045 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7fc2cc0971e0 con 0x7fc2d40b9040 2024-06-29T14:23:50.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.955+0000 7fc2d2ffd640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fc2c00761a0 0x7fc2c00786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:50.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.956+0000 7fc2d2ffd640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fc2c00761a0 0x7fc2c00786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4005fd0 tx=0x7fc2c400f360 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:50.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:50.960+0000 7fc2bb7fe640 1 -- 172.21.15.81:0/2879776045 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc2cc060d80 con 0x7fc2d40b9040 2024-06-29T14:23:51.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.161+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2879776045 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc2d40b0820 con 0x7fc2c00761a0 2024-06-29T14:23:51.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.250+0000 7fc2bb7fe640 1 -- 172.21.15.81:0/2879776045 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fc2d40b0820 con 0x7fc2c00761a0 2024-06-29T14:23:51.249 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/dir1 2024-06-29T14:23:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.252+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fc2c00761a0 msgr2=0x7fc2c00786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.252+0000 7fc2d93f5640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fc2c00761a0 0x7fc2c00786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4005fd0 tx=0x7fc2c400f360 comp rx=0 tx=0).stop 2024-06-29T14:23:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.252+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2879776045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d40b9040 msgr2=0x7fc2d40b2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.252+0000 7fc2d93f5640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d40b9040 0x7fc2d40b2c50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00c760 tx=0x7fc2cc00cc20 comp rx=0 tx=0).stop 2024-06-29T14:23:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.252+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2879776045 shutdown_connections 2024-06-29T14:23:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.252+0000 7fc2d93f5640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fc2c00761a0 0x7fc2c00786b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.252+0000 7fc2d93f5640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d40b9040 0x7fc2d40b2c50 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.252+0000 7fc2d93f5640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2d40b7290 0x7fc2d41580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.252+0000 7fc2d93f5640 1 --2- 172.21.15.81:0/2879776045 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2d40b5250 0x7fc2d41f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.252+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2879776045 >> 172.21.15.81:0/2879776045 conn(0x7fc2d409cf30 msgr2=0x7fc2d4160070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.253+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2879776045 shutdown_connections 2024-06-29T14:23:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.253+0000 7fc2d93f5640 1 -- 172.21.15.81:0/2879776045 wait complete. 2024-06-29T14:23:51.262 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2024-06-29T14:23:51.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:51.262 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2024-06-29T14:23:51.279 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2024-06-29T14:23:51.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.436+0000 7fda332ef640 1 -- 172.21.15.81:0/516264114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda2c0d2a90 msgr2=0x7fda2c0d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:51.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.436+0000 7fda332ef640 1 --2- 172.21.15.81:0/516264114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda2c0d2a90 0x7fda2c0d09c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fda24005fd0 tx=0x7fda2402a940 comp rx=0 tx=0).stop 2024-06-29T14:23:51.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.436+0000 7fda332ef640 1 -- 172.21.15.81:0/516264114 shutdown_connections 2024-06-29T14:23:51.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.436+0000 7fda332ef640 1 --2- 172.21.15.81:0/516264114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda2c0d2a90 0x7fda2c0d09c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.436+0000 7fda332ef640 1 --2- 172.21.15.81:0/516264114 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda2c0b9040 0x7fda2c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.436+0000 7fda332ef640 1 -- 172.21.15.81:0/516264114 >> 172.21.15.81:0/516264114 conn(0x7fda2c09cf30 msgr2=0x7fda2c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:51.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.436+0000 7fda332ef640 1 -- 172.21.15.81:0/516264114 shutdown_connections 2024-06-29T14:23:51.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.436+0000 7fda332ef640 1 -- 172.21.15.81:0/516264114 wait complete. 2024-06-29T14:23:51.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.438+0000 7fda332ef640 1 Processor -- start 2024-06-29T14:23:51.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.438+0000 7fda332ef640 1 -- start start 2024-06-29T14:23:51.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.439+0000 7fda332ef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda2c0b5250 0x7fda2c152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.439+0000 7fda332ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda2c0b9040 0x7fda2c0c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.439+0000 7fda332ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda2c0d2a90 0x7fda2c0c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.439+0000 7fda332ef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c0b7d30 con 0x7fda2c0b9040 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.439+0000 7fda332ef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c0b7bb0 con 0x7fda2c0b5250 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.439+0000 7fda332ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fda2c0b7eb0 con 0x7fda2c0d2a90 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.439+0000 7fda3088a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda2c0b9040 0x7fda2c0c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.439+0000 7fda3088a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda2c0b9040 0x7fda2c0c9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35896/0 (socket says 172.21.15.81:35896) 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.439+0000 7fda3088a640 1 -- 172.21.15.81:0/4096252288 learned_addr learned my addr 172.21.15.81:0/4096252288 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda3188c640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda2c0d2a90 0x7fda2c0c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda3088a640 1 -- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda2c0d2a90 msgr2=0x7fda2c0c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda3108b640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda2c0b5250 0x7fda2c152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:51.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda3088a640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda2c0d2a90 0x7fda2c0c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda3088a640 1 -- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda2c0b5250 msgr2=0x7fda2c152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:51.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda3088a640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda2c0b5250 0x7fda2c152f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda3088a640 1 -- 172.21.15.81:0/4096252288 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2c0b11d0 con 0x7fda2c0b9040 2024-06-29T14:23:51.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda3088a640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda2c0b9040 0x7fda2c0c9670 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fda2000c3a0 tx=0x7fda2000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:51.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda29ffb640 1 -- 172.21.15.81:0/4096252288 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda20011790 con 0x7fda2c0b9040 2024-06-29T14:23:51.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda29ffb640 1 -- 172.21.15.81:0/4096252288 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda20017070 con 0x7fda2c0b9040 2024-06-29T14:23:51.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda332ef640 1 -- 172.21.15.81:0/4096252288 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda2c0b14b0 con 0x7fda2c0b9040 2024-06-29T14:23:51.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda29ffb640 1 -- 172.21.15.81:0/4096252288 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda20011ca0 con 0x7fda2c0b9040 2024-06-29T14:23:51.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.440+0000 7fda332ef640 1 -- 172.21.15.81:0/4096252288 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda2c0b1a70 con 0x7fda2c0b9040 2024-06-29T14:23:51.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.441+0000 7fda332ef640 1 -- 172.21.15.81:0/4096252288 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda2c0b4e40 con 0x7fda2c0b9040 2024-06-29T14:23:51.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.442+0000 7fda29ffb640 1 -- 172.21.15.81:0/4096252288 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 98056+0+0 (secure 0 0 0) 0x7fda2001c460 con 0x7fda2c0b9040 2024-06-29T14:23:51.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.442+0000 7fda29ffb640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fda180761a0 0x7fda180786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:51.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.442+0000 7fda29ffb640 1 -- 172.21.15.81:0/4096252288 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7fda200972e0 con 0x7fda2c0b9040 2024-06-29T14:23:51.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.443+0000 7fda3108b640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fda180761a0 0x7fda180786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:51.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.443+0000 7fda3108b640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fda180761a0 0x7fda180786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00b9a0 tx=0x7fda1c005c20 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:51.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.446+0000 7fda29ffb640 1 -- 172.21.15.81:0/4096252288 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fda2c0b4e40 con 0x7fda2c0b9040 2024-06-29T14:23:51.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.653+0000 7fda332ef640 1 -- 172.21.15.81:0/4096252288 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fda2c0b0f60 con 0x7fda180761a0 2024-06-29T14:23:51.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.661+0000 7fda29ffb640 1 -- 172.21.15.81:0/4096252288 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 9) ==== 98321+0+0 (secure 0 0 0) 0x7fda2005c980 con 0x7fda2c0b9040 2024-06-29T14:23:51.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.735+0000 7fda29ffb640 1 -- 172.21.15.81:0/4096252288 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fda2c0b0f60 con 0x7fda180761a0 2024-06-29T14:23:51.734 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/dir2 2024-06-29T14:23:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.737+0000 7fda332ef640 1 -- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fda180761a0 msgr2=0x7fda180786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.737+0000 7fda332ef640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fda180761a0 0x7fda180786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00b9a0 tx=0x7fda1c005c20 comp rx=0 tx=0).stop 2024-06-29T14:23:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.737+0000 7fda332ef640 1 -- 172.21.15.81:0/4096252288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda2c0b9040 msgr2=0x7fda2c0c9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.737+0000 7fda332ef640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda2c0b9040 0x7fda2c0c9670 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fda2000c3a0 tx=0x7fda2000c860 comp rx=0 tx=0).stop 2024-06-29T14:23:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.738+0000 7fda332ef640 1 -- 172.21.15.81:0/4096252288 shutdown_connections 2024-06-29T14:23:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.738+0000 7fda332ef640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fda180761a0 0x7fda180786b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.738+0000 7fda332ef640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda2c0d2a90 0x7fda2c0c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.738+0000 7fda332ef640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda2c0b9040 0x7fda2c0c9670 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.738+0000 7fda332ef640 1 --2- 172.21.15.81:0/4096252288 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda2c0b5250 0x7fda2c152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.738+0000 7fda332ef640 1 -- 172.21.15.81:0/4096252288 >> 172.21.15.81:0/4096252288 conn(0x7fda2c09cf30 msgr2=0x7fda2c15ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.738+0000 7fda332ef640 1 -- 172.21.15.81:0/4096252288 shutdown_connections 2024-06-29T14:23:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.738+0000 7fda332ef640 1 -- 172.21.15.81:0/4096252288 wait complete. 2024-06-29T14:23:51.747 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2024-06-29T14:23:51.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:51.747 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2024-06-29T14:23:51.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2024-06-29T14:23:51.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.923+0000 7efd3853c640 1 -- 172.21.15.81:0/3463272512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd3015b230 msgr2=0x7efd3006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:51.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.923+0000 7efd3853c640 1 --2- 172.21.15.81:0/3463272512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd3015b230 0x7efd3006f660 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7efd300703b0 tx=0x7efd2002a560 comp rx=0 tx=0).stop 2024-06-29T14:23:51.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.923+0000 7efd3853c640 1 -- 172.21.15.81:0/3463272512 shutdown_connections 2024-06-29T14:23:51.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.923+0000 7efd3853c640 1 --2- 172.21.15.81:0/3463272512 >> v2:172.21.15.167:3300/0 conn(0x7efd3006fba0 0x7efd30071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.923+0000 7efd3853c640 1 --2- 172.21.15.81:0/3463272512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd3015b230 0x7efd3006f660 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.923+0000 7efd3853c640 1 -- 172.21.15.81:0/3463272512 >> 172.21.15.81:0/3463272512 conn(0x7efd3006ccc0 msgr2=0x7efd300b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:51.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.923+0000 7efd3853c640 1 -- 172.21.15.81:0/3463272512 shutdown_connections 2024-06-29T14:23:51.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.923+0000 7efd3853c640 1 -- 172.21.15.81:0/3463272512 wait complete. 2024-06-29T14:23:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.924+0000 7efd3853c640 1 Processor -- start 2024-06-29T14:23:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.924+0000 7efd3853c640 1 -- start start 2024-06-29T14:23:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd3853c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd3006fba0 0x7efd30154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd3853c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd300b3640 0x7efd301492e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd3853c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd3015b230 0x7efd300b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd3853c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd3006edd0 con 0x7efd3006fba0 2024-06-29T14:23:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd3853c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efd300cf7d0 con 0x7efd300b3640 2024-06-29T14:23:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd3853c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efd3006ef50 con 0x7efd3015b230 2024-06-29T14:23:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd362d8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd3006fba0 0x7efd30154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd362d8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd3006fba0 0x7efd30154600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35926/0 (socket says 172.21.15.81:35926) 2024-06-29T14:23:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd362d8640 1 -- 172.21.15.81:0/3796217312 learned_addr learned my addr 172.21.15.81:0/3796217312 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd35ad7640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd300b3640 0x7efd301492e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd36ad9640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd3015b230 0x7efd300b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd362d8640 1 -- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd3015b230 msgr2=0x7efd300b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd362d8640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd3015b230 0x7efd300b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd362d8640 1 -- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd300b3640 msgr2=0x7efd301492e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd362d8640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd300b3640 0x7efd301492e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd362d8640 1 -- 172.21.15.81:0/3796217312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3014d2a0 con 0x7efd3006fba0 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd362d8640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd3006fba0 0x7efd30154600 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00c3d0 tx=0x7efd2c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.926+0000 7efd26ffd640 1 -- 172.21.15.81:0/3796217312 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c017070 con 0x7efd3006fba0 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.927+0000 7efd26ffd640 1 -- 172.21.15.81:0/3796217312 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd2c0053c0 con 0x7efd3006fba0 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.927+0000 7efd26ffd640 1 -- 172.21.15.81:0/3796217312 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd2c003850 con 0x7efd3006fba0 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.927+0000 7efd3853c640 1 -- 172.21.15.81:0/3796217312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3014a6f0 con 0x7efd3006fba0 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.927+0000 7efd3853c640 1 -- 172.21.15.81:0/3796217312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd300aa390 con 0x7efd3006fba0 2024-06-29T14:23:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.927+0000 7efd3853c640 1 -- 172.21.15.81:0/3796217312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd3006f4b0 con 0x7efd3006fba0 2024-06-29T14:23:51.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.928+0000 7efd26ffd640 1 -- 172.21.15.81:0/3796217312 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 9) ==== 98321+0+0 (secure 0 0 0) 0x7efd2c01a070 con 0x7efd3006fba0 2024-06-29T14:23:51.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.929+0000 7efd26ffd640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efd14076470 0x7efd14078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:51.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.929+0000 7efd26ffd640 1 -- 172.21.15.81:0/3796217312 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7efd2c096e20 con 0x7efd3006fba0 2024-06-29T14:23:51.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.929+0000 7efd35ad7640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efd14076470 0x7efd14078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:51.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.929+0000 7efd35ad7640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efd14076470 0x7efd14078980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efd20033000 tx=0x7efd20004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:51.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:51.933+0000 7efd26ffd640 1 -- 172.21.15.81:0/3796217312 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efd3006ffb0 con 0x7efd3006fba0 2024-06-29T14:23:52.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.133+0000 7efd3853c640 1 -- 172.21.15.81:0/3796217312 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7efd300aa630 con 0x7efd14076470 2024-06-29T14:23:52.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.210+0000 7efd26ffd640 1 -- 172.21.15.81:0/3796217312 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7efd300aa630 con 0x7efd14076470 2024-06-29T14:23:52.208 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/dir3 2024-06-29T14:23:52.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 -- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efd14076470 msgr2=0x7efd14078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efd14076470 0x7efd14078980 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efd20033000 tx=0x7efd20004730 comp rx=0 tx=0).stop 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 -- 172.21.15.81:0/3796217312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd3006fba0 msgr2=0x7efd30154600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd3006fba0 0x7efd30154600 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efd2c00c3d0 tx=0x7efd2c00c890 comp rx=0 tx=0).stop 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 -- 172.21.15.81:0/3796217312 shutdown_connections 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efd14076470 0x7efd14078980 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd3015b230 0x7efd300b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd300b3640 0x7efd301492e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 --2- 172.21.15.81:0/3796217312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd3006fba0 0x7efd30154600 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 -- 172.21.15.81:0/3796217312 >> 172.21.15.81:0/3796217312 conn(0x7efd3006ccc0 msgr2=0x7efd3006e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 -- 172.21.15.81:0/3796217312 shutdown_connections 2024-06-29T14:23:52.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.212+0000 7efd3853c640 1 -- 172.21.15.81:0/3796217312 wait complete. 2024-06-29T14:23:52.221 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2024-06-29T14:23:52.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:52.222 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2024-06-29T14:23:52.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2024-06-29T14:23:52.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.396+0000 7f5f05e65640 1 -- 172.21.15.81:0/1064424246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f000ad0c0 msgr2=0x7f5f0015bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:52.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.396+0000 7f5f05e65640 1 --2- 172.21.15.81:0/1064424246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f000ad0c0 0x7f5f0015bdf0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5f00073520 tx=0x7f5ef002a4d0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.396+0000 7f5f05e65640 1 -- 172.21.15.81:0/1064424246 shutdown_connections 2024-06-29T14:23:52.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.396+0000 7f5f05e65640 1 --2- 172.21.15.81:0/1064424246 >> v2:172.21.15.167:3300/0 conn(0x7f5f00074630 0x7f5f0006c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.396+0000 7f5f05e65640 1 --2- 172.21.15.81:0/1064424246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f000ad0c0 0x7f5f0015bdf0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.396+0000 7f5f05e65640 1 -- 172.21.15.81:0/1064424246 >> 172.21.15.81:0/1064424246 conn(0x7f5f00072950 msgr2=0x7f5f000b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:52.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.396+0000 7f5f05e65640 1 -- 172.21.15.81:0/1064424246 shutdown_connections 2024-06-29T14:23:52.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.396+0000 7f5f05e65640 1 -- 172.21.15.81:0/1064424246 wait complete. 2024-06-29T14:23:52.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.398+0000 7f5f05e65640 1 Processor -- start 2024-06-29T14:23:52.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.398+0000 7f5f05e65640 1 -- start start 2024-06-29T14:23:52.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5f05e65640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f00073b10 0x7f5f00154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:52.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5f05e65640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f00074630 0x7f5f00150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:52.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5f05e65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f000ad0c0 0x7f5f00151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:52.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5f05e65640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0006d000 con 0x7f5f000ad0c0 2024-06-29T14:23:52.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5f05e65640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0006ce80 con 0x7f5f00073b10 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5f05e65640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5f0006d180 con 0x7f5f00074630 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5efffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f000ad0c0 0x7f5f00151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5efffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f000ad0c0 0x7f5f00151340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35940/0 (socket says 172.21.15.81:35940) 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5efffff640 1 -- 172.21.15.81:0/1955105062 learned_addr learned my addr 172.21.15.81:0/1955105062 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5eff7fe640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f00073b10 0x7f5f00154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5efeffd640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f00074630 0x7f5f00150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5efffff640 1 -- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f00074630 msgr2=0x7f5f00150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5efffff640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f00074630 0x7f5f00150e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5efffff640 1 -- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f00073b10 msgr2=0x7f5f00154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5efffff640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f00073b10 0x7f5f00154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5efffff640 1 -- 172.21.15.81:0/1955105062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f00151c10 con 0x7f5f000ad0c0 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.400+0000 7f5efffff640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f000ad0c0 0x7f5f00151340 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800b9e0 tx=0x7f5ef800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.401+0000 7f5ee7fff640 1 -- 172.21.15.81:0/1955105062 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8019070 con 0x7f5f000ad0c0 2024-06-29T14:23:52.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.401+0000 7f5ee7fff640 1 -- 172.21.15.81:0/1955105062 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ef800cc10 con 0x7f5f000ad0c0 2024-06-29T14:23:52.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.401+0000 7f5f05e65640 1 -- 172.21.15.81:0/1955105062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f00151ef0 con 0x7f5f000ad0c0 2024-06-29T14:23:52.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.401+0000 7f5ee7fff640 1 -- 172.21.15.81:0/1955105062 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef80146a0 con 0x7f5f000ad0c0 2024-06-29T14:23:52.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.401+0000 7f5f05e65640 1 -- 172.21.15.81:0/1955105062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f001524b0 con 0x7f5f000ad0c0 2024-06-29T14:23:52.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.402+0000 7f5f05e65640 1 -- 172.21.15.81:0/1955105062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0006d6c0 con 0x7f5f000ad0c0 2024-06-29T14:23:52.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.402+0000 7f5ee7fff640 1 -- 172.21.15.81:0/1955105062 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 9) ==== 98321+0+0 (secure 0 0 0) 0x7f5ef801c070 con 0x7f5f000ad0c0 2024-06-29T14:23:52.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.403+0000 7f5ee7fff640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5eec076470 0x7f5eec078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:52.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.403+0000 7f5ee7fff640 1 -- 172.21.15.81:0/1955105062 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f5ef8097780 con 0x7f5f000ad0c0 2024-06-29T14:23:52.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.403+0000 7f5eff7fe640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5eec076470 0x7f5eec078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:52.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.403+0000 7f5eff7fe640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5eec076470 0x7f5eec078980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0033000 tx=0x7f5ef0004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:52.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.407+0000 7f5ee7fff640 1 -- 172.21.15.81:0/1955105062 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5ef80607f0 con 0x7f5f000ad0c0 2024-06-29T14:23:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.608+0000 7f5f05e65640 1 -- 172.21.15.81:0/1955105062 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f5f000d0a50 con 0x7f5eec076470 2024-06-29T14:23:52.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.700+0000 7f5ee7fff640 1 -- 172.21.15.81:0/1955105062 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f5f000d0a50 con 0x7f5eec076470 2024-06-29T14:23:52.698 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/dir4 2024-06-29T14:23:52.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 -- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5eec076470 msgr2=0x7f5eec078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:52.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5eec076470 0x7f5eec078980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0033000 tx=0x7f5ef0004730 comp rx=0 tx=0).stop 2024-06-29T14:23:52.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 -- 172.21.15.81:0/1955105062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f000ad0c0 msgr2=0x7f5f00151340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:52.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f000ad0c0 0x7f5f00151340 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800b9e0 tx=0x7f5ef800bea0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 -- 172.21.15.81:0/1955105062 shutdown_connections 2024-06-29T14:23:52.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5eec076470 0x7f5eec078980 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f000ad0c0 0x7f5f00151340 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f00074630 0x7f5f00150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 --2- 172.21.15.81:0/1955105062 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f00073b10 0x7f5f00154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 -- 172.21.15.81:0/1955105062 >> 172.21.15.81:0/1955105062 conn(0x7f5f00072950 msgr2=0x7f5f000b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 -- 172.21.15.81:0/1955105062 shutdown_connections 2024-06-29T14:23:52.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.702+0000 7f5f05e65640 1 -- 172.21.15.81:0/1955105062 wait complete. 2024-06-29T14:23:52.712 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2024-06-29T14:23:52.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:52.712 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2024-06-29T14:23:52.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2024-06-29T14:23:52.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.887+0000 7fd2acbe4640 1 -- 172.21.15.81:0/3096443510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a806fa90 msgr2=0x7fd2a806c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:52.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.887+0000 7fd2acbe4640 1 --2- 172.21.15.81:0/3096443510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a806fa90 0x7fd2a806c830 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8072d60 tx=0x7fd2a002a560 comp rx=0 tx=0).stop 2024-06-29T14:23:52.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.887+0000 7fd2acbe4640 1 -- 172.21.15.81:0/3096443510 shutdown_connections 2024-06-29T14:23:52.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.887+0000 7fd2acbe4640 1 --2- 172.21.15.81:0/3096443510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a806fa90 0x7fd2a806c830 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.887+0000 7fd2acbe4640 1 --2- 172.21.15.81:0/3096443510 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2a806f680 0x7fd2a806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.887+0000 7fd2acbe4640 1 --2- 172.21.15.81:0/3096443510 >> v2:172.21.15.167:3300/0 conn(0x7fd2a8074520 0x7fd2a806f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.887+0000 7fd2acbe4640 1 -- 172.21.15.81:0/3096443510 >> 172.21.15.81:0/3096443510 conn(0x7fd2a8072950 msgr2=0x7fd2a80b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:52.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.887+0000 7fd2acbe4640 1 -- 172.21.15.81:0/3096443510 shutdown_connections 2024-06-29T14:23:52.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.887+0000 7fd2acbe4640 1 -- 172.21.15.81:0/3096443510 wait complete. 2024-06-29T14:23:52.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.889+0000 7fd2acbe4640 1 Processor -- start 2024-06-29T14:23:52.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.889+0000 7fd2acbe4640 1 -- start start 2024-06-29T14:23:52.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2acbe4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2a806f680 0x7fd2a814e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:52.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2acbe4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a806fa90 0x7fd2a81f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:52.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2acbe4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2a8074520 0x7fd2a8143c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:52.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2acbe4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a806cd70 con 0x7fd2a806fa90 2024-06-29T14:23:52.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2acbe4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a815b220 con 0x7fd2a806f680 2024-06-29T14:23:52.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2acbe4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd2a806cef0 con 0x7fd2a8074520 2024-06-29T14:23:52.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2a5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a806fa90 0x7fd2a81f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:52.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2a5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a806fa90 0x7fd2a81f1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35960/0 (socket says 172.21.15.81:35960) 2024-06-29T14:23:52.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2a659c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2a806f680 0x7fd2a814e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:52.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2a5d9b640 1 -- 172.21.15.81:0/2205978673 learned_addr learned my addr 172.21.15.81:0/2205978673 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:52.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2a6d9d640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2a8074520 0x7fd2a8143c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:52.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2a5d9b640 1 -- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2a8074520 msgr2=0x7fd2a8143c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:52.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2a5d9b640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2a8074520 0x7fd2a8143c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2a5d9b640 1 -- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2a806f680 msgr2=0x7fd2a814e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:52.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2a5d9b640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2a806f680 0x7fd2a814e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:52.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.890+0000 7fd2a5d9b640 1 -- 172.21.15.81:0/2205978673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a814fc50 con 0x7fd2a806fa90 2024-06-29T14:23:52.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.891+0000 7fd2a5d9b640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a806fa90 0x7fd2a81f1760 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd29000d720 tx=0x7fd29000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:52.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.891+0000 7fd29effd640 1 -- 172.21.15.81:0/2205978673 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290017070 con 0x7fd2a806fa90 2024-06-29T14:23:52.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.891+0000 7fd29effd640 1 -- 172.21.15.81:0/2205978673 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd290009ab0 con 0x7fd2a806fa90 2024-06-29T14:23:52.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.891+0000 7fd29effd640 1 -- 172.21.15.81:0/2205978673 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290010460 con 0x7fd2a806fa90 2024-06-29T14:23:52.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.891+0000 7fd2acbe4640 1 -- 172.21.15.81:0/2205978673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a814ff30 con 0x7fd2a806fa90 2024-06-29T14:23:52.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.891+0000 7fd2acbe4640 1 -- 172.21.15.81:0/2205978673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a81504f0 con 0x7fd2a806fa90 2024-06-29T14:23:52.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.892+0000 7fd2acbe4640 1 -- 172.21.15.81:0/2205978673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a806f140 con 0x7fd2a806fa90 2024-06-29T14:23:52.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.893+0000 7fd29effd640 1 -- 172.21.15.81:0/2205978673 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 9) ==== 98321+0+0 (secure 0 0 0) 0x7fd29001a070 con 0x7fd2a806fa90 2024-06-29T14:23:52.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.893+0000 7fd29effd640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd288076470 0x7fd288078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:52.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.893+0000 7fd29effd640 1 -- 172.21.15.81:0/2205978673 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7fd290097ae0 con 0x7fd2a806fa90 2024-06-29T14:23:52.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.893+0000 7fd2a659c640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd288076470 0x7fd288078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:52.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.894+0000 7fd2a659c640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd288076470 0x7fd288078980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd298005fd0 tx=0x7fd2980095e0 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:52.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:52.897+0000 7fd29effd640 1 -- 172.21.15.81:0/2205978673 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd290060a40 con 0x7fd2a806fa90 2024-06-29T14:23:53.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.106+0000 7fd2acbe4640 1 -- 172.21.15.81:0/2205978673 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd2a8157440 con 0x7fd288076470 2024-06-29T14:23:53.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.176+0000 7fd29effd640 1 -- 172.21.15.81:0/2205978673 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fd2a8157440 con 0x7fd288076470 2024-06-29T14:23:53.175 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/dir5 2024-06-29T14:23:53.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 -- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd288076470 msgr2=0x7fd288078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd288076470 0x7fd288078980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd298005fd0 tx=0x7fd2980095e0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 -- 172.21.15.81:0/2205978673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a806fa90 msgr2=0x7fd2a81f1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a806fa90 0x7fd2a81f1760 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd29000d720 tx=0x7fd29000dbe0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 -- 172.21.15.81:0/2205978673 shutdown_connections 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd288076470 0x7fd288078980 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2a8074520 0x7fd2a8143c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a806fa90 0x7fd2a81f1760 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 --2- 172.21.15.81:0/2205978673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2a806f680 0x7fd2a814e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 -- 172.21.15.81:0/2205978673 >> 172.21.15.81:0/2205978673 conn(0x7fd2a8072950 msgr2=0x7fd2a8157d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 -- 172.21.15.81:0/2205978673 shutdown_connections 2024-06-29T14:23:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.178+0000 7fd2acbe4640 1 -- 172.21.15.81:0/2205978673 wait complete. 2024-06-29T14:23:53.187 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2024-06-29T14:23:53.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:53.188 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2024-06-29T14:23:53.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2024-06-29T14:23:53.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.364+0000 7f9252fe0640 1 -- 172.21.15.81:0/3312180667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f924c0cea10 msgr2=0x7f924c0b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:53.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.364+0000 7f9252fe0640 1 --2- 172.21.15.81:0/3312180667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f924c0cea10 0x7f924c0b47b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9240005fd0 tx=0x7f924002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.364+0000 7f9252fe0640 1 -- 172.21.15.81:0/3312180667 shutdown_connections 2024-06-29T14:23:53.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.364+0000 7f9252fe0640 1 --2- 172.21.15.81:0/3312180667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f924c0cea10 0x7f924c0b47b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.364+0000 7f9252fe0640 1 --2- 172.21.15.81:0/3312180667 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f924c0b6d60 0x7f924c0ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.364+0000 7f9252fe0640 1 -- 172.21.15.81:0/3312180667 >> 172.21.15.81:0/3312180667 conn(0x7f924c0d1960 msgr2=0x7f924c0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:53.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.364+0000 7f9252fe0640 1 -- 172.21.15.81:0/3312180667 shutdown_connections 2024-06-29T14:23:53.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.364+0000 7f9252fe0640 1 -- 172.21.15.81:0/3312180667 wait complete. 2024-06-29T14:23:53.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.366+0000 7f9252fe0640 1 Processor -- start 2024-06-29T14:23:53.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.366+0000 7f9252fe0640 1 -- start start 2024-06-29T14:23:53.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f9252fe0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f924c0b4cf0 0x7f924c15b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:53.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f9252fe0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f924c0b6d60 0x7f924c0c9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:53.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f9252fe0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f924c0cea10 0x7f924c1620b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:53.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f9252fe0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f924c0cd540 con 0x7f924c0b4cf0 2024-06-29T14:23:53.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f9252fe0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f924c0d0a10 con 0x7f924c0cea10 2024-06-29T14:23:53.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f9252fe0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f924c0cd6c0 con 0x7f924c0b6d60 2024-06-29T14:23:53.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f9250d7c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f924c0b4cf0 0x7f924c15b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f9250d7c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f924c0b4cf0 0x7f924c15b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35978/0 (socket says 172.21.15.81:35978) 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f9250d7c640 1 -- 172.21.15.81:0/3092190981 learned_addr learned my addr 172.21.15.81:0/3092190981 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f924bfff640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f924c0b6d60 0x7f924c0c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.367+0000 7f925157d640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f924c0cea10 0x7f924c1620b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f9250d7c640 1 -- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f924c0b6d60 msgr2=0x7f924c0c9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f9250d7c640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f924c0b6d60 0x7f924c0c9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f9250d7c640 1 -- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f924c0cea10 msgr2=0x7f924c1620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f9250d7c640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f924c0cea10 0x7f924c1620b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f9250d7c640 1 -- 172.21.15.81:0/3092190981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c0af840 con 0x7f924c0b4cf0 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f9250d7c640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f924c0b4cf0 0x7f924c15b790 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f923c009fd0 tx=0x7f923c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f92497fa640 1 -- 172.21.15.81:0/3092190981 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c00ccc0 con 0x7f924c0b4cf0 2024-06-29T14:23:53.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f92497fa640 1 -- 172.21.15.81:0/3092190981 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f923c00ee70 con 0x7f924c0b4cf0 2024-06-29T14:23:53.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f9252fe0640 1 -- 172.21.15.81:0/3092190981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924c0afd40 con 0x7f924c0b4cf0 2024-06-29T14:23:53.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f92497fa640 1 -- 172.21.15.81:0/3092190981 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c00ce60 con 0x7f924c0b4cf0 2024-06-29T14:23:53.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.368+0000 7f9252fe0640 1 -- 172.21.15.81:0/3092190981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924c0b0300 con 0x7f924c0b4cf0 2024-06-29T14:23:53.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.369+0000 7f9252fe0640 1 -- 172.21.15.81:0/3092190981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924c0ce4d0 con 0x7f924c0b4cf0 2024-06-29T14:23:53.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.370+0000 7f92497fa640 1 -- 172.21.15.81:0/3092190981 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 9) ==== 98321+0+0 (secure 0 0 0) 0x7f923c004280 con 0x7f924c0b4cf0 2024-06-29T14:23:53.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.370+0000 7f92497fa640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9238076470 0x7f9238078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:53.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.370+0000 7f92497fa640 1 -- 172.21.15.81:0/3092190981 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f923c098280 con 0x7f924c0b4cf0 2024-06-29T14:23:53.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.370+0000 7f924bfff640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9238076470 0x7f9238078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:53.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.371+0000 7f924bfff640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9238076470 0x7f9238078980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9240033000 tx=0x7f9240004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:53.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.375+0000 7f92497fa640 1 -- 172.21.15.81:0/3092190981 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f923c061270 con 0x7f924c0b4cf0 2024-06-29T14:23:53.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.574+0000 7f9252fe0640 1 -- 172.21.15.81:0/3092190981 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f924c0cdec0 con 0x7f9238076470 2024-06-29T14:23:53.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.663+0000 7f92497fa640 1 -- 172.21.15.81:0/3092190981 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f924c0cdec0 con 0x7f9238076470 2024-06-29T14:23:53.661 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/dir6 2024-06-29T14:23:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 -- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9238076470 msgr2=0x7f9238078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9238076470 0x7f9238078980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9240033000 tx=0x7f9240004730 comp rx=0 tx=0).stop 2024-06-29T14:23:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 -- 172.21.15.81:0/3092190981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f924c0b4cf0 msgr2=0x7f924c15b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f924c0b4cf0 0x7f924c15b790 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f923c009fd0 tx=0x7f923c00ec20 comp rx=0 tx=0).stop 2024-06-29T14:23:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 -- 172.21.15.81:0/3092190981 shutdown_connections 2024-06-29T14:23:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9238076470 0x7f9238078980 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f924c0cea10 0x7f924c1620b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f924c0b6d60 0x7f924c0c9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 --2- 172.21.15.81:0/3092190981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f924c0b4cf0 0x7f924c15b790 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 -- 172.21.15.81:0/3092190981 >> 172.21.15.81:0/3092190981 conn(0x7f924c0d1960 msgr2=0x7f924c0b5e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 -- 172.21.15.81:0/3092190981 shutdown_connections 2024-06-29T14:23:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.665+0000 7f9252fe0640 1 -- 172.21.15.81:0/3092190981 wait complete. 2024-06-29T14:23:53.676 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2024-06-29T14:23:53.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:53.676 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2024-06-29T14:23:53.693 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2024-06-29T14:23:53.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.851+0000 7fd02743f640 1 -- 172.21.15.81:0/2485950900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd02015c440 msgr2=0x7fd02006c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:53.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.851+0000 7fd02743f640 1 --2- 172.21.15.81:0/2485950900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd02015c440 0x7fd02006c830 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd0200cd380 tx=0x7fd01802a9e0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.851+0000 7fd02743f640 1 -- 172.21.15.81:0/2485950900 shutdown_connections 2024-06-29T14:23:53.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.851+0000 7fd02743f640 1 --2- 172.21.15.81:0/2485950900 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd02015c030 0x7fd02006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.851+0000 7fd02743f640 1 --2- 172.21.15.81:0/2485950900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd02015c440 0x7fd02006c830 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.851+0000 7fd02743f640 1 -- 172.21.15.81:0/2485950900 >> 172.21.15.81:0/2485950900 conn(0x7fd0200d0930 msgr2=0x7fd0200d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:53.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.851+0000 7fd02743f640 1 -- 172.21.15.81:0/2485950900 shutdown_connections 2024-06-29T14:23:53.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.851+0000 7fd02743f640 1 -- 172.21.15.81:0/2485950900 wait complete. 2024-06-29T14:23:53.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.853+0000 7fd02743f640 1 Processor -- start 2024-06-29T14:23:53.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.853+0000 7fd02743f640 1 -- start start 2024-06-29T14:23:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.854+0000 7fd02743f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd02015c030 0x7fd020159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.854+0000 7fd02743f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd02015c440 0x7fd020157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.854+0000 7fd02743f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd02015d750 0x7fd0201473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.854+0000 7fd02743f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0200cfda0 con 0x7fd02015c440 2024-06-29T14:23:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.854+0000 7fd02743f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd0200cfc20 con 0x7fd02015d750 2024-06-29T14:23:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.854+0000 7fd02743f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd0200cff20 con 0x7fd02015c030 2024-06-29T14:23:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.854+0000 7fd0249da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd02015c440 0x7fd020157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.854+0000 7fd0249da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd02015c440 0x7fd020157cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36002/0 (socket says 172.21.15.81:36002) 2024-06-29T14:23:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.854+0000 7fd0249da640 1 -- 172.21.15.81:0/953212772 learned_addr learned my addr 172.21.15.81:0/953212772 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.854+0000 7fd0251db640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd02015c030 0x7fd020159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd0259dc640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd02015d750 0x7fd0201473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd0249da640 1 -- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd02015c030 msgr2=0x7fd020159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd0249da640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd02015c030 0x7fd020159690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd0249da640 1 -- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd02015d750 msgr2=0x7fd0201473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd0249da640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd02015d750 0x7fd0201473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd0249da640 1 -- 172.21.15.81:0/953212772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0201478e0 con 0x7fd02015c440 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd0251db640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd02015c030 0x7fd020159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd0259dc640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd02015d750 0x7fd0201473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd0249da640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd02015c440 0x7fd020157cf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd01000d770 tx=0x7fd01000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd015ffb640 1 -- 172.21.15.81:0/953212772 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd010017070 con 0x7fd02015c440 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd015ffb640 1 -- 172.21.15.81:0/953212772 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd010009d20 con 0x7fd02015c440 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd015ffb640 1 -- 172.21.15.81:0/953212772 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0100106c0 con 0x7fd02015c440 2024-06-29T14:23:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd02743f640 1 -- 172.21.15.81:0/953212772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0201585f0 con 0x7fd02015c440 2024-06-29T14:23:53.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.855+0000 7fd02743f640 1 -- 172.21.15.81:0/953212772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd020158bb0 con 0x7fd02015c440 2024-06-29T14:23:53.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.856+0000 7fd02743f640 1 -- 172.21.15.81:0/953212772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd020158360 con 0x7fd02015c440 2024-06-29T14:23:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.857+0000 7fd015ffb640 1 -- 172.21.15.81:0/953212772 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7fd01001a070 con 0x7fd02015c440 2024-06-29T14:23:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.857+0000 7fd015ffb640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd0040764d0 0x7fd0040789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.857+0000 7fd015ffb640 1 -- 172.21.15.81:0/953212772 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7fd010097e40 con 0x7fd02015c440 2024-06-29T14:23:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.858+0000 7fd0251db640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd0040764d0 0x7fd0040789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.858+0000 7fd0251db640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd0040764d0 0x7fd0040789e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd01c006fd0 tx=0x7fd01c008040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:53.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:53.862+0000 7fd015ffb640 1 -- 172.21.15.81:0/953212772 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd010061280 con 0x7fd02015c440 2024-06-29T14:23:54.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.066+0000 7fd02743f640 1 -- 172.21.15.81:0/953212772 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd02006e2a0 con 0x7fd0040764d0 2024-06-29T14:23:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.135+0000 7fd015ffb640 1 -- 172.21.15.81:0/953212772 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fd02006e2a0 con 0x7fd0040764d0 2024-06-29T14:23:54.134 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/dir7 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 -- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd0040764d0 msgr2=0x7fd0040789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd0040764d0 0x7fd0040789e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd01c006fd0 tx=0x7fd01c008040 comp rx=0 tx=0).stop 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 -- 172.21.15.81:0/953212772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd02015c440 msgr2=0x7fd020157cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd02015c440 0x7fd020157cf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd01000d770 tx=0x7fd01000dc30 comp rx=0 tx=0).stop 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 -- 172.21.15.81:0/953212772 shutdown_connections 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd0040764d0 0x7fd0040789e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd02015d750 0x7fd0201473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd02015c440 0x7fd020157cf0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 --2- 172.21.15.81:0/953212772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd02015c030 0x7fd020159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 -- 172.21.15.81:0/953212772 >> 172.21.15.81:0/953212772 conn(0x7fd0200d0930 msgr2=0x7fd02015b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.137+0000 7fd02743f640 1 -- 172.21.15.81:0/953212772 shutdown_connections 2024-06-29T14:23:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.138+0000 7fd02743f640 1 -- 172.21.15.81:0/953212772 wait complete. 2024-06-29T14:23:54.146 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2024-06-29T14:23:54.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:54.147 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2024-06-29T14:23:54.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2024-06-29T14:23:54.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.321+0000 7f52bdd63640 1 -- 172.21.15.81:0/823870054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52b8074520 msgr2=0x7f52b806f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:54.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.321+0000 7f52bdd63640 1 --2- 172.21.15.81:0/823870054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52b8074520 0x7f52b806f140 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f52b806e210 tx=0x7f52a802a560 comp rx=0 tx=0).stop 2024-06-29T14:23:54.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.321+0000 7f52bdd63640 1 -- 172.21.15.81:0/823870054 shutdown_connections 2024-06-29T14:23:54.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.321+0000 7f52bdd63640 1 --2- 172.21.15.81:0/823870054 >> v2:172.21.15.167:3300/0 conn(0x7f52b806f680 0x7f52b806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.321+0000 7f52bdd63640 1 --2- 172.21.15.81:0/823870054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52b8074520 0x7f52b806f140 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.321+0000 7f52bdd63640 1 -- 172.21.15.81:0/823870054 >> 172.21.15.81:0/823870054 conn(0x7f52b8072950 msgr2=0x7f52b80b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:54.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.322+0000 7f52bdd63640 1 -- 172.21.15.81:0/823870054 shutdown_connections 2024-06-29T14:23:54.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.322+0000 7f52bdd63640 1 -- 172.21.15.81:0/823870054 wait complete. 2024-06-29T14:23:54.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.323+0000 7f52bdd63640 1 Processor -- start 2024-06-29T14:23:54.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.323+0000 7f52bdd63640 1 -- start start 2024-06-29T14:23:54.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52bdd63640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52b806f680 0x7f52b8152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:54.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52bdd63640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52b806fa90 0x7f52b81e5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:54.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52bdd63640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52b8074520 0x7f52b81e61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:54.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52bdd63640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f52b815b010 con 0x7f52b806f680 2024-06-29T14:23:54.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52bdd63640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f52b806d150 con 0x7f52b806fa90 2024-06-29T14:23:54.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52bdd63640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f52b815b190 con 0x7f52b8074520 2024-06-29T14:23:54.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52b77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52b806f680 0x7f52b8152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:54.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52b77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52b806f680 0x7f52b8152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36026/0 (socket says 172.21.15.81:36026) 2024-06-29T14:23:54.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52b77fe640 1 -- 172.21.15.81:0/3856354647 learned_addr learned my addr 172.21.15.81:0/3856354647 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:54.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52b77fe640 1 -- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52b8074520 msgr2=0x7f52b81e61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52b77fe640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52b8074520 0x7f52b81e61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52b6ffd640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52b806fa90 0x7f52b81e5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52b77fe640 1 -- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52b806fa90 msgr2=0x7f52b81e5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52b77fe640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52b806fa90 0x7f52b81e5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52b77fe640 1 -- 172.21.15.81:0/3856354647 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b81e5790 con 0x7f52b806f680 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52b77fe640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52b806f680 0x7f52b8152f10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f52a8034040 tx=0x7f52a80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f529ffff640 1 -- 172.21.15.81:0/3856354647 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a803e030 con 0x7f52b806f680 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f529ffff640 1 -- 172.21.15.81:0/3856354647 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52a80047a0 con 0x7f52b806f680 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f52bdd63640 1 -- 172.21.15.81:0/3856354647 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b8153bd0 con 0x7f52b806f680 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.325+0000 7f529ffff640 1 -- 172.21.15.81:0/3856354647 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a8035040 con 0x7f52b806f680 2024-06-29T14:23:54.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.326+0000 7f52bdd63640 1 -- 172.21.15.81:0/3856354647 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b8154190 con 0x7f52b806f680 2024-06-29T14:23:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.326+0000 7f52bdd63640 1 -- 172.21.15.81:0/3856354647 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b806dac0 con 0x7f52b806f680 2024-06-29T14:23:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.327+0000 7f529ffff640 1 -- 172.21.15.81:0/3856354647 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f52a8041030 con 0x7f52b806f680 2024-06-29T14:23:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.328+0000 7f529ffff640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f52a4076500 0x7f52a4078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.328+0000 7f529ffff640 1 -- 172.21.15.81:0/3856354647 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f52a80b84f0 con 0x7f52b806f680 2024-06-29T14:23:54.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.328+0000 7f52b6ffd640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f52a4076500 0x7f52a4078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:54.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.328+0000 7f52b6ffd640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f52a4076500 0x7f52a4078a10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f52b81e72b0 tx=0x7f52ac008040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.332+0000 7f529ffff640 1 -- 172.21.15.81:0/3856354647 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f52b806dac0 con 0x7f52b806f680 2024-06-29T14:23:54.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.532+0000 7f52bdd63640 1 -- 172.21.15.81:0/3856354647 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f52b81543d0 con 0x7f52a4076500 2024-06-29T14:23:54.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.620+0000 7f529ffff640 1 -- 172.21.15.81:0/3856354647 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f52ac00a380 con 0x7f52a4076500 2024-06-29T14:23:54.618 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/dir8 2024-06-29T14:23:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 -- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f52a4076500 msgr2=0x7f52a4078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f52a4076500 0x7f52a4078a10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f52b81e72b0 tx=0x7f52ac008040 comp rx=0 tx=0).stop 2024-06-29T14:23:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 -- 172.21.15.81:0/3856354647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52b806f680 msgr2=0x7f52b8152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52b806f680 0x7f52b8152f10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f52a8034040 tx=0x7f52a80039e0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 -- 172.21.15.81:0/3856354647 shutdown_connections 2024-06-29T14:23:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f52a4076500 0x7f52a4078a10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52b8074520 0x7f52b81e61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52b806fa90 0x7f52b81e5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 --2- 172.21.15.81:0/3856354647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52b806f680 0x7f52b8152f10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 -- 172.21.15.81:0/3856354647 >> 172.21.15.81:0/3856354647 conn(0x7f52b8072950 msgr2=0x7f52b8158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 -- 172.21.15.81:0/3856354647 shutdown_connections 2024-06-29T14:23:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.622+0000 7f52bdd63640 1 -- 172.21.15.81:0/3856354647 wait complete. 2024-06-29T14:23:54.631 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2024-06-29T14:23:54.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:23:54.631 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2024-06-29T14:23:54.648 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2024-06-29T14:23:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.806+0000 7fd0698cc640 1 -- 172.21.15.81:0/3585753346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0640d0b90 msgr2=0x7fd0640d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.806+0000 7fd0698cc640 1 --2- 172.21.15.81:0/3585753346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0640d0b90 0x7fd0640d1a80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd04c005fd0 tx=0x7fd04c02a670 comp rx=0 tx=0).stop 2024-06-29T14:23:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.806+0000 7fd0698cc640 1 -- 172.21.15.81:0/3585753346 shutdown_connections 2024-06-29T14:23:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.806+0000 7fd0698cc640 1 --2- 172.21.15.81:0/3585753346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0640d0b90 0x7fd0640d1a80 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.806+0000 7fd0698cc640 1 -- 172.21.15.81:0/3585753346 >> 172.21.15.81:0/3585753346 conn(0x7fd06409cf30 msgr2=0x7fd0640d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.806+0000 7fd0698cc640 1 -- 172.21.15.81:0/3585753346 shutdown_connections 2024-06-29T14:23:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.806+0000 7fd0698cc640 1 -- 172.21.15.81:0/3585753346 wait complete. 2024-06-29T14:23:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.808+0000 7fd0698cc640 1 Processor -- start 2024-06-29T14:23:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.808+0000 7fd0698cc640 1 -- start start 2024-06-29T14:23:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.809+0000 7fd0698cc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0640b5250 0x7fd0640b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0698cc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0640b9040 0x7fd0641f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0698cc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0640c7550 0x7fd0640a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0698cc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0640d26d0 con 0x7fd0640c7550 2024-06-29T14:23:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0698cc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd0640b85c0 con 0x7fd0640b9040 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0698cc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd0640d2850 con 0x7fd0640b5250 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0637fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0640c7550 0x7fd0640a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0637fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0640c7550 0x7fd0640a0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36046/0 (socket says 172.21.15.81:36046) 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0637fe640 1 -- 172.21.15.81:0/637180387 learned_addr learned my addr 172.21.15.81:0/637180387 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd05a7fc640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0640b9040 0x7fd0641f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd062ffd640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0640b5250 0x7fd0640b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0637fe640 1 -- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0640b5250 msgr2=0x7fd0640b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0637fe640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0640b5250 0x7fd0640b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0637fe640 1 -- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0640b9040 msgr2=0x7fd0641f7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0637fe640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0640b9040 0x7fd0641f7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0637fe640 1 -- 172.21.15.81:0/637180387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0640b2030 con 0x7fd0640c7550 2024-06-29T14:23:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd062ffd640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0640b5250 0x7fd0640b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0637fe640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0640c7550 0x7fd0640a0a00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00be90 tx=0x7fd05c00bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd060ff9640 1 -- 172.21.15.81:0/637180387 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c019070 con 0x7fd0640c7550 2024-06-29T14:23:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd060ff9640 1 -- 172.21.15.81:0/637180387 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd05c014440 con 0x7fd0640c7550 2024-06-29T14:23:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0698cc640 1 -- 172.21.15.81:0/637180387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0640b2310 con 0x7fd0640c7550 2024-06-29T14:23:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd060ff9640 1 -- 172.21.15.81:0/637180387 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c0148d0 con 0x7fd0640c7550 2024-06-29T14:23:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.810+0000 7fd0698cc640 1 -- 172.21.15.81:0/637180387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0640b28d0 con 0x7fd0640c7550 2024-06-29T14:23:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.811+0000 7fd0698cc640 1 -- 172.21.15.81:0/637180387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0640d2de0 con 0x7fd0640c7550 2024-06-29T14:23:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.812+0000 7fd060ff9640 1 -- 172.21.15.81:0/637180387 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7fd05c01c070 con 0x7fd0640c7550 2024-06-29T14:23:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.812+0000 7fd060ff9640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd044076500 0x7fd044078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.813+0000 7fd060ff9640 1 -- 172.21.15.81:0/637180387 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7fd05c097c90 con 0x7fd0640c7550 2024-06-29T14:23:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.813+0000 7fd062ffd640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd044076500 0x7fd044078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.813+0000 7fd062ffd640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd044076500 0x7fd044078a10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd054000f60 tx=0x7fd054009040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:54.817+0000 7fd060ff9640 1 -- 172.21.15.81:0/637180387 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd05c060b90 con 0x7fd0640c7550 2024-06-29T14:23:55.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.021+0000 7fd0698cc640 1 -- 172.21.15.81:0/637180387 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd0640c7100 con 0x7fd044076500 2024-06-29T14:23:55.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.104+0000 7fd060ff9640 1 -- 172.21.15.81:0/637180387 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fd0640c7100 con 0x7fd044076500 2024-06-29T14:23:55.102 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/dir9 2024-06-29T14:23:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 -- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd044076500 msgr2=0x7fd044078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd044076500 0x7fd044078a10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd054000f60 tx=0x7fd054009040 comp rx=0 tx=0).stop 2024-06-29T14:23:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 -- 172.21.15.81:0/637180387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0640c7550 msgr2=0x7fd0640a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0640c7550 0x7fd0640a0a00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00be90 tx=0x7fd05c00bf80 comp rx=0 tx=0).stop 2024-06-29T14:23:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 -- 172.21.15.81:0/637180387 shutdown_connections 2024-06-29T14:23:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd044076500 0x7fd044078a10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0640c7550 0x7fd0640a0a00 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0640b9040 0x7fd0641f7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 --2- 172.21.15.81:0/637180387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0640b5250 0x7fd0640b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 -- 172.21.15.81:0/637180387 >> 172.21.15.81:0/637180387 conn(0x7fd06409cf30 msgr2=0x7fd06406e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.106+0000 7fd0698cc640 1 -- 172.21.15.81:0/637180387 shutdown_connections 2024-06-29T14:23:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.107+0000 7fd0698cc640 1 -- 172.21.15.81:0/637180387 wait complete. 2024-06-29T14:23:55.116 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2024-06-29T14:23:55.116 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2024-06-29T14:23:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.234+0000 7f4f2f18c640 1 -- 172.21.15.81:0/3690263821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f280ceb80 msgr2=0x7f4f280b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.234+0000 7f4f2f18c640 1 --2- 172.21.15.81:0/3690263821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f280ceb80 0x7f4f280b3b00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4f2808c2f0 tx=0x7f4f1c02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.234+0000 7f4f2f18c640 1 -- 172.21.15.81:0/3690263821 shutdown_connections 2024-06-29T14:23:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.234+0000 7f4f2f18c640 1 --2- 172.21.15.81:0/3690263821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f280ceb80 0x7f4f280b3b00 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.234+0000 7f4f2f18c640 1 --2- 172.21.15.81:0/3690263821 >> v2:172.21.15.167:3300/0 conn(0x7f4f280ce770 0x7f4f280b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.234+0000 7f4f2f18c640 1 -- 172.21.15.81:0/3690263821 >> 172.21.15.81:0/3690263821 conn(0x7f4f280d3c60 msgr2=0x7f4f280d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.234+0000 7f4f2f18c640 1 -- 172.21.15.81:0/3690263821 shutdown_connections 2024-06-29T14:23:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.234+0000 7f4f2f18c640 1 -- 172.21.15.81:0/3690263821 wait complete. 2024-06-29T14:23:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.236+0000 7f4f2f18c640 1 Processor -- start 2024-06-29T14:23:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.236+0000 7f4f2f18c640 1 -- start start 2024-06-29T14:23:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f2f18c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f280ce770 0x7f4f281dae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f2f18c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f280ceb80 0x7f4f2809dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f2f18c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f280d2240 0x7f4f2809e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f2f18c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4f280d15a0 con 0x7f4f280ceb80 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f2f18c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4f280d1420 con 0x7f4f280ce770 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f2f18c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4f280d4990 con 0x7f4f280d2240 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f27fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f280ceb80 0x7f4f2809dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f27fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f280ceb80 0x7f4f2809dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36056/0 (socket says 172.21.15.81:36056) 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f27fff640 1 -- 172.21.15.81:0/2972365541 learned_addr learned my addr 172.21.15.81:0/2972365541 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f2cf28640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f280ce770 0x7f4f281dae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f2d729640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f280d2240 0x7f4f2809e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f27fff640 1 -- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f280d2240 msgr2=0x7f4f2809e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f27fff640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f280d2240 0x7f4f2809e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f27fff640 1 -- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f280ce770 msgr2=0x7f4f281dae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f27fff640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f280ce770 0x7f4f281dae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f27fff640 1 -- 172.21.15.81:0/2972365541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f281db350 con 0x7f4f280ceb80 2024-06-29T14:23:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.237+0000 7f4f2d729640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f280d2240 0x7f4f2809e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.238+0000 7f4f27fff640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f280ceb80 0x7f4f2809dbb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c02aa90 tx=0x7f4f1c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.238+0000 7f4f257fa640 1 -- 172.21.15.81:0/2972365541 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c031d60 con 0x7f4f280ceb80 2024-06-29T14:23:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.238+0000 7f4f257fa640 1 -- 172.21.15.81:0/2972365541 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f1c033e10 con 0x7f4f280ceb80 2024-06-29T14:23:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.238+0000 7f4f2f18c640 1 -- 172.21.15.81:0/2972365541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f280ba5f0 con 0x7f4f280ceb80 2024-06-29T14:23:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.238+0000 7f4f257fa640 1 -- 172.21.15.81:0/2972365541 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c005260 con 0x7f4f280ceb80 2024-06-29T14:23:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.238+0000 7f4f2f18c640 1 -- 172.21.15.81:0/2972365541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f280a14e0 con 0x7f4f280ceb80 2024-06-29T14:23:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.239+0000 7f4f2f18c640 1 -- 172.21.15.81:0/2972365541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f280d2650 con 0x7f4f280ceb80 2024-06-29T14:23:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.240+0000 7f4f257fa640 1 -- 172.21.15.81:0/2972365541 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f4f1c005400 con 0x7f4f280ceb80 2024-06-29T14:23:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.240+0000 7f4f257fa640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4f14076500 0x7f4f14078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.240+0000 7f4f257fa640 1 -- 172.21.15.81:0/2972365541 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f4f1c0b8e60 con 0x7f4f280ceb80 2024-06-29T14:23:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.241+0000 7f4f2cf28640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4f14076500 0x7f4f14078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.241+0000 7f4f2cf28640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4f14076500 0x7f4f14078a10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4f280d0120 tx=0x7f4f1800b2e0 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.245+0000 7f4f257fa640 1 -- 172.21.15.81:0/2972365541 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f1c03f030 con 0x7f4f280ceb80 2024-06-29T14:23:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.448+0000 7f4f2f18c640 1 -- 172.21.15.81:0/2972365541 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4f280b24e0 con 0x7f4f14076500 2024-06-29T14:23:55.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.506+0000 7f4f257fa640 1 -- 172.21.15.81:0/2972365541 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f4f280b24e0 con 0x7f4f14076500 2024-06-29T14:23:55.504 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/dir0 2024-06-29T14:23:55.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 -- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4f14076500 msgr2=0x7f4f14078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4f14076500 0x7f4f14078a10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4f280d0120 tx=0x7f4f1800b2e0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 -- 172.21.15.81:0/2972365541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f280ceb80 msgr2=0x7f4f2809dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f280ceb80 0x7f4f2809dbb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c02aa90 tx=0x7f4f1c033c20 comp rx=0 tx=0).stop 2024-06-29T14:23:55.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 -- 172.21.15.81:0/2972365541 shutdown_connections 2024-06-29T14:23:55.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4f14076500 0x7f4f14078a10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f280d2240 0x7f4f2809e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f280ceb80 0x7f4f2809dbb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 --2- 172.21.15.81:0/2972365541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f280ce770 0x7f4f281dae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 -- 172.21.15.81:0/2972365541 >> 172.21.15.81:0/2972365541 conn(0x7f4f280d3c60 msgr2=0x7f4f280b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:55.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 -- 172.21.15.81:0/2972365541 shutdown_connections 2024-06-29T14:23:55.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.508+0000 7f4f2f18c640 1 -- 172.21.15.81:0/2972365541 wait complete. 2024-06-29T14:23:55.518 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2024-06-29T14:23:55.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2024-06-29T14:23:55.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.636+0000 7fa78da12640 1 -- 172.21.15.81:0/2192452182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa78808ccd0 msgr2=0x7fa7880b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.636+0000 7fa78da12640 1 --2- 172.21.15.81:0/2192452182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa78808ccd0 0x7fa7880b3240 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa770009830 tx=0x7fa77002a3f0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.636+0000 7fa78da12640 1 -- 172.21.15.81:0/2192452182 shutdown_connections 2024-06-29T14:23:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.636+0000 7fa78da12640 1 --2- 172.21.15.81:0/2192452182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa78808ccd0 0x7fa7880b3240 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.636+0000 7fa78da12640 1 -- 172.21.15.81:0/2192452182 >> 172.21.15.81:0/2192452182 conn(0x7fa7880b56d0 msgr2=0x7fa7880b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.636+0000 7fa78da12640 1 -- 172.21.15.81:0/2192452182 shutdown_connections 2024-06-29T14:23:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.636+0000 7fa78da12640 1 -- 172.21.15.81:0/2192452182 wait complete. 2024-06-29T14:23:55.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.637+0000 7fa78da12640 1 Processor -- start 2024-06-29T14:23:55.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.638+0000 7fa78da12640 1 -- start start 2024-06-29T14:23:55.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa78da12640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa78808ccd0 0x7fa7880cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:55.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa78da12640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7880cf040 0x7fa78814edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:55.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa78da12640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa788167ed0 0x7fa7881682e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa78da12640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa7880b4410 con 0x7fa788167ed0 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa78da12640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa7880b4290 con 0x7fa7880cf040 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa78da12640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa7880b4590 con 0x7fa78808ccd0 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa7877fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa788167ed0 0x7fa7881682e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa7877fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa788167ed0 0x7fa7881682e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36082/0 (socket says 172.21.15.81:36082) 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa7877fe640 1 -- 172.21.15.81:0/2384775572 learned_addr learned my addr 172.21.15.81:0/2384775572 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa7867fc640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7880cf040 0x7fa78814edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa786ffd640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa78808ccd0 0x7fa7880cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa7877fe640 1 -- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa78808ccd0 msgr2=0x7fa7880cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa7877fe640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa78808ccd0 0x7fa7880cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa7877fe640 1 -- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7880cf040 msgr2=0x7fa78814edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa7877fe640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7880cf040 0x7fa78814edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa7877fe640 1 -- 172.21.15.81:0/2384775572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa788165760 con 0x7fa788167ed0 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa786ffd640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa78808ccd0 0x7fa7880cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.639+0000 7fa7867fc640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7880cf040 0x7fa78814edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.640+0000 7fa7877fe640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa788167ed0 0x7fa7881682e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa78000b410 tx=0x7fa78000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.640+0000 7fa77ffff640 1 -- 172.21.15.81:0/2384775572 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa780015020 con 0x7fa788167ed0 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.640+0000 7fa77ffff640 1 -- 172.21.15.81:0/2384775572 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa780011920 con 0x7fa788167ed0 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.640+0000 7fa78da12640 1 -- 172.21.15.81:0/2384775572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa788168820 con 0x7fa788167ed0 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.640+0000 7fa77ffff640 1 -- 172.21.15.81:0/2384775572 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa780011db0 con 0x7fa788167ed0 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.640+0000 7fa78da12640 1 -- 172.21.15.81:0/2384775572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa78814fca0 con 0x7fa788167ed0 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.641+0000 7fa78da12640 1 -- 172.21.15.81:0/2384775572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa78808c270 con 0x7fa788167ed0 2024-06-29T14:23:55.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.641+0000 7fa77ffff640 1 -- 172.21.15.81:0/2384775572 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7fa780018020 con 0x7fa788167ed0 2024-06-29T14:23:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.642+0000 7fa77ffff640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa7680764d0 0x7fa7680789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.642+0000 7fa77ffff640 1 -- 172.21.15.81:0/2384775572 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7fa780097a00 con 0x7fa788167ed0 2024-06-29T14:23:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.642+0000 7fa786ffd640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa7680764d0 0x7fa7680789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:55.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.643+0000 7fa786ffd640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa7680764d0 0x7fa7680789e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa778007fd0 tx=0x7fa778009040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:55.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.646+0000 7fa77ffff640 1 -- 172.21.15.81:0/2384775572 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa78001f030 con 0x7fa788167ed0 2024-06-29T14:23:55.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.850+0000 7fa78da12640 1 -- 172.21.15.81:0/2384775572 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fa7880cea90 con 0x7fa7680764d0 2024-06-29T14:23:55.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.906+0000 7fa77ffff640 1 -- 172.21.15.81:0/2384775572 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa7880cea90 con 0x7fa7680764d0 2024-06-29T14:23:55.904 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/dir1 2024-06-29T14:23:55.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 -- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa7680764d0 msgr2=0x7fa7680789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa7680764d0 0x7fa7680789e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa778007fd0 tx=0x7fa778009040 comp rx=0 tx=0).stop 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 -- 172.21.15.81:0/2384775572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa788167ed0 msgr2=0x7fa7881682e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa788167ed0 0x7fa7881682e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa78000b410 tx=0x7fa78000b8d0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 -- 172.21.15.81:0/2384775572 shutdown_connections 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa7680764d0 0x7fa7680789e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa788167ed0 0x7fa7881682e0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7880cf040 0x7fa78814edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 --2- 172.21.15.81:0/2384775572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa78808ccd0 0x7fa7880cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 -- 172.21.15.81:0/2384775572 >> 172.21.15.81:0/2384775572 conn(0x7fa7880b56d0 msgr2=0x7fa7880b8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 -- 172.21.15.81:0/2384775572 shutdown_connections 2024-06-29T14:23:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:55.908+0000 7fa78da12640 1 -- 172.21.15.81:0/2384775572 wait complete. 2024-06-29T14:23:55.917 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2024-06-29T14:23:55.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2024-06-29T14:23:56.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.035+0000 7f1a0197a640 1 -- 172.21.15.81:0/493633849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19fc06ca00 msgr2=0x7f19fc15fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.035+0000 7f1a0197a640 1 --2- 172.21.15.81:0/493633849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19fc06ca00 0x7f19fc15fcb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f19e4005fd0 tx=0x7f19e402a530 comp rx=0 tx=0).stop 2024-06-29T14:23:56.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.035+0000 7f1a0197a640 1 -- 172.21.15.81:0/493633849 shutdown_connections 2024-06-29T14:23:56.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.035+0000 7f1a0197a640 1 --2- 172.21.15.81:0/493633849 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19fc1601f0 0x7f19fc0cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.035+0000 7f1a0197a640 1 --2- 172.21.15.81:0/493633849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19fc06ca00 0x7f19fc15fcb0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.035+0000 7f1a0197a640 1 -- 172.21.15.81:0/493633849 >> 172.21.15.81:0/493633849 conn(0x7f19fc0d0b70 msgr2=0x7f19fc0d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:56.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.035+0000 7f1a0197a640 1 -- 172.21.15.81:0/493633849 shutdown_connections 2024-06-29T14:23:56.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.035+0000 7f1a0197a640 1 -- 172.21.15.81:0/493633849 wait complete. 2024-06-29T14:23:56.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.037+0000 7f1a0197a640 1 Processor -- start 2024-06-29T14:23:56.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.037+0000 7f1a0197a640 1 -- start start 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.038+0000 7f1a0197a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19fc13ec50 0x7f19fc0af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.038+0000 7f1a0197a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19fc1601f0 0x7f19fc13d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.038+0000 7f1a0197a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19fc1f3740 0x7f19fc0701b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.038+0000 7f1a0197a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc0cfc00 con 0x7f19fc1601f0 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.038+0000 7f1a0197a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f19fc0cfa80 con 0x7f19fc1f3740 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.038+0000 7f1a0197a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f19fc0cfd80 con 0x7f19fc13ec50 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fa7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19fc1601f0 0x7f19fc13d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fa7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19fc1601f0 0x7f19fc13d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36100/0 (socket says 172.21.15.81:36100) 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fa7fc640 1 -- 172.21.15.81:0/2060148537 learned_addr learned my addr 172.21.15.81:0/2060148537 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19faffd640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19fc13ec50 0x7f19fc0af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fb7fe640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19fc1f3740 0x7f19fc0701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fa7fc640 1 -- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19fc13ec50 msgr2=0x7f19fc0af7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fa7fc640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19fc13ec50 0x7f19fc0af7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fa7fc640 1 -- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19fc1f3740 msgr2=0x7f19fc0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fa7fc640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19fc1f3740 0x7f19fc0701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fa7fc640 1 -- 172.21.15.81:0/2060148537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19fc0b02d0 con 0x7f19fc1601f0 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fb7fe640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19fc1f3740 0x7f19fc0701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19fa7fc640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19fc1601f0 0x7f19fc13d660 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f19e4034040 tx=0x7f19e4003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19f3fff640 1 -- 172.21.15.81:0/2060148537 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e403e030 con 0x7f19fc1601f0 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19f3fff640 1 -- 172.21.15.81:0/2060148537 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19e40047a0 con 0x7f19fc1601f0 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f19f3fff640 1 -- 172.21.15.81:0/2060148537 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e4035040 con 0x7f19fc1601f0 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f1a0197a640 1 -- 172.21.15.81:0/2060148537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19fc0afff0 con 0x7f19fc1601f0 2024-06-29T14:23:56.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.039+0000 7f1a0197a640 1 -- 172.21.15.81:0/2060148537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19fc1f2f40 con 0x7f19fc1601f0 2024-06-29T14:23:56.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.040+0000 7f1a0197a640 1 -- 172.21.15.81:0/2060148537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19fc1f2a80 con 0x7f19fc1601f0 2024-06-29T14:23:56.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.041+0000 7f19f3fff640 1 -- 172.21.15.81:0/2060148537 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f19e4041030 con 0x7f19fc1601f0 2024-06-29T14:23:56.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.041+0000 7f19f3fff640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f19dc0764d0 0x7f19dc0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.042+0000 7f19f3fff640 1 -- 172.21.15.81:0/2060148537 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f19e40b8490 con 0x7f19fc1601f0 2024-06-29T14:23:56.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.042+0000 7f19faffd640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f19dc0764d0 0x7f19dc0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.042+0000 7f19faffd640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f19dc0764d0 0x7f19dc0789e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f19ec009fd0 tx=0x7f19ec00d040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:56.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.046+0000 7f19f3fff640 1 -- 172.21.15.81:0/2060148537 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f19fc1f2a80 con 0x7f19fc1601f0 2024-06-29T14:23:56.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.249+0000 7f1a0197a640 1 -- 172.21.15.81:0/2060148537 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f19fc0d02b0 con 0x7f19dc0764d0 2024-06-29T14:23:56.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.320+0000 7f19f3fff640 1 -- 172.21.15.81:0/2060148537 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f19fc0d02b0 con 0x7f19dc0764d0 2024-06-29T14:23:56.319 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/dir2 2024-06-29T14:23:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.322+0000 7f1a0197a640 1 -- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f19dc0764d0 msgr2=0x7f19dc0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.322+0000 7f1a0197a640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f19dc0764d0 0x7f19dc0789e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f19ec009fd0 tx=0x7f19ec00d040 comp rx=0 tx=0).stop 2024-06-29T14:23:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.322+0000 7f1a0197a640 1 -- 172.21.15.81:0/2060148537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19fc1601f0 msgr2=0x7f19fc13d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.322+0000 7f1a0197a640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19fc1601f0 0x7f19fc13d660 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f19e4034040 tx=0x7f19e4003970 comp rx=0 tx=0).stop 2024-06-29T14:23:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.322+0000 7f1a0197a640 1 -- 172.21.15.81:0/2060148537 shutdown_connections 2024-06-29T14:23:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.322+0000 7f1a0197a640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f19dc0764d0 0x7f19dc0789e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.322+0000 7f1a0197a640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19fc1f3740 0x7f19fc0701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.323+0000 7f1a0197a640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19fc1601f0 0x7f19fc13d660 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.323+0000 7f1a0197a640 1 --2- 172.21.15.81:0/2060148537 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19fc13ec50 0x7f19fc0af7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.323+0000 7f1a0197a640 1 -- 172.21.15.81:0/2060148537 >> 172.21.15.81:0/2060148537 conn(0x7f19fc0d0b70 msgr2=0x7f19fc15e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.323+0000 7f1a0197a640 1 -- 172.21.15.81:0/2060148537 shutdown_connections 2024-06-29T14:23:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.323+0000 7f1a0197a640 1 -- 172.21.15.81:0/2060148537 wait complete. 2024-06-29T14:23:56.332 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2024-06-29T14:23:56.332 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2024-06-29T14:23:56.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.450+0000 7efc74c5b640 1 -- 172.21.15.81:0/603993820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc7006f680 msgr2=0x7efc7006c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.450+0000 7efc74c5b640 1 --2- 172.21.15.81:0/603993820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc7006f680 0x7efc7006c2f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7efc58009830 tx=0x7efc5802a510 comp rx=0 tx=0).stop 2024-06-29T14:23:56.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.450+0000 7efc74c5b640 1 -- 172.21.15.81:0/603993820 shutdown_connections 2024-06-29T14:23:56.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.450+0000 7efc74c5b640 1 --2- 172.21.15.81:0/603993820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc7006f680 0x7efc7006c2f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.450+0000 7efc74c5b640 1 --2- 172.21.15.81:0/603993820 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc70074520 0x7efc7006f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.450+0000 7efc74c5b640 1 -- 172.21.15.81:0/603993820 >> 172.21.15.81:0/603993820 conn(0x7efc70072950 msgr2=0x7efc700b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:56.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.450+0000 7efc74c5b640 1 -- 172.21.15.81:0/603993820 shutdown_connections 2024-06-29T14:23:56.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.450+0000 7efc74c5b640 1 -- 172.21.15.81:0/603993820 wait complete. 2024-06-29T14:23:56.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.452+0000 7efc74c5b640 1 Processor -- start 2024-06-29T14:23:56.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.452+0000 7efc74c5b640 1 -- start start 2024-06-29T14:23:56.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.453+0000 7efc74c5b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc7006f680 0x7efc701f6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.453+0000 7efc74c5b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc7006fa90 0x7efc7009d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.453+0000 7efc74c5b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc70074520 0x7efc700d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.453+0000 7efc74c5b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc7015b010 con 0x7efc7006f680 2024-06-29T14:23:56.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.453+0000 7efc74c5b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efc7006d150 con 0x7efc70074520 2024-06-29T14:23:56.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.453+0000 7efc74c5b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efc7015b190 con 0x7efc7006fa90 2024-06-29T14:23:56.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.453+0000 7efc6e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc7006f680 0x7efc701f6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.453+0000 7efc6e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc7006f680 0x7efc701f6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36108/0 (socket says 172.21.15.81:36108) 2024-06-29T14:23:56.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.453+0000 7efc6e59c640 1 -- 172.21.15.81:0/2452511624 learned_addr learned my addr 172.21.15.81:0/2452511624 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:56.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc67fff640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc7006fa90 0x7efc7009d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc6e59c640 1 -- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc7006fa90 msgr2=0x7efc7009d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc6ed9d640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc70074520 0x7efc700d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc6e59c640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc7006fa90 0x7efc7009d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc6e59c640 1 -- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc70074520 msgr2=0x7efc700d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc6e59c640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc70074520 0x7efc700d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc6e59c640 1 -- 172.21.15.81:0/2452511624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc701f6ca0 con 0x7efc7006f680 2024-06-29T14:23:56.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc6e59c640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc7006f680 0x7efc701f6220 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efc6000d740 tx=0x7efc6000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:56.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc66ffd640 1 -- 172.21.15.81:0/2452511624 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc60017070 con 0x7efc7006f680 2024-06-29T14:23:56.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc66ffd640 1 -- 172.21.15.81:0/2452511624 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc60009d20 con 0x7efc7006f680 2024-06-29T14:23:56.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc74c5b640 1 -- 172.21.15.81:0/2452511624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc701f6f80 con 0x7efc7006f680 2024-06-29T14:23:56.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc66ffd640 1 -- 172.21.15.81:0/2452511624 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc600106c0 con 0x7efc7006f680 2024-06-29T14:23:56.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.454+0000 7efc74c5b640 1 -- 172.21.15.81:0/2452511624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc7009ab40 con 0x7efc7006f680 2024-06-29T14:23:56.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.455+0000 7efc74c5b640 1 -- 172.21.15.81:0/2452511624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc70068ef0 con 0x7efc7006f680 2024-06-29T14:23:56.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.456+0000 7efc66ffd640 1 -- 172.21.15.81:0/2452511624 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7efc6001a070 con 0x7efc7006f680 2024-06-29T14:23:56.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.456+0000 7efc66ffd640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efc500764d0 0x7efc500789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.457+0000 7efc66ffd640 1 -- 172.21.15.81:0/2452511624 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7efc60097e30 con 0x7efc7006f680 2024-06-29T14:23:56.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.457+0000 7efc67fff640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efc500764d0 0x7efc500789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.457+0000 7efc67fff640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efc500764d0 0x7efc500789e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efc58034000 tx=0x7efc58004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:56.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.461+0000 7efc66ffd640 1 -- 172.21.15.81:0/2452511624 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efc60061270 con 0x7efc7006f680 2024-06-29T14:23:56.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.665+0000 7efc74c5b640 1 -- 172.21.15.81:0/2452511624 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7efc700af5b0 con 0x7efc500764d0 2024-06-29T14:23:56.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.730+0000 7efc66ffd640 1 -- 172.21.15.81:0/2452511624 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7efc700af5b0 con 0x7efc500764d0 2024-06-29T14:23:56.729 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/dir3 2024-06-29T14:23:56.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 -- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efc500764d0 msgr2=0x7efc500789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efc500764d0 0x7efc500789e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efc58034000 tx=0x7efc58004730 comp rx=0 tx=0).stop 2024-06-29T14:23:56.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 -- 172.21.15.81:0/2452511624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc7006f680 msgr2=0x7efc701f6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc7006f680 0x7efc701f6220 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efc6000d740 tx=0x7efc6000dc00 comp rx=0 tx=0).stop 2024-06-29T14:23:56.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 -- 172.21.15.81:0/2452511624 shutdown_connections 2024-06-29T14:23:56.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efc500764d0 0x7efc500789e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efc70074520 0x7efc700d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc7006fa90 0x7efc7009d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 --2- 172.21.15.81:0/2452511624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc7006f680 0x7efc701f6220 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 -- 172.21.15.81:0/2452511624 >> 172.21.15.81:0/2452511624 conn(0x7efc70072950 msgr2=0x7efc7006c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:56.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 -- 172.21.15.81:0/2452511624 shutdown_connections 2024-06-29T14:23:56.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.733+0000 7efc74c5b640 1 -- 172.21.15.81:0/2452511624 wait complete. 2024-06-29T14:23:56.742 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2024-06-29T14:23:56.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2024-06-29T14:23:56.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.861+0000 7ff038865640 1 -- 172.21.15.81:0/3586251207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff03415aab0 msgr2=0x7ff03415aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.861+0000 7ff038865640 1 --2- 172.21.15.81:0/3586251207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff03415aab0 0x7ff03415aec0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff02c005fd0 tx=0x7ff02c02a780 comp rx=0 tx=0).stop 2024-06-29T14:23:56.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.861+0000 7ff038865640 1 -- 172.21.15.81:0/3586251207 shutdown_connections 2024-06-29T14:23:56.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.861+0000 7ff038865640 1 --2- 172.21.15.81:0/3586251207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff03415aab0 0x7ff03415aec0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.861+0000 7ff038865640 1 --2- 172.21.15.81:0/3586251207 >> v2:172.21.15.167:3300/0 conn(0x7ff03406fd70 0x7ff0340b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.861+0000 7ff038865640 1 -- 172.21.15.81:0/3586251207 >> 172.21.15.81:0/3586251207 conn(0x7ff03406ccc0 msgr2=0x7ff0340b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:56.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.861+0000 7ff038865640 1 -- 172.21.15.81:0/3586251207 shutdown_connections 2024-06-29T14:23:56.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.861+0000 7ff038865640 1 -- 172.21.15.81:0/3586251207 wait complete. 2024-06-29T14:23:56.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.863+0000 7ff038865640 1 Processor -- start 2024-06-29T14:23:56.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.863+0000 7ff038865640 1 -- start start 2024-06-29T14:23:56.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.864+0000 7ff038865640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0340cf4c0 0x7ff034146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.864+0000 7ff038865640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff03415aab0 0x7ff0340ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.864+0000 7ff038865640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0341563b0 0x7ff0340aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.864+0000 7ff038865640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff03406f160 con 0x7ff0340cf4c0 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.864+0000 7ff038865640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff03406efe0 con 0x7ff03415aab0 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.864+0000 7ff038865640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff03406f2e0 con 0x7ff0341563b0 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.864+0000 7ff03259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0340cf4c0 0x7ff034146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.864+0000 7ff03259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0340cf4c0 0x7ff034146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36124/0 (socket says 172.21.15.81:36124) 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.864+0000 7ff03259c640 1 -- 172.21.15.81:0/1391304503 learned_addr learned my addr 172.21.15.81:0/1391304503 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff032d9d640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0341563b0 0x7ff0340aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff031d9b640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff03415aab0 0x7ff0340ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff03259c640 1 -- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0341563b0 msgr2=0x7ff0340aed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff03259c640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0341563b0 0x7ff0340aed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff03259c640 1 -- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff03415aab0 msgr2=0x7ff0340ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:56.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff03259c640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff03415aab0 0x7ff0340ae7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:56.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff03259c640 1 -- 172.21.15.81:0/1391304503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0340af240 con 0x7ff0340cf4c0 2024-06-29T14:23:56.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff03259c640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0340cf4c0 0x7ff034146340 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff02400c3a0 tx=0x7ff02400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:56.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff022ffd640 1 -- 172.21.15.81:0/1391304503 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024011790 con 0x7ff0340cf4c0 2024-06-29T14:23:56.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff022ffd640 1 -- 172.21.15.81:0/1391304503 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff024017070 con 0x7ff0340cf4c0 2024-06-29T14:23:56.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff038865640 1 -- 172.21.15.81:0/1391304503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03415ec40 con 0x7ff0340cf4c0 2024-06-29T14:23:56.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff022ffd640 1 -- 172.21.15.81:0/1391304503 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024011cc0 con 0x7ff0340cf4c0 2024-06-29T14:23:56.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.865+0000 7ff038865640 1 -- 172.21.15.81:0/1391304503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0340adce0 con 0x7ff0340cf4c0 2024-06-29T14:23:56.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.866+0000 7ff038865640 1 -- 172.21.15.81:0/1391304503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0340b3640 con 0x7ff0340cf4c0 2024-06-29T14:23:56.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.867+0000 7ff022ffd640 1 -- 172.21.15.81:0/1391304503 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7ff02401c460 con 0x7ff0340cf4c0 2024-06-29T14:23:56.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.868+0000 7ff022ffd640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff0140764d0 0x7ff0140789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:56.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.868+0000 7ff022ffd640 1 -- 172.21.15.81:0/1391304503 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7ff024097a10 con 0x7ff0340cf4c0 2024-06-29T14:23:56.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.868+0000 7ff031d9b640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff0140764d0 0x7ff0140789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:56.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.868+0000 7ff031d9b640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff0140764d0 0x7ff0140789e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff02800a700 tx=0x7ff028010040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:56.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:56.872+0000 7ff022ffd640 1 -- 172.21.15.81:0/1391304503 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff024060900 con 0x7ff0340cf4c0 2024-06-29T14:23:57.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.074+0000 7ff038865640 1 -- 172.21.15.81:0/1391304503 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7ff03415c510 con 0x7ff0140764d0 2024-06-29T14:23:57.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.134+0000 7ff022ffd640 1 -- 172.21.15.81:0/1391304503 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7ff02800ec30 con 0x7ff0140764d0 2024-06-29T14:23:57.132 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/dir4 2024-06-29T14:23:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 -- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff0140764d0 msgr2=0x7ff0140789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff0140764d0 0x7ff0140789e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff02800a700 tx=0x7ff028010040 comp rx=0 tx=0).stop 2024-06-29T14:23:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 -- 172.21.15.81:0/1391304503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0340cf4c0 msgr2=0x7ff034146340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0340cf4c0 0x7ff034146340 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff02400c3a0 tx=0x7ff02400c860 comp rx=0 tx=0).stop 2024-06-29T14:23:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 -- 172.21.15.81:0/1391304503 shutdown_connections 2024-06-29T14:23:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff0140764d0 0x7ff0140789e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0341563b0 0x7ff0340aed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff03415aab0 0x7ff0340ae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 --2- 172.21.15.81:0/1391304503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0340cf4c0 0x7ff034146340 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 -- 172.21.15.81:0/1391304503 >> 172.21.15.81:0/1391304503 conn(0x7ff03406ccc0 msgr2=0x7ff034158640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 -- 172.21.15.81:0/1391304503 shutdown_connections 2024-06-29T14:23:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.136+0000 7ff038865640 1 -- 172.21.15.81:0/1391304503 wait complete. 2024-06-29T14:23:57.145 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2024-06-29T14:23:57.145 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2024-06-29T14:23:57.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.265+0000 7f60a14d6640 1 -- 172.21.15.81:0/2987280027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088004800 msgr2=0x7f6088004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.265+0000 7f60a14d6640 1 --2- 172.21.15.81:0/2987280027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088004800 0x7f6088004c10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f609c0696f0 tx=0x7f6088005830 comp rx=0 tx=0).stop 2024-06-29T14:23:57.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.265+0000 7f609b7fe640 1 -- 172.21.15.81:0/2987280027 reap_dead start 2024-06-29T14:23:57.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.265+0000 7f60a14d6640 1 -- 172.21.15.81:0/2987280027 shutdown_connections 2024-06-29T14:23:57.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.265+0000 7f60a14d6640 1 -- 172.21.15.81:0/2987280027 >> 172.21.15.81:0/2987280027 conn(0x7f609c0b3680 msgr2=0x7f609c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:57.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.265+0000 7f60a14d6640 1 -- 172.21.15.81:0/2987280027 shutdown_connections 2024-06-29T14:23:57.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.265+0000 7f60a14d6640 1 -- 172.21.15.81:0/2987280027 wait complete. 2024-06-29T14:23:57.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.266+0000 7f60a14d6640 1 Processor -- start 2024-06-29T14:23:57.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.267+0000 7f60a14d6640 1 -- start start 2024-06-29T14:23:57.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f60a14d6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088004800 0x7f609c157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f60a14d6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f609c157cb0 0x7f609c153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f60a14d6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f609c153ad0 0x7f609c153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f60a14d6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f609c0b59e0 con 0x7f6088004800 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f60a14d6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f609c0b5860 con 0x7f609c153ad0 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f60a14d6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f609c0b5b60 con 0x7f609c157cb0 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088004800 0x7f609c157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088004800 0x7f609c157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36136/0 (socket says 172.21.15.81:36136) 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609affd640 1 -- 172.21.15.81:0/2145089868 learned_addr learned my addr 172.21.15.81:0/2145089868 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f60927fc640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f609c157cb0 0x7f609c153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609b7fe640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f609c153ad0 0x7f609c153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609affd640 1 -- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f609c157cb0 msgr2=0x7f609c153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609affd640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f609c157cb0 0x7f609c153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609affd640 1 -- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f609c153ad0 msgr2=0x7f609c153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609affd640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f609c153ad0 0x7f609c153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609affd640 1 -- 172.21.15.81:0/2145089868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609c09e370 con 0x7f6088004800 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f60927fc640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f609c157cb0 0x7f609c153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609b7fe640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f609c153ad0 0x7f609c153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.268+0000 7f609affd640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088004800 0x7f609c157770 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f608c011b10 tx=0x7f608c009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.269+0000 7f6093fff640 1 -- 172.21.15.81:0/2145089868 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c01e070 con 0x7f6088004800 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.269+0000 7f6093fff640 1 -- 172.21.15.81:0/2145089868 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f608c004050 con 0x7f6088004800 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.269+0000 7f6093fff640 1 -- 172.21.15.81:0/2145089868 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c00fea0 con 0x7f6088004800 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.269+0000 7f60a14d6640 1 -- 172.21.15.81:0/2145089868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609c09e650 con 0x7f6088004800 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.269+0000 7f60a14d6640 1 -- 172.21.15.81:0/2145089868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f609c09ec10 con 0x7f6088004800 2024-06-29T14:23:57.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.270+0000 7f60a14d6640 1 -- 172.21.15.81:0/2145089868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f609c0b60f0 con 0x7f6088004800 2024-06-29T14:23:57.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.270+0000 7f6093fff640 1 -- 172.21.15.81:0/2145089868 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f608c021070 con 0x7f6088004800 2024-06-29T14:23:57.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.271+0000 7f6093fff640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f607c0764d0 0x7f607c0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:57.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.271+0000 7f6093fff640 1 -- 172.21.15.81:0/2145089868 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f608c0986e0 con 0x7f6088004800 2024-06-29T14:23:57.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.271+0000 7f60927fc640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f607c0764d0 0x7f607c0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:57.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.272+0000 7f60927fc640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f607c0764d0 0x7f607c0789e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6084033000 tx=0x7f6084000fe0 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:57.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.275+0000 7f6093fff640 1 -- 172.21.15.81:0/2145089868 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f608c0044a0 con 0x7f6088004800 2024-06-29T14:23:57.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.480+0000 7f60a14d6640 1 -- 172.21.15.81:0/2145089868 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f609c0d0260 con 0x7f607c0764d0 2024-06-29T14:23:57.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.548+0000 7f6093fff640 1 -- 172.21.15.81:0/2145089868 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f609c0d0260 con 0x7f607c0764d0 2024-06-29T14:23:57.547 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/dir5 2024-06-29T14:23:57.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 -- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f607c0764d0 msgr2=0x7f607c0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f607c0764d0 0x7f607c0789e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6084033000 tx=0x7f6084000fe0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 -- 172.21.15.81:0/2145089868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088004800 msgr2=0x7f609c157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088004800 0x7f609c157770 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f608c011b10 tx=0x7f608c009e90 comp rx=0 tx=0).stop 2024-06-29T14:23:57.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 -- 172.21.15.81:0/2145089868 shutdown_connections 2024-06-29T14:23:57.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f607c0764d0 0x7f607c0789e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f609c153ad0 0x7f609c153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f609c157cb0 0x7f609c153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 --2- 172.21.15.81:0/2145089868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088004800 0x7f609c157770 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 -- 172.21.15.81:0/2145089868 >> 172.21.15.81:0/2145089868 conn(0x7f609c0b3680 msgr2=0x7f609c070610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:57.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 -- 172.21.15.81:0/2145089868 shutdown_connections 2024-06-29T14:23:57.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.550+0000 7f60a14d6640 1 -- 172.21.15.81:0/2145089868 wait complete. 2024-06-29T14:23:57.559 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2024-06-29T14:23:57.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2024-06-29T14:23:57.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.681+0000 7fb07bd11640 1 -- 172.21.15.81:0/3631669108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb060003fd0 msgr2=0x7fb060004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.681+0000 7fb07bd11640 1 --2- 172.21.15.81:0/3631669108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb060003fd0 0x7fb060004800 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb0740514c0 tx=0x7fb060005830 comp rx=0 tx=0).stop 2024-06-29T14:23:57.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.681+0000 7fb07a2ae640 1 -- 172.21.15.81:0/3631669108 reap_dead start 2024-06-29T14:23:57.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.681+0000 7fb07bd11640 1 -- 172.21.15.81:0/3631669108 shutdown_connections 2024-06-29T14:23:57.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.681+0000 7fb07bd11640 1 -- 172.21.15.81:0/3631669108 >> 172.21.15.81:0/3631669108 conn(0x7fb0740d0930 msgr2=0x7fb0740d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:57.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.681+0000 7fb07bd11640 1 -- 172.21.15.81:0/3631669108 shutdown_connections 2024-06-29T14:23:57.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.681+0000 7fb07bd11640 1 -- 172.21.15.81:0/3631669108 wait complete. 2024-06-29T14:23:57.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.683+0000 7fb07bd11640 1 Processor -- start 2024-06-29T14:23:57.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.683+0000 7fb07bd11640 1 -- start start 2024-06-29T14:23:57.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb07bd11640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb060003fd0 0x7fb0740ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:57.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb07bd11640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb074151f20 0x7fb0740c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:57.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb07bd11640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0741ef8a0 0x7fb0741f5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:57.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb07bd11640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb07415aed0 con 0x7fb060003fd0 2024-06-29T14:23:57.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb07bd11640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb0740b4b80 con 0x7fb074151f20 2024-06-29T14:23:57.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb07bd11640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb07415b050 con 0x7fb0741ef8a0 2024-06-29T14:23:57.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb079aad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb060003fd0 0x7fb0740ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb079aad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb060003fd0 0x7fb0740ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36154/0 (socket says 172.21.15.81:36154) 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb079aad640 1 -- 172.21.15.81:0/3440567148 learned_addr learned my addr 172.21.15.81:0/3440567148 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb0792ac640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb074151f20 0x7fb0740c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.684+0000 7fb07a2ae640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0741ef8a0 0x7fb0741f5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb079aad640 1 -- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0741ef8a0 msgr2=0x7fb0741f5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb079aad640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0741ef8a0 0x7fb0741f5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb079aad640 1 -- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb074151f20 msgr2=0x7fb0740c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb079aad640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb074151f20 0x7fb0740c79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb079aad640 1 -- 172.21.15.81:0/3440567148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb074152690 con 0x7fb060003fd0 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb079aad640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb060003fd0 0x7fb0740ceff0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb070011a50 tx=0x7fb070011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb06a7fc640 1 -- 172.21.15.81:0/3440567148 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07001f070 con 0x7fb060003fd0 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb06a7fc640 1 -- 172.21.15.81:0/3440567148 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb07000fe20 con 0x7fb060003fd0 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb07bd11640 1 -- 172.21.15.81:0/3440567148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb074152970 con 0x7fb060003fd0 2024-06-29T14:23:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb06a7fc640 1 -- 172.21.15.81:0/3440567148 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070012690 con 0x7fb060003fd0 2024-06-29T14:23:57.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.685+0000 7fb07bd11640 1 -- 172.21.15.81:0/3440567148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0741f6470 con 0x7fb060003fd0 2024-06-29T14:23:57.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.686+0000 7fb07bd11640 1 -- 172.21.15.81:0/3440567148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb07406e2a0 con 0x7fb060003fd0 2024-06-29T14:23:57.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.687+0000 7fb06a7fc640 1 -- 172.21.15.81:0/3440567148 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7fb070022070 con 0x7fb060003fd0 2024-06-29T14:23:57.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.687+0000 7fb06a7fc640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb0580764d0 0x7fb0580789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:57.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.687+0000 7fb06a7fc640 1 -- 172.21.15.81:0/3440567148 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7fb0700985e0 con 0x7fb060003fd0 2024-06-29T14:23:57.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.687+0000 7fb0792ac640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb0580764d0 0x7fb0580789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:57.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.688+0000 7fb0792ac640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb0580764d0 0x7fb0580789e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb064035000 tx=0x7fb0640046c0 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:57.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.692+0000 7fb06a7fc640 1 -- 172.21.15.81:0/3440567148 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb07406e2a0 con 0x7fb060003fd0 2024-06-29T14:23:57.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.897+0000 7fb07bd11640 1 -- 172.21.15.81:0/3440567148 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb0741f6710 con 0x7fb0580764d0 2024-06-29T14:23:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.966+0000 7fb06a7fc640 1 -- 172.21.15.81:0/3440567148 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb0741f6710 con 0x7fb0580764d0 2024-06-29T14:23:57.964 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/dir6 2024-06-29T14:23:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.968+0000 7fb07bd11640 1 -- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb0580764d0 msgr2=0x7fb0580789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.968+0000 7fb07bd11640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb0580764d0 0x7fb0580789e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb064035000 tx=0x7fb0640046c0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.968+0000 7fb07bd11640 1 -- 172.21.15.81:0/3440567148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb060003fd0 msgr2=0x7fb0740ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.968+0000 7fb07bd11640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb060003fd0 0x7fb0740ceff0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb070011a50 tx=0x7fb070011f10 comp rx=0 tx=0).stop 2024-06-29T14:23:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.968+0000 7fb07bd11640 1 -- 172.21.15.81:0/3440567148 shutdown_connections 2024-06-29T14:23:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.968+0000 7fb07bd11640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb0580764d0 0x7fb0580789e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.968+0000 7fb07bd11640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0741ef8a0 0x7fb0741f5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.968+0000 7fb07bd11640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb074151f20 0x7fb0740c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.968+0000 7fb07bd11640 1 --2- 172.21.15.81:0/3440567148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb060003fd0 0x7fb0740ceff0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:57.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.968+0000 7fb07bd11640 1 -- 172.21.15.81:0/3440567148 >> 172.21.15.81:0/3440567148 conn(0x7fb0740d0930 msgr2=0x7fb0740b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:57.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.969+0000 7fb07bd11640 1 -- 172.21.15.81:0/3440567148 shutdown_connections 2024-06-29T14:23:57.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:57.969+0000 7fb07bd11640 1 -- 172.21.15.81:0/3440567148 wait complete. 2024-06-29T14:23:57.977 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2024-06-29T14:23:57.978 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2024-06-29T14:23:58.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.097+0000 7f95fe7d4640 1 -- 172.21.15.81:0/1004519671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95e0004800 msgr2=0x7f95e0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.097+0000 7f95fe7d4640 1 --2- 172.21.15.81:0/1004519671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95e0004800 0x7f95e0004c10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00c260 tx=0x7f95e0005830 comp rx=0 tx=0).stop 2024-06-29T14:23:58.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.097+0000 7f95fcd71640 1 -- 172.21.15.81:0/1004519671 reap_dead start 2024-06-29T14:23:58.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.097+0000 7f95fe7d4640 1 -- 172.21.15.81:0/1004519671 shutdown_connections 2024-06-29T14:23:58.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.097+0000 7f95fe7d4640 1 -- 172.21.15.81:0/1004519671 >> 172.21.15.81:0/1004519671 conn(0x7f95f806ccc0 msgr2=0x7f95f80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:58.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.098+0000 7f95fe7d4640 1 -- 172.21.15.81:0/1004519671 shutdown_connections 2024-06-29T14:23:58.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.098+0000 7f95fe7d4640 1 -- 172.21.15.81:0/1004519671 wait complete. 2024-06-29T14:23:58.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.099+0000 7f95fe7d4640 1 Processor -- start 2024-06-29T14:23:58.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.099+0000 7f95fe7d4640 1 -- start start 2024-06-29T14:23:58.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95fe7d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95e0004800 0x7f95f815f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95fe7d4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95f81f68b0 0x7f95f80b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95fe7d4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95f81f6cc0 0x7f95f80b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95fe7d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f95f8072910 con 0x7f95e0004800 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95fe7d4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f95f8072790 con 0x7f95f81f6cc0 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95fe7d4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f95f8072a90 con 0x7f95f81f68b0 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95e0004800 0x7f95f815f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95e0004800 0x7f95f815f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36162/0 (socket says 172.21.15.81:36162) 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f7fff640 1 -- 172.21.15.81:0/2985835085 learned_addr learned my addr 172.21.15.81:0/2985835085 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f77fe640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95f81f68b0 0x7f95f80b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f7fff640 1 -- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95f81f68b0 msgr2=0x7f95f80b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95fcd71640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95f81f6cc0 0x7f95f80b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f7fff640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95f81f68b0 0x7f95f80b0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f7fff640 1 -- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95f81f6cc0 msgr2=0x7f95f80b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f7fff640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95f81f6cc0 0x7f95f80b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f7fff640 1 -- 172.21.15.81:0/2985835085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f81f60b0 con 0x7f95e0004800 2024-06-29T14:23:58.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f7fff640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95e0004800 0x7f95f815f970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f95e8011a80 tx=0x7f95e8011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:58.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f4ff9640 1 -- 172.21.15.81:0/2985835085 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e800fd50 con 0x7f95e0004800 2024-06-29T14:23:58.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95f4ff9640 1 -- 172.21.15.81:0/2985835085 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95e8016070 con 0x7f95e0004800 2024-06-29T14:23:58.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.101+0000 7f95fe7d4640 1 -- 172.21.15.81:0/2985835085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f81592e0 con 0x7f95e0004800 2024-06-29T14:23:58.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.102+0000 7f95f4ff9640 1 -- 172.21.15.81:0/2985835085 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e80126b0 con 0x7f95e0004800 2024-06-29T14:23:58.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.102+0000 7f95fe7d4640 1 -- 172.21.15.81:0/2985835085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f81598a0 con 0x7f95e0004800 2024-06-29T14:23:58.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.103+0000 7f95fe7d4640 1 -- 172.21.15.81:0/2985835085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95f815b5b0 con 0x7f95e0004800 2024-06-29T14:23:58.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.103+0000 7f95f4ff9640 1 -- 172.21.15.81:0/2985835085 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f95e8018030 con 0x7f95e0004800 2024-06-29T14:23:58.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.104+0000 7f95f4ff9640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f95e40764d0 0x7f95e40789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.104+0000 7f95f4ff9640 1 -- 172.21.15.81:0/2985835085 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f95e8098480 con 0x7f95e0004800 2024-06-29T14:23:58.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.104+0000 7f95f77fe640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f95e40764d0 0x7f95e40789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.105+0000 7f95f77fe640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f95e40764d0 0x7f95e40789e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00c260 tx=0x7f95ec02d000 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:58.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.108+0000 7f95f4ff9640 1 -- 172.21.15.81:0/2985835085 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f95e8061a60 con 0x7f95e0004800 2024-06-29T14:23:58.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.311+0000 7f95fe7d4640 1 -- 172.21.15.81:0/2985835085 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f95f80cf2d0 con 0x7f95e40764d0 2024-06-29T14:23:58.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.373+0000 7f95f4ff9640 1 -- 172.21.15.81:0/2985835085 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f95f80cf2d0 con 0x7f95e40764d0 2024-06-29T14:23:58.371 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/dir7 2024-06-29T14:23:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 -- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f95e40764d0 msgr2=0x7f95e40789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f95e40764d0 0x7f95e40789e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00c260 tx=0x7f95ec02d000 comp rx=0 tx=0).stop 2024-06-29T14:23:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 -- 172.21.15.81:0/2985835085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95e0004800 msgr2=0x7f95f815f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95e0004800 0x7f95f815f970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f95e8011a80 tx=0x7f95e8011f40 comp rx=0 tx=0).stop 2024-06-29T14:23:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 -- 172.21.15.81:0/2985835085 shutdown_connections 2024-06-29T14:23:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f95e40764d0 0x7f95e40789e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95f81f6cc0 0x7f95f80b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95f81f68b0 0x7f95f80b0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 --2- 172.21.15.81:0/2985835085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95e0004800 0x7f95f815f970 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 -- 172.21.15.81:0/2985835085 >> 172.21.15.81:0/2985835085 conn(0x7f95f806ccc0 msgr2=0x7f95f80ce740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 -- 172.21.15.81:0/2985835085 shutdown_connections 2024-06-29T14:23:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.375+0000 7f95fe7d4640 1 -- 172.21.15.81:0/2985835085 wait complete. 2024-06-29T14:23:58.384 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2024-06-29T14:23:58.384 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2024-06-29T14:23:58.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.503+0000 7f65238fd640 1 -- 172.21.15.81:0/1305405045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6508005250 msgr2=0x7f650800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.503+0000 7f65238fd640 1 --2- 172.21.15.81:0/1305405045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6508005250 0x7f650800b9c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f65140013d0 tx=0x7f6508005830 comp rx=0 tx=0).stop 2024-06-29T14:23:58.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.503+0000 7f65238fd640 1 -- 172.21.15.81:0/1305405045 shutdown_connections 2024-06-29T14:23:58.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.503+0000 7f65238fd640 1 --2- 172.21.15.81:0/1305405045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6508005250 0x7f650800b9c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.503+0000 7f65238fd640 1 -- 172.21.15.81:0/1305405045 >> 172.21.15.81:0/1305405045 conn(0x7f651c09cf30 msgr2=0x7f651c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:58.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.503+0000 7f65238fd640 1 -- 172.21.15.81:0/1305405045 shutdown_connections 2024-06-29T14:23:58.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.503+0000 7f65238fd640 1 -- 172.21.15.81:0/1305405045 wait complete. 2024-06-29T14:23:58.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.505+0000 7f65238fd640 1 Processor -- start 2024-06-29T14:23:58.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.505+0000 7f65238fd640 1 -- start start 2024-06-29T14:23:58.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f65238fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6508005250 0x7f651c154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f65238fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f651c1584d0 0x7f651c1588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f65238fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f651c15da00 0x7f651c15de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f65238fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f651c0d0860 con 0x7f651c1584d0 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f65238fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f651c0d06e0 con 0x7f6508005250 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f65238fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f651c0d09e0 con 0x7f651c15da00 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f6520e98640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f651c1584d0 0x7f651c1588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f6520e98640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f651c1584d0 0x7f651c1588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36180/0 (socket says 172.21.15.81:36180) 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f6520e98640 1 -- 172.21.15.81:0/2289633393 learned_addr learned my addr 172.21.15.81:0/2289633393 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f6521699640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6508005250 0x7f651c154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.506+0000 7f6521e9a640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f651c15da00 0x7f651c15de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f6520e98640 1 -- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f651c15da00 msgr2=0x7f651c15de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f6520e98640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f651c15da00 0x7f651c15de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f6520e98640 1 -- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6508005250 msgr2=0x7f651c154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f6520e98640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6508005250 0x7f651c154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f6520e98640 1 -- 172.21.15.81:0/2289633393 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c15e350 con 0x7f651c1584d0 2024-06-29T14:23:58.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f6521e9a640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f651c15da00 0x7f651c15de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:58.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f6520e98640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f651c1584d0 0x7f651c1588e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f650c011800 tx=0x7f650c011c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:58.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f6511ffb640 1 -- 172.21.15.81:0/2289633393 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c004460 con 0x7f651c1584d0 2024-06-29T14:23:58.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f6511ffb640 1 -- 172.21.15.81:0/2289633393 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f650c016070 con 0x7f651c1584d0 2024-06-29T14:23:58.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f6511ffb640 1 -- 172.21.15.81:0/2289633393 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c00fb60 con 0x7f651c1584d0 2024-06-29T14:23:58.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f65238fd640 1 -- 172.21.15.81:0/2289633393 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651c15b9e0 con 0x7f651c1584d0 2024-06-29T14:23:58.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.507+0000 7f65238fd640 1 -- 172.21.15.81:0/2289633393 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f651c15bfa0 con 0x7f651c1584d0 2024-06-29T14:23:58.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.508+0000 7f65238fd640 1 -- 172.21.15.81:0/2289633393 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f651c0b7290 con 0x7f651c1584d0 2024-06-29T14:23:58.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.509+0000 7f6511ffb640 1 -- 172.21.15.81:0/2289633393 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f650c00fd00 con 0x7f651c1584d0 2024-06-29T14:23:58.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.509+0000 7f6511ffb640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6500076500 0x7f6500078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.510+0000 7f6511ffb640 1 -- 172.21.15.81:0/2289633393 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f650c098840 con 0x7f651c1584d0 2024-06-29T14:23:58.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.510+0000 7f6521699640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6500076500 0x7f6500078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.510+0000 7f6521699640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6500076500 0x7f6500078a10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6518006680 tx=0x7f6518006b70 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:58.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.514+0000 7f6511ffb640 1 -- 172.21.15.81:0/2289633393 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f651c15e350 con 0x7f651c1584d0 2024-06-29T14:23:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.718+0000 7f65238fd640 1 -- 172.21.15.81:0/2289633393 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f651c15b730 con 0x7f6500076500 2024-06-29T14:23:58.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.777+0000 7f6511ffb640 1 -- 172.21.15.81:0/2289633393 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f651c15b730 con 0x7f6500076500 2024-06-29T14:23:58.776 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/dir8 2024-06-29T14:23:58.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 -- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6500076500 msgr2=0x7f6500078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6500076500 0x7f6500078a10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6518006680 tx=0x7f6518006b70 comp rx=0 tx=0).stop 2024-06-29T14:23:58.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 -- 172.21.15.81:0/2289633393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f651c1584d0 msgr2=0x7f651c1588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f651c1584d0 0x7f651c1588e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f650c011800 tx=0x7f650c011c00 comp rx=0 tx=0).stop 2024-06-29T14:23:58.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 -- 172.21.15.81:0/2289633393 shutdown_connections 2024-06-29T14:23:58.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6500076500 0x7f6500078a10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f651c15da00 0x7f651c15de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f651c1584d0 0x7f651c1588e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 --2- 172.21.15.81:0/2289633393 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6508005250 0x7f651c154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 -- 172.21.15.81:0/2289633393 >> 172.21.15.81:0/2289633393 conn(0x7f651c09cf30 msgr2=0x7f651c160e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:58.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 -- 172.21.15.81:0/2289633393 shutdown_connections 2024-06-29T14:23:58.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.780+0000 7f65238fd640 1 -- 172.21.15.81:0/2289633393 wait complete. 2024-06-29T14:23:58.789 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2024-06-29T14:23:58.789 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2024-06-29T14:23:58.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.907+0000 7f95c0dfb640 1 -- 172.21.15.81:0/223345146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95bc073030 msgr2=0x7f95bc073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.907+0000 7f95c0dfb640 1 --2- 172.21.15.81:0/223345146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95bc073030 0x7f95bc073440 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f95bc071d20 tx=0x7f95b002a560 comp rx=0 tx=0).stop 2024-06-29T14:23:58.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.907+0000 7f95c0dfb640 1 -- 172.21.15.81:0/223345146 shutdown_connections 2024-06-29T14:23:58.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.907+0000 7f95c0dfb640 1 --2- 172.21.15.81:0/223345146 >> v2:172.21.15.167:3300/0 conn(0x7f95bc15e960 0x7f95bc15ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.907+0000 7f95c0dfb640 1 --2- 172.21.15.81:0/223345146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95bc073030 0x7f95bc073440 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.907+0000 7f95c0dfb640 1 -- 172.21.15.81:0/223345146 >> 172.21.15.81:0/223345146 conn(0x7f95bc06ccc0 msgr2=0x7f95bc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:58.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.907+0000 7f95c0dfb640 1 -- 172.21.15.81:0/223345146 shutdown_connections 2024-06-29T14:23:58.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.907+0000 7f95c0dfb640 1 -- 172.21.15.81:0/223345146 wait complete. 2024-06-29T14:23:58.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.909+0000 7f95c0dfb640 1 Processor -- start 2024-06-29T14:23:58.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.909+0000 7f95c0dfb640 1 -- start start 2024-06-29T14:23:58.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95c0dfb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95bc073030 0x7f95bc1572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95c0dfb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95bc1412f0 0x7f95bc1f8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95c0dfb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95bc15e960 0x7f95bc0cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95c0dfb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc0b3210 con 0x7f95bc073030 2024-06-29T14:23:58.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95c0dfb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc141740 con 0x7f95bc15e960 2024-06-29T14:23:58.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95c0dfb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f95bc0b3390 con 0x7f95bc1412f0 2024-06-29T14:23:58.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95ba59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95bc073030 0x7f95bc1572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95ba59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95bc073030 0x7f95bc1572b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36204/0 (socket says 172.21.15.81:36204) 2024-06-29T14:23:58.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95ba59c640 1 -- 172.21.15.81:0/3779574254 learned_addr learned my addr 172.21.15.81:0/3779574254 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:23:58.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95b9d9b640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95bc1412f0 0x7f95bc1f8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95bad9d640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95bc15e960 0x7f95bc0cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95ba59c640 1 -- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95bc1412f0 msgr2=0x7f95bc1f8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.910+0000 7f95ba59c640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95bc1412f0 0x7f95bc1f8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.911+0000 7f95ba59c640 1 -- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95bc15e960 msgr2=0x7f95bc0cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.911+0000 7f95ba59c640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95bc15e960 0x7f95bc0cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.911+0000 7f95ba59c640 1 -- 172.21.15.81:0/3779574254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95bc1f8820 con 0x7f95bc073030 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.911+0000 7f95bad9d640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95bc15e960 0x7f95bc0cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.911+0000 7f95ba59c640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95bc073030 0x7f95bc1572b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00b3f0 tx=0x7f95ac00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.911+0000 7f95aaffd640 1 -- 172.21.15.81:0/3779574254 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac015020 con 0x7f95bc073030 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.911+0000 7f95aaffd640 1 -- 172.21.15.81:0/3779574254 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95ac011920 con 0x7f95bc073030 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.911+0000 7f95aaffd640 1 -- 172.21.15.81:0/3779574254 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac011d70 con 0x7f95bc073030 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.911+0000 7f95c0dfb640 1 -- 172.21.15.81:0/3779574254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95bc157a50 con 0x7f95bc073030 2024-06-29T14:23:58.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.911+0000 7f95c0dfb640 1 -- 172.21.15.81:0/3779574254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95bc1f9520 con 0x7f95bc073030 2024-06-29T14:23:58.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.912+0000 7f95c0dfb640 1 -- 172.21.15.81:0/3779574254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95bc0b38f0 con 0x7f95bc073030 2024-06-29T14:23:58.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.913+0000 7f95aaffd640 1 -- 172.21.15.81:0/3779574254 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 10) ==== 98377+0+0 (secure 0 0 0) 0x7f95ac018020 con 0x7f95bc073030 2024-06-29T14:23:58.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.913+0000 7f95aaffd640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f959c0764d0 0x7f959c0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:23:58.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.913+0000 7f95aaffd640 1 -- 172.21.15.81:0/3779574254 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f95ac0977c0 con 0x7f95bc073030 2024-06-29T14:23:58.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.914+0000 7f95b9d9b640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f959c0764d0 0x7f959c0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:23:58.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.914+0000 7f95b9d9b640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f959c0764d0 0x7f959c0789e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f95b0033000 tx=0x7f95b0004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:23:58.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:58.918+0000 7f95aaffd640 1 -- 172.21.15.81:0/3779574254 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f95ac01f030 con 0x7f95bc073030 2024-06-29T14:23:59.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.123+0000 7f95c0dfb640 1 -- 172.21.15.81:0/3779574254 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f95bc0b0ff0 con 0x7f959c0764d0 2024-06-29T14:23:59.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.198+0000 7f95aaffd640 1 -- 172.21.15.81:0/3779574254 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f95bc0b0ff0 con 0x7f959c0764d0 2024-06-29T14:23:59.196 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/dir9 2024-06-29T14:23:59.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 -- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f959c0764d0 msgr2=0x7f959c0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f959c0764d0 0x7f959c0789e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f95b0033000 tx=0x7f95b0004730 comp rx=0 tx=0).stop 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 -- 172.21.15.81:0/3779574254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95bc073030 msgr2=0x7f95bc1572b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95bc073030 0x7f95bc1572b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00b3f0 tx=0x7f95ac00b8b0 comp rx=0 tx=0).stop 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 -- 172.21.15.81:0/3779574254 shutdown_connections 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f959c0764d0 0x7f959c0789e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95bc15e960 0x7f95bc0cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95bc1412f0 0x7f95bc1f8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 --2- 172.21.15.81:0/3779574254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95bc073030 0x7f95bc1572b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 -- 172.21.15.81:0/3779574254 >> 172.21.15.81:0/3779574254 conn(0x7f95bc06ccc0 msgr2=0x7f95bc0b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 -- 172.21.15.81:0/3779574254 shutdown_connections 2024-06-29T14:23:59.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:23:59.200+0000 7f95c0dfb640 1 -- 172.21.15.81:0/3779574254 wait complete. 2024-06-29T14:23:59.209 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2024-06-29T14:24:00.143 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:24:00.144+0000 7f1d4429e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:24:00.143 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:24:00.144+0000 7f1d4429e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:24:00.143 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:24:00.144+0000 7f1d4429e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:24:00.143 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:24:00.144+0000 7f1d4429e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:24:00.143 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:24:00.144+0000 7f1d4429e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:24:07.089 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:24:07.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:24:37.165 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:24:37.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:25:07.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:25:07.333 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:25:37.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:25:37.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:26:07.526 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:26:07.529 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:26:37.824 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:26:37.828 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:27:07.971 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:27:07.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:27:38.118 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:27:38.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:28:00.676 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:28:00.677+0000 7f1d3c28e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:28:00.676 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:28:00.677+0000 7f1d3c28e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:28:00.676 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:28:00.677+0000 7f1d3c28e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:28:00.676 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:28:00.677+0000 7f1d3c28e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:28:00.676 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:28:00.677+0000 7f1d3c28e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:28:08.248 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:28:08.252 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:28:38.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:28:38.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:29:08.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:29:08.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:29:38.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:29:38.682 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:30:08.852 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:30:08.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:30:39.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:30:39.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:31:09.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:31:09.083 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:31:39.217 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:31:39.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:32:09.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:32:09.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:32:39.497 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:32:39.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:33:09.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:33:09.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:33:39.749 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:33:39.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:34:09.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:34:09.852 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:34:19.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2024-06-29T14:34:19.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.343+0000 7ffadbfa1640 1 -- 172.21.15.81:0/1691624743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffad40b9040 msgr2=0x7ffad40d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.343+0000 7ffadbfa1640 1 --2- 172.21.15.81:0/1691624743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffad40b9040 0x7ffad40d0480 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ffad408e290 tx=0x7ffad002a610 comp rx=0 tx=0).stop 2024-06-29T14:34:19.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.343+0000 7ffadbfa1640 1 -- 172.21.15.81:0/1691624743 shutdown_connections 2024-06-29T14:34:19.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.343+0000 7ffadbfa1640 1 --2- 172.21.15.81:0/1691624743 >> v2:172.21.15.167:3300/0 conn(0x7ffad40b7290 0x7ffad40b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.343+0000 7ffadbfa1640 1 --2- 172.21.15.81:0/1691624743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffad40b5250 0x7ffad40d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.343+0000 7ffadbfa1640 1 --2- 172.21.15.81:0/1691624743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffad40b9040 0x7ffad40d0480 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.343+0000 7ffadbfa1640 1 -- 172.21.15.81:0/1691624743 >> 172.21.15.81:0/1691624743 conn(0x7ffad409cf30 msgr2=0x7ffad40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:19.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.344+0000 7ffadbfa1640 1 -- 172.21.15.81:0/1691624743 shutdown_connections 2024-06-29T14:34:19.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.344+0000 7ffadbfa1640 1 -- 172.21.15.81:0/1691624743 wait complete. 2024-06-29T14:34:19.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.345+0000 7ffadbfa1640 1 Processor -- start 2024-06-29T14:34:19.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.345+0000 7ffadbfa1640 1 -- start start 2024-06-29T14:34:19.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffadbfa1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffad40b5250 0x7ffad40c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffadbfa1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffad40b7290 0x7ffad40a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffadbfa1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffad40b9040 0x7ffad40a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffadbfa1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffad40b7ff0 con 0x7ffad40b7290 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffadbfa1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ffad40b7e70 con 0x7ffad40b5250 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffadbfa1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ffad40b8170 con 0x7ffad40b9040 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffad953c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffad40b7290 0x7ffad40a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffad953c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffad40b7290 0x7ffad40a09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41114/0 (socket says 172.21.15.81:41114) 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffad953c640 1 -- 172.21.15.81:0/3507969192 learned_addr learned my addr 172.21.15.81:0/3507969192 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffad9d3d640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffad40b5250 0x7ffad40c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffada53e640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffad40b9040 0x7ffad40a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffad953c640 1 -- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffad40b9040 msgr2=0x7ffad40a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffad953c640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffad40b9040 0x7ffad40a0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffad953c640 1 -- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffad40b5250 msgr2=0x7ffad40c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffad953c640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffad40b5250 0x7ffad40c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffad953c640 1 -- 172.21.15.81:0/3507969192 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad409e600 con 0x7ffad40b7290 2024-06-29T14:34:19.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffad953c640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffad40b7290 0x7ffad40a09d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ffac400c9a0 tx=0x7ffac400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:19.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.347+0000 7ffaca7fc640 1 -- 172.21.15.81:0/3507969192 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac4007b50 con 0x7ffad40b7290 2024-06-29T14:34:19.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.348+0000 7ffaca7fc640 1 -- 172.21.15.81:0/3507969192 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffac4007de0 con 0x7ffad40b7290 2024-06-29T14:34:19.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.348+0000 7ffadbfa1640 1 -- 172.21.15.81:0/3507969192 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad409e8e0 con 0x7ffad40b7290 2024-06-29T14:34:19.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.348+0000 7ffaca7fc640 1 -- 172.21.15.81:0/3507969192 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac400f6b0 con 0x7ffad40b7290 2024-06-29T14:34:19.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.348+0000 7ffadbfa1640 1 -- 172.21.15.81:0/3507969192 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad409eea0 con 0x7ffad40b7290 2024-06-29T14:34:19.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.348+0000 7ffadbfa1640 1 -- 172.21.15.81:0/3507969192 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad40b76a0 con 0x7ffad40b7290 2024-06-29T14:34:19.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.349+0000 7ffaca7fc640 1 -- 172.21.15.81:0/3507969192 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7ffac400f850 con 0x7ffad40b7290 2024-06-29T14:34:19.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.350+0000 7ffaca7fc640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ffab80764d0 0x7ffab80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.350+0000 7ffaca7fc640 1 -- 172.21.15.81:0/3507969192 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7ffac4098280 con 0x7ffad40b7290 2024-06-29T14:34:19.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.350+0000 7ffad9d3d640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ffab80764d0 0x7ffab80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.351+0000 7ffad9d3d640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ffab80764d0 0x7ffab80789e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffad0033000 tx=0x7ffad0004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:19.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.354+0000 7ffaca7fc640 1 -- 172.21.15.81:0/3507969192 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffac4014030 con 0x7ffad40b7290 2024-06-29T14:34:19.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.557+0000 7ffadbfa1640 1 -- 172.21.15.81:0/3507969192 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7ffad409e390 con 0x7ffab80764d0 2024-06-29T14:34:19.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.634+0000 7ffaca7fc640 1 -- 172.21.15.81:0/3507969192 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7ffad409e390 con 0x7ffab80764d0 2024-06-29T14:34:19.632 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2024-06-29T14:34:19.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.636+0000 7ffadbfa1640 1 -- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ffab80764d0 msgr2=0x7ffab80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.636+0000 7ffadbfa1640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ffab80764d0 0x7ffab80789e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ffad0033000 tx=0x7ffad0004730 comp rx=0 tx=0).stop 2024-06-29T14:34:19.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.636+0000 7ffadbfa1640 1 -- 172.21.15.81:0/3507969192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffad40b7290 msgr2=0x7ffad40a09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.636+0000 7ffadbfa1640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffad40b7290 0x7ffad40a09d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ffac400c9a0 tx=0x7ffac400ce60 comp rx=0 tx=0).stop 2024-06-29T14:34:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.637+0000 7ffadbfa1640 1 -- 172.21.15.81:0/3507969192 shutdown_connections 2024-06-29T14:34:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.637+0000 7ffadbfa1640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ffab80764d0 0x7ffab80789e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.637+0000 7ffadbfa1640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffad40b9040 0x7ffad40a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.637+0000 7ffadbfa1640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffad40b7290 0x7ffad40a09d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.637+0000 7ffadbfa1640 1 --2- 172.21.15.81:0/3507969192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffad40b5250 0x7ffad40c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.637+0000 7ffadbfa1640 1 -- 172.21.15.81:0/3507969192 >> 172.21.15.81:0/3507969192 conn(0x7ffad409cf30 msgr2=0x7ffad415d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.637+0000 7ffadbfa1640 1 -- 172.21.15.81:0/3507969192 shutdown_connections 2024-06-29T14:34:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.637+0000 7ffadbfa1640 1 -- 172.21.15.81:0/3507969192 wait complete. 2024-06-29T14:34:19.645 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2024-06-29T14:34:19.646 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2024-06-29T14:34:19.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.764+0000 7fdbe4490640 1 -- 172.21.15.81:0/2629051683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbc80039e0 msgr2=0x7fdbc8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.764+0000 7fdbe4490640 1 --2- 172.21.15.81:0/2629051683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbc80039e0 0x7fdbc8003e30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdbd800c310 tx=0x7fdbc8005830 comp rx=0 tx=0).stop 2024-06-29T14:34:19.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.764+0000 7fdbe2a2d640 1 -- 172.21.15.81:0/2629051683 reap_dead start 2024-06-29T14:34:19.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.764+0000 7fdbe4490640 1 -- 172.21.15.81:0/2629051683 shutdown_connections 2024-06-29T14:34:19.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.764+0000 7fdbe4490640 1 -- 172.21.15.81:0/2629051683 >> 172.21.15.81:0/2629051683 conn(0x7fdbdc0b2920 msgr2=0x7fdbdc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:19.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.764+0000 7fdbe4490640 1 -- 172.21.15.81:0/2629051683 shutdown_connections 2024-06-29T14:34:19.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.764+0000 7fdbe4490640 1 -- 172.21.15.81:0/2629051683 wait complete. 2024-06-29T14:34:19.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.766+0000 7fdbe4490640 1 Processor -- start 2024-06-29T14:34:19.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.766+0000 7fdbe4490640 1 -- start start 2024-06-29T14:34:19.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe4490640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbc80039e0 0x7fdbdc1462f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe4490640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbdc0ce560 0x7fdbdc0ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe4490640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbdc0cf400 0x7fdbdc0d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe4490640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc163c20 con 0x7fdbdc0cf400 2024-06-29T14:34:19.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe4490640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc163aa0 con 0x7fdbc80039e0 2024-06-29T14:34:19.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe4490640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdbdc163da0 con 0x7fdbdc0ce560 2024-06-29T14:34:19.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe2a2d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbdc0cf400 0x7fdbdc0d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe2a2d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbdc0cf400 0x7fdbdc0d05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41134/0 (socket says 172.21.15.81:41134) 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe2a2d640 1 -- 172.21.15.81:0/1050975748 learned_addr learned my addr 172.21.15.81:0/1050975748 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe1a2b640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbdc0ce560 0x7fdbdc0ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe2a2d640 1 -- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbdc0ce560 msgr2=0x7fdbdc0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe2a2d640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbdc0ce560 0x7fdbdc0ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.767+0000 7fdbe222c640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbc80039e0 0x7fdbdc1462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.768+0000 7fdbe2a2d640 1 -- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbc80039e0 msgr2=0x7fdbdc1462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.768+0000 7fdbe2a2d640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbc80039e0 0x7fdbdc1462f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.768+0000 7fdbe2a2d640 1 -- 172.21.15.81:0/1050975748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbdc0cf810 con 0x7fdbdc0cf400 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.768+0000 7fdbe222c640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbc80039e0 0x7fdbdc1462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.768+0000 7fdbe2a2d640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbdc0cf400 0x7fdbdc0d05a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4005fd0 tx=0x7fdbd400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.768+0000 7fdbd2ffd640 1 -- 172.21.15.81:0/1050975748 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4004b20 con 0x7fdbdc0cf400 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.768+0000 7fdbd2ffd640 1 -- 172.21.15.81:0/1050975748 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdbd4004cc0 con 0x7fdbdc0cf400 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.768+0000 7fdbe4490640 1 -- 172.21.15.81:0/1050975748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbdc148b90 con 0x7fdbdc0cf400 2024-06-29T14:34:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.768+0000 7fdbd2ffd640 1 -- 172.21.15.81:0/1050975748 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4003bd0 con 0x7fdbdc0cf400 2024-06-29T14:34:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.768+0000 7fdbe4490640 1 -- 172.21.15.81:0/1050975748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbdc149110 con 0x7fdbdc0cf400 2024-06-29T14:34:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.769+0000 7fdbe4490640 1 -- 172.21.15.81:0/1050975748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbdc1497d0 con 0x7fdbdc0cf400 2024-06-29T14:34:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.770+0000 7fdbd2ffd640 1 -- 172.21.15.81:0/1050975748 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fdbd4004030 con 0x7fdbdc0cf400 2024-06-29T14:34:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.770+0000 7fdbd2ffd640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fdbc0076500 0x7fdbc0078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.770+0000 7fdbd2ffd640 1 -- 172.21.15.81:0/1050975748 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7fdbd4098070 con 0x7fdbdc0cf400 2024-06-29T14:34:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.770+0000 7fdbe222c640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fdbc0076500 0x7fdbc0078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.771+0000 7fdbe222c640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fdbc0076500 0x7fdbc0078a10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8033000 tx=0x7fdbd802d000 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:19.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.775+0000 7fdbd2ffd640 1 -- 172.21.15.81:0/1050975748 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdbdc1497d0 con 0x7fdbdc0cf400 2024-06-29T14:34:19.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:19.977+0000 7fdbe4490640 1 -- 172.21.15.81:0/1050975748 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fdbdc15b6e0 con 0x7fdbc0076500 2024-06-29T14:34:20.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.059+0000 7fdbd2ffd640 1 -- 172.21.15.81:0/1050975748 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fdbdc15b6e0 con 0x7fdbc0076500 2024-06-29T14:34:20.056 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2024-06-29T14:34:20.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.061+0000 7fdbe4490640 1 -- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fdbc0076500 msgr2=0x7fdbc0078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.061+0000 7fdbe4490640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fdbc0076500 0x7fdbc0078a10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8033000 tx=0x7fdbd802d000 comp rx=0 tx=0).stop 2024-06-29T14:34:20.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.061+0000 7fdbe4490640 1 -- 172.21.15.81:0/1050975748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbdc0cf400 msgr2=0x7fdbdc0d05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.061+0000 7fdbe4490640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbdc0cf400 0x7fdbdc0d05a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4005fd0 tx=0x7fdbd400f3c0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.062+0000 7fdbe4490640 1 -- 172.21.15.81:0/1050975748 shutdown_connections 2024-06-29T14:34:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.062+0000 7fdbe4490640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fdbc0076500 0x7fdbc0078a10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.062+0000 7fdbe4490640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbdc0cf400 0x7fdbdc0d05a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.062+0000 7fdbe4490640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbdc0ce560 0x7fdbdc0ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.062+0000 7fdbe4490640 1 --2- 172.21.15.81:0/1050975748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbc80039e0 0x7fdbdc1462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.062+0000 7fdbe4490640 1 -- 172.21.15.81:0/1050975748 >> 172.21.15.81:0/1050975748 conn(0x7fdbdc0b2920 msgr2=0x7fdbdc14fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.062+0000 7fdbe4490640 1 -- 172.21.15.81:0/1050975748 shutdown_connections 2024-06-29T14:34:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.062+0000 7fdbe4490640 1 -- 172.21.15.81:0/1050975748 wait complete. 2024-06-29T14:34:20.070 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2024-06-29T14:34:20.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2024-06-29T14:34:20.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.188+0000 7fa1cfc65640 1 -- 172.21.15.81:0/679633750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c80b3960 msgr2=0x7fa1c80b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.188+0000 7fa1cfc65640 1 --2- 172.21.15.81:0/679633750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c80b3960 0x7fa1c80b3d70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8070960 tx=0x7fa1c002aac0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.188+0000 7fa1cfc65640 1 -- 172.21.15.81:0/679633750 shutdown_connections 2024-06-29T14:34:20.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.188+0000 7fa1cfc65640 1 --2- 172.21.15.81:0/679633750 >> v2:172.21.15.167:3300/0 conn(0x7fa1c806f6a0 0x7fa1c815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.188+0000 7fa1cfc65640 1 --2- 172.21.15.81:0/679633750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c80b3960 0x7fa1c80b3d70 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.189+0000 7fa1cfc65640 1 -- 172.21.15.81:0/679633750 >> 172.21.15.81:0/679633750 conn(0x7fa1c80d0930 msgr2=0x7fa1c80d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:20.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.189+0000 7fa1cfc65640 1 -- 172.21.15.81:0/679633750 shutdown_connections 2024-06-29T14:34:20.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.189+0000 7fa1cfc65640 1 -- 172.21.15.81:0/679633750 wait complete. 2024-06-29T14:34:20.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.190+0000 7fa1cfc65640 1 Processor -- start 2024-06-29T14:34:20.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.190+0000 7fa1cfc65640 1 -- start start 2024-06-29T14:34:20.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cfc65640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1c806f6a0 0x7fa1c814e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cfc65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c80b3960 0x7fa1c8150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cfc65640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1c80b57b0 0x7fa1c81e1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cfc65640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c80b4430 con 0x7fa1c80b3960 2024-06-29T14:34:20.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cfc65640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c80b42b0 con 0x7fa1c80b57b0 2024-06-29T14:34:20.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cfc65640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c80b45b0 con 0x7fa1c806f6a0 2024-06-29T14:34:20.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c80b3960 0x7fa1c8150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c80b3960 0x7fa1c8150670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42540/0 (socket says 172.21.15.81:42540) 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cd200640 1 -- 172.21.15.81:0/3957642205 learned_addr learned my addr 172.21.15.81:0/3957642205 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cd200640 1 -- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1c806f6a0 msgr2=0x7fa1c814e6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cd200640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1c806f6a0 0x7fa1c814e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cd200640 1 -- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1c80b57b0 msgr2=0x7fa1c81e1710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cd200640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1c80b57b0 0x7fa1c81e1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cd200640 1 -- 172.21.15.81:0/3957642205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c81e0670 con 0x7fa1c80b3960 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cd200640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c80b3960 0x7fa1c8150670 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800e7f0 tx=0x7fa1b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1be7fc640 1 -- 172.21.15.81:0/3957642205 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b800ccc0 con 0x7fa1c80b3960 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1be7fc640 1 -- 172.21.15.81:0/3957642205 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1b800ce80 con 0x7fa1c80b3960 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cfc65640 1 -- 172.21.15.81:0/3957642205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c81e0950 con 0x7fa1c80b3960 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1be7fc640 1 -- 172.21.15.81:0/3957642205 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8005280 con 0x7fa1c80b3960 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.192+0000 7fa1cfc65640 1 -- 172.21.15.81:0/3957642205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c81e0f10 con 0x7fa1c80b3960 2024-06-29T14:34:20.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.193+0000 7fa1cfc65640 1 -- 172.21.15.81:0/3957642205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c806e2a0 con 0x7fa1c80b3960 2024-06-29T14:34:20.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.194+0000 7fa1be7fc640 1 -- 172.21.15.81:0/3957642205 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fa1b8005420 con 0x7fa1c80b3960 2024-06-29T14:34:20.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.195+0000 7fa1be7fc640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa1ac0764d0 0x7fa1ac0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.195+0000 7fa1be7fc640 1 -- 172.21.15.81:0/3957642205 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7fa1b8098700 con 0x7fa1c80b3960 2024-06-29T14:34:20.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.195+0000 7fa1cda01640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa1ac0764d0 0x7fa1ac0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.196+0000 7fa1cda01640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa1ac0764d0 0x7fa1ac0789e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4001480 tx=0x7fa1c40079b0 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:20.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.199+0000 7fa1be7fc640 1 -- 172.21.15.81:0/3957642205 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa1c806e2a0 con 0x7fa1c80b3960 2024-06-29T14:34:20.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.404+0000 7fa1cfc65640 1 -- 172.21.15.81:0/3957642205 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7fa1c814ec00 con 0x7fa1ac0764d0 2024-06-29T14:34:20.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.488+0000 7fa1be7fc640 1 -- 172.21.15.81:0/3957642205 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fa1c4005080 con 0x7fa1ac0764d0 2024-06-29T14:34:20.486 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2024-06-29T14:34:20.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 -- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa1ac0764d0 msgr2=0x7fa1ac0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa1ac0764d0 0x7fa1ac0789e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4001480 tx=0x7fa1c40079b0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 -- 172.21.15.81:0/3957642205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c80b3960 msgr2=0x7fa1c8150670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c80b3960 0x7fa1c8150670 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800e7f0 tx=0x7fa1b800ecb0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 -- 172.21.15.81:0/3957642205 shutdown_connections 2024-06-29T14:34:20.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fa1ac0764d0 0x7fa1ac0789e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1c80b57b0 0x7fa1c81e1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c80b3960 0x7fa1c8150670 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 --2- 172.21.15.81:0/3957642205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1c806f6a0 0x7fa1c814e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 -- 172.21.15.81:0/3957642205 >> 172.21.15.81:0/3957642205 conn(0x7fa1c80d0930 msgr2=0x7fa1c806ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:20.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 -- 172.21.15.81:0/3957642205 shutdown_connections 2024-06-29T14:34:20.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.490+0000 7fa1cfc65640 1 -- 172.21.15.81:0/3957642205 wait complete. 2024-06-29T14:34:20.499 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2024-06-29T14:34:20.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2024-06-29T14:34:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.617+0000 7f73b27f5640 1 -- 172.21.15.81:0/3377540270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac0b57b0 msgr2=0x7f73ac15aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.617+0000 7f73b27f5640 1 --2- 172.21.15.81:0/3377540270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac0b57b0 0x7f73ac15aed0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f73ac070960 tx=0x7f739c02a560 comp rx=0 tx=0).stop 2024-06-29T14:34:20.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.617+0000 7f73b27f5640 1 -- 172.21.15.81:0/3377540270 shutdown_connections 2024-06-29T14:34:20.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.617+0000 7f73b27f5640 1 --2- 172.21.15.81:0/3377540270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73ac0ce1d0 0x7f73ac0ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.617+0000 7f73b27f5640 1 --2- 172.21.15.81:0/3377540270 >> v2:172.21.15.167:3300/0 conn(0x7f73ac06ef90 0x7f73ac06f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.617+0000 7f73b27f5640 1 --2- 172.21.15.81:0/3377540270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac0b57b0 0x7f73ac15aed0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.617+0000 7f73b27f5640 1 -- 172.21.15.81:0/3377540270 >> 172.21.15.81:0/3377540270 conn(0x7f73ac0d0930 msgr2=0x7f73ac0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:20.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.617+0000 7f73b27f5640 1 -- 172.21.15.81:0/3377540270 shutdown_connections 2024-06-29T14:34:20.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.617+0000 7f73b27f5640 1 -- 172.21.15.81:0/3377540270 wait complete. 2024-06-29T14:34:20.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.619+0000 7f73b27f5640 1 Processor -- start 2024-06-29T14:34:20.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.619+0000 7f73b27f5640 1 -- start start 2024-06-29T14:34:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73b27f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac06ef90 0x7f73ac152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73b27f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73ac0b57b0 0x7f73ac1e3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73b27f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73ac0ce1d0 0x7f73ac155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73b27f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac0ceeb0 con 0x7f73ac06ef90 2024-06-29T14:34:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73b27f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac0ced30 con 0x7f73ac0ce1d0 2024-06-29T14:34:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73b27f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f73ac0cf030 con 0x7f73ac0b57b0 2024-06-29T14:34:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73abfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac06ef90 0x7f73ac152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73abfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac06ef90 0x7f73ac152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42564/0 (socket says 172.21.15.81:42564) 2024-06-29T14:34:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73abfff640 1 -- 172.21.15.81:0/1301309997 learned_addr learned my addr 172.21.15.81:0/1301309997 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73ab7fe640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73ac0b57b0 0x7f73ac1e3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73b0d92640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73ac0ce1d0 0x7f73ac155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73abfff640 1 -- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73ac0b57b0 msgr2=0x7f73ac1e3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73abfff640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73ac0b57b0 0x7f73ac1e3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73abfff640 1 -- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73ac0ce1d0 msgr2=0x7f73ac155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73abfff640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73ac0ce1d0 0x7f73ac155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73abfff640 1 -- 172.21.15.81:0/1301309997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac1f3500 con 0x7f73ac06ef90 2024-06-29T14:34:20.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73b0d92640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73ac0ce1d0 0x7f73ac155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:20.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73abfff640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac06ef90 0x7f73ac152470 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f739c034040 tx=0x7f739c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:20.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73a8ff9640 1 -- 172.21.15.81:0/1301309997 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c03e030 con 0x7f73ac06ef90 2024-06-29T14:34:20.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73a8ff9640 1 -- 172.21.15.81:0/1301309997 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f739c0047a0 con 0x7f73ac06ef90 2024-06-29T14:34:20.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73b27f5640 1 -- 172.21.15.81:0/1301309997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ac1f37e0 con 0x7f73ac06ef90 2024-06-29T14:34:20.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73a8ff9640 1 -- 172.21.15.81:0/1301309997 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c035040 con 0x7f73ac06ef90 2024-06-29T14:34:20.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.621+0000 7f73b27f5640 1 -- 172.21.15.81:0/1301309997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ac1f3da0 con 0x7f73ac06ef90 2024-06-29T14:34:20.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.622+0000 7f73b27f5640 1 -- 172.21.15.81:0/1301309997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73ac06e2a0 con 0x7f73ac06ef90 2024-06-29T14:34:20.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.623+0000 7f73a8ff9640 1 -- 172.21.15.81:0/1301309997 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f739c041030 con 0x7f73ac06ef90 2024-06-29T14:34:20.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.624+0000 7f73a8ff9640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f738c076500 0x7f738c078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:20.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.624+0000 7f73a8ff9640 1 -- 172.21.15.81:0/1301309997 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f739c0b8410 con 0x7f73ac06ef90 2024-06-29T14:34:20.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.624+0000 7f73ab7fe640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f738c076500 0x7f738c078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:20.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.625+0000 7f73ab7fe640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f738c076500 0x7f738c078a10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f73ac1e2930 tx=0x7f7394009060 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:20.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.628+0000 7f73a8ff9640 1 -- 172.21.15.81:0/1301309997 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f73ac06e2a0 con 0x7f73ac06ef90 2024-06-29T14:34:20.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.831+0000 7f73b27f5640 1 -- 172.21.15.81:0/1301309997 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f73ac15aed0 con 0x7f738c076500 2024-06-29T14:34:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.918+0000 7f73a8ff9640 1 -- 172.21.15.81:0/1301309997 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f73ac15aed0 con 0x7f738c076500 2024-06-29T14:34:20.916 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2024-06-29T14:34:20.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 -- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f738c076500 msgr2=0x7f738c078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f738c076500 0x7f738c078a10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f73ac1e2930 tx=0x7f7394009060 comp rx=0 tx=0).stop 2024-06-29T14:34:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 -- 172.21.15.81:0/1301309997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac06ef90 msgr2=0x7f73ac152470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac06ef90 0x7f73ac152470 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f739c034040 tx=0x7f739c0039e0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 -- 172.21.15.81:0/1301309997 shutdown_connections 2024-06-29T14:34:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f738c076500 0x7f738c078a10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73ac0ce1d0 0x7f73ac155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73ac0b57b0 0x7f73ac1e3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 --2- 172.21.15.81:0/1301309997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac06ef90 0x7f73ac152470 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 -- 172.21.15.81:0/1301309997 >> 172.21.15.81:0/1301309997 conn(0x7f73ac0d0930 msgr2=0x7f73ac159500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 -- 172.21.15.81:0/1301309997 shutdown_connections 2024-06-29T14:34:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:20.920+0000 7f73b27f5640 1 -- 172.21.15.81:0/1301309997 wait complete. 2024-06-29T14:34:20.929 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2024-06-29T14:34:20.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2024-06-29T14:34:21.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.047+0000 7f43d15f5640 1 -- 172.21.15.81:0/2193155969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43cc06c2f0 msgr2=0x7f43cc06c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.047+0000 7f43d15f5640 1 --2- 172.21.15.81:0/2193155969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43cc06c2f0 0x7f43cc06c700 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f43cc072e80 tx=0x7f43c002a530 comp rx=0 tx=0).stop 2024-06-29T14:34:21.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.047+0000 7f43d15f5640 1 -- 172.21.15.81:0/2193155969 shutdown_connections 2024-06-29T14:34:21.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.047+0000 7f43d15f5640 1 --2- 172.21.15.81:0/2193155969 >> v2:172.21.15.167:3300/0 conn(0x7f43cc06f9a0 0x7f43cc06fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.047+0000 7f43d15f5640 1 --2- 172.21.15.81:0/2193155969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43cc06c2f0 0x7f43cc06c700 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.047+0000 7f43d15f5640 1 -- 172.21.15.81:0/2193155969 >> 172.21.15.81:0/2193155969 conn(0x7f43cc0b2920 msgr2=0x7f43cc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:21.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.047+0000 7f43d15f5640 1 -- 172.21.15.81:0/2193155969 shutdown_connections 2024-06-29T14:34:21.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.047+0000 7f43d15f5640 1 -- 172.21.15.81:0/2193155969 wait complete. 2024-06-29T14:34:21.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.049+0000 7f43d15f5640 1 Processor -- start 2024-06-29T14:34:21.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.049+0000 7f43d15f5640 1 -- start start 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.050+0000 7f43d15f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43cc06c2f0 0x7f43cc0afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.050+0000 7f43d15f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43cc06f9a0 0x7f43cc0acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.050+0000 7f43d15f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43cc159bd0 0x7f43cc0b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.050+0000 7f43d15f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f43cc15fc10 con 0x7f43cc159bd0 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.050+0000 7f43d15f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f43cc15fa90 con 0x7f43cc06c2f0 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.050+0000 7f43d15f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f43cc15fd90 con 0x7f43cc06f9a0 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.050+0000 7f43cb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43cc159bd0 0x7f43cc0b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.050+0000 7f43cb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43cc159bd0 0x7f43cc0b0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42580/0 (socket says 172.21.15.81:42580) 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.050+0000 7f43cb7fe640 1 -- 172.21.15.81:0/528550432 learned_addr learned my addr 172.21.15.81:0/528550432 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:21.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43caffd640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43cc06c2f0 0x7f43cc0afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43ca7fc640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43cc06f9a0 0x7f43cc0acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43cb7fe640 1 -- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43cc06f9a0 msgr2=0x7f43cc0acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43cb7fe640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43cc06f9a0 0x7f43cc0acaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43cb7fe640 1 -- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43cc06c2f0 msgr2=0x7f43cc0afb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43cb7fe640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43cc06c2f0 0x7f43cc0afb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43cb7fe640 1 -- 172.21.15.81:0/528550432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43cc0ad030 con 0x7f43cc159bd0 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43cb7fe640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43cc159bd0 0x7f43cc0b0d70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f43c400d9a0 tx=0x7f43c400de60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43b37fe640 1 -- 172.21.15.81:0/528550432 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4017070 con 0x7f43cc159bd0 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43b37fe640 1 -- 172.21.15.81:0/528550432 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43c400be00 con 0x7f43cc159bd0 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43d15f5640 1 -- 172.21.15.81:0/528550432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43cc1f6ae0 con 0x7f43cc159bd0 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43b37fe640 1 -- 172.21.15.81:0/528550432 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c40106c0 con 0x7f43cc159bd0 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.051+0000 7f43d15f5640 1 -- 172.21.15.81:0/528550432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43cc1f70a0 con 0x7f43cc159bd0 2024-06-29T14:34:21.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.052+0000 7f43d15f5640 1 -- 172.21.15.81:0/528550432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43cc1601b0 con 0x7f43cc159bd0 2024-06-29T14:34:21.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.053+0000 7f43b37fe640 1 -- 172.21.15.81:0/528550432 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f43c4010860 con 0x7f43cc159bd0 2024-06-29T14:34:21.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.053+0000 7f43b37fe640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43b80764d0 0x7f43b80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.053+0000 7f43b37fe640 1 -- 172.21.15.81:0/528550432 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f43c40981b0 con 0x7f43cc159bd0 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.053+0000 7f43caffd640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43b80764d0 0x7f43b80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.054+0000 7f43caffd640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43b80764d0 0x7f43b80789e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43bc000f60 tx=0x7f43bc009040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.058+0000 7f43b37fe640 1 -- 172.21.15.81:0/528550432 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f43c40610e0 con 0x7f43cc159bd0 2024-06-29T14:34:21.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.261+0000 7f43d15f5640 1 -- 172.21.15.81:0/528550432 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f43cc1f6890 con 0x7f43b80764d0 2024-06-29T14:34:21.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.343+0000 7f43b37fe640 1 -- 172.21.15.81:0/528550432 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f43cc1f6890 con 0x7f43b80764d0 2024-06-29T14:34:21.341 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2024-06-29T14:34:21.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 -- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43b80764d0 msgr2=0x7f43b80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43b80764d0 0x7f43b80789e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43bc000f60 tx=0x7f43bc009040 comp rx=0 tx=0).stop 2024-06-29T14:34:21.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 -- 172.21.15.81:0/528550432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43cc159bd0 msgr2=0x7f43cc0b0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43cc159bd0 0x7f43cc0b0d70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f43c400d9a0 tx=0x7f43c400de60 comp rx=0 tx=0).stop 2024-06-29T14:34:21.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 -- 172.21.15.81:0/528550432 shutdown_connections 2024-06-29T14:34:21.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43b80764d0 0x7f43b80789e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43cc159bd0 0x7f43cc0b0d70 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43cc06f9a0 0x7f43cc0acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 --2- 172.21.15.81:0/528550432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43cc06c2f0 0x7f43cc0afb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 -- 172.21.15.81:0/528550432 >> 172.21.15.81:0/528550432 conn(0x7f43cc0b2920 msgr2=0x7f43cc0d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:21.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 -- 172.21.15.81:0/528550432 shutdown_connections 2024-06-29T14:34:21.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.345+0000 7f43d15f5640 1 -- 172.21.15.81:0/528550432 wait complete. 2024-06-29T14:34:21.354 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2024-06-29T14:34:21.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2024-06-29T14:34:21.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.473+0000 7f90d759a640 1 -- 172.21.15.81:0/2388544774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d00b35b0 msgr2=0x7f90d015b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.473+0000 7f90d759a640 1 --2- 172.21.15.81:0/2388544774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d00b35b0 0x7f90d015b5b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f90d00cc260 tx=0x7f90cc02a4d0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.473+0000 7f90d759a640 1 -- 172.21.15.81:0/2388544774 shutdown_connections 2024-06-29T14:34:21.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.473+0000 7f90d759a640 1 --2- 172.21.15.81:0/2388544774 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90d006f660 0x7f90d006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.473+0000 7f90d759a640 1 --2- 172.21.15.81:0/2388544774 >> v2:172.21.15.167:3300/0 conn(0x7f90d00b4430 0x7f90d00cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.473+0000 7f90d759a640 1 --2- 172.21.15.81:0/2388544774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d00b35b0 0x7f90d015b5b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.473+0000 7f90d759a640 1 -- 172.21.15.81:0/2388544774 >> 172.21.15.81:0/2388544774 conn(0x7f90d006ccc0 msgr2=0x7f90d00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:21.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.473+0000 7f90d759a640 1 -- 172.21.15.81:0/2388544774 shutdown_connections 2024-06-29T14:34:21.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.473+0000 7f90d759a640 1 -- 172.21.15.81:0/2388544774 wait complete. 2024-06-29T14:34:21.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.474+0000 7f90d759a640 1 Processor -- start 2024-06-29T14:34:21.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.475+0000 7f90d759a640 1 -- start start 2024-06-29T14:34:21.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d759a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90d006f660 0x7f90d014aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d759a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d00b35b0 0x7f90d0151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d759a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90d00b4430 0x7f90d0142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d759a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f90d006f160 con 0x7f90d00b35b0 2024-06-29T14:34:21.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d759a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f90d006efe0 con 0x7f90d006f660 2024-06-29T14:34:21.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d759a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f90d006f2e0 con 0x7f90d00b4430 2024-06-29T14:34:21.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d4b35640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d00b35b0 0x7f90d0151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d4b35640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d00b35b0 0x7f90d0151580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42592/0 (socket says 172.21.15.81:42592) 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d4b35640 1 -- 172.21.15.81:0/4082141 learned_addr learned my addr 172.21.15.81:0/4082141 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d5336640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90d006f660 0x7f90d014aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d5b37640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90d00b4430 0x7f90d0142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d4b35640 1 -- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90d00b4430 msgr2=0x7f90d0142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d4b35640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90d00b4430 0x7f90d0142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d4b35640 1 -- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90d006f660 msgr2=0x7f90d014aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d4b35640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90d006f660 0x7f90d014aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d4b35640 1 -- 172.21.15.81:0/4082141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d00af5a0 con 0x7f90d00b35b0 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d5b37640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90d00b4430 0x7f90d0142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.476+0000 7f90d4b35640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d00b35b0 0x7f90d0151580 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f90c000d700 tx=0x7f90c000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.477+0000 7f90c5ffb640 1 -- 172.21.15.81:0/4082141 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c0017070 con 0x7f90d00b35b0 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.477+0000 7f90c5ffb640 1 -- 172.21.15.81:0/4082141 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90c0009b10 con 0x7f90d00b35b0 2024-06-29T14:34:21.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.477+0000 7f90d759a640 1 -- 172.21.15.81:0/4082141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90d0150560 con 0x7f90d00b35b0 2024-06-29T14:34:21.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.477+0000 7f90c5ffb640 1 -- 172.21.15.81:0/4082141 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c0010460 con 0x7f90d00b35b0 2024-06-29T14:34:21.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.477+0000 7f90d759a640 1 -- 172.21.15.81:0/4082141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90d00af860 con 0x7f90d00b35b0 2024-06-29T14:34:21.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.478+0000 7f90d759a640 1 -- 172.21.15.81:0/4082141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90d015b5b0 con 0x7f90d00b35b0 2024-06-29T14:34:21.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.478+0000 7f90c5ffb640 1 -- 172.21.15.81:0/4082141 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f90c001a070 con 0x7f90d00b35b0 2024-06-29T14:34:21.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.479+0000 7f90c5ffb640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f90b40764d0 0x7f90b40789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.479+0000 7f90c5ffb640 1 -- 172.21.15.81:0/4082141 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f90c0097de0 con 0x7f90d00b35b0 2024-06-29T14:34:21.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.479+0000 7f90d5336640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f90b40764d0 0x7f90b40789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.480+0000 7f90d5336640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f90b40764d0 0x7f90b40789e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f90cc033000 tx=0x7f90cc004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.483+0000 7f90c5ffb640 1 -- 172.21.15.81:0/4082141 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f90c0060bd0 con 0x7f90d00b35b0 2024-06-29T14:34:21.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.685+0000 7f90d759a640 1 -- 172.21.15.81:0/4082141 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f90d014b8e0 con 0x7f90b40764d0 2024-06-29T14:34:21.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.763+0000 7f90c5ffb640 1 -- 172.21.15.81:0/4082141 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f90d014b8e0 con 0x7f90b40764d0 2024-06-29T14:34:21.761 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2024-06-29T14:34:21.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 -- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f90b40764d0 msgr2=0x7f90b40789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f90b40764d0 0x7f90b40789e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f90cc033000 tx=0x7f90cc004730 comp rx=0 tx=0).stop 2024-06-29T14:34:21.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 -- 172.21.15.81:0/4082141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d00b35b0 msgr2=0x7f90d0151580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d00b35b0 0x7f90d0151580 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f90c000d700 tx=0x7f90c000dbc0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 -- 172.21.15.81:0/4082141 shutdown_connections 2024-06-29T14:34:21.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f90b40764d0 0x7f90b40789e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90d00b4430 0x7f90d0142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d00b35b0 0x7f90d0151580 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 --2- 172.21.15.81:0/4082141 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90d006f660 0x7f90d014aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 -- 172.21.15.81:0/4082141 >> 172.21.15.81:0/4082141 conn(0x7f90d006ccc0 msgr2=0x7f90d014e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:21.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 -- 172.21.15.81:0/4082141 shutdown_connections 2024-06-29T14:34:21.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.766+0000 7f90d759a640 1 -- 172.21.15.81:0/4082141 wait complete. 2024-06-29T14:34:21.774 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2024-06-29T14:34:21.774 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2024-06-29T14:34:21.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.894+0000 7f46b128f640 1 -- 172.21.15.81:0/3150946627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46ac13ed60 msgr2=0x7f46ac06c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.894+0000 7f46b128f640 1 --2- 172.21.15.81:0/3150946627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46ac13ed60 0x7f46ac06c2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f469c005fd0 tx=0x7f469c02aa90 comp rx=0 tx=0).stop 2024-06-29T14:34:21.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.894+0000 7f46b128f640 1 -- 172.21.15.81:0/3150946627 shutdown_connections 2024-06-29T14:34:21.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.894+0000 7f46b128f640 1 --2- 172.21.15.81:0/3150946627 >> v2:172.21.15.167:3300/0 conn(0x7f46ac160360 0x7f46ac0cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.894+0000 7f46b128f640 1 --2- 172.21.15.81:0/3150946627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46ac13ed60 0x7f46ac06c2f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.894+0000 7f46b128f640 1 -- 172.21.15.81:0/3150946627 >> 172.21.15.81:0/3150946627 conn(0x7f46ac0d0b70 msgr2=0x7f46ac0d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:21.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.894+0000 7f46b128f640 1 -- 172.21.15.81:0/3150946627 shutdown_connections 2024-06-29T14:34:21.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.894+0000 7f46b128f640 1 -- 172.21.15.81:0/3150946627 wait complete. 2024-06-29T14:34:21.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.896+0000 7f46b128f640 1 Processor -- start 2024-06-29T14:34:21.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.896+0000 7f46b128f640 1 -- start start 2024-06-29T14:34:21.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46b128f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46ac06c830 0x7f46ac139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46b128f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ac13ed60 0x7f46ac0707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46b128f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46ac160360 0x7f46ac070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46b128f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac0cfb70 con 0x7f46ac13ed60 2024-06-29T14:34:21.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46b128f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac0cf9f0 con 0x7f46ac160360 2024-06-29T14:34:21.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46b128f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f46ac0cfcf0 con 0x7f46ac06c830 2024-06-29T14:34:21.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46aa7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ac13ed60 0x7f46ac0707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46aa7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ac13ed60 0x7f46ac0707a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42598/0 (socket says 172.21.15.81:42598) 2024-06-29T14:34:21.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46aa7fc640 1 -- 172.21.15.81:0/481042841 learned_addr learned my addr 172.21.15.81:0/481042841 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:21.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46aaffd640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46ac06c830 0x7f46ac139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46ab7fe640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46ac160360 0x7f46ac070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46aa7fc640 1 -- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46ac06c830 msgr2=0x7f46ac139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46aa7fc640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46ac06c830 0x7f46ac139ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46aa7fc640 1 -- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46ac160360 msgr2=0x7f46ac070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46aa7fc640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46ac160360 0x7f46ac070ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46aa7fc640 1 -- 172.21.15.81:0/481042841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ac15a8f0 con 0x7f46ac13ed60 2024-06-29T14:34:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46aaffd640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46ac06c830 0x7f46ac139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.897+0000 7f46ab7fe640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46ac160360 0x7f46ac070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:21.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.898+0000 7f46aa7fc640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ac13ed60 0x7f46ac0707a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f46a000c400 tx=0x7f46a000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.898+0000 7f46937fe640 1 -- 172.21.15.81:0/481042841 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a0017070 con 0x7f46ac13ed60 2024-06-29T14:34:21.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.898+0000 7f46937fe640 1 -- 172.21.15.81:0/481042841 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46a00053c0 con 0x7f46ac13ed60 2024-06-29T14:34:21.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.898+0000 7f46b128f640 1 -- 172.21.15.81:0/481042841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46ac15ab90 con 0x7f46ac13ed60 2024-06-29T14:34:21.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.898+0000 7f46937fe640 1 -- 172.21.15.81:0/481042841 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a0003850 con 0x7f46ac13ed60 2024-06-29T14:34:21.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.898+0000 7f46b128f640 1 -- 172.21.15.81:0/481042841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46ac141150 con 0x7f46ac13ed60 2024-06-29T14:34:21.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.899+0000 7f46b128f640 1 -- 172.21.15.81:0/481042841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46ac15c040 con 0x7f46ac13ed60 2024-06-29T14:34:21.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.900+0000 7f46937fe640 1 -- 172.21.15.81:0/481042841 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f46a001a070 con 0x7f46ac13ed60 2024-06-29T14:34:21.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.900+0000 7f46937fe640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4698076500 0x7f4698078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:21.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.900+0000 7f46937fe640 1 -- 172.21.15.81:0/481042841 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f46a0096f40 con 0x7f46ac13ed60 2024-06-29T14:34:21.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.901+0000 7f46aaffd640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4698076500 0x7f4698078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:21.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.901+0000 7f46aaffd640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4698076500 0x7f4698078a10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f469c035000 tx=0x7f469c038040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:21.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:21.904+0000 7f46937fe640 1 -- 172.21.15.81:0/481042841 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f46ac06cc40 con 0x7f46ac13ed60 2024-06-29T14:34:22.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.109+0000 7f46b128f640 1 -- 172.21.15.81:0/481042841 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f46ac13a400 con 0x7f4698076500 2024-06-29T14:34:22.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.198+0000 7f46937fe640 1 -- 172.21.15.81:0/481042841 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f469c02e6d0 con 0x7f4698076500 2024-06-29T14:34:22.195 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2024-06-29T14:34:22.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 -- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4698076500 msgr2=0x7f4698078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4698076500 0x7f4698078a10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f469c035000 tx=0x7f469c038040 comp rx=0 tx=0).stop 2024-06-29T14:34:22.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 -- 172.21.15.81:0/481042841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ac13ed60 msgr2=0x7f46ac0707a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ac13ed60 0x7f46ac0707a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f46a000c400 tx=0x7f46a000c800 comp rx=0 tx=0).stop 2024-06-29T14:34:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 -- 172.21.15.81:0/481042841 shutdown_connections 2024-06-29T14:34:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4698076500 0x7f4698078a10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46ac160360 0x7f46ac070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ac13ed60 0x7f46ac0707a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 --2- 172.21.15.81:0/481042841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46ac06c830 0x7f46ac139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 -- 172.21.15.81:0/481042841 >> 172.21.15.81:0/481042841 conn(0x7f46ac0d0b70 msgr2=0x7f46ac15fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 -- 172.21.15.81:0/481042841 shutdown_connections 2024-06-29T14:34:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.200+0000 7f46b128f640 1 -- 172.21.15.81:0/481042841 wait complete. 2024-06-29T14:34:22.208 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2024-06-29T14:34:22.208 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2024-06-29T14:34:22.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.328+0000 7f20877a5640 1 -- 172.21.15.81:0/4252897149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f206c004d40 msgr2=0x7f206c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.328+0000 7f20877a5640 1 --2- 172.21.15.81:0/4252897149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f206c004d40 0x7f206c005150 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2078009ed0 tx=0x7f206c005830 comp rx=0 tx=0).stop 2024-06-29T14:34:22.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.328+0000 7f2085d42640 1 -- 172.21.15.81:0/4252897149 reap_dead start 2024-06-29T14:34:22.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.328+0000 7f20877a5640 1 -- 172.21.15.81:0/4252897149 shutdown_connections 2024-06-29T14:34:22.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.328+0000 7f20877a5640 1 -- 172.21.15.81:0/4252897149 >> 172.21.15.81:0/4252897149 conn(0x7f20800b3680 msgr2=0x7f20800b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:22.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.328+0000 7f20877a5640 1 -- 172.21.15.81:0/4252897149 shutdown_connections 2024-06-29T14:34:22.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.328+0000 7f20877a5640 1 -- 172.21.15.81:0/4252897149 wait complete. 2024-06-29T14:34:22.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.330+0000 7f20877a5640 1 Processor -- start 2024-06-29T14:34:22.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.330+0000 7f20877a5640 1 -- start start 2024-06-29T14:34:22.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.331+0000 7f20877a5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f206c004d40 0x7f20801e0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.331+0000 7f20877a5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20800b17f0 0x7f20800b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.331+0000 7f20877a5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20801e68f0 0x7f20801e6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.331+0000 7f20877a5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f20800d09d0 con 0x7f20801e68f0 2024-06-29T14:34:22.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.331+0000 7f20877a5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f20800d0850 con 0x7f206c004d40 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.331+0000 7f20877a5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f20800b5890 con 0x7f20800b17f0 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085d42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20801e68f0 0x7f20801e6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085d42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20801e68f0 0x7f20801e6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42616/0 (socket says 172.21.15.81:42616) 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085d42640 1 -- 172.21.15.81:0/2599273424 learned_addr learned my addr 172.21.15.81:0/2599273424 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2084d40640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20800b17f0 0x7f20800b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085541640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f206c004d40 0x7f20801e0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085d42640 1 -- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20800b17f0 msgr2=0x7f20800b1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085d42640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20800b17f0 0x7f20800b1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085d42640 1 -- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f206c004d40 msgr2=0x7f20801e0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085d42640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f206c004d40 0x7f20801e0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085d42640 1 -- 172.21.15.81:0/2599273424 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20801e7240 con 0x7f20801e68f0 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2084d40640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20800b17f0 0x7f20800b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085541640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f206c004d40 0x7f20801e0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:22.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2085d42640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20801e68f0 0x7f20801e6d00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f207802af20 tx=0x7f2078033de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:22.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2075ffb640 1 -- 172.21.15.81:0/2599273424 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078038040 con 0x7f20801e68f0 2024-06-29T14:34:22.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2075ffb640 1 -- 172.21.15.81:0/2599273424 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2078003c60 con 0x7f20801e68f0 2024-06-29T14:34:22.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f2075ffb640 1 -- 172.21.15.81:0/2599273424 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078009ad0 con 0x7f20801e68f0 2024-06-29T14:34:22.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f20877a5640 1 -- 172.21.15.81:0/2599273424 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20800b2ac0 con 0x7f20801e68f0 2024-06-29T14:34:22.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.332+0000 7f20877a5640 1 -- 172.21.15.81:0/2599273424 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20800b27e0 con 0x7f20801e68f0 2024-06-29T14:34:22.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.333+0000 7f20877a5640 1 -- 172.21.15.81:0/2599273424 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20800b5350 con 0x7f20801e68f0 2024-06-29T14:34:22.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.334+0000 7f2075ffb640 1 -- 172.21.15.81:0/2599273424 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f2078004290 con 0x7f20801e68f0 2024-06-29T14:34:22.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.334+0000 7f2075ffb640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2064076500 0x7f2064078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.335+0000 7f2075ffb640 1 -- 172.21.15.81:0/2599273424 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f20780b8a00 con 0x7f20801e68f0 2024-06-29T14:34:22.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.335+0000 7f2085541640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2064076500 0x7f2064078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.335+0000 7f2085541640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2064076500 0x7f2064078a10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f207c001740 tx=0x7f207c001240 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:22.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.339+0000 7f2075ffb640 1 -- 172.21.15.81:0/2599273424 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f20801e7240 con 0x7f20801e68f0 2024-06-29T14:34:22.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.541+0000 7f20877a5640 1 -- 172.21.15.81:0/2599273424 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f20800cbfd0 con 0x7f2064076500 2024-06-29T14:34:22.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.624+0000 7f2075ffb640 1 -- 172.21.15.81:0/2599273424 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f20800cbfd0 con 0x7f2064076500 2024-06-29T14:34:22.622 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 -- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2064076500 msgr2=0x7f2064078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2064076500 0x7f2064078a10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f207c001740 tx=0x7f207c001240 comp rx=0 tx=0).stop 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 -- 172.21.15.81:0/2599273424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20801e68f0 msgr2=0x7f20801e6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20801e68f0 0x7f20801e6d00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f207802af20 tx=0x7f2078033de0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 -- 172.21.15.81:0/2599273424 shutdown_connections 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2064076500 0x7f2064078a10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20801e68f0 0x7f20801e6d00 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20800b17f0 0x7f20800b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 --2- 172.21.15.81:0/2599273424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f206c004d40 0x7f20801e0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 -- 172.21.15.81:0/2599273424 >> 172.21.15.81:0/2599273424 conn(0x7f20800b3680 msgr2=0x7f20801e8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:22.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 -- 172.21.15.81:0/2599273424 shutdown_connections 2024-06-29T14:34:22.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.626+0000 7f20877a5640 1 -- 172.21.15.81:0/2599273424 wait complete. 2024-06-29T14:34:22.634 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2024-06-29T14:34:22.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2024-06-29T14:34:22.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.754+0000 7f35a6c1f640 1 -- 172.21.15.81:0/322480505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f358c003a30 msgr2=0x7f358c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.754+0000 7f35a6c1f640 1 --2- 172.21.15.81:0/322480505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f358c003a30 0x7f358c003ea0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3590001dc0 tx=0x7f358c005830 comp rx=0 tx=0).stop 2024-06-29T14:34:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.754+0000 7f35a51bc640 1 -- 172.21.15.81:0/322480505 reap_dead start 2024-06-29T14:34:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.754+0000 7f35a6c1f640 1 -- 172.21.15.81:0/322480505 shutdown_connections 2024-06-29T14:34:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.754+0000 7f35a6c1f640 1 -- 172.21.15.81:0/322480505 >> 172.21.15.81:0/322480505 conn(0x7f35a00b1ce0 msgr2=0x7f35a00b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.754+0000 7f35a6c1f640 1 -- 172.21.15.81:0/322480505 shutdown_connections 2024-06-29T14:34:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.754+0000 7f35a6c1f640 1 -- 172.21.15.81:0/322480505 wait complete. 2024-06-29T14:34:22.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.755+0000 7f35a6c1f640 1 Processor -- start 2024-06-29T14:34:22.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.755+0000 7f35a6c1f640 1 -- start start 2024-06-29T14:34:22.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f35a6c1f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358c003a30 0x7f35a013c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f35a6c1f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a01e3650 0x7f35a01e3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f35a6c1f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35a013df70 0x7f35a01510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f35a6c1f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f35a00743c0 con 0x7f35a01e3650 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f35a6c1f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f35a0074240 con 0x7f35a013df70 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f35a6c1f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f35a006fc00 con 0x7f358c003a30 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f359ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a01e3650 0x7f35a01e3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f359ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a01e3650 0x7f35a01e3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42636/0 (socket says 172.21.15.81:42636) 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f359ffff640 1 -- 172.21.15.81:0/271735506 learned_addr learned my addr 172.21.15.81:0/271735506 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f359ffff640 1 -- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358c003a30 msgr2=0x7f35a013c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f359ffff640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358c003a30 0x7f35a013c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f359ffff640 1 -- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35a013df70 msgr2=0x7f35a01510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f359ffff640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35a013df70 0x7f35a01510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.757+0000 7f359ffff640 1 -- 172.21.15.81:0/271735506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a013ced0 con 0x7f35a01e3650 2024-06-29T14:34:22.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.758+0000 7f359ffff640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a01e3650 0x7f35a01e3f60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3588011ab0 tx=0x7f3588011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:22.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.758+0000 7f359d7fa640 1 -- 172.21.15.81:0/271735506 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588010e20 con 0x7f35a01e3650 2024-06-29T14:34:22.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.758+0000 7f359d7fa640 1 -- 172.21.15.81:0/271735506 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3588016030 con 0x7f35a01e3650 2024-06-29T14:34:22.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.758+0000 7f35a6c1f640 1 -- 172.21.15.81:0/271735506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a013d1b0 con 0x7f35a01e3650 2024-06-29T14:34:22.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.758+0000 7f359d7fa640 1 -- 172.21.15.81:0/271735506 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35880042b0 con 0x7f35a01e3650 2024-06-29T14:34:22.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.758+0000 7f35a6c1f640 1 -- 172.21.15.81:0/271735506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a013d770 con 0x7f35a01e3650 2024-06-29T14:34:22.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.759+0000 7f35a6c1f640 1 -- 172.21.15.81:0/271735506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35a006eb40 con 0x7f35a01e3650 2024-06-29T14:34:22.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.759+0000 7f359d7fa640 1 -- 172.21.15.81:0/271735506 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f35880035f0 con 0x7f35a01e3650 2024-06-29T14:34:22.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.760+0000 7f359d7fa640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f35800764d0 0x7f35800789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:22.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.760+0000 7f359d7fa640 1 -- 172.21.15.81:0/271735506 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f3588098b30 con 0x7f35a01e3650 2024-06-29T14:34:22.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.760+0000 7f35a49bb640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f35800764d0 0x7f35800789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:22.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.761+0000 7f35a49bb640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f35800764d0 0x7f35800789e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3590032000 tx=0x7f35900014a0 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.764+0000 7f359d7fa640 1 -- 172.21.15.81:0/271735506 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f35880621d0 con 0x7f35a01e3650 2024-06-29T14:34:22.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:22.966+0000 7f35a6c1f640 1 -- 172.21.15.81:0/271735506 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f35a006a4b0 con 0x7f35800764d0 2024-06-29T14:34:23.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.048+0000 7f359d7fa640 1 -- 172.21.15.81:0/271735506 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f35a006a4b0 con 0x7f35800764d0 2024-06-29T14:34:23.046 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2024-06-29T14:34:23.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 -- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f35800764d0 msgr2=0x7f35800789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f35800764d0 0x7f35800789e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3590032000 tx=0x7f35900014a0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 -- 172.21.15.81:0/271735506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a01e3650 msgr2=0x7f35a01e3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a01e3650 0x7f35a01e3f60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3588011ab0 tx=0x7f3588011f70 comp rx=0 tx=0).stop 2024-06-29T14:34:23.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 -- 172.21.15.81:0/271735506 shutdown_connections 2024-06-29T14:34:23.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f35800764d0 0x7f35800789e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35a013df70 0x7f35a01510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a01e3650 0x7f35a01e3f60 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 --2- 172.21.15.81:0/271735506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f358c003a30 0x7f35a013c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 -- 172.21.15.81:0/271735506 >> 172.21.15.81:0/271735506 conn(0x7f35a00b1ce0 msgr2=0x7f35a006f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 -- 172.21.15.81:0/271735506 shutdown_connections 2024-06-29T14:34:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.050+0000 7f35a6c1f640 1 -- 172.21.15.81:0/271735506 wait complete. 2024-06-29T14:34:23.062 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2024-06-29T14:34:23.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2024-06-29T14:34:23.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.182+0000 7f437baf6640 1 -- 172.21.15.81:0/3303264512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43740b3960 msgr2=0x7f43740b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.182+0000 7f437baf6640 1 --2- 172.21.15.81:0/3303264512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43740b3960 0x7f43740b3d70 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f436c005fd0 tx=0x7f436c02aa90 comp rx=0 tx=0).stop 2024-06-29T14:34:23.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.182+0000 7f437baf6640 1 -- 172.21.15.81:0/3303264512 shutdown_connections 2024-06-29T14:34:23.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.182+0000 7f437baf6640 1 --2- 172.21.15.81:0/3303264512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43740b3960 0x7f43740b3d70 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.182+0000 7f437baf6640 1 --2- 172.21.15.81:0/3303264512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43740b57b0 0x7f437415aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.182+0000 7f437baf6640 1 -- 172.21.15.81:0/3303264512 >> 172.21.15.81:0/3303264512 conn(0x7f43740d0930 msgr2=0x7f43740d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:23.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.182+0000 7f437baf6640 1 -- 172.21.15.81:0/3303264512 shutdown_connections 2024-06-29T14:34:23.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.182+0000 7f437baf6640 1 -- 172.21.15.81:0/3303264512 wait complete. 2024-06-29T14:34:23.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.183+0000 7f437baf6640 1 Processor -- start 2024-06-29T14:34:23.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.183+0000 7f437baf6640 1 -- start start 2024-06-29T14:34:23.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f437baf6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437406ef90 0x7f43741f57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f437baf6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43740b3960 0x7f43741f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f437baf6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43740b57b0 0x7f43741f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f437baf6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f43740b4640 con 0x7f437406ef90 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f437baf6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f43740b44c0 con 0x7f43740b3960 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f437baf6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f43740b47c0 con 0x7f43740b57b0 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379892640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437406ef90 0x7f43741f57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379892640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437406ef90 0x7f43741f57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42646/0 (socket says 172.21.15.81:42646) 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379892640 1 -- 172.21.15.81:0/4242614877 learned_addr learned my addr 172.21.15.81:0/4242614877 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f437a093640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43740b57b0 0x7f43741f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379091640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43740b3960 0x7f43741f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379892640 1 -- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43740b57b0 msgr2=0x7f43741f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379892640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43740b57b0 0x7f43741f18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379892640 1 -- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43740b3960 msgr2=0x7f43741f13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379892640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43740b3960 0x7f43741f13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379892640 1 -- 172.21.15.81:0/4242614877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43740aad90 con 0x7f437406ef90 2024-06-29T14:34:23.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379091640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43740b3960 0x7f43741f13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:23.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f4379892640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437406ef90 0x7f43741f57a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4370007900 tx=0x7f4370007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:23.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f436a7fc640 1 -- 172.21.15.81:0/4242614877 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437000e6f0 con 0x7f437406ef90 2024-06-29T14:34:23.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f436a7fc640 1 -- 172.21.15.81:0/4242614877 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f437000e890 con 0x7f437406ef90 2024-06-29T14:34:23.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f437baf6640 1 -- 172.21.15.81:0/4242614877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43740ac230 con 0x7f437406ef90 2024-06-29T14:34:23.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.185+0000 7f436a7fc640 1 -- 172.21.15.81:0/4242614877 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437000ece0 con 0x7f437406ef90 2024-06-29T14:34:23.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.186+0000 7f437baf6640 1 -- 172.21.15.81:0/4242614877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43740ac7f0 con 0x7f437406ef90 2024-06-29T14:34:23.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.187+0000 7f437baf6640 1 -- 172.21.15.81:0/4242614877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f437406e2a0 con 0x7f437406ef90 2024-06-29T14:34:23.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.187+0000 7f436a7fc640 1 -- 172.21.15.81:0/4242614877 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f437000a440 con 0x7f437406ef90 2024-06-29T14:34:23.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.188+0000 7f436a7fc640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43580764d0 0x7f43580789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.188+0000 7f436a7fc640 1 -- 172.21.15.81:0/4242614877 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f4370099ac0 con 0x7f437406ef90 2024-06-29T14:34:23.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.188+0000 7f4379091640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43580764d0 0x7f43580789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.189+0000 7f4379091640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43580764d0 0x7f43580789e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f43740ab170 tx=0x7f4364009040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:23.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.193+0000 7f436a7fc640 1 -- 172.21.15.81:0/4242614877 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f437406e2a0 con 0x7f437406ef90 2024-06-29T14:34:23.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.396+0000 7f437baf6640 1 -- 172.21.15.81:0/4242614877 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f43740b3d70 con 0x7f43580764d0 2024-06-29T14:34:23.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.476+0000 7f436a7fc640 1 -- 172.21.15.81:0/4242614877 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f43740b3d70 con 0x7f43580764d0 2024-06-29T14:34:23.474 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 -- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43580764d0 msgr2=0x7f43580789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43580764d0 0x7f43580789e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f43740ab170 tx=0x7f4364009040 comp rx=0 tx=0).stop 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 -- 172.21.15.81:0/4242614877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437406ef90 msgr2=0x7f43741f57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437406ef90 0x7f43741f57a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4370007900 tx=0x7f4370007dc0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 -- 172.21.15.81:0/4242614877 shutdown_connections 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f43580764d0 0x7f43580789e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f43740b57b0 0x7f43741f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f43740b3960 0x7f43741f13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 --2- 172.21.15.81:0/4242614877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437406ef90 0x7f43741f57a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 -- 172.21.15.81:0/4242614877 >> 172.21.15.81:0/4242614877 conn(0x7f43740d0930 msgr2=0x7f43740cf970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 -- 172.21.15.81:0/4242614877 shutdown_connections 2024-06-29T14:34:23.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.478+0000 7f437baf6640 1 -- 172.21.15.81:0/4242614877 wait complete. 2024-06-29T14:34:23.486 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2024-06-29T14:34:23.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:23.487 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-06-29T14:34:23.503 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:23.504 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:23.504 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:23.504 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:23.504 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:23.504 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:23.504 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:23.504 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_01_UTC 2024-06-29T14:34:23.504 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:23.504 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:23.504 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:23.504 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_01_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:23.505 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2024-06-29T14:34:23.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.662+0000 7f846e66d640 1 -- 172.21.15.81:0/1780184034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84680b35b0 msgr2=0x7f846815b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.662+0000 7f846e66d640 1 --2- 172.21.15.81:0/1780184034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84680b35b0 0x7f846815b5b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f84680cd500 tx=0x7f845802a560 comp rx=0 tx=0).stop 2024-06-29T14:34:23.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.662+0000 7f846e66d640 1 -- 172.21.15.81:0/1780184034 shutdown_connections 2024-06-29T14:34:23.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.662+0000 7f846e66d640 1 --2- 172.21.15.81:0/1780184034 >> v2:172.21.15.167:3300/0 conn(0x7f846806f660 0x7f846806fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.662+0000 7f846e66d640 1 --2- 172.21.15.81:0/1780184034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84680b35b0 0x7f846815b5b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.662+0000 7f846e66d640 1 -- 172.21.15.81:0/1780184034 >> 172.21.15.81:0/1780184034 conn(0x7f846806ccc0 msgr2=0x7f84680b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:23.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.663+0000 7f846e66d640 1 -- 172.21.15.81:0/1780184034 shutdown_connections 2024-06-29T14:34:23.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.663+0000 7f846e66d640 1 -- 172.21.15.81:0/1780184034 wait complete. 2024-06-29T14:34:23.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.664+0000 7f846e66d640 1 Processor -- start 2024-06-29T14:34:23.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.664+0000 7f846e66d640 1 -- start start 2024-06-29T14:34:23.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f846e66d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f846806f660 0x7f8468147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f846e66d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84680b35b0 0x7f84680b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f846e66d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84680b4430 0x7f846813a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f846e66d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f846806f1e0 con 0x7f846806f660 2024-06-29T14:34:23.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f846e66d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f846806f060 con 0x7f84680b35b0 2024-06-29T14:34:23.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f846e66d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f846806f360 con 0x7f84680b4430 2024-06-29T14:34:23.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8467fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f846806f660 0x7f8468147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8467fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f846806f660 0x7f8468147ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42670/0 (socket says 172.21.15.81:42670) 2024-06-29T14:34:23.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8467fff640 1 -- 172.21.15.81:0/521718608 learned_addr learned my addr 172.21.15.81:0/521718608 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:23.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f84677fe640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84680b35b0 0x7f84680b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8467fff640 1 -- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84680b4430 msgr2=0x7f846813a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8467fff640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84680b4430 0x7f846813a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8467fff640 1 -- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84680b35b0 msgr2=0x7f84680b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8467fff640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84680b35b0 0x7f84680b1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8467fff640 1 -- 172.21.15.81:0/521718608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f846813a730 con 0x7f846806f660 2024-06-29T14:34:23.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8467fff640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f846806f660 0x7f8468147ee0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8458034040 tx=0x7f84580039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:23.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8464ff9640 1 -- 172.21.15.81:0/521718608 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845803e030 con 0x7f846806f660 2024-06-29T14:34:23.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8464ff9640 1 -- 172.21.15.81:0/521718608 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84580047a0 con 0x7f846806f660 2024-06-29T14:34:23.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f846e66d640 1 -- 172.21.15.81:0/521718608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84680b00b0 con 0x7f846806f660 2024-06-29T14:34:23.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f8464ff9640 1 -- 172.21.15.81:0/521718608 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8458035040 con 0x7f846806f660 2024-06-29T14:34:23.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.666+0000 7f846e66d640 1 -- 172.21.15.81:0/521718608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84680b0670 con 0x7f846806f660 2024-06-29T14:34:23.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.667+0000 7f846e66d640 1 -- 172.21.15.81:0/521718608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f846806fa70 con 0x7f846806f660 2024-06-29T14:34:23.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.668+0000 7f8464ff9640 1 -- 172.21.15.81:0/521718608 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f8458041030 con 0x7f846806f660 2024-06-29T14:34:23.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.669+0000 7f8464ff9640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f84540764d0 0x7f84540789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:23.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.669+0000 7f8464ff9640 1 -- 172.21.15.81:0/521718608 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f84580b8950 con 0x7f846806f660 2024-06-29T14:34:23.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.669+0000 7f84677fe640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f84540764d0 0x7f84540789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:23.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.670+0000 7f84677fe640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f84540764d0 0x7f84540789e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f846814e850 tx=0x7f845c009040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:23.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.672+0000 7f8464ff9640 1 -- 172.21.15.81:0/521718608 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8458081f00 con 0x7f846806f660 2024-06-29T14:34:23.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.874+0000 7f846e66d640 1 -- 172.21.15.81:0/521718608 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f846815b700 con 0x7f84540764d0 2024-06-29T14:34:23.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.884+0000 7f8464ff9640 1 -- 172.21.15.81:0/521718608 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f846815b700 con 0x7f84540764d0 2024-06-29T14:34:23.882 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:23.882 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:50", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:23.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 -- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f84540764d0 msgr2=0x7f84540789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f84540764d0 0x7f84540789e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f846814e850 tx=0x7f845c009040 comp rx=0 tx=0).stop 2024-06-29T14:34:23.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 -- 172.21.15.81:0/521718608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f846806f660 msgr2=0x7f8468147ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:23.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f846806f660 0x7f8468147ee0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8458034040 tx=0x7f84580039e0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 -- 172.21.15.81:0/521718608 shutdown_connections 2024-06-29T14:34:23.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f84540764d0 0x7f84540789e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84680b4430 0x7f846813a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84680b35b0 0x7f84680b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 --2- 172.21.15.81:0/521718608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f846806f660 0x7f8468147ee0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:23.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 -- 172.21.15.81:0/521718608 >> 172.21.15.81:0/521718608 conn(0x7f846806ccc0 msgr2=0x7f84681543a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:23.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 -- 172.21.15.81:0/521718608 shutdown_connections 2024-06-29T14:34:23.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:23.886+0000 7f846e66d640 1 -- 172.21.15.81:0/521718608 wait complete. 2024-06-29T14:34:23.895 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:50", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:23.895 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:50', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:23.895 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:23.895 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:23.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:23.895 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-06-29T14:34:23.911 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:23.911 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:23.911 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:23.911 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:23.911 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:23.911 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:23.911 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:23.911 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:23.911 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:23.912 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:23.912 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:23.912 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:23.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2024-06-29T14:34:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.071+0000 7f1b23aa4640 1 -- 172.21.15.81:0/1014360969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b1c162710 msgr2=0x7f1b1c147940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.071+0000 7f1b23aa4640 1 --2- 172.21.15.81:0/1014360969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b1c162710 0x7f1b1c147940 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c071780 tx=0x7f1b1802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.071+0000 7f1b23aa4640 1 -- 172.21.15.81:0/1014360969 shutdown_connections 2024-06-29T14:34:24.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.071+0000 7f1b23aa4640 1 --2- 172.21.15.81:0/1014360969 >> v2:172.21.15.167:3300/0 conn(0x7f1b1c0b4cc0 0x7f1b1c0732a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.071+0000 7f1b23aa4640 1 --2- 172.21.15.81:0/1014360969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b1c162710 0x7f1b1c147940 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.071+0000 7f1b23aa4640 1 -- 172.21.15.81:0/1014360969 >> 172.21.15.81:0/1014360969 conn(0x7f1b1c0b1ce0 msgr2=0x7f1b1c0b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:24.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.071+0000 7f1b23aa4640 1 -- 172.21.15.81:0/1014360969 shutdown_connections 2024-06-29T14:34:24.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.071+0000 7f1b23aa4640 1 -- 172.21.15.81:0/1014360969 wait complete. 2024-06-29T14:34:24.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.072+0000 7f1b23aa4640 1 Processor -- start 2024-06-29T14:34:24.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.072+0000 7f1b23aa4640 1 -- start start 2024-06-29T14:34:24.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b23aa4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b1c0737e0 0x7f1b1c15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b23aa4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b1c0b4cc0 0x7f1b1c140db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b23aa4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b1c162710 0x7f1b1c1412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b23aa4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c0b5250 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b23aa4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1c0b50d0 con 0x7f1b1c162710 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b23aa4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1b1c0adfe0 con 0x7f1b1c0737e0 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b2103f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b1c0b4cc0 0x7f1b1c140db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b2103f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b1c0b4cc0 0x7f1b1c140db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42684/0 (socket says 172.21.15.81:42684) 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b2103f640 1 -- 172.21.15.81:0/273431694 learned_addr learned my addr 172.21.15.81:0/273431694 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b21840640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b1c0737e0 0x7f1b1c15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b22041640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b1c162710 0x7f1b1c1412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b2103f640 1 -- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b1c0737e0 msgr2=0x7f1b1c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b2103f640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b1c0737e0 0x7f1b1c15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b2103f640 1 -- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b1c162710 msgr2=0x7f1b1c1412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b2103f640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b1c162710 0x7f1b1c1412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b2103f640 1 -- 172.21.15.81:0/273431694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b1c13e300 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b2103f640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b1c0b4cc0 0x7f1b1c140db0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00d770 tx=0x7f1b0c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:24.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b127fc640 1 -- 172.21.15.81:0/273431694 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c017070 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b127fc640 1 -- 172.21.15.81:0/273431694 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b0c009d20 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.074+0000 7f1b23aa4640 1 -- 172.21.15.81:0/273431694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b1c13e5e0 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.075+0000 7f1b127fc640 1 -- 172.21.15.81:0/273431694 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c004280 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.075+0000 7f1b23aa4640 1 -- 172.21.15.81:0/273431694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b1c13eba0 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.076+0000 7f1b23aa4640 1 -- 172.21.15.81:0/273431694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b1c162450 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.076+0000 7f1b127fc640 1 -- 172.21.15.81:0/273431694 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f1b0c005020 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.077+0000 7f1b127fc640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1b000764d0 0x7f1b000789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.077+0000 7f1b127fc640 1 -- 172.21.15.81:0/273431694 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f1b0c0980e0 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.077+0000 7f1b21840640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1b000764d0 0x7f1b000789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.078+0000 7f1b21840640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1b000764d0 0x7f1b000789e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b18034000 tx=0x7f1b18004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:24.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.081+0000 7f1b127fc640 1 -- 172.21.15.81:0/273431694 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1b1c162450 con 0x7f1b1c0b4cc0 2024-06-29T14:34:24.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.286+0000 7f1b23aa4640 1 -- 172.21.15.81:0/273431694 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1b1c13cea0 con 0x7f1b000764d0 2024-06-29T14:34:24.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.296+0000 7f1b127fc640 1 -- 172.21.15.81:0/273431694 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f1b1c13cea0 con 0x7f1b000764d0 2024-06-29T14:34:24.294 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:24.295 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:51", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 -- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1b000764d0 msgr2=0x7f1b000789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1b000764d0 0x7f1b000789e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1b18034000 tx=0x7f1b18004730 comp rx=0 tx=0).stop 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 -- 172.21.15.81:0/273431694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b1c0b4cc0 msgr2=0x7f1b1c140db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b1c0b4cc0 0x7f1b1c140db0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00d770 tx=0x7f1b0c00dc30 comp rx=0 tx=0).stop 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 -- 172.21.15.81:0/273431694 shutdown_connections 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1b000764d0 0x7f1b000789e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b1c162710 0x7f1b1c1412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b1c0b4cc0 0x7f1b1c140db0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 --2- 172.21.15.81:0/273431694 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b1c0737e0 0x7f1b1c15cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 -- 172.21.15.81:0/273431694 >> 172.21.15.81:0/273431694 conn(0x7f1b1c0b1ce0 msgr2=0x7f1b1c072270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 -- 172.21.15.81:0/273431694 shutdown_connections 2024-06-29T14:34:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.298+0000 7f1b23aa4640 1 -- 172.21.15.81:0/273431694 wait complete. 2024-06-29T14:34:24.306 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:51", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:24.307 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:51', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:24.307 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:24.307 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:24.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:24.307 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-06-29T14:34:24.322 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:24.323 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:24.323 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:24.323 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:24.323 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:24.323 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:24.323 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:24.323 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:24.323 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:24.323 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:24.323 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:24.324 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:24.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.482+0000 7ff1c5c40640 1 -- 172.21.15.81:0/901561332 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1c00b6eb0 msgr2=0x7ff1c00d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.482+0000 7ff1c5c40640 1 --2- 172.21.15.81:0/901561332 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1c00b6eb0 0x7ff1c00d02d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0005fd0 tx=0x7ff1b002a540 comp rx=0 tx=0).stop 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.482+0000 7ff1c5c40640 1 -- 172.21.15.81:0/901561332 shutdown_connections 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.482+0000 7ff1c5c40640 1 --2- 172.21.15.81:0/901561332 >> v2:172.21.15.167:3300/0 conn(0x7ff1c00ce920 0x7ff1c00cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.482+0000 7ff1c5c40640 1 --2- 172.21.15.81:0/901561332 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1c00b6eb0 0x7ff1c00d02d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.482+0000 7ff1c5c40640 1 -- 172.21.15.81:0/901561332 >> 172.21.15.81:0/901561332 conn(0x7ff1c00b3510 msgr2=0x7ff1c00b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.482+0000 7ff1c5c40640 1 -- 172.21.15.81:0/901561332 shutdown_connections 2024-06-29T14:34:24.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.482+0000 7ff1c5c40640 1 -- 172.21.15.81:0/901561332 wait complete. 2024-06-29T14:34:24.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.484+0000 7ff1c5c40640 1 Processor -- start 2024-06-29T14:34:24.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.484+0000 7ff1c5c40640 1 -- start start 2024-06-29T14:34:24.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1c5c40640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1c00b6eb0 0x7ff1c014a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1c5c40640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1c00ce510 0x7ff1c015f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1c5c40640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1c00ce920 0x7ff1c00cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1c5c40640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c00b50a0 con 0x7ff1c00ce510 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1c5c40640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c00b4f20 con 0x7ff1c00b6eb0 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1c5c40640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff1c00b5220 con 0x7ff1c00ce920 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1beffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1c00ce510 0x7ff1c015f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1beffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1c00ce510 0x7ff1c015f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42686/0 (socket says 172.21.15.81:42686) 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1beffd640 1 -- 172.21.15.81:0/404564661 learned_addr learned my addr 172.21.15.81:0/404564661 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1bffff640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1c00ce920 0x7ff1c00cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1bf7fe640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1c00b6eb0 0x7ff1c014a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1beffd640 1 -- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1c00ce920 msgr2=0x7ff1c00cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1beffd640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1c00ce920 0x7ff1c00cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1beffd640 1 -- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1c00b6eb0 msgr2=0x7ff1c014a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1beffd640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1c00b6eb0 0x7ff1c014a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1beffd640 1 -- 172.21.15.81:0/404564661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c01581e0 con 0x7ff1c00ce510 2024-06-29T14:34:24.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1beffd640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1c00ce510 0x7ff1c015f130 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff1b400c430 tx=0x7ff1b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:24.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1a7fff640 1 -- 172.21.15.81:0/404564661 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4017070 con 0x7ff1c00ce510 2024-06-29T14:34:24.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1a7fff640 1 -- 172.21.15.81:0/404564661 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1b40053e0 con 0x7ff1c00ce510 2024-06-29T14:34:24.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1c5c40640 1 -- 172.21.15.81:0/404564661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1c015dd60 con 0x7ff1c00ce510 2024-06-29T14:34:24.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.486+0000 7ff1a7fff640 1 -- 172.21.15.81:0/404564661 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4003850 con 0x7ff1c00ce510 2024-06-29T14:34:24.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.487+0000 7ff1c5c40640 1 -- 172.21.15.81:0/404564661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1c015e320 con 0x7ff1c00ce510 2024-06-29T14:34:24.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.488+0000 7ff1c5c40640 1 -- 172.21.15.81:0/404564661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1c00d02d0 con 0x7ff1c00ce510 2024-06-29T14:34:24.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.488+0000 7ff1a7fff640 1 -- 172.21.15.81:0/404564661 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7ff1b401a070 con 0x7ff1c00ce510 2024-06-29T14:34:24.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.489+0000 7ff1a7fff640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff1ac076500 0x7ff1ac078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.489+0000 7ff1a7fff640 1 -- 172.21.15.81:0/404564661 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7ff1b4097080 con 0x7ff1c00ce510 2024-06-29T14:34:24.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.489+0000 7ff1bf7fe640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff1ac076500 0x7ff1ac078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.490+0000 7ff1bf7fe640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff1ac076500 0x7ff1ac078a10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0033000 tx=0x7ff1b0004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:24.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.493+0000 7ff1a7fff640 1 -- 172.21.15.81:0/404564661 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff1b40605d0 con 0x7ff1c00ce510 2024-06-29T14:34:24.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.694+0000 7ff1c5c40640 1 -- 172.21.15.81:0/404564661 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1c01583b0 con 0x7ff1ac076500 2024-06-29T14:34:24.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.704+0000 7ff1a7fff640 1 -- 172.21.15.81:0/404564661 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7ff1c01583b0 con 0x7ff1ac076500 2024-06-29T14:34:24.701 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:24.702 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:51", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:24.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 -- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff1ac076500 msgr2=0x7ff1ac078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff1ac076500 0x7ff1ac078a10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0033000 tx=0x7ff1b0004730 comp rx=0 tx=0).stop 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 -- 172.21.15.81:0/404564661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1c00ce510 msgr2=0x7ff1c015f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1c00ce510 0x7ff1c015f130 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff1b400c430 tx=0x7ff1b400c8f0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 -- 172.21.15.81:0/404564661 shutdown_connections 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff1ac076500 0x7ff1ac078a10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1c00ce920 0x7ff1c00cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1c00ce510 0x7ff1c015f130 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 --2- 172.21.15.81:0/404564661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1c00b6eb0 0x7ff1c014a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 -- 172.21.15.81:0/404564661 >> 172.21.15.81:0/404564661 conn(0x7ff1c00b3510 msgr2=0x7ff1c015fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 -- 172.21.15.81:0/404564661 shutdown_connections 2024-06-29T14:34:24.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.706+0000 7ff1c5c40640 1 -- 172.21.15.81:0/404564661 wait complete. 2024-06-29T14:34:24.714 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:51", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:24.714 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:51', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:24.714 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:24.714 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:24.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:24.715 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-06-29T14:34:24.731 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:24.731 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:24.731 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:24.731 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:24.731 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:24.731 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:24.731 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:24.731 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:24.732 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:24.732 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:24.732 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:24.732 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:24.732 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2024-06-29T14:34:24.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.888+0000 7f884383c640 1 -- 172.21.15.81:0/3164684278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8828003a30 msgr2=0x7f8828003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.888+0000 7f884383c640 1 --2- 172.21.15.81:0/3164684278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8828003a30 0x7f8828003ea0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8838001dc0 tx=0x7f8828005830 comp rx=0 tx=0).stop 2024-06-29T14:34:24.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.888+0000 7f8841dd9640 1 -- 172.21.15.81:0/3164684278 reap_dead start 2024-06-29T14:34:24.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.888+0000 7f884383c640 1 -- 172.21.15.81:0/3164684278 shutdown_connections 2024-06-29T14:34:24.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.888+0000 7f884383c640 1 -- 172.21.15.81:0/3164684278 >> 172.21.15.81:0/3164684278 conn(0x7f883c0d0930 msgr2=0x7f883c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:24.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.888+0000 7f884383c640 1 -- 172.21.15.81:0/3164684278 shutdown_connections 2024-06-29T14:34:24.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.888+0000 7f884383c640 1 -- 172.21.15.81:0/3164684278 wait complete. 2024-06-29T14:34:24.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.890+0000 7f884383c640 1 Processor -- start 2024-06-29T14:34:24.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.890+0000 7f884383c640 1 -- start start 2024-06-29T14:34:24.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.891+0000 7f884383c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8828003a30 0x7f883c147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.891+0000 7f884383c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f883c144200 0x7f883c0b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.891+0000 7f884383c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f883c0b1d80 0x7f883c144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.891+0000 7f884383c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f883c06c2f0 con 0x7f8828003a30 2024-06-29T14:34:24.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.891+0000 7f884383c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f883c0b5060 con 0x7f883c144200 2024-06-29T14:34:24.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.891+0000 7f884383c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f883c06c470 con 0x7f883c0b1d80 2024-06-29T14:34:24.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.891+0000 7f88415d8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8828003a30 0x7f883c147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.891+0000 7f88415d8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8828003a30 0x7f883c147200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42696/0 (socket says 172.21.15.81:42696) 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.891+0000 7f88415d8640 1 -- 172.21.15.81:0/503603926 learned_addr learned my addr 172.21.15.81:0/503603926 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f8841dd9640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f883c0b1d80 0x7f883c144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f88415d8640 1 -- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f883c0b1d80 msgr2=0x7f883c144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f88415d8640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f883c0b1d80 0x7f883c144760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f8840dd7640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f883c144200 0x7f883c0b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f88415d8640 1 -- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f883c144200 msgr2=0x7f883c0b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f88415d8640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f883c144200 0x7f883c0b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f88415d8640 1 -- 172.21.15.81:0/503603926 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883c0ce040 con 0x7f8828003a30 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f88415d8640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8828003a30 0x7f883c147200 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f883802acc0 tx=0x7f8838006db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f8831ffb640 1 -- 172.21.15.81:0/503603926 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838006250 con 0x7f8828003a30 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f8831ffb640 1 -- 172.21.15.81:0/503603926 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88380064e0 con 0x7f8828003a30 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f884383c640 1 -- 172.21.15.81:0/503603926 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f883c147740 con 0x7f8828003a30 2024-06-29T14:34:24.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f8831ffb640 1 -- 172.21.15.81:0/503603926 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88380068b0 con 0x7f8828003a30 2024-06-29T14:34:24.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.892+0000 7f884383c640 1 -- 172.21.15.81:0/503603926 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f883c1f5360 con 0x7f8828003a30 2024-06-29T14:34:24.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.893+0000 7f884383c640 1 -- 172.21.15.81:0/503603926 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f883c068bd0 con 0x7f8828003a30 2024-06-29T14:34:24.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.894+0000 7f8831ffb640 1 -- 172.21.15.81:0/503603926 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f8838033480 con 0x7f8828003a30 2024-06-29T14:34:24.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.894+0000 7f8831ffb640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f8820076670 0x7f8820078b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:24.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.894+0000 7f8831ffb640 1 -- 172.21.15.81:0/503603926 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f88380b8400 con 0x7f8828003a30 2024-06-29T14:34:24.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.895+0000 7f8840dd7640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f8820076670 0x7f8820078b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:24.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.895+0000 7f8840dd7640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f8820076670 0x7f8820078b80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f883c0b23e0 tx=0x7f882c005f90 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:24.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:24.899+0000 7f8831ffb640 1 -- 172.21.15.81:0/503603926 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f883c068bd0 con 0x7f8828003a30 2024-06-29T14:34:25.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.100+0000 7f884383c640 1 -- 172.21.15.81:0/503603926 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f883c158d40 con 0x7f8820076670 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.110+0000 7f8831ffb640 1 -- 172.21.15.81:0/503603926 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f882c012aa0 con 0x7f8820076670 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:25.107 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:52", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 -- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f8820076670 msgr2=0x7f8820078b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f8820076670 0x7f8820078b80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f883c0b23e0 tx=0x7f882c005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 -- 172.21.15.81:0/503603926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8828003a30 msgr2=0x7f883c147200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8828003a30 0x7f883c147200 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f883802acc0 tx=0x7f8838006db0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 -- 172.21.15.81:0/503603926 shutdown_connections 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f8820076670 0x7f8820078b80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f883c0b1d80 0x7f883c144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f883c144200 0x7f883c0b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 --2- 172.21.15.81:0/503603926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8828003a30 0x7f883c147200 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 -- 172.21.15.81:0/503603926 >> 172.21.15.81:0/503603926 conn(0x7f883c0d0930 msgr2=0x7f883c1614f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 -- 172.21.15.81:0/503603926 shutdown_connections 2024-06-29T14:34:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.112+0000 7f884383c640 1 -- 172.21.15.81:0/503603926 wait complete. 2024-06-29T14:34:25.120 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:52", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:25.120 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:52', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:25.120 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:25.120 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:25.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:25.121 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-06-29T14:34:25.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:25.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:25.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:25.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:25.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:25.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:25.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:25.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:25.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:25.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:25.138 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:25.138 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:25.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2024-06-29T14:34:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.295+0000 7f691d7eb640 1 -- 172.21.15.81:0/3074709153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69180cea50 msgr2=0x7f69180b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.295+0000 7f691d7eb640 1 --2- 172.21.15.81:0/3074709153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69180cea50 0x7f69180b4810 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6918073550 tx=0x7f691002a560 comp rx=0 tx=0).stop 2024-06-29T14:34:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.295+0000 7f691d7eb640 1 -- 172.21.15.81:0/3074709153 shutdown_connections 2024-06-29T14:34:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.295+0000 7f691d7eb640 1 --2- 172.21.15.81:0/3074709153 >> v2:172.21.15.167:3300/0 conn(0x7f69180cd480 0x7f69180ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.295+0000 7f691d7eb640 1 --2- 172.21.15.81:0/3074709153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69180cea50 0x7f69180b4810 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.295+0000 7f691d7eb640 1 -- 172.21.15.81:0/3074709153 >> 172.21.15.81:0/3074709153 conn(0x7f69180b3510 msgr2=0x7f69180b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.295+0000 7f691d7eb640 1 -- 172.21.15.81:0/3074709153 shutdown_connections 2024-06-29T14:34:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.295+0000 7f691d7eb640 1 -- 172.21.15.81:0/3074709153 wait complete. 2024-06-29T14:34:25.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.297+0000 7f691d7eb640 1 Processor -- start 2024-06-29T14:34:25.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.297+0000 7f691d7eb640 1 -- start start 2024-06-29T14:34:25.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.298+0000 7f691d7eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f69180b6eb0 0x7f69181e72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.298+0000 7f691d7eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f69180cd480 0x7f69180ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.298+0000 7f691d7eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69180cea50 0x7f69180ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.298+0000 7f691d7eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f69180b4ed0 con 0x7f69180cea50 2024-06-29T14:34:25.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.298+0000 7f691d7eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f69180b4d50 con 0x7f69180cd480 2024-06-29T14:34:25.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.298+0000 7f691d7eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f69180b5050 con 0x7f69180b6eb0 2024-06-29T14:34:25.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.298+0000 7f69177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69180cea50 0x7f69180ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.298+0000 7f69177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69180cea50 0x7f69180ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42724/0 (socket says 172.21.15.81:42724) 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.298+0000 7f69177fe640 1 -- 172.21.15.81:0/4121762998 learned_addr learned my addr 172.21.15.81:0/4121762998 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f6916ffd640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f69180b6eb0 0x7f69181e72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f69167fc640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f69180cd480 0x7f69180ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f69177fe640 1 -- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f69180b6eb0 msgr2=0x7f69181e72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f69177fe640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f69180b6eb0 0x7f69181e72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f69177fe640 1 -- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f69180cd480 msgr2=0x7f69180ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f69177fe640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f69180cd480 0x7f69180ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f69177fe640 1 -- 172.21.15.81:0/4121762998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69181f4f70 con 0x7f69180cea50 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f69167fc640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f69180cd480 0x7f69180ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f69177fe640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69180cea50 0x7f69180ca760 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6910009230 tx=0x7f6910003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f68ff7fe640 1 -- 172.21.15.81:0/4121762998 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691003e030 con 0x7f69180cea50 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f68ff7fe640 1 -- 172.21.15.81:0/4121762998 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6910003e50 con 0x7f69180cea50 2024-06-29T14:34:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f68ff7fe640 1 -- 172.21.15.81:0/4121762998 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910035040 con 0x7f69180cea50 2024-06-29T14:34:25.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f691d7eb640 1 -- 172.21.15.81:0/4121762998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69180aeb60 con 0x7f69180cea50 2024-06-29T14:34:25.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.299+0000 7f691d7eb640 1 -- 172.21.15.81:0/4121762998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69180cb990 con 0x7f69180cea50 2024-06-29T14:34:25.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.300+0000 7f691d7eb640 1 -- 172.21.15.81:0/4121762998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69180cb4d0 con 0x7f69180cea50 2024-06-29T14:34:25.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.301+0000 7f68ff7fe640 1 -- 172.21.15.81:0/4121762998 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f6910041030 con 0x7f69180cea50 2024-06-29T14:34:25.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.301+0000 7f68ff7fe640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f69040764d0 0x7f69040789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.301+0000 7f68ff7fe640 1 -- 172.21.15.81:0/4121762998 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f69100b8320 con 0x7f69180cea50 2024-06-29T14:34:25.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.301+0000 7f6916ffd640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f69040764d0 0x7f69040789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.302+0000 7f6916ffd640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f69040764d0 0x7f69040789e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6908001ce0 tx=0x7f6908009040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:25.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.305+0000 7f68ff7fe640 1 -- 172.21.15.81:0/4121762998 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f69180cb4d0 con 0x7f69180cea50 2024-06-29T14:34:25.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.504+0000 7f691d7eb640 1 -- 172.21.15.81:0/4121762998 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f69180ae8f0 con 0x7f69040764d0 2024-06-29T14:34:25.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.514+0000 7f68ff7fe640 1 -- 172.21.15.81:0/4121762998 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f69180ae8f0 con 0x7f69040764d0 2024-06-29T14:34:25.512 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:25.512 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:52", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 -- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f69040764d0 msgr2=0x7f69040789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f69040764d0 0x7f69040789e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6908001ce0 tx=0x7f6908009040 comp rx=0 tx=0).stop 2024-06-29T14:34:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 -- 172.21.15.81:0/4121762998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69180cea50 msgr2=0x7f69180ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69180cea50 0x7f69180ca760 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6910009230 tx=0x7f6910003960 comp rx=0 tx=0).stop 2024-06-29T14:34:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 -- 172.21.15.81:0/4121762998 shutdown_connections 2024-06-29T14:34:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f69040764d0 0x7f69040789e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69180cea50 0x7f69180ca760 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f69180cd480 0x7f69180ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 --2- 172.21.15.81:0/4121762998 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f69180b6eb0 0x7f69181e72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 -- 172.21.15.81:0/4121762998 >> 172.21.15.81:0/4121762998 conn(0x7f69180b3510 msgr2=0x7f6918070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:25.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 -- 172.21.15.81:0/4121762998 shutdown_connections 2024-06-29T14:34:25.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.516+0000 7f691d7eb640 1 -- 172.21.15.81:0/4121762998 wait complete. 2024-06-29T14:34:25.527 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:52", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:25.527 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:52', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:25.527 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:25.527 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:25.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:25.527 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-06-29T14:34:25.544 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:25.544 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:25.544 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:25.544 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:25.544 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:25.544 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:25.544 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:25.544 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:25.544 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:25.544 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:25.545 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:25.545 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:25.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2024-06-29T14:34:25.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.701+0000 7f93c93c7640 1 -- 172.21.15.81:0/3531406867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93c40d2a90 msgr2=0x7f93c40d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.701+0000 7f93c93c7640 1 --2- 172.21.15.81:0/3531406867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93c40d2a90 0x7f93c40d09c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f93bc005fd0 tx=0x7f93bc02a940 comp rx=0 tx=0).stop 2024-06-29T14:34:25.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.702+0000 7f93c93c7640 1 -- 172.21.15.81:0/3531406867 shutdown_connections 2024-06-29T14:34:25.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.702+0000 7f93c93c7640 1 --2- 172.21.15.81:0/3531406867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93c40d2a90 0x7f93c40d09c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.702+0000 7f93c93c7640 1 --2- 172.21.15.81:0/3531406867 >> v2:172.21.15.167:3300/0 conn(0x7f93c40b9040 0x7f93c40d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.702+0000 7f93c93c7640 1 -- 172.21.15.81:0/3531406867 >> 172.21.15.81:0/3531406867 conn(0x7f93c409cf30 msgr2=0x7f93c40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:25.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.702+0000 7f93c93c7640 1 -- 172.21.15.81:0/3531406867 shutdown_connections 2024-06-29T14:34:25.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.702+0000 7f93c93c7640 1 -- 172.21.15.81:0/3531406867 wait complete. 2024-06-29T14:34:25.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.703+0000 7f93c93c7640 1 Processor -- start 2024-06-29T14:34:25.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.703+0000 7f93c93c7640 1 -- start start 2024-06-29T14:34:25.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c93c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c40b5250 0x7f93c415c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c93c7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93c40b9040 0x7f93c40a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c93c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93c40d2a90 0x7f93c40a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c93c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f93c40b7d30 con 0x7f93c40b5250 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c93c7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f93c40b7bb0 con 0x7f93c40d2a90 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c93c7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f93c40b7eb0 con 0x7f93c40b9040 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c40b5250 0x7f93c415c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c40b5250 0x7f93c415c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42736/0 (socket says 172.21.15.81:42736) 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c2ffd640 1 -- 172.21.15.81:0/3165442420 learned_addr learned my addr 172.21.15.81:0/3165442420 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93ba7fc640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93c40b9040 0x7f93c40a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c37fe640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93c40d2a90 0x7f93c40a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c2ffd640 1 -- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93c40b9040 msgr2=0x7f93c40a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c2ffd640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93c40b9040 0x7f93c40a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c2ffd640 1 -- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93c40d2a90 msgr2=0x7f93c40a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c2ffd640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93c40d2a90 0x7f93c40a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c2ffd640 1 -- 172.21.15.81:0/3165442420 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c40b24f0 con 0x7f93c40b5250 2024-06-29T14:34:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c37fe640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93c40d2a90 0x7f93c40a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.705+0000 7f93c2ffd640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c40b5250 0x7f93c415c4c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f93b400ea20 tx=0x7f93b400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.706+0000 7f93bbfff640 1 -- 172.21.15.81:0/3165442420 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b400ce40 con 0x7f93c40b5250 2024-06-29T14:34:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.706+0000 7f93bbfff640 1 -- 172.21.15.81:0/3165442420 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93b4004020 con 0x7f93c40b5250 2024-06-29T14:34:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.706+0000 7f93c93c7640 1 -- 172.21.15.81:0/3165442420 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c40c80d0 con 0x7f93c40b5250 2024-06-29T14:34:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.706+0000 7f93bbfff640 1 -- 172.21.15.81:0/3165442420 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b4004470 con 0x7f93c40b5250 2024-06-29T14:34:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.706+0000 7f93c93c7640 1 -- 172.21.15.81:0/3165442420 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c40b2a70 con 0x7f93c40b5250 2024-06-29T14:34:25.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.707+0000 7f93c93c7640 1 -- 172.21.15.81:0/3165442420 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93c40cec90 con 0x7f93c40b5250 2024-06-29T14:34:25.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.707+0000 7f93bbfff640 1 -- 172.21.15.81:0/3165442420 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f93b40035f0 con 0x7f93c40b5250 2024-06-29T14:34:25.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.708+0000 7f93bbfff640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f93a40764d0 0x7f93a40789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:25.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.708+0000 7f93bbfff640 1 -- 172.21.15.81:0/3165442420 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f93b4098610 con 0x7f93c40b5250 2024-06-29T14:34:25.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.708+0000 7f93ba7fc640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f93a40764d0 0x7f93a40789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:25.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.708+0000 7f93ba7fc640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f93a40764d0 0x7f93a40789e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93c415e350 tx=0x7f93ac007e90 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:25.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.712+0000 7f93bbfff640 1 -- 172.21.15.81:0/3165442420 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f93c40cec90 con 0x7f93c40b5250 2024-06-29T14:34:25.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.913+0000 7f93c93c7640 1 -- 172.21.15.81:0/3165442420 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f93c40b8440 con 0x7f93a40764d0 2024-06-29T14:34:25.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.923+0000 7f93bbfff640 1 -- 172.21.15.81:0/3165442420 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f93c40b8440 con 0x7f93a40764d0 2024-06-29T14:34:25.920 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:25.920 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:53", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:25.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 -- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f93a40764d0 msgr2=0x7f93a40789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f93a40764d0 0x7f93a40789e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93c415e350 tx=0x7f93ac007e90 comp rx=0 tx=0).stop 2024-06-29T14:34:25.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 -- 172.21.15.81:0/3165442420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c40b5250 msgr2=0x7f93c415c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:25.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c40b5250 0x7f93c415c4c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f93b400ea20 tx=0x7f93b400eee0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 -- 172.21.15.81:0/3165442420 shutdown_connections 2024-06-29T14:34:25.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f93a40764d0 0x7f93a40789e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f93c40d2a90 0x7f93c40a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93c40b9040 0x7f93c40a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 --2- 172.21.15.81:0/3165442420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c40b5250 0x7f93c415c4c0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:25.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 -- 172.21.15.81:0/3165442420 >> 172.21.15.81:0/3165442420 conn(0x7f93c409cf30 msgr2=0x7f93c40d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:25.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 -- 172.21.15.81:0/3165442420 shutdown_connections 2024-06-29T14:34:25.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:25.925+0000 7f93c93c7640 1 -- 172.21.15.81:0/3165442420 wait complete. 2024-06-29T14:34:25.933 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:53", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:25.933 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:53', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:25.933 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:25.934 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:25.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:25.934 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-06-29T14:34:25.954 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:25.954 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:25.954 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:25.954 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:25.954 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:25.954 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:25.954 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:25.954 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:25.955 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:25.955 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:25.955 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:25.955 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:25.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2024-06-29T14:34:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.114+0000 7f015c6c5640 1 -- 172.21.15.81:0/3141237233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01540cd540 msgr2=0x7f01540d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.114+0000 7f015c6c5640 1 --2- 172.21.15.81:0/3141237233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01540cd540 0x7f01540d00f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f014c005fd0 tx=0x7f014c02aa90 comp rx=0 tx=0).stop 2024-06-29T14:34:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.114+0000 7f015c6c5640 1 -- 172.21.15.81:0/3141237233 shutdown_connections 2024-06-29T14:34:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.114+0000 7f015c6c5640 1 --2- 172.21.15.81:0/3141237233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01540cd540 0x7f01540d00f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.114+0000 7f015c6c5640 1 --2- 172.21.15.81:0/3141237233 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01540b6d60 0x7f01540ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.114+0000 7f015c6c5640 1 -- 172.21.15.81:0/3141237233 >> 172.21.15.81:0/3141237233 conn(0x7f01540d1960 msgr2=0x7f01540d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.114+0000 7f015c6c5640 1 -- 172.21.15.81:0/3141237233 shutdown_connections 2024-06-29T14:34:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.114+0000 7f015c6c5640 1 -- 172.21.15.81:0/3141237233 wait complete. 2024-06-29T14:34:26.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.116+0000 7f015c6c5640 1 Processor -- start 2024-06-29T14:34:26.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.116+0000 7f015c6c5640 1 -- start start 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f015c6c5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01540b6d60 0x7f01540cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f015c6c5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01540cd540 0x7f015415c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f015c6c5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01540cea10 0x7f015415c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f015c6c5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f01540d07b0 con 0x7f01540cd540 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f015c6c5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f01540d0630 con 0x7f01540b6d60 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f015c6c5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f01540d0930 con 0x7f01540cea10 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f0159c60640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01540cd540 0x7f015415c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f0159c60640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01540cd540 0x7f015415c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42748/0 (socket says 172.21.15.81:42748) 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f0159c60640 1 -- 172.21.15.81:0/1046161873 learned_addr learned my addr 172.21.15.81:0/1046161873 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f015ac62640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01540cea10 0x7f015415c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f0159c60640 1 -- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01540cea10 msgr2=0x7f015415c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f015a461640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01540b6d60 0x7f01540cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f0159c60640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01540cea10 0x7f015415c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f0159c60640 1 -- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01540b6d60 msgr2=0x7f01540cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f0159c60640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01540b6d60 0x7f01540cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f0159c60640 1 -- 172.21.15.81:0/1046161873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01540c8ab0 con 0x7f01540cd540 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f015ac62640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01540cea10 0x7f015415c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f015a461640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01540b6d60 0x7f01540cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.117+0000 7f0159c60640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01540cd540 0x7f015415c170 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f014400c3c0 tx=0x7f014400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.118+0000 7f014affd640 1 -- 172.21.15.81:0/1046161873 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144017070 con 0x7f01540cd540 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.118+0000 7f014affd640 1 -- 172.21.15.81:0/1046161873 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01440035f0 con 0x7f01540cd540 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.118+0000 7f015c6c5640 1 -- 172.21.15.81:0/1046161873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01540c8d50 con 0x7f01540cd540 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.118+0000 7f014affd640 1 -- 172.21.15.81:0/1046161873 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144003a20 con 0x7f01540cd540 2024-06-29T14:34:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.118+0000 7f015c6c5640 1 -- 172.21.15.81:0/1046161873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01540c9310 con 0x7f01540cd540 2024-06-29T14:34:26.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.119+0000 7f015c6c5640 1 -- 172.21.15.81:0/1046161873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01540cb9c0 con 0x7f01540cd540 2024-06-29T14:34:26.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.119+0000 7f014affd640 1 -- 172.21.15.81:0/1046161873 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f014401a070 con 0x7f01540cd540 2024-06-29T14:34:26.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.120+0000 7f014affd640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0138076500 0x7f0138078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.120+0000 7f014affd640 1 -- 172.21.15.81:0/1046161873 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f0144096ed0 con 0x7f01540cd540 2024-06-29T14:34:26.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.120+0000 7f015a461640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0138076500 0x7f0138078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.121+0000 7f015a461640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0138076500 0x7f0138078a10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f01540c5220 tx=0x7f0150005d10 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:26.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.124+0000 7f014affd640 1 -- 172.21.15.81:0/1046161873 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f01440604b0 con 0x7f01540cd540 2024-06-29T14:34:26.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.325+0000 7f015c6c5640 1 -- 172.21.15.81:0/1046161873 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f01540ce4d0 con 0x7f0138076500 2024-06-29T14:34:26.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.335+0000 7f014affd640 1 -- 172.21.15.81:0/1046161873 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f01540ce4d0 con 0x7f0138076500 2024-06-29T14:34:26.332 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:26.333 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:53", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:26.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 -- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0138076500 msgr2=0x7f0138078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0138076500 0x7f0138078a10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f01540c5220 tx=0x7f0150005d10 comp rx=0 tx=0).stop 2024-06-29T14:34:26.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 -- 172.21.15.81:0/1046161873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01540cd540 msgr2=0x7f015415c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01540cd540 0x7f015415c170 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f014400c3c0 tx=0x7f014400c880 comp rx=0 tx=0).stop 2024-06-29T14:34:26.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 -- 172.21.15.81:0/1046161873 shutdown_connections 2024-06-29T14:34:26.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0138076500 0x7f0138078a10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01540cea10 0x7f015415c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01540cd540 0x7f015415c170 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 --2- 172.21.15.81:0/1046161873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01540b6d60 0x7f01540cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 -- 172.21.15.81:0/1046161873 >> 172.21.15.81:0/1046161873 conn(0x7f01540d1960 msgr2=0x7f01540d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:26.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 -- 172.21.15.81:0/1046161873 shutdown_connections 2024-06-29T14:34:26.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.337+0000 7f015c6c5640 1 -- 172.21.15.81:0/1046161873 wait complete. 2024-06-29T14:34:26.345 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:53", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:26.345 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:53', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:26.345 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:26.345 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:26.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:26.346 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-06-29T14:34:26.362 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:26.362 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:26.362 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:26.362 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:26.362 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:26.362 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:26.362 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:26.363 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:26.363 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:26.363 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:26.363 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:26.363 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:26.363 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2024-06-29T14:34:26.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.520+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1457293063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdcc15b5b0 msgr2=0x7fcdcc0b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.520+0000 7fcdd2eac640 1 --2- 172.21.15.81:0/1457293063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdcc15b5b0 0x7fcdcc0b4430 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc005fd0 tx=0x7fcdbc02b880 comp rx=0 tx=0).stop 2024-06-29T14:34:26.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.521+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1457293063 shutdown_connections 2024-06-29T14:34:26.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.521+0000 7fcdd2eac640 1 --2- 172.21.15.81:0/1457293063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdcc15b5b0 0x7fcdcc0b4430 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.521+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1457293063 >> 172.21.15.81:0/1457293063 conn(0x7fcdcc06ccc0 msgr2=0x7fcdcc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:26.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.521+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1457293063 shutdown_connections 2024-06-29T14:34:26.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.521+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1457293063 wait complete. 2024-06-29T14:34:26.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.522+0000 7fcdd2eac640 1 Processor -- start 2024-06-29T14:34:26.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.522+0000 7fcdd2eac640 1 -- start start 2024-06-29T14:34:26.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdd2eac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcdcc06f660 0x7fcdcc150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdd2eac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdcc0b35b0 0x7fcdcc14a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdd2eac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcdcc15b5b0 0x7fcdcc159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdd2eac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcdcc06f460 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdd2eac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcdcc06f2e0 con 0x7fcdcc15b5b0 2024-06-29T14:34:26.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdd2eac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcdcc071f20 con 0x7fcdcc06f660 2024-06-29T14:34:26.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdc3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdcc0b35b0 0x7fcdcc14a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdc3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdcc0b35b0 0x7fcdcc14a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42770/0 (socket says 172.21.15.81:42770) 2024-06-29T14:34:26.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdc3fff640 1 -- 172.21.15.81:0/1289035567 learned_addr learned my addr 172.21.15.81:0/1289035567 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:26.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdd0c48640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcdcc06f660 0x7fcdcc150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdd1449640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcdcc15b5b0 0x7fcdcc159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdc3fff640 1 -- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcdcc06f660 msgr2=0x7fcdcc150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdc3fff640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcdcc06f660 0x7fcdcc150e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdc3fff640 1 -- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcdcc15b5b0 msgr2=0x7fcdcc159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdc3fff640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcdcc15b5b0 0x7fcdcc159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdc3fff640 1 -- 172.21.15.81:0/1289035567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdcc14f9e0 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdd1449640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcdcc15b5b0 0x7fcdcc159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:26.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.524+0000 7fcdc3fff640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdcc0b35b0 0x7fcdcc14a6b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc000c00 tx=0x7fcdbc003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:26.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.525+0000 7fcdc17fa640 1 -- 172.21.15.81:0/1289035567 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc040030 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.525+0000 7fcdc17fa640 1 -- 172.21.15.81:0/1289035567 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdbc003e00 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.525+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1289035567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdcc14fcc0 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.525+0000 7fcdc17fa640 1 -- 172.21.15.81:0/1289035567 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0047a0 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.525+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1289035567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdcc14abf0 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.526+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1289035567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdcc14f750 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.526+0000 7fcdc17fa640 1 -- 172.21.15.81:0/1289035567 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7fcdbc043030 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.527+0000 7fcdc17fa640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fcdb80764d0 0x7fcdb80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.527+0000 7fcdc17fa640 1 -- 172.21.15.81:0/1289035567 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7fcdbc0b8c90 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.527+0000 7fcdd0c48640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fcdb80764d0 0x7fcdb80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.528+0000 7fcdd0c48640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fcdb80764d0 0x7fcdb80789e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8006fd0 tx=0x7fcdc8008040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:26.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.531+0000 7fcdc17fa640 1 -- 172.21.15.81:0/1289035567 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcdbc02f050 con 0x7fcdcc0b35b0 2024-06-29T14:34:26.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.734+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1289035567 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcdcc1531e0 con 0x7fcdb80764d0 2024-06-29T14:34:26.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.743+0000 7fcdc17fa640 1 -- 172.21.15.81:0/1289035567 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7fcdcc1531e0 con 0x7fcdb80764d0 2024-06-29T14:34:26.741 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:26.741 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:54", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:26.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fcdb80764d0 msgr2=0x7fcdb80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fcdb80764d0 0x7fcdb80789e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8006fd0 tx=0x7fcdc8008040 comp rx=0 tx=0).stop 2024-06-29T14:34:26.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1289035567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdcc0b35b0 msgr2=0x7fcdcc14a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdcc0b35b0 0x7fcdcc14a6b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc000c00 tx=0x7fcdbc003910 comp rx=0 tx=0).stop 2024-06-29T14:34:26.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1289035567 shutdown_connections 2024-06-29T14:34:26.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fcdb80764d0 0x7fcdb80789e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcdcc15b5b0 0x7fcdcc159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdcc0b35b0 0x7fcdcc14a6b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 --2- 172.21.15.81:0/1289035567 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcdcc06f660 0x7fcdcc150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1289035567 >> 172.21.15.81:0/1289035567 conn(0x7fcdcc06ccc0 msgr2=0x7fcdcc14b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:26.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1289035567 shutdown_connections 2024-06-29T14:34:26.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.745+0000 7fcdd2eac640 1 -- 172.21.15.81:0/1289035567 wait complete. 2024-06-29T14:34:26.755 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:54", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:26.755 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:54', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:26.755 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:26.755 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:26.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:26.755 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-06-29T14:34:26.771 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:26.772 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:26.772 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:26.772 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:26.772 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:26.772 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:26.772 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:26.772 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:26.772 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:26.773 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:26.773 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:26.773 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:26.773 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2024-06-29T14:34:26.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.930+0000 7f50c7247640 1 -- 172.21.15.81:0/465459920 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00b9040 msgr2=0x7f50c00d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.930+0000 7f50c7247640 1 --2- 172.21.15.81:0/465459920 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00b9040 0x7f50c00d0480 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f50bc009830 tx=0x7f50bc02a5a0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.930+0000 7f50c7247640 1 -- 172.21.15.81:0/465459920 shutdown_connections 2024-06-29T14:34:26.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.930+0000 7f50c7247640 1 --2- 172.21.15.81:0/465459920 >> v2:172.21.15.167:3300/0 conn(0x7f50c00d2a90 0x7f50c00d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.930+0000 7f50c7247640 1 --2- 172.21.15.81:0/465459920 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00b9040 0x7f50c00d0480 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.930+0000 7f50c7247640 1 -- 172.21.15.81:0/465459920 >> 172.21.15.81:0/465459920 conn(0x7f50c009cf30 msgr2=0x7f50c00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:26.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.930+0000 7f50c7247640 1 -- 172.21.15.81:0/465459920 shutdown_connections 2024-06-29T14:34:26.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.930+0000 7f50c7247640 1 -- 172.21.15.81:0/465459920 wait complete. 2024-06-29T14:34:26.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.931+0000 7f50c7247640 1 Processor -- start 2024-06-29T14:34:26.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.932+0000 7f50c7247640 1 -- start start 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50c7247640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00b5250 0x7f50c0158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50c7247640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b9040 0x7f50c00cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50c7247640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50c00d2a90 0x7f50c00cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50c7247640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f50c00b7d30 con 0x7f50c00b9040 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50c7247640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f50c00b7bb0 con 0x7f50c00d2a90 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50c7247640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f50c00b7eb0 con 0x7f50c00b5250 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50b7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b9040 0x7f50c00cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50b7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b9040 0x7f50c00cc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42776/0 (socket says 172.21.15.81:42776) 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50b7fff640 1 -- 172.21.15.81:0/1377006474 learned_addr learned my addr 172.21.15.81:0/1377006474 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50c4fe3640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00b5250 0x7f50c0158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50c57e4640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50c00d2a90 0x7f50c00cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50b7fff640 1 -- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00b5250 msgr2=0x7f50c0158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50b7fff640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00b5250 0x7f50c0158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50b7fff640 1 -- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50c00d2a90 msgr2=0x7f50c00cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:26.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50b7fff640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50c00d2a90 0x7f50c00cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:26.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.933+0000 7f50b7fff640 1 -- 172.21.15.81:0/1377006474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c015bbe0 con 0x7f50c00b9040 2024-06-29T14:34:26.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.934+0000 7f50b7fff640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b9040 0x7f50c00cc3d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f50b000c3a0 tx=0x7f50b000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:26.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.934+0000 7f50b57fa640 1 -- 172.21.15.81:0/1377006474 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b0017070 con 0x7f50c00b9040 2024-06-29T14:34:26.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.934+0000 7f50b57fa640 1 -- 172.21.15.81:0/1377006474 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f50b00053c0 con 0x7f50c00b9040 2024-06-29T14:34:26.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.934+0000 7f50c7247640 1 -- 172.21.15.81:0/1377006474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c01594f0 con 0x7f50c00b9040 2024-06-29T14:34:26.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.934+0000 7f50b57fa640 1 -- 172.21.15.81:0/1377006474 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b0003850 con 0x7f50c00b9040 2024-06-29T14:34:26.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.934+0000 7f50c7247640 1 -- 172.21.15.81:0/1377006474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c0159ab0 con 0x7f50c00b9040 2024-06-29T14:34:26.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.935+0000 7f50c7247640 1 -- 172.21.15.81:0/1377006474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50c00b83b0 con 0x7f50c00b9040 2024-06-29T14:34:26.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.936+0000 7f50b57fa640 1 -- 172.21.15.81:0/1377006474 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f50b001a070 con 0x7f50c00b9040 2024-06-29T14:34:26.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.936+0000 7f50b57fa640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f50ac076500 0x7f50ac078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:26.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.936+0000 7f50b57fa640 1 -- 172.21.15.81:0/1377006474 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f50b0096e90 con 0x7f50c00b9040 2024-06-29T14:34:26.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.937+0000 7f50c4fe3640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f50ac076500 0x7f50ac078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:26.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.937+0000 7f50c4fe3640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f50ac076500 0x7f50ac078a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f50bc034000 tx=0x7f50bc004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:26.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:26.941+0000 7f50b57fa640 1 -- 172.21.15.81:0/1377006474 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f50b005fe50 con 0x7f50c00b9040 2024-06-29T14:34:27.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.140+0000 7f50c7247640 1 -- 172.21.15.81:0/1377006474 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f50c00b8b90 con 0x7f50ac076500 2024-06-29T14:34:27.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.149+0000 7f50b57fa640 1 -- 172.21.15.81:0/1377006474 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f50c00b8b90 con 0x7f50ac076500 2024-06-29T14:34:27.147 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:27.147 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:54", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:27.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 -- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f50ac076500 msgr2=0x7f50ac078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f50ac076500 0x7f50ac078a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f50bc034000 tx=0x7f50bc004730 comp rx=0 tx=0).stop 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 -- 172.21.15.81:0/1377006474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b9040 msgr2=0x7f50c00cc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b9040 0x7f50c00cc3d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f50b000c3a0 tx=0x7f50b000c860 comp rx=0 tx=0).stop 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 -- 172.21.15.81:0/1377006474 shutdown_connections 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f50ac076500 0x7f50ac078a10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50c00d2a90 0x7f50c00cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b9040 0x7f50c00cc3d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 --2- 172.21.15.81:0/1377006474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00b5250 0x7f50c0158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 -- 172.21.15.81:0/1377006474 >> 172.21.15.81:0/1377006474 conn(0x7f50c009cf30 msgr2=0x7f50c006ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 -- 172.21.15.81:0/1377006474 shutdown_connections 2024-06-29T14:34:27.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.151+0000 7f50c7247640 1 -- 172.21.15.81:0/1377006474 wait complete. 2024-06-29T14:34:27.161 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:54", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:27.162 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:54', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:27.162 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:27.162 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:27.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:27.162 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-06-29T14:34:27.178 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:27.179 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:27.179 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:27.179 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:27.179 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:27.179 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:27.179 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:27.179 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:27.179 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:27.179 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:27.179 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:27.180 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_24_00_UTC', 'scheduled-2024-06-29-14_25_00_UTC', 'scheduled-2024-06-29-14_26_00_UTC', 'scheduled-2024-06-29-14_27_00_UTC', 'scheduled-2024-06-29-14_28_00_UTC', 'scheduled-2024-06-29-14_29_00_UTC', 'scheduled-2024-06-29-14_30_00_UTC', 'scheduled-2024-06-29-14_31_00_UTC', 'scheduled-2024-06-29-14_32_00_UTC', 'scheduled-2024-06-29-14_33_00_UTC', 'scheduled-2024-06-29-14_34_00_UTC'] 2024-06-29T14:34:27.180 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2024-06-29T14:34:27.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.337+0000 7f0786ed6640 1 -- 172.21.15.81:0/3514024639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800b6eb0 msgr2=0x7f07800d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.337+0000 7f0786ed6640 1 --2- 172.21.15.81:0/3514024639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800b6eb0 0x7f07800d02d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f07800b41b0 tx=0x7f077c02a490 comp rx=0 tx=0).stop 2024-06-29T14:34:27.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.337+0000 7f0786ed6640 1 -- 172.21.15.81:0/3514024639 shutdown_connections 2024-06-29T14:34:27.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.337+0000 7f0786ed6640 1 --2- 172.21.15.81:0/3514024639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07800ce510 0x7f07800ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.337+0000 7f0786ed6640 1 --2- 172.21.15.81:0/3514024639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800b6eb0 0x7f07800d02d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.337+0000 7f0786ed6640 1 -- 172.21.15.81:0/3514024639 >> 172.21.15.81:0/3514024639 conn(0x7f07800b3510 msgr2=0x7f07800b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:27.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.337+0000 7f0786ed6640 1 -- 172.21.15.81:0/3514024639 shutdown_connections 2024-06-29T14:34:27.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.337+0000 7f0786ed6640 1 -- 172.21.15.81:0/3514024639 wait complete. 2024-06-29T14:34:27.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.339+0000 7f0786ed6640 1 Processor -- start 2024-06-29T14:34:27.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.339+0000 7f0786ed6640 1 -- start start 2024-06-29T14:34:27.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.340+0000 7f0786ed6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07800b4810 0x7f078015cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.340+0000 7f0786ed6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07800b6eb0 0x7f078009ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.340+0000 7f0786ed6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ce510 0x7f078009f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.340+0000 7f0786ed6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f07800cd480 con 0x7f07800ce510 2024-06-29T14:34:27.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.340+0000 7f0786ed6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f07800b29c0 con 0x7f07800b6eb0 2024-06-29T14:34:27.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.340+0000 7f0786ed6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f07800cd600 con 0x7f07800b4810 2024-06-29T14:34:27.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.340+0000 7f0785473640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ce510 0x7f078009f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.340+0000 7f0785473640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ce510 0x7f078009f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42788/0 (socket says 172.21.15.81:42788) 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f0785473640 1 -- 172.21.15.81:0/2713652831 learned_addr learned my addr 172.21.15.81:0/2713652831 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f0777fff640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07800b6eb0 0x7f078009ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f0785473640 1 -- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07800b4810 msgr2=0x7f078015cca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f0785473640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07800b4810 0x7f078015cca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f0785473640 1 -- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07800b6eb0 msgr2=0x7f078009ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f0785473640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07800b6eb0 0x7f078009ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f0785473640 1 -- 172.21.15.81:0/2713652831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078015fca0 con 0x7f07800ce510 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f0785473640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ce510 0x7f078009f380 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f077800b9a0 tx=0x7f077800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f07757fa640 1 -- 172.21.15.81:0/2713652831 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778019070 con 0x7f07800ce510 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f07757fa640 1 -- 172.21.15.81:0/2713652831 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f077800cc90 con 0x7f07800ce510 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f0786ed6640 1 -- 172.21.15.81:0/2713652831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078015ff80 con 0x7f07800ce510 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f07757fa640 1 -- 172.21.15.81:0/2713652831 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780146a0 con 0x7f07800ce510 2024-06-29T14:34:27.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.341+0000 7f0786ed6640 1 -- 172.21.15.81:0/2713652831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0780160540 con 0x7f07800ce510 2024-06-29T14:34:27.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.342+0000 7f0786ed6640 1 -- 172.21.15.81:0/2713652831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07800d02d0 con 0x7f07800ce510 2024-06-29T14:34:27.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.343+0000 7f07757fa640 1 -- 172.21.15.81:0/2713652831 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f077801c070 con 0x7f07800ce510 2024-06-29T14:34:27.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.343+0000 7f07757fa640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f076c0764d0 0x7f076c0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.344+0000 7f07757fa640 1 -- 172.21.15.81:0/2713652831 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f0778097960 con 0x7f07800ce510 2024-06-29T14:34:27.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.344+0000 7f0784c72640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f076c0764d0 0x7f076c0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.344+0000 7f0784c72640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f076c0764d0 0x7f076c0789e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f077c033000 tx=0x7f077c004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:27.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.348+0000 7f07757fa640 1 -- 172.21.15.81:0/2713652831 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0778060eb0 con 0x7f07800ce510 2024-06-29T14:34:27.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.551+0000 7f0786ed6640 1 -- 172.21.15.81:0/2713652831 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f07800cdf00 con 0x7f076c0764d0 2024-06-29T14:34:27.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.561+0000 7f07757fa640 1 -- 172.21.15.81:0/2713652831 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+360 (secure 0 0 0) 0x7f07800cdf00 con 0x7f076c0764d0 2024-06-29T14:34:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:27.559 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:55", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 -- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f076c0764d0 msgr2=0x7f076c0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f076c0764d0 0x7f076c0789e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f077c033000 tx=0x7f077c004730 comp rx=0 tx=0).stop 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 -- 172.21.15.81:0/2713652831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ce510 msgr2=0x7f078009f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ce510 0x7f078009f380 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f077800b9a0 tx=0x7f077800be60 comp rx=0 tx=0).stop 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 -- 172.21.15.81:0/2713652831 shutdown_connections 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f076c0764d0 0x7f076c0789e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ce510 0x7f078009f380 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07800b6eb0 0x7f078009ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 --2- 172.21.15.81:0/2713652831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07800b4810 0x7f078015cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 -- 172.21.15.81:0/2713652831 >> 172.21.15.81:0/2713652831 conn(0x7f07800b3510 msgr2=0x7f07800b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 -- 172.21.15.81:0/2713652831 shutdown_connections 2024-06-29T14:34:27.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.563+0000 7f0786ed6640 1 -- 172.21.15.81:0/2713652831 wait complete. 2024-06-29T14:34:27.571 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:23:55", "first": "2024-06-29T14:24:00", "last": "2024-06-29T14:34:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2024-06-29T14:34:27.572 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:23:55', 'first': '2024-06-29T14:24:00', 'last': '2024-06-29T14:34:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2024-06-29T14:34:27.572 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T14:34:27.572 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T14:34:27.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2024-06-29T14:34:27.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.691+0000 7f77dc1a9640 1 -- 172.21.15.81:0/2242056247 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77d4073740 msgr2=0x7f77d415e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.691+0000 7f77dc1a9640 1 --2- 172.21.15.81:0/2242056247 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77d4073740 0x7f77d415e960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f77cc005fd0 tx=0x7f77cc02aa90 comp rx=0 tx=0).stop 2024-06-29T14:34:27.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.691+0000 7f77dc1a9640 1 -- 172.21.15.81:0/2242056247 shutdown_connections 2024-06-29T14:34:27.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.691+0000 7f77dc1a9640 1 --2- 172.21.15.81:0/2242056247 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77d4073740 0x7f77d415e960 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.691+0000 7f77dc1a9640 1 --2- 172.21.15.81:0/2242056247 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77d41412f0 0x7f77d40cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.691+0000 7f77dc1a9640 1 -- 172.21.15.81:0/2242056247 >> 172.21.15.81:0/2242056247 conn(0x7f77d406ccc0 msgr2=0x7f77d40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:27.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.691+0000 7f77dc1a9640 1 -- 172.21.15.81:0/2242056247 shutdown_connections 2024-06-29T14:34:27.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.691+0000 7f77dc1a9640 1 -- 172.21.15.81:0/2242056247 wait complete. 2024-06-29T14:34:27.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.693+0000 7f77dc1a9640 1 Processor -- start 2024-06-29T14:34:27.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.693+0000 7f77dc1a9640 1 -- start start 2024-06-29T14:34:27.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77dc1a9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77d40cf4d0 0x7f77d41f4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77dc1a9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77d41412f0 0x7f77d41f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77dc1a9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77d413fdb0 0x7f77d41401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77dc1a9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f77d40b3210 con 0x7f77d413fdb0 2024-06-29T14:34:27.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77dc1a9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f77d415f240 con 0x7f77d40cf4d0 2024-06-29T14:34:27.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77dc1a9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f77d40b3390 con 0x7f77d41412f0 2024-06-29T14:34:27.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77da746640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77d413fdb0 0x7f77d41401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77da746640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77d413fdb0 0x7f77d41401c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42792/0 (socket says 172.21.15.81:42792) 2024-06-29T14:34:27.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77da746640 1 -- 172.21.15.81:0/725352933 learned_addr learned my addr 172.21.15.81:0/725352933 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:27.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77d9f45640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77d40cf4d0 0x7f77d41f4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.694+0000 7f77d9744640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77d41412f0 0x7f77d41f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77da746640 1 -- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77d41412f0 msgr2=0x7f77d41f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77da746640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77d41412f0 0x7f77d41f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77da746640 1 -- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77d40cf4d0 msgr2=0x7f77d41f4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77da746640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77d40cf4d0 0x7f77d41f4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77da746640 1 -- 172.21.15.81:0/725352933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d41f5d90 con 0x7f77d413fdb0 2024-06-29T14:34:27.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77da746640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77d413fdb0 0x7f77d41401c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f77cc009bf0 tx=0x7f77cc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:27.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77ca7fc640 1 -- 172.21.15.81:0/725352933 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77cc044070 con 0x7f77d413fdb0 2024-06-29T14:34:27.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77dc1a9640 1 -- 172.21.15.81:0/725352933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d4137080 con 0x7f77d413fdb0 2024-06-29T14:34:27.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77ca7fc640 1 -- 172.21.15.81:0/725352933 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77cc003e00 con 0x7f77d413fdb0 2024-06-29T14:34:27.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77ca7fc640 1 -- 172.21.15.81:0/725352933 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77cc0044b0 con 0x7f77d413fdb0 2024-06-29T14:34:27.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.695+0000 7f77dc1a9640 1 -- 172.21.15.81:0/725352933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d40af680 con 0x7f77d413fdb0 2024-06-29T14:34:27.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.696+0000 7f77dc1a9640 1 -- 172.21.15.81:0/725352933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77d415b400 con 0x7f77d413fdb0 2024-06-29T14:34:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.697+0000 7f77ca7fc640 1 -- 172.21.15.81:0/725352933 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f77cc047070 con 0x7f77d413fdb0 2024-06-29T14:34:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.697+0000 7f77ca7fc640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f77b80764d0 0x7f77b80789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.697+0000 7f77ca7fc640 1 -- 172.21.15.81:0/725352933 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f77cc0b8c00 con 0x7f77d413fdb0 2024-06-29T14:34:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.697+0000 7f77d9f45640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f77b80764d0 0x7f77b80789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.698+0000 7f77d9f45640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f77b80764d0 0x7f77b80789e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009ac0 tx=0x7f77d0005f90 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:27.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.701+0000 7f77ca7fc640 1 -- 172.21.15.81:0/725352933 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f77d415b400 con 0x7f77d413fdb0 2024-06-29T14:34:27.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.904+0000 7f77dc1a9640 1 -- 172.21.15.81:0/725352933 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f77d40b38b0 con 0x7f77b80764d0 2024-06-29T14:34:27.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.997+0000 7f77ca7fc640 1 -- 172.21.15.81:0/725352933 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f77d40b38b0 con 0x7f77b80764d0 2024-06-29T14:34:27.995 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/dir0 2024-06-29T14:34:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.999+0000 7f77dc1a9640 1 -- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f77b80764d0 msgr2=0x7f77b80789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.999+0000 7f77dc1a9640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f77b80764d0 0x7f77b80789e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009ac0 tx=0x7f77d0005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:27.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.999+0000 7f77dc1a9640 1 -- 172.21.15.81:0/725352933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77d413fdb0 msgr2=0x7f77d41401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:27.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:27.999+0000 7f77dc1a9640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77d413fdb0 0x7f77d41401c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f77cc009bf0 tx=0x7f77cc003c10 comp rx=0 tx=0).stop 2024-06-29T14:34:27.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.000+0000 7f77dc1a9640 1 -- 172.21.15.81:0/725352933 shutdown_connections 2024-06-29T14:34:27.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.000+0000 7f77dc1a9640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f77b80764d0 0x7f77b80789e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.000+0000 7f77dc1a9640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77d413fdb0 0x7f77d41401c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.000+0000 7f77dc1a9640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77d41412f0 0x7f77d41f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.000+0000 7f77dc1a9640 1 --2- 172.21.15.81:0/725352933 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77d40cf4d0 0x7f77d41f4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:27.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.000+0000 7f77dc1a9640 1 -- 172.21.15.81:0/725352933 >> 172.21.15.81:0/725352933 conn(0x7f77d406ccc0 msgr2=0x7f77d406df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:27.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.000+0000 7f77dc1a9640 1 -- 172.21.15.81:0/725352933 shutdown_connections 2024-06-29T14:34:27.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.000+0000 7f77dc1a9640 1 -- 172.21.15.81:0/725352933 wait complete. 2024-06-29T14:34:28.008 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2024-06-29T14:34:28.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.008 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2024-06-29T14:34:28.024 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:28.024 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:28.024 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:28.024 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:28.025 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:28.025 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:28.025 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:28.025 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_01_UTC 2024-06-29T14:34:28.025 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:28.025 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:28.025 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:28.025 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:28.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.025 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:28.086 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:28.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.086 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:28.148 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:28.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.148 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:28.212 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:28.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.212 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:28.274 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:28.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.274 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:28.338 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:28.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.338 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:28.400 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:28.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.401 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:28.462 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_31_01_UTC 2024-06-29T14:34:28.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.462 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_31_01_UTC) 2024-06-29T14:34:28.524 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:28.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.524 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:28.586 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:28.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.586 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:28.648 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:28.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.648 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:28.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:28.710 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2024-06-29T14:34:28.769 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2024-06-29T14:34:28.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.927+0000 7f0d20124640 1 -- 172.21.15.81:0/1966483329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d18073990 msgr2=0x7f0d1815c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.927+0000 7f0d20124640 1 --2- 172.21.15.81:0/1966483329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d18073990 0x7f0d1815c5f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0d08005fd0 tx=0x7f0d0802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.927+0000 7f0d20124640 1 -- 172.21.15.81:0/1966483329 shutdown_connections 2024-06-29T14:34:28.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.927+0000 7f0d20124640 1 --2- 172.21.15.81:0/1966483329 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d1815cb30 0x7f0d1815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.927+0000 7f0d20124640 1 --2- 172.21.15.81:0/1966483329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d18073990 0x7f0d1815c5f0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.927+0000 7f0d20124640 1 -- 172.21.15.81:0/1966483329 >> 172.21.15.81:0/1966483329 conn(0x7f0d1806ccc0 msgr2=0x7f0d180b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:28.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.927+0000 7f0d20124640 1 -- 172.21.15.81:0/1966483329 shutdown_connections 2024-06-29T14:34:28.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.927+0000 7f0d20124640 1 -- 172.21.15.81:0/1966483329 wait complete. 2024-06-29T14:34:28.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.929+0000 7f0d20124640 1 Processor -- start 2024-06-29T14:34:28.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.929+0000 7f0d20124640 1 -- start start 2024-06-29T14:34:28.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.930+0000 7f0d20124640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d18145530 0x7f0d18157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.930+0000 7f0d20124640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1815cb30 0x7f0d181f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.930+0000 7f0d20124640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d181f3380 0x7f0d180a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.930+0000 7f0d20124640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1815dde0 con 0x7f0d1815cb30 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.930+0000 7f0d20124640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1815dc60 con 0x7f0d181f3380 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.930+0000 7f0d20124640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1815df60 con 0x7f0d18145530 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1d6bf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1815cb30 0x7f0d181f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1d6bf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1815cb30 0x7f0d181f2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42806/0 (socket says 172.21.15.81:42806) 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1d6bf640 1 -- 172.21.15.81:0/3960289070 learned_addr learned my addr 172.21.15.81:0/3960289070 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1dec0640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d18145530 0x7f0d18157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1e6c1640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d181f3380 0x7f0d180a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1d6bf640 1 -- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d18145530 msgr2=0x7f0d18157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1d6bf640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d18145530 0x7f0d18157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1d6bf640 1 -- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d181f3380 msgr2=0x7f0d180a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:28.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1d6bf640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d181f3380 0x7f0d180a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:28.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1d6bf640 1 -- 172.21.15.81:0/3960289070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d18158360 con 0x7f0d1815cb30 2024-06-29T14:34:28.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1dec0640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d18145530 0x7f0d18157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:28.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d1d6bf640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1815cb30 0x7f0d181f2e40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0d08034040 tx=0x7f0d080038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:28.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d0e7fc640 1 -- 172.21.15.81:0/3960289070 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0803e030 con 0x7f0d1815cb30 2024-06-29T14:34:28.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d0e7fc640 1 -- 172.21.15.81:0/3960289070 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d08003ca0 con 0x7f0d1815cb30 2024-06-29T14:34:28.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d20124640 1 -- 172.21.15.81:0/3960289070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d18156d80 con 0x7f0d1815cb30 2024-06-29T14:34:28.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d0e7fc640 1 -- 172.21.15.81:0/3960289070 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08035040 con 0x7f0d1815cb30 2024-06-29T14:34:28.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.931+0000 7f0d20124640 1 -- 172.21.15.81:0/3960289070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d18157340 con 0x7f0d1815cb30 2024-06-29T14:34:28.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.932+0000 7f0d20124640 1 -- 172.21.15.81:0/3960289070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d18156ad0 con 0x7f0d1815cb30 2024-06-29T14:34:28.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.933+0000 7f0d0e7fc640 1 -- 172.21.15.81:0/3960289070 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f0d08041030 con 0x7f0d1815cb30 2024-06-29T14:34:28.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.934+0000 7f0d0e7fc640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0cfc076500 0x7f0cfc078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:28.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.934+0000 7f0d0e7fc640 1 -- 172.21.15.81:0/3960289070 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5967+0+0 (secure 0 0 0) 0x7f0d080b8270 con 0x7f0d1815cb30 2024-06-29T14:34:28.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.934+0000 7f0d1dec0640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0cfc076500 0x7f0cfc078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:28.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.934+0000 7f0d1dec0640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0cfc076500 0x7f0cfc078a10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0d14007fd0 tx=0x7f0d14009040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:28.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:28.938+0000 7f0d0e7fc640 1 -- 172.21.15.81:0/3960289070 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0d08081740 con 0x7f0d1815cb30 2024-06-29T14:34:29.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.139+0000 7f0d20124640 1 -- 172.21.15.81:0/3960289070 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f0d18144ef0 con 0x7f0cfc076500 2024-06-29T14:34:29.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.237+0000 7f0d0e7fc640 1 -- 172.21.15.81:0/3960289070 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f0d18144ef0 con 0x7f0cfc076500 2024-06-29T14:34:29.235 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/dir1 2024-06-29T14:34:29.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 -- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0cfc076500 msgr2=0x7f0cfc078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0cfc076500 0x7f0cfc078a10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0d14007fd0 tx=0x7f0d14009040 comp rx=0 tx=0).stop 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 -- 172.21.15.81:0/3960289070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1815cb30 msgr2=0x7f0d181f2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1815cb30 0x7f0d181f2e40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0d08034040 tx=0x7f0d080038a0 comp rx=0 tx=0).stop 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 -- 172.21.15.81:0/3960289070 shutdown_connections 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0cfc076500 0x7f0cfc078a10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d181f3380 0x7f0d180a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1815cb30 0x7f0d181f2e40 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 --2- 172.21.15.81:0/3960289070 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d18145530 0x7f0d18157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 -- 172.21.15.81:0/3960289070 >> 172.21.15.81:0/3960289070 conn(0x7f0d1806ccc0 msgr2=0x7f0d1815f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 -- 172.21.15.81:0/3960289070 shutdown_connections 2024-06-29T14:34:29.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:29.239+0000 7f0d20124640 1 -- 172.21.15.81:0/3960289070 wait complete. 2024-06-29T14:34:29.247 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2024-06-29T14:34:29.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.247 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2024-06-29T14:34:29.263 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:29.263 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:29.264 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:29.264 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:29.264 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:29.264 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:29.264 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:29.264 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:29.264 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:29.264 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:29.264 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:29.264 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:29.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.265 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:29.327 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:29.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.328 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:29.388 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:29.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.389 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:29.451 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:29.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.451 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:29.513 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:29.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.514 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:29.577 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:29.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.577 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:29.639 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:29.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.640 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:29.701 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:29.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.701 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:29.762 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:29.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.763 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:29.826 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:29.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.827 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:29.888 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:29.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.889 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:29.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:29.950 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2024-06-29T14:34:30.009 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2024-06-29T14:34:30.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.171+0000 7f6e22847640 1 -- 172.21.15.81:0/4229236728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e1c0b5250 msgr2=0x7f6e1c0d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.171+0000 7f6e22847640 1 --2- 172.21.15.81:0/4229236728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e1c0b5250 0x7f6e1c0d1fc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6e14005fd0 tx=0x7f6e1402a930 comp rx=0 tx=0).stop 2024-06-29T14:34:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.171+0000 7f6e22847640 1 -- 172.21.15.81:0/4229236728 shutdown_connections 2024-06-29T14:34:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.171+0000 7f6e22847640 1 --2- 172.21.15.81:0/4229236728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e1c0b5250 0x7f6e1c0d1fc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.171+0000 7f6e22847640 1 --2- 172.21.15.81:0/4229236728 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e1c0d0b90 0x7f6e1c0d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.171+0000 7f6e22847640 1 -- 172.21.15.81:0/4229236728 >> 172.21.15.81:0/4229236728 conn(0x7f6e1c09cf30 msgr2=0x7f6e1c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.171+0000 7f6e22847640 1 -- 172.21.15.81:0/4229236728 shutdown_connections 2024-06-29T14:34:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.171+0000 7f6e22847640 1 -- 172.21.15.81:0/4229236728 wait complete. 2024-06-29T14:34:30.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.174+0000 7f6e22847640 1 Processor -- start 2024-06-29T14:34:30.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.174+0000 7f6e22847640 1 -- start start 2024-06-29T14:34:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e22847640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e1c0b5250 0x7f6e1c0a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e22847640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e1c0b9040 0x7f6e1c15c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e22847640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e1c0a1250 0x7f6e1c15c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e22847640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1c0d2680 con 0x7f6e1c0b9040 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e22847640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1c0d2500 con 0x7f6e1c0b5250 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e22847640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6e1c0d2800 con 0x7f6e1c0a1250 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e1b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e1c0b9040 0x7f6e1c15c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e1b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e1c0b9040 0x7f6e1c15c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59222/0 (socket says 172.21.15.81:59222) 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e1b7fe640 1 -- 172.21.15.81:0/3881733028 learned_addr learned my addr 172.21.15.81:0/3881733028 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e20de4640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e1c0a1250 0x7f6e1c15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.176+0000 7f6e1bfff640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e1c0b5250 0x7f6e1c0a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.177+0000 7f6e1b7fe640 1 -- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e1c0a1250 msgr2=0x7f6e1c15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.177+0000 7f6e1b7fe640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e1c0a1250 0x7f6e1c15c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.177+0000 7f6e1b7fe640 1 -- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e1c0b5250 msgr2=0x7f6e1c0a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.177+0000 7f6e1b7fe640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e1c0b5250 0x7f6e1c0a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.177+0000 7f6e1b7fe640 1 -- 172.21.15.81:0/3881733028 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e1c0c9bd0 con 0x7f6e1c0b9040 2024-06-29T14:34:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.177+0000 7f6e1b7fe640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e1c0b9040 0x7f6e1c15c2e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6e1000e710 tx=0x7f6e1000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:30.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.177+0000 7f6e18ff9640 1 -- 172.21.15.81:0/3881733028 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1000cab0 con 0x7f6e1c0b9040 2024-06-29T14:34:30.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.178+0000 7f6e18ff9640 1 -- 172.21.15.81:0/3881733028 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e1000cd40 con 0x7f6e1c0b9040 2024-06-29T14:34:30.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.178+0000 7f6e22847640 1 -- 172.21.15.81:0/3881733028 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e1c09db80 con 0x7f6e1c0b9040 2024-06-29T14:34:30.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.178+0000 7f6e18ff9640 1 -- 172.21.15.81:0/3881733028 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e10010690 con 0x7f6e1c0b9040 2024-06-29T14:34:30.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.178+0000 7f6e22847640 1 -- 172.21.15.81:0/3881733028 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e1c0a1820 con 0x7f6e1c0b9040 2024-06-29T14:34:30.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.180+0000 7f6e22847640 1 -- 172.21.15.81:0/3881733028 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e1c09d9a0 con 0x7f6e1c0b9040 2024-06-29T14:34:30.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.180+0000 7f6e18ff9640 1 -- 172.21.15.81:0/3881733028 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f6e10010830 con 0x7f6e1c0b9040 2024-06-29T14:34:30.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.181+0000 7f6e18ff9640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6e08076520 0x7f6e08078a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:30.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.181+0000 7f6e18ff9640 1 -- 172.21.15.81:0/3881733028 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6631+0+0 (secure 0 0 0) 0x7f6e100987b0 con 0x7f6e1c0b9040 2024-06-29T14:34:30.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.182+0000 7f6e1bfff640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6e08076520 0x7f6e08078a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:30.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.183+0000 7f6e1bfff640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6e08076520 0x7f6e08078a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c0015e0 tx=0x7f6e0c009040 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:30.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.189+0000 7f6e18ff9640 1 -- 172.21.15.81:0/3881733028 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6e1c09d9a0 con 0x7f6e1c0b9040 2024-06-29T14:34:30.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.459+0000 7f6e22847640 1 -- 172.21.15.81:0/3881733028 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6e1c0d2d90 con 0x7f6e08076520 2024-06-29T14:34:30.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.601+0000 7f6e18ff9640 1 -- 172.21.15.81:0/3881733028 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f6e1c0d2d90 con 0x7f6e08076520 2024-06-29T14:34:30.599 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/dir2 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 -- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6e08076520 msgr2=0x7f6e08078a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6e08076520 0x7f6e08078a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c0015e0 tx=0x7f6e0c009040 comp rx=0 tx=0).stop 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 -- 172.21.15.81:0/3881733028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e1c0b9040 msgr2=0x7f6e1c15c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e1c0b9040 0x7f6e1c15c2e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6e1000e710 tx=0x7f6e1000ebd0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 -- 172.21.15.81:0/3881733028 shutdown_connections 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6e08076520 0x7f6e08078a30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e1c0a1250 0x7f6e1c15c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e1c0b9040 0x7f6e1c15c2e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 --2- 172.21.15.81:0/3881733028 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e1c0b5250 0x7f6e1c0a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 -- 172.21.15.81:0/3881733028 >> 172.21.15.81:0/3881733028 conn(0x7f6e1c09cf30 msgr2=0x7f6e1c15e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 -- 172.21.15.81:0/3881733028 shutdown_connections 2024-06-29T14:34:30.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:30.604+0000 7f6e22847640 1 -- 172.21.15.81:0/3881733028 wait complete. 2024-06-29T14:34:30.612 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2024-06-29T14:34:30.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.612 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2024-06-29T14:34:30.629 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:30.629 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:30.629 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:30.629 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:30.629 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:30.629 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:30.629 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:30.630 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:30.630 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:30.630 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:30.630 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:30.630 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:30.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.630 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:30.691 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:30.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.692 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:30.754 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:30.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.755 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:30.818 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:30.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.818 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:30.880 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:30.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.880 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:30.942 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:30.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:30.942 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:31.004 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:31.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.004 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:31.066 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:31.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.066 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:31.128 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:31.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.128 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:31.192 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:31.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.192 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:31.254 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:31.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.254 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:31.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.316 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2024-06-29T14:34:31.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2024-06-29T14:34:31.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.534+0000 7ff5fee74640 1 -- 172.21.15.81:0/498059029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5e00053a0 msgr2=0x7ff5e000ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:31.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.534+0000 7ff5fee74640 1 --2- 172.21.15.81:0/498059029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5e00053a0 0x7ff5e000ba50 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff5f806e210 tx=0x7ff5e0005830 comp rx=0 tx=0).stop 2024-06-29T14:34:31.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.534+0000 7ff5fee74640 1 -- 172.21.15.81:0/498059029 shutdown_connections 2024-06-29T14:34:31.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.534+0000 7ff5fee74640 1 --2- 172.21.15.81:0/498059029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5e00053a0 0x7ff5e000ba50 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.534+0000 7ff5fee74640 1 -- 172.21.15.81:0/498059029 >> 172.21.15.81:0/498059029 conn(0x7ff5f8072950 msgr2=0x7ff5f80b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:31.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.534+0000 7ff5fee74640 1 -- 172.21.15.81:0/498059029 shutdown_connections 2024-06-29T14:34:31.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.534+0000 7ff5fee74640 1 -- 172.21.15.81:0/498059029 wait complete. 2024-06-29T14:34:31.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.536+0000 7ff5fee74640 1 Processor -- start 2024-06-29T14:34:31.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.536+0000 7ff5fee74640 1 -- start start 2024-06-29T14:34:31.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.537+0000 7ff5fee74640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5e00053a0 0x7ff5f814d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:31.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.537+0000 7ff5fee74640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5f815cfd0 0x7ff5f815d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:31.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.537+0000 7ff5fee74640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5f8153b60 0x7ff5f8153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:31.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.537+0000 7ff5fee74640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f815bdf0 con 0x7ff5f8153b60 2024-06-29T14:34:31.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.537+0000 7ff5fee74640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f8074460 con 0x7ff5f815cfd0 2024-06-29T14:34:31.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.537+0000 7ff5fee74640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f815bf70 con 0x7ff5e00053a0 2024-06-29T14:34:31.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fd411640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5f8153b60 0x7ff5f8153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fd411640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5f8153b60 0x7ff5f8153f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59230/0 (socket says 172.21.15.81:59230) 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fd411640 1 -- 172.21.15.81:0/1076352927 learned_addr learned my addr 172.21.15.81:0/1076352927 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fcc10640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5e00053a0 0x7ff5f814d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5f7fff640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5f815cfd0 0x7ff5f815d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fd411640 1 -- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5e00053a0 msgr2=0x7ff5f814d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fd411640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5e00053a0 0x7ff5f814d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fd411640 1 -- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5f815cfd0 msgr2=0x7ff5f815d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fd411640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5f815cfd0 0x7ff5f815d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fd411640 1 -- 172.21.15.81:0/1076352927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f81544b0 con 0x7ff5f8153b60 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fd411640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5f8153b60 0x7ff5f8153f70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff5f002ac50 tx=0x7ff5f0006d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5f57fa640 1 -- 172.21.15.81:0/1076352927 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0003d80 con 0x7ff5f8153b60 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5f57fa640 1 -- 172.21.15.81:0/1076352927 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5f0005a90 con 0x7ff5f8153b60 2024-06-29T14:34:31.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fee74640 1 -- 172.21.15.81:0/1076352927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f814e5d0 con 0x7ff5f8153b60 2024-06-29T14:34:31.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5f57fa640 1 -- 172.21.15.81:0/1076352927 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0005ee0 con 0x7ff5f8153b60 2024-06-29T14:34:31.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.538+0000 7ff5fee74640 1 -- 172.21.15.81:0/1076352927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f814e870 con 0x7ff5f8153b60 2024-06-29T14:34:31.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.539+0000 7ff5fee74640 1 -- 172.21.15.81:0/1076352927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f8159760 con 0x7ff5f8153b60 2024-06-29T14:34:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.540+0000 7ff5f57fa640 1 -- 172.21.15.81:0/1076352927 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7ff5f0006080 con 0x7ff5f8153b60 2024-06-29T14:34:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.540+0000 7ff5f57fa640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff5e4076500 0x7ff5e4078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.541+0000 7ff5f57fa640 1 -- 172.21.15.81:0/1076352927 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6359+0+0 (secure 0 0 0) 0x7ff5f00b80e0 con 0x7ff5f8153b60 2024-06-29T14:34:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.541+0000 7ff5fcc10640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff5e4076500 0x7ff5e4078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.541+0000 7ff5fcc10640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff5e4076500 0x7ff5e4078a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8003700 tx=0x7ff5e800b260 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:31.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.545+0000 7ff5f57fa640 1 -- 172.21.15.81:0/1076352927 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff5f8159760 con 0x7ff5f8153b60 2024-06-29T14:34:31.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.752+0000 7ff5fee74640 1 -- 172.21.15.81:0/1076352927 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7ff5f80aea70 con 0x7ff5e4076500 2024-06-29T14:34:31.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.849+0000 7ff5f57fa640 1 -- 172.21.15.81:0/1076352927 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7ff5f80aea70 con 0x7ff5e4076500 2024-06-29T14:34:31.847 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/dir3 2024-06-29T14:34:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 -- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff5e4076500 msgr2=0x7ff5e4078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff5e4076500 0x7ff5e4078a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8003700 tx=0x7ff5e800b260 comp rx=0 tx=0).stop 2024-06-29T14:34:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 -- 172.21.15.81:0/1076352927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5f8153b60 msgr2=0x7ff5f8153f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5f8153b60 0x7ff5f8153f70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff5f002ac50 tx=0x7ff5f0006d20 comp rx=0 tx=0).stop 2024-06-29T14:34:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 -- 172.21.15.81:0/1076352927 shutdown_connections 2024-06-29T14:34:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7ff5e4076500 0x7ff5e4078a10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5f8153b60 0x7ff5f8153f70 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5f815cfd0 0x7ff5f815d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 --2- 172.21.15.81:0/1076352927 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5e00053a0 0x7ff5f814d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 -- 172.21.15.81:0/1076352927 >> 172.21.15.81:0/1076352927 conn(0x7ff5f8072950 msgr2=0x7ff5f8157b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 -- 172.21.15.81:0/1076352927 shutdown_connections 2024-06-29T14:34:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:31.851+0000 7ff5fee74640 1 -- 172.21.15.81:0/1076352927 wait complete. 2024-06-29T14:34:31.860 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2024-06-29T14:34:31.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.860 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2024-06-29T14:34:31.876 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:31.877 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:31.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.878 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:31.940 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:31.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:31.940 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:32.002 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:32.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.002 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:32.064 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:32.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.064 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:32.126 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:32.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.126 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:32.206 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:32.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.206 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:32.268 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:32.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.269 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:32.334 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:32.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.334 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:32.396 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:32.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.396 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:32.457 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:32.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.458 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:32.520 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:32.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.520 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:32.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:32.583 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2024-06-29T14:34:32.641 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2024-06-29T14:34:32.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.800+0000 7feb15352640 1 -- 172.21.15.81:0/2355532025 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb100ce770 msgr2=0x7feb100b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:32.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.800+0000 7feb15352640 1 --2- 172.21.15.81:0/2355532025 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb100ce770 0x7feb100b35c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feb00009830 tx=0x7feb0002a480 comp rx=0 tx=0).stop 2024-06-29T14:34:32.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.801+0000 7feb15352640 1 -- 172.21.15.81:0/2355532025 shutdown_connections 2024-06-29T14:34:32.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.801+0000 7feb15352640 1 --2- 172.21.15.81:0/2355532025 >> v2:172.21.15.167:3300/0 conn(0x7feb100d2950 0x7feb100b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.801+0000 7feb15352640 1 --2- 172.21.15.81:0/2355532025 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb100ce770 0x7feb100b35c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.801+0000 7feb15352640 1 -- 172.21.15.81:0/2355532025 >> 172.21.15.81:0/2355532025 conn(0x7feb100d3c60 msgr2=0x7feb100d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:32.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.801+0000 7feb15352640 1 -- 172.21.15.81:0/2355532025 shutdown_connections 2024-06-29T14:34:32.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.801+0000 7feb15352640 1 -- 172.21.15.81:0/2355532025 wait complete. 2024-06-29T14:34:32.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.803+0000 7feb15352640 1 Processor -- start 2024-06-29T14:34:32.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.803+0000 7feb15352640 1 -- start start 2024-06-29T14:34:32.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb15352640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb100ce770 0x7feb100a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:32.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb15352640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb100ceb80 0x7feb101e96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:32.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb15352640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb100d2950 0x7feb101e3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:32.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb15352640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feb100d15a0 con 0x7feb100d2950 2024-06-29T14:34:32.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb15352640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7feb100d1420 con 0x7feb100ceb80 2024-06-29T14:34:32.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb15352640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7feb100d4990 con 0x7feb100ce770 2024-06-29T14:34:32.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb100d2950 0x7feb101e3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:32.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb100d2950 0x7feb101e3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59236/0 (socket says 172.21.15.81:59236) 2024-06-29T14:34:32.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0f7fe640 1 -- 172.21.15.81:0/814294701 learned_addr learned my addr 172.21.15.81:0/814294701 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0effd640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb100ce770 0x7feb100a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0e7fc640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb100ceb80 0x7feb101e96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0f7fe640 1 -- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb100ce770 msgr2=0x7feb100a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0f7fe640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb100ce770 0x7feb100a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0f7fe640 1 -- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb100ceb80 msgr2=0x7feb101e96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0f7fe640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb100ceb80 0x7feb101e96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0f7fe640 1 -- 172.21.15.81:0/814294701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb101e9f00 con 0x7feb100d2950 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.804+0000 7feb0f7fe640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb100d2950 0x7feb101e3120 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7feb0800ec20 tx=0x7feb0800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.805+0000 7feaf77fe640 1 -- 172.21.15.81:0/814294701 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08003b70 con 0x7feb100d2950 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.805+0000 7feb15352640 1 -- 172.21.15.81:0/814294701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1009dc50 con 0x7feb100d2950 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.805+0000 7feaf77fe640 1 -- 172.21.15.81:0/814294701 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb08003e00 con 0x7feb100d2950 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.805+0000 7feaf77fe640 1 -- 172.21.15.81:0/814294701 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08004280 con 0x7feb100d2950 2024-06-29T14:34:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.805+0000 7feb15352640 1 -- 172.21.15.81:0/814294701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb1009e210 con 0x7feb100d2950 2024-06-29T14:34:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.806+0000 7feb15352640 1 -- 172.21.15.81:0/814294701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb101e9c70 con 0x7feb100d2950 2024-06-29T14:34:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.806+0000 7feaf77fe640 1 -- 172.21.15.81:0/814294701 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7feb08004420 con 0x7feb100d2950 2024-06-29T14:34:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.807+0000 7feaf77fe640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7feafc0764d0 0x7feafc0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.807+0000 7feaf77fe640 1 -- 172.21.15.81:0/814294701 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6567+0+0 (secure 0 0 0) 0x7feb08098cc0 con 0x7feb100d2950 2024-06-29T14:34:32.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.807+0000 7feb0effd640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7feafc0764d0 0x7feafc0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:32.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.808+0000 7feb0effd640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7feafc0764d0 0x7feafc0789e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb00034000 tx=0x7feb00004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:32.811+0000 7feaf77fe640 1 -- 172.21.15.81:0/814294701 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feb08061860 con 0x7feb100d2950 2024-06-29T14:34:33.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.015+0000 7feb15352640 1 -- 172.21.15.81:0/814294701 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7feb100b16f0 con 0x7feafc0764d0 2024-06-29T14:34:33.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.105+0000 7feaf77fe640 1 -- 172.21.15.81:0/814294701 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7feb100b16f0 con 0x7feafc0764d0 2024-06-29T14:34:33.103 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/dir4 2024-06-29T14:34:33.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 -- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7feafc0764d0 msgr2=0x7feafc0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:33.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7feafc0764d0 0x7feafc0789e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb00034000 tx=0x7feb00004730 comp rx=0 tx=0).stop 2024-06-29T14:34:33.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 -- 172.21.15.81:0/814294701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb100d2950 msgr2=0x7feb101e3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:33.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb100d2950 0x7feb101e3120 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7feb0800ec20 tx=0x7feb0800c6a0 comp rx=0 tx=0).stop 2024-06-29T14:34:33.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 -- 172.21.15.81:0/814294701 shutdown_connections 2024-06-29T14:34:33.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7feafc0764d0 0x7feafc0789e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:33.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb100d2950 0x7feb101e3120 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:33.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb100ceb80 0x7feb101e96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:33.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 --2- 172.21.15.81:0/814294701 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb100ce770 0x7feb100a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:33.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 -- 172.21.15.81:0/814294701 >> 172.21.15.81:0/814294701 conn(0x7feb100d3c60 msgr2=0x7feb100d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:33.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 -- 172.21.15.81:0/814294701 shutdown_connections 2024-06-29T14:34:33.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:33.107+0000 7feb15352640 1 -- 172.21.15.81:0/814294701 wait complete. 2024-06-29T14:34:33.116 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2024-06-29T14:34:33.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.116 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2024-06-29T14:34:33.132 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:33.133 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:33.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.134 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:33.205 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:33.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.205 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:33.269 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:33.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.269 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:33.331 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:33.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.332 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:33.399 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:33.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.399 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:33.467 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:33.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.467 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:33.531 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:33.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.531 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:33.593 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:33.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.594 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:33.657 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:33.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.657 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:33.720 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:33.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.721 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:33.783 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:33.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.783 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:33.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:33.845 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2024-06-29T14:34:33.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2024-06-29T14:34:34.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.066+0000 7efd04cce640 1 -- 172.21.15.81:0/2087585421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcec003a00 msgr2=0x7efcec003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:34.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.066+0000 7efd04cce640 1 --2- 172.21.15.81:0/2087585421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcec003a00 0x7efcec003e70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009d00 tx=0x7efcec005830 comp rx=0 tx=0).stop 2024-06-29T14:34:34.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.066+0000 7efcfed9d640 1 -- 172.21.15.81:0/2087585421 reap_dead start 2024-06-29T14:34:34.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.066+0000 7efd04cce640 1 -- 172.21.15.81:0/2087585421 shutdown_connections 2024-06-29T14:34:34.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.066+0000 7efd04cce640 1 -- 172.21.15.81:0/2087585421 >> 172.21.15.81:0/2087585421 conn(0x7efd0009cf30 msgr2=0x7efd000d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:34.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.066+0000 7efd04cce640 1 -- 172.21.15.81:0/2087585421 shutdown_connections 2024-06-29T14:34:34.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.066+0000 7efd04cce640 1 -- 172.21.15.81:0/2087585421 wait complete. 2024-06-29T14:34:34.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.068+0000 7efd04cce640 1 Processor -- start 2024-06-29T14:34:34.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.068+0000 7efd04cce640 1 -- start start 2024-06-29T14:34:34.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efd04cce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efcec003a00 0x7efd000b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:34.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efd04cce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd0009f1c0 0x7efd0015d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:34.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efd04cce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd000b0c80 0x7efd0009e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:34.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efd04cce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd000d09c0 con 0x7efd000b0c80 2024-06-29T14:34:34.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efd04cce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efd000d2d80 con 0x7efd0009f1c0 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efd04cce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efd000d0b40 con 0x7efcec003a00 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcfed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd000b0c80 0x7efd0009e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcfed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd000b0c80 0x7efd0009e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59262/0 (socket says 172.21.15.81:59262) 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcfed9d640 1 -- 172.21.15.81:0/3875708912 learned_addr learned my addr 172.21.15.81:0/3875708912 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcfe59c640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efcec003a00 0x7efd000b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcf5d9b640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd0009f1c0 0x7efd0015d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcfed9d640 1 -- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efcec003a00 msgr2=0x7efd000b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcfed9d640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efcec003a00 0x7efd000b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcfed9d640 1 -- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd0009f1c0 msgr2=0x7efd0015d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcfed9d640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd0009f1c0 0x7efd0015d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcfed9d640 1 -- 172.21.15.81:0/3875708912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0009f9a0 con 0x7efd000b0c80 2024-06-29T14:34:34.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.069+0000 7efcf5d9b640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd0009f1c0 0x7efd0015d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:34.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.070+0000 7efcfed9d640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd000b0c80 0x7efd0009e700 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efd000d4460 tx=0x7efcf8006790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:34.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.070+0000 7efcf77fe640 1 -- 172.21.15.81:0/3875708912 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf800fc10 con 0x7efd000b0c80 2024-06-29T14:34:34.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.070+0000 7efcf77fe640 1 -- 172.21.15.81:0/3875708912 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcf8018070 con 0x7efd000b0c80 2024-06-29T14:34:34.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.070+0000 7efd04cce640 1 -- 172.21.15.81:0/3875708912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0009d940 con 0x7efd000b0c80 2024-06-29T14:34:34.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.070+0000 7efcf77fe640 1 -- 172.21.15.81:0/3875708912 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8011430 con 0x7efd000b0c80 2024-06-29T14:34:34.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.070+0000 7efd04cce640 1 -- 172.21.15.81:0/3875708912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0009df00 con 0x7efd000b0c80 2024-06-29T14:34:34.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.071+0000 7efd04cce640 1 -- 172.21.15.81:0/3875708912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd00161600 con 0x7efd000b0c80 2024-06-29T14:34:34.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.072+0000 7efcf77fe640 1 -- 172.21.15.81:0/3875708912 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7efcf80115d0 con 0x7efd000b0c80 2024-06-29T14:34:34.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.072+0000 7efcf77fe640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efce00764c0 0x7efce00789d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:34.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.072+0000 7efcf77fe640 1 -- 172.21.15.81:0/3875708912 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6675+0+0 (secure 0 0 0) 0x7efcf8098940 con 0x7efd000b0c80 2024-06-29T14:34:34.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.072+0000 7efcfe59c640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efce00764c0 0x7efce00789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:34.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.073+0000 7efcfe59c640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efce00764c0 0x7efce00789d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efcf0035000 tx=0x7efcf00046c0 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:34.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.076+0000 7efcf77fe640 1 -- 172.21.15.81:0/3875708912 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efcf8061b40 con 0x7efd000b0c80 2024-06-29T14:34:34.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.303+0000 7efd04cce640 1 -- 172.21.15.81:0/3875708912 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7efd0015dc40 con 0x7efce00764c0 2024-06-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.420+0000 7efcf77fe640 1 -- 172.21.15.81:0/3875708912 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7efcf002bb80 con 0x7efce00764c0 2024-06-29T14:34:34.418 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/dir5 2024-06-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 -- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efce00764c0 msgr2=0x7efce00789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efce00764c0 0x7efce00789d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efcf0035000 tx=0x7efcf00046c0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 -- 172.21.15.81:0/3875708912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd000b0c80 msgr2=0x7efd0009e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:34.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd000b0c80 0x7efd0009e700 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efd000d4460 tx=0x7efcf8006790 comp rx=0 tx=0).stop 2024-06-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 -- 172.21.15.81:0/3875708912 shutdown_connections 2024-06-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7efce00764c0 0x7efce00789d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd000b0c80 0x7efd0009e700 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd0009f1c0 0x7efd0015d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 --2- 172.21.15.81:0/3875708912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efcec003a00 0x7efd000b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 -- 172.21.15.81:0/3875708912 >> 172.21.15.81:0/3875708912 conn(0x7efd0009cf30 msgr2=0x7efd0006e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 -- 172.21.15.81:0/3875708912 shutdown_connections 2024-06-29T14:34:34.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:34.422+0000 7efd04cce640 1 -- 172.21.15.81:0/3875708912 wait complete. 2024-06-29T14:34:34.431 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2024-06-29T14:34:34.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.431 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2024-06-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:34.447 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:34.448 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:34.448 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:34.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.448 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:34.510 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:34.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.511 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:34.572 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:34.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.572 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:34.633 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:34.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.634 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:34.694 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:34.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.695 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:34.759 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:34.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.759 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:34.820 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:34.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.821 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:34.881 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:34.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.881 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:34.942 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:34.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:34.943 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:35.004 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:35.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.004 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:35.065 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:35.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.065 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:35.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.128 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2024-06-29T14:34:35.194 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2024-06-29T14:34:35.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.395+0000 7f62da732640 1 -- 172.21.15.81:0/1583660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62d4073480 msgr2=0x7f62d415b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:35.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.395+0000 7f62da732640 1 --2- 172.21.15.81:0/1583660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62d4073480 0x7f62d415b6e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f62d4072f10 tx=0x7f62c802a4d0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.395+0000 7f62da732640 1 -- 172.21.15.81:0/1583660 shutdown_connections 2024-06-29T14:34:35.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.395+0000 7f62da732640 1 --2- 172.21.15.81:0/1583660 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62d415bc20 0x7f62d415e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.395+0000 7f62da732640 1 --2- 172.21.15.81:0/1583660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62d4073480 0x7f62d415b6e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.395+0000 7f62da732640 1 --2- 172.21.15.81:0/1583660 >> v2:172.21.15.167:3300/0 conn(0x7f62d40cd150 0x7f62d4072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.395+0000 7f62da732640 1 -- 172.21.15.81:0/1583660 >> 172.21.15.81:0/1583660 conn(0x7f62d40b2920 msgr2=0x7f62d40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:35.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.395+0000 7f62da732640 1 -- 172.21.15.81:0/1583660 shutdown_connections 2024-06-29T14:34:35.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.395+0000 7f62da732640 1 -- 172.21.15.81:0/1583660 wait complete. 2024-06-29T14:34:35.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.397+0000 7f62da732640 1 Processor -- start 2024-06-29T14:34:35.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.397+0000 7f62da732640 1 -- start start 2024-06-29T14:34:35.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62da732640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62d4073480 0x7f62d40af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:35.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62da732640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62d40cd150 0x7f62d406c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:35.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62da732640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62d415bc20 0x7f62d406c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:35.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62da732640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62d415f010 con 0x7f62d40cd150 2024-06-29T14:34:35.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62da732640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f62d415ee90 con 0x7f62d4073480 2024-06-29T14:34:35.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62da732640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f62d415f190 con 0x7f62d415bc20 2024-06-29T14:34:35.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62d40cd150 0x7f62d406c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:35.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62d40cd150 0x7f62d406c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59274/0 (socket says 172.21.15.81:59274) 2024-06-29T14:34:35.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d37fe640 1 -- 172.21.15.81:0/3941765911 learned_addr learned my addr 172.21.15.81:0/3941765911 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:35.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d8ccf640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62d415bc20 0x7f62d406c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:35.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d3fff640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62d4073480 0x7f62d40af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:35.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d37fe640 1 -- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62d415bc20 msgr2=0x7f62d406c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:35.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d37fe640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62d415bc20 0x7f62d406c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d37fe640 1 -- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62d4073480 msgr2=0x7f62d40af590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:35.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d37fe640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62d4073480 0x7f62d40af590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d37fe640 1 -- 172.21.15.81:0/3941765911 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d4156100 con 0x7f62d40cd150 2024-06-29T14:34:35.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d37fe640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62d40cd150 0x7f62d406c2f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f62c80099c0 tx=0x7f62c80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:35.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d0ff9640 1 -- 172.21.15.81:0/3941765911 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c803e030 con 0x7f62d40cd150 2024-06-29T14:34:35.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d0ff9640 1 -- 172.21.15.81:0/3941765911 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62c8003ca0 con 0x7f62d40cd150 2024-06-29T14:34:35.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62da732640 1 -- 172.21.15.81:0/3941765911 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d40adb20 con 0x7f62d40cd150 2024-06-29T14:34:35.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.399+0000 7f62d0ff9640 1 -- 172.21.15.81:0/3941765911 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8035040 con 0x7f62d40cd150 2024-06-29T14:34:35.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.400+0000 7f62da732640 1 -- 172.21.15.81:0/3941765911 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d40ae0e0 con 0x7f62d40cd150 2024-06-29T14:34:35.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.401+0000 7f62da732640 1 -- 172.21.15.81:0/3941765911 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d40b10a0 con 0x7f62d40cd150 2024-06-29T14:34:35.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.401+0000 7f62d0ff9640 1 -- 172.21.15.81:0/3941765911 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f62c8041030 con 0x7f62d40cd150 2024-06-29T14:34:35.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.402+0000 7f62d0ff9640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f62c00764d0 0x7f62c00789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:35.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.402+0000 7f62d0ff9640 1 -- 172.21.15.81:0/3941765911 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6839+0+0 (secure 0 0 0) 0x7f62c80b84d0 con 0x7f62d40cd150 2024-06-29T14:34:35.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.402+0000 7f62d3fff640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f62c00764d0 0x7f62c00789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:35.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.402+0000 7f62d3fff640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f62c00764d0 0x7f62c00789e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009ab0 tx=0x7f62c4005f90 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:35.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.406+0000 7f62d0ff9640 1 -- 172.21.15.81:0/3941765911 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f62d40b10a0 con 0x7f62d40cd150 2024-06-29T14:34:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.607+0000 7f62da732640 1 -- 172.21.15.81:0/3941765911 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f62d40ad8d0 con 0x7f62c00764d0 2024-06-29T14:34:35.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.696+0000 7f62d0ff9640 1 -- 172.21.15.81:0/3941765911 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f62d40ad8d0 con 0x7f62c00764d0 2024-06-29T14:34:35.694 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/dir6 2024-06-29T14:34:35.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 -- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f62c00764d0 msgr2=0x7f62c00789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:35.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f62c00764d0 0x7f62c00789e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009ab0 tx=0x7f62c4005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:35.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 -- 172.21.15.81:0/3941765911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62d40cd150 msgr2=0x7f62d406c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:35.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62d40cd150 0x7f62d406c2f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f62c80099c0 tx=0x7f62c80038a0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 -- 172.21.15.81:0/3941765911 shutdown_connections 2024-06-29T14:34:35.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f62c00764d0 0x7f62c00789e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62d415bc20 0x7f62d406c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62d40cd150 0x7f62d406c2f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 --2- 172.21.15.81:0/3941765911 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62d4073480 0x7f62d40af590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:35.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 -- 172.21.15.81:0/3941765911 >> 172.21.15.81:0/3941765911 conn(0x7f62d40b2920 msgr2=0x7f62d4154a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:35.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 -- 172.21.15.81:0/3941765911 shutdown_connections 2024-06-29T14:34:35.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:35.698+0000 7f62da732640 1 -- 172.21.15.81:0/3941765911 wait complete. 2024-06-29T14:34:35.706 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2024-06-29T14:34:35.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.707 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2024-06-29T14:34:35.723 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:35.723 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:35.723 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:35.723 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:35.723 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:35.723 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:35.723 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:35.723 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:35.724 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:35.724 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:35.724 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:35.724 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:35.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.724 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:35.785 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:35.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.786 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:35.847 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:35.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.847 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:35.908 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:35.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.909 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:35.971 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:35.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:35.971 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:36.033 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:36.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.034 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:36.094 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:36.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.095 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:36.167 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:36.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.167 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:36.228 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:36.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.228 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:36.289 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:36.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.289 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:36.351 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:36.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.351 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:36.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.413 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2024-06-29T14:34:36.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2024-06-29T14:34:36.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.631+0000 7f9bcab7f640 1 -- 172.21.15.81:0/2046989982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac004800 msgr2=0x7f9bac004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:36.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.631+0000 7f9bcab7f640 1 --2- 172.21.15.81:0/2046989982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac004800 0x7f9bac004c10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4145b70 tx=0x7f9bac005830 comp rx=0 tx=0).stop 2024-06-29T14:34:36.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.631+0000 7f9bc911c640 1 -- 172.21.15.81:0/2046989982 reap_dead start 2024-06-29T14:34:36.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.631+0000 7f9bcab7f640 1 -- 172.21.15.81:0/2046989982 shutdown_connections 2024-06-29T14:34:36.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.631+0000 7f9bcab7f640 1 -- 172.21.15.81:0/2046989982 >> 172.21.15.81:0/2046989982 conn(0x7f9bc406ccc0 msgr2=0x7f9bc40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:36.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.631+0000 7f9bcab7f640 1 -- 172.21.15.81:0/2046989982 shutdown_connections 2024-06-29T14:34:36.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.631+0000 7f9bcab7f640 1 -- 172.21.15.81:0/2046989982 wait complete. 2024-06-29T14:34:36.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.633+0000 7f9bcab7f640 1 Processor -- start 2024-06-29T14:34:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.633+0000 7f9bcab7f640 1 -- start start 2024-06-29T14:34:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bcab7f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9bac004800 0x7f9bc413bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bcab7f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9bc40af200 0x7f9bc4142f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bcab7f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc40ad110 0x7f9bc41434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bcab7f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc4160520 con 0x7f9bc40ad110 2024-06-29T14:34:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bcab7f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc41603a0 con 0x7f9bac004800 2024-06-29T14:34:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bcab7f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9bc41606a0 con 0x7f9bc40af200 2024-06-29T14:34:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bc911c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc40ad110 0x7f9bc41434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bc911c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc40ad110 0x7f9bc41434c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59298/0 (socket says 172.21.15.81:59298) 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bc911c640 1 -- 172.21.15.81:0/3366087680 learned_addr learned my addr 172.21.15.81:0/3366087680 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bbbfff640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9bc40af200 0x7f9bc4142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bc891b640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9bac004800 0x7f9bc413bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bc911c640 1 -- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9bc40af200 msgr2=0x7f9bc4142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bc911c640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9bc40af200 0x7f9bc4142f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bc911c640 1 -- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9bac004800 msgr2=0x7f9bc413bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bc911c640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9bac004800 0x7f9bc413bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bc911c640 1 -- 172.21.15.81:0/3366087680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc40a9560 con 0x7f9bc40ad110 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bc911c640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc40ad110 0x7f9bc41434c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9bac004780 tx=0x7f9bbc006420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bb97fa640 1 -- 172.21.15.81:0/3366087680 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc011870 con 0x7f9bc40ad110 2024-06-29T14:34:36.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bb97fa640 1 -- 172.21.15.81:0/3366087680 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9bbc017070 con 0x7f9bc40ad110 2024-06-29T14:34:36.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bcab7f640 1 -- 172.21.15.81:0/3366087680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc40ab420 con 0x7f9bc40ad110 2024-06-29T14:34:36.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bb97fa640 1 -- 172.21.15.81:0/3366087680 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc011de0 con 0x7f9bc40ad110 2024-06-29T14:34:36.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.635+0000 7f9bcab7f640 1 -- 172.21.15.81:0/3366087680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bc413cd90 con 0x7f9bc40ad110 2024-06-29T14:34:36.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.636+0000 7f9bcab7f640 1 -- 172.21.15.81:0/3366087680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bc4144a20 con 0x7f9bc40ad110 2024-06-29T14:34:36.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.637+0000 7f9bb97fa640 1 -- 172.21.15.81:0/3366087680 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f9bbc01c460 con 0x7f9bc40ad110 2024-06-29T14:34:36.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.638+0000 7f9bb97fa640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9bb0076500 0x7f9bb0078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:36.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.638+0000 7f9bb97fa640 1 -- 172.21.15.81:0/3366087680 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6567+0+0 (secure 0 0 0) 0x7f9bbc098790 con 0x7f9bc40ad110 2024-06-29T14:34:36.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.638+0000 7f9bc891b640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9bb0076500 0x7f9bb0078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:36.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.638+0000 7f9bc891b640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9bb0076500 0x7f9bb0078a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009c50 tx=0x7f9bc0005f90 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.642+0000 7f9bb97fa640 1 -- 172.21.15.81:0/3366087680 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9bc415c350 con 0x7f9bc40ad110 2024-06-29T14:34:36.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.840+0000 7f9bcab7f640 1 -- 172.21.15.81:0/3366087680 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f9bc40ad580 con 0x7f9bb0076500 2024-06-29T14:34:36.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.937+0000 7f9bb97fa640 1 -- 172.21.15.81:0/3366087680 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f9bc40ad580 con 0x7f9bb0076500 2024-06-29T14:34:36.935 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/dir7 2024-06-29T14:34:36.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 -- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9bb0076500 msgr2=0x7f9bb0078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9bb0076500 0x7f9bb0078a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009c50 tx=0x7f9bc0005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 -- 172.21.15.81:0/3366087680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc40ad110 msgr2=0x7f9bc41434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc40ad110 0x7f9bc41434c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9bac004780 tx=0x7f9bbc006420 comp rx=0 tx=0).stop 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 -- 172.21.15.81:0/3366087680 shutdown_connections 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9bb0076500 0x7f9bb0078a10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc40ad110 0x7f9bc41434c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9bc40af200 0x7f9bc4142f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 --2- 172.21.15.81:0/3366087680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9bac004800 0x7f9bc413bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 -- 172.21.15.81:0/3366087680 >> 172.21.15.81:0/3366087680 conn(0x7f9bc406ccc0 msgr2=0x7f9bc40afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 -- 172.21.15.81:0/3366087680 shutdown_connections 2024-06-29T14:34:36.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:36.939+0000 7f9bcab7f640 1 -- 172.21.15.81:0/3366087680 wait complete. 2024-06-29T14:34:36.948 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2024-06-29T14:34:36.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.948 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2024-06-29T14:34:36.966 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:36.966 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:36.966 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:36.966 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:36.966 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:36.966 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:36.966 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:36.966 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:36.967 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:36.967 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:36.967 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:36.967 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:36.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:36.967 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:37.029 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:37.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.029 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:37.090 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:37.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.091 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:37.155 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:37.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.156 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:37.225 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:37.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.226 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:37.288 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:37.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.288 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:37.349 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:37.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.350 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:37.411 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:37.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.412 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:37.472 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:37.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.472 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:37.534 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:37.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.534 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:37.595 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:37.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.595 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:37.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:37.658 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2024-06-29T14:34:37.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2024-06-29T14:34:37.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.877+0000 7f9589119640 1 -- 172.21.15.81:0/462159401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95840b57b0 msgr2=0x7f958415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:37.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.877+0000 7f9589119640 1 --2- 172.21.15.81:0/462159401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95840b57b0 0x7f958415aed0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9574005fd0 tx=0x7f957402a530 comp rx=0 tx=0).stop 2024-06-29T14:34:37.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.877+0000 7f9589119640 1 -- 172.21.15.81:0/462159401 shutdown_connections 2024-06-29T14:34:37.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.877+0000 7f9589119640 1 --2- 172.21.15.81:0/462159401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f958406ef90 0x7f958406f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.877+0000 7f9589119640 1 --2- 172.21.15.81:0/462159401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95840b57b0 0x7f958415aed0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.877+0000 7f9589119640 1 -- 172.21.15.81:0/462159401 >> 172.21.15.81:0/462159401 conn(0x7f95840d0930 msgr2=0x7f95840d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:37.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.877+0000 7f9589119640 1 -- 172.21.15.81:0/462159401 shutdown_connections 2024-06-29T14:34:37.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.877+0000 7f9589119640 1 -- 172.21.15.81:0/462159401 wait complete. 2024-06-29T14:34:37.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.879+0000 7f9589119640 1 Processor -- start 2024-06-29T14:34:37.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.879+0000 7f9589119640 1 -- start start 2024-06-29T14:34:37.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f9589119640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f958406ef90 0x7f95840a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f9589119640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95840b57b0 0x7f95841f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f9589119640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95840ce1d0 0x7f95840b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f9589119640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f95840cecf0 con 0x7f95840ce1d0 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f9589119640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f95840b4b80 con 0x7f958406ef90 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f9589119640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f95840cee70 con 0x7f95840b57b0 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f958359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95840ce1d0 0x7f95840b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f958359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95840ce1d0 0x7f95840b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59314/0 (socket says 172.21.15.81:59314) 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f958359e640 1 -- 172.21.15.81:0/1785101365 learned_addr learned my addr 172.21.15.81:0/1785101365 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f9582d9d640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f958406ef90 0x7f95840a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.880+0000 7f958259c640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95840b57b0 0x7f95841f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f958359e640 1 -- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95840b57b0 msgr2=0x7f95841f7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f958359e640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95840b57b0 0x7f95841f7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f958359e640 1 -- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f958406ef90 msgr2=0x7f95840a9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:37.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f958359e640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f958406ef90 0x7f95840a9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:37.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f958359e640 1 -- 172.21.15.81:0/1785101365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95841f8260 con 0x7f95840ce1d0 2024-06-29T14:34:37.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f958359e640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95840ce1d0 0x7f95840b07f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f957c00b9a0 tx=0x7f957c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:37.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f956b7fe640 1 -- 172.21.15.81:0/1785101365 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c019070 con 0x7f95840ce1d0 2024-06-29T14:34:37.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f956b7fe640 1 -- 172.21.15.81:0/1785101365 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f957c00cc90 con 0x7f95840ce1d0 2024-06-29T14:34:37.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f9589119640 1 -- 172.21.15.81:0/1785101365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95841e6e30 con 0x7f95840ce1d0 2024-06-29T14:34:37.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f956b7fe640 1 -- 172.21.15.81:0/1785101365 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c0146a0 con 0x7f95840ce1d0 2024-06-29T14:34:37.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.881+0000 7f9589119640 1 -- 172.21.15.81:0/1785101365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95841e73f0 con 0x7f95840ce1d0 2024-06-29T14:34:37.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.882+0000 7f9589119640 1 -- 172.21.15.81:0/1785101365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f958406e2a0 con 0x7f95840ce1d0 2024-06-29T14:34:37.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.883+0000 7f956b7fe640 1 -- 172.21.15.81:0/1785101365 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f957c01c070 con 0x7f95840ce1d0 2024-06-29T14:34:37.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.883+0000 7f956b7fe640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9570076500 0x7f9570078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:37.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.883+0000 7f956b7fe640 1 -- 172.21.15.81:0/1785101365 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6615+0+0 (secure 0 0 0) 0x7f957c0979e0 con 0x7f95840ce1d0 2024-06-29T14:34:37.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.883+0000 7f9582d9d640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9570076500 0x7f9570078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:37.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.884+0000 7f9582d9d640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9570076500 0x7f9570078a10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9574033000 tx=0x7f9574004730 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:37.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:37.888+0000 7f956b7fe640 1 -- 172.21.15.81:0/1785101365 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f958406e2a0 con 0x7f95840ce1d0 2024-06-29T14:34:38.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.089+0000 7f9589119640 1 -- 172.21.15.81:0/1785101365 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f958406f3a0 con 0x7f9570076500 2024-06-29T14:34:38.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.238+0000 7f956b7fe640 1 -- 172.21.15.81:0/1785101365 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f958406f3a0 con 0x7f9570076500 2024-06-29T14:34:38.237 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/dir8 2024-06-29T14:34:38.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 -- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9570076500 msgr2=0x7f9570078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:38.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9570076500 0x7f9570078a10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9574033000 tx=0x7f9574004730 comp rx=0 tx=0).stop 2024-06-29T14:34:38.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 -- 172.21.15.81:0/1785101365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95840ce1d0 msgr2=0x7f95840b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:38.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95840ce1d0 0x7f95840b07f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f957c00b9a0 tx=0x7f957c00be60 comp rx=0 tx=0).stop 2024-06-29T14:34:38.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 -- 172.21.15.81:0/1785101365 shutdown_connections 2024-06-29T14:34:38.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f9570076500 0x7f9570078a10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:38.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95840ce1d0 0x7f95840b07f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:38.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95840b57b0 0x7f95841f7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:38.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 --2- 172.21.15.81:0/1785101365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f958406ef90 0x7f95840a9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:38.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 -- 172.21.15.81:0/1785101365 >> 172.21.15.81:0/1785101365 conn(0x7f95840d0930 msgr2=0x7f95841586d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:38.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 -- 172.21.15.81:0/1785101365 shutdown_connections 2024-06-29T14:34:38.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:38.240+0000 7f9589119640 1 -- 172.21.15.81:0/1785101365 wait complete. 2024-06-29T14:34:38.250 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2024-06-29T14:34:38.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.250 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2024-06-29T14:34:38.267 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:38.267 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:38.267 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:38.267 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:38.267 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:38.267 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:38.267 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:38.267 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:38.267 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:38.268 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:38.268 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:38.268 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:38.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.268 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:38.331 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:38.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.331 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:38.392 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:38.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.393 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:38.454 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:38.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.454 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:38.516 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:38.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.516 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:38.578 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:38.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.578 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:38.640 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:38.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.640 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:38.702 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:38.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.702 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:38.764 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:38.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.765 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:38.826 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:38.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.826 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:38.888 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:38.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.888 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:38.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:38.951 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2024-06-29T14:34:39.009 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2024-06-29T14:34:39.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.179+0000 7f07a484e640 1 -- 172.21.15.81:0/1272684160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07a00d0940 msgr2=0x7f07a00b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:39.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.179+0000 7f07a484e640 1 --2- 172.21.15.81:0/1272684160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07a00d0940 0x7f07a00b4f40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f07a0073550 tx=0x7f079802aa30 comp rx=0 tx=0).stop 2024-06-29T14:34:39.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.179+0000 7f07a484e640 1 -- 172.21.15.81:0/1272684160 shutdown_connections 2024-06-29T14:34:39.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.179+0000 7f07a484e640 1 --2- 172.21.15.81:0/1272684160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07a00d0940 0x7f07a00b4f40 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.179+0000 7f07a484e640 1 --2- 172.21.15.81:0/1272684160 >> v2:172.21.15.167:3300/0 conn(0x7f07a00b3200 0x7f07a00cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.179+0000 7f07a484e640 1 -- 172.21.15.81:0/1272684160 >> 172.21.15.81:0/1272684160 conn(0x7f07a00b3680 msgr2=0x7f07a00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:39.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.179+0000 7f07a484e640 1 -- 172.21.15.81:0/1272684160 shutdown_connections 2024-06-29T14:34:39.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.179+0000 7f07a484e640 1 -- 172.21.15.81:0/1272684160 wait complete. 2024-06-29T14:34:39.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.181+0000 7f07a484e640 1 Processor -- start 2024-06-29T14:34:39.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.181+0000 7f07a484e640 1 -- start start 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f07a484e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07a00b3200 0x7f07a01e5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f07a484e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07a00cccf0 0x7f07a01e3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f07a484e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07a006e650 0x7f07a00c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f07a484e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f07a00b5bb0 con 0x7f07a00cccf0 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f07a484e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f07a00b5a30 con 0x7f07a00b3200 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f07a484e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f07a00b5d30 con 0x7f07a006e650 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f0797fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07a00cccf0 0x7f07a01e3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f0797fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07a00cccf0 0x7f07a01e3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59334/0 (socket says 172.21.15.81:59334) 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f0797fff640 1 -- 172.21.15.81:0/3226592586 learned_addr learned my addr 172.21.15.81:0/3226592586 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f079e59c640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07a00b3200 0x7f07a01e5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f079ed9d640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07a006e650 0x7f07a00c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f0797fff640 1 -- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07a006e650 msgr2=0x7f07a00c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:39.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f0797fff640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07a006e650 0x7f07a00c7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f0797fff640 1 -- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07a00b3200 msgr2=0x7f07a01e5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:39.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f0797fff640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07a00b3200 0x7f07a01e5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.183+0000 7f0797fff640 1 -- 172.21.15.81:0/3226592586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a006d580 con 0x7f07a00cccf0 2024-06-29T14:34:39.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.184+0000 7f0797fff640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07a00cccf0 0x7f07a01e3030 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f078800c370 tx=0x7f078800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:39.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.184+0000 7f0796ffd640 1 -- 172.21.15.81:0/3226592586 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0788017070 con 0x7f07a00cccf0 2024-06-29T14:34:39.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.184+0000 7f0796ffd640 1 -- 172.21.15.81:0/3226592586 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07880053c0 con 0x7f07a00cccf0 2024-06-29T14:34:39.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.184+0000 7f07a484e640 1 -- 172.21.15.81:0/3226592586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a006e320 con 0x7f07a00cccf0 2024-06-29T14:34:39.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.184+0000 7f0796ffd640 1 -- 172.21.15.81:0/3226592586 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0788003850 con 0x7f07a00cccf0 2024-06-29T14:34:39.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.184+0000 7f07a484e640 1 -- 172.21.15.81:0/3226592586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a006db20 con 0x7f07a00cccf0 2024-06-29T14:34:39.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.185+0000 7f07a484e640 1 -- 172.21.15.81:0/3226592586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07a00cc7b0 con 0x7f07a00cccf0 2024-06-29T14:34:39.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.186+0000 7f0796ffd640 1 -- 172.21.15.81:0/3226592586 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f078801a070 con 0x7f07a00cccf0 2024-06-29T14:34:39.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.186+0000 7f0796ffd640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0780076500 0x7f0780078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:39.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.186+0000 7f0796ffd640 1 -- 172.21.15.81:0/3226592586 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6487+0+0 (secure 0 0 0) 0x7f07880971d0 con 0x7f07a00cccf0 2024-06-29T14:34:39.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.186+0000 7f079e59c640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0780076500 0x7f0780078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:39.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.187+0000 7f079e59c640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0780076500 0x7f0780078a10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0790004630 tx=0x7f079000b2e0 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:39.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.191+0000 7f0796ffd640 1 -- 172.21.15.81:0/3226592586 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f078805fef0 con 0x7f07a00cccf0 2024-06-29T14:34:39.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.426+0000 7f07a484e640 1 -- 172.21.15.81:0/3226592586 --> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f07a01e6670 con 0x7f0780076500 2024-06-29T14:34:39.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.513+0000 7f0796ffd640 1 -- 172.21.15.81:0/3226592586 <== mgr.5122 v2:172.21.15.167:6832/189854401 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f0790004280 con 0x7f0780076500 2024-06-29T14:34:39.512 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/dir9 2024-06-29T14:34:39.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 -- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0780076500 msgr2=0x7f0780078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0780076500 0x7f0780078a10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0790004630 tx=0x7f079000b2e0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 -- 172.21.15.81:0/3226592586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07a00cccf0 msgr2=0x7f07a01e3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07a00cccf0 0x7f07a01e3030 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f078800c370 tx=0x7f078800c830 comp rx=0 tx=0).stop 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 -- 172.21.15.81:0/3226592586 shutdown_connections 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f0780076500 0x7f0780078a10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07a006e650 0x7f07a00c7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07a00cccf0 0x7f07a01e3030 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 --2- 172.21.15.81:0/3226592586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f07a00b3200 0x7f07a01e5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 -- 172.21.15.81:0/3226592586 >> 172.21.15.81:0/3226592586 conn(0x7f07a00b3680 msgr2=0x7f07a00caf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 -- 172.21.15.81:0/3226592586 shutdown_connections 2024-06-29T14:34:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:39.515+0000 7f07a484e640 1 -- 172.21.15.81:0/3226592586 wait complete. 2024-06-29T14:34:39.524 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2024-06-29T14:34:39.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.525 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2024-06-29T14:34:39.541 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:39.542 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:39.542 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:39.542 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:39.542 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:39.542 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:39.542 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:39.542 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:39.542 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:39.542 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:39.542 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:39.543 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_24_00_UTC 2024-06-29T14:34:39.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.543 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_24_00_UTC) 2024-06-29T14:34:39.606 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_25_00_UTC 2024-06-29T14:34:39.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.606 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_25_00_UTC) 2024-06-29T14:34:39.668 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_26_00_UTC 2024-06-29T14:34:39.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.668 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_26_00_UTC) 2024-06-29T14:34:39.729 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_27_00_UTC 2024-06-29T14:34:39.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.730 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_27_00_UTC) 2024-06-29T14:34:39.790 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_28_00_UTC 2024-06-29T14:34:39.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.791 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_28_00_UTC) 2024-06-29T14:34:39.852 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_29_00_UTC 2024-06-29T14:34:39.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.852 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_29_00_UTC) 2024-06-29T14:34:39.914 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_30_00_UTC 2024-06-29T14:34:39.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.914 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_30_00_UTC) 2024-06-29T14:34:39.975 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_31_00_UTC 2024-06-29T14:34:39.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:39.975 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_31_00_UTC) 2024-06-29T14:34:39.995 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:34:40.019 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:34:40.037 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_32_00_UTC 2024-06-29T14:34:40.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.037 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_32_00_UTC) 2024-06-29T14:34:40.099 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_33_00_UTC 2024-06-29T14:34:40.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.099 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_33_00_UTC) 2024-06-29T14:34:40.172 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_34_00_UTC 2024-06-29T14:34:40.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.172 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2024-06-29-14_34_00_UTC) 2024-06-29T14:34:40.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:40.235 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2024-06-29T14:34:40.318 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:34:40.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.530+0000 7f6dc59e0640 1 -- 172.21.15.81:0/2557459392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dc015b400 msgr2=0x7f6dc015b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:40.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.530+0000 7f6dc59e0640 1 --2- 172.21.15.81:0/2557459392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dc015b400 0x7f6dc015b810 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00d0a00 tx=0x7f6db802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.530+0000 7f6dc59e0640 1 -- 172.21.15.81:0/2557459392 shutdown_connections 2024-06-29T14:34:40.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.530+0000 7f6dc59e0640 1 --2- 172.21.15.81:0/2557459392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dc015b400 0x7f6dc015b810 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.530+0000 7f6dc59e0640 1 --2- 172.21.15.81:0/2557459392 >> v2:172.21.15.167:3300/0 conn(0x7f6dc0073690 0x7f6dc0144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.530+0000 7f6dc59e0640 1 -- 172.21.15.81:0/2557459392 >> 172.21.15.81:0/2557459392 conn(0x7f6dc00b2920 msgr2=0x7f6dc00b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:40.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.530+0000 7f6dc59e0640 1 -- 172.21.15.81:0/2557459392 shutdown_connections 2024-06-29T14:34:40.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.530+0000 7f6dc59e0640 1 -- 172.21.15.81:0/2557459392 wait complete. 2024-06-29T14:34:40.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.531+0000 7f6dc59e0640 1 Processor -- start 2024-06-29T14:34:40.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.532+0000 7f6dc59e0640 1 -- start start 2024-06-29T14:34:40.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dc59e0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6dc00b4800 0x7f6dc013b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dc59e0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6dc015b400 0x7f6dc013b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dc59e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dc0159200 0x7f6dc013bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dc59e0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc006f200 con 0x7f6dc0159200 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dc59e0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc006f080 con 0x7f6dc00b4800 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dc59e0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc006f380 con 0x7f6dc015b400 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dbf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dc0159200 0x7f6dc013bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dbf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dc0159200 0x7f6dc013bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37608/0 (socket says 172.21.15.81:37608) 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dbf7fe640 1 -- 172.21.15.81:0/3491932580 learned_addr learned my addr 172.21.15.81:0/3491932580 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dbeffd640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6dc00b4800 0x7f6dc013b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.533+0000 7f6dbe7fc640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6dc015b400 0x7f6dc013b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6dbf7fe640 1 -- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6dc015b400 msgr2=0x7f6dc013b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:40.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6dbf7fe640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6dc015b400 0x7f6dc013b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6dbf7fe640 1 -- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6dc00b4800 msgr2=0x7f6dc013b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:40.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6dbf7fe640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6dc00b4800 0x7f6dc013b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:40.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6dbf7fe640 1 -- 172.21.15.81:0/3491932580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc01404c0 con 0x7f6dc0159200 2024-06-29T14:34:40.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6dbeffd640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6dc00b4800 0x7f6dc013b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:40.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6dbf7fe640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dc0159200 0x7f6dc013bdc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6db802aea0 tx=0x7f6db8033f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:40.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6da7fff640 1 -- 172.21.15.81:0/3491932580 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db8031990 con 0x7f6dc0159200 2024-06-29T14:34:40.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6da7fff640 1 -- 172.21.15.81:0/3491932580 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6db8031c20 con 0x7f6dc0159200 2024-06-29T14:34:40.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6da7fff640 1 -- 172.21.15.81:0/3491932580 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db80330d0 con 0x7f6dc0159200 2024-06-29T14:34:40.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6dc59e0640 1 -- 172.21.15.81:0/3491932580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc01407a0 con 0x7f6dc0159200 2024-06-29T14:34:40.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.534+0000 7f6dc59e0640 1 -- 172.21.15.81:0/3491932580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc0140d60 con 0x7f6dc0159200 2024-06-29T14:34:40.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.535+0000 7f6dc59e0640 1 -- 172.21.15.81:0/3491932580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc006f830 con 0x7f6dc0159200 2024-06-29T14:34:40.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.536+0000 7f6da7fff640 1 -- 172.21.15.81:0/3491932580 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 98389+0+0 (secure 0 0 0) 0x7f6db8033270 con 0x7f6dc0159200 2024-06-29T14:34:40.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.537+0000 7f6da7fff640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6dac0764d0 0x7f6dac0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:40.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.537+0000 7f6da7fff640 1 -- 172.21.15.81:0/3491932580 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6951+0+0 (secure 0 0 0) 0x7f6db80b9370 con 0x7f6dc0159200 2024-06-29T14:34:40.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.537+0000 7f6dbeffd640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6dac0764d0 0x7f6dac0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:40.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.537+0000 7f6dbeffd640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6dac0764d0 0x7f6dac0789e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6db0001b10 tx=0x7f6db0007e90 comp rx=0 tx=0).ready entity=mgr.5122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:40.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.541+0000 7f6da7fff640 1 -- 172.21.15.81:0/3491932580 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6db803f030 con 0x7f6dc0159200 2024-06-29T14:34:40.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:40.769+0000 7f6dc59e0640 1 -- 172.21.15.81:0/3491932580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f6dc00cf890 con 0x7f6dc0159200 2024-06-29T14:34:41.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.181+0000 7f6da7fff640 1 -- 172.21.15.81:0/3491932580 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) ==== 93+0+0 (secure 0 0 0) 0x7f6db8081e10 con 0x7f6dc0159200 2024-06-29T14:34:41.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.185+0000 7f6dc59e0640 1 -- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6dac0764d0 msgr2=0x7f6dac0789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.185+0000 7f6dc59e0640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6dac0764d0 0x7f6dac0789e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6db0001b10 tx=0x7f6db0007e90 comp rx=0 tx=0).stop 2024-06-29T14:34:41.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.185+0000 7f6dc59e0640 1 -- 172.21.15.81:0/3491932580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dc0159200 msgr2=0x7f6dc013bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.185+0000 7f6dc59e0640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dc0159200 0x7f6dc013bdc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6db802aea0 tx=0x7f6db8033f70 comp rx=0 tx=0).stop 2024-06-29T14:34:41.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.185+0000 7f6dc59e0640 1 -- 172.21.15.81:0/3491932580 shutdown_connections 2024-06-29T14:34:41.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.185+0000 7f6dc59e0640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f6dac0764d0 0x7f6dac0789e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.185+0000 7f6dc59e0640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dc0159200 0x7f6dc013bdc0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.185+0000 7f6dc59e0640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6dc015b400 0x7f6dc013b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.185+0000 7f6dc59e0640 1 --2- 172.21.15.81:0/3491932580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6dc00b4800 0x7f6dc013b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.185+0000 7f6dc59e0640 1 -- 172.21.15.81:0/3491932580 >> 172.21.15.81:0/3491932580 conn(0x7f6dc00b2920 msgr2=0x7f6dc015e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:41.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.186+0000 7f6dc59e0640 1 -- 172.21.15.81:0/3491932580 shutdown_connections 2024-06-29T14:34:41.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.186+0000 7f6dc59e0640 1 -- 172.21.15.81:0/3491932580 wait complete. 2024-06-29T14:34:41.198 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:34:41.198 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:34:41.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:34:41.293 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:34:41.293 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:34:41.293 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:34:41.293 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:41.309 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:34:41.310 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:41.366 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:41.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.524+0000 7f17a51f5640 1 -- 172.21.15.81:0/3101300652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17a00b5250 msgr2=0x7f17a00d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.524+0000 7f17a51f5640 1 --2- 172.21.15.81:0/3101300652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17a00b5250 0x7f17a00d09c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1788009830 tx=0x7f178802a5a0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.524+0000 7f17a51f5640 1 -- 172.21.15.81:0/3101300652 shutdown_connections 2024-06-29T14:34:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.524+0000 7f17a51f5640 1 --2- 172.21.15.81:0/3101300652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17a00b5250 0x7f17a00d09c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.524+0000 7f17a51f5640 1 --2- 172.21.15.81:0/3101300652 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f17a00b9040 0x7f17a00d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.524+0000 7f17a51f5640 1 -- 172.21.15.81:0/3101300652 >> 172.21.15.81:0/3101300652 conn(0x7f17a009cf30 msgr2=0x7f17a00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.524+0000 7f17a51f5640 1 -- 172.21.15.81:0/3101300652 shutdown_connections 2024-06-29T14:34:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.525+0000 7f17a51f5640 1 -- 172.21.15.81:0/3101300652 wait complete. 2024-06-29T14:34:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.526+0000 7f17a51f5640 1 Processor -- start 2024-06-29T14:34:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.526+0000 7f17a51f5640 1 -- start start 2024-06-29T14:34:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f17a51f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f17a00b5250 0x7f17a0151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f17a51f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17a00b7290 0x7f17a00b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f17a51f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17a00b9040 0x7f17a0160c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f17a51f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f17a00b7db0 con 0x7f17a00b7290 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f17a51f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f17a00d2d50 con 0x7f17a00b9040 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f17a51f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f17a00b7f30 con 0x7f17a00b5250 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f1797fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17a00b7290 0x7f17a00b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f1797fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17a00b7290 0x7f17a00b09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37638/0 (socket says 172.21.15.81:37638) 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f1797fff640 1 -- 172.21.15.81:0/4009171533 learned_addr learned my addr 172.21.15.81:0/4009171533 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f179ed9d640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f17a00b5250 0x7f17a0151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f179f59e640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17a00b9040 0x7f17a0160c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f1797fff640 1 -- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f17a00b5250 msgr2=0x7f17a0151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f1797fff640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f17a00b5250 0x7f17a0151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f1797fff640 1 -- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17a00b9040 msgr2=0x7f17a0160c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f1797fff640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17a00b9040 0x7f17a0160c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f1797fff640 1 -- 172.21.15.81:0/4009171533 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a00b3150 con 0x7f17a00b7290 2024-06-29T14:34:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.528+0000 7f1797fff640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17a00b7290 0x7f17a00b09b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f178802ab90 tx=0x7f1788033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.529+0000 7f17977fe640 1 -- 172.21.15.81:0/4009171533 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788005c90 con 0x7f17a00b7290 2024-06-29T14:34:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.529+0000 7f17977fe640 1 -- 172.21.15.81:0/4009171533 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1788031e50 con 0x7f17a00b7290 2024-06-29T14:34:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.529+0000 7f17977fe640 1 -- 172.21.15.81:0/4009171533 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788005260 con 0x7f17a00b7290 2024-06-29T14:34:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.529+0000 7f17a51f5640 1 -- 172.21.15.81:0/4009171533 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a01516c0 con 0x7f17a00b7290 2024-06-29T14:34:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.529+0000 7f17a51f5640 1 -- 172.21.15.81:0/4009171533 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a00b38a0 con 0x7f17a00b7290 2024-06-29T14:34:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.530+0000 7f17a51f5640 1 -- 172.21.15.81:0/4009171533 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a00b84c0 con 0x7f17a00b7290 2024-06-29T14:34:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.531+0000 7f17977fe640 1 -- 172.21.15.81:0/4009171533 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f1788005400 con 0x7f17a00b7290 2024-06-29T14:34:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.532+0000 7f17977fe640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1780076480 0x7f1780078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.532+0000 7f179ed9d640 1 -- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1780076480 msgr2=0x7f1780078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.532+0000 7f17977fe640 1 -- 172.21.15.81:0/4009171533 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6695+0+0 (secure 0 0 0) 0x7f17880b8e90 con 0x7f17a00b7290 2024-06-29T14:34:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.532+0000 7f179ed9d640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1780076480 0x7f1780078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:41.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.536+0000 7f17977fe640 1 -- 172.21.15.81:0/4009171533 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f178803f030 con 0x7f17a00b7290 2024-06-29T14:34:41.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.730+0000 7f17a51f5640 1 -- 172.21.15.81:0/4009171533 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f17a00b3320 con 0x7f17a00b7290 2024-06-29T14:34:41.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.732+0000 7f17977fe640 1 -- 172.21.15.81:0/4009171533 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+18164 (secure 0 0 0) 0x7f17a00b3320 con 0x7f17a00b7290 2024-06-29T14:34:41.730 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:41.731 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":46,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:34:41.144731+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:34.937915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:36.258024+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":218,"snap_epoch":46,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3793734276":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/3134616648":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6833/3101542440":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6832/3101542440":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/3819768847":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/2999883788":"2024-06-30T14:23:49.594083+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":9,"length":1},{"begin":11,"length":2},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":20,"length":1},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":27,"length":1},{"begin":31,"length":2},{"begin":34,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":43,"length":1},{"begin":45,"length":1},{"begin":47,"length":1},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":55,"length":1},{"begin":57,"length":1},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":65,"length":1},{"begin":67,"length":1},{"begin":69,"length":1},{"begin":74,"length":1},{"begin":76,"length":1},{"begin":78,"length":1},{"begin":80,"length":1},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":87,"length":1},{"begin":91,"length":2},{"begin":94,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":103,"length":1},{"begin":105,"length":1},{"begin":107,"length":1},{"begin":109,"length":1},{"begin":175,"length":44}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":78,"length":1},{"begin":218,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:41.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.733+0000 7f179ed9d640 1 -- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1780076480 msgr2=0x7f1780078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:41.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.733+0000 7f179ed9d640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1780076480 0x7f1780078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:41.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 -- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1780076480 msgr2=0x7f1780078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1780076480 0x7f1780078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 -- 172.21.15.81:0/4009171533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17a00b7290 msgr2=0x7f17a00b09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17a00b7290 0x7f17a00b09b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f178802ab90 tx=0x7f1788033c20 comp rx=0 tx=0).stop 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 -- 172.21.15.81:0/4009171533 shutdown_connections 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1780076480 0x7f1780078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f17a00b9040 0x7f17a0160c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17a00b7290 0x7f17a00b09b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 --2- 172.21.15.81:0/4009171533 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f17a00b5250 0x7f17a0151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 -- 172.21.15.81:0/4009171533 >> 172.21.15.81:0/4009171533 conn(0x7f17a009cf30 msgr2=0x7f17a015f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 -- 172.21.15.81:0/4009171533 shutdown_connections 2024-06-29T14:34:41.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.734+0000 7f17a51f5640 1 -- 172.21.15.81:0/4009171533 wait complete. 2024-06-29T14:34:41.744 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:23:34.937915+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:23:36.258024+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 218, 'snap_epoch': 46, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:34:41.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:34:41.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.865+0000 7fd92d931640 1 -- 172.21.15.81:0/489150236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928073030 msgr2=0x7fd928073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.865+0000 7fd92d931640 1 --2- 172.21.15.81:0/489150236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928073030 0x7fd928073440 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd910005fd0 tx=0x7fd91002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.865+0000 7fd92d931640 1 -- 172.21.15.81:0/489150236 shutdown_connections 2024-06-29T14:34:41.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.865+0000 7fd92d931640 1 --2- 172.21.15.81:0/489150236 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd92815e960 0x7fd92815ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.865+0000 7fd92d931640 1 --2- 172.21.15.81:0/489150236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928073030 0x7fd928073440 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.865+0000 7fd92d931640 1 -- 172.21.15.81:0/489150236 >> 172.21.15.81:0/489150236 conn(0x7fd92806ccc0 msgr2=0x7fd9280b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:41.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.865+0000 7fd92d931640 1 -- 172.21.15.81:0/489150236 shutdown_connections 2024-06-29T14:34:41.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.865+0000 7fd92d931640 1 -- 172.21.15.81:0/489150236 wait complete. 2024-06-29T14:34:41.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.867+0000 7fd92d931640 1 Processor -- start 2024-06-29T14:34:41.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.867+0000 7fd92d931640 1 -- start start 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd92d931640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928073030 0x7fd928157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd92d931640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9281412f0 0x7fd9280c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd92d931640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd92815e960 0x7fd92813dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd92d931640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd9280b3390 con 0x7fd928073030 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd92d931640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd9280b3210 con 0x7fd9281412f0 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd92d931640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd9280b3510 con 0x7fd92815e960 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd926ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928073030 0x7fd928157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd926ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928073030 0x7fd928157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37660/0 (socket says 172.21.15.81:37660) 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd926ffd640 1 -- 172.21.15.81:0/731168177 learned_addr learned my addr 172.21.15.81:0/731168177 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd91ffff640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9281412f0 0x7fd9280c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.868+0000 7fd9277fe640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd92815e960 0x7fd92813dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd926ffd640 1 -- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd92815e960 msgr2=0x7fd92813dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd926ffd640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd92815e960 0x7fd92813dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd926ffd640 1 -- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9281412f0 msgr2=0x7fd9280c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:41.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd926ffd640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9281412f0 0x7fd9280c57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:41.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd926ffd640 1 -- 172.21.15.81:0/731168177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9280c4a40 con 0x7fd928073030 2024-06-29T14:34:41.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd926ffd640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928073030 0x7fd928157f50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c3a0 tx=0x7fd91800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:41.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd924ff9640 1 -- 172.21.15.81:0/731168177 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918017070 con 0x7fd928073030 2024-06-29T14:34:41.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd924ff9640 1 -- 172.21.15.81:0/731168177 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9180053c0 con 0x7fd928073030 2024-06-29T14:34:41.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd92d931640 1 -- 172.21.15.81:0/731168177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9280c4ce0 con 0x7fd928073030 2024-06-29T14:34:41.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd924ff9640 1 -- 172.21.15.81:0/731168177 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918003850 con 0x7fd928073030 2024-06-29T14:34:41.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.869+0000 7fd92d931640 1 -- 172.21.15.81:0/731168177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9280c52a0 con 0x7fd928073030 2024-06-29T14:34:41.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.870+0000 7fd92d931640 1 -- 172.21.15.81:0/731168177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9280b3a10 con 0x7fd928073030 2024-06-29T14:34:41.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.871+0000 7fd924ff9640 1 -- 172.21.15.81:0/731168177 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7fd91801a070 con 0x7fd928073030 2024-06-29T14:34:41.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.872+0000 7fd924ff9640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd908076480 0x7fd908078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:41.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.872+0000 7fd91ffff640 1 -- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd908076480 msgr2=0x7fd908078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:41.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.872+0000 7fd924ff9640 1 -- 172.21.15.81:0/731168177 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6695+0+0 (secure 0 0 0) 0x7fd9180970c0 con 0x7fd928073030 2024-06-29T14:34:41.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.872+0000 7fd91ffff640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd908076480 0x7fd908078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:41.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:41.876+0000 7fd924ff9640 1 -- 172.21.15.81:0/731168177 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd928073980 con 0x7fd928073030 2024-06-29T14:34:42.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.066+0000 7fd92d931640 1 -- 172.21.15.81:0/731168177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd928072330 con 0x7fd928073030 2024-06-29T14:34:42.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.066+0000 7fd924ff9640 1 -- 172.21.15.81:0/731168177 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) ==== 93+0+10 (secure 0 0 0) 0x7fd91805fe10 con 0x7fd928073030 2024-06-29T14:34:42.065 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:34:42.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.068+0000 7fd92d931640 1 -- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd908076480 msgr2=0x7fd908078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:42.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.068+0000 7fd92d931640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd908076480 0x7fd908078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.068+0000 7fd92d931640 1 -- 172.21.15.81:0/731168177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928073030 msgr2=0x7fd928157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.068+0000 7fd92d931640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928073030 0x7fd928157f50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c3a0 tx=0x7fd91800c860 comp rx=0 tx=0).stop 2024-06-29T14:34:42.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.068+0000 7fd92d931640 1 -- 172.21.15.81:0/731168177 shutdown_connections 2024-06-29T14:34:42.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.068+0000 7fd92d931640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fd908076480 0x7fd908078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.068+0000 7fd92d931640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd92815e960 0x7fd92813dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.068+0000 7fd92d931640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9281412f0 0x7fd9280c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.069+0000 7fd92d931640 1 --2- 172.21.15.81:0/731168177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928073030 0x7fd928157f50 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.069+0000 7fd92d931640 1 -- 172.21.15.81:0/731168177 >> 172.21.15.81:0/731168177 conn(0x7fd92806ccc0 msgr2=0x7fd92815d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:42.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.069+0000 7fd92d931640 1 -- 172.21.15.81:0/731168177 shutdown_connections 2024-06-29T14:34:42.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.069+0000 7fd92d931640 1 -- 172.21.15.81:0/731168177 wait complete. 2024-06-29T14:34:42.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:34:42.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.209+0000 7f7f4e078640 1 -- 172.21.15.81:0/942558156 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f480b46a0 msgr2=0x7f7f48072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.209+0000 7f7f4e078640 1 --2- 172.21.15.81:0/942558156 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f480b46a0 0x7f7f48072f40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7f40005fd0 tx=0x7f7f4002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.209+0000 7f7f4e078640 1 -- 172.21.15.81:0/942558156 shutdown_connections 2024-06-29T14:34:42.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.209+0000 7f7f4e078640 1 --2- 172.21.15.81:0/942558156 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f4815b8f0 0x7f7f4806edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.209+0000 7f7f4e078640 1 --2- 172.21.15.81:0/942558156 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f480b46a0 0x7f7f48072f40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.209+0000 7f7f4e078640 1 -- 172.21.15.81:0/942558156 >> 172.21.15.81:0/942558156 conn(0x7f7f480d0b70 msgr2=0x7f7f480d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:42.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.209+0000 7f7f4e078640 1 -- 172.21.15.81:0/942558156 shutdown_connections 2024-06-29T14:34:42.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.209+0000 7f7f4e078640 1 -- 172.21.15.81:0/942558156 wait complete. 2024-06-29T14:34:42.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.212+0000 7f7f4e078640 1 Processor -- start 2024-06-29T14:34:42.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.212+0000 7f7f4e078640 1 -- start start 2024-06-29T14:34:42.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f4e078640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f48073480 0x7f7f481eccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f4e078640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f480b46a0 0x7f7f480ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f4e078640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f4815b8f0 0x7f7f4815c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f4e078640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4806f680 con 0x7f7f480b46a0 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f4e078640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4806f500 con 0x7f7f4815b8f0 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f4e078640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7f4806f800 con 0x7f7f48073480 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f46ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f480b46a0 0x7f7f480ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f46ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f480b46a0 0x7f7f480ae160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37672/0 (socket says 172.21.15.81:37672) 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3378983667 learned_addr learned my addr 172.21.15.81:0/3378983667 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f477fe640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f48073480 0x7f7f481eccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.214+0000 7f7f47fff640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f4815b8f0 0x7f7f4815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f48073480 msgr2=0x7f7f481eccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f46ffd640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f48073480 0x7f7f481eccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f4815b8f0 msgr2=0x7f7f4815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f46ffd640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f4815b8f0 0x7f7f4815c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3378983667 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f480aeb40 con 0x7f7f480b46a0 2024-06-29T14:34:42.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f46ffd640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f480b46a0 0x7f7f480ae160 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c00c3a0 tx=0x7f7f3c00c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:42.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f2ffff640 1 -- 172.21.15.81:0/3378983667 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c011790 con 0x7f7f480b46a0 2024-06-29T14:34:42.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f2ffff640 1 -- 172.21.15.81:0/3378983667 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f3c017070 con 0x7f7f480b46a0 2024-06-29T14:34:42.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f4e078640 1 -- 172.21.15.81:0/3378983667 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f480aee20 con 0x7f7f480b46a0 2024-06-29T14:34:42.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f2ffff640 1 -- 172.21.15.81:0/3378983667 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c011c30 con 0x7f7f480b46a0 2024-06-29T14:34:42.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.215+0000 7f7f4e078640 1 -- 172.21.15.81:0/3378983667 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f480af3e0 con 0x7f7f480b46a0 2024-06-29T14:34:42.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.216+0000 7f7f4e078640 1 -- 172.21.15.81:0/3378983667 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f48072f40 con 0x7f7f480b46a0 2024-06-29T14:34:42.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.217+0000 7f7f2ffff640 1 -- 172.21.15.81:0/3378983667 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f7f3c01c460 con 0x7f7f480b46a0 2024-06-29T14:34:42.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.217+0000 7f7f2ffff640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f7f34076480 0x7f7f34078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.217+0000 7f7f2ffff640 1 -- 172.21.15.81:0/3378983667 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6743+0+0 (secure 0 0 0) 0x7f7f3c097a80 con 0x7f7f480b46a0 2024-06-29T14:34:42.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.217+0000 7f7f477fe640 1 -- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f7f34076480 msgr2=0x7f7f34078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:42.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.217+0000 7f7f477fe640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f7f34076480 0x7f7f34078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:42.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.222+0000 7f7f2ffff640 1 -- 172.21.15.81:0/3378983667 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7f48073890 con 0x7f7f480b46a0 2024-06-29T14:34:42.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.409+0000 7f7f4e078640 1 -- 172.21.15.81:0/3378983667 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7f480d0730 con 0x7f7f480b46a0 2024-06-29T14:34:42.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.409+0000 7f7f2ffff640 1 -- 172.21.15.81:0/3378983667 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v47) ==== 104+0+11 (secure 0 0 0) 0x7f7f3c060de0 con 0x7f7f480b46a0 2024-06-29T14:34:42.408 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:34:42.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 -- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f7f34076480 msgr2=0x7f7f34078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:42.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f7f34076480 0x7f7f34078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 -- 172.21.15.81:0/3378983667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f480b46a0 msgr2=0x7f7f480ae160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f480b46a0 0x7f7f480ae160 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c00c3a0 tx=0x7f7f3c00c7a0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 -- 172.21.15.81:0/3378983667 shutdown_connections 2024-06-29T14:34:42.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f7f34076480 0x7f7f34078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f4815b8f0 0x7f7f4815c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f480b46a0 0x7f7f480ae160 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 --2- 172.21.15.81:0/3378983667 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f48073480 0x7f7f481eccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 -- 172.21.15.81:0/3378983667 >> 172.21.15.81:0/3378983667 conn(0x7f7f480d0b70 msgr2=0x7f7f480b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:42.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 -- 172.21.15.81:0/3378983667 shutdown_connections 2024-06-29T14:34:42.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.411+0000 7f7f4e078640 1 -- 172.21.15.81:0/3378983667 wait complete. 2024-06-29T14:34:42.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:34:42.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.542+0000 7f794be9e640 1 -- 172.21.15.81:0/2435542552 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7930005300 msgr2=0x7f793000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.542+0000 7f794be9e640 1 --2- 172.21.15.81:0/2435542552 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7930005300 0x7f793000b9c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f793c001ed0 tx=0x7f7930005830 comp rx=0 tx=0).stop 2024-06-29T14:34:42.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.542+0000 7f794a43b640 1 -- 172.21.15.81:0/2435542552 reap_dead start 2024-06-29T14:34:42.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.542+0000 7f794be9e640 1 -- 172.21.15.81:0/2435542552 shutdown_connections 2024-06-29T14:34:42.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.542+0000 7f794be9e640 1 -- 172.21.15.81:0/2435542552 >> 172.21.15.81:0/2435542552 conn(0x7f79440b2920 msgr2=0x7f79440b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:42.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.542+0000 7f794be9e640 1 -- 172.21.15.81:0/2435542552 shutdown_connections 2024-06-29T14:34:42.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.542+0000 7f794be9e640 1 -- 172.21.15.81:0/2435542552 wait complete. 2024-06-29T14:34:42.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.544+0000 7f794be9e640 1 Processor -- start 2024-06-29T14:34:42.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.544+0000 7f794be9e640 1 -- start start 2024-06-29T14:34:42.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.545+0000 7f794be9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79440cc9c0 0x7f79440c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.545+0000 7f794be9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79441f4ce0 0x7f79441f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.545+0000 7f794be9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79441f5630 0x7f794414e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.545+0000 7f794be9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f794415c1c0 con 0x7f79441f4ce0 2024-06-29T14:34:42.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.545+0000 7f794be9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7944073860 con 0x7f79441f5630 2024-06-29T14:34:42.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.545+0000 7f794be9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f794415b6e0 con 0x7f79440cc9c0 2024-06-29T14:34:42.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f7949439640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79441f4ce0 0x7f79441f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f7949439640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79441f4ce0 0x7f79441f50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37694/0 (socket says 172.21.15.81:37694) 2024-06-29T14:34:42.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f7949439640 1 -- 172.21.15.81:0/2603744092 learned_addr learned my addr 172.21.15.81:0/2603744092 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:42.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f7949c3a640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79440cc9c0 0x7f79440c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f794a43b640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79441f5630 0x7f794414e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f7949439640 1 -- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79440cc9c0 msgr2=0x7f79440c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f7949439640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79440cc9c0 0x7f79440c9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f7949439640 1 -- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79441f5630 msgr2=0x7f794414e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f7949439640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79441f5630 0x7f794414e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f7949439640 1 -- 172.21.15.81:0/2603744092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79440cbe10 con 0x7f79441f4ce0 2024-06-29T14:34:42.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f7949439640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79441f4ce0 0x7f79441f50f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f79400119f0 tx=0x7f7940011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:42.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f793a7fc640 1 -- 172.21.15.81:0/2603744092 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794000fc50 con 0x7f79441f4ce0 2024-06-29T14:34:42.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f793a7fc640 1 -- 172.21.15.81:0/2603744092 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7940004030 con 0x7f79441f4ce0 2024-06-29T14:34:42.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f794be9e640 1 -- 172.21.15.81:0/2603744092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7944149de0 con 0x7f79441f4ce0 2024-06-29T14:34:42.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f793a7fc640 1 -- 172.21.15.81:0/2603744092 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940004460 con 0x7f79441f4ce0 2024-06-29T14:34:42.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.546+0000 7f794be9e640 1 -- 172.21.15.81:0/2603744092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79440ca730 con 0x7f79441f4ce0 2024-06-29T14:34:42.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.547+0000 7f794be9e640 1 -- 172.21.15.81:0/2603744092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7944072f40 con 0x7f79441f4ce0 2024-06-29T14:34:42.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.548+0000 7f793a7fc640 1 -- 172.21.15.81:0/2603744092 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f7940005020 con 0x7f79441f4ce0 2024-06-29T14:34:42.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.549+0000 7f793a7fc640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f79280764b0 0x7f79280789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.549+0000 7f7949c3a640 1 -- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f79280764b0 msgr2=0x7f79280789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:42.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.549+0000 7f793a7fc640 1 -- 172.21.15.81:0/2603744092 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6743+0+0 (secure 0 0 0) 0x7f7940098a70 con 0x7f79441f4ce0 2024-06-29T14:34:42.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.549+0000 7f7949c3a640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f79280764b0 0x7f79280789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:42.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.553+0000 7f793a7fc640 1 -- 172.21.15.81:0/2603744092 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7940061690 con 0x7f79441f4ce0 2024-06-29T14:34:42.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.740+0000 7f794be9e640 1 -- 172.21.15.81:0/2603744092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f794415f5f0 con 0x7f79441f4ce0 2024-06-29T14:34:42.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.740+0000 7f793a7fc640 1 -- 172.21.15.81:0/2603744092 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v47) ==== 100+0+11 (secure 0 0 0) 0x7f7940065990 con 0x7f79441f4ce0 2024-06-29T14:34:42.739 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:34:42.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 -- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f79280764b0 msgr2=0x7f79280789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:42.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f79280764b0 0x7f79280789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 -- 172.21.15.81:0/2603744092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79441f4ce0 msgr2=0x7f79441f50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79441f4ce0 0x7f79441f50f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f79400119f0 tx=0x7f7940011eb0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 -- 172.21.15.81:0/2603744092 shutdown_connections 2024-06-29T14:34:42.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f79280764b0 0x7f79280789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f79441f5630 0x7f794414e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79441f4ce0 0x7f79441f50f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 --2- 172.21.15.81:0/2603744092 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f79440cc9c0 0x7f79440c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 -- 172.21.15.81:0/2603744092 >> 172.21.15.81:0/2603744092 conn(0x7f79440b2920 msgr2=0x7f79440b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:42.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 -- 172.21.15.81:0/2603744092 shutdown_connections 2024-06-29T14:34:42.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.742+0000 7f794be9e640 1 -- 172.21.15.81:0/2603744092 wait complete. 2024-06-29T14:34:42.752 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:34:42.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.870+0000 7f5e57111640 1 -- 172.21.15.81:0/4065966882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e500b3200 msgr2=0x7f5e500cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.870+0000 7f5e57111640 1 --2- 172.21.15.81:0/4065966882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e500b3200 0x7f5e500cc7b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5e500c5590 tx=0x7f5e4402a560 comp rx=0 tx=0).stop 2024-06-29T14:34:42.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.870+0000 7f5e57111640 1 -- 172.21.15.81:0/4065966882 shutdown_connections 2024-06-29T14:34:42.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.870+0000 7f5e57111640 1 --2- 172.21.15.81:0/4065966882 >> v2:172.21.15.167:3300/0 conn(0x7f5e500d0940 0x7f5e500b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.870+0000 7f5e57111640 1 --2- 172.21.15.81:0/4065966882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e500b3200 0x7f5e500cc7b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.870+0000 7f5e57111640 1 -- 172.21.15.81:0/4065966882 >> 172.21.15.81:0/4065966882 conn(0x7f5e500b3680 msgr2=0x7f5e500b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:42.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.870+0000 7f5e57111640 1 -- 172.21.15.81:0/4065966882 shutdown_connections 2024-06-29T14:34:42.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.870+0000 7f5e57111640 1 -- 172.21.15.81:0/4065966882 wait complete. 2024-06-29T14:34:42.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.872+0000 7f5e57111640 1 Processor -- start 2024-06-29T14:34:42.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.872+0000 7f5e57111640 1 -- start start 2024-06-29T14:34:42.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.873+0000 7f5e57111640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e500b3200 0x7f5e500c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.873+0000 7f5e57111640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e500cccf0 0x7f5e500cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.873+0000 7f5e57111640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e501f7a50 0x7f5e501e5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.873+0000 7f5e57111640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5e500b5bb0 con 0x7f5e500cccf0 2024-06-29T14:34:42.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.873+0000 7f5e57111640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5e500b5a30 con 0x7f5e500b3200 2024-06-29T14:34:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.873+0000 7f5e57111640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5e500b5d30 con 0x7f5e501f7a50 2024-06-29T14:34:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.873+0000 7f5e4ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e500cccf0 0x7f5e500cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.873+0000 7f5e4ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e500cccf0 0x7f5e500cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37704/0 (socket says 172.21.15.81:37704) 2024-06-29T14:34:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.873+0000 7f5e4ffff640 1 -- 172.21.15.81:0/3476147663 learned_addr learned my addr 172.21.15.81:0/3476147663 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.873+0000 7f5e54ead640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e500b3200 0x7f5e500c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e556ae640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e501f7a50 0x7f5e501e5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e4ffff640 1 -- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e501f7a50 msgr2=0x7f5e501e5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e4ffff640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e501f7a50 0x7f5e501e5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e4ffff640 1 -- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e500b3200 msgr2=0x7f5e500c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e4ffff640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e500b3200 0x7f5e500c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e4ffff640 1 -- 172.21.15.81:0/3476147663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e500c94a0 con 0x7f5e500cccf0 2024-06-29T14:34:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e4ffff640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e500cccf0 0x7f5e500cafc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00c3a0 tx=0x7f5e3c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e4d7fa640 1 -- 172.21.15.81:0/3476147663 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c017070 con 0x7f5e500cccf0 2024-06-29T14:34:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e4d7fa640 1 -- 172.21.15.81:0/3476147663 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e3c0053c0 con 0x7f5e500cccf0 2024-06-29T14:34:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e4d7fa640 1 -- 172.21.15.81:0/3476147663 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c003850 con 0x7f5e500cccf0 2024-06-29T14:34:42.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e57111640 1 -- 172.21.15.81:0/3476147663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e500c9780 con 0x7f5e500cccf0 2024-06-29T14:34:42.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.874+0000 7f5e57111640 1 -- 172.21.15.81:0/3476147663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e500c9d40 con 0x7f5e500cccf0 2024-06-29T14:34:42.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.875+0000 7f5e57111640 1 -- 172.21.15.81:0/3476147663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e500ca120 con 0x7f5e500cccf0 2024-06-29T14:34:42.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.876+0000 7f5e4d7fa640 1 -- 172.21.15.81:0/3476147663 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f5e3c01a070 con 0x7f5e500cccf0 2024-06-29T14:34:42.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.876+0000 7f5e4d7fa640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5e34076480 0x7f5e34078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:42.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.877+0000 7f5e54ead640 1 -- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5e34076480 msgr2=0x7f5e34078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:42.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.877+0000 7f5e4d7fa640 1 -- 172.21.15.81:0/3476147663 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6743+0+0 (secure 0 0 0) 0x7f5e3c0970f0 con 0x7f5e500cccf0 2024-06-29T14:34:42.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.877+0000 7f5e54ead640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5e34076480 0x7f5e34078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:42.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:42.881+0000 7f5e4d7fa640 1 -- 172.21.15.81:0/3476147663 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5e500ca120 con 0x7f5e500cccf0 2024-06-29T14:34:43.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.076+0000 7f5e57111640 1 -- 172.21.15.81:0/3476147663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5e500cc7b0 con 0x7f5e500cccf0 2024-06-29T14:34:43.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.076+0000 7f5e4d7fa640 1 -- 172.21.15.81:0/3476147663 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v47) ==== 80+0+360 (secure 0 0 0) 0x7f5e3c05fe20 con 0x7f5e500cccf0 2024-06-29T14:34:43.075 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3793734276 2024-06-30T14:23:49.594083+0000 2024-06-29T14:34:43.075 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3134616648 2024-06-30T14:23:49.594083+0000 2024-06-29T14:34:43.075 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/3101542440 2024-06-30T14:23:49.594083+0000 2024-06-29T14:34:43.075 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/3101542440 2024-06-30T14:23:49.594083+0000 2024-06-29T14:34:43.075 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3819768847 2024-06-30T14:23:49.594083+0000 2024-06-29T14:34:43.075 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2999883788 2024-06-30T14:23:49.594083+0000 2024-06-29T14:34:43.076 INFO:teuthology.orchestra.run.smithi081.stderr:listed 6 entries 2024-06-29T14:34:43.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.078+0000 7f5e54ead640 1 -- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5e34076480 msgr2=0x7f5e34078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:43.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.078+0000 7f5e54ead640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5e34076480 0x7f5e34078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:43.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.078+0000 7f5e57111640 1 -- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5e34076480 msgr2=0x7f5e34078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:43.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.078+0000 7f5e57111640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5e34076480 0x7f5e34078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.078+0000 7f5e57111640 1 -- 172.21.15.81:0/3476147663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e500cccf0 msgr2=0x7f5e500cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.079+0000 7f5e57111640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e500cccf0 0x7f5e500cafc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c00c3a0 tx=0x7f5e3c00c860 comp rx=0 tx=0).stop 2024-06-29T14:34:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.079+0000 7f5e57111640 1 -- 172.21.15.81:0/3476147663 shutdown_connections 2024-06-29T14:34:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.079+0000 7f5e57111640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f5e34076480 0x7f5e34078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.079+0000 7f5e57111640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e501f7a50 0x7f5e501e5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.079+0000 7f5e57111640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e500cccf0 0x7f5e500cafc0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.079+0000 7f5e57111640 1 --2- 172.21.15.81:0/3476147663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e500b3200 0x7f5e500c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.079+0000 7f5e57111640 1 -- 172.21.15.81:0/3476147663 >> 172.21.15.81:0/3476147663 conn(0x7f5e500b3680 msgr2=0x7f5e50070300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.079+0000 7f5e57111640 1 -- 172.21.15.81:0/3476147663 shutdown_connections 2024-06-29T14:34:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.079+0000 7f5e57111640 1 -- 172.21.15.81:0/3476147663 wait complete. 2024-06-29T14:34:43.089 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:34:43.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:43.089 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:34:43.121 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:34:43.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:34:43.122 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:34:43.176 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:34:43.177 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:34:43.177 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:43.193 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:34:43.193 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:34:43.193 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:34:43.193 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:43.232 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:34:43.232 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:34:43.232 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:34:43.232 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:43.249 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:34:43.250 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:34:43.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.389+0000 7f2009363640 1 -- 172.21.15.81:0/233048599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff0003a00 msgr2=0x7f1ff0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.389+0000 7f2009363640 1 --2- 172.21.15.81:0/233048599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff0003a00 0x7f1ff0003e70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009b50 tx=0x7f1ff0005830 comp rx=0 tx=0).stop 2024-06-29T14:34:43.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.389+0000 7f20037fe640 1 -- 172.21.15.81:0/233048599 reap_dead start 2024-06-29T14:34:43.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.389+0000 7f2009363640 1 -- 172.21.15.81:0/233048599 shutdown_connections 2024-06-29T14:34:43.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.389+0000 7f2009363640 1 -- 172.21.15.81:0/233048599 >> 172.21.15.81:0/233048599 conn(0x7f20040b3680 msgr2=0x7f20040b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.389+0000 7f2009363640 1 -- 172.21.15.81:0/233048599 shutdown_connections 2024-06-29T14:34:43.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.389+0000 7f2009363640 1 -- 172.21.15.81:0/233048599 wait complete. 2024-06-29T14:34:43.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.391+0000 7f2009363640 1 Processor -- start 2024-06-29T14:34:43.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.391+0000 7f2009363640 1 -- start start 2024-06-29T14:34:43.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f2009363640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff0003a00 0x7f20040c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f2009363640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2004156780 0x7f200409f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f2009363640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2004157ab0 0x7f2004157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f2009363640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f20040d03b0 con 0x7f2004156780 2024-06-29T14:34:43.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f2009363640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f20040d0230 con 0x7f1ff0003a00 2024-06-29T14:34:43.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f2009363640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f20040d0530 con 0x7f2004157ab0 2024-06-29T14:34:43.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f1ffbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2004156780 0x7f200409f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f1ffbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2004156780 0x7f200409f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37716/0 (socket says 172.21.15.81:37716) 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f1ffbfff640 1 -- 172.21.15.81:0/479501874 learned_addr learned my addr 172.21.15.81:0/479501874 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f2002ffd640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff0003a00 0x7f20040c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.392+0000 7f20037fe640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2004157ab0 0x7f2004157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f1ffbfff640 1 -- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2004157ab0 msgr2=0x7f2004157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f1ffbfff640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2004157ab0 0x7f2004157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f1ffbfff640 1 -- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff0003a00 msgr2=0x7f20040c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f1ffbfff640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff0003a00 0x7f20040c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f1ffbfff640 1 -- 172.21.15.81:0/479501874 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200406e570 con 0x7f2004156780 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f2002ffd640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff0003a00 0x7f20040c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f1ffbfff640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2004156780 0x7f200409f2b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1fec011ae0 tx=0x7f1fec011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f1ffb7fe640 1 -- 172.21.15.81:0/479501874 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec004440 con 0x7f2004156780 2024-06-29T14:34:43.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f1ffb7fe640 1 -- 172.21.15.81:0/479501874 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fec00f9a0 con 0x7f2004156780 2024-06-29T14:34:43.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f2009363640 1 -- 172.21.15.81:0/479501874 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200406eaa0 con 0x7f2004156780 2024-06-29T14:34:43.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f1ffb7fe640 1 -- 172.21.15.81:0/479501874 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00fdd0 con 0x7f2004156780 2024-06-29T14:34:43.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.393+0000 7f2009363640 1 -- 172.21.15.81:0/479501874 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20041572b0 con 0x7f2004156780 2024-06-29T14:34:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.394+0000 7f2009363640 1 -- 172.21.15.81:0/479501874 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20040cc7b0 con 0x7f2004156780 2024-06-29T14:34:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.395+0000 7f1ffb7fe640 1 -- 172.21.15.81:0/479501874 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f1fec0035f0 con 0x7f2004156780 2024-06-29T14:34:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.395+0000 7f1ffb7fe640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1fe40764b0 0x7f1fe40789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.396+0000 7f2002ffd640 1 -- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1fe40764b0 msgr2=0x7f1fe40789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.396+0000 7f1ffb7fe640 1 -- 172.21.15.81:0/479501874 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6743+0+0 (secure 0 0 0) 0x7f1fec098d60 con 0x7f2004156780 2024-06-29T14:34:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.396+0000 7f2002ffd640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1fe40764b0 0x7f1fe40789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:43.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.400+0000 7f1ffb7fe640 1 -- 172.21.15.81:0/479501874 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1fec061fb0 con 0x7f2004156780 2024-06-29T14:34:43.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.597+0000 7f2002ffd640 1 -- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1fe40764b0 msgr2=0x7f1fe40789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:43.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.597+0000 7f2002ffd640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1fe40764b0 0x7f1fe40789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:34:43.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.619+0000 7f2009363640 1 -- 172.21.15.81:0/479501874 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20040b3e10 con 0x7f2004156780 2024-06-29T14:34:43.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.620+0000 7f1ffb7fe640 1 -- 172.21.15.81:0/479501874 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5220 (secure 0 0 0) 0x7f1fec0618b0 con 0x7f2004156780 2024-06-29T14:34:43.619 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:43.619 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"btime":"2024-06-29T14:23:40:952859+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3513859413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3513859413},{"type":"v1","addr":"172.21.15.81:6835","nonce":3513859413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4775,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/763942647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":763942647},{"type":"v1","addr":"172.21.15.167:6836","nonce":763942647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1148290480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1148290480},{"type":"v1","addr":"172.21.15.167:6837","nonce":1148290480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:37.343197+0000","modified":"2024-06-29T14:23:40.325922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5172":{"gid":5172,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/58595811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":58595811},{"type":"v1","addr":"172.21.15.81:6834","nonce":58595811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5172,"qdb_cluster":[5172]},"id":2}]} 2024-06-29T14:34:43.619 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 16 2024-06-29T14:34:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 -- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1fe40764b0 msgr2=0x7f1fe40789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1fe40764b0 0x7f1fe40789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 -- 172.21.15.81:0/479501874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2004156780 msgr2=0x7f200409f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2004156780 0x7f200409f2b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1fec011ae0 tx=0x7f1fec011ee0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 -- 172.21.15.81:0/479501874 shutdown_connections 2024-06-29T14:34:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f1fe40764b0 0x7f1fe40789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2004157ab0 0x7f2004157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2004156780 0x7f200409f2b0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 --2- 172.21.15.81:0/479501874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff0003a00 0x7f20040c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 -- 172.21.15.81:0/479501874 >> 172.21.15.81:0/479501874 conn(0x7f20040b3680 msgr2=0x7f2004071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 -- 172.21.15.81:0/479501874 shutdown_connections 2024-06-29T14:34:43.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.622+0000 7f2009363640 1 -- 172.21.15.81:0/479501874 wait complete. 2024-06-29T14:34:43.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:43.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.747+0000 7f61ebb0a640 1 -- 172.21.15.81:0/37016251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61e40d0b90 msgr2=0x7f61e40d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.747+0000 7f61ebb0a640 1 --2- 172.21.15.81:0/37016251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61e40d0b90 0x7f61e40d1a80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009830 tx=0x7f61e002a640 comp rx=0 tx=0).stop 2024-06-29T14:34:43.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.747+0000 7f61ebb0a640 1 -- 172.21.15.81:0/37016251 shutdown_connections 2024-06-29T14:34:43.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.747+0000 7f61ebb0a640 1 --2- 172.21.15.81:0/37016251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61e40d0b90 0x7f61e40d1a80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.747+0000 7f61ebb0a640 1 -- 172.21.15.81:0/37016251 >> 172.21.15.81:0/37016251 conn(0x7f61e409cf30 msgr2=0x7f61e40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.747+0000 7f61ebb0a640 1 -- 172.21.15.81:0/37016251 shutdown_connections 2024-06-29T14:34:43.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.747+0000 7f61ebb0a640 1 -- 172.21.15.81:0/37016251 wait complete. 2024-06-29T14:34:43.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.749+0000 7f61ebb0a640 1 Processor -- start 2024-06-29T14:34:43.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.749+0000 7f61ebb0a640 1 -- start start 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ebb0a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61e40b5250 0x7f61e4150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ebb0a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61e40b9040 0x7f61e41f8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ebb0a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61e41f86c0 0x7f61e41573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ebb0a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61e40d26d0 con 0x7f61e41f86c0 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ebb0a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f61e40b85c0 con 0x7f61e40b9040 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ebb0a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f61e40d2850 con 0x7f61e40b5250 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ea0a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61e41f86c0 0x7f61e41573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ea0a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61e41f86c0 0x7f61e41573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37732/0 (socket says 172.21.15.81:37732) 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ea0a7640 1 -- 172.21.15.81:0/993982828 learned_addr learned my addr 172.21.15.81:0/993982828 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61e90a5640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61e40b9040 0x7f61e41f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61e98a6640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61e40b5250 0x7f61e4150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ea0a7640 1 -- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61e40b5250 msgr2=0x7f61e4150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ea0a7640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61e40b5250 0x7f61e4150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ea0a7640 1 -- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61e40b9040 msgr2=0x7f61e41f8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ea0a7640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61e40b9040 0x7f61e41f8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61ea0a7640 1 -- 172.21.15.81:0/993982828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e41f8e50 con 0x7f61e41f86c0 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.750+0000 7f61e90a5640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61e40b9040 0x7f61e41f8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.751+0000 7f61ea0a7640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61e41f86c0 0x7f61e41573b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00c6b0 tx=0x7f61dc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.751+0000 7f61da7fc640 1 -- 172.21.15.81:0/993982828 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc003b70 con 0x7f61e41f86c0 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.751+0000 7f61da7fc640 1 -- 172.21.15.81:0/993982828 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61dc003d10 con 0x7f61e41f86c0 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.751+0000 7f61da7fc640 1 -- 172.21.15.81:0/993982828 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc005330 con 0x7f61e41f86c0 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.751+0000 7f61ebb0a640 1 -- 172.21.15.81:0/993982828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61e4154930 con 0x7f61e41f86c0 2024-06-29T14:34:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.751+0000 7f61ebb0a640 1 -- 172.21.15.81:0/993982828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61e40a1790 con 0x7f61e41f86c0 2024-06-29T14:34:43.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.752+0000 7f61ebb0a640 1 -- 172.21.15.81:0/993982828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61e40d46a0 con 0x7f61e41f86c0 2024-06-29T14:34:43.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.753+0000 7f61da7fc640 1 -- 172.21.15.81:0/993982828 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f61dc0054d0 con 0x7f61e41f86c0 2024-06-29T14:34:43.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.753+0000 7f61da7fc640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f61c8076480 0x7f61c8078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:43.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.753+0000 7f61e98a6640 1 -- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f61c8076480 msgr2=0x7f61c8078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:43.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.753+0000 7f61da7fc640 1 -- 172.21.15.81:0/993982828 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6743+0+0 (secure 0 0 0) 0x7f61dc097b80 con 0x7f61e41f86c0 2024-06-29T14:34:43.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.753+0000 7f61e98a6640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f61c8076480 0x7f61c8078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:43.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.762+0000 7f61da7fc640 1 -- 172.21.15.81:0/993982828 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f61e40d46a0 con 0x7f61e41f86c0 2024-06-29T14:34:43.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.941+0000 7f61ebb0a640 1 -- 172.21.15.81:0/993982828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61e40d2de0 con 0x7f61e41f86c0 2024-06-29T14:34:43.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.943+0000 7f61da7fc640 1 -- 172.21.15.81:0/993982828 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+18238 (secure 0 0 0) 0x7f61dc060810 con 0x7f61e41f86c0 2024-06-29T14:34:43.942 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:43.942 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":47,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:34:42.150514+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:34.937915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:36.258024+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":47,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3793734276":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/3134616648":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6833/3101542440":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6832/3101542440":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/3819768847":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/2999883788":"2024-06-30T14:23:49.594083+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":9,"length":1},{"begin":11,"length":2},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":20,"length":1},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":27,"length":1},{"begin":31,"length":2},{"begin":34,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":43,"length":1},{"begin":45,"length":1},{"begin":47,"length":1},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":55,"length":1},{"begin":57,"length":1},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":65,"length":1},{"begin":67,"length":1},{"begin":69,"length":1},{"begin":74,"length":1},{"begin":76,"length":1},{"begin":78,"length":1},{"begin":80,"length":1},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":87,"length":1},{"begin":89,"length":1},{"begin":91,"length":2},{"begin":94,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":105,"length":1},{"begin":107,"length":1},{"begin":109,"length":1},{"begin":175,"length":46}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":89,"length":1},{"begin":100,"length":1},{"begin":219,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:43.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 -- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f61c8076480 msgr2=0x7f61c8078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:43.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f61c8076480 0x7f61c8078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 -- 172.21.15.81:0/993982828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61e41f86c0 msgr2=0x7f61e41573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:43.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61e41f86c0 0x7f61e41573b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00c6b0 tx=0x7f61dc00cb70 comp rx=0 tx=0).stop 2024-06-29T14:34:43.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 -- 172.21.15.81:0/993982828 shutdown_connections 2024-06-29T14:34:43.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f61c8076480 0x7f61c8078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61e41f86c0 0x7f61e41573b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61e40b9040 0x7f61e41f8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 --2- 172.21.15.81:0/993982828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61e40b5250 0x7f61e4150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:43.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 -- 172.21.15.81:0/993982828 >> 172.21.15.81:0/993982828 conn(0x7f61e409cf30 msgr2=0x7f61e415e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:43.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 -- 172.21.15.81:0/993982828 shutdown_connections 2024-06-29T14:34:43.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:43.945+0000 7f61ebb0a640 1 -- 172.21.15.81:0/993982828 wait complete. 2024-06-29T14:34:43.955 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:23:34.937915+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:23:36.258024+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 220, 'snap_epoch': 47, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:34:43.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:34:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.075+0000 7f405b69e640 1 -- 172.21.15.81:0/1403993841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40540cd150 msgr2=0x7f4054072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.075+0000 7f405b69e640 1 --2- 172.21.15.81:0/1403993841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40540cd150 0x7f4054072f40 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4044009830 tx=0x7f404402a530 comp rx=0 tx=0).stop 2024-06-29T14:34:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.075+0000 7f405b69e640 1 -- 172.21.15.81:0/1403993841 shutdown_connections 2024-06-29T14:34:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.075+0000 7f405b69e640 1 --2- 172.21.15.81:0/1403993841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40540cd150 0x7f4054072f40 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.075+0000 7f405b69e640 1 -- 172.21.15.81:0/1403993841 >> 172.21.15.81:0/1403993841 conn(0x7f40540b2920 msgr2=0x7f40540b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.075+0000 7f405b69e640 1 -- 172.21.15.81:0/1403993841 shutdown_connections 2024-06-29T14:34:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.075+0000 7f405b69e640 1 -- 172.21.15.81:0/1403993841 wait complete. 2024-06-29T14:34:44.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.077+0000 7f405b69e640 1 Processor -- start 2024-06-29T14:34:44.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.077+0000 7f405b69e640 1 -- start start 2024-06-29T14:34:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f405b69e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4054073480 0x7f4054145410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f405b69e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40540cd150 0x7f405406c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f405b69e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f405415bdf0 0x7f405406c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f405b69e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f405406f830 con 0x7f405415bdf0 2024-06-29T14:34:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f405b69e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f405406f6b0 con 0x7f40540cd150 2024-06-29T14:34:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f405b69e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f405406f9b0 con 0x7f4054073480 2024-06-29T14:34:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f4059c3b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f405415bdf0 0x7f405406c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f4059c3b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f405415bdf0 0x7f405406c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37748/0 (socket says 172.21.15.81:37748) 2024-06-29T14:34:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f4059c3b640 1 -- 172.21.15.81:0/2313777506 learned_addr learned my addr 172.21.15.81:0/2313777506 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f405943a640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4054073480 0x7f4054145410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.078+0000 7f4058c39640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40540cd150 0x7f405406c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f4059c3b640 1 -- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4054073480 msgr2=0x7f4054145410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f4059c3b640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4054073480 0x7f4054145410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f4059c3b640 1 -- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40540cd150 msgr2=0x7f405406c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f4059c3b640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40540cd150 0x7f405406c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f4059c3b640 1 -- 172.21.15.81:0/2313777506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4054157160 con 0x7f405415bdf0 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f4059c3b640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f405415bdf0 0x7f405406c830 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f404c00cb10 tx=0x7f404c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f4049ffb640 1 -- 172.21.15.81:0/2313777506 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c016020 con 0x7f405415bdf0 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f4049ffb640 1 -- 172.21.15.81:0/2313777506 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f404c00ad00 con 0x7f405415bdf0 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f405b69e640 1 -- 172.21.15.81:0/2313777506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40541573e0 con 0x7f405415bdf0 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f4049ffb640 1 -- 172.21.15.81:0/2313777506 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c012680 con 0x7f405415bdf0 2024-06-29T14:34:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.079+0000 7f405b69e640 1 -- 172.21.15.81:0/2313777506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40541579a0 con 0x7f405415bdf0 2024-06-29T14:34:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.080+0000 7f405b69e640 1 -- 172.21.15.81:0/2313777506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4054072f40 con 0x7f405415bdf0 2024-06-29T14:34:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.081+0000 7f4049ffb640 1 -- 172.21.15.81:0/2313777506 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f404c019020 con 0x7f405415bdf0 2024-06-29T14:34:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.082+0000 7f4049ffb640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4038076480 0x7f4038078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.082+0000 7f405943a640 1 -- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4038076480 msgr2=0x7f4038078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:44.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.082+0000 7f4049ffb640 1 -- 172.21.15.81:0/2313777506 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6743+0+0 (secure 0 0 0) 0x7f404c098520 con 0x7f405415bdf0 2024-06-29T14:34:44.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.082+0000 7f405943a640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4038076480 0x7f4038078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:44.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.086+0000 7f4049ffb640 1 -- 172.21.15.81:0/2313777506 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f404c020020 con 0x7f405415bdf0 2024-06-29T14:34:44.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.270+0000 7f405b69e640 1 -- 172.21.15.81:0/2313777506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f405415b0e0 con 0x7f405415bdf0 2024-06-29T14:34:44.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.271+0000 7f4049ffb640 1 -- 172.21.15.81:0/2313777506 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) ==== 93+0+10 (secure 0 0 0) 0x7f4054157160 con 0x7f405415bdf0 2024-06-29T14:34:44.269 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:34:44.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 -- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4038076480 msgr2=0x7f4038078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4038076480 0x7f4038078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 -- 172.21.15.81:0/2313777506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f405415bdf0 msgr2=0x7f405406c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f405415bdf0 0x7f405406c830 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f404c00cb10 tx=0x7f404c00a060 comp rx=0 tx=0).stop 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 -- 172.21.15.81:0/2313777506 shutdown_connections 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f4038076480 0x7f4038078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f405415bdf0 0x7f405406c830 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40540cd150 0x7f405406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 --2- 172.21.15.81:0/2313777506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4054073480 0x7f4054145410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 -- 172.21.15.81:0/2313777506 >> 172.21.15.81:0/2313777506 conn(0x7f40540b2920 msgr2=0x7f405415de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 -- 172.21.15.81:0/2313777506 shutdown_connections 2024-06-29T14:34:44.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.273+0000 7f405b69e640 1 -- 172.21.15.81:0/2313777506 wait complete. 2024-06-29T14:34:44.283 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:34:44.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.401+0000 7f2ddbaef640 1 -- 172.21.15.81:0/74123056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dd415bdf0 msgr2=0x7f2dd406ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.401+0000 7f2ddbaef640 1 --2- 172.21.15.81:0/74123056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dd415bdf0 0x7f2dd406ed90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4072f10 tx=0x7f2dcc02aa30 comp rx=0 tx=0).stop 2024-06-29T14:34:44.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.402+0000 7f2ddbaef640 1 -- 172.21.15.81:0/74123056 shutdown_connections 2024-06-29T14:34:44.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.402+0000 7f2ddbaef640 1 --2- 172.21.15.81:0/74123056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dd415bdf0 0x7f2dd406ed90 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.402+0000 7f2ddbaef640 1 --2- 172.21.15.81:0/74123056 >> v2:172.21.15.167:3300/0 conn(0x7f2dd40cd150 0x7f2dd4072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.402+0000 7f2ddbaef640 1 -- 172.21.15.81:0/74123056 >> 172.21.15.81:0/74123056 conn(0x7f2dd40b2920 msgr2=0x7f2dd40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.402+0000 7f2ddbaef640 1 -- 172.21.15.81:0/74123056 shutdown_connections 2024-06-29T14:34:44.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.402+0000 7f2ddbaef640 1 -- 172.21.15.81:0/74123056 wait complete. 2024-06-29T14:34:44.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.405+0000 7f2ddbaef640 1 Processor -- start 2024-06-29T14:34:44.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.405+0000 7f2ddbaef640 1 -- start start 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2ddbaef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dd4073480 0x7f2dd40ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2ddbaef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2dd40cd150 0x7f2dd41413c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2ddbaef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dd415bdf0 0x7f2dd415e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2ddbaef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd406f450 con 0x7f2dd4073480 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2ddbaef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd406f2d0 con 0x7f2dd40cd150 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2ddbaef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2dd406f5d0 con 0x7f2dd415bdf0 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2dd988b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dd4073480 0x7f2dd40ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2dd988b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dd4073480 0x7f2dd40ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37758/0 (socket says 172.21.15.81:37758) 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2dd988b640 1 -- 172.21.15.81:0/214211169 learned_addr learned my addr 172.21.15.81:0/214211169 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2dda08c640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dd415bdf0 0x7f2dd415e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2dd908a640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2dd40cd150 0x7f2dd41413c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2dd988b640 1 -- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dd415bdf0 msgr2=0x7f2dd415e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2dd988b640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dd415bdf0 0x7f2dd415e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2dd988b640 1 -- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2dd40cd150 msgr2=0x7f2dd41413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2dd988b640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2dd40cd150 0x7f2dd41413c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.407+0000 7f2dd988b640 1 -- 172.21.15.81:0/214211169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd40ceb30 con 0x7f2dd4073480 2024-06-29T14:34:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.408+0000 7f2dd988b640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dd4073480 0x7f2dd40ce250 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000d770 tx=0x7f2dd000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.408+0000 7f2dca7fc640 1 -- 172.21.15.81:0/214211169 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0017070 con 0x7f2dd4073480 2024-06-29T14:34:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.408+0000 7f2dca7fc640 1 -- 172.21.15.81:0/214211169 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2dd0009d20 con 0x7f2dd4073480 2024-06-29T14:34:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.408+0000 7f2ddbaef640 1 -- 172.21.15.81:0/214211169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd40cee10 con 0x7f2dd4073480 2024-06-29T14:34:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.408+0000 7f2dca7fc640 1 -- 172.21.15.81:0/214211169 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd00106c0 con 0x7f2dd4073480 2024-06-29T14:34:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.408+0000 7f2ddbaef640 1 -- 172.21.15.81:0/214211169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd40cf3d0 con 0x7f2dd4073480 2024-06-29T14:34:44.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.409+0000 7f2ddbaef640 1 -- 172.21.15.81:0/214211169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd4072f40 con 0x7f2dd4073480 2024-06-29T14:34:44.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.410+0000 7f2dca7fc640 1 -- 172.21.15.81:0/214211169 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7f2dd001a070 con 0x7f2dd4073480 2024-06-29T14:34:44.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.410+0000 7f2dca7fc640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2db8076480 0x7f2db8078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.411+0000 7f2dd908a640 1 -- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2db8076480 msgr2=0x7f2db8078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:44.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.411+0000 7f2dca7fc640 1 -- 172.21.15.81:0/214211169 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6743+0+0 (secure 0 0 0) 0x7f2dd0097f00 con 0x7f2dd4073480 2024-06-29T14:34:44.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.411+0000 7f2dd908a640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2db8076480 0x7f2db8078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:44.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.415+0000 7f2dca7fc640 1 -- 172.21.15.81:0/214211169 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2dd0060b20 con 0x7f2dd4073480 2024-06-29T14:34:44.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.597+0000 7f2ddbaef640 1 -- 172.21.15.81:0/214211169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2dd415f560 con 0x7f2dd4073480 2024-06-29T14:34:44.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.597+0000 7f2dca7fc640 1 -- 172.21.15.81:0/214211169 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v47) ==== 104+0+11 (secure 0 0 0) 0x7f2dd0064e20 con 0x7f2dd4073480 2024-06-29T14:34:44.596 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:34:44.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 -- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2db8076480 msgr2=0x7f2db8078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:44.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2db8076480 0x7f2db8078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 -- 172.21.15.81:0/214211169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dd4073480 msgr2=0x7f2dd40ce250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dd4073480 0x7f2dd40ce250 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000d770 tx=0x7f2dd000dc30 comp rx=0 tx=0).stop 2024-06-29T14:34:44.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 -- 172.21.15.81:0/214211169 shutdown_connections 2024-06-29T14:34:44.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7f2db8076480 0x7f2db8078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dd415bdf0 0x7f2dd415e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2dd40cd150 0x7f2dd41413c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 --2- 172.21.15.81:0/214211169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dd4073480 0x7f2dd40ce250 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 -- 172.21.15.81:0/214211169 >> 172.21.15.81:0/214211169 conn(0x7f2dd40b2920 msgr2=0x7f2dd415ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 -- 172.21.15.81:0/214211169 shutdown_connections 2024-06-29T14:34:44.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.599+0000 7f2ddbaef640 1 -- 172.21.15.81:0/214211169 wait complete. 2024-06-29T14:34:44.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:34:44.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.728+0000 7fb7c3efe640 1 -- 172.21.15.81:0/3328572696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7bc0cd150 msgr2=0x7fb7bc072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.728+0000 7fb7c3efe640 1 --2- 172.21.15.81:0/3328572696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7bc0cd150 0x7fb7bc072f40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009830 tx=0x7fb7b802a480 comp rx=0 tx=0).stop 2024-06-29T14:34:44.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.728+0000 7fb7c3efe640 1 -- 172.21.15.81:0/3328572696 shutdown_connections 2024-06-29T14:34:44.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.728+0000 7fb7c3efe640 1 --2- 172.21.15.81:0/3328572696 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc073480 0x7fb7bc15b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.728+0000 7fb7c3efe640 1 --2- 172.21.15.81:0/3328572696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7bc0cd150 0x7fb7bc072f40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.728+0000 7fb7c3efe640 1 -- 172.21.15.81:0/3328572696 >> 172.21.15.81:0/3328572696 conn(0x7fb7bc0b2920 msgr2=0x7fb7bc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.728+0000 7fb7c3efe640 1 -- 172.21.15.81:0/3328572696 shutdown_connections 2024-06-29T14:34:44.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.728+0000 7fb7c3efe640 1 -- 172.21.15.81:0/3328572696 wait complete. 2024-06-29T14:34:44.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.730+0000 7fb7c3efe640 1 Processor -- start 2024-06-29T14:34:44.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.730+0000 7fb7c3efe640 1 -- start start 2024-06-29T14:34:44.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c3efe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7bc073480 0x7fb7bc1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c3efe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7bc0cd150 0x7fb7bc147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c3efe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc15bc20 0x7fb7bc0d0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c3efe640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc15c1b0 con 0x7fb7bc0cd150 2024-06-29T14:34:44.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c3efe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb7bc15c030 con 0x7fb7bc073480 2024-06-29T14:34:44.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c3efe640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb7bc15f060 con 0x7fb7bc15bc20 2024-06-29T14:34:44.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c1499640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7bc0cd150 0x7fb7bc147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c1499640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7bc0cd150 0x7fb7bc147550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37762/0 (socket says 172.21.15.81:37762) 2024-06-29T14:34:44.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c1499640 1 -- 172.21.15.81:0/675930889 learned_addr learned my addr 172.21.15.81:0/675930889 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:44.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c1c9a640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7bc073480 0x7fb7bc1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c249b640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc15bc20 0x7fb7bc0d0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c1499640 1 -- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc15bc20 msgr2=0x7fb7bc0d0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.731+0000 7fb7c1499640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc15bc20 0x7fb7bc0d0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.732+0000 7fb7c1499640 1 -- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7bc073480 msgr2=0x7fb7bc1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.732+0000 7fb7c1499640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7bc073480 0x7fb7bc1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.732+0000 7fb7c1499640 1 -- 172.21.15.81:0/675930889 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7bc0ce380 con 0x7fb7bc0cd150 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.732+0000 7fb7c249b640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc15bc20 0x7fb7bc0d0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.732+0000 7fb7c1499640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7bc0cd150 0x7fb7bc147550 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac00e940 tx=0x7fb7ac00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.732+0000 7fb7b27fc640 1 -- 172.21.15.81:0/675930889 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac00cd20 con 0x7fb7bc0cd150 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.732+0000 7fb7b27fc640 1 -- 172.21.15.81:0/675930889 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7ac010430 con 0x7fb7bc0cd150 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.732+0000 7fb7c3efe640 1 -- 172.21.15.81:0/675930889 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7bc0ce660 con 0x7fb7bc0cd150 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.732+0000 7fb7b27fc640 1 -- 172.21.15.81:0/675930889 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac010880 con 0x7fb7bc0cd150 2024-06-29T14:34:44.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.732+0000 7fb7c3efe640 1 -- 172.21.15.81:0/675930889 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7bc148230 con 0x7fb7bc0cd150 2024-06-29T14:34:44.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.733+0000 7fb7c3efe640 1 -- 172.21.15.81:0/675930889 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7bc15f580 con 0x7fb7bc0cd150 2024-06-29T14:34:44.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.734+0000 7fb7b27fc640 1 -- 172.21.15.81:0/675930889 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 14) ==== 98372+0+0 (secure 0 0 0) 0x7fb7ac010a20 con 0x7fb7bc0cd150 2024-06-29T14:34:44.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.735+0000 7fb7b27fc640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb7a00764b0 0x7fb7a00789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:44.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.735+0000 7fb7c1c9a640 1 -- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb7a00764b0 msgr2=0x7fb7a00789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/189854401 2024-06-29T14:34:44.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.735+0000 7fb7b27fc640 1 -- 172.21.15.81:0/675930889 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6743+0+0 (secure 0 0 0) 0x7fb7ac0989b0 con 0x7fb7bc0cd150 2024-06-29T14:34:44.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.735+0000 7fb7c1c9a640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb7a00764b0 0x7fb7a00789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:34:44.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.739+0000 7fb7b27fc640 1 -- 172.21.15.81:0/675930889 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb7ac054af0 con 0x7fb7bc0cd150 2024-06-29T14:34:44.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.833+0000 7fb7b27fc640 1 -- 172.21.15.81:0/675930889 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 15) ==== 97912+0+0 (secure 0 0 0) 0x7fb7ac010d10 con 0x7fb7bc0cd150 2024-06-29T14:34:44.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.833+0000 7fb7b27fc640 1 -- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb7a00764b0 msgr2=0x7fb7a00789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:34:44.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.833+0000 7fb7b27fc640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb7a00764b0 0x7fb7a00789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.884 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:34:44.886+0000 7f246257c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:44.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:34:44.886+0000 7f246257c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:44.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:34:44.886+0000 7f246257c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:44.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:34:44.887+0000 7f246257c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:44.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:34:44.887+0000 7f246257c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:44.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:34:44.887+0000 7f2465582640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:44.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:34:44.887+0000 7f2465582640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:44.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:34:44.887+0000 7f2465582640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:44.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:34:44.887+0000 7f2465582640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:44.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:34:44.887+0000 7f2465582640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:34:44.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.930+0000 7fb7c3efe640 1 -- 172.21.15.81:0/675930889 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb7bc154f00 con 0x7fb7bc0cd150 2024-06-29T14:34:44.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.930+0000 7fb7b27fc640 1 -- 172.21.15.81:0/675930889 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7fb7ac061ad0 con 0x7fb7bc0cd150 2024-06-29T14:34:44.929 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:34:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.932+0000 7fb7c3efe640 1 -- 172.21.15.81:0/675930889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7bc0cd150 msgr2=0x7fb7bc147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.932+0000 7fb7c3efe640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7bc0cd150 0x7fb7bc147550 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac00e940 tx=0x7fb7ac00ee00 comp rx=0 tx=0).stop 2024-06-29T14:34:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.932+0000 7fb7c3efe640 1 -- 172.21.15.81:0/675930889 shutdown_connections 2024-06-29T14:34:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.932+0000 7fb7c3efe640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:6832/189854401,v1:172.21.15.167:6833/189854401] conn(0x7fb7a00764b0 0x7fb7a00789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.932+0000 7fb7c3efe640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc15bc20 0x7fb7bc0d0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.932+0000 7fb7c3efe640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7bc0cd150 0x7fb7bc147550 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.932+0000 7fb7c3efe640 1 --2- 172.21.15.81:0/675930889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7bc073480 0x7fb7bc1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.932+0000 7fb7c3efe640 1 -- 172.21.15.81:0/675930889 >> 172.21.15.81:0/675930889 conn(0x7fb7bc0b2920 msgr2=0x7fb7bc070ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.932+0000 7fb7c3efe640 1 -- 172.21.15.81:0/675930889 shutdown_connections 2024-06-29T14:34:44.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:44.932+0000 7fb7c3efe640 1 -- 172.21.15.81:0/675930889 wait complete. 2024-06-29T14:34:44.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:34:45.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.060+0000 7fe686417640 1 -- 172.21.15.81:0/3366976054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68015d750 msgr2=0x7fe6800afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.060+0000 7fe686417640 1 --2- 172.21.15.81:0/3366976054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68015d750 0x7fe6800afb60 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe670009830 tx=0x7fe67002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.060+0000 7fe686417640 1 -- 172.21.15.81:0/3366976054 shutdown_connections 2024-06-29T14:34:45.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.060+0000 7fe686417640 1 --2- 172.21.15.81:0/3366976054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68015d750 0x7fe6800afb60 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.060+0000 7fe686417640 1 -- 172.21.15.81:0/3366976054 >> 172.21.15.81:0/3366976054 conn(0x7fe6800d0930 msgr2=0x7fe6800d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.060+0000 7fe686417640 1 -- 172.21.15.81:0/3366976054 shutdown_connections 2024-06-29T14:34:45.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.060+0000 7fe686417640 1 -- 172.21.15.81:0/3366976054 wait complete. 2024-06-29T14:34:45.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.062+0000 7fe686417640 1 Processor -- start 2024-06-29T14:34:45.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.062+0000 7fe686417640 1 -- start start 2024-06-29T14:34:45.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe686417640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68015c030 0x7fe68014fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe686417640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe68015d750 0x7fe680157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe686417640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe680150230 0x7fe680158190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe686417640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6800d0030 con 0x7fe68015c030 2024-06-29T14:34:45.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe686417640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe6800cfeb0 con 0x7fe68015d750 2024-06-29T14:34:45.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe686417640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe6800d01b0 con 0x7fe680150230 2024-06-29T14:34:45.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe67ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68015c030 0x7fe68014fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe67ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68015c030 0x7fe68014fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37776/0 (socket says 172.21.15.81:37776) 2024-06-29T14:34:45.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe67ffff640 1 -- 172.21.15.81:0/3417586543 learned_addr learned my addr 172.21.15.81:0/3417586543 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:45.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe6849b4640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe680150230 0x7fe680158190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe67f7fe640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe68015d750 0x7fe680157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe67ffff640 1 -- 172.21.15.81:0/3417586543 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe680150230 msgr2=0x7fe680158190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.063+0000 7fe67ffff640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe680150230 0x7fe680158190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe67ffff640 1 -- 172.21.15.81:0/3417586543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe68015d750 msgr2=0x7fe680157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe67ffff640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe68015d750 0x7fe680157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe67ffff640 1 -- 172.21.15.81:0/3417586543 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe680158b20 con 0x7fe68015c030 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe6849b4640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe680150230 0x7fe680158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe67f7fe640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe68015d750 0x7fe680157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe67ffff640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68015c030 0x7fe68014fb90 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe670000c00 tx=0x7fe670033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe67cff9640 1 -- 172.21.15.81:0/3417586543 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670031d60 con 0x7fe68015c030 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe67cff9640 1 -- 172.21.15.81:0/3417586543 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6700330d0 con 0x7fe68015c030 2024-06-29T14:34:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe686417640 1 -- 172.21.15.81:0/3417586543 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe680158e00 con 0x7fe68015c030 2024-06-29T14:34:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe67cff9640 1 -- 172.21.15.81:0/3417586543 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670033520 con 0x7fe68015c030 2024-06-29T14:34:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.064+0000 7fe686417640 1 -- 172.21.15.81:0/3417586543 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6801593c0 con 0x7fe68015c030 2024-06-29T14:34:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.065+0000 7fe686417640 1 -- 172.21.15.81:0/3417586543 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68006e2a0 con 0x7fe68015c030 2024-06-29T14:34:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.066+0000 7fe67cff9640 1 -- 172.21.15.81:0/3417586543 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 15) ==== 97912+0+0 (secure 0 0 0) 0x7fe6700035f0 con 0x7fe68015c030 2024-06-29T14:34:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.066+0000 7fe67cff9640 1 -- 172.21.15.81:0/3417586543 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 7070+0+0 (secure 0 0 0) 0x7fe6700b8bf0 con 0x7fe68015c030 2024-06-29T14:34:45.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.071+0000 7fe67cff9640 1 -- 172.21.15.81:0/3417586543 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe68006e2a0 con 0x7fe68015c030 2024-06-29T14:34:45.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.287+0000 7fe686417640 1 -- 172.21.15.81:0/3417586543 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe68015b950 con 0x7fe68015c030 2024-06-29T14:34:45.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.288+0000 7fe67cff9640 1 -- 172.21.15.81:0/3417586543 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5220 (secure 0 0 0) 0x7fe67003f030 con 0x7fe68015c030 2024-06-29T14:34:45.287 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:45.287 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"btime":"2024-06-29T14:23:40:952859+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3513859413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3513859413},{"type":"v1","addr":"172.21.15.81:6835","nonce":3513859413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4775,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/763942647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":763942647},{"type":"v1","addr":"172.21.15.167:6836","nonce":763942647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1148290480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1148290480},{"type":"v1","addr":"172.21.15.167:6837","nonce":1148290480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:37.343197+0000","modified":"2024-06-29T14:23:40.325922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5172":{"gid":5172,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/58595811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":58595811},{"type":"v1","addr":"172.21.15.81:6834","nonce":58595811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5172,"qdb_cluster":[5172]},"id":2}]} 2024-06-29T14:34:45.287 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 16 2024-06-29T14:34:45.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.290+0000 7fe686417640 1 -- 172.21.15.81:0/3417586543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68015c030 msgr2=0x7fe68014fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.290+0000 7fe686417640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68015c030 0x7fe68014fb90 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe670000c00 tx=0x7fe670033d60 comp rx=0 tx=0).stop 2024-06-29T14:34:45.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.290+0000 7fe686417640 1 -- 172.21.15.81:0/3417586543 shutdown_connections 2024-06-29T14:34:45.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.290+0000 7fe686417640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe680150230 0x7fe680158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.290+0000 7fe686417640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe68015d750 0x7fe680157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.290+0000 7fe686417640 1 --2- 172.21.15.81:0/3417586543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68015c030 0x7fe68014fb90 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.290+0000 7fe686417640 1 -- 172.21.15.81:0/3417586543 >> 172.21.15.81:0/3417586543 conn(0x7fe6800d0930 msgr2=0x7fe680152490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.290+0000 7fe686417640 1 -- 172.21.15.81:0/3417586543 shutdown_connections 2024-06-29T14:34:45.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.290+0000 7fe686417640 1 -- 172.21.15.81:0/3417586543 wait complete. 2024-06-29T14:34:45.300 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:45.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.415+0000 7f2bb8df0640 1 -- 172.21.15.81:0/56667737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ba0004800 msgr2=0x7f2ba0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.415+0000 7f2bb8df0640 1 --2- 172.21.15.81:0/56667737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ba0004800 0x7f2ba0004c10 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c0017d0 tx=0x7f2ba0005830 comp rx=0 tx=0).stop 2024-06-29T14:34:45.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.415+0000 7f2bb2d9d640 1 -- 172.21.15.81:0/56667737 reap_dead start 2024-06-29T14:34:45.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.415+0000 7f2bb8df0640 1 -- 172.21.15.81:0/56667737 shutdown_connections 2024-06-29T14:34:45.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.415+0000 7f2bb8df0640 1 -- 172.21.15.81:0/56667737 >> 172.21.15.81:0/56667737 conn(0x7f2bb40d1960 msgr2=0x7f2bb40d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.415+0000 7f2bb8df0640 1 -- 172.21.15.81:0/56667737 shutdown_connections 2024-06-29T14:34:45.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.415+0000 7f2bb8df0640 1 -- 172.21.15.81:0/56667737 wait complete. 2024-06-29T14:34:45.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.417+0000 7f2bb8df0640 1 Processor -- start 2024-06-29T14:34:45.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.417+0000 7f2bb8df0640 1 -- start start 2024-06-29T14:34:45.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bb8df0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ba0004800 0x7f2bb41d7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bb8df0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bb40cb690 0x7f2bb40c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bb8df0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2bb40cbaa0 0x7f2bb40c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bb8df0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb40b5020 con 0x7f2bb40cb690 2024-06-29T14:34:45.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bb8df0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb40b2790 con 0x7f2ba0004800 2024-06-29T14:34:45.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bb8df0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2bb40d0500 con 0x7f2bb40cbaa0 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2ba9d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bb40cb690 0x7f2bb40c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2ba9d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bb40cb690 0x7f2bb40c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37802/0 (socket says 172.21.15.81:37802) 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2ba9d9b640 1 -- 172.21.15.81:0/301327341 learned_addr learned my addr 172.21.15.81:0/301327341 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bb2d9d640 1 --2- 172.21.15.81:0/301327341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2bb40cbaa0 0x7f2bb40c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bb259c640 1 --2- 172.21.15.81:0/301327341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ba0004800 0x7f2bb41d7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2ba9d9b640 1 -- 172.21.15.81:0/301327341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2bb40cbaa0 msgr2=0x7f2bb40c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2ba9d9b640 1 --2- 172.21.15.81:0/301327341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2bb40cbaa0 0x7f2bb40c6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2ba9d9b640 1 -- 172.21.15.81:0/301327341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ba0004800 msgr2=0x7f2bb41d7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2ba9d9b640 1 --2- 172.21.15.81:0/301327341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ba0004800 0x7f2bb41d7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2ba9d9b640 1 -- 172.21.15.81:0/301327341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb41f2dd0 con 0x7f2bb40cb690 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2ba9d9b640 1 --2- 172.21.15.81:0/301327341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bb40cb690 0x7f2bb40c6920 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c000f60 tx=0x7f2b9c006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bab7fe640 1 -- 172.21.15.81:0/301327341 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c043070 con 0x7f2bb40cb690 2024-06-29T14:34:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bab7fe640 1 -- 172.21.15.81:0/301327341 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b9c0038e0 con 0x7f2bb40cb690 2024-06-29T14:34:45.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.419+0000 7f2bb8df0640 1 -- 172.21.15.81:0/301327341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb41f63c0 con 0x7f2bb40cb690 2024-06-29T14:34:45.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.420+0000 7f2bab7fe640 1 -- 172.21.15.81:0/301327341 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c003d30 con 0x7f2bb40cb690 2024-06-29T14:34:45.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.420+0000 7f2bb8df0640 1 -- 172.21.15.81:0/301327341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb41f6ba0 con 0x7f2bb40cb690 2024-06-29T14:34:45.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.421+0000 7f2bb8df0640 1 -- 172.21.15.81:0/301327341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bb41f6630 con 0x7f2bb40cb690 2024-06-29T14:34:45.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.421+0000 7f2bab7fe640 1 -- 172.21.15.81:0/301327341 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 15) ==== 97912+0+0 (secure 0 0 0) 0x7f2b9c0303c0 con 0x7f2bb40cb690 2024-06-29T14:34:45.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.422+0000 7f2bab7fe640 1 -- 172.21.15.81:0/301327341 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 7070+0+0 (secure 0 0 0) 0x7f2b9c0b7530 con 0x7f2bb40cb690 2024-06-29T14:34:45.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.426+0000 7f2bab7fe640 1 -- 172.21.15.81:0/301327341 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2bb41f2dd0 con 0x7f2bb40cb690 2024-06-29T14:34:45.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.606+0000 7f2bb8df0640 1 -- 172.21.15.81:0/301327341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bb40ce4d0 con 0x7f2bb40cb690 2024-06-29T14:34:45.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.607+0000 7f2bab7fe640 1 -- 172.21.15.81:0/301327341 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+18714 (secure 0 0 0) 0x7f2b9c0806b0 con 0x7f2bb40cb690 2024-06-29T14:34:45.606 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:45.607 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:34:44.823772+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:34.937915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:36.258024+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":220,"snap_epoch":47,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2999883788":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/3190548120":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/2974208992":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/3819768847":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6832/3101542440":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6833/3101542440":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/3134616648":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6833/189854401":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/1701819343":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/3793734276":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/154739969":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/2258575888":"2024-06-30T14:34:44.823719+0000","172.21.15.167:6832/189854401":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/1556709624":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/3972870238":"2024-06-30T14:34:44.823719+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":9,"length":1},{"begin":11,"length":2},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":20,"length":1},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":27,"length":1},{"begin":31,"length":2},{"begin":34,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":43,"length":1},{"begin":45,"length":1},{"begin":47,"length":1},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":55,"length":1},{"begin":57,"length":1},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":65,"length":1},{"begin":67,"length":1},{"begin":69,"length":1},{"begin":74,"length":1},{"begin":76,"length":1},{"begin":78,"length":1},{"begin":80,"length":1},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":87,"length":1},{"begin":89,"length":1},{"begin":91,"length":2},{"begin":94,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":105,"length":1},{"begin":107,"length":1},{"begin":109,"length":1},{"begin":175,"length":46}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:45.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.609+0000 7f2bb8df0640 1 -- 172.21.15.81:0/301327341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bb40cb690 msgr2=0x7f2bb40c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.609+0000 7f2bb8df0640 1 --2- 172.21.15.81:0/301327341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bb40cb690 0x7f2bb40c6920 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c000f60 tx=0x7f2b9c006940 comp rx=0 tx=0).stop 2024-06-29T14:34:45.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.609+0000 7f2bb8df0640 1 -- 172.21.15.81:0/301327341 shutdown_connections 2024-06-29T14:34:45.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.609+0000 7f2bb8df0640 1 --2- 172.21.15.81:0/301327341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2bb40cbaa0 0x7f2bb40c6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.609+0000 7f2bb8df0640 1 --2- 172.21.15.81:0/301327341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bb40cb690 0x7f2bb40c6920 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.609+0000 7f2bb8df0640 1 --2- 172.21.15.81:0/301327341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ba0004800 0x7f2bb41d7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.609+0000 7f2bb8df0640 1 -- 172.21.15.81:0/301327341 >> 172.21.15.81:0/301327341 conn(0x7f2bb40d1960 msgr2=0x7f2bb40b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.609+0000 7f2bb8df0640 1 -- 172.21.15.81:0/301327341 shutdown_connections 2024-06-29T14:34:45.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.609+0000 7f2bb8df0640 1 -- 172.21.15.81:0/301327341 wait complete. 2024-06-29T14:34:45.619 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:34:45.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:34:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.734+0000 7f44e0743640 1 -- 172.21.15.81:0/357287709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44d80cd480 msgr2=0x7f44d80ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.734+0000 7f44e0743640 1 --2- 172.21.15.81:0/357287709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44d80cd480 0x7f44d80ce510 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f44c8005fd0 tx=0x7f44c802a540 comp rx=0 tx=0).stop 2024-06-29T14:34:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.734+0000 7f44e0743640 1 -- 172.21.15.81:0/357287709 shutdown_connections 2024-06-29T14:34:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.734+0000 7f44e0743640 1 --2- 172.21.15.81:0/357287709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44d80cd480 0x7f44d80ce510 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.734+0000 7f44e0743640 1 -- 172.21.15.81:0/357287709 >> 172.21.15.81:0/357287709 conn(0x7f44d80b3510 msgr2=0x7f44d80b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.734+0000 7f44e0743640 1 -- 172.21.15.81:0/357287709 shutdown_connections 2024-06-29T14:34:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.734+0000 7f44e0743640 1 -- 172.21.15.81:0/357287709 wait complete. 2024-06-29T14:34:45.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.735+0000 7f44e0743640 1 Processor -- start 2024-06-29T14:34:45.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.736+0000 7f44e0743640 1 -- start start 2024-06-29T14:34:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44e0743640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44d80b6eb0 0x7f44d80b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44e0743640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44d80cd480 0x7f44d8153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44e0743640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44d80cea50 0x7f44d8165790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44e0743640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f44d80b52b0 con 0x7f44d80b6eb0 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44e0743640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f44d80b5130 con 0x7f44d80cd480 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44e0743640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f44d8167660 con 0x7f44d80cea50 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44de4df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44d80b6eb0 0x7f44d80b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44de4df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44d80b6eb0 0x7f44d80b14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37812/0 (socket says 172.21.15.81:37812) 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44de4df640 1 -- 172.21.15.81:0/3644803017 learned_addr learned my addr 172.21.15.81:0/3644803017 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44ddcde640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44d80cd480 0x7f44d8153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44dece0640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44d80cea50 0x7f44d8165790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44de4df640 1 -- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44d80cea50 msgr2=0x7f44d8165790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44de4df640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44d80cea50 0x7f44d8165790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44de4df640 1 -- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44d80cd480 msgr2=0x7f44d8153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44de4df640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44d80cd480 0x7f44d8153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44de4df640 1 -- 172.21.15.81:0/3644803017 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d8164f40 con 0x7f44d80b6eb0 2024-06-29T14:34:45.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.737+0000 7f44de4df640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44d80b6eb0 0x7f44d80b14a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f44d400b3f0 tx=0x7f44d400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:45.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.738+0000 7f44ceffd640 1 -- 172.21.15.81:0/3644803017 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4015020 con 0x7f44d80b6eb0 2024-06-29T14:34:45.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.738+0000 7f44ceffd640 1 -- 172.21.15.81:0/3644803017 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44d4011920 con 0x7f44d80b6eb0 2024-06-29T14:34:45.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.738+0000 7f44ceffd640 1 -- 172.21.15.81:0/3644803017 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4011d70 con 0x7f44d80b6eb0 2024-06-29T14:34:45.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.738+0000 7f44e0743640 1 -- 172.21.15.81:0/3644803017 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d8165220 con 0x7f44d80b6eb0 2024-06-29T14:34:45.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.738+0000 7f44e0743640 1 -- 172.21.15.81:0/3644803017 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d8154720 con 0x7f44d80b6eb0 2024-06-29T14:34:45.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.739+0000 7f44e0743640 1 -- 172.21.15.81:0/3644803017 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44d80ce510 con 0x7f44d80b6eb0 2024-06-29T14:34:45.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.739+0000 7f44ceffd640 1 -- 172.21.15.81:0/3644803017 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 15) ==== 97912+0+0 (secure 0 0 0) 0x7f44d4018020 con 0x7f44d80b6eb0 2024-06-29T14:34:45.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.740+0000 7f44ceffd640 1 -- 172.21.15.81:0/3644803017 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 7070+0+0 (secure 0 0 0) 0x7f44d40973c0 con 0x7f44d80b6eb0 2024-06-29T14:34:45.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.744+0000 7f44ceffd640 1 -- 172.21.15.81:0/3644803017 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f44d401f030 con 0x7f44d80b6eb0 2024-06-29T14:34:45.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.843+0000 7f44ceffd640 1 -- 172.21.15.81:0/3644803017 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 16) ==== 98039+0+0 (secure 0 0 0) 0x7f44d405beb0 con 0x7f44d80b6eb0 2024-06-29T14:34:45.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.843+0000 7f44ceffd640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f44bc07c2d0 0x7f44bc07e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:45.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.848+0000 7f44ddcde640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f44bc07c2d0 0x7f44bc07e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:45.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.850+0000 7f44ddcde640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f44bc07c2d0 0x7f44bc07e6f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f44c8033000 tx=0x7f44c8004730 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:45.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.972+0000 7f44e0743640 1 -- 172.21.15.81:0/3644803017 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44d80cdf00 con 0x7f44d80b6eb0 2024-06-29T14:34:45.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.973+0000 7f44ceffd640 1 -- 172.21.15.81:0/3644803017 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5220 (secure 0 0 0) 0x7f44d4011d50 con 0x7f44d80b6eb0 2024-06-29T14:34:45.971 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:45.972 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"btime":"2024-06-29T14:23:40:952859+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3513859413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3513859413},{"type":"v1","addr":"172.21.15.81:6835","nonce":3513859413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4775,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/763942647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":763942647},{"type":"v1","addr":"172.21.15.167:6836","nonce":763942647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1148290480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1148290480},{"type":"v1","addr":"172.21.15.167:6837","nonce":1148290480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:37.343197+0000","modified":"2024-06-29T14:23:40.325922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5172":{"gid":5172,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/58595811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":58595811},{"type":"v1","addr":"172.21.15.81:6834","nonce":58595811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5172,"qdb_cluster":[5172]},"id":2}]} 2024-06-29T14:34:45.972 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 16 2024-06-29T14:34:45.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 -- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f44bc07c2d0 msgr2=0x7f44bc07e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f44bc07c2d0 0x7f44bc07e6f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f44c8033000 tx=0x7f44c8004730 comp rx=0 tx=0).stop 2024-06-29T14:34:45.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 -- 172.21.15.81:0/3644803017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44d80b6eb0 msgr2=0x7f44d80b14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:45.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44d80b6eb0 0x7f44d80b14a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f44d400b3f0 tx=0x7f44d400b8b0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 -- 172.21.15.81:0/3644803017 shutdown_connections 2024-06-29T14:34:45.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44d80cea50 0x7f44d8165790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44d80cd480 0x7f44d8153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f44bc07c2d0 0x7f44bc07e6f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 --2- 172.21.15.81:0/3644803017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44d80b6eb0 0x7f44d80b14a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:45.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 -- 172.21.15.81:0/3644803017 >> 172.21.15.81:0/3644803017 conn(0x7f44d80b3510 msgr2=0x7f44d8163860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:45.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 -- 172.21.15.81:0/3644803017 shutdown_connections 2024-06-29T14:34:45.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:45.975+0000 7f44e0743640 1 -- 172.21.15.81:0/3644803017 wait complete. 2024-06-29T14:34:45.985 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:34:46.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.100+0000 7f64e70df640 1 -- 172.21.15.81:0/1712232360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64c8004800 msgr2=0x7f64c8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.100+0000 7f64e70df640 1 --2- 172.21.15.81:0/1712232360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64c8004800 0x7f64c8004c10 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f64d40016c0 tx=0x7f64c8005830 comp rx=0 tx=0).stop 2024-06-29T14:34:46.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.100+0000 7f64e70df640 1 -- 172.21.15.81:0/1712232360 shutdown_connections 2024-06-29T14:34:46.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.100+0000 7f64e70df640 1 --2- 172.21.15.81:0/1712232360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64c8004800 0x7f64c8004c10 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.100+0000 7f64e70df640 1 -- 172.21.15.81:0/1712232360 >> 172.21.15.81:0/1712232360 conn(0x7f64e00d0930 msgr2=0x7f64e00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.100+0000 7f64e70df640 1 -- 172.21.15.81:0/1712232360 shutdown_connections 2024-06-29T14:34:46.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.100+0000 7f64e70df640 1 -- 172.21.15.81:0/1712232360 wait complete. 2024-06-29T14:34:46.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.102+0000 7f64e70df640 1 Processor -- start 2024-06-29T14:34:46.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.102+0000 7f64e70df640 1 -- start start 2024-06-29T14:34:46.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e70df640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64c8004800 0x7f64e01e4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e70df640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64e00aed10 0x7f64e00af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e70df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e00b3a60 0x7f64e00b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e70df640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64e00affa0 con 0x7f64e00b3a60 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e70df640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f64e00afe20 con 0x7f64e00aed10 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e70df640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f64e00b0120 con 0x7f64c8004800 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e567c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e00b3a60 0x7f64e00b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e567c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e00b3a60 0x7f64e00b3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37828/0 (socket says 172.21.15.81:37828) 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e567c640 1 -- 172.21.15.81:0/103756484 learned_addr learned my addr 172.21.15.81:0/103756484 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64dffff640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64e00aed10 0x7f64e00af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e4e7b640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64c8004800 0x7f64e01e4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e567c640 1 -- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64c8004800 msgr2=0x7f64e01e4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e567c640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64c8004800 0x7f64e01e4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e567c640 1 -- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64e00aed10 msgr2=0x7f64e00af120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e567c640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64e00aed10 0x7f64e00af120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e567c640 1 -- 172.21.15.81:0/103756484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e01e70b0 con 0x7f64e00b3a60 2024-06-29T14:34:46.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.104+0000 7f64e567c640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e00b3a60 0x7f64e00b3e70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f64e00b36f0 tx=0x7f64d8006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:46.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.105+0000 7f64dd7fa640 1 -- 172.21.15.81:0/103756484 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d800fa20 con 0x7f64e00b3a60 2024-06-29T14:34:46.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.105+0000 7f64dd7fa640 1 -- 172.21.15.81:0/103756484 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64d800fbe0 con 0x7f64e00b3a60 2024-06-29T14:34:46.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.105+0000 7f64dd7fa640 1 -- 172.21.15.81:0/103756484 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d8011430 con 0x7f64e00b3a60 2024-06-29T14:34:46.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.105+0000 7f64e70df640 1 -- 172.21.15.81:0/103756484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64e00adc70 con 0x7f64e00b3a60 2024-06-29T14:34:46.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.105+0000 7f64e70df640 1 -- 172.21.15.81:0/103756484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64e00adf50 con 0x7f64e00b3a60 2024-06-29T14:34:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.106+0000 7f64e70df640 1 -- 172.21.15.81:0/103756484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64e006e2a0 con 0x7f64e00b3a60 2024-06-29T14:34:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.106+0000 7f64dd7fa640 1 -- 172.21.15.81:0/103756484 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 98039+0+0 (secure 0 0 0) 0x7f64d80115f0 con 0x7f64e00b3a60 2024-06-29T14:34:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.107+0000 7f64dd7fa640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f64cc076120 0x7f64cc078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.107+0000 7f64dd7fa640 1 -- 172.21.15.81:0/103756484 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 7130+0+0 (secure 0 0 0) 0x7f64d80982e0 con 0x7f64e00b3a60 2024-06-29T14:34:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.107+0000 7f64e4e7b640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f64cc076120 0x7f64cc078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.108+0000 7f64e4e7b640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f64cc076120 0x7f64cc078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f64e006adf0 tx=0x7f64d0005f90 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:46.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.111+0000 7f64dd7fa640 1 -- 172.21.15.81:0/103756484 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f64e006e2a0 con 0x7f64e00b3a60 2024-06-29T14:34:46.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.331+0000 7f64e70df640 1 -- 172.21.15.81:0/103756484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64e006c570 con 0x7f64e00b3a60 2024-06-29T14:34:46.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.332+0000 7f64dd7fa640 1 -- 172.21.15.81:0/103756484 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5220 (secure 0 0 0) 0x7f64e01e70b0 con 0x7f64e00b3a60 2024-06-29T14:34:46.331 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:46.331 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"btime":"2024-06-29T14:23:40:952859+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3513859413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3513859413},{"type":"v1","addr":"172.21.15.81:6835","nonce":3513859413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4775,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/763942647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":763942647},{"type":"v1","addr":"172.21.15.167:6836","nonce":763942647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1148290480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1148290480},{"type":"v1","addr":"172.21.15.167:6837","nonce":1148290480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:23:37.343197+0000","modified":"2024-06-29T14:23:40.325922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5172},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5172":{"gid":5172,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/58595811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":58595811},{"type":"v1","addr":"172.21.15.81:6834","nonce":58595811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5172,"qdb_cluster":[5172]},"id":2}]} 2024-06-29T14:34:46.331 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 16 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 -- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f64cc076120 msgr2=0x7f64cc078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f64cc076120 0x7f64cc078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f64e006adf0 tx=0x7f64d0005f90 comp rx=0 tx=0).stop 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 -- 172.21.15.81:0/103756484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e00b3a60 msgr2=0x7f64e00b3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e00b3a60 0x7f64e00b3e70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f64e00b36f0 tx=0x7f64d8006660 comp rx=0 tx=0).stop 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 -- 172.21.15.81:0/103756484 shutdown_connections 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f64cc076120 0x7f64cc078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e00b3a60 0x7f64e00b3e70 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64e00aed10 0x7f64e00af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 --2- 172.21.15.81:0/103756484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64c8004800 0x7f64e01e4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 -- 172.21.15.81:0/103756484 >> 172.21.15.81:0/103756484 conn(0x7f64e00d0930 msgr2=0x7f64e015db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 -- 172.21.15.81:0/103756484 shutdown_connections 2024-06-29T14:34:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.334+0000 7f64e70df640 1 -- 172.21.15.81:0/103756484 wait complete. 2024-06-29T14:34:46.343 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:46.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.457+0000 7f1b79112640 1 -- 172.21.15.81:0/2495731250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b740b35b0 msgr2=0x7f1b7415b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.457+0000 7f1b79112640 1 --2- 172.21.15.81:0/2495731250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b740b35b0 0x7f1b7415b5b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1b740cc260 tx=0x7f1b6402a560 comp rx=0 tx=0).stop 2024-06-29T14:34:46.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.457+0000 7f1b79112640 1 -- 172.21.15.81:0/2495731250 shutdown_connections 2024-06-29T14:34:46.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.457+0000 7f1b79112640 1 --2- 172.21.15.81:0/2495731250 >> v2:172.21.15.167:3300/0 conn(0x7f1b740b4430 0x7f1b740cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.457+0000 7f1b79112640 1 --2- 172.21.15.81:0/2495731250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b740b35b0 0x7f1b7415b5b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.457+0000 7f1b79112640 1 -- 172.21.15.81:0/2495731250 >> 172.21.15.81:0/2495731250 conn(0x7f1b7406ccc0 msgr2=0x7f1b740b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.457+0000 7f1b79112640 1 -- 172.21.15.81:0/2495731250 shutdown_connections 2024-06-29T14:34:46.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.457+0000 7f1b79112640 1 -- 172.21.15.81:0/2495731250 wait complete. 2024-06-29T14:34:46.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.459+0000 7f1b79112640 1 Processor -- start 2024-06-29T14:34:46.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.459+0000 7f1b79112640 1 -- start start 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b79112640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b7406f660 0x7f1b74159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b79112640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b740b35b0 0x7f1b741465f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b79112640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b740b4430 0x7f1b74157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b79112640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7406f1e0 con 0x7f1b740b4430 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b79112640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7406f060 con 0x7f1b740b35b0 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b79112640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1b7406f360 con 0x7f1b7406f660 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b7359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b740b4430 0x7f1b74157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b7359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b740b4430 0x7f1b74157810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37848/0 (socket says 172.21.15.81:37848) 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b7359e640 1 -- 172.21.15.81:0/3407745031 learned_addr learned my addr 172.21.15.81:0/3407745031 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b72d9d640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b7406f660 0x7f1b74159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b7259c640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b740b35b0 0x7f1b741465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b7359e640 1 -- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b7406f660 msgr2=0x7f1b74159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b7359e640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b7406f660 0x7f1b74159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b7359e640 1 -- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b740b35b0 msgr2=0x7f1b741465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b7359e640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b740b35b0 0x7f1b741465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.460+0000 7f1b7359e640 1 -- 172.21.15.81:0/3407745031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b74158490 con 0x7f1b740b4430 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.461+0000 7f1b7359e640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b740b4430 0x7f1b74157810 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00c730 tx=0x7f1b6c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.461+0000 7f1b5b7fe640 1 -- 172.21.15.81:0/3407745031 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c003b70 con 0x7f1b740b4430 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.461+0000 7f1b5b7fe640 1 -- 172.21.15.81:0/3407745031 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b6c003d10 con 0x7f1b740b4430 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.461+0000 7f1b79112640 1 -- 172.21.15.81:0/3407745031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b74158770 con 0x7f1b740b4430 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.461+0000 7f1b5b7fe640 1 -- 172.21.15.81:0/3407745031 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c00aab0 con 0x7f1b740b4430 2024-06-29T14:34:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.461+0000 7f1b79112640 1 -- 172.21.15.81:0/3407745031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b74158d30 con 0x7f1b740b4430 2024-06-29T14:34:46.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.462+0000 7f1b79112640 1 -- 172.21.15.81:0/3407745031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b7415b5b0 con 0x7f1b740b4430 2024-06-29T14:34:46.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.463+0000 7f1b5b7fe640 1 -- 172.21.15.81:0/3407745031 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 98039+0+0 (secure 0 0 0) 0x7f1b6c00ac50 con 0x7f1b740b4430 2024-06-29T14:34:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.463+0000 7f1b5b7fe640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1b60076120 0x7f1b60078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.463+0000 7f1b5b7fe640 1 -- 172.21.15.81:0/3407745031 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 7130+0+0 (secure 0 0 0) 0x7f1b6c0977a0 con 0x7f1b740b4430 2024-06-29T14:34:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.463+0000 7f1b72d9d640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1b60076120 0x7f1b60078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.464+0000 7f1b72d9d640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1b60076120 0x7f1b60078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1b64033000 tx=0x7f1b64004730 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:46.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.468+0000 7f1b5b7fe640 1 -- 172.21.15.81:0/3407745031 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1b6c060eb0 con 0x7f1b740b4430 2024-06-29T14:34:46.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.648+0000 7f1b79112640 1 -- 172.21.15.81:0/3407745031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b74158220 con 0x7f1b740b4430 2024-06-29T14:34:46.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.649+0000 7f1b5b7fe640 1 -- 172.21.15.81:0/3407745031 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+18809 (secure 0 0 0) 0x7f1b6c0607b0 con 0x7f1b740b4430 2024-06-29T14:34:46.648 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:46.648 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:34:45.828767+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:23:34.937915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-06-29T14:23:36.258024+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":221,"snap_epoch":49,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2999883788":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/3190548120":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/2974208992":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/3819768847":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6832/3101542440":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6833/3101542440":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/3134616648":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6833/189854401":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/1701819343":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/3793734276":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/154739969":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/2258575888":"2024-06-30T14:34:44.823719+0000","172.21.15.167:6832/189854401":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/1556709624":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/3972870238":"2024-06-30T14:34:44.823719+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":9,"length":1},{"begin":11,"length":2},{"begin":14,"length":1},{"begin":16,"length":1},{"begin":20,"length":1},{"begin":23,"length":1},{"begin":25,"length":1},{"begin":27,"length":1},{"begin":31,"length":2},{"begin":34,"length":1},{"begin":36,"length":1},{"begin":38,"length":1},{"begin":43,"length":1},{"begin":45,"length":1},{"begin":47,"length":1},{"begin":49,"length":1},{"begin":53,"length":1},{"begin":55,"length":1},{"begin":57,"length":1},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":65,"length":1},{"begin":67,"length":1},{"begin":69,"length":1},{"begin":74,"length":1},{"begin":76,"length":1},{"begin":78,"length":1},{"begin":80,"length":1},{"begin":83,"length":1},{"begin":85,"length":1},{"begin":87,"length":1},{"begin":89,"length":1},{"begin":91,"length":2},{"begin":94,"length":1},{"begin":96,"length":1},{"begin":98,"length":1},{"begin":100,"length":1},{"begin":103,"length":1},{"begin":105,"length":1},{"begin":107,"length":1},{"begin":109,"length":1},{"begin":111,"length":1},{"begin":175,"length":47}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:46.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 -- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1b60076120 msgr2=0x7f1b60078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1b60076120 0x7f1b60078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1b64033000 tx=0x7f1b64004730 comp rx=0 tx=0).stop 2024-06-29T14:34:46.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 -- 172.21.15.81:0/3407745031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b740b4430 msgr2=0x7f1b74157810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b740b4430 0x7f1b74157810 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00c730 tx=0x7f1b6c00cbf0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 -- 172.21.15.81:0/3407745031 shutdown_connections 2024-06-29T14:34:46.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1b60076120 0x7f1b60078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b740b4430 0x7f1b74157810 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b740b35b0 0x7f1b741465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 --2- 172.21.15.81:0/3407745031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b7406f660 0x7f1b74159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 -- 172.21.15.81:0/3407745031 >> 172.21.15.81:0/3407745031 conn(0x7f1b7406ccc0 msgr2=0x7f1b7415c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 -- 172.21.15.81:0/3407745031 shutdown_connections 2024-06-29T14:34:46.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.651+0000 7f1b79112640 1 -- 172.21.15.81:0/3407745031 wait complete. 2024-06-29T14:34:46.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:34:46.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.778+0000 7f0fe408f640 1 -- 172.21.15.81:0/2756106191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fc8003a50 msgr2=0x7f0fc8003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.778+0000 7f0fe408f640 1 --2- 172.21.15.81:0/2756106191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fc8003a50 0x7f0fc8003ec0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8009b50 tx=0x7f0fc8005830 comp rx=0 tx=0).stop 2024-06-29T14:34:46.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.778+0000 7f0fe408f640 1 -- 172.21.15.81:0/2756106191 shutdown_connections 2024-06-29T14:34:46.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.778+0000 7f0fe408f640 1 --2- 172.21.15.81:0/2756106191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fc8003a50 0x7f0fc8003ec0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.778+0000 7f0fe408f640 1 -- 172.21.15.81:0/2756106191 >> 172.21.15.81:0/2756106191 conn(0x7f0fdc09cf30 msgr2=0x7f0fdc0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:46.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.778+0000 7f0fe408f640 1 -- 172.21.15.81:0/2756106191 shutdown_connections 2024-06-29T14:34:46.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.778+0000 7f0fe408f640 1 -- 172.21.15.81:0/2756106191 wait complete. 2024-06-29T14:34:46.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.780+0000 7f0fe408f640 1 Processor -- start 2024-06-29T14:34:46.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.780+0000 7f0fe408f640 1 -- start start 2024-06-29T14:34:46.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe408f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fc8003a50 0x7f0fdc0a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe408f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fdc14ec10 0x7f0fdc0a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe408f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0a0be0 0x7f0fdc151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe408f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc0d0810 con 0x7f0fc8003a50 2024-06-29T14:34:46.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe408f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc0d0690 con 0x7f0fdc0a0be0 2024-06-29T14:34:46.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe408f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0fdc0d0990 con 0x7f0fdc14ec10 2024-06-29T14:34:46.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe1e2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fc8003a50 0x7f0fdc0a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe1e2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fc8003a50 0x7f0fdc0a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37870/0 (socket says 172.21.15.81:37870) 2024-06-29T14:34:46.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe1e2b640 1 -- 172.21.15.81:0/258129592 learned_addr learned my addr 172.21.15.81:0/258129592 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:46.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe162a640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fdc14ec10 0x7f0fdc0a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe262c640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0a0be0 0x7f0fdc151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe1e2b640 1 -- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fdc14ec10 msgr2=0x7f0fdc0a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe1e2b640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fdc14ec10 0x7f0fdc0a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe1e2b640 1 -- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0a0be0 msgr2=0x7f0fdc151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe1e2b640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0a0be0 0x7f0fdc151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe1e2b640 1 -- 172.21.15.81:0/258129592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fdc0b3e40 con 0x7f0fc8003a50 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.781+0000 7f0fe262c640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0a0be0 0x7f0fdc151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.782+0000 7f0fe1e2b640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fc8003a50 0x7f0fdc0a0160 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8036040 tx=0x7f0fd80336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.782+0000 7f0fd27fc640 1 -- 172.21.15.81:0/258129592 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8038040 con 0x7f0fc8003a50 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.782+0000 7f0fd27fc640 1 -- 172.21.15.81:0/258129592 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0fd8041070 con 0x7f0fc8003a50 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.782+0000 7f0fd27fc640 1 -- 172.21.15.81:0/258129592 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd803c430 con 0x7f0fc8003a50 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.782+0000 7f0fe408f640 1 -- 172.21.15.81:0/258129592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fdc0b4120 con 0x7f0fc8003a50 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.782+0000 7f0fe408f640 1 -- 172.21.15.81:0/258129592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fdc0b46e0 con 0x7f0fc8003a50 2024-06-29T14:34:46.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.783+0000 7f0fe408f640 1 -- 172.21.15.81:0/258129592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fdc0d1a80 con 0x7f0fc8003a50 2024-06-29T14:34:46.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.784+0000 7f0fd27fc640 1 -- 172.21.15.81:0/258129592 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 98039+0+0 (secure 0 0 0) 0x7f0fd803c5d0 con 0x7f0fc8003a50 2024-06-29T14:34:46.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.784+0000 7f0fd27fc640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0fc0076120 0x7f0fc0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:46.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.784+0000 7f0fd27fc640 1 -- 172.21.15.81:0/258129592 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 7130+0+0 (secure 0 0 0) 0x7f0fd80b84e0 con 0x7f0fc8003a50 2024-06-29T14:34:46.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.784+0000 7f0fe162a640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0fc0076120 0x7f0fc0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:46.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.785+0000 7f0fe162a640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0fc0076120 0x7f0fc0078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0018a0 tx=0x7f0fcc000ed0 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:46.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:46.788+0000 7f0fd27fc640 1 -- 172.21.15.81:0/258129592 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0fd8081b60 con 0x7f0fc8003a50 2024-06-29T14:34:47.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.011+0000 7f0fe408f640 1 -- 172.21.15.81:0/258129592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0fdc160650 con 0x7f0fc8003a50 2024-06-29T14:34:47.010 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T14:34:47.012+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:34:47.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.013+0000 7f0fd27fc640 1 -- 172.21.15.81:0/258129592 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7f0fd8081460 con 0x7f0fc8003a50 2024-06-29T14:34:47.012 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:34:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.016+0000 7f0fe408f640 1 -- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0fc0076120 msgr2=0x7f0fc0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.016+0000 7f0fe408f640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0fc0076120 0x7f0fc0078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0018a0 tx=0x7f0fcc000ed0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.016+0000 7f0fe408f640 1 -- 172.21.15.81:0/258129592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fc8003a50 msgr2=0x7f0fdc0a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.016+0000 7f0fe408f640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fc8003a50 0x7f0fdc0a0160 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8036040 tx=0x7f0fd80336f0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.016+0000 7f0fe408f640 1 -- 172.21.15.81:0/258129592 shutdown_connections 2024-06-29T14:34:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.016+0000 7f0fe408f640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0a0be0 0x7f0fdc151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.016+0000 7f0fe408f640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fdc14ec10 0x7f0fdc0a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.016+0000 7f0fe408f640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0fc0076120 0x7f0fc0078630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.016+0000 7f0fe408f640 1 --2- 172.21.15.81:0/258129592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fc8003a50 0x7f0fdc0a0160 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.016+0000 7f0fe408f640 1 -- 172.21.15.81:0/258129592 >> 172.21.15.81:0/258129592 conn(0x7f0fdc09cf30 msgr2=0x7f0fdc0ca560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.017+0000 7f0fe408f640 1 -- 172.21.15.81:0/258129592 shutdown_connections 2024-06-29T14:34:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.017+0000 7f0fe408f640 1 -- 172.21.15.81:0/258129592 wait complete. 2024-06-29T14:34:47.027 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:34:47.075 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:34:47.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.147+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/1698034715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb240050c0 msgr2=0x7fbb2400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.147+0000 7fbb3d2e7640 1 --2- 172.21.15.81:0/1698034715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb240050c0 0x7fbb2400b9c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000c530 tx=0x7fbb24005830 comp rx=0 tx=0).stop 2024-06-29T14:34:47.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.147+0000 7fbb377fe640 1 -- 172.21.15.81:0/1698034715 reap_dead start 2024-06-29T14:34:47.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.147+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/1698034715 shutdown_connections 2024-06-29T14:34:47.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.147+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/1698034715 >> 172.21.15.81:0/1698034715 conn(0x7fbb380d3bd0 msgr2=0x7fbb380d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:47.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.147+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/1698034715 shutdown_connections 2024-06-29T14:34:47.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.147+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/1698034715 wait complete. 2024-06-29T14:34:47.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.149+0000 7fbb3d2e7640 1 Processor -- start 2024-06-29T14:34:47.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.149+0000 7fbb3d2e7640 1 -- start start 2024-06-29T14:34:47.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.150+0000 7fbb3d2e7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb240050c0 0x7fbb381e3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.150+0000 7fbb3d2e7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb380b1840 0x7fbb381d3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.150+0000 7fbb3d2e7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb381f4b10 0x7fbb381f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.150+0000 7fbb3d2e7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb380d46f0 con 0x7fbb240050c0 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.150+0000 7fbb3d2e7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbb380d4870 con 0x7fbb381f4b10 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.150+0000 7fbb3d2e7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbb380b7130 con 0x7fbb380b1840 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb36ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb240050c0 0x7fbb381e3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb36ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb240050c0 0x7fbb381e3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37896/0 (socket says 172.21.15.81:37896) 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb36ffd640 1 -- 172.21.15.81:0/2465096071 learned_addr learned my addr 172.21.15.81:0/2465096071 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb377fe640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb381f4b10 0x7fbb381f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb2e7fc640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb380b1840 0x7fbb381d3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb36ffd640 1 -- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb380b1840 msgr2=0x7fbb381d3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb36ffd640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb380b1840 0x7fbb381d3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb36ffd640 1 -- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb381f4b10 msgr2=0x7fbb381f4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb36ffd640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb381f4b10 0x7fbb381f4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:47.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb36ffd640 1 -- 172.21.15.81:0/2465096071 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb381e2cc0 con 0x7fbb240050c0 2024-06-29T14:34:47.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb36ffd640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb240050c0 0x7fbb381e3d60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009fd0 tx=0x7fbb28011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:47.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb2ffff640 1 -- 172.21.15.81:0/2465096071 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28004460 con 0x7fbb240050c0 2024-06-29T14:34:47.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb2ffff640 1 -- 172.21.15.81:0/2465096071 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb28016070 con 0x7fbb240050c0 2024-06-29T14:34:47.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/2465096071 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb381e2fa0 con 0x7fbb240050c0 2024-06-29T14:34:47.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb2ffff640 1 -- 172.21.15.81:0/2465096071 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800fb60 con 0x7fbb240050c0 2024-06-29T14:34:47.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.151+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/2465096071 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb381e3560 con 0x7fbb240050c0 2024-06-29T14:34:47.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.152+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/2465096071 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb380b6a20 con 0x7fbb240050c0 2024-06-29T14:34:47.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.153+0000 7fbb2ffff640 1 -- 172.21.15.81:0/2465096071 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 98039+0+0 (secure 0 0 0) 0x7fbb28018030 con 0x7fbb240050c0 2024-06-29T14:34:47.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.153+0000 7fbb2ffff640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbb18076150 0x7fbb18078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:47.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.153+0000 7fbb2ffff640 1 -- 172.21.15.81:0/2465096071 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 7172+0+0 (secure 0 0 0) 0x7fbb28098220 con 0x7fbb240050c0 2024-06-29T14:34:47.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.154+0000 7fbb2e7fc640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbb18076150 0x7fbb18078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:47.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.154+0000 7fbb2e7fc640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbb18076150 0x7fbb18078660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb380b40c0 tx=0x7fbb20006b80 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:47.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.158+0000 7fbb2ffff640 1 -- 172.21.15.81:0/2465096071 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb280617f0 con 0x7fbb240050c0 2024-06-29T14:34:47.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.169+0000 7fbb2ffff640 1 -- 172.21.15.81:0/2465096071 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 17) ==== 98247+0+0 (secure 0 0 0) 0x7fbb2805d380 con 0x7fbb240050c0 2024-06-29T14:34:47.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:47.380+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/2465096071 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fbb380b8bd0 con 0x7fbb240050c0 2024-06-29T14:34:48.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.016+0000 7fbb2ffff640 1 -- 172.21.15.81:0/2465096071 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7fbb2806bd50 con 0x7fbb240050c0 2024-06-29T14:34:48.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.018+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbb18076150 msgr2=0x7fbb18078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.018+0000 7fbb3d2e7640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbb18076150 0x7fbb18078660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb380b40c0 tx=0x7fbb20006b80 comp rx=0 tx=0).stop 2024-06-29T14:34:48.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.018+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/2465096071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb240050c0 msgr2=0x7fbb381e3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.018+0000 7fbb3d2e7640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb240050c0 0x7fbb381e3d60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009fd0 tx=0x7fbb28011cb0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.019+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/2465096071 shutdown_connections 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.019+0000 7fbb3d2e7640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb381f4b10 0x7fbb381f4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.019+0000 7fbb3d2e7640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb380b1840 0x7fbb381d3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.019+0000 7fbb3d2e7640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbb18076150 0x7fbb18078660 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.019+0000 7fbb3d2e7640 1 --2- 172.21.15.81:0/2465096071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb240050c0 0x7fbb381e3d60 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.019+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/2465096071 >> 172.21.15.81:0/2465096071 conn(0x7fbb380d3bd0 msgr2=0x7fbb380cda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.019+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/2465096071 shutdown_connections 2024-06-29T14:34:48.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.019+0000 7fbb3d2e7640 1 -- 172.21.15.81:0/2465096071 wait complete. 2024-06-29T14:34:48.030 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:34:48.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.152+0000 7f6f94cf6640 1 -- 172.21.15.81:0/3450726357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f9015ace0 msgr2=0x7f6f90072ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.152+0000 7f6f94cf6640 1 --2- 172.21.15.81:0/3450726357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f9015ace0 0x7f6f90072ef0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6f900d0770 tx=0x7f6f8402a560 comp rx=0 tx=0).stop 2024-06-29T14:34:48.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.152+0000 7f6f94cf6640 1 -- 172.21.15.81:0/3450726357 shutdown_connections 2024-06-29T14:34:48.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.152+0000 7f6f94cf6640 1 --2- 172.21.15.81:0/3450726357 >> v2:172.21.15.167:3300/0 conn(0x7f6f90073430 0x7f6f9015ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.152+0000 7f6f94cf6640 1 --2- 172.21.15.81:0/3450726357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f9015ace0 0x7f6f90072ef0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.152+0000 7f6f94cf6640 1 --2- 172.21.15.81:0/3450726357 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f900b5a60 0x7f6f900affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.152+0000 7f6f94cf6640 1 -- 172.21.15.81:0/3450726357 >> 172.21.15.81:0/3450726357 conn(0x7f6f900b2580 msgr2=0x7f6f900b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:48.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.152+0000 7f6f94cf6640 1 -- 172.21.15.81:0/3450726357 shutdown_connections 2024-06-29T14:34:48.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.152+0000 7f6f94cf6640 1 -- 172.21.15.81:0/3450726357 wait complete. 2024-06-29T14:34:48.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.154+0000 7f6f94cf6640 1 Processor -- start 2024-06-29T14:34:48.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.154+0000 7f6f94cf6640 1 -- start start 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.155+0000 7f6f94cf6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f90073430 0x7f6f90151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f94cf6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f900b5a60 0x7f6f900c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f94cf6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f9015ace0 0x7f6f900c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f94cf6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9015f2e0 con 0x7f6f90073430 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f94cf6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9015f160 con 0x7f6f900b5a60 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f94cf6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6f9015f460 con 0x7f6f9015ace0 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f90073430 0x7f6f90151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f90073430 0x7f6f90151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37910/0 (socket says 172.21.15.81:37910) 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8e59c640 1 -- 172.21.15.81:0/974524688 learned_addr learned my addr 172.21.15.81:0/974524688 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8ed9d640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f9015ace0 0x7f6f900c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8dd9b640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f900b5a60 0x7f6f900c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8e59c640 1 -- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f9015ace0 msgr2=0x7f6f900c5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8e59c640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f9015ace0 0x7f6f900c5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8e59c640 1 -- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f900b5a60 msgr2=0x7f6f900c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8e59c640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f900b5a60 0x7f6f900c5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8e59c640 1 -- 172.21.15.81:0/974524688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f900c6600 con 0x7f6f90073430 2024-06-29T14:34:48.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f8e59c640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f90073430 0x7f6f90151ef0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000d740 tx=0x7f6f8000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:48.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f7effd640 1 -- 172.21.15.81:0/974524688 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f80017070 con 0x7f6f90073430 2024-06-29T14:34:48.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f7effd640 1 -- 172.21.15.81:0/974524688 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f80009d20 con 0x7f6f90073430 2024-06-29T14:34:48.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f94cf6640 1 -- 172.21.15.81:0/974524688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f900c68e0 con 0x7f6f90073430 2024-06-29T14:34:48.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f7effd640 1 -- 172.21.15.81:0/974524688 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f800106c0 con 0x7f6f90073430 2024-06-29T14:34:48.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.156+0000 7f6f94cf6640 1 -- 172.21.15.81:0/974524688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f900c6ea0 con 0x7f6f90073430 2024-06-29T14:34:48.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.157+0000 7f6f94cf6640 1 -- 172.21.15.81:0/974524688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f9015ea10 con 0x7f6f90073430 2024-06-29T14:34:48.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.158+0000 7f6f7effd640 1 -- 172.21.15.81:0/974524688 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 98247+0+0 (secure 0 0 0) 0x7f6f8001a070 con 0x7f6f90073430 2024-06-29T14:34:48.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.159+0000 7f6f7effd640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6f70076360 0x7f6f70078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:48.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.159+0000 7f6f7effd640 1 -- 172.21.15.81:0/974524688 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 7200+0+0 (secure 0 0 0) 0x7f6f80097ed0 con 0x7f6f90073430 2024-06-29T14:34:48.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.159+0000 7f6f8dd9b640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6f70076360 0x7f6f70078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:48.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.160+0000 7f6f8dd9b640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6f70076360 0x7f6f70078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6f84033000 tx=0x7f6f84004730 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:48.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.163+0000 7f6f7effd640 1 -- 172.21.15.81:0/974524688 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6f80060aa0 con 0x7f6f90073430 2024-06-29T14:34:48.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:48.346+0000 7f6f94cf6640 1 -- 172.21.15.81:0/974524688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6f90157e40 con 0x7f6f90073430 2024-06-29T14:34:49.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.185+0000 7f6f7effd640 1 -- 172.21.15.81:0/974524688 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v52) ==== 181+0+0 (secure 0 0 0) 0x7f6f80064da0 con 0x7f6f90073430 2024-06-29T14:34:49.184 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:34:49.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.189+0000 7f6f94cf6640 1 -- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6f70076360 msgr2=0x7f6f70078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:49.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.189+0000 7f6f94cf6640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6f70076360 0x7f6f70078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6f84033000 tx=0x7f6f84004730 comp rx=0 tx=0).stop 2024-06-29T14:34:49.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.189+0000 7f6f94cf6640 1 -- 172.21.15.81:0/974524688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f90073430 msgr2=0x7f6f90151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:49.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.190+0000 7f6f7effd640 1 -- 172.21.15.81:0/974524688 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f6f8005cd30 con 0x7f6f90073430 2024-06-29T14:34:49.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.190+0000 7f6f94cf6640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f90073430 0x7f6f90151ef0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000d740 tx=0x7f6f8000dc00 comp rx=0 tx=0).stop 2024-06-29T14:34:49.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.191+0000 7f6f7effd640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6f70076360 0x7f6f70078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:49.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.191+0000 7f6f94cf6640 1 -- 172.21.15.81:0/974524688 shutdown_connections 2024-06-29T14:34:49.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.191+0000 7f6f94cf6640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f9015ace0 0x7f6f900c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.191+0000 7f6f94cf6640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f900b5a60 0x7f6f900c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.191+0000 7f6f94cf6640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6f70076360 0x7f6f70078870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.192+0000 7f6f7effd640 1 -- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6f70076360 msgr2=0x7f6f70078870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-29T14:34:49.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.192+0000 7f6f7effd640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6f70076360 0x7f6f70078870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.192+0000 7f6f94cf6640 1 --2- 172.21.15.81:0/974524688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f90073430 0x7f6f90151ef0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.192+0000 7f6f94cf6640 1 -- 172.21.15.81:0/974524688 >> 172.21.15.81:0/974524688 conn(0x7f6f900b2580 msgr2=0x7f6f90152cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:49.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.192+0000 7f6f94cf6640 1 -- 172.21.15.81:0/974524688 shutdown_connections 2024-06-29T14:34:49.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.192+0000 7f6f94cf6640 1 -- 172.21.15.81:0/974524688 wait complete. 2024-06-29T14:34:49.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:34:49.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.329+0000 7faec55a1640 1 -- 172.21.15.81:0/2443527621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec00d0420 msgr2=0x7faec00cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:49.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.329+0000 7faec55a1640 1 --2- 172.21.15.81:0/2443527621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec00d0420 0x7faec00cd2b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009830 tx=0x7faeb802a510 comp rx=0 tx=0).stop 2024-06-29T14:34:49.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.329+0000 7faec55a1640 1 -- 172.21.15.81:0/2443527621 shutdown_connections 2024-06-29T14:34:49.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.329+0000 7faec55a1640 1 --2- 172.21.15.81:0/2443527621 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec006ca00 0x7faec006f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.329+0000 7faec55a1640 1 --2- 172.21.15.81:0/2443527621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec00d0420 0x7faec00cd2b0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.329+0000 7faec55a1640 1 -- 172.21.15.81:0/2443527621 >> 172.21.15.81:0/2443527621 conn(0x7faec00b2580 msgr2=0x7faec00b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:49.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.329+0000 7faec55a1640 1 -- 172.21.15.81:0/2443527621 shutdown_connections 2024-06-29T14:34:49.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.329+0000 7faec55a1640 1 -- 172.21.15.81:0/2443527621 wait complete. 2024-06-29T14:34:49.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.331+0000 7faec55a1640 1 Processor -- start 2024-06-29T14:34:49.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.331+0000 7faec55a1640 1 -- start start 2024-06-29T14:34:49.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.332+0000 7faec55a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec00cd7f0 0x7faec013e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:49.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.332+0000 7faec55a1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faec00d0420 0x7faec015ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:49.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.332+0000 7faec55a1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec0144840 0x7faec0144c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:49.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.332+0000 7faec55a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faec006ffe0 con 0x7faec00cd7f0 2024-06-29T14:34:49.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.332+0000 7faec55a1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faec006fe60 con 0x7faec00d0420 2024-06-29T14:34:49.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.332+0000 7faec55a1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faec0070160 con 0x7faec0144840 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.332+0000 7faebeffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec00cd7f0 0x7faec013e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.332+0000 7faebeffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec00cd7f0 0x7faec013e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37920/0 (socket says 172.21.15.81:37920) 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.332+0000 7faebeffd640 1 -- 172.21.15.81:0/2343660742 learned_addr learned my addr 172.21.15.81:0/2343660742 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faebf7fe640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec0144840 0x7faec0144c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faebeffd640 1 -- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec0144840 msgr2=0x7faec0144c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faebeffd640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec0144840 0x7faec0144c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faebeffd640 1 -- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faec00d0420 msgr2=0x7faec015ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faebe7fc640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faec00d0420 0x7faec015ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faebeffd640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faec00d0420 0x7faec015ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faebeffd640 1 -- 172.21.15.81:0/2343660742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec0145190 con 0x7faec00cd7f0 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faebf7fe640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec0144840 0x7faec0144c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faebeffd640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec00cd7f0 0x7faec013e2b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7faeb802ab00 tx=0x7faeb8033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:49.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faea77fe640 1 -- 172.21.15.81:0/2343660742 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb8031d60 con 0x7faec00cd7f0 2024-06-29T14:34:49.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faea77fe640 1 -- 172.21.15.81:0/2343660742 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faeb8005020 con 0x7faec00cd7f0 2024-06-29T14:34:49.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faec55a1640 1 -- 172.21.15.81:0/2343660742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faec0142d10 con 0x7faec00cd7f0 2024-06-29T14:34:49.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faea77fe640 1 -- 172.21.15.81:0/2343660742 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb8005470 con 0x7faec00cd7f0 2024-06-29T14:34:49.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.333+0000 7faec55a1640 1 -- 172.21.15.81:0/2343660742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faec01432d0 con 0x7faec00cd7f0 2024-06-29T14:34:49.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.334+0000 7faec55a1640 1 -- 172.21.15.81:0/2343660742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faec00706f0 con 0x7faec00cd7f0 2024-06-29T14:34:49.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.335+0000 7faea77fe640 1 -- 172.21.15.81:0/2343660742 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7faeb80047a0 con 0x7faec00cd7f0 2024-06-29T14:34:49.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.336+0000 7faea77fe640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7faeac0763f0 0x7faeac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:49.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.336+0000 7faea77fe640 1 -- 172.21.15.81:0/2343660742 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 6046+0+0 (secure 0 0 0) 0x7faeb80b8ec0 con 0x7faec00cd7f0 2024-06-29T14:34:49.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.336+0000 7faebe7fc640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7faeac0763f0 0x7faeac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:49.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.337+0000 7faebe7fc640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7faeac0763f0 0x7faeac078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faec00b20a0 tx=0x7faeb4009040 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:49.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.340+0000 7faea77fe640 1 -- 172.21.15.81:0/2343660742 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faec0145190 con 0x7faec00cd7f0 2024-06-29T14:34:49.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:49.518+0000 7faec55a1640 1 -- 172.21.15.81:0/2343660742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7faec0071520 con 0x7faec00cd7f0 2024-06-29T14:34:50.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.193+0000 7faea77fe640 1 -- 172.21.15.81:0/2343660742 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v53) ==== 169+0+0 (secure 0 0 0) 0x7faeb803f030 con 0x7faec00cd7f0 2024-06-29T14:34:50.192 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T14:34:50.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.196+0000 7faec55a1640 1 -- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7faeac0763f0 msgr2=0x7faeac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.196+0000 7faec55a1640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7faeac0763f0 0x7faeac078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faec00b20a0 tx=0x7faeb4009040 comp rx=0 tx=0).stop 2024-06-29T14:34:50.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.196+0000 7faec55a1640 1 -- 172.21.15.81:0/2343660742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec00cd7f0 msgr2=0x7faec013e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.196+0000 7faec55a1640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec00cd7f0 0x7faec013e2b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7faeb802ab00 tx=0x7faeb8033ce0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.196+0000 7faec55a1640 1 -- 172.21.15.81:0/2343660742 shutdown_connections 2024-06-29T14:34:50.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.196+0000 7faec55a1640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faec0144840 0x7faec0144c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.196+0000 7faec55a1640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faec00d0420 0x7faec015ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.196+0000 7faec55a1640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7faeac0763f0 0x7faeac078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.196+0000 7faec55a1640 1 --2- 172.21.15.81:0/2343660742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faec00cd7f0 0x7faec013e2b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.196+0000 7faec55a1640 1 -- 172.21.15.81:0/2343660742 >> 172.21.15.81:0/2343660742 conn(0x7faec00b2580 msgr2=0x7faec0070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:50.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.197+0000 7faec55a1640 1 -- 172.21.15.81:0/2343660742 shutdown_connections 2024-06-29T14:34:50.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.197+0000 7faec55a1640 1 -- 172.21.15.81:0/2343660742 wait complete. 2024-06-29T14:34:50.209 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:34:50.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.323+0000 7f4e91154640 1 -- 172.21.15.81:0/2077359336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c073740 msgr2=0x7f4e8c15e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.323+0000 7f4e91154640 1 --2- 172.21.15.81:0/2077359336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c073740 0x7f4e8c15e960 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c070e50 tx=0x7f4e8402aa30 comp rx=0 tx=0).stop 2024-06-29T14:34:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.324+0000 7f4e91154640 1 -- 172.21.15.81:0/2077359336 shutdown_connections 2024-06-29T14:34:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.324+0000 7f4e91154640 1 --2- 172.21.15.81:0/2077359336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c073740 0x7f4e8c15e960 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.324+0000 7f4e91154640 1 --2- 172.21.15.81:0/2077359336 >> v2:172.21.15.167:3300/0 conn(0x7f4e8c1412f0 0x7f4e8c0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.324+0000 7f4e91154640 1 -- 172.21.15.81:0/2077359336 >> 172.21.15.81:0/2077359336 conn(0x7f4e8c06ccc0 msgr2=0x7f4e8c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.324+0000 7f4e91154640 1 -- 172.21.15.81:0/2077359336 shutdown_connections 2024-06-29T14:34:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.324+0000 7f4e91154640 1 -- 172.21.15.81:0/2077359336 wait complete. 2024-06-29T14:34:50.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.325+0000 7f4e91154640 1 Processor -- start 2024-06-29T14:34:50.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.326+0000 7f4e91154640 1 -- start start 2024-06-29T14:34:50.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e91154640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0cf4d0 0x7f4e8c13fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e91154640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e8c1412f0 0x7f4e8c156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e91154640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e8c1404b0 0x7f4e8c1f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e91154640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c0b3210 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e91154640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c15f1f0 con 0x7f4e8c1404b0 2024-06-29T14:34:50.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e91154640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8c0b3390 con 0x7f4e8c1412f0 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0cf4d0 0x7f4e8c13fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0cf4d0 0x7f4e8c13fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60116/0 (socket says 172.21.15.81:60116) 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8ad9d640 1 -- 172.21.15.81:0/3945114435 learned_addr learned my addr 172.21.15.81:0/3945114435 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8b59e640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e8c1404b0 0x7f4e8c1f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8259c640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e8c1412f0 0x7f4e8c156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8ad9d640 1 -- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e8c1412f0 msgr2=0x7f4e8c156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8ad9d640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e8c1412f0 0x7f4e8c156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8ad9d640 1 -- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e8c1404b0 msgr2=0x7f4e8c1f3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8ad9d640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e8c1404b0 0x7f4e8c1f3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8ad9d640 1 -- 172.21.15.81:0/3945114435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c1549b0 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.327+0000 7f4e8b59e640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e8c1404b0 0x7f4e8c1f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:50.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.328+0000 7f4e8ad9d640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0cf4d0 0x7f4e8c13fd90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00b410 tx=0x7f4e7c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:50.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.328+0000 7f4e83fff640 1 -- 172.21.15.81:0/3945114435 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c015020 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.328+0000 7f4e83fff640 1 -- 172.21.15.81:0/3945114435 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e7c011940 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.328+0000 7f4e91154640 1 -- 172.21.15.81:0/3945114435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8c155c70 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.328+0000 7f4e83fff640 1 -- 172.21.15.81:0/3945114435 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c011d90 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.328+0000 7f4e91154640 1 -- 172.21.15.81:0/3945114435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e8c156420 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.329+0000 7f4e91154640 1 -- 172.21.15.81:0/3945114435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e8c0b3930 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.330+0000 7f4e83fff640 1 -- 172.21.15.81:0/3945114435 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f4e7c018020 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.330+0000 7f4e83fff640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f4e6c0763f0 0x7f4e6c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.330+0000 7f4e83fff640 1 -- 172.21.15.81:0/3945114435 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5632+0+0 (secure 0 0 0) 0x7f4e7c0978c0 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.330+0000 7f4e8259c640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f4e6c0763f0 0x7f4e6c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.331+0000 7f4e8259c640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f4e6c0763f0 0x7f4e6c078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4e74006fd0 tx=0x7f4e74008040 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:50.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.334+0000 7f4e83fff640 1 -- 172.21.15.81:0/3945114435 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4e7c01f020 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.514+0000 7f4e91154640 1 -- 172.21.15.81:0/3945114435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f4e8c159c80 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.518+0000 7f4e83fff640 1 -- 172.21.15.81:0/3945114435 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v31) ==== 71+0+0 (secure 0 0 0) 0x7f4e7c061160 con 0x7f4e8c0cf4d0 2024-06-29T14:34:50.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 -- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f4e6c0763f0 msgr2=0x7f4e6c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f4e6c0763f0 0x7f4e6c078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4e74006fd0 tx=0x7f4e74008040 comp rx=0 tx=0).stop 2024-06-29T14:34:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 -- 172.21.15.81:0/3945114435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0cf4d0 msgr2=0x7f4e8c13fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0cf4d0 0x7f4e8c13fd90 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00b410 tx=0x7f4e7c00b8d0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 -- 172.21.15.81:0/3945114435 shutdown_connections 2024-06-29T14:34:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e8c1404b0 0x7f4e8c1f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e8c1412f0 0x7f4e8c156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f4e6c0763f0 0x7f4e6c078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 --2- 172.21.15.81:0/3945114435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0cf4d0 0x7f4e8c13fd90 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 -- 172.21.15.81:0/3945114435 >> 172.21.15.81:0/3945114435 conn(0x7f4e8c06ccc0 msgr2=0x7f4e8c06df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 -- 172.21.15.81:0/3945114435 shutdown_connections 2024-06-29T14:34:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.520+0000 7f4e91154640 1 -- 172.21.15.81:0/3945114435 wait complete. 2024-06-29T14:34:50.530 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2024-06-29T14:34:50.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.644+0000 7f8c4303c640 1 -- 172.21.15.81:0/599930699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c3c073740 msgr2=0x7f8c3c15e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.644+0000 7f8c4303c640 1 --2- 172.21.15.81:0/599930699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c3c073740 0x7f8c3c15e960 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c070e50 tx=0x7f8c3402a560 comp rx=0 tx=0).stop 2024-06-29T14:34:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.644+0000 7f8c4303c640 1 -- 172.21.15.81:0/599930699 shutdown_connections 2024-06-29T14:34:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.644+0000 7f8c4303c640 1 --2- 172.21.15.81:0/599930699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c3c073740 0x7f8c3c15e960 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.644+0000 7f8c4303c640 1 --2- 172.21.15.81:0/599930699 >> v2:172.21.15.167:3300/0 conn(0x7f8c3c1412f0 0x7f8c3c0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.644+0000 7f8c4303c640 1 -- 172.21.15.81:0/599930699 >> 172.21.15.81:0/599930699 conn(0x7f8c3c06ccc0 msgr2=0x7f8c3c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.644+0000 7f8c4303c640 1 -- 172.21.15.81:0/599930699 shutdown_connections 2024-06-29T14:34:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.644+0000 7f8c4303c640 1 -- 172.21.15.81:0/599930699 wait complete. 2024-06-29T14:34:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.645+0000 7f8c4303c640 1 Processor -- start 2024-06-29T14:34:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.646+0000 7f8c4303c640 1 -- start start 2024-06-29T14:34:50.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c4303c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c3c0cf4d0 0x7f8c3c134ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c4303c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c3c1412f0 0x7f8c3c158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c4303c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c3c158a60 0x7f8c3c158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c4303c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c0b3210 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c4303c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c15f1f0 con 0x7f8c3c1412f0 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c4303c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8c3c0b3390 con 0x7f8c3c158a60 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c40dd8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c3c0cf4d0 0x7f8c3c134ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c40dd8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c3c0cf4d0 0x7f8c3c134ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60130/0 (socket says 172.21.15.81:60130) 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c40dd8640 1 -- 172.21.15.81:0/718461892 learned_addr learned my addr 172.21.15.81:0/718461892 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c415d9640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c3c158a60 0x7f8c3c158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c3bfff640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c3c1412f0 0x7f8c3c158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c40dd8640 1 -- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c3c158a60 msgr2=0x7f8c3c158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c40dd8640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c3c158a60 0x7f8c3c158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c40dd8640 1 -- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c3c1412f0 msgr2=0x7f8c3c158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c40dd8640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c3c1412f0 0x7f8c3c158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.647+0000 7f8c40dd8640 1 -- 172.21.15.81:0/718461892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3c1593b0 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.648+0000 7f8c40dd8640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c3c0cf4d0 0x7f8c3c134ef0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c00d700 tx=0x7f8c2c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:50.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.648+0000 7f8c397fa640 1 -- 172.21.15.81:0/718461892 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c016070 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.648+0000 7f8c397fa640 1 -- 172.21.15.81:0/718461892 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c2c005220 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.648+0000 7f8c397fa640 1 -- 172.21.15.81:0/718461892 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c009820 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.648+0000 7f8c4303c640 1 -- 172.21.15.81:0/718461892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c3c158060 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.648+0000 7f8c4303c640 1 -- 172.21.15.81:0/718461892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c3c156b60 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.649+0000 7f8c4303c640 1 -- 172.21.15.81:0/718461892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c3c0cce40 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.650+0000 7f8c397fa640 1 -- 172.21.15.81:0/718461892 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f8c2c0099c0 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.650+0000 7f8c397fa640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f8c28076420 0x7f8c28078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:50.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.650+0000 7f8c397fa640 1 -- 172.21.15.81:0/718461892 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5632+0+0 (secure 0 0 0) 0x7f8c2c099970 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.650+0000 7f8c3bfff640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f8c28076420 0x7f8c28078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:50.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.651+0000 7f8c3bfff640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f8c28076420 0x7f8c28078930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c1e5980 tx=0x7f8c300094d0 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:50.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.654+0000 7f8c397fa640 1 -- 172.21.15.81:0/718461892 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8c3c0cce40 con 0x7f8c3c0cf4d0 2024-06-29T14:34:50.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:50.909+0000 7f8c4303c640 1 -- 172.21.15.81:0/718461892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) -- 0x7f8c3c070280 con 0x7f8c3c0cf4d0 2024-06-29T14:34:51.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.194+0000 7f8c397fa640 1 -- 172.21.15.81:0/718461892 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v436) ==== 158+0+0 (secure 0 0 0) 0x7f8c3c070280 con 0x7f8c3c0cf4d0 2024-06-29T14:34:51.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.196+0000 7f8c4303c640 1 -- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f8c28076420 msgr2=0x7f8c28078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f8c28076420 0x7f8c28078930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c1e5980 tx=0x7f8c300094d0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 -- 172.21.15.81:0/718461892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c3c0cf4d0 msgr2=0x7f8c3c134ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c3c0cf4d0 0x7f8c3c134ef0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c00d700 tx=0x7f8c2c00dbc0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 -- 172.21.15.81:0/718461892 shutdown_connections 2024-06-29T14:34:51.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c3c158a60 0x7f8c3c158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c3c1412f0 0x7f8c3c158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f8c28076420 0x7f8c28078930 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 --2- 172.21.15.81:0/718461892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c3c0cf4d0 0x7f8c3c134ef0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 -- 172.21.15.81:0/718461892 >> 172.21.15.81:0/718461892 conn(0x7f8c3c06ccc0 msgr2=0x7f8c3c153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:51.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 -- 172.21.15.81:0/718461892 shutdown_connections 2024-06-29T14:34:51.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.197+0000 7f8c4303c640 1 -- 172.21.15.81:0/718461892 wait complete. 2024-06-29T14:34:51.206 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2024-06-29T14:34:51.207 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2024-06-29T14:34:51.207 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:34:51.207 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:51.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.322+0000 7f7167939640 1 -- 172.21.15.81:0/1302535870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600cea10 msgr2=0x7f71600b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.322+0000 7f7167939640 1 --2- 172.21.15.81:0/1302535870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600cea10 0x7f71600b47b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f71600b2c10 tx=0x7f715402a540 comp rx=0 tx=0).stop 2024-06-29T14:34:51.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.322+0000 7f7167939640 1 -- 172.21.15.81:0/1302535870 shutdown_connections 2024-06-29T14:34:51.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.322+0000 7f7167939640 1 --2- 172.21.15.81:0/1302535870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600cea10 0x7f71600b47b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.322+0000 7f7167939640 1 --2- 172.21.15.81:0/1302535870 >> v2:172.21.15.167:3300/0 conn(0x7f71600b6d60 0x7f71600ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.322+0000 7f7167939640 1 -- 172.21.15.81:0/1302535870 >> 172.21.15.81:0/1302535870 conn(0x7f71600d1960 msgr2=0x7f71600d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.322+0000 7f7167939640 1 -- 172.21.15.81:0/1302535870 shutdown_connections 2024-06-29T14:34:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.322+0000 7f7167939640 1 -- 172.21.15.81:0/1302535870 wait complete. 2024-06-29T14:34:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.324+0000 7f7167939640 1 Processor -- start 2024-06-29T14:34:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.324+0000 7f7167939640 1 -- start start 2024-06-29T14:34:51.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f7167939640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b4cf0 0x7f716015cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f7167939640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71600b6d60 0x7f7160165950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f7167939640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71600cea10 0x7f7160165e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f7167939640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f71600cd540 con 0x7f71600b4cf0 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f7167939640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f71600d0a10 con 0x7f71600cea10 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f7167939640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f71600cd6c0 con 0x7f71600b6d60 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f71656d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b4cf0 0x7f716015cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f71656d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b4cf0 0x7f716015cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60158/0 (socket says 172.21.15.81:60158) 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f71656d5640 1 -- 172.21.15.81:0/4126368387 learned_addr learned my addr 172.21.15.81:0/4126368387 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f7165ed6640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71600cea10 0x7f7160165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.325+0000 7f7164ed4640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71600b6d60 0x7f7160165950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f71656d5640 1 -- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71600b6d60 msgr2=0x7f7160165950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f71656d5640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71600b6d60 0x7f7160165950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f71656d5640 1 -- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71600cea10 msgr2=0x7f7160165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f71656d5640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71600cea10 0x7f7160165e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f71656d5640 1 -- 172.21.15.81:0/4126368387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71601f6590 con 0x7f71600b4cf0 2024-06-29T14:34:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f71656d5640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b4cf0 0x7f716015cad0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f715000b9f0 tx=0x7f715000beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f715dffb640 1 -- 172.21.15.81:0/4126368387 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150019070 con 0x7f71600b4cf0 2024-06-29T14:34:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f715dffb640 1 -- 172.21.15.81:0/4126368387 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f715000ccc0 con 0x7f71600b4cf0 2024-06-29T14:34:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f7167939640 1 -- 172.21.15.81:0/4126368387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716015e490 con 0x7f71600b4cf0 2024-06-29T14:34:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f715dffb640 1 -- 172.21.15.81:0/4126368387 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71500146a0 con 0x7f71600b4cf0 2024-06-29T14:34:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.326+0000 7f7167939640 1 -- 172.21.15.81:0/4126368387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71601f6b50 con 0x7f71600b4cf0 2024-06-29T14:34:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.327+0000 7f7167939640 1 -- 172.21.15.81:0/4126368387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71600c8a10 con 0x7f71600b4cf0 2024-06-29T14:34:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.328+0000 7f715dffb640 1 -- 172.21.15.81:0/4126368387 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f715001c070 con 0x7f71600b4cf0 2024-06-29T14:34:51.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.328+0000 7f715dffb640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f714c0763f0 0x7f714c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.328+0000 7f715dffb640 1 -- 172.21.15.81:0/4126368387 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5632+0+0 (secure 0 0 0) 0x7f7150097690 con 0x7f71600b4cf0 2024-06-29T14:34:51.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.328+0000 7f7164ed4640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f714c0763f0 0x7f714c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.329+0000 7f7164ed4640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f714c0763f0 0x7f714c078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7154033000 tx=0x7f7154004730 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:51.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.333+0000 7f715dffb640 1 -- 172.21.15.81:0/4126368387 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7150060fc0 con 0x7f71600b4cf0 2024-06-29T14:34:51.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.511+0000 7f7167939640 1 -- 172.21.15.81:0/4126368387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71600ce4d0 con 0x7f71600b4cf0 2024-06-29T14:34:51.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.512+0000 7f715dffb640 1 -- 172.21.15.81:0/4126368387 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) ==== 74+0+14028 (secure 0 0 0) 0x7f71500608c0 con 0x7f71600b4cf0 2024-06-29T14:34:51.511 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:51.511 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":53,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:34:50.187385+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6832/58595811":"2024-06-30T14:34:47.013504+0000","172.21.15.167:0/2999883788":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/3190548120":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/2974208992":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/3819768847":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6832/3101542440":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6833/3101542440":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/3134616648":"2024-06-30T14:23:49.594083+0000","172.21.15.167:6833/189854401":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/1701819343":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/3793734276":"2024-06-30T14:23:49.594083+0000","172.21.15.167:0/154739969":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/2258575888":"2024-06-30T14:34:44.823719+0000","172.21.15.167:6832/189854401":"2024-06-30T14:34:44.823719+0000","172.21.15.81:6834/58595811":"2024-06-30T14:34:47.013504+0000","172.21.15.167:0/1556709624":"2024-06-30T14:34:44.823719+0000","172.21.15.167:0/3972870238":"2024-06-30T14:34:44.823719+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:51.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 -- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f714c0763f0 msgr2=0x7f714c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f714c0763f0 0x7f714c078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7154033000 tx=0x7f7154004730 comp rx=0 tx=0).stop 2024-06-29T14:34:51.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 -- 172.21.15.81:0/4126368387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b4cf0 msgr2=0x7f716015cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b4cf0 0x7f716015cad0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f715000b9f0 tx=0x7f715000beb0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 -- 172.21.15.81:0/4126368387 shutdown_connections 2024-06-29T14:34:51.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71600cea10 0x7f7160165e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71600b6d60 0x7f7160165950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f714c0763f0 0x7f714c078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 --2- 172.21.15.81:0/4126368387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b4cf0 0x7f716015cad0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 -- 172.21.15.81:0/4126368387 >> 172.21.15.81:0/4126368387 conn(0x7f71600d1960 msgr2=0x7f716015f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:51.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 -- 172.21.15.81:0/4126368387 shutdown_connections 2024-06-29T14:34:51.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.514+0000 7f7167939640 1 -- 172.21.15.81:0/4126368387 wait complete. 2024-06-29T14:34:51.525 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:34:51.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:34:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.642+0000 7f3a1203d640 1 -- 172.21.15.81:0/793832422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a0c0ad0c0 msgr2=0x7f3a0c15bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.642+0000 7f3a1203d640 1 --2- 172.21.15.81:0/793832422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a0c0ad0c0 0x7f3a0c15bdf0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c072e90 tx=0x7f39fc02a4d0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.643+0000 7f3a1203d640 1 -- 172.21.15.81:0/793832422 shutdown_connections 2024-06-29T14:34:51.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.643+0000 7f3a1203d640 1 --2- 172.21.15.81:0/793832422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a0c074460 0x7f3a0c0aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.643+0000 7f3a1203d640 1 --2- 172.21.15.81:0/793832422 >> v2:172.21.15.167:3300/0 conn(0x7f3a0c073b10 0x7f3a0c073f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.643+0000 7f3a1203d640 1 --2- 172.21.15.81:0/793832422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a0c0ad0c0 0x7f3a0c15bdf0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.643+0000 7f3a1203d640 1 -- 172.21.15.81:0/793832422 >> 172.21.15.81:0/793832422 conn(0x7f3a0c072950 msgr2=0x7f3a0c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:51.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.643+0000 7f3a1203d640 1 -- 172.21.15.81:0/793832422 shutdown_connections 2024-06-29T14:34:51.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.643+0000 7f3a1203d640 1 -- 172.21.15.81:0/793832422 wait complete. 2024-06-29T14:34:51.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.644+0000 7f3a1203d640 1 Processor -- start 2024-06-29T14:34:51.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.645+0000 7f3a1203d640 1 -- start start 2024-06-29T14:34:51.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a1203d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a0c073b10 0x7f3a0c150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a1203d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a0c074460 0x7f3a0c145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a1203d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a0c0ad0c0 0x7f3a0c0b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a1203d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c0af340 con 0x7f3a0c074460 2024-06-29T14:34:51.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a1203d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c0af1c0 con 0x7f3a0c073b10 2024-06-29T14:34:51.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a1203d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3a0c0af4c0 con 0x7f3a0c0ad0c0 2024-06-29T14:34:51.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a0affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a0c074460 0x7f3a0c145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a0affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a0c074460 0x7f3a0c145470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60188/0 (socket says 172.21.15.81:60188) 2024-06-29T14:34:51.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a0affd640 1 -- 172.21.15.81:0/512321452 learned_addr learned my addr 172.21.15.81:0/512321452 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:51.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a0b7fe640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a0c073b10 0x7f3a0c150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a0bfff640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a0c0ad0c0 0x7f3a0c0b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a0affd640 1 -- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a0c0ad0c0 msgr2=0x7f3a0c0b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a0affd640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a0c0ad0c0 0x7f3a0c0b1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a0affd640 1 -- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a0c073b10 msgr2=0x7f3a0c150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a0affd640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a0c073b10 0x7f3a0c150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.646+0000 7f3a0affd640 1 -- 172.21.15.81:0/512321452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0c153330 con 0x7f3a0c074460 2024-06-29T14:34:51.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.647+0000 7f3a0affd640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a0c074460 0x7f3a0c145470 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000d750 tx=0x7f3a0000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:51.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.647+0000 7f39f3fff640 1 -- 172.21.15.81:0/512321452 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a00017070 con 0x7f3a0c074460 2024-06-29T14:34:51.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.647+0000 7f39f3fff640 1 -- 172.21.15.81:0/512321452 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a00009cc0 con 0x7f3a0c074460 2024-06-29T14:34:51.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.647+0000 7f39f3fff640 1 -- 172.21.15.81:0/512321452 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a000106c0 con 0x7f3a0c074460 2024-06-29T14:34:51.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.647+0000 7f3a1203d640 1 -- 172.21.15.81:0/512321452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0c151250 con 0x7f3a0c074460 2024-06-29T14:34:51.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.647+0000 7f3a1203d640 1 -- 172.21.15.81:0/512321452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a0c1535f0 con 0x7f3a0c074460 2024-06-29T14:34:51.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.648+0000 7f3a1203d640 1 -- 172.21.15.81:0/512321452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a0c15c410 con 0x7f3a0c074460 2024-06-29T14:34:51.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.648+0000 7f39f3fff640 1 -- 172.21.15.81:0/512321452 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f3a0001a070 con 0x7f3a0c074460 2024-06-29T14:34:51.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.649+0000 7f39f3fff640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f39f8076420 0x7f39f8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.649+0000 7f39f3fff640 1 -- 172.21.15.81:0/512321452 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5632+0+0 (secure 0 0 0) 0x7f3a00097a30 con 0x7f3a0c074460 2024-06-29T14:34:51.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.649+0000 7f3a0b7fe640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f39f8076420 0x7f39f8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.650+0000 7f3a0b7fe640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f39f8076420 0x7f39f8078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f39fc033000 tx=0x7f39fc004730 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:51.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.653+0000 7f39f3fff640 1 -- 172.21.15.81:0/512321452 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3a0c15c410 con 0x7f3a0c074460 2024-06-29T14:34:51.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.830+0000 7f3a1203d640 1 -- 172.21.15.81:0/512321452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3a0c0aea70 con 0x7f3a0c074460 2024-06-29T14:34:51.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.831+0000 7f39f3fff640 1 -- 172.21.15.81:0/512321452 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) ==== 93+0+10 (secure 0 0 0) 0x7f3a0000de60 con 0x7f3a0c074460 2024-06-29T14:34:51.830 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:34:51.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 -- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f39f8076420 msgr2=0x7f39f8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f39f8076420 0x7f39f8078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f39fc033000 tx=0x7f39fc004730 comp rx=0 tx=0).stop 2024-06-29T14:34:51.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 -- 172.21.15.81:0/512321452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a0c074460 msgr2=0x7f3a0c145470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a0c074460 0x7f3a0c145470 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000d750 tx=0x7f3a0000dc10 comp rx=0 tx=0).stop 2024-06-29T14:34:51.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 -- 172.21.15.81:0/512321452 shutdown_connections 2024-06-29T14:34:51.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a0c0ad0c0 0x7f3a0c0b1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f39f8076420 0x7f39f8078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a0c074460 0x7f3a0c145470 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 --2- 172.21.15.81:0/512321452 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a0c073b10 0x7f3a0c150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 -- 172.21.15.81:0/512321452 >> 172.21.15.81:0/512321452 conn(0x7f3a0c072950 msgr2=0x7f3a0c156610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:51.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 -- 172.21.15.81:0/512321452 shutdown_connections 2024-06-29T14:34:51.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.833+0000 7f3a1203d640 1 -- 172.21.15.81:0/512321452 wait complete. 2024-06-29T14:34:51.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-06-29T14:34:51.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.957+0000 7f7591d31640 1 -- 172.21.15.81:0/1112339485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f758c0b6d60 msgr2=0x7f758c0ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.957+0000 7f7591d31640 1 --2- 172.21.15.81:0/1112339485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f758c0b6d60 0x7f758c0ce4d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f757c005fd0 tx=0x7f757c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.957+0000 7f7591d31640 1 -- 172.21.15.81:0/1112339485 shutdown_connections 2024-06-29T14:34:51.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.957+0000 7f7591d31640 1 --2- 172.21.15.81:0/1112339485 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f758c0cea10 0x7f758c0b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.957+0000 7f7591d31640 1 --2- 172.21.15.81:0/1112339485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f758c0b6d60 0x7f758c0ce4d0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.957+0000 7f7591d31640 1 -- 172.21.15.81:0/1112339485 >> 172.21.15.81:0/1112339485 conn(0x7f758c0d1960 msgr2=0x7f758c0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:51.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.957+0000 7f7591d31640 1 -- 172.21.15.81:0/1112339485 shutdown_connections 2024-06-29T14:34:51.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.957+0000 7f7591d31640 1 -- 172.21.15.81:0/1112339485 wait complete. 2024-06-29T14:34:51.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.959+0000 7f7591d31640 1 Processor -- start 2024-06-29T14:34:51.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.959+0000 7f7591d31640 1 -- start start 2024-06-29T14:34:51.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.960+0000 7f7591d31640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f758c0b4cf0 0x7f758c145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.960+0000 7f7591d31640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f758c0b6d60 0x7f758c159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.960+0000 7f7591d31640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f758c0cea10 0x7f758c163780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.960+0000 7f7591d31640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f758c0cd540 con 0x7f758c0cea10 2024-06-29T14:34:51.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.960+0000 7f7591d31640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f758c0d0a10 con 0x7f758c0b4cf0 2024-06-29T14:34:51.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.960+0000 7f7591d31640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f758c0cd6c0 con 0x7f758c0b6d60 2024-06-29T14:34:51.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.960+0000 7f758bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f758c0cea10 0x7f758c163780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.960+0000 7f758bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f758c0cea10 0x7f758c163780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60206/0 (socket says 172.21.15.81:60206) 2024-06-29T14:34:51.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.960+0000 7f758bfff640 1 -- 172.21.15.81:0/3097684363 learned_addr learned my addr 172.21.15.81:0/3097684363 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:51.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f758bfff640 1 -- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f758c0b6d60 msgr2=0x7f758c159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f758bfff640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f758c0b6d60 0x7f758c159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f758b7fe640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f758c0b4cf0 0x7f758c145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f758bfff640 1 -- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f758c0b4cf0 msgr2=0x7f758c145550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:51.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f758bfff640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f758c0b4cf0 0x7f758c145550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:51.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f758bfff640 1 -- 172.21.15.81:0/3097684363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758c162ff0 con 0x7f758c0cea10 2024-06-29T14:34:51.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f758bfff640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f758c0cea10 0x7f758c163780 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f758400c730 tx=0x7f758400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:51.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f7583fff640 1 -- 172.21.15.81:0/3097684363 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7584003b70 con 0x7f758c0cea10 2024-06-29T14:34:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f7583fff640 1 -- 172.21.15.81:0/3097684363 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7584003d10 con 0x7f758c0cea10 2024-06-29T14:34:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f7591d31640 1 -- 172.21.15.81:0/3097684363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f758c145a90 con 0x7f758c0cea10 2024-06-29T14:34:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f7583fff640 1 -- 172.21.15.81:0/3097684363 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7584005330 con 0x7f758c0cea10 2024-06-29T14:34:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.961+0000 7f7591d31640 1 -- 172.21.15.81:0/3097684363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f758c15aad0 con 0x7f758c0cea10 2024-06-29T14:34:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.962+0000 7f7591d31640 1 -- 172.21.15.81:0/3097684363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f758c0b6510 con 0x7f758c0cea10 2024-06-29T14:34:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.963+0000 7f7583fff640 1 -- 172.21.15.81:0/3097684363 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f75840054d0 con 0x7f758c0cea10 2024-06-29T14:34:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.963+0000 7f7583fff640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f756c076420 0x7f756c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.963+0000 7f7583fff640 1 -- 172.21.15.81:0/3097684363 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5632+0+0 (secure 0 0 0) 0x7f75840977a0 con 0x7f758c0cea10 2024-06-29T14:34:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.963+0000 7f758b7fe640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f756c076420 0x7f756c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:51.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.964+0000 7f758b7fe640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f756c076420 0x7f756c078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f757c033000 tx=0x7f757c004730 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:51.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:51.967+0000 7f7583fff640 1 -- 172.21.15.81:0/3097684363 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f758c0b6510 con 0x7f758c0cea10 2024-06-29T14:34:52.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:52.222+0000 7f7591d31640 1 -- 172.21.15.81:0/3097684363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) -- 0x7f758c0ce4d0 con 0x7f758c0cea10 2024-06-29T14:34:53.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.202+0000 7f7583fff640 1 -- 172.21.15.81:0/3097684363 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v438) ==== 157+0+0 (secure 0 0 0) 0x7f7584060f30 con 0x7f758c0cea10 2024-06-29T14:34:53.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 -- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f756c076420 msgr2=0x7f756c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f756c076420 0x7f756c078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f757c033000 tx=0x7f757c004730 comp rx=0 tx=0).stop 2024-06-29T14:34:53.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 -- 172.21.15.81:0/3097684363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f758c0cea10 msgr2=0x7f758c163780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f758c0cea10 0x7f758c163780 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f758400c730 tx=0x7f758400cb30 comp rx=0 tx=0).stop 2024-06-29T14:34:53.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 -- 172.21.15.81:0/3097684363 shutdown_connections 2024-06-29T14:34:53.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f756c076420 0x7f756c078930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f758c0cea10 0x7f758c163780 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f758c0b6d60 0x7f758c159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 --2- 172.21.15.81:0/3097684363 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f758c0b4cf0 0x7f758c145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 -- 172.21.15.81:0/3097684363 >> 172.21.15.81:0/3097684363 conn(0x7f758c0d1960 msgr2=0x7f758c165e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:53.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 -- 172.21.15.81:0/3097684363 shutdown_connections 2024-06-29T14:34:53.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.205+0000 7f7591d31640 1 -- 172.21.15.81:0/3097684363 wait complete. 2024-06-29T14:34:53.217 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:34:53.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.338+0000 7f109ad7a640 1 -- 172.21.15.81:0/3123784284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10940cd480 msgr2=0x7f10940ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.338+0000 7f109ad7a640 1 --2- 172.21.15.81:0/3123784284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10940cd480 0x7f10940ce510 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1084009830 tx=0x7f108402a480 comp rx=0 tx=0).stop 2024-06-29T14:34:53.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.338+0000 7f109ad7a640 1 -- 172.21.15.81:0/3123784284 shutdown_connections 2024-06-29T14:34:53.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.338+0000 7f109ad7a640 1 --2- 172.21.15.81:0/3123784284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10940cd480 0x7f10940ce510 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.338+0000 7f109ad7a640 1 -- 172.21.15.81:0/3123784284 >> 172.21.15.81:0/3123784284 conn(0x7f10940b3510 msgr2=0x7f10940b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:53.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.338+0000 7f109ad7a640 1 -- 172.21.15.81:0/3123784284 shutdown_connections 2024-06-29T14:34:53.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.338+0000 7f109ad7a640 1 -- 172.21.15.81:0/3123784284 wait complete. 2024-06-29T14:34:53.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.340+0000 7f109ad7a640 1 Processor -- start 2024-06-29T14:34:53.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.340+0000 7f109ad7a640 1 -- start start 2024-06-29T14:34:53.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f109ad7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10940b6eb0 0x7f1094149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f109ad7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10940cd480 0x7f10941f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f109ad7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10940cea50 0x7f1094161ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f109ad7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f10940b52b0 con 0x7f10940b6eb0 2024-06-29T14:34:53.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f109ad7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f10940b5130 con 0x7f10940cea50 2024-06-29T14:34:53.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f109ad7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1094167660 con 0x7f10940cd480 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f1098b16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10940b6eb0 0x7f1094149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f1098b16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10940b6eb0 0x7f1094149a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60222/0 (socket says 172.21.15.81:60222) 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f1098b16640 1 -- 172.21.15.81:0/2866372275 learned_addr learned my addr 172.21.15.81:0/2866372275 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f1099317640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10940cea50 0x7f1094161ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.341+0000 7f108bfff640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10940cd480 0x7f10941f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f1098b16640 1 -- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10940cd480 msgr2=0x7f10941f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f1098b16640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10940cd480 0x7f10941f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f1098b16640 1 -- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10940cea50 msgr2=0x7f1094161ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f1098b16640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10940cea50 0x7f1094161ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f1098b16640 1 -- 172.21.15.81:0/2866372275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1094149fa0 con 0x7f10940b6eb0 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f1098b16640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10940b6eb0 0x7f1094149a60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f109000c3c0 tx=0x7f109000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f10897fa640 1 -- 172.21.15.81:0/2866372275 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1090017070 con 0x7f10940b6eb0 2024-06-29T14:34:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f10897fa640 1 -- 172.21.15.81:0/2866372275 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10900035f0 con 0x7f10940b6eb0 2024-06-29T14:34:53.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f109ad7a640 1 -- 172.21.15.81:0/2866372275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10940ae7f0 con 0x7f10940b6eb0 2024-06-29T14:34:53.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f10897fa640 1 -- 172.21.15.81:0/2866372275 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1090003a40 con 0x7f10940b6eb0 2024-06-29T14:34:53.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.342+0000 7f109ad7a640 1 -- 172.21.15.81:0/2866372275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10941f3d20 con 0x7f10940b6eb0 2024-06-29T14:34:53.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.343+0000 7f109ad7a640 1 -- 172.21.15.81:0/2866372275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10940ce510 con 0x7f10940b6eb0 2024-06-29T14:34:53.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.344+0000 7f10897fa640 1 -- 172.21.15.81:0/2866372275 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f109001a070 con 0x7f10940b6eb0 2024-06-29T14:34:53.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.344+0000 7f10897fa640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f10800763f0 0x7f1080078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.344+0000 7f10897fa640 1 -- 172.21.15.81:0/2866372275 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5632+0+0 (secure 0 0 0) 0x7f1090096c50 con 0x7f10940b6eb0 2024-06-29T14:34:53.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.344+0000 7f108bfff640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f10800763f0 0x7f1080078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.345+0000 7f108bfff640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f10800763f0 0x7f1080078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1084034000 tx=0x7f1084004730 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:53.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.349+0000 7f10897fa640 1 -- 172.21.15.81:0/2866372275 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f109009c020 con 0x7f10940b6eb0 2024-06-29T14:34:53.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.529+0000 7f109ad7a640 1 -- 172.21.15.81:0/2866372275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f10940cdf00 con 0x7f10940b6eb0 2024-06-29T14:34:53.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.529+0000 7f10897fa640 1 -- 172.21.15.81:0/2866372275 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v31) ==== 59+0+1998 (secure 0 0 0) 0x7f1090060460 con 0x7f10940b6eb0 2024-06-29T14:34:53.528 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:34:53.528 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:34:53.528 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:34:53.528 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:34:53.528 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T14:34:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T14:34:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T14:34:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T14:34:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T14:34:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T14:34:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T14:34:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T14:34:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T14:34:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:34:53.529 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T14:34:53.530 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:34:53.531 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 -- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f10800763f0 msgr2=0x7f1080078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f10800763f0 0x7f1080078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1084034000 tx=0x7f1084004730 comp rx=0 tx=0).stop 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 -- 172.21.15.81:0/2866372275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10940b6eb0 msgr2=0x7f1094149a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10940b6eb0 0x7f1094149a60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f109000c3c0 tx=0x7f109000c880 comp rx=0 tx=0).stop 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 -- 172.21.15.81:0/2866372275 shutdown_connections 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10940cea50 0x7f1094161ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10940cd480 0x7f10941f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f10800763f0 0x7f1080078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 --2- 172.21.15.81:0/2866372275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10940b6eb0 0x7f1094149a60 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 -- 172.21.15.81:0/2866372275 >> 172.21.15.81:0/2866372275 conn(0x7f10940b3510 msgr2=0x7f1094070970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 -- 172.21.15.81:0/2866372275 shutdown_connections 2024-06-29T14:34:53.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.532+0000 7f109ad7a640 1 -- 172.21.15.81:0/2866372275 wait complete. 2024-06-29T14:34:53.541 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:34:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.659+0000 7fe29d017640 1 -- 172.21.15.81:0/1990254969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2980b3200 msgr2=0x7fe2980cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.659+0000 7fe29d017640 1 --2- 172.21.15.81:0/1990254969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2980b3200 0x7fe2980cc7b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe2980c5590 tx=0x7fe28802a820 comp rx=0 tx=0).stop 2024-06-29T14:34:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.659+0000 7fe29d017640 1 -- 172.21.15.81:0/1990254969 shutdown_connections 2024-06-29T14:34:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.659+0000 7fe29d017640 1 --2- 172.21.15.81:0/1990254969 >> v2:172.21.15.167:3300/0 conn(0x7fe2980d0940 0x7fe2980b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.659+0000 7fe29d017640 1 --2- 172.21.15.81:0/1990254969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2980b3200 0x7fe2980cc7b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.659+0000 7fe29d017640 1 -- 172.21.15.81:0/1990254969 >> 172.21.15.81:0/1990254969 conn(0x7fe2980b3680 msgr2=0x7fe2980b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.659+0000 7fe29d017640 1 -- 172.21.15.81:0/1990254969 shutdown_connections 2024-06-29T14:34:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.659+0000 7fe29d017640 1 -- 172.21.15.81:0/1990254969 wait complete. 2024-06-29T14:34:53.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.661+0000 7fe29d017640 1 Processor -- start 2024-06-29T14:34:53.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.661+0000 7fe29d017640 1 -- start start 2024-06-29T14:34:53.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.662+0000 7fe29d017640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2980b3200 0x7fe2981e2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.662+0000 7fe29d017640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2980cccf0 0x7fe2980c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.662+0000 7fe29d017640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2981e5330 0x7fe2981e5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.662+0000 7fe29d017640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe2980b5b70 con 0x7fe2980b3200 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.662+0000 7fe29d017640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe2980b59f0 con 0x7fe2981e5330 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe29d017640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe2980b5cf0 con 0x7fe2980cccf0 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe296d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2980b3200 0x7fe2981e2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe296d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2980b3200 0x7fe2981e2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60244/0 (socket says 172.21.15.81:60244) 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe296d9d640 1 -- 172.21.15.81:0/3411200322 learned_addr learned my addr 172.21.15.81:0/3411200322 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe29759e640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2981e5330 0x7fe2981e5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe29659c640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2980cccf0 0x7fe2980c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe296d9d640 1 -- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2980cccf0 msgr2=0x7fe2980c82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe296d9d640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2980cccf0 0x7fe2980c82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe296d9d640 1 -- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2981e5330 msgr2=0x7fe2981e5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe296d9d640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2981e5330 0x7fe2981e5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe296d9d640 1 -- 172.21.15.81:0/3411200322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2981e5c80 con 0x7fe2980b3200 2024-06-29T14:34:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe296d9d640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2980b3200 0x7fe2981e2b70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe288009ca0 tx=0x7fe288032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe27f7fe640 1 -- 172.21.15.81:0/3411200322 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288037040 con 0x7fe2980b3200 2024-06-29T14:34:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe27f7fe640 1 -- 172.21.15.81:0/3411200322 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe288042070 con 0x7fe2980b3200 2024-06-29T14:34:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe29d017640 1 -- 172.21.15.81:0/3411200322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2981f2a90 con 0x7fe2980b3200 2024-06-29T14:34:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe27f7fe640 1 -- 172.21.15.81:0/3411200322 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288004280 con 0x7fe2980b3200 2024-06-29T14:34:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.663+0000 7fe29d017640 1 -- 172.21.15.81:0/3411200322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2981f3050 con 0x7fe2980b3200 2024-06-29T14:34:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.664+0000 7fe29d017640 1 -- 172.21.15.81:0/3411200322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2980cc7b0 con 0x7fe2980b3200 2024-06-29T14:34:53.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.665+0000 7fe27f7fe640 1 -- 172.21.15.81:0/3411200322 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fe288004420 con 0x7fe2980b3200 2024-06-29T14:34:53.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.665+0000 7fe27f7fe640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fe2840763f0 0x7fe284078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:53.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.666+0000 7fe27f7fe640 1 -- 172.21.15.81:0/3411200322 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5632+0+0 (secure 0 0 0) 0x7fe2880b8910 con 0x7fe2980b3200 2024-06-29T14:34:53.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.666+0000 7fe29659c640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fe2840763f0 0x7fe284078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:53.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.666+0000 7fe29659c640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fe2840763f0 0x7fe284078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe28c000f60 tx=0x7fe28c009040 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:53.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.670+0000 7fe27f7fe640 1 -- 172.21.15.81:0/3411200322 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe2981e5c80 con 0x7fe2980b3200 2024-06-29T14:34:53.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.849+0000 7fe29d017640 1 -- 172.21.15.81:0/3411200322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe2981e3310 con 0x7fe2980b3200 2024-06-29T14:34:53.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.852+0000 7fe27f7fe640 1 -- 172.21.15.81:0/3411200322 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7fe288082010 con 0x7fe2980b3200 2024-06-29T14:34:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 -- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fe2840763f0 msgr2=0x7fe284078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fe2840763f0 0x7fe284078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe28c000f60 tx=0x7fe28c009040 comp rx=0 tx=0).stop 2024-06-29T14:34:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 -- 172.21.15.81:0/3411200322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2980b3200 msgr2=0x7fe2981e2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2980b3200 0x7fe2981e2b70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe288009ca0 tx=0x7fe288032d10 comp rx=0 tx=0).stop 2024-06-29T14:34:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 -- 172.21.15.81:0/3411200322 shutdown_connections 2024-06-29T14:34:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2981e5330 0x7fe2981e5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe2980cccf0 0x7fe2980c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fe2840763f0 0x7fe284078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 --2- 172.21.15.81:0/3411200322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2980b3200 0x7fe2981e2b70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 -- 172.21.15.81:0/3411200322 >> 172.21.15.81:0/3411200322 conn(0x7fe2980b3680 msgr2=0x7fe2981e41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 -- 172.21.15.81:0/3411200322 shutdown_connections 2024-06-29T14:34:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:53.855+0000 7fe29d017640 1 -- 172.21.15.81:0/3411200322 wait complete. 2024-06-29T14:34:53.864 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:34:53.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:34:53.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:34:53.954 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:34:53.954 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:34:53.954 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:53.970 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:34:53.970 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:34:53.970 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:53.987 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:34:53.987 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:34:53.987 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:54.025 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:34:54.025 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:34:54.025 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:34:54.044 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:34:54.044 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:34:54.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:34:54.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.179+0000 7f9f9b64d640 1 -- 172.21.15.81:0/789218678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f9406f850 msgr2=0x7f9f940ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.179+0000 7f9f9b64d640 1 --2- 172.21.15.81:0/789218678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f9406f850 0x7f9f940ae300 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009830 tx=0x7f9f8402a510 comp rx=0 tx=0).stop 2024-06-29T14:34:54.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.179+0000 7f9f9b64d640 1 -- 172.21.15.81:0/789218678 shutdown_connections 2024-06-29T14:34:54.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.179+0000 7f9f9b64d640 1 --2- 172.21.15.81:0/789218678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f940ae840 0x7f9f940aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.179+0000 7f9f9b64d640 1 --2- 172.21.15.81:0/789218678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f9406f850 0x7f9f940ae300 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.179+0000 7f9f9b64d640 1 -- 172.21.15.81:0/789218678 >> 172.21.15.81:0/789218678 conn(0x7f9f94072950 msgr2=0x7f9f940b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:54.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.179+0000 7f9f9b64d640 1 -- 172.21.15.81:0/789218678 shutdown_connections 2024-06-29T14:34:54.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.179+0000 7f9f9b64d640 1 -- 172.21.15.81:0/789218678 wait complete. 2024-06-29T14:34:54.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.181+0000 7f9f9b64d640 1 Processor -- start 2024-06-29T14:34:54.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.181+0000 7f9f9b64d640 1 -- start start 2024-06-29T14:34:54.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.182+0000 7f9f9b64d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f9406f850 0x7f9f9414f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.182+0000 7f9f9b64d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f94074520 0x7f9f94156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.182+0000 7f9f9b64d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f940ae840 0x7f9f94156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.182+0000 7f9f9b64d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9415b190 con 0x7f9f940ae840 2024-06-29T14:34:54.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.182+0000 7f9f9b64d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9415b010 con 0x7f9f94074520 2024-06-29T14:34:54.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.182+0000 7f9f9b64d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9f940af190 con 0x7f9f9406f850 2024-06-29T14:34:54.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f99bea640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f940ae840 0x7f9f94156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f99bea640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f940ae840 0x7f9f94156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60254/0 (socket says 172.21.15.81:60254) 2024-06-29T14:34:54.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f99bea640 1 -- 172.21.15.81:0/416273869 learned_addr learned my addr 172.21.15.81:0/416273869 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:54.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f993e9640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f9406f850 0x7f9f9414f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f98be8640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f94074520 0x7f9f94156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f99bea640 1 -- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f9406f850 msgr2=0x7f9f9414f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f99bea640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f9406f850 0x7f9f9414f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f99bea640 1 -- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f94074520 msgr2=0x7f9f94156810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f99bea640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f94074520 0x7f9f94156810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f99bea640 1 -- 172.21.15.81:0/416273869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f94151d90 con 0x7f9f940ae840 2024-06-29T14:34:54.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f99bea640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f940ae840 0x7f9f94156d50 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c00ebf0 tx=0x7f9f8c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:54.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f89ffb640 1 -- 172.21.15.81:0/416273869 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c003b70 con 0x7f9f940ae840 2024-06-29T14:34:54.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f89ffb640 1 -- 172.21.15.81:0/416273869 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f8c003d10 con 0x7f9f940ae840 2024-06-29T14:34:54.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f9b64d640 1 -- 172.21.15.81:0/416273869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f94152000 con 0x7f9f940ae840 2024-06-29T14:34:54.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f89ffb640 1 -- 172.21.15.81:0/416273869 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c004280 con 0x7f9f940ae840 2024-06-29T14:34:54.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.183+0000 7f9f9b64d640 1 -- 172.21.15.81:0/416273869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f941525c0 con 0x7f9f940ae840 2024-06-29T14:34:54.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.184+0000 7f9f9b64d640 1 -- 172.21.15.81:0/416273869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f941572f0 con 0x7f9f940ae840 2024-06-29T14:34:54.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.185+0000 7f9f89ffb640 1 -- 172.21.15.81:0/416273869 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f9f8c004420 con 0x7f9f940ae840 2024-06-29T14:34:54.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.185+0000 7f9f89ffb640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9f780763f0 0x7f9f78078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.186+0000 7f9f89ffb640 1 -- 172.21.15.81:0/416273869 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5632+0+0 (secure 0 0 0) 0x7f9f8c098630 con 0x7f9f940ae840 2024-06-29T14:34:54.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.186+0000 7f9f993e9640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9f780763f0 0x7f9f78078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.186+0000 7f9f993e9640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9f780763f0 0x7f9f78078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f900015e0 tx=0x7f9f90009040 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:54.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.190+0000 7f9f89ffb640 1 -- 172.21.15.81:0/416273869 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9f8c061d30 con 0x7f9f940ae840 2024-06-29T14:34:54.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.405+0000 7f9f9b64d640 1 -- 172.21.15.81:0/416273869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f940ae300 con 0x7f9f940ae840 2024-06-29T14:34:54.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.406+0000 7f9f89ffb640 1 -- 172.21.15.81:0/416273869 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+3914 (secure 0 0 0) 0x7f9f8c061630 con 0x7f9f940ae840 2024-06-29T14:34:54.405 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:54.406 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":18,"btime":"2024-06-29T14:34:48:015199+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3513859413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3513859413},{"type":"v1","addr":"172.21.15.81:6835","nonce":3513859413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4775,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/763942647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":763942647},{"type":"v1","addr":"172.21.15.167:6836","nonce":763942647}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4966,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1148290480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1148290480},{"type":"v1","addr":"172.21.15.167:6837","nonce":1148290480}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5949,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3836783219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3836783219},{"type":"v1","addr":"172.21.15.81:6834","nonce":3836783219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[]} 2024-06-29T14:34:54.407 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 18 2024-06-29T14:34:54.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 -- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9f780763f0 msgr2=0x7f9f78078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9f780763f0 0x7f9f78078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f900015e0 tx=0x7f9f90009040 comp rx=0 tx=0).stop 2024-06-29T14:34:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 -- 172.21.15.81:0/416273869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f940ae840 msgr2=0x7f9f94156d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f940ae840 0x7f9f94156d50 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c00ebf0 tx=0x7f9f8c00c6a0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 -- 172.21.15.81:0/416273869 shutdown_connections 2024-06-29T14:34:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9f780763f0 0x7f9f78078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f940ae840 0x7f9f94156d50 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f94074520 0x7f9f94156810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 --2- 172.21.15.81:0/416273869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f9406f850 0x7f9f9414f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 -- 172.21.15.81:0/416273869 >> 172.21.15.81:0/416273869 conn(0x7f9f94072950 msgr2=0x7f9f940b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:54.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 -- 172.21.15.81:0/416273869 shutdown_connections 2024-06-29T14:34:54.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.408+0000 7f9f9b64d640 1 -- 172.21.15.81:0/416273869 wait complete. 2024-06-29T14:34:54.418 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:34:54.419 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:34:54.419 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:34:54.419 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:34:54.420 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:34:54.420 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:34:54.420 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:34:54.420 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:34:54.421 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:34:54.421 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:34:54.421 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:34:54.421 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:34:54.422 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:34:54.422 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:34:54.422 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:34:54.423 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:34:54.431 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:34:54.431 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:34:54.433 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:34:54.433 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:34:54.437 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:34:54.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:34:54.439 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:34:54.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:34:54.474 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:34:54.474 DEBUG:teuthology.parallel:result is None 2024-06-29T14:34:54.475 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:34:54.475 DEBUG:teuthology.parallel:result is None 2024-06-29T14:34:54.480 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:34:54.481 DEBUG:teuthology.parallel:result is None 2024-06-29T14:34:54.481 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:34:54.482 DEBUG:teuthology.parallel:result is None 2024-06-29T14:34:54.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:34:54.560 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:34:54.560 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T14:34:54.572 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:34:54.572 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T14:34:54.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.603+0000 7f0d553ba640 1 -- 172.21.15.81:0/3379474949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d500d2950 msgr2=0x7f0d500b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.603+0000 7f0d553ba640 1 --2- 172.21.15.81:0/3379474949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d500d2950 0x7f0d500b3b00 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0d48005fd0 tx=0x7f0d4802aa10 comp rx=0 tx=0).stop 2024-06-29T14:34:54.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.603+0000 7f0d553ba640 1 -- 172.21.15.81:0/3379474949 shutdown_connections 2024-06-29T14:34:54.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.603+0000 7f0d553ba640 1 --2- 172.21.15.81:0/3379474949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d500d2950 0x7f0d500b3b00 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.603+0000 7f0d553ba640 1 --2- 172.21.15.81:0/3379474949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d500ce770 0x7f0d500b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.603+0000 7f0d553ba640 1 -- 172.21.15.81:0/3379474949 >> 172.21.15.81:0/3379474949 conn(0x7f0d500d3c60 msgr2=0x7f0d500d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:54.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.603+0000 7f0d553ba640 1 -- 172.21.15.81:0/3379474949 shutdown_connections 2024-06-29T14:34:54.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.603+0000 7f0d553ba640 1 -- 172.21.15.81:0/3379474949 wait complete. 2024-06-29T14:34:54.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.605+0000 7f0d553ba640 1 Processor -- start 2024-06-29T14:34:54.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.605+0000 7f0d553ba640 1 -- start start 2024-06-29T14:34:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d553ba640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d500ce770 0x7f0d501e9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d553ba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d500ceb80 0x7f0d501e4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d553ba640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d500d2950 0x7f0d500b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d553ba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0d500d15a0 con 0x7f0d500ceb80 2024-06-29T14:34:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d553ba640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0d500d1420 con 0x7f0d500d2950 2024-06-29T14:34:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d553ba640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0d500d4990 con 0x7f0d500ce770 2024-06-29T14:34:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d47fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d500ceb80 0x7f0d501e4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d47fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d500ceb80 0x7f0d501e4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60308/0 (socket says 172.21.15.81:60308) 2024-06-29T14:34:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d47fff640 1 -- 172.21.15.81:0/907038083 learned_addr learned my addr 172.21.15.81:0/907038083 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d4effd640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d500ce770 0x7f0d501e9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d4f7fe640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d500d2950 0x7f0d500b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d47fff640 1 -- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d500ce770 msgr2=0x7f0d501e9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d47fff640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d500ce770 0x7f0d501e9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d47fff640 1 -- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d500d2950 msgr2=0x7f0d500b7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d47fff640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d500d2950 0x7f0d500b7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d47fff640 1 -- 172.21.15.81:0/907038083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d501e7f70 con 0x7f0d500ceb80 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d47fff640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d500ceb80 0x7f0d501e4b50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800c370 tx=0x7f0d3800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d477fe640 1 -- 172.21.15.81:0/907038083 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38017070 con 0x7f0d500ceb80 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d477fe640 1 -- 172.21.15.81:0/907038083 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d380053c0 con 0x7f0d500ceb80 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d553ba640 1 -- 172.21.15.81:0/907038083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d501e8250 con 0x7f0d500ceb80 2024-06-29T14:34:54.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d477fe640 1 -- 172.21.15.81:0/907038083 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38003850 con 0x7f0d500ceb80 2024-06-29T14:34:54.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.607+0000 7f0d553ba640 1 -- 172.21.15.81:0/907038083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d501e8810 con 0x7f0d500ceb80 2024-06-29T14:34:54.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.608+0000 7f0d553ba640 1 -- 172.21.15.81:0/907038083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d501e7ce0 con 0x7f0d500ceb80 2024-06-29T14:34:54.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.609+0000 7f0d477fe640 1 -- 172.21.15.81:0/907038083 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f0d3801a070 con 0x7f0d500ceb80 2024-06-29T14:34:54.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.610+0000 7f0d477fe640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0d30076420 0x7f0d30078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:54.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.610+0000 7f0d477fe640 1 -- 172.21.15.81:0/907038083 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5632+0+0 (secure 0 0 0) 0x7f0d38096ae0 con 0x7f0d500ceb80 2024-06-29T14:34:54.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.610+0000 7f0d4effd640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0d30076420 0x7f0d30078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:54.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.610+0000 7f0d4effd640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0d30076420 0x7f0d30078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0d500d0120 tx=0x7f0d4000b2e0 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:54.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.614+0000 7f0d477fe640 1 -- 172.21.15.81:0/907038083 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0d3805fd30 con 0x7f0d500ceb80 2024-06-29T14:34:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:54.804+0000 7f0d553ba640 1 -- 172.21.15.81:0/907038083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f0d500d2240 con 0x7f0d500ceb80 2024-06-29T14:34:55.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.213+0000 7f0d477fe640 1 -- 172.21.15.81:0/907038083 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v55) ==== 97+0+0 (secure 0 0 0) 0x7f0d38068370 con 0x7f0d500ceb80 2024-06-29T14:34:55.211 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 -- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0d30076420 msgr2=0x7f0d30078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0d30076420 0x7f0d30078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0d500d0120 tx=0x7f0d4000b2e0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 -- 172.21.15.81:0/907038083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d500ceb80 msgr2=0x7f0d501e4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d500ceb80 0x7f0d501e4b50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800c370 tx=0x7f0d3800c830 comp rx=0 tx=0).stop 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 -- 172.21.15.81:0/907038083 shutdown_connections 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d500d2950 0x7f0d500b7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0d30076420 0x7f0d30078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d500ceb80 0x7f0d501e4b50 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 --2- 172.21.15.81:0/907038083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d500ce770 0x7f0d501e9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 -- 172.21.15.81:0/907038083 >> 172.21.15.81:0/907038083 conn(0x7f0d500d3c60 msgr2=0x7f0d500cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 -- 172.21.15.81:0/907038083 shutdown_connections 2024-06-29T14:34:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.216+0000 7f0d553ba640 1 -- 172.21.15.81:0/907038083 wait complete. 2024-06-29T14:34:55.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:34:55.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.342+0000 7f11ae19e640 1 -- 172.21.15.81:0/2558695840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11a80d0b90 msgr2=0x7f11a80d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.342+0000 7f11ae19e640 1 --2- 172.21.15.81:0/2558695840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11a80d0b90 0x7f11a80d1a80 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f119c009830 tx=0x7f119c02a5a0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.342+0000 7f11ae19e640 1 -- 172.21.15.81:0/2558695840 shutdown_connections 2024-06-29T14:34:55.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.342+0000 7f11ae19e640 1 --2- 172.21.15.81:0/2558695840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11a80d0b90 0x7f11a80d1a80 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.342+0000 7f11ae19e640 1 -- 172.21.15.81:0/2558695840 >> 172.21.15.81:0/2558695840 conn(0x7f11a809cf30 msgr2=0x7f11a80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:55.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.342+0000 7f11ae19e640 1 -- 172.21.15.81:0/2558695840 shutdown_connections 2024-06-29T14:34:55.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.342+0000 7f11ae19e640 1 -- 172.21.15.81:0/2558695840 wait complete. 2024-06-29T14:34:55.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.344+0000 7f11ae19e640 1 Processor -- start 2024-06-29T14:34:55.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.344+0000 7f11ae19e640 1 -- start start 2024-06-29T14:34:55.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11ae19e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f11a80b5250 0x7f11a81589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11ae19e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f11a80b9040 0x7f11a8155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11ae19e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11a809d8a0 0x7f11a809dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11ae19e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f11a80d26d0 con 0x7f11a809d8a0 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11ae19e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f11a80b85c0 con 0x7f11a80b9040 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11ae19e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f11a80d2850 con 0x7f11a80b5250 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11a809d8a0 0x7f11a809dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11a809d8a0 0x7f11a809dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60336/0 (socket says 172.21.15.81:60336) 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a7fff640 1 -- 172.21.15.81:0/3427530853 learned_addr learned my addr 172.21.15.81:0/3427530853 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a77fe640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f11a80b5250 0x7f11a81589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a6ffd640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f11a80b9040 0x7f11a8155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a7fff640 1 -- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f11a80b5250 msgr2=0x7f11a81589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a7fff640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f11a80b5250 0x7f11a81589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a7fff640 1 -- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f11a80b9040 msgr2=0x7f11a8155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a7fff640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f11a80b9040 0x7f11a8155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a7fff640 1 -- 172.21.15.81:0/3427530853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a809e1f0 con 0x7f11a809d8a0 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11a7fff640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11a809d8a0 0x7f11a809dcb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f11a000b440 tx=0x7f11a000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:55.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f118ffff640 1 -- 172.21.15.81:0/3427530853 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a0015020 con 0x7f11a809d8a0 2024-06-29T14:34:55.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f118ffff640 1 -- 172.21.15.81:0/3427530853 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11a0011920 con 0x7f11a809d8a0 2024-06-29T14:34:55.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f11ae19e640 1 -- 172.21.15.81:0/3427530853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a80a1880 con 0x7f11a809d8a0 2024-06-29T14:34:55.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.346+0000 7f118ffff640 1 -- 172.21.15.81:0/3427530853 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a0011d70 con 0x7f11a809d8a0 2024-06-29T14:34:55.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.347+0000 7f11ae19e640 1 -- 172.21.15.81:0/3427530853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a8157040 con 0x7f11a809d8a0 2024-06-29T14:34:55.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.347+0000 7f11ae19e640 1 -- 172.21.15.81:0/3427530853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a80d2de0 con 0x7f11a809d8a0 2024-06-29T14:34:55.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.348+0000 7f118ffff640 1 -- 172.21.15.81:0/3427530853 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f11a0018020 con 0x7f11a809d8a0 2024-06-29T14:34:55.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.349+0000 7f118ffff640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1194076420 0x7f1194078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.349+0000 7f118ffff640 1 -- 172.21.15.81:0/3427530853 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 4901+0+0 (secure 0 0 0) 0x7f11a0097450 con 0x7f11a809d8a0 2024-06-29T14:34:55.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.349+0000 7f11a77fe640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1194076420 0x7f1194078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.350+0000 7f11a77fe640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1194076420 0x7f1194078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f11980012e0 tx=0x7f1198009040 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:55.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.353+0000 7f118ffff640 1 -- 172.21.15.81:0/3427530853 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f11a809e1f0 con 0x7f11a809d8a0 2024-06-29T14:34:55.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.601+0000 7f11ae19e640 1 -- 172.21.15.81:0/3427530853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f11a8156dd0 con 0x7f11a809d8a0 2024-06-29T14:34:55.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.601+0000 7f118ffff640 1 -- 172.21.15.81:0/3427530853 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v29) ==== 80+0+5304 (secure 0 0 0) 0x7f11a0060ae0 con 0x7f11a809d8a0 2024-06-29T14:34:55.601 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:55.601 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:34:55.601 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T14:34:55.601 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.602 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T14:34:55.602 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T14:34:55.602 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.602 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:34:55.602 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:34:55.603 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:34:55.603 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.603 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.603 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.603 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.603 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T14:34:55.604 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T14:34:55.604 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.604 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:34:55.604 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:34:55.604 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:34:55.604 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.605 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.605 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.605 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.605 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T14:34:55.605 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T14:34:55.605 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.606 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:34:55.606 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:34:55.606 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:34:55.606 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.606 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.607 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.607 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T14:34:55.607 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T14:34:55.607 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.607 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:34:55.608 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:34:55.608 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:34:55.608 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.608 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.608 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.609 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.609 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T14:34:55.609 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T14:34:55.609 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.609 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:34:55.610 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:34:55.610 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.610 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.610 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.610 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.611 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T14:34:55.611 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T14:34:55.611 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.611 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:34:55.611 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:34:55.612 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.612 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.612 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.612 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.612 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T14:34:55.612 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T14:34:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:34:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:34:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T14:34:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T14:34:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:34:55.615 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:34:55.615 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.615 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.615 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.615 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.615 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T14:34:55.616 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T14:34:55.616 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.616 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:34:55.616 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:34:55.616 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.616 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.616 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.616 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T14:34:55.617 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T14:34:55.617 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.617 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:34:55.617 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:34:55.617 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.617 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.618 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.618 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T14:34:55.618 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T14:34:55.618 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.618 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:34:55.618 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:34:55.618 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.618 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.619 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.619 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T14:34:55.619 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T14:34:55.619 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.619 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:34:55.619 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:34:55.620 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.620 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.620 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.620 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T14:34:55.620 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T14:34:55.620 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.620 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:34:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.622 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.623 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:34:55.624 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:55.625 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T14:34:55.626 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:34:55.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 -- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1194076420 msgr2=0x7f1194078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1194076420 0x7f1194078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f11980012e0 tx=0x7f1198009040 comp rx=0 tx=0).stop 2024-06-29T14:34:55.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 -- 172.21.15.81:0/3427530853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11a809d8a0 msgr2=0x7f11a809dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11a809d8a0 0x7f11a809dcb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f11a000b440 tx=0x7f11a000b900 comp rx=0 tx=0).stop 2024-06-29T14:34:55.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 -- 172.21.15.81:0/3427530853 shutdown_connections 2024-06-29T14:34:55.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1194076420 0x7f1194078930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11a809d8a0 0x7f11a809dcb0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f11a80b9040 0x7f11a8155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 --2- 172.21.15.81:0/3427530853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f11a80b5250 0x7f11a81589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 -- 172.21.15.81:0/3427530853 >> 172.21.15.81:0/3427530853 conn(0x7f11a809cf30 msgr2=0x7f11a8160e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:55.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 -- 172.21.15.81:0/3427530853 shutdown_connections 2024-06-29T14:34:55.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.604+0000 7f11ae19e640 1 -- 172.21.15.81:0/3427530853 wait complete. 2024-06-29T14:34:55.629 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:55.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.744+0000 7f3e920e6640 1 -- 172.21.15.81:0/690894141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e8c06f4e0 msgr2=0x7f3e8c166aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.744+0000 7f3e920e6640 1 --2- 172.21.15.81:0/690894141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e8c06f4e0 0x7f3e8c166aa0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3e84005fd0 tx=0x7f3e8402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.745+0000 7f3e920e6640 1 -- 172.21.15.81:0/690894141 shutdown_connections 2024-06-29T14:34:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.745+0000 7f3e920e6640 1 --2- 172.21.15.81:0/690894141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e8c06f4e0 0x7f3e8c166aa0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.745+0000 7f3e920e6640 1 -- 172.21.15.81:0/690894141 >> 172.21.15.81:0/690894141 conn(0x7f3e8c06ccc0 msgr2=0x7f3e8c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.745+0000 7f3e920e6640 1 -- 172.21.15.81:0/690894141 shutdown_connections 2024-06-29T14:34:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.745+0000 7f3e920e6640 1 -- 172.21.15.81:0/690894141 wait complete. 2024-06-29T14:34:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.746+0000 7f3e920e6640 1 Processor -- start 2024-06-29T14:34:55.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.746+0000 7f3e920e6640 1 -- start start 2024-06-29T14:34:55.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e920e6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e8c071f20 0x7f3e8c15b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e920e6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e8c144a20 0x7f3e8c12fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e920e6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e8c13b720 0x7f3e8c13bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e920e6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c167370 con 0x7f3e8c13b720 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e920e6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c1671f0 con 0x7f3e8c144a20 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e920e6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3e8c1674f0 con 0x7f3e8c071f20 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e8c13b720 0x7f3e8c13bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e8c13b720 0x7f3e8c13bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60346/0 (socket says 172.21.15.81:60346) 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8bfff640 1 -- 172.21.15.81:0/3653529342 learned_addr learned my addr 172.21.15.81:0/3653529342 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8b7fe640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e8c071f20 0x7f3e8c15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8affd640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e8c144a20 0x7f3e8c12fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8bfff640 1 -- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e8c071f20 msgr2=0x7f3e8c15b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8bfff640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e8c071f20 0x7f3e8c15b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8bfff640 1 -- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e8c144a20 msgr2=0x7f3e8c12fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8bfff640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e8c144a20 0x7f3e8c12fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8bfff640 1 -- 172.21.15.81:0/3653529342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c13c070 con 0x7f3e8c13b720 2024-06-29T14:34:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8affd640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e8c144a20 0x7f3e8c12fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e8bfff640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e8c13b720 0x7f3e8c13bb30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009cd0 tx=0x7f3e84003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.748+0000 7f3e73fff640 1 -- 172.21.15.81:0/3653529342 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e84031b90 con 0x7f3e8c13b720 2024-06-29T14:34:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.749+0000 7f3e73fff640 1 -- 172.21.15.81:0/3653529342 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e84031e20 con 0x7f3e8c13b720 2024-06-29T14:34:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.749+0000 7f3e73fff640 1 -- 172.21.15.81:0/3653529342 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e840332b0 con 0x7f3e8c13b720 2024-06-29T14:34:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.749+0000 7f3e920e6640 1 -- 172.21.15.81:0/3653529342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8c1312a0 con 0x7f3e8c13b720 2024-06-29T14:34:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.749+0000 7f3e920e6640 1 -- 172.21.15.81:0/3653529342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8c143d00 con 0x7f3e8c13b720 2024-06-29T14:34:55.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.749+0000 7f3e920e6640 1 -- 172.21.15.81:0/3653529342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e8c131810 con 0x7f3e8c13b720 2024-06-29T14:34:55.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.750+0000 7f3e73fff640 1 -- 172.21.15.81:0/3653529342 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f3e84031660 con 0x7f3e8c13b720 2024-06-29T14:34:55.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.751+0000 7f3e73fff640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f3e780763f0 0x7f3e78078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.751+0000 7f3e73fff640 1 -- 172.21.15.81:0/3653529342 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 4901+0+0 (secure 0 0 0) 0x7f3e840b8c30 con 0x7f3e8c13b720 2024-06-29T14:34:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.751+0000 7f3e8b7fe640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f3e780763f0 0x7f3e78078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.752+0000 7f3e8b7fe640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f3e780763f0 0x7f3e78078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c000f60 tx=0x7f3e7c009040 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:55.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.755+0000 7f3e73fff640 1 -- 172.21.15.81:0/3653529342 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3e8c13c070 con 0x7f3e8c13b720 2024-06-29T14:34:55.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.936+0000 7f3e920e6640 1 -- 172.21.15.81:0/3653529342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e8c131030 con 0x7f3e8c13b720 2024-06-29T14:34:55.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.937+0000 7f3e73fff640 1 -- 172.21.15.81:0/3653529342 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+12949 (secure 0 0 0) 0x7f3e8403f030 con 0x7f3e8c13b720 2024-06-29T14:34:55.935 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:55.935 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":57,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:34:55.222811+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:55.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 -- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f3e780763f0 msgr2=0x7f3e78078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f3e780763f0 0x7f3e78078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c000f60 tx=0x7f3e7c009040 comp rx=0 tx=0).stop 2024-06-29T14:34:55.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 -- 172.21.15.81:0/3653529342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e8c13b720 msgr2=0x7f3e8c13bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:55.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e8c13b720 0x7f3e8c13bb30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009cd0 tx=0x7f3e84003ba0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 -- 172.21.15.81:0/3653529342 shutdown_connections 2024-06-29T14:34:55.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f3e780763f0 0x7f3e78078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e8c13b720 0x7f3e8c13bb30 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e8c144a20 0x7f3e8c12fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 --2- 172.21.15.81:0/3653529342 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e8c071f20 0x7f3e8c15b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:55.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 -- 172.21.15.81:0/3653529342 >> 172.21.15.81:0/3653529342 conn(0x7f3e8c06ccc0 msgr2=0x7f3e8c06ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:55.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 -- 172.21.15.81:0/3653529342 shutdown_connections 2024-06-29T14:34:55.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:55.939+0000 7f3e920e6640 1 -- 172.21.15.81:0/3653529342 wait complete. 2024-06-29T14:34:55.948 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:34:55.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:34:56.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.068+0000 7feb222c5640 1 -- 172.21.15.81:0/2284801810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb1c0cea10 msgr2=0x7feb1c0b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.068+0000 7feb222c5640 1 --2- 172.21.15.81:0/2284801810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb1c0cea10 0x7feb1c0b47b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7feb1c0737a0 tx=0x7feb0402a560 comp rx=0 tx=0).stop 2024-06-29T14:34:56.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.068+0000 7feb222c5640 1 -- 172.21.15.81:0/2284801810 shutdown_connections 2024-06-29T14:34:56.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.068+0000 7feb222c5640 1 --2- 172.21.15.81:0/2284801810 >> v2:172.21.15.167:3300/0 conn(0x7feb1c0b4cf0 0x7feb1c0d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.068+0000 7feb222c5640 1 --2- 172.21.15.81:0/2284801810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb1c0cea10 0x7feb1c0b47b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.068+0000 7feb222c5640 1 --2- 172.21.15.81:0/2284801810 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb1c0b6d60 0x7feb1c0ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.068+0000 7feb222c5640 1 -- 172.21.15.81:0/2284801810 >> 172.21.15.81:0/2284801810 conn(0x7feb1c0d1960 msgr2=0x7feb1c0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:56.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.068+0000 7feb222c5640 1 -- 172.21.15.81:0/2284801810 shutdown_connections 2024-06-29T14:34:56.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.068+0000 7feb222c5640 1 -- 172.21.15.81:0/2284801810 wait complete. 2024-06-29T14:34:56.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.070+0000 7feb222c5640 1 Processor -- start 2024-06-29T14:34:56.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.070+0000 7feb222c5640 1 -- start start 2024-06-29T14:34:56.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.071+0000 7feb222c5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb1c0b4cf0 0x7feb1c0c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.071+0000 7feb222c5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb1c0b6d60 0x7feb1c1d9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.071+0000 7feb222c5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb1c0cea10 0x7feb1c06d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.071+0000 7feb222c5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c0d09c0 con 0x7feb1c0cea10 2024-06-29T14:34:56.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.071+0000 7feb222c5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c0d0840 con 0x7feb1c0b4cf0 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.071+0000 7feb222c5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7feb1c0cc530 con 0x7feb1c0b6d60 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.071+0000 7feb20862640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb1c0cea10 0x7feb1c06d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.071+0000 7feb20862640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb1c0cea10 0x7feb1c06d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60362/0 (socket says 172.21.15.81:60362) 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.071+0000 7feb20862640 1 -- 172.21.15.81:0/517096614 learned_addr learned my addr 172.21.15.81:0/517096614 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb13fff640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb1c0b6d60 0x7feb1c1d9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb1bfff640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb1c0b4cf0 0x7feb1c0c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb20862640 1 -- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb1c0b6d60 msgr2=0x7feb1c1d9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb20862640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb1c0b6d60 0x7feb1c1d9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb20862640 1 -- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb1c0b4cf0 msgr2=0x7feb1c0c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb20862640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb1c0b4cf0 0x7feb1c0c7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb20862640 1 -- 172.21.15.81:0/517096614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1c0c8470 con 0x7feb1c0cea10 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb1bfff640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb1c0b4cf0 0x7feb1c0c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb20862640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb1c0cea10 0x7feb1c06d830 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7feb1400cab0 tx=0x7feb1400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:56.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb197fa640 1 -- 172.21.15.81:0/517096614 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14007df0 con 0x7feb1c0cea10 2024-06-29T14:34:56.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb197fa640 1 -- 172.21.15.81:0/517096614 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb1400f560 con 0x7feb1c0cea10 2024-06-29T14:34:56.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb222c5640 1 -- 172.21.15.81:0/517096614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1c0c8b40 con 0x7feb1c0cea10 2024-06-29T14:34:56.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb197fa640 1 -- 172.21.15.81:0/517096614 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1400f9b0 con 0x7feb1c0cea10 2024-06-29T14:34:56.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.072+0000 7feb222c5640 1 -- 172.21.15.81:0/517096614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb1c0c9100 con 0x7feb1c0cea10 2024-06-29T14:34:56.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.073+0000 7feb222c5640 1 -- 172.21.15.81:0/517096614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb1c0ce4d0 con 0x7feb1c0cea10 2024-06-29T14:34:56.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.074+0000 7feb197fa640 1 -- 172.21.15.81:0/517096614 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7feb1400fb50 con 0x7feb1c0cea10 2024-06-29T14:34:56.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.074+0000 7feb197fa640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7feafc0763f0 0x7feafc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.074+0000 7feb197fa640 1 -- 172.21.15.81:0/517096614 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 4901+0+0 (secure 0 0 0) 0x7feb14098000 con 0x7feb1c0cea10 2024-06-29T14:34:56.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.074+0000 7feb1bfff640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7feafc0763f0 0x7feafc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:56.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.075+0000 7feb1bfff640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7feafc0763f0 0x7feafc078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feb0c005fd0 tx=0x7feb0c00f360 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:56.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.079+0000 7feb197fa640 1 -- 172.21.15.81:0/517096614 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feb14014070 con 0x7feb1c0cea10 2024-06-29T14:34:56.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.258+0000 7feb222c5640 1 -- 172.21.15.81:0/517096614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feb1c0c8200 con 0x7feb1c0cea10 2024-06-29T14:34:56.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.259+0000 7feb197fa640 1 -- 172.21.15.81:0/517096614 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7feb14061410 con 0x7feb1c0cea10 2024-06-29T14:34:56.256 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:34:56.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 -- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7feafc0763f0 msgr2=0x7feafc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7feafc0763f0 0x7feafc078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feb0c005fd0 tx=0x7feb0c00f360 comp rx=0 tx=0).stop 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 -- 172.21.15.81:0/517096614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb1c0cea10 msgr2=0x7feb1c06d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb1c0cea10 0x7feb1c06d830 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7feb1400cab0 tx=0x7feb1400cf70 comp rx=0 tx=0).stop 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 -- 172.21.15.81:0/517096614 shutdown_connections 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7feafc0763f0 0x7feafc078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb1c0cea10 0x7feb1c06d830 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feb1c0b6d60 0x7feb1c1d9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 --2- 172.21.15.81:0/517096614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feb1c0b4cf0 0x7feb1c0c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 -- 172.21.15.81:0/517096614 >> 172.21.15.81:0/517096614 conn(0x7feb1c0d1960 msgr2=0x7feb1c0b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 -- 172.21.15.81:0/517096614 shutdown_connections 2024-06-29T14:34:56.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.261+0000 7feb222c5640 1 -- 172.21.15.81:0/517096614 wait complete. 2024-06-29T14:34:56.270 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:34:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.387+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/911231269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f00005e50 msgr2=0x7f7f00006300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.387+0000 7f7f1d0ce640 1 --2- 172.21.15.81:0/911231269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f00005e50 0x7f7f00006300 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7f10001d90 tx=0x7f7f00006940 comp rx=0 tx=0).stop 2024-06-29T14:34:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.387+0000 7f7f1759e640 1 -- 172.21.15.81:0/911231269 reap_dead start 2024-06-29T14:34:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.387+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/911231269 shutdown_connections 2024-06-29T14:34:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.387+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/911231269 >> 172.21.15.81:0/911231269 conn(0x7f7f1806ccc0 msgr2=0x7f7f180b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.387+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/911231269 shutdown_connections 2024-06-29T14:34:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.387+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/911231269 wait complete. 2024-06-29T14:34:56.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.389+0000 7f7f1d0ce640 1 Processor -- start 2024-06-29T14:34:56.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.389+0000 7f7f1d0ce640 1 -- start start 2024-06-29T14:34:56.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1d0ce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f00005e50 0x7f7f181f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1d0ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f1814e6e0 0x7f7f180a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1d0ce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f180ab5a0 0x7f7f180a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1d0ce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1806ef50 con 0x7f7f180ab5a0 2024-06-29T14:34:56.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1d0ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1806edd0 con 0x7f7f1814e6e0 2024-06-29T14:34:56.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1d0ce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7f1806f0d0 con 0x7f7f00005e50 2024-06-29T14:34:56.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1759e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f180ab5a0 0x7f7f180a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1759e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f180ab5a0 0x7f7f180a9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60384/0 (socket says 172.21.15.81:60384) 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1759e640 1 -- 172.21.15.81:0/1683766352 learned_addr learned my addr 172.21.15.81:0/1683766352 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1759e640 1 -- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f00005e50 msgr2=0x7f7f181f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1759e640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f00005e50 0x7f7f181f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1759e640 1 -- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f1814e6e0 msgr2=0x7f7f180a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1759e640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f1814e6e0 0x7f7f180a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1759e640 1 -- 172.21.15.81:0/1683766352 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f180acdb0 con 0x7f7f180ab5a0 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1759e640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f180ab5a0 0x7f7f180a9a00 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7f1002ac80 tx=0x7f7f10001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7eff7fe640 1 -- 172.21.15.81:0/1683766352 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f100060b0 con 0x7f7f180ab5a0 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7eff7fe640 1 -- 172.21.15.81:0/1683766352 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f10006250 con 0x7f7f180ab5a0 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/1683766352 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f180abd10 con 0x7f7f180ab5a0 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.391+0000 7f7eff7fe640 1 -- 172.21.15.81:0/1683766352 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10006660 con 0x7f7f180ab5a0 2024-06-29T14:34:56.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.392+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/1683766352 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f180ac2d0 con 0x7f7f180ab5a0 2024-06-29T14:34:56.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.393+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/1683766352 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f180b4860 con 0x7f7f180ab5a0 2024-06-29T14:34:56.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.393+0000 7f7eff7fe640 1 -- 172.21.15.81:0/1683766352 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f7f10033480 con 0x7f7f180ab5a0 2024-06-29T14:34:56.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.394+0000 7f7eff7fe640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7f04076420 0x7f7f04078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.394+0000 7f7eff7fe640 1 -- 172.21.15.81:0/1683766352 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 4901+0+0 (secure 0 0 0) 0x7f7f100b7350 con 0x7f7f180ab5a0 2024-06-29T14:34:56.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.394+0000 7f7f16d9d640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7f04076420 0x7f7f04078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:56.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.395+0000 7f7f16d9d640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7f04076420 0x7f7f04078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7f08001860 tx=0x7f7f08001e40 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:56.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.398+0000 7f7eff7fe640 1 -- 172.21.15.81:0/1683766352 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7f10080db0 con 0x7f7f180ab5a0 2024-06-29T14:34:56.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.579+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/1683766352 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7f1814eb50 con 0x7f7f180ab5a0 2024-06-29T14:34:56.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.580+0000 7f7eff7fe640 1 -- 172.21.15.81:0/1683766352 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v57) ==== 84+0+65 (secure 0 0 0) 0x7f7f100806b0 con 0x7f7f180ab5a0 2024-06-29T14:34:56.577 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:56.577 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:34:56.578 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:34:56.578 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T14:34:56.578 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T14:34:56.578 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:34:56.578 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:34:56.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7f04076420 msgr2=0x7f7f04078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7f04076420 0x7f7f04078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7f08001860 tx=0x7f7f08001e40 comp rx=0 tx=0).stop 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/1683766352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f180ab5a0 msgr2=0x7f7f180a9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f180ab5a0 0x7f7f180a9a00 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7f1002ac80 tx=0x7f7f10001b10 comp rx=0 tx=0).stop 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/1683766352 shutdown_connections 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7f04076420 0x7f7f04078930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f180ab5a0 0x7f7f180a9a00 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f1814e6e0 0x7f7f180a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 --2- 172.21.15.81:0/1683766352 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f00005e50 0x7f7f181f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/1683766352 >> 172.21.15.81:0/1683766352 conn(0x7f7f1806ccc0 msgr2=0x7f7f18071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/1683766352 shutdown_connections 2024-06-29T14:34:56.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.582+0000 7f7f1d0ce640 1 -- 172.21.15.81:0/1683766352 wait complete. 2024-06-29T14:34:56.591 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:34:56.591 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:34:56.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.711+0000 7f5c30f63640 1 -- 172.21.15.81:0/1064327140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c0b21a0 msgr2=0x7f5c2c15c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.711+0000 7f5c30f63640 1 --2- 172.21.15.81:0/1064327140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c0b21a0 0x7f5c2c15c0e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009830 tx=0x7f5c2002a510 comp rx=0 tx=0).stop 2024-06-29T14:34:56.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.711+0000 7f5c30f63640 1 -- 172.21.15.81:0/1064327140 shutdown_connections 2024-06-29T14:34:56.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.711+0000 7f5c30f63640 1 --2- 172.21.15.81:0/1064327140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c0b21a0 0x7f5c2c15c0e0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.711+0000 7f5c30f63640 1 -- 172.21.15.81:0/1064327140 >> 172.21.15.81:0/1064327140 conn(0x7f5c2c09aa60 msgr2=0x7f5c2c09ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:56.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.711+0000 7f5c30f63640 1 -- 172.21.15.81:0/1064327140 shutdown_connections 2024-06-29T14:34:56.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.711+0000 7f5c30f63640 1 -- 172.21.15.81:0/1064327140 wait complete. 2024-06-29T14:34:56.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.713+0000 7f5c30f63640 1 Processor -- start 2024-06-29T14:34:56.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.713+0000 7f5c30f63640 1 -- start start 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c30f63640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c0ae8f0 0x7f5c2c14ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c30f63640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c2c0b2cb0 0x7f5c2c159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c30f63640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c2c152e90 0x7f5c2c1543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c30f63640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c160290 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c30f63640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c0d1b70 con 0x7f5c2c152e90 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c30f63640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5c2c160410 con 0x7f5c2c0b2cb0 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c2a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c0ae8f0 0x7f5c2c14ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c2a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c0ae8f0 0x7f5c2c14ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60400/0 (socket says 172.21.15.81:60400) 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c2a59c640 1 -- 172.21.15.81:0/4213671573 learned_addr learned my addr 172.21.15.81:0/4213671573 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c2a59c640 1 -- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c2c0b2cb0 msgr2=0x7f5c2c159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c2a59c640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c2c0b2cb0 0x7f5c2c159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c2a59c640 1 -- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c2c152e90 msgr2=0x7f5c2c1543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c2a59c640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c2c152e90 0x7f5c2c1543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c2a59c640 1 -- 172.21.15.81:0/4213671573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2c151df0 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.714+0000 7f5c2a59c640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c0ae8f0 0x7f5c2c14ff00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00c3f0 tx=0x7f5c1c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.715+0000 7f5c1affd640 1 -- 172.21.15.81:0/4213671573 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c017070 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.715+0000 7f5c1affd640 1 -- 172.21.15.81:0/4213671573 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c1c0035f0 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.715+0000 7f5c30f63640 1 -- 172.21.15.81:0/4213671573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c2c1520d0 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.715+0000 7f5c1affd640 1 -- 172.21.15.81:0/4213671573 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c003a40 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.715+0000 7f5c30f63640 1 -- 172.21.15.81:0/4213671573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c2c152690 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.716+0000 7f5c30f63640 1 -- 172.21.15.81:0/4213671573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c2c160940 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.716+0000 7f5c1affd640 1 -- 172.21.15.81:0/4213671573 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f5c1c01a070 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.717+0000 7f5c1affd640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5c0c0763f0 0x7f5c0c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:56.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.717+0000 7f5c1affd640 1 -- 172.21.15.81:0/4213671573 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 4901+0+0 (secure 0 0 0) 0x7f5c1c0969c0 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.717+0000 7f5c29d9b640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5c0c0763f0 0x7f5c0c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:56.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.718+0000 7f5c29d9b640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5c0c0763f0 0x7f5c0c078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c20034000 tx=0x7f5c200046c0 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:56.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.721+0000 7f5c1affd640 1 -- 172.21.15.81:0/4213671573 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5c1c098e20 con 0x7f5c2c0ae8f0 2024-06-29T14:34:56.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:56.906+0000 7f5c30f63640 1 -- 172.21.15.81:0/4213671573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5c2c0712a0 con 0x7f5c2c0ae8f0 2024-06-29T14:34:57.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.221+0000 7f5c1affd640 1 -- 172.21.15.81:0/4213671573 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v58) ==== 138+0+0 (secure 0 0 0) 0x7f5c1c0604a0 con 0x7f5c2c0ae8f0 2024-06-29T14:34:57.219 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T14:34:57.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 -- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5c0c0763f0 msgr2=0x7f5c0c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5c0c0763f0 0x7f5c0c078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c20034000 tx=0x7f5c200046c0 comp rx=0 tx=0).stop 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 -- 172.21.15.81:0/4213671573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c0ae8f0 msgr2=0x7f5c2c14ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c0ae8f0 0x7f5c2c14ff00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00c3f0 tx=0x7f5c1c00c8b0 comp rx=0 tx=0).stop 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 -- 172.21.15.81:0/4213671573 shutdown_connections 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c2c152e90 0x7f5c2c1543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c2c0b2cb0 0x7f5c2c159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5c0c0763f0 0x7f5c0c078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 --2- 172.21.15.81:0/4213671573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c0ae8f0 0x7f5c2c14ff00 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 -- 172.21.15.81:0/4213671573 >> 172.21.15.81:0/4213671573 conn(0x7f5c2c09aa60 msgr2=0x7f5c2c151470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 -- 172.21.15.81:0/4213671573 shutdown_connections 2024-06-29T14:34:57.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.224+0000 7f5c30f63640 1 -- 172.21.15.81:0/4213671573 wait complete. 2024-06-29T14:34:57.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:34:57.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.359+0000 7f6d4085b640 1 -- 172.21.15.81:0/3088942758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d28004d40 msgr2=0x7f6d28005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:57.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.359+0000 7f6d4085b640 1 --2- 172.21.15.81:0/3088942758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d28004d40 0x7f6d28005150 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0d4460 tx=0x7f6d28005830 comp rx=0 tx=0).stop 2024-06-29T14:34:57.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.359+0000 7f6d3ad9d640 1 -- 172.21.15.81:0/3088942758 reap_dead start 2024-06-29T14:34:57.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.359+0000 7f6d4085b640 1 -- 172.21.15.81:0/3088942758 shutdown_connections 2024-06-29T14:34:57.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.359+0000 7f6d4085b640 1 -- 172.21.15.81:0/3088942758 >> 172.21.15.81:0/3088942758 conn(0x7f6d3c09cf30 msgr2=0x7f6d3c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:57.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.359+0000 7f6d4085b640 1 -- 172.21.15.81:0/3088942758 shutdown_connections 2024-06-29T14:34:57.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.359+0000 7f6d4085b640 1 -- 172.21.15.81:0/3088942758 wait complete. 2024-06-29T14:34:57.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.361+0000 7f6d4085b640 1 Processor -- start 2024-06-29T14:34:57.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.361+0000 7f6d4085b640 1 -- start start 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d4085b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d28004d40 0x7f6d3c150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d4085b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d3c0a0be0 0x7f6d3c0a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d4085b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d3c0cd600 0x7f6d3c0cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d4085b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3c0d0d50 con 0x7f6d28004d40 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d4085b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3c0d0bd0 con 0x7f6d3c0cd600 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d4085b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6d3c0b79a0 con 0x7f6d3c0a0be0 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d3a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d28004d40 0x7f6d3c150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d3a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d28004d40 0x7f6d3c150890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60426/0 (socket says 172.21.15.81:60426) 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d3a59c640 1 -- 172.21.15.81:0/739918345 learned_addr learned my addr 172.21.15.81:0/739918345 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d31d9b640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d3c0a0be0 0x7f6d3c0a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.362+0000 7f6d3ad9d640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d3c0cd600 0x7f6d3c0cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d3a59c640 1 -- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d3c0a0be0 msgr2=0x7f6d3c0a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d3a59c640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d3c0a0be0 0x7f6d3c0a0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d3a59c640 1 -- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d3c0cd600 msgr2=0x7f6d3c0cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d3a59c640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d3c0cd600 0x7f6d3c0cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d3a59c640 1 -- 172.21.15.81:0/739918345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d3c150dd0 con 0x7f6d28004d40 2024-06-29T14:34:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d3a59c640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d28004d40 0x7f6d3c150890 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c011a50 tx=0x7f6d2c011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d337fe640 1 -- 172.21.15.81:0/739918345 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c01f070 con 0x7f6d28004d40 2024-06-29T14:34:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d337fe640 1 -- 172.21.15.81:0/739918345 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d2c00fe80 con 0x7f6d28004d40 2024-06-29T14:34:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d4085b640 1 -- 172.21.15.81:0/739918345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d3c0b0e00 con 0x7f6d28004d40 2024-06-29T14:34:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d337fe640 1 -- 172.21.15.81:0/739918345 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c012690 con 0x7f6d28004d40 2024-06-29T14:34:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.363+0000 7f6d4085b640 1 -- 172.21.15.81:0/739918345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d3c0b13c0 con 0x7f6d28004d40 2024-06-29T14:34:57.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.364+0000 7f6d4085b640 1 -- 172.21.15.81:0/739918345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d3c14efc0 con 0x7f6d28004d40 2024-06-29T14:34:57.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.365+0000 7f6d337fe640 1 -- 172.21.15.81:0/739918345 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f6d2c022070 con 0x7f6d28004d40 2024-06-29T14:34:57.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.365+0000 7f6d337fe640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6d1c0763f0 0x7f6d1c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:57.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.366+0000 7f6d337fe640 1 -- 172.21.15.81:0/739918345 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5271+0+0 (secure 0 0 0) 0x7f6d2c098240 con 0x7f6d28004d40 2024-06-29T14:34:57.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.366+0000 7f6d31d9b640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6d1c0763f0 0x7f6d1c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:57.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.366+0000 7f6d31d9b640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6d1c0763f0 0x7f6d1c078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6d24001860 tx=0x7f6d24000ed0 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:57.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.370+0000 7f6d337fe640 1 -- 172.21.15.81:0/739918345 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6d3c150dd0 con 0x7f6d28004d40 2024-06-29T14:34:57.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:57.550+0000 7f6d4085b640 1 -- 172.21.15.81:0/739918345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6d3c0d0480 con 0x7f6d28004d40 2024-06-29T14:34:58.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.229+0000 7f6d337fe640 1 -- 172.21.15.81:0/739918345 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v59) ==== 170+0+0 (secure 0 0 0) 0x7f6d2c0615f0 con 0x7f6d28004d40 2024-06-29T14:34:58.226 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T14:34:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.231+0000 7f6d4085b640 1 -- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6d1c0763f0 msgr2=0x7f6d1c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.231+0000 7f6d4085b640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6d1c0763f0 0x7f6d1c078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6d24001860 tx=0x7f6d24000ed0 comp rx=0 tx=0).stop 2024-06-29T14:34:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.231+0000 7f6d4085b640 1 -- 172.21.15.81:0/739918345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d28004d40 msgr2=0x7f6d3c150890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.231+0000 7f6d4085b640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d28004d40 0x7f6d3c150890 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c011a50 tx=0x7f6d2c011f10 comp rx=0 tx=0).stop 2024-06-29T14:34:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.231+0000 7f6d4085b640 1 -- 172.21.15.81:0/739918345 shutdown_connections 2024-06-29T14:34:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.231+0000 7f6d4085b640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d3c0cd600 0x7f6d3c0cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:58.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.231+0000 7f6d4085b640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d3c0a0be0 0x7f6d3c0a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:58.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.231+0000 7f6d4085b640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f6d1c0763f0 0x7f6d1c078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:58.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.231+0000 7f6d4085b640 1 --2- 172.21.15.81:0/739918345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d28004d40 0x7f6d3c150890 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:58.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.231+0000 7f6d4085b640 1 -- 172.21.15.81:0/739918345 >> 172.21.15.81:0/739918345 conn(0x7f6d3c09cf30 msgr2=0x7f6d3c0b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:58.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.232+0000 7f6d4085b640 1 -- 172.21.15.81:0/739918345 shutdown_connections 2024-06-29T14:34:58.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.232+0000 7f6d4085b640 1 -- 172.21.15.81:0/739918345 wait complete. 2024-06-29T14:34:58.252 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:34:58.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.371+0000 7fbe372e3640 1 -- 172.21.15.81:0/1803589846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe30159bd0 msgr2=0x7fbe300d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:58.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.371+0000 7fbe372e3640 1 --2- 172.21.15.81:0/1803589846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe30159bd0 0x7fbe300d0010 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbe30072100 tx=0x7fbe2c02a530 comp rx=0 tx=0).stop 2024-06-29T14:34:58.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.371+0000 7fbe372e3640 1 -- 172.21.15.81:0/1803589846 shutdown_connections 2024-06-29T14:34:58.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.371+0000 7fbe372e3640 1 --2- 172.21.15.81:0/1803589846 >> v2:172.21.15.167:3300/0 conn(0x7fbe300d0550 0x7fbe3006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:58.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.371+0000 7fbe372e3640 1 --2- 172.21.15.81:0/1803589846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe30159bd0 0x7fbe300d0010 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:58.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.371+0000 7fbe372e3640 1 -- 172.21.15.81:0/1803589846 >> 172.21.15.81:0/1803589846 conn(0x7fbe300b2920 msgr2=0x7fbe300b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:58.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.371+0000 7fbe372e3640 1 -- 172.21.15.81:0/1803589846 shutdown_connections 2024-06-29T14:34:58.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.371+0000 7fbe372e3640 1 -- 172.21.15.81:0/1803589846 wait complete. 2024-06-29T14:34:58.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.373+0000 7fbe372e3640 1 Processor -- start 2024-06-29T14:34:58.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.373+0000 7fbe372e3640 1 -- start start 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.374+0000 7fbe372e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe3006c830 0x7fbe3013f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.374+0000 7fbe372e3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe300d0550 0x7fbe30142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.374+0000 7fbe372e3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe30159bd0 0x7fbe30142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.374+0000 7fbe372e3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3015fa90 con 0x7fbe3006c830 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.374+0000 7fbe372e3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbe300703a0 con 0x7fbe300d0550 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.374+0000 7fbe372e3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbe3015fc10 con 0x7fbe30159bd0 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.374+0000 7fbe3507f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe3006c830 0x7fbe3013f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.374+0000 7fbe3507f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe3006c830 0x7fbe3013f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60444/0 (socket says 172.21.15.81:60444) 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.374+0000 7fbe3507f640 1 -- 172.21.15.81:0/249206578 learned_addr learned my addr 172.21.15.81:0/249206578 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe3507f640 1 -- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe30159bd0 msgr2=0x7fbe30142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe3507f640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe30159bd0 0x7fbe30142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:58.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe3507f640 1 -- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe300d0550 msgr2=0x7fbe30142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe3507f640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe300d0550 0x7fbe30142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe3507f640 1 -- 172.21.15.81:0/249206578 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe300a9c10 con 0x7fbe3006c830 2024-06-29T14:34:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe3507f640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe3006c830 0x7fbe3013f6d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009780 tx=0x7fbe2c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe25ffb640 1 -- 172.21.15.81:0/249206578 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c031d60 con 0x7fbe3006c830 2024-06-29T14:34:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe25ffb640 1 -- 172.21.15.81:0/249206578 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe2c033e10 con 0x7fbe3006c830 2024-06-29T14:34:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe372e3640 1 -- 172.21.15.81:0/249206578 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe300a9ef0 con 0x7fbe3006c830 2024-06-29T14:34:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe25ffb640 1 -- 172.21.15.81:0/249206578 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c005260 con 0x7fbe3006c830 2024-06-29T14:34:58.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.375+0000 7fbe372e3640 1 -- 172.21.15.81:0/249206578 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe300aa4b0 con 0x7fbe3006c830 2024-06-29T14:34:58.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.376+0000 7fbe372e3640 1 -- 172.21.15.81:0/249206578 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe3006c2f0 con 0x7fbe3006c830 2024-06-29T14:34:58.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.377+0000 7fbe25ffb640 1 -- 172.21.15.81:0/249206578 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fbe2c005400 con 0x7fbe3006c830 2024-06-29T14:34:58.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.378+0000 7fbe25ffb640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbe140763f0 0x7fbe14078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:58.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.378+0000 7fbe25ffb640 1 -- 172.21.15.81:0/249206578 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5653+0+0 (secure 0 0 0) 0x7fbe2c0b90b0 con 0x7fbe3006c830 2024-06-29T14:34:58.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.378+0000 7fbe3487e640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbe140763f0 0x7fbe14078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:58.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.378+0000 7fbe3487e640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbe140763f0 0x7fbe14078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe200058c0 tx=0x7fbe20005850 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:58.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.382+0000 7fbe25ffb640 1 -- 172.21.15.81:0/249206578 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbe2c03f030 con 0x7fbe3006c830 2024-06-29T14:34:58.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:58.601+0000 7fbe372e3640 1 -- 172.21.15.81:0/249206578 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbe3006f330 con 0x7fbe3006c830 2024-06-29T14:34:59.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.243+0000 7fbe25ffb640 1 -- 172.21.15.81:0/249206578 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) ==== 172+0+0 (secure 0 0 0) 0x7fbe2c082870 con 0x7fbe3006c830 2024-06-29T14:34:59.242 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 6 and data pool 7 2024-06-29T14:34:59.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.249+0000 7fbe372e3640 1 -- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbe140763f0 msgr2=0x7fbe14078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.249+0000 7fbe372e3640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbe140763f0 0x7fbe14078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe200058c0 tx=0x7fbe20005850 comp rx=0 tx=0).stop 2024-06-29T14:34:59.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.249+0000 7fbe372e3640 1 -- 172.21.15.81:0/249206578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe3006c830 msgr2=0x7fbe3013f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.249+0000 7fbe372e3640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe3006c830 0x7fbe3013f6d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009780 tx=0x7fbe2c033c20 comp rx=0 tx=0).stop 2024-06-29T14:34:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.249+0000 7fbe372e3640 1 -- 172.21.15.81:0/249206578 shutdown_connections 2024-06-29T14:34:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.249+0000 7fbe372e3640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe30159bd0 0x7fbe30142d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.249+0000 7fbe372e3640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe300d0550 0x7fbe30142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.249+0000 7fbe372e3640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fbe140763f0 0x7fbe14078900 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe200058c0 tx=0x7fbe20005850 comp rx=0 tx=0).stop 2024-06-29T14:34:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.249+0000 7fbe372e3640 1 --2- 172.21.15.81:0/249206578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe3006c830 0x7fbe3013f6d0 secure :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c009780 tx=0x7fbe2c033c20 comp rx=0 tx=0).stop 2024-06-29T14:34:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.249+0000 7fbe372e3640 1 -- 172.21.15.81:0/249206578 >> 172.21.15.81:0/249206578 conn(0x7fbe300b2920 msgr2=0x7fbe301454c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:59.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.254+0000 7fbe372e3640 1 -- 172.21.15.81:0/249206578 shutdown_connections 2024-06-29T14:34:59.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.261+0000 7fbe372e3640 1 -- 172.21.15.81:0/249206578 wait complete. 2024-06-29T14:34:59.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:59.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.395+0000 7f7ec47d4640 1 -- 172.21.15.81:0/3824596456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ebc15b5b0 msgr2=0x7f7ebc0b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.395+0000 7f7ec47d4640 1 --2- 172.21.15.81:0/3824596456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ebc15b5b0 0x7f7ebc0b4430 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7eb000a940 tx=0x7f7eb002b850 comp rx=0 tx=0).stop 2024-06-29T14:34:59.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.395+0000 7f7ec47d4640 1 -- 172.21.15.81:0/3824596456 shutdown_connections 2024-06-29T14:34:59.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.395+0000 7f7ec47d4640 1 --2- 172.21.15.81:0/3824596456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ebc15b5b0 0x7f7ebc0b4430 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.395+0000 7f7ec47d4640 1 -- 172.21.15.81:0/3824596456 >> 172.21.15.81:0/3824596456 conn(0x7f7ebc06ccc0 msgr2=0x7f7ebc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:59.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.395+0000 7f7ec47d4640 1 -- 172.21.15.81:0/3824596456 shutdown_connections 2024-06-29T14:34:59.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.395+0000 7f7ec47d4640 1 -- 172.21.15.81:0/3824596456 wait complete. 2024-06-29T14:34:59.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.396+0000 7f7ec47d4640 1 Processor -- start 2024-06-29T14:34:59.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.396+0000 7f7ec47d4640 1 -- start start 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec47d4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ebc06f660 0x7f7ebc14c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec47d4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ebc0b35b0 0x7f7ebc14d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec47d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ebc15b5b0 0x7f7ebc146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec47d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ebc06f460 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec47d4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7ebc06f2e0 con 0x7f7ebc06f660 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec47d4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7ebc071f20 con 0x7f7ebc0b35b0 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec2d71640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ebc15b5b0 0x7f7ebc146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec2d71640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ebc15b5b0 0x7f7ebc146240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60474/0 (socket says 172.21.15.81:60474) 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec2d71640 1 -- 172.21.15.81:0/670395154 learned_addr learned my addr 172.21.15.81:0/670395154 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec2570640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ebc06f660 0x7f7ebc14c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:59.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec1d6f640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ebc0b35b0 0x7f7ebc14d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec2d71640 1 -- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ebc0b35b0 msgr2=0x7f7ebc14d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec2d71640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ebc0b35b0 0x7f7ebc14d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec2d71640 1 -- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ebc06f660 msgr2=0x7f7ebc14c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec2d71640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ebc06f660 0x7f7ebc14c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec2d71640 1 -- 172.21.15.81:0/670395154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ebc14da50 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ec2d71640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ebc15b5b0 0x7f7ebc146240 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f7eb400c3c0 tx=0x7f7eb400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.398+0000 7f7ebaffd640 1 -- 172.21.15.81:0/670395154 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb4017070 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.399+0000 7f7ebaffd640 1 -- 172.21.15.81:0/670395154 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7eb40035f0 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.399+0000 7f7ec47d4640 1 -- 172.21.15.81:0/670395154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ebc14df60 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.399+0000 7f7ebaffd640 1 -- 172.21.15.81:0/670395154 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb4003a40 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.399+0000 7f7ec47d4640 1 -- 172.21.15.81:0/670395154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ebc14c0c0 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.400+0000 7f7ec47d4640 1 -- 172.21.15.81:0/670395154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ebc14dcd0 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.400+0000 7f7ebaffd640 1 -- 172.21.15.81:0/670395154 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f7eb401a070 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.401+0000 7f7ebaffd640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7ea80763f0 0x7f7ea8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:59.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.401+0000 7f7ebaffd640 1 -- 172.21.15.81:0/670395154 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5753+0+0 (secure 0 0 0) 0x7f7eb4096bc0 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.401+0000 7f7ec2570640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7ea80763f0 0x7f7ea8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:59.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.402+0000 7f7ec2570640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7ea80763f0 0x7f7ea8078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7eac007fd0 tx=0x7f7eac009040 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:59.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.406+0000 7f7ebaffd640 1 -- 172.21.15.81:0/670395154 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7eb4060410 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.589+0000 7f7ec47d4640 1 -- 172.21.15.81:0/670395154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ebc14bbd0 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.590+0000 7f7ebaffd640 1 -- 172.21.15.81:0/670395154 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+16684 (secure 0 0 0) 0x7f7eb405fd10 con 0x7f7ebc15b5b0 2024-06-29T14:34:59.588 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:59.589 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":60,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:34:58.602155+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:34:56.911140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-06-29T14:34:57.555296+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 -- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7ea80763f0 msgr2=0x7f7ea8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7ea80763f0 0x7f7ea8078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7eac007fd0 tx=0x7f7eac009040 comp rx=0 tx=0).stop 2024-06-29T14:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 -- 172.21.15.81:0/670395154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ebc15b5b0 msgr2=0x7f7ebc146240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ebc15b5b0 0x7f7ebc146240 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f7eb400c3c0 tx=0x7f7eb400c880 comp rx=0 tx=0).stop 2024-06-29T14:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 -- 172.21.15.81:0/670395154 shutdown_connections 2024-06-29T14:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7ea80763f0 0x7f7ea8078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ebc15b5b0 0x7f7ebc146240 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ebc0b35b0 0x7f7ebc14d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 --2- 172.21.15.81:0/670395154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ebc06f660 0x7f7ebc14c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 -- 172.21.15.81:0/670395154 >> 172.21.15.81:0/670395154 conn(0x7f7ebc06ccc0 msgr2=0x7f7ebc14b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:59.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 -- 172.21.15.81:0/670395154 shutdown_connections 2024-06-29T14:34:59.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.592+0000 7f7ec47d4640 1 -- 172.21.15.81:0/670395154 wait complete. 2024-06-29T14:34:59.601 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:34:59.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.717+0000 7fece7a26640 1 -- 172.21.15.81:0/2063484848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece00b3200 msgr2=0x7fece00cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.717+0000 7fece7a26640 1 --2- 172.21.15.81:0/2063484848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece00b3200 0x7fece00cc7b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009830 tx=0x7fecdc02a480 comp rx=0 tx=0).stop 2024-06-29T14:34:59.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.717+0000 7fece7a26640 1 -- 172.21.15.81:0/2063484848 shutdown_connections 2024-06-29T14:34:59.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.717+0000 7fece7a26640 1 --2- 172.21.15.81:0/2063484848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece00b3200 0x7fece00cc7b0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.717+0000 7fece7a26640 1 -- 172.21.15.81:0/2063484848 >> 172.21.15.81:0/2063484848 conn(0x7fece00b3680 msgr2=0x7fece00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:59.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.717+0000 7fece7a26640 1 -- 172.21.15.81:0/2063484848 shutdown_connections 2024-06-29T14:34:59.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.717+0000 7fece7a26640 1 -- 172.21.15.81:0/2063484848 wait complete. 2024-06-29T14:34:59.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.719+0000 7fece7a26640 1 Processor -- start 2024-06-29T14:34:59.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.719+0000 7fece7a26640 1 -- start start 2024-06-29T14:34:59.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.720+0000 7fece7a26640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece00b3200 0x7fece00ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:59.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.720+0000 7fece7a26640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fece00cccf0 0x7fece009e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:59.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.720+0000 7fece7a26640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fece009e630 0x7fece009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:59.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.720+0000 7fece7a26640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fece00b5e50 con 0x7fece00b3200 2024-06-29T14:34:59.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.720+0000 7fece7a26640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fece00b5cd0 con 0x7fece00cccf0 2024-06-29T14:34:59.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.720+0000 7fece7a26640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fece00b5fd0 con 0x7fece009e630 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.720+0000 7fece57c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece00b3200 0x7fece00ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece57c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece00b3200 0x7fece00ca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60500/0 (socket says 172.21.15.81:60500) 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece57c2640 1 -- 172.21.15.81:0/1850177305 learned_addr learned my addr 172.21.15.81:0/1850177305 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece4fc1640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fece00cccf0 0x7fece009e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece5fc3640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fece009e630 0x7fece009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece57c2640 1 -- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fece009e630 msgr2=0x7fece009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece57c2640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fece009e630 0x7fece009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece57c2640 1 -- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fece00cccf0 msgr2=0x7fece009e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece57c2640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fece00cccf0 0x7fece009e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece57c2640 1 -- 172.21.15.81:0/1850177305 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece009ef80 con 0x7fece00b3200 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece5fc3640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fece009e630 0x7fece009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece57c2640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece00b3200 0x7fece00ca770 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fecdc02aa70 tx=0x7fecdc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fecd5ffb640 1 -- 172.21.15.81:0/1850177305 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc031d60 con 0x7fece00b3200 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fecd5ffb640 1 -- 172.21.15.81:0/1850177305 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecdc033e10 con 0x7fece00b3200 2024-06-29T14:34:59.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece7a26640 1 -- 172.21.15.81:0/1850177305 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece00cb150 con 0x7fece00b3200 2024-06-29T14:34:59.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fecd5ffb640 1 -- 172.21.15.81:0/1850177305 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc005260 con 0x7fece00b3200 2024-06-29T14:34:59.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.721+0000 7fece7a26640 1 -- 172.21.15.81:0/1850177305 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece009d8f0 con 0x7fece00b3200 2024-06-29T14:34:59.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.722+0000 7fece7a26640 1 -- 172.21.15.81:0/1850177305 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece00cb670 con 0x7fece00b3200 2024-06-29T14:34:59.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.723+0000 7fecd5ffb640 1 -- 172.21.15.81:0/1850177305 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fecdc005400 con 0x7fece00b3200 2024-06-29T14:34:59.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.724+0000 7fecd5ffb640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fecc40763f0 0x7fecc4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:34:59.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.724+0000 7fecd5ffb640 1 -- 172.21.15.81:0/1850177305 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5753+0+0 (secure 0 0 0) 0x7fecdc0b8c40 con 0x7fece00b3200 2024-06-29T14:34:59.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.724+0000 7fece4fc1640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fecc40763f0 0x7fecc4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:34:59.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.725+0000 7fece4fc1640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fecc40763f0 0x7fecc4078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fecd0000f60 tx=0x7fecd0009040 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:34:59.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.728+0000 7fecd5ffb640 1 -- 172.21.15.81:0/1850177305 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fece00cb670 con 0x7fece00b3200 2024-06-29T14:34:59.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.909+0000 7fece7a26640 1 -- 172.21.15.81:0/1850177305 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fece00b3da0 con 0x7fece00b3200 2024-06-29T14:34:59.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.910+0000 7fecd5ffb640 1 -- 172.21.15.81:0/1850177305 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+16684 (secure 0 0 0) 0x7fecdc03f030 con 0x7fece00b3200 2024-06-29T14:34:59.907 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:34:59.907 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":60,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:34:58.602155+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:34:56.911140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-06-29T14:34:57.555296+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:34:59.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 -- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fecc40763f0 msgr2=0x7fecc4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fecc40763f0 0x7fecc4078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fecd0000f60 tx=0x7fecd0009040 comp rx=0 tx=0).stop 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 -- 172.21.15.81:0/1850177305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece00b3200 msgr2=0x7fece00ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece00b3200 0x7fece00ca770 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fecdc02aa70 tx=0x7fecdc033c20 comp rx=0 tx=0).stop 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 -- 172.21.15.81:0/1850177305 shutdown_connections 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fece009e630 0x7fece009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fece00cccf0 0x7fece009e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fecc40763f0 0x7fecc4078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 --2- 172.21.15.81:0/1850177305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece00b3200 0x7fece00ca770 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 -- 172.21.15.81:0/1850177305 >> 172.21.15.81:0/1850177305 conn(0x7fece00b3680 msgr2=0x7fece00cf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 -- 172.21.15.81:0/1850177305 shutdown_connections 2024-06-29T14:34:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:34:59.912+0000 7fece7a26640 1 -- 172.21.15.81:0/1850177305 wait complete. 2024-06-29T14:34:59.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:35:00.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.041+0000 7f8527626640 1 -- 172.21.15.81:0/2532204838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8520073570 msgr2=0x7f852015e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:00.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.041+0000 7f8527626640 1 --2- 172.21.15.81:0/2532204838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8520073570 0x7f852015e960 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f85200b2700 tx=0x7f851802aa30 comp rx=0 tx=0).stop 2024-06-29T14:35:00.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.041+0000 7f8527626640 1 -- 172.21.15.81:0/2532204838 shutdown_connections 2024-06-29T14:35:00.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.041+0000 7f8527626640 1 --2- 172.21.15.81:0/2532204838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8520073570 0x7f852015e960 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:00.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.041+0000 7f8527626640 1 --2- 172.21.15.81:0/2532204838 >> v2:172.21.15.167:3300/0 conn(0x7f85200cf4d0 0x7f8520073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:00.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.041+0000 7f8527626640 1 --2- 172.21.15.81:0/2532204838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f85201412f0 0x7f85200cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:00.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.041+0000 7f8527626640 1 -- 172.21.15.81:0/2532204838 >> 172.21.15.81:0/2532204838 conn(0x7f852006ccc0 msgr2=0x7f85200b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:00.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.041+0000 7f8527626640 1 -- 172.21.15.81:0/2532204838 shutdown_connections 2024-06-29T14:35:00.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.041+0000 7f8527626640 1 -- 172.21.15.81:0/2532204838 wait complete. 2024-06-29T14:35:00.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.043+0000 7f8527626640 1 Processor -- start 2024-06-29T14:35:00.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.043+0000 7f8527626640 1 -- start start 2024-06-29T14:35:00.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8527626640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8520073570 0x7f85200c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:00.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8527626640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85200cf4d0 0x7f85200ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:00.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8527626640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85201412f0 0x7f85200c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:00.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8527626640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f852015f230 con 0x7f85200cf4d0 2024-06-29T14:35:00.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8527626640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f852015f0b0 con 0x7f85201412f0 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8527626640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f85200b3210 con 0x7f8520073570 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8524bc1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85200cf4d0 0x7f85200ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8524bc1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85200cf4d0 0x7f85200ac880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50284/0 (socket says 172.21.15.81:50284) 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8524bc1640 1 -- 172.21.15.81:0/2510737760 learned_addr learned my addr 172.21.15.81:0/2510737760 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f85253c2640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8520073570 0x7f85200c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8525bc3640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85201412f0 0x7f85200c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8524bc1640 1 -- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8520073570 msgr2=0x7f85200c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8524bc1640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8520073570 0x7f85200c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8524bc1640 1 -- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85201412f0 msgr2=0x7f85200c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8524bc1640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85201412f0 0x7f85200c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8524bc1640 1 -- 172.21.15.81:0/2510737760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85200af2e0 con 0x7f85200cf4d0 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.044+0000 7f8525bc3640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85201412f0 0x7f85200c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:00.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.045+0000 7f8524bc1640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85200cf4d0 0x7f85200ac880 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f851400d720 tx=0x7f851400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:00.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.045+0000 7f851dffb640 1 -- 172.21.15.81:0/2510737760 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8514017070 con 0x7f85200cf4d0 2024-06-29T14:35:00.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.045+0000 7f851dffb640 1 -- 172.21.15.81:0/2510737760 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8514009ab0 con 0x7f85200cf4d0 2024-06-29T14:35:00.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.045+0000 7f8527626640 1 -- 172.21.15.81:0/2510737760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85200adf20 con 0x7f85200cf4d0 2024-06-29T14:35:00.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.045+0000 7f851dffb640 1 -- 172.21.15.81:0/2510737760 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8514010460 con 0x7f85200cf4d0 2024-06-29T14:35:00.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.045+0000 7f8527626640 1 -- 172.21.15.81:0/2510737760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85200ac080 con 0x7f85200cf4d0 2024-06-29T14:35:00.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.046+0000 7f8527626640 1 -- 172.21.15.81:0/2510737760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85200ae440 con 0x7f85200cf4d0 2024-06-29T14:35:00.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.047+0000 7f851dffb640 1 -- 172.21.15.81:0/2510737760 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f851401a070 con 0x7f85200cf4d0 2024-06-29T14:35:00.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.047+0000 7f851dffb640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f850c0763f0 0x7f850c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:00.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.047+0000 7f851dffb640 1 -- 172.21.15.81:0/2510737760 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5753+0+0 (secure 0 0 0) 0x7f8514097a40 con 0x7f85200cf4d0 2024-06-29T14:35:00.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.048+0000 7f85253c2640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f850c0763f0 0x7f850c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:00.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.048+0000 7f85253c2640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f850c0763f0 0x7f850c078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8510004630 tx=0x7f851000b2e0 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:00.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.052+0000 7f851dffb640 1 -- 172.21.15.81:0/2510737760 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f85200ae440 con 0x7f85200cf4d0 2024-06-29T14:35:00.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:00.274+0000 7f8527626640 1 -- 172.21.15.81:0/2510737760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f85200cd5e0 con 0x7f85200cf4d0 2024-06-29T14:35:01.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.252+0000 7f851dffb640 1 -- 172.21.15.81:0/2510737760 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f85200cd5e0 con 0x7f85200cf4d0 2024-06-29T14:35:01.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.255+0000 7f8527626640 1 -- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f850c0763f0 msgr2=0x7f850c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:01.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.255+0000 7f8527626640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f850c0763f0 0x7f850c078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8510004630 tx=0x7f851000b2e0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.255+0000 7f8527626640 1 -- 172.21.15.81:0/2510737760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85200cf4d0 msgr2=0x7f85200ac880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:01.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.255+0000 7f8527626640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85200cf4d0 0x7f85200ac880 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f851400d720 tx=0x7f851400dbe0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.256+0000 7f8527626640 1 -- 172.21.15.81:0/2510737760 shutdown_connections 2024-06-29T14:35:01.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.256+0000 7f8527626640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f85201412f0 0x7f85200c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.256+0000 7f8527626640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f850c0763f0 0x7f850c078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.256+0000 7f8527626640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85200cf4d0 0x7f85200ac880 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.256+0000 7f8527626640 1 --2- 172.21.15.81:0/2510737760 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8520073570 0x7f85200c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.256+0000 7f8527626640 1 -- 172.21.15.81:0/2510737760 >> 172.21.15.81:0/2510737760 conn(0x7f852006ccc0 msgr2=0x7f852015e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:01.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.256+0000 7f8527626640 1 -- 172.21.15.81:0/2510737760 shutdown_connections 2024-06-29T14:35:01.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.256+0000 7f8527626640 1 -- 172.21.15.81:0/2510737760 wait complete. 2024-06-29T14:35:01.268 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:35:01.268 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:35:01.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.394+0000 7f748895b640 1 -- 172.21.15.81:0/2706823209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f748406ef90 msgr2=0x7f748406f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:01.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.394+0000 7f748895b640 1 --2- 172.21.15.81:0/2706823209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f748406ef90 0x7f748406f3a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f7484071e60 tx=0x7f747802a560 comp rx=0 tx=0).stop 2024-06-29T14:35:01.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.394+0000 7f748895b640 1 -- 172.21.15.81:0/2706823209 shutdown_connections 2024-06-29T14:35:01.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.394+0000 7f748895b640 1 --2- 172.21.15.81:0/2706823209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f748406ef90 0x7f748406f3a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.394+0000 7f748895b640 1 --2- 172.21.15.81:0/2706823209 >> v2:172.21.15.167:3300/0 conn(0x7f74840b57b0 0x7f748415aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.394+0000 7f748895b640 1 -- 172.21.15.81:0/2706823209 >> 172.21.15.81:0/2706823209 conn(0x7f74840d0930 msgr2=0x7f74840d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:01.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.394+0000 7f748895b640 1 -- 172.21.15.81:0/2706823209 shutdown_connections 2024-06-29T14:35:01.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.394+0000 7f748895b640 1 -- 172.21.15.81:0/2706823209 wait complete. 2024-06-29T14:35:01.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.396+0000 7f748895b640 1 Processor -- start 2024-06-29T14:35:01.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.396+0000 7f748895b640 1 -- start start 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f748895b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f748406ef90 0x7f74841f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f748895b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b57b0 0x7f74840adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f748895b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74840ce1d0 0x7f74841f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f748895b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74840b4b30 con 0x7f74840b57b0 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f748895b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f74840b49b0 con 0x7f74840ce1d0 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f748895b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f74841f4580 con 0x7f748406ef90 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f7481d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b57b0 0x7f74840adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f7481d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b57b0 0x7f74840adbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50316/0 (socket says 172.21.15.81:50316) 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f7481d9b640 1 -- 172.21.15.81:0/3205553321 learned_addr learned my addr 172.21.15.81:0/3205553321 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f748259c640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f748406ef90 0x7f74841f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.397+0000 7f7482d9d640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74840ce1d0 0x7f74841f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f7481d9b640 1 -- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f748406ef90 msgr2=0x7f74841f7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:01.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f7481d9b640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f748406ef90 0x7f74841f7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f7481d9b640 1 -- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74840ce1d0 msgr2=0x7f74841f5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:01.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f7481d9b640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74840ce1d0 0x7f74841f5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:01.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f7481d9b640 1 -- 172.21.15.81:0/3205553321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74841f40b0 con 0x7f74840b57b0 2024-06-29T14:35:01.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f7481d9b640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b57b0 0x7f74840adbf0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7478034040 tx=0x7f7478003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:01.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f7472ffd640 1 -- 172.21.15.81:0/3205553321 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747803e030 con 0x7f74840b57b0 2024-06-29T14:35:01.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f7472ffd640 1 -- 172.21.15.81:0/3205553321 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7478003e00 con 0x7f74840b57b0 2024-06-29T14:35:01.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f748895b640 1 -- 172.21.15.81:0/3205553321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74840ae600 con 0x7f74840b57b0 2024-06-29T14:35:01.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f7472ffd640 1 -- 172.21.15.81:0/3205553321 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7478035040 con 0x7f74840b57b0 2024-06-29T14:35:01.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.398+0000 7f748895b640 1 -- 172.21.15.81:0/3205553321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74840ae320 con 0x7f74840b57b0 2024-06-29T14:35:01.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.399+0000 7f748895b640 1 -- 172.21.15.81:0/3205553321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f748406e2a0 con 0x7f74840b57b0 2024-06-29T14:35:01.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.400+0000 7f7472ffd640 1 -- 172.21.15.81:0/3205553321 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f7478041030 con 0x7f74840b57b0 2024-06-29T14:35:01.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.400+0000 7f7472ffd640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7464076420 0x7f7464078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:01.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.400+0000 7f7472ffd640 1 -- 172.21.15.81:0/3205553321 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f74780b8230 con 0x7f74840b57b0 2024-06-29T14:35:01.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.400+0000 7f748259c640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7464076420 0x7f7464078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:01.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.401+0000 7f748259c640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7464076420 0x7f7464078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7474005fd0 tx=0x7f7474005850 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:01.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.405+0000 7f7472ffd640 1 -- 172.21.15.81:0/3205553321 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f748406e2a0 con 0x7f74840b57b0 2024-06-29T14:35:01.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:01.622+0000 7f748895b640 1 -- 172.21.15.81:0/3205553321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f74840cf020 con 0x7f74840b57b0 2024-06-29T14:35:02.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.257+0000 7f7472ffd640 1 -- 172.21.15.81:0/3205553321 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7f74780818c0 con 0x7f74840b57b0 2024-06-29T14:35:02.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 -- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7464076420 msgr2=0x7f7464078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7464076420 0x7f7464078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7474005fd0 tx=0x7f7474005850 comp rx=0 tx=0).stop 2024-06-29T14:35:02.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 -- 172.21.15.81:0/3205553321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b57b0 msgr2=0x7f74840adbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b57b0 0x7f74840adbf0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7478034040 tx=0x7f7478003910 comp rx=0 tx=0).stop 2024-06-29T14:35:02.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 -- 172.21.15.81:0/3205553321 shutdown_connections 2024-06-29T14:35:02.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74840ce1d0 0x7f74841f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f7464076420 0x7f7464078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b57b0 0x7f74840adbf0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 --2- 172.21.15.81:0/3205553321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f748406ef90 0x7f74841f7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 -- 172.21.15.81:0/3205553321 >> 172.21.15.81:0/3205553321 conn(0x7f74840d0930 msgr2=0x7f74840cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:02.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 -- 172.21.15.81:0/3205553321 shutdown_connections 2024-06-29T14:35:02.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.261+0000 7f748895b640 1 -- 172.21.15.81:0/3205553321 wait complete. 2024-06-29T14:35:02.271 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:35:02.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:35:02.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.389+0000 7f2f2e963640 1 -- 172.21.15.81:0/4029045005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f280cf4d0 msgr2=0x7f2f28073030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.389+0000 7f2f2e963640 1 --2- 172.21.15.81:0/4029045005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f280cf4d0 0x7f2f28073030 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2f28070e50 tx=0x7f2f1c02a530 comp rx=0 tx=0).stop 2024-06-29T14:35:02.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.389+0000 7f2f2e963640 1 -- 172.21.15.81:0/4029045005 shutdown_connections 2024-06-29T14:35:02.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.389+0000 7f2f2e963640 1 --2- 172.21.15.81:0/4029045005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f280cf4d0 0x7f2f28073030 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.389+0000 7f2f2e963640 1 --2- 172.21.15.81:0/4029045005 >> v2:172.21.15.167:3300/0 conn(0x7f2f281412f0 0x7f2f280cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.389+0000 7f2f2e963640 1 -- 172.21.15.81:0/4029045005 >> 172.21.15.81:0/4029045005 conn(0x7f2f2806ccc0 msgr2=0x7f2f280b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:02.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.389+0000 7f2f2e963640 1 -- 172.21.15.81:0/4029045005 shutdown_connections 2024-06-29T14:35:02.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.389+0000 7f2f2e963640 1 -- 172.21.15.81:0/4029045005 wait complete. 2024-06-29T14:35:02.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.391+0000 7f2f2e963640 1 Processor -- start 2024-06-29T14:35:02.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.391+0000 7f2f2e963640 1 -- start start 2024-06-29T14:35:02.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f2e963640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f28073570 0x7f2f280a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:02.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f2e963640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f280cf4d0 0x7f2f28152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:02.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f2e963640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f281412f0 0x7f2f28152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:02.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f2e963640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2f280b3210 con 0x7f2f280cf4d0 2024-06-29T14:35:02.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f2e963640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2815f280 con 0x7f2f28073570 2024-06-29T14:35:02.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f2e963640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2f280b3390 con 0x7f2f281412f0 2024-06-29T14:35:02.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f280cf4d0 0x7f2f28152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:02.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f280cf4d0 0x7f2f28152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50342/0 (socket says 172.21.15.81:50342) 2024-06-29T14:35:02.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f277fe640 1 -- 172.21.15.81:0/207171362 learned_addr learned my addr 172.21.15.81:0/207171362 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f2cf00640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f281412f0 0x7f2f28152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f27fff640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f28073570 0x7f2f280a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f277fe640 1 -- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f281412f0 msgr2=0x7f2f28152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f277fe640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f281412f0 0x7f2f28152b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f277fe640 1 -- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f28073570 msgr2=0x7f2f280a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f277fe640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f28073570 0x7f2f280a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f277fe640 1 -- 172.21.15.81:0/207171362 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f28157ec0 con 0x7f2f280cf4d0 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f27fff640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f28073570 0x7f2f280a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f277fe640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f280cf4d0 0x7f2f28152600 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009920 tx=0x7f2f1c033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f24ff9640 1 -- 172.21.15.81:0/207171362 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c031d60 con 0x7f2f280cf4d0 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f24ff9640 1 -- 172.21.15.81:0/207171362 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f1c005020 con 0x7f2f280cf4d0 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.393+0000 7f2f2e963640 1 -- 172.21.15.81:0/207171362 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f280aec60 con 0x7f2f280cf4d0 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.394+0000 7f2f24ff9640 1 -- 172.21.15.81:0/207171362 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c005470 con 0x7f2f280cf4d0 2024-06-29T14:35:02.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.394+0000 7f2f2e963640 1 -- 172.21.15.81:0/207171362 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f28156080 con 0x7f2f280cf4d0 2024-06-29T14:35:02.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.395+0000 7f2f2e963640 1 -- 172.21.15.81:0/207171362 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f28155b10 con 0x7f2f280cf4d0 2024-06-29T14:35:02.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.395+0000 7f2f24ff9640 1 -- 172.21.15.81:0/207171362 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f2f1c0047a0 con 0x7f2f280cf4d0 2024-06-29T14:35:02.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.396+0000 7f2f24ff9640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f2f140763f0 0x7f2f14078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:02.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.396+0000 7f2f24ff9640 1 -- 172.21.15.81:0/207171362 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f2f1c0b8ca0 con 0x7f2f280cf4d0 2024-06-29T14:35:02.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.396+0000 7f2f27fff640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f2f140763f0 0x7f2f14078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:02.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.396+0000 7f2f27fff640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f2f140763f0 0x7f2f14078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800a870 tx=0x7f2f18008040 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:02.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.400+0000 7f2f24ff9640 1 -- 172.21.15.81:0/207171362 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2f1c03f030 con 0x7f2f280cf4d0 2024-06-29T14:35:02.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.615+0000 7f2f2e963640 1 -- 172.21.15.81:0/207171362 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f280b38b0 con 0x7f2f280cf4d0 2024-06-29T14:35:02.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.615+0000 7f2f24ff9640 1 -- 172.21.15.81:0/207171362 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5229 (secure 0 0 0) 0x7f2f1c0823b0 con 0x7f2f280cf4d0 2024-06-29T14:35:02.613 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:35:02.613 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":25,"btime":"2024-06-29T14:35:02:255148+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2980523942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2980523942},{"type":"v1","addr":"172.21.15.167:6836","nonce":2980523942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5635,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2317403375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2317403375},{"type":"v1","addr":"172.21.15.167:6837","nonce":2317403375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/2346221458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2346221458},{"type":"v1","addr":"172.21.15.81:6834","nonce":2346221458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:34:58.602103+0000","modified":"2024-06-29T14:35:01.623553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6033":{"gid":6033,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/2669979711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2669979711},{"type":"v1","addr":"172.21.15.81:6835","nonce":2669979711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6033,"qdb_cluster":[6033]},"id":3}]} 2024-06-29T14:35:02.613 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 25 2024-06-29T14:35:02.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 -- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f2f140763f0 msgr2=0x7f2f14078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f2f140763f0 0x7f2f14078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800a870 tx=0x7f2f18008040 comp rx=0 tx=0).stop 2024-06-29T14:35:02.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 -- 172.21.15.81:0/207171362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f280cf4d0 msgr2=0x7f2f28152600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f280cf4d0 0x7f2f28152600 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009920 tx=0x7f2f1c033d60 comp rx=0 tx=0).stop 2024-06-29T14:35:02.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 -- 172.21.15.81:0/207171362 shutdown_connections 2024-06-29T14:35:02.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f281412f0 0x7f2f28152b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f2f140763f0 0x7f2f14078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f280cf4d0 0x7f2f28152600 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 --2- 172.21.15.81:0/207171362 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f28073570 0x7f2f280a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 -- 172.21.15.81:0/207171362 >> 172.21.15.81:0/207171362 conn(0x7f2f2806ccc0 msgr2=0x7f2f280af6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:02.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 -- 172.21.15.81:0/207171362 shutdown_connections 2024-06-29T14:35:02.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.617+0000 7f2f2e963640 1 -- 172.21.15.81:0/207171362 wait complete. 2024-06-29T14:35:02.627 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:35:02.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.743+0000 7f5a7852a640 1 -- 172.21.15.81:0/44816381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a58004800 msgr2=0x7f5a58004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.743+0000 7f5a7852a640 1 --2- 172.21.15.81:0/44816381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a58004800 0x7f5a58004c10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f5a640016c0 tx=0x7f5a58005830 comp rx=0 tx=0).stop 2024-06-29T14:35:02.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.744+0000 7f5a7852a640 1 -- 172.21.15.81:0/44816381 shutdown_connections 2024-06-29T14:35:02.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.744+0000 7f5a7852a640 1 --2- 172.21.15.81:0/44816381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a58004800 0x7f5a58004c10 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.744+0000 7f5a7852a640 1 -- 172.21.15.81:0/44816381 >> 172.21.15.81:0/44816381 conn(0x7f5a700d0930 msgr2=0x7f5a700d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:02.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.744+0000 7f5a7852a640 1 -- 172.21.15.81:0/44816381 shutdown_connections 2024-06-29T14:35:02.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.744+0000 7f5a7852a640 1 -- 172.21.15.81:0/44816381 wait complete. 2024-06-29T14:35:02.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.745+0000 7f5a7852a640 1 Processor -- start 2024-06-29T14:35:02.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.745+0000 7f5a7852a640 1 -- start start 2024-06-29T14:35:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a7852a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a58004800 0x7f5a700aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a7852a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5a700ae0c0 0x7f5a7014e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a7852a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a7014f800 0x7f5a7014fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a7852a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700affa0 con 0x7f5a58004800 2024-06-29T14:35:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a7852a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700afe20 con 0x7f5a7014f800 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a7852a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5a700b0120 con 0x7f5a700ae0c0 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a762c6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a58004800 0x7f5a700aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a762c6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a58004800 0x7f5a700aef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50356/0 (socket says 172.21.15.81:50356) 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a762c6640 1 -- 172.21.15.81:0/964364357 learned_addr learned my addr 172.21.15.81:0/964364357 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a75ac5640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5a700ae0c0 0x7f5a7014e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a76ac7640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a7014f800 0x7f5a7014fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a762c6640 1 -- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5a700ae0c0 msgr2=0x7f5a7014e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a762c6640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5a700ae0c0 0x7f5a7014e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a762c6640 1 -- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a7014f800 msgr2=0x7f5a7014fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a762c6640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a7014f800 0x7f5a7014fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a762c6640 1 -- 172.21.15.81:0/964364357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a70150150 con 0x7f5a58004800 2024-06-29T14:35:02.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a762c6640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a58004800 0x7f5a700aef40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5a600117b0 tx=0x7f5a60011c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:02.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a6effd640 1 -- 172.21.15.81:0/964364357 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000fcf0 con 0x7f5a58004800 2024-06-29T14:35:02.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a6effd640 1 -- 172.21.15.81:0/964364357 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a60004030 con 0x7f5a58004800 2024-06-29T14:35:02.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a7852a640 1 -- 172.21.15.81:0/964364357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a700af870 con 0x7f5a58004800 2024-06-29T14:35:02.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a6effd640 1 -- 172.21.15.81:0/964364357 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a600044a0 con 0x7f5a58004800 2024-06-29T14:35:02.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.747+0000 7f5a7852a640 1 -- 172.21.15.81:0/964364357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a700b3de0 con 0x7f5a58004800 2024-06-29T14:35:02.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.749+0000 7f5a7852a640 1 -- 172.21.15.81:0/964364357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a7006e2a0 con 0x7f5a58004800 2024-06-29T14:35:02.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.749+0000 7f5a6effd640 1 -- 172.21.15.81:0/964364357 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f5a600035f0 con 0x7f5a58004800 2024-06-29T14:35:02.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.750+0000 7f5a6effd640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5a5c0763f0 0x7f5a5c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:02.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.750+0000 7f5a6effd640 1 -- 172.21.15.81:0/964364357 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f5a600986c0 con 0x7f5a58004800 2024-06-29T14:35:02.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.750+0000 7f5a75ac5640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5a5c0763f0 0x7f5a5c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:02.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.750+0000 7f5a75ac5640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5a5c0763f0 0x7f5a5c078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5a64032000 tx=0x7f5a64001190 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:02.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.754+0000 7f5a6effd640 1 -- 172.21.15.81:0/964364357 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5a7006e2a0 con 0x7f5a58004800 2024-06-29T14:35:02.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.934+0000 7f5a7852a640 1 -- 172.21.15.81:0/964364357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a7006c570 con 0x7f5a58004800 2024-06-29T14:35:02.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.935+0000 7f5a6effd640 1 -- 172.21.15.81:0/964364357 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+16671 (secure 0 0 0) 0x7f5a70150150 con 0x7f5a58004800 2024-06-29T14:35:02.933 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:35:02.933 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":61,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:35:00.231153+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:34:56.911140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-06-29T14:34:57.555296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:35:02.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 -- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5a5c0763f0 msgr2=0x7f5a5c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5a5c0763f0 0x7f5a5c078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5a64032000 tx=0x7f5a64001190 comp rx=0 tx=0).stop 2024-06-29T14:35:02.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 -- 172.21.15.81:0/964364357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a58004800 msgr2=0x7f5a700aef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:02.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a58004800 0x7f5a700aef40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5a600117b0 tx=0x7f5a60011c70 comp rx=0 tx=0).stop 2024-06-29T14:35:02.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 -- 172.21.15.81:0/964364357 shutdown_connections 2024-06-29T14:35:02.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a7014f800 0x7f5a7014fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5a700ae0c0 0x7f5a7014e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f5a5c0763f0 0x7f5a5c078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 --2- 172.21.15.81:0/964364357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a58004800 0x7f5a700aef40 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:02.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 -- 172.21.15.81:0/964364357 >> 172.21.15.81:0/964364357 conn(0x7f5a700d0930 msgr2=0x7f5a700b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:02.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 -- 172.21.15.81:0/964364357 shutdown_connections 2024-06-29T14:35:02.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:02.937+0000 7f5a7852a640 1 -- 172.21.15.81:0/964364357 wait complete. 2024-06-29T14:35:02.947 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:35:02.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:35:03.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.063+0000 7f09484c0640 1 -- 172.21.15.81:0/1089249676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0940074520 msgr2=0x7f094006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.063+0000 7f09484c0640 1 --2- 172.21.15.81:0/1089249676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0940074520 0x7f094006f140 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f093c005fd0 tx=0x7f093c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.063+0000 7f09484c0640 1 -- 172.21.15.81:0/1089249676 shutdown_connections 2024-06-29T14:35:03.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.063+0000 7f09484c0640 1 --2- 172.21.15.81:0/1089249676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0940074520 0x7f094006f140 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.063+0000 7f09484c0640 1 -- 172.21.15.81:0/1089249676 >> 172.21.15.81:0/1089249676 conn(0x7f0940072950 msgr2=0x7f09400b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.063+0000 7f09484c0640 1 -- 172.21.15.81:0/1089249676 shutdown_connections 2024-06-29T14:35:03.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.063+0000 7f09484c0640 1 -- 172.21.15.81:0/1089249676 wait complete. 2024-06-29T14:35:03.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.064+0000 7f09484c0640 1 Processor -- start 2024-06-29T14:35:03.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.064+0000 7f09484c0640 1 -- start start 2024-06-29T14:35:03.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f09484c0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f094006f680 0x7f0940157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f09484c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f094006fa90 0x7f094015d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f09484c0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0940074520 0x7f0940146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f09484c0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f094015b190 con 0x7f094006fa90 2024-06-29T14:35:03.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f09484c0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f094015b010 con 0x7f094006f680 2024-06-29T14:35:03.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f09484c0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f09400af310 con 0x7f0940074520 2024-06-29T14:35:03.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f0945a5b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f094006fa90 0x7f094015d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f0945a5b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f094006fa90 0x7f094015d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50376/0 (socket says 172.21.15.81:50376) 2024-06-29T14:35:03.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f0945a5b640 1 -- 172.21.15.81:0/157103479 learned_addr learned my addr 172.21.15.81:0/157103479 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:03.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f094625c640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f094006f680 0x7f0940157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f0946a5d640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0940074520 0x7f0940146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f0945a5b640 1 -- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0940074520 msgr2=0x7f0940146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f0945a5b640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0940074520 0x7f0940146070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f0945a5b640 1 -- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f094006f680 msgr2=0x7f0940157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f0945a5b640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f094006f680 0x7f0940157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f0945a5b640 1 -- 172.21.15.81:0/157103479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0940153390 con 0x7f094006fa90 2024-06-29T14:35:03.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.066+0000 7f0945a5b640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f094006fa90 0x7f094015d030 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f093000b410 tx=0x7f093000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.067+0000 7f0936ffd640 1 -- 172.21.15.81:0/157103479 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930015020 con 0x7f094006fa90 2024-06-29T14:35:03.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.067+0000 7f0936ffd640 1 -- 172.21.15.81:0/157103479 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0930011920 con 0x7f094006fa90 2024-06-29T14:35:03.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.067+0000 7f09484c0640 1 -- 172.21.15.81:0/157103479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0940153670 con 0x7f094006fa90 2024-06-29T14:35:03.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.067+0000 7f0936ffd640 1 -- 172.21.15.81:0/157103479 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0930011d70 con 0x7f094006fa90 2024-06-29T14:35:03.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.067+0000 7f09484c0640 1 -- 172.21.15.81:0/157103479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0940153c30 con 0x7f094006fa90 2024-06-29T14:35:03.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.068+0000 7f09484c0640 1 -- 172.21.15.81:0/157103479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f094015a610 con 0x7f094006fa90 2024-06-29T14:35:03.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.068+0000 7f0936ffd640 1 -- 172.21.15.81:0/157103479 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f0930018020 con 0x7f094006fa90 2024-06-29T14:35:03.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.069+0000 7f0936ffd640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f09240763f0 0x7f0924078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.069+0000 7f0936ffd640 1 -- 172.21.15.81:0/157103479 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f09300978c0 con 0x7f094006fa90 2024-06-29T14:35:03.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.069+0000 7f094625c640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f09240763f0 0x7f0924078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.070+0000 7f094625c640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f09240763f0 0x7f0924078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f093c033000 tx=0x7f093c004730 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.073+0000 7f0936ffd640 1 -- 172.21.15.81:0/157103479 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f094015a610 con 0x7f094006fa90 2024-06-29T14:35:03.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.251+0000 7f09484c0640 1 -- 172.21.15.81:0/157103479 --> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0940153120 con 0x7f09240763f0 2024-06-29T14:35:03.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.257+0000 7f0936ffd640 1 -- 172.21.15.81:0/157103479 <== mgr.5425 v2:172.21.15.167:6832/1090782941 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251952 (secure 0 0 0) 0x7f0940153120 con 0x7f09240763f0 2024-06-29T14:35:03.258 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:35:03.260 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:35:03.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 -- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f09240763f0 msgr2=0x7f0924078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f09240763f0 0x7f0924078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f093c033000 tx=0x7f093c004730 comp rx=0 tx=0).stop 2024-06-29T14:35:03.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 -- 172.21.15.81:0/157103479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f094006fa90 msgr2=0x7f094015d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f094006fa90 0x7f094015d030 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f093000b410 tx=0x7f093000b8d0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 -- 172.21.15.81:0/157103479 shutdown_connections 2024-06-29T14:35:03.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0940074520 0x7f0940146070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f09240763f0 0x7f0924078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f094006fa90 0x7f094015d030 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 --2- 172.21.15.81:0/157103479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f094006f680 0x7f0940157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 -- 172.21.15.81:0/157103479 >> 172.21.15.81:0/157103479 conn(0x7f0940072950 msgr2=0x7f0940068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 -- 172.21.15.81:0/157103479 shutdown_connections 2024-06-29T14:35:03.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.260+0000 7f09484c0640 1 -- 172.21.15.81:0/157103479 wait complete. 2024-06-29T14:35:03.269 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-06-29T14:35:02.850922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":170,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":300720,"kb_used_data":6828,"kb_used_omap":12,"kb_used_meta":293747,"kb_avail":754674000,"statfs":{"total":773094113280,"available":772786176000,"internally_reserved":0,"allocated":6991872,"data_stored":3881388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":300797490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.627567"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242215+0000","last_change":"2024-06-29T14:34:59.254713+0000","last_active":"2024-06-29T14:35:00.242215+0000","last_peered":"2024-06-29T14:35:00.242215+0000","last_clean":"2024-06-29T14:35:00.242215+0000","last_became_active":"2024-06-29T14:34:59.254343+0000","last_became_peered":"2024-06-29T14:34:59.254343+0000","last_unstale":"2024-06-29T14:35:00.242215+0000","last_undegraded":"2024-06-29T14:35:00.242215+0000","last_fullsized":"2024-06-29T14:35:00.242215+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:43:44.063626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236545+0000","last_change":"2024-06-29T14:34:59.252912+0000","last_active":"2024-06-29T14:35:00.236545+0000","last_peered":"2024-06-29T14:35:00.236545+0000","last_clean":"2024-06-29T14:35:00.236545+0000","last_became_active":"2024-06-29T14:34:59.252563+0000","last_became_peered":"2024-06-29T14:34:59.252563+0000","last_unstale":"2024-06-29T14:35:00.236545+0000","last_undegraded":"2024-06-29T14:35:00.236545+0000","last_fullsized":"2024-06-29T14:35:00.236545+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:53.321271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235425+0000","last_change":"2024-06-29T14:34:59.244690+0000","last_active":"2024-06-29T14:35:00.235425+0000","last_peered":"2024-06-29T14:35:00.235425+0000","last_clean":"2024-06-29T14:35:00.235425+0000","last_became_active":"2024-06-29T14:34:59.244314+0000","last_became_peered":"2024-06-29T14:34:59.244314+0000","last_unstale":"2024-06-29T14:35:00.235425+0000","last_undegraded":"2024-06-29T14:35:00.235425+0000","last_fullsized":"2024-06-29T14:35:00.235425+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:00.875369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241978+0000","last_change":"2024-06-29T14:34:59.255746+0000","last_active":"2024-06-29T14:35:00.241978+0000","last_peered":"2024-06-29T14:35:00.241978+0000","last_clean":"2024-06-29T14:35:00.241978+0000","last_became_active":"2024-06-29T14:34:59.255366+0000","last_became_peered":"2024-06-29T14:34:59.255366+0000","last_unstale":"2024-06-29T14:35:00.241978+0000","last_undegraded":"2024-06-29T14:35:00.241978+0000","last_fullsized":"2024-06-29T14:35:00.241978+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:04.477358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236207+0000","last_change":"2024-06-29T14:34:59.249674+0000","last_active":"2024-06-29T14:35:00.236207+0000","last_peered":"2024-06-29T14:35:00.236207+0000","last_clean":"2024-06-29T14:35:00.236207+0000","last_became_active":"2024-06-29T14:34:59.249376+0000","last_became_peered":"2024-06-29T14:34:59.249376+0000","last_unstale":"2024-06-29T14:35:00.236207+0000","last_undegraded":"2024-06-29T14:35:00.236207+0000","last_fullsized":"2024-06-29T14:35:00.236207+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:44.772104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241544+0000","last_change":"2024-06-29T14:34:59.253473+0000","last_active":"2024-06-29T14:35:00.241544+0000","last_peered":"2024-06-29T14:35:00.241544+0000","last_clean":"2024-06-29T14:35:00.241544+0000","last_became_active":"2024-06-29T14:34:59.253152+0000","last_became_peered":"2024-06-29T14:34:59.253152+0000","last_unstale":"2024-06-29T14:35:00.241544+0000","last_undegraded":"2024-06-29T14:35:00.241544+0000","last_fullsized":"2024-06-29T14:35:00.241544+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:52.278087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241501+0000","last_change":"2024-06-29T14:34:59.262000+0000","last_active":"2024-06-29T14:35:00.241501+0000","last_peered":"2024-06-29T14:35:00.241501+0000","last_clean":"2024-06-29T14:35:00.241501+0000","last_became_active":"2024-06-29T14:34:59.261666+0000","last_became_peered":"2024-06-29T14:34:59.261666+0000","last_unstale":"2024-06-29T14:35:00.241501+0000","last_undegraded":"2024-06-29T14:35:00.241501+0000","last_fullsized":"2024-06-29T14:35:00.241501+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:37.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240761+0000","last_change":"2024-06-29T14:34:59.254813+0000","last_active":"2024-06-29T14:35:00.240761+0000","last_peered":"2024-06-29T14:35:00.240761+0000","last_clean":"2024-06-29T14:35:00.240761+0000","last_became_active":"2024-06-29T14:34:59.254484+0000","last_became_peered":"2024-06-29T14:34:59.254484+0000","last_unstale":"2024-06-29T14:35:00.240761+0000","last_undegraded":"2024-06-29T14:35:00.240761+0000","last_fullsized":"2024-06-29T14:35:00.240761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:40.033180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241285+0000","last_change":"2024-06-29T14:34:59.262064+0000","last_active":"2024-06-29T14:35:00.241285+0000","last_peered":"2024-06-29T14:35:00.241285+0000","last_clean":"2024-06-29T14:35:00.241285+0000","last_became_active":"2024-06-29T14:34:59.261648+0000","last_became_peered":"2024-06-29T14:34:59.261648+0000","last_unstale":"2024-06-29T14:35:00.241285+0000","last_undegraded":"2024-06-29T14:35:00.241285+0000","last_fullsized":"2024-06-29T14:35:00.241285+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:03.348772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241708+0000","last_change":"2024-06-29T14:34:59.255791+0000","last_active":"2024-06-29T14:35:00.241708+0000","last_peered":"2024-06-29T14:35:00.241708+0000","last_clean":"2024-06-29T14:35:00.241708+0000","last_became_active":"2024-06-29T14:34:59.255326+0000","last_became_peered":"2024-06-29T14:34:59.255326+0000","last_unstale":"2024-06-29T14:35:00.241708+0000","last_undegraded":"2024-06-29T14:35:00.241708+0000","last_fullsized":"2024-06-29T14:35:00.241708+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:51:43.053123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241167+0000","last_change":"2024-06-29T14:34:59.261510+0000","last_active":"2024-06-29T14:35:00.241167+0000","last_peered":"2024-06-29T14:35:00.241167+0000","last_clean":"2024-06-29T14:35:00.241167+0000","last_became_active":"2024-06-29T14:34:59.261182+0000","last_became_peered":"2024-06-29T14:34:59.261182+0000","last_unstale":"2024-06-29T14:35:00.241167+0000","last_undegraded":"2024-06-29T14:35:00.241167+0000","last_fullsized":"2024-06-29T14:35:00.241167+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:31:47.110087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236894+0000","last_change":"2024-06-29T14:34:59.254087+0000","last_active":"2024-06-29T14:35:00.236894+0000","last_peered":"2024-06-29T14:35:00.236894+0000","last_clean":"2024-06-29T14:35:00.236894+0000","last_became_active":"2024-06-29T14:34:59.253563+0000","last_became_peered":"2024-06-29T14:34:59.253563+0000","last_unstale":"2024-06-29T14:35:00.236894+0000","last_undegraded":"2024-06-29T14:35:00.236894+0000","last_fullsized":"2024-06-29T14:35:00.236894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:34:35.863173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235307+0000","last_change":"2024-06-29T14:34:59.259565+0000","last_active":"2024-06-29T14:35:00.235307+0000","last_peered":"2024-06-29T14:35:00.235307+0000","last_clean":"2024-06-29T14:35:00.235307+0000","last_became_active":"2024-06-29T14:34:59.259231+0000","last_became_peered":"2024-06-29T14:34:59.259231+0000","last_unstale":"2024-06-29T14:35:00.235307+0000","last_undegraded":"2024-06-29T14:35:00.235307+0000","last_fullsized":"2024-06-29T14:35:00.235307+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:44.349582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240591+0000","last_change":"2024-06-29T14:34:59.261496+0000","last_active":"2024-06-29T14:35:00.240591+0000","last_peered":"2024-06-29T14:35:00.240591+0000","last_clean":"2024-06-29T14:35:00.240591+0000","last_became_active":"2024-06-29T14:34:59.261153+0000","last_became_peered":"2024-06-29T14:34:59.261153+0000","last_unstale":"2024-06-29T14:35:00.240591+0000","last_undegraded":"2024-06-29T14:35:00.240591+0000","last_fullsized":"2024-06-29T14:35:00.240591+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:37.045557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241520+0000","last_change":"2024-06-29T14:34:59.239398+0000","last_active":"2024-06-29T14:35:00.241520+0000","last_peered":"2024-06-29T14:35:00.241520+0000","last_clean":"2024-06-29T14:35:00.241520+0000","last_became_active":"2024-06-29T14:34:59.239175+0000","last_became_peered":"2024-06-29T14:34:59.239175+0000","last_unstale":"2024-06-29T14:35:00.241520+0000","last_undegraded":"2024-06-29T14:35:00.241520+0000","last_fullsized":"2024-06-29T14:35:00.241520+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:41:11.700373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241083+0000","last_change":"2024-06-29T14:34:59.250840+0000","last_active":"2024-06-29T14:35:00.241083+0000","last_peered":"2024-06-29T14:35:00.241083+0000","last_clean":"2024-06-29T14:35:00.241083+0000","last_became_active":"2024-06-29T14:34:59.250518+0000","last_became_peered":"2024-06-29T14:34:59.250518+0000","last_unstale":"2024-06-29T14:35:00.241083+0000","last_undegraded":"2024-06-29T14:35:00.241083+0000","last_fullsized":"2024-06-29T14:35:00.241083+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:41:50.447752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235940+0000","last_change":"2024-06-29T14:34:59.245145+0000","last_active":"2024-06-29T14:35:00.235940+0000","last_peered":"2024-06-29T14:35:00.235940+0000","last_clean":"2024-06-29T14:35:00.235940+0000","last_became_active":"2024-06-29T14:34:59.244820+0000","last_became_peered":"2024-06-29T14:34:59.244820+0000","last_unstale":"2024-06-29T14:35:00.235940+0000","last_undegraded":"2024-06-29T14:35:00.235940+0000","last_fullsized":"2024-06-29T14:35:00.235940+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:19:20.489371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236167+0000","last_change":"2024-06-29T14:34:59.261158+0000","last_active":"2024-06-29T14:35:00.236167+0000","last_peered":"2024-06-29T14:35:00.236167+0000","last_clean":"2024-06-29T14:35:00.236167+0000","last_became_active":"2024-06-29T14:34:59.260819+0000","last_became_peered":"2024-06-29T14:34:59.260819+0000","last_unstale":"2024-06-29T14:35:00.236167+0000","last_undegraded":"2024-06-29T14:35:00.236167+0000","last_fullsized":"2024-06-29T14:35:00.236167+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:07:52.274841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241998+0000","last_change":"2024-06-29T14:34:59.252099+0000","last_active":"2024-06-29T14:35:00.241998+0000","last_peered":"2024-06-29T14:35:00.241998+0000","last_clean":"2024-06-29T14:35:00.241998+0000","last_became_active":"2024-06-29T14:34:59.251588+0000","last_became_peered":"2024-06-29T14:34:59.251588+0000","last_unstale":"2024-06-29T14:35:00.241998+0000","last_undegraded":"2024-06-29T14:35:00.241998+0000","last_fullsized":"2024-06-29T14:35:00.241998+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:03.348772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235289+0000","last_change":"2024-06-29T14:34:59.250569+0000","last_active":"2024-06-29T14:35:00.235289+0000","last_peered":"2024-06-29T14:35:00.235289+0000","last_clean":"2024-06-29T14:35:00.235289+0000","last_became_active":"2024-06-29T14:34:59.250252+0000","last_became_peered":"2024-06-29T14:34:59.250252+0000","last_unstale":"2024-06-29T14:35:00.235289+0000","last_undegraded":"2024-06-29T14:35:00.235289+0000","last_fullsized":"2024-06-29T14:35:00.235289+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:55:44.770475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241982+0000","last_change":"2024-06-29T14:34:59.254219+0000","last_active":"2024-06-29T14:35:00.241982+0000","last_peered":"2024-06-29T14:35:00.241982+0000","last_clean":"2024-06-29T14:35:00.241982+0000","last_became_active":"2024-06-29T14:34:59.253940+0000","last_became_peered":"2024-06-29T14:34:59.253940+0000","last_unstale":"2024-06-29T14:35:00.241982+0000","last_undegraded":"2024-06-29T14:35:00.241982+0000","last_fullsized":"2024-06-29T14:35:00.241982+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:32.408127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240755+0000","last_change":"2024-06-29T14:34:59.255662+0000","last_active":"2024-06-29T14:35:00.240755+0000","last_peered":"2024-06-29T14:35:00.240755+0000","last_clean":"2024-06-29T14:35:00.240755+0000","last_became_active":"2024-06-29T14:34:59.255214+0000","last_became_peered":"2024-06-29T14:34:59.255214+0000","last_unstale":"2024-06-29T14:35:00.240755+0000","last_undegraded":"2024-06-29T14:35:00.240755+0000","last_fullsized":"2024-06-29T14:35:00.240755+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:41:50.447752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241030+0000","last_change":"2024-06-29T14:34:59.254888+0000","last_active":"2024-06-29T14:35:00.241030+0000","last_peered":"2024-06-29T14:35:00.241030+0000","last_clean":"2024-06-29T14:35:00.241030+0000","last_became_active":"2024-06-29T14:34:59.254162+0000","last_became_peered":"2024-06-29T14:34:59.254162+0000","last_unstale":"2024-06-29T14:35:00.241030+0000","last_undegraded":"2024-06-29T14:35:00.241030+0000","last_fullsized":"2024-06-29T14:35:00.241030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:40.033180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240914+0000","last_change":"2024-06-29T14:34:59.249930+0000","last_active":"2024-06-29T14:35:00.240914+0000","last_peered":"2024-06-29T14:35:00.240914+0000","last_clean":"2024-06-29T14:35:00.240914+0000","last_became_active":"2024-06-29T14:34:59.249613+0000","last_became_peered":"2024-06-29T14:34:59.249613+0000","last_unstale":"2024-06-29T14:35:00.240914+0000","last_undegraded":"2024-06-29T14:35:00.240914+0000","last_fullsized":"2024-06-29T14:35:00.240914+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:29.407691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242018+0000","last_change":"2024-06-29T14:34:59.255472+0000","last_active":"2024-06-29T14:35:00.242018+0000","last_peered":"2024-06-29T14:35:00.242018+0000","last_clean":"2024-06-29T14:35:00.242018+0000","last_became_active":"2024-06-29T14:34:59.252469+0000","last_became_peered":"2024-06-29T14:34:59.252469+0000","last_unstale":"2024-06-29T14:35:00.242018+0000","last_undegraded":"2024-06-29T14:35:00.242018+0000","last_fullsized":"2024-06-29T14:35:00.242018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:34:57.120364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235100+0000","last_change":"2024-06-29T14:34:59.258912+0000","last_active":"2024-06-29T14:35:00.235100+0000","last_peered":"2024-06-29T14:35:00.235100+0000","last_clean":"2024-06-29T14:35:00.235100+0000","last_became_active":"2024-06-29T14:34:59.258575+0000","last_became_peered":"2024-06-29T14:34:59.258575+0000","last_unstale":"2024-06-29T14:35:00.235100+0000","last_undegraded":"2024-06-29T14:35:00.235100+0000","last_fullsized":"2024-06-29T14:35:00.235100+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:05.947594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235555+0000","last_change":"2024-06-29T14:34:59.245507+0000","last_active":"2024-06-29T14:35:00.235555+0000","last_peered":"2024-06-29T14:35:00.235555+0000","last_clean":"2024-06-29T14:35:00.235555+0000","last_became_active":"2024-06-29T14:34:59.245231+0000","last_became_peered":"2024-06-29T14:34:59.245231+0000","last_unstale":"2024-06-29T14:35:00.235555+0000","last_undegraded":"2024-06-29T14:35:00.235555+0000","last_fullsized":"2024-06-29T14:35:00.235555+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:20:01.535487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.237028+0000","last_change":"2024-06-29T14:34:59.254379+0000","last_active":"2024-06-29T14:35:00.237028+0000","last_peered":"2024-06-29T14:35:00.237028+0000","last_clean":"2024-06-29T14:35:00.237028+0000","last_became_active":"2024-06-29T14:34:59.253497+0000","last_became_peered":"2024-06-29T14:34:59.253497+0000","last_unstale":"2024-06-29T14:35:00.237028+0000","last_undegraded":"2024-06-29T14:35:00.237028+0000","last_fullsized":"2024-06-29T14:35:00.237028+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:49:38.824249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241755+0000","last_change":"2024-06-29T14:34:59.250999+0000","last_active":"2024-06-29T14:35:00.241755+0000","last_peered":"2024-06-29T14:35:00.241755+0000","last_clean":"2024-06-29T14:35:00.241755+0000","last_became_active":"2024-06-29T14:34:59.250633+0000","last_became_peered":"2024-06-29T14:34:59.250633+0000","last_unstale":"2024-06-29T14:35:00.241755+0000","last_undegraded":"2024-06-29T14:35:00.241755+0000","last_fullsized":"2024-06-29T14:35:00.241755+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:52:52.718656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235709+0000","last_change":"2024-06-29T14:34:59.247093+0000","last_active":"2024-06-29T14:35:00.235709+0000","last_peered":"2024-06-29T14:35:00.235709+0000","last_clean":"2024-06-29T14:35:00.235709+0000","last_became_active":"2024-06-29T14:34:59.246722+0000","last_became_peered":"2024-06-29T14:34:59.246722+0000","last_unstale":"2024-06-29T14:35:00.235709+0000","last_undegraded":"2024-06-29T14:35:00.235709+0000","last_fullsized":"2024-06-29T14:35:00.235709+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:21.719173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241654+0000","last_change":"2024-06-29T14:34:59.245003+0000","last_active":"2024-06-29T14:35:00.241654+0000","last_peered":"2024-06-29T14:35:00.241654+0000","last_clean":"2024-06-29T14:35:00.241654+0000","last_became_active":"2024-06-29T14:34:59.244673+0000","last_became_peered":"2024-06-29T14:34:59.244673+0000","last_unstale":"2024-06-29T14:35:00.241654+0000","last_undegraded":"2024-06-29T14:35:00.241654+0000","last_fullsized":"2024-06-29T14:35:00.241654+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:12.388505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235084+0000","last_change":"2024-06-29T14:34:59.257956+0000","last_active":"2024-06-29T14:35:00.235084+0000","last_peered":"2024-06-29T14:35:00.235084+0000","last_clean":"2024-06-29T14:35:00.235084+0000","last_became_active":"2024-06-29T14:34:59.257666+0000","last_became_peered":"2024-06-29T14:34:59.257666+0000","last_unstale":"2024-06-29T14:35:00.235084+0000","last_undegraded":"2024-06-29T14:35:00.235084+0000","last_fullsized":"2024-06-29T14:35:00.235084+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:38.152457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240506+0000","last_change":"2024-06-29T14:34:59.258398+0000","last_active":"2024-06-29T14:35:00.240506+0000","last_peered":"2024-06-29T14:35:00.240506+0000","last_clean":"2024-06-29T14:35:00.240506+0000","last_became_active":"2024-06-29T14:34:59.258078+0000","last_became_peered":"2024-06-29T14:34:59.258078+0000","last_unstale":"2024-06-29T14:35:00.240506+0000","last_undegraded":"2024-06-29T14:35:00.240506+0000","last_fullsized":"2024-06-29T14:35:00.240506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:21.389411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"60'4","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242094+0000","last_change":"2024-06-29T14:34:58.248413+0000","last_active":"2024-06-29T14:35:00.242094+0000","last_peered":"2024-06-29T14:35:00.242094+0000","last_clean":"2024-06-29T14:35:00.242094+0000","last_became_active":"2024-06-29T14:34:58.248108+0000","last_became_peered":"2024-06-29T14:34:58.248108+0000","last_unstale":"2024-06-29T14:35:00.242094+0000","last_undegraded":"2024-06-29T14:35:00.242094+0000","last_fullsized":"2024-06-29T14:35:00.242094+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:28.882848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241702+0000","last_change":"2024-06-29T14:34:59.257583+0000","last_active":"2024-06-29T14:35:00.241702+0000","last_peered":"2024-06-29T14:35:00.241702+0000","last_clean":"2024-06-29T14:35:00.241702+0000","last_became_active":"2024-06-29T14:34:59.257226+0000","last_became_peered":"2024-06-29T14:34:59.257226+0000","last_unstale":"2024-06-29T14:35:00.241702+0000","last_undegraded":"2024-06-29T14:35:00.241702+0000","last_fullsized":"2024-06-29T14:35:00.241702+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:18.320387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"60'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236172+0000","last_change":"2024-06-29T14:34:58.246164+0000","last_active":"2024-06-29T14:35:00.236172+0000","last_peered":"2024-06-29T14:35:00.236172+0000","last_clean":"2024-06-29T14:35:00.236172+0000","last_became_active":"2024-06-29T14:34:58.245857+0000","last_became_peered":"2024-06-29T14:34:58.245857+0000","last_unstale":"2024-06-29T14:35:00.236172+0000","last_undegraded":"2024-06-29T14:35:00.236172+0000","last_fullsized":"2024-06-29T14:35:00.236172+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:49.115624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.237055+0000","last_change":"2024-06-29T14:34:59.257401+0000","last_active":"2024-06-29T14:35:00.237055+0000","last_peered":"2024-06-29T14:35:00.237055+0000","last_clean":"2024-06-29T14:35:00.237055+0000","last_became_active":"2024-06-29T14:34:59.257049+0000","last_became_peered":"2024-06-29T14:34:59.257049+0000","last_unstale":"2024-06-29T14:35:00.237055+0000","last_undegraded":"2024-06-29T14:35:00.237055+0000","last_fullsized":"2024-06-29T14:35:00.237055+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:41.763922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241031+0000","last_change":"2024-06-29T14:34:58.245297+0000","last_active":"2024-06-29T14:35:00.241031+0000","last_peered":"2024-06-29T14:35:00.241031+0000","last_clean":"2024-06-29T14:35:00.241031+0000","last_became_active":"2024-06-29T14:34:58.244945+0000","last_became_peered":"2024-06-29T14:34:58.244945+0000","last_unstale":"2024-06-29T14:35:00.241031+0000","last_undegraded":"2024-06-29T14:35:00.241031+0000","last_fullsized":"2024-06-29T14:35:00.241031+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:15.366522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240902+0000","last_change":"2024-06-29T14:34:59.255529+0000","last_active":"2024-06-29T14:35:00.240902+0000","last_peered":"2024-06-29T14:35:00.240902+0000","last_clean":"2024-06-29T14:35:00.240902+0000","last_became_active":"2024-06-29T14:34:59.255224+0000","last_became_peered":"2024-06-29T14:34:59.255224+0000","last_unstale":"2024-06-29T14:35:00.240902+0000","last_undegraded":"2024-06-29T14:35:00.240902+0000","last_fullsized":"2024-06-29T14:35:00.240902+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:36.215441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241803+0000","last_change":"2024-06-29T14:34:58.246382+0000","last_active":"2024-06-29T14:35:00.241803+0000","last_peered":"2024-06-29T14:35:00.241803+0000","last_clean":"2024-06-29T14:35:00.241803+0000","last_became_active":"2024-06-29T14:34:58.246079+0000","last_became_peered":"2024-06-29T14:34:58.246079+0000","last_unstale":"2024-06-29T14:35:00.241803+0000","last_undegraded":"2024-06-29T14:35:00.241803+0000","last_fullsized":"2024-06-29T14:35:00.241803+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:17.782537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235837+0000","last_change":"2024-06-29T14:34:59.250681+0000","last_active":"2024-06-29T14:35:00.235837+0000","last_peered":"2024-06-29T14:35:00.235837+0000","last_clean":"2024-06-29T14:35:00.235837+0000","last_became_active":"2024-06-29T14:34:59.247904+0000","last_became_peered":"2024-06-29T14:34:59.247904+0000","last_unstale":"2024-06-29T14:35:00.235837+0000","last_undegraded":"2024-06-29T14:35:00.235837+0000","last_fullsized":"2024-06-29T14:35:00.235837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:08.456231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"60'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242137+0000","last_change":"2024-06-29T14:34:58.245028+0000","last_active":"2024-06-29T14:35:00.242137+0000","last_peered":"2024-06-29T14:35:00.242137+0000","last_clean":"2024-06-29T14:35:00.242137+0000","last_became_active":"2024-06-29T14:34:58.244722+0000","last_became_peered":"2024-06-29T14:34:58.244722+0000","last_unstale":"2024-06-29T14:35:00.242137+0000","last_undegraded":"2024-06-29T14:35:00.242137+0000","last_fullsized":"2024-06-29T14:35:00.242137+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:52.316090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240884+0000","last_change":"2024-06-29T14:34:59.261932+0000","last_active":"2024-06-29T14:35:00.240884+0000","last_peered":"2024-06-29T14:35:00.240884+0000","last_clean":"2024-06-29T14:35:00.240884+0000","last_became_active":"2024-06-29T14:34:59.260450+0000","last_became_peered":"2024-06-29T14:34:59.260450+0000","last_unstale":"2024-06-29T14:35:00.240884+0000","last_undegraded":"2024-06-29T14:35:00.240884+0000","last_fullsized":"2024-06-29T14:35:00.240884+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:09.077088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"60'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235837+0000","last_change":"2024-06-29T14:34:58.240292+0000","last_active":"2024-06-29T14:35:00.235837+0000","last_peered":"2024-06-29T14:35:00.235837+0000","last_clean":"2024-06-29T14:35:00.235837+0000","last_became_active":"2024-06-29T14:34:58.239947+0000","last_became_peered":"2024-06-29T14:34:58.239947+0000","last_unstale":"2024-06-29T14:35:00.235837+0000","last_undegraded":"2024-06-29T14:35:00.235837+0000","last_fullsized":"2024-06-29T14:35:00.235837+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:17.431157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240908+0000","last_change":"2024-06-29T14:34:59.255535+0000","last_active":"2024-06-29T14:35:00.240908+0000","last_peered":"2024-06-29T14:35:00.240908+0000","last_clean":"2024-06-29T14:35:00.240908+0000","last_became_active":"2024-06-29T14:34:59.255152+0000","last_became_peered":"2024-06-29T14:34:59.255152+0000","last_unstale":"2024-06-29T14:35:00.240908+0000","last_undegraded":"2024-06-29T14:35:00.240908+0000","last_fullsized":"2024-06-29T14:35:00.240908+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:18.320387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235749+0000","last_change":"2024-06-29T14:34:58.246611+0000","last_active":"2024-06-29T14:35:00.235749+0000","last_peered":"2024-06-29T14:35:00.235749+0000","last_clean":"2024-06-29T14:35:00.235749+0000","last_became_active":"2024-06-29T14:34:58.246261+0000","last_became_peered":"2024-06-29T14:34:58.246261+0000","last_unstale":"2024-06-29T14:35:00.235749+0000","last_undegraded":"2024-06-29T14:35:00.235749+0000","last_fullsized":"2024-06-29T14:35:00.235749+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:09:45.439479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240951+0000","last_change":"2024-06-29T14:34:59.257352+0000","last_active":"2024-06-29T14:35:00.240951+0000","last_peered":"2024-06-29T14:35:00.240951+0000","last_clean":"2024-06-29T14:35:00.240951+0000","last_became_active":"2024-06-29T14:34:59.254553+0000","last_became_peered":"2024-06-29T14:34:59.254553+0000","last_unstale":"2024-06-29T14:35:00.240951+0000","last_undegraded":"2024-06-29T14:35:00.240951+0000","last_fullsized":"2024-06-29T14:35:00.240951+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:37.045557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241008+0000","last_change":"2024-06-29T14:34:58.252416+0000","last_active":"2024-06-29T14:35:00.241008+0000","last_peered":"2024-06-29T14:35:00.241008+0000","last_clean":"2024-06-29T14:35:00.241008+0000","last_became_active":"2024-06-29T14:34:58.252134+0000","last_became_peered":"2024-06-29T14:34:58.252134+0000","last_unstale":"2024-06-29T14:35:00.241008+0000","last_undegraded":"2024-06-29T14:35:00.241008+0000","last_fullsized":"2024-06-29T14:35:00.241008+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:35:30.509733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236034+0000","last_change":"2024-06-29T14:34:59.252708+0000","last_active":"2024-06-29T14:35:00.236034+0000","last_peered":"2024-06-29T14:35:00.236034+0000","last_clean":"2024-06-29T14:35:00.236034+0000","last_became_active":"2024-06-29T14:34:59.252379+0000","last_became_peered":"2024-06-29T14:34:59.252379+0000","last_unstale":"2024-06-29T14:35:00.236034+0000","last_undegraded":"2024-06-29T14:35:00.236034+0000","last_fullsized":"2024-06-29T14:35:00.236034+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:04.197860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236167+0000","last_change":"2024-06-29T14:34:58.247292+0000","last_active":"2024-06-29T14:35:00.236167+0000","last_peered":"2024-06-29T14:35:00.236167+0000","last_clean":"2024-06-29T14:35:00.236167+0000","last_became_active":"2024-06-29T14:34:58.246897+0000","last_became_peered":"2024-06-29T14:34:58.246897+0000","last_unstale":"2024-06-29T14:35:00.236167+0000","last_undegraded":"2024-06-29T14:35:00.236167+0000","last_fullsized":"2024-06-29T14:35:00.236167+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:10.240446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242213+0000","last_change":"2024-06-29T14:34:59.249506+0000","last_active":"2024-06-29T14:35:00.242213+0000","last_peered":"2024-06-29T14:35:00.242213+0000","last_clean":"2024-06-29T14:35:00.242213+0000","last_became_active":"2024-06-29T14:34:59.248397+0000","last_became_peered":"2024-06-29T14:34:59.248397+0000","last_unstale":"2024-06-29T14:35:00.242213+0000","last_undegraded":"2024-06-29T14:35:00.242213+0000","last_fullsized":"2024-06-29T14:35:00.242213+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:09.991806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241961+0000","last_change":"2024-06-29T14:34:58.247221+0000","last_active":"2024-06-29T14:35:00.241961+0000","last_peered":"2024-06-29T14:35:00.241961+0000","last_clean":"2024-06-29T14:35:00.241961+0000","last_became_active":"2024-06-29T14:34:58.246901+0000","last_became_peered":"2024-06-29T14:34:58.246901+0000","last_unstale":"2024-06-29T14:35:00.241961+0000","last_undegraded":"2024-06-29T14:35:00.241961+0000","last_fullsized":"2024-06-29T14:35:00.241961+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:30:01.369451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241827+0000","last_change":"2024-06-29T14:34:59.257606+0000","last_active":"2024-06-29T14:35:00.241827+0000","last_peered":"2024-06-29T14:35:00.241827+0000","last_clean":"2024-06-29T14:35:00.241827+0000","last_became_active":"2024-06-29T14:34:59.257289+0000","last_became_peered":"2024-06-29T14:34:59.257289+0000","last_unstale":"2024-06-29T14:35:00.241827+0000","last_undegraded":"2024-06-29T14:35:00.241827+0000","last_fullsized":"2024-06-29T14:35:00.241827+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:37.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"60'3","reported_seq":17,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242232+0000","last_change":"2024-06-29T14:34:58.249391+0000","last_active":"2024-06-29T14:35:00.242232+0000","last_peered":"2024-06-29T14:35:00.242232+0000","last_clean":"2024-06-29T14:35:00.242232+0000","last_became_active":"2024-06-29T14:34:58.248695+0000","last_became_peered":"2024-06-29T14:34:58.248695+0000","last_unstale":"2024-06-29T14:35:00.242232+0000","last_undegraded":"2024-06-29T14:35:00.242232+0000","last_fullsized":"2024-06-29T14:35:00.242232+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:09:51.574526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242146+0000","last_change":"2024-06-29T14:34:59.262489+0000","last_active":"2024-06-29T14:35:00.242146+0000","last_peered":"2024-06-29T14:35:00.242146+0000","last_clean":"2024-06-29T14:35:00.242146+0000","last_became_active":"2024-06-29T14:34:59.262074+0000","last_became_peered":"2024-06-29T14:34:59.262074+0000","last_unstale":"2024-06-29T14:35:00.242146+0000","last_undegraded":"2024-06-29T14:35:00.242146+0000","last_fullsized":"2024-06-29T14:35:00.242146+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:25.665738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"60'6","reported_seq":20,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235311+0000","last_change":"2024-06-29T14:34:58.245922+0000","last_active":"2024-06-29T14:35:00.235311+0000","last_peered":"2024-06-29T14:35:00.235311+0000","last_clean":"2024-06-29T14:35:00.235311+0000","last_became_active":"2024-06-29T14:34:58.245623+0000","last_became_peered":"2024-06-29T14:34:58.245623+0000","last_unstale":"2024-06-29T14:35:00.235311+0000","last_undegraded":"2024-06-29T14:35:00.235311+0000","last_fullsized":"2024-06-29T14:35:00.235311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:08.594875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241213+0000","last_change":"2024-06-29T14:34:59.254977+0000","last_active":"2024-06-29T14:35:00.241213+0000","last_peered":"2024-06-29T14:35:00.241213+0000","last_clean":"2024-06-29T14:35:00.241213+0000","last_became_active":"2024-06-29T14:34:59.252975+0000","last_became_peered":"2024-06-29T14:34:59.252975+0000","last_unstale":"2024-06-29T14:35:00.241213+0000","last_undegraded":"2024-06-29T14:35:00.241213+0000","last_fullsized":"2024-06-29T14:35:00.241213+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:09.077088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242233+0000","last_change":"2024-06-29T14:34:58.250625+0000","last_active":"2024-06-29T14:35:00.242233+0000","last_peered":"2024-06-29T14:35:00.242233+0000","last_clean":"2024-06-29T14:35:00.242233+0000","last_became_active":"2024-06-29T14:34:58.250100+0000","last_became_peered":"2024-06-29T14:34:58.250100+0000","last_unstale":"2024-06-29T14:35:00.242233+0000","last_undegraded":"2024-06-29T14:35:00.242233+0000","last_fullsized":"2024-06-29T14:35:00.242233+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:34:34.857992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241031+0000","last_change":"2024-06-29T14:34:59.253640+0000","last_active":"2024-06-29T14:35:00.241031+0000","last_peered":"2024-06-29T14:35:00.241031+0000","last_clean":"2024-06-29T14:35:00.241031+0000","last_became_active":"2024-06-29T14:34:59.253343+0000","last_became_peered":"2024-06-29T14:34:59.253343+0000","last_unstale":"2024-06-29T14:35:00.241031+0000","last_undegraded":"2024-06-29T14:35:00.241031+0000","last_fullsized":"2024-06-29T14:35:00.241031+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:52:52.718656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241350+0000","last_change":"2024-06-29T14:34:58.247199+0000","last_active":"2024-06-29T14:35:00.241350+0000","last_peered":"2024-06-29T14:35:00.241350+0000","last_clean":"2024-06-29T14:35:00.241350+0000","last_became_active":"2024-06-29T14:34:58.246809+0000","last_became_peered":"2024-06-29T14:34:58.246809+0000","last_unstale":"2024-06-29T14:35:00.241350+0000","last_undegraded":"2024-06-29T14:35:00.241350+0000","last_fullsized":"2024-06-29T14:35:00.241350+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:23.103200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241459+0000","last_change":"2024-06-29T14:34:59.250883+0000","last_active":"2024-06-29T14:35:00.241459+0000","last_peered":"2024-06-29T14:35:00.241459+0000","last_clean":"2024-06-29T14:35:00.241459+0000","last_became_active":"2024-06-29T14:34:59.250561+0000","last_became_peered":"2024-06-29T14:34:59.250561+0000","last_unstale":"2024-06-29T14:35:00.241459+0000","last_undegraded":"2024-06-29T14:35:00.241459+0000","last_fullsized":"2024-06-29T14:35:00.241459+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:52.377285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242030+0000","last_change":"2024-06-29T14:34:58.249932+0000","last_active":"2024-06-29T14:35:00.242030+0000","last_peered":"2024-06-29T14:35:00.242030+0000","last_clean":"2024-06-29T14:35:00.242030+0000","last_became_active":"2024-06-29T14:34:58.249209+0000","last_became_peered":"2024-06-29T14:34:58.249209+0000","last_unstale":"2024-06-29T14:35:00.242030+0000","last_undegraded":"2024-06-29T14:35:00.242030+0000","last_fullsized":"2024-06-29T14:35:00.242030+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:30.049435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242040+0000","last_change":"2024-06-29T14:34:59.259241+0000","last_active":"2024-06-29T14:35:00.242040+0000","last_peered":"2024-06-29T14:35:00.242040+0000","last_clean":"2024-06-29T14:35:00.242040+0000","last_became_active":"2024-06-29T14:34:59.258928+0000","last_became_peered":"2024-06-29T14:34:59.258928+0000","last_unstale":"2024-06-29T14:35:00.242040+0000","last_undegraded":"2024-06-29T14:35:00.242040+0000","last_fullsized":"2024-06-29T14:35:00.242040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:32.408127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"60'8","reported_seq":22,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241111+0000","last_change":"2024-06-29T14:34:58.249139+0000","last_active":"2024-06-29T14:35:00.241111+0000","last_peered":"2024-06-29T14:35:00.241111+0000","last_clean":"2024-06-29T14:35:00.241111+0000","last_became_active":"2024-06-29T14:34:58.248847+0000","last_became_peered":"2024-06-29T14:34:58.248847+0000","last_unstale":"2024-06-29T14:35:00.241111+0000","last_undegraded":"2024-06-29T14:35:00.241111+0000","last_fullsized":"2024-06-29T14:35:00.241111+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:36.410979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235189+0000","last_change":"2024-06-29T14:34:59.251266+0000","last_active":"2024-06-29T14:35:00.235189+0000","last_peered":"2024-06-29T14:35:00.235189+0000","last_clean":"2024-06-29T14:35:00.235189+0000","last_became_active":"2024-06-29T14:34:59.250928+0000","last_became_peered":"2024-06-29T14:34:59.250928+0000","last_unstale":"2024-06-29T14:35:00.235189+0000","last_undegraded":"2024-06-29T14:35:00.235189+0000","last_fullsized":"2024-06-29T14:35:00.235189+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:52.124526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"60'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235218+0000","last_change":"2024-06-29T14:34:58.247513+0000","last_active":"2024-06-29T14:35:00.235218+0000","last_peered":"2024-06-29T14:35:00.235218+0000","last_clean":"2024-06-29T14:35:00.235218+0000","last_became_active":"2024-06-29T14:34:58.246500+0000","last_became_peered":"2024-06-29T14:34:58.246500+0000","last_unstale":"2024-06-29T14:35:00.235218+0000","last_undegraded":"2024-06-29T14:35:00.235218+0000","last_fullsized":"2024-06-29T14:35:00.235218+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:14.428931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241407+0000","last_change":"2024-06-29T14:34:59.259355+0000","last_active":"2024-06-29T14:35:00.241407+0000","last_peered":"2024-06-29T14:35:00.241407+0000","last_clean":"2024-06-29T14:35:00.241407+0000","last_became_active":"2024-06-29T14:34:59.259033+0000","last_became_peered":"2024-06-29T14:34:59.259033+0000","last_unstale":"2024-06-29T14:35:00.241407+0000","last_undegraded":"2024-06-29T14:35:00.241407+0000","last_fullsized":"2024-06-29T14:35:00.241407+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:34:57.120364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236293+0000","last_change":"2024-06-29T14:34:58.245944+0000","last_active":"2024-06-29T14:35:00.236293+0000","last_peered":"2024-06-29T14:35:00.236293+0000","last_clean":"2024-06-29T14:35:00.236293+0000","last_became_active":"2024-06-29T14:34:58.245609+0000","last_became_peered":"2024-06-29T14:34:58.245609+0000","last_unstale":"2024-06-29T14:35:00.236293+0000","last_undegraded":"2024-06-29T14:35:00.236293+0000","last_fullsized":"2024-06-29T14:35:00.236293+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:15.366522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241786+0000","last_change":"2024-06-29T14:34:59.257340+0000","last_active":"2024-06-29T14:35:00.241786+0000","last_peered":"2024-06-29T14:35:00.241786+0000","last_clean":"2024-06-29T14:35:00.241786+0000","last_became_active":"2024-06-29T14:34:59.256832+0000","last_became_peered":"2024-06-29T14:34:59.256832+0000","last_unstale":"2024-06-29T14:35:00.241786+0000","last_undegraded":"2024-06-29T14:35:00.241786+0000","last_fullsized":"2024-06-29T14:35:00.241786+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:31:47.110087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"60'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236844+0000","last_change":"2024-06-29T14:34:58.249091+0000","last_active":"2024-06-29T14:35:00.236844+0000","last_peered":"2024-06-29T14:35:00.236844+0000","last_clean":"2024-06-29T14:35:00.236844+0000","last_became_active":"2024-06-29T14:34:58.248782+0000","last_became_peered":"2024-06-29T14:34:58.248782+0000","last_unstale":"2024-06-29T14:35:00.236844+0000","last_undegraded":"2024-06-29T14:35:00.236844+0000","last_fullsized":"2024-06-29T14:35:00.236844+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:33:22.688773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241702+0000","last_change":"2024-06-29T14:34:59.256831+0000","last_active":"2024-06-29T14:35:00.241702+0000","last_peered":"2024-06-29T14:35:00.241702+0000","last_clean":"2024-06-29T14:35:00.241702+0000","last_became_active":"2024-06-29T14:34:59.255345+0000","last_became_peered":"2024-06-29T14:34:59.255345+0000","last_unstale":"2024-06-29T14:35:00.241702+0000","last_undegraded":"2024-06-29T14:35:00.241702+0000","last_fullsized":"2024-06-29T14:35:00.241702+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:15:43.422063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241923+0000","last_change":"2024-06-29T14:34:58.248329+0000","last_active":"2024-06-29T14:35:00.241923+0000","last_peered":"2024-06-29T14:35:00.241923+0000","last_clean":"2024-06-29T14:35:00.241923+0000","last_became_active":"2024-06-29T14:34:58.248027+0000","last_became_peered":"2024-06-29T14:34:58.248027+0000","last_unstale":"2024-06-29T14:35:00.241923+0000","last_undegraded":"2024-06-29T14:35:00.241923+0000","last_fullsized":"2024-06-29T14:35:00.241923+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:43.766922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241728+0000","last_change":"2024-06-29T14:34:59.245926+0000","last_active":"2024-06-29T14:35:00.241728+0000","last_peered":"2024-06-29T14:35:00.241728+0000","last_clean":"2024-06-29T14:35:00.241728+0000","last_became_active":"2024-06-29T14:34:59.245622+0000","last_became_peered":"2024-06-29T14:34:59.245622+0000","last_unstale":"2024-06-29T14:35:00.241728+0000","last_undegraded":"2024-06-29T14:35:00.241728+0000","last_fullsized":"2024-06-29T14:35:00.241728+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:32.817173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242293+0000","last_change":"2024-06-29T14:34:58.245125+0000","last_active":"2024-06-29T14:35:00.242293+0000","last_peered":"2024-06-29T14:35:00.242293+0000","last_clean":"2024-06-29T14:35:00.242293+0000","last_became_active":"2024-06-29T14:34:58.244855+0000","last_became_peered":"2024-06-29T14:34:58.244855+0000","last_unstale":"2024-06-29T14:35:00.242293+0000","last_undegraded":"2024-06-29T14:35:00.242293+0000","last_fullsized":"2024-06-29T14:35:00.242293+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:49:37.819067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240467+0000","last_change":"2024-06-29T14:34:59.260429+0000","last_active":"2024-06-29T14:35:00.240467+0000","last_peered":"2024-06-29T14:35:00.240467+0000","last_clean":"2024-06-29T14:35:00.240467+0000","last_became_active":"2024-06-29T14:34:59.260124+0000","last_became_peered":"2024-06-29T14:34:59.260124+0000","last_unstale":"2024-06-29T14:35:00.240467+0000","last_undegraded":"2024-06-29T14:35:00.240467+0000","last_fullsized":"2024-06-29T14:35:00.240467+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:05:55.273115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235599+0000","last_change":"2024-06-29T14:34:58.248755+0000","last_active":"2024-06-29T14:35:00.235599+0000","last_peered":"2024-06-29T14:35:00.235599+0000","last_clean":"2024-06-29T14:35:00.235599+0000","last_became_active":"2024-06-29T14:34:58.248339+0000","last_became_peered":"2024-06-29T14:34:58.248339+0000","last_unstale":"2024-06-29T14:35:00.235599+0000","last_undegraded":"2024-06-29T14:35:00.235599+0000","last_fullsized":"2024-06-29T14:35:00.235599+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:56.808469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235240+0000","last_change":"2024-06-29T14:34:59.250934+0000","last_active":"2024-06-29T14:35:00.235240+0000","last_peered":"2024-06-29T14:35:00.235240+0000","last_clean":"2024-06-29T14:35:00.235240+0000","last_became_active":"2024-06-29T14:34:59.250245+0000","last_became_peered":"2024-06-29T14:34:59.250245+0000","last_unstale":"2024-06-29T14:35:00.235240+0000","last_undegraded":"2024-06-29T14:35:00.235240+0000","last_fullsized":"2024-06-29T14:35:00.235240+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:09.337433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240707+0000","last_change":"2024-06-29T14:34:58.247657+0000","last_active":"2024-06-29T14:35:00.240707+0000","last_peered":"2024-06-29T14:35:00.240707+0000","last_clean":"2024-06-29T14:35:00.240707+0000","last_became_active":"2024-06-29T14:34:58.247368+0000","last_became_peered":"2024-06-29T14:34:58.247368+0000","last_unstale":"2024-06-29T14:35:00.240707+0000","last_undegraded":"2024-06-29T14:35:00.240707+0000","last_fullsized":"2024-06-29T14:35:00.240707+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:04.574489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240907+0000","last_change":"2024-06-29T14:34:59.261666+0000","last_active":"2024-06-29T14:35:00.240907+0000","last_peered":"2024-06-29T14:35:00.240907+0000","last_clean":"2024-06-29T14:35:00.240907+0000","last_became_active":"2024-06-29T14:34:59.261268+0000","last_became_peered":"2024-06-29T14:34:59.261268+0000","last_unstale":"2024-06-29T14:35:00.240907+0000","last_undegraded":"2024-06-29T14:35:00.240907+0000","last_fullsized":"2024-06-29T14:35:00.240907+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:15:43.422063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241212+0000","last_change":"2024-06-29T14:34:58.250753+0000","last_active":"2024-06-29T14:35:00.241212+0000","last_peered":"2024-06-29T14:35:00.241212+0000","last_clean":"2024-06-29T14:35:00.241212+0000","last_became_active":"2024-06-29T14:34:58.250412+0000","last_became_peered":"2024-06-29T14:34:58.250412+0000","last_unstale":"2024-06-29T14:35:00.241212+0000","last_undegraded":"2024-06-29T14:35:00.241212+0000","last_fullsized":"2024-06-29T14:35:00.241212+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:29:39.596110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242164+0000","last_change":"2024-06-29T14:34:59.253459+0000","last_active":"2024-06-29T14:35:00.242164+0000","last_peered":"2024-06-29T14:35:00.242164+0000","last_clean":"2024-06-29T14:35:00.242164+0000","last_became_active":"2024-06-29T14:34:59.252828+0000","last_became_peered":"2024-06-29T14:34:59.252828+0000","last_unstale":"2024-06-29T14:35:00.242164+0000","last_undegraded":"2024-06-29T14:35:00.242164+0000","last_fullsized":"2024-06-29T14:35:00.242164+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:51:43.053123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"61'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.249894+0000","last_change":"2024-06-29T14:34:58.250190+0000","last_active":"2024-06-29T14:35:00.249894+0000","last_peered":"2024-06-29T14:35:00.249894+0000","last_clean":"2024-06-29T14:35:00.249894+0000","last_became_active":"2024-06-29T14:34:58.249847+0000","last_became_peered":"2024-06-29T14:34:58.249847+0000","last_unstale":"2024-06-29T14:35:00.249894+0000","last_undegraded":"2024-06-29T14:35:00.249894+0000","last_fullsized":"2024-06-29T14:35:00.249894+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:33.533408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242010+0000","last_change":"2024-06-29T14:34:59.254990+0000","last_active":"2024-06-29T14:35:00.242010+0000","last_peered":"2024-06-29T14:35:00.242010+0000","last_clean":"2024-06-29T14:35:00.242010+0000","last_became_active":"2024-06-29T14:34:59.254683+0000","last_became_peered":"2024-06-29T14:34:59.254683+0000","last_unstale":"2024-06-29T14:35:00.242010+0000","last_undegraded":"2024-06-29T14:35:00.242010+0000","last_fullsized":"2024-06-29T14:35:00.242010+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:52:16.345341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242296+0000","last_change":"2024-06-29T14:34:58.249410+0000","last_active":"2024-06-29T14:35:00.242296+0000","last_peered":"2024-06-29T14:35:00.242296+0000","last_clean":"2024-06-29T14:35:00.242296+0000","last_became_active":"2024-06-29T14:34:58.249106+0000","last_became_peered":"2024-06-29T14:34:58.249106+0000","last_unstale":"2024-06-29T14:35:00.242296+0000","last_undegraded":"2024-06-29T14:35:00.242296+0000","last_fullsized":"2024-06-29T14:35:00.242296+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:09:07.727419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242093+0000","last_change":"2024-06-29T14:34:59.250049+0000","last_active":"2024-06-29T14:35:00.242093+0000","last_peered":"2024-06-29T14:35:00.242093+0000","last_clean":"2024-06-29T14:35:00.242093+0000","last_became_active":"2024-06-29T14:34:59.249688+0000","last_became_peered":"2024-06-29T14:34:59.249688+0000","last_unstale":"2024-06-29T14:35:00.242093+0000","last_undegraded":"2024-06-29T14:35:00.242093+0000","last_fullsized":"2024-06-29T14:35:00.242093+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:46.170143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240380+0000","last_change":"2024-06-29T14:34:58.244944+0000","last_active":"2024-06-29T14:35:00.240380+0000","last_peered":"2024-06-29T14:35:00.240380+0000","last_clean":"2024-06-29T14:35:00.240380+0000","last_became_active":"2024-06-29T14:34:58.244695+0000","last_became_peered":"2024-06-29T14:34:58.244695+0000","last_unstale":"2024-06-29T14:35:00.240380+0000","last_undegraded":"2024-06-29T14:35:00.240380+0000","last_fullsized":"2024-06-29T14:35:00.240380+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:49.115624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241766+0000","last_change":"2024-06-29T14:34:59.251433+0000","last_active":"2024-06-29T14:35:00.241766+0000","last_peered":"2024-06-29T14:35:00.241766+0000","last_clean":"2024-06-29T14:35:00.241766+0000","last_became_active":"2024-06-29T14:34:59.251117+0000","last_became_peered":"2024-06-29T14:34:59.251117+0000","last_unstale":"2024-06-29T14:35:00.241766+0000","last_undegraded":"2024-06-29T14:35:00.241766+0000","last_fullsized":"2024-06-29T14:35:00.241766+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:20.698906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241565+0000","last_change":"2024-06-29T14:34:58.248932+0000","last_active":"2024-06-29T14:35:00.241565+0000","last_peered":"2024-06-29T14:35:00.241565+0000","last_clean":"2024-06-29T14:35:00.241565+0000","last_became_active":"2024-06-29T14:34:58.248576+0000","last_became_peered":"2024-06-29T14:34:58.248576+0000","last_unstale":"2024-06-29T14:35:00.241565+0000","last_undegraded":"2024-06-29T14:35:00.241565+0000","last_fullsized":"2024-06-29T14:35:00.241565+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:27.920891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242155+0000","last_change":"2024-06-29T14:34:59.256887+0000","last_active":"2024-06-29T14:35:00.242155+0000","last_peered":"2024-06-29T14:35:00.242155+0000","last_clean":"2024-06-29T14:35:00.242155+0000","last_became_active":"2024-06-29T14:34:59.256525+0000","last_became_peered":"2024-06-29T14:34:59.256525+0000","last_unstale":"2024-06-29T14:35:00.242155+0000","last_undegraded":"2024-06-29T14:35:00.242155+0000","last_fullsized":"2024-06-29T14:35:00.242155+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:46.170143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235977+0000","last_change":"2024-06-29T14:34:58.245383+0000","last_active":"2024-06-29T14:35:00.235977+0000","last_peered":"2024-06-29T14:35:00.235977+0000","last_clean":"2024-06-29T14:35:00.235977+0000","last_became_active":"2024-06-29T14:34:58.245057+0000","last_became_peered":"2024-06-29T14:34:58.245057+0000","last_unstale":"2024-06-29T14:35:00.235977+0000","last_undegraded":"2024-06-29T14:35:00.235977+0000","last_fullsized":"2024-06-29T14:35:00.235977+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:49:48.013292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235477+0000","last_change":"2024-06-29T14:34:59.251278+0000","last_active":"2024-06-29T14:35:00.235477+0000","last_peered":"2024-06-29T14:35:00.235477+0000","last_clean":"2024-06-29T14:35:00.235477+0000","last_became_active":"2024-06-29T14:34:59.250390+0000","last_became_peered":"2024-06-29T14:34:59.250390+0000","last_unstale":"2024-06-29T14:35:00.235477+0000","last_undegraded":"2024-06-29T14:35:00.235477+0000","last_fullsized":"2024-06-29T14:35:00.235477+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:29.888030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240315+0000","last_change":"2024-06-29T14:34:58.250852+0000","last_active":"2024-06-29T14:35:00.240315+0000","last_peered":"2024-06-29T14:35:00.240315+0000","last_clean":"2024-06-29T14:35:00.240315+0000","last_became_active":"2024-06-29T14:34:58.250341+0000","last_became_peered":"2024-06-29T14:34:58.250341+0000","last_unstale":"2024-06-29T14:35:00.240315+0000","last_undegraded":"2024-06-29T14:35:00.240315+0000","last_fullsized":"2024-06-29T14:35:00.240315+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:17.431157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235352+0000","last_change":"2024-06-29T14:34:59.253737+0000","last_active":"2024-06-29T14:35:00.235352+0000","last_peered":"2024-06-29T14:35:00.235352+0000","last_clean":"2024-06-29T14:35:00.235352+0000","last_became_active":"2024-06-29T14:34:59.253361+0000","last_became_peered":"2024-06-29T14:34:59.253361+0000","last_unstale":"2024-06-29T14:35:00.235352+0000","last_undegraded":"2024-06-29T14:35:00.235352+0000","last_fullsized":"2024-06-29T14:35:00.235352+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:08.498050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241216+0000","last_change":"2024-06-29T14:34:58.247157+0000","last_active":"2024-06-29T14:35:00.241216+0000","last_peered":"2024-06-29T14:35:00.241216+0000","last_clean":"2024-06-29T14:35:00.241216+0000","last_became_active":"2024-06-29T14:34:58.246836+0000","last_became_peered":"2024-06-29T14:34:58.246836+0000","last_unstale":"2024-06-29T14:35:00.241216+0000","last_undegraded":"2024-06-29T14:35:00.241216+0000","last_fullsized":"2024-06-29T14:35:00.241216+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:26.334699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241870+0000","last_change":"2024-06-29T14:34:59.256741+0000","last_active":"2024-06-29T14:35:00.241870+0000","last_peered":"2024-06-29T14:35:00.241870+0000","last_clean":"2024-06-29T14:35:00.241870+0000","last_became_active":"2024-06-29T14:34:59.256439+0000","last_became_peered":"2024-06-29T14:34:59.256439+0000","last_unstale":"2024-06-29T14:35:00.241870+0000","last_undegraded":"2024-06-29T14:35:00.241870+0000","last_fullsized":"2024-06-29T14:35:00.241870+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:42:18.547863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236885+0000","last_change":"2024-06-29T14:34:58.241817+0000","last_active":"2024-06-29T14:35:00.236885+0000","last_peered":"2024-06-29T14:35:00.236885+0000","last_clean":"2024-06-29T14:35:00.236885+0000","last_became_active":"2024-06-29T14:34:58.241457+0000","last_became_peered":"2024-06-29T14:34:58.241457+0000","last_unstale":"2024-06-29T14:35:00.236885+0000","last_undegraded":"2024-06-29T14:35:00.236885+0000","last_fullsized":"2024-06-29T14:35:00.236885+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:54:34.801214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"49'86","reported_seq":244,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241314+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:35:00.241314+0000","last_peered":"2024-06-29T14:35:00.241314+0000","last_clean":"2024-06-29T14:35:00.241314+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:35:00.241314+0000","last_undegraded":"2024-06-29T14:35:00.241314+0000","last_fullsized":"2024-06-29T14:35:00.241314+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575002,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1310720,"data_stored":925058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":720896,"data_stored":335272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575001,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":334670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575002,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54100,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317740,"statfs":{"total":96636764160,"available":96581365760,"internally_reserved":0,"allocated":724992,"data_stored":336212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575002,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":737280,"data_stored":341582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36056,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335784,"statfs":{"total":96636764160,"available":96599842816,"internally_reserved":0,"allocated":729088,"data_stored":336234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49944,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321896,"statfs":{"total":96636764160,"available":96585621504,"internally_reserved":0,"allocated":729088,"data_stored":341014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575002,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1323008,"data_stored":931346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:35:03.274 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:35:03.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.424+0000 7fcb95881640 1 -- 172.21.15.81:0/4118832267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb900d2a90 msgr2=0x7fcb900d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.424+0000 7fcb95881640 1 --2- 172.21.15.81:0/4118832267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb900d2a90 0x7fcb900d09c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fcb9008e230 tx=0x7fcb8802aac0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.424+0000 7fcb95881640 1 -- 172.21.15.81:0/4118832267 shutdown_connections 2024-06-29T14:35:03.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.424+0000 7fcb95881640 1 --2- 172.21.15.81:0/4118832267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb900d2a90 0x7fcb900d09c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.424+0000 7fcb95881640 1 --2- 172.21.15.81:0/4118832267 >> v2:172.21.15.167:3300/0 conn(0x7fcb900b9040 0x7fcb900d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.424+0000 7fcb95881640 1 -- 172.21.15.81:0/4118832267 >> 172.21.15.81:0/4118832267 conn(0x7fcb9009cf30 msgr2=0x7fcb900d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.424+0000 7fcb95881640 1 -- 172.21.15.81:0/4118832267 shutdown_connections 2024-06-29T14:35:03.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.424+0000 7fcb95881640 1 -- 172.21.15.81:0/4118832267 wait complete. 2024-06-29T14:35:03.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.425+0000 7fcb95881640 1 Processor -- start 2024-06-29T14:35:03.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.426+0000 7fcb95881640 1 -- start start 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb95881640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb900b5250 0x7fcb900cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb95881640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb900b9040 0x7fcb901f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb95881640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb900d2a90 0x7fcb901584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb95881640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcb900b7d30 con 0x7fcb900d2a90 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb95881640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcb900b7bb0 con 0x7fcb900b9040 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb95881640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcb900b7eb0 con 0x7fcb900b5250 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb900d2a90 0x7fcb901584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb900d2a90 0x7fcb901584e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50396/0 (socket says 172.21.15.81:50396) 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8f7fe640 1 -- 172.21.15.81:0/325156588 learned_addr learned my addr 172.21.15.81:0/325156588 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8effd640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb900b5250 0x7fcb900cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8e7fc640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb900b9040 0x7fcb901f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8f7fe640 1 -- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb900b5250 msgr2=0x7fcb900cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8f7fe640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb900b5250 0x7fcb900cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8f7fe640 1 -- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb900b9040 msgr2=0x7fcb901f5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8f7fe640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb900b9040 0x7fcb901f5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8f7fe640 1 -- 172.21.15.81:0/325156588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb901f55d0 con 0x7fcb900d2a90 2024-06-29T14:35:03.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8e7fc640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb900b9040 0x7fcb901f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:03.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.427+0000 7fcb8f7fe640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb900d2a90 0x7fcb901584e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009d60 tx=0x7fcb88003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.428+0000 7fcb77fff640 1 -- 172.21.15.81:0/325156588 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88044070 con 0x7fcb900d2a90 2024-06-29T14:35:03.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.428+0000 7fcb77fff640 1 -- 172.21.15.81:0/325156588 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb88004250 con 0x7fcb900d2a90 2024-06-29T14:35:03.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.428+0000 7fcb95881640 1 -- 172.21.15.81:0/325156588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb901f5ad0 con 0x7fcb900d2a90 2024-06-29T14:35:03.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.428+0000 7fcb77fff640 1 -- 172.21.15.81:0/325156588 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88004680 con 0x7fcb900d2a90 2024-06-29T14:35:03.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.428+0000 7fcb95881640 1 -- 172.21.15.81:0/325156588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb901f6090 con 0x7fcb900d2a90 2024-06-29T14:35:03.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.429+0000 7fcb95881640 1 -- 172.21.15.81:0/325156588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb900b8440 con 0x7fcb900d2a90 2024-06-29T14:35:03.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.429+0000 7fcb77fff640 1 -- 172.21.15.81:0/325156588 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7fcb88036400 con 0x7fcb900d2a90 2024-06-29T14:35:03.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.430+0000 7fcb77fff640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fcb7c0763f0 0x7fcb7c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.430+0000 7fcb77fff640 1 -- 172.21.15.81:0/325156588 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7fcb880b7f40 con 0x7fcb900d2a90 2024-06-29T14:35:03.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.430+0000 7fcb8effd640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fcb7c0763f0 0x7fcb7c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.431+0000 7fcb8effd640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fcb7c0763f0 0x7fcb7c078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcb80004630 tx=0x7fcb8000b2e0 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.435+0000 7fcb77fff640 1 -- 172.21.15.81:0/325156588 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcb88032030 con 0x7fcb900d2a90 2024-06-29T14:35:03.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.613+0000 7fcb95881640 1 -- 172.21.15.81:0/325156588 --> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcb9006d840 con 0x7fcb7c0763f0 2024-06-29T14:35:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.618+0000 7fcb77fff640 1 -- 172.21.15.81:0/325156588 <== mgr.5425 v2:172.21.15.167:6832/1090782941 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251952 (secure 0 0 0) 0x7fcb9006d840 con 0x7fcb7c0763f0 2024-06-29T14:35:03.618 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:35:03.620 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:35:03.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.621+0000 7fcb95881640 1 -- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fcb7c0763f0 msgr2=0x7fcb7c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.621+0000 7fcb95881640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fcb7c0763f0 0x7fcb7c078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcb80004630 tx=0x7fcb8000b2e0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.622+0000 7fcb95881640 1 -- 172.21.15.81:0/325156588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb900d2a90 msgr2=0x7fcb901584e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.622+0000 7fcb95881640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb900d2a90 0x7fcb901584e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009d60 tx=0x7fcb88003f10 comp rx=0 tx=0).stop 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.622+0000 7fcb95881640 1 -- 172.21.15.81:0/325156588 shutdown_connections 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.622+0000 7fcb95881640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fcb7c0763f0 0x7fcb7c078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.622+0000 7fcb95881640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb900d2a90 0x7fcb901584e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.622+0000 7fcb95881640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb900b9040 0x7fcb901f5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.622+0000 7fcb95881640 1 --2- 172.21.15.81:0/325156588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb900b5250 0x7fcb900cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.622+0000 7fcb95881640 1 -- 172.21.15.81:0/325156588 >> 172.21.15.81:0/325156588 conn(0x7fcb9009cf30 msgr2=0x7fcb90161400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.622+0000 7fcb95881640 1 -- 172.21.15.81:0/325156588 shutdown_connections 2024-06-29T14:35:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.622+0000 7fcb95881640 1 -- 172.21.15.81:0/325156588 wait complete. 2024-06-29T14:35:03.631 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-06-29T14:35:02.850922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":170,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":300720,"kb_used_data":6828,"kb_used_omap":12,"kb_used_meta":293747,"kb_avail":754674000,"statfs":{"total":773094113280,"available":772786176000,"internally_reserved":0,"allocated":6991872,"data_stored":3881388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":300797490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.627567"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242215+0000","last_change":"2024-06-29T14:34:59.254713+0000","last_active":"2024-06-29T14:35:00.242215+0000","last_peered":"2024-06-29T14:35:00.242215+0000","last_clean":"2024-06-29T14:35:00.242215+0000","last_became_active":"2024-06-29T14:34:59.254343+0000","last_became_peered":"2024-06-29T14:34:59.254343+0000","last_unstale":"2024-06-29T14:35:00.242215+0000","last_undegraded":"2024-06-29T14:35:00.242215+0000","last_fullsized":"2024-06-29T14:35:00.242215+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:43:44.063626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236545+0000","last_change":"2024-06-29T14:34:59.252912+0000","last_active":"2024-06-29T14:35:00.236545+0000","last_peered":"2024-06-29T14:35:00.236545+0000","last_clean":"2024-06-29T14:35:00.236545+0000","last_became_active":"2024-06-29T14:34:59.252563+0000","last_became_peered":"2024-06-29T14:34:59.252563+0000","last_unstale":"2024-06-29T14:35:00.236545+0000","last_undegraded":"2024-06-29T14:35:00.236545+0000","last_fullsized":"2024-06-29T14:35:00.236545+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:53.321271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235425+0000","last_change":"2024-06-29T14:34:59.244690+0000","last_active":"2024-06-29T14:35:00.235425+0000","last_peered":"2024-06-29T14:35:00.235425+0000","last_clean":"2024-06-29T14:35:00.235425+0000","last_became_active":"2024-06-29T14:34:59.244314+0000","last_became_peered":"2024-06-29T14:34:59.244314+0000","last_unstale":"2024-06-29T14:35:00.235425+0000","last_undegraded":"2024-06-29T14:35:00.235425+0000","last_fullsized":"2024-06-29T14:35:00.235425+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:00.875369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241978+0000","last_change":"2024-06-29T14:34:59.255746+0000","last_active":"2024-06-29T14:35:00.241978+0000","last_peered":"2024-06-29T14:35:00.241978+0000","last_clean":"2024-06-29T14:35:00.241978+0000","last_became_active":"2024-06-29T14:34:59.255366+0000","last_became_peered":"2024-06-29T14:34:59.255366+0000","last_unstale":"2024-06-29T14:35:00.241978+0000","last_undegraded":"2024-06-29T14:35:00.241978+0000","last_fullsized":"2024-06-29T14:35:00.241978+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:04.477358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236207+0000","last_change":"2024-06-29T14:34:59.249674+0000","last_active":"2024-06-29T14:35:00.236207+0000","last_peered":"2024-06-29T14:35:00.236207+0000","last_clean":"2024-06-29T14:35:00.236207+0000","last_became_active":"2024-06-29T14:34:59.249376+0000","last_became_peered":"2024-06-29T14:34:59.249376+0000","last_unstale":"2024-06-29T14:35:00.236207+0000","last_undegraded":"2024-06-29T14:35:00.236207+0000","last_fullsized":"2024-06-29T14:35:00.236207+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:44.772104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241544+0000","last_change":"2024-06-29T14:34:59.253473+0000","last_active":"2024-06-29T14:35:00.241544+0000","last_peered":"2024-06-29T14:35:00.241544+0000","last_clean":"2024-06-29T14:35:00.241544+0000","last_became_active":"2024-06-29T14:34:59.253152+0000","last_became_peered":"2024-06-29T14:34:59.253152+0000","last_unstale":"2024-06-29T14:35:00.241544+0000","last_undegraded":"2024-06-29T14:35:00.241544+0000","last_fullsized":"2024-06-29T14:35:00.241544+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:52.278087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241501+0000","last_change":"2024-06-29T14:34:59.262000+0000","last_active":"2024-06-29T14:35:00.241501+0000","last_peered":"2024-06-29T14:35:00.241501+0000","last_clean":"2024-06-29T14:35:00.241501+0000","last_became_active":"2024-06-29T14:34:59.261666+0000","last_became_peered":"2024-06-29T14:34:59.261666+0000","last_unstale":"2024-06-29T14:35:00.241501+0000","last_undegraded":"2024-06-29T14:35:00.241501+0000","last_fullsized":"2024-06-29T14:35:00.241501+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:37.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240761+0000","last_change":"2024-06-29T14:34:59.254813+0000","last_active":"2024-06-29T14:35:00.240761+0000","last_peered":"2024-06-29T14:35:00.240761+0000","last_clean":"2024-06-29T14:35:00.240761+0000","last_became_active":"2024-06-29T14:34:59.254484+0000","last_became_peered":"2024-06-29T14:34:59.254484+0000","last_unstale":"2024-06-29T14:35:00.240761+0000","last_undegraded":"2024-06-29T14:35:00.240761+0000","last_fullsized":"2024-06-29T14:35:00.240761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:40.033180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241285+0000","last_change":"2024-06-29T14:34:59.262064+0000","last_active":"2024-06-29T14:35:00.241285+0000","last_peered":"2024-06-29T14:35:00.241285+0000","last_clean":"2024-06-29T14:35:00.241285+0000","last_became_active":"2024-06-29T14:34:59.261648+0000","last_became_peered":"2024-06-29T14:34:59.261648+0000","last_unstale":"2024-06-29T14:35:00.241285+0000","last_undegraded":"2024-06-29T14:35:00.241285+0000","last_fullsized":"2024-06-29T14:35:00.241285+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:03.348772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241708+0000","last_change":"2024-06-29T14:34:59.255791+0000","last_active":"2024-06-29T14:35:00.241708+0000","last_peered":"2024-06-29T14:35:00.241708+0000","last_clean":"2024-06-29T14:35:00.241708+0000","last_became_active":"2024-06-29T14:34:59.255326+0000","last_became_peered":"2024-06-29T14:34:59.255326+0000","last_unstale":"2024-06-29T14:35:00.241708+0000","last_undegraded":"2024-06-29T14:35:00.241708+0000","last_fullsized":"2024-06-29T14:35:00.241708+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:51:43.053123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241167+0000","last_change":"2024-06-29T14:34:59.261510+0000","last_active":"2024-06-29T14:35:00.241167+0000","last_peered":"2024-06-29T14:35:00.241167+0000","last_clean":"2024-06-29T14:35:00.241167+0000","last_became_active":"2024-06-29T14:34:59.261182+0000","last_became_peered":"2024-06-29T14:34:59.261182+0000","last_unstale":"2024-06-29T14:35:00.241167+0000","last_undegraded":"2024-06-29T14:35:00.241167+0000","last_fullsized":"2024-06-29T14:35:00.241167+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:31:47.110087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236894+0000","last_change":"2024-06-29T14:34:59.254087+0000","last_active":"2024-06-29T14:35:00.236894+0000","last_peered":"2024-06-29T14:35:00.236894+0000","last_clean":"2024-06-29T14:35:00.236894+0000","last_became_active":"2024-06-29T14:34:59.253563+0000","last_became_peered":"2024-06-29T14:34:59.253563+0000","last_unstale":"2024-06-29T14:35:00.236894+0000","last_undegraded":"2024-06-29T14:35:00.236894+0000","last_fullsized":"2024-06-29T14:35:00.236894+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:34:35.863173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235307+0000","last_change":"2024-06-29T14:34:59.259565+0000","last_active":"2024-06-29T14:35:00.235307+0000","last_peered":"2024-06-29T14:35:00.235307+0000","last_clean":"2024-06-29T14:35:00.235307+0000","last_became_active":"2024-06-29T14:34:59.259231+0000","last_became_peered":"2024-06-29T14:34:59.259231+0000","last_unstale":"2024-06-29T14:35:00.235307+0000","last_undegraded":"2024-06-29T14:35:00.235307+0000","last_fullsized":"2024-06-29T14:35:00.235307+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:44.349582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240591+0000","last_change":"2024-06-29T14:34:59.261496+0000","last_active":"2024-06-29T14:35:00.240591+0000","last_peered":"2024-06-29T14:35:00.240591+0000","last_clean":"2024-06-29T14:35:00.240591+0000","last_became_active":"2024-06-29T14:34:59.261153+0000","last_became_peered":"2024-06-29T14:34:59.261153+0000","last_unstale":"2024-06-29T14:35:00.240591+0000","last_undegraded":"2024-06-29T14:35:00.240591+0000","last_fullsized":"2024-06-29T14:35:00.240591+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:37.045557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241520+0000","last_change":"2024-06-29T14:34:59.239398+0000","last_active":"2024-06-29T14:35:00.241520+0000","last_peered":"2024-06-29T14:35:00.241520+0000","last_clean":"2024-06-29T14:35:00.241520+0000","last_became_active":"2024-06-29T14:34:59.239175+0000","last_became_peered":"2024-06-29T14:34:59.239175+0000","last_unstale":"2024-06-29T14:35:00.241520+0000","last_undegraded":"2024-06-29T14:35:00.241520+0000","last_fullsized":"2024-06-29T14:35:00.241520+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:41:11.700373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241083+0000","last_change":"2024-06-29T14:34:59.250840+0000","last_active":"2024-06-29T14:35:00.241083+0000","last_peered":"2024-06-29T14:35:00.241083+0000","last_clean":"2024-06-29T14:35:00.241083+0000","last_became_active":"2024-06-29T14:34:59.250518+0000","last_became_peered":"2024-06-29T14:34:59.250518+0000","last_unstale":"2024-06-29T14:35:00.241083+0000","last_undegraded":"2024-06-29T14:35:00.241083+0000","last_fullsized":"2024-06-29T14:35:00.241083+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:41:50.447752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235940+0000","last_change":"2024-06-29T14:34:59.245145+0000","last_active":"2024-06-29T14:35:00.235940+0000","last_peered":"2024-06-29T14:35:00.235940+0000","last_clean":"2024-06-29T14:35:00.235940+0000","last_became_active":"2024-06-29T14:34:59.244820+0000","last_became_peered":"2024-06-29T14:34:59.244820+0000","last_unstale":"2024-06-29T14:35:00.235940+0000","last_undegraded":"2024-06-29T14:35:00.235940+0000","last_fullsized":"2024-06-29T14:35:00.235940+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:19:20.489371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236167+0000","last_change":"2024-06-29T14:34:59.261158+0000","last_active":"2024-06-29T14:35:00.236167+0000","last_peered":"2024-06-29T14:35:00.236167+0000","last_clean":"2024-06-29T14:35:00.236167+0000","last_became_active":"2024-06-29T14:34:59.260819+0000","last_became_peered":"2024-06-29T14:34:59.260819+0000","last_unstale":"2024-06-29T14:35:00.236167+0000","last_undegraded":"2024-06-29T14:35:00.236167+0000","last_fullsized":"2024-06-29T14:35:00.236167+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:07:52.274841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241998+0000","last_change":"2024-06-29T14:34:59.252099+0000","last_active":"2024-06-29T14:35:00.241998+0000","last_peered":"2024-06-29T14:35:00.241998+0000","last_clean":"2024-06-29T14:35:00.241998+0000","last_became_active":"2024-06-29T14:34:59.251588+0000","last_became_peered":"2024-06-29T14:34:59.251588+0000","last_unstale":"2024-06-29T14:35:00.241998+0000","last_undegraded":"2024-06-29T14:35:00.241998+0000","last_fullsized":"2024-06-29T14:35:00.241998+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:03.348772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235289+0000","last_change":"2024-06-29T14:34:59.250569+0000","last_active":"2024-06-29T14:35:00.235289+0000","last_peered":"2024-06-29T14:35:00.235289+0000","last_clean":"2024-06-29T14:35:00.235289+0000","last_became_active":"2024-06-29T14:34:59.250252+0000","last_became_peered":"2024-06-29T14:34:59.250252+0000","last_unstale":"2024-06-29T14:35:00.235289+0000","last_undegraded":"2024-06-29T14:35:00.235289+0000","last_fullsized":"2024-06-29T14:35:00.235289+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:55:44.770475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241982+0000","last_change":"2024-06-29T14:34:59.254219+0000","last_active":"2024-06-29T14:35:00.241982+0000","last_peered":"2024-06-29T14:35:00.241982+0000","last_clean":"2024-06-29T14:35:00.241982+0000","last_became_active":"2024-06-29T14:34:59.253940+0000","last_became_peered":"2024-06-29T14:34:59.253940+0000","last_unstale":"2024-06-29T14:35:00.241982+0000","last_undegraded":"2024-06-29T14:35:00.241982+0000","last_fullsized":"2024-06-29T14:35:00.241982+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:32.408127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240755+0000","last_change":"2024-06-29T14:34:59.255662+0000","last_active":"2024-06-29T14:35:00.240755+0000","last_peered":"2024-06-29T14:35:00.240755+0000","last_clean":"2024-06-29T14:35:00.240755+0000","last_became_active":"2024-06-29T14:34:59.255214+0000","last_became_peered":"2024-06-29T14:34:59.255214+0000","last_unstale":"2024-06-29T14:35:00.240755+0000","last_undegraded":"2024-06-29T14:35:00.240755+0000","last_fullsized":"2024-06-29T14:35:00.240755+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:41:50.447752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241030+0000","last_change":"2024-06-29T14:34:59.254888+0000","last_active":"2024-06-29T14:35:00.241030+0000","last_peered":"2024-06-29T14:35:00.241030+0000","last_clean":"2024-06-29T14:35:00.241030+0000","last_became_active":"2024-06-29T14:34:59.254162+0000","last_became_peered":"2024-06-29T14:34:59.254162+0000","last_unstale":"2024-06-29T14:35:00.241030+0000","last_undegraded":"2024-06-29T14:35:00.241030+0000","last_fullsized":"2024-06-29T14:35:00.241030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:09:40.033180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240914+0000","last_change":"2024-06-29T14:34:59.249930+0000","last_active":"2024-06-29T14:35:00.240914+0000","last_peered":"2024-06-29T14:35:00.240914+0000","last_clean":"2024-06-29T14:35:00.240914+0000","last_became_active":"2024-06-29T14:34:59.249613+0000","last_became_peered":"2024-06-29T14:34:59.249613+0000","last_unstale":"2024-06-29T14:35:00.240914+0000","last_undegraded":"2024-06-29T14:35:00.240914+0000","last_fullsized":"2024-06-29T14:35:00.240914+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:29.407691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242018+0000","last_change":"2024-06-29T14:34:59.255472+0000","last_active":"2024-06-29T14:35:00.242018+0000","last_peered":"2024-06-29T14:35:00.242018+0000","last_clean":"2024-06-29T14:35:00.242018+0000","last_became_active":"2024-06-29T14:34:59.252469+0000","last_became_peered":"2024-06-29T14:34:59.252469+0000","last_unstale":"2024-06-29T14:35:00.242018+0000","last_undegraded":"2024-06-29T14:35:00.242018+0000","last_fullsized":"2024-06-29T14:35:00.242018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:34:57.120364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235100+0000","last_change":"2024-06-29T14:34:59.258912+0000","last_active":"2024-06-29T14:35:00.235100+0000","last_peered":"2024-06-29T14:35:00.235100+0000","last_clean":"2024-06-29T14:35:00.235100+0000","last_became_active":"2024-06-29T14:34:59.258575+0000","last_became_peered":"2024-06-29T14:34:59.258575+0000","last_unstale":"2024-06-29T14:35:00.235100+0000","last_undegraded":"2024-06-29T14:35:00.235100+0000","last_fullsized":"2024-06-29T14:35:00.235100+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:05.947594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235555+0000","last_change":"2024-06-29T14:34:59.245507+0000","last_active":"2024-06-29T14:35:00.235555+0000","last_peered":"2024-06-29T14:35:00.235555+0000","last_clean":"2024-06-29T14:35:00.235555+0000","last_became_active":"2024-06-29T14:34:59.245231+0000","last_became_peered":"2024-06-29T14:34:59.245231+0000","last_unstale":"2024-06-29T14:35:00.235555+0000","last_undegraded":"2024-06-29T14:35:00.235555+0000","last_fullsized":"2024-06-29T14:35:00.235555+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:20:01.535487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.237028+0000","last_change":"2024-06-29T14:34:59.254379+0000","last_active":"2024-06-29T14:35:00.237028+0000","last_peered":"2024-06-29T14:35:00.237028+0000","last_clean":"2024-06-29T14:35:00.237028+0000","last_became_active":"2024-06-29T14:34:59.253497+0000","last_became_peered":"2024-06-29T14:34:59.253497+0000","last_unstale":"2024-06-29T14:35:00.237028+0000","last_undegraded":"2024-06-29T14:35:00.237028+0000","last_fullsized":"2024-06-29T14:35:00.237028+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:49:38.824249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241755+0000","last_change":"2024-06-29T14:34:59.250999+0000","last_active":"2024-06-29T14:35:00.241755+0000","last_peered":"2024-06-29T14:35:00.241755+0000","last_clean":"2024-06-29T14:35:00.241755+0000","last_became_active":"2024-06-29T14:34:59.250633+0000","last_became_peered":"2024-06-29T14:34:59.250633+0000","last_unstale":"2024-06-29T14:35:00.241755+0000","last_undegraded":"2024-06-29T14:35:00.241755+0000","last_fullsized":"2024-06-29T14:35:00.241755+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:52:52.718656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235709+0000","last_change":"2024-06-29T14:34:59.247093+0000","last_active":"2024-06-29T14:35:00.235709+0000","last_peered":"2024-06-29T14:35:00.235709+0000","last_clean":"2024-06-29T14:35:00.235709+0000","last_became_active":"2024-06-29T14:34:59.246722+0000","last_became_peered":"2024-06-29T14:34:59.246722+0000","last_unstale":"2024-06-29T14:35:00.235709+0000","last_undegraded":"2024-06-29T14:35:00.235709+0000","last_fullsized":"2024-06-29T14:35:00.235709+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:21.719173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241654+0000","last_change":"2024-06-29T14:34:59.245003+0000","last_active":"2024-06-29T14:35:00.241654+0000","last_peered":"2024-06-29T14:35:00.241654+0000","last_clean":"2024-06-29T14:35:00.241654+0000","last_became_active":"2024-06-29T14:34:59.244673+0000","last_became_peered":"2024-06-29T14:34:59.244673+0000","last_unstale":"2024-06-29T14:35:00.241654+0000","last_undegraded":"2024-06-29T14:35:00.241654+0000","last_fullsized":"2024-06-29T14:35:00.241654+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:12.388505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235084+0000","last_change":"2024-06-29T14:34:59.257956+0000","last_active":"2024-06-29T14:35:00.235084+0000","last_peered":"2024-06-29T14:35:00.235084+0000","last_clean":"2024-06-29T14:35:00.235084+0000","last_became_active":"2024-06-29T14:34:59.257666+0000","last_became_peered":"2024-06-29T14:34:59.257666+0000","last_unstale":"2024-06-29T14:35:00.235084+0000","last_undegraded":"2024-06-29T14:35:00.235084+0000","last_fullsized":"2024-06-29T14:35:00.235084+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:38.152457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240506+0000","last_change":"2024-06-29T14:34:59.258398+0000","last_active":"2024-06-29T14:35:00.240506+0000","last_peered":"2024-06-29T14:35:00.240506+0000","last_clean":"2024-06-29T14:35:00.240506+0000","last_became_active":"2024-06-29T14:34:59.258078+0000","last_became_peered":"2024-06-29T14:34:59.258078+0000","last_unstale":"2024-06-29T14:35:00.240506+0000","last_undegraded":"2024-06-29T14:35:00.240506+0000","last_fullsized":"2024-06-29T14:35:00.240506+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:47:21.389411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"60'4","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242094+0000","last_change":"2024-06-29T14:34:58.248413+0000","last_active":"2024-06-29T14:35:00.242094+0000","last_peered":"2024-06-29T14:35:00.242094+0000","last_clean":"2024-06-29T14:35:00.242094+0000","last_became_active":"2024-06-29T14:34:58.248108+0000","last_became_peered":"2024-06-29T14:34:58.248108+0000","last_unstale":"2024-06-29T14:35:00.242094+0000","last_undegraded":"2024-06-29T14:35:00.242094+0000","last_fullsized":"2024-06-29T14:35:00.242094+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:28.882848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241702+0000","last_change":"2024-06-29T14:34:59.257583+0000","last_active":"2024-06-29T14:35:00.241702+0000","last_peered":"2024-06-29T14:35:00.241702+0000","last_clean":"2024-06-29T14:35:00.241702+0000","last_became_active":"2024-06-29T14:34:59.257226+0000","last_became_peered":"2024-06-29T14:34:59.257226+0000","last_unstale":"2024-06-29T14:35:00.241702+0000","last_undegraded":"2024-06-29T14:35:00.241702+0000","last_fullsized":"2024-06-29T14:35:00.241702+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:18.320387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"60'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236172+0000","last_change":"2024-06-29T14:34:58.246164+0000","last_active":"2024-06-29T14:35:00.236172+0000","last_peered":"2024-06-29T14:35:00.236172+0000","last_clean":"2024-06-29T14:35:00.236172+0000","last_became_active":"2024-06-29T14:34:58.245857+0000","last_became_peered":"2024-06-29T14:34:58.245857+0000","last_unstale":"2024-06-29T14:35:00.236172+0000","last_undegraded":"2024-06-29T14:35:00.236172+0000","last_fullsized":"2024-06-29T14:35:00.236172+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:49.115624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.237055+0000","last_change":"2024-06-29T14:34:59.257401+0000","last_active":"2024-06-29T14:35:00.237055+0000","last_peered":"2024-06-29T14:35:00.237055+0000","last_clean":"2024-06-29T14:35:00.237055+0000","last_became_active":"2024-06-29T14:34:59.257049+0000","last_became_peered":"2024-06-29T14:34:59.257049+0000","last_unstale":"2024-06-29T14:35:00.237055+0000","last_undegraded":"2024-06-29T14:35:00.237055+0000","last_fullsized":"2024-06-29T14:35:00.237055+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:41.763922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241031+0000","last_change":"2024-06-29T14:34:58.245297+0000","last_active":"2024-06-29T14:35:00.241031+0000","last_peered":"2024-06-29T14:35:00.241031+0000","last_clean":"2024-06-29T14:35:00.241031+0000","last_became_active":"2024-06-29T14:34:58.244945+0000","last_became_peered":"2024-06-29T14:34:58.244945+0000","last_unstale":"2024-06-29T14:35:00.241031+0000","last_undegraded":"2024-06-29T14:35:00.241031+0000","last_fullsized":"2024-06-29T14:35:00.241031+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:15.366522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240902+0000","last_change":"2024-06-29T14:34:59.255529+0000","last_active":"2024-06-29T14:35:00.240902+0000","last_peered":"2024-06-29T14:35:00.240902+0000","last_clean":"2024-06-29T14:35:00.240902+0000","last_became_active":"2024-06-29T14:34:59.255224+0000","last_became_peered":"2024-06-29T14:34:59.255224+0000","last_unstale":"2024-06-29T14:35:00.240902+0000","last_undegraded":"2024-06-29T14:35:00.240902+0000","last_fullsized":"2024-06-29T14:35:00.240902+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:36.215441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241803+0000","last_change":"2024-06-29T14:34:58.246382+0000","last_active":"2024-06-29T14:35:00.241803+0000","last_peered":"2024-06-29T14:35:00.241803+0000","last_clean":"2024-06-29T14:35:00.241803+0000","last_became_active":"2024-06-29T14:34:58.246079+0000","last_became_peered":"2024-06-29T14:34:58.246079+0000","last_unstale":"2024-06-29T14:35:00.241803+0000","last_undegraded":"2024-06-29T14:35:00.241803+0000","last_fullsized":"2024-06-29T14:35:00.241803+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:01:17.782537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235837+0000","last_change":"2024-06-29T14:34:59.250681+0000","last_active":"2024-06-29T14:35:00.235837+0000","last_peered":"2024-06-29T14:35:00.235837+0000","last_clean":"2024-06-29T14:35:00.235837+0000","last_became_active":"2024-06-29T14:34:59.247904+0000","last_became_peered":"2024-06-29T14:34:59.247904+0000","last_unstale":"2024-06-29T14:35:00.235837+0000","last_undegraded":"2024-06-29T14:35:00.235837+0000","last_fullsized":"2024-06-29T14:35:00.235837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:08.456231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"60'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242137+0000","last_change":"2024-06-29T14:34:58.245028+0000","last_active":"2024-06-29T14:35:00.242137+0000","last_peered":"2024-06-29T14:35:00.242137+0000","last_clean":"2024-06-29T14:35:00.242137+0000","last_became_active":"2024-06-29T14:34:58.244722+0000","last_became_peered":"2024-06-29T14:34:58.244722+0000","last_unstale":"2024-06-29T14:35:00.242137+0000","last_undegraded":"2024-06-29T14:35:00.242137+0000","last_fullsized":"2024-06-29T14:35:00.242137+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:42:52.316090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240884+0000","last_change":"2024-06-29T14:34:59.261932+0000","last_active":"2024-06-29T14:35:00.240884+0000","last_peered":"2024-06-29T14:35:00.240884+0000","last_clean":"2024-06-29T14:35:00.240884+0000","last_became_active":"2024-06-29T14:34:59.260450+0000","last_became_peered":"2024-06-29T14:34:59.260450+0000","last_unstale":"2024-06-29T14:35:00.240884+0000","last_undegraded":"2024-06-29T14:35:00.240884+0000","last_fullsized":"2024-06-29T14:35:00.240884+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:09.077088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"60'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235837+0000","last_change":"2024-06-29T14:34:58.240292+0000","last_active":"2024-06-29T14:35:00.235837+0000","last_peered":"2024-06-29T14:35:00.235837+0000","last_clean":"2024-06-29T14:35:00.235837+0000","last_became_active":"2024-06-29T14:34:58.239947+0000","last_became_peered":"2024-06-29T14:34:58.239947+0000","last_unstale":"2024-06-29T14:35:00.235837+0000","last_undegraded":"2024-06-29T14:35:00.235837+0000","last_fullsized":"2024-06-29T14:35:00.235837+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:17.431157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240908+0000","last_change":"2024-06-29T14:34:59.255535+0000","last_active":"2024-06-29T14:35:00.240908+0000","last_peered":"2024-06-29T14:35:00.240908+0000","last_clean":"2024-06-29T14:35:00.240908+0000","last_became_active":"2024-06-29T14:34:59.255152+0000","last_became_peered":"2024-06-29T14:34:59.255152+0000","last_unstale":"2024-06-29T14:35:00.240908+0000","last_undegraded":"2024-06-29T14:35:00.240908+0000","last_fullsized":"2024-06-29T14:35:00.240908+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:11:18.320387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235749+0000","last_change":"2024-06-29T14:34:58.246611+0000","last_active":"2024-06-29T14:35:00.235749+0000","last_peered":"2024-06-29T14:35:00.235749+0000","last_clean":"2024-06-29T14:35:00.235749+0000","last_became_active":"2024-06-29T14:34:58.246261+0000","last_became_peered":"2024-06-29T14:34:58.246261+0000","last_unstale":"2024-06-29T14:35:00.235749+0000","last_undegraded":"2024-06-29T14:35:00.235749+0000","last_fullsized":"2024-06-29T14:35:00.235749+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:09:45.439479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240951+0000","last_change":"2024-06-29T14:34:59.257352+0000","last_active":"2024-06-29T14:35:00.240951+0000","last_peered":"2024-06-29T14:35:00.240951+0000","last_clean":"2024-06-29T14:35:00.240951+0000","last_became_active":"2024-06-29T14:34:59.254553+0000","last_became_peered":"2024-06-29T14:34:59.254553+0000","last_unstale":"2024-06-29T14:35:00.240951+0000","last_undegraded":"2024-06-29T14:35:00.240951+0000","last_fullsized":"2024-06-29T14:35:00.240951+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:37.045557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241008+0000","last_change":"2024-06-29T14:34:58.252416+0000","last_active":"2024-06-29T14:35:00.241008+0000","last_peered":"2024-06-29T14:35:00.241008+0000","last_clean":"2024-06-29T14:35:00.241008+0000","last_became_active":"2024-06-29T14:34:58.252134+0000","last_became_peered":"2024-06-29T14:34:58.252134+0000","last_unstale":"2024-06-29T14:35:00.241008+0000","last_undegraded":"2024-06-29T14:35:00.241008+0000","last_fullsized":"2024-06-29T14:35:00.241008+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:35:30.509733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236034+0000","last_change":"2024-06-29T14:34:59.252708+0000","last_active":"2024-06-29T14:35:00.236034+0000","last_peered":"2024-06-29T14:35:00.236034+0000","last_clean":"2024-06-29T14:35:00.236034+0000","last_became_active":"2024-06-29T14:34:59.252379+0000","last_became_peered":"2024-06-29T14:34:59.252379+0000","last_unstale":"2024-06-29T14:35:00.236034+0000","last_undegraded":"2024-06-29T14:35:00.236034+0000","last_fullsized":"2024-06-29T14:35:00.236034+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:04.197860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236167+0000","last_change":"2024-06-29T14:34:58.247292+0000","last_active":"2024-06-29T14:35:00.236167+0000","last_peered":"2024-06-29T14:35:00.236167+0000","last_clean":"2024-06-29T14:35:00.236167+0000","last_became_active":"2024-06-29T14:34:58.246897+0000","last_became_peered":"2024-06-29T14:34:58.246897+0000","last_unstale":"2024-06-29T14:35:00.236167+0000","last_undegraded":"2024-06-29T14:35:00.236167+0000","last_fullsized":"2024-06-29T14:35:00.236167+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:10.240446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242213+0000","last_change":"2024-06-29T14:34:59.249506+0000","last_active":"2024-06-29T14:35:00.242213+0000","last_peered":"2024-06-29T14:35:00.242213+0000","last_clean":"2024-06-29T14:35:00.242213+0000","last_became_active":"2024-06-29T14:34:59.248397+0000","last_became_peered":"2024-06-29T14:34:59.248397+0000","last_unstale":"2024-06-29T14:35:00.242213+0000","last_undegraded":"2024-06-29T14:35:00.242213+0000","last_fullsized":"2024-06-29T14:35:00.242213+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:09.991806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241961+0000","last_change":"2024-06-29T14:34:58.247221+0000","last_active":"2024-06-29T14:35:00.241961+0000","last_peered":"2024-06-29T14:35:00.241961+0000","last_clean":"2024-06-29T14:35:00.241961+0000","last_became_active":"2024-06-29T14:34:58.246901+0000","last_became_peered":"2024-06-29T14:34:58.246901+0000","last_unstale":"2024-06-29T14:35:00.241961+0000","last_undegraded":"2024-06-29T14:35:00.241961+0000","last_fullsized":"2024-06-29T14:35:00.241961+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:30:01.369451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241827+0000","last_change":"2024-06-29T14:34:59.257606+0000","last_active":"2024-06-29T14:35:00.241827+0000","last_peered":"2024-06-29T14:35:00.241827+0000","last_clean":"2024-06-29T14:35:00.241827+0000","last_became_active":"2024-06-29T14:34:59.257289+0000","last_became_peered":"2024-06-29T14:34:59.257289+0000","last_unstale":"2024-06-29T14:35:00.241827+0000","last_undegraded":"2024-06-29T14:35:00.241827+0000","last_fullsized":"2024-06-29T14:35:00.241827+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:48:37.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"60'3","reported_seq":17,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242232+0000","last_change":"2024-06-29T14:34:58.249391+0000","last_active":"2024-06-29T14:35:00.242232+0000","last_peered":"2024-06-29T14:35:00.242232+0000","last_clean":"2024-06-29T14:35:00.242232+0000","last_became_active":"2024-06-29T14:34:58.248695+0000","last_became_peered":"2024-06-29T14:34:58.248695+0000","last_unstale":"2024-06-29T14:35:00.242232+0000","last_undegraded":"2024-06-29T14:35:00.242232+0000","last_fullsized":"2024-06-29T14:35:00.242232+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:09:51.574526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242146+0000","last_change":"2024-06-29T14:34:59.262489+0000","last_active":"2024-06-29T14:35:00.242146+0000","last_peered":"2024-06-29T14:35:00.242146+0000","last_clean":"2024-06-29T14:35:00.242146+0000","last_became_active":"2024-06-29T14:34:59.262074+0000","last_became_peered":"2024-06-29T14:34:59.262074+0000","last_unstale":"2024-06-29T14:35:00.242146+0000","last_undegraded":"2024-06-29T14:35:00.242146+0000","last_fullsized":"2024-06-29T14:35:00.242146+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:25.665738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"60'6","reported_seq":20,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235311+0000","last_change":"2024-06-29T14:34:58.245922+0000","last_active":"2024-06-29T14:35:00.235311+0000","last_peered":"2024-06-29T14:35:00.235311+0000","last_clean":"2024-06-29T14:35:00.235311+0000","last_became_active":"2024-06-29T14:34:58.245623+0000","last_became_peered":"2024-06-29T14:34:58.245623+0000","last_unstale":"2024-06-29T14:35:00.235311+0000","last_undegraded":"2024-06-29T14:35:00.235311+0000","last_fullsized":"2024-06-29T14:35:00.235311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:08.594875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241213+0000","last_change":"2024-06-29T14:34:59.254977+0000","last_active":"2024-06-29T14:35:00.241213+0000","last_peered":"2024-06-29T14:35:00.241213+0000","last_clean":"2024-06-29T14:35:00.241213+0000","last_became_active":"2024-06-29T14:34:59.252975+0000","last_became_peered":"2024-06-29T14:34:59.252975+0000","last_unstale":"2024-06-29T14:35:00.241213+0000","last_undegraded":"2024-06-29T14:35:00.241213+0000","last_fullsized":"2024-06-29T14:35:00.241213+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:09.077088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242233+0000","last_change":"2024-06-29T14:34:58.250625+0000","last_active":"2024-06-29T14:35:00.242233+0000","last_peered":"2024-06-29T14:35:00.242233+0000","last_clean":"2024-06-29T14:35:00.242233+0000","last_became_active":"2024-06-29T14:34:58.250100+0000","last_became_peered":"2024-06-29T14:34:58.250100+0000","last_unstale":"2024-06-29T14:35:00.242233+0000","last_undegraded":"2024-06-29T14:35:00.242233+0000","last_fullsized":"2024-06-29T14:35:00.242233+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:34:34.857992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241031+0000","last_change":"2024-06-29T14:34:59.253640+0000","last_active":"2024-06-29T14:35:00.241031+0000","last_peered":"2024-06-29T14:35:00.241031+0000","last_clean":"2024-06-29T14:35:00.241031+0000","last_became_active":"2024-06-29T14:34:59.253343+0000","last_became_peered":"2024-06-29T14:34:59.253343+0000","last_unstale":"2024-06-29T14:35:00.241031+0000","last_undegraded":"2024-06-29T14:35:00.241031+0000","last_fullsized":"2024-06-29T14:35:00.241031+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:52:52.718656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241350+0000","last_change":"2024-06-29T14:34:58.247199+0000","last_active":"2024-06-29T14:35:00.241350+0000","last_peered":"2024-06-29T14:35:00.241350+0000","last_clean":"2024-06-29T14:35:00.241350+0000","last_became_active":"2024-06-29T14:34:58.246809+0000","last_became_peered":"2024-06-29T14:34:58.246809+0000","last_unstale":"2024-06-29T14:35:00.241350+0000","last_undegraded":"2024-06-29T14:35:00.241350+0000","last_fullsized":"2024-06-29T14:35:00.241350+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:23.103200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241459+0000","last_change":"2024-06-29T14:34:59.250883+0000","last_active":"2024-06-29T14:35:00.241459+0000","last_peered":"2024-06-29T14:35:00.241459+0000","last_clean":"2024-06-29T14:35:00.241459+0000","last_became_active":"2024-06-29T14:34:59.250561+0000","last_became_peered":"2024-06-29T14:34:59.250561+0000","last_unstale":"2024-06-29T14:35:00.241459+0000","last_undegraded":"2024-06-29T14:35:00.241459+0000","last_fullsized":"2024-06-29T14:35:00.241459+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:52.377285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242030+0000","last_change":"2024-06-29T14:34:58.249932+0000","last_active":"2024-06-29T14:35:00.242030+0000","last_peered":"2024-06-29T14:35:00.242030+0000","last_clean":"2024-06-29T14:35:00.242030+0000","last_became_active":"2024-06-29T14:34:58.249209+0000","last_became_peered":"2024-06-29T14:34:58.249209+0000","last_unstale":"2024-06-29T14:35:00.242030+0000","last_undegraded":"2024-06-29T14:35:00.242030+0000","last_fullsized":"2024-06-29T14:35:00.242030+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:30.049435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242040+0000","last_change":"2024-06-29T14:34:59.259241+0000","last_active":"2024-06-29T14:35:00.242040+0000","last_peered":"2024-06-29T14:35:00.242040+0000","last_clean":"2024-06-29T14:35:00.242040+0000","last_became_active":"2024-06-29T14:34:59.258928+0000","last_became_peered":"2024-06-29T14:34:59.258928+0000","last_unstale":"2024-06-29T14:35:00.242040+0000","last_undegraded":"2024-06-29T14:35:00.242040+0000","last_fullsized":"2024-06-29T14:35:00.242040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:32.408127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"60'8","reported_seq":22,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241111+0000","last_change":"2024-06-29T14:34:58.249139+0000","last_active":"2024-06-29T14:35:00.241111+0000","last_peered":"2024-06-29T14:35:00.241111+0000","last_clean":"2024-06-29T14:35:00.241111+0000","last_became_active":"2024-06-29T14:34:58.248847+0000","last_became_peered":"2024-06-29T14:34:58.248847+0000","last_unstale":"2024-06-29T14:35:00.241111+0000","last_undegraded":"2024-06-29T14:35:00.241111+0000","last_fullsized":"2024-06-29T14:35:00.241111+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:36.410979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235189+0000","last_change":"2024-06-29T14:34:59.251266+0000","last_active":"2024-06-29T14:35:00.235189+0000","last_peered":"2024-06-29T14:35:00.235189+0000","last_clean":"2024-06-29T14:35:00.235189+0000","last_became_active":"2024-06-29T14:34:59.250928+0000","last_became_peered":"2024-06-29T14:34:59.250928+0000","last_unstale":"2024-06-29T14:35:00.235189+0000","last_undegraded":"2024-06-29T14:35:00.235189+0000","last_fullsized":"2024-06-29T14:35:00.235189+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:28:52.124526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"60'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235218+0000","last_change":"2024-06-29T14:34:58.247513+0000","last_active":"2024-06-29T14:35:00.235218+0000","last_peered":"2024-06-29T14:35:00.235218+0000","last_clean":"2024-06-29T14:35:00.235218+0000","last_became_active":"2024-06-29T14:34:58.246500+0000","last_became_peered":"2024-06-29T14:34:58.246500+0000","last_unstale":"2024-06-29T14:35:00.235218+0000","last_undegraded":"2024-06-29T14:35:00.235218+0000","last_fullsized":"2024-06-29T14:35:00.235218+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:14.428931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241407+0000","last_change":"2024-06-29T14:34:59.259355+0000","last_active":"2024-06-29T14:35:00.241407+0000","last_peered":"2024-06-29T14:35:00.241407+0000","last_clean":"2024-06-29T14:35:00.241407+0000","last_became_active":"2024-06-29T14:34:59.259033+0000","last_became_peered":"2024-06-29T14:34:59.259033+0000","last_unstale":"2024-06-29T14:35:00.241407+0000","last_undegraded":"2024-06-29T14:35:00.241407+0000","last_fullsized":"2024-06-29T14:35:00.241407+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:34:57.120364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236293+0000","last_change":"2024-06-29T14:34:58.245944+0000","last_active":"2024-06-29T14:35:00.236293+0000","last_peered":"2024-06-29T14:35:00.236293+0000","last_clean":"2024-06-29T14:35:00.236293+0000","last_became_active":"2024-06-29T14:34:58.245609+0000","last_became_peered":"2024-06-29T14:34:58.245609+0000","last_unstale":"2024-06-29T14:35:00.236293+0000","last_undegraded":"2024-06-29T14:35:00.236293+0000","last_fullsized":"2024-06-29T14:35:00.236293+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:15.366522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241786+0000","last_change":"2024-06-29T14:34:59.257340+0000","last_active":"2024-06-29T14:35:00.241786+0000","last_peered":"2024-06-29T14:35:00.241786+0000","last_clean":"2024-06-29T14:35:00.241786+0000","last_became_active":"2024-06-29T14:34:59.256832+0000","last_became_peered":"2024-06-29T14:34:59.256832+0000","last_unstale":"2024-06-29T14:35:00.241786+0000","last_undegraded":"2024-06-29T14:35:00.241786+0000","last_fullsized":"2024-06-29T14:35:00.241786+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:31:47.110087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"60'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236844+0000","last_change":"2024-06-29T14:34:58.249091+0000","last_active":"2024-06-29T14:35:00.236844+0000","last_peered":"2024-06-29T14:35:00.236844+0000","last_clean":"2024-06-29T14:35:00.236844+0000","last_became_active":"2024-06-29T14:34:58.248782+0000","last_became_peered":"2024-06-29T14:34:58.248782+0000","last_unstale":"2024-06-29T14:35:00.236844+0000","last_undegraded":"2024-06-29T14:35:00.236844+0000","last_fullsized":"2024-06-29T14:35:00.236844+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:33:22.688773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241702+0000","last_change":"2024-06-29T14:34:59.256831+0000","last_active":"2024-06-29T14:35:00.241702+0000","last_peered":"2024-06-29T14:35:00.241702+0000","last_clean":"2024-06-29T14:35:00.241702+0000","last_became_active":"2024-06-29T14:34:59.255345+0000","last_became_peered":"2024-06-29T14:34:59.255345+0000","last_unstale":"2024-06-29T14:35:00.241702+0000","last_undegraded":"2024-06-29T14:35:00.241702+0000","last_fullsized":"2024-06-29T14:35:00.241702+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:15:43.422063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241923+0000","last_change":"2024-06-29T14:34:58.248329+0000","last_active":"2024-06-29T14:35:00.241923+0000","last_peered":"2024-06-29T14:35:00.241923+0000","last_clean":"2024-06-29T14:35:00.241923+0000","last_became_active":"2024-06-29T14:34:58.248027+0000","last_became_peered":"2024-06-29T14:34:58.248027+0000","last_unstale":"2024-06-29T14:35:00.241923+0000","last_undegraded":"2024-06-29T14:35:00.241923+0000","last_fullsized":"2024-06-29T14:35:00.241923+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:43.766922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241728+0000","last_change":"2024-06-29T14:34:59.245926+0000","last_active":"2024-06-29T14:35:00.241728+0000","last_peered":"2024-06-29T14:35:00.241728+0000","last_clean":"2024-06-29T14:35:00.241728+0000","last_became_active":"2024-06-29T14:34:59.245622+0000","last_became_peered":"2024-06-29T14:34:59.245622+0000","last_unstale":"2024-06-29T14:35:00.241728+0000","last_undegraded":"2024-06-29T14:35:00.241728+0000","last_fullsized":"2024-06-29T14:35:00.241728+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:32.817173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242293+0000","last_change":"2024-06-29T14:34:58.245125+0000","last_active":"2024-06-29T14:35:00.242293+0000","last_peered":"2024-06-29T14:35:00.242293+0000","last_clean":"2024-06-29T14:35:00.242293+0000","last_became_active":"2024-06-29T14:34:58.244855+0000","last_became_peered":"2024-06-29T14:34:58.244855+0000","last_unstale":"2024-06-29T14:35:00.242293+0000","last_undegraded":"2024-06-29T14:35:00.242293+0000","last_fullsized":"2024-06-29T14:35:00.242293+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:49:37.819067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240467+0000","last_change":"2024-06-29T14:34:59.260429+0000","last_active":"2024-06-29T14:35:00.240467+0000","last_peered":"2024-06-29T14:35:00.240467+0000","last_clean":"2024-06-29T14:35:00.240467+0000","last_became_active":"2024-06-29T14:34:59.260124+0000","last_became_peered":"2024-06-29T14:34:59.260124+0000","last_unstale":"2024-06-29T14:35:00.240467+0000","last_undegraded":"2024-06-29T14:35:00.240467+0000","last_fullsized":"2024-06-29T14:35:00.240467+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:05:55.273115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235599+0000","last_change":"2024-06-29T14:34:58.248755+0000","last_active":"2024-06-29T14:35:00.235599+0000","last_peered":"2024-06-29T14:35:00.235599+0000","last_clean":"2024-06-29T14:35:00.235599+0000","last_became_active":"2024-06-29T14:34:58.248339+0000","last_became_peered":"2024-06-29T14:34:58.248339+0000","last_unstale":"2024-06-29T14:35:00.235599+0000","last_undegraded":"2024-06-29T14:35:00.235599+0000","last_fullsized":"2024-06-29T14:35:00.235599+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:56.808469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235240+0000","last_change":"2024-06-29T14:34:59.250934+0000","last_active":"2024-06-29T14:35:00.235240+0000","last_peered":"2024-06-29T14:35:00.235240+0000","last_clean":"2024-06-29T14:35:00.235240+0000","last_became_active":"2024-06-29T14:34:59.250245+0000","last_became_peered":"2024-06-29T14:34:59.250245+0000","last_unstale":"2024-06-29T14:35:00.235240+0000","last_undegraded":"2024-06-29T14:35:00.235240+0000","last_fullsized":"2024-06-29T14:35:00.235240+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:09.337433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240707+0000","last_change":"2024-06-29T14:34:58.247657+0000","last_active":"2024-06-29T14:35:00.240707+0000","last_peered":"2024-06-29T14:35:00.240707+0000","last_clean":"2024-06-29T14:35:00.240707+0000","last_became_active":"2024-06-29T14:34:58.247368+0000","last_became_peered":"2024-06-29T14:34:58.247368+0000","last_unstale":"2024-06-29T14:35:00.240707+0000","last_undegraded":"2024-06-29T14:35:00.240707+0000","last_fullsized":"2024-06-29T14:35:00.240707+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:04.574489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240907+0000","last_change":"2024-06-29T14:34:59.261666+0000","last_active":"2024-06-29T14:35:00.240907+0000","last_peered":"2024-06-29T14:35:00.240907+0000","last_clean":"2024-06-29T14:35:00.240907+0000","last_became_active":"2024-06-29T14:34:59.261268+0000","last_became_peered":"2024-06-29T14:34:59.261268+0000","last_unstale":"2024-06-29T14:35:00.240907+0000","last_undegraded":"2024-06-29T14:35:00.240907+0000","last_fullsized":"2024-06-29T14:35:00.240907+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:15:43.422063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241212+0000","last_change":"2024-06-29T14:34:58.250753+0000","last_active":"2024-06-29T14:35:00.241212+0000","last_peered":"2024-06-29T14:35:00.241212+0000","last_clean":"2024-06-29T14:35:00.241212+0000","last_became_active":"2024-06-29T14:34:58.250412+0000","last_became_peered":"2024-06-29T14:34:58.250412+0000","last_unstale":"2024-06-29T14:35:00.241212+0000","last_undegraded":"2024-06-29T14:35:00.241212+0000","last_fullsized":"2024-06-29T14:35:00.241212+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:29:39.596110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242164+0000","last_change":"2024-06-29T14:34:59.253459+0000","last_active":"2024-06-29T14:35:00.242164+0000","last_peered":"2024-06-29T14:35:00.242164+0000","last_clean":"2024-06-29T14:35:00.242164+0000","last_became_active":"2024-06-29T14:34:59.252828+0000","last_became_peered":"2024-06-29T14:34:59.252828+0000","last_unstale":"2024-06-29T14:35:00.242164+0000","last_undegraded":"2024-06-29T14:35:00.242164+0000","last_fullsized":"2024-06-29T14:35:00.242164+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:51:43.053123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"61'1","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.249894+0000","last_change":"2024-06-29T14:34:58.250190+0000","last_active":"2024-06-29T14:35:00.249894+0000","last_peered":"2024-06-29T14:35:00.249894+0000","last_clean":"2024-06-29T14:35:00.249894+0000","last_became_active":"2024-06-29T14:34:58.249847+0000","last_became_peered":"2024-06-29T14:34:58.249847+0000","last_unstale":"2024-06-29T14:35:00.249894+0000","last_undegraded":"2024-06-29T14:35:00.249894+0000","last_fullsized":"2024-06-29T14:35:00.249894+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:33.533408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242010+0000","last_change":"2024-06-29T14:34:59.254990+0000","last_active":"2024-06-29T14:35:00.242010+0000","last_peered":"2024-06-29T14:35:00.242010+0000","last_clean":"2024-06-29T14:35:00.242010+0000","last_became_active":"2024-06-29T14:34:59.254683+0000","last_became_peered":"2024-06-29T14:34:59.254683+0000","last_unstale":"2024-06-29T14:35:00.242010+0000","last_undegraded":"2024-06-29T14:35:00.242010+0000","last_fullsized":"2024-06-29T14:35:00.242010+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:52:16.345341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242296+0000","last_change":"2024-06-29T14:34:58.249410+0000","last_active":"2024-06-29T14:35:00.242296+0000","last_peered":"2024-06-29T14:35:00.242296+0000","last_clean":"2024-06-29T14:35:00.242296+0000","last_became_active":"2024-06-29T14:34:58.249106+0000","last_became_peered":"2024-06-29T14:34:58.249106+0000","last_unstale":"2024-06-29T14:35:00.242296+0000","last_undegraded":"2024-06-29T14:35:00.242296+0000","last_fullsized":"2024-06-29T14:35:00.242296+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:09:07.727419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242093+0000","last_change":"2024-06-29T14:34:59.250049+0000","last_active":"2024-06-29T14:35:00.242093+0000","last_peered":"2024-06-29T14:35:00.242093+0000","last_clean":"2024-06-29T14:35:00.242093+0000","last_became_active":"2024-06-29T14:34:59.249688+0000","last_became_peered":"2024-06-29T14:34:59.249688+0000","last_unstale":"2024-06-29T14:35:00.242093+0000","last_undegraded":"2024-06-29T14:35:00.242093+0000","last_fullsized":"2024-06-29T14:35:00.242093+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:46.170143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240380+0000","last_change":"2024-06-29T14:34:58.244944+0000","last_active":"2024-06-29T14:35:00.240380+0000","last_peered":"2024-06-29T14:35:00.240380+0000","last_clean":"2024-06-29T14:35:00.240380+0000","last_became_active":"2024-06-29T14:34:58.244695+0000","last_became_peered":"2024-06-29T14:34:58.244695+0000","last_unstale":"2024-06-29T14:35:00.240380+0000","last_undegraded":"2024-06-29T14:35:00.240380+0000","last_fullsized":"2024-06-29T14:35:00.240380+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:49.115624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241766+0000","last_change":"2024-06-29T14:34:59.251433+0000","last_active":"2024-06-29T14:35:00.241766+0000","last_peered":"2024-06-29T14:35:00.241766+0000","last_clean":"2024-06-29T14:35:00.241766+0000","last_became_active":"2024-06-29T14:34:59.251117+0000","last_became_peered":"2024-06-29T14:34:59.251117+0000","last_unstale":"2024-06-29T14:35:00.241766+0000","last_undegraded":"2024-06-29T14:35:00.241766+0000","last_fullsized":"2024-06-29T14:35:00.241766+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:20.698906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241565+0000","last_change":"2024-06-29T14:34:58.248932+0000","last_active":"2024-06-29T14:35:00.241565+0000","last_peered":"2024-06-29T14:35:00.241565+0000","last_clean":"2024-06-29T14:35:00.241565+0000","last_became_active":"2024-06-29T14:34:58.248576+0000","last_became_peered":"2024-06-29T14:34:58.248576+0000","last_unstale":"2024-06-29T14:35:00.241565+0000","last_undegraded":"2024-06-29T14:35:00.241565+0000","last_fullsized":"2024-06-29T14:35:00.241565+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:27.920891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.242155+0000","last_change":"2024-06-29T14:34:59.256887+0000","last_active":"2024-06-29T14:35:00.242155+0000","last_peered":"2024-06-29T14:35:00.242155+0000","last_clean":"2024-06-29T14:35:00.242155+0000","last_became_active":"2024-06-29T14:34:59.256525+0000","last_became_peered":"2024-06-29T14:34:59.256525+0000","last_unstale":"2024-06-29T14:35:00.242155+0000","last_undegraded":"2024-06-29T14:35:00.242155+0000","last_fullsized":"2024-06-29T14:35:00.242155+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:46.170143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235977+0000","last_change":"2024-06-29T14:34:58.245383+0000","last_active":"2024-06-29T14:35:00.235977+0000","last_peered":"2024-06-29T14:35:00.235977+0000","last_clean":"2024-06-29T14:35:00.235977+0000","last_became_active":"2024-06-29T14:34:58.245057+0000","last_became_peered":"2024-06-29T14:34:58.245057+0000","last_unstale":"2024-06-29T14:35:00.235977+0000","last_undegraded":"2024-06-29T14:35:00.235977+0000","last_fullsized":"2024-06-29T14:35:00.235977+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:49:48.013292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235477+0000","last_change":"2024-06-29T14:34:59.251278+0000","last_active":"2024-06-29T14:35:00.235477+0000","last_peered":"2024-06-29T14:35:00.235477+0000","last_clean":"2024-06-29T14:35:00.235477+0000","last_became_active":"2024-06-29T14:34:59.250390+0000","last_became_peered":"2024-06-29T14:34:59.250390+0000","last_unstale":"2024-06-29T14:35:00.235477+0000","last_undegraded":"2024-06-29T14:35:00.235477+0000","last_fullsized":"2024-06-29T14:35:00.235477+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:29.888030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.240315+0000","last_change":"2024-06-29T14:34:58.250852+0000","last_active":"2024-06-29T14:35:00.240315+0000","last_peered":"2024-06-29T14:35:00.240315+0000","last_clean":"2024-06-29T14:35:00.240315+0000","last_became_active":"2024-06-29T14:34:58.250341+0000","last_became_peered":"2024-06-29T14:34:58.250341+0000","last_unstale":"2024-06-29T14:35:00.240315+0000","last_undegraded":"2024-06-29T14:35:00.240315+0000","last_fullsized":"2024-06-29T14:35:00.240315+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:17.431157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.235352+0000","last_change":"2024-06-29T14:34:59.253737+0000","last_active":"2024-06-29T14:35:00.235352+0000","last_peered":"2024-06-29T14:35:00.235352+0000","last_clean":"2024-06-29T14:35:00.235352+0000","last_became_active":"2024-06-29T14:34:59.253361+0000","last_became_peered":"2024-06-29T14:34:59.253361+0000","last_unstale":"2024-06-29T14:35:00.235352+0000","last_undegraded":"2024-06-29T14:35:00.235352+0000","last_fullsized":"2024-06-29T14:35:00.235352+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:08.498050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241216+0000","last_change":"2024-06-29T14:34:58.247157+0000","last_active":"2024-06-29T14:35:00.241216+0000","last_peered":"2024-06-29T14:35:00.241216+0000","last_clean":"2024-06-29T14:35:00.241216+0000","last_became_active":"2024-06-29T14:34:58.246836+0000","last_became_peered":"2024-06-29T14:34:58.246836+0000","last_unstale":"2024-06-29T14:35:00.241216+0000","last_undegraded":"2024-06-29T14:35:00.241216+0000","last_fullsized":"2024-06-29T14:35:00.241216+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:26.334699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241870+0000","last_change":"2024-06-29T14:34:59.256741+0000","last_active":"2024-06-29T14:35:00.241870+0000","last_peered":"2024-06-29T14:35:00.241870+0000","last_clean":"2024-06-29T14:35:00.241870+0000","last_became_active":"2024-06-29T14:34:59.256439+0000","last_became_peered":"2024-06-29T14:34:59.256439+0000","last_unstale":"2024-06-29T14:35:00.241870+0000","last_undegraded":"2024-06-29T14:35:00.241870+0000","last_fullsized":"2024-06-29T14:35:00.241870+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:58.222696+0000","last_clean_scrub_stamp":"2024-06-29T14:34:58.222696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:42:18.547863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"60'2","reported_seq":16,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.236885+0000","last_change":"2024-06-29T14:34:58.241817+0000","last_active":"2024-06-29T14:35:00.236885+0000","last_peered":"2024-06-29T14:35:00.236885+0000","last_clean":"2024-06-29T14:35:00.236885+0000","last_became_active":"2024-06-29T14:34:58.241457+0000","last_became_peered":"2024-06-29T14:34:58.241457+0000","last_unstale":"2024-06-29T14:35:00.236885+0000","last_undegraded":"2024-06-29T14:35:00.236885+0000","last_fullsized":"2024-06-29T14:35:00.236885+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:34:57.217515+0000","last_clean_scrub_stamp":"2024-06-29T14:34:57.217515+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:54:34.801214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"49'86","reported_seq":244,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-06-29T14:35:00.241314+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:35:00.241314+0000","last_peered":"2024-06-29T14:35:00.241314+0000","last_clean":"2024-06-29T14:35:00.241314+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:35:00.241314+0000","last_undegraded":"2024-06-29T14:35:00.241314+0000","last_fullsized":"2024-06-29T14:35:00.241314+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":407,"num_write":125,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575002,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1310720,"data_stored":925058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":720896,"data_stored":335272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575001,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":334670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575002,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54100,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317740,"statfs":{"total":96636764160,"available":96581365760,"internally_reserved":0,"allocated":724992,"data_stored":336212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575002,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":737280,"data_stored":341582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36056,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335784,"statfs":{"total":96636764160,"available":96599842816,"internally_reserved":0,"allocated":729088,"data_stored":336234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575001,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49944,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321896,"statfs":{"total":96636764160,"available":96585621504,"internally_reserved":0,"allocated":729088,"data_stored":341014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575002,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32472,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1323008,"data_stored":931346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:35:03.635 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:35:03.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:35:03.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.795+0000 7f4802cc3640 1 -- 172.21.15.81:0/416676569 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47fc0d2950 msgr2=0x7f47fc0b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.795+0000 7f4802cc3640 1 --2- 172.21.15.81:0/416676569 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47fc0d2950 0x7f47fc0b3b00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f47f4005fd0 tx=0x7f47f402a820 comp rx=0 tx=0).stop 2024-06-29T14:35:03.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.795+0000 7f4802cc3640 1 -- 172.21.15.81:0/416676569 shutdown_connections 2024-06-29T14:35:03.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.795+0000 7f4802cc3640 1 --2- 172.21.15.81:0/416676569 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47fc0d2950 0x7f47fc0b3b00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.795+0000 7f4802cc3640 1 --2- 172.21.15.81:0/416676569 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47fc0ce770 0x7f47fc0b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.795+0000 7f4802cc3640 1 -- 172.21.15.81:0/416676569 >> 172.21.15.81:0/416676569 conn(0x7f47fc0d3c60 msgr2=0x7f47fc0d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:03.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.795+0000 7f4802cc3640 1 -- 172.21.15.81:0/416676569 shutdown_connections 2024-06-29T14:35:03.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.795+0000 7f4802cc3640 1 -- 172.21.15.81:0/416676569 wait complete. 2024-06-29T14:35:03.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.797+0000 7f4802cc3640 1 Processor -- start 2024-06-29T14:35:03.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.797+0000 7f4802cc3640 1 -- start start 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f4802cc3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47fc0ce770 0x7f47fc1d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f4802cc3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47fc0ceb80 0x7f47fc0b8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f4802cc3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47fc0d2950 0x7f47fc1e10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f4802cc3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc0d15a0 con 0x7f47fc0ce770 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f4802cc3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc0d1420 con 0x7f47fc0ceb80 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f4802cc3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f47fc0d4990 con 0x7f47fc0d2950 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f4800a5f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47fc0ce770 0x7f47fc1d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f4800a5f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47fc0ce770 0x7f47fc1d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50400/0 (socket says 172.21.15.81:50400) 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f4800a5f640 1 -- 172.21.15.81:0/794393072 learned_addr learned my addr 172.21.15.81:0/794393072 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f47f3fff640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47fc0ceb80 0x7f47fc0b8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.798+0000 7f4801260640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47fc0d2950 0x7f47fc1e10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f4800a5f640 1 -- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47fc0d2950 msgr2=0x7f47fc1e10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f4800a5f640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47fc0d2950 0x7f47fc1e10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f4800a5f640 1 -- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47fc0ceb80 msgr2=0x7f47fc0b8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f4800a5f640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47fc0ceb80 0x7f47fc0b8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f4800a5f640 1 -- 172.21.15.81:0/794393072 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc1e1dc0 con 0x7f47fc0ce770 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f4800a5f640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47fc0ce770 0x7f47fc1d0b20 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009fd0 tx=0x7f47f800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f47f17fa640 1 -- 172.21.15.81:0/794393072 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f800ccc0 con 0x7f47fc0ce770 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f4802cc3640 1 -- 172.21.15.81:0/794393072 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47fc1e20a0 con 0x7f47fc0ce770 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f47f17fa640 1 -- 172.21.15.81:0/794393072 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47f800ee70 con 0x7f47fc0ce770 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f47f17fa640 1 -- 172.21.15.81:0/794393072 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f800ce60 con 0x7f47fc0ce770 2024-06-29T14:35:03.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.799+0000 7f4802cc3640 1 -- 172.21.15.81:0/794393072 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47fc0b8870 con 0x7f47fc0ce770 2024-06-29T14:35:03.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.800+0000 7f4802cc3640 1 -- 172.21.15.81:0/794393072 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47fc0d2240 con 0x7f47fc0ce770 2024-06-29T14:35:03.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.801+0000 7f47f17fa640 1 -- 172.21.15.81:0/794393072 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f47f8004280 con 0x7f47fc0ce770 2024-06-29T14:35:03.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.801+0000 7f47f17fa640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f47e80763f0 0x7f47e8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:03.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.801+0000 7f47f17fa640 1 -- 172.21.15.81:0/794393072 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f47f8098280 con 0x7f47fc0ce770 2024-06-29T14:35:03.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.802+0000 7f47f3fff640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f47e80763f0 0x7f47e8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:03.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.802+0000 7f47f3fff640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f47e80763f0 0x7f47e8078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f47fc06e530 tx=0x7f47ec007e90 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:03.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:03.806+0000 7f47f17fa640 1 -- 172.21.15.81:0/794393072 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f47f8061230 con 0x7f47fc0ce770 2024-06-29T14:35:04.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.053+0000 7f4802cc3640 1 -- 172.21.15.81:0/794393072 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f47fc0b1f00 con 0x7f47fc0ce770 2024-06-29T14:35:04.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.056+0000 7f47f17fa640 1 -- 172.21.15.81:0/794393072 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v30) ==== 190+0+147 (secure 0 0 0) 0x7f47f8065530 con 0x7f47fc0ce770 2024-06-29T14:35:04.053 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T14:35:04.054 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T14:35:04.054 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T14:35:04.054 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T14:35:04.054 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:35:04.054 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T14:35:04.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 -- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f47e80763f0 msgr2=0x7f47e8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f47e80763f0 0x7f47e8078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f47fc06e530 tx=0x7f47ec007e90 comp rx=0 tx=0).stop 2024-06-29T14:35:04.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 -- 172.21.15.81:0/794393072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47fc0ce770 msgr2=0x7f47fc1d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47fc0ce770 0x7f47fc1d0b20 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009fd0 tx=0x7f47f800ec20 comp rx=0 tx=0).stop 2024-06-29T14:35:04.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 -- 172.21.15.81:0/794393072 shutdown_connections 2024-06-29T14:35:04.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47fc0d2950 0x7f47fc1e10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47fc0ceb80 0x7f47fc0b8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f47e80763f0 0x7f47e8078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 --2- 172.21.15.81:0/794393072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47fc0ce770 0x7f47fc1d0b20 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 -- 172.21.15.81:0/794393072 >> 172.21.15.81:0/794393072 conn(0x7f47fc0d3c60 msgr2=0x7f47fc1e7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 -- 172.21.15.81:0/794393072 shutdown_connections 2024-06-29T14:35:04.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.058+0000 7f4802cc3640 1 -- 172.21.15.81:0/794393072 wait complete. 2024-06-29T14:35:04.067 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:35:04.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.183+0000 7f0e91ecc640 1 -- 172.21.15.81:0/30952404 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e8c1412f0 msgr2=0x7f0e8c0cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.183+0000 7f0e91ecc640 1 --2- 172.21.15.81:0/30952404 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e8c1412f0 0x7f0e8c0cef90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c009830 tx=0x7f0e7c02a510 comp rx=0 tx=0).stop 2024-06-29T14:35:04.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.183+0000 7f0e91ecc640 1 -- 172.21.15.81:0/30952404 shutdown_connections 2024-06-29T14:35:04.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.183+0000 7f0e91ecc640 1 --2- 172.21.15.81:0/30952404 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e8c0cf4d0 0x7f0e8c073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.183+0000 7f0e91ecc640 1 --2- 172.21.15.81:0/30952404 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e8c1412f0 0x7f0e8c0cef90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.183+0000 7f0e91ecc640 1 -- 172.21.15.81:0/30952404 >> 172.21.15.81:0/30952404 conn(0x7f0e8c06ccc0 msgr2=0x7f0e8c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.183+0000 7f0e91ecc640 1 -- 172.21.15.81:0/30952404 shutdown_connections 2024-06-29T14:35:04.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.183+0000 7f0e91ecc640 1 -- 172.21.15.81:0/30952404 wait complete. 2024-06-29T14:35:04.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.185+0000 7f0e91ecc640 1 Processor -- start 2024-06-29T14:35:04.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.185+0000 7f0e91ecc640 1 -- start start 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.186+0000 7f0e91ecc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e8c073570 0x7f0e8c1f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.186+0000 7f0e91ecc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e8c0cf4d0 0x7f0e8c1533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.186+0000 7f0e91ecc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e8c1412f0 0x7f0e8c0abc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.186+0000 7f0e91ecc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c0b3210 con 0x7f0e8c073570 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.186+0000 7f0e91ecc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c15f280 con 0x7f0e8c1412f0 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.186+0000 7f0e91ecc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0e8c0b3390 con 0x7f0e8c0cf4d0 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.186+0000 7f0e8b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e8c073570 0x7f0e8c1f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.186+0000 7f0e8b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e8c073570 0x7f0e8c1f24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50412/0 (socket says 172.21.15.81:50412) 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.186+0000 7f0e8b7fe640 1 -- 172.21.15.81:0/851870122 learned_addr learned my addr 172.21.15.81:0/851870122 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e8affd640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e8c0cf4d0 0x7f0e8c1533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e8bfff640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e8c1412f0 0x7f0e8c0abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e8b7fe640 1 -- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e8c0cf4d0 msgr2=0x7f0e8c1533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e8b7fe640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e8c0cf4d0 0x7f0e8c1533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e8b7fe640 1 -- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e8c1412f0 msgr2=0x7f0e8c0abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e8b7fe640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e8c1412f0 0x7f0e8c0abc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e8b7fe640 1 -- 172.21.15.81:0/851870122 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c1f2a10 con 0x7f0e8c073570 2024-06-29T14:35:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e8b7fe640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e8c073570 0x7f0e8c1f24d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c000c00 tx=0x7f0e7c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e73fff640 1 -- 172.21.15.81:0/851870122 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c031d60 con 0x7f0e8c073570 2024-06-29T14:35:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e73fff640 1 -- 172.21.15.81:0/851870122 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e7c033e10 con 0x7f0e8c073570 2024-06-29T14:35:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e91ecc640 1 -- 172.21.15.81:0/851870122 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8c0ade50 con 0x7f0e8c073570 2024-06-29T14:35:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e73fff640 1 -- 172.21.15.81:0/851870122 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c033310 con 0x7f0e8c073570 2024-06-29T14:35:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.187+0000 7f0e91ecc640 1 -- 172.21.15.81:0/851870122 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e8c0ae660 con 0x7f0e8c073570 2024-06-29T14:35:04.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.188+0000 7f0e91ecc640 1 -- 172.21.15.81:0/851870122 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e8c0af060 con 0x7f0e8c073570 2024-06-29T14:35:04.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.189+0000 7f0e73fff640 1 -- 172.21.15.81:0/851870122 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f0e7c0334b0 con 0x7f0e8c073570 2024-06-29T14:35:04.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.189+0000 7f0e73fff640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0e78076420 0x7f0e78078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.190+0000 7f0e73fff640 1 -- 172.21.15.81:0/851870122 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f0e7c0b89b0 con 0x7f0e8c073570 2024-06-29T14:35:04.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.190+0000 7f0e8affd640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0e78076420 0x7f0e78078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.190+0000 7f0e8affd640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0e78076420 0x7f0e78078930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c153ea0 tx=0x7f0e80005f90 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.194+0000 7f0e73fff640 1 -- 172.21.15.81:0/851870122 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0e7c04a030 con 0x7f0e8c073570 2024-06-29T14:35:04.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.412+0000 7f0e91ecc640 1 -- 172.21.15.81:0/851870122 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e8c0b3880 con 0x7f0e8c073570 2024-06-29T14:35:04.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.413+0000 7f0e73fff640 1 -- 172.21.15.81:0/851870122 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5229 (secure 0 0 0) 0x7f0e8c1f2a10 con 0x7f0e8c073570 2024-06-29T14:35:04.410 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:35:04.411 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":25,"btime":"2024-06-29T14:35:02:255148+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2980523942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2980523942},{"type":"v1","addr":"172.21.15.167:6836","nonce":2980523942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5635,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2317403375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2317403375},{"type":"v1","addr":"172.21.15.167:6837","nonce":2317403375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/2346221458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2346221458},{"type":"v1","addr":"172.21.15.81:6834","nonce":2346221458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:34:58.602103+0000","modified":"2024-06-29T14:35:01.623553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6033":{"gid":6033,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/2669979711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2669979711},{"type":"v1","addr":"172.21.15.81:6835","nonce":2669979711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6033,"qdb_cluster":[6033]},"id":3}]} 2024-06-29T14:35:04.411 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 25 2024-06-29T14:35:04.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 -- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0e78076420 msgr2=0x7f0e78078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0e78076420 0x7f0e78078930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c153ea0 tx=0x7f0e80005f90 comp rx=0 tx=0).stop 2024-06-29T14:35:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 -- 172.21.15.81:0/851870122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e8c073570 msgr2=0x7f0e8c1f24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e8c073570 0x7f0e8c1f24d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c000c00 tx=0x7f0e7c033c20 comp rx=0 tx=0).stop 2024-06-29T14:35:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 -- 172.21.15.81:0/851870122 shutdown_connections 2024-06-29T14:35:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e8c1412f0 0x7f0e8c0abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e8c0cf4d0 0x7f0e8c1533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f0e78076420 0x7f0e78078930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 --2- 172.21.15.81:0/851870122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e8c073570 0x7f0e8c1f24d0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 -- 172.21.15.81:0/851870122 >> 172.21.15.81:0/851870122 conn(0x7f0e8c06ccc0 msgr2=0x7f0e8c0b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 -- 172.21.15.81:0/851870122 shutdown_connections 2024-06-29T14:35:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.415+0000 7f0e91ecc640 1 -- 172.21.15.81:0/851870122 wait complete. 2024-06-29T14:35:04.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:34:58.602103+0000', 'modified': '2024-06-29T14:35:01.623553+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6033}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6033': {'gid': 6033, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6835/2669979711', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6833', 'nonce': 2669979711}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 2669979711}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6033, 'qdb_cluster': [6033]} 2024-06-29T14:35:04.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:35:04.827 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-06-29T14:35:04.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.945+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3740522472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b580b9040 msgr2=0x7f9b580d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.945+0000 7f9b5d37a640 1 --2- 172.21.15.81:0/3740522472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b580b9040 0x7f9b580d0480 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009830 tx=0x7f9b4802a5a0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.945+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3740522472 shutdown_connections 2024-06-29T14:35:04.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.945+0000 7f9b5d37a640 1 --2- 172.21.15.81:0/3740522472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b580b9040 0x7f9b580d0480 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.945+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3740522472 >> 172.21.15.81:0/3740522472 conn(0x7f9b5809cf30 msgr2=0x7f9b580d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.945+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3740522472 shutdown_connections 2024-06-29T14:35:04.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.945+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3740522472 wait complete. 2024-06-29T14:35:04.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.947+0000 7f9b5d37a640 1 Processor -- start 2024-06-29T14:35:04.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.947+0000 7f9b5d37a640 1 -- start start 2024-06-29T14:35:04.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b5d37a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b580b5250 0x7f9b5809fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b5d37a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b580b9040 0x7f9b580cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b5d37a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b580d2a90 0x7f9b580b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b5d37a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9b580d0d50 con 0x7f9b580d2a90 2024-06-29T14:35:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b5d37a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9b580d0bd0 con 0x7f9b580b9040 2024-06-29T14:35:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b5d37a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9b580b80b0 con 0x7f9b580b5250 2024-06-29T14:35:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b580d2a90 0x7f9b580b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b580d2a90 0x7f9b580b2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50426/0 (socket says 172.21.15.81:50426) 2024-06-29T14:35:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b577fe640 1 -- 172.21.15.81:0/3363084568 learned_addr learned my addr 172.21.15.81:0/3363084568 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b580b5250 0x7f9b5809fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b580b9040 0x7f9b580cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b577fe640 1 -- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b580b5250 msgr2=0x7f9b5809fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b577fe640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b580b5250 0x7f9b5809fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b577fe640 1 -- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b580b9040 msgr2=0x7f9b580cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b577fe640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b580b9040 0x7f9b580cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b577fe640 1 -- 172.21.15.81:0/3363084568 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b580b38a0 con 0x7f9b580d2a90 2024-06-29T14:35:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b577fe640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b580d2a90 0x7f9b580b2140 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000be90 tx=0x7f9b5000bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b4ffff640 1 -- 172.21.15.81:0/3363084568 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50019070 con 0x7f9b580d2a90 2024-06-29T14:35:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b4ffff640 1 -- 172.21.15.81:0/3363084568 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b50014440 con 0x7f9b580d2a90 2024-06-29T14:35:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b4ffff640 1 -- 172.21.15.81:0/3363084568 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b500148d0 con 0x7f9b580d2a90 2024-06-29T14:35:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3363084568 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b580b1b40 con 0x7f9b580d2a90 2024-06-29T14:35:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.949+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3363084568 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b580b3530 con 0x7f9b580d2a90 2024-06-29T14:35:04.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.951+0000 7f9b4ffff640 1 -- 172.21.15.81:0/3363084568 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f9b5001c070 con 0x7f9b580d2a90 2024-06-29T14:35:04.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.952+0000 7f9b4ffff640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9b380763f0 0x7f9b38078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.952+0000 7f9b4ffff640 1 -- 172.21.15.81:0/3363084568 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f9b50097930 con 0x7f9b580d2a90 2024-06-29T14:35:04.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.952+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9b380763f0 0x7f9b38078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.952+0000 7f9b64233740 1 Processor -- start 2024-06-29T14:35:04.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.952+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9b380763f0 0x7f9b38078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b48034000 tx=0x7f9b48004730 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.953+0000 7f9b64233740 1 -- start start 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b64233740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a9758026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b64233740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 0x55a9757e63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b64233740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757e68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b64233740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a97577ebc0 con 0x55a9753f5590 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b64233740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55a97572d630 con 0x55a9753ed3b0 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b64233740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55a97571cbd0 con 0x55a975416270 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757e68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757e68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50432/0 (socket says 172.21.15.81:50432) 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b577fe640 1 -- 172.21.15.81:0/2111156225 learned_addr learned my addr 172.21.15.81:0/2111156225 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/2111156225 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a9758026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/2111156225 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 0x55a9757e63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b577fe640 1 -- 172.21.15.81:0/2111156225 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 msgr2=0x55a9758026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b577fe640 1 --2- 172.21.15.81:0/2111156225 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a9758026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b577fe640 1 -- 172.21.15.81:0/2111156225 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 msgr2=0x55a9757e63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b577fe640 1 --2- 172.21.15.81:0/2111156225 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 0x55a9757e63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b577fe640 1 -- 172.21.15.81:0/2111156225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a975719ca0 con 0x55a9753f5590 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/2111156225 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a9758026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b577fe640 1 --2- 172.21.15.81:0/2111156225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757e68e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9b5005d040 tx=0x7f9b500606a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44e5bc0f79db30d server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.954+0000 7f9b4cff9640 1 -- 172.21.15.81:0/2111156225 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50069170 con 0x55a9753f5590 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b4cff9640 1 -- 172.21.15.81:0/2111156225 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b50064e00 con 0x55a9753f5590 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b4cff9640 1 -- 172.21.15.81:0/2111156225 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50067330 con 0x55a9753f5590 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b64233740 1 -- 172.21.15.81:0/2111156225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 msgr2=0x55a9757e68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b64233740 1 --2- 172.21.15.81:0/2111156225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757e68e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9b5005d040 tx=0x7f9b500606a0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b64233740 1 -- 172.21.15.81:0/2111156225 shutdown_connections 2024-06-29T14:35:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b64233740 1 --2- 172.21.15.81:0/2111156225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757e68e0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b64233740 1 --2- 172.21.15.81:0/2111156225 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 0x55a9757e63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b64233740 1 --2- 172.21.15.81:0/2111156225 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a9758026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b64233740 1 -- 172.21.15.81:0/2111156225 >> 172.21.15.81:0/2111156225 conn(0x55a97540a3c0 msgr2=0x55a9757b9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b64233740 1 -- 172.21.15.81:0/2111156225 shutdown_connections 2024-06-29T14:35:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.955+0000 7f9b64233740 1 -- 172.21.15.81:0/2111156225 wait complete. 2024-06-29T14:35:04.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.956+0000 7f9b64233740 1 -- start start 2024-06-29T14:35:04.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.956+0000 7f9b64233740 1 Processor -- start 2024-06-29T14:35:04.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b64233740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 0x55a9757d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b64233740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b64233740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a9757cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b64233740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a9756baae0 con 0x55a9753f5590 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b64233740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55a975720c10 con 0x55a9753ed3b0 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b64233740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55a975490620 con 0x55a975416270 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b64233740 10 client.0 _ll_register_callbacks cb 0x55a9756c7480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b4e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757fdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b4e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757fdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50444/0 (socket says 172.21.15.81:50444) 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b4e7fc640 1 -- 172.21.15.81:0/35864170 learned_addr learned my addr 172.21.15.81:0/35864170 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 0x55a9757d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b577fe640 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a9757cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b4e7fc640 1 -- 172.21.15.81:0/35864170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 msgr2=0x55a9757cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a9757cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b4e7fc640 1 -- 172.21.15.81:0/35864170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 msgr2=0x55a9757d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 0x55a9757d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b4e7fc640 1 -- 172.21.15.81:0/35864170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a975481850 con 0x55a9753f5590 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b577fe640 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a9757cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 0x55a9757d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757fdc20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9b40001d30 tx=0x7f9b40010610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622805b55a7ef23f server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b35ffb640 1 -- 172.21.15.81:0/35864170 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b400052d0 con 0x55a9753f5590 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b35ffb640 1 -- 172.21.15.81:0/35864170 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f9b400035f0 con 0x55a9753f5590 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.958+0000 7f9b64233740 1 -- 172.21.15.81:0/35864170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b50064e00 con 0x55a9753f5590 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b35ffb640 1 -- 172.21.15.81:0/35864170 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9b40021050 con 0x55a9753f5590 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b64233740 10 client.6192 fetch_fsmap learned FSMap version 25 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b64233740 1 -- 172.21.15.81:0/35864170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55a9754f86b0 con 0x55a9753f5590 2024-06-29T14:35:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b35ffb640 1 -- 172.21.15.81:0/35864170 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b50064e00 con 0x55a9753f5590 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b35ffb640 1 -- 172.21.15.81:0/35864170 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7f9b40024020 con 0x55a9753f5590 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b64233740 10 client.6192 fetch_fsmap finished waiting for FSMap version 25 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b64233740 10 client.6192 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b64233740 20 client.6192 populate_metadata read hostname 'smithi081' 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b64233740 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.81:6833/2669979711,v1:172.21.15.81:6835/2669979711] conn(0x55a9753d4e40 0x55a9757d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b64233740 4 client.6192 mds_command: new command op to 6033 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b64233740 1 -- 172.21.15.81:0/35864170 --> [v2:172.21.15.81:6833/2669979711,v1:172.21.15.81:6835/2669979711] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a97572f580 con 0x55a9753d4e40 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.81:6833/2669979711,v1:172.21.15.81:6835/2669979711] conn(0x55a9753d4e40 0x55a9757d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.81:6833/2669979711,v1:172.21.15.81:6835/2669979711] conn(0x55a9753d4e40 0x55a9757d0130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f800a50004c49dd server_cookie=313865349617d476 in_seq=0 out_seq=0 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.959+0000 7f9b35ffb640 10 client.6192 ms_handle_connect on v2:172.21.15.81:6833/2669979711 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b35ffb640 1 -- 172.21.15.81:0/35864170 <== mds.0 v2:172.21.15.81:6833/2669979711 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a97572f580 con 0x55a9753d4e40 2024-06-29T14:35:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b35ffb640 10 client.6192 handle_command_reply: tid=0 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 client.6192 shutdown 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 -- 172.21.15.81:0/35864170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 msgr2=0x55a9757fdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757fdc20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9b40001d30 tx=0x7f9b40010610 comp rx=0 tx=0).stop 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 -- 172.21.15.81:0/35864170 shutdown_connections 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a9757cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.81:6833/2669979711,v1:172.21.15.81:6835/2669979711] conn(0x55a9753d4e40 0x55a9757d0130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753f5590 0x55a9757fdc20 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 --2- 172.21.15.81:0/35864170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753ed3b0 0x55a9757d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b35ffb640 0 client.6192 ms_handle_reset on v2:172.21.15.81:6833/2669979711 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 -- 172.21.15.81:0/35864170 >> 172.21.15.81:0/35864170 conn(0x55a97540a3c0 msgr2=0x55a9757e2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 -- 172.21.15.81:0/35864170 shutdown_connections 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 1 -- 172.21.15.81:0/35864170 wait complete. 2024-06-29T14:35:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.960+0000 7f9b64233740 20 client.6192 trim_cache size 0 max 16384 2024-06-29T14:35:04.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.979+0000 7f9b64233740 1 Processor -- start 2024-06-29T14:35:04.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.979+0000 7f9b64233740 1 -- start start 2024-06-29T14:35:04.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.980+0000 7f9b64233740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a9753ed3b0 0x55a9758018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.980+0000 7f9b64233740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 0x55a9757b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.980+0000 7f9b64233740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a975416270 0x55a9757b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.980+0000 7f9b64233740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58200850 con 0x55a975416270 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.980+0000 7f9b64233740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581ffbe0 con 0x55a9753f5590 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.980+0000 7f9b64233740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9b58200200 con 0x55a9753ed3b0 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.980+0000 7f9b577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a975416270 0x55a9757b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a975416270 0x55a9757b71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50450/0 (socket says 172.21.15.81:50450) 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b577fe640 1 -- 172.21.15.81:0/369483632 learned_addr learned my addr 172.21.15.81:0/369483632 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a9753ed3b0 0x55a9758018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 0x55a9757b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b577fe640 1 -- 172.21.15.81:0/369483632 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a9753ed3b0 msgr2=0x55a9758018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:04.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b577fe640 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a9753ed3b0 0x55a9758018f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b577fe640 1 -- 172.21.15.81:0/369483632 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 msgr2=0x55a9757b6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b577fe640 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 0x55a9757b6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b577fe640 1 -- 172.21.15.81:0/369483632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9757e3440 con 0x55a975416270 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 0x55a9757b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a9753ed3b0 0x55a9758018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b577fe640 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a975416270 0x55a9757b71b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000cfd0 tx=0x7f9b5000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47ed14a51c5097e server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b37fff640 1 -- 172.21.15.81:0/369483632 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50069170 con 0x55a975416270 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b37fff640 1 -- 172.21.15.81:0/369483632 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b50064e00 con 0x55a975416270 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.981+0000 7f9b37fff640 1 -- 172.21.15.81:0/369483632 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b50067330 con 0x55a975416270 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.982+0000 7f9b64233740 1 -- 172.21.15.81:0/369483632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a975416270 msgr2=0x55a9757b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.982+0000 7f9b64233740 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a975416270 0x55a9757b71b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000cfd0 tx=0x7f9b5000cf40 comp rx=0 tx=0).stop 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.982+0000 7f9b64233740 1 -- 172.21.15.81:0/369483632 shutdown_connections 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.982+0000 7f9b64233740 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a975416270 0x55a9757b71b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.982+0000 7f9b64233740 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 0x55a9757b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.982+0000 7f9b64233740 1 --2- 172.21.15.81:0/369483632 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a9753ed3b0 0x55a9758018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.982+0000 7f9b64233740 1 -- 172.21.15.81:0/369483632 >> 172.21.15.81:0/369483632 conn(0x55a97540a3c0 msgr2=0x55a9757bd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.982+0000 7f9b64233740 1 -- 172.21.15.81:0/369483632 shutdown_connections 2024-06-29T14:35:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.982+0000 7f9b64233740 1 -- 172.21.15.81:0/369483632 wait complete. 2024-06-29T14:35:04.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.983+0000 7f9b64233740 1 -- start start 2024-06-29T14:35:04.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.983+0000 7f9b64233740 1 Processor -- start 2024-06-29T14:35:04.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b64233740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753ed3b0 0x55a9757fedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b64233740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 0x55a9757ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b64233740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a97587db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b64233740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a9757ce0a0 con 0x55a9753ed3b0 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b64233740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55a9757cefe0 con 0x55a9753f5590 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b64233740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55a9757cfe80 con 0x55a975416270 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b64233740 10 client.0 _ll_register_callbacks cb 0x7f9b40001990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b56ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753ed3b0 0x55a9757fedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b56ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753ed3b0 0x55a9757fedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50452/0 (socket says 172.21.15.81:50452) 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b56ffd640 1 -- 172.21.15.81:0/2774420984 learned_addr learned my addr 172.21.15.81:0/2774420984 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b577fe640 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a97587db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 0x55a9757ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b56ffd640 1 -- 172.21.15.81:0/2774420984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 msgr2=0x55a97587db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a97587db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b56ffd640 1 -- 172.21.15.81:0/2774420984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 msgr2=0x55a9757ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:35:04.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 0x55a9757ccf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b56ffd640 1 -- 172.21.15.81:0/2774420984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a97571a6b0 con 0x55a9753ed3b0 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.984+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 0x55a9757ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b56ffd640 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753ed3b0 0x55a9757fedf0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009830 tx=0x7f9b48005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a433ddc0017a0f1f server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b35ffb640 1 -- 172.21.15.81:0/2774420984 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b48058060 con 0x55a9753ed3b0 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b35ffb640 1 -- 172.21.15.81:0/2774420984 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f9b480523b0 con 0x55a9753ed3b0 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b64233740 1 -- 172.21.15.81:0/2774420984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b50064e00 con 0x55a9753ed3b0 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b35ffb640 1 -- 172.21.15.81:0/2774420984 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9b4805e050 con 0x55a9753ed3b0 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b64233740 10 client.6198 fetch_fsmap learned FSMap version 25 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b64233740 1 -- 172.21.15.81:0/2774420984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55a9757b2b20 con 0x55a9753ed3b0 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b35ffb640 1 -- 172.21.15.81:0/2774420984 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b50064e00 con 0x55a9753ed3b0 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b35ffb640 1 -- 172.21.15.81:0/2774420984 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7f9b48061020 con 0x55a9753ed3b0 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b64233740 10 client.6198 fetch_fsmap finished waiting for FSMap version 25 2024-06-29T14:35:04.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b64233740 10 client.6198 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b64233740 20 client.6198 populate_metadata read hostname 'smithi081' 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b64233740 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.81:6833/2669979711,v1:172.21.15.81:6835/2669979711] conn(0x55a9757e0990 0x55a9757e0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b64233740 4 client.6198 mds_command: new command op to 6033 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b64233740 1 -- 172.21.15.81:0/2774420984 --> [v2:172.21.15.81:6833/2669979711,v1:172.21.15.81:6835/2669979711] -- command(tid 0: {"prefix": "status"}) -- 0x55a97572b020 con 0x55a9757e0990 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.985+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.81:6833/2669979711,v1:172.21.15.81:6835/2669979711] conn(0x55a9757e0990 0x55a9757e0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b35ffb640 10 client.6198 ms_handle_connect on v2:172.21.15.81:6833/2669979711 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b4e7fc640 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.81:6833/2669979711,v1:172.21.15.81:6835/2669979711] conn(0x55a9757e0990 0x55a9757e0da0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81b80996ac202d7 server_cookie=6b3d2956cbd6ecbf in_seq=0 out_seq=0 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b35ffb640 1 -- 172.21.15.81:0/2774420984 <== mds.0 v2:172.21.15.81:6833/2669979711 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55a97572b020 con 0x55a9757e0990 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b35ffb640 10 client.6198 handle_command_reply: tid=0 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 client.6198 shutdown 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 -- 172.21.15.81:0/2774420984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753ed3b0 msgr2=0x55a9757fedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753ed3b0 0x55a9757fedf0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009830 tx=0x7f9b48005f90 comp rx=0 tx=0).stop 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 -- 172.21.15.81:0/2774420984 shutdown_connections 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55a975416270 0x55a97587db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55a9753f5590 0x55a9757ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.81:6833/2669979711,v1:172.21.15.81:6835/2669979711] conn(0x55a9757e0990 0x55a9757e0da0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 --2- 172.21.15.81:0/2774420984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a9753ed3b0 0x55a9757fedf0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b35ffb640 0 client.6198 ms_handle_reset on v2:172.21.15.81:6833/2669979711 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 -- 172.21.15.81:0/2774420984 >> 172.21.15.81:0/2774420984 conn(0x55a97540a3c0 msgr2=0x55a9757e29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 -- 172.21.15.81:0/2774420984 shutdown_connections 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 1 -- 172.21.15.81:0/2774420984 wait complete. 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.986+0000 7f9b64233740 20 client.6198 trim_cache size 0 max 16384 2024-06-29T14:35:04.990 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 6033, 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7429379000000003, 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 25, 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 61, 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 61, 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.422361974999999 2024-06-29T14:35:04.991 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:35:04.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9b380763f0 msgr2=0x7f9b38078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9b380763f0 0x7f9b38078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b48034000 tx=0x7f9b48004730 comp rx=0 tx=0).stop 2024-06-29T14:35:04.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3363084568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b580d2a90 msgr2=0x7f9b580b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:04.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b580d2a90 0x7f9b580b2140 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000be90 tx=0x7f9b5000bf80 comp rx=0 tx=0).stop 2024-06-29T14:35:04.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3363084568 shutdown_connections 2024-06-29T14:35:04.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f9b380763f0 0x7f9b38078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b580d2a90 0x7f9b580b2140 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b580b9040 0x7f9b580cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 --2- 172.21.15.81:0/3363084568 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b580b5250 0x7f9b5809fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:04.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3363084568 >> 172.21.15.81:0/3363084568 conn(0x7f9b5809cf30 msgr2=0x7f9b581613a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:04.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3363084568 shutdown_connections 2024-06-29T14:35:04.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:04.987+0000 7f9b5d37a640 1 -- 172.21.15.81:0/3363084568 wait complete. 2024-06-29T14:35:04.995 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:35:05.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.111+0000 7f6186636640 1 -- 172.21.15.81:0/989373337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f616c004800 msgr2=0x7f616c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:05.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.111+0000 7f6186636640 1 --2- 172.21.15.81:0/989373337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f616c004800 0x7f616c004c10 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f61680016c0 tx=0x7f616c005830 comp rx=0 tx=0).stop 2024-06-29T14:35:05.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.111+0000 7f6186636640 1 -- 172.21.15.81:0/989373337 shutdown_connections 2024-06-29T14:35:05.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.111+0000 7f6186636640 1 --2- 172.21.15.81:0/989373337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f616c004800 0x7f616c004c10 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.111+0000 7f6186636640 1 -- 172.21.15.81:0/989373337 >> 172.21.15.81:0/989373337 conn(0x7f618006ccc0 msgr2=0x7f61800b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:05.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.112+0000 7f6186636640 1 -- 172.21.15.81:0/989373337 shutdown_connections 2024-06-29T14:35:05.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.112+0000 7f6186636640 1 -- 172.21.15.81:0/989373337 wait complete. 2024-06-29T14:35:05.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.113+0000 7f6186636640 1 Processor -- start 2024-06-29T14:35:05.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.113+0000 7f6186636640 1 -- start start 2024-06-29T14:35:05.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f6186636640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f616c004800 0x7f61801443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:05.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f6186636640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6180159670 0x7f61801392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:05.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f6186636640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6180136c10 0x7f6180139810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:05.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f6186636640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f618015f100 con 0x7f616c004800 2024-06-29T14:35:05.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f6186636640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f618015ef80 con 0x7f6180136c10 2024-06-29T14:35:05.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f6186636640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6180073440 con 0x7f6180159670 2024-06-29T14:35:05.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f616c004800 0x7f61801443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:05.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f616c004800 0x7f61801443b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50480/0 (socket says 172.21.15.81:50480) 2024-06-29T14:35:05.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617ffff640 1 -- 172.21.15.81:0/1210857683 learned_addr learned my addr 172.21.15.81:0/1210857683 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:05.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f6184bd3640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6180136c10 0x7f6180139810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:05.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f6177fff640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6180159670 0x7f61801392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:05.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617ffff640 1 -- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6180159670 msgr2=0x7f61801392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:05.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617ffff640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6180159670 0x7f61801392d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617ffff640 1 -- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6180136c10 msgr2=0x7f6180139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:05.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617ffff640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6180136c10 0x7f6180139810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617ffff640 1 -- 172.21.15.81:0/1210857683 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61801448f0 con 0x7f616c004800 2024-06-29T14:35:05.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f6177fff640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6180159670 0x7f61801392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:05.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617ffff640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f616c004800 0x7f61801443b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f617000e5e0 tx=0x7f617000e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:05.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617d7fa640 1 -- 172.21.15.81:0/1210857683 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170015070 con 0x7f616c004800 2024-06-29T14:35:05.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617d7fa640 1 -- 172.21.15.81:0/1210857683 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f61700109a0 con 0x7f616c004800 2024-06-29T14:35:05.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f617d7fa640 1 -- 172.21.15.81:0/1210857683 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170010df0 con 0x7f616c004800 2024-06-29T14:35:05.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.115+0000 7f6186636640 1 -- 172.21.15.81:0/1210857683 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6180137800 con 0x7f616c004800 2024-06-29T14:35:05.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.116+0000 7f6186636640 1 -- 172.21.15.81:0/1210857683 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618013cf90 con 0x7f616c004800 2024-06-29T14:35:05.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.117+0000 7f6186636640 1 -- 172.21.15.81:0/1210857683 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6180073910 con 0x7f616c004800 2024-06-29T14:35:05.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.117+0000 7f617d7fa640 1 -- 172.21.15.81:0/1210857683 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f6170018070 con 0x7f616c004800 2024-06-29T14:35:05.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.118+0000 7f617d7fa640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f61600763f0 0x7f6160078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:05.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.118+0000 7f617d7fa640 1 -- 172.21.15.81:0/1210857683 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f6170097d10 con 0x7f616c004800 2024-06-29T14:35:05.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.118+0000 7f6177fff640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f61600763f0 0x7f6160078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:05.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.119+0000 7f6177fff640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f61600763f0 0x7f6160078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6168032000 tx=0x7f6168001190 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:05.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.122+0000 7f617d7fa640 1 -- 172.21.15.81:0/1210857683 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f61801448f0 con 0x7f616c004800 2024-06-29T14:35:05.215 INFO:tasks.ceph.mds.c.smithi081.stderr:2024-06-29T14:35:05.218+0000 7f708836e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:35:05.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.339+0000 7f6186636640 1 -- 172.21.15.81:0/1210857683 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6180137d20 con 0x7f616c004800 2024-06-29T14:35:05.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.339+0000 7f617d7fa640 1 -- 172.21.15.81:0/1210857683 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f6170061420 con 0x7f616c004800 2024-06-29T14:35:05.338 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:35:05.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 -- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f61600763f0 msgr2=0x7f6160078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:05.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f61600763f0 0x7f6160078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6168032000 tx=0x7f6168001190 comp rx=0 tx=0).stop 2024-06-29T14:35:05.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 -- 172.21.15.81:0/1210857683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f616c004800 msgr2=0x7f61801443b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:05.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f616c004800 0x7f61801443b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f617000e5e0 tx=0x7f617000e9e0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 -- 172.21.15.81:0/1210857683 shutdown_connections 2024-06-29T14:35:05.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6180136c10 0x7f6180139810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6180159670 0x7f61801392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f61600763f0 0x7f6160078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 --2- 172.21.15.81:0/1210857683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f616c004800 0x7f61801443b0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:05.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 -- 172.21.15.81:0/1210857683 >> 172.21.15.81:0/1210857683 conn(0x7f618006ccc0 msgr2=0x7f618006e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:05.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 -- 172.21.15.81:0/1210857683 shutdown_connections 2024-06-29T14:35:05.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05.341+0000 7f6186636640 1 -- 172.21.15.81:0/1210857683 wait complete. 2024-06-29T14:35:05.350 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:35:05.350 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:35:05.350 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T14:35:05.350 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:35:05.350 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:35:05.351 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:35:05.351 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:35:05.351 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:35:05.351 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:35:05.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:05.351 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T14:35:05.366 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T14:35:05.367 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 42014sec preferred_lft 42014sec 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:05.368 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:35:05.369 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:35:05.369 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T14:35:05.369 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:05.369 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:35:05.369 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:35:05.369 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T14:35:05.369 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:35:05.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:05.369 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:35:05.431 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T14:35:05.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:05.433 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T14:35:05.490 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T14:35:05.490 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:35:05.490 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:35:05.490 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:35:05.490 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:35:05.491 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:35:05.491 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:35:05.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:35:05.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:35:05.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:35:05.491 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:35:05.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:35:05.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:35:05.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:05.617 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:35:05.632 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:35:05.632 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:35:05.632 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:35:05.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:05.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T14:35:05.698 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:35:05.698 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:35:05.753 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:35:05.754 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:35:05.754 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:35:05.809 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T14:35:05.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:05.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:35:05.903 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:35:05.903 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:35:05.911 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:35:05.911 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:35:05.911 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:35:05.911 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:35:05.911 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:35:05.912 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:35:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:35:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:35:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:35:05.912 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:35:05.914 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:35:05.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:05.914 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:35:05.939 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:35:06.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:35:06.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:35:06.040 INFO:teuthology.orchestra.run.smithi081.stdout:client6207 2024-06-29T14:35:06.040 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:35:06.040 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:35:06.056 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:35:06.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.212+0000 7f89ffae6640 1 -- 172.21.15.81:0/262943192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89f815e960 msgr2=0x7f89f815ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.212+0000 7f89ffae6640 1 --2- 172.21.15.81:0/262943192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89f815e960 0x7f89f815ed70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f89f0005fd0 tx=0x7f89f002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.212+0000 7f89ffae6640 1 -- 172.21.15.81:0/262943192 shutdown_connections 2024-06-29T14:35:06.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.212+0000 7f89ffae6640 1 --2- 172.21.15.81:0/262943192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89f815e960 0x7f89f815ed70 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.212+0000 7f89ffae6640 1 --2- 172.21.15.81:0/262943192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f89f8073030 0x7f89f8073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.212+0000 7f89ffae6640 1 -- 172.21.15.81:0/262943192 >> 172.21.15.81:0/262943192 conn(0x7f89f806ccc0 msgr2=0x7f89f80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:06.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.212+0000 7f89ffae6640 1 -- 172.21.15.81:0/262943192 shutdown_connections 2024-06-29T14:35:06.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.212+0000 7f89ffae6640 1 -- 172.21.15.81:0/262943192 wait complete. 2024-06-29T14:35:06.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.214+0000 7f89ffae6640 1 Processor -- start 2024-06-29T14:35:06.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.214+0000 7f89ffae6640 1 -- start start 2024-06-29T14:35:06.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.215+0000 7f89ffae6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f89f8073030 0x7f89f80a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.215+0000 7f89ffae6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89f81412f0 0x7f89f8140590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.215+0000 7f89ffae6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89f815e960 0x7f89f80ab790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.215+0000 7f89ffae6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f89f80b3390 con 0x7f89f815e960 2024-06-29T14:35:06.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.215+0000 7f89ffae6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f89f80b3210 con 0x7f89f81412f0 2024-06-29T14:35:06.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.215+0000 7f89ffae6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f89f80b3510 con 0x7f89f8073030 2024-06-29T14:35:06.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fe083640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89f815e960 0x7f89f80ab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fe083640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89f815e960 0x7f89f80ab790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50510/0 (socket says 172.21.15.81:50510) 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fe083640 1 -- 172.21.15.81:0/1094577976 learned_addr learned my addr 172.21.15.81:0/1094577976 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fd882640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f89f8073030 0x7f89f80a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fd081640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89f81412f0 0x7f89f8140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fe083640 1 -- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f89f8073030 msgr2=0x7f89f80a9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fe083640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f89f8073030 0x7f89f80a9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fe083640 1 -- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89f81412f0 msgr2=0x7f89f8140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fe083640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89f81412f0 0x7f89f8140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fe083640 1 -- 172.21.15.81:0/1094577976 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f80aa700 con 0x7f89f815e960 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fd081640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89f81412f0 0x7f89f8140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fd882640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f89f8073030 0x7f89f80a9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89fe083640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89f815e960 0x7f89f80ab790 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009cd0 tx=0x7f89f00316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89ee7fc640 1 -- 172.21.15.81:0/1094577976 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0031990 con 0x7f89f815e960 2024-06-29T14:35:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89ee7fc640 1 -- 172.21.15.81:0/1094577976 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89f0031c20 con 0x7f89f815e960 2024-06-29T14:35:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89ffae6640 1 -- 172.21.15.81:0/1094577976 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f80ad800 con 0x7f89f815e960 2024-06-29T14:35:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89ee7fc640 1 -- 172.21.15.81:0/1094577976 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0033070 con 0x7f89f815e960 2024-06-29T14:35:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.216+0000 7f89ffae6640 1 -- 172.21.15.81:0/1094577976 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f80addc0 con 0x7f89f815e960 2024-06-29T14:35:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.217+0000 7f89ffae6640 1 -- 172.21.15.81:0/1094577976 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89f8068f50 con 0x7f89f815e960 2024-06-29T14:35:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.218+0000 7f89ee7fc640 1 -- 172.21.15.81:0/1094577976 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f89f0033210 con 0x7f89f815e960 2024-06-29T14:35:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.219+0000 7f89ee7fc640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f89dc0763e0 0x7f89dc0788f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.219+0000 7f89ee7fc640 1 -- 172.21.15.81:0/1094577976 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f89f00b8db0 con 0x7f89f815e960 2024-06-29T14:35:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.219+0000 7f89fd882640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f89dc0763e0 0x7f89dc0788f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.220+0000 7f89fd882640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f89dc0763e0 0x7f89dc0788f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f89f40013f0 tx=0x7f89f4009040 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:06.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.223+0000 7f89ee7fc640 1 -- 172.21.15.81:0/1094577976 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f89f8068f50 con 0x7f89f815e960 2024-06-29T14:35:06.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.430+0000 7f89ffae6640 1 -- 172.21.15.81:0/1094577976 --> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f89f8140ad0 con 0x7f89dc0763e0 2024-06-29T14:35:06.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.432+0000 7f89ee7fc640 1 -- 172.21.15.81:0/1094577976 <== mgr.5425 v2:172.21.15.167:6832/1090782941 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f89f8140ad0 con 0x7f89dc0763e0 2024-06-29T14:35:06.430 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:35:06.430 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:35:06.430 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T14:35:06.430 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:35:06.430 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 -- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f89dc0763e0 msgr2=0x7f89dc0788f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f89dc0763e0 0x7f89dc0788f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f89f40013f0 tx=0x7f89f4009040 comp rx=0 tx=0).stop 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 -- 172.21.15.81:0/1094577976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89f815e960 msgr2=0x7f89f80ab790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89f815e960 0x7f89f80ab790 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009cd0 tx=0x7f89f00316d0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 -- 172.21.15.81:0/1094577976 shutdown_connections 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f89dc0763e0 0x7f89dc0788f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89f815e960 0x7f89f80ab790 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f89f81412f0 0x7f89f8140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 --2- 172.21.15.81:0/1094577976 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f89f8073030 0x7f89f80a9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 -- 172.21.15.81:0/1094577976 >> 172.21.15.81:0/1094577976 conn(0x7f89f806ccc0 msgr2=0x7f89f815de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 -- 172.21.15.81:0/1094577976 shutdown_connections 2024-06-29T14:35:06.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.434+0000 7f89ffae6640 1 -- 172.21.15.81:0/1094577976 wait complete. 2024-06-29T14:35:06.443 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:35:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.560+0000 7f1ca65b2640 1 -- 172.21.15.81:0/3047744129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ca00cea50 msgr2=0x7f1ca00b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.560+0000 7f1ca65b2640 1 --2- 172.21.15.81:0/3047744129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ca00cea50 0x7f1ca00b4810 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1ca0073550 tx=0x7f1c9802a980 comp rx=0 tx=0).stop 2024-06-29T14:35:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.560+0000 7f1ca65b2640 1 -- 172.21.15.81:0/3047744129 shutdown_connections 2024-06-29T14:35:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.560+0000 7f1ca65b2640 1 --2- 172.21.15.81:0/3047744129 >> v2:172.21.15.167:3300/0 conn(0x7f1ca00cd480 0x7f1ca00ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.560+0000 7f1ca65b2640 1 --2- 172.21.15.81:0/3047744129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ca00cea50 0x7f1ca00b4810 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.560+0000 7f1ca65b2640 1 -- 172.21.15.81:0/3047744129 >> 172.21.15.81:0/3047744129 conn(0x7f1ca00b3510 msgr2=0x7f1ca00b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.560+0000 7f1ca65b2640 1 -- 172.21.15.81:0/3047744129 shutdown_connections 2024-06-29T14:35:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.560+0000 7f1ca65b2640 1 -- 172.21.15.81:0/3047744129 wait complete. 2024-06-29T14:35:06.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.562+0000 7f1ca65b2640 1 Processor -- start 2024-06-29T14:35:06.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.562+0000 7f1ca65b2640 1 -- start start 2024-06-29T14:35:06.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1ca65b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ca00b6eb0 0x7f1ca01dee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1ca65b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ca00cd480 0x7f1ca00c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1ca65b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ca00cea50 0x7f1ca00c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1ca65b2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca00b4ed0 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1ca65b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca00b4d50 con 0x7f1ca00cd480 2024-06-29T14:35:06.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1ca65b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca00b5050 con 0x7f1ca00cea50 2024-06-29T14:35:06.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1c9ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ca00b6eb0 0x7f1ca01dee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1c9ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ca00b6eb0 0x7f1ca01dee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50540/0 (socket says 172.21.15.81:50540) 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1c9ffff640 1 -- 172.21.15.81:0/4257582615 learned_addr learned my addr 172.21.15.81:0/4257582615 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1ca4b4f640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ca00cea50 0x7f1ca00c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.563+0000 7f1c9f7fe640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ca00cd480 0x7f1ca00c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1c9ffff640 1 -- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ca00cea50 msgr2=0x7f1ca00c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1c9ffff640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ca00cea50 0x7f1ca00c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1c9ffff640 1 -- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ca00cd480 msgr2=0x7f1ca00c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1c9ffff640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ca00cd480 0x7f1ca00c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1c9ffff640 1 -- 172.21.15.81:0/4257582615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca01e7440 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1c9f7fe640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ca00cd480 0x7f1ca00c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1c9ffff640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ca00b6eb0 0x7f1ca01dee20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000b3f0 tx=0x7f1c9000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1c9cff9640 1 -- 172.21.15.81:0/4257582615 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90015020 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1c9cff9640 1 -- 172.21.15.81:0/4257582615 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c90011920 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1ca65b2640 1 -- 172.21.15.81:0/4257582615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca01e5bf0 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1c9cff9640 1 -- 172.21.15.81:0/4257582615 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90011d70 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.564+0000 7f1ca65b2640 1 -- 172.21.15.81:0/4257582615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca01e61b0 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.565+0000 7f1ca65b2640 1 -- 172.21.15.81:0/4257582615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca00ce510 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.566+0000 7f1c9cff9640 1 -- 172.21.15.81:0/4257582615 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 98303+0+0 (secure 0 0 0) 0x7f1c90018020 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.566+0000 7f1c9cff9640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1c8c0763f0 0x7f1c8c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:06.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.566+0000 7f1c9cff9640 1 -- 172.21.15.81:0/4257582615 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f1c90097940 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.567+0000 7f1c9f7fe640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1c8c0763f0 0x7f1c8c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:06.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.567+0000 7f1c9f7fe640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1c8c0763f0 0x7f1c8c078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1ca01e70c0 tx=0x7f1c94005f10 comp rx=0 tx=0).ready entity=mgr.5425 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:06.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.570+0000 7f1c9cff9640 1 -- 172.21.15.81:0/4257582615 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1c9001f030 con 0x7f1ca00b6eb0 2024-06-29T14:35:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:06.797+0000 7f1ca65b2640 1 -- 172.21.15.81:0/4257582615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f1ca00cdf00 con 0x7f1ca00b6eb0 2024-06-29T14:35:07.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.281+0000 7f1c9cff9640 1 -- 172.21.15.81:0/4257582615 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f1c900611b0 con 0x7f1ca00b6eb0 2024-06-29T14:35:07.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.284+0000 7f1ca65b2640 1 -- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1c8c0763f0 msgr2=0x7f1c8c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.284+0000 7f1ca65b2640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1c8c0763f0 0x7f1c8c078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1ca01e70c0 tx=0x7f1c94005f10 comp rx=0 tx=0).stop 2024-06-29T14:35:07.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.284+0000 7f1ca65b2640 1 -- 172.21.15.81:0/4257582615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ca00b6eb0 msgr2=0x7f1ca01dee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.284+0000 7f1ca65b2640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ca00b6eb0 0x7f1ca01dee20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000b3f0 tx=0x7f1c9000b8b0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.285+0000 7f1ca65b2640 1 -- 172.21.15.81:0/4257582615 shutdown_connections 2024-06-29T14:35:07.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.285+0000 7f1ca65b2640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ca00cea50 0x7f1ca00c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.285+0000 7f1ca65b2640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ca00cd480 0x7f1ca00c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.285+0000 7f1ca65b2640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f1c8c0763f0 0x7f1c8c078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.285+0000 7f1ca65b2640 1 --2- 172.21.15.81:0/4257582615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ca00b6eb0 0x7f1ca01dee20 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.285+0000 7f1ca65b2640 1 -- 172.21.15.81:0/4257582615 >> 172.21.15.81:0/4257582615 conn(0x7f1ca00b3510 msgr2=0x7f1ca01e4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.286+0000 7f1ca65b2640 1 -- 172.21.15.81:0/4257582615 shutdown_connections 2024-06-29T14:35:07.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.286+0000 7f1ca65b2640 1 -- 172.21.15.81:0/4257582615 wait complete. 2024-06-29T14:35:07.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:35:07.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.420+0000 7fd76b111640 1 -- 172.21.15.81:0/4024600845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76415bc20 msgr2=0x7fd76406ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.420+0000 7fd76b111640 1 --2- 172.21.15.81:0/4024600845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76415bc20 0x7fd76406ed90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd764074310 tx=0x7fd75c02aa30 comp rx=0 tx=0).stop 2024-06-29T14:35:07.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.420+0000 7fd76b111640 1 -- 172.21.15.81:0/4024600845 shutdown_connections 2024-06-29T14:35:07.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.420+0000 7fd76b111640 1 --2- 172.21.15.81:0/4024600845 >> v2:172.21.15.167:3300/0 conn(0x7fd764073650 0x7fd76415b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.420+0000 7fd76b111640 1 --2- 172.21.15.81:0/4024600845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76415bc20 0x7fd76406ed90 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.420+0000 7fd76b111640 1 -- 172.21.15.81:0/4024600845 >> 172.21.15.81:0/4024600845 conn(0x7fd7640b2920 msgr2=0x7fd7640b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.421+0000 7fd76b111640 1 -- 172.21.15.81:0/4024600845 shutdown_connections 2024-06-29T14:35:07.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.421+0000 7fd76b111640 1 -- 172.21.15.81:0/4024600845 wait complete. 2024-06-29T14:35:07.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.422+0000 7fd76b111640 1 Processor -- start 2024-06-29T14:35:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.422+0000 7fd76b111640 1 -- start start 2024-06-29T14:35:07.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd76b111640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7640cd150 0x7fd764156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd76b111640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76415bc20 0x7fd76413bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd76b111640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd764152c10 0x7fd764153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd76b111640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd76406f450 con 0x7fd76415bc20 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd76b111640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd76406f2d0 con 0x7fd764152c10 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd76b111640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd76406f5d0 con 0x7fd7640cd150 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd763fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76415bc20 0x7fd76413bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd763fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76415bc20 0x7fd76413bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50580/0 (socket says 172.21.15.81:50580) 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd763fff640 1 -- 172.21.15.81:0/221068041 learned_addr learned my addr 172.21.15.81:0/221068041 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd7696ae640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd764152c10 0x7fd764153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd768ead640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7640cd150 0x7fd764156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd763fff640 1 -- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7640cd150 msgr2=0x7fd764156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd763fff640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7640cd150 0x7fd764156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd763fff640 1 -- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd764152c10 msgr2=0x7fd764153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd763fff640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd764152c10 0x7fd764153020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd763fff640 1 -- 172.21.15.81:0/221068041 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76413c980 con 0x7fd76415bc20 2024-06-29T14:35:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.424+0000 7fd763fff640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76415bc20 0x7fd76413bec0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd75000e790 tx=0x7fd75000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.425+0000 7fd7617fa640 1 -- 172.21.15.81:0/221068041 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75000eea0 con 0x7fd76415bc20 2024-06-29T14:35:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.425+0000 7fd7617fa640 1 -- 172.21.15.81:0/221068041 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd75000ccc0 con 0x7fd76415bc20 2024-06-29T14:35:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.425+0000 7fd7617fa640 1 -- 172.21.15.81:0/221068041 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750005280 con 0x7fd76415bc20 2024-06-29T14:35:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.425+0000 7fd76b111640 1 -- 172.21.15.81:0/221068041 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76413cc60 con 0x7fd76415bc20 2024-06-29T14:35:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.425+0000 7fd76b111640 1 -- 172.21.15.81:0/221068041 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd764156970 con 0x7fd76415bc20 2024-06-29T14:35:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.426+0000 7fd76b111640 1 -- 172.21.15.81:0/221068041 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7640b5b80 con 0x7fd76415bc20 2024-06-29T14:35:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.426+0000 7fd7617fa640 1 -- 172.21.15.81:0/221068041 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 19) ==== 98320+0+0 (secure 0 0 0) 0x7fd750005420 con 0x7fd76415bc20 2024-06-29T14:35:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.427+0000 7fd7617fa640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd748076440 0x7fd748078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.427+0000 7fd768ead640 1 -- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd748076440 msgr2=0x7fd748078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1090782941 2024-06-29T14:35:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.427+0000 7fd7617fa640 1 -- 172.21.15.81:0/221068041 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7fd750098260 con 0x7fd76415bc20 2024-06-29T14:35:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.427+0000 7fd768ead640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd748076440 0x7fd748078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:35:07.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.431+0000 7fd7617fa640 1 -- 172.21.15.81:0/221068041 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd7640b5b80 con 0x7fd76415bc20 2024-06-29T14:35:07.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.621+0000 7fd76b111640 1 -- 172.21.15.81:0/221068041 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fd764155a20 con 0x7fd76415bc20 2024-06-29T14:35:07.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.625+0000 7fd7617fa640 1 -- 172.21.15.81:0/221068041 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v33)=0 v33) ==== 138+0+0 (secure 0 0 0) 0x7fd7500612a0 con 0x7fd76415bc20 2024-06-29T14:35:07.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 -- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd748076440 msgr2=0x7fd748078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:35:07.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd748076440 0x7fd748078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 -- 172.21.15.81:0/221068041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76415bc20 msgr2=0x7fd76413bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76415bc20 0x7fd76413bec0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd75000e790 tx=0x7fd75000ec50 comp rx=0 tx=0).stop 2024-06-29T14:35:07.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 -- 172.21.15.81:0/221068041 shutdown_connections 2024-06-29T14:35:07.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd764152c10 0x7fd764153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd748076440 0x7fd748078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76415bc20 0x7fd76413bec0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 --2- 172.21.15.81:0/221068041 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7640cd150 0x7fd764156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 -- 172.21.15.81:0/221068041 >> 172.21.15.81:0/221068041 conn(0x7fd7640b2920 msgr2=0x7fd76415f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 -- 172.21.15.81:0/221068041 shutdown_connections 2024-06-29T14:35:07.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.627+0000 7fd76b111640 1 -- 172.21.15.81:0/221068041 wait complete. 2024-06-29T14:35:07.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:35:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.758+0000 7fd776f7a640 1 -- 172.21.15.81:0/491585736 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7700b5f50 msgr2=0x7fd770166870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.758+0000 7fd776f7a640 1 --2- 172.21.15.81:0/491585736 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7700b5f50 0x7fd770166870 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd768005fd0 tx=0x7fd76802a840 comp rx=0 tx=0).stop 2024-06-29T14:35:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.758+0000 7fd776f7a640 1 -- 172.21.15.81:0/491585736 shutdown_connections 2024-06-29T14:35:07.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.758+0000 7fd776f7a640 1 --2- 172.21.15.81:0/491585736 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7700b5f50 0x7fd770166870 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.758+0000 7fd776f7a640 1 -- 172.21.15.81:0/491585736 >> 172.21.15.81:0/491585736 conn(0x7fd7700b3680 msgr2=0x7fd7700b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.758+0000 7fd776f7a640 1 -- 172.21.15.81:0/491585736 shutdown_connections 2024-06-29T14:35:07.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.758+0000 7fd776f7a640 1 -- 172.21.15.81:0/491585736 wait complete. 2024-06-29T14:35:07.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.760+0000 7fd776f7a640 1 Processor -- start 2024-06-29T14:35:07.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.760+0000 7fd776f7a640 1 -- start start 2024-06-29T14:35:07.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd776f7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7700b3200 0x7fd7700c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd776f7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7700b5f50 0x7fd7701514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd776f7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7700d0230 0x7fd77014d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd776f7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd7701672d0 con 0x7fd7700b5f50 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd776f7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd770167150 con 0x7fd7700d0230 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd776f7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd770167450 con 0x7fd7700b3200 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd76ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7700b5f50 0x7fd7701514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd76ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7700b5f50 0x7fd7701514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50590/0 (socket says 172.21.15.81:50590) 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd76ffff640 1 -- 172.21.15.81:0/1939458225 learned_addr learned my addr 172.21.15.81:0/1939458225 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd774d16640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7700b3200 0x7fd7700c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd775517640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7700d0230 0x7fd77014d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd76ffff640 1 -- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7700b3200 msgr2=0x7fd7700c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd76ffff640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7700b3200 0x7fd7700c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd76ffff640 1 -- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7700d0230 msgr2=0x7fd77014d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd76ffff640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7700d0230 0x7fd77014d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd76ffff640 1 -- 172.21.15.81:0/1939458225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd770152110 con 0x7fd7700b5f50 2024-06-29T14:35:07.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.762+0000 7fd76ffff640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7700b5f50 0x7fd7701514f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00c430 tx=0x7fd75c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:07.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.763+0000 7fd76d7fa640 1 -- 172.21.15.81:0/1939458225 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c017070 con 0x7fd7700b5f50 2024-06-29T14:35:07.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.763+0000 7fd776f7a640 1 -- 172.21.15.81:0/1939458225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7700c8950 con 0x7fd7700b5f50 2024-06-29T14:35:07.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.763+0000 7fd76d7fa640 1 -- 172.21.15.81:0/1939458225 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd75c0053c0 con 0x7fd7700b5f50 2024-06-29T14:35:07.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.763+0000 7fd76d7fa640 1 -- 172.21.15.81:0/1939458225 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c003850 con 0x7fd7700b5f50 2024-06-29T14:35:07.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.763+0000 7fd776f7a640 1 -- 172.21.15.81:0/1939458225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77006eb80 con 0x7fd7700b5f50 2024-06-29T14:35:07.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.763+0000 7fd776f7a640 1 -- 172.21.15.81:0/1939458225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd77006e610 con 0x7fd7700b5f50 2024-06-29T14:35:07.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.764+0000 7fd76d7fa640 1 -- 172.21.15.81:0/1939458225 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 19) ==== 98320+0+0 (secure 0 0 0) 0x7fd75c01a070 con 0x7fd7700b5f50 2024-06-29T14:35:07.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.765+0000 7fd76d7fa640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd754076440 0x7fd754078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:07.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.765+0000 7fd774d16640 1 -- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd754076440 msgr2=0x7fd754078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1090782941 2024-06-29T14:35:07.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.765+0000 7fd76d7fa640 1 -- 172.21.15.81:0/1939458225 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7fd75c097260 con 0x7fd7700b5f50 2024-06-29T14:35:07.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.765+0000 7fd774d16640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd754076440 0x7fd754078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:35:07.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.769+0000 7fd76d7fa640 1 -- 172.21.15.81:0/1939458225 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd75c021020 con 0x7fd7700b5f50 2024-06-29T14:35:07.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.956+0000 7fd776f7a640 1 -- 172.21.15.81:0/1939458225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fd7700cc7b0 con 0x7fd7700b5f50 2024-06-29T14:35:07.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.960+0000 7fd76d7fa640 1 -- 172.21.15.81:0/1939458225 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v34)=0 v34) ==== 133+0+0 (secure 0 0 0) 0x7fd75c0603a0 con 0x7fd7700b5f50 2024-06-29T14:35:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 -- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd754076440 msgr2=0x7fd754078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:35:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd754076440 0x7fd754078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 -- 172.21.15.81:0/1939458225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7700b5f50 msgr2=0x7fd7701514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7700b5f50 0x7fd7701514f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00c430 tx=0x7fd75c00c8f0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 -- 172.21.15.81:0/1939458225 shutdown_connections 2024-06-29T14:35:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd7700d0230 0x7fd77014d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7fd754076440 0x7fd754078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7700b5f50 0x7fd7701514f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 --2- 172.21.15.81:0/1939458225 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7700b3200 0x7fd7700c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 -- 172.21.15.81:0/1939458225 >> 172.21.15.81:0/1939458225 conn(0x7fd7700b3680 msgr2=0x7fd7700cf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 -- 172.21.15.81:0/1939458225 shutdown_connections 2024-06-29T14:35:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:07.962+0000 7fd776f7a640 1 -- 172.21.15.81:0/1939458225 wait complete. 2024-06-29T14:35:07.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:07.971 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T14:35:07.991 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-06-29T14:35:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.152+0000 7f80cbb7a640 1 -- 172.21.15.81:0/3530727405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80b0003c10 msgr2=0x7f80b0004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.152+0000 7f80cbb7a640 1 --2- 172.21.15.81:0/3530727405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80b0003c10 0x7f80b0004060 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f80c0001830 tx=0x7f80b0005830 comp rx=0 tx=0).stop 2024-06-29T14:35:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.152+0000 7f80ca117640 1 -- 172.21.15.81:0/3530727405 reap_dead start 2024-06-29T14:35:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.152+0000 7f80cbb7a640 1 -- 172.21.15.81:0/3530727405 shutdown_connections 2024-06-29T14:35:08.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.152+0000 7f80cbb7a640 1 -- 172.21.15.81:0/3530727405 >> 172.21.15.81:0/3530727405 conn(0x7f80c40b2920 msgr2=0x7f80c40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:08.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.153+0000 7f80cbb7a640 1 -- 172.21.15.81:0/3530727405 shutdown_connections 2024-06-29T14:35:08.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.153+0000 7f80cbb7a640 1 -- 172.21.15.81:0/3530727405 wait complete. 2024-06-29T14:35:08.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.155+0000 7f80cbb7a640 1 Processor -- start 2024-06-29T14:35:08.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.156+0000 7f80cbb7a640 1 -- start start 2024-06-29T14:35:08.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80cbb7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80b0003c10 0x7f80c4145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:08.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80cbb7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80c4158720 0x7f80c414bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:08.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80cbb7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c414c2a0 0x7f80c41f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80cbb7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f80c406c470 con 0x7f80c4158720 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80cbb7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f80c406c2f0 con 0x7f80b0003c10 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80cbb7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f80c406c5f0 con 0x7f80c414c2a0 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80c9115640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80c4158720 0x7f80c414bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80c9115640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80c4158720 0x7f80c414bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50608/0 (socket says 172.21.15.81:50608) 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80c9115640 1 -- 172.21.15.81:0/747696724 learned_addr learned my addr 172.21.15.81:0/747696724 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80ca117640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c414c2a0 0x7f80c41f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80c9916640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80b0003c10 0x7f80c4145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80c9115640 1 -- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c414c2a0 msgr2=0x7f80c41f3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80c9115640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80c414c2a0 0x7f80c41f3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80c9115640 1 -- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80b0003c10 msgr2=0x7f80c4145da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80c9115640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80b0003c10 0x7f80c4145da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:08.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80c9115640 1 -- 172.21.15.81:0/747696724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c414c6b0 con 0x7f80c4158720 2024-06-29T14:35:08.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80c9115640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80c4158720 0x7f80c414bd60 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f80b400e5c0 tx=0x7f80b400ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:08.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b4015070 con 0x7f80c4158720 2024-06-29T14:35:08.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80b4010980 con 0x7f80c4158720 2024-06-29T14:35:08.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.158+0000 7f80cbb7a640 1 -- 172.21.15.81:0/747696724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c414ad40 con 0x7f80c4158720 2024-06-29T14:35:08.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.159+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b4010dd0 con 0x7f80c4158720 2024-06-29T14:35:08.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.159+0000 7f80cbb7a640 1 -- 172.21.15.81:0/747696724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c414b020 con 0x7f80c4158720 2024-06-29T14:35:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.160+0000 7f80cbb7a640 1 -- 172.21.15.81:0/747696724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c415f5f0 con 0x7f80c4158720 2024-06-29T14:35:08.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.160+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 19) ==== 98320+0+0 (secure 0 0 0) 0x7f80b4018070 con 0x7f80c4158720 2024-06-29T14:35:08.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.161+0000 7f80ba7fc640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 0x7f80a8078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:08.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.161+0000 7f80c9916640 1 -- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 msgr2=0x7f80a8078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1090782941 2024-06-29T14:35:08.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.161+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5753+0+0 (secure 0 0 0) 0x7f80b4097ca0 con 0x7f80c4158720 2024-06-29T14:35:08.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.161+0000 7f80c9916640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 0x7f80a8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:35:08.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.165+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f80c414c6b0 con 0x7f80c4158720 2024-06-29T14:35:08.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.362+0000 7f80c9916640 1 -- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 msgr2=0x7f80a8078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1090782941 2024-06-29T14:35:08.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.362+0000 7f80c9916640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 0x7f80a8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:35:08.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.376+0000 7f80cbb7a640 1 -- 172.21.15.81:0/747696724 --> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f80c415f310 con 0x7f80a8076470 2024-06-29T14:35:08.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.763+0000 7f80c9916640 1 -- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 msgr2=0x7f80a8078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1090782941 2024-06-29T14:35:08.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:08.763+0000 7f80c9916640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 0x7f80a8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:35:09.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:09.564+0000 7f80c9916640 1 -- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 msgr2=0x7f80a8078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1090782941 2024-06-29T14:35:09.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:09.564+0000 7f80c9916640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 0x7f80a8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:35:10.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:35:10.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:35:10.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:10.914+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 20) ==== 97929+0+0 (secure 0 0 0) 0x7f80b4096190 con 0x7f80c4158720 2024-06-29T14:35:10.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:10.914+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 msgr2=0x7f80a8078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:35:10.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:10.914+0000 7f80ba7fc640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/1090782941,v1:172.21.15.167:6833/1090782941] conn(0x7f80a8076470 0x7f80a8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:11.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:35:11.061+0000 7faf86ccc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:11.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:35:11.061+0000 7faf86ccc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:11.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:35:11.061+0000 7faf86ccc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:11.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:35:11.061+0000 7faf86ccc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:11.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:35:11.061+0000 7faf86ccc640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:11.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:35:11.061+0000 7faf83cc6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:11.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:35:11.061+0000 7faf83cc6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:11.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:35:11.061+0000 7faf83cc6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:11.060 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:35:11.061+0000 7faf83cc6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:11.060 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:35:11.061+0000 7faf83cc6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:35:11.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:11.919+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 21) ==== 98056+0+0 (secure 0 0 0) 0x7f80b4005d30 con 0x7f80c4158720 2024-06-29T14:35:11.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:11.920+0000 7f80ba7fc640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f80a803ad60 0x7f80a8080400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:11.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:11.920+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 --> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f80b4066090 con 0x7f80a803ad60 2024-06-29T14:35:11.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:11.924+0000 7f80c9916640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f80a803ad60 0x7f80a8080400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:11.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:11.928+0000 7f80c9916640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f80a803ad60 0x7f80a8080400 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f80c0032000 tx=0x7f80c00012f0 comp rx=0 tx=0).ready entity=mgr.5752 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:11.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.000+0000 7f80ba7fc640 1 -- 172.21.15.81:0/747696724 <== mgr.5752 v2:172.21.15.167:6832/2493228728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f80b4066090 con 0x7f80a803ad60 2024-06-29T14:35:11.998 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1 2024-06-29T14:35:12.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.002+0000 7f80cbb7a640 1 -- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f80a803ad60 msgr2=0x7f80a8080400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.002+0000 7f80cbb7a640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f80a803ad60 0x7f80a8080400 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f80c0032000 tx=0x7f80c00012f0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.002+0000 7f80cbb7a640 1 -- 172.21.15.81:0/747696724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80c4158720 msgr2=0x7f80c414bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.002+0000 7f80cbb7a640 1 --2- 172.21.15.81:0/747696724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80c4158720 0x7f80c414bd60 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f80b400e5c0 tx=0x7f80b400ea80 comp rx=0 tx=0).stop 2024-06-29T14:35:12.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.002+0000 7f80ca117640 1 -- 172.21.15.81:0/747696724 reap_dead start 2024-06-29T14:35:12.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.002+0000 7f80cbb7a640 1 -- 172.21.15.81:0/747696724 shutdown_connections 2024-06-29T14:35:12.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.002+0000 7f80cbb7a640 1 -- 172.21.15.81:0/747696724 >> 172.21.15.81:0/747696724 conn(0x7f80c40b2920 msgr2=0x7f80c4153990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:12.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.003+0000 7f80cbb7a640 1 -- 172.21.15.81:0/747696724 shutdown_connections 2024-06-29T14:35:12.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.003+0000 7f80cbb7a640 1 -- 172.21.15.81:0/747696724 wait complete. 2024-06-29T14:35:12.012 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T14:35:12.012 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2024-06-29T14:35:12.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.132+0000 7f0d9601f640 1 -- 172.21.15.81:0/2422973166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d7c005300 msgr2=0x7f0d7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.132+0000 7f0d9601f640 1 --2- 172.21.15.81:0/2422973166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d7c005300 0x7f0d7c00b9c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0d88002230 tx=0x7f0d7c005830 comp rx=0 tx=0).stop 2024-06-29T14:35:12.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.132+0000 7f0d9601f640 1 -- 172.21.15.81:0/2422973166 shutdown_connections 2024-06-29T14:35:12.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.132+0000 7f0d9601f640 1 --2- 172.21.15.81:0/2422973166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d7c005300 0x7f0d7c00b9c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.132+0000 7f0d9601f640 1 -- 172.21.15.81:0/2422973166 >> 172.21.15.81:0/2422973166 conn(0x7f0d9009cf30 msgr2=0x7f0d900d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:12.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.132+0000 7f0d9601f640 1 -- 172.21.15.81:0/2422973166 shutdown_connections 2024-06-29T14:35:12.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.132+0000 7f0d9601f640 1 -- 172.21.15.81:0/2422973166 wait complete. 2024-06-29T14:35:12.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.134+0000 7f0d9601f640 1 Processor -- start 2024-06-29T14:35:12.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.134+0000 7f0d9601f640 1 -- start start 2024-06-29T14:35:12.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.135+0000 7f0d9601f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d7c005300 0x7f0d901f7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.135+0000 7f0d9601f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d9009d9a0 0x7f0d9009f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.135+0000 7f0d9601f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d9015a240 0x7f0d9009f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.135+0000 7f0d9601f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0d900d0810 con 0x7f0d7c005300 2024-06-29T14:35:12.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.135+0000 7f0d9601f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0d900d0690 con 0x7f0d9015a240 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.135+0000 7f0d9601f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0d900d0990 con 0x7f0d9009d9a0 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.135+0000 7f0d8f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d7c005300 0x7f0d901f7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.135+0000 7f0d8f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d7c005300 0x7f0d901f7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42130/0 (socket says 172.21.15.81:42130) 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.135+0000 7f0d8f7fe640 1 -- 172.21.15.81:0/401311487 learned_addr learned my addr 172.21.15.81:0/401311487 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d86ffd640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d9009d9a0 0x7f0d9009f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8ffff640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d9015a240 0x7f0d9009f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8f7fe640 1 -- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d9009d9a0 msgr2=0x7f0d9009f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8f7fe640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d9009d9a0 0x7f0d9009f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8f7fe640 1 -- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d9015a240 msgr2=0x7f0d9009f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8f7fe640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d9015a240 0x7f0d9009f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8f7fe640 1 -- 172.21.15.81:0/401311487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9009e180 con 0x7f0d7c005300 2024-06-29T14:35:12.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8ffff640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d9015a240 0x7f0d9009f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:35:12.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8f7fe640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d7c005300 0x7f0d901f7b90 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0d80010a70 tx=0x7f0d80010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:12.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8cff9640 1 -- 172.21.15.81:0/401311487 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8000ead0 con 0x7f0d7c005300 2024-06-29T14:35:12.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8cff9640 1 -- 172.21.15.81:0/401311487 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d8000ec70 con 0x7f0d7c005300 2024-06-29T14:35:12.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d9601f640 1 -- 172.21.15.81:0/401311487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9009e460 con 0x7f0d7c005300 2024-06-29T14:35:12.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d8cff9640 1 -- 172.21.15.81:0/401311487 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d80011440 con 0x7f0d7c005300 2024-06-29T14:35:12.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.136+0000 7f0d9601f640 1 -- 172.21.15.81:0/401311487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9009ea20 con 0x7f0d7c005300 2024-06-29T14:35:12.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.137+0000 7f0d9601f640 1 -- 172.21.15.81:0/401311487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d900d1a80 con 0x7f0d7c005300 2024-06-29T14:35:12.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.138+0000 7f0d8cff9640 1 -- 172.21.15.81:0/401311487 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 21) ==== 98056+0+0 (secure 0 0 0) 0x7f0d800115e0 con 0x7f0d7c005300 2024-06-29T14:35:12.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.138+0000 7f0d8cff9640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0d70076170 0x7f0d70078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.139+0000 7f0d8cff9640 1 -- 172.21.15.81:0/401311487 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6097+0+0 (secure 0 0 0) 0x7f0d80098440 con 0x7f0d7c005300 2024-06-29T14:35:12.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.139+0000 7f0d86ffd640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0d70076170 0x7f0d70078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.139+0000 7f0d86ffd640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0d70076170 0x7f0d70078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0d9015aa00 tx=0x7f0d78006b70 comp rx=0 tx=0).ready entity=mgr.5752 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:12.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.143+0000 7f0d8cff9640 1 -- 172.21.15.81:0/401311487 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0d80061f50 con 0x7f0d7c005300 2024-06-29T14:35:12.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.344+0000 7f0d9601f640 1 -- 172.21.15.81:0/401311487 --> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) -- 0x7f0d9006a5d0 con 0x7f0d70076170 2024-06-29T14:35:12.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.405+0000 7f0d8cff9640 1 -- 172.21.15.81:0/401311487 <== mgr.5752 v2:172.21.15.167:6832/2493228728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0d9006a5d0 con 0x7f0d70076170 2024-06-29T14:35:12.403 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1 2024-06-29T14:35:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 -- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0d70076170 msgr2=0x7f0d70078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0d70076170 0x7f0d70078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0d9015aa00 tx=0x7f0d78006b70 comp rx=0 tx=0).stop 2024-06-29T14:35:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 -- 172.21.15.81:0/401311487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d7c005300 msgr2=0x7f0d901f7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d7c005300 0x7f0d901f7b90 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0d80010a70 tx=0x7f0d80010e70 comp rx=0 tx=0).stop 2024-06-29T14:35:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 -- 172.21.15.81:0/401311487 shutdown_connections 2024-06-29T14:35:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0d70076170 0x7f0d70078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d9015a240 0x7f0d9009f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d9009d9a0 0x7f0d9009f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 --2- 172.21.15.81:0/401311487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d7c005300 0x7f0d901f7b90 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 -- 172.21.15.81:0/401311487 >> 172.21.15.81:0/401311487 conn(0x7f0d9009cf30 msgr2=0x7f0d90161300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:12.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 -- 172.21.15.81:0/401311487 shutdown_connections 2024-06-29T14:35:12.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.407+0000 7f0d9601f640 1 -- 172.21.15.81:0/401311487 wait complete. 2024-06-29T14:35:12.416 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T14:35:12.416 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-06-29-14_36_00 in ~48s... 2024-06-29T14:35:12.416 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-06-29-14_37_00 in ~108s... 2024-06-29T14:35:12.416 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2024-06-29T14:35:12.416 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-06-29T14:35:12.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.533+0000 7f9480ef6640 1 -- 172.21.15.81:0/3071928222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947c073030 msgr2=0x7f947c073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.533+0000 7f9480ef6640 1 --2- 172.21.15.81:0/3071928222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947c073030 0x7f947c073440 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9464009830 tx=0x7f946402a540 comp rx=0 tx=0).stop 2024-06-29T14:35:12.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.533+0000 7f9480ef6640 1 -- 172.21.15.81:0/3071928222 shutdown_connections 2024-06-29T14:35:12.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.533+0000 7f9480ef6640 1 --2- 172.21.15.81:0/3071928222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947c073030 0x7f947c073440 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.533+0000 7f9480ef6640 1 -- 172.21.15.81:0/3071928222 >> 172.21.15.81:0/3071928222 conn(0x7f947c06ccc0 msgr2=0x7f947c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:12.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.533+0000 7f9480ef6640 1 -- 172.21.15.81:0/3071928222 shutdown_connections 2024-06-29T14:35:12.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.534+0000 7f9480ef6640 1 -- 172.21.15.81:0/3071928222 wait complete. 2024-06-29T14:35:12.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.536+0000 7f9480ef6640 1 Processor -- start 2024-06-29T14:35:12.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.536+0000 7f9480ef6640 1 -- start start 2024-06-29T14:35:12.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f9480ef6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f947c073030 0x7f947c157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f9480ef6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947c1412f0 0x7f947c151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f9480ef6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f947c15e960 0x7f947c155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f9480ef6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f947c0b35a0 con 0x7f947c15e960 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f9480ef6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f947c0b3420 con 0x7f947c1412f0 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f9480ef6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f947c0b3720 con 0x7f947c073030 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f947ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f947c15e960 0x7f947c155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f947ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f947c15e960 0x7f947c155e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42134/0 (socket says 172.21.15.81:42134) 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f947ad9d640 1 -- 172.21.15.81:0/2364978602 learned_addr learned my addr 172.21.15.81:0/2364978602 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f9471d9b640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947c1412f0 0x7f947c151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f947a59c640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f947c073030 0x7f947c157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f947ad9d640 1 -- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f947c073030 msgr2=0x7f947c157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f947ad9d640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f947c073030 0x7f947c157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f947ad9d640 1 -- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947c1412f0 msgr2=0x7f947c151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f947ad9d640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947c1412f0 0x7f947c151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.537+0000 7f947ad9d640 1 -- 172.21.15.81:0/2364978602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c156ad0 con 0x7f947c15e960 2024-06-29T14:35:12.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.538+0000 7f947ad9d640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f947c15e960 0x7f947c155e80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f947400c700 tx=0x7f947400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:12.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.538+0000 7f94737fe640 1 -- 172.21.15.81:0/2364978602 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474003b70 con 0x7f947c15e960 2024-06-29T14:35:12.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.538+0000 7f94737fe640 1 -- 172.21.15.81:0/2364978602 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9474003d10 con 0x7f947c15e960 2024-06-29T14:35:12.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.538+0000 7f9480ef6640 1 -- 172.21.15.81:0/2364978602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947c156db0 con 0x7f947c15e960 2024-06-29T14:35:12.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.538+0000 7f94737fe640 1 -- 172.21.15.81:0/2364978602 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947400aa50 con 0x7f947c15e960 2024-06-29T14:35:12.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.538+0000 7f9480ef6640 1 -- 172.21.15.81:0/2364978602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947c157370 con 0x7f947c15e960 2024-06-29T14:35:12.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.539+0000 7f9480ef6640 1 -- 172.21.15.81:0/2364978602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f947c073440 con 0x7f947c15e960 2024-06-29T14:35:12.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.540+0000 7f94737fe640 1 -- 172.21.15.81:0/2364978602 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 21) ==== 98056+0+0 (secure 0 0 0) 0x7f947400abf0 con 0x7f947c15e960 2024-06-29T14:35:12.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.540+0000 7f94737fe640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f945c0761a0 0x7f945c0786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:35:12.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.540+0000 7f94737fe640 1 -- 172.21.15.81:0/2364978602 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6097+0+0 (secure 0 0 0) 0x7f94740972b0 con 0x7f947c15e960 2024-06-29T14:35:12.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.540+0000 7f947a59c640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f945c0761a0 0x7f945c0786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:35:12.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.541+0000 7f947a59c640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f945c0761a0 0x7f945c0786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f946c001590 tx=0x7f946c001830 comp rx=0 tx=0).ready entity=mgr.5752 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:35:12.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.544+0000 7f94737fe640 1 -- 172.21.15.81:0/2364978602 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f94740606e0 con 0x7f947c15e960 2024-06-29T14:35:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.745+0000 7f9480ef6640 1 -- 172.21.15.81:0/2364978602 --> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f947c15e170 con 0x7f945c0761a0 2024-06-29T14:35:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.752+0000 7f94737fe640 1 -- 172.21.15.81:0/2364978602 <== mgr.5752 v2:172.21.15.167:6832/2493228728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+76 (secure 0 0 0) 0x7f947c15e170 con 0x7f945c0761a0 2024-06-29T14:35:12.750 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:35:12.750 INFO:teuthology.orchestra.run.smithi081.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-06-29T14:35:12.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 -- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f945c0761a0 msgr2=0x7f945c0786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f945c0761a0 0x7f945c0786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f946c001590 tx=0x7f946c001830 comp rx=0 tx=0).stop 2024-06-29T14:35:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 -- 172.21.15.81:0/2364978602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f947c15e960 msgr2=0x7f947c155e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:35:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f947c15e960 0x7f947c155e80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f947400c700 tx=0x7f947400cbc0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 -- 172.21.15.81:0/2364978602 shutdown_connections 2024-06-29T14:35:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f945c0761a0 0x7f945c0786b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f947c15e960 0x7f947c155e80 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f947c1412f0 0x7f947c151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 --2- 172.21.15.81:0/2364978602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f947c073030 0x7f947c157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:35:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 -- 172.21.15.81:0/2364978602 >> 172.21.15.81:0/2364978602 conn(0x7f947c06ccc0 msgr2=0x7f947c06e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:35:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 -- 172.21.15.81:0/2364978602 shutdown_connections 2024-06-29T14:35:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:35:12.755+0000 7f9480ef6640 1 -- 172.21.15.81:0/2364978602 wait complete. 2024-06-29T14:35:12.764 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2024-06-29T14:35:12.764 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2024-06-29T14:35:12.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:12.764 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:12.780 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:12.780 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:12.780 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:13.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:13.782 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:13.801 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:13.801 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:13.801 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:14.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:14.802 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:14.823 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:14.823 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:14.823 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:15.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:15.824 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:15.845 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:15.845 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:15.845 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:16.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:16.847 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:16.867 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:16.867 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:16.867 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:17.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:17.868 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:17.888 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:17.888 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:17.888 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:18.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:18.890 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:18.910 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:18.910 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:18.911 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:19.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:19.912 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:19.931 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:19.931 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:19.932 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:20.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:20.933 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:20.953 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:20.954 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:20.954 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:21.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:21.955 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:21.975 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:21.975 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:21.975 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:22.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:22.976 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:22.996 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:22.997 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:22.997 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:23.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:23.998 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:24.018 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:24.018 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:24.018 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:25.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:25.019 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:25.039 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:25.039 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:25.040 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:26.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:26.040 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:26.060 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:26.061 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:26.061 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:27.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:27.061 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:27.081 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:27.081 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:27.081 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:28.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:28.083 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:28.102 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:28.103 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:28.103 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:29.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:29.103 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:29.123 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:29.124 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:29.124 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:30.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:30.124 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:30.145 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:30.145 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:30.145 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:31.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:31.146 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:31.166 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:31.166 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:31.166 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:32.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:32.167 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:32.187 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:32.187 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:32.187 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:33.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:33.188 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:33.208 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:33.208 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:33.208 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:34.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:34.209 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:34.230 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:34.230 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:34.230 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:35.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:35.231 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:35.251 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:35.251 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:35.251 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:36.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:36.252 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:36.273 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:36.273 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:36.273 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:37.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:37.274 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:37.294 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:37.294 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:37.294 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:38.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:38.295 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:38.314 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:38.314 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:38.315 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:39.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:39.316 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:39.336 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:39.336 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:39.336 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:40.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:40.338 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:40.358 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:40.358 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:40.358 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:40.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:35:40.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:35:41.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:41.359 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:41.379 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:41.379 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:41.379 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:42.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:42.380 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:42.399 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:42.400 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:42.400 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:43.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:43.401 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:43.422 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:43.422 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:43.422 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:44.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:44.423 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:44.443 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:44.444 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:44.444 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:45.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:45.444 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:45.465 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:45.465 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:45.465 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:46.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:46.466 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:46.487 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:46.487 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:46.487 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:47.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:47.488 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:47.508 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:47.508 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:47.508 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:48.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:48.509 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:48.530 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:48.530 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:48.530 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:49.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:49.533 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:49.554 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:49.554 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:49.554 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:50.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:50.555 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:50.575 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:50.575 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:50.575 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:51.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:51.576 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:51.596 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:51.596 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:51.596 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:52.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:52.598 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:52.618 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:52.618 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:52.618 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:53.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:53.620 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:53.640 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:53.640 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:53.640 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:54.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:54.642 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:54.662 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:54.663 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:54.663 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:55.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:55.664 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:55.684 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:55.684 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:55.684 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:56.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:56.685 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:56.705 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:56.705 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:56.706 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:57.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:57.707 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:57.727 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:57.727 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:57.727 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:58.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:58.728 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:58.749 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:58.749 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:58.749 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:35:59.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:35:59.750 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:35:59.770 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:35:59.770 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:35:59.770 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:00.414 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:36:00.416+0000 7faf75caa640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:36:00.414 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:36:00.416+0000 7faf75caa640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:36:00.414 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:36:00.416+0000 7faf75caa640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:36:00.414 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:36:00.416+0000 7faf75caa640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:36:00.415 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:36:00.416+0000 7faf75caa640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:36:00.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:00.771 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:00.791 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:00.791 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:00.791 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:00.791 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:00.791 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-06-29-14_36_00_UTC'] 2024-06-29T14:36:00.792 DEBUG:tasks.cephfs.test_snap_schedules:exec=1719671712.416468, now = 1719671760.7920058, timo = 48 2024-06-29T14:36:01.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:01.792 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:01.812 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:01.813 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:01.813 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:01.813 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:02.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:02.814 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:02.835 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:02.835 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:02.835 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:02.835 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:03.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:03.836 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:03.856 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:03.857 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:03.857 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:03.857 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:04.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:04.858 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:04.878 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:04.878 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:04.878 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:04.878 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:05.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:05.879 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:05.899 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:05.900 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:05.900 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:05.900 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:06.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:06.901 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:06.921 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:06.922 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:06.922 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:06.922 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:07.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:07.923 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:07.943 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:07.943 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:07.943 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:07.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:08.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:08.945 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:08.965 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:08.965 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:08.965 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:08.965 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:09.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:09.967 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:09.987 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:09.987 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:09.987 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:09.987 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:10.670 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:36:10.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:36:10.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:10.988 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:11.008 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:11.008 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:11.008 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:11.009 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:12.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:12.010 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:12.030 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:12.030 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:12.030 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:12.030 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:13.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:13.032 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:13.051 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:13.051 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:13.052 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:13.052 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:14.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:14.053 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:14.074 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:14.074 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:14.074 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:14.075 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:15.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:15.075 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:15.095 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:15.096 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:15.096 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:15.096 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:16.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:16.097 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:16.116 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:16.116 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:16.116 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:16.116 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:17.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:17.117 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:17.137 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:17.138 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:17.138 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:17.138 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:18.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:18.139 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:18.159 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:18.159 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:18.159 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:18.159 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:19.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:19.160 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:19.180 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:19.181 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:19.181 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:19.181 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:20.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:20.182 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:20.201 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:20.202 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:20.202 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:20.202 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:21.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:21.203 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:21.223 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:21.223 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:21.224 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:21.224 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:22.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:22.225 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:22.245 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:22.246 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:22.246 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:22.246 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:23.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:23.247 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:23.266 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:23.267 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:23.268 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:23.268 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:24.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:24.268 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:24.289 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:24.289 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:24.289 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:24.289 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:25.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:25.291 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:25.311 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:25.311 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:25.311 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:25.311 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:26.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:26.312 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:26.332 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:26.333 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:26.333 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:26.333 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:27.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:27.334 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:27.353 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:27.354 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:27.354 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:27.354 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:28.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:28.355 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:28.375 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:28.376 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:28.376 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:28.376 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:29.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:29.377 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:29.396 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:29.397 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:29.397 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:29.397 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:30.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:30.398 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:30.418 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:30.418 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:30.418 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:30.419 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:31.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:31.420 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:31.440 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:31.441 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:31.441 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:31.441 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:32.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:32.442 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:32.461 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:32.461 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:32.462 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:32.462 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:33.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:33.463 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:33.483 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:33.483 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:33.483 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:33.483 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:34.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:34.485 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:34.505 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:34.505 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:34.505 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:34.505 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:35.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:35.506 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:35.527 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:35.527 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:35.527 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:35.527 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:36.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:36.528 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:36.548 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:36.549 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:36.549 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:36.549 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:37.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:37.550 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:37.569 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:37.570 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:37.571 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:37.571 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:38.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:38.572 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:38.592 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:38.592 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:38.592 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:38.592 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:39.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:39.594 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:39.614 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:39.614 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:39.614 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:39.614 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:40.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:40.615 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:40.634 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:40.635 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:40.636 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:40.636 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:40.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:36:40.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:36:41.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:41.636 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:41.656 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:41.657 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:41.657 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:41.657 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:42.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:42.657 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:42.677 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:42.678 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:42.678 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:42.678 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:43.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:43.679 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:43.699 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:43.700 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:43.700 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:43.700 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:44.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:44.701 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:44.721 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:44.721 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:44.721 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:44.722 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:45.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:45.723 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:45.744 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:45.744 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:45.744 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:45.744 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:46.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:46.745 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:46.765 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:46.766 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:46.766 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:46.766 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:47.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:47.767 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:47.787 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:47.787 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:47.787 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:47.787 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:48.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:48.788 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:48.808 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:48.808 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:48.808 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:48.808 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:49.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:49.810 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:49.830 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:49.830 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:49.830 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:49.830 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:50.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:50.831 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:50.851 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:50.852 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:50.852 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:50.852 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:51.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:51.852 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:51.872 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:51.873 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:51.873 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:51.873 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:52.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:52.874 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:52.894 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:52.894 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:52.894 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:52.894 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:53.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:53.895 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:53.915 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:53.916 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:53.916 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:53.916 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:54.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:54.918 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:54.938 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:54.938 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:54.938 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:54.938 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:55.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:55.940 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:55.960 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:55.960 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:55.960 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:56.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:56.961 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:56.980 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:56.981 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:56.981 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:56.981 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:57.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:57.982 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:58.002 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:58.003 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:58.003 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:58.003 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:36:59.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:36:59.004 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:36:59.024 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:36:59.025 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:36:59.025 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:36:59.025 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:37:00.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:00.026 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:37:00.046 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:37:00.047 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:37:00.047 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:37:00.047 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:37:01.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:01.048 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:37:01.068 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:37:01.068 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_37_00_UTC 2024-06-29T14:37:01.069 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_37_00_UTC', 'scheduled-2024-06-29-14_36_00_UTC'} 2024-06-29T14:37:01.069 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-06-29-14_37_00_UTC'} 2024-06-29T14:37:01.069 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:37:01.069 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-06-29-14_37_00_UTC'] 2024-06-29T14:37:01.069 DEBUG:tasks.cephfs.test_snap_schedules:exec=1719671712.416468, now = 1719671821.0695393, timo = 108 2024-06-29T14:37:02.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-06-29T14:37:02.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.201+0000 7f4776227640 1 -- 172.21.15.81:0/4129558172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47700b35b0 msgr2=0x7f477015b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.201+0000 7f4776227640 1 --2- 172.21.15.81:0/4129558172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47700b35b0 0x7f477015b5b0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f47700cd500 tx=0x7f476002a4d0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.201+0000 7f4776227640 1 -- 172.21.15.81:0/4129558172 shutdown_connections 2024-06-29T14:37:02.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.201+0000 7f4776227640 1 --2- 172.21.15.81:0/4129558172 >> v2:172.21.15.167:3300/0 conn(0x7f477006f660 0x7f477006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.201+0000 7f4776227640 1 --2- 172.21.15.81:0/4129558172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47700b35b0 0x7f477015b5b0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.201+0000 7f4776227640 1 -- 172.21.15.81:0/4129558172 >> 172.21.15.81:0/4129558172 conn(0x7f477006ccc0 msgr2=0x7f47700b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:02.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.201+0000 7f4776227640 1 -- 172.21.15.81:0/4129558172 shutdown_connections 2024-06-29T14:37:02.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.201+0000 7f4776227640 1 -- 172.21.15.81:0/4129558172 wait complete. 2024-06-29T14:37:02.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.203+0000 7f4776227640 1 Processor -- start 2024-06-29T14:37:02.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.203+0000 7f4776227640 1 -- start start 2024-06-29T14:37:02.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.204+0000 7f4776227640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f477006f660 0x7f4770143f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.204+0000 7f4776227640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47700b35b0 0x7f47700aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f4776227640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47700b4430 0x7f4770156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f4776227640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f477006f160 con 0x7f47700b35b0 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f4776227640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f477006efe0 con 0x7f47700b4430 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f4776227640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f477006f2e0 con 0x7f477006f660 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47700b35b0 0x7f47700aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47700b35b0 0x7f47700aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39708/0 (socket says 172.21.15.81:39708) 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476effd640 1 -- 172.21.15.81:0/3743305831 learned_addr learned my addr 172.21.15.81:0/3743305831 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476ffff640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47700b4430 0x7f4770156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476f7fe640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f477006f660 0x7f4770143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476effd640 1 -- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f477006f660 msgr2=0x7f4770143f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476effd640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f477006f660 0x7f4770143f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476effd640 1 -- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47700b4430 msgr2=0x7f4770156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476effd640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47700b4430 0x7f4770156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476effd640 1 -- 172.21.15.81:0/3743305831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4770155380 con 0x7f47700b35b0 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476ffff640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47700b4430 0x7f4770156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f476effd640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47700b35b0 0x7f47700aee70 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f476400b3f0 tx=0x7f476400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f4757fff640 1 -- 172.21.15.81:0/3743305831 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764015020 con 0x7f47700b35b0 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f4757fff640 1 -- 172.21.15.81:0/3743305831 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4764011920 con 0x7f47700b35b0 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f4757fff640 1 -- 172.21.15.81:0/3743305831 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764011d50 con 0x7f47700b35b0 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f4776227640 1 -- 172.21.15.81:0/3743305831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4770155ca0 con 0x7f47700b35b0 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.205+0000 7f4776227640 1 -- 172.21.15.81:0/3743305831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4770156260 con 0x7f47700b35b0 2024-06-29T14:37:02.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.206+0000 7f4776227640 1 -- 172.21.15.81:0/3743305831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f477006fa70 con 0x7f47700b35b0 2024-06-29T14:37:02.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.207+0000 7f4757fff640 1 -- 172.21.15.81:0/3743305831 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 25) ==== 98332+0+0 (secure 0 0 0) 0x7f4764018020 con 0x7f47700b35b0 2024-06-29T14:37:02.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.207+0000 7f4757fff640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f475c076440 0x7f475c078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.208+0000 7f4757fff640 1 -- 172.21.15.81:0/3743305831 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6155+0+0 (secure 0 0 0) 0x7f4764097ac0 con 0x7f47700b35b0 2024-06-29T14:37:02.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.208+0000 7f476f7fe640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f475c076440 0x7f475c078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.208+0000 7f476f7fe640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f475c076440 0x7f475c078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4760033000 tx=0x7f4760004730 comp rx=0 tx=0).ready entity=mgr.5752 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:02.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.212+0000 7f4757fff640 1 -- 172.21.15.81:0/3743305831 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f476401f020 con 0x7f47700b35b0 2024-06-29T14:37:02.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.418+0000 7f4776227640 1 -- 172.21.15.81:0/3743305831 --> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f47700ce080 con 0x7f475c076440 2024-06-29T14:37:02.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.495+0000 7f4757fff640 1 -- 172.21.15.81:0/3743305831 <== mgr.5752 v2:172.21.15.167:6832/2493228728 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f47700ce080 con 0x7f475c076440 2024-06-29T14:37:02.492 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path snap_test_dir1 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 -- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f475c076440 msgr2=0x7f475c078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f475c076440 0x7f475c078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4760033000 tx=0x7f4760004730 comp rx=0 tx=0).stop 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 -- 172.21.15.81:0/3743305831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47700b35b0 msgr2=0x7f47700aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47700b35b0 0x7f47700aee70 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f476400b3f0 tx=0x7f476400b8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 -- 172.21.15.81:0/3743305831 shutdown_connections 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f475c076440 0x7f475c078950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47700b4430 0x7f4770156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47700b35b0 0x7f47700aee70 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 --2- 172.21.15.81:0/3743305831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f477006f660 0x7f4770143f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 -- 172.21.15.81:0/3743305831 >> 172.21.15.81:0/3743305831 conn(0x7f477006ccc0 msgr2=0x7f47700ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 -- 172.21.15.81:0/3743305831 shutdown_connections 2024-06-29T14:37:02.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.497+0000 7f4776227640 1 -- 172.21.15.81:0/3743305831 wait complete. 2024-06-29T14:37:02.505 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-06-29T14:37:02.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:02.506 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:37:02.521 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:37:02.521 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_37_00_UTC 2024-06-29T14:37:02.521 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_36_00_UTC 2024-06-29T14:37:02.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:02.522 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_36_00_UTC) 2024-06-29T14:37:02.581 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_37_00_UTC 2024-06-29T14:37:02.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:02.581 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_37_00_UTC) 2024-06-29T14:37:02.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:02.642 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T14:37:02.701 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:37:02.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.857+0000 7f1fd56ce640 1 -- 172.21.15.81:0/1047132988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd006fd70 msgr2=0x7f1fd00b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.857+0000 7f1fd56ce640 1 --2- 172.21.15.81:0/1047132988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd006fd70 0x7f1fd00b4430 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8005fd0 tx=0x7f1fc802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.857+0000 7f1fd56ce640 1 -- 172.21.15.81:0/1047132988 shutdown_connections 2024-06-29T14:37:02.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.857+0000 7f1fd56ce640 1 --2- 172.21.15.81:0/1047132988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd006fd70 0x7f1fd00b4430 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.857+0000 7f1fd56ce640 1 -- 172.21.15.81:0/1047132988 >> 172.21.15.81:0/1047132988 conn(0x7f1fd006ccc0 msgr2=0x7f1fd00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:02.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.857+0000 7f1fd56ce640 1 -- 172.21.15.81:0/1047132988 shutdown_connections 2024-06-29T14:37:02.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.857+0000 7f1fd56ce640 1 -- 172.21.15.81:0/1047132988 wait complete. 2024-06-29T14:37:02.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.859+0000 7f1fd56ce640 1 Processor -- start 2024-06-29T14:37:02.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.859+0000 7f1fd56ce640 1 -- start start 2024-06-29T14:37:02.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.860+0000 7f1fd56ce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fd00b35b0 0x7f1fd013ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.860+0000 7f1fd56ce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd015b5b0 0x7f1fd013b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.860+0000 7f1fd56ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fd013be00 0x7f1fd00cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.860+0000 7f1fd56ce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd00725b0 con 0x7f1fd015b5b0 2024-06-29T14:37:02.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.860+0000 7f1fd56ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd0072430 con 0x7f1fd013be00 2024-06-29T14:37:02.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.860+0000 7f1fd56ce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1fd0072730 con 0x7f1fd00b35b0 2024-06-29T14:37:02.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd015b5b0 0x7f1fd013b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd015b5b0 0x7f1fd013b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39726/0 (socket says 172.21.15.81:39726) 2024-06-29T14:37:02.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc7fff640 1 -- 172.21.15.81:0/430169660 learned_addr learned my addr 172.21.15.81:0/430169660 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fcf7fe640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fd013be00 0x7f1fd00cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fceffd640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fd00b35b0 0x7f1fd013ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc7fff640 1 -- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fd00b35b0 msgr2=0x7f1fd013ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc7fff640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fd00b35b0 0x7f1fd013ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc7fff640 1 -- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fd013be00 msgr2=0x7f1fd00cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc7fff640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fd013be00 0x7f1fd00cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc7fff640 1 -- 172.21.15.81:0/430169660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd013c610 con 0x7f1fd015b5b0 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc7fff640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd015b5b0 0x7f1fd013b8c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800c370 tx=0x7f1fb800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc77fe640 1 -- 172.21.15.81:0/430169660 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8017070 con 0x7f1fd015b5b0 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc77fe640 1 -- 172.21.15.81:0/430169660 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fb80035f0 con 0x7f1fd015b5b0 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fd56ce640 1 -- 172.21.15.81:0/430169660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd013c8f0 con 0x7f1fd015b5b0 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fc77fe640 1 -- 172.21.15.81:0/430169660 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8003a40 con 0x7f1fd015b5b0 2024-06-29T14:37:02.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.861+0000 7f1fd56ce640 1 -- 172.21.15.81:0/430169660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd013ceb0 con 0x7f1fd015b5b0 2024-06-29T14:37:02.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.862+0000 7f1fd56ce640 1 -- 172.21.15.81:0/430169660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd0072c50 con 0x7f1fd015b5b0 2024-06-29T14:37:02.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.863+0000 7f1fc77fe640 1 -- 172.21.15.81:0/430169660 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 25) ==== 98332+0+0 (secure 0 0 0) 0x7f1fb801a070 con 0x7f1fd015b5b0 2024-06-29T14:37:02.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.863+0000 7f1fc77fe640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f1fb0076440 0x7f1fb0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:02.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.864+0000 7f1fc77fe640 1 -- 172.21.15.81:0/430169660 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6155+0+0 (secure 0 0 0) 0x7f1fb8096f10 con 0x7f1fd015b5b0 2024-06-29T14:37:02.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.864+0000 7f1fceffd640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f1fb0076440 0x7f1fb0078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:02.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.864+0000 7f1fceffd640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f1fb0076440 0x7f1fb0078950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0000f60 tx=0x7f1fc0009040 comp rx=0 tx=0).ready entity=mgr.5752 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:02.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:02.868+0000 7f1fc77fe640 1 -- 172.21.15.81:0/430169660 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1fb8060490 con 0x7f1fd015b5b0 2024-06-29T14:37:03.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.093+0000 7f1fd56ce640 1 -- 172.21.15.81:0/430169660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f1fd00cbdc0 con 0x7f1fd015b5b0 2024-06-29T14:37:03.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.224+0000 7f1fc77fe640 1 -- 172.21.15.81:0/430169660 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v26) ==== 93+0+0 (secure 0 0 0) 0x7f1fb805fd90 con 0x7f1fd015b5b0 2024-06-29T14:37:03.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.227+0000 7f1fd56ce640 1 -- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f1fb0076440 msgr2=0x7f1fb0078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.227+0000 7f1fd56ce640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f1fb0076440 0x7f1fb0078950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0000f60 tx=0x7f1fc0009040 comp rx=0 tx=0).stop 2024-06-29T14:37:03.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.227+0000 7f1fd56ce640 1 -- 172.21.15.81:0/430169660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd015b5b0 msgr2=0x7f1fd013b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.227+0000 7f1fd56ce640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd015b5b0 0x7f1fd013b8c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800c370 tx=0x7f1fb800c830 comp rx=0 tx=0).stop 2024-06-29T14:37:03.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.228+0000 7f1fd56ce640 1 -- 172.21.15.81:0/430169660 shutdown_connections 2024-06-29T14:37:03.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.228+0000 7f1fd56ce640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f1fb0076440 0x7f1fb0078950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.228+0000 7f1fd56ce640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fd013be00 0x7f1fd00cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.228+0000 7f1fd56ce640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd015b5b0 0x7f1fd013b8c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.228+0000 7f1fd56ce640 1 --2- 172.21.15.81:0/430169660 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fd00b35b0 0x7f1fd013ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.228+0000 7f1fd56ce640 1 -- 172.21.15.81:0/430169660 >> 172.21.15.81:0/430169660 conn(0x7f1fd006ccc0 msgr2=0x7f1fd00b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:03.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.228+0000 7f1fd56ce640 1 -- 172.21.15.81:0/430169660 shutdown_connections 2024-06-29T14:37:03.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.228+0000 7f1fd56ce640 1 -- 172.21.15.81:0/430169660 wait complete. 2024-06-29T14:37:03.238 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:37:03.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:03.272 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:03.328 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:37:03.328 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:03.328 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:03.329 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:03.344 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:03.344 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:03.400 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:03.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.557+0000 7f0a7a081640 1 -- 172.21.15.81:0/53790293 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a740b5250 msgr2=0x7f0a740d2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.557+0000 7f0a7a081640 1 --2- 172.21.15.81:0/53790293 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a740b5250 0x7f0a740d2190 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009be0 tx=0x7f0a6c02a940 comp rx=0 tx=0).stop 2024-06-29T14:37:03.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.557+0000 7f0a7a081640 1 -- 172.21.15.81:0/53790293 shutdown_connections 2024-06-29T14:37:03.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.557+0000 7f0a7a081640 1 --2- 172.21.15.81:0/53790293 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a740b5250 0x7f0a740d2190 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.557+0000 7f0a7a081640 1 -- 172.21.15.81:0/53790293 >> 172.21.15.81:0/53790293 conn(0x7f0a7409cf30 msgr2=0x7f0a740d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:03.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.557+0000 7f0a7a081640 1 -- 172.21.15.81:0/53790293 shutdown_connections 2024-06-29T14:37:03.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.557+0000 7f0a7a081640 1 -- 172.21.15.81:0/53790293 wait complete. 2024-06-29T14:37:03.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.559+0000 7f0a7a081640 1 Processor -- start 2024-06-29T14:37:03.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.559+0000 7f0a7a081640 1 -- start start 2024-06-29T14:37:03.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a7a081640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a740b5250 0x7f0a74155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a7a081640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a740b9040 0x7f0a740b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a7a081640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a740b4420 0x7f0a740b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a7a081640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0a740d2850 con 0x7f0a740b4420 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a7a081640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0a740d26d0 con 0x7f0a740b9040 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a7a081640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0a740d29d0 con 0x7f0a740b5250 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a73fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a740b4420 0x7f0a740b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a73fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a740b4420 0x7f0a740b1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39766/0 (socket says 172.21.15.81:39766) 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a73fff640 1 -- 172.21.15.81:0/1970584512 learned_addr learned my addr 172.21.15.81:0/1970584512 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a737fe640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a740b5250 0x7f0a74155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a6affd640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a740b9040 0x7f0a740b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a73fff640 1 -- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a740b5250 msgr2=0x7f0a74155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a73fff640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a740b5250 0x7f0a74155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a73fff640 1 -- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a740b9040 msgr2=0x7f0a740b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a73fff640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a740b9040 0x7f0a740b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a73fff640 1 -- 172.21.15.81:0/1970584512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a741f8340 con 0x7f0a740b4420 2024-06-29T14:37:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a73fff640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a740b4420 0x7f0a740b1d80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c005fd0 tx=0x7f0a6c004660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:03.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a70ff9640 1 -- 172.21.15.81:0/1970584512 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c003e60 con 0x7f0a740b4420 2024-06-29T14:37:03.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a70ff9640 1 -- 172.21.15.81:0/1970584512 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a6c031ac0 con 0x7f0a740b4420 2024-06-29T14:37:03.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a70ff9640 1 -- 172.21.15.81:0/1970584512 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c004850 con 0x7f0a740b4420 2024-06-29T14:37:03.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.561+0000 7f0a7a081640 1 -- 172.21.15.81:0/1970584512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a741f8620 con 0x7f0a740b4420 2024-06-29T14:37:03.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.562+0000 7f0a7a081640 1 -- 172.21.15.81:0/1970584512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a741f8be0 con 0x7f0a740b4420 2024-06-29T14:37:03.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.563+0000 7f0a7a081640 1 -- 172.21.15.81:0/1970584512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a741f8090 con 0x7f0a740b4420 2024-06-29T14:37:03.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.563+0000 7f0a70ff9640 1 -- 172.21.15.81:0/1970584512 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7f0a6c03b410 con 0x7f0a740b4420 2024-06-29T14:37:03.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.564+0000 7f0a70ff9640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0a540763f0 0x7f0a54078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.564+0000 7f0a737fe640 1 -- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0a540763f0 msgr2=0x7f0a54078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:03.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.564+0000 7f0a70ff9640 1 -- 172.21.15.81:0/1970584512 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6155+0+0 (secure 0 0 0) 0x7f0a6c0b8b20 con 0x7f0a740b4420 2024-06-29T14:37:03.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.564+0000 7f0a737fe640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0a540763f0 0x7f0a54078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:03.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.568+0000 7f0a70ff9640 1 -- 172.21.15.81:0/1970584512 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0a6c04a030 con 0x7f0a740b4420 2024-06-29T14:37:03.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.752+0000 7f0a7a081640 1 -- 172.21.15.81:0/1970584512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a74068620 con 0x7f0a740b4420 2024-06-29T14:37:03.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.753+0000 7f0a70ff9640 1 -- 172.21.15.81:0/1970584512 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+17274 (secure 0 0 0) 0x7f0a6c03f030 con 0x7f0a740b4420 2024-06-29T14:37:03.751 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:03.751 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":64,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:36:12.073468+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:34:56.911140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-06-29T14:34:57.555296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.12","mappings":[{"from":0,"to":5}]},{"pgid":"7.2c","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/1090782941":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6832/1090782941":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/1626573527":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/799531669":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/2483549331":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/2831660161":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/3380278063":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/605629739":"2024-06-30T14:35:10.908137+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:03.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.755+0000 7f0a7a081640 1 -- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0a540763f0 msgr2=0x7f0a54078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:03.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.755+0000 7f0a7a081640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0a540763f0 0x7f0a54078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.755+0000 7f0a7a081640 1 -- 172.21.15.81:0/1970584512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a740b4420 msgr2=0x7f0a740b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.755+0000 7f0a7a081640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a740b4420 0x7f0a740b1d80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c005fd0 tx=0x7f0a6c004660 comp rx=0 tx=0).stop 2024-06-29T14:37:03.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.755+0000 7f0a7a081640 1 -- 172.21.15.81:0/1970584512 shutdown_connections 2024-06-29T14:37:03.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.755+0000 7f0a7a081640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0a540763f0 0x7f0a54078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.755+0000 7f0a7a081640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a740b4420 0x7f0a740b1d80 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.755+0000 7f0a7a081640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0a740b9040 0x7f0a740b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.755+0000 7f0a7a081640 1 --2- 172.21.15.81:0/1970584512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0a740b5250 0x7f0a74155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.755+0000 7f0a7a081640 1 -- 172.21.15.81:0/1970584512 >> 172.21.15.81:0/1970584512 conn(0x7f0a7409cf30 msgr2=0x7f0a7415f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:03.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.756+0000 7f0a7a081640 1 -- 172.21.15.81:0/1970584512 shutdown_connections 2024-06-29T14:37:03.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.756+0000 7f0a7a081640 1 -- 172.21.15.81:0/1970584512 wait complete. 2024-06-29T14:37:03.765 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:34:56.911140+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:34:57.555296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:03.766 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:03.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.887+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2435415489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80ce770 msgr2=0x7fa6c80b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.887+0000 7fa6cfe5e640 1 --2- 172.21.15.81:0/2435415489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80ce770 0x7fa6c80b35c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8005fd0 tx=0x7fa6b802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.887+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2435415489 shutdown_connections 2024-06-29T14:37:03.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.887+0000 7fa6cfe5e640 1 --2- 172.21.15.81:0/2435415489 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6c80d2950 0x7fa6c80b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.887+0000 7fa6cfe5e640 1 --2- 172.21.15.81:0/2435415489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80ce770 0x7fa6c80b35c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.887+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2435415489 >> 172.21.15.81:0/2435415489 conn(0x7fa6c80d3c60 msgr2=0x7fa6c80d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:03.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.887+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2435415489 shutdown_connections 2024-06-29T14:37:03.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.887+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2435415489 wait complete. 2024-06-29T14:37:03.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.889+0000 7fa6cfe5e640 1 Processor -- start 2024-06-29T14:37:03.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.889+0000 7fa6cfe5e640 1 -- start start 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cfe5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80ce770 0x7fa6c8156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cfe5e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6c80ceb80 0x7fa6c80b6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cfe5e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6c80d2950 0x7fa6c80b7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cfe5e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c80d15a0 con 0x7fa6c80ce770 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cfe5e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c80d1420 con 0x7fa6c80ceb80 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cfe5e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa6c80d4990 con 0x7fa6c80d2950 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cdbfa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80ce770 0x7fa6c8156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cdbfa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80ce770 0x7fa6c8156a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39780/0 (socket says 172.21.15.81:39780) 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cdbfa640 1 -- 172.21.15.81:0/2783066943 learned_addr learned my addr 172.21.15.81:0/2783066943 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6ce3fb640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6c80d2950 0x7fa6c80b7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cd3f9640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6c80ceb80 0x7fa6c80b6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cdbfa640 1 -- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6c80d2950 msgr2=0x7fa6c80b7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cdbfa640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6c80d2950 0x7fa6c80b7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cdbfa640 1 -- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6c80ceb80 msgr2=0x7fa6c80b6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cdbfa640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6c80ceb80 0x7fa6c80b6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.890+0000 7fa6cdbfa640 1 -- 172.21.15.81:0/2783066943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c8160d50 con 0x7fa6c80ce770 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.891+0000 7fa6cdbfa640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80ce770 0x7fa6c8156a20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8034040 tx=0x7fa6b80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.891+0000 7fa6be7fc640 1 -- 172.21.15.81:0/2783066943 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b803e030 con 0x7fa6c80ce770 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.891+0000 7fa6be7fc640 1 -- 172.21.15.81:0/2783066943 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa6b8003ae0 con 0x7fa6c80ce770 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.891+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2783066943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6c8161030 con 0x7fa6c80ce770 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.891+0000 7fa6be7fc640 1 -- 172.21.15.81:0/2783066943 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8035040 con 0x7fa6c80ce770 2024-06-29T14:37:03.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.891+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2783066943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6c8166660 con 0x7fa6c80ce770 2024-06-29T14:37:03.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.892+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2783066943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6c80d2240 con 0x7fa6c80ce770 2024-06-29T14:37:03.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.892+0000 7fa6be7fc640 1 -- 172.21.15.81:0/2783066943 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7fa6b8041030 con 0x7fa6c80ce770 2024-06-29T14:37:03.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.893+0000 7fa6be7fc640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa6ac076420 0x7fa6ac078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:03.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.893+0000 7fa6cd3f9640 1 -- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa6ac076420 msgr2=0x7fa6ac078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:03.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.893+0000 7fa6be7fc640 1 -- 172.21.15.81:0/2783066943 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6155+0+0 (secure 0 0 0) 0x7fa6b80b8290 con 0x7fa6c80ce770 2024-06-29T14:37:03.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.893+0000 7fa6cd3f9640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa6ac076420 0x7fa6ac078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:03.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:03.897+0000 7fa6be7fc640 1 -- 172.21.15.81:0/2783066943 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa6b80811c0 con 0x7fa6c80ce770 2024-06-29T14:37:04.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.084+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2783066943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa6c8166070 con 0x7fa6c80ce770 2024-06-29T14:37:04.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.084+0000 7fa6be7fc640 1 -- 172.21.15.81:0/2783066943 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) ==== 93+0+10 (secure 0 0 0) 0x7fa6b80854c0 con 0x7fa6c80ce770 2024-06-29T14:37:04.082 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:37:04.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa6ac076420 msgr2=0x7fa6ac078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:04.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa6ac076420 0x7fa6ac078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2783066943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80ce770 msgr2=0x7fa6c8156a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80ce770 0x7fa6c8156a20 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8034040 tx=0x7fa6b80035f0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2783066943 shutdown_connections 2024-06-29T14:37:04.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa6ac076420 0x7fa6ac078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6c80d2950 0x7fa6c80b7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6c80ceb80 0x7fa6c80b6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 --2- 172.21.15.81:0/2783066943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80ce770 0x7fa6c8156a20 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2783066943 >> 172.21.15.81:0/2783066943 conn(0x7fa6c80d3c60 msgr2=0x7fa6c806d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2783066943 shutdown_connections 2024-06-29T14:37:04.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.086+0000 7fa6cfe5e640 1 -- 172.21.15.81:0/2783066943 wait complete. 2024-06-29T14:37:04.096 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:37:04.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.218+0000 7fb75fe22640 1 -- 172.21.15.81:0/1093193712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7580d2a90 msgr2=0x7fb7580d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.218+0000 7fb75fe22640 1 --2- 172.21.15.81:0/1093193712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7580d2a90 0x7fb7580d09c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fb75808e230 tx=0x7fb75002a940 comp rx=0 tx=0).stop 2024-06-29T14:37:04.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.218+0000 7fb75fe22640 1 -- 172.21.15.81:0/1093193712 shutdown_connections 2024-06-29T14:37:04.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.218+0000 7fb75fe22640 1 --2- 172.21.15.81:0/1093193712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7580d2a90 0x7fb7580d09c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.218+0000 7fb75fe22640 1 --2- 172.21.15.81:0/1093193712 >> v2:172.21.15.167:3300/0 conn(0x7fb7580b9040 0x7fb7580d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.218+0000 7fb75fe22640 1 -- 172.21.15.81:0/1093193712 >> 172.21.15.81:0/1093193712 conn(0x7fb75809cf30 msgr2=0x7fb7580d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.218+0000 7fb75fe22640 1 -- 172.21.15.81:0/1093193712 shutdown_connections 2024-06-29T14:37:04.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.218+0000 7fb75fe22640 1 -- 172.21.15.81:0/1093193712 wait complete. 2024-06-29T14:37:04.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.219+0000 7fb75fe22640 1 Processor -- start 2024-06-29T14:37:04.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.220+0000 7fb75fe22640 1 -- start start 2024-06-29T14:37:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75fe22640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7580b5250 0x7fb7580b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75fe22640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7580b9040 0x7fb75815bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75fe22640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7580d2a90 0x7fb7580b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75fe22640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb7580b7d30 con 0x7fb7580b5250 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75fe22640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb7580b7bb0 con 0x7fb7580b9040 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75fe22640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb7580b7eb0 con 0x7fb7580d2a90 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75dbbe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7580b5250 0x7fb7580b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75dbbe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7580b5250 0x7fb7580b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39800/0 (socket says 172.21.15.81:39800) 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75dbbe640 1 -- 172.21.15.81:0/1429549422 learned_addr learned my addr 172.21.15.81:0/1429549422 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75d3bd640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7580b9040 0x7fb75815bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75e3bf640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7580d2a90 0x7fb7580b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75dbbe640 1 -- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7580d2a90 msgr2=0x7fb7580b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75dbbe640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7580d2a90 0x7fb7580b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75dbbe640 1 -- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7580b9040 msgr2=0x7fb75815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75dbbe640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7580b9040 0x7fb75815bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75dbbe640 1 -- 172.21.15.81:0/1429549422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7581f5520 con 0x7fb7580b5250 2024-06-29T14:37:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.221+0000 7fb75dbbe640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7580b5250 0x7fb7580b0820 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb74800e7c0 tx=0x7fb74800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.222+0000 7fb74e7fc640 1 -- 172.21.15.81:0/1429549422 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74800ccc0 con 0x7fb7580b5250 2024-06-29T14:37:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.222+0000 7fb74e7fc640 1 -- 172.21.15.81:0/1429549422 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb748010430 con 0x7fb7580b5250 2024-06-29T14:37:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.222+0000 7fb74e7fc640 1 -- 172.21.15.81:0/1429549422 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb748010860 con 0x7fb7580b5250 2024-06-29T14:37:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.222+0000 7fb75fe22640 1 -- 172.21.15.81:0/1429549422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7581f5990 con 0x7fb7580b5250 2024-06-29T14:37:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.222+0000 7fb75fe22640 1 -- 172.21.15.81:0/1429549422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7580b2aa0 con 0x7fb7580b5250 2024-06-29T14:37:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.223+0000 7fb75fe22640 1 -- 172.21.15.81:0/1429549422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7580b8440 con 0x7fb7580b5250 2024-06-29T14:37:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.224+0000 7fb74e7fc640 1 -- 172.21.15.81:0/1429549422 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7fb748010a00 con 0x7fb7580b5250 2024-06-29T14:37:04.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.225+0000 7fb74e7fc640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb73c0763f0 0x7fb73c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.225+0000 7fb75d3bd640 1 -- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb73c0763f0 msgr2=0x7fb73c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:04.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.225+0000 7fb74e7fc640 1 -- 172.21.15.81:0/1429549422 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6155+0+0 (secure 0 0 0) 0x7fb7480985f0 con 0x7fb7580b5250 2024-06-29T14:37:04.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.225+0000 7fb75d3bd640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb73c0763f0 0x7fb73c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:04.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.229+0000 7fb74e7fc640 1 -- 172.21.15.81:0/1429549422 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb748061490 con 0x7fb7580b5250 2024-06-29T14:37:04.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.417+0000 7fb75fe22640 1 -- 172.21.15.81:0/1429549422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb7580b0d60 con 0x7fb7580b5250 2024-06-29T14:37:04.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.417+0000 7fb74e7fc640 1 -- 172.21.15.81:0/1429549422 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v64) ==== 104+0+11 (secure 0 0 0) 0x7fb748065790 con 0x7fb7580b5250 2024-06-29T14:37:04.415 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:37:04.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 -- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb73c0763f0 msgr2=0x7fb73c078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:04.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb73c0763f0 0x7fb73c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 -- 172.21.15.81:0/1429549422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7580b5250 msgr2=0x7fb7580b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7580b5250 0x7fb7580b0820 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb74800e7c0 tx=0x7fb74800ec80 comp rx=0 tx=0).stop 2024-06-29T14:37:04.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 -- 172.21.15.81:0/1429549422 shutdown_connections 2024-06-29T14:37:04.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb73c0763f0 0x7fb73c078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7580d2a90 0x7fb7580b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7580b9040 0x7fb75815bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 --2- 172.21.15.81:0/1429549422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7580b5250 0x7fb7580b0820 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 -- 172.21.15.81:0/1429549422 >> 172.21.15.81:0/1429549422 conn(0x7fb75809cf30 msgr2=0x7fb7581612e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 -- 172.21.15.81:0/1429549422 shutdown_connections 2024-06-29T14:37:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.419+0000 7fb75fe22640 1 -- 172.21.15.81:0/1429549422 wait complete. 2024-06-29T14:37:04.428 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:37:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.549+0000 7fa8f7952640 1 -- 172.21.15.81:0/2921870011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8f00ceb80 msgr2=0x7fa8f00b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.549+0000 7fa8f7952640 1 --2- 172.21.15.81:0/2921870011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8f00ceb80 0x7fa8f00b3b00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa8f008c2f0 tx=0x7fa8ec02a4d0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.549+0000 7fa8f7952640 1 -- 172.21.15.81:0/2921870011 shutdown_connections 2024-06-29T14:37:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.549+0000 7fa8f7952640 1 --2- 172.21.15.81:0/2921870011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8f00ceb80 0x7fa8f00b3b00 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.549+0000 7fa8f7952640 1 --2- 172.21.15.81:0/2921870011 >> v2:172.21.15.167:3300/0 conn(0x7fa8f00ce770 0x7fa8f00b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.549+0000 7fa8f7952640 1 -- 172.21.15.81:0/2921870011 >> 172.21.15.81:0/2921870011 conn(0x7fa8f00d3c60 msgr2=0x7fa8f00d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.549+0000 7fa8f7952640 1 -- 172.21.15.81:0/2921870011 shutdown_connections 2024-06-29T14:37:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.549+0000 7fa8f7952640 1 -- 172.21.15.81:0/2921870011 wait complete. 2024-06-29T14:37:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.551+0000 7fa8f7952640 1 Processor -- start 2024-06-29T14:37:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.551+0000 7fa8f7952640 1 -- start start 2024-06-29T14:37:04.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f7952640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8f00ce770 0x7fa8f01f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f7952640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8f00ceb80 0x7fa8f01e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f7952640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8f00d2240 0x7fa8f009e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f7952640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f00d15a0 con 0x7fa8f00ceb80 2024-06-29T14:37:04.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f7952640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f00d1420 con 0x7fa8f00ce770 2024-06-29T14:37:04.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f7952640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f00d4990 con 0x7fa8f00d2240 2024-06-29T14:37:04.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f4eed640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8f00ceb80 0x7fa8f01e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f4eed640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8f00ceb80 0x7fa8f01e5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39810/0 (socket says 172.21.15.81:39810) 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f4eed640 1 -- 172.21.15.81:0/1732684970 learned_addr learned my addr 172.21.15.81:0/1732684970 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f56ee640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8f00ce770 0x7fa8f01f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f5eef640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8f00d2240 0x7fa8f009e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f4eed640 1 -- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8f00d2240 msgr2=0x7fa8f009e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f4eed640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8f00d2240 0x7fa8f009e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f4eed640 1 -- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8f00ce770 msgr2=0x7fa8f01f6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f4eed640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8f00ce770 0x7fa8f01f6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f4eed640 1 -- 172.21.15.81:0/1732684970 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f009edc0 con 0x7fa8f00ceb80 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8f4eed640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8f00ceb80 0x7fa8f01e5f70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0099c0 tx=0x7fa8ec0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.553+0000 7fa8e5ffb640 1 -- 172.21.15.81:0/1732684970 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec03e030 con 0x7fa8f00ceb80 2024-06-29T14:37:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.554+0000 7fa8e5ffb640 1 -- 172.21.15.81:0/1732684970 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8ec0047a0 con 0x7fa8f00ceb80 2024-06-29T14:37:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.554+0000 7fa8e5ffb640 1 -- 172.21.15.81:0/1732684970 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec035040 con 0x7fa8f00ceb80 2024-06-29T14:37:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.554+0000 7fa8f7952640 1 -- 172.21.15.81:0/1732684970 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f009ead0 con 0x7fa8f00ceb80 2024-06-29T14:37:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.554+0000 7fa8f7952640 1 -- 172.21.15.81:0/1732684970 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f00a14b0 con 0x7fa8f00ceb80 2024-06-29T14:37:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.555+0000 7fa8f7952640 1 -- 172.21.15.81:0/1732684970 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8f00d2650 con 0x7fa8f00ceb80 2024-06-29T14:37:04.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.555+0000 7fa8e5ffb640 1 -- 172.21.15.81:0/1732684970 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7fa8ec041030 con 0x7fa8f00ceb80 2024-06-29T14:37:04.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.556+0000 7fa8e5ffb640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa8d40763f0 0x7fa8d4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.556+0000 7fa8f56ee640 1 -- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa8d40763f0 msgr2=0x7fa8d4078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:04.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.556+0000 7fa8e5ffb640 1 -- 172.21.15.81:0/1732684970 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6155+0+0 (secure 0 0 0) 0x7fa8ec0b83b0 con 0x7fa8f00ceb80 2024-06-29T14:37:04.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.556+0000 7fa8f56ee640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa8d40763f0 0x7fa8d4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:04.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.565+0000 7fa8e5ffb640 1 -- 172.21.15.81:0/1732684970 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa8f009ead0 con 0x7fa8f00ceb80 2024-06-29T14:37:04.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.754+0000 7fa8f7952640 1 -- 172.21.15.81:0/1732684970 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa8f00b3e20 con 0x7fa8f00ceb80 2024-06-29T14:37:04.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.754+0000 7fa8e5ffb640 1 -- 172.21.15.81:0/1732684970 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v64) ==== 100+0+11 (secure 0 0 0) 0x7fa8ec0811f0 con 0x7fa8f00ceb80 2024-06-29T14:37:04.752 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:37:04.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 -- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa8d40763f0 msgr2=0x7fa8d4078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:04.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa8d40763f0 0x7fa8d4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 -- 172.21.15.81:0/1732684970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8f00ceb80 msgr2=0x7fa8f01e5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8f00ceb80 0x7fa8f01e5f70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0099c0 tx=0x7fa8ec0039e0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 -- 172.21.15.81:0/1732684970 shutdown_connections 2024-06-29T14:37:04.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fa8d40763f0 0x7fa8d4078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8f00d2240 0x7fa8f009e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8f00ceb80 0x7fa8f01e5f70 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 --2- 172.21.15.81:0/1732684970 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8f00ce770 0x7fa8f01f6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 -- 172.21.15.81:0/1732684970 >> 172.21.15.81:0/1732684970 conn(0x7fa8f00d3c60 msgr2=0x7fa8f00b1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 -- 172.21.15.81:0/1732684970 shutdown_connections 2024-06-29T14:37:04.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.756+0000 7fa8f7952640 1 -- 172.21.15.81:0/1732684970 wait complete. 2024-06-29T14:37:04.765 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:37:04.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.884+0000 7f83b2c4d640 1 -- 172.21.15.81:0/790697366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83ac0cd150 msgr2=0x7f83ac072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.884+0000 7f83b2c4d640 1 --2- 172.21.15.81:0/790697366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83ac0cd150 0x7f83ac072f40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f83ac072cf0 tx=0x7f83a002a560 comp rx=0 tx=0).stop 2024-06-29T14:37:04.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.884+0000 7f83b2c4d640 1 -- 172.21.15.81:0/790697366 shutdown_connections 2024-06-29T14:37:04.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.884+0000 7f83b2c4d640 1 --2- 172.21.15.81:0/790697366 >> v2:172.21.15.167:3300/0 conn(0x7f83ac15bdf0 0x7f83ac06ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.884+0000 7f83b2c4d640 1 --2- 172.21.15.81:0/790697366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83ac0cd150 0x7f83ac072f40 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.884+0000 7f83b2c4d640 1 -- 172.21.15.81:0/790697366 >> 172.21.15.81:0/790697366 conn(0x7f83ac0b2920 msgr2=0x7f83ac0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:04.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.884+0000 7f83b2c4d640 1 -- 172.21.15.81:0/790697366 shutdown_connections 2024-06-29T14:37:04.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.884+0000 7f83b2c4d640 1 -- 172.21.15.81:0/790697366 wait complete. 2024-06-29T14:37:04.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.887+0000 7f83b2c4d640 1 Processor -- start 2024-06-29T14:37:04.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.887+0000 7f83b2c4d640 1 -- start start 2024-06-29T14:37:04.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.889+0000 7f83b2c4d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83ac073480 0x7f83ac158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.889+0000 7f83b2c4d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83ac0cd150 0x7f83ac13ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.889+0000 7f83b2c4d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83ac15bdf0 0x7f83ac0ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.889+0000 7f83b2c4d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac06f450 con 0x7f83ac073480 2024-06-29T14:37:04.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.889+0000 7f83b2c4d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac06f2d0 con 0x7f83ac15bdf0 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.889+0000 7f83b2c4d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f83ac06f5d0 con 0x7f83ac0cd150 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b09e9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83ac073480 0x7f83ac158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b09e9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83ac073480 0x7f83ac158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39836/0 (socket says 172.21.15.81:39836) 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b09e9640 1 -- 172.21.15.81:0/1419732111 learned_addr learned my addr 172.21.15.81:0/1419732111 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b11ea640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83ac15bdf0 0x7f83ac0ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83abfff640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83ac0cd150 0x7f83ac13ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b09e9640 1 -- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83ac0cd150 msgr2=0x7f83ac13ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b09e9640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83ac0cd150 0x7f83ac13ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b09e9640 1 -- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83ac15bdf0 msgr2=0x7f83ac0ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b09e9640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83ac15bdf0 0x7f83ac0ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b09e9640 1 -- 172.21.15.81:0/1419732111 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ac156a70 con 0x7f83ac073480 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b09e9640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83ac073480 0x7f83ac158010 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f83a0034040 tx=0x7f83a00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:04.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83a97fa640 1 -- 172.21.15.81:0/1419732111 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a0035040 con 0x7f83ac073480 2024-06-29T14:37:04.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83a97fa640 1 -- 172.21.15.81:0/1419732111 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83a00047a0 con 0x7f83ac073480 2024-06-29T14:37:04.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b2c4d640 1 -- 172.21.15.81:0/1419732111 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83ac156f70 con 0x7f83ac073480 2024-06-29T14:37:04.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83a97fa640 1 -- 172.21.15.81:0/1419732111 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a0004bf0 con 0x7f83ac073480 2024-06-29T14:37:04.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.890+0000 7f83b2c4d640 1 -- 172.21.15.81:0/1419732111 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83ac157530 con 0x7f83ac073480 2024-06-29T14:37:04.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.892+0000 7f83b2c4d640 1 -- 172.21.15.81:0/1419732111 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83ac072f40 con 0x7f83ac073480 2024-06-29T14:37:04.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.892+0000 7f83a97fa640 1 -- 172.21.15.81:0/1419732111 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7f83a0004d90 con 0x7f83ac073480 2024-06-29T14:37:04.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.892+0000 7f83a97fa640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f8390076420 0x7f8390078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:04.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.892+0000 7f83abfff640 1 -- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f8390076420 msgr2=0x7f8390078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:04.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.893+0000 7f83a97fa640 1 -- 172.21.15.81:0/1419732111 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6155+0+0 (secure 0 0 0) 0x7f83a00ba460 con 0x7f83ac073480 2024-06-29T14:37:04.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.893+0000 7f83abfff640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f8390076420 0x7f8390078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:04.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:04.897+0000 7f83a97fa640 1 -- 172.21.15.81:0/1419732111 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f83a00833a0 con 0x7f83ac073480 2024-06-29T14:37:05.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.094+0000 7f83abfff640 1 -- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f8390076420 msgr2=0x7f8390078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:05.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.094+0000 7f83abfff640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f8390076420 0x7f8390078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:05.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.098+0000 7f83b2c4d640 1 -- 172.21.15.81:0/1419732111 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f83ac0b2270 con 0x7f83ac073480 2024-06-29T14:37:05.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.098+0000 7f83a97fa640 1 -- 172.21.15.81:0/1419732111 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v64) ==== 80+0+476 (secure 0 0 0) 0x7f83a008c030 con 0x7f83ac073480 2024-06-29T14:37:05.095 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/1090782941 2024-06-30T14:35:10.908137+0000 2024-06-29T14:37:05.096 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/1090782941 2024-06-30T14:35:10.908137+0000 2024-06-29T14:37:05.096 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1626573527 2024-06-30T14:35:10.908137+0000 2024-06-29T14:37:05.096 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/799531669 2024-06-30T14:35:10.908137+0000 2024-06-29T14:37:05.096 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2483549331 2024-06-30T14:35:10.908137+0000 2024-06-29T14:37:05.096 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2831660161 2024-06-30T14:35:10.908137+0000 2024-06-29T14:37:05.096 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3380278063 2024-06-30T14:35:10.908137+0000 2024-06-29T14:37:05.096 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/605629739 2024-06-30T14:35:10.908137+0000 2024-06-29T14:37:05.096 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T14:37:05.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 -- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f8390076420 msgr2=0x7f8390078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:05.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f8390076420 0x7f8390078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 -- 172.21.15.81:0/1419732111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83ac073480 msgr2=0x7f83ac158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83ac073480 0x7f83ac158010 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f83a0034040 tx=0x7f83a00038a0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 -- 172.21.15.81:0/1419732111 shutdown_connections 2024-06-29T14:37:05.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f8390076420 0x7f8390078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83ac15bdf0 0x7f83ac0ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83ac0cd150 0x7f83ac13ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 --2- 172.21.15.81:0/1419732111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83ac073480 0x7f83ac158010 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 -- 172.21.15.81:0/1419732111 >> 172.21.15.81:0/1419732111 conn(0x7f83ac0b2920 msgr2=0x7f83ac154eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:05.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 -- 172.21.15.81:0/1419732111 shutdown_connections 2024-06-29T14:37:05.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.100+0000 7f83b2c4d640 1 -- 172.21.15.81:0/1419732111 wait complete. 2024-06-29T14:37:05.109 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:37:05.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:05.109 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:05.139 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:37:05.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:05.139 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:37:05.194 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:05.195 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:37:05.195 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:05.211 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:05.211 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:05.211 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:05.211 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:05.250 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:05.251 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:05.251 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:37:05.251 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:05.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:05.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.410+0000 7f5bc0103640 1 -- 172.21.15.81:0/295311582 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb80b5f50 msgr2=0x7f5bb8166870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.410+0000 7f5bc0103640 1 --2- 172.21.15.81:0/295311582 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb80b5f50 0x7f5bb8166870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0005fd0 tx=0x7f5bb002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.410+0000 7f5bc0103640 1 -- 172.21.15.81:0/295311582 shutdown_connections 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.410+0000 7f5bc0103640 1 --2- 172.21.15.81:0/295311582 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb80b5f50 0x7f5bb8166870 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.410+0000 7f5bc0103640 1 -- 172.21.15.81:0/295311582 >> 172.21.15.81:0/295311582 conn(0x7f5bb80b3680 msgr2=0x7f5bb80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.410+0000 7f5bc0103640 1 -- 172.21.15.81:0/295311582 shutdown_connections 2024-06-29T14:37:05.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.410+0000 7f5bc0103640 1 -- 172.21.15.81:0/295311582 wait complete. 2024-06-29T14:37:05.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.412+0000 7f5bc0103640 1 Processor -- start 2024-06-29T14:37:05.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.412+0000 7f5bc0103640 1 -- start start 2024-06-29T14:37:05.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bc0103640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bb80b3200 0x7f5bb80c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bc0103640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb80b5f50 0x7f5bb8159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bc0103640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb80d0230 0x7f5bb8159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bc0103640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb81672d0 con 0x7f5bb80d0230 2024-06-29T14:37:05.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bc0103640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb8167150 con 0x7f5bb80b5f50 2024-06-29T14:37:05.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bc0103640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb8167450 con 0x7f5bb80b3200 2024-06-29T14:37:05.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bbe6a0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb80d0230 0x7f5bb8159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bbe6a0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb80d0230 0x7f5bb8159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39850/0 (socket says 172.21.15.81:39850) 2024-06-29T14:37:05.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bbe6a0640 1 -- 172.21.15.81:0/1141179355 learned_addr learned my addr 172.21.15.81:0/1141179355 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:05.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bbde9f640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bb80b3200 0x7f5bb80c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.413+0000 7f5bbd69e640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb80b5f50 0x7f5bb8159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bbe6a0640 1 -- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bb80b3200 msgr2=0x7f5bb80c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bbe6a0640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bb80b3200 0x7f5bb80c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bbe6a0640 1 -- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb80b5f50 msgr2=0x7f5bb8159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bbe6a0640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb80b5f50 0x7f5bb8159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bbe6a0640 1 -- 172.21.15.81:0/1141179355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb814f220 con 0x7f5bb80d0230 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bbd69e640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb80b5f50 0x7f5bb8159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bbe6a0640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb80d0230 0x7f5bb8159b90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5bb002aea0 tx=0x7f5bb0033f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bae7fc640 1 -- 172.21.15.81:0/1141179355 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0031990 con 0x7f5bb80d0230 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bae7fc640 1 -- 172.21.15.81:0/1141179355 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bb0031c20 con 0x7f5bb80d0230 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bc0103640 1 -- 172.21.15.81:0/1141179355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb8151dc0 con 0x7f5bb80d0230 2024-06-29T14:37:05.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bae7fc640 1 -- 172.21.15.81:0/1141179355 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb00330d0 con 0x7f5bb80d0230 2024-06-29T14:37:05.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.414+0000 7f5bc0103640 1 -- 172.21.15.81:0/1141179355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb806e3d0 con 0x7f5bb80d0230 2024-06-29T14:37:05.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.415+0000 7f5bc0103640 1 -- 172.21.15.81:0/1141179355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb8151b30 con 0x7f5bb80d0230 2024-06-29T14:37:05.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.416+0000 7f5bae7fc640 1 -- 172.21.15.81:0/1141179355 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7f5bb0033270 con 0x7f5bb80d0230 2024-06-29T14:37:05.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.417+0000 7f5bae7fc640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f5b9c076420 0x7f5b9c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.417+0000 7f5bbde9f640 1 -- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f5b9c076420 msgr2=0x7f5b9c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:05.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.417+0000 7f5bae7fc640 1 -- 172.21.15.81:0/1141179355 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6243+0+0 (secure 0 0 0) 0x7f5bb00b8f20 con 0x7f5bb80d0230 2024-06-29T14:37:05.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.417+0000 7f5bbde9f640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f5b9c076420 0x7f5b9c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:05.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.421+0000 7f5bae7fc640 1 -- 172.21.15.81:0/1141179355 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5bb003f030 con 0x7f5bb80d0230 2024-06-29T14:37:05.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.618+0000 7f5bbde9f640 1 -- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f5b9c076420 msgr2=0x7f5b9c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:05.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.618+0000 7f5bbde9f640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f5b9c076420 0x7f5b9c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:05.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.650+0000 7f5bc0103640 1 -- 172.21.15.81:0/1141179355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bb80cc7b0 con 0x7f5bb80d0230 2024-06-29T14:37:05.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.651+0000 7f5bae7fc640 1 -- 172.21.15.81:0/1141179355 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5229 (secure 0 0 0) 0x7f5bb0081d60 con 0x7f5bb80d0230 2024-06-29T14:37:05.648 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:05.648 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":25,"btime":"2024-06-29T14:35:02:255148+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2980523942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2980523942},{"type":"v1","addr":"172.21.15.167:6836","nonce":2980523942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5635,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2317403375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2317403375},{"type":"v1","addr":"172.21.15.167:6837","nonce":2317403375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/2346221458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2346221458},{"type":"v1","addr":"172.21.15.81:6834","nonce":2346221458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:34:58.602103+0000","modified":"2024-06-29T14:35:01.623553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6033":{"gid":6033,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/2669979711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2669979711},{"type":"v1","addr":"172.21.15.81:6835","nonce":2669979711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6033,"qdb_cluster":[6033]},"id":3}]} 2024-06-29T14:37:05.648 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 25 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 -- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f5b9c076420 msgr2=0x7f5b9c078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f5b9c076420 0x7f5b9c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 -- 172.21.15.81:0/1141179355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb80d0230 msgr2=0x7f5bb8159b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb80d0230 0x7f5bb8159b90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5bb002aea0 tx=0x7f5bb0033f70 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 -- 172.21.15.81:0/1141179355 shutdown_connections 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f5b9c076420 0x7f5b9c078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb80d0230 0x7f5bb8159b90 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb80b5f50 0x7f5bb8159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 --2- 172.21.15.81:0/1141179355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bb80b3200 0x7f5bb80c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 -- 172.21.15.81:0/1141179355 >> 172.21.15.81:0/1141179355 conn(0x7f5bb80b3680 msgr2=0x7f5bb80d22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:05.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 -- 172.21.15.81:0/1141179355 shutdown_connections 2024-06-29T14:37:05.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.653+0000 7f5bc0103640 1 -- 172.21.15.81:0/1141179355 wait complete. 2024-06-29T14:37:05.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:05.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.779+0000 7fc34fdee640 1 -- 172.21.15.81:0/2773676003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3480b3200 msgr2=0x7fc3480cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.779+0000 7fc34fdee640 1 --2- 172.21.15.81:0/2773676003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3480b3200 0x7fc3480cc7b0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fc348074b10 tx=0x7fc33c02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.779+0000 7fc34fdee640 1 -- 172.21.15.81:0/2773676003 shutdown_connections 2024-06-29T14:37:05.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.779+0000 7fc34fdee640 1 --2- 172.21.15.81:0/2773676003 >> v2:172.21.15.167:3300/0 conn(0x7fc3480cccf0 0x7fc3480d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.779+0000 7fc34fdee640 1 --2- 172.21.15.81:0/2773676003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3480b3200 0x7fc3480cc7b0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.779+0000 7fc34fdee640 1 -- 172.21.15.81:0/2773676003 >> 172.21.15.81:0/2773676003 conn(0x7fc3480b3680 msgr2=0x7fc3480b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:05.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.779+0000 7fc34fdee640 1 -- 172.21.15.81:0/2773676003 shutdown_connections 2024-06-29T14:37:05.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.779+0000 7fc34fdee640 1 -- 172.21.15.81:0/2773676003 wait complete. 2024-06-29T14:37:05.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.782+0000 7fc34fdee640 1 Processor -- start 2024-06-29T14:37:05.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.782+0000 7fc34fdee640 1 -- start start 2024-06-29T14:37:05.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34fdee640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3480b3200 0x7fc3480c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34fdee640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3480cccf0 0x7fc3481dd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34fdee640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc3480d0770 0x7fc3480c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34fdee640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc3480cedb0 con 0x7fc3480cccf0 2024-06-29T14:37:05.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34fdee640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc3480cec30 con 0x7fc3480b3200 2024-06-29T14:37:05.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34fdee640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc3480cef30 con 0x7fc3480d0770 2024-06-29T14:37:05.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34d389640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3480cccf0 0x7fc3481dd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34d389640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3480cccf0 0x7fc3481dd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39864/0 (socket says 172.21.15.81:39864) 2024-06-29T14:37:05.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34d389640 1 -- 172.21.15.81:0/715052867 learned_addr learned my addr 172.21.15.81:0/715052867 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34db8a640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3480b3200 0x7fc3480c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.784+0000 7fc34e38b640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc3480d0770 0x7fc3480c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc34d389640 1 -- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc3480d0770 msgr2=0x7fc3480c7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc34d389640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc3480d0770 0x7fc3480c7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc34d389640 1 -- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3480b3200 msgr2=0x7fc3480c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc34d389640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3480b3200 0x7fc3480c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc34d389640 1 -- 172.21.15.81:0/715052867 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3480ca4a0 con 0x7fc3480cccf0 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc34d389640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3480cccf0 0x7fc3481dd850 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc34400e790 tx=0x7fc34400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc33a7fc640 1 -- 172.21.15.81:0/715052867 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34400eea0 con 0x7fc3480cccf0 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc33a7fc640 1 -- 172.21.15.81:0/715052867 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc34400ccc0 con 0x7fc3480cccf0 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc34fdee640 1 -- 172.21.15.81:0/715052867 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3481e1850 con 0x7fc3480cccf0 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc33a7fc640 1 -- 172.21.15.81:0/715052867 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc344005280 con 0x7fc3480cccf0 2024-06-29T14:37:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.785+0000 7fc34fdee640 1 -- 172.21.15.81:0/715052867 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3481e1e10 con 0x7fc3480cccf0 2024-06-29T14:37:05.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.786+0000 7fc34fdee640 1 -- 172.21.15.81:0/715052867 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3480cc7b0 con 0x7fc3480cccf0 2024-06-29T14:37:05.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.787+0000 7fc33a7fc640 1 -- 172.21.15.81:0/715052867 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7fc344005420 con 0x7fc3480cccf0 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.787+0000 7fc33a7fc640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fc32c0763f0 0x7fc32c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.787+0000 7fc34db8a640 1 -- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fc32c0763f0 msgr2=0x7fc32c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.787+0000 7fc33a7fc640 1 -- 172.21.15.81:0/715052867 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6243+0+0 (secure 0 0 0) 0x7fc3440983e0 con 0x7fc3480cccf0 2024-06-29T14:37:05.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.787+0000 7fc34db8a640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fc32c0763f0 0x7fc32c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:05.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.791+0000 7fc33a7fc640 1 -- 172.21.15.81:0/715052867 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc344061220 con 0x7fc3480cccf0 2024-06-29T14:37:05.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.981+0000 7fc34fdee640 1 -- 172.21.15.81:0/715052867 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3480d1b10 con 0x7fc3480cccf0 2024-06-29T14:37:05.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.982+0000 7fc33a7fc640 1 -- 172.21.15.81:0/715052867 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) ==== 74+0+17429 (secure 0 0 0) 0x7fc344065520 con 0x7fc3480cccf0 2024-06-29T14:37:05.979 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:05.980 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":65,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:05.230637+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:34:56.911140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-06-29T14:34:57.555296+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.12","mappings":[{"from":0,"to":5}]},{"pgid":"7.2c","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/1090782941":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6832/1090782941":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/1626573527":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/799531669":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/2483549331":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/2831660161":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/3380278063":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/605629739":"2024-06-30T14:35:10.908137+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:05.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 -- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fc32c0763f0 msgr2=0x7fc32c078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:05.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fc32c0763f0 0x7fc32c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 -- 172.21.15.81:0/715052867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3480cccf0 msgr2=0x7fc3481dd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:05.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3480cccf0 0x7fc3481dd850 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc34400e790 tx=0x7fc34400ec50 comp rx=0 tx=0).stop 2024-06-29T14:37:05.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 -- 172.21.15.81:0/715052867 shutdown_connections 2024-06-29T14:37:05.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fc32c0763f0 0x7fc32c078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc3480d0770 0x7fc3480c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3480cccf0 0x7fc3481dd850 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 --2- 172.21.15.81:0/715052867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3480b3200 0x7fc3480c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:05.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 -- 172.21.15.81:0/715052867 >> 172.21.15.81:0/715052867 conn(0x7fc3480b3680 msgr2=0x7fc348070150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:05.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 -- 172.21.15.81:0/715052867 shutdown_connections 2024-06-29T14:37:05.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:05.984+0000 7fc34fdee640 1 -- 172.21.15.81:0/715052867 wait complete. 2024-06-29T14:37:05.992 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:34:56.911140+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:34:57.555296+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 65, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:05.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:06.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.114+0000 7f0f87bbd640 1 -- 172.21.15.81:0/791624324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f800b3200 msgr2=0x7f0f800cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.114+0000 7f0f87bbd640 1 --2- 172.21.15.81:0/791624324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f800b3200 0x7f0f800cc7b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f0f800c5590 tx=0x7f0f7002a4a0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.114+0000 7f0f87bbd640 1 -- 172.21.15.81:0/791624324 shutdown_connections 2024-06-29T14:37:06.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.114+0000 7f0f87bbd640 1 --2- 172.21.15.81:0/791624324 >> v2:172.21.15.167:3300/0 conn(0x7f0f800d0940 0x7f0f800b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.114+0000 7f0f87bbd640 1 --2- 172.21.15.81:0/791624324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f800b3200 0x7f0f800cc7b0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.114+0000 7f0f87bbd640 1 -- 172.21.15.81:0/791624324 >> 172.21.15.81:0/791624324 conn(0x7f0f800b3680 msgr2=0x7f0f800b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:06.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.114+0000 7f0f87bbd640 1 -- 172.21.15.81:0/791624324 shutdown_connections 2024-06-29T14:37:06.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.114+0000 7f0f87bbd640 1 -- 172.21.15.81:0/791624324 wait complete. 2024-06-29T14:37:06.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.116+0000 7f0f87bbd640 1 Processor -- start 2024-06-29T14:37:06.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.116+0000 7f0f87bbd640 1 -- start start 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f87bbd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f800b3200 0x7f0f801f41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f87bbd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f800cccf0 0x7f0f8006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f87bbd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f801e2ea0 0x7f0f801e32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f87bbd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0f800b5bb0 con 0x7f0f801e2ea0 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f87bbd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0f800b5a30 con 0x7f0f800b3200 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f87bbd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0f800b5d30 con 0x7f0f800cccf0 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f8615a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f801e2ea0 0x7f0f801e32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f8615a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f801e2ea0 0x7f0f801e32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39886/0 (socket says 172.21.15.81:39886) 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f8615a640 1 -- 172.21.15.81:0/3372786688 learned_addr learned my addr 172.21.15.81:0/3372786688 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f85959640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f800b3200 0x7f0f801f41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.117+0000 7f0f85158640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f800cccf0 0x7f0f8006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f8615a640 1 -- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f800cccf0 msgr2=0x7f0f8006ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f8615a640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f800cccf0 0x7f0f8006ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f8615a640 1 -- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f800b3200 msgr2=0x7f0f801f41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f8615a640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f800b3200 0x7f0f801f41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f8615a640 1 -- 172.21.15.81:0/3372786688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f801e37f0 con 0x7f0f801e2ea0 2024-06-29T14:37:06.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f8615a640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f801e2ea0 0x7f0f801e32b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800d9e0 tx=0x7f0f7800dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:06.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f767fc640 1 -- 172.21.15.81:0/3372786688 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78017070 con 0x7f0f801e2ea0 2024-06-29T14:37:06.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f767fc640 1 -- 172.21.15.81:0/3372786688 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f7800be30 con 0x7f0f801e2ea0 2024-06-29T14:37:06.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f87bbd640 1 -- 172.21.15.81:0/3372786688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f8006d8e0 con 0x7f0f801e2ea0 2024-06-29T14:37:06.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f767fc640 1 -- 172.21.15.81:0/3372786688 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78003bb0 con 0x7f0f801e2ea0 2024-06-29T14:37:06.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.118+0000 7f0f87bbd640 1 -- 172.21.15.81:0/3372786688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f8006dea0 con 0x7f0f801e2ea0 2024-06-29T14:37:06.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.119+0000 7f0f87bbd640 1 -- 172.21.15.81:0/3372786688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f800cc7b0 con 0x7f0f801e2ea0 2024-06-29T14:37:06.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.120+0000 7f0f767fc640 1 -- 172.21.15.81:0/3372786688 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7f0f78010460 con 0x7f0f801e2ea0 2024-06-29T14:37:06.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.120+0000 7f0f767fc640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0f640763f0 0x7f0f64078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.120+0000 7f0f85959640 1 -- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0f640763f0 msgr2=0x7f0f64078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:06.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.120+0000 7f0f767fc640 1 -- 172.21.15.81:0/3372786688 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6243+0+0 (secure 0 0 0) 0x7f0f78097ee0 con 0x7f0f801e2ea0 2024-06-29T14:37:06.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.120+0000 7f0f85959640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0f640763f0 0x7f0f64078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:06.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.129+0000 7f0f767fc640 1 -- 172.21.15.81:0/3372786688 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f801e37f0 con 0x7f0f801e2ea0 2024-06-29T14:37:06.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.322+0000 7f0f85959640 1 -- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0f640763f0 msgr2=0x7f0f64078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:06.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.322+0000 7f0f85959640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0f640763f0 0x7f0f64078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:06.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.330+0000 7f0f87bbd640 1 -- 172.21.15.81:0/3372786688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0f800c8be0 con 0x7f0f801e2ea0 2024-06-29T14:37:06.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.331+0000 7f0f767fc640 1 -- 172.21.15.81:0/3372786688 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) ==== 93+0+10 (secure 0 0 0) 0x7f0f78060dc0 con 0x7f0f801e2ea0 2024-06-29T14:37:06.328 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 -- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0f640763f0 msgr2=0x7f0f64078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0f640763f0 0x7f0f64078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 -- 172.21.15.81:0/3372786688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f801e2ea0 msgr2=0x7f0f801e32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f801e2ea0 0x7f0f801e32b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800d9e0 tx=0x7f0f7800dea0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 -- 172.21.15.81:0/3372786688 shutdown_connections 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7f0f640763f0 0x7f0f64078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f801e2ea0 0x7f0f801e32b0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f800cccf0 0x7f0f8006ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 --2- 172.21.15.81:0/3372786688 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f800b3200 0x7f0f801f41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 -- 172.21.15.81:0/3372786688 >> 172.21.15.81:0/3372786688 conn(0x7f0f800b3680 msgr2=0x7f0f801daa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 -- 172.21.15.81:0/3372786688 shutdown_connections 2024-06-29T14:37:06.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.333+0000 7f0f87bbd640 1 -- 172.21.15.81:0/3372786688 wait complete. 2024-06-29T14:37:06.341 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:37:06.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.463+0000 7ff3ac539640 1 -- 172.21.15.81:0/1278404944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3a40cfd10 msgr2=0x7ff3a40ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.463+0000 7ff3ac539640 1 --2- 172.21.15.81:0/1278404944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3a40cfd10 0x7ff3a40ce7b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff39c005fd0 tx=0x7ff39c02a940 comp rx=0 tx=0).stop 2024-06-29T14:37:06.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.463+0000 7ff3ac539640 1 -- 172.21.15.81:0/1278404944 shutdown_connections 2024-06-29T14:37:06.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.463+0000 7ff3ac539640 1 --2- 172.21.15.81:0/1278404944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3a40cfd10 0x7ff3a40ce7b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.463+0000 7ff3ac539640 1 -- 172.21.15.81:0/1278404944 >> 172.21.15.81:0/1278404944 conn(0x7ff3a40b35a0 msgr2=0x7ff3a40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:06.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.463+0000 7ff3ac539640 1 -- 172.21.15.81:0/1278404944 shutdown_connections 2024-06-29T14:37:06.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.463+0000 7ff3ac539640 1 -- 172.21.15.81:0/1278404944 wait complete. 2024-06-29T14:37:06.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.465+0000 7ff3ac539640 1 Processor -- start 2024-06-29T14:37:06.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.465+0000 7ff3ac539640 1 -- start start 2024-06-29T14:37:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3ac539640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3a40ce3a0 0x7ff3a409d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3ac539640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3a40cfd10 0x7ff3a414c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3ac539640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3a40c5560 0x7ff3a4165790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3ac539640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a40b6df0 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3ac539640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a40b28c0 con 0x7ff3a40cfd10 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3ac539640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a40b6f70 con 0x7ff3a40c5560 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3aa2d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3a40ce3a0 0x7ff3a409d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3aa2d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3a40ce3a0 0x7ff3a409d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39890/0 (socket says 172.21.15.81:39890) 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3aa2d5640 1 -- 172.21.15.81:0/522707027 learned_addr learned my addr 172.21.15.81:0/522707027 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3aaad6640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3a40c5560 0x7ff3a4165790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3a9ad4640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3a40cfd10 0x7ff3a414c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3aa2d5640 1 -- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3a40c5560 msgr2=0x7ff3a4165790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3aa2d5640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3a40c5560 0x7ff3a4165790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3aa2d5640 1 -- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3a40cfd10 msgr2=0x7ff3a414c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3aa2d5640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3a40cfd10 0x7ff3a414c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.466+0000 7ff3aa2d5640 1 -- 172.21.15.81:0/522707027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a414cfb0 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.467+0000 7ff3aa2d5640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3a40ce3a0 0x7ff3a409d000 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000c3d0 tx=0x7ff3a000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.467+0000 7ff39affd640 1 -- 172.21.15.81:0/522707027 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0017070 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.467+0000 7ff3ac539640 1 -- 172.21.15.81:0/522707027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a414d290 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.467+0000 7ff39affd640 1 -- 172.21.15.81:0/522707027 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3a00053c0 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.467+0000 7ff39affd640 1 -- 172.21.15.81:0/522707027 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0003850 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.467+0000 7ff3ac539640 1 -- 172.21.15.81:0/522707027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a414d850 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.468+0000 7ff3ac539640 1 -- 172.21.15.81:0/522707027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a40d11a0 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.469+0000 7ff39affd640 1 -- 172.21.15.81:0/522707027 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7ff3a001a070 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.470+0000 7ff39affd640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7ff3880763f0 0x7ff388078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.470+0000 7ff3a9ad4640 1 -- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7ff3880763f0 msgr2=0x7ff388078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:06.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.470+0000 7ff39affd640 1 -- 172.21.15.81:0/522707027 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6227+0+0 (secure 0 0 0) 0x7ff3a0096f50 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.470+0000 7ff3a9ad4640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7ff3880763f0 0x7ff388078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:06.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.474+0000 7ff39affd640 1 -- 172.21.15.81:0/522707027 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff3a40d11a0 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.660+0000 7ff3ac539640 1 -- 172.21.15.81:0/522707027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff3a40b0cf0 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.660+0000 7ff39affd640 1 -- 172.21.15.81:0/522707027 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v66) ==== 104+0+11 (secure 0 0 0) 0x7ff3a40b0cf0 con 0x7ff3a40ce3a0 2024-06-29T14:37:06.658 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:37:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 -- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7ff3880763f0 msgr2=0x7ff388078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7ff3880763f0 0x7ff388078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 -- 172.21.15.81:0/522707027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3a40ce3a0 msgr2=0x7ff3a409d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3a40ce3a0 0x7ff3a409d000 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000c3d0 tx=0x7ff3a000c890 comp rx=0 tx=0).stop 2024-06-29T14:37:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 -- 172.21.15.81:0/522707027 shutdown_connections 2024-06-29T14:37:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7ff3880763f0 0x7ff388078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3a40c5560 0x7ff3a4165790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3a40cfd10 0x7ff3a414c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 --2- 172.21.15.81:0/522707027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3a40ce3a0 0x7ff3a409d000 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 -- 172.21.15.81:0/522707027 >> 172.21.15.81:0/522707027 conn(0x7ff3a40b35a0 msgr2=0x7ff3a40ae880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 -- 172.21.15.81:0/522707027 shutdown_connections 2024-06-29T14:37:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.662+0000 7ff3ac539640 1 -- 172.21.15.81:0/522707027 wait complete. 2024-06-29T14:37:06.671 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:37:06.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.791+0000 7fb111f68640 1 -- 172.21.15.81:0/3596995353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb10c0b9040 msgr2=0x7fb10c0d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.791+0000 7fb111f68640 1 --2- 172.21.15.81:0/3596995353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb10c0b9040 0x7fb10c0d0480 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0a1b70 tx=0x7fb10002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.791+0000 7fb111f68640 1 -- 172.21.15.81:0/3596995353 shutdown_connections 2024-06-29T14:37:06.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.791+0000 7fb111f68640 1 --2- 172.21.15.81:0/3596995353 >> v2:172.21.15.167:3300/0 conn(0x7fb10c0b5250 0x7fb10c0d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.791+0000 7fb111f68640 1 --2- 172.21.15.81:0/3596995353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb10c0b9040 0x7fb10c0d0480 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.791+0000 7fb111f68640 1 -- 172.21.15.81:0/3596995353 >> 172.21.15.81:0/3596995353 conn(0x7fb10c09cf30 msgr2=0x7fb10c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:06.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.791+0000 7fb111f68640 1 -- 172.21.15.81:0/3596995353 shutdown_connections 2024-06-29T14:37:06.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.791+0000 7fb111f68640 1 -- 172.21.15.81:0/3596995353 wait complete. 2024-06-29T14:37:06.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.793+0000 7fb111f68640 1 Processor -- start 2024-06-29T14:37:06.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.793+0000 7fb111f68640 1 -- start start 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.794+0000 7fb111f68640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb10c0b5250 0x7fb10c1f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.794+0000 7fb111f68640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb10c0b7290 0x7fb10c1f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.794+0000 7fb111f68640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb10c0b9040 0x7fb10c0a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.794+0000 7fb111f68640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c0d2e20 con 0x7fb10c0b5250 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.794+0000 7fb111f68640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c0d2ca0 con 0x7fb10c0b7290 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.794+0000 7fb111f68640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb10c0b7db0 con 0x7fb10c0b9040 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.794+0000 7fb10b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb10c0b5250 0x7fb10c1f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.794+0000 7fb10b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb10c0b5250 0x7fb10c1f8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39916/0 (socket says 172.21.15.81:39916) 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.794+0000 7fb10b7fe640 1 -- 172.21.15.81:0/2943741202 learned_addr learned my addr 172.21.15.81:0/2943741202 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb10affd640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb10c0b7290 0x7fb10c1f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb10bfff640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb10c0b9040 0x7fb10c0a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb10b7fe640 1 -- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb10c0b9040 msgr2=0x7fb10c0a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb10b7fe640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb10c0b9040 0x7fb10c0a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb10b7fe640 1 -- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb10c0b7290 msgr2=0x7fb10c1f47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb10b7fe640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb10c0b7290 0x7fb10c1f47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb10b7fe640 1 -- 172.21.15.81:0/2943741202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c1f94f0 con 0x7fb10c0b5250 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb10bfff640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb10c0b9040 0x7fb10c0a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb10b7fe640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb10c0b5250 0x7fb10c1f8fb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb100000c00 tx=0x7fb100003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb0f3fff640 1 -- 172.21.15.81:0/2943741202 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10003e030 con 0x7fb10c0b5250 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb111f68640 1 -- 172.21.15.81:0/2943741202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10c1e6cf0 con 0x7fb10c0b5250 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb0f3fff640 1 -- 172.21.15.81:0/2943741202 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb100003e00 con 0x7fb10c0b5250 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb0f3fff640 1 -- 172.21.15.81:0/2943741202 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb100035040 con 0x7fb10c0b5250 2024-06-29T14:37:06.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.795+0000 7fb111f68640 1 -- 172.21.15.81:0/2943741202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10c1e72b0 con 0x7fb10c0b5250 2024-06-29T14:37:06.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.797+0000 7fb111f68640 1 -- 172.21.15.81:0/2943741202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb10c0b82f0 con 0x7fb10c0b5250 2024-06-29T14:37:06.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.797+0000 7fb0f3fff640 1 -- 172.21.15.81:0/2943741202 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 26) ==== 98315+0+0 (secure 0 0 0) 0x7fb100041030 con 0x7fb10c0b5250 2024-06-29T14:37:06.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.797+0000 7fb0f3fff640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb0f80763f0 0x7fb0f8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:06.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.798+0000 7fb10affd640 1 -- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb0f80763f0 msgr2=0x7fb0f8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2493228728 2024-06-29T14:37:06.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.798+0000 7fb0f3fff640 1 -- 172.21.15.81:0/2943741202 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6227+0+0 (secure 0 0 0) 0x7fb1000b8440 con 0x7fb10c0b5250 2024-06-29T14:37:06.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.798+0000 7fb10affd640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb0f80763f0 0x7fb0f8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:06.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.802+0000 7fb0f3fff640 1 -- 172.21.15.81:0/2943741202 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb100081210 con 0x7fb10c0b5250 2024-06-29T14:37:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.916+0000 7fb0f3fff640 1 -- 172.21.15.81:0/2943741202 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 27) ==== 97912+0+0 (secure 0 0 0) 0x7fb1000b6750 con 0x7fb10c0b5250 2024-06-29T14:37:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.916+0000 7fb0f3fff640 1 -- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb0f80763f0 msgr2=0x7fb0f8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:06.916+0000 7fb0f3fff640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb0f80763f0 0x7fb0f8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:06.966 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:06.968+0000 7f02bab82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.966 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:06.968+0000 7f02bab82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.966 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:06.968+0000 7f02bab82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.966 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:06.968+0000 7f02bab82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.966 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:06.968+0000 7f02bab82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:06.968+0000 7f02b7b7c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:06.968+0000 7f02b7b7c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:06.968+0000 7f02b7b7c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:06.968+0000 7f02b7b7c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:06.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:06.968+0000 7f02b7b7c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:07.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.015+0000 7fb111f68640 1 -- 172.21.15.81:0/2943741202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb10c0b8730 con 0x7fb10c0b5250 2024-06-29T14:37:07.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.016+0000 7fb0f3fff640 1 -- 172.21.15.81:0/2943741202 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7fb10009b360 con 0x7fb10c0b5250 2024-06-29T14:37:07.013 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:37:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.018+0000 7fb111f68640 1 -- 172.21.15.81:0/2943741202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb10c0b5250 msgr2=0x7fb10c1f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.018+0000 7fb111f68640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb10c0b5250 0x7fb10c1f8fb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb100000c00 tx=0x7fb100003910 comp rx=0 tx=0).stop 2024-06-29T14:37:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.018+0000 7fb111f68640 1 -- 172.21.15.81:0/2943741202 shutdown_connections 2024-06-29T14:37:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.018+0000 7fb111f68640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:6832/2493228728,v1:172.21.15.167:6833/2493228728] conn(0x7fb0f80763f0 0x7fb0f8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.018+0000 7fb111f68640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb10c0b9040 0x7fb10c0a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.018+0000 7fb111f68640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb10c0b7290 0x7fb10c1f47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.018+0000 7fb111f68640 1 --2- 172.21.15.81:0/2943741202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb10c0b5250 0x7fb10c1f8fb0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.018+0000 7fb111f68640 1 -- 172.21.15.81:0/2943741202 >> 172.21.15.81:0/2943741202 conn(0x7fb10c09cf30 msgr2=0x7fb10c160320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.018+0000 7fb111f68640 1 -- 172.21.15.81:0/2943741202 shutdown_connections 2024-06-29T14:37:07.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.018+0000 7fb111f68640 1 -- 172.21.15.81:0/2943741202 wait complete. 2024-06-29T14:37:07.027 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:07.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.143+0000 7f0403a74640 1 -- 172.21.15.81:0/422937484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03fc0b3200 msgr2=0x7f03fc0cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.143+0000 7f0403a74640 1 --2- 172.21.15.81:0/422937484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03fc0b3200 0x7f03fc0cc7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009830 tx=0x7f03ec02a510 comp rx=0 tx=0).stop 2024-06-29T14:37:07.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.143+0000 7f0403a74640 1 -- 172.21.15.81:0/422937484 shutdown_connections 2024-06-29T14:37:07.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.143+0000 7f0403a74640 1 --2- 172.21.15.81:0/422937484 >> v2:172.21.15.167:3300/0 conn(0x7f03fc0cccf0 0x7f03fc0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.143+0000 7f0403a74640 1 --2- 172.21.15.81:0/422937484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03fc0b3200 0x7f03fc0cc7b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.143+0000 7f0403a74640 1 -- 172.21.15.81:0/422937484 >> 172.21.15.81:0/422937484 conn(0x7f03fc0b3680 msgr2=0x7f03fc0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.143+0000 7f0403a74640 1 -- 172.21.15.81:0/422937484 shutdown_connections 2024-06-29T14:37:07.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.143+0000 7f0403a74640 1 -- 172.21.15.81:0/422937484 wait complete. 2024-06-29T14:37:07.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.145+0000 7f0403a74640 1 Processor -- start 2024-06-29T14:37:07.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.145+0000 7f0403a74640 1 -- start start 2024-06-29T14:37:07.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.146+0000 7f0403a74640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03fc0b3200 0x7f03fc09ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0403a74640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03fc0cccf0 0x7f03fc1558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0403a74640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03fc0d0770 0x7f03fc1f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0403a74640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f03fc0cedb0 con 0x7f03fc0b3200 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0403a74640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f03fc0cec30 con 0x7f03fc0cccf0 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0403a74640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f03fc0cef30 con 0x7f03fc0d0770 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0401810640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03fc0b3200 0x7f03fc09ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0401810640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03fc0b3200 0x7f03fc09ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39930/0 (socket says 172.21.15.81:39930) 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0401810640 1 -- 172.21.15.81:0/2288635200 learned_addr learned my addr 172.21.15.81:0/2288635200 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f040100f640 1 --2- 172.21.15.81:0/2288635200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03fc0cccf0 0x7f03fc1558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0402011640 1 --2- 172.21.15.81:0/2288635200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03fc0d0770 0x7f03fc1f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0401810640 1 -- 172.21.15.81:0/2288635200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03fc0d0770 msgr2=0x7f03fc1f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0401810640 1 --2- 172.21.15.81:0/2288635200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03fc0d0770 0x7f03fc1f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0401810640 1 -- 172.21.15.81:0/2288635200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03fc0cccf0 msgr2=0x7f03fc1558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0401810640 1 --2- 172.21.15.81:0/2288635200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03fc0cccf0 0x7f03fc1558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0401810640 1 -- 172.21.15.81:0/2288635200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03fc1e2cb0 con 0x7f03fc0b3200 2024-06-29T14:37:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0401810640 1 --2- 172.21.15.81:0/2288635200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03fc0b3200 0x7f03fc09ee30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f03ec02ab00 tx=0x7f03ec033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f03fa7fc640 1 -- 172.21.15.81:0/2288635200 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec005c90 con 0x7f03fc0b3200 2024-06-29T14:37:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f03fa7fc640 1 -- 172.21.15.81:0/2288635200 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03ec031e50 con 0x7f03fc0b3200 2024-06-29T14:37:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f03fa7fc640 1 -- 172.21.15.81:0/2288635200 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ec005260 con 0x7f03fc0b3200 2024-06-29T14:37:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0403a74640 1 -- 172.21.15.81:0/2288635200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03fc1e2f90 con 0x7f03fc0b3200 2024-06-29T14:37:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.147+0000 7f0403a74640 1 -- 172.21.15.81:0/2288635200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03fc1e3550 con 0x7f03fc0b3200 2024-06-29T14:37:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.148+0000 7f0403a74640 1 -- 172.21.15.81:0/2288635200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03fc0cc7b0 con 0x7f03fc0b3200 2024-06-29T14:37:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.149+0000 7f03fa7fc640 1 -- 172.21.15.81:0/2288635200 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 27) ==== 97912+0+0 (secure 0 0 0) 0x7f03ec005400 con 0x7f03fc0b3200 2024-06-29T14:37:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.150+0000 7f03fa7fc640 1 -- 172.21.15.81:0/2288635200 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6527+0+0 (secure 0 0 0) 0x7f03ec0b84a0 con 0x7f03fc0b3200 2024-06-29T14:37:07.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.154+0000 7f03fa7fc640 1 -- 172.21.15.81:0/2288635200 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f03ec03f030 con 0x7f03fc0b3200 2024-06-29T14:37:07.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.373+0000 7f0403a74640 1 -- 172.21.15.81:0/2288635200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03fc1e5660 con 0x7f03fc0b3200 2024-06-29T14:37:07.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.374+0000 7f03fa7fc640 1 -- 172.21.15.81:0/2288635200 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5229 (secure 0 0 0) 0x7f03ec081850 con 0x7f03fc0b3200 2024-06-29T14:37:07.372 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:07.372 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":25,"btime":"2024-06-29T14:35:02:255148+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2980523942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2980523942},{"type":"v1","addr":"172.21.15.167:6836","nonce":2980523942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5635,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2317403375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2317403375},{"type":"v1","addr":"172.21.15.167:6837","nonce":2317403375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/2346221458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2346221458},{"type":"v1","addr":"172.21.15.81:6834","nonce":2346221458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:34:58.602103+0000","modified":"2024-06-29T14:35:01.623553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6033":{"gid":6033,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/2669979711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2669979711},{"type":"v1","addr":"172.21.15.81:6835","nonce":2669979711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6033,"qdb_cluster":[6033]},"id":3}]} 2024-06-29T14:37:07.372 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 25 2024-06-29T14:37:07.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.376+0000 7f0403a74640 1 -- 172.21.15.81:0/2288635200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03fc0b3200 msgr2=0x7f03fc09ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.376+0000 7f0403a74640 1 --2- 172.21.15.81:0/2288635200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03fc0b3200 0x7f03fc09ee30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f03ec02ab00 tx=0x7f03ec033c20 comp rx=0 tx=0).stop 2024-06-29T14:37:07.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.376+0000 7f0403a74640 1 -- 172.21.15.81:0/2288635200 shutdown_connections 2024-06-29T14:37:07.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.376+0000 7f0403a74640 1 --2- 172.21.15.81:0/2288635200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03fc0d0770 0x7f03fc1f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.376+0000 7f0403a74640 1 --2- 172.21.15.81:0/2288635200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03fc0cccf0 0x7f03fc1558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.376+0000 7f0403a74640 1 --2- 172.21.15.81:0/2288635200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03fc0b3200 0x7f03fc09ee30 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.376+0000 7f0403a74640 1 -- 172.21.15.81:0/2288635200 >> 172.21.15.81:0/2288635200 conn(0x7f03fc0b3680 msgr2=0x7f03fc1e8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.376+0000 7f0403a74640 1 -- 172.21.15.81:0/2288635200 shutdown_connections 2024-06-29T14:37:07.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.376+0000 7f0403a74640 1 -- 172.21.15.81:0/2288635200 wait complete. 2024-06-29T14:37:07.384 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:07.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.501+0000 7f78404a5640 1 -- 172.21.15.81:0/2686095887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7824004d40 msgr2=0x7f7824005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.501+0000 7f78404a5640 1 --2- 172.21.15.81:0/2686095887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7824004d40 0x7f7824005150 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f78380b61c0 tx=0x7f7824005830 comp rx=0 tx=0).stop 2024-06-29T14:37:07.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.501+0000 7f783ea42640 1 -- 172.21.15.81:0/2686095887 reap_dead start 2024-06-29T14:37:07.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.501+0000 7f78404a5640 1 -- 172.21.15.81:0/2686095887 shutdown_connections 2024-06-29T14:37:07.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.501+0000 7f78404a5640 1 -- 172.21.15.81:0/2686095887 >> 172.21.15.81:0/2686095887 conn(0x7f7838072950 msgr2=0x7f78380b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.501+0000 7f78404a5640 1 -- 172.21.15.81:0/2686095887 shutdown_connections 2024-06-29T14:37:07.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.501+0000 7f78404a5640 1 -- 172.21.15.81:0/2686095887 wait complete. 2024-06-29T14:37:07.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.503+0000 7f78404a5640 1 Processor -- start 2024-06-29T14:37:07.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.503+0000 7f78404a5640 1 -- start start 2024-06-29T14:37:07.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.504+0000 7f78404a5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7824004d40 0x7f7838155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.504+0000 7f78404a5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7838154e00 0x7f78381523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.504+0000 7f78404a5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f783809d640 0x7f783809da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.504+0000 7f78404a5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f783815b010 con 0x7f783809d640 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.504+0000 7f78404a5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f783806d150 con 0x7f7824004d40 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.504+0000 7f78404a5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f783815b190 con 0x7f7838154e00 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783ea42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f783809d640 0x7f783809da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783ea42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f783809d640 0x7f783809da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39952/0 (socket says 172.21.15.81:39952) 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783ea42640 1 -- 172.21.15.81:0/3360627344 learned_addr learned my addr 172.21.15.81:0/3360627344 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783da40640 1 --2- 172.21.15.81:0/3360627344 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7838154e00 0x7f78381523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783e241640 1 --2- 172.21.15.81:0/3360627344 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7824004d40 0x7f7838155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783ea42640 1 -- 172.21.15.81:0/3360627344 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7838154e00 msgr2=0x7f78381523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783ea42640 1 --2- 172.21.15.81:0/3360627344 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7838154e00 0x7f78381523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783ea42640 1 -- 172.21.15.81:0/3360627344 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7824004d40 msgr2=0x7f7838155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783ea42640 1 --2- 172.21.15.81:0/3360627344 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7824004d40 0x7f7838155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783ea42640 1 -- 172.21.15.81:0/3360627344 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783809df90 con 0x7f783809d640 2024-06-29T14:37:07.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783e241640 1 --2- 172.21.15.81:0/3360627344 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7824004d40 0x7f7838155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:07.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f783ea42640 1 --2- 172.21.15.81:0/3360627344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f783809d640 0x7f783809da50 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7830004bb0 tx=0x7f7830002100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:07.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f782effd640 1 -- 172.21.15.81:0/3360627344 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783002b9b0 con 0x7f783809d640 2024-06-29T14:37:07.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f782effd640 1 -- 172.21.15.81:0/3360627344 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f783002bb50 con 0x7f783809d640 2024-06-29T14:37:07.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f78404a5640 1 -- 172.21.15.81:0/3360627344 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78381538a0 con 0x7f783809d640 2024-06-29T14:37:07.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f782effd640 1 -- 172.21.15.81:0/3360627344 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7830006070 con 0x7f783809d640 2024-06-29T14:37:07.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.505+0000 7f78404a5640 1 -- 172.21.15.81:0/3360627344 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7838153e60 con 0x7f783809d640 2024-06-29T14:37:07.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.506+0000 7f78404a5640 1 -- 172.21.15.81:0/3360627344 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f783806f140 con 0x7f783809d640 2024-06-29T14:37:07.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.507+0000 7f782effd640 1 -- 172.21.15.81:0/3360627344 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 27) ==== 97912+0+0 (secure 0 0 0) 0x7f783003d030 con 0x7f783809d640 2024-06-29T14:37:07.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.507+0000 7f782effd640 1 -- 172.21.15.81:0/3360627344 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6527+0+0 (secure 0 0 0) 0x7f78300b7170 con 0x7f783809d640 2024-06-29T14:37:07.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.512+0000 7f782effd640 1 -- 172.21.15.81:0/3360627344 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f783809df90 con 0x7f783809d640 2024-06-29T14:37:07.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.691+0000 7f78404a5640 1 -- 172.21.15.81:0/3360627344 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7838155d00 con 0x7f783809d640 2024-06-29T14:37:07.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.692+0000 7f782effd640 1 -- 172.21.15.81:0/3360627344 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+17848 (secure 0 0 0) 0x7f7830080b60 con 0x7f783809d640 2024-06-29T14:37:07.690 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:07.690 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":67,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:06.907868+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:34:56.911140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-06-29T14:34:57.555296+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.12","mappings":[{"from":0,"to":5}]},{"pgid":"7.2c","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/375340026":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/3632236021":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/1756060870":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/1790794665":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/183639693":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/3418120263":"2024-06-30T14:37:06.907833+0000","172.21.15.167:6833/1090782941":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6832/1090782941":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/1626573527":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6833/2493228728":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/799531669":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/2483549331":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6832/2493228728":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/2831660161":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/3380278063":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/605629739":"2024-06-30T14:35:10.908137+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:07.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.694+0000 7f78404a5640 1 -- 172.21.15.81:0/3360627344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f783809d640 msgr2=0x7f783809da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.694+0000 7f78404a5640 1 --2- 172.21.15.81:0/3360627344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f783809d640 0x7f783809da50 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7830004bb0 tx=0x7f7830002100 comp rx=0 tx=0).stop 2024-06-29T14:37:07.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.694+0000 7f78404a5640 1 -- 172.21.15.81:0/3360627344 shutdown_connections 2024-06-29T14:37:07.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.694+0000 7f78404a5640 1 --2- 172.21.15.81:0/3360627344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f783809d640 0x7f783809da50 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.694+0000 7f78404a5640 1 --2- 172.21.15.81:0/3360627344 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7838154e00 0x7f78381523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.694+0000 7f78404a5640 1 --2- 172.21.15.81:0/3360627344 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7824004d40 0x7f7838155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.694+0000 7f78404a5640 1 -- 172.21.15.81:0/3360627344 >> 172.21.15.81:0/3360627344 conn(0x7f7838072950 msgr2=0x7f7838156270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.694+0000 7f78404a5640 1 -- 172.21.15.81:0/3360627344 shutdown_connections 2024-06-29T14:37:07.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.694+0000 7f78404a5640 1 -- 172.21.15.81:0/3360627344 wait complete. 2024-06-29T14:37:07.703 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:37:07.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:07.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.819+0000 7f9f36cb9640 1 -- 172.21.15.81:0/2932454517 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f300b9040 msgr2=0x7f9f300d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.819+0000 7f9f36cb9640 1 --2- 172.21.15.81:0/2932454517 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f300b9040 0x7f9f300d0480 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c009830 tx=0x7f9f2c02a5a0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.820+0000 7f9f36cb9640 1 -- 172.21.15.81:0/2932454517 shutdown_connections 2024-06-29T14:37:07.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.820+0000 7f9f36cb9640 1 --2- 172.21.15.81:0/2932454517 >> v2:172.21.15.167:3300/0 conn(0x7f9f300b5250 0x7f9f300d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.820+0000 7f9f36cb9640 1 --2- 172.21.15.81:0/2932454517 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f300b9040 0x7f9f300d0480 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.820+0000 7f9f36cb9640 1 -- 172.21.15.81:0/2932454517 >> 172.21.15.81:0/2932454517 conn(0x7f9f3009cf30 msgr2=0x7f9f300d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:07.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.820+0000 7f9f36cb9640 1 -- 172.21.15.81:0/2932454517 shutdown_connections 2024-06-29T14:37:07.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.820+0000 7f9f36cb9640 1 -- 172.21.15.81:0/2932454517 wait complete. 2024-06-29T14:37:07.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.821+0000 7f9f36cb9640 1 Processor -- start 2024-06-29T14:37:07.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.821+0000 7f9f36cb9640 1 -- start start 2024-06-29T14:37:07.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f36cb9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f300b5250 0x7f9f300a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f36cb9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300b7290 0x7f9f30152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f36cb9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f300b9040 0x7f9f300cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f36cb9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f300b7db0 con 0x7f9f300b7290 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f36cb9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9f300d2d50 con 0x7f9f300b9040 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f36cb9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9f300b7f30 con 0x7f9f300b5250 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f27fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300b7290 0x7f9f30152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f34a55640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f300b5250 0x7f9f300a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f27fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300b7290 0x7f9f30152070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39958/0 (socket says 172.21.15.81:39958) 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f27fff640 1 -- 172.21.15.81:0/3878324784 learned_addr learned my addr 172.21.15.81:0/3878324784 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f34a55640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f300b5250 0x7f9f300a13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:44526/0 (socket says 172.21.15.81:44526) 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f35256640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f300b9040 0x7f9f300cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f27fff640 1 -- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f300b5250 msgr2=0x7f9f300a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f27fff640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f300b5250 0x7f9f300a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f27fff640 1 -- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f300b9040 msgr2=0x7f9f300cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f27fff640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f300b9040 0x7f9f300cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f27fff640 1 -- 172.21.15.81:0/3878324784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f301525b0 con 0x7f9f300b7290 2024-06-29T14:37:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f35256640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f300b9040 0x7f9f300cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.823+0000 7f9f27fff640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300b7290 0x7f9f30152070 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000e7c0 tx=0x7f9f2000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.824+0000 7f9f257fa640 1 -- 172.21.15.81:0/3878324784 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2000ccc0 con 0x7f9f300b7290 2024-06-29T14:37:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.824+0000 7f9f257fa640 1 -- 172.21.15.81:0/3878324784 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f2000ce60 con 0x7f9f300b7290 2024-06-29T14:37:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.824+0000 7f9f257fa640 1 -- 172.21.15.81:0/3878324784 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f20005280 con 0x7f9f300b7290 2024-06-29T14:37:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.824+0000 7f9f36cb9640 1 -- 172.21.15.81:0/3878324784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3009e030 con 0x7f9f300b7290 2024-06-29T14:37:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.824+0000 7f9f36cb9640 1 -- 172.21.15.81:0/3878324784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f3009e5f0 con 0x7f9f300b7290 2024-06-29T14:37:07.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.825+0000 7f9f36cb9640 1 -- 172.21.15.81:0/3878324784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f3009dda0 con 0x7f9f300b7290 2024-06-29T14:37:07.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.825+0000 7f9f257fa640 1 -- 172.21.15.81:0/3878324784 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 27) ==== 97912+0+0 (secure 0 0 0) 0x7f9f20005420 con 0x7f9f300b7290 2024-06-29T14:37:07.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.826+0000 7f9f257fa640 1 -- 172.21.15.81:0/3878324784 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6527+0+0 (secure 0 0 0) 0x7f9f20097c80 con 0x7f9f300b7290 2024-06-29T14:37:07.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.830+0000 7f9f257fa640 1 -- 172.21.15.81:0/3878324784 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9f20061020 con 0x7f9f300b7290 2024-06-29T14:37:07.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.917+0000 7f9f257fa640 1 -- 172.21.15.81:0/3878324784 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 28) ==== 98039+0+0 (secure 0 0 0) 0x7f9f2005c990 con 0x7f9f300b7290 2024-06-29T14:37:07.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.918+0000 7f9f257fa640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9f1c07bbc0 0x7f9f1c07dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:07.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.919+0000 7f9f34a55640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9f1c07bbc0 0x7f9f1c07dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:07.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:07.922+0000 7f9f34a55640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9f1c07bbc0 0x7f9f1c07dfe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c034000 tx=0x7f9f2c004730 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.051+0000 7f9f36cb9640 1 -- 172.21.15.81:0/3878324784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f300b84c0 con 0x7f9f300b7290 2024-06-29T14:37:08.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.052+0000 7f9f257fa640 1 -- 172.21.15.81:0/3878324784 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5229 (secure 0 0 0) 0x7f9f20054e40 con 0x7f9f300b7290 2024-06-29T14:37:08.050 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:08.050 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":25,"btime":"2024-06-29T14:35:02:255148+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2980523942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2980523942},{"type":"v1","addr":"172.21.15.167:6836","nonce":2980523942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5635,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2317403375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2317403375},{"type":"v1","addr":"172.21.15.167:6837","nonce":2317403375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/2346221458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2346221458},{"type":"v1","addr":"172.21.15.81:6834","nonce":2346221458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:34:58.602103+0000","modified":"2024-06-29T14:35:01.623553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6033":{"gid":6033,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/2669979711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2669979711},{"type":"v1","addr":"172.21.15.81:6835","nonce":2669979711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6033,"qdb_cluster":[6033]},"id":3}]} 2024-06-29T14:37:08.050 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 25 2024-06-29T14:37:08.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 -- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9f1c07bbc0 msgr2=0x7f9f1c07dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9f1c07bbc0 0x7f9f1c07dfe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c034000 tx=0x7f9f2c004730 comp rx=0 tx=0).stop 2024-06-29T14:37:08.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 -- 172.21.15.81:0/3878324784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300b7290 msgr2=0x7f9f30152070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300b7290 0x7f9f30152070 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000e7c0 tx=0x7f9f2000ec80 comp rx=0 tx=0).stop 2024-06-29T14:37:08.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 -- 172.21.15.81:0/3878324784 shutdown_connections 2024-06-29T14:37:08.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9f1c07bbc0 0x7f9f1c07dfe0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f300b9040 0x7f9f300cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300b7290 0x7f9f30152070 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 --2- 172.21.15.81:0/3878324784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f300b5250 0x7f9f300a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 -- 172.21.15.81:0/3878324784 >> 172.21.15.81:0/3878324784 conn(0x7f9f3009cf30 msgr2=0x7f9f300d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 -- 172.21.15.81:0/3878324784 shutdown_connections 2024-06-29T14:37:08.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.054+0000 7f9f36cb9640 1 -- 172.21.15.81:0/3878324784 wait complete. 2024-06-29T14:37:08.062 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:08.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.178+0000 7f8bad19e640 1 -- 172.21.15.81:0/2785682455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ba806f660 msgr2=0x7f8ba806fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.178+0000 7f8bad19e640 1 --2- 172.21.15.81:0/2785682455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ba806f660 0x7f8ba806fa70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0005fd0 tx=0x7f8ba002aa10 comp rx=0 tx=0).stop 2024-06-29T14:37:08.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.178+0000 7f8bad19e640 1 -- 172.21.15.81:0/2785682455 shutdown_connections 2024-06-29T14:37:08.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.178+0000 7f8bad19e640 1 --2- 172.21.15.81:0/2785682455 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ba815b5b0 0x7f8ba80b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.178+0000 7f8bad19e640 1 --2- 172.21.15.81:0/2785682455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ba806f660 0x7f8ba806fa70 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.178+0000 7f8bad19e640 1 -- 172.21.15.81:0/2785682455 >> 172.21.15.81:0/2785682455 conn(0x7f8ba806ccc0 msgr2=0x7f8ba80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.178+0000 7f8bad19e640 1 -- 172.21.15.81:0/2785682455 shutdown_connections 2024-06-29T14:37:08.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.178+0000 7f8bad19e640 1 -- 172.21.15.81:0/2785682455 wait complete. 2024-06-29T14:37:08.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.180+0000 7f8bad19e640 1 Processor -- start 2024-06-29T14:37:08.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.180+0000 7f8bad19e640 1 -- start start 2024-06-29T14:37:08.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8bad19e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ba806f660 0x7f8ba8149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8bad19e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ba80b35b0 0x7f8ba8157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8bad19e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ba815b5b0 0x7f8ba8159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8bad19e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba806f160 con 0x7f8ba80b35b0 2024-06-29T14:37:08.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8bad19e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba806efe0 con 0x7f8ba815b5b0 2024-06-29T14:37:08.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8bad19e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba806f2e0 con 0x7f8ba806f660 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8b9ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ba80b35b0 0x7f8ba8157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8b9ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ba80b35b0 0x7f8ba8157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39982/0 (socket says 172.21.15.81:39982) 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8b9ffff640 1 -- 172.21.15.81:0/3075444633 learned_addr learned my addr 172.21.15.81:0/3075444633 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8ba759e640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ba815b5b0 0x7f8ba8159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.181+0000 7f8ba6d9d640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ba806f660 0x7f8ba8149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8b9ffff640 1 -- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ba806f660 msgr2=0x7f8ba8149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8b9ffff640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ba806f660 0x7f8ba8149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8b9ffff640 1 -- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ba815b5b0 msgr2=0x7f8ba8159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8b9ffff640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ba815b5b0 0x7f8ba8159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8b9ffff640 1 -- 172.21.15.81:0/3075444633 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba80aa7b0 con 0x7f8ba80b35b0 2024-06-29T14:37:08.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8b9ffff640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ba80b35b0 0x7f8ba8157100 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000db90 tx=0x7f8b9000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8b9f7fe640 1 -- 172.21.15.81:0/3075444633 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9001b070 con 0x7f8ba80b35b0 2024-06-29T14:37:08.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8b9f7fe640 1 -- 172.21.15.81:0/3075444633 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b9000ae50 con 0x7f8ba80b35b0 2024-06-29T14:37:08.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8b9f7fe640 1 -- 172.21.15.81:0/3075444633 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9000e9e0 con 0x7f8ba80b35b0 2024-06-29T14:37:08.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8bad19e640 1 -- 172.21.15.81:0/3075444633 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba8157ce0 con 0x7f8ba80b35b0 2024-06-29T14:37:08.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.182+0000 7f8bad19e640 1 -- 172.21.15.81:0/3075444633 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba81582a0 con 0x7f8ba80b35b0 2024-06-29T14:37:08.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.183+0000 7f8bad19e640 1 -- 172.21.15.81:0/3075444633 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba8157a50 con 0x7f8ba80b35b0 2024-06-29T14:37:08.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.184+0000 7f8b9f7fe640 1 -- 172.21.15.81:0/3075444633 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 28) ==== 98039+0+0 (secure 0 0 0) 0x7f8b9001e070 con 0x7f8ba80b35b0 2024-06-29T14:37:08.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.184+0000 7f8b9f7fe640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8b88076150 0x7f8b88078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.184+0000 7f8b9f7fe640 1 -- 172.21.15.81:0/3075444633 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6527+0+0 (secure 0 0 0) 0x7f8b90097c20 con 0x7f8ba80b35b0 2024-06-29T14:37:08.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.184+0000 7f8ba6d9d640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8b88076150 0x7f8b88078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.185+0000 7f8ba6d9d640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8b88076150 0x7f8b88078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8b98001720 tx=0x7f8b98009040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.189+0000 7f8b9f7fe640 1 -- 172.21.15.81:0/3075444633 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8ba80aa7b0 con 0x7f8ba80b35b0 2024-06-29T14:37:08.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.407+0000 7f8bad19e640 1 -- 172.21.15.81:0/3075444633 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ba806fa70 con 0x7f8ba80b35b0 2024-06-29T14:37:08.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.408+0000 7f8b9f7fe640 1 -- 172.21.15.81:0/3075444633 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5229 (secure 0 0 0) 0x7f8b90060eb0 con 0x7f8ba80b35b0 2024-06-29T14:37:08.405 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:08.405 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":25,"btime":"2024-06-29T14:35:02:255148+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2980523942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2980523942},{"type":"v1","addr":"172.21.15.167:6836","nonce":2980523942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5635,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2317403375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2317403375},{"type":"v1","addr":"172.21.15.167:6837","nonce":2317403375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/2346221458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2346221458},{"type":"v1","addr":"172.21.15.81:6834","nonce":2346221458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:34:58.602103+0000","modified":"2024-06-29T14:35:01.623553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6033},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6033":{"gid":6033,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/2669979711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2669979711},{"type":"v1","addr":"172.21.15.81:6835","nonce":2669979711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6033,"qdb_cluster":[6033]},"id":3}]} 2024-06-29T14:37:08.406 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 25 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 -- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8b88076150 msgr2=0x7f8b88078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8b88076150 0x7f8b88078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8b98001720 tx=0x7f8b98009040 comp rx=0 tx=0).stop 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 -- 172.21.15.81:0/3075444633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ba80b35b0 msgr2=0x7f8ba8157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ba80b35b0 0x7f8ba8157100 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000db90 tx=0x7f8b9000a620 comp rx=0 tx=0).stop 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 -- 172.21.15.81:0/3075444633 shutdown_connections 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8b88076150 0x7f8b88078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ba815b5b0 0x7f8ba8159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ba80b35b0 0x7f8ba8157100 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 --2- 172.21.15.81:0/3075444633 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ba806f660 0x7f8ba8149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 -- 172.21.15.81:0/3075444633 >> 172.21.15.81:0/3075444633 conn(0x7f8ba806ccc0 msgr2=0x7f8ba8151300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 -- 172.21.15.81:0/3075444633 shutdown_connections 2024-06-29T14:37:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.410+0000 7f8bad19e640 1 -- 172.21.15.81:0/3075444633 wait complete. 2024-06-29T14:37:08.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:08.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.535+0000 7f7ea77e0640 1 -- 172.21.15.81:0/2142855774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ea006f680 msgr2=0x7f7ea006c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.535+0000 7f7ea77e0640 1 --2- 172.21.15.81:0/2142855774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ea006f680 0x7f7ea006c2f0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7e90005fd0 tx=0x7f7e9002a530 comp rx=0 tx=0).stop 2024-06-29T14:37:08.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.535+0000 7f7ea77e0640 1 -- 172.21.15.81:0/2142855774 shutdown_connections 2024-06-29T14:37:08.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.535+0000 7f7ea77e0640 1 --2- 172.21.15.81:0/2142855774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ea006f680 0x7f7ea006c2f0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.535+0000 7f7ea77e0640 1 --2- 172.21.15.81:0/2142855774 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ea0074520 0x7f7ea006f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.535+0000 7f7ea77e0640 1 -- 172.21.15.81:0/2142855774 >> 172.21.15.81:0/2142855774 conn(0x7f7ea0072950 msgr2=0x7f7ea00b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.535+0000 7f7ea77e0640 1 -- 172.21.15.81:0/2142855774 shutdown_connections 2024-06-29T14:37:08.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.535+0000 7f7ea77e0640 1 -- 172.21.15.81:0/2142855774 wait complete. 2024-06-29T14:37:08.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.536+0000 7f7ea77e0640 1 Processor -- start 2024-06-29T14:37:08.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.537+0000 7f7ea77e0640 1 -- start start 2024-06-29T14:37:08.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea77e0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ea006f680 0x7f7ea009d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea77e0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ea006fa90 0x7f7ea015c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea77e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ea0074520 0x7f7ea015cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea77e0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea015b010 con 0x7f7ea0074520 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea77e0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea006d150 con 0x7f7ea006f680 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea77e0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7ea015b190 con 0x7f7ea006fa90 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea5d7d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ea0074520 0x7f7ea015cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea5d7d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ea0074520 0x7f7ea015cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40000/0 (socket says 172.21.15.81:40000) 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea5d7d640 1 -- 172.21.15.81:0/518660368 learned_addr learned my addr 172.21.15.81:0/518660368 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea4d7b640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ea006fa90 0x7f7ea015c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea557c640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ea006f680 0x7f7ea009d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea5d7d640 1 -- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ea006fa90 msgr2=0x7f7ea015c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea5d7d640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ea006fa90 0x7f7ea015c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea5d7d640 1 -- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ea006f680 msgr2=0x7f7ea009d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea5d7d640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ea006f680 0x7f7ea009d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea5d7d640 1 -- 172.21.15.81:0/518660368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea009bc50 con 0x7f7ea0074520 2024-06-29T14:37:08.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea557c640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ea006f680 0x7f7ea009d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:08.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.538+0000 7f7ea5d7d640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ea0074520 0x7f7ea015cda0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800b930 tx=0x7f7e9800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.539+0000 7f7e95ffb640 1 -- 172.21.15.81:0/518660368 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9801b070 con 0x7f7ea0074520 2024-06-29T14:37:08.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.539+0000 7f7e95ffb640 1 -- 172.21.15.81:0/518660368 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e98016440 con 0x7f7ea0074520 2024-06-29T14:37:08.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.539+0000 7f7ea77e0640 1 -- 172.21.15.81:0/518660368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea009bf30 con 0x7f7ea0074520 2024-06-29T14:37:08.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.539+0000 7f7e95ffb640 1 -- 172.21.15.81:0/518660368 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800ee60 con 0x7f7ea0074520 2024-06-29T14:37:08.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.539+0000 7f7ea77e0640 1 -- 172.21.15.81:0/518660368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea009c4f0 con 0x7f7ea0074520 2024-06-29T14:37:08.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.540+0000 7f7ea77e0640 1 -- 172.21.15.81:0/518660368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea009df80 con 0x7f7ea0074520 2024-06-29T14:37:08.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.540+0000 7f7e95ffb640 1 -- 172.21.15.81:0/518660368 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 28) ==== 98039+0+0 (secure 0 0 0) 0x7f7e9801e070 con 0x7f7ea0074520 2024-06-29T14:37:08.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.541+0000 7f7e95ffb640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7e84076120 0x7f7e84078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.541+0000 7f7e95ffb640 1 -- 172.21.15.81:0/518660368 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6527+0+0 (secure 0 0 0) 0x7f7e98099490 con 0x7f7ea0074520 2024-06-29T14:37:08.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.541+0000 7f7ea557c640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7e84076120 0x7f7e84078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.542+0000 7f7ea557c640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7e84076120 0x7f7e84078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009890 tx=0x7f7e9c008040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.546+0000 7f7e95ffb640 1 -- 172.21.15.81:0/518660368 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7e980627b0 con 0x7f7ea0074520 2024-06-29T14:37:08.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.724+0000 7f7ea77e0640 1 -- 172.21.15.81:0/518660368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ea009d680 con 0x7f7ea0074520 2024-06-29T14:37:08.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.725+0000 7f7e95ffb640 1 -- 172.21.15.81:0/518660368 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+17848 (secure 0 0 0) 0x7f7e98066ab0 con 0x7f7ea0074520 2024-06-29T14:37:08.724 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:08.724 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":67,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:06.907868+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:34:56.911140+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-06-29T14:34:57.555296+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.12","mappings":[{"from":0,"to":5}]},{"pgid":"7.2c","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/375340026":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/3632236021":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/1756060870":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/1790794665":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/183639693":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/3418120263":"2024-06-30T14:37:06.907833+0000","172.21.15.167:6833/1090782941":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6832/1090782941":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/1626573527":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6833/2493228728":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/799531669":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/2483549331":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6832/2493228728":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/2831660161":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/3380278063":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/605629739":"2024-06-30T14:35:10.908137+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:08.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 -- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7e84076120 msgr2=0x7f7e84078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7e84076120 0x7f7e84078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009890 tx=0x7f7e9c008040 comp rx=0 tx=0).stop 2024-06-29T14:37:08.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 -- 172.21.15.81:0/518660368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ea0074520 msgr2=0x7f7ea015cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ea0074520 0x7f7ea015cda0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800b930 tx=0x7f7e9800bdf0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 -- 172.21.15.81:0/518660368 shutdown_connections 2024-06-29T14:37:08.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7e84076120 0x7f7e84078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ea0074520 0x7f7ea015cda0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ea006fa90 0x7f7ea015c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 --2- 172.21.15.81:0/518660368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ea006f680 0x7f7ea009d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 -- 172.21.15.81:0/518660368 >> 172.21.15.81:0/518660368 conn(0x7f7ea0072950 msgr2=0x7f7ea00b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 -- 172.21.15.81:0/518660368 shutdown_connections 2024-06-29T14:37:08.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.727+0000 7f7ea77e0640 1 -- 172.21.15.81:0/518660368 wait complete. 2024-06-29T14:37:08.737 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:37:08.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.852+0000 7fbc0a438640 1 -- 172.21.15.81:0/1004517810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc040b9040 msgr2=0x7fbc040d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.852+0000 7fbc0a438640 1 --2- 172.21.15.81:0/1004517810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc040b9040 0x7fbc040d0480 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4005fd0 tx=0x7fbbf402a670 comp rx=0 tx=0).stop 2024-06-29T14:37:08.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.852+0000 7fbc0a438640 1 -- 172.21.15.81:0/1004517810 shutdown_connections 2024-06-29T14:37:08.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.852+0000 7fbc0a438640 1 --2- 172.21.15.81:0/1004517810 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc040d2a90 0x7fbc040d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.852+0000 7fbc0a438640 1 --2- 172.21.15.81:0/1004517810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc040b9040 0x7fbc040d0480 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.852+0000 7fbc0a438640 1 -- 172.21.15.81:0/1004517810 >> 172.21.15.81:0/1004517810 conn(0x7fbc0409cf30 msgr2=0x7fbc040d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:08.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.852+0000 7fbc0a438640 1 -- 172.21.15.81:0/1004517810 shutdown_connections 2024-06-29T14:37:08.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.852+0000 7fbc0a438640 1 -- 172.21.15.81:0/1004517810 wait complete. 2024-06-29T14:37:08.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.854+0000 7fbc0a438640 1 Processor -- start 2024-06-29T14:37:08.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.854+0000 7fbc0a438640 1 -- start start 2024-06-29T14:37:08.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.855+0000 7fbc0a438640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc040b5250 0x7fbc040a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.855+0000 7fbc0a438640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbc040b9040 0x7fbc040b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.855+0000 7fbc0a438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc040d2a90 0x7fbc040b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.855+0000 7fbc0a438640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbc040b7d30 con 0x7fbc040d2a90 2024-06-29T14:37:08.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.855+0000 7fbc0a438640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbc040b7bb0 con 0x7fbc040b9040 2024-06-29T14:37:08.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.855+0000 7fbc0a438640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbc040b7eb0 con 0x7fbc040b5250 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc089d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc040d2a90 0x7fbc040b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc089d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc040d2a90 0x7fbc040b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40012/0 (socket says 172.21.15.81:40012) 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc089d5640 1 -- 172.21.15.81:0/448573271 learned_addr learned my addr 172.21.15.81:0/448573271 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc03fff640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc040b5250 0x7fbc040a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc037fe640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbc040b9040 0x7fbc040b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc089d5640 1 -- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc040b5250 msgr2=0x7fbc040a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc089d5640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc040b5250 0x7fbc040a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc089d5640 1 -- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbc040b9040 msgr2=0x7fbc040b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc089d5640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbc040b9040 0x7fbc040b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc089d5640 1 -- 172.21.15.81:0/448573271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc040a10f0 con 0x7fbc040d2a90 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc089d5640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc040d2a90 0x7fbc040b1f80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00da10 tx=0x7fbbfc00ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc00ff9640 1 -- 172.21.15.81:0/448573271 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc017070 con 0x7fbc040d2a90 2024-06-29T14:37:08.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc00ff9640 1 -- 172.21.15.81:0/448573271 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbfc010460 con 0x7fbc040d2a90 2024-06-29T14:37:08.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc0a438640 1 -- 172.21.15.81:0/448573271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc040a13d0 con 0x7fbc040d2a90 2024-06-29T14:37:08.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc00ff9640 1 -- 172.21.15.81:0/448573271 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0108b0 con 0x7fbc040d2a90 2024-06-29T14:37:08.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.856+0000 7fbc0a438640 1 -- 172.21.15.81:0/448573271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc040b1240 con 0x7fbc040d2a90 2024-06-29T14:37:08.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.857+0000 7fbc0a438640 1 -- 172.21.15.81:0/448573271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc040b8440 con 0x7fbc040d2a90 2024-06-29T14:37:08.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.858+0000 7fbc00ff9640 1 -- 172.21.15.81:0/448573271 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 28) ==== 98039+0+0 (secure 0 0 0) 0x7fbbfc01a070 con 0x7fbc040d2a90 2024-06-29T14:37:08.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.859+0000 7fbc00ff9640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbbf0076120 0x7fbbf0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:08.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.859+0000 7fbc00ff9640 1 -- 172.21.15.81:0/448573271 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6527+0+0 (secure 0 0 0) 0x7fbbfc097ad0 con 0x7fbc040d2a90 2024-06-29T14:37:08.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.859+0000 7fbc03fff640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbbf0076120 0x7fbbf0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:08.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.859+0000 7fbc03fff640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbbf0076120 0x7fbbf0078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4033000 tx=0x7fbbf4004730 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:08.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:08.863+0000 7fbc00ff9640 1 -- 172.21.15.81:0/448573271 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbbfc0613b0 con 0x7fbc040d2a90 2024-06-29T14:37:09.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.085+0000 7fbc0a438640 1 -- 172.21.15.81:0/448573271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbc040688e0 con 0x7fbc040d2a90 2024-06-29T14:37:09.083 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T14:37:09.085+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:37:09.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.087+0000 7fbc00ff9640 1 -- 172.21.15.81:0/448573271 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7fbbfc060cb0 con 0x7fbc040d2a90 2024-06-29T14:37:09.085 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 -- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbbf0076120 msgr2=0x7fbbf0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbbf0076120 0x7fbbf0078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4033000 tx=0x7fbbf4004730 comp rx=0 tx=0).stop 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 -- 172.21.15.81:0/448573271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc040d2a90 msgr2=0x7fbc040b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc040d2a90 0x7fbc040b1f80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00da10 tx=0x7fbbfc00ded0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 -- 172.21.15.81:0/448573271 shutdown_connections 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbbf0076120 0x7fbbf0078630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc040d2a90 0x7fbc040b1f80 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbc040b9040 0x7fbc040b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 --2- 172.21.15.81:0/448573271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc040b5250 0x7fbc040a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 -- 172.21.15.81:0/448573271 >> 172.21.15.81:0/448573271 conn(0x7fbc0409cf30 msgr2=0x7fbc0415cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:09.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 -- 172.21.15.81:0/448573271 shutdown_connections 2024-06-29T14:37:09.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.090+0000 7fbc0a438640 1 -- 172.21.15.81:0/448573271 wait complete. 2024-06-29T14:37:09.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:37:09.148 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T14:37:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.220+0000 7f87d703f640 1 -- 172.21.15.81:0/2869880922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87bc003c60 msgr2=0x7f87bc0040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.220+0000 7f87d703f640 1 --2- 172.21.15.81:0/2869880922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87bc003c60 0x7f87bc0040b0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f87c0001830 tx=0x7f87bc005830 comp rx=0 tx=0).stop 2024-06-29T14:37:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.220+0000 7f87d55dc640 1 -- 172.21.15.81:0/2869880922 reap_dead start 2024-06-29T14:37:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.220+0000 7f87d703f640 1 -- 172.21.15.81:0/2869880922 shutdown_connections 2024-06-29T14:37:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.220+0000 7f87d703f640 1 -- 172.21.15.81:0/2869880922 >> 172.21.15.81:0/2869880922 conn(0x7f87d00b28b0 msgr2=0x7f87d00b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.220+0000 7f87d703f640 1 -- 172.21.15.81:0/2869880922 shutdown_connections 2024-06-29T14:37:09.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.220+0000 7f87d703f640 1 -- 172.21.15.81:0/2869880922 wait complete. 2024-06-29T14:37:09.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.222+0000 7f87d703f640 1 Processor -- start 2024-06-29T14:37:09.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.222+0000 7f87d703f640 1 -- start start 2024-06-29T14:37:09.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d703f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87bc003c60 0x7f87d0151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d703f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87d0153510 0x7f87d0156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d703f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d0158210 0x7f87d0158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d703f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87d015ec10 con 0x7f87d0158210 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d703f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f87d00cfa10 con 0x7f87bc003c60 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d703f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f87d015ed90 con 0x7f87d0153510 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d55dc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d0158210 0x7f87d0158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d55dc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d0158210 0x7f87d0158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40040/0 (socket says 172.21.15.81:40040) 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d55dc640 1 -- 172.21.15.81:0/2995974175 learned_addr learned my addr 172.21.15.81:0/2995974175 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d55dc640 1 -- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87d0153510 msgr2=0x7f87d0156880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87cffff640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87d0153510 0x7f87d0156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.223+0000 7f87d4ddb640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87bc003c60 0x7f87d0151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:09.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.224+0000 7f87d55dc640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87d0153510 0x7f87d0156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.224+0000 7f87d55dc640 1 -- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87bc003c60 msgr2=0x7f87d0151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:09.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.224+0000 7f87d55dc640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87bc003c60 0x7f87d0151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:09.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.224+0000 7f87d55dc640 1 -- 172.21.15.81:0/2995974175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d0139d50 con 0x7f87d0158210 2024-06-29T14:37:09.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.224+0000 7f87d55dc640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d0158210 0x7f87d0158620 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f87c8001350 tx=0x7f87c800d6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:09.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.224+0000 7f87cd7fa640 1 -- 172.21.15.81:0/2995974175 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c800ee50 con 0x7f87d0158210 2024-06-29T14:37:09.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.224+0000 7f87cd7fa640 1 -- 172.21.15.81:0/2995974175 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87c8003a60 con 0x7f87d0158210 2024-06-29T14:37:09.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.224+0000 7f87cd7fa640 1 -- 172.21.15.81:0/2995974175 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8006070 con 0x7f87d0158210 2024-06-29T14:37:09.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.224+0000 7f87d703f640 1 -- 172.21.15.81:0/2995974175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d0152470 con 0x7f87d0158210 2024-06-29T14:37:09.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.224+0000 7f87d703f640 1 -- 172.21.15.81:0/2995974175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d0152a30 con 0x7f87d0158210 2024-06-29T14:37:09.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.225+0000 7f87d703f640 1 -- 172.21.15.81:0/2995974175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87d00b41c0 con 0x7f87d0158210 2024-06-29T14:37:09.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.225+0000 7f87cd7fa640 1 -- 172.21.15.81:0/2995974175 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 28) ==== 98039+0+0 (secure 0 0 0) 0x7f87c8006210 con 0x7f87d0158210 2024-06-29T14:37:09.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.226+0000 7f87cd7fa640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f87b00762c0 0x7f87b00787d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:09.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.226+0000 7f87cd7fa640 1 -- 172.21.15.81:0/2995974175 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6613+0+0 (secure 0 0 0) 0x7f87c8097dd0 con 0x7f87d0158210 2024-06-29T14:37:09.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.226+0000 7f87d4ddb640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f87b00762c0 0x7f87b00787d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:09.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.227+0000 7f87d4ddb640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f87b00762c0 0x7f87b00787d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f87c003b000 tx=0x7f87c0001360 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:09.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.230+0000 7f87cd7fa640 1 -- 172.21.15.81:0/2995974175 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f87d015f150 con 0x7f87d0158210 2024-06-29T14:37:09.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.260+0000 7f87cd7fa640 1 -- 172.21.15.81:0/2995974175 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 29) ==== 98247+0+0 (secure 0 0 0) 0x7f87c805d270 con 0x7f87d0158210 2024-06-29T14:37:09.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:09.453+0000 7f87d703f640 1 -- 172.21.15.81:0/2995974175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f87d01521c0 con 0x7f87d0158210 2024-06-29T14:37:10.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.089+0000 7f87cd7fa640 1 -- 172.21.15.81:0/2995974175 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7f87c80616e0 con 0x7f87d0158210 2024-06-29T14:37:10.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 -- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f87b00762c0 msgr2=0x7f87b00787d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f87b00762c0 0x7f87b00787d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f87c003b000 tx=0x7f87c0001360 comp rx=0 tx=0).stop 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 -- 172.21.15.81:0/2995974175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d0158210 msgr2=0x7f87d0158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d0158210 0x7f87d0158620 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f87c8001350 tx=0x7f87c800d6a0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 -- 172.21.15.81:0/2995974175 shutdown_connections 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f87b00762c0 0x7f87b00787d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d0158210 0x7f87d0158620 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87d0153510 0x7f87d0156880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 --2- 172.21.15.81:0/2995974175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87bc003c60 0x7f87d0151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 -- 172.21.15.81:0/2995974175 >> 172.21.15.81:0/2995974175 conn(0x7f87d00b28b0 msgr2=0x7f87d00b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 -- 172.21.15.81:0/2995974175 shutdown_connections 2024-06-29T14:37:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.091+0000 7f87d703f640 1 -- 172.21.15.81:0/2995974175 wait complete. 2024-06-29T14:37:10.102 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:37:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.224+0000 7f1b64281640 1 -- 172.21.15.81:0/3341118812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b5c0af730 msgr2=0x7f1b5c0afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.224+0000 7f1b64281640 1 --2- 172.21.15.81:0/3341118812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b5c0af730 0x7f1b5c0afb40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1b54005fd0 tx=0x7f1b5402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.224+0000 7f1b64281640 1 -- 172.21.15.81:0/3341118812 shutdown_connections 2024-06-29T14:37:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.224+0000 7f1b64281640 1 --2- 172.21.15.81:0/3341118812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b5c0af730 0x7f1b5c0afb40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.224+0000 7f1b64281640 1 --2- 172.21.15.81:0/3341118812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b5c0708f0 0x7f1b5c15bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.224+0000 7f1b64281640 1 -- 172.21.15.81:0/3341118812 >> 172.21.15.81:0/3341118812 conn(0x7f1b5c0d08f0 msgr2=0x7f1b5c0d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.224+0000 7f1b64281640 1 -- 172.21.15.81:0/3341118812 shutdown_connections 2024-06-29T14:37:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.224+0000 7f1b64281640 1 -- 172.21.15.81:0/3341118812 wait complete. 2024-06-29T14:37:10.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.226+0000 7f1b64281640 1 Processor -- start 2024-06-29T14:37:10.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.226+0000 7f1b64281640 1 -- start start 2024-06-29T14:37:10.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b64281640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b5c0708f0 0x7f1b5c153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:10.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b64281640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b5c0af730 0x7f1b5c1e6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:10.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b64281640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b5c1555e0 0x7f1b5c1f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:10.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b64281640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c06ff10 con 0x7f1b5c0708f0 2024-06-29T14:37:10.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b64281640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c06fd90 con 0x7f1b5c0af730 2024-06-29T14:37:10.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b64281640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1b5c070090 con 0x7f1b5c1555e0 2024-06-29T14:37:10.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b6201d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b5c0708f0 0x7f1b5c153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b6201d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b5c0708f0 0x7f1b5c153ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42306/0 (socket says 172.21.15.81:42306) 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b6201d640 1 -- 172.21.15.81:0/309043073 learned_addr learned my addr 172.21.15.81:0/309043073 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b6281e640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b5c1555e0 0x7f1b5c1f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.227+0000 7f1b6181c640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b5c0af730 0x7f1b5c1e6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b6201d640 1 -- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b5c1555e0 msgr2=0x7f1b5c1f3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b6201d640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b5c1555e0 0x7f1b5c1f3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b6201d640 1 -- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b5c0af730 msgr2=0x7f1b5c1e6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b6201d640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b5c0af730 0x7f1b5c1e6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b6201d640 1 -- 172.21.15.81:0/309043073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c0af4c0 con 0x7f1b5c0708f0 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b6201d640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b5c0708f0 0x7f1b5c153ba0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800c760 tx=0x7f1b5800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b52ffd640 1 -- 172.21.15.81:0/309043073 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b580050d0 con 0x7f1b5c0708f0 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b52ffd640 1 -- 172.21.15.81:0/309043073 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b58005290 con 0x7f1b5c0708f0 2024-06-29T14:37:10.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b52ffd640 1 -- 172.21.15.81:0/309043073 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5800a8a0 con 0x7f1b5c0708f0 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b64281640 1 -- 172.21.15.81:0/309043073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5c1e4fc0 con 0x7f1b5c0708f0 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.228+0000 7f1b64281640 1 -- 172.21.15.81:0/309043073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5c1e5580 con 0x7f1b5c0708f0 2024-06-29T14:37:10.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.229+0000 7f1b64281640 1 -- 172.21.15.81:0/309043073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5c075c90 con 0x7f1b5c0708f0 2024-06-29T14:37:10.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.230+0000 7f1b52ffd640 1 -- 172.21.15.81:0/309043073 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 29) ==== 98247+0+0 (secure 0 0 0) 0x7f1b5800aa40 con 0x7f1b5c0708f0 2024-06-29T14:37:10.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.230+0000 7f1b52ffd640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f1b40076360 0x7f1b40078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:10.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.230+0000 7f1b52ffd640 1 -- 172.21.15.81:0/309043073 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6613+0+0 (secure 0 0 0) 0x7f1b58097620 con 0x7f1b5c0708f0 2024-06-29T14:37:10.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.230+0000 7f1b6181c640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f1b40076360 0x7f1b40078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:10.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.231+0000 7f1b6181c640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f1b40076360 0x7f1b40078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0096c0 tx=0x7f1b4c009260 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:10.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.234+0000 7f1b52ffd640 1 -- 172.21.15.81:0/309043073 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1b5c075c90 con 0x7f1b5c0708f0 2024-06-29T14:37:10.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:10.419+0000 7f1b64281640 1 -- 172.21.15.81:0/309043073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1b5c070620 con 0x7f1b5c0708f0 2024-06-29T14:37:10.978 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:37:10.981 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:37:11.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.263+0000 7f1b52ffd640 1 -- 172.21.15.81:0/309043073 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v69) ==== 181+0+0 (secure 0 0 0) 0x7f1b5c0b03f0 con 0x7f1b5c0708f0 2024-06-29T14:37:11.261 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:37:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 -- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f1b40076360 msgr2=0x7f1b40078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f1b40076360 0x7f1b40078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0096c0 tx=0x7f1b4c009260 comp rx=0 tx=0).stop 2024-06-29T14:37:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 -- 172.21.15.81:0/309043073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b5c0708f0 msgr2=0x7f1b5c153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b5c0708f0 0x7f1b5c153ba0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800c760 tx=0x7f1b5800cc20 comp rx=0 tx=0).stop 2024-06-29T14:37:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 -- 172.21.15.81:0/309043073 shutdown_connections 2024-06-29T14:37:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f1b40076360 0x7f1b40078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1b5c1555e0 0x7f1b5c1f3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1b5c0af730 0x7f1b5c1e6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 --2- 172.21.15.81:0/309043073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b5c0708f0 0x7f1b5c153ba0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 -- 172.21.15.81:0/309043073 >> 172.21.15.81:0/309043073 conn(0x7f1b5c0d08f0 msgr2=0x7f1b5c0b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:11.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 -- 172.21.15.81:0/309043073 shutdown_connections 2024-06-29T14:37:11.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.265+0000 7f1b64281640 1 -- 172.21.15.81:0/309043073 wait complete. 2024-06-29T14:37:11.280 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:37:11.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.401+0000 7faf66aac640 1 -- 172.21.15.81:0/784800755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf60074070 msgr2=0x7faf60074480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:11.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.401+0000 7faf66aac640 1 --2- 172.21.15.81:0/784800755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf60074070 0x7faf60074480 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7faf60070e30 tx=0x7faf4c02a440 comp rx=0 tx=0).stop 2024-06-29T14:37:11.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.401+0000 7faf66aac640 1 -- 172.21.15.81:0/784800755 shutdown_connections 2024-06-29T14:37:11.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.401+0000 7faf66aac640 1 --2- 172.21.15.81:0/784800755 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf6006c2f0 0x7faf6006c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.401+0000 7faf66aac640 1 --2- 172.21.15.81:0/784800755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf60074070 0x7faf60074480 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.401+0000 7faf66aac640 1 --2- 172.21.15.81:0/784800755 >> v2:172.21.15.167:3300/0 conn(0x7faf600b04b0 0x7faf60073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.401+0000 7faf66aac640 1 -- 172.21.15.81:0/784800755 >> 172.21.15.81:0/784800755 conn(0x7faf60072970 msgr2=0x7faf600b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:11.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.401+0000 7faf66aac640 1 -- 172.21.15.81:0/784800755 shutdown_connections 2024-06-29T14:37:11.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.401+0000 7faf66aac640 1 -- 172.21.15.81:0/784800755 wait complete. 2024-06-29T14:37:11.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.403+0000 7faf66aac640 1 Processor -- start 2024-06-29T14:37:11.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.403+0000 7faf66aac640 1 -- start start 2024-06-29T14:37:11.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf66aac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf6006c2f0 0x7faf601504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:11.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf66aac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf60074070 0x7faf600ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:11.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf66aac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf600b04b0 0x7faf600cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf66aac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf6006cdc0 con 0x7faf6006c2f0 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf66aac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faf6006cc40 con 0x7faf60074070 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf66aac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faf6006cf40 con 0x7faf600b04b0 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf64848640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf6006c2f0 0x7faf601504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf64848640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf6006c2f0 0x7faf601504f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42326/0 (socket says 172.21.15.81:42326) 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf64848640 1 -- 172.21.15.81:0/3880606902 learned_addr learned my addr 172.21.15.81:0/3880606902 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf5ffff640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf60074070 0x7faf600ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.404+0000 7faf65049640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf600b04b0 0x7faf600cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf64848640 1 -- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf600b04b0 msgr2=0x7faf600cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf64848640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf600b04b0 0x7faf600cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf64848640 1 -- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf60074070 msgr2=0x7faf600ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf64848640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf60074070 0x7faf600ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf64848640 1 -- 172.21.15.81:0/3880606902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf601f6550 con 0x7faf6006c2f0 2024-06-29T14:37:11.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf5ffff640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf60074070 0x7faf600ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf64848640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf6006c2f0 0x7faf601504f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7faf5400b440 tx=0x7faf5400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf5d7fa640 1 -- 172.21.15.81:0/3880606902 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf54015020 con 0x7faf6006c2f0 2024-06-29T14:37:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf5d7fa640 1 -- 172.21.15.81:0/3880606902 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf54011980 con 0x7faf6006c2f0 2024-06-29T14:37:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf66aac640 1 -- 172.21.15.81:0/3880606902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf601f6830 con 0x7faf6006c2f0 2024-06-29T14:37:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf5d7fa640 1 -- 172.21.15.81:0/3880606902 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf54011dd0 con 0x7faf6006c2f0 2024-06-29T14:37:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.405+0000 7faf66aac640 1 -- 172.21.15.81:0/3880606902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf601f6df0 con 0x7faf6006c2f0 2024-06-29T14:37:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.406+0000 7faf66aac640 1 -- 172.21.15.81:0/3880606902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf600747a0 con 0x7faf6006c2f0 2024-06-29T14:37:11.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.407+0000 7faf5d7fa640 1 -- 172.21.15.81:0/3880606902 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7faf54018020 con 0x7faf6006c2f0 2024-06-29T14:37:11.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.407+0000 7faf5d7fa640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7faf440763f0 0x7faf44078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:11.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.407+0000 7faf5d7fa640 1 -- 172.21.15.81:0/3880606902 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6175+0+0 (secure 0 0 0) 0x7faf54097ce0 con 0x7faf6006c2f0 2024-06-29T14:37:11.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.407+0000 7faf5ffff640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7faf440763f0 0x7faf44078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:11.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.408+0000 7faf5ffff640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7faf440763f0 0x7faf44078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faf4c033000 tx=0x7faf4c004730 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:11.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.412+0000 7faf5d7fa640 1 -- 172.21.15.81:0/3880606902 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faf600747a0 con 0x7faf6006c2f0 2024-06-29T14:37:11.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:11.590+0000 7faf66aac640 1 -- 172.21.15.81:0/3880606902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7faf60074480 con 0x7faf6006c2f0 2024-06-29T14:37:12.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.264+0000 7faf5d7fa640 1 -- 172.21.15.81:0/3880606902 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v70) ==== 169+0+0 (secure 0 0 0) 0x7faf5401f020 con 0x7faf6006c2f0 2024-06-29T14:37:12.264 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T14:37:12.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.268+0000 7faf66aac640 1 -- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7faf440763f0 msgr2=0x7faf44078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.268+0000 7faf66aac640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7faf440763f0 0x7faf44078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faf4c033000 tx=0x7faf4c004730 comp rx=0 tx=0).stop 2024-06-29T14:37:12.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.268+0000 7faf66aac640 1 -- 172.21.15.81:0/3880606902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf6006c2f0 msgr2=0x7faf601504f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.268+0000 7faf66aac640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf6006c2f0 0x7faf601504f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7faf5400b440 tx=0x7faf5400b900 comp rx=0 tx=0).stop 2024-06-29T14:37:12.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.268+0000 7faf66aac640 1 -- 172.21.15.81:0/3880606902 shutdown_connections 2024-06-29T14:37:12.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.268+0000 7faf66aac640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7faf440763f0 0x7faf44078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.268+0000 7faf66aac640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf600b04b0 0x7faf600cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.268+0000 7faf66aac640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf60074070 0x7faf600ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.268+0000 7faf66aac640 1 --2- 172.21.15.81:0/3880606902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf6006c2f0 0x7faf601504f0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.268+0000 7faf66aac640 1 -- 172.21.15.81:0/3880606902 >> 172.21.15.81:0/3880606902 conn(0x7faf60072970 msgr2=0x7faf6015ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:12.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.269+0000 7faf66aac640 1 -- 172.21.15.81:0/3880606902 shutdown_connections 2024-06-29T14:37:12.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.269+0000 7faf66aac640 1 -- 172.21.15.81:0/3880606902 wait complete. 2024-06-29T14:37:12.281 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:37:12.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.401+0000 7f4c12370640 1 -- 172.21.15.81:0/1562590051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c0c15c030 msgr2=0x7f4c0c06c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.401+0000 7f4c12370640 1 --2- 172.21.15.81:0/1562590051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c0c15c030 0x7f4c0c06c2f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c074e30 tx=0x7f4c0002a560 comp rx=0 tx=0).stop 2024-06-29T14:37:12.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.401+0000 7f4c12370640 1 -- 172.21.15.81:0/1562590051 shutdown_connections 2024-06-29T14:37:12.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.401+0000 7f4c12370640 1 --2- 172.21.15.81:0/1562590051 >> v2:172.21.15.167:3300/0 conn(0x7f4c0c15c440 0x7f4c0c06c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.401+0000 7f4c12370640 1 --2- 172.21.15.81:0/1562590051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c0c15c030 0x7f4c0c06c2f0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.401+0000 7f4c12370640 1 -- 172.21.15.81:0/1562590051 >> 172.21.15.81:0/1562590051 conn(0x7f4c0c0d0930 msgr2=0x7f4c0c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:12.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.401+0000 7f4c12370640 1 -- 172.21.15.81:0/1562590051 shutdown_connections 2024-06-29T14:37:12.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.401+0000 7f4c12370640 1 -- 172.21.15.81:0/1562590051 wait complete. 2024-06-29T14:37:12.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.403+0000 7f4c12370640 1 Processor -- start 2024-06-29T14:37:12.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.403+0000 7f4c12370640 1 -- start start 2024-06-29T14:37:12.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.404+0000 7f4c12370640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c0c15c030 0x7f4c0c1f7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.404+0000 7f4c12370640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c0c15c440 0x7f4c0c151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.404+0000 7f4c12370640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c0c15d750 0x7f4c0c1f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.404+0000 7f4c12370640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0c0cfca0 con 0x7f4c0c15c440 2024-06-29T14:37:12.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.404+0000 7f4c12370640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0c0cfb20 con 0x7f4c0c15d750 2024-06-29T14:37:12.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.404+0000 7f4c12370640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4c0c0cfe20 con 0x7f4c0c15c030 2024-06-29T14:37:12.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c0b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c0c15c440 0x7f4c0c151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c0b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c0c15c440 0x7f4c0c151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42358/0 (socket says 172.21.15.81:42358) 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c0b7fe640 1 -- 172.21.15.81:0/930714608 learned_addr learned my addr 172.21.15.81:0/930714608 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c0b7fe640 1 -- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c0c15c030 msgr2=0x7f4c0c1f7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c0b7fe640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c0c15c030 0x7f4c0c1f7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c0b7fe640 1 -- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c0c15d750 msgr2=0x7f4c0c1f65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c0b7fe640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c0c15d750 0x7f4c0c1f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c0b7fe640 1 -- 172.21.15.81:0/930714608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0c157a00 con 0x7f4c0c15c440 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c0b7fe640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c0c15c440 0x7f4c0c151f90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4c00034040 tx=0x7f4c000039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c08ff9640 1 -- 172.21.15.81:0/930714608 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0003e030 con 0x7f4c0c15c440 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c12370640 1 -- 172.21.15.81:0/930714608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0c157ce0 con 0x7f4c0c15c440 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c08ff9640 1 -- 172.21.15.81:0/930714608 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c000047a0 con 0x7f4c0c15c440 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c08ff9640 1 -- 172.21.15.81:0/930714608 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00035040 con 0x7f4c0c15c440 2024-06-29T14:37:12.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.405+0000 7f4c12370640 1 -- 172.21.15.81:0/930714608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c0c1582a0 con 0x7f4c0c15c440 2024-06-29T14:37:12.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.406+0000 7f4c12370640 1 -- 172.21.15.81:0/930714608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c0c06e2a0 con 0x7f4c0c15c440 2024-06-29T14:37:12.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.407+0000 7f4c08ff9640 1 -- 172.21.15.81:0/930714608 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f4c00041030 con 0x7f4c0c15c440 2024-06-29T14:37:12.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.407+0000 7f4c08ff9640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f4bf80763f0 0x7f4bf8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.408+0000 7f4c08ff9640 1 -- 172.21.15.81:0/930714608 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5703+0+0 (secure 0 0 0) 0x7f4c000b82a0 con 0x7f4c0c15c440 2024-06-29T14:37:12.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.408+0000 7f4c0bfff640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f4bf80763f0 0x7f4bf8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.408+0000 7f4c0bfff640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f4bf80763f0 0x7f4bf8078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009650 tx=0x7f4bfc006ba0 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:12.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.412+0000 7f4c08ff9640 1 -- 172.21.15.81:0/930714608 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4c0c06e2a0 con 0x7f4c0c15c440 2024-06-29T14:37:12.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.591+0000 7f4c12370640 1 -- 172.21.15.81:0/930714608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f4c0c0d02d0 con 0x7f4c0c15c440 2024-06-29T14:37:12.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.595+0000 7f4c08ff9640 1 -- 172.21.15.81:0/930714608 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v35) ==== 71+0+0 (secure 0 0 0) 0x7f4c00081960 con 0x7f4c0c15c440 2024-06-29T14:37:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 -- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f4bf80763f0 msgr2=0x7f4bf8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f4bf80763f0 0x7f4bf8078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009650 tx=0x7f4bfc006ba0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 -- 172.21.15.81:0/930714608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c0c15c440 msgr2=0x7f4c0c151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c0c15c440 0x7f4c0c151f90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4c00034040 tx=0x7f4c000039e0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 -- 172.21.15.81:0/930714608 shutdown_connections 2024-06-29T14:37:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f4bf80763f0 0x7f4bf8078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c0c15d750 0x7f4c0c1f65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c0c15c440 0x7f4c0c151f90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 --2- 172.21.15.81:0/930714608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c0c15c030 0x7f4c0c1f7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 -- 172.21.15.81:0/930714608 >> 172.21.15.81:0/930714608 conn(0x7f4c0c0d0930 msgr2=0x7f4c0c15db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:12.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 -- 172.21.15.81:0/930714608 shutdown_connections 2024-06-29T14:37:12.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.597+0000 7f4c12370640 1 -- 172.21.15.81:0/930714608 wait complete. 2024-06-29T14:37:12.607 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2024-06-29T14:37:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.723+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1305168310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbd4073570 msgr2=0x7fcbd415b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.723+0000 7fcbdc7e1640 1 --2- 172.21.15.81:0/1305168310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbd4073570 0x7fcbd415b4d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40b2320 tx=0x7fcbc402a530 comp rx=0 tx=0).stop 2024-06-29T14:37:12.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.724+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1305168310 shutdown_connections 2024-06-29T14:37:12.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.724+0000 7fcbdc7e1640 1 --2- 172.21.15.81:0/1305168310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbd4073570 0x7fcbd415b4d0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.724+0000 7fcbdc7e1640 1 --2- 172.21.15.81:0/1305168310 >> v2:172.21.15.167:3300/0 conn(0x7fcbd413dd70 0x7fcbd4073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.724+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1305168310 >> 172.21.15.81:0/1305168310 conn(0x7fcbd406ccc0 msgr2=0x7fcbd40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.724+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1305168310 shutdown_connections 2024-06-29T14:37:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.724+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1305168310 wait complete. 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.725+0000 7fcbdc7e1640 1 Processor -- start 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.725+0000 7fcbdc7e1640 1 -- start start 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdc7e1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbd4073570 0x7fcbd413a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdc7e1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbd413dd70 0x7fcbd413e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdc7e1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbd415ba10 0x7fcbd413ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdc7e1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd406f4e0 con 0x7fcbd415ba10 2024-06-29T14:37:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdc7e1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd40cf3d0 con 0x7fcbd413dd70 2024-06-29T14:37:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdc7e1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd406f660 con 0x7fcbd4073570 2024-06-29T14:37:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdad7e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbd415ba10 0x7fcbd413ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdad7e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbd415ba10 0x7fcbd413ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42370/0 (socket says 172.21.15.81:42370) 2024-06-29T14:37:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdad7e640 1 -- 172.21.15.81:0/1756260452 learned_addr learned my addr 172.21.15.81:0/1756260452 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbda57d640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbd4073570 0x7fcbd413a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbd9d7c640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbd413dd70 0x7fcbd413e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdad7e640 1 -- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbd4073570 msgr2=0x7fcbd413a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdad7e640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbd4073570 0x7fcbd413a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdad7e640 1 -- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbd413dd70 msgr2=0x7fcbd413e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdad7e640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbd413dd70 0x7fcbd413e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdad7e640 1 -- 172.21.15.81:0/1756260452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd41f4410 con 0x7fcbd415ba10 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdad7e640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbd415ba10 0x7fcbd413ed30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00c390 tx=0x7fcbcc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbcaffd640 1 -- 172.21.15.81:0/1756260452 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc011790 con 0x7fcbd415ba10 2024-06-29T14:37:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbcaffd640 1 -- 172.21.15.81:0/1756260452 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbcc017070 con 0x7fcbd415ba10 2024-06-29T14:37:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbcaffd640 1 -- 172.21.15.81:0/1756260452 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc011c80 con 0x7fcbd415ba10 2024-06-29T14:37:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.727+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1756260452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd41f4680 con 0x7fcbd415ba10 2024-06-29T14:37:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.728+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1756260452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd41f4c40 con 0x7fcbd415ba10 2024-06-29T14:37:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.729+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1756260452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd41f0df0 con 0x7fcbd415ba10 2024-06-29T14:37:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.729+0000 7fcbcaffd640 1 -- 172.21.15.81:0/1756260452 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fcbcc01c460 con 0x7fcbd415ba10 2024-06-29T14:37:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.730+0000 7fcbcaffd640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcbb80763f0 0x7fcbb8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.730+0000 7fcbcaffd640 1 -- 172.21.15.81:0/1756260452 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5703+0+0 (secure 0 0 0) 0x7fcbcc097740 con 0x7fcbd415ba10 2024-06-29T14:37:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.730+0000 7fcbda57d640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcbb80763f0 0x7fcbb8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.730+0000 7fcbda57d640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcbb80763f0 0x7fcbb8078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0009f20 tx=0x7fcbd00094f0 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:12.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.734+0000 7fcbcaffd640 1 -- 172.21.15.81:0/1756260452 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcbd41f0df0 con 0x7fcbd415ba10 2024-06-29T14:37:12.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:12.986+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1756260452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) -- 0x7fcbd406fb10 con 0x7fcbd415ba10 2024-06-29T14:37:13.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.274+0000 7fcbcaffd640 1 -- 172.21.15.81:0/1756260452 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v525) ==== 154+0+0 (secure 0 0 0) 0x7fcbd4073980 con 0x7fcbd415ba10 2024-06-29T14:37:13.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.276+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcbb80763f0 msgr2=0x7fcbb8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.276+0000 7fcbdc7e1640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcbb80763f0 0x7fcbb8078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0009f20 tx=0x7fcbd00094f0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.276+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1756260452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbd415ba10 msgr2=0x7fcbd413ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.276+0000 7fcbdc7e1640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbd415ba10 0x7fcbd413ed30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00c390 tx=0x7fcbcc00c850 comp rx=0 tx=0).stop 2024-06-29T14:37:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.276+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1756260452 shutdown_connections 2024-06-29T14:37:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.276+0000 7fcbdc7e1640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcbb80763f0 0x7fcbb8078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.276+0000 7fcbdc7e1640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbd415ba10 0x7fcbd413ed30 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.276+0000 7fcbdc7e1640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbd413dd70 0x7fcbd413e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.276+0000 7fcbdc7e1640 1 --2- 172.21.15.81:0/1756260452 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbd4073570 0x7fcbd413a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.277+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1756260452 >> 172.21.15.81:0/1756260452 conn(0x7fcbd406ccc0 msgr2=0x7fcbd406ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.277+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1756260452 shutdown_connections 2024-06-29T14:37:13.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.277+0000 7fcbdc7e1640 1 -- 172.21.15.81:0/1756260452 wait complete. 2024-06-29T14:37:13.287 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2024-06-29T14:37:13.287 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2024-06-29T14:37:13.287 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:37:13.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:13.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.405+0000 7f50abcd6640 1 -- 172.21.15.81:0/1628679612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5090003f90 msgr2=0x7f5090004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.405+0000 7f50abcd6640 1 --2- 172.21.15.81:0/1628679612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5090003f90 0x7f5090004800 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5094001740 tx=0x7f5090005830 comp rx=0 tx=0).stop 2024-06-29T14:37:13.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.405+0000 7f50aa273640 1 -- 172.21.15.81:0/1628679612 reap_dead start 2024-06-29T14:37:13.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.406+0000 7f50abcd6640 1 -- 172.21.15.81:0/1628679612 shutdown_connections 2024-06-29T14:37:13.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.406+0000 7f50abcd6640 1 -- 172.21.15.81:0/1628679612 >> 172.21.15.81:0/1628679612 conn(0x7f50a406ccc0 msgr2=0x7f50a40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:13.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.406+0000 7f50abcd6640 1 -- 172.21.15.81:0/1628679612 shutdown_connections 2024-06-29T14:37:13.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.406+0000 7f50abcd6640 1 -- 172.21.15.81:0/1628679612 wait complete. 2024-06-29T14:37:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.407+0000 7f50abcd6640 1 Processor -- start 2024-06-29T14:37:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.407+0000 7f50abcd6640 1 -- start start 2024-06-29T14:37:13.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.409+0000 7f50abcd6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5090003f90 0x7f50a40a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.409+0000 7f50abcd6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50a41400c0 0x7f50a4157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.409+0000 7f50abcd6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50a4137fa0 0x7f50a41f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.409+0000 7f50abcd6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f50a40cf320 con 0x7f5090003f90 2024-06-29T14:37:13.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.409+0000 7f50abcd6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f50a40cf1a0 con 0x7f50a41400c0 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.409+0000 7f50abcd6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f50a415f2e0 con 0x7f50a4137fa0 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.409+0000 7f50a9a72640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5090003f90 0x7f50a40a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.409+0000 7f50a9a72640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5090003f90 0x7f50a40a9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42394/0 (socket says 172.21.15.81:42394) 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.409+0000 7f50a9a72640 1 -- 172.21.15.81:0/3053052779 learned_addr learned my addr 172.21.15.81:0/3053052779 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50a9271640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50a41400c0 0x7f50a4157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50aa273640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50a4137fa0 0x7f50a41f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50a9a72640 1 -- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50a4137fa0 msgr2=0x7f50a41f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50a9a72640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50a4137fa0 0x7f50a41f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50a9a72640 1 -- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50a41400c0 msgr2=0x7f50a4157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50a9a72640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50a41400c0 0x7f50a4157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50a9a72640 1 -- 172.21.15.81:0/3053052779 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a4157640 con 0x7f5090003f90 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50a9271640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50a41400c0 0x7f50a4157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50aa273640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50a4137fa0 0x7f50a41f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:13.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50a9a72640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5090003f90 0x7f50a40a9eb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f50a0011ae0 tx=0x7f50a0009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:13.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f509a7fc640 1 -- 172.21.15.81:0/3053052779 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a001e070 con 0x7f5090003f90 2024-06-29T14:37:13.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f509a7fc640 1 -- 172.21.15.81:0/3053052779 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f50a000fe70 con 0x7f5090003f90 2024-06-29T14:37:13.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50abcd6640 1 -- 172.21.15.81:0/3053052779 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a4156060 con 0x7f5090003f90 2024-06-29T14:37:13.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f509a7fc640 1 -- 172.21.15.81:0/3053052779 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0004290 con 0x7f5090003f90 2024-06-29T14:37:13.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.410+0000 7f50abcd6640 1 -- 172.21.15.81:0/3053052779 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a4156620 con 0x7f5090003f90 2024-06-29T14:37:13.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.411+0000 7f50abcd6640 1 -- 172.21.15.81:0/3053052779 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a40cce40 con 0x7f5090003f90 2024-06-29T14:37:13.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.412+0000 7f509a7fc640 1 -- 172.21.15.81:0/3053052779 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f50a0021070 con 0x7f5090003f90 2024-06-29T14:37:13.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.412+0000 7f509a7fc640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f50880763f0 0x7f5088078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.412+0000 7f509a7fc640 1 -- 172.21.15.81:0/3053052779 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5703+0+0 (secure 0 0 0) 0x7f50a0098490 con 0x7f5090003f90 2024-06-29T14:37:13.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.412+0000 7f50a9271640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f50880763f0 0x7f5088078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.413+0000 7f50a9271640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f50880763f0 0x7f5088078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5094001710 tx=0x7f5094001270 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:13.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.417+0000 7f509a7fc640 1 -- 172.21.15.81:0/3053052779 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f50a40cce40 con 0x7f5090003f90 2024-06-29T14:37:13.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.598+0000 7f50abcd6640 1 -- 172.21.15.81:0/3053052779 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50a415e960 con 0x7f5090003f90 2024-06-29T14:37:13.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.599+0000 7f509a7fc640 1 -- 172.21.15.81:0/3053052779 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+14138 (secure 0 0 0) 0x7f50a00615b0 con 0x7f5090003f90 2024-06-29T14:37:13.598 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:13.598 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":70,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:12.264467+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6835/2669979711":"2024-06-30T14:37:09.087036+0000","172.21.15.167:0/375340026":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/3632236021":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/1756060870":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/1790794665":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/183639693":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/3418120263":"2024-06-30T14:37:06.907833+0000","172.21.15.167:6833/1090782941":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6832/1090782941":"2024-06-30T14:35:10.908137+0000","172.21.15.81:6833/2669979711":"2024-06-30T14:37:09.087036+0000","172.21.15.167:0/1626573527":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6833/2493228728":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/799531669":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/2483549331":"2024-06-30T14:35:10.908137+0000","172.21.15.167:6832/2493228728":"2024-06-30T14:37:06.907833+0000","172.21.15.167:0/2831660161":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/3380278063":"2024-06-30T14:35:10.908137+0000","172.21.15.167:0/605629739":"2024-06-30T14:35:10.908137+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:13.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 -- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f50880763f0 msgr2=0x7f5088078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f50880763f0 0x7f5088078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5094001710 tx=0x7f5094001270 comp rx=0 tx=0).stop 2024-06-29T14:37:13.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 -- 172.21.15.81:0/3053052779 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5090003f90 msgr2=0x7f50a40a9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5090003f90 0x7f50a40a9eb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f50a0011ae0 tx=0x7f50a0009e90 comp rx=0 tx=0).stop 2024-06-29T14:37:13.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 -- 172.21.15.81:0/3053052779 shutdown_connections 2024-06-29T14:37:13.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f50880763f0 0x7f5088078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50a4137fa0 0x7f50a41f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50a41400c0 0x7f50a4157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 --2- 172.21.15.81:0/3053052779 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5090003f90 0x7f50a40a9eb0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 -- 172.21.15.81:0/3053052779 >> 172.21.15.81:0/3053052779 conn(0x7f50a406ccc0 msgr2=0x7f50a406f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:13.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 -- 172.21.15.81:0/3053052779 shutdown_connections 2024-06-29T14:37:13.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.601+0000 7f50abcd6640 1 -- 172.21.15.81:0/3053052779 wait complete. 2024-06-29T14:37:13.611 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:13.611 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.729+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1331406528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b0003c10 msgr2=0x7f72b0004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.729+0000 7f72cf1eb640 1 --2- 172.21.15.81:0/1331406528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b0003c10 0x7f72b0004060 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f72c4006f70 tx=0x7f72b0005830 comp rx=0 tx=0).stop 2024-06-29T14:37:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.729+0000 7f72cd788640 1 -- 172.21.15.81:0/1331406528 reap_dead start 2024-06-29T14:37:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.729+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1331406528 shutdown_connections 2024-06-29T14:37:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.729+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1331406528 >> 172.21.15.81:0/1331406528 conn(0x7f72c80d0930 msgr2=0x7f72c80d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.729+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1331406528 shutdown_connections 2024-06-29T14:37:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.729+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1331406528 wait complete. 2024-06-29T14:37:13.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.731+0000 7f72cf1eb640 1 Processor -- start 2024-06-29T14:37:13.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.731+0000 7f72cf1eb640 1 -- start start 2024-06-29T14:37:13.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72cf1eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b0003c10 0x7f72c80c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72cf1eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72c81579d0 0x7f72c80ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72cf1eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72c80aebb0 0x7f72c80cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72cf1eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72c80b41f0 con 0x7f72b0003c10 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72cf1eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f72c80b4070 con 0x7f72c81579d0 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72cf1eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f72c80b4370 con 0x7f72c80aebb0 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72ccf87640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b0003c10 0x7f72c80c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72ccf87640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b0003c10 0x7f72c80c9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42420/0 (socket says 172.21.15.81:42420) 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72ccf87640 1 -- 172.21.15.81:0/1003767263 learned_addr learned my addr 172.21.15.81:0/1003767263 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72bffff640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72c81579d0 0x7f72c80ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72cd788640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72c80aebb0 0x7f72c80cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72ccf87640 1 -- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72c80aebb0 msgr2=0x7f72c80cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72ccf87640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72c80aebb0 0x7f72c80cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72ccf87640 1 -- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72c81579d0 msgr2=0x7f72c80ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72ccf87640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72c81579d0 0x7f72c80ae670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72ccf87640 1 -- 172.21.15.81:0/1003767263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c80c5970 con 0x7f72b0003c10 2024-06-29T14:37:13.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72ccf87640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b0003c10 0x7f72c80c9aa0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f72c4006f70 tx=0x7f72c4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:13.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72bd7fa640 1 -- 172.21.15.81:0/1003767263 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4032030 con 0x7f72b0003c10 2024-06-29T14:37:13.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72bd7fa640 1 -- 172.21.15.81:0/1003767263 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72c4003af0 con 0x7f72b0003c10 2024-06-29T14:37:13.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72bd7fa640 1 -- 172.21.15.81:0/1003767263 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4042480 con 0x7f72b0003c10 2024-06-29T14:37:13.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.733+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1003767263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c80c5c10 con 0x7f72b0003c10 2024-06-29T14:37:13.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.734+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1003767263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c80c61d0 con 0x7f72b0003c10 2024-06-29T14:37:13.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.735+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1003767263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72c806e2a0 con 0x7f72b0003c10 2024-06-29T14:37:13.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.735+0000 7f72bd7fa640 1 -- 172.21.15.81:0/1003767263 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f72c4035020 con 0x7f72b0003c10 2024-06-29T14:37:13.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.736+0000 7f72bd7fa640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72b40763f0 0x7f72b4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:13.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.736+0000 7f72bd7fa640 1 -- 172.21.15.81:0/1003767263 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5703+0+0 (secure 0 0 0) 0x7f72c40b8400 con 0x7f72b0003c10 2024-06-29T14:37:13.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.736+0000 7f72bffff640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72b40763f0 0x7f72b4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:13.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.737+0000 7f72bffff640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72b40763f0 0x7f72b4078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f72c80cf9a0 tx=0x7f72b8006b10 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.740+0000 7f72bd7fa640 1 -- 172.21.15.81:0/1003767263 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f72c806e2a0 con 0x7f72b0003c10 2024-06-29T14:37:13.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.921+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1003767263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f72c80b4870 con 0x7f72b0003c10 2024-06-29T14:37:13.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.921+0000 7f72bd7fa640 1 -- 172.21.15.81:0/1003767263 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) ==== 93+0+10 (secure 0 0 0) 0x7f72c4081ac0 con 0x7f72b0003c10 2024-06-29T14:37:13.920 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:37:13.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72b40763f0 msgr2=0x7f72b4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72b40763f0 0x7f72b4078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f72c80cf9a0 tx=0x7f72b8006b10 comp rx=0 tx=0).stop 2024-06-29T14:37:13.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1003767263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b0003c10 msgr2=0x7f72c80c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:13.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b0003c10 0x7f72c80c9aa0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f72c4006f70 tx=0x7f72c4001cb0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1003767263 shutdown_connections 2024-06-29T14:37:13.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72b40763f0 0x7f72b4078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72c80aebb0 0x7f72c80cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72c81579d0 0x7f72c80ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 --2- 172.21.15.81:0/1003767263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b0003c10 0x7f72c80c9aa0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:13.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1003767263 >> 172.21.15.81:0/1003767263 conn(0x7f72c80d0930 msgr2=0x7f72c80d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:13.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1003767263 shutdown_connections 2024-06-29T14:37:13.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:13.923+0000 7f72cf1eb640 1 -- 172.21.15.81:0/1003767263 wait complete. 2024-06-29T14:37:13.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.048+0000 7f90e56fd640 1 -- 172.21.15.81:0/4068691581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90cc004800 msgr2=0x7f90cc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.048+0000 7f90e56fd640 1 --2- 172.21.15.81:0/4068691581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90cc004800 0x7f90cc004c10 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f90e00cac90 tx=0x7f90cc005830 comp rx=0 tx=0).stop 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.048+0000 7f90df7fe640 1 -- 172.21.15.81:0/4068691581 reap_dead start 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.048+0000 7f90e56fd640 1 -- 172.21.15.81:0/4068691581 shutdown_connections 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.048+0000 7f90e56fd640 1 -- 172.21.15.81:0/4068691581 >> 172.21.15.81:0/4068691581 conn(0x7f90e009cf30 msgr2=0x7f90e00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.048+0000 7f90e56fd640 1 -- 172.21.15.81:0/4068691581 shutdown_connections 2024-06-29T14:37:14.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.048+0000 7f90e56fd640 1 -- 172.21.15.81:0/4068691581 wait complete. 2024-06-29T14:37:14.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.050+0000 7f90e56fd640 1 Processor -- start 2024-06-29T14:37:14.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.050+0000 7f90e56fd640 1 -- start start 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.051+0000 7f90e56fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90cc004800 0x7f90e0153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.051+0000 7f90e56fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90e01f40d0 0x7f90e015f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.051+0000 7f90e56fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90e015f980 0x7f90e014dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90e56fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f90e00b8440 con 0x7f90e01f40d0 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90e56fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f90e00b82c0 con 0x7f90e015f980 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90e56fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f90e00b85c0 con 0x7f90cc004800 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90de7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90e01f40d0 0x7f90e015f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90de7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90e01f40d0 0x7f90e015f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42452/0 (socket says 172.21.15.81:42452) 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90de7fc640 1 -- 172.21.15.81:0/4142241610 learned_addr learned my addr 172.21.15.81:0/4142241610 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90deffd640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90cc004800 0x7f90e0153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90df7fe640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90e015f980 0x7f90e014dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:14.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90de7fc640 1 -- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90cc004800 msgr2=0x7f90e0153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90de7fc640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90cc004800 0x7f90e0153d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90de7fc640 1 -- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90e015f980 msgr2=0x7f90e014dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90de7fc640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90e015f980 0x7f90e014dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90de7fc640 1 -- 172.21.15.81:0/4142241610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e015fd90 con 0x7f90e01f40d0 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90de7fc640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90e01f40d0 0x7f90e015f440 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f90c8000c00 tx=0x7f90c8001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90d77fe640 1 -- 172.21.15.81:0/4142241610 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c8031040 con 0x7f90e01f40d0 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90d77fe640 1 -- 172.21.15.81:0/4142241610 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90c80035f0 con 0x7f90e01f40d0 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90e56fd640 1 -- 172.21.15.81:0/4142241610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e01500d0 con 0x7f90e01f40d0 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90d77fe640 1 -- 172.21.15.81:0/4142241610 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c80039e0 con 0x7f90e01f40d0 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.052+0000 7f90e56fd640 1 -- 172.21.15.81:0/4142241610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e0150690 con 0x7f90e01f40d0 2024-06-29T14:37:14.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.054+0000 7f90e56fd640 1 -- 172.21.15.81:0/4142241610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e014fe40 con 0x7f90e01f40d0 2024-06-29T14:37:14.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.054+0000 7f90d77fe640 1 -- 172.21.15.81:0/4142241610 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f90c8034070 con 0x7f90e01f40d0 2024-06-29T14:37:14.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.054+0000 7f90d77fe640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f90c00763f0 0x7f90c0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:14.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.054+0000 7f90d77fe640 1 -- 172.21.15.81:0/4142241610 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5703+0+0 (secure 0 0 0) 0x7f90c80b7a90 con 0x7f90e01f40d0 2024-06-29T14:37:14.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.055+0000 7f90deffd640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f90c00763f0 0x7f90c0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:14.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.055+0000 7f90deffd640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f90c00763f0 0x7f90c0078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f90d8005fd0 tx=0x7f90d8009500 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:14.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.059+0000 7f90d77fe640 1 -- 172.21.15.81:0/4142241610 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f90e015fd90 con 0x7f90e01f40d0 2024-06-29T14:37:14.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:14.308+0000 7f90e56fd640 1 -- 172.21.15.81:0/4142241610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) -- 0x7f90e00d08c0 con 0x7f90e01f40d0 2024-06-29T14:37:15.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.283+0000 7f90d77fe640 1 -- 172.21.15.81:0/4142241610 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v527) ==== 159+0+0 (secure 0 0 0) 0x7f90c8081340 con 0x7f90e01f40d0 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 -- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f90c00763f0 msgr2=0x7f90c0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f90c00763f0 0x7f90c0078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f90d8005fd0 tx=0x7f90d8009500 comp rx=0 tx=0).stop 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 -- 172.21.15.81:0/4142241610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90e01f40d0 msgr2=0x7f90e015f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90e01f40d0 0x7f90e015f440 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f90c8000c00 tx=0x7f90c8001d00 comp rx=0 tx=0).stop 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 -- 172.21.15.81:0/4142241610 shutdown_connections 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f90c00763f0 0x7f90c0078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90e015f980 0x7f90e014dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90e01f40d0 0x7f90e015f440 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 --2- 172.21.15.81:0/4142241610 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90cc004800 0x7f90e0153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 -- 172.21.15.81:0/4142241610 >> 172.21.15.81:0/4142241610 conn(0x7f90e009cf30 msgr2=0x7f90e015e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 -- 172.21.15.81:0/4142241610 shutdown_connections 2024-06-29T14:37:15.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.286+0000 7f90e56fd640 1 -- 172.21.15.81:0/4142241610 wait complete. 2024-06-29T14:37:15.298 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:37:15.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.417+0000 7fbb22c47640 1 -- 172.21.15.81:0/3578631730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb1c071f20 msgr2=0x7fbb1c144a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.417+0000 7fbb22c47640 1 --2- 172.21.15.81:0/3578631730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb1c071f20 0x7fbb1c144a20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009830 tx=0x7fbb1002a560 comp rx=0 tx=0).stop 2024-06-29T14:37:15.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.417+0000 7fbb22c47640 1 -- 172.21.15.81:0/3578631730 shutdown_connections 2024-06-29T14:37:15.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.417+0000 7fbb22c47640 1 --2- 172.21.15.81:0/3578631730 >> v2:172.21.15.167:3300/0 conn(0x7fbb1c145180 0x7fbb1c06edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.417+0000 7fbb22c47640 1 --2- 172.21.15.81:0/3578631730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb1c071f20 0x7fbb1c144a20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.417+0000 7fbb22c47640 1 -- 172.21.15.81:0/3578631730 >> 172.21.15.81:0/3578631730 conn(0x7fbb1c06ccc0 msgr2=0x7fbb1c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:15.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.417+0000 7fbb22c47640 1 -- 172.21.15.81:0/3578631730 shutdown_connections 2024-06-29T14:37:15.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.417+0000 7fbb22c47640 1 -- 172.21.15.81:0/3578631730 wait complete. 2024-06-29T14:37:15.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.419+0000 7fbb22c47640 1 Processor -- start 2024-06-29T14:37:15.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.419+0000 7fbb22c47640 1 -- start start 2024-06-29T14:37:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.420+0000 7fbb22c47640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb1c071f20 0x7fbb1c15a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.420+0000 7fbb22c47640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb1c145180 0x7fbb1c0aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.420+0000 7fbb22c47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb1c145590 0x7fbb1c0abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.420+0000 7fbb22c47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c06f720 con 0x7fbb1c145590 2024-06-29T14:37:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.420+0000 7fbb22c47640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c06f5a0 con 0x7fbb1c071f20 2024-06-29T14:37:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.420+0000 7fbb22c47640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1c160050 con 0x7fbb1c145180 2024-06-29T14:37:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.420+0000 7fbb211e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb1c145590 0x7fbb1c0abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.420+0000 7fbb211e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb1c145590 0x7fbb1c0abfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42464/0 (socket says 172.21.15.81:42464) 2024-06-29T14:37:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb211e4640 1 -- 172.21.15.81:0/3930726643 learned_addr learned my addr 172.21.15.81:0/3930726643 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb1bfff640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb1c145180 0x7fbb1c0aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb209e3640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb1c071f20 0x7fbb1c15a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb211e4640 1 -- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb1c145180 msgr2=0x7fbb1c0aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb211e4640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb1c145180 0x7fbb1c0aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb211e4640 1 -- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb1c071f20 msgr2=0x7fbb1c15a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb211e4640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb1c071f20 0x7fbb1c15a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb211e4640 1 -- 172.21.15.81:0/3930726643 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c13e450 con 0x7fbb1c145590 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb209e3640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb1c071f20 0x7fbb1c15a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb211e4640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb1c145590 0x7fbb1c0abfb0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400cad0 tx=0x7fbb1400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb197fa640 1 -- 172.21.15.81:0/3930726643 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb14016020 con 0x7fbb1c145590 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb197fa640 1 -- 172.21.15.81:0/3930726643 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb1400ad20 con 0x7fbb1c145590 2024-06-29T14:37:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb197fa640 1 -- 172.21.15.81:0/3930726643 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb14012680 con 0x7fbb1c145590 2024-06-29T14:37:15.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb22c47640 1 -- 172.21.15.81:0/3930726643 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1c15b130 con 0x7fbb1c145590 2024-06-29T14:37:15.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.421+0000 7fbb22c47640 1 -- 172.21.15.81:0/3930726643 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb1c13e710 con 0x7fbb1c145590 2024-06-29T14:37:15.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.422+0000 7fbb22c47640 1 -- 172.21.15.81:0/3930726643 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb1c160510 con 0x7fbb1c145590 2024-06-29T14:37:15.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.423+0000 7fbb197fa640 1 -- 172.21.15.81:0/3930726643 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fbb14019020 con 0x7fbb1c145590 2024-06-29T14:37:15.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.423+0000 7fbb197fa640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbb000763f0 0x7fbb00078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.423+0000 7fbb197fa640 1 -- 172.21.15.81:0/3930726643 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5703+0+0 (secure 0 0 0) 0x7fbb14097f30 con 0x7fbb1c145590 2024-06-29T14:37:15.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.423+0000 7fbb209e3640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbb000763f0 0x7fbb00078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.424+0000 7fbb209e3640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbb000763f0 0x7fbb00078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb10034000 tx=0x7fbb10004730 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:15.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.428+0000 7fbb197fa640 1 -- 172.21.15.81:0/3930726643 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb14020020 con 0x7fbb1c145590 2024-06-29T14:37:15.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.613+0000 7fbb22c47640 1 -- 172.21.15.81:0/3930726643 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fbb1c06eba0 con 0x7fbb1c145590 2024-06-29T14:37:15.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.613+0000 7fbb197fa640 1 -- 172.21.15.81:0/3930726643 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v35) ==== 59+0+1998 (secure 0 0 0) 0x7fbb1c15c350 con 0x7fbb1c145590 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T14:37:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T14:37:15.613 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:37:15.614 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:37:15.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 -- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbb000763f0 msgr2=0x7fbb00078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbb000763f0 0x7fbb00078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb10034000 tx=0x7fbb10004730 comp rx=0 tx=0).stop 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 -- 172.21.15.81:0/3930726643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb1c145590 msgr2=0x7fbb1c0abfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb1c145590 0x7fbb1c0abfb0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400cad0 tx=0x7fbb1400cf90 comp rx=0 tx=0).stop 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 -- 172.21.15.81:0/3930726643 shutdown_connections 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fbb000763f0 0x7fbb00078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb1c145590 0x7fbb1c0abfb0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb1c145180 0x7fbb1c0aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 --2- 172.21.15.81:0/3930726643 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb1c071f20 0x7fbb1c15a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 -- 172.21.15.81:0/3930726643 >> 172.21.15.81:0/3930726643 conn(0x7fbb1c06ccc0 msgr2=0x7fbb1c0b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.615+0000 7fbb22c47640 1 -- 172.21.15.81:0/3930726643 shutdown_connections 2024-06-29T14:37:15.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.616+0000 7fbb22c47640 1 -- 172.21.15.81:0/3930726643 wait complete. 2024-06-29T14:37:15.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:37:15.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.742+0000 7f52619e0640 1 -- 172.21.15.81:0/2587224936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f525c0b9040 msgr2=0x7f525c0d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.742+0000 7f52619e0640 1 --2- 172.21.15.81:0/2587224936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f525c0b9040 0x7f525c0d0480 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f524c009830 tx=0x7f524c02a5a0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.742+0000 7f52619e0640 1 -- 172.21.15.81:0/2587224936 shutdown_connections 2024-06-29T14:37:15.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.742+0000 7f52619e0640 1 --2- 172.21.15.81:0/2587224936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f525c0b9040 0x7f525c0d0480 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.743+0000 7f52619e0640 1 -- 172.21.15.81:0/2587224936 >> 172.21.15.81:0/2587224936 conn(0x7f525c09cf30 msgr2=0x7f525c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:15.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.743+0000 7f52619e0640 1 -- 172.21.15.81:0/2587224936 shutdown_connections 2024-06-29T14:37:15.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.743+0000 7f52619e0640 1 -- 172.21.15.81:0/2587224936 wait complete. 2024-06-29T14:37:15.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.744+0000 7f52619e0640 1 Processor -- start 2024-06-29T14:37:15.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.744+0000 7f52619e0640 1 -- start start 2024-06-29T14:37:15.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f52619e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f525c0b5250 0x7f525c15c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f52619e0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f525c0b9040 0x7f525c0b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f52619e0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f525c0d2a90 0x7f525c0a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f52619e0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f525c0d0d50 con 0x7f525c0b5250 2024-06-29T14:37:15.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f52619e0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f525c0d0bd0 con 0x7f525c0d2a90 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f52619e0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f525c0b80b0 con 0x7f525c0b9040 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f525affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f525c0b5250 0x7f525c15c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f525affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f525c0b5250 0x7f525c15c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42484/0 (socket says 172.21.15.81:42484) 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f525affd640 1 -- 172.21.15.81:0/3986066175 learned_addr learned my addr 172.21.15.81:0/3986066175 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f525b7fe640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f525c0d2a90 0x7f525c0a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f52527fc640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f525c0b9040 0x7f525c0b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f525affd640 1 -- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f525c0b9040 msgr2=0x7f525c0b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f525affd640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f525c0b9040 0x7f525c0b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f525affd640 1 -- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f525c0d2a90 msgr2=0x7f525c0a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f525affd640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f525c0d2a90 0x7f525c0a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f525affd640 1 -- 172.21.15.81:0/3986066175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c1f5420 con 0x7f525c0b5250 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f52527fc640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f525c0b9040 0x7f525c0b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:15.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.746+0000 7f525affd640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f525c0b5250 0x7f525c15c110 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f524c000c00 tx=0x7f524c033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:15.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.747+0000 7f5258ff9640 1 -- 172.21.15.81:0/3986066175 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c031d60 con 0x7f525c0b5250 2024-06-29T14:37:15.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.747+0000 7f5258ff9640 1 -- 172.21.15.81:0/3986066175 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f524c043070 con 0x7f525c0b5250 2024-06-29T14:37:15.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.747+0000 7f5258ff9640 1 -- 172.21.15.81:0/3986066175 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c0332b0 con 0x7f525c0b5250 2024-06-29T14:37:15.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.747+0000 7f52619e0640 1 -- 172.21.15.81:0/3986066175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525c0b1a20 con 0x7f525c0b5250 2024-06-29T14:37:15.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.747+0000 7f52619e0640 1 -- 172.21.15.81:0/3986066175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525c0b45e0 con 0x7f525c0b5250 2024-06-29T14:37:15.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.748+0000 7f52619e0640 1 -- 172.21.15.81:0/3986066175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f525c0d0480 con 0x7f525c0b5250 2024-06-29T14:37:15.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.748+0000 7f5258ff9640 1 -- 172.21.15.81:0/3986066175 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f524c033450 con 0x7f525c0b5250 2024-06-29T14:37:15.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.749+0000 7f5258ff9640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f523c076420 0x7f523c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:15.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.749+0000 7f5258ff9640 1 -- 172.21.15.81:0/3986066175 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5703+0+0 (secure 0 0 0) 0x7f524c0b89d0 con 0x7f525c0b5250 2024-06-29T14:37:15.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.749+0000 7f52527fc640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f523c076420 0x7f523c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:15.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.750+0000 7f52527fc640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f523c076420 0x7f523c078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f525c0d2660 tx=0x7f5244009040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:15.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.753+0000 7f5258ff9640 1 -- 172.21.15.81:0/3986066175 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f524c082080 con 0x7f525c0b5250 2024-06-29T14:37:15.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.931+0000 7f52619e0640 1 -- 172.21.15.81:0/3986066175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f525c0d0480 con 0x7f525c0b5250 2024-06-29T14:37:15.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.935+0000 7f5258ff9640 1 -- 172.21.15.81:0/3986066175 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) ==== 122+0+0 (secure 0 0 0) 0x7f524c08a100 con 0x7f525c0b5250 2024-06-29T14:37:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 -- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f523c076420 msgr2=0x7f523c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f523c076420 0x7f523c078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f525c0d2660 tx=0x7f5244009040 comp rx=0 tx=0).stop 2024-06-29T14:37:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 -- 172.21.15.81:0/3986066175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f525c0b5250 msgr2=0x7f525c15c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f525c0b5250 0x7f525c15c110 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f524c000c00 tx=0x7f524c033d60 comp rx=0 tx=0).stop 2024-06-29T14:37:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 -- 172.21.15.81:0/3986066175 shutdown_connections 2024-06-29T14:37:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f523c076420 0x7f523c078930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f525c0d2a90 0x7f525c0a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f525c0b9040 0x7f525c0b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 --2- 172.21.15.81:0/3986066175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f525c0b5250 0x7f525c15c110 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:15.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 -- 172.21.15.81:0/3986066175 >> 172.21.15.81:0/3986066175 conn(0x7f525c09cf30 msgr2=0x7f525c15d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:15.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 -- 172.21.15.81:0/3986066175 shutdown_connections 2024-06-29T14:37:15.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:15.937+0000 7f52619e0640 1 -- 172.21.15.81:0/3986066175 wait complete. 2024-06-29T14:37:15.947 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:37:15.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:15.983 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:16.025 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:37:16.025 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:16.025 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:16.041 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:16.041 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:37:16.041 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:16.085 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:16.085 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:16.086 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:16.101 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:16.102 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:37:16.102 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:16.144 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:16.144 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:37:16.144 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.258+0000 7fd7a428f640 1 -- 172.21.15.81:0/3459279479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c15e890 msgr2=0x7fd79c15b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.258+0000 7fd7a428f640 1 --2- 172.21.15.81:0/3459279479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c15e890 0x7fd79c15b9d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd78c005fd0 tx=0x7fd78c02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.258+0000 7fd7a428f640 1 -- 172.21.15.81:0/3459279479 shutdown_connections 2024-06-29T14:37:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.258+0000 7fd7a428f640 1 --2- 172.21.15.81:0/3459279479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c15bf10 0x7fd79c0732a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.258+0000 7fd7a428f640 1 --2- 172.21.15.81:0/3459279479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c15e890 0x7fd79c15b9d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.258+0000 7fd7a428f640 1 -- 172.21.15.81:0/3459279479 >> 172.21.15.81:0/3459279479 conn(0x7fd79c0b1ce0 msgr2=0x7fd79c0b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:16.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.258+0000 7fd7a428f640 1 -- 172.21.15.81:0/3459279479 shutdown_connections 2024-06-29T14:37:16.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.258+0000 7fd7a428f640 1 -- 172.21.15.81:0/3459279479 wait complete. 2024-06-29T14:37:16.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.260+0000 7fd7a428f640 1 Processor -- start 2024-06-29T14:37:16.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.260+0000 7fd7a428f640 1 -- start start 2024-06-29T14:37:16.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a428f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0737e0 0x7fd79c157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a428f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd79c15bf10 0x7fd79c0b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a428f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c15e890 0x7fd79c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a428f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c1412c0 con 0x7fd79c15e890 2024-06-29T14:37:16.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a428f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c141140 con 0x7fd79c15bf10 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a428f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd79c0ce550 con 0x7fd79c0737e0 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a282c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c15e890 0x7fd79c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a282c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c15e890 0x7fd79c157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42496/0 (socket says 172.21.15.81:42496) 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a282c640 1 -- 172.21.15.81:0/1629574479 learned_addr learned my addr 172.21.15.81:0/1629574479 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a202b640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0737e0 0x7fd79c157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.261+0000 7fd7a182a640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd79c15bf10 0x7fd79c0b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd7a282c640 1 -- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0737e0 msgr2=0x7fd79c157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd7a282c640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0737e0 0x7fd79c157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd7a282c640 1 -- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd79c15bf10 msgr2=0x7fd79c0b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd7a282c640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd79c15bf10 0x7fd79c0b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd7a282c640 1 -- 172.21.15.81:0/1629574479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c0b3e20 con 0x7fd79c15e890 2024-06-29T14:37:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd7a282c640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c15e890 0x7fd79c157fd0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c3f0 tx=0x7fd79400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:16.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd79affd640 1 -- 172.21.15.81:0/1629574479 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794017070 con 0x7fd79c15e890 2024-06-29T14:37:16.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd79affd640 1 -- 172.21.15.81:0/1629574479 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7940035f0 con 0x7fd79c15e890 2024-06-29T14:37:16.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd7a428f640 1 -- 172.21.15.81:0/1629574479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd79c13d650 con 0x7fd79c15e890 2024-06-29T14:37:16.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd79affd640 1 -- 172.21.15.81:0/1629574479 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794003a40 con 0x7fd79c15e890 2024-06-29T14:37:16.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.262+0000 7fd7a428f640 1 -- 172.21.15.81:0/1629574479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd79c155a00 con 0x7fd79c15e890 2024-06-29T14:37:16.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.263+0000 7fd7a428f640 1 -- 172.21.15.81:0/1629574479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd79c155830 con 0x7fd79c15e890 2024-06-29T14:37:16.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.264+0000 7fd79affd640 1 -- 172.21.15.81:0/1629574479 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fd79401a070 con 0x7fd79c15e890 2024-06-29T14:37:16.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.264+0000 7fd79affd640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd7880763f0 0x7fd788078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.264+0000 7fd79affd640 1 -- 172.21.15.81:0/1629574479 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5703+0+0 (secure 0 0 0) 0x7fd794096dd0 con 0x7fd79c15e890 2024-06-29T14:37:16.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.265+0000 7fd7a202b640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd7880763f0 0x7fd788078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.265+0000 7fd7a202b640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd7880763f0 0x7fd788078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd78c034000 tx=0x7fd78c004ab0 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:16.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.268+0000 7fd79affd640 1 -- 172.21.15.81:0/1629574479 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd794021020 con 0x7fd79c15e890 2024-06-29T14:37:16.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.483+0000 7fd7a428f640 1 -- 172.21.15.81:0/1629574479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd79c15bcc0 con 0x7fd79c15e890 2024-06-29T14:37:16.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.484+0000 7fd79affd640 1 -- 172.21.15.81:0/1629574479 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+3914 (secure 0 0 0) 0x7fd79c15bcc0 con 0x7fd79c15e890 2024-06-29T14:37:16.483 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:16.483 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":27,"btime":"2024-06-29T14:37:10:089066+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5632,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2980523942","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2980523942},{"type":"v1","addr":"172.21.15.167:6836","nonce":2980523942}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5635,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2317403375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2317403375},{"type":"v1","addr":"172.21.15.167:6837","nonce":2317403375}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6030,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/2346221458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2346221458},{"type":"v1","addr":"172.21.15.81:6834","nonce":2346221458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":6468,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/265244901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":265244901},{"type":"v1","addr":"172.21.15.81:6835","nonce":265244901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[]} 2024-06-29T14:37:16.483 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 27 2024-06-29T14:37:16.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 -- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd7880763f0 msgr2=0x7fd788078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd7880763f0 0x7fd788078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd78c034000 tx=0x7fd78c004ab0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 -- 172.21.15.81:0/1629574479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c15e890 msgr2=0x7fd79c157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c15e890 0x7fd79c157fd0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c3f0 tx=0x7fd79400c8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 -- 172.21.15.81:0/1629574479 shutdown_connections 2024-06-29T14:37:16.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd7880763f0 0x7fd788078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c15e890 0x7fd79c157fd0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd79c15bf10 0x7fd79c0b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 --2- 172.21.15.81:0/1629574479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0737e0 0x7fd79c157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 -- 172.21.15.81:0/1629574479 >> 172.21.15.81:0/1629574479 conn(0x7fd79c0b1ce0 msgr2=0x7fd79c0b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:16.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 -- 172.21.15.81:0/1629574479 shutdown_connections 2024-06-29T14:37:16.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.486+0000 7fd7a428f640 1 -- 172.21.15.81:0/1629574479 wait complete. 2024-06-29T14:37:16.496 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:37:16.496 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:37:16.496 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:37:16.496 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:16.497 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:37:16.497 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:37:16.497 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:37:16.497 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:16.497 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:37:16.497 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:37:16.497 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:37:16.498 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:16.498 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:37:16.498 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:37:16.498 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:37:16.498 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:16.506 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:37:16.506 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:37:16.509 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:37:16.509 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:37:16.510 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:37:16.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:37:16.511 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:37:16.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:37:16.551 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:37:16.552 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:16.553 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:37:16.554 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:16.554 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:37:16.555 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:16.555 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:37:16.555 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:16.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:37:16.639 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T14:37:16.639 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:37:16.647 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T14:37:16.647 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:37:16.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.678+0000 7fc0faffd640 1 -- 172.21.15.81:0/1308003104 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0062d0 con 0x7fc0e4003a80 2024-06-29T14:37:16.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.679+0000 7fc102552640 1 -- 172.21.15.81:0/1308003104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0e4003a80 msgr2=0x7fc0e4003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.679+0000 7fc102552640 1 --2- 172.21.15.81:0/1308003104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0e4003a80 0x7fc0e4003ef0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0016c0 tx=0x7fc0e4005830 comp rx=0 tx=0).stop 2024-06-29T14:37:16.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.679+0000 7fc100aef640 1 -- 172.21.15.81:0/1308003104 reap_dead start 2024-06-29T14:37:16.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.679+0000 7fc102552640 1 -- 172.21.15.81:0/1308003104 shutdown_connections 2024-06-29T14:37:16.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.679+0000 7fc102552640 1 -- 172.21.15.81:0/1308003104 >> 172.21.15.81:0/1308003104 conn(0x7fc0fc09cf30 msgr2=0x7fc0fc0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:16.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.679+0000 7fc102552640 1 -- 172.21.15.81:0/1308003104 shutdown_connections 2024-06-29T14:37:16.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.679+0000 7fc102552640 1 -- 172.21.15.81:0/1308003104 wait complete. 2024-06-29T14:37:16.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.681+0000 7fc102552640 1 Processor -- start 2024-06-29T14:37:16.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.681+0000 7fc102552640 1 -- start start 2024-06-29T14:37:16.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.682+0000 7fc102552640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0e4003a80 0x7fc0fc0cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc102552640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0fc072af0 0x7fc0fc0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc102552640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0fc15a180 0x7fc0fc0cca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc102552640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc0d0e00 con 0x7fc0fc072af0 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc102552640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc0b84b0 con 0x7fc0fc15a180 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc102552640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc0fc0d1a80 con 0x7fc0e4003a80 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0fc072af0 0x7fc0fc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0fc072af0 0x7fc0fc0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42524/0 (socket says 172.21.15.81:42524) 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fb7fe640 1 -- 172.21.15.81:0/2832129165 learned_addr learned my addr 172.21.15.81:0/2832129165 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fbfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0e4003a80 0x7fc0fc0cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc100aef640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0fc15a180 0x7fc0fc0cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fb7fe640 1 -- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0e4003a80 msgr2=0x7fc0fc0cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fb7fe640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0e4003a80 0x7fc0fc0cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fb7fe640 1 -- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0fc15a180 msgr2=0x7fc0fc0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fb7fe640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0fc15a180 0x7fc0fc0cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fb7fe640 1 -- 172.21.15.81:0/2832129165 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0fc0732d0 con 0x7fc0fc072af0 2024-06-29T14:37:16.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc100aef640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0fc15a180 0x7fc0fc0cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:16.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fbfff640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0e4003a80 0x7fc0fc0cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:16.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0fb7fe640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0fc072af0 0x7fc0fc0a1710 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc0f00118b0 tx=0x7fc0f0011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:16.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0f8ff9640 1 -- 172.21.15.81:0/2832129165 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f000f940 con 0x7fc0fc072af0 2024-06-29T14:37:16.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0f8ff9640 1 -- 172.21.15.81:0/2832129165 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0f000fae0 con 0x7fc0fc072af0 2024-06-29T14:37:16.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc0f8ff9640 1 -- 172.21.15.81:0/2832129165 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0005020 con 0x7fc0fc072af0 2024-06-29T14:37:16.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc102552640 1 -- 172.21.15.81:0/2832129165 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0fc0735b0 con 0x7fc0fc072af0 2024-06-29T14:37:16.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.683+0000 7fc102552640 1 -- 172.21.15.81:0/2832129165 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0fc0ccf70 con 0x7fc0fc072af0 2024-06-29T14:37:16.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.684+0000 7fc102552640 1 -- 172.21.15.81:0/2832129165 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0fc0d0480 con 0x7fc0fc072af0 2024-06-29T14:37:16.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.685+0000 7fc0f8ff9640 1 -- 172.21.15.81:0/2832129165 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fc0f00051c0 con 0x7fc0fc072af0 2024-06-29T14:37:16.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.686+0000 7fc0f8ff9640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fc0e8076420 0x7fc0e8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:16.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.686+0000 7fc0f8ff9640 1 -- 172.21.15.81:0/2832129165 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5703+0+0 (secure 0 0 0) 0x7fc0f00988f0 con 0x7fc0fc072af0 2024-06-29T14:37:16.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.686+0000 7fc0fbfff640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fc0e8076420 0x7fc0e8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:16.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.687+0000 7fc0fbfff640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fc0e8076420 0x7fc0e8078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec032000 tx=0x7fc0ec001190 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:16.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.690+0000 7fc0f8ff9640 1 -- 172.21.15.81:0/2832129165 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0f0062110 con 0x7fc0fc072af0 2024-06-29T14:37:16.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-06-29T14:37:16.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:16.880+0000 7fc102552640 1 -- 172.21.15.81:0/2832129165 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fc0fc1616f0 con 0x7fc0fc072af0 2024-06-29T14:37:17.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.295+0000 7fc0f8ff9640 1 -- 172.21.15.81:0/2832129165 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v72) ==== 97+0+0 (secure 0 0 0) 0x7fc0f006a100 con 0x7fc0fc072af0 2024-06-29T14:37:17.294 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T14:37:17.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 -- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fc0e8076420 msgr2=0x7fc0e8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fc0e8076420 0x7fc0e8078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec032000 tx=0x7fc0ec001190 comp rx=0 tx=0).stop 2024-06-29T14:37:17.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 -- 172.21.15.81:0/2832129165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0fc072af0 msgr2=0x7fc0fc0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0fc072af0 0x7fc0fc0a1710 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc0f00118b0 tx=0x7fc0f0011d70 comp rx=0 tx=0).stop 2024-06-29T14:37:17.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 -- 172.21.15.81:0/2832129165 shutdown_connections 2024-06-29T14:37:17.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fc0e8076420 0x7fc0e8078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0fc15a180 0x7fc0fc0cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0fc072af0 0x7fc0fc0a1710 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 --2- 172.21.15.81:0/2832129165 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0e4003a80 0x7fc0fc0cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 -- 172.21.15.81:0/2832129165 >> 172.21.15.81:0/2832129165 conn(0x7fc0fc09cf30 msgr2=0x7fc0fc15d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:17.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 -- 172.21.15.81:0/2832129165 shutdown_connections 2024-06-29T14:37:17.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.298+0000 7fc102552640 1 -- 172.21.15.81:0/2832129165 wait complete. 2024-06-29T14:37:17.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:37:17.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.424+0000 7fde30bcb640 1 -- 172.21.15.81:0/3382275699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2c15c030 msgr2=0x7fde2c06c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.424+0000 7fde30bcb640 1 --2- 172.21.15.81:0/3382275699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2c15c030 0x7fde2c06c2f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0cd380 tx=0x7fde1402a560 comp rx=0 tx=0).stop 2024-06-29T14:37:17.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.424+0000 7fde30bcb640 1 -- 172.21.15.81:0/3382275699 shutdown_connections 2024-06-29T14:37:17.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.424+0000 7fde30bcb640 1 --2- 172.21.15.81:0/3382275699 >> v2:172.21.15.167:3300/0 conn(0x7fde2c15c440 0x7fde2c06c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.424+0000 7fde30bcb640 1 --2- 172.21.15.81:0/3382275699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2c15c030 0x7fde2c06c2f0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.424+0000 7fde30bcb640 1 --2- 172.21.15.81:0/3382275699 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2c15d750 0x7fde2c0afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.424+0000 7fde30bcb640 1 -- 172.21.15.81:0/3382275699 >> 172.21.15.81:0/3382275699 conn(0x7fde2c0d0930 msgr2=0x7fde2c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:17.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.424+0000 7fde30bcb640 1 -- 172.21.15.81:0/3382275699 shutdown_connections 2024-06-29T14:37:17.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.424+0000 7fde30bcb640 1 -- 172.21.15.81:0/3382275699 wait complete. 2024-06-29T14:37:17.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.426+0000 7fde30bcb640 1 Processor -- start 2024-06-29T14:37:17.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.426+0000 7fde30bcb640 1 -- start start 2024-06-29T14:37:17.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.427+0000 7fde30bcb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2c15c030 0x7fde2c14b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.427+0000 7fde30bcb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde2c15c440 0x7fde2c0ae7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde30bcb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2c15d750 0x7fde2c0b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde30bcb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c15fff0 con 0x7fde2c15d750 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde30bcb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c15fe70 con 0x7fde2c15c440 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde30bcb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fde2c160170 con 0x7fde2c15c030 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde2ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2c15d750 0x7fde2c0b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde2ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2c15d750 0x7fde2c0b4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42538/0 (socket says 172.21.15.81:42538) 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde2ad9d640 1 -- 172.21.15.81:0/1683875389 learned_addr learned my addr 172.21.15.81:0/1683875389 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde2a59c640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2c15c030 0x7fde2c14b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde21d9b640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde2c15c440 0x7fde2c0ae7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde2ad9d640 1 -- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2c15c030 msgr2=0x7fde2c14b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde2ad9d640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2c15c030 0x7fde2c14b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde2ad9d640 1 -- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde2c15c440 msgr2=0x7fde2c0ae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde2ad9d640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde2c15c440 0x7fde2c0ae7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde2ad9d640 1 -- 172.21.15.81:0/1683875389 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2c1566d0 con 0x7fde2c15d750 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde2ad9d640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2c15d750 0x7fde2c0b4790 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fde2400cac0 tx=0x7fde2400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:17.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde237fe640 1 -- 172.21.15.81:0/1683875389 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24007d90 con 0x7fde2c15d750 2024-06-29T14:37:17.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde237fe640 1 -- 172.21.15.81:0/1683875389 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde24003970 con 0x7fde2c15d750 2024-06-29T14:37:17.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde30bcb640 1 -- 172.21.15.81:0/1683875389 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2c1569b0 con 0x7fde2c15d750 2024-06-29T14:37:17.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde237fe640 1 -- 172.21.15.81:0/1683875389 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24003de0 con 0x7fde2c15d750 2024-06-29T14:37:17.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.428+0000 7fde30bcb640 1 -- 172.21.15.81:0/1683875389 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2c156f70 con 0x7fde2c15d750 2024-06-29T14:37:17.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.430+0000 7fde30bcb640 1 -- 172.21.15.81:0/1683875389 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde2c06e2a0 con 0x7fde2c15d750 2024-06-29T14:37:17.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.430+0000 7fde237fe640 1 -- 172.21.15.81:0/1683875389 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fde24004030 con 0x7fde2c15d750 2024-06-29T14:37:17.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.430+0000 7fde237fe640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fde0c076420 0x7fde0c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.431+0000 7fde237fe640 1 -- 172.21.15.81:0/1683875389 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 4929+0+0 (secure 0 0 0) 0x7fde24097c70 con 0x7fde2c15d750 2024-06-29T14:37:17.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.431+0000 7fde2a59c640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fde0c076420 0x7fde0c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.431+0000 7fde2a59c640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fde0c076420 0x7fde0c078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde1c005fd0 tx=0x7fde1c00a250 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:17.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.435+0000 7fde237fe640 1 -- 172.21.15.81:0/1683875389 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fde2c06e2a0 con 0x7fde2c15d750 2024-06-29T14:37:17.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.685+0000 7fde30bcb640 1 -- 172.21.15.81:0/1683875389 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fde2c0afb60 con 0x7fde2c15d750 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.685+0000 7fde237fe640 1 -- 172.21.15.81:0/1683875389 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v30) ==== 80+0+5304 (secure 0 0 0) 0x7fde24014070 con 0x7fde2c15d750 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:37:17.684 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.685 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T14:37:17.686 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.687 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.688 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.689 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.690 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.691 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T14:37:17.692 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:37:17.693 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.694 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T14:37:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:37:17.696 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T14:37:17.697 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:37:17.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 -- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fde0c076420 msgr2=0x7fde0c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fde0c076420 0x7fde0c078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde1c005fd0 tx=0x7fde1c00a250 comp rx=0 tx=0).stop 2024-06-29T14:37:17.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 -- 172.21.15.81:0/1683875389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2c15d750 msgr2=0x7fde2c0b4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2c15d750 0x7fde2c0b4790 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fde2400cac0 tx=0x7fde2400cf80 comp rx=0 tx=0).stop 2024-06-29T14:37:17.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 -- 172.21.15.81:0/1683875389 shutdown_connections 2024-06-29T14:37:17.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fde0c076420 0x7fde0c078930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2c15d750 0x7fde2c0b4790 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde2c15c440 0x7fde2c0ae7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 --2- 172.21.15.81:0/1683875389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2c15c030 0x7fde2c14b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 -- 172.21.15.81:0/1683875389 >> 172.21.15.81:0/1683875389 conn(0x7fde2c0d0930 msgr2=0x7fde2c0b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:17.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 -- 172.21.15.81:0/1683875389 shutdown_connections 2024-06-29T14:37:17.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.687+0000 7fde30bcb640 1 -- 172.21.15.81:0/1683875389 wait complete. 2024-06-29T14:37:17.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.814+0000 7f5ff45ac640 1 -- 172.21.15.81:0/1863292087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fec0b4cf0 msgr2=0x7f5fec0d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.814+0000 7f5ff45ac640 1 --2- 172.21.15.81:0/1863292087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fec0b4cf0 0x7f5fec0d00f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5fec0b2c10 tx=0x7f5fe402aa00 comp rx=0 tx=0).stop 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.815+0000 7f5ff45ac640 1 -- 172.21.15.81:0/1863292087 shutdown_connections 2024-06-29T14:37:17.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.815+0000 7f5ff45ac640 1 --2- 172.21.15.81:0/1863292087 >> v2:172.21.15.167:3300/0 conn(0x7f5fec0cd540 0x7f5fec0b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.815+0000 7f5ff45ac640 1 --2- 172.21.15.81:0/1863292087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fec0b4cf0 0x7f5fec0d00f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.815+0000 7f5ff45ac640 1 -- 172.21.15.81:0/1863292087 >> 172.21.15.81:0/1863292087 conn(0x7f5fec0d1960 msgr2=0x7f5fec0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.815+0000 7f5ff45ac640 1 -- 172.21.15.81:0/1863292087 shutdown_connections 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.815+0000 7f5ff45ac640 1 -- 172.21.15.81:0/1863292087 wait complete. 2024-06-29T14:37:17.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.816+0000 7f5ff45ac640 1 Processor -- start 2024-06-29T14:37:17.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.816+0000 7f5ff45ac640 1 -- start start 2024-06-29T14:37:17.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff45ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fec0b4cf0 0x7f5fec160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff45ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fec0b6d60 0x7f5fec0c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff45ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fec0cd540 0x7f5fec163990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff45ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5fec0d0630 con 0x7f5fec0cd540 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff45ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5fec0b2790 con 0x7f5fec0b6d60 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff45ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5fec0d07b0 con 0x7f5fec0b4cf0 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2b49640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fec0cd540 0x7f5fec163990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2b49640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fec0cd540 0x7f5fec163990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42554/0 (socket says 172.21.15.81:42554) 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2b49640 1 -- 172.21.15.81:0/3021533223 learned_addr learned my addr 172.21.15.81:0/3021533223 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff1b47640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fec0b6d60 0x7f5fec0c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2348640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fec0b4cf0 0x7f5fec160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2b49640 1 -- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fec0b4cf0 msgr2=0x7f5fec160940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2b49640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fec0b4cf0 0x7f5fec160940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2b49640 1 -- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fec0b6d60 msgr2=0x7f5fec0c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2b49640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fec0b6d60 0x7f5fec0c5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2b49640 1 -- 172.21.15.81:0/3021533223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fec0c65b0 con 0x7f5fec0cd540 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff1b47640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fec0b6d60 0x7f5fec0c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2348640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fec0b4cf0 0x7f5fec160940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff2b49640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fec0cd540 0x7f5fec163990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4005fd0 tx=0x7f5fe4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5fe2ffd640 1 -- 172.21.15.81:0/3021533223 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4033050 con 0x7f5fec0cd540 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5fe2ffd640 1 -- 172.21.15.81:0/3021533223 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5fe4004450 con 0x7f5fec0cd540 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.818+0000 7f5ff45ac640 1 -- 172.21.15.81:0/3021533223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fec0ca4d0 con 0x7f5fec0cd540 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.819+0000 7f5fe2ffd640 1 -- 172.21.15.81:0/3021533223 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4038450 con 0x7f5fec0cd540 2024-06-29T14:37:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.819+0000 7f5ff45ac640 1 -- 172.21.15.81:0/3021533223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fec0caa90 con 0x7f5fec0cd540 2024-06-29T14:37:17.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.820+0000 7f5ff45ac640 1 -- 172.21.15.81:0/3021533223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fec0d00f0 con 0x7f5fec0cd540 2024-06-29T14:37:17.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.820+0000 7f5fe2ffd640 1 -- 172.21.15.81:0/3021533223 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f5fe4036070 con 0x7f5fec0cd540 2024-06-29T14:37:17.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.821+0000 7f5fe2ffd640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5fd00763f0 0x7f5fd0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:17.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.821+0000 7f5fe2ffd640 1 -- 172.21.15.81:0/3021533223 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 4929+0+0 (secure 0 0 0) 0x7f5fe40b8b40 con 0x7f5fec0cd540 2024-06-29T14:37:17.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.821+0000 7f5ff2348640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5fd00763f0 0x7f5fd0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:17.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.822+0000 7f5ff2348640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5fd00763f0 0x7f5fd0078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8006fd0 tx=0x7f5fe8008040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:17.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:17.825+0000 7f5fe2ffd640 1 -- 172.21.15.81:0/3021533223 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5fe40824d0 con 0x7f5fec0cd540 2024-06-29T14:37:18.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.005+0000 7f5ff45ac640 1 -- 172.21.15.81:0/3021533223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5fec0cdec0 con 0x7f5fec0cd540 2024-06-29T14:37:18.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.006+0000 7f5fe2ffd640 1 -- 172.21.15.81:0/3021533223 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+12993 (secure 0 0 0) 0x7f5fe4086180 con 0x7f5fec0cd540 2024-06-29T14:37:18.005 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:18.005 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":74,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:17.310827+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 -- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5fd00763f0 msgr2=0x7f5fd0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5fd00763f0 0x7f5fd0078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8006fd0 tx=0x7f5fe8008040 comp rx=0 tx=0).stop 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 -- 172.21.15.81:0/3021533223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fec0cd540 msgr2=0x7f5fec163990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fec0cd540 0x7f5fec163990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4005fd0 tx=0x7f5fe4003f10 comp rx=0 tx=0).stop 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 -- 172.21.15.81:0/3021533223 shutdown_connections 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5fd00763f0 0x7f5fd0078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fec0cd540 0x7f5fec163990 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fec0b6d60 0x7f5fec0c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 --2- 172.21.15.81:0/3021533223 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fec0b4cf0 0x7f5fec160940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 -- 172.21.15.81:0/3021533223 >> 172.21.15.81:0/3021533223 conn(0x7f5fec0d1960 msgr2=0x7f5fec070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 -- 172.21.15.81:0/3021533223 shutdown_connections 2024-06-29T14:37:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.008+0000 7f5ff45ac640 1 -- 172.21.15.81:0/3021533223 wait complete. 2024-06-29T14:37:18.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:18.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.136+0000 7f91d69eb640 1 -- 172.21.15.81:0/1752163462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91b8004800 msgr2=0x7f91b8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.136+0000 7f91d69eb640 1 --2- 172.21.15.81:0/1752163462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91b8004800 0x7f91b8004c10 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009d20 tx=0x7f91b8005830 comp rx=0 tx=0).stop 2024-06-29T14:37:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.136+0000 7f91d4f88640 1 -- 172.21.15.81:0/1752163462 reap_dead start 2024-06-29T14:37:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.136+0000 7f91d69eb640 1 -- 172.21.15.81:0/1752163462 shutdown_connections 2024-06-29T14:37:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.136+0000 7f91d69eb640 1 -- 172.21.15.81:0/1752163462 >> 172.21.15.81:0/1752163462 conn(0x7f91d009cf30 msgr2=0x7f91d00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.136+0000 7f91d69eb640 1 -- 172.21.15.81:0/1752163462 shutdown_connections 2024-06-29T14:37:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.136+0000 7f91d69eb640 1 -- 172.21.15.81:0/1752163462 wait complete. 2024-06-29T14:37:18.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.138+0000 7f91d69eb640 1 Processor -- start 2024-06-29T14:37:18.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.138+0000 7f91d69eb640 1 -- start start 2024-06-29T14:37:18.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.139+0000 7f91d69eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91b8004800 0x7f91d00b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.139+0000 7f91d69eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91d00b1260 0x7f91d015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.139+0000 7f91d69eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91d00b36f0 0x7f91d00b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.139+0000 7f91d69eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f91d00d1fc0 con 0x7f91d00b36f0 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.139+0000 7f91d69eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f91d00b8500 con 0x7f91d00b1260 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.139+0000 7f91d69eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f91d00d2140 con 0x7f91b8004800 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.139+0000 7f91d4f88640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91d00b36f0 0x7f91d00b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.139+0000 7f91d4f88640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91d00b36f0 0x7f91d00b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42558/0 (socket says 172.21.15.81:42558) 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.139+0000 7f91d4f88640 1 -- 172.21.15.81:0/2475562846 learned_addr learned my addr 172.21.15.81:0/2475562846 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91cf7fe640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91d00b1260 0x7f91d015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91cffff640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91b8004800 0x7f91d00b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91d4f88640 1 -- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91b8004800 msgr2=0x7f91d00b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91d4f88640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91b8004800 0x7f91d00b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91d4f88640 1 -- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91d00b1260 msgr2=0x7f91d015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91d4f88640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91d00b1260 0x7f91d015c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91d4f88640 1 -- 172.21.15.81:0/2475562846 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d00b4040 con 0x7f91d00b36f0 2024-06-29T14:37:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91d4f88640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91d00b36f0 0x7f91d00b3b00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f91c8005fd0 tx=0x7f91c800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91ccff9640 1 -- 172.21.15.81:0/2475562846 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8017070 con 0x7f91d00b36f0 2024-06-29T14:37:18.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91ccff9640 1 -- 172.21.15.81:0/2475562846 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91c8015070 con 0x7f91d00b36f0 2024-06-29T14:37:18.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91d69eb640 1 -- 172.21.15.81:0/2475562846 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d009fa50 con 0x7f91d00b36f0 2024-06-29T14:37:18.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91ccff9640 1 -- 172.21.15.81:0/2475562846 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8005020 con 0x7f91d00b36f0 2024-06-29T14:37:18.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.140+0000 7f91d69eb640 1 -- 172.21.15.81:0/2475562846 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d009ffd0 con 0x7f91d00b36f0 2024-06-29T14:37:18.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.141+0000 7f91d69eb640 1 -- 172.21.15.81:0/2475562846 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91d006a450 con 0x7f91d00b36f0 2024-06-29T14:37:18.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.142+0000 7f91ccff9640 1 -- 172.21.15.81:0/2475562846 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f91c80051c0 con 0x7f91d00b36f0 2024-06-29T14:37:18.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.142+0000 7f91ccff9640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f91bc0763f0 0x7f91bc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.143+0000 7f91ccff9640 1 -- 172.21.15.81:0/2475562846 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 4929+0+0 (secure 0 0 0) 0x7f91c8098120 con 0x7f91d00b36f0 2024-06-29T14:37:18.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.143+0000 7f91cffff640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f91bc0763f0 0x7f91bc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.143+0000 7f91cffff640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f91bc0763f0 0x7f91bc078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f91c0001eb0 tx=0x7f91c0000f30 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.147+0000 7f91ccff9640 1 -- 172.21.15.81:0/2475562846 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f91d00b4040 con 0x7f91d00b36f0 2024-06-29T14:37:18.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.332+0000 7f91d69eb640 1 -- 172.21.15.81:0/2475562846 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f91d0069230 con 0x7f91d00b36f0 2024-06-29T14:37:18.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.332+0000 7f91ccff9640 1 -- 172.21.15.81:0/2475562846 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7f91c8061be0 con 0x7f91d00b36f0 2024-06-29T14:37:18.331 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:37:18.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 -- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f91bc0763f0 msgr2=0x7f91bc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f91bc0763f0 0x7f91bc078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f91c0001eb0 tx=0x7f91c0000f30 comp rx=0 tx=0).stop 2024-06-29T14:37:18.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 -- 172.21.15.81:0/2475562846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91d00b36f0 msgr2=0x7f91d00b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91d00b36f0 0x7f91d00b3b00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f91c8005fd0 tx=0x7f91c800e720 comp rx=0 tx=0).stop 2024-06-29T14:37:18.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 -- 172.21.15.81:0/2475562846 shutdown_connections 2024-06-29T14:37:18.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f91bc0763f0 0x7f91bc078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91d00b36f0 0x7f91d00b3b00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91d00b1260 0x7f91d015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 --2- 172.21.15.81:0/2475562846 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91b8004800 0x7f91d00b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 -- 172.21.15.81:0/2475562846 >> 172.21.15.81:0/2475562846 conn(0x7f91d009cf30 msgr2=0x7f91d015f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 -- 172.21.15.81:0/2475562846 shutdown_connections 2024-06-29T14:37:18.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.334+0000 7f91d69eb640 1 -- 172.21.15.81:0/2475562846 wait complete. 2024-06-29T14:37:18.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:37:18.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.460+0000 7f7478ee3640 1 -- 172.21.15.81:0/2865308033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74740b3200 msgr2=0x7f74740cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.460+0000 7f7478ee3640 1 --2- 172.21.15.81:0/2865308033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74740b3200 0x7f74740cc7b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7464005fd0 tx=0x7f746402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.460+0000 7f7478ee3640 1 -- 172.21.15.81:0/2865308033 shutdown_connections 2024-06-29T14:37:18.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.460+0000 7f7478ee3640 1 --2- 172.21.15.81:0/2865308033 >> v2:172.21.15.167:3300/0 conn(0x7f74740cccf0 0x7f74740d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.460+0000 7f7478ee3640 1 --2- 172.21.15.81:0/2865308033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74740b3200 0x7f74740cc7b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.460+0000 7f7478ee3640 1 -- 172.21.15.81:0/2865308033 >> 172.21.15.81:0/2865308033 conn(0x7f74740b3680 msgr2=0x7f74740b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.460+0000 7f7478ee3640 1 -- 172.21.15.81:0/2865308033 shutdown_connections 2024-06-29T14:37:18.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.460+0000 7f7478ee3640 1 -- 172.21.15.81:0/2865308033 wait complete. 2024-06-29T14:37:18.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.462+0000 7f7478ee3640 1 Processor -- start 2024-06-29T14:37:18.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.462+0000 7f7478ee3640 1 -- start start 2024-06-29T14:37:18.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f7478ee3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74740b3200 0x7f747406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f7478ee3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74740cccf0 0x7f7474159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f7478ee3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740d0770 0x7f74741595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f7478ee3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74740cedb0 con 0x7f74740b3200 2024-06-29T14:37:18.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f7478ee3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f74740cec30 con 0x7f74740d0770 2024-06-29T14:37:18.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f7478ee3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f74740cef30 con 0x7f74740cccf0 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f747259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74740b3200 0x7f747406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f747259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74740b3200 0x7f747406e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42574/0 (socket says 172.21.15.81:42574) 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f747259c640 1 -- 172.21.15.81:0/276815079 learned_addr learned my addr 172.21.15.81:0/276815079 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f7469d9b640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74740cccf0 0x7f7474159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f7472d9d640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740d0770 0x7f74741595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f747259c640 1 -- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74740cccf0 msgr2=0x7f7474159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.463+0000 7f747259c640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74740cccf0 0x7f7474159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.464+0000 7f747259c640 1 -- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740d0770 msgr2=0x7f74741595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.464+0000 7f747259c640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740d0770 0x7f74741595d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.464+0000 7f747259c640 1 -- 172.21.15.81:0/276815079 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747406ed70 con 0x7f74740b3200 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.464+0000 7f747259c640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74740b3200 0x7f747406e2f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f7464034040 tx=0x7f74640035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.464+0000 7f746b7fe640 1 -- 172.21.15.81:0/276815079 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746403e030 con 0x7f74740b3200 2024-06-29T14:37:18.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.464+0000 7f746b7fe640 1 -- 172.21.15.81:0/276815079 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7464003ae0 con 0x7f74740b3200 2024-06-29T14:37:18.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.464+0000 7f7478ee3640 1 -- 172.21.15.81:0/276815079 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74741582d0 con 0x7f74740b3200 2024-06-29T14:37:18.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.464+0000 7f746b7fe640 1 -- 172.21.15.81:0/276815079 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464035040 con 0x7f74740b3200 2024-06-29T14:37:18.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.464+0000 7f7478ee3640 1 -- 172.21.15.81:0/276815079 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7474158890 con 0x7f74740b3200 2024-06-29T14:37:18.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.465+0000 7f7478ee3640 1 -- 172.21.15.81:0/276815079 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74741e82b0 con 0x7f74740b3200 2024-06-29T14:37:18.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.466+0000 7f746b7fe640 1 -- 172.21.15.81:0/276815079 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f7464041030 con 0x7f74740b3200 2024-06-29T14:37:18.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.466+0000 7f746b7fe640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7454076420 0x7f7454078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.466+0000 7f746b7fe640 1 -- 172.21.15.81:0/276815079 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 4929+0+0 (secure 0 0 0) 0x7f74640b7ba0 con 0x7f74740b3200 2024-06-29T14:37:18.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.466+0000 7f7469d9b640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7454076420 0x7f7454078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.467+0000 7f7469d9b640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7454076420 0x7f7454078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f74741e7740 tx=0x7f745c005f90 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.471+0000 7f746b7fe640 1 -- 172.21.15.81:0/276815079 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f74741e82b0 con 0x7f74740b3200 2024-06-29T14:37:18.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.655+0000 7f7478ee3640 1 -- 172.21.15.81:0/276815079 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f74740cc7b0 con 0x7f74740b3200 2024-06-29T14:37:18.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.656+0000 7f746b7fe640 1 -- 172.21.15.81:0/276815079 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v74) ==== 84+0+65 (secure 0 0 0) 0x7f7464080e80 con 0x7f74740b3200 2024-06-29T14:37:18.654 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:18.654 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:37:18.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:18.655 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T14:37:18.655 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T14:37:18.655 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:18.655 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:37:18.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.657+0000 7f7478ee3640 1 -- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7454076420 msgr2=0x7f7454078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.657+0000 7f7478ee3640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7454076420 0x7f7454078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f74741e7740 tx=0x7f745c005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:18.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.658+0000 7f7478ee3640 1 -- 172.21.15.81:0/276815079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74740b3200 msgr2=0x7f747406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.658+0000 7f7478ee3640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74740b3200 0x7f747406e2f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f7464034040 tx=0x7f74640035f0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.658+0000 7f7478ee3640 1 -- 172.21.15.81:0/276815079 shutdown_connections 2024-06-29T14:37:18.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.658+0000 7f7478ee3640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f7454076420 0x7f7454078930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.658+0000 7f7478ee3640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74740d0770 0x7f74741595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.658+0000 7f7478ee3640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74740cccf0 0x7f7474159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.658+0000 7f7478ee3640 1 --2- 172.21.15.81:0/276815079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74740b3200 0x7f747406e2f0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.658+0000 7f7478ee3640 1 -- 172.21.15.81:0/276815079 >> 172.21.15.81:0/276815079 conn(0x7f74740b3680 msgr2=0x7f7474070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.658+0000 7f7478ee3640 1 -- 172.21.15.81:0/276815079 shutdown_connections 2024-06-29T14:37:18.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.658+0000 7f7478ee3640 1 -- 172.21.15.81:0/276815079 wait complete. 2024-06-29T14:37:18.667 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:37:18.668 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:37:18.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.786+0000 7f63e14c0640 1 -- 172.21.15.81:0/1055856796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63dc0b6bc0 msgr2=0x7f63dc0b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.786+0000 7f63e14c0640 1 --2- 172.21.15.81:0/1055856796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63dc0b6bc0 0x7f63dc0b4980 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0cc540 tx=0x7f63c402a4b0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.786+0000 7f63e14c0640 1 -- 172.21.15.81:0/1055856796 shutdown_connections 2024-06-29T14:37:18.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.786+0000 7f63e14c0640 1 --2- 172.21.15.81:0/1055856796 >> v2:172.21.15.167:3300/0 conn(0x7f63dc0b2fe0 0x7f63dc160580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.786+0000 7f63e14c0640 1 --2- 172.21.15.81:0/1055856796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63dc0b6bc0 0x7f63dc0b4980 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.786+0000 7f63e14c0640 1 -- 172.21.15.81:0/1055856796 >> 172.21.15.81:0/1055856796 conn(0x7f63dc0b3680 msgr2=0x7f63dc0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:18.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.786+0000 7f63e14c0640 1 -- 172.21.15.81:0/1055856796 shutdown_connections 2024-06-29T14:37:18.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.786+0000 7f63e14c0640 1 -- 172.21.15.81:0/1055856796 wait complete. 2024-06-29T14:37:18.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.788+0000 7f63e14c0640 1 Processor -- start 2024-06-29T14:37:18.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.788+0000 7f63e14c0640 1 -- start start 2024-06-29T14:37:18.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.789+0000 7f63e14c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63dc0b2fe0 0x7f63dc1563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.789+0000 7f63e14c0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63dc0b6bc0 0x7f63dc06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.789+0000 7f63e14c0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63dc0b71f0 0x7f63dc06eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.789+0000 7f63e14c0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc0cd110 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.789+0000 7f63e14c0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc0ccf90 con 0x7f63dc0b6bc0 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.789+0000 7f63e14c0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f63dc0cd290 con 0x7f63dc0b71f0 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.789+0000 7f63daffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63dc0b2fe0 0x7f63dc1563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.789+0000 7f63daffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63dc0b2fe0 0x7f63dc1563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42604/0 (socket says 172.21.15.81:42604) 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.789+0000 7f63daffd640 1 -- 172.21.15.81:0/1823305962 learned_addr learned my addr 172.21.15.81:0/1823305962 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63da7fc640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63dc0b6bc0 0x7f63dc06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63db7fe640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63dc0b71f0 0x7f63dc06eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63daffd640 1 -- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63dc0b71f0 msgr2=0x7f63dc06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63daffd640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63dc0b71f0 0x7f63dc06eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63daffd640 1 -- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63dc0b6bc0 msgr2=0x7f63dc06e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63daffd640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63dc0b6bc0 0x7f63dc06e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63daffd640 1 -- 172.21.15.81:0/1823305962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc0c6c40 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63da7fc640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63dc0b6bc0 0x7f63dc06e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:18.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63db7fe640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63dc0b71f0 0x7f63dc06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:18.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63daffd640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63dc0b2fe0 0x7f63dc1563c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00ba50 tx=0x7f63cc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63d37fe640 1 -- 172.21.15.81:0/1823305962 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc019070 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63d37fe640 1 -- 172.21.15.81:0/1823305962 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63cc00cd00 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63d37fe640 1 -- 172.21.15.81:0/1823305962 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc0146a0 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63e14c0640 1 -- 172.21.15.81:0/1823305962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63dc159580 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.790+0000 7f63e14c0640 1 -- 172.21.15.81:0/1823305962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63dc06db70 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.791+0000 7f63e14c0640 1 -- 172.21.15.81:0/1823305962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63dc0cd870 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.792+0000 7f63d37fe640 1 -- 172.21.15.81:0/1823305962 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f63cc01c070 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.792+0000 7f63d37fe640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63bc0763f0 0x7f63bc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:18.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.792+0000 7f63d37fe640 1 -- 172.21.15.81:0/1823305962 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 4929+0+0 (secure 0 0 0) 0x7f63cc097520 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.793+0000 7f63da7fc640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63bc0763f0 0x7f63bc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:18.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.793+0000 7f63da7fc640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63bc0763f0 0x7f63bc078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f63c4033000 tx=0x7f63c4004730 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:18.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.797+0000 7f63d37fe640 1 -- 172.21.15.81:0/1823305962 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f63cc061070 con 0x7f63dc0b2fe0 2024-06-29T14:37:18.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:18.977+0000 7f63e14c0640 1 -- 172.21.15.81:0/1823305962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f63dc0c53a0 con 0x7f63dc0b2fe0 2024-06-29T14:37:19.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.304+0000 7f63d37fe640 1 -- 172.21.15.81:0/1823305962 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v75) ==== 138+0+0 (secure 0 0 0) 0x7f63cc060970 con 0x7f63dc0b2fe0 2024-06-29T14:37:19.303 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T14:37:19.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.307+0000 7f63e14c0640 1 -- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63bc0763f0 msgr2=0x7f63bc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:19.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.307+0000 7f63e14c0640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63bc0763f0 0x7f63bc078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f63c4033000 tx=0x7f63c4004730 comp rx=0 tx=0).stop 2024-06-29T14:37:19.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.307+0000 7f63e14c0640 1 -- 172.21.15.81:0/1823305962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63dc0b2fe0 msgr2=0x7f63dc1563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:19.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.307+0000 7f63e14c0640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63dc0b2fe0 0x7f63dc1563c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00ba50 tx=0x7f63cc00bf10 comp rx=0 tx=0).stop 2024-06-29T14:37:19.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.307+0000 7f63e14c0640 1 -- 172.21.15.81:0/1823305962 shutdown_connections 2024-06-29T14:37:19.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.308+0000 7f63e14c0640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63bc0763f0 0x7f63bc078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.308+0000 7f63e14c0640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63dc0b71f0 0x7f63dc06eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.308+0000 7f63e14c0640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63dc0b6bc0 0x7f63dc06e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.308+0000 7f63e14c0640 1 --2- 172.21.15.81:0/1823305962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63dc0b2fe0 0x7f63dc1563c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.308+0000 7f63e14c0640 1 -- 172.21.15.81:0/1823305962 >> 172.21.15.81:0/1823305962 conn(0x7f63dc0b3680 msgr2=0x7f63dc0c9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:19.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.308+0000 7f63e14c0640 1 -- 172.21.15.81:0/1823305962 shutdown_connections 2024-06-29T14:37:19.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.308+0000 7f63e14c0640 1 -- 172.21.15.81:0/1823305962 wait complete. 2024-06-29T14:37:19.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:37:19.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.443+0000 7f336613c640 1 -- 172.21.15.81:0/4178452165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33480053a0 msgr2=0x7f334800ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:19.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.443+0000 7f336613c640 1 --2- 172.21.15.81:0/4178452165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33480053a0 0x7f334800ba50 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f336006e210 tx=0x7f3348005830 comp rx=0 tx=0).stop 2024-06-29T14:37:19.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.443+0000 7f336613c640 1 -- 172.21.15.81:0/4178452165 shutdown_connections 2024-06-29T14:37:19.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.443+0000 7f336613c640 1 --2- 172.21.15.81:0/4178452165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33480053a0 0x7f334800ba50 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.443+0000 7f336613c640 1 -- 172.21.15.81:0/4178452165 >> 172.21.15.81:0/4178452165 conn(0x7f3360072950 msgr2=0x7f33600b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:19.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.443+0000 7f336613c640 1 -- 172.21.15.81:0/4178452165 shutdown_connections 2024-06-29T14:37:19.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.443+0000 7f336613c640 1 -- 172.21.15.81:0/4178452165 wait complete. 2024-06-29T14:37:19.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.445+0000 7f336613c640 1 Processor -- start 2024-06-29T14:37:19.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.445+0000 7f336613c640 1 -- start start 2024-06-29T14:37:19.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.446+0000 7f336613c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33480053a0 0x7f3360148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:19.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.446+0000 7f336613c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33600cc730 0x7f336015e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:19.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.446+0000 7f336613c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33600ccb40 0x7f336015e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:19.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.446+0000 7f336613c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f336015bdf0 con 0x7f33600ccb40 2024-06-29T14:37:19.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.446+0000 7f336613c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f33600af360 con 0x7f33480053a0 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.446+0000 7f336613c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f336015bf70 con 0x7f33600cc730 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.446+0000 7f335ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33600ccb40 0x7f336015e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.446+0000 7f335ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33600ccb40 0x7f336015e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42622/0 (socket says 172.21.15.81:42622) 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.446+0000 7f335ffff640 1 -- 172.21.15.81:0/2409738409 learned_addr learned my addr 172.21.15.81:0/2409738409 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f335effd640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33600cc730 0x7f336015e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f335f7fe640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33480053a0 0x7f3360148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f335ffff640 1 -- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33600cc730 msgr2=0x7f336015e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f335ffff640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33600cc730 0x7f336015e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f335ffff640 1 -- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33480053a0 msgr2=0x7f3360148350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f335ffff640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33480053a0 0x7f3360148350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f335ffff640 1 -- 172.21.15.81:0/2409738409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3360154a70 con 0x7f33600ccb40 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f335ffff640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33600ccb40 0x7f336015e9d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3358009e50 tx=0x7f335802cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f3347fff640 1 -- 172.21.15.81:0/2409738409 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3358009b00 con 0x7f33600ccb40 2024-06-29T14:37:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f3347fff640 1 -- 172.21.15.81:0/2409738409 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3358041070 con 0x7f33600ccb40 2024-06-29T14:37:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f336613c640 1 -- 172.21.15.81:0/2409738409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33600cf0b0 con 0x7f33600ccb40 2024-06-29T14:37:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f3347fff640 1 -- 172.21.15.81:0/2409738409 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33580339c0 con 0x7f33600ccb40 2024-06-29T14:37:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.447+0000 7f336613c640 1 -- 172.21.15.81:0/2409738409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33600cf670 con 0x7f33600ccb40 2024-06-29T14:37:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.448+0000 7f336613c640 1 -- 172.21.15.81:0/2409738409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3360073b10 con 0x7f33600ccb40 2024-06-29T14:37:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.449+0000 7f3347fff640 1 -- 172.21.15.81:0/2409738409 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f3358033b60 con 0x7f33600ccb40 2024-06-29T14:37:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.449+0000 7f3347fff640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f334c0763f0 0x7f334c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.450+0000 7f3347fff640 1 -- 172.21.15.81:0/2409738409 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5299+0+0 (secure 0 0 0) 0x7f33580b8310 con 0x7f33600ccb40 2024-06-29T14:37:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.450+0000 7f335f7fe640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f334c0763f0 0x7f334c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.450+0000 7f335f7fe640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f334c0763f0 0x7f334c078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3350006680 tx=0x7f3350006b70 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:19.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.454+0000 7f3347fff640 1 -- 172.21.15.81:0/2409738409 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3358081d80 con 0x7f33600ccb40 2024-06-29T14:37:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:19.636+0000 7f336613c640 1 -- 172.21.15.81:0/2409738409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f33600b0b70 con 0x7f33600ccb40 2024-06-29T14:37:20.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.308+0000 7f3347fff640 1 -- 172.21.15.81:0/2409738409 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) ==== 170+0+0 (secure 0 0 0) 0x7f3358081680 con 0x7f33600ccb40 2024-06-29T14:37:20.307 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T14:37:20.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.311+0000 7f336613c640 1 -- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f334c0763f0 msgr2=0x7f334c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:20.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.311+0000 7f336613c640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f334c0763f0 0x7f334c078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3350006680 tx=0x7f3350006b70 comp rx=0 tx=0).stop 2024-06-29T14:37:20.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.311+0000 7f336613c640 1 -- 172.21.15.81:0/2409738409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33600ccb40 msgr2=0x7f336015e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:20.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.311+0000 7f336613c640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33600ccb40 0x7f336015e9d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3358009e50 tx=0x7f335802cf40 comp rx=0 tx=0).stop 2024-06-29T14:37:20.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.311+0000 7f336613c640 1 -- 172.21.15.81:0/2409738409 shutdown_connections 2024-06-29T14:37:20.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.311+0000 7f336613c640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f334c0763f0 0x7f334c078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.311+0000 7f336613c640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33600ccb40 0x7f336015e9d0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.311+0000 7f336613c640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33600cc730 0x7f336015e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.311+0000 7f336613c640 1 --2- 172.21.15.81:0/2409738409 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33480053a0 0x7f3360148350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.312+0000 7f336613c640 1 -- 172.21.15.81:0/2409738409 >> 172.21.15.81:0/2409738409 conn(0x7f3360072950 msgr2=0x7f336015fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.312+0000 7f336613c640 1 -- 172.21.15.81:0/2409738409 shutdown_connections 2024-06-29T14:37:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.312+0000 7f336613c640 1 -- 172.21.15.81:0/2409738409 wait complete. 2024-06-29T14:37:20.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:37:20.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.459+0000 7fa19a963640 1 -- 172.21.15.81:0/1149334202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa17c005250 msgr2=0x7fa17c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:20.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.459+0000 7fa19a963640 1 --2- 172.21.15.81:0/1149334202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa17c005250 0x7fa17c00b9c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009ed0 tx=0x7fa17c005830 comp rx=0 tx=0).stop 2024-06-29T14:37:20.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.460+0000 7fa19a963640 1 -- 172.21.15.81:0/1149334202 shutdown_connections 2024-06-29T14:37:20.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.460+0000 7fa19a963640 1 --2- 172.21.15.81:0/1149334202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa17c005250 0x7fa17c00b9c0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.460+0000 7fa19a963640 1 -- 172.21.15.81:0/1149334202 >> 172.21.15.81:0/1149334202 conn(0x7fa1940d1960 msgr2=0x7fa1940d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:20.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.460+0000 7fa19a963640 1 -- 172.21.15.81:0/1149334202 shutdown_connections 2024-06-29T14:37:20.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.460+0000 7fa19a963640 1 -- 172.21.15.81:0/1149334202 wait complete. 2024-06-29T14:37:20.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.461+0000 7fa19a963640 1 Processor -- start 2024-06-29T14:37:20.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.462+0000 7fa19a963640 1 -- start start 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa19a963640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa17c005250 0x7fa1941e7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa19a963640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1941e7aa0 0x7fa1940ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa19a963640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1940aeed0 0x7fa1941dec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa19a963640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa1940d0890 con 0x7fa1940aeed0 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa19a963640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa1940d0710 con 0x7fa1941e7aa0 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa19a963640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa1940d0a10 con 0x7fa17c005250 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa198f00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1940aeed0 0x7fa1941dec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa198f00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1940aeed0 0x7fa1941dec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41028/0 (socket says 172.21.15.81:41028) 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa198f00640 1 -- 172.21.15.81:0/502414103 learned_addr learned my addr 172.21.15.81:0/502414103 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa193fff640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa17c005250 0x7fa1941e7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa1937fe640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1941e7aa0 0x7fa1940ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa198f00640 1 -- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa17c005250 msgr2=0x7fa1941e7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa198f00640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa17c005250 0x7fa1941e7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa198f00640 1 -- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1941e7aa0 msgr2=0x7fa1940ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa198f00640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1941e7aa0 0x7fa1940ae990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa198f00640 1 -- 172.21.15.81:0/502414103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1940af700 con 0x7fa1940aeed0 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.463+0000 7fa198f00640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1940aeed0 0x7fa1941dec40 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009e20 tx=0x7fa18c034ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.464+0000 7fa190ff9640 1 -- 172.21.15.81:0/502414103 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c041070 con 0x7fa1940aeed0 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.464+0000 7fa190ff9640 1 -- 172.21.15.81:0/502414103 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa18c0325f0 con 0x7fa1940aeed0 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.464+0000 7fa19a963640 1 -- 172.21.15.81:0/502414103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1940af9e0 con 0x7fa1940aeed0 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.464+0000 7fa190ff9640 1 -- 172.21.15.81:0/502414103 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c009ad0 con 0x7fa1940aeed0 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.464+0000 7fa19a963640 1 -- 172.21.15.81:0/502414103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1940affa0 con 0x7fa1940aeed0 2024-06-29T14:37:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.465+0000 7fa19a963640 1 -- 172.21.15.81:0/502414103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1940ce770 con 0x7fa1940aeed0 2024-06-29T14:37:20.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.465+0000 7fa190ff9640 1 -- 172.21.15.81:0/502414103 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fa18c044070 con 0x7fa1940aeed0 2024-06-29T14:37:20.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.466+0000 7fa190ff9640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa1800763f0 0x7fa180078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:20.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.466+0000 7fa190ff9640 1 -- 172.21.15.81:0/502414103 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5681+0+0 (secure 0 0 0) 0x7fa18c0b8330 con 0x7fa1940aeed0 2024-06-29T14:37:20.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.467+0000 7fa193fff640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa1800763f0 0x7fa180078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:20.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.467+0000 7fa193fff640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa1800763f0 0x7fa180078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa184000fb0 tx=0x7fa184001130 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:20.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.470+0000 7fa190ff9640 1 -- 172.21.15.81:0/502414103 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa18c081be0 con 0x7fa1940aeed0 2024-06-29T14:37:20.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:20.690+0000 7fa19a963640 1 -- 172.21.15.81:0/502414103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa1940af490 con 0x7fa1940aeed0 2024-06-29T14:37:21.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.328+0000 7fa190ff9640 1 -- 172.21.15.81:0/502414103 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) ==== 172+0+0 (secure 0 0 0) 0x7fa18c0814e0 con 0x7fa1940aeed0 2024-06-29T14:37:21.327 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 8 and data pool 9 2024-06-29T14:37:21.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.332+0000 7fa19a963640 1 -- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa1800763f0 msgr2=0x7fa180078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.332+0000 7fa19a963640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa1800763f0 0x7fa180078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa184000fb0 tx=0x7fa184001130 comp rx=0 tx=0).stop 2024-06-29T14:37:21.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.332+0000 7fa19a963640 1 -- 172.21.15.81:0/502414103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1940aeed0 msgr2=0x7fa1941dec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.332+0000 7fa19a963640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1940aeed0 0x7fa1941dec40 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009e20 tx=0x7fa18c034ca0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.333+0000 7fa19a963640 1 -- 172.21.15.81:0/502414103 shutdown_connections 2024-06-29T14:37:21.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.333+0000 7fa19a963640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa1800763f0 0x7fa180078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.333+0000 7fa19a963640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1940aeed0 0x7fa1941dec40 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.333+0000 7fa19a963640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1941e7aa0 0x7fa1940ae990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.333+0000 7fa19a963640 1 --2- 172.21.15.81:0/502414103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa17c005250 0x7fa1941e7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.333+0000 7fa19a963640 1 -- 172.21.15.81:0/502414103 >> 172.21.15.81:0/502414103 conn(0x7fa1940d1960 msgr2=0x7fa1941e42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:21.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.334+0000 7fa19a963640 1 -- 172.21.15.81:0/502414103 shutdown_connections 2024-06-29T14:37:21.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.335+0000 7fa19a963640 1 -- 172.21.15.81:0/502414103 wait complete. 2024-06-29T14:37:21.356 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:21.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.472+0000 7fdc919e0640 1 -- 172.21.15.81:0/4259224114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc8c0cea10 msgr2=0x7fdc8c0b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.472+0000 7fdc919e0640 1 --2- 172.21.15.81:0/4259224114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc8c0cea10 0x7fdc8c0b47b0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0739c0 tx=0x7fdc8002a560 comp rx=0 tx=0).stop 2024-06-29T14:37:21.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.472+0000 7fdc919e0640 1 -- 172.21.15.81:0/4259224114 shutdown_connections 2024-06-29T14:37:21.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.472+0000 7fdc919e0640 1 --2- 172.21.15.81:0/4259224114 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c0b4cf0 0x7fdc8c0d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.472+0000 7fdc919e0640 1 --2- 172.21.15.81:0/4259224114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc8c0cea10 0x7fdc8c0b47b0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.472+0000 7fdc919e0640 1 --2- 172.21.15.81:0/4259224114 >> v2:172.21.15.167:3300/0 conn(0x7fdc8c0b6d60 0x7fdc8c0ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.472+0000 7fdc919e0640 1 -- 172.21.15.81:0/4259224114 >> 172.21.15.81:0/4259224114 conn(0x7fdc8c0d1960 msgr2=0x7fdc8c0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:21.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.472+0000 7fdc919e0640 1 -- 172.21.15.81:0/4259224114 shutdown_connections 2024-06-29T14:37:21.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.472+0000 7fdc919e0640 1 -- 172.21.15.81:0/4259224114 wait complete. 2024-06-29T14:37:21.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.474+0000 7fdc919e0640 1 Processor -- start 2024-06-29T14:37:21.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.474+0000 7fdc919e0640 1 -- start start 2024-06-29T14:37:21.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.475+0000 7fdc919e0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c0b4cf0 0x7fdc8c1dbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.475+0000 7fdc919e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc8c0b6d60 0x7fdc8c09d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.475+0000 7fdc919e0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc8c0cea10 0x7fdc8c09d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.475+0000 7fdc919e0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c0d09c0 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.475+0000 7fdc919e0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8c0d0840 con 0x7fdc8c0cea10 2024-06-29T14:37:21.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.475+0000 7fdc919e0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdc8c0cc530 con 0x7fdc8c0b4cf0 2024-06-29T14:37:21.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.475+0000 7fdc8a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc8c0b6d60 0x7fdc8c09d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.475+0000 7fdc8a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc8c0b6d60 0x7fdc8c09d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41036/0 (socket says 172.21.15.81:41036) 2024-06-29T14:37:21.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.475+0000 7fdc8a7fc640 1 -- 172.21.15.81:0/3635709083 learned_addr learned my addr 172.21.15.81:0/3635709083 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:21.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc8affd640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c0b4cf0 0x7fdc8c1dbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc8b7fe640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc8c0cea10 0x7fdc8c09d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc8a7fc640 1 -- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c0b4cf0 msgr2=0x7fdc8c1dbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc8a7fc640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c0b4cf0 0x7fdc8c1dbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc8a7fc640 1 -- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc8c0cea10 msgr2=0x7fdc8c09d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc8a7fc640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc8c0cea10 0x7fdc8c09d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc8a7fc640 1 -- 172.21.15.81:0/3635709083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc8c1dc080 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc8a7fc640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc8c0b6d60 0x7fdc8c09d120 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fdc800099c0 tx=0x7fdc800038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:21.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc73fff640 1 -- 172.21.15.81:0/3635709083 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8003e030 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc73fff640 1 -- 172.21.15.81:0/3635709083 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc80003db0 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc919e0640 1 -- 172.21.15.81:0/3635709083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc8c09c610 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc73fff640 1 -- 172.21.15.81:0/3635709083 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc80035040 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.476+0000 7fdc919e0640 1 -- 172.21.15.81:0/3635709083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc8c0cb270 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.477+0000 7fdc919e0640 1 -- 172.21.15.81:0/3635709083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc8c0ce4d0 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.478+0000 7fdc73fff640 1 -- 172.21.15.81:0/3635709083 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fdc80041030 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.478+0000 7fdc73fff640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fdc78076420 0x7fdc78078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.478+0000 7fdc73fff640 1 -- 172.21.15.81:0/3635709083 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5781+0+0 (secure 0 0 0) 0x7fdc800b8470 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.478+0000 7fdc8affd640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fdc78076420 0x7fdc78078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.479+0000 7fdc8affd640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fdc78076420 0x7fdc78078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0c5220 tx=0x7fdc7c009500 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:21.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.483+0000 7fdc73fff640 1 -- 172.21.15.81:0/3635709083 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc80081ad0 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.666+0000 7fdc919e0640 1 -- 172.21.15.81:0/3635709083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc8c09c3a0 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.667+0000 7fdc73fff640 1 -- 172.21.15.81:0/3635709083 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+16780 (secure 0 0 0) 0x7fdc800813d0 con 0x7fdc8c0b6d60 2024-06-29T14:37:21.666 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:21.667 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":77,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:20.692116+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:18.983445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.642471+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:21.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 -- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fdc78076420 msgr2=0x7fdc78078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fdc78076420 0x7fdc78078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0c5220 tx=0x7fdc7c009500 comp rx=0 tx=0).stop 2024-06-29T14:37:21.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 -- 172.21.15.81:0/3635709083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc8c0b6d60 msgr2=0x7fdc8c09d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc8c0b6d60 0x7fdc8c09d120 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fdc800099c0 tx=0x7fdc800038a0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 -- 172.21.15.81:0/3635709083 shutdown_connections 2024-06-29T14:37:21.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fdc78076420 0x7fdc78078930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc8c0cea10 0x7fdc8c09d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc8c0b6d60 0x7fdc8c09d120 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 --2- 172.21.15.81:0/3635709083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc8c0b4cf0 0x7fdc8c1dbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 -- 172.21.15.81:0/3635709083 >> 172.21.15.81:0/3635709083 conn(0x7fdc8c0d1960 msgr2=0x7fdc8c1e4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:21.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 -- 172.21.15.81:0/3635709083 shutdown_connections 2024-06-29T14:37:21.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.670+0000 7fdc919e0640 1 -- 172.21.15.81:0/3635709083 wait complete. 2024-06-29T14:37:21.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:21.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.794+0000 7f3571f1c640 1 -- 172.21.15.81:0/1547447886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f356c0d0b90 msgr2=0x7f356c0d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.794+0000 7f3571f1c640 1 --2- 172.21.15.81:0/1547447886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f356c0d0b90 0x7f356c0d1a80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3560009830 tx=0x7f356002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.794+0000 7f3571f1c640 1 -- 172.21.15.81:0/1547447886 shutdown_connections 2024-06-29T14:37:21.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.794+0000 7f3571f1c640 1 --2- 172.21.15.81:0/1547447886 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f356c0b5250 0x7f356c0d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.794+0000 7f3571f1c640 1 --2- 172.21.15.81:0/1547447886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f356c0d0b90 0x7f356c0d1a80 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.794+0000 7f3571f1c640 1 -- 172.21.15.81:0/1547447886 >> 172.21.15.81:0/1547447886 conn(0x7f356c09cf30 msgr2=0x7f356c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:21.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.794+0000 7f3571f1c640 1 -- 172.21.15.81:0/1547447886 shutdown_connections 2024-06-29T14:37:21.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.794+0000 7f3571f1c640 1 -- 172.21.15.81:0/1547447886 wait complete. 2024-06-29T14:37:21.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.796+0000 7f3571f1c640 1 Processor -- start 2024-06-29T14:37:21.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.796+0000 7f3571f1c640 1 -- start start 2024-06-29T14:37:21.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3571f1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f356c0b5250 0x7f356c15b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3571f1c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f356c0b9040 0x7f356c0ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3571f1c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f356c0c8ff0 0x7f356c0cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3571f1c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f356c0d2500 con 0x7f356c0b5250 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3571f1c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f356c0b8560 con 0x7f356c0b9040 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3571f1c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f356c0d2680 con 0x7f356c0c8ff0 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f356c0b5250 0x7f356c15b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f356c0b5250 0x7f356c15b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41064/0 (socket says 172.21.15.81:41064) 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356b7fe640 1 -- 172.21.15.81:0/315622575 learned_addr learned my addr 172.21.15.81:0/315622575 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356bfff640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f356c0c8ff0 0x7f356c0cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356affd640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f356c0b9040 0x7f356c0ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356b7fe640 1 -- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f356c0c8ff0 msgr2=0x7f356c0cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356b7fe640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f356c0c8ff0 0x7f356c0cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356b7fe640 1 -- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f356c0b9040 msgr2=0x7f356c0ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356b7fe640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f356c0b9040 0x7f356c0ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356b7fe640 1 -- 172.21.15.81:0/315622575 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356c0c7ef0 con 0x7f356c0b5250 2024-06-29T14:37:21.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f356b7fe640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f356c0b5250 0x7f356c15b990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f355c00c3c0 tx=0x7f355c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:21.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3553fff640 1 -- 172.21.15.81:0/315622575 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355c017070 con 0x7f356c0b5250 2024-06-29T14:37:21.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3553fff640 1 -- 172.21.15.81:0/315622575 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f355c0035f0 con 0x7f356c0b5250 2024-06-29T14:37:21.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3553fff640 1 -- 172.21.15.81:0/315622575 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355c003a80 con 0x7f356c0b5250 2024-06-29T14:37:21.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3571f1c640 1 -- 172.21.15.81:0/315622575 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f356c0c81d0 con 0x7f356c0b5250 2024-06-29T14:37:21.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.798+0000 7f3571f1c640 1 -- 172.21.15.81:0/315622575 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f356c0b1f60 con 0x7f356c0b5250 2024-06-29T14:37:21.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.799+0000 7f3571f1c640 1 -- 172.21.15.81:0/315622575 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f356c0b21c0 con 0x7f356c0b5250 2024-06-29T14:37:21.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.800+0000 7f3553fff640 1 -- 172.21.15.81:0/315622575 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f355c01a070 con 0x7f356c0b5250 2024-06-29T14:37:21.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.801+0000 7f3553fff640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f35580763f0 0x7f3558078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:21.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.801+0000 7f3553fff640 1 -- 172.21.15.81:0/315622575 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5781+0+0 (secure 0 0 0) 0x7f355c096fa0 con 0x7f356c0b5250 2024-06-29T14:37:21.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.801+0000 7f356affd640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f35580763f0 0x7f3558078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:21.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.801+0000 7f356affd640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f35580763f0 0x7f3558078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3560034000 tx=0x7f3560004730 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:21.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.804+0000 7f3553fff640 1 -- 172.21.15.81:0/315622575 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f355c021020 con 0x7f356c0b5250 2024-06-29T14:37:21.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.984+0000 7f3571f1c640 1 -- 172.21.15.81:0/315622575 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f356c0d2c10 con 0x7f356c0b5250 2024-06-29T14:37:21.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.986+0000 7f3553fff640 1 -- 172.21.15.81:0/315622575 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+16780 (secure 0 0 0) 0x7f355c060910 con 0x7f356c0b5250 2024-06-29T14:37:21.984 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:21.984 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":77,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:20.692116+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:18.983445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.642471+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 -- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f35580763f0 msgr2=0x7f3558078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f35580763f0 0x7f3558078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3560034000 tx=0x7f3560004730 comp rx=0 tx=0).stop 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 -- 172.21.15.81:0/315622575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f356c0b5250 msgr2=0x7f356c15b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f356c0b5250 0x7f356c15b990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f355c00c3c0 tx=0x7f355c00c880 comp rx=0 tx=0).stop 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 -- 172.21.15.81:0/315622575 shutdown_connections 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f35580763f0 0x7f3558078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f356c0c8ff0 0x7f356c0cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f356c0b9040 0x7f356c0ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 --2- 172.21.15.81:0/315622575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f356c0b5250 0x7f356c15b990 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 -- 172.21.15.81:0/315622575 >> 172.21.15.81:0/315622575 conn(0x7f356c09cf30 msgr2=0x7f356c068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:21.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 -- 172.21.15.81:0/315622575 shutdown_connections 2024-06-29T14:37:21.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:21.988+0000 7f3571f1c640 1 -- 172.21.15.81:0/315622575 wait complete. 2024-06-29T14:37:21.998 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:37:22.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.116+0000 7f4683847640 1 -- 172.21.15.81:0/3118330188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f467c15aab0 msgr2=0x7f467c15aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:22.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.116+0000 7f4683847640 1 --2- 172.21.15.81:0/3118330188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f467c15aab0 0x7f467c15aec0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4674005fd0 tx=0x7f467402a4b0 comp rx=0 tx=0).stop 2024-06-29T14:37:22.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.116+0000 7f4683847640 1 -- 172.21.15.81:0/3118330188 shutdown_connections 2024-06-29T14:37:22.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.116+0000 7f4683847640 1 --2- 172.21.15.81:0/3118330188 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f467c06fd70 0x7f467c0b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:22.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.116+0000 7f4683847640 1 --2- 172.21.15.81:0/3118330188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f467c15aab0 0x7f467c15aec0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:22.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.116+0000 7f4683847640 1 -- 172.21.15.81:0/3118330188 >> 172.21.15.81:0/3118330188 conn(0x7f467c06ccc0 msgr2=0x7f467c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:22.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.116+0000 7f4683847640 1 -- 172.21.15.81:0/3118330188 shutdown_connections 2024-06-29T14:37:22.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.116+0000 7f4683847640 1 -- 172.21.15.81:0/3118330188 wait complete. 2024-06-29T14:37:22.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.118+0000 7f4683847640 1 Processor -- start 2024-06-29T14:37:22.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.118+0000 7f4683847640 1 -- start start 2024-06-29T14:37:22.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.119+0000 7f4683847640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f467c0cf4c0 0x7f467c1439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:22.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.119+0000 7f4683847640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f467c15aab0 0x7f467c1558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:22.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.119+0000 7f4683847640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f467c155de0 0x7f467c1561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.119+0000 7f4683847640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f467c06f270 con 0x7f467c155de0 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.119+0000 7f4683847640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f467c06f0f0 con 0x7f467c0cf4c0 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.119+0000 7f4683847640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f467c06f3f0 con 0x7f467c15aab0 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.119+0000 7f4681de4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f467c155de0 0x7f467c1561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.119+0000 7f4681de4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f467c155de0 0x7f467c1561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41074/0 (socket says 172.21.15.81:41074) 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.119+0000 7f4681de4640 1 -- 172.21.15.81:0/2844456741 learned_addr learned my addr 172.21.15.81:0/2844456741 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4680de2640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f467c15aab0 0x7f467c1558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f46815e3640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f467c0cf4c0 0x7f467c1439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4681de4640 1 -- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f467c15aab0 msgr2=0x7f467c1558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4681de4640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f467c15aab0 0x7f467c1558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4681de4640 1 -- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f467c0cf4c0 msgr2=0x7f467c1439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4681de4640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f467c0cf4c0 0x7f467c1439c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:22.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4681de4640 1 -- 172.21.15.81:0/2844456741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467c156730 con 0x7f467c155de0 2024-06-29T14:37:22.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4681de4640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f467c155de0 0x7f467c1561f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4674034040 tx=0x7f4674003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:22.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4679ffb640 1 -- 172.21.15.81:0/2844456741 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f467403e030 con 0x7f467c155de0 2024-06-29T14:37:22.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4679ffb640 1 -- 172.21.15.81:0/2844456741 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4674003d70 con 0x7f467c155de0 2024-06-29T14:37:22.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4683847640 1 -- 172.21.15.81:0/2844456741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f467c149e20 con 0x7f467c155de0 2024-06-29T14:37:22.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4679ffb640 1 -- 172.21.15.81:0/2844456741 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4674035040 con 0x7f467c155de0 2024-06-29T14:37:22.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.120+0000 7f4683847640 1 -- 172.21.15.81:0/2844456741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f467c0aeda0 con 0x7f467c155de0 2024-06-29T14:37:22.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.121+0000 7f4683847640 1 -- 172.21.15.81:0/2844456741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f467c14a020 con 0x7f467c155de0 2024-06-29T14:37:22.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.122+0000 7f4679ffb640 1 -- 172.21.15.81:0/2844456741 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f4674041030 con 0x7f467c155de0 2024-06-29T14:37:22.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.122+0000 7f4679ffb640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f46680763f0 0x7f4668078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:22.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.122+0000 7f4679ffb640 1 -- 172.21.15.81:0/2844456741 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5781+0+0 (secure 0 0 0) 0x7f46740b8160 con 0x7f467c155de0 2024-06-29T14:37:22.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.123+0000 7f46815e3640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f46680763f0 0x7f4668078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:22.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.123+0000 7f46815e3640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f46680763f0 0x7f4668078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f466c001ce0 tx=0x7f466c009040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:22.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.127+0000 7f4679ffb640 1 -- 172.21.15.81:0/2844456741 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f467c14a020 con 0x7f467c155de0 2024-06-29T14:37:22.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:22.346+0000 7f4683847640 1 -- 172.21.15.81:0/2844456741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f467c0b3640 con 0x7f467c155de0 2024-06-29T14:37:23.298 INFO:tasks.ceph.mds.b.smithi167.stderr:2024-06-29T14:37:23.301+0000 7f57460dc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:37:23.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.338+0000 7f4679ffb640 1 -- 172.21.15.81:0/2844456741 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v33) ==== 118+0+0 (secure 0 0 0) 0x7f4674081850 con 0x7f467c155de0 2024-06-29T14:37:23.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.341+0000 7f4683847640 1 -- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f46680763f0 msgr2=0x7f4668078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:23.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.341+0000 7f4683847640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f46680763f0 0x7f4668078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f466c001ce0 tx=0x7f466c009040 comp rx=0 tx=0).stop 2024-06-29T14:37:23.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.341+0000 7f4683847640 1 -- 172.21.15.81:0/2844456741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f467c155de0 msgr2=0x7f467c1561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:23.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.341+0000 7f4683847640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f467c155de0 0x7f467c1561f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4674034040 tx=0x7f4674003880 comp rx=0 tx=0).stop 2024-06-29T14:37:23.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.342+0000 7f4683847640 1 -- 172.21.15.81:0/2844456741 shutdown_connections 2024-06-29T14:37:23.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.342+0000 7f4683847640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f46680763f0 0x7f4668078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.342+0000 7f4683847640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f467c155de0 0x7f467c1561f0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.342+0000 7f4683847640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f467c15aab0 0x7f467c1558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.342+0000 7f4683847640 1 --2- 172.21.15.81:0/2844456741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f467c0cf4c0 0x7f467c1439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.342+0000 7f4683847640 1 -- 172.21.15.81:0/2844456741 >> 172.21.15.81:0/2844456741 conn(0x7f467c06ccc0 msgr2=0x7f467c159e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:23.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.342+0000 7f4683847640 1 -- 172.21.15.81:0/2844456741 shutdown_connections 2024-06-29T14:37:23.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.342+0000 7f4683847640 1 -- 172.21.15.81:0/2844456741 wait complete. 2024-06-29T14:37:23.354 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:37:23.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:37:23.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-06-29T14:37:23.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.480+0000 7fcb5c455640 1 -- 172.21.15.81:0/3914941322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb540b3cd0 msgr2=0x7fcb540d2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.480+0000 7fcb5c455640 1 --2- 172.21.15.81:0/3914941322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb540b3cd0 0x7fcb540d2240 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009830 tx=0x7fcb4402a480 comp rx=0 tx=0).stop 2024-06-29T14:37:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.480+0000 7fcb5c455640 1 -- 172.21.15.81:0/3914941322 shutdown_connections 2024-06-29T14:37:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.480+0000 7fcb5c455640 1 --2- 172.21.15.81:0/3914941322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb540ceb80 0x7fcb540d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.480+0000 7fcb5c455640 1 --2- 172.21.15.81:0/3914941322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb540b3cd0 0x7fcb540d2240 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.480+0000 7fcb5c455640 1 -- 172.21.15.81:0/3914941322 >> 172.21.15.81:0/3914941322 conn(0x7fcb540d3c60 msgr2=0x7fcb540d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.480+0000 7fcb5c455640 1 -- 172.21.15.81:0/3914941322 shutdown_connections 2024-06-29T14:37:23.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.480+0000 7fcb5c455640 1 -- 172.21.15.81:0/3914941322 wait complete. 2024-06-29T14:37:23.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.481+0000 7fcb5c455640 1 Processor -- start 2024-06-29T14:37:23.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.481+0000 7fcb5c455640 1 -- start start 2024-06-29T14:37:23.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb5c455640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb540ce770 0x7fcb541520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:23.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb5c455640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb540ceb80 0x7fcb540cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb5c455640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb540cbe90 0x7fcb54167180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb5c455640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcb540d15a0 con 0x7fcb540ceb80 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb5c455640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcb540d1420 con 0x7fcb540ce770 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb5c455640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcb540d4990 con 0x7fcb540cbe90 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb599f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb540ceb80 0x7fcb540cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb599f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb540ceb80 0x7fcb540cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41084/0 (socket says 172.21.15.81:41084) 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb599f0640 1 -- 172.21.15.81:0/3688611026 learned_addr learned my addr 172.21.15.81:0/3688611026 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb5a9f2640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb540cbe90 0x7fcb54167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb5a1f1640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb540ce770 0x7fcb541520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb599f0640 1 -- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb540cbe90 msgr2=0x7fcb54167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb599f0640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb540cbe90 0x7fcb54167180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb599f0640 1 -- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb540ce770 msgr2=0x7fcb541520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb599f0640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb540ce770 0x7fcb541520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:23.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb599f0640 1 -- 172.21.15.81:0/3688611026 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb54152600 con 0x7fcb540ceb80 2024-06-29T14:37:23.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb5a1f1640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb540ce770 0x7fcb541520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:23.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.483+0000 7fcb599f0640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb540ceb80 0x7fcb540cb950 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009950 tx=0x7fcb44033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:23.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.484+0000 7fcb4affd640 1 -- 172.21.15.81:0/3688611026 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44031d60 con 0x7fcb540ceb80 2024-06-29T14:37:23.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.484+0000 7fcb5c455640 1 -- 172.21.15.81:0/3688611026 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb541630f0 con 0x7fcb540ceb80 2024-06-29T14:37:23.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.484+0000 7fcb4affd640 1 -- 172.21.15.81:0/3688611026 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb44033e10 con 0x7fcb540ceb80 2024-06-29T14:37:23.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.484+0000 7fcb4affd640 1 -- 172.21.15.81:0/3688611026 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44005260 con 0x7fcb540ceb80 2024-06-29T14:37:23.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.484+0000 7fcb5c455640 1 -- 172.21.15.81:0/3688611026 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb541636b0 con 0x7fcb540ceb80 2024-06-29T14:37:23.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.485+0000 7fcb5c455640 1 -- 172.21.15.81:0/3688611026 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb540d2240 con 0x7fcb540ceb80 2024-06-29T14:37:23.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.485+0000 7fcb4affd640 1 -- 172.21.15.81:0/3688611026 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fcb44005400 con 0x7fcb540ceb80 2024-06-29T14:37:23.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.486+0000 7fcb4affd640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcb380763f0 0x7fcb38078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:23.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.486+0000 7fcb4affd640 1 -- 172.21.15.81:0/3688611026 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7fcb440b8bd0 con 0x7fcb540ceb80 2024-06-29T14:37:23.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.486+0000 7fcb5a1f1640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcb380763f0 0x7fcb38078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:23.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.486+0000 7fcb5a1f1640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcb380763f0 0x7fcb38078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb540d0120 tx=0x7fcb50009040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:23.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.490+0000 7fcb4affd640 1 -- 172.21.15.81:0/3688611026 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcb4403f030 con 0x7fcb540ceb80 2024-06-29T14:37:23.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:23.707+0000 7fcb5c455640 1 -- 172.21.15.81:0/3688611026 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fcb54162510 con 0x7fcb540ceb80 2024-06-29T14:37:24.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.340+0000 7fcb4affd640 1 -- 172.21.15.81:0/3688611026 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7fcb440821b0 con 0x7fcb540ceb80 2024-06-29T14:37:24.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 -- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcb380763f0 msgr2=0x7fcb38078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcb380763f0 0x7fcb38078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb540d0120 tx=0x7fcb50009040 comp rx=0 tx=0).stop 2024-06-29T14:37:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 -- 172.21.15.81:0/3688611026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb540ceb80 msgr2=0x7fcb540cb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb540ceb80 0x7fcb540cb950 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009950 tx=0x7fcb44033c20 comp rx=0 tx=0).stop 2024-06-29T14:37:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 -- 172.21.15.81:0/3688611026 shutdown_connections 2024-06-29T14:37:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fcb380763f0 0x7fcb38078900 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb540d0120 tx=0x7fcb50009040 comp rx=0 tx=0).stop 2024-06-29T14:37:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcb540cbe90 0x7fcb54167180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb540ceb80 0x7fcb540cb950 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 --2- 172.21.15.81:0/3688611026 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcb540ce770 0x7fcb541520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 -- 172.21.15.81:0/3688611026 >> 172.21.15.81:0/3688611026 conn(0x7fcb540d3c60 msgr2=0x7fcb5406ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 -- 172.21.15.81:0/3688611026 shutdown_connections 2024-06-29T14:37:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.343+0000 7fcb5c455640 1 -- 172.21.15.81:0/3688611026 wait complete. 2024-06-29T14:37:24.355 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:37:24.355 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:24.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.474+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2803857051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff698004800 msgr2=0x7ff698004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.474+0000 7ff6b0dbd640 1 --2- 172.21.15.81:0/2803857051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff698004800 0x7ff698004c10 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff694009b50 tx=0x7ff698005830 comp rx=0 tx=0).stop 2024-06-29T14:37:24.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.474+0000 7ff6aad9d640 1 -- 172.21.15.81:0/2803857051 reap_dead start 2024-06-29T14:37:24.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.474+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2803857051 shutdown_connections 2024-06-29T14:37:24.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.474+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2803857051 >> 172.21.15.81:0/2803857051 conn(0x7ff6ac0d0930 msgr2=0x7ff6ac0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:24.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.474+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2803857051 shutdown_connections 2024-06-29T14:37:24.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.474+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2803857051 wait complete. 2024-06-29T14:37:24.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.476+0000 7ff6b0dbd640 1 Processor -- start 2024-06-29T14:37:24.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.476+0000 7ff6b0dbd640 1 -- start start 2024-06-29T14:37:24.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.477+0000 7ff6b0dbd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff698004800 0x7ff6ac156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.477+0000 7ff6b0dbd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6ac1f5710 0x7ff6ac1f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.477+0000 7ff6b0dbd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6ac1f5b20 0x7ff6ac1f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.477+0000 7ff6b0dbd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac0d0650 con 0x7ff698004800 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.477+0000 7ff6b0dbd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac0d04d0 con 0x7ff6ac1f5710 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.477+0000 7ff6b0dbd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff6ac06c2f0 con 0x7ff6ac1f5b20 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6aa59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff698004800 0x7ff6ac156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6aa59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff698004800 0x7ff6ac156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41108/0 (socket says 172.21.15.81:41108) 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6aa59c640 1 -- 172.21.15.81:0/2454261886 learned_addr learned my addr 172.21.15.81:0/2454261886 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6a9d9b640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6ac1f5710 0x7ff6ac1f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6aad9d640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6ac1f5b20 0x7ff6ac1f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6aa59c640 1 -- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6ac1f5b20 msgr2=0x7ff6ac1f30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6aa59c640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6ac1f5b20 0x7ff6ac1f30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6aa59c640 1 -- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6ac1f5710 msgr2=0x7ff6ac1f2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6aa59c640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6ac1f5710 0x7ff6ac1f2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6aa59c640 1 -- 172.21.15.81:0/2454261886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ac1f4670 con 0x7ff698004800 2024-06-29T14:37:24.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6aa59c640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff698004800 0x7ff6ac156880 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4004fd0 tx=0x7ff6a4001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:24.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6a2ffd640 1 -- 172.21.15.81:0/2454261886 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4015070 con 0x7ff698004800 2024-06-29T14:37:24.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6a2ffd640 1 -- 172.21.15.81:0/2454261886 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6a400ea90 con 0x7ff698004800 2024-06-29T14:37:24.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2454261886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ac1f4950 con 0x7ff698004800 2024-06-29T14:37:24.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6a2ffd640 1 -- 172.21.15.81:0/2454261886 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a400eea0 con 0x7ff698004800 2024-06-29T14:37:24.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.478+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2454261886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ac1f4f10 con 0x7ff698004800 2024-06-29T14:37:24.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.479+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2454261886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ac06e2a0 con 0x7ff698004800 2024-06-29T14:37:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.480+0000 7ff6a2ffd640 1 -- 172.21.15.81:0/2454261886 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7ff6a4018070 con 0x7ff698004800 2024-06-29T14:37:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.480+0000 7ff6a2ffd640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7ff68c0763f0 0x7ff68c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.481+0000 7ff6a2ffd640 1 -- 172.21.15.81:0/2454261886 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7ff6a4098330 con 0x7ff698004800 2024-06-29T14:37:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.481+0000 7ff6a9d9b640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7ff68c0763f0 0x7ff68c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.481+0000 7ff6a9d9b640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7ff68c0763f0 0x7ff68c078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff694035000 tx=0x7ff69402d040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:24.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.485+0000 7ff6a2ffd640 1 -- 172.21.15.81:0/2454261886 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff6ac06e2a0 con 0x7ff698004800 2024-06-29T14:37:24.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.701+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2454261886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6ac06c7e0 con 0x7ff698004800 2024-06-29T14:37:24.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.701+0000 7ff6a2ffd640 1 -- 172.21.15.81:0/2454261886 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5226 (secure 0 0 0) 0x7ff6a4061a20 con 0x7ff698004800 2024-06-29T14:37:24.700 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:24.700 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"btime":"2024-06-29T14:37:24:340402+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3148404249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3148404249},{"type":"v1","addr":"172.21.15.167:6836","nonce":3148404249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6549,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/704367026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":704367026},{"type":"v1","addr":"172.21.15.81:6834","nonce":704367026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6552,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2316096753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2316096753},{"type":"v1","addr":"172.21.15.81:6835","nonce":2316096753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.692072+0000","modified":"2024-06-29T14:37:23.709021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6567":{"gid":6567,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/2723173791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2723173791},{"type":"v1","addr":"172.21.15.167:6837","nonce":2723173791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6567,"qdb_cluster":[6567]},"id":4}]} 2024-06-29T14:37:24.700 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 34 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7ff68c0763f0 msgr2=0x7ff68c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7ff68c0763f0 0x7ff68c078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff694035000 tx=0x7ff69402d040 comp rx=0 tx=0).stop 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2454261886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff698004800 msgr2=0x7ff6ac156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff698004800 0x7ff6ac156880 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4004fd0 tx=0x7ff6a4001d20 comp rx=0 tx=0).stop 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2454261886 shutdown_connections 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7ff68c0763f0 0x7ff68c078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6ac1f5b20 0x7ff6ac1f30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6ac1f5710 0x7ff6ac1f2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 --2- 172.21.15.81:0/2454261886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff698004800 0x7ff6ac156880 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2454261886 >> 172.21.15.81:0/2454261886 conn(0x7ff6ac0d0930 msgr2=0x7ff6ac06ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2454261886 shutdown_connections 2024-06-29T14:37:24.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.703+0000 7ff6b0dbd640 1 -- 172.21.15.81:0/2454261886 wait complete. 2024-06-29T14:37:24.713 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:24.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.828+0000 7f9b58822640 1 -- 172.21.15.81:0/3638104409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b5408ccd0 msgr2=0x7f9b540cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.828+0000 7f9b58822640 1 --2- 172.21.15.81:0/3638104409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b5408ccd0 0x7f9b540cee70 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005fd0 tx=0x7f9b3c02a4b0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.828+0000 7f9b58822640 1 -- 172.21.15.81:0/3638104409 shutdown_connections 2024-06-29T14:37:24.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.828+0000 7f9b58822640 1 --2- 172.21.15.81:0/3638104409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b5408ccd0 0x7f9b540cee70 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.828+0000 7f9b58822640 1 --2- 172.21.15.81:0/3638104409 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b5408c270 0x7f9b540ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.828+0000 7f9b58822640 1 -- 172.21.15.81:0/3638104409 >> 172.21.15.81:0/3638104409 conn(0x7f9b540b56d0 msgr2=0x7f9b540b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:24.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.828+0000 7f9b58822640 1 -- 172.21.15.81:0/3638104409 shutdown_connections 2024-06-29T14:37:24.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.828+0000 7f9b58822640 1 -- 172.21.15.81:0/3638104409 wait complete. 2024-06-29T14:37:24.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.830+0000 7f9b58822640 1 Processor -- start 2024-06-29T14:37:24.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.830+0000 7f9b58822640 1 -- start start 2024-06-29T14:37:24.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b58822640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b5408ccd0 0x7f9b5409e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b58822640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b540d17b0 0x7f9b54153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b58822640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b54167950 0x7f9b54165140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b58822640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9b540d2450 con 0x7f9b540d17b0 2024-06-29T14:37:24.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b58822640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9b540d22d0 con 0x7f9b5408ccd0 2024-06-29T14:37:24.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b58822640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9b540d25d0 con 0x7f9b54167950 2024-06-29T14:37:24.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b540d17b0 0x7f9b54153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b540d17b0 0x7f9b54153070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41124/0 (socket says 172.21.15.81:41124) 2024-06-29T14:37:24.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4bfff640 1 -- 172.21.15.81:0/2355302868 learned_addr learned my addr 172.21.15.81:0/2355302868 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:24.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b52d9d640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b54167950 0x7f9b54165140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b5259c640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b5408ccd0 0x7f9b5409e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4bfff640 1 -- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b54167950 msgr2=0x7f9b54165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4bfff640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b54167950 0x7f9b54165140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4bfff640 1 -- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b5408ccd0 msgr2=0x7f9b5409e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4bfff640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b5408ccd0 0x7f9b5409e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4bfff640 1 -- 172.21.15.81:0/2355302868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b540a15d0 con 0x7f9b540d17b0 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4bfff640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b540d17b0 0x7f9b54153070 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c034040 tx=0x7f9b3c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4affd640 1 -- 172.21.15.81:0/2355302868 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c03e030 con 0x7f9b540d17b0 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4affd640 1 -- 172.21.15.81:0/2355302868 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b3c003db0 con 0x7f9b540d17b0 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b58822640 1 -- 172.21.15.81:0/2355302868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b54168530 con 0x7f9b540d17b0 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b4affd640 1 -- 172.21.15.81:0/2355302868 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c035040 con 0x7f9b540d17b0 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.832+0000 7f9b58822640 1 -- 172.21.15.81:0/2355302868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b5409dee0 con 0x7f9b540d17b0 2024-06-29T14:37:24.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.833+0000 7f9b58822640 1 -- 172.21.15.81:0/2355302868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b541682a0 con 0x7f9b540d17b0 2024-06-29T14:37:24.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.834+0000 7f9b4affd640 1 -- 172.21.15.81:0/2355302868 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f9b3c041030 con 0x7f9b540d17b0 2024-06-29T14:37:24.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.835+0000 7f9b4affd640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9b340763f0 0x7f9b34078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:24.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.835+0000 7f9b4affd640 1 -- 172.21.15.81:0/2355302868 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7f9b3c0b8480 con 0x7f9b540d17b0 2024-06-29T14:37:24.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.835+0000 7f9b5259c640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9b340763f0 0x7f9b34078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:24.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.835+0000 7f9b5259c640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9b340763f0 0x7f9b34078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009ac0 tx=0x7f9b44005f90 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:24.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:24.838+0000 7f9b4affd640 1 -- 172.21.15.81:0/2355302868 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9b540a15d0 con 0x7f9b540d17b0 2024-06-29T14:37:25.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.019+0000 7f9b58822640 1 -- 172.21.15.81:0/2355302868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b5408c270 con 0x7f9b540d17b0 2024-06-29T14:37:25.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.020+0000 7f9b4affd640 1 -- 172.21.15.81:0/2355302868 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+16767 (secure 0 0 0) 0x7f9b3c081c50 con 0x7f9b540d17b0 2024-06-29T14:37:25.019 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:25.019 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":78,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:22.327323+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:18.983445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.642471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 -- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9b340763f0 msgr2=0x7f9b34078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9b340763f0 0x7f9b34078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009ac0 tx=0x7f9b44005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 -- 172.21.15.81:0/2355302868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b540d17b0 msgr2=0x7f9b54153070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b540d17b0 0x7f9b54153070 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c034040 tx=0x7f9b3c0038a0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 -- 172.21.15.81:0/2355302868 shutdown_connections 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f9b340763f0 0x7f9b34078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b54167950 0x7f9b54165140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b540d17b0 0x7f9b54153070 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 --2- 172.21.15.81:0/2355302868 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b5408ccd0 0x7f9b5409e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 -- 172.21.15.81:0/2355302868 >> 172.21.15.81:0/2355302868 conn(0x7f9b540b56d0 msgr2=0x7f9b5406e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 -- 172.21.15.81:0/2355302868 shutdown_connections 2024-06-29T14:37:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.022+0000 7f9b58822640 1 -- 172.21.15.81:0/2355302868 wait complete. 2024-06-29T14:37:25.032 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:37:25.033 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:37:25.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.146+0000 7fb3739bd640 1 -- 172.21.15.81:0/1756314463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb36c06f4e0 msgr2=0x7fb36c151030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.146+0000 7fb3739bd640 1 --2- 172.21.15.81:0/1756314463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb36c06f4e0 0x7fb36c151030 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb35c005fd0 tx=0x7fb35c02b9a0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.146+0000 7fb3739bd640 1 -- 172.21.15.81:0/1756314463 shutdown_connections 2024-06-29T14:37:25.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.146+0000 7fb3739bd640 1 --2- 172.21.15.81:0/1756314463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb36c06f4e0 0x7fb36c151030 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.146+0000 7fb3739bd640 1 -- 172.21.15.81:0/1756314463 >> 172.21.15.81:0/1756314463 conn(0x7fb36c06ccc0 msgr2=0x7fb36c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.146+0000 7fb3739bd640 1 -- 172.21.15.81:0/1756314463 shutdown_connections 2024-06-29T14:37:25.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.146+0000 7fb3739bd640 1 -- 172.21.15.81:0/1756314463 wait complete. 2024-06-29T14:37:25.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.148+0000 7fb3739bd640 1 Processor -- start 2024-06-29T14:37:25.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.148+0000 7fb3739bd640 1 -- start start 2024-06-29T14:37:25.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.149+0000 7fb3739bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb36c0cefc0 0x7fb36c1f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.149+0000 7fb3739bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb36c15e5f0 0x7fb36c149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.149+0000 7fb3739bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb36c148ef0 0x7fb36c14b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.149+0000 7fb3739bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c0cdcd0 con 0x7fb36c148ef0 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.149+0000 7fb3739bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c070220 con 0x7fb36c15e5f0 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.149+0000 7fb3739bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb36c0cde50 con 0x7fb36c0cefc0 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb371f5a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb36c148ef0 0x7fb36c14b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb371f5a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb36c148ef0 0x7fb36c14b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41156/0 (socket says 172.21.15.81:41156) 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb371f5a640 1 -- 172.21.15.81:0/3195539281 learned_addr learned my addr 172.21.15.81:0/3195539281 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb371759640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb36c0cefc0 0x7fb36c1f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb370f58640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb36c15e5f0 0x7fb36c149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb371f5a640 1 -- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb36c0cefc0 msgr2=0x7fb36c1f2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb371f5a640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb36c0cefc0 0x7fb36c1f2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb371f5a640 1 -- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb36c15e5f0 msgr2=0x7fb36c149570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb371f5a640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb36c15e5f0 0x7fb36c149570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb371f5a640 1 -- 172.21.15.81:0/3195539281 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c1f6110 con 0x7fb36c148ef0 2024-06-29T14:37:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb371f5a640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb36c148ef0 0x7fb36c14b6e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb36400c730 tx=0x7fb36400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb361ffb640 1 -- 172.21.15.81:0/3195539281 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364003b70 con 0x7fb36c148ef0 2024-06-29T14:37:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb361ffb640 1 -- 172.21.15.81:0/3195539281 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb364003d30 con 0x7fb36c148ef0 2024-06-29T14:37:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb3739bd640 1 -- 172.21.15.81:0/3195539281 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36c149ab0 con 0x7fb36c148ef0 2024-06-29T14:37:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.150+0000 7fb361ffb640 1 -- 172.21.15.81:0/3195539281 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3640052d0 con 0x7fb36c148ef0 2024-06-29T14:37:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.151+0000 7fb3739bd640 1 -- 172.21.15.81:0/3195539281 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb36c1f3630 con 0x7fb36c148ef0 2024-06-29T14:37:25.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.151+0000 7fb3739bd640 1 -- 172.21.15.81:0/3195539281 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb36c0ce710 con 0x7fb36c148ef0 2024-06-29T14:37:25.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.152+0000 7fb361ffb640 1 -- 172.21.15.81:0/3195539281 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fb364005680 con 0x7fb36c148ef0 2024-06-29T14:37:25.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.153+0000 7fb361ffb640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fb3500763f0 0x7fb350078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.153+0000 7fb361ffb640 1 -- 172.21.15.81:0/3195539281 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7fb3640977e0 con 0x7fb36c148ef0 2024-06-29T14:37:25.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.153+0000 7fb371759640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fb3500763f0 0x7fb350078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.154+0000 7fb371759640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fb3500763f0 0x7fb350078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb368000f60 tx=0x7fb368009040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.157+0000 7fb361ffb640 1 -- 172.21.15.81:0/3195539281 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb36c0ce710 con 0x7fb36c148ef0 2024-06-29T14:37:25.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.334+0000 7fb3739bd640 1 -- 172.21.15.81:0/3195539281 --> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb36c1f30e0 con 0x7fb3500763f0 2024-06-29T14:37:25.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.340+0000 7fb361ffb640 1 -- 172.21.15.81:0/3195539281 <== mgr.5860 v2:172.21.15.167:6832/3693496302 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251923 (secure 0 0 0) 0x7fb36c1f30e0 con 0x7fb3500763f0 2024-06-29T14:37:25.340 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.342+0000 7fb3739bd640 1 -- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fb3500763f0 msgr2=0x7fb350078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.342+0000 7fb3739bd640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fb3500763f0 0x7fb350078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb368000f60 tx=0x7fb368009040 comp rx=0 tx=0).stop 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.342+0000 7fb3739bd640 1 -- 172.21.15.81:0/3195539281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb36c148ef0 msgr2=0x7fb36c14b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.342+0000 7fb3739bd640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb36c148ef0 0x7fb36c14b6e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb36400c730 tx=0x7fb36400cbf0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.342+0000 7fb3739bd640 1 -- 172.21.15.81:0/3195539281 shutdown_connections 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.342+0000 7fb3739bd640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fb3500763f0 0x7fb350078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.342+0000 7fb3739bd640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb36c148ef0 0x7fb36c14b6e0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.343+0000 7fb3739bd640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb36c15e5f0 0x7fb36c149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.343+0000 7fb3739bd640 1 --2- 172.21.15.81:0/3195539281 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb36c0cefc0 0x7fb36c1f2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.343+0000 7fb3739bd640 1 -- 172.21.15.81:0/3195539281 >> 172.21.15.81:0/3195539281 conn(0x7fb36c06ccc0 msgr2=0x7fb36c06ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.343+0000 7fb3739bd640 1 -- 172.21.15.81:0/3195539281 shutdown_connections 2024-06-29T14:37:25.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.343+0000 7fb3739bd640 1 -- 172.21.15.81:0/3195539281 wait complete. 2024-06-29T14:37:25.352 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:37:24.927747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":194,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310728,"kb_used_data":8500,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754663992,"statfs":{"total":773094113280,"available":772775927808,"internally_reserved":0,"allocated":8704000,"data_stored":4729960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":309317170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.618450"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337424+0000","last_change":"2024-06-29T14:37:21.339050+0000","last_active":"2024-06-29T14:37:22.337424+0000","last_peered":"2024-06-29T14:37:22.337424+0000","last_clean":"2024-06-29T14:37:22.337424+0000","last_became_active":"2024-06-29T14:37:21.338754+0000","last_became_peered":"2024-06-29T14:37:21.338754+0000","last_unstale":"2024-06-29T14:37:22.337424+0000","last_undegraded":"2024-06-29T14:37:22.337424+0000","last_fullsized":"2024-06-29T14:37:22.337424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:14.663303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601638+0000","last_change":"2024-06-29T14:37:21.335806+0000","last_active":"2024-06-29T14:37:22.601638+0000","last_peered":"2024-06-29T14:37:22.601638+0000","last_clean":"2024-06-29T14:37:22.601638+0000","last_became_active":"2024-06-29T14:37:21.335501+0000","last_became_peered":"2024-06-29T14:37:21.335501+0000","last_unstale":"2024-06-29T14:37:22.601638+0000","last_undegraded":"2024-06-29T14:37:22.601638+0000","last_fullsized":"2024-06-29T14:37:22.601638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:48.114271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341610+0000","last_change":"2024-06-29T14:37:21.345040+0000","last_active":"2024-06-29T14:37:22.341610+0000","last_peered":"2024-06-29T14:37:22.341610+0000","last_clean":"2024-06-29T14:37:22.341610+0000","last_became_active":"2024-06-29T14:37:21.344620+0000","last_became_peered":"2024-06-29T14:37:21.344620+0000","last_unstale":"2024-06-29T14:37:22.341610+0000","last_undegraded":"2024-06-29T14:37:22.341610+0000","last_fullsized":"2024-06-29T14:37:22.341610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:56.796698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.337489+0000","last_change":"2024-06-29T14:37:21.330537+0000","last_active":"2024-06-29T14:37:21.337489+0000","last_peered":"2024-06-29T14:37:21.337489+0000","last_clean":"2024-06-29T14:37:21.337489+0000","last_became_active":"2024-06-29T14:37:21.330184+0000","last_became_peered":"2024-06-29T14:37:21.330184+0000","last_unstale":"2024-06-29T14:37:21.337489+0000","last_undegraded":"2024-06-29T14:37:21.337489+0000","last_fullsized":"2024-06-29T14:37:21.337489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:28.461310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337484+0000","last_change":"2024-06-29T14:37:21.339553+0000","last_active":"2024-06-29T14:37:22.337484+0000","last_peered":"2024-06-29T14:37:22.337484+0000","last_clean":"2024-06-29T14:37:22.337484+0000","last_became_active":"2024-06-29T14:37:21.339247+0000","last_became_peered":"2024-06-29T14:37:21.339247+0000","last_unstale":"2024-06-29T14:37:22.337484+0000","last_undegraded":"2024-06-29T14:37:22.337484+0000","last_fullsized":"2024-06-29T14:37:22.337484+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:36.736621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.350868+0000","last_change":"2024-06-29T14:37:21.345589+0000","last_active":"2024-06-29T14:37:21.350868+0000","last_peered":"2024-06-29T14:37:21.350868+0000","last_clean":"2024-06-29T14:37:21.350868+0000","last_became_active":"2024-06-29T14:37:21.345292+0000","last_became_peered":"2024-06-29T14:37:21.345292+0000","last_unstale":"2024-06-29T14:37:21.350868+0000","last_undegraded":"2024-06-29T14:37:21.350868+0000","last_fullsized":"2024-06-29T14:37:21.350868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:22:54.491723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.343626+0000","last_change":"2024-06-29T14:37:21.342997+0000","last_active":"2024-06-29T14:37:21.343626+0000","last_peered":"2024-06-29T14:37:21.343626+0000","last_clean":"2024-06-29T14:37:21.343626+0000","last_became_active":"2024-06-29T14:37:21.342480+0000","last_became_peered":"2024-06-29T14:37:21.342480+0000","last_unstale":"2024-06-29T14:37:21.343626+0000","last_undegraded":"2024-06-29T14:37:21.343626+0000","last_fullsized":"2024-06-29T14:37:21.343626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:13.066212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601254+0000","last_change":"2024-06-29T14:37:21.342958+0000","last_active":"2024-06-29T14:37:22.601254+0000","last_peered":"2024-06-29T14:37:22.601254+0000","last_clean":"2024-06-29T14:37:22.601254+0000","last_became_active":"2024-06-29T14:37:21.342649+0000","last_became_peered":"2024-06-29T14:37:21.342649+0000","last_unstale":"2024-06-29T14:37:22.601254+0000","last_undegraded":"2024-06-29T14:37:22.601254+0000","last_fullsized":"2024-06-29T14:37:22.601254+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:26.622470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.348747+0000","last_change":"2024-06-29T14:37:21.346904+0000","last_active":"2024-06-29T14:37:21.348747+0000","last_peered":"2024-06-29T14:37:21.348747+0000","last_clean":"2024-06-29T14:37:21.348747+0000","last_became_active":"2024-06-29T14:37:21.346479+0000","last_became_peered":"2024-06-29T14:37:21.346479+0000","last_unstale":"2024-06-29T14:37:21.348747+0000","last_undegraded":"2024-06-29T14:37:21.348747+0000","last_fullsized":"2024-06-29T14:37:21.348747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:07.258003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.327035+0000","last_change":"2024-06-29T14:37:21.325952+0000","last_active":"2024-06-29T14:37:21.327035+0000","last_peered":"2024-06-29T14:37:21.327035+0000","last_clean":"2024-06-29T14:37:21.327035+0000","last_became_active":"2024-06-29T14:37:21.325631+0000","last_became_peered":"2024-06-29T14:37:21.325631+0000","last_unstale":"2024-06-29T14:37:21.327035+0000","last_undegraded":"2024-06-29T14:37:21.327035+0000","last_fullsized":"2024-06-29T14:37:21.327035+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:14.460881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.339263+0000","last_change":"2024-06-29T14:37:21.338195+0000","last_active":"2024-06-29T14:37:21.339263+0000","last_peered":"2024-06-29T14:37:21.339263+0000","last_clean":"2024-06-29T14:37:21.339263+0000","last_became_active":"2024-06-29T14:37:21.337893+0000","last_became_peered":"2024-06-29T14:37:21.337893+0000","last_unstale":"2024-06-29T14:37:21.339263+0000","last_undegraded":"2024-06-29T14:37:21.339263+0000","last_fullsized":"2024-06-29T14:37:21.339263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:14.361683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.341155+0000","last_change":"2024-06-29T14:37:21.337735+0000","last_active":"2024-06-29T14:37:21.341155+0000","last_peered":"2024-06-29T14:37:21.341155+0000","last_clean":"2024-06-29T14:37:21.341155+0000","last_became_active":"2024-06-29T14:37:21.337305+0000","last_became_peered":"2024-06-29T14:37:21.337305+0000","last_unstale":"2024-06-29T14:37:21.341155+0000","last_undegraded":"2024-06-29T14:37:21.341155+0000","last_fullsized":"2024-06-29T14:37:21.341155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:17.356643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341402+0000","last_change":"2024-06-29T14:37:21.338020+0000","last_active":"2024-06-29T14:37:22.341402+0000","last_peered":"2024-06-29T14:37:22.341402+0000","last_clean":"2024-06-29T14:37:22.341402+0000","last_became_active":"2024-06-29T14:37:21.337700+0000","last_became_peered":"2024-06-29T14:37:21.337700+0000","last_unstale":"2024-06-29T14:37:22.341402+0000","last_undegraded":"2024-06-29T14:37:22.341402+0000","last_fullsized":"2024-06-29T14:37:22.341402+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:03.847518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347056+0000","last_change":"2024-06-29T14:37:21.344057+0000","last_active":"2024-06-29T14:37:21.347056+0000","last_peered":"2024-06-29T14:37:21.347056+0000","last_clean":"2024-06-29T14:37:21.347056+0000","last_became_active":"2024-06-29T14:37:21.343389+0000","last_became_peered":"2024-06-29T14:37:21.343389+0000","last_unstale":"2024-06-29T14:37:21.347056+0000","last_undegraded":"2024-06-29T14:37:21.347056+0000","last_fullsized":"2024-06-29T14:37:21.347056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:08:47.167252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.343159+0000","last_change":"2024-06-29T14:37:21.341095+0000","last_active":"2024-06-29T14:37:21.343159+0000","last_peered":"2024-06-29T14:37:21.343159+0000","last_clean":"2024-06-29T14:37:21.343159+0000","last_became_active":"2024-06-29T14:37:21.340267+0000","last_became_peered":"2024-06-29T14:37:21.340267+0000","last_unstale":"2024-06-29T14:37:21.343159+0000","last_undegraded":"2024-06-29T14:37:21.343159+0000","last_fullsized":"2024-06-29T14:37:21.343159+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:24.879428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.342656+0000","last_change":"2024-06-29T14:37:21.336893+0000","last_active":"2024-06-29T14:37:21.342656+0000","last_peered":"2024-06-29T14:37:21.342656+0000","last_clean":"2024-06-29T14:37:21.342656+0000","last_became_active":"2024-06-29T14:37:21.336025+0000","last_became_peered":"2024-06-29T14:37:21.336025+0000","last_unstale":"2024-06-29T14:37:21.342656+0000","last_undegraded":"2024-06-29T14:37:21.342656+0000","last_fullsized":"2024-06-29T14:37:21.342656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:36.736621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601706+0000","last_change":"2024-06-29T14:37:21.340901+0000","last_active":"2024-06-29T14:37:22.601706+0000","last_peered":"2024-06-29T14:37:22.601706+0000","last_clean":"2024-06-29T14:37:22.601706+0000","last_became_active":"2024-06-29T14:37:21.336428+0000","last_became_peered":"2024-06-29T14:37:21.336428+0000","last_unstale":"2024-06-29T14:37:22.601706+0000","last_undegraded":"2024-06-29T14:37:22.601706+0000","last_fullsized":"2024-06-29T14:37:22.601706+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:49:54.532144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.342355+0000","last_change":"2024-06-29T14:37:21.344574+0000","last_active":"2024-06-29T14:37:22.342355+0000","last_peered":"2024-06-29T14:37:22.342355+0000","last_clean":"2024-06-29T14:37:22.342355+0000","last_became_active":"2024-06-29T14:37:21.344265+0000","last_became_peered":"2024-06-29T14:37:21.344265+0000","last_unstale":"2024-06-29T14:37:22.342355+0000","last_undegraded":"2024-06-29T14:37:22.342355+0000","last_fullsized":"2024-06-29T14:37:22.342355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:52.721920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337431+0000","last_change":"2024-06-29T14:37:21.337973+0000","last_active":"2024-06-29T14:37:22.337431+0000","last_peered":"2024-06-29T14:37:22.337431+0000","last_clean":"2024-06-29T14:37:22.337431+0000","last_became_active":"2024-06-29T14:37:21.337569+0000","last_became_peered":"2024-06-29T14:37:21.337569+0000","last_unstale":"2024-06-29T14:37:22.337431+0000","last_undegraded":"2024-06-29T14:37:22.337431+0000","last_fullsized":"2024-06-29T14:37:22.337431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:30.816196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601562+0000","last_change":"2024-06-29T14:37:21.341065+0000","last_active":"2024-06-29T14:37:22.601562+0000","last_peered":"2024-06-29T14:37:22.601562+0000","last_clean":"2024-06-29T14:37:22.601562+0000","last_became_active":"2024-06-29T14:37:21.336714+0000","last_became_peered":"2024-06-29T14:37:21.336714+0000","last_unstale":"2024-06-29T14:37:22.601562+0000","last_undegraded":"2024-06-29T14:37:22.601562+0000","last_fullsized":"2024-06-29T14:37:22.601562+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:09.757075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341287+0000","last_change":"2024-06-29T14:37:21.346405+0000","last_active":"2024-06-29T14:37:22.341287+0000","last_peered":"2024-06-29T14:37:22.341287+0000","last_clean":"2024-06-29T14:37:22.341287+0000","last_became_active":"2024-06-29T14:37:21.346086+0000","last_became_peered":"2024-06-29T14:37:21.346086+0000","last_unstale":"2024-06-29T14:37:22.341287+0000","last_undegraded":"2024-06-29T14:37:22.341287+0000","last_fullsized":"2024-06-29T14:37:22.341287+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:27:14.903405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601276+0000","last_change":"2024-06-29T14:37:21.336857+0000","last_active":"2024-06-29T14:37:22.601276+0000","last_peered":"2024-06-29T14:37:22.601276+0000","last_clean":"2024-06-29T14:37:22.601276+0000","last_became_active":"2024-06-29T14:37:21.336257+0000","last_became_peered":"2024-06-29T14:37:21.336257+0000","last_unstale":"2024-06-29T14:37:22.601276+0000","last_undegraded":"2024-06-29T14:37:22.601276+0000","last_fullsized":"2024-06-29T14:37:22.601276+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:39:40.381701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.340982+0000","last_change":"2024-06-29T14:37:21.339712+0000","last_active":"2024-06-29T14:37:22.340982+0000","last_peered":"2024-06-29T14:37:22.340982+0000","last_clean":"2024-06-29T14:37:22.340982+0000","last_became_active":"2024-06-29T14:37:21.339354+0000","last_became_peered":"2024-06-29T14:37:21.339354+0000","last_unstale":"2024-06-29T14:37:22.340982+0000","last_undegraded":"2024-06-29T14:37:22.340982+0000","last_fullsized":"2024-06-29T14:37:22.340982+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:26:01.150918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.340811+0000","last_change":"2024-06-29T14:37:21.341879+0000","last_active":"2024-06-29T14:37:22.340811+0000","last_peered":"2024-06-29T14:37:22.340811+0000","last_clean":"2024-06-29T14:37:22.340811+0000","last_became_active":"2024-06-29T14:37:21.341542+0000","last_became_peered":"2024-06-29T14:37:21.341542+0000","last_unstale":"2024-06-29T14:37:22.340811+0000","last_undegraded":"2024-06-29T14:37:22.340811+0000","last_fullsized":"2024-06-29T14:37:22.340811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:57.946769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337586+0000","last_change":"2024-06-29T14:37:21.328524+0000","last_active":"2024-06-29T14:37:22.337586+0000","last_peered":"2024-06-29T14:37:22.337586+0000","last_clean":"2024-06-29T14:37:22.337586+0000","last_became_active":"2024-06-29T14:37:21.328200+0000","last_became_peered":"2024-06-29T14:37:21.328200+0000","last_unstale":"2024-06-29T14:37:22.337586+0000","last_undegraded":"2024-06-29T14:37:22.337586+0000","last_fullsized":"2024-06-29T14:37:22.337586+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:14.460881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.344778+0000","last_change":"2024-06-29T14:37:21.341751+0000","last_active":"2024-06-29T14:37:21.344778+0000","last_peered":"2024-06-29T14:37:21.344778+0000","last_clean":"2024-06-29T14:37:21.344778+0000","last_became_active":"2024-06-29T14:37:21.341413+0000","last_became_peered":"2024-06-29T14:37:21.341413+0000","last_unstale":"2024-06-29T14:37:21.344778+0000","last_undegraded":"2024-06-29T14:37:21.344778+0000","last_fullsized":"2024-06-29T14:37:21.344778+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:52.816582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347933+0000","last_change":"2024-06-29T14:37:21.342502+0000","last_active":"2024-06-29T14:37:21.347933+0000","last_peered":"2024-06-29T14:37:21.347933+0000","last_clean":"2024-06-29T14:37:21.347933+0000","last_became_active":"2024-06-29T14:37:21.341363+0000","last_became_peered":"2024-06-29T14:37:21.341363+0000","last_unstale":"2024-06-29T14:37:21.347933+0000","last_undegraded":"2024-06-29T14:37:21.347933+0000","last_fullsized":"2024-06-29T14:37:21.347933+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:54.900769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337669+0000","last_change":"2024-06-29T14:37:21.340399+0000","last_active":"2024-06-29T14:37:22.337669+0000","last_peered":"2024-06-29T14:37:22.337669+0000","last_clean":"2024-06-29T14:37:22.337669+0000","last_became_active":"2024-06-29T14:37:21.340111+0000","last_became_peered":"2024-06-29T14:37:21.340111+0000","last_unstale":"2024-06-29T14:37:22.337669+0000","last_undegraded":"2024-06-29T14:37:22.337669+0000","last_fullsized":"2024-06-29T14:37:22.337669+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:43:33.783969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601643+0000","last_change":"2024-06-29T14:37:21.335288+0000","last_active":"2024-06-29T14:37:22.601643+0000","last_peered":"2024-06-29T14:37:22.601643+0000","last_clean":"2024-06-29T14:37:22.601643+0000","last_became_active":"2024-06-29T14:37:21.334896+0000","last_became_peered":"2024-06-29T14:37:21.334896+0000","last_unstale":"2024-06-29T14:37:22.601643+0000","last_undegraded":"2024-06-29T14:37:22.601643+0000","last_fullsized":"2024-06-29T14:37:22.601643+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:47.742605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341196+0000","last_change":"2024-06-29T14:37:21.345910+0000","last_active":"2024-06-29T14:37:22.341196+0000","last_peered":"2024-06-29T14:37:22.341196+0000","last_clean":"2024-06-29T14:37:22.341196+0000","last_became_active":"2024-06-29T14:37:21.345327+0000","last_became_peered":"2024-06-29T14:37:21.345327+0000","last_unstale":"2024-06-29T14:37:22.341196+0000","last_undegraded":"2024-06-29T14:37:22.341196+0000","last_fullsized":"2024-06-29T14:37:22.341196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:01.286008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337288+0000","last_change":"2024-06-29T14:37:21.329735+0000","last_active":"2024-06-29T14:37:22.337288+0000","last_peered":"2024-06-29T14:37:22.337288+0000","last_clean":"2024-06-29T14:37:22.337288+0000","last_became_active":"2024-06-29T14:37:21.329423+0000","last_became_peered":"2024-06-29T14:37:21.329423+0000","last_unstale":"2024-06-29T14:37:22.337288+0000","last_undegraded":"2024-06-29T14:37:22.337288+0000","last_fullsized":"2024-06-29T14:37:22.337288+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:51.009669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.342677+0000","last_change":"2024-06-29T14:37:21.338099+0000","last_active":"2024-06-29T14:37:21.342677+0000","last_peered":"2024-06-29T14:37:21.342677+0000","last_clean":"2024-06-29T14:37:21.342677+0000","last_became_active":"2024-06-29T14:37:21.337248+0000","last_became_peered":"2024-06-29T14:37:21.337248+0000","last_unstale":"2024-06-29T14:37:21.342677+0000","last_undegraded":"2024-06-29T14:37:21.342677+0000","last_fullsized":"2024-06-29T14:37:21.342677+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:43:33.783969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347135+0000","last_change":"2024-06-29T14:37:21.343236+0000","last_active":"2024-06-29T14:37:21.347135+0000","last_peered":"2024-06-29T14:37:21.347135+0000","last_clean":"2024-06-29T14:37:21.347135+0000","last_became_active":"2024-06-29T14:37:21.342939+0000","last_became_peered":"2024-06-29T14:37:21.342939+0000","last_unstale":"2024-06-29T14:37:21.347135+0000","last_undegraded":"2024-06-29T14:37:21.347135+0000","last_fullsized":"2024-06-29T14:37:21.347135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:59.527370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"77'4","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.353920+0000","last_change":"2024-06-29T14:37:20.332839+0000","last_active":"2024-06-29T14:37:21.353920+0000","last_peered":"2024-06-29T14:37:21.353920+0000","last_clean":"2024-06-29T14:37:21.353920+0000","last_became_active":"2024-06-29T14:37:20.332510+0000","last_became_peered":"2024-06-29T14:37:20.332510+0000","last_unstale":"2024-06-29T14:37:21.353920+0000","last_undegraded":"2024-06-29T14:37:21.353920+0000","last_fullsized":"2024-06-29T14:37:21.353920+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:43:42.846682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.346727+0000","last_change":"2024-06-29T14:37:21.342890+0000","last_active":"2024-06-29T14:37:21.346727+0000","last_peered":"2024-06-29T14:37:21.346727+0000","last_clean":"2024-06-29T14:37:21.346727+0000","last_became_active":"2024-06-29T14:37:21.342595+0000","last_became_peered":"2024-06-29T14:37:21.342595+0000","last_unstale":"2024-06-29T14:37:21.346727+0000","last_undegraded":"2024-06-29T14:37:21.346727+0000","last_fullsized":"2024-06-29T14:37:21.346727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:54.900769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601671+0000","last_change":"2024-06-29T14:37:20.332603+0000","last_active":"2024-06-29T14:37:22.601671+0000","last_peered":"2024-06-29T14:37:22.601671+0000","last_clean":"2024-06-29T14:37:22.601671+0000","last_became_active":"2024-06-29T14:37:20.330978+0000","last_became_peered":"2024-06-29T14:37:20.330978+0000","last_unstale":"2024-06-29T14:37:22.601671+0000","last_undegraded":"2024-06-29T14:37:22.601671+0000","last_fullsized":"2024-06-29T14:37:22.601671+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:18:44.692648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.341808+0000","last_change":"2024-06-29T14:37:21.340205+0000","last_active":"2024-06-29T14:37:21.341808+0000","last_peered":"2024-06-29T14:37:21.341808+0000","last_clean":"2024-06-29T14:37:21.341808+0000","last_became_active":"2024-06-29T14:37:21.339261+0000","last_became_peered":"2024-06-29T14:37:21.339261+0000","last_unstale":"2024-06-29T14:37:21.341808+0000","last_undegraded":"2024-06-29T14:37:21.341808+0000","last_fullsized":"2024-06-29T14:37:21.341808+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:34.472101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.352347+0000","last_change":"2024-06-29T14:37:20.333105+0000","last_active":"2024-06-29T14:37:21.352347+0000","last_peered":"2024-06-29T14:37:21.352347+0000","last_clean":"2024-06-29T14:37:21.352347+0000","last_became_active":"2024-06-29T14:37:20.331980+0000","last_became_peered":"2024-06-29T14:37:20.331980+0000","last_unstale":"2024-06-29T14:37:21.352347+0000","last_undegraded":"2024-06-29T14:37:21.352347+0000","last_fullsized":"2024-06-29T14:37:21.352347+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:13.658291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.342878+0000","last_change":"2024-06-29T14:37:21.338556+0000","last_active":"2024-06-29T14:37:21.342878+0000","last_peered":"2024-06-29T14:37:21.342878+0000","last_clean":"2024-06-29T14:37:21.342878+0000","last_became_active":"2024-06-29T14:37:21.338264+0000","last_became_peered":"2024-06-29T14:37:21.338264+0000","last_unstale":"2024-06-29T14:37:21.342878+0000","last_undegraded":"2024-06-29T14:37:21.342878+0000","last_fullsized":"2024-06-29T14:37:21.342878+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:14.361683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.353205+0000","last_change":"2024-06-29T14:37:20.328019+0000","last_active":"2024-06-29T14:37:21.353205+0000","last_peered":"2024-06-29T14:37:21.353205+0000","last_clean":"2024-06-29T14:37:21.353205+0000","last_became_active":"2024-06-29T14:37:20.327688+0000","last_became_peered":"2024-06-29T14:37:20.327688+0000","last_unstale":"2024-06-29T14:37:21.353205+0000","last_undegraded":"2024-06-29T14:37:21.353205+0000","last_fullsized":"2024-06-29T14:37:21.353205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:58.968350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601309+0000","last_change":"2024-06-29T14:37:21.337144+0000","last_active":"2024-06-29T14:37:22.601309+0000","last_peered":"2024-06-29T14:37:22.601309+0000","last_clean":"2024-06-29T14:37:22.601309+0000","last_became_active":"2024-06-29T14:37:21.336538+0000","last_became_peered":"2024-06-29T14:37:21.336538+0000","last_unstale":"2024-06-29T14:37:22.601309+0000","last_undegraded":"2024-06-29T14:37:22.601309+0000","last_fullsized":"2024-06-29T14:37:22.601309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:40:50.731137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.348705+0000","last_change":"2024-06-29T14:37:20.327124+0000","last_active":"2024-06-29T14:37:21.348705+0000","last_peered":"2024-06-29T14:37:21.348705+0000","last_clean":"2024-06-29T14:37:21.348705+0000","last_became_active":"2024-06-29T14:37:20.326807+0000","last_became_peered":"2024-06-29T14:37:20.326807+0000","last_unstale":"2024-06-29T14:37:21.348705+0000","last_undegraded":"2024-06-29T14:37:21.348705+0000","last_fullsized":"2024-06-29T14:37:21.348705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:02:22.616858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601585+0000","last_change":"2024-06-29T14:37:21.338958+0000","last_active":"2024-06-29T14:37:22.601585+0000","last_peered":"2024-06-29T14:37:22.601585+0000","last_clean":"2024-06-29T14:37:22.601585+0000","last_became_active":"2024-06-29T14:37:21.338607+0000","last_became_peered":"2024-06-29T14:37:21.338607+0000","last_unstale":"2024-06-29T14:37:22.601585+0000","last_undegraded":"2024-06-29T14:37:22.601585+0000","last_fullsized":"2024-06-29T14:37:22.601585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:51.451894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.352161+0000","last_change":"2024-06-29T14:37:20.336366+0000","last_active":"2024-06-29T14:37:21.352161+0000","last_peered":"2024-06-29T14:37:21.352161+0000","last_clean":"2024-06-29T14:37:21.352161+0000","last_became_active":"2024-06-29T14:37:20.335996+0000","last_became_peered":"2024-06-29T14:37:20.335996+0000","last_unstale":"2024-06-29T14:37:21.352161+0000","last_undegraded":"2024-06-29T14:37:21.352161+0000","last_fullsized":"2024-06-29T14:37:21.352161+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:50.004657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.340866+0000","last_change":"2024-06-29T14:37:21.335781+0000","last_active":"2024-06-29T14:37:22.340866+0000","last_peered":"2024-06-29T14:37:22.340866+0000","last_clean":"2024-06-29T14:37:22.340866+0000","last_became_active":"2024-06-29T14:37:21.335395+0000","last_became_peered":"2024-06-29T14:37:21.335395+0000","last_unstale":"2024-06-29T14:37:22.340866+0000","last_undegraded":"2024-06-29T14:37:22.340866+0000","last_fullsized":"2024-06-29T14:37:22.340866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:52:00.907845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.324089+0000","last_change":"2024-06-29T14:37:20.324816+0000","last_active":"2024-06-29T14:37:21.324089+0000","last_peered":"2024-06-29T14:37:21.324089+0000","last_clean":"2024-06-29T14:37:21.324089+0000","last_became_active":"2024-06-29T14:37:20.324298+0000","last_became_peered":"2024-06-29T14:37:20.324298+0000","last_unstale":"2024-06-29T14:37:21.324089+0000","last_undegraded":"2024-06-29T14:37:21.324089+0000","last_fullsized":"2024-06-29T14:37:21.324089+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:07.296498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601333+0000","last_change":"2024-06-29T14:37:21.337674+0000","last_active":"2024-06-29T14:37:22.601333+0000","last_peered":"2024-06-29T14:37:22.601333+0000","last_clean":"2024-06-29T14:37:22.601333+0000","last_became_active":"2024-06-29T14:37:21.335834+0000","last_became_peered":"2024-06-29T14:37:21.335834+0000","last_unstale":"2024-06-29T14:37:22.601333+0000","last_undegraded":"2024-06-29T14:37:22.601333+0000","last_fullsized":"2024-06-29T14:37:22.601333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:27:15.828871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.330690+0000","last_change":"2024-06-29T14:37:20.328434+0000","last_active":"2024-06-29T14:37:21.330690+0000","last_peered":"2024-06-29T14:37:21.330690+0000","last_clean":"2024-06-29T14:37:21.330690+0000","last_became_active":"2024-06-29T14:37:20.327785+0000","last_became_peered":"2024-06-29T14:37:20.327785+0000","last_unstale":"2024-06-29T14:37:21.330690+0000","last_undegraded":"2024-06-29T14:37:21.330690+0000","last_fullsized":"2024-06-29T14:37:21.330690+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:43.272884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.351162+0000","last_change":"2024-06-29T14:37:21.349641+0000","last_active":"2024-06-29T14:37:21.351162+0000","last_peered":"2024-06-29T14:37:21.351162+0000","last_clean":"2024-06-29T14:37:21.351162+0000","last_became_active":"2024-06-29T14:37:21.349316+0000","last_became_peered":"2024-06-29T14:37:21.349316+0000","last_unstale":"2024-06-29T14:37:21.351162+0000","last_undegraded":"2024-06-29T14:37:21.351162+0000","last_fullsized":"2024-06-29T14:37:21.351162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:38.951897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.342105+0000","last_change":"2024-06-29T14:37:20.328496+0000","last_active":"2024-06-29T14:37:22.342105+0000","last_peered":"2024-06-29T14:37:22.342105+0000","last_clean":"2024-06-29T14:37:22.342105+0000","last_became_active":"2024-06-29T14:37:20.328145+0000","last_became_peered":"2024-06-29T14:37:20.328145+0000","last_unstale":"2024-06-29T14:37:22.342105+0000","last_undegraded":"2024-06-29T14:37:22.342105+0000","last_fullsized":"2024-06-29T14:37:22.342105+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:21.758521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341994+0000","last_change":"2024-06-29T14:37:21.350645+0000","last_active":"2024-06-29T14:37:22.341994+0000","last_peered":"2024-06-29T14:37:22.341994+0000","last_clean":"2024-06-29T14:37:22.341994+0000","last_became_active":"2024-06-29T14:37:21.349911+0000","last_became_peered":"2024-06-29T14:37:21.349911+0000","last_unstale":"2024-06-29T14:37:22.341994+0000","last_undegraded":"2024-06-29T14:37:22.341994+0000","last_fullsized":"2024-06-29T14:37:22.341994+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:29.306968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601731+0000","last_change":"2024-06-29T14:37:20.336008+0000","last_active":"2024-06-29T14:37:22.601731+0000","last_peered":"2024-06-29T14:37:22.601731+0000","last_clean":"2024-06-29T14:37:22.601731+0000","last_became_active":"2024-06-29T14:37:20.335655+0000","last_became_peered":"2024-06-29T14:37:20.335655+0000","last_unstale":"2024-06-29T14:37:22.601731+0000","last_undegraded":"2024-06-29T14:37:22.601731+0000","last_fullsized":"2024-06-29T14:37:22.601731+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:10:58.466853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347539+0000","last_change":"2024-06-29T14:37:21.344101+0000","last_active":"2024-06-29T14:37:21.347539+0000","last_peered":"2024-06-29T14:37:21.347539+0000","last_clean":"2024-06-29T14:37:21.347539+0000","last_became_active":"2024-06-29T14:37:21.343805+0000","last_became_peered":"2024-06-29T14:37:21.343805+0000","last_unstale":"2024-06-29T14:37:21.347539+0000","last_undegraded":"2024-06-29T14:37:21.347539+0000","last_fullsized":"2024-06-29T14:37:21.347539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:10:42.782502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"77'3","reported_seq":17,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341864+0000","last_change":"2024-06-29T14:37:20.331719+0000","last_active":"2024-06-29T14:37:22.341864+0000","last_peered":"2024-06-29T14:37:22.341864+0000","last_clean":"2024-06-29T14:37:22.341864+0000","last_became_active":"2024-06-29T14:37:20.331434+0000","last_became_peered":"2024-06-29T14:37:20.331434+0000","last_unstale":"2024-06-29T14:37:22.341864+0000","last_undegraded":"2024-06-29T14:37:22.341864+0000","last_fullsized":"2024-06-29T14:37:22.341864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:00:13.196568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341737+0000","last_change":"2024-06-29T14:37:21.349455+0000","last_active":"2024-06-29T14:37:22.341737+0000","last_peered":"2024-06-29T14:37:22.341737+0000","last_clean":"2024-06-29T14:37:22.341737+0000","last_became_active":"2024-06-29T14:37:21.349137+0000","last_became_peered":"2024-06-29T14:37:21.349137+0000","last_unstale":"2024-06-29T14:37:22.341737+0000","last_undegraded":"2024-06-29T14:37:22.341737+0000","last_fullsized":"2024-06-29T14:37:22.341737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:40:22.699838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"77'6","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.355736+0000","last_change":"2024-06-29T14:37:20.333936+0000","last_active":"2024-06-29T14:37:21.355736+0000","last_peered":"2024-06-29T14:37:21.355736+0000","last_clean":"2024-06-29T14:37:21.355736+0000","last_became_active":"2024-06-29T14:37:20.333638+0000","last_became_peered":"2024-06-29T14:37:20.333638+0000","last_unstale":"2024-06-29T14:37:21.355736+0000","last_undegraded":"2024-06-29T14:37:21.355736+0000","last_fullsized":"2024-06-29T14:37:21.355736+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:52:28.519994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.337294+0000","last_change":"2024-06-29T14:37:21.332515+0000","last_active":"2024-06-29T14:37:21.337294+0000","last_peered":"2024-06-29T14:37:21.337294+0000","last_clean":"2024-06-29T14:37:21.337294+0000","last_became_active":"2024-06-29T14:37:21.332214+0000","last_became_peered":"2024-06-29T14:37:21.332214+0000","last_unstale":"2024-06-29T14:37:21.337294+0000","last_undegraded":"2024-06-29T14:37:21.337294+0000","last_fullsized":"2024-06-29T14:37:21.337294+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:02.199586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601461+0000","last_change":"2024-06-29T14:37:20.332067+0000","last_active":"2024-06-29T14:37:22.601461+0000","last_peered":"2024-06-29T14:37:22.601461+0000","last_clean":"2024-06-29T14:37:22.601461+0000","last_became_active":"2024-06-29T14:37:20.330978+0000","last_became_peered":"2024-06-29T14:37:20.330978+0000","last_unstale":"2024-06-29T14:37:22.601461+0000","last_undegraded":"2024-06-29T14:37:22.601461+0000","last_fullsized":"2024-06-29T14:37:22.601461+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:08:23.406581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341429+0000","last_change":"2024-06-29T14:37:21.339874+0000","last_active":"2024-06-29T14:37:22.341429+0000","last_peered":"2024-06-29T14:37:22.341429+0000","last_clean":"2024-06-29T14:37:22.341429+0000","last_became_active":"2024-06-29T14:37:21.339538+0000","last_became_peered":"2024-06-29T14:37:21.339538+0000","last_unstale":"2024-06-29T14:37:22.341429+0000","last_undegraded":"2024-06-29T14:37:22.341429+0000","last_fullsized":"2024-06-29T14:37:22.341429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:44:53.221807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341621+0000","last_change":"2024-06-29T14:37:20.327726+0000","last_active":"2024-06-29T14:37:22.341621+0000","last_peered":"2024-06-29T14:37:22.341621+0000","last_clean":"2024-06-29T14:37:22.341621+0000","last_became_active":"2024-06-29T14:37:20.327419+0000","last_became_peered":"2024-06-29T14:37:20.327419+0000","last_unstale":"2024-06-29T14:37:22.341621+0000","last_undegraded":"2024-06-29T14:37:22.341621+0000","last_fullsized":"2024-06-29T14:37:22.341621+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:52.133200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.353334+0000","last_change":"2024-06-29T14:37:21.352035+0000","last_active":"2024-06-29T14:37:21.353334+0000","last_peered":"2024-06-29T14:37:21.353334+0000","last_clean":"2024-06-29T14:37:21.353334+0000","last_became_active":"2024-06-29T14:37:21.351683+0000","last_became_peered":"2024-06-29T14:37:21.351683+0000","last_unstale":"2024-06-29T14:37:21.353334+0000","last_undegraded":"2024-06-29T14:37:21.353334+0000","last_fullsized":"2024-06-29T14:37:21.353334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:07.732295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.324673+0000","last_change":"2024-06-29T14:37:20.330644+0000","last_active":"2024-06-29T14:37:21.324673+0000","last_peered":"2024-06-29T14:37:21.324673+0000","last_clean":"2024-06-29T14:37:21.324673+0000","last_became_active":"2024-06-29T14:37:20.329982+0000","last_became_peered":"2024-06-29T14:37:20.329982+0000","last_unstale":"2024-06-29T14:37:21.324673+0000","last_undegraded":"2024-06-29T14:37:21.324673+0000","last_fullsized":"2024-06-29T14:37:21.324673+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:28:20.994101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341127+0000","last_change":"2024-06-29T14:37:21.340382+0000","last_active":"2024-06-29T14:37:22.341127+0000","last_peered":"2024-06-29T14:37:22.341127+0000","last_clean":"2024-06-29T14:37:22.341127+0000","last_became_active":"2024-06-29T14:37:21.340108+0000","last_became_peered":"2024-06-29T14:37:21.340108+0000","last_unstale":"2024-06-29T14:37:22.341127+0000","last_undegraded":"2024-06-29T14:37:22.341127+0000","last_fullsized":"2024-06-29T14:37:22.341127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:42:45.066789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"77'8","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341549+0000","last_change":"2024-06-29T14:37:20.333200+0000","last_active":"2024-06-29T14:37:22.341549+0000","last_peered":"2024-06-29T14:37:22.341549+0000","last_clean":"2024-06-29T14:37:22.341549+0000","last_became_active":"2024-06-29T14:37:20.332809+0000","last_became_peered":"2024-06-29T14:37:20.332809+0000","last_unstale":"2024-06-29T14:37:22.341549+0000","last_undegraded":"2024-06-29T14:37:22.341549+0000","last_fullsized":"2024-06-29T14:37:22.341549+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:18:46.403873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.342517+0000","last_change":"2024-06-29T14:37:21.345719+0000","last_active":"2024-06-29T14:37:22.342517+0000","last_peered":"2024-06-29T14:37:22.342517+0000","last_clean":"2024-06-29T14:37:22.342517+0000","last_became_active":"2024-06-29T14:37:21.344358+0000","last_became_peered":"2024-06-29T14:37:21.344358+0000","last_unstale":"2024-06-29T14:37:22.342517+0000","last_undegraded":"2024-06-29T14:37:22.342517+0000","last_fullsized":"2024-06-29T14:37:22.342517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:26.622470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601748+0000","last_change":"2024-06-29T14:37:20.331191+0000","last_active":"2024-06-29T14:37:22.601748+0000","last_peered":"2024-06-29T14:37:22.601748+0000","last_clean":"2024-06-29T14:37:22.601748+0000","last_became_active":"2024-06-29T14:37:20.330799+0000","last_became_peered":"2024-06-29T14:37:20.330799+0000","last_unstale":"2024-06-29T14:37:22.601748+0000","last_undegraded":"2024-06-29T14:37:22.601748+0000","last_fullsized":"2024-06-29T14:37:22.601748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:55.068765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.336983+0000","last_change":"2024-06-29T14:37:21.334231+0000","last_active":"2024-06-29T14:37:21.336983+0000","last_peered":"2024-06-29T14:37:21.336983+0000","last_clean":"2024-06-29T14:37:21.336983+0000","last_became_active":"2024-06-29T14:37:21.333878+0000","last_became_peered":"2024-06-29T14:37:21.333878+0000","last_unstale":"2024-06-29T14:37:21.336983+0000","last_undegraded":"2024-06-29T14:37:21.336983+0000","last_fullsized":"2024-06-29T14:37:21.336983+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:49.784323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.352258+0000","last_change":"2024-06-29T14:37:20.331529+0000","last_active":"2024-06-29T14:37:21.352258+0000","last_peered":"2024-06-29T14:37:21.352258+0000","last_clean":"2024-06-29T14:37:21.352258+0000","last_became_active":"2024-06-29T14:37:20.331200+0000","last_became_peered":"2024-06-29T14:37:20.331200+0000","last_unstale":"2024-06-29T14:37:21.352258+0000","last_undegraded":"2024-06-29T14:37:21.352258+0000","last_fullsized":"2024-06-29T14:37:21.352258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:32:01.679875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601649+0000","last_change":"2024-06-29T14:37:21.344219+0000","last_active":"2024-06-29T14:37:22.601649+0000","last_peered":"2024-06-29T14:37:22.601649+0000","last_clean":"2024-06-29T14:37:22.601649+0000","last_became_active":"2024-06-29T14:37:21.343911+0000","last_became_peered":"2024-06-29T14:37:21.343911+0000","last_unstale":"2024-06-29T14:37:22.601649+0000","last_undegraded":"2024-06-29T14:37:22.601649+0000","last_fullsized":"2024-06-29T14:37:22.601649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:55.557195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.352771+0000","last_change":"2024-06-29T14:37:20.329583+0000","last_active":"2024-06-29T14:37:21.352771+0000","last_peered":"2024-06-29T14:37:21.352771+0000","last_clean":"2024-06-29T14:37:21.352771+0000","last_became_active":"2024-06-29T14:37:20.329219+0000","last_became_peered":"2024-06-29T14:37:20.329219+0000","last_unstale":"2024-06-29T14:37:21.352771+0000","last_undegraded":"2024-06-29T14:37:21.352771+0000","last_fullsized":"2024-06-29T14:37:21.352771+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:00:18.929987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341790+0000","last_change":"2024-06-29T14:37:21.351369+0000","last_active":"2024-06-29T14:37:22.341790+0000","last_peered":"2024-06-29T14:37:22.341790+0000","last_clean":"2024-06-29T14:37:22.341790+0000","last_became_active":"2024-06-29T14:37:21.351082+0000","last_became_peered":"2024-06-29T14:37:21.351082+0000","last_unstale":"2024-06-29T14:37:22.341790+0000","last_undegraded":"2024-06-29T14:37:22.341790+0000","last_fullsized":"2024-06-29T14:37:22.341790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:14:09.378628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.330706+0000","last_change":"2024-06-29T14:37:20.334328+0000","last_active":"2024-06-29T14:37:21.330706+0000","last_peered":"2024-06-29T14:37:21.330706+0000","last_clean":"2024-06-29T14:37:21.330706+0000","last_became_active":"2024-06-29T14:37:20.333959+0000","last_became_peered":"2024-06-29T14:37:20.333959+0000","last_unstale":"2024-06-29T14:37:21.330706+0000","last_undegraded":"2024-06-29T14:37:21.330706+0000","last_fullsized":"2024-06-29T14:37:21.330706+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:39.866303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.337272+0000","last_change":"2024-06-29T14:37:21.331835+0000","last_active":"2024-06-29T14:37:21.337272+0000","last_peered":"2024-06-29T14:37:21.337272+0000","last_clean":"2024-06-29T14:37:21.337272+0000","last_became_active":"2024-06-29T14:37:21.331459+0000","last_became_peered":"2024-06-29T14:37:21.331459+0000","last_unstale":"2024-06-29T14:37:21.337272+0000","last_undegraded":"2024-06-29T14:37:21.337272+0000","last_fullsized":"2024-06-29T14:37:21.337272+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:31.348583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601703+0000","last_change":"2024-06-29T14:37:20.331768+0000","last_active":"2024-06-29T14:37:22.601703+0000","last_peered":"2024-06-29T14:37:22.601703+0000","last_clean":"2024-06-29T14:37:22.601703+0000","last_became_active":"2024-06-29T14:37:20.331463+0000","last_became_peered":"2024-06-29T14:37:20.331463+0000","last_unstale":"2024-06-29T14:37:22.601703+0000","last_undegraded":"2024-06-29T14:37:22.601703+0000","last_fullsized":"2024-06-29T14:37:22.601703+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:02:25.719152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.339576+0000","last_change":"2024-06-29T14:37:21.334081+0000","last_active":"2024-06-29T14:37:21.339576+0000","last_peered":"2024-06-29T14:37:21.339576+0000","last_clean":"2024-06-29T14:37:21.339576+0000","last_became_active":"2024-06-29T14:37:21.333798+0000","last_became_peered":"2024-06-29T14:37:21.333798+0000","last_unstale":"2024-06-29T14:37:21.339576+0000","last_undegraded":"2024-06-29T14:37:21.339576+0000","last_fullsized":"2024-06-29T14:37:21.339576+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:30.816196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.321025+0000","last_change":"2024-06-29T14:37:20.321965+0000","last_active":"2024-06-29T14:37:21.321025+0000","last_peered":"2024-06-29T14:37:21.321025+0000","last_clean":"2024-06-29T14:37:21.321025+0000","last_became_active":"2024-06-29T14:37:20.321651+0000","last_became_peered":"2024-06-29T14:37:20.321651+0000","last_unstale":"2024-06-29T14:37:21.321025+0000","last_undegraded":"2024-06-29T14:37:21.321025+0000","last_fullsized":"2024-06-29T14:37:21.321025+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:50.004657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347164+0000","last_change":"2024-06-29T14:37:21.342704+0000","last_active":"2024-06-29T14:37:21.347164+0000","last_peered":"2024-06-29T14:37:21.347164+0000","last_clean":"2024-06-29T14:37:21.347164+0000","last_became_active":"2024-06-29T14:37:21.342411+0000","last_became_peered":"2024-06-29T14:37:21.342411+0000","last_unstale":"2024-06-29T14:37:21.347164+0000","last_undegraded":"2024-06-29T14:37:21.347164+0000","last_fullsized":"2024-06-29T14:37:21.347164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:10:42.782502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.323435+0000","last_change":"2024-06-29T14:37:20.331301+0000","last_active":"2024-06-29T14:37:21.323435+0000","last_peered":"2024-06-29T14:37:21.323435+0000","last_clean":"2024-06-29T14:37:21.323435+0000","last_became_active":"2024-06-29T14:37:20.331006+0000","last_became_peered":"2024-06-29T14:37:20.331006+0000","last_unstale":"2024-06-29T14:37:21.323435+0000","last_undegraded":"2024-06-29T14:37:21.323435+0000","last_fullsized":"2024-06-29T14:37:21.323435+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:44:41.313263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.339315+0000","last_change":"2024-06-29T14:37:21.330028+0000","last_active":"2024-06-29T14:37:21.339315+0000","last_peered":"2024-06-29T14:37:21.339315+0000","last_clean":"2024-06-29T14:37:21.339315+0000","last_became_active":"2024-06-29T14:37:21.329385+0000","last_became_peered":"2024-06-29T14:37:21.329385+0000","last_unstale":"2024-06-29T14:37:21.339315+0000","last_undegraded":"2024-06-29T14:37:21.339315+0000","last_fullsized":"2024-06-29T14:37:21.339315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:38:34.058779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.349457+0000","last_change":"2024-06-29T14:37:20.335098+0000","last_active":"2024-06-29T14:37:21.349457+0000","last_peered":"2024-06-29T14:37:21.349457+0000","last_clean":"2024-06-29T14:37:21.349457+0000","last_became_active":"2024-06-29T14:37:20.334764+0000","last_became_peered":"2024-06-29T14:37:20.334764+0000","last_unstale":"2024-06-29T14:37:21.349457+0000","last_undegraded":"2024-06-29T14:37:21.349457+0000","last_fullsized":"2024-06-29T14:37:21.349457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:35.731609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.344769+0000","last_change":"2024-06-29T14:37:21.341948+0000","last_active":"2024-06-29T14:37:21.344769+0000","last_peered":"2024-06-29T14:37:21.344769+0000","last_clean":"2024-06-29T14:37:21.344769+0000","last_became_active":"2024-06-29T14:37:21.341579+0000","last_became_peered":"2024-06-29T14:37:21.341579+0000","last_unstale":"2024-06-29T14:37:21.344769+0000","last_undegraded":"2024-06-29T14:37:21.344769+0000","last_fullsized":"2024-06-29T14:37:21.344769+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:50.328395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.334554+0000","last_change":"2024-06-29T14:37:20.329392+0000","last_active":"2024-06-29T14:37:22.334554+0000","last_peered":"2024-06-29T14:37:22.334554+0000","last_clean":"2024-06-29T14:37:22.334554+0000","last_became_active":"2024-06-29T14:37:20.328262+0000","last_became_peered":"2024-06-29T14:37:20.328262+0000","last_unstale":"2024-06-29T14:37:22.334554+0000","last_undegraded":"2024-06-29T14:37:22.334554+0000","last_fullsized":"2024-06-29T14:37:22.334554+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:45.186965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347514+0000","last_change":"2024-06-29T14:37:21.340775+0000","last_active":"2024-06-29T14:37:21.347514+0000","last_peered":"2024-06-29T14:37:21.347514+0000","last_clean":"2024-06-29T14:37:21.347514+0000","last_became_active":"2024-06-29T14:37:21.340459+0000","last_became_peered":"2024-06-29T14:37:21.340459+0000","last_unstale":"2024-06-29T14:37:21.347514+0000","last_undegraded":"2024-06-29T14:37:21.347514+0000","last_fullsized":"2024-06-29T14:37:21.347514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:34.472101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341831+0000","last_change":"2024-06-29T14:37:20.330320+0000","last_active":"2024-06-29T14:37:22.341831+0000","last_peered":"2024-06-29T14:37:22.341831+0000","last_clean":"2024-06-29T14:37:22.341831+0000","last_became_active":"2024-06-29T14:37:20.330015+0000","last_became_peered":"2024-06-29T14:37:20.330015+0000","last_unstale":"2024-06-29T14:37:22.341831+0000","last_undegraded":"2024-06-29T14:37:22.341831+0000","last_fullsized":"2024-06-29T14:37:22.341831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:46:27.470117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.342855+0000","last_change":"2024-06-29T14:37:21.337846+0000","last_active":"2024-06-29T14:37:21.342855+0000","last_peered":"2024-06-29T14:37:21.342855+0000","last_clean":"2024-06-29T14:37:21.342855+0000","last_became_active":"2024-06-29T14:37:21.337554+0000","last_became_peered":"2024-06-29T14:37:21.337554+0000","last_unstale":"2024-06-29T14:37:21.342855+0000","last_undegraded":"2024-06-29T14:37:21.342855+0000","last_fullsized":"2024-06-29T14:37:21.342855+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:14.460881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.324774+0000","last_change":"2024-06-29T14:37:20.325501+0000","last_active":"2024-06-29T14:37:21.324774+0000","last_peered":"2024-06-29T14:37:21.324774+0000","last_clean":"2024-06-29T14:37:21.324774+0000","last_became_active":"2024-06-29T14:37:20.325177+0000","last_became_peered":"2024-06-29T14:37:20.325177+0000","last_unstale":"2024-06-29T14:37:21.324774+0000","last_undegraded":"2024-06-29T14:37:21.324774+0000","last_fullsized":"2024-06-29T14:37:21.324774+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:37:20.299114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601492+0000","last_change":"2024-06-29T14:37:21.336648+0000","last_active":"2024-06-29T14:37:22.601492+0000","last_peered":"2024-06-29T14:37:22.601492+0000","last_clean":"2024-06-29T14:37:22.601492+0000","last_became_active":"2024-06-29T14:37:21.336213+0000","last_became_peered":"2024-06-29T14:37:21.336213+0000","last_unstale":"2024-06-29T14:37:22.601492+0000","last_undegraded":"2024-06-29T14:37:22.601492+0000","last_fullsized":"2024-06-29T14:37:22.601492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:42:18.286112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.321050+0000","last_change":"2024-06-29T14:37:20.322453+0000","last_active":"2024-06-29T14:37:21.321050+0000","last_peered":"2024-06-29T14:37:21.321050+0000","last_clean":"2024-06-29T14:37:21.321050+0000","last_became_active":"2024-06-29T14:37:20.322147+0000","last_became_peered":"2024-06-29T14:37:20.322147+0000","last_unstale":"2024-06-29T14:37:21.321050+0000","last_undegraded":"2024-06-29T14:37:21.321050+0000","last_fullsized":"2024-06-29T14:37:21.321050+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:29.811184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341503+0000","last_change":"2024-06-29T14:37:21.347234+0000","last_active":"2024-06-29T14:37:22.341503+0000","last_peered":"2024-06-29T14:37:22.341503+0000","last_clean":"2024-06-29T14:37:22.341503+0000","last_became_active":"2024-06-29T14:37:21.346123+0000","last_became_peered":"2024-06-29T14:37:21.346123+0000","last_unstale":"2024-06-29T14:37:22.341503+0000","last_undegraded":"2024-06-29T14:37:22.341503+0000","last_fullsized":"2024-06-29T14:37:22.341503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:12.328298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.330392+0000","last_change":"2024-06-29T14:37:20.333964+0000","last_active":"2024-06-29T14:37:21.330392+0000","last_peered":"2024-06-29T14:37:21.330392+0000","last_clean":"2024-06-29T14:37:21.330392+0000","last_became_active":"2024-06-29T14:37:20.333661+0000","last_became_peered":"2024-06-29T14:37:20.333661+0000","last_unstale":"2024-06-29T14:37:21.330392+0000","last_undegraded":"2024-06-29T14:37:21.330392+0000","last_fullsized":"2024-06-29T14:37:21.330392+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:02:44.976401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.332440+0000","last_change":"2024-06-29T14:37:21.327530+0000","last_active":"2024-06-29T14:37:21.332440+0000","last_peered":"2024-06-29T14:37:21.332440+0000","last_clean":"2024-06-29T14:37:21.332440+0000","last_became_active":"2024-06-29T14:37:21.327184+0000","last_became_peered":"2024-06-29T14:37:21.327184+0000","last_unstale":"2024-06-29T14:37:21.332440+0000","last_undegraded":"2024-06-29T14:37:21.332440+0000","last_fullsized":"2024-06-29T14:37:21.332440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:43:33.783969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.321033+0000","last_change":"2024-06-29T14:37:20.326915+0000","last_active":"2024-06-29T14:37:21.321033+0000","last_peered":"2024-06-29T14:37:21.321033+0000","last_clean":"2024-06-29T14:37:21.321033+0000","last_became_active":"2024-06-29T14:37:20.326569+0000","last_became_peered":"2024-06-29T14:37:20.326569+0000","last_unstale":"2024-06-29T14:37:21.321033+0000","last_undegraded":"2024-06-29T14:37:21.321033+0000","last_fullsized":"2024-06-29T14:37:21.321033+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:13.658291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601397+0000","last_change":"2024-06-29T14:37:21.335794+0000","last_active":"2024-06-29T14:37:22.601397+0000","last_peered":"2024-06-29T14:37:22.601397+0000","last_clean":"2024-06-29T14:37:22.601397+0000","last_became_active":"2024-06-29T14:37:21.335461+0000","last_became_peered":"2024-06-29T14:37:21.335461+0000","last_unstale":"2024-06-29T14:37:22.601397+0000","last_undegraded":"2024-06-29T14:37:22.601397+0000","last_fullsized":"2024-06-29T14:37:22.601397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:17.833749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601442+0000","last_change":"2024-06-29T14:37:20.328161+0000","last_active":"2024-06-29T14:37:22.601442+0000","last_peered":"2024-06-29T14:37:22.601442+0000","last_clean":"2024-06-29T14:37:22.601442+0000","last_became_active":"2024-06-29T14:37:20.327853+0000","last_became_peered":"2024-06-29T14:37:20.327853+0000","last_unstale":"2024-06-29T14:37:22.601442+0000","last_undegraded":"2024-06-29T14:37:22.601442+0000","last_fullsized":"2024-06-29T14:37:22.601442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:58.959487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601358+0000","last_change":"2024-06-29T14:37:21.340956+0000","last_active":"2024-06-29T14:37:22.601358+0000","last_peered":"2024-06-29T14:37:22.601358+0000","last_clean":"2024-06-29T14:37:22.601358+0000","last_became_active":"2024-06-29T14:37:21.340681+0000","last_became_peered":"2024-06-29T14:37:21.340681+0000","last_unstale":"2024-06-29T14:37:22.601358+0000","last_undegraded":"2024-06-29T14:37:22.601358+0000","last_fullsized":"2024-06-29T14:37:22.601358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:31:15.861788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341726+0000","last_change":"2024-06-29T14:37:20.323931+0000","last_active":"2024-06-29T14:37:22.341726+0000","last_peered":"2024-06-29T14:37:22.341726+0000","last_clean":"2024-06-29T14:37:22.341726+0000","last_became_active":"2024-06-29T14:37:20.323595+0000","last_became_peered":"2024-06-29T14:37:20.323595+0000","last_unstale":"2024-06-29T14:37:22.341726+0000","last_undegraded":"2024-06-29T14:37:22.341726+0000","last_fullsized":"2024-06-29T14:37:22.341726+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:02:55.617173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"67'100","reported_seq":326,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341264+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:37:22.341264+0000","last_peered":"2024-06-29T14:37:22.341264+0000","last_clean":"2024-06-29T14:37:22.341264+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:37:22.341264+0000","last_undegraded":"2024-06-29T14:37:22.341264+0000","last_fullsized":"2024-06-29T14:37:22.341264+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1531904,"data_stored":1032782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575031,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":929792,"data_stored":442254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575029,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":442906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575030,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58472,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313368,"statfs":{"total":96636764160,"available":96576888832,"internally_reserved":0,"allocated":942080,"data_stored":443866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575030,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36260,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335580,"statfs":{"total":96636764160,"available":96599633920,"internally_reserved":0,"allocated":937984,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36276,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335564,"statfs":{"total":96636764160,"available":96599617536,"internally_reserved":0,"allocated":950272,"data_stored":448682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50148,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321692,"statfs":{"total":96636764160,"available":96585412608,"internally_reserved":0,"allocated":933888,"data_stored":442344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575030,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1540096,"data_stored":1034816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:37:25.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:37:25.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.508+0000 7f63cfce4640 1 -- 172.21.15.81:0/4172010561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80d0230 msgr2=0x7f63c80d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.508+0000 7f63cfce4640 1 --2- 172.21.15.81:0/4172010561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80d0230 0x7f63c80d0640 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f63b8005fd0 tx=0x7f63b802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.508+0000 7f63cfce4640 1 -- 172.21.15.81:0/4172010561 shutdown_connections 2024-06-29T14:37:25.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.508+0000 7f63cfce4640 1 --2- 172.21.15.81:0/4172010561 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c80b4f40 0x7f63c80b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.508+0000 7f63cfce4640 1 --2- 172.21.15.81:0/4172010561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80d0230 0x7f63c80d0640 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.508+0000 7f63cfce4640 1 -- 172.21.15.81:0/4172010561 >> 172.21.15.81:0/4172010561 conn(0x7f63c80b3680 msgr2=0x7f63c80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.508+0000 7f63cfce4640 1 -- 172.21.15.81:0/4172010561 shutdown_connections 2024-06-29T14:37:25.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.508+0000 7f63cfce4640 1 -- 172.21.15.81:0/4172010561 wait complete. 2024-06-29T14:37:25.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.510+0000 7f63cfce4640 1 Processor -- start 2024-06-29T14:37:25.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.510+0000 7f63cfce4640 1 -- start start 2024-06-29T14:37:25.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63cfce4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c80b3200 0x7f63c815bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63cfce4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c80b4f40 0x7f63c809ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63cfce4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80d0230 0x7f63c8070a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63cfce4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f63c80b5c20 con 0x7f63c80d0230 2024-06-29T14:37:25.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63cfce4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f63c80b5aa0 con 0x7f63c80b4f40 2024-06-29T14:37:25.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63cfce4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f63c80b5da0 con 0x7f63c80b3200 2024-06-29T14:37:25.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63ce281640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80d0230 0x7f63c8070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63ce281640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80d0230 0x7f63c8070a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41170/0 (socket says 172.21.15.81:41170) 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63ce281640 1 -- 172.21.15.81:0/3938115184 learned_addr learned my addr 172.21.15.81:0/3938115184 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63cd27f640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c80b4f40 0x7f63c809ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.511+0000 7f63cda80640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c80b3200 0x7f63c815bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63ce281640 1 -- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c80b3200 msgr2=0x7f63c815bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63ce281640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c80b3200 0x7f63c815bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63ce281640 1 -- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c80b4f40 msgr2=0x7f63c809ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63ce281640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c80b4f40 0x7f63c809ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63ce281640 1 -- 172.21.15.81:0/3938115184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c81f4df0 con 0x7f63c80d0230 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63ce281640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80d0230 0x7f63c8070a40 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f63c000d9a0 tx=0x7f63c000de60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63be7fc640 1 -- 172.21.15.81:0/3938115184 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0017070 con 0x7f63c80d0230 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63be7fc640 1 -- 172.21.15.81:0/3938115184 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63c000be00 con 0x7f63c80d0230 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63be7fc640 1 -- 172.21.15.81:0/3938115184 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c00106c0 con 0x7f63c80d0230 2024-06-29T14:37:25.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63cfce4640 1 -- 172.21.15.81:0/3938115184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c81f50b0 con 0x7f63c80d0230 2024-06-29T14:37:25.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.512+0000 7f63cfce4640 1 -- 172.21.15.81:0/3938115184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c81f5670 con 0x7f63c80d0230 2024-06-29T14:37:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.513+0000 7f63cfce4640 1 -- 172.21.15.81:0/3938115184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c80b5350 con 0x7f63c80d0230 2024-06-29T14:37:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.514+0000 7f63be7fc640 1 -- 172.21.15.81:0/3938115184 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f63c0010860 con 0x7f63c80d0230 2024-06-29T14:37:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.514+0000 7f63be7fc640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63ac0763f0 0x7f63ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.515+0000 7f63be7fc640 1 -- 172.21.15.81:0/3938115184 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7f63c0097fe0 con 0x7f63c80d0230 2024-06-29T14:37:25.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.515+0000 7f63cda80640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63ac0763f0 0x7f63ac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.515+0000 7f63cda80640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63ac0763f0 0x7f63ac078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63c4001720 tx=0x7f63c4009040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.519+0000 7f63be7fc640 1 -- 172.21.15.81:0/3938115184 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f63c0060ff0 con 0x7f63c80d0230 2024-06-29T14:37:25.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.695+0000 7f63cfce4640 1 -- 172.21.15.81:0/3938115184 --> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f63c81e1c30 con 0x7f63ac0763f0 2024-06-29T14:37:25.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.701+0000 7f63be7fc640 1 -- 172.21.15.81:0/3938115184 <== mgr.5860 v2:172.21.15.167:6832/3693496302 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251923 (secure 0 0 0) 0x7f63c81e1c30 con 0x7f63ac0763f0 2024-06-29T14:37:25.701 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:25.702 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:37:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 -- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63ac0763f0 msgr2=0x7f63ac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63ac0763f0 0x7f63ac078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63c4001720 tx=0x7f63c4009040 comp rx=0 tx=0).stop 2024-06-29T14:37:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 -- 172.21.15.81:0/3938115184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80d0230 msgr2=0x7f63c8070a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80d0230 0x7f63c8070a40 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f63c000d9a0 tx=0x7f63c000de60 comp rx=0 tx=0).stop 2024-06-29T14:37:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 -- 172.21.15.81:0/3938115184 shutdown_connections 2024-06-29T14:37:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f63ac0763f0 0x7f63ac078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63c80d0230 0x7f63c8070a40 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63c80b4f40 0x7f63c809ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 --2- 172.21.15.81:0/3938115184 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63c80b3200 0x7f63c815bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 -- 172.21.15.81:0/3938115184 >> 172.21.15.81:0/3938115184 conn(0x7f63c80b3680 msgr2=0x7f63c81e3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 -- 172.21.15.81:0/3938115184 shutdown_connections 2024-06-29T14:37:25.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.703+0000 7f63cfce4640 1 -- 172.21.15.81:0/3938115184 wait complete. 2024-06-29T14:37:25.713 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:37:24.927747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":194,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310728,"kb_used_data":8500,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754663992,"statfs":{"total":773094113280,"available":772775927808,"internally_reserved":0,"allocated":8704000,"data_stored":4729960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":309317170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.618450"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337424+0000","last_change":"2024-06-29T14:37:21.339050+0000","last_active":"2024-06-29T14:37:22.337424+0000","last_peered":"2024-06-29T14:37:22.337424+0000","last_clean":"2024-06-29T14:37:22.337424+0000","last_became_active":"2024-06-29T14:37:21.338754+0000","last_became_peered":"2024-06-29T14:37:21.338754+0000","last_unstale":"2024-06-29T14:37:22.337424+0000","last_undegraded":"2024-06-29T14:37:22.337424+0000","last_fullsized":"2024-06-29T14:37:22.337424+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:14.663303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601638+0000","last_change":"2024-06-29T14:37:21.335806+0000","last_active":"2024-06-29T14:37:22.601638+0000","last_peered":"2024-06-29T14:37:22.601638+0000","last_clean":"2024-06-29T14:37:22.601638+0000","last_became_active":"2024-06-29T14:37:21.335501+0000","last_became_peered":"2024-06-29T14:37:21.335501+0000","last_unstale":"2024-06-29T14:37:22.601638+0000","last_undegraded":"2024-06-29T14:37:22.601638+0000","last_fullsized":"2024-06-29T14:37:22.601638+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:48.114271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341610+0000","last_change":"2024-06-29T14:37:21.345040+0000","last_active":"2024-06-29T14:37:22.341610+0000","last_peered":"2024-06-29T14:37:22.341610+0000","last_clean":"2024-06-29T14:37:22.341610+0000","last_became_active":"2024-06-29T14:37:21.344620+0000","last_became_peered":"2024-06-29T14:37:21.344620+0000","last_unstale":"2024-06-29T14:37:22.341610+0000","last_undegraded":"2024-06-29T14:37:22.341610+0000","last_fullsized":"2024-06-29T14:37:22.341610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:56.796698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.337489+0000","last_change":"2024-06-29T14:37:21.330537+0000","last_active":"2024-06-29T14:37:21.337489+0000","last_peered":"2024-06-29T14:37:21.337489+0000","last_clean":"2024-06-29T14:37:21.337489+0000","last_became_active":"2024-06-29T14:37:21.330184+0000","last_became_peered":"2024-06-29T14:37:21.330184+0000","last_unstale":"2024-06-29T14:37:21.337489+0000","last_undegraded":"2024-06-29T14:37:21.337489+0000","last_fullsized":"2024-06-29T14:37:21.337489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:28.461310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337484+0000","last_change":"2024-06-29T14:37:21.339553+0000","last_active":"2024-06-29T14:37:22.337484+0000","last_peered":"2024-06-29T14:37:22.337484+0000","last_clean":"2024-06-29T14:37:22.337484+0000","last_became_active":"2024-06-29T14:37:21.339247+0000","last_became_peered":"2024-06-29T14:37:21.339247+0000","last_unstale":"2024-06-29T14:37:22.337484+0000","last_undegraded":"2024-06-29T14:37:22.337484+0000","last_fullsized":"2024-06-29T14:37:22.337484+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:36.736621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.350868+0000","last_change":"2024-06-29T14:37:21.345589+0000","last_active":"2024-06-29T14:37:21.350868+0000","last_peered":"2024-06-29T14:37:21.350868+0000","last_clean":"2024-06-29T14:37:21.350868+0000","last_became_active":"2024-06-29T14:37:21.345292+0000","last_became_peered":"2024-06-29T14:37:21.345292+0000","last_unstale":"2024-06-29T14:37:21.350868+0000","last_undegraded":"2024-06-29T14:37:21.350868+0000","last_fullsized":"2024-06-29T14:37:21.350868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:22:54.491723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.343626+0000","last_change":"2024-06-29T14:37:21.342997+0000","last_active":"2024-06-29T14:37:21.343626+0000","last_peered":"2024-06-29T14:37:21.343626+0000","last_clean":"2024-06-29T14:37:21.343626+0000","last_became_active":"2024-06-29T14:37:21.342480+0000","last_became_peered":"2024-06-29T14:37:21.342480+0000","last_unstale":"2024-06-29T14:37:21.343626+0000","last_undegraded":"2024-06-29T14:37:21.343626+0000","last_fullsized":"2024-06-29T14:37:21.343626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:13.066212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601254+0000","last_change":"2024-06-29T14:37:21.342958+0000","last_active":"2024-06-29T14:37:22.601254+0000","last_peered":"2024-06-29T14:37:22.601254+0000","last_clean":"2024-06-29T14:37:22.601254+0000","last_became_active":"2024-06-29T14:37:21.342649+0000","last_became_peered":"2024-06-29T14:37:21.342649+0000","last_unstale":"2024-06-29T14:37:22.601254+0000","last_undegraded":"2024-06-29T14:37:22.601254+0000","last_fullsized":"2024-06-29T14:37:22.601254+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:26.622470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.348747+0000","last_change":"2024-06-29T14:37:21.346904+0000","last_active":"2024-06-29T14:37:21.348747+0000","last_peered":"2024-06-29T14:37:21.348747+0000","last_clean":"2024-06-29T14:37:21.348747+0000","last_became_active":"2024-06-29T14:37:21.346479+0000","last_became_peered":"2024-06-29T14:37:21.346479+0000","last_unstale":"2024-06-29T14:37:21.348747+0000","last_undegraded":"2024-06-29T14:37:21.348747+0000","last_fullsized":"2024-06-29T14:37:21.348747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:07.258003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.327035+0000","last_change":"2024-06-29T14:37:21.325952+0000","last_active":"2024-06-29T14:37:21.327035+0000","last_peered":"2024-06-29T14:37:21.327035+0000","last_clean":"2024-06-29T14:37:21.327035+0000","last_became_active":"2024-06-29T14:37:21.325631+0000","last_became_peered":"2024-06-29T14:37:21.325631+0000","last_unstale":"2024-06-29T14:37:21.327035+0000","last_undegraded":"2024-06-29T14:37:21.327035+0000","last_fullsized":"2024-06-29T14:37:21.327035+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:14.460881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.339263+0000","last_change":"2024-06-29T14:37:21.338195+0000","last_active":"2024-06-29T14:37:21.339263+0000","last_peered":"2024-06-29T14:37:21.339263+0000","last_clean":"2024-06-29T14:37:21.339263+0000","last_became_active":"2024-06-29T14:37:21.337893+0000","last_became_peered":"2024-06-29T14:37:21.337893+0000","last_unstale":"2024-06-29T14:37:21.339263+0000","last_undegraded":"2024-06-29T14:37:21.339263+0000","last_fullsized":"2024-06-29T14:37:21.339263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:14.361683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.341155+0000","last_change":"2024-06-29T14:37:21.337735+0000","last_active":"2024-06-29T14:37:21.341155+0000","last_peered":"2024-06-29T14:37:21.341155+0000","last_clean":"2024-06-29T14:37:21.341155+0000","last_became_active":"2024-06-29T14:37:21.337305+0000","last_became_peered":"2024-06-29T14:37:21.337305+0000","last_unstale":"2024-06-29T14:37:21.341155+0000","last_undegraded":"2024-06-29T14:37:21.341155+0000","last_fullsized":"2024-06-29T14:37:21.341155+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:17.356643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341402+0000","last_change":"2024-06-29T14:37:21.338020+0000","last_active":"2024-06-29T14:37:22.341402+0000","last_peered":"2024-06-29T14:37:22.341402+0000","last_clean":"2024-06-29T14:37:22.341402+0000","last_became_active":"2024-06-29T14:37:21.337700+0000","last_became_peered":"2024-06-29T14:37:21.337700+0000","last_unstale":"2024-06-29T14:37:22.341402+0000","last_undegraded":"2024-06-29T14:37:22.341402+0000","last_fullsized":"2024-06-29T14:37:22.341402+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:03.847518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347056+0000","last_change":"2024-06-29T14:37:21.344057+0000","last_active":"2024-06-29T14:37:21.347056+0000","last_peered":"2024-06-29T14:37:21.347056+0000","last_clean":"2024-06-29T14:37:21.347056+0000","last_became_active":"2024-06-29T14:37:21.343389+0000","last_became_peered":"2024-06-29T14:37:21.343389+0000","last_unstale":"2024-06-29T14:37:21.347056+0000","last_undegraded":"2024-06-29T14:37:21.347056+0000","last_fullsized":"2024-06-29T14:37:21.347056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:08:47.167252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.343159+0000","last_change":"2024-06-29T14:37:21.341095+0000","last_active":"2024-06-29T14:37:21.343159+0000","last_peered":"2024-06-29T14:37:21.343159+0000","last_clean":"2024-06-29T14:37:21.343159+0000","last_became_active":"2024-06-29T14:37:21.340267+0000","last_became_peered":"2024-06-29T14:37:21.340267+0000","last_unstale":"2024-06-29T14:37:21.343159+0000","last_undegraded":"2024-06-29T14:37:21.343159+0000","last_fullsized":"2024-06-29T14:37:21.343159+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:24.879428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.342656+0000","last_change":"2024-06-29T14:37:21.336893+0000","last_active":"2024-06-29T14:37:21.342656+0000","last_peered":"2024-06-29T14:37:21.342656+0000","last_clean":"2024-06-29T14:37:21.342656+0000","last_became_active":"2024-06-29T14:37:21.336025+0000","last_became_peered":"2024-06-29T14:37:21.336025+0000","last_unstale":"2024-06-29T14:37:21.342656+0000","last_undegraded":"2024-06-29T14:37:21.342656+0000","last_fullsized":"2024-06-29T14:37:21.342656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:36.736621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601706+0000","last_change":"2024-06-29T14:37:21.340901+0000","last_active":"2024-06-29T14:37:22.601706+0000","last_peered":"2024-06-29T14:37:22.601706+0000","last_clean":"2024-06-29T14:37:22.601706+0000","last_became_active":"2024-06-29T14:37:21.336428+0000","last_became_peered":"2024-06-29T14:37:21.336428+0000","last_unstale":"2024-06-29T14:37:22.601706+0000","last_undegraded":"2024-06-29T14:37:22.601706+0000","last_fullsized":"2024-06-29T14:37:22.601706+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:49:54.532144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.342355+0000","last_change":"2024-06-29T14:37:21.344574+0000","last_active":"2024-06-29T14:37:22.342355+0000","last_peered":"2024-06-29T14:37:22.342355+0000","last_clean":"2024-06-29T14:37:22.342355+0000","last_became_active":"2024-06-29T14:37:21.344265+0000","last_became_peered":"2024-06-29T14:37:21.344265+0000","last_unstale":"2024-06-29T14:37:22.342355+0000","last_undegraded":"2024-06-29T14:37:22.342355+0000","last_fullsized":"2024-06-29T14:37:22.342355+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:52.721920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337431+0000","last_change":"2024-06-29T14:37:21.337973+0000","last_active":"2024-06-29T14:37:22.337431+0000","last_peered":"2024-06-29T14:37:22.337431+0000","last_clean":"2024-06-29T14:37:22.337431+0000","last_became_active":"2024-06-29T14:37:21.337569+0000","last_became_peered":"2024-06-29T14:37:21.337569+0000","last_unstale":"2024-06-29T14:37:22.337431+0000","last_undegraded":"2024-06-29T14:37:22.337431+0000","last_fullsized":"2024-06-29T14:37:22.337431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:30.816196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601562+0000","last_change":"2024-06-29T14:37:21.341065+0000","last_active":"2024-06-29T14:37:22.601562+0000","last_peered":"2024-06-29T14:37:22.601562+0000","last_clean":"2024-06-29T14:37:22.601562+0000","last_became_active":"2024-06-29T14:37:21.336714+0000","last_became_peered":"2024-06-29T14:37:21.336714+0000","last_unstale":"2024-06-29T14:37:22.601562+0000","last_undegraded":"2024-06-29T14:37:22.601562+0000","last_fullsized":"2024-06-29T14:37:22.601562+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:09.757075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341287+0000","last_change":"2024-06-29T14:37:21.346405+0000","last_active":"2024-06-29T14:37:22.341287+0000","last_peered":"2024-06-29T14:37:22.341287+0000","last_clean":"2024-06-29T14:37:22.341287+0000","last_became_active":"2024-06-29T14:37:21.346086+0000","last_became_peered":"2024-06-29T14:37:21.346086+0000","last_unstale":"2024-06-29T14:37:22.341287+0000","last_undegraded":"2024-06-29T14:37:22.341287+0000","last_fullsized":"2024-06-29T14:37:22.341287+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:27:14.903405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601276+0000","last_change":"2024-06-29T14:37:21.336857+0000","last_active":"2024-06-29T14:37:22.601276+0000","last_peered":"2024-06-29T14:37:22.601276+0000","last_clean":"2024-06-29T14:37:22.601276+0000","last_became_active":"2024-06-29T14:37:21.336257+0000","last_became_peered":"2024-06-29T14:37:21.336257+0000","last_unstale":"2024-06-29T14:37:22.601276+0000","last_undegraded":"2024-06-29T14:37:22.601276+0000","last_fullsized":"2024-06-29T14:37:22.601276+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:39:40.381701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.340982+0000","last_change":"2024-06-29T14:37:21.339712+0000","last_active":"2024-06-29T14:37:22.340982+0000","last_peered":"2024-06-29T14:37:22.340982+0000","last_clean":"2024-06-29T14:37:22.340982+0000","last_became_active":"2024-06-29T14:37:21.339354+0000","last_became_peered":"2024-06-29T14:37:21.339354+0000","last_unstale":"2024-06-29T14:37:22.340982+0000","last_undegraded":"2024-06-29T14:37:22.340982+0000","last_fullsized":"2024-06-29T14:37:22.340982+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:26:01.150918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.340811+0000","last_change":"2024-06-29T14:37:21.341879+0000","last_active":"2024-06-29T14:37:22.340811+0000","last_peered":"2024-06-29T14:37:22.340811+0000","last_clean":"2024-06-29T14:37:22.340811+0000","last_became_active":"2024-06-29T14:37:21.341542+0000","last_became_peered":"2024-06-29T14:37:21.341542+0000","last_unstale":"2024-06-29T14:37:22.340811+0000","last_undegraded":"2024-06-29T14:37:22.340811+0000","last_fullsized":"2024-06-29T14:37:22.340811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:57.946769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337586+0000","last_change":"2024-06-29T14:37:21.328524+0000","last_active":"2024-06-29T14:37:22.337586+0000","last_peered":"2024-06-29T14:37:22.337586+0000","last_clean":"2024-06-29T14:37:22.337586+0000","last_became_active":"2024-06-29T14:37:21.328200+0000","last_became_peered":"2024-06-29T14:37:21.328200+0000","last_unstale":"2024-06-29T14:37:22.337586+0000","last_undegraded":"2024-06-29T14:37:22.337586+0000","last_fullsized":"2024-06-29T14:37:22.337586+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:14.460881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.344778+0000","last_change":"2024-06-29T14:37:21.341751+0000","last_active":"2024-06-29T14:37:21.344778+0000","last_peered":"2024-06-29T14:37:21.344778+0000","last_clean":"2024-06-29T14:37:21.344778+0000","last_became_active":"2024-06-29T14:37:21.341413+0000","last_became_peered":"2024-06-29T14:37:21.341413+0000","last_unstale":"2024-06-29T14:37:21.344778+0000","last_undegraded":"2024-06-29T14:37:21.344778+0000","last_fullsized":"2024-06-29T14:37:21.344778+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:52.816582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347933+0000","last_change":"2024-06-29T14:37:21.342502+0000","last_active":"2024-06-29T14:37:21.347933+0000","last_peered":"2024-06-29T14:37:21.347933+0000","last_clean":"2024-06-29T14:37:21.347933+0000","last_became_active":"2024-06-29T14:37:21.341363+0000","last_became_peered":"2024-06-29T14:37:21.341363+0000","last_unstale":"2024-06-29T14:37:21.347933+0000","last_undegraded":"2024-06-29T14:37:21.347933+0000","last_fullsized":"2024-06-29T14:37:21.347933+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:54.900769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337669+0000","last_change":"2024-06-29T14:37:21.340399+0000","last_active":"2024-06-29T14:37:22.337669+0000","last_peered":"2024-06-29T14:37:22.337669+0000","last_clean":"2024-06-29T14:37:22.337669+0000","last_became_active":"2024-06-29T14:37:21.340111+0000","last_became_peered":"2024-06-29T14:37:21.340111+0000","last_unstale":"2024-06-29T14:37:22.337669+0000","last_undegraded":"2024-06-29T14:37:22.337669+0000","last_fullsized":"2024-06-29T14:37:22.337669+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:43:33.783969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601643+0000","last_change":"2024-06-29T14:37:21.335288+0000","last_active":"2024-06-29T14:37:22.601643+0000","last_peered":"2024-06-29T14:37:22.601643+0000","last_clean":"2024-06-29T14:37:22.601643+0000","last_became_active":"2024-06-29T14:37:21.334896+0000","last_became_peered":"2024-06-29T14:37:21.334896+0000","last_unstale":"2024-06-29T14:37:22.601643+0000","last_undegraded":"2024-06-29T14:37:22.601643+0000","last_fullsized":"2024-06-29T14:37:22.601643+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:13:47.742605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341196+0000","last_change":"2024-06-29T14:37:21.345910+0000","last_active":"2024-06-29T14:37:22.341196+0000","last_peered":"2024-06-29T14:37:22.341196+0000","last_clean":"2024-06-29T14:37:22.341196+0000","last_became_active":"2024-06-29T14:37:21.345327+0000","last_became_peered":"2024-06-29T14:37:21.345327+0000","last_unstale":"2024-06-29T14:37:22.341196+0000","last_undegraded":"2024-06-29T14:37:22.341196+0000","last_fullsized":"2024-06-29T14:37:22.341196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:01.286008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.337288+0000","last_change":"2024-06-29T14:37:21.329735+0000","last_active":"2024-06-29T14:37:22.337288+0000","last_peered":"2024-06-29T14:37:22.337288+0000","last_clean":"2024-06-29T14:37:22.337288+0000","last_became_active":"2024-06-29T14:37:21.329423+0000","last_became_peered":"2024-06-29T14:37:21.329423+0000","last_unstale":"2024-06-29T14:37:22.337288+0000","last_undegraded":"2024-06-29T14:37:22.337288+0000","last_fullsized":"2024-06-29T14:37:22.337288+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:51.009669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.342677+0000","last_change":"2024-06-29T14:37:21.338099+0000","last_active":"2024-06-29T14:37:21.342677+0000","last_peered":"2024-06-29T14:37:21.342677+0000","last_clean":"2024-06-29T14:37:21.342677+0000","last_became_active":"2024-06-29T14:37:21.337248+0000","last_became_peered":"2024-06-29T14:37:21.337248+0000","last_unstale":"2024-06-29T14:37:21.342677+0000","last_undegraded":"2024-06-29T14:37:21.342677+0000","last_fullsized":"2024-06-29T14:37:21.342677+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:43:33.783969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347135+0000","last_change":"2024-06-29T14:37:21.343236+0000","last_active":"2024-06-29T14:37:21.347135+0000","last_peered":"2024-06-29T14:37:21.347135+0000","last_clean":"2024-06-29T14:37:21.347135+0000","last_became_active":"2024-06-29T14:37:21.342939+0000","last_became_peered":"2024-06-29T14:37:21.342939+0000","last_unstale":"2024-06-29T14:37:21.347135+0000","last_undegraded":"2024-06-29T14:37:21.347135+0000","last_fullsized":"2024-06-29T14:37:21.347135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:59.527370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"77'4","reported_seq":16,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.353920+0000","last_change":"2024-06-29T14:37:20.332839+0000","last_active":"2024-06-29T14:37:21.353920+0000","last_peered":"2024-06-29T14:37:21.353920+0000","last_clean":"2024-06-29T14:37:21.353920+0000","last_became_active":"2024-06-29T14:37:20.332510+0000","last_became_peered":"2024-06-29T14:37:20.332510+0000","last_unstale":"2024-06-29T14:37:21.353920+0000","last_undegraded":"2024-06-29T14:37:21.353920+0000","last_fullsized":"2024-06-29T14:37:21.353920+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:43:42.846682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.346727+0000","last_change":"2024-06-29T14:37:21.342890+0000","last_active":"2024-06-29T14:37:21.346727+0000","last_peered":"2024-06-29T14:37:21.346727+0000","last_clean":"2024-06-29T14:37:21.346727+0000","last_became_active":"2024-06-29T14:37:21.342595+0000","last_became_peered":"2024-06-29T14:37:21.342595+0000","last_unstale":"2024-06-29T14:37:21.346727+0000","last_undegraded":"2024-06-29T14:37:21.346727+0000","last_fullsized":"2024-06-29T14:37:21.346727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:54.900769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601671+0000","last_change":"2024-06-29T14:37:20.332603+0000","last_active":"2024-06-29T14:37:22.601671+0000","last_peered":"2024-06-29T14:37:22.601671+0000","last_clean":"2024-06-29T14:37:22.601671+0000","last_became_active":"2024-06-29T14:37:20.330978+0000","last_became_peered":"2024-06-29T14:37:20.330978+0000","last_unstale":"2024-06-29T14:37:22.601671+0000","last_undegraded":"2024-06-29T14:37:22.601671+0000","last_fullsized":"2024-06-29T14:37:22.601671+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:18:44.692648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.341808+0000","last_change":"2024-06-29T14:37:21.340205+0000","last_active":"2024-06-29T14:37:21.341808+0000","last_peered":"2024-06-29T14:37:21.341808+0000","last_clean":"2024-06-29T14:37:21.341808+0000","last_became_active":"2024-06-29T14:37:21.339261+0000","last_became_peered":"2024-06-29T14:37:21.339261+0000","last_unstale":"2024-06-29T14:37:21.341808+0000","last_undegraded":"2024-06-29T14:37:21.341808+0000","last_fullsized":"2024-06-29T14:37:21.341808+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:34.472101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.352347+0000","last_change":"2024-06-29T14:37:20.333105+0000","last_active":"2024-06-29T14:37:21.352347+0000","last_peered":"2024-06-29T14:37:21.352347+0000","last_clean":"2024-06-29T14:37:21.352347+0000","last_became_active":"2024-06-29T14:37:20.331980+0000","last_became_peered":"2024-06-29T14:37:20.331980+0000","last_unstale":"2024-06-29T14:37:21.352347+0000","last_undegraded":"2024-06-29T14:37:21.352347+0000","last_fullsized":"2024-06-29T14:37:21.352347+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:13.658291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.342878+0000","last_change":"2024-06-29T14:37:21.338556+0000","last_active":"2024-06-29T14:37:21.342878+0000","last_peered":"2024-06-29T14:37:21.342878+0000","last_clean":"2024-06-29T14:37:21.342878+0000","last_became_active":"2024-06-29T14:37:21.338264+0000","last_became_peered":"2024-06-29T14:37:21.338264+0000","last_unstale":"2024-06-29T14:37:21.342878+0000","last_undegraded":"2024-06-29T14:37:21.342878+0000","last_fullsized":"2024-06-29T14:37:21.342878+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:14.361683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.353205+0000","last_change":"2024-06-29T14:37:20.328019+0000","last_active":"2024-06-29T14:37:21.353205+0000","last_peered":"2024-06-29T14:37:21.353205+0000","last_clean":"2024-06-29T14:37:21.353205+0000","last_became_active":"2024-06-29T14:37:20.327688+0000","last_became_peered":"2024-06-29T14:37:20.327688+0000","last_unstale":"2024-06-29T14:37:21.353205+0000","last_undegraded":"2024-06-29T14:37:21.353205+0000","last_fullsized":"2024-06-29T14:37:21.353205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:44:58.968350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601309+0000","last_change":"2024-06-29T14:37:21.337144+0000","last_active":"2024-06-29T14:37:22.601309+0000","last_peered":"2024-06-29T14:37:22.601309+0000","last_clean":"2024-06-29T14:37:22.601309+0000","last_became_active":"2024-06-29T14:37:21.336538+0000","last_became_peered":"2024-06-29T14:37:21.336538+0000","last_unstale":"2024-06-29T14:37:22.601309+0000","last_undegraded":"2024-06-29T14:37:22.601309+0000","last_fullsized":"2024-06-29T14:37:22.601309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:40:50.731137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.348705+0000","last_change":"2024-06-29T14:37:20.327124+0000","last_active":"2024-06-29T14:37:21.348705+0000","last_peered":"2024-06-29T14:37:21.348705+0000","last_clean":"2024-06-29T14:37:21.348705+0000","last_became_active":"2024-06-29T14:37:20.326807+0000","last_became_peered":"2024-06-29T14:37:20.326807+0000","last_unstale":"2024-06-29T14:37:21.348705+0000","last_undegraded":"2024-06-29T14:37:21.348705+0000","last_fullsized":"2024-06-29T14:37:21.348705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:02:22.616858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601585+0000","last_change":"2024-06-29T14:37:21.338958+0000","last_active":"2024-06-29T14:37:22.601585+0000","last_peered":"2024-06-29T14:37:22.601585+0000","last_clean":"2024-06-29T14:37:22.601585+0000","last_became_active":"2024-06-29T14:37:21.338607+0000","last_became_peered":"2024-06-29T14:37:21.338607+0000","last_unstale":"2024-06-29T14:37:22.601585+0000","last_undegraded":"2024-06-29T14:37:22.601585+0000","last_fullsized":"2024-06-29T14:37:22.601585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:51.451894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.352161+0000","last_change":"2024-06-29T14:37:20.336366+0000","last_active":"2024-06-29T14:37:21.352161+0000","last_peered":"2024-06-29T14:37:21.352161+0000","last_clean":"2024-06-29T14:37:21.352161+0000","last_became_active":"2024-06-29T14:37:20.335996+0000","last_became_peered":"2024-06-29T14:37:20.335996+0000","last_unstale":"2024-06-29T14:37:21.352161+0000","last_undegraded":"2024-06-29T14:37:21.352161+0000","last_fullsized":"2024-06-29T14:37:21.352161+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:50.004657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.340866+0000","last_change":"2024-06-29T14:37:21.335781+0000","last_active":"2024-06-29T14:37:22.340866+0000","last_peered":"2024-06-29T14:37:22.340866+0000","last_clean":"2024-06-29T14:37:22.340866+0000","last_became_active":"2024-06-29T14:37:21.335395+0000","last_became_peered":"2024-06-29T14:37:21.335395+0000","last_unstale":"2024-06-29T14:37:22.340866+0000","last_undegraded":"2024-06-29T14:37:22.340866+0000","last_fullsized":"2024-06-29T14:37:22.340866+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:52:00.907845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.324089+0000","last_change":"2024-06-29T14:37:20.324816+0000","last_active":"2024-06-29T14:37:21.324089+0000","last_peered":"2024-06-29T14:37:21.324089+0000","last_clean":"2024-06-29T14:37:21.324089+0000","last_became_active":"2024-06-29T14:37:20.324298+0000","last_became_peered":"2024-06-29T14:37:20.324298+0000","last_unstale":"2024-06-29T14:37:21.324089+0000","last_undegraded":"2024-06-29T14:37:21.324089+0000","last_fullsized":"2024-06-29T14:37:21.324089+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:07.296498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601333+0000","last_change":"2024-06-29T14:37:21.337674+0000","last_active":"2024-06-29T14:37:22.601333+0000","last_peered":"2024-06-29T14:37:22.601333+0000","last_clean":"2024-06-29T14:37:22.601333+0000","last_became_active":"2024-06-29T14:37:21.335834+0000","last_became_peered":"2024-06-29T14:37:21.335834+0000","last_unstale":"2024-06-29T14:37:22.601333+0000","last_undegraded":"2024-06-29T14:37:22.601333+0000","last_fullsized":"2024-06-29T14:37:22.601333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:27:15.828871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.330690+0000","last_change":"2024-06-29T14:37:20.328434+0000","last_active":"2024-06-29T14:37:21.330690+0000","last_peered":"2024-06-29T14:37:21.330690+0000","last_clean":"2024-06-29T14:37:21.330690+0000","last_became_active":"2024-06-29T14:37:20.327785+0000","last_became_peered":"2024-06-29T14:37:20.327785+0000","last_unstale":"2024-06-29T14:37:21.330690+0000","last_undegraded":"2024-06-29T14:37:21.330690+0000","last_fullsized":"2024-06-29T14:37:21.330690+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:43.272884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.351162+0000","last_change":"2024-06-29T14:37:21.349641+0000","last_active":"2024-06-29T14:37:21.351162+0000","last_peered":"2024-06-29T14:37:21.351162+0000","last_clean":"2024-06-29T14:37:21.351162+0000","last_became_active":"2024-06-29T14:37:21.349316+0000","last_became_peered":"2024-06-29T14:37:21.349316+0000","last_unstale":"2024-06-29T14:37:21.351162+0000","last_undegraded":"2024-06-29T14:37:21.351162+0000","last_fullsized":"2024-06-29T14:37:21.351162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:38.951897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.342105+0000","last_change":"2024-06-29T14:37:20.328496+0000","last_active":"2024-06-29T14:37:22.342105+0000","last_peered":"2024-06-29T14:37:22.342105+0000","last_clean":"2024-06-29T14:37:22.342105+0000","last_became_active":"2024-06-29T14:37:20.328145+0000","last_became_peered":"2024-06-29T14:37:20.328145+0000","last_unstale":"2024-06-29T14:37:22.342105+0000","last_undegraded":"2024-06-29T14:37:22.342105+0000","last_fullsized":"2024-06-29T14:37:22.342105+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:21.758521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341994+0000","last_change":"2024-06-29T14:37:21.350645+0000","last_active":"2024-06-29T14:37:22.341994+0000","last_peered":"2024-06-29T14:37:22.341994+0000","last_clean":"2024-06-29T14:37:22.341994+0000","last_became_active":"2024-06-29T14:37:21.349911+0000","last_became_peered":"2024-06-29T14:37:21.349911+0000","last_unstale":"2024-06-29T14:37:22.341994+0000","last_undegraded":"2024-06-29T14:37:22.341994+0000","last_fullsized":"2024-06-29T14:37:22.341994+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:29.306968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601731+0000","last_change":"2024-06-29T14:37:20.336008+0000","last_active":"2024-06-29T14:37:22.601731+0000","last_peered":"2024-06-29T14:37:22.601731+0000","last_clean":"2024-06-29T14:37:22.601731+0000","last_became_active":"2024-06-29T14:37:20.335655+0000","last_became_peered":"2024-06-29T14:37:20.335655+0000","last_unstale":"2024-06-29T14:37:22.601731+0000","last_undegraded":"2024-06-29T14:37:22.601731+0000","last_fullsized":"2024-06-29T14:37:22.601731+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:10:58.466853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347539+0000","last_change":"2024-06-29T14:37:21.344101+0000","last_active":"2024-06-29T14:37:21.347539+0000","last_peered":"2024-06-29T14:37:21.347539+0000","last_clean":"2024-06-29T14:37:21.347539+0000","last_became_active":"2024-06-29T14:37:21.343805+0000","last_became_peered":"2024-06-29T14:37:21.343805+0000","last_unstale":"2024-06-29T14:37:21.347539+0000","last_undegraded":"2024-06-29T14:37:21.347539+0000","last_fullsized":"2024-06-29T14:37:21.347539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:10:42.782502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"77'3","reported_seq":17,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341864+0000","last_change":"2024-06-29T14:37:20.331719+0000","last_active":"2024-06-29T14:37:22.341864+0000","last_peered":"2024-06-29T14:37:22.341864+0000","last_clean":"2024-06-29T14:37:22.341864+0000","last_became_active":"2024-06-29T14:37:20.331434+0000","last_became_peered":"2024-06-29T14:37:20.331434+0000","last_unstale":"2024-06-29T14:37:22.341864+0000","last_undegraded":"2024-06-29T14:37:22.341864+0000","last_fullsized":"2024-06-29T14:37:22.341864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:00:13.196568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341737+0000","last_change":"2024-06-29T14:37:21.349455+0000","last_active":"2024-06-29T14:37:22.341737+0000","last_peered":"2024-06-29T14:37:22.341737+0000","last_clean":"2024-06-29T14:37:22.341737+0000","last_became_active":"2024-06-29T14:37:21.349137+0000","last_became_peered":"2024-06-29T14:37:21.349137+0000","last_unstale":"2024-06-29T14:37:22.341737+0000","last_undegraded":"2024-06-29T14:37:22.341737+0000","last_fullsized":"2024-06-29T14:37:22.341737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:40:22.699838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"77'6","reported_seq":18,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.355736+0000","last_change":"2024-06-29T14:37:20.333936+0000","last_active":"2024-06-29T14:37:21.355736+0000","last_peered":"2024-06-29T14:37:21.355736+0000","last_clean":"2024-06-29T14:37:21.355736+0000","last_became_active":"2024-06-29T14:37:20.333638+0000","last_became_peered":"2024-06-29T14:37:20.333638+0000","last_unstale":"2024-06-29T14:37:21.355736+0000","last_undegraded":"2024-06-29T14:37:21.355736+0000","last_fullsized":"2024-06-29T14:37:21.355736+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:52:28.519994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.337294+0000","last_change":"2024-06-29T14:37:21.332515+0000","last_active":"2024-06-29T14:37:21.337294+0000","last_peered":"2024-06-29T14:37:21.337294+0000","last_clean":"2024-06-29T14:37:21.337294+0000","last_became_active":"2024-06-29T14:37:21.332214+0000","last_became_peered":"2024-06-29T14:37:21.332214+0000","last_unstale":"2024-06-29T14:37:21.337294+0000","last_undegraded":"2024-06-29T14:37:21.337294+0000","last_fullsized":"2024-06-29T14:37:21.337294+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:02.199586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601461+0000","last_change":"2024-06-29T14:37:20.332067+0000","last_active":"2024-06-29T14:37:22.601461+0000","last_peered":"2024-06-29T14:37:22.601461+0000","last_clean":"2024-06-29T14:37:22.601461+0000","last_became_active":"2024-06-29T14:37:20.330978+0000","last_became_peered":"2024-06-29T14:37:20.330978+0000","last_unstale":"2024-06-29T14:37:22.601461+0000","last_undegraded":"2024-06-29T14:37:22.601461+0000","last_fullsized":"2024-06-29T14:37:22.601461+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:08:23.406581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341429+0000","last_change":"2024-06-29T14:37:21.339874+0000","last_active":"2024-06-29T14:37:22.341429+0000","last_peered":"2024-06-29T14:37:22.341429+0000","last_clean":"2024-06-29T14:37:22.341429+0000","last_became_active":"2024-06-29T14:37:21.339538+0000","last_became_peered":"2024-06-29T14:37:21.339538+0000","last_unstale":"2024-06-29T14:37:22.341429+0000","last_undegraded":"2024-06-29T14:37:22.341429+0000","last_fullsized":"2024-06-29T14:37:22.341429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:44:53.221807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341621+0000","last_change":"2024-06-29T14:37:20.327726+0000","last_active":"2024-06-29T14:37:22.341621+0000","last_peered":"2024-06-29T14:37:22.341621+0000","last_clean":"2024-06-29T14:37:22.341621+0000","last_became_active":"2024-06-29T14:37:20.327419+0000","last_became_peered":"2024-06-29T14:37:20.327419+0000","last_unstale":"2024-06-29T14:37:22.341621+0000","last_undegraded":"2024-06-29T14:37:22.341621+0000","last_fullsized":"2024-06-29T14:37:22.341621+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:52.133200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.353334+0000","last_change":"2024-06-29T14:37:21.352035+0000","last_active":"2024-06-29T14:37:21.353334+0000","last_peered":"2024-06-29T14:37:21.353334+0000","last_clean":"2024-06-29T14:37:21.353334+0000","last_became_active":"2024-06-29T14:37:21.351683+0000","last_became_peered":"2024-06-29T14:37:21.351683+0000","last_unstale":"2024-06-29T14:37:21.353334+0000","last_undegraded":"2024-06-29T14:37:21.353334+0000","last_fullsized":"2024-06-29T14:37:21.353334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:07.732295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.324673+0000","last_change":"2024-06-29T14:37:20.330644+0000","last_active":"2024-06-29T14:37:21.324673+0000","last_peered":"2024-06-29T14:37:21.324673+0000","last_clean":"2024-06-29T14:37:21.324673+0000","last_became_active":"2024-06-29T14:37:20.329982+0000","last_became_peered":"2024-06-29T14:37:20.329982+0000","last_unstale":"2024-06-29T14:37:21.324673+0000","last_undegraded":"2024-06-29T14:37:21.324673+0000","last_fullsized":"2024-06-29T14:37:21.324673+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:28:20.994101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341127+0000","last_change":"2024-06-29T14:37:21.340382+0000","last_active":"2024-06-29T14:37:22.341127+0000","last_peered":"2024-06-29T14:37:22.341127+0000","last_clean":"2024-06-29T14:37:22.341127+0000","last_became_active":"2024-06-29T14:37:21.340108+0000","last_became_peered":"2024-06-29T14:37:21.340108+0000","last_unstale":"2024-06-29T14:37:22.341127+0000","last_undegraded":"2024-06-29T14:37:22.341127+0000","last_fullsized":"2024-06-29T14:37:22.341127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:42:45.066789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"77'8","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341549+0000","last_change":"2024-06-29T14:37:20.333200+0000","last_active":"2024-06-29T14:37:22.341549+0000","last_peered":"2024-06-29T14:37:22.341549+0000","last_clean":"2024-06-29T14:37:22.341549+0000","last_became_active":"2024-06-29T14:37:20.332809+0000","last_became_peered":"2024-06-29T14:37:20.332809+0000","last_unstale":"2024-06-29T14:37:22.341549+0000","last_undegraded":"2024-06-29T14:37:22.341549+0000","last_fullsized":"2024-06-29T14:37:22.341549+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:18:46.403873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.342517+0000","last_change":"2024-06-29T14:37:21.345719+0000","last_active":"2024-06-29T14:37:22.342517+0000","last_peered":"2024-06-29T14:37:22.342517+0000","last_clean":"2024-06-29T14:37:22.342517+0000","last_became_active":"2024-06-29T14:37:21.344358+0000","last_became_peered":"2024-06-29T14:37:21.344358+0000","last_unstale":"2024-06-29T14:37:22.342517+0000","last_undegraded":"2024-06-29T14:37:22.342517+0000","last_fullsized":"2024-06-29T14:37:22.342517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:26.622470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"77'1","reported_seq":15,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601748+0000","last_change":"2024-06-29T14:37:20.331191+0000","last_active":"2024-06-29T14:37:22.601748+0000","last_peered":"2024-06-29T14:37:22.601748+0000","last_clean":"2024-06-29T14:37:22.601748+0000","last_became_active":"2024-06-29T14:37:20.330799+0000","last_became_peered":"2024-06-29T14:37:20.330799+0000","last_unstale":"2024-06-29T14:37:22.601748+0000","last_undegraded":"2024-06-29T14:37:22.601748+0000","last_fullsized":"2024-06-29T14:37:22.601748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:55.068765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.336983+0000","last_change":"2024-06-29T14:37:21.334231+0000","last_active":"2024-06-29T14:37:21.336983+0000","last_peered":"2024-06-29T14:37:21.336983+0000","last_clean":"2024-06-29T14:37:21.336983+0000","last_became_active":"2024-06-29T14:37:21.333878+0000","last_became_peered":"2024-06-29T14:37:21.333878+0000","last_unstale":"2024-06-29T14:37:21.336983+0000","last_undegraded":"2024-06-29T14:37:21.336983+0000","last_fullsized":"2024-06-29T14:37:21.336983+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:32:49.784323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.352258+0000","last_change":"2024-06-29T14:37:20.331529+0000","last_active":"2024-06-29T14:37:21.352258+0000","last_peered":"2024-06-29T14:37:21.352258+0000","last_clean":"2024-06-29T14:37:21.352258+0000","last_became_active":"2024-06-29T14:37:20.331200+0000","last_became_peered":"2024-06-29T14:37:20.331200+0000","last_unstale":"2024-06-29T14:37:21.352258+0000","last_undegraded":"2024-06-29T14:37:21.352258+0000","last_fullsized":"2024-06-29T14:37:21.352258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:32:01.679875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601649+0000","last_change":"2024-06-29T14:37:21.344219+0000","last_active":"2024-06-29T14:37:22.601649+0000","last_peered":"2024-06-29T14:37:22.601649+0000","last_clean":"2024-06-29T14:37:22.601649+0000","last_became_active":"2024-06-29T14:37:21.343911+0000","last_became_peered":"2024-06-29T14:37:21.343911+0000","last_unstale":"2024-06-29T14:37:22.601649+0000","last_undegraded":"2024-06-29T14:37:22.601649+0000","last_fullsized":"2024-06-29T14:37:22.601649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:31:55.557195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.352771+0000","last_change":"2024-06-29T14:37:20.329583+0000","last_active":"2024-06-29T14:37:21.352771+0000","last_peered":"2024-06-29T14:37:21.352771+0000","last_clean":"2024-06-29T14:37:21.352771+0000","last_became_active":"2024-06-29T14:37:20.329219+0000","last_became_peered":"2024-06-29T14:37:20.329219+0000","last_unstale":"2024-06-29T14:37:21.352771+0000","last_undegraded":"2024-06-29T14:37:21.352771+0000","last_fullsized":"2024-06-29T14:37:21.352771+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:00:18.929987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341790+0000","last_change":"2024-06-29T14:37:21.351369+0000","last_active":"2024-06-29T14:37:22.341790+0000","last_peered":"2024-06-29T14:37:22.341790+0000","last_clean":"2024-06-29T14:37:22.341790+0000","last_became_active":"2024-06-29T14:37:21.351082+0000","last_became_peered":"2024-06-29T14:37:21.351082+0000","last_unstale":"2024-06-29T14:37:22.341790+0000","last_undegraded":"2024-06-29T14:37:22.341790+0000","last_fullsized":"2024-06-29T14:37:22.341790+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:14:09.378628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.330706+0000","last_change":"2024-06-29T14:37:20.334328+0000","last_active":"2024-06-29T14:37:21.330706+0000","last_peered":"2024-06-29T14:37:21.330706+0000","last_clean":"2024-06-29T14:37:21.330706+0000","last_became_active":"2024-06-29T14:37:20.333959+0000","last_became_peered":"2024-06-29T14:37:20.333959+0000","last_unstale":"2024-06-29T14:37:21.330706+0000","last_undegraded":"2024-06-29T14:37:21.330706+0000","last_fullsized":"2024-06-29T14:37:21.330706+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:39.866303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.337272+0000","last_change":"2024-06-29T14:37:21.331835+0000","last_active":"2024-06-29T14:37:21.337272+0000","last_peered":"2024-06-29T14:37:21.337272+0000","last_clean":"2024-06-29T14:37:21.337272+0000","last_became_active":"2024-06-29T14:37:21.331459+0000","last_became_peered":"2024-06-29T14:37:21.331459+0000","last_unstale":"2024-06-29T14:37:21.337272+0000","last_undegraded":"2024-06-29T14:37:21.337272+0000","last_fullsized":"2024-06-29T14:37:21.337272+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:38:31.348583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"77'2","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601703+0000","last_change":"2024-06-29T14:37:20.331768+0000","last_active":"2024-06-29T14:37:22.601703+0000","last_peered":"2024-06-29T14:37:22.601703+0000","last_clean":"2024-06-29T14:37:22.601703+0000","last_became_active":"2024-06-29T14:37:20.331463+0000","last_became_peered":"2024-06-29T14:37:20.331463+0000","last_unstale":"2024-06-29T14:37:22.601703+0000","last_undegraded":"2024-06-29T14:37:22.601703+0000","last_fullsized":"2024-06-29T14:37:22.601703+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:02:25.719152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.339576+0000","last_change":"2024-06-29T14:37:21.334081+0000","last_active":"2024-06-29T14:37:21.339576+0000","last_peered":"2024-06-29T14:37:21.339576+0000","last_clean":"2024-06-29T14:37:21.339576+0000","last_became_active":"2024-06-29T14:37:21.333798+0000","last_became_peered":"2024-06-29T14:37:21.333798+0000","last_unstale":"2024-06-29T14:37:21.339576+0000","last_undegraded":"2024-06-29T14:37:21.339576+0000","last_fullsized":"2024-06-29T14:37:21.339576+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:30.816196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.321025+0000","last_change":"2024-06-29T14:37:20.321965+0000","last_active":"2024-06-29T14:37:21.321025+0000","last_peered":"2024-06-29T14:37:21.321025+0000","last_clean":"2024-06-29T14:37:21.321025+0000","last_became_active":"2024-06-29T14:37:20.321651+0000","last_became_peered":"2024-06-29T14:37:20.321651+0000","last_unstale":"2024-06-29T14:37:21.321025+0000","last_undegraded":"2024-06-29T14:37:21.321025+0000","last_fullsized":"2024-06-29T14:37:21.321025+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:32:50.004657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347164+0000","last_change":"2024-06-29T14:37:21.342704+0000","last_active":"2024-06-29T14:37:21.347164+0000","last_peered":"2024-06-29T14:37:21.347164+0000","last_clean":"2024-06-29T14:37:21.347164+0000","last_became_active":"2024-06-29T14:37:21.342411+0000","last_became_peered":"2024-06-29T14:37:21.342411+0000","last_unstale":"2024-06-29T14:37:21.347164+0000","last_undegraded":"2024-06-29T14:37:21.347164+0000","last_fullsized":"2024-06-29T14:37:21.347164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:10:42.782502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.323435+0000","last_change":"2024-06-29T14:37:20.331301+0000","last_active":"2024-06-29T14:37:21.323435+0000","last_peered":"2024-06-29T14:37:21.323435+0000","last_clean":"2024-06-29T14:37:21.323435+0000","last_became_active":"2024-06-29T14:37:20.331006+0000","last_became_peered":"2024-06-29T14:37:20.331006+0000","last_unstale":"2024-06-29T14:37:21.323435+0000","last_undegraded":"2024-06-29T14:37:21.323435+0000","last_fullsized":"2024-06-29T14:37:21.323435+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:44:41.313263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.339315+0000","last_change":"2024-06-29T14:37:21.330028+0000","last_active":"2024-06-29T14:37:21.339315+0000","last_peered":"2024-06-29T14:37:21.339315+0000","last_clean":"2024-06-29T14:37:21.339315+0000","last_became_active":"2024-06-29T14:37:21.329385+0000","last_became_peered":"2024-06-29T14:37:21.329385+0000","last_unstale":"2024-06-29T14:37:21.339315+0000","last_undegraded":"2024-06-29T14:37:21.339315+0000","last_fullsized":"2024-06-29T14:37:21.339315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:38:34.058779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"77'2","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.349457+0000","last_change":"2024-06-29T14:37:20.335098+0000","last_active":"2024-06-29T14:37:21.349457+0000","last_peered":"2024-06-29T14:37:21.349457+0000","last_clean":"2024-06-29T14:37:21.349457+0000","last_became_active":"2024-06-29T14:37:20.334764+0000","last_became_peered":"2024-06-29T14:37:20.334764+0000","last_unstale":"2024-06-29T14:37:21.349457+0000","last_undegraded":"2024-06-29T14:37:21.349457+0000","last_fullsized":"2024-06-29T14:37:21.349457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:35.731609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.344769+0000","last_change":"2024-06-29T14:37:21.341948+0000","last_active":"2024-06-29T14:37:21.344769+0000","last_peered":"2024-06-29T14:37:21.344769+0000","last_clean":"2024-06-29T14:37:21.344769+0000","last_became_active":"2024-06-29T14:37:21.341579+0000","last_became_peered":"2024-06-29T14:37:21.341579+0000","last_unstale":"2024-06-29T14:37:21.344769+0000","last_undegraded":"2024-06-29T14:37:21.344769+0000","last_fullsized":"2024-06-29T14:37:21.344769+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:50.328395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"77'1","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.334554+0000","last_change":"2024-06-29T14:37:20.329392+0000","last_active":"2024-06-29T14:37:22.334554+0000","last_peered":"2024-06-29T14:37:22.334554+0000","last_clean":"2024-06-29T14:37:22.334554+0000","last_became_active":"2024-06-29T14:37:20.328262+0000","last_became_peered":"2024-06-29T14:37:20.328262+0000","last_unstale":"2024-06-29T14:37:22.334554+0000","last_undegraded":"2024-06-29T14:37:22.334554+0000","last_fullsized":"2024-06-29T14:37:22.334554+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:04:45.186965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.347514+0000","last_change":"2024-06-29T14:37:21.340775+0000","last_active":"2024-06-29T14:37:21.347514+0000","last_peered":"2024-06-29T14:37:21.347514+0000","last_clean":"2024-06-29T14:37:21.347514+0000","last_became_active":"2024-06-29T14:37:21.340459+0000","last_became_peered":"2024-06-29T14:37:21.340459+0000","last_unstale":"2024-06-29T14:37:21.347514+0000","last_undegraded":"2024-06-29T14:37:21.347514+0000","last_fullsized":"2024-06-29T14:37:21.347514+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:34.472101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341831+0000","last_change":"2024-06-29T14:37:20.330320+0000","last_active":"2024-06-29T14:37:22.341831+0000","last_peered":"2024-06-29T14:37:22.341831+0000","last_clean":"2024-06-29T14:37:22.341831+0000","last_became_active":"2024-06-29T14:37:20.330015+0000","last_became_peered":"2024-06-29T14:37:20.330015+0000","last_unstale":"2024-06-29T14:37:22.341831+0000","last_undegraded":"2024-06-29T14:37:22.341831+0000","last_fullsized":"2024-06-29T14:37:22.341831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:46:27.470117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.342855+0000","last_change":"2024-06-29T14:37:21.337846+0000","last_active":"2024-06-29T14:37:21.342855+0000","last_peered":"2024-06-29T14:37:21.342855+0000","last_clean":"2024-06-29T14:37:21.342855+0000","last_became_active":"2024-06-29T14:37:21.337554+0000","last_became_peered":"2024-06-29T14:37:21.337554+0000","last_unstale":"2024-06-29T14:37:21.342855+0000","last_undegraded":"2024-06-29T14:37:21.342855+0000","last_fullsized":"2024-06-29T14:37:21.342855+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:14.460881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.324774+0000","last_change":"2024-06-29T14:37:20.325501+0000","last_active":"2024-06-29T14:37:21.324774+0000","last_peered":"2024-06-29T14:37:21.324774+0000","last_clean":"2024-06-29T14:37:21.324774+0000","last_became_active":"2024-06-29T14:37:20.325177+0000","last_became_peered":"2024-06-29T14:37:20.325177+0000","last_unstale":"2024-06-29T14:37:21.324774+0000","last_undegraded":"2024-06-29T14:37:21.324774+0000","last_fullsized":"2024-06-29T14:37:21.324774+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:37:20.299114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601492+0000","last_change":"2024-06-29T14:37:21.336648+0000","last_active":"2024-06-29T14:37:22.601492+0000","last_peered":"2024-06-29T14:37:22.601492+0000","last_clean":"2024-06-29T14:37:22.601492+0000","last_became_active":"2024-06-29T14:37:21.336213+0000","last_became_peered":"2024-06-29T14:37:21.336213+0000","last_unstale":"2024-06-29T14:37:22.601492+0000","last_undegraded":"2024-06-29T14:37:22.601492+0000","last_fullsized":"2024-06-29T14:37:22.601492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:42:18.286112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.321050+0000","last_change":"2024-06-29T14:37:20.322453+0000","last_active":"2024-06-29T14:37:21.321050+0000","last_peered":"2024-06-29T14:37:21.321050+0000","last_clean":"2024-06-29T14:37:21.321050+0000","last_became_active":"2024-06-29T14:37:20.322147+0000","last_became_peered":"2024-06-29T14:37:20.322147+0000","last_unstale":"2024-06-29T14:37:21.321050+0000","last_undegraded":"2024-06-29T14:37:21.321050+0000","last_fullsized":"2024-06-29T14:37:21.321050+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:29.811184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341503+0000","last_change":"2024-06-29T14:37:21.347234+0000","last_active":"2024-06-29T14:37:22.341503+0000","last_peered":"2024-06-29T14:37:22.341503+0000","last_clean":"2024-06-29T14:37:22.341503+0000","last_became_active":"2024-06-29T14:37:21.346123+0000","last_became_peered":"2024-06-29T14:37:21.346123+0000","last_unstale":"2024-06-29T14:37:22.341503+0000","last_undegraded":"2024-06-29T14:37:22.341503+0000","last_fullsized":"2024-06-29T14:37:22.341503+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:12.328298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.330392+0000","last_change":"2024-06-29T14:37:20.333964+0000","last_active":"2024-06-29T14:37:21.330392+0000","last_peered":"2024-06-29T14:37:21.330392+0000","last_clean":"2024-06-29T14:37:21.330392+0000","last_became_active":"2024-06-29T14:37:20.333661+0000","last_became_peered":"2024-06-29T14:37:20.333661+0000","last_unstale":"2024-06-29T14:37:21.330392+0000","last_undegraded":"2024-06-29T14:37:21.330392+0000","last_fullsized":"2024-06-29T14:37:21.330392+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:02:44.976401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.332440+0000","last_change":"2024-06-29T14:37:21.327530+0000","last_active":"2024-06-29T14:37:21.332440+0000","last_peered":"2024-06-29T14:37:21.332440+0000","last_clean":"2024-06-29T14:37:21.332440+0000","last_became_active":"2024-06-29T14:37:21.327184+0000","last_became_peered":"2024-06-29T14:37:21.327184+0000","last_unstale":"2024-06-29T14:37:21.332440+0000","last_undegraded":"2024-06-29T14:37:21.332440+0000","last_fullsized":"2024-06-29T14:37:21.332440+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:43:33.783969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-06-29T14:37:21.321033+0000","last_change":"2024-06-29T14:37:20.326915+0000","last_active":"2024-06-29T14:37:21.321033+0000","last_peered":"2024-06-29T14:37:21.321033+0000","last_clean":"2024-06-29T14:37:21.321033+0000","last_became_active":"2024-06-29T14:37:20.326569+0000","last_became_peered":"2024-06-29T14:37:20.326569+0000","last_unstale":"2024-06-29T14:37:21.321033+0000","last_undegraded":"2024-06-29T14:37:21.321033+0000","last_fullsized":"2024-06-29T14:37:21.321033+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:13.658291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601397+0000","last_change":"2024-06-29T14:37:21.335794+0000","last_active":"2024-06-29T14:37:22.601397+0000","last_peered":"2024-06-29T14:37:22.601397+0000","last_clean":"2024-06-29T14:37:22.601397+0000","last_became_active":"2024-06-29T14:37:21.335461+0000","last_became_peered":"2024-06-29T14:37:21.335461+0000","last_unstale":"2024-06-29T14:37:22.601397+0000","last_undegraded":"2024-06-29T14:37:22.601397+0000","last_fullsized":"2024-06-29T14:37:22.601397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:17.833749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601442+0000","last_change":"2024-06-29T14:37:20.328161+0000","last_active":"2024-06-29T14:37:22.601442+0000","last_peered":"2024-06-29T14:37:22.601442+0000","last_clean":"2024-06-29T14:37:22.601442+0000","last_became_active":"2024-06-29T14:37:20.327853+0000","last_became_peered":"2024-06-29T14:37:20.327853+0000","last_unstale":"2024-06-29T14:37:22.601442+0000","last_undegraded":"2024-06-29T14:37:22.601442+0000","last_fullsized":"2024-06-29T14:37:22.601442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:11:58.959487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.601358+0000","last_change":"2024-06-29T14:37:21.340956+0000","last_active":"2024-06-29T14:37:22.601358+0000","last_peered":"2024-06-29T14:37:22.601358+0000","last_clean":"2024-06-29T14:37:22.601358+0000","last_became_active":"2024-06-29T14:37:21.340681+0000","last_became_peered":"2024-06-29T14:37:21.340681+0000","last_unstale":"2024-06-29T14:37:22.601358+0000","last_undegraded":"2024-06-29T14:37:22.601358+0000","last_fullsized":"2024-06-29T14:37:22.601358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:20.306292+0000","last_clean_scrub_stamp":"2024-06-29T14:37:20.306292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:31:15.861788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341726+0000","last_change":"2024-06-29T14:37:20.323931+0000","last_active":"2024-06-29T14:37:22.341726+0000","last_peered":"2024-06-29T14:37:22.341726+0000","last_clean":"2024-06-29T14:37:22.341726+0000","last_became_active":"2024-06-29T14:37:20.323595+0000","last_became_peered":"2024-06-29T14:37:20.323595+0000","last_unstale":"2024-06-29T14:37:22.341726+0000","last_undegraded":"2024-06-29T14:37:22.341726+0000","last_fullsized":"2024-06-29T14:37:22.341726+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:19.301280+0000","last_clean_scrub_stamp":"2024-06-29T14:37:19.301280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:02:55.617173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"67'100","reported_seq":326,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-06-29T14:37:22.341264+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:37:22.341264+0000","last_peered":"2024-06-29T14:37:22.341264+0000","last_clean":"2024-06-29T14:37:22.341264+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:37:22.341264+0000","last_undegraded":"2024-06-29T14:37:22.341264+0000","last_fullsized":"2024-06-29T14:37:22.341264+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575030,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1531904,"data_stored":1032782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575031,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":929792,"data_stored":442254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575029,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":442906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575030,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58472,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313368,"statfs":{"total":96636764160,"available":96576888832,"internally_reserved":0,"allocated":942080,"data_stored":443866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575030,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36260,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335580,"statfs":{"total":96636764160,"available":96599633920,"internally_reserved":0,"allocated":937984,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36276,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335564,"statfs":{"total":96636764160,"available":96599617536,"internally_reserved":0,"allocated":950272,"data_stored":448682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50148,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321692,"statfs":{"total":96636764160,"available":96585412608,"internally_reserved":0,"allocated":933888,"data_stored":442344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575030,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1540096,"data_stored":1034816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:37:25.716 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:37:25.717 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:37:25.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.875+0000 7f5b2e089640 1 -- 172.21.15.81:0/2013113883 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b28073480 msgr2=0x7f5b2815b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.875+0000 7f5b2e089640 1 --2- 172.21.15.81:0/2013113883 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b28073480 0x7f5b2815b6e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c005fd0 tx=0x7f5b1c02a4a0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.876+0000 7f5b2e089640 1 -- 172.21.15.81:0/2013113883 shutdown_connections 2024-06-29T14:37:25.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.876+0000 7f5b2e089640 1 --2- 172.21.15.81:0/2013113883 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b28073480 0x7f5b2815b6e0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.876+0000 7f5b2e089640 1 --2- 172.21.15.81:0/2013113883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b280cd150 0x7f5b28072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.876+0000 7f5b2e089640 1 -- 172.21.15.81:0/2013113883 >> 172.21.15.81:0/2013113883 conn(0x7f5b280b2920 msgr2=0x7f5b280b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:25.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.876+0000 7f5b2e089640 1 -- 172.21.15.81:0/2013113883 shutdown_connections 2024-06-29T14:37:25.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.876+0000 7f5b2e089640 1 -- 172.21.15.81:0/2013113883 wait complete. 2024-06-29T14:37:25.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.877+0000 7f5b2e089640 1 Processor -- start 2024-06-29T14:37:25.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.878+0000 7f5b2e089640 1 -- start start 2024-06-29T14:37:25.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b2e089640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b28073480 0x7f5b281f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b2e089640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b280cd150 0x7f5b2814f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b2e089640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2815bc20 0x7f5b2806c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b2e089640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2815c1b0 con 0x7f5b2815bc20 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b2e089640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2815c030 con 0x7f5b280cd150 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b2e089640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5b2815f060 con 0x7f5b28073480 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b27fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2815bc20 0x7f5b2806c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b27fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2815bc20 0x7f5b2806c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41186/0 (socket says 172.21.15.81:41186) 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b27fff640 1 -- 172.21.15.81:0/1911844924 learned_addr learned my addr 172.21.15.81:0/1911844924 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b277fe640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b28073480 0x7f5b281f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b26ffd640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b280cd150 0x7f5b2814f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b27fff640 1 -- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b28073480 msgr2=0x7f5b281f4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b27fff640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b28073480 0x7f5b281f4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b27fff640 1 -- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b280cd150 msgr2=0x7f5b2814f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b27fff640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b280cd150 0x7f5b2814f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.879+0000 7f5b27fff640 1 -- 172.21.15.81:0/1911844924 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b280adc40 con 0x7f5b2815bc20 2024-06-29T14:37:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.880+0000 7f5b27fff640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2815bc20 0x7f5b2806c320 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000b9a0 tx=0x7f5b2000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.880+0000 7f5b0ffff640 1 -- 172.21.15.81:0/1911844924 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20019070 con 0x7f5b2815bc20 2024-06-29T14:37:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.880+0000 7f5b2e089640 1 -- 172.21.15.81:0/1911844924 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b281f51e0 con 0x7f5b2815bc20 2024-06-29T14:37:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.880+0000 7f5b0ffff640 1 -- 172.21.15.81:0/1911844924 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b2000cc90 con 0x7f5b2815bc20 2024-06-29T14:37:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.880+0000 7f5b0ffff640 1 -- 172.21.15.81:0/1911844924 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b200146a0 con 0x7f5b2815bc20 2024-06-29T14:37:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.880+0000 7f5b2e089640 1 -- 172.21.15.81:0/1911844924 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b28152090 con 0x7f5b2815bc20 2024-06-29T14:37:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.881+0000 7f5b2e089640 1 -- 172.21.15.81:0/1911844924 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b280714d0 con 0x7f5b2815bc20 2024-06-29T14:37:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.882+0000 7f5b0ffff640 1 -- 172.21.15.81:0/1911844924 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f5b2001c070 con 0x7f5b2815bc20 2024-06-29T14:37:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.882+0000 7f5b0ffff640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5b140763f0 0x7f5b14078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.882+0000 7f5b0ffff640 1 -- 172.21.15.81:0/1911844924 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7f5b200976c0 con 0x7f5b2815bc20 2024-06-29T14:37:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.883+0000 7f5b277fe640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5b140763f0 0x7f5b14078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.883+0000 7f5b277fe640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5b140763f0 0x7f5b14078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5b18004630 tx=0x7f5b1800b2e0 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:25.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:25.887+0000 7f5b0ffff640 1 -- 172.21.15.81:0/1911844924 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5b280714d0 con 0x7f5b2815bc20 2024-06-29T14:37:26.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.135+0000 7f5b2e089640 1 -- 172.21.15.81:0/1911844924 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f5b2815f580 con 0x7f5b2815bc20 2024-06-29T14:37:26.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.137+0000 7f5b0ffff640 1 -- 172.21.15.81:0/1911844924 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v31) ==== 190+0+147 (secure 0 0 0) 0x7f5b20060760 con 0x7f5b2815bc20 2024-06-29T14:37:26.136 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T14:37:26.136 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T14:37:26.136 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T14:37:26.136 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T14:37:26.136 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:37:26.136 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T14:37:26.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.139+0000 7f5b2e089640 1 -- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5b140763f0 msgr2=0x7f5b14078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.139+0000 7f5b2e089640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5b140763f0 0x7f5b14078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5b18004630 tx=0x7f5b1800b2e0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.139+0000 7f5b2e089640 1 -- 172.21.15.81:0/1911844924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2815bc20 msgr2=0x7f5b2806c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.139+0000 7f5b2e089640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2815bc20 0x7f5b2806c320 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000b9a0 tx=0x7f5b2000be60 comp rx=0 tx=0).stop 2024-06-29T14:37:26.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.139+0000 7f5b2e089640 1 -- 172.21.15.81:0/1911844924 shutdown_connections 2024-06-29T14:37:26.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.139+0000 7f5b2e089640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f5b140763f0 0x7f5b14078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.139+0000 7f5b2e089640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2815bc20 0x7f5b2806c320 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.139+0000 7f5b2e089640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b280cd150 0x7f5b2814f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.139+0000 7f5b2e089640 1 --2- 172.21.15.81:0/1911844924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5b28073480 0x7f5b281f4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.139+0000 7f5b2e089640 1 -- 172.21.15.81:0/1911844924 >> 172.21.15.81:0/1911844924 conn(0x7f5b280b2920 msgr2=0x7f5b280b0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:26.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.140+0000 7f5b2e089640 1 -- 172.21.15.81:0/1911844924 shutdown_connections 2024-06-29T14:37:26.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.140+0000 7f5b2e089640 1 -- 172.21.15.81:0/1911844924 wait complete. 2024-06-29T14:37:26.149 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:26.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.264+0000 7f72a1c3f640 1 -- 172.21.15.81:0/4053664991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729c074520 msgr2=0x7f729c06f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.264+0000 7f72a1c3f640 1 --2- 172.21.15.81:0/4053664991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729c074520 0x7f729c06f140 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f729c06e210 tx=0x7f728c02a560 comp rx=0 tx=0).stop 2024-06-29T14:37:26.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.264+0000 7f72a1c3f640 1 -- 172.21.15.81:0/4053664991 shutdown_connections 2024-06-29T14:37:26.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.264+0000 7f72a1c3f640 1 --2- 172.21.15.81:0/4053664991 >> v2:172.21.15.167:3300/0 conn(0x7f729c06f680 0x7f729c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.264+0000 7f72a1c3f640 1 --2- 172.21.15.81:0/4053664991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729c074520 0x7f729c06f140 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.264+0000 7f72a1c3f640 1 -- 172.21.15.81:0/4053664991 >> 172.21.15.81:0/4053664991 conn(0x7f729c072950 msgr2=0x7f729c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:26.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.264+0000 7f72a1c3f640 1 -- 172.21.15.81:0/4053664991 shutdown_connections 2024-06-29T14:37:26.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.264+0000 7f72a1c3f640 1 -- 172.21.15.81:0/4053664991 wait complete. 2024-06-29T14:37:26.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.266+0000 7f72a1c3f640 1 Processor -- start 2024-06-29T14:37:26.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.266+0000 7f72a1c3f640 1 -- start start 2024-06-29T14:37:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.267+0000 7f72a1c3f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729c06f680 0x7f729c155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.267+0000 7f72a1c3f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f729c06fa90 0x7f729c1f15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.267+0000 7f72a1c3f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f729c074520 0x7f729c09c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.267+0000 7f72a1c3f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f729c15b010 con 0x7f729c06f680 2024-06-29T14:37:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.267+0000 7f72a1c3f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f729c06d150 con 0x7f729c06fa90 2024-06-29T14:37:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.267+0000 7f72a1c3f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f729c15b190 con 0x7f729c074520 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.267+0000 7f729b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729c06f680 0x7f729c155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.267+0000 7f729b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729c06f680 0x7f729c155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41206/0 (socket says 172.21.15.81:41206) 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.267+0000 7f729b7fe640 1 -- 172.21.15.81:0/3377691515 learned_addr learned my addr 172.21.15.81:0/3377691515 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f729bfff640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f729c074520 0x7f729c09c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f729affd640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f729c06fa90 0x7f729c1f15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f729b7fe640 1 -- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f729c074520 msgr2=0x7f729c09c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f729b7fe640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f729c074520 0x7f729c09c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f729b7fe640 1 -- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f729c06fa90 msgr2=0x7f729c1f15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f729b7fe640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f729c06fa90 0x7f729c1f15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f729b7fe640 1 -- 172.21.15.81:0/3377691515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729c153810 con 0x7f729c06f680 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f729b7fe640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729c06f680 0x7f729c155940 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f728c034040 tx=0x7f728c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f7283fff640 1 -- 172.21.15.81:0/3377691515 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c03e030 con 0x7f729c06f680 2024-06-29T14:37:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f7283fff640 1 -- 172.21.15.81:0/3377691515 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f728c0047a0 con 0x7f729c06f680 2024-06-29T14:37:26.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f72a1c3f640 1 -- 172.21.15.81:0/3377691515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729c153ab0 con 0x7f729c06f680 2024-06-29T14:37:26.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f7283fff640 1 -- 172.21.15.81:0/3377691515 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c035040 con 0x7f729c06f680 2024-06-29T14:37:26.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.268+0000 7f72a1c3f640 1 -- 172.21.15.81:0/3377691515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729c154070 con 0x7f729c06f680 2024-06-29T14:37:26.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.269+0000 7f72a1c3f640 1 -- 172.21.15.81:0/3377691515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f729c0af540 con 0x7f729c06f680 2024-06-29T14:37:26.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.270+0000 7f7283fff640 1 -- 172.21.15.81:0/3377691515 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f728c041030 con 0x7f729c06f680 2024-06-29T14:37:26.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.270+0000 7f7283fff640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72880763f0 0x7f7288078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:26.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.271+0000 7f7283fff640 1 -- 172.21.15.81:0/3377691515 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7f728c0b8350 con 0x7f729c06f680 2024-06-29T14:37:26.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.271+0000 7f729affd640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72880763f0 0x7f7288078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:26.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.271+0000 7f729affd640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72880763f0 0x7f7288078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7290005fd0 tx=0x7f7290005d80 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:26.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.275+0000 7f7283fff640 1 -- 172.21.15.81:0/3377691515 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f728c0819b0 con 0x7f729c06f680 2024-06-29T14:37:26.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.493+0000 7f72a1c3f640 1 -- 172.21.15.81:0/3377691515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f729c1582e0 con 0x7f729c06f680 2024-06-29T14:37:26.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.493+0000 7f7283fff640 1 -- 172.21.15.81:0/3377691515 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5226 (secure 0 0 0) 0x7f728c0812b0 con 0x7f729c06f680 2024-06-29T14:37:26.492 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:26.492 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"btime":"2024-06-29T14:37:24:340402+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3148404249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3148404249},{"type":"v1","addr":"172.21.15.167:6836","nonce":3148404249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6549,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/704367026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":704367026},{"type":"v1","addr":"172.21.15.81:6834","nonce":704367026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6552,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2316096753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2316096753},{"type":"v1","addr":"172.21.15.81:6835","nonce":2316096753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.692072+0000","modified":"2024-06-29T14:37:23.709021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6567":{"gid":6567,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/2723173791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2723173791},{"type":"v1","addr":"172.21.15.167:6837","nonce":2723173791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6567,"qdb_cluster":[6567]},"id":4}]} 2024-06-29T14:37:26.492 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 34 2024-06-29T14:37:26.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 -- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72880763f0 msgr2=0x7f7288078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72880763f0 0x7f7288078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7290005fd0 tx=0x7f7290005d80 comp rx=0 tx=0).stop 2024-06-29T14:37:26.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 -- 172.21.15.81:0/3377691515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729c06f680 msgr2=0x7f729c155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:26.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729c06f680 0x7f729c155940 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f728c034040 tx=0x7f728c0039e0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 -- 172.21.15.81:0/3377691515 shutdown_connections 2024-06-29T14:37:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f72880763f0 0x7f7288078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f729c074520 0x7f729c09c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f729c06fa90 0x7f729c1f15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 --2- 172.21.15.81:0/3377691515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729c06f680 0x7f729c155940 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 -- 172.21.15.81:0/3377691515 >> 172.21.15.81:0/3377691515 conn(0x7f729c072950 msgr2=0x7f729c06c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 -- 172.21.15.81:0/3377691515 shutdown_connections 2024-06-29T14:37:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:26.496+0000 7f72a1c3f640 1 -- 172.21.15.81:0/3377691515 wait complete. 2024-06-29T14:37:26.908 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 34, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:37:20.692072+0000', 'modified': '2024-06-29T14:37:23.709021+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6567}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6567': {'gid': 6567, 'name': 'b', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.167:6837/2723173791', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6835', 'nonce': 2723173791}, {'type': 'v1', 'addr': '172.21.15.167:6837', 'nonce': 2723173791}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6567, 'qdb_cluster': [6567]} 2024-06-29T14:37:26.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:37:26.909 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-06-29T14:37:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.027+0000 7f8ffc847640 1 -- 172.21.15.81:0/3909545786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ff80b35b0 msgr2=0x7f8ff815b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.027+0000 7f8ffc847640 1 --2- 172.21.15.81:0/3909545786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ff80b35b0 0x7f8ff815b5b0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f8ff815aac0 tx=0x7f8fe802a980 comp rx=0 tx=0).stop 2024-06-29T14:37:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.027+0000 7f8ffc847640 1 -- 172.21.15.81:0/3909545786 shutdown_connections 2024-06-29T14:37:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.027+0000 7f8ffc847640 1 --2- 172.21.15.81:0/3909545786 >> v2:172.21.15.167:3300/0 conn(0x7f8ff806f660 0x7f8ff806fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.027+0000 7f8ffc847640 1 --2- 172.21.15.81:0/3909545786 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ff80b4430 0x7f8ff80cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.027+0000 7f8ffc847640 1 --2- 172.21.15.81:0/3909545786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ff80b35b0 0x7f8ff815b5b0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.027+0000 7f8ffc847640 1 -- 172.21.15.81:0/3909545786 >> 172.21.15.81:0/3909545786 conn(0x7f8ff806ccc0 msgr2=0x7f8ff80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.027+0000 7f8ffc847640 1 -- 172.21.15.81:0/3909545786 shutdown_connections 2024-06-29T14:37:27.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.027+0000 7f8ffc847640 1 -- 172.21.15.81:0/3909545786 wait complete. 2024-06-29T14:37:27.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.029+0000 7f8ffc847640 1 Processor -- start 2024-06-29T14:37:27.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.029+0000 7f8ffc847640 1 -- start start 2024-06-29T14:37:27.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ffc847640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ff806f660 0x7f8ff8150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ffc847640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ff80b35b0 0x7f8ff8148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ffc847640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ff80b4430 0x7f8ff8148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ffc847640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff806f160 con 0x7f8ff80b4430 2024-06-29T14:37:27.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ffc847640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff806efe0 con 0x7f8ff80b35b0 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ffc847640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8ff806f2e0 con 0x7f8ff806f660 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ff6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ff80b4430 0x7f8ff8148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ff6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ff80b4430 0x7f8ff8148570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41228/0 (socket says 172.21.15.81:41228) 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/1181976874 learned_addr learned my addr 172.21.15.81:0/1181976874 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ff659c640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ff806f660 0x7f8ff8150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.030+0000 7f8ff5d9b640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ff80b35b0 0x7f8ff8148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ff806f660 msgr2=0x7f8ff8150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8ff6d9d640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ff806f660 0x7f8ff8150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ff80b35b0 msgr2=0x7f8ff8148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8ff6d9d640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ff80b35b0 0x7f8ff8148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/1181976874 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff8144860 con 0x7f8ff80b4430 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8ff6d9d640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ff80b4430 0x7f8ff8148570 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000d5e0 tx=0x7f8ff000daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8fe6ffd640 1 -- 172.21.15.81:0/1181976874 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0017070 con 0x7f8ff80b4430 2024-06-29T14:37:27.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8fe6ffd640 1 -- 172.21.15.81:0/1181976874 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ff0005310 con 0x7f8ff80b4430 2024-06-29T14:37:27.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8ffc847640 1 -- 172.21.15.81:0/1181976874 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff8150f70 con 0x7f8ff80b4430 2024-06-29T14:37:27.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8fe6ffd640 1 -- 172.21.15.81:0/1181976874 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00047a0 con 0x7f8ff80b4430 2024-06-29T14:37:27.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.031+0000 7f8ffc847640 1 -- 172.21.15.81:0/1181976874 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff8152880 con 0x7f8ff80b4430 2024-06-29T14:37:27.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.033+0000 7f8fe6ffd640 1 -- 172.21.15.81:0/1181976874 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f8ff0004940 con 0x7f8ff80b4430 2024-06-29T14:37:27.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.033+0000 7f8fe6ffd640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8fd8076420 0x7f8fd8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.033+0000 7f8fe6ffd640 1 -- 172.21.15.81:0/1181976874 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7f8ff0097860 con 0x7f8ff80b4430 2024-06-29T14:37:27.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.034+0000 7f8ff659c640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8fd8076420 0x7f8fd8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.034+0000 7f8ff659c640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8fd8076420 0x7f8fd8078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8035000 tx=0x7f8fe8004ab0 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.034+0000 7f90037e3740 1 Processor -- start 2024-06-29T14:37:27.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.034+0000 7f90037e3740 1 -- start start 2024-06-29T14:37:27.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f90037e3740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2079270 0x555bf2469af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f90037e3740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf20503b0 0x555bf244d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f90037e3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2058590 0x555bf244dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f90037e3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555bf23e1bc0 con 0x555bf2058590 2024-06-29T14:37:27.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f90037e3740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x555bf2390630 con 0x555bf20503b0 2024-06-29T14:37:27.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f90037e3740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x555bf237fbd0 con 0x555bf2079270 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2058590 0x555bf244dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2058590 0x555bf244dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41242/0 (socket says 172.21.15.81:41242) 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/1316910250 learned_addr learned my addr 172.21.15.81:0/1316910250 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff659c640 1 --2- 172.21.15.81:0/1316910250 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2079270 0x555bf2469af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff5d9b640 1 --2- 172.21.15.81:0/1316910250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf20503b0 0x555bf244d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/1316910250 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2079270 msgr2=0x555bf2469af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff6d9d640 1 --2- 172.21.15.81:0/1316910250 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2079270 0x555bf2469af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/1316910250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf20503b0 msgr2=0x555bf244d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff6d9d640 1 --2- 172.21.15.81:0/1316910250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf20503b0 0x555bf244d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/1316910250 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bf237cca0 con 0x555bf2058590 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff5d9b640 1 --2- 172.21.15.81:0/1316910250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf20503b0 0x555bf244d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8ff6d9d640 1 --2- 172.21.15.81:0/1316910250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2058590 0x555bf244dcf0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f8ff005cef0 tx=0x7f8ff0060250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede244870c7caa2b server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8fd7fff640 1 -- 172.21.15.81:0/1316910250 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0020040 con 0x555bf2058590 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8fd7fff640 1 -- 172.21.15.81:0/1316910250 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ff0060900 con 0x555bf2058590 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.036+0000 7f8fd7fff640 1 -- 172.21.15.81:0/1316910250 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0064c00 con 0x555bf2058590 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.037+0000 7f90037e3740 1 -- 172.21.15.81:0/1316910250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2058590 msgr2=0x555bf244dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.037+0000 7f90037e3740 1 --2- 172.21.15.81:0/1316910250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2058590 0x555bf244dcf0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f8ff005cef0 tx=0x7f8ff0060250 comp rx=0 tx=0).stop 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.037+0000 7f90037e3740 1 -- 172.21.15.81:0/1316910250 shutdown_connections 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.037+0000 7f90037e3740 1 --2- 172.21.15.81:0/1316910250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2058590 0x555bf244dcf0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.037+0000 7f90037e3740 1 --2- 172.21.15.81:0/1316910250 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf20503b0 0x555bf244d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.037+0000 7f90037e3740 1 --2- 172.21.15.81:0/1316910250 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2079270 0x555bf2469af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.037+0000 7f90037e3740 1 -- 172.21.15.81:0/1316910250 >> 172.21.15.81:0/1316910250 conn(0x555bf206d3c0 msgr2=0x555bf2421380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.037+0000 7f90037e3740 1 -- 172.21.15.81:0/1316910250 shutdown_connections 2024-06-29T14:37:27.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.037+0000 7f90037e3740 1 -- 172.21.15.81:0/1316910250 wait complete. 2024-06-29T14:37:27.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.038+0000 7f90037e3740 1 -- start start 2024-06-29T14:37:27.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.038+0000 7f90037e3740 1 Processor -- start 2024-06-29T14:37:27.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f90037e3740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf20503b0 0x555bf2444fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f90037e3740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2058590 0x555bf2460c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f90037e3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2079270 0x555bf2461170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f90037e3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555bf231dae0 con 0x555bf2079270 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f90037e3740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x555bf2383c10 con 0x555bf2058590 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f90037e3740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x555bf20f3620 con 0x555bf20503b0 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f90037e3740 10 client.0 _ll_register_callbacks cb 0x555bf232a480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2079270 0x555bf2461170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2079270 0x555bf2461170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41254/0 (socket says 172.21.15.81:41254) 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/2570835427 learned_addr learned my addr 172.21.15.81:0/2570835427 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff659c640 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf20503b0 0x555bf2444fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff5d9b640 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2058590 0x555bf2460c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf20503b0 msgr2=0x555bf2444fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff6d9d640 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf20503b0 0x555bf2444fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2058590 msgr2=0x555bf2460c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff6d9d640 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2058590 0x555bf2460c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff6d9d640 1 -- 172.21.15.81:0/2570835427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bf20e4850 con 0x555bf2079270 2024-06-29T14:37:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8ff6d9d640 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2079270 0x555bf2461170 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000e010 tx=0x7f8ff0005c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11523526131de1f server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8fd57fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2570835427 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0020040 con 0x555bf2079270 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2570835427 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7f8ff00693a0 con 0x555bf2079270 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.040+0000 7f90037e3740 1 -- 172.21.15.81:0/2570835427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ff0060900 con 0x555bf2079270 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2570835427 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8ff005cf50 con 0x555bf2079270 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f90037e3740 10 client.6690 fetch_fsmap learned FSMap version 34 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f90037e3740 1 -- 172.21.15.81:0/2570835427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x555bf215b6b0 con 0x555bf2079270 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2570835427 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ff0060900 con 0x555bf2079270 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2570835427 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 34) ==== 3502+0+0 (secure 0 0 0) 0x7f8ff0061b80 con 0x555bf2079270 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f90037e3740 10 client.6690 fetch_fsmap finished waiting for FSMap version 34 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f90037e3740 10 client.6690 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f90037e3740 20 client.6690 populate_metadata read hostname 'smithi081' 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f90037e3740 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:6835/2723173791,v1:172.21.15.167:6837/2723173791] conn(0x555bf2037e40 0x555bf247c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f90037e3740 4 client.6690 mds_command: new command op to 6567 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f90037e3740 1 -- 172.21.15.81:0/2570835427 --> [v2:172.21.15.167:6835/2723173791,v1:172.21.15.167:6837/2723173791] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bf2392580 con 0x555bf2037e40 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.041+0000 7f8ff659c640 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:6835/2723173791,v1:172.21.15.167:6837/2723173791] conn(0x555bf2037e40 0x555bf247c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.042+0000 7f8fd57fa640 10 client.6690 ms_handle_connect on v2:172.21.15.167:6835/2723173791 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.042+0000 7f8ff659c640 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:6835/2723173791,v1:172.21.15.167:6837/2723173791] conn(0x555bf2037e40 0x555bf247c2e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34afa6c89d1f88cc server_cookie=3426347d1fe7dbec in_seq=0 out_seq=0 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.042+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2570835427 <== mds.0 v2:172.21.15.167:6835/2723173791 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x555bf2392580 con 0x555bf2037e40 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.042+0000 7f8fd57fa640 10 client.6690 handle_command_reply: tid=0 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 client.6690 shutdown 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 -- 172.21.15.81:0/2570835427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2079270 msgr2=0x555bf2461170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2079270 0x555bf2461170 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000e010 tx=0x7f8ff0005c20 comp rx=0 tx=0).stop 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 -- 172.21.15.81:0/2570835427 shutdown_connections 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:6835/2723173791,v1:172.21.15.167:6837/2723173791] conn(0x555bf2037e40 0x555bf247c2e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf2079270 0x555bf2461170 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2058590 0x555bf2460c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f8fd57fa640 0 client.6690 ms_handle_reset on v2:172.21.15.167:6835/2723173791 2024-06-29T14:37:27.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 --2- 172.21.15.81:0/2570835427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf20503b0 0x555bf2444fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 -- 172.21.15.81:0/2570835427 >> 172.21.15.81:0/2570835427 conn(0x555bf206d3c0 msgr2=0x555bf244a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 -- 172.21.15.81:0/2570835427 shutdown_connections 2024-06-29T14:37:27.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 1 -- 172.21.15.81:0/2570835427 wait complete. 2024-06-29T14:37:27.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.043+0000 7f90037e3740 20 client.6690 trim_cache size 0 max 16384 2024-06-29T14:37:27.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.062+0000 7f90037e3740 1 Processor -- start 2024-06-29T14:37:27.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.062+0000 7f90037e3740 1 -- start start 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f90037e3740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2037e40 0x555bf248c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f90037e3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf2424c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f90037e3740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2058590 0x555bf24232e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f90037e3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81669d0 con 0x555bf20503b0 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f90037e3740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff814a900 con 0x555bf2037e40 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f90037e3740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8ff8166340 con 0x555bf2058590 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f8ff5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf2424c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f8ff5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf2424c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41264/0 (socket says 172.21.15.81:41264) 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f8ff5d9b640 1 -- 172.21.15.81:0/2616902012 learned_addr learned my addr 172.21.15.81:0/2616902012 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f8ff5d9b640 1 -- 172.21.15.81:0/2616902012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2058590 msgr2=0x555bf24232e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f8ff5d9b640 1 --2- 172.21.15.81:0/2616902012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2058590 0x555bf24232e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f8ff5d9b640 1 -- 172.21.15.81:0/2616902012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2037e40 msgr2=0x555bf248c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f8ff5d9b640 1 --2- 172.21.15.81:0/2616902012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2037e40 0x555bf248c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f8ff5d9b640 1 -- 172.21.15.81:0/2616902012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bf241f3e0 con 0x555bf20503b0 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.063+0000 7f8ff5d9b640 1 --2- 172.21.15.81:0/2616902012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf2424c50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8ff81f5fd0 tx=0x7f8fec003ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74197bfa9a038bb server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f8fd77fe640 1 -- 172.21.15.81:0/2616902012 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00ba60 con 0x555bf20503b0 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f8fd77fe640 1 -- 172.21.15.81:0/2616902012 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fec00bcf0 con 0x555bf20503b0 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f8fd77fe640 1 -- 172.21.15.81:0/2616902012 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec005240 con 0x555bf20503b0 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f90037e3740 1 -- 172.21.15.81:0/2616902012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 msgr2=0x555bf2424c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f90037e3740 1 --2- 172.21.15.81:0/2616902012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf2424c50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f8ff81f5fd0 tx=0x7f8fec003ad0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f90037e3740 1 -- 172.21.15.81:0/2616902012 shutdown_connections 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f90037e3740 1 --2- 172.21.15.81:0/2616902012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2058590 0x555bf24232e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f90037e3740 1 --2- 172.21.15.81:0/2616902012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf2424c50 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f90037e3740 1 --2- 172.21.15.81:0/2616902012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2037e40 0x555bf248c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f90037e3740 1 -- 172.21.15.81:0/2616902012 >> 172.21.15.81:0/2616902012 conn(0x555bf206d3c0 msgr2=0x555bf2415260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f90037e3740 1 -- 172.21.15.81:0/2616902012 shutdown_connections 2024-06-29T14:37:27.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.064+0000 7f90037e3740 1 -- 172.21.15.81:0/2616902012 wait complete. 2024-06-29T14:37:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.065+0000 7f90037e3740 1 -- start start 2024-06-29T14:37:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.065+0000 7f90037e3740 1 Processor -- start 2024-06-29T14:37:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f90037e3740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2037e40 0x555bf244a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f90037e3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf24e58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f90037e3740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2058590 0x555bf24e94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f90037e3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555bf231dae0 con 0x555bf20503b0 2024-06-29T14:37:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f90037e3740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x555bf2383c10 con 0x555bf2058590 2024-06-29T14:37:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f90037e3740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x555bf24618a0 con 0x555bf2037e40 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f90037e3740 10 client.0 _ll_register_callbacks cb 0x7f8ff0078070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf24e58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf24e58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41278/0 (socket says 172.21.15.81:41278) 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff5d9b640 1 -- 172.21.15.81:0/2964640603 learned_addr learned my addr 172.21.15.81:0/2964640603 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff659c640 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2037e40 0x555bf244a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff6d9d640 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2058590 0x555bf24e94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff5d9b640 1 -- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2037e40 msgr2=0x555bf244a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff5d9b640 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2037e40 0x555bf244a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff5d9b640 1 -- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2058590 msgr2=0x555bf24e94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff5d9b640 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2058590 0x555bf24e94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff5d9b640 1 -- 172.21.15.81:0/2964640603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bf2420100 con 0x555bf20503b0 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8ff5d9b640 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf24e58f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x555bf2162360 tx=0x7f8fec00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e759a52dbf972aa server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.067+0000 7f8fd57fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2964640603 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec029060 con 0x555bf20503b0 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2964640603 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7f8fec005230 con 0x555bf20503b0 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f90037e3740 1 -- 172.21.15.81:0/2964640603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fec00bcf0 con 0x555bf20503b0 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2964640603 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8fec02f050 con 0x555bf20503b0 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f90037e3740 10 client.6696 fetch_fsmap learned FSMap version 34 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f90037e3740 1 -- 172.21.15.81:0/2964640603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x555bf244e360 con 0x555bf20503b0 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2964640603 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fec00bcf0 con 0x555bf20503b0 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2964640603 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 34) ==== 3502+0+0 (secure 0 0 0) 0x7f8fec032020 con 0x555bf20503b0 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f90037e3740 10 client.6696 fetch_fsmap finished waiting for FSMap version 34 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f90037e3740 10 client.6696 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f90037e3740 20 client.6696 populate_metadata read hostname 'smithi081' 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f90037e3740 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:6835/2723173791,v1:172.21.15.167:6837/2723173791] conn(0x555bf200d280 0x555bf2419450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f90037e3740 4 client.6696 mds_command: new command op to 6567 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:37:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f90037e3740 1 -- 172.21.15.81:0/2964640603 --> [v2:172.21.15.167:6835/2723173791,v1:172.21.15.167:6837/2723173791] -- command(tid 0: {"prefix": "status"}) -- 0x555bf2464570 con 0x555bf200d280 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.068+0000 7f8ff659c640 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:6835/2723173791,v1:172.21.15.167:6837/2723173791] conn(0x555bf200d280 0x555bf2419450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f8ff659c640 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:6835/2723173791,v1:172.21.15.167:6837/2723173791] conn(0x555bf200d280 0x555bf2419450 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb84232bd2347c41 server_cookie=d20a336fef4a204 in_seq=0 out_seq=0 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f8fd57fa640 10 client.6696 ms_handle_connect on v2:172.21.15.167:6835/2723173791 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f8fd57fa640 1 -- 172.21.15.81:0/2964640603 <== mds.0 v2:172.21.15.167:6835/2723173791 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x555bf2464570 con 0x555bf200d280 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f8fd57fa640 10 client.6696 handle_command_reply: tid=0 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 client.6696 shutdown 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 -- 172.21.15.81:0/2964640603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 msgr2=0x555bf24e58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf24e58f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x555bf2162360 tx=0x7f8fec00ba90 comp rx=0 tx=0).stop 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 -- 172.21.15.81:0/2964640603 shutdown_connections 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:6835/2723173791,v1:172.21.15.167:6837/2723173791] conn(0x555bf200d280 0x555bf2419450 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x555bf2058590 0x555bf24e94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf20503b0 0x555bf24e58f0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 --2- 172.21.15.81:0/2964640603 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x555bf2037e40 0x555bf244a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f8fd57fa640 0 client.6696 ms_handle_reset on v2:172.21.15.167:6835/2723173791 2024-06-29T14:37:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 -- 172.21.15.81:0/2964640603 >> 172.21.15.81:0/2964640603 conn(0x555bf206d3c0 msgr2=0x555bf248c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 -- 172.21.15.81:0/2964640603 shutdown_connections 2024-06-29T14:37:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 1 -- 172.21.15.81:0/2964640603 wait complete. 2024-06-29T14:37:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.069+0000 7f90037e3740 20 client.6696 trim_cache size 0 max 16384 2024-06-29T14:37:27.073 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:37:27.073 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T14:37:27.073 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T14:37:27.073 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 6567, 2024-06-29T14:37:27.073 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T14:37:27.074 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T14:37:27.074 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T14:37:27.074 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7428131990000004, 2024-06-29T14:37:27.074 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 34, 2024-06-29T14:37:27.074 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 78, 2024-06-29T14:37:27.074 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 78, 2024-06-29T14:37:27.074 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.418445451 2024-06-29T14:37:27.074 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:37:27.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 -- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8fd8076420 msgr2=0x7f8fd8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8fd8076420 0x7f8fd8078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8035000 tx=0x7f8fe8004ab0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 -- 172.21.15.81:0/1181976874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ff80b4430 msgr2=0x7f8ff8148570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ff80b4430 0x7f8ff8148570 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000d5e0 tx=0x7f8ff000daa0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 -- 172.21.15.81:0/1181976874 shutdown_connections 2024-06-29T14:37:27.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f8fd8076420 0x7f8fd8078930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ff80b4430 0x7f8ff8148570 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8ff80b35b0 0x7f8ff8148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 --2- 172.21.15.81:0/1181976874 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8ff806f660 0x7f8ff8150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 -- 172.21.15.81:0/1181976874 >> 172.21.15.81:0/1181976874 conn(0x7f8ff806ccc0 msgr2=0x7f8ff8071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 -- 172.21.15.81:0/1181976874 shutdown_connections 2024-06-29T14:37:27.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.070+0000 7f8ffc847640 1 -- 172.21.15.81:0/1181976874 wait complete. 2024-06-29T14:37:27.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:37:27.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.195+0000 7f8039bcb640 1 -- 172.21.15.81:0/1580501193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f801c003a00 msgr2=0x7f801c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.195+0000 7f8039bcb640 1 --2- 172.21.15.81:0/1580501193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f801c003a00 0x7f801c003e30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f8024001740 tx=0x7f801c005830 comp rx=0 tx=0).stop 2024-06-29T14:37:27.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.195+0000 7f8039bcb640 1 -- 172.21.15.81:0/1580501193 shutdown_connections 2024-06-29T14:37:27.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.195+0000 7f8039bcb640 1 --2- 172.21.15.81:0/1580501193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f801c003a00 0x7f801c003e30 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.195+0000 7f8039bcb640 1 -- 172.21.15.81:0/1580501193 >> 172.21.15.81:0/1580501193 conn(0x7f80340d0930 msgr2=0x7f80340d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.195+0000 7f8039bcb640 1 -- 172.21.15.81:0/1580501193 shutdown_connections 2024-06-29T14:37:27.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.195+0000 7f8039bcb640 1 -- 172.21.15.81:0/1580501193 wait complete. 2024-06-29T14:37:27.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.197+0000 7f8039bcb640 1 Processor -- start 2024-06-29T14:37:27.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.197+0000 7f8039bcb640 1 -- start start 2024-06-29T14:37:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.198+0000 7f8039bcb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f801c003a00 0x7f80340ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.198+0000 7f8039bcb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80340ae960 0x7f80341537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.198+0000 7f8039bcb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8034152b50 0x7f8034153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.198+0000 7f8039bcb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f803415b370 con 0x7f801c003a00 2024-06-29T14:37:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.198+0000 7f8039bcb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f803415b1f0 con 0x7f8034152b50 2024-06-29T14:37:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.198+0000 7f8039bcb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f803406ef90 con 0x7f80340ae960 2024-06-29T14:37:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.198+0000 7f80337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f801c003a00 0x7f80340ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.198+0000 7f80337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f801c003a00 0x7f80340ae420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41308/0 (socket says 172.21.15.81:41308) 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.198+0000 7f80337fe640 1 -- 172.21.15.81:0/2696623588 learned_addr learned my addr 172.21.15.81:0/2696623588 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f80337fe640 1 -- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80340ae960 msgr2=0x7f80341537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f8032ffd640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80340ae960 0x7f80341537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f8033fff640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8034152b50 0x7f8034153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f80337fe640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80340ae960 0x7f80341537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f80337fe640 1 -- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8034152b50 msgr2=0x7f8034153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f80337fe640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8034152b50 0x7f8034153ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f80337fe640 1 -- 172.21.15.81:0/2696623588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803415c270 con 0x7f801c003a00 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f80337fe640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f801c003a00 0x7f80340ae420 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f8024001860 tx=0x7f8024006130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f801bfff640 1 -- 172.21.15.81:0/2696623588 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8024040030 con 0x7f801c003a00 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f801bfff640 1 -- 172.21.15.81:0/2696623588 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8024004640 con 0x7f801c003a00 2024-06-29T14:37:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f8039bcb640 1 -- 172.21.15.81:0/2696623588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f803415c550 con 0x7f801c003a00 2024-06-29T14:37:27.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f801bfff640 1 -- 172.21.15.81:0/2696623588 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8024037040 con 0x7f801c003a00 2024-06-29T14:37:27.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.199+0000 7f8039bcb640 1 -- 172.21.15.81:0/2696623588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f803415cb10 con 0x7f801c003a00 2024-06-29T14:37:27.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.200+0000 7f8039bcb640 1 -- 172.21.15.81:0/2696623588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8034071680 con 0x7f801c003a00 2024-06-29T14:37:27.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.201+0000 7f801bfff640 1 -- 172.21.15.81:0/2696623588 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f8024043070 con 0x7f801c003a00 2024-06-29T14:37:27.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.201+0000 7f801bfff640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f80200763f0 0x7f8020078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:27.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.201+0000 7f801bfff640 1 -- 172.21.15.81:0/2696623588 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7f80240b72b0 con 0x7f801c003a00 2024-06-29T14:37:27.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.202+0000 7f8032ffd640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f80200763f0 0x7f8020078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:27.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.202+0000 7f8032ffd640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f80200763f0 0x7f8020078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f80341ef990 tx=0x7f8028005f90 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:27.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.206+0000 7f801bfff640 1 -- 172.21.15.81:0/2696623588 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8034071680 con 0x7f801c003a00 2024-06-29T14:37:27.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.427+0000 7f8039bcb640 1 -- 172.21.15.81:0/2696623588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f80340d06c0 con 0x7f801c003a00 2024-06-29T14:37:27.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.427+0000 7f801bfff640 1 -- 172.21.15.81:0/2696623588 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v34) ==== 53+0+73 (secure 0 0 0) 0x7f80340d06c0 con 0x7f801c003a00 2024-06-29T14:37:27.426 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:37:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.429+0000 7f8039bcb640 1 -- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f80200763f0 msgr2=0x7f8020078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.429+0000 7f8039bcb640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f80200763f0 0x7f8020078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f80341ef990 tx=0x7f8028005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.429+0000 7f8039bcb640 1 -- 172.21.15.81:0/2696623588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f801c003a00 msgr2=0x7f80340ae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.429+0000 7f8039bcb640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f801c003a00 0x7f80340ae420 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f8024001860 tx=0x7f8024006130 comp rx=0 tx=0).stop 2024-06-29T14:37:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.429+0000 7f8039bcb640 1 -- 172.21.15.81:0/2696623588 shutdown_connections 2024-06-29T14:37:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.429+0000 7f8039bcb640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f80200763f0 0x7f8020078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.429+0000 7f8039bcb640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8034152b50 0x7f8034153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.429+0000 7f8039bcb640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f80340ae960 0x7f80341537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.429+0000 7f8039bcb640 1 --2- 172.21.15.81:0/2696623588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f801c003a00 0x7f80340ae420 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:27.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.429+0000 7f8039bcb640 1 -- 172.21.15.81:0/2696623588 >> 172.21.15.81:0/2696623588 conn(0x7f80340d0930 msgr2=0x7f8034068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:27.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.430+0000 7f8039bcb640 1 -- 172.21.15.81:0/2696623588 shutdown_connections 2024-06-29T14:37:27.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27.430+0000 7f8039bcb640 1 -- 172.21.15.81:0/2696623588 wait complete. 2024-06-29T14:37:27.439 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:37:27.440 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:37:27.440 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T14:37:27.440 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:37:27.440 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:27.440 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:37:27.440 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:37:27.440 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:37:27.440 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:27.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.441 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T14:37:27.456 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:37:27.456 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:37:27.456 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:37:27.457 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 41872sec preferred_lft 41872sec 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T14:37:27.458 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.459 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:37:27.459 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:37:27.459 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T14:37:27.459 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:27.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.459 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:37:27.520 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T14:37:27.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.522 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T14:37:27.579 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T14:37:27.579 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:37:27.579 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:37:27.579 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:37:27.579 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:37:27.580 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:37:27.580 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:37:27.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:37:27.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:37:27.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:37:27.580 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:37:27.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:37:27.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:37:27.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.706 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:37:27.721 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:37:27.721 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:37:27.722 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:27.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.722 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T14:37:27.787 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:37:27.787 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:27.843 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:27.843 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:37:27.843 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:27.898 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T14:37:27.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:27.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:37:27.991 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:37:27.991 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:37:28.000 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:37:28.000 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:37:28.000 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:37:28.000 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:37:28.000 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:37:28.000 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:37:28.001 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:37:28.001 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:37:28.001 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:37:28.001 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:37:28.003 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:37:28.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:28.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:28.029 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:37:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:37:28.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:37:28.132 INFO:teuthology.orchestra.run.smithi081.stdout:client6705 2024-06-29T14:37:28.132 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:28.133 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:28.149 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:37:28.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.305+0000 7fa0ef511640 1 -- 172.21.15.81:0/1283598270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e80ce920 msgr2=0x7fa0e80cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.305+0000 7fa0ef511640 1 --2- 172.21.15.81:0/1283598270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e80ce920 0x7fa0e80cd480 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8072aa0 tx=0x7fa0e002a950 comp rx=0 tx=0).stop 2024-06-29T14:37:28.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.305+0000 7fa0ef511640 1 -- 172.21.15.81:0/1283598270 shutdown_connections 2024-06-29T14:37:28.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.305+0000 7fa0ef511640 1 --2- 172.21.15.81:0/1283598270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e80ce920 0x7fa0e80cd480 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.305+0000 7fa0ef511640 1 --2- 172.21.15.81:0/1283598270 >> v2:172.21.15.167:3300/0 conn(0x7fa0e80b6eb0 0x7fa0e80d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.305+0000 7fa0ef511640 1 -- 172.21.15.81:0/1283598270 >> 172.21.15.81:0/1283598270 conn(0x7fa0e80b3510 msgr2=0x7fa0e80b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:28.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.305+0000 7fa0ef511640 1 -- 172.21.15.81:0/1283598270 shutdown_connections 2024-06-29T14:37:28.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.305+0000 7fa0ef511640 1 -- 172.21.15.81:0/1283598270 wait complete. 2024-06-29T14:37:28.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.307+0000 7fa0ef511640 1 Processor -- start 2024-06-29T14:37:28.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.307+0000 7fa0ef511640 1 -- start start 2024-06-29T14:37:28.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ef511640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e80b6eb0 0x7fa0e81f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ef511640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0e80ce510 0x7fa0e81dbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ef511640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e80ce920 0x7fa0e80afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ef511640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e80b4f20 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ef511640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e80b29c0 con 0x7fa0e80ce920 2024-06-29T14:37:28.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ef511640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa0e80b50a0 con 0x7fa0e80ce510 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ed2ad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e80b6eb0 0x7fa0e81f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ed2ad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e80b6eb0 0x7fa0e81f30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41324/0 (socket says 172.21.15.81:41324) 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ed2ad640 1 -- 172.21.15.81:0/3327457341 learned_addr learned my addr 172.21.15.81:0/3327457341 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0edaae640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e80ce920 0x7fa0e80afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ecaac640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0e80ce510 0x7fa0e81dbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ed2ad640 1 -- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0e80ce510 msgr2=0x7fa0e81dbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ed2ad640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0e80ce510 0x7fa0e81dbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ed2ad640 1 -- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e80ce920 msgr2=0x7fa0e80afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ed2ad640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e80ce920 0x7fa0e80afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0ed2ad640 1 -- 172.21.15.81:0/3327457341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e80aeb60 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.308+0000 7fa0edaae640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e80ce920 0x7fa0e80afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.309+0000 7fa0ed2ad640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e80b6eb0 0x7fa0e81f30b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa0e400c9a0 tx=0x7fa0e400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:28.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.309+0000 7fa0ddffb640 1 -- 172.21.15.81:0/3327457341 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4007bb0 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.309+0000 7fa0ddffb640 1 -- 172.21.15.81:0/3327457341 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0e4007e40 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.309+0000 7fa0ef511640 1 -- 172.21.15.81:0/3327457341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e80aee40 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.309+0000 7fa0ddffb640 1 -- 172.21.15.81:0/3327457341 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4005280 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.309+0000 7fa0ef511640 1 -- 172.21.15.81:0/3327457341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e80af400 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.310+0000 7fa0ef511640 1 -- 172.21.15.81:0/3327457341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0e80d02d0 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.310+0000 7fa0ddffb640 1 -- 172.21.15.81:0/3327457341 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7fa0e4005420 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.311+0000 7fa0ddffb640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa0cc0763f0 0x7fa0cc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.311+0000 7fa0ddffb640 1 -- 172.21.15.81:0/3327457341 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7fa0e4097dd0 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.311+0000 7fa0ecaac640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa0cc0763f0 0x7fa0cc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.312+0000 7fa0ecaac640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa0cc0763f0 0x7fa0cc078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8006fd0 tx=0x7fa0d8008040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:28.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.316+0000 7fa0ddffb640 1 -- 172.21.15.81:0/3327457341 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa0e4014030 con 0x7fa0e80b6eb0 2024-06-29T14:37:28.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.523+0000 7fa0ef511640 1 -- 172.21.15.81:0/3327457341 --> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fa0e80cdf00 con 0x7fa0cc0763f0 2024-06-29T14:37:28.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.526+0000 7fa0ddffb640 1 -- 172.21.15.81:0/3327457341 <== mgr.5860 v2:172.21.15.167:6832/3693496302 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fa0e80cdf00 con 0x7fa0cc0763f0 2024-06-29T14:37:28.524 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:37:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T14:37:28.524 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:28.524 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:37:28.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 -- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa0cc0763f0 msgr2=0x7fa0cc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa0cc0763f0 0x7fa0cc078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8006fd0 tx=0x7fa0d8008040 comp rx=0 tx=0).stop 2024-06-29T14:37:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 -- 172.21.15.81:0/3327457341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e80b6eb0 msgr2=0x7fa0e81f30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e80b6eb0 0x7fa0e81f30b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa0e400c9a0 tx=0x7fa0e400ce60 comp rx=0 tx=0).stop 2024-06-29T14:37:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 -- 172.21.15.81:0/3327457341 shutdown_connections 2024-06-29T14:37:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fa0cc0763f0 0x7fa0cc078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e80ce920 0x7fa0e80afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0e80ce510 0x7fa0e81dbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 --2- 172.21.15.81:0/3327457341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e80b6eb0 0x7fa0e81f30b0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 -- 172.21.15.81:0/3327457341 >> 172.21.15.81:0/3327457341 conn(0x7fa0e80b3510 msgr2=0x7fa0e80b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 -- 172.21.15.81:0/3327457341 shutdown_connections 2024-06-29T14:37:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.528+0000 7fa0ef511640 1 -- 172.21.15.81:0/3327457341 wait complete. 2024-06-29T14:37:28.537 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:37:28.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.656+0000 7f17322c1640 1 -- 172.21.15.81:0/2151934448 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f172c06f6a0 msgr2=0x7f172c15aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.656+0000 7f17322c1640 1 --2- 172.21.15.81:0/2151934448 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f172c06f6a0 0x7f172c15aed0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1720005fd0 tx=0x7f172002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.656+0000 7f17322c1640 1 -- 172.21.15.81:0/2151934448 shutdown_connections 2024-06-29T14:37:28.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.656+0000 7f17322c1640 1 --2- 172.21.15.81:0/2151934448 >> v2:172.21.15.167:3300/0 conn(0x7f172c0b3960 0x7f172c0b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.656+0000 7f17322c1640 1 --2- 172.21.15.81:0/2151934448 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f172c06f6a0 0x7f172c15aed0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.656+0000 7f17322c1640 1 -- 172.21.15.81:0/2151934448 >> 172.21.15.81:0/2151934448 conn(0x7f172c0d0930 msgr2=0x7f172c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:28.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.656+0000 7f17322c1640 1 -- 172.21.15.81:0/2151934448 shutdown_connections 2024-06-29T14:37:28.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.656+0000 7f17322c1640 1 -- 172.21.15.81:0/2151934448 wait complete. 2024-06-29T14:37:28.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.657+0000 7f17322c1640 1 Processor -- start 2024-06-29T14:37:28.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.658+0000 7f17322c1640 1 -- start start 2024-06-29T14:37:28.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f17322c1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f172c06f6a0 0x7f172c1f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f17322c1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f172c0b3960 0x7f172c0aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f17322c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172c0b57b0 0x7f172c1518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f17322c1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f172c0b4640 con 0x7f172c0b57b0 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f17322c1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f172c0b44c0 con 0x7f172c06f6a0 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f17322c1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f172c0b47c0 con 0x7f172c0b3960 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f173085e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172c0b57b0 0x7f172c1518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f173085e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172c0b57b0 0x7f172c1518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41330/0 (socket says 172.21.15.81:41330) 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f173085e640 1 -- 172.21.15.81:0/1905906443 learned_addr learned my addr 172.21.15.81:0/1905906443 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f172b7fe640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f172c0b3960 0x7f172c0aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f172bfff640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f172c06f6a0 0x7f172c1f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f173085e640 1 -- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f172c0b3960 msgr2=0x7f172c0aa9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f173085e640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f172c0b3960 0x7f172c0aa9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f173085e640 1 -- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f172c06f6a0 msgr2=0x7f172c1f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f173085e640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f172c06f6a0 0x7f172c1f1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.659+0000 7f173085e640 1 -- 172.21.15.81:0/1905906443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f172c1f2440 con 0x7f172c0b57b0 2024-06-29T14:37:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.660+0000 7f173085e640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172c0b57b0 0x7f172c1518e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f172400ec20 tx=0x7f172400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.660+0000 7f1728ff9640 1 -- 172.21.15.81:0/1905906443 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724003b70 con 0x7f172c0b57b0 2024-06-29T14:37:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.660+0000 7f1728ff9640 1 -- 172.21.15.81:0/1905906443 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1724003e00 con 0x7f172c0b57b0 2024-06-29T14:37:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.660+0000 7f1728ff9640 1 -- 172.21.15.81:0/1905906443 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724004280 con 0x7f172c0b57b0 2024-06-29T14:37:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.660+0000 7f17322c1640 1 -- 172.21.15.81:0/1905906443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f172c1f2720 con 0x7f172c0b57b0 2024-06-29T14:37:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.660+0000 7f17322c1640 1 -- 172.21.15.81:0/1905906443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f172c151f90 con 0x7f172c0b57b0 2024-06-29T14:37:28.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.661+0000 7f17322c1640 1 -- 172.21.15.81:0/1905906443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f172c06e2a0 con 0x7f172c0b57b0 2024-06-29T14:37:28.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.661+0000 7f1728ff9640 1 -- 172.21.15.81:0/1905906443 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 30) ==== 98303+0+0 (secure 0 0 0) 0x7f1724004420 con 0x7f172c0b57b0 2024-06-29T14:37:28.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.662+0000 7f1728ff9640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f17180763f0 0x7f1718078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:28.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.662+0000 7f1728ff9640 1 -- 172.21.15.81:0/1905906443 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7f1724098740 con 0x7f172c0b57b0 2024-06-29T14:37:28.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.662+0000 7f172bfff640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f17180763f0 0x7f1718078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:28.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.663+0000 7f172bfff640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f17180763f0 0x7f1718078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f171c001720 tx=0x7f171c009040 comp rx=0 tx=0).ready entity=mgr.5860 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:28.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.666+0000 7f1728ff9640 1 -- 172.21.15.81:0/1905906443 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f172c06e2a0 con 0x7f172c0b57b0 2024-06-29T14:37:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:28.893+0000 7f17322c1640 1 -- 172.21.15.81:0/1905906443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f172c1f51e0 con 0x7f172c0b57b0 2024-06-29T14:37:29.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.365+0000 7f1728ff9640 1 -- 172.21.15.81:0/1905906443 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v31) ==== 92+0+0 (secure 0 0 0) 0x7f1724061740 con 0x7f172c0b57b0 2024-06-29T14:37:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.367+0000 7f17322c1640 1 -- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f17180763f0 msgr2=0x7f1718078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.367+0000 7f17322c1640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f17180763f0 0x7f1718078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f171c001720 tx=0x7f171c009040 comp rx=0 tx=0).stop 2024-06-29T14:37:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.367+0000 7f17322c1640 1 -- 172.21.15.81:0/1905906443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172c0b57b0 msgr2=0x7f172c1518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.367+0000 7f17322c1640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172c0b57b0 0x7f172c1518e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f172400ec20 tx=0x7f172400c6a0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.368+0000 7f17322c1640 1 -- 172.21.15.81:0/1905906443 shutdown_connections 2024-06-29T14:37:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.368+0000 7f17322c1640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f17180763f0 0x7f1718078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.368+0000 7f17322c1640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172c0b57b0 0x7f172c1518e0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.368+0000 7f17322c1640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f172c0b3960 0x7f172c0aa9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.368+0000 7f17322c1640 1 --2- 172.21.15.81:0/1905906443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f172c06f6a0 0x7f172c1f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.368+0000 7f17322c1640 1 -- 172.21.15.81:0/1905906443 >> 172.21.15.81:0/1905906443 conn(0x7f172c0d0930 msgr2=0x7f172c0cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:29.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.369+0000 7f17322c1640 1 -- 172.21.15.81:0/1905906443 shutdown_connections 2024-06-29T14:37:29.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.369+0000 7f17322c1640 1 -- 172.21.15.81:0/1905906443 wait complete. 2024-06-29T14:37:29.379 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.504+0000 7fd8ee447640 1 -- 172.21.15.81:0/4116543165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8d4003a00 msgr2=0x7fd8d4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.504+0000 7fd8ee447640 1 --2- 172.21.15.81:0/4116543165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8d4003a00 0x7fd8d4003e70 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800c4a0 tx=0x7fd8d4005830 comp rx=0 tx=0).stop 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.504+0000 7fd8ec9e4640 1 -- 172.21.15.81:0/4116543165 reap_dead start 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.504+0000 7fd8ee447640 1 -- 172.21.15.81:0/4116543165 shutdown_connections 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.504+0000 7fd8ee447640 1 -- 172.21.15.81:0/4116543165 >> 172.21.15.81:0/4116543165 conn(0x7fd8e80b3680 msgr2=0x7fd8e80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:29.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.504+0000 7fd8ee447640 1 -- 172.21.15.81:0/4116543165 shutdown_connections 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.504+0000 7fd8ee447640 1 -- 172.21.15.81:0/4116543165 wait complete. 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.506+0000 7fd8ee447640 1 Processor -- start 2024-06-29T14:37:29.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.506+0000 7fd8ee447640 1 -- start start 2024-06-29T14:37:29.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.507+0000 7fd8ee447640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8d4003a00 0x7fd8e80cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8ee447640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8e806de10 0x7fd8e806ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8ee447640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8e81f8ac0 0x7fd8e81f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8ee447640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e80b56e0 con 0x7fd8d4003a00 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8ee447640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e80b5560 con 0x7fd8e806de10 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8ee447640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e80b5860 con 0x7fd8e81f8ac0 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8d4003a00 0x7fd8e80cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8d4003a00 0x7fd8e80cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41358/0 (socket says 172.21.15.81:41358) 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e7fff640 1 -- 172.21.15.81:0/2642155590 learned_addr learned my addr 172.21.15.81:0/2642155590 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e77fe640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8e806de10 0x7fd8e806ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8ec9e4640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8e81f8ac0 0x7fd8e81f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e7fff640 1 -- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8e81f8ac0 msgr2=0x7fd8e81f8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e7fff640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8e81f8ac0 0x7fd8e81f8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e7fff640 1 -- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8e806de10 msgr2=0x7fd8e806ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e7fff640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8e806de10 0x7fd8e806ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e7fff640 1 -- 172.21.15.81:0/2642155590 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e806e220 con 0x7fd8d4003a00 2024-06-29T14:37:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e7fff640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8d4003a00 0x7fd8e80cb990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8001b30 tx=0x7fd8d8006290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e4ff9640 1 -- 172.21.15.81:0/2642155590 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8042070 con 0x7fd8d4003a00 2024-06-29T14:37:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e4ff9640 1 -- 172.21.15.81:0/2642155590 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8d8001180 con 0x7fd8d4003a00 2024-06-29T14:37:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8ee447640 1 -- 172.21.15.81:0/2642155590 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e81f54f0 con 0x7fd8d4003a00 2024-06-29T14:37:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.508+0000 7fd8e4ff9640 1 -- 172.21.15.81:0/2642155590 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8d802b8a0 con 0x7fd8d4003a00 2024-06-29T14:37:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.509+0000 7fd8ee447640 1 -- 172.21.15.81:0/2642155590 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e81f5ab0 con 0x7fd8d4003a00 2024-06-29T14:37:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.510+0000 7fd8ee447640 1 -- 172.21.15.81:0/2642155590 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e80d0640 con 0x7fd8d4003a00 2024-06-29T14:37:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.510+0000 7fd8e4ff9640 1 -- 172.21.15.81:0/2642155590 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 31) ==== 98320+0+0 (secure 0 0 0) 0x7fd8d802ba40 con 0x7fd8d4003a00 2024-06-29T14:37:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.511+0000 7fd8e4ff9640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd8c8076440 0x7fd8c8078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.511+0000 7fd8e4ff9640 1 -- 172.21.15.81:0/2642155590 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7fd8d80b71b0 con 0x7fd8d4003a00 2024-06-29T14:37:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.511+0000 7fd8e77fe640 1 -- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd8c8076440 msgr2=0x7fd8c8078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3693496302 2024-06-29T14:37:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.511+0000 7fd8e77fe640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd8c8076440 0x7fd8c8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:29.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.515+0000 7fd8e4ff9640 1 -- 172.21.15.81:0/2642155590 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd8e80d0b80 con 0x7fd8d4003a00 2024-06-29T14:37:29.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.703+0000 7fd8ee447640 1 -- 172.21.15.81:0/2642155590 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fd8e81e8ab0 con 0x7fd8d4003a00 2024-06-29T14:37:29.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.707+0000 7fd8e4ff9640 1 -- 172.21.15.81:0/2642155590 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v37)=0 v37) ==== 138+0+0 (secure 0 0 0) 0x7fd8d8080360 con 0x7fd8d4003a00 2024-06-29T14:37:29.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 -- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd8c8076440 msgr2=0x7fd8c8078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:29.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd8c8076440 0x7fd8c8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 -- 172.21.15.81:0/2642155590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8d4003a00 msgr2=0x7fd8e80cb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8d4003a00 0x7fd8e80cb990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8001b30 tx=0x7fd8d8006290 comp rx=0 tx=0).stop 2024-06-29T14:37:29.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 -- 172.21.15.81:0/2642155590 shutdown_connections 2024-06-29T14:37:29.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd8c8076440 0x7fd8c8078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8e81f8ac0 0x7fd8e81f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8e806de10 0x7fd8e806ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 --2- 172.21.15.81:0/2642155590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8d4003a00 0x7fd8e80cb990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 -- 172.21.15.81:0/2642155590 >> 172.21.15.81:0/2642155590 conn(0x7fd8e80b3680 msgr2=0x7fd8e8070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 -- 172.21.15.81:0/2642155590 shutdown_connections 2024-06-29T14:37:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.709+0000 7fd8ee447640 1 -- 172.21.15.81:0/2642155590 wait complete. 2024-06-29T14:37:29.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:37:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.839+0000 7fd089cce640 1 -- 172.21.15.81:0/2500409012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0840b3200 msgr2=0x7fd0840cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.839+0000 7fd089cce640 1 --2- 172.21.15.81:0/2500409012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0840b3200 0x7fd0840cc7b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fd0840c5590 tx=0x7fd07402a560 comp rx=0 tx=0).stop 2024-06-29T14:37:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.839+0000 7fd089cce640 1 -- 172.21.15.81:0/2500409012 shutdown_connections 2024-06-29T14:37:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.839+0000 7fd089cce640 1 --2- 172.21.15.81:0/2500409012 >> v2:172.21.15.167:3300/0 conn(0x7fd0840d0940 0x7fd0840b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.839+0000 7fd089cce640 1 --2- 172.21.15.81:0/2500409012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0840b3200 0x7fd0840cc7b0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.839+0000 7fd089cce640 1 -- 172.21.15.81:0/2500409012 >> 172.21.15.81:0/2500409012 conn(0x7fd0840b3680 msgr2=0x7fd0840b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.839+0000 7fd089cce640 1 -- 172.21.15.81:0/2500409012 shutdown_connections 2024-06-29T14:37:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.839+0000 7fd089cce640 1 -- 172.21.15.81:0/2500409012 wait complete. 2024-06-29T14:37:29.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.841+0000 7fd089cce640 1 Processor -- start 2024-06-29T14:37:29.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.841+0000 7fd089cce640 1 -- start start 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd089cce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0840b3200 0x7fd0841cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd089cce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0840cccf0 0x7fd0841dce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd089cce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0841dae50 0x7fd0841dd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd089cce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0840b5bb0 con 0x7fd0840cccf0 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd089cce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd0840b5a30 con 0x7fd0841dae50 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd089cce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd0840b5d30 con 0x7fd0840b3200 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd07bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0840cccf0 0x7fd0841dce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd0837fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0840b3200 0x7fd0841cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd07bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0840cccf0 0x7fd0841dce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41382/0 (socket says 172.21.15.81:41382) 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd083fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0841dae50 0x7fd0841dd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.842+0000 7fd07bfff640 1 -- 172.21.15.81:0/3719920394 learned_addr learned my addr 172.21.15.81:0/3719920394 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd07bfff640 1 -- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0840b3200 msgr2=0x7fd0841cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd07bfff640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0840b3200 0x7fd0841cec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd07bfff640 1 -- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0841dae50 msgr2=0x7fd0841dd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd07bfff640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0841dae50 0x7fd0841dd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd07bfff640 1 -- 172.21.15.81:0/3719920394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0841e8110 con 0x7fd0840cccf0 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd07bfff640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0840cccf0 0x7fd0841dce40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c370 tx=0x7fd06c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd080ff9640 1 -- 172.21.15.81:0/3719920394 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c017070 con 0x7fd0840cccf0 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd080ff9640 1 -- 172.21.15.81:0/3719920394 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd06c0053c0 con 0x7fd0840cccf0 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd089cce640 1 -- 172.21.15.81:0/3719920394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0841ddb10 con 0x7fd0840cccf0 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd080ff9640 1 -- 172.21.15.81:0/3719920394 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c003850 con 0x7fd0840cccf0 2024-06-29T14:37:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.843+0000 7fd089cce640 1 -- 172.21.15.81:0/3719920394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0841de0d0 con 0x7fd0840cccf0 2024-06-29T14:37:29.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.844+0000 7fd089cce640 1 -- 172.21.15.81:0/3719920394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0840cc7b0 con 0x7fd0840cccf0 2024-06-29T14:37:29.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.846+0000 7fd080ff9640 1 -- 172.21.15.81:0/3719920394 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 31) ==== 98320+0+0 (secure 0 0 0) 0x7fd06c01a070 con 0x7fd0840cccf0 2024-06-29T14:37:29.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.846+0000 7fd080ff9640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd064076440 0x7fd064078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.846+0000 7fd0837fe640 1 -- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd064076440 msgr2=0x7fd064078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3693496302 2024-06-29T14:37:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.846+0000 7fd080ff9640 1 -- 172.21.15.81:0/3719920394 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7fd06c096c50 con 0x7fd0840cccf0 2024-06-29T14:37:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.846+0000 7fd0837fe640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd064076440 0x7fd064078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:29.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:29.850+0000 7fd080ff9640 1 -- 172.21.15.81:0/3719920394 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd06c09c020 con 0x7fd0840cccf0 2024-06-29T14:37:30.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.033+0000 7fd089cce640 1 -- 172.21.15.81:0/3719920394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fd0840b6a50 con 0x7fd0840cccf0 2024-06-29T14:37:30.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.039+0000 7fd080ff9640 1 -- 172.21.15.81:0/3719920394 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v38)=0 v38) ==== 133+0+0 (secure 0 0 0) 0x7fd06c05fe00 con 0x7fd0840cccf0 2024-06-29T14:37:30.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.041+0000 7fd089cce640 1 -- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd064076440 msgr2=0x7fd064078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:30.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd064076440 0x7fd064078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 -- 172.21.15.81:0/3719920394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0840cccf0 msgr2=0x7fd0841dce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0840cccf0 0x7fd0841dce40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c370 tx=0x7fd06c00c830 comp rx=0 tx=0).stop 2024-06-29T14:37:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 -- 172.21.15.81:0/3719920394 shutdown_connections 2024-06-29T14:37:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7fd064076440 0x7fd064078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0841dae50 0x7fd0841dd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0840cccf0 0x7fd0841dce40 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 --2- 172.21.15.81:0/3719920394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0840b3200 0x7fd0841cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 -- 172.21.15.81:0/3719920394 >> 172.21.15.81:0/3719920394 conn(0x7fd0840b3680 msgr2=0x7fd0841e8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 -- 172.21.15.81:0/3719920394 shutdown_connections 2024-06-29T14:37:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.042+0000 7fd089cce640 1 -- 172.21.15.81:0/3719920394 wait complete. 2024-06-29T14:37:30.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:30.051 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T14:37:30.071 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-06-29T14:37:30.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.231+0000 7f0577db9640 1 -- 172.21.15.81:0/3693194792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f057015b400 msgr2=0x7f057015b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:30.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.231+0000 7f0577db9640 1 --2- 172.21.15.81:0/3693194792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f057015b400 0x7f057015b810 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0570073c30 tx=0x7f056802aa30 comp rx=0 tx=0).stop 2024-06-29T14:37:30.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.231+0000 7f0577db9640 1 -- 172.21.15.81:0/3693194792 shutdown_connections 2024-06-29T14:37:30.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.231+0000 7f0577db9640 1 --2- 172.21.15.81:0/3693194792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f057015b400 0x7f057015b810 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.231+0000 7f0577db9640 1 --2- 172.21.15.81:0/3693194792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0570072f80 0x7f0570073390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.231+0000 7f0577db9640 1 --2- 172.21.15.81:0/3693194792 >> v2:172.21.15.167:3300/0 conn(0x7f05700b4800 0x7f0570144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.231+0000 7f0577db9640 1 -- 172.21.15.81:0/3693194792 >> 172.21.15.81:0/3693194792 conn(0x7f05700b2920 msgr2=0x7f05700b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.231+0000 7f0577db9640 1 -- 172.21.15.81:0/3693194792 shutdown_connections 2024-06-29T14:37:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.231+0000 7f0577db9640 1 -- 172.21.15.81:0/3693194792 wait complete. 2024-06-29T14:37:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.233+0000 7f0577db9640 1 Processor -- start 2024-06-29T14:37:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.233+0000 7f0577db9640 1 -- start start 2024-06-29T14:37:30.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.234+0000 7f0577db9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0570072f80 0x7f057015a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:30.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.234+0000 7f0577db9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05700b4800 0x7f05701f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:30.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.234+0000 7f0577db9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f057015b400 0x7f05701f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:30.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.234+0000 7f0577db9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f057006f140 con 0x7f057015b400 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.234+0000 7f0577db9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f057006efc0 con 0x7f0570072f80 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.234+0000 7f0577db9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f057006f2c0 con 0x7f05700b4800 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.234+0000 7f0576356640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f057015b400 0x7f05701f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0576356640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f057015b400 0x7f05701f3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57580/0 (socket says 172.21.15.81:57580) 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0576356640 1 -- 172.21.15.81:0/1537077391 learned_addr learned my addr 172.21.15.81:0/1537077391 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0575354640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05700b4800 0x7f05701f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0575b55640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0570072f80 0x7f057015a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0576356640 1 -- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05700b4800 msgr2=0x7f05701f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0576356640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05700b4800 0x7f05701f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0576356640 1 -- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0570072f80 msgr2=0x7f057015a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0576356640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0570072f80 0x7f057015a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0576356640 1 -- 172.21.15.81:0/1537077391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05700aaaf0 con 0x7f057015b400 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0575354640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05700b4800 0x7f05701f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0576356640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f057015b400 0x7f05701f3170 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0568009d60 tx=0x7f0568004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0568047070 con 0x7f057015b400 2024-06-29T14:37:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f056802f690 con 0x7f057015b400 2024-06-29T14:37:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0577db9640 1 -- 172.21.15.81:0/1537077391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05700aadd0 con 0x7f057015b400 2024-06-29T14:37:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0568033070 con 0x7f057015b400 2024-06-29T14:37:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.235+0000 7f0577db9640 1 -- 172.21.15.81:0/1537077391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05700ab390 con 0x7f057015b400 2024-06-29T14:37:30.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.236+0000 7f0577db9640 1 -- 172.21.15.81:0/1537077391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05700aa860 con 0x7f057015b400 2024-06-29T14:37:30.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.237+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 31) ==== 98320+0+0 (secure 0 0 0) 0x7f056804a070 con 0x7f057015b400 2024-06-29T14:37:30.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.238+0000 7f05667fc640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 0x7f0554078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:30.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.238+0000 7f0575b55640 1 -- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 msgr2=0x7f0554078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3693496302 2024-06-29T14:37:30.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.238+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5781+0+0 (secure 0 0 0) 0x7f05680bab30 con 0x7f057015b400 2024-06-29T14:37:30.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.238+0000 7f0575b55640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 0x7f0554078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:30.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.242+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0568083bd0 con 0x7f057015b400 2024-06-29T14:37:30.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.439+0000 7f0575b55640 1 -- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 msgr2=0x7f0554078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3693496302 2024-06-29T14:37:30.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.439+0000 7f0575b55640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 0x7f0554078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:30.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.455+0000 7f0577db9640 1 -- 172.21.15.81:0/1537077391 --> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f0570144590 con 0x7f0554076440 2024-06-29T14:37:30.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.840+0000 7f0575b55640 1 -- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 msgr2=0x7f0554078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3693496302 2024-06-29T14:37:30.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:30.840+0000 7f0575b55640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 0x7f0554078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:37:31.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:31.641+0000 7f0575b55640 1 -- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 msgr2=0x7f0554078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3693496302 2024-06-29T14:37:31.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:31.641+0000 7f0575b55640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 0x7f0554078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:37:32.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:33.000+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 32) ==== 97929+0+0 (secure 0 0 0) 0x7f0568036070 con 0x7f057015b400 2024-06-29T14:37:32.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:33.000+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 msgr2=0x7f0554078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:32.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:33.000+0000 7f05667fc640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/3693496302,v1:172.21.15.167:6833/3693496302] conn(0x7f0554076440 0x7f0554078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:33.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:33.163+0000 7f4538b77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:33.163+0000 7f4538b77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:33.163+0000 7f4538b77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:33.163+0000 7f4538b77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:33.163+0000 7f4538b77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:33.163+0000 7f4535b71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:33.163+0000 7f4535b71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:33.163+0000 7f4535b71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:33.163+0000 7f4535b71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:33.162 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:33.163+0000 7f4535b71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:34.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.003+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 33) ==== 98056+0+0 (secure 0 0 0) 0x7f056802d070 con 0x7f057015b400 2024-06-29T14:37:34.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.004+0000 7f05667fc640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f055407e420 0x7f0554080840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:34.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.004+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 --> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f0568049ec0 con 0x7f055407e420 2024-06-29T14:37:34.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.005+0000 7f0575b55640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f055407e420 0x7f0554080840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:34.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.008+0000 7f0575b55640 1 --2- 172.21.15.81:0/1537077391 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f055407e420 0x7f0554080840 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0560005fd0 tx=0x7f056000b2e0 comp rx=0 tx=0).ready entity=mgr.6217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:34.013 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:34.015+0000 7f453fb85640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-06-29T14:37:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.016+0000 7f05667fc640 1 -- 172.21.15.81:0/1537077391 <== mgr.6217 v2:172.21.15.167:6832/2940523935 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) ==== 49+0+0 (secure 0 0 0) 0x7f0568049ec0 con 0x7f055407e420 2024-06-29T14:37:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-06-29T14:37:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:37:34.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:34.020 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T14:37:34.039 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:37:34.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.200+0000 7f9c5c48f640 1 -- 172.21.15.81:0/323299638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c400051c0 msgr2=0x7f9c4000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.200+0000 7f9c5c48f640 1 --2- 172.21.15.81:0/323299638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c400051c0 0x7f9c4000b9c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c001e10 tx=0x7f9c40005830 comp rx=0 tx=0).stop 2024-06-29T14:37:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.200+0000 7f9c5c48f640 1 -- 172.21.15.81:0/323299638 shutdown_connections 2024-06-29T14:37:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.200+0000 7f9c5c48f640 1 --2- 172.21.15.81:0/323299638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c400051c0 0x7f9c4000b9c0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.200+0000 7f9c5c48f640 1 -- 172.21.15.81:0/323299638 >> 172.21.15.81:0/323299638 conn(0x7f9c5406ccc0 msgr2=0x7f9c540b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.200+0000 7f9c5c48f640 1 -- 172.21.15.81:0/323299638 shutdown_connections 2024-06-29T14:37:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.200+0000 7f9c5c48f640 1 -- 172.21.15.81:0/323299638 wait complete. 2024-06-29T14:37:34.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.201+0000 7f9c5c48f640 1 Processor -- start 2024-06-29T14:37:34.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.201+0000 7f9c5c48f640 1 -- start start 2024-06-29T14:37:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5c48f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c400051c0 0x7f9c54072100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5c48f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c54072640 0x7f9c541f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5c48f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c541f52a0 0x7f9c541f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5c48f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9c540731b0 con 0x7f9c541f52a0 2024-06-29T14:37:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5c48f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9c54073030 con 0x7f9c54072640 2024-06-29T14:37:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5c48f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9c54073330 con 0x7f9c400051c0 2024-06-29T14:37:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5aa2c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c541f52a0 0x7f9c541f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5aa2c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c541f52a0 0x7f9c541f3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57598/0 (socket says 172.21.15.81:57598) 2024-06-29T14:37:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5aa2c640 1 -- 172.21.15.81:0/1342857954 learned_addr learned my addr 172.21.15.81:0/1342857954 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c59a2a640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c54072640 0x7f9c541f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5a22b640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c400051c0 0x7f9c54072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5aa2c640 1 -- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c400051c0 msgr2=0x7f9c54072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5aa2c640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c400051c0 0x7f9c54072100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5aa2c640 1 -- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c54072640 msgr2=0x7f9c541f39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:34.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5aa2c640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c54072640 0x7f9c541f39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:34.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5aa2c640 1 -- 172.21.15.81:0/1342857954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c541f88b0 con 0x7f9c541f52a0 2024-06-29T14:37:34.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5aa2c640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c541f52a0 0x7f9c541f3f20 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c02ad60 tx=0x7f9c4c001b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:34.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c4affd640 1 -- 172.21.15.81:0/1342857954 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c003d80 con 0x7f9c541f52a0 2024-06-29T14:37:34.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c4affd640 1 -- 172.21.15.81:0/1342857954 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c4c032420 con 0x7f9c541f52a0 2024-06-29T14:37:34.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.203+0000 7f9c5c48f640 1 -- 172.21.15.81:0/1342857954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c541f47c0 con 0x7f9c541f52a0 2024-06-29T14:37:34.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.204+0000 7f9c4affd640 1 -- 172.21.15.81:0/1342857954 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c032870 con 0x7f9c541f52a0 2024-06-29T14:37:34.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.204+0000 7f9c5c48f640 1 -- 172.21.15.81:0/1342857954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c541f4460 con 0x7f9c541f52a0 2024-06-29T14:37:34.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.204+0000 7f9c5c48f640 1 -- 172.21.15.81:0/1342857954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c541f8620 con 0x7f9c541f52a0 2024-06-29T14:37:34.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.205+0000 7f9c4affd640 1 -- 172.21.15.81:0/1342857954 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 33) ==== 98056+0+0 (secure 0 0 0) 0x7f9c4c03d030 con 0x7f9c541f52a0 2024-06-29T14:37:34.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.206+0000 7f9c4affd640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c38076170 0x7f9c38078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:34.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.206+0000 7f9c4affd640 1 -- 172.21.15.81:0/1342857954 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7f9c4c0b7180 con 0x7f9c541f52a0 2024-06-29T14:37:34.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.206+0000 7f9c5a22b640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c38076170 0x7f9c38078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:34.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.206+0000 7f9c5a22b640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c38076170 0x7f9c38078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9c50009bd0 tx=0x7f9c50005f90 comp rx=0 tx=0).ready entity=mgr.6217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:34.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.210+0000 7f9c4affd640 1 -- 172.21.15.81:0/1342857954 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9c4c006260 con 0x7f9c541f52a0 2024-06-29T14:37:34.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:34.437+0000 7f9c5c48f640 1 -- 172.21.15.81:0/1342857954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f9c54073890 con 0x7f9c541f52a0 2024-06-29T14:37:35.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.005+0000 7f9c4affd640 1 -- 172.21.15.81:0/1342857954 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v34) ==== 93+0+0 (secure 0 0 0) 0x7f9c4c080630 con 0x7f9c541f52a0 2024-06-29T14:37:35.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.009+0000 7f9c5c48f640 1 -- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c38076170 msgr2=0x7f9c38078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.009+0000 7f9c5c48f640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c38076170 0x7f9c38078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9c50009bd0 tx=0x7f9c50005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:35.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.009+0000 7f9c5c48f640 1 -- 172.21.15.81:0/1342857954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c541f52a0 msgr2=0x7f9c541f3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.009+0000 7f9c5c48f640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c541f52a0 0x7f9c541f3f20 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c02ad60 tx=0x7f9c4c001b60 comp rx=0 tx=0).stop 2024-06-29T14:37:35.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.010+0000 7f9c5c48f640 1 -- 172.21.15.81:0/1342857954 shutdown_connections 2024-06-29T14:37:35.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.010+0000 7f9c5c48f640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c541f52a0 0x7f9c541f3f20 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.010+0000 7f9c5c48f640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c54072640 0x7f9c541f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.010+0000 7f9c5c48f640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c38076170 0x7f9c38078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.010+0000 7f9c5c48f640 1 --2- 172.21.15.81:0/1342857954 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c400051c0 0x7f9c54072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.010+0000 7f9c5c48f640 1 -- 172.21.15.81:0/1342857954 >> 172.21.15.81:0/1342857954 conn(0x7f9c5406ccc0 msgr2=0x7f9c5406ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:35.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.010+0000 7f9c5c48f640 1 -- 172.21.15.81:0/1342857954 shutdown_connections 2024-06-29T14:37:35.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.010+0000 7f9c5c48f640 1 -- 172.21.15.81:0/1342857954 wait complete. 2024-06-29T14:37:35.023 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:37:35.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:35.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:35.103 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:37:35.103 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:35.103 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:35.104 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:35.119 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:35.119 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:35.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.332+0000 7f8e2b281640 1 -- 172.21.15.81:0/2883435787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e240b9040 msgr2=0x7f8e240d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.332+0000 7f8e2b281640 1 --2- 172.21.15.81:0/2883435787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e240b9040 0x7f8e240d0480 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8e240a1b70 tx=0x7f8e1402a680 comp rx=0 tx=0).stop 2024-06-29T14:37:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.332+0000 7f8e2b281640 1 -- 172.21.15.81:0/2883435787 shutdown_connections 2024-06-29T14:37:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.332+0000 7f8e2b281640 1 --2- 172.21.15.81:0/2883435787 >> v2:172.21.15.167:3300/0 conn(0x7f8e240b5250 0x7f8e240d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.332+0000 7f8e2b281640 1 --2- 172.21.15.81:0/2883435787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e240b9040 0x7f8e240d0480 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.332+0000 7f8e2b281640 1 -- 172.21.15.81:0/2883435787 >> 172.21.15.81:0/2883435787 conn(0x7f8e2409cf30 msgr2=0x7f8e240d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.332+0000 7f8e2b281640 1 -- 172.21.15.81:0/2883435787 shutdown_connections 2024-06-29T14:37:35.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.332+0000 7f8e2b281640 1 -- 172.21.15.81:0/2883435787 wait complete. 2024-06-29T14:37:35.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.334+0000 7f8e2b281640 1 Processor -- start 2024-06-29T14:37:35.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.334+0000 7f8e2b281640 1 -- start start 2024-06-29T14:37:35.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2b281640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e240b5250 0x7f8e24155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2b281640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8e240b7290 0x7f8e2409ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2b281640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8e240b9040 0x7f8e240c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2b281640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e240d2e20 con 0x7f8e240b5250 2024-06-29T14:37:35.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2b281640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8e240d2ca0 con 0x7f8e240b9040 2024-06-29T14:37:35.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2b281640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8e240b7db0 con 0x7f8e240b7290 2024-06-29T14:37:35.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2901d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e240b5250 0x7f8e24155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2901d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e240b5250 0x7f8e24155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57618/0 (socket says 172.21.15.81:57618) 2024-06-29T14:37:35.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2901d640 1 -- 172.21.15.81:0/2906862988 learned_addr learned my addr 172.21.15.81:0/2906862988 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:35.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2981e640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8e240b9040 0x7f8e240c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.335+0000 7f8e2881c640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8e240b7290 0x7f8e2409ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e2901d640 1 -- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8e240b7290 msgr2=0x7f8e2409ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e2901d640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8e240b7290 0x7f8e2409ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e2901d640 1 -- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8e240b9040 msgr2=0x7f8e240c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e2901d640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8e240b9040 0x7f8e240c7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e2901d640 1 -- 172.21.15.81:0/2906862988 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e240a07d0 con 0x7f8e240b5250 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e2901d640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e240b5250 0x7f8e24155fb0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f8e14034040 tx=0x7f8e14003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e19ffb640 1 -- 172.21.15.81:0/2906862988 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1403e030 con 0x7f8e240b5250 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e19ffb640 1 -- 172.21.15.81:0/2906862988 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e14003d10 con 0x7f8e240b5250 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e2b281640 1 -- 172.21.15.81:0/2906862988 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e240a0ab0 con 0x7f8e240b5250 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e19ffb640 1 -- 172.21.15.81:0/2906862988 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e14035040 con 0x7f8e240b5250 2024-06-29T14:37:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.336+0000 7f8e2b281640 1 -- 172.21.15.81:0/2906862988 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e240a1070 con 0x7f8e240b5250 2024-06-29T14:37:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.337+0000 7f8e2b281640 1 -- 172.21.15.81:0/2906862988 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e240b82f0 con 0x7f8e240b5250 2024-06-29T14:37:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.338+0000 7f8e19ffb640 1 -- 172.21.15.81:0/2906862988 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f8e14041030 con 0x7f8e240b5250 2024-06-29T14:37:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.339+0000 7f8e19ffb640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f8e08076420 0x7f8e08078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.339+0000 7f8e2881c640 1 -- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f8e08076420 msgr2=0x7f8e08078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.339+0000 7f8e19ffb640 1 -- 172.21.15.81:0/2906862988 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7f8e140b8530 con 0x7f8e240b5250 2024-06-29T14:37:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.339+0000 7f8e2881c640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f8e08076420 0x7f8e08078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:35.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.347+0000 7f8e19ffb640 1 -- 172.21.15.81:0/2906862988 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8e14081560 con 0x7f8e240b5250 2024-06-29T14:37:35.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.532+0000 7f8e2b281640 1 -- 172.21.15.81:0/2906862988 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e240b4bd0 con 0x7f8e240b5250 2024-06-29T14:37:35.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.533+0000 7f8e19ffb640 1 -- 172.21.15.81:0/2906862988 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+17273 (secure 0 0 0) 0x7f8e14085860 con 0x7f8e240b5250 2024-06-29T14:37:35.531 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:35.531 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":79,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:32.992934+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:18.983445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.642471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/3693496302":"2024-06-30T14:37:32.992869+0000","172.21.15.167:6832/3693496302":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/1220706198":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/538343225":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/1189302834":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/3217872433":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/2826882843":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/95469329":"2024-06-30T14:37:32.992869+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 -- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f8e08076420 msgr2=0x7f8e08078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f8e08076420 0x7f8e08078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 -- 172.21.15.81:0/2906862988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e240b5250 msgr2=0x7f8e24155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e240b5250 0x7f8e24155fb0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f8e14034040 tx=0x7f8e14003820 comp rx=0 tx=0).stop 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 -- 172.21.15.81:0/2906862988 shutdown_connections 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8e240b9040 0x7f8e240c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f8e08076420 0x7f8e08078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8e240b7290 0x7f8e2409ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 --2- 172.21.15.81:0/2906862988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e240b5250 0x7f8e24155fb0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 -- 172.21.15.81:0/2906862988 >> 172.21.15.81:0/2906862988 conn(0x7f8e2409cf30 msgr2=0x7f8e240d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 -- 172.21.15.81:0/2906862988 shutdown_connections 2024-06-29T14:37:35.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.535+0000 7f8e2b281640 1 -- 172.21.15.81:0/2906862988 wait complete. 2024-06-29T14:37:35.544 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:37:18.983445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:37:19.642471+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:35.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:35.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.666+0000 7f6b49bb2640 1 -- 172.21.15.81:0/2032672892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b440b5250 msgr2=0x7f6b440d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.666+0000 7f6b49bb2640 1 --2- 172.21.15.81:0/2032672892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b440b5250 0x7f6b440d1fc0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c005fd0 tx=0x7f6b3c02a940 comp rx=0 tx=0).stop 2024-06-29T14:37:35.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.666+0000 7f6b49bb2640 1 -- 172.21.15.81:0/2032672892 shutdown_connections 2024-06-29T14:37:35.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.666+0000 7f6b49bb2640 1 --2- 172.21.15.81:0/2032672892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b440b5250 0x7f6b440d1fc0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.666+0000 7f6b49bb2640 1 --2- 172.21.15.81:0/2032672892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b440d0b90 0x7f6b440d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.666+0000 7f6b49bb2640 1 -- 172.21.15.81:0/2032672892 >> 172.21.15.81:0/2032672892 conn(0x7f6b4409cf30 msgr2=0x7f6b440d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:35.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.666+0000 7f6b49bb2640 1 -- 172.21.15.81:0/2032672892 shutdown_connections 2024-06-29T14:37:35.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.666+0000 7f6b49bb2640 1 -- 172.21.15.81:0/2032672892 wait complete. 2024-06-29T14:37:35.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.668+0000 7f6b49bb2640 1 Processor -- start 2024-06-29T14:37:35.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.668+0000 7f6b49bb2640 1 -- start start 2024-06-29T14:37:35.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b49bb2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b440b5250 0x7f6b4415c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b49bb2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b440b9040 0x7f6b440c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b49bb2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b440ccf10 0x7f6b440cd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b49bb2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6b440d2680 con 0x7f6b440b5250 2024-06-29T14:37:35.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b49bb2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6b440d2500 con 0x7f6b440ccf10 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b49bb2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6b440d2800 con 0x7f6b440b9040 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b437fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b440b5250 0x7f6b4415c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b437fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b440b5250 0x7f6b4415c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57644/0 (socket says 172.21.15.81:57644) 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b437fe640 1 -- 172.21.15.81:0/79115848 learned_addr learned my addr 172.21.15.81:0/79115848 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b3bfff640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b440b9040 0x7f6b440c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.669+0000 7f6b43fff640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b440ccf10 0x7f6b440cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b437fe640 1 -- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b440b9040 msgr2=0x7f6b440c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b437fe640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b440b9040 0x7f6b440c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b437fe640 1 -- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b440ccf10 msgr2=0x7f6b440cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b437fe640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b440ccf10 0x7f6b440cd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b437fe640 1 -- 172.21.15.81:0/79115848 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b440cd860 con 0x7f6b440b5250 2024-06-29T14:37:35.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b3bfff640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b440b9040 0x7f6b440c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:35.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b43fff640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b440ccf10 0x7f6b440cd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:35.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b437fe640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b440b5250 0x7f6b4415c880 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400c3c0 tx=0x7f6b3400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:35.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b40ff9640 1 -- 172.21.15.81:0/79115848 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34017070 con 0x7f6b440b5250 2024-06-29T14:37:35.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b40ff9640 1 -- 172.21.15.81:0/79115848 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b340035f0 con 0x7f6b440b5250 2024-06-29T14:37:35.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b40ff9640 1 -- 172.21.15.81:0/79115848 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34003a40 con 0x7f6b440b5250 2024-06-29T14:37:35.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b49bb2640 1 -- 172.21.15.81:0/79115848 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b441f4cd0 con 0x7f6b440b5250 2024-06-29T14:37:35.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.670+0000 7f6b49bb2640 1 -- 172.21.15.81:0/79115848 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b441f54b0 con 0x7f6b440b5250 2024-06-29T14:37:35.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.671+0000 7f6b49bb2640 1 -- 172.21.15.81:0/79115848 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b440d2d90 con 0x7f6b440b5250 2024-06-29T14:37:35.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.672+0000 7f6b40ff9640 1 -- 172.21.15.81:0/79115848 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f6b3401a070 con 0x7f6b440b5250 2024-06-29T14:37:35.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.673+0000 7f6b40ff9640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f6b240763f0 0x7f6b24078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:35.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.673+0000 7f6b3bfff640 1 -- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f6b240763f0 msgr2=0x7f6b24078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:35.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.673+0000 7f6b40ff9640 1 -- 172.21.15.81:0/79115848 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7f6b34096dd0 con 0x7f6b440b5250 2024-06-29T14:37:35.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.673+0000 7f6b3bfff640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f6b240763f0 0x7f6b24078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:35.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.677+0000 7f6b40ff9640 1 -- 172.21.15.81:0/79115848 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6b440cd860 con 0x7f6b440b5250 2024-06-29T14:37:35.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.863+0000 7f6b49bb2640 1 -- 172.21.15.81:0/79115848 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6b440d09f0 con 0x7f6b440b5250 2024-06-29T14:37:35.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.863+0000 7f6b40ff9640 1 -- 172.21.15.81:0/79115848 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7f6b34098020 con 0x7f6b440b5250 2024-06-29T14:37:35.861 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:37:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 -- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f6b240763f0 msgr2=0x7f6b24078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f6b240763f0 0x7f6b24078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 -- 172.21.15.81:0/79115848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b440b5250 msgr2=0x7f6b4415c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b440b5250 0x7f6b4415c880 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400c3c0 tx=0x7f6b3400c880 comp rx=0 tx=0).stop 2024-06-29T14:37:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 -- 172.21.15.81:0/79115848 shutdown_connections 2024-06-29T14:37:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b440ccf10 0x7f6b440cd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f6b240763f0 0x7f6b24078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b440b9040 0x7f6b440c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 --2- 172.21.15.81:0/79115848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b440b5250 0x7f6b4415c880 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 -- 172.21.15.81:0/79115848 >> 172.21.15.81:0/79115848 conn(0x7f6b4409cf30 msgr2=0x7f6b4406e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 -- 172.21.15.81:0/79115848 shutdown_connections 2024-06-29T14:37:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.866+0000 7f6b49bb2640 1 -- 172.21.15.81:0/79115848 wait complete. 2024-06-29T14:37:35.875 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:37:35.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.997+0000 7f1438ba1640 1 -- 172.21.15.81:0/2406484392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14340ce770 msgr2=0x7f14340b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:35.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.997+0000 7f1438ba1640 1 --2- 172.21.15.81:0/2406484392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14340ce770 0x7f14340b35c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f14340c7400 tx=0x7f142402a4a0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.998+0000 7f1438ba1640 1 -- 172.21.15.81:0/2406484392 shutdown_connections 2024-06-29T14:37:35.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.998+0000 7f1438ba1640 1 --2- 172.21.15.81:0/2406484392 >> v2:172.21.15.167:3300/0 conn(0x7f14340ceb80 0x7f14340b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.998+0000 7f1438ba1640 1 --2- 172.21.15.81:0/2406484392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14340ce770 0x7f14340b35c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:35.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.998+0000 7f1438ba1640 1 -- 172.21.15.81:0/2406484392 >> 172.21.15.81:0/2406484392 conn(0x7f14340d3c60 msgr2=0x7f14340d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:35.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.998+0000 7f1438ba1640 1 -- 172.21.15.81:0/2406484392 shutdown_connections 2024-06-29T14:37:35.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:35.998+0000 7f1438ba1640 1 -- 172.21.15.81:0/2406484392 wait complete. 2024-06-29T14:37:35.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.000+0000 7f1438ba1640 1 Processor -- start 2024-06-29T14:37:35.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.000+0000 7f1438ba1640 1 -- start start 2024-06-29T14:37:35.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1438ba1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f14340ce770 0x7f14340cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1438ba1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14340ceb80 0x7f14340a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1438ba1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14340d2240 0x7f14341f8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1438ba1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f14340d15a0 con 0x7f14340ceb80 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1438ba1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f14340d1420 con 0x7f14340d2240 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1438ba1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f14340d4990 con 0x7f14340ce770 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1431d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14340ceb80 0x7f14340a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1431d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14340ceb80 0x7f14340a1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57660/0 (socket says 172.21.15.81:57660) 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1431d9b640 1 -- 172.21.15.81:0/3872003195 learned_addr learned my addr 172.21.15.81:0/3872003195 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1432d9d640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14340d2240 0x7f14341f8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f143259c640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f14340ce770 0x7f14340cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1431d9b640 1 -- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f14340ce770 msgr2=0x7f14340cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1431d9b640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f14340ce770 0x7f14340cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1431d9b640 1 -- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14340d2240 msgr2=0x7f14341f8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1431d9b640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14340d2240 0x7f14341f8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.001+0000 7f1431d9b640 1 -- 172.21.15.81:0/3872003195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14340cb4d0 con 0x7f14340ceb80 2024-06-29T14:37:36.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.002+0000 7f1431d9b640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14340ceb80 0x7f14340a1350 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f142800e7c0 tx=0x7f142800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:36.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.002+0000 7f1422ffd640 1 -- 172.21.15.81:0/3872003195 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142800ccc0 con 0x7f14340ceb80 2024-06-29T14:37:36.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.002+0000 7f1422ffd640 1 -- 172.21.15.81:0/3872003195 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1428010430 con 0x7f14340ceb80 2024-06-29T14:37:36.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.002+0000 7f1422ffd640 1 -- 172.21.15.81:0/3872003195 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1428010860 con 0x7f14340ceb80 2024-06-29T14:37:36.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.002+0000 7f1438ba1640 1 -- 172.21.15.81:0/3872003195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14340cb7b0 con 0x7f14340ceb80 2024-06-29T14:37:36.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.002+0000 7f1438ba1640 1 -- 172.21.15.81:0/3872003195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14340cbd70 con 0x7f14340ceb80 2024-06-29T14:37:36.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.003+0000 7f1438ba1640 1 -- 172.21.15.81:0/3872003195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14340cb240 con 0x7f14340ceb80 2024-06-29T14:37:36.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.003+0000 7f1422ffd640 1 -- 172.21.15.81:0/3872003195 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f1428010a00 con 0x7f14340ceb80 2024-06-29T14:37:36.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.004+0000 7f1422ffd640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f14140763f0 0x7f1414078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.004+0000 7f143259c640 1 -- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f14140763f0 msgr2=0x7f1414078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:36.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.004+0000 7f1422ffd640 1 -- 172.21.15.81:0/3872003195 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7f1428098420 con 0x7f14340ceb80 2024-06-29T14:37:36.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.004+0000 7f143259c640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f14140763f0 0x7f1414078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:36.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.008+0000 7f1422ffd640 1 -- 172.21.15.81:0/3872003195 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1428061370 con 0x7f14340ceb80 2024-06-29T14:37:36.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.206+0000 7f143259c640 1 -- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f14140763f0 msgr2=0x7f1414078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:36.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.206+0000 7f143259c640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f14140763f0 0x7f1414078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:36.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.227+0000 7f1438ba1640 1 -- 172.21.15.81:0/3872003195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f14340d2650 con 0x7f14340ceb80 2024-06-29T14:37:36.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.227+0000 7f1422ffd640 1 -- 172.21.15.81:0/3872003195 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7f1428065670 con 0x7f14340ceb80 2024-06-29T14:37:36.225 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:37:36.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 -- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f14140763f0 msgr2=0x7f1414078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:36.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f14140763f0 0x7f1414078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 -- 172.21.15.81:0/3872003195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14340ceb80 msgr2=0x7f14340a1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14340ceb80 0x7f14340a1350 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f142800e7c0 tx=0x7f142800ec80 comp rx=0 tx=0).stop 2024-06-29T14:37:36.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 -- 172.21.15.81:0/3872003195 shutdown_connections 2024-06-29T14:37:36.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f14340d2240 0x7f14341f8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14340ceb80 0x7f14340a1350 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f14140763f0 0x7f1414078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 --2- 172.21.15.81:0/3872003195 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f14340ce770 0x7f14340cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 -- 172.21.15.81:0/3872003195 >> 172.21.15.81:0/3872003195 conn(0x7f14340d3c60 msgr2=0x7f14341e6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 -- 172.21.15.81:0/3872003195 shutdown_connections 2024-06-29T14:37:36.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.230+0000 7f1438ba1640 1 -- 172.21.15.81:0/3872003195 wait complete. 2024-06-29T14:37:36.239 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:37:36.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.361+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/2433678405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7806c2f0 msgr2=0x7fdc7806c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.361+0000 7fdc7cb7a640 1 --2- 172.21.15.81:0/2433678405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7806c2f0 0x7fdc7806c700 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c005fd0 tx=0x7fdc6c02a670 comp rx=0 tx=0).stop 2024-06-29T14:37:36.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.361+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/2433678405 shutdown_connections 2024-06-29T14:37:36.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.361+0000 7fdc7cb7a640 1 --2- 172.21.15.81:0/2433678405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7806c2f0 0x7fdc7806c700 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.361+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/2433678405 >> 172.21.15.81:0/2433678405 conn(0x7fdc780b2920 msgr2=0x7fdc780b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.361+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/2433678405 shutdown_connections 2024-06-29T14:37:36.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.361+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/2433678405 wait complete. 2024-06-29T14:37:36.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.363+0000 7fdc7cb7a640 1 Processor -- start 2024-06-29T14:37:36.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.363+0000 7fdc7cb7a640 1 -- start start 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc7cb7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7806c2f0 0x7fdc780a9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc7cb7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7806f9a0 0x7fdc781f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc7cb7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc78159bd0 0x7fdc781f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc7cb7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7815fe20 con 0x7fdc7806f9a0 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc7cb7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7815fca0 con 0x7fdc78159bd0 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc7cb7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdc7815ffa0 con 0x7fdc7806c2f0 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc75d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7806f9a0 0x7fdc781f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc75d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7806f9a0 0x7fdc781f6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57678/0 (socket says 172.21.15.81:57678) 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc75d9b640 1 -- 172.21.15.81:0/1611997981 learned_addr learned my addr 172.21.15.81:0/1611997981 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc7659c640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7806c2f0 0x7fdc780a9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc76d9d640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc78159bd0 0x7fdc781f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc75d9b640 1 -- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7806c2f0 msgr2=0x7fdc780a9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc75d9b640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7806c2f0 0x7fdc780a9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc75d9b640 1 -- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc78159bd0 msgr2=0x7fdc781f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc75d9b640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc78159bd0 0x7fdc781f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.364+0000 7fdc75d9b640 1 -- 172.21.15.81:0/1611997981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7813ed80 con 0x7fdc7806f9a0 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.365+0000 7fdc75d9b640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7806f9a0 0x7fdc781f6a90 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c000c00 tx=0x7fdc6c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.365+0000 7fdc66ffd640 1 -- 172.21.15.81:0/1611997981 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c03e030 con 0x7fdc7806f9a0 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.365+0000 7fdc66ffd640 1 -- 172.21.15.81:0/1611997981 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc6c003e60 con 0x7fdc7806f9a0 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.365+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/1611997981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc7813f060 con 0x7fdc7806f9a0 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.365+0000 7fdc66ffd640 1 -- 172.21.15.81:0/1611997981 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c035040 con 0x7fdc7806f9a0 2024-06-29T14:37:36.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.365+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/1611997981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc7813f620 con 0x7fdc7806f9a0 2024-06-29T14:37:36.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.366+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/1611997981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc7806cf40 con 0x7fdc7806f9a0 2024-06-29T14:37:36.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.367+0000 7fdc66ffd640 1 -- 172.21.15.81:0/1611997981 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7fdc6c041030 con 0x7fdc7806f9a0 2024-06-29T14:37:36.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.367+0000 7fdc66ffd640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fdc580763f0 0x7fdc58078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.367+0000 7fdc7659c640 1 -- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fdc580763f0 msgr2=0x7fdc58078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:36.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.367+0000 7fdc66ffd640 1 -- 172.21.15.81:0/1611997981 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7fdc6c0b83a0 con 0x7fdc7806f9a0 2024-06-29T14:37:36.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.367+0000 7fdc7659c640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fdc580763f0 0x7fdc58078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:36.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.372+0000 7fdc66ffd640 1 -- 172.21.15.81:0/1611997981 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc7806cf40 con 0x7fdc7806f9a0 2024-06-29T14:37:36.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.560+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/1611997981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdc781604c0 con 0x7fdc7806f9a0 2024-06-29T14:37:36.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.561+0000 7fdc66ffd640 1 -- 172.21.15.81:0/1611997981 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v79) ==== 100+0+11 (secure 0 0 0) 0x7fdc6c081260 con 0x7fdc7806f9a0 2024-06-29T14:37:36.558 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fdc580763f0 msgr2=0x7fdc58078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fdc580763f0 0x7fdc58078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/1611997981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7806f9a0 msgr2=0x7fdc781f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7806f9a0 0x7fdc781f6a90 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c000c00 tx=0x7fdc6c003970 comp rx=0 tx=0).stop 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/1611997981 shutdown_connections 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc78159bd0 0x7fdc781f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7806f9a0 0x7fdc781f6a90 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fdc580763f0 0x7fdc58078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 --2- 172.21.15.81:0/1611997981 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7806c2f0 0x7fdc780a9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/1611997981 >> 172.21.15.81:0/1611997981 conn(0x7fdc780b2920 msgr2=0x7fdc780b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/1611997981 shutdown_connections 2024-06-29T14:37:36.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.563+0000 7fdc7cb7a640 1 -- 172.21.15.81:0/1611997981 wait complete. 2024-06-29T14:37:36.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:37:36.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.692+0000 7fc5add35640 1 -- 172.21.15.81:0/1405825446 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5a8073480 msgr2=0x7fc5a815b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.692+0000 7fc5add35640 1 --2- 172.21.15.81:0/1405825446 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5a8073480 0x7fc5a815b6e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc590009830 tx=0x7fc59002a480 comp rx=0 tx=0).stop 2024-06-29T14:37:36.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.692+0000 7fc5add35640 1 -- 172.21.15.81:0/1405825446 shutdown_connections 2024-06-29T14:37:36.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.692+0000 7fc5add35640 1 --2- 172.21.15.81:0/1405825446 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5a8073480 0x7fc5a815b6e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.692+0000 7fc5add35640 1 --2- 172.21.15.81:0/1405825446 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5a80cd150 0x7fc5a8072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.692+0000 7fc5add35640 1 -- 172.21.15.81:0/1405825446 >> 172.21.15.81:0/1405825446 conn(0x7fc5a80b2920 msgr2=0x7fc5a80b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.692+0000 7fc5add35640 1 -- 172.21.15.81:0/1405825446 shutdown_connections 2024-06-29T14:37:36.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.692+0000 7fc5add35640 1 -- 172.21.15.81:0/1405825446 wait complete. 2024-06-29T14:37:36.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.694+0000 7fc5add35640 1 Processor -- start 2024-06-29T14:37:36.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.694+0000 7fc5add35640 1 -- start start 2024-06-29T14:37:36.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.695+0000 7fc5add35640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a8073480 0x7fc5a814e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.695+0000 7fc5add35640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5a80cd150 0x7fc5a81f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.695+0000 7fc5add35640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5a815bc20 0x7fc5a80af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.695+0000 7fc5add35640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a815c1b0 con 0x7fc5a8073480 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.695+0000 7fc5add35640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a815c030 con 0x7fc5a815bc20 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.695+0000 7fc5add35640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a815f060 con 0x7fc5a80cd150 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.695+0000 7fc5a77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a8073480 0x7fc5a814e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5a77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a8073480 0x7fc5a814e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57680/0 (socket says 172.21.15.81:57680) 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5a77fe640 1 -- 172.21.15.81:0/1686996652 learned_addr learned my addr 172.21.15.81:0/1686996652 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5a7fff640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5a815bc20 0x7fc5a80af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5a6ffd640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5a80cd150 0x7fc5a81f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5a77fe640 1 -- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5a80cd150 msgr2=0x7fc5a81f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5a77fe640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5a80cd150 0x7fc5a81f5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5a77fe640 1 -- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5a815bc20 msgr2=0x7fc5a80af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5a77fe640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5a815bc20 0x7fc5a80af980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5a77fe640 1 -- 172.21.15.81:0/1686996652 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a81f5ee0 con 0x7fc5a8073480 2024-06-29T14:37:36.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5a77fe640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a8073480 0x7fc5a814e830 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc59800e790 tx=0x7fc59800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:36.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc59ffff640 1 -- 172.21.15.81:0/1686996652 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59800eea0 con 0x7fc5a8073480 2024-06-29T14:37:36.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc59ffff640 1 -- 172.21.15.81:0/1686996652 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc59800cd50 con 0x7fc5a8073480 2024-06-29T14:37:36.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5add35640 1 -- 172.21.15.81:0/1686996652 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a81f61c0 con 0x7fc5a8073480 2024-06-29T14:37:36.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc59ffff640 1 -- 172.21.15.81:0/1686996652 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598010690 con 0x7fc5a8073480 2024-06-29T14:37:36.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.696+0000 7fc5add35640 1 -- 172.21.15.81:0/1686996652 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a80adc50 con 0x7fc5a8073480 2024-06-29T14:37:36.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.697+0000 7fc5add35640 1 -- 172.21.15.81:0/1686996652 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5a80b31d0 con 0x7fc5a8073480 2024-06-29T14:37:36.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.698+0000 7fc59ffff640 1 -- 172.21.15.81:0/1686996652 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7fc598010830 con 0x7fc5a8073480 2024-06-29T14:37:36.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.699+0000 7fc59ffff640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fc588076420 0x7fc588078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:36.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.699+0000 7fc5a6ffd640 1 -- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fc588076420 msgr2=0x7fc588078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:36.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.699+0000 7fc59ffff640 1 -- 172.21.15.81:0/1686996652 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7fc598098c00 con 0x7fc5a8073480 2024-06-29T14:37:36.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.699+0000 7fc5a6ffd640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fc588076420 0x7fc588078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:36.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.702+0000 7fc59ffff640 1 -- 172.21.15.81:0/1686996652 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc5a80b31d0 con 0x7fc5a8073480 2024-06-29T14:37:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.900+0000 7fc5a6ffd640 1 -- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fc588076420 msgr2=0x7fc588078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.900+0000 7fc5a6ffd640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fc588076420 0x7fc588078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:36.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.902+0000 7fc5add35640 1 -- 172.21.15.81:0/1686996652 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc5a815f580 con 0x7fc5a8073480 2024-06-29T14:37:36.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.902+0000 7fc59ffff640 1 -- 172.21.15.81:0/1686996652 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v79) ==== 80+0+475 (secure 0 0 0) 0x7fc598061ba0 con 0x7fc5a8073480 2024-06-29T14:37:36.900 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/3693496302 2024-06-30T14:37:32.992869+0000 2024-06-29T14:37:36.900 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/3693496302 2024-06-30T14:37:32.992869+0000 2024-06-29T14:37:36.900 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1220706198 2024-06-30T14:37:32.992869+0000 2024-06-29T14:37:36.900 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/538343225 2024-06-30T14:37:32.992869+0000 2024-06-29T14:37:36.901 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1189302834 2024-06-30T14:37:32.992869+0000 2024-06-29T14:37:36.901 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3217872433 2024-06-30T14:37:32.992869+0000 2024-06-29T14:37:36.901 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2826882843 2024-06-30T14:37:32.992869+0000 2024-06-29T14:37:36.901 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/95469329 2024-06-30T14:37:32.992869+0000 2024-06-29T14:37:36.901 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T14:37:36.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.904+0000 7fc5add35640 1 -- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fc588076420 msgr2=0x7fc588078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:36.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.904+0000 7fc5add35640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fc588076420 0x7fc588078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.904+0000 7fc5add35640 1 -- 172.21.15.81:0/1686996652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a8073480 msgr2=0x7fc5a814e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:36.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.904+0000 7fc5add35640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a8073480 0x7fc5a814e830 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc59800e790 tx=0x7fc59800ec50 comp rx=0 tx=0).stop 2024-06-29T14:37:36.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.904+0000 7fc5add35640 1 -- 172.21.15.81:0/1686996652 shutdown_connections 2024-06-29T14:37:36.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.904+0000 7fc5add35640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5a815bc20 0x7fc5a80af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.904+0000 7fc5add35640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fc588076420 0x7fc588078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.904+0000 7fc5add35640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5a80cd150 0x7fc5a81f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.904+0000 7fc5add35640 1 --2- 172.21.15.81:0/1686996652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a8073480 0x7fc5a814e830 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:36.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.904+0000 7fc5add35640 1 -- 172.21.15.81:0/1686996652 >> 172.21.15.81:0/1686996652 conn(0x7fc5a80b2920 msgr2=0x7fc5a815ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:36.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.905+0000 7fc5add35640 1 -- 172.21.15.81:0/1686996652 shutdown_connections 2024-06-29T14:37:36.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:36.905+0000 7fc5add35640 1 -- 172.21.15.81:0/1686996652 wait complete. 2024-06-29T14:37:36.914 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:37:36.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:36.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:36.943 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:37:36.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:36.943 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:37:36.998 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:36.998 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:37:36.998 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:37.014 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:37.014 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:37.014 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:37.015 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:37.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:37.054 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:37:37.054 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:37:37.054 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:37.070 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:37.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:37.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.211+0000 7fbd69ea4640 1 -- 172.21.15.81:0/1040431018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd640cd150 msgr2=0x7fbd64072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.211+0000 7fbd69ea4640 1 --2- 172.21.15.81:0/1040431018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd640cd150 0x7fbd64072f40 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbd54005fd0 tx=0x7fbd5402a540 comp rx=0 tx=0).stop 2024-06-29T14:37:37.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.211+0000 7fbd69ea4640 1 -- 172.21.15.81:0/1040431018 shutdown_connections 2024-06-29T14:37:37.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.211+0000 7fbd69ea4640 1 --2- 172.21.15.81:0/1040431018 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd64073480 0x7fbd6415b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.211+0000 7fbd69ea4640 1 --2- 172.21.15.81:0/1040431018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd640cd150 0x7fbd64072f40 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.211+0000 7fbd69ea4640 1 -- 172.21.15.81:0/1040431018 >> 172.21.15.81:0/1040431018 conn(0x7fbd640b2920 msgr2=0x7fbd640b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:37.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.211+0000 7fbd69ea4640 1 -- 172.21.15.81:0/1040431018 shutdown_connections 2024-06-29T14:37:37.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.211+0000 7fbd69ea4640 1 -- 172.21.15.81:0/1040431018 wait complete. 2024-06-29T14:37:37.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.213+0000 7fbd69ea4640 1 Processor -- start 2024-06-29T14:37:37.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.213+0000 7fbd69ea4640 1 -- start start 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd69ea4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd64073480 0x7fbd640a9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd69ea4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd640cd150 0x7fbd640cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd69ea4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd6415bc20 0x7fbd640cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd69ea4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6415c1b0 con 0x7fbd640cd150 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd69ea4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6415c030 con 0x7fbd6415bc20 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd69ea4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbd6415f060 con 0x7fbd64073480 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd5affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd640cd150 0x7fbd640cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd5affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd640cd150 0x7fbd640cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57702/0 (socket says 172.21.15.81:57702) 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd5affd640 1 -- 172.21.15.81:0/522947888 learned_addr learned my addr 172.21.15.81:0/522947888 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd637fe640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd64073480 0x7fbd640a9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.214+0000 7fbd63fff640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd6415bc20 0x7fbd640cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd5affd640 1 -- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd64073480 msgr2=0x7fbd640a9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd5affd640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd64073480 0x7fbd640a9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd5affd640 1 -- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd6415bc20 msgr2=0x7fbd640cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd5affd640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd6415bc20 0x7fbd640cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd5affd640 1 -- 172.21.15.81:0/522947888 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd640aa4b0 con 0x7fbd640cd150 2024-06-29T14:37:37.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd5affd640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd640cd150 0x7fbd640cf2a0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00e790 tx=0x7fbd4c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:37.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd60ff9640 1 -- 172.21.15.81:0/522947888 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c00eea0 con 0x7fbd640cd150 2024-06-29T14:37:37.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd60ff9640 1 -- 172.21.15.81:0/522947888 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd4c00ccc0 con 0x7fbd640cd150 2024-06-29T14:37:37.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd69ea4640 1 -- 172.21.15.81:0/522947888 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd640aa790 con 0x7fbd640cd150 2024-06-29T14:37:37.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd60ff9640 1 -- 172.21.15.81:0/522947888 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c004280 con 0x7fbd640cd150 2024-06-29T14:37:37.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.215+0000 7fbd69ea4640 1 -- 172.21.15.81:0/522947888 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd640aad50 con 0x7fbd640cd150 2024-06-29T14:37:37.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.216+0000 7fbd69ea4640 1 -- 172.21.15.81:0/522947888 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd6415f580 con 0x7fbd640cd150 2024-06-29T14:37:37.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.217+0000 7fbd60ff9640 1 -- 172.21.15.81:0/522947888 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7fbd4c0035f0 con 0x7fbd640cd150 2024-06-29T14:37:37.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.217+0000 7fbd60ff9640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fbd440763f0 0x7fbd44078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.217+0000 7fbd637fe640 1 -- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fbd440763f0 msgr2=0x7fbd44078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:37.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.217+0000 7fbd60ff9640 1 -- 172.21.15.81:0/522947888 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7fbd4c098360 con 0x7fbd640cd150 2024-06-29T14:37:37.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.217+0000 7fbd637fe640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fbd440763f0 0x7fbd44078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:37.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.222+0000 7fbd60ff9640 1 -- 172.21.15.81:0/522947888 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbd4c061220 con 0x7fbd640cd150 2024-06-29T14:37:37.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.418+0000 7fbd637fe640 1 -- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fbd440763f0 msgr2=0x7fbd44078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:37.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.418+0000 7fbd637fe640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fbd440763f0 0x7fbd44078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:37.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.449+0000 7fbd69ea4640 1 -- 172.21.15.81:0/522947888 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd640aa1d0 con 0x7fbd640cd150 2024-06-29T14:37:37.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.450+0000 7fbd60ff9640 1 -- 172.21.15.81:0/522947888 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5226 (secure 0 0 0) 0x7fbd4c065520 con 0x7fbd640cd150 2024-06-29T14:37:37.447 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:37.447 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"btime":"2024-06-29T14:37:24:340402+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3148404249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3148404249},{"type":"v1","addr":"172.21.15.167:6836","nonce":3148404249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6549,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/704367026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":704367026},{"type":"v1","addr":"172.21.15.81:6834","nonce":704367026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6552,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2316096753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2316096753},{"type":"v1","addr":"172.21.15.81:6835","nonce":2316096753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.692072+0000","modified":"2024-06-29T14:37:23.709021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6567":{"gid":6567,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/2723173791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2723173791},{"type":"v1","addr":"172.21.15.167:6837","nonce":2723173791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6567,"qdb_cluster":[6567]},"id":4}]} 2024-06-29T14:37:37.448 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 34 2024-06-29T14:37:37.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 -- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fbd440763f0 msgr2=0x7fbd44078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:37.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fbd440763f0 0x7fbd44078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 -- 172.21.15.81:0/522947888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd640cd150 msgr2=0x7fbd640cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd640cd150 0x7fbd640cf2a0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00e790 tx=0x7fbd4c00ec50 comp rx=0 tx=0).stop 2024-06-29T14:37:37.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 -- 172.21.15.81:0/522947888 shutdown_connections 2024-06-29T14:37:37.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd6415bc20 0x7fbd640cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd640cd150 0x7fbd640cf2a0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fbd440763f0 0x7fbd44078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 --2- 172.21.15.81:0/522947888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd64073480 0x7fbd640a9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 -- 172.21.15.81:0/522947888 >> 172.21.15.81:0/522947888 conn(0x7fbd640b2920 msgr2=0x7fbd640709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:37.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 -- 172.21.15.81:0/522947888 shutdown_connections 2024-06-29T14:37:37.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.452+0000 7fbd69ea4640 1 -- 172.21.15.81:0/522947888 wait complete. 2024-06-29T14:37:37.461 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:37.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.580+0000 7f611f8f7640 1 -- 172.21.15.81:0/2575444640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61040041c0 msgr2=0x7f6104004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.580+0000 7f611f8f7640 1 --2- 172.21.15.81:0/2575444640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61040041c0 0x7f6104004800 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f6114006f60 tx=0x7f6104005830 comp rx=0 tx=0).stop 2024-06-29T14:37:37.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.580+0000 7f611de94640 1 -- 172.21.15.81:0/2575444640 reap_dead start 2024-06-29T14:37:37.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.580+0000 7f611f8f7640 1 -- 172.21.15.81:0/2575444640 shutdown_connections 2024-06-29T14:37:37.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.580+0000 7f611f8f7640 1 -- 172.21.15.81:0/2575444640 >> 172.21.15.81:0/2575444640 conn(0x7f61180b3680 msgr2=0x7f61180b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:37.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.580+0000 7f611f8f7640 1 -- 172.21.15.81:0/2575444640 shutdown_connections 2024-06-29T14:37:37.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.580+0000 7f611f8f7640 1 -- 172.21.15.81:0/2575444640 wait complete. 2024-06-29T14:37:37.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.582+0000 7f611f8f7640 1 Processor -- start 2024-06-29T14:37:37.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.582+0000 7f611f8f7640 1 -- start start 2024-06-29T14:37:37.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611f8f7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61040041c0 0x7f61180c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611f8f7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61180c5fb0 0x7f611806e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611f8f7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61180c63c0 0x7f61180c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611f8f7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6118164d00 con 0x7f61180c5fb0 2024-06-29T14:37:37.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611f8f7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6118164b80 con 0x7f61040041c0 2024-06-29T14:37:37.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611f8f7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6118164e80 con 0x7f61180c63c0 2024-06-29T14:37:37.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611ce92640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61180c5fb0 0x7f611806e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611ce92640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61180c5fb0 0x7f611806e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57716/0 (socket says 172.21.15.81:57716) 2024-06-29T14:37:37.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611ce92640 1 -- 172.21.15.81:0/959105153 learned_addr learned my addr 172.21.15.81:0/959105153 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:37.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611d693640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61040041c0 0x7f61180c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611de94640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61180c63c0 0x7f61180c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611ce92640 1 -- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61180c63c0 msgr2=0x7f61180c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611ce92640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61180c63c0 0x7f61180c7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611ce92640 1 -- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61040041c0 msgr2=0x7f61180c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611ce92640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61040041c0 0x7f61180c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611ce92640 1 -- 172.21.15.81:0/959105153 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61180c9120 con 0x7f61180c5fb0 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.583+0000 7f611ce92640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61180c5fb0 0x7f611806e640 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6114031040 tx=0x7f6114006cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.584+0000 7f610dffb640 1 -- 172.21.15.81:0/959105153 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114032030 con 0x7f61180c5fb0 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.584+0000 7f610dffb640 1 -- 172.21.15.81:0/959105153 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6114003af0 con 0x7f61180c5fb0 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.584+0000 7f610dffb640 1 -- 172.21.15.81:0/959105153 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6114037450 con 0x7f61180c5fb0 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.584+0000 7f611f8f7640 1 -- 172.21.15.81:0/959105153 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f611806d5a0 con 0x7f61180c5fb0 2024-06-29T14:37:37.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.584+0000 7f611f8f7640 1 -- 172.21.15.81:0/959105153 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f611806db60 con 0x7f61180c5fb0 2024-06-29T14:37:37.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.585+0000 7f611f8f7640 1 -- 172.21.15.81:0/959105153 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61180cc7b0 con 0x7f61180c5fb0 2024-06-29T14:37:37.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.586+0000 7f610dffb640 1 -- 172.21.15.81:0/959105153 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f6114035020 con 0x7f61180c5fb0 2024-06-29T14:37:37.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.586+0000 7f610dffb640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f60fc0763f0 0x7f60fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.586+0000 7f611d693640 1 -- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f60fc0763f0 msgr2=0x7f60fc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:37.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.586+0000 7f610dffb640 1 -- 172.21.15.81:0/959105153 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7f61140b7fb0 con 0x7f61180c5fb0 2024-06-29T14:37:37.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.586+0000 7f611d693640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f60fc0763f0 0x7f60fc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:37.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.591+0000 7f610dffb640 1 -- 172.21.15.81:0/959105153 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6114047030 con 0x7f61180c5fb0 2024-06-29T14:37:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.778+0000 7f611f8f7640 1 -- 172.21.15.81:0/959105153 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f611815b020 con 0x7f61180c5fb0 2024-06-29T14:37:37.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.779+0000 7f610dffb640 1 -- 172.21.15.81:0/959105153 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+17273 (secure 0 0 0) 0x7f6114081550 con 0x7f61180c5fb0 2024-06-29T14:37:37.777 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:37.777 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":79,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:32.992934+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:18.983445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.642471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/3693496302":"2024-06-30T14:37:32.992869+0000","172.21.15.167:6832/3693496302":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/1220706198":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/538343225":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/1189302834":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/3217872433":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/2826882843":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/95469329":"2024-06-30T14:37:32.992869+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:37.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.781+0000 7f611f8f7640 1 -- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f60fc0763f0 msgr2=0x7f60fc078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:37.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.781+0000 7f611f8f7640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f60fc0763f0 0x7f60fc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.781+0000 7f611f8f7640 1 -- 172.21.15.81:0/959105153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61180c5fb0 msgr2=0x7f611806e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.781+0000 7f611f8f7640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61180c5fb0 0x7f611806e640 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6114031040 tx=0x7f6114006cc0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.782+0000 7f611f8f7640 1 -- 172.21.15.81:0/959105153 shutdown_connections 2024-06-29T14:37:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.782+0000 7f611f8f7640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f60fc0763f0 0x7f60fc078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.782+0000 7f611f8f7640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f61180c63c0 0x7f61180c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.782+0000 7f611f8f7640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61180c5fb0 0x7f611806e640 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.782+0000 7f611f8f7640 1 --2- 172.21.15.81:0/959105153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f61040041c0 0x7f61180c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.782+0000 7f611f8f7640 1 -- 172.21.15.81:0/959105153 >> 172.21.15.81:0/959105153 conn(0x7f61180b3680 msgr2=0x7f611815a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.782+0000 7f611f8f7640 1 -- 172.21.15.81:0/959105153 shutdown_connections 2024-06-29T14:37:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.782+0000 7f611f8f7640 1 -- 172.21.15.81:0/959105153 wait complete. 2024-06-29T14:37:37.791 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:37:18.983445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:37:19.642471+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:37.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:37.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.912+0000 7fb43f511640 1 -- 172.21.15.81:0/3597389633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4380cd150 msgr2=0x7fb438072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.912+0000 7fb43f511640 1 --2- 172.21.15.81:0/3597389633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4380cd150 0x7fb438072f40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb438074310 tx=0x7fb42802a560 comp rx=0 tx=0).stop 2024-06-29T14:37:37.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.912+0000 7fb43f511640 1 -- 172.21.15.81:0/3597389633 shutdown_connections 2024-06-29T14:37:37.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.912+0000 7fb43f511640 1 --2- 172.21.15.81:0/3597389633 >> v2:172.21.15.167:3300/0 conn(0x7fb438073480 0x7fb43815b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.912+0000 7fb43f511640 1 --2- 172.21.15.81:0/3597389633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4380cd150 0x7fb438072f40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.912+0000 7fb43f511640 1 -- 172.21.15.81:0/3597389633 >> 172.21.15.81:0/3597389633 conn(0x7fb4380b2920 msgr2=0x7fb4380b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:37.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.912+0000 7fb43f511640 1 -- 172.21.15.81:0/3597389633 shutdown_connections 2024-06-29T14:37:37.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.912+0000 7fb43f511640 1 -- 172.21.15.81:0/3597389633 wait complete. 2024-06-29T14:37:37.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.914+0000 7fb43f511640 1 Processor -- start 2024-06-29T14:37:37.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.914+0000 7fb43f511640 1 -- start start 2024-06-29T14:37:37.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43f511640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb438073480 0x7fb43814e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43f511640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4380cd150 0x7fb4380af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43f511640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb43815bc20 0x7fb43806c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43f511640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb43815c1b0 con 0x7fb438073480 2024-06-29T14:37:37.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43f511640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb43815c030 con 0x7fb4380cd150 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43f511640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb43815f060 con 0x7fb43815bc20 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43d2ad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb438073480 0x7fb43814e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43d2ad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb438073480 0x7fb43814e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57746/0 (socket says 172.21.15.81:57746) 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43d2ad640 1 -- 172.21.15.81:0/842627044 learned_addr learned my addr 172.21.15.81:0/842627044 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43caac640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4380cd150 0x7fb4380af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.915+0000 7fb43daae640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb43815bc20 0x7fb43806c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb43d2ad640 1 -- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb43815bc20 msgr2=0x7fb43806c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb43d2ad640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb43815bc20 0x7fb43806c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb43d2ad640 1 -- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4380cd150 msgr2=0x7fb4380af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb43d2ad640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4380cd150 0x7fb4380af980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:37.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb43d2ad640 1 -- 172.21.15.81:0/842627044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43814e9e0 con 0x7fb438073480 2024-06-29T14:37:37.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb43daae640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb43815bc20 0x7fb43806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:37.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb43d2ad640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb438073480 0x7fb43814e4a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fb428034040 tx=0x7fb4280039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:37.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb42dffb640 1 -- 172.21.15.81:0/842627044 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42803e030 con 0x7fb438073480 2024-06-29T14:37:37.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb42dffb640 1 -- 172.21.15.81:0/842627044 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4280047a0 con 0x7fb438073480 2024-06-29T14:37:37.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb43f511640 1 -- 172.21.15.81:0/842627044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4380ce3f0 con 0x7fb438073480 2024-06-29T14:37:37.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb42dffb640 1 -- 172.21.15.81:0/842627044 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb428035040 con 0x7fb438073480 2024-06-29T14:37:37.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.916+0000 7fb43f511640 1 -- 172.21.15.81:0/842627044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4380ce9b0 con 0x7fb438073480 2024-06-29T14:37:37.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.917+0000 7fb43f511640 1 -- 172.21.15.81:0/842627044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb43815f4c0 con 0x7fb438073480 2024-06-29T14:37:37.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.918+0000 7fb42dffb640 1 -- 172.21.15.81:0/842627044 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7fb428041030 con 0x7fb438073480 2024-06-29T14:37:37.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.919+0000 7fb42dffb640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fb41c076420 0x7fb41c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:37.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.919+0000 7fb42dffb640 1 -- 172.21.15.81:0/842627044 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7fb4280b82a0 con 0x7fb438073480 2024-06-29T14:37:37.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.919+0000 7fb43caac640 1 -- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fb41c076420 msgr2=0x7fb41c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:37.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.919+0000 7fb43caac640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fb41c076420 0x7fb41c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:37.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:37.923+0000 7fb42dffb640 1 -- 172.21.15.81:0/842627044 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb428081160 con 0x7fb438073480 2024-06-29T14:37:38.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.119+0000 7fb43f511640 1 -- 172.21.15.81:0/842627044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb438070cc0 con 0x7fb438073480 2024-06-29T14:37:38.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.119+0000 7fb42dffb640 1 -- 172.21.15.81:0/842627044 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7fb428085460 con 0x7fb438073480 2024-06-29T14:37:38.117 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:37:38.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.120+0000 7fb43caac640 1 -- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fb41c076420 msgr2=0x7fb41c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:38.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.120+0000 7fb43caac640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fb41c076420 0x7fb41c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:37:38.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 -- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fb41c076420 msgr2=0x7fb41c078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:38.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fb41c076420 0x7fb41c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 -- 172.21.15.81:0/842627044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb438073480 msgr2=0x7fb43814e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb438073480 0x7fb43814e4a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fb428034040 tx=0x7fb4280039e0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 -- 172.21.15.81:0/842627044 shutdown_connections 2024-06-29T14:37:38.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fb41c076420 0x7fb41c078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb43815bc20 0x7fb43806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb4380cd150 0x7fb4380af980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 --2- 172.21.15.81:0/842627044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb438073480 0x7fb43814e4a0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 -- 172.21.15.81:0/842627044 >> 172.21.15.81:0/842627044 conn(0x7fb4380b2920 msgr2=0x7fb4380b0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 -- 172.21.15.81:0/842627044 shutdown_connections 2024-06-29T14:37:38.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.121+0000 7fb43f511640 1 -- 172.21.15.81:0/842627044 wait complete. 2024-06-29T14:37:38.131 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:37:38.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.252+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2874599135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c94073480 msgr2=0x7f9c9415b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.252+0000 7f9c9bb9e640 1 --2- 172.21.15.81:0/2874599135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c94073480 0x7f9c9415b6e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f9c94072f10 tx=0x7f9c9002a560 comp rx=0 tx=0).stop 2024-06-29T14:37:38.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.252+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2874599135 shutdown_connections 2024-06-29T14:37:38.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.252+0000 7f9c9bb9e640 1 --2- 172.21.15.81:0/2874599135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c94073480 0x7f9c9415b6e0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.252+0000 7f9c9bb9e640 1 --2- 172.21.15.81:0/2874599135 >> v2:172.21.15.167:3300/0 conn(0x7f9c940cd150 0x7f9c94072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.252+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2874599135 >> 172.21.15.81:0/2874599135 conn(0x7f9c940b2920 msgr2=0x7f9c940b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.252+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2874599135 shutdown_connections 2024-06-29T14:37:38.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.252+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2874599135 wait complete. 2024-06-29T14:37:38.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.254+0000 7f9c9bb9e640 1 Processor -- start 2024-06-29T14:37:38.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.254+0000 7f9c9bb9e640 1 -- start start 2024-06-29T14:37:38.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.255+0000 7f9c9bb9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c94073480 0x7f9c9414a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.255+0000 7f9c9bb9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c940cd150 0x7f9c940a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.255+0000 7f9c9bb9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9415bc20 0x7f9c940d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.255+0000 7f9c9bb9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9415c1b0 con 0x7f9c94073480 2024-06-29T14:37:38.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.255+0000 7f9c9bb9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9415c030 con 0x7f9c940cd150 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.255+0000 7f9c9bb9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9c9415f060 con 0x7f9c9415bc20 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9993a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c94073480 0x7f9c9414a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9993a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c94073480 0x7f9c9414a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57752/0 (socket says 172.21.15.81:57752) 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9993a640 1 -- 172.21.15.81:0/2778847205 learned_addr learned my addr 172.21.15.81:0/2778847205 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c99139640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c940cd150 0x7f9c940a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9a13b640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9415bc20 0x7f9c940d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9993a640 1 -- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9415bc20 msgr2=0x7f9c940d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9993a640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9415bc20 0x7f9c940d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9993a640 1 -- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c940cd150 msgr2=0x7f9c940a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9993a640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c940cd150 0x7f9c940a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9993a640 1 -- 172.21.15.81:0/2778847205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c940aa920 con 0x7f9c94073480 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9993a640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c94073480 0x7f9c9414a010 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400d770 tx=0x7f9c8400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c8a7fc640 1 -- 172.21.15.81:0/2778847205 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c84017070 con 0x7f9c94073480 2024-06-29T14:37:38.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c8a7fc640 1 -- 172.21.15.81:0/2778847205 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c84009d20 con 0x7f9c94073480 2024-06-29T14:37:38.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2778847205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c940a9da0 con 0x7f9c94073480 2024-06-29T14:37:38.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c8a7fc640 1 -- 172.21.15.81:0/2778847205 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c840106c0 con 0x7f9c94073480 2024-06-29T14:37:38.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.256+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2778847205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c940aa360 con 0x7f9c94073480 2024-06-29T14:37:38.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.258+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2778847205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c94154de0 con 0x7f9c94073480 2024-06-29T14:37:38.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.258+0000 7f9c8a7fc640 1 -- 172.21.15.81:0/2778847205 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7f9c8401a070 con 0x7f9c94073480 2024-06-29T14:37:38.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.259+0000 7f9c8a7fc640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c780763f0 0x7f9c78078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.259+0000 7f9c99139640 1 -- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c780763f0 msgr2=0x7f9c78078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:38.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.259+0000 7f9c8a7fc640 1 -- 172.21.15.81:0/2778847205 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7f9c84097ba0 con 0x7f9c94073480 2024-06-29T14:37:38.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.259+0000 7f9c99139640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c780763f0 0x7f9c78078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:38.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.263+0000 7f9c8a7fc640 1 -- 172.21.15.81:0/2778847205 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9c84060af0 con 0x7f9c94073480 2024-06-29T14:37:38.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.451+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2778847205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9c9406fa30 con 0x7f9c94073480 2024-06-29T14:37:38.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.451+0000 7f9c8a7fc640 1 -- 172.21.15.81:0/2778847205 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v79) ==== 104+0+11 (secure 0 0 0) 0x7f9c84064df0 con 0x7f9c94073480 2024-06-29T14:37:38.449 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:37:38.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.453+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c780763f0 msgr2=0x7f9c78078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:38.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.453+0000 7f9c9bb9e640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c780763f0 0x7f9c78078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.453+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2778847205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c94073480 msgr2=0x7f9c9414a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.453+0000 7f9c9bb9e640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c94073480 0x7f9c9414a010 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400d770 tx=0x7f9c8400dc30 comp rx=0 tx=0).stop 2024-06-29T14:37:38.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.453+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2778847205 shutdown_connections 2024-06-29T14:37:38.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.453+0000 7f9c9bb9e640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7f9c780763f0 0x7f9c78078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.453+0000 7f9c9bb9e640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9415bc20 0x7f9c940d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.453+0000 7f9c9bb9e640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c940cd150 0x7f9c940a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.453+0000 7f9c9bb9e640 1 --2- 172.21.15.81:0/2778847205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c94073480 0x7f9c9414a010 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.454+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2778847205 >> 172.21.15.81:0/2778847205 conn(0x7f9c940b2920 msgr2=0x7f9c9415d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.454+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2778847205 shutdown_connections 2024-06-29T14:37:38.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.454+0000 7f9c9bb9e640 1 -- 172.21.15.81:0/2778847205 wait complete. 2024-06-29T14:37:38.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:37:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.586+0000 7fd342009640 1 -- 172.21.15.81:0/88242419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3280051c0 msgr2=0x7fd32800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.586+0000 7fd342009640 1 --2- 172.21.15.81:0/88242419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3280051c0 0x7fd32800b9c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd334001e90 tx=0x7fd328005830 comp rx=0 tx=0).stop 2024-06-29T14:37:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.586+0000 7fd342009640 1 -- 172.21.15.81:0/88242419 shutdown_connections 2024-06-29T14:37:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.586+0000 7fd342009640 1 --2- 172.21.15.81:0/88242419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3280051c0 0x7fd32800b9c0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.586+0000 7fd342009640 1 -- 172.21.15.81:0/88242419 >> 172.21.15.81:0/88242419 conn(0x7fd33c0d0b70 msgr2=0x7fd33c0d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.586+0000 7fd342009640 1 -- 172.21.15.81:0/88242419 shutdown_connections 2024-06-29T14:37:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.586+0000 7fd342009640 1 -- 172.21.15.81:0/88242419 wait complete. 2024-06-29T14:37:38.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.588+0000 7fd342009640 1 Processor -- start 2024-06-29T14:37:38.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.588+0000 7fd342009640 1 -- start start 2024-06-29T14:37:38.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd342009640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3280051c0 0x7fd33c0ab120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd342009640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd33c0ad180 0x7fd33c0ab660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd342009640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd33c1e6b40 0x7fd33c1e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd342009640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c15b570 con 0x7fd33c0ad180 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd342009640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c15b3f0 con 0x7fd33c1e6b40 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd342009640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd33c072f40 con 0x7fd3280051c0 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd33affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd33c0ad180 0x7fd33c0ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd33affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd33c0ad180 0x7fd33c0ab660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57770/0 (socket says 172.21.15.81:57770) 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd33affd640 1 -- 172.21.15.81:0/3568554138 learned_addr learned my addr 172.21.15.81:0/3568554138 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd33bfff640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd33c1e6b40 0x7fd33c1e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.589+0000 7fd33b7fe640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3280051c0 0x7fd33c0ab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd33affd640 1 -- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3280051c0 msgr2=0x7fd33c0ab120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd33affd640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3280051c0 0x7fd33c0ab120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd33affd640 1 -- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd33c1e6b40 msgr2=0x7fd33c1e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd33affd640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd33c1e6b40 0x7fd33c1e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd33affd640 1 -- 172.21.15.81:0/3568554138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33c1e3d40 con 0x7fd33c0ad180 2024-06-29T14:37:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd33affd640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd33c0ad180 0x7fd33c0ab660 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fd324011ae0 tx=0x7fd324009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd333fff640 1 -- 172.21.15.81:0/3568554138 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32401e070 con 0x7fd33c0ad180 2024-06-29T14:37:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd333fff640 1 -- 172.21.15.81:0/3568554138 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd324004480 con 0x7fd33c0ad180 2024-06-29T14:37:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd342009640 1 -- 172.21.15.81:0/3568554138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd33c1e4020 con 0x7fd33c0ad180 2024-06-29T14:37:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd333fff640 1 -- 172.21.15.81:0/3568554138 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32400fc60 con 0x7fd33c0ad180 2024-06-29T14:37:38.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.590+0000 7fd342009640 1 -- 172.21.15.81:0/3568554138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd33c0ad5c0 con 0x7fd33c0ad180 2024-06-29T14:37:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.591+0000 7fd342009640 1 -- 172.21.15.81:0/3568554138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd33c06fdd0 con 0x7fd33c0ad180 2024-06-29T14:37:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.592+0000 7fd333fff640 1 -- 172.21.15.81:0/3568554138 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 34) ==== 98304+0+0 (secure 0 0 0) 0x7fd324021070 con 0x7fd33c0ad180 2024-06-29T14:37:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.592+0000 7fd333fff640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fd31c0763f0 0x7fd31c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.592+0000 7fd33b7fe640 1 -- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fd31c0763f0 msgr2=0x7fd31c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2940523935 2024-06-29T14:37:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.592+0000 7fd333fff640 1 -- 172.21.15.81:0/3568554138 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6125+0+0 (secure 0 0 0) 0x7fd324098550 con 0x7fd33c0ad180 2024-06-29T14:37:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.592+0000 7fd33b7fe640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fd31c0763f0 0x7fd31c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:37:38.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.597+0000 7fd333fff640 1 -- 172.21.15.81:0/3568554138 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd33c06fdd0 con 0x7fd33c0ad180 2024-06-29T14:37:38.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.677+0000 7fd333fff640 1 -- 172.21.15.81:0/3568554138 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 35) ==== 97912+0+0 (secure 0 0 0) 0x7fd3240968d0 con 0x7fd33c0ad180 2024-06-29T14:37:38.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.677+0000 7fd333fff640 1 -- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fd31c0763f0 msgr2=0x7fd31c078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:37:38.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.677+0000 7fd333fff640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fd31c0763f0 0x7fd31c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.719 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:38.721+0000 7fdc6b771640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.719 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:38.721+0000 7fdc6b771640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.719 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:38.721+0000 7fdc6b771640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.720 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:38.722+0000 7fdc6b771640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.720 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:38.722+0000 7fdc6b771640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.720 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:38.722+0000 7fdc6876b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.720 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:38.722+0000 7fdc6876b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.720 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:38.722+0000 7fdc6876b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.720 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:38.722+0000 7fdc6876b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.720 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:37:38.722+0000 7fdc6876b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:37:38.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.788+0000 7fd342009640 1 -- 172.21.15.81:0/3568554138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd33c0734a0 con 0x7fd33c0ad180 2024-06-29T14:37:38.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.789+0000 7fd333fff640 1 -- 172.21.15.81:0/3568554138 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7fd324094390 con 0x7fd33c0ad180 2024-06-29T14:37:38.786 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:37:38.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.791+0000 7fd342009640 1 -- 172.21.15.81:0/3568554138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd33c0ad180 msgr2=0x7fd33c0ab660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.791+0000 7fd342009640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd33c0ad180 0x7fd33c0ab660 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fd324011ae0 tx=0x7fd324009e40 comp rx=0 tx=0).stop 2024-06-29T14:37:38.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.791+0000 7fd342009640 1 -- 172.21.15.81:0/3568554138 shutdown_connections 2024-06-29T14:37:38.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.791+0000 7fd342009640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd33c1e6b40 0x7fd33c1e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.791+0000 7fd342009640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd33c0ad180 0x7fd33c0ab660 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.791+0000 7fd342009640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:6832/2940523935,v1:172.21.15.167:6833/2940523935] conn(0x7fd31c0763f0 0x7fd31c078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.791+0000 7fd342009640 1 --2- 172.21.15.81:0/3568554138 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3280051c0 0x7fd33c0ab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.791+0000 7fd342009640 1 -- 172.21.15.81:0/3568554138 >> 172.21.15.81:0/3568554138 conn(0x7fd33c0d0b70 msgr2=0x7fd33c0d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.791+0000 7fd342009640 1 -- 172.21.15.81:0/3568554138 shutdown_connections 2024-06-29T14:37:38.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.791+0000 7fd342009640 1 -- 172.21.15.81:0/3568554138 wait complete. 2024-06-29T14:37:38.800 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:38.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.918+0000 7f3f1932a640 1 -- 172.21.15.81:0/1819570796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3efc004d40 msgr2=0x7f3efc0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.918+0000 7f3f1932a640 1 --2- 172.21.15.81:0/1819570796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3efc004d40 0x7f3efc0051f0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c005fd0 tx=0x7f3efc005830 comp rx=0 tx=0).stop 2024-06-29T14:37:38.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.918+0000 7f3f137fe640 1 -- 172.21.15.81:0/1819570796 reap_dead start 2024-06-29T14:37:38.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.918+0000 7f3f1932a640 1 -- 172.21.15.81:0/1819570796 shutdown_connections 2024-06-29T14:37:38.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.918+0000 7f3f1932a640 1 -- 172.21.15.81:0/1819570796 >> 172.21.15.81:0/1819570796 conn(0x7f3f140d0930 msgr2=0x7f3f140d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:38.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.918+0000 7f3f1932a640 1 -- 172.21.15.81:0/1819570796 shutdown_connections 2024-06-29T14:37:38.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.918+0000 7f3f1932a640 1 -- 172.21.15.81:0/1819570796 wait complete. 2024-06-29T14:37:38.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.920+0000 7f3f1932a640 1 Processor -- start 2024-06-29T14:37:38.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.920+0000 7f3f1932a640 1 -- start start 2024-06-29T14:37:38.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.921+0000 7f3f1932a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3efc004d40 0x7f3f1414e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.921+0000 7f3f1932a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f141f4fd0 0x7f3f140b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.921+0000 7f3f1932a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f141f53e0 0x7f3f140b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:38.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.921+0000 7f3f1932a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f140b41f0 con 0x7f3efc004d40 2024-06-29T14:37:38.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.921+0000 7f3f1932a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3f140b4070 con 0x7f3f141f53e0 2024-06-29T14:37:38.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.921+0000 7f3f1932a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3f140b4370 con 0x7f3f141f4fd0 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.921+0000 7f3f12ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3efc004d40 0x7f3f1414e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.921+0000 7f3f12ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3efc004d40 0x7f3f1414e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57782/0 (socket says 172.21.15.81:57782) 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.921+0000 7f3f12ffd640 1 -- 172.21.15.81:0/2221831993 learned_addr learned my addr 172.21.15.81:0/2221831993 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3f12ffd640 1 -- 172.21.15.81:0/2221831993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f141f4fd0 msgr2=0x7f3f140b09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3f127fc640 1 --2- 172.21.15.81:0/2221831993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f141f4fd0 0x7f3f140b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3f137fe640 1 --2- 172.21.15.81:0/2221831993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f141f53e0 0x7f3f140b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3f12ffd640 1 --2- 172.21.15.81:0/2221831993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f141f4fd0 0x7f3f140b09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3f12ffd640 1 -- 172.21.15.81:0/2221831993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f141f53e0 msgr2=0x7f3f140b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3f12ffd640 1 --2- 172.21.15.81:0/2221831993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f141f53e0 0x7f3f140b0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3f12ffd640 1 -- 172.21.15.81:0/2221831993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f141537f0 con 0x7f3efc004d40 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3f12ffd640 1 --2- 172.21.15.81:0/2221831993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3efc004d40 0x7f3f1414e330 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f3f04003900 tx=0x7f3f04003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3efb7fe640 1 -- 172.21.15.81:0/2221831993 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04011b60 con 0x7f3efc004d40 2024-06-29T14:37:38.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3efb7fe640 1 -- 172.21.15.81:0/2221831993 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f04018020 con 0x7f3efc004d40 2024-06-29T14:37:38.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3f1932a640 1 -- 172.21.15.81:0/2221831993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f1414e870 con 0x7f3efc004d40 2024-06-29T14:37:38.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3efb7fe640 1 -- 172.21.15.81:0/2221831993 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04013450 con 0x7f3efc004d40 2024-06-29T14:37:38.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.922+0000 7f3f1932a640 1 -- 172.21.15.81:0/2221831993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f140ab4d0 con 0x7f3efc004d40 2024-06-29T14:37:38.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.923+0000 7f3f1932a640 1 -- 172.21.15.81:0/2221831993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f1406e2a0 con 0x7f3efc004d40 2024-06-29T14:37:38.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.924+0000 7f3efb7fe640 1 -- 172.21.15.81:0/2221831993 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 35) ==== 97912+0+0 (secure 0 0 0) 0x7f3f040135f0 con 0x7f3efc004d40 2024-06-29T14:37:38.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.924+0000 7f3efb7fe640 1 -- 172.21.15.81:0/2221831993 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6469+0+0 (secure 0 0 0) 0x7f3f0409a7c0 con 0x7f3efc004d40 2024-06-29T14:37:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:38.929+0000 7f3efb7fe640 1 -- 172.21.15.81:0/2221831993 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3f1406e2a0 con 0x7f3efc004d40 2024-06-29T14:37:39.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.149+0000 7f3f1932a640 1 -- 172.21.15.81:0/2221831993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f140b4900 con 0x7f3efc004d40 2024-06-29T14:37:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.149+0000 7f3efb7fe640 1 -- 172.21.15.81:0/2221831993 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5226 (secure 0 0 0) 0x7f3f04064e20 con 0x7f3efc004d40 2024-06-29T14:37:39.147 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:39.147 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"btime":"2024-06-29T14:37:24:340402+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3148404249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3148404249},{"type":"v1","addr":"172.21.15.167:6836","nonce":3148404249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6549,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/704367026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":704367026},{"type":"v1","addr":"172.21.15.81:6834","nonce":704367026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6552,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2316096753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2316096753},{"type":"v1","addr":"172.21.15.81:6835","nonce":2316096753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.692072+0000","modified":"2024-06-29T14:37:23.709021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6567":{"gid":6567,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/2723173791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2723173791},{"type":"v1","addr":"172.21.15.167:6837","nonce":2723173791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6567,"qdb_cluster":[6567]},"id":4}]} 2024-06-29T14:37:39.147 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 34 2024-06-29T14:37:39.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.152+0000 7f3f1932a640 1 -- 172.21.15.81:0/2221831993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3efc004d40 msgr2=0x7f3f1414e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.152+0000 7f3f1932a640 1 --2- 172.21.15.81:0/2221831993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3efc004d40 0x7f3f1414e330 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f3f04003900 tx=0x7f3f04003dc0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.152+0000 7f3f1932a640 1 -- 172.21.15.81:0/2221831993 shutdown_connections 2024-06-29T14:37:39.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.152+0000 7f3f1932a640 1 --2- 172.21.15.81:0/2221831993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f141f53e0 0x7f3f140b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.152+0000 7f3f1932a640 1 --2- 172.21.15.81:0/2221831993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f141f4fd0 0x7f3f140b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.152+0000 7f3f1932a640 1 --2- 172.21.15.81:0/2221831993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3efc004d40 0x7f3f1414e330 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.152+0000 7f3f1932a640 1 -- 172.21.15.81:0/2221831993 >> 172.21.15.81:0/2221831993 conn(0x7f3f140d0930 msgr2=0x7f3f1415a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.152+0000 7f3f1932a640 1 -- 172.21.15.81:0/2221831993 shutdown_connections 2024-06-29T14:37:39.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.152+0000 7f3f1932a640 1 -- 172.21.15.81:0/2221831993 wait complete. 2024-06-29T14:37:39.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:39.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.276+0000 7f9fa0240640 1 -- 172.21.15.81:0/1267516873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84004dd0 msgr2=0x7f9f840052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.276+0000 7f9fa0240640 1 --2- 172.21.15.81:0/1267516873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84004dd0 0x7f9f840052a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9f90001f30 tx=0x7f9f84005830 comp rx=0 tx=0).stop 2024-06-29T14:37:39.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.276+0000 7f9f9e7dd640 1 -- 172.21.15.81:0/1267516873 reap_dead start 2024-06-29T14:37:39.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.276+0000 7f9fa0240640 1 -- 172.21.15.81:0/1267516873 shutdown_connections 2024-06-29T14:37:39.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.276+0000 7f9fa0240640 1 -- 172.21.15.81:0/1267516873 >> 172.21.15.81:0/1267516873 conn(0x7f9f980d0930 msgr2=0x7f9f980d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.276+0000 7f9fa0240640 1 -- 172.21.15.81:0/1267516873 shutdown_connections 2024-06-29T14:37:39.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.276+0000 7f9fa0240640 1 -- 172.21.15.81:0/1267516873 wait complete. 2024-06-29T14:37:39.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.278+0000 7f9fa0240640 1 Processor -- start 2024-06-29T14:37:39.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.278+0000 7f9fa0240640 1 -- start start 2024-06-29T14:37:39.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.279+0000 7f9fa0240640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f84004dd0 0x7f9f980ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.279+0000 7f9fa0240640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f980aed70 0x7f9f98153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.279+0000 7f9fa0240640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f98153eb0 0x7f9f981542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.279+0000 7f9fa0240640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9815b260 con 0x7f9f98153eb0 2024-06-29T14:37:39.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.279+0000 7f9fa0240640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9815b0e0 con 0x7f9f980aed70 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.279+0000 7f9fa0240640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9f9815b3e0 con 0x7f9f84004dd0 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.279+0000 7f9f9e7dd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f98153eb0 0x7f9f981542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.279+0000 7f9f9e7dd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f98153eb0 0x7f9f981542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57794/0 (socket says 172.21.15.81:57794) 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.279+0000 7f9f9e7dd640 1 -- 172.21.15.81:0/3601703916 learned_addr learned my addr 172.21.15.81:0/3601703916 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.279+0000 7f9f9d7db640 1 --2- 172.21.15.81:0/3601703916 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f980aed70 0x7f9f98153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f9e7dd640 1 -- 172.21.15.81:0/3601703916 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f84004dd0 msgr2=0x7f9f980ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f9dfdc640 1 --2- 172.21.15.81:0/3601703916 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f84004dd0 0x7f9f980ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f9e7dd640 1 --2- 172.21.15.81:0/3601703916 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f84004dd0 0x7f9f980ae830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f9e7dd640 1 -- 172.21.15.81:0/3601703916 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f980aed70 msgr2=0x7f9f98153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f9e7dd640 1 --2- 172.21.15.81:0/3601703916 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f980aed70 0x7f9f98153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f9e7dd640 1 -- 172.21.15.81:0/3601703916 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f98154800 con 0x7f9f98153eb0 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f9d7db640 1 --2- 172.21.15.81:0/3601703916 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f980aed70 0x7f9f98153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f9e7dd640 1 --2- 172.21.15.81:0/3601703916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f98153eb0 0x7f9f981542c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9f90000f80 tx=0x7f9f90006cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f8e7fc640 1 -- 172.21.15.81:0/3601703916 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90037030 con 0x7f9f98153eb0 2024-06-29T14:37:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f8e7fc640 1 -- 172.21.15.81:0/3601703916 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f900060b0 con 0x7f9f98153eb0 2024-06-29T14:37:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9fa0240640 1 -- 172.21.15.81:0/3601703916 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f980af560 con 0x7f9f98153eb0 2024-06-29T14:37:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9f8e7fc640 1 -- 172.21.15.81:0/3601703916 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f900064c0 con 0x7f9f98153eb0 2024-06-29T14:37:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.280+0000 7f9fa0240640 1 -- 172.21.15.81:0/3601703916 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f98153170 con 0x7f9f98153eb0 2024-06-29T14:37:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.281+0000 7f9fa0240640 1 -- 172.21.15.81:0/3601703916 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f9806e2a0 con 0x7f9f98153eb0 2024-06-29T14:37:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.282+0000 7f9f8e7fc640 1 -- 172.21.15.81:0/3601703916 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 35) ==== 97912+0+0 (secure 0 0 0) 0x7f9f90003a30 con 0x7f9f98153eb0 2024-06-29T14:37:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.282+0000 7f9f8e7fc640 1 -- 172.21.15.81:0/3601703916 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6469+0+0 (secure 0 0 0) 0x7f9f900b7150 con 0x7f9f98153eb0 2024-06-29T14:37:39.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.287+0000 7f9f8e7fc640 1 -- 172.21.15.81:0/3601703916 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9f9806e2a0 con 0x7f9f98153eb0 2024-06-29T14:37:39.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.465+0000 7f9fa0240640 1 -- 172.21.15.81:0/3601703916 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f981549d0 con 0x7f9f98153eb0 2024-06-29T14:37:39.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.466+0000 7f9f8e7fc640 1 -- 172.21.15.81:0/3601703916 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+17778 (secure 0 0 0) 0x7f9f981549d0 con 0x7f9f98153eb0 2024-06-29T14:37:39.464 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:39.464 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":80,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:38.662535+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:18.983445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.642471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3387399405":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/665800877":"2024-06-30T14:37:38.662497+0000","172.21.15.167:6833/2940523935":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/470342636":"2024-06-30T14:37:38.662497+0000","172.21.15.167:6832/2940523935":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/2903530506":"2024-06-30T14:37:38.662497+0000","172.21.15.167:6833/3693496302":"2024-06-30T14:37:32.992869+0000","172.21.15.167:6832/3693496302":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/912055670":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/1220706198":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/538343225":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/1189302834":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/3217872433":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/53505028":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/2826882843":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/95469329":"2024-06-30T14:37:32.992869+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:39.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.468+0000 7f9fa0240640 1 -- 172.21.15.81:0/3601703916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f98153eb0 msgr2=0x7f9f981542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.468+0000 7f9fa0240640 1 --2- 172.21.15.81:0/3601703916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f98153eb0 0x7f9f981542c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9f90000f80 tx=0x7f9f90006cd0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.468+0000 7f9fa0240640 1 -- 172.21.15.81:0/3601703916 shutdown_connections 2024-06-29T14:37:39.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.468+0000 7f9fa0240640 1 --2- 172.21.15.81:0/3601703916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f98153eb0 0x7f9f981542c0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.468+0000 7f9fa0240640 1 --2- 172.21.15.81:0/3601703916 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f980aed70 0x7f9f98153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.468+0000 7f9fa0240640 1 --2- 172.21.15.81:0/3601703916 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f84004dd0 0x7f9f980ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.468+0000 7f9fa0240640 1 -- 172.21.15.81:0/3601703916 >> 172.21.15.81:0/3601703916 conn(0x7f9f980d0930 msgr2=0x7f9f98159500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.468+0000 7f9fa0240640 1 -- 172.21.15.81:0/3601703916 shutdown_connections 2024-06-29T14:37:39.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.468+0000 7f9fa0240640 1 -- 172.21.15.81:0/3601703916 wait complete. 2024-06-29T14:37:39.477 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:37:39.477 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:39.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.593+0000 7fae11baa640 1 -- 172.21.15.81:0/3265261845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fadf4003a50 msgr2=0x7fadf4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.593+0000 7fae11baa640 1 --2- 172.21.15.81:0/3265261845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fadf4003a50 0x7fadf4003ec0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fadfc0016c0 tx=0x7fadf4005830 comp rx=0 tx=0).stop 2024-06-29T14:37:39.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.593+0000 7fae11baa640 1 -- 172.21.15.81:0/3265261845 shutdown_connections 2024-06-29T14:37:39.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.593+0000 7fae11baa640 1 --2- 172.21.15.81:0/3265261845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fadf4003a50 0x7fadf4003ec0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.593+0000 7fae11baa640 1 -- 172.21.15.81:0/3265261845 >> 172.21.15.81:0/3265261845 conn(0x7fae0c0d3bd0 msgr2=0x7fae0c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.593+0000 7fae11baa640 1 -- 172.21.15.81:0/3265261845 shutdown_connections 2024-06-29T14:37:39.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.594+0000 7fae11baa640 1 -- 172.21.15.81:0/3265261845 wait complete. 2024-06-29T14:37:39.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.595+0000 7fae11baa640 1 Processor -- start 2024-06-29T14:37:39.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.595+0000 7fae11baa640 1 -- start start 2024-06-29T14:37:39.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae11baa640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fadf4003a50 0x7fae0c0caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae11baa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae0c0cdc30 0x7fae0c151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae11baa640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae0c09fa70 0x7fae0c09fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae11baa640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c0b9410 con 0x7fae0c0cdc30 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae11baa640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fae0c0b9290 con 0x7fadf4003a50 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae11baa640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fae0c0d0410 con 0x7fae0c09fa70 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae0c0cdc30 0x7fae0c151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae0c0cdc30 0x7fae0c151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57814/0 (socket says 172.21.15.81:57814) 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0affd640 1 -- 172.21.15.81:0/2428192199 learned_addr learned my addr 172.21.15.81:0/2428192199 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0b7fe640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fadf4003a50 0x7fae0c0caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0bfff640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae0c09fa70 0x7fae0c09fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0affd640 1 -- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae0c09fa70 msgr2=0x7fae0c09fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0affd640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae0c09fa70 0x7fae0c09fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0affd640 1 -- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fadf4003a50 msgr2=0x7fae0c0caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0affd640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fadf4003a50 0x7fae0c0caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0affd640 1 -- 172.21.15.81:0/2428192199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0c0a03c0 con 0x7fae0c0cdc30 2024-06-29T14:37:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0bfff640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae0c09fa70 0x7fae0c09fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae0affd640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae0c0cdc30 0x7fae0c151e20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fae00010890 tx=0x7fae00010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fadf3fff640 1 -- 172.21.15.81:0/2428192199 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0000ece0 con 0x7fae0c0cdc30 2024-06-29T14:37:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fadf3fff640 1 -- 172.21.15.81:0/2428192199 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae00012450 con 0x7fae0c0cdc30 2024-06-29T14:37:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fae11baa640 1 -- 172.21.15.81:0/2428192199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae0c0cb790 con 0x7fae0c0cdc30 2024-06-29T14:37:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.597+0000 7fadf3fff640 1 -- 172.21.15.81:0/2428192199 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00012860 con 0x7fae0c0cdc30 2024-06-29T14:37:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.598+0000 7fae11baa640 1 -- 172.21.15.81:0/2428192199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae0c0cbd50 con 0x7fae0c0cdc30 2024-06-29T14:37:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.598+0000 7fae11baa640 1 -- 172.21.15.81:0/2428192199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae0c0b9700 con 0x7fae0c0cdc30 2024-06-29T14:37:39.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.599+0000 7fadf3fff640 1 -- 172.21.15.81:0/2428192199 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 35) ==== 97912+0+0 (secure 0 0 0) 0x7fae00012a00 con 0x7fae0c0cdc30 2024-06-29T14:37:39.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.600+0000 7fadf3fff640 1 -- 172.21.15.81:0/2428192199 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6469+0+0 (secure 0 0 0) 0x7fae00098110 con 0x7fae0c0cdc30 2024-06-29T14:37:39.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.604+0000 7fadf3fff640 1 -- 172.21.15.81:0/2428192199 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fae0c0a03c0 con 0x7fae0c0cdc30 2024-06-29T14:37:39.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.674+0000 7fadf3fff640 1 -- 172.21.15.81:0/2428192199 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 36) ==== 98039+0+0 (secure 0 0 0) 0x7fae00012cf0 con 0x7fae0c0cdc30 2024-06-29T14:37:39.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.674+0000 7fadf3fff640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fadf807bae0 0x7fadf807df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.675+0000 7fae0b7fe640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fadf807bae0 0x7fadf807df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.676+0000 7fae0b7fe640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fadf807bae0 0x7fadf807df00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fadfc032000 tx=0x7fadfc02c040 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.824+0000 7fae11baa640 1 -- 172.21.15.81:0/2428192199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae0c0cb520 con 0x7fae0c0cdc30 2024-06-29T14:37:39.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.824+0000 7fadf3fff640 1 -- 172.21.15.81:0/2428192199 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5226 (secure 0 0 0) 0x7fae00012820 con 0x7fae0c0cdc30 2024-06-29T14:37:39.822 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:39.822 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"btime":"2024-06-29T14:37:24:340402+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3148404249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3148404249},{"type":"v1","addr":"172.21.15.167:6836","nonce":3148404249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6549,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/704367026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":704367026},{"type":"v1","addr":"172.21.15.81:6834","nonce":704367026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6552,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2316096753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2316096753},{"type":"v1","addr":"172.21.15.81:6835","nonce":2316096753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.692072+0000","modified":"2024-06-29T14:37:23.709021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6567":{"gid":6567,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/2723173791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2723173791},{"type":"v1","addr":"172.21.15.167:6837","nonce":2723173791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6567,"qdb_cluster":[6567]},"id":4}]} 2024-06-29T14:37:39.822 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 34 2024-06-29T14:37:39.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.826+0000 7fae11baa640 1 -- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fadf807bae0 msgr2=0x7fadf807df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.826+0000 7fae11baa640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fadf807bae0 0x7fadf807df00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fadfc032000 tx=0x7fadfc02c040 comp rx=0 tx=0).stop 2024-06-29T14:37:39.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.826+0000 7fae11baa640 1 -- 172.21.15.81:0/2428192199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae0c0cdc30 msgr2=0x7fae0c151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.826+0000 7fae11baa640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae0c0cdc30 0x7fae0c151e20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fae00010890 tx=0x7fae00010d50 comp rx=0 tx=0).stop 2024-06-29T14:37:39.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.826+0000 7fae11baa640 1 -- 172.21.15.81:0/2428192199 shutdown_connections 2024-06-29T14:37:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.826+0000 7fae11baa640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fadf807bae0 0x7fadf807df00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.826+0000 7fae11baa640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fae0c09fa70 0x7fae0c09fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.826+0000 7fae11baa640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae0c0cdc30 0x7fae0c151e20 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.826+0000 7fae11baa640 1 --2- 172.21.15.81:0/2428192199 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fadf4003a50 0x7fae0c0caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.826+0000 7fae11baa640 1 -- 172.21.15.81:0/2428192199 >> 172.21.15.81:0/2428192199 conn(0x7fae0c0d3bd0 msgr2=0x7fae0c1618a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.827+0000 7fae11baa640 1 -- 172.21.15.81:0/2428192199 shutdown_connections 2024-06-29T14:37:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.827+0000 7fae11baa640 1 -- 172.21.15.81:0/2428192199 wait complete. 2024-06-29T14:37:39.835 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:39.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.950+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/2642803297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c780ce510 msgr2=0x7f8c780ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.950+0000 7f8c7fe8f640 1 --2- 172.21.15.81:0/2642803297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c780ce510 0x7f8c780ce920 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f8c780b41b0 tx=0x7f8c6802a520 comp rx=0 tx=0).stop 2024-06-29T14:37:39.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.950+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/2642803297 shutdown_connections 2024-06-29T14:37:39.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.951+0000 7f8c7fe8f640 1 --2- 172.21.15.81:0/2642803297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c780ce510 0x7f8c780ce920 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.951+0000 7f8c7fe8f640 1 --2- 172.21.15.81:0/2642803297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c780b6eb0 0x7f8c780d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.951+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/2642803297 >> 172.21.15.81:0/2642803297 conn(0x7f8c780b3510 msgr2=0x7f8c780b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:39.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.951+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/2642803297 shutdown_connections 2024-06-29T14:37:39.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.951+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/2642803297 wait complete. 2024-06-29T14:37:39.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.952+0000 7f8c7fe8f640 1 Processor -- start 2024-06-29T14:37:39.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.952+0000 7f8c7fe8f640 1 -- start start 2024-06-29T14:37:39.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7fe8f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c780b4810 0x7f8c7814aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7fe8f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c780b6eb0 0x7f8c7815f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7fe8f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c780ce510 0x7f8c780b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7fe8f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c780cd480 con 0x7f8c780b4810 2024-06-29T14:37:39.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7fe8f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8c780b29c0 con 0x7f8c780ce510 2024-06-29T14:37:39.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7fe8f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8c780cd600 con 0x7f8c780b6eb0 2024-06-29T14:37:39.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7dc2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c780b4810 0x7f8c7814aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7dc2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c780b4810 0x7f8c7814aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57828/0 (socket says 172.21.15.81:57828) 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7dc2b640 1 -- 172.21.15.81:0/416932154 learned_addr learned my addr 172.21.15.81:0/416932154 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7e42c640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c780ce510 0x7f8c780b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7d42a640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c780b6eb0 0x7f8c7815f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7dc2b640 1 -- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c780b6eb0 msgr2=0x7f8c7815f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7dc2b640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c780b6eb0 0x7f8c7815f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7dc2b640 1 -- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c780ce510 msgr2=0x7f8c780b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7dc2b640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c780ce510 0x7f8c780b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7dc2b640 1 -- 172.21.15.81:0/416932154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7815f820 con 0x7f8c780b4810 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7dc2b640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c780b4810 0x7f8c7814aeb0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d780 tx=0x7f8c7400dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c6e7fc640 1 -- 172.21.15.81:0/416932154 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74017070 con 0x7f8c780b4810 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c6e7fc640 1 -- 172.21.15.81:0/416932154 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c7400de90 con 0x7f8c780b4810 2024-06-29T14:37:39.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/416932154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c780c9ad0 con 0x7f8c780b4810 2024-06-29T14:37:39.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.954+0000 7f8c6e7fc640 1 -- 172.21.15.81:0/416932154 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c740099d0 con 0x7f8c780b4810 2024-06-29T14:37:39.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.955+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/416932154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c78164790 con 0x7f8c780b4810 2024-06-29T14:37:39.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.956+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/416932154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c780d02d0 con 0x7f8c780b4810 2024-06-29T14:37:39.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.956+0000 7f8c6e7fc640 1 -- 172.21.15.81:0/416932154 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 36) ==== 98039+0+0 (secure 0 0 0) 0x7f8c74009b70 con 0x7f8c780b4810 2024-06-29T14:37:39.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.957+0000 7f8c6e7fc640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8c5c076120 0x7f8c5c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:39.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.957+0000 7f8c6e7fc640 1 -- 172.21.15.81:0/416932154 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6469+0+0 (secure 0 0 0) 0x7f8c74097880 con 0x7f8c780b4810 2024-06-29T14:37:39.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.957+0000 7f8c7d42a640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8c5c076120 0x7f8c5c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:39.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.958+0000 7f8c7d42a640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8c5c076120 0x7f8c5c078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8c68033000 tx=0x7f8c68004730 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:39.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:39.961+0000 7f8c6e7fc640 1 -- 172.21.15.81:0/416932154 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8c74020020 con 0x7f8c780b4810 2024-06-29T14:37:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.180+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/416932154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c780cdf00 con 0x7f8c780b4810 2024-06-29T14:37:40.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.180+0000 7f8c6e7fc640 1 -- 172.21.15.81:0/416932154 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5226 (secure 0 0 0) 0x7f8c74069210 con 0x7f8c780b4810 2024-06-29T14:37:40.178 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:40.178 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"btime":"2024-06-29T14:37:24:340402+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3148404249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3148404249},{"type":"v1","addr":"172.21.15.167:6836","nonce":3148404249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6549,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/704367026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":704367026},{"type":"v1","addr":"172.21.15.81:6834","nonce":704367026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6552,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2316096753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2316096753},{"type":"v1","addr":"172.21.15.81:6835","nonce":2316096753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:20.692072+0000","modified":"2024-06-29T14:37:23.709021+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6567":{"gid":6567,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/2723173791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2723173791},{"type":"v1","addr":"172.21.15.167:6837","nonce":2723173791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6567,"qdb_cluster":[6567]},"id":4}]} 2024-06-29T14:37:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 34 2024-06-29T14:37:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8c5c076120 msgr2=0x7f8c5c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8c5c076120 0x7f8c5c078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8c68033000 tx=0x7f8c68004730 comp rx=0 tx=0).stop 2024-06-29T14:37:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/416932154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c780b4810 msgr2=0x7f8c7814aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c780b4810 0x7f8c7814aeb0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d780 tx=0x7f8c7400dc40 comp rx=0 tx=0).stop 2024-06-29T14:37:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/416932154 shutdown_connections 2024-06-29T14:37:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8c5c076120 0x7f8c5c078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c780ce510 0x7f8c780b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c780b6eb0 0x7f8c7815f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 --2- 172.21.15.81:0/416932154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c780b4810 0x7f8c7814aeb0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/416932154 >> 172.21.15.81:0/416932154 conn(0x7f8c780b3510 msgr2=0x7f8c780cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/416932154 shutdown_connections 2024-06-29T14:37:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.182+0000 7f8c7fe8f640 1 -- 172.21.15.81:0/416932154 wait complete. 2024-06-29T14:37:40.191 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:40.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.306+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/2631516969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cbc0b35b0 msgr2=0x7f0cbc15b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.306+0000 7f0cc3b7f640 1 --2- 172.21.15.81:0/2631516969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cbc0b35b0 0x7f0cbc15b5b0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8005fd0 tx=0x7f0cb802b910 comp rx=0 tx=0).stop 2024-06-29T14:37:40.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.306+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/2631516969 shutdown_connections 2024-06-29T14:37:40.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.306+0000 7f0cc3b7f640 1 --2- 172.21.15.81:0/2631516969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cbc0b35b0 0x7f0cbc15b5b0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.306+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/2631516969 >> 172.21.15.81:0/2631516969 conn(0x7f0cbc06ccc0 msgr2=0x7f0cbc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.306+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/2631516969 shutdown_connections 2024-06-29T14:37:40.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.306+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/2631516969 wait complete. 2024-06-29T14:37:40.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.307+0000 7f0cc3b7f640 1 Processor -- start 2024-06-29T14:37:40.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.308+0000 7f0cc3b7f640 1 -- start start 2024-06-29T14:37:40.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc3b7f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cbc06f660 0x7f0cbc150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc3b7f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0cbc0b35b0 0x7f0cbc1f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc3b7f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cbc0b4430 0x7f0cbc0b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc3b7f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc06f3d0 con 0x7f0cbc06f660 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc3b7f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc06f250 con 0x7f0cbc0b4430 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc3b7f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0cbc071f20 con 0x7f0cbc0b35b0 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc191b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cbc06f660 0x7f0cbc150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc191b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cbc06f660 0x7f0cbc150880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45890/0 (socket says 172.21.15.81:45890) 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc191b640 1 -- 172.21.15.81:0/1956966897 learned_addr learned my addr 172.21.15.81:0/1956966897 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc111a640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0cbc0b35b0 0x7f0cbc1f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc191b640 1 -- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0cbc0b35b0 msgr2=0x7f0cbc1f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc211c640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cbc0b4430 0x7f0cbc0b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc191b640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0cbc0b35b0 0x7f0cbc1f62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc191b640 1 -- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cbc0b4430 msgr2=0x7f0cbc0b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc191b640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cbc0b4430 0x7f0cbc0b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc191b640 1 -- 172.21.15.81:0/1956966897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cbc1f6830 con 0x7f0cbc06f660 2024-06-29T14:37:40.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc111a640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0cbc0b35b0 0x7f0cbc1f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:40.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc211c640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cbc0b4430 0x7f0cbc0b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:40.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.309+0000 7f0cc191b640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cbc06f660 0x7f0cbc150880 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8034040 tx=0x7f0cb8004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:40.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.310+0000 7f0cb27fc640 1 -- 172.21.15.81:0/1956966897 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb803e070 con 0x7f0cbc06f660 2024-06-29T14:37:40.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.310+0000 7f0cb27fc640 1 -- 172.21.15.81:0/1956966897 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cb8031900 con 0x7f0cbc06f660 2024-06-29T14:37:40.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.310+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/1956966897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cbc1f60d0 con 0x7f0cbc06f660 2024-06-29T14:37:40.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.310+0000 7f0cb27fc640 1 -- 172.21.15.81:0/1956966897 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8031cf0 con 0x7f0cbc06f660 2024-06-29T14:37:40.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.310+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/1956966897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cbc1f5b10 con 0x7f0cbc06f660 2024-06-29T14:37:40.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.311+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/1956966897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cbc1f6a00 con 0x7f0cbc06f660 2024-06-29T14:37:40.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.311+0000 7f0cb27fc640 1 -- 172.21.15.81:0/1956966897 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 36) ==== 98039+0+0 (secure 0 0 0) 0x7f0cb8038070 con 0x7f0cbc06f660 2024-06-29T14:37:40.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.312+0000 7f0cb27fc640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0ca0076120 0x7f0ca0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.312+0000 7f0cb27fc640 1 -- 172.21.15.81:0/1956966897 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6469+0+0 (secure 0 0 0) 0x7f0cb80b88c0 con 0x7f0cbc06f660 2024-06-29T14:37:40.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.312+0000 7f0cc111a640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0ca0076120 0x7f0ca0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.313+0000 7f0cc111a640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0ca0076120 0x7f0ca0078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc14cc90 tx=0x7f0cac009040 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:40.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.316+0000 7f0cb27fc640 1 -- 172.21.15.81:0/1956966897 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0cb8082260 con 0x7f0cbc06f660 2024-06-29T14:37:40.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.496+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/1956966897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0cbc15b5b0 con 0x7f0cbc06f660 2024-06-29T14:37:40.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.497+0000 7f0cb27fc640 1 -- 172.21.15.81:0/1956966897 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+17778 (secure 0 0 0) 0x7f0cb808a250 con 0x7f0cbc06f660 2024-06-29T14:37:40.494 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:40.495 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":80,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:38.662535+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:18.983445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:19.642471+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3387399405":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/665800877":"2024-06-30T14:37:38.662497+0000","172.21.15.167:6833/2940523935":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/470342636":"2024-06-30T14:37:38.662497+0000","172.21.15.167:6832/2940523935":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/2903530506":"2024-06-30T14:37:38.662497+0000","172.21.15.167:6833/3693496302":"2024-06-30T14:37:32.992869+0000","172.21.15.167:6832/3693496302":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/912055670":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/1220706198":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/538343225":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/1189302834":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/3217872433":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/53505028":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/2826882843":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/95469329":"2024-06-30T14:37:32.992869+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:40.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0ca0076120 msgr2=0x7f0ca0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0ca0076120 0x7f0ca0078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc14cc90 tx=0x7f0cac009040 comp rx=0 tx=0).stop 2024-06-29T14:37:40.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/1956966897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cbc06f660 msgr2=0x7f0cbc150880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cbc06f660 0x7f0cbc150880 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8034040 tx=0x7f0cb8004a50 comp rx=0 tx=0).stop 2024-06-29T14:37:40.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/1956966897 shutdown_connections 2024-06-29T14:37:40.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0ca0076120 0x7f0ca0078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0cbc0b4430 0x7f0cbc0b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0cbc0b35b0 0x7f0cbc1f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 --2- 172.21.15.81:0/1956966897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cbc06f660 0x7f0cbc150880 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/1956966897 >> 172.21.15.81:0/1956966897 conn(0x7f0cbc06ccc0 msgr2=0x7f0cbc15bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/1956966897 shutdown_connections 2024-06-29T14:37:40.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.499+0000 7f0cc3b7f640 1 -- 172.21.15.81:0/1956966897 wait complete. 2024-06-29T14:37:40.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:37:40.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.624+0000 7f3bcaa10640 1 -- 172.21.15.81:0/368328842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bc406f680 msgr2=0x7f3bc406c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.624+0000 7f3bcaa10640 1 --2- 172.21.15.81:0/368328842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bc406f680 0x7f3bc406c2f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3bc40726b0 tx=0x7f3bb802a560 comp rx=0 tx=0).stop 2024-06-29T14:37:40.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.624+0000 7f3bcaa10640 1 -- 172.21.15.81:0/368328842 shutdown_connections 2024-06-29T14:37:40.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.624+0000 7f3bcaa10640 1 --2- 172.21.15.81:0/368328842 >> v2:172.21.15.167:3300/0 conn(0x7f3bc406fa90 0x7f3bc406c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.624+0000 7f3bcaa10640 1 --2- 172.21.15.81:0/368328842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bc406f680 0x7f3bc406c2f0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.624+0000 7f3bcaa10640 1 --2- 172.21.15.81:0/368328842 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3bc4074520 0x7f3bc406f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.624+0000 7f3bcaa10640 1 -- 172.21.15.81:0/368328842 >> 172.21.15.81:0/368328842 conn(0x7f3bc4072950 msgr2=0x7f3bc40b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.624+0000 7f3bcaa10640 1 -- 172.21.15.81:0/368328842 shutdown_connections 2024-06-29T14:37:40.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.624+0000 7f3bcaa10640 1 -- 172.21.15.81:0/368328842 wait complete. 2024-06-29T14:37:40.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.626+0000 7f3bcaa10640 1 Processor -- start 2024-06-29T14:37:40.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.626+0000 7f3bcaa10640 1 -- start start 2024-06-29T14:37:40.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.627+0000 7f3bcaa10640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3bc406f680 0x7f3bc414f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.627+0000 7f3bcaa10640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3bc406fa90 0x7f3bc4154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.627+0000 7f3bcaa10640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bc4074520 0x7f3bc41482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.627+0000 7f3bcaa10640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc406cd70 con 0x7f3bc4074520 2024-06-29T14:37:40.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.627+0000 7f3bcaa10640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc415b220 con 0x7f3bc406fa90 2024-06-29T14:37:40.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.627+0000 7f3bcaa10640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc406cef0 con 0x7f3bc406f680 2024-06-29T14:37:40.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.627+0000 7f3bc8fad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bc4074520 0x7f3bc41482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.627+0000 7f3bc8fad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bc4074520 0x7f3bc41482e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45900/0 (socket says 172.21.15.81:45900) 2024-06-29T14:37:40.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.627+0000 7f3bc8fad640 1 -- 172.21.15.81:0/2193600749 learned_addr learned my addr 172.21.15.81:0/2193600749 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc3fff640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3bc406f680 0x7f3bc414f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc37fe640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3bc406fa90 0x7f3bc4154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc8fad640 1 -- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3bc406f680 msgr2=0x7f3bc414f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc8fad640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3bc406f680 0x7f3bc414f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc8fad640 1 -- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3bc406fa90 msgr2=0x7f3bc4154250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc8fad640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3bc406fa90 0x7f3bc4154250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc8fad640 1 -- 172.21.15.81:0/2193600749 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc4150580 con 0x7f3bc4074520 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc8fad640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bc4074520 0x7f3bc41482e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00d9b0 tx=0x7f3bbc00de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc0ff9640 1 -- 172.21.15.81:0/2193600749 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc016070 con 0x7f3bc4074520 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc0ff9640 1 -- 172.21.15.81:0/2193600749 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bbc00bce0 con 0x7f3bc4074520 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bcaa10640 1 -- 172.21.15.81:0/2193600749 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc4150860 con 0x7f3bc4074520 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bc0ff9640 1 -- 172.21.15.81:0/2193600749 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc003bd0 con 0x7f3bc4074520 2024-06-29T14:37:40.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.628+0000 7f3bcaa10640 1 -- 172.21.15.81:0/2193600749 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc4150e20 con 0x7f3bc4074520 2024-06-29T14:37:40.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.629+0000 7f3bcaa10640 1 -- 172.21.15.81:0/2193600749 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc41502f0 con 0x7f3bc4074520 2024-06-29T14:37:40.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.630+0000 7f3bc0ff9640 1 -- 172.21.15.81:0/2193600749 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 36) ==== 98039+0+0 (secure 0 0 0) 0x7f3bbc010460 con 0x7f3bc4074520 2024-06-29T14:37:40.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.630+0000 7f3bc0ff9640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3bb0076150 0x7f3bb0078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.630+0000 7f3bc0ff9640 1 -- 172.21.15.81:0/2193600749 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6469+0+0 (secure 0 0 0) 0x7f3bbc097b60 con 0x7f3bc4074520 2024-06-29T14:37:40.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.631+0000 7f3bc3fff640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3bb0076150 0x7f3bb0078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.631+0000 7f3bc3fff640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3bb0076150 0x7f3bb0078660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009b30 tx=0x7f3bb4005f90 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:40.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.635+0000 7f3bc0ff9640 1 -- 172.21.15.81:0/2193600749 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3bbc060e20 con 0x7f3bc4074520 2024-06-29T14:37:40.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.854+0000 7f3bcaa10640 1 -- 172.21.15.81:0/2193600749 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3bc406f140 con 0x7f3bc4074520 2024-06-29T14:37:40.852 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T14:37:40.855+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:37:40.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.856+0000 7f3bc0ff9640 1 -- 172.21.15.81:0/2193600749 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) ==== 161+0+0 (secure 0 0 0) 0x7f3bbc097070 con 0x7f3bc4074520 2024-06-29T14:37:40.853 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:37:40.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 -- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3bb0076150 msgr2=0x7f3bb0078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3bb0076150 0x7f3bb0078660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009b30 tx=0x7f3bb4005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:40.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 -- 172.21.15.81:0/2193600749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bc4074520 msgr2=0x7f3bc41482e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bc4074520 0x7f3bc41482e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00d9b0 tx=0x7f3bbc00de70 comp rx=0 tx=0).stop 2024-06-29T14:37:40.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 -- 172.21.15.81:0/2193600749 shutdown_connections 2024-06-29T14:37:40.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3bb0076150 0x7f3bb0078660 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bc4074520 0x7f3bc41482e0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3bc406fa90 0x7f3bc4154250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 --2- 172.21.15.81:0/2193600749 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3bc406f680 0x7f3bc414f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 -- 172.21.15.81:0/2193600749 >> 172.21.15.81:0/2193600749 conn(0x7f3bc4072950 msgr2=0x7f3bc40afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 -- 172.21.15.81:0/2193600749 shutdown_connections 2024-06-29T14:37:40.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.859+0000 7f3bcaa10640 1 -- 172.21.15.81:0/2193600749 wait complete. 2024-06-29T14:37:40.859 INFO:tasks.ceph.mds.b.smithi167.stderr: -4121> 2024-06-29T14:37:23.301+0000 7f57460dc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:37:40.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:37:40.915 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:37:40.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.986+0000 7f4e379d4640 1 -- 172.21.15.81:0/3549535788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e30071f20 msgr2=0x7f4e30155480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.986+0000 7f4e379d4640 1 --2- 172.21.15.81:0/3549535788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e30071f20 0x7f4e30155480 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f4e30071130 tx=0x7f4e2c02a5f0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.986+0000 7f4e379d4640 1 -- 172.21.15.81:0/3549535788 shutdown_connections 2024-06-29T14:37:40.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.986+0000 7f4e379d4640 1 --2- 172.21.15.81:0/3549535788 >> v2:172.21.15.167:3300/0 conn(0x7f4e3015bf20 0x7f4e3006f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.986+0000 7f4e379d4640 1 --2- 172.21.15.81:0/3549535788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e30071f20 0x7f4e30155480 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.986+0000 7f4e379d4640 1 -- 172.21.15.81:0/3549535788 >> 172.21.15.81:0/3549535788 conn(0x7f4e300b1730 msgr2=0x7f4e300b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:40.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.986+0000 7f4e379d4640 1 -- 172.21.15.81:0/3549535788 shutdown_connections 2024-06-29T14:37:40.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.986+0000 7f4e379d4640 1 -- 172.21.15.81:0/3549535788 wait complete. 2024-06-29T14:37:40.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.988+0000 7f4e379d4640 1 Processor -- start 2024-06-29T14:37:40.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.988+0000 7f4e379d4640 1 -- start start 2024-06-29T14:37:40.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.989+0000 7f4e379d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e3006fca0 0x7f4e300aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.989+0000 7f4e379d4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e30071f20 0x7f4e300ab200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.989+0000 7f4e379d4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e3015bf20 0x7f4e301f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.989+0000 7f4e379d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3015f1a0 con 0x7f4e3006fca0 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.989+0000 7f4e379d4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4e300b40a0 con 0x7f4e30071f20 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.989+0000 7f4e379d4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4e3015f320 con 0x7f4e3015bf20 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.989+0000 7f4e35770640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e3006fca0 0x7f4e300aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.989+0000 7f4e35770640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e3006fca0 0x7f4e300aacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45922/0 (socket says 172.21.15.81:45922) 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.989+0000 7f4e35770640 1 -- 172.21.15.81:0/3580450724 learned_addr learned my addr 172.21.15.81:0/3580450724 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e34f6f640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e30071f20 0x7f4e300ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e35f71640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e3015bf20 0x7f4e301f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e35770640 1 -- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e3015bf20 msgr2=0x7f4e301f5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e35770640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e3015bf20 0x7f4e301f5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e35770640 1 -- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e30071f20 msgr2=0x7f4e300ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e35770640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e30071f20 0x7f4e300ab200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e35770640 1 -- 172.21.15.81:0/3580450724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e300d08a0 con 0x7f4e3006fca0 2024-06-29T14:37:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e34f6f640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e30071f20 0x7f4e300ab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e35f71640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e3015bf20 0x7f4e301f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e35770640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e3006fca0 0x7f4e300aacc0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c034040 tx=0x7f4e2c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e25ffb640 1 -- 172.21.15.81:0/3580450724 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c035040 con 0x7f4e3006fca0 2024-06-29T14:37:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e25ffb640 1 -- 172.21.15.81:0/3580450724 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e2c003dd0 con 0x7f4e3006fca0 2024-06-29T14:37:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e379d4640 1 -- 172.21.15.81:0/3580450724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e300cfc10 con 0x7f4e3006fca0 2024-06-29T14:37:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e25ffb640 1 -- 172.21.15.81:0/3580450724 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0049c0 con 0x7f4e3006fca0 2024-06-29T14:37:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.990+0000 7f4e379d4640 1 -- 172.21.15.81:0/3580450724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e301f1ab0 con 0x7f4e3006fca0 2024-06-29T14:37:40.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.991+0000 7f4e379d4640 1 -- 172.21.15.81:0/3580450724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e3015f840 con 0x7f4e3006fca0 2024-06-29T14:37:40.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.992+0000 7f4e25ffb640 1 -- 172.21.15.81:0/3580450724 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 36) ==== 98039+0+0 (secure 0 0 0) 0x7f4e2c004b60 con 0x7f4e3006fca0 2024-06-29T14:37:40.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.992+0000 7f4e25ffb640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f4e14076120 0x7f4e14078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:40.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.993+0000 7f4e25ffb640 1 -- 172.21.15.81:0/3580450724 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6555+0+0 (secure 0 0 0) 0x7f4e2c0ba480 con 0x7f4e3006fca0 2024-06-29T14:37:40.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.993+0000 7f4e34f6f640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f4e14076120 0x7f4e14078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:40.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.993+0000 7f4e34f6f640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f4e14076120 0x7f4e14078630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4e20005fd0 tx=0x7f4e20005ea0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:40.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:40.997+0000 7f4e25ffb640 1 -- 172.21.15.81:0/3580450724 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4e300d08a0 con 0x7f4e3006fca0 2024-06-29T14:37:41.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.030+0000 7f4e25ffb640 1 -- 172.21.15.81:0/3580450724 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 37) ==== 98247+0+0 (secure 0 0 0) 0x7f4e2c07f8d0 con 0x7f4e3006fca0 2024-06-29T14:37:41.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:37:41.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:37:41.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.221+0000 7f4e379d4640 1 -- 172.21.15.81:0/3580450724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f4e300d0380 con 0x7f4e3006fca0 2024-06-29T14:37:41.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.857+0000 7f4e25ffb640 1 -- 172.21.15.81:0/3580450724 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) ==== 104+0+0 (secure 0 0 0) 0x7f4e2c083d40 con 0x7f4e3006fca0 2024-06-29T14:37:41.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 -- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f4e14076120 msgr2=0x7f4e14078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f4e14076120 0x7f4e14078630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4e20005fd0 tx=0x7f4e20005ea0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 -- 172.21.15.81:0/3580450724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e3006fca0 msgr2=0x7f4e300aacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e3006fca0 0x7f4e300aacc0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c034040 tx=0x7f4e2c0039e0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 -- 172.21.15.81:0/3580450724 shutdown_connections 2024-06-29T14:37:41.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f4e14076120 0x7f4e14078630 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4e3015bf20 0x7f4e301f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4e30071f20 0x7f4e300ab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 --2- 172.21.15.81:0/3580450724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e3006fca0 0x7f4e300aacc0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 -- 172.21.15.81:0/3580450724 >> 172.21.15.81:0/3580450724 conn(0x7f4e300b1730 msgr2=0x7f4e30159010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:41.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 -- 172.21.15.81:0/3580450724 shutdown_connections 2024-06-29T14:37:41.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.860+0000 7f4e379d4640 1 -- 172.21.15.81:0/3580450724 wait complete. 2024-06-29T14:37:41.870 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:37:41.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.992+0000 7f3fb3a81640 1 -- 172.21.15.81:0/1592139406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f980041c0 msgr2=0x7f3f98004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.992+0000 7f3fb3a81640 1 --2- 172.21.15.81:0/1592139406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f980041c0 0x7f3f98004800 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c001830 tx=0x7f3f98005830 comp rx=0 tx=0).stop 2024-06-29T14:37:41.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.992+0000 7f3fb201e640 1 -- 172.21.15.81:0/1592139406 reap_dead start 2024-06-29T14:37:41.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.992+0000 7f3fb3a81640 1 -- 172.21.15.81:0/1592139406 shutdown_connections 2024-06-29T14:37:41.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.992+0000 7f3fb3a81640 1 -- 172.21.15.81:0/1592139406 >> 172.21.15.81:0/1592139406 conn(0x7f3fac09aa60 msgr2=0x7f3fac09ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:41.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.992+0000 7f3fb3a81640 1 -- 172.21.15.81:0/1592139406 shutdown_connections 2024-06-29T14:37:41.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.992+0000 7f3fb3a81640 1 -- 172.21.15.81:0/1592139406 wait complete. 2024-06-29T14:37:41.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.994+0000 7f3fb3a81640 1 Processor -- start 2024-06-29T14:37:41.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.994+0000 7f3fb3a81640 1 -- start start 2024-06-29T14:37:41.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.995+0000 7f3fb3a81640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f980041c0 0x7f3fac1e7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.995+0000 7f3fb3a81640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3fac0b5ab0 0x7f3fac0b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.995+0000 7f3fb3a81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fac1e90c0 0x7f3fac1e94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.995+0000 7f3fb3a81640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3fac0ae8f0 con 0x7f3fac1e90c0 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.995+0000 7f3fb3a81640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3fac15e650 con 0x7f3fac0b5ab0 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.995+0000 7f3fb3a81640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3fac0aea70 con 0x7f3f980041c0 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.995+0000 7f3fb201e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fac1e90c0 0x7f3fac1e94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.995+0000 7f3fb201e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fac1e90c0 0x7f3fac1e94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45948/0 (socket says 172.21.15.81:45948) 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.995+0000 7f3fb201e640 1 -- 172.21.15.81:0/4264725573 learned_addr learned my addr 172.21.15.81:0/4264725573 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fb101c640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3fac0b5ab0 0x7f3fac0b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fb181d640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f980041c0 0x7f3fac1e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fb201e640 1 -- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f980041c0 msgr2=0x7f3fac1e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fb201e640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f980041c0 0x7f3fac1e7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fb201e640 1 -- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3fac0b5ab0 msgr2=0x7f3fac0b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fb201e640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3fac0b5ab0 0x7f3fac0b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fb201e640 1 -- 172.21.15.81:0/4264725573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fac1f8150 con 0x7f3fac1e90c0 2024-06-29T14:37:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fb201e640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fac1e90c0 0x7f3fac1e94d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3fac074900 tx=0x7f3fa400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fa27fc640 1 -- 172.21.15.81:0/4264725573 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4004b20 con 0x7f3fac1e90c0 2024-06-29T14:37:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fa27fc640 1 -- 172.21.15.81:0/4264725573 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fa4004cc0 con 0x7f3fac1e90c0 2024-06-29T14:37:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fb3a81640 1 -- 172.21.15.81:0/4264725573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fac1f83f0 con 0x7f3fac1e90c0 2024-06-29T14:37:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fa27fc640 1 -- 172.21.15.81:0/4264725573 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4003bd0 con 0x7f3fac1e90c0 2024-06-29T14:37:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.996+0000 7f3fb3a81640 1 -- 172.21.15.81:0/4264725573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fac1f89b0 con 0x7f3fac1e90c0 2024-06-29T14:37:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.997+0000 7f3fb3a81640 1 -- 172.21.15.81:0/4264725573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fac0713c0 con 0x7f3fac1e90c0 2024-06-29T14:37:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.998+0000 7f3fa27fc640 1 -- 172.21.15.81:0/4264725573 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 37) ==== 98247+0+0 (secure 0 0 0) 0x7f3fa4004030 con 0x7f3fac1e90c0 2024-06-29T14:37:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.998+0000 7f3fa27fc640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3f90076390 0x7f3f900788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.998+0000 7f3fa27fc640 1 -- 172.21.15.81:0/4264725573 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6555+0+0 (secure 0 0 0) 0x7f3fa4097f70 con 0x7f3fac1e90c0 2024-06-29T14:37:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.999+0000 7f3fb181d640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3f90076390 0x7f3f900788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:41.999+0000 7f3fb181d640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3f90076390 0x7f3f900788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8001b40 tx=0x7f3fa8001ad0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:42.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:42.003+0000 7f3fa27fc640 1 -- 172.21.15.81:0/4264725573 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3fa4060da0 con 0x7f3fac1e90c0 2024-06-29T14:37:42.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:42.183+0000 7f3fb3a81640 1 -- 172.21.15.81:0/4264725573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3fac0710e0 con 0x7f3fac1e90c0 2024-06-29T14:37:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.043+0000 7f3fa27fc640 1 -- 172.21.15.81:0/4264725573 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v82) ==== 181+0+0 (secure 0 0 0) 0x7f3fa40650a0 con 0x7f3fac1e90c0 2024-06-29T14:37:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:37:43.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.047+0000 7f3fb3a81640 1 -- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3f90076390 msgr2=0x7f3f900788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:43.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.047+0000 7f3fb3a81640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3f90076390 0x7f3f900788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8001b40 tx=0x7f3fa8001ad0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.047+0000 7f3fb3a81640 1 -- 172.21.15.81:0/4264725573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fac1e90c0 msgr2=0x7f3fac1e94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:43.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.047+0000 7f3fb3a81640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fac1e90c0 0x7f3fac1e94d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3fac074900 tx=0x7f3fa400f3c0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.047+0000 7f3fb3a81640 1 -- 172.21.15.81:0/4264725573 shutdown_connections 2024-06-29T14:37:43.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.047+0000 7f3fb3a81640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3f90076390 0x7f3f900788a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.047+0000 7f3fb3a81640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fac1e90c0 0x7f3fac1e94d0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.047+0000 7f3fb3a81640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3fac0b5ab0 0x7f3fac0b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.047+0000 7f3fb3a81640 1 --2- 172.21.15.81:0/4264725573 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f980041c0 0x7f3fac1e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.047+0000 7f3fb3a81640 1 -- 172.21.15.81:0/4264725573 >> 172.21.15.81:0/4264725573 conn(0x7f3fac09aa60 msgr2=0x7f3fac0718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:43.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.049+0000 7f3fb3a81640 1 -- 172.21.15.81:0/4264725573 shutdown_connections 2024-06-29T14:37:43.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.049+0000 7f3fb3a81640 1 -- 172.21.15.81:0/4264725573 wait complete. 2024-06-29T14:37:43.062 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:37:43.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.183+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/1740962814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9b8003a00 msgr2=0x7fb9b8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:43.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.183+0000 7fb9d0fa1640 1 --2- 172.21.15.81:0/1740962814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9b8003a00 0x7fb9b8003e70 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00bb20 tx=0x7fb9b8005830 comp rx=0 tx=0).stop 2024-06-29T14:37:43.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.183+0000 7fb9cad9d640 1 -- 172.21.15.81:0/1740962814 reap_dead start 2024-06-29T14:37:43.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.183+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/1740962814 shutdown_connections 2024-06-29T14:37:43.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.183+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/1740962814 >> 172.21.15.81:0/1740962814 conn(0x7fb9cc06cd60 msgr2=0x7fb9cc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:43.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.183+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/1740962814 shutdown_connections 2024-06-29T14:37:43.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.183+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/1740962814 wait complete. 2024-06-29T14:37:43.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.185+0000 7fb9d0fa1640 1 Processor -- start 2024-06-29T14:37:43.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.185+0000 7fb9d0fa1640 1 -- start start 2024-06-29T14:37:43.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9d0fa1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9b8003a00 0x7fb9cc1e6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9d0fa1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9cc1f5420 0x7fb9cc1f5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9d0fa1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9cc06e650 0x7fb9cc06ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9d0fa1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc144630 con 0x7fb9cc06e650 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9d0fa1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc072e70 con 0x7fb9cc1f5420 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9d0fa1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb9cc1447b0 con 0x7fb9b8003a00 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9cad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9cc06e650 0x7fb9cc06ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9cad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9cc06e650 0x7fb9cc06ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45964/0 (socket says 172.21.15.81:45964) 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9cad9d640 1 -- 172.21.15.81:0/3153385977 learned_addr learned my addr 172.21.15.81:0/3153385977 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9ca59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9b8003a00 0x7fb9cc1e6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9c1d9b640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9cc1f5420 0x7fb9cc1f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9cad9d640 1 -- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9b8003a00 msgr2=0x7fb9cc1e6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:43.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9cad9d640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9b8003a00 0x7fb9cc1e6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9cad9d640 1 -- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9cc1f5420 msgr2=0x7fb9cc1f5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:43.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9cad9d640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9cc1f5420 0x7fb9cc1f5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:43.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9cad9d640 1 -- 172.21.15.81:0/3153385977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc1f96d0 con 0x7fb9cc06e650 2024-06-29T14:37:43.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.187+0000 7fb9cad9d640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9cc06e650 0x7fb9cc06ea60 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc071d20 tx=0x7fb9c4006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:43.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.188+0000 7fb9c37fe640 1 -- 172.21.15.81:0/3153385977 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c400fbb0 con 0x7fb9cc06e650 2024-06-29T14:37:43.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.188+0000 7fb9c37fe640 1 -- 172.21.15.81:0/3153385977 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9c4018070 con 0x7fb9cc06e650 2024-06-29T14:37:43.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.188+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/3153385977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9cc06efa0 con 0x7fb9cc06e650 2024-06-29T14:37:43.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.188+0000 7fb9c37fe640 1 -- 172.21.15.81:0/3153385977 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4006970 con 0x7fb9cc06e650 2024-06-29T14:37:43.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.188+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/3153385977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9cc1f8bf0 con 0x7fb9cc06e650 2024-06-29T14:37:43.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.189+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/3153385977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9cc1f5f00 con 0x7fb9cc06e650 2024-06-29T14:37:43.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.189+0000 7fb9c37fe640 1 -- 172.21.15.81:0/3153385977 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fb9c401d460 con 0x7fb9cc06e650 2024-06-29T14:37:43.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.190+0000 7fb9c37fe640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb9ac076590 0x7fb9ac078aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:43.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.190+0000 7fb9c37fe640 1 -- 172.21.15.81:0/3153385977 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 6117+0+0 (secure 0 0 0) 0x7fb9c40982d0 con 0x7fb9cc06e650 2024-06-29T14:37:43.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.190+0000 7fb9ca59c640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb9ac076590 0x7fb9ac078aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:43.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.191+0000 7fb9ca59c640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb9ac076590 0x7fb9ac078aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc036000 tx=0x7fb9bc03c040 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:43.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.194+0000 7fb9c37fe640 1 -- 172.21.15.81:0/3153385977 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb9c40617f0 con 0x7fb9cc06e650 2024-06-29T14:37:43.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:43.375+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/3153385977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb9cc0cc5b0 con 0x7fb9cc06e650 2024-06-29T14:37:44.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.050+0000 7fb9c37fe640 1 -- 172.21.15.81:0/3153385977 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v83) ==== 169+0+0 (secure 0 0 0) 0x7fb9cc0cc5b0 con 0x7fb9cc06e650 2024-06-29T14:37:44.048 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T14:37:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb9ac076590 msgr2=0x7fb9ac078aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb9ac076590 0x7fb9ac078aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc036000 tx=0x7fb9bc03c040 comp rx=0 tx=0).stop 2024-06-29T14:37:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/3153385977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9cc06e650 msgr2=0x7fb9cc06ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9cc06e650 0x7fb9cc06ea60 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc071d20 tx=0x7fb9c4006660 comp rx=0 tx=0).stop 2024-06-29T14:37:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/3153385977 shutdown_connections 2024-06-29T14:37:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb9ac076590 0x7fb9ac078aa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9cc06e650 0x7fb9cc06ea60 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb9cc1f5420 0x7fb9cc1f5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 --2- 172.21.15.81:0/3153385977 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb9b8003a00 0x7fb9cc1e6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/3153385977 >> 172.21.15.81:0/3153385977 conn(0x7fb9cc06cd60 msgr2=0x7fb9cc13f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/3153385977 shutdown_connections 2024-06-29T14:37:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.052+0000 7fb9d0fa1640 1 -- 172.21.15.81:0/3153385977 wait complete. 2024-06-29T14:37:44.065 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:37:44.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.186+0000 7facac0ac640 1 -- 172.21.15.81:0/2211414787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac90003a30 msgr2=0x7fac90003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.186+0000 7facac0ac640 1 --2- 172.21.15.81:0/2211414787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac90003a30 0x7fac90003ea0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7faca00016c0 tx=0x7fac90005830 comp rx=0 tx=0).stop 2024-06-29T14:37:44.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.186+0000 7facaa649640 1 -- 172.21.15.81:0/2211414787 reap_dead start 2024-06-29T14:37:44.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.186+0000 7facac0ac640 1 -- 172.21.15.81:0/2211414787 shutdown_connections 2024-06-29T14:37:44.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.186+0000 7facac0ac640 1 -- 172.21.15.81:0/2211414787 >> 172.21.15.81:0/2211414787 conn(0x7faca40d0930 msgr2=0x7faca40d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:44.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.186+0000 7facac0ac640 1 -- 172.21.15.81:0/2211414787 shutdown_connections 2024-06-29T14:37:44.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.186+0000 7facac0ac640 1 -- 172.21.15.81:0/2211414787 wait complete. 2024-06-29T14:37:44.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.188+0000 7facac0ac640 1 Processor -- start 2024-06-29T14:37:44.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.188+0000 7facac0ac640 1 -- start start 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7facac0ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac90003a30 0x7faca40aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7facac0ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faca41f2c10 0x7faca4146180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7facac0ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faca40abaa0 0x7faca40aca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7facac0ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faca415b260 con 0x7fac90003a30 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7facac0ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faca415b0e0 con 0x7faca41f2c10 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7facac0ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faca415b3e0 con 0x7faca40abaa0 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7faca9e48640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac90003a30 0x7faca40aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7faca9e48640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac90003a30 0x7faca40aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45986/0 (socket says 172.21.15.81:45986) 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7faca9e48640 1 -- 172.21.15.81:0/652654529 learned_addr learned my addr 172.21.15.81:0/652654529 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7faca9e48640 1 -- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faca40abaa0 msgr2=0x7faca40aca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7faca9e48640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faca40abaa0 0x7faca40aca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7faca9e48640 1 -- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faca41f2c10 msgr2=0x7faca4146180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.189+0000 7faca9e48640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faca41f2c10 0x7faca4146180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.190+0000 7faca9e48640 1 -- 172.21.15.81:0/652654529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca40aaa00 con 0x7fac90003a30 2024-06-29T14:37:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.190+0000 7faca9e48640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac90003a30 0x7faca40aef60 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7faca002a930 tx=0x7faca0006360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.190+0000 7fac9a7fc640 1 -- 172.21.15.81:0/652654529 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca0038070 con 0x7fac90003a30 2024-06-29T14:37:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.190+0000 7fac9a7fc640 1 -- 172.21.15.81:0/652654529 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faca0003af0 con 0x7fac90003a30 2024-06-29T14:37:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.190+0000 7facac0ac640 1 -- 172.21.15.81:0/652654529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faca40aace0 con 0x7fac90003a30 2024-06-29T14:37:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.190+0000 7fac9a7fc640 1 -- 172.21.15.81:0/652654529 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca003d460 con 0x7fac90003a30 2024-06-29T14:37:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.190+0000 7facac0ac640 1 -- 172.21.15.81:0/652654529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faca40ab2a0 con 0x7fac90003a30 2024-06-29T14:37:44.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.191+0000 7facac0ac640 1 -- 172.21.15.81:0/652654529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faca406e2a0 con 0x7fac90003a30 2024-06-29T14:37:44.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.192+0000 7fac9a7fc640 1 -- 172.21.15.81:0/652654529 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7faca0030070 con 0x7fac90003a30 2024-06-29T14:37:44.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.192+0000 7fac9a7fc640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fac880763f0 0x7fac88078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.192+0000 7fac9a7fc640 1 -- 172.21.15.81:0/652654529 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5703+0+0 (secure 0 0 0) 0x7faca00b76e0 con 0x7fac90003a30 2024-06-29T14:37:44.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.193+0000 7faca9647640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fac880763f0 0x7fac88078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.193+0000 7faca9647640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fac880763f0 0x7fac88078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faca40af660 tx=0x7fac9400d3f0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:44.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.197+0000 7fac9a7fc640 1 -- 172.21.15.81:0/652654529 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faca406e2a0 con 0x7fac90003a30 2024-06-29T14:37:44.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.378+0000 7facac0ac640 1 -- 172.21.15.81:0/652654529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7faca40b4890 con 0x7fac90003a30 2024-06-29T14:37:44.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.382+0000 7fac9a7fc640 1 -- 172.21.15.81:0/652654529 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v39) ==== 71+0+0 (secure 0 0 0) 0x7faca0042060 con 0x7fac90003a30 2024-06-29T14:37:44.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 -- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fac880763f0 msgr2=0x7fac88078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fac880763f0 0x7fac88078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faca40af660 tx=0x7fac9400d3f0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 -- 172.21.15.81:0/652654529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac90003a30 msgr2=0x7faca40aef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac90003a30 0x7faca40aef60 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7faca002a930 tx=0x7faca0006360 comp rx=0 tx=0).stop 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 -- 172.21.15.81:0/652654529 shutdown_connections 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fac880763f0 0x7fac88078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faca40abaa0 0x7faca40aca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faca41f2c10 0x7faca4146180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 --2- 172.21.15.81:0/652654529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac90003a30 0x7faca40aef60 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 -- 172.21.15.81:0/652654529 >> 172.21.15.81:0/652654529 conn(0x7faca40d0930 msgr2=0x7faca40b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 -- 172.21.15.81:0/652654529 shutdown_connections 2024-06-29T14:37:44.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.384+0000 7facac0ac640 1 -- 172.21.15.81:0/652654529 wait complete. 2024-06-29T14:37:44.392 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2024-06-29T14:37:44.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.509+0000 7f8db69bd640 1 -- 172.21.15.81:0/1447106248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db00d0230 msgr2=0x7f8db00d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.509+0000 7f8db69bd640 1 --2- 172.21.15.81:0/1447106248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db00d0230 0x7f8db00d0640 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009830 tx=0x7f8da402a480 comp rx=0 tx=0).stop 2024-06-29T14:37:44.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.509+0000 7f8db69bd640 1 -- 172.21.15.81:0/1447106248 shutdown_connections 2024-06-29T14:37:44.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.509+0000 7f8db69bd640 1 --2- 172.21.15.81:0/1447106248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db00d0230 0x7f8db00d0640 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.509+0000 7f8db69bd640 1 -- 172.21.15.81:0/1447106248 >> 172.21.15.81:0/1447106248 conn(0x7f8db00b3680 msgr2=0x7f8db00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:44.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.509+0000 7f8db69bd640 1 -- 172.21.15.81:0/1447106248 shutdown_connections 2024-06-29T14:37:44.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.509+0000 7f8db69bd640 1 -- 172.21.15.81:0/1447106248 wait complete. 2024-06-29T14:37:44.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.511+0000 7f8db69bd640 1 Processor -- start 2024-06-29T14:37:44.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.511+0000 7f8db69bd640 1 -- start start 2024-06-29T14:37:44.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.512+0000 7f8db69bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8db00b3200 0x7f8db014d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.512+0000 7f8db69bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db00b4f40 0x7f8db01f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.512+0000 7f8db69bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8db00d0230 0x7f8db009e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.512+0000 7f8db69bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8db00b5e50 con 0x7f8db00b4f40 2024-06-29T14:37:44.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.512+0000 7f8db69bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8db00b5cd0 con 0x7f8db00b3200 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.512+0000 7f8db69bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8db00b5fd0 con 0x7f8db00d0230 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8daf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db00b4f40 0x7f8db01f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8daf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db00b4f40 0x7f8db01f39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46008/0 (socket says 172.21.15.81:46008) 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8daf7fe640 1 -- 172.21.15.81:0/1081622435 learned_addr learned my addr 172.21.15.81:0/1081622435 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8daf7fe640 1 -- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8db00d0230 msgr2=0x7f8db009e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8daf7fe640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8db00d0230 0x7f8db009e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8daf7fe640 1 -- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8db00b3200 msgr2=0x7f8db014d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8daf7fe640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8db00b3200 0x7f8db014d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8daf7fe640 1 -- 172.21.15.81:0/1081622435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db014bfd0 con 0x7f8db00b4f40 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8daf7fe640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db00b4f40 0x7f8db01f39f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009950 tx=0x7f8da4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8dacff9640 1 -- 172.21.15.81:0/1081622435 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da4031d60 con 0x7f8db00b4f40 2024-06-29T14:37:44.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8dacff9640 1 -- 172.21.15.81:0/1081622435 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8da4033e10 con 0x7f8db00b4f40 2024-06-29T14:37:44.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8db69bd640 1 -- 172.21.15.81:0/1081622435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db014c2b0 con 0x7f8db00b4f40 2024-06-29T14:37:44.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8dacff9640 1 -- 172.21.15.81:0/1081622435 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da4005260 con 0x7f8db00b4f40 2024-06-29T14:37:44.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.513+0000 7f8db69bd640 1 -- 172.21.15.81:0/1081622435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db014c870 con 0x7f8db00b4f40 2024-06-29T14:37:44.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.514+0000 7f8db69bd640 1 -- 172.21.15.81:0/1081622435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db014bd40 con 0x7f8db00b4f40 2024-06-29T14:37:44.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.515+0000 7f8dacff9640 1 -- 172.21.15.81:0/1081622435 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f8da4005400 con 0x7f8db00b4f40 2024-06-29T14:37:44.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.516+0000 7f8dacff9640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8d9c0763f0 0x7f8d9c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:44.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.516+0000 7f8dacff9640 1 -- 172.21.15.81:0/1081622435 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5703+0+0 (secure 0 0 0) 0x7f8da40b8c40 con 0x7f8db00b4f40 2024-06-29T14:37:44.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.516+0000 7f8daffff640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8d9c0763f0 0x7f8d9c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:44.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.516+0000 7f8daffff640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8d9c0763f0 0x7f8d9c078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009650 tx=0x7f8da0006ba0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:44.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.523+0000 7f8dacff9640 1 -- 172.21.15.81:0/1081622435 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8da403f030 con 0x7f8db00b4f40 2024-06-29T14:37:44.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:44.777+0000 7f8db69bd640 1 -- 172.21.15.81:0/1081622435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) -- 0x7f8db01f3f30 con 0x7f8db00b4f40 2024-06-29T14:37:45.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.051+0000 7f8dacff9640 1 -- 172.21.15.81:0/1081622435 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v554) ==== 156+0+0 (secure 0 0 0) 0x7f8da4082380 con 0x7f8db00b4f40 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 -- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8d9c0763f0 msgr2=0x7f8d9c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8d9c0763f0 0x7f8d9c078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009650 tx=0x7f8da0006ba0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 -- 172.21.15.81:0/1081622435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db00b4f40 msgr2=0x7f8db01f39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db00b4f40 0x7f8db01f39f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009950 tx=0x7f8da4033c20 comp rx=0 tx=0).stop 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 -- 172.21.15.81:0/1081622435 shutdown_connections 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8d9c0763f0 0x7f8d9c078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8db00d0230 0x7f8db009e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db00b4f40 0x7f8db01f39f0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 --2- 172.21.15.81:0/1081622435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8db00b3200 0x7f8db014d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 -- 172.21.15.81:0/1081622435 >> 172.21.15.81:0/1081622435 conn(0x7f8db00b3680 msgr2=0x7f8db00718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 -- 172.21.15.81:0/1081622435 shutdown_connections 2024-06-29T14:37:45.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.055+0000 7f8db69bd640 1 -- 172.21.15.81:0/1081622435 wait complete. 2024-06-29T14:37:45.064 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2024-06-29T14:37:45.064 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2024-06-29T14:37:45.064 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:37:45.065 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.179+0000 7fd4dd046640 1 -- 172.21.15.81:0/3457627252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4d81412f0 msgr2=0x7fd4d80cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.179+0000 7fd4dd046640 1 --2- 172.21.15.81:0/3457627252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4d81412f0 0x7fd4d80cef90 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80b2dd0 tx=0x7fd4c802aa30 comp rx=0 tx=0).stop 2024-06-29T14:37:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.179+0000 7fd4dd046640 1 -- 172.21.15.81:0/3457627252 shutdown_connections 2024-06-29T14:37:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.179+0000 7fd4dd046640 1 --2- 172.21.15.81:0/3457627252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4d8073570 0x7fd4d815e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.179+0000 7fd4dd046640 1 --2- 172.21.15.81:0/3457627252 >> v2:172.21.15.167:3300/0 conn(0x7fd4d80cf4d0 0x7fd4d8073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.179+0000 7fd4dd046640 1 --2- 172.21.15.81:0/3457627252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4d81412f0 0x7fd4d80cef90 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.179+0000 7fd4dd046640 1 -- 172.21.15.81:0/3457627252 >> 172.21.15.81:0/3457627252 conn(0x7fd4d806ccc0 msgr2=0x7fd4d80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.179+0000 7fd4dd046640 1 -- 172.21.15.81:0/3457627252 shutdown_connections 2024-06-29T14:37:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.179+0000 7fd4dd046640 1 -- 172.21.15.81:0/3457627252 wait complete. 2024-06-29T14:37:45.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.181+0000 7fd4dd046640 1 Processor -- start 2024-06-29T14:37:45.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.181+0000 7fd4dd046640 1 -- start start 2024-06-29T14:37:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.182+0000 7fd4dd046640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4d8073570 0x7fd4d81564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.182+0000 7fd4dd046640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4d80cf4d0 0x7fd4d80ab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.182+0000 7fd4dd046640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4d81412f0 0x7fd4d80b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.182+0000 7fd4dd046640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d815f230 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.182+0000 7fd4dd046640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d815f0b0 con 0x7fd4d81412f0 2024-06-29T14:37:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.182+0000 7fd4dd046640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d80b3210 con 0x7fd4d8073570 2024-06-29T14:37:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.182+0000 7fd4d659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4d80cf4d0 0x7fd4d80ab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4d80cf4d0 0x7fd4d80ab4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46030/0 (socket says 172.21.15.81:46030) 2024-06-29T14:37:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d659c640 1 -- 172.21.15.81:0/476170407 learned_addr learned my addr 172.21.15.81:0/476170407 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d6d9d640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4d8073570 0x7fd4d81564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d759e640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4d81412f0 0x7fd4d80b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d659c640 1 -- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4d8073570 msgr2=0x7fd4d81564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d659c640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4d8073570 0x7fd4d81564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d659c640 1 -- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4d81412f0 msgr2=0x7fd4d80b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d659c640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4d81412f0 0x7fd4d80b0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d659c640 1 -- 172.21.15.81:0/476170407 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d8157910 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d6d9d640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4d8073570 0x7fd4d81564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d759e640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4d81412f0 0x7fd4d80b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4d659c640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4d80cf4d0 0x7fd4d80ab4e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00c970 tx=0x7fd4cc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4bf7fe640 1 -- 172.21.15.81:0/476170407 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc007bb0 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4bf7fe640 1 -- 172.21.15.81:0/476170407 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4cc007e40 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4dd046640 1 -- 172.21.15.81:0/476170407 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d8157bf0 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4bf7fe640 1 -- 172.21.15.81:0/476170407 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc00f690 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.183+0000 7fd4dd046640 1 -- 172.21.15.81:0/476170407 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d8155fc0 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.184+0000 7fd4dd046640 1 -- 172.21.15.81:0/476170407 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d8155ad0 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.185+0000 7fd4bf7fe640 1 -- 172.21.15.81:0/476170407 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fd4cc00f830 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.186+0000 7fd4bf7fe640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd4c40763f0 0x7fd4c4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.186+0000 7fd4bf7fe640 1 -- 172.21.15.81:0/476170407 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5703+0+0 (secure 0 0 0) 0x7fd4cc0982c0 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.186+0000 7fd4d6d9d640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd4c40763f0 0x7fd4c4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.187+0000 7fd4d6d9d640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd4c40763f0 0x7fd4c4078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8035000 tx=0x7fd4c8004ab0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:45.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.190+0000 7fd4bf7fe640 1 -- 172.21.15.81:0/476170407 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd4cc014030 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.370+0000 7fd4dd046640 1 -- 172.21.15.81:0/476170407 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4d8073030 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.371+0000 7fd4bf7fe640 1 -- 172.21.15.81:0/476170407 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+14136 (secure 0 0 0) 0x7fd4cc0618f0 con 0x7fd4d80cf4d0 2024-06-29T14:37:45.369 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:45.369 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":83,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:44.044006+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6837/2723173791":"2024-06-30T14:37:40.855324+0000","172.21.15.167:0/3387399405":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/665800877":"2024-06-30T14:37:38.662497+0000","172.21.15.167:6833/2940523935":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/470342636":"2024-06-30T14:37:38.662497+0000","172.21.15.167:6832/2940523935":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/2903530506":"2024-06-30T14:37:38.662497+0000","172.21.15.167:6833/3693496302":"2024-06-30T14:37:32.992869+0000","172.21.15.167:6832/3693496302":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/912055670":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/1220706198":"2024-06-30T14:37:32.992869+0000","172.21.15.167:6835/2723173791":"2024-06-30T14:37:40.855324+0000","172.21.15.167:0/538343225":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/1189302834":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/3217872433":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/53505028":"2024-06-30T14:37:38.662497+0000","172.21.15.167:0/2826882843":"2024-06-30T14:37:32.992869+0000","172.21.15.167:0/95469329":"2024-06-30T14:37:32.992869+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 -- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd4c40763f0 msgr2=0x7fd4c4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd4c40763f0 0x7fd4c4078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8035000 tx=0x7fd4c8004ab0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 -- 172.21.15.81:0/476170407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4d80cf4d0 msgr2=0x7fd4d80ab4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4d80cf4d0 0x7fd4d80ab4e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00c970 tx=0x7fd4cc00ce30 comp rx=0 tx=0).stop 2024-06-29T14:37:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 -- 172.21.15.81:0/476170407 shutdown_connections 2024-06-29T14:37:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd4c40763f0 0x7fd4c4078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4d81412f0 0x7fd4d80b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4d80cf4d0 0x7fd4d80ab4e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 --2- 172.21.15.81:0/476170407 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4d8073570 0x7fd4d81564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 -- 172.21.15.81:0/476170407 >> 172.21.15.81:0/476170407 conn(0x7fd4d806ccc0 msgr2=0x7fd4d815ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 -- 172.21.15.81:0/476170407 shutdown_connections 2024-06-29T14:37:45.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.373+0000 7fd4dd046640 1 -- 172.21.15.81:0/476170407 wait complete. 2024-06-29T14:37:45.382 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:45.383 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:45.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.502+0000 7f9708503640 1 -- 172.21.15.81:0/720841033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f970006f6a0 msgr2=0x7f970015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.502+0000 7f9708503640 1 --2- 172.21.15.81:0/720841033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f970006f6a0 0x7f970015aed0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f96f0009830 tx=0x7f96f002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.502+0000 7f9708503640 1 -- 172.21.15.81:0/720841033 shutdown_connections 2024-06-29T14:37:45.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.502+0000 7f9708503640 1 --2- 172.21.15.81:0/720841033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f970006f6a0 0x7f970015aed0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.502+0000 7f9708503640 1 -- 172.21.15.81:0/720841033 >> 172.21.15.81:0/720841033 conn(0x7f97000d0930 msgr2=0x7f97000d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.502+0000 7f9708503640 1 -- 172.21.15.81:0/720841033 shutdown_connections 2024-06-29T14:37:45.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.502+0000 7f9708503640 1 -- 172.21.15.81:0/720841033 wait complete. 2024-06-29T14:37:45.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.504+0000 7f9708503640 1 Processor -- start 2024-06-29T14:37:45.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.504+0000 7f9708503640 1 -- start start 2024-06-29T14:37:45.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f9708503640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f970006f6a0 0x7f97000cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f9708503640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000b3960 0x7f97000adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f9708503640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97000b57b0 0x7f9700157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f9708503640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f97000b4600 con 0x7f970006f6a0 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f9708503640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f97000b4480 con 0x7f97000b3960 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f9708503640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f97000b4780 con 0x7f97000b57b0 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f970629f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f970006f6a0 0x7f97000cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f970629f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f970006f6a0 0x7f97000cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46042/0 (socket says 172.21.15.81:46042) 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f970629f640 1 -- 172.21.15.81:0/3082309822 learned_addr learned my addr 172.21.15.81:0/3082309822 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f9706aa0640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97000b57b0 0x7f9700157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f9705a9e640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000b3960 0x7f97000adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f970629f640 1 -- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97000b57b0 msgr2=0x7f9700157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f970629f640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97000b57b0 0x7f9700157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f970629f640 1 -- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000b3960 msgr2=0x7f97000adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f970629f640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000b3960 0x7f97000adb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.505+0000 7f970629f640 1 -- 172.21.15.81:0/3082309822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97001f6570 con 0x7f970006f6a0 2024-06-29T14:37:45.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.506+0000 7f970629f640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f970006f6a0 0x7f97000cc1f0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c3d0 tx=0x7f96fc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:45.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.506+0000 7f96f6ffd640 1 -- 172.21.15.81:0/3082309822 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc017070 con 0x7f970006f6a0 2024-06-29T14:37:45.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.506+0000 7f96f6ffd640 1 -- 172.21.15.81:0/3082309822 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96fc0053c0 con 0x7f970006f6a0 2024-06-29T14:37:45.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.506+0000 7f9708503640 1 -- 172.21.15.81:0/3082309822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97000cc730 con 0x7f970006f6a0 2024-06-29T14:37:45.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.506+0000 7f96f6ffd640 1 -- 172.21.15.81:0/3082309822 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc003850 con 0x7f970006f6a0 2024-06-29T14:37:45.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.506+0000 7f9708503640 1 -- 172.21.15.81:0/3082309822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97001f23e0 con 0x7f970006f6a0 2024-06-29T14:37:45.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.507+0000 7f9708503640 1 -- 172.21.15.81:0/3082309822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f970006e2a0 con 0x7f970006f6a0 2024-06-29T14:37:45.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.507+0000 7f96f6ffd640 1 -- 172.21.15.81:0/3082309822 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f96fc01a070 con 0x7f970006f6a0 2024-06-29T14:37:45.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.508+0000 7f96f6ffd640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f96e40763f0 0x7f96e4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.508+0000 7f96f6ffd640 1 -- 172.21.15.81:0/3082309822 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5703+0+0 (secure 0 0 0) 0x7f96fc096d10 con 0x7f970006f6a0 2024-06-29T14:37:45.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.508+0000 7f9705a9e640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f96e40763f0 0x7f96e4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.509+0000 7f9705a9e640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f96e40763f0 0x7f96e4078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f96f0034000 tx=0x7f96f0004730 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:45.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.512+0000 7f96f6ffd640 1 -- 172.21.15.81:0/3082309822 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f970006e2a0 con 0x7f970006f6a0 2024-06-29T14:37:45.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.691+0000 7f9708503640 1 -- 172.21.15.81:0/3082309822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f97000b4bd0 con 0x7f970006f6a0 2024-06-29T14:37:45.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.692+0000 7f96f6ffd640 1 -- 172.21.15.81:0/3082309822 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f96fc060560 con 0x7f970006f6a0 2024-06-29T14:37:45.689 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:37:45.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.693+0000 7f9708503640 1 -- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f96e40763f0 msgr2=0x7f96e4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.693+0000 7f9708503640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f96e40763f0 0x7f96e4078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f96f0034000 tx=0x7f96f0004730 comp rx=0 tx=0).stop 2024-06-29T14:37:45.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.693+0000 7f9708503640 1 -- 172.21.15.81:0/3082309822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f970006f6a0 msgr2=0x7f97000cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.693+0000 7f9708503640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f970006f6a0 0x7f97000cc1f0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00c3d0 tx=0x7f96fc00c890 comp rx=0 tx=0).stop 2024-06-29T14:37:45.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.694+0000 7f9708503640 1 -- 172.21.15.81:0/3082309822 shutdown_connections 2024-06-29T14:37:45.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.694+0000 7f9708503640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f96e40763f0 0x7f96e4078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.694+0000 7f9708503640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97000b57b0 0x7f9700157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.694+0000 7f9708503640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97000b3960 0x7f97000adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.694+0000 7f9708503640 1 --2- 172.21.15.81:0/3082309822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f970006f6a0 0x7f97000cc1f0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.694+0000 7f9708503640 1 -- 172.21.15.81:0/3082309822 >> 172.21.15.81:0/3082309822 conn(0x7f97000d0930 msgr2=0x7f9700158c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.694+0000 7f9708503640 1 -- 172.21.15.81:0/3082309822 shutdown_connections 2024-06-29T14:37:45.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.694+0000 7f9708503640 1 -- 172.21.15.81:0/3082309822 wait complete. 2024-06-29T14:37:45.702 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-06-29T14:37:45.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.818+0000 7fe961427640 1 -- 172.21.15.81:0/1222719639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95c0b3cd0 msgr2=0x7fe95c0d2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.818+0000 7fe961427640 1 --2- 172.21.15.81:0/1222719639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95c0b3cd0 0x7fe95c0d2240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe950005fd0 tx=0x7fe95002a540 comp rx=0 tx=0).stop 2024-06-29T14:37:45.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.819+0000 7fe961427640 1 -- 172.21.15.81:0/1222719639 shutdown_connections 2024-06-29T14:37:45.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.819+0000 7fe961427640 1 --2- 172.21.15.81:0/1222719639 >> v2:172.21.15.167:3300/0 conn(0x7fe95c0ceb80 0x7fe95c0d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.819+0000 7fe961427640 1 --2- 172.21.15.81:0/1222719639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95c0b3cd0 0x7fe95c0d2240 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.819+0000 7fe961427640 1 -- 172.21.15.81:0/1222719639 >> 172.21.15.81:0/1222719639 conn(0x7fe95c0d3c60 msgr2=0x7fe95c0d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:45.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.819+0000 7fe961427640 1 -- 172.21.15.81:0/1222719639 shutdown_connections 2024-06-29T14:37:45.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.819+0000 7fe961427640 1 -- 172.21.15.81:0/1222719639 wait complete. 2024-06-29T14:37:45.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.820+0000 7fe961427640 1 Processor -- start 2024-06-29T14:37:45.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.821+0000 7fe961427640 1 -- start start 2024-06-29T14:37:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe961427640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe95c0ce770 0x7fe95c15ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe961427640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95c0ceb80 0x7fe95c162c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe961427640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95c163170 0x7fe95c163580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe961427640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe95c0d15a0 con 0x7fe95c0ce770 2024-06-29T14:37:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe961427640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe95c0d1420 con 0x7fe95c163170 2024-06-29T14:37:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe961427640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe95c0d4990 con 0x7fe95c0ceb80 2024-06-29T14:37:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe95c0ce770 0x7fe95c15ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe95c0ce770 0x7fe95c15ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46052/0 (socket says 172.21.15.81:46052) 2024-06-29T14:37:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95affd640 1 -- 172.21.15.81:0/2630101606 learned_addr learned my addr 172.21.15.81:0/2630101606 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95a7fc640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95c0ceb80 0x7fe95c162c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95b7fe640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95c163170 0x7fe95c163580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95affd640 1 -- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95c0ceb80 msgr2=0x7fe95c162c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95affd640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95c0ceb80 0x7fe95c162c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95affd640 1 -- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95c163170 msgr2=0x7fe95c163580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95affd640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95c163170 0x7fe95c163580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95affd640 1 -- 172.21.15.81:0/2630101606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95c167100 con 0x7fe95c0ce770 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.822+0000 7fe95affd640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe95c0ce770 0x7fe95c15ec00 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00c3a0 tx=0x7fe94c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.823+0000 7fe9437fe640 1 -- 172.21.15.81:0/2630101606 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c017070 con 0x7fe95c0ce770 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.823+0000 7fe9437fe640 1 -- 172.21.15.81:0/2630101606 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe94c0053c0 con 0x7fe95c0ce770 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.823+0000 7fe961427640 1 -- 172.21.15.81:0/2630101606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe95c1673a0 con 0x7fe95c0ce770 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.823+0000 7fe9437fe640 1 -- 172.21.15.81:0/2630101606 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c003850 con 0x7fe95c0ce770 2024-06-29T14:37:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.823+0000 7fe961427640 1 -- 172.21.15.81:0/2630101606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe95c167960 con 0x7fe95c0ce770 2024-06-29T14:37:45.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.824+0000 7fe961427640 1 -- 172.21.15.81:0/2630101606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe95c0d4540 con 0x7fe95c0ce770 2024-06-29T14:37:45.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.824+0000 7fe9437fe640 1 -- 172.21.15.81:0/2630101606 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fe94c01a070 con 0x7fe95c0ce770 2024-06-29T14:37:45.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.825+0000 7fe9437fe640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe9480763f0 0x7fe948078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:45.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.825+0000 7fe9437fe640 1 -- 172.21.15.81:0/2630101606 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5703+0+0 (secure 0 0 0) 0x7fe94c096c40 con 0x7fe95c0ce770 2024-06-29T14:37:45.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.825+0000 7fe95a7fc640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe9480763f0 0x7fe948078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:45.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.826+0000 7fe95a7fc640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe9480763f0 0x7fe948078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe950033000 tx=0x7fe950004730 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:45.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:45.829+0000 7fe9437fe640 1 -- 172.21.15.81:0/2630101606 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe95c0d4540 con 0x7fe95c0ce770 2024-06-29T14:37:46.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:46.086+0000 7fe961427640 1 -- 172.21.15.81:0/2630101606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) -- 0x7fe95c162830 con 0x7fe95c0ce770 2024-06-29T14:37:47.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.059+0000 7fe9437fe640 1 -- 172.21.15.81:0/2630101606 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v556) ==== 169+0+0 (secure 0 0 0) 0x7fe94c060490 con 0x7fe95c0ce770 2024-06-29T14:37:47.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.061+0000 7fe961427640 1 -- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe9480763f0 msgr2=0x7fe948078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.061+0000 7fe961427640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe9480763f0 0x7fe948078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe950033000 tx=0x7fe950004730 comp rx=0 tx=0).stop 2024-06-29T14:37:47.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.061+0000 7fe961427640 1 -- 172.21.15.81:0/2630101606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe95c0ce770 msgr2=0x7fe95c15ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.061+0000 7fe961427640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe95c0ce770 0x7fe95c15ec00 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00c3a0 tx=0x7fe94c00c860 comp rx=0 tx=0).stop 2024-06-29T14:37:47.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.062+0000 7fe961427640 1 -- 172.21.15.81:0/2630101606 shutdown_connections 2024-06-29T14:37:47.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.062+0000 7fe961427640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe9480763f0 0x7fe948078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.062+0000 7fe961427640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe95c163170 0x7fe95c163580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.062+0000 7fe961427640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe95c0ceb80 0x7fe95c162c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.062+0000 7fe961427640 1 --2- 172.21.15.81:0/2630101606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe95c0ce770 0x7fe95c15ec00 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.062+0000 7fe961427640 1 -- 172.21.15.81:0/2630101606 >> 172.21.15.81:0/2630101606 conn(0x7fe95c0d3c60 msgr2=0x7fe95c0d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:47.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.062+0000 7fe961427640 1 -- 172.21.15.81:0/2630101606 shutdown_connections 2024-06-29T14:37:47.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.062+0000 7fe961427640 1 -- 172.21.15.81:0/2630101606 wait complete. 2024-06-29T14:37:47.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:37:47.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.195+0000 7f90186b2640 1 -- 172.21.15.81:0/1874345341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ffc004800 msgr2=0x7f8ffc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.195+0000 7f90186b2640 1 --2- 172.21.15.81:0/1874345341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ffc004800 0x7f8ffc004c10 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f9000009b50 tx=0x7f8ffc005830 comp rx=0 tx=0).stop 2024-06-29T14:37:47.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.195+0000 7f9016c4f640 1 -- 172.21.15.81:0/1874345341 reap_dead start 2024-06-29T14:37:47.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.195+0000 7f90186b2640 1 -- 172.21.15.81:0/1874345341 shutdown_connections 2024-06-29T14:37:47.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.195+0000 7f90186b2640 1 -- 172.21.15.81:0/1874345341 >> 172.21.15.81:0/1874345341 conn(0x7f901006ccc0 msgr2=0x7f90100b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:47.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.195+0000 7f90186b2640 1 -- 172.21.15.81:0/1874345341 shutdown_connections 2024-06-29T14:37:47.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.196+0000 7f90186b2640 1 -- 172.21.15.81:0/1874345341 wait complete. 2024-06-29T14:37:47.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.197+0000 7f90186b2640 1 Processor -- start 2024-06-29T14:37:47.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.197+0000 7f90186b2640 1 -- start start 2024-06-29T14:37:47.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.198+0000 7f90186b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ffc004800 0x7f9010158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.198+0000 7f90186b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90100aa2b0 0x7f901013f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f90186b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90101400c0 0x7f90101545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f90186b2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f901015ed70 con 0x7f8ffc004800 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f90186b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f90100cf720 con 0x7f90101400c0 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f90186b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f901015eef0 con 0x7f90100aa2b0 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f901644e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ffc004800 0x7f9010158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f901644e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ffc004800 0x7f9010158570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46080/0 (socket says 172.21.15.81:46080) 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f901644e640 1 -- 172.21.15.81:0/3774057536 learned_addr learned my addr 172.21.15.81:0/3774057536 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f9015c4d640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90100aa2b0 0x7f901013f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f9016c4f640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90101400c0 0x7f90101545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f901644e640 1 -- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90100aa2b0 msgr2=0x7f901013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f901644e640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90100aa2b0 0x7f901013f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f901644e640 1 -- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90101400c0 msgr2=0x7f90101545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f901644e640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90101400c0 0x7f90101545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f901644e640 1 -- 172.21.15.81:0/3774057536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90100a9560 con 0x7f8ffc004800 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f9015c4d640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90100aa2b0 0x7f901013f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f9016c4f640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90101400c0 0x7f90101545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f901644e640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ffc004800 0x7f9010158570 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f900c00e630 tx=0x7f900c00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f9006ffd640 1 -- 172.21.15.81:0/3774057536 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900c015070 con 0x7f8ffc004800 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f9006ffd640 1 -- 172.21.15.81:0/3774057536 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f900c00ca90 con 0x7f8ffc004800 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f90186b2640 1 -- 172.21.15.81:0/3774057536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90100a97d0 con 0x7f8ffc004800 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f9006ffd640 1 -- 172.21.15.81:0/3774057536 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900c00ce80 con 0x7f8ffc004800 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.199+0000 7f90186b2640 1 -- 172.21.15.81:0/3774057536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90100a9d90 con 0x7f8ffc004800 2024-06-29T14:37:47.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.200+0000 7f90186b2640 1 -- 172.21.15.81:0/3774057536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9010073440 con 0x7f8ffc004800 2024-06-29T14:37:47.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.201+0000 7f9006ffd640 1 -- 172.21.15.81:0/3774057536 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f900c018070 con 0x7f8ffc004800 2024-06-29T14:37:47.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.202+0000 7f9006ffd640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8ff40763f0 0x7f8ff4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.202+0000 7f9006ffd640 1 -- 172.21.15.81:0/3774057536 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5703+0+0 (secure 0 0 0) 0x7f900c097e30 con 0x7f8ffc004800 2024-06-29T14:37:47.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.202+0000 7f9015c4d640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8ff40763f0 0x7f8ff4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.202+0000 7f9015c4d640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8ff40763f0 0x7f8ff4078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9000009b50 tx=0x7f900002d040 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:47.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.206+0000 7f9006ffd640 1 -- 172.21.15.81:0/3774057536 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9010073980 con 0x7f8ffc004800 2024-06-29T14:37:47.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.389+0000 7f90186b2640 1 -- 172.21.15.81:0/3774057536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f9010159c80 con 0x7f8ffc004800 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.389+0000 7f9006ffd640 1 -- 172.21.15.81:0/3774057536 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v39) ==== 59+0+1998 (secure 0 0 0) 0x7f900c061570 con 0x7f8ffc004800 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T14:37:47.387 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T14:37:47.388 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:37:47.389 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 -- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8ff40763f0 msgr2=0x7f8ff4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8ff40763f0 0x7f8ff4078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9000009b50 tx=0x7f900002d040 comp rx=0 tx=0).stop 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 -- 172.21.15.81:0/3774057536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ffc004800 msgr2=0x7f9010158570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ffc004800 0x7f9010158570 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f900c00e630 tx=0x7f900c00eaf0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 -- 172.21.15.81:0/3774057536 shutdown_connections 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8ff40763f0 0x7f8ff4078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90101400c0 0x7f90101545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90100aa2b0 0x7f901013f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 --2- 172.21.15.81:0/3774057536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ffc004800 0x7f9010158570 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 -- 172.21.15.81:0/3774057536 >> 172.21.15.81:0/3774057536 conn(0x7f901006ccc0 msgr2=0x7f90101590d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 -- 172.21.15.81:0/3774057536 shutdown_connections 2024-06-29T14:37:47.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.391+0000 7f90186b2640 1 -- 172.21.15.81:0/3774057536 wait complete. 2024-06-29T14:37:47.400 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:37:47.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.519+0000 7f06daf19640 1 -- 172.21.15.81:0/2976832569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c0005150 msgr2=0x7f06c000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.519+0000 7f06daf19640 1 --2- 172.21.15.81:0/2976832569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c0005150 0x7f06c000b9c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f06d408cf40 tx=0x7f06c0005830 comp rx=0 tx=0).stop 2024-06-29T14:37:47.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.519+0000 7f06d94b6640 1 -- 172.21.15.81:0/2976832569 reap_dead start 2024-06-29T14:37:47.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.519+0000 7f06daf19640 1 -- 172.21.15.81:0/2976832569 shutdown_connections 2024-06-29T14:37:47.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.519+0000 7f06daf19640 1 -- 172.21.15.81:0/2976832569 >> 172.21.15.81:0/2976832569 conn(0x7f06d409cf30 msgr2=0x7f06d40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:47.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.519+0000 7f06daf19640 1 -- 172.21.15.81:0/2976832569 shutdown_connections 2024-06-29T14:37:47.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.519+0000 7f06daf19640 1 -- 172.21.15.81:0/2976832569 wait complete. 2024-06-29T14:37:47.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.521+0000 7f06daf19640 1 Processor -- start 2024-06-29T14:37:47.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.521+0000 7f06daf19640 1 -- start start 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06daf19640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c0005150 0x7f06d415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06daf19640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06d40a1350 0x7f06d415ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06daf19640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06d409e4d0 0x7f06d409eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06daf19640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06d40b82a0 con 0x7f06d409e4d0 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06daf19640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f06d40b8420 con 0x7f06d40a1350 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06daf19640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f06d40b85a0 con 0x7f06c0005150 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d94b6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06d409e4d0 0x7f06d409eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d94b6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06d409e4d0 0x7f06d409eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46094/0 (socket says 172.21.15.81:46094) 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d94b6640 1 -- 172.21.15.81:0/2379959368 learned_addr learned my addr 172.21.15.81:0/2379959368 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d8cb5640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c0005150 0x7f06d415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d3fff640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06d40a1350 0x7f06d415ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d94b6640 1 -- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c0005150 msgr2=0x7f06d415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d94b6640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c0005150 0x7f06d415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d94b6640 1 -- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06d40a1350 msgr2=0x7f06d415ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d94b6640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06d40a1350 0x7f06d415ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d94b6640 1 -- 172.21.15.81:0/2379959368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d415b4c0 con 0x7f06d409e4d0 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.522+0000 7f06d8cb5640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c0005150 0x7f06d415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.523+0000 7f06d3fff640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06d40a1350 0x7f06d415ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.523+0000 7f06d94b6640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06d409e4d0 0x7f06d409eba0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00bbe0 tx=0x7f06bc0326a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.523+0000 7f06d17fa640 1 -- 172.21.15.81:0/2379959368 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc041070 con 0x7f06d409e4d0 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.523+0000 7f06d17fa640 1 -- 172.21.15.81:0/2379959368 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06bc032c90 con 0x7f06d409e4d0 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.523+0000 7f06daf19640 1 -- 172.21.15.81:0/2379959368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d415b7a0 con 0x7f06d409e4d0 2024-06-29T14:37:47.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.523+0000 7f06d17fa640 1 -- 172.21.15.81:0/2379959368 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc045640 con 0x7f06d409e4d0 2024-06-29T14:37:47.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.523+0000 7f06daf19640 1 -- 172.21.15.81:0/2379959368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d415bd60 con 0x7f06d409e4d0 2024-06-29T14:37:47.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.524+0000 7f06daf19640 1 -- 172.21.15.81:0/2379959368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d40a1810 con 0x7f06d409e4d0 2024-06-29T14:37:47.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.525+0000 7f06d17fa640 1 -- 172.21.15.81:0/2379959368 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f06bc0457e0 con 0x7f06d409e4d0 2024-06-29T14:37:47.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.525+0000 7f06d17fa640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f06b4076420 0x7f06b4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:47.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.525+0000 7f06d17fa640 1 -- 172.21.15.81:0/2379959368 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5703+0+0 (secure 0 0 0) 0x7f06bc0b85a0 con 0x7f06d409e4d0 2024-06-29T14:37:47.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.525+0000 7f06d8cb5640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f06b4076420 0x7f06b4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:47.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.526+0000 7f06d8cb5640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f06b4076420 0x7f06b4078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f06cc001eb0 tx=0x7f06cc000ed0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:47.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.529+0000 7f06d17fa640 1 -- 172.21.15.81:0/2379959368 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f06bc081d50 con 0x7f06d409e4d0 2024-06-29T14:37:47.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.708+0000 7f06daf19640 1 -- 172.21.15.81:0/2379959368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f06d40d46a0 con 0x7f06d409e4d0 2024-06-29T14:37:47.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.711+0000 7f06d17fa640 1 -- 172.21.15.81:0/2379959368 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) ==== 122+0+0 (secure 0 0 0) 0x7f06d40d46a0 con 0x7f06d409e4d0 2024-06-29T14:37:47.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.713+0000 7f06daf19640 1 -- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f06b4076420 msgr2=0x7f06b4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.713+0000 7f06daf19640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f06b4076420 0x7f06b4078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f06cc001eb0 tx=0x7f06cc000ed0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.714+0000 7f06daf19640 1 -- 172.21.15.81:0/2379959368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06d409e4d0 msgr2=0x7f06d409eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.714+0000 7f06daf19640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06d409e4d0 0x7f06d409eba0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f06bc00bbe0 tx=0x7f06bc0326a0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.714+0000 7f06daf19640 1 -- 172.21.15.81:0/2379959368 shutdown_connections 2024-06-29T14:37:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.714+0000 7f06daf19640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f06b4076420 0x7f06b4078930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.714+0000 7f06daf19640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06d409e4d0 0x7f06d409eba0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.714+0000 7f06daf19640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06d40a1350 0x7f06d415ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.714+0000 7f06daf19640 1 --2- 172.21.15.81:0/2379959368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c0005150 0x7f06d415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.714+0000 7f06daf19640 1 -- 172.21.15.81:0/2379959368 >> 172.21.15.81:0/2379959368 conn(0x7f06d409cf30 msgr2=0x7f06d4161270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.714+0000 7f06daf19640 1 -- 172.21.15.81:0/2379959368 shutdown_connections 2024-06-29T14:37:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:47.714+0000 7f06daf19640 1 -- 172.21.15.81:0/2379959368 wait complete. 2024-06-29T14:37:47.722 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:37:47.722 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:47.771 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:37:47.817 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:37:47.817 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:47.817 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:47.833 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:47.833 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:37:47.833 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:47.851 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:47.851 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:47.851 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:47.888 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:37:47.889 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:37:47.889 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:47.908 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:37:47.909 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:37:47.909 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:48.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.044+0000 7f152b19e640 1 -- 172.21.15.81:0/159382424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f152415b5b0 msgr2=0x7f15240b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.044+0000 7f152b19e640 1 --2- 172.21.15.81:0/159382424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f152415b5b0 0x7f15240b4430 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1514009830 tx=0x7f151402a480 comp rx=0 tx=0).stop 2024-06-29T14:37:48.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.044+0000 7f152b19e640 1 -- 172.21.15.81:0/159382424 shutdown_connections 2024-06-29T14:37:48.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.044+0000 7f152b19e640 1 --2- 172.21.15.81:0/159382424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f152415b5b0 0x7f15240b4430 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.044+0000 7f152b19e640 1 -- 172.21.15.81:0/159382424 >> 172.21.15.81:0/159382424 conn(0x7f152406ccc0 msgr2=0x7f15240b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:48.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.044+0000 7f152b19e640 1 -- 172.21.15.81:0/159382424 shutdown_connections 2024-06-29T14:37:48.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.044+0000 7f152b19e640 1 -- 172.21.15.81:0/159382424 wait complete. 2024-06-29T14:37:48.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.046+0000 7f152b19e640 1 Processor -- start 2024-06-29T14:37:48.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.046+0000 7f152b19e640 1 -- start start 2024-06-29T14:37:48.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f152b19e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f152406f660 0x7f15241596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f152b19e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15240b35b0 0x7f1524148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f152b19e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f152415b5b0 0x7f15241f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f152b19e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f15240725b0 con 0x7f152406f660 2024-06-29T14:37:48.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f152b19e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1524072430 con 0x7f152415b5b0 2024-06-29T14:37:48.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f152b19e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1524072730 con 0x7f15240b35b0 2024-06-29T14:37:48.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f1528f3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f152406f660 0x7f15241596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f1528f3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f152406f660 0x7f15241596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46120/0 (socket says 172.21.15.81:46120) 2024-06-29T14:37:48.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f1528f3a640 1 -- 172.21.15.81:0/1180059757 learned_addr learned my addr 172.21.15.81:0/1180059757 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:48.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f152973b640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f152415b5b0 0x7f15241f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f151bfff640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15240b35b0 0x7f1524148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f1528f3a640 1 -- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15240b35b0 msgr2=0x7f1524148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f1528f3a640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15240b35b0 0x7f1524148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f1528f3a640 1 -- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f152415b5b0 msgr2=0x7f15241f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f1528f3a640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f152415b5b0 0x7f15241f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f1528f3a640 1 -- 172.21.15.81:0/1180059757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1524159e70 con 0x7f152406f660 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f152973b640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f152415b5b0 0x7f15241f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f151bfff640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15240b35b0 0x7f1524148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f1528f3a640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f152406f660 0x7f15241596b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f152000c3c0 tx=0x7f152000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f15197fa640 1 -- 172.21.15.81:0/1180059757 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1520017070 con 0x7f152406f660 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f15197fa640 1 -- 172.21.15.81:0/1180059757 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15200035f0 con 0x7f152406f660 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.048+0000 7f152b19e640 1 -- 172.21.15.81:0/1180059757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15240a9560 con 0x7f152406f660 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.049+0000 7f15197fa640 1 -- 172.21.15.81:0/1180059757 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1520003a80 con 0x7f152406f660 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.049+0000 7f152b19e640 1 -- 172.21.15.81:0/1180059757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15240a9b20 con 0x7f152406f660 2024-06-29T14:37:48.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.049+0000 7f152b19e640 1 -- 172.21.15.81:0/1180059757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15240b4430 con 0x7f152406f660 2024-06-29T14:37:48.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.050+0000 7f15197fa640 1 -- 172.21.15.81:0/1180059757 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f152001a070 con 0x7f152406f660 2024-06-29T14:37:48.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.051+0000 7f15197fa640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1510076420 0x7f1510078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.051+0000 7f15197fa640 1 -- 172.21.15.81:0/1180059757 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5703+0+0 (secure 0 0 0) 0x7f1520096ca0 con 0x7f152406f660 2024-06-29T14:37:48.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.051+0000 7f151bfff640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1510076420 0x7f1510078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.052+0000 7f151bfff640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1510076420 0x7f1510078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1514034000 tx=0x7f1514004730 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:48.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.055+0000 7f15197fa640 1 -- 172.21.15.81:0/1180059757 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1520060460 con 0x7f152406f660 2024-06-29T14:37:48.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.272+0000 7f152b19e640 1 -- 172.21.15.81:0/1180059757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1524069550 con 0x7f152406f660 2024-06-29T14:37:48.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.273+0000 7f15197fa640 1 -- 172.21.15.81:0/1180059757 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+3914 (secure 0 0 0) 0x7f152005fd60 con 0x7f152406f660 2024-06-29T14:37:48.270 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:48.270 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":36,"btime":"2024-06-29T14:37:41:857001+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5681,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3148404249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3148404249},{"type":"v1","addr":"172.21.15.167:6836","nonce":3148404249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6430,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/4072587146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":4072587146},{"type":"v1","addr":"172.21.15.167:6837","nonce":4072587146}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":6549,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/704367026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":704367026},{"type":"v1","addr":"172.21.15.81:6834","nonce":704367026}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6552,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2316096753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2316096753},{"type":"v1","addr":"172.21.15.81:6835","nonce":2316096753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[]} 2024-06-29T14:37:48.271 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 36 2024-06-29T14:37:48.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 -- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1510076420 msgr2=0x7f1510078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1510076420 0x7f1510078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1514034000 tx=0x7f1514004730 comp rx=0 tx=0).stop 2024-06-29T14:37:48.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 -- 172.21.15.81:0/1180059757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f152406f660 msgr2=0x7f15241596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f152406f660 0x7f15241596b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f152000c3c0 tx=0x7f152000c880 comp rx=0 tx=0).stop 2024-06-29T14:37:48.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 -- 172.21.15.81:0/1180059757 shutdown_connections 2024-06-29T14:37:48.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1510076420 0x7f1510078930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f152415b5b0 0x7f15241f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15240b35b0 0x7f1524148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 --2- 172.21.15.81:0/1180059757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f152406f660 0x7f15241596b0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 -- 172.21.15.81:0/1180059757 >> 172.21.15.81:0/1180059757 conn(0x7f152406ccc0 msgr2=0x7f152415d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:48.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 -- 172.21.15.81:0/1180059757 shutdown_connections 2024-06-29T14:37:48.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.275+0000 7f152b19e640 1 -- 172.21.15.81:0/1180059757 wait complete. 2024-06-29T14:37:48.285 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:37:48.285 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:37:48.285 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:37:48.285 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:48.285 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:37:48.285 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:37:48.286 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:37:48.286 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:48.286 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:37:48.286 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:37:48.286 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:37:48.286 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:48.286 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:37:48.286 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:37:48.287 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:37:48.287 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:37:48.295 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:37:48.295 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:37:48.296 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:37:48.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:37:48.298 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:37:48.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:37:48.300 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:37:48.300 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:37:48.337 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:37:48.338 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:48.338 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:37:48.339 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:48.342 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:37:48.342 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:48.343 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:37:48.344 DEBUG:teuthology.parallel:result is None 2024-06-29T14:37:48.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:37:48.423 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:37:48.423 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T14:37:48.437 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T14:37:48.437 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:37:48.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.467+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1616219728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8be0004800 msgr2=0x7f8be0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.467+0000 7f8bfed9e640 1 --2- 172.21.15.81:0/1616219728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8be0004800 0x7f8be0004c10 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f8bec001830 tx=0x7f8be0005830 comp rx=0 tx=0).stop 2024-06-29T14:37:48.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.467+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1616219728 shutdown_connections 2024-06-29T14:37:48.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.467+0000 7f8bfed9e640 1 --2- 172.21.15.81:0/1616219728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8be0004800 0x7f8be0004c10 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.467+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1616219728 >> 172.21.15.81:0/1616219728 conn(0x7f8bf809cf30 msgr2=0x7f8bf80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:48.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.467+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1616219728 shutdown_connections 2024-06-29T14:37:48.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.468+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1616219728 wait complete. 2024-06-29T14:37:48.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.469+0000 7f8bfed9e640 1 Processor -- start 2024-06-29T14:37:48.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.469+0000 7f8bfed9e640 1 -- start start 2024-06-29T14:37:48.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfed9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8be0004800 0x7f8bf81550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfed9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf81f4bc0 0x7f8bf81f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfed9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf806d170 0x7f8bf806d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfed9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf80d0810 con 0x7f8bf806d170 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfed9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf80d0690 con 0x7f8be0004800 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfed9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf80d0990 con 0x7f8bf81f4bc0 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfd33b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf806d170 0x7f8bf806d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfd33b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf806d170 0x7f8bf806d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46168/0 (socket says 172.21.15.81:46168) 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfd33b640 1 -- 172.21.15.81:0/1674558435 learned_addr learned my addr 172.21.15.81:0/1674558435 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfcb3a640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8be0004800 0x7f8bf81550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bf7fff640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf81f4bc0 0x7f8bf81f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfd33b640 1 -- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf81f4bc0 msgr2=0x7f8bf81f4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfd33b640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf81f4bc0 0x7f8bf81f4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfd33b640 1 -- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8be0004800 msgr2=0x7f8bf81550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfd33b640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8be0004800 0x7f8bf81550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfd33b640 1 -- 172.21.15.81:0/1674558435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf8155600 con 0x7f8bf806d170 2024-06-29T14:37:48.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfcb3a640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8be0004800 0x7f8bf81550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:37:48.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bfd33b640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf806d170 0x7f8bf806d580 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80d43d0 tx=0x7f8bf00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:48.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.471+0000 7f8bf57fa640 1 -- 172.21.15.81:0/1674558435 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0011840 con 0x7f8bf806d170 2024-06-29T14:37:48.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.472+0000 7f8bf57fa640 1 -- 172.21.15.81:0/1674558435 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bf0017070 con 0x7f8bf806d170 2024-06-29T14:37:48.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.472+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1674558435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf806dac0 con 0x7f8bf806d170 2024-06-29T14:37:48.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.472+0000 7f8bf57fa640 1 -- 172.21.15.81:0/1674558435 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0011c80 con 0x7f8bf806d170 2024-06-29T14:37:48.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.472+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1674558435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf81f5b60 con 0x7f8bf806d170 2024-06-29T14:37:48.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.473+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1674558435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf80d1a80 con 0x7f8bf806d170 2024-06-29T14:37:48.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.473+0000 7f8bf57fa640 1 -- 172.21.15.81:0/1674558435 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f8bf001c460 con 0x7f8bf806d170 2024-06-29T14:37:48.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.474+0000 7f8bf57fa640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8be40763f0 0x7f8be4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:48.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.474+0000 7f8bf57fa640 1 -- 172.21.15.81:0/1674558435 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5703+0+0 (secure 0 0 0) 0x7f8bf0098060 con 0x7f8bf806d170 2024-06-29T14:37:48.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.474+0000 7f8bfcb3a640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8be40763f0 0x7f8be4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:48.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.475+0000 7f8bfcb3a640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8be40763f0 0x7f8be4078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009fd0 tx=0x7f8be8009500 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:48.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.478+0000 7f8bf57fa640 1 -- 172.21.15.81:0/1674558435 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8bf806dac0 con 0x7f8bf806d170 2024-06-29T14:37:48.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:48.673+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1674558435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f8bf81f5610 con 0x7f8bf806d170 2024-06-29T14:37:49.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.070+0000 7f8bf57fa640 1 -- 172.21.15.81:0/1674558435 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v85) ==== 97+0+0 (secure 0 0 0) 0x7f8bf8155600 con 0x7f8bf806d170 2024-06-29T14:37:49.068 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T14:37:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8be40763f0 msgr2=0x7f8be4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8be40763f0 0x7f8be4078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009fd0 tx=0x7f8be8009500 comp rx=0 tx=0).stop 2024-06-29T14:37:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1674558435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf806d170 msgr2=0x7f8bf806d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf806d170 0x7f8bf806d580 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80d43d0 tx=0x7f8bf00039c0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1674558435 shutdown_connections 2024-06-29T14:37:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f8be40763f0 0x7f8be4078900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf806d170 0x7f8bf806d580 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8bf81f4bc0 0x7f8bf81f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 --2- 172.21.15.81:0/1674558435 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8be0004800 0x7f8bf81550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1674558435 >> 172.21.15.81:0/1674558435 conn(0x7f8bf809cf30 msgr2=0x7f8bf806ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:49.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.073+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1674558435 shutdown_connections 2024-06-29T14:37:49.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.074+0000 7f8bfed9e640 1 -- 172.21.15.81:0/1674558435 wait complete. 2024-06-29T14:37:49.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:37:49.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.199+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1415692661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa070003a00 msgr2=0x7fa070003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.199+0000 7fa08d8fd640 1 --2- 172.21.15.81:0/1415692661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa070003a00 0x7fa070003e70 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa07800bb20 tx=0x7fa070005830 comp rx=0 tx=0).stop 2024-06-29T14:37:49.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.199+0000 7fa0877fe640 1 -- 172.21.15.81:0/1415692661 reap_dead start 2024-06-29T14:37:49.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.199+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1415692661 shutdown_connections 2024-06-29T14:37:49.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.199+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1415692661 >> 172.21.15.81:0/1415692661 conn(0x7fa08809cf30 msgr2=0x7fa0880d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:49.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.199+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1415692661 shutdown_connections 2024-06-29T14:37:49.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.199+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1415692661 wait complete. 2024-06-29T14:37:49.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.201+0000 7fa08d8fd640 1 Processor -- start 2024-06-29T14:37:49.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.201+0000 7fa08d8fd640 1 -- start start 2024-06-29T14:37:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa08d8fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa070003a00 0x7fa0880b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa08d8fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0881f7ca0 0x7fa0881f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa08d8fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0880b41d0 0x7fa0880b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa08d8fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa0880d2a90 con 0x7fa0881f7ca0 2024-06-29T14:37:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa08d8fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa0880d06e0 con 0x7fa0880b41d0 2024-06-29T14:37:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa08d8fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa0880d2c10 con 0x7fa070003a00 2024-06-29T14:37:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa0867fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0881f7ca0 0x7fa0881f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa0867fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0881f7ca0 0x7fa0881f80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46198/0 (socket says 172.21.15.81:46198) 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa0867fc640 1 -- 172.21.15.81:0/1477459587 learned_addr learned my addr 172.21.15.81:0/1477459587 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa086ffd640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa070003a00 0x7fa0880b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.202+0000 7fa0877fe640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0880b41d0 0x7fa0880b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa0867fc640 1 -- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa070003a00 msgr2=0x7fa0880b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa0867fc640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa070003a00 0x7fa0880b0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa0867fc640 1 -- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0880b41d0 msgr2=0x7fa0880b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa0867fc640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0880b41d0 0x7fa0880b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa0867fc640 1 -- 172.21.15.81:0/1477459587 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0881f85f0 con 0x7fa0881f7ca0 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa0867fc640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0881f7ca0 0x7fa0881f80b0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fa07c00f9d0 tx=0x7fa07c00fe90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa06ffff640 1 -- 172.21.15.81:0/1477459587 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c015070 con 0x7fa0881f7ca0 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa06ffff640 1 -- 172.21.15.81:0/1477459587 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa07c00b820 con 0x7fa0881f7ca0 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1477459587 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0880b1200 con 0x7fa0881f7ca0 2024-06-29T14:37:49.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa06ffff640 1 -- 172.21.15.81:0/1477459587 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c00bc10 con 0x7fa0881f7ca0 2024-06-29T14:37:49.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.203+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1477459587 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0880b17c0 con 0x7fa0881f7ca0 2024-06-29T14:37:49.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.204+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1477459587 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0880b0f70 con 0x7fa0881f7ca0 2024-06-29T14:37:49.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.205+0000 7fa06ffff640 1 -- 172.21.15.81:0/1477459587 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fa07c0035f0 con 0x7fa0881f7ca0 2024-06-29T14:37:49.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.205+0000 7fa06ffff640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa074076420 0x7fa074078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.205+0000 7fa06ffff640 1 -- 172.21.15.81:0/1477459587 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 4929+0+0 (secure 0 0 0) 0x7fa07c098260 con 0x7fa0881f7ca0 2024-06-29T14:37:49.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.206+0000 7fa086ffd640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa074076420 0x7fa074078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.206+0000 7fa086ffd640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa074076420 0x7fa074078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa078036000 tx=0x7fa07803c040 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:49.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.209+0000 7fa06ffff640 1 -- 172.21.15.81:0/1477459587 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa0881f85f0 con 0x7fa0881f7ca0 2024-06-29T14:37:49.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.458+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1477459587 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa0880d0940 con 0x7fa0881f7ca0 2024-06-29T14:37:49.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.459+0000 7fa06ffff640 1 -- 172.21.15.81:0/1477459587 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v31) ==== 80+0+5304 (secure 0 0 0) 0x7fa07c0615c0 con 0x7fa0881f7ca0 2024-06-29T14:37:49.456 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:49.456 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:37:49.457 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T14:37:49.457 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.457 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T14:37:49.457 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T14:37:49.457 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.457 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:49.457 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:37:49.459 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:37:49.459 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.459 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.459 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.459 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T14:37:49.460 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:37:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T14:37:49.462 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T14:37:49.463 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.463 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.463 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.463 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.463 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.463 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.463 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T14:37:49.463 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.464 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T14:37:49.465 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.467 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.468 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:37:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T14:37:49.471 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:37:49.472 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:37:49.473 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.473 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:49.473 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T14:37:49.473 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:37:49.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa074076420 msgr2=0x7fa074078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa074076420 0x7fa074078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa078036000 tx=0x7fa07803c040 comp rx=0 tx=0).stop 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1477459587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0881f7ca0 msgr2=0x7fa0881f80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0881f7ca0 0x7fa0881f80b0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fa07c00f9d0 tx=0x7fa07c00fe90 comp rx=0 tx=0).stop 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1477459587 shutdown_connections 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa074076420 0x7fa074078930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0880b41d0 0x7fa0880b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0881f7ca0 0x7fa0881f80b0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 --2- 172.21.15.81:0/1477459587 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa070003a00 0x7fa0880b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1477459587 >> 172.21.15.81:0/1477459587 conn(0x7fa08809cf30 msgr2=0x7fa08815f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1477459587 shutdown_connections 2024-06-29T14:37:49.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.461+0000 7fa08d8fd640 1 -- 172.21.15.81:0/1477459587 wait complete. 2024-06-29T14:37:49.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:49.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.592+0000 7eff3fd7a640 1 -- 172.21.15.81:0/4038152876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff38074520 msgr2=0x7eff3806f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.592+0000 7eff3fd7a640 1 --2- 172.21.15.81:0/4038152876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff38074520 0x7eff3806f140 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7eff28005fd0 tx=0x7eff2802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.592+0000 7eff3fd7a640 1 -- 172.21.15.81:0/4038152876 shutdown_connections 2024-06-29T14:37:49.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.592+0000 7eff3fd7a640 1 --2- 172.21.15.81:0/4038152876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff3806fa90 0x7eff3806ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.592+0000 7eff3fd7a640 1 --2- 172.21.15.81:0/4038152876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff38074520 0x7eff3806f140 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.592+0000 7eff3fd7a640 1 -- 172.21.15.81:0/4038152876 >> 172.21.15.81:0/4038152876 conn(0x7eff38072950 msgr2=0x7eff380b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:49.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.592+0000 7eff3fd7a640 1 -- 172.21.15.81:0/4038152876 shutdown_connections 2024-06-29T14:37:49.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.592+0000 7eff3fd7a640 1 -- 172.21.15.81:0/4038152876 wait complete. 2024-06-29T14:37:49.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.594+0000 7eff3fd7a640 1 Processor -- start 2024-06-29T14:37:49.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.594+0000 7eff3fd7a640 1 -- start start 2024-06-29T14:37:49.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3fd7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff3806f680 0x7eff3814ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3fd7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff3806fa90 0x7eff381f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3fd7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff38074520 0x7eff380d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3fd7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7eff3815b010 con 0x7eff38074520 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3fd7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7eff3806d150 con 0x7eff3806fa90 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3fd7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7eff3815b190 con 0x7eff3806f680 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3e317640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff38074520 0x7eff380d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3e317640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff38074520 0x7eff380d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46220/0 (socket says 172.21.15.81:46220) 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3e317640 1 -- 172.21.15.81:0/3599105497 learned_addr learned my addr 172.21.15.81:0/3599105497 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3db16640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff3806f680 0x7eff3814ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3d315640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff3806fa90 0x7eff381f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3e317640 1 -- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff3806f680 msgr2=0x7eff3814ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3e317640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff3806f680 0x7eff3814ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3e317640 1 -- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff3806fa90 msgr2=0x7eff381f71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3e317640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff3806fa90 0x7eff381f71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.595+0000 7eff3e317640 1 -- 172.21.15.81:0/3599105497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff381f7720 con 0x7eff38074520 2024-06-29T14:37:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.596+0000 7eff3e317640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff38074520 0x7eff380d0030 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7eff3000e9c0 tx=0x7eff3000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.596+0000 7eff367fc640 1 -- 172.21.15.81:0/3599105497 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30004d20 con 0x7eff38074520 2024-06-29T14:37:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.596+0000 7eff367fc640 1 -- 172.21.15.81:0/3599105497 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff3000cb20 con 0x7eff38074520 2024-06-29T14:37:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.596+0000 7eff3fd7a640 1 -- 172.21.15.81:0/3599105497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff3814f7c0 con 0x7eff38074520 2024-06-29T14:37:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.596+0000 7eff367fc640 1 -- 172.21.15.81:0/3599105497 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30005020 con 0x7eff38074520 2024-06-29T14:37:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.596+0000 7eff3fd7a640 1 -- 172.21.15.81:0/3599105497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff38151ea0 con 0x7eff38074520 2024-06-29T14:37:49.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.597+0000 7eff3fd7a640 1 -- 172.21.15.81:0/3599105497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff3814f530 con 0x7eff38074520 2024-06-29T14:37:49.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.598+0000 7eff367fc640 1 -- 172.21.15.81:0/3599105497 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7eff300051c0 con 0x7eff38074520 2024-06-29T14:37:49.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.598+0000 7eff367fc640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7eff24076420 0x7eff24078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.598+0000 7eff367fc640 1 -- 172.21.15.81:0/3599105497 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 4929+0+0 (secure 0 0 0) 0x7eff30097eb0 con 0x7eff38074520 2024-06-29T14:37:49.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.598+0000 7eff3db16640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7eff24076420 0x7eff24078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.599+0000 7eff3db16640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7eff24076420 0x7eff24078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff28033000 tx=0x7eff28004730 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:49.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.602+0000 7eff367fc640 1 -- 172.21.15.81:0/3599105497 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7eff30016030 con 0x7eff38074520 2024-06-29T14:37:49.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.784+0000 7eff3fd7a640 1 -- 172.21.15.81:0/3599105497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff380af4d0 con 0x7eff38074520 2024-06-29T14:37:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.785+0000 7eff367fc640 1 -- 172.21.15.81:0/3599105497 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+12993 (secure 0 0 0) 0x7eff30061210 con 0x7eff38074520 2024-06-29T14:37:49.783 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:49.783 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":87,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:49.082133+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:49.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 -- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7eff24076420 msgr2=0x7eff24078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7eff24076420 0x7eff24078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff28033000 tx=0x7eff28004730 comp rx=0 tx=0).stop 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 -- 172.21.15.81:0/3599105497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff38074520 msgr2=0x7eff380d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff38074520 0x7eff380d0030 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7eff3000e9c0 tx=0x7eff3000ee80 comp rx=0 tx=0).stop 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 -- 172.21.15.81:0/3599105497 shutdown_connections 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7eff24076420 0x7eff24078930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff38074520 0x7eff380d0030 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff3806fa90 0x7eff381f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 --2- 172.21.15.81:0/3599105497 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff3806f680 0x7eff3814ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 -- 172.21.15.81:0/3599105497 >> 172.21.15.81:0/3599105497 conn(0x7eff38072950 msgr2=0x7eff380b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 -- 172.21.15.81:0/3599105497 shutdown_connections 2024-06-29T14:37:49.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.787+0000 7eff3fd7a640 1 -- 172.21.15.81:0/3599105497 wait complete. 2024-06-29T14:37:49.796 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:37:49.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:37:49.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.914+0000 7f53710a4640 1 -- 172.21.15.81:0/1195417031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f536c073480 msgr2=0x7f536c15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.914+0000 7f53710a4640 1 --2- 172.21.15.81:0/1195417031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f536c073480 0x7f536c15b6e0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f536c072e80 tx=0x7f535402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.914+0000 7f53710a4640 1 -- 172.21.15.81:0/1195417031 shutdown_connections 2024-06-29T14:37:49.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.914+0000 7f53710a4640 1 --2- 172.21.15.81:0/1195417031 >> v2:172.21.15.167:3300/0 conn(0x7f536c15bc20 0x7f536c15e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.915+0000 7f53710a4640 1 --2- 172.21.15.81:0/1195417031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f536c073480 0x7f536c15b6e0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.915+0000 7f53710a4640 1 --2- 172.21.15.81:0/1195417031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f536c0cd150 0x7f536c072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.915+0000 7f53710a4640 1 -- 172.21.15.81:0/1195417031 >> 172.21.15.81:0/1195417031 conn(0x7f536c0b2920 msgr2=0x7f536c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:49.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.915+0000 7f53710a4640 1 -- 172.21.15.81:0/1195417031 shutdown_connections 2024-06-29T14:37:49.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.915+0000 7f53710a4640 1 -- 172.21.15.81:0/1195417031 wait complete. 2024-06-29T14:37:49.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.917+0000 7f53710a4640 1 Processor -- start 2024-06-29T14:37:49.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.917+0000 7f53710a4640 1 -- start start 2024-06-29T14:37:49.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f53710a4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f536c073480 0x7f536c146590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f53710a4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f536c0cd150 0x7f536c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f53710a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f536c15bc20 0x7f536c158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f53710a4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f536c15f010 con 0x7f536c15bc20 2024-06-29T14:37:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f53710a4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f536c15ee90 con 0x7f536c073480 2024-06-29T14:37:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f53710a4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f536c15f190 con 0x7f536c0cd150 2024-06-29T14:37:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f536b59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f536c15bc20 0x7f536c158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f536b59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f536c15bc20 0x7f536c158510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46228/0 (socket says 172.21.15.81:46228) 2024-06-29T14:37:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f536b59e640 1 -- 172.21.15.81:0/666002978 learned_addr learned my addr 172.21.15.81:0/666002978 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f536a59c640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f536c0cd150 0x7f536c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f536ad9d640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f536c073480 0x7f536c146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f536b59e640 1 -- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f536c0cd150 msgr2=0x7f536c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f536b59e640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f536c0cd150 0x7f536c157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f536b59e640 1 -- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f536c073480 msgr2=0x7f536c146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f536b59e640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f536c073480 0x7f536c146590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.918+0000 7f536b59e640 1 -- 172.21.15.81:0/666002978 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536c150b30 con 0x7f536c15bc20 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.919+0000 7f536b59e640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f536c15bc20 0x7f536c158510 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f535c00cae0 tx=0x7f535c007220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.919+0000 7f53637fe640 1 -- 172.21.15.81:0/666002978 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c007da0 con 0x7f536c15bc20 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.919+0000 7f53637fe640 1 -- 172.21.15.81:0/666002978 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f535c004c30 con 0x7f536c15bc20 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.919+0000 7f53710a4640 1 -- 172.21.15.81:0/666002978 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f536c0d0e90 con 0x7f536c15bc20 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.919+0000 7f53637fe640 1 -- 172.21.15.81:0/666002978 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c005280 con 0x7f536c15bc20 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.919+0000 7f53710a4640 1 -- 172.21.15.81:0/666002978 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f536c0d1450 con 0x7f536c15bc20 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.920+0000 7f53710a4640 1 -- 172.21.15.81:0/666002978 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f536c15f6b0 con 0x7f536c15bc20 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.921+0000 7f53637fe640 1 -- 172.21.15.81:0/666002978 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f535c005420 con 0x7f536c15bc20 2024-06-29T14:37:49.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.921+0000 7f53637fe640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f534c0763f0 0x7f534c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:49.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.921+0000 7f53637fe640 1 -- 172.21.15.81:0/666002978 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 4929+0+0 (secure 0 0 0) 0x7f535c097fd0 con 0x7f536c15bc20 2024-06-29T14:37:49.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.921+0000 7f536ad9d640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f534c0763f0 0x7f534c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:49.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.922+0000 7f536ad9d640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f534c0763f0 0x7f534c078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f536400aa10 tx=0x7f5364005ed0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:49.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:49.926+0000 7f53637fe640 1 -- 172.21.15.81:0/666002978 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f535c014070 con 0x7f536c15bc20 2024-06-29T14:37:50.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.106+0000 7f53710a4640 1 -- 172.21.15.81:0/666002978 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f536c1508c0 con 0x7f536c15bc20 2024-06-29T14:37:50.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.106+0000 7f53637fe640 1 -- 172.21.15.81:0/666002978 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f535c061450 con 0x7f536c15bc20 2024-06-29T14:37:50.104 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:37:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.108+0000 7f53710a4640 1 -- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f534c0763f0 msgr2=0x7f534c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.108+0000 7f53710a4640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f534c0763f0 0x7f534c078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f536400aa10 tx=0x7f5364005ed0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.108+0000 7f53710a4640 1 -- 172.21.15.81:0/666002978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f536c15bc20 msgr2=0x7f536c158510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.108+0000 7f53710a4640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f536c15bc20 0x7f536c158510 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f535c00cae0 tx=0x7f535c007220 comp rx=0 tx=0).stop 2024-06-29T14:37:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.108+0000 7f53710a4640 1 -- 172.21.15.81:0/666002978 shutdown_connections 2024-06-29T14:37:50.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.108+0000 7f53710a4640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f534c0763f0 0x7f534c078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.108+0000 7f53710a4640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f536c15bc20 0x7f536c158510 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.108+0000 7f53710a4640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f536c0cd150 0x7f536c157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.108+0000 7f53710a4640 1 --2- 172.21.15.81:0/666002978 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f536c073480 0x7f536c146590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.108+0000 7f53710a4640 1 -- 172.21.15.81:0/666002978 >> 172.21.15.81:0/666002978 conn(0x7f536c0b2920 msgr2=0x7f536c154a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:50.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.109+0000 7f53710a4640 1 -- 172.21.15.81:0/666002978 shutdown_connections 2024-06-29T14:37:50.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.109+0000 7f53710a4640 1 -- 172.21.15.81:0/666002978 wait complete. 2024-06-29T14:37:50.117 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:37:50.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.233+0000 7fccec25b640 1 -- 172.21.15.81:0/3546290232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcce40cea10 msgr2=0x7fcce40b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.233+0000 7fccec25b640 1 --2- 172.21.15.81:0/3546290232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcce40cea10 0x7fcce40b47b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fcce40b2c10 tx=0x7fccd802a540 comp rx=0 tx=0).stop 2024-06-29T14:37:50.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.233+0000 7fccec25b640 1 -- 172.21.15.81:0/3546290232 shutdown_connections 2024-06-29T14:37:50.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.233+0000 7fccec25b640 1 --2- 172.21.15.81:0/3546290232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcce40cea10 0x7fcce40b47b0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.233+0000 7fccec25b640 1 --2- 172.21.15.81:0/3546290232 >> v2:172.21.15.167:3300/0 conn(0x7fcce40b6d60 0x7fcce40ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.233+0000 7fccec25b640 1 -- 172.21.15.81:0/3546290232 >> 172.21.15.81:0/3546290232 conn(0x7fcce40d1960 msgr2=0x7fcce40d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:50.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.233+0000 7fccec25b640 1 -- 172.21.15.81:0/3546290232 shutdown_connections 2024-06-29T14:37:50.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.233+0000 7fccec25b640 1 -- 172.21.15.81:0/3546290232 wait complete. 2024-06-29T14:37:50.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.235+0000 7fccec25b640 1 Processor -- start 2024-06-29T14:37:50.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.235+0000 7fccec25b640 1 -- start start 2024-06-29T14:37:50.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.236+0000 7fccec25b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcce40b4cf0 0x7fcce41f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.236+0000 7fccec25b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcce40b6d60 0x7fcce409d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.236+0000 7fccec25b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcce40cea10 0x7fcce40b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.236+0000 7fccec25b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcce40cd540 con 0x7fcce40b4cf0 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.236+0000 7fccec25b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcce40d0a10 con 0x7fcce40b6d60 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.236+0000 7fccec25b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcce40cd6c0 con 0x7fcce40cea10 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce9ff7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcce40b4cf0 0x7fcce41f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce9ff7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcce40b4cf0 0x7fcce41f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47624/0 (socket says 172.21.15.81:47624) 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce9ff7640 1 -- 172.21.15.81:0/76881179 learned_addr learned my addr 172.21.15.81:0/76881179 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce97f6640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcce40b6d60 0x7fcce409d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fccea7f8640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcce40cea10 0x7fcce40b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce9ff7640 1 -- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcce40cea10 msgr2=0x7fcce40b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce9ff7640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcce40cea10 0x7fcce40b0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce9ff7640 1 -- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcce40b6d60 msgr2=0x7fcce409d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce9ff7640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcce40b6d60 0x7fcce409d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce9ff7640 1 -- 172.21.15.81:0/76881179 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce409d810 con 0x7fcce40b4cf0 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fccea7f8640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcce40cea10 0x7fcce40b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce9ff7640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcce40b4cf0 0x7fcce41f2660 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fccd400d700 tx=0x7fccd400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce27fc640 1 -- 172.21.15.81:0/76881179 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd4017070 con 0x7fcce40b4cf0 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce27fc640 1 -- 172.21.15.81:0/76881179 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fccd4004310 con 0x7fcce40b4cf0 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fccec25b640 1 -- 172.21.15.81:0/76881179 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcce40af840 con 0x7fcce40b4cf0 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fcce27fc640 1 -- 172.21.15.81:0/76881179 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd4009a20 con 0x7fcce40b4cf0 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.237+0000 7fccec25b640 1 -- 172.21.15.81:0/76881179 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcce40afe00 con 0x7fcce40b4cf0 2024-06-29T14:37:50.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.238+0000 7fccec25b640 1 -- 172.21.15.81:0/76881179 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcce40ce4d0 con 0x7fcce40b4cf0 2024-06-29T14:37:50.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.239+0000 7fcce27fc640 1 -- 172.21.15.81:0/76881179 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fccd4009bc0 con 0x7fcce40b4cf0 2024-06-29T14:37:50.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.240+0000 7fcce27fc640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fccd00763f0 0x7fccd0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.240+0000 7fcce27fc640 1 -- 172.21.15.81:0/76881179 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 4929+0+0 (secure 0 0 0) 0x7fccd4097610 con 0x7fcce40b4cf0 2024-06-29T14:37:50.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.240+0000 7fcce97f6640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fccd00763f0 0x7fccd0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.241+0000 7fcce97f6640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fccd00763f0 0x7fccd0078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fccd8033000 tx=0x7fccd8004730 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:50.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.244+0000 7fcce27fc640 1 -- 172.21.15.81:0/76881179 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fccd4061050 con 0x7fcce40b4cf0 2024-06-29T14:37:50.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.425+0000 7fccec25b640 1 -- 172.21.15.81:0/76881179 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fcce40cdec0 con 0x7fcce40b4cf0 2024-06-29T14:37:50.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.426+0000 7fcce27fc640 1 -- 172.21.15.81:0/76881179 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v87) ==== 84+0+65 (secure 0 0 0) 0x7fccd4060950 con 0x7fcce40b4cf0 2024-06-29T14:37:50.423 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:50.423 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:37:50.424 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:37:50.424 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T14:37:50.424 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T14:37:50.424 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:37:50.424 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:37:50.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.427+0000 7fccec25b640 1 -- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fccd00763f0 msgr2=0x7fccd0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.427+0000 7fccec25b640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fccd00763f0 0x7fccd0078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fccd8033000 tx=0x7fccd8004730 comp rx=0 tx=0).stop 2024-06-29T14:37:50.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.427+0000 7fccec25b640 1 -- 172.21.15.81:0/76881179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcce40b4cf0 msgr2=0x7fcce41f2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.427+0000 7fccec25b640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcce40b4cf0 0x7fcce41f2660 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fccd400d700 tx=0x7fccd400dbc0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.427+0000 7fccec25b640 1 -- 172.21.15.81:0/76881179 shutdown_connections 2024-06-29T14:37:50.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.428+0000 7fccec25b640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fccd00763f0 0x7fccd0078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.428+0000 7fccec25b640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcce40cea10 0x7fcce40b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.428+0000 7fccec25b640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcce40b6d60 0x7fcce409d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.428+0000 7fccec25b640 1 --2- 172.21.15.81:0/76881179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcce40b4cf0 0x7fcce41f2660 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.428+0000 7fccec25b640 1 -- 172.21.15.81:0/76881179 >> 172.21.15.81:0/76881179 conn(0x7fcce40d1960 msgr2=0x7fcce40717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:50.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.428+0000 7fccec25b640 1 -- 172.21.15.81:0/76881179 shutdown_connections 2024-06-29T14:37:50.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.428+0000 7fccec25b640 1 -- 172.21.15.81:0/76881179 wait complete. 2024-06-29T14:37:50.436 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:37:50.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:37:50.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.557+0000 7f08a5265640 1 -- 172.21.15.81:0/2920957981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08a00d1e80 msgr2=0x7f08a00d2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.557+0000 7f08a5265640 1 --2- 172.21.15.81:0/2920957981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08a00d1e80 0x7f08a00d2290 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f0890005fd0 tx=0x7f089002a9f0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.557+0000 7f08a5265640 1 -- 172.21.15.81:0/2920957981 shutdown_connections 2024-06-29T14:37:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.557+0000 7f08a5265640 1 --2- 172.21.15.81:0/2920957981 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08a00cea50 0x7f08a00cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.557+0000 7f08a5265640 1 --2- 172.21.15.81:0/2920957981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08a00d1e80 0x7f08a00d2290 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.557+0000 7f08a5265640 1 -- 172.21.15.81:0/2920957981 >> 172.21.15.81:0/2920957981 conn(0x7f08a00b5700 msgr2=0x7f08a00b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.557+0000 7f08a5265640 1 -- 172.21.15.81:0/2920957981 shutdown_connections 2024-06-29T14:37:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.557+0000 7f08a5265640 1 -- 172.21.15.81:0/2920957981 wait complete. 2024-06-29T14:37:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.559+0000 7f08a5265640 1 Processor -- start 2024-06-29T14:37:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.559+0000 7f08a5265640 1 -- start start 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f08a5265640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08a00cea50 0x7f08a009eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f08a5265640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08a00cf3a0 0x7f08a01dfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f08a5265640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08a00d1e80 0x7f08a00cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f08a5265640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f08a00d1960 con 0x7f08a00cf3a0 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f08a5265640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f08a00d17e0 con 0x7f08a00d1e80 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f08a5265640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f08a00cf7b0 con 0x7f08a00cea50 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f089e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08a00cf3a0 0x7f08a01dfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f089e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08a00cf3a0 0x7f08a01dfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47650/0 (socket says 172.21.15.81:47650) 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f089e7fc640 1 -- 172.21.15.81:0/3904182853 learned_addr learned my addr 172.21.15.81:0/3904182853 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f089effd640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08a00cea50 0x7f08a009eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f089e7fc640 1 -- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08a00cea50 msgr2=0x7f08a009eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f089e7fc640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08a00cea50 0x7f08a009eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f089e7fc640 1 -- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08a00d1e80 msgr2=0x7f08a00cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f089e7fc640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08a00d1e80 0x7f08a00cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.560+0000 7f089e7fc640 1 -- 172.21.15.81:0/3904182853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a00a1650 con 0x7f08a00cf3a0 2024-06-29T14:37:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.561+0000 7f089e7fc640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08a00cf3a0 0x7f08a01dfdc0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f089400d680 tx=0x7f089400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.561+0000 7f08877fe640 1 -- 172.21.15.81:0/3904182853 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894017070 con 0x7f08a00cf3a0 2024-06-29T14:37:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.561+0000 7f08877fe640 1 -- 172.21.15.81:0/3904182853 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0894009ab0 con 0x7f08a00cf3a0 2024-06-29T14:37:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.561+0000 7f08a5265640 1 -- 172.21.15.81:0/3904182853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a01e0c30 con 0x7f08a00cf3a0 2024-06-29T14:37:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.561+0000 7f08877fe640 1 -- 172.21.15.81:0/3904182853 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894010460 con 0x7f08a00cf3a0 2024-06-29T14:37:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.561+0000 7f08a5265640 1 -- 172.21.15.81:0/3904182853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a01e11f0 con 0x7f08a00cf3a0 2024-06-29T14:37:50.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.562+0000 7f08a5265640 1 -- 172.21.15.81:0/3904182853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a00ca460 con 0x7f08a00cf3a0 2024-06-29T14:37:50.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.563+0000 7f08877fe640 1 -- 172.21.15.81:0/3904182853 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f089401a070 con 0x7f08a00cf3a0 2024-06-29T14:37:50.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.563+0000 7f08877fe640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f088c0763f0 0x7f088c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:50.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.563+0000 7f08877fe640 1 -- 172.21.15.81:0/3904182853 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 4929+0+0 (secure 0 0 0) 0x7f0894097710 con 0x7f08a00cf3a0 2024-06-29T14:37:50.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.563+0000 7f089effd640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f088c0763f0 0x7f088c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:50.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.564+0000 7f089effd640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f088c0763f0 0x7f088c078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0890035000 tx=0x7f0890004ab0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:50.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.568+0000 7f08877fe640 1 -- 172.21.15.81:0/3904182853 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f08a00ca460 con 0x7f08a00cf3a0 2024-06-29T14:37:50.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:50.748+0000 7f08a5265640 1 -- 172.21.15.81:0/3904182853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f08a00cee60 con 0x7f08a00cf3a0 2024-06-29T14:37:51.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.077+0000 7f08877fe640 1 -- 172.21.15.81:0/3904182853 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v88) ==== 138+0+0 (secure 0 0 0) 0x7f0894061150 con 0x7f08a00cf3a0 2024-06-29T14:37:51.075 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T14:37:51.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 -- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f088c0763f0 msgr2=0x7f088c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:51.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f088c0763f0 0x7f088c078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0890035000 tx=0x7f0890004ab0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 -- 172.21.15.81:0/3904182853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08a00cf3a0 msgr2=0x7f08a01dfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:51.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08a00cf3a0 0x7f08a01dfdc0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f089400d680 tx=0x7f089400db40 comp rx=0 tx=0).stop 2024-06-29T14:37:51.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 -- 172.21.15.81:0/3904182853 shutdown_connections 2024-06-29T14:37:51.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f088c0763f0 0x7f088c078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08a00d1e80 0x7f08a00cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08a00cf3a0 0x7f08a01dfdc0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 --2- 172.21.15.81:0/3904182853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08a00cea50 0x7f08a009eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 -- 172.21.15.81:0/3904182853 >> 172.21.15.81:0/3904182853 conn(0x7f08a00b5700 msgr2=0x7f08a01e6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:51.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 -- 172.21.15.81:0/3904182853 shutdown_connections 2024-06-29T14:37:51.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.080+0000 7f08a5265640 1 -- 172.21.15.81:0/3904182853 wait complete. 2024-06-29T14:37:51.092 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:37:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.215+0000 7f82d2d9e640 1 -- 172.21.15.81:0/2368429811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82cc0d0b90 msgr2=0x7f82cc0d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.215+0000 7f82d2d9e640 1 --2- 172.21.15.81:0/2368429811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82cc0d0b90 0x7f82cc0d1a80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f82cc08e330 tx=0x7f82c002ab40 comp rx=0 tx=0).stop 2024-06-29T14:37:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.215+0000 7f82d2d9e640 1 -- 172.21.15.81:0/2368429811 shutdown_connections 2024-06-29T14:37:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.215+0000 7f82d2d9e640 1 --2- 172.21.15.81:0/2368429811 >> v2:172.21.15.167:3300/0 conn(0x7f82cc0b5250 0x7f82cc0d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.215+0000 7f82d2d9e640 1 --2- 172.21.15.81:0/2368429811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82cc0d0b90 0x7f82cc0d1a80 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.215+0000 7f82d2d9e640 1 -- 172.21.15.81:0/2368429811 >> 172.21.15.81:0/2368429811 conn(0x7f82cc09cf30 msgr2=0x7f82cc0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.215+0000 7f82d2d9e640 1 -- 172.21.15.81:0/2368429811 shutdown_connections 2024-06-29T14:37:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.215+0000 7f82d2d9e640 1 -- 172.21.15.81:0/2368429811 wait complete. 2024-06-29T14:37:51.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.217+0000 7f82d2d9e640 1 Processor -- start 2024-06-29T14:37:51.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.217+0000 7f82d2d9e640 1 -- start start 2024-06-29T14:37:51.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.218+0000 7f82d2d9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82cc0b5250 0x7f82cc0a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:51.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.218+0000 7f82d2d9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82cc0b9040 0x7f82cc1f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:51.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.218+0000 7f82d2d9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82cc0b3880 0x7f82cc0b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:51.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.218+0000 7f82d2d9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82cc0d2500 con 0x7f82cc0b3880 2024-06-29T14:37:51.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.218+0000 7f82d2d9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f82cc0b84b0 con 0x7f82cc0b9040 2024-06-29T14:37:51.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.218+0000 7f82d2d9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f82cc0d2680 con 0x7f82cc0b5250 2024-06-29T14:37:51.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.218+0000 7f82d133b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82cc0b3880 0x7f82cc0b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.218+0000 7f82d133b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82cc0b3880 0x7f82cc0b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47678/0 (socket says 172.21.15.81:47678) 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.218+0000 7f82d133b640 1 -- 172.21.15.81:0/317267947 learned_addr learned my addr 172.21.15.81:0/317267947 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82cbfff640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82cc0b9040 0x7f82cc1f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82d133b640 1 -- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82cc0b5250 msgr2=0x7f82cc0a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82d133b640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82cc0b5250 0x7f82cc0a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82d133b640 1 -- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82cc0b9040 msgr2=0x7f82cc1f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82d133b640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82cc0b9040 0x7f82cc1f5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82d133b640 1 -- 172.21.15.81:0/317267947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82cc0b41d0 con 0x7f82cc0b3880 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82d133b640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82cc0b3880 0x7f82cc0b3c90 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f82c400d5f0 tx=0x7f82c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82c97fa640 1 -- 172.21.15.81:0/317267947 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c4017070 con 0x7f82cc0b3880 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82c97fa640 1 -- 172.21.15.81:0/317267947 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82c4005330 con 0x7f82cc0b3880 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82d2d9e640 1 -- 172.21.15.81:0/317267947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82cc0c7970 con 0x7f82cc0b3880 2024-06-29T14:37:51.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82c97fa640 1 -- 172.21.15.81:0/317267947 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c40047a0 con 0x7f82cc0b3880 2024-06-29T14:37:51.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.219+0000 7f82d2d9e640 1 -- 172.21.15.81:0/317267947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82cc151300 con 0x7f82cc0b3880 2024-06-29T14:37:51.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.220+0000 7f82d2d9e640 1 -- 172.21.15.81:0/317267947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82cc0d2c10 con 0x7f82cc0b3880 2024-06-29T14:37:51.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.221+0000 7f82c97fa640 1 -- 172.21.15.81:0/317267947 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f82c4004940 con 0x7f82cc0b3880 2024-06-29T14:37:51.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.221+0000 7f82c97fa640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f82b8076420 0x7f82b8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:51.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.222+0000 7f82c97fa640 1 -- 172.21.15.81:0/317267947 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5299+0+0 (secure 0 0 0) 0x7f82c4097780 con 0x7f82cc0b3880 2024-06-29T14:37:51.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.222+0000 7f82d0b3a640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f82b8076420 0x7f82b8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:51.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.222+0000 7f82d0b3a640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f82b8076420 0x7f82b8078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f82bc000f60 tx=0x7f82bc0079b0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:51.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.226+0000 7f82c97fa640 1 -- 172.21.15.81:0/317267947 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f82c4060fc0 con 0x7f82cc0b3880 2024-06-29T14:37:51.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:51.405+0000 7f82d2d9e640 1 -- 172.21.15.81:0/317267947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f82cc06dfb0 con 0x7f82cc0b3880 2024-06-29T14:37:52.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.086+0000 7f82c97fa640 1 -- 172.21.15.81:0/317267947 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v89) ==== 170+0+0 (secure 0 0 0) 0x7f82c4069040 con 0x7f82cc0b3880 2024-06-29T14:37:52.084 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T14:37:52.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.089+0000 7f82d2d9e640 1 -- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f82b8076420 msgr2=0x7f82b8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:52.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.089+0000 7f82d2d9e640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f82b8076420 0x7f82b8078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f82bc000f60 tx=0x7f82bc0079b0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.089+0000 7f82d2d9e640 1 -- 172.21.15.81:0/317267947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82cc0b3880 msgr2=0x7f82cc0b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:52.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.089+0000 7f82d2d9e640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82cc0b3880 0x7f82cc0b3c90 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f82c400d5f0 tx=0x7f82c400dab0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.090+0000 7f82d2d9e640 1 -- 172.21.15.81:0/317267947 shutdown_connections 2024-06-29T14:37:52.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.090+0000 7f82d2d9e640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f82b8076420 0x7f82b8078930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.090+0000 7f82d2d9e640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82cc0b3880 0x7f82cc0b3c90 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.090+0000 7f82d2d9e640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82cc0b9040 0x7f82cc1f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.090+0000 7f82d2d9e640 1 --2- 172.21.15.81:0/317267947 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82cc0b5250 0x7f82cc0a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.090+0000 7f82d2d9e640 1 -- 172.21.15.81:0/317267947 >> 172.21.15.81:0/317267947 conn(0x7f82cc09cf30 msgr2=0x7f82cc0b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:52.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.090+0000 7f82d2d9e640 1 -- 172.21.15.81:0/317267947 shutdown_connections 2024-06-29T14:37:52.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.090+0000 7f82d2d9e640 1 -- 172.21.15.81:0/317267947 wait complete. 2024-06-29T14:37:52.109 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:37:52.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.232+0000 7fa6a7e26640 1 -- 172.21.15.81:0/3482987229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6a00cf4d0 msgr2=0x7fa6a0073030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:52.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.232+0000 7fa6a7e26640 1 --2- 172.21.15.81:0/3482987229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6a00cf4d0 0x7fa6a0073030 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fa690005fd0 tx=0x7fa69002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.232+0000 7fa6a7e26640 1 -- 172.21.15.81:0/3482987229 shutdown_connections 2024-06-29T14:37:52.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.232+0000 7fa6a7e26640 1 --2- 172.21.15.81:0/3482987229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6a00cf4d0 0x7fa6a0073030 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.232+0000 7fa6a7e26640 1 --2- 172.21.15.81:0/3482987229 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6a01412f0 0x7fa6a00cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.232+0000 7fa6a7e26640 1 -- 172.21.15.81:0/3482987229 >> 172.21.15.81:0/3482987229 conn(0x7fa6a006ccc0 msgr2=0x7fa6a00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:52.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.232+0000 7fa6a7e26640 1 -- 172.21.15.81:0/3482987229 shutdown_connections 2024-06-29T14:37:52.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.232+0000 7fa6a7e26640 1 -- 172.21.15.81:0/3482987229 wait complete. 2024-06-29T14:37:52.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.234+0000 7fa6a7e26640 1 Processor -- start 2024-06-29T14:37:52.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.234+0000 7fa6a7e26640 1 -- start start 2024-06-29T14:37:52.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a7e26640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6a0073570 0x7fa6a0140600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:52.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a7e26640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6a00cf4d0 0x7fa6a00aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:52.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a7e26640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6a01412f0 0x7fa6a0071f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:52.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a7e26640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a00b3210 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a7e26640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a015f280 con 0x7fa6a01412f0 2024-06-29T14:37:52.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a7e26640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa6a00b3390 con 0x7fa6a0073570 2024-06-29T14:37:52.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a53c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6a00cf4d0 0x7fa6a00aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:52.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a53c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6a00cf4d0 0x7fa6a00aa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47692/0 (socket says 172.21.15.81:47692) 2024-06-29T14:37:52.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a53c1640 1 -- 172.21.15.81:0/4176185345 learned_addr learned my addr 172.21.15.81:0/4176185345 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:52.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a5bc2640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6a0073570 0x7fa6a0140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a63c3640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6a01412f0 0x7fa6a0071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a53c1640 1 -- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6a0073570 msgr2=0x7fa6a0140600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a53c1640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6a0073570 0x7fa6a0140600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a53c1640 1 -- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6a01412f0 msgr2=0x7fa6a0071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a53c1640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6a01412f0 0x7fa6a0071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.235+0000 7fa6a53c1640 1 -- 172.21.15.81:0/4176185345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a00ab1e0 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.236+0000 7fa6a53c1640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6a00cf4d0 0x7fa6a00aa8c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa690000c00 tx=0x7fa6900038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.236+0000 7fa6967fc640 1 -- 172.21.15.81:0/4176185345 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69003e030 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.236+0000 7fa6967fc640 1 -- 172.21.15.81:0/4176185345 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa690003ca0 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.236+0000 7fa6967fc640 1 -- 172.21.15.81:0/4176185345 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690035040 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.236+0000 7fa6a7e26640 1 -- 172.21.15.81:0/4176185345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a00aa690 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.236+0000 7fa6a7e26640 1 -- 172.21.15.81:0/4176185345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a00a95f0 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.237+0000 7fa6a7e26640 1 -- 172.21.15.81:0/4176185345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6a00b3880 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.238+0000 7fa6967fc640 1 -- 172.21.15.81:0/4176185345 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fa690041030 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.238+0000 7fa6967fc640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa6840763f0 0x7fa684078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:52.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.238+0000 7fa6967fc640 1 -- 172.21.15.81:0/4176185345 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5681+0+0 (secure 0 0 0) 0x7fa6900b7ef0 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.238+0000 7fa6a5bc2640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa6840763f0 0x7fa684078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:52.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.239+0000 7fa6a5bc2640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa6840763f0 0x7fa684078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa69c004630 tx=0x7fa69c00b2e0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:52.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.243+0000 7fa6967fc640 1 -- 172.21.15.81:0/4176185345 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa690080f70 con 0x7fa6a00cf4d0 2024-06-29T14:37:52.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:52.461+0000 7fa6a7e26640 1 -- 172.21.15.81:0/4176185345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa6a0159eb0 con 0x7fa6a00cf4d0 2024-06-29T14:37:53.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.105+0000 7fa6967fc640 1 -- 172.21.15.81:0/4176185345 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v39) ==== 174+0+0 (secure 0 0 0) 0x7fa690085270 con 0x7fa6a00cf4d0 2024-06-29T14:37:53.102 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 10 and data pool 11 2024-06-29T14:37:53.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.112+0000 7fa6a7e26640 1 -- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa6840763f0 msgr2=0x7fa684078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.112+0000 7fa6a7e26640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa6840763f0 0x7fa684078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa69c004630 tx=0x7fa69c00b2e0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.112+0000 7fa6a7e26640 1 -- 172.21.15.81:0/4176185345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6a00cf4d0 msgr2=0x7fa6a00aa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.112+0000 7fa6a7e26640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6a00cf4d0 0x7fa6a00aa8c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fa690000c00 tx=0x7fa6900038a0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.113+0000 7fa6a7e26640 1 -- 172.21.15.81:0/4176185345 shutdown_connections 2024-06-29T14:37:53.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.113+0000 7fa6a7e26640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fa6840763f0 0x7fa684078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.113+0000 7fa6a7e26640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa6a01412f0 0x7fa6a0071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.113+0000 7fa6a7e26640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6a00cf4d0 0x7fa6a00aa8c0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.113+0000 7fa6a7e26640 1 --2- 172.21.15.81:0/4176185345 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6a0073570 0x7fa6a0140600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.113+0000 7fa6a7e26640 1 -- 172.21.15.81:0/4176185345 >> 172.21.15.81:0/4176185345 conn(0x7fa6a006ccc0 msgr2=0x7fa6a015a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:53.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.113+0000 7fa6a7e26640 1 -- 172.21.15.81:0/4176185345 shutdown_connections 2024-06-29T14:37:53.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.113+0000 7fa6a7e26640 1 -- 172.21.15.81:0/4176185345 wait complete. 2024-06-29T14:37:53.130 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:53.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.250+0000 7fdc0edb9640 1 -- 172.21.15.81:0/496505984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc080cf4d0 msgr2=0x7fdc08073030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.250+0000 7fdc0edb9640 1 --2- 172.21.15.81:0/496505984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc080cf4d0 0x7fdc08073030 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fdc08070e50 tx=0x7fdbf402a5f0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.250+0000 7fdc0edb9640 1 -- 172.21.15.81:0/496505984 shutdown_connections 2024-06-29T14:37:53.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.250+0000 7fdc0edb9640 1 --2- 172.21.15.81:0/496505984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc080cf4d0 0x7fdc08073030 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.250+0000 7fdc0edb9640 1 --2- 172.21.15.81:0/496505984 >> v2:172.21.15.167:3300/0 conn(0x7fdc081412f0 0x7fdc080cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.250+0000 7fdc0edb9640 1 -- 172.21.15.81:0/496505984 >> 172.21.15.81:0/496505984 conn(0x7fdc0806ccc0 msgr2=0x7fdc080b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:53.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.250+0000 7fdc0edb9640 1 -- 172.21.15.81:0/496505984 shutdown_connections 2024-06-29T14:37:53.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.250+0000 7fdc0edb9640 1 -- 172.21.15.81:0/496505984 wait complete. 2024-06-29T14:37:53.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.252+0000 7fdc0edb9640 1 Processor -- start 2024-06-29T14:37:53.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.252+0000 7fdc0edb9640 1 -- start start 2024-06-29T14:37:53.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.253+0000 7fdc0edb9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc08073570 0x7fdc080ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.253+0000 7fdc0edb9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc080cf4d0 0x7fdc080c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.253+0000 7fdc0edb9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc081412f0 0x7fdc08158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0edb9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc080b3210 con 0x7fdc081412f0 2024-06-29T14:37:53.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0edb9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0815f280 con 0x7fdc080cf4d0 2024-06-29T14:37:53.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0edb9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdc080b3390 con 0x7fdc08073570 2024-06-29T14:37:53.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0d356640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc081412f0 0x7fdc08158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0d356640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc081412f0 0x7fdc08158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47700/0 (socket says 172.21.15.81:47700) 2024-06-29T14:37:53.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0d356640 1 -- 172.21.15.81:0/1416019712 learned_addr learned my addr 172.21.15.81:0/1416019712 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0d356640 1 -- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc08073570 msgr2=0x7fdc080ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0d356640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc08073570 0x7fdc080ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0d356640 1 -- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc080cf4d0 msgr2=0x7fdc080c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0d356640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc080cf4d0 0x7fdc080c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0d356640 1 -- 172.21.15.81:0/1416019712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc08159f30 con 0x7fdc081412f0 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0d356640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc081412f0 0x7fdc08158480 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000e9a0 tx=0x7fdc0000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc057fa640 1 -- 172.21.15.81:0/1416019712 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0000cd20 con 0x7fdc081412f0 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc057fa640 1 -- 172.21.15.81:0/1416019712 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc00003970 con 0x7fdc081412f0 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0edb9640 1 -- 172.21.15.81:0/1416019712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc080ade20 con 0x7fdc081412f0 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc057fa640 1 -- 172.21.15.81:0/1416019712 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc00003d80 con 0x7fdc081412f0 2024-06-29T14:37:53.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.254+0000 7fdc0edb9640 1 -- 172.21.15.81:0/1416019712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc080abf80 con 0x7fdc081412f0 2024-06-29T14:37:53.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.255+0000 7fdc0edb9640 1 -- 172.21.15.81:0/1416019712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc080b38b0 con 0x7fdc081412f0 2024-06-29T14:37:53.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.256+0000 7fdc057fa640 1 -- 172.21.15.81:0/1416019712 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fdc00011470 con 0x7fdc081412f0 2024-06-29T14:37:53.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.256+0000 7fdc057fa640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fdbec0763f0 0x7fdbec078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.256+0000 7fdc057fa640 1 -- 172.21.15.81:0/1416019712 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5781+0+0 (secure 0 0 0) 0x7fdc0009a330 con 0x7fdc081412f0 2024-06-29T14:37:53.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.257+0000 7fdc0cb55640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fdbec0763f0 0x7fdbec078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.257+0000 7fdc0cb55640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fdbec0763f0 0x7fdbec078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00a860 tx=0x7fdbfc005b50 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:53.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.261+0000 7fdc057fa640 1 -- 172.21.15.81:0/1416019712 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc00016020 con 0x7fdc081412f0 2024-06-29T14:37:53.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.442+0000 7fdc0edb9640 1 -- 172.21.15.81:0/1416019712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc0815a1c0 con 0x7fdc081412f0 2024-06-29T14:37:53.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.443+0000 7fdc057fa640 1 -- 172.21.15.81:0/1416019712 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+16783 (secure 0 0 0) 0x7fdc08073980 con 0x7fdc081412f0 2024-06-29T14:37:53.441 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:53.441 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":90,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:52.462659+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:50.753390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.410843+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 -- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fdbec0763f0 msgr2=0x7fdbec078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fdbec0763f0 0x7fdbec078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00a860 tx=0x7fdbfc005b50 comp rx=0 tx=0).stop 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 -- 172.21.15.81:0/1416019712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc081412f0 msgr2=0x7fdc08158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc081412f0 0x7fdc08158480 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fdc0000e9a0 tx=0x7fdc0000ee60 comp rx=0 tx=0).stop 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 -- 172.21.15.81:0/1416019712 shutdown_connections 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fdbec0763f0 0x7fdbec078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc081412f0 0x7fdc08158480 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc080cf4d0 0x7fdc080c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 --2- 172.21.15.81:0/1416019712 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc08073570 0x7fdc080ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 -- 172.21.15.81:0/1416019712 >> 172.21.15.81:0/1416019712 conn(0x7fdc0806ccc0 msgr2=0x7fdc08140540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 -- 172.21.15.81:0/1416019712 shutdown_connections 2024-06-29T14:37:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.445+0000 7fdc0edb9640 1 -- 172.21.15.81:0/1416019712 wait complete. 2024-06-29T14:37:53.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:53.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.571+0000 7fb3362e4640 1 -- 172.21.15.81:0/1921702028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb318004800 msgr2=0x7fb318004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.571+0000 7fb3362e4640 1 --2- 172.21.15.81:0/1921702028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb318004800 0x7fb318004c10 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb324006fc0 tx=0x7fb318005830 comp rx=0 tx=0).stop 2024-06-29T14:37:53.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.571+0000 7fb3362e4640 1 -- 172.21.15.81:0/1921702028 shutdown_connections 2024-06-29T14:37:53.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.571+0000 7fb3362e4640 1 --2- 172.21.15.81:0/1921702028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb318004800 0x7fb318004c10 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.571+0000 7fb3362e4640 1 -- 172.21.15.81:0/1921702028 >> 172.21.15.81:0/1921702028 conn(0x7fb33006ccc0 msgr2=0x7fb3300b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:53.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.571+0000 7fb3362e4640 1 -- 172.21.15.81:0/1921702028 shutdown_connections 2024-06-29T14:37:53.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.572+0000 7fb3362e4640 1 -- 172.21.15.81:0/1921702028 wait complete. 2024-06-29T14:37:53.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.573+0000 7fb3362e4640 1 Processor -- start 2024-06-29T14:37:53.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.573+0000 7fb3362e4640 1 -- start start 2024-06-29T14:37:53.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb3362e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb318004800 0x7fb33014ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb3362e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb330136440 0x7fb330143cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb3362e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb330136960 0x7fb33013f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb3362e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb3300731b0 con 0x7fb330136960 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb3362e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb330073030 con 0x7fb330136440 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb3362e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb330073330 con 0x7fb318004800 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb334881640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb330136960 0x7fb33013f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb334881640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb330136960 0x7fb33013f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47724/0 (socket says 172.21.15.81:47724) 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb334881640 1 -- 172.21.15.81:0/1494023567 learned_addr learned my addr 172.21.15.81:0/1494023567 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb32ffff640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb318004800 0x7fb33014ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb32f7fe640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb330136440 0x7fb330143cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb334881640 1 -- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb318004800 msgr2=0x7fb33014ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb334881640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb318004800 0x7fb33014ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb334881640 1 -- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb330136440 msgr2=0x7fb330143cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb334881640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb330136440 0x7fb330143cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb334881640 1 -- 172.21.15.81:0/1494023567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb330159d60 con 0x7fb330136960 2024-06-29T14:37:53.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb334881640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb330136960 0x7fb33013f630 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb3300b4ee0 tx=0x7fb3280019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:53.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.575+0000 7fb32cff9640 1 -- 172.21.15.81:0/1494023567 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328011660 con 0x7fb330136960 2024-06-29T14:37:53.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.576+0000 7fb32cff9640 1 -- 172.21.15.81:0/1494023567 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb328017070 con 0x7fb330136960 2024-06-29T14:37:53.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.576+0000 7fb3362e4640 1 -- 172.21.15.81:0/1494023567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb330143250 con 0x7fb330136960 2024-06-29T14:37:53.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.576+0000 7fb32cff9640 1 -- 172.21.15.81:0/1494023567 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328011bb0 con 0x7fb330136960 2024-06-29T14:37:53.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.576+0000 7fb3362e4640 1 -- 172.21.15.81:0/1494023567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb330137820 con 0x7fb330136960 2024-06-29T14:37:53.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.577+0000 7fb3362e4640 1 -- 172.21.15.81:0/1494023567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb330073890 con 0x7fb330136960 2024-06-29T14:37:53.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.577+0000 7fb32cff9640 1 -- 172.21.15.81:0/1494023567 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fb32801c460 con 0x7fb330136960 2024-06-29T14:37:53.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.578+0000 7fb32cff9640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb31c076420 0x7fb31c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.578+0000 7fb32cff9640 1 -- 172.21.15.81:0/1494023567 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5781+0+0 (secure 0 0 0) 0x7fb328098060 con 0x7fb330136960 2024-06-29T14:37:53.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.578+0000 7fb32ffff640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb31c076420 0x7fb31c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.579+0000 7fb32ffff640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb31c076420 0x7fb31c078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb320009cd0 tx=0x7fb320005f90 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:53.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.582+0000 7fb32cff9640 1 -- 172.21.15.81:0/1494023567 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb328061750 con 0x7fb330136960 2024-06-29T14:37:53.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.763+0000 7fb3362e4640 1 -- 172.21.15.81:0/1494023567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb330159f70 con 0x7fb330136960 2024-06-29T14:37:53.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.764+0000 7fb32cff9640 1 -- 172.21.15.81:0/1494023567 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+16783 (secure 0 0 0) 0x7fb328061050 con 0x7fb330136960 2024-06-29T14:37:53.762 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:53.762 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":90,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:52.462659+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:50.753390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.410843+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:53.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.766+0000 7fb3362e4640 1 -- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb31c076420 msgr2=0x7fb31c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.766+0000 7fb3362e4640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb31c076420 0x7fb31c078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb320009cd0 tx=0x7fb320005f90 comp rx=0 tx=0).stop 2024-06-29T14:37:53.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.766+0000 7fb3362e4640 1 -- 172.21.15.81:0/1494023567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb330136960 msgr2=0x7fb33013f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.766+0000 7fb3362e4640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb330136960 0x7fb33013f630 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fb3300b4ee0 tx=0x7fb3280019b0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.767+0000 7fb3362e4640 1 -- 172.21.15.81:0/1494023567 shutdown_connections 2024-06-29T14:37:53.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.767+0000 7fb3362e4640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb31c076420 0x7fb31c078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.767+0000 7fb3362e4640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb330136960 0x7fb33013f630 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.767+0000 7fb3362e4640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb330136440 0x7fb330143cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.767+0000 7fb3362e4640 1 --2- 172.21.15.81:0/1494023567 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb318004800 0x7fb33014ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.767+0000 7fb3362e4640 1 -- 172.21.15.81:0/1494023567 >> 172.21.15.81:0/1494023567 conn(0x7fb33006ccc0 msgr2=0x7fb3300b1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:53.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.767+0000 7fb3362e4640 1 -- 172.21.15.81:0/1494023567 shutdown_connections 2024-06-29T14:37:53.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.767+0000 7fb3362e4640 1 -- 172.21.15.81:0/1494023567 wait complete. 2024-06-29T14:37:53.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:37:53.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.895+0000 7fb6b0773640 1 -- 172.21.15.81:0/3974493081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb694004800 msgr2=0x7fb694004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.895+0000 7fb6b0773640 1 --2- 172.21.15.81:0/3974493081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb694004800 0x7fb694004c10 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80cac90 tx=0x7fb694005830 comp rx=0 tx=0).stop 2024-06-29T14:37:53.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.895+0000 7fb6aed10640 1 -- 172.21.15.81:0/3974493081 reap_dead start 2024-06-29T14:37:53.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.895+0000 7fb6b0773640 1 -- 172.21.15.81:0/3974493081 shutdown_connections 2024-06-29T14:37:53.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.895+0000 7fb6b0773640 1 -- 172.21.15.81:0/3974493081 >> 172.21.15.81:0/3974493081 conn(0x7fb6a809cf30 msgr2=0x7fb6a80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:53.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.895+0000 7fb6b0773640 1 -- 172.21.15.81:0/3974493081 shutdown_connections 2024-06-29T14:37:53.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.895+0000 7fb6b0773640 1 -- 172.21.15.81:0/3974493081 wait complete. 2024-06-29T14:37:53.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.897+0000 7fb6b0773640 1 Processor -- start 2024-06-29T14:37:53.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.897+0000 7fb6b0773640 1 -- start start 2024-06-29T14:37:53.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6b0773640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb694004800 0x7fb6a81f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6b0773640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6a8147770 0x7fb6a8147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6b0773640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6a80a0de0 0x7fb6a80a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6b0773640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a80b8420 con 0x7fb694004800 2024-06-29T14:37:53.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6b0773640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a80b82a0 con 0x7fb6a8147770 2024-06-29T14:37:53.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6b0773640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a80b85a0 con 0x7fb6a80a0de0 2024-06-29T14:37:53.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6ae50f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb694004800 0x7fb6a81f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6ae50f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb694004800 0x7fb6a81f4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47740/0 (socket says 172.21.15.81:47740) 2024-06-29T14:37:53.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6ae50f640 1 -- 172.21.15.81:0/1496485721 learned_addr learned my addr 172.21.15.81:0/1496485721 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:53.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6add0e640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6a8147770 0x7fb6a8147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6aed10640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6a80a0de0 0x7fb6a80a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6ae50f640 1 -- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6a80a0de0 msgr2=0x7fb6a80a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6ae50f640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6a80a0de0 0x7fb6a80a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6ae50f640 1 -- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6a8147770 msgr2=0x7fb6a8147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6ae50f640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6a8147770 0x7fb6a8147b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6ae50f640 1 -- 172.21.15.81:0/1496485721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a81480c0 con 0x7fb694004800 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.898+0000 7fb6ae50f640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb694004800 0x7fb6a81f4260 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4000fb0 tx=0x7fb6a40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.899+0000 7fb69effd640 1 -- 172.21.15.81:0/1496485721 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4015070 con 0x7fb694004800 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.899+0000 7fb69effd640 1 -- 172.21.15.81:0/1496485721 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6a4011920 con 0x7fb694004800 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.899+0000 7fb69effd640 1 -- 172.21.15.81:0/1496485721 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4011d70 con 0x7fb694004800 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.899+0000 7fb6b0773640 1 -- 172.21.15.81:0/1496485721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a8158ed0 con 0x7fb694004800 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.899+0000 7fb6b0773640 1 -- 172.21.15.81:0/1496485721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a8159490 con 0x7fb694004800 2024-06-29T14:37:53.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.900+0000 7fb6b0773640 1 -- 172.21.15.81:0/1496485721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a80d0480 con 0x7fb694004800 2024-06-29T14:37:53.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.901+0000 7fb69effd640 1 -- 172.21.15.81:0/1496485721 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fb6a4018070 con 0x7fb694004800 2024-06-29T14:37:53.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.901+0000 7fb69effd640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb68c076420 0x7fb68c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:53.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.901+0000 7fb69effd640 1 -- 172.21.15.81:0/1496485721 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5781+0+0 (secure 0 0 0) 0x7fb6a40986f0 con 0x7fb694004800 2024-06-29T14:37:53.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.901+0000 7fb6add0e640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb68c076420 0x7fb68c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:53.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.902+0000 7fb6add0e640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb68c076420 0x7fb68c078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb698036000 tx=0x7fb6980046c0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:53.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:53.905+0000 7fb69effd640 1 -- 172.21.15.81:0/1496485721 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb6a81480c0 con 0x7fb694004800 2024-06-29T14:37:54.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:54.128+0000 7fb6b0773640 1 -- 172.21.15.81:0/1496485721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb6a80d2b20 con 0x7fb694004800 2024-06-29T14:37:55.077 INFO:tasks.ceph.mds.b.smithi167.stderr:2024-06-29T14:37:55.079+0000 7fbc7353e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:37:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.110+0000 7fb69effd640 1 -- 172.21.15.81:0/1496485721 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) ==== 118+0+0 (secure 0 0 0) 0x7fb6a4061de0 con 0x7fb694004800 2024-06-29T14:37:55.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 -- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb68c076420 msgr2=0x7fb68c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:55.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb68c076420 0x7fb68c078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb698036000 tx=0x7fb6980046c0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 -- 172.21.15.81:0/1496485721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb694004800 msgr2=0x7fb6a81f4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:55.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb694004800 0x7fb6a81f4260 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4000fb0 tx=0x7fb6a40039c0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 -- 172.21.15.81:0/1496485721 shutdown_connections 2024-06-29T14:37:55.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fb68c076420 0x7fb68c078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6a80a0de0 0x7fb6a80a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6a8147770 0x7fb6a8147b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 --2- 172.21.15.81:0/1496485721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb694004800 0x7fb6a81f4260 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 -- 172.21.15.81:0/1496485721 >> 172.21.15.81:0/1496485721 conn(0x7fb6a809cf30 msgr2=0x7fb6a8068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:55.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 -- 172.21.15.81:0/1496485721 shutdown_connections 2024-06-29T14:37:55.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.113+0000 7fb6b0773640 1 -- 172.21.15.81:0/1496485721 wait complete. 2024-06-29T14:37:55.124 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:37:55.124 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:37:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.248+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/1380621390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f64004800 msgr2=0x7f1f64004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.248+0000 7f1f7d9d2640 1 --2- 172.21.15.81:0/1380621390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f64004800 0x7f1f64004c10 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f1f780c8bb0 tx=0x7f1f64005830 comp rx=0 tx=0).stop 2024-06-29T14:37:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.248+0000 7f1f777fe640 1 -- 172.21.15.81:0/1380621390 reap_dead start 2024-06-29T14:37:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.248+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/1380621390 shutdown_connections 2024-06-29T14:37:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.248+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/1380621390 >> 172.21.15.81:0/1380621390 conn(0x7f1f780b3680 msgr2=0x7f1f780b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.248+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/1380621390 shutdown_connections 2024-06-29T14:37:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.248+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/1380621390 wait complete. 2024-06-29T14:37:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.250+0000 7f1f7d9d2640 1 Processor -- start 2024-06-29T14:37:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.250+0000 7f1f7d9d2640 1 -- start start 2024-06-29T14:37:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f7d9d2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f64004800 0x7f1f780b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f7d9d2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f78157b10 0x7f1f780b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f7d9d2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f780b21c0 0x7f1f7815a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f7d9d2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1f78164d90 con 0x7f1f780b21c0 2024-06-29T14:37:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f7d9d2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1f78164c10 con 0x7f1f64004800 2024-06-29T14:37:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f7d9d2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1f78164f10 con 0x7f1f78157b10 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f780b21c0 0x7f1f7815a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f780b21c0 0x7f1f7815a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47754/0 (socket says 172.21.15.81:47754) 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f777fe640 1 -- 172.21.15.81:0/3622102027 learned_addr learned my addr 172.21.15.81:0/3622102027 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f76ffd640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f64004800 0x7f1f780b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f767fc640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f78157b10 0x7f1f780b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.251+0000 7f1f777fe640 1 -- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f78157b10 msgr2=0x7f1f780b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.252+0000 7f1f777fe640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f78157b10 0x7f1f780b1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.252+0000 7f1f777fe640 1 -- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f64004800 msgr2=0x7f1f780b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.252+0000 7f1f777fe640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f64004800 0x7f1f780b1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.252+0000 7f1f777fe640 1 -- 172.21.15.81:0/3622102027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f780b29f0 con 0x7f1f780b21c0 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.252+0000 7f1f777fe640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f780b21c0 0x7f1f7815a370 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1f780b3650 tx=0x7f1f68011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.252+0000 7f1f6ffff640 1 -- 172.21.15.81:0/3622102027 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f68004230 con 0x7f1f780b21c0 2024-06-29T14:37:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.252+0000 7f1f6ffff640 1 -- 172.21.15.81:0/3622102027 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f68016070 con 0x7f1f780b21c0 2024-06-29T14:37:55.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.252+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/3622102027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f78156d50 con 0x7f1f780b21c0 2024-06-29T14:37:55.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.252+0000 7f1f6ffff640 1 -- 172.21.15.81:0/3622102027 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800fac0 con 0x7f1f780b21c0 2024-06-29T14:37:55.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.252+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/3622102027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f78157310 con 0x7f1f780b21c0 2024-06-29T14:37:55.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.253+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/3622102027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f780cc7b0 con 0x7f1f780b21c0 2024-06-29T14:37:55.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.254+0000 7f1f6ffff640 1 -- 172.21.15.81:0/3622102027 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f1f6800fc60 con 0x7f1f780b21c0 2024-06-29T14:37:55.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.254+0000 7f1f6ffff640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1f58076420 0x7f1f58078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:55.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.254+0000 7f1f6ffff640 1 -- 172.21.15.81:0/3622102027 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f1f68098200 con 0x7f1f780b21c0 2024-06-29T14:37:55.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.254+0000 7f1f76ffd640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1f58076420 0x7f1f58078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:55.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.255+0000 7f1f76ffd640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1f58076420 0x7f1f58078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1f70000f80 tx=0x7f1f70006ef0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:55.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.259+0000 7f1f6ffff640 1 -- 172.21.15.81:0/3622102027 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1f68061320 con 0x7f1f780b21c0 2024-06-29T14:37:55.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:55.477+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/3622102027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f1f780d07e0 con 0x7f1f780b21c0 2024-06-29T14:37:56.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.111+0000 7f1f6ffff640 1 -- 172.21.15.81:0/3622102027 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v42) ==== 122+0+0 (secure 0 0 0) 0x7f1f68098020 con 0x7f1f780b21c0 2024-06-29T14:37:56.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.113+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1f58076420 msgr2=0x7f1f58078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.113+0000 7f1f7d9d2640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1f58076420 0x7f1f58078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1f70000f80 tx=0x7f1f70006ef0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.114+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/3622102027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f780b21c0 msgr2=0x7f1f7815a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.114+0000 7f1f7d9d2640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f780b21c0 0x7f1f7815a370 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f1f780b3650 tx=0x7f1f68011b10 comp rx=0 tx=0).stop 2024-06-29T14:37:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.114+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/3622102027 shutdown_connections 2024-06-29T14:37:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.114+0000 7f1f7d9d2640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1f58076420 0x7f1f58078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.114+0000 7f1f7d9d2640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f780b21c0 0x7f1f7815a370 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.114+0000 7f1f7d9d2640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1f78157b10 0x7f1f780b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.114+0000 7f1f7d9d2640 1 --2- 172.21.15.81:0/3622102027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1f64004800 0x7f1f780b1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.114+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/3622102027 >> 172.21.15.81:0/3622102027 conn(0x7f1f780b3680 msgr2=0x7f1f7815b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.114+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/3622102027 shutdown_connections 2024-06-29T14:37:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.114+0000 7f1f7d9d2640 1 -- 172.21.15.81:0/3622102027 wait complete. 2024-06-29T14:37:56.124 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:37:56.124 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:56.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.242+0000 7f0eac303640 1 -- 172.21.15.81:0/2283672134 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ea406f850 msgr2=0x7f0ea40ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.242+0000 7f0eac303640 1 --2- 172.21.15.81:0/2283672134 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ea406f850 0x7f0ea40ae300 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0e98005fd0 tx=0x7f0e9802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.243+0000 7f0eac303640 1 -- 172.21.15.81:0/2283672134 shutdown_connections 2024-06-29T14:37:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.243+0000 7f0eac303640 1 --2- 172.21.15.81:0/2283672134 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ea40ae840 0x7f0ea40aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.243+0000 7f0eac303640 1 --2- 172.21.15.81:0/2283672134 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ea406f850 0x7f0ea40ae300 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.243+0000 7f0eac303640 1 -- 172.21.15.81:0/2283672134 >> 172.21.15.81:0/2283672134 conn(0x7f0ea4072950 msgr2=0x7f0ea40b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.243+0000 7f0eac303640 1 -- 172.21.15.81:0/2283672134 shutdown_connections 2024-06-29T14:37:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.243+0000 7f0eac303640 1 -- 172.21.15.81:0/2283672134 wait complete. 2024-06-29T14:37:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.244+0000 7f0eac303640 1 Processor -- start 2024-06-29T14:37:56.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.244+0000 7f0eac303640 1 -- start start 2024-06-29T14:37:56.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0eac303640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ea406f850 0x7f0ea41522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0eac303640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ea4074520 0x7f0ea4156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0eac303640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ea40ae840 0x7f0ea415d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0eac303640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea415b190 con 0x7f0ea4074520 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0eac303640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea415b010 con 0x7f0ea406f850 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0eac303640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0ea40af190 con 0x7f0ea40ae840 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0ea989e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ea4074520 0x7f0ea4156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0ea989e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ea4074520 0x7f0ea4156d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47766/0 (socket says 172.21.15.81:47766) 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0ea989e640 1 -- 172.21.15.81:0/568673563 learned_addr learned my addr 172.21.15.81:0/568673563 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0eaa8a0640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ea40ae840 0x7f0ea415d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0eaa09f640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ea406f850 0x7f0ea41522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0ea989e640 1 -- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ea40ae840 msgr2=0x7f0ea415d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0ea989e640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ea40ae840 0x7f0ea415d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0ea989e640 1 -- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ea406f850 msgr2=0x7f0ea41522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0ea989e640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ea406f850 0x7f0ea41522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0ea989e640 1 -- 172.21.15.81:0/568673563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea4156940 con 0x7f0ea4074520 2024-06-29T14:37:56.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.246+0000 7f0ea989e640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ea4074520 0x7f0ea4156d30 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f0e98034040 tx=0x7f0e980035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:56.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.247+0000 7f0ea2ffd640 1 -- 172.21.15.81:0/568673563 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9803e030 con 0x7f0ea4074520 2024-06-29T14:37:56.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.247+0000 7f0ea2ffd640 1 -- 172.21.15.81:0/568673563 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e98003ae0 con 0x7f0ea4074520 2024-06-29T14:37:56.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.247+0000 7f0ea2ffd640 1 -- 172.21.15.81:0/568673563 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e98035040 con 0x7f0ea4074520 2024-06-29T14:37:56.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.247+0000 7f0eac303640 1 -- 172.21.15.81:0/568673563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea41558a0 con 0x7f0ea4074520 2024-06-29T14:37:56.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.247+0000 7f0eac303640 1 -- 172.21.15.81:0/568673563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea4155e60 con 0x7f0ea4074520 2024-06-29T14:37:56.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.248+0000 7f0eac303640 1 -- 172.21.15.81:0/568673563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea40ae300 con 0x7f0ea4074520 2024-06-29T14:37:56.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.248+0000 7f0ea2ffd640 1 -- 172.21.15.81:0/568673563 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f0e98041030 con 0x7f0ea4074520 2024-06-29T14:37:56.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.249+0000 7f0ea2ffd640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e900763f0 0x7f0e90078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.249+0000 7f0ea2ffd640 1 -- 172.21.15.81:0/568673563 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f0e980b8240 con 0x7f0ea4074520 2024-06-29T14:37:56.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.249+0000 7f0eaa09f640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e900763f0 0x7f0e90078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.250+0000 7f0eaa09f640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e900763f0 0x7f0e90078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e940017a0 tx=0x7f0e94009040 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.253+0000 7f0ea2ffd640 1 -- 172.21.15.81:0/568673563 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0ea4156940 con 0x7f0ea4074520 2024-06-29T14:37:56.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.470+0000 7f0eac303640 1 -- 172.21.15.81:0/568673563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ea415a580 con 0x7f0ea4074520 2024-06-29T14:37:56.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.470+0000 7f0ea2ffd640 1 -- 172.21.15.81:0/568673563 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5225 (secure 0 0 0) 0x7f0e98081250 con 0x7f0ea4074520 2024-06-29T14:37:56.468 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:56.468 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":42,"btime":"2024-06-29T14:37:56:110726+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6002,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3644828063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3644828063},{"type":"v1","addr":"172.21.15.167:6836","nonce":3644828063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":7026,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2532767666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2532767666},{"type":"v1","addr":"172.21.15.81:6835","nonce":2532767666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":7029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/818227798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":818227798},{"type":"v1","addr":"172.21.15.81:6834","nonce":818227798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.462608+0000","modified":"2024-06-29T14:37:55.478070+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"b","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/910129539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":910129539},{"type":"v1","addr":"172.21.15.167:6837","nonce":910129539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":5}]} 2024-06-29T14:37:56.468 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 42 2024-06-29T14:37:56.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 -- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e900763f0 msgr2=0x7f0e90078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e900763f0 0x7f0e90078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e940017a0 tx=0x7f0e94009040 comp rx=0 tx=0).stop 2024-06-29T14:37:56.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 -- 172.21.15.81:0/568673563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ea4074520 msgr2=0x7f0ea4156d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ea4074520 0x7f0ea4156d30 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f0e98034040 tx=0x7f0e980035f0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 -- 172.21.15.81:0/568673563 shutdown_connections 2024-06-29T14:37:56.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e900763f0 0x7f0e90078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ea40ae840 0x7f0ea415d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ea4074520 0x7f0ea4156d30 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 --2- 172.21.15.81:0/568673563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ea406f850 0x7f0ea41522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 -- 172.21.15.81:0/568673563 >> 172.21.15.81:0/568673563 conn(0x7f0ea4072950 msgr2=0x7f0ea40b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:56.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 -- 172.21.15.81:0/568673563 shutdown_connections 2024-06-29T14:37:56.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.472+0000 7f0eac303640 1 -- 172.21.15.81:0/568673563 wait complete. 2024-06-29T14:37:56.481 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:37:56.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.597+0000 7f9a163a1640 1 -- 172.21.15.81:0/2078105059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8005250 msgr2=0x7f99f800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.597+0000 7f9a163a1640 1 --2- 172.21.15.81:0/2078105059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8005250 0x7f99f800b9c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f9a100b4cb0 tx=0x7f99f8005830 comp rx=0 tx=0).stop 2024-06-29T14:37:56.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.597+0000 7f9a163a1640 1 -- 172.21.15.81:0/2078105059 shutdown_connections 2024-06-29T14:37:56.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.597+0000 7f9a163a1640 1 --2- 172.21.15.81:0/2078105059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8005250 0x7f99f800b9c0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.597+0000 7f9a163a1640 1 -- 172.21.15.81:0/2078105059 >> 172.21.15.81:0/2078105059 conn(0x7f9a1006ccc0 msgr2=0x7f9a100b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:56.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.597+0000 7f9a163a1640 1 -- 172.21.15.81:0/2078105059 shutdown_connections 2024-06-29T14:37:56.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.597+0000 7f9a163a1640 1 -- 172.21.15.81:0/2078105059 wait complete. 2024-06-29T14:37:56.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.599+0000 7f9a163a1640 1 Processor -- start 2024-06-29T14:37:56.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.599+0000 7f9a163a1640 1 -- start start 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a163a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8005250 0x7f9a101efed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a163a1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a10131d30 0x7f9a1012e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a163a1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a10132280 0x7f9a1012e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a163a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1015eae0 con 0x7f99f8005250 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a163a1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1015e960 con 0x7f9a10131d30 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a163a1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9a1015ec60 con 0x7f9a10132280 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8005250 0x7f9a101efed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8005250 0x7f9a101efed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47792/0 (socket says 172.21.15.81:47792) 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0ffff640 1 -- 172.21.15.81:0/797975470 learned_addr learned my addr 172.21.15.81:0/797975470 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0ffff640 1 -- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a10132280 msgr2=0x7f9a1012e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0ffff640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a10132280 0x7f9a1012e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0ffff640 1 -- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a10131d30 msgr2=0x7f9a1012e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0ffff640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a10131d30 0x7f9a1012e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0ffff640 1 -- 172.21.15.81:0/797975470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1012f080 con 0x7f99f8005250 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0ffff640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8005250 0x7f9a101efed0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f9a00011ab0 tx=0x7f9a00011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0cff9640 1 -- 172.21.15.81:0/797975470 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0001e070 con 0x7f99f8005250 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a163a1640 1 -- 172.21.15.81:0/797975470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a10140af0 con 0x7f99f8005250 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0cff9640 1 -- 172.21.15.81:0/797975470 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a00004460 con 0x7f99f8005250 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a0cff9640 1 -- 172.21.15.81:0/797975470 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0000fc60 con 0x7f99f8005250 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.601+0000 7f9a163a1640 1 -- 172.21.15.81:0/797975470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a1012c280 con 0x7f99f8005250 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.602+0000 7f9a163a1640 1 -- 172.21.15.81:0/797975470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a1015f1c0 con 0x7f99f8005250 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.603+0000 7f9a0cff9640 1 -- 172.21.15.81:0/797975470 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f9a00021070 con 0x7f99f8005250 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.604+0000 7f9a0cff9640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f99fc076420 0x7f99fc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.604+0000 7f9a0cff9640 1 -- 172.21.15.81:0/797975470 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f9a00098450 con 0x7f99f8005250 2024-06-29T14:37:56.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.604+0000 7f9a0f7fe640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f99fc076420 0x7f99fc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.604+0000 7f9a0f7fe640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f99fc076420 0x7f99fc078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a04006fd0 tx=0x7f9a040069d0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:56.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.608+0000 7f9a0cff9640 1 -- 172.21.15.81:0/797975470 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9a000613e0 con 0x7f99f8005250 2024-06-29T14:37:56.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.789+0000 7f9a163a1640 1 -- 172.21.15.81:0/797975470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a1015a3c0 con 0x7f99f8005250 2024-06-29T14:37:56.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.790+0000 7f9a0cff9640 1 -- 172.21.15.81:0/797975470 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+16770 (secure 0 0 0) 0x7f9a000656e0 con 0x7f99f8005250 2024-06-29T14:37:56.788 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:56.788 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":91,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:37:54.098469+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:50.753390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.410843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 -- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f99fc076420 msgr2=0x7f99fc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f99fc076420 0x7f99fc078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a04006fd0 tx=0x7f9a040069d0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 -- 172.21.15.81:0/797975470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8005250 msgr2=0x7f9a101efed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8005250 0x7f9a101efed0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f9a00011ab0 tx=0x7f9a00011f70 comp rx=0 tx=0).stop 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 -- 172.21.15.81:0/797975470 shutdown_connections 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f99fc076420 0x7f99fc078930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a10132280 0x7f9a1012e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a10131d30 0x7f9a1012e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 --2- 172.21.15.81:0/797975470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8005250 0x7f9a101efed0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 -- 172.21.15.81:0/797975470 >> 172.21.15.81:0/797975470 conn(0x7f9a1006ccc0 msgr2=0x7f9a1015b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:56.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 -- 172.21.15.81:0/797975470 shutdown_connections 2024-06-29T14:37:56.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.792+0000 7f9a163a1640 1 -- 172.21.15.81:0/797975470 wait complete. 2024-06-29T14:37:56.801 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:37:56.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:37:56.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.918+0000 7f0eb1994640 1 -- 172.21.15.81:0/4219339890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0eac0b4250 msgr2=0x7f0eac0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.918+0000 7f0eb1994640 1 --2- 172.21.15.81:0/4219339890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0eac0b4250 0x7f0eac0cee70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4005fd0 tx=0x7f0ea402a780 comp rx=0 tx=0).stop 2024-06-29T14:37:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.918+0000 7f0eb1994640 1 -- 172.21.15.81:0/4219339890 shutdown_connections 2024-06-29T14:37:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.918+0000 7f0eb1994640 1 --2- 172.21.15.81:0/4219339890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0eac0b4250 0x7f0eac0cee70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.918+0000 7f0eb1994640 1 --2- 172.21.15.81:0/4219339890 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0eac08c270 0x7f0eac0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.918+0000 7f0eb1994640 1 -- 172.21.15.81:0/4219339890 >> 172.21.15.81:0/4219339890 conn(0x7f0eac0b56d0 msgr2=0x7f0eac0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.918+0000 7f0eb1994640 1 -- 172.21.15.81:0/4219339890 shutdown_connections 2024-06-29T14:37:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.918+0000 7f0eb1994640 1 -- 172.21.15.81:0/4219339890 wait complete. 2024-06-29T14:37:56.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.920+0000 7f0eb1994640 1 Processor -- start 2024-06-29T14:37:56.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.920+0000 7f0eb1994640 1 -- start start 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eb1994640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0eac08ccd0 0x7f0eac149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eb1994640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0eac0b4250 0x7f0eac149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eb1994640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0eac0b13f0 0x7f0eac0b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eb1994640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac0d2330 con 0x7f0eac0b4250 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eb1994640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0eac0d21b0 con 0x7f0eac0b13f0 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eb1994640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0eac0d24b0 con 0x7f0eac08ccd0 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eaa7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0eac0b4250 0x7f0eac149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eaaffd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0eac08ccd0 0x7f0eac149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eaa7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0eac0b4250 0x7f0eac149f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47796/0 (socket says 172.21.15.81:47796) 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eaa7fc640 1 -- 172.21.15.81:0/3746432980 learned_addr learned my addr 172.21.15.81:0/3746432980 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.921+0000 7f0eab7fe640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0eac0b13f0 0x7f0eac0b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0eaa7fc640 1 -- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0eac08ccd0 msgr2=0x7f0eac149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0eaa7fc640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0eac08ccd0 0x7f0eac149a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0eaa7fc640 1 -- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0eac0b13f0 msgr2=0x7f0eac0b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:56.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0eaa7fc640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0eac0b13f0 0x7f0eac0b1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:56.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0eaa7fc640 1 -- 172.21.15.81:0/3746432980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac1f8480 con 0x7f0eac0b4250 2024-06-29T14:37:56.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0eaa7fc640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0eac0b4250 0x7f0eac149f40 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000c3f0 tx=0x7f0ea000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:56.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0e93fff640 1 -- 172.21.15.81:0/3746432980 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0017070 con 0x7f0eac0b4250 2024-06-29T14:37:56.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0e93fff640 1 -- 172.21.15.81:0/3746432980 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ea00053c0 con 0x7f0eac0b4250 2024-06-29T14:37:56.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0e93fff640 1 -- 172.21.15.81:0/3746432980 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0003850 con 0x7f0eac0b4250 2024-06-29T14:37:56.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0eb1994640 1 -- 172.21.15.81:0/3746432980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eac1f8760 con 0x7f0eac0b4250 2024-06-29T14:37:56.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.922+0000 7f0eb1994640 1 -- 172.21.15.81:0/3746432980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eac1f8d20 con 0x7f0eac0b4250 2024-06-29T14:37:56.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.923+0000 7f0eb1994640 1 -- 172.21.15.81:0/3746432980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eac08c270 con 0x7f0eac0b4250 2024-06-29T14:37:56.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.924+0000 7f0e93fff640 1 -- 172.21.15.81:0/3746432980 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f0ea001a070 con 0x7f0eac0b4250 2024-06-29T14:37:56.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.924+0000 7f0e93fff640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e98076420 0x7f0e98078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:56.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.925+0000 7f0e93fff640 1 -- 172.21.15.81:0/3746432980 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f0ea0096d30 con 0x7f0eac0b4250 2024-06-29T14:37:56.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.925+0000 7f0eaaffd640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e98076420 0x7f0e98078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:56.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.925+0000 7f0eaaffd640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e98076420 0x7f0e98078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009530 tx=0x7f0e9c009260 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:56.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:56.929+0000 7f0e93fff640 1 -- 172.21.15.81:0/3746432980 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0ea005fdd0 con 0x7f0eac0b4250 2024-06-29T14:37:57.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.103+0000 7f0eb1994640 1 -- 172.21.15.81:0/3746432980 --> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0eac0b2e70 con 0x7f0e98076420 2024-06-29T14:37:57.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.108+0000 7f0e93fff640 1 -- 172.21.15.81:0/3746432980 <== mgr.6298 v2:172.21.15.167:6832/1959721003 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252082 (secure 0 0 0) 0x7f0eac0b2e70 con 0x7f0e98076420 2024-06-29T14:37:57.109 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:57.110 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.111+0000 7f0eb1994640 1 -- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e98076420 msgr2=0x7f0e98078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.111+0000 7f0eb1994640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e98076420 0x7f0e98078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009530 tx=0x7f0e9c009260 comp rx=0 tx=0).stop 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.111+0000 7f0eb1994640 1 -- 172.21.15.81:0/3746432980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0eac0b4250 msgr2=0x7f0eac149f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.111+0000 7f0eb1994640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0eac0b4250 0x7f0eac149f40 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000c3f0 tx=0x7f0ea000c8b0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.111+0000 7f0eb1994640 1 -- 172.21.15.81:0/3746432980 shutdown_connections 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.111+0000 7f0eb1994640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f0e98076420 0x7f0e98078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.111+0000 7f0eb1994640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0eac0b13f0 0x7f0eac0b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.111+0000 7f0eb1994640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0eac0b4250 0x7f0eac149f40 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.111+0000 7f0eb1994640 1 --2- 172.21.15.81:0/3746432980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0eac08ccd0 0x7f0eac149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.111+0000 7f0eb1994640 1 -- 172.21.15.81:0/3746432980 >> 172.21.15.81:0/3746432980 conn(0x7f0eac0b56d0 msgr2=0x7f0eac0b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:57.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.112+0000 7f0eb1994640 1 -- 172.21.15.81:0/3746432980 shutdown_connections 2024-06-29T14:37:57.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.112+0000 7f0eb1994640 1 -- 172.21.15.81:0/3746432980 wait complete. 2024-06-29T14:37:57.120 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:37:56.684035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":366,"num_read_kb":935,"num_write":206,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":150,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":312076,"kb_used_data":9860,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754662644,"statfs":{"total":773094113280,"available":772774547456,"internally_reserved":0,"allocated":10096640,"data_stored":5403336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":309317170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.603900"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107588+0000","last_change":"2024-06-29T14:37:53.111578+0000","last_active":"2024-06-29T14:37:54.107588+0000","last_peered":"2024-06-29T14:37:54.107588+0000","last_clean":"2024-06-29T14:37:54.107588+0000","last_became_active":"2024-06-29T14:37:53.111191+0000","last_became_peered":"2024-06-29T14:37:53.111191+0000","last_unstale":"2024-06-29T14:37:54.107588+0000","last_undegraded":"2024-06-29T14:37:54.107588+0000","last_fullsized":"2024-06-29T14:37:54.107588+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:34.305864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.224020+0000","last_change":"2024-06-29T14:37:53.116767+0000","last_active":"2024-06-29T14:37:54.224020+0000","last_peered":"2024-06-29T14:37:54.224020+0000","last_clean":"2024-06-29T14:37:54.224020+0000","last_became_active":"2024-06-29T14:37:53.116392+0000","last_became_peered":"2024-06-29T14:37:53.116392+0000","last_unstale":"2024-06-29T14:37:54.224020+0000","last_undegraded":"2024-06-29T14:37:54.224020+0000","last_fullsized":"2024-06-29T14:37:54.224020+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:30.643237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223482+0000","last_change":"2024-06-29T14:37:53.104489+0000","last_active":"2024-06-29T14:37:54.223482+0000","last_peered":"2024-06-29T14:37:54.223482+0000","last_clean":"2024-06-29T14:37:54.223482+0000","last_became_active":"2024-06-29T14:37:53.104192+0000","last_became_peered":"2024-06-29T14:37:53.104192+0000","last_unstale":"2024-06-29T14:37:54.223482+0000","last_undegraded":"2024-06-29T14:37:54.223482+0000","last_fullsized":"2024-06-29T14:37:54.223482+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:53:33.687273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222866+0000","last_change":"2024-06-29T14:37:53.112173+0000","last_active":"2024-06-29T14:37:54.222866+0000","last_peered":"2024-06-29T14:37:54.222866+0000","last_clean":"2024-06-29T14:37:54.222866+0000","last_became_active":"2024-06-29T14:37:53.111842+0000","last_became_peered":"2024-06-29T14:37:53.111842+0000","last_unstale":"2024-06-29T14:37:54.222866+0000","last_undegraded":"2024-06-29T14:37:54.222866+0000","last_fullsized":"2024-06-29T14:37:54.222866+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:20.219581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106519+0000","last_change":"2024-06-29T14:37:53.114238+0000","last_active":"2024-06-29T14:37:54.106519+0000","last_peered":"2024-06-29T14:37:54.106519+0000","last_clean":"2024-06-29T14:37:54.106519+0000","last_became_active":"2024-06-29T14:37:53.113937+0000","last_became_peered":"2024-06-29T14:37:53.113937+0000","last_unstale":"2024-06-29T14:37:54.106519+0000","last_undegraded":"2024-06-29T14:37:54.106519+0000","last_fullsized":"2024-06-29T14:37:54.106519+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:16:03.492357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223314+0000","last_change":"2024-06-29T14:37:53.108976+0000","last_active":"2024-06-29T14:37:54.223314+0000","last_peered":"2024-06-29T14:37:54.223314+0000","last_clean":"2024-06-29T14:37:54.223314+0000","last_became_active":"2024-06-29T14:37:53.108335+0000","last_became_peered":"2024-06-29T14:37:53.108335+0000","last_unstale":"2024-06-29T14:37:54.223314+0000","last_undegraded":"2024-06-29T14:37:54.223314+0000","last_fullsized":"2024-06-29T14:37:54.223314+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:13.401787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110342+0000","last_change":"2024-06-29T14:37:53.099815+0000","last_active":"2024-06-29T14:37:54.110342+0000","last_peered":"2024-06-29T14:37:54.110342+0000","last_clean":"2024-06-29T14:37:54.110342+0000","last_became_active":"2024-06-29T14:37:53.099520+0000","last_became_peered":"2024-06-29T14:37:53.099520+0000","last_unstale":"2024-06-29T14:37:54.110342+0000","last_undegraded":"2024-06-29T14:37:54.110342+0000","last_fullsized":"2024-06-29T14:37:54.110342+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:49.606231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223190+0000","last_change":"2024-06-29T14:37:53.109983+0000","last_active":"2024-06-29T14:37:54.223190+0000","last_peered":"2024-06-29T14:37:54.223190+0000","last_clean":"2024-06-29T14:37:54.223190+0000","last_became_active":"2024-06-29T14:37:53.108642+0000","last_became_peered":"2024-06-29T14:37:53.108642+0000","last_unstale":"2024-06-29T14:37:54.223190+0000","last_undegraded":"2024-06-29T14:37:54.223190+0000","last_fullsized":"2024-06-29T14:37:54.223190+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:16:03.492357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107320+0000","last_change":"2024-06-29T14:37:53.115824+0000","last_active":"2024-06-29T14:37:54.107320+0000","last_peered":"2024-06-29T14:37:54.107320+0000","last_clean":"2024-06-29T14:37:54.107320+0000","last_became_active":"2024-06-29T14:37:53.115485+0000","last_became_peered":"2024-06-29T14:37:53.115485+0000","last_unstale":"2024-06-29T14:37:54.107320+0000","last_undegraded":"2024-06-29T14:37:54.107320+0000","last_fullsized":"2024-06-29T14:37:54.107320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:52.622865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223156+0000","last_change":"2024-06-29T14:37:53.110102+0000","last_active":"2024-06-29T14:37:54.223156+0000","last_peered":"2024-06-29T14:37:54.223156+0000","last_clean":"2024-06-29T14:37:54.223156+0000","last_became_active":"2024-06-29T14:37:53.109724+0000","last_became_peered":"2024-06-29T14:37:53.109724+0000","last_unstale":"2024-06-29T14:37:54.223156+0000","last_undegraded":"2024-06-29T14:37:54.223156+0000","last_fullsized":"2024-06-29T14:37:54.223156+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:24.589063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.108213+0000","last_change":"2024-06-29T14:37:53.116306+0000","last_active":"2024-06-29T14:37:54.108213+0000","last_peered":"2024-06-29T14:37:54.108213+0000","last_clean":"2024-06-29T14:37:54.108213+0000","last_became_active":"2024-06-29T14:37:53.115422+0000","last_became_peered":"2024-06-29T14:37:53.115422+0000","last_unstale":"2024-06-29T14:37:54.108213+0000","last_undegraded":"2024-06-29T14:37:54.108213+0000","last_fullsized":"2024-06-29T14:37:54.108213+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:53:33.687273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223021+0000","last_change":"2024-06-29T14:37:53.109833+0000","last_active":"2024-06-29T14:37:54.223021+0000","last_peered":"2024-06-29T14:37:54.223021+0000","last_clean":"2024-06-29T14:37:54.223021+0000","last_became_active":"2024-06-29T14:37:53.109370+0000","last_became_peered":"2024-06-29T14:37:53.109370+0000","last_unstale":"2024-06-29T14:37:54.223021+0000","last_undegraded":"2024-06-29T14:37:54.223021+0000","last_fullsized":"2024-06-29T14:37:54.223021+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:38.086834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106978+0000","last_change":"2024-06-29T14:37:53.117154+0000","last_active":"2024-06-29T14:37:54.106978+0000","last_peered":"2024-06-29T14:37:54.106978+0000","last_clean":"2024-06-29T14:37:54.106978+0000","last_became_active":"2024-06-29T14:37:53.116863+0000","last_became_peered":"2024-06-29T14:37:53.116863+0000","last_unstale":"2024-06-29T14:37:54.106978+0000","last_undegraded":"2024-06-29T14:37:54.106978+0000","last_fullsized":"2024-06-29T14:37:54.106978+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:05.329593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106655+0000","last_change":"2024-06-29T14:37:53.112761+0000","last_active":"2024-06-29T14:37:54.106655+0000","last_peered":"2024-06-29T14:37:54.106655+0000","last_clean":"2024-06-29T14:37:54.106655+0000","last_became_active":"2024-06-29T14:37:53.112312+0000","last_became_peered":"2024-06-29T14:37:53.112312+0000","last_unstale":"2024-06-29T14:37:54.106655+0000","last_undegraded":"2024-06-29T14:37:54.106655+0000","last_fullsized":"2024-06-29T14:37:54.106655+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:20.219581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106937+0000","last_change":"2024-06-29T14:37:53.112281+0000","last_active":"2024-06-29T14:37:54.106937+0000","last_peered":"2024-06-29T14:37:54.106937+0000","last_clean":"2024-06-29T14:37:54.106937+0000","last_became_active":"2024-06-29T14:37:53.111959+0000","last_became_peered":"2024-06-29T14:37:53.111959+0000","last_unstale":"2024-06-29T14:37:54.106937+0000","last_undegraded":"2024-06-29T14:37:54.106937+0000","last_fullsized":"2024-06-29T14:37:54.106937+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:21.172170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222372+0000","last_change":"2024-06-29T14:37:53.113332+0000","last_active":"2024-06-29T14:37:54.222372+0000","last_peered":"2024-06-29T14:37:54.222372+0000","last_clean":"2024-06-29T14:37:54.222372+0000","last_became_active":"2024-06-29T14:37:53.112962+0000","last_became_peered":"2024-06-29T14:37:53.112962+0000","last_unstale":"2024-06-29T14:37:54.222372+0000","last_undegraded":"2024-06-29T14:37:54.222372+0000","last_fullsized":"2024-06-29T14:37:54.222372+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:39:05.831261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107440+0000","last_change":"2024-06-29T14:37:53.106745+0000","last_active":"2024-06-29T14:37:54.107440+0000","last_peered":"2024-06-29T14:37:54.107440+0000","last_clean":"2024-06-29T14:37:54.107440+0000","last_became_active":"2024-06-29T14:37:53.106415+0000","last_became_peered":"2024-06-29T14:37:53.106415+0000","last_unstale":"2024-06-29T14:37:54.107440+0000","last_undegraded":"2024-06-29T14:37:54.107440+0000","last_fullsized":"2024-06-29T14:37:54.107440+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:12.272265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223064+0000","last_change":"2024-06-29T14:37:53.112191+0000","last_active":"2024-06-29T14:37:54.223064+0000","last_peered":"2024-06-29T14:37:54.223064+0000","last_clean":"2024-06-29T14:37:54.223064+0000","last_became_active":"2024-06-29T14:37:53.111896+0000","last_became_peered":"2024-06-29T14:37:53.111896+0000","last_unstale":"2024-06-29T14:37:54.223064+0000","last_undegraded":"2024-06-29T14:37:54.223064+0000","last_fullsized":"2024-06-29T14:37:54.223064+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:42.695790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223809+0000","last_change":"2024-06-29T14:37:53.108911+0000","last_active":"2024-06-29T14:37:54.223809+0000","last_peered":"2024-06-29T14:37:54.223809+0000","last_clean":"2024-06-29T14:37:54.223809+0000","last_became_active":"2024-06-29T14:37:53.108568+0000","last_became_peered":"2024-06-29T14:37:53.108568+0000","last_unstale":"2024-06-29T14:37:54.223809+0000","last_undegraded":"2024-06-29T14:37:54.223809+0000","last_fullsized":"2024-06-29T14:37:54.223809+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:12.272265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106903+0000","last_change":"2024-06-29T14:37:53.111196+0000","last_active":"2024-06-29T14:37:54.106903+0000","last_peered":"2024-06-29T14:37:54.106903+0000","last_clean":"2024-06-29T14:37:54.106903+0000","last_became_active":"2024-06-29T14:37:53.110895+0000","last_became_peered":"2024-06-29T14:37:53.110895+0000","last_unstale":"2024-06-29T14:37:54.106903+0000","last_undegraded":"2024-06-29T14:37:54.106903+0000","last_fullsized":"2024-06-29T14:37:54.106903+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:16.484148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222753+0000","last_change":"2024-06-29T14:37:53.109750+0000","last_active":"2024-06-29T14:37:54.222753+0000","last_peered":"2024-06-29T14:37:54.222753+0000","last_clean":"2024-06-29T14:37:54.222753+0000","last_became_active":"2024-06-29T14:37:53.105224+0000","last_became_peered":"2024-06-29T14:37:53.105224+0000","last_unstale":"2024-06-29T14:37:54.222753+0000","last_undegraded":"2024-06-29T14:37:54.222753+0000","last_fullsized":"2024-06-29T14:37:54.222753+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:21.556805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222624+0000","last_change":"2024-06-29T14:37:53.109842+0000","last_active":"2024-06-29T14:37:54.222624+0000","last_peered":"2024-06-29T14:37:54.222624+0000","last_clean":"2024-06-29T14:37:54.222624+0000","last_became_active":"2024-06-29T14:37:53.109471+0000","last_became_peered":"2024-06-29T14:37:53.109471+0000","last_unstale":"2024-06-29T14:37:54.222624+0000","last_undegraded":"2024-06-29T14:37:54.222624+0000","last_fullsized":"2024-06-29T14:37:54.222624+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:00.233792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107724+0000","last_change":"2024-06-29T14:37:53.111759+0000","last_active":"2024-06-29T14:37:54.107724+0000","last_peered":"2024-06-29T14:37:54.107724+0000","last_clean":"2024-06-29T14:37:54.107724+0000","last_became_active":"2024-06-29T14:37:53.111383+0000","last_became_peered":"2024-06-29T14:37:53.111383+0000","last_unstale":"2024-06-29T14:37:54.107724+0000","last_undegraded":"2024-06-29T14:37:54.107724+0000","last_fullsized":"2024-06-29T14:37:54.107724+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:28.744782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107850+0000","last_change":"2024-06-29T14:37:53.112395+0000","last_active":"2024-06-29T14:37:54.107850+0000","last_peered":"2024-06-29T14:37:54.107850+0000","last_clean":"2024-06-29T14:37:54.107850+0000","last_became_active":"2024-06-29T14:37:53.112050+0000","last_became_peered":"2024-06-29T14:37:53.112050+0000","last_unstale":"2024-06-29T14:37:54.107850+0000","last_undegraded":"2024-06-29T14:37:54.107850+0000","last_fullsized":"2024-06-29T14:37:54.107850+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:40:28.605585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222911+0000","last_change":"2024-06-29T14:37:53.113480+0000","last_active":"2024-06-29T14:37:54.222911+0000","last_peered":"2024-06-29T14:37:54.222911+0000","last_clean":"2024-06-29T14:37:54.222911+0000","last_became_active":"2024-06-29T14:37:53.113148+0000","last_became_peered":"2024-06-29T14:37:53.113148+0000","last_unstale":"2024-06-29T14:37:54.222911+0000","last_undegraded":"2024-06-29T14:37:54.222911+0000","last_fullsized":"2024-06-29T14:37:54.222911+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:49.129124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110627+0000","last_change":"2024-06-29T14:37:53.108702+0000","last_active":"2024-06-29T14:37:54.110627+0000","last_peered":"2024-06-29T14:37:54.110627+0000","last_clean":"2024-06-29T14:37:54.110627+0000","last_became_active":"2024-06-29T14:37:53.108416+0000","last_became_peered":"2024-06-29T14:37:53.108416+0000","last_unstale":"2024-06-29T14:37:54.110627+0000","last_undegraded":"2024-06-29T14:37:54.110627+0000","last_fullsized":"2024-06-29T14:37:54.110627+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:27.329677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223511+0000","last_change":"2024-06-29T14:37:53.117067+0000","last_active":"2024-06-29T14:37:54.223511+0000","last_peered":"2024-06-29T14:37:54.223511+0000","last_clean":"2024-06-29T14:37:54.223511+0000","last_became_active":"2024-06-29T14:37:53.116704+0000","last_became_peered":"2024-06-29T14:37:53.116704+0000","last_unstale":"2024-06-29T14:37:54.223511+0000","last_undegraded":"2024-06-29T14:37:54.223511+0000","last_fullsized":"2024-06-29T14:37:54.223511+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:57.417821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222622+0000","last_change":"2024-06-29T14:37:53.106087+0000","last_active":"2024-06-29T14:37:54.222622+0000","last_peered":"2024-06-29T14:37:54.222622+0000","last_clean":"2024-06-29T14:37:54.222622+0000","last_became_active":"2024-06-29T14:37:53.105775+0000","last_became_peered":"2024-06-29T14:37:53.105775+0000","last_unstale":"2024-06-29T14:37:54.222622+0000","last_undegraded":"2024-06-29T14:37:54.222622+0000","last_fullsized":"2024-06-29T14:37:54.222622+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:39:03.121064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223259+0000","last_change":"2024-06-29T14:37:53.112320+0000","last_active":"2024-06-29T14:37:54.223259+0000","last_peered":"2024-06-29T14:37:54.223259+0000","last_clean":"2024-06-29T14:37:54.223259+0000","last_became_active":"2024-06-29T14:37:53.111998+0000","last_became_peered":"2024-06-29T14:37:53.111998+0000","last_unstale":"2024-06-29T14:37:54.223259+0000","last_undegraded":"2024-06-29T14:37:54.223259+0000","last_fullsized":"2024-06-29T14:37:54.223259+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:36:44.236489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110336+0000","last_change":"2024-06-29T14:37:53.107995+0000","last_active":"2024-06-29T14:37:54.110336+0000","last_peered":"2024-06-29T14:37:54.110336+0000","last_clean":"2024-06-29T14:37:54.110336+0000","last_became_active":"2024-06-29T14:37:53.107601+0000","last_became_peered":"2024-06-29T14:37:53.107601+0000","last_unstale":"2024-06-29T14:37:54.110336+0000","last_undegraded":"2024-06-29T14:37:54.110336+0000","last_fullsized":"2024-06-29T14:37:54.110336+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:42:50.058593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110461+0000","last_change":"2024-06-29T14:37:53.107318+0000","last_active":"2024-06-29T14:37:54.110461+0000","last_peered":"2024-06-29T14:37:54.110461+0000","last_clean":"2024-06-29T14:37:54.110461+0000","last_became_active":"2024-06-29T14:37:53.107009+0000","last_became_peered":"2024-06-29T14:37:53.107009+0000","last_unstale":"2024-06-29T14:37:54.110461+0000","last_undegraded":"2024-06-29T14:37:54.110461+0000","last_fullsized":"2024-06-29T14:37:54.110461+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:41.529557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223190+0000","last_change":"2024-06-29T14:37:53.118793+0000","last_active":"2024-06-29T14:37:54.223190+0000","last_peered":"2024-06-29T14:37:54.223190+0000","last_clean":"2024-06-29T14:37:54.223190+0000","last_became_active":"2024-06-29T14:37:53.118266+0000","last_became_peered":"2024-06-29T14:37:53.118266+0000","last_unstale":"2024-06-29T14:37:54.223190+0000","last_undegraded":"2024-06-29T14:37:54.223190+0000","last_fullsized":"2024-06-29T14:37:54.223190+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:20.219581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107660+0000","last_change":"2024-06-29T14:37:53.112129+0000","last_active":"2024-06-29T14:37:54.107660+0000","last_peered":"2024-06-29T14:37:54.107660+0000","last_clean":"2024-06-29T14:37:54.107660+0000","last_became_active":"2024-06-29T14:37:53.111566+0000","last_became_peered":"2024-06-29T14:37:53.111566+0000","last_unstale":"2024-06-29T14:37:54.107660+0000","last_undegraded":"2024-06-29T14:37:54.107660+0000","last_fullsized":"2024-06-29T14:37:54.107660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:42:48.680994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"90'4","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.108206+0000","last_change":"2024-06-29T14:37:52.110214+0000","last_active":"2024-06-29T14:37:54.108206+0000","last_peered":"2024-06-29T14:37:54.108206+0000","last_clean":"2024-06-29T14:37:54.108206+0000","last_became_active":"2024-06-29T14:37:52.109890+0000","last_became_peered":"2024-06-29T14:37:52.109890+0000","last_unstale":"2024-06-29T14:37:54.108206+0000","last_undegraded":"2024-06-29T14:37:54.108206+0000","last_fullsized":"2024-06-29T14:37:54.108206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:47:06.667446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107544+0000","last_change":"2024-06-29T14:37:53.105885+0000","last_active":"2024-06-29T14:37:54.107544+0000","last_peered":"2024-06-29T14:37:54.107544+0000","last_clean":"2024-06-29T14:37:54.107544+0000","last_became_active":"2024-06-29T14:37:53.105537+0000","last_became_peered":"2024-06-29T14:37:53.105537+0000","last_unstale":"2024-06-29T14:37:54.107544+0000","last_undegraded":"2024-06-29T14:37:54.107544+0000","last_fullsized":"2024-06-29T14:37:54.107544+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:30.643237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"90'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107229+0000","last_change":"2024-06-29T14:37:52.101360+0000","last_active":"2024-06-29T14:37:54.107229+0000","last_peered":"2024-06-29T14:37:54.107229+0000","last_clean":"2024-06-29T14:37:54.107229+0000","last_became_active":"2024-06-29T14:37:52.100991+0000","last_became_peered":"2024-06-29T14:37:52.100991+0000","last_unstale":"2024-06-29T14:37:54.107229+0000","last_undegraded":"2024-06-29T14:37:54.107229+0000","last_fullsized":"2024-06-29T14:37:54.107229+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:12.396428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223790+0000","last_change":"2024-06-29T14:37:53.120868+0000","last_active":"2024-06-29T14:37:54.223790+0000","last_peered":"2024-06-29T14:37:54.223790+0000","last_clean":"2024-06-29T14:37:54.223790+0000","last_became_active":"2024-06-29T14:37:53.119351+0000","last_became_peered":"2024-06-29T14:37:53.119351+0000","last_unstale":"2024-06-29T14:37:54.223790+0000","last_undegraded":"2024-06-29T14:37:54.223790+0000","last_fullsized":"2024-06-29T14:37:54.223790+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:42.695790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222833+0000","last_change":"2024-06-29T14:37:52.112731+0000","last_active":"2024-06-29T14:37:54.222833+0000","last_peered":"2024-06-29T14:37:54.222833+0000","last_clean":"2024-06-29T14:37:54.222833+0000","last_became_active":"2024-06-29T14:37:52.112436+0000","last_became_peered":"2024-06-29T14:37:52.112436+0000","last_unstale":"2024-06-29T14:37:54.222833+0000","last_undegraded":"2024-06-29T14:37:54.222833+0000","last_fullsized":"2024-06-29T14:37:54.222833+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:59:38.479274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.108134+0000","last_change":"2024-06-29T14:37:53.116910+0000","last_active":"2024-06-29T14:37:54.108134+0000","last_peered":"2024-06-29T14:37:54.108134+0000","last_clean":"2024-06-29T14:37:54.108134+0000","last_became_active":"2024-06-29T14:37:53.115676+0000","last_became_peered":"2024-06-29T14:37:53.115676+0000","last_unstale":"2024-06-29T14:37:54.108134+0000","last_undegraded":"2024-06-29T14:37:54.108134+0000","last_fullsized":"2024-06-29T14:37:54.108134+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:27.808263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223917+0000","last_change":"2024-06-29T14:37:52.112031+0000","last_active":"2024-06-29T14:37:54.223917+0000","last_peered":"2024-06-29T14:37:54.223917+0000","last_clean":"2024-06-29T14:37:54.223917+0000","last_became_active":"2024-06-29T14:37:52.111699+0000","last_became_peered":"2024-06-29T14:37:52.111699+0000","last_unstale":"2024-06-29T14:37:54.223917+0000","last_undegraded":"2024-06-29T14:37:54.223917+0000","last_fullsized":"2024-06-29T14:37:54.223917+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:28.495591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110664+0000","last_change":"2024-06-29T14:37:53.101327+0000","last_active":"2024-06-29T14:37:54.110664+0000","last_peered":"2024-06-29T14:37:54.110664+0000","last_clean":"2024-06-29T14:37:54.110664+0000","last_became_active":"2024-06-29T14:37:53.100903+0000","last_became_peered":"2024-06-29T14:37:53.100903+0000","last_unstale":"2024-06-29T14:37:54.110664+0000","last_undegraded":"2024-06-29T14:37:54.110664+0000","last_fullsized":"2024-06-29T14:37:54.110664+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:50:26.304625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"90'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223399+0000","last_change":"2024-06-29T14:37:52.112889+0000","last_active":"2024-06-29T14:37:54.223399+0000","last_peered":"2024-06-29T14:37:54.223399+0000","last_clean":"2024-06-29T14:37:54.223399+0000","last_became_active":"2024-06-29T14:37:52.112544+0000","last_became_peered":"2024-06-29T14:37:52.112544+0000","last_unstale":"2024-06-29T14:37:54.223399+0000","last_undegraded":"2024-06-29T14:37:54.223399+0000","last_fullsized":"2024-06-29T14:37:54.223399+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:22.505425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222727+0000","last_change":"2024-06-29T14:37:53.112364+0000","last_active":"2024-06-29T14:37:54.222727+0000","last_peered":"2024-06-29T14:37:54.222727+0000","last_clean":"2024-06-29T14:37:54.222727+0000","last_became_active":"2024-06-29T14:37:53.112023+0000","last_became_peered":"2024-06-29T14:37:53.112023+0000","last_unstale":"2024-06-29T14:37:54.222727+0000","last_undegraded":"2024-06-29T14:37:54.222727+0000","last_fullsized":"2024-06-29T14:37:54.222727+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:33.972067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"90'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223399+0000","last_change":"2024-06-29T14:37:52.109509+0000","last_active":"2024-06-29T14:37:54.223399+0000","last_peered":"2024-06-29T14:37:54.223399+0000","last_clean":"2024-06-29T14:37:54.223399+0000","last_became_active":"2024-06-29T14:37:52.109194+0000","last_became_peered":"2024-06-29T14:37:52.109194+0000","last_unstale":"2024-06-29T14:37:54.223399+0000","last_undegraded":"2024-06-29T14:37:54.223399+0000","last_fullsized":"2024-06-29T14:37:54.223399+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:32.361654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106193+0000","last_change":"2024-06-29T14:37:53.103525+0000","last_active":"2024-06-29T14:37:54.106193+0000","last_peered":"2024-06-29T14:37:54.106193+0000","last_clean":"2024-06-29T14:37:54.106193+0000","last_became_active":"2024-06-29T14:37:53.103214+0000","last_became_peered":"2024-06-29T14:37:53.103214+0000","last_unstale":"2024-06-29T14:37:54.106193+0000","last_undegraded":"2024-06-29T14:37:54.106193+0000","last_fullsized":"2024-06-29T14:37:54.106193+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:26.164210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222781+0000","last_change":"2024-06-29T14:37:52.109066+0000","last_active":"2024-06-29T14:37:54.222781+0000","last_peered":"2024-06-29T14:37:54.222781+0000","last_clean":"2024-06-29T14:37:54.222781+0000","last_became_active":"2024-06-29T14:37:52.108736+0000","last_became_peered":"2024-06-29T14:37:52.108736+0000","last_unstale":"2024-06-29T14:37:54.222781+0000","last_undegraded":"2024-06-29T14:37:54.222781+0000","last_fullsized":"2024-06-29T14:37:54.222781+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:09.719020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107115+0000","last_change":"2024-06-29T14:37:53.118647+0000","last_active":"2024-06-29T14:37:54.107115+0000","last_peered":"2024-06-29T14:37:54.107115+0000","last_clean":"2024-06-29T14:37:54.107115+0000","last_became_active":"2024-06-29T14:37:53.117679+0000","last_became_peered":"2024-06-29T14:37:53.117679+0000","last_unstale":"2024-06-29T14:37:54.107115+0000","last_undegraded":"2024-06-29T14:37:54.107115+0000","last_fullsized":"2024-06-29T14:37:54.107115+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:58.594346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106185+0000","last_change":"2024-06-29T14:37:52.102090+0000","last_active":"2024-06-29T14:37:54.106185+0000","last_peered":"2024-06-29T14:37:54.106185+0000","last_clean":"2024-06-29T14:37:54.106185+0000","last_became_active":"2024-06-29T14:37:52.101617+0000","last_became_peered":"2024-06-29T14:37:52.101617+0000","last_unstale":"2024-06-29T14:37:54.106185+0000","last_undegraded":"2024-06-29T14:37:54.106185+0000","last_fullsized":"2024-06-29T14:37:54.106185+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:05.193575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110626+0000","last_change":"2024-06-29T14:37:53.106169+0000","last_active":"2024-06-29T14:37:54.110626+0000","last_peered":"2024-06-29T14:37:54.110626+0000","last_clean":"2024-06-29T14:37:54.110626+0000","last_became_active":"2024-06-29T14:37:53.105816+0000","last_became_peered":"2024-06-29T14:37:53.105816+0000","last_unstale":"2024-06-29T14:37:54.110626+0000","last_undegraded":"2024-06-29T14:37:54.110626+0000","last_fullsized":"2024-06-29T14:37:54.110626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:19.515086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222842+0000","last_change":"2024-06-29T14:37:52.111657+0000","last_active":"2024-06-29T14:37:54.222842+0000","last_peered":"2024-06-29T14:37:54.222842+0000","last_clean":"2024-06-29T14:37:54.222842+0000","last_became_active":"2024-06-29T14:37:52.111212+0000","last_became_peered":"2024-06-29T14:37:52.111212+0000","last_unstale":"2024-06-29T14:37:54.222842+0000","last_undegraded":"2024-06-29T14:37:54.222842+0000","last_fullsized":"2024-06-29T14:37:54.222842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:27.663108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107896+0000","last_change":"2024-06-29T14:37:53.118101+0000","last_active":"2024-06-29T14:37:54.107896+0000","last_peered":"2024-06-29T14:37:54.107896+0000","last_clean":"2024-06-29T14:37:54.107896+0000","last_became_active":"2024-06-29T14:37:53.117761+0000","last_became_peered":"2024-06-29T14:37:53.117761+0000","last_unstale":"2024-06-29T14:37:54.107896+0000","last_undegraded":"2024-06-29T14:37:54.107896+0000","last_fullsized":"2024-06-29T14:37:54.107896+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:36:44.236489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.224042+0000","last_change":"2024-06-29T14:37:52.107836+0000","last_active":"2024-06-29T14:37:54.224042+0000","last_peered":"2024-06-29T14:37:54.224042+0000","last_clean":"2024-06-29T14:37:54.224042+0000","last_became_active":"2024-06-29T14:37:52.107495+0000","last_became_peered":"2024-06-29T14:37:52.107495+0000","last_unstale":"2024-06-29T14:37:54.224042+0000","last_undegraded":"2024-06-29T14:37:54.224042+0000","last_fullsized":"2024-06-29T14:37:54.224042+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:55.646551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223794+0000","last_change":"2024-06-29T14:37:53.111413+0000","last_active":"2024-06-29T14:37:54.223794+0000","last_peered":"2024-06-29T14:37:54.223794+0000","last_clean":"2024-06-29T14:37:54.223794+0000","last_became_active":"2024-06-29T14:37:53.109137+0000","last_became_peered":"2024-06-29T14:37:53.109137+0000","last_unstale":"2024-06-29T14:37:54.223794+0000","last_undegraded":"2024-06-29T14:37:54.223794+0000","last_fullsized":"2024-06-29T14:37:54.223794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:52.622865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"90'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223561+0000","last_change":"2024-06-29T14:37:52.114208+0000","last_active":"2024-06-29T14:37:54.223561+0000","last_peered":"2024-06-29T14:37:54.223561+0000","last_clean":"2024-06-29T14:37:54.223561+0000","last_became_active":"2024-06-29T14:37:52.113884+0000","last_became_peered":"2024-06-29T14:37:52.113884+0000","last_unstale":"2024-06-29T14:37:54.223561+0000","last_undegraded":"2024-06-29T14:37:54.223561+0000","last_fullsized":"2024-06-29T14:37:54.223561+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:37.912012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107137+0000","last_change":"2024-06-29T14:37:53.109776+0000","last_active":"2024-06-29T14:37:54.107137+0000","last_peered":"2024-06-29T14:37:54.107137+0000","last_clean":"2024-06-29T14:37:54.107137+0000","last_became_active":"2024-06-29T14:37:53.109474+0000","last_became_peered":"2024-06-29T14:37:53.109474+0000","last_unstale":"2024-06-29T14:37:54.107137+0000","last_undegraded":"2024-06-29T14:37:54.107137+0000","last_fullsized":"2024-06-29T14:37:54.107137+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:38.086834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"90'6","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223128+0000","last_change":"2024-06-29T14:37:52.098924+0000","last_active":"2024-06-29T14:37:54.223128+0000","last_peered":"2024-06-29T14:37:54.223128+0000","last_clean":"2024-06-29T14:37:54.223128+0000","last_became_active":"2024-06-29T14:37:52.098646+0000","last_became_peered":"2024-06-29T14:37:52.098646+0000","last_unstale":"2024-06-29T14:37:54.223128+0000","last_undegraded":"2024-06-29T14:37:54.223128+0000","last_fullsized":"2024-06-29T14:37:54.223128+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:08.728523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107358+0000","last_change":"2024-06-29T14:37:53.111620+0000","last_active":"2024-06-29T14:37:54.107358+0000","last_peered":"2024-06-29T14:37:54.107358+0000","last_clean":"2024-06-29T14:37:54.107358+0000","last_became_active":"2024-06-29T14:37:53.111239+0000","last_became_peered":"2024-06-29T14:37:53.111239+0000","last_unstale":"2024-06-29T14:37:54.107358+0000","last_undegraded":"2024-06-29T14:37:54.107358+0000","last_fullsized":"2024-06-29T14:37:54.107358+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:00:58.319938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222958+0000","last_change":"2024-06-29T14:37:52.104019+0000","last_active":"2024-06-29T14:37:54.222958+0000","last_peered":"2024-06-29T14:37:54.222958+0000","last_clean":"2024-06-29T14:37:54.222958+0000","last_became_active":"2024-06-29T14:37:52.103581+0000","last_became_peered":"2024-06-29T14:37:52.103581+0000","last_unstale":"2024-06-29T14:37:54.222958+0000","last_undegraded":"2024-06-29T14:37:54.222958+0000","last_fullsized":"2024-06-29T14:37:54.222958+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:38.499417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110580+0000","last_change":"2024-06-29T14:37:53.103766+0000","last_active":"2024-06-29T14:37:54.110580+0000","last_peered":"2024-06-29T14:37:54.110580+0000","last_clean":"2024-06-29T14:37:54.110580+0000","last_became_active":"2024-06-29T14:37:53.103339+0000","last_became_peered":"2024-06-29T14:37:53.103339+0000","last_unstale":"2024-06-29T14:37:54.110580+0000","last_undegraded":"2024-06-29T14:37:54.110580+0000","last_fullsized":"2024-06-29T14:37:54.110580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:23.224376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223675+0000","last_change":"2024-06-29T14:37:52.106538+0000","last_active":"2024-06-29T14:37:54.223675+0000","last_peered":"2024-06-29T14:37:54.223675+0000","last_clean":"2024-06-29T14:37:54.223675+0000","last_became_active":"2024-06-29T14:37:52.106146+0000","last_became_peered":"2024-06-29T14:37:52.106146+0000","last_unstale":"2024-06-29T14:37:54.223675+0000","last_undegraded":"2024-06-29T14:37:54.223675+0000","last_fullsized":"2024-06-29T14:37:54.223675+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:45.083202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106691+0000","last_change":"2024-06-29T14:37:53.118627+0000","last_active":"2024-06-29T14:37:54.106691+0000","last_peered":"2024-06-29T14:37:54.106691+0000","last_clean":"2024-06-29T14:37:54.106691+0000","last_became_active":"2024-06-29T14:37:53.118062+0000","last_became_peered":"2024-06-29T14:37:53.118062+0000","last_unstale":"2024-06-29T14:37:54.106691+0000","last_undegraded":"2024-06-29T14:37:54.106691+0000","last_fullsized":"2024-06-29T14:37:54.106691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:24.041354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110233+0000","last_change":"2024-06-29T14:37:52.103863+0000","last_active":"2024-06-29T14:37:54.110233+0000","last_peered":"2024-06-29T14:37:54.110233+0000","last_clean":"2024-06-29T14:37:54.110233+0000","last_became_active":"2024-06-29T14:37:52.103546+0000","last_became_peered":"2024-06-29T14:37:52.103546+0000","last_unstale":"2024-06-29T14:37:54.110233+0000","last_undegraded":"2024-06-29T14:37:54.110233+0000","last_fullsized":"2024-06-29T14:37:54.110233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:43.095421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106484+0000","last_change":"2024-06-29T14:37:53.116125+0000","last_active":"2024-06-29T14:37:54.106484+0000","last_peered":"2024-06-29T14:37:54.106484+0000","last_clean":"2024-06-29T14:37:54.106484+0000","last_became_active":"2024-06-29T14:37:53.115798+0000","last_became_peered":"2024-06-29T14:37:53.115798+0000","last_unstale":"2024-06-29T14:37:54.106484+0000","last_undegraded":"2024-06-29T14:37:54.106484+0000","last_fullsized":"2024-06-29T14:37:54.106484+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:59.235978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222374+0000","last_change":"2024-06-29T14:37:52.109753+0000","last_active":"2024-06-29T14:37:54.222374+0000","last_peered":"2024-06-29T14:37:54.222374+0000","last_clean":"2024-06-29T14:37:54.222374+0000","last_became_active":"2024-06-29T14:37:52.108953+0000","last_became_peered":"2024-06-29T14:37:52.108953+0000","last_unstale":"2024-06-29T14:37:54.222374+0000","last_undegraded":"2024-06-29T14:37:54.222374+0000","last_fullsized":"2024-06-29T14:37:54.222374+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:34:35.163677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223338+0000","last_change":"2024-06-29T14:37:53.100142+0000","last_active":"2024-06-29T14:37:54.223338+0000","last_peered":"2024-06-29T14:37:54.223338+0000","last_clean":"2024-06-29T14:37:54.223338+0000","last_became_active":"2024-06-29T14:37:53.099794+0000","last_became_peered":"2024-06-29T14:37:53.099794+0000","last_unstale":"2024-06-29T14:37:54.223338+0000","last_undegraded":"2024-06-29T14:37:54.223338+0000","last_fullsized":"2024-06-29T14:37:54.223338+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:22.100876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"90'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223916+0000","last_change":"2024-06-29T14:37:52.111533+0000","last_active":"2024-06-29T14:37:54.223916+0000","last_peered":"2024-06-29T14:37:54.223916+0000","last_clean":"2024-06-29T14:37:54.223916+0000","last_became_active":"2024-06-29T14:37:52.110801+0000","last_became_peered":"2024-06-29T14:37:52.110801+0000","last_unstale":"2024-06-29T14:37:54.223916+0000","last_undegraded":"2024-06-29T14:37:54.223916+0000","last_fullsized":"2024-06-29T14:37:54.223916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:15.045019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.108084+0000","last_change":"2024-06-29T14:37:53.115001+0000","last_active":"2024-06-29T14:37:54.108084+0000","last_peered":"2024-06-29T14:37:54.108084+0000","last_clean":"2024-06-29T14:37:54.108084+0000","last_became_active":"2024-06-29T14:37:53.114676+0000","last_became_peered":"2024-06-29T14:37:53.114676+0000","last_unstale":"2024-06-29T14:37:54.108084+0000","last_undegraded":"2024-06-29T14:37:54.108084+0000","last_fullsized":"2024-06-29T14:37:54.108084+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:26.272219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223474+0000","last_change":"2024-06-29T14:37:52.115847+0000","last_active":"2024-06-29T14:37:54.223474+0000","last_peered":"2024-06-29T14:37:54.223474+0000","last_clean":"2024-06-29T14:37:54.223474+0000","last_became_active":"2024-06-29T14:37:52.115526+0000","last_became_peered":"2024-06-29T14:37:52.115526+0000","last_unstale":"2024-06-29T14:37:54.223474+0000","last_undegraded":"2024-06-29T14:37:54.223474+0000","last_fullsized":"2024-06-29T14:37:54.223474+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:26.133687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223077+0000","last_change":"2024-06-29T14:37:53.104206+0000","last_active":"2024-06-29T14:37:54.223077+0000","last_peered":"2024-06-29T14:37:54.223077+0000","last_clean":"2024-06-29T14:37:54.223077+0000","last_became_active":"2024-06-29T14:37:53.103873+0000","last_became_peered":"2024-06-29T14:37:53.103873+0000","last_unstale":"2024-06-29T14:37:54.223077+0000","last_undegraded":"2024-06-29T14:37:54.223077+0000","last_fullsized":"2024-06-29T14:37:54.223077+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:52.622865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"90'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223782+0000","last_change":"2024-06-29T14:37:52.112758+0000","last_active":"2024-06-29T14:37:54.223782+0000","last_peered":"2024-06-29T14:37:54.223782+0000","last_clean":"2024-06-29T14:37:54.223782+0000","last_became_active":"2024-06-29T14:37:52.112438+0000","last_became_peered":"2024-06-29T14:37:52.112438+0000","last_unstale":"2024-06-29T14:37:54.223782+0000","last_undegraded":"2024-06-29T14:37:54.223782+0000","last_fullsized":"2024-06-29T14:37:54.223782+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:25:52.179698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107320+0000","last_change":"2024-06-29T14:37:53.112345+0000","last_active":"2024-06-29T14:37:54.107320+0000","last_peered":"2024-06-29T14:37:54.107320+0000","last_clean":"2024-06-29T14:37:54.107320+0000","last_became_active":"2024-06-29T14:37:53.111719+0000","last_became_peered":"2024-06-29T14:37:53.111719+0000","last_unstale":"2024-06-29T14:37:54.107320+0000","last_undegraded":"2024-06-29T14:37:54.107320+0000","last_fullsized":"2024-06-29T14:37:54.107320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:42.695790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107819+0000","last_change":"2024-06-29T14:37:52.113119+0000","last_active":"2024-06-29T14:37:54.107819+0000","last_peered":"2024-06-29T14:37:54.107819+0000","last_clean":"2024-06-29T14:37:54.107819+0000","last_became_active":"2024-06-29T14:37:52.112787+0000","last_became_peered":"2024-06-29T14:37:52.112787+0000","last_unstale":"2024-06-29T14:37:54.107819+0000","last_undegraded":"2024-06-29T14:37:54.107819+0000","last_fullsized":"2024-06-29T14:37:54.107819+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:23.559283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223456+0000","last_change":"2024-06-29T14:37:53.103357+0000","last_active":"2024-06-29T14:37:54.223456+0000","last_peered":"2024-06-29T14:37:54.223456+0000","last_clean":"2024-06-29T14:37:54.223456+0000","last_became_active":"2024-06-29T14:37:53.103036+0000","last_became_peered":"2024-06-29T14:37:53.103036+0000","last_unstale":"2024-06-29T14:37:54.223456+0000","last_undegraded":"2024-06-29T14:37:54.223456+0000","last_fullsized":"2024-06-29T14:37:54.223456+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:27.808263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223551+0000","last_change":"2024-06-29T14:37:52.108612+0000","last_active":"2024-06-29T14:37:54.223551+0000","last_peered":"2024-06-29T14:37:54.223551+0000","last_clean":"2024-06-29T14:37:54.223551+0000","last_became_active":"2024-06-29T14:37:52.108264+0000","last_became_peered":"2024-06-29T14:37:52.108264+0000","last_unstale":"2024-06-29T14:37:54.223551+0000","last_undegraded":"2024-06-29T14:37:54.223551+0000","last_fullsized":"2024-06-29T14:37:54.223551+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:16.748536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223183+0000","last_change":"2024-06-29T14:37:53.110153+0000","last_active":"2024-06-29T14:37:54.223183+0000","last_peered":"2024-06-29T14:37:54.223183+0000","last_clean":"2024-06-29T14:37:54.223183+0000","last_became_active":"2024-06-29T14:37:53.109833+0000","last_became_peered":"2024-06-29T14:37:53.109833+0000","last_unstale":"2024-06-29T14:37:54.223183+0000","last_undegraded":"2024-06-29T14:37:54.223183+0000","last_fullsized":"2024-06-29T14:37:54.223183+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:56:29.828443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110510+0000","last_change":"2024-06-29T14:37:52.104682+0000","last_active":"2024-06-29T14:37:54.110510+0000","last_peered":"2024-06-29T14:37:54.110510+0000","last_clean":"2024-06-29T14:37:54.110510+0000","last_became_active":"2024-06-29T14:37:52.104388+0000","last_became_peered":"2024-06-29T14:37:52.104388+0000","last_unstale":"2024-06-29T14:37:54.110510+0000","last_undegraded":"2024-06-29T14:37:54.110510+0000","last_fullsized":"2024-06-29T14:37:54.110510+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:27.563821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223348+0000","last_change":"2024-06-29T14:37:53.109361+0000","last_active":"2024-06-29T14:37:54.223348+0000","last_peered":"2024-06-29T14:37:54.223348+0000","last_clean":"2024-06-29T14:37:54.223348+0000","last_became_active":"2024-06-29T14:37:53.109028+0000","last_became_peered":"2024-06-29T14:37:53.109028+0000","last_unstale":"2024-06-29T14:37:54.223348+0000","last_undegraded":"2024-06-29T14:37:54.223348+0000","last_fullsized":"2024-06-29T14:37:54.223348+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:26.272219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106636+0000","last_change":"2024-06-29T14:37:52.104697+0000","last_active":"2024-06-29T14:37:54.106636+0000","last_peered":"2024-06-29T14:37:54.106636+0000","last_clean":"2024-06-29T14:37:54.106636+0000","last_became_active":"2024-06-29T14:37:52.104373+0000","last_became_peered":"2024-06-29T14:37:52.104373+0000","last_unstale":"2024-06-29T14:37:54.106636+0000","last_undegraded":"2024-06-29T14:37:54.106636+0000","last_fullsized":"2024-06-29T14:37:54.106636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:19.214222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107059+0000","last_change":"2024-06-29T14:37:53.113192+0000","last_active":"2024-06-29T14:37:54.107059+0000","last_peered":"2024-06-29T14:37:54.107059+0000","last_clean":"2024-06-29T14:37:54.107059+0000","last_became_active":"2024-06-29T14:37:53.112869+0000","last_became_peered":"2024-06-29T14:37:53.112869+0000","last_unstale":"2024-06-29T14:37:54.107059+0000","last_undegraded":"2024-06-29T14:37:54.107059+0000","last_fullsized":"2024-06-29T14:37:54.107059+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:09:10.493935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223939+0000","last_change":"2024-06-29T14:37:52.111123+0000","last_active":"2024-06-29T14:37:54.223939+0000","last_peered":"2024-06-29T14:37:54.223939+0000","last_clean":"2024-06-29T14:37:54.223939+0000","last_became_active":"2024-06-29T14:37:52.110622+0000","last_became_peered":"2024-06-29T14:37:52.110622+0000","last_unstale":"2024-06-29T14:37:54.223939+0000","last_undegraded":"2024-06-29T14:37:54.223939+0000","last_fullsized":"2024-06-29T14:37:54.223939+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:53:00.292129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110613+0000","last_change":"2024-06-29T14:37:53.102032+0000","last_active":"2024-06-29T14:37:54.110613+0000","last_peered":"2024-06-29T14:37:54.110613+0000","last_clean":"2024-06-29T14:37:54.110613+0000","last_became_active":"2024-06-29T14:37:53.101733+0000","last_became_peered":"2024-06-29T14:37:53.101733+0000","last_unstale":"2024-06-29T14:37:54.110613+0000","last_undegraded":"2024-06-29T14:37:54.110613+0000","last_fullsized":"2024-06-29T14:37:54.110613+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:19.886753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"91'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.111935+0000","last_change":"2024-06-29T14:37:52.103694+0000","last_active":"2024-06-29T14:37:54.111935+0000","last_peered":"2024-06-29T14:37:54.111935+0000","last_clean":"2024-06-29T14:37:54.111935+0000","last_became_active":"2024-06-29T14:37:52.103319+0000","last_became_peered":"2024-06-29T14:37:52.103319+0000","last_unstale":"2024-06-29T14:37:54.111935+0000","last_undegraded":"2024-06-29T14:37:54.111935+0000","last_fullsized":"2024-06-29T14:37:54.111935+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:40:53.466961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107762+0000","last_change":"2024-06-29T14:37:53.116211+0000","last_active":"2024-06-29T14:37:54.107762+0000","last_peered":"2024-06-29T14:37:54.107762+0000","last_clean":"2024-06-29T14:37:54.107762+0000","last_became_active":"2024-06-29T14:37:53.115841+0000","last_became_peered":"2024-06-29T14:37:53.115841+0000","last_unstale":"2024-06-29T14:37:54.107762+0000","last_undegraded":"2024-06-29T14:37:54.107762+0000","last_fullsized":"2024-06-29T14:37:54.107762+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:56:29.828443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107377+0000","last_change":"2024-06-29T14:37:52.102785+0000","last_active":"2024-06-29T14:37:54.107377+0000","last_peered":"2024-06-29T14:37:54.107377+0000","last_clean":"2024-06-29T14:37:54.107377+0000","last_became_active":"2024-06-29T14:37:52.102461+0000","last_became_peered":"2024-06-29T14:37:52.102461+0000","last_unstale":"2024-06-29T14:37:54.107377+0000","last_undegraded":"2024-06-29T14:37:54.107377+0000","last_fullsized":"2024-06-29T14:37:54.107377+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:16.748536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.108310+0000","last_change":"2024-06-29T14:37:53.116666+0000","last_active":"2024-06-29T14:37:54.108310+0000","last_peered":"2024-06-29T14:37:54.108310+0000","last_clean":"2024-06-29T14:37:54.108310+0000","last_became_active":"2024-06-29T14:37:53.115938+0000","last_became_peered":"2024-06-29T14:37:53.115938+0000","last_unstale":"2024-06-29T14:37:54.108310+0000","last_undegraded":"2024-06-29T14:37:54.108310+0000","last_fullsized":"2024-06-29T14:37:54.108310+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:57.417821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107019+0000","last_change":"2024-06-29T14:37:52.109330+0000","last_active":"2024-06-29T14:37:54.107019+0000","last_peered":"2024-06-29T14:37:54.107019+0000","last_clean":"2024-06-29T14:37:54.107019+0000","last_became_active":"2024-06-29T14:37:52.109025+0000","last_became_peered":"2024-06-29T14:37:52.109025+0000","last_unstale":"2024-06-29T14:37:54.107019+0000","last_undegraded":"2024-06-29T14:37:54.107019+0000","last_fullsized":"2024-06-29T14:37:54.107019+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:16:02.486998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106839+0000","last_change":"2024-06-29T14:37:53.116204+0000","last_active":"2024-06-29T14:37:54.106839+0000","last_peered":"2024-06-29T14:37:54.106839+0000","last_clean":"2024-06-29T14:37:54.106839+0000","last_became_active":"2024-06-29T14:37:53.115875+0000","last_became_peered":"2024-06-29T14:37:53.115875+0000","last_unstale":"2024-06-29T14:37:54.106839+0000","last_undegraded":"2024-06-29T14:37:54.106839+0000","last_fullsized":"2024-06-29T14:37:54.106839+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:13.401787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107371+0000","last_change":"2024-06-29T14:37:52.106510+0000","last_active":"2024-06-29T14:37:54.107371+0000","last_peered":"2024-06-29T14:37:54.107371+0000","last_clean":"2024-06-29T14:37:54.107371+0000","last_became_active":"2024-06-29T14:37:52.106162+0000","last_became_peered":"2024-06-29T14:37:52.106162+0000","last_unstale":"2024-06-29T14:37:54.107371+0000","last_undegraded":"2024-06-29T14:37:54.107371+0000","last_fullsized":"2024-06-29T14:37:54.107371+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:15.045019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223355+0000","last_change":"2024-06-29T14:37:53.119291+0000","last_active":"2024-06-29T14:37:54.223355+0000","last_peered":"2024-06-29T14:37:54.223355+0000","last_clean":"2024-06-29T14:37:54.223355+0000","last_became_active":"2024-06-29T14:37:53.118604+0000","last_became_peered":"2024-06-29T14:37:53.118604+0000","last_unstale":"2024-06-29T14:37:54.223355+0000","last_undegraded":"2024-06-29T14:37:54.223355+0000","last_fullsized":"2024-06-29T14:37:54.223355+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:38.086834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222704+0000","last_change":"2024-06-29T14:37:52.110474+0000","last_active":"2024-06-29T14:37:54.222704+0000","last_peered":"2024-06-29T14:37:54.222704+0000","last_clean":"2024-06-29T14:37:54.222704+0000","last_became_active":"2024-06-29T14:37:52.110163+0000","last_became_peered":"2024-06-29T14:37:52.110163+0000","last_unstale":"2024-06-29T14:37:54.222704+0000","last_undegraded":"2024-06-29T14:37:54.222704+0000","last_fullsized":"2024-06-29T14:37:54.222704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:38.025126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222932+0000","last_change":"2024-06-29T14:37:53.105314+0000","last_active":"2024-06-29T14:37:54.222932+0000","last_peered":"2024-06-29T14:37:54.222932+0000","last_clean":"2024-06-29T14:37:54.222932+0000","last_became_active":"2024-06-29T14:37:53.105004+0000","last_became_peered":"2024-06-29T14:37:53.105004+0000","last_unstale":"2024-06-29T14:37:54.222932+0000","last_undegraded":"2024-06-29T14:37:54.222932+0000","last_fullsized":"2024-06-29T14:37:54.222932+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:13.401787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222306+0000","last_change":"2024-06-29T14:37:52.113877+0000","last_active":"2024-06-29T14:37:54.222306+0000","last_peered":"2024-06-29T14:37:54.222306+0000","last_clean":"2024-06-29T14:37:54.222306+0000","last_became_active":"2024-06-29T14:37:52.113535+0000","last_became_peered":"2024-06-29T14:37:52.113535+0000","last_unstale":"2024-06-29T14:37:54.222306+0000","last_undegraded":"2024-06-29T14:37:54.222306+0000","last_fullsized":"2024-06-29T14:37:54.222306+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:30:21.560571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223172+0000","last_change":"2024-06-29T14:37:53.117832+0000","last_active":"2024-06-29T14:37:54.223172+0000","last_peered":"2024-06-29T14:37:54.223172+0000","last_clean":"2024-06-29T14:37:54.223172+0000","last_became_active":"2024-06-29T14:37:53.116160+0000","last_became_peered":"2024-06-29T14:37:53.116160+0000","last_unstale":"2024-06-29T14:37:54.223172+0000","last_undegraded":"2024-06-29T14:37:54.223172+0000","last_fullsized":"2024-06-29T14:37:54.223172+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:59.235978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223329+0000","last_change":"2024-06-29T14:37:52.112695+0000","last_active":"2024-06-29T14:37:54.223329+0000","last_peered":"2024-06-29T14:37:54.223329+0000","last_clean":"2024-06-29T14:37:54.223329+0000","last_became_active":"2024-06-29T14:37:52.112377+0000","last_became_peered":"2024-06-29T14:37:52.112377+0000","last_unstale":"2024-06-29T14:37:54.223329+0000","last_undegraded":"2024-06-29T14:37:54.223329+0000","last_fullsized":"2024-06-29T14:37:54.223329+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:05.193575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223102+0000","last_change":"2024-06-29T14:37:53.111427+0000","last_active":"2024-06-29T14:37:54.223102+0000","last_peered":"2024-06-29T14:37:54.223102+0000","last_clean":"2024-06-29T14:37:54.223102+0000","last_became_active":"2024-06-29T14:37:53.111124+0000","last_became_peered":"2024-06-29T14:37:53.111124+0000","last_unstale":"2024-06-29T14:37:54.223102+0000","last_undegraded":"2024-06-29T14:37:54.223102+0000","last_fullsized":"2024-06-29T14:37:54.223102+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:28.744782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223511+0000","last_change":"2024-06-29T14:37:52.111603+0000","last_active":"2024-06-29T14:37:54.223511+0000","last_peered":"2024-06-29T14:37:54.223511+0000","last_clean":"2024-06-29T14:37:54.223511+0000","last_became_active":"2024-06-29T14:37:52.111262+0000","last_became_peered":"2024-06-29T14:37:52.111262+0000","last_unstale":"2024-06-29T14:37:54.223511+0000","last_undegraded":"2024-06-29T14:37:54.223511+0000","last_fullsized":"2024-06-29T14:37:54.223511+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:02:54.388993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"80'107","reported_seq":376,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107238+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:37:54.107238+0000","last_peered":"2024-06-29T14:37:54.107238+0000","last_clean":"2024-06-29T14:37:54.107238+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:37:54.107238+0000","last_undegraded":"2024-06-29T14:37:54.107238+0000","last_fullsized":"2024-06-29T14:37:54.107238+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":366,"num_read_kb":935,"num_write":161,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":366,"num_read_kb":935,"num_write":161,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575036,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1699840,"data_stored":1114224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575037,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1118208,"data_stored":529704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575035,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1114112,"data_stored":529702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58624,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313216,"statfs":{"total":96636764160,"available":96576733184,"internally_reserved":0,"allocated":1101824,"data_stored":523310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575036,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36420,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335420,"statfs":{"total":96636764160,"available":96599470080,"internally_reserved":0,"allocated":1105920,"data_stored":523352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575036,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36440,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335400,"statfs":{"total":96636764160,"available":96599449600,"internally_reserved":0,"allocated":1122304,"data_stored":531224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575035,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50340,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321500,"statfs":{"total":96636764160,"available":96585216000,"internally_reserved":0,"allocated":1130496,"data_stored":531840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575036,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":1119980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:37:57.124 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:37:57.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.276+0000 7fe51e505640 1 -- 172.21.15.81:0/2129083254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe518073570 msgr2=0x7fe51815e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.276+0000 7fe51e505640 1 --2- 172.21.15.81:0/2129083254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe518073570 0x7fe51815e960 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe5180b2700 tx=0x7fe51002aa30 comp rx=0 tx=0).stop 2024-06-29T14:37:57.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.276+0000 7fe51e505640 1 -- 172.21.15.81:0/2129083254 shutdown_connections 2024-06-29T14:37:57.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.276+0000 7fe51e505640 1 --2- 172.21.15.81:0/2129083254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe518073570 0x7fe51815e960 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.276+0000 7fe51e505640 1 --2- 172.21.15.81:0/2129083254 >> v2:172.21.15.167:3300/0 conn(0x7fe5180cf4d0 0x7fe518073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.276+0000 7fe51e505640 1 --2- 172.21.15.81:0/2129083254 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5181412f0 0x7fe5180cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.276+0000 7fe51e505640 1 -- 172.21.15.81:0/2129083254 >> 172.21.15.81:0/2129083254 conn(0x7fe51806ccc0 msgr2=0x7fe5180b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:57.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.276+0000 7fe51e505640 1 -- 172.21.15.81:0/2129083254 shutdown_connections 2024-06-29T14:37:57.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.276+0000 7fe51e505640 1 -- 172.21.15.81:0/2129083254 wait complete. 2024-06-29T14:37:57.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.278+0000 7fe51e505640 1 Processor -- start 2024-06-29T14:37:57.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.278+0000 7fe51e505640 1 -- start start 2024-06-29T14:37:57.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe51e505640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe518073570 0x7fe5180a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe51e505640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5180cf4d0 0x7fe51815d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe51e505640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5181412f0 0x7fe51815d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe51e505640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe51815f230 con 0x7fe5180cf4d0 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe51e505640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe51815f0b0 con 0x7fe518073570 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe51e505640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe5180b3210 con 0x7fe5181412f0 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe5177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5180cf4d0 0x7fe51815d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe5177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5180cf4d0 0x7fe51815d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47802/0 (socket says 172.21.15.81:47802) 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe5177fe640 1 -- 172.21.15.81:0/3832485075 learned_addr learned my addr 172.21.15.81:0/3832485075 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe517fff640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe518073570 0x7fe5180a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.279+0000 7fe51caa2640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5181412f0 0x7fe51815d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe5177fe640 1 -- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5181412f0 msgr2=0x7fe51815d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe5177fe640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5181412f0 0x7fe51815d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe5177fe640 1 -- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe518073570 msgr2=0x7fe5180a9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe5177fe640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe518073570 0x7fe5180a9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe5177fe640 1 -- 172.21.15.81:0/3832485075 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5180adf30 con 0x7fe5180cf4d0 2024-06-29T14:37:57.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe5177fe640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5180cf4d0 0x7fe51815d190 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe50c00b410 tx=0x7fe50c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:57.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe514ff9640 1 -- 172.21.15.81:0/3832485075 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c015020 con 0x7fe5180cf4d0 2024-06-29T14:37:57.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe514ff9640 1 -- 172.21.15.81:0/3832485075 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe50c011920 con 0x7fe5180cf4d0 2024-06-29T14:37:57.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe51e505640 1 -- 172.21.15.81:0/3832485075 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5180ab330 con 0x7fe5180cf4d0 2024-06-29T14:37:57.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe514ff9640 1 -- 172.21.15.81:0/3832485075 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c011d70 con 0x7fe5180cf4d0 2024-06-29T14:37:57.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.280+0000 7fe51e505640 1 -- 172.21.15.81:0/3832485075 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5180ab8f0 con 0x7fe5180cf4d0 2024-06-29T14:37:57.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.281+0000 7fe51e505640 1 -- 172.21.15.81:0/3832485075 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe518073030 con 0x7fe5180cf4d0 2024-06-29T14:37:57.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.282+0000 7fe514ff9640 1 -- 172.21.15.81:0/3832485075 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fe50c018020 con 0x7fe5180cf4d0 2024-06-29T14:37:57.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.282+0000 7fe514ff9640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe504076420 0x7fe504078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.282+0000 7fe514ff9640 1 -- 172.21.15.81:0/3832485075 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7fe50c097680 con 0x7fe5180cf4d0 2024-06-29T14:37:57.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.282+0000 7fe517fff640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe504076420 0x7fe504078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.283+0000 7fe517fff640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe504076420 0x7fe504078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe5080098e0 tx=0x7fe508009060 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:57.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.286+0000 7fe514ff9640 1 -- 172.21.15.81:0/3832485075 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe50c01f020 con 0x7fe5180cf4d0 2024-06-29T14:37:57.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.463+0000 7fe51e505640 1 -- 172.21.15.81:0/3832485075 --> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe51813fd40 con 0x7fe504076420 2024-06-29T14:37:57.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.468+0000 7fe514ff9640 1 -- 172.21.15.81:0/3832485075 <== mgr.6298 v2:172.21.15.167:6832/1959721003 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252082 (secure 0 0 0) 0x7fe508009530 con 0x7fe504076420 2024-06-29T14:37:57.468 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:57.469 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:37:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 -- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe504076420 msgr2=0x7fe504078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe504076420 0x7fe504078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe5080098e0 tx=0x7fe508009060 comp rx=0 tx=0).stop 2024-06-29T14:37:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 -- 172.21.15.81:0/3832485075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5180cf4d0 msgr2=0x7fe51815d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5180cf4d0 0x7fe51815d190 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe50c00b410 tx=0x7fe50c00b8d0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 -- 172.21.15.81:0/3832485075 shutdown_connections 2024-06-29T14:37:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe504076420 0x7fe504078930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5181412f0 0x7fe51815d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5180cf4d0 0x7fe51815d190 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 --2- 172.21.15.81:0/3832485075 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe518073570 0x7fe5180a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 -- 172.21.15.81:0/3832485075 >> 172.21.15.81:0/3832485075 conn(0x7fe51806ccc0 msgr2=0x7fe51806f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:57.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 -- 172.21.15.81:0/3832485075 shutdown_connections 2024-06-29T14:37:57.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.471+0000 7fe51e505640 1 -- 172.21.15.81:0/3832485075 wait complete. 2024-06-29T14:37:57.480 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:37:56.684035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":366,"num_read_kb":935,"num_write":206,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":150,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":312076,"kb_used_data":9860,"kb_used_omap":12,"kb_used_meta":302067,"kb_avail":754662644,"statfs":{"total":773094113280,"available":772774547456,"internally_reserved":0,"allocated":10096640,"data_stored":5403336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":309317170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.603900"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107588+0000","last_change":"2024-06-29T14:37:53.111578+0000","last_active":"2024-06-29T14:37:54.107588+0000","last_peered":"2024-06-29T14:37:54.107588+0000","last_clean":"2024-06-29T14:37:54.107588+0000","last_became_active":"2024-06-29T14:37:53.111191+0000","last_became_peered":"2024-06-29T14:37:53.111191+0000","last_unstale":"2024-06-29T14:37:54.107588+0000","last_undegraded":"2024-06-29T14:37:54.107588+0000","last_fullsized":"2024-06-29T14:37:54.107588+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:34.305864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.224020+0000","last_change":"2024-06-29T14:37:53.116767+0000","last_active":"2024-06-29T14:37:54.224020+0000","last_peered":"2024-06-29T14:37:54.224020+0000","last_clean":"2024-06-29T14:37:54.224020+0000","last_became_active":"2024-06-29T14:37:53.116392+0000","last_became_peered":"2024-06-29T14:37:53.116392+0000","last_unstale":"2024-06-29T14:37:54.224020+0000","last_undegraded":"2024-06-29T14:37:54.224020+0000","last_fullsized":"2024-06-29T14:37:54.224020+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:30.643237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223482+0000","last_change":"2024-06-29T14:37:53.104489+0000","last_active":"2024-06-29T14:37:54.223482+0000","last_peered":"2024-06-29T14:37:54.223482+0000","last_clean":"2024-06-29T14:37:54.223482+0000","last_became_active":"2024-06-29T14:37:53.104192+0000","last_became_peered":"2024-06-29T14:37:53.104192+0000","last_unstale":"2024-06-29T14:37:54.223482+0000","last_undegraded":"2024-06-29T14:37:54.223482+0000","last_fullsized":"2024-06-29T14:37:54.223482+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:53:33.687273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222866+0000","last_change":"2024-06-29T14:37:53.112173+0000","last_active":"2024-06-29T14:37:54.222866+0000","last_peered":"2024-06-29T14:37:54.222866+0000","last_clean":"2024-06-29T14:37:54.222866+0000","last_became_active":"2024-06-29T14:37:53.111842+0000","last_became_peered":"2024-06-29T14:37:53.111842+0000","last_unstale":"2024-06-29T14:37:54.222866+0000","last_undegraded":"2024-06-29T14:37:54.222866+0000","last_fullsized":"2024-06-29T14:37:54.222866+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:20.219581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106519+0000","last_change":"2024-06-29T14:37:53.114238+0000","last_active":"2024-06-29T14:37:54.106519+0000","last_peered":"2024-06-29T14:37:54.106519+0000","last_clean":"2024-06-29T14:37:54.106519+0000","last_became_active":"2024-06-29T14:37:53.113937+0000","last_became_peered":"2024-06-29T14:37:53.113937+0000","last_unstale":"2024-06-29T14:37:54.106519+0000","last_undegraded":"2024-06-29T14:37:54.106519+0000","last_fullsized":"2024-06-29T14:37:54.106519+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:16:03.492357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223314+0000","last_change":"2024-06-29T14:37:53.108976+0000","last_active":"2024-06-29T14:37:54.223314+0000","last_peered":"2024-06-29T14:37:54.223314+0000","last_clean":"2024-06-29T14:37:54.223314+0000","last_became_active":"2024-06-29T14:37:53.108335+0000","last_became_peered":"2024-06-29T14:37:53.108335+0000","last_unstale":"2024-06-29T14:37:54.223314+0000","last_undegraded":"2024-06-29T14:37:54.223314+0000","last_fullsized":"2024-06-29T14:37:54.223314+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:13.401787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110342+0000","last_change":"2024-06-29T14:37:53.099815+0000","last_active":"2024-06-29T14:37:54.110342+0000","last_peered":"2024-06-29T14:37:54.110342+0000","last_clean":"2024-06-29T14:37:54.110342+0000","last_became_active":"2024-06-29T14:37:53.099520+0000","last_became_peered":"2024-06-29T14:37:53.099520+0000","last_unstale":"2024-06-29T14:37:54.110342+0000","last_undegraded":"2024-06-29T14:37:54.110342+0000","last_fullsized":"2024-06-29T14:37:54.110342+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:49.606231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223190+0000","last_change":"2024-06-29T14:37:53.109983+0000","last_active":"2024-06-29T14:37:54.223190+0000","last_peered":"2024-06-29T14:37:54.223190+0000","last_clean":"2024-06-29T14:37:54.223190+0000","last_became_active":"2024-06-29T14:37:53.108642+0000","last_became_peered":"2024-06-29T14:37:53.108642+0000","last_unstale":"2024-06-29T14:37:54.223190+0000","last_undegraded":"2024-06-29T14:37:54.223190+0000","last_fullsized":"2024-06-29T14:37:54.223190+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:16:03.492357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107320+0000","last_change":"2024-06-29T14:37:53.115824+0000","last_active":"2024-06-29T14:37:54.107320+0000","last_peered":"2024-06-29T14:37:54.107320+0000","last_clean":"2024-06-29T14:37:54.107320+0000","last_became_active":"2024-06-29T14:37:53.115485+0000","last_became_peered":"2024-06-29T14:37:53.115485+0000","last_unstale":"2024-06-29T14:37:54.107320+0000","last_undegraded":"2024-06-29T14:37:54.107320+0000","last_fullsized":"2024-06-29T14:37:54.107320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:52.622865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223156+0000","last_change":"2024-06-29T14:37:53.110102+0000","last_active":"2024-06-29T14:37:54.223156+0000","last_peered":"2024-06-29T14:37:54.223156+0000","last_clean":"2024-06-29T14:37:54.223156+0000","last_became_active":"2024-06-29T14:37:53.109724+0000","last_became_peered":"2024-06-29T14:37:53.109724+0000","last_unstale":"2024-06-29T14:37:54.223156+0000","last_undegraded":"2024-06-29T14:37:54.223156+0000","last_fullsized":"2024-06-29T14:37:54.223156+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:24.589063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.108213+0000","last_change":"2024-06-29T14:37:53.116306+0000","last_active":"2024-06-29T14:37:54.108213+0000","last_peered":"2024-06-29T14:37:54.108213+0000","last_clean":"2024-06-29T14:37:54.108213+0000","last_became_active":"2024-06-29T14:37:53.115422+0000","last_became_peered":"2024-06-29T14:37:53.115422+0000","last_unstale":"2024-06-29T14:37:54.108213+0000","last_undegraded":"2024-06-29T14:37:54.108213+0000","last_fullsized":"2024-06-29T14:37:54.108213+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:53:33.687273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223021+0000","last_change":"2024-06-29T14:37:53.109833+0000","last_active":"2024-06-29T14:37:54.223021+0000","last_peered":"2024-06-29T14:37:54.223021+0000","last_clean":"2024-06-29T14:37:54.223021+0000","last_became_active":"2024-06-29T14:37:53.109370+0000","last_became_peered":"2024-06-29T14:37:53.109370+0000","last_unstale":"2024-06-29T14:37:54.223021+0000","last_undegraded":"2024-06-29T14:37:54.223021+0000","last_fullsized":"2024-06-29T14:37:54.223021+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:38.086834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106978+0000","last_change":"2024-06-29T14:37:53.117154+0000","last_active":"2024-06-29T14:37:54.106978+0000","last_peered":"2024-06-29T14:37:54.106978+0000","last_clean":"2024-06-29T14:37:54.106978+0000","last_became_active":"2024-06-29T14:37:53.116863+0000","last_became_peered":"2024-06-29T14:37:53.116863+0000","last_unstale":"2024-06-29T14:37:54.106978+0000","last_undegraded":"2024-06-29T14:37:54.106978+0000","last_fullsized":"2024-06-29T14:37:54.106978+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:05.329593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106655+0000","last_change":"2024-06-29T14:37:53.112761+0000","last_active":"2024-06-29T14:37:54.106655+0000","last_peered":"2024-06-29T14:37:54.106655+0000","last_clean":"2024-06-29T14:37:54.106655+0000","last_became_active":"2024-06-29T14:37:53.112312+0000","last_became_peered":"2024-06-29T14:37:53.112312+0000","last_unstale":"2024-06-29T14:37:54.106655+0000","last_undegraded":"2024-06-29T14:37:54.106655+0000","last_fullsized":"2024-06-29T14:37:54.106655+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:20.219581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106937+0000","last_change":"2024-06-29T14:37:53.112281+0000","last_active":"2024-06-29T14:37:54.106937+0000","last_peered":"2024-06-29T14:37:54.106937+0000","last_clean":"2024-06-29T14:37:54.106937+0000","last_became_active":"2024-06-29T14:37:53.111959+0000","last_became_peered":"2024-06-29T14:37:53.111959+0000","last_unstale":"2024-06-29T14:37:54.106937+0000","last_undegraded":"2024-06-29T14:37:54.106937+0000","last_fullsized":"2024-06-29T14:37:54.106937+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:21.172170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222372+0000","last_change":"2024-06-29T14:37:53.113332+0000","last_active":"2024-06-29T14:37:54.222372+0000","last_peered":"2024-06-29T14:37:54.222372+0000","last_clean":"2024-06-29T14:37:54.222372+0000","last_became_active":"2024-06-29T14:37:53.112962+0000","last_became_peered":"2024-06-29T14:37:53.112962+0000","last_unstale":"2024-06-29T14:37:54.222372+0000","last_undegraded":"2024-06-29T14:37:54.222372+0000","last_fullsized":"2024-06-29T14:37:54.222372+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:39:05.831261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107440+0000","last_change":"2024-06-29T14:37:53.106745+0000","last_active":"2024-06-29T14:37:54.107440+0000","last_peered":"2024-06-29T14:37:54.107440+0000","last_clean":"2024-06-29T14:37:54.107440+0000","last_became_active":"2024-06-29T14:37:53.106415+0000","last_became_peered":"2024-06-29T14:37:53.106415+0000","last_unstale":"2024-06-29T14:37:54.107440+0000","last_undegraded":"2024-06-29T14:37:54.107440+0000","last_fullsized":"2024-06-29T14:37:54.107440+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:12.272265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223064+0000","last_change":"2024-06-29T14:37:53.112191+0000","last_active":"2024-06-29T14:37:54.223064+0000","last_peered":"2024-06-29T14:37:54.223064+0000","last_clean":"2024-06-29T14:37:54.223064+0000","last_became_active":"2024-06-29T14:37:53.111896+0000","last_became_peered":"2024-06-29T14:37:53.111896+0000","last_unstale":"2024-06-29T14:37:54.223064+0000","last_undegraded":"2024-06-29T14:37:54.223064+0000","last_fullsized":"2024-06-29T14:37:54.223064+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:42.695790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223809+0000","last_change":"2024-06-29T14:37:53.108911+0000","last_active":"2024-06-29T14:37:54.223809+0000","last_peered":"2024-06-29T14:37:54.223809+0000","last_clean":"2024-06-29T14:37:54.223809+0000","last_became_active":"2024-06-29T14:37:53.108568+0000","last_became_peered":"2024-06-29T14:37:53.108568+0000","last_unstale":"2024-06-29T14:37:54.223809+0000","last_undegraded":"2024-06-29T14:37:54.223809+0000","last_fullsized":"2024-06-29T14:37:54.223809+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:12.272265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106903+0000","last_change":"2024-06-29T14:37:53.111196+0000","last_active":"2024-06-29T14:37:54.106903+0000","last_peered":"2024-06-29T14:37:54.106903+0000","last_clean":"2024-06-29T14:37:54.106903+0000","last_became_active":"2024-06-29T14:37:53.110895+0000","last_became_peered":"2024-06-29T14:37:53.110895+0000","last_unstale":"2024-06-29T14:37:54.106903+0000","last_undegraded":"2024-06-29T14:37:54.106903+0000","last_fullsized":"2024-06-29T14:37:54.106903+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:16.484148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222753+0000","last_change":"2024-06-29T14:37:53.109750+0000","last_active":"2024-06-29T14:37:54.222753+0000","last_peered":"2024-06-29T14:37:54.222753+0000","last_clean":"2024-06-29T14:37:54.222753+0000","last_became_active":"2024-06-29T14:37:53.105224+0000","last_became_peered":"2024-06-29T14:37:53.105224+0000","last_unstale":"2024-06-29T14:37:54.222753+0000","last_undegraded":"2024-06-29T14:37:54.222753+0000","last_fullsized":"2024-06-29T14:37:54.222753+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:21.556805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222624+0000","last_change":"2024-06-29T14:37:53.109842+0000","last_active":"2024-06-29T14:37:54.222624+0000","last_peered":"2024-06-29T14:37:54.222624+0000","last_clean":"2024-06-29T14:37:54.222624+0000","last_became_active":"2024-06-29T14:37:53.109471+0000","last_became_peered":"2024-06-29T14:37:53.109471+0000","last_unstale":"2024-06-29T14:37:54.222624+0000","last_undegraded":"2024-06-29T14:37:54.222624+0000","last_fullsized":"2024-06-29T14:37:54.222624+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:00.233792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107724+0000","last_change":"2024-06-29T14:37:53.111759+0000","last_active":"2024-06-29T14:37:54.107724+0000","last_peered":"2024-06-29T14:37:54.107724+0000","last_clean":"2024-06-29T14:37:54.107724+0000","last_became_active":"2024-06-29T14:37:53.111383+0000","last_became_peered":"2024-06-29T14:37:53.111383+0000","last_unstale":"2024-06-29T14:37:54.107724+0000","last_undegraded":"2024-06-29T14:37:54.107724+0000","last_fullsized":"2024-06-29T14:37:54.107724+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:28.744782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107850+0000","last_change":"2024-06-29T14:37:53.112395+0000","last_active":"2024-06-29T14:37:54.107850+0000","last_peered":"2024-06-29T14:37:54.107850+0000","last_clean":"2024-06-29T14:37:54.107850+0000","last_became_active":"2024-06-29T14:37:53.112050+0000","last_became_peered":"2024-06-29T14:37:53.112050+0000","last_unstale":"2024-06-29T14:37:54.107850+0000","last_undegraded":"2024-06-29T14:37:54.107850+0000","last_fullsized":"2024-06-29T14:37:54.107850+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:40:28.605585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222911+0000","last_change":"2024-06-29T14:37:53.113480+0000","last_active":"2024-06-29T14:37:54.222911+0000","last_peered":"2024-06-29T14:37:54.222911+0000","last_clean":"2024-06-29T14:37:54.222911+0000","last_became_active":"2024-06-29T14:37:53.113148+0000","last_became_peered":"2024-06-29T14:37:53.113148+0000","last_unstale":"2024-06-29T14:37:54.222911+0000","last_undegraded":"2024-06-29T14:37:54.222911+0000","last_fullsized":"2024-06-29T14:37:54.222911+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:49.129124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110627+0000","last_change":"2024-06-29T14:37:53.108702+0000","last_active":"2024-06-29T14:37:54.110627+0000","last_peered":"2024-06-29T14:37:54.110627+0000","last_clean":"2024-06-29T14:37:54.110627+0000","last_became_active":"2024-06-29T14:37:53.108416+0000","last_became_peered":"2024-06-29T14:37:53.108416+0000","last_unstale":"2024-06-29T14:37:54.110627+0000","last_undegraded":"2024-06-29T14:37:54.110627+0000","last_fullsized":"2024-06-29T14:37:54.110627+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:27.329677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223511+0000","last_change":"2024-06-29T14:37:53.117067+0000","last_active":"2024-06-29T14:37:54.223511+0000","last_peered":"2024-06-29T14:37:54.223511+0000","last_clean":"2024-06-29T14:37:54.223511+0000","last_became_active":"2024-06-29T14:37:53.116704+0000","last_became_peered":"2024-06-29T14:37:53.116704+0000","last_unstale":"2024-06-29T14:37:54.223511+0000","last_undegraded":"2024-06-29T14:37:54.223511+0000","last_fullsized":"2024-06-29T14:37:54.223511+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:57.417821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222622+0000","last_change":"2024-06-29T14:37:53.106087+0000","last_active":"2024-06-29T14:37:54.222622+0000","last_peered":"2024-06-29T14:37:54.222622+0000","last_clean":"2024-06-29T14:37:54.222622+0000","last_became_active":"2024-06-29T14:37:53.105775+0000","last_became_peered":"2024-06-29T14:37:53.105775+0000","last_unstale":"2024-06-29T14:37:54.222622+0000","last_undegraded":"2024-06-29T14:37:54.222622+0000","last_fullsized":"2024-06-29T14:37:54.222622+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:39:03.121064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223259+0000","last_change":"2024-06-29T14:37:53.112320+0000","last_active":"2024-06-29T14:37:54.223259+0000","last_peered":"2024-06-29T14:37:54.223259+0000","last_clean":"2024-06-29T14:37:54.223259+0000","last_became_active":"2024-06-29T14:37:53.111998+0000","last_became_peered":"2024-06-29T14:37:53.111998+0000","last_unstale":"2024-06-29T14:37:54.223259+0000","last_undegraded":"2024-06-29T14:37:54.223259+0000","last_fullsized":"2024-06-29T14:37:54.223259+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:36:44.236489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110336+0000","last_change":"2024-06-29T14:37:53.107995+0000","last_active":"2024-06-29T14:37:54.110336+0000","last_peered":"2024-06-29T14:37:54.110336+0000","last_clean":"2024-06-29T14:37:54.110336+0000","last_became_active":"2024-06-29T14:37:53.107601+0000","last_became_peered":"2024-06-29T14:37:53.107601+0000","last_unstale":"2024-06-29T14:37:54.110336+0000","last_undegraded":"2024-06-29T14:37:54.110336+0000","last_fullsized":"2024-06-29T14:37:54.110336+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:42:50.058593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110461+0000","last_change":"2024-06-29T14:37:53.107318+0000","last_active":"2024-06-29T14:37:54.110461+0000","last_peered":"2024-06-29T14:37:54.110461+0000","last_clean":"2024-06-29T14:37:54.110461+0000","last_became_active":"2024-06-29T14:37:53.107009+0000","last_became_peered":"2024-06-29T14:37:53.107009+0000","last_unstale":"2024-06-29T14:37:54.110461+0000","last_undegraded":"2024-06-29T14:37:54.110461+0000","last_fullsized":"2024-06-29T14:37:54.110461+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:41.529557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223190+0000","last_change":"2024-06-29T14:37:53.118793+0000","last_active":"2024-06-29T14:37:54.223190+0000","last_peered":"2024-06-29T14:37:54.223190+0000","last_clean":"2024-06-29T14:37:54.223190+0000","last_became_active":"2024-06-29T14:37:53.118266+0000","last_became_peered":"2024-06-29T14:37:53.118266+0000","last_unstale":"2024-06-29T14:37:54.223190+0000","last_undegraded":"2024-06-29T14:37:54.223190+0000","last_fullsized":"2024-06-29T14:37:54.223190+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:20.219581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107660+0000","last_change":"2024-06-29T14:37:53.112129+0000","last_active":"2024-06-29T14:37:54.107660+0000","last_peered":"2024-06-29T14:37:54.107660+0000","last_clean":"2024-06-29T14:37:54.107660+0000","last_became_active":"2024-06-29T14:37:53.111566+0000","last_became_peered":"2024-06-29T14:37:53.111566+0000","last_unstale":"2024-06-29T14:37:54.107660+0000","last_undegraded":"2024-06-29T14:37:54.107660+0000","last_fullsized":"2024-06-29T14:37:54.107660+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:42:48.680994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"90'4","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.108206+0000","last_change":"2024-06-29T14:37:52.110214+0000","last_active":"2024-06-29T14:37:54.108206+0000","last_peered":"2024-06-29T14:37:54.108206+0000","last_clean":"2024-06-29T14:37:54.108206+0000","last_became_active":"2024-06-29T14:37:52.109890+0000","last_became_peered":"2024-06-29T14:37:52.109890+0000","last_unstale":"2024-06-29T14:37:54.108206+0000","last_undegraded":"2024-06-29T14:37:54.108206+0000","last_fullsized":"2024-06-29T14:37:54.108206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:47:06.667446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107544+0000","last_change":"2024-06-29T14:37:53.105885+0000","last_active":"2024-06-29T14:37:54.107544+0000","last_peered":"2024-06-29T14:37:54.107544+0000","last_clean":"2024-06-29T14:37:54.107544+0000","last_became_active":"2024-06-29T14:37:53.105537+0000","last_became_peered":"2024-06-29T14:37:53.105537+0000","last_unstale":"2024-06-29T14:37:54.107544+0000","last_undegraded":"2024-06-29T14:37:54.107544+0000","last_fullsized":"2024-06-29T14:37:54.107544+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:30.643237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"90'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107229+0000","last_change":"2024-06-29T14:37:52.101360+0000","last_active":"2024-06-29T14:37:54.107229+0000","last_peered":"2024-06-29T14:37:54.107229+0000","last_clean":"2024-06-29T14:37:54.107229+0000","last_became_active":"2024-06-29T14:37:52.100991+0000","last_became_peered":"2024-06-29T14:37:52.100991+0000","last_unstale":"2024-06-29T14:37:54.107229+0000","last_undegraded":"2024-06-29T14:37:54.107229+0000","last_fullsized":"2024-06-29T14:37:54.107229+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:12.396428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223790+0000","last_change":"2024-06-29T14:37:53.120868+0000","last_active":"2024-06-29T14:37:54.223790+0000","last_peered":"2024-06-29T14:37:54.223790+0000","last_clean":"2024-06-29T14:37:54.223790+0000","last_became_active":"2024-06-29T14:37:53.119351+0000","last_became_peered":"2024-06-29T14:37:53.119351+0000","last_unstale":"2024-06-29T14:37:54.223790+0000","last_undegraded":"2024-06-29T14:37:54.223790+0000","last_fullsized":"2024-06-29T14:37:54.223790+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:42.695790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222833+0000","last_change":"2024-06-29T14:37:52.112731+0000","last_active":"2024-06-29T14:37:54.222833+0000","last_peered":"2024-06-29T14:37:54.222833+0000","last_clean":"2024-06-29T14:37:54.222833+0000","last_became_active":"2024-06-29T14:37:52.112436+0000","last_became_peered":"2024-06-29T14:37:52.112436+0000","last_unstale":"2024-06-29T14:37:54.222833+0000","last_undegraded":"2024-06-29T14:37:54.222833+0000","last_fullsized":"2024-06-29T14:37:54.222833+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:59:38.479274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.108134+0000","last_change":"2024-06-29T14:37:53.116910+0000","last_active":"2024-06-29T14:37:54.108134+0000","last_peered":"2024-06-29T14:37:54.108134+0000","last_clean":"2024-06-29T14:37:54.108134+0000","last_became_active":"2024-06-29T14:37:53.115676+0000","last_became_peered":"2024-06-29T14:37:53.115676+0000","last_unstale":"2024-06-29T14:37:54.108134+0000","last_undegraded":"2024-06-29T14:37:54.108134+0000","last_fullsized":"2024-06-29T14:37:54.108134+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:27.808263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223917+0000","last_change":"2024-06-29T14:37:52.112031+0000","last_active":"2024-06-29T14:37:54.223917+0000","last_peered":"2024-06-29T14:37:54.223917+0000","last_clean":"2024-06-29T14:37:54.223917+0000","last_became_active":"2024-06-29T14:37:52.111699+0000","last_became_peered":"2024-06-29T14:37:52.111699+0000","last_unstale":"2024-06-29T14:37:54.223917+0000","last_undegraded":"2024-06-29T14:37:54.223917+0000","last_fullsized":"2024-06-29T14:37:54.223917+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:51:28.495591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110664+0000","last_change":"2024-06-29T14:37:53.101327+0000","last_active":"2024-06-29T14:37:54.110664+0000","last_peered":"2024-06-29T14:37:54.110664+0000","last_clean":"2024-06-29T14:37:54.110664+0000","last_became_active":"2024-06-29T14:37:53.100903+0000","last_became_peered":"2024-06-29T14:37:53.100903+0000","last_unstale":"2024-06-29T14:37:54.110664+0000","last_undegraded":"2024-06-29T14:37:54.110664+0000","last_fullsized":"2024-06-29T14:37:54.110664+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:50:26.304625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"90'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223399+0000","last_change":"2024-06-29T14:37:52.112889+0000","last_active":"2024-06-29T14:37:54.223399+0000","last_peered":"2024-06-29T14:37:54.223399+0000","last_clean":"2024-06-29T14:37:54.223399+0000","last_became_active":"2024-06-29T14:37:52.112544+0000","last_became_peered":"2024-06-29T14:37:52.112544+0000","last_unstale":"2024-06-29T14:37:54.223399+0000","last_undegraded":"2024-06-29T14:37:54.223399+0000","last_fullsized":"2024-06-29T14:37:54.223399+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:22.505425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222727+0000","last_change":"2024-06-29T14:37:53.112364+0000","last_active":"2024-06-29T14:37:54.222727+0000","last_peered":"2024-06-29T14:37:54.222727+0000","last_clean":"2024-06-29T14:37:54.222727+0000","last_became_active":"2024-06-29T14:37:53.112023+0000","last_became_peered":"2024-06-29T14:37:53.112023+0000","last_unstale":"2024-06-29T14:37:54.222727+0000","last_undegraded":"2024-06-29T14:37:54.222727+0000","last_fullsized":"2024-06-29T14:37:54.222727+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:33.972067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"90'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223399+0000","last_change":"2024-06-29T14:37:52.109509+0000","last_active":"2024-06-29T14:37:54.223399+0000","last_peered":"2024-06-29T14:37:54.223399+0000","last_clean":"2024-06-29T14:37:54.223399+0000","last_became_active":"2024-06-29T14:37:52.109194+0000","last_became_peered":"2024-06-29T14:37:52.109194+0000","last_unstale":"2024-06-29T14:37:54.223399+0000","last_undegraded":"2024-06-29T14:37:54.223399+0000","last_fullsized":"2024-06-29T14:37:54.223399+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:21:32.361654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106193+0000","last_change":"2024-06-29T14:37:53.103525+0000","last_active":"2024-06-29T14:37:54.106193+0000","last_peered":"2024-06-29T14:37:54.106193+0000","last_clean":"2024-06-29T14:37:54.106193+0000","last_became_active":"2024-06-29T14:37:53.103214+0000","last_became_peered":"2024-06-29T14:37:53.103214+0000","last_unstale":"2024-06-29T14:37:54.106193+0000","last_undegraded":"2024-06-29T14:37:54.106193+0000","last_fullsized":"2024-06-29T14:37:54.106193+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:26.164210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222781+0000","last_change":"2024-06-29T14:37:52.109066+0000","last_active":"2024-06-29T14:37:54.222781+0000","last_peered":"2024-06-29T14:37:54.222781+0000","last_clean":"2024-06-29T14:37:54.222781+0000","last_became_active":"2024-06-29T14:37:52.108736+0000","last_became_peered":"2024-06-29T14:37:52.108736+0000","last_unstale":"2024-06-29T14:37:54.222781+0000","last_undegraded":"2024-06-29T14:37:54.222781+0000","last_fullsized":"2024-06-29T14:37:54.222781+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:41:09.719020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107115+0000","last_change":"2024-06-29T14:37:53.118647+0000","last_active":"2024-06-29T14:37:54.107115+0000","last_peered":"2024-06-29T14:37:54.107115+0000","last_clean":"2024-06-29T14:37:54.107115+0000","last_became_active":"2024-06-29T14:37:53.117679+0000","last_became_peered":"2024-06-29T14:37:53.117679+0000","last_unstale":"2024-06-29T14:37:54.107115+0000","last_undegraded":"2024-06-29T14:37:54.107115+0000","last_fullsized":"2024-06-29T14:37:54.107115+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:58.594346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106185+0000","last_change":"2024-06-29T14:37:52.102090+0000","last_active":"2024-06-29T14:37:54.106185+0000","last_peered":"2024-06-29T14:37:54.106185+0000","last_clean":"2024-06-29T14:37:54.106185+0000","last_became_active":"2024-06-29T14:37:52.101617+0000","last_became_peered":"2024-06-29T14:37:52.101617+0000","last_unstale":"2024-06-29T14:37:54.106185+0000","last_undegraded":"2024-06-29T14:37:54.106185+0000","last_fullsized":"2024-06-29T14:37:54.106185+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:05.193575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110626+0000","last_change":"2024-06-29T14:37:53.106169+0000","last_active":"2024-06-29T14:37:54.110626+0000","last_peered":"2024-06-29T14:37:54.110626+0000","last_clean":"2024-06-29T14:37:54.110626+0000","last_became_active":"2024-06-29T14:37:53.105816+0000","last_became_peered":"2024-06-29T14:37:53.105816+0000","last_unstale":"2024-06-29T14:37:54.110626+0000","last_undegraded":"2024-06-29T14:37:54.110626+0000","last_fullsized":"2024-06-29T14:37:54.110626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:19.515086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222842+0000","last_change":"2024-06-29T14:37:52.111657+0000","last_active":"2024-06-29T14:37:54.222842+0000","last_peered":"2024-06-29T14:37:54.222842+0000","last_clean":"2024-06-29T14:37:54.222842+0000","last_became_active":"2024-06-29T14:37:52.111212+0000","last_became_peered":"2024-06-29T14:37:52.111212+0000","last_unstale":"2024-06-29T14:37:54.222842+0000","last_undegraded":"2024-06-29T14:37:54.222842+0000","last_fullsized":"2024-06-29T14:37:54.222842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:20:27.663108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107896+0000","last_change":"2024-06-29T14:37:53.118101+0000","last_active":"2024-06-29T14:37:54.107896+0000","last_peered":"2024-06-29T14:37:54.107896+0000","last_clean":"2024-06-29T14:37:54.107896+0000","last_became_active":"2024-06-29T14:37:53.117761+0000","last_became_peered":"2024-06-29T14:37:53.117761+0000","last_unstale":"2024-06-29T14:37:54.107896+0000","last_undegraded":"2024-06-29T14:37:54.107896+0000","last_fullsized":"2024-06-29T14:37:54.107896+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:36:44.236489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.224042+0000","last_change":"2024-06-29T14:37:52.107836+0000","last_active":"2024-06-29T14:37:54.224042+0000","last_peered":"2024-06-29T14:37:54.224042+0000","last_clean":"2024-06-29T14:37:54.224042+0000","last_became_active":"2024-06-29T14:37:52.107495+0000","last_became_peered":"2024-06-29T14:37:52.107495+0000","last_unstale":"2024-06-29T14:37:54.224042+0000","last_undegraded":"2024-06-29T14:37:54.224042+0000","last_fullsized":"2024-06-29T14:37:54.224042+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:47:55.646551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223794+0000","last_change":"2024-06-29T14:37:53.111413+0000","last_active":"2024-06-29T14:37:54.223794+0000","last_peered":"2024-06-29T14:37:54.223794+0000","last_clean":"2024-06-29T14:37:54.223794+0000","last_became_active":"2024-06-29T14:37:53.109137+0000","last_became_peered":"2024-06-29T14:37:53.109137+0000","last_unstale":"2024-06-29T14:37:54.223794+0000","last_undegraded":"2024-06-29T14:37:54.223794+0000","last_fullsized":"2024-06-29T14:37:54.223794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:52.622865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"90'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223561+0000","last_change":"2024-06-29T14:37:52.114208+0000","last_active":"2024-06-29T14:37:54.223561+0000","last_peered":"2024-06-29T14:37:54.223561+0000","last_clean":"2024-06-29T14:37:54.223561+0000","last_became_active":"2024-06-29T14:37:52.113884+0000","last_became_peered":"2024-06-29T14:37:52.113884+0000","last_unstale":"2024-06-29T14:37:54.223561+0000","last_undegraded":"2024-06-29T14:37:54.223561+0000","last_fullsized":"2024-06-29T14:37:54.223561+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:37.912012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107137+0000","last_change":"2024-06-29T14:37:53.109776+0000","last_active":"2024-06-29T14:37:54.107137+0000","last_peered":"2024-06-29T14:37:54.107137+0000","last_clean":"2024-06-29T14:37:54.107137+0000","last_became_active":"2024-06-29T14:37:53.109474+0000","last_became_peered":"2024-06-29T14:37:53.109474+0000","last_unstale":"2024-06-29T14:37:54.107137+0000","last_undegraded":"2024-06-29T14:37:54.107137+0000","last_fullsized":"2024-06-29T14:37:54.107137+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:38.086834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"90'6","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223128+0000","last_change":"2024-06-29T14:37:52.098924+0000","last_active":"2024-06-29T14:37:54.223128+0000","last_peered":"2024-06-29T14:37:54.223128+0000","last_clean":"2024-06-29T14:37:54.223128+0000","last_became_active":"2024-06-29T14:37:52.098646+0000","last_became_peered":"2024-06-29T14:37:52.098646+0000","last_unstale":"2024-06-29T14:37:54.223128+0000","last_undegraded":"2024-06-29T14:37:54.223128+0000","last_fullsized":"2024-06-29T14:37:54.223128+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:08.728523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107358+0000","last_change":"2024-06-29T14:37:53.111620+0000","last_active":"2024-06-29T14:37:54.107358+0000","last_peered":"2024-06-29T14:37:54.107358+0000","last_clean":"2024-06-29T14:37:54.107358+0000","last_became_active":"2024-06-29T14:37:53.111239+0000","last_became_peered":"2024-06-29T14:37:53.111239+0000","last_unstale":"2024-06-29T14:37:54.107358+0000","last_undegraded":"2024-06-29T14:37:54.107358+0000","last_fullsized":"2024-06-29T14:37:54.107358+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:00:58.319938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222958+0000","last_change":"2024-06-29T14:37:52.104019+0000","last_active":"2024-06-29T14:37:54.222958+0000","last_peered":"2024-06-29T14:37:54.222958+0000","last_clean":"2024-06-29T14:37:54.222958+0000","last_became_active":"2024-06-29T14:37:52.103581+0000","last_became_peered":"2024-06-29T14:37:52.103581+0000","last_unstale":"2024-06-29T14:37:54.222958+0000","last_undegraded":"2024-06-29T14:37:54.222958+0000","last_fullsized":"2024-06-29T14:37:54.222958+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:38.499417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110580+0000","last_change":"2024-06-29T14:37:53.103766+0000","last_active":"2024-06-29T14:37:54.110580+0000","last_peered":"2024-06-29T14:37:54.110580+0000","last_clean":"2024-06-29T14:37:54.110580+0000","last_became_active":"2024-06-29T14:37:53.103339+0000","last_became_peered":"2024-06-29T14:37:53.103339+0000","last_unstale":"2024-06-29T14:37:54.110580+0000","last_undegraded":"2024-06-29T14:37:54.110580+0000","last_fullsized":"2024-06-29T14:37:54.110580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:23.224376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223675+0000","last_change":"2024-06-29T14:37:52.106538+0000","last_active":"2024-06-29T14:37:54.223675+0000","last_peered":"2024-06-29T14:37:54.223675+0000","last_clean":"2024-06-29T14:37:54.223675+0000","last_became_active":"2024-06-29T14:37:52.106146+0000","last_became_peered":"2024-06-29T14:37:52.106146+0000","last_unstale":"2024-06-29T14:37:54.223675+0000","last_undegraded":"2024-06-29T14:37:54.223675+0000","last_fullsized":"2024-06-29T14:37:54.223675+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:45.083202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106691+0000","last_change":"2024-06-29T14:37:53.118627+0000","last_active":"2024-06-29T14:37:54.106691+0000","last_peered":"2024-06-29T14:37:54.106691+0000","last_clean":"2024-06-29T14:37:54.106691+0000","last_became_active":"2024-06-29T14:37:53.118062+0000","last_became_peered":"2024-06-29T14:37:53.118062+0000","last_unstale":"2024-06-29T14:37:54.106691+0000","last_undegraded":"2024-06-29T14:37:54.106691+0000","last_fullsized":"2024-06-29T14:37:54.106691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:24.041354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110233+0000","last_change":"2024-06-29T14:37:52.103863+0000","last_active":"2024-06-29T14:37:54.110233+0000","last_peered":"2024-06-29T14:37:54.110233+0000","last_clean":"2024-06-29T14:37:54.110233+0000","last_became_active":"2024-06-29T14:37:52.103546+0000","last_became_peered":"2024-06-29T14:37:52.103546+0000","last_unstale":"2024-06-29T14:37:54.110233+0000","last_undegraded":"2024-06-29T14:37:54.110233+0000","last_fullsized":"2024-06-29T14:37:54.110233+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:43.095421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106484+0000","last_change":"2024-06-29T14:37:53.116125+0000","last_active":"2024-06-29T14:37:54.106484+0000","last_peered":"2024-06-29T14:37:54.106484+0000","last_clean":"2024-06-29T14:37:54.106484+0000","last_became_active":"2024-06-29T14:37:53.115798+0000","last_became_peered":"2024-06-29T14:37:53.115798+0000","last_unstale":"2024-06-29T14:37:54.106484+0000","last_undegraded":"2024-06-29T14:37:54.106484+0000","last_fullsized":"2024-06-29T14:37:54.106484+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:59.235978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"90'8","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222374+0000","last_change":"2024-06-29T14:37:52.109753+0000","last_active":"2024-06-29T14:37:54.222374+0000","last_peered":"2024-06-29T14:37:54.222374+0000","last_clean":"2024-06-29T14:37:54.222374+0000","last_became_active":"2024-06-29T14:37:52.108953+0000","last_became_peered":"2024-06-29T14:37:52.108953+0000","last_unstale":"2024-06-29T14:37:54.222374+0000","last_undegraded":"2024-06-29T14:37:54.222374+0000","last_fullsized":"2024-06-29T14:37:54.222374+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:34:35.163677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223338+0000","last_change":"2024-06-29T14:37:53.100142+0000","last_active":"2024-06-29T14:37:54.223338+0000","last_peered":"2024-06-29T14:37:54.223338+0000","last_clean":"2024-06-29T14:37:54.223338+0000","last_became_active":"2024-06-29T14:37:53.099794+0000","last_became_peered":"2024-06-29T14:37:53.099794+0000","last_unstale":"2024-06-29T14:37:54.223338+0000","last_undegraded":"2024-06-29T14:37:54.223338+0000","last_fullsized":"2024-06-29T14:37:54.223338+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:22.100876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"90'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223916+0000","last_change":"2024-06-29T14:37:52.111533+0000","last_active":"2024-06-29T14:37:54.223916+0000","last_peered":"2024-06-29T14:37:54.223916+0000","last_clean":"2024-06-29T14:37:54.223916+0000","last_became_active":"2024-06-29T14:37:52.110801+0000","last_became_peered":"2024-06-29T14:37:52.110801+0000","last_unstale":"2024-06-29T14:37:54.223916+0000","last_undegraded":"2024-06-29T14:37:54.223916+0000","last_fullsized":"2024-06-29T14:37:54.223916+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:15.045019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.108084+0000","last_change":"2024-06-29T14:37:53.115001+0000","last_active":"2024-06-29T14:37:54.108084+0000","last_peered":"2024-06-29T14:37:54.108084+0000","last_clean":"2024-06-29T14:37:54.108084+0000","last_became_active":"2024-06-29T14:37:53.114676+0000","last_became_peered":"2024-06-29T14:37:53.114676+0000","last_unstale":"2024-06-29T14:37:54.108084+0000","last_undegraded":"2024-06-29T14:37:54.108084+0000","last_fullsized":"2024-06-29T14:37:54.108084+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:26.272219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223474+0000","last_change":"2024-06-29T14:37:52.115847+0000","last_active":"2024-06-29T14:37:54.223474+0000","last_peered":"2024-06-29T14:37:54.223474+0000","last_clean":"2024-06-29T14:37:54.223474+0000","last_became_active":"2024-06-29T14:37:52.115526+0000","last_became_peered":"2024-06-29T14:37:52.115526+0000","last_unstale":"2024-06-29T14:37:54.223474+0000","last_undegraded":"2024-06-29T14:37:54.223474+0000","last_fullsized":"2024-06-29T14:37:54.223474+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:26.133687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223077+0000","last_change":"2024-06-29T14:37:53.104206+0000","last_active":"2024-06-29T14:37:54.223077+0000","last_peered":"2024-06-29T14:37:54.223077+0000","last_clean":"2024-06-29T14:37:54.223077+0000","last_became_active":"2024-06-29T14:37:53.103873+0000","last_became_peered":"2024-06-29T14:37:53.103873+0000","last_unstale":"2024-06-29T14:37:54.223077+0000","last_undegraded":"2024-06-29T14:37:54.223077+0000","last_fullsized":"2024-06-29T14:37:54.223077+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:52.622865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"90'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223782+0000","last_change":"2024-06-29T14:37:52.112758+0000","last_active":"2024-06-29T14:37:54.223782+0000","last_peered":"2024-06-29T14:37:54.223782+0000","last_clean":"2024-06-29T14:37:54.223782+0000","last_became_active":"2024-06-29T14:37:52.112438+0000","last_became_peered":"2024-06-29T14:37:52.112438+0000","last_unstale":"2024-06-29T14:37:54.223782+0000","last_undegraded":"2024-06-29T14:37:54.223782+0000","last_fullsized":"2024-06-29T14:37:54.223782+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:25:52.179698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107320+0000","last_change":"2024-06-29T14:37:53.112345+0000","last_active":"2024-06-29T14:37:54.107320+0000","last_peered":"2024-06-29T14:37:54.107320+0000","last_clean":"2024-06-29T14:37:54.107320+0000","last_became_active":"2024-06-29T14:37:53.111719+0000","last_became_peered":"2024-06-29T14:37:53.111719+0000","last_unstale":"2024-06-29T14:37:54.107320+0000","last_undegraded":"2024-06-29T14:37:54.107320+0000","last_fullsized":"2024-06-29T14:37:54.107320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:42.695790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107819+0000","last_change":"2024-06-29T14:37:52.113119+0000","last_active":"2024-06-29T14:37:54.107819+0000","last_peered":"2024-06-29T14:37:54.107819+0000","last_clean":"2024-06-29T14:37:54.107819+0000","last_became_active":"2024-06-29T14:37:52.112787+0000","last_became_peered":"2024-06-29T14:37:52.112787+0000","last_unstale":"2024-06-29T14:37:54.107819+0000","last_undegraded":"2024-06-29T14:37:54.107819+0000","last_fullsized":"2024-06-29T14:37:54.107819+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:23.559283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223456+0000","last_change":"2024-06-29T14:37:53.103357+0000","last_active":"2024-06-29T14:37:54.223456+0000","last_peered":"2024-06-29T14:37:54.223456+0000","last_clean":"2024-06-29T14:37:54.223456+0000","last_became_active":"2024-06-29T14:37:53.103036+0000","last_became_peered":"2024-06-29T14:37:53.103036+0000","last_unstale":"2024-06-29T14:37:54.223456+0000","last_undegraded":"2024-06-29T14:37:54.223456+0000","last_fullsized":"2024-06-29T14:37:54.223456+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:27.808263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223551+0000","last_change":"2024-06-29T14:37:52.108612+0000","last_active":"2024-06-29T14:37:54.223551+0000","last_peered":"2024-06-29T14:37:54.223551+0000","last_clean":"2024-06-29T14:37:54.223551+0000","last_became_active":"2024-06-29T14:37:52.108264+0000","last_became_peered":"2024-06-29T14:37:52.108264+0000","last_unstale":"2024-06-29T14:37:54.223551+0000","last_undegraded":"2024-06-29T14:37:54.223551+0000","last_fullsized":"2024-06-29T14:37:54.223551+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:16.748536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223183+0000","last_change":"2024-06-29T14:37:53.110153+0000","last_active":"2024-06-29T14:37:54.223183+0000","last_peered":"2024-06-29T14:37:54.223183+0000","last_clean":"2024-06-29T14:37:54.223183+0000","last_became_active":"2024-06-29T14:37:53.109833+0000","last_became_peered":"2024-06-29T14:37:53.109833+0000","last_unstale":"2024-06-29T14:37:54.223183+0000","last_undegraded":"2024-06-29T14:37:54.223183+0000","last_fullsized":"2024-06-29T14:37:54.223183+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:56:29.828443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110510+0000","last_change":"2024-06-29T14:37:52.104682+0000","last_active":"2024-06-29T14:37:54.110510+0000","last_peered":"2024-06-29T14:37:54.110510+0000","last_clean":"2024-06-29T14:37:54.110510+0000","last_became_active":"2024-06-29T14:37:52.104388+0000","last_became_peered":"2024-06-29T14:37:52.104388+0000","last_unstale":"2024-06-29T14:37:54.110510+0000","last_undegraded":"2024-06-29T14:37:54.110510+0000","last_fullsized":"2024-06-29T14:37:54.110510+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:27.563821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223348+0000","last_change":"2024-06-29T14:37:53.109361+0000","last_active":"2024-06-29T14:37:54.223348+0000","last_peered":"2024-06-29T14:37:54.223348+0000","last_clean":"2024-06-29T14:37:54.223348+0000","last_became_active":"2024-06-29T14:37:53.109028+0000","last_became_peered":"2024-06-29T14:37:53.109028+0000","last_unstale":"2024-06-29T14:37:54.223348+0000","last_undegraded":"2024-06-29T14:37:54.223348+0000","last_fullsized":"2024-06-29T14:37:54.223348+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:26.272219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106636+0000","last_change":"2024-06-29T14:37:52.104697+0000","last_active":"2024-06-29T14:37:54.106636+0000","last_peered":"2024-06-29T14:37:54.106636+0000","last_clean":"2024-06-29T14:37:54.106636+0000","last_became_active":"2024-06-29T14:37:52.104373+0000","last_became_peered":"2024-06-29T14:37:52.104373+0000","last_unstale":"2024-06-29T14:37:54.106636+0000","last_undegraded":"2024-06-29T14:37:54.106636+0000","last_fullsized":"2024-06-29T14:37:54.106636+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:19.214222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107059+0000","last_change":"2024-06-29T14:37:53.113192+0000","last_active":"2024-06-29T14:37:54.107059+0000","last_peered":"2024-06-29T14:37:54.107059+0000","last_clean":"2024-06-29T14:37:54.107059+0000","last_became_active":"2024-06-29T14:37:53.112869+0000","last_became_peered":"2024-06-29T14:37:53.112869+0000","last_unstale":"2024-06-29T14:37:54.107059+0000","last_undegraded":"2024-06-29T14:37:54.107059+0000","last_fullsized":"2024-06-29T14:37:54.107059+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:09:10.493935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223939+0000","last_change":"2024-06-29T14:37:52.111123+0000","last_active":"2024-06-29T14:37:54.223939+0000","last_peered":"2024-06-29T14:37:54.223939+0000","last_clean":"2024-06-29T14:37:54.223939+0000","last_became_active":"2024-06-29T14:37:52.110622+0000","last_became_peered":"2024-06-29T14:37:52.110622+0000","last_unstale":"2024-06-29T14:37:54.223939+0000","last_undegraded":"2024-06-29T14:37:54.223939+0000","last_fullsized":"2024-06-29T14:37:54.223939+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:53:00.292129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.110613+0000","last_change":"2024-06-29T14:37:53.102032+0000","last_active":"2024-06-29T14:37:54.110613+0000","last_peered":"2024-06-29T14:37:54.110613+0000","last_clean":"2024-06-29T14:37:54.110613+0000","last_became_active":"2024-06-29T14:37:53.101733+0000","last_became_peered":"2024-06-29T14:37:53.101733+0000","last_unstale":"2024-06-29T14:37:54.110613+0000","last_undegraded":"2024-06-29T14:37:54.110613+0000","last_fullsized":"2024-06-29T14:37:54.110613+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:19.886753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"91'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.111935+0000","last_change":"2024-06-29T14:37:52.103694+0000","last_active":"2024-06-29T14:37:54.111935+0000","last_peered":"2024-06-29T14:37:54.111935+0000","last_clean":"2024-06-29T14:37:54.111935+0000","last_became_active":"2024-06-29T14:37:52.103319+0000","last_became_peered":"2024-06-29T14:37:52.103319+0000","last_unstale":"2024-06-29T14:37:54.111935+0000","last_undegraded":"2024-06-29T14:37:54.111935+0000","last_fullsized":"2024-06-29T14:37:54.111935+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:40:53.466961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107762+0000","last_change":"2024-06-29T14:37:53.116211+0000","last_active":"2024-06-29T14:37:54.107762+0000","last_peered":"2024-06-29T14:37:54.107762+0000","last_clean":"2024-06-29T14:37:54.107762+0000","last_became_active":"2024-06-29T14:37:53.115841+0000","last_became_peered":"2024-06-29T14:37:53.115841+0000","last_unstale":"2024-06-29T14:37:54.107762+0000","last_undegraded":"2024-06-29T14:37:54.107762+0000","last_fullsized":"2024-06-29T14:37:54.107762+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:56:29.828443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107377+0000","last_change":"2024-06-29T14:37:52.102785+0000","last_active":"2024-06-29T14:37:54.107377+0000","last_peered":"2024-06-29T14:37:54.107377+0000","last_clean":"2024-06-29T14:37:54.107377+0000","last_became_active":"2024-06-29T14:37:52.102461+0000","last_became_peered":"2024-06-29T14:37:52.102461+0000","last_unstale":"2024-06-29T14:37:54.107377+0000","last_undegraded":"2024-06-29T14:37:54.107377+0000","last_fullsized":"2024-06-29T14:37:54.107377+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:16.748536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.108310+0000","last_change":"2024-06-29T14:37:53.116666+0000","last_active":"2024-06-29T14:37:54.108310+0000","last_peered":"2024-06-29T14:37:54.108310+0000","last_clean":"2024-06-29T14:37:54.108310+0000","last_became_active":"2024-06-29T14:37:53.115938+0000","last_became_peered":"2024-06-29T14:37:53.115938+0000","last_unstale":"2024-06-29T14:37:54.108310+0000","last_undegraded":"2024-06-29T14:37:54.108310+0000","last_fullsized":"2024-06-29T14:37:54.108310+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:57.417821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107019+0000","last_change":"2024-06-29T14:37:52.109330+0000","last_active":"2024-06-29T14:37:54.107019+0000","last_peered":"2024-06-29T14:37:54.107019+0000","last_clean":"2024-06-29T14:37:54.107019+0000","last_became_active":"2024-06-29T14:37:52.109025+0000","last_became_peered":"2024-06-29T14:37:52.109025+0000","last_unstale":"2024-06-29T14:37:54.107019+0000","last_undegraded":"2024-06-29T14:37:54.107019+0000","last_fullsized":"2024-06-29T14:37:54.107019+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:16:02.486998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.106839+0000","last_change":"2024-06-29T14:37:53.116204+0000","last_active":"2024-06-29T14:37:54.106839+0000","last_peered":"2024-06-29T14:37:54.106839+0000","last_clean":"2024-06-29T14:37:54.106839+0000","last_became_active":"2024-06-29T14:37:53.115875+0000","last_became_peered":"2024-06-29T14:37:53.115875+0000","last_unstale":"2024-06-29T14:37:54.106839+0000","last_undegraded":"2024-06-29T14:37:54.106839+0000","last_fullsized":"2024-06-29T14:37:54.106839+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:13.401787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107371+0000","last_change":"2024-06-29T14:37:52.106510+0000","last_active":"2024-06-29T14:37:54.107371+0000","last_peered":"2024-06-29T14:37:54.107371+0000","last_clean":"2024-06-29T14:37:54.107371+0000","last_became_active":"2024-06-29T14:37:52.106162+0000","last_became_peered":"2024-06-29T14:37:52.106162+0000","last_unstale":"2024-06-29T14:37:54.107371+0000","last_undegraded":"2024-06-29T14:37:54.107371+0000","last_fullsized":"2024-06-29T14:37:54.107371+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:15.045019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223355+0000","last_change":"2024-06-29T14:37:53.119291+0000","last_active":"2024-06-29T14:37:54.223355+0000","last_peered":"2024-06-29T14:37:54.223355+0000","last_clean":"2024-06-29T14:37:54.223355+0000","last_became_active":"2024-06-29T14:37:53.118604+0000","last_became_peered":"2024-06-29T14:37:53.118604+0000","last_unstale":"2024-06-29T14:37:54.223355+0000","last_undegraded":"2024-06-29T14:37:54.223355+0000","last_fullsized":"2024-06-29T14:37:54.223355+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:23:38.086834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222704+0000","last_change":"2024-06-29T14:37:52.110474+0000","last_active":"2024-06-29T14:37:54.222704+0000","last_peered":"2024-06-29T14:37:54.222704+0000","last_clean":"2024-06-29T14:37:54.222704+0000","last_became_active":"2024-06-29T14:37:52.110163+0000","last_became_peered":"2024-06-29T14:37:52.110163+0000","last_unstale":"2024-06-29T14:37:54.222704+0000","last_undegraded":"2024-06-29T14:37:54.222704+0000","last_fullsized":"2024-06-29T14:37:54.222704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:38.025126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222932+0000","last_change":"2024-06-29T14:37:53.105314+0000","last_active":"2024-06-29T14:37:54.222932+0000","last_peered":"2024-06-29T14:37:54.222932+0000","last_clean":"2024-06-29T14:37:54.222932+0000","last_became_active":"2024-06-29T14:37:53.105004+0000","last_became_peered":"2024-06-29T14:37:53.105004+0000","last_unstale":"2024-06-29T14:37:54.222932+0000","last_undegraded":"2024-06-29T14:37:54.222932+0000","last_fullsized":"2024-06-29T14:37:54.222932+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:13.401787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.222306+0000","last_change":"2024-06-29T14:37:52.113877+0000","last_active":"2024-06-29T14:37:54.222306+0000","last_peered":"2024-06-29T14:37:54.222306+0000","last_clean":"2024-06-29T14:37:54.222306+0000","last_became_active":"2024-06-29T14:37:52.113535+0000","last_became_peered":"2024-06-29T14:37:52.113535+0000","last_unstale":"2024-06-29T14:37:54.222306+0000","last_undegraded":"2024-06-29T14:37:54.222306+0000","last_fullsized":"2024-06-29T14:37:54.222306+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:30:21.560571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223172+0000","last_change":"2024-06-29T14:37:53.117832+0000","last_active":"2024-06-29T14:37:54.223172+0000","last_peered":"2024-06-29T14:37:54.223172+0000","last_clean":"2024-06-29T14:37:54.223172+0000","last_became_active":"2024-06-29T14:37:53.116160+0000","last_became_peered":"2024-06-29T14:37:53.116160+0000","last_unstale":"2024-06-29T14:37:54.223172+0000","last_undegraded":"2024-06-29T14:37:54.223172+0000","last_fullsized":"2024-06-29T14:37:54.223172+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:59.235978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223329+0000","last_change":"2024-06-29T14:37:52.112695+0000","last_active":"2024-06-29T14:37:54.223329+0000","last_peered":"2024-06-29T14:37:54.223329+0000","last_clean":"2024-06-29T14:37:54.223329+0000","last_became_active":"2024-06-29T14:37:52.112377+0000","last_became_peered":"2024-06-29T14:37:52.112377+0000","last_unstale":"2024-06-29T14:37:54.223329+0000","last_undegraded":"2024-06-29T14:37:54.223329+0000","last_fullsized":"2024-06-29T14:37:54.223329+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:05.193575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223102+0000","last_change":"2024-06-29T14:37:53.111427+0000","last_active":"2024-06-29T14:37:54.223102+0000","last_peered":"2024-06-29T14:37:54.223102+0000","last_clean":"2024-06-29T14:37:54.223102+0000","last_became_active":"2024-06-29T14:37:53.111124+0000","last_became_peered":"2024-06-29T14:37:53.111124+0000","last_unstale":"2024-06-29T14:37:54.223102+0000","last_undegraded":"2024-06-29T14:37:54.223102+0000","last_fullsized":"2024-06-29T14:37:54.223102+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:52.078774+0000","last_clean_scrub_stamp":"2024-06-29T14:37:52.078774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:28.744782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"90'2","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.223511+0000","last_change":"2024-06-29T14:37:52.111603+0000","last_active":"2024-06-29T14:37:54.223511+0000","last_peered":"2024-06-29T14:37:54.223511+0000","last_clean":"2024-06-29T14:37:54.223511+0000","last_became_active":"2024-06-29T14:37:52.111262+0000","last_became_peered":"2024-06-29T14:37:52.111262+0000","last_unstale":"2024-06-29T14:37:54.223511+0000","last_undegraded":"2024-06-29T14:37:54.223511+0000","last_fullsized":"2024-06-29T14:37:54.223511+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:37:51.073415+0000","last_clean_scrub_stamp":"2024-06-29T14:37:51.073415+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:02:54.388993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"80'107","reported_seq":376,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-06-29T14:37:54.107238+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:37:54.107238+0000","last_peered":"2024-06-29T14:37:54.107238+0000","last_clean":"2024-06-29T14:37:54.107238+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:37:54.107238+0000","last_undegraded":"2024-06-29T14:37:54.107238+0000","last_fullsized":"2024-06-29T14:37:54.107238+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":366,"num_read_kb":935,"num_write":161,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":366,"num_read_kb":935,"num_write":161,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575036,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1699840,"data_stored":1114224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575037,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1118208,"data_stored":529704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575035,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1114112,"data_stored":529702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58624,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313216,"statfs":{"total":96636764160,"available":96576733184,"internally_reserved":0,"allocated":1101824,"data_stored":523310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575036,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36420,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335420,"statfs":{"total":96636764160,"available":96599470080,"internally_reserved":0,"allocated":1105920,"data_stored":523352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575036,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36440,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335400,"statfs":{"total":96636764160,"available":96599449600,"internally_reserved":0,"allocated":1122304,"data_stored":531224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575035,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50340,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321500,"statfs":{"total":96636764160,"available":96585216000,"internally_reserved":0,"allocated":1130496,"data_stored":531840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575036,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":1119980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:37:57.484 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:37:57.485 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:37:57.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.644+0000 7f2913b63640 1 -- 172.21.15.81:0/2395722518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c06ef90 msgr2=0x7f290c06f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.644+0000 7f2913b63640 1 --2- 172.21.15.81:0/2395722518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c06ef90 0x7f290c06f3a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f290c071e60 tx=0x7f290002a560 comp rx=0 tx=0).stop 2024-06-29T14:37:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.644+0000 7f2913b63640 1 -- 172.21.15.81:0/2395722518 shutdown_connections 2024-06-29T14:37:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.644+0000 7f2913b63640 1 --2- 172.21.15.81:0/2395722518 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c0ce1d0 0x7f290c0ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.644+0000 7f2913b63640 1 --2- 172.21.15.81:0/2395722518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c06ef90 0x7f290c06f3a0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.644+0000 7f2913b63640 1 --2- 172.21.15.81:0/2395722518 >> v2:172.21.15.167:3300/0 conn(0x7f290c0b57b0 0x7f290c15aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.644+0000 7f2913b63640 1 -- 172.21.15.81:0/2395722518 >> 172.21.15.81:0/2395722518 conn(0x7f290c0d0930 msgr2=0x7f290c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.644+0000 7f2913b63640 1 -- 172.21.15.81:0/2395722518 shutdown_connections 2024-06-29T14:37:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.644+0000 7f2913b63640 1 -- 172.21.15.81:0/2395722518 wait complete. 2024-06-29T14:37:57.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.646+0000 7f2913b63640 1 Processor -- start 2024-06-29T14:37:57.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.646+0000 7f2913b63640 1 -- start start 2024-06-29T14:37:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f2913b63640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c06ef90 0x7f290c0ae730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f2913b63640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0b57b0 0x7f290c1f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f2913b63640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0ce1d0 0x7f290c06c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f2913b63640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f290c0ceeb0 con 0x7f290c0b57b0 2024-06-29T14:37:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f2913b63640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f290c0ced30 con 0x7f290c0ce1d0 2024-06-29T14:37:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f2913b63640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f290c0cf030 con 0x7f290c06ef90 2024-06-29T14:37:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f29110fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0b57b0 0x7f290c1f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f29110fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0b57b0 0x7f290c1f5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47824/0 (socket says 172.21.15.81:47824) 2024-06-29T14:37:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f29110fe640 1 -- 172.21.15.81:0/1748156997 learned_addr learned my addr 172.21.15.81:0/1748156997 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f29118ff640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c06ef90 0x7f290c0ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f2912100640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0ce1d0 0x7f290c06c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f29110fe640 1 -- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c06ef90 msgr2=0x7f290c0ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f29110fe640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c06ef90 0x7f290c0ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f29110fe640 1 -- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0ce1d0 msgr2=0x7f290c06c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f29110fe640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0ce1d0 0x7f290c06c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.647+0000 7f29110fe640 1 -- 172.21.15.81:0/1748156997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290c0c6a30 con 0x7f290c0b57b0 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.648+0000 7f29110fe640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0b57b0 0x7f290c1f5c10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f2900000c00 tx=0x7f29000039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.648+0000 7f290a7fc640 1 -- 172.21.15.81:0/1748156997 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290003e030 con 0x7f290c0b57b0 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.648+0000 7f290a7fc640 1 -- 172.21.15.81:0/1748156997 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29000047a0 con 0x7f290c0b57b0 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.648+0000 7f2913b63640 1 -- 172.21.15.81:0/1748156997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290c0c6d10 con 0x7f290c0b57b0 2024-06-29T14:37:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.648+0000 7f290a7fc640 1 -- 172.21.15.81:0/1748156997 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900035040 con 0x7f290c0b57b0 2024-06-29T14:37:57.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.648+0000 7f2913b63640 1 -- 172.21.15.81:0/1748156997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f290c0c72d0 con 0x7f290c0b57b0 2024-06-29T14:37:57.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.649+0000 7f2913b63640 1 -- 172.21.15.81:0/1748156997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f290c06e2a0 con 0x7f290c0b57b0 2024-06-29T14:37:57.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.649+0000 7f290a7fc640 1 -- 172.21.15.81:0/1748156997 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f2900041030 con 0x7f290c0b57b0 2024-06-29T14:37:57.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.650+0000 7f290a7fc640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f28f80763f0 0x7f28f8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:57.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.650+0000 7f290a7fc640 1 -- 172.21.15.81:0/1748156997 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f29000b8270 con 0x7f290c0b57b0 2024-06-29T14:37:57.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.650+0000 7f29118ff640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f28f80763f0 0x7f28f8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.651+0000 7f29118ff640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f28f80763f0 0x7f28f8078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f28fc0098e0 tx=0x7f28fc009060 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:57.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.655+0000 7f290a7fc640 1 -- 172.21.15.81:0/1748156997 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f290c06e2a0 con 0x7f290c0b57b0 2024-06-29T14:37:57.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.907+0000 7f2913b63640 1 -- 172.21.15.81:0/1748156997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f290c15aed0 con 0x7f290c0b57b0 2024-06-29T14:37:57.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.910+0000 7f290a7fc640 1 -- 172.21.15.81:0/1748156997 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v32) ==== 190+0+147 (secure 0 0 0) 0x7f2900081280 con 0x7f290c0b57b0 2024-06-29T14:37:57.907 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T14:37:57.907 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T14:37:57.908 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T14:37:57.908 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T14:37:57.908 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:37:57.908 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T14:37:57.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 -- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f28f80763f0 msgr2=0x7f28f8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f28f80763f0 0x7f28f8078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f28fc0098e0 tx=0x7f28fc009060 comp rx=0 tx=0).stop 2024-06-29T14:37:57.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 -- 172.21.15.81:0/1748156997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0b57b0 msgr2=0x7f290c1f5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:57.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0b57b0 0x7f290c1f5c10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f2900000c00 tx=0x7f29000039e0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 -- 172.21.15.81:0/1748156997 shutdown_connections 2024-06-29T14:37:57.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f28f80763f0 0x7f28f8078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0ce1d0 0x7f290c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0b57b0 0x7f290c1f5c10 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 --2- 172.21.15.81:0/1748156997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c06ef90 0x7f290c0ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 -- 172.21.15.81:0/1748156997 >> 172.21.15.81:0/1748156997 conn(0x7f290c0d0930 msgr2=0x7f290c0cf440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 -- 172.21.15.81:0/1748156997 shutdown_connections 2024-06-29T14:37:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:57.912+0000 7f2913b63640 1 -- 172.21.15.81:0/1748156997 wait complete. 2024-06-29T14:37:57.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:37:58.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.036+0000 7f1c879b2640 1 -- 172.21.15.81:0/824955428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c800b7290 msgr2=0x7f1c800b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.036+0000 7f1c879b2640 1 --2- 172.21.15.81:0/824955428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c800b7290 0x7f1c800b76a0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f1c8008e230 tx=0x7f1c7802aac0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.036+0000 7f1c879b2640 1 -- 172.21.15.81:0/824955428 shutdown_connections 2024-06-29T14:37:58.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.036+0000 7f1c879b2640 1 --2- 172.21.15.81:0/824955428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c800b7290 0x7f1c800b76a0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.036+0000 7f1c879b2640 1 --2- 172.21.15.81:0/824955428 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c800b5250 0x7f1c800d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.036+0000 7f1c879b2640 1 --2- 172.21.15.81:0/824955428 >> v2:172.21.15.167:3300/0 conn(0x7f1c800b9040 0x7f1c800d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.036+0000 7f1c879b2640 1 -- 172.21.15.81:0/824955428 >> 172.21.15.81:0/824955428 conn(0x7f1c8009cf30 msgr2=0x7f1c800d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.036+0000 7f1c879b2640 1 -- 172.21.15.81:0/824955428 shutdown_connections 2024-06-29T14:37:58.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.036+0000 7f1c879b2640 1 -- 172.21.15.81:0/824955428 wait complete. 2024-06-29T14:37:58.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.038+0000 7f1c879b2640 1 Processor -- start 2024-06-29T14:37:58.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.038+0000 7f1c879b2640 1 -- start start 2024-06-29T14:37:58.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.039+0000 7f1c879b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c800b5250 0x7f1c8009d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.039+0000 7f1c879b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c800b7290 0x7f1c80154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.039+0000 7f1c879b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c800b9040 0x7f1c8009f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.039+0000 7f1c879b2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1c800b7f70 con 0x7f1c800b5250 2024-06-29T14:37:58.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.039+0000 7f1c879b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1c800b7df0 con 0x7f1c800b9040 2024-06-29T14:37:58.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.039+0000 7f1c879b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1c800b80f0 con 0x7f1c800b7290 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c8574e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c800b5250 0x7f1c8009d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c8574e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c800b5250 0x7f1c8009d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47838/0 (socket says 172.21.15.81:47838) 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c8574e640 1 -- 172.21.15.81:0/3085611040 learned_addr learned my addr 172.21.15.81:0/3085611040 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c85f4f640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c800b9040 0x7f1c8009f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c84f4d640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c800b7290 0x7f1c80154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c8574e640 1 -- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c800b7290 msgr2=0x7f1c80154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c8574e640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c800b7290 0x7f1c80154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c8574e640 1 -- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c800b9040 msgr2=0x7f1c8009f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c8574e640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c800b9040 0x7f1c8009f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c8574e640 1 -- 172.21.15.81:0/3085611040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8009e2a0 con 0x7f1c800b5250 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c84f4d640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c800b7290 0x7f1c80154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c8574e640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c800b5250 0x7f1c8009d9a0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c970 tx=0x7f1c7c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c75ffb640 1 -- 172.21.15.81:0/3085611040 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c007bb0 con 0x7f1c800b5250 2024-06-29T14:37:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c75ffb640 1 -- 172.21.15.81:0/3085611040 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c7c007e40 con 0x7f1c800b5250 2024-06-29T14:37:58.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c75ffb640 1 -- 172.21.15.81:0/3085611040 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00f690 con 0x7f1c800b5250 2024-06-29T14:37:58.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c879b2640 1 -- 172.21.15.81:0/3085611040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8009e580 con 0x7f1c800b5250 2024-06-29T14:37:58.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.040+0000 7f1c879b2640 1 -- 172.21.15.81:0/3085611040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c800c8bb0 con 0x7f1c800b5250 2024-06-29T14:37:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.042+0000 7f1c879b2640 1 -- 172.21.15.81:0/3085611040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c800c8e10 con 0x7f1c800b5250 2024-06-29T14:37:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.042+0000 7f1c75ffb640 1 -- 172.21.15.81:0/3085611040 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f1c7c00f830 con 0x7f1c800b5250 2024-06-29T14:37:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.043+0000 7f1c75ffb640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1c640763f0 0x7f1c64078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.043+0000 7f1c75ffb640 1 -- 172.21.15.81:0/3085611040 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f1c7c0980f0 con 0x7f1c800b5250 2024-06-29T14:37:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.043+0000 7f1c84f4d640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1c640763f0 0x7f1c64078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.043+0000 7f1c84f4d640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1c640763f0 0x7f1c64078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1c70005fd0 tx=0x7f1c70005e00 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.047+0000 7f1c75ffb640 1 -- 172.21.15.81:0/3085611040 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1c7c014030 con 0x7f1c800b5250 2024-06-29T14:37:58.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.266+0000 7f1c879b2640 1 -- 172.21.15.81:0/3085611040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c800b76a0 con 0x7f1c800b5250 2024-06-29T14:37:58.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.266+0000 7f1c75ffb640 1 -- 172.21.15.81:0/3085611040 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5225 (secure 0 0 0) 0x7f1c7c0617e0 con 0x7f1c800b5250 2024-06-29T14:37:58.264 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:37:58.264 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":42,"btime":"2024-06-29T14:37:56:110726+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6002,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3644828063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3644828063},{"type":"v1","addr":"172.21.15.167:6836","nonce":3644828063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":7026,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2532767666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2532767666},{"type":"v1","addr":"172.21.15.81:6835","nonce":2532767666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":7029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/818227798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":818227798},{"type":"v1","addr":"172.21.15.81:6834","nonce":818227798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.462608+0000","modified":"2024-06-29T14:37:55.478070+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"b","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/910129539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":910129539},{"type":"v1","addr":"172.21.15.167:6837","nonce":910129539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":5}]} 2024-06-29T14:37:58.264 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 42 2024-06-29T14:37:58.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 -- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1c640763f0 msgr2=0x7f1c64078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1c640763f0 0x7f1c64078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1c70005fd0 tx=0x7f1c70005e00 comp rx=0 tx=0).stop 2024-06-29T14:37:58.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 -- 172.21.15.81:0/3085611040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c800b5250 msgr2=0x7f1c8009d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c800b5250 0x7f1c8009d9a0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c970 tx=0x7f1c7c00ce30 comp rx=0 tx=0).stop 2024-06-29T14:37:58.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 -- 172.21.15.81:0/3085611040 shutdown_connections 2024-06-29T14:37:58.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f1c640763f0 0x7f1c64078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c800b9040 0x7f1c8009f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c800b7290 0x7f1c80154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 --2- 172.21.15.81:0/3085611040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c800b5250 0x7f1c8009d9a0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 -- 172.21.15.81:0/3085611040 >> 172.21.15.81:0/3085611040 conn(0x7f1c8009cf30 msgr2=0x7f1c800d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 -- 172.21.15.81:0/3085611040 shutdown_connections 2024-06-29T14:37:58.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.269+0000 7f1c879b2640 1 -- 172.21.15.81:0/3085611040 wait complete. 2024-06-29T14:37:58.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:37:52.462608+0000', 'modified': '2024-06-29T14:37:55.478070+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7041}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7041': {'gid': 7041, 'name': 'b', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.167:6837/910129539', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6835', 'nonce': 910129539}, {'type': 'v1', 'addr': '172.21.15.167:6837', 'nonce': 910129539}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7041, 'qdb_cluster': [7041]} 2024-06-29T14:37:58.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:37:58.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-06-29T14:37:58.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.798+0000 7f243e29e640 1 -- 172.21.15.81:0/540508882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2420004e40 msgr2=0x7f242000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.798+0000 7f243e29e640 1 --2- 172.21.15.81:0/540508882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2420004e40 0x7f242000b9c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f243000ad20 tx=0x7f2420005830 comp rx=0 tx=0).stop 2024-06-29T14:37:58.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.798+0000 7f243c83b640 1 -- 172.21.15.81:0/540508882 reap_dead start 2024-06-29T14:37:58.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.798+0000 7f243e29e640 1 -- 172.21.15.81:0/540508882 shutdown_connections 2024-06-29T14:37:58.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.798+0000 7f243e29e640 1 -- 172.21.15.81:0/540508882 >> 172.21.15.81:0/540508882 conn(0x7f24380d3c60 msgr2=0x7f24380d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.798+0000 7f243e29e640 1 -- 172.21.15.81:0/540508882 shutdown_connections 2024-06-29T14:37:58.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.798+0000 7f243e29e640 1 -- 172.21.15.81:0/540508882 wait complete. 2024-06-29T14:37:58.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.800+0000 7f243e29e640 1 Processor -- start 2024-06-29T14:37:58.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.800+0000 7f243e29e640 1 -- start start 2024-06-29T14:37:58.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.801+0000 7f243e29e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2420004e40 0x7f24381da950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.801+0000 7f243e29e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24381e0660 0x7f24381ea280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.801+0000 7f243e29e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24381e2400 0x7f24381e2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.801+0000 7f243e29e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f24380cfb70 con 0x7f2420004e40 2024-06-29T14:37:58.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.801+0000 7f243e29e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f24380d1630 con 0x7f24381e0660 2024-06-29T14:37:58.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.801+0000 7f243e29e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f24380d4990 con 0x7f24381e2400 2024-06-29T14:37:58.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.801+0000 7f2437fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2420004e40 0x7f24381da950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.801+0000 7f2437fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2420004e40 0x7f24381da950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47852/0 (socket says 172.21.15.81:47852) 2024-06-29T14:37:58.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.801+0000 7f2437fff640 1 -- 172.21.15.81:0/3529711375 learned_addr learned my addr 172.21.15.81:0/3529711375 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f2437fff640 1 -- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24381e2400 msgr2=0x7f24381e2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f2437fff640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24381e2400 0x7f24381e2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f2437fff640 1 -- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24381e0660 msgr2=0x7f24381ea280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f24377fe640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24381e0660 0x7f24381ea280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f2437fff640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24381e0660 0x7f24381ea280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f2437fff640 1 -- 172.21.15.81:0/3529711375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24381e0f20 con 0x7f2420004e40 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f2437fff640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2420004e40 0x7f24381da950 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f24280065e0 tx=0x7f24280039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f2434ff9640 1 -- 172.21.15.81:0/3529711375 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2428015070 con 0x7f2420004e40 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f2434ff9640 1 -- 172.21.15.81:0/3529711375 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f242800f960 con 0x7f2420004e40 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f243e29e640 1 -- 172.21.15.81:0/3529711375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24381e1200 con 0x7f2420004e40 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f2434ff9640 1 -- 172.21.15.81:0/3529711375 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242800fdb0 con 0x7f2420004e40 2024-06-29T14:37:58.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.802+0000 7f243e29e640 1 -- 172.21.15.81:0/3529711375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24381e17c0 con 0x7f2420004e40 2024-06-29T14:37:58.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.804+0000 7f2434ff9640 1 -- 172.21.15.81:0/3529711375 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f2428011430 con 0x7f2420004e40 2024-06-29T14:37:58.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.804+0000 7f2434ff9640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f24240763f0 0x7f2424078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.804+0000 7f2434ff9640 1 -- 172.21.15.81:0/3529711375 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f24280987a0 con 0x7f2420004e40 2024-06-29T14:37:58.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.805+0000 7f24377fe640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f24240763f0 0x7f2424078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.805+0000 7f24377fe640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f24240763f0 0x7f2424078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f24381e7500 tx=0x7f242c006b70 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.805+0000 7f2445239740 1 Processor -- start 2024-06-29T14:37:58.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.805+0000 7f2445239740 1 -- start start 2024-06-29T14:37:58.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.806+0000 7f2445239740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ecd270 0x560f312b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.806+0000 7f2445239740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ea43b0 0x560f3129d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.806+0000 7f2445239740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30eac590 0x560f3129d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.806+0000 7f2445239740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f31235bc0 con 0x560f30ea43b0 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.806+0000 7f2445239740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x560f311e4630 con 0x560f30eac590 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.806+0000 7f2445239740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x560f311d3bd0 con 0x560f30ecd270 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f24377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ea43b0 0x560f3129d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f24377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ea43b0 0x560f3129d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47862/0 (socket says 172.21.15.81:47862) 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f24377fe640 1 -- 172.21.15.81:0/969815470 learned_addr learned my addr 172.21.15.81:0/969815470 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f243c83b640 1 --2- 172.21.15.81:0/969815470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30eac590 0x560f3129d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f2437fff640 1 --2- 172.21.15.81:0/969815470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ecd270 0x560f312b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f24377fe640 1 -- 172.21.15.81:0/969815470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ecd270 msgr2=0x560f312b96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f24377fe640 1 --2- 172.21.15.81:0/969815470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ecd270 0x560f312b96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f24377fe640 1 -- 172.21.15.81:0/969815470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30eac590 msgr2=0x560f3129d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:58.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f24377fe640 1 --2- 172.21.15.81:0/969815470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30eac590 0x560f3129d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f24377fe640 1 -- 172.21.15.81:0/969815470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f311d0ca0 con 0x560f30ea43b0 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f2437fff640 1 --2- 172.21.15.81:0/969815470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ecd270 0x560f312b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f24377fe640 1 --2- 172.21.15.81:0/969815470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ea43b0 0x560f3129d3a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f242c00fb40 tx=0x7f242c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261a2563e4b163a4 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f241dffb640 1 -- 172.21.15.81:0/969815470 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c021070 con 0x560f30ea43b0 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f241dffb640 1 -- 172.21.15.81:0/969815470 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f242c011b30 con 0x560f30ea43b0 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.807+0000 7f241dffb640 1 -- 172.21.15.81:0/969815470 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c015070 con 0x560f30ea43b0 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.808+0000 7f2445239740 1 -- 172.21.15.81:0/969815470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ea43b0 msgr2=0x560f3129d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.808+0000 7f2445239740 1 --2- 172.21.15.81:0/969815470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ea43b0 0x560f3129d3a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f242c00fb40 tx=0x7f242c012000 comp rx=0 tx=0).stop 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.808+0000 7f2445239740 1 -- 172.21.15.81:0/969815470 shutdown_connections 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.808+0000 7f2445239740 1 --2- 172.21.15.81:0/969815470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30eac590 0x560f3129d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.808+0000 7f2445239740 1 --2- 172.21.15.81:0/969815470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ea43b0 0x560f3129d3a0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.808+0000 7f2445239740 1 --2- 172.21.15.81:0/969815470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ecd270 0x560f312b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.808+0000 7f2445239740 1 -- 172.21.15.81:0/969815470 >> 172.21.15.81:0/969815470 conn(0x560f30ec13c0 msgr2=0x560f31270f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.808+0000 7f2445239740 1 -- 172.21.15.81:0/969815470 shutdown_connections 2024-06-29T14:37:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.808+0000 7f2445239740 1 -- 172.21.15.81:0/969815470 wait complete. 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.809+0000 7f2445239740 1 -- start start 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.809+0000 7f2445239740 1 Processor -- start 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.810+0000 7f2445239740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30ea43b0 0x560f31294980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.810+0000 7f2445239740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30eac590 0x560f3129d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.810+0000 7f2445239740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f3129d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.810+0000 7f2445239740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f31171ae0 con 0x560f30ecd270 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.810+0000 7f2445239740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x560f311d7c10 con 0x560f30ea43b0 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.810+0000 7f2445239740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x560f30f47620 con 0x560f30eac590 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.810+0000 7f2445239740 10 client.0 _ll_register_callbacks cb 0x560f3117e480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f243c83b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f3129d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f243c83b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f3129d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47868/0 (socket says 172.21.15.81:47868) 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f243c83b640 1 -- 172.21.15.81:0/2411676454 learned_addr learned my addr 172.21.15.81:0/2411676454 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f2437fff640 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30ea43b0 0x560f31294980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f24377fe640 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30eac590 0x560f3129d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f243c83b640 1 -- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30eac590 msgr2=0x560f3129d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f243c83b640 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30eac590 0x560f3129d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f243c83b640 1 -- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30ea43b0 msgr2=0x560f31294980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f243c83b640 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30ea43b0 0x560f31294980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f243c83b640 1 -- 172.21.15.81:0/2411676454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f30f38850 con 0x560f30ecd270 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f243c83b640 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f3129d6e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f2430002230 tx=0x7f243002af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149d861491259639 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f2416ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f2416ffd640 1 -- 172.21.15.81:0/2411676454 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430046060 con 0x560f30ecd270 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f2416ffd640 1 -- 172.21.15.81:0/2411676454 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f243003d670 con 0x560f30ecd270 2024-06-29T14:37:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f2445239740 1 -- 172.21.15.81:0/2411676454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f242c011b30 con 0x560f30ecd270 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f2416ffd640 1 -- 172.21.15.81:0/2411676454 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f243004c050 con 0x560f30ecd270 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f2445239740 10 client.7158 fetch_fsmap learned FSMap version 42 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.811+0000 7f2445239740 1 -- 172.21.15.81:0/2411676454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) -- 0x560f30faf6b0 con 0x560f30ecd270 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2416ffd640 1 -- 172.21.15.81:0/2411676454 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f242c011b30 con 0x560f30ecd270 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2416ffd640 1 -- 172.21.15.81:0/2411676454 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 42) ==== 3502+0+0 (secure 0 0 0) 0x7f243003d960 con 0x560f30ecd270 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2445239740 10 client.7158 fetch_fsmap finished waiting for FSMap version 42 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2445239740 10 client.7158 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2445239740 20 client.7158 populate_metadata read hostname 'smithi081' 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2445239740 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:6835/910129539,v1:172.21.15.167:6837/910129539] conn(0x560f30e8be40 0x560f312c8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2445239740 4 client.7158 mds_command: new command op to 7041 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2445239740 1 -- 172.21.15.81:0/2411676454 --> [v2:172.21.15.167:6835/910129539,v1:172.21.15.167:6837/910129539] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f311e6580 con 0x560f30e8be40 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2437fff640 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:6835/910129539,v1:172.21.15.167:6837/910129539] conn(0x560f30e8be40 0x560f312c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2416ffd640 10 client.7158 ms_handle_connect on v2:172.21.15.167:6835/910129539 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.812+0000 7f2437fff640 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:6835/910129539,v1:172.21.15.167:6837/910129539] conn(0x560f30e8be40 0x560f312c8690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca2d23da0a684ae3 server_cookie=43b09d9d71b6740f in_seq=0 out_seq=0 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2416ffd640 1 -- 172.21.15.81:0/2411676454 <== mds.0 v2:172.21.15.167:6835/910129539 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x560f311e6580 con 0x560f30e8be40 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2416ffd640 10 client.7158 handle_command_reply: tid=0 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2445239740 1 client.7158 shutdown 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2445239740 1 -- 172.21.15.81:0/2411676454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 msgr2=0x560f3129d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2445239740 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f3129d6e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f2430002230 tx=0x7f243002af20 comp rx=0 tx=0).stop 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2445239740 1 -- 172.21.15.81:0/2411676454 shutdown_connections 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2445239740 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f3129d6e0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2445239740 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:6835/910129539,v1:172.21.15.167:6837/910129539] conn(0x560f30e8be40 0x560f312c8690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2445239740 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30eac590 0x560f3129d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2445239740 1 --2- 172.21.15.81:0/2411676454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30ea43b0 0x560f31294980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2416ffd640 0 client.7158 ms_handle_reset on v2:172.21.15.167:6835/910129539 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.813+0000 7f2445239740 1 -- 172.21.15.81:0/2411676454 >> 172.21.15.81:0/2411676454 conn(0x560f30ec13c0 msgr2=0x560f3126cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.814+0000 7f2445239740 1 -- 172.21.15.81:0/2411676454 shutdown_connections 2024-06-29T14:37:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.814+0000 7f2445239740 1 -- 172.21.15.81:0/2411676454 wait complete. 2024-06-29T14:37:58.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.814+0000 7f2445239740 20 client.7158 trim_cache size 0 max 16384 2024-06-29T14:37:58.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.833+0000 7f2445239740 1 Processor -- start 2024-06-29T14:37:58.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.833+0000 7f2445239740 1 -- start start 2024-06-29T14:37:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f2445239740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ea43b0 0x560f31299b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f2445239740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30eac590 0x560f31298cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f2445239740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30ecd270 0x560f31273070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f2445239740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f24382017c0 con 0x560f30eac590 2024-06-29T14:37:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f2445239740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2438200860 con 0x560f30ecd270 2024-06-29T14:37:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f2445239740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f24381e8950 con 0x560f30ea43b0 2024-06-29T14:37:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f24377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30eac590 0x560f31298cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f24377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30eac590 0x560f31298cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47876/0 (socket says 172.21.15.81:47876) 2024-06-29T14:37:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f24377fe640 1 -- 172.21.15.81:0/1433270327 learned_addr learned my addr 172.21.15.81:0/1433270327 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f24377fe640 1 -- 172.21.15.81:0/1433270327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ea43b0 msgr2=0x560f31299b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f24377fe640 1 --2- 172.21.15.81:0/1433270327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ea43b0 0x560f31299b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f24377fe640 1 -- 172.21.15.81:0/1433270327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30ecd270 msgr2=0x560f31273070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f24377fe640 1 --2- 172.21.15.81:0/1433270327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30ecd270 0x560f31273070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f24377fe640 1 -- 172.21.15.81:0/1433270327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f3126fd00 con 0x560f30eac590 2024-06-29T14:37:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.834+0000 7f24377fe640 1 --2- 172.21.15.81:0/1433270327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30eac590 0x560f31298cf0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f242c0066b0 tx=0x7f242c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf1f2c0972b6cc2 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f241d7fa640 1 -- 172.21.15.81:0/1433270327 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c021040 con 0x560f30eac590 2024-06-29T14:37:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f241d7fa640 1 -- 172.21.15.81:0/1433270327 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f242c011b30 con 0x560f30eac590 2024-06-29T14:37:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f241d7fa640 1 -- 172.21.15.81:0/1433270327 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c015040 con 0x560f30eac590 2024-06-29T14:37:58.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f2445239740 1 -- 172.21.15.81:0/1433270327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30eac590 msgr2=0x560f31298cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f2445239740 1 --2- 172.21.15.81:0/1433270327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30eac590 0x560f31298cf0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f242c0066b0 tx=0x7f242c0036f0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f2445239740 1 -- 172.21.15.81:0/1433270327 shutdown_connections 2024-06-29T14:37:58.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f2445239740 1 --2- 172.21.15.81:0/1433270327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30ecd270 0x560f31273070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f2445239740 1 --2- 172.21.15.81:0/1433270327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30eac590 0x560f31298cf0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f2445239740 1 --2- 172.21.15.81:0/1433270327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ea43b0 0x560f31299b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f2445239740 1 -- 172.21.15.81:0/1433270327 >> 172.21.15.81:0/1433270327 conn(0x560f30ec13c0 msgr2=0x560f31274650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f2445239740 1 -- 172.21.15.81:0/1433270327 shutdown_connections 2024-06-29T14:37:58.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.835+0000 7f2445239740 1 -- 172.21.15.81:0/1433270327 wait complete. 2024-06-29T14:37:58.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.836+0000 7f2445239740 1 -- start start 2024-06-29T14:37:58.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.836+0000 7f2445239740 1 Processor -- start 2024-06-29T14:37:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2445239740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ea43b0 0x560f31335b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2445239740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30eac590 0x560f312d5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2445239740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f312d6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2445239740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f3129c480 con 0x560f30ecd270 2024-06-29T14:37:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2445239740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2438200210 con 0x560f30eac590 2024-06-29T14:37:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2445239740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x560f312952c0 con 0x560f30ea43b0 2024-06-29T14:37:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2445239740 10 client.0 _ll_register_callbacks cb 0x7f24200036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:37:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f243c83b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f312d6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f243c83b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f312d6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47878/0 (socket says 172.21.15.81:47878) 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f243c83b640 1 -- 172.21.15.81:0/3633077833 learned_addr learned my addr 172.21.15.81:0/3633077833 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2437fff640 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ea43b0 0x560f31335b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f24377fe640 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30eac590 0x560f312d5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f243c83b640 1 -- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ea43b0 msgr2=0x560f31335b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f243c83b640 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ea43b0 0x560f31335b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f243c83b640 1 -- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30eac590 msgr2=0x560f312d5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f243c83b640 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30eac590 0x560f312d5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f243c83b640 1 -- 172.21.15.81:0/3633077833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f31338ae0 con 0x560f30ecd270 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f243c83b640 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f312d6290 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f243000a3c0 tx=0x7f243003d670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4403255f45354f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2416ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2416ffd640 1 -- 172.21.15.81:0/3633077833 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430055060 con 0x560f30ecd270 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2416ffd640 1 -- 172.21.15.81:0/3633077833 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f2430031640 con 0x560f30ecd270 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.838+0000 7f2445239740 1 -- 172.21.15.81:0/3633077833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f242c011b30 con 0x560f30ecd270 2024-06-29T14:37:58.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2416ffd640 1 -- 172.21.15.81:0/3633077833 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2430036040 con 0x560f30ecd270 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2445239740 10 client.7164 fetch_fsmap learned FSMap version 42 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2445239740 1 -- 172.21.15.81:0/3633077833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) -- 0x560f31336370 con 0x560f30ecd270 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2416ffd640 1 -- 172.21.15.81:0/3633077833 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f242c011b30 con 0x560f30ecd270 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2416ffd640 1 -- 172.21.15.81:0/3633077833 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 42) ==== 3502+0+0 (secure 0 0 0) 0x7f2430031880 con 0x560f30ecd270 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2445239740 10 client.7164 fetch_fsmap finished waiting for FSMap version 42 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2445239740 10 client.7164 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2445239740 20 client.7164 populate_metadata read hostname 'smithi081' 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2445239740 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:6835/910129539,v1:172.21.15.167:6837/910129539] conn(0x560f31298910 0x560f3129c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2445239740 4 client.7164 mds_command: new command op to 7041 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2445239740 1 -- 172.21.15.81:0/3633077833 --> [v2:172.21.15.167:6835/910129539,v1:172.21.15.167:6837/910129539] -- command(tid 0: {"prefix": "status"}) -- 0x560f311e2020 con 0x560f31298910 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2437fff640 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:6835/910129539,v1:172.21.15.167:6837/910129539] conn(0x560f31298910 0x560f3129c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2416ffd640 10 client.7164 ms_handle_connect on v2:172.21.15.167:6835/910129539 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.839+0000 7f2437fff640 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:6835/910129539,v1:172.21.15.167:6837/910129539] conn(0x560f31298910 0x560f3129c9d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78c786e403b56e0a server_cookie=3ac6de534141be3a in_seq=0 out_seq=0 2024-06-29T14:37:58.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2416ffd640 1 -- 172.21.15.81:0/3633077833 <== mds.0 v2:172.21.15.167:6835/910129539 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x560f311e2020 con 0x560f31298910 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2416ffd640 10 client.7164 handle_command_reply: tid=0 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 client.7164 shutdown 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 -- 172.21.15.81:0/3633077833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 msgr2=0x560f312d6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f312d6290 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f243000a3c0 tx=0x7f243003d670 comp rx=0 tx=0).stop 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 -- 172.21.15.81:0/3633077833 shutdown_connections 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f30ecd270 0x560f312d6290 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x560f30eac590 0x560f312d5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:6835/910129539,v1:172.21.15.167:6837/910129539] conn(0x560f31298910 0x560f3129c9d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 --2- 172.21.15.81:0/3633077833 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x560f30ea43b0 0x560f31335b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2416ffd640 0 client.7164 ms_handle_reset on v2:172.21.15.167:6835/910129539 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 -- 172.21.15.81:0/3633077833 >> 172.21.15.81:0/3633077833 conn(0x560f30ec13c0 msgr2=0x560f312b85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 -- 172.21.15.81:0/3633077833 shutdown_connections 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 1 -- 172.21.15.81:0/3633077833 wait complete. 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f2445239740 20 client.7164 trim_cache size 0 max 16384 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 7041, 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.740919839, 2024-06-29T14:37:58.843 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 42, 2024-06-29T14:37:58.844 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 91, 2024-06-29T14:37:58.844 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 91, 2024-06-29T14:37:58.844 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.398038603 2024-06-29T14:37:58.844 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f243e29e640 1 -- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f24240763f0 msgr2=0x7f2424078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f243e29e640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f24240763f0 0x7f2424078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f24381e7500 tx=0x7f242c006b70 comp rx=0 tx=0).stop 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f243e29e640 1 -- 172.21.15.81:0/3529711375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2420004e40 msgr2=0x7f24381da950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.840+0000 7f243e29e640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2420004e40 0x7f24381da950 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f24280065e0 tx=0x7f24280039c0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.841+0000 7f243e29e640 1 -- 172.21.15.81:0/3529711375 shutdown_connections 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.841+0000 7f243e29e640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f24240763f0 0x7f2424078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.841+0000 7f243e29e640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24381e2400 0x7f24381e2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.841+0000 7f243e29e640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24381e0660 0x7f24381ea280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.841+0000 7f243e29e640 1 --2- 172.21.15.81:0/3529711375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2420004e40 0x7f24381da950 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.841+0000 7f243e29e640 1 -- 172.21.15.81:0/3529711375 >> 172.21.15.81:0/3529711375 conn(0x7f24380d3c60 msgr2=0x7f24381e64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.841+0000 7f243e29e640 1 -- 172.21.15.81:0/3529711375 shutdown_connections 2024-06-29T14:37:58.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.841+0000 7f243e29e640 1 -- 172.21.15.81:0/3529711375 wait complete. 2024-06-29T14:37:58.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:37:58.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.964+0000 7fe08caef640 1 -- 172.21.15.81:0/797807113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe074004800 msgr2=0x7fe074004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.964+0000 7fe08caef640 1 --2- 172.21.15.81:0/797807113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe074004800 0x7fe074004c10 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fe070001740 tx=0x7fe074005830 comp rx=0 tx=0).stop 2024-06-29T14:37:58.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.964+0000 7fe086d9d640 1 -- 172.21.15.81:0/797807113 reap_dead start 2024-06-29T14:37:58.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.964+0000 7fe08caef640 1 -- 172.21.15.81:0/797807113 shutdown_connections 2024-06-29T14:37:58.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.964+0000 7fe08caef640 1 -- 172.21.15.81:0/797807113 >> 172.21.15.81:0/797807113 conn(0x7fe088072950 msgr2=0x7fe0880b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:58.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.964+0000 7fe08caef640 1 -- 172.21.15.81:0/797807113 shutdown_connections 2024-06-29T14:37:58.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.964+0000 7fe08caef640 1 -- 172.21.15.81:0/797807113 wait complete. 2024-06-29T14:37:58.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.966+0000 7fe08caef640 1 Processor -- start 2024-06-29T14:37:58.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.966+0000 7fe08caef640 1 -- start start 2024-06-29T14:37:58.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.967+0000 7fe08caef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe074004800 0x7fe088150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.967+0000 7fe08caef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe088151470 0x7fe08815c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.967+0000 7fe08caef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe08815c8a0 0x7fe08815ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.967+0000 7fe08caef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe08815b190 con 0x7fe074004800 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.967+0000 7fe08caef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe08815b010 con 0x7fe088151470 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.967+0000 7fe08caef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe0880aee20 con 0x7fe08815c8a0 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe074004800 0x7fe088150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe074004800 0x7fe088150f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47904/0 (socket says 172.21.15.81:47904) 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08659c640 1 -- 172.21.15.81:0/2387574609 learned_addr learned my addr 172.21.15.81:0/2387574609 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe086d9d640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe08815c8a0 0x7fe08815ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe07dd9b640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe088151470 0x7fe08815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08659c640 1 -- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe08815c8a0 msgr2=0x7fe08815ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08659c640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe08815c8a0 0x7fe08815ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08659c640 1 -- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe088151470 msgr2=0x7fe08815c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08659c640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe088151470 0x7fe08815c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08659c640 1 -- 172.21.15.81:0/2387574609 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe088152010 con 0x7fe074004800 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe086d9d640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe08815c8a0 0x7fe08815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe07dd9b640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe088151470 0x7fe08815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08659c640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe074004800 0x7fe088150f30 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fe0780066e0 tx=0x7fe0780039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe07f7fe640 1 -- 172.21.15.81:0/2387574609 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07800f760 con 0x7fe074004800 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe07f7fe640 1 -- 172.21.15.81:0/2387574609 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe078016020 con 0x7fe074004800 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe07f7fe640 1 -- 172.21.15.81:0/2387574609 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07800fbc0 con 0x7fe074004800 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08caef640 1 -- 172.21.15.81:0/2387574609 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0881543f0 con 0x7fe074004800 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.968+0000 7fe08caef640 1 -- 172.21.15.81:0/2387574609 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0881549b0 con 0x7fe074004800 2024-06-29T14:37:58.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.969+0000 7fe08caef640 1 -- 172.21.15.81:0/2387574609 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0880af380 con 0x7fe074004800 2024-06-29T14:37:58.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.970+0000 7fe07f7fe640 1 -- 172.21.15.81:0/2387574609 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fe078011430 con 0x7fe074004800 2024-06-29T14:37:58.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.971+0000 7fe07f7fe640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe0680763f0 0x7fe068078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:37:58.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.971+0000 7fe07f7fe640 1 -- 172.21.15.81:0/2387574609 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7fe078098560 con 0x7fe074004800 2024-06-29T14:37:58.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.971+0000 7fe07dd9b640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe0680763f0 0x7fe068078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:37:58.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.971+0000 7fe07dd9b640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe0680763f0 0x7fe068078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe070001710 tx=0x7fe070001270 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:37:58.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:58.975+0000 7fe07f7fe640 1 -- 172.21.15.81:0/2387574609 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe078020020 con 0x7fe074004800 2024-06-29T14:37:59.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.192+0000 7fe08caef640 1 -- 172.21.15.81:0/2387574609 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe08815a230 con 0x7fe074004800 2024-06-29T14:37:59.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.192+0000 7fe07f7fe640 1 -- 172.21.15.81:0/2387574609 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) ==== 53+0+73 (secure 0 0 0) 0x7fe078061c60 con 0x7fe074004800 2024-06-29T14:37:59.190 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:37:59.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.194+0000 7fe08caef640 1 -- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe0680763f0 msgr2=0x7fe068078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.194+0000 7fe08caef640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe0680763f0 0x7fe068078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe070001710 tx=0x7fe070001270 comp rx=0 tx=0).stop 2024-06-29T14:37:59.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.194+0000 7fe08caef640 1 -- 172.21.15.81:0/2387574609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe074004800 msgr2=0x7fe088150f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:37:59.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.194+0000 7fe08caef640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe074004800 0x7fe088150f30 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fe0780066e0 tx=0x7fe0780039c0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.194+0000 7fe08caef640 1 -- 172.21.15.81:0/2387574609 shutdown_connections 2024-06-29T14:37:59.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.194+0000 7fe08caef640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fe0680763f0 0x7fe068078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.194+0000 7fe08caef640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe08815c8a0 0x7fe08815ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.194+0000 7fe08caef640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe088151470 0x7fe08815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.194+0000 7fe08caef640 1 --2- 172.21.15.81:0/2387574609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe074004800 0x7fe088150f30 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:37:59.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.194+0000 7fe08caef640 1 -- 172.21.15.81:0/2387574609 >> 172.21.15.81:0/2387574609 conn(0x7fe088072950 msgr2=0x7fe088158860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:37:59.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.195+0000 7fe08caef640 1 -- 172.21.15.81:0/2387574609 shutdown_connections 2024-06-29T14:37:59.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59.195+0000 7fe08caef640 1 -- 172.21.15.81:0/2387574609 wait complete. 2024-06-29T14:37:59.203 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:37:59.203 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:37:59.203 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T14:37:59.203 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:37:59.203 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:59.204 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:37:59.204 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:37:59.204 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:37:59.204 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:59.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.204 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T14:37:59.220 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:37:59.220 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:37:59.220 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:37:59.220 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.220 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:37:59.220 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.220 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:59.220 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:59.220 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T14:37:59.220 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 41840sec preferred_lft 41840sec 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:37:59.221 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:37:59.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.223 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:37:59.283 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T14:37:59.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.285 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T14:37:59.341 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T14:37:59.341 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:37:59.341 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:37:59.341 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:37:59.341 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:37:59.341 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:37:59.341 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:37:59.341 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:37:59.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:37:59.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:37:59.342 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:37:59.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:37:59.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:37:59.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.467 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:37:59.482 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:37:59.482 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:37:59.483 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:59.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.483 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T14:37:59.547 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:37:59.547 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:59.602 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:37:59.603 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:37:59.603 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:59.658 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T14:37:59.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:37:59.752 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:37:59.752 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:37:59.761 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:37:59.761 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:37:59.761 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:37:59.761 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:37:59.762 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:37:59.762 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:37:59.762 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:37:59.762 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:37:59.762 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:37:59.762 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:37:59.764 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:37:59.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:37:59.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:37:59.789 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:37:59.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:37:59.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:37:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:37:59.892 INFO:teuthology.orchestra.run.smithi081.stdout:client6601 2024-06-29T14:37:59.892 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:37:59.892 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:37:59.908 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:38:00.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.064+0000 7fd1d9d63640 1 -- 172.21.15.81:0/3463031063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1d41451c0 msgr2=0x7fd1d4144c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.064+0000 7fd1d9d63640 1 --2- 172.21.15.81:0/3463031063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1d41451c0 0x7fd1d4144c40 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4005fd0 tx=0x7fd1c402aa90 comp rx=0 tx=0).stop 2024-06-29T14:38:00.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.064+0000 7fd1d9d63640 1 -- 172.21.15.81:0/3463031063 shutdown_connections 2024-06-29T14:38:00.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.064+0000 7fd1d9d63640 1 --2- 172.21.15.81:0/3463031063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1d41451c0 0x7fd1d4144c40 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.064+0000 7fd1d9d63640 1 -- 172.21.15.81:0/3463031063 >> 172.21.15.81:0/3463031063 conn(0x7fd1d40b1ce0 msgr2=0x7fd1d40b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:00.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.064+0000 7fd1d9d63640 1 -- 172.21.15.81:0/3463031063 shutdown_connections 2024-06-29T14:38:00.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.064+0000 7fd1d9d63640 1 -- 172.21.15.81:0/3463031063 wait complete. 2024-06-29T14:38:00.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.065+0000 7fd1d9d63640 1 Processor -- start 2024-06-29T14:38:00.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.066+0000 7fd1d9d63640 1 -- start start 2024-06-29T14:38:00.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d9d63640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1d40732a0 0x7fd1d41f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d9d63640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1d40cf6e0 0x7fd1d41e2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d9d63640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1d41451c0 0x7fd1d41f6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d9d63640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d415c690 con 0x7fd1d41451c0 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d9d63640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d415c510 con 0x7fd1d40cf6e0 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d9d63640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d415c810 con 0x7fd1d40732a0 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1d41451c0 0x7fd1d41f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1d41451c0 0x7fd1d41f6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43880/0 (socket says 172.21.15.81:43880) 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d3fff640 1 -- 172.21.15.81:0/2377483715 learned_addr learned my addr 172.21.15.81:0/2377483715 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d2ffd640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1d40cf6e0 0x7fd1d41e2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d37fe640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1d40732a0 0x7fd1d41f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d3fff640 1 -- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1d40732a0 msgr2=0x7fd1d41f0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d3fff640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1d40732a0 0x7fd1d41f0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d3fff640 1 -- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1d40cf6e0 msgr2=0x7fd1d41e2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d3fff640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1d40cf6e0 0x7fd1d41e2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.067+0000 7fd1d3fff640 1 -- 172.21.15.81:0/2377483715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d41f4350 con 0x7fd1d41451c0 2024-06-29T14:38:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.068+0000 7fd1d3fff640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1d41451c0 0x7fd1d41f6690 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00b410 tx=0x7fd1cc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.068+0000 7fd1bbfff640 1 -- 172.21.15.81:0/2377483715 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc015030 con 0x7fd1d41451c0 2024-06-29T14:38:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.068+0000 7fd1bbfff640 1 -- 172.21.15.81:0/2377483715 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1cc004020 con 0x7fd1d41451c0 2024-06-29T14:38:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.068+0000 7fd1d9d63640 1 -- 172.21.15.81:0/2377483715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d41e3370 con 0x7fd1d41451c0 2024-06-29T14:38:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.068+0000 7fd1bbfff640 1 -- 172.21.15.81:0/2377483715 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc004470 con 0x7fd1d41451c0 2024-06-29T14:38:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.068+0000 7fd1d9d63640 1 -- 172.21.15.81:0/2377483715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d41f5e90 con 0x7fd1d41451c0 2024-06-29T14:38:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.069+0000 7fd1d9d63640 1 -- 172.21.15.81:0/2377483715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d41f60f0 con 0x7fd1d41451c0 2024-06-29T14:38:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.069+0000 7fd1bbfff640 1 -- 172.21.15.81:0/2377483715 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7fd1cc018070 con 0x7fd1d41451c0 2024-06-29T14:38:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.070+0000 7fd1bbfff640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd1c00763f0 0x7fd1c0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.070+0000 7fd1bbfff640 1 -- 172.21.15.81:0/2377483715 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7fd1cc097530 con 0x7fd1d41451c0 2024-06-29T14:38:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.070+0000 7fd1d37fe640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd1c00763f0 0x7fd1c0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.071+0000 7fd1d37fe640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd1c00763f0 0x7fd1c0078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4035000 tx=0x7fd1c4038040 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:00.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.074+0000 7fd1bbfff640 1 -- 172.21.15.81:0/2377483715 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd1cc060ca0 con 0x7fd1d41451c0 2024-06-29T14:38:00.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.283+0000 7fd1d9d63640 1 -- 172.21.15.81:0/2377483715 --> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fd1d41f0a70 con 0x7fd1c00763f0 2024-06-29T14:38:00.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.285+0000 7fd1bbfff640 1 -- 172.21.15.81:0/2377483715 <== mgr.6298 v2:172.21.15.167:6832/1959721003 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fd1c402eb60 con 0x7fd1c00763f0 2024-06-29T14:38:00.283 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:38:00.283 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:00.283 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T14:38:00.283 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:00.283 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:38:00.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 -- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd1c00763f0 msgr2=0x7fd1c0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd1c00763f0 0x7fd1c0078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4035000 tx=0x7fd1c4038040 comp rx=0 tx=0).stop 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 -- 172.21.15.81:0/2377483715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1d41451c0 msgr2=0x7fd1d41f6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1d41451c0 0x7fd1d41f6690 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00b410 tx=0x7fd1cc00b8d0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 -- 172.21.15.81:0/2377483715 shutdown_connections 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7fd1c00763f0 0x7fd1c0078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1d41451c0 0x7fd1d41f6690 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1d40cf6e0 0x7fd1d41e2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 --2- 172.21.15.81:0/2377483715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1d40732a0 0x7fd1d41f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 -- 172.21.15.81:0/2377483715 >> 172.21.15.81:0/2377483715 conn(0x7fd1d40b1ce0 msgr2=0x7fd1d40af970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 -- 172.21.15.81:0/2377483715 shutdown_connections 2024-06-29T14:38:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.287+0000 7fd1d9d63640 1 -- 172.21.15.81:0/2377483715 wait complete. 2024-06-29T14:38:00.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:38:00.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.413+0000 7f64bfda1640 1 -- 172.21.15.81:0/1853159871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8073570 msgr2=0x7f64b815e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.413+0000 7f64bfda1640 1 --2- 172.21.15.81:0/1853159871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8073570 0x7f64b815e960 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f64b80b2700 tx=0x7f64b002aa30 comp rx=0 tx=0).stop 2024-06-29T14:38:00.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.413+0000 7f64bfda1640 1 -- 172.21.15.81:0/1853159871 shutdown_connections 2024-06-29T14:38:00.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.413+0000 7f64bfda1640 1 --2- 172.21.15.81:0/1853159871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8073570 0x7f64b815e960 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.413+0000 7f64bfda1640 1 --2- 172.21.15.81:0/1853159871 >> v2:172.21.15.167:3300/0 conn(0x7f64b80cf4d0 0x7f64b8073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.413+0000 7f64bfda1640 1 --2- 172.21.15.81:0/1853159871 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64b81412f0 0x7f64b80cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.413+0000 7f64bfda1640 1 -- 172.21.15.81:0/1853159871 >> 172.21.15.81:0/1853159871 conn(0x7f64b806ccc0 msgr2=0x7f64b80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:00.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.414+0000 7f64bfda1640 1 -- 172.21.15.81:0/1853159871 shutdown_connections 2024-06-29T14:38:00.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.414+0000 7f64bfda1640 1 -- 172.21.15.81:0/1853159871 wait complete. 2024-06-29T14:38:00.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.415+0000 7f64bfda1640 1 Processor -- start 2024-06-29T14:38:00.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.415+0000 7f64bfda1640 1 -- start start 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bfda1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8073570 0x7f64b81578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bfda1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64b80cf4d0 0x7f64b80ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bfda1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64b81412f0 0x7f64b81f1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bfda1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64b815f230 con 0x7f64b8073570 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bfda1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f64b815f0b0 con 0x7f64b81412f0 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bfda1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f64b80b3210 con 0x7f64b80cf4d0 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bdb3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8073570 0x7f64b81578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bdb3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8073570 0x7f64b81578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43894/0 (socket says 172.21.15.81:43894) 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bdb3d640 1 -- 172.21.15.81:0/1873098124 learned_addr learned my addr 172.21.15.81:0/1873098124 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64be33e640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64b81412f0 0x7f64b81f1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bd33c640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64b80cf4d0 0x7f64b80ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bdb3d640 1 -- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64b80cf4d0 msgr2=0x7f64b80ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bdb3d640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64b80cf4d0 0x7f64b80ae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bdb3d640 1 -- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64b81412f0 msgr2=0x7f64b81f1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bdb3d640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64b81412f0 0x7f64b81f1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bdb3d640 1 -- 172.21.15.81:0/1873098124 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b80addf0 con 0x7f64b8073570 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64be33e640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64b81412f0 0x7f64b81f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bd33c640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64b80cf4d0 0x7f64b80ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bdb3d640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8073570 0x7f64b81578f0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f64b4007850 tx=0x7f64b4007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64ae7fc640 1 -- 172.21.15.81:0/1873098124 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b4019020 con 0x7f64b8073570 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64ae7fc640 1 -- 172.21.15.81:0/1873098124 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64b4005020 con 0x7f64b8073570 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bfda1640 1 -- 172.21.15.81:0/1873098124 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b80acd50 con 0x7f64b8073570 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64ae7fc640 1 -- 172.21.15.81:0/1873098124 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b4005490 con 0x7f64b8073570 2024-06-29T14:38:00.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.417+0000 7f64bfda1640 1 -- 172.21.15.81:0/1873098124 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b80ad310 con 0x7f64b8073570 2024-06-29T14:38:00.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.418+0000 7f64bfda1640 1 -- 172.21.15.81:0/1873098124 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b806f270 con 0x7f64b8073570 2024-06-29T14:38:00.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.419+0000 7f64ae7fc640 1 -- 172.21.15.81:0/1873098124 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 38) ==== 98303+0+0 (secure 0 0 0) 0x7f64b40047a0 con 0x7f64b8073570 2024-06-29T14:38:00.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.420+0000 7f64ae7fc640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f649c0763f0 0x7f649c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:00.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.420+0000 7f64ae7fc640 1 -- 172.21.15.81:0/1873098124 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f64b4099670 con 0x7f64b8073570 2024-06-29T14:38:00.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.420+0000 7f64bd33c640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f649c0763f0 0x7f649c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:00.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.420+0000 7f64bd33c640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f649c0763f0 0x7f649c078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64b815dee0 tx=0x7f64a8005ca0 comp rx=0 tx=0).ready entity=mgr.6298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.424+0000 7f64ae7fc640 1 -- 172.21.15.81:0/1873098124 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f64b4023020 con 0x7f64b8073570 2024-06-29T14:38:00.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:00.653+0000 7f64bfda1640 1 -- 172.21.15.81:0/1873098124 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f64b806ef90 con 0x7f64b8073570 2024-06-29T14:38:01.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.140+0000 7f64ae7fc640 1 -- 172.21.15.81:0/1873098124 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v39) ==== 92+0+0 (secure 0 0 0) 0x7f64b4062d60 con 0x7f64b8073570 2024-06-29T14:38:01.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.144+0000 7f64bfda1640 1 -- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f649c0763f0 msgr2=0x7f649c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.144+0000 7f64bfda1640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f649c0763f0 0x7f649c078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64b815dee0 tx=0x7f64a8005ca0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.144+0000 7f64bfda1640 1 -- 172.21.15.81:0/1873098124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8073570 msgr2=0x7f64b81578f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.144+0000 7f64bfda1640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8073570 0x7f64b81578f0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f64b4007850 tx=0x7f64b4007d10 comp rx=0 tx=0).stop 2024-06-29T14:38:01.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.144+0000 7f64bfda1640 1 -- 172.21.15.81:0/1873098124 shutdown_connections 2024-06-29T14:38:01.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.144+0000 7f64bfda1640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f649c0763f0 0x7f649c078900 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64b815dee0 tx=0x7f64a8005ca0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.144+0000 7f64bfda1640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64b81412f0 0x7f64b81f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.144+0000 7f64bfda1640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64b80cf4d0 0x7f64b80ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.144+0000 7f64bfda1640 1 --2- 172.21.15.81:0/1873098124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8073570 0x7f64b81578f0 secure :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f64b4007850 tx=0x7f64b4007d10 comp rx=0 tx=0).stop 2024-06-29T14:38:01.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.144+0000 7f64bfda1640 1 -- 172.21.15.81:0/1873098124 >> 172.21.15.81:0/1873098124 conn(0x7f64b806ccc0 msgr2=0x7f64b815aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:01.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.145+0000 7f64bfda1640 1 -- 172.21.15.81:0/1873098124 shutdown_connections 2024-06-29T14:38:01.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.145+0000 7f64bfda1640 1 -- 172.21.15.81:0/1873098124 wait complete. 2024-06-29T14:38:01.158 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:38:01.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.282+0000 7f20a7b11640 1 -- 172.21.15.81:0/2376672853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20a00b3200 msgr2=0x7f20a00cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.282+0000 7f20a7b11640 1 --2- 172.21.15.81:0/2376672853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20a00b3200 0x7f20a00cc7b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2098009830 tx=0x7f209802a510 comp rx=0 tx=0).stop 2024-06-29T14:38:01.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.282+0000 7f20a7b11640 1 -- 172.21.15.81:0/2376672853 shutdown_connections 2024-06-29T14:38:01.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.282+0000 7f20a7b11640 1 --2- 172.21.15.81:0/2376672853 >> v2:172.21.15.167:3300/0 conn(0x7f20a00cccf0 0x7f20a00d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.282+0000 7f20a7b11640 1 --2- 172.21.15.81:0/2376672853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20a00b3200 0x7f20a00cc7b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.282+0000 7f20a7b11640 1 -- 172.21.15.81:0/2376672853 >> 172.21.15.81:0/2376672853 conn(0x7f20a00b3680 msgr2=0x7f20a00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:01.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.282+0000 7f20a7b11640 1 -- 172.21.15.81:0/2376672853 shutdown_connections 2024-06-29T14:38:01.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.282+0000 7f20a7b11640 1 -- 172.21.15.81:0/2376672853 wait complete. 2024-06-29T14:38:01.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.283+0000 7f20a7b11640 1 Processor -- start 2024-06-29T14:38:01.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.284+0000 7f20a7b11640 1 -- start start 2024-06-29T14:38:01.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a7b11640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20a00b3200 0x7f20a00b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a7b11640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20a00cccf0 0x7f20a01511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a7b11640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20a00d0770 0x7f20a00af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a7b11640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f20a00cedb0 con 0x7f20a00b3200 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a7b11640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f20a00cec30 con 0x7f20a00cccf0 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a7b11640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f20a00cef30 con 0x7f20a00d0770 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a58ad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20a00b3200 0x7f20a00b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a58ad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20a00b3200 0x7f20a00b25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43922/0 (socket says 172.21.15.81:43922) 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a58ad640 1 -- 172.21.15.81:0/2860540992 learned_addr learned my addr 172.21.15.81:0/2860540992 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a50ac640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20a00cccf0 0x7f20a01511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a60ae640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20a00d0770 0x7f20a00af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a58ad640 1 -- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20a00d0770 msgr2=0x7f20a00af7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a58ad640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20a00d0770 0x7f20a00af7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a58ad640 1 -- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20a00cccf0 msgr2=0x7f20a01511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a58ad640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20a00cccf0 0x7f20a01511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a58ad640 1 -- 172.21.15.81:0/2860540992 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a009daa0 con 0x7f20a00b3200 2024-06-29T14:38:01.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.285+0000 7f20a58ad640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20a00b3200 0x7f20a00b25a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f209802ab00 tx=0x7f2098033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:01.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.286+0000 7f20967fc640 1 -- 172.21.15.81:0/2860540992 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098005c90 con 0x7f20a00b3200 2024-06-29T14:38:01.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.286+0000 7f20967fc640 1 -- 172.21.15.81:0/2860540992 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2098031e50 con 0x7f20a00b3200 2024-06-29T14:38:01.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.286+0000 7f20967fc640 1 -- 172.21.15.81:0/2860540992 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098005260 con 0x7f20a00b3200 2024-06-29T14:38:01.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.286+0000 7f20a7b11640 1 -- 172.21.15.81:0/2860540992 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20a009f260 con 0x7f20a00b3200 2024-06-29T14:38:01.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.286+0000 7f20a7b11640 1 -- 172.21.15.81:0/2860540992 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20a009d4e0 con 0x7f20a00b3200 2024-06-29T14:38:01.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.287+0000 7f20a7b11640 1 -- 172.21.15.81:0/2860540992 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20a00cc7b0 con 0x7f20a00b3200 2024-06-29T14:38:01.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.287+0000 7f20967fc640 1 -- 172.21.15.81:0/2860540992 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 39) ==== 98320+0+0 (secure 0 0 0) 0x7f2098005400 con 0x7f20a00b3200 2024-06-29T14:38:01.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.288+0000 7f20967fc640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f2084076440 0x7f2084078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.288+0000 7f20a50ac640 1 -- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f2084076440 msgr2=0x7f2084078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1959721003 2024-06-29T14:38:01.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.288+0000 7f20967fc640 1 -- 172.21.15.81:0/2860540992 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f20980b8a10 con 0x7f20a00b3200 2024-06-29T14:38:01.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.288+0000 7f20a50ac640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f2084076440 0x7f2084078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:01.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.292+0000 7f20967fc640 1 -- 172.21.15.81:0/2860540992 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f209803f030 con 0x7f20a00b3200 2024-06-29T14:38:01.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.478+0000 7f20a7b11640 1 -- 172.21.15.81:0/2860540992 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f20a00d2680 con 0x7f20a00b3200 2024-06-29T14:38:01.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.481+0000 7f20967fc640 1 -- 172.21.15.81:0/2860540992 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) ==== 138+0+0 (secure 0 0 0) 0x7f209808a0f0 con 0x7f20a00b3200 2024-06-29T14:38:01.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.483+0000 7f20a7b11640 1 -- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f2084076440 msgr2=0x7f2084078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:01.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.483+0000 7f20a7b11640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f2084076440 0x7f2084078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.483+0000 7f20a7b11640 1 -- 172.21.15.81:0/2860540992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20a00b3200 msgr2=0x7f20a00b25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.483+0000 7f20a7b11640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20a00b3200 0x7f20a00b25a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f209802ab00 tx=0x7f2098033c20 comp rx=0 tx=0).stop 2024-06-29T14:38:01.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.483+0000 7f20a7b11640 1 -- 172.21.15.81:0/2860540992 shutdown_connections 2024-06-29T14:38:01.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.483+0000 7f20a7b11640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f2084076440 0x7f2084078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.483+0000 7f20a7b11640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20a00d0770 0x7f20a00af7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.483+0000 7f20a7b11640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20a00cccf0 0x7f20a01511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.483+0000 7f20a7b11640 1 --2- 172.21.15.81:0/2860540992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20a00b3200 0x7f20a00b25a0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.484+0000 7f20a7b11640 1 -- 172.21.15.81:0/2860540992 >> 172.21.15.81:0/2860540992 conn(0x7f20a00b3680 msgr2=0x7f20a01597f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:01.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.484+0000 7f20a7b11640 1 -- 172.21.15.81:0/2860540992 shutdown_connections 2024-06-29T14:38:01.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.484+0000 7f20a7b11640 1 -- 172.21.15.81:0/2860540992 wait complete. 2024-06-29T14:38:01.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:38:01.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.613+0000 7f3cf2573640 1 -- 172.21.15.81:0/3498320430 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cec0c5870 msgr2=0x7f3cec06c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.613+0000 7f3cf2573640 1 --2- 172.21.15.81:0/3498320430 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cec0c5870 0x7f3cec06c2f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4009830 tx=0x7f3cd402a5a0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.613+0000 7f3cf2573640 1 -- 172.21.15.81:0/3498320430 shutdown_connections 2024-06-29T14:38:01.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.613+0000 7f3cf2573640 1 --2- 172.21.15.81:0/3498320430 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3cec0cf860 0x7f3cec06c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.613+0000 7f3cf2573640 1 --2- 172.21.15.81:0/3498320430 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cec0c5870 0x7f3cec06c2f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.613+0000 7f3cf2573640 1 -- 172.21.15.81:0/3498320430 >> 172.21.15.81:0/3498320430 conn(0x7f3cec0d00b0 msgr2=0x7f3cec0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:01.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.613+0000 7f3cf2573640 1 -- 172.21.15.81:0/3498320430 shutdown_connections 2024-06-29T14:38:01.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.613+0000 7f3cf2573640 1 -- 172.21.15.81:0/3498320430 wait complete. 2024-06-29T14:38:01.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.615+0000 7f3cf2573640 1 Processor -- start 2024-06-29T14:38:01.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.615+0000 7f3cf2573640 1 -- start start 2024-06-29T14:38:01.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cf2573640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cec0cf860 0x7f3cec14faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cf2573640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cec15ae90 0x7f3cec15d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cf2573640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3cec156bd0 0x7f3cec15d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cf2573640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec06d2c0 con 0x7f3cec15ae90 2024-06-29T14:38:01.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cf2573640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3cec06d140 con 0x7f3cec0cf860 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cf2573640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3cec06d440 con 0x7f3cec156bd0 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3ceb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cec15ae90 0x7f3cec15d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3ceb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cec15ae90 0x7f3cec15d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43936/0 (socket says 172.21.15.81:43936) 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3ceb7fe640 1 -- 172.21.15.81:0/253371397 learned_addr learned my addr 172.21.15.81:0/253371397 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cebfff640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cec0cf860 0x7f3cec14faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cf0b10640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3cec156bd0 0x7f3cec15d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cebfff640 1 -- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3cec156bd0 msgr2=0x7f3cec15d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cebfff640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3cec156bd0 0x7f3cec15d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cebfff640 1 -- 172.21.15.81:0/253371397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cec15ae90 msgr2=0x7f3cec15d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cebfff640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cec15ae90 0x7f3cec15d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cebfff640 1 -- 172.21.15.81:0/253371397 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cec0c9da0 con 0x7f3cec0cf860 2024-06-29T14:38:01.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3cebfff640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cec0cf860 0x7f3cec14faa0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00c3f0 tx=0x7f3cdc00c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:01.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.617+0000 7f3ce8ff9640 1 -- 172.21.15.81:0/253371397 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc017070 con 0x7f3cec0cf860 2024-06-29T14:38:01.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.618+0000 7f3ce8ff9640 1 -- 172.21.15.81:0/253371397 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3cdc0053c0 con 0x7f3cec0cf860 2024-06-29T14:38:01.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.618+0000 7f3ce8ff9640 1 -- 172.21.15.81:0/253371397 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc003850 con 0x7f3cec0cf860 2024-06-29T14:38:01.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.618+0000 7f3cf2573640 1 -- 172.21.15.81:0/253371397 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cec0b5410 con 0x7f3cec0cf860 2024-06-29T14:38:01.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.618+0000 7f3cf2573640 1 -- 172.21.15.81:0/253371397 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cec0b59d0 con 0x7f3cec0cf860 2024-06-29T14:38:01.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.618+0000 7f3cf2573640 1 -- 172.21.15.81:0/253371397 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cec0b5230 con 0x7f3cec0cf860 2024-06-29T14:38:01.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.619+0000 7f3ce8ff9640 1 -- 172.21.15.81:0/253371397 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 39) ==== 98320+0+0 (secure 0 0 0) 0x7f3cdc01a070 con 0x7f3cec0cf860 2024-06-29T14:38:01.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.620+0000 7f3ce8ff9640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3ccc076470 0x7f3ccc078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.620+0000 7f3ceb7fe640 1 -- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3ccc076470 msgr2=0x7f3ccc078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1959721003 2024-06-29T14:38:01.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.620+0000 7f3ce8ff9640 1 -- 172.21.15.81:0/253371397 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f3cdc096c60 con 0x7f3cec0cf860 2024-06-29T14:38:01.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.620+0000 7f3ceb7fe640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3ccc076470 0x7f3ccc078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:01.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.625+0000 7f3ce8ff9640 1 -- 172.21.15.81:0/253371397 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3cec0b5230 con 0x7f3cec0cf860 2024-06-29T14:38:01.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.814+0000 7f3cf2573640 1 -- 172.21.15.81:0/253371397 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f3cec06d9d0 con 0x7f3cec0cf860 2024-06-29T14:38:01.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.820+0000 7f3ce8ff9640 1 -- 172.21.15.81:0/253371397 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) ==== 133+0+0 (secure 0 0 0) 0x7f3cdc068430 con 0x7f3cec0cf860 2024-06-29T14:38:01.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.821+0000 7f3ceb7fe640 1 -- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3ccc076470 msgr2=0x7f3ccc078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1959721003 2024-06-29T14:38:01.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.821+0000 7f3ceb7fe640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3ccc076470 0x7f3ccc078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 -- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3ccc076470 msgr2=0x7f3ccc078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3ccc076470 0x7f3ccc078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 -- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cec0cf860 msgr2=0x7f3cec14faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cec0cf860 0x7f3cec14faa0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00c3f0 tx=0x7f3cdc00c8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 -- 172.21.15.81:0/253371397 shutdown_connections 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f3ccc076470 0x7f3ccc078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3cec156bd0 0x7f3cec15d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cec15ae90 0x7f3cec15d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 --2- 172.21.15.81:0/253371397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3cec0cf860 0x7f3cec14faa0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 -- 172.21.15.81:0/253371397 >> 172.21.15.81:0/253371397 conn(0x7f3cec0d00b0 msgr2=0x7f3cec0afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 -- 172.21.15.81:0/253371397 shutdown_connections 2024-06-29T14:38:01.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.822+0000 7f3cf2573640 1 -- 172.21.15.81:0/253371397 wait complete. 2024-06-29T14:38:01.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-06-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.949+0000 7f78a6b3c640 1 -- 172.21.15.81:0/3956559808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78a00d2a90 msgr2=0x7f78a00d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.949+0000 7f78a6b3c640 1 --2- 172.21.15.81:0/3956559808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78a00d2a90 0x7f78a00d09c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f78a008e230 tx=0x7f789802a940 comp rx=0 tx=0).stop 2024-06-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.950+0000 7f78a6b3c640 1 -- 172.21.15.81:0/3956559808 shutdown_connections 2024-06-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.950+0000 7f78a6b3c640 1 --2- 172.21.15.81:0/3956559808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78a00d2a90 0x7f78a00d09c0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.950+0000 7f78a6b3c640 1 --2- 172.21.15.81:0/3956559808 >> v2:172.21.15.167:3300/0 conn(0x7f78a00b9040 0x7f78a00d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.950+0000 7f78a6b3c640 1 -- 172.21.15.81:0/3956559808 >> 172.21.15.81:0/3956559808 conn(0x7f78a009cf30 msgr2=0x7f78a00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.950+0000 7f78a6b3c640 1 -- 172.21.15.81:0/3956559808 shutdown_connections 2024-06-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.950+0000 7f78a6b3c640 1 -- 172.21.15.81:0/3956559808 wait complete. 2024-06-29T14:38:01.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.951+0000 7f78a6b3c640 1 Processor -- start 2024-06-29T14:38:01.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.951+0000 7f78a6b3c640 1 -- start start 2024-06-29T14:38:01.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f78a6b3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78a00b5250 0x7f78a014fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f78a6b3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78a00b9040 0x7f78a014e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f78a6b3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78a00d2a90 0x7f78a014d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f78a6b3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f78a00b7d30 con 0x7f78a00b9040 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f78a6b3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f78a00b7bb0 con 0x7f78a00d2a90 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f78a6b3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f78a00b7eb0 con 0x7f78a00b5250 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f789ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78a00b9040 0x7f78a014e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f789ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78a00b9040 0x7f78a014e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43954/0 (socket says 172.21.15.81:43954) 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f789ffff640 1 -- 172.21.15.81:0/2616755101 learned_addr learned my addr 172.21.15.81:0/2616755101 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f78a48d8640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78a00b5250 0x7f78a014fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f78a50d9640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78a00d2a90 0x7f78a014d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f789ffff640 1 -- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78a00b5250 msgr2=0x7f78a014fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f789ffff640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78a00b5250 0x7f78a014fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f789ffff640 1 -- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78a00d2a90 msgr2=0x7f78a014d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f789ffff640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78a00d2a90 0x7f78a014d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f789ffff640 1 -- 172.21.15.81:0/2616755101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a014de80 con 0x7f78a00b9040 2024-06-29T14:38:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f78a48d8640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78a00b5250 0x7f78a014fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.953+0000 7f78a50d9640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78a00d2a90 0x7f78a014d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.954+0000 7f789ffff640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78a00b9040 0x7f78a014e190 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f788c00c3a0 tx=0x7f788c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.954+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c011790 con 0x7f78a00b9040 2024-06-29T14:38:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.954+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f788c017070 con 0x7f78a00b9040 2024-06-29T14:38:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.954+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c011cc0 con 0x7f78a00b9040 2024-06-29T14:38:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.954+0000 7f78a6b3c640 1 -- 172.21.15.81:0/2616755101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a014eac0 con 0x7f78a00b9040 2024-06-29T14:38:01.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.954+0000 7f78a6b3c640 1 -- 172.21.15.81:0/2616755101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a014f080 con 0x7f78a00b9040 2024-06-29T14:38:01.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.955+0000 7f78a6b3c640 1 -- 172.21.15.81:0/2616755101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a00b8440 con 0x7f78a00b9040 2024-06-29T14:38:01.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.956+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 39) ==== 98320+0+0 (secure 0 0 0) 0x7f788c01c460 con 0x7f78a00b9040 2024-06-29T14:38:01.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.956+0000 7f789d7fa640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 0x7f7884078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:01.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.956+0000 7f78a48d8640 1 -- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 msgr2=0x7f7884078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1959721003 2024-06-29T14:38:01.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.956+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5781+0+0 (secure 0 0 0) 0x7f788c097750 con 0x7f78a00b9040 2024-06-29T14:38:01.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.956+0000 7f78a48d8640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 0x7f7884078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:01.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:01.961+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f788c060870 con 0x7f78a00b9040 2024-06-29T14:38:02.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:02.157+0000 7f78a48d8640 1 -- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 msgr2=0x7f7884078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1959721003 2024-06-29T14:38:02.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:02.157+0000 7f78a48d8640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 0x7f7884078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:02.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:02.171+0000 7f78a6b3c640 1 -- 172.21.15.81:0/2616755101 --> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f78a014e7e0 con 0x7f7884076440 2024-06-29T14:38:02.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:02.559+0000 7f78a48d8640 1 -- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 msgr2=0x7f7884078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1959721003 2024-06-29T14:38:02.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:02.559+0000 7f78a48d8640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 0x7f7884078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:38:03.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:03.361+0000 7f78a48d8640 1 -- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 msgr2=0x7f7884078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1959721003 2024-06-29T14:38:03.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:03.361+0000 7f78a48d8640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 0x7f7884078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:38:04.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:04.765+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 40) ==== 97929+0+0 (secure 0 0 0) 0x7f788c095c20 con 0x7f78a00b9040 2024-06-29T14:38:04.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:04.765+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 msgr2=0x7f7884078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:04.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:04.765+0000 7f789d7fa640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1959721003,v1:172.21.15.167:6833/1959721003] conn(0x7f7884076440 0x7f7884078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:04.911 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:04.913+0000 7fe0be9fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:04.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:04.913+0000 7fe0be9fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:04.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:04.913+0000 7fe0be9fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:04.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:04.913+0000 7fe0be9fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:04.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:04.913+0000 7fe0be9fd640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:04.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:04.913+0000 7fe0bb9f7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:04.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:04.914+0000 7fe0bb9f7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:04.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:04.914+0000 7fe0bb9f7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:04.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:04.914+0000 7fe0bb9f7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:04.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:04.914+0000 7fe0bb9f7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:05.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.769+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 41) ==== 98056+0+0 (secure 0 0 0) 0x7f788c05cb00 con 0x7f78a00b9040 2024-06-29T14:38:05.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.770+0000 7f789d7fa640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f788403afd0 0x7f7884080430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:05.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.770+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 --> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f788c0999a0 con 0x7f788403afd0 2024-06-29T14:38:05.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.771+0000 7f78a48d8640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f788403afd0 0x7f7884080430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:05.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.773+0000 7f78a48d8640 1 --2- 172.21.15.81:0/2616755101 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f788403afd0 0x7f7884080430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f78940099d0 tx=0x7f7894005f90 comp rx=0 tx=0).ready entity=mgr.6092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:05.781 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:05.784+0000 7fe0c5a0b640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-06-29T14:38:05.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.783+0000 7f789d7fa640 1 -- 172.21.15.81:0/2616755101 <== mgr.6092 v2:172.21.15.167:6832/1921154940 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) ==== 49+0+0 (secure 0 0 0) 0x7f788c0999a0 con 0x7f788403afd0 2024-06-29T14:38:05.782 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-06-29T14:38:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:38:05.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:38:05.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.910+0000 7f2f881a5640 1 -- 172.21.15.81:0/2692076341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f80074520 msgr2=0x7f2f8006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:05.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.910+0000 7f2f881a5640 1 --2- 172.21.15.81:0/2692076341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f80074520 0x7f2f8006f140 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f2f8006e210 tx=0x7f2f7c02a5f0 comp rx=0 tx=0).stop 2024-06-29T14:38:05.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.910+0000 7f2f881a5640 1 -- 172.21.15.81:0/2692076341 shutdown_connections 2024-06-29T14:38:05.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.910+0000 7f2f881a5640 1 --2- 172.21.15.81:0/2692076341 >> v2:172.21.15.167:3300/0 conn(0x7f2f8006f680 0x7f2f8006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:05.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.910+0000 7f2f881a5640 1 --2- 172.21.15.81:0/2692076341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f80074520 0x7f2f8006f140 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:05.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.910+0000 7f2f881a5640 1 -- 172.21.15.81:0/2692076341 >> 172.21.15.81:0/2692076341 conn(0x7f2f80072950 msgr2=0x7f2f800b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:05.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.910+0000 7f2f881a5640 1 -- 172.21.15.81:0/2692076341 shutdown_connections 2024-06-29T14:38:05.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.910+0000 7f2f881a5640 1 -- 172.21.15.81:0/2692076341 wait complete. 2024-06-29T14:38:05.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.911+0000 7f2f881a5640 1 Processor -- start 2024-06-29T14:38:05.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.912+0000 7f2f881a5640 1 -- start start 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f881a5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f8006f680 0x7f2f801548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f881a5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f8006fa90 0x7f2f80152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f881a5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f80074520 0x7f2f8009dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f881a5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8015b010 con 0x7f2f8006fa90 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f881a5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8006d150 con 0x7f2f8006f680 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f881a5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2f8015b190 con 0x7f2f80074520 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f85740640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f8006fa90 0x7f2f80152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f85740640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f8006fa90 0x7f2f80152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43970/0 (socket says 172.21.15.81:43970) 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f85740640 1 -- 172.21.15.81:0/3705899401 learned_addr learned my addr 172.21.15.81:0/3705899401 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f85f41640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f8006f680 0x7f2f801548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f86742640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f80074520 0x7f2f8009dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:05.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f85740640 1 -- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f80074520 msgr2=0x7f2f8009dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f85740640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f80074520 0x7f2f8009dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f85740640 1 -- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f8006f680 msgr2=0x7f2f801548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f85740640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f8006f680 0x7f2f801548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f85740640 1 -- 172.21.15.81:0/3705899401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f80153810 con 0x7f2f8006fa90 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f86742640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f80074520 0x7f2f8009dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.913+0000 7f2f85740640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f8006fa90 0x7f2f80152f10 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000b440 tx=0x7f2f7000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.914+0000 7f2f767fc640 1 -- 172.21.15.81:0/3705899401 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70015020 con 0x7f2f8006fa90 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.914+0000 7f2f767fc640 1 -- 172.21.15.81:0/3705899401 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f70011920 con 0x7f2f8006fa90 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.914+0000 7f2f881a5640 1 -- 172.21.15.81:0/3705899401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f80153af0 con 0x7f2f8006fa90 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.914+0000 7f2f767fc640 1 -- 172.21.15.81:0/3705899401 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70011d70 con 0x7f2f8006fa90 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.914+0000 7f2f881a5640 1 -- 172.21.15.81:0/3705899401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f801540b0 con 0x7f2f8006fa90 2024-06-29T14:38:05.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.915+0000 7f2f881a5640 1 -- 172.21.15.81:0/3705899401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f800af4f0 con 0x7f2f8006fa90 2024-06-29T14:38:05.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.915+0000 7f2f767fc640 1 -- 172.21.15.81:0/3705899401 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 41) ==== 98056+0+0 (secure 0 0 0) 0x7f2f70018020 con 0x7f2f8006fa90 2024-06-29T14:38:05.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.916+0000 7f2f767fc640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2f640761a0 0x7f2f640786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:05.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.916+0000 7f2f767fc640 1 -- 172.21.15.81:0/3705899401 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7f2f70097480 con 0x7f2f8006fa90 2024-06-29T14:38:05.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.916+0000 7f2f85f41640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2f640761a0 0x7f2f640786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:05.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.917+0000 7f2f85f41640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2f640761a0 0x7f2f640786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c033000 tx=0x7f2f7c004730 comp rx=0 tx=0).ready entity=mgr.6092 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:05.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:05.921+0000 7f2f767fc640 1 -- 172.21.15.81:0/3705899401 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2f7001f020 con 0x7f2f8006fa90 2024-06-29T14:38:06.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.147+0000 7f2f881a5640 1 -- 172.21.15.81:0/3705899401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f2f800b5780 con 0x7f2f8006fa90 2024-06-29T14:38:06.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.774+0000 7f2f767fc640 1 -- 172.21.15.81:0/3705899401 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v42) ==== 93+0+0 (secure 0 0 0) 0x7f2f70060f80 con 0x7f2f8006fa90 2024-06-29T14:38:06.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 -- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2f640761a0 msgr2=0x7f2f640786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:06.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2f640761a0 0x7f2f640786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c033000 tx=0x7f2f7c004730 comp rx=0 tx=0).stop 2024-06-29T14:38:06.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 -- 172.21.15.81:0/3705899401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f8006fa90 msgr2=0x7f2f80152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:06.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f8006fa90 0x7f2f80152f10 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000b440 tx=0x7f2f7000b900 comp rx=0 tx=0).stop 2024-06-29T14:38:06.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 -- 172.21.15.81:0/3705899401 shutdown_connections 2024-06-29T14:38:06.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2f640761a0 0x7f2f640786b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:06.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f80074520 0x7f2f8009dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:06.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f8006fa90 0x7f2f80152f10 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:06.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 --2- 172.21.15.81:0/3705899401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f8006f680 0x7f2f801548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:06.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 -- 172.21.15.81:0/3705899401 >> 172.21.15.81:0/3705899401 conn(0x7f2f80072950 msgr2=0x7f2f801595f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:06.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 -- 172.21.15.81:0/3705899401 shutdown_connections 2024-06-29T14:38:06.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:06.777+0000 7f2f881a5640 1 -- 172.21.15.81:0/3705899401 wait complete. 2024-06-29T14:38:06.790 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:38:06.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:38:06.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:38:06.877 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:38:06.877 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:38:06.877 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:38:06.877 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:06.893 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:38:06.893 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:06.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:07.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.105+0000 7f46e7acc640 1 -- 172.21.15.81:0/968272258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46e00d0230 msgr2=0x7f46e00d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.105+0000 7f46e7acc640 1 --2- 172.21.15.81:0/968272258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46e00d0230 0x7f46e00d0640 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f46dc009860 tx=0x7f46dc02a5a0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.105+0000 7f46e7acc640 1 -- 172.21.15.81:0/968272258 shutdown_connections 2024-06-29T14:38:07.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.105+0000 7f46e7acc640 1 --2- 172.21.15.81:0/968272258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46e00d0230 0x7f46e00d0640 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.105+0000 7f46e7acc640 1 -- 172.21.15.81:0/968272258 >> 172.21.15.81:0/968272258 conn(0x7f46e00b3680 msgr2=0x7f46e00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.105+0000 7f46e7acc640 1 -- 172.21.15.81:0/968272258 shutdown_connections 2024-06-29T14:38:07.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.105+0000 7f46e7acc640 1 -- 172.21.15.81:0/968272258 wait complete. 2024-06-29T14:38:07.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.107+0000 7f46e7acc640 1 Processor -- start 2024-06-29T14:38:07.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.107+0000 7f46e7acc640 1 -- start start 2024-06-29T14:38:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e7acc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46e00b3200 0x7f46e009c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e7acc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e00b4f40 0x7f46e00b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e7acc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46e00d0230 0x7f46e0070050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e7acc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46e00b5e20 con 0x7f46e00d0230 2024-06-29T14:38:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e7acc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f46e00b5ca0 con 0x7f46e00b4f40 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e7acc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f46e00b5fa0 con 0x7f46e00b3200 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e6069640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46e00d0230 0x7f46e0070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e6069640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46e00d0230 0x7f46e0070050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43998/0 (socket says 172.21.15.81:43998) 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e5868640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46e00b3200 0x7f46e009c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e6069640 1 -- 172.21.15.81:0/2077457859 learned_addr learned my addr 172.21.15.81:0/2077457859 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.108+0000 7f46e5067640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e00b4f40 0x7f46e00b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46e6069640 1 -- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46e00b3200 msgr2=0x7f46e009c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46e6069640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46e00b3200 0x7f46e009c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46e6069640 1 -- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e00b4f40 msgr2=0x7f46e00b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46e6069640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e00b4f40 0x7f46e00b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46e6069640 1 -- 172.21.15.81:0/2077457859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e009ed40 con 0x7f46e00d0230 2024-06-29T14:38:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46e5067640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e00b4f40 0x7f46e00b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:07.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46e6069640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46e00d0230 0x7f46e0070050 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f46d800c6b0 tx=0x7f46d800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:07.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46d67fc640 1 -- 172.21.15.81:0/2077457859 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8003b70 con 0x7f46e00d0230 2024-06-29T14:38:07.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46d67fc640 1 -- 172.21.15.81:0/2077457859 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46d8003d30 con 0x7f46e00d0230 2024-06-29T14:38:07.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46e7acc640 1 -- 172.21.15.81:0/2077457859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e009f020 con 0x7f46e00d0230 2024-06-29T14:38:07.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46d67fc640 1 -- 172.21.15.81:0/2077457859 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8005330 con 0x7f46e00d0230 2024-06-29T14:38:07.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.109+0000 7f46e7acc640 1 -- 172.21.15.81:0/2077457859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e009f5e0 con 0x7f46e00d0230 2024-06-29T14:38:07.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.110+0000 7f46e7acc640 1 -- 172.21.15.81:0/2077457859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e00d0640 con 0x7f46e00d0230 2024-06-29T14:38:07.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.112+0000 7f46d67fc640 1 -- 172.21.15.81:0/2077457859 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f46d80054d0 con 0x7f46e00d0230 2024-06-29T14:38:07.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.112+0000 7f46d67fc640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f46c40763f0 0x7f46c4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.112+0000 7f46e5868640 1 -- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f46c40763f0 msgr2=0x7f46c4078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:07.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.112+0000 7f46d67fc640 1 -- 172.21.15.81:0/2077457859 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7f46d8097760 con 0x7f46e00d0230 2024-06-29T14:38:07.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.112+0000 7f46e5868640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f46c40763f0 0x7f46c4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:07.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.117+0000 7f46d67fc640 1 -- 172.21.15.81:0/2077457859 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f46d80607c0 con 0x7f46e00d0230 2024-06-29T14:38:07.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.301+0000 7f46e7acc640 1 -- 172.21.15.81:0/2077457859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46e015aec0 con 0x7f46e00d0230 2024-06-29T14:38:07.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.302+0000 7f46d67fc640 1 -- 172.21.15.81:0/2077457859 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+17278 (secure 0 0 0) 0x7f46d8064ac0 con 0x7f46e00d0230 2024-06-29T14:38:07.301 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:07.301 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":92,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:38:04.758114+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:50.753390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.410843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3028513012":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/3977971920":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/2335426752":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/4214568326":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/707343224":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/1821058896":"2024-06-30T14:38:04.758075+0000","172.21.15.167:6833/1959721003":"2024-06-30T14:38:04.758075+0000","172.21.15.167:6832/1959721003":"2024-06-30T14:38:04.758075+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:07.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 -- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f46c40763f0 msgr2=0x7f46c4078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:07.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f46c40763f0 0x7f46c4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 -- 172.21.15.81:0/2077457859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46e00d0230 msgr2=0x7f46e0070050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46e00d0230 0x7f46e0070050 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f46d800c6b0 tx=0x7f46d800cb70 comp rx=0 tx=0).stop 2024-06-29T14:38:07.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 -- 172.21.15.81:0/2077457859 shutdown_connections 2024-06-29T14:38:07.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f46c40763f0 0x7f46c4078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46e00d0230 0x7f46e0070050 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46e00b4f40 0x7f46e00b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 --2- 172.21.15.81:0/2077457859 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46e00b3200 0x7f46e009c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 -- 172.21.15.81:0/2077457859 >> 172.21.15.81:0/2077457859 conn(0x7f46e00b3680 msgr2=0x7f46e00cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 -- 172.21.15.81:0/2077457859 shutdown_connections 2024-06-29T14:38:07.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.305+0000 7f46e7acc640 1 -- 172.21.15.81:0/2077457859 wait complete. 2024-06-29T14:38:07.315 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:37:50.753390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:37:51.410843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:38:07.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:38:07.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.437+0000 7fb84a881640 1 -- 172.21.15.81:0/1831892275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb82c005260 msgr2=0x7fb82c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.437+0000 7fb84a881640 1 --2- 172.21.15.81:0/1831892275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb82c005260 0x7fb82c00b9c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fb84408cf40 tx=0x7fb82c005830 comp rx=0 tx=0).stop 2024-06-29T14:38:07.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.437+0000 7fb848e1e640 1 -- 172.21.15.81:0/1831892275 reap_dead start 2024-06-29T14:38:07.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.437+0000 7fb84a881640 1 -- 172.21.15.81:0/1831892275 shutdown_connections 2024-06-29T14:38:07.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.437+0000 7fb84a881640 1 -- 172.21.15.81:0/1831892275 >> 172.21.15.81:0/1831892275 conn(0x7fb84409cf30 msgr2=0x7fb8440d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.437+0000 7fb84a881640 1 -- 172.21.15.81:0/1831892275 shutdown_connections 2024-06-29T14:38:07.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.437+0000 7fb84a881640 1 -- 172.21.15.81:0/1831892275 wait complete. 2024-06-29T14:38:07.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.439+0000 7fb84a881640 1 Processor -- start 2024-06-29T14:38:07.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.439+0000 7fb84a881640 1 -- start start 2024-06-29T14:38:07.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.440+0000 7fb84a881640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb82c005260 0x7fb8440b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.440+0000 7fb84a881640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8441f4260 0x7fb8440a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.440+0000 7fb84a881640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb84409d9a0 0x7fb84409ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.440+0000 7fb84a881640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8440d0db0 con 0x7fb8441f4260 2024-06-29T14:38:07.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.440+0000 7fb84a881640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb8440b84b0 con 0x7fb82c005260 2024-06-29T14:38:07.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.440+0000 7fb84a881640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb8440d1a80 con 0x7fb84409d9a0 2024-06-29T14:38:07.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.440+0000 7fb8437fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8441f4260 0x7fb8440a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb8437fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8441f4260 0x7fb8440a1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44008/0 (socket says 172.21.15.81:44008) 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb8437fe640 1 -- 172.21.15.81:0/2361143033 learned_addr learned my addr 172.21.15.81:0/2361143033 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb843fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb82c005260 0x7fb8440b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb848e1e640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb84409d9a0 0x7fb84409ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb843fff640 1 -- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb84409d9a0 msgr2=0x7fb84409ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb843fff640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb84409d9a0 0x7fb84409ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb843fff640 1 -- 172.21.15.81:0/2361143033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8441f4260 msgr2=0x7fb8440a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb843fff640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8441f4260 0x7fb8440a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb843fff640 1 -- 172.21.15.81:0/2361143033 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8440a02c0 con 0x7fb82c005260 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb8437fe640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8441f4260 0x7fb8440a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb843fff640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb82c005260 0x7fb8440b4400 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb834000c00 tx=0x7fb834001800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb840ff9640 1 -- 172.21.15.81:0/2361143033 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834011640 con 0x7fb82c005260 2024-06-29T14:38:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb84a881640 1 -- 172.21.15.81:0/2361143033 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8440a05a0 con 0x7fb82c005260 2024-06-29T14:38:07.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb840ff9640 1 -- 172.21.15.81:0/2361143033 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb834016020 con 0x7fb82c005260 2024-06-29T14:38:07.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb840ff9640 1 -- 172.21.15.81:0/2361143033 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834011aa0 con 0x7fb82c005260 2024-06-29T14:38:07.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.441+0000 7fb84a881640 1 -- 172.21.15.81:0/2361143033 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8440a0b60 con 0x7fb82c005260 2024-06-29T14:38:07.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.443+0000 7fb84a881640 1 -- 172.21.15.81:0/2361143033 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8440d0480 con 0x7fb82c005260 2024-06-29T14:38:07.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.443+0000 7fb840ff9640 1 -- 172.21.15.81:0/2361143033 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7fb834011c40 con 0x7fb82c005260 2024-06-29T14:38:07.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.444+0000 7fb840ff9640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fb830076420 0x7fb830078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.444+0000 7fb8437fe640 1 -- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fb830076420 msgr2=0x7fb830078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:07.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.444+0000 7fb840ff9640 1 -- 172.21.15.81:0/2361143033 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7fb834098730 con 0x7fb82c005260 2024-06-29T14:38:07.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.444+0000 7fb8437fe640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fb830076420 0x7fb830078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:07.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.453+0000 7fb840ff9640 1 -- 172.21.15.81:0/2361143033 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb834020020 con 0x7fb82c005260 2024-06-29T14:38:07.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.638+0000 7fb84a881640 1 -- 172.21.15.81:0/2361143033 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb84415f160 con 0x7fb82c005260 2024-06-29T14:38:07.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.639+0000 7fb840ff9640 1 -- 172.21.15.81:0/2361143033 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7fb8340615e0 con 0x7fb82c005260 2024-06-29T14:38:07.638 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:38:07.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.641+0000 7fb84a881640 1 -- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fb830076420 msgr2=0x7fb830078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:07.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.641+0000 7fb84a881640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fb830076420 0x7fb830078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.641+0000 7fb84a881640 1 -- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb82c005260 msgr2=0x7fb8440b4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.641+0000 7fb84a881640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb82c005260 0x7fb8440b4400 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb834000c00 tx=0x7fb834001800 comp rx=0 tx=0).stop 2024-06-29T14:38:07.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.641+0000 7fb84a881640 1 -- 172.21.15.81:0/2361143033 shutdown_connections 2024-06-29T14:38:07.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.641+0000 7fb84a881640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fb830076420 0x7fb830078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.641+0000 7fb84a881640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb84409d9a0 0x7fb84409ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.641+0000 7fb84a881640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8441f4260 0x7fb8440a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.641+0000 7fb84a881640 1 --2- 172.21.15.81:0/2361143033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb82c005260 0x7fb8440b4400 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.641+0000 7fb84a881640 1 -- 172.21.15.81:0/2361143033 >> 172.21.15.81:0/2361143033 conn(0x7fb84409cf30 msgr2=0x7fb84406ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.642+0000 7fb84a881640 1 -- 172.21.15.81:0/2361143033 shutdown_connections 2024-06-29T14:38:07.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.642+0000 7fb84a881640 1 -- 172.21.15.81:0/2361143033 wait complete. 2024-06-29T14:38:07.652 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:38:07.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.776+0000 7f2b0e135640 1 -- 172.21.15.81:0/2932542564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b080b5250 msgr2=0x7f2b080d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.776+0000 7f2b0e135640 1 --2- 172.21.15.81:0/2932542564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b080b5250 0x7f2b080d09c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f2b0808e230 tx=0x7f2af002a680 comp rx=0 tx=0).stop 2024-06-29T14:38:07.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.776+0000 7f2b0e135640 1 -- 172.21.15.81:0/2932542564 shutdown_connections 2024-06-29T14:38:07.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.776+0000 7f2b0e135640 1 --2- 172.21.15.81:0/2932542564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b080b5250 0x7f2b080d09c0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.776+0000 7f2b0e135640 1 --2- 172.21.15.81:0/2932542564 >> v2:172.21.15.167:3300/0 conn(0x7f2b080b9040 0x7f2b080d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.776+0000 7f2b0e135640 1 -- 172.21.15.81:0/2932542564 >> 172.21.15.81:0/2932542564 conn(0x7f2b0809cf30 msgr2=0x7f2b080d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.776+0000 7f2b0e135640 1 -- 172.21.15.81:0/2932542564 shutdown_connections 2024-06-29T14:38:07.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.776+0000 7f2b0e135640 1 -- 172.21.15.81:0/2932542564 wait complete. 2024-06-29T14:38:07.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.779+0000 7f2b0e135640 1 Processor -- start 2024-06-29T14:38:07.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.779+0000 7f2b0e135640 1 -- start start 2024-06-29T14:38:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b0e135640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b080b5250 0x7f2b0815d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b0e135640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b080b7290 0x7f2b081e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b0e135640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b080b9040 0x7f2b081f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b0e135640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2b080d2e20 con 0x7f2b080b9040 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b0e135640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2b080d2ca0 con 0x7f2b080b5250 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b0e135640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2b080b7db0 con 0x7f2b080b7290 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b07fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b080b9040 0x7f2b081f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b06ffd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b080b7290 0x7f2b081e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b07fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b080b9040 0x7f2b081f55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44024/0 (socket says 172.21.15.81:44024) 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b07fff640 1 -- 172.21.15.81:0/3506049661 learned_addr learned my addr 172.21.15.81:0/3506049661 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b06ffd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b080b7290 0x7f2b081e82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:48956/0 (socket says 172.21.15.81:48956) 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b077fe640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b080b5250 0x7f2b0815d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b06ffd640 1 -- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b080b5250 msgr2=0x7f2b0815d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b06ffd640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b080b5250 0x7f2b0815d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b06ffd640 1 -- 172.21.15.81:0/3506049661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b080b9040 msgr2=0x7f2b081f55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b06ffd640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b080b9040 0x7f2b081f55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b06ffd640 1 -- 172.21.15.81:0/3506049661 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b081e7240 con 0x7f2b080b7290 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b077fe640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b080b5250 0x7f2b0815d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.781+0000 7f2b07fff640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b080b9040 0x7f2b081f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.782+0000 7f2b06ffd640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b080b7290 0x7f2b081e82a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2b080b9cc0 tx=0x7f2af00038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.782+0000 7f2afffff640 1 -- 172.21.15.81:0/3506049661 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af003e030 con 0x7f2b080b7290 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.782+0000 7f2afffff640 1 -- 172.21.15.81:0/3506049661 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2af0003d90 con 0x7f2b080b7290 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.782+0000 7f2b0e135640 1 -- 172.21.15.81:0/3506049661 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0815e540 con 0x7f2b080b7290 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.782+0000 7f2afffff640 1 -- 172.21.15.81:0/3506049661 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0035040 con 0x7f2b080b7290 2024-06-29T14:38:07.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.782+0000 7f2b0e135640 1 -- 172.21.15.81:0/3506049661 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b081e77c0 con 0x7f2b080b7290 2024-06-29T14:38:07.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.783+0000 7f2b0e135640 1 -- 172.21.15.81:0/3506049661 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b080b82f0 con 0x7f2b080b7290 2024-06-29T14:38:07.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.783+0000 7f2afffff640 1 -- 172.21.15.81:0/3506049661 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f2af0041030 con 0x7f2b080b7290 2024-06-29T14:38:07.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.784+0000 7f2afffff640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2ae80763f0 0x7f2ae8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:07.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.784+0000 7f2b077fe640 1 -- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2ae80763f0 msgr2=0x7f2ae8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:07.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.784+0000 7f2afffff640 1 -- 172.21.15.81:0/3506049661 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7f2af00b80e0 con 0x7f2b080b7290 2024-06-29T14:38:07.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.784+0000 7f2b077fe640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2ae80763f0 0x7f2ae8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:07.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.788+0000 7f2afffff640 1 -- 172.21.15.81:0/3506049661 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2af0081030 con 0x7f2b080b7290 2024-06-29T14:38:07.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.984+0000 7f2b0e135640 1 -- 172.21.15.81:0/3506049661 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2b080b3be0 con 0x7f2b080b7290 2024-06-29T14:38:07.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.985+0000 7f2afffff640 1 -- 172.21.15.81:0/3506049661 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f2af0085330 con 0x7f2b080b7290 2024-06-29T14:38:07.984 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:38:07.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.985+0000 7f2b077fe640 1 -- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2ae80763f0 msgr2=0x7f2ae8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:07.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.985+0000 7f2b077fe640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2ae80763f0 0x7f2ae8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:07.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.987+0000 7f2b0e135640 1 -- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2ae80763f0 msgr2=0x7f2ae8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:07.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.987+0000 7f2b0e135640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2ae80763f0 0x7f2ae8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.987+0000 7f2b0e135640 1 -- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b080b7290 msgr2=0x7f2b081e82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:07.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.987+0000 7f2b0e135640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b080b7290 0x7f2b081e82a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2b080b9cc0 tx=0x7f2af00038a0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.987+0000 7f2b0e135640 1 -- 172.21.15.81:0/3506049661 shutdown_connections 2024-06-29T14:38:07.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.987+0000 7f2b0e135640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f2ae80763f0 0x7f2ae8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.987+0000 7f2b0e135640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b080b9040 0x7f2b081f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.987+0000 7f2b0e135640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2b080b7290 0x7f2b081e82a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.987+0000 7f2b0e135640 1 --2- 172.21.15.81:0/3506049661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2b080b5250 0x7f2b0815d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:07.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.987+0000 7f2b0e135640 1 -- 172.21.15.81:0/3506049661 >> 172.21.15.81:0/3506049661 conn(0x7f2b0809cf30 msgr2=0x7f2b0806ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:07.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.988+0000 7f2b0e135640 1 -- 172.21.15.81:0/3506049661 shutdown_connections 2024-06-29T14:38:07.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:07.988+0000 7f2b0e135640 1 -- 172.21.15.81:0/3506049661 wait complete. 2024-06-29T14:38:07.997 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:38:08.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.119+0000 7f5aa33ac640 1 -- 172.21.15.81:0/2570611089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a9c0b5f50 msgr2=0x7f5a9c166870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.119+0000 7f5aa33ac640 1 --2- 172.21.15.81:0/2570611089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a9c0b5f50 0x7f5a9c166870 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5a94005fd0 tx=0x7f5a9402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.119+0000 7f5aa33ac640 1 -- 172.21.15.81:0/2570611089 shutdown_connections 2024-06-29T14:38:08.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.119+0000 7f5aa33ac640 1 --2- 172.21.15.81:0/2570611089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a9c0b5f50 0x7f5a9c166870 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.119+0000 7f5aa33ac640 1 -- 172.21.15.81:0/2570611089 >> 172.21.15.81:0/2570611089 conn(0x7f5a9c0b3680 msgr2=0x7f5a9c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:08.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.119+0000 7f5aa33ac640 1 -- 172.21.15.81:0/2570611089 shutdown_connections 2024-06-29T14:38:08.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.119+0000 7f5aa33ac640 1 -- 172.21.15.81:0/2570611089 wait complete. 2024-06-29T14:38:08.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.121+0000 7f5aa33ac640 1 Processor -- start 2024-06-29T14:38:08.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.121+0000 7f5aa33ac640 1 -- start start 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa33ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a9c0b3200 0x7f5a9c15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa33ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5a9c0b5f50 0x7f5a9c14e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa33ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a9c0d0230 0x7f5a9c0cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa33ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c1672d0 con 0x7f5a9c0b3200 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa33ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c167150 con 0x7f5a9c0d0230 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa33ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5a9c167450 con 0x7f5a9c0b5f50 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa1148640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a9c0b3200 0x7f5a9c15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa1148640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a9c0b3200 0x7f5a9c15bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44040/0 (socket says 172.21.15.81:44040) 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa1148640 1 -- 172.21.15.81:0/3930564440 learned_addr learned my addr 172.21.15.81:0/3930564440 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa1949640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a9c0d0230 0x7f5a9c0cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.122+0000 7f5aa0947640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5a9c0b5f50 0x7f5a9c14e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5aa1148640 1 -- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5a9c0b5f50 msgr2=0x7f5a9c14e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5aa1148640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5a9c0b5f50 0x7f5a9c14e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5aa1148640 1 -- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a9c0d0230 msgr2=0x7f5a9c0cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5aa1148640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a9c0d0230 0x7f5a9c0cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5aa1148640 1 -- 172.21.15.81:0/3930564440 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9c15a870 con 0x7f5a9c0b3200 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5aa1148640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a9c0b3200 0x7f5a9c15bd80 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800c3a0 tx=0x7f5a9800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5a8dffb640 1 -- 172.21.15.81:0/3930564440 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98017070 con 0x7f5a9c0b3200 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5aa33ac640 1 -- 172.21.15.81:0/3930564440 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a9c15ace0 con 0x7f5a9c0b3200 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5a8dffb640 1 -- 172.21.15.81:0/3930564440 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a980053c0 con 0x7f5a9c0b3200 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5a8dffb640 1 -- 172.21.15.81:0/3930564440 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98003850 con 0x7f5a9c0b3200 2024-06-29T14:38:08.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.123+0000 7f5aa33ac640 1 -- 172.21.15.81:0/3930564440 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a9c15b2a0 con 0x7f5a9c0b3200 2024-06-29T14:38:08.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.124+0000 7f5aa33ac640 1 -- 172.21.15.81:0/3930564440 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a9c0b7460 con 0x7f5a9c0b3200 2024-06-29T14:38:08.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.125+0000 7f5a8dffb640 1 -- 172.21.15.81:0/3930564440 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f5a9801a070 con 0x7f5a9c0b3200 2024-06-29T14:38:08.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.126+0000 7f5a8dffb640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f5a800763f0 0x7f5a80078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.126+0000 7f5aa0947640 1 -- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f5a800763f0 msgr2=0x7f5a80078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:08.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.126+0000 7f5a8dffb640 1 -- 172.21.15.81:0/3930564440 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7f5a98096d50 con 0x7f5a9c0b3200 2024-06-29T14:38:08.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.126+0000 7f5aa0947640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f5a800763f0 0x7f5a80078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:08.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.134+0000 7f5a8dffb640 1 -- 172.21.15.81:0/3930564440 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5a9c0b7460 con 0x7f5a9c0b3200 2024-06-29T14:38:08.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.322+0000 7f5aa33ac640 1 -- 172.21.15.81:0/3930564440 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5a9c0cc7b0 con 0x7f5a9c0b3200 2024-06-29T14:38:08.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.322+0000 7f5a8dffb640 1 -- 172.21.15.81:0/3930564440 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f5a9809a030 con 0x7f5a9c0b3200 2024-06-29T14:38:08.321 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:38:08.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 -- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f5a800763f0 msgr2=0x7f5a80078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:08.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f5a800763f0 0x7f5a80078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 -- 172.21.15.81:0/3930564440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a9c0b3200 msgr2=0x7f5a9c15bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a9c0b3200 0x7f5a9c15bd80 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800c3a0 tx=0x7f5a9800c860 comp rx=0 tx=0).stop 2024-06-29T14:38:08.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 -- 172.21.15.81:0/3930564440 shutdown_connections 2024-06-29T14:38:08.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f5a800763f0 0x7f5a80078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5a9c0d0230 0x7f5a9c0cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5a9c0b5f50 0x7f5a9c14e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 --2- 172.21.15.81:0/3930564440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a9c0b3200 0x7f5a9c15bd80 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 -- 172.21.15.81:0/3930564440 >> 172.21.15.81:0/3930564440 conn(0x7f5a9c0b3680 msgr2=0x7f5a9c0b55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:08.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 -- 172.21.15.81:0/3930564440 shutdown_connections 2024-06-29T14:38:08.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.324+0000 7f5aa33ac640 1 -- 172.21.15.81:0/3930564440 wait complete. 2024-06-29T14:38:08.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:38:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.452+0000 7f96b30fd640 1 -- 172.21.15.81:0/52819558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96ac0d2a90 msgr2=0x7f96ac0d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.452+0000 7f96b30fd640 1 --2- 172.21.15.81:0/52819558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96ac0d2a90 0x7f96ac0d09c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f96ac08e230 tx=0x7f96a402ab40 comp rx=0 tx=0).stop 2024-06-29T14:38:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.452+0000 7f96b30fd640 1 -- 172.21.15.81:0/52819558 shutdown_connections 2024-06-29T14:38:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.452+0000 7f96b30fd640 1 --2- 172.21.15.81:0/52819558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96ac0d2a90 0x7f96ac0d09c0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.452+0000 7f96b30fd640 1 --2- 172.21.15.81:0/52819558 >> v2:172.21.15.167:3300/0 conn(0x7f96ac0b9040 0x7f96ac0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.452+0000 7f96b30fd640 1 -- 172.21.15.81:0/52819558 >> 172.21.15.81:0/52819558 conn(0x7f96ac09cf30 msgr2=0x7f96ac0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.452+0000 7f96b30fd640 1 -- 172.21.15.81:0/52819558 shutdown_connections 2024-06-29T14:38:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.452+0000 7f96b30fd640 1 -- 172.21.15.81:0/52819558 wait complete. 2024-06-29T14:38:08.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.454+0000 7f96b30fd640 1 Processor -- start 2024-06-29T14:38:08.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.454+0000 7f96b30fd640 1 -- start start 2024-06-29T14:38:08.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b30fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96ac0b5250 0x7f96ac0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b30fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96ac0b9040 0x7f96ac0a1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b30fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96ac0d2a90 0x7f96ac0b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b30fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f96ac0b7d30 con 0x7f96ac0d2a90 2024-06-29T14:38:08.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b30fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f96ac0b7bb0 con 0x7f96ac0b5250 2024-06-29T14:38:08.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b30fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f96ac0b7eb0 con 0x7f96ac0b9040 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b169a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96ac0d2a90 0x7f96ac0b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b169a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96ac0d2a90 0x7f96ac0b2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44060/0 (socket says 172.21.15.81:44060) 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b169a640 1 -- 172.21.15.81:0/2830211772 learned_addr learned my addr 172.21.15.81:0/2830211772 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b0e99640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96ac0b5250 0x7f96ac0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96abfff640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96ac0b9040 0x7f96ac0a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b169a640 1 -- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96ac0b9040 msgr2=0x7f96ac0a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b169a640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96ac0b9040 0x7f96ac0a1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b169a640 1 -- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96ac0b5250 msgr2=0x7f96ac0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b169a640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96ac0b5250 0x7f96ac0cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b169a640 1 -- 172.21.15.81:0/2830211772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ac1596a0 con 0x7f96ac0d2a90 2024-06-29T14:38:08.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b169a640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96ac0d2a90 0x7f96ac0b2a00 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f96a4009d60 tx=0x7f96a402c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:08.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96a97fa640 1 -- 172.21.15.81:0/2830211772 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a4044070 con 0x7f96ac0d2a90 2024-06-29T14:38:08.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96b30fd640 1 -- 172.21.15.81:0/2830211772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96ac0cd210 con 0x7f96ac0d2a90 2024-06-29T14:38:08.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.456+0000 7f96a97fa640 1 -- 172.21.15.81:0/2830211772 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96a4004250 con 0x7f96ac0d2a90 2024-06-29T14:38:08.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.457+0000 7f96a97fa640 1 -- 172.21.15.81:0/2830211772 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a4004680 con 0x7f96ac0d2a90 2024-06-29T14:38:08.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.457+0000 7f96b30fd640 1 -- 172.21.15.81:0/2830211772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96ac09fc20 con 0x7f96ac0d2a90 2024-06-29T14:38:08.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.458+0000 7f96b30fd640 1 -- 172.21.15.81:0/2830211772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96ac09f6b0 con 0x7f96ac0d2a90 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.459+0000 7f96a97fa640 1 -- 172.21.15.81:0/2830211772 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f96a403f410 con 0x7f96ac0d2a90 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.459+0000 7f96a97fa640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f96900763f0 0x7f9690078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.459+0000 7f96a97fa640 1 -- 172.21.15.81:0/2830211772 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7f96a40b8650 con 0x7f96ac0d2a90 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.459+0000 7f96b0e99640 1 -- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f96900763f0 msgr2=0x7f9690078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:08.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.459+0000 7f96b0e99640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f96900763f0 0x7f9690078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.463+0000 7f96a97fa640 1 -- 172.21.15.81:0/2830211772 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f96a4032030 con 0x7f96ac0d2a90 2024-06-29T14:38:08.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.661+0000 7f96b0e99640 1 -- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f96900763f0 msgr2=0x7f9690078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:08.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.661+0000 7f96b0e99640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f96900763f0 0x7f9690078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:08.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.664+0000 7f96b30fd640 1 -- 172.21.15.81:0/2830211772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f96ac0b8870 con 0x7f96ac0d2a90 2024-06-29T14:38:08.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.664+0000 7f96a97fa640 1 -- 172.21.15.81:0/2830211772 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v92) ==== 80+0+477 (secure 0 0 0) 0x7f96a4004820 con 0x7f96ac0d2a90 2024-06-29T14:38:08.663 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3028513012 2024-06-30T14:38:04.758075+0000 2024-06-29T14:38:08.663 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3977971920 2024-06-30T14:38:04.758075+0000 2024-06-29T14:38:08.663 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2335426752 2024-06-30T14:38:04.758075+0000 2024-06-29T14:38:08.663 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/4214568326 2024-06-30T14:38:04.758075+0000 2024-06-29T14:38:08.663 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/707343224 2024-06-30T14:38:04.758075+0000 2024-06-29T14:38:08.664 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1821058896 2024-06-30T14:38:04.758075+0000 2024-06-29T14:38:08.664 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/1959721003 2024-06-30T14:38:04.758075+0000 2024-06-29T14:38:08.664 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/1959721003 2024-06-30T14:38:04.758075+0000 2024-06-29T14:38:08.664 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T14:38:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 -- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f96900763f0 msgr2=0x7f9690078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f96900763f0 0x7f9690078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 -- 172.21.15.81:0/2830211772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96ac0d2a90 msgr2=0x7f96ac0b2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96ac0d2a90 0x7f96ac0b2a00 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f96a4009d60 tx=0x7f96a402c8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 -- 172.21.15.81:0/2830211772 shutdown_connections 2024-06-29T14:38:08.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f96900763f0 0x7f9690078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96ac0d2a90 0x7f96ac0b2a00 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96ac0b9040 0x7f96ac0a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 --2- 172.21.15.81:0/2830211772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96ac0b5250 0x7f96ac0cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 -- 172.21.15.81:0/2830211772 >> 172.21.15.81:0/2830211772 conn(0x7f96ac09cf30 msgr2=0x7f96ac06cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:08.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 -- 172.21.15.81:0/2830211772 shutdown_connections 2024-06-29T14:38:08.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.667+0000 7f96b30fd640 1 -- 172.21.15.81:0/2830211772 wait complete. 2024-06-29T14:38:08.677 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:38:08.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:08.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:08.706 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:38:08.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:08.707 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:38:08.762 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:38:08.762 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:38:08.762 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:08.778 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:38:08.778 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:38:08.778 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:38:08.779 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:08.819 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:38:08.819 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:38:08.819 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:38:08.819 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:08.835 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:38:08.835 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:08.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.976+0000 7fe7fb111640 1 -- 172.21.15.81:0/2617144560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7f40b3200 msgr2=0x7fe7f40cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.976+0000 7fe7fb111640 1 --2- 172.21.15.81:0/2617144560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7f40b3200 0x7fe7f40cc7b0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8005fd0 tx=0x7fe7e802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.976+0000 7fe7fb111640 1 -- 172.21.15.81:0/2617144560 shutdown_connections 2024-06-29T14:38:08.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.976+0000 7fe7fb111640 1 --2- 172.21.15.81:0/2617144560 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7f40cccf0 0x7fe7f40d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.976+0000 7fe7fb111640 1 --2- 172.21.15.81:0/2617144560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7f40b3200 0x7fe7f40cc7b0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.976+0000 7fe7fb111640 1 -- 172.21.15.81:0/2617144560 >> 172.21.15.81:0/2617144560 conn(0x7fe7f40b3680 msgr2=0x7fe7f40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:08.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.976+0000 7fe7fb111640 1 -- 172.21.15.81:0/2617144560 shutdown_connections 2024-06-29T14:38:08.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.976+0000 7fe7fb111640 1 -- 172.21.15.81:0/2617144560 wait complete. 2024-06-29T14:38:08.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.978+0000 7fe7fb111640 1 Processor -- start 2024-06-29T14:38:08.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.978+0000 7fe7fb111640 1 -- start start 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7fb111640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7f40b3200 0x7fe7f4156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7fb111640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe7f40cccf0 0x7fe7f41e19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7fb111640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7f40d0770 0x7fe7f409dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7fb111640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f40cedb0 con 0x7fe7f40b3200 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7fb111640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f40cec30 con 0x7fe7f40cccf0 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7fb111640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe7f40cef30 con 0x7fe7f40d0770 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7f8ead640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7f40b3200 0x7fe7f4156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7f8ead640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7f40b3200 0x7fe7f4156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44088/0 (socket says 172.21.15.81:44088) 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7f8ead640 1 -- 172.21.15.81:0/317781608 learned_addr learned my addr 172.21.15.81:0/317781608 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7f3fff640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe7f40cccf0 0x7fe7f41e19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.979+0000 7fe7f96ae640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7f40d0770 0x7fe7f409dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7f8ead640 1 -- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7f40d0770 msgr2=0x7fe7f409dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7f8ead640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7f40d0770 0x7fe7f409dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7f8ead640 1 -- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe7f40cccf0 msgr2=0x7fe7f41e19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7f8ead640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe7f40cccf0 0x7fe7f41e19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7f8ead640 1 -- 172.21.15.81:0/317781608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f409f170 con 0x7fe7f40b3200 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7f8ead640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7f40b3200 0x7fe7f4156080 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8034040 tx=0x7fe7e80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7f17fa640 1 -- 172.21.15.81:0/317781608 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e803e030 con 0x7fe7f40b3200 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7f17fa640 1 -- 172.21.15.81:0/317781608 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7e8003ae0 con 0x7fe7f40b3200 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7fb111640 1 -- 172.21.15.81:0/317781608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f409ee90 con 0x7fe7f40b3200 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7f17fa640 1 -- 172.21.15.81:0/317781608 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8035040 con 0x7fe7f40b3200 2024-06-29T14:38:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.980+0000 7fe7fb111640 1 -- 172.21.15.81:0/317781608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f409d2c0 con 0x7fe7f40b3200 2024-06-29T14:38:08.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.981+0000 7fe7fb111640 1 -- 172.21.15.81:0/317781608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7f406a370 con 0x7fe7f40b3200 2024-06-29T14:38:08.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.982+0000 7fe7f17fa640 1 -- 172.21.15.81:0/317781608 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7fe7e8041030 con 0x7fe7f40b3200 2024-06-29T14:38:08.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.983+0000 7fe7f17fa640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fe7d8076420 0x7fe7d8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:08.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.983+0000 7fe7f3fff640 1 -- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fe7d8076420 msgr2=0x7fe7d8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:08.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.983+0000 7fe7f17fa640 1 -- 172.21.15.81:0/317781608 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7fe7e80b8550 con 0x7fe7f40b3200 2024-06-29T14:38:08.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.983+0000 7fe7f3fff640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fe7d8076420 0x7fe7d8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:08.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:08.990+0000 7fe7f17fa640 1 -- 172.21.15.81:0/317781608 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe7f406a370 con 0x7fe7f40b3200 2024-06-29T14:38:09.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.184+0000 7fe7f3fff640 1 -- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fe7d8076420 msgr2=0x7fe7d8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:09.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.184+0000 7fe7f3fff640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fe7d8076420 0x7fe7d8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.216+0000 7fe7fb111640 1 -- 172.21.15.81:0/317781608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7f40cc7b0 con 0x7fe7f40b3200 2024-06-29T14:38:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.217+0000 7fe7f17fa640 1 -- 172.21.15.81:0/317781608 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5225 (secure 0 0 0) 0x7fe7e80814f0 con 0x7fe7f40b3200 2024-06-29T14:38:09.215 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:09.215 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":42,"btime":"2024-06-29T14:37:56:110726+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6002,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3644828063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3644828063},{"type":"v1","addr":"172.21.15.167:6836","nonce":3644828063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":7026,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2532767666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2532767666},{"type":"v1","addr":"172.21.15.81:6835","nonce":2532767666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":7029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/818227798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":818227798},{"type":"v1","addr":"172.21.15.81:6834","nonce":818227798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.462608+0000","modified":"2024-06-29T14:37:55.478070+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"b","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/910129539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":910129539},{"type":"v1","addr":"172.21.15.167:6837","nonce":910129539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":5}]} 2024-06-29T14:38:09.216 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 42 2024-06-29T14:38:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 -- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fe7d8076420 msgr2=0x7fe7d8078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fe7d8076420 0x7fe7d8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 -- 172.21.15.81:0/317781608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7f40b3200 msgr2=0x7fe7f4156080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7f40b3200 0x7fe7f4156080 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8034040 tx=0x7fe7e80035f0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 -- 172.21.15.81:0/317781608 shutdown_connections 2024-06-29T14:38:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fe7d8076420 0x7fe7d8078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7f40d0770 0x7fe7f409dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe7f40cccf0 0x7fe7f41e19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 --2- 172.21.15.81:0/317781608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7f40b3200 0x7fe7f4156080 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 -- 172.21.15.81:0/317781608 >> 172.21.15.81:0/317781608 conn(0x7fe7f40b3680 msgr2=0x7fe7f41e8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 -- 172.21.15.81:0/317781608 shutdown_connections 2024-06-29T14:38:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.219+0000 7fe7fb111640 1 -- 172.21.15.81:0/317781608 wait complete. 2024-06-29T14:38:09.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.345+0000 7fc2e8c22640 1 -- 172.21.15.81:0/2342431629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e40b3200 msgr2=0x7fc2e40cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.345+0000 7fc2e8c22640 1 --2- 172.21.15.81:0/2342431629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e40b3200 0x7fc2e40cc7b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40c5590 tx=0x7fc2d802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.345+0000 7fc2e8c22640 1 -- 172.21.15.81:0/2342431629 shutdown_connections 2024-06-29T14:38:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.345+0000 7fc2e8c22640 1 --2- 172.21.15.81:0/2342431629 >> v2:172.21.15.167:3300/0 conn(0x7fc2e40d0940 0x7fc2e40b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.345+0000 7fc2e8c22640 1 --2- 172.21.15.81:0/2342431629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e40b3200 0x7fc2e40cc7b0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.345+0000 7fc2e8c22640 1 -- 172.21.15.81:0/2342431629 >> 172.21.15.81:0/2342431629 conn(0x7fc2e40b3680 msgr2=0x7fc2e40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.345+0000 7fc2e8c22640 1 -- 172.21.15.81:0/2342431629 shutdown_connections 2024-06-29T14:38:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.345+0000 7fc2e8c22640 1 -- 172.21.15.81:0/2342431629 wait complete. 2024-06-29T14:38:09.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.347+0000 7fc2e8c22640 1 Processor -- start 2024-06-29T14:38:09.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.347+0000 7fc2e8c22640 1 -- start start 2024-06-29T14:38:09.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e8c22640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2e40b3200 0x7fc2e40caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e8c22640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e40cccf0 0x7fc2e41d8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e8c22640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e40cffa0 0x7fc2e40d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e8c22640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e40b5bb0 con 0x7fc2e40cffa0 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e8c22640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e40b5a30 con 0x7fc2e40b3200 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e8c22640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc2e40b5d30 con 0x7fc2e40cccf0 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e40cffa0 0x7fc2e40d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e40cffa0 0x7fc2e40d03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44120/0 (socket says 172.21.15.81:44120) 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e2d9d640 1 -- 172.21.15.81:0/3185932811 learned_addr learned my addr 172.21.15.81:0/3185932811 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e259c640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2e40b3200 0x7fc2e40caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.348+0000 7fc2e1d9b640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e40cccf0 0x7fc2e41d8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2e2d9d640 1 -- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e40cccf0 msgr2=0x7fc2e41d8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2e2d9d640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e40cccf0 0x7fc2e41d8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2e2d9d640 1 -- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2e40b3200 msgr2=0x7fc2e40caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2e2d9d640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2e40b3200 0x7fc2e40caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2e2d9d640 1 -- 172.21.15.81:0/3185932811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e40d08f0 con 0x7fc2e40cffa0 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2e2d9d640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e40cffa0 0x7fc2e40d03b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e990 tx=0x7fc2dc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2d2ffd640 1 -- 172.21.15.81:0/3185932811 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc004d20 con 0x7fc2e40cffa0 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2d2ffd640 1 -- 172.21.15.81:0/3185932811 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2dc00cb00 con 0x7fc2e40cffa0 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2e8c22640 1 -- 172.21.15.81:0/3185932811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e40c9e60 con 0x7fc2e40cffa0 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2d2ffd640 1 -- 172.21.15.81:0/3185932811 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc005020 con 0x7fc2e40cffa0 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.349+0000 7fc2e8c22640 1 -- 172.21.15.81:0/3185932811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e40c95c0 con 0x7fc2e40cffa0 2024-06-29T14:38:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.350+0000 7fc2e8c22640 1 -- 172.21.15.81:0/3185932811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2e40b52a0 con 0x7fc2e40cffa0 2024-06-29T14:38:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.351+0000 7fc2d2ffd640 1 -- 172.21.15.81:0/3185932811 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7fc2dc0051c0 con 0x7fc2e40cffa0 2024-06-29T14:38:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.351+0000 7fc2d2ffd640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fc2c40763f0 0x7fc2c4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.351+0000 7fc2e259c640 1 -- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fc2c40763f0 msgr2=0x7fc2c4078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.352+0000 7fc2d2ffd640 1 -- 172.21.15.81:0/3185932811 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7fc2dc0983b0 con 0x7fc2e40cffa0 2024-06-29T14:38:09.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.352+0000 7fc2e259c640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fc2c40763f0 0x7fc2c4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:09.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.356+0000 7fc2d2ffd640 1 -- 172.21.15.81:0/3185932811 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc2e40b52a0 con 0x7fc2e40cffa0 2024-06-29T14:38:09.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.546+0000 7fc2e8c22640 1 -- 172.21.15.81:0/3185932811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2e40cc7b0 con 0x7fc2e40cffa0 2024-06-29T14:38:09.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.547+0000 7fc2d2ffd640 1 -- 172.21.15.81:0/3185932811 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+17278 (secure 0 0 0) 0x7fc2e40d08f0 con 0x7fc2e40cffa0 2024-06-29T14:38:09.546 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:09.546 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":92,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:38:04.758114+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:50.753390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.410843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3028513012":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/3977971920":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/2335426752":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/4214568326":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/707343224":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/1821058896":"2024-06-30T14:38:04.758075+0000","172.21.15.167:6833/1959721003":"2024-06-30T14:38:04.758075+0000","172.21.15.167:6832/1959721003":"2024-06-30T14:38:04.758075+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 -- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fc2c40763f0 msgr2=0x7fc2c4078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fc2c40763f0 0x7fc2c4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 -- 172.21.15.81:0/3185932811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e40cffa0 msgr2=0x7fc2e40d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e40cffa0 0x7fc2e40d03b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e990 tx=0x7fc2dc00ee50 comp rx=0 tx=0).stop 2024-06-29T14:38:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 -- 172.21.15.81:0/3185932811 shutdown_connections 2024-06-29T14:38:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7fc2c40763f0 0x7fc2c4078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e40cffa0 0x7fc2e40d03b0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e40cccf0 0x7fc2e41d8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 --2- 172.21.15.81:0/3185932811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2e40b3200 0x7fc2e40caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 -- 172.21.15.81:0/3185932811 >> 172.21.15.81:0/3185932811 conn(0x7fc2e40b3680 msgr2=0x7fc2e41e81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 -- 172.21.15.81:0/3185932811 shutdown_connections 2024-06-29T14:38:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.550+0000 7fc2e8c22640 1 -- 172.21.15.81:0/3185932811 wait complete. 2024-06-29T14:38:09.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:37:50.753390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:37:51.410843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:38:09.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:38:09.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.680+0000 7f031412a640 1 -- 172.21.15.81:0/3361178371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f030c073480 msgr2=0x7f030c15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.680+0000 7f031412a640 1 --2- 172.21.15.81:0/3361178371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f030c073480 0x7f030c15b6e0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f0308005fd0 tx=0x7f030802a540 comp rx=0 tx=0).stop 2024-06-29T14:38:09.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.680+0000 7f031412a640 1 -- 172.21.15.81:0/3361178371 shutdown_connections 2024-06-29T14:38:09.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.680+0000 7f031412a640 1 --2- 172.21.15.81:0/3361178371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f030c073480 0x7f030c15b6e0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.680+0000 7f031412a640 1 --2- 172.21.15.81:0/3361178371 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c0cd150 0x7f030c072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.680+0000 7f031412a640 1 -- 172.21.15.81:0/3361178371 >> 172.21.15.81:0/3361178371 conn(0x7f030c0b2920 msgr2=0x7f030c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:09.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.680+0000 7f031412a640 1 -- 172.21.15.81:0/3361178371 shutdown_connections 2024-06-29T14:38:09.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.680+0000 7f031412a640 1 -- 172.21.15.81:0/3361178371 wait complete. 2024-06-29T14:38:09.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.683+0000 7f031412a640 1 Processor -- start 2024-06-29T14:38:09.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.683+0000 7f031412a640 1 -- start start 2024-06-29T14:38:09.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.685+0000 7f031412a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f030c073480 0x7f030c0d0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.685+0000 7f031412a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f030c0cd150 0x7f030c1532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.685+0000 7f031412a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c15bc20 0x7f030c157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.685+0000 7f031412a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f030c15c1b0 con 0x7f030c073480 2024-06-29T14:38:09.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.685+0000 7f031412a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f030c15c030 con 0x7f030c0cd150 2024-06-29T14:38:09.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.685+0000 7f031412a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f030c15f060 con 0x7f030c15bc20 2024-06-29T14:38:09.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f0311ec6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f030c073480 0x7f030c0d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f0311ec6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f030c073480 0x7f030c0d0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44138/0 (socket says 172.21.15.81:44138) 2024-06-29T14:38:09.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f0311ec6640 1 -- 172.21.15.81:0/3809632443 learned_addr learned my addr 172.21.15.81:0/3809632443 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f03126c7640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c15bc20 0x7f030c157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f03116c5640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f030c0cd150 0x7f030c1532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f0311ec6640 1 -- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c15bc20 msgr2=0x7f030c157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f0311ec6640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c15bc20 0x7f030c157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f0311ec6640 1 -- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f030c0cd150 msgr2=0x7f030c1532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f0311ec6640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f030c0cd150 0x7f030c1532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f0311ec6640 1 -- 172.21.15.81:0/3809632443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030c0ce530 con 0x7f030c073480 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f0311ec6640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f030c073480 0x7f030c0d0090 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f030000d740 tx=0x7f030000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f02fe7fc640 1 -- 172.21.15.81:0/3809632443 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0300017070 con 0x7f030c073480 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f02fe7fc640 1 -- 172.21.15.81:0/3809632443 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0300009d20 con 0x7f030c073480 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f02fe7fc640 1 -- 172.21.15.81:0/3809632443 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03000106c0 con 0x7f030c073480 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f031412a640 1 -- 172.21.15.81:0/3809632443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f030c0ce810 con 0x7f030c073480 2024-06-29T14:38:09.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.686+0000 7f031412a640 1 -- 172.21.15.81:0/3809632443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f030c0cedd0 con 0x7f030c073480 2024-06-29T14:38:09.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.687+0000 7f031412a640 1 -- 172.21.15.81:0/3809632443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f030c15f580 con 0x7f030c073480 2024-06-29T14:38:09.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.688+0000 7f02fe7fc640 1 -- 172.21.15.81:0/3809632443 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f030001a070 con 0x7f030c073480 2024-06-29T14:38:09.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.689+0000 7f02fe7fc640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f02f0076420 0x7f02f0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:09.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.689+0000 7f03116c5640 1 -- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f02f0076420 msgr2=0x7f02f0078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:09.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.689+0000 7f02fe7fc640 1 -- 172.21.15.81:0/3809632443 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7f03000980b0 con 0x7f030c073480 2024-06-29T14:38:09.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.689+0000 7f03116c5640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f02f0076420 0x7f02f0078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:09.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.697+0000 7f02fe7fc640 1 -- 172.21.15.81:0/3809632443 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0300060fe0 con 0x7f030c073480 2024-06-29T14:38:09.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.882+0000 7f031412a640 1 -- 172.21.15.81:0/3809632443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f030c15e0c0 con 0x7f030c073480 2024-06-29T14:38:09.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.883+0000 7f02fe7fc640 1 -- 172.21.15.81:0/3809632443 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f03000652e0 con 0x7f030c073480 2024-06-29T14:38:09.881 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:38:09.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 -- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f02f0076420 msgr2=0x7f02f0078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:09.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f02f0076420 0x7f02f0078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 -- 172.21.15.81:0/3809632443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f030c073480 msgr2=0x7f030c0d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:09.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f030c073480 0x7f030c0d0090 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f030000d740 tx=0x7f030000dc00 comp rx=0 tx=0).stop 2024-06-29T14:38:09.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 -- 172.21.15.81:0/3809632443 shutdown_connections 2024-06-29T14:38:09.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f02f0076420 0x7f02f0078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f030c15bc20 0x7f030c157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f030c0cd150 0x7f030c1532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 --2- 172.21.15.81:0/3809632443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f030c073480 0x7f030c0d0090 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:09.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 -- 172.21.15.81:0/3809632443 >> 172.21.15.81:0/3809632443 conn(0x7f030c0b2920 msgr2=0x7f030c0709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:09.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 -- 172.21.15.81:0/3809632443 shutdown_connections 2024-06-29T14:38:09.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:09.885+0000 7f031412a640 1 -- 172.21.15.81:0/3809632443 wait complete. 2024-06-29T14:38:09.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:38:10.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.016+0000 7f74a0363640 1 -- 172.21.15.81:0/2327606885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74980d0770 msgr2=0x7f74980ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.016+0000 7f74a0363640 1 --2- 172.21.15.81:0/2327606885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74980d0770 0x7f74980ce520 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f7498073550 tx=0x7f749002aa30 comp rx=0 tx=0).stop 2024-06-29T14:38:10.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.016+0000 7f74a0363640 1 -- 172.21.15.81:0/2327606885 shutdown_connections 2024-06-29T14:38:10.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.016+0000 7f74a0363640 1 --2- 172.21.15.81:0/2327606885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74980d0770 0x7f74980ce520 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.016+0000 7f74a0363640 1 --2- 172.21.15.81:0/2327606885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74980cccf0 0x7f74980d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.016+0000 7f74a0363640 1 --2- 172.21.15.81:0/2327606885 >> v2:172.21.15.167:3300/0 conn(0x7f74980b3200 0x7f74980cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.016+0000 7f74a0363640 1 -- 172.21.15.81:0/2327606885 >> 172.21.15.81:0/2327606885 conn(0x7f74980b3680 msgr2=0x7f74980b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.016+0000 7f74a0363640 1 -- 172.21.15.81:0/2327606885 shutdown_connections 2024-06-29T14:38:10.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.016+0000 7f74a0363640 1 -- 172.21.15.81:0/2327606885 wait complete. 2024-06-29T14:38:10.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.018+0000 7f74a0363640 1 Processor -- start 2024-06-29T14:38:10.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.018+0000 7f74a0363640 1 -- start start 2024-06-29T14:38:10.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f74a0363640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74980b3200 0x7f749809bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f74a0363640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74980cccf0 0x7f7498159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f74a0363640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74980d0770 0x7f7498155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f74a0363640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74980cef90 con 0x7f74980cccf0 2024-06-29T14:38:10.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f74a0363640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f74980cee10 con 0x7f74980b3200 2024-06-29T14:38:10.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f74a0363640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f74980cf110 con 0x7f74980d0770 2024-06-29T14:38:10.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f749d8fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74980cccf0 0x7f7498159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f749d8fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74980cccf0 0x7f7498159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36272/0 (socket says 172.21.15.81:36272) 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f749d8fe640 1 -- 172.21.15.81:0/954026781 learned_addr learned my addr 172.21.15.81:0/954026781 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f749e0ff640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74980b3200 0x7f749809bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f749e900640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74980d0770 0x7f7498155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f749d8fe640 1 -- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74980d0770 msgr2=0x7f7498155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f749d8fe640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74980d0770 0x7f7498155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f749d8fe640 1 -- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74980b3200 msgr2=0x7f749809bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f749d8fe640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74980b3200 0x7f749809bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.019+0000 7f749d8fe640 1 -- 172.21.15.81:0/954026781 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7498156b10 con 0x7f74980cccf0 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.020+0000 7f749d8fe640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74980cccf0 0x7f7498159aa0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f749400d770 tx=0x7f749400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.020+0000 7f748effd640 1 -- 172.21.15.81:0/954026781 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494017070 con 0x7f74980cccf0 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.020+0000 7f748effd640 1 -- 172.21.15.81:0/954026781 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7494009d20 con 0x7f74980cccf0 2024-06-29T14:38:10.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.020+0000 7f74a0363640 1 -- 172.21.15.81:0/954026781 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7498156db0 con 0x7f74980cccf0 2024-06-29T14:38:10.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.020+0000 7f748effd640 1 -- 172.21.15.81:0/954026781 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74940106c0 con 0x7f74980cccf0 2024-06-29T14:38:10.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.020+0000 7f74a0363640 1 -- 172.21.15.81:0/954026781 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7498157370 con 0x7f74980cccf0 2024-06-29T14:38:10.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.021+0000 7f74a0363640 1 -- 172.21.15.81:0/954026781 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74980d0230 con 0x7f74980cccf0 2024-06-29T14:38:10.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.022+0000 7f748effd640 1 -- 172.21.15.81:0/954026781 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7f749401a070 con 0x7f74980cccf0 2024-06-29T14:38:10.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.022+0000 7f748effd640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f747c0763f0 0x7f747c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.022+0000 7f749e0ff640 1 -- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f747c0763f0 msgr2=0x7f747c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:10.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.022+0000 7f748effd640 1 -- 172.21.15.81:0/954026781 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7f7494097b00 con 0x7f74980cccf0 2024-06-29T14:38:10.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.022+0000 7f749e0ff640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f747c0763f0 0x7f747c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:10.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.031+0000 7f748effd640 1 -- 172.21.15.81:0/954026781 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7494060ae0 con 0x7f74980cccf0 2024-06-29T14:38:10.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.217+0000 7f74a0363640 1 -- 172.21.15.81:0/954026781 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f74981e77c0 con 0x7f74980cccf0 2024-06-29T14:38:10.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.218+0000 7f748effd640 1 -- 172.21.15.81:0/954026781 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f7494064de0 con 0x7f74980cccf0 2024-06-29T14:38:10.216 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:38:10.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 -- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f747c0763f0 msgr2=0x7f747c078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f747c0763f0 0x7f747c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 -- 172.21.15.81:0/954026781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74980cccf0 msgr2=0x7f7498159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74980cccf0 0x7f7498159aa0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f749400d770 tx=0x7f749400dc30 comp rx=0 tx=0).stop 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 -- 172.21.15.81:0/954026781 shutdown_connections 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7f747c0763f0 0x7f747c078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74980d0770 0x7f7498155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74980cccf0 0x7f7498159aa0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 --2- 172.21.15.81:0/954026781 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74980b3200 0x7f749809bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 -- 172.21.15.81:0/954026781 >> 172.21.15.81:0/954026781 conn(0x7f74980b3680 msgr2=0x7f74980cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 -- 172.21.15.81:0/954026781 shutdown_connections 2024-06-29T14:38:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.220+0000 7f74a0363640 1 -- 172.21.15.81:0/954026781 wait complete. 2024-06-29T14:38:10.230 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:38:10.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.348+0000 7efe7c4ac640 1 -- 172.21.15.81:0/235478933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe740b9040 msgr2=0x7efe740d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.348+0000 7efe7c4ac640 1 --2- 172.21.15.81:0/235478933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe740b9040 0x7efe740d0480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efe70009860 tx=0x7efe7002a620 comp rx=0 tx=0).stop 2024-06-29T14:38:10.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.349+0000 7efe7c4ac640 1 -- 172.21.15.81:0/235478933 shutdown_connections 2024-06-29T14:38:10.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.349+0000 7efe7c4ac640 1 --2- 172.21.15.81:0/235478933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe740b9040 0x7efe740d0480 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.349+0000 7efe7c4ac640 1 -- 172.21.15.81:0/235478933 >> 172.21.15.81:0/235478933 conn(0x7efe7409cf30 msgr2=0x7efe740d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.349+0000 7efe7c4ac640 1 -- 172.21.15.81:0/235478933 shutdown_connections 2024-06-29T14:38:10.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.349+0000 7efe7c4ac640 1 -- 172.21.15.81:0/235478933 wait complete. 2024-06-29T14:38:10.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.350+0000 7efe7c4ac640 1 Processor -- start 2024-06-29T14:38:10.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.350+0000 7efe7c4ac640 1 -- start start 2024-06-29T14:38:10.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe7c4ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe740b5250 0x7efe7409e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe7c4ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740b9040 0x7efe7414fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe7c4ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe740d2a90 0x7efe740cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe7c4ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe740d0e00 con 0x7efe740b9040 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe7c4ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efe740d0c80 con 0x7efe740d2a90 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe7c4ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efe740b80b0 con 0x7efe740b5250 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe79a47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740b9040 0x7efe7414fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe79a47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740b9040 0x7efe7414fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36274/0 (socket says 172.21.15.81:36274) 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe79a47640 1 -- 172.21.15.81:0/2709224644 learned_addr learned my addr 172.21.15.81:0/2709224644 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe7a248640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe740b5250 0x7efe7409e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe7aa49640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe740d2a90 0x7efe740cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe79a47640 1 -- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe740b5250 msgr2=0x7efe7409e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe79a47640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe740b5250 0x7efe7409e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe79a47640 1 -- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe740d2a90 msgr2=0x7efe740cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe79a47640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe740d2a90 0x7efe740cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe79a47640 1 -- 172.21.15.81:0/2709224644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe74150b00 con 0x7efe740b9040 2024-06-29T14:38:10.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe79a47640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740b9040 0x7efe7414fd40 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7efe6800c390 tx=0x7efe6800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:10.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.352+0000 7efe66ffd640 1 -- 172.21.15.81:0/2709224644 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68011790 con 0x7efe740b9040 2024-06-29T14:38:10.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.353+0000 7efe66ffd640 1 -- 172.21.15.81:0/2709224644 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe68017070 con 0x7efe740b9040 2024-06-29T14:38:10.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.353+0000 7efe7c4ac640 1 -- 172.21.15.81:0/2709224644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe74150de0 con 0x7efe740b9040 2024-06-29T14:38:10.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.353+0000 7efe66ffd640 1 -- 172.21.15.81:0/2709224644 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe68011cc0 con 0x7efe740b9040 2024-06-29T14:38:10.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.353+0000 7efe7c4ac640 1 -- 172.21.15.81:0/2709224644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe741513a0 con 0x7efe740b9040 2024-06-29T14:38:10.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.354+0000 7efe7c4ac640 1 -- 172.21.15.81:0/2709224644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe740d0480 con 0x7efe740b9040 2024-06-29T14:38:10.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.354+0000 7efe66ffd640 1 -- 172.21.15.81:0/2709224644 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 42) ==== 98304+0+0 (secure 0 0 0) 0x7efe6801c460 con 0x7efe740b9040 2024-06-29T14:38:10.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.355+0000 7efe66ffd640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7efe58076420 0x7efe58078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.355+0000 7efe7a248640 1 -- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7efe58076420 msgr2=0x7efe58078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1921154940 2024-06-29T14:38:10.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.355+0000 7efe66ffd640 1 -- 172.21.15.81:0/2709224644 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6125+0+0 (secure 0 0 0) 0x7efe68097820 con 0x7efe740b9040 2024-06-29T14:38:10.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.355+0000 7efe7a248640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7efe58076420 0x7efe58078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:10.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.359+0000 7efe66ffd640 1 -- 172.21.15.81:0/2709224644 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efe68060770 con 0x7efe740b9040 2024-06-29T14:38:10.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.431+0000 7efe66ffd640 1 -- 172.21.15.81:0/2709224644 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 43) ==== 97912+0+0 (secure 0 0 0) 0x7efe68095ba0 con 0x7efe740b9040 2024-06-29T14:38:10.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.432+0000 7efe66ffd640 1 -- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7efe58076420 msgr2=0x7efe58078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:10.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.432+0000 7efe66ffd640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7efe58076420 0x7efe58078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.477 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:10.479+0000 7f8322b3c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.477 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:10.479+0000 7f8322b3c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.477 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:10.479+0000 7f8322b3c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.477 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:10.479+0000 7f8322b3c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.477 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:10.479+0000 7f8322b3c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.478 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:10.479+0000 7f8325b42640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.478 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:10.479+0000 7f8325b42640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.478 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:10.479+0000 7f8325b42640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.478 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:10.479+0000 7f8325b42640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.478 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:10.479+0000 7f8325b42640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:10.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.556+0000 7efe7c4ac640 1 -- 172.21.15.81:0/2709224644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efe74150850 con 0x7efe740b9040 2024-06-29T14:38:10.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.556+0000 7efe66ffd640 1 -- 172.21.15.81:0/2709224644 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7efe6806bed0 con 0x7efe740b9040 2024-06-29T14:38:10.555 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:38:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.558+0000 7efe7c4ac640 1 -- 172.21.15.81:0/2709224644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740b9040 msgr2=0x7efe7414fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.558+0000 7efe7c4ac640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740b9040 0x7efe7414fd40 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7efe6800c390 tx=0x7efe6800c850 comp rx=0 tx=0).stop 2024-06-29T14:38:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.558+0000 7efe7c4ac640 1 -- 172.21.15.81:0/2709224644 shutdown_connections 2024-06-29T14:38:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.558+0000 7efe7c4ac640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:6832/1921154940,v1:172.21.15.167:6833/1921154940] conn(0x7efe58076420 0x7efe58078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.558+0000 7efe7c4ac640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe740d2a90 0x7efe740cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.558+0000 7efe7c4ac640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740b9040 0x7efe7414fd40 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.558+0000 7efe7c4ac640 1 --2- 172.21.15.81:0/2709224644 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe740b5250 0x7efe7409e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.558+0000 7efe7c4ac640 1 -- 172.21.15.81:0/2709224644 >> 172.21.15.81:0/2709224644 conn(0x7efe7409cf30 msgr2=0x7efe7406dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.558+0000 7efe7c4ac640 1 -- 172.21.15.81:0/2709224644 shutdown_connections 2024-06-29T14:38:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.558+0000 7efe7c4ac640 1 -- 172.21.15.81:0/2709224644 wait complete. 2024-06-29T14:38:10.569 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:10.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.684+0000 7fa903752640 1 -- 172.21.15.81:0/828177207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8fc06f850 msgr2=0x7fa8fc0ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.684+0000 7fa903752640 1 --2- 172.21.15.81:0/828177207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8fc06f850 0x7fa8fc0ae300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009860 tx=0x7fa8ec02a5a0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.685+0000 7fa903752640 1 -- 172.21.15.81:0/828177207 shutdown_connections 2024-06-29T14:38:10.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.685+0000 7fa903752640 1 --2- 172.21.15.81:0/828177207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8fc06f850 0x7fa8fc0ae300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.685+0000 7fa903752640 1 -- 172.21.15.81:0/828177207 >> 172.21.15.81:0/828177207 conn(0x7fa8fc072950 msgr2=0x7fa8fc0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.685+0000 7fa903752640 1 -- 172.21.15.81:0/828177207 shutdown_connections 2024-06-29T14:38:10.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.685+0000 7fa903752640 1 -- 172.21.15.81:0/828177207 wait complete. 2024-06-29T14:38:10.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.686+0000 7fa903752640 1 Processor -- start 2024-06-29T14:38:10.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.686+0000 7fa903752640 1 -- start start 2024-06-29T14:38:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa903752640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8fc06f850 0x7fa8fc1f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa903752640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8fc074520 0x7fa8fc1f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa903752640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8fc0ae840 0x7fa8fc0d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa903752640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa8fc15b190 con 0x7fa8fc074520 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa903752640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa8fc15b010 con 0x7fa8fc06f850 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa903752640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa8fc0af360 con 0x7fa8fc0ae840 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa900ced640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8fc074520 0x7fa8fc1f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa900ced640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8fc074520 0x7fa8fc1f28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36282/0 (socket says 172.21.15.81:36282) 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa900ced640 1 -- 172.21.15.81:0/997533619 learned_addr learned my addr 172.21.15.81:0/997533619 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa901cef640 1 --2- 172.21.15.81:0/997533619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8fc0ae840 0x7fa8fc0d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa9014ee640 1 --2- 172.21.15.81:0/997533619 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8fc06f850 0x7fa8fc1f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa900ced640 1 -- 172.21.15.81:0/997533619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8fc0ae840 msgr2=0x7fa8fc0d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa900ced640 1 --2- 172.21.15.81:0/997533619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8fc0ae840 0x7fa8fc0d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa900ced640 1 -- 172.21.15.81:0/997533619 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8fc06f850 msgr2=0x7fa8fc1f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa900ced640 1 --2- 172.21.15.81:0/997533619 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8fc06f850 0x7fa8fc1f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa900ced640 1 -- 172.21.15.81:0/997533619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8fc1521e0 con 0x7fa8fc074520 2024-06-29T14:38:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa9014ee640 1 --2- 172.21.15.81:0/997533619 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8fc06f850 0x7fa8fc1f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa900ced640 1 --2- 172.21.15.81:0/997533619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8fc074520 0x7fa8fc1f28f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec000c00 tx=0x7fa8ec033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa8f1ffb640 1 -- 172.21.15.81:0/997533619 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec031d60 con 0x7fa8fc074520 2024-06-29T14:38:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa8f1ffb640 1 -- 172.21.15.81:0/997533619 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8ec033e10 con 0x7fa8fc074520 2024-06-29T14:38:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa8f1ffb640 1 -- 172.21.15.81:0/997533619 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec033310 con 0x7fa8fc074520 2024-06-29T14:38:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.688+0000 7fa903752640 1 -- 172.21.15.81:0/997533619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8fc1524a0 con 0x7fa8fc074520 2024-06-29T14:38:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.689+0000 7fa903752640 1 -- 172.21.15.81:0/997533619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8fc152a60 con 0x7fa8fc074520 2024-06-29T14:38:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.690+0000 7fa903752640 1 -- 172.21.15.81:0/997533619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8fc0ae300 con 0x7fa8fc074520 2024-06-29T14:38:10.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.690+0000 7fa8f1ffb640 1 -- 172.21.15.81:0/997533619 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 43) ==== 97912+0+0 (secure 0 0 0) 0x7fa8ec0334b0 con 0x7fa8fc074520 2024-06-29T14:38:10.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.691+0000 7fa8f1ffb640 1 -- 172.21.15.81:0/997533619 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6469+0+0 (secure 0 0 0) 0x7fa8ec0b8490 con 0x7fa8fc074520 2024-06-29T14:38:10.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.695+0000 7fa8f1ffb640 1 -- 172.21.15.81:0/997533619 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa8ec04a030 con 0x7fa8fc074520 2024-06-29T14:38:10.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.914+0000 7fa903752640 1 -- 172.21.15.81:0/997533619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8fc1530b0 con 0x7fa8fc074520 2024-06-29T14:38:10.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.915+0000 7fa8f1ffb640 1 -- 172.21.15.81:0/997533619 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5225 (secure 0 0 0) 0x7fa8ec03f030 con 0x7fa8fc074520 2024-06-29T14:38:10.914 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:10.914 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":42,"btime":"2024-06-29T14:37:56:110726+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6002,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3644828063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3644828063},{"type":"v1","addr":"172.21.15.167:6836","nonce":3644828063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":7026,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2532767666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2532767666},{"type":"v1","addr":"172.21.15.81:6835","nonce":2532767666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":7029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/818227798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":818227798},{"type":"v1","addr":"172.21.15.81:6834","nonce":818227798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.462608+0000","modified":"2024-06-29T14:37:55.478070+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"b","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/910129539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":910129539},{"type":"v1","addr":"172.21.15.167:6837","nonce":910129539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":5}]} 2024-06-29T14:38:10.914 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 42 2024-06-29T14:38:10.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.917+0000 7fa903752640 1 -- 172.21.15.81:0/997533619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8fc074520 msgr2=0x7fa8fc1f28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:10.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.917+0000 7fa903752640 1 --2- 172.21.15.81:0/997533619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8fc074520 0x7fa8fc1f28f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec000c00 tx=0x7fa8ec033c20 comp rx=0 tx=0).stop 2024-06-29T14:38:10.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.917+0000 7fa903752640 1 -- 172.21.15.81:0/997533619 shutdown_connections 2024-06-29T14:38:10.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.917+0000 7fa903752640 1 --2- 172.21.15.81:0/997533619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8fc0ae840 0x7fa8fc0d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.917+0000 7fa903752640 1 --2- 172.21.15.81:0/997533619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8fc074520 0x7fa8fc1f28f0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.917+0000 7fa903752640 1 --2- 172.21.15.81:0/997533619 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8fc06f850 0x7fa8fc1f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:10.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.917+0000 7fa903752640 1 -- 172.21.15.81:0/997533619 >> 172.21.15.81:0/997533619 conn(0x7fa8fc072950 msgr2=0x7fa8fc06c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:10.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.917+0000 7fa903752640 1 -- 172.21.15.81:0/997533619 shutdown_connections 2024-06-29T14:38:10.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:10.917+0000 7fa903752640 1 -- 172.21.15.81:0/997533619 wait complete. 2024-06-29T14:38:10.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.041+0000 7f52fa810640 1 -- 172.21.15.81:0/3139651712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52dc004800 msgr2=0x7f52dc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.041+0000 7f52fa810640 1 --2- 172.21.15.81:0/3139651712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52dc004800 0x7f52dc004c10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f52e80016c0 tx=0x7f52dc005830 comp rx=0 tx=0).stop 2024-06-29T14:38:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.041+0000 7f52fa810640 1 -- 172.21.15.81:0/3139651712 shutdown_connections 2024-06-29T14:38:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.041+0000 7f52fa810640 1 --2- 172.21.15.81:0/3139651712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52dc004800 0x7f52dc004c10 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.041+0000 7f52fa810640 1 -- 172.21.15.81:0/3139651712 >> 172.21.15.81:0/3139651712 conn(0x7f52f40b3510 msgr2=0x7f52f40b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.041+0000 7f52fa810640 1 -- 172.21.15.81:0/3139651712 shutdown_connections 2024-06-29T14:38:11.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.041+0000 7f52fa810640 1 -- 172.21.15.81:0/3139651712 wait complete. 2024-06-29T14:38:11.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.043+0000 7f52fa810640 1 Processor -- start 2024-06-29T14:38:11.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.043+0000 7f52fa810640 1 -- start start 2024-06-29T14:38:11.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52fa810640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52dc004800 0x7f52f41df6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52fa810640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52f41f8d70 0x7f52f40c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52fa810640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52f40c9bb0 0x7f52f406e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52fa810640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f52f40d0660 con 0x7f52dc004800 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52fa810640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f52f40d04e0 con 0x7f52f41f8d70 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52fa810640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f52f40d07e0 con 0x7f52f40c9bb0 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52dc004800 0x7f52f41df6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52dc004800 0x7f52f41df6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36302/0 (socket says 172.21.15.81:36302) 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f3fff640 1 -- 172.21.15.81:0/341732736 learned_addr learned my addr 172.21.15.81:0/341732736 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f8dad640 1 --2- 172.21.15.81:0/341732736 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52f40c9bb0 0x7f52f406e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f37fe640 1 --2- 172.21.15.81:0/341732736 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52f41f8d70 0x7f52f40c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f3fff640 1 -- 172.21.15.81:0/341732736 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52f40c9bb0 msgr2=0x7f52f406e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f3fff640 1 --2- 172.21.15.81:0/341732736 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52f40c9bb0 0x7f52f406e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f3fff640 1 -- 172.21.15.81:0/341732736 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52f41f8d70 msgr2=0x7f52f40c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f3fff640 1 --2- 172.21.15.81:0/341732736 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52f41f8d70 0x7f52f40c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f3fff640 1 -- 172.21.15.81:0/341732736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f40c9fc0 con 0x7f52dc004800 2024-06-29T14:38:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f3fff640 1 --2- 172.21.15.81:0/341732736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52dc004800 0x7f52f41df6e0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f52e4011ae0 tx=0x7f52e4011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:11.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f0ff9640 1 -- 172.21.15.81:0/341732736 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e401e070 con 0x7f52dc004800 2024-06-29T14:38:11.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.045+0000 7f52f0ff9640 1 -- 172.21.15.81:0/341732736 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52e400fe70 con 0x7f52dc004800 2024-06-29T14:38:11.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.046+0000 7f52f0ff9640 1 -- 172.21.15.81:0/341732736 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e4019690 con 0x7f52dc004800 2024-06-29T14:38:11.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.046+0000 7f52fa810640 1 -- 172.21.15.81:0/341732736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f41e5a20 con 0x7f52dc004800 2024-06-29T14:38:11.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.046+0000 7f52fa810640 1 -- 172.21.15.81:0/341732736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f406dc70 con 0x7f52dc004800 2024-06-29T14:38:11.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.046+0000 7f52fa810640 1 -- 172.21.15.81:0/341732736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f40b49d0 con 0x7f52dc004800 2024-06-29T14:38:11.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.047+0000 7f52f0ff9640 1 -- 172.21.15.81:0/341732736 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 43) ==== 97912+0+0 (secure 0 0 0) 0x7f52e4021070 con 0x7f52dc004800 2024-06-29T14:38:11.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.048+0000 7f52f0ff9640 1 -- 172.21.15.81:0/341732736 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6469+0+0 (secure 0 0 0) 0x7f52e4097fb0 con 0x7f52dc004800 2024-06-29T14:38:11.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.052+0000 7f52f0ff9640 1 -- 172.21.15.81:0/341732736 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f52f40c9fc0 con 0x7f52dc004800 2024-06-29T14:38:11.232 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:38:11.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.234+0000 7f52fa810640 1 -- 172.21.15.81:0/341732736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52f40cdf00 con 0x7f52dc004800 2024-06-29T14:38:11.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.235+0000 7f52f0ff9640 1 -- 172.21.15.81:0/341732736 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+17787 (secure 0 0 0) 0x7f52e4061410 con 0x7f52dc004800 2024-06-29T14:38:11.234 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:11.234 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":93,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:38:10.421159+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:50.753390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.410843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3236355037":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/227247568":"2024-06-30T14:38:10.421119+0000","172.21.15.167:6833/1921154940":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/1196105043":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/1751731779":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/1307625241":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/3028513012":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/2042442118":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/3977971920":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/2335426752":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/4214568326":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/707343224":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/1821058896":"2024-06-30T14:38:04.758075+0000","172.21.15.167:6832/1921154940":"2024-06-30T14:38:10.421119+0000","172.21.15.167:6833/1959721003":"2024-06-30T14:38:04.758075+0000","172.21.15.167:6832/1959721003":"2024-06-30T14:38:04.758075+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:11.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:38:11.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.237+0000 7f52fa810640 1 -- 172.21.15.81:0/341732736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52dc004800 msgr2=0x7f52f41df6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.237+0000 7f52fa810640 1 --2- 172.21.15.81:0/341732736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52dc004800 0x7f52f41df6e0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f52e4011ae0 tx=0x7f52e4011ee0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.237+0000 7f52fa810640 1 -- 172.21.15.81:0/341732736 shutdown_connections 2024-06-29T14:38:11.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.237+0000 7f52fa810640 1 --2- 172.21.15.81:0/341732736 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52f40c9bb0 0x7f52f406e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.237+0000 7f52fa810640 1 --2- 172.21.15.81:0/341732736 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52f41f8d70 0x7f52f40c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.237+0000 7f52fa810640 1 --2- 172.21.15.81:0/341732736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52dc004800 0x7f52f41df6e0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.237+0000 7f52fa810640 1 -- 172.21.15.81:0/341732736 >> 172.21.15.81:0/341732736 conn(0x7f52f40b3510 msgr2=0x7f52f40b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:11.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.237+0000 7f52fa810640 1 -- 172.21.15.81:0/341732736 shutdown_connections 2024-06-29T14:38:11.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.237+0000 7f52fa810640 1 -- 172.21.15.81:0/341732736 wait complete. 2024-06-29T14:38:11.247 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:38:11.248 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:11.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.439+0000 7f87566c0640 1 -- 172.21.15.81:0/1930007520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87500b5250 msgr2=0x7f87500d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.439+0000 7f87566c0640 1 --2- 172.21.15.81:0/1930007520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87500b5250 0x7f87500d09c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f875008e230 tx=0x7f874402a610 comp rx=0 tx=0).stop 2024-06-29T14:38:11.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.439+0000 7f87566c0640 1 -- 172.21.15.81:0/1930007520 shutdown_connections 2024-06-29T14:38:11.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.439+0000 7f87566c0640 1 --2- 172.21.15.81:0/1930007520 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87500b7290 0x7f87500b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.439+0000 7f87566c0640 1 --2- 172.21.15.81:0/1930007520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87500b5250 0x7f87500d09c0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.439+0000 7f87566c0640 1 --2- 172.21.15.81:0/1930007520 >> v2:172.21.15.167:3300/0 conn(0x7f87500b9040 0x7f87500d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.439+0000 7f87566c0640 1 -- 172.21.15.81:0/1930007520 >> 172.21.15.81:0/1930007520 conn(0x7f875009cf30 msgr2=0x7f87500d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:11.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.439+0000 7f87566c0640 1 -- 172.21.15.81:0/1930007520 shutdown_connections 2024-06-29T14:38:11.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.439+0000 7f87566c0640 1 -- 172.21.15.81:0/1930007520 wait complete. 2024-06-29T14:38:11.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.441+0000 7f87566c0640 1 Processor -- start 2024-06-29T14:38:11.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.441+0000 7f87566c0640 1 -- start start 2024-06-29T14:38:11.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f87566c0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87500b5250 0x7f875009e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f87566c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87500b7290 0x7f87500cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f87566c0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87500b9040 0x7f8750160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f87566c0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87500b7f70 con 0x7f87500b7290 2024-06-29T14:38:11.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f87566c0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f87500b7df0 con 0x7f87500b9040 2024-06-29T14:38:11.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f87566c0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f87500b80f0 con 0x7f87500b5250 2024-06-29T14:38:11.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f874f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87500b7290 0x7f87500cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f874f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87500b7290 0x7f87500cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36326/0 (socket says 172.21.15.81:36326) 2024-06-29T14:38:11.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f874f7fe640 1 -- 172.21.15.81:0/617271758 learned_addr learned my addr 172.21.15.81:0/617271758 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f874ffff640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87500b5250 0x7f875009e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.442+0000 7f8754c5d640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87500b9040 0x7f8750160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f874f7fe640 1 -- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87500b5250 msgr2=0x7f875009e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f874f7fe640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87500b5250 0x7f875009e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f874f7fe640 1 -- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87500b9040 msgr2=0x7f8750160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f874f7fe640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87500b9040 0x7f8750160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f874f7fe640 1 -- 172.21.15.81:0/617271758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87500a0280 con 0x7f87500b7290 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f8754c5d640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87500b9040 0x7f8750160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f874f7fe640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87500b7290 0x7f87500cda50 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f8744034040 tx=0x7f87440038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f874cff9640 1 -- 172.21.15.81:0/617271758 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874403e030 con 0x7f87500b7290 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f874cff9640 1 -- 172.21.15.81:0/617271758 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8744003d90 con 0x7f87500b7290 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f87566c0640 1 -- 172.21.15.81:0/617271758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f875009ec30 con 0x7f87500b7290 2024-06-29T14:38:11.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f874cff9640 1 -- 172.21.15.81:0/617271758 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744035040 con 0x7f87500b7290 2024-06-29T14:38:11.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.443+0000 7f87566c0640 1 -- 172.21.15.81:0/617271758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87500a0800 con 0x7f87500b7290 2024-06-29T14:38:11.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.444+0000 7f87566c0640 1 -- 172.21.15.81:0/617271758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87500b76a0 con 0x7f87500b7290 2024-06-29T14:38:11.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.445+0000 7f874cff9640 1 -- 172.21.15.81:0/617271758 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 44) ==== 98039+0+0 (secure 0 0 0) 0x7f8744041030 con 0x7f87500b7290 2024-06-29T14:38:11.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.445+0000 7f874cff9640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f873c076120 0x7f873c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.446+0000 7f874cff9640 1 -- 172.21.15.81:0/617271758 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6469+0+0 (secure 0 0 0) 0x7f87440b7ca0 con 0x7f87500b7290 2024-06-29T14:38:11.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.446+0000 7f874ffff640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f873c076120 0x7f873c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.446+0000 7f874ffff640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f873c076120 0x7f873c078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8740009b30 tx=0x7f8740005f90 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:11.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.450+0000 7f874cff9640 1 -- 172.21.15.81:0/617271758 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8744080ff0 con 0x7f87500b7290 2024-06-29T14:38:11.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.666+0000 7f87566c0640 1 -- 172.21.15.81:0/617271758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8750155380 con 0x7f87500b7290 2024-06-29T14:38:11.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.667+0000 7f874cff9640 1 -- 172.21.15.81:0/617271758 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5225 (secure 0 0 0) 0x7f87440852f0 con 0x7f87500b7290 2024-06-29T14:38:11.665 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:11.665 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":42,"btime":"2024-06-29T14:37:56:110726+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6002,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3644828063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3644828063},{"type":"v1","addr":"172.21.15.167:6836","nonce":3644828063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":7026,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2532767666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2532767666},{"type":"v1","addr":"172.21.15.81:6835","nonce":2532767666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":7029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/818227798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":818227798},{"type":"v1","addr":"172.21.15.81:6834","nonce":818227798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.462608+0000","modified":"2024-06-29T14:37:55.478070+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"b","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/910129539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":910129539},{"type":"v1","addr":"172.21.15.167:6837","nonce":910129539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":5}]} 2024-06-29T14:38:11.666 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 42 2024-06-29T14:38:11.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 -- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f873c076120 msgr2=0x7f873c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f873c076120 0x7f873c078630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8740009b30 tx=0x7f8740005f90 comp rx=0 tx=0).stop 2024-06-29T14:38:11.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 -- 172.21.15.81:0/617271758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87500b7290 msgr2=0x7f87500cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87500b7290 0x7f87500cda50 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f8744034040 tx=0x7f87440038a0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 -- 172.21.15.81:0/617271758 shutdown_connections 2024-06-29T14:38:11.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f873c076120 0x7f873c078630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87500b9040 0x7f8750160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87500b7290 0x7f87500cda50 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 --2- 172.21.15.81:0/617271758 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87500b5250 0x7f875009e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 -- 172.21.15.81:0/617271758 >> 172.21.15.81:0/617271758 conn(0x7f875009cf30 msgr2=0x7f87501610b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:11.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 -- 172.21.15.81:0/617271758 shutdown_connections 2024-06-29T14:38:11.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.669+0000 7f87566c0640 1 -- 172.21.15.81:0/617271758 wait complete. 2024-06-29T14:38:11.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:11.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.793+0000 7f509cc81640 1 -- 172.21.15.81:0/2728254018 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5098159bd0 msgr2=0x7f50980d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.793+0000 7f509cc81640 1 --2- 172.21.15.81:0/2728254018 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5098159bd0 0x7f50980d0010 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5088009830 tx=0x7f508802a510 comp rx=0 tx=0).stop 2024-06-29T14:38:11.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.793+0000 7f509cc81640 1 -- 172.21.15.81:0/2728254018 shutdown_connections 2024-06-29T14:38:11.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.793+0000 7f509cc81640 1 --2- 172.21.15.81:0/2728254018 >> v2:172.21.15.167:3300/0 conn(0x7f50980d0550 0x7f509806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.793+0000 7f509cc81640 1 --2- 172.21.15.81:0/2728254018 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5098159bd0 0x7f50980d0010 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.793+0000 7f509cc81640 1 -- 172.21.15.81:0/2728254018 >> 172.21.15.81:0/2728254018 conn(0x7f50980b2920 msgr2=0x7f50980b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:11.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.793+0000 7f509cc81640 1 -- 172.21.15.81:0/2728254018 shutdown_connections 2024-06-29T14:38:11.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.793+0000 7f509cc81640 1 -- 172.21.15.81:0/2728254018 wait complete. 2024-06-29T14:38:11.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.795+0000 7f509cc81640 1 Processor -- start 2024-06-29T14:38:11.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.795+0000 7f509cc81640 1 -- start start 2024-06-29T14:38:11.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f509cc81640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f509806c830 0x7f5098141bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f509cc81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50980d0550 0x7f509813c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f509cc81640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5098159bd0 0x7f509813c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f509cc81640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f509815fc10 con 0x7f50980d0550 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f509cc81640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f509815fa90 con 0x7f5098159bd0 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f509cc81640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f509815fd90 con 0x7f509806c830 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5095d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50980d0550 0x7f509813c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5095d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50980d0550 0x7f509813c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36330/0 (socket says 172.21.15.81:36330) 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5095d9b640 1 -- 172.21.15.81:0/1599297134 learned_addr learned my addr 172.21.15.81:0/1599297134 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5096d9d640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5098159bd0 0x7f509813c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f509659c640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f509806c830 0x7f5098141bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5095d9b640 1 -- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f509806c830 msgr2=0x7f5098141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5095d9b640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f509806c830 0x7f5098141bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5095d9b640 1 -- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5098159bd0 msgr2=0x7f509813c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5095d9b640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5098159bd0 0x7f509813c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:11.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5095d9b640 1 -- 172.21.15.81:0/1599297134 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509813e010 con 0x7f50980d0550 2024-06-29T14:38:11.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5095d9b640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50980d0550 0x7f509813c100 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f508c00ea20 tx=0x7f508c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:11.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.797+0000 7f5086ffd640 1 -- 172.21.15.81:0/1599297134 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c00ccf0 con 0x7f50980d0550 2024-06-29T14:38:11.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.798+0000 7f5086ffd640 1 -- 172.21.15.81:0/1599297134 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f508c00ce90 con 0x7f50980d0550 2024-06-29T14:38:11.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.798+0000 7f509cc81640 1 -- 172.21.15.81:0/1599297134 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509813e2f0 con 0x7f50980d0550 2024-06-29T14:38:11.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.798+0000 7f5086ffd640 1 -- 172.21.15.81:0/1599297134 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c004260 con 0x7f50980d0550 2024-06-29T14:38:11.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.798+0000 7f509cc81640 1 -- 172.21.15.81:0/1599297134 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f509813f960 con 0x7f50980d0550 2024-06-29T14:38:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.798+0000 7f509cc81640 1 -- 172.21.15.81:0/1599297134 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5098160240 con 0x7f50980d0550 2024-06-29T14:38:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.799+0000 7f5086ffd640 1 -- 172.21.15.81:0/1599297134 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 44) ==== 98039+0+0 (secure 0 0 0) 0x7f508c0035f0 con 0x7f50980d0550 2024-06-29T14:38:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.800+0000 7f5086ffd640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5078076150 0x7f5078078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.800+0000 7f5086ffd640 1 -- 172.21.15.81:0/1599297134 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6469+0+0 (secure 0 0 0) 0x7f508c097e40 con 0x7f50980d0550 2024-06-29T14:38:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.800+0000 7f509659c640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5078076150 0x7f5078078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.800+0000 7f509659c640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5078076150 0x7f5078078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5088034000 tx=0x7f5088004730 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:11.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:11.804+0000 7f5086ffd640 1 -- 172.21.15.81:0/1599297134 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f508c061110 con 0x7f50980d0550 2024-06-29T14:38:12.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.023+0000 7f509cc81640 1 -- 172.21.15.81:0/1599297134 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f509806f330 con 0x7f50980d0550 2024-06-29T14:38:12.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.024+0000 7f5086ffd640 1 -- 172.21.15.81:0/1599297134 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5225 (secure 0 0 0) 0x7f508c065410 con 0x7f50980d0550 2024-06-29T14:38:12.022 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:12.022 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":42,"btime":"2024-06-29T14:37:56:110726+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6002,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3644828063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3644828063},{"type":"v1","addr":"172.21.15.167:6836","nonce":3644828063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":7026,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2532767666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2532767666},{"type":"v1","addr":"172.21.15.81:6835","nonce":2532767666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":7029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/818227798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":818227798},{"type":"v1","addr":"172.21.15.81:6834","nonce":818227798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:37:52.462608+0000","modified":"2024-06-29T14:37:55.478070+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7041":{"gid":7041,"name":"b","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/910129539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":910129539},{"type":"v1","addr":"172.21.15.167:6837","nonce":910129539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7041,"qdb_cluster":[7041]},"id":5}]} 2024-06-29T14:38:12.023 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 42 2024-06-29T14:38:12.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 -- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5078076150 msgr2=0x7f5078078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5078076150 0x7f5078078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5088034000 tx=0x7f5088004730 comp rx=0 tx=0).stop 2024-06-29T14:38:12.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 -- 172.21.15.81:0/1599297134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50980d0550 msgr2=0x7f509813c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50980d0550 0x7f509813c100 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f508c00ea20 tx=0x7f508c00eee0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 -- 172.21.15.81:0/1599297134 shutdown_connections 2024-06-29T14:38:12.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5078076150 0x7f5078078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5098159bd0 0x7f509813c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50980d0550 0x7f509813c100 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 --2- 172.21.15.81:0/1599297134 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f509806c830 0x7f5098141bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 -- 172.21.15.81:0/1599297134 >> 172.21.15.81:0/1599297134 conn(0x7f50980b2920 msgr2=0x7f50980d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 -- 172.21.15.81:0/1599297134 shutdown_connections 2024-06-29T14:38:12.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.026+0000 7f509cc81640 1 -- 172.21.15.81:0/1599297134 wait complete. 2024-06-29T14:38:12.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:12.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.149+0000 7f201f01f640 1 -- 172.21.15.81:0/599125699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f201806ef90 msgr2=0x7f201806f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.149+0000 7f201f01f640 1 --2- 172.21.15.81:0/599125699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f201806ef90 0x7f201806f3a0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2008009830 tx=0x7f200802a510 comp rx=0 tx=0).stop 2024-06-29T14:38:12.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.149+0000 7f201f01f640 1 -- 172.21.15.81:0/599125699 shutdown_connections 2024-06-29T14:38:12.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.149+0000 7f201f01f640 1 --2- 172.21.15.81:0/599125699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f201806ef90 0x7f201806f3a0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.149+0000 7f201f01f640 1 --2- 172.21.15.81:0/599125699 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20180b57b0 0x7f201815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.149+0000 7f201f01f640 1 -- 172.21.15.81:0/599125699 >> 172.21.15.81:0/599125699 conn(0x7f20180d0930 msgr2=0x7f20180d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.150+0000 7f201f01f640 1 -- 172.21.15.81:0/599125699 shutdown_connections 2024-06-29T14:38:12.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.150+0000 7f201f01f640 1 -- 172.21.15.81:0/599125699 wait complete. 2024-06-29T14:38:12.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.151+0000 7f201f01f640 1 Processor -- start 2024-06-29T14:38:12.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.151+0000 7f201f01f640 1 -- start start 2024-06-29T14:38:12.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201f01f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f201806ef90 0x7f20181f5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201f01f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20180b57b0 0x7f20180a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201f01f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20180ce1d0 0x7f2018154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201f01f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f20180cecf0 con 0x7f201806ef90 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201f01f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f20180b4b80 con 0x7f20180b57b0 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201f01f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f20180cee70 con 0x7f20180ce1d0 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201cdbb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f201806ef90 0x7f20181f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201cdbb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f201806ef90 0x7f20181f5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36338/0 (socket says 172.21.15.81:36338) 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201cdbb640 1 -- 172.21.15.81:0/1130963936 learned_addr learned my addr 172.21.15.81:0/1130963936 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201cdbb640 1 -- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20180ce1d0 msgr2=0x7f2018154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201d5bc640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20180ce1d0 0x7f2018154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201cdbb640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20180ce1d0 0x7f2018154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f200ffff640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20180b57b0 0x7f20180a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201cdbb640 1 -- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20180b57b0 msgr2=0x7f20180a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201cdbb640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20180b57b0 0x7f20180a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201cdbb640 1 -- 172.21.15.81:0/1130963936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20181f86a0 con 0x7f201806ef90 2024-06-29T14:38:12.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f201cdbb640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f201806ef90 0x7f20181f5720 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f201400e7c0 tx=0x7f201400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.153+0000 7f200d7fa640 1 -- 172.21.15.81:0/1130963936 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201400ccc0 con 0x7f201806ef90 2024-06-29T14:38:12.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.154+0000 7f200d7fa640 1 -- 172.21.15.81:0/1130963936 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2014010430 con 0x7f201806ef90 2024-06-29T14:38:12.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.154+0000 7f201f01f640 1 -- 172.21.15.81:0/1130963936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20181f8980 con 0x7f201806ef90 2024-06-29T14:38:12.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.154+0000 7f200d7fa640 1 -- 172.21.15.81:0/1130963936 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014010860 con 0x7f201806ef90 2024-06-29T14:38:12.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.154+0000 7f201f01f640 1 -- 172.21.15.81:0/1130963936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20181f8f40 con 0x7f201806ef90 2024-06-29T14:38:12.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.155+0000 7f201f01f640 1 -- 172.21.15.81:0/1130963936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201806e2a0 con 0x7f201806ef90 2024-06-29T14:38:12.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.155+0000 7f200d7fa640 1 -- 172.21.15.81:0/1130963936 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 44) ==== 98039+0+0 (secure 0 0 0) 0x7f2014010a00 con 0x7f201806ef90 2024-06-29T14:38:12.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.156+0000 7f200d7fa640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f2004076120 0x7f2004078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.156+0000 7f200d7fa640 1 -- 172.21.15.81:0/1130963936 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6469+0+0 (secure 0 0 0) 0x7f20140980f0 con 0x7f201806ef90 2024-06-29T14:38:12.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.156+0000 7f200ffff640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f2004076120 0x7f2004078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.157+0000 7f200ffff640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f2004076120 0x7f2004078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2008034000 tx=0x7f2008004730 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.160+0000 7f200d7fa640 1 -- 172.21.15.81:0/1130963936 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f201806e2a0 con 0x7f201806ef90 2024-06-29T14:38:12.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.338+0000 7f201f01f640 1 -- 172.21.15.81:0/1130963936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f201806f3a0 con 0x7f201806ef90 2024-06-29T14:38:12.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.339+0000 7f200d7fa640 1 -- 172.21.15.81:0/1130963936 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+17787 (secure 0 0 0) 0x7f2014061a00 con 0x7f201806ef90 2024-06-29T14:38:12.338 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:12.338 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":93,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:38:10.421159+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:37:50.753390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-06-29T14:37:51.410843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3236355037":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/227247568":"2024-06-30T14:38:10.421119+0000","172.21.15.167:6833/1921154940":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/1196105043":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/1751731779":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/1307625241":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/3028513012":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/2042442118":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/3977971920":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/2335426752":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/4214568326":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/707343224":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/1821058896":"2024-06-30T14:38:04.758075+0000","172.21.15.167:6832/1921154940":"2024-06-30T14:38:10.421119+0000","172.21.15.167:6833/1959721003":"2024-06-30T14:38:04.758075+0000","172.21.15.167:6832/1959721003":"2024-06-30T14:38:04.758075+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:12.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.341+0000 7f201f01f640 1 -- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f2004076120 msgr2=0x7f2004078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.341+0000 7f201f01f640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f2004076120 0x7f2004078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2008034000 tx=0x7f2008004730 comp rx=0 tx=0).stop 2024-06-29T14:38:12.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.341+0000 7f201f01f640 1 -- 172.21.15.81:0/1130963936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f201806ef90 msgr2=0x7f20181f5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.341+0000 7f201f01f640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f201806ef90 0x7f20181f5720 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f201400e7c0 tx=0x7f201400ec80 comp rx=0 tx=0).stop 2024-06-29T14:38:12.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.341+0000 7f201f01f640 1 -- 172.21.15.81:0/1130963936 shutdown_connections 2024-06-29T14:38:12.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.341+0000 7f201f01f640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f2004076120 0x7f2004078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.341+0000 7f201f01f640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20180ce1d0 0x7f2018154550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.341+0000 7f201f01f640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20180b57b0 0x7f20180a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.341+0000 7f201f01f640 1 --2- 172.21.15.81:0/1130963936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f201806ef90 0x7f20181f5720 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.342+0000 7f201f01f640 1 -- 172.21.15.81:0/1130963936 >> 172.21.15.81:0/1130963936 conn(0x7f20180d0930 msgr2=0x7f2018153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.342+0000 7f201f01f640 1 -- 172.21.15.81:0/1130963936 shutdown_connections 2024-06-29T14:38:12.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.342+0000 7f201f01f640 1 -- 172.21.15.81:0/1130963936 wait complete. 2024-06-29T14:38:12.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:38:12.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.468+0000 7fc8a681f640 1 -- 172.21.15.81:0/2220805703 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8a0074520 msgr2=0x7fc8a006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.468+0000 7fc8a681f640 1 --2- 172.21.15.81:0/2220805703 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8a0074520 0x7fc8a006f140 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc890009830 tx=0x7fc89002a510 comp rx=0 tx=0).stop 2024-06-29T14:38:12.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.468+0000 7fc8a681f640 1 -- 172.21.15.81:0/2220805703 shutdown_connections 2024-06-29T14:38:12.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.468+0000 7fc8a681f640 1 --2- 172.21.15.81:0/2220805703 >> v2:172.21.15.167:3300/0 conn(0x7fc8a006fa90 0x7fc8a006ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.468+0000 7fc8a681f640 1 --2- 172.21.15.81:0/2220805703 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8a0074520 0x7fc8a006f140 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.468+0000 7fc8a681f640 1 -- 172.21.15.81:0/2220805703 >> 172.21.15.81:0/2220805703 conn(0x7fc8a0072950 msgr2=0x7fc8a00b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.469+0000 7fc8a681f640 1 -- 172.21.15.81:0/2220805703 shutdown_connections 2024-06-29T14:38:12.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.469+0000 7fc8a681f640 1 -- 172.21.15.81:0/2220805703 wait complete. 2024-06-29T14:38:12.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.470+0000 7fc8a681f640 1 Processor -- start 2024-06-29T14:38:12.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.471+0000 7fc8a681f640 1 -- start start 2024-06-29T14:38:12.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc8a681f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8a006f680 0x7fc8a0153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc8a681f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8a006fa90 0x7fc8a00c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc8a681f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8a0074520 0x7fc8a0156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc8a681f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a015b010 con 0x7fc8a006f680 2024-06-29T14:38:12.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc8a681f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a006d150 con 0x7fc8a0074520 2024-06-29T14:38:12.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc8a681f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc8a015b190 con 0x7fc8a006fa90 2024-06-29T14:38:12.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc89ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8a006f680 0x7fc8a0153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc89ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8a006f680 0x7fc8a0153690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36354/0 (socket says 172.21.15.81:36354) 2024-06-29T14:38:12.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc89ffff640 1 -- 172.21.15.81:0/3084266967 learned_addr learned my addr 172.21.15.81:0/3084266967 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:12.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc89f7fe640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8a006fa90 0x7fc8a00c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc8a4dbc640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8a0074520 0x7fc8a0156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc89ffff640 1 -- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8a006fa90 msgr2=0x7fc8a00c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc89ffff640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8a006fa90 0x7fc8a00c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc89ffff640 1 -- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8a0074520 msgr2=0x7fc8a0156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc89ffff640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8a0074520 0x7fc8a0156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc89ffff640 1 -- 172.21.15.81:0/3084266967 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a00c5050 con 0x7fc8a006f680 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.472+0000 7fc89ffff640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8a006f680 0x7fc8a0153690 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fc89002ab00 tx=0x7fc890033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.473+0000 7fc89cff9640 1 -- 172.21.15.81:0/3084266967 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890005c90 con 0x7fc8a006f680 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.473+0000 7fc89cff9640 1 -- 172.21.15.81:0/3084266967 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc890031e50 con 0x7fc8a006f680 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.473+0000 7fc8a681f640 1 -- 172.21.15.81:0/3084266967 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8a009bff0 con 0x7fc8a006f680 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.473+0000 7fc89cff9640 1 -- 172.21.15.81:0/3084266967 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890005260 con 0x7fc8a006f680 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.473+0000 7fc8a681f640 1 -- 172.21.15.81:0/3084266967 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8a00c55f0 con 0x7fc8a006f680 2024-06-29T14:38:12.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.474+0000 7fc8a681f640 1 -- 172.21.15.81:0/3084266967 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8a0068ef0 con 0x7fc8a006f680 2024-06-29T14:38:12.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.474+0000 7fc89cff9640 1 -- 172.21.15.81:0/3084266967 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 44) ==== 98039+0+0 (secure 0 0 0) 0x7fc890005400 con 0x7fc8a006f680 2024-06-29T14:38:12.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.475+0000 7fc89cff9640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc88c076120 0x7fc88c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.475+0000 7fc89cff9640 1 -- 172.21.15.81:0/3084266967 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6469+0+0 (secure 0 0 0) 0x7fc8900b8660 con 0x7fc8a006f680 2024-06-29T14:38:12.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.475+0000 7fc89f7fe640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc88c076120 0x7fc88c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.476+0000 7fc89f7fe640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc88c076120 0x7fc88c078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc894007fd0 tx=0x7fc894001390 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.480+0000 7fc89cff9640 1 -- 172.21.15.81:0/3084266967 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc89003f030 con 0x7fc8a006f680 2024-06-29T14:38:12.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.696+0000 7fc8a681f640 1 -- 172.21.15.81:0/3084266967 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc8a00af5b0 con 0x7fc8a006f680 2024-06-29T14:38:12.695 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T14:38:12.697+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:38:12.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.699+0000 7fc89cff9640 1 -- 172.21.15.81:0/3084266967 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) ==== 161+0+0 (secure 0 0 0) 0x7fc890081920 con 0x7fc8a006f680 2024-06-29T14:38:12.697 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 -- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc88c076120 msgr2=0x7fc88c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc88c076120 0x7fc88c078630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc894007fd0 tx=0x7fc894001390 comp rx=0 tx=0).stop 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 -- 172.21.15.81:0/3084266967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8a006f680 msgr2=0x7fc8a0153690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8a006f680 0x7fc8a0153690 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fc89002ab00 tx=0x7fc890033c20 comp rx=0 tx=0).stop 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 -- 172.21.15.81:0/3084266967 shutdown_connections 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc88c076120 0x7fc88c078630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8a0074520 0x7fc8a0156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8a006fa90 0x7fc8a00c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 --2- 172.21.15.81:0/3084266967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8a006f680 0x7fc8a0153690 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 -- 172.21.15.81:0/3084266967 >> 172.21.15.81:0/3084266967 conn(0x7fc8a0072950 msgr2=0x7fc8a006c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 -- 172.21.15.81:0/3084266967 shutdown_connections 2024-06-29T14:38:12.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.701+0000 7fc8a681f640 1 -- 172.21.15.81:0/3084266967 wait complete. 2024-06-29T14:38:12.703 INFO:tasks.ceph.mds.b.smithi167.stderr: -2857> 2024-06-29T14:37:55.079+0000 7fbc7353e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:38:12.711 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:38:12.757 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:38:12.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.833+0000 7f8417cce640 1 -- 172.21.15.81:0/1632733209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8410071f20 msgr2=0x7f8410155480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.833+0000 7f8417cce640 1 --2- 172.21.15.81:0/1632733209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8410071f20 0x7f8410155480 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f840c005fd0 tx=0x7f840c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.833+0000 7f8417cce640 1 -- 172.21.15.81:0/1632733209 shutdown_connections 2024-06-29T14:38:12.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.833+0000 7f8417cce640 1 --2- 172.21.15.81:0/1632733209 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f841015bf20 0x7f841006f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.833+0000 7f8417cce640 1 --2- 172.21.15.81:0/1632733209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8410071f20 0x7f8410155480 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.833+0000 7f8417cce640 1 -- 172.21.15.81:0/1632733209 >> 172.21.15.81:0/1632733209 conn(0x7f84100b1730 msgr2=0x7f84100b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:12.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.833+0000 7f8417cce640 1 -- 172.21.15.81:0/1632733209 shutdown_connections 2024-06-29T14:38:12.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.833+0000 7f8417cce640 1 -- 172.21.15.81:0/1632733209 wait complete. 2024-06-29T14:38:12.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.835+0000 7f8417cce640 1 Processor -- start 2024-06-29T14:38:12.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.835+0000 7f8417cce640 1 -- start start 2024-06-29T14:38:12.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.836+0000 7f8417cce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f841006fca0 0x7f84101f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.836+0000 7f8417cce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8410071f20 0x7f84100ced10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.836+0000 7f8417cce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f841015bf20 0x7f8410142580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.836+0000 7f8417cce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f841015f1a0 con 0x7f841015bf20 2024-06-29T14:38:12.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.836+0000 7f8417cce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f84100b40a0 con 0x7f841006fca0 2024-06-29T14:38:12.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.836+0000 7f8417cce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f841015f320 con 0x7f8410071f20 2024-06-29T14:38:12.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f841626b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f841015bf20 0x7f8410142580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f841626b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f841015bf20 0x7f8410142580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36378/0 (socket says 172.21.15.81:36378) 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f841626b640 1 -- 172.21.15.81:0/4104579110 learned_addr learned my addr 172.21.15.81:0/4104579110 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f8415269640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8410071f20 0x7f84100ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f8415a6a640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f841006fca0 0x7f84101f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f841626b640 1 -- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8410071f20 msgr2=0x7f84100ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f841626b640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8410071f20 0x7f84100ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f841626b640 1 -- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f841006fca0 msgr2=0x7f84101f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f841626b640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f841006fca0 0x7f84101f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f841626b640 1 -- 172.21.15.81:0/4104579110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84100a9560 con 0x7f841015bf20 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f841626b640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f841015bf20 0x7f8410142580 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8408002230 tx=0x7f840800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f84067fc640 1 -- 172.21.15.81:0/4104579110 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8408003b70 con 0x7f841015bf20 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f84067fc640 1 -- 172.21.15.81:0/4104579110 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8408003d10 con 0x7f841015bf20 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f8417cce640 1 -- 172.21.15.81:0/4104579110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84100a9840 con 0x7f841015bf20 2024-06-29T14:38:12.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f84067fc640 1 -- 172.21.15.81:0/4104579110 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8408005330 con 0x7f841015bf20 2024-06-29T14:38:12.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.837+0000 7f8417cce640 1 -- 172.21.15.81:0/4104579110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84100a9e00 con 0x7f841015bf20 2024-06-29T14:38:12.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.838+0000 7f8417cce640 1 -- 172.21.15.81:0/4104579110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8410157f60 con 0x7f841015bf20 2024-06-29T14:38:12.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.839+0000 7f84067fc640 1 -- 172.21.15.81:0/4104579110 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 45) ==== 98247+0+0 (secure 0 0 0) 0x7f84080054d0 con 0x7f841015bf20 2024-06-29T14:38:12.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.840+0000 7f84067fc640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f83f4076360 0x7f83f4078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:12.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.840+0000 7f84067fc640 1 -- 172.21.15.81:0/4104579110 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6555+0+0 (secure 0 0 0) 0x7f8408097930 con 0x7f841015bf20 2024-06-29T14:38:12.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.840+0000 7f8415a6a640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f83f4076360 0x7f83f4078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:12.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.840+0000 7f8415a6a640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f83f4076360 0x7f83f4078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f840c033000 tx=0x7f840c004730 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:12.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:12.844+0000 7f84067fc640 1 -- 172.21.15.81:0/4104579110 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f84100700b0 con 0x7f841015bf20 2024-06-29T14:38:13.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.060+0000 7f8417cce640 1 -- 172.21.15.81:0/4104579110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f841015f7f0 con 0x7f841015bf20 2024-06-29T14:38:13.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.700+0000 7f84067fc640 1 -- 172.21.15.81:0/4104579110 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) ==== 104+0+0 (secure 0 0 0) 0x7f84080607f0 con 0x7f841015bf20 2024-06-29T14:38:13.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 -- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f83f4076360 msgr2=0x7f83f4078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:13.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f83f4076360 0x7f83f4078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f840c033000 tx=0x7f840c004730 comp rx=0 tx=0).stop 2024-06-29T14:38:13.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 -- 172.21.15.81:0/4104579110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f841015bf20 msgr2=0x7f8410142580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:13.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f841015bf20 0x7f8410142580 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8408002230 tx=0x7f840800cb90 comp rx=0 tx=0).stop 2024-06-29T14:38:13.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 -- 172.21.15.81:0/4104579110 shutdown_connections 2024-06-29T14:38:13.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f83f4076360 0x7f83f4078870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f841015bf20 0x7f8410142580 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8410071f20 0x7f84100ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 --2- 172.21.15.81:0/4104579110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f841006fca0 0x7f84101f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 -- 172.21.15.81:0/4104579110 >> 172.21.15.81:0/4104579110 conn(0x7f84100b1730 msgr2=0x7f8410068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:13.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.703+0000 7f8417cce640 1 -- 172.21.15.81:0/4104579110 shutdown_connections 2024-06-29T14:38:13.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.704+0000 7f8417cce640 1 -- 172.21.15.81:0/4104579110 wait complete. 2024-06-29T14:38:13.715 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:38:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.837+0000 7f0787e64640 1 -- 172.21.15.81:0/4035230560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f078015ac90 msgr2=0x7f07800aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.837+0000 7f0787e64640 1 --2- 172.21.15.81:0/4035230560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f078015ac90 0x7f07800aec10 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f078006eae0 tx=0x7f077402a980 comp rx=0 tx=0).stop 2024-06-29T14:38:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.837+0000 7f0787e64640 1 -- 172.21.15.81:0/4035230560 shutdown_connections 2024-06-29T14:38:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.837+0000 7f0787e64640 1 --2- 172.21.15.81:0/4035230560 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f07800ace00 0x7f07800ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.837+0000 7f0787e64640 1 --2- 172.21.15.81:0/4035230560 >> v2:172.21.15.167:3300/0 conn(0x7f078015e990 0x7f078015eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.837+0000 7f0787e64640 1 --2- 172.21.15.81:0/4035230560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f078015ac90 0x7f07800aec10 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.837+0000 7f0787e64640 1 -- 172.21.15.81:0/4035230560 >> 172.21.15.81:0/4035230560 conn(0x7f07800b1da0 msgr2=0x7f07800b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.837+0000 7f0787e64640 1 -- 172.21.15.81:0/4035230560 shutdown_connections 2024-06-29T14:38:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.837+0000 7f0787e64640 1 -- 172.21.15.81:0/4035230560 wait complete. 2024-06-29T14:38:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.839+0000 7f0787e64640 1 Processor -- start 2024-06-29T14:38:13.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.839+0000 7f0787e64640 1 -- start start 2024-06-29T14:38:13.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.840+0000 7f0787e64640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ace00 0x7f078014a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:13.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.840+0000 7f0787e64640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f078015ac90 0x7f078009b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:13.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.840+0000 7f0787e64640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f078015e990 0x7f078009c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:13.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.840+0000 7f0787e64640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f07800b2e30 con 0x7f07800ace00 2024-06-29T14:38:13.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.840+0000 7f0787e64640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f07800b2cb0 con 0x7f078015e990 2024-06-29T14:38:13.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.840+0000 7f0787e64640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f07800b2fb0 con 0x7f078015ac90 2024-06-29T14:38:13.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.840+0000 7f0785c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ace00 0x7f078014a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:13.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0785c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ace00 0x7f078014a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36396/0 (socket says 172.21.15.81:36396) 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0785c00640 1 -- 172.21.15.81:0/2476539123 learned_addr learned my addr 172.21.15.81:0/2476539123 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0786401640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f078015e990 0x7f078009c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f07853ff640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f078015ac90 0x7f078009b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0785c00640 1 -- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f078015ac90 msgr2=0x7f078009b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0785c00640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f078015ac90 0x7f078009b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0785c00640 1 -- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f078015e990 msgr2=0x7f078009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0785c00640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f078015e990 0x7f078009c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0785c00640 1 -- 172.21.15.81:0/2476539123 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078015c730 con 0x7f07800ace00 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f07853ff640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f078015ac90 0x7f078009b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0785c00640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ace00 0x7f078014a840 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0774009d60 tx=0x7f0774003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f077e7fc640 1 -- 172.21.15.81:0/2476539123 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774048070 con 0x7f07800ace00 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f077e7fc640 1 -- 172.21.15.81:0/2476539123 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0774004400 con 0x7f07800ace00 2024-06-29T14:38:13.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0787e64640 1 -- 172.21.15.81:0/2476539123 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078015ca10 con 0x7f07800ace00 2024-06-29T14:38:13.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f077e7fc640 1 -- 172.21.15.81:0/2476539123 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774034040 con 0x7f07800ace00 2024-06-29T14:38:13.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.841+0000 7f0787e64640 1 -- 172.21.15.81:0/2476539123 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f078015cfd0 con 0x7f07800ace00 2024-06-29T14:38:13.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.842+0000 7f0787e64640 1 -- 172.21.15.81:0/2476539123 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07800aec10 con 0x7f07800ace00 2024-06-29T14:38:13.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.843+0000 7f077e7fc640 1 -- 172.21.15.81:0/2476539123 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 45) ==== 98247+0+0 (secure 0 0 0) 0x7f077404b070 con 0x7f07800ace00 2024-06-29T14:38:13.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.844+0000 7f077e7fc640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f0764076360 0x7f0764078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:13.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.844+0000 7f077e7fc640 1 -- 172.21.15.81:0/2476539123 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6555+0+0 (secure 0 0 0) 0x7f07740baa90 con 0x7f07800ace00 2024-06-29T14:38:13.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.844+0000 7f07853ff640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f0764076360 0x7f0764078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:13.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.845+0000 7f07853ff640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f0764076360 0x7f0764078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07800c4ed0 tx=0x7f076c005f10 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:13.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:13.849+0000 7f077e7fc640 1 -- 172.21.15.81:0/2476539123 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f07740838d0 con 0x7f07800ace00 2024-06-29T14:38:14.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.032+0000 7f0787e64640 1 -- 172.21.15.81:0/2476539123 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f07800b24f0 con 0x7f07800ace00 2024-06-29T14:38:14.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.801+0000 7f077e7fc640 1 -- 172.21.15.81:0/2476539123 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) ==== 181+0+0 (secure 0 0 0) 0x7f0774087bd0 con 0x7f07800ace00 2024-06-29T14:38:14.799 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:38:14.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.803+0000 7f0787e64640 1 -- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f0764076360 msgr2=0x7f0764078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:14.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.803+0000 7f0787e64640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f0764076360 0x7f0764078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07800c4ed0 tx=0x7f076c005f10 comp rx=0 tx=0).stop 2024-06-29T14:38:14.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.803+0000 7f0787e64640 1 -- 172.21.15.81:0/2476539123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ace00 msgr2=0x7f078014a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:14.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.803+0000 7f0787e64640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ace00 0x7f078014a840 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0774009d60 tx=0x7f0774003f90 comp rx=0 tx=0).stop 2024-06-29T14:38:14.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.803+0000 7f0787e64640 1 -- 172.21.15.81:0/2476539123 shutdown_connections 2024-06-29T14:38:14.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.803+0000 7f0787e64640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f0764076360 0x7f0764078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.803+0000 7f0787e64640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f078015e990 0x7f078009c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.803+0000 7f0787e64640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f078015ac90 0x7f078009b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.803+0000 7f0787e64640 1 --2- 172.21.15.81:0/2476539123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07800ace00 0x7f078014a840 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.803+0000 7f0787e64640 1 -- 172.21.15.81:0/2476539123 >> 172.21.15.81:0/2476539123 conn(0x7f07800b1da0 msgr2=0x7f0780156940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:14.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.804+0000 7f0787e64640 1 -- 172.21.15.81:0/2476539123 shutdown_connections 2024-06-29T14:38:14.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.804+0000 7f0787e64640 1 -- 172.21.15.81:0/2476539123 wait complete. 2024-06-29T14:38:14.820 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:38:14.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.941+0000 7f8286827640 1 -- 172.21.15.81:0/2689508088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f826c0053c0 msgr2=0x7f826c00ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:14.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.941+0000 7f8286827640 1 --2- 172.21.15.81:0/2689508088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f826c0053c0 0x7f826c00ba70 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f8278009ed0 tx=0x7f826c005830 comp rx=0 tx=0).stop 2024-06-29T14:38:14.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.941+0000 7f8286827640 1 -- 172.21.15.81:0/2689508088 shutdown_connections 2024-06-29T14:38:14.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.941+0000 7f8286827640 1 --2- 172.21.15.81:0/2689508088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f826c0053c0 0x7f826c00ba70 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.942+0000 7f8286827640 1 -- 172.21.15.81:0/2689508088 >> 172.21.15.81:0/2689508088 conn(0x7f8280072970 msgr2=0x7f82800b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:14.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.942+0000 7f8286827640 1 -- 172.21.15.81:0/2689508088 shutdown_connections 2024-06-29T14:38:14.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.942+0000 7f8286827640 1 -- 172.21.15.81:0/2689508088 wait complete. 2024-06-29T14:38:14.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.943+0000 7f8286827640 1 Processor -- start 2024-06-29T14:38:14.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.943+0000 7f8286827640 1 -- start start 2024-06-29T14:38:14.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8286827640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f826c0053c0 0x7f828009d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:14.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8286827640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8280153dc0 0x7f828014f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:14.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8286827640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f828014c6b0 0x7f82801f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:14.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8286827640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8280073cb0 con 0x7f8280153dc0 2024-06-29T14:38:14.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8286827640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8280073b30 con 0x7f828014c6b0 2024-06-29T14:38:14.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8286827640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8280073e30 con 0x7f826c0053c0 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8277fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8280153dc0 0x7f828014f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8277fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8280153dc0 0x7f828014f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36408/0 (socket says 172.21.15.81:36408) 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8277fff640 1 -- 172.21.15.81:0/3867431807 learned_addr learned my addr 172.21.15.81:0/3867431807 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f827ffff640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f826c0053c0 0x7f828009d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8284dc4640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f828014c6b0 0x7f82801f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8277fff640 1 -- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f826c0053c0 msgr2=0x7f828009d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8277fff640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f826c0053c0 0x7f828009d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8277fff640 1 -- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f828014c6b0 msgr2=0x7f82801f36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8277fff640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f828014c6b0 0x7f82801f36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8277fff640 1 -- 172.21.15.81:0/3867431807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82801f31e0 con 0x7f8280153dc0 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8277fff640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8280153dc0 0x7f828014f130 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f8268003cd0 tx=0x7f8268001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f827d7fa640 1 -- 172.21.15.81:0/3867431807 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268015070 con 0x7f8280153dc0 2024-06-29T14:38:14.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f827d7fa640 1 -- 172.21.15.81:0/3867431807 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8268011860 con 0x7f8280153dc0 2024-06-29T14:38:14.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8286827640 1 -- 172.21.15.81:0/3867431807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8280151d80 con 0x7f8280153dc0 2024-06-29T14:38:14.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f827d7fa640 1 -- 172.21.15.81:0/3867431807 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268011cb0 con 0x7f8280153dc0 2024-06-29T14:38:14.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.945+0000 7f8286827640 1 -- 172.21.15.81:0/3867431807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8280151aa0 con 0x7f8280153dc0 2024-06-29T14:38:14.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.946+0000 7f8286827640 1 -- 172.21.15.81:0/3867431807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82801f2fd0 con 0x7f8280153dc0 2024-06-29T14:38:14.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.947+0000 7f827d7fa640 1 -- 172.21.15.81:0/3867431807 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f8268018070 con 0x7f8280153dc0 2024-06-29T14:38:14.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.948+0000 7f827d7fa640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8260076420 0x7f8260078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:14.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.948+0000 7f827d7fa640 1 -- 172.21.15.81:0/3867431807 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6117+0+0 (secure 0 0 0) 0x7f8268098700 con 0x7f8280153dc0 2024-06-29T14:38:14.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.948+0000 7f827ffff640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8260076420 0x7f8260078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:14.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.948+0000 7f827ffff640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8260076420 0x7f8260078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8270001850 tx=0x7f8270006720 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:14.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:14.952+0000 7f827d7fa640 1 -- 172.21.15.81:0/3867431807 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f82680615c0 con 0x7f8280153dc0 2024-06-29T14:38:15.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.133+0000 7f8286827640 1 -- 172.21.15.81:0/3867431807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f828006c2f0 con 0x7f8280153dc0 2024-06-29T14:38:15.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.806+0000 7f827d7fa640 1 -- 172.21.15.81:0/3867431807 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) ==== 169+0+0 (secure 0 0 0) 0x7f82680658c0 con 0x7f8280153dc0 2024-06-29T14:38:15.805 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T14:38:15.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.810+0000 7f8286827640 1 -- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8260076420 msgr2=0x7f8260078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:15.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.810+0000 7f8286827640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8260076420 0x7f8260078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8270001850 tx=0x7f8270006720 comp rx=0 tx=0).stop 2024-06-29T14:38:15.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.810+0000 7f8286827640 1 -- 172.21.15.81:0/3867431807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8280153dc0 msgr2=0x7f828014f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:15.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.810+0000 7f8286827640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8280153dc0 0x7f828014f130 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f8268003cd0 tx=0x7f8268001800 comp rx=0 tx=0).stop 2024-06-29T14:38:15.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.811+0000 7f8286827640 1 -- 172.21.15.81:0/3867431807 shutdown_connections 2024-06-29T14:38:15.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.811+0000 7f8286827640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8260076420 0x7f8260078930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.811+0000 7f8286827640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f828014c6b0 0x7f82801f36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.811+0000 7f8286827640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8280153dc0 0x7f828014f130 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.811+0000 7f8286827640 1 --2- 172.21.15.81:0/3867431807 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f826c0053c0 0x7f828009d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.811+0000 7f8286827640 1 -- 172.21.15.81:0/3867431807 >> 172.21.15.81:0/3867431807 conn(0x7f8280072970 msgr2=0x7f82800af0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:15.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.811+0000 7f8286827640 1 -- 172.21.15.81:0/3867431807 shutdown_connections 2024-06-29T14:38:15.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.812+0000 7f8286827640 1 -- 172.21.15.81:0/3867431807 wait complete. 2024-06-29T14:38:15.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:38:15.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.945+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2016179148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ddc004000 msgr2=0x7f9ddc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:15.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.945+0000 7f9dfc1c7640 1 --2- 172.21.15.81:0/2016179148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ddc004000 0x7f9ddc004800 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009b50 tx=0x7f9ddc005830 comp rx=0 tx=0).stop 2024-06-29T14:38:15.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.945+0000 7f9dfa764640 1 -- 172.21.15.81:0/2016179148 reap_dead start 2024-06-29T14:38:15.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.945+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2016179148 shutdown_connections 2024-06-29T14:38:15.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.945+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2016179148 >> 172.21.15.81:0/2016179148 conn(0x7f9df406ccc0 msgr2=0x7f9df40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:15.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.945+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2016179148 shutdown_connections 2024-06-29T14:38:15.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.945+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2016179148 wait complete. 2024-06-29T14:38:15.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.947+0000 7f9dfc1c7640 1 Processor -- start 2024-06-29T14:38:15.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.947+0000 7f9dfc1c7640 1 -- start start 2024-06-29T14:38:15.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.948+0000 7f9dfc1c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ddc004000 0x7f9df40cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.948+0000 7f9dfc1c7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9df41464d0 0x7f9df40ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.948+0000 7f9dfc1c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9df41468e0 0x7f9df40aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.948+0000 7f9dfc1c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9df406ef50 con 0x7f9df41468e0 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfc1c7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9df406edd0 con 0x7f9ddc004000 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfc1c7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9df406f0d0 con 0x7f9df41464d0 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfa764640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9df41468e0 0x7f9df40aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfa764640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9df41468e0 0x7f9df40aed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36420/0 (socket says 172.21.15.81:36420) 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfa764640 1 -- 172.21.15.81:0/2611848944 learned_addr learned my addr 172.21.15.81:0/2611848944 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9df9f63640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ddc004000 0x7f9df40cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9df9762640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9df41464d0 0x7f9df40ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfa764640 1 -- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9df41464d0 msgr2=0x7f9df40ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:15.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfa764640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9df41464d0 0x7f9df40ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfa764640 1 -- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ddc004000 msgr2=0x7f9df40cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfa764640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ddc004000 0x7f9df40cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfa764640 1 -- 172.21.15.81:0/2611848944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df40aaad0 con 0x7f9df41468e0 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9df9762640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9df41464d0 0x7f9df40ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfa764640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9df41468e0 0x7f9df40aed80 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f9df4073ce0 tx=0x7f9dec001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9df27fc640 1 -- 172.21.15.81:0/2611848944 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec011870 con 0x7f9df41468e0 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9df27fc640 1 -- 172.21.15.81:0/2611848944 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9dec017070 con 0x7f9df41468e0 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2611848944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df40aadb0 con 0x7f9df41468e0 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9df27fc640 1 -- 172.21.15.81:0/2611848944 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec011da0 con 0x7f9df41468e0 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.949+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2611848944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df40ab370 con 0x7f9df41468e0 2024-06-29T14:38:15.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.950+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2611848944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df40b4830 con 0x7f9df41468e0 2024-06-29T14:38:15.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.951+0000 7f9df27fc640 1 -- 172.21.15.81:0/2611848944 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f9dec01c460 con 0x7f9df41468e0 2024-06-29T14:38:15.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.952+0000 7f9df27fc640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9de00763f0 0x7f9de0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:15.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.952+0000 7f9df27fc640 1 -- 172.21.15.81:0/2611848944 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5703+0+0 (secure 0 0 0) 0x7f9dec098280 con 0x7f9df41468e0 2024-06-29T14:38:15.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.952+0000 7f9df9f63640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9de00763f0 0x7f9de0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:15.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.953+0000 7f9df9f63640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9de00763f0 0x7f9de0078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9de40018a0 tx=0x7f9de4000ed0 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:15.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:15.959+0000 7f9df27fc640 1 -- 172.21.15.81:0/2611848944 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9dec061930 con 0x7f9df41468e0 2024-06-29T14:38:16.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.145+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2611848944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f9df4072f30 con 0x7f9df41468e0 2024-06-29T14:38:16.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.149+0000 7f9df27fc640 1 -- 172.21.15.81:0/2611848944 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v43) ==== 71+0+0 (secure 0 0 0) 0x7f9dec061230 con 0x7f9df41468e0 2024-06-29T14:38:16.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.151+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9de00763f0 msgr2=0x7f9de0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.151+0000 7f9dfc1c7640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9de00763f0 0x7f9de0078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9de40018a0 tx=0x7f9de4000ed0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.151+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2611848944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9df41468e0 msgr2=0x7f9df40aed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.151+0000 7f9dfc1c7640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9df41468e0 0x7f9df40aed80 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f9df4073ce0 tx=0x7f9dec001a70 comp rx=0 tx=0).stop 2024-06-29T14:38:16.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.152+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2611848944 shutdown_connections 2024-06-29T14:38:16.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.152+0000 7f9dfc1c7640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9de00763f0 0x7f9de0078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.152+0000 7f9dfc1c7640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9df41468e0 0x7f9df40aed80 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.152+0000 7f9dfc1c7640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9df41464d0 0x7f9df40ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.152+0000 7f9dfc1c7640 1 --2- 172.21.15.81:0/2611848944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ddc004000 0x7f9df40cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.152+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2611848944 >> 172.21.15.81:0/2611848944 conn(0x7f9df406ccc0 msgr2=0x7f9df415cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:16.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.152+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2611848944 shutdown_connections 2024-06-29T14:38:16.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.152+0000 7f9dfc1c7640 1 -- 172.21.15.81:0/2611848944 wait complete. 2024-06-29T14:38:16.161 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2024-06-29T14:38:16.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.276+0000 7f3529417640 1 -- 172.21.15.81:0/2916446825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35240b9040 msgr2=0x7f35240d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.277+0000 7f3529417640 1 --2- 172.21.15.81:0/2916446825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35240b9040 0x7f35240d0480 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f35240a1b70 tx=0x7f351402a5f0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.277+0000 7f3529417640 1 -- 172.21.15.81:0/2916446825 shutdown_connections 2024-06-29T14:38:16.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.277+0000 7f3529417640 1 --2- 172.21.15.81:0/2916446825 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35240b7290 0x7f35240b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.277+0000 7f3529417640 1 --2- 172.21.15.81:0/2916446825 >> v2:172.21.15.167:3300/0 conn(0x7f35240b5250 0x7f35240d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.277+0000 7f3529417640 1 --2- 172.21.15.81:0/2916446825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35240b9040 0x7f35240d0480 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.277+0000 7f3529417640 1 -- 172.21.15.81:0/2916446825 >> 172.21.15.81:0/2916446825 conn(0x7f352409cf30 msgr2=0x7f35240d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:16.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.277+0000 7f3529417640 1 -- 172.21.15.81:0/2916446825 shutdown_connections 2024-06-29T14:38:16.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.277+0000 7f3529417640 1 -- 172.21.15.81:0/2916446825 wait complete. 2024-06-29T14:38:16.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.278+0000 7f3529417640 1 Processor -- start 2024-06-29T14:38:16.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.279+0000 7f3529417640 1 -- start start 2024-06-29T14:38:16.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f3529417640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35240b5250 0x7f3524156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f3529417640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35240b7290 0x7f35240b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f3529417640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35240b9040 0x7f352415fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f3529417640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f35240b7f70 con 0x7f35240b7290 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f3529417640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f35240b7df0 con 0x7f35240b9040 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f3529417640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f35240b80f0 con 0x7f35240b5250 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f351a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35240b7290 0x7f35240b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f351a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35240b7290 0x7f35240b3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36442/0 (socket says 172.21.15.81:36442) 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f351a7fc640 1 -- 172.21.15.81:0/2915415309 learned_addr learned my addr 172.21.15.81:0/2915415309 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f35237fe640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35240b9040 0x7f352415fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f3522ffd640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35240b5250 0x7f3524156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f351a7fc640 1 -- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35240b5250 msgr2=0x7f3524156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f351a7fc640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35240b5250 0x7f3524156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f351a7fc640 1 -- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35240b9040 msgr2=0x7f352415fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f351a7fc640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35240b9040 0x7f352415fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f351a7fc640 1 -- 172.21.15.81:0/2915415309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35240c82d0 con 0x7f35240b7290 2024-06-29T14:38:16.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.280+0000 7f351a7fc640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35240b7290 0x7f35240b3b70 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f350c00d770 tx=0x7f350c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:16.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.281+0000 7f351bfff640 1 -- 172.21.15.81:0/2915415309 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c017070 con 0x7f35240b7290 2024-06-29T14:38:16.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.281+0000 7f351bfff640 1 -- 172.21.15.81:0/2915415309 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f350c009d20 con 0x7f35240b7290 2024-06-29T14:38:16.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.281+0000 7f3529417640 1 -- 172.21.15.81:0/2915415309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35240c85b0 con 0x7f35240b7290 2024-06-29T14:38:16.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.281+0000 7f351bfff640 1 -- 172.21.15.81:0/2915415309 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350c0106c0 con 0x7f35240b7290 2024-06-29T14:38:16.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.281+0000 7f3529417640 1 -- 172.21.15.81:0/2915415309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35240c8b70 con 0x7f35240b7290 2024-06-29T14:38:16.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.281+0000 7f3529417640 1 -- 172.21.15.81:0/2915415309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35240c8040 con 0x7f35240b7290 2024-06-29T14:38:16.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.282+0000 7f351bfff640 1 -- 172.21.15.81:0/2915415309 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f350c01a070 con 0x7f35240b7290 2024-06-29T14:38:16.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.283+0000 7f351bfff640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f35040763f0 0x7f3504078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.283+0000 7f351bfff640 1 -- 172.21.15.81:0/2915415309 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5703+0+0 (secure 0 0 0) 0x7f350c097f10 con 0x7f35240b7290 2024-06-29T14:38:16.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.283+0000 7f3522ffd640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f35040763f0 0x7f3504078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.284+0000 7f3522ffd640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f35040763f0 0x7f3504078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3514033000 tx=0x7f3514004730 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:16.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.286+0000 7f351bfff640 1 -- 172.21.15.81:0/2915415309 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f350c061730 con 0x7f35240b7290 2024-06-29T14:38:16.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.538+0000 7f3529417640 1 -- 172.21.15.81:0/2915415309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) -- 0x7f35240b4b90 con 0x7f35240b7290 2024-06-29T14:38:16.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.808+0000 7f351bfff640 1 -- 172.21.15.81:0/2915415309 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v583) ==== 166+0+0 (secure 0 0 0) 0x7f35240b4b90 con 0x7f35240b7290 2024-06-29T14:38:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 -- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f35040763f0 msgr2=0x7f3504078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f35040763f0 0x7f3504078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3514033000 tx=0x7f3514004730 comp rx=0 tx=0).stop 2024-06-29T14:38:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 -- 172.21.15.81:0/2915415309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35240b7290 msgr2=0x7f35240b3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35240b7290 0x7f35240b3b70 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f350c00d770 tx=0x7f350c00dc30 comp rx=0 tx=0).stop 2024-06-29T14:38:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 -- 172.21.15.81:0/2915415309 shutdown_connections 2024-06-29T14:38:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f35040763f0 0x7f3504078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35240b9040 0x7f352415fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35240b7290 0x7f35240b3b70 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 --2- 172.21.15.81:0/2915415309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35240b5250 0x7f3524156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 -- 172.21.15.81:0/2915415309 >> 172.21.15.81:0/2915415309 conn(0x7f352409cf30 msgr2=0x7f3524068330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:16.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 -- 172.21.15.81:0/2915415309 shutdown_connections 2024-06-29T14:38:16.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.810+0000 7f3529417640 1 -- 172.21.15.81:0/2915415309 wait complete. 2024-06-29T14:38:16.820 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2024-06-29T14:38:16.820 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2024-06-29T14:38:16.821 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:38:16.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:16.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.935+0000 7f91e57f0640 1 -- 172.21.15.81:0/1953278257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91e00d2a90 msgr2=0x7f91e00d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.935+0000 7f91e57f0640 1 --2- 172.21.15.81:0/1953278257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91e00d2a90 0x7f91e00d09c0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f91d8005fd0 tx=0x7f91d802aa90 comp rx=0 tx=0).stop 2024-06-29T14:38:16.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.935+0000 7f91e57f0640 1 -- 172.21.15.81:0/1953278257 shutdown_connections 2024-06-29T14:38:16.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.935+0000 7f91e57f0640 1 --2- 172.21.15.81:0/1953278257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91e00d2a90 0x7f91e00d09c0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.935+0000 7f91e57f0640 1 --2- 172.21.15.81:0/1953278257 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91e00b9040 0x7f91e00d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.935+0000 7f91e57f0640 1 -- 172.21.15.81:0/1953278257 >> 172.21.15.81:0/1953278257 conn(0x7f91e009cf30 msgr2=0x7f91e00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:16.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.936+0000 7f91e57f0640 1 -- 172.21.15.81:0/1953278257 shutdown_connections 2024-06-29T14:38:16.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.936+0000 7f91e57f0640 1 -- 172.21.15.81:0/1953278257 wait complete. 2024-06-29T14:38:16.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.937+0000 7f91e57f0640 1 Processor -- start 2024-06-29T14:38:16.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.938+0000 7f91e57f0640 1 -- start start 2024-06-29T14:38:16.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91e57f0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91e00b5250 0x7f91e0153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91e57f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91e00b9040 0x7f91e00c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91e57f0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91e00d2a90 0x7f91e0151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91e57f0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f91e00b7d30 con 0x7f91e00b9040 2024-06-29T14:38:16.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91e57f0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f91e00b7bb0 con 0x7f91e00d2a90 2024-06-29T14:38:16.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91e57f0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f91e00b7eb0 con 0x7f91e00b5250 2024-06-29T14:38:16.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91d67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91e00b9040 0x7f91e00c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91d67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91e00b9040 0x7f91e00c9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36458/0 (socket says 172.21.15.81:36458) 2024-06-29T14:38:16.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91d67fc640 1 -- 172.21.15.81:0/870545581 learned_addr learned my addr 172.21.15.81:0/870545581 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:16.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91deffd640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91e00b5250 0x7f91e0153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91df7fe640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91e00d2a90 0x7f91e0151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91d67fc640 1 -- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91e00b5250 msgr2=0x7f91e0153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91d67fc640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91e00b5250 0x7f91e0153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91d67fc640 1 -- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91e00d2a90 msgr2=0x7f91e0151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91d67fc640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91e00d2a90 0x7f91e0151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91d67fc640 1 -- 172.21.15.81:0/870545581 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e00c9fc0 con 0x7f91e00b9040 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.939+0000 7f91d67fc640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91e00b9040 0x7f91e00c9a80 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f91c800b440 tx=0x7f91c800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.940+0000 7f91d7fff640 1 -- 172.21.15.81:0/870545581 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8015020 con 0x7f91e00b9040 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.940+0000 7f91d7fff640 1 -- 172.21.15.81:0/870545581 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91c8011940 con 0x7f91e00b9040 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.940+0000 7f91e57f0640 1 -- 172.21.15.81:0/870545581 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e0150360 con 0x7f91e00b9040 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.940+0000 7f91d7fff640 1 -- 172.21.15.81:0/870545581 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8011d90 con 0x7f91e00b9040 2024-06-29T14:38:16.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.940+0000 7f91e57f0640 1 -- 172.21.15.81:0/870545581 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e0150920 con 0x7f91e00b9040 2024-06-29T14:38:16.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.941+0000 7f91e57f0640 1 -- 172.21.15.81:0/870545581 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e01500b0 con 0x7f91e00b9040 2024-06-29T14:38:16.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.941+0000 7f91d7fff640 1 -- 172.21.15.81:0/870545581 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f91c8018020 con 0x7f91e00b9040 2024-06-29T14:38:16.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.942+0000 7f91d7fff640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f91c0076420 0x7f91c0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:16.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.942+0000 7f91d7fff640 1 -- 172.21.15.81:0/870545581 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5703+0+0 (secure 0 0 0) 0x7f91c8097400 con 0x7f91e00b9040 2024-06-29T14:38:16.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.942+0000 7f91deffd640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f91c0076420 0x7f91c0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:16.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.943+0000 7f91deffd640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f91c0076420 0x7f91c0078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f91d0004630 tx=0x7f91d000b2e0 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:16.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:16.947+0000 7f91d7fff640 1 -- 172.21.15.81:0/870545581 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f91c801f030 con 0x7f91e00b9040 2024-06-29T14:38:17.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.127+0000 7f91e57f0640 1 -- 172.21.15.81:0/870545581 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91e00b8440 con 0x7f91e00b9040 2024-06-29T14:38:17.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.128+0000 7f91d7fff640 1 -- 172.21.15.81:0/870545581 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+14141 (secure 0 0 0) 0x7f91c8060570 con 0x7f91e00b9040 2024-06-29T14:38:17.127 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:17.127 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":96,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:38:15.802903+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6835/910129539":"2024-06-30T14:38:12.698802+0000","172.21.15.167:0/3236355037":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/227247568":"2024-06-30T14:38:10.421119+0000","172.21.15.167:6833/1921154940":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/1196105043":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/1751731779":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/1307625241":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/3028513012":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/2042442118":"2024-06-30T14:38:10.421119+0000","172.21.15.167:0/3977971920":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/2335426752":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/4214568326":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/707343224":"2024-06-30T14:38:04.758075+0000","172.21.15.167:0/1821058896":"2024-06-30T14:38:04.758075+0000","172.21.15.167:6832/1921154940":"2024-06-30T14:38:10.421119+0000","172.21.15.167:6833/1959721003":"2024-06-30T14:38:04.758075+0000","172.21.15.167:6837/910129539":"2024-06-30T14:38:12.698802+0000","172.21.15.167:6832/1959721003":"2024-06-30T14:38:04.758075+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 -- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f91c0076420 msgr2=0x7f91c0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f91c0076420 0x7f91c0078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f91d0004630 tx=0x7f91d000b2e0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 -- 172.21.15.81:0/870545581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91e00b9040 msgr2=0x7f91e00c9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91e00b9040 0x7f91e00c9a80 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f91c800b440 tx=0x7f91c800b900 comp rx=0 tx=0).stop 2024-06-29T14:38:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 -- 172.21.15.81:0/870545581 shutdown_connections 2024-06-29T14:38:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f91c0076420 0x7f91c0078930 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f91e00d2a90 0x7f91e0151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91e00b9040 0x7f91e00c9a80 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 --2- 172.21.15.81:0/870545581 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f91e00b5250 0x7f91e0153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 -- 172.21.15.81:0/870545581 >> 172.21.15.81:0/870545581 conn(0x7f91e009cf30 msgr2=0x7f91e00d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 -- 172.21.15.81:0/870545581 shutdown_connections 2024-06-29T14:38:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.130+0000 7f91e57f0640 1 -- 172.21.15.81:0/870545581 wait complete. 2024-06-29T14:38:17.140 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:38:17.140 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:38:17.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.258+0000 7fd35cf9f640 1 -- 172.21.15.81:0/1434627958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd35808ccd0 msgr2=0x7fd3580cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.258+0000 7fd35cf9f640 1 --2- 172.21.15.81:0/1434627958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd35808ccd0 0x7fd3580cee70 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009830 tx=0x7fd34c02a3f0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.258+0000 7fd35cf9f640 1 -- 172.21.15.81:0/1434627958 shutdown_connections 2024-06-29T14:38:17.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.258+0000 7fd35cf9f640 1 --2- 172.21.15.81:0/1434627958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd35808ccd0 0x7fd3580cee70 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.258+0000 7fd35cf9f640 1 --2- 172.21.15.81:0/1434627958 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd35808c270 0x7fd3580ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.258+0000 7fd35cf9f640 1 -- 172.21.15.81:0/1434627958 >> 172.21.15.81:0/1434627958 conn(0x7fd3580b56d0 msgr2=0x7fd3580b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:17.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.258+0000 7fd35cf9f640 1 -- 172.21.15.81:0/1434627958 shutdown_connections 2024-06-29T14:38:17.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.258+0000 7fd35cf9f640 1 -- 172.21.15.81:0/1434627958 wait complete. 2024-06-29T14:38:17.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.260+0000 7fd35cf9f640 1 Processor -- start 2024-06-29T14:38:17.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.260+0000 7fd35cf9f640 1 -- start start 2024-06-29T14:38:17.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.261+0000 7fd35cf9f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd35808ccd0 0x7fd3580cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.261+0000 7fd35cf9f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3580d17b0 0x7fd3580b7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.261+0000 7fd35cf9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd358161f50 0x7fd358162360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.261+0000 7fd35cf9f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd3580d2450 con 0x7fd358161f50 2024-06-29T14:38:17.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.261+0000 7fd35cf9f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd3580d22d0 con 0x7fd35808ccd0 2024-06-29T14:38:17.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.261+0000 7fd35cf9f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd3580d25d0 con 0x7fd3580d17b0 2024-06-29T14:38:17.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.261+0000 7fd356d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd358161f50 0x7fd358162360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.261+0000 7fd356d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd358161f50 0x7fd358162360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36482/0 (socket says 172.21.15.81:36482) 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.261+0000 7fd356d9d640 1 -- 172.21.15.81:0/3985983243 learned_addr learned my addr 172.21.15.81:0/3985983243 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd355d9b640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3580d17b0 0x7fd3580b7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd35659c640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd35808ccd0 0x7fd3580cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd356d9d640 1 -- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3580d17b0 msgr2=0x7fd3580b7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd356d9d640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3580d17b0 0x7fd3580b7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd356d9d640 1 -- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd35808ccd0 msgr2=0x7fd3580cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd356d9d640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd35808ccd0 0x7fd3580cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd356d9d640 1 -- 172.21.15.81:0/3985983243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3580b7af0 con 0x7fd358161f50 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd356d9d640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd358161f50 0x7fd358162360 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd35000c760 tx=0x7fd35000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd346ffd640 1 -- 172.21.15.81:0/3985983243 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350003da0 con 0x7fd358161f50 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd346ffd640 1 -- 172.21.15.81:0/3985983243 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd35000ce10 con 0x7fd358161f50 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd35cf9f640 1 -- 172.21.15.81:0/3985983243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3580b7dd0 con 0x7fd358161f50 2024-06-29T14:38:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd346ffd640 1 -- 172.21.15.81:0/3985983243 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3500052d0 con 0x7fd358161f50 2024-06-29T14:38:17.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.262+0000 7fd35cf9f640 1 -- 172.21.15.81:0/3985983243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3580b8390 con 0x7fd358161f50 2024-06-29T14:38:17.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.263+0000 7fd35cf9f640 1 -- 172.21.15.81:0/3985983243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd35808c270 con 0x7fd358161f50 2024-06-29T14:38:17.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.264+0000 7fd346ffd640 1 -- 172.21.15.81:0/3985983243 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fd350005470 con 0x7fd358161f50 2024-06-29T14:38:17.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.264+0000 7fd346ffd640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd3380763f0 0x7fd338078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.265+0000 7fd346ffd640 1 -- 172.21.15.81:0/3985983243 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5703+0+0 (secure 0 0 0) 0x7fd3500975c0 con 0x7fd358161f50 2024-06-29T14:38:17.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.265+0000 7fd35659c640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd3380763f0 0x7fd338078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.265+0000 7fd35659c640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd3380763f0 0x7fd338078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd348005fd0 tx=0x7fd34800a250 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.269+0000 7fd346ffd640 1 -- 172.21.15.81:0/3985983243 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd350060740 con 0x7fd358161f50 2024-06-29T14:38:17.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.448+0000 7fd35cf9f640 1 -- 172.21.15.81:0/3985983243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd3580b9aa0 con 0x7fd358161f50 2024-06-29T14:38:17.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.449+0000 7fd346ffd640 1 -- 172.21.15.81:0/3985983243 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) ==== 93+0+10 (secure 0 0 0) 0x7fd350064a40 con 0x7fd358161f50 2024-06-29T14:38:17.448 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:38:17.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 -- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd3380763f0 msgr2=0x7fd338078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd3380763f0 0x7fd338078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd348005fd0 tx=0x7fd34800a250 comp rx=0 tx=0).stop 2024-06-29T14:38:17.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 -- 172.21.15.81:0/3985983243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd358161f50 msgr2=0x7fd358162360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd358161f50 0x7fd358162360 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd35000c760 tx=0x7fd35000cc20 comp rx=0 tx=0).stop 2024-06-29T14:38:17.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 -- 172.21.15.81:0/3985983243 shutdown_connections 2024-06-29T14:38:17.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd3380763f0 0x7fd338078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd358161f50 0x7fd358162360 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3580d17b0 0x7fd3580b7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 --2- 172.21.15.81:0/3985983243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd35808ccd0 0x7fd3580cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 -- 172.21.15.81:0/3985983243 >> 172.21.15.81:0/3985983243 conn(0x7fd3580b56d0 msgr2=0x7fd3580d05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:17.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 -- 172.21.15.81:0/3985983243 shutdown_connections 2024-06-29T14:38:17.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.451+0000 7fd35cf9f640 1 -- 172.21.15.81:0/3985983243 wait complete. 2024-06-29T14:38:17.461 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-06-29T14:38:17.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.576+0000 7f731369e640 1 -- 172.21.15.81:0/3569463691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c073030 msgr2=0x7f730c073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.576+0000 7f731369e640 1 --2- 172.21.15.81:0/3569463691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c073030 0x7f730c073440 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f730c071d20 tx=0x7f72fc02a5f0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.576+0000 7f731369e640 1 -- 172.21.15.81:0/3569463691 shutdown_connections 2024-06-29T14:38:17.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.576+0000 7f731369e640 1 --2- 172.21.15.81:0/3569463691 >> v2:172.21.15.167:3300/0 conn(0x7f730c15e960 0x7f730c15ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.576+0000 7f731369e640 1 --2- 172.21.15.81:0/3569463691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c073030 0x7f730c073440 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.576+0000 7f731369e640 1 -- 172.21.15.81:0/3569463691 >> 172.21.15.81:0/3569463691 conn(0x7f730c06ccc0 msgr2=0x7f730c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:17.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.576+0000 7f731369e640 1 -- 172.21.15.81:0/3569463691 shutdown_connections 2024-06-29T14:38:17.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.576+0000 7f731369e640 1 -- 172.21.15.81:0/3569463691 wait complete. 2024-06-29T14:38:17.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.578+0000 7f731369e640 1 Processor -- start 2024-06-29T14:38:17.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.578+0000 7f731369e640 1 -- start start 2024-06-29T14:38:17.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f731369e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c073030 0x7f730c151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f731369e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c1412f0 0x7f730c153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f731369e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c15e960 0x7f730c153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f731369e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f730c0b3210 con 0x7f730c1412f0 2024-06-29T14:38:17.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f731369e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f730c141740 con 0x7f730c073030 2024-06-29T14:38:17.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f731369e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f730c0b3390 con 0x7f730c15e960 2024-06-29T14:38:17.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f7310c39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c1412f0 0x7f730c153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f7310c39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c1412f0 0x7f730c153110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36498/0 (socket says 172.21.15.81:36498) 2024-06-29T14:38:17.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f7310c39640 1 -- 172.21.15.81:0/503077378 learned_addr learned my addr 172.21.15.81:0/503077378 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f731143a640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c073030 0x7f730c151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.579+0000 7f7311c3b640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c15e960 0x7f730c153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f7310c39640 1 -- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c15e960 msgr2=0x7f730c153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f7310c39640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c15e960 0x7f730c153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f7310c39640 1 -- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c073030 msgr2=0x7f730c151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f7310c39640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c073030 0x7f730c151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f7310c39640 1 -- 172.21.15.81:0/503077378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730c1402d0 con 0x7f730c1412f0 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f731143a640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c073030 0x7f730c151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f7311c3b640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c15e960 0x7f730c153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f7310c39640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c1412f0 0x7f730c153110 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f72fc034040 tx=0x7f72fc003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f7301ffb640 1 -- 172.21.15.81:0/503077378 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc03e030 con 0x7f730c1412f0 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f7301ffb640 1 -- 172.21.15.81:0/503077378 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72fc003e00 con 0x7f730c1412f0 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f731369e640 1 -- 172.21.15.81:0/503077378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f730c151eb0 con 0x7f730c1412f0 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f7301ffb640 1 -- 172.21.15.81:0/503077378 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc035040 con 0x7f730c1412f0 2024-06-29T14:38:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.580+0000 7f731369e640 1 -- 172.21.15.81:0/503077378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f730c159cd0 con 0x7f730c1412f0 2024-06-29T14:38:17.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.581+0000 7f731369e640 1 -- 172.21.15.81:0/503077378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f730c159a40 con 0x7f730c1412f0 2024-06-29T14:38:17.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.582+0000 7f7301ffb640 1 -- 172.21.15.81:0/503077378 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f72fc041030 con 0x7f730c1412f0 2024-06-29T14:38:17.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.582+0000 7f7301ffb640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f72f00763f0 0x7f72f0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:17.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.582+0000 7f7301ffb640 1 -- 172.21.15.81:0/503077378 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5703+0+0 (secure 0 0 0) 0x7f72fc0b8280 con 0x7f730c1412f0 2024-06-29T14:38:17.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.583+0000 7f731143a640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f72f00763f0 0x7f72f0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:17.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.583+0000 7f731143a640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f72f00763f0 0x7f72f0078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7308007fd0 tx=0x7f7308009040 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:17.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.587+0000 7f7301ffb640 1 -- 172.21.15.81:0/503077378 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f730c1402d0 con 0x7f730c1412f0 2024-06-29T14:38:17.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:17.837+0000 7f731369e640 1 -- 172.21.15.81:0/503077378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) -- 0x7f730c0b38f0 con 0x7f730c1412f0 2024-06-29T14:38:18.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.815+0000 7f7301ffb640 1 -- 172.21.15.81:0/503077378 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v585) ==== 186+0+0 (secure 0 0 0) 0x7f72fc081930 con 0x7f730c1412f0 2024-06-29T14:38:18.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 -- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f72f00763f0 msgr2=0x7f72f0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:18.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f72f00763f0 0x7f72f0078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7308007fd0 tx=0x7f7308009040 comp rx=0 tx=0).stop 2024-06-29T14:38:18.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 -- 172.21.15.81:0/503077378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c1412f0 msgr2=0x7f730c153110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:18.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c1412f0 0x7f730c153110 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f72fc034040 tx=0x7f72fc003910 comp rx=0 tx=0).stop 2024-06-29T14:38:18.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 -- 172.21.15.81:0/503077378 shutdown_connections 2024-06-29T14:38:18.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f72f00763f0 0x7f72f0078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:18.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f730c15e960 0x7f730c153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:18.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f730c1412f0 0x7f730c153110 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:18.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 --2- 172.21.15.81:0/503077378 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f730c073030 0x7f730c151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:18.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 -- 172.21.15.81:0/503077378 >> 172.21.15.81:0/503077378 conn(0x7f730c06ccc0 msgr2=0x7f730c068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:18.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 -- 172.21.15.81:0/503077378 shutdown_connections 2024-06-29T14:38:18.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.818+0000 7f731369e640 1 -- 172.21.15.81:0/503077378 wait complete. 2024-06-29T14:38:18.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:38:18.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.955+0000 7ff0c50e4640 1 -- 172.21.15.81:0/1596569049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0c00b9040 msgr2=0x7ff0c00d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:18.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.955+0000 7ff0c50e4640 1 --2- 172.21.15.81:0/1596569049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0c00b9040 0x7ff0c00d0480 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00a1b70 tx=0x7ff0b002a680 comp rx=0 tx=0).stop 2024-06-29T14:38:18.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.955+0000 7ff0c50e4640 1 -- 172.21.15.81:0/1596569049 shutdown_connections 2024-06-29T14:38:18.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.955+0000 7ff0c50e4640 1 --2- 172.21.15.81:0/1596569049 >> v2:172.21.15.167:3300/0 conn(0x7ff0c00b5250 0x7ff0c00d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:18.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.955+0000 7ff0c50e4640 1 --2- 172.21.15.81:0/1596569049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0c00b9040 0x7ff0c00d0480 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:18.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.955+0000 7ff0c50e4640 1 -- 172.21.15.81:0/1596569049 >> 172.21.15.81:0/1596569049 conn(0x7ff0c009cf30 msgr2=0x7ff0c00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:18.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.955+0000 7ff0c50e4640 1 -- 172.21.15.81:0/1596569049 shutdown_connections 2024-06-29T14:38:18.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.955+0000 7ff0c50e4640 1 -- 172.21.15.81:0/1596569049 wait complete. 2024-06-29T14:38:18.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.957+0000 7ff0c50e4640 1 Processor -- start 2024-06-29T14:38:18.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.957+0000 7ff0c50e4640 1 -- start start 2024-06-29T14:38:18.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0c50e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0c00b5250 0x7ff0c01f8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:18.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0c50e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0c00b7290 0x7ff0c009d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:18.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0c50e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0c00b9040 0x7ff0c00c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:18.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0c50e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c00d2e20 con 0x7ff0c00b9040 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0c50e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c00d2ca0 con 0x7ff0c00b5250 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0c50e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff0c00b7db0 con 0x7ff0c00b7290 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0bf59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0c00b9040 0x7ff0c00c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0bf59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0c00b9040 0x7ff0c00c74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36522/0 (socket says 172.21.15.81:36522) 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0bf59e640 1 -- 172.21.15.81:0/246451477 learned_addr learned my addr 172.21.15.81:0/246451477 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0b7fff640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0c00b7290 0x7ff0c009d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0bed9d640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0c00b5250 0x7ff0c01f8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0bf59e640 1 -- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0c00b7290 msgr2=0x7ff0c009d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0bf59e640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0c00b7290 0x7ff0c009d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0bf59e640 1 -- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0c00b5250 msgr2=0x7ff0c01f8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0bf59e640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0c00b5250 0x7ff0c01f8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0bf59e640 1 -- 172.21.15.81:0/246451477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c009e510 con 0x7ff0c00b9040 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0bf59e640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0c00b9040 0x7ff0c00c74e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800c6e0 tx=0x7ff0b800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:18.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0b77fe640 1 -- 172.21.15.81:0/246451477 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8003b70 con 0x7ff0c00b9040 2024-06-29T14:38:18.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0b77fe640 1 -- 172.21.15.81:0/246451477 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0b8003d10 con 0x7ff0c00b9040 2024-06-29T14:38:18.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0c50e4640 1 -- 172.21.15.81:0/246451477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c009e7f0 con 0x7ff0c00b9040 2024-06-29T14:38:18.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0b77fe640 1 -- 172.21.15.81:0/246451477 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8005330 con 0x7ff0c00b9040 2024-06-29T14:38:18.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.959+0000 7ff0c50e4640 1 -- 172.21.15.81:0/246451477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c009edb0 con 0x7ff0c00b9040 2024-06-29T14:38:18.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.960+0000 7ff0c50e4640 1 -- 172.21.15.81:0/246451477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c00b82f0 con 0x7ff0c00b9040 2024-06-29T14:38:18.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.961+0000 7ff0b77fe640 1 -- 172.21.15.81:0/246451477 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7ff0b80054d0 con 0x7ff0c00b9040 2024-06-29T14:38:18.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.961+0000 7ff0b77fe640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff0a00763f0 0x7ff0a0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:18.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.962+0000 7ff0b77fe640 1 -- 172.21.15.81:0/246451477 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5703+0+0 (secure 0 0 0) 0x7ff0b80975a0 con 0x7ff0c00b9040 2024-06-29T14:38:18.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.962+0000 7ff0bed9d640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff0a00763f0 0x7ff0a0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:18.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.962+0000 7ff0bed9d640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff0a00763f0 0x7ff0a0078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0033000 tx=0x7ff0b0004730 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:18.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:18.967+0000 7ff0b77fe640 1 -- 172.21.15.81:0/246451477 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff0b80607b0 con 0x7ff0c00b9040 2024-06-29T14:38:19.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.151+0000 7ff0c50e4640 1 -- 172.21.15.81:0/246451477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7ff0c00b8930 con 0x7ff0c00b9040 2024-06-29T14:38:19.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.151+0000 7ff0b77fe640 1 -- 172.21.15.81:0/246451477 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v43) ==== 59+0+1998 (secure 0 0 0) 0x7ff0b8064ab0 con 0x7ff0c00b9040 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T14:38:19.150 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:38:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T14:38:19.152 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T14:38:19.152 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T14:38:19.152 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:38:19.152 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 -- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff0a00763f0 msgr2=0x7ff0a0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff0a00763f0 0x7ff0a0078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0033000 tx=0x7ff0b0004730 comp rx=0 tx=0).stop 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 -- 172.21.15.81:0/246451477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0c00b9040 msgr2=0x7ff0c00c74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0c00b9040 0x7ff0c00c74e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800c6e0 tx=0x7ff0b800cba0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 -- 172.21.15.81:0/246451477 shutdown_connections 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff0a00763f0 0x7ff0a0078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0c00b9040 0x7ff0c00c74e0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0c00b7290 0x7ff0c009d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 --2- 172.21.15.81:0/246451477 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0c00b5250 0x7ff0c01f8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 -- 172.21.15.81:0/246451477 >> 172.21.15.81:0/246451477 conn(0x7ff0c009cf30 msgr2=0x7ff0c00d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 -- 172.21.15.81:0/246451477 shutdown_connections 2024-06-29T14:38:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.153+0000 7ff0c50e4640 1 -- 172.21.15.81:0/246451477 wait complete. 2024-06-29T14:38:19.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:38:19.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.281+0000 7f4eb4363640 1 -- 172.21.15.81:0/670466494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eac0b5250 msgr2=0x7f4eac0d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.281+0000 7f4eb4363640 1 --2- 172.21.15.81:0/670466494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eac0b5250 0x7f4eac0d1fc0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009830 tx=0x7f4ea402a5a0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.281+0000 7f4eb4363640 1 -- 172.21.15.81:0/670466494 shutdown_connections 2024-06-29T14:38:19.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.281+0000 7f4eb4363640 1 --2- 172.21.15.81:0/670466494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eac0b5250 0x7f4eac0d1fc0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.281+0000 7f4eb4363640 1 --2- 172.21.15.81:0/670466494 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eac0d0b90 0x7f4eac0d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.281+0000 7f4eb4363640 1 -- 172.21.15.81:0/670466494 >> 172.21.15.81:0/670466494 conn(0x7f4eac09cf30 msgr2=0x7f4eac0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:19.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.281+0000 7f4eb4363640 1 -- 172.21.15.81:0/670466494 shutdown_connections 2024-06-29T14:38:19.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.281+0000 7f4eb4363640 1 -- 172.21.15.81:0/670466494 wait complete. 2024-06-29T14:38:19.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.284+0000 7f4eb4363640 1 Processor -- start 2024-06-29T14:38:19.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.284+0000 7f4eb4363640 1 -- start start 2024-06-29T14:38:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb4363640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eac0b5250 0x7f4eac154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb4363640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eac0b9040 0x7f4eac0b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb4363640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eac0cce90 0x7f4eac0cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb4363640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac0d2680 con 0x7f4eac0b9040 2024-06-29T14:38:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb4363640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac0d2500 con 0x7f4eac0b5250 2024-06-29T14:38:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb4363640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4eac0d2800 con 0x7f4eac0cce90 2024-06-29T14:38:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb18fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eac0b9040 0x7f4eac0b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb18fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eac0b9040 0x7f4eac0b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36532/0 (socket says 172.21.15.81:36532) 2024-06-29T14:38:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb18fe640 1 -- 172.21.15.81:0/2046612792 learned_addr learned my addr 172.21.15.81:0/2046612792 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb2900640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eac0cce90 0x7f4eac0cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb20ff640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eac0b5250 0x7f4eac154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb18fe640 1 -- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eac0cce90 msgr2=0x7f4eac0cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb18fe640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eac0cce90 0x7f4eac0cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb18fe640 1 -- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eac0b5250 msgr2=0x7f4eac154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb18fe640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eac0b5250 0x7f4eac154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb18fe640 1 -- 172.21.15.81:0/2046612792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eac0cd7e0 con 0x7f4eac0b9040 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb18fe640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eac0b9040 0x7f4eac0b0820 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00d680 tx=0x7f4e9c00db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4ea2ffd640 1 -- 172.21.15.81:0/2046612792 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c017070 con 0x7f4eac0b9040 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4ea2ffd640 1 -- 172.21.15.81:0/2046612792 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e9c009ab0 con 0x7f4eac0b9040 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4ea2ffd640 1 -- 172.21.15.81:0/2046612792 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c010460 con 0x7f4eac0b9040 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb4363640 1 -- 172.21.15.81:0/2046612792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eac0b3980 con 0x7f4eac0b9040 2024-06-29T14:38:19.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.285+0000 7f4eb4363640 1 -- 172.21.15.81:0/2046612792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eac0b3f40 con 0x7f4eac0b9040 2024-06-29T14:38:19.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.286+0000 7f4eb4363640 1 -- 172.21.15.81:0/2046612792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eac0d2d90 con 0x7f4eac0b9040 2024-06-29T14:38:19.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.287+0000 7f4ea2ffd640 1 -- 172.21.15.81:0/2046612792 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f4e9c01a070 con 0x7f4eac0b9040 2024-06-29T14:38:19.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.287+0000 7f4ea2ffd640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4e90076420 0x7f4e90078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.287+0000 7f4ea2ffd640 1 -- 172.21.15.81:0/2046612792 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5703+0+0 (secure 0 0 0) 0x7f4e9c097960 con 0x7f4eac0b9040 2024-06-29T14:38:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.288+0000 7f4eb20ff640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4e90076420 0x7f4e90078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.288+0000 7f4eb20ff640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4e90076420 0x7f4e90078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ea80020f0 tx=0x7f4ea8009040 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:19.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.293+0000 7f4ea2ffd640 1 -- 172.21.15.81:0/2046612792 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4eac0cd7e0 con 0x7f4eac0b9040 2024-06-29T14:38:19.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.471+0000 7f4eb4363640 1 -- 172.21.15.81:0/2046612792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f4eac06d560 con 0x7f4eac0b9040 2024-06-29T14:38:19.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.475+0000 7f4ea2ffd640 1 -- 172.21.15.81:0/2046612792 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7f4e9c0610a0 con 0x7f4eac0b9040 2024-06-29T14:38:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 -- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4e90076420 msgr2=0x7f4e90078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4e90076420 0x7f4e90078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ea80020f0 tx=0x7f4ea8009040 comp rx=0 tx=0).stop 2024-06-29T14:38:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 -- 172.21.15.81:0/2046612792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eac0b9040 msgr2=0x7f4eac0b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eac0b9040 0x7f4eac0b0820 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00d680 tx=0x7f4e9c00db40 comp rx=0 tx=0).stop 2024-06-29T14:38:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 -- 172.21.15.81:0/2046612792 shutdown_connections 2024-06-29T14:38:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4e90076420 0x7f4e90078930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eac0cce90 0x7f4eac0cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eac0b9040 0x7f4eac0b0820 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 --2- 172.21.15.81:0/2046612792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eac0b5250 0x7f4eac154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 -- 172.21.15.81:0/2046612792 >> 172.21.15.81:0/2046612792 conn(0x7f4eac09cf30 msgr2=0x7f4eac160f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:19.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 -- 172.21.15.81:0/2046612792 shutdown_connections 2024-06-29T14:38:19.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.477+0000 7f4eb4363640 1 -- 172.21.15.81:0/2046612792 wait complete. 2024-06-29T14:38:19.487 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:38:19.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:38:19.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:38:19.574 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:38:19.575 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:38:19.575 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:19.590 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:38:19.590 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:38:19.591 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:19.608 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:38:19.608 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:38:19.609 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:19.646 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:38:19.647 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:38:19.647 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:19.665 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:38:19.665 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:38:19.665 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:19.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.803+0000 7f75094fd640 1 -- 172.21.15.81:0/4126500717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75040b4810 msgr2=0x7f75040b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.803+0000 7f75094fd640 1 --2- 172.21.15.81:0/4126500717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75040b4810 0x7f75040b4c20 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f74fc005fd0 tx=0x7f74fc02a820 comp rx=0 tx=0).stop 2024-06-29T14:38:19.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.803+0000 7f75094fd640 1 -- 172.21.15.81:0/4126500717 shutdown_connections 2024-06-29T14:38:19.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.803+0000 7f75094fd640 1 --2- 172.21.15.81:0/4126500717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75040b4810 0x7f75040b4c20 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.803+0000 7f75094fd640 1 -- 172.21.15.81:0/4126500717 >> 172.21.15.81:0/4126500717 conn(0x7f75040b3510 msgr2=0x7f75040b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:19.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.803+0000 7f75094fd640 1 -- 172.21.15.81:0/4126500717 shutdown_connections 2024-06-29T14:38:19.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.803+0000 7f75094fd640 1 -- 172.21.15.81:0/4126500717 wait complete. 2024-06-29T14:38:19.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.804+0000 7f75094fd640 1 Processor -- start 2024-06-29T14:38:19.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.805+0000 7f75094fd640 1 -- start start 2024-06-29T14:38:19.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75094fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75040b4810 0x7f750415aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75094fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75040b6eb0 0x7f7504163330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75094fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75040cd480 0x7f7504163870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75094fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f75040b5160 con 0x7f75040cd480 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75094fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f75040b29c0 con 0x7f75040b6eb0 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75094fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f75040b52e0 con 0x7f75040b4810 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75040cd480 0x7f7504163870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75040cd480 0x7f7504163870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36546/0 (socket says 172.21.15.81:36546) 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75037fe640 1 -- 172.21.15.81:0/1636335412 learned_addr learned my addr 172.21.15.81:0/1636335412 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f74fa7fc640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75040b6eb0 0x7f7504163330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f7502ffd640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75040b4810 0x7f750415aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75037fe640 1 -- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75040b4810 msgr2=0x7f750415aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75037fe640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75040b4810 0x7f750415aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75037fe640 1 -- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75040b6eb0 msgr2=0x7f7504163330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:19.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75037fe640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75040b6eb0 0x7f7504163330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:19.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.806+0000 7f75037fe640 1 -- 172.21.15.81:0/1636335412 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7504166e10 con 0x7f75040cd480 2024-06-29T14:38:19.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.807+0000 7f75037fe640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75040cd480 0x7f7504163870 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009cd0 tx=0x7f74fc032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:19.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.807+0000 7f74fbfff640 1 -- 172.21.15.81:0/1636335412 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc037040 con 0x7f75040cd480 2024-06-29T14:38:19.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.807+0000 7f74fbfff640 1 -- 172.21.15.81:0/1636335412 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74fc042070 con 0x7f75040cd480 2024-06-29T14:38:19.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.807+0000 7f74fbfff640 1 -- 172.21.15.81:0/1636335412 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74fc004260 con 0x7f75040cd480 2024-06-29T14:38:19.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.807+0000 7f75094fd640 1 -- 172.21.15.81:0/1636335412 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75041670f0 con 0x7f75040cd480 2024-06-29T14:38:19.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.807+0000 7f75094fd640 1 -- 172.21.15.81:0/1636335412 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f750415b390 con 0x7f75040cd480 2024-06-29T14:38:19.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.808+0000 7f75094fd640 1 -- 172.21.15.81:0/1636335412 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75040b4c20 con 0x7f75040cd480 2024-06-29T14:38:19.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.808+0000 7f74fbfff640 1 -- 172.21.15.81:0/1636335412 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f74fc004400 con 0x7f75040cd480 2024-06-29T14:38:19.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.809+0000 7f74fbfff640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f74e40763f0 0x7f74e4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:19.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.809+0000 7f74fbfff640 1 -- 172.21.15.81:0/1636335412 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5703+0+0 (secure 0 0 0) 0x7f74fc0b8820 con 0x7f75040cd480 2024-06-29T14:38:19.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.809+0000 7f7502ffd640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f74e40763f0 0x7f74e4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:19.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.810+0000 7f7502ffd640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f74e40763f0 0x7f74e4078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f74f4006fd0 tx=0x7f74f4008040 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:19.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:19.813+0000 7f74fbfff640 1 -- 172.21.15.81:0/1636335412 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f74fc081f60 con 0x7f75040cd480 2024-06-29T14:38:20.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.029+0000 7f75094fd640 1 -- 172.21.15.81:0/1636335412 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75040cdf00 con 0x7f75040cd480 2024-06-29T14:38:20.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.029+0000 7f74fbfff640 1 -- 172.21.15.81:0/1636335412 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+3914 (secure 0 0 0) 0x7f74fc081860 con 0x7f75040cd480 2024-06-29T14:38:20.027 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:20.027 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":44,"btime":"2024-06-29T14:38:13:700592+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6002,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3644828063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3644828063},{"type":"v1","addr":"172.21.15.167:6836","nonce":3644828063}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":6272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/3998286446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":3998286446},{"type":"v1","addr":"172.21.15.167:6837","nonce":3998286446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":7026,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2532767666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2532767666},{"type":"v1","addr":"172.21.15.81:6835","nonce":2532767666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37},{"gid":7029,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/818227798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":818227798},{"type":"v1","addr":"172.21.15.81:6834","nonce":818227798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[]} 2024-06-29T14:38:20.027 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 44 2024-06-29T14:38:20.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 -- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f74e40763f0 msgr2=0x7f74e4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f74e40763f0 0x7f74e4078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f74f4006fd0 tx=0x7f74f4008040 comp rx=0 tx=0).stop 2024-06-29T14:38:20.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 -- 172.21.15.81:0/1636335412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75040cd480 msgr2=0x7f7504163870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75040cd480 0x7f7504163870 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009cd0 tx=0x7f74fc032d10 comp rx=0 tx=0).stop 2024-06-29T14:38:20.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 -- 172.21.15.81:0/1636335412 shutdown_connections 2024-06-29T14:38:20.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f74e40763f0 0x7f74e4078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75040cd480 0x7f7504163870 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75040b6eb0 0x7f7504163330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 --2- 172.21.15.81:0/1636335412 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75040b4810 0x7f750415aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 -- 172.21.15.81:0/1636335412 >> 172.21.15.81:0/1636335412 conn(0x7f75040b3510 msgr2=0x7f7504068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:20.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 -- 172.21.15.81:0/1636335412 shutdown_connections 2024-06-29T14:38:20.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.031+0000 7f75094fd640 1 -- 172.21.15.81:0/1636335412 wait complete. 2024-06-29T14:38:20.040 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:38:20.040 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:38:20.041 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:38:20.041 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:38:20.041 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:38:20.041 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:38:20.041 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:38:20.041 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:38:20.041 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:38:20.041 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:38:20.042 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:38:20.042 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:38:20.042 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:38:20.042 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:38:20.042 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:38:20.042 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:38:20.050 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:38:20.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:38:20.051 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:38:20.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:38:20.054 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:38:20.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:38:20.056 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:38:20.056 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:38:20.093 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:38:20.093 DEBUG:teuthology.parallel:result is None 2024-06-29T14:38:20.093 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:38:20.093 DEBUG:teuthology.parallel:result is None 2024-06-29T14:38:20.098 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:38:20.099 DEBUG:teuthology.parallel:result is None 2024-06-29T14:38:20.100 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:38:20.100 DEBUG:teuthology.parallel:result is None 2024-06-29T14:38:20.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:38:20.177 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:38:20.178 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T14:38:20.194 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T14:38:20.194 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:38:20.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.223+0000 7f89613bd640 1 -- 172.21.15.81:0/1135093235 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f895c0d0b20 msgr2=0x7f895c0cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.223+0000 7f89613bd640 1 --2- 172.21.15.81:0/1135093235 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f895c0d0b20 0x7f895c0cf5c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8954005fd0 tx=0x7f895402ab20 comp rx=0 tx=0).stop 2024-06-29T14:38:20.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.223+0000 7f89613bd640 1 -- 172.21.15.81:0/1135093235 shutdown_connections 2024-06-29T14:38:20.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.223+0000 7f89613bd640 1 --2- 172.21.15.81:0/1135093235 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f895c0d0b20 0x7f895c0cf5c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.223+0000 7f89613bd640 1 --2- 172.21.15.81:0/1135093235 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f895c0b8ba0 0x7f895c0d46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.223+0000 7f89613bd640 1 -- 172.21.15.81:0/1135093235 >> 172.21.15.81:0/1135093235 conn(0x7f895c0d3bd0 msgr2=0x7f895c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:20.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.223+0000 7f89613bd640 1 -- 172.21.15.81:0/1135093235 shutdown_connections 2024-06-29T14:38:20.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.223+0000 7f89613bd640 1 -- 172.21.15.81:0/1135093235 wait complete. 2024-06-29T14:38:20.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.225+0000 7f89613bd640 1 Processor -- start 2024-06-29T14:38:20.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.225+0000 7f89613bd640 1 -- start start 2024-06-29T14:38:20.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.226+0000 7f89613bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f895c0b6a20 0x7f895c158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.226+0000 7f89613bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f895c0b8ba0 0x7f895c09fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.226+0000 7f89613bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f895c160fd0 0x7f895c1613e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.226+0000 7f89613bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f895c0b9410 con 0x7f895c160fd0 2024-06-29T14:38:20.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.226+0000 7f89613bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f895c0b9290 con 0x7f895c0b6a20 2024-06-29T14:38:20.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.226+0000 7f89613bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f895c0cfb00 con 0x7f895c0b8ba0 2024-06-29T14:38:20.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.226+0000 7f895b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f895c160fd0 0x7f895c1613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.226+0000 7f895b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f895c160fd0 0x7f895c1613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38520/0 (socket says 172.21.15.81:38520) 2024-06-29T14:38:20.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.226+0000 7f895b7fe640 1 -- 172.21.15.81:0/1197116627 learned_addr learned my addr 172.21.15.81:0/1197116627 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:20.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f895affd640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f895c0b6a20 0x7f895c158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f89527fc640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f895c0b8ba0 0x7f895c09fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f895b7fe640 1 -- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f895c0b8ba0 msgr2=0x7f895c09fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f895b7fe640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f895c0b8ba0 0x7f895c09fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f895b7fe640 1 -- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f895c0b6a20 msgr2=0x7f895c158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f895b7fe640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f895c0b6a20 0x7f895c158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f895b7fe640 1 -- 172.21.15.81:0/1197116627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c161920 con 0x7f895c160fd0 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f895b7fe640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f895c160fd0 0x7f895c1613e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f8954009d90 tx=0x7f8954003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f8953fff640 1 -- 172.21.15.81:0/1197116627 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954044070 con 0x7f895c160fd0 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f8953fff640 1 -- 172.21.15.81:0/1197116627 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8954003e00 con 0x7f895c160fd0 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f8953fff640 1 -- 172.21.15.81:0/1197116627 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89540044b0 con 0x7f895c160fd0 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f89613bd640 1 -- 172.21.15.81:0/1197116627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895c0a0720 con 0x7f895c160fd0 2024-06-29T14:38:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.227+0000 7f89613bd640 1 -- 172.21.15.81:0/1197116627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895c0a0ce0 con 0x7f895c160fd0 2024-06-29T14:38:20.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.228+0000 7f89613bd640 1 -- 172.21.15.81:0/1197116627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895c0a0490 con 0x7f895c160fd0 2024-06-29T14:38:20.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.229+0000 7f8953fff640 1 -- 172.21.15.81:0/1197116627 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f8954004650 con 0x7f895c160fd0 2024-06-29T14:38:20.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.229+0000 7f8953fff640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f893c076420 0x7f893c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.229+0000 7f8953fff640 1 -- 172.21.15.81:0/1197116627 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5703+0+0 (secure 0 0 0) 0x7f89540b8210 con 0x7f895c160fd0 2024-06-29T14:38:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.230+0000 7f895affd640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f893c076420 0x7f893c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.230+0000 7f895affd640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f893c076420 0x7f893c078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f894c0095f0 tx=0x7f894c005f90 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:20.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.234+0000 7f8953fff640 1 -- 172.21.15.81:0/1197116627 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f895c161920 con 0x7f895c160fd0 2024-06-29T14:38:20.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.426+0000 7f89613bd640 1 -- 172.21.15.81:0/1197116627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f895c0cffb0 con 0x7f895c160fd0 2024-06-29T14:38:20.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.830+0000 7f8953fff640 1 -- 172.21.15.81:0/1197116627 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v98) ==== 97+0+0 (secure 0 0 0) 0x7f8954032030 con 0x7f895c160fd0 2024-06-29T14:38:20.828 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T14:38:20.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.833+0000 7f89613bd640 1 -- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f893c076420 msgr2=0x7f893c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.833+0000 7f89613bd640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f893c076420 0x7f893c078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f894c0095f0 tx=0x7f894c005f90 comp rx=0 tx=0).stop 2024-06-29T14:38:20.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.833+0000 7f89613bd640 1 -- 172.21.15.81:0/1197116627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f895c160fd0 msgr2=0x7f895c1613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.833+0000 7f89613bd640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f895c160fd0 0x7f895c1613e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f8954009d90 tx=0x7f8954003c10 comp rx=0 tx=0).stop 2024-06-29T14:38:20.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.833+0000 7f89613bd640 1 -- 172.21.15.81:0/1197116627 shutdown_connections 2024-06-29T14:38:20.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.833+0000 7f89613bd640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f893c076420 0x7f893c078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.833+0000 7f89613bd640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f895c160fd0 0x7f895c1613e0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.833+0000 7f89613bd640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f895c0b8ba0 0x7f895c09fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.833+0000 7f89613bd640 1 --2- 172.21.15.81:0/1197116627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f895c0b6a20 0x7f895c158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.833+0000 7f89613bd640 1 -- 172.21.15.81:0/1197116627 >> 172.21.15.81:0/1197116627 conn(0x7f895c0d3bd0 msgr2=0x7f895c06cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:20.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.834+0000 7f89613bd640 1 -- 172.21.15.81:0/1197116627 shutdown_connections 2024-06-29T14:38:20.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.834+0000 7f89613bd640 1 -- 172.21.15.81:0/1197116627 wait complete. 2024-06-29T14:38:20.845 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:38:20.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.961+0000 7f4f7e368640 1 -- 172.21.15.81:0/563220443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f781412f0 msgr2=0x7f4f780cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.961+0000 7f4f7e368640 1 --2- 172.21.15.81:0/563220443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f781412f0 0x7f4f780cef90 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f4f68005fd0 tx=0x7f4f6802a530 comp rx=0 tx=0).stop 2024-06-29T14:38:20.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.961+0000 7f4f7e368640 1 -- 172.21.15.81:0/563220443 shutdown_connections 2024-06-29T14:38:20.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.961+0000 7f4f7e368640 1 --2- 172.21.15.81:0/563220443 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f780cf4d0 0x7f4f78073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.961+0000 7f4f7e368640 1 --2- 172.21.15.81:0/563220443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f781412f0 0x7f4f780cef90 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.961+0000 7f4f7e368640 1 -- 172.21.15.81:0/563220443 >> 172.21.15.81:0/563220443 conn(0x7f4f7806ccc0 msgr2=0x7f4f780b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:20.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.961+0000 7f4f7e368640 1 -- 172.21.15.81:0/563220443 shutdown_connections 2024-06-29T14:38:20.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.961+0000 7f4f7e368640 1 -- 172.21.15.81:0/563220443 wait complete. 2024-06-29T14:38:20.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.963+0000 7f4f7e368640 1 Processor -- start 2024-06-29T14:38:20.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.963+0000 7f4f7e368640 1 -- start start 2024-06-29T14:38:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.964+0000 7f4f7e368640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f78073570 0x7f4f7813fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.964+0000 7f4f7e368640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f780cf4d0 0x7f4f781f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.964+0000 7f4f7e368640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f781412f0 0x7f4f781f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.964+0000 7f4f7e368640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4f780b3210 con 0x7f4f781412f0 2024-06-29T14:38:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.964+0000 7f4f7e368640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7815f280 con 0x7f4f780cf4d0 2024-06-29T14:38:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.964+0000 7f4f7e368640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4f780b3390 con 0x7f4f78073570 2024-06-29T14:38:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.964+0000 7f4f7c905640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f781412f0 0x7f4f781f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.964+0000 7f4f7c905640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f781412f0 0x7f4f781f6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38548/0 (socket says 172.21.15.81:38548) 2024-06-29T14:38:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.964+0000 7f4f7c905640 1 -- 172.21.15.81:0/3792912291 learned_addr learned my addr 172.21.15.81:0/3792912291 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f77fff640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f78073570 0x7f4f7813fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f777fe640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f780cf4d0 0x7f4f781f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f7c905640 1 -- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f78073570 msgr2=0x7f4f7813fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f7c905640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f78073570 0x7f4f7813fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f7c905640 1 -- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f780cf4d0 msgr2=0x7f4f781f6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f7c905640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f780cf4d0 0x7f4f781f6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f7c905640 1 -- 172.21.15.81:0/3792912291 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f780a94c0 con 0x7f4f781412f0 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f77fff640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f78073570 0x7f4f7813fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f777fe640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f780cf4d0 0x7f4f781f6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f7c905640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f781412f0 0x7f4f781f6a90 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000b9a0 tx=0x7f4f7000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f74ff9640 1 -- 172.21.15.81:0/3792912291 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f70019070 con 0x7f4f781412f0 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f74ff9640 1 -- 172.21.15.81:0/3792912291 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f7000cc30 con 0x7f4f781412f0 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f7e368640 1 -- 172.21.15.81:0/3792912291 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f78158ee0 con 0x7f4f781412f0 2024-06-29T14:38:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f74ff9640 1 -- 172.21.15.81:0/3792912291 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f700146a0 con 0x7f4f781412f0 2024-06-29T14:38:20.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.965+0000 7f4f7e368640 1 -- 172.21.15.81:0/3792912291 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f780af060 con 0x7f4f781412f0 2024-06-29T14:38:20.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.966+0000 7f4f7e368640 1 -- 172.21.15.81:0/3792912291 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f780a9690 con 0x7f4f781412f0 2024-06-29T14:38:20.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.967+0000 7f4f74ff9640 1 -- 172.21.15.81:0/3792912291 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f4f7001c070 con 0x7f4f781412f0 2024-06-29T14:38:20.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.967+0000 7f4f74ff9640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4f64076420 0x7f4f64078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:20.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.967+0000 7f4f74ff9640 1 -- 172.21.15.81:0/3792912291 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 4929+0+0 (secure 0 0 0) 0x7f4f70097100 con 0x7f4f781412f0 2024-06-29T14:38:20.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.968+0000 7f4f77fff640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4f64076420 0x7f4f64078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:20.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.968+0000 7f4f77fff640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4f64076420 0x7f4f64078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4f68033000 tx=0x7f4f68004730 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:20.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:20.972+0000 7f4f74ff9640 1 -- 172.21.15.81:0/3792912291 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f7000ce20 con 0x7f4f781412f0 2024-06-29T14:38:21.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.225+0000 7f4f7e368640 1 -- 172.21.15.81:0/3792912291 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f4f780b38a0 con 0x7f4f781412f0 2024-06-29T14:38:21.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.226+0000 7f4f74ff9640 1 -- 172.21.15.81:0/3792912291 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v32) ==== 80+0+5304 (secure 0 0 0) 0x7f4f70060ce0 con 0x7f4f781412f0 2024-06-29T14:38:21.223 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.224 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T14:38:21.225 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:38:21.226 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.227 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.228 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.229 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.230 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.231 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T14:38:21.232 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.233 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.234 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.235 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T14:38:21.236 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T14:38:21.237 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:38:21.237 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:38:21.237 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:38:21.237 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:38:21.237 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.237 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:21.237 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T14:38:21.237 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:38:21.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 -- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4f64076420 msgr2=0x7f4f64078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4f64076420 0x7f4f64078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4f68033000 tx=0x7f4f68004730 comp rx=0 tx=0).stop 2024-06-29T14:38:21.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 -- 172.21.15.81:0/3792912291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f781412f0 msgr2=0x7f4f781f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f781412f0 0x7f4f781f6a90 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000b9a0 tx=0x7f4f7000be60 comp rx=0 tx=0).stop 2024-06-29T14:38:21.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 -- 172.21.15.81:0/3792912291 shutdown_connections 2024-06-29T14:38:21.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4f64076420 0x7f4f64078930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f781412f0 0x7f4f781f6a90 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f780cf4d0 0x7f4f781f6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 --2- 172.21.15.81:0/3792912291 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f78073570 0x7f4f7813fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 -- 172.21.15.81:0/3792912291 >> 172.21.15.81:0/3792912291 conn(0x7f4f7806ccc0 msgr2=0x7f4f7806e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 -- 172.21.15.81:0/3792912291 shutdown_connections 2024-06-29T14:38:21.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.228+0000 7f4f7e368640 1 -- 172.21.15.81:0/3792912291 wait complete. 2024-06-29T14:38:21.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:21.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.356+0000 7f1a268f6640 1 -- 172.21.15.81:0/2450068191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a08004000 msgr2=0x7f1a08004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.356+0000 7f1a268f6640 1 --2- 172.21.15.81:0/2450068191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a08004000 0x7f1a08004800 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f1a14006fc0 tx=0x7f1a08005830 comp rx=0 tx=0).stop 2024-06-29T14:38:21.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.356+0000 7f1a24e93640 1 -- 172.21.15.81:0/2450068191 reap_dead start 2024-06-29T14:38:21.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.356+0000 7f1a268f6640 1 -- 172.21.15.81:0/2450068191 shutdown_connections 2024-06-29T14:38:21.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.356+0000 7f1a268f6640 1 -- 172.21.15.81:0/2450068191 >> 172.21.15.81:0/2450068191 conn(0x7f1a200d0930 msgr2=0x7f1a200d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.356+0000 7f1a268f6640 1 -- 172.21.15.81:0/2450068191 shutdown_connections 2024-06-29T14:38:21.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.356+0000 7f1a268f6640 1 -- 172.21.15.81:0/2450068191 wait complete. 2024-06-29T14:38:21.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.358+0000 7f1a268f6640 1 Processor -- start 2024-06-29T14:38:21.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.358+0000 7f1a268f6640 1 -- start start 2024-06-29T14:38:21.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.359+0000 7f1a268f6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a08004000 0x7f1a20146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.359+0000 7f1a268f6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a20146160 0x7f1a20156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.359+0000 7f1a268f6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a200b4ba0 0x7f1a200b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.359+0000 7f1a268f6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a200b0050 con 0x7f1a20146160 2024-06-29T14:38:21.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.359+0000 7f1a268f6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a200afed0 con 0x7f1a08004000 2024-06-29T14:38:21.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.359+0000 7f1a268f6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a200b01d0 con 0x7f1a200b4ba0 2024-06-29T14:38:21.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a20146160 0x7f1a20156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a20146160 0x7f1a20156140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38564/0 (socket says 172.21.15.81:38564) 2024-06-29T14:38:21.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1f7fe640 1 -- 172.21.15.81:0/3715278470 learned_addr learned my addr 172.21.15.81:0/3715278470 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a24e93640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a200b4ba0 0x7f1a200b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1ffff640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a08004000 0x7f1a20146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1f7fe640 1 -- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a200b4ba0 msgr2=0x7f1a200b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1f7fe640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a200b4ba0 0x7f1a200b4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1f7fe640 1 -- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a08004000 msgr2=0x7f1a20146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1f7fe640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a08004000 0x7f1a20146a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1f7fe640 1 -- 172.21.15.81:0/3715278470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a200b54f0 con 0x7f1a20146160 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1f7fe640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a20146160 0x7f1a20156140 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1a1400c320 tx=0x7f1a14006c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1cff9640 1 -- 172.21.15.81:0/3715278470 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a14031040 con 0x7f1a20146160 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1cff9640 1 -- 172.21.15.81:0/3715278470 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a14003610 con 0x7f1a20146160 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a268f6640 1 -- 172.21.15.81:0/3715278470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a200b3b00 con 0x7f1a20146160 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a1cff9640 1 -- 172.21.15.81:0/3715278470 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a14003a40 con 0x7f1a20146160 2024-06-29T14:38:21.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.360+0000 7f1a268f6640 1 -- 172.21.15.81:0/3715278470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a200b40c0 con 0x7f1a20146160 2024-06-29T14:38:21.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.361+0000 7f1a268f6640 1 -- 172.21.15.81:0/3715278470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a2006e2a0 con 0x7f1a20146160 2024-06-29T14:38:21.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.362+0000 7f1a1cff9640 1 -- 172.21.15.81:0/3715278470 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f1a14034070 con 0x7f1a20146160 2024-06-29T14:38:21.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.362+0000 7f1a1cff9640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1a0c0763f0 0x7f1a0c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.362+0000 7f1a1cff9640 1 -- 172.21.15.81:0/3715278470 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 4929+0+0 (secure 0 0 0) 0x7f1a140b7760 con 0x7f1a20146160 2024-06-29T14:38:21.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.363+0000 7f1a1ffff640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1a0c0763f0 0x7f1a0c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.363+0000 7f1a1ffff640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1a0c0763f0 0x7f1a0c078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009c50 tx=0x7f1a10005f90 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:21.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.367+0000 7f1a1cff9640 1 -- 172.21.15.81:0/3715278470 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a2006e2a0 con 0x7f1a20146160 2024-06-29T14:38:21.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.546+0000 7f1a268f6640 1 -- 172.21.15.81:0/3715278470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a200d01a0 con 0x7f1a20146160 2024-06-29T14:38:21.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.547+0000 7f1a1cff9640 1 -- 172.21.15.81:0/3715278470 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+12995 (secure 0 0 0) 0x7f1a14081230 con 0x7f1a20146160 2024-06-29T14:38:21.544 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:21.545 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":100,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:38:20.839261+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:21.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 -- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1a0c0763f0 msgr2=0x7f1a0c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1a0c0763f0 0x7f1a0c078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009c50 tx=0x7f1a10005f90 comp rx=0 tx=0).stop 2024-06-29T14:38:21.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 -- 172.21.15.81:0/3715278470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a20146160 msgr2=0x7f1a20156140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a20146160 0x7f1a20156140 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1a1400c320 tx=0x7f1a14006c80 comp rx=0 tx=0).stop 2024-06-29T14:38:21.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 -- 172.21.15.81:0/3715278470 shutdown_connections 2024-06-29T14:38:21.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1a0c0763f0 0x7f1a0c078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a200b4ba0 0x7f1a200b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a20146160 0x7f1a20156140 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 --2- 172.21.15.81:0/3715278470 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a08004000 0x7f1a20146a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 -- 172.21.15.81:0/3715278470 >> 172.21.15.81:0/3715278470 conn(0x7f1a200d0930 msgr2=0x7f1a2015b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 -- 172.21.15.81:0/3715278470 shutdown_connections 2024-06-29T14:38:21.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.549+0000 7f1a268f6640 1 -- 172.21.15.81:0/3715278470 wait complete. 2024-06-29T14:38:21.558 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:38:21.558 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:38:21.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.677+0000 7f1640864640 1 -- 172.21.15.81:0/1695641699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1628004800 msgr2=0x7f1628004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.677+0000 7f1640864640 1 --2- 172.21.15.81:0/1695641699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1628004800 0x7f1628004c10 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f163c0cac90 tx=0x7f1628005830 comp rx=0 tx=0).stop 2024-06-29T14:38:21.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.677+0000 7f163ad9d640 1 -- 172.21.15.81:0/1695641699 reap_dead start 2024-06-29T14:38:21.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.677+0000 7f1640864640 1 -- 172.21.15.81:0/1695641699 shutdown_connections 2024-06-29T14:38:21.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.677+0000 7f1640864640 1 -- 172.21.15.81:0/1695641699 >> 172.21.15.81:0/1695641699 conn(0x7f163c09cf30 msgr2=0x7f163c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.677+0000 7f1640864640 1 -- 172.21.15.81:0/1695641699 shutdown_connections 2024-06-29T14:38:21.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.677+0000 7f1640864640 1 -- 172.21.15.81:0/1695641699 wait complete. 2024-06-29T14:38:21.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.679+0000 7f1640864640 1 Processor -- start 2024-06-29T14:38:21.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.679+0000 7f1640864640 1 -- start start 2024-06-29T14:38:21.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.680+0000 7f1640864640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1628004800 0x7f163c142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.680+0000 7f1640864640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f163c144cf0 0x7f163c145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.680+0000 7f1640864640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f163c15b980 0x7f163c15bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.680+0000 7f1640864640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f163c0d0db0 con 0x7f163c144cf0 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.680+0000 7f1640864640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f163c0b84b0 con 0x7f163c15b980 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.680+0000 7f1640864640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f163c0d1a80 con 0x7f1628004800 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.680+0000 7f1633fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f163c144cf0 0x7f163c145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.680+0000 7f1633fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f163c144cf0 0x7f163c145100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38578/0 (socket says 172.21.15.81:38578) 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.680+0000 7f1633fff640 1 -- 172.21.15.81:0/621556086 learned_addr learned my addr 172.21.15.81:0/621556086 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f163a59c640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1628004800 0x7f163c142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f163ad9d640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f163c15b980 0x7f163c15bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1633fff640 1 -- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1628004800 msgr2=0x7f163c142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1633fff640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1628004800 0x7f163c142c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1633fff640 1 -- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f163c15b980 msgr2=0x7f163c15bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1633fff640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f163c15b980 0x7f163c15bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1633fff640 1 -- 172.21.15.81:0/621556086 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c145a50 con 0x7f163c144cf0 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f163ad9d640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f163c15b980 0x7f163c15bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f163a59c640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1628004800 0x7f163c142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1633fff640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f163c144cf0 0x7f163c145100 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f1624009a70 tx=0x7f1624033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1632ffd640 1 -- 172.21.15.81:0/621556086 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1624043070 con 0x7f163c144cf0 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1632ffd640 1 -- 172.21.15.81:0/621556086 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1624004120 con 0x7f163c144cf0 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1640864640 1 -- 172.21.15.81:0/621556086 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f163c15aea0 con 0x7f163c144cf0 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1632ffd640 1 -- 172.21.15.81:0/621556086 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1624038040 con 0x7f163c144cf0 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.681+0000 7f1640864640 1 -- 172.21.15.81:0/621556086 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f163c1431d0 con 0x7f163c144cf0 2024-06-29T14:38:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.682+0000 7f1640864640 1 -- 172.21.15.81:0/621556086 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f163c0d0480 con 0x7f163c144cf0 2024-06-29T14:38:21.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.683+0000 7f1632ffd640 1 -- 172.21.15.81:0/621556086 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f1624046070 con 0x7f163c144cf0 2024-06-29T14:38:21.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.683+0000 7f1632ffd640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f161c0763f0 0x7f161c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.683+0000 7f1632ffd640 1 -- 172.21.15.81:0/621556086 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 4929+0+0 (secure 0 0 0) 0x7f16240b7c50 con 0x7f163c144cf0 2024-06-29T14:38:21.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.684+0000 7f163a59c640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f161c0763f0 0x7f161c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.684+0000 7f163a59c640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f161c0763f0 0x7f161c078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f162c0036b0 tx=0x7f162c006b70 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:21.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.688+0000 7f1632ffd640 1 -- 172.21.15.81:0/621556086 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f16240bd020 con 0x7f163c144cf0 2024-06-29T14:38:21.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.868+0000 7f1640864640 1 -- 172.21.15.81:0/621556086 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f163c1457e0 con 0x7f163c144cf0 2024-06-29T14:38:21.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.868+0000 7f1632ffd640 1 -- 172.21.15.81:0/621556086 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) ==== 93+0+10 (secure 0 0 0) 0x7f16240810c0 con 0x7f163c144cf0 2024-06-29T14:38:21.866 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:38:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 -- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f161c0763f0 msgr2=0x7f161c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f161c0763f0 0x7f161c078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f162c0036b0 tx=0x7f162c006b70 comp rx=0 tx=0).stop 2024-06-29T14:38:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 -- 172.21.15.81:0/621556086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f163c144cf0 msgr2=0x7f163c145100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f163c144cf0 0x7f163c145100 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f1624009a70 tx=0x7f1624033670 comp rx=0 tx=0).stop 2024-06-29T14:38:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 -- 172.21.15.81:0/621556086 shutdown_connections 2024-06-29T14:38:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f161c0763f0 0x7f161c078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f163c15b980 0x7f163c15bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f163c144cf0 0x7f163c145100 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 --2- 172.21.15.81:0/621556086 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1628004800 0x7f163c142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 -- 172.21.15.81:0/621556086 >> 172.21.15.81:0/621556086 conn(0x7f163c09cf30 msgr2=0x7f163c161200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.870+0000 7f1640864640 1 -- 172.21.15.81:0/621556086 shutdown_connections 2024-06-29T14:38:21.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.871+0000 7f1640864640 1 -- 172.21.15.81:0/621556086 wait complete. 2024-06-29T14:38:21.879 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:38:21.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.995+0000 7f64c1ce4640 1 -- 172.21.15.81:0/1315162547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0aea10 msgr2=0x7f64bc0aee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.995+0000 7f64c1ce4640 1 --2- 172.21.15.81:0/1315162547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0aea10 0x7f64bc0aee20 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f64b4005fd0 tx=0x7f64b402aa90 comp rx=0 tx=0).stop 2024-06-29T14:38:21.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.995+0000 7f64c1ce4640 1 -- 172.21.15.81:0/1315162547 shutdown_connections 2024-06-29T14:38:21.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.995+0000 7f64c1ce4640 1 --2- 172.21.15.81:0/1315162547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0aea10 0x7f64bc0aee20 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.995+0000 7f64c1ce4640 1 -- 172.21.15.81:0/1315162547 >> 172.21.15.81:0/1315162547 conn(0x7f64bc072950 msgr2=0x7f64bc0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:21.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.995+0000 7f64c1ce4640 1 -- 172.21.15.81:0/1315162547 shutdown_connections 2024-06-29T14:38:21.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.995+0000 7f64c1ce4640 1 -- 172.21.15.81:0/1315162547 wait complete. 2024-06-29T14:38:21.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.996+0000 7f64c1ce4640 1 Processor -- start 2024-06-29T14:38:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.997+0000 7f64c1ce4640 1 -- start start 2024-06-29T14:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64c1ce4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64bc06f850 0x7f64bc1f1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64c1ce4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64bc074520 0x7f64bc0b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64c1ce4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0aea10 0x7f64bc0b1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64c1ce4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc15b190 con 0x7f64bc0aea10 2024-06-29T14:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64c1ce4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc15b010 con 0x7f64bc06f850 2024-06-29T14:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64c1ce4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f64bc0af360 con 0x7f64bc074520 2024-06-29T14:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64bbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0aea10 0x7f64bc0b1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64bbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0aea10 0x7f64bc0b1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38596/0 (socket says 172.21.15.81:38596) 2024-06-29T14:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64bbfff640 1 -- 172.21.15.81:0/3336255196 learned_addr learned my addr 172.21.15.81:0/3336255196 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64bb7fe640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64bc06f850 0x7f64bc1f1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64baffd640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64bc074520 0x7f64bc0b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64bbfff640 1 -- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64bc074520 msgr2=0x7f64bc0b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64bbfff640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64bc074520 0x7f64bc0b1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64bbfff640 1 -- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64bc06f850 msgr2=0x7f64bc1f1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64bbfff640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64bc06f850 0x7f64bc1f1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64bbfff640 1 -- 172.21.15.81:0/3336255196 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64bc143360 con 0x7f64bc0aea10 2024-06-29T14:38:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.998+0000 7f64bbfff640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0aea10 0x7f64bc0b1540 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009bf0 tx=0x7f64b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.999+0000 7f64a3fff640 1 -- 172.21.15.81:0/3336255196 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b4046070 con 0x7f64bc0aea10 2024-06-29T14:38:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.999+0000 7f64a3fff640 1 -- 172.21.15.81:0/3336255196 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64b4030ea0 con 0x7f64bc0aea10 2024-06-29T14:38:21.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.999+0000 7f64c1ce4640 1 -- 172.21.15.81:0/3336255196 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64bc143640 con 0x7f64bc0aea10 2024-06-29T14:38:21.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.999+0000 7f64a3fff640 1 -- 172.21.15.81:0/3336255196 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b4041410 con 0x7f64bc0aea10 2024-06-29T14:38:21.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:21.999+0000 7f64c1ce4640 1 -- 172.21.15.81:0/3336255196 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64bc14faa0 con 0x7f64bc0aea10 2024-06-29T14:38:22.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.000+0000 7f64c1ce4640 1 -- 172.21.15.81:0/3336255196 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64bc0af180 con 0x7f64bc0aea10 2024-06-29T14:38:22.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.000+0000 7f64a3fff640 1 -- 172.21.15.81:0/3336255196 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f64b4049070 con 0x7f64bc0aea10 2024-06-29T14:38:22.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.001+0000 7f64a3fff640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64a80763f0 0x7f64a8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.001+0000 7f64a3fff640 1 -- 172.21.15.81:0/3336255196 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 4929+0+0 (secure 0 0 0) 0x7f64b40ba4c0 con 0x7f64bc0aea10 2024-06-29T14:38:22.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.001+0000 7f64bb7fe640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64a80763f0 0x7f64a8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.002+0000 7f64bb7fe640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64a80763f0 0x7f64a8078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64ac000f60 tx=0x7f64ac009040 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.005+0000 7f64a3fff640 1 -- 172.21.15.81:0/3336255196 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f64bc0af180 con 0x7f64bc0aea10 2024-06-29T14:38:22.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.188+0000 7f64c1ce4640 1 -- 172.21.15.81:0/3336255196 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f64bc0aee20 con 0x7f64bc0aea10 2024-06-29T14:38:22.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.188+0000 7f64a3fff640 1 -- 172.21.15.81:0/3336255196 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v100) ==== 84+0+65 (secure 0 0 0) 0x7f64b4032030 con 0x7f64bc0aea10 2024-06-29T14:38:22.186 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:22.186 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:38:22.186 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:22.186 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T14:38:22.186 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T14:38:22.187 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:22.187 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:38:22.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.190+0000 7f64c1ce4640 1 -- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64a80763f0 msgr2=0x7f64a8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.190+0000 7f64c1ce4640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64a80763f0 0x7f64a8078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f64ac000f60 tx=0x7f64ac009040 comp rx=0 tx=0).stop 2024-06-29T14:38:22.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.190+0000 7f64c1ce4640 1 -- 172.21.15.81:0/3336255196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0aea10 msgr2=0x7f64bc0b1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.190+0000 7f64c1ce4640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0aea10 0x7f64bc0b1540 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009bf0 tx=0x7f64b4004270 comp rx=0 tx=0).stop 2024-06-29T14:38:22.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.190+0000 7f64c1ce4640 1 -- 172.21.15.81:0/3336255196 shutdown_connections 2024-06-29T14:38:22.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.190+0000 7f64c1ce4640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64a80763f0 0x7f64a8078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.190+0000 7f64c1ce4640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0aea10 0x7f64bc0b1540 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.190+0000 7f64c1ce4640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64bc074520 0x7f64bc0b1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.190+0000 7f64c1ce4640 1 --2- 172.21.15.81:0/3336255196 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64bc06f850 0x7f64bc1f1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.190+0000 7f64c1ce4640 1 -- 172.21.15.81:0/3336255196 >> 172.21.15.81:0/3336255196 conn(0x7f64bc072950 msgr2=0x7f64bc0cead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:22.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.191+0000 7f64c1ce4640 1 -- 172.21.15.81:0/3336255196 shutdown_connections 2024-06-29T14:38:22.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.191+0000 7f64c1ce4640 1 -- 172.21.15.81:0/3336255196 wait complete. 2024-06-29T14:38:22.199 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:38:22.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:38:22.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.319+0000 7fcfe3912640 1 -- 172.21.15.81:0/1707089892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfdc0b1070 msgr2=0x7fcfdc0b1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.319+0000 7fcfe3912640 1 --2- 172.21.15.81:0/1707089892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfdc0b1070 0x7fcfdc0b1480 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc073020 tx=0x7fcfd402aa00 comp rx=0 tx=0).stop 2024-06-29T14:38:22.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.319+0000 7fcfe3912640 1 -- 172.21.15.81:0/1707089892 shutdown_connections 2024-06-29T14:38:22.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.319+0000 7fcfe3912640 1 --2- 172.21.15.81:0/1707089892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfdc0b1070 0x7fcfdc0b1480 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.319+0000 7fcfe3912640 1 -- 172.21.15.81:0/1707089892 >> 172.21.15.81:0/1707089892 conn(0x7fcfdc072970 msgr2=0x7fcfdc0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:22.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.319+0000 7fcfe3912640 1 -- 172.21.15.81:0/1707089892 shutdown_connections 2024-06-29T14:38:22.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.319+0000 7fcfe3912640 1 -- 172.21.15.81:0/1707089892 wait complete. 2024-06-29T14:38:22.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.321+0000 7fcfe3912640 1 Processor -- start 2024-06-29T14:38:22.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.321+0000 7fcfe3912640 1 -- start start 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe3912640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfdc073b30 0x7fcfdc09b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe3912640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfdc0b1070 0x7fcfdc09c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe3912640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcfdc15cd70 0x7fcfdc09ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe3912640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc0747b0 con 0x7fcfdc073b30 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe3912640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc159cc0 con 0x7fcfdc15cd70 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe3912640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcfdc074930 con 0x7fcfdc0b1070 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe16ae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfdc073b30 0x7fcfdc09b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe16ae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfdc073b30 0x7fcfdc09b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38606/0 (socket says 172.21.15.81:38606) 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe16ae640 1 -- 172.21.15.81:0/2781587878 learned_addr learned my addr 172.21.15.81:0/2781587878 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe1eaf640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcfdc15cd70 0x7fcfdc09ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe0ead640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfdc0b1070 0x7fcfdc09c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe16ae640 1 -- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfdc0b1070 msgr2=0x7fcfdc09c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe16ae640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfdc0b1070 0x7fcfdc09c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe16ae640 1 -- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcfdc15cd70 msgr2=0x7fcfdc09ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe16ae640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcfdc15cd70 0x7fcfdc09ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe16ae640 1 -- 172.21.15.81:0/2781587878 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc0cd700 con 0x7fcfdc073b30 2024-06-29T14:38:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.322+0000 7fcfe0ead640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfdc0b1070 0x7fcfdc09c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.323+0000 7fcfe16ae640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfdc073b30 0x7fcfdc09b080 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800b410 tx=0x7fcfd800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.323+0000 7fcfd1ffb640 1 -- 172.21.15.81:0/2781587878 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8015020 con 0x7fcfdc073b30 2024-06-29T14:38:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.323+0000 7fcfd1ffb640 1 -- 172.21.15.81:0/2781587878 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfd8011920 con 0x7fcfdc073b30 2024-06-29T14:38:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.323+0000 7fcfe3912640 1 -- 172.21.15.81:0/2781587878 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc09b9f0 con 0x7fcfdc073b30 2024-06-29T14:38:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.323+0000 7fcfd1ffb640 1 -- 172.21.15.81:0/2781587878 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8011d70 con 0x7fcfdc073b30 2024-06-29T14:38:22.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.323+0000 7fcfe3912640 1 -- 172.21.15.81:0/2781587878 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfdc1572a0 con 0x7fcfdc073b30 2024-06-29T14:38:22.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.324+0000 7fcfe3912640 1 -- 172.21.15.81:0/2781587878 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfdc0cd8d0 con 0x7fcfdc073b30 2024-06-29T14:38:22.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.325+0000 7fcfd1ffb640 1 -- 172.21.15.81:0/2781587878 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fcfd8018020 con 0x7fcfdc073b30 2024-06-29T14:38:22.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.325+0000 7fcfd1ffb640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fcfc00763f0 0x7fcfc0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.325+0000 7fcfd1ffb640 1 -- 172.21.15.81:0/2781587878 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 4929+0+0 (secure 0 0 0) 0x7fcfd8097680 con 0x7fcfdc073b30 2024-06-29T14:38:22.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.325+0000 7fcfe0ead640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fcfc00763f0 0x7fcfc0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.326+0000 7fcfe0ead640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fcfc00763f0 0x7fcfc0078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc000f60 tx=0x7fcfcc009040 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.330+0000 7fcfd1ffb640 1 -- 172.21.15.81:0/2781587878 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcfd801f020 con 0x7fcfdc073b30 2024-06-29T14:38:22.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.509+0000 7fcfe3912640 1 -- 172.21.15.81:0/2781587878 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fcfdc0b1480 con 0x7fcfdc073b30 2024-06-29T14:38:22.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.840+0000 7fcfd1ffb640 1 -- 172.21.15.81:0/2781587878 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v101) ==== 138+0+0 (secure 0 0 0) 0x7fcfd8061140 con 0x7fcfdc073b30 2024-06-29T14:38:22.838 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T14:38:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 -- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fcfc00763f0 msgr2=0x7fcfc0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fcfc00763f0 0x7fcfc0078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc000f60 tx=0x7fcfcc009040 comp rx=0 tx=0).stop 2024-06-29T14:38:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 -- 172.21.15.81:0/2781587878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfdc073b30 msgr2=0x7fcfdc09b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfdc073b30 0x7fcfdc09b080 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800b410 tx=0x7fcfd800b8d0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 -- 172.21.15.81:0/2781587878 shutdown_connections 2024-06-29T14:38:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fcfc00763f0 0x7fcfc0078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcfdc15cd70 0x7fcfdc09ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfdc0b1070 0x7fcfdc09c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 --2- 172.21.15.81:0/2781587878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfdc073b30 0x7fcfdc09b080 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 -- 172.21.15.81:0/2781587878 >> 172.21.15.81:0/2781587878 conn(0x7fcfdc072970 msgr2=0x7fcfdc15f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 -- 172.21.15.81:0/2781587878 shutdown_connections 2024-06-29T14:38:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.842+0000 7fcfe3912640 1 -- 172.21.15.81:0/2781587878 wait complete. 2024-06-29T14:38:22.854 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:38:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.978+0000 7f464be3c640 1 -- 172.21.15.81:0/3980015316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f464406f6a0 msgr2=0x7f464415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.978+0000 7f464be3c640 1 --2- 172.21.15.81:0/3980015316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f464406f6a0 0x7f464415aed0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f46440743b0 tx=0x7f463402a5f0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.978+0000 7f464be3c640 1 -- 172.21.15.81:0/3980015316 shutdown_connections 2024-06-29T14:38:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.978+0000 7f464be3c640 1 --2- 172.21.15.81:0/3980015316 >> v2:172.21.15.167:3300/0 conn(0x7f46440b3960 0x7f46440b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.978+0000 7f464be3c640 1 --2- 172.21.15.81:0/3980015316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f464406f6a0 0x7f464415aed0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.978+0000 7f464be3c640 1 -- 172.21.15.81:0/3980015316 >> 172.21.15.81:0/3980015316 conn(0x7f46440d0930 msgr2=0x7f46440d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.978+0000 7f464be3c640 1 -- 172.21.15.81:0/3980015316 shutdown_connections 2024-06-29T14:38:22.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.978+0000 7f464be3c640 1 -- 172.21.15.81:0/3980015316 wait complete. 2024-06-29T14:38:22.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.980+0000 7f464be3c640 1 Processor -- start 2024-06-29T14:38:22.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.980+0000 7f464be3c640 1 -- start start 2024-06-29T14:38:22.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f464be3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f464406f6a0 0x7f4644150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f464be3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46440b3960 0x7f46441f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f464be3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46440b57b0 0x7f46440ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f464be3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46440b4430 con 0x7f464406f6a0 2024-06-29T14:38:22.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f464be3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f46440b42b0 con 0x7f46440b57b0 2024-06-29T14:38:22.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f464be3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f46440b45b0 con 0x7f46440b3960 2024-06-29T14:38:22.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f4649bd8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f464406f6a0 0x7f4644150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f4649bd8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f464406f6a0 0x7f4644150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38624/0 (socket says 172.21.15.81:38624) 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f4649bd8640 1 -- 172.21.15.81:0/2502119078 learned_addr learned my addr 172.21.15.81:0/2502119078 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f46493d7640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46440b3960 0x7f46441f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.981+0000 7f464a3d9640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46440b57b0 0x7f46440ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f4649bd8640 1 -- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46440b3960 msgr2=0x7f46441f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f4649bd8640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46440b3960 0x7f46441f25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f4649bd8640 1 -- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46440b57b0 msgr2=0x7f46440ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f4649bd8640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46440b57b0 0x7f46440ce1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f4649bd8640 1 -- 172.21.15.81:0/2502119078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4644150e90 con 0x7f464406f6a0 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f4649bd8640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f464406f6a0 0x7f4644150950 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f464000c3a0 tx=0x7f464000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f463a7fc640 1 -- 172.21.15.81:0/2502119078 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640017070 con 0x7f464406f6a0 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f463a7fc640 1 -- 172.21.15.81:0/2502119078 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46400053c0 con 0x7f464406f6a0 2024-06-29T14:38:22.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f464be3c640 1 -- 172.21.15.81:0/2502119078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46441f2b00 con 0x7f464406f6a0 2024-06-29T14:38:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f463a7fc640 1 -- 172.21.15.81:0/2502119078 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640003850 con 0x7f464406f6a0 2024-06-29T14:38:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.982+0000 7f464be3c640 1 -- 172.21.15.81:0/2502119078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4644151680 con 0x7f464406f6a0 2024-06-29T14:38:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.983+0000 7f464be3c640 1 -- 172.21.15.81:0/2502119078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f464406e2a0 con 0x7f464406f6a0 2024-06-29T14:38:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.984+0000 7f463a7fc640 1 -- 172.21.15.81:0/2502119078 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f464001a070 con 0x7f464406f6a0 2024-06-29T14:38:22.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.984+0000 7f463a7fc640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4628076420 0x7f4628078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:22.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.984+0000 7f463a7fc640 1 -- 172.21.15.81:0/2502119078 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5299+0+0 (secure 0 0 0) 0x7f4640096b00 con 0x7f464406f6a0 2024-06-29T14:38:22.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.984+0000 7f46493d7640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4628076420 0x7f4628078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:22.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.985+0000 7f46493d7640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4628076420 0x7f4628078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4634033000 tx=0x7f4634004730 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:22.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:22.989+0000 7f463a7fc640 1 -- 172.21.15.81:0/2502119078 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f464406e2a0 con 0x7f464406f6a0 2024-06-29T14:38:23.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.170+0000 7f464be3c640 1 -- 172.21.15.81:0/2502119078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f46440b4a60 con 0x7f464406f6a0 2024-06-29T14:38:23.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.844+0000 7f463a7fc640 1 -- 172.21.15.81:0/2502119078 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v102) ==== 170+0+0 (secure 0 0 0) 0x7f464005fe00 con 0x7f464406f6a0 2024-06-29T14:38:23.842 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T14:38:23.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 -- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4628076420 msgr2=0x7f4628078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:23.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4628076420 0x7f4628078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4634033000 tx=0x7f4634004730 comp rx=0 tx=0).stop 2024-06-29T14:38:23.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 -- 172.21.15.81:0/2502119078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f464406f6a0 msgr2=0x7f4644150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:23.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f464406f6a0 0x7f4644150950 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f464000c3a0 tx=0x7f464000c860 comp rx=0 tx=0).stop 2024-06-29T14:38:23.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 -- 172.21.15.81:0/2502119078 shutdown_connections 2024-06-29T14:38:23.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4628076420 0x7f4628078930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46440b57b0 0x7f46440ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46440b3960 0x7f46441f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 --2- 172.21.15.81:0/2502119078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f464406f6a0 0x7f4644150950 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 -- 172.21.15.81:0/2502119078 >> 172.21.15.81:0/2502119078 conn(0x7f46440d0930 msgr2=0x7f4644159500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:23.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 -- 172.21.15.81:0/2502119078 shutdown_connections 2024-06-29T14:38:23.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.848+0000 7f464be3c640 1 -- 172.21.15.81:0/2502119078 wait complete. 2024-06-29T14:38:23.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:38:23.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.988+0000 7f10ec311640 1 -- 172.21.15.81:0/1673764923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e40cf600 msgr2=0x7f10e40cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:23.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.988+0000 7f10ec311640 1 --2- 172.21.15.81:0/1673764923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e40cf600 0x7f10e40cfa10 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f10dc005fd0 tx=0x7f10dc02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.988+0000 7f10ec311640 1 -- 172.21.15.81:0/1673764923 shutdown_connections 2024-06-29T14:38:23.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.988+0000 7f10ec311640 1 --2- 172.21.15.81:0/1673764923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e40cf600 0x7f10e40cfa10 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.988+0000 7f10ec311640 1 --2- 172.21.15.81:0/1673764923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e40ce3a0 0x7f10e40ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.988+0000 7f10ec311640 1 -- 172.21.15.81:0/1673764923 >> 172.21.15.81:0/1673764923 conn(0x7f10e40b35a0 msgr2=0x7f10e40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:23.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.988+0000 7f10ec311640 1 -- 172.21.15.81:0/1673764923 shutdown_connections 2024-06-29T14:38:23.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.988+0000 7f10ec311640 1 -- 172.21.15.81:0/1673764923 wait complete. 2024-06-29T14:38:23.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.990+0000 7f10ec311640 1 Processor -- start 2024-06-29T14:38:23.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.990+0000 7f10ec311640 1 -- start start 2024-06-29T14:38:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.991+0000 7f10ec311640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e40ce3a0 0x7f10e409c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.991+0000 7f10ec311640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e40cf600 0x7f10e41d3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.991+0000 7f10ec311640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e41dddb0 0x7f10e40b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.991+0000 7f10ec311640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f10e40b2830 con 0x7f10e41dddb0 2024-06-29T14:38:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.991+0000 7f10ec311640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f10e40b26b0 con 0x7f10e40cf600 2024-06-29T14:38:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.991+0000 7f10ec311640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f10e40b6df0 con 0x7f10e40ce3a0 2024-06-29T14:38:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea8ae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e41dddb0 0x7f10e40b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea8ae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e41dddb0 0x7f10e40b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38646/0 (socket says 172.21.15.81:38646) 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea8ae640 1 -- 172.21.15.81:0/738058940 learned_addr learned my addr 172.21.15.81:0/738058940 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea0ad640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e40ce3a0 0x7f10e409c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10e98ac640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e40cf600 0x7f10e41d3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea8ae640 1 -- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e40ce3a0 msgr2=0x7f10e409c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea8ae640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e40ce3a0 0x7f10e409c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea8ae640 1 -- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e40cf600 msgr2=0x7f10e41d3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea8ae640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e40cf600 0x7f10e41d3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea8ae640 1 -- 172.21.15.81:0/738058940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e409eab0 con 0x7f10e41dddb0 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea0ad640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e40ce3a0 0x7f10e409c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ea8ae640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e41dddb0 0x7f10e40b58f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f10dc006010 tx=0x7f10dc003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10e2ffd640 1 -- 172.21.15.81:0/738058940 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc03e030 con 0x7f10e41dddb0 2024-06-29T14:38:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10e2ffd640 1 -- 172.21.15.81:0/738058940 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10dc003c20 con 0x7f10e41dddb0 2024-06-29T14:38:23.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ec311640 1 -- 172.21.15.81:0/738058940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e409ed90 con 0x7f10e41dddb0 2024-06-29T14:38:23.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10e2ffd640 1 -- 172.21.15.81:0/738058940 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc035040 con 0x7f10e41dddb0 2024-06-29T14:38:23.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.992+0000 7f10ec311640 1 -- 172.21.15.81:0/738058940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e409e780 con 0x7f10e41dddb0 2024-06-29T14:38:23.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.993+0000 7f10ec311640 1 -- 172.21.15.81:0/738058940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e40b7360 con 0x7f10e41dddb0 2024-06-29T14:38:23.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.994+0000 7f10e2ffd640 1 -- 172.21.15.81:0/738058940 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f10dc041030 con 0x7f10e41dddb0 2024-06-29T14:38:23.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.995+0000 7f10e2ffd640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10d00763f0 0x7f10d0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:23.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.995+0000 7f10e2ffd640 1 -- 172.21.15.81:0/738058940 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 5681+0+0 (secure 0 0 0) 0x7f10dc0b8320 con 0x7f10e41dddb0 2024-06-29T14:38:23.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.995+0000 7f10ea0ad640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10d00763f0 0x7f10d0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:23.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.995+0000 7f10ea0ad640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10d00763f0 0x7f10d0078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f10d4007fd0 tx=0x7f10d4009040 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:23.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:23.999+0000 7f10e2ffd640 1 -- 172.21.15.81:0/738058940 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f10dc081ac0 con 0x7f10e41dddb0 2024-06-29T14:38:24.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.215+0000 7f10ec311640 1 -- 172.21.15.81:0/738058940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f10e40d0900 con 0x7f10e41dddb0 2024-06-29T14:38:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.872+0000 7f10e2ffd640 1 -- 172.21.15.81:0/738058940 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v47) ==== 174+0+0 (secure 0 0 0) 0x7f10dc0813c0 con 0x7f10e41dddb0 2024-06-29T14:38:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 12 and data pool 13 2024-06-29T14:38:24.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.882+0000 7f10ec311640 1 -- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10d00763f0 msgr2=0x7f10d0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:24.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.882+0000 7f10ec311640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10d00763f0 0x7f10d0078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f10d4007fd0 tx=0x7f10d4009040 comp rx=0 tx=0).stop 2024-06-29T14:38:24.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.882+0000 7f10ec311640 1 -- 172.21.15.81:0/738058940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e41dddb0 msgr2=0x7f10e40b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:24.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.882+0000 7f10ec311640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e41dddb0 0x7f10e40b58f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f10dc006010 tx=0x7f10dc003820 comp rx=0 tx=0).stop 2024-06-29T14:38:24.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.884+0000 7f10ec311640 1 -- 172.21.15.81:0/738058940 shutdown_connections 2024-06-29T14:38:24.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.884+0000 7f10ec311640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10d00763f0 0x7f10d0078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.884+0000 7f10ec311640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e41dddb0 0x7f10e40b58f0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.884+0000 7f10ec311640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e40cf600 0x7f10e41d3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.884+0000 7f10ec311640 1 --2- 172.21.15.81:0/738058940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e40ce3a0 0x7f10e409c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:24.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.884+0000 7f10ec311640 1 -- 172.21.15.81:0/738058940 >> 172.21.15.81:0/738058940 conn(0x7f10e40b35a0 msgr2=0x7f10e40b19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:24.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.884+0000 7f10ec311640 1 -- 172.21.15.81:0/738058940 shutdown_connections 2024-06-29T14:38:24.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:24.884+0000 7f10ec311640 1 -- 172.21.15.81:0/738058940 wait complete. 2024-06-29T14:38:24.902 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:25.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.019+0000 7fca79538640 1 -- 172.21.15.81:0/3694575038 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca740b5250 msgr2=0x7fca740d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.019+0000 7fca79538640 1 --2- 172.21.15.81:0/3694575038 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca740b5250 0x7fca740d09c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009830 tx=0x7fca5c02a5a0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.019+0000 7fca79538640 1 -- 172.21.15.81:0/3694575038 shutdown_connections 2024-06-29T14:38:25.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.019+0000 7fca79538640 1 --2- 172.21.15.81:0/3694575038 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca740b5250 0x7fca740d09c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.019+0000 7fca79538640 1 --2- 172.21.15.81:0/3694575038 >> v2:172.21.15.167:3300/0 conn(0x7fca740b9040 0x7fca740d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.019+0000 7fca79538640 1 -- 172.21.15.81:0/3694575038 >> 172.21.15.81:0/3694575038 conn(0x7fca7409cf30 msgr2=0x7fca740d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:25.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.019+0000 7fca79538640 1 -- 172.21.15.81:0/3694575038 shutdown_connections 2024-06-29T14:38:25.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.019+0000 7fca79538640 1 -- 172.21.15.81:0/3694575038 wait complete. 2024-06-29T14:38:25.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.021+0000 7fca79538640 1 Processor -- start 2024-06-29T14:38:25.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.021+0000 7fca79538640 1 -- start start 2024-06-29T14:38:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca79538640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca740b5250 0x7fca74153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca79538640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca740b7290 0x7fca740c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca79538640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca740b9040 0x7fca74155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca79538640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fca740b7db0 con 0x7fca740b5250 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca79538640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fca740d2d50 con 0x7fca740b9040 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca79538640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fca740b7f30 con 0x7fca740b7290 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca72ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca740b5250 0x7fca74153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca72ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca740b5250 0x7fca74153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38658/0 (socket says 172.21.15.81:38658) 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca72ffd640 1 -- 172.21.15.81:0/1383497586 learned_addr learned my addr 172.21.15.81:0/1383497586 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca727fc640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca740b7290 0x7fca740c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca737fe640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca740b9040 0x7fca74155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca72ffd640 1 -- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca740b7290 msgr2=0x7fca740c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca72ffd640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca740b7290 0x7fca740c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca72ffd640 1 -- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca740b9040 msgr2=0x7fca74155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca72ffd640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca740b9040 0x7fca74155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca72ffd640 1 -- 172.21.15.81:0/1383497586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca740c87d0 con 0x7fca740b5250 2024-06-29T14:38:25.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca727fc640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca740b7290 0x7fca740c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:25.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca72ffd640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca740b5250 0x7fca74153140 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00e9f0 tx=0x7fca6c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:25.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca6b7fe640 1 -- 172.21.15.81:0/1383497586 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c00cd80 con 0x7fca740b5250 2024-06-29T14:38:25.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca6b7fe640 1 -- 172.21.15.81:0/1383497586 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca6c004020 con 0x7fca740b5250 2024-06-29T14:38:25.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca79538640 1 -- 172.21.15.81:0/1383497586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca740c84f0 con 0x7fca740b5250 2024-06-29T14:38:25.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca6b7fe640 1 -- 172.21.15.81:0/1383497586 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c004450 con 0x7fca740b5250 2024-06-29T14:38:25.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.023+0000 7fca79538640 1 -- 172.21.15.81:0/1383497586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca740b1050 con 0x7fca740b5250 2024-06-29T14:38:25.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.025+0000 7fca79538640 1 -- 172.21.15.81:0/1383497586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca7406e830 con 0x7fca740b5250 2024-06-29T14:38:25.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.025+0000 7fca6b7fe640 1 -- 172.21.15.81:0/1383497586 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fca6c0035f0 con 0x7fca740b5250 2024-06-29T14:38:25.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.025+0000 7fca6b7fe640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fca54076420 0x7fca54078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.026+0000 7fca6b7fe640 1 -- 172.21.15.81:0/1383497586 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5781+0+0 (secure 0 0 0) 0x7fca6c098280 con 0x7fca740b5250 2024-06-29T14:38:25.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.026+0000 7fca727fc640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fca54076420 0x7fca54078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.026+0000 7fca727fc640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fca54076420 0x7fca54078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fca5c034000 tx=0x7fca5c004730 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.030+0000 7fca6b7fe640 1 -- 172.21.15.81:0/1383497586 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fca7406e830 con 0x7fca740b5250 2024-06-29T14:38:25.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.211+0000 7fca79538640 1 -- 172.21.15.81:0/1383497586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca740b1490 con 0x7fca740b5250 2024-06-29T14:38:25.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.212+0000 7fca6b7fe640 1 -- 172.21.15.81:0/1383497586 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16786 (secure 0 0 0) 0x7fca6c061a00 con 0x7fca740b5250 2024-06-29T14:38:25.210 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:25.210 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":103,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:38:24.216432+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.514776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.175620+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:25.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 -- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fca54076420 msgr2=0x7fca54078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fca54076420 0x7fca54078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fca5c034000 tx=0x7fca5c004730 comp rx=0 tx=0).stop 2024-06-29T14:38:25.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 -- 172.21.15.81:0/1383497586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca740b5250 msgr2=0x7fca74153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca740b5250 0x7fca74153140 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00e9f0 tx=0x7fca6c00eeb0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 -- 172.21.15.81:0/1383497586 shutdown_connections 2024-06-29T14:38:25.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fca54076420 0x7fca54078930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fca740b9040 0x7fca74155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fca740b7290 0x7fca740c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 --2- 172.21.15.81:0/1383497586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca740b5250 0x7fca74153140 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 -- 172.21.15.81:0/1383497586 >> 172.21.15.81:0/1383497586 conn(0x7fca7409cf30 msgr2=0x7fca740d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:25.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 -- 172.21.15.81:0/1383497586 shutdown_connections 2024-06-29T14:38:25.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.214+0000 7fca79538640 1 -- 172.21.15.81:0/1383497586 wait complete. 2024-06-29T14:38:25.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:25.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.337+0000 7f13ab281640 1 -- 172.21.15.81:0/2630574755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a40b9040 msgr2=0x7f13a40d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.337+0000 7f13ab281640 1 --2- 172.21.15.81:0/2630574755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a40b9040 0x7f13a40d0480 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f13a40a1b70 tx=0x7f13a002a940 comp rx=0 tx=0).stop 2024-06-29T14:38:25.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.337+0000 7f13ab281640 1 -- 172.21.15.81:0/2630574755 shutdown_connections 2024-06-29T14:38:25.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.337+0000 7f13ab281640 1 --2- 172.21.15.81:0/2630574755 >> v2:172.21.15.167:3300/0 conn(0x7f13a40b5250 0x7f13a40d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.337+0000 7f13ab281640 1 --2- 172.21.15.81:0/2630574755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a40b9040 0x7f13a40d0480 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.337+0000 7f13ab281640 1 -- 172.21.15.81:0/2630574755 >> 172.21.15.81:0/2630574755 conn(0x7f13a409cf30 msgr2=0x7f13a40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:25.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.337+0000 7f13ab281640 1 -- 172.21.15.81:0/2630574755 shutdown_connections 2024-06-29T14:38:25.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.337+0000 7f13ab281640 1 -- 172.21.15.81:0/2630574755 wait complete. 2024-06-29T14:38:25.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.339+0000 7f13ab281640 1 Processor -- start 2024-06-29T14:38:25.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.339+0000 7f13ab281640 1 -- start start 2024-06-29T14:38:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.340+0000 7f13ab281640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13a40b5250 0x7f13a414eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.340+0000 7f13ab281640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13a40b7290 0x7f13a4151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.340+0000 7f13ab281640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a40b9040 0x7f13a40a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.340+0000 7f13ab281640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f13a40d2e20 con 0x7f13a40b9040 2024-06-29T14:38:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.340+0000 7f13ab281640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f13a40d2ca0 con 0x7f13a40b7290 2024-06-29T14:38:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.340+0000 7f13ab281640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f13a40b7db0 con 0x7f13a40b5250 2024-06-29T14:38:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.340+0000 7f13a981e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a40b9040 0x7f13a40a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.340+0000 7f13a981e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a40b9040 0x7f13a40a0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38678/0 (socket says 172.21.15.81:38678) 2024-06-29T14:38:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.340+0000 7f13a981e640 1 -- 172.21.15.81:0/44363454 learned_addr learned my addr 172.21.15.81:0/44363454 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:25.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13a901d640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13a40b5250 0x7f13a414eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13a981e640 1 -- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13a40b5250 msgr2=0x7f13a414eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13a981e640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13a40b5250 0x7f13a414eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13a881c640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13a40b7290 0x7f13a4151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13a981e640 1 -- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13a40b7290 msgr2=0x7f13a4151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13a981e640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13a40b7290 0x7f13a4151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13a981e640 1 -- 172.21.15.81:0/44363454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a414f3f0 con 0x7f13a40b9040 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13a881c640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13a40b7290 0x7f13a4151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13a981e640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a40b9040 0x7f13a40a0a00 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f139c00c3c0 tx=0x7f139c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f1399ffb640 1 -- 172.21.15.81:0/44363454 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c017070 con 0x7f13a40b9040 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f1399ffb640 1 -- 172.21.15.81:0/44363454 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f139c0035f0 con 0x7f13a40b9040 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13ab281640 1 -- 172.21.15.81:0/44363454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a414f6d0 con 0x7f13a40b9040 2024-06-29T14:38:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f1399ffb640 1 -- 172.21.15.81:0/44363454 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c003a40 con 0x7f13a40b9040 2024-06-29T14:38:25.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.341+0000 7f13ab281640 1 -- 172.21.15.81:0/44363454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a414fc90 con 0x7f13a40b9040 2024-06-29T14:38:25.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.342+0000 7f13ab281640 1 -- 172.21.15.81:0/44363454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a40b82f0 con 0x7f13a40b9040 2024-06-29T14:38:25.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.343+0000 7f1399ffb640 1 -- 172.21.15.81:0/44363454 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f139c01a070 con 0x7f13a40b9040 2024-06-29T14:38:25.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.343+0000 7f1399ffb640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1388076420 0x7f1388078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.343+0000 7f1399ffb640 1 -- 172.21.15.81:0/44363454 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5781+0+0 (secure 0 0 0) 0x7f139c096bc0 con 0x7f13a40b9040 2024-06-29T14:38:25.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.344+0000 7f13a901d640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1388076420 0x7f1388078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.344+0000 7f13a901d640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1388076420 0x7f1388078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f13a0034000 tx=0x7f13a0004ab0 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:25.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.348+0000 7f1399ffb640 1 -- 172.21.15.81:0/44363454 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f139c060330 con 0x7f13a40b9040 2024-06-29T14:38:25.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.530+0000 7f13ab281640 1 -- 172.21.15.81:0/44363454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13a41521c0 con 0x7f13a40b9040 2024-06-29T14:38:25.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.531+0000 7f1399ffb640 1 -- 172.21.15.81:0/44363454 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+16786 (secure 0 0 0) 0x7f139c0683b0 con 0x7f13a40b9040 2024-06-29T14:38:25.529 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:25.529 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":103,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:38:24.216432+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.514776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.175620+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:25.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 -- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1388076420 msgr2=0x7f1388078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1388076420 0x7f1388078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f13a0034000 tx=0x7f13a0004ab0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 -- 172.21.15.81:0/44363454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a40b9040 msgr2=0x7f13a40a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a40b9040 0x7f13a40a0a00 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f139c00c3c0 tx=0x7f139c00c880 comp rx=0 tx=0).stop 2024-06-29T14:38:25.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 -- 172.21.15.81:0/44363454 shutdown_connections 2024-06-29T14:38:25.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1388076420 0x7f1388078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a40b9040 0x7f13a40a0a00 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13a40b7290 0x7f13a4151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 --2- 172.21.15.81:0/44363454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13a40b5250 0x7f13a414eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 -- 172.21.15.81:0/44363454 >> 172.21.15.81:0/44363454 conn(0x7f13a409cf30 msgr2=0x7f13a40682a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:25.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 -- 172.21.15.81:0/44363454 shutdown_connections 2024-06-29T14:38:25.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.533+0000 7f13ab281640 1 -- 172.21.15.81:0/44363454 wait complete. 2024-06-29T14:38:25.543 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:38:25.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.661+0000 7f10efe3d640 1 -- 172.21.15.81:0/566911436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e80d2a90 msgr2=0x7f10e80d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.661+0000 7f10efe3d640 1 --2- 172.21.15.81:0/566911436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e80d2a90 0x7f10e80d09c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f10e808e230 tx=0x7f10e002ab40 comp rx=0 tx=0).stop 2024-06-29T14:38:25.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.661+0000 7f10efe3d640 1 -- 172.21.15.81:0/566911436 shutdown_connections 2024-06-29T14:38:25.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.661+0000 7f10efe3d640 1 --2- 172.21.15.81:0/566911436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e80d2a90 0x7f10e80d09c0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.661+0000 7f10efe3d640 1 --2- 172.21.15.81:0/566911436 >> v2:172.21.15.167:3300/0 conn(0x7f10e80b9040 0x7f10e80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.661+0000 7f10efe3d640 1 -- 172.21.15.81:0/566911436 >> 172.21.15.81:0/566911436 conn(0x7f10e809cf30 msgr2=0x7f10e80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:25.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.661+0000 7f10efe3d640 1 -- 172.21.15.81:0/566911436 shutdown_connections 2024-06-29T14:38:25.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.661+0000 7f10efe3d640 1 -- 172.21.15.81:0/566911436 wait complete. 2024-06-29T14:38:25.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.663+0000 7f10efe3d640 1 Processor -- start 2024-06-29T14:38:25.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.663+0000 7f10efe3d640 1 -- start start 2024-06-29T14:38:25.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.664+0000 7f10efe3d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e80b5250 0x7f10e80c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.664+0000 7f10efe3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e80b9040 0x7f10e814eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10efe3d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e80d2a90 0x7f10e80a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10efe3d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f10e80b7d30 con 0x7f10e80b9040 2024-06-29T14:38:25.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10efe3d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f10e80b7bb0 con 0x7f10e80b5250 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10efe3d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f10e80b7eb0 con 0x7f10e80d2a90 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10ed3d8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e80b9040 0x7f10e814eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10ed3d8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e80b9040 0x7f10e814eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38700/0 (socket says 172.21.15.81:38700) 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10ed3d8640 1 -- 172.21.15.81:0/1500609475 learned_addr learned my addr 172.21.15.81:0/1500609475 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10edbd9640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e80b5250 0x7f10e80c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10ee3da640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e80d2a90 0x7f10e80a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10ed3d8640 1 -- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e80d2a90 msgr2=0x7f10e80a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10ed3d8640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e80d2a90 0x7f10e80a0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10ed3d8640 1 -- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e80b5250 msgr2=0x7f10e80c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10ed3d8640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e80b5250 0x7f10e80c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10ed3d8640 1 -- 172.21.15.81:0/1500609475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e81518d0 con 0x7f10e80b9040 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10edbd9640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e80b5250 0x7f10e80c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10ed3d8640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e80b9040 0x7f10e814eca0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f10d800c430 tx=0x7f10d800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:25.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10de7fc640 1 -- 172.21.15.81:0/1500609475 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d8017070 con 0x7f10e80b9040 2024-06-29T14:38:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10de7fc640 1 -- 172.21.15.81:0/1500609475 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10d80053c0 con 0x7f10e80b9040 2024-06-29T14:38:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10de7fc640 1 -- 172.21.15.81:0/1500609475 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d8003850 con 0x7f10e80b9040 2024-06-29T14:38:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10efe3d640 1 -- 172.21.15.81:0/1500609475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e8151bb0 con 0x7f10e80b9040 2024-06-29T14:38:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.665+0000 7f10efe3d640 1 -- 172.21.15.81:0/1500609475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e8152170 con 0x7f10e80b9040 2024-06-29T14:38:25.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.666+0000 7f10efe3d640 1 -- 172.21.15.81:0/1500609475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e80b8440 con 0x7f10e80b9040 2024-06-29T14:38:25.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.667+0000 7f10de7fc640 1 -- 172.21.15.81:0/1500609475 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f10d801a070 con 0x7f10e80b9040 2024-06-29T14:38:25.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.667+0000 7f10de7fc640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10cc076420 0x7f10cc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:25.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.668+0000 7f10de7fc640 1 -- 172.21.15.81:0/1500609475 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 5781+0+0 (secure 0 0 0) 0x7f10d8096da0 con 0x7f10e80b9040 2024-06-29T14:38:25.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.668+0000 7f10edbd9640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10cc076420 0x7f10cc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:25.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.668+0000 7f10edbd9640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10cc076420 0x7f10cc078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f10e4009aa0 tx=0x7f10e4005db0 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:25.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.672+0000 7f10de7fc640 1 -- 172.21.15.81:0/1500609475 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f10d8060520 con 0x7f10e80b9040 2024-06-29T14:38:25.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:25.896+0000 7f10efe3d640 1 -- 172.21.15.81:0/1500609475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f10e815e3a0 con 0x7f10e80b9040 2024-06-29T14:38:26.836 INFO:tasks.ceph.mds.b.smithi167.stderr:2024-06-29T14:38:26.839+0000 7fcc620e0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:38:26.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.883+0000 7f10de7fc640 1 -- 172.21.15.81:0/1500609475 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) ==== 118+0+0 (secure 0 0 0) 0x7f10d805fe20 con 0x7f10e80b9040 2024-06-29T14:38:26.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.885+0000 7f10efe3d640 1 -- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10cc076420 msgr2=0x7f10cc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:26.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.885+0000 7f10efe3d640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10cc076420 0x7f10cc078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f10e4009aa0 tx=0x7f10e4005db0 comp rx=0 tx=0).stop 2024-06-29T14:38:26.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.885+0000 7f10efe3d640 1 -- 172.21.15.81:0/1500609475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e80b9040 msgr2=0x7f10e814eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:26.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.885+0000 7f10efe3d640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e80b9040 0x7f10e814eca0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f10d800c430 tx=0x7f10d800c8f0 comp rx=0 tx=0).stop 2024-06-29T14:38:26.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.885+0000 7f10efe3d640 1 -- 172.21.15.81:0/1500609475 shutdown_connections 2024-06-29T14:38:26.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.885+0000 7f10efe3d640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f10cc076420 0x7f10cc078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:26.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.885+0000 7f10efe3d640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f10e80d2a90 0x7f10e80a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:26.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.885+0000 7f10efe3d640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10e80b9040 0x7f10e814eca0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:26.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.885+0000 7f10efe3d640 1 --2- 172.21.15.81:0/1500609475 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f10e80b5250 0x7f10e80c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:26.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.885+0000 7f10efe3d640 1 -- 172.21.15.81:0/1500609475 >> 172.21.15.81:0/1500609475 conn(0x7f10e809cf30 msgr2=0x7f10e806e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:26.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.886+0000 7f10efe3d640 1 -- 172.21.15.81:0/1500609475 shutdown_connections 2024-06-29T14:38:26.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:26.886+0000 7f10efe3d640 1 -- 172.21.15.81:0/1500609475 wait complete. 2024-06-29T14:38:26.896 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:38:26.897 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:38:27.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.022+0000 7fd9fa49e640 1 -- 172.21.15.81:0/1076847040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9f415bf10 msgr2=0x7fd9f40732a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:27.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.022+0000 7fd9fa49e640 1 --2- 172.21.15.81:0/1076847040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9f415bf10 0x7fd9f40732a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc005fd0 tx=0x7fd9dc02a530 comp rx=0 tx=0).stop 2024-06-29T14:38:27.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.022+0000 7fd9fa49e640 1 -- 172.21.15.81:0/1076847040 shutdown_connections 2024-06-29T14:38:27.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.022+0000 7fd9fa49e640 1 --2- 172.21.15.81:0/1076847040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9f415bf10 0x7fd9f40732a0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.022+0000 7fd9fa49e640 1 --2- 172.21.15.81:0/1076847040 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9f415e890 0x7fd9f415b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.022+0000 7fd9fa49e640 1 -- 172.21.15.81:0/1076847040 >> 172.21.15.81:0/1076847040 conn(0x7fd9f40b1ce0 msgr2=0x7fd9f40b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:27.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.022+0000 7fd9fa49e640 1 -- 172.21.15.81:0/1076847040 shutdown_connections 2024-06-29T14:38:27.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.022+0000 7fd9fa49e640 1 -- 172.21.15.81:0/1076847040 wait complete. 2024-06-29T14:38:27.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.024+0000 7fd9fa49e640 1 Processor -- start 2024-06-29T14:38:27.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.024+0000 7fd9fa49e640 1 -- start start 2024-06-29T14:38:27.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9fa49e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9f40737e0 0x7fd9f413d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:27.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9fa49e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9f415bf10 0x7fd9f41567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:27.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9fa49e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9f415e890 0x7fd9f4156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:27.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9fa49e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f41412c0 con 0x7fd9f415bf10 2024-06-29T14:38:27.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9fa49e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f4141140 con 0x7fd9f415e890 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9fa49e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f40b2f00 con 0x7fd9f40737e0 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9ebfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9f415bf10 0x7fd9f41567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9ebfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9f415bf10 0x7fd9f41567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38716/0 (socket says 172.21.15.81:38716) 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9ebfff640 1 -- 172.21.15.81:0/3990803110 learned_addr learned my addr 172.21.15.81:0/3990803110 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9f3fff640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9f40737e0 0x7fd9f413d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9f8a3b640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9f415e890 0x7fd9f4156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9ebfff640 1 -- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9f40737e0 msgr2=0x7fd9f413d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9ebfff640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9f40737e0 0x7fd9f413d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9ebfff640 1 -- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9f415e890 msgr2=0x7fd9f4156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.025+0000 7fd9ebfff640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9f415e890 0x7fd9f4156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.026+0000 7fd9ebfff640 1 -- 172.21.15.81:0/3990803110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f41415e0 con 0x7fd9f415bf10 2024-06-29T14:38:27.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.026+0000 7fd9f8a3b640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9f415e890 0x7fd9f4156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.026+0000 7fd9ebfff640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9f415bf10 0x7fd9f41567e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc006010 tx=0x7fd9dc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.026+0000 7fd9f17fa640 1 -- 172.21.15.81:0/3990803110 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc03e030 con 0x7fd9f415bf10 2024-06-29T14:38:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.026+0000 7fd9f17fa640 1 -- 172.21.15.81:0/3990803110 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9dc003d90 con 0x7fd9f415bf10 2024-06-29T14:38:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.026+0000 7fd9fa49e640 1 -- 172.21.15.81:0/3990803110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f41418c0 con 0x7fd9f415bf10 2024-06-29T14:38:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.026+0000 7fd9f17fa640 1 -- 172.21.15.81:0/3990803110 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc035040 con 0x7fd9f415bf10 2024-06-29T14:38:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.026+0000 7fd9fa49e640 1 -- 172.21.15.81:0/3990803110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f4141e80 con 0x7fd9f415bf10 2024-06-29T14:38:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.027+0000 7fd9fa49e640 1 -- 172.21.15.81:0/3990803110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f413dc50 con 0x7fd9f415bf10 2024-06-29T14:38:27.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.028+0000 7fd9f17fa640 1 -- 172.21.15.81:0/3990803110 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7fd9dc041030 con 0x7fd9f415bf10 2024-06-29T14:38:27.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.028+0000 7fd9f17fa640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd9d4076420 0x7fd9d4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:27.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.028+0000 7fd9f17fa640 1 -- 172.21.15.81:0/3990803110 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7fd9dc0b80b0 con 0x7fd9f415bf10 2024-06-29T14:38:27.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.028+0000 7fd9f3fff640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd9d4076420 0x7fd9d4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:27.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.029+0000 7fd9f3fff640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd9d4076420 0x7fd9d4078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4004630 tx=0x7fd9e400b2e0 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:27.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.032+0000 7fd9f17fa640 1 -- 172.21.15.81:0/3990803110 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd9f413dc50 con 0x7fd9f415bf10 2024-06-29T14:38:27.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.251+0000 7fd9fa49e640 1 -- 172.21.15.81:0/3990803110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd9f40732a0 con 0x7fd9f415bf10 2024-06-29T14:38:27.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.884+0000 7fd9f17fa640 1 -- 172.21.15.81:0/3990803110 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v51) ==== 122+0+0 (secure 0 0 0) 0x7fd9f4073bf0 con 0x7fd9f415bf10 2024-06-29T14:38:27.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.887+0000 7fd9fa49e640 1 -- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd9d4076420 msgr2=0x7fd9d4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:27.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.887+0000 7fd9fa49e640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd9d4076420 0x7fd9d4078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4004630 tx=0x7fd9e400b2e0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.887+0000 7fd9fa49e640 1 -- 172.21.15.81:0/3990803110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9f415bf10 msgr2=0x7fd9f41567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:27.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.887+0000 7fd9fa49e640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9f415bf10 0x7fd9f41567e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc006010 tx=0x7fd9dc0038a0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.887+0000 7fd9fa49e640 1 -- 172.21.15.81:0/3990803110 shutdown_connections 2024-06-29T14:38:27.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.887+0000 7fd9fa49e640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fd9d4076420 0x7fd9d4078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.887+0000 7fd9fa49e640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9f415e890 0x7fd9f4156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.887+0000 7fd9fa49e640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9f415bf10 0x7fd9f41567e0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.887+0000 7fd9fa49e640 1 --2- 172.21.15.81:0/3990803110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9f40737e0 0x7fd9f413d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:27.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.887+0000 7fd9fa49e640 1 -- 172.21.15.81:0/3990803110 >> 172.21.15.81:0/3990803110 conn(0x7fd9f40b1ce0 msgr2=0x7fd9f4165cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:27.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.888+0000 7fd9fa49e640 1 -- 172.21.15.81:0/3990803110 shutdown_connections 2024-06-29T14:38:27.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:27.888+0000 7fd9fa49e640 1 -- 172.21.15.81:0/3990803110 wait complete. 2024-06-29T14:38:27.898 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:38:27.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:28.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.014+0000 7f13cf394640 1 -- 172.21.15.81:0/1923582208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13b4003a00 msgr2=0x7f13b4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.014+0000 7f13cf394640 1 --2- 172.21.15.81:0/1923582208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13b4003a00 0x7f13b4003e70 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009b50 tx=0x7f13b4005830 comp rx=0 tx=0).stop 2024-06-29T14:38:28.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.014+0000 7f13cd931640 1 -- 172.21.15.81:0/1923582208 reap_dead start 2024-06-29T14:38:28.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.014+0000 7f13cf394640 1 -- 172.21.15.81:0/1923582208 shutdown_connections 2024-06-29T14:38:28.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.014+0000 7f13cf394640 1 -- 172.21.15.81:0/1923582208 >> 172.21.15.81:0/1923582208 conn(0x7f13c806ccc0 msgr2=0x7f13c80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.015+0000 7f13cf394640 1 -- 172.21.15.81:0/1923582208 shutdown_connections 2024-06-29T14:38:28.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.015+0000 7f13cf394640 1 -- 172.21.15.81:0/1923582208 wait complete. 2024-06-29T14:38:28.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.016+0000 7f13cf394640 1 Processor -- start 2024-06-29T14:38:28.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.016+0000 7f13cf394640 1 -- start start 2024-06-29T14:38:28.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cf394640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13b4003a00 0x7f13c80cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cf394640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13c81330c0 0x7f13c81334d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cf394640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13c815d360 0x7f13c815d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cf394640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f13c815d0c0 con 0x7f13b4003a00 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cf394640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f13c815cf40 con 0x7f13c815d360 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cf394640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f13c8073280 con 0x7f13c81330c0 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cd130640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13b4003a00 0x7f13c80cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cd130640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13b4003a00 0x7f13c80cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38736/0 (socket says 172.21.15.81:38736) 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cd130640 1 -- 172.21.15.81:0/1670006678 learned_addr learned my addr 172.21.15.81:0/1670006678 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cc92f640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13c81330c0 0x7f13c81334d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cd931640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13c815d360 0x7f13c815d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cd130640 1 -- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13c81330c0 msgr2=0x7f13c81334d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cd130640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13c81330c0 0x7f13c81334d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cd130640 1 -- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13c815d360 msgr2=0x7f13c815d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cd130640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13c815d360 0x7f13c815d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cd130640 1 -- 172.21.15.81:0/1670006678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c815dcb0 con 0x7f13b4003a00 2024-06-29T14:38:28.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cd130640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13b4003a00 0x7f13c80cd880 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009a70 tx=0x7f13c4033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:28.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13bdffb640 1 -- 172.21.15.81:0/1670006678 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4038040 con 0x7f13b4003a00 2024-06-29T14:38:28.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13bdffb640 1 -- 172.21.15.81:0/1670006678 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13c4004020 con 0x7f13b4003a00 2024-06-29T14:38:28.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13cf394640 1 -- 172.21.15.81:0/1670006678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c80cddc0 con 0x7f13b4003a00 2024-06-29T14:38:28.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.018+0000 7f13bdffb640 1 -- 172.21.15.81:0/1670006678 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4004470 con 0x7f13b4003a00 2024-06-29T14:38:28.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.019+0000 7f13cf394640 1 -- 172.21.15.81:0/1670006678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c81311b0 con 0x7f13b4003a00 2024-06-29T14:38:28.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.020+0000 7f13cf394640 1 -- 172.21.15.81:0/1670006678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c80af7d0 con 0x7f13b4003a00 2024-06-29T14:38:28.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.020+0000 7f13bdffb640 1 -- 172.21.15.81:0/1670006678 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f13c4045020 con 0x7f13b4003a00 2024-06-29T14:38:28.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.021+0000 7f13bdffb640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f13ac0763f0 0x7f13ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.021+0000 7f13bdffb640 1 -- 172.21.15.81:0/1670006678 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f13c40b8570 con 0x7f13b4003a00 2024-06-29T14:38:28.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.021+0000 7f13cc92f640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f13ac0763f0 0x7f13ac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.022+0000 7f13cc92f640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f13ac0763f0 0x7f13ac078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f13c80af610 tx=0x7f13b8005f90 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:28.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.025+0000 7f13bdffb640 1 -- 172.21.15.81:0/1670006678 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f13c815dcb0 con 0x7f13b4003a00 2024-06-29T14:38:28.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.244+0000 7f13cf394640 1 -- 172.21.15.81:0/1670006678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13c815e030 con 0x7f13b4003a00 2024-06-29T14:38:28.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.244+0000 7f13bdffb640 1 -- 172.21.15.81:0/1670006678 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5231 (secure 0 0 0) 0x7f13c4081c60 con 0x7f13b4003a00 2024-06-29T14:38:28.242 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:28.242 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":51,"btime":"2024-06-29T14:38:27:883643+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1895620554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1895620554},{"type":"v1","addr":"172.21.15.167:6836","nonce":1895620554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":7485,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3391563230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3391563230},{"type":"v1","addr":"172.21.15.81:6834","nonce":3391563230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":7488,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3932182351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3932182351},{"type":"v1","addr":"172.21.15.81:6835","nonce":3932182351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.216390+0000","modified":"2024-06-29T14:38:27.252097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/3193613343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":3193613343},{"type":"v1","addr":"172.21.15.167:6837","nonce":3193613343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":6}]} 2024-06-29T14:38:28.242 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 51 2024-06-29T14:38:28.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 -- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f13ac0763f0 msgr2=0x7f13ac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f13ac0763f0 0x7f13ac078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f13c80af610 tx=0x7f13b8005f90 comp rx=0 tx=0).stop 2024-06-29T14:38:28.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 -- 172.21.15.81:0/1670006678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13b4003a00 msgr2=0x7f13c80cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13b4003a00 0x7f13c80cd880 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009a70 tx=0x7f13c4033670 comp rx=0 tx=0).stop 2024-06-29T14:38:28.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 -- 172.21.15.81:0/1670006678 shutdown_connections 2024-06-29T14:38:28.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f13ac0763f0 0x7f13ac078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13c815d360 0x7f13c815d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13c81330c0 0x7f13c81334d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 --2- 172.21.15.81:0/1670006678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13b4003a00 0x7f13c80cd880 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 -- 172.21.15.81:0/1670006678 >> 172.21.15.81:0/1670006678 conn(0x7f13c806ccc0 msgr2=0x7f13c81410e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 -- 172.21.15.81:0/1670006678 shutdown_connections 2024-06-29T14:38:28.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.246+0000 7f13cf394640 1 -- 172.21.15.81:0/1670006678 wait complete. 2024-06-29T14:38:28.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:38:28.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.370+0000 7f2805065640 1 -- 172.21.15.81:0/1294114442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27ec004d40 msgr2=0x7f27ec005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.370+0000 7f2805065640 1 --2- 172.21.15.81:0/1294114442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27ec004d40 0x7f27ec005150 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009ed0 tx=0x7f27ec005830 comp rx=0 tx=0).stop 2024-06-29T14:38:28.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.370+0000 7f27ff59e640 1 -- 172.21.15.81:0/1294114442 reap_dead start 2024-06-29T14:38:28.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.371+0000 7f2805065640 1 -- 172.21.15.81:0/1294114442 shutdown_connections 2024-06-29T14:38:28.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.371+0000 7f2805065640 1 -- 172.21.15.81:0/1294114442 >> 172.21.15.81:0/1294114442 conn(0x7f28000d3c60 msgr2=0x7f28000d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.371+0000 7f2805065640 1 -- 172.21.15.81:0/1294114442 shutdown_connections 2024-06-29T14:38:28.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.371+0000 7f2805065640 1 -- 172.21.15.81:0/1294114442 wait complete. 2024-06-29T14:38:28.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.372+0000 7f2805065640 1 Processor -- start 2024-06-29T14:38:28.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.373+0000 7f2805065640 1 -- start start 2024-06-29T14:38:28.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f2805065640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27ec004d40 0x7f28001dc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f2805065640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28001dd110 0x7f28001e8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f2805065640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28001e8da0 0x7f28001e13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f2805065640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28000d15a0 con 0x7f27ec004d40 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f2805065640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f28000d1420 con 0x7f28001e8da0 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f2805065640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f28000d4990 con 0x7f28001dd110 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f27fed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27ec004d40 0x7f28001dc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f27fed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27ec004d40 0x7f28001dc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38744/0 (socket says 172.21.15.81:38744) 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f27fed9d640 1 -- 172.21.15.81:0/2323118725 learned_addr learned my addr 172.21.15.81:0/2323118725 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f27ff59e640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28001e8da0 0x7f28001e13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f27f659c640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28001dd110 0x7f28001e8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f27fed9d640 1 -- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28001dd110 msgr2=0x7f28001e8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f27fed9d640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28001dd110 0x7f28001e8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f27fed9d640 1 -- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28001e8da0 msgr2=0x7f28001e13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f27fed9d640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28001e8da0 0x7f28001e13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.374+0000 7f27fed9d640 1 -- 172.21.15.81:0/2323118725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28001e5090 con 0x7f27ec004d40 2024-06-29T14:38:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.375+0000 7f27fed9d640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27ec004d40 0x7f28001dc7c0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f27f0006640 tx=0x7f27f0006730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.375+0000 7f27f7fff640 1 -- 172.21.15.81:0/2323118725 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f0015070 con 0x7f27ec004d40 2024-06-29T14:38:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.375+0000 7f27f7fff640 1 -- 172.21.15.81:0/2323118725 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27f0011920 con 0x7f27ec004d40 2024-06-29T14:38:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.375+0000 7f2805065640 1 -- 172.21.15.81:0/2323118725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28001e5370 con 0x7f27ec004d40 2024-06-29T14:38:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.375+0000 7f27f7fff640 1 -- 172.21.15.81:0/2323118725 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f0011d70 con 0x7f27ec004d40 2024-06-29T14:38:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.375+0000 7f2805065640 1 -- 172.21.15.81:0/2323118725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28001e5930 con 0x7f27ec004d40 2024-06-29T14:38:28.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.376+0000 7f2805065640 1 -- 172.21.15.81:0/2323118725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28000d2240 con 0x7f27ec004d40 2024-06-29T14:38:28.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.376+0000 7f27f7fff640 1 -- 172.21.15.81:0/2323118725 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f27f0010420 con 0x7f27ec004d40 2024-06-29T14:38:28.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.377+0000 7f27f7fff640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f27e0076420 0x7f27e0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.377+0000 7f27f7fff640 1 -- 172.21.15.81:0/2323118725 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f27f0098740 con 0x7f27ec004d40 2024-06-29T14:38:28.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.377+0000 7f27f659c640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f27e0076420 0x7f27e0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.378+0000 7f27f659c640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f27e0076420 0x7f27e0078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28000b2230 tx=0x7f27e8005f90 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:28.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.381+0000 7f27f7fff640 1 -- 172.21.15.81:0/2323118725 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f27f0061750 con 0x7f27ec004d40 2024-06-29T14:38:28.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.561+0000 7f2805065640 1 -- 172.21.15.81:0/2323118725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28001e7100 con 0x7f27ec004d40 2024-06-29T14:38:28.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.562+0000 7f27f7fff640 1 -- 172.21.15.81:0/2323118725 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+16773 (secure 0 0 0) 0x7f27f0065a50 con 0x7f27ec004d40 2024-06-29T14:38:28.562 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:28.562 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":104,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:38:25.861684+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.514776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.175620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:38:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.564+0000 7f2805065640 1 -- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f27e0076420 msgr2=0x7f27e0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.564+0000 7f2805065640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f27e0076420 0x7f27e0078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28000b2230 tx=0x7f27e8005f90 comp rx=0 tx=0).stop 2024-06-29T14:38:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.564+0000 7f2805065640 1 -- 172.21.15.81:0/2323118725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27ec004d40 msgr2=0x7f28001dc7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.564+0000 7f2805065640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27ec004d40 0x7f28001dc7c0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f27f0006640 tx=0x7f27f0006730 comp rx=0 tx=0).stop 2024-06-29T14:38:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.564+0000 7f2805065640 1 -- 172.21.15.81:0/2323118725 shutdown_connections 2024-06-29T14:38:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.564+0000 7f2805065640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f27e0076420 0x7f27e0078930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.564+0000 7f2805065640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28001e8da0 0x7f28001e13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.564+0000 7f2805065640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28001dd110 0x7f28001e8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.564+0000 7f2805065640 1 --2- 172.21.15.81:0/2323118725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27ec004d40 0x7f28001dc7c0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.564+0000 7f2805065640 1 -- 172.21.15.81:0/2323118725 >> 172.21.15.81:0/2323118725 conn(0x7f28000d3c60 msgr2=0x7f28001e0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.565+0000 7f2805065640 1 -- 172.21.15.81:0/2323118725 shutdown_connections 2024-06-29T14:38:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.565+0000 7f2805065640 1 -- 172.21.15.81:0/2323118725 wait complete. 2024-06-29T14:38:28.573 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:38:28.574 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:38:28.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.690+0000 7f9f52a38640 1 -- 172.21.15.81:0/3219647174 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f4c073740 msgr2=0x7f9f4c15b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.690+0000 7f9f52a38640 1 --2- 172.21.15.81:0/3219647174 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f4c073740 0x7f9f4c15b4d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009830 tx=0x7f9f4002a510 comp rx=0 tx=0).stop 2024-06-29T14:38:28.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.690+0000 7f9f52a38640 1 -- 172.21.15.81:0/3219647174 shutdown_connections 2024-06-29T14:38:28.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.690+0000 7f9f52a38640 1 --2- 172.21.15.81:0/3219647174 >> v2:172.21.15.167:3300/0 conn(0x7f9f4c15ba10 0x7f9f4c0ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.690+0000 7f9f52a38640 1 --2- 172.21.15.81:0/3219647174 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f4c073740 0x7f9f4c15b4d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.690+0000 7f9f52a38640 1 -- 172.21.15.81:0/3219647174 >> 172.21.15.81:0/3219647174 conn(0x7f9f4c06ccc0 msgr2=0x7f9f4c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.691+0000 7f9f52a38640 1 -- 172.21.15.81:0/3219647174 shutdown_connections 2024-06-29T14:38:28.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.691+0000 7f9f52a38640 1 -- 172.21.15.81:0/3219647174 wait complete. 2024-06-29T14:38:28.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.692+0000 7f9f52a38640 1 Processor -- start 2024-06-29T14:38:28.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.692+0000 7f9f52a38640 1 -- start start 2024-06-29T14:38:28.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f52a38640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f4c13dd70 0x7f9f4c0b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f52a38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f4c15ba10 0x7f9f4c133170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f52a38640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f4c0ae420 0x7f9f4c1f8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f52a38640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c0ceb40 con 0x7f9f4c15ba10 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f52a38640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c0ce9c0 con 0x7f9f4c13dd70 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f52a38640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9f4c0cecc0 con 0x7f9f4c0ae420 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f4b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f4c15ba10 0x7f9f4c133170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f4b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f4c15ba10 0x7f9f4c133170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38746/0 (socket says 172.21.15.81:38746) 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f4b7fe640 1 -- 172.21.15.81:0/471921621 learned_addr learned my addr 172.21.15.81:0/471921621 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f50fd5640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f4c0ae420 0x7f9f4c1f8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f4bfff640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f4c13dd70 0x7f9f4c0b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f4b7fe640 1 -- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f4c0ae420 msgr2=0x7f9f4c1f8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f4b7fe640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f4c0ae420 0x7f9f4c1f8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f4b7fe640 1 -- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f4c13dd70 msgr2=0x7f9f4c0b2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f4b7fe640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f4c13dd70 0x7f9f4c0b2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f4b7fe640 1 -- 172.21.15.81:0/471921621 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4c0724e0 con 0x7f9f4c15ba10 2024-06-29T14:38:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.694+0000 7f9f4b7fe640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f4c15ba10 0x7f9f4c133170 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009950 tx=0x7f9f40033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:28.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.695+0000 7f9f48ff9640 1 -- 172.21.15.81:0/471921621 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40031d60 con 0x7f9f4c15ba10 2024-06-29T14:38:28.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.695+0000 7f9f48ff9640 1 -- 172.21.15.81:0/471921621 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f40033e10 con 0x7f9f4c15ba10 2024-06-29T14:38:28.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.695+0000 7f9f48ff9640 1 -- 172.21.15.81:0/471921621 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40005260 con 0x7f9f4c15ba10 2024-06-29T14:38:28.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.695+0000 7f9f52a38640 1 -- 172.21.15.81:0/471921621 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f4c0727c0 con 0x7f9f4c15ba10 2024-06-29T14:38:28.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.695+0000 7f9f52a38640 1 -- 172.21.15.81:0/471921621 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f4c1f84e0 con 0x7f9f4c15ba10 2024-06-29T14:38:28.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.696+0000 7f9f52a38640 1 -- 172.21.15.81:0/471921621 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f4c0cf1e0 con 0x7f9f4c15ba10 2024-06-29T14:38:28.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.696+0000 7f9f48ff9640 1 -- 172.21.15.81:0/471921621 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f9f40005400 con 0x7f9f4c15ba10 2024-06-29T14:38:28.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.697+0000 7f9f48ff9640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9f38076420 0x7f9f38078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:28.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.697+0000 7f9f48ff9640 1 -- 172.21.15.81:0/471921621 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f9f400b8a30 con 0x7f9f4c15ba10 2024-06-29T14:38:28.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.697+0000 7f9f4bfff640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9f38076420 0x7f9f38078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:28.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.697+0000 7f9f4bfff640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9f38076420 0x7f9f38078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c0015e0 tx=0x7f9f3c009040 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.701+0000 7f9f48ff9640 1 -- 172.21.15.81:0/471921621 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9f4003f030 con 0x7f9f4c15ba10 2024-06-29T14:38:28.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.882+0000 7f9f52a38640 1 -- 172.21.15.81:0/471921621 --> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f4c06fa70 con 0x7f9f38076420 2024-06-29T14:38:28.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.887+0000 7f9f48ff9640 1 -- 172.21.15.81:0/471921621 <== mgr.6158 v2:172.21.15.167:6832/2583731609 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252485 (secure 0 0 0) 0x7f9f4c06fa70 con 0x7f9f38076420 2024-06-29T14:38:28.887 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:28.888 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:38:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 -- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9f38076420 msgr2=0x7f9f38078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9f38076420 0x7f9f38078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c0015e0 tx=0x7f9f3c009040 comp rx=0 tx=0).stop 2024-06-29T14:38:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 -- 172.21.15.81:0/471921621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f4c15ba10 msgr2=0x7f9f4c133170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f4c15ba10 0x7f9f4c133170 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009950 tx=0x7f9f40033c20 comp rx=0 tx=0).stop 2024-06-29T14:38:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 -- 172.21.15.81:0/471921621 shutdown_connections 2024-06-29T14:38:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9f38076420 0x7f9f38078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f4c0ae420 0x7f9f4c1f8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f4c15ba10 0x7f9f4c133170 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 --2- 172.21.15.81:0/471921621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f4c13dd70 0x7f9f4c0b2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:28.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 -- 172.21.15.81:0/471921621 >> 172.21.15.81:0/471921621 conn(0x7f9f4c06ccc0 msgr2=0x7f9f4c15a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:28.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 -- 172.21.15.81:0/471921621 shutdown_connections 2024-06-29T14:38:28.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:28.890+0000 7f9f52a38640 1 -- 172.21.15.81:0/471921621 wait complete. 2024-06-29T14:38:28.899 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:38:28.441243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":218,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":317524,"kb_used_data":11160,"kb_used_omap":12,"kb_used_meta":306227,"kb_avail":754657196,"statfs":{"total":773094113280,"available":772768968704,"internally_reserved":0,"allocated":11427840,"data_stored":6044852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":313577010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.599151"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868615+0000","last_change":"2024-06-29T14:38:24.895717+0000","last_active":"2024-06-29T14:38:25.868615+0000","last_peered":"2024-06-29T14:38:25.868615+0000","last_clean":"2024-06-29T14:38:25.868615+0000","last_became_active":"2024-06-29T14:38:24.895364+0000","last_became_peered":"2024-06-29T14:38:24.895364+0000","last_unstale":"2024-06-29T14:38:25.868615+0000","last_undegraded":"2024-06-29T14:38:25.868615+0000","last_fullsized":"2024-06-29T14:38:25.868615+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:30.259008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865436+0000","last_change":"2024-06-29T14:38:24.890785+0000","last_active":"2024-06-29T14:38:25.865436+0000","last_peered":"2024-06-29T14:38:25.865436+0000","last_clean":"2024-06-29T14:38:25.865436+0000","last_became_active":"2024-06-29T14:38:24.890352+0000","last_became_peered":"2024-06-29T14:38:24.890352+0000","last_unstale":"2024-06-29T14:38:25.865436+0000","last_undegraded":"2024-06-29T14:38:25.865436+0000","last_fullsized":"2024-06-29T14:38:25.865436+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:59.321242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045559+0000","last_change":"2024-06-29T14:38:24.886809+0000","last_active":"2024-06-29T14:38:26.045559+0000","last_peered":"2024-06-29T14:38:26.045559+0000","last_clean":"2024-06-29T14:38:26.045559+0000","last_became_active":"2024-06-29T14:38:24.886402+0000","last_became_peered":"2024-06-29T14:38:24.886402+0000","last_unstale":"2024-06-29T14:38:26.045559+0000","last_undegraded":"2024-06-29T14:38:26.045559+0000","last_fullsized":"2024-06-29T14:38:26.045559+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:55.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045231+0000","last_change":"2024-06-29T14:38:24.882495+0000","last_active":"2024-06-29T14:38:26.045231+0000","last_peered":"2024-06-29T14:38:26.045231+0000","last_clean":"2024-06-29T14:38:26.045231+0000","last_became_active":"2024-06-29T14:38:24.881796+0000","last_became_peered":"2024-06-29T14:38:24.881796+0000","last_unstale":"2024-06-29T14:38:26.045231+0000","last_undegraded":"2024-06-29T14:38:26.045231+0000","last_fullsized":"2024-06-29T14:38:26.045231+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:56.351425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045310+0000","last_change":"2024-06-29T14:38:24.875308+0000","last_active":"2024-06-29T14:38:26.045310+0000","last_peered":"2024-06-29T14:38:26.045310+0000","last_clean":"2024-06-29T14:38:26.045310+0000","last_became_active":"2024-06-29T14:38:24.873008+0000","last_became_peered":"2024-06-29T14:38:24.873008+0000","last_unstale":"2024-06-29T14:38:26.045310+0000","last_undegraded":"2024-06-29T14:38:26.045310+0000","last_fullsized":"2024-06-29T14:38:26.045310+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:40:43.916544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867170+0000","last_change":"2024-06-29T14:38:24.881179+0000","last_active":"2024-06-29T14:38:25.867170+0000","last_peered":"2024-06-29T14:38:25.867170+0000","last_clean":"2024-06-29T14:38:25.867170+0000","last_became_active":"2024-06-29T14:38:24.880860+0000","last_became_peered":"2024-06-29T14:38:24.880860+0000","last_unstale":"2024-06-29T14:38:25.867170+0000","last_undegraded":"2024-06-29T14:38:25.867170+0000","last_fullsized":"2024-06-29T14:38:25.867170+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:58.034581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044703+0000","last_change":"2024-06-29T14:38:24.886592+0000","last_active":"2024-06-29T14:38:26.044703+0000","last_peered":"2024-06-29T14:38:26.044703+0000","last_clean":"2024-06-29T14:38:26.044703+0000","last_became_active":"2024-06-29T14:38:24.886266+0000","last_became_peered":"2024-06-29T14:38:24.886266+0000","last_unstale":"2024-06-29T14:38:26.044703+0000","last_undegraded":"2024-06-29T14:38:26.044703+0000","last_fullsized":"2024-06-29T14:38:26.044703+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:31.996153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867000+0000","last_change":"2024-06-29T14:38:24.874632+0000","last_active":"2024-06-29T14:38:25.867000+0000","last_peered":"2024-06-29T14:38:25.867000+0000","last_clean":"2024-06-29T14:38:25.867000+0000","last_became_active":"2024-06-29T14:38:24.874334+0000","last_became_peered":"2024-06-29T14:38:24.874334+0000","last_unstale":"2024-06-29T14:38:25.867000+0000","last_undegraded":"2024-06-29T14:38:25.867000+0000","last_fullsized":"2024-06-29T14:38:25.867000+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:00.507144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867450+0000","last_change":"2024-06-29T14:38:24.878758+0000","last_active":"2024-06-29T14:38:25.867450+0000","last_peered":"2024-06-29T14:38:25.867450+0000","last_clean":"2024-06-29T14:38:25.867450+0000","last_became_active":"2024-06-29T14:38:24.878456+0000","last_became_peered":"2024-06-29T14:38:24.878456+0000","last_unstale":"2024-06-29T14:38:25.867450+0000","last_undegraded":"2024-06-29T14:38:25.867450+0000","last_fullsized":"2024-06-29T14:38:25.867450+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:54:05.449634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868435+0000","last_change":"2024-06-29T14:38:24.887705+0000","last_active":"2024-06-29T14:38:25.868435+0000","last_peered":"2024-06-29T14:38:25.868435+0000","last_clean":"2024-06-29T14:38:25.868435+0000","last_became_active":"2024-06-29T14:38:24.887398+0000","last_became_peered":"2024-06-29T14:38:24.887398+0000","last_unstale":"2024-06-29T14:38:25.868435+0000","last_undegraded":"2024-06-29T14:38:25.868435+0000","last_fullsized":"2024-06-29T14:38:25.868435+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:20:53.669315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865218+0000","last_change":"2024-06-29T14:38:24.875948+0000","last_active":"2024-06-29T14:38:25.865218+0000","last_peered":"2024-06-29T14:38:25.865218+0000","last_clean":"2024-06-29T14:38:25.865218+0000","last_became_active":"2024-06-29T14:38:24.875639+0000","last_became_peered":"2024-06-29T14:38:24.875639+0000","last_unstale":"2024-06-29T14:38:25.865218+0000","last_undegraded":"2024-06-29T14:38:25.865218+0000","last_fullsized":"2024-06-29T14:38:25.865218+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:53.863238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868589+0000","last_change":"2024-06-29T14:38:24.889183+0000","last_active":"2024-06-29T14:38:25.868589+0000","last_peered":"2024-06-29T14:38:25.868589+0000","last_clean":"2024-06-29T14:38:25.868589+0000","last_became_active":"2024-06-29T14:38:24.888843+0000","last_became_peered":"2024-06-29T14:38:24.888843+0000","last_unstale":"2024-06-29T14:38:25.868589+0000","last_undegraded":"2024-06-29T14:38:25.868589+0000","last_fullsized":"2024-06-29T14:38:25.868589+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:09:27.946436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046149+0000","last_change":"2024-06-29T14:38:24.882021+0000","last_active":"2024-06-29T14:38:26.046149+0000","last_peered":"2024-06-29T14:38:26.046149+0000","last_clean":"2024-06-29T14:38:26.046149+0000","last_became_active":"2024-06-29T14:38:24.881682+0000","last_became_peered":"2024-06-29T14:38:24.881682+0000","last_unstale":"2024-06-29T14:38:26.046149+0000","last_undegraded":"2024-06-29T14:38:26.046149+0000","last_fullsized":"2024-06-29T14:38:26.046149+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:40:06.228389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044930+0000","last_change":"2024-06-29T14:38:24.883867+0000","last_active":"2024-06-29T14:38:26.044930+0000","last_peered":"2024-06-29T14:38:26.044930+0000","last_clean":"2024-06-29T14:38:26.044930+0000","last_became_active":"2024-06-29T14:38:24.883097+0000","last_became_peered":"2024-06-29T14:38:24.883097+0000","last_unstale":"2024-06-29T14:38:26.044930+0000","last_undegraded":"2024-06-29T14:38:26.044930+0000","last_fullsized":"2024-06-29T14:38:26.044930+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:53.319167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044828+0000","last_change":"2024-06-29T14:38:24.882354+0000","last_active":"2024-06-29T14:38:26.044828+0000","last_peered":"2024-06-29T14:38:26.044828+0000","last_clean":"2024-06-29T14:38:26.044828+0000","last_became_active":"2024-06-29T14:38:24.882071+0000","last_became_peered":"2024-06-29T14:38:24.882071+0000","last_unstale":"2024-06-29T14:38:26.044828+0000","last_undegraded":"2024-06-29T14:38:26.044828+0000","last_fullsized":"2024-06-29T14:38:26.044828+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:38:49.127792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045143+0000","last_change":"2024-06-29T14:38:24.889841+0000","last_active":"2024-06-29T14:38:26.045143+0000","last_peered":"2024-06-29T14:38:26.045143+0000","last_clean":"2024-06-29T14:38:26.045143+0000","last_became_active":"2024-06-29T14:38:24.889397+0000","last_became_peered":"2024-06-29T14:38:24.889397+0000","last_unstale":"2024-06-29T14:38:26.045143+0000","last_undegraded":"2024-06-29T14:38:26.045143+0000","last_fullsized":"2024-06-29T14:38:26.045143+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:43:56.256764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045651+0000","last_change":"2024-06-29T14:38:24.878218+0000","last_active":"2024-06-29T14:38:26.045651+0000","last_peered":"2024-06-29T14:38:26.045651+0000","last_clean":"2024-06-29T14:38:26.045651+0000","last_became_active":"2024-06-29T14:38:24.877846+0000","last_became_peered":"2024-06-29T14:38:24.877846+0000","last_unstale":"2024-06-29T14:38:26.045651+0000","last_undegraded":"2024-06-29T14:38:26.045651+0000","last_fullsized":"2024-06-29T14:38:26.045651+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:28:19.363714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045300+0000","last_change":"2024-06-29T14:38:24.887628+0000","last_active":"2024-06-29T14:38:26.045300+0000","last_peered":"2024-06-29T14:38:26.045300+0000","last_clean":"2024-06-29T14:38:26.045300+0000","last_became_active":"2024-06-29T14:38:24.887317+0000","last_became_peered":"2024-06-29T14:38:24.887317+0000","last_unstale":"2024-06-29T14:38:26.045300+0000","last_undegraded":"2024-06-29T14:38:26.045300+0000","last_fullsized":"2024-06-29T14:38:26.045300+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:53.863238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881427+0000","last_change":"2024-06-29T14:38:24.885786+0000","last_active":"2024-06-29T14:38:25.881427+0000","last_peered":"2024-06-29T14:38:25.881427+0000","last_clean":"2024-06-29T14:38:25.881427+0000","last_became_active":"2024-06-29T14:38:24.885416+0000","last_became_peered":"2024-06-29T14:38:24.885416+0000","last_unstale":"2024-06-29T14:38:25.881427+0000","last_undegraded":"2024-06-29T14:38:25.881427+0000","last_fullsized":"2024-06-29T14:38:25.881427+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:20:53.669315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045381+0000","last_change":"2024-06-29T14:38:24.882092+0000","last_active":"2024-06-29T14:38:26.045381+0000","last_peered":"2024-06-29T14:38:26.045381+0000","last_clean":"2024-06-29T14:38:26.045381+0000","last_became_active":"2024-06-29T14:38:24.881227+0000","last_became_peered":"2024-06-29T14:38:24.881227+0000","last_unstale":"2024-06-29T14:38:26.045381+0000","last_undegraded":"2024-06-29T14:38:26.045381+0000","last_fullsized":"2024-06-29T14:38:26.045381+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:16:34.915868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045057+0000","last_change":"2024-06-29T14:38:24.877861+0000","last_active":"2024-06-29T14:38:26.045057+0000","last_peered":"2024-06-29T14:38:26.045057+0000","last_clean":"2024-06-29T14:38:26.045057+0000","last_became_active":"2024-06-29T14:38:24.877536+0000","last_became_peered":"2024-06-29T14:38:24.877536+0000","last_unstale":"2024-06-29T14:38:26.045057+0000","last_undegraded":"2024-06-29T14:38:26.045057+0000","last_fullsized":"2024-06-29T14:38:26.045057+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:20.891486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865261+0000","last_change":"2024-06-29T14:38:24.877853+0000","last_active":"2024-06-29T14:38:25.865261+0000","last_peered":"2024-06-29T14:38:25.865261+0000","last_clean":"2024-06-29T14:38:25.865261+0000","last_became_active":"2024-06-29T14:38:24.877495+0000","last_became_peered":"2024-06-29T14:38:24.877495+0000","last_unstale":"2024-06-29T14:38:25.865261+0000","last_undegraded":"2024-06-29T14:38:25.865261+0000","last_fullsized":"2024-06-29T14:38:25.865261+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:56:41.084977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868159+0000","last_change":"2024-06-29T14:38:24.884777+0000","last_active":"2024-06-29T14:38:25.868159+0000","last_peered":"2024-06-29T14:38:25.868159+0000","last_clean":"2024-06-29T14:38:25.868159+0000","last_became_active":"2024-06-29T14:38:24.884371+0000","last_became_peered":"2024-06-29T14:38:24.884371+0000","last_unstale":"2024-06-29T14:38:25.868159+0000","last_undegraded":"2024-06-29T14:38:25.868159+0000","last_fullsized":"2024-06-29T14:38:25.868159+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:17.736423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044708+0000","last_change":"2024-06-29T14:38:24.872900+0000","last_active":"2024-06-29T14:38:26.044708+0000","last_peered":"2024-06-29T14:38:26.044708+0000","last_clean":"2024-06-29T14:38:26.044708+0000","last_became_active":"2024-06-29T14:38:24.872578+0000","last_became_peered":"2024-06-29T14:38:24.872578+0000","last_unstale":"2024-06-29T14:38:26.044708+0000","last_undegraded":"2024-06-29T14:38:26.044708+0000","last_fullsized":"2024-06-29T14:38:26.044708+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:39:34.883426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865311+0000","last_change":"2024-06-29T14:38:24.890265+0000","last_active":"2024-06-29T14:38:25.865311+0000","last_peered":"2024-06-29T14:38:25.865311+0000","last_clean":"2024-06-29T14:38:25.865311+0000","last_became_active":"2024-06-29T14:38:24.889933+0000","last_became_peered":"2024-06-29T14:38:24.889933+0000","last_unstale":"2024-06-29T14:38:25.865311+0000","last_undegraded":"2024-06-29T14:38:25.865311+0000","last_fullsized":"2024-06-29T14:38:25.865311+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:21.850352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045577+0000","last_change":"2024-06-29T14:38:24.882267+0000","last_active":"2024-06-29T14:38:26.045577+0000","last_peered":"2024-06-29T14:38:26.045577+0000","last_clean":"2024-06-29T14:38:26.045577+0000","last_became_active":"2024-06-29T14:38:24.881889+0000","last_became_peered":"2024-06-29T14:38:24.881889+0000","last_unstale":"2024-06-29T14:38:26.045577+0000","last_undegraded":"2024-06-29T14:38:26.045577+0000","last_fullsized":"2024-06-29T14:38:26.045577+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:11.829543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045877+0000","last_change":"2024-06-29T14:38:24.872736+0000","last_active":"2024-06-29T14:38:26.045877+0000","last_peered":"2024-06-29T14:38:26.045877+0000","last_clean":"2024-06-29T14:38:26.045877+0000","last_became_active":"2024-06-29T14:38:24.872408+0000","last_became_peered":"2024-06-29T14:38:24.872408+0000","last_unstale":"2024-06-29T14:38:26.045877+0000","last_undegraded":"2024-06-29T14:38:26.045877+0000","last_fullsized":"2024-06-29T14:38:26.045877+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:23:21.368593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868683+0000","last_change":"2024-06-29T14:38:24.893858+0000","last_active":"2024-06-29T14:38:25.868683+0000","last_peered":"2024-06-29T14:38:25.868683+0000","last_clean":"2024-06-29T14:38:25.868683+0000","last_became_active":"2024-06-29T14:38:24.893341+0000","last_became_peered":"2024-06-29T14:38:24.893341+0000","last_unstale":"2024-06-29T14:38:25.868683+0000","last_undegraded":"2024-06-29T14:38:25.868683+0000","last_fullsized":"2024-06-29T14:38:25.868683+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:19:49.232504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865464+0000","last_change":"2024-06-29T14:38:24.877965+0000","last_active":"2024-06-29T14:38:25.865464+0000","last_peered":"2024-06-29T14:38:25.865464+0000","last_clean":"2024-06-29T14:38:25.865464+0000","last_became_active":"2024-06-29T14:38:24.877474+0000","last_became_peered":"2024-06-29T14:38:24.877474+0000","last_unstale":"2024-06-29T14:38:25.865464+0000","last_undegraded":"2024-06-29T14:38:25.865464+0000","last_fullsized":"2024-06-29T14:38:25.865464+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:56.351425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865495+0000","last_change":"2024-06-29T14:38:24.882576+0000","last_active":"2024-06-29T14:38:25.865495+0000","last_peered":"2024-06-29T14:38:25.865495+0000","last_clean":"2024-06-29T14:38:25.865495+0000","last_became_active":"2024-06-29T14:38:24.882270+0000","last_became_peered":"2024-06-29T14:38:24.882270+0000","last_unstale":"2024-06-29T14:38:25.865495+0000","last_undegraded":"2024-06-29T14:38:25.865495+0000","last_fullsized":"2024-06-29T14:38:25.865495+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:19:50.943728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045243+0000","last_change":"2024-06-29T14:38:24.879841+0000","last_active":"2024-06-29T14:38:26.045243+0000","last_peered":"2024-06-29T14:38:26.045243+0000","last_clean":"2024-06-29T14:38:26.045243+0000","last_became_active":"2024-06-29T14:38:24.879479+0000","last_became_peered":"2024-06-29T14:38:24.879479+0000","last_unstale":"2024-06-29T14:38:26.045243+0000","last_undegraded":"2024-06-29T14:38:26.045243+0000","last_fullsized":"2024-06-29T14:38:26.045243+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:30.157313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868125+0000","last_change":"2024-06-29T14:38:24.885493+0000","last_active":"2024-06-29T14:38:25.868125+0000","last_peered":"2024-06-29T14:38:25.868125+0000","last_clean":"2024-06-29T14:38:25.868125+0000","last_became_active":"2024-06-29T14:38:24.884701+0000","last_became_peered":"2024-06-29T14:38:24.884701+0000","last_unstale":"2024-06-29T14:38:25.868125+0000","last_undegraded":"2024-06-29T14:38:25.868125+0000","last_fullsized":"2024-06-29T14:38:25.868125+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:26.298376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046168+0000","last_change":"2024-06-29T14:38:24.891981+0000","last_active":"2024-06-29T14:38:26.046168+0000","last_peered":"2024-06-29T14:38:26.046168+0000","last_clean":"2024-06-29T14:38:26.046168+0000","last_became_active":"2024-06-29T14:38:24.891615+0000","last_became_peered":"2024-06-29T14:38:24.891615+0000","last_unstale":"2024-06-29T14:38:26.046168+0000","last_undegraded":"2024-06-29T14:38:26.046168+0000","last_fullsized":"2024-06-29T14:38:26.046168+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:54.986737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"103'4","reported_seq":18,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868776+0000","last_change":"2024-06-29T14:38:23.869892+0000","last_active":"2024-06-29T14:38:25.868776+0000","last_peered":"2024-06-29T14:38:25.868776+0000","last_clean":"2024-06-29T14:38:25.868776+0000","last_became_active":"2024-06-29T14:38:23.869503+0000","last_became_peered":"2024-06-29T14:38:23.869503+0000","last_unstale":"2024-06-29T14:38:25.868776+0000","last_undegraded":"2024-06-29T14:38:25.868776+0000","last_fullsized":"2024-06-29T14:38:25.868776+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:01:22.462383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868585+0000","last_change":"2024-06-29T14:38:24.891255+0000","last_active":"2024-06-29T14:38:25.868585+0000","last_peered":"2024-06-29T14:38:25.868585+0000","last_clean":"2024-06-29T14:38:25.868585+0000","last_became_active":"2024-06-29T14:38:24.890949+0000","last_became_peered":"2024-06-29T14:38:24.890949+0000","last_unstale":"2024-06-29T14:38:25.868585+0000","last_undegraded":"2024-06-29T14:38:25.868585+0000","last_fullsized":"2024-06-29T14:38:25.868585+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:01.826665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"103'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046299+0000","last_change":"2024-06-29T14:38:23.861675+0000","last_active":"2024-06-29T14:38:26.046299+0000","last_peered":"2024-06-29T14:38:26.046299+0000","last_clean":"2024-06-29T14:38:26.046299+0000","last_became_active":"2024-06-29T14:38:23.861353+0000","last_became_peered":"2024-06-29T14:38:23.861353+0000","last_unstale":"2024-06-29T14:38:26.046299+0000","last_undegraded":"2024-06-29T14:38:26.046299+0000","last_fullsized":"2024-06-29T14:38:26.046299+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:13:02.491883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868790+0000","last_change":"2024-06-29T14:38:24.886169+0000","last_active":"2024-06-29T14:38:25.868790+0000","last_peered":"2024-06-29T14:38:25.868790+0000","last_clean":"2024-06-29T14:38:25.868790+0000","last_became_active":"2024-06-29T14:38:24.885816+0000","last_became_peered":"2024-06-29T14:38:24.885816+0000","last_unstale":"2024-06-29T14:38:25.868790+0000","last_undegraded":"2024-06-29T14:38:25.868790+0000","last_fullsized":"2024-06-29T14:38:25.868790+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:12:03.006708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881440+0000","last_change":"2024-06-29T14:38:23.858739+0000","last_active":"2024-06-29T14:38:25.881440+0000","last_peered":"2024-06-29T14:38:25.881440+0000","last_clean":"2024-06-29T14:38:25.881440+0000","last_became_active":"2024-06-29T14:38:23.858435+0000","last_became_peered":"2024-06-29T14:38:23.858435+0000","last_unstale":"2024-06-29T14:38:25.881440+0000","last_undegraded":"2024-06-29T14:38:25.881440+0000","last_fullsized":"2024-06-29T14:38:25.881440+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:56:40.077517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045683+0000","last_change":"2024-06-29T14:38:24.881961+0000","last_active":"2024-06-29T14:38:26.045683+0000","last_peered":"2024-06-29T14:38:26.045683+0000","last_clean":"2024-06-29T14:38:26.045683+0000","last_became_active":"2024-06-29T14:38:24.881548+0000","last_became_peered":"2024-06-29T14:38:24.881548+0000","last_unstale":"2024-06-29T14:38:26.045683+0000","last_undegraded":"2024-06-29T14:38:26.045683+0000","last_fullsized":"2024-06-29T14:38:26.045683+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:32.034964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881370+0000","last_change":"2024-06-29T14:38:23.860453+0000","last_active":"2024-06-29T14:38:25.881370+0000","last_peered":"2024-06-29T14:38:25.881370+0000","last_clean":"2024-06-29T14:38:25.881370+0000","last_became_active":"2024-06-29T14:38:23.860140+0000","last_became_peered":"2024-06-29T14:38:23.860140+0000","last_unstale":"2024-06-29T14:38:25.881370+0000","last_undegraded":"2024-06-29T14:38:25.881370+0000","last_fullsized":"2024-06-29T14:38:25.881370+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:19.884026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865571+0000","last_change":"2024-06-29T14:38:24.878860+0000","last_active":"2024-06-29T14:38:25.865571+0000","last_peered":"2024-06-29T14:38:25.865571+0000","last_clean":"2024-06-29T14:38:25.865571+0000","last_became_active":"2024-06-29T14:38:24.878533+0000","last_became_peered":"2024-06-29T14:38:24.878533+0000","last_unstale":"2024-06-29T14:38:25.865571+0000","last_undegraded":"2024-06-29T14:38:25.865571+0000","last_fullsized":"2024-06-29T14:38:25.865571+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:26.298376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"103'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046009+0000","last_change":"2024-06-29T14:38:23.863690+0000","last_active":"2024-06-29T14:38:26.046009+0000","last_peered":"2024-06-29T14:38:26.046009+0000","last_clean":"2024-06-29T14:38:26.046009+0000","last_became_active":"2024-06-29T14:38:23.863345+0000","last_became_peered":"2024-06-29T14:38:23.863345+0000","last_unstale":"2024-06-29T14:38:26.046009+0000","last_undegraded":"2024-06-29T14:38:26.046009+0000","last_fullsized":"2024-06-29T14:38:26.046009+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:24.011231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045526+0000","last_change":"2024-06-29T14:38:24.876900+0000","last_active":"2024-06-29T14:38:26.045526+0000","last_peered":"2024-06-29T14:38:26.045526+0000","last_clean":"2024-06-29T14:38:26.045526+0000","last_became_active":"2024-06-29T14:38:24.876603+0000","last_became_peered":"2024-06-29T14:38:24.876603+0000","last_unstale":"2024-06-29T14:38:26.045526+0000","last_undegraded":"2024-06-29T14:38:26.045526+0000","last_fullsized":"2024-06-29T14:38:26.045526+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:41:54.265980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"103'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045160+0000","last_change":"2024-06-29T14:38:23.862520+0000","last_active":"2024-06-29T14:38:26.045160+0000","last_peered":"2024-06-29T14:38:26.045160+0000","last_clean":"2024-06-29T14:38:26.045160+0000","last_became_active":"2024-06-29T14:38:23.862199+0000","last_became_peered":"2024-06-29T14:38:23.862199+0000","last_unstale":"2024-06-29T14:38:26.045160+0000","last_undegraded":"2024-06-29T14:38:26.045160+0000","last_fullsized":"2024-06-29T14:38:26.045160+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:12:01.999248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045369+0000","last_change":"2024-06-29T14:38:24.877423+0000","last_active":"2024-06-29T14:38:26.045369+0000","last_peered":"2024-06-29T14:38:26.045369+0000","last_clean":"2024-06-29T14:38:26.045369+0000","last_became_active":"2024-06-29T14:38:24.877069+0000","last_became_peered":"2024-06-29T14:38:24.877069+0000","last_unstale":"2024-06-29T14:38:26.045369+0000","last_undegraded":"2024-06-29T14:38:26.045369+0000","last_fullsized":"2024-06-29T14:38:26.045369+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:25.968927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881096+0000","last_change":"2024-06-29T14:38:23.864149+0000","last_active":"2024-06-29T14:38:25.881096+0000","last_peered":"2024-06-29T14:38:25.881096+0000","last_clean":"2024-06-29T14:38:25.881096+0000","last_became_active":"2024-06-29T14:38:23.863834+0000","last_became_peered":"2024-06-29T14:38:23.863834+0000","last_unstale":"2024-06-29T14:38:25.881096+0000","last_undegraded":"2024-06-29T14:38:25.881096+0000","last_fullsized":"2024-06-29T14:38:25.881096+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:52.311707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044935+0000","last_change":"2024-06-29T14:38:24.881598+0000","last_active":"2024-06-29T14:38:26.044935+0000","last_peered":"2024-06-29T14:38:26.044935+0000","last_clean":"2024-06-29T14:38:26.044935+0000","last_became_active":"2024-06-29T14:38:24.881289+0000","last_became_peered":"2024-06-29T14:38:24.881289+0000","last_unstale":"2024-06-29T14:38:26.044935+0000","last_undegraded":"2024-06-29T14:38:26.044935+0000","last_fullsized":"2024-06-29T14:38:26.044935+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:29.602452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865194+0000","last_change":"2024-06-29T14:38:23.862900+0000","last_active":"2024-06-29T14:38:25.865194+0000","last_peered":"2024-06-29T14:38:25.865194+0000","last_clean":"2024-06-29T14:38:25.865194+0000","last_became_active":"2024-06-29T14:38:23.862562+0000","last_became_peered":"2024-06-29T14:38:23.862562+0000","last_unstale":"2024-06-29T14:38:25.865194+0000","last_undegraded":"2024-06-29T14:38:25.865194+0000","last_fullsized":"2024-06-29T14:38:25.865194+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:57.893947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045357+0000","last_change":"2024-06-29T14:38:24.871083+0000","last_active":"2024-06-29T14:38:26.045357+0000","last_peered":"2024-06-29T14:38:26.045357+0000","last_clean":"2024-06-29T14:38:26.045357+0000","last_became_active":"2024-06-29T14:38:24.870762+0000","last_became_peered":"2024-06-29T14:38:24.870762+0000","last_unstale":"2024-06-29T14:38:26.045357+0000","last_undegraded":"2024-06-29T14:38:26.045357+0000","last_fullsized":"2024-06-29T14:38:26.045357+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:25.968927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868900+0000","last_change":"2024-06-29T14:38:23.864993+0000","last_active":"2024-06-29T14:38:25.868900+0000","last_peered":"2024-06-29T14:38:25.868900+0000","last_clean":"2024-06-29T14:38:25.868900+0000","last_became_active":"2024-06-29T14:38:23.863274+0000","last_became_peered":"2024-06-29T14:38:23.863274+0000","last_unstale":"2024-06-29T14:38:25.868900+0000","last_undegraded":"2024-06-29T14:38:25.868900+0000","last_fullsized":"2024-06-29T14:38:25.868900+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:39:33.875966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867492+0000","last_change":"2024-06-29T14:38:24.880283+0000","last_active":"2024-06-29T14:38:25.867492+0000","last_peered":"2024-06-29T14:38:25.867492+0000","last_clean":"2024-06-29T14:38:25.867492+0000","last_became_active":"2024-06-29T14:38:24.879943+0000","last_became_peered":"2024-06-29T14:38:24.879943+0000","last_unstale":"2024-06-29T14:38:25.867492+0000","last_undegraded":"2024-06-29T14:38:25.867492+0000","last_fullsized":"2024-06-29T14:38:25.867492+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:29.180183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868611+0000","last_change":"2024-06-29T14:38:23.865407+0000","last_active":"2024-06-29T14:38:25.868611+0000","last_peered":"2024-06-29T14:38:25.868611+0000","last_clean":"2024-06-29T14:38:25.868611+0000","last_became_active":"2024-06-29T14:38:23.865081+0000","last_became_peered":"2024-06-29T14:38:23.865081+0000","last_unstale":"2024-06-29T14:38:25.868611+0000","last_undegraded":"2024-06-29T14:38:25.868611+0000","last_fullsized":"2024-06-29T14:38:25.868611+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:39:36.586162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881314+0000","last_change":"2024-06-29T14:38:24.886685+0000","last_active":"2024-06-29T14:38:25.881314+0000","last_peered":"2024-06-29T14:38:25.881314+0000","last_clean":"2024-06-29T14:38:25.881314+0000","last_became_active":"2024-06-29T14:38:24.886319+0000","last_became_peered":"2024-06-29T14:38:24.886319+0000","last_unstale":"2024-06-29T14:38:25.881314+0000","last_undegraded":"2024-06-29T14:38:25.881314+0000","last_fullsized":"2024-06-29T14:38:25.881314+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:57.459300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"103'3","reported_seq":17,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868462+0000","last_change":"2024-06-29T14:38:23.866309+0000","last_active":"2024-06-29T14:38:25.868462+0000","last_peered":"2024-06-29T14:38:25.868462+0000","last_clean":"2024-06-29T14:38:25.868462+0000","last_became_active":"2024-06-29T14:38:23.866007+0000","last_became_peered":"2024-06-29T14:38:23.866007+0000","last_unstale":"2024-06-29T14:38:25.868462+0000","last_undegraded":"2024-06-29T14:38:25.868462+0000","last_fullsized":"2024-06-29T14:38:25.868462+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:00:10.828893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045734+0000","last_change":"2024-06-29T14:38:24.880224+0000","last_active":"2024-06-29T14:38:26.045734+0000","last_peered":"2024-06-29T14:38:26.045734+0000","last_clean":"2024-06-29T14:38:26.045734+0000","last_became_active":"2024-06-29T14:38:24.879888+0000","last_became_peered":"2024-06-29T14:38:24.879888+0000","last_unstale":"2024-06-29T14:38:26.045734+0000","last_undegraded":"2024-06-29T14:38:26.045734+0000","last_fullsized":"2024-06-29T14:38:26.045734+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:32:19.396631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"103'6","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867266+0000","last_change":"2024-06-29T14:38:23.859256+0000","last_active":"2024-06-29T14:38:25.867266+0000","last_peered":"2024-06-29T14:38:25.867266+0000","last_clean":"2024-06-29T14:38:25.867266+0000","last_became_active":"2024-06-29T14:38:23.858825+0000","last_became_peered":"2024-06-29T14:38:23.858825+0000","last_unstale":"2024-06-29T14:38:25.867266+0000","last_undegraded":"2024-06-29T14:38:25.867266+0000","last_fullsized":"2024-06-29T14:38:25.867266+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:54.265685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881249+0000","last_change":"2024-06-29T14:38:24.881099+0000","last_active":"2024-06-29T14:38:25.881249+0000","last_peered":"2024-06-29T14:38:25.881249+0000","last_clean":"2024-06-29T14:38:25.881249+0000","last_became_active":"2024-06-29T14:38:24.880538+0000","last_became_peered":"2024-06-29T14:38:24.880538+0000","last_unstale":"2024-06-29T14:38:25.881249+0000","last_undegraded":"2024-06-29T14:38:25.881249+0000","last_fullsized":"2024-06-29T14:38:25.881249+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:26.298376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865560+0000","last_change":"2024-06-29T14:38:23.859990+0000","last_active":"2024-06-29T14:38:25.865560+0000","last_peered":"2024-06-29T14:38:25.865560+0000","last_clean":"2024-06-29T14:38:25.865560+0000","last_became_active":"2024-06-29T14:38:23.859684+0000","last_became_peered":"2024-06-29T14:38:23.859684+0000","last_unstale":"2024-06-29T14:38:25.865560+0000","last_undegraded":"2024-06-29T14:38:25.865560+0000","last_fullsized":"2024-06-29T14:38:25.865560+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:44.845658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046258+0000","last_change":"2024-06-29T14:38:24.880308+0000","last_active":"2024-06-29T14:38:26.046258+0000","last_peered":"2024-06-29T14:38:26.046258+0000","last_clean":"2024-06-29T14:38:26.046258+0000","last_became_active":"2024-06-29T14:38:24.879979+0000","last_became_peered":"2024-06-29T14:38:24.879979+0000","last_unstale":"2024-06-29T14:38:26.046258+0000","last_undegraded":"2024-06-29T14:38:26.046258+0000","last_fullsized":"2024-06-29T14:38:26.046258+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:47:25.308681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045114+0000","last_change":"2024-06-29T14:38:23.864842+0000","last_active":"2024-06-29T14:38:26.045114+0000","last_peered":"2024-06-29T14:38:26.045114+0000","last_clean":"2024-06-29T14:38:26.045114+0000","last_became_active":"2024-06-29T14:38:23.864431+0000","last_became_peered":"2024-06-29T14:38:23.864431+0000","last_unstale":"2024-06-29T14:38:26.045114+0000","last_undegraded":"2024-06-29T14:38:26.045114+0000","last_fullsized":"2024-06-29T14:38:26.045114+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:06:05.845482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865091+0000","last_change":"2024-06-29T14:38:24.874217+0000","last_active":"2024-06-29T14:38:25.865091+0000","last_peered":"2024-06-29T14:38:25.865091+0000","last_clean":"2024-06-29T14:38:25.865091+0000","last_became_active":"2024-06-29T14:38:24.873913+0000","last_became_peered":"2024-06-29T14:38:24.873913+0000","last_unstale":"2024-06-29T14:38:25.865091+0000","last_undegraded":"2024-06-29T14:38:25.865091+0000","last_fullsized":"2024-06-29T14:38:25.865091+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:20.891486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867006+0000","last_change":"2024-06-29T14:38:23.861597+0000","last_active":"2024-06-29T14:38:25.867006+0000","last_peered":"2024-06-29T14:38:25.867006+0000","last_clean":"2024-06-29T14:38:25.867006+0000","last_became_active":"2024-06-29T14:38:23.861290+0000","last_became_peered":"2024-06-29T14:38:23.861290+0000","last_unstale":"2024-06-29T14:38:25.867006+0000","last_undegraded":"2024-06-29T14:38:25.867006+0000","last_fullsized":"2024-06-29T14:38:25.867006+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:35:09.672272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865378+0000","last_change":"2024-06-29T14:38:24.887998+0000","last_active":"2024-06-29T14:38:25.865378+0000","last_peered":"2024-06-29T14:38:25.865378+0000","last_clean":"2024-06-29T14:38:25.865378+0000","last_became_active":"2024-06-29T14:38:24.887703+0000","last_became_peered":"2024-06-29T14:38:24.887703+0000","last_unstale":"2024-06-29T14:38:25.865378+0000","last_undegraded":"2024-06-29T14:38:25.865378+0000","last_fullsized":"2024-06-29T14:38:25.865378+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:33:13.047537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"103'8","reported_seq":22,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881054+0000","last_change":"2024-06-29T14:38:23.864330+0000","last_active":"2024-06-29T14:38:25.881054+0000","last_peered":"2024-06-29T14:38:25.881054+0000","last_clean":"2024-06-29T14:38:25.881054+0000","last_became_active":"2024-06-29T14:38:23.863064+0000","last_became_peered":"2024-06-29T14:38:23.863064+0000","last_unstale":"2024-06-29T14:38:25.881054+0000","last_undegraded":"2024-06-29T14:38:25.881054+0000","last_fullsized":"2024-06-29T14:38:25.881054+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:04.726969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045749+0000","last_change":"2024-06-29T14:38:24.883075+0000","last_active":"2024-06-29T14:38:26.045749+0000","last_peered":"2024-06-29T14:38:26.045749+0000","last_clean":"2024-06-29T14:38:26.045749+0000","last_became_active":"2024-06-29T14:38:24.882674+0000","last_became_peered":"2024-06-29T14:38:24.882674+0000","last_unstale":"2024-06-29T14:38:26.045749+0000","last_undegraded":"2024-06-29T14:38:26.045749+0000","last_fullsized":"2024-06-29T14:38:26.045749+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:49.579212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"103'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868981+0000","last_change":"2024-06-29T14:38:23.860074+0000","last_active":"2024-06-29T14:38:25.868981+0000","last_peered":"2024-06-29T14:38:25.868981+0000","last_clean":"2024-06-29T14:38:25.868981+0000","last_became_active":"2024-06-29T14:38:23.859730+0000","last_became_peered":"2024-06-29T14:38:23.859730+0000","last_unstale":"2024-06-29T14:38:25.868981+0000","last_undegraded":"2024-06-29T14:38:25.868981+0000","last_fullsized":"2024-06-29T14:38:25.868981+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:30.988693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868828+0000","last_change":"2024-06-29T14:38:24.889550+0000","last_active":"2024-06-29T14:38:25.868828+0000","last_peered":"2024-06-29T14:38:25.868828+0000","last_clean":"2024-06-29T14:38:25.868828+0000","last_became_active":"2024-06-29T14:38:24.889270+0000","last_became_peered":"2024-06-29T14:38:24.889270+0000","last_unstale":"2024-06-29T14:38:25.868828+0000","last_undegraded":"2024-06-29T14:38:25.868828+0000","last_fullsized":"2024-06-29T14:38:25.868828+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:21:59.608621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881518+0000","last_change":"2024-06-29T14:38:23.863982+0000","last_active":"2024-06-29T14:38:25.881518+0000","last_peered":"2024-06-29T14:38:25.881518+0000","last_clean":"2024-06-29T14:38:25.881518+0000","last_became_active":"2024-06-29T14:38:23.863705+0000","last_became_peered":"2024-06-29T14:38:23.863705+0000","last_unstale":"2024-06-29T14:38:25.881518+0000","last_undegraded":"2024-06-29T14:38:25.881518+0000","last_fullsized":"2024-06-29T14:38:25.881518+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:16:33.908408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045968+0000","last_change":"2024-06-29T14:38:24.870357+0000","last_active":"2024-06-29T14:38:26.045968+0000","last_peered":"2024-06-29T14:38:26.045968+0000","last_clean":"2024-06-29T14:38:26.045968+0000","last_became_active":"2024-06-29T14:38:24.870004+0000","last_became_peered":"2024-06-29T14:38:24.870004+0000","last_unstale":"2024-06-29T14:38:26.045968+0000","last_undegraded":"2024-06-29T14:38:26.045968+0000","last_fullsized":"2024-06-29T14:38:26.045968+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:43:21.820955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881468+0000","last_change":"2024-06-29T14:38:23.861238+0000","last_active":"2024-06-29T14:38:25.881468+0000","last_peered":"2024-06-29T14:38:25.881468+0000","last_clean":"2024-06-29T14:38:25.881468+0000","last_became_active":"2024-06-29T14:38:23.860831+0000","last_became_peered":"2024-06-29T14:38:23.860831+0000","last_unstale":"2024-06-29T14:38:25.881468+0000","last_undegraded":"2024-06-29T14:38:25.881468+0000","last_fullsized":"2024-06-29T14:38:25.881468+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:55.343965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045333+0000","last_change":"2024-06-29T14:38:24.874105+0000","last_active":"2024-06-29T14:38:26.045333+0000","last_peered":"2024-06-29T14:38:26.045333+0000","last_clean":"2024-06-29T14:38:26.045333+0000","last_became_active":"2024-06-29T14:38:24.873820+0000","last_became_peered":"2024-06-29T14:38:24.873820+0000","last_unstale":"2024-06-29T14:38:26.045333+0000","last_undegraded":"2024-06-29T14:38:26.045333+0000","last_fullsized":"2024-06-29T14:38:26.045333+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:56:41.084977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045473+0000","last_change":"2024-06-29T14:38:23.870453+0000","last_active":"2024-06-29T14:38:26.045473+0000","last_peered":"2024-06-29T14:38:26.045473+0000","last_clean":"2024-06-29T14:38:26.045473+0000","last_became_active":"2024-06-29T14:38:23.870126+0000","last_became_peered":"2024-06-29T14:38:23.870126+0000","last_unstale":"2024-06-29T14:38:26.045473+0000","last_undegraded":"2024-06-29T14:38:26.045473+0000","last_fullsized":"2024-06-29T14:38:26.045473+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:20.908520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867126+0000","last_change":"2024-06-29T14:38:24.875963+0000","last_active":"2024-06-29T14:38:25.867126+0000","last_peered":"2024-06-29T14:38:25.867126+0000","last_clean":"2024-06-29T14:38:25.867126+0000","last_became_active":"2024-06-29T14:38:24.875616+0000","last_became_peered":"2024-06-29T14:38:24.875616+0000","last_unstale":"2024-06-29T14:38:25.867126+0000","last_undegraded":"2024-06-29T14:38:25.867126+0000","last_fullsized":"2024-06-29T14:38:25.867126+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:15.998851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045052+0000","last_change":"2024-06-29T14:38:23.867352+0000","last_active":"2024-06-29T14:38:26.045052+0000","last_peered":"2024-06-29T14:38:26.045052+0000","last_clean":"2024-06-29T14:38:26.045052+0000","last_became_active":"2024-06-29T14:38:23.866429+0000","last_became_peered":"2024-06-29T14:38:23.866429+0000","last_unstale":"2024-06-29T14:38:26.045052+0000","last_undegraded":"2024-06-29T14:38:26.045052+0000","last_fullsized":"2024-06-29T14:38:26.045052+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:52.183288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044751+0000","last_change":"2024-06-29T14:38:24.881521+0000","last_active":"2024-06-29T14:38:26.044751+0000","last_peered":"2024-06-29T14:38:26.044751+0000","last_clean":"2024-06-29T14:38:26.044751+0000","last_became_active":"2024-06-29T14:38:24.881215+0000","last_became_peered":"2024-06-29T14:38:24.881215+0000","last_unstale":"2024-06-29T14:38:26.044751+0000","last_undegraded":"2024-06-29T14:38:26.044751+0000","last_fullsized":"2024-06-29T14:38:26.044751+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:05.734429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044810+0000","last_change":"2024-06-29T14:38:23.866866+0000","last_active":"2024-06-29T14:38:26.044810+0000","last_peered":"2024-06-29T14:38:26.044810+0000","last_clean":"2024-06-29T14:38:26.044810+0000","last_became_active":"2024-06-29T14:38:23.866561+0000","last_became_peered":"2024-06-29T14:38:23.866561+0000","last_unstale":"2024-06-29T14:38:26.044810+0000","last_undegraded":"2024-06-29T14:38:26.044810+0000","last_fullsized":"2024-06-29T14:38:26.044810+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:00:10.239535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867104+0000","last_change":"2024-06-29T14:38:24.878172+0000","last_active":"2024-06-29T14:38:25.867104+0000","last_peered":"2024-06-29T14:38:25.867104+0000","last_clean":"2024-06-29T14:38:25.867104+0000","last_became_active":"2024-06-29T14:38:24.877839+0000","last_became_peered":"2024-06-29T14:38:24.877839+0000","last_unstale":"2024-06-29T14:38:25.867104+0000","last_undegraded":"2024-06-29T14:38:25.867104+0000","last_fullsized":"2024-06-29T14:38:25.867104+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:01.590805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044581+0000","last_change":"2024-06-29T14:38:23.870477+0000","last_active":"2024-06-29T14:38:26.044581+0000","last_peered":"2024-06-29T14:38:26.044581+0000","last_clean":"2024-06-29T14:38:26.044581+0000","last_became_active":"2024-06-29T14:38:23.870172+0000","last_became_peered":"2024-06-29T14:38:23.870172+0000","last_unstale":"2024-06-29T14:38:26.044581+0000","last_undegraded":"2024-06-29T14:38:26.044581+0000","last_fullsized":"2024-06-29T14:38:26.044581+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:42.862786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046169+0000","last_change":"2024-06-29T14:38:24.870586+0000","last_active":"2024-06-29T14:38:26.046169+0000","last_peered":"2024-06-29T14:38:26.046169+0000","last_clean":"2024-06-29T14:38:26.046169+0000","last_became_active":"2024-06-29T14:38:24.869888+0000","last_became_peered":"2024-06-29T14:38:24.869888+0000","last_unstale":"2024-06-29T14:38:26.046169+0000","last_undegraded":"2024-06-29T14:38:26.046169+0000","last_fullsized":"2024-06-29T14:38:26.046169+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:51.649114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865375+0000","last_change":"2024-06-29T14:38:23.862060+0000","last_active":"2024-06-29T14:38:25.865375+0000","last_peered":"2024-06-29T14:38:25.865375+0000","last_clean":"2024-06-29T14:38:25.865375+0000","last_became_active":"2024-06-29T14:38:23.861757+0000","last_became_peered":"2024-06-29T14:38:23.861757+0000","last_unstale":"2024-06-29T14:38:25.865375+0000","last_undegraded":"2024-06-29T14:38:25.865375+0000","last_fullsized":"2024-06-29T14:38:25.865375+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:52.183288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045993+0000","last_change":"2024-06-29T14:38:24.893919+0000","last_active":"2024-06-29T14:38:26.045993+0000","last_peered":"2024-06-29T14:38:26.045993+0000","last_clean":"2024-06-29T14:38:26.045993+0000","last_became_active":"2024-06-29T14:38:24.893427+0000","last_became_peered":"2024-06-29T14:38:24.893427+0000","last_unstale":"2024-06-29T14:38:26.045993+0000","last_undegraded":"2024-06-29T14:38:26.045993+0000","last_fullsized":"2024-06-29T14:38:26.045993+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:14.728843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"104'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.880381+0000","last_change":"2024-06-29T14:38:23.860797+0000","last_active":"2024-06-29T14:38:25.880381+0000","last_peered":"2024-06-29T14:38:25.880381+0000","last_clean":"2024-06-29T14:38:25.880381+0000","last_became_active":"2024-06-29T14:38:23.860469+0000","last_became_peered":"2024-06-29T14:38:23.860469+0000","last_unstale":"2024-06-29T14:38:25.880381+0000","last_undegraded":"2024-06-29T14:38:25.880381+0000","last_fullsized":"2024-06-29T14:38:25.880381+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:37.100494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868932+0000","last_change":"2024-06-29T14:38:24.886902+0000","last_active":"2024-06-29T14:38:25.868932+0000","last_peered":"2024-06-29T14:38:25.868932+0000","last_clean":"2024-06-29T14:38:25.868932+0000","last_became_active":"2024-06-29T14:38:24.886526+0000","last_became_peered":"2024-06-29T14:38:24.886526+0000","last_unstale":"2024-06-29T14:38:25.868932+0000","last_undegraded":"2024-06-29T14:38:25.868932+0000","last_fullsized":"2024-06-29T14:38:25.868932+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:13:03.499343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868792+0000","last_change":"2024-06-29T14:38:23.869524+0000","last_active":"2024-06-29T14:38:25.868792+0000","last_peered":"2024-06-29T14:38:25.868792+0000","last_clean":"2024-06-29T14:38:25.868792+0000","last_became_active":"2024-06-29T14:38:23.869285+0000","last_became_peered":"2024-06-29T14:38:23.869285+0000","last_unstale":"2024-06-29T14:38:25.868792+0000","last_undegraded":"2024-06-29T14:38:25.868792+0000","last_fullsized":"2024-06-29T14:38:25.868792+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:24.526496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046110+0000","last_change":"2024-06-29T14:38:24.881811+0000","last_active":"2024-06-29T14:38:26.046110+0000","last_peered":"2024-06-29T14:38:26.046110+0000","last_clean":"2024-06-29T14:38:26.046110+0000","last_became_active":"2024-06-29T14:38:24.880561+0000","last_became_peered":"2024-06-29T14:38:24.880561+0000","last_unstale":"2024-06-29T14:38:26.046110+0000","last_undegraded":"2024-06-29T14:38:26.046110+0000","last_fullsized":"2024-06-29T14:38:26.046110+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:13.291919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868931+0000","last_change":"2024-06-29T14:38:23.861543+0000","last_active":"2024-06-29T14:38:25.868931+0000","last_peered":"2024-06-29T14:38:25.868931+0000","last_clean":"2024-06-29T14:38:25.868931+0000","last_became_active":"2024-06-29T14:38:23.861172+0000","last_became_peered":"2024-06-29T14:38:23.861172+0000","last_unstale":"2024-06-29T14:38:25.868931+0000","last_undegraded":"2024-06-29T14:38:25.868931+0000","last_fullsized":"2024-06-29T14:38:25.868931+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:46:02.500746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867348+0000","last_change":"2024-06-29T14:38:24.876023+0000","last_active":"2024-06-29T14:38:25.867348+0000","last_peered":"2024-06-29T14:38:25.867348+0000","last_clean":"2024-06-29T14:38:25.867348+0000","last_became_active":"2024-06-29T14:38:24.875716+0000","last_became_peered":"2024-06-29T14:38:24.875716+0000","last_unstale":"2024-06-29T14:38:25.867348+0000","last_undegraded":"2024-06-29T14:38:25.867348+0000","last_fullsized":"2024-06-29T14:38:25.867348+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:59.570624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881426+0000","last_change":"2024-06-29T14:38:23.863495+0000","last_active":"2024-06-29T14:38:25.881426+0000","last_peered":"2024-06-29T14:38:25.881426+0000","last_clean":"2024-06-29T14:38:25.881426+0000","last_became_active":"2024-06-29T14:38:23.863135+0000","last_became_peered":"2024-06-29T14:38:23.863135+0000","last_unstale":"2024-06-29T14:38:25.881426+0000","last_undegraded":"2024-06-29T14:38:25.881426+0000","last_fullsized":"2024-06-29T14:38:25.881426+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:52.855778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881257+0000","last_change":"2024-06-29T14:38:24.883262+0000","last_active":"2024-06-29T14:38:25.881257+0000","last_peered":"2024-06-29T14:38:25.881257+0000","last_clean":"2024-06-29T14:38:25.881257+0000","last_became_active":"2024-06-29T14:38:24.882492+0000","last_became_peered":"2024-06-29T14:38:24.882492+0000","last_unstale":"2024-06-29T14:38:25.881257+0000","last_undegraded":"2024-06-29T14:38:25.881257+0000","last_fullsized":"2024-06-29T14:38:25.881257+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:17.736423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045280+0000","last_change":"2024-06-29T14:38:23.864312+0000","last_active":"2024-06-29T14:38:26.045280+0000","last_peered":"2024-06-29T14:38:26.045280+0000","last_clean":"2024-06-29T14:38:26.045280+0000","last_became_active":"2024-06-29T14:38:23.863953+0000","last_became_peered":"2024-06-29T14:38:23.863953+0000","last_unstale":"2024-06-29T14:38:26.045280+0000","last_undegraded":"2024-06-29T14:38:26.045280+0000","last_fullsized":"2024-06-29T14:38:26.045280+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:21:58.601161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045591+0000","last_change":"2024-06-29T14:38:24.881400+0000","last_active":"2024-06-29T14:38:26.045591+0000","last_peered":"2024-06-29T14:38:26.045591+0000","last_clean":"2024-06-29T14:38:26.045591+0000","last_became_active":"2024-06-29T14:38:24.881103+0000","last_became_peered":"2024-06-29T14:38:24.881103+0000","last_unstale":"2024-06-29T14:38:26.045591+0000","last_undegraded":"2024-06-29T14:38:26.045591+0000","last_fullsized":"2024-06-29T14:38:26.045591+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:17:22.571127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868360+0000","last_change":"2024-06-29T14:38:23.864138+0000","last_active":"2024-06-29T14:38:25.868360+0000","last_peered":"2024-06-29T14:38:25.868360+0000","last_clean":"2024-06-29T14:38:25.868360+0000","last_became_active":"2024-06-29T14:38:23.863429+0000","last_became_peered":"2024-06-29T14:38:23.863429+0000","last_unstale":"2024-06-29T14:38:25.868360+0000","last_undegraded":"2024-06-29T14:38:25.868360+0000","last_fullsized":"2024-06-29T14:38:25.868360+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:38:23.831509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868221+0000","last_change":"2024-06-29T14:38:24.884561+0000","last_active":"2024-06-29T14:38:25.868221+0000","last_peered":"2024-06-29T14:38:25.868221+0000","last_clean":"2024-06-29T14:38:25.868221+0000","last_became_active":"2024-06-29T14:38:24.884185+0000","last_became_peered":"2024-06-29T14:38:24.884185+0000","last_unstale":"2024-06-29T14:38:25.868221+0000","last_undegraded":"2024-06-29T14:38:25.868221+0000","last_fullsized":"2024-06-29T14:38:25.868221+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:57.459300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045142+0000","last_change":"2024-06-29T14:38:23.865707+0000","last_active":"2024-06-29T14:38:26.045142+0000","last_peered":"2024-06-29T14:38:26.045142+0000","last_clean":"2024-06-29T14:38:26.045142+0000","last_became_active":"2024-06-29T14:38:23.865398+0000","last_became_peered":"2024-06-29T14:38:23.865398+0000","last_unstale":"2024-06-29T14:38:26.045142+0000","last_undegraded":"2024-06-29T14:38:26.045142+0000","last_fullsized":"2024-06-29T14:38:26.045142+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:43:46.596358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865300+0000","last_change":"2024-06-29T14:38:24.876616+0000","last_active":"2024-06-29T14:38:25.865300+0000","last_peered":"2024-06-29T14:38:25.865300+0000","last_clean":"2024-06-29T14:38:25.865300+0000","last_became_active":"2024-06-29T14:38:24.875729+0000","last_became_peered":"2024-06-29T14:38:24.875729+0000","last_unstale":"2024-06-29T14:38:25.865300+0000","last_undegraded":"2024-06-29T14:38:25.865300+0000","last_fullsized":"2024-06-29T14:38:25.865300+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:16:34.915868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"103'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045773+0000","last_change":"2024-06-29T14:38:23.867208+0000","last_active":"2024-06-29T14:38:26.045773+0000","last_peered":"2024-06-29T14:38:26.045773+0000","last_clean":"2024-06-29T14:38:26.045773+0000","last_became_active":"2024-06-29T14:38:23.866521+0000","last_became_peered":"2024-06-29T14:38:23.866521+0000","last_unstale":"2024-06-29T14:38:26.045773+0000","last_undegraded":"2024-06-29T14:38:26.045773+0000","last_fullsized":"2024-06-29T14:38:26.045773+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:11.028911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"93'114","reported_seq":423,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045667+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:38:26.045667+0000","last_peered":"2024-06-29T14:38:26.045667+0000","last_clean":"2024-06-29T14:38:26.045667+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:38:26.045667+0000","last_undegraded":"2024-06-29T14:38:26.045667+0000","last_fullsized":"2024-06-29T14:38:26.045667+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":173,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":173,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1863680,"data_stored":1194664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575043,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1277952,"data_stored":604920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36592,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335248,"statfs":{"total":96636764160,"available":96599293952,"internally_reserved":0,"allocated":1282048,"data_stored":605860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58804,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313036,"statfs":{"total":96636764160,"available":96576548864,"internally_reserved":0,"allocated":1277952,"data_stored":610598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36596,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335244,"statfs":{"total":96636764160,"available":96599289856,"internally_reserved":0,"allocated":1277952,"data_stored":605838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575042,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36592,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335248,"statfs":{"total":96636764160,"available":96599293952,"internally_reserved":0,"allocated":1282048,"data_stored":611180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50492,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321348,"statfs":{"total":96636764160,"available":96585060352,"internally_reserved":0,"allocated":1290240,"data_stored":610792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575042,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1875968,"data_stored":1201000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:38:28.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.057+0000 7f5f9f672640 1 -- 172.21.15.81:0/1253964760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980cea50 msgr2=0x7f5f980b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.057+0000 7f5f9f672640 1 --2- 172.21.15.81:0/1253964760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980cea50 0x7f5f980b4810 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f5f98073550 tx=0x7f5f9002a820 comp rx=0 tx=0).stop 2024-06-29T14:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.057+0000 7f5f9f672640 1 -- 172.21.15.81:0/1253964760 shutdown_connections 2024-06-29T14:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.057+0000 7f5f9f672640 1 --2- 172.21.15.81:0/1253964760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980cea50 0x7f5f980b4810 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.057+0000 7f5f9f672640 1 --2- 172.21.15.81:0/1253964760 >> v2:172.21.15.167:3300/0 conn(0x7f5f980cd480 0x7f5f980ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.057+0000 7f5f9f672640 1 -- 172.21.15.81:0/1253964760 >> 172.21.15.81:0/1253964760 conn(0x7f5f980b3510 msgr2=0x7f5f980b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.057+0000 7f5f9f672640 1 -- 172.21.15.81:0/1253964760 shutdown_connections 2024-06-29T14:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.057+0000 7f5f9f672640 1 -- 172.21.15.81:0/1253964760 wait complete. 2024-06-29T14:38:29.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.059+0000 7f5f9f672640 1 Processor -- start 2024-06-29T14:38:29.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.059+0000 7f5f9f672640 1 -- start start 2024-06-29T14:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.060+0000 7f5f9f672640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f980b6eb0 0x7f5f980b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.060+0000 7f5f9f672640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980cd480 0x7f5f980b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.060+0000 7f5f9f672640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f980cea50 0x7f5f981d4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.060+0000 7f5f9f672640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f980b4ed0 con 0x7f5f980cd480 2024-06-29T14:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.060+0000 7f5f9f672640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5f980b4d50 con 0x7f5f980cea50 2024-06-29T14:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.060+0000 7f5f9f672640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5f980b5050 con 0x7f5f980b6eb0 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.060+0000 7f5f9cc0d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980cd480 0x7f5f980b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.060+0000 7f5f9cc0d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980cd480 0x7f5f980b0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38758/0 (socket says 172.21.15.81:38758) 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.060+0000 7f5f9cc0d640 1 -- 172.21.15.81:0/2505979589 learned_addr learned my addr 172.21.15.81:0/2505979589 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9d40e640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f980b6eb0 0x7f5f980b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9dc0f640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f980cea50 0x7f5f981d4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9cc0d640 1 -- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f980b6eb0 msgr2=0x7f5f980b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9cc0d640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f980b6eb0 0x7f5f980b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9cc0d640 1 -- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f980cea50 msgr2=0x7f5f981d4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9cc0d640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f980cea50 0x7f5f981d4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9cc0d640 1 -- 172.21.15.81:0/2505979589 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f980b1200 con 0x7f5f980cd480 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9d40e640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f980b6eb0 0x7f5f980b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9dc0f640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f980cea50 0x7f5f981d4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9cc0d640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980cd480 0x7f5f980b0b10 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f5f8800c970 tx=0x7f5f8800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f8dffb640 1 -- 172.21.15.81:0/2505979589 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f88007bb0 con 0x7f5f980cd480 2024-06-29T14:38:29.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9f672640 1 -- 172.21.15.81:0/2505979589 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f980b14e0 con 0x7f5f980cd480 2024-06-29T14:38:29.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f8dffb640 1 -- 172.21.15.81:0/2505979589 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f88007e40 con 0x7f5f980cd480 2024-06-29T14:38:29.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f8dffb640 1 -- 172.21.15.81:0/2505979589 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f88005280 con 0x7f5f980cd480 2024-06-29T14:38:29.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.061+0000 7f5f9f672640 1 -- 172.21.15.81:0/2505979589 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f980ca1e0 con 0x7f5f980cd480 2024-06-29T14:38:29.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.063+0000 7f5f9f672640 1 -- 172.21.15.81:0/2505979589 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f980aed40 con 0x7f5f980cd480 2024-06-29T14:38:29.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.063+0000 7f5f8dffb640 1 -- 172.21.15.81:0/2505979589 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f5f88005420 con 0x7f5f980cd480 2024-06-29T14:38:29.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.063+0000 7f5f8dffb640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5f7c0763f0 0x7f5f7c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.063+0000 7f5f8dffb640 1 -- 172.21.15.81:0/2505979589 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f5f88097ef0 con 0x7f5f980cd480 2024-06-29T14:38:29.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.064+0000 7f5f9d40e640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5f7c0763f0 0x7f5f7c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.064+0000 7f5f9d40e640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5f7c0763f0 0x7f5f7c078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000f60 tx=0x7f5f94009040 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.068+0000 7f5f8dffb640 1 -- 172.21.15.81:0/2505979589 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5f88014030 con 0x7f5f980cd480 2024-06-29T14:38:29.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.245+0000 7f5f9f672640 1 -- 172.21.15.81:0/2505979589 --> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5f980c8a30 con 0x7f5f7c0763f0 2024-06-29T14:38:29.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.250+0000 7f5f8dffb640 1 -- 172.21.15.81:0/2505979589 <== mgr.6158 v2:172.21.15.167:6832/2583731609 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252485 (secure 0 0 0) 0x7f5f980c8a30 con 0x7f5f7c0763f0 2024-06-29T14:38:29.249 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:29.251 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:38:29.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 -- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5f7c0763f0 msgr2=0x7f5f7c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5f7c0763f0 0x7f5f7c078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5f94000f60 tx=0x7f5f94009040 comp rx=0 tx=0).stop 2024-06-29T14:38:29.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 -- 172.21.15.81:0/2505979589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980cd480 msgr2=0x7f5f980b0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980cd480 0x7f5f980b0b10 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f5f8800c970 tx=0x7f5f8800ce30 comp rx=0 tx=0).stop 2024-06-29T14:38:29.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 -- 172.21.15.81:0/2505979589 shutdown_connections 2024-06-29T14:38:29.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f5f7c0763f0 0x7f5f7c078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f980cea50 0x7f5f981d4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980cd480 0x7f5f980b0b10 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 --2- 172.21.15.81:0/2505979589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f980b6eb0 0x7f5f980b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 -- 172.21.15.81:0/2505979589 >> 172.21.15.81:0/2505979589 conn(0x7f5f980b3510 msgr2=0x7f5f980b54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:29.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 -- 172.21.15.81:0/2505979589 shutdown_connections 2024-06-29T14:38:29.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.253+0000 7f5f9f672640 1 -- 172.21.15.81:0/2505979589 wait complete. 2024-06-29T14:38:29.261 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:38:28.441243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":218,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":317524,"kb_used_data":11160,"kb_used_omap":12,"kb_used_meta":306227,"kb_avail":754657196,"statfs":{"total":773094113280,"available":772768968704,"internally_reserved":0,"allocated":11427840,"data_stored":6044852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":313577010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.599151"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868615+0000","last_change":"2024-06-29T14:38:24.895717+0000","last_active":"2024-06-29T14:38:25.868615+0000","last_peered":"2024-06-29T14:38:25.868615+0000","last_clean":"2024-06-29T14:38:25.868615+0000","last_became_active":"2024-06-29T14:38:24.895364+0000","last_became_peered":"2024-06-29T14:38:24.895364+0000","last_unstale":"2024-06-29T14:38:25.868615+0000","last_undegraded":"2024-06-29T14:38:25.868615+0000","last_fullsized":"2024-06-29T14:38:25.868615+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:30.259008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865436+0000","last_change":"2024-06-29T14:38:24.890785+0000","last_active":"2024-06-29T14:38:25.865436+0000","last_peered":"2024-06-29T14:38:25.865436+0000","last_clean":"2024-06-29T14:38:25.865436+0000","last_became_active":"2024-06-29T14:38:24.890352+0000","last_became_peered":"2024-06-29T14:38:24.890352+0000","last_unstale":"2024-06-29T14:38:25.865436+0000","last_undegraded":"2024-06-29T14:38:25.865436+0000","last_fullsized":"2024-06-29T14:38:25.865436+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:59.321242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045559+0000","last_change":"2024-06-29T14:38:24.886809+0000","last_active":"2024-06-29T14:38:26.045559+0000","last_peered":"2024-06-29T14:38:26.045559+0000","last_clean":"2024-06-29T14:38:26.045559+0000","last_became_active":"2024-06-29T14:38:24.886402+0000","last_became_peered":"2024-06-29T14:38:24.886402+0000","last_unstale":"2024-06-29T14:38:26.045559+0000","last_undegraded":"2024-06-29T14:38:26.045559+0000","last_fullsized":"2024-06-29T14:38:26.045559+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:01:55.368956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045231+0000","last_change":"2024-06-29T14:38:24.882495+0000","last_active":"2024-06-29T14:38:26.045231+0000","last_peered":"2024-06-29T14:38:26.045231+0000","last_clean":"2024-06-29T14:38:26.045231+0000","last_became_active":"2024-06-29T14:38:24.881796+0000","last_became_peered":"2024-06-29T14:38:24.881796+0000","last_unstale":"2024-06-29T14:38:26.045231+0000","last_undegraded":"2024-06-29T14:38:26.045231+0000","last_fullsized":"2024-06-29T14:38:26.045231+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:56.351425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045310+0000","last_change":"2024-06-29T14:38:24.875308+0000","last_active":"2024-06-29T14:38:26.045310+0000","last_peered":"2024-06-29T14:38:26.045310+0000","last_clean":"2024-06-29T14:38:26.045310+0000","last_became_active":"2024-06-29T14:38:24.873008+0000","last_became_peered":"2024-06-29T14:38:24.873008+0000","last_unstale":"2024-06-29T14:38:26.045310+0000","last_undegraded":"2024-06-29T14:38:26.045310+0000","last_fullsized":"2024-06-29T14:38:26.045310+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:40:43.916544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867170+0000","last_change":"2024-06-29T14:38:24.881179+0000","last_active":"2024-06-29T14:38:25.867170+0000","last_peered":"2024-06-29T14:38:25.867170+0000","last_clean":"2024-06-29T14:38:25.867170+0000","last_became_active":"2024-06-29T14:38:24.880860+0000","last_became_peered":"2024-06-29T14:38:24.880860+0000","last_unstale":"2024-06-29T14:38:25.867170+0000","last_undegraded":"2024-06-29T14:38:25.867170+0000","last_fullsized":"2024-06-29T14:38:25.867170+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:58.034581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044703+0000","last_change":"2024-06-29T14:38:24.886592+0000","last_active":"2024-06-29T14:38:26.044703+0000","last_peered":"2024-06-29T14:38:26.044703+0000","last_clean":"2024-06-29T14:38:26.044703+0000","last_became_active":"2024-06-29T14:38:24.886266+0000","last_became_peered":"2024-06-29T14:38:24.886266+0000","last_unstale":"2024-06-29T14:38:26.044703+0000","last_undegraded":"2024-06-29T14:38:26.044703+0000","last_fullsized":"2024-06-29T14:38:26.044703+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:31.996153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867000+0000","last_change":"2024-06-29T14:38:24.874632+0000","last_active":"2024-06-29T14:38:25.867000+0000","last_peered":"2024-06-29T14:38:25.867000+0000","last_clean":"2024-06-29T14:38:25.867000+0000","last_became_active":"2024-06-29T14:38:24.874334+0000","last_became_peered":"2024-06-29T14:38:24.874334+0000","last_unstale":"2024-06-29T14:38:25.867000+0000","last_undegraded":"2024-06-29T14:38:25.867000+0000","last_fullsized":"2024-06-29T14:38:25.867000+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:00.507144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867450+0000","last_change":"2024-06-29T14:38:24.878758+0000","last_active":"2024-06-29T14:38:25.867450+0000","last_peered":"2024-06-29T14:38:25.867450+0000","last_clean":"2024-06-29T14:38:25.867450+0000","last_became_active":"2024-06-29T14:38:24.878456+0000","last_became_peered":"2024-06-29T14:38:24.878456+0000","last_unstale":"2024-06-29T14:38:25.867450+0000","last_undegraded":"2024-06-29T14:38:25.867450+0000","last_fullsized":"2024-06-29T14:38:25.867450+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:54:05.449634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868435+0000","last_change":"2024-06-29T14:38:24.887705+0000","last_active":"2024-06-29T14:38:25.868435+0000","last_peered":"2024-06-29T14:38:25.868435+0000","last_clean":"2024-06-29T14:38:25.868435+0000","last_became_active":"2024-06-29T14:38:24.887398+0000","last_became_peered":"2024-06-29T14:38:24.887398+0000","last_unstale":"2024-06-29T14:38:25.868435+0000","last_undegraded":"2024-06-29T14:38:25.868435+0000","last_fullsized":"2024-06-29T14:38:25.868435+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:20:53.669315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865218+0000","last_change":"2024-06-29T14:38:24.875948+0000","last_active":"2024-06-29T14:38:25.865218+0000","last_peered":"2024-06-29T14:38:25.865218+0000","last_clean":"2024-06-29T14:38:25.865218+0000","last_became_active":"2024-06-29T14:38:24.875639+0000","last_became_peered":"2024-06-29T14:38:24.875639+0000","last_unstale":"2024-06-29T14:38:25.865218+0000","last_undegraded":"2024-06-29T14:38:25.865218+0000","last_fullsized":"2024-06-29T14:38:25.865218+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:53.863238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868589+0000","last_change":"2024-06-29T14:38:24.889183+0000","last_active":"2024-06-29T14:38:25.868589+0000","last_peered":"2024-06-29T14:38:25.868589+0000","last_clean":"2024-06-29T14:38:25.868589+0000","last_became_active":"2024-06-29T14:38:24.888843+0000","last_became_peered":"2024-06-29T14:38:24.888843+0000","last_unstale":"2024-06-29T14:38:25.868589+0000","last_undegraded":"2024-06-29T14:38:25.868589+0000","last_fullsized":"2024-06-29T14:38:25.868589+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:09:27.946436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046149+0000","last_change":"2024-06-29T14:38:24.882021+0000","last_active":"2024-06-29T14:38:26.046149+0000","last_peered":"2024-06-29T14:38:26.046149+0000","last_clean":"2024-06-29T14:38:26.046149+0000","last_became_active":"2024-06-29T14:38:24.881682+0000","last_became_peered":"2024-06-29T14:38:24.881682+0000","last_unstale":"2024-06-29T14:38:26.046149+0000","last_undegraded":"2024-06-29T14:38:26.046149+0000","last_fullsized":"2024-06-29T14:38:26.046149+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:40:06.228389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044930+0000","last_change":"2024-06-29T14:38:24.883867+0000","last_active":"2024-06-29T14:38:26.044930+0000","last_peered":"2024-06-29T14:38:26.044930+0000","last_clean":"2024-06-29T14:38:26.044930+0000","last_became_active":"2024-06-29T14:38:24.883097+0000","last_became_peered":"2024-06-29T14:38:24.883097+0000","last_unstale":"2024-06-29T14:38:26.044930+0000","last_undegraded":"2024-06-29T14:38:26.044930+0000","last_fullsized":"2024-06-29T14:38:26.044930+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:53.319167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044828+0000","last_change":"2024-06-29T14:38:24.882354+0000","last_active":"2024-06-29T14:38:26.044828+0000","last_peered":"2024-06-29T14:38:26.044828+0000","last_clean":"2024-06-29T14:38:26.044828+0000","last_became_active":"2024-06-29T14:38:24.882071+0000","last_became_peered":"2024-06-29T14:38:24.882071+0000","last_unstale":"2024-06-29T14:38:26.044828+0000","last_undegraded":"2024-06-29T14:38:26.044828+0000","last_fullsized":"2024-06-29T14:38:26.044828+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:38:49.127792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045143+0000","last_change":"2024-06-29T14:38:24.889841+0000","last_active":"2024-06-29T14:38:26.045143+0000","last_peered":"2024-06-29T14:38:26.045143+0000","last_clean":"2024-06-29T14:38:26.045143+0000","last_became_active":"2024-06-29T14:38:24.889397+0000","last_became_peered":"2024-06-29T14:38:24.889397+0000","last_unstale":"2024-06-29T14:38:26.045143+0000","last_undegraded":"2024-06-29T14:38:26.045143+0000","last_fullsized":"2024-06-29T14:38:26.045143+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:43:56.256764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045651+0000","last_change":"2024-06-29T14:38:24.878218+0000","last_active":"2024-06-29T14:38:26.045651+0000","last_peered":"2024-06-29T14:38:26.045651+0000","last_clean":"2024-06-29T14:38:26.045651+0000","last_became_active":"2024-06-29T14:38:24.877846+0000","last_became_peered":"2024-06-29T14:38:24.877846+0000","last_unstale":"2024-06-29T14:38:26.045651+0000","last_undegraded":"2024-06-29T14:38:26.045651+0000","last_fullsized":"2024-06-29T14:38:26.045651+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:28:19.363714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045300+0000","last_change":"2024-06-29T14:38:24.887628+0000","last_active":"2024-06-29T14:38:26.045300+0000","last_peered":"2024-06-29T14:38:26.045300+0000","last_clean":"2024-06-29T14:38:26.045300+0000","last_became_active":"2024-06-29T14:38:24.887317+0000","last_became_peered":"2024-06-29T14:38:24.887317+0000","last_unstale":"2024-06-29T14:38:26.045300+0000","last_undegraded":"2024-06-29T14:38:26.045300+0000","last_fullsized":"2024-06-29T14:38:26.045300+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:53.863238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881427+0000","last_change":"2024-06-29T14:38:24.885786+0000","last_active":"2024-06-29T14:38:25.881427+0000","last_peered":"2024-06-29T14:38:25.881427+0000","last_clean":"2024-06-29T14:38:25.881427+0000","last_became_active":"2024-06-29T14:38:24.885416+0000","last_became_peered":"2024-06-29T14:38:24.885416+0000","last_unstale":"2024-06-29T14:38:25.881427+0000","last_undegraded":"2024-06-29T14:38:25.881427+0000","last_fullsized":"2024-06-29T14:38:25.881427+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:20:53.669315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045381+0000","last_change":"2024-06-29T14:38:24.882092+0000","last_active":"2024-06-29T14:38:26.045381+0000","last_peered":"2024-06-29T14:38:26.045381+0000","last_clean":"2024-06-29T14:38:26.045381+0000","last_became_active":"2024-06-29T14:38:24.881227+0000","last_became_peered":"2024-06-29T14:38:24.881227+0000","last_unstale":"2024-06-29T14:38:26.045381+0000","last_undegraded":"2024-06-29T14:38:26.045381+0000","last_fullsized":"2024-06-29T14:38:26.045381+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:16:34.915868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045057+0000","last_change":"2024-06-29T14:38:24.877861+0000","last_active":"2024-06-29T14:38:26.045057+0000","last_peered":"2024-06-29T14:38:26.045057+0000","last_clean":"2024-06-29T14:38:26.045057+0000","last_became_active":"2024-06-29T14:38:24.877536+0000","last_became_peered":"2024-06-29T14:38:24.877536+0000","last_unstale":"2024-06-29T14:38:26.045057+0000","last_undegraded":"2024-06-29T14:38:26.045057+0000","last_fullsized":"2024-06-29T14:38:26.045057+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:20.891486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865261+0000","last_change":"2024-06-29T14:38:24.877853+0000","last_active":"2024-06-29T14:38:25.865261+0000","last_peered":"2024-06-29T14:38:25.865261+0000","last_clean":"2024-06-29T14:38:25.865261+0000","last_became_active":"2024-06-29T14:38:24.877495+0000","last_became_peered":"2024-06-29T14:38:24.877495+0000","last_unstale":"2024-06-29T14:38:25.865261+0000","last_undegraded":"2024-06-29T14:38:25.865261+0000","last_fullsized":"2024-06-29T14:38:25.865261+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:56:41.084977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868159+0000","last_change":"2024-06-29T14:38:24.884777+0000","last_active":"2024-06-29T14:38:25.868159+0000","last_peered":"2024-06-29T14:38:25.868159+0000","last_clean":"2024-06-29T14:38:25.868159+0000","last_became_active":"2024-06-29T14:38:24.884371+0000","last_became_peered":"2024-06-29T14:38:24.884371+0000","last_unstale":"2024-06-29T14:38:25.868159+0000","last_undegraded":"2024-06-29T14:38:25.868159+0000","last_fullsized":"2024-06-29T14:38:25.868159+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:17.736423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044708+0000","last_change":"2024-06-29T14:38:24.872900+0000","last_active":"2024-06-29T14:38:26.044708+0000","last_peered":"2024-06-29T14:38:26.044708+0000","last_clean":"2024-06-29T14:38:26.044708+0000","last_became_active":"2024-06-29T14:38:24.872578+0000","last_became_peered":"2024-06-29T14:38:24.872578+0000","last_unstale":"2024-06-29T14:38:26.044708+0000","last_undegraded":"2024-06-29T14:38:26.044708+0000","last_fullsized":"2024-06-29T14:38:26.044708+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:39:34.883426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865311+0000","last_change":"2024-06-29T14:38:24.890265+0000","last_active":"2024-06-29T14:38:25.865311+0000","last_peered":"2024-06-29T14:38:25.865311+0000","last_clean":"2024-06-29T14:38:25.865311+0000","last_became_active":"2024-06-29T14:38:24.889933+0000","last_became_peered":"2024-06-29T14:38:24.889933+0000","last_unstale":"2024-06-29T14:38:25.865311+0000","last_undegraded":"2024-06-29T14:38:25.865311+0000","last_fullsized":"2024-06-29T14:38:25.865311+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:21.850352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045577+0000","last_change":"2024-06-29T14:38:24.882267+0000","last_active":"2024-06-29T14:38:26.045577+0000","last_peered":"2024-06-29T14:38:26.045577+0000","last_clean":"2024-06-29T14:38:26.045577+0000","last_became_active":"2024-06-29T14:38:24.881889+0000","last_became_peered":"2024-06-29T14:38:24.881889+0000","last_unstale":"2024-06-29T14:38:26.045577+0000","last_undegraded":"2024-06-29T14:38:26.045577+0000","last_fullsized":"2024-06-29T14:38:26.045577+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:05:11.829543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045877+0000","last_change":"2024-06-29T14:38:24.872736+0000","last_active":"2024-06-29T14:38:26.045877+0000","last_peered":"2024-06-29T14:38:26.045877+0000","last_clean":"2024-06-29T14:38:26.045877+0000","last_became_active":"2024-06-29T14:38:24.872408+0000","last_became_peered":"2024-06-29T14:38:24.872408+0000","last_unstale":"2024-06-29T14:38:26.045877+0000","last_undegraded":"2024-06-29T14:38:26.045877+0000","last_fullsized":"2024-06-29T14:38:26.045877+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:23:21.368593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868683+0000","last_change":"2024-06-29T14:38:24.893858+0000","last_active":"2024-06-29T14:38:25.868683+0000","last_peered":"2024-06-29T14:38:25.868683+0000","last_clean":"2024-06-29T14:38:25.868683+0000","last_became_active":"2024-06-29T14:38:24.893341+0000","last_became_peered":"2024-06-29T14:38:24.893341+0000","last_unstale":"2024-06-29T14:38:25.868683+0000","last_undegraded":"2024-06-29T14:38:25.868683+0000","last_fullsized":"2024-06-29T14:38:25.868683+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:19:49.232504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865464+0000","last_change":"2024-06-29T14:38:24.877965+0000","last_active":"2024-06-29T14:38:25.865464+0000","last_peered":"2024-06-29T14:38:25.865464+0000","last_clean":"2024-06-29T14:38:25.865464+0000","last_became_active":"2024-06-29T14:38:24.877474+0000","last_became_peered":"2024-06-29T14:38:24.877474+0000","last_unstale":"2024-06-29T14:38:25.865464+0000","last_undegraded":"2024-06-29T14:38:25.865464+0000","last_fullsized":"2024-06-29T14:38:25.865464+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:56.351425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865495+0000","last_change":"2024-06-29T14:38:24.882576+0000","last_active":"2024-06-29T14:38:25.865495+0000","last_peered":"2024-06-29T14:38:25.865495+0000","last_clean":"2024-06-29T14:38:25.865495+0000","last_became_active":"2024-06-29T14:38:24.882270+0000","last_became_peered":"2024-06-29T14:38:24.882270+0000","last_unstale":"2024-06-29T14:38:25.865495+0000","last_undegraded":"2024-06-29T14:38:25.865495+0000","last_fullsized":"2024-06-29T14:38:25.865495+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:19:50.943728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045243+0000","last_change":"2024-06-29T14:38:24.879841+0000","last_active":"2024-06-29T14:38:26.045243+0000","last_peered":"2024-06-29T14:38:26.045243+0000","last_clean":"2024-06-29T14:38:26.045243+0000","last_became_active":"2024-06-29T14:38:24.879479+0000","last_became_peered":"2024-06-29T14:38:24.879479+0000","last_unstale":"2024-06-29T14:38:26.045243+0000","last_undegraded":"2024-06-29T14:38:26.045243+0000","last_fullsized":"2024-06-29T14:38:26.045243+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:30.157313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868125+0000","last_change":"2024-06-29T14:38:24.885493+0000","last_active":"2024-06-29T14:38:25.868125+0000","last_peered":"2024-06-29T14:38:25.868125+0000","last_clean":"2024-06-29T14:38:25.868125+0000","last_became_active":"2024-06-29T14:38:24.884701+0000","last_became_peered":"2024-06-29T14:38:24.884701+0000","last_unstale":"2024-06-29T14:38:25.868125+0000","last_undegraded":"2024-06-29T14:38:25.868125+0000","last_fullsized":"2024-06-29T14:38:25.868125+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:26.298376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046168+0000","last_change":"2024-06-29T14:38:24.891981+0000","last_active":"2024-06-29T14:38:26.046168+0000","last_peered":"2024-06-29T14:38:26.046168+0000","last_clean":"2024-06-29T14:38:26.046168+0000","last_became_active":"2024-06-29T14:38:24.891615+0000","last_became_peered":"2024-06-29T14:38:24.891615+0000","last_unstale":"2024-06-29T14:38:26.046168+0000","last_undegraded":"2024-06-29T14:38:26.046168+0000","last_fullsized":"2024-06-29T14:38:26.046168+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:54.986737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"103'4","reported_seq":18,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868776+0000","last_change":"2024-06-29T14:38:23.869892+0000","last_active":"2024-06-29T14:38:25.868776+0000","last_peered":"2024-06-29T14:38:25.868776+0000","last_clean":"2024-06-29T14:38:25.868776+0000","last_became_active":"2024-06-29T14:38:23.869503+0000","last_became_peered":"2024-06-29T14:38:23.869503+0000","last_unstale":"2024-06-29T14:38:25.868776+0000","last_undegraded":"2024-06-29T14:38:25.868776+0000","last_fullsized":"2024-06-29T14:38:25.868776+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:01:22.462383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868585+0000","last_change":"2024-06-29T14:38:24.891255+0000","last_active":"2024-06-29T14:38:25.868585+0000","last_peered":"2024-06-29T14:38:25.868585+0000","last_clean":"2024-06-29T14:38:25.868585+0000","last_became_active":"2024-06-29T14:38:24.890949+0000","last_became_peered":"2024-06-29T14:38:24.890949+0000","last_unstale":"2024-06-29T14:38:25.868585+0000","last_undegraded":"2024-06-29T14:38:25.868585+0000","last_fullsized":"2024-06-29T14:38:25.868585+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:58:01.826665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"103'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046299+0000","last_change":"2024-06-29T14:38:23.861675+0000","last_active":"2024-06-29T14:38:26.046299+0000","last_peered":"2024-06-29T14:38:26.046299+0000","last_clean":"2024-06-29T14:38:26.046299+0000","last_became_active":"2024-06-29T14:38:23.861353+0000","last_became_peered":"2024-06-29T14:38:23.861353+0000","last_unstale":"2024-06-29T14:38:26.046299+0000","last_undegraded":"2024-06-29T14:38:26.046299+0000","last_fullsized":"2024-06-29T14:38:26.046299+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:13:02.491883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868790+0000","last_change":"2024-06-29T14:38:24.886169+0000","last_active":"2024-06-29T14:38:25.868790+0000","last_peered":"2024-06-29T14:38:25.868790+0000","last_clean":"2024-06-29T14:38:25.868790+0000","last_became_active":"2024-06-29T14:38:24.885816+0000","last_became_peered":"2024-06-29T14:38:24.885816+0000","last_unstale":"2024-06-29T14:38:25.868790+0000","last_undegraded":"2024-06-29T14:38:25.868790+0000","last_fullsized":"2024-06-29T14:38:25.868790+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:12:03.006708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881440+0000","last_change":"2024-06-29T14:38:23.858739+0000","last_active":"2024-06-29T14:38:25.881440+0000","last_peered":"2024-06-29T14:38:25.881440+0000","last_clean":"2024-06-29T14:38:25.881440+0000","last_became_active":"2024-06-29T14:38:23.858435+0000","last_became_peered":"2024-06-29T14:38:23.858435+0000","last_unstale":"2024-06-29T14:38:25.881440+0000","last_undegraded":"2024-06-29T14:38:25.881440+0000","last_fullsized":"2024-06-29T14:38:25.881440+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:56:40.077517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045683+0000","last_change":"2024-06-29T14:38:24.881961+0000","last_active":"2024-06-29T14:38:26.045683+0000","last_peered":"2024-06-29T14:38:26.045683+0000","last_clean":"2024-06-29T14:38:26.045683+0000","last_became_active":"2024-06-29T14:38:24.881548+0000","last_became_peered":"2024-06-29T14:38:24.881548+0000","last_unstale":"2024-06-29T14:38:26.045683+0000","last_undegraded":"2024-06-29T14:38:26.045683+0000","last_fullsized":"2024-06-29T14:38:26.045683+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:32.034964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881370+0000","last_change":"2024-06-29T14:38:23.860453+0000","last_active":"2024-06-29T14:38:25.881370+0000","last_peered":"2024-06-29T14:38:25.881370+0000","last_clean":"2024-06-29T14:38:25.881370+0000","last_became_active":"2024-06-29T14:38:23.860140+0000","last_became_peered":"2024-06-29T14:38:23.860140+0000","last_unstale":"2024-06-29T14:38:25.881370+0000","last_undegraded":"2024-06-29T14:38:25.881370+0000","last_fullsized":"2024-06-29T14:38:25.881370+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:19.884026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865571+0000","last_change":"2024-06-29T14:38:24.878860+0000","last_active":"2024-06-29T14:38:25.865571+0000","last_peered":"2024-06-29T14:38:25.865571+0000","last_clean":"2024-06-29T14:38:25.865571+0000","last_became_active":"2024-06-29T14:38:24.878533+0000","last_became_peered":"2024-06-29T14:38:24.878533+0000","last_unstale":"2024-06-29T14:38:25.865571+0000","last_undegraded":"2024-06-29T14:38:25.865571+0000","last_fullsized":"2024-06-29T14:38:25.865571+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:26.298376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"103'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046009+0000","last_change":"2024-06-29T14:38:23.863690+0000","last_active":"2024-06-29T14:38:26.046009+0000","last_peered":"2024-06-29T14:38:26.046009+0000","last_clean":"2024-06-29T14:38:26.046009+0000","last_became_active":"2024-06-29T14:38:23.863345+0000","last_became_peered":"2024-06-29T14:38:23.863345+0000","last_unstale":"2024-06-29T14:38:26.046009+0000","last_undegraded":"2024-06-29T14:38:26.046009+0000","last_fullsized":"2024-06-29T14:38:26.046009+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:24.011231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045526+0000","last_change":"2024-06-29T14:38:24.876900+0000","last_active":"2024-06-29T14:38:26.045526+0000","last_peered":"2024-06-29T14:38:26.045526+0000","last_clean":"2024-06-29T14:38:26.045526+0000","last_became_active":"2024-06-29T14:38:24.876603+0000","last_became_peered":"2024-06-29T14:38:24.876603+0000","last_unstale":"2024-06-29T14:38:26.045526+0000","last_undegraded":"2024-06-29T14:38:26.045526+0000","last_fullsized":"2024-06-29T14:38:26.045526+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:41:54.265980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"103'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045160+0000","last_change":"2024-06-29T14:38:23.862520+0000","last_active":"2024-06-29T14:38:26.045160+0000","last_peered":"2024-06-29T14:38:26.045160+0000","last_clean":"2024-06-29T14:38:26.045160+0000","last_became_active":"2024-06-29T14:38:23.862199+0000","last_became_peered":"2024-06-29T14:38:23.862199+0000","last_unstale":"2024-06-29T14:38:26.045160+0000","last_undegraded":"2024-06-29T14:38:26.045160+0000","last_fullsized":"2024-06-29T14:38:26.045160+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:12:01.999248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045369+0000","last_change":"2024-06-29T14:38:24.877423+0000","last_active":"2024-06-29T14:38:26.045369+0000","last_peered":"2024-06-29T14:38:26.045369+0000","last_clean":"2024-06-29T14:38:26.045369+0000","last_became_active":"2024-06-29T14:38:24.877069+0000","last_became_peered":"2024-06-29T14:38:24.877069+0000","last_unstale":"2024-06-29T14:38:26.045369+0000","last_undegraded":"2024-06-29T14:38:26.045369+0000","last_fullsized":"2024-06-29T14:38:26.045369+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:25.968927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881096+0000","last_change":"2024-06-29T14:38:23.864149+0000","last_active":"2024-06-29T14:38:25.881096+0000","last_peered":"2024-06-29T14:38:25.881096+0000","last_clean":"2024-06-29T14:38:25.881096+0000","last_became_active":"2024-06-29T14:38:23.863834+0000","last_became_peered":"2024-06-29T14:38:23.863834+0000","last_unstale":"2024-06-29T14:38:25.881096+0000","last_undegraded":"2024-06-29T14:38:25.881096+0000","last_fullsized":"2024-06-29T14:38:25.881096+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:52.311707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044935+0000","last_change":"2024-06-29T14:38:24.881598+0000","last_active":"2024-06-29T14:38:26.044935+0000","last_peered":"2024-06-29T14:38:26.044935+0000","last_clean":"2024-06-29T14:38:26.044935+0000","last_became_active":"2024-06-29T14:38:24.881289+0000","last_became_peered":"2024-06-29T14:38:24.881289+0000","last_unstale":"2024-06-29T14:38:26.044935+0000","last_undegraded":"2024-06-29T14:38:26.044935+0000","last_fullsized":"2024-06-29T14:38:26.044935+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:29.602452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865194+0000","last_change":"2024-06-29T14:38:23.862900+0000","last_active":"2024-06-29T14:38:25.865194+0000","last_peered":"2024-06-29T14:38:25.865194+0000","last_clean":"2024-06-29T14:38:25.865194+0000","last_became_active":"2024-06-29T14:38:23.862562+0000","last_became_peered":"2024-06-29T14:38:23.862562+0000","last_unstale":"2024-06-29T14:38:25.865194+0000","last_undegraded":"2024-06-29T14:38:25.865194+0000","last_fullsized":"2024-06-29T14:38:25.865194+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:57.893947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045357+0000","last_change":"2024-06-29T14:38:24.871083+0000","last_active":"2024-06-29T14:38:26.045357+0000","last_peered":"2024-06-29T14:38:26.045357+0000","last_clean":"2024-06-29T14:38:26.045357+0000","last_became_active":"2024-06-29T14:38:24.870762+0000","last_became_peered":"2024-06-29T14:38:24.870762+0000","last_unstale":"2024-06-29T14:38:26.045357+0000","last_undegraded":"2024-06-29T14:38:26.045357+0000","last_fullsized":"2024-06-29T14:38:26.045357+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:25.968927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868900+0000","last_change":"2024-06-29T14:38:23.864993+0000","last_active":"2024-06-29T14:38:25.868900+0000","last_peered":"2024-06-29T14:38:25.868900+0000","last_clean":"2024-06-29T14:38:25.868900+0000","last_became_active":"2024-06-29T14:38:23.863274+0000","last_became_peered":"2024-06-29T14:38:23.863274+0000","last_unstale":"2024-06-29T14:38:25.868900+0000","last_undegraded":"2024-06-29T14:38:25.868900+0000","last_fullsized":"2024-06-29T14:38:25.868900+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:39:33.875966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867492+0000","last_change":"2024-06-29T14:38:24.880283+0000","last_active":"2024-06-29T14:38:25.867492+0000","last_peered":"2024-06-29T14:38:25.867492+0000","last_clean":"2024-06-29T14:38:25.867492+0000","last_became_active":"2024-06-29T14:38:24.879943+0000","last_became_peered":"2024-06-29T14:38:24.879943+0000","last_unstale":"2024-06-29T14:38:25.867492+0000","last_undegraded":"2024-06-29T14:38:25.867492+0000","last_fullsized":"2024-06-29T14:38:25.867492+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:29.180183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868611+0000","last_change":"2024-06-29T14:38:23.865407+0000","last_active":"2024-06-29T14:38:25.868611+0000","last_peered":"2024-06-29T14:38:25.868611+0000","last_clean":"2024-06-29T14:38:25.868611+0000","last_became_active":"2024-06-29T14:38:23.865081+0000","last_became_peered":"2024-06-29T14:38:23.865081+0000","last_unstale":"2024-06-29T14:38:25.868611+0000","last_undegraded":"2024-06-29T14:38:25.868611+0000","last_fullsized":"2024-06-29T14:38:25.868611+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:39:36.586162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881314+0000","last_change":"2024-06-29T14:38:24.886685+0000","last_active":"2024-06-29T14:38:25.881314+0000","last_peered":"2024-06-29T14:38:25.881314+0000","last_clean":"2024-06-29T14:38:25.881314+0000","last_became_active":"2024-06-29T14:38:24.886319+0000","last_became_peered":"2024-06-29T14:38:24.886319+0000","last_unstale":"2024-06-29T14:38:25.881314+0000","last_undegraded":"2024-06-29T14:38:25.881314+0000","last_fullsized":"2024-06-29T14:38:25.881314+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:57.459300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"103'3","reported_seq":17,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868462+0000","last_change":"2024-06-29T14:38:23.866309+0000","last_active":"2024-06-29T14:38:25.868462+0000","last_peered":"2024-06-29T14:38:25.868462+0000","last_clean":"2024-06-29T14:38:25.868462+0000","last_became_active":"2024-06-29T14:38:23.866007+0000","last_became_peered":"2024-06-29T14:38:23.866007+0000","last_unstale":"2024-06-29T14:38:25.868462+0000","last_undegraded":"2024-06-29T14:38:25.868462+0000","last_fullsized":"2024-06-29T14:38:25.868462+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:00:10.828893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045734+0000","last_change":"2024-06-29T14:38:24.880224+0000","last_active":"2024-06-29T14:38:26.045734+0000","last_peered":"2024-06-29T14:38:26.045734+0000","last_clean":"2024-06-29T14:38:26.045734+0000","last_became_active":"2024-06-29T14:38:24.879888+0000","last_became_peered":"2024-06-29T14:38:24.879888+0000","last_unstale":"2024-06-29T14:38:26.045734+0000","last_undegraded":"2024-06-29T14:38:26.045734+0000","last_fullsized":"2024-06-29T14:38:26.045734+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:32:19.396631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"103'6","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867266+0000","last_change":"2024-06-29T14:38:23.859256+0000","last_active":"2024-06-29T14:38:25.867266+0000","last_peered":"2024-06-29T14:38:25.867266+0000","last_clean":"2024-06-29T14:38:25.867266+0000","last_became_active":"2024-06-29T14:38:23.858825+0000","last_became_peered":"2024-06-29T14:38:23.858825+0000","last_unstale":"2024-06-29T14:38:25.867266+0000","last_undegraded":"2024-06-29T14:38:25.867266+0000","last_fullsized":"2024-06-29T14:38:25.867266+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:54.265685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881249+0000","last_change":"2024-06-29T14:38:24.881099+0000","last_active":"2024-06-29T14:38:25.881249+0000","last_peered":"2024-06-29T14:38:25.881249+0000","last_clean":"2024-06-29T14:38:25.881249+0000","last_became_active":"2024-06-29T14:38:24.880538+0000","last_became_peered":"2024-06-29T14:38:24.880538+0000","last_unstale":"2024-06-29T14:38:25.881249+0000","last_undegraded":"2024-06-29T14:38:25.881249+0000","last_fullsized":"2024-06-29T14:38:25.881249+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:26.298376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865560+0000","last_change":"2024-06-29T14:38:23.859990+0000","last_active":"2024-06-29T14:38:25.865560+0000","last_peered":"2024-06-29T14:38:25.865560+0000","last_clean":"2024-06-29T14:38:25.865560+0000","last_became_active":"2024-06-29T14:38:23.859684+0000","last_became_peered":"2024-06-29T14:38:23.859684+0000","last_unstale":"2024-06-29T14:38:25.865560+0000","last_undegraded":"2024-06-29T14:38:25.865560+0000","last_fullsized":"2024-06-29T14:38:25.865560+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:44.845658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046258+0000","last_change":"2024-06-29T14:38:24.880308+0000","last_active":"2024-06-29T14:38:26.046258+0000","last_peered":"2024-06-29T14:38:26.046258+0000","last_clean":"2024-06-29T14:38:26.046258+0000","last_became_active":"2024-06-29T14:38:24.879979+0000","last_became_peered":"2024-06-29T14:38:24.879979+0000","last_unstale":"2024-06-29T14:38:26.046258+0000","last_undegraded":"2024-06-29T14:38:26.046258+0000","last_fullsized":"2024-06-29T14:38:26.046258+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:47:25.308681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045114+0000","last_change":"2024-06-29T14:38:23.864842+0000","last_active":"2024-06-29T14:38:26.045114+0000","last_peered":"2024-06-29T14:38:26.045114+0000","last_clean":"2024-06-29T14:38:26.045114+0000","last_became_active":"2024-06-29T14:38:23.864431+0000","last_became_peered":"2024-06-29T14:38:23.864431+0000","last_unstale":"2024-06-29T14:38:26.045114+0000","last_undegraded":"2024-06-29T14:38:26.045114+0000","last_fullsized":"2024-06-29T14:38:26.045114+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:06:05.845482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865091+0000","last_change":"2024-06-29T14:38:24.874217+0000","last_active":"2024-06-29T14:38:25.865091+0000","last_peered":"2024-06-29T14:38:25.865091+0000","last_clean":"2024-06-29T14:38:25.865091+0000","last_became_active":"2024-06-29T14:38:24.873913+0000","last_became_peered":"2024-06-29T14:38:24.873913+0000","last_unstale":"2024-06-29T14:38:25.865091+0000","last_undegraded":"2024-06-29T14:38:25.865091+0000","last_fullsized":"2024-06-29T14:38:25.865091+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:20.891486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867006+0000","last_change":"2024-06-29T14:38:23.861597+0000","last_active":"2024-06-29T14:38:25.867006+0000","last_peered":"2024-06-29T14:38:25.867006+0000","last_clean":"2024-06-29T14:38:25.867006+0000","last_became_active":"2024-06-29T14:38:23.861290+0000","last_became_peered":"2024-06-29T14:38:23.861290+0000","last_unstale":"2024-06-29T14:38:25.867006+0000","last_undegraded":"2024-06-29T14:38:25.867006+0000","last_fullsized":"2024-06-29T14:38:25.867006+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:35:09.672272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865378+0000","last_change":"2024-06-29T14:38:24.887998+0000","last_active":"2024-06-29T14:38:25.865378+0000","last_peered":"2024-06-29T14:38:25.865378+0000","last_clean":"2024-06-29T14:38:25.865378+0000","last_became_active":"2024-06-29T14:38:24.887703+0000","last_became_peered":"2024-06-29T14:38:24.887703+0000","last_unstale":"2024-06-29T14:38:25.865378+0000","last_undegraded":"2024-06-29T14:38:25.865378+0000","last_fullsized":"2024-06-29T14:38:25.865378+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:33:13.047537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"103'8","reported_seq":22,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881054+0000","last_change":"2024-06-29T14:38:23.864330+0000","last_active":"2024-06-29T14:38:25.881054+0000","last_peered":"2024-06-29T14:38:25.881054+0000","last_clean":"2024-06-29T14:38:25.881054+0000","last_became_active":"2024-06-29T14:38:23.863064+0000","last_became_peered":"2024-06-29T14:38:23.863064+0000","last_unstale":"2024-06-29T14:38:25.881054+0000","last_undegraded":"2024-06-29T14:38:25.881054+0000","last_fullsized":"2024-06-29T14:38:25.881054+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:04.726969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045749+0000","last_change":"2024-06-29T14:38:24.883075+0000","last_active":"2024-06-29T14:38:26.045749+0000","last_peered":"2024-06-29T14:38:26.045749+0000","last_clean":"2024-06-29T14:38:26.045749+0000","last_became_active":"2024-06-29T14:38:24.882674+0000","last_became_peered":"2024-06-29T14:38:24.882674+0000","last_unstale":"2024-06-29T14:38:26.045749+0000","last_undegraded":"2024-06-29T14:38:26.045749+0000","last_fullsized":"2024-06-29T14:38:26.045749+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:49.579212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"103'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868981+0000","last_change":"2024-06-29T14:38:23.860074+0000","last_active":"2024-06-29T14:38:25.868981+0000","last_peered":"2024-06-29T14:38:25.868981+0000","last_clean":"2024-06-29T14:38:25.868981+0000","last_became_active":"2024-06-29T14:38:23.859730+0000","last_became_peered":"2024-06-29T14:38:23.859730+0000","last_unstale":"2024-06-29T14:38:25.868981+0000","last_undegraded":"2024-06-29T14:38:25.868981+0000","last_fullsized":"2024-06-29T14:38:25.868981+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:30.988693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868828+0000","last_change":"2024-06-29T14:38:24.889550+0000","last_active":"2024-06-29T14:38:25.868828+0000","last_peered":"2024-06-29T14:38:25.868828+0000","last_clean":"2024-06-29T14:38:25.868828+0000","last_became_active":"2024-06-29T14:38:24.889270+0000","last_became_peered":"2024-06-29T14:38:24.889270+0000","last_unstale":"2024-06-29T14:38:25.868828+0000","last_undegraded":"2024-06-29T14:38:25.868828+0000","last_fullsized":"2024-06-29T14:38:25.868828+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:21:59.608621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881518+0000","last_change":"2024-06-29T14:38:23.863982+0000","last_active":"2024-06-29T14:38:25.881518+0000","last_peered":"2024-06-29T14:38:25.881518+0000","last_clean":"2024-06-29T14:38:25.881518+0000","last_became_active":"2024-06-29T14:38:23.863705+0000","last_became_peered":"2024-06-29T14:38:23.863705+0000","last_unstale":"2024-06-29T14:38:25.881518+0000","last_undegraded":"2024-06-29T14:38:25.881518+0000","last_fullsized":"2024-06-29T14:38:25.881518+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:16:33.908408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045968+0000","last_change":"2024-06-29T14:38:24.870357+0000","last_active":"2024-06-29T14:38:26.045968+0000","last_peered":"2024-06-29T14:38:26.045968+0000","last_clean":"2024-06-29T14:38:26.045968+0000","last_became_active":"2024-06-29T14:38:24.870004+0000","last_became_peered":"2024-06-29T14:38:24.870004+0000","last_unstale":"2024-06-29T14:38:26.045968+0000","last_undegraded":"2024-06-29T14:38:26.045968+0000","last_fullsized":"2024-06-29T14:38:26.045968+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:43:21.820955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881468+0000","last_change":"2024-06-29T14:38:23.861238+0000","last_active":"2024-06-29T14:38:25.881468+0000","last_peered":"2024-06-29T14:38:25.881468+0000","last_clean":"2024-06-29T14:38:25.881468+0000","last_became_active":"2024-06-29T14:38:23.860831+0000","last_became_peered":"2024-06-29T14:38:23.860831+0000","last_unstale":"2024-06-29T14:38:25.881468+0000","last_undegraded":"2024-06-29T14:38:25.881468+0000","last_fullsized":"2024-06-29T14:38:25.881468+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:55.343965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045333+0000","last_change":"2024-06-29T14:38:24.874105+0000","last_active":"2024-06-29T14:38:26.045333+0000","last_peered":"2024-06-29T14:38:26.045333+0000","last_clean":"2024-06-29T14:38:26.045333+0000","last_became_active":"2024-06-29T14:38:24.873820+0000","last_became_peered":"2024-06-29T14:38:24.873820+0000","last_unstale":"2024-06-29T14:38:26.045333+0000","last_undegraded":"2024-06-29T14:38:26.045333+0000","last_fullsized":"2024-06-29T14:38:26.045333+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:56:41.084977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045473+0000","last_change":"2024-06-29T14:38:23.870453+0000","last_active":"2024-06-29T14:38:26.045473+0000","last_peered":"2024-06-29T14:38:26.045473+0000","last_clean":"2024-06-29T14:38:26.045473+0000","last_became_active":"2024-06-29T14:38:23.870126+0000","last_became_peered":"2024-06-29T14:38:23.870126+0000","last_unstale":"2024-06-29T14:38:26.045473+0000","last_undegraded":"2024-06-29T14:38:26.045473+0000","last_fullsized":"2024-06-29T14:38:26.045473+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:20.908520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867126+0000","last_change":"2024-06-29T14:38:24.875963+0000","last_active":"2024-06-29T14:38:25.867126+0000","last_peered":"2024-06-29T14:38:25.867126+0000","last_clean":"2024-06-29T14:38:25.867126+0000","last_became_active":"2024-06-29T14:38:24.875616+0000","last_became_peered":"2024-06-29T14:38:24.875616+0000","last_unstale":"2024-06-29T14:38:25.867126+0000","last_undegraded":"2024-06-29T14:38:25.867126+0000","last_fullsized":"2024-06-29T14:38:25.867126+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:15.998851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045052+0000","last_change":"2024-06-29T14:38:23.867352+0000","last_active":"2024-06-29T14:38:26.045052+0000","last_peered":"2024-06-29T14:38:26.045052+0000","last_clean":"2024-06-29T14:38:26.045052+0000","last_became_active":"2024-06-29T14:38:23.866429+0000","last_became_peered":"2024-06-29T14:38:23.866429+0000","last_unstale":"2024-06-29T14:38:26.045052+0000","last_undegraded":"2024-06-29T14:38:26.045052+0000","last_fullsized":"2024-06-29T14:38:26.045052+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:52.183288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044751+0000","last_change":"2024-06-29T14:38:24.881521+0000","last_active":"2024-06-29T14:38:26.044751+0000","last_peered":"2024-06-29T14:38:26.044751+0000","last_clean":"2024-06-29T14:38:26.044751+0000","last_became_active":"2024-06-29T14:38:24.881215+0000","last_became_peered":"2024-06-29T14:38:24.881215+0000","last_unstale":"2024-06-29T14:38:26.044751+0000","last_undegraded":"2024-06-29T14:38:26.044751+0000","last_fullsized":"2024-06-29T14:38:26.044751+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:39:05.734429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044810+0000","last_change":"2024-06-29T14:38:23.866866+0000","last_active":"2024-06-29T14:38:26.044810+0000","last_peered":"2024-06-29T14:38:26.044810+0000","last_clean":"2024-06-29T14:38:26.044810+0000","last_became_active":"2024-06-29T14:38:23.866561+0000","last_became_peered":"2024-06-29T14:38:23.866561+0000","last_unstale":"2024-06-29T14:38:26.044810+0000","last_undegraded":"2024-06-29T14:38:26.044810+0000","last_fullsized":"2024-06-29T14:38:26.044810+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:00:10.239535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867104+0000","last_change":"2024-06-29T14:38:24.878172+0000","last_active":"2024-06-29T14:38:25.867104+0000","last_peered":"2024-06-29T14:38:25.867104+0000","last_clean":"2024-06-29T14:38:25.867104+0000","last_became_active":"2024-06-29T14:38:24.877839+0000","last_became_peered":"2024-06-29T14:38:24.877839+0000","last_unstale":"2024-06-29T14:38:25.867104+0000","last_undegraded":"2024-06-29T14:38:25.867104+0000","last_fullsized":"2024-06-29T14:38:25.867104+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:57:01.590805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.044581+0000","last_change":"2024-06-29T14:38:23.870477+0000","last_active":"2024-06-29T14:38:26.044581+0000","last_peered":"2024-06-29T14:38:26.044581+0000","last_clean":"2024-06-29T14:38:26.044581+0000","last_became_active":"2024-06-29T14:38:23.870172+0000","last_became_peered":"2024-06-29T14:38:23.870172+0000","last_unstale":"2024-06-29T14:38:26.044581+0000","last_undegraded":"2024-06-29T14:38:26.044581+0000","last_fullsized":"2024-06-29T14:38:26.044581+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:42.862786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046169+0000","last_change":"2024-06-29T14:38:24.870586+0000","last_active":"2024-06-29T14:38:26.046169+0000","last_peered":"2024-06-29T14:38:26.046169+0000","last_clean":"2024-06-29T14:38:26.046169+0000","last_became_active":"2024-06-29T14:38:24.869888+0000","last_became_peered":"2024-06-29T14:38:24.869888+0000","last_unstale":"2024-06-29T14:38:26.046169+0000","last_undegraded":"2024-06-29T14:38:26.046169+0000","last_fullsized":"2024-06-29T14:38:26.046169+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:51.649114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"103'2","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865375+0000","last_change":"2024-06-29T14:38:23.862060+0000","last_active":"2024-06-29T14:38:25.865375+0000","last_peered":"2024-06-29T14:38:25.865375+0000","last_clean":"2024-06-29T14:38:25.865375+0000","last_became_active":"2024-06-29T14:38:23.861757+0000","last_became_peered":"2024-06-29T14:38:23.861757+0000","last_unstale":"2024-06-29T14:38:25.865375+0000","last_undegraded":"2024-06-29T14:38:25.865375+0000","last_fullsized":"2024-06-29T14:38:25.865375+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:56:52.183288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045993+0000","last_change":"2024-06-29T14:38:24.893919+0000","last_active":"2024-06-29T14:38:26.045993+0000","last_peered":"2024-06-29T14:38:26.045993+0000","last_clean":"2024-06-29T14:38:26.045993+0000","last_became_active":"2024-06-29T14:38:24.893427+0000","last_became_peered":"2024-06-29T14:38:24.893427+0000","last_unstale":"2024-06-29T14:38:26.045993+0000","last_undegraded":"2024-06-29T14:38:26.045993+0000","last_fullsized":"2024-06-29T14:38:26.045993+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:14.728843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"104'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.880381+0000","last_change":"2024-06-29T14:38:23.860797+0000","last_active":"2024-06-29T14:38:25.880381+0000","last_peered":"2024-06-29T14:38:25.880381+0000","last_clean":"2024-06-29T14:38:25.880381+0000","last_became_active":"2024-06-29T14:38:23.860469+0000","last_became_peered":"2024-06-29T14:38:23.860469+0000","last_unstale":"2024-06-29T14:38:25.880381+0000","last_undegraded":"2024-06-29T14:38:25.880381+0000","last_fullsized":"2024-06-29T14:38:25.880381+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:31:37.100494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868932+0000","last_change":"2024-06-29T14:38:24.886902+0000","last_active":"2024-06-29T14:38:25.868932+0000","last_peered":"2024-06-29T14:38:25.868932+0000","last_clean":"2024-06-29T14:38:25.868932+0000","last_became_active":"2024-06-29T14:38:24.886526+0000","last_became_peered":"2024-06-29T14:38:24.886526+0000","last_unstale":"2024-06-29T14:38:25.868932+0000","last_undegraded":"2024-06-29T14:38:25.868932+0000","last_fullsized":"2024-06-29T14:38:25.868932+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:13:03.499343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868792+0000","last_change":"2024-06-29T14:38:23.869524+0000","last_active":"2024-06-29T14:38:25.868792+0000","last_peered":"2024-06-29T14:38:25.868792+0000","last_clean":"2024-06-29T14:38:25.868792+0000","last_became_active":"2024-06-29T14:38:23.869285+0000","last_became_peered":"2024-06-29T14:38:23.869285+0000","last_unstale":"2024-06-29T14:38:25.868792+0000","last_undegraded":"2024-06-29T14:38:25.868792+0000","last_fullsized":"2024-06-29T14:38:25.868792+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:29:24.526496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.046110+0000","last_change":"2024-06-29T14:38:24.881811+0000","last_active":"2024-06-29T14:38:26.046110+0000","last_peered":"2024-06-29T14:38:26.046110+0000","last_clean":"2024-06-29T14:38:26.046110+0000","last_became_active":"2024-06-29T14:38:24.880561+0000","last_became_peered":"2024-06-29T14:38:24.880561+0000","last_unstale":"2024-06-29T14:38:26.046110+0000","last_undegraded":"2024-06-29T14:38:26.046110+0000","last_fullsized":"2024-06-29T14:38:26.046110+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:13.291919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868931+0000","last_change":"2024-06-29T14:38:23.861543+0000","last_active":"2024-06-29T14:38:25.868931+0000","last_peered":"2024-06-29T14:38:25.868931+0000","last_clean":"2024-06-29T14:38:25.868931+0000","last_became_active":"2024-06-29T14:38:23.861172+0000","last_became_peered":"2024-06-29T14:38:23.861172+0000","last_unstale":"2024-06-29T14:38:25.868931+0000","last_undegraded":"2024-06-29T14:38:25.868931+0000","last_fullsized":"2024-06-29T14:38:25.868931+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:46:02.500746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.867348+0000","last_change":"2024-06-29T14:38:24.876023+0000","last_active":"2024-06-29T14:38:25.867348+0000","last_peered":"2024-06-29T14:38:25.867348+0000","last_clean":"2024-06-29T14:38:25.867348+0000","last_became_active":"2024-06-29T14:38:24.875716+0000","last_became_peered":"2024-06-29T14:38:24.875716+0000","last_unstale":"2024-06-29T14:38:25.867348+0000","last_undegraded":"2024-06-29T14:38:25.867348+0000","last_fullsized":"2024-06-29T14:38:25.867348+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:59.570624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881426+0000","last_change":"2024-06-29T14:38:23.863495+0000","last_active":"2024-06-29T14:38:25.881426+0000","last_peered":"2024-06-29T14:38:25.881426+0000","last_clean":"2024-06-29T14:38:25.881426+0000","last_became_active":"2024-06-29T14:38:23.863135+0000","last_became_peered":"2024-06-29T14:38:23.863135+0000","last_unstale":"2024-06-29T14:38:25.881426+0000","last_undegraded":"2024-06-29T14:38:25.881426+0000","last_fullsized":"2024-06-29T14:38:25.881426+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:52.855778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.881257+0000","last_change":"2024-06-29T14:38:24.883262+0000","last_active":"2024-06-29T14:38:25.881257+0000","last_peered":"2024-06-29T14:38:25.881257+0000","last_clean":"2024-06-29T14:38:25.881257+0000","last_became_active":"2024-06-29T14:38:24.882492+0000","last_became_peered":"2024-06-29T14:38:24.882492+0000","last_unstale":"2024-06-29T14:38:25.881257+0000","last_undegraded":"2024-06-29T14:38:25.881257+0000","last_fullsized":"2024-06-29T14:38:25.881257+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:17.736423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045280+0000","last_change":"2024-06-29T14:38:23.864312+0000","last_active":"2024-06-29T14:38:26.045280+0000","last_peered":"2024-06-29T14:38:26.045280+0000","last_clean":"2024-06-29T14:38:26.045280+0000","last_became_active":"2024-06-29T14:38:23.863953+0000","last_became_peered":"2024-06-29T14:38:23.863953+0000","last_unstale":"2024-06-29T14:38:26.045280+0000","last_undegraded":"2024-06-29T14:38:26.045280+0000","last_fullsized":"2024-06-29T14:38:26.045280+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:21:58.601161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045591+0000","last_change":"2024-06-29T14:38:24.881400+0000","last_active":"2024-06-29T14:38:26.045591+0000","last_peered":"2024-06-29T14:38:26.045591+0000","last_clean":"2024-06-29T14:38:26.045591+0000","last_became_active":"2024-06-29T14:38:24.881103+0000","last_became_peered":"2024-06-29T14:38:24.881103+0000","last_unstale":"2024-06-29T14:38:26.045591+0000","last_undegraded":"2024-06-29T14:38:26.045591+0000","last_fullsized":"2024-06-29T14:38:26.045591+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:17:22.571127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868360+0000","last_change":"2024-06-29T14:38:23.864138+0000","last_active":"2024-06-29T14:38:25.868360+0000","last_peered":"2024-06-29T14:38:25.868360+0000","last_clean":"2024-06-29T14:38:25.868360+0000","last_became_active":"2024-06-29T14:38:23.863429+0000","last_became_peered":"2024-06-29T14:38:23.863429+0000","last_unstale":"2024-06-29T14:38:25.868360+0000","last_undegraded":"2024-06-29T14:38:25.868360+0000","last_fullsized":"2024-06-29T14:38:25.868360+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:38:23.831509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.868221+0000","last_change":"2024-06-29T14:38:24.884561+0000","last_active":"2024-06-29T14:38:25.868221+0000","last_peered":"2024-06-29T14:38:25.868221+0000","last_clean":"2024-06-29T14:38:25.868221+0000","last_became_active":"2024-06-29T14:38:24.884185+0000","last_became_peered":"2024-06-29T14:38:24.884185+0000","last_unstale":"2024-06-29T14:38:25.868221+0000","last_undegraded":"2024-06-29T14:38:25.868221+0000","last_fullsized":"2024-06-29T14:38:25.868221+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:57.459300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045142+0000","last_change":"2024-06-29T14:38:23.865707+0000","last_active":"2024-06-29T14:38:26.045142+0000","last_peered":"2024-06-29T14:38:26.045142+0000","last_clean":"2024-06-29T14:38:26.045142+0000","last_became_active":"2024-06-29T14:38:23.865398+0000","last_became_peered":"2024-06-29T14:38:23.865398+0000","last_unstale":"2024-06-29T14:38:26.045142+0000","last_undegraded":"2024-06-29T14:38:26.045142+0000","last_fullsized":"2024-06-29T14:38:26.045142+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:43:46.596358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:25.865300+0000","last_change":"2024-06-29T14:38:24.876616+0000","last_active":"2024-06-29T14:38:25.865300+0000","last_peered":"2024-06-29T14:38:25.865300+0000","last_clean":"2024-06-29T14:38:25.865300+0000","last_became_active":"2024-06-29T14:38:24.875729+0000","last_became_peered":"2024-06-29T14:38:24.875729+0000","last_unstale":"2024-06-29T14:38:25.865300+0000","last_undegraded":"2024-06-29T14:38:25.865300+0000","last_fullsized":"2024-06-29T14:38:25.865300+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:23.841135+0000","last_clean_scrub_stamp":"2024-06-29T14:38:23.841135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:16:34.915868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"103'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045773+0000","last_change":"2024-06-29T14:38:23.867208+0000","last_active":"2024-06-29T14:38:26.045773+0000","last_peered":"2024-06-29T14:38:26.045773+0000","last_clean":"2024-06-29T14:38:26.045773+0000","last_became_active":"2024-06-29T14:38:23.866521+0000","last_became_peered":"2024-06-29T14:38:23.866521+0000","last_unstale":"2024-06-29T14:38:26.045773+0000","last_undegraded":"2024-06-29T14:38:26.045773+0000","last_fullsized":"2024-06-29T14:38:26.045773+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:38:22.833675+0000","last_clean_scrub_stamp":"2024-06-29T14:38:22.833675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:11.028911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"93'114","reported_seq":423,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-06-29T14:38:26.045667+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:38:26.045667+0000","last_peered":"2024-06-29T14:38:26.045667+0000","last_clean":"2024-06-29T14:38:26.045667+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:38:26.045667+0000","last_undegraded":"2024-06-29T14:38:26.045667+0000","last_fullsized":"2024-06-29T14:38:26.045667+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":173,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":426,"num_read_kb":1111,"num_write":173,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575042,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1863680,"data_stored":1194664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575043,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1277952,"data_stored":604920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36592,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335248,"statfs":{"total":96636764160,"available":96599293952,"internally_reserved":0,"allocated":1282048,"data_stored":605860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575042,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58804,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313036,"statfs":{"total":96636764160,"available":96576548864,"internally_reserved":0,"allocated":1277952,"data_stored":610598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575042,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36596,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335244,"statfs":{"total":96636764160,"available":96599289856,"internally_reserved":0,"allocated":1277952,"data_stored":605838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575042,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36592,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335248,"statfs":{"total":96636764160,"available":96599293952,"internally_reserved":0,"allocated":1282048,"data_stored":611180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50492,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321348,"statfs":{"total":96636764160,"available":96585060352,"internally_reserved":0,"allocated":1290240,"data_stored":610792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575042,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1875968,"data_stored":1201000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:38:29.265 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:38:29.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:38:29.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.426+0000 7f8defc3c640 1 -- 172.21.15.81:0/3533203609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dd0004800 msgr2=0x7f8dd0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.426+0000 7f8defc3c640 1 --2- 172.21.15.81:0/3533203609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dd0004800 0x7f8dd0004c10 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc006f90 tx=0x7f8dd0005830 comp rx=0 tx=0).stop 2024-06-29T14:38:29.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.426+0000 7f8dee1d9640 1 -- 172.21.15.81:0/3533203609 reap_dead start 2024-06-29T14:38:29.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.426+0000 7f8defc3c640 1 -- 172.21.15.81:0/3533203609 shutdown_connections 2024-06-29T14:38:29.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.426+0000 7f8defc3c640 1 -- 172.21.15.81:0/3533203609 >> 172.21.15.81:0/3533203609 conn(0x7f8de80d0930 msgr2=0x7f8de80d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:29.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.427+0000 7f8defc3c640 1 -- 172.21.15.81:0/3533203609 shutdown_connections 2024-06-29T14:38:29.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.427+0000 7f8defc3c640 1 -- 172.21.15.81:0/3533203609 wait complete. 2024-06-29T14:38:29.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.428+0000 7f8defc3c640 1 Processor -- start 2024-06-29T14:38:29.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.428+0000 7f8defc3c640 1 -- start start 2024-06-29T14:38:29.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8defc3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dd0004800 0x7f8de81530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8defc3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8de813b9e0 0x7f8de80b0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8defc3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8de80b0ea0 0x7f8de81518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8defc3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8de806f230 con 0x7f8de813b9e0 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8defc3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8de80b50a0 con 0x7f8de80b0ea0 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8defc3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8de806f3b0 con 0x7f8dd0004800 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded1d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8de813b9e0 0x7f8de80b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded1d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8de813b9e0 0x7f8de80b0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38770/0 (socket says 172.21.15.81:38770) 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded1d7640 1 -- 172.21.15.81:0/2345660385 learned_addr learned my addr 172.21.15.81:0/2345660385 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8dee1d9640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8de80b0ea0 0x7f8de81518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded9d8640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dd0004800 0x7f8de81530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded1d7640 1 -- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dd0004800 msgr2=0x7f8de81530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded1d7640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dd0004800 0x7f8de81530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded1d7640 1 -- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8de80b0ea0 msgr2=0x7f8de81518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded1d7640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8de80b0ea0 0x7f8de81518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded1d7640 1 -- 172.21.15.81:0/2345660385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de80b12b0 con 0x7f8de813b9e0 2024-06-29T14:38:29.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8dee1d9640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8de80b0ea0 0x7f8de81518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:29.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded9d8640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dd0004800 0x7f8de81530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:29.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8ded1d7640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8de813b9e0 0x7f8de80b0960 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc031040 tx=0x7f8ddc006ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8de67fc640 1 -- 172.21.15.81:0/2345660385 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc032030 con 0x7f8de813b9e0 2024-06-29T14:38:29.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8de67fc640 1 -- 172.21.15.81:0/2345660385 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ddc0038e0 con 0x7f8de813b9e0 2024-06-29T14:38:29.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8defc3c640 1 -- 172.21.15.81:0/2345660385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de8153b60 con 0x7f8de813b9e0 2024-06-29T14:38:29.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8de67fc640 1 -- 172.21.15.81:0/2345660385 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc003d10 con 0x7f8de813b9e0 2024-06-29T14:38:29.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.430+0000 7f8defc3c640 1 -- 172.21.15.81:0/2345660385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de8154120 con 0x7f8de813b9e0 2024-06-29T14:38:29.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.431+0000 7f8defc3c640 1 -- 172.21.15.81:0/2345660385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de80ad990 con 0x7f8de813b9e0 2024-06-29T14:38:29.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.432+0000 7f8de67fc640 1 -- 172.21.15.81:0/2345660385 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f8ddc035020 con 0x7f8de813b9e0 2024-06-29T14:38:29.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.433+0000 7f8de67fc640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8dd40763f0 0x7f8dd4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.433+0000 7f8de67fc640 1 -- 172.21.15.81:0/2345660385 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f8ddc0b7e00 con 0x7f8de813b9e0 2024-06-29T14:38:29.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.433+0000 7f8ded9d8640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8dd40763f0 0x7f8dd4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.434+0000 7f8ded9d8640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8dd40763f0 0x7f8dd4078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8005fd0 tx=0x7f8dd8009500 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.437+0000 7f8de67fc640 1 -- 172.21.15.81:0/2345660385 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8de80ad990 con 0x7f8de813b9e0 2024-06-29T14:38:29.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.686+0000 7f8defc3c640 1 -- 172.21.15.81:0/2345660385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8de81538f0 con 0x7f8de813b9e0 2024-06-29T14:38:29.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.689+0000 7f8de67fc640 1 -- 172.21.15.81:0/2345660385 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v34) ==== 190+0+147 (secure 0 0 0) 0x7f8ddc047030 con 0x7f8de813b9e0 2024-06-29T14:38:29.686 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T14:38:29.686 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T14:38:29.686 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T14:38:29.686 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T14:38:29.687 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:38:29.687 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T14:38:29.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 -- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8dd40763f0 msgr2=0x7f8dd4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8dd40763f0 0x7f8dd4078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8005fd0 tx=0x7f8dd8009500 comp rx=0 tx=0).stop 2024-06-29T14:38:29.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 -- 172.21.15.81:0/2345660385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8de813b9e0 msgr2=0x7f8de80b0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8de813b9e0 0x7f8de80b0960 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc031040 tx=0x7f8ddc006ce0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 -- 172.21.15.81:0/2345660385 shutdown_connections 2024-06-29T14:38:29.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8dd40763f0 0x7f8dd4078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8de80b0ea0 0x7f8de81518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8de813b9e0 0x7f8de80b0960 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 --2- 172.21.15.81:0/2345660385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dd0004800 0x7f8de81530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 -- 172.21.15.81:0/2345660385 >> 172.21.15.81:0/2345660385 conn(0x7f8de80d0930 msgr2=0x7f8de80ccad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:29.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 -- 172.21.15.81:0/2345660385 shutdown_connections 2024-06-29T14:38:29.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.691+0000 7f8defc3c640 1 -- 172.21.15.81:0/2345660385 wait complete. 2024-06-29T14:38:29.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:38:29.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.816+0000 7ff55561f640 1 -- 172.21.15.81:0/2609777217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff53c003a00 msgr2=0x7ff53c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.816+0000 7ff55561f640 1 --2- 172.21.15.81:0/2609777217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff53c003a00 0x7ff53c003e70 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7ff540009b50 tx=0x7ff53c005830 comp rx=0 tx=0).stop 2024-06-29T14:38:29.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.816+0000 7ff54f7fe640 1 -- 172.21.15.81:0/2609777217 reap_dead start 2024-06-29T14:38:29.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.816+0000 7ff55561f640 1 -- 172.21.15.81:0/2609777217 shutdown_connections 2024-06-29T14:38:29.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.816+0000 7ff55561f640 1 -- 172.21.15.81:0/2609777217 >> 172.21.15.81:0/2609777217 conn(0x7ff55009cf30 msgr2=0x7ff5500d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:29.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.816+0000 7ff55561f640 1 -- 172.21.15.81:0/2609777217 shutdown_connections 2024-06-29T14:38:29.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.816+0000 7ff55561f640 1 -- 172.21.15.81:0/2609777217 wait complete. 2024-06-29T14:38:29.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.818+0000 7ff55561f640 1 Processor -- start 2024-06-29T14:38:29.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.818+0000 7ff55561f640 1 -- start start 2024-06-29T14:38:29.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff55561f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff53c003a00 0x7ff5501f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff55561f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5500cb940 0x7ff5501e7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff55561f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5501e7e80 0x7ff55015c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff55561f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff5500d09c0 con 0x7ff5501e7e80 2024-06-29T14:38:29.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff55561f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff5500d2d80 con 0x7ff5500cb940 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff55561f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff5500d0b40 con 0x7ff53c003a00 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff54f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5501e7e80 0x7ff55015c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff54f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5501e7e80 0x7ff55015c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38794/0 (socket says 172.21.15.81:38794) 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff54f7fe640 1 -- 172.21.15.81:0/2904447864 learned_addr learned my addr 172.21.15.81:0/2904447864 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff547fff640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5500cb940 0x7ff5501e7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff54effd640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff53c003a00 0x7ff5501f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff54f7fe640 1 -- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff53c003a00 msgr2=0x7ff5501f45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff54f7fe640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff53c003a00 0x7ff5501f45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff54f7fe640 1 -- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5500cb940 msgr2=0x7ff5501e7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff54f7fe640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5500cb940 0x7ff5501e7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.819+0000 7ff54f7fe640 1 -- 172.21.15.81:0/2904447864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5501e8670 con 0x7ff5501e7e80 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.820+0000 7ff54f7fe640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5501e7e80 0x7ff55015c9e0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ff548002230 tx=0x7ff548006790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.820+0000 7ff5477fe640 1 -- 172.21.15.81:0/2904447864 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54800fc30 con 0x7ff5501e7e80 2024-06-29T14:38:29.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.820+0000 7ff5477fe640 1 -- 172.21.15.81:0/2904447864 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff548018070 con 0x7ff5501e7e80 2024-06-29T14:38:29.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.820+0000 7ff5477fe640 1 -- 172.21.15.81:0/2904447864 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548011430 con 0x7ff5501e7e80 2024-06-29T14:38:29.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.820+0000 7ff55561f640 1 -- 172.21.15.81:0/2904447864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5500cae60 con 0x7ff5501e7e80 2024-06-29T14:38:29.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.820+0000 7ff55561f640 1 -- 172.21.15.81:0/2904447864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5500cb420 con 0x7ff5501e7e80 2024-06-29T14:38:29.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.821+0000 7ff55561f640 1 -- 172.21.15.81:0/2904447864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5501e8390 con 0x7ff5501e7e80 2024-06-29T14:38:29.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.821+0000 7ff5477fe640 1 -- 172.21.15.81:0/2904447864 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7ff5480115d0 con 0x7ff5501e7e80 2024-06-29T14:38:29.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.822+0000 7ff5477fe640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff530076420 0x7ff530078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:29.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.822+0000 7ff5477fe640 1 -- 172.21.15.81:0/2904447864 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7ff548098430 con 0x7ff5501e7e80 2024-06-29T14:38:29.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.822+0000 7ff54effd640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff530076420 0x7ff530078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:29.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.823+0000 7ff54effd640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff530076420 0x7ff530078930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff540035000 tx=0x7ff54002d040 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:29.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:29.826+0000 7ff5477fe640 1 -- 172.21.15.81:0/2904447864 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff548061a90 con 0x7ff5501e7e80 2024-06-29T14:38:30.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.043+0000 7ff55561f640 1 -- 172.21.15.81:0/2904447864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5501ea5f0 con 0x7ff5501e7e80 2024-06-29T14:38:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.044+0000 7ff5477fe640 1 -- 172.21.15.81:0/2904447864 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5231 (secure 0 0 0) 0x7ff548061390 con 0x7ff5501e7e80 2024-06-29T14:38:30.041 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:30.041 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":51,"btime":"2024-06-29T14:38:27:883643+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1895620554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1895620554},{"type":"v1","addr":"172.21.15.167:6836","nonce":1895620554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":7485,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3391563230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3391563230},{"type":"v1","addr":"172.21.15.81:6834","nonce":3391563230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":7488,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3932182351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3932182351},{"type":"v1","addr":"172.21.15.81:6835","nonce":3932182351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.216390+0000","modified":"2024-06-29T14:38:27.252097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/3193613343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":3193613343},{"type":"v1","addr":"172.21.15.167:6837","nonce":3193613343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":6}]} 2024-06-29T14:38:30.042 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 51 2024-06-29T14:38:30.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 -- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff530076420 msgr2=0x7ff530078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff530076420 0x7ff530078930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff540035000 tx=0x7ff54002d040 comp rx=0 tx=0).stop 2024-06-29T14:38:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 -- 172.21.15.81:0/2904447864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5501e7e80 msgr2=0x7ff55015c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5501e7e80 0x7ff55015c9e0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ff548002230 tx=0x7ff548006790 comp rx=0 tx=0).stop 2024-06-29T14:38:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 -- 172.21.15.81:0/2904447864 shutdown_connections 2024-06-29T14:38:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7ff530076420 0x7ff530078930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5501e7e80 0x7ff55015c9e0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5500cb940 0x7ff5501e7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 --2- 172.21.15.81:0/2904447864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff53c003a00 0x7ff5501f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 -- 172.21.15.81:0/2904447864 >> 172.21.15.81:0/2904447864 conn(0x7ff55009cf30 msgr2=0x7ff55015df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 -- 172.21.15.81:0/2904447864 shutdown_connections 2024-06-29T14:38:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.046+0000 7ff55561f640 1 -- 172.21.15.81:0/2904447864 wait complete. 2024-06-29T14:38:30.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:38:24.216390+0000', 'modified': '2024-06-29T14:38:27.252097+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7503}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7503': {'gid': 7503, 'name': 'b', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.167:6837/3193613343', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6835', 'nonce': 3193613343}, {'type': 'v1', 'addr': '172.21.15.167:6837', 'nonce': 3193613343}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7503, 'qdb_cluster': [7503]} 2024-06-29T14:38:30.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:38:30.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-06-29T14:38:30.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.576+0000 7f8dc3124640 1 -- 172.21.15.81:0/2162469703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dbc0d0b90 msgr2=0x7f8dbc0d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.576+0000 7f8dc3124640 1 --2- 172.21.15.81:0/2162469703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dbc0d0b90 0x7f8dbc0d1a80 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc08e330 tx=0x7f8da802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.576+0000 7f8dc3124640 1 -- 172.21.15.81:0/2162469703 shutdown_connections 2024-06-29T14:38:30.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.576+0000 7f8dc3124640 1 --2- 172.21.15.81:0/2162469703 >> v2:172.21.15.167:3300/0 conn(0x7f8dbc0b5250 0x7f8dbc0d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.576+0000 7f8dc3124640 1 --2- 172.21.15.81:0/2162469703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dbc0d0b90 0x7f8dbc0d1a80 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.576+0000 7f8dc3124640 1 -- 172.21.15.81:0/2162469703 >> 172.21.15.81:0/2162469703 conn(0x7f8dbc09cf30 msgr2=0x7f8dbc0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.576+0000 7f8dc3124640 1 -- 172.21.15.81:0/2162469703 shutdown_connections 2024-06-29T14:38:30.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.576+0000 7f8dc3124640 1 -- 172.21.15.81:0/2162469703 wait complete. 2024-06-29T14:38:30.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.578+0000 7f8dc3124640 1 Processor -- start 2024-06-29T14:38:30.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.578+0000 7f8dc3124640 1 -- start start 2024-06-29T14:38:30.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.579+0000 7f8dc3124640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dbc0b5250 0x7f8dbc14e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.579+0000 7f8dc3124640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dbc0b9040 0x7f8dbc0a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.579+0000 7f8dc3124640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8dbc0c74e0 0x7f8dbc0c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.579+0000 7f8dc3124640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc0d2500 con 0x7f8dbc0b9040 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.579+0000 7f8dc3124640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc0b8530 con 0x7f8dbc0c74e0 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.579+0000 7f8dc3124640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8dbc0d2680 con 0x7f8dbc0b5250 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dbbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dbc0b9040 0x7f8dbc0a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dbbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dbc0b9040 0x7f8dbc0a0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36876/0 (socket says 172.21.15.81:36876) 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dbbfff640 1 -- 172.21.15.81:0/774499146 learned_addr learned my addr 172.21.15.81:0/774499146 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dc16c1640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8dbc0c74e0 0x7f8dbc0c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dbc0b5250 0x7f8dbc14e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dbbfff640 1 -- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dbc0b5250 msgr2=0x7f8dbc14e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dbc0b5250 0x7f8dbc14e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dbbfff640 1 -- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8dbc0c74e0 msgr2=0x7f8dbc0c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8dbc0c74e0 0x7f8dbc0c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dbbfff640 1 -- 172.21.15.81:0/774499146 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dbc151f70 con 0x7f8dbc0b9040 2024-06-29T14:38:30.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dbc0b5250 0x7f8dbc14e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:30.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dc16c1640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8dbc0c74e0 0x7f8dbc0c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:30.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dbc0b9040 0x7f8dbc0a0b40 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009920 tx=0x7f8da8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:30.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8db97fa640 1 -- 172.21.15.81:0/774499146 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8033e10 con 0x7f8dbc0b9040 2024-06-29T14:38:30.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8db97fa640 1 -- 172.21.15.81:0/774499146 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8da8031b60 con 0x7f8dbc0b9040 2024-06-29T14:38:30.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dc3124640 1 -- 172.21.15.81:0/774499146 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dbc0c7e30 con 0x7f8dbc0b9040 2024-06-29T14:38:30.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8db97fa640 1 -- 172.21.15.81:0/774499146 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8005020 con 0x7f8dbc0b9040 2024-06-29T14:38:30.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.580+0000 7f8dc3124640 1 -- 172.21.15.81:0/774499146 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dbc14f470 con 0x7f8dbc0b9040 2024-06-29T14:38:30.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.582+0000 7f8db97fa640 1 -- 172.21.15.81:0/774499146 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f8da8045020 con 0x7f8dbc0b9040 2024-06-29T14:38:30.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.583+0000 7f8db97fa640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8da0076420 0x7f8da0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.583+0000 7f8db97fa640 1 -- 172.21.15.81:0/774499146 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f8da80b83f0 con 0x7f8dbc0b9040 2024-06-29T14:38:30.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.583+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8da0076420 0x7f8da0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.584+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8da0076420 0x7f8da0078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8db00016a0 tx=0x7f8db0009040 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:30.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.584+0000 7f8dc9fdc740 1 Processor -- start 2024-06-29T14:38:30.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.584+0000 7f8dc9fdc740 1 -- start start 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c8c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4ae3b0 0x561a9c8a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c8a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561a9c83fbc0 con 0x561a9c4d7270 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x561a9c7ee630 con 0x561a9c4ae3b0 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x561a9c7ddbd0 con 0x561a9c4b6590 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc0ec0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c8c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc0ec0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c8c36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36880/0 (socket says 172.21.15.81:36880) 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/2084356344 learned_addr learned my addr 172.21.15.81:0/2084356344 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc16c1640 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c8a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4ae3b0 0x561a9c8a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/2084356344 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 msgr2=0x561a9c8a78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c8a78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/2084356344 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4ae3b0 msgr2=0x561a9c8a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4ae3b0 0x561a9c8a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/2084356344 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a9c7daca0 con 0x561a9c4d7270 2024-06-29T14:38:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4ae3b0 0x561a9c8a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.585+0000 7f8dc16c1640 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c8a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c8c36e0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f8db0001e80 tx=0x7f8db0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443dfe90f3311583 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8da67fc640 1 -- 172.21.15.81:0/2084356344 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0005020 con 0x561a9c4d7270 2024-06-29T14:38:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8da67fc640 1 -- 172.21.15.81:0/2084356344 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8db00052b0 con 0x561a9c4d7270 2024-06-29T14:38:30.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8da67fc640 1 -- 172.21.15.81:0/2084356344 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0015940 con 0x561a9c4d7270 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/2084356344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 msgr2=0x561a9c8c36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c8c36e0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f8db0001e80 tx=0x7f8db0011000 comp rx=0 tx=0).stop 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/2084356344 shutdown_connections 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c8a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4ae3b0 0x561a9c8a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/2084356344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c8c36e0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.586+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/2084356344 >> 172.21.15.81:0/2084356344 conn(0x561a9c4cb3c0 msgr2=0x561a9c87af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.587+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/2084356344 shutdown_connections 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.587+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/2084356344 wait complete. 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.587+0000 7f8dc9fdc740 1 -- start start 2024-06-29T14:38:30.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.588+0000 7f8dc9fdc740 1 Processor -- start 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c8c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c895080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 0x561a9c8a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561a9c77bae0 con 0x561a9c4ae3b0 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x561a9c7e1c10 con 0x561a9c4d7270 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x561a9c551620 con 0x561a9c4b6590 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc9fdc740 10 client.0 _ll_register_callbacks cb 0x561a9c788480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc0ec0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c8c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc0ec0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c8c1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36896/0 (socket says 172.21.15.81:36896) 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/738196602 learned_addr learned my addr 172.21.15.81:0/738196602 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c895080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc16c1640 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 0x561a9c8a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 msgr2=0x561a9c895080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c895080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 msgr2=0x561a9c8a4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 0x561a9c8a4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.589+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/738196602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a9c542850 con 0x561a9c4ae3b0 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c8c1d60 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8db0022040 tx=0x7f8db0015b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59b480ef4c548cf server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8d9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/738196602 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db002f020 con 0x561a9c4ae3b0 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/738196602 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f8db002b660 con 0x561a9c4ae3b0 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/738196602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8db00052b0 con 0x561a9c4ae3b0 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/738196602 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f8db0039050 con 0x561a9c4ae3b0 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dc9fdc740 10 client.7635 fetch_fsmap learned FSMap version 51 2024-06-29T14:38:30.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/738196602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x561a9c5b96b0 con 0x561a9c4ae3b0 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/738196602 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8db00052b0 con 0x561a9c4ae3b0 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/738196602 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 51) ==== 3502+0+0 (secure 0 0 0) 0x7f8db003c020 con 0x561a9c4ae3b0 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dc9fdc740 10 client.7635 fetch_fsmap finished waiting for FSMap version 51 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dc9fdc740 10 client.7635 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dc9fdc740 20 client.7635 populate_metadata read hostname 'smithi081' 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:6835/3193613343,v1:172.21.15.167:6837/3193613343] conn(0x561a9c495e40 0x561a9c8d1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dc9fdc740 4 client.7635 mds_command: new command op to 7503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/738196602 --> [v2:172.21.15.167:6835/3193613343,v1:172.21.15.167:6837/3193613343] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a9c7f0580 con 0x561a9c495e40 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.590+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:6835/3193613343,v1:172.21.15.167:6837/3193613343] conn(0x561a9c495e40 0x561a9c8d1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.591+0000 7f8d9b7fe640 10 client.7635 ms_handle_connect on v2:172.21.15.167:6835/3193613343 2024-06-29T14:38:30.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.591+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:6835/3193613343,v1:172.21.15.167:6837/3193613343] conn(0x561a9c495e40 0x561a9c8d1f00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91d7b24a88980a2d server_cookie=752502f0cd35ed12 in_seq=0 out_seq=0 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/738196602 <== mds.0 v2:172.21.15.167:6835/3193613343 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x561a9c7f0580 con 0x561a9c495e40 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8d9b7fe640 10 client.7635 handle_command_reply: tid=0 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 client.7635 shutdown 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/738196602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 msgr2=0x561a9c8c1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c8c1d60 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8db0022040 tx=0x7f8db0015b70 comp rx=0 tx=0).stop 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/738196602 shutdown_connections 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:6835/3193613343,v1:172.21.15.167:6837/3193613343] conn(0x561a9c495e40 0x561a9c8d1f00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 0x561a9c8a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c895080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/738196602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c8c1d60 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8d9b7fe640 0 client.7635 ms_handle_reset on v2:172.21.15.167:6835/3193613343 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/738196602 >> 172.21.15.81:0/738196602 conn(0x561a9c4cb3c0 msgr2=0x561a9c8a3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/738196602 shutdown_connections 2024-06-29T14:38:30.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/738196602 wait complete. 2024-06-29T14:38:30.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.592+0000 7f8dc9fdc740 20 client.7635 trim_cache size 0 max 16384 2024-06-29T14:38:30.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.610+0000 7f8dc9fdc740 1 Processor -- start 2024-06-29T14:38:30.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.611+0000 7f8dc9fdc740 1 -- start start 2024-06-29T14:38:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4ae3b0 0x561a9c8a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c89e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c89ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc200a60 con 0x561a9c4d7270 2024-06-29T14:38:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc0b1130 con 0x561a9c4ae3b0 2024-06-29T14:38:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8dbc200530 con 0x561a9c4b6590 2024-06-29T14:38:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc16c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c89ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc16c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c89ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36912/0 (socket says 172.21.15.81:36912) 2024-06-29T14:38:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc16c1640 1 -- 172.21.15.81:0/3836002949 learned_addr learned my addr 172.21.15.81:0/3836002949 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc16c1640 1 -- 172.21.15.81:0/3836002949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 msgr2=0x561a9c89e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc16c1640 1 --2- 172.21.15.81:0/3836002949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c89e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc16c1640 1 -- 172.21.15.81:0/3836002949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4ae3b0 msgr2=0x561a9c8a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc16c1640 1 --2- 172.21.15.81:0/3836002949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4ae3b0 0x561a9c8a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc16c1640 1 -- 172.21.15.81:0/3836002949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a9c8a4440 con 0x561a9c4d7270 2024-06-29T14:38:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8dc16c1640 1 --2- 172.21.15.81:0/3836002949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c89ee50 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f8db40043b0 tx=0x7f8db4004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d313f9083c3a074 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8da5ffb640 1 -- 172.21.15.81:0/3836002949 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4004b20 con 0x561a9c4d7270 2024-06-29T14:38:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8da5ffb640 1 -- 172.21.15.81:0/3836002949 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8db4004cc0 con 0x561a9c4d7270 2024-06-29T14:38:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.612+0000 7f8da5ffb640 1 -- 172.21.15.81:0/3836002949 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4005220 con 0x561a9c4d7270 2024-06-29T14:38:30.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.613+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/3836002949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 msgr2=0x561a9c89ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.613+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/3836002949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c89ee50 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f8db40043b0 tx=0x7f8db4004870 comp rx=0 tx=0).stop 2024-06-29T14:38:30.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.613+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/3836002949 shutdown_connections 2024-06-29T14:38:30.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.613+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/3836002949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4d7270 0x561a9c89ee50 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.613+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/3836002949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c89e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.613+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/3836002949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4ae3b0 0x561a9c8a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.613+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/3836002949 >> 172.21.15.81:0/3836002949 conn(0x561a9c4cb3c0 msgr2=0x561a9c87e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.613+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/3836002949 shutdown_connections 2024-06-29T14:38:30.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.613+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/3836002949 wait complete. 2024-06-29T14:38:30.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.614+0000 7f8dc9fdc740 1 -- start start 2024-06-29T14:38:30.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.614+0000 7f8dc9fdc740 1 Processor -- start 2024-06-29T14:38:30.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c9400f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c8d3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc9fdc740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 0x561a9c8d4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561a9c8a5a90 con 0x561a9c4ae3b0 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x561a9c89f4b0 con 0x561a9c4d7270 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc9fdc740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x561a9c8a55e0 con 0x561a9c4b6590 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc9fdc740 10 client.0 _ll_register_callbacks cb 0x7f8db0001d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc0ec0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c9400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc0ec0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c9400f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36926/0 (socket says 172.21.15.81:36926) 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/4050503741 learned_addr learned my addr 172.21.15.81:0/4050503741 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc16c1640 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 0x561a9c8d4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c8d3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 msgr2=0x561a9c8d3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c8d3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 msgr2=0x561a9c8d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 0x561a9c8d4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc0ec0640 1 -- 172.21.15.81:0/4050503741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a9c8a80a0 con 0x561a9c4ae3b0 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c8d3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc16c1640 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 0x561a9c8d4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc0ec0640 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c9400f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f8db000e760 tx=0x7f8db00012e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e62e45dc3a4f26 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8d9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/4050503741 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db002f070 con 0x561a9c4ae3b0 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/4050503741 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f8db002b870 con 0x561a9c4ae3b0 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/4050503741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8db4004cc0 con 0x561a9c4ae3b0 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/4050503741 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f8db0004620 con 0x561a9c4ae3b0 2024-06-29T14:38:30.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc9fdc740 10 client.7641 fetch_fsmap learned FSMap version 51 2024-06-29T14:38:30.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.616+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/4050503741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x561a9c8a78a0 con 0x561a9c4ae3b0 2024-06-29T14:38:30.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/4050503741 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8db4004cc0 con 0x561a9c4ae3b0 2024-06-29T14:38:30.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/4050503741 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 51) ==== 3502+0+0 (secure 0 0 0) 0x7f8db003c040 con 0x561a9c4ae3b0 2024-06-29T14:38:30.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8dc9fdc740 10 client.7641 fetch_fsmap finished waiting for FSMap version 51 2024-06-29T14:38:30.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8dc9fdc740 10 client.7641 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:38:30.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8dc9fdc740 20 client.7641 populate_metadata read hostname 'smithi081' 2024-06-29T14:38:30.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:6835/3193613343,v1:172.21.15.167:6837/3193613343] conn(0x561a9c943320 0x561a9c8bdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8dc9fdc740 4 client.7641 mds_command: new command op to 7503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:38:30.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/4050503741 --> [v2:172.21.15.167:6835/3193613343,v1:172.21.15.167:6837/3193613343] -- command(tid 0: {"prefix": "status"}) -- 0x561a9c7ec020 con 0x561a9c943320 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:6835/3193613343,v1:172.21.15.167:6837/3193613343] conn(0x561a9c943320 0x561a9c8bdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8d9b7fe640 10 client.7641 ms_handle_connect on v2:172.21.15.167:6835/3193613343 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.617+0000 7f8dbbfff640 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:6835/3193613343,v1:172.21.15.167:6837/3193613343] conn(0x561a9c943320 0x561a9c8bdc10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef979923d533784e server_cookie=eeef75b21a75e2ad in_seq=0 out_seq=0 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8d9b7fe640 1 -- 172.21.15.81:0/4050503741 <== mds.0 v2:172.21.15.167:6835/3193613343 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x561a9c7ec020 con 0x561a9c943320 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8d9b7fe640 10 client.7641 handle_command_reply: tid=0 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 client.7641 shutdown 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/4050503741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 msgr2=0x561a9c9400f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c9400f0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f8db000e760 tx=0x7f8db00012e0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/4050503741 shutdown_connections 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:6835/3193613343,v1:172.21.15.167:6837/3193613343] conn(0x561a9c943320 0x561a9c8bdc10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x561a9c4d7270 0x561a9c8d4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8d9b7fe640 0 client.7641 ms_handle_reset on v2:172.21.15.167:6835/3193613343 2024-06-29T14:38:30.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x561a9c4b6590 0x561a9c8d3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 --2- 172.21.15.81:0/4050503741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a9c4ae3b0 0x561a9c9400f0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/4050503741 >> 172.21.15.81:0/4050503741 conn(0x561a9c4cb3c0 msgr2=0x561a9c878ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/4050503741 shutdown_connections 2024-06-29T14:38:30.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 1 -- 172.21.15.81:0/4050503741 wait complete. 2024-06-29T14:38:30.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.618+0000 7f8dc9fdc740 20 client.7641 trim_cache size 0 max 16384 2024-06-29T14:38:30.622 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:38:30.622 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T14:38:30.622 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T14:38:30.623 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 7503, 2024-06-29T14:38:30.623 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T14:38:30.623 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T14:38:30.623 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T14:38:30.623 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7443243480000001, 2024-06-29T14:38:30.623 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 51, 2024-06-29T14:38:30.623 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 104, 2024-06-29T14:38:30.623 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 104, 2024-06-29T14:38:30.623 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.419461482999999 2024-06-29T14:38:30.623 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:38:30.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 -- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8da0076420 msgr2=0x7f8da0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8da0076420 0x7f8da0078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8db00016a0 tx=0x7f8db0009040 comp rx=0 tx=0).stop 2024-06-29T14:38:30.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 -- 172.21.15.81:0/774499146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dbc0b9040 msgr2=0x7f8dbc0a0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dbc0b9040 0x7f8dbc0a0b40 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009920 tx=0x7f8da8033c20 comp rx=0 tx=0).stop 2024-06-29T14:38:30.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 -- 172.21.15.81:0/774499146 shutdown_connections 2024-06-29T14:38:30.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f8da0076420 0x7f8da0078930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8dbc0c74e0 0x7f8dbc0c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dbc0b9040 0x7f8dbc0a0b40 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 --2- 172.21.15.81:0/774499146 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8dbc0b5250 0x7f8dbc14e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 -- 172.21.15.81:0/774499146 >> 172.21.15.81:0/774499146 conn(0x7f8dbc09cf30 msgr2=0x7f8dbc15f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 -- 172.21.15.81:0/774499146 shutdown_connections 2024-06-29T14:38:30.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.619+0000 7f8dc3124640 1 -- 172.21.15.81:0/774499146 wait complete. 2024-06-29T14:38:30.627 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:38:30.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.743+0000 7f64e29f5640 1 -- 172.21.15.81:0/2935537341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64dc0ce770 msgr2=0x7f64dc0b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.743+0000 7f64e29f5640 1 --2- 172.21.15.81:0/2935537341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64dc0ce770 0x7f64dc0b35c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f64cc005fd0 tx=0x7f64cc02a540 comp rx=0 tx=0).stop 2024-06-29T14:38:30.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.743+0000 7f64e29f5640 1 -- 172.21.15.81:0/2935537341 shutdown_connections 2024-06-29T14:38:30.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.743+0000 7f64e29f5640 1 --2- 172.21.15.81:0/2935537341 >> v2:172.21.15.167:3300/0 conn(0x7f64dc0ceb80 0x7f64dc0b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.743+0000 7f64e29f5640 1 --2- 172.21.15.81:0/2935537341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64dc0ce770 0x7f64dc0b35c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.743+0000 7f64e29f5640 1 -- 172.21.15.81:0/2935537341 >> 172.21.15.81:0/2935537341 conn(0x7f64dc0d3c60 msgr2=0x7f64dc0d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.743+0000 7f64e29f5640 1 -- 172.21.15.81:0/2935537341 shutdown_connections 2024-06-29T14:38:30.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.743+0000 7f64e29f5640 1 -- 172.21.15.81:0/2935537341 wait complete. 2024-06-29T14:38:30.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.744+0000 7f64e29f5640 1 Processor -- start 2024-06-29T14:38:30.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.745+0000 7f64e29f5640 1 -- start start 2024-06-29T14:38:30.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64e29f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64dc0ce770 0x7f64dc1d9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64e29f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64dc0ceb80 0x7f64dc1f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64e29f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64dc0d2240 0x7f64dc0b77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64e29f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc0d15a0 con 0x7f64dc0ceb80 2024-06-29T14:38:30.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64e29f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc0d1420 con 0x7f64dc0d2240 2024-06-29T14:38:30.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64e29f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f64dc0d4990 con 0x7f64dc0ce770 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64dc0ceb80 0x7f64dc1f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64dc0ceb80 0x7f64dc1f52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36932/0 (socket says 172.21.15.81:36932) 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64d3fff640 1 -- 172.21.15.81:0/204097191 learned_addr learned my addr 172.21.15.81:0/204097191 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64dc0ce770 0x7f64dc1d9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64e0f92640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64dc0d2240 0x7f64dc0b77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64d3fff640 1 -- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64dc0ce770 msgr2=0x7f64dc1d9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64d3fff640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64dc0ce770 0x7f64dc1d9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64d3fff640 1 -- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64dc0d2240 msgr2=0x7f64dc0b77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64d3fff640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64dc0d2240 0x7f64dc0b77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.746+0000 7f64d3fff640 1 -- 172.21.15.81:0/204097191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc1f5de0 con 0x7f64dc0ceb80 2024-06-29T14:38:30.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.747+0000 7f64d3fff640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64dc0ceb80 0x7f64dc1f52d0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f64c400e790 tx=0x7f64c400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.747+0000 7f64d97fa640 1 -- 172.21.15.81:0/204097191 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c400eea0 con 0x7f64dc0ceb80 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.747+0000 7f64d97fa640 1 -- 172.21.15.81:0/204097191 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64c400ccc0 con 0x7f64dc0ceb80 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.747+0000 7f64e29f5640 1 -- 172.21.15.81:0/204097191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64dc1f60c0 con 0x7f64dc0ceb80 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.747+0000 7f64d97fa640 1 -- 172.21.15.81:0/204097191 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4005280 con 0x7f64dc0ceb80 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.747+0000 7f64e29f5640 1 -- 172.21.15.81:0/204097191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64dc1da340 con 0x7f64dc0ceb80 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.748+0000 7f64e29f5640 1 -- 172.21.15.81:0/204097191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64dc0d2650 con 0x7f64dc0ceb80 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.748+0000 7f64d97fa640 1 -- 172.21.15.81:0/204097191 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f64c4005420 con 0x7f64dc0ceb80 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.749+0000 7f64d97fa640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64bc0763f0 0x7f64bc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.749+0000 7f64d97fa640 1 -- 172.21.15.81:0/204097191 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f64c4098190 con 0x7f64dc0ceb80 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.749+0000 7f64dbfff640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64bc0763f0 0x7f64bc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:30.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.750+0000 7f64dbfff640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64bc0763f0 0x7f64bc078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64cc033000 tx=0x7f64cc004730 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:30.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.753+0000 7f64d97fa640 1 -- 172.21.15.81:0/204097191 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f64c4061240 con 0x7f64dc0ceb80 2024-06-29T14:38:30.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.970+0000 7f64e29f5640 1 -- 172.21.15.81:0/204097191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f64dc06e2c0 con 0x7f64dc0ceb80 2024-06-29T14:38:30.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.971+0000 7f64d97fa640 1 -- 172.21.15.81:0/204097191 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) ==== 53+0+73 (secure 0 0 0) 0x7f64c4065540 con 0x7f64dc0ceb80 2024-06-29T14:38:30.968 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:38:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 -- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64bc0763f0 msgr2=0x7f64bc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64bc0763f0 0x7f64bc078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64cc033000 tx=0x7f64cc004730 comp rx=0 tx=0).stop 2024-06-29T14:38:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 -- 172.21.15.81:0/204097191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64dc0ceb80 msgr2=0x7f64dc1f52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64dc0ceb80 0x7f64dc1f52d0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f64c400e790 tx=0x7f64c400ec50 comp rx=0 tx=0).stop 2024-06-29T14:38:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 -- 172.21.15.81:0/204097191 shutdown_connections 2024-06-29T14:38:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f64bc0763f0 0x7f64bc078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64dc0d2240 0x7f64dc0b77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64dc0ceb80 0x7f64dc1f52d0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 --2- 172.21.15.81:0/204097191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64dc0ce770 0x7f64dc1d9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 -- 172.21.15.81:0/204097191 >> 172.21.15.81:0/204097191 conn(0x7f64dc0d3c60 msgr2=0x7f64dc1e62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 -- 172.21.15.81:0/204097191 shutdown_connections 2024-06-29T14:38:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:30.973+0000 7f64e29f5640 1 -- 172.21.15.81:0/204097191 wait complete. 2024-06-29T14:38:30.981 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:38:30.982 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:38:30.982 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T14:38:30.982 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:38:30.982 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:30.982 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:38:30.982 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:38:30.982 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:38:30.982 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:38:30.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:30.982 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T14:38:30.998 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:38:30.998 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:38:30.998 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:38:30.999 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 41809sec preferred_lft 41809sec 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.000 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:38:31.001 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:38:31.001 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T14:38:31.001 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:38:31.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.001 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:38:31.063 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T14:38:31.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.065 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T14:38:31.121 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T14:38:31.121 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:38:31.121 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:38:31.121 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:38:31.121 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:38:31.122 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:38:31.122 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:38:31.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:38:31.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:38:31.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:38:31.122 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:38:31.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:38:31.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:38:31.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.247 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:38:31.262 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:38:31.262 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:38:31.262 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:38:31.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.262 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T14:38:31.327 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:38:31.327 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:31.382 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:38:31.383 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:38:31.383 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:31.438 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T14:38:31.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:38:31.532 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:38:31.532 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:38:31.545 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:38:31.545 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:38:31.545 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:38:31.545 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:38:31.545 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:38:31.546 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:38:31.546 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:38:31.546 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:38:31.546 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:38:31.546 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:38:31.547 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:38:31.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:38:31.548 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:38:31.572 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:38:31.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:38:31.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:38:31.676 INFO:teuthology.orchestra.run.smithi081.stdout:client6428 2024-06-29T14:38:31.676 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:38:31.676 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:38:31.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:38:31.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.849+0000 7f9599297640 1 -- 172.21.15.81:0/1264111172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9580004800 msgr2=0x7f9580004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.849+0000 7f9599297640 1 --2- 172.21.15.81:0/1264111172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9580004800 0x7f9580004c10 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f957c0016c0 tx=0x7f9580005830 comp rx=0 tx=0).stop 2024-06-29T14:38:31.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.849+0000 7f9599297640 1 -- 172.21.15.81:0/1264111172 shutdown_connections 2024-06-29T14:38:31.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.849+0000 7f9599297640 1 --2- 172.21.15.81:0/1264111172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9580004800 0x7f9580004c10 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.849+0000 7f9599297640 1 -- 172.21.15.81:0/1264111172 >> 172.21.15.81:0/1264111172 conn(0x7f959409cf30 msgr2=0x7f95940d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:31.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.849+0000 7f9599297640 1 -- 172.21.15.81:0/1264111172 shutdown_connections 2024-06-29T14:38:31.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.849+0000 7f9599297640 1 -- 172.21.15.81:0/1264111172 wait complete. 2024-06-29T14:38:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.851+0000 7f9599297640 1 Processor -- start 2024-06-29T14:38:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.851+0000 7f9599297640 1 -- start start 2024-06-29T14:38:31.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.852+0000 7f9599297640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9580004800 0x7f959415d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.852+0000 7f9599297640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f959409e9f0 0x7f95940b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.852+0000 7f9599297640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95940b1d40 0x7f95940b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.852+0000 7f9599297640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f95940d0810 con 0x7f9580004800 2024-06-29T14:38:31.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.852+0000 7f9599297640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f95940d0690 con 0x7f959409e9f0 2024-06-29T14:38:31.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.852+0000 7f9599297640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f95940d0990 con 0x7f95940b1d40 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.852+0000 7f9592ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9580004800 0x7f959415d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f9592ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9580004800 0x7f959415d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36936/0 (socket says 172.21.15.81:36936) 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f9592ffd640 1 -- 172.21.15.81:0/59828628 learned_addr learned my addr 172.21.15.81:0/59828628 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f9592ffd640 1 -- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95940b1d40 msgr2=0x7f95940b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f9592ffd640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95940b1d40 0x7f95940b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f9592ffd640 1 -- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f959409e9f0 msgr2=0x7f95940b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f9592ffd640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f959409e9f0 0x7f95940b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f9592ffd640 1 -- 172.21.15.81:0/59828628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95941f4aa0 con 0x7f9580004800 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f9592ffd640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9580004800 0x7f959415d440 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f9584010a20 tx=0x7f9584010ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f958b7fe640 1 -- 172.21.15.81:0/59828628 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958400ec50 con 0x7f9580004800 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f958b7fe640 1 -- 172.21.15.81:0/59828628 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f958400edf0 con 0x7f9580004800 2024-06-29T14:38:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f9599297640 1 -- 172.21.15.81:0/59828628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95940a06f0 con 0x7f9580004800 2024-06-29T14:38:31.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f958b7fe640 1 -- 172.21.15.81:0/59828628 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95840116a0 con 0x7f9580004800 2024-06-29T14:38:31.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.853+0000 7f9599297640 1 -- 172.21.15.81:0/59828628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95940a0cb0 con 0x7f9580004800 2024-06-29T14:38:31.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.854+0000 7f9599297640 1 -- 172.21.15.81:0/59828628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95940ce540 con 0x7f9580004800 2024-06-29T14:38:31.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.855+0000 7f958b7fe640 1 -- 172.21.15.81:0/59828628 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f9584011840 con 0x7f9580004800 2024-06-29T14:38:31.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.855+0000 7f958b7fe640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9574076420 0x7f9574078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:31.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.856+0000 7f958b7fe640 1 -- 172.21.15.81:0/59828628 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f9584098670 con 0x7f9580004800 2024-06-29T14:38:31.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.856+0000 7f95927fc640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9574076420 0x7f9574078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:31.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.856+0000 7f95927fc640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9574076420 0x7f9574078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f957c032000 tx=0x7f957c001120 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:31.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:31.860+0000 7f958b7fe640 1 -- 172.21.15.81:0/59828628 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f95940ce540 con 0x7f9580004800 2024-06-29T14:38:32.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.066+0000 7f9599297640 1 -- 172.21.15.81:0/59828628 --> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f95940a0470 con 0x7f9574076420 2024-06-29T14:38:32.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.068+0000 7f958b7fe640 1 -- 172.21.15.81:0/59828628 <== mgr.6158 v2:172.21.15.167:6832/2583731609 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f957c02e040 con 0x7f9574076420 2024-06-29T14:38:32.065 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:38:32.065 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:38:32.066 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T14:38:32.066 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:38:32.066 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:38:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.069+0000 7f9599297640 1 -- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9574076420 msgr2=0x7f9574078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.069+0000 7f9599297640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9574076420 0x7f9574078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f957c032000 tx=0x7f957c001120 comp rx=0 tx=0).stop 2024-06-29T14:38:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.069+0000 7f9599297640 1 -- 172.21.15.81:0/59828628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9580004800 msgr2=0x7f959415d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.069+0000 7f9599297640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9580004800 0x7f959415d440 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f9584010a20 tx=0x7f9584010ee0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.069+0000 7f9599297640 1 -- 172.21.15.81:0/59828628 shutdown_connections 2024-06-29T14:38:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.069+0000 7f9599297640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f9574076420 0x7f9574078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.069+0000 7f9599297640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95940b1d40 0x7f95940b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.069+0000 7f9599297640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f959409e9f0 0x7f95940b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.069+0000 7f9599297640 1 --2- 172.21.15.81:0/59828628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9580004800 0x7f959415d440 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.070+0000 7f9599297640 1 -- 172.21.15.81:0/59828628 >> 172.21.15.81:0/59828628 conn(0x7f959409cf30 msgr2=0x7f95941611f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.070+0000 7f9599297640 1 -- 172.21.15.81:0/59828628 shutdown_connections 2024-06-29T14:38:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.070+0000 7f9599297640 1 -- 172.21.15.81:0/59828628 wait complete. 2024-06-29T14:38:32.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:38:32.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.195+0000 7f4ce58fd640 1 -- 172.21.15.81:0/2282088006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce0072ac0 msgr2=0x7f4ce00ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.195+0000 7f4ce58fd640 1 --2- 172.21.15.81:0/2282088006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce0072ac0 0x7f4ce00ccb50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f4ce00cd710 tx=0x7f4cc802a560 comp rx=0 tx=0).stop 2024-06-29T14:38:32.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.195+0000 7f4ce58fd640 1 -- 172.21.15.81:0/2282088006 shutdown_connections 2024-06-29T14:38:32.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.195+0000 7f4ce58fd640 1 --2- 172.21.15.81:0/2282088006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce0072ac0 0x7f4ce00ccb50 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.195+0000 7f4ce58fd640 1 --2- 172.21.15.81:0/2282088006 >> v2:172.21.15.167:3300/0 conn(0x7f4ce00726b0 0x7f4ce00cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.195+0000 7f4ce58fd640 1 -- 172.21.15.81:0/2282088006 >> 172.21.15.81:0/2282088006 conn(0x7f4ce00b1ec0 msgr2=0x7f4ce00b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:32.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.195+0000 7f4ce58fd640 1 -- 172.21.15.81:0/2282088006 shutdown_connections 2024-06-29T14:38:32.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.195+0000 7f4ce58fd640 1 -- 172.21.15.81:0/2282088006 wait complete. 2024-06-29T14:38:32.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.197+0000 7f4ce58fd640 1 Processor -- start 2024-06-29T14:38:32.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.197+0000 7f4ce58fd640 1 -- start start 2024-06-29T14:38:32.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4ce58fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ce00726b0 0x7f4ce014bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4ce58fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ce0072ac0 0x7f4ce00b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4ce58fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce00cf1f0 0x7f4ce0150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4ce58fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce006d690 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4ce58fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce006d510 con 0x7f4ce00726b0 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4ce58fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4ce015a950 con 0x7f4ce0072ac0 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce00cf1f0 0x7f4ce0150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce00cf1f0 0x7f4ce0150a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36962/0 (socket says 172.21.15.81:36962) 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdf7fe640 1 -- 172.21.15.81:0/3831615402 learned_addr learned my addr 172.21.15.81:0/3831615402 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdeffd640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ce00726b0 0x7f4ce014bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cde7fc640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ce0072ac0 0x7f4ce00b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdf7fe640 1 -- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ce0072ac0 msgr2=0x7f4ce00b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdf7fe640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ce0072ac0 0x7f4ce00b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdf7fe640 1 -- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ce00726b0 msgr2=0x7f4ce014bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdf7fe640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ce00726b0 0x7f4ce014bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdf7fe640 1 -- 172.21.15.81:0/3831615402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce0158460 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cde7fc640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ce0072ac0 0x7f4ce00b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdeffd640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ce00726b0 0x7f4ce014bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cdf7fe640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce00cf1f0 0x7f4ce0150a00 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000b410 tx=0x7f4cd000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cd7fff640 1 -- 172.21.15.81:0/3831615402 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0015030 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cd7fff640 1 -- 172.21.15.81:0/3831615402 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4cd0004110 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4ce58fd640 1 -- 172.21.15.81:0/3831615402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce0158740 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.199+0000 7f4cd7fff640 1 -- 172.21.15.81:0/3831615402 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0011630 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.200+0000 7f4ce58fd640 1 -- 172.21.15.81:0/3831615402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce0158d00 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.201+0000 7f4ce58fd640 1 -- 172.21.15.81:0/3831615402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce006db40 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.201+0000 7f4cd7fff640 1 -- 172.21.15.81:0/3831615402 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 46) ==== 98303+0+0 (secure 0 0 0) 0x7f4cd0018070 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.202+0000 7f4cd7fff640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4cc00763f0 0x7f4cc0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:32.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.202+0000 7f4cd7fff640 1 -- 172.21.15.81:0/3831615402 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f4cd00975d0 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.202+0000 7f4cdeffd640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4cc00763f0 0x7f4cc0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:32.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.203+0000 7f4cdeffd640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4cc00763f0 0x7f4cc0078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8005c40 tx=0x7f4cd800a410 comp rx=0 tx=0).ready entity=mgr.6158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:32.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.206+0000 7f4cd7fff640 1 -- 172.21.15.81:0/3831615402 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4ce006db40 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.434+0000 7f4ce58fd640 1 -- 172.21.15.81:0/3831615402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f4ce006fb30 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.911+0000 7f4cd7fff640 1 -- 172.21.15.81:0/3831615402 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v47) ==== 92+0+0 (secure 0 0 0) 0x7f4ce006fb30 con 0x7f4ce00cf1f0 2024-06-29T14:38:32.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.915+0000 7f4ce58fd640 1 -- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4cc00763f0 msgr2=0x7f4cc0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.915+0000 7f4ce58fd640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4cc00763f0 0x7f4cc0078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8005c40 tx=0x7f4cd800a410 comp rx=0 tx=0).stop 2024-06-29T14:38:32.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.915+0000 7f4ce58fd640 1 -- 172.21.15.81:0/3831615402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce00cf1f0 msgr2=0x7f4ce0150a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:32.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.915+0000 7f4ce58fd640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce00cf1f0 0x7f4ce0150a00 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000b410 tx=0x7f4cd000b8d0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.915+0000 7f4ce58fd640 1 -- 172.21.15.81:0/3831615402 shutdown_connections 2024-06-29T14:38:32.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.915+0000 7f4ce58fd640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f4cc00763f0 0x7f4cc0078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.915+0000 7f4ce58fd640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce00cf1f0 0x7f4ce0150a00 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.915+0000 7f4ce58fd640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ce0072ac0 0x7f4ce00b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.915+0000 7f4ce58fd640 1 --2- 172.21.15.81:0/3831615402 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ce00726b0 0x7f4ce014bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:32.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.915+0000 7f4ce58fd640 1 -- 172.21.15.81:0/3831615402 >> 172.21.15.81:0/3831615402 conn(0x7f4ce00b1ec0 msgr2=0x7f4ce01553b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:32.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.916+0000 7f4ce58fd640 1 -- 172.21.15.81:0/3831615402 shutdown_connections 2024-06-29T14:38:32.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:32.916+0000 7f4ce58fd640 1 -- 172.21.15.81:0/3831615402 wait complete. 2024-06-29T14:38:32.929 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:38:33.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.064+0000 7f7c9c773640 1 -- 172.21.15.81:0/476070258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c9415bdf0 msgr2=0x7f7c94165ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.064+0000 7f7c9c773640 1 --2- 172.21.15.81:0/476070258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c9415bdf0 0x7f7c94165ab0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009860 tx=0x7f7c9002a510 comp rx=0 tx=0).stop 2024-06-29T14:38:33.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.065+0000 7f7c9c773640 1 -- 172.21.15.81:0/476070258 shutdown_connections 2024-06-29T14:38:33.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.065+0000 7f7c9c773640 1 --2- 172.21.15.81:0/476070258 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c9415bdf0 0x7f7c94165ab0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.065+0000 7f7c9c773640 1 -- 172.21.15.81:0/476070258 >> 172.21.15.81:0/476070258 conn(0x7f7c940b2920 msgr2=0x7f7c940b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:33.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.065+0000 7f7c9c773640 1 -- 172.21.15.81:0/476070258 shutdown_connections 2024-06-29T14:38:33.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.065+0000 7f7c9c773640 1 -- 172.21.15.81:0/476070258 wait complete. 2024-06-29T14:38:33.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.066+0000 7f7c9c773640 1 Processor -- start 2024-06-29T14:38:33.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.066+0000 7f7c9c773640 1 -- start start 2024-06-29T14:38:33.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c9c773640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c940cd150 0x7f7c941f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c9c773640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9415bdf0 0x7f7c9414b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c9c773640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c9414f040 0x7f7c9414bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c9c773640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7c94166200 con 0x7f7c9415bdf0 2024-06-29T14:38:33.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c9c773640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9406fe80 con 0x7f7c9414f040 2024-06-29T14:38:33.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c9c773640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7c94166380 con 0x7f7c940cd150 2024-06-29T14:38:33.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c99d0e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9415bdf0 0x7f7c9414b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c9a50f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c940cd150 0x7f7c941f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c99d0e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9415bdf0 0x7f7c9414b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36992/0 (socket says 172.21.15.81:36992) 2024-06-29T14:38:33.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c99d0e640 1 -- 172.21.15.81:0/3042440219 learned_addr learned my addr 172.21.15.81:0/3042440219 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c9ad10640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c9414f040 0x7f7c9414bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c99d0e640 1 -- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c940cd150 msgr2=0x7f7c941f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c99d0e640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c940cd150 0x7f7c941f2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c99d0e640 1 -- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c9414f040 msgr2=0x7f7c9414bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c99d0e640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c9414f040 0x7f7c9414bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c99d0e640 1 -- 172.21.15.81:0/3042440219 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c941f2ad0 con 0x7f7c9415bdf0 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c9a50f640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c940cd150 0x7f7c941f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.068+0000 7f7c99d0e640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9415bdf0 0x7f7c9414b7c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400c400 tx=0x7f7c8400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.069+0000 7f7c8affd640 1 -- 172.21.15.81:0/3042440219 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84017070 con 0x7f7c9415bdf0 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.069+0000 7f7c8affd640 1 -- 172.21.15.81:0/3042440219 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c840053c0 con 0x7f7c9415bdf0 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.069+0000 7f7c9c773640 1 -- 172.21.15.81:0/3042440219 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c9414e2a0 con 0x7f7c9415bdf0 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.069+0000 7f7c8affd640 1 -- 172.21.15.81:0/3042440219 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84003850 con 0x7f7c9415bdf0 2024-06-29T14:38:33.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.069+0000 7f7c9c773640 1 -- 172.21.15.81:0/3042440219 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c941502f0 con 0x7f7c9415bdf0 2024-06-29T14:38:33.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.070+0000 7f7c9c773640 1 -- 172.21.15.81:0/3042440219 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c9415b6e0 con 0x7f7c9415bdf0 2024-06-29T14:38:33.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.071+0000 7f7c8affd640 1 -- 172.21.15.81:0/3042440219 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 47) ==== 98320+0+0 (secure 0 0 0) 0x7f7c8401a070 con 0x7f7c9415bdf0 2024-06-29T14:38:33.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.071+0000 7f7c8affd640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f7c78076470 0x7f7c78078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.071+0000 7f7c9a50f640 1 -- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f7c78076470 msgr2=0x7f7c78078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2583731609 2024-06-29T14:38:33.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.071+0000 7f7c8affd640 1 -- 172.21.15.81:0/3042440219 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f7c84096c30 con 0x7f7c9415bdf0 2024-06-29T14:38:33.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.071+0000 7f7c9a50f640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f7c78076470 0x7f7c78078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:33.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.075+0000 7f7c8affd640 1 -- 172.21.15.81:0/3042440219 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7c941f2ad0 con 0x7f7c9415bdf0 2024-06-29T14:38:33.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.260+0000 7f7c9c773640 1 -- 172.21.15.81:0/3042440219 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f7c94155cd0 con 0x7f7c9415bdf0 2024-06-29T14:38:33.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.264+0000 7f7c8affd640 1 -- 172.21.15.81:0/3042440219 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v45)=0 v45) ==== 138+0+0 (secure 0 0 0) 0x7f7c8405fe70 con 0x7f7c9415bdf0 2024-06-29T14:38:33.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 -- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f7c78076470 msgr2=0x7f7c78078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f7c78076470 0x7f7c78078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 -- 172.21.15.81:0/3042440219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9415bdf0 msgr2=0x7f7c9414b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9415bdf0 0x7f7c9414b7c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400c400 tx=0x7f7c8400c8c0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 -- 172.21.15.81:0/3042440219 shutdown_connections 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f7c78076470 0x7f7c78078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c9414f040 0x7f7c9414bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9415bdf0 0x7f7c9414b7c0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 --2- 172.21.15.81:0/3042440219 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c940cd150 0x7f7c941f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 -- 172.21.15.81:0/3042440219 >> 172.21.15.81:0/3042440219 conn(0x7f7c940b2920 msgr2=0x7f7c94072f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 -- 172.21.15.81:0/3042440219 shutdown_connections 2024-06-29T14:38:33.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.267+0000 7f7c9c773640 1 -- 172.21.15.81:0/3042440219 wait complete. 2024-06-29T14:38:33.280 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:38:33.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.401+0000 7f1446d8c640 1 -- 172.21.15.81:0/2770633526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f144006c2f0 msgr2=0x7f144006c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.401+0000 7f1446d8c640 1 --2- 172.21.15.81:0/2770633526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f144006c2f0 0x7f144006c700 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f1438005fd0 tx=0x7f143802a9e0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.401+0000 7f1446d8c640 1 -- 172.21.15.81:0/2770633526 shutdown_connections 2024-06-29T14:38:33.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.401+0000 7f1446d8c640 1 --2- 172.21.15.81:0/2770633526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f144006c2f0 0x7f144006c700 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.401+0000 7f1446d8c640 1 --2- 172.21.15.81:0/2770633526 >> v2:172.21.15.167:3300/0 conn(0x7f1440070d60 0x7f144006fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.401+0000 7f1446d8c640 1 -- 172.21.15.81:0/2770633526 >> 172.21.15.81:0/2770633526 conn(0x7f14400b2920 msgr2=0x7f14400b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:33.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.401+0000 7f1446d8c640 1 -- 172.21.15.81:0/2770633526 shutdown_connections 2024-06-29T14:38:33.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.401+0000 7f1446d8c640 1 -- 172.21.15.81:0/2770633526 wait complete. 2024-06-29T14:38:33.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.403+0000 7f1446d8c640 1 Processor -- start 2024-06-29T14:38:33.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.403+0000 7f1446d8c640 1 -- start start 2024-06-29T14:38:33.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f1446d8c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f144006c2f0 0x7f1440144680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f1446d8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1440070020 0x7f14400ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f1446d8c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1440070d60 0x7f1440159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f1446d8c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f144015fb30 con 0x7f1440070020 2024-06-29T14:38:33.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f1446d8c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f14400d0880 con 0x7f144006c2f0 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f1446d8c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f144015fcb0 con 0x7f1440070d60 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f143ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1440070020 0x7f14400ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f143ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1440070020 0x7f14400ad300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37016/0 (socket says 172.21.15.81:37016) 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f143ffff640 1 -- 172.21.15.81:0/3262277756 learned_addr learned my addr 172.21.15.81:0/3262277756 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f1445329640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1440070d60 0x7f1440159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.404+0000 7f1444b28640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f144006c2f0 0x7f1440144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f143ffff640 1 -- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1440070d60 msgr2=0x7f1440159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f143ffff640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1440070d60 0x7f1440159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f143ffff640 1 -- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f144006c2f0 msgr2=0x7f1440144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f143ffff640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f144006c2f0 0x7f1440144680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f143ffff640 1 -- 172.21.15.81:0/3262277756 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1440144bc0 con 0x7f1440070020 2024-06-29T14:38:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f143ffff640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1440070020 0x7f14400ad300 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f142c00c3c0 tx=0x7f142c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:33.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f143d7fa640 1 -- 172.21.15.81:0/3262277756 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c017070 con 0x7f1440070020 2024-06-29T14:38:33.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f143d7fa640 1 -- 172.21.15.81:0/3262277756 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f142c0035f0 con 0x7f1440070020 2024-06-29T14:38:33.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f1446d8c640 1 -- 172.21.15.81:0/3262277756 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14401f63f0 con 0x7f1440070020 2024-06-29T14:38:33.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f143d7fa640 1 -- 172.21.15.81:0/3262277756 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f142c003a80 con 0x7f1440070020 2024-06-29T14:38:33.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.405+0000 7f1446d8c640 1 -- 172.21.15.81:0/3262277756 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14401f69b0 con 0x7f1440070020 2024-06-29T14:38:33.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.406+0000 7f1446d8c640 1 -- 172.21.15.81:0/3262277756 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14400add90 con 0x7f1440070020 2024-06-29T14:38:33.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.407+0000 7f143d7fa640 1 -- 172.21.15.81:0/3262277756 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 47) ==== 98320+0+0 (secure 0 0 0) 0x7f142c01a070 con 0x7f1440070020 2024-06-29T14:38:33.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.407+0000 7f143d7fa640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1424076440 0x7f1424078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.407+0000 7f1444b28640 1 -- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1424076440 msgr2=0x7f1424078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2583731609 2024-06-29T14:38:33.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.407+0000 7f143d7fa640 1 -- 172.21.15.81:0/3262277756 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7f142c096c80 con 0x7f1440070020 2024-06-29T14:38:33.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.407+0000 7f1444b28640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1424076440 0x7f1424078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:33.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.412+0000 7f143d7fa640 1 -- 172.21.15.81:0/3262277756 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f144006cc40 con 0x7f1440070020 2024-06-29T14:38:33.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.608+0000 7f1444b28640 1 -- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1424076440 msgr2=0x7f1424078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2583731609 2024-06-29T14:38:33.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.608+0000 7f1444b28640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1424076440 0x7f1424078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:33.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.643+0000 7f1446d8c640 1 -- 172.21.15.81:0/3262277756 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f1440071670 con 0x7f1440070020 2024-06-29T14:38:33.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.647+0000 7f143d7fa640 1 -- 172.21.15.81:0/3262277756 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v46)=0 v46) ==== 133+0+0 (secure 0 0 0) 0x7f142c05fe30 con 0x7f1440070020 2024-06-29T14:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 -- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1424076440 msgr2=0x7f1424078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1424076440 0x7f1424078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 -- 172.21.15.81:0/3262277756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1440070020 msgr2=0x7f14400ad300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1440070020 0x7f14400ad300 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f142c00c3c0 tx=0x7f142c00c880 comp rx=0 tx=0).stop 2024-06-29T14:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 -- 172.21.15.81:0/3262277756 shutdown_connections 2024-06-29T14:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7f1424076440 0x7f1424078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1440070d60 0x7f1440159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1440070020 0x7f14400ad300 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 --2- 172.21.15.81:0/3262277756 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f144006c2f0 0x7f1440144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 -- 172.21.15.81:0/3262277756 >> 172.21.15.81:0/3262277756 conn(0x7f14400b2920 msgr2=0x7f144015d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:33.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 -- 172.21.15.81:0/3262277756 shutdown_connections 2024-06-29T14:38:33.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.649+0000 7f1446d8c640 1 -- 172.21.15.81:0/3262277756 wait complete. 2024-06-29T14:38:33.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2024-06-29T14:38:33.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.780+0000 7fc5535c7640 1 -- 172.21.15.81:0/3594865111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc54c0b5250 msgr2=0x7fc54c0d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.780+0000 7fc5535c7640 1 --2- 172.21.15.81:0/3594865111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc54c0b5250 0x7fc54c0d1fc0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0a1b70 tx=0x7fc54402a940 comp rx=0 tx=0).stop 2024-06-29T14:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.780+0000 7fc5535c7640 1 -- 172.21.15.81:0/3594865111 shutdown_connections 2024-06-29T14:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.780+0000 7fc5535c7640 1 --2- 172.21.15.81:0/3594865111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc54c0b5250 0x7fc54c0d1fc0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.780+0000 7fc5535c7640 1 --2- 172.21.15.81:0/3594865111 >> v2:172.21.15.167:3300/0 conn(0x7fc54c0d0b90 0x7fc54c0d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.780+0000 7fc5535c7640 1 -- 172.21.15.81:0/3594865111 >> 172.21.15.81:0/3594865111 conn(0x7fc54c09cf30 msgr2=0x7fc54c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.780+0000 7fc5535c7640 1 -- 172.21.15.81:0/3594865111 shutdown_connections 2024-06-29T14:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.780+0000 7fc5535c7640 1 -- 172.21.15.81:0/3594865111 wait complete. 2024-06-29T14:38:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.782+0000 7fc5535c7640 1 Processor -- start 2024-06-29T14:38:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.782+0000 7fc5535c7640 1 -- start start 2024-06-29T14:38:33.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc5535c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc54c0b5250 0x7fc54c0a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc5535c7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc54c0b9040 0x7fc54c09e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc5535c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc54c1e8840 0x7fc54c1e8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc5535c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc54c0d2500 con 0x7fc54c0b5250 2024-06-29T14:38:33.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc5535c7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc54c0b8530 con 0x7fc54c1e8840 2024-06-29T14:38:33.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc5535c7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc54c0d2680 con 0x7fc54c0b9040 2024-06-29T14:38:33.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc551363640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc54c0b5250 0x7fc54c0a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc551363640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc54c0b5250 0x7fc54c0a1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37022/0 (socket says 172.21.15.81:37022) 2024-06-29T14:38:33.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc551363640 1 -- 172.21.15.81:0/2287827073 learned_addr learned my addr 172.21.15.81:0/2287827073 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:33.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc551b64640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc54c1e8840 0x7fc54c1e8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc550b62640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc54c0b9040 0x7fc54c09e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc551363640 1 -- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc54c0b9040 msgr2=0x7fc54c09e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc551363640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc54c0b9040 0x7fc54c09e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc551363640 1 -- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc54c1e8840 msgr2=0x7fc54c1e8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc551363640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc54c1e8840 0x7fc54c1e8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc551363640 1 -- 172.21.15.81:0/2287827073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc54c0a1880 con 0x7fc54c0b5250 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc551b64640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc54c1e8840 0x7fc54c1e8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.783+0000 7fc550b62640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc54c0b9040 0x7fc54c09e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.784+0000 7fc551363640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc54c0b5250 0x7fc54c0a1340 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00c3c0 tx=0x7fc53c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.784+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53c017070 con 0x7fc54c0b5250 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.784+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc53c0035f0 con 0x7fc54c0b5250 2024-06-29T14:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.784+0000 7fc5535c7640 1 -- 172.21.15.81:0/2287827073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc54c1e9190 con 0x7fc54c0b5250 2024-06-29T14:38:33.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.784+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53c003a80 con 0x7fc54c0b5250 2024-06-29T14:38:33.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.784+0000 7fc5535c7640 1 -- 172.21.15.81:0/2287827073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc54c14ae80 con 0x7fc54c0b5250 2024-06-29T14:38:33.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.785+0000 7fc5535c7640 1 -- 172.21.15.81:0/2287827073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc54c0d2c10 con 0x7fc54c0b5250 2024-06-29T14:38:33.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.786+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 47) ==== 98320+0+0 (secure 0 0 0) 0x7fc53c01a070 con 0x7fc54c0b5250 2024-06-29T14:38:33.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.787+0000 7fc541ffb640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 0x7fc530078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:33.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.787+0000 7fc550b62640 1 -- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 msgr2=0x7fc530078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2583731609 2024-06-29T14:38:33.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.787+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5781+0+0 (secure 0 0 0) 0x7fc53c096d60 con 0x7fc54c0b5250 2024-06-29T14:38:33.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.787+0000 7fc550b62640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 0x7fc530078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:38:33.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.791+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc54c1e9190 con 0x7fc54c0b5250 2024-06-29T14:38:33.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.988+0000 7fc550b62640 1 -- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 msgr2=0x7fc530078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2583731609 2024-06-29T14:38:33.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:33.989+0000 7fc550b62640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 0x7fc530078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:38:33.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:34.002+0000 7fc5535c7640 1 -- 172.21.15.81:0/2287827073 --> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc54c14bbc0 con 0x7fc530076470 2024-06-29T14:38:34.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:34.390+0000 7fc550b62640 1 -- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 msgr2=0x7fc530078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2583731609 2024-06-29T14:38:34.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:34.390+0000 7fc550b62640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 0x7fc530078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:38:35.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:35.191+0000 7fc550b62640 1 -- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 msgr2=0x7fc530078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2583731609 2024-06-29T14:38:35.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:35.191+0000 7fc550b62640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 0x7fc530078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:38:36.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:36.595+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 48) ==== 97929+0+0 (secure 0 0 0) 0x7fc53c09c020 con 0x7fc54c0b5250 2024-06-29T14:38:36.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:36.595+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 msgr2=0x7fc530078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:38:36.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:36.595+0000 7fc541ffb640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/2583731609,v1:172.21.15.167:6833/2583731609] conn(0x7fc530076470 0x7fc530078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:36.754 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:36.756+0000 7f21f3160640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:36.754 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:36.756+0000 7f21f3160640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:36.757 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:36.756+0000 7f21f3160640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:36.757 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:36.756+0000 7f21f3160640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:36.757 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:36.756+0000 7f21f3160640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:36.757 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:36.756+0000 7f21f6166640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:36.757 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:36.756+0000 7f21f6166640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:36.758 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:36.756+0000 7f21f6166640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:36.758 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:36.756+0000 7f21f6166640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:36.758 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:38:36.756+0000 7f21f6166640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:38:37.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.600+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 49) ==== 98056+0+0 (secure 0 0 0) 0x7fc53c095230 con 0x7fc54c0b5250 2024-06-29T14:38:37.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.600+0000 7fc541ffb640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fc53003ad60 0x7fc530080460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:37.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.600+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 --> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fc53c047b40 con 0x7fc53003ad60 2024-06-29T14:38:37.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.602+0000 7fc550b62640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fc53003ad60 0x7fc530080460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:37.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.603+0000 7fc550b62640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fc53003ad60 0x7fc530080460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc54c08da30 tx=0x7fc5480072e0 comp rx=0 tx=0).ready entity=mgr.6443 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:37.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.669+0000 7fc541ffb640 1 -- 172.21.15.81:0/2287827073 <== mgr.6443 v2:172.21.15.167:6832/295682379 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fc53c047b40 con 0x7fc53003ad60 2024-06-29T14:38:37.666 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /bad-path 2024-06-29T14:38:37.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.671+0000 7fc5535c7640 1 -- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fc53003ad60 msgr2=0x7fc530080460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:37.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.671+0000 7fc5535c7640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fc53003ad60 0x7fc530080460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc54c08da30 tx=0x7fc5480072e0 comp rx=0 tx=0).stop 2024-06-29T14:38:37.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.671+0000 7fc5535c7640 1 -- 172.21.15.81:0/2287827073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc54c0b5250 msgr2=0x7fc54c0a1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:37.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.671+0000 7fc5535c7640 1 --2- 172.21.15.81:0/2287827073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc54c0b5250 0x7fc54c0a1340 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00c3c0 tx=0x7fc53c00c880 comp rx=0 tx=0).stop 2024-06-29T14:38:37.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.671+0000 7fc551b64640 1 -- 172.21.15.81:0/2287827073 reap_dead start 2024-06-29T14:38:37.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.671+0000 7fc5535c7640 1 -- 172.21.15.81:0/2287827073 shutdown_connections 2024-06-29T14:38:37.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.671+0000 7fc5535c7640 1 -- 172.21.15.81:0/2287827073 >> 172.21.15.81:0/2287827073 conn(0x7fc54c09cf30 msgr2=0x7fc54c06d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:37.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.671+0000 7fc5535c7640 1 -- 172.21.15.81:0/2287827073 shutdown_connections 2024-06-29T14:38:37.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.671+0000 7fc5535c7640 1 -- 172.21.15.81:0/2287827073 wait complete. 2024-06-29T14:38:37.679 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2024-06-29T14:38:37.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-06-29T14:38:37.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.797+0000 7f8b828cc640 1 -- 172.21.15.81:0/2218285473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b7c06fa90 msgr2=0x7f8b7c06ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:37.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.797+0000 7f8b828cc640 1 --2- 172.21.15.81:0/2218285473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b7c06fa90 0x7f8b7c06ca00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8b74005fd0 tx=0x7f8b7402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:38:37.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.797+0000 7f8b828cc640 1 -- 172.21.15.81:0/2218285473 shutdown_connections 2024-06-29T14:38:37.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.797+0000 7f8b828cc640 1 --2- 172.21.15.81:0/2218285473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b7c06fa90 0x7f8b7c06ca00 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:37.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.797+0000 7f8b828cc640 1 --2- 172.21.15.81:0/2218285473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b7c074520 0x7f8b7c06f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:37.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.797+0000 7f8b828cc640 1 -- 172.21.15.81:0/2218285473 >> 172.21.15.81:0/2218285473 conn(0x7f8b7c072950 msgr2=0x7f8b7c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:37.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.797+0000 7f8b828cc640 1 -- 172.21.15.81:0/2218285473 shutdown_connections 2024-06-29T14:38:37.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.797+0000 7f8b828cc640 1 -- 172.21.15.81:0/2218285473 wait complete. 2024-06-29T14:38:37.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.799+0000 7f8b828cc640 1 Processor -- start 2024-06-29T14:38:37.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.799+0000 7f8b828cc640 1 -- start start 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.800+0000 7f8b828cc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b7c06f680 0x7f8b7c14f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.800+0000 7f8b828cc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b7c06fa90 0x7f8b7c1f5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.800+0000 7f8b828cc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c074520 0x7f8b7c1f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.800+0000 7f8b828cc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7c15b010 con 0x7f8b7c074520 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.800+0000 7f8b828cc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7c06d150 con 0x7f8b7c06f680 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.800+0000 7f8b828cc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8b7c15b190 con 0x7f8b7c06fa90 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.800+0000 7f8b80e69640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c074520 0x7f8b7c1f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.800+0000 7f8b80e69640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c074520 0x7f8b7c1f5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37026/0 (socket says 172.21.15.81:37026) 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.800+0000 7f8b80e69640 1 -- 172.21.15.81:0/2700437983 learned_addr learned my addr 172.21.15.81:0/2700437983 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b7bfff640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b7c06f680 0x7f8b7c14f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:37.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b7b7fe640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b7c06fa90 0x7f8b7c1f5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b80e69640 1 -- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b7c06fa90 msgr2=0x7f8b7c1f5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b80e69640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b7c06fa90 0x7f8b7c1f5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b80e69640 1 -- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b7c06f680 msgr2=0x7f8b7c14f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b80e69640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b7c06f680 0x7f8b7c14f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b80e69640 1 -- 172.21.15.81:0/2700437983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7c150380 con 0x7f8b7c074520 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b7bfff640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b7c06f680 0x7f8b7c14f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b80e69640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c074520 0x7f8b7c1f5dd0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f8b7402aea0 tx=0x7f8b740316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b78ff9640 1 -- 172.21.15.81:0/2700437983 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74031990 con 0x7f8b7c074520 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b78ff9640 1 -- 172.21.15.81:0/2700437983 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b74031c20 con 0x7f8b7c074520 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b78ff9640 1 -- 172.21.15.81:0/2700437983 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b740330d0 con 0x7f8b7c074520 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b828cc640 1 -- 172.21.15.81:0/2700437983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b7c150660 con 0x7f8b7c074520 2024-06-29T14:38:37.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.801+0000 7f8b828cc640 1 -- 172.21.15.81:0/2700437983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b7c150c20 con 0x7f8b7c074520 2024-06-29T14:38:37.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.802+0000 7f8b828cc640 1 -- 172.21.15.81:0/2700437983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b7c068ef0 con 0x7f8b7c074520 2024-06-29T14:38:37.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.803+0000 7f8b78ff9640 1 -- 172.21.15.81:0/2700437983 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 49) ==== 98056+0+0 (secure 0 0 0) 0x7f8b74033270 con 0x7f8b7c074520 2024-06-29T14:38:37.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.803+0000 7f8b78ff9640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8b68076160 0x7f8b68078670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:38:37.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.803+0000 7f8b78ff9640 1 -- 172.21.15.81:0/2700437983 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6125+0+0 (secure 0 0 0) 0x7f8b740b8a20 con 0x7f8b7c074520 2024-06-29T14:38:37.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.803+0000 7f8b7bfff640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8b68076160 0x7f8b68078670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:38:37.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.804+0000 7f8b7bfff640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8b68076160 0x7f8b68078670 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c009b80 tx=0x7f8b6c005f90 comp rx=0 tx=0).ready entity=mgr.6443 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:38:37.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:37.808+0000 7f8b78ff9640 1 -- 172.21.15.81:0/2700437983 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8b7403f030 con 0x7f8b7c074520 2024-06-29T14:38:38.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.005+0000 7f8b828cc640 1 -- 172.21.15.81:0/2700437983 --> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8b7c0af5b0 con 0x7f8b68076160 2024-06-29T14:38:38.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.012+0000 7f8b78ff9640 1 -- 172.21.15.81:0/2700437983 <== mgr.6443 v2:172.21.15.167:6832/295682379 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+302 (secure 0 0 0) 0x7f8b7c0af5b0 con 0x7f8b68076160 2024-06-29T14:38:38.010 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:38:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:38:37", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-06-29T14:38:38.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 -- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8b68076160 msgr2=0x7f8b68078670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:38.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8b68076160 0x7f8b68078670 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c009b80 tx=0x7f8b6c005f90 comp rx=0 tx=0).stop 2024-06-29T14:38:38.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 -- 172.21.15.81:0/2700437983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c074520 msgr2=0x7f8b7c1f5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:38:38.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c074520 0x7f8b7c1f5dd0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f8b7402aea0 tx=0x7f8b740316d0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 -- 172.21.15.81:0/2700437983 shutdown_connections 2024-06-29T14:38:38.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8b68076160 0x7f8b68078670 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c074520 0x7f8b7c1f5dd0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b7c06fa90 0x7f8b7c1f5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 --2- 172.21.15.81:0/2700437983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b7c06f680 0x7f8b7c14f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:38:38.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 -- 172.21.15.81:0/2700437983 >> 172.21.15.81:0/2700437983 conn(0x7f8b7c072950 msgr2=0x7f8b7c0b0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:38:38.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 -- 172.21.15.81:0/2700437983 shutdown_connections 2024-06-29T14:38:38.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:38:38.014+0000 7f8b828cc640 1 -- 172.21.15.81:0/2700437983 wait complete. 2024-06-29T14:38:38.023 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:38:37", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-06-29T14:38:41.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:38:41.363 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:39:00.679 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:00.680+0000 7f21e5945640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:00.679 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:00.680+0000 7f21e5945640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:00.679 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:00.680+0000 7f21e5945640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:00.679 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:00.680+0000 7f21e5945640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:00.679 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:00.680+0000 7f21e5945640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:11.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:39:11.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:39:38.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2024-06-29T14:39:38.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.157+0000 7f1e3fcba640 1 -- 172.21.15.81:0/1981738023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e38073030 msgr2=0x7f1e38073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.157+0000 7f1e3fcba640 1 --2- 172.21.15.81:0/1981738023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e38073030 0x7f1e38073440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c005fd0 tx=0x7f1e2c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.157+0000 7f1e3fcba640 1 -- 172.21.15.81:0/1981738023 shutdown_connections 2024-06-29T14:39:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.157+0000 7f1e3fcba640 1 --2- 172.21.15.81:0/1981738023 >> v2:172.21.15.167:3300/0 conn(0x7f1e3815e960 0x7f1e3815ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.157+0000 7f1e3fcba640 1 --2- 172.21.15.81:0/1981738023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e38073030 0x7f1e38073440 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.157+0000 7f1e3fcba640 1 -- 172.21.15.81:0/1981738023 >> 172.21.15.81:0/1981738023 conn(0x7f1e3806ccc0 msgr2=0x7f1e380b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.157+0000 7f1e3fcba640 1 -- 172.21.15.81:0/1981738023 shutdown_connections 2024-06-29T14:39:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.157+0000 7f1e3fcba640 1 -- 172.21.15.81:0/1981738023 wait complete. 2024-06-29T14:39:38.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.158+0000 7f1e3fcba640 1 Processor -- start 2024-06-29T14:39:38.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.158+0000 7f1e3fcba640 1 -- start start 2024-06-29T14:39:38.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3fcba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e38073030 0x7f1e380a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3fcba640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e381412f0 0x7f1e381f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3fcba640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e3815e960 0x7f1e380ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3fcba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1e380b3390 con 0x7f1e38073030 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3fcba640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1e380b3210 con 0x7f1e3815e960 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3fcba640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1e380b3510 con 0x7f1e381412f0 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3da56640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e38073030 0x7f1e380a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3da56640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e38073030 0x7f1e380a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54010/0 (socket says 172.21.15.81:54010) 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3da56640 1 -- 172.21.15.81:0/4140185816 learned_addr learned my addr 172.21.15.81:0/4140185816 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3d255640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e381412f0 0x7f1e381f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3e257640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e3815e960 0x7f1e380ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3da56640 1 -- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e381412f0 msgr2=0x7f1e381f3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3da56640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e381412f0 0x7f1e381f3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3da56640 1 -- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e3815e960 msgr2=0x7f1e380ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3da56640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e3815e960 0x7f1e380ab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3da56640 1 -- 172.21.15.81:0/4140185816 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e380a9d10 con 0x7f1e38073030 2024-06-29T14:39:38.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3e257640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e3815e960 0x7f1e380ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:38.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e3da56640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e38073030 0x7f1e380a94c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800c3a0 tx=0x7f1e2800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:38.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.160+0000 7f1e367fc640 1 -- 172.21.15.81:0/4140185816 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e28017070 con 0x7f1e38073030 2024-06-29T14:39:38.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.161+0000 7f1e367fc640 1 -- 172.21.15.81:0/4140185816 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e280053c0 con 0x7f1e38073030 2024-06-29T14:39:38.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.161+0000 7f1e3fcba640 1 -- 172.21.15.81:0/4140185816 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e380a9ff0 con 0x7f1e38073030 2024-06-29T14:39:38.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.161+0000 7f1e367fc640 1 -- 172.21.15.81:0/4140185816 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e28003850 con 0x7f1e38073030 2024-06-29T14:39:38.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.161+0000 7f1e3fcba640 1 -- 172.21.15.81:0/4140185816 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e380aa5b0 con 0x7f1e38073030 2024-06-29T14:39:38.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.162+0000 7f1e3fcba640 1 -- 172.21.15.81:0/4140185816 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e380b3a10 con 0x7f1e38073030 2024-06-29T14:39:38.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.162+0000 7f1e367fc640 1 -- 172.21.15.81:0/4140185816 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 52) ==== 98434+0+0 (secure 0 0 0) 0x7f1e2801a070 con 0x7f1e38073030 2024-06-29T14:39:38.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.163+0000 7f1e367fc640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f1e24076560 0x7f1e24078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.163+0000 7f1e367fc640 1 -- 172.21.15.81:0/4140185816 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7f1e28097140 con 0x7f1e38073030 2024-06-29T14:39:38.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.163+0000 7f1e3d255640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f1e24076560 0x7f1e24078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.164+0000 7f1e3d255640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f1e24076560 0x7f1e24078a70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c033000 tx=0x7f1e2c004730 comp rx=0 tx=0).ready entity=mgr.6443 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:38.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.168+0000 7f1e367fc640 1 -- 172.21.15.81:0/4140185816 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1e38073980 con 0x7f1e38073030 2024-06-29T14:39:38.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.371+0000 7f1e3fcba640 1 -- 172.21.15.81:0/4140185816 --> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1e3815c6f0 con 0x7f1e24076560 2024-06-29T14:39:38.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.379+0000 7f1e367fc640 1 -- 172.21.15.81:0/4140185816 <== mgr.6443 v2:172.21.15.167:6832/295682379 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+303 (secure 0 0 0) 0x7f1e3815c6f0 con 0x7f1e24076560 2024-06-29T14:39:38.378 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:38.378 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:38:37", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-06-29T14:39:38.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 -- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f1e24076560 msgr2=0x7f1e24078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f1e24076560 0x7f1e24078a70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c033000 tx=0x7f1e2c004730 comp rx=0 tx=0).stop 2024-06-29T14:39:38.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 -- 172.21.15.81:0/4140185816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e38073030 msgr2=0x7f1e380a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e38073030 0x7f1e380a94c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800c3a0 tx=0x7f1e2800c860 comp rx=0 tx=0).stop 2024-06-29T14:39:38.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 -- 172.21.15.81:0/4140185816 shutdown_connections 2024-06-29T14:39:38.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f1e24076560 0x7f1e24078a70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e3815e960 0x7f1e380ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e381412f0 0x7f1e381f3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 --2- 172.21.15.81:0/4140185816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e38073030 0x7f1e380a94c0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 -- 172.21.15.81:0/4140185816 >> 172.21.15.81:0/4140185816 conn(0x7f1e3806ccc0 msgr2=0x7f1e3806f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:38.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 -- 172.21.15.81:0/4140185816 shutdown_connections 2024-06-29T14:39:38.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.381+0000 7f1e3fcba640 1 -- 172.21.15.81:0/4140185816 wait complete. 2024-06-29T14:39:38.390 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:38:37", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2024-06-29T14:39:38.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2024-06-29T14:39:38.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.508+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3377781172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc9415e710 msgr2=0x7fbc9406f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.508+0000 7fbc9bc3d640 1 --2- 172.21.15.81:0/3377781172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc9415e710 0x7fbc9406f230 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fbc94070ea0 tx=0x7fbc9002a4d0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.508+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3377781172 shutdown_connections 2024-06-29T14:39:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.508+0000 7fbc9bc3d640 1 --2- 172.21.15.81:0/3377781172 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc9406c830 0x7fbc940af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.508+0000 7fbc9bc3d640 1 --2- 172.21.15.81:0/3377781172 >> v2:172.21.15.167:3300/0 conn(0x7fbc94154fc0 0x7fbc9406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.508+0000 7fbc9bc3d640 1 --2- 172.21.15.81:0/3377781172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc9415e710 0x7fbc9406f230 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.508+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3377781172 >> 172.21.15.81:0/3377781172 conn(0x7fbc940d0930 msgr2=0x7fbc940d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.508+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3377781172 shutdown_connections 2024-06-29T14:39:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.508+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3377781172 wait complete. 2024-06-29T14:39:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.509+0000 7fbc9bc3d640 1 Processor -- start 2024-06-29T14:39:38.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.510+0000 7fbc9bc3d640 1 -- start start 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc9bc3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc9406c830 0x7fbc94156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc9bc3d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc94154fc0 0x7fbc9414b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc9bc3d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbc9415e710 0x7fbc94157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc9bc3d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbc940affb0 con 0x7fbc9406c830 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc9bc3d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbc940afe30 con 0x7fbc9415e710 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc9bc3d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbc940b0130 con 0x7fbc94154fc0 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc999d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc9406c830 0x7fbc94156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc999d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc9406c830 0x7fbc94156ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54022/0 (socket says 172.21.15.81:54022) 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc999d9640 1 -- 172.21.15.81:0/3187165732 learned_addr learned my addr 172.21.15.81:0/3187165732 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc9a1da640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbc9415e710 0x7fbc94157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc991d8640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc94154fc0 0x7fbc9414b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc999d9640 1 -- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc94154fc0 msgr2=0x7fbc9414b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc999d9640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc94154fc0 0x7fbc9414b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc999d9640 1 -- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbc9415e710 msgr2=0x7fbc94157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc999d9640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbc9415e710 0x7fbc94157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc999d9640 1 -- 172.21.15.81:0/3187165732 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc94155a40 con 0x7fbc9406c830 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.511+0000 7fbc999d9640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc9406c830 0x7fbc94156ae0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fbc90000c00 tx=0x7fbc900039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.512+0000 7fbc8a7fc640 1 -- 172.21.15.81:0/3187165732 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9003e030 con 0x7fbc9406c830 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.512+0000 7fbc8a7fc640 1 -- 172.21.15.81:0/3187165732 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc900047a0 con 0x7fbc9406c830 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.512+0000 7fbc8a7fc640 1 -- 172.21.15.81:0/3187165732 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc90035040 con 0x7fbc9406c830 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.512+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3187165732 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc94155d20 con 0x7fbc9406c830 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.512+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3187165732 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc941562e0 con 0x7fbc9406c830 2024-06-29T14:39:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.513+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3187165732 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc9406e2a0 con 0x7fbc9406c830 2024-06-29T14:39:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.513+0000 7fbc8a7fc640 1 -- 172.21.15.81:0/3187165732 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 52) ==== 98434+0+0 (secure 0 0 0) 0x7fbc90041030 con 0x7fbc9406c830 2024-06-29T14:39:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.514+0000 7fbc8a7fc640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fbc78076590 0x7fbc78078aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.514+0000 7fbc8a7fc640 1 -- 172.21.15.81:0/3187165732 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7fbc900b8570 con 0x7fbc9406c830 2024-06-29T14:39:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.514+0000 7fbc991d8640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fbc78076590 0x7fbc78078aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.515+0000 7fbc991d8640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fbc78076590 0x7fbc78078aa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400a490 tx=0x7fbc8400a420 comp rx=0 tx=0).ready entity=mgr.6443 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:38.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.519+0000 7fbc8a7fc640 1 -- 172.21.15.81:0/3187165732 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbc9406e2a0 con 0x7fbc9406c830 2024-06-29T14:39:38.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.721+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3187165732 --> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) -- 0x7fbc9406c2f0 con 0x7fbc78076590 2024-06-29T14:39:38.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.800+0000 7fbc8a7fc640 1 -- 172.21.15.81:0/3187165732 <== mgr.6443 v2:172.21.15.167:6832/295682379 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+36 (secure 0 0 0) 0x7fbc9406c2f0 con 0x7fbc78076590 2024-06-29T14:39:38.797 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /bad-path 2024-06-29T14:39:38.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fbc78076590 msgr2=0x7fbc78078aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fbc78076590 0x7fbc78078aa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400a490 tx=0x7fbc8400a420 comp rx=0 tx=0).stop 2024-06-29T14:39:38.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3187165732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc9406c830 msgr2=0x7fbc94156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc9406c830 0x7fbc94156ae0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fbc90000c00 tx=0x7fbc900039e0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3187165732 shutdown_connections 2024-06-29T14:39:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fbc78076590 0x7fbc78078aa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbc9415e710 0x7fbc94157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbc94154fc0 0x7fbc9414b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 --2- 172.21.15.81:0/3187165732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc9406c830 0x7fbc94156ae0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3187165732 >> 172.21.15.81:0/3187165732 conn(0x7fbc940d0930 msgr2=0x7fbc94161540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3187165732 shutdown_connections 2024-06-29T14:39:38.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.802+0000 7fbc9bc3d640 1 -- 172.21.15.81:0/3187165732 wait complete. 2024-06-29T14:39:38.810 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2024-06-29T14:39:38.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:39:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.928+0000 7f59e5d9a640 1 -- 172.21.15.81:0/4088359921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59e015c030 msgr2=0x7f59e006c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.928+0000 7f59e5d9a640 1 --2- 172.21.15.81:0/4088359921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59e015c030 0x7f59e006c2f0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f59e0071e60 tx=0x7f59d402a560 comp rx=0 tx=0).stop 2024-06-29T14:39:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.928+0000 7f59e5d9a640 1 -- 172.21.15.81:0/4088359921 shutdown_connections 2024-06-29T14:39:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.928+0000 7f59e5d9a640 1 --2- 172.21.15.81:0/4088359921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59e015c030 0x7f59e006c2f0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.928+0000 7f59e5d9a640 1 --2- 172.21.15.81:0/4088359921 >> v2:172.21.15.167:3300/0 conn(0x7f59e015d750 0x7f59e00afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.928+0000 7f59e5d9a640 1 -- 172.21.15.81:0/4088359921 >> 172.21.15.81:0/4088359921 conn(0x7f59e00d0930 msgr2=0x7f59e00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.928+0000 7f59e5d9a640 1 -- 172.21.15.81:0/4088359921 shutdown_connections 2024-06-29T14:39:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.928+0000 7f59e5d9a640 1 -- 172.21.15.81:0/4088359921 wait complete. 2024-06-29T14:39:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.930+0000 7f59e5d9a640 1 Processor -- start 2024-06-29T14:39:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.930+0000 7f59e5d9a640 1 -- start start 2024-06-29T14:39:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59e5d9a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59e015c030 0x7f59e0147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59e5d9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59e015c440 0x7f59e0154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59e5d9a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59e015d750 0x7f59e0070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59e5d9a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f59e006cbc0 con 0x7f59e015c440 2024-06-29T14:39:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59e5d9a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f59e006ca40 con 0x7f59e015d750 2024-06-29T14:39:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59e5d9a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f59e0160290 con 0x7f59e015c030 2024-06-29T14:39:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59deffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59e015c440 0x7f59e0154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59deffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59e015c440 0x7f59e0154bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54026/0 (socket says 172.21.15.81:54026) 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59deffd640 1 -- 172.21.15.81:0/2574968360 learned_addr learned my addr 172.21.15.81:0/2574968360 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59dffff640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59e015d750 0x7f59e0070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59df7fe640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59e015c030 0x7f59e0147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59deffd640 1 -- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59e015c030 msgr2=0x7f59e0147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59deffd640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59e015c030 0x7f59e0147ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59deffd640 1 -- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59e015d750 msgr2=0x7f59e0070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59deffd640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59e015d750 0x7f59e0070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.931+0000 7f59deffd640 1 -- 172.21.15.81:0/2574968360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e0148530 con 0x7f59e015c440 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.932+0000 7f59df7fe640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59e015c030 0x7f59e0147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.932+0000 7f59deffd640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59e015c440 0x7f59e0154bb0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f59d4034040 tx=0x7f59d40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.932+0000 7f59c7fff640 1 -- 172.21.15.81:0/2574968360 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d403e030 con 0x7f59e015c440 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.932+0000 7f59c7fff640 1 -- 172.21.15.81:0/2574968360 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59d40047a0 con 0x7f59e015c440 2024-06-29T14:39:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.932+0000 7f59c7fff640 1 -- 172.21.15.81:0/2574968360 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4035040 con 0x7f59e015c440 2024-06-29T14:39:38.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.932+0000 7f59e5d9a640 1 -- 172.21.15.81:0/2574968360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e01f6a60 con 0x7f59e015c440 2024-06-29T14:39:38.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.932+0000 7f59e5d9a640 1 -- 172.21.15.81:0/2574968360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e01f7020 con 0x7f59e015c440 2024-06-29T14:39:38.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.933+0000 7f59e5d9a640 1 -- 172.21.15.81:0/2574968360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e006e2a0 con 0x7f59e015c440 2024-06-29T14:39:38.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.934+0000 7f59c7fff640 1 -- 172.21.15.81:0/2574968360 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 52) ==== 98434+0+0 (secure 0 0 0) 0x7f59d4041030 con 0x7f59e015c440 2024-06-29T14:39:38.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.934+0000 7f59c7fff640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f59cc076560 0x7f59cc078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:38.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.934+0000 7f59c7fff640 1 -- 172.21.15.81:0/2574968360 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7f59d40b8540 con 0x7f59e015c440 2024-06-29T14:39:38.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.934+0000 7f59df7fe640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f59cc076560 0x7f59cc078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:38.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.935+0000 7f59df7fe640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f59cc076560 0x7f59cc078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f59d0005fd0 tx=0x7f59d000f360 comp rx=0 tx=0).ready entity=mgr.6443 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:38.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:38.939+0000 7f59c7fff640 1 -- 172.21.15.81:0/2574968360 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f59e006e2a0 con 0x7f59e015c440 2024-06-29T14:39:39.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.169+0000 7f59e5d9a640 1 -- 172.21.15.81:0/2574968360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f59e0160740 con 0x7f59e015c440 2024-06-29T14:39:39.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.742+0000 7f59c7fff640 1 -- 172.21.15.81:0/2574968360 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v53) ==== 93+0+0 (secure 0 0 0) 0x7f59d4081240 con 0x7f59e015c440 2024-06-29T14:39:39.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.745+0000 7f59e5d9a640 1 -- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f59cc076560 msgr2=0x7f59cc078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:39.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.745+0000 7f59e5d9a640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f59cc076560 0x7f59cc078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f59d0005fd0 tx=0x7f59d000f360 comp rx=0 tx=0).stop 2024-06-29T14:39:39.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.745+0000 7f59e5d9a640 1 -- 172.21.15.81:0/2574968360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59e015c440 msgr2=0x7f59e0154bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:39.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.745+0000 7f59e5d9a640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59e015c440 0x7f59e0154bb0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f59d4034040 tx=0x7f59d40039e0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.745+0000 7f59e5d9a640 1 -- 172.21.15.81:0/2574968360 shutdown_connections 2024-06-29T14:39:39.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.745+0000 7f59e5d9a640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f59cc076560 0x7f59cc078a70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.745+0000 7f59e5d9a640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59e015d750 0x7f59e0070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.745+0000 7f59e5d9a640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59e015c440 0x7f59e0154bb0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.745+0000 7f59e5d9a640 1 --2- 172.21.15.81:0/2574968360 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59e015c030 0x7f59e0147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:39.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.745+0000 7f59e5d9a640 1 -- 172.21.15.81:0/2574968360 >> 172.21.15.81:0/2574968360 conn(0x7f59e00d0930 msgr2=0x7f59e0068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:39.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.746+0000 7f59e5d9a640 1 -- 172.21.15.81:0/2574968360 shutdown_connections 2024-06-29T14:39:39.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:39.746+0000 7f59e5d9a640 1 -- 172.21.15.81:0/2574968360 wait complete. 2024-06-29T14:39:39.756 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:39:39.756 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:39:39.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:39:39.837 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:39:39.837 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:39:39.837 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:39:39.837 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:39.853 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:39:39.853 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:39.908 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.065+0000 7ff05399a640 1 -- 172.21.15.81:0/3670151738 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff04c0b9040 msgr2=0x7ff04c0d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.065+0000 7ff05399a640 1 --2- 172.21.15.81:0/3670151738 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff04c0b9040 0x7ff04c0d0480 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009860 tx=0x7ff03c02a620 comp rx=0 tx=0).stop 2024-06-29T14:39:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.065+0000 7ff05399a640 1 -- 172.21.15.81:0/3670151738 shutdown_connections 2024-06-29T14:39:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.065+0000 7ff05399a640 1 --2- 172.21.15.81:0/3670151738 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff04c0b9040 0x7ff04c0d0480 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.065+0000 7ff05399a640 1 -- 172.21.15.81:0/3670151738 >> 172.21.15.81:0/3670151738 conn(0x7ff04c09cf30 msgr2=0x7ff04c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.065+0000 7ff05399a640 1 -- 172.21.15.81:0/3670151738 shutdown_connections 2024-06-29T14:39:40.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.065+0000 7ff05399a640 1 -- 172.21.15.81:0/3670151738 wait complete. 2024-06-29T14:39:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.067+0000 7ff05399a640 1 Processor -- start 2024-06-29T14:39:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.067+0000 7ff05399a640 1 -- start start 2024-06-29T14:39:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff05399a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff04c0b5250 0x7ff04c09f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff05399a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff04c0b9040 0x7ff04c15c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff05399a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff04c0d2a90 0x7ff04c0a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff05399a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c0d0e00 con 0x7ff04c0d2a90 2024-06-29T14:39:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff05399a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c0d0c80 con 0x7ff04c0b5250 2024-06-29T14:39:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff05399a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff04c0b80b0 con 0x7ff04c0b9040 2024-06-29T14:39:40.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff051f37640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff04c0d2a90 0x7ff04c0a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff051f37640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff04c0d2a90 0x7ff04c0a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53910/0 (socket says 172.21.15.81:53910) 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff051f37640 1 -- 172.21.15.81:0/3164810831 learned_addr learned my addr 172.21.15.81:0/3164810831 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff051736640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff04c0b5250 0x7ff04c09f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.068+0000 7ff050f35640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff04c0b9040 0x7ff04c15c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff051f37640 1 -- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff04c0b9040 msgr2=0x7ff04c15c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff051f37640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff04c0b9040 0x7ff04c15c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff051f37640 1 -- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff04c0b5250 msgr2=0x7ff04c09f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff051f37640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff04c0b5250 0x7ff04c09f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff051f37640 1 -- 172.21.15.81:0/3164810831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c0c7730 con 0x7ff04c0d2a90 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff051736640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff04c0b5250 0x7ff04c09f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff051f37640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff04c0d2a90 0x7ff04c0a1150 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7ff04400c760 tx=0x7ff04400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff041ffb640 1 -- 172.21.15.81:0/3164810831 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044003b70 con 0x7ff04c0d2a90 2024-06-29T14:39:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff041ffb640 1 -- 172.21.15.81:0/3164810831 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff044003d50 con 0x7ff04c0d2a90 2024-06-29T14:39:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff05399a640 1 -- 172.21.15.81:0/3164810831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff04c0c7a10 con 0x7ff04c0d2a90 2024-06-29T14:39:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff041ffb640 1 -- 172.21.15.81:0/3164810831 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04400aab0 con 0x7ff04c0d2a90 2024-06-29T14:39:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.069+0000 7ff05399a640 1 -- 172.21.15.81:0/3164810831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff04c15c770 con 0x7ff04c0d2a90 2024-06-29T14:39:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.070+0000 7ff05399a640 1 -- 172.21.15.81:0/3164810831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff04c0c74c0 con 0x7ff04c0d2a90 2024-06-29T14:39:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.072+0000 7ff041ffb640 1 -- 172.21.15.81:0/3164810831 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7ff04400ac50 con 0x7ff04c0d2a90 2024-06-29T14:39:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.072+0000 7ff041ffb640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7ff030076540 0x7ff030078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.072+0000 7ff051736640 1 -- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7ff030076540 msgr2=0x7ff030078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.072+0000 7ff041ffb640 1 -- 172.21.15.81:0/3164810831 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7ff044097b70 con 0x7ff04c0d2a90 2024-06-29T14:39:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.072+0000 7ff051736640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7ff030076540 0x7ff030078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:40.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.077+0000 7ff041ffb640 1 -- 172.21.15.81:0/3164810831 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff0440608c0 con 0x7ff04c0d2a90 2024-06-29T14:39:40.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.263+0000 7ff05399a640 1 -- 172.21.15.81:0/3164810831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff04c09fa30 con 0x7ff04c0d2a90 2024-06-29T14:39:40.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.264+0000 7ff041ffb640 1 -- 172.21.15.81:0/3164810831 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+17377 (secure 0 0 0) 0x7ff044064bc0 con 0x7ff04c0d2a90 2024-06-29T14:39:40.262 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:40.262 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":107,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:39:37.720730+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.514776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.175620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.7","mappings":[{"from":6,"to":3}]},{"pgid":"13.20","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3578595575":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2870419655":"2024-06-30T14:38:36.587419+0000","172.21.15.167:6833/2583731609":"2024-06-30T14:38:36.587419+0000","172.21.15.167:6832/2583731609":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/1601099482":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2667854206":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/32033638":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/4125253667":"2024-06-30T14:38:36.587419+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:40.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.266+0000 7ff05399a640 1 -- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7ff030076540 msgr2=0x7ff030078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:40.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.266+0000 7ff05399a640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7ff030076540 0x7ff030078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.266+0000 7ff05399a640 1 -- 172.21.15.81:0/3164810831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff04c0d2a90 msgr2=0x7ff04c0a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.266+0000 7ff05399a640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff04c0d2a90 0x7ff04c0a1150 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7ff04400c760 tx=0x7ff04400cc20 comp rx=0 tx=0).stop 2024-06-29T14:39:40.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.266+0000 7ff05399a640 1 -- 172.21.15.81:0/3164810831 shutdown_connections 2024-06-29T14:39:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.266+0000 7ff05399a640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7ff030076540 0x7ff030078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.266+0000 7ff05399a640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff04c0d2a90 0x7ff04c0a1150 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.266+0000 7ff05399a640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff04c0b9040 0x7ff04c15c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.266+0000 7ff05399a640 1 --2- 172.21.15.81:0/3164810831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff04c0b5250 0x7ff04c09f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.266+0000 7ff05399a640 1 -- 172.21.15.81:0/3164810831 >> 172.21.15.81:0/3164810831 conn(0x7ff04c09cf30 msgr2=0x7ff04c15ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.267+0000 7ff05399a640 1 -- 172.21.15.81:0/3164810831 shutdown_connections 2024-06-29T14:39:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.267+0000 7ff05399a640 1 -- 172.21.15.81:0/3164810831 wait complete. 2024-06-29T14:39:40.276 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:38:22.514776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:38:23.175620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:39:40.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:39:40.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.398+0000 7fe496968640 1 -- 172.21.15.81:0/4235531636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4900cd150 msgr2=0x7fe490072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.398+0000 7fe496968640 1 --2- 172.21.15.81:0/4235531636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4900cd150 0x7fe490072f40 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fe490072cf0 tx=0x7fe48002a560 comp rx=0 tx=0).stop 2024-06-29T14:39:40.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.398+0000 7fe496968640 1 -- 172.21.15.81:0/4235531636 shutdown_connections 2024-06-29T14:39:40.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.398+0000 7fe496968640 1 --2- 172.21.15.81:0/4235531636 >> v2:172.21.15.167:3300/0 conn(0x7fe49015bdf0 0x7fe49006ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.398+0000 7fe496968640 1 --2- 172.21.15.81:0/4235531636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4900cd150 0x7fe490072f40 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.398+0000 7fe496968640 1 -- 172.21.15.81:0/4235531636 >> 172.21.15.81:0/4235531636 conn(0x7fe4900b2920 msgr2=0x7fe4900b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:40.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.398+0000 7fe496968640 1 -- 172.21.15.81:0/4235531636 shutdown_connections 2024-06-29T14:39:40.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.398+0000 7fe496968640 1 -- 172.21.15.81:0/4235531636 wait complete. 2024-06-29T14:39:40.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.400+0000 7fe496968640 1 Processor -- start 2024-06-29T14:39:40.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.400+0000 7fe496968640 1 -- start start 2024-06-29T14:39:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.401+0000 7fe496968640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe490073480 0x7fe4901571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.401+0000 7fe496968640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4900cd150 0x7fe4901f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.401+0000 7fe496968640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe49015bdf0 0x7fe4900cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.401+0000 7fe496968640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe49006f450 con 0x7fe49015bdf0 2024-06-29T14:39:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.401+0000 7fe496968640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe49006f2d0 con 0x7fe490073480 2024-06-29T14:39:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.401+0000 7fe496968640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe49006f5d0 con 0x7fe4900cd150 2024-06-29T14:39:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe494f05640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe49015bdf0 0x7fe4900cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe494f05640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe49015bdf0 0x7fe4900cfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53922/0 (socket says 172.21.15.81:53922) 2024-06-29T14:39:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe494f05640 1 -- 172.21.15.81:0/2524104297 learned_addr learned my addr 172.21.15.81:0/2524104297 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:40.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe48ffff640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe490073480 0x7fe4901571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe487fff640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4900cd150 0x7fe4901f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe494f05640 1 -- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4900cd150 msgr2=0x7fe4901f5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe494f05640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4900cd150 0x7fe4901f5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe494f05640 1 -- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe490073480 msgr2=0x7fe4901571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe494f05640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe490073480 0x7fe4901571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe494f05640 1 -- 172.21.15.81:0/2524104297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe490157710 con 0x7fe49015bdf0 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe494f05640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe49015bdf0 0x7fe4900cfcc0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fe48800ea70 tx=0x7fe48800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe48d7fa640 1 -- 172.21.15.81:0/2524104297 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48800ce40 con 0x7fe49015bdf0 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe48d7fa640 1 -- 172.21.15.81:0/2524104297 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe488004c10 con 0x7fe49015bdf0 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe496968640 1 -- 172.21.15.81:0/2524104297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4901f61f0 con 0x7fe49015bdf0 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe48d7fa640 1 -- 172.21.15.81:0/2524104297 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe488005020 con 0x7fe49015bdf0 2024-06-29T14:39:40.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.402+0000 7fe496968640 1 -- 172.21.15.81:0/2524104297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe490156410 con 0x7fe49015bdf0 2024-06-29T14:39:40.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.403+0000 7fe496968640 1 -- 172.21.15.81:0/2524104297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe490072f40 con 0x7fe49015bdf0 2024-06-29T14:39:40.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.404+0000 7fe48d7fa640 1 -- 172.21.15.81:0/2524104297 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7fe4880051c0 con 0x7fe49015bdf0 2024-06-29T14:39:40.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.404+0000 7fe48d7fa640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fe470076540 0x7fe470078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.404+0000 7fe48ffff640 1 -- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fe470076540 msgr2=0x7fe470078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:40.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.404+0000 7fe48d7fa640 1 -- 172.21.15.81:0/2524104297 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7fe488098c30 con 0x7fe49015bdf0 2024-06-29T14:39:40.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.405+0000 7fe48ffff640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fe470076540 0x7fe470078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:40.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.409+0000 7fe48d7fa640 1 -- 172.21.15.81:0/2524104297 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe4901f61f0 con 0x7fe49015bdf0 2024-06-29T14:39:40.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.594+0000 7fe496968640 1 -- 172.21.15.81:0/2524104297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe4900b3130 con 0x7fe49015bdf0 2024-06-29T14:39:40.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.595+0000 7fe48d7fa640 1 -- 172.21.15.81:0/2524104297 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) ==== 93+0+10 (secure 0 0 0) 0x7fe4880617e0 con 0x7fe49015bdf0 2024-06-29T14:39:40.592 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:39:40.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 -- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fe470076540 msgr2=0x7fe470078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:40.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fe470076540 0x7fe470078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 -- 172.21.15.81:0/2524104297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe49015bdf0 msgr2=0x7fe4900cfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe49015bdf0 0x7fe4900cfcc0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fe48800ea70 tx=0x7fe48800ef30 comp rx=0 tx=0).stop 2024-06-29T14:39:40.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 -- 172.21.15.81:0/2524104297 shutdown_connections 2024-06-29T14:39:40.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fe470076540 0x7fe470078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe49015bdf0 0x7fe4900cfcc0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4900cd150 0x7fe4901f5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 --2- 172.21.15.81:0/2524104297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe490073480 0x7fe4901571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 -- 172.21.15.81:0/2524104297 >> 172.21.15.81:0/2524104297 conn(0x7fe4900b2920 msgr2=0x7fe4900b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:40.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 -- 172.21.15.81:0/2524104297 shutdown_connections 2024-06-29T14:39:40.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.597+0000 7fe496968640 1 -- 172.21.15.81:0/2524104297 wait complete. 2024-06-29T14:39:40.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:39:40.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.730+0000 7f5b05132640 1 -- 172.21.15.81:0/1964810012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5aec005280 msgr2=0x7f5aec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.730+0000 7f5b05132640 1 --2- 172.21.15.81:0/1964810012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5aec005280 0x7f5aec00b9c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f5b000d43d0 tx=0x7f5aec005830 comp rx=0 tx=0).stop 2024-06-29T14:39:40.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.730+0000 7f5b05132640 1 -- 172.21.15.81:0/1964810012 shutdown_connections 2024-06-29T14:39:40.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.730+0000 7f5b05132640 1 --2- 172.21.15.81:0/1964810012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5aec005280 0x7f5aec00b9c0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.730+0000 7f5b05132640 1 -- 172.21.15.81:0/1964810012 >> 172.21.15.81:0/1964810012 conn(0x7f5b0009cf30 msgr2=0x7f5b000d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:40.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.730+0000 7f5b05132640 1 -- 172.21.15.81:0/1964810012 shutdown_connections 2024-06-29T14:39:40.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.730+0000 7f5b05132640 1 -- 172.21.15.81:0/1964810012 wait complete. 2024-06-29T14:39:40.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.732+0000 7f5b05132640 1 Processor -- start 2024-06-29T14:39:40.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.732+0000 7f5b05132640 1 -- start start 2024-06-29T14:39:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5b05132640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5aec005280 0x7f5b001e9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5b05132640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00148300 0x7f5b001f8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5b05132640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b001f8660 0x7f5b001f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5b05132640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5b000d0810 con 0x7f5b00148300 2024-06-29T14:39:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5b05132640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5b000d0690 con 0x7f5b001f8660 2024-06-29T14:39:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5b05132640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5b000d0990 con 0x7f5aec005280 2024-06-29T14:39:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5afe59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00148300 0x7f5b001f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5afe59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00148300 0x7f5b001f8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53954/0 (socket says 172.21.15.81:53954) 2024-06-29T14:39:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5afe59c640 1 -- 172.21.15.81:0/3560692091 learned_addr learned my addr 172.21.15.81:0/3560692091 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5afed9d640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5aec005280 0x7f5b001e9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.733+0000 7f5aff59e640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b001f8660 0x7f5b001f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5afe59c640 1 -- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5aec005280 msgr2=0x7f5b001e9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5afe59c640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5aec005280 0x7f5b001e9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5afe59c640 1 -- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b001f8660 msgr2=0x7f5b001f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5afe59c640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b001f8660 0x7f5b001f3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5afe59c640 1 -- 172.21.15.81:0/3560692091 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b001f8a70 con 0x7f5b00148300 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5afe59c640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00148300 0x7f5b001f8120 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5af0011ab0 tx=0x7f5af0011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5af77fe640 1 -- 172.21.15.81:0/3560692091 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af001e070 con 0x7f5b00148300 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5af77fe640 1 -- 172.21.15.81:0/3560692091 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5af0004460 con 0x7f5b00148300 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5b05132640 1 -- 172.21.15.81:0/3560692091 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0015fad0 con 0x7f5b00148300 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5af77fe640 1 -- 172.21.15.81:0/3560692091 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af000fc00 con 0x7f5b00148300 2024-06-29T14:39:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.734+0000 7f5b05132640 1 -- 172.21.15.81:0/3560692091 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b00160090 con 0x7f5b00148300 2024-06-29T14:39:40.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.735+0000 7f5b05132640 1 -- 172.21.15.81:0/3560692091 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b000b8410 con 0x7f5b00148300 2024-06-29T14:39:40.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.736+0000 7f5af77fe640 1 -- 172.21.15.81:0/3560692091 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f5af0021070 con 0x7f5b00148300 2024-06-29T14:39:40.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.736+0000 7f5af77fe640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f5ae0076540 0x7f5ae0078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:40.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.736+0000 7f5afed9d640 1 -- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f5ae0076540 msgr2=0x7f5ae0078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:40.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.736+0000 7f5af77fe640 1 -- 172.21.15.81:0/3560692091 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7f5af0098770 con 0x7f5b00148300 2024-06-29T14:39:40.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.736+0000 7f5afed9d640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f5ae0076540 0x7f5ae0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.741+0000 7f5af77fe640 1 -- 172.21.15.81:0/3560692091 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5b001f8a70 con 0x7f5b00148300 2024-06-29T14:39:40.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.937+0000 7f5afed9d640 1 -- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f5ae0076540 msgr2=0x7f5ae0078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:40.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.937+0000 7f5afed9d640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f5ae0076540 0x7f5ae0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:39:40.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.947+0000 7f5b05132640 1 -- 172.21.15.81:0/3560692091 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5b0015e8c0 con 0x7f5b00148300 2024-06-29T14:39:40.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.948+0000 7f5af77fe640 1 -- 172.21.15.81:0/3560692091 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) ==== 104+0+11 (secure 0 0 0) 0x7f5af0061430 con 0x7f5b00148300 2024-06-29T14:39:40.945 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:39:40.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 -- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f5ae0076540 msgr2=0x7f5ae0078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f5ae0076540 0x7f5ae0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 -- 172.21.15.81:0/3560692091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00148300 msgr2=0x7f5b001f8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00148300 0x7f5b001f8120 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f5af0011ab0 tx=0x7f5af0011eb0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 -- 172.21.15.81:0/3560692091 shutdown_connections 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f5ae0076540 0x7f5ae0078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5b001f8660 0x7f5b001f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00148300 0x7f5b001f8120 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 --2- 172.21.15.81:0/3560692091 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5aec005280 0x7f5b001e9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 -- 172.21.15.81:0/3560692091 >> 172.21.15.81:0/3560692091 conn(0x7f5b0009cf30 msgr2=0x7f5b00158430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 -- 172.21.15.81:0/3560692091 shutdown_connections 2024-06-29T14:39:40.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:40.950+0000 7f5b05132640 1 -- 172.21.15.81:0/3560692091 wait complete. 2024-06-29T14:39:40.959 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:39:41.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.080+0000 7f8c7bac3640 1 -- 172.21.15.81:0/1662427229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c740cd150 msgr2=0x7f8c74072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.080+0000 7f8c7bac3640 1 --2- 172.21.15.81:0/1662427229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c740cd150 0x7f8c74072f40 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f8c74074310 tx=0x7f8c7002a4a0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.080+0000 7f8c7bac3640 1 -- 172.21.15.81:0/1662427229 shutdown_connections 2024-06-29T14:39:41.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.080+0000 7f8c7bac3640 1 --2- 172.21.15.81:0/1662427229 >> v2:172.21.15.167:3300/0 conn(0x7f8c74073480 0x7f8c7415b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.080+0000 7f8c7bac3640 1 --2- 172.21.15.81:0/1662427229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c740cd150 0x7f8c74072f40 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.080+0000 7f8c7bac3640 1 -- 172.21.15.81:0/1662427229 >> 172.21.15.81:0/1662427229 conn(0x7f8c740b2920 msgr2=0x7f8c740b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.081+0000 7f8c7bac3640 1 -- 172.21.15.81:0/1662427229 shutdown_connections 2024-06-29T14:39:41.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.081+0000 7f8c7bac3640 1 -- 172.21.15.81:0/1662427229 wait complete. 2024-06-29T14:39:41.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.082+0000 7f8c7bac3640 1 Processor -- start 2024-06-29T14:39:41.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.082+0000 7f8c7bac3640 1 -- start start 2024-06-29T14:39:41.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.083+0000 7f8c7bac3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c74073480 0x7f8c74151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7bac3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c740cd150 0x7f8c741f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7bac3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c7415bc20 0x7f8c7406c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7bac3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7415c1b0 con 0x7f8c740cd150 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7bac3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7415c030 con 0x7f8c74073480 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7bac3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8c7415f060 con 0x7f8c7415bc20 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7905e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c740cd150 0x7f8c741f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7905e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c740cd150 0x7f8c741f4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53974/0 (socket says 172.21.15.81:53974) 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7905e640 1 -- 172.21.15.81:0/3613533673 learned_addr learned my addr 172.21.15.81:0/3613533673 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7985f640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c74073480 0x7f8c74151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7a060640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c7415bc20 0x7f8c7406c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7905e640 1 -- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c7415bc20 msgr2=0x7f8c7406c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7905e640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c7415bc20 0x7f8c7406c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7905e640 1 -- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c74073480 msgr2=0x7f8c74151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7905e640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c74073480 0x7f8c74151c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7905e640 1 -- 172.21.15.81:0/3613533673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c740af880 con 0x7f8c740cd150 2024-06-29T14:39:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7a060640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c7415bc20 0x7f8c7406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7905e640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c740cd150 0x7f8c741f4df0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009fd0 tx=0x7f8c6400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c6a7fc640 1 -- 172.21.15.81:0/3613533673 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6400ccc0 con 0x7f8c740cd150 2024-06-29T14:39:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c6a7fc640 1 -- 172.21.15.81:0/3613533673 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c6400ee70 con 0x7f8c740cd150 2024-06-29T14:39:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7bac3640 1 -- 172.21.15.81:0/3613533673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c740afaf0 con 0x7f8c740cd150 2024-06-29T14:39:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c6a7fc640 1 -- 172.21.15.81:0/3613533673 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6400ce60 con 0x7f8c740cd150 2024-06-29T14:39:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.084+0000 7f8c7bac3640 1 -- 172.21.15.81:0/3613533673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c740b00b0 con 0x7f8c740cd150 2024-06-29T14:39:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.085+0000 7f8c7bac3640 1 -- 172.21.15.81:0/3613533673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c7415f4e0 con 0x7f8c740cd150 2024-06-29T14:39:41.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.086+0000 7f8c6a7fc640 1 -- 172.21.15.81:0/3613533673 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f8c64005260 con 0x7f8c740cd150 2024-06-29T14:39:41.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.087+0000 7f8c6a7fc640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8c58076510 0x7f8c58078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.087+0000 7f8c7985f640 1 -- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8c58076510 msgr2=0x7f8c58078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:41.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.087+0000 7f8c6a7fc640 1 -- 172.21.15.81:0/3613533673 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7f8c640986d0 con 0x7f8c740cd150 2024-06-29T14:39:41.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.087+0000 7f8c7985f640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8c58076510 0x7f8c58078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:41.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.091+0000 7f8c6a7fc640 1 -- 172.21.15.81:0/3613533673 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8c64061950 con 0x7f8c740cd150 2024-06-29T14:39:41.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.278+0000 7f8c7bac3640 1 -- 172.21.15.81:0/3613533673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8c74149ac0 con 0x7f8c740cd150 2024-06-29T14:39:41.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.278+0000 7f8c6a7fc640 1 -- 172.21.15.81:0/3613533673 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) ==== 100+0+11 (secure 0 0 0) 0x7f8c64061250 con 0x7f8c740cd150 2024-06-29T14:39:41.276 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:39:41.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.280+0000 7f8c7bac3640 1 -- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8c58076510 msgr2=0x7f8c58078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:41.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.280+0000 7f8c7bac3640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8c58076510 0x7f8c58078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.280+0000 7f8c7bac3640 1 -- 172.21.15.81:0/3613533673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c740cd150 msgr2=0x7f8c741f4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.280+0000 7f8c7bac3640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c740cd150 0x7f8c741f4df0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009fd0 tx=0x7f8c6400ec20 comp rx=0 tx=0).stop 2024-06-29T14:39:41.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.281+0000 7f8c7bac3640 1 -- 172.21.15.81:0/3613533673 shutdown_connections 2024-06-29T14:39:41.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.281+0000 7f8c7bac3640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f8c58076510 0x7f8c58078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.281+0000 7f8c7bac3640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c7415bc20 0x7f8c7406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.281+0000 7f8c7bac3640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c740cd150 0x7f8c741f4df0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.281+0000 7f8c7bac3640 1 --2- 172.21.15.81:0/3613533673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c74073480 0x7f8c74151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.281+0000 7f8c7bac3640 1 -- 172.21.15.81:0/3613533673 >> 172.21.15.81:0/3613533673 conn(0x7f8c740b2920 msgr2=0x7f8c740b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.281+0000 7f8c7bac3640 1 -- 172.21.15.81:0/3613533673 shutdown_connections 2024-06-29T14:39:41.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.281+0000 7f8c7bac3640 1 -- 172.21.15.81:0/3613533673 wait complete. 2024-06-29T14:39:41.290 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:39:41.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.408+0000 7f741fedf640 1 -- 172.21.15.81:0/2763633619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7404004800 msgr2=0x7f7404004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.408+0000 7f741fedf640 1 --2- 172.21.15.81:0/2763633619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7404004800 0x7f7404004c10 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f7408006fc0 tx=0x7f7404005830 comp rx=0 tx=0).stop 2024-06-29T14:39:41.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.408+0000 7f741fedf640 1 -- 172.21.15.81:0/2763633619 shutdown_connections 2024-06-29T14:39:41.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.408+0000 7f741fedf640 1 --2- 172.21.15.81:0/2763633619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7404004800 0x7f7404004c10 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.408+0000 7f741fedf640 1 -- 172.21.15.81:0/2763633619 >> 172.21.15.81:0/2763633619 conn(0x7f74180b2920 msgr2=0x7f74180b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.408+0000 7f741fedf640 1 -- 172.21.15.81:0/2763633619 shutdown_connections 2024-06-29T14:39:41.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.408+0000 7f741fedf640 1 -- 172.21.15.81:0/2763633619 wait complete. 2024-06-29T14:39:41.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.410+0000 7f741fedf640 1 Processor -- start 2024-06-29T14:39:41.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.410+0000 7f741fedf640 1 -- start start 2024-06-29T14:39:41.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.411+0000 7f741fedf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7404004800 0x7f7418158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.411+0000 7f741fedf640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7418143f90 0x7f74180af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.411+0000 7f741fedf640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74180afe90 0x7f741813ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.411+0000 7f741fedf640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f741806c700 con 0x7f7404004800 2024-06-29T14:39:41.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.411+0000 7f741fedf640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f74180700d0 con 0x7f7418143f90 2024-06-29T14:39:41.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.411+0000 7f741fedf640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f741806c880 con 0x7f74180afe90 2024-06-29T14:39:41.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741dc7b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7404004800 0x7f7418158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741dc7b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7404004800 0x7f7418158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53990/0 (socket says 172.21.15.81:53990) 2024-06-29T14:39:41.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741dc7b640 1 -- 172.21.15.81:0/1149002947 learned_addr learned my addr 172.21.15.81:0/1149002947 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:41.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741e47c640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74180afe90 0x7f741813ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741d47a640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7418143f90 0x7f74180af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741dc7b640 1 -- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74180afe90 msgr2=0x7f741813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741dc7b640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74180afe90 0x7f741813ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741dc7b640 1 -- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7418143f90 msgr2=0x7f74180af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741dc7b640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7418143f90 0x7f74180af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741dc7b640 1 -- 172.21.15.81:0/1149002947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74180b02a0 con 0x7f7404004800 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741d47a640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7418143f90 0x7f74180af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741dc7b640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7404004800 0x7f7418158790 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f7414003f00 tx=0x7f7414001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f740e7fc640 1 -- 172.21.15.81:0/1149002947 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414011640 con 0x7f7404004800 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741fedf640 1 -- 172.21.15.81:0/1149002947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74181f2f20 con 0x7f7404004800 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f740e7fc640 1 -- 172.21.15.81:0/1149002947 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7414016020 con 0x7f7404004800 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f740e7fc640 1 -- 172.21.15.81:0/1149002947 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414011a60 con 0x7f7404004800 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.412+0000 7f741fedf640 1 -- 172.21.15.81:0/1149002947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f741813c3f0 con 0x7f7404004800 2024-06-29T14:39:41.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.413+0000 7f741fedf640 1 -- 172.21.15.81:0/1149002947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7418070250 con 0x7f7404004800 2024-06-29T14:39:41.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.414+0000 7f740e7fc640 1 -- 172.21.15.81:0/1149002947 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f7414011c00 con 0x7f7404004800 2024-06-29T14:39:41.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.415+0000 7f740e7fc640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f73fc076540 0x7f73fc078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.415+0000 7f741d47a640 1 -- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f73fc076540 msgr2=0x7f73fc078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:41.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.415+0000 7f740e7fc640 1 -- 172.21.15.81:0/1149002947 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7f7414098ed0 con 0x7f7404004800 2024-06-29T14:39:41.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.415+0000 7f741d47a640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f73fc076540 0x7f73fc078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:41.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.422+0000 7f740e7fc640 1 -- 172.21.15.81:0/1149002947 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7414020020 con 0x7f7404004800 2024-06-29T14:39:41.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:39:41.582 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:39:41.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.615+0000 7f741d47a640 1 -- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f73fc076540 msgr2=0x7f73fc078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:41.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.615+0000 7f741d47a640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f73fc076540 0x7f73fc078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:39:41.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.619+0000 7f741fedf640 1 -- 172.21.15.81:0/1149002947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f741806f330 con 0x7f7404004800 2024-06-29T14:39:41.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.619+0000 7f740e7fc640 1 -- 172.21.15.81:0/1149002947 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v107) ==== 80+0+476 (secure 0 0 0) 0x7f74180b02a0 con 0x7f7404004800 2024-06-29T14:39:41.618 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3578595575 2024-06-30T14:38:36.587419+0000 2024-06-29T14:39:41.619 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2870419655 2024-06-30T14:38:36.587419+0000 2024-06-29T14:39:41.619 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/2583731609 2024-06-30T14:38:36.587419+0000 2024-06-29T14:39:41.619 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/2583731609 2024-06-30T14:38:36.587419+0000 2024-06-29T14:39:41.619 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1601099482 2024-06-30T14:38:36.587419+0000 2024-06-29T14:39:41.619 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2667854206 2024-06-30T14:38:36.587419+0000 2024-06-29T14:39:41.620 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/32033638 2024-06-30T14:38:36.587419+0000 2024-06-29T14:39:41.620 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/4125253667 2024-06-30T14:38:36.587419+0000 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 -- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f73fc076540 msgr2=0x7f73fc078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f73fc076540 0x7f73fc078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 -- 172.21.15.81:0/1149002947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7404004800 msgr2=0x7f7418158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7404004800 0x7f7418158790 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f7414003f00 tx=0x7f7414001630 comp rx=0 tx=0).stop 2024-06-29T14:39:41.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 -- 172.21.15.81:0/1149002947 shutdown_connections 2024-06-29T14:39:41.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f73fc076540 0x7f73fc078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74180afe90 0x7f741813ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7418143f90 0x7f74180af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 --2- 172.21.15.81:0/1149002947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7404004800 0x7f7418158790 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 -- 172.21.15.81:0/1149002947 >> 172.21.15.81:0/1149002947 conn(0x7f74180b2920 msgr2=0x7f741815ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 -- 172.21.15.81:0/1149002947 shutdown_connections 2024-06-29T14:39:41.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.621+0000 7f741fedf640 1 -- 172.21.15.81:0/1149002947 wait complete. 2024-06-29T14:39:41.632 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:39:41.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:39:41.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:39:41.701 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:39:41.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:39:41.702 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:39:41.758 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:39:41.758 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:39:41.758 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:41.774 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:39:41.774 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:39:41.774 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:39:41.774 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:41.814 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:39:41.814 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:39:41.814 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:39:41.814 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:41.830 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:39:41.831 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:39:41.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.973+0000 7f545fab3640 1 -- 172.21.15.81:0/2519719612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54580b4f40 msgr2=0x7f54580b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.973+0000 7f545fab3640 1 --2- 172.21.15.81:0/2519719612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54580b4f40 0x7f54580b5350 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5458074b10 tx=0x7f545002a820 comp rx=0 tx=0).stop 2024-06-29T14:39:41.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.973+0000 7f545fab3640 1 -- 172.21.15.81:0/2519719612 shutdown_connections 2024-06-29T14:39:41.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.973+0000 7f545fab3640 1 --2- 172.21.15.81:0/2519719612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54580b4f40 0x7f54580b5350 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.973+0000 7f545fab3640 1 --2- 172.21.15.81:0/2519719612 >> v2:172.21.15.167:3300/0 conn(0x7f54580d0230 0x7f54580d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.973+0000 7f545fab3640 1 -- 172.21.15.81:0/2519719612 >> 172.21.15.81:0/2519719612 conn(0x7f54580b3680 msgr2=0x7f54580b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:41.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.973+0000 7f545fab3640 1 -- 172.21.15.81:0/2519719612 shutdown_connections 2024-06-29T14:39:41.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.973+0000 7f545fab3640 1 -- 172.21.15.81:0/2519719612 wait complete. 2024-06-29T14:39:41.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.975+0000 7f545fab3640 1 Processor -- start 2024-06-29T14:39:41.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.975+0000 7f545fab3640 1 -- start start 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545fab3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54580b3200 0x7f54581f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545fab3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54580b4f40 0x7f54581e5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545fab3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54580d0230 0x7f545809ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545fab3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f54580b5c20 con 0x7f54580d0230 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545fab3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f54580b5aa0 con 0x7f54580b4f40 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545fab3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f54580b5da0 con 0x7f54580b3200 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545e050640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54580d0230 0x7f545809ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545e050640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54580d0230 0x7f545809ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54006/0 (socket says 172.21.15.81:54006) 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545e050640 1 -- 172.21.15.81:0/2623283424 learned_addr learned my addr 172.21.15.81:0/2623283424 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545d84f640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54580b3200 0x7f54581f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545d04e640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54580b4f40 0x7f54581e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:41.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545e050640 1 -- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54580b3200 msgr2=0x7f54581f2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545e050640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54580b3200 0x7f54581f2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545e050640 1 -- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54580b4f40 msgr2=0x7f54581e5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:41.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545e050640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54580b4f40 0x7f54581e5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:41.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.976+0000 7f545e050640 1 -- 172.21.15.81:0/2623283424 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54581f36e0 con 0x7f54580d0230 2024-06-29T14:39:41.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.977+0000 7f545e050640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54580d0230 0x7f545809ce20 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f5450009b30 tx=0x7f5450032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:41.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.977+0000 7f544e7fc640 1 -- 172.21.15.81:0/2623283424 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450037040 con 0x7f54580d0230 2024-06-29T14:39:41.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.977+0000 7f544e7fc640 1 -- 172.21.15.81:0/2623283424 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5450042070 con 0x7f54580d0230 2024-06-29T14:39:41.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.977+0000 7f545fab3640 1 -- 172.21.15.81:0/2623283424 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54581f39c0 con 0x7f54580d0230 2024-06-29T14:39:41.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.977+0000 7f544e7fc640 1 -- 172.21.15.81:0/2623283424 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54500042a0 con 0x7f54580d0230 2024-06-29T14:39:41.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.977+0000 7f545fab3640 1 -- 172.21.15.81:0/2623283424 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54581f31f0 con 0x7f54580d0230 2024-06-29T14:39:41.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.978+0000 7f545fab3640 1 -- 172.21.15.81:0/2623283424 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54581f3450 con 0x7f54580d0230 2024-06-29T14:39:41.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.979+0000 7f544e7fc640 1 -- 172.21.15.81:0/2623283424 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f5450004440 con 0x7f54580d0230 2024-06-29T14:39:41.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.979+0000 7f544e7fc640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f543c076540 0x7f543c078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:41.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.979+0000 7f545d84f640 1 -- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f543c076540 msgr2=0x7f543c078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:41.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.979+0000 7f544e7fc640 1 -- 172.21.15.81:0/2623283424 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7f54500b8b80 con 0x7f54580d0230 2024-06-29T14:39:41.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.979+0000 7f545d84f640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f543c076540 0x7f543c078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:41.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:41.984+0000 7f544e7fc640 1 -- 172.21.15.81:0/2623283424 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5450081840 con 0x7f54580d0230 2024-06-29T14:39:42.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.180+0000 7f545d84f640 1 -- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f543c076540 msgr2=0x7f543c078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:42.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.180+0000 7f545d84f640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f543c076540 0x7f543c078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:39:42.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.202+0000 7f545fab3640 1 -- 172.21.15.81:0/2623283424 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54580b5350 con 0x7f54580d0230 2024-06-29T14:39:42.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.203+0000 7f544e7fc640 1 -- 172.21.15.81:0/2623283424 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5231 (secure 0 0 0) 0x7f5450085b40 con 0x7f54580d0230 2024-06-29T14:39:42.202 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:42.202 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":51,"btime":"2024-06-29T14:38:27:883643+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1895620554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1895620554},{"type":"v1","addr":"172.21.15.167:6836","nonce":1895620554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":7485,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3391563230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3391563230},{"type":"v1","addr":"172.21.15.81:6834","nonce":3391563230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":7488,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3932182351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3932182351},{"type":"v1","addr":"172.21.15.81:6835","nonce":3932182351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.216390+0000","modified":"2024-06-29T14:38:27.252097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/3193613343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":3193613343},{"type":"v1","addr":"172.21.15.167:6837","nonce":3193613343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":6}]} 2024-06-29T14:39:42.202 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 51 2024-06-29T14:39:42.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 -- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f543c076540 msgr2=0x7f543c078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:42.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f543c076540 0x7f543c078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 -- 172.21.15.81:0/2623283424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54580d0230 msgr2=0x7f545809ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54580d0230 0x7f545809ce20 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f5450009b30 tx=0x7f5450032d10 comp rx=0 tx=0).stop 2024-06-29T14:39:42.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 -- 172.21.15.81:0/2623283424 shutdown_connections 2024-06-29T14:39:42.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f543c076540 0x7f543c078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54580d0230 0x7f545809ce20 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54580b4f40 0x7f54581e5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 --2- 172.21.15.81:0/2623283424 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54580b3200 0x7f54581f2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 -- 172.21.15.81:0/2623283424 >> 172.21.15.81:0/2623283424 conn(0x7f54580b3680 msgr2=0x7f54580d1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:42.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 -- 172.21.15.81:0/2623283424 shutdown_connections 2024-06-29T14:39:42.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.205+0000 7f545fab3640 1 -- 172.21.15.81:0/2623283424 wait complete. 2024-06-29T14:39:42.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:42.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.332+0000 7f1471b35640 1 -- 172.21.15.81:0/1412020308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146c0d0230 msgr2=0x7f146c0d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.332+0000 7f1471b35640 1 --2- 172.21.15.81:0/1412020308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146c0d0230 0x7f146c0d0640 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1454009830 tx=0x7f145402a510 comp rx=0 tx=0).stop 2024-06-29T14:39:42.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.333+0000 7f1471b35640 1 -- 172.21.15.81:0/1412020308 shutdown_connections 2024-06-29T14:39:42.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.333+0000 7f1471b35640 1 --2- 172.21.15.81:0/1412020308 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f146c0b4f40 0x7f146c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.333+0000 7f1471b35640 1 --2- 172.21.15.81:0/1412020308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146c0d0230 0x7f146c0d0640 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.333+0000 7f1471b35640 1 -- 172.21.15.81:0/1412020308 >> 172.21.15.81:0/1412020308 conn(0x7f146c0b3680 msgr2=0x7f146c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:42.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.333+0000 7f1471b35640 1 -- 172.21.15.81:0/1412020308 shutdown_connections 2024-06-29T14:39:42.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.333+0000 7f1471b35640 1 -- 172.21.15.81:0/1412020308 wait complete. 2024-06-29T14:39:42.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.334+0000 7f1471b35640 1 Processor -- start 2024-06-29T14:39:42.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.334+0000 7f1471b35640 1 -- start start 2024-06-29T14:39:42.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1471b35640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f146c0b3200 0x7f146c09dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1471b35640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f146c0b4f40 0x7f146c155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1471b35640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146c0d0230 0x7f146c1592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1471b35640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f146c0b5c20 con 0x7f146c0d0230 2024-06-29T14:39:42.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1471b35640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f146c0b5aa0 con 0x7f146c0b4f40 2024-06-29T14:39:42.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1471b35640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f146c0b5da0 con 0x7f146c0b3200 2024-06-29T14:39:42.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f146bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146c0d0230 0x7f146c1592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f146bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146c0d0230 0x7f146c1592c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54020/0 (socket says 172.21.15.81:54020) 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f146bfff640 1 -- 172.21.15.81:0/4186306495 learned_addr learned my addr 172.21.15.81:0/4186306495 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f146b7fe640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f146c0b3200 0x7f146c09dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1463fff640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f146c0b4f40 0x7f146c155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f146bfff640 1 -- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f146c0b3200 msgr2=0x7f146c09dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f146bfff640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f146c0b3200 0x7f146c09dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f146bfff640 1 -- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f146c0b4f40 msgr2=0x7f146c155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f146bfff640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f146c0b4f40 0x7f146c155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f146bfff640 1 -- 172.21.15.81:0/4186306495 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146c09e190 con 0x7f146c0d0230 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1463fff640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f146c0b4f40 0x7f146c155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f146bfff640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146c0d0230 0x7f146c1592c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f146400ea70 tx=0x7f146400ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1468ff9640 1 -- 172.21.15.81:0/4186306495 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146400ce40 con 0x7f146c0d0230 2024-06-29T14:39:42.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1468ff9640 1 -- 172.21.15.81:0/4186306495 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1464004c10 con 0x7f146c0d0230 2024-06-29T14:39:42.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1471b35640 1 -- 172.21.15.81:0/4186306495 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f146c09e460 con 0x7f146c0d0230 2024-06-29T14:39:42.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.336+0000 7f1468ff9640 1 -- 172.21.15.81:0/4186306495 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464005020 con 0x7f146c0d0230 2024-06-29T14:39:42.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.337+0000 7f1471b35640 1 -- 172.21.15.81:0/4186306495 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f146c09ea20 con 0x7f146c0d0230 2024-06-29T14:39:42.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.338+0000 7f1471b35640 1 -- 172.21.15.81:0/4186306495 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f146c0b5350 con 0x7f146c0d0230 2024-06-29T14:39:42.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.338+0000 7f1468ff9640 1 -- 172.21.15.81:0/4186306495 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f14640051c0 con 0x7f146c0d0230 2024-06-29T14:39:42.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.339+0000 7f1468ff9640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f144c076510 0x7f144c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.339+0000 7f146b7fe640 1 -- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f144c076510 msgr2=0x7f144c078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:42.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.339+0000 7f1468ff9640 1 -- 172.21.15.81:0/4186306495 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7f1464098b70 con 0x7f146c0d0230 2024-06-29T14:39:42.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.339+0000 7f146b7fe640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f144c076510 0x7f144c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:42.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.343+0000 7f1468ff9640 1 -- 172.21.15.81:0/4186306495 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1464061e80 con 0x7f146c0d0230 2024-06-29T14:39:42.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.530+0000 7f1471b35640 1 -- 172.21.15.81:0/4186306495 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f146c15ae60 con 0x7f146c0d0230 2024-06-29T14:39:42.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.531+0000 7f1468ff9640 1 -- 172.21.15.81:0/4186306495 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+17377 (secure 0 0 0) 0x7f1464068e20 con 0x7f146c0d0230 2024-06-29T14:39:42.531 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:42.531 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":107,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:39:37.720730+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.514776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.175620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.7","mappings":[{"from":6,"to":3}]},{"pgid":"13.20","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3578595575":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2870419655":"2024-06-30T14:38:36.587419+0000","172.21.15.167:6833/2583731609":"2024-06-30T14:38:36.587419+0000","172.21.15.167:6832/2583731609":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/1601099482":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2667854206":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/32033638":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/4125253667":"2024-06-30T14:38:36.587419+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:42.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.533+0000 7f1471b35640 1 -- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f144c076510 msgr2=0x7f144c078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:42.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.533+0000 7f1471b35640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f144c076510 0x7f144c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.534+0000 7f1471b35640 1 -- 172.21.15.81:0/4186306495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146c0d0230 msgr2=0x7f146c1592c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.534+0000 7f1471b35640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146c0d0230 0x7f146c1592c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f146400ea70 tx=0x7f146400ef30 comp rx=0 tx=0).stop 2024-06-29T14:39:42.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.534+0000 7f1471b35640 1 -- 172.21.15.81:0/4186306495 shutdown_connections 2024-06-29T14:39:42.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.534+0000 7f1471b35640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f144c076510 0x7f144c078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.534+0000 7f1471b35640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146c0d0230 0x7f146c1592c0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.534+0000 7f1471b35640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f146c0b4f40 0x7f146c155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.534+0000 7f1471b35640 1 --2- 172.21.15.81:0/4186306495 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f146c0b3200 0x7f146c09dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.534+0000 7f1471b35640 1 -- 172.21.15.81:0/4186306495 >> 172.21.15.81:0/4186306495 conn(0x7f146c0b3680 msgr2=0x7f146c0ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:42.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.534+0000 7f1471b35640 1 -- 172.21.15.81:0/4186306495 shutdown_connections 2024-06-29T14:39:42.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.534+0000 7f1471b35640 1 -- 172.21.15.81:0/4186306495 wait complete. 2024-06-29T14:39:42.544 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:38:22.514776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:38:23.175620+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:39:42.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:39:42.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.664+0000 7f0676bba640 1 -- 172.21.15.81:0/3246246945 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0670073650 msgr2=0x7f067015b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.664+0000 7f0676bba640 1 --2- 172.21.15.81:0/3246246945 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0670073650 0x7f067015b1e0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f065c009830 tx=0x7f065c02a5c0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.664+0000 7f0676bba640 1 -- 172.21.15.81:0/3246246945 shutdown_connections 2024-06-29T14:39:42.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.664+0000 7f0676bba640 1 --2- 172.21.15.81:0/3246246945 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0670073650 0x7f067015b1e0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.664+0000 7f0676bba640 1 -- 172.21.15.81:0/3246246945 >> 172.21.15.81:0/3246246945 conn(0x7f06700d0b70 msgr2=0x7f06700d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:42.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.664+0000 7f0676bba640 1 -- 172.21.15.81:0/3246246945 shutdown_connections 2024-06-29T14:39:42.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.664+0000 7f0676bba640 1 -- 172.21.15.81:0/3246246945 wait complete. 2024-06-29T14:39:42.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.666+0000 7f0676bba640 1 Processor -- start 2024-06-29T14:39:42.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.666+0000 7f0676bba640 1 -- start start 2024-06-29T14:39:42.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f0676bba640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06700b46a0 0x7f0670139c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f0676bba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015b720 0x7f0670158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f0676bba640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f067013f980 0x7f067013e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f0676bba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f067006f850 con 0x7f067015b720 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f0676bba640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f067006f6d0 con 0x7f067013f980 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f0676bba640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f067006f9d0 con 0x7f06700b46a0 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f066ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015b720 0x7f0670158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f066ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015b720 0x7f0670158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54046/0 (socket says 172.21.15.81:54046) 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f066ffff640 1 -- 172.21.15.81:0/3295423915 learned_addr learned my addr 172.21.15.81:0/3295423915 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f0675157640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f067013f980 0x7f067013e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f0674956640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06700b46a0 0x7f0670139c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f066ffff640 1 -- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06700b46a0 msgr2=0x7f0670139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f066ffff640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06700b46a0 0x7f0670139c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f066ffff640 1 -- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f067013f980 msgr2=0x7f067013e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f066ffff640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f067013f980 0x7f067013e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f066ffff640 1 -- 172.21.15.81:0/3295423915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067013a180 con 0x7f067015b720 2024-06-29T14:39:42.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.668+0000 7f066ffff640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015b720 0x7f0670158210 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f065c009950 tx=0x7f065c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:42.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.669+0000 7f066d7fa640 1 -- 172.21.15.81:0/3295423915 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c003a80 con 0x7f067015b720 2024-06-29T14:39:42.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.669+0000 7f066d7fa640 1 -- 172.21.15.81:0/3295423915 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f065c003d10 con 0x7f067015b720 2024-06-29T14:39:42.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.669+0000 7f0676bba640 1 -- 172.21.15.81:0/3295423915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06700b00c0 con 0x7f067015b720 2024-06-29T14:39:42.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.669+0000 7f066d7fa640 1 -- 172.21.15.81:0/3295423915 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c005280 con 0x7f067015b720 2024-06-29T14:39:42.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.669+0000 7f0676bba640 1 -- 172.21.15.81:0/3295423915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f067013f3e0 con 0x7f067015b720 2024-06-29T14:39:42.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.670+0000 7f0676bba640 1 -- 172.21.15.81:0/3295423915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f067015b1e0 con 0x7f067015b720 2024-06-29T14:39:42.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.671+0000 7f066d7fa640 1 -- 172.21.15.81:0/3295423915 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f065c005420 con 0x7f067015b720 2024-06-29T14:39:42.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.671+0000 7f066d7fa640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f0654076540 0x7f0654078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.671+0000 7f0674956640 1 -- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f0654076540 msgr2=0x7f0654078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:42.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.672+0000 7f066d7fa640 1 -- 172.21.15.81:0/3295423915 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7f065c0bbbe0 con 0x7f067015b720 2024-06-29T14:39:42.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.672+0000 7f0674956640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f0654076540 0x7f0654078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:42.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.675+0000 7f066d7fa640 1 -- 172.21.15.81:0/3295423915 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f065c041030 con 0x7f067015b720 2024-06-29T14:39:42.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.860+0000 7f0676bba640 1 -- 172.21.15.81:0/3295423915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0670141e80 con 0x7f067015b720 2024-06-29T14:39:42.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.860+0000 7f066d7fa640 1 -- 172.21.15.81:0/3295423915 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) ==== 93+0+10 (secure 0 0 0) 0x7f065c005710 con 0x7f067015b720 2024-06-29T14:39:42.859 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:39:42.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 -- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f0654076540 msgr2=0x7f0654078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:42.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f0654076540 0x7f0654078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 -- 172.21.15.81:0/3295423915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015b720 msgr2=0x7f0670158210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015b720 0x7f0670158210 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f065c009950 tx=0x7f065c0045e0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 -- 172.21.15.81:0/3295423915 shutdown_connections 2024-06-29T14:39:42.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f0654076540 0x7f0654078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f067013f980 0x7f067013e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015b720 0x7f0670158210 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 --2- 172.21.15.81:0/3295423915 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06700b46a0 0x7f0670139c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 -- 172.21.15.81:0/3295423915 >> 172.21.15.81:0/3295423915 conn(0x7f06700d0b70 msgr2=0x7f06701423a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:42.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 -- 172.21.15.81:0/3295423915 shutdown_connections 2024-06-29T14:39:42.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.863+0000 7f0676bba640 1 -- 172.21.15.81:0/3295423915 wait complete. 2024-06-29T14:39:42.873 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:39:42.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.991+0000 7f7c65cef640 1 -- 172.21.15.81:0/371525931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6015bdf0 msgr2=0x7f7c60165ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.991+0000 7f7c65cef640 1 --2- 172.21.15.81:0/371525931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6015bdf0 0x7f7c60165ab0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f7c58005fd0 tx=0x7f7c5802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.991+0000 7f7c65cef640 1 -- 172.21.15.81:0/371525931 shutdown_connections 2024-06-29T14:39:42.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.991+0000 7f7c65cef640 1 --2- 172.21.15.81:0/371525931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6015bdf0 0x7f7c60165ab0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.991+0000 7f7c65cef640 1 -- 172.21.15.81:0/371525931 >> 172.21.15.81:0/371525931 conn(0x7f7c600b2920 msgr2=0x7f7c600b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:42.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.991+0000 7f7c65cef640 1 -- 172.21.15.81:0/371525931 shutdown_connections 2024-06-29T14:39:42.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.991+0000 7f7c65cef640 1 -- 172.21.15.81:0/371525931 wait complete. 2024-06-29T14:39:42.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.994+0000 7f7c65cef640 1 Processor -- start 2024-06-29T14:39:42.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.995+0000 7f7c65cef640 1 -- start start 2024-06-29T14:39:42.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c65cef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c600cd150 0x7f7c601f4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c65cef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6015bdf0 0x7f7c6013d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c65cef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c601656f0 0x7f7c60155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c65cef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7c60166170 con 0x7f7c6015bdf0 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c65cef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7c60165ff0 con 0x7f7c601656f0 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c65cef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7c601662f0 con 0x7f7c600cd150 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c5effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6015bdf0 0x7f7c6013d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c5effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6015bdf0 0x7f7c6013d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54066/0 (socket says 172.21.15.81:54066) 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c5effd640 1 -- 172.21.15.81:0/89033311 learned_addr learned my addr 172.21.15.81:0/89033311 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c5ffff640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c601656f0 0x7f7c60155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c5f7fe640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c600cd150 0x7f7c601f4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c5effd640 1 -- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c600cd150 msgr2=0x7f7c601f4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c5effd640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c600cd150 0x7f7c601f4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c5effd640 1 -- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c601656f0 msgr2=0x7f7c60155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:42.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c5effd640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c601656f0 0x7f7c60155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:42.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.997+0000 7f7c5effd640 1 -- 172.21.15.81:0/89033311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c60156b10 con 0x7f7c6015bdf0 2024-06-29T14:39:42.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.998+0000 7f7c5effd640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6015bdf0 0x7f7c6013d1b0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800c400 tx=0x7f7c4800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:42.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.998+0000 7f7c57fff640 1 -- 172.21.15.81:0/89033311 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48011790 con 0x7f7c6015bdf0 2024-06-29T14:39:42.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.998+0000 7f7c57fff640 1 -- 172.21.15.81:0/89033311 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c48017070 con 0x7f7c6015bdf0 2024-06-29T14:39:42.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.998+0000 7f7c65cef640 1 -- 172.21.15.81:0/89033311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c6013d6f0 con 0x7f7c6015bdf0 2024-06-29T14:39:42.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.998+0000 7f7c57fff640 1 -- 172.21.15.81:0/89033311 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48011c60 con 0x7f7c6015bdf0 2024-06-29T14:39:42.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.998+0000 7f7c65cef640 1 -- 172.21.15.81:0/89033311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c60164c10 con 0x7f7c6015bdf0 2024-06-29T14:39:42.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:42.999+0000 7f7c65cef640 1 -- 172.21.15.81:0/89033311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c6015b6e0 con 0x7f7c6015bdf0 2024-06-29T14:39:42.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.000+0000 7f7c57fff640 1 -- 172.21.15.81:0/89033311 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7f7c4801c460 con 0x7f7c6015bdf0 2024-06-29T14:39:42.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.000+0000 7f7c57fff640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f7c40076510 0x7f7c40078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:42.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.000+0000 7f7c5f7fe640 1 -- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f7c40076510 msgr2=0x7f7c40078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:42.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.000+0000 7f7c57fff640 1 -- 172.21.15.81:0/89033311 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7f7c48097ad0 con 0x7f7c6015bdf0 2024-06-29T14:39:42.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.000+0000 7f7c5f7fe640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f7c40076510 0x7f7c40078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:43.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.004+0000 7f7c57fff640 1 -- 172.21.15.81:0/89033311 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7c480608c0 con 0x7f7c6015bdf0 2024-06-29T14:39:43.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.193+0000 7f7c65cef640 1 -- 172.21.15.81:0/89033311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7c60156ce0 con 0x7f7c6015bdf0 2024-06-29T14:39:43.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.193+0000 7f7c57fff640 1 -- 172.21.15.81:0/89033311 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) ==== 104+0+11 (secure 0 0 0) 0x7f7c48064bc0 con 0x7f7c6015bdf0 2024-06-29T14:39:43.192 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:39:43.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 -- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f7c40076510 msgr2=0x7f7c40078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f7c40076510 0x7f7c40078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 -- 172.21.15.81:0/89033311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6015bdf0 msgr2=0x7f7c6013d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6015bdf0 0x7f7c6013d1b0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800c400 tx=0x7f7c4800c8c0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 -- 172.21.15.81:0/89033311 shutdown_connections 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7f7c40076510 0x7f7c40078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7c601656f0 0x7f7c60155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6015bdf0 0x7f7c6013d1b0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 --2- 172.21.15.81:0/89033311 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7c600cd150 0x7f7c601f4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 -- 172.21.15.81:0/89033311 >> 172.21.15.81:0/89033311 conn(0x7f7c600b2920 msgr2=0x7f7c6015ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 -- 172.21.15.81:0/89033311 shutdown_connections 2024-06-29T14:39:43.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.196+0000 7f7c65cef640 1 -- 172.21.15.81:0/89033311 wait complete. 2024-06-29T14:39:43.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:39:43.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.326+0000 7fa0e4c17640 1 -- 172.21.15.81:0/2636355771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e015bdf0 msgr2=0x7fa0e006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.326+0000 7fa0e4c17640 1 --2- 172.21.15.81:0/2636355771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e015bdf0 0x7fa0e006ed90 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0072f10 tx=0x7fa0d802aa30 comp rx=0 tx=0).stop 2024-06-29T14:39:43.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.326+0000 7fa0e4c17640 1 -- 172.21.15.81:0/2636355771 shutdown_connections 2024-06-29T14:39:43.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.326+0000 7fa0e4c17640 1 --2- 172.21.15.81:0/2636355771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e015bdf0 0x7fa0e006ed90 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.326+0000 7fa0e4c17640 1 --2- 172.21.15.81:0/2636355771 >> v2:172.21.15.167:3300/0 conn(0x7fa0e00cd150 0x7fa0e0072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.326+0000 7fa0e4c17640 1 -- 172.21.15.81:0/2636355771 >> 172.21.15.81:0/2636355771 conn(0x7fa0e00b2920 msgr2=0x7fa0e00b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.326+0000 7fa0e4c17640 1 -- 172.21.15.81:0/2636355771 shutdown_connections 2024-06-29T14:39:43.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.326+0000 7fa0e4c17640 1 -- 172.21.15.81:0/2636355771 wait complete. 2024-06-29T14:39:43.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.328+0000 7fa0e4c17640 1 Processor -- start 2024-06-29T14:39:43.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.328+0000 7fa0e4c17640 1 -- start start 2024-06-29T14:39:43.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0e4c17640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0e0073480 0x7fa0e014b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0e4c17640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e00cd150 0x7fa0e013f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0e4c17640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e015bdf0 0x7fa0e00d0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0e4c17640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e006f450 con 0x7fa0e00cd150 2024-06-29T14:39:43.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0e4c17640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e006f2d0 con 0x7fa0e015bdf0 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0e4c17640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa0e006f5d0 con 0x7fa0e0073480 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0ddd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e00cd150 0x7fa0e013f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0ddd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e00cd150 0x7fa0e013f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54086/0 (socket says 172.21.15.81:54086) 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0ddd9b640 1 -- 172.21.15.81:0/1026396212 learned_addr learned my addr 172.21.15.81:0/1026396212 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0de59c640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0e0073480 0x7fa0e014b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0ded9d640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e015bdf0 0x7fa0e00d0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0ddd9b640 1 -- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0e0073480 msgr2=0x7fa0e014b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0ddd9b640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0e0073480 0x7fa0e014b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0ddd9b640 1 -- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e015bdf0 msgr2=0x7fa0e00d0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0ddd9b640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e015bdf0 0x7fa0e00d0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0ddd9b640 1 -- 172.21.15.81:0/1026396212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e0156270 con 0x7fa0e00cd150 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.330+0000 7fa0ddd9b640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e00cd150 0x7fa0e013f390 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000c370 tx=0x7fa0d000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:43.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.331+0000 7fa0d6ffd640 1 -- 172.21.15.81:0/1026396212 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0017070 con 0x7fa0e00cd150 2024-06-29T14:39:43.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.331+0000 7fa0d6ffd640 1 -- 172.21.15.81:0/1026396212 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0d00053c0 con 0x7fa0e00cd150 2024-06-29T14:39:43.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.331+0000 7fa0e4c17640 1 -- 172.21.15.81:0/1026396212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e0156550 con 0x7fa0e00cd150 2024-06-29T14:39:43.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.331+0000 7fa0d6ffd640 1 -- 172.21.15.81:0/1026396212 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0003850 con 0x7fa0e00cd150 2024-06-29T14:39:43.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.331+0000 7fa0e4c17640 1 -- 172.21.15.81:0/1026396212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e0156b10 con 0x7fa0e00cd150 2024-06-29T14:39:43.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.332+0000 7fa0e4c17640 1 -- 172.21.15.81:0/1026396212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0e0072f40 con 0x7fa0e00cd150 2024-06-29T14:39:43.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.333+0000 7fa0d6ffd640 1 -- 172.21.15.81:0/1026396212 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 53) ==== 98417+0+0 (secure 0 0 0) 0x7fa0d001a070 con 0x7fa0e00cd150 2024-06-29T14:39:43.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.334+0000 7fa0d6ffd640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fa0c0076510 0x7fa0c0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.334+0000 7fa0de59c640 1 -- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fa0c0076510 msgr2=0x7fa0c0078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/295682379 2024-06-29T14:39:43.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.334+0000 7fa0d6ffd640 1 -- 172.21.15.81:0/1026396212 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6183+0+0 (secure 0 0 0) 0x7fa0d0097070 con 0x7fa0e00cd150 2024-06-29T14:39:43.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.334+0000 7fa0de59c640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fa0c0076510 0x7fa0c0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:39:43.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.338+0000 7fa0d6ffd640 1 -- 172.21.15.81:0/1026396212 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa0e0073890 con 0x7fa0e00cd150 2024-06-29T14:39:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.396+0000 7fa0d6ffd640 1 -- 172.21.15.81:0/1026396212 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 54) ==== 97912+0+0 (secure 0 0 0) 0x7fa0d005c120 con 0x7fa0e00cd150 2024-06-29T14:39:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.396+0000 7fa0d6ffd640 1 -- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fa0c0076510 msgr2=0x7fa0c0078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:39:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.396+0000 7fa0d6ffd640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fa0c0076510 0x7fa0c0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.445 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:43.447+0000 7f86a5a82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:43.445 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:43.447+0000 7f86a5a82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:43.445 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:43.447+0000 7f86a5a82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:43.445 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:43.447+0000 7f86a5a82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:43.445 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:43.447+0000 7f86a5a82640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:43.446 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:43.448+0000 7f86a8a88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:43.446 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:43.448+0000 7f86a8a88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:43.446 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:43.448+0000 7f86a8a88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:43.446 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:43.448+0000 7f86a8a88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:43.446 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:39:43.448+0000 7f86a8a88640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:39:43.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.533+0000 7fa0e4c17640 1 -- 172.21.15.81:0/1026396212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa0e00d1890 con 0x7fa0e00cd150 2024-06-29T14:39:43.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.534+0000 7fa0d6ffd640 1 -- 172.21.15.81:0/1026396212 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) ==== 100+0+11 (secure 0 0 0) 0x7fa0d001b3b0 con 0x7fa0e00cd150 2024-06-29T14:39:43.532 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:39:43.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.536+0000 7fa0e4c17640 1 -- 172.21.15.81:0/1026396212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e00cd150 msgr2=0x7fa0e013f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.536+0000 7fa0e4c17640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e00cd150 0x7fa0e013f390 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000c370 tx=0x7fa0d000c830 comp rx=0 tx=0).stop 2024-06-29T14:39:43.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.536+0000 7fa0e4c17640 1 -- 172.21.15.81:0/1026396212 shutdown_connections 2024-06-29T14:39:43.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.536+0000 7fa0e4c17640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:6832/295682379,v1:172.21.15.167:6833/295682379] conn(0x7fa0c0076510 0x7fa0c0078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.536+0000 7fa0e4c17640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0e015bdf0 0x7fa0e00d0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.536+0000 7fa0e4c17640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0e00cd150 0x7fa0e013f390 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.536+0000 7fa0e4c17640 1 --2- 172.21.15.81:0/1026396212 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0e0073480 0x7fa0e014b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.536+0000 7fa0e4c17640 1 -- 172.21.15.81:0/1026396212 >> 172.21.15.81:0/1026396212 conn(0x7fa0e00b2920 msgr2=0x7fa0e0154900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.536+0000 7fa0e4c17640 1 -- 172.21.15.81:0/1026396212 shutdown_connections 2024-06-29T14:39:43.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.536+0000 7fa0e4c17640 1 -- 172.21.15.81:0/1026396212 wait complete. 2024-06-29T14:39:43.546 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:39:43.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.661+0000 7f3497217640 1 -- 172.21.15.81:0/1286047945 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34900b6a20 msgr2=0x7f34900d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.661+0000 7f3497217640 1 --2- 172.21.15.81:0/1286047945 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34900b6a20 0x7f34900d0410 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f349008c7e0 tx=0x7f348402a680 comp rx=0 tx=0).stop 2024-06-29T14:39:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.661+0000 7f3497217640 1 -- 172.21.15.81:0/1286047945 shutdown_connections 2024-06-29T14:39:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.661+0000 7f3497217640 1 --2- 172.21.15.81:0/1286047945 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34900b6a20 0x7f34900d0410 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.661+0000 7f3497217640 1 --2- 172.21.15.81:0/1286047945 >> v2:172.21.15.167:3300/0 conn(0x7f34900b8ba0 0x7f34900d46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.661+0000 7f3497217640 1 -- 172.21.15.81:0/1286047945 >> 172.21.15.81:0/1286047945 conn(0x7f34900d3bd0 msgr2=0x7f34900d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.661+0000 7f3497217640 1 -- 172.21.15.81:0/1286047945 shutdown_connections 2024-06-29T14:39:43.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.661+0000 7f3497217640 1 -- 172.21.15.81:0/1286047945 wait complete. 2024-06-29T14:39:43.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.663+0000 7f3497217640 1 Processor -- start 2024-06-29T14:39:43.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.663+0000 7f3497217640 1 -- start start 2024-06-29T14:39:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3497217640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34900b6a20 0x7f34901eb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3497217640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f34900b8ba0 0x7f34901ea160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3497217640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f34900d0950 0x7f34901ea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3497217640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f34900b9290 con 0x7f34900b6a20 2024-06-29T14:39:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3497217640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f34900b7040 con 0x7f34900b8ba0 2024-06-29T14:39:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3497217640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f34900b9410 con 0x7f34900d0950 2024-06-29T14:39:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3494fb3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34900b6a20 0x7f34901eb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3494fb3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34900b6a20 0x7f34901eb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54102/0 (socket says 172.21.15.81:54102) 2024-06-29T14:39:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3494fb3640 1 -- 172.21.15.81:0/133055438 learned_addr learned my addr 172.21.15.81:0/133055438 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:43.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f348ffff640 1 --2- 172.21.15.81:0/133055438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f34900b8ba0 0x7f34901ea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f34957b4640 1 --2- 172.21.15.81:0/133055438 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f34900d0950 0x7f34901ea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3494fb3640 1 -- 172.21.15.81:0/133055438 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f34900d0950 msgr2=0x7f34901ea6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3494fb3640 1 --2- 172.21.15.81:0/133055438 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f34900d0950 0x7f34901ea6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3494fb3640 1 -- 172.21.15.81:0/133055438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f34900b8ba0 msgr2=0x7f34901ea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.664+0000 7f3494fb3640 1 --2- 172.21.15.81:0/133055438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f34900b8ba0 0x7f34901ea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.665+0000 7f3494fb3640 1 -- 172.21.15.81:0/133055438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349014a090 con 0x7f34900b6a20 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.665+0000 7f34957b4640 1 --2- 172.21.15.81:0/133055438 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f34900d0950 0x7f34901ea6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.665+0000 7f3494fb3640 1 --2- 172.21.15.81:0/133055438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34900b6a20 0x7f34901eb580 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f348000d6d0 tx=0x7f348000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.665+0000 7f348d7fa640 1 -- 172.21.15.81:0/133055438 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480017070 con 0x7f34900b6a20 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.665+0000 7f348d7fa640 1 -- 172.21.15.81:0/133055438 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3480005330 con 0x7f34900b6a20 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.665+0000 7f3497217640 1 -- 172.21.15.81:0/133055438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f349014a370 con 0x7f34900b6a20 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.665+0000 7f348d7fa640 1 -- 172.21.15.81:0/133055438 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480009820 con 0x7f34900b6a20 2024-06-29T14:39:43.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.665+0000 7f3497217640 1 -- 172.21.15.81:0/133055438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f349014bbf0 con 0x7f34900b6a20 2024-06-29T14:39:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.666+0000 7f3497217640 1 -- 172.21.15.81:0/133055438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3490149e80 con 0x7f34900b6a20 2024-06-29T14:39:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.666+0000 7f348d7fa640 1 -- 172.21.15.81:0/133055438 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 54) ==== 97912+0+0 (secure 0 0 0) 0x7f3480009bd0 con 0x7f34900b6a20 2024-06-29T14:39:43.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.667+0000 7f348d7fa640 1 -- 172.21.15.81:0/133055438 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6613+0+0 (secure 0 0 0) 0x7f3480097870 con 0x7f34900b6a20 2024-06-29T14:39:43.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.672+0000 7f348d7fa640 1 -- 172.21.15.81:0/133055438 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3480060bb0 con 0x7f34900b6a20 2024-06-29T14:39:43.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.893+0000 7f3497217640 1 -- 172.21.15.81:0/133055438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34900cff00 con 0x7f34900b6a20 2024-06-29T14:39:43.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.894+0000 7f348d7fa640 1 -- 172.21.15.81:0/133055438 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5231 (secure 0 0 0) 0x7f3480069280 con 0x7f34900b6a20 2024-06-29T14:39:43.892 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:43.893 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":51,"btime":"2024-06-29T14:38:27:883643+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1895620554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1895620554},{"type":"v1","addr":"172.21.15.167:6836","nonce":1895620554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":7485,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3391563230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3391563230},{"type":"v1","addr":"172.21.15.81:6834","nonce":3391563230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":7488,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3932182351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3932182351},{"type":"v1","addr":"172.21.15.81:6835","nonce":3932182351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.216390+0000","modified":"2024-06-29T14:38:27.252097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/3193613343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":3193613343},{"type":"v1","addr":"172.21.15.167:6837","nonce":3193613343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":6}]} 2024-06-29T14:39:43.893 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 51 2024-06-29T14:39:43.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.896+0000 7f3497217640 1 -- 172.21.15.81:0/133055438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34900b6a20 msgr2=0x7f34901eb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:43.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.896+0000 7f3497217640 1 --2- 172.21.15.81:0/133055438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34900b6a20 0x7f34901eb580 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f348000d6d0 tx=0x7f348000db90 comp rx=0 tx=0).stop 2024-06-29T14:39:43.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.896+0000 7f3497217640 1 -- 172.21.15.81:0/133055438 shutdown_connections 2024-06-29T14:39:43.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.896+0000 7f3497217640 1 --2- 172.21.15.81:0/133055438 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f34900d0950 0x7f34901ea6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.896+0000 7f3497217640 1 --2- 172.21.15.81:0/133055438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f34900b8ba0 0x7f34901ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.896+0000 7f3497217640 1 --2- 172.21.15.81:0/133055438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34900b6a20 0x7f34901eb580 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:43.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.896+0000 7f3497217640 1 -- 172.21.15.81:0/133055438 >> 172.21.15.81:0/133055438 conn(0x7f34900d3bd0 msgr2=0x7f349006d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:43.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.896+0000 7f3497217640 1 -- 172.21.15.81:0/133055438 shutdown_connections 2024-06-29T14:39:43.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:43.896+0000 7f3497217640 1 -- 172.21.15.81:0/133055438 wait complete. 2024-06-29T14:39:43.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:44.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.021+0000 7f7274f2a640 1 -- 172.21.15.81:0/1104243391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72700b57b0 msgr2=0x7f727015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.021+0000 7f7274f2a640 1 --2- 172.21.15.81:0/1104243391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72700b57b0 0x7f727015aed0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f72700743b0 tx=0x7f726002aac0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.021+0000 7f7274f2a640 1 -- 172.21.15.81:0/1104243391 shutdown_connections 2024-06-29T14:39:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.021+0000 7f7274f2a640 1 --2- 172.21.15.81:0/1104243391 >> v2:172.21.15.167:3300/0 conn(0x7f72700b3960 0x7f72700b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.021+0000 7f7274f2a640 1 --2- 172.21.15.81:0/1104243391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72700b57b0 0x7f727015aed0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.021+0000 7f7274f2a640 1 -- 172.21.15.81:0/1104243391 >> 172.21.15.81:0/1104243391 conn(0x7f72700d0930 msgr2=0x7f72700d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.021+0000 7f7274f2a640 1 -- 172.21.15.81:0/1104243391 shutdown_connections 2024-06-29T14:39:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.021+0000 7f7274f2a640 1 -- 172.21.15.81:0/1104243391 wait complete. 2024-06-29T14:39:44.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.023+0000 7f7274f2a640 1 Processor -- start 2024-06-29T14:39:44.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.023+0000 7f7274f2a640 1 -- start start 2024-06-29T14:39:44.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f7274f2a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727006ef90 0x7f7270150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f7274f2a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72700b3960 0x7f72700cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f7274f2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72700b57b0 0x7f727006c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f7274f2a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72700b4430 con 0x7f72700b57b0 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f7274f2a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f72700b42b0 con 0x7f727006ef90 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f7274f2a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f72700b45b0 con 0x7f72700b3960 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72700b57b0 0x7f727006c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72700b57b0 0x7f727006c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54118/0 (socket says 172.21.15.81:54118) 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726ed9d640 1 -- 172.21.15.81:0/2995684890 learned_addr learned my addr 172.21.15.81:0/2995684890 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726e59c640 1 --2- 172.21.15.81:0/2995684890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727006ef90 0x7f7270150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726dd9b640 1 --2- 172.21.15.81:0/2995684890 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72700b3960 0x7f72700cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726ed9d640 1 -- 172.21.15.81:0/2995684890 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72700b3960 msgr2=0x7f72700cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726ed9d640 1 --2- 172.21.15.81:0/2995684890 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72700b3960 0x7f72700cb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726ed9d640 1 -- 172.21.15.81:0/2995684890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727006ef90 msgr2=0x7f7270150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726ed9d640 1 --2- 172.21.15.81:0/2995684890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727006ef90 0x7f7270150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726ed9d640 1 -- 172.21.15.81:0/2995684890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72700cc0c0 con 0x7f72700b57b0 2024-06-29T14:39:44.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.024+0000 7f726e59c640 1 --2- 172.21.15.81:0/2995684890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727006ef90 0x7f7270150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:44.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.025+0000 7f726ed9d640 1 --2- 172.21.15.81:0/2995684890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72700b57b0 0x7f727006c2f0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f726800c970 tx=0x7f726800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:44.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.025+0000 7f725effd640 1 -- 172.21.15.81:0/2995684890 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268004460 con 0x7f72700b57b0 2024-06-29T14:39:44.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.025+0000 7f725effd640 1 -- 172.21.15.81:0/2995684890 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72680077a0 con 0x7f72700b57b0 2024-06-29T14:39:44.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.025+0000 7f7274f2a640 1 -- 172.21.15.81:0/2995684890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72700cc3a0 con 0x7f72700b57b0 2024-06-29T14:39:44.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.025+0000 7f725effd640 1 -- 172.21.15.81:0/2995684890 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268007bd0 con 0x7f72700b57b0 2024-06-29T14:39:44.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.025+0000 7f7274f2a640 1 -- 172.21.15.81:0/2995684890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72701549d0 con 0x7f72700b57b0 2024-06-29T14:39:44.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.026+0000 7f7274f2a640 1 -- 172.21.15.81:0/2995684890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f727006e2a0 con 0x7f72700b57b0 2024-06-29T14:39:44.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.026+0000 7f725effd640 1 -- 172.21.15.81:0/2995684890 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 54) ==== 97912+0+0 (secure 0 0 0) 0x7f72680035f0 con 0x7f72700b57b0 2024-06-29T14:39:44.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.027+0000 7f725effd640 1 -- 172.21.15.81:0/2995684890 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6613+0+0 (secure 0 0 0) 0x7f7268097bf0 con 0x7f72700b57b0 2024-06-29T14:39:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.032+0000 7f725effd640 1 -- 172.21.15.81:0/2995684890 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f727006e2a0 con 0x7f72700b57b0 2024-06-29T14:39:44.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.214+0000 7f7274f2a640 1 -- 172.21.15.81:0/2995684890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72700b49d0 con 0x7f72700b57b0 2024-06-29T14:39:44.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.216+0000 7f725effd640 1 -- 172.21.15.81:0/2995684890 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+18009 (secure 0 0 0) 0x7f7268013030 con 0x7f72700b57b0 2024-06-29T14:39:44.214 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:44.215 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":108,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:39:43.383226+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.514776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.175620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.7","mappings":[{"from":6,"to":3}]},{"pgid":"13.20","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/51861160":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/3916841092":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/4191187352":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/3979799678":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/2383909496":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/2133120885":"2024-06-30T14:39:43.383162+0000","172.21.15.167:6833/295682379":"2024-06-30T14:39:43.383162+0000","172.21.15.167:6832/295682379":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/3578595575":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2870419655":"2024-06-30T14:38:36.587419+0000","172.21.15.167:6833/2583731609":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/3737328049":"2024-06-30T14:39:43.383162+0000","172.21.15.167:6832/2583731609":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/1601099482":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2667854206":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/32033638":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2697428082":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/4125253667":"2024-06-30T14:38:36.587419+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:44.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.218+0000 7f7274f2a640 1 -- 172.21.15.81:0/2995684890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72700b57b0 msgr2=0x7f727006c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.218+0000 7f7274f2a640 1 --2- 172.21.15.81:0/2995684890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72700b57b0 0x7f727006c2f0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f726800c970 tx=0x7f726800ce30 comp rx=0 tx=0).stop 2024-06-29T14:39:44.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.218+0000 7f7274f2a640 1 -- 172.21.15.81:0/2995684890 shutdown_connections 2024-06-29T14:39:44.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.218+0000 7f7274f2a640 1 --2- 172.21.15.81:0/2995684890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72700b57b0 0x7f727006c2f0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.218+0000 7f7274f2a640 1 --2- 172.21.15.81:0/2995684890 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72700b3960 0x7f72700cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.218+0000 7f7274f2a640 1 --2- 172.21.15.81:0/2995684890 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727006ef90 0x7f7270150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.218+0000 7f7274f2a640 1 -- 172.21.15.81:0/2995684890 >> 172.21.15.81:0/2995684890 conn(0x7f72700d0930 msgr2=0x7f727015a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.218+0000 7f7274f2a640 1 -- 172.21.15.81:0/2995684890 shutdown_connections 2024-06-29T14:39:44.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.218+0000 7f7274f2a640 1 -- 172.21.15.81:0/2995684890 wait complete. 2024-06-29T14:39:44.228 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:39:44.228 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:39:44.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.342+0000 7f4616612640 1 -- 172.21.15.81:0/3915783765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100cf4d0 msgr2=0x7f4610073030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.342+0000 7f4616612640 1 --2- 172.21.15.81:0/3915783765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100cf4d0 0x7f4610073030 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f4610070e50 tx=0x7f45f802a5f0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.342+0000 7f4616612640 1 -- 172.21.15.81:0/3915783765 shutdown_connections 2024-06-29T14:39:44.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.342+0000 7f4616612640 1 --2- 172.21.15.81:0/3915783765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100cf4d0 0x7f4610073030 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.342+0000 7f4616612640 1 --2- 172.21.15.81:0/3915783765 >> v2:172.21.15.167:3300/0 conn(0x7f46101412f0 0x7f46100cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.342+0000 7f4616612640 1 -- 172.21.15.81:0/3915783765 >> 172.21.15.81:0/3915783765 conn(0x7f461006ccc0 msgr2=0x7f46100b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.342+0000 7f4616612640 1 -- 172.21.15.81:0/3915783765 shutdown_connections 2024-06-29T14:39:44.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.342+0000 7f4616612640 1 -- 172.21.15.81:0/3915783765 wait complete. 2024-06-29T14:39:44.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.344+0000 7f4616612640 1 Processor -- start 2024-06-29T14:39:44.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.344+0000 7f4616612640 1 -- start start 2024-06-29T14:39:44.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4616612640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4610073570 0x7f46101404c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4616612640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100cf4d0 0x7f46100adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4616612640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46101412f0 0x7f4610071f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4616612640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46100b3210 con 0x7f46100cf4d0 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4616612640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f461015f280 con 0x7f4610073570 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4616612640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f46100b3390 con 0x7f46101412f0 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4607fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100cf4d0 0x7f46100adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4607fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100cf4d0 0x7f46100adbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54130/0 (socket says 172.21.15.81:54130) 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4607fff640 1 -- 172.21.15.81:0/2907827506 learned_addr learned my addr 172.21.15.81:0/2907827506 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f460ffff640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4610073570 0x7f46101404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4614baf640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46101412f0 0x7f4610071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4607fff640 1 -- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46101412f0 msgr2=0x7f4610071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4607fff640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46101412f0 0x7f4610071f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4607fff640 1 -- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4610073570 msgr2=0x7f46101404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4607fff640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4610073570 0x7f46101404c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4607fff640 1 -- 172.21.15.81:0/2907827506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4610155a20 con 0x7f46100cf4d0 2024-06-29T14:39:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.345+0000 7f4614baf640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46101412f0 0x7f4610071f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.346+0000 7f4607fff640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100cf4d0 0x7f46100adbe0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f45f80099c0 tx=0x7f45f80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.346+0000 7f460d7fa640 1 -- 172.21.15.81:0/2907827506 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f803e030 con 0x7f46100cf4d0 2024-06-29T14:39:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.346+0000 7f460d7fa640 1 -- 172.21.15.81:0/2907827506 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45f8003db0 con 0x7f46100cf4d0 2024-06-29T14:39:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.346+0000 7f4616612640 1 -- 172.21.15.81:0/2907827506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4610157da0 con 0x7f46100cf4d0 2024-06-29T14:39:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.346+0000 7f460d7fa640 1 -- 172.21.15.81:0/2907827506 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f8035040 con 0x7f46100cf4d0 2024-06-29T14:39:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.346+0000 7f4616612640 1 -- 172.21.15.81:0/2907827506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4610154c60 con 0x7f46100cf4d0 2024-06-29T14:39:44.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.347+0000 7f4616612640 1 -- 172.21.15.81:0/2907827506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4610068bd0 con 0x7f46100cf4d0 2024-06-29T14:39:44.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.347+0000 7f460d7fa640 1 -- 172.21.15.81:0/2907827506 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 54) ==== 97912+0+0 (secure 0 0 0) 0x7f45f8041030 con 0x7f46100cf4d0 2024-06-29T14:39:44.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.348+0000 7f460d7fa640 1 -- 172.21.15.81:0/2907827506 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6613+0+0 (secure 0 0 0) 0x7f45f80b7f00 con 0x7f46100cf4d0 2024-06-29T14:39:44.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.352+0000 7f460d7fa640 1 -- 172.21.15.81:0/2907827506 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4610068bd0 con 0x7f46100cf4d0 2024-06-29T14:39:44.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.395+0000 7f460d7fa640 1 -- 172.21.15.81:0/2907827506 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 55) ==== 98039+0+0 (secure 0 0 0) 0x7f45f807cc40 con 0x7f46100cf4d0 2024-06-29T14:39:44.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.396+0000 7f460d7fa640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f45f007bc40 0x7f45f007e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.397+0000 7f460ffff640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f45f007bc40 0x7f45f007e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.398+0000 7f460ffff640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f45f007bc40 0x7f45f007e060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4600009b30 tx=0x7f4600005f90 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:44.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.575+0000 7f4616612640 1 -- 172.21.15.81:0/2907827506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46100b38b0 con 0x7f46100cf4d0 2024-06-29T14:39:44.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.575+0000 7f460d7fa640 1 -- 172.21.15.81:0/2907827506 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5231 (secure 0 0 0) 0x7f45f80af690 con 0x7f46100cf4d0 2024-06-29T14:39:44.574 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:44.574 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":51,"btime":"2024-06-29T14:38:27:883643+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1895620554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1895620554},{"type":"v1","addr":"172.21.15.167:6836","nonce":1895620554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":7485,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3391563230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3391563230},{"type":"v1","addr":"172.21.15.81:6834","nonce":3391563230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":7488,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3932182351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3932182351},{"type":"v1","addr":"172.21.15.81:6835","nonce":3932182351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.216390+0000","modified":"2024-06-29T14:38:27.252097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/3193613343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":3193613343},{"type":"v1","addr":"172.21.15.167:6837","nonce":3193613343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":6}]} 2024-06-29T14:39:44.574 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 51 2024-06-29T14:39:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 -- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f45f007bc40 msgr2=0x7f45f007e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f45f007bc40 0x7f45f007e060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4600009b30 tx=0x7f4600005f90 comp rx=0 tx=0).stop 2024-06-29T14:39:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 -- 172.21.15.81:0/2907827506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100cf4d0 msgr2=0x7f46100adbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100cf4d0 0x7f46100adbe0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f45f80099c0 tx=0x7f45f80038a0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 -- 172.21.15.81:0/2907827506 shutdown_connections 2024-06-29T14:39:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f45f007bc40 0x7f45f007e060 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46101412f0 0x7f4610071f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100cf4d0 0x7f46100adbe0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 --2- 172.21.15.81:0/2907827506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4610073570 0x7f46101404c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 -- 172.21.15.81:0/2907827506 >> 172.21.15.81:0/2907827506 conn(0x7f461006ccc0 msgr2=0x7f461015ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 -- 172.21.15.81:0/2907827506 shutdown_connections 2024-06-29T14:39:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.577+0000 7f4616612640 1 -- 172.21.15.81:0/2907827506 wait complete. 2024-06-29T14:39:44.587 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:39:44.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.702+0000 7fa40acba640 1 -- 172.21.15.81:0/4187521089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa40415b5b0 msgr2=0x7fa4040b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.702+0000 7fa40acba640 1 --2- 172.21.15.81:0/4187521089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa40415b5b0 0x7fa4040b4430 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009830 tx=0x7fa3f402a480 comp rx=0 tx=0).stop 2024-06-29T14:39:44.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.702+0000 7fa40acba640 1 -- 172.21.15.81:0/4187521089 shutdown_connections 2024-06-29T14:39:44.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.702+0000 7fa40acba640 1 --2- 172.21.15.81:0/4187521089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa40406f660 0x7fa40406fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.702+0000 7fa40acba640 1 --2- 172.21.15.81:0/4187521089 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa40415b5b0 0x7fa4040b4430 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.702+0000 7fa40acba640 1 -- 172.21.15.81:0/4187521089 >> 172.21.15.81:0/4187521089 conn(0x7fa40406ccc0 msgr2=0x7fa4040b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.702+0000 7fa40acba640 1 -- 172.21.15.81:0/4187521089 shutdown_connections 2024-06-29T14:39:44.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.702+0000 7fa40acba640 1 -- 172.21.15.81:0/4187521089 wait complete. 2024-06-29T14:39:44.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.704+0000 7fa40acba640 1 Processor -- start 2024-06-29T14:39:44.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.704+0000 7fa40acba640 1 -- start start 2024-06-29T14:39:44.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa40acba640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa40406f660 0x7fa404145aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa40acba640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4040b35b0 0x7fa40414ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa40acba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa40415b5b0 0x7fa4041f50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa40acba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa404072360 con 0x7fa40415b5b0 2024-06-29T14:39:44.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa40acba640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa4040721e0 con 0x7fa4040b35b0 2024-06-29T14:39:44.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa40acba640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa4040724e0 con 0x7fa40406f660 2024-06-29T14:39:44.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa409257640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa40415b5b0 0x7fa4041f50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa409257640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa40415b5b0 0x7fa4041f50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54140/0 (socket says 172.21.15.81:54140) 2024-06-29T14:39:44.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa409257640 1 -- 172.21.15.81:0/1274374105 learned_addr learned my addr 172.21.15.81:0/1274374105 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:44.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa3fbfff640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4040b35b0 0x7fa40414ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa408a56640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa40406f660 0x7fa404145aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa409257640 1 -- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa40406f660 msgr2=0x7fa404145aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa409257640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa40406f660 0x7fa404145aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa409257640 1 -- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4040b35b0 msgr2=0x7fa40414ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa409257640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4040b35b0 0x7fa40414ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa409257640 1 -- 172.21.15.81:0/1274374105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4040b0b90 con 0x7fa40415b5b0 2024-06-29T14:39:44.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa409257640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa40415b5b0 0x7fa4041f50e0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc00ec50 tx=0x7fa3fc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:44.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa3f97fa640 1 -- 172.21.15.81:0/1274374105 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc003b70 con 0x7fa40415b5b0 2024-06-29T14:39:44.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa3f97fa640 1 -- 172.21.15.81:0/1274374105 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3fc003d10 con 0x7fa40415b5b0 2024-06-29T14:39:44.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa3f97fa640 1 -- 172.21.15.81:0/1274374105 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc004280 con 0x7fa40415b5b0 2024-06-29T14:39:44.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.706+0000 7fa40acba640 1 -- 172.21.15.81:0/1274374105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa40414ac40 con 0x7fa40415b5b0 2024-06-29T14:39:44.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.707+0000 7fa40acba640 1 -- 172.21.15.81:0/1274374105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa40414b200 con 0x7fa40415b5b0 2024-06-29T14:39:44.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.707+0000 7fa40acba640 1 -- 172.21.15.81:0/1274374105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa404072a00 con 0x7fa40415b5b0 2024-06-29T14:39:44.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.708+0000 7fa3f97fa640 1 -- 172.21.15.81:0/1274374105 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 55) ==== 98039+0+0 (secure 0 0 0) 0x7fa3fc004420 con 0x7fa40415b5b0 2024-06-29T14:39:44.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.709+0000 7fa3f97fa640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fa3f0076120 0x7fa3f0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:44.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.709+0000 7fa3f97fa640 1 -- 172.21.15.81:0/1274374105 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6613+0+0 (secure 0 0 0) 0x7fa3fc098420 con 0x7fa40415b5b0 2024-06-29T14:39:44.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.709+0000 7fa408a56640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fa3f0076120 0x7fa3f0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:44.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.709+0000 7fa408a56640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fa3f0076120 0x7fa3f0078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa400001720 tx=0x7fa400007e90 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:44.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.713+0000 7fa3f97fa640 1 -- 172.21.15.81:0/1274374105 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa3fc0616e0 con 0x7fa40415b5b0 2024-06-29T14:39:44.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.934+0000 7fa40acba640 1 -- 172.21.15.81:0/1274374105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa40406f150 con 0x7fa40415b5b0 2024-06-29T14:39:44.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.934+0000 7fa3f97fa640 1 -- 172.21.15.81:0/1274374105 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5231 (secure 0 0 0) 0x7fa3fc0659e0 con 0x7fa40415b5b0 2024-06-29T14:39:44.933 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:44.934 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":51,"btime":"2024-06-29T14:38:27:883643+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1895620554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1895620554},{"type":"v1","addr":"172.21.15.167:6836","nonce":1895620554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":7485,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3391563230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3391563230},{"type":"v1","addr":"172.21.15.81:6834","nonce":3391563230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":7488,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3932182351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3932182351},{"type":"v1","addr":"172.21.15.81:6835","nonce":3932182351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:38:24.216390+0000","modified":"2024-06-29T14:38:27.252097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7503":{"gid":7503,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/3193613343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":3193613343},{"type":"v1","addr":"172.21.15.167:6837","nonce":3193613343}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7503,"qdb_cluster":[7503]},"id":6}]} 2024-06-29T14:39:44.934 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 51 2024-06-29T14:39:44.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.936+0000 7fa40acba640 1 -- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fa3f0076120 msgr2=0x7fa3f0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.936+0000 7fa40acba640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fa3f0076120 0x7fa3f0078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa400001720 tx=0x7fa400007e90 comp rx=0 tx=0).stop 2024-06-29T14:39:44.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.936+0000 7fa40acba640 1 -- 172.21.15.81:0/1274374105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa40415b5b0 msgr2=0x7fa4041f50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:44.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.936+0000 7fa40acba640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa40415b5b0 0x7fa4041f50e0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc00ec50 tx=0x7fa3fc00c6a0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.936+0000 7fa40acba640 1 -- 172.21.15.81:0/1274374105 shutdown_connections 2024-06-29T14:39:44.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.936+0000 7fa40acba640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fa3f0076120 0x7fa3f0078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.937+0000 7fa40acba640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa40415b5b0 0x7fa4041f50e0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.937+0000 7fa40acba640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4040b35b0 0x7fa40414ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.937+0000 7fa40acba640 1 --2- 172.21.15.81:0/1274374105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa40406f660 0x7fa404145aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:44.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.937+0000 7fa40acba640 1 -- 172.21.15.81:0/1274374105 >> 172.21.15.81:0/1274374105 conn(0x7fa40406ccc0 msgr2=0x7fa4041511e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:44.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.937+0000 7fa40acba640 1 -- 172.21.15.81:0/1274374105 shutdown_connections 2024-06-29T14:39:44.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:44.937+0000 7fa40acba640 1 -- 172.21.15.81:0/1274374105 wait complete. 2024-06-29T14:39:44.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:45.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.061+0000 7fd9c9fee640 1 -- 172.21.15.81:0/1191058811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac004800 msgr2=0x7fd9ac004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.061+0000 7fd9c9fee640 1 --2- 172.21.15.81:0/1191058811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac004800 0x7fd9ac004c10 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800c260 tx=0x7fd9ac005830 comp rx=0 tx=0).stop 2024-06-29T14:39:45.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.061+0000 7fd9c9fee640 1 -- 172.21.15.81:0/1191058811 shutdown_connections 2024-06-29T14:39:45.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.061+0000 7fd9c9fee640 1 --2- 172.21.15.81:0/1191058811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac004800 0x7fd9ac004c10 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.061+0000 7fd9c9fee640 1 -- 172.21.15.81:0/1191058811 >> 172.21.15.81:0/1191058811 conn(0x7fd9c40d0930 msgr2=0x7fd9c40d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:45.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.061+0000 7fd9c9fee640 1 -- 172.21.15.81:0/1191058811 shutdown_connections 2024-06-29T14:39:45.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.061+0000 7fd9c9fee640 1 -- 172.21.15.81:0/1191058811 wait complete. 2024-06-29T14:39:45.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.063+0000 7fd9c9fee640 1 Processor -- start 2024-06-29T14:39:45.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.063+0000 7fd9c9fee640 1 -- start start 2024-06-29T14:39:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.064+0000 7fd9c9fee640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9ac004800 0x7fd9c415d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.064+0000 7fd9c9fee640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9c415da70 0x7fd9c41e4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.064+0000 7fd9c9fee640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9c41e4670 0x7fd9c4147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c9fee640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c406f3a0 con 0x7fd9c415da70 2024-06-29T14:39:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c9fee640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c415b2f0 con 0x7fd9ac004800 2024-06-29T14:39:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c9fee640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c406f520 con 0x7fd9c41e4670 2024-06-29T14:39:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9c415da70 0x7fd9c41e4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9c415da70 0x7fd9c41e4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54168/0 (socket says 172.21.15.81:54168) 2024-06-29T14:39:45.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c2ffd640 1 -- 172.21.15.81:0/2421669081 learned_addr learned my addr 172.21.15.81:0/2421669081 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c37fe640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9ac004800 0x7fd9c415d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c3fff640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9c41e4670 0x7fd9c4147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c2ffd640 1 -- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9c41e4670 msgr2=0x7fd9c4147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c2ffd640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9c41e4670 0x7fd9c4147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c2ffd640 1 -- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9ac004800 msgr2=0x7fd9c415d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c2ffd640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9ac004800 0x7fd9c415d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c2ffd640 1 -- 172.21.15.81:0/2421669081 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c41e4df0 con 0x7fd9c415da70 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c37fe640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9ac004800 0x7fd9c415d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c3fff640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9c41e4670 0x7fd9c4147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c2ffd640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9c415da70 0x7fd9c41e4130 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800c380 tx=0x7fd9b8004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9abfff640 1 -- 172.21.15.81:0/2421669081 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8031040 con 0x7fd9c415da70 2024-06-29T14:39:45.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9abfff640 1 -- 172.21.15.81:0/2421669081 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9b8006420 con 0x7fd9c415da70 2024-06-29T14:39:45.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9abfff640 1 -- 172.21.15.81:0/2421669081 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8006c50 con 0x7fd9c415da70 2024-06-29T14:39:45.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c9fee640 1 -- 172.21.15.81:0/2421669081 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c41f2110 con 0x7fd9c415da70 2024-06-29T14:39:45.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.065+0000 7fd9c9fee640 1 -- 172.21.15.81:0/2421669081 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c41f3c70 con 0x7fd9c415da70 2024-06-29T14:39:45.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.066+0000 7fd9c9fee640 1 -- 172.21.15.81:0/2421669081 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c4068ed0 con 0x7fd9c415da70 2024-06-29T14:39:45.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.067+0000 7fd9abfff640 1 -- 172.21.15.81:0/2421669081 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 55) ==== 98039+0+0 (secure 0 0 0) 0x7fd9b8034070 con 0x7fd9c415da70 2024-06-29T14:39:45.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.068+0000 7fd9abfff640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd9b0076120 0x7fd9b0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.068+0000 7fd9abfff640 1 -- 172.21.15.81:0/2421669081 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6613+0+0 (secure 0 0 0) 0x7fd9b80b7b20 con 0x7fd9c415da70 2024-06-29T14:39:45.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.068+0000 7fd9c37fe640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd9b0076120 0x7fd9b0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.068+0000 7fd9c37fe640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd9b0076120 0x7fd9b0078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd9c406adf0 tx=0x7fd9b4005ef0 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.072+0000 7fd9abfff640 1 -- 172.21.15.81:0/2421669081 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd9c4068ed0 con 0x7fd9c415da70 2024-06-29T14:39:45.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.254+0000 7fd9c9fee640 1 -- 172.21.15.81:0/2421669081 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9c41e4b80 con 0x7fd9c415da70 2024-06-29T14:39:45.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.255+0000 7fd9abfff640 1 -- 172.21.15.81:0/2421669081 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+18009 (secure 0 0 0) 0x7fd9b8045060 con 0x7fd9c415da70 2024-06-29T14:39:45.253 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:45.254 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":108,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:39:43.383226+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:38:22.514776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-06-29T14:38:23.175620+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.7","mappings":[{"from":6,"to":3}]},{"pgid":"13.20","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/51861160":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/3916841092":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/4191187352":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/3979799678":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/2383909496":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/2133120885":"2024-06-30T14:39:43.383162+0000","172.21.15.167:6833/295682379":"2024-06-30T14:39:43.383162+0000","172.21.15.167:6832/295682379":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/3578595575":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2870419655":"2024-06-30T14:38:36.587419+0000","172.21.15.167:6833/2583731609":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/3737328049":"2024-06-30T14:39:43.383162+0000","172.21.15.167:6832/2583731609":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/1601099482":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2667854206":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/32033638":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2697428082":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/4125253667":"2024-06-30T14:38:36.587419+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:45.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 -- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd9b0076120 msgr2=0x7fd9b0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd9b0076120 0x7fd9b0078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd9c406adf0 tx=0x7fd9b4005ef0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 -- 172.21.15.81:0/2421669081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9c415da70 msgr2=0x7fd9c41e4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9c415da70 0x7fd9c41e4130 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800c380 tx=0x7fd9b8004be0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 -- 172.21.15.81:0/2421669081 shutdown_connections 2024-06-29T14:39:45.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd9b0076120 0x7fd9b0078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9c41e4670 0x7fd9c4147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9c415da70 0x7fd9c41e4130 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 --2- 172.21.15.81:0/2421669081 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9ac004800 0x7fd9c415d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 -- 172.21.15.81:0/2421669081 >> 172.21.15.81:0/2421669081 conn(0x7fd9c40d0930 msgr2=0x7fd9c41555c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:45.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 -- 172.21.15.81:0/2421669081 shutdown_connections 2024-06-29T14:39:45.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.257+0000 7fd9c9fee640 1 -- 172.21.15.81:0/2421669081 wait complete. 2024-06-29T14:39:45.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:39:45.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.383+0000 7fb7d8664640 1 -- 172.21.15.81:0/3881500335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7bc004d90 msgr2=0x7fb7bc005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.383+0000 7fb7d8664640 1 --2- 172.21.15.81:0/3881500335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7bc004d90 0x7fb7bc005220 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009b20 tx=0x7fb7bc005830 comp rx=0 tx=0).stop 2024-06-29T14:39:45.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.383+0000 7fb7d6c01640 1 -- 172.21.15.81:0/3881500335 reap_dead start 2024-06-29T14:39:45.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.383+0000 7fb7d8664640 1 -- 172.21.15.81:0/3881500335 shutdown_connections 2024-06-29T14:39:45.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.383+0000 7fb7d8664640 1 -- 172.21.15.81:0/3881500335 >> 172.21.15.81:0/3881500335 conn(0x7fb7d0072950 msgr2=0x7fb7d00b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:45.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.383+0000 7fb7d8664640 1 -- 172.21.15.81:0/3881500335 shutdown_connections 2024-06-29T14:39:45.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.383+0000 7fb7d8664640 1 -- 172.21.15.81:0/3881500335 wait complete. 2024-06-29T14:39:45.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.385+0000 7fb7d8664640 1 Processor -- start 2024-06-29T14:39:45.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.385+0000 7fb7d8664640 1 -- start start 2024-06-29T14:39:45.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.386+0000 7fb7d8664640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc004d90 0x7fb7d01f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.386+0000 7fb7d8664640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d009d940 0x7fb7d0151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.386+0000 7fb7d8664640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d009dd50 0x7fb7d0152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.386+0000 7fb7d8664640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d015b190 con 0x7fb7d009dd50 2024-06-29T14:39:45.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.386+0000 7fb7d8664640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d015b010 con 0x7fb7d009d940 2024-06-29T14:39:45.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.386+0000 7fb7d8664640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb7d00aee20 con 0x7fb7bc004d90 2024-06-29T14:39:45.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.386+0000 7fb7d6c01640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d009dd50 0x7fb7d0152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.386+0000 7fb7d6c01640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d009dd50 0x7fb7d0152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54192/0 (socket says 172.21.15.81:54192) 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7d6c01640 1 -- 172.21.15.81:0/1967408265 learned_addr learned my addr 172.21.15.81:0/1967408265 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7d6400640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc004d90 0x7fb7d01f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7d6c01640 1 -- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc004d90 msgr2=0x7fb7d01f2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7d6c01640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc004d90 0x7fb7d01f2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7d6c01640 1 -- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d009d940 msgr2=0x7fb7d0151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7d6c01640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d009d940 0x7fb7d0151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7d6c01640 1 -- 172.21.15.81:0/1967408265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d01f32e0 con 0x7fb7d009dd50 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7d6c01640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d009dd50 0x7fb7d0152180 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009b20 tx=0x7fb7c8033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7c6ffd640 1 -- 172.21.15.81:0/1967408265 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8038040 con 0x7fb7d009dd50 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7c6ffd640 1 -- 172.21.15.81:0/1967408265 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7c8004020 con 0x7fb7d009dd50 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7d8664640 1 -- 172.21.15.81:0/1967408265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d009aff0 con 0x7fb7d009dd50 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7c6ffd640 1 -- 172.21.15.81:0/1967408265 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8004430 con 0x7fb7d009dd50 2024-06-29T14:39:45.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.387+0000 7fb7d8664640 1 -- 172.21.15.81:0/1967408265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d009b5b0 con 0x7fb7d009dd50 2024-06-29T14:39:45.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.388+0000 7fb7d8664640 1 -- 172.21.15.81:0/1967408265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7d00af380 con 0x7fb7d009dd50 2024-06-29T14:39:45.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.389+0000 7fb7c6ffd640 1 -- 172.21.15.81:0/1967408265 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 55) ==== 98039+0+0 (secure 0 0 0) 0x7fb7c8044030 con 0x7fb7d009dd50 2024-06-29T14:39:45.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.390+0000 7fb7c6ffd640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb7b4076120 0x7fb7b4078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.390+0000 7fb7c6ffd640 1 -- 172.21.15.81:0/1967408265 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6613+0+0 (secure 0 0 0) 0x7fb7c80b78a0 con 0x7fb7d009dd50 2024-06-29T14:39:45.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.390+0000 7fb7d6400640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb7b4076120 0x7fb7b4078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.390+0000 7fb7d6400640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb7b4076120 0x7fb7b4078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc006fd0 tx=0x7fb7cc001e90 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.394+0000 7fb7c6ffd640 1 -- 172.21.15.81:0/1967408265 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb7c804b020 con 0x7fb7d009dd50 2024-06-29T14:39:45.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.616+0000 7fb7d8664640 1 -- 172.21.15.81:0/1967408265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb7d0159500 con 0x7fb7d009dd50 2024-06-29T14:39:45.615 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T14:39:45.617+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:39:45.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.618+0000 7fb7c6ffd640 1 -- 172.21.15.81:0/1967408265 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v52) ==== 161+0+0 (secure 0 0 0) 0x7fb7c80810e0 con 0x7fb7d009dd50 2024-06-29T14:39:45.617 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:39:45.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 -- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb7b4076120 msgr2=0x7fb7b4078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb7b4076120 0x7fb7b4078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc006fd0 tx=0x7fb7cc001e90 comp rx=0 tx=0).stop 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 -- 172.21.15.81:0/1967408265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d009dd50 msgr2=0x7fb7d0152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d009dd50 0x7fb7d0152180 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009b20 tx=0x7fb7c8033670 comp rx=0 tx=0).stop 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 -- 172.21.15.81:0/1967408265 shutdown_connections 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb7b4076120 0x7fb7b4078630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d009dd50 0x7fb7d0152180 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d009d940 0x7fb7d0151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 --2- 172.21.15.81:0/1967408265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7bc004d90 0x7fb7d01f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 -- 172.21.15.81:0/1967408265 >> 172.21.15.81:0/1967408265 conn(0x7fb7d0072950 msgr2=0x7fb7d0068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 -- 172.21.15.81:0/1967408265 shutdown_connections 2024-06-29T14:39:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.621+0000 7fb7d8664640 1 -- 172.21.15.81:0/1967408265 wait complete. 2024-06-29T14:39:45.625 INFO:tasks.ceph.mds.b.smithi167.stderr: -7910> 2024-06-29T14:38:26.839+0000 7fcc620e0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:39:45.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:39:45.684 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:39:45.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.749+0000 7f2ef6505640 1 -- 172.21.15.81:0/1241257267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef0142370 msgr2=0x7f2ef00cf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.749+0000 7f2ef6505640 1 --2- 172.21.15.81:0/1241257267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef0142370 0x7f2ef00cf8e0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f2ef006f8e0 tx=0x7f2ee002a530 comp rx=0 tx=0).stop 2024-06-29T14:39:45.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.749+0000 7f2ef6505640 1 -- 172.21.15.81:0/1241257267 shutdown_connections 2024-06-29T14:39:45.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.749+0000 7f2ef6505640 1 --2- 172.21.15.81:0/1241257267 >> v2:172.21.15.167:3300/0 conn(0x7f2ef015fb10 0x7f2ef015ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.749+0000 7f2ef6505640 1 --2- 172.21.15.81:0/1241257267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef0142370 0x7f2ef00cf8e0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.749+0000 7f2ef6505640 1 -- 172.21.15.81:0/1241257267 >> 172.21.15.81:0/1241257267 conn(0x7f2ef009aa60 msgr2=0x7f2ef009ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:45.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.749+0000 7f2ef6505640 1 -- 172.21.15.81:0/1241257267 shutdown_connections 2024-06-29T14:39:45.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.749+0000 7f2ef6505640 1 -- 172.21.15.81:0/1241257267 wait complete. 2024-06-29T14:39:45.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.751+0000 7f2ef6505640 1 Processor -- start 2024-06-29T14:39:45.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.751+0000 7f2ef6505640 1 -- start start 2024-06-29T14:39:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2ef6505640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef00ce3f0 0x7f2ef0142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2ef6505640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ef0142370 0x7f2ef00b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2ef6505640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ef015fb10 0x7f2ef015a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2ef6505640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef00d2140 con 0x7f2ef00ce3f0 2024-06-29T14:39:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2ef6505640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef00b2e30 con 0x7f2ef0142370 2024-06-29T14:39:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2ef6505640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2ef00b2fb0 con 0x7f2ef015fb10 2024-06-29T14:39:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2eeffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef00ce3f0 0x7f2ef0142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2eeffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef00ce3f0 0x7f2ef0142b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54210/0 (socket says 172.21.15.81:54210) 2024-06-29T14:39:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2eeffff640 1 -- 172.21.15.81:0/1293661689 learned_addr learned my addr 172.21.15.81:0/1293661689 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2ef4aa2640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ef015fb10 0x7f2ef015a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.753+0000 7f2eef7fe640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ef0142370 0x7f2ef00b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.754+0000 7f2ef4aa2640 1 -- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ef0142370 msgr2=0x7f2ef00b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.754+0000 7f2ef4aa2640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ef0142370 0x7f2ef00b5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.754+0000 7f2ef4aa2640 1 -- 172.21.15.81:0/1293661689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef00ce3f0 msgr2=0x7f2ef0142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.754+0000 7f2ef4aa2640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef00ce3f0 0x7f2ef0142b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.754+0000 7f2ef4aa2640 1 -- 172.21.15.81:0/1293661689 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef01597a0 con 0x7f2ef015fb10 2024-06-29T14:39:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.754+0000 7f2ef4aa2640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ef015fb10 0x7f2ef015a840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00d0d60 tx=0x7f2ee800cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.760+0000 7f2eecff9640 1 -- 172.21.15.81:0/1293661689 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8003b70 con 0x7f2ef015fb10 2024-06-29T14:39:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.760+0000 7f2eecff9640 1 -- 172.21.15.81:0/1293661689 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ee8003d30 con 0x7f2ef015fb10 2024-06-29T14:39:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.760+0000 7f2ef6505640 1 -- 172.21.15.81:0/1293661689 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef0159a80 con 0x7f2ef015fb10 2024-06-29T14:39:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.760+0000 7f2eecff9640 1 -- 172.21.15.81:0/1293661689 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee80052d0 con 0x7f2ef015fb10 2024-06-29T14:39:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.760+0000 7f2ef6505640 1 -- 172.21.15.81:0/1293661689 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef015a040 con 0x7f2ef015fb10 2024-06-29T14:39:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.761+0000 7f2ef6505640 1 -- 172.21.15.81:0/1293661689 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ef00cf120 con 0x7f2ef015fb10 2024-06-29T14:39:45.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.762+0000 7f2eecff9640 1 -- 172.21.15.81:0/1293661689 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 56) ==== 98247+0+0 (secure 0 0 0) 0x7f2ee8005470 con 0x7f2ef015fb10 2024-06-29T14:39:45.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.762+0000 7f2eecff9640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f2edc076360 0x7f2edc078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:45.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.763+0000 7f2eeffff640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f2edc076360 0x7f2edc078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:45.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.763+0000 7f2eeffff640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f2edc076360 0x7f2edc078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0005eb0 tx=0x7f2ee0004730 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:45.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.766+0000 7f2eecff9640 1 -- 172.21.15.81:0/1293661689 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(109..109 src has 1..109) ==== 6699+0+0 (secure 0 0 0) 0x7f2ee8005760 con 0x7f2ef015fb10 2024-06-29T14:39:45.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.771+0000 7f2eecff9640 1 -- 172.21.15.81:0/1293661689 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2ef0160460 con 0x7f2ef015fb10 2024-06-29T14:39:45.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:45.990+0000 7f2ef6505640 1 -- 172.21.15.81:0/1293661689 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2ef0159530 con 0x7f2ef015fb10 2024-06-29T14:39:46.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.626+0000 7f2eecff9640 1 -- 172.21.15.81:0/1293661689 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v53) ==== 104+0+0 (secure 0 0 0) 0x7f2ee8060660 con 0x7f2ef015fb10 2024-06-29T14:39:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 -- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f2edc076360 msgr2=0x7f2edc078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f2edc076360 0x7f2edc078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0005eb0 tx=0x7f2ee0004730 comp rx=0 tx=0).stop 2024-06-29T14:39:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 -- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ef015fb10 msgr2=0x7f2ef015a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ef015fb10 0x7f2ef015a840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00d0d60 tx=0x7f2ee800cb90 comp rx=0 tx=0).stop 2024-06-29T14:39:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 -- 172.21.15.81:0/1293661689 shutdown_connections 2024-06-29T14:39:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f2edc076360 0x7f2edc078870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ef015fb10 0x7f2ef015a840 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ef0142370 0x7f2ef00b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 --2- 172.21.15.81:0/1293661689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef00ce3f0 0x7f2ef0142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 -- 172.21.15.81:0/1293661689 >> 172.21.15.81:0/1293661689 conn(0x7f2ef009aa60 msgr2=0x7f2ef00ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:46.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 -- 172.21.15.81:0/1293661689 shutdown_connections 2024-06-29T14:39:46.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.628+0000 7f2ef6505640 1 -- 172.21.15.81:0/1293661689 wait complete. 2024-06-29T14:39:46.638 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:39:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.759+0000 7fe027c88640 1 -- 172.21.15.81:0/781204539 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0201555e0 msgr2=0x7fe02015bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.759+0000 7fe027c88640 1 --2- 172.21.15.81:0/781204539 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0201555e0 0x7fe02015bea0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fe010005fd0 tx=0x7fe01002a8b0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.759+0000 7fe027c88640 1 -- 172.21.15.81:0/781204539 shutdown_connections 2024-06-29T14:39:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.759+0000 7fe027c88640 1 --2- 172.21.15.81:0/781204539 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0201555e0 0x7fe02015bea0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.759+0000 7fe027c88640 1 -- 172.21.15.81:0/781204539 >> 172.21.15.81:0/781204539 conn(0x7fe0200d08f0 msgr2=0x7fe0200d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.759+0000 7fe027c88640 1 -- 172.21.15.81:0/781204539 shutdown_connections 2024-06-29T14:39:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.759+0000 7fe027c88640 1 -- 172.21.15.81:0/781204539 wait complete. 2024-06-29T14:39:46.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.760+0000 7fe027c88640 1 Processor -- start 2024-06-29T14:39:46.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.760+0000 7fe027c88640 1 -- start start 2024-06-29T14:39:46.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe027c88640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0200708f0 0x7fe020148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe027c88640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0200af730 0x7fe0201504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe027c88640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0201555e0 0x7fe020150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe027c88640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe020070050 con 0x7fe0200708f0 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe027c88640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe02006fed0 con 0x7fe0201555e0 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe027c88640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe0200701d0 con 0x7fe0200af730 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe025a24640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0200708f0 0x7fe020148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe025a24640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0200708f0 0x7fe020148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54230/0 (socket says 172.21.15.81:54230) 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe025a24640 1 -- 172.21.15.81:0/3193408004 learned_addr learned my addr 172.21.15.81:0/3193408004 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe026225640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0201555e0 0x7fe020150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.762+0000 7fe025223640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0200af730 0x7fe0201504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.763+0000 7fe025a24640 1 -- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0200af730 msgr2=0x7fe0201504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.763+0000 7fe025a24640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0200af730 0x7fe0201504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.763+0000 7fe025a24640 1 -- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0201555e0 msgr2=0x7fe020150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.763+0000 7fe025a24640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0201555e0 0x7fe020150a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.763+0000 7fe025a24640 1 -- 172.21.15.81:0/3193408004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe02014dbe0 con 0x7fe0200708f0 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.763+0000 7fe026225640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0201555e0 0x7fe020150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.763+0000 7fe025223640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0200af730 0x7fe0201504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.763+0000 7fe025a24640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0200708f0 0x7fe020148d80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fe010009cd0 tx=0x7fe010003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.765+0000 7fe01e7fc640 1 -- 172.21.15.81:0/3193408004 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe010031da0 con 0x7fe0200708f0 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.765+0000 7fe01e7fc640 1 -- 172.21.15.81:0/3193408004 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe010033070 con 0x7fe0200708f0 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.765+0000 7fe027c88640 1 -- 172.21.15.81:0/3193408004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe02014b210 con 0x7fe0200708f0 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.765+0000 7fe01e7fc640 1 -- 172.21.15.81:0/3193408004 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0100334c0 con 0x7fe0200708f0 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.765+0000 7fe027c88640 1 -- 172.21.15.81:0/3193408004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe02015c5f0 con 0x7fe0200708f0 2024-06-29T14:39:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.766+0000 7fe027c88640 1 -- 172.21.15.81:0/3193408004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe02014ddb0 con 0x7fe0200708f0 2024-06-29T14:39:46.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.766+0000 7fe01e7fc640 1 -- 172.21.15.81:0/3193408004 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 56) ==== 98247+0+0 (secure 0 0 0) 0x7fe010033660 con 0x7fe0200708f0 2024-06-29T14:39:46.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.767+0000 7fe01e7fc640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fe00c076390 0x7fe00c0788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:46.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.767+0000 7fe01e7fc640 1 -- 172.21.15.81:0/3193408004 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 6699+0+0 (secure 0 0 0) 0x7fe0100b9270 con 0x7fe0200708f0 2024-06-29T14:39:46.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.767+0000 7fe025223640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fe00c076390 0x7fe00c0788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:46.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.767+0000 7fe025223640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fe00c076390 0x7fe00c0788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe014006fd0 tx=0x7fe014008040 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:46.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.771+0000 7fe01e7fc640 1 -- 172.21.15.81:0/3193408004 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe01003f030 con 0x7fe0200708f0 2024-06-29T14:39:46.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:46.953+0000 7fe027c88640 1 -- 172.21.15.81:0/3193408004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe020075c90 con 0x7fe0200708f0 2024-06-29T14:39:47.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.771+0000 7fe01e7fc640 1 -- 172.21.15.81:0/3193408004 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v110) ==== 181+0+0 (secure 0 0 0) 0x7fe020075c90 con 0x7fe0200708f0 2024-06-29T14:39:47.770 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:39:47.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.776+0000 7fe027c88640 1 -- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fe00c076390 msgr2=0x7fe00c0788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:47.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.776+0000 7fe027c88640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fe00c076390 0x7fe00c0788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe014006fd0 tx=0x7fe014008040 comp rx=0 tx=0).stop 2024-06-29T14:39:47.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.776+0000 7fe027c88640 1 -- 172.21.15.81:0/3193408004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0200708f0 msgr2=0x7fe020148d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:47.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.776+0000 7fe027c88640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0200708f0 0x7fe020148d80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fe010009cd0 tx=0x7fe010003c20 comp rx=0 tx=0).stop 2024-06-29T14:39:47.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.778+0000 7fe027c88640 1 -- 172.21.15.81:0/3193408004 shutdown_connections 2024-06-29T14:39:47.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.778+0000 7fe027c88640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fe00c076390 0x7fe00c0788a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.778+0000 7fe027c88640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0201555e0 0x7fe020150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.778+0000 7fe027c88640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0200af730 0x7fe0201504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.778+0000 7fe027c88640 1 --2- 172.21.15.81:0/3193408004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0200708f0 0x7fe020148d80 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.778+0000 7fe027c88640 1 -- 172.21.15.81:0/3193408004 >> 172.21.15.81:0/3193408004 conn(0x7fe0200d08f0 msgr2=0x7fe0200b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:47.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.778+0000 7fe027c88640 1 -- 172.21.15.81:0/3193408004 shutdown_connections 2024-06-29T14:39:47.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.779+0000 7fe027c88640 1 -- 172.21.15.81:0/3193408004 wait complete. 2024-06-29T14:39:47.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:39:47.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.915+0000 7fda52711640 1 -- 172.21.15.81:0/297625706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0d1e80 msgr2=0x7fda4c0d2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:47.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.915+0000 7fda52711640 1 --2- 172.21.15.81:0/297625706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0d1e80 0x7fda4c0d2290 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fda4c08cec0 tx=0x7fda3c02a530 comp rx=0 tx=0).stop 2024-06-29T14:39:47.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.915+0000 7fda52711640 1 -- 172.21.15.81:0/297625706 shutdown_connections 2024-06-29T14:39:47.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.915+0000 7fda52711640 1 --2- 172.21.15.81:0/297625706 >> v2:172.21.15.167:3300/0 conn(0x7fda4c0cea50 0x7fda4c0cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.915+0000 7fda52711640 1 --2- 172.21.15.81:0/297625706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0d1e80 0x7fda4c0d2290 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.915+0000 7fda52711640 1 -- 172.21.15.81:0/297625706 >> 172.21.15.81:0/297625706 conn(0x7fda4c0b5700 msgr2=0x7fda4c0b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:47.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.915+0000 7fda52711640 1 -- 172.21.15.81:0/297625706 shutdown_connections 2024-06-29T14:39:47.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.915+0000 7fda52711640 1 -- 172.21.15.81:0/297625706 wait complete. 2024-06-29T14:39:47.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.917+0000 7fda52711640 1 Processor -- start 2024-06-29T14:39:47.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.917+0000 7fda52711640 1 -- start start 2024-06-29T14:39:47.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda52711640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda4c0cea50 0x7fda4c1e1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:47.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda52711640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda4c0cf3a0 0x7fda4c0a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda52711640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0d1e80 0x7fda4c0c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda52711640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c0d17e0 con 0x7fda4c0d1e80 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda52711640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c08d400 con 0x7fda4c0cea50 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda52711640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fda4c0d1960 con 0x7fda4c0cf3a0 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda50cae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0d1e80 0x7fda4c0c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda50cae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0d1e80 0x7fda4c0c9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54260/0 (socket says 172.21.15.81:54260) 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda50cae640 1 -- 172.21.15.81:0/3597478211 learned_addr learned my addr 172.21.15.81:0/3597478211 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda4b7fe640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda4c0cf3a0 0x7fda4c0a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda4bfff640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda4c0cea50 0x7fda4c1e1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda50cae640 1 -- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda4c0cf3a0 msgr2=0x7fda4c0a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda50cae640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda4c0cf3a0 0x7fda4c0a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda50cae640 1 -- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda4c0cea50 msgr2=0x7fda4c1e1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:47.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda50cae640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda4c0cea50 0x7fda4c1e1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:47.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda50cae640 1 -- 172.21.15.81:0/3597478211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c09ff60 con 0x7fda4c0d1e80 2024-06-29T14:39:47.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda4bfff640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda4c0cea50 0x7fda4c1e1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:47.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda50cae640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0d1e80 0x7fda4c0c9e70 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fda4400c730 tx=0x7fda4400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:47.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda48ff9640 1 -- 172.21.15.81:0/3597478211 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44003b70 con 0x7fda4c0d1e80 2024-06-29T14:39:47.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.919+0000 7fda48ff9640 1 -- 172.21.15.81:0/3597478211 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda44003d10 con 0x7fda4c0d1e80 2024-06-29T14:39:47.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.920+0000 7fda52711640 1 -- 172.21.15.81:0/3597478211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda4c0a16c0 con 0x7fda4c0d1e80 2024-06-29T14:39:47.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.920+0000 7fda48ff9640 1 -- 172.21.15.81:0/3597478211 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda440052d0 con 0x7fda4c0d1e80 2024-06-29T14:39:47.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.920+0000 7fda52711640 1 -- 172.21.15.81:0/3597478211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4c09de50 con 0x7fda4c0d1e80 2024-06-29T14:39:47.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.921+0000 7fda52711640 1 -- 172.21.15.81:0/3597478211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda4c0cee60 con 0x7fda4c0d1e80 2024-06-29T14:39:47.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.921+0000 7fda48ff9640 1 -- 172.21.15.81:0/3597478211 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fda44005470 con 0x7fda4c0d1e80 2024-06-29T14:39:47.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.922+0000 7fda48ff9640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fda380763f0 0x7fda38078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:47.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.922+0000 7fda48ff9640 1 -- 172.21.15.81:0/3597478211 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6261+0+0 (secure 0 0 0) 0x7fda440979a0 con 0x7fda4c0d1e80 2024-06-29T14:39:47.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.922+0000 7fda4bfff640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fda380763f0 0x7fda38078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:47.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.923+0000 7fda4bfff640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fda380763f0 0x7fda38078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fda3c034000 tx=0x7fda3c004730 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:47.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:47.926+0000 7fda48ff9640 1 -- 172.21.15.81:0/3597478211 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fda4c09ff60 con 0x7fda4c0d1e80 2024-06-29T14:39:48.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.106+0000 7fda52711640 1 -- 172.21.15.81:0/3597478211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fda4c0b64d0 con 0x7fda4c0d1e80 2024-06-29T14:39:48.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.772+0000 7fda48ff9640 1 -- 172.21.15.81:0/3597478211 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v111) ==== 169+0+0 (secure 0 0 0) 0x7fda44060f30 con 0x7fda4c0d1e80 2024-06-29T14:39:48.771 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T14:39:48.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.775+0000 7fda52711640 1 -- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fda380763f0 msgr2=0x7fda38078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:48.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.775+0000 7fda52711640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fda380763f0 0x7fda38078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fda3c034000 tx=0x7fda3c004730 comp rx=0 tx=0).stop 2024-06-29T14:39:48.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.775+0000 7fda52711640 1 -- 172.21.15.81:0/3597478211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0d1e80 msgr2=0x7fda4c0c9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:48.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.775+0000 7fda52711640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0d1e80 0x7fda4c0c9e70 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fda4400c730 tx=0x7fda4400cb30 comp rx=0 tx=0).stop 2024-06-29T14:39:48.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.777+0000 7fda52711640 1 -- 172.21.15.81:0/3597478211 shutdown_connections 2024-06-29T14:39:48.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.777+0000 7fda52711640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fda380763f0 0x7fda38078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.777+0000 7fda52711640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0d1e80 0x7fda4c0c9e70 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.777+0000 7fda52711640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda4c0cf3a0 0x7fda4c0a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.778+0000 7fda52711640 1 --2- 172.21.15.81:0/3597478211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda4c0cea50 0x7fda4c1e1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.778+0000 7fda52711640 1 -- 172.21.15.81:0/3597478211 >> 172.21.15.81:0/3597478211 conn(0x7fda4c0b5700 msgr2=0x7fda4c0cdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:48.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.780+0000 7fda52711640 1 -- 172.21.15.81:0/3597478211 shutdown_connections 2024-06-29T14:39:48.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.780+0000 7fda52711640 1 -- 172.21.15.81:0/3597478211 wait complete. 2024-06-29T14:39:48.794 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:39:48.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.911+0000 7fd0154b3640 1 -- 172.21.15.81:0/3542753976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0100d2a90 msgr2=0x7fd0100d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:48.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.911+0000 7fd0154b3640 1 --2- 172.21.15.81:0/3542753976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0100d2a90 0x7fd0100d09c0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fd008005fd0 tx=0x7fd00802aa90 comp rx=0 tx=0).stop 2024-06-29T14:39:48.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.911+0000 7fd0154b3640 1 -- 172.21.15.81:0/3542753976 shutdown_connections 2024-06-29T14:39:48.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.911+0000 7fd0154b3640 1 --2- 172.21.15.81:0/3542753976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0100d2a90 0x7fd0100d09c0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.911+0000 7fd0154b3640 1 --2- 172.21.15.81:0/3542753976 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0100b9040 0x7fd0100d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.911+0000 7fd0154b3640 1 -- 172.21.15.81:0/3542753976 >> 172.21.15.81:0/3542753976 conn(0x7fd01009cf30 msgr2=0x7fd0100d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:48.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.911+0000 7fd0154b3640 1 -- 172.21.15.81:0/3542753976 shutdown_connections 2024-06-29T14:39:48.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.911+0000 7fd0154b3640 1 -- 172.21.15.81:0/3542753976 wait complete. 2024-06-29T14:39:48.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.913+0000 7fd0154b3640 1 Processor -- start 2024-06-29T14:39:48.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.913+0000 7fd0154b3640 1 -- start start 2024-06-29T14:39:48.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.914+0000 7fd0154b3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0100b5250 0x7fd0100ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:48.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.914+0000 7fd0154b3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0100b9040 0x7fd01015cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:48.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.914+0000 7fd0154b3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0100d2a90 0x7fd01015d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:48.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.914+0000 7fd0154b3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0100b7d30 con 0x7fd0100b5250 2024-06-29T14:39:48.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.914+0000 7fd0154b3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd0100b7bb0 con 0x7fd0100d2a90 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.914+0000 7fd0154b3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd0100b7eb0 con 0x7fd0100b9040 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd00effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0100b5250 0x7fd0100ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd00effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0100b5250 0x7fd0100ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54268/0 (socket says 172.21.15.81:54268) 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd00effd640 1 -- 172.21.15.81:0/2925647023 learned_addr learned my addr 172.21.15.81:0/2925647023 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd0067fc640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0100b9040 0x7fd01015cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd00f7fe640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0100d2a90 0x7fd01015d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd00effd640 1 -- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0100b9040 msgr2=0x7fd01015cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd00effd640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0100b9040 0x7fd01015cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd00effd640 1 -- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0100d2a90 msgr2=0x7fd01015d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd00effd640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0100d2a90 0x7fd01015d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd00effd640 1 -- 172.21.15.81:0/2925647023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0100c8380 con 0x7fd0100b5250 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd0067fc640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0100b9040 0x7fd01015cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd00effd640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0100b5250 0x7fd0100ca690 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd00000b410 tx=0x7fd00000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd007fff640 1 -- 172.21.15.81:0/2925647023 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000015020 con 0x7fd0100b5250 2024-06-29T14:39:48.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd007fff640 1 -- 172.21.15.81:0/2925647023 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd000011940 con 0x7fd0100b5250 2024-06-29T14:39:48.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd0154b3640 1 -- 172.21.15.81:0/2925647023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0100c8620 con 0x7fd0100b5250 2024-06-29T14:39:48.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd007fff640 1 -- 172.21.15.81:0/2925647023 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000011d90 con 0x7fd0100b5250 2024-06-29T14:39:48.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.915+0000 7fd0154b3640 1 -- 172.21.15.81:0/2925647023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0100c8be0 con 0x7fd0100b5250 2024-06-29T14:39:48.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.916+0000 7fd0154b3640 1 -- 172.21.15.81:0/2925647023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0100b8440 con 0x7fd0100b5250 2024-06-29T14:39:48.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.917+0000 7fd007fff640 1 -- 172.21.15.81:0/2925647023 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fd000018020 con 0x7fd0100b5250 2024-06-29T14:39:48.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.918+0000 7fd007fff640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcff00763f0 0x7fcff0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:48.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.918+0000 7fd007fff640 1 -- 172.21.15.81:0/2925647023 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5789+0+0 (secure 0 0 0) 0x7fd000097940 con 0x7fd0100b5250 2024-06-29T14:39:48.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.918+0000 7fd0067fc640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcff00763f0 0x7fcff0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:48.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.919+0000 7fd0067fc640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcff00763f0 0x7fcff0078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd010068380 tx=0x7fcff8008040 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:48.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:48.922+0000 7fd007fff640 1 -- 172.21.15.81:0/2925647023 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd00001f020 con 0x7fd0100b5250 2024-06-29T14:39:49.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.110+0000 7fd0154b3640 1 -- 172.21.15.81:0/2925647023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fd01015f860 con 0x7fd0100b5250 2024-06-29T14:39:49.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.114+0000 7fd007fff640 1 -- 172.21.15.81:0/2925647023 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v47) ==== 71+0+0 (secure 0 0 0) 0x7fd0100c8380 con 0x7fd0100b5250 2024-06-29T14:39:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 -- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcff00763f0 msgr2=0x7fcff0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcff00763f0 0x7fcff0078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd010068380 tx=0x7fcff8008040 comp rx=0 tx=0).stop 2024-06-29T14:39:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 -- 172.21.15.81:0/2925647023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0100b5250 msgr2=0x7fd0100ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0100b5250 0x7fd0100ca690 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fd00000b410 tx=0x7fd00000b8d0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 -- 172.21.15.81:0/2925647023 shutdown_connections 2024-06-29T14:39:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcff00763f0 0x7fcff0078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0100d2a90 0x7fd01015d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0100b9040 0x7fd01015cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 --2- 172.21.15.81:0/2925647023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0100b5250 0x7fd0100ca690 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 -- 172.21.15.81:0/2925647023 >> 172.21.15.81:0/2925647023 conn(0x7fd01009cf30 msgr2=0x7fd0100d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:49.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 -- 172.21.15.81:0/2925647023 shutdown_connections 2024-06-29T14:39:49.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.116+0000 7fd0154b3640 1 -- 172.21.15.81:0/2925647023 wait complete. 2024-06-29T14:39:49.126 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2024-06-29T14:39:49.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.242+0000 7f5fc24ce640 1 -- 172.21.15.81:0/2637105722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fa4004800 msgr2=0x7f5fa4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.242+0000 7f5fc24ce640 1 --2- 172.21.15.81:0/2637105722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fa4004800 0x7f5fa4004c10 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00016c0 tx=0x7f5fa4005830 comp rx=0 tx=0).stop 2024-06-29T14:39:49.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.242+0000 7f5fc24ce640 1 -- 172.21.15.81:0/2637105722 shutdown_connections 2024-06-29T14:39:49.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.242+0000 7f5fc24ce640 1 --2- 172.21.15.81:0/2637105722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fa4004800 0x7f5fa4004c10 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.242+0000 7f5fc24ce640 1 -- 172.21.15.81:0/2637105722 >> 172.21.15.81:0/2637105722 conn(0x7f5fbc0b3510 msgr2=0x7f5fbc0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:49.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.242+0000 7f5fc24ce640 1 -- 172.21.15.81:0/2637105722 shutdown_connections 2024-06-29T14:39:49.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.242+0000 7f5fc24ce640 1 -- 172.21.15.81:0/2637105722 wait complete. 2024-06-29T14:39:49.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.244+0000 7f5fc24ce640 1 Processor -- start 2024-06-29T14:39:49.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.244+0000 7f5fc24ce640 1 -- start start 2024-06-29T14:39:49.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fc24ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fa4004800 0x7f5fbc1f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fc24ce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fbc0affa0 0x7f5fbc1dfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fc24ce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fbc070100 0x7f5fbc070510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fc24ce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc0d0660 con 0x7f5fbc0affa0 2024-06-29T14:39:49.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fc24ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5fbc0d04e0 con 0x7f5fa4004800 2024-06-29T14:39:49.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fc24ce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5fbc0d07e0 con 0x7f5fbc070100 2024-06-29T14:39:49.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fbb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fbc0affa0 0x7f5fbc1dfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fbb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fbc0affa0 0x7f5fbc1dfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54280/0 (socket says 172.21.15.81:54280) 2024-06-29T14:39:49.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fbb7fe640 1 -- 172.21.15.81:0/1003647437 learned_addr learned my addr 172.21.15.81:0/1003647437 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:49.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fbbfff640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fa4004800 0x7f5fbc1f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.245+0000 7f5fc0a6b640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fbc070100 0x7f5fbc070510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fbb7fe640 1 -- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fbc070100 msgr2=0x7f5fbc070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fbb7fe640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fbc070100 0x7f5fbc070510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fbb7fe640 1 -- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fa4004800 msgr2=0x7f5fbc1f6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fbb7fe640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fa4004800 0x7f5fbc1f6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fbb7fe640 1 -- 172.21.15.81:0/1003647437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fbc1f7370 con 0x7f5fbc0affa0 2024-06-29T14:39:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fc0a6b640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fbc070100 0x7f5fbc070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fbb7fe640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fbc0affa0 0x7f5fbc1dfd50 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f5fb002a910 tx=0x7f5fb00063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fb8ff9640 1 -- 172.21.15.81:0/1003647437 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0003a00 con 0x7f5fbc0affa0 2024-06-29T14:39:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fb8ff9640 1 -- 172.21.15.81:0/1003647437 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5fb0039070 con 0x7f5fbc0affa0 2024-06-29T14:39:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fc24ce640 1 -- 172.21.15.81:0/1003647437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fbc070a50 con 0x7f5fbc0affa0 2024-06-29T14:39:49.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fb8ff9640 1 -- 172.21.15.81:0/1003647437 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0033420 con 0x7f5fbc0affa0 2024-06-29T14:39:49.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.246+0000 7f5fc24ce640 1 -- 172.21.15.81:0/1003647437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fbc0ae920 con 0x7f5fbc0affa0 2024-06-29T14:39:49.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.247+0000 7f5fc24ce640 1 -- 172.21.15.81:0/1003647437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fbc0b49d0 con 0x7f5fbc0affa0 2024-06-29T14:39:49.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.248+0000 7f5fb8ff9640 1 -- 172.21.15.81:0/1003647437 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f5fb0044020 con 0x7f5fbc0affa0 2024-06-29T14:39:49.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.248+0000 7f5fb8ff9640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5fa80763f0 0x7f5fa8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.248+0000 7f5fb8ff9640 1 -- 172.21.15.81:0/1003647437 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5789+0+0 (secure 0 0 0) 0x7f5fb00b71c0 con 0x7f5fbc0affa0 2024-06-29T14:39:49.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.248+0000 7f5fbbfff640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5fa80763f0 0x7f5fa8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.249+0000 7f5fbbfff640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5fa80763f0 0x7f5fa8078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5fac009cd0 tx=0x7f5fac005f90 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:49.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.253+0000 7f5fb8ff9640 1 -- 172.21.15.81:0/1003647437 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5fbc070a50 con 0x7f5fbc0affa0 2024-06-29T14:39:49.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.500+0000 7f5fc24ce640 1 -- 172.21.15.81:0/1003647437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) -- 0x7f5fbc0b1d90 con 0x7f5fbc0affa0 2024-06-29T14:39:49.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.781+0000 7f5fb8ff9640 1 -- 172.21.15.81:0/1003647437 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v645) ==== 183+0+0 (secure 0 0 0) 0x7f5fbc1f7370 con 0x7f5fbc0affa0 2024-06-29T14:39:49.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 -- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5fa80763f0 msgr2=0x7f5fa8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5fa80763f0 0x7f5fa8078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5fac009cd0 tx=0x7f5fac005f90 comp rx=0 tx=0).stop 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 -- 172.21.15.81:0/1003647437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fbc0affa0 msgr2=0x7f5fbc1dfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fbc0affa0 0x7f5fbc1dfd50 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f5fb002a910 tx=0x7f5fb00063a0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 -- 172.21.15.81:0/1003647437 shutdown_connections 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5fa80763f0 0x7f5fa8078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5fbc070100 0x7f5fbc070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fbc0affa0 0x7f5fbc1dfd50 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 --2- 172.21.15.81:0/1003647437 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5fa4004800 0x7f5fbc1f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 -- 172.21.15.81:0/1003647437 >> 172.21.15.81:0/1003647437 conn(0x7f5fbc0b3510 msgr2=0x7f5fbc0b6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:49.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 -- 172.21.15.81:0/1003647437 shutdown_connections 2024-06-29T14:39:49.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.783+0000 7f5fc24ce640 1 -- 172.21.15.81:0/1003647437 wait complete. 2024-06-29T14:39:49.794 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2024-06-29T14:39:49.794 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2024-06-29T14:39:49.795 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:39:49.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:49.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.912+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/3352803715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b406ef90 msgr2=0x7fc5b406f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.912+0000 7fc5bb9bd640 1 --2- 172.21.15.81:0/3352803715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b406ef90 0x7fc5b406f3a0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4071e60 tx=0x7fc5a402a560 comp rx=0 tx=0).stop 2024-06-29T14:39:49.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.912+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/3352803715 shutdown_connections 2024-06-29T14:39:49.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.912+0000 7fc5bb9bd640 1 --2- 172.21.15.81:0/3352803715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b406ef90 0x7fc5b406f3a0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.912+0000 7fc5bb9bd640 1 --2- 172.21.15.81:0/3352803715 >> v2:172.21.15.167:3300/0 conn(0x7fc5b40b57b0 0x7fc5b415aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.912+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/3352803715 >> 172.21.15.81:0/3352803715 conn(0x7fc5b40d0930 msgr2=0x7fc5b40d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:49.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.912+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/3352803715 shutdown_connections 2024-06-29T14:39:49.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.912+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/3352803715 wait complete. 2024-06-29T14:39:49.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.914+0000 7fc5bb9bd640 1 Processor -- start 2024-06-29T14:39:49.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.914+0000 7fc5bb9bd640 1 -- start start 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.915+0000 7fc5bb9bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5b406ef90 0x7fc5b41f6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.915+0000 7fc5bb9bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5b40b57b0 0x7fc5b40cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.915+0000 7fc5bb9bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b40ce1d0 0x7fc5b40c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.915+0000 7fc5bb9bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b40b4b30 con 0x7fc5b40ce1d0 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.915+0000 7fc5bb9bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b40b49b0 con 0x7fc5b40b57b0 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.915+0000 7fc5bb9bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b41f6890 con 0x7fc5b406ef90 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.915+0000 7fc5b9f5a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b40ce1d0 0x7fc5b40c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.915+0000 7fc5b9f5a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b40ce1d0 0x7fc5b40c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54304/0 (socket says 172.21.15.81:54304) 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.915+0000 7fc5b9f5a640 1 -- 172.21.15.81:0/2806456337 learned_addr learned my addr 172.21.15.81:0/2806456337 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5b8f58640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5b40b57b0 0x7fc5b40cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5b9759640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5b406ef90 0x7fc5b41f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5b9f5a640 1 -- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5b406ef90 msgr2=0x7fc5b41f6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5b9f5a640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5b406ef90 0x7fc5b41f6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5b9f5a640 1 -- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5b40b57b0 msgr2=0x7fc5b40cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5b9f5a640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5b40b57b0 0x7fc5b40cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5b9f5a640 1 -- 172.21.15.81:0/2806456337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b40cac80 con 0x7fc5b40ce1d0 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5b9f5a640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b40ce1d0 0x7fc5b40c9eb0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00b410 tx=0x7fc5ac00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5a9ffb640 1 -- 172.21.15.81:0/2806456337 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac015030 con 0x7fc5b40ce1d0 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5a9ffb640 1 -- 172.21.15.81:0/2806456337 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5ac004020 con 0x7fc5b40ce1d0 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/2806456337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b4153440 con 0x7fc5b40ce1d0 2024-06-29T14:39:49.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5a9ffb640 1 -- 172.21.15.81:0/2806456337 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac011630 con 0x7fc5b40ce1d0 2024-06-29T14:39:49.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.916+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/2806456337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b4153a00 con 0x7fc5b40ce1d0 2024-06-29T14:39:49.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.917+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/2806456337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b406e2a0 con 0x7fc5b40ce1d0 2024-06-29T14:39:49.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.918+0000 7fc5a9ffb640 1 -- 172.21.15.81:0/2806456337 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fc5ac018070 con 0x7fc5b40ce1d0 2024-06-29T14:39:49.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.918+0000 7fc5a9ffb640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc5980763f0 0x7fc598078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:49.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.918+0000 7fc5a9ffb640 1 -- 172.21.15.81:0/2806456337 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5789+0+0 (secure 0 0 0) 0x7fc5ac097480 con 0x7fc5b40ce1d0 2024-06-29T14:39:49.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.919+0000 7fc5b9759640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc5980763f0 0x7fc598078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:49.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.919+0000 7fc5b9759640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc5980763f0 0x7fc598078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0005fd0 tx=0x7fc5b00097d0 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:49.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:49.923+0000 7fc5a9ffb640 1 -- 172.21.15.81:0/2806456337 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc5b406e2a0 con 0x7fc5b40ce1d0 2024-06-29T14:39:50.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.104+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/2806456337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc5b40cf020 con 0x7fc5b40ce1d0 2024-06-29T14:39:50.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.105+0000 7fc5a9ffb640 1 -- 172.21.15.81:0/2806456337 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+14274 (secure 0 0 0) 0x7fc5ac060c80 con 0x7fc5b40ce1d0 2024-06-29T14:39:50.103 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:50.103 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":111,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:39:48.772694+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6835/3193613343":"2024-06-30T14:39:45.618850+0000","172.21.15.167:0/51861160":"2024-06-30T14:39:43.383162+0000","172.21.15.167:6837/3193613343":"2024-06-30T14:39:45.618850+0000","172.21.15.167:0/3916841092":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/4191187352":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/3979799678":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/2383909496":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/2133120885":"2024-06-30T14:39:43.383162+0000","172.21.15.167:6833/295682379":"2024-06-30T14:39:43.383162+0000","172.21.15.167:6832/295682379":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/3578595575":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2870419655":"2024-06-30T14:38:36.587419+0000","172.21.15.167:6833/2583731609":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/3737328049":"2024-06-30T14:39:43.383162+0000","172.21.15.167:6832/2583731609":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/1601099482":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2667854206":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/32033638":"2024-06-30T14:38:36.587419+0000","172.21.15.167:0/2697428082":"2024-06-30T14:39:43.383162+0000","172.21.15.167:0/4125253667":"2024-06-30T14:38:36.587419+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc5980763f0 msgr2=0x7fc598078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc5980763f0 0x7fc598078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0005fd0 tx=0x7fc5b00097d0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/2806456337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b40ce1d0 msgr2=0x7fc5b40c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b40ce1d0 0x7fc5b40c9eb0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00b410 tx=0x7fc5ac00b8d0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/2806456337 shutdown_connections 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc5980763f0 0x7fc598078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b40ce1d0 0x7fc5b40c9eb0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc5b40b57b0 0x7fc5b40cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 --2- 172.21.15.81:0/2806456337 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc5b406ef90 0x7fc5b41f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/2806456337 >> 172.21.15.81:0/2806456337 conn(0x7fc5b40d0930 msgr2=0x7fc5b40cf640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:50.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/2806456337 shutdown_connections 2024-06-29T14:39:50.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.107+0000 7fc5bb9bd640 1 -- 172.21.15.81:0/2806456337 wait complete. 2024-06-29T14:39:50.117 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:39:50.117 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:39:50.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.235+0000 7f5bccbf0640 1 -- 172.21.15.81:0/2823004687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb4004d40 msgr2=0x7f5bb4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.235+0000 7f5bccbf0640 1 --2- 172.21.15.81:0/2823004687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb4004d40 0x7f5bb4005190 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80b61c0 tx=0x7f5bb4005830 comp rx=0 tx=0).stop 2024-06-29T14:39:50.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.235+0000 7f5bc6d9d640 1 -- 172.21.15.81:0/2823004687 reap_dead start 2024-06-29T14:39:50.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.235+0000 7f5bccbf0640 1 -- 172.21.15.81:0/2823004687 shutdown_connections 2024-06-29T14:39:50.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.235+0000 7f5bccbf0640 1 -- 172.21.15.81:0/2823004687 >> 172.21.15.81:0/2823004687 conn(0x7f5bc8072950 msgr2=0x7f5bc80b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:50.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.236+0000 7f5bccbf0640 1 -- 172.21.15.81:0/2823004687 shutdown_connections 2024-06-29T14:39:50.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.236+0000 7f5bccbf0640 1 -- 172.21.15.81:0/2823004687 wait complete. 2024-06-29T14:39:50.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.238+0000 7f5bccbf0640 1 Processor -- start 2024-06-29T14:39:50.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.238+0000 7f5bccbf0640 1 -- start start 2024-06-29T14:39:50.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bccbf0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb4004d40 0x7f5bc8158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bccbf0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bc81f78a0 0x7f5bc81531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bccbf0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bc8153730 0x7f5bc80c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bccbf0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc815b190 con 0x7f5bc8153730 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bccbf0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc815b010 con 0x7f5bb4004d40 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bccbf0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc80af360 con 0x7f5bc81f78a0 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bc8153730 0x7f5bc80c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bc8153730 0x7f5bc80c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44528/0 (socket says 172.21.15.81:44528) 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc6d9d640 1 -- 172.21.15.81:0/1647260376 learned_addr learned my addr 172.21.15.81:0/1647260376 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bbdd9b640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bc81f78a0 0x7f5bc81531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc659c640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb4004d40 0x7f5bc8158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc6d9d640 1 -- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bc81f78a0 msgr2=0x7f5bc81531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc6d9d640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bc81f78a0 0x7f5bc81531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc6d9d640 1 -- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb4004d40 msgr2=0x7f5bc8158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc6d9d640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb4004d40 0x7f5bc8158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc6d9d640 1 -- 172.21.15.81:0/1647260376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc81598f0 con 0x7f5bc8153730 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc659c640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb4004d40 0x7f5bc8158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bbdd9b640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bc81f78a0 0x7f5bc81531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.239+0000 7f5bc6d9d640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bc8153730 0x7f5bc80c9460 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000c700 tx=0x7f5bc0004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.240+0000 7f5bbf7fe640 1 -- 172.21.15.81:0/1647260376 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc002bb70 con 0x7f5bc8153730 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.240+0000 7f5bbf7fe640 1 -- 172.21.15.81:0/1647260376 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bc002bd10 con 0x7f5bc8153730 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.240+0000 7f5bccbf0640 1 -- 172.21.15.81:0/1647260376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc8159bd0 con 0x7f5bc8153730 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.240+0000 7f5bbf7fe640 1 -- 172.21.15.81:0/1647260376 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00062b0 con 0x7f5bc8153730 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.240+0000 7f5bccbf0640 1 -- 172.21.15.81:0/1647260376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc8153b40 con 0x7f5bc8153730 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.241+0000 7f5bccbf0640 1 -- 172.21.15.81:0/1647260376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc80ae390 con 0x7f5bc8153730 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.241+0000 7f5bbf7fe640 1 -- 172.21.15.81:0/1647260376 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f5bc0006450 con 0x7f5bc8153730 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.242+0000 7f5bbf7fe640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5ba80763f0 0x7f5ba8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.242+0000 7f5bbf7fe640 1 -- 172.21.15.81:0/1647260376 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5789+0+0 (secure 0 0 0) 0x7f5bc00b7df0 con 0x7f5bc8153730 2024-06-29T14:39:50.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.242+0000 7f5bc659c640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5ba80763f0 0x7f5ba8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.243+0000 7f5bc659c640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5ba80763f0 0x7f5ba8078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5bb80011a0 tx=0x7f5bb8006b10 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:50.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.246+0000 7f5bbf7fe640 1 -- 172.21.15.81:0/1647260376 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5bc00815f0 con 0x7f5bc8153730 2024-06-29T14:39:50.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.426+0000 7f5bccbf0640 1 -- 172.21.15.81:0/1647260376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5bc80cc470 con 0x7f5bc8153730 2024-06-29T14:39:50.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.426+0000 7f5bbf7fe640 1 -- 172.21.15.81:0/1647260376 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) ==== 93+0+10 (secure 0 0 0) 0x7f5bc0080ef0 con 0x7f5bc8153730 2024-06-29T14:39:50.425 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:39:50.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.428+0000 7f5bccbf0640 1 -- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5ba80763f0 msgr2=0x7f5ba8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.428+0000 7f5bccbf0640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5ba80763f0 0x7f5ba8078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5bb80011a0 tx=0x7f5bb8006b10 comp rx=0 tx=0).stop 2024-06-29T14:39:50.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.428+0000 7f5bccbf0640 1 -- 172.21.15.81:0/1647260376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bc8153730 msgr2=0x7f5bc80c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.428+0000 7f5bccbf0640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bc8153730 0x7f5bc80c9460 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000c700 tx=0x7f5bc0004bb0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.429+0000 7f5bccbf0640 1 -- 172.21.15.81:0/1647260376 shutdown_connections 2024-06-29T14:39:50.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.429+0000 7f5bccbf0640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f5ba80763f0 0x7f5ba8078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.429+0000 7f5bccbf0640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bc8153730 0x7f5bc80c9460 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.429+0000 7f5bccbf0640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5bc81f78a0 0x7f5bc81531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.429+0000 7f5bccbf0640 1 --2- 172.21.15.81:0/1647260376 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5bb4004d40 0x7f5bc8158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.429+0000 7f5bccbf0640 1 -- 172.21.15.81:0/1647260376 >> 172.21.15.81:0/1647260376 conn(0x7f5bc8072950 msgr2=0x7f5bc809d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:50.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.429+0000 7f5bccbf0640 1 -- 172.21.15.81:0/1647260376 shutdown_connections 2024-06-29T14:39:50.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.429+0000 7f5bccbf0640 1 -- 172.21.15.81:0/1647260376 wait complete. 2024-06-29T14:39:50.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-06-29T14:39:50.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.555+0000 7f058f0e3640 1 -- 172.21.15.81:0/1193590565 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05880cea50 msgr2=0x7f05880b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.555+0000 7f058f0e3640 1 --2- 172.21.15.81:0/1193590565 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05880cea50 0x7f05880b4810 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f0580005fd0 tx=0x7f058002a820 comp rx=0 tx=0).stop 2024-06-29T14:39:50.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.555+0000 7f058f0e3640 1 -- 172.21.15.81:0/1193590565 shutdown_connections 2024-06-29T14:39:50.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.555+0000 7f058f0e3640 1 --2- 172.21.15.81:0/1193590565 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05880cea50 0x7f05880b4810 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.555+0000 7f058f0e3640 1 --2- 172.21.15.81:0/1193590565 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05880cd480 0x7f05880ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.555+0000 7f058f0e3640 1 -- 172.21.15.81:0/1193590565 >> 172.21.15.81:0/1193590565 conn(0x7f05880b3510 msgr2=0x7f05880b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:50.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.555+0000 7f058f0e3640 1 -- 172.21.15.81:0/1193590565 shutdown_connections 2024-06-29T14:39:50.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.555+0000 7f058f0e3640 1 -- 172.21.15.81:0/1193590565 wait complete. 2024-06-29T14:39:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.557+0000 7f058f0e3640 1 Processor -- start 2024-06-29T14:39:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.557+0000 7f058f0e3640 1 -- start start 2024-06-29T14:39:50.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.558+0000 7f058f0e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05880b6eb0 0x7f058814b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.558+0000 7f058f0e3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05880cd480 0x7f058815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.558+0000 7f058f0e3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05880cea50 0x7f058815e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.558+0000 7f058f0e3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f05880b50e0 con 0x7f05880b6eb0 2024-06-29T14:39:50.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.558+0000 7f058f0e3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f05880b4f60 con 0x7f05880cea50 2024-06-29T14:39:50.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.558+0000 7f058f0e3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f05880b5260 con 0x7f05880cd480 2024-06-29T14:39:50.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058ce7f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05880b6eb0 0x7f058814b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058ce7f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05880b6eb0 0x7f058814b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44530/0 (socket says 172.21.15.81:44530) 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058ce7f640 1 -- 172.21.15.81:0/3988526194 learned_addr learned my addr 172.21.15.81:0/3988526194 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058d680640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05880cea50 0x7f058815e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f0587fff640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05880cd480 0x7f058815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058ce7f640 1 -- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05880cd480 msgr2=0x7f058815dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058ce7f640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05880cd480 0x7f058815dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058ce7f640 1 -- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05880cea50 msgr2=0x7f058815e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058ce7f640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05880cea50 0x7f058815e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058ce7f640 1 -- 172.21.15.81:0/3988526194 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0588160ea0 con 0x7f05880b6eb0 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058d680640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05880cea50 0x7f058815e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058ce7f640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05880b6eb0 0x7f058814b6b0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f057800c3a0 tx=0x7f057800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f05857fa640 1 -- 172.21.15.81:0/3988526194 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05780052d0 con 0x7f05880b6eb0 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f05857fa640 1 -- 172.21.15.81:0/3988526194 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0578007020 con 0x7f05880b6eb0 2024-06-29T14:39:50.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f05857fa640 1 -- 172.21.15.81:0/3988526194 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578007470 con 0x7f05880b6eb0 2024-06-29T14:39:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.559+0000 7f058f0e3640 1 -- 172.21.15.81:0/3988526194 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058814bbf0 con 0x7f05880b6eb0 2024-06-29T14:39:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.560+0000 7f058f0e3640 1 -- 172.21.15.81:0/3988526194 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0588161460 con 0x7f05880b6eb0 2024-06-29T14:39:50.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.561+0000 7f058f0e3640 1 -- 172.21.15.81:0/3988526194 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05881616a0 con 0x7f05880b6eb0 2024-06-29T14:39:50.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.561+0000 7f05857fa640 1 -- 172.21.15.81:0/3988526194 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f05780067d0 con 0x7f05880b6eb0 2024-06-29T14:39:50.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.561+0000 7f05857fa640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0574076420 0x7f0574078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:50.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.562+0000 7f05857fa640 1 -- 172.21.15.81:0/3988526194 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5789+0+0 (secure 0 0 0) 0x7f0578096ed0 con 0x7f05880b6eb0 2024-06-29T14:39:50.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.562+0000 7f0587fff640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0574076420 0x7f0574078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:50.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.562+0000 7f0587fff640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0574076420 0x7f0574078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f05881590f0 tx=0x7f057c009260 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:50.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.566+0000 7f05857fa640 1 -- 172.21.15.81:0/3988526194 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f05881616a0 con 0x7f05880b6eb0 2024-06-29T14:39:50.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:50.817+0000 7f058f0e3640 1 -- 172.21.15.81:0/3988526194 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) -- 0x7f05880ce510 con 0x7f05880b6eb0 2024-06-29T14:39:51.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.787+0000 7f05857fa640 1 -- 172.21.15.81:0/3988526194 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v647) ==== 151+0+0 (secure 0 0 0) 0x7f0578060640 con 0x7f05880b6eb0 2024-06-29T14:39:51.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.790+0000 7f058f0e3640 1 -- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0574076420 msgr2=0x7f0574078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:51.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.790+0000 7f058f0e3640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0574076420 0x7f0574078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f05881590f0 tx=0x7f057c009260 comp rx=0 tx=0).stop 2024-06-29T14:39:51.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.790+0000 7f058f0e3640 1 -- 172.21.15.81:0/3988526194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05880b6eb0 msgr2=0x7f058814b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:51.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.790+0000 7f058f0e3640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05880b6eb0 0x7f058814b6b0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f057800c3a0 tx=0x7f057800c860 comp rx=0 tx=0).stop 2024-06-29T14:39:51.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.791+0000 7f058f0e3640 1 -- 172.21.15.81:0/3988526194 shutdown_connections 2024-06-29T14:39:51.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.791+0000 7f058f0e3640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0574076420 0x7f0574078930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.791+0000 7f058f0e3640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05880cea50 0x7f058815e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.791+0000 7f058f0e3640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05880cd480 0x7f058815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.791+0000 7f058f0e3640 1 --2- 172.21.15.81:0/3988526194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05880b6eb0 0x7f058814b6b0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.791+0000 7f058f0e3640 1 -- 172.21.15.81:0/3988526194 >> 172.21.15.81:0/3988526194 conn(0x7f05880b3510 msgr2=0x7f0588165730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:51.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.791+0000 7f058f0e3640 1 -- 172.21.15.81:0/3988526194 shutdown_connections 2024-06-29T14:39:51.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.791+0000 7f058f0e3640 1 -- 172.21.15.81:0/3988526194 wait complete. 2024-06-29T14:39:51.803 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:39:51.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.924+0000 7fcf50dbd640 1 -- 172.21.15.81:0/3159436094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0b6eb0 msgr2=0x7fcf4c0d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.924+0000 7fcf50dbd640 1 --2- 172.21.15.81:0/3159436094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0b6eb0 0x7fcf4c0d02d0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c073550 tx=0x7fcf3c02a820 comp rx=0 tx=0).stop 2024-06-29T14:39:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.924+0000 7fcf50dbd640 1 -- 172.21.15.81:0/3159436094 shutdown_connections 2024-06-29T14:39:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.924+0000 7fcf50dbd640 1 --2- 172.21.15.81:0/3159436094 >> v2:172.21.15.167:3300/0 conn(0x7fcf4c0ce510 0x7fcf4c0ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.924+0000 7fcf50dbd640 1 --2- 172.21.15.81:0/3159436094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0b6eb0 0x7fcf4c0d02d0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.924+0000 7fcf50dbd640 1 -- 172.21.15.81:0/3159436094 >> 172.21.15.81:0/3159436094 conn(0x7fcf4c0b3510 msgr2=0x7fcf4c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.924+0000 7fcf50dbd640 1 -- 172.21.15.81:0/3159436094 shutdown_connections 2024-06-29T14:39:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.924+0000 7fcf50dbd640 1 -- 172.21.15.81:0/3159436094 wait complete. 2024-06-29T14:39:51.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.926+0000 7fcf50dbd640 1 Processor -- start 2024-06-29T14:39:51.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.926+0000 7fcf50dbd640 1 -- start start 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf50dbd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf4c0b4810 0x7fcf4c0c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf50dbd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf4c0b6eb0 0x7fcf4c09ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf50dbd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0ce510 0x7fcf4c09f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf50dbd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c0cd480 con 0x7fcf4c0ce510 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf50dbd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c0b29c0 con 0x7fcf4c0b6eb0 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf50dbd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4c0cd600 con 0x7fcf4c0b4810 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf4ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0ce510 0x7fcf4c09f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf4ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0ce510 0x7fcf4c09f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44554/0 (socket says 172.21.15.81:44554) 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf4ad9d640 1 -- 172.21.15.81:0/2684948651 learned_addr learned my addr 172.21.15.81:0/2684948651 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf49d9b640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf4c0b6eb0 0x7fcf4c09ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:51.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf4a59c640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf4c0b4810 0x7fcf4c0c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf4ad9d640 1 -- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf4c0b4810 msgr2=0x7fcf4c0c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf4ad9d640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf4c0b4810 0x7fcf4c0c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf4ad9d640 1 -- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf4c0b6eb0 msgr2=0x7fcf4c09ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.927+0000 7fcf4ad9d640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf4c0b6eb0 0x7fcf4c09ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.928+0000 7fcf4ad9d640 1 -- 172.21.15.81:0/2684948651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c09b900 con 0x7fcf4c0ce510 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.928+0000 7fcf4ad9d640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0ce510 0x7fcf4c09f1f0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400c390 tx=0x7fcf4400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.928+0000 7fcf3affd640 1 -- 172.21.15.81:0/2684948651 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44017070 con 0x7fcf4c0ce510 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.928+0000 7fcf3affd640 1 -- 172.21.15.81:0/2684948651 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf440035f0 con 0x7fcf4c0ce510 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.928+0000 7fcf50dbd640 1 -- 172.21.15.81:0/2684948651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf4c09def0 con 0x7fcf4c0ce510 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.928+0000 7fcf3affd640 1 -- 172.21.15.81:0/2684948651 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44003a40 con 0x7fcf4c0ce510 2024-06-29T14:39:51.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.928+0000 7fcf50dbd640 1 -- 172.21.15.81:0/2684948651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf4c09e4b0 con 0x7fcf4c0ce510 2024-06-29T14:39:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.929+0000 7fcf50dbd640 1 -- 172.21.15.81:0/2684948651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf4c09bb70 con 0x7fcf4c0ce510 2024-06-29T14:39:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.929+0000 7fcf3affd640 1 -- 172.21.15.81:0/2684948651 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fcf4401a070 con 0x7fcf4c0ce510 2024-06-29T14:39:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.930+0000 7fcf3affd640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcf2c0763f0 0x7fcf2c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:51.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.930+0000 7fcf3affd640 1 -- 172.21.15.81:0/2684948651 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5789+0+0 (secure 0 0 0) 0x7fcf44096b70 con 0x7fcf4c0ce510 2024-06-29T14:39:51.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.930+0000 7fcf4a59c640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcf2c0763f0 0x7fcf2c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:51.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.931+0000 7fcf4a59c640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcf2c0763f0 0x7fcf2c078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c034000 tx=0x7fcf3c004ab0 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:51.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:51.934+0000 7fcf3affd640 1 -- 172.21.15.81:0/2684948651 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcf4405fd20 con 0x7fcf4c0ce510 2024-06-29T14:39:52.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.118+0000 7fcf50dbd640 1 -- 172.21.15.81:0/2684948651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fcf4c0d02d0 con 0x7fcf4c0ce510 2024-06-29T14:39:52.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.119+0000 7fcf3affd640 1 -- 172.21.15.81:0/2684948651 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v47) ==== 59+0+1998 (secure 0 0 0) 0x7fcf44066030 con 0x7fcf4c0ce510 2024-06-29T14:39:52.117 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:39:52.117 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:39:52.117 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:39:52.117 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T14:39:52.118 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:39:52.119 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:39:52.119 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T14:39:52.119 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:39:52.119 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:39:52.119 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:39:52.119 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T14:39:52.119 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T14:39:52.119 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T14:39:52.119 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:39:52.119 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:39:52.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 -- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcf2c0763f0 msgr2=0x7fcf2c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcf2c0763f0 0x7fcf2c078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c034000 tx=0x7fcf3c004ab0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 -- 172.21.15.81:0/2684948651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0ce510 msgr2=0x7fcf4c09f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0ce510 0x7fcf4c09f1f0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400c390 tx=0x7fcf4400c850 comp rx=0 tx=0).stop 2024-06-29T14:39:52.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 -- 172.21.15.81:0/2684948651 shutdown_connections 2024-06-29T14:39:52.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fcf2c0763f0 0x7fcf2c078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0ce510 0x7fcf4c09f1f0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf4c0b6eb0 0x7fcf4c09ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 --2- 172.21.15.81:0/2684948651 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf4c0b4810 0x7fcf4c0c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 -- 172.21.15.81:0/2684948651 >> 172.21.15.81:0/2684948651 conn(0x7fcf4c0b3510 msgr2=0x7fcf4c068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:52.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 -- 172.21.15.81:0/2684948651 shutdown_connections 2024-06-29T14:39:52.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.121+0000 7fcf50dbd640 1 -- 172.21.15.81:0/2684948651 wait complete. 2024-06-29T14:39:52.130 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:39:52.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.248+0000 7f5715664640 1 -- 172.21.15.81:0/2820524613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571015e960 msgr2=0x7f571015ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.248+0000 7f5715664640 1 --2- 172.21.15.81:0/2820524613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571015e960 0x7f571015ed70 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f5708009830 tx=0x7f570802a510 comp rx=0 tx=0).stop 2024-06-29T14:39:52.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.248+0000 7f5715664640 1 -- 172.21.15.81:0/2820524613 shutdown_connections 2024-06-29T14:39:52.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.248+0000 7f5715664640 1 --2- 172.21.15.81:0/2820524613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571015e960 0x7f571015ed70 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.248+0000 7f5715664640 1 --2- 172.21.15.81:0/2820524613 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5710073030 0x7f5710073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.248+0000 7f5715664640 1 -- 172.21.15.81:0/2820524613 >> 172.21.15.81:0/2820524613 conn(0x7f571006ccc0 msgr2=0x7f57100b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:52.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.248+0000 7f5715664640 1 -- 172.21.15.81:0/2820524613 shutdown_connections 2024-06-29T14:39:52.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.248+0000 7f5715664640 1 -- 172.21.15.81:0/2820524613 wait complete. 2024-06-29T14:39:52.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.250+0000 7f5715664640 1 Processor -- start 2024-06-29T14:39:52.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.250+0000 7f5715664640 1 -- start start 2024-06-29T14:39:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f5715664640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5710073030 0x7f571013f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f5715664640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57101412f0 0x7f57100b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f5715664640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571015e960 0x7f57100cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f5715664640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f57100b3390 con 0x7f571015e960 2024-06-29T14:39:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f5715664640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f57100b3210 con 0x7f57101412f0 2024-06-29T14:39:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f5715664640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f57100b3510 con 0x7f5710073030 2024-06-29T14:39:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f570f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571015e960 0x7f57100cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f570f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571015e960 0x7f57100cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44574/0 (socket says 172.21.15.81:44574) 2024-06-29T14:39:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f570f7fe640 1 -- 172.21.15.81:0/2988582247 learned_addr learned my addr 172.21.15.81:0/2988582247 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f570effd640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5710073030 0x7f571013f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f570e7fc640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57101412f0 0x7f57100b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f570f7fe640 1 -- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5710073030 msgr2=0x7f571013f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f570f7fe640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5710073030 0x7f571013f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f570f7fe640 1 -- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57101412f0 msgr2=0x7f57100b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f570f7fe640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57101412f0 0x7f57100b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.251+0000 7f570f7fe640 1 -- 172.21.15.81:0/2988582247 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5710158e30 con 0x7f571015e960 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.252+0000 7f570f7fe640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571015e960 0x7f57100cc220 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f5708009920 tx=0x7f5708033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.252+0000 7f56f77fe640 1 -- 172.21.15.81:0/2988582247 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708031d60 con 0x7f571015e960 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.252+0000 7f56f77fe640 1 -- 172.21.15.81:0/2988582247 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5708033e10 con 0x7f571015e960 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.252+0000 7f5715664640 1 -- 172.21.15.81:0/2988582247 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5710159110 con 0x7f571015e960 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.252+0000 7f56f77fe640 1 -- 172.21.15.81:0/2988582247 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5708005220 con 0x7f571015e960 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.252+0000 7f5715664640 1 -- 172.21.15.81:0/2988582247 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57101596d0 con 0x7f571015e960 2024-06-29T14:39:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.253+0000 7f5715664640 1 -- 172.21.15.81:0/2988582247 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57100b0e60 con 0x7f571015e960 2024-06-29T14:39:52.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.254+0000 7f56f77fe640 1 -- 172.21.15.81:0/2988582247 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f57080053c0 con 0x7f571015e960 2024-06-29T14:39:52.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.254+0000 7f56f77fe640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f56fc076420 0x7f56fc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.254+0000 7f56f77fe640 1 -- 172.21.15.81:0/2988582247 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5789+0+0 (secure 0 0 0) 0x7f57080b8ae0 con 0x7f571015e960 2024-06-29T14:39:52.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.254+0000 7f570effd640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f56fc076420 0x7f56fc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.255+0000 7f570effd640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f56fc076420 0x7f56fc078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f57000012e0 tx=0x7f5700009040 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:52.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.258+0000 7f56f77fe640 1 -- 172.21.15.81:0/2988582247 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f570803f030 con 0x7f571015e960 2024-06-29T14:39:52.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.438+0000 7f5715664640 1 -- 172.21.15.81:0/2988582247 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f5710068bd0 con 0x7f571015e960 2024-06-29T14:39:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.441+0000 7f56f77fe640 1 -- 172.21.15.81:0/2988582247 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) ==== 122+0+0 (secure 0 0 0) 0x7f5710068bd0 con 0x7f571015e960 2024-06-29T14:39:52.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 -- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f56fc076420 msgr2=0x7f56fc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f56fc076420 0x7f56fc078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f57000012e0 tx=0x7f5700009040 comp rx=0 tx=0).stop 2024-06-29T14:39:52.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 -- 172.21.15.81:0/2988582247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571015e960 msgr2=0x7f57100cc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571015e960 0x7f57100cc220 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f5708009920 tx=0x7f5708033c20 comp rx=0 tx=0).stop 2024-06-29T14:39:52.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 -- 172.21.15.81:0/2988582247 shutdown_connections 2024-06-29T14:39:52.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f56fc076420 0x7f56fc078930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571015e960 0x7f57100cc220 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57101412f0 0x7f57100b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 --2- 172.21.15.81:0/2988582247 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5710073030 0x7f571013f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 -- 172.21.15.81:0/2988582247 >> 172.21.15.81:0/2988582247 conn(0x7f571006ccc0 msgr2=0x7f571006e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:52.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 -- 172.21.15.81:0/2988582247 shutdown_connections 2024-06-29T14:39:52.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.444+0000 7f5715664640 1 -- 172.21.15.81:0/2988582247 wait complete. 2024-06-29T14:39:52.454 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:39:52.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:39:52.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:39:52.533 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:39:52.533 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:39:52.533 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:52.549 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:39:52.549 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:39:52.549 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:52.592 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:39:52.592 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:39:52.592 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:52.608 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:39:52.608 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:39:52.608 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:39:52.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:39:52.650 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:39:52.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:39:52.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.764+0000 7fd7a394b640 1 -- 172.21.15.81:0/1427202840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c0cf600 msgr2=0x7fd79c0cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.764+0000 7fd7a394b640 1 --2- 172.21.15.81:0/1427202840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c0cf600 0x7fd79c0cfa10 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fd798009830 tx=0x7fd79802a510 comp rx=0 tx=0).stop 2024-06-29T14:39:52.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.764+0000 7fd7a394b640 1 -- 172.21.15.81:0/1427202840 shutdown_connections 2024-06-29T14:39:52.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.764+0000 7fd7a394b640 1 --2- 172.21.15.81:0/1427202840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c0cf600 0x7fd79c0cfa10 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.764+0000 7fd7a394b640 1 --2- 172.21.15.81:0/1427202840 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0ce3a0 0x7fd79c0ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.764+0000 7fd7a394b640 1 -- 172.21.15.81:0/1427202840 >> 172.21.15.81:0/1427202840 conn(0x7fd79c0b35a0 msgr2=0x7fd79c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:52.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.764+0000 7fd7a394b640 1 -- 172.21.15.81:0/1427202840 shutdown_connections 2024-06-29T14:39:52.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.764+0000 7fd7a394b640 1 -- 172.21.15.81:0/1427202840 wait complete. 2024-06-29T14:39:52.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.766+0000 7fd7a394b640 1 Processor -- start 2024-06-29T14:39:52.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.766+0000 7fd7a394b640 1 -- start start 2024-06-29T14:39:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.767+0000 7fd7a394b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0ce3a0 0x7fd79c1df700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.767+0000 7fd7a394b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c0cf600 0x7fd79c1f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a394b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd79c1d8fa0 0x7fd79c1d8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a394b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c0b2830 con 0x7fd79c0cf600 2024-06-29T14:39:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a394b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c0b26b0 con 0x7fd79c1d8fa0 2024-06-29T14:39:52.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a394b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd79c0b6df0 con 0x7fd79c0ce3a0 2024-06-29T14:39:52.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a0ee6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c0cf600 0x7fd79c1f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a0ee6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c0cf600 0x7fd79c1f29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44598/0 (socket says 172.21.15.81:44598) 2024-06-29T14:39:52.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a0ee6640 1 -- 172.21.15.81:0/429095666 learned_addr learned my addr 172.21.15.81:0/429095666 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:52.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a16e7640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0ce3a0 0x7fd79c1df700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a1ee8640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd79c1d8fa0 0x7fd79c1d8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a0ee6640 1 -- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0ce3a0 msgr2=0x7fd79c1df700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a0ee6640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0ce3a0 0x7fd79c1df700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a0ee6640 1 -- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd79c1d8fa0 msgr2=0x7fd79c1d8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a0ee6640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd79c1d8fa0 0x7fd79c1d8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a0ee6640 1 -- 172.21.15.81:0/429095666 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c1e58f0 con 0x7fd79c0cf600 2024-06-29T14:39:52.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a0ee6640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c0cf600 0x7fd79c1f29c0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd78c00d740 tx=0x7fd78c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:52.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd791ffb640 1 -- 172.21.15.81:0/429095666 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78c017070 con 0x7fd79c0cf600 2024-06-29T14:39:52.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd791ffb640 1 -- 172.21.15.81:0/429095666 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd78c009cc0 con 0x7fd79c0cf600 2024-06-29T14:39:52.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd7a394b640 1 -- 172.21.15.81:0/429095666 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd79c1e5bd0 con 0x7fd79c0cf600 2024-06-29T14:39:52.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.768+0000 7fd791ffb640 1 -- 172.21.15.81:0/429095666 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0106c0 con 0x7fd79c0cf600 2024-06-29T14:39:52.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.769+0000 7fd7a394b640 1 -- 172.21.15.81:0/429095666 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd79c1e6190 con 0x7fd79c0cf600 2024-06-29T14:39:52.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.769+0000 7fd7a394b640 1 -- 172.21.15.81:0/429095666 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd79c0b7360 con 0x7fd79c0cf600 2024-06-29T14:39:52.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.770+0000 7fd791ffb640 1 -- 172.21.15.81:0/429095666 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fd78c01a070 con 0x7fd79c0cf600 2024-06-29T14:39:52.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.771+0000 7fd791ffb640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd7800763f0 0x7fd780078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:52.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.771+0000 7fd791ffb640 1 -- 172.21.15.81:0/429095666 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5789+0+0 (secure 0 0 0) 0x7fd78c097ae0 con 0x7fd79c0cf600 2024-06-29T14:39:52.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.771+0000 7fd7a16e7640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd7800763f0 0x7fd780078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:52.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.772+0000 7fd7a16e7640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd7800763f0 0x7fd780078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7940012e0 tx=0x7fd794009040 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:52.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.775+0000 7fd791ffb640 1 -- 172.21.15.81:0/429095666 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd78c060b00 con 0x7fd79c0cf600 2024-06-29T14:39:52.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.994+0000 7fd7a394b640 1 -- 172.21.15.81:0/429095666 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd79c0cdee0 con 0x7fd79c0cf600 2024-06-29T14:39:52.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.995+0000 7fd791ffb640 1 -- 172.21.15.81:0/429095666 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+3914 (secure 0 0 0) 0x7fd78c064e00 con 0x7fd79c0cf600 2024-06-29T14:39:52.993 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:52.993 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":53,"btime":"2024-06-29T14:39:46:620549+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1895620554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1895620554},{"type":"v1","addr":"172.21.15.167:6836","nonce":1895620554}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":7288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/897523091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":897523091},{"type":"v1","addr":"172.21.15.167:6837","nonce":897523091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53},{"gid":7485,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3391563230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3391563230},{"type":"v1","addr":"172.21.15.81:6834","nonce":3391563230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":7488,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3932182351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3932182351},{"type":"v1","addr":"172.21.15.81:6835","nonce":3932182351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[]} 2024-06-29T14:39:52.994 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 53 2024-06-29T14:39:52.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 -- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd7800763f0 msgr2=0x7fd780078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd7800763f0 0x7fd780078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7940012e0 tx=0x7fd794009040 comp rx=0 tx=0).stop 2024-06-29T14:39:52.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 -- 172.21.15.81:0/429095666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c0cf600 msgr2=0x7fd79c1f29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:52.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c0cf600 0x7fd79c1f29c0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd78c00d740 tx=0x7fd78c00dc00 comp rx=0 tx=0).stop 2024-06-29T14:39:52.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 -- 172.21.15.81:0/429095666 shutdown_connections 2024-06-29T14:39:52.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd7800763f0 0x7fd780078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd79c1d8fa0 0x7fd79c1d8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd79c0cf600 0x7fd79c1f29c0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 --2- 172.21.15.81:0/429095666 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd79c0ce3a0 0x7fd79c1df700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:52.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 -- 172.21.15.81:0/429095666 >> 172.21.15.81:0/429095666 conn(0x7fd79c0b35a0 msgr2=0x7fd79c0afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:52.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 -- 172.21.15.81:0/429095666 shutdown_connections 2024-06-29T14:39:52.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:52.997+0000 7fd7a394b640 1 -- 172.21.15.81:0/429095666 wait complete. 2024-06-29T14:39:53.007 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:39:53.007 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:39:53.007 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:39:53.007 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:39:53.007 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:39:53.008 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:39:53.008 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:39:53.008 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:39:53.008 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:39:53.008 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:39:53.008 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:39:53.008 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:39:53.009 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:39:53.009 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:39:53.009 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:39:53.009 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:39:53.017 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:39:53.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:39:53.018 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:39:53.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:39:53.021 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:39:53.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:39:53.023 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:39:53.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:39:53.060 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:39:53.060 DEBUG:teuthology.parallel:result is None 2024-06-29T14:39:53.060 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:39:53.061 DEBUG:teuthology.parallel:result is None 2024-06-29T14:39:53.065 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:39:53.066 DEBUG:teuthology.parallel:result is None 2024-06-29T14:39:53.067 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:39:53.067 DEBUG:teuthology.parallel:result is None 2024-06-29T14:39:53.067 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:39:53.145 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:39:53.146 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T14:39:53.158 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T14:39:53.158 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:39:53.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.190+0000 7febde09e640 1 -- 172.21.15.81:0/3936735969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc4003a50 msgr2=0x7febc4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.190+0000 7febde09e640 1 --2- 172.21.15.81:0/3936735969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc4003a50 0x7febc4003ec0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7febc800c610 tx=0x7febc4005830 comp rx=0 tx=0).stop 2024-06-29T14:39:53.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.190+0000 7febde09e640 1 -- 172.21.15.81:0/3936735969 shutdown_connections 2024-06-29T14:39:53.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.190+0000 7febde09e640 1 --2- 172.21.15.81:0/3936735969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc4003a50 0x7febc4003ec0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.190+0000 7febde09e640 1 -- 172.21.15.81:0/3936735969 >> 172.21.15.81:0/3936735969 conn(0x7febd80b3680 msgr2=0x7febd80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:53.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.190+0000 7febde09e640 1 -- 172.21.15.81:0/3936735969 shutdown_connections 2024-06-29T14:39:53.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.190+0000 7febde09e640 1 -- 172.21.15.81:0/3936735969 wait complete. 2024-06-29T14:39:53.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.192+0000 7febde09e640 1 Processor -- start 2024-06-29T14:39:53.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.192+0000 7febde09e640 1 -- start start 2024-06-29T14:39:53.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febde09e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febc4003a50 0x7febd81f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febde09e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7febd81f6290 0x7febd8156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febde09e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febd80ca6e0 0x7febd80caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febde09e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7febd80cc930 con 0x7febd80ca6e0 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febde09e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7febd80cc7b0 con 0x7febc4003a50 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febde09e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7febd80ccab0 con 0x7febd81f6290 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febd7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febd80ca6e0 0x7febd80caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febd7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febd80ca6e0 0x7febd80caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44642/0 (socket says 172.21.15.81:44642) 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febd7fff640 1 -- 172.21.15.81:0/2021561209 learned_addr learned my addr 172.21.15.81:0/2021561209 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febd7fff640 1 -- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7febd81f6290 msgr2=0x7febd8156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febd7fff640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7febd81f6290 0x7febd8156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febd7fff640 1 -- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febc4003a50 msgr2=0x7febd81f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febd77fe640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febc4003a50 0x7febd81f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febd7fff640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febc4003a50 0x7febd81f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.193+0000 7febd7fff640 1 -- 172.21.15.81:0/2021561209 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd80cb030 con 0x7febd80ca6e0 2024-06-29T14:39:53.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.194+0000 7febd7fff640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febd80ca6e0 0x7febd80caaf0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7febd80d0fd0 tx=0x7febd0006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.194+0000 7febd4ff9640 1 -- 172.21.15.81:0/2021561209 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd0011900 con 0x7febd80ca6e0 2024-06-29T14:39:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.194+0000 7febd4ff9640 1 -- 172.21.15.81:0/2021561209 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febd0018070 con 0x7febd80ca6e0 2024-06-29T14:39:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.194+0000 7febde09e640 1 -- 172.21.15.81:0/2021561209 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd81f2ca0 con 0x7febd80ca6e0 2024-06-29T14:39:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.194+0000 7febd4ff9640 1 -- 172.21.15.81:0/2021561209 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd0011d40 con 0x7febd80ca6e0 2024-06-29T14:39:53.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.194+0000 7febde09e640 1 -- 172.21.15.81:0/2021561209 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd81f6ff0 con 0x7febd80ca6e0 2024-06-29T14:39:53.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.195+0000 7febde09e640 1 -- 172.21.15.81:0/2021561209 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd81f6a80 con 0x7febd80ca6e0 2024-06-29T14:39:53.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.196+0000 7febd4ff9640 1 -- 172.21.15.81:0/2021561209 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7febd001d460 con 0x7febd80ca6e0 2024-06-29T14:39:53.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.196+0000 7febd4ff9640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7febb80763f0 0x7febb8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.196+0000 7febd4ff9640 1 -- 172.21.15.81:0/2021561209 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5789+0+0 (secure 0 0 0) 0x7febd0098300 con 0x7febd80ca6e0 2024-06-29T14:39:53.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.196+0000 7febd77fe640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7febb80763f0 0x7febb8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.197+0000 7febd77fe640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7febb80763f0 0x7febb8078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7febc8033000 tx=0x7febc8001bc0 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:53.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.201+0000 7febd4ff9640 1 -- 172.21.15.81:0/2021561209 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7febd80cb030 con 0x7febd80ca6e0 2024-06-29T14:39:53.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.392+0000 7febde09e640 1 -- 172.21.15.81:0/2021561209 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7febd81f71c0 con 0x7febd80ca6e0 2024-06-29T14:39:53.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.802+0000 7febd4ff9640 1 -- 172.21.15.81:0/2021561209 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v113) ==== 97+0+0 (secure 0 0 0) 0x7febd0061960 con 0x7febd80ca6e0 2024-06-29T14:39:53.800 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T14:39:53.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 -- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7febb80763f0 msgr2=0x7febb8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7febb80763f0 0x7febb8078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7febc8033000 tx=0x7febc8001bc0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 -- 172.21.15.81:0/2021561209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febd80ca6e0 msgr2=0x7febd80caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febd80ca6e0 0x7febd80caaf0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7febd80d0fd0 tx=0x7febd0006430 comp rx=0 tx=0).stop 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 -- 172.21.15.81:0/2021561209 shutdown_connections 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7febb80763f0 0x7febb8078900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febd80ca6e0 0x7febd80caaf0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7febd81f6290 0x7febd8156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 --2- 172.21.15.81:0/2021561209 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febc4003a50 0x7febd81f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 -- 172.21.15.81:0/2021561209 >> 172.21.15.81:0/2021561209 conn(0x7febd80b3680 msgr2=0x7febd80c7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 -- 172.21.15.81:0/2021561209 shutdown_connections 2024-06-29T14:39:53.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.804+0000 7febde09e640 1 -- 172.21.15.81:0/2021561209 wait complete. 2024-06-29T14:39:53.815 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:39:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.930+0000 7f3b934ac640 1 -- 172.21.15.81:0/347948775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b8c06f660 msgr2=0x7f3b8c06fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.930+0000 7f3b934ac640 1 --2- 172.21.15.81:0/347948775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b8c06f660 0x7f3b8c06fa70 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400a940 tx=0x7f3b8402b7b0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.930+0000 7f3b934ac640 1 -- 172.21.15.81:0/347948775 shutdown_connections 2024-06-29T14:39:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.930+0000 7f3b934ac640 1 --2- 172.21.15.81:0/347948775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b8c06f660 0x7f3b8c06fa70 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.930+0000 7f3b934ac640 1 --2- 172.21.15.81:0/347948775 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b8c15b5b0 0x7f3b8c0b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.930+0000 7f3b934ac640 1 -- 172.21.15.81:0/347948775 >> 172.21.15.81:0/347948775 conn(0x7f3b8c06ccc0 msgr2=0x7f3b8c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.930+0000 7f3b934ac640 1 -- 172.21.15.81:0/347948775 shutdown_connections 2024-06-29T14:39:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.930+0000 7f3b934ac640 1 -- 172.21.15.81:0/347948775 wait complete. 2024-06-29T14:39:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.932+0000 7f3b934ac640 1 Processor -- start 2024-06-29T14:39:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.932+0000 7f3b934ac640 1 -- start start 2024-06-29T14:39:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.933+0000 7f3b934ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b8c06f660 0x7f3b8c158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.933+0000 7f3b934ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b8c0b35b0 0x7f3b8c158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.933+0000 7f3b934ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b8c15b5b0 0x7f3b8c0b0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.933+0000 7f3b934ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c06f160 con 0x7f3b8c0b35b0 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.933+0000 7f3b934ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c06efe0 con 0x7f3b8c15b5b0 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.933+0000 7f3b934ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3b8c06f2e0 con 0x7f3b8c06f660 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b90a47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b8c0b35b0 0x7f3b8c158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b90a47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b8c0b35b0 0x7f3b8c158850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44658/0 (socket says 172.21.15.81:44658) 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b90a47640 1 -- 172.21.15.81:0/994747739 learned_addr learned my addr 172.21.15.81:0/994747739 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b91248640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b8c06f660 0x7f3b8c158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b91a49640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b8c15b5b0 0x7f3b8c0b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b90a47640 1 -- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b8c06f660 msgr2=0x7f3b8c158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b90a47640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b8c06f660 0x7f3b8c158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b90a47640 1 -- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b8c15b5b0 msgr2=0x7f3b8c0b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b90a47640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b8c15b5b0 0x7f3b8c0b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b90a47640 1 -- 172.21.15.81:0/994747739 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8c159060 con 0x7f3b8c0b35b0 2024-06-29T14:39:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b91a49640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b8c15b5b0 0x7f3b8c0b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b90a47640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b8c0b35b0 0x7f3b8c158850 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000c9a0 tx=0x7f3b8000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b89ffb640 1 -- 172.21.15.81:0/994747739 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b80007bb0 con 0x7f3b8c0b35b0 2024-06-29T14:39:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b89ffb640 1 -- 172.21.15.81:0/994747739 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b80007e40 con 0x7f3b8c0b35b0 2024-06-29T14:39:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b89ffb640 1 -- 172.21.15.81:0/994747739 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b80005280 con 0x7f3b8c0b35b0 2024-06-29T14:39:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.934+0000 7f3b934ac640 1 -- 172.21.15.81:0/994747739 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8c159340 con 0x7f3b8c0b35b0 2024-06-29T14:39:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.935+0000 7f3b934ac640 1 -- 172.21.15.81:0/994747739 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b8c0add50 con 0x7f3b8c0b35b0 2024-06-29T14:39:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.935+0000 7f3b934ac640 1 -- 172.21.15.81:0/994747739 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b8c158dd0 con 0x7f3b8c0b35b0 2024-06-29T14:39:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.936+0000 7f3b89ffb640 1 -- 172.21.15.81:0/994747739 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f3b80005420 con 0x7f3b8c0b35b0 2024-06-29T14:39:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.937+0000 7f3b89ffb640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f3b78076420 0x7f3b78078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.937+0000 7f3b89ffb640 1 -- 172.21.15.81:0/994747739 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4929+0+0 (secure 0 0 0) 0x7f3b80097a70 con 0x7f3b8c0b35b0 2024-06-29T14:39:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.937+0000 7f3b91248640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f3b78076420 0x7f3b78078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.937+0000 7f3b91248640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f3b78076420 0x7f3b78078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0012e0 tx=0x7f3b7c009040 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:53.941+0000 7f3b89ffb640 1 -- 172.21.15.81:0/994747739 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3b80014030 con 0x7f3b8c0b35b0 2024-06-29T14:39:54.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.192+0000 7f3b934ac640 1 -- 172.21.15.81:0/994747739 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f3b8c06fa70 con 0x7f3b8c0b35b0 2024-06-29T14:39:54.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.193+0000 7f3b89ffb640 1 -- 172.21.15.81:0/994747739 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v34) ==== 80+0+5304 (secure 0 0 0) 0x7f3b80060e90 con 0x7f3b8c0b35b0 2024-06-29T14:39:54.191 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:54.191 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:39:54.191 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:39:54.193 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.194 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.196 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.197 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.198 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:39:54.199 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.200 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T14:39:54.201 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:39:54.202 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.203 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T14:39:54.204 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:39:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:39:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:39:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:39:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:54.205 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T14:39:54.205 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:39:54.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 -- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f3b78076420 msgr2=0x7f3b78078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f3b78076420 0x7f3b78078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0012e0 tx=0x7f3b7c009040 comp rx=0 tx=0).stop 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 -- 172.21.15.81:0/994747739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b8c0b35b0 msgr2=0x7f3b8c158850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b8c0b35b0 0x7f3b8c158850 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000c9a0 tx=0x7f3b8000ce60 comp rx=0 tx=0).stop 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 -- 172.21.15.81:0/994747739 shutdown_connections 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f3b78076420 0x7f3b78078930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b8c15b5b0 0x7f3b8c0b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b8c0b35b0 0x7f3b8c158850 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 --2- 172.21.15.81:0/994747739 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b8c06f660 0x7f3b8c158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 -- 172.21.15.81:0/994747739 >> 172.21.15.81:0/994747739 conn(0x7f3b8c06ccc0 msgr2=0x7f3b8c072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 -- 172.21.15.81:0/994747739 shutdown_connections 2024-06-29T14:39:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.195+0000 7f3b934ac640 1 -- 172.21.15.81:0/994747739 wait complete. 2024-06-29T14:39:54.208 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:54.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.322+0000 7faee6610640 1 -- 172.21.15.81:0/1671912964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee00cccf0 msgr2=0x7faee00d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.322+0000 7faee6610640 1 --2- 172.21.15.81:0/1671912964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee00cccf0 0x7faee00d0230 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7faee0073550 tx=0x7faed402a4a0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.322+0000 7faee6610640 1 -- 172.21.15.81:0/1671912964 shutdown_connections 2024-06-29T14:39:54.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.322+0000 7faee6610640 1 --2- 172.21.15.81:0/1671912964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee00cccf0 0x7faee00d0230 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.322+0000 7faee6610640 1 --2- 172.21.15.81:0/1671912964 >> v2:172.21.15.167:3300/0 conn(0x7faee00b3200 0x7faee00cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.322+0000 7faee6610640 1 -- 172.21.15.81:0/1671912964 >> 172.21.15.81:0/1671912964 conn(0x7faee00b3680 msgr2=0x7faee00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:54.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.322+0000 7faee6610640 1 -- 172.21.15.81:0/1671912964 shutdown_connections 2024-06-29T14:39:54.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.323+0000 7faee6610640 1 -- 172.21.15.81:0/1671912964 wait complete. 2024-06-29T14:39:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.324+0000 7faee6610640 1 Processor -- start 2024-06-29T14:39:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.324+0000 7faee6610640 1 -- start start 2024-06-29T14:39:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faee6610640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faee00b3200 0x7faee00af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faee6610640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee00cccf0 0x7faee01f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faee6610640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faee00d0770 0x7faee01e3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faee6610640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faee00cedb0 con 0x7faee00cccf0 2024-06-29T14:39:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faee6610640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faee00cec30 con 0x7faee00d0770 2024-06-29T14:39:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faee6610640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faee00cef30 con 0x7faee00b3200 2024-06-29T14:39:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee00cccf0 0x7faee01f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee00cccf0 0x7faee01f24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44674/0 (socket says 172.21.15.81:44674) 2024-06-29T14:39:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedf7fe640 1 -- 172.21.15.81:0/3120586368 learned_addr learned my addr 172.21.15.81:0/3120586368 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedffff640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faee00b3200 0x7faee00af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faee4bad640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faee00d0770 0x7faee01e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedf7fe640 1 -- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faee00b3200 msgr2=0x7faee00af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedf7fe640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faee00b3200 0x7faee00af270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedf7fe640 1 -- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faee00d0770 msgr2=0x7faee01e3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedf7fe640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faee00d0770 0x7faee01e3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedf7fe640 1 -- 172.21.15.81:0/3120586368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee009de30 con 0x7faee00cccf0 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedf7fe640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee00cccf0 0x7faee01f24f0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7faed402aa90 tx=0x7faed4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.326+0000 7faedcff9640 1 -- 172.21.15.81:0/3120586368 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed4031d60 con 0x7faee00cccf0 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.327+0000 7faedcff9640 1 -- 172.21.15.81:0/3120586368 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faed4033e10 con 0x7faee00cccf0 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.327+0000 7faedcff9640 1 -- 172.21.15.81:0/3120586368 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed4005260 con 0x7faee00cccf0 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.327+0000 7faee6610640 1 -- 172.21.15.81:0/3120586368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee009e110 con 0x7faee00cccf0 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.327+0000 7faee6610640 1 -- 172.21.15.81:0/3120586368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee01f2a30 con 0x7faee00cccf0 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.328+0000 7faee6610640 1 -- 172.21.15.81:0/3120586368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faee00aedd0 con 0x7faee00cccf0 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.329+0000 7faedcff9640 1 -- 172.21.15.81:0/3120586368 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7faed4005400 con 0x7faee00cccf0 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.329+0000 7faedcff9640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7faecc076420 0x7faecc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.329+0000 7faedcff9640 1 -- 172.21.15.81:0/3120586368 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4929+0+0 (secure 0 0 0) 0x7faed40b8940 con 0x7faee00cccf0 2024-06-29T14:39:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.329+0000 7faedffff640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7faecc076420 0x7faecc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.330+0000 7faedffff640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7faecc076420 0x7faecc078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faed00058c0 tx=0x7faed0005d80 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.333+0000 7faedcff9640 1 -- 172.21.15.81:0/3120586368 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faed403f030 con 0x7faee00cccf0 2024-06-29T14:39:54.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.515+0000 7faee6610640 1 -- 172.21.15.81:0/3120586368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faee00cc7b0 con 0x7faee00cccf0 2024-06-29T14:39:54.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.516+0000 7faedcff9640 1 -- 172.21.15.81:0/3120586368 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) ==== 74+0+13003 (secure 0 0 0) 0x7faed4031770 con 0x7faee00cccf0 2024-06-29T14:39:54.514 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:54.514 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":115,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:39:53.817617+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 -- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7faecc076420 msgr2=0x7faecc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7faecc076420 0x7faecc078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faed00058c0 tx=0x7faed0005d80 comp rx=0 tx=0).stop 2024-06-29T14:39:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 -- 172.21.15.81:0/3120586368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee00cccf0 msgr2=0x7faee01f24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee00cccf0 0x7faee01f24f0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7faed402aa90 tx=0x7faed4033c20 comp rx=0 tx=0).stop 2024-06-29T14:39:54.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 -- 172.21.15.81:0/3120586368 shutdown_connections 2024-06-29T14:39:54.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7faecc076420 0x7faecc078930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faee00d0770 0x7faee01e3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee00cccf0 0x7faee01f24f0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 --2- 172.21.15.81:0/3120586368 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faee00b3200 0x7faee00af270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 -- 172.21.15.81:0/3120586368 >> 172.21.15.81:0/3120586368 conn(0x7faee00b3680 msgr2=0x7faee00b5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:54.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 -- 172.21.15.81:0/3120586368 shutdown_connections 2024-06-29T14:39:54.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.518+0000 7faee6610640 1 -- 172.21.15.81:0/3120586368 wait complete. 2024-06-29T14:39:54.527 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:39:54.528 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:39:54.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.645+0000 7f4d14a9e640 1 -- 172.21.15.81:0/3508919363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d10073030 msgr2=0x7f4d10073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.645+0000 7f4d14a9e640 1 --2- 172.21.15.81:0/3508919363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d10073030 0x7f4d10073440 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f4d10071d20 tx=0x7f4d0402a560 comp rx=0 tx=0).stop 2024-06-29T14:39:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.645+0000 7f4d14a9e640 1 -- 172.21.15.81:0/3508919363 shutdown_connections 2024-06-29T14:39:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.645+0000 7f4d14a9e640 1 --2- 172.21.15.81:0/3508919363 >> v2:172.21.15.167:3300/0 conn(0x7f4d1015e960 0x7f4d1015ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.645+0000 7f4d14a9e640 1 --2- 172.21.15.81:0/3508919363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d10073030 0x7f4d10073440 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.645+0000 7f4d14a9e640 1 -- 172.21.15.81:0/3508919363 >> 172.21.15.81:0/3508919363 conn(0x7f4d1006ccc0 msgr2=0x7f4d100b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.645+0000 7f4d14a9e640 1 -- 172.21.15.81:0/3508919363 shutdown_connections 2024-06-29T14:39:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.645+0000 7f4d14a9e640 1 -- 172.21.15.81:0/3508919363 wait complete. 2024-06-29T14:39:54.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.647+0000 7f4d14a9e640 1 Processor -- start 2024-06-29T14:39:54.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.647+0000 7f4d14a9e640 1 -- start start 2024-06-29T14:39:54.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.648+0000 7f4d14a9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d10073030 0x7f4d10155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.648+0000 7f4d14a9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d101412f0 0x7f4d101f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.648+0000 7f4d14a9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d1015e960 0x7f4d101537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.648+0000 7f4d14a9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4d100b3210 con 0x7f4d101412f0 2024-06-29T14:39:54.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.648+0000 7f4d14a9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4d10141740 con 0x7f4d10073030 2024-06-29T14:39:54.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.648+0000 7f4d14a9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4d100b3390 con 0x7f4d1015e960 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.648+0000 7f4d0dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d101412f0 0x7f4d101f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.648+0000 7f4d0dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d101412f0 0x7f4d101f29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44688/0 (socket says 172.21.15.81:44688) 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.648+0000 7f4d0dd9b640 1 -- 172.21.15.81:0/1126027059 learned_addr learned my addr 172.21.15.81:0/1126027059 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d0e59c640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d10073030 0x7f4d10155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d0ed9d640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d1015e960 0x7f4d101537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d0dd9b640 1 -- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d1015e960 msgr2=0x7f4d101537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d0dd9b640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d1015e960 0x7f4d101537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d0dd9b640 1 -- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d10073030 msgr2=0x7f4d10155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d0dd9b640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d10073030 0x7f4d10155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d0dd9b640 1 -- 172.21.15.81:0/1126027059 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d10152750 con 0x7f4d101412f0 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d0e59c640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d10073030 0x7f4d10155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d0dd9b640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d101412f0 0x7f4d101f29b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f4d04034040 tx=0x7f4d040039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4cfeffd640 1 -- 172.21.15.81:0/1126027059 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0403e030 con 0x7f4d101412f0 2024-06-29T14:39:54.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4cfeffd640 1 -- 172.21.15.81:0/1126027059 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d040047a0 con 0x7f4d101412f0 2024-06-29T14:39:54.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d14a9e640 1 -- 172.21.15.81:0/1126027059 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d10152a30 con 0x7f4d101412f0 2024-06-29T14:39:54.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4cfeffd640 1 -- 172.21.15.81:0/1126027059 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d04035040 con 0x7f4d101412f0 2024-06-29T14:39:54.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.649+0000 7f4d14a9e640 1 -- 172.21.15.81:0/1126027059 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d10152ff0 con 0x7f4d101412f0 2024-06-29T14:39:54.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.650+0000 7f4d14a9e640 1 -- 172.21.15.81:0/1126027059 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d100b38f0 con 0x7f4d101412f0 2024-06-29T14:39:54.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.651+0000 7f4cfeffd640 1 -- 172.21.15.81:0/1126027059 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f4d04041030 con 0x7f4d101412f0 2024-06-29T14:39:54.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.651+0000 7f4cfeffd640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4cf00763f0 0x7f4cf0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.651+0000 7f4cfeffd640 1 -- 172.21.15.81:0/1126027059 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4929+0+0 (secure 0 0 0) 0x7f4d040b7e60 con 0x7f4d101412f0 2024-06-29T14:39:54.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.652+0000 7f4d0e59c640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4cf00763f0 0x7f4cf0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.652+0000 7f4d0e59c640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4cf00763f0 0x7f4cf0078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000a830 tx=0x7f4d00008040 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:54.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.655+0000 7f4cfeffd640 1 -- 172.21.15.81:0/1126027059 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4d10073980 con 0x7f4d101412f0 2024-06-29T14:39:54.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.839+0000 7f4d14a9e640 1 -- 172.21.15.81:0/1126027059 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4d1015c8b0 con 0x7f4d101412f0 2024-06-29T14:39:54.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.839+0000 7f4cfeffd640 1 -- 172.21.15.81:0/1126027059 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) ==== 93+0+10 (secure 0 0 0) 0x7f4d04081930 con 0x7f4d101412f0 2024-06-29T14:39:54.838 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:39:54.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 -- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4cf00763f0 msgr2=0x7f4cf0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4cf00763f0 0x7f4cf0078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000a830 tx=0x7f4d00008040 comp rx=0 tx=0).stop 2024-06-29T14:39:54.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 -- 172.21.15.81:0/1126027059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d101412f0 msgr2=0x7f4d101f29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d101412f0 0x7f4d101f29b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f4d04034040 tx=0x7f4d040039e0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 -- 172.21.15.81:0/1126027059 shutdown_connections 2024-06-29T14:39:54.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4cf00763f0 0x7f4cf0078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d1015e960 0x7f4d101537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d101412f0 0x7f4d101f29b0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 --2- 172.21.15.81:0/1126027059 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d10073030 0x7f4d10155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 -- 172.21.15.81:0/1126027059 >> 172.21.15.81:0/1126027059 conn(0x7f4d1006ccc0 msgr2=0x7f4d1006de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:54.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 -- 172.21.15.81:0/1126027059 shutdown_connections 2024-06-29T14:39:54.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.842+0000 7f4d14a9e640 1 -- 172.21.15.81:0/1126027059 wait complete. 2024-06-29T14:39:54.851 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:39:54.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.966+0000 7fd4a5318640 1 -- 172.21.15.81:0/1664280521 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4a00d0230 msgr2=0x7fd4a00d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.966+0000 7fd4a5318640 1 --2- 172.21.15.81:0/1664280521 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4a00d0230 0x7fd4a00d0640 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd494009830 tx=0x7fd49402a510 comp rx=0 tx=0).stop 2024-06-29T14:39:54.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.966+0000 7fd4a5318640 1 -- 172.21.15.81:0/1664280521 shutdown_connections 2024-06-29T14:39:54.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.966+0000 7fd4a5318640 1 --2- 172.21.15.81:0/1664280521 >> v2:172.21.15.167:3300/0 conn(0x7fd4a00b4f40 0x7fd4a00b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.966+0000 7fd4a5318640 1 --2- 172.21.15.81:0/1664280521 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4a00d0230 0x7fd4a00d0640 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.966+0000 7fd4a5318640 1 -- 172.21.15.81:0/1664280521 >> 172.21.15.81:0/1664280521 conn(0x7fd4a00b3680 msgr2=0x7fd4a00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:54.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.966+0000 7fd4a5318640 1 -- 172.21.15.81:0/1664280521 shutdown_connections 2024-06-29T14:39:54.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.966+0000 7fd4a5318640 1 -- 172.21.15.81:0/1664280521 wait complete. 2024-06-29T14:39:54.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.968+0000 7fd4a5318640 1 Processor -- start 2024-06-29T14:39:54.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.968+0000 7fd4a5318640 1 -- start start 2024-06-29T14:39:54.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd4a5318640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4a00b3200 0x7fd4a0156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd4a5318640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4a00b4f40 0x7fd4a01e65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd4a5318640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4a00d0230 0x7fd4a0159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd4a5318640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a00b5c20 con 0x7fd4a00d0230 2024-06-29T14:39:54.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd4a5318640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a00b5aa0 con 0x7fd4a00b3200 2024-06-29T14:39:54.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd4a5318640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd4a00b5da0 con 0x7fd4a00b4f40 2024-06-29T14:39:54.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4a00d0230 0x7fd4a0159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4a00d0230 0x7fd4a0159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44700/0 (socket says 172.21.15.81:44700) 2024-06-29T14:39:54.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49f7fe640 1 -- 172.21.15.81:0/1534695441 learned_addr learned my addr 172.21.15.81:0/1534695441 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:54.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49effd640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4a00b3200 0x7fd4a0156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49e7fc640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4a00b4f40 0x7fd4a01e65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49f7fe640 1 -- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4a00b4f40 msgr2=0x7fd4a01e65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49f7fe640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4a00b4f40 0x7fd4a01e65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49f7fe640 1 -- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4a00b3200 msgr2=0x7fd4a0156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:54.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49f7fe640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4a00b3200 0x7fd4a0156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:54.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49f7fe640 1 -- 172.21.15.81:0/1534695441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a00cb040 con 0x7fd4a00d0230 2024-06-29T14:39:54.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd49f7fe640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4a00d0230 0x7fd4a0159460 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fd49800e8d0 tx=0x7fd49800ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:54.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd4877fe640 1 -- 172.21.15.81:0/1534695441 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49800ccc0 con 0x7fd4a00d0230 2024-06-29T14:39:54.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd4a5318640 1 -- 172.21.15.81:0/1534695441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a00cb320 con 0x7fd4a00d0230 2024-06-29T14:39:54.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd4877fe640 1 -- 172.21.15.81:0/1534695441 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd49800ce60 con 0x7fd4a00d0230 2024-06-29T14:39:54.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.970+0000 7fd4877fe640 1 -- 172.21.15.81:0/1534695441 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498004280 con 0x7fd4a00d0230 2024-06-29T14:39:54.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.971+0000 7fd4a5318640 1 -- 172.21.15.81:0/1534695441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a00cb8e0 con 0x7fd4a00d0230 2024-06-29T14:39:54.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.972+0000 7fd4a5318640 1 -- 172.21.15.81:0/1534695441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4a00b5350 con 0x7fd4a00d0230 2024-06-29T14:39:54.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.972+0000 7fd4877fe640 1 -- 172.21.15.81:0/1534695441 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fd4980035f0 con 0x7fd4a00d0230 2024-06-29T14:39:54.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.973+0000 7fd4877fe640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd48c076420 0x7fd48c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:54.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.973+0000 7fd4877fe640 1 -- 172.21.15.81:0/1534695441 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4929+0+0 (secure 0 0 0) 0x7fd4980980e0 con 0x7fd4a00d0230 2024-06-29T14:39:54.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.973+0000 7fd49effd640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd48c076420 0x7fd48c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:54.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.973+0000 7fd49effd640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd48c076420 0x7fd48c078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd490001770 tx=0x7fd490009040 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:54.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:54.977+0000 7fd4877fe640 1 -- 172.21.15.81:0/1534695441 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd498061a10 con 0x7fd4a00d0230 2024-06-29T14:39:55.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.161+0000 7fd4a5318640 1 -- 172.21.15.81:0/1534695441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd4a00c62e0 con 0x7fd4a00d0230 2024-06-29T14:39:55.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.162+0000 7fd4877fe640 1 -- 172.21.15.81:0/1534695441 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v115) ==== 84+0+65 (secure 0 0 0) 0x7fd498061310 con 0x7fd4a00d0230 2024-06-29T14:39:55.161 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:55.161 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:39:55.161 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:39:55.161 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T14:39:55.161 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T14:39:55.162 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:39:55.162 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:39:55.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.163+0000 7fd4a5318640 1 -- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd48c076420 msgr2=0x7fd48c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.163+0000 7fd4a5318640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd48c076420 0x7fd48c078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd490001770 tx=0x7fd490009040 comp rx=0 tx=0).stop 2024-06-29T14:39:55.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.163+0000 7fd4a5318640 1 -- 172.21.15.81:0/1534695441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4a00d0230 msgr2=0x7fd4a0159460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.163+0000 7fd4a5318640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4a00d0230 0x7fd4a0159460 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fd49800e8d0 tx=0x7fd49800ed90 comp rx=0 tx=0).stop 2024-06-29T14:39:55.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.164+0000 7fd4a5318640 1 -- 172.21.15.81:0/1534695441 shutdown_connections 2024-06-29T14:39:55.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.164+0000 7fd4a5318640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd48c076420 0x7fd48c078930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.164+0000 7fd4a5318640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4a00d0230 0x7fd4a0159460 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.164+0000 7fd4a5318640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4a00b4f40 0x7fd4a01e65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.164+0000 7fd4a5318640 1 --2- 172.21.15.81:0/1534695441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4a00b3200 0x7fd4a0156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.164+0000 7fd4a5318640 1 -- 172.21.15.81:0/1534695441 >> 172.21.15.81:0/1534695441 conn(0x7fd4a00b3680 msgr2=0x7fd4a00d1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:55.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.164+0000 7fd4a5318640 1 -- 172.21.15.81:0/1534695441 shutdown_connections 2024-06-29T14:39:55.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.164+0000 7fd4a5318640 1 -- 172.21.15.81:0/1534695441 wait complete. 2024-06-29T14:39:55.174 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:39:55.174 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:39:55.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.293+0000 7fc865a3f640 1 -- 172.21.15.81:0/546375429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8600b2ea0 msgr2=0x7fc8600cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.293+0000 7fc865a3f640 1 --2- 172.21.15.81:0/546375429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8600b2ea0 0x7fc8600cf510 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fc86008c7f0 tx=0x7fc85402a4d0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.294+0000 7fc865a3f640 1 -- 172.21.15.81:0/546375429 shutdown_connections 2024-06-29T14:39:55.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.294+0000 7fc865a3f640 1 --2- 172.21.15.81:0/546375429 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8600d03b0 0x7fc8600d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.294+0000 7fc865a3f640 1 --2- 172.21.15.81:0/546375429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8600b2ea0 0x7fc8600cf510 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.294+0000 7fc865a3f640 1 --2- 172.21.15.81:0/546375429 >> v2:172.21.15.167:3300/0 conn(0x7fc8600d2440 0x7fc8600d4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.294+0000 7fc865a3f640 1 -- 172.21.15.81:0/546375429 >> 172.21.15.81:0/546375429 conn(0x7fc8600d3c50 msgr2=0x7fc8600d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.294+0000 7fc865a3f640 1 -- 172.21.15.81:0/546375429 shutdown_connections 2024-06-29T14:39:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.294+0000 7fc865a3f640 1 -- 172.21.15.81:0/546375429 wait complete. 2024-06-29T14:39:55.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.295+0000 7fc865a3f640 1 Processor -- start 2024-06-29T14:39:55.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.296+0000 7fc865a3f640 1 -- start start 2024-06-29T14:39:55.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc865a3f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8600b2ea0 0x7fc860161120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc865a3f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8600d03b0 0x7fc8600b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc865a3f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8600d2440 0x7fc8600b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc865a3f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600cfbd0 con 0x7fc8600b2ea0 2024-06-29T14:39:55.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc865a3f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600cfa50 con 0x7fc8600d2440 2024-06-29T14:39:55.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc865a3f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc86006cb10 con 0x7fc8600d03b0 2024-06-29T14:39:55.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8600b2ea0 0x7fc860161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8600b2ea0 0x7fc860161120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44724/0 (socket says 172.21.15.81:44724) 2024-06-29T14:39:55.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85effd640 1 -- 172.21.15.81:0/1197831713 learned_addr learned my addr 172.21.15.81:0/1197831713 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:55.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85e7fc640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8600d03b0 0x7fc8600b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85f7fe640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8600d2440 0x7fc8600b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85effd640 1 -- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8600d03b0 msgr2=0x7fc8600b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85effd640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8600d03b0 0x7fc8600b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85effd640 1 -- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8600d2440 msgr2=0x7fc8600b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85effd640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8600d2440 0x7fc8600b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85effd640 1 -- 172.21.15.81:0/1197831713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8600c9d00 con 0x7fc8600b2ea0 2024-06-29T14:39:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85e7fc640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8600d03b0 0x7fc8600b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:55.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc85effd640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8600b2ea0 0x7fc860161120 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc85000c9a0 tx=0x7fc85000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:55.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.297+0000 7fc847fff640 1 -- 172.21.15.81:0/1197831713 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850007bb0 con 0x7fc8600b2ea0 2024-06-29T14:39:55.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.298+0000 7fc847fff640 1 -- 172.21.15.81:0/1197831713 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc850007e40 con 0x7fc8600b2ea0 2024-06-29T14:39:55.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.298+0000 7fc865a3f640 1 -- 172.21.15.81:0/1197831713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8600caa80 con 0x7fc8600b2ea0 2024-06-29T14:39:55.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.298+0000 7fc847fff640 1 -- 172.21.15.81:0/1197831713 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85000f690 con 0x7fc8600b2ea0 2024-06-29T14:39:55.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.298+0000 7fc865a3f640 1 -- 172.21.15.81:0/1197831713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8600c9fa0 con 0x7fc8600b2ea0 2024-06-29T14:39:55.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.299+0000 7fc865a3f640 1 -- 172.21.15.81:0/1197831713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8600c7700 con 0x7fc8600b2ea0 2024-06-29T14:39:55.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.299+0000 7fc847fff640 1 -- 172.21.15.81:0/1197831713 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fc850018400 con 0x7fc8600b2ea0 2024-06-29T14:39:55.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.300+0000 7fc847fff640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc84c076420 0x7fc84c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.300+0000 7fc847fff640 1 -- 172.21.15.81:0/1197831713 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4929+0+0 (secure 0 0 0) 0x7fc850097e30 con 0x7fc8600b2ea0 2024-06-29T14:39:55.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.300+0000 7fc85e7fc640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc84c076420 0x7fc84c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.301+0000 7fc85e7fc640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc84c076420 0x7fc84c078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc854033000 tx=0x7fc854004730 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:55.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.304+0000 7fc847fff640 1 -- 172.21.15.81:0/1197831713 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc850014030 con 0x7fc8600b2ea0 2024-06-29T14:39:55.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.485+0000 7fc865a3f640 1 -- 172.21.15.81:0/1197831713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc8600d14a0 con 0x7fc8600b2ea0 2024-06-29T14:39:55.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.806+0000 7fc847fff640 1 -- 172.21.15.81:0/1197831713 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v116) ==== 138+0+0 (secure 0 0 0) 0x7fc850061870 con 0x7fc8600b2ea0 2024-06-29T14:39:55.805 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T14:39:55.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 -- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc84c076420 msgr2=0x7fc84c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc84c076420 0x7fc84c078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc854033000 tx=0x7fc854004730 comp rx=0 tx=0).stop 2024-06-29T14:39:55.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 -- 172.21.15.81:0/1197831713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8600b2ea0 msgr2=0x7fc860161120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8600b2ea0 0x7fc860161120 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fc85000c9a0 tx=0x7fc85000ce60 comp rx=0 tx=0).stop 2024-06-29T14:39:55.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 -- 172.21.15.81:0/1197831713 shutdown_connections 2024-06-29T14:39:55.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc84c076420 0x7fc84c078930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8600d2440 0x7fc8600b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8600d03b0 0x7fc8600b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 --2- 172.21.15.81:0/1197831713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8600b2ea0 0x7fc860161120 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 -- 172.21.15.81:0/1197831713 >> 172.21.15.81:0/1197831713 conn(0x7fc8600d3c50 msgr2=0x7fc8600b7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:55.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 -- 172.21.15.81:0/1197831713 shutdown_connections 2024-06-29T14:39:55.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.809+0000 7fc865a3f640 1 -- 172.21.15.81:0/1197831713 wait complete. 2024-06-29T14:39:55.823 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:39:55.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.946+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2585363184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e80b35b0 msgr2=0x7fc2e815b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.946+0000 7fc2efd2a640 1 --2- 172.21.15.81:0/2585363184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e80b35b0 0x7fc2e815b5b0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009830 tx=0x7fc2e402a530 comp rx=0 tx=0).stop 2024-06-29T14:39:55.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.946+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2585363184 shutdown_connections 2024-06-29T14:39:55.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.946+0000 7fc2efd2a640 1 --2- 172.21.15.81:0/2585363184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e80b35b0 0x7fc2e815b5b0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.946+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2585363184 >> 172.21.15.81:0/2585363184 conn(0x7fc2e806ccc0 msgr2=0x7fc2e80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:55.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.946+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2585363184 shutdown_connections 2024-06-29T14:39:55.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.946+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2585363184 wait complete. 2024-06-29T14:39:55.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.948+0000 7fc2efd2a640 1 Processor -- start 2024-06-29T14:39:55.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.948+0000 7fc2efd2a640 1 -- start start 2024-06-29T14:39:55.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2efd2a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e806f660 0x7fc2e81524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2efd2a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2e80b35b0 0x7fc2e81461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2efd2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e80b4430 0x7fc2e814acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2efd2a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e8072550 con 0x7fc2e80b4430 2024-06-29T14:39:55.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2efd2a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e80723d0 con 0x7fc2e80b35b0 2024-06-29T14:39:55.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2efd2a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc2e80726d0 con 0x7fc2e806f660 2024-06-29T14:39:55.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2ee2c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e80b4430 0x7fc2e814acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2ee2c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e80b4430 0x7fc2e814acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44740/0 (socket says 172.21.15.81:44740) 2024-06-29T14:39:55.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2ee2c7640 1 -- 172.21.15.81:0/2274233427 learned_addr learned my addr 172.21.15.81:0/2274233427 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:55.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2edac6640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e806f660 0x7fc2e81524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2ed2c5640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2e80b35b0 0x7fc2e81461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2ee2c7640 1 -- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e806f660 msgr2=0x7fc2e81524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2ee2c7640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e806f660 0x7fc2e81524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2ee2c7640 1 -- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2e80b35b0 msgr2=0x7fc2e81461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2ee2c7640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2e80b35b0 0x7fc2e81461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2ee2c7640 1 -- 172.21.15.81:0/2274233427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e81535e0 con 0x7fc2e80b4430 2024-06-29T14:39:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.949+0000 7fc2edac6640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e806f660 0x7fc2e81524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.950+0000 7fc2ee2c7640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e80b4430 0x7fc2e814acc0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000c760 tx=0x7fc2e000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.950+0000 7fc2da7fc640 1 -- 172.21.15.81:0/2274233427 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0003b70 con 0x7fc2e80b4430 2024-06-29T14:39:55.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.950+0000 7fc2da7fc640 1 -- 172.21.15.81:0/2274233427 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2e0003d10 con 0x7fc2e80b4430 2024-06-29T14:39:55.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.950+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2274233427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e81538c0 con 0x7fc2e80b4430 2024-06-29T14:39:55.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.950+0000 7fc2da7fc640 1 -- 172.21.15.81:0/2274233427 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00052d0 con 0x7fc2e80b4430 2024-06-29T14:39:55.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.950+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2274233427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e8151fb0 con 0x7fc2e80b4430 2024-06-29T14:39:55.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.951+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2274233427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2e8153350 con 0x7fc2e80b4430 2024-06-29T14:39:55.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.952+0000 7fc2da7fc640 1 -- 172.21.15.81:0/2274233427 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fc2e0005470 con 0x7fc2e80b4430 2024-06-29T14:39:55.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.952+0000 7fc2da7fc640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc2cc0763f0 0x7fc2cc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:55.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.952+0000 7fc2da7fc640 1 -- 172.21.15.81:0/2274233427 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5299+0+0 (secure 0 0 0) 0x7fc2e0097590 con 0x7fc2e80b4430 2024-06-29T14:39:55.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.952+0000 7fc2edac6640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc2cc0763f0 0x7fc2cc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:55.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.953+0000 7fc2edac6640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc2cc0763f0 0x7fc2cc078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4034000 tx=0x7fc2e4004730 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:55.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:55.957+0000 7fc2da7fc640 1 -- 172.21.15.81:0/2274233427 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc2e0060f70 con 0x7fc2e80b4430 2024-06-29T14:39:56.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.139+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2274233427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc2e8153a90 con 0x7fc2e80b4430 2024-06-29T14:39:56.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.812+0000 7fc2da7fc640 1 -- 172.21.15.81:0/2274233427 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v117) ==== 170+0+0 (secure 0 0 0) 0x7fc2e0060870 con 0x7fc2e80b4430 2024-06-29T14:39:56.810 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T14:39:56.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc2cc0763f0 msgr2=0x7fc2cc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc2cc0763f0 0x7fc2cc078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4034000 tx=0x7fc2e4004730 comp rx=0 tx=0).stop 2024-06-29T14:39:56.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2274233427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e80b4430 msgr2=0x7fc2e814acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e80b4430 0x7fc2e814acc0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000c760 tx=0x7fc2e000cb60 comp rx=0 tx=0).stop 2024-06-29T14:39:56.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2274233427 shutdown_connections 2024-06-29T14:39:56.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fc2cc0763f0 0x7fc2cc078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2e80b4430 0x7fc2e814acc0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2e80b35b0 0x7fc2e81461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 --2- 172.21.15.81:0/2274233427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2e806f660 0x7fc2e81524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2274233427 >> 172.21.15.81:0/2274233427 conn(0x7fc2e806ccc0 msgr2=0x7fc2e8150d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:56.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2274233427 shutdown_connections 2024-06-29T14:39:56.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.815+0000 7fc2efd2a640 1 -- 172.21.15.81:0/2274233427 wait complete. 2024-06-29T14:39:56.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:39:56.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.953+0000 7f7165acc640 1 -- 172.21.15.81:0/1190633618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600d0230 msgr2=0x7f71600d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.953+0000 7f7165acc640 1 --2- 172.21.15.81:0/1190633618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600d0230 0x7f71600d0640 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f7148009830 tx=0x7f714802a480 comp rx=0 tx=0).stop 2024-06-29T14:39:56.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.953+0000 7f7165acc640 1 -- 172.21.15.81:0/1190633618 shutdown_connections 2024-06-29T14:39:56.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.953+0000 7f7165acc640 1 --2- 172.21.15.81:0/1190633618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600d0230 0x7f71600d0640 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.953+0000 7f7165acc640 1 -- 172.21.15.81:0/1190633618 >> 172.21.15.81:0/1190633618 conn(0x7f71600b3680 msgr2=0x7f71600b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:56.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.953+0000 7f7165acc640 1 -- 172.21.15.81:0/1190633618 shutdown_connections 2024-06-29T14:39:56.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.953+0000 7f7165acc640 1 -- 172.21.15.81:0/1190633618 wait complete. 2024-06-29T14:39:56.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.955+0000 7f7165acc640 1 Processor -- start 2024-06-29T14:39:56.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.955+0000 7f7165acc640 1 -- start start 2024-06-29T14:39:56.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.956+0000 7f7165acc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b3200 0x7f716014eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.956+0000 7f7165acc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71600b4f40 0x7f71600b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.956+0000 7f7165acc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71600d0230 0x7f71601f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.956+0000 7f7165acc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f71600b5e50 con 0x7f71600b3200 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.956+0000 7f7165acc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f71600b5cd0 con 0x7f71600d0230 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.956+0000 7f7165acc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f71600b5fd0 con 0x7f71600b4f40 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b3200 0x7f716014eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b3200 0x7f716014eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44766/0 (socket says 172.21.15.81:44766) 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715f7fe640 1 -- 172.21.15.81:0/786798327 learned_addr learned my addr 172.21.15.81:0/786798327 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715effd640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71600b4f40 0x7f71600b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715ffff640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71600d0230 0x7f71601f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715f7fe640 1 -- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71600b4f40 msgr2=0x7f71600b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715f7fe640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71600b4f40 0x7f71600b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715f7fe640 1 -- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71600d0230 msgr2=0x7f71601f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715f7fe640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71600d0230 0x7f71601f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:56.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715f7fe640 1 -- 172.21.15.81:0/786798327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71600b1370 con 0x7f71600b3200 2024-06-29T14:39:56.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f715f7fe640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b3200 0x7f716014eac0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f715000b410 tx=0x7f715000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:56.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f7157fff640 1 -- 172.21.15.81:0/786798327 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150015020 con 0x7f71600b3200 2024-06-29T14:39:56.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f7157fff640 1 -- 172.21.15.81:0/786798327 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7150011920 con 0x7f71600b3200 2024-06-29T14:39:56.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f7165acc640 1 -- 172.21.15.81:0/786798327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71600b1650 con 0x7f71600b3200 2024-06-29T14:39:56.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f7157fff640 1 -- 172.21.15.81:0/786798327 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150011d70 con 0x7f71600b3200 2024-06-29T14:39:56.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.957+0000 7f7165acc640 1 -- 172.21.15.81:0/786798327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71600b1c10 con 0x7f71600b3200 2024-06-29T14:39:56.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.958+0000 7f7165acc640 1 -- 172.21.15.81:0/786798327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f716009f670 con 0x7f71600b3200 2024-06-29T14:39:56.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.959+0000 7f7157fff640 1 -- 172.21.15.81:0/786798327 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f7150018020 con 0x7f71600b3200 2024-06-29T14:39:56.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.960+0000 7f7157fff640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7140076420 0x7f7140078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:56.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.960+0000 7f7157fff640 1 -- 172.21.15.81:0/786798327 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 5681+0+0 (secure 0 0 0) 0x7f7150097df0 con 0x7f71600b3200 2024-06-29T14:39:56.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.960+0000 7f715effd640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7140076420 0x7f7140078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:56.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.961+0000 7f715effd640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7140076420 0x7f7140078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7148034000 tx=0x7f7148004730 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:56.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:56.964+0000 7f7157fff640 1 -- 172.21.15.81:0/786798327 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f715001f020 con 0x7f71600b3200 2024-06-29T14:39:57.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.182+0000 7f7165acc640 1 -- 172.21.15.81:0/786798327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f71600cedb0 con 0x7f71600b3200 2024-06-29T14:39:57.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.834+0000 7f7157fff640 1 -- 172.21.15.81:0/786798327 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v56) ==== 174+0+0 (secure 0 0 0) 0x7f71500615c0 con 0x7f71600b3200 2024-06-29T14:39:57.832 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 14 and data pool 15 2024-06-29T14:39:57.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.839+0000 7f7165acc640 1 -- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7140076420 msgr2=0x7f7140078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:57.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.840+0000 7f7165acc640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7140076420 0x7f7140078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7148034000 tx=0x7f7148004730 comp rx=0 tx=0).stop 2024-06-29T14:39:57.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.840+0000 7f7165acc640 1 -- 172.21.15.81:0/786798327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b3200 msgr2=0x7f716014eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:57.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.840+0000 7f7165acc640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b3200 0x7f716014eac0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f715000b410 tx=0x7f715000b8d0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.842+0000 7f7165acc640 1 -- 172.21.15.81:0/786798327 shutdown_connections 2024-06-29T14:39:57.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.842+0000 7f7165acc640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7140076420 0x7f7140078930 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.842+0000 7f7165acc640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71600d0230 0x7f71601f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.842+0000 7f7165acc640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71600b4f40 0x7f71600b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.842+0000 7f7165acc640 1 --2- 172.21.15.81:0/786798327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71600b3200 0x7f716014eac0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.842+0000 7f7165acc640 1 -- 172.21.15.81:0/786798327 >> 172.21.15.81:0/786798327 conn(0x7f71600b3680 msgr2=0x7f71600702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:57.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.847+0000 7f7165acc640 1 -- 172.21.15.81:0/786798327 shutdown_connections 2024-06-29T14:39:57.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.851+0000 7f7165acc640 1 -- 172.21.15.81:0/786798327 wait complete. 2024-06-29T14:39:57.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.981+0000 7f47b0555640 1 -- 172.21.15.81:0/2545274090 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a80b5250 msgr2=0x7f47a80d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.981+0000 7f47b0555640 1 --2- 172.21.15.81:0/2545274090 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a80b5250 0x7f47a80d09c0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f47a808e230 tx=0x7f479c02a5f0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.981+0000 7f47b0555640 1 -- 172.21.15.81:0/2545274090 shutdown_connections 2024-06-29T14:39:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.981+0000 7f47b0555640 1 --2- 172.21.15.81:0/2545274090 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a80b5250 0x7f47a80d09c0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.981+0000 7f47b0555640 1 --2- 172.21.15.81:0/2545274090 >> v2:172.21.15.167:3300/0 conn(0x7f47a80b9040 0x7f47a80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.981+0000 7f47b0555640 1 -- 172.21.15.81:0/2545274090 >> 172.21.15.81:0/2545274090 conn(0x7f47a809cf30 msgr2=0x7f47a80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.981+0000 7f47b0555640 1 -- 172.21.15.81:0/2545274090 shutdown_connections 2024-06-29T14:39:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.981+0000 7f47b0555640 1 -- 172.21.15.81:0/2545274090 wait complete. 2024-06-29T14:39:57.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.983+0000 7f47b0555640 1 Processor -- start 2024-06-29T14:39:57.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.983+0000 7f47b0555640 1 -- start start 2024-06-29T14:39:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.984+0000 7f47b0555640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47a80b5250 0x7f47a80b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.984+0000 7f47b0555640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47a80b7290 0x7f47a80b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.984+0000 7f47b0555640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a80b9040 0x7f47a80c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.984+0000 7f47b0555640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f47a80d2e20 con 0x7f47a80b9040 2024-06-29T14:39:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.984+0000 7f47b0555640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f47a80d2ca0 con 0x7f47a80b7290 2024-06-29T14:39:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.984+0000 7f47b0555640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f47a80b7db0 con 0x7f47a80b5250 2024-06-29T14:39:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.984+0000 7f47aeaf2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a80b9040 0x7f47a80c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.984+0000 7f47aeaf2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a80b9040 0x7f47a80c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44780/0 (socket says 172.21.15.81:44780) 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.984+0000 7f47aeaf2640 1 -- 172.21.15.81:0/2824989121 learned_addr learned my addr 172.21.15.81:0/2824989121 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47adaf0640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47a80b7290 0x7f47a80b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47ae2f1640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47a80b5250 0x7f47a80b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47aeaf2640 1 -- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47a80b5250 msgr2=0x7f47a80b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47aeaf2640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47a80b5250 0x7f47a80b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47aeaf2640 1 -- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47a80b7290 msgr2=0x7f47a80b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47aeaf2640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47a80b7290 0x7f47a80b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47aeaf2640 1 -- 172.21.15.81:0/2824989121 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a80b0d60 con 0x7f47a80b9040 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47aeaf2640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a80b9040 0x7f47a80c7cf0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f47a000b3e0 tx=0x7f47a000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47a6ffd640 1 -- 172.21.15.81:0/2824989121 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0015030 con 0x7f47a80b9040 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47a6ffd640 1 -- 172.21.15.81:0/2824989121 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47a0004110 con 0x7f47a80b9040 2024-06-29T14:39:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47b0555640 1 -- 172.21.15.81:0/2824989121 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a80b1270 con 0x7f47a80b9040 2024-06-29T14:39:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47a6ffd640 1 -- 172.21.15.81:0/2824989121 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0011630 con 0x7f47a80b9040 2024-06-29T14:39:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.985+0000 7f47b0555640 1 -- 172.21.15.81:0/2824989121 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a80b1830 con 0x7f47a80b9040 2024-06-29T14:39:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.986+0000 7f47b0555640 1 -- 172.21.15.81:0/2824989121 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a80b0fe0 con 0x7f47a80b9040 2024-06-29T14:39:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.987+0000 7f47a6ffd640 1 -- 172.21.15.81:0/2824989121 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f47a0018070 con 0x7f47a80b9040 2024-06-29T14:39:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.988+0000 7f47a6ffd640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f47940763f0 0x7f4794078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:57.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.988+0000 7f47a6ffd640 1 -- 172.21.15.81:0/2824989121 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5781+0+0 (secure 0 0 0) 0x7f47a0097740 con 0x7f47a80b9040 2024-06-29T14:39:57.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.988+0000 7f47ae2f1640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f47940763f0 0x7f4794078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:57.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.989+0000 7f47ae2f1640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f47940763f0 0x7f4794078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4798005fd0 tx=0x7f4798005f40 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:57.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:57.991+0000 7f47a6ffd640 1 -- 172.21.15.81:0/2824989121 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f47a0061020 con 0x7f47a80b9040 2024-06-29T14:39:58.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.172+0000 7f47b0555640 1 -- 172.21.15.81:0/2824989121 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47a80b82f0 con 0x7f47a80b9040 2024-06-29T14:39:58.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.173+0000 7f47a6ffd640 1 -- 172.21.15.81:0/2824989121 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+16734 (secure 0 0 0) 0x7f47a0060920 con 0x7f47a80b9040 2024-06-29T14:39:58.172 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:58.172 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":118,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:39:57.184020+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:55.492112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.145353+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:58.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 -- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f47940763f0 msgr2=0x7f4794078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f47940763f0 0x7f4794078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4798005fd0 tx=0x7f4798005f40 comp rx=0 tx=0).stop 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 -- 172.21.15.81:0/2824989121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a80b9040 msgr2=0x7f47a80c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a80b9040 0x7f47a80c7cf0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f47a000b3e0 tx=0x7f47a000b8a0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 -- 172.21.15.81:0/2824989121 shutdown_connections 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f47940763f0 0x7f4794078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a80b9040 0x7f47a80c7cf0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47a80b7290 0x7f47a80b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 --2- 172.21.15.81:0/2824989121 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47a80b5250 0x7f47a80b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 -- 172.21.15.81:0/2824989121 >> 172.21.15.81:0/2824989121 conn(0x7f47a809cf30 msgr2=0x7f47a80d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 -- 172.21.15.81:0/2824989121 shutdown_connections 2024-06-29T14:39:58.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.175+0000 7f47b0555640 1 -- 172.21.15.81:0/2824989121 wait complete. 2024-06-29T14:39:58.185 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:39:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.301+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2832934565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f481412f0 msgr2=0x7f0f480cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.301+0000 7f0f4f97f640 1 --2- 172.21.15.81:0/2832934565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f481412f0 0x7f0f480cef90 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f0f480b2dd0 tx=0x7f0f4402a560 comp rx=0 tx=0).stop 2024-06-29T14:39:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.301+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2832934565 shutdown_connections 2024-06-29T14:39:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.301+0000 7f0f4f97f640 1 --2- 172.21.15.81:0/2832934565 >> v2:172.21.15.167:3300/0 conn(0x7f0f480cf4d0 0x7f0f48073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.301+0000 7f0f4f97f640 1 --2- 172.21.15.81:0/2832934565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f481412f0 0x7f0f480cef90 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.301+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2832934565 >> 172.21.15.81:0/2832934565 conn(0x7f0f4806ccc0 msgr2=0x7f0f480b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.301+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2832934565 shutdown_connections 2024-06-29T14:39:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.301+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2832934565 wait complete. 2024-06-29T14:39:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.303+0000 7f0f4f97f640 1 Processor -- start 2024-06-29T14:39:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.303+0000 7f0f4f97f640 1 -- start start 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.304+0000 7f0f4f97f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f48073570 0x7f0f481f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.304+0000 7f0f4f97f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f480cf4d0 0x7f0f48155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.304+0000 7f0f4f97f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f481412f0 0x7f0f4813fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.304+0000 7f0f4f97f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0f480b3210 con 0x7f0f481412f0 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.304+0000 7f0f4f97f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4815f280 con 0x7f0f480cf4d0 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.304+0000 7f0f4f97f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0f480b3390 con 0x7f0f48073570 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.304+0000 7f0f4df1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f481412f0 0x7f0f4813fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.304+0000 7f0f4df1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f481412f0 0x7f0f4813fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44804/0 (socket says 172.21.15.81:44804) 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.304+0000 7f0f4df1c640 1 -- 172.21.15.81:0/2551678904 learned_addr learned my addr 172.21.15.81:0/2551678904 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4cf1a640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f480cf4d0 0x7f0f48155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4d71b640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f48073570 0x7f0f481f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4df1c640 1 -- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f48073570 msgr2=0x7f0f481f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4df1c640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f48073570 0x7f0f481f5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4df1c640 1 -- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f480cf4d0 msgr2=0x7f0f48155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4df1c640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f480cf4d0 0x7f0f48155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4df1c640 1 -- 172.21.15.81:0/2551678904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f481400f0 con 0x7f0f481412f0 2024-06-29T14:39:58.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4cf1a640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f480cf4d0 0x7f0f48155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:58.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4df1c640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f481412f0 0x7f0f4813fbb0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000b9e0 tx=0x7f0f4000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:58.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f3dffb640 1 -- 172.21.15.81:0/2551678904 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f40003b70 con 0x7f0f481412f0 2024-06-29T14:39:58.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f3dffb640 1 -- 172.21.15.81:0/2551678904 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f40003e00 con 0x7f0f481412f0 2024-06-29T14:39:58.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2551678904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f480aa100 con 0x7f0f481412f0 2024-06-29T14:39:58.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f3dffb640 1 -- 172.21.15.81:0/2551678904 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4000cb20 con 0x7f0f481412f0 2024-06-29T14:39:58.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.305+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2551678904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f48154e10 con 0x7f0f481412f0 2024-06-29T14:39:58.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.306+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2551678904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f480b38d0 con 0x7f0f481412f0 2024-06-29T14:39:58.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.307+0000 7f0f3dffb640 1 -- 172.21.15.81:0/2551678904 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f0f4000ccc0 con 0x7f0f481412f0 2024-06-29T14:39:58.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.307+0000 7f0f3dffb640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0f2c0763f0 0x7f0f2c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.307+0000 7f0f3dffb640 1 -- 172.21.15.81:0/2551678904 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5781+0+0 (secure 0 0 0) 0x7f0f400978d0 con 0x7f0f481412f0 2024-06-29T14:39:58.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.308+0000 7f0f4d71b640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0f2c0763f0 0x7f0f2c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.308+0000 7f0f4d71b640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0f2c0763f0 0x7f0f2c078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0f44033000 tx=0x7f0f44004730 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:58.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.312+0000 7f0f3dffb640 1 -- 172.21.15.81:0/2551678904 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f40061040 con 0x7f0f481412f0 2024-06-29T14:39:58.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.493+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2551678904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f4815c540 con 0x7f0f481412f0 2024-06-29T14:39:58.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.495+0000 7f0f3dffb640 1 -- 172.21.15.81:0/2551678904 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) ==== 74+0+16734 (secure 0 0 0) 0x7f0f40060940 con 0x7f0f481412f0 2024-06-29T14:39:58.493 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:39:58.493 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":118,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:39:57.184020+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:55.492112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.145353+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:39:58.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0f2c0763f0 msgr2=0x7f0f2c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0f2c0763f0 0x7f0f2c078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0f44033000 tx=0x7f0f44004730 comp rx=0 tx=0).stop 2024-06-29T14:39:58.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2551678904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f481412f0 msgr2=0x7f0f4813fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f481412f0 0x7f0f4813fbb0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000b9e0 tx=0x7f0f4000bea0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2551678904 shutdown_connections 2024-06-29T14:39:58.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f0f2c0763f0 0x7f0f2c078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f481412f0 0x7f0f4813fbb0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f480cf4d0 0x7f0f48155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 --2- 172.21.15.81:0/2551678904 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f48073570 0x7f0f481f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2551678904 >> 172.21.15.81:0/2551678904 conn(0x7f0f4806ccc0 msgr2=0x7f0f4815ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:58.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2551678904 shutdown_connections 2024-06-29T14:39:58.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.497+0000 7f0f4f97f640 1 -- 172.21.15.81:0/2551678904 wait complete. 2024-06-29T14:39:58.507 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:39:58.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.624+0000 7f6691cc5640 1 -- 172.21.15.81:0/1222646717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668c0b3200 msgr2=0x7f668c0cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.624+0000 7f6691cc5640 1 --2- 172.21.15.81:0/1222646717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668c0b3200 0x7f668c0cc7b0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f668c0c5590 tx=0x7f667c02a440 comp rx=0 tx=0).stop 2024-06-29T14:39:58.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.624+0000 7f6691cc5640 1 -- 172.21.15.81:0/1222646717 shutdown_connections 2024-06-29T14:39:58.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.624+0000 7f6691cc5640 1 --2- 172.21.15.81:0/1222646717 >> v2:172.21.15.167:3300/0 conn(0x7f668c0d0940 0x7f668c0b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.624+0000 7f6691cc5640 1 --2- 172.21.15.81:0/1222646717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668c0b3200 0x7f668c0cc7b0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.624+0000 7f6691cc5640 1 -- 172.21.15.81:0/1222646717 >> 172.21.15.81:0/1222646717 conn(0x7f668c0b3680 msgr2=0x7f668c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:58.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.624+0000 7f6691cc5640 1 -- 172.21.15.81:0/1222646717 shutdown_connections 2024-06-29T14:39:58.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.624+0000 7f6691cc5640 1 -- 172.21.15.81:0/1222646717 wait complete. 2024-06-29T14:39:58.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.626+0000 7f6691cc5640 1 Processor -- start 2024-06-29T14:39:58.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.626+0000 7f6691cc5640 1 -- start start 2024-06-29T14:39:58.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.627+0000 7f6691cc5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f668c0b3200 0x7f668c1de880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f6691cc5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668c0cccf0 0x7f668c0cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f6691cc5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f668c1dee80 0x7f668c1e6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f6691cc5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f668c0b5b60 con 0x7f668c0cccf0 2024-06-29T14:39:58.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f6691cc5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f668c0b59e0 con 0x7f668c1dee80 2024-06-29T14:39:58.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f6691cc5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f668c0b5ce0 con 0x7f668c0b3200 2024-06-29T14:39:58.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668c0cccf0 0x7f668c0cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668c0cccf0 0x7f668c0cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44810/0 (socket says 172.21.15.81:44810) 2024-06-29T14:39:58.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668affd640 1 -- 172.21.15.81:0/564076619 learned_addr learned my addr 172.21.15.81:0/564076619 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:58.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668b7fe640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f668c0b3200 0x7f668c1de880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668bfff640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f668c1dee80 0x7f668c1e6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668affd640 1 -- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f668c0b3200 msgr2=0x7f668c1de880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668affd640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f668c0b3200 0x7f668c1de880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668affd640 1 -- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f668c1dee80 msgr2=0x7f668c1e6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:58.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668affd640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f668c1dee80 0x7f668c1e6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:58.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668affd640 1 -- 172.21.15.81:0/564076619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668c0c5780 con 0x7f668c0cccf0 2024-06-29T14:39:58.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668bfff640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f668c1dee80 0x7f668c1e6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:58.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668b7fe640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f668c0b3200 0x7f668c1de880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:39:58.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f668affd640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668c0cccf0 0x7f668c0cb530 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f667400c530 tx=0x7f667400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:58.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f6683fff640 1 -- 172.21.15.81:0/564076619 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674016020 con 0x7f668c0cccf0 2024-06-29T14:39:58.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f6683fff640 1 -- 172.21.15.81:0/564076619 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6674012920 con 0x7f668c0cccf0 2024-06-29T14:39:58.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f6691cc5640 1 -- 172.21.15.81:0/564076619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668c0c5a60 con 0x7f668c0cccf0 2024-06-29T14:39:58.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f6683fff640 1 -- 172.21.15.81:0/564076619 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674012d70 con 0x7f668c0cccf0 2024-06-29T14:39:58.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.628+0000 7f6691cc5640 1 -- 172.21.15.81:0/564076619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668c0c6020 con 0x7f668c0cccf0 2024-06-29T14:39:58.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.629+0000 7f6691cc5640 1 -- 172.21.15.81:0/564076619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f668c0cc7b0 con 0x7f668c0cccf0 2024-06-29T14:39:58.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.630+0000 7f6683fff640 1 -- 172.21.15.81:0/564076619 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f6674019020 con 0x7f668c0cccf0 2024-06-29T14:39:58.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.631+0000 7f6683fff640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f666c0763f0 0x7f666c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:58.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.631+0000 7f6683fff640 1 -- 172.21.15.81:0/564076619 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 5781+0+0 (secure 0 0 0) 0x7f6674097a90 con 0x7f668c0cccf0 2024-06-29T14:39:58.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.631+0000 7f668b7fe640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f666c0763f0 0x7f666c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:58.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.631+0000 7f668b7fe640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f666c0763f0 0x7f666c078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f667c034000 tx=0x7f667c004730 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:58.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.635+0000 7f6683fff640 1 -- 172.21.15.81:0/564076619 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6674020020 con 0x7f668c0cccf0 2024-06-29T14:39:58.800 INFO:tasks.ceph.mds.a.smithi081.stderr:2024-06-29T14:39:58.802+0000 7f8317a23640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:39:58.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:58.855+0000 7f6691cc5640 1 -- 172.21.15.81:0/564076619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f668c0d2820 con 0x7f668c0cccf0 2024-06-29T14:39:59.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.845+0000 7f6683fff640 1 -- 172.21.15.81:0/564076619 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v58) ==== 118+0+0 (secure 0 0 0) 0x7f6674069020 con 0x7f668c0cccf0 2024-06-29T14:39:59.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 -- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f666c0763f0 msgr2=0x7f666c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:59.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f666c0763f0 0x7f666c078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f667c034000 tx=0x7f667c004730 comp rx=0 tx=0).stop 2024-06-29T14:39:59.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 -- 172.21.15.81:0/564076619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668c0cccf0 msgr2=0x7f668c0cb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:59.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668c0cccf0 0x7f668c0cb530 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f667400c530 tx=0x7f667400c9f0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 -- 172.21.15.81:0/564076619 shutdown_connections 2024-06-29T14:39:59.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f666c0763f0 0x7f666c078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f668c1dee80 0x7f668c1e6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f668c0cccf0 0x7f668c0cb530 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 --2- 172.21.15.81:0/564076619 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f668c0b3200 0x7f668c1de880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 -- 172.21.15.81:0/564076619 >> 172.21.15.81:0/564076619 conn(0x7f668c0b3680 msgr2=0x7f668c1e3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:59.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.848+0000 7f6691cc5640 1 -- 172.21.15.81:0/564076619 shutdown_connections 2024-06-29T14:39:59.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.849+0000 7f6691cc5640 1 -- 172.21.15.81:0/564076619 wait complete. 2024-06-29T14:39:59.860 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:39:59.861 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:39:59.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.986+0000 7f95e048f640 1 -- 172.21.15.81:0/504860580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d80cd540 msgr2=0x7f95d80d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:59.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.986+0000 7f95e048f640 1 --2- 172.21.15.81:0/504860580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d80cd540 0x7f95d80d00f0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f95d0005fd0 tx=0x7f95d002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.986+0000 7f95e048f640 1 -- 172.21.15.81:0/504860580 shutdown_connections 2024-06-29T14:39:59.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.986+0000 7f95e048f640 1 --2- 172.21.15.81:0/504860580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d80cd540 0x7f95d80d00f0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.986+0000 7f95e048f640 1 --2- 172.21.15.81:0/504860580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d80b6d60 0x7f95d80ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.986+0000 7f95e048f640 1 -- 172.21.15.81:0/504860580 >> 172.21.15.81:0/504860580 conn(0x7f95d80d1960 msgr2=0x7f95d80d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:39:59.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.986+0000 7f95e048f640 1 -- 172.21.15.81:0/504860580 shutdown_connections 2024-06-29T14:39:59.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.986+0000 7f95e048f640 1 -- 172.21.15.81:0/504860580 wait complete. 2024-06-29T14:39:59.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.988+0000 7f95e048f640 1 Processor -- start 2024-06-29T14:39:59.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.988+0000 7f95e048f640 1 -- start start 2024-06-29T14:39:59.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95e048f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d80b6d60 0x7f95d815d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:59.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95e048f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d80cd540 0x7f95d80af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:59.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95e048f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d80cea10 0x7f95d815fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:59.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95e048f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f95d80d07b0 con 0x7f95d80cd540 2024-06-29T14:39:59.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95e048f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f95d80d0630 con 0x7f95d80b6d60 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95e048f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f95d80d0930 con 0x7f95d80cea10 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95dda2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d80cd540 0x7f95d80af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95dda2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d80cd540 0x7f95d80af920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46206/0 (socket says 172.21.15.81:46206) 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95dda2a640 1 -- 172.21.15.81:0/3609575167 learned_addr learned my addr 172.21.15.81:0/3609575167 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95de22b640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d80b6d60 0x7f95d815d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95dea2c640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d80cea10 0x7f95d815fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95dda2a640 1 -- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d80cea10 msgr2=0x7f95d815fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95dda2a640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d80cea10 0x7f95d815fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95dda2a640 1 -- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d80b6d60 msgr2=0x7f95d815d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95dda2a640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d80b6d60 0x7f95d815d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:39:59.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95dda2a640 1 -- 172.21.15.81:0/3609575167 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d80afe60 con 0x7f95d80cd540 2024-06-29T14:39:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.989+0000 7f95de22b640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d80b6d60 0x7f95d815d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:39:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.990+0000 7f95dda2a640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d80cd540 0x7f95d80af920 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f95c800c3a0 tx=0x7f95c800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.990+0000 7f95ceffd640 1 -- 172.21.15.81:0/3609575167 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8017070 con 0x7f95d80cd540 2024-06-29T14:39:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.990+0000 7f95ceffd640 1 -- 172.21.15.81:0/3609575167 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95c80053c0 con 0x7f95d80cd540 2024-06-29T14:39:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.990+0000 7f95e048f640 1 -- 172.21.15.81:0/3609575167 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95d80c68f0 con 0x7f95d80cd540 2024-06-29T14:39:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.990+0000 7f95ceffd640 1 -- 172.21.15.81:0/3609575167 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8003850 con 0x7f95d80cd540 2024-06-29T14:39:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.990+0000 7f95e048f640 1 -- 172.21.15.81:0/3609575167 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95d80c6eb0 con 0x7f95d80cd540 2024-06-29T14:39:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.991+0000 7f95e048f640 1 -- 172.21.15.81:0/3609575167 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95d80c6660 con 0x7f95d80cd540 2024-06-29T14:39:59.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.991+0000 7f95ceffd640 1 -- 172.21.15.81:0/3609575167 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f95c801a070 con 0x7f95d80cd540 2024-06-29T14:39:59.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.992+0000 7f95ceffd640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f95bc076420 0x7f95bc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:39:59.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.992+0000 7f95ceffd640 1 -- 172.21.15.81:0/3609575167 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7f95c8096ce0 con 0x7f95d80cd540 2024-06-29T14:39:59.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.992+0000 7f95de22b640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f95bc076420 0x7f95bc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:39:59.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.993+0000 7f95de22b640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f95bc076420 0x7f95bc078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95d80c5220 tx=0x7f95d40094a0 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:39:59.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:39:59.997+0000 7f95ceffd640 1 -- 172.21.15.81:0/3609575167 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f95c8060450 con 0x7f95d80cd540 2024-06-29T14:40:00.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.215+0000 7f95e048f640 1 -- 172.21.15.81:0/3609575167 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f95d80ce4d0 con 0x7f95d80cd540 2024-06-29T14:40:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.847+0000 7f95ceffd640 1 -- 172.21.15.81:0/3609575167 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v59) ==== 122+0+0 (secure 0 0 0) 0x7f95c805fd50 con 0x7f95d80cd540 2024-06-29T14:40:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 -- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f95bc076420 msgr2=0x7f95bc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f95bc076420 0x7f95bc078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95d80c5220 tx=0x7f95d40094a0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 -- 172.21.15.81:0/3609575167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d80cd540 msgr2=0x7f95d80af920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d80cd540 0x7f95d80af920 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f95c800c3a0 tx=0x7f95c800c860 comp rx=0 tx=0).stop 2024-06-29T14:40:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 -- 172.21.15.81:0/3609575167 shutdown_connections 2024-06-29T14:40:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f95bc076420 0x7f95bc078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d80cea10 0x7f95d815fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d80cd540 0x7f95d80af920 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 --2- 172.21.15.81:0/3609575167 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d80b6d60 0x7f95d815d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 -- 172.21.15.81:0/3609575167 >> 172.21.15.81:0/3609575167 conn(0x7f95d80d1960 msgr2=0x7f95d8071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:00.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 -- 172.21.15.81:0/3609575167 shutdown_connections 2024-06-29T14:40:00.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.850+0000 7f95e048f640 1 -- 172.21.15.81:0/3609575167 wait complete. 2024-06-29T14:40:00.862 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:40:00.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:40:00.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.987+0000 7f3157dac640 1 -- 172.21.15.81:0/1364370130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313c004040 msgr2=0x7f313c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:00.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.987+0000 7f3157dac640 1 --2- 172.21.15.81:0/1364370130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313c004040 0x7f313c004800 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f314000c710 tx=0x7f313c005830 comp rx=0 tx=0).stop 2024-06-29T14:40:00.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.987+0000 7f3156349640 1 -- 172.21.15.81:0/1364370130 reap_dead start 2024-06-29T14:40:00.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.987+0000 7f3157dac640 1 -- 172.21.15.81:0/1364370130 shutdown_connections 2024-06-29T14:40:00.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.987+0000 7f3157dac640 1 -- 172.21.15.81:0/1364370130 >> 172.21.15.81:0/1364370130 conn(0x7f31500b3510 msgr2=0x7f31500b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:00.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.987+0000 7f3157dac640 1 -- 172.21.15.81:0/1364370130 shutdown_connections 2024-06-29T14:40:00.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.987+0000 7f3157dac640 1 -- 172.21.15.81:0/1364370130 wait complete. 2024-06-29T14:40:00.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.989+0000 7f3157dac640 1 Processor -- start 2024-06-29T14:40:00.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.989+0000 7f3157dac640 1 -- start start 2024-06-29T14:40:00.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.990+0000 7f3157dac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313c004040 0x7f31500ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:00.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.990+0000 7f3157dac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31501db4c0 0x7f31501df050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:00.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.990+0000 7f3157dac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31501df590 0x7f31501f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.990+0000 7f3157dac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f31500b4c20 con 0x7f313c004040 2024-06-29T14:40:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.990+0000 7f3157dac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f31500b29c0 con 0x7f31501db4c0 2024-06-29T14:40:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.990+0000 7f3157dac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f31500b4da0 con 0x7f31501df590 2024-06-29T14:40:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155b48640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313c004040 0x7f31500ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155b48640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313c004040 0x7f31500ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46226/0 (socket says 172.21.15.81:46226) 2024-06-29T14:40:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155b48640 1 -- 172.21.15.81:0/2444475145 learned_addr learned my addr 172.21.15.81:0/2444475145 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155347640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31501db4c0 0x7f31501df050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3156349640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31501df590 0x7f31501f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155b48640 1 -- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31501df590 msgr2=0x7f31501f2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:00.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155b48640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31501df590 0x7f31501f2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155b48640 1 -- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31501db4c0 msgr2=0x7f31501df050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155b48640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31501db4c0 0x7f31501df050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155b48640 1 -- 172.21.15.81:0/2444475145 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31500caf90 con 0x7f313c004040 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155347640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31501db4c0 0x7f31501df050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3155b48640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313c004040 0x7f31500ca760 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f314c006590 tx=0x7f314c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f31467fc640 1 -- 172.21.15.81:0/2444475145 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c015070 con 0x7f313c004040 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f31467fc640 1 -- 172.21.15.81:0/2444475145 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f314c011900 con 0x7f313c004040 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3157dac640 1 -- 172.21.15.81:0/2444475145 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31501dff20 con 0x7f313c004040 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f31467fc640 1 -- 172.21.15.81:0/2444475145 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c011d30 con 0x7f313c004040 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.991+0000 7f3157dac640 1 -- 172.21.15.81:0/2444475145 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31500c9e50 con 0x7f313c004040 2024-06-29T14:40:00.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.992+0000 7f3157dac640 1 -- 172.21.15.81:0/2444475145 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31500cdf00 con 0x7f313c004040 2024-06-29T14:40:00.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.993+0000 7f31467fc640 1 -- 172.21.15.81:0/2444475145 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f314c010420 con 0x7f313c004040 2024-06-29T14:40:00.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.993+0000 7f31467fc640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f31340763f0 0x7f3134078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:00.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.993+0000 7f31467fc640 1 -- 172.21.15.81:0/2444475145 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7f314c098720 con 0x7f313c004040 2024-06-29T14:40:00.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.994+0000 7f3155347640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f31340763f0 0x7f3134078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:00.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.994+0000 7f3155347640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f31340763f0 0x7f3134078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3140032000 tx=0x7f3140003760 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:00.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:00.998+0000 7f31467fc640 1 -- 172.21.15.81:0/2444475145 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f314c061e20 con 0x7f313c004040 2024-06-29T14:40:01.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.219+0000 7f3157dac640 1 -- 172.21.15.81:0/2444475145 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31500b5300 con 0x7f313c004040 2024-06-29T14:40:01.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.221+0000 7f31467fc640 1 -- 172.21.15.81:0/2444475145 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5231 (secure 0 0 0) 0x7f314c061720 con 0x7f313c004040 2024-06-29T14:40:01.218 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:40:01.218 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":59,"btime":"2024-06-29T14:40:00:847465+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1563799167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1563799167},{"type":"v1","addr":"172.21.15.167:6837","nonce":1563799167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7354,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3062579579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3062579579},{"type":"v1","addr":"172.21.15.81:6835","nonce":3062579579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7363,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3427077146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3427077146},{"type":"v1","addr":"172.21.15.167:6836","nonce":3427077146}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.183979+0000","modified":"2024-06-29T14:40:00.217351+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/1049868964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1049868964},{"type":"v1","addr":"172.21.15.81:6833","nonce":1049868964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8022,"qdb_cluster":[8022]},"id":7}]} 2024-06-29T14:40:01.219 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 59 2024-06-29T14:40:01.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 -- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f31340763f0 msgr2=0x7f3134078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f31340763f0 0x7f3134078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3140032000 tx=0x7f3140003760 comp rx=0 tx=0).stop 2024-06-29T14:40:01.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 -- 172.21.15.81:0/2444475145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313c004040 msgr2=0x7f31500ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313c004040 0x7f31500ca760 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f314c006590 tx=0x7f314c0039c0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 -- 172.21.15.81:0/2444475145 shutdown_connections 2024-06-29T14:40:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f31340763f0 0x7f3134078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31501df590 0x7f31501f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31501db4c0 0x7f31501df050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 --2- 172.21.15.81:0/2444475145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313c004040 0x7f31500ca760 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 -- 172.21.15.81:0/2444475145 >> 172.21.15.81:0/2444475145 conn(0x7f31500b3510 msgr2=0x7f31501e4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 -- 172.21.15.81:0/2444475145 shutdown_connections 2024-06-29T14:40:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.223+0000 7f3157dac640 1 -- 172.21.15.81:0/2444475145 wait complete. 2024-06-29T14:40:01.232 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:40:01.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.347+0000 7fb245e12640 1 -- 172.21.15.81:0/2124573748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb228004800 msgr2=0x7fb228004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.347+0000 7fb245e12640 1 --2- 172.21.15.81:0/2124573748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb228004800 0x7fb228004c10 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fb234009b50 tx=0x7fb228005830 comp rx=0 tx=0).stop 2024-06-29T14:40:01.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.347+0000 7fb23ffff640 1 -- 172.21.15.81:0/2124573748 reap_dead start 2024-06-29T14:40:01.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.347+0000 7fb245e12640 1 -- 172.21.15.81:0/2124573748 shutdown_connections 2024-06-29T14:40:01.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.347+0000 7fb245e12640 1 -- 172.21.15.81:0/2124573748 >> 172.21.15.81:0/2124573748 conn(0x7fb2400d0930 msgr2=0x7fb2400d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:01.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.347+0000 7fb245e12640 1 -- 172.21.15.81:0/2124573748 shutdown_connections 2024-06-29T14:40:01.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.347+0000 7fb245e12640 1 -- 172.21.15.81:0/2124573748 wait complete. 2024-06-29T14:40:01.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.349+0000 7fb245e12640 1 Processor -- start 2024-06-29T14:40:01.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.349+0000 7fb245e12640 1 -- start start 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb245e12640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb228004800 0x7fb24014a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb245e12640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2400ae780 0x7fb2400aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb245e12640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb240072f30 0x7fb240073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb245e12640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2400b3ef0 con 0x7fb2400ae780 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb245e12640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb2400b3d70 con 0x7fb240072f30 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb245e12640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb2400b4070 con 0x7fb228004800 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2400ae780 0x7fb2400aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2400ae780 0x7fb2400aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46232/0 (socket says 172.21.15.81:46232) 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23effd640 1 -- 172.21.15.81:0/204895884 learned_addr learned my addr 172.21.15.81:0/204895884 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23f7fe640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb228004800 0x7fb24014a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23effd640 1 -- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb228004800 msgr2=0x7fb24014a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23ffff640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb240072f30 0x7fb240073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23effd640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb228004800 0x7fb24014a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23effd640 1 -- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb240072f30 msgr2=0x7fb240073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23effd640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb240072f30 0x7fb240073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23effd640 1 -- 172.21.15.81:0/204895884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb240073880 con 0x7fb2400ae780 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23ffff640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb240072f30 0x7fb240073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb23effd640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2400ae780 0x7fb2400aeb90 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fb234036040 tx=0x7fb234009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb227fff640 1 -- 172.21.15.81:0/204895884 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb234038050 con 0x7fb2400ae780 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb227fff640 1 -- 172.21.15.81:0/204895884 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb234039040 con 0x7fb2400ae780 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb245e12640 1 -- 172.21.15.81:0/204895884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2400af9c0 con 0x7fb2400ae780 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.351+0000 7fb227fff640 1 -- 172.21.15.81:0/204895884 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb234033e40 con 0x7fb2400ae780 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.353+0000 7fb245e12640 1 -- 172.21.15.81:0/204895884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2400c7500 con 0x7fb2400ae780 2024-06-29T14:40:01.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.353+0000 7fb245e12640 1 -- 172.21.15.81:0/204895884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb24006e2a0 con 0x7fb2400ae780 2024-06-29T14:40:01.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.354+0000 7fb227fff640 1 -- 172.21.15.81:0/204895884 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fb23402b8c0 con 0x7fb2400ae780 2024-06-29T14:40:01.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.355+0000 7fb227fff640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb22c076420 0x7fb22c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.355+0000 7fb227fff640 1 -- 172.21.15.81:0/204895884 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7fb2340b8c70 con 0x7fb2400ae780 2024-06-29T14:40:01.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.355+0000 7fb23f7fe640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb22c076420 0x7fb22c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.355+0000 7fb23f7fe640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb22c076420 0x7fb22c078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb2300047a0 tx=0x7fb230000ed0 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:01.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.359+0000 7fb227fff640 1 -- 172.21.15.81:0/204895884 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb24006e2a0 con 0x7fb2400ae780 2024-06-29T14:40:01.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.541+0000 7fb245e12640 1 -- 172.21.15.81:0/204895884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb24015aed0 con 0x7fb2400ae780 2024-06-29T14:40:01.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.542+0000 7fb227fff640 1 -- 172.21.15.81:0/204895884 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) ==== 74+0+16721 (secure 0 0 0) 0x7fb240073880 con 0x7fb2400ae780 2024-06-29T14:40:01.540 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:40:01.540 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":119,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:39:58.835175+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:55.492112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.145353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:40:01.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 -- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb22c076420 msgr2=0x7fb22c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb22c076420 0x7fb22c078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb2300047a0 tx=0x7fb230000ed0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 -- 172.21.15.81:0/204895884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2400ae780 msgr2=0x7fb2400aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2400ae780 0x7fb2400aeb90 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fb234036040 tx=0x7fb234009850 comp rx=0 tx=0).stop 2024-06-29T14:40:01.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 -- 172.21.15.81:0/204895884 shutdown_connections 2024-06-29T14:40:01.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fb22c076420 0x7fb22c078930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb240072f30 0x7fb240073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2400ae780 0x7fb2400aeb90 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 --2- 172.21.15.81:0/204895884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb228004800 0x7fb24014a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 -- 172.21.15.81:0/204895884 >> 172.21.15.81:0/204895884 conn(0x7fb2400d0930 msgr2=0x7fb2400cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:01.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 -- 172.21.15.81:0/204895884 shutdown_connections 2024-06-29T14:40:01.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.544+0000 7fb245e12640 1 -- 172.21.15.81:0/204895884 wait complete. 2024-06-29T14:40:01.553 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:40:01.553 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:40:01.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.669+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3767738232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da806f6a0 msgr2=0x7f4da815aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.669+0000 7f4dadbf0640 1 --2- 172.21.15.81:0/3767738232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da806f6a0 0x7f4da815aed0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f4da80743b0 tx=0x7f4d9c02a530 comp rx=0 tx=0).stop 2024-06-29T14:40:01.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.669+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3767738232 shutdown_connections 2024-06-29T14:40:01.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.669+0000 7f4dadbf0640 1 --2- 172.21.15.81:0/3767738232 >> v2:172.21.15.167:3300/0 conn(0x7f4da80b3960 0x7f4da80b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.669+0000 7f4dadbf0640 1 --2- 172.21.15.81:0/3767738232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da806f6a0 0x7f4da815aed0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.669+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3767738232 >> 172.21.15.81:0/3767738232 conn(0x7f4da80d0930 msgr2=0x7f4da80d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:01.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.669+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3767738232 shutdown_connections 2024-06-29T14:40:01.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.669+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3767738232 wait complete. 2024-06-29T14:40:01.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.670+0000 7f4dadbf0640 1 Processor -- start 2024-06-29T14:40:01.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.671+0000 7f4dadbf0640 1 -- start start 2024-06-29T14:40:01.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4dadbf0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4da806f6a0 0x7f4da81e2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4dadbf0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da80b3960 0x7f4da8147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4dadbf0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4da80b57b0 0x7f4da8156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4dadbf0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4da80b4430 con 0x7f4da80b3960 2024-06-29T14:40:01.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4dadbf0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4da80b42b0 con 0x7f4da80b57b0 2024-06-29T14:40:01.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4dadbf0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4da80b45b0 con 0x7f4da806f6a0 2024-06-29T14:40:01.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4da6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da80b3960 0x7f4da8147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4da6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da80b3960 0x7f4da8147330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46252/0 (socket says 172.21.15.81:46252) 2024-06-29T14:40:01.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4da6ffd640 1 -- 172.21.15.81:0/3681825426 learned_addr learned my addr 172.21.15.81:0/3681825426 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4da77fe640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4da806f6a0 0x7f4da81e2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4da7fff640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4da80b57b0 0x7f4da8156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4da6ffd640 1 -- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4da806f6a0 msgr2=0x7f4da81e2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4da6ffd640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4da806f6a0 0x7f4da81e2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4da6ffd640 1 -- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4da80b57b0 msgr2=0x7f4da8156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4da6ffd640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4da80b57b0 0x7f4da8156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.672+0000 7f4da6ffd640 1 -- 172.21.15.81:0/3681825426 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da81f1f70 con 0x7f4da80b3960 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.673+0000 7f4da6ffd640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da80b3960 0x7f4da8147330 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0098f0 tx=0x7f4d9c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.673+0000 7f4d8ffff640 1 -- 172.21.15.81:0/3681825426 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c005220 con 0x7f4da80b3960 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.673+0000 7f4d8ffff640 1 -- 172.21.15.81:0/3681825426 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d9c0054b0 con 0x7f4da80b3960 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.673+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3681825426 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da81f2250 con 0x7f4da80b3960 2024-06-29T14:40:01.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.673+0000 7f4d8ffff640 1 -- 172.21.15.81:0/3681825426 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c003a40 con 0x7f4da80b3960 2024-06-29T14:40:01.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.673+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3681825426 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da81f2810 con 0x7f4da80b3960 2024-06-29T14:40:01.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.674+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3681825426 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da81e3500 con 0x7f4da80b3960 2024-06-29T14:40:01.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.675+0000 7f4d8ffff640 1 -- 172.21.15.81:0/3681825426 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f4d9c003be0 con 0x7f4da80b3960 2024-06-29T14:40:01.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.675+0000 7f4d8ffff640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4d94076420 0x7f4d94078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:01.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.675+0000 7f4d8ffff640 1 -- 172.21.15.81:0/3681825426 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7f4d9c0bb3f0 con 0x7f4da80b3960 2024-06-29T14:40:01.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.675+0000 7f4da77fe640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4d94076420 0x7f4d94078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:01.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.676+0000 7f4da77fe640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4d94076420 0x7f4d94078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4d98007fd0 tx=0x7f4d98009040 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:01.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.680+0000 7f4d8ffff640 1 -- 172.21.15.81:0/3681825426 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4d9c041030 con 0x7f4da80b3960 2024-06-29T14:40:01.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.855+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3681825426 --> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4da806e2a0 con 0x7f4d94076420 2024-06-29T14:40:01.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.860+0000 7f4d8ffff640 1 -- 172.21.15.81:0/3681825426 <== mgr.6533 v2:172.21.15.167:6832/473361950 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252488 (secure 0 0 0) 0x7f4da806e2a0 con 0x7f4d94076420 2024-06-29T14:40:01.861 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:40:01.863 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:40:01.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.863+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4d94076420 msgr2=0x7f4d94078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.863+0000 7f4dadbf0640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4d94076420 0x7f4d94078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4d98007fd0 tx=0x7f4d98009040 comp rx=0 tx=0).stop 2024-06-29T14:40:01.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.863+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3681825426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da80b3960 msgr2=0x7f4da8147330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:01.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.863+0000 7f4dadbf0640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da80b3960 0x7f4da8147330 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0098f0 tx=0x7f4d9c004360 comp rx=0 tx=0).stop 2024-06-29T14:40:01.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.863+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3681825426 shutdown_connections 2024-06-29T14:40:01.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.863+0000 7f4dadbf0640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4d94076420 0x7f4d94078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.863+0000 7f4dadbf0640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4da80b57b0 0x7f4da8156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.863+0000 7f4dadbf0640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da80b3960 0x7f4da8147330 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.864+0000 7f4dadbf0640 1 --2- 172.21.15.81:0/3681825426 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4da806f6a0 0x7f4da81e2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:01.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.864+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3681825426 >> 172.21.15.81:0/3681825426 conn(0x7f4da80d0930 msgr2=0x7f4da80d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:01.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.864+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3681825426 shutdown_connections 2024-06-29T14:40:01.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:01.864+0000 7f4dadbf0640 1 -- 172.21.15.81:0/3681825426 wait complete. 2024-06-29T14:40:01.872 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:40:01.405279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":242,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167,"ondisk_log_size":167,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331508,"kb_used_data":12664,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754643212,"statfs":{"total":773094113280,"available":772754649088,"internally_reserved":0,"allocated":12967936,"data_stored":6801812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":326356530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.596026"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039848+0000","last_change":"2024-06-29T14:39:57.835654+0000","last_active":"2024-06-29T14:39:59.039848+0000","last_peered":"2024-06-29T14:39:59.039848+0000","last_clean":"2024-06-29T14:39:59.039848+0000","last_became_active":"2024-06-29T14:39:57.835297+0000","last_became_peered":"2024-06-29T14:39:57.835297+0000","last_unstale":"2024-06-29T14:39:59.039848+0000","last_undegraded":"2024-06-29T14:39:59.039848+0000","last_fullsized":"2024-06-29T14:39:59.039848+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:33.335092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848259+0000","last_change":"2024-06-29T14:39:57.840717+0000","last_active":"2024-06-29T14:39:58.848259+0000","last_peered":"2024-06-29T14:39:58.848259+0000","last_clean":"2024-06-29T14:39:58.848259+0000","last_became_active":"2024-06-29T14:39:57.840233+0000","last_became_peered":"2024-06-29T14:39:57.840233+0000","last_unstale":"2024-06-29T14:39:58.848259+0000","last_undegraded":"2024-06-29T14:39:58.848259+0000","last_fullsized":"2024-06-29T14:39:58.848259+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:18.320916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039814+0000","last_change":"2024-06-29T14:39:57.835220+0000","last_active":"2024-06-29T14:39:59.039814+0000","last_peered":"2024-06-29T14:39:59.039814+0000","last_clean":"2024-06-29T14:39:59.039814+0000","last_became_active":"2024-06-29T14:39:57.834763+0000","last_became_peered":"2024-06-29T14:39:57.834763+0000","last_unstale":"2024-06-29T14:39:59.039814+0000","last_undegraded":"2024-06-29T14:39:59.039814+0000","last_fullsized":"2024-06-29T14:39:59.039814+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:30.633816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040140+0000","last_change":"2024-06-29T14:39:57.839122+0000","last_active":"2024-06-29T14:39:59.040140+0000","last_peered":"2024-06-29T14:39:59.040140+0000","last_clean":"2024-06-29T14:39:59.040140+0000","last_became_active":"2024-06-29T14:39:57.838817+0000","last_became_peered":"2024-06-29T14:39:57.838817+0000","last_unstale":"2024-06-29T14:39:59.040140+0000","last_undegraded":"2024-06-29T14:39:59.040140+0000","last_fullsized":"2024-06-29T14:39:59.040140+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:24.366124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.844165+0000","last_change":"2024-06-29T14:39:57.855460+0000","last_active":"2024-06-29T14:39:58.844165+0000","last_peered":"2024-06-29T14:39:58.844165+0000","last_clean":"2024-06-29T14:39:58.844165+0000","last_became_active":"2024-06-29T14:39:57.854498+0000","last_became_peered":"2024-06-29T14:39:57.854498+0000","last_unstale":"2024-06-29T14:39:58.844165+0000","last_undegraded":"2024-06-29T14:39:58.844165+0000","last_fullsized":"2024-06-29T14:39:58.844165+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:58.680752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843680+0000","last_change":"2024-06-29T14:39:57.838238+0000","last_active":"2024-06-29T14:39:58.843680+0000","last_peered":"2024-06-29T14:39:58.843680+0000","last_clean":"2024-06-29T14:39:58.843680+0000","last_became_active":"2024-06-29T14:39:57.837888+0000","last_became_peered":"2024-06-29T14:39:57.837888+0000","last_unstale":"2024-06-29T14:39:58.843680+0000","last_undegraded":"2024-06-29T14:39:58.843680+0000","last_fullsized":"2024-06-29T14:39:58.843680+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:02.523081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843051+0000","last_change":"2024-06-29T14:39:57.838453+0000","last_active":"2024-06-29T14:39:58.843051+0000","last_peered":"2024-06-29T14:39:58.843051+0000","last_clean":"2024-06-29T14:39:58.843051+0000","last_became_active":"2024-06-29T14:39:57.838010+0000","last_became_peered":"2024-06-29T14:39:57.838010+0000","last_unstale":"2024-06-29T14:39:58.843051+0000","last_undegraded":"2024-06-29T14:39:58.843051+0000","last_fullsized":"2024-06-29T14:39:58.843051+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:04:14.381186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847819+0000","last_change":"2024-06-29T14:39:57.841723+0000","last_active":"2024-06-29T14:39:58.847819+0000","last_peered":"2024-06-29T14:39:58.847819+0000","last_clean":"2024-06-29T14:39:58.847819+0000","last_became_active":"2024-06-29T14:39:57.841394+0000","last_became_peered":"2024-06-29T14:39:57.841394+0000","last_unstale":"2024-06-29T14:39:58.847819+0000","last_undegraded":"2024-06-29T14:39:58.847819+0000","last_fullsized":"2024-06-29T14:39:58.847819+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:51.814707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040109+0000","last_change":"2024-06-29T14:39:57.846519+0000","last_active":"2024-06-29T14:39:59.040109+0000","last_peered":"2024-06-29T14:39:59.040109+0000","last_clean":"2024-06-29T14:39:59.040109+0000","last_became_active":"2024-06-29T14:39:57.846214+0000","last_became_peered":"2024-06-29T14:39:57.846214+0000","last_unstale":"2024-06-29T14:39:59.040109+0000","last_undegraded":"2024-06-29T14:39:59.040109+0000","last_fullsized":"2024-06-29T14:39:59.040109+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:38:02.569597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843767+0000","last_change":"2024-06-29T14:39:57.850573+0000","last_active":"2024-06-29T14:39:58.843767+0000","last_peered":"2024-06-29T14:39:58.843767+0000","last_clean":"2024-06-29T14:39:58.843767+0000","last_became_active":"2024-06-29T14:39:57.850253+0000","last_became_peered":"2024-06-29T14:39:57.850253+0000","last_unstale":"2024-06-29T14:39:58.843767+0000","last_undegraded":"2024-06-29T14:39:58.843767+0000","last_fullsized":"2024-06-29T14:39:58.843767+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:37.398231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843144+0000","last_change":"2024-06-29T14:39:57.846625+0000","last_active":"2024-06-29T14:39:58.843144+0000","last_peered":"2024-06-29T14:39:58.843144+0000","last_clean":"2024-06-29T14:39:58.843144+0000","last_became_active":"2024-06-29T14:39:57.846093+0000","last_became_peered":"2024-06-29T14:39:57.846093+0000","last_unstale":"2024-06-29T14:39:58.843144+0000","last_undegraded":"2024-06-29T14:39:58.843144+0000","last_fullsized":"2024-06-29T14:39:58.843144+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:47:41.982090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040259+0000","last_change":"2024-06-29T14:39:57.846795+0000","last_active":"2024-06-29T14:39:59.040259+0000","last_peered":"2024-06-29T14:39:59.040259+0000","last_clean":"2024-06-29T14:39:59.040259+0000","last_became_active":"2024-06-29T14:39:57.845938+0000","last_became_peered":"2024-06-29T14:39:57.845938+0000","last_unstale":"2024-06-29T14:39:59.040259+0000","last_undegraded":"2024-06-29T14:39:59.040259+0000","last_fullsized":"2024-06-29T14:39:59.040259+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:13:58.936072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040191+0000","last_change":"2024-06-29T14:39:57.837125+0000","last_active":"2024-06-29T14:39:59.040191+0000","last_peered":"2024-06-29T14:39:59.040191+0000","last_clean":"2024-06-29T14:39:59.040191+0000","last_became_active":"2024-06-29T14:39:57.836819+0000","last_became_peered":"2024-06-29T14:39:57.836819+0000","last_unstale":"2024-06-29T14:39:59.040191+0000","last_undegraded":"2024-06-29T14:39:59.040191+0000","last_fullsized":"2024-06-29T14:39:59.040191+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:46:50.642335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842589+0000","last_change":"2024-06-29T14:39:57.848231+0000","last_active":"2024-06-29T14:39:58.842589+0000","last_peered":"2024-06-29T14:39:58.842589+0000","last_clean":"2024-06-29T14:39:58.842589+0000","last_became_active":"2024-06-29T14:39:57.847950+0000","last_became_peered":"2024-06-29T14:39:57.847950+0000","last_unstale":"2024-06-29T14:39:58.842589+0000","last_undegraded":"2024-06-29T14:39:58.842589+0000","last_fullsized":"2024-06-29T14:39:58.842589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:47.844050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039425+0000","last_change":"2024-06-29T14:39:57.837808+0000","last_active":"2024-06-29T14:39:59.039425+0000","last_peered":"2024-06-29T14:39:59.039425+0000","last_clean":"2024-06-29T14:39:59.039425+0000","last_became_active":"2024-06-29T14:39:57.837478+0000","last_became_peered":"2024-06-29T14:39:57.837478+0000","last_unstale":"2024-06-29T14:39:59.039425+0000","last_undegraded":"2024-06-29T14:39:59.039425+0000","last_fullsized":"2024-06-29T14:39:59.039425+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:35.457935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843626+0000","last_change":"2024-06-29T14:39:57.841970+0000","last_active":"2024-06-29T14:39:58.843626+0000","last_peered":"2024-06-29T14:39:58.843626+0000","last_clean":"2024-06-29T14:39:58.843626+0000","last_became_active":"2024-06-29T14:39:57.841656+0000","last_became_peered":"2024-06-29T14:39:57.841656+0000","last_unstale":"2024-06-29T14:39:58.843626+0000","last_undegraded":"2024-06-29T14:39:58.843626+0000","last_fullsized":"2024-06-29T14:39:58.843626+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:21.019105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848253+0000","last_change":"2024-06-29T14:39:57.842839+0000","last_active":"2024-06-29T14:39:58.848253+0000","last_peered":"2024-06-29T14:39:58.848253+0000","last_clean":"2024-06-29T14:39:58.848253+0000","last_became_active":"2024-06-29T14:39:57.842549+0000","last_became_peered":"2024-06-29T14:39:57.842549+0000","last_unstale":"2024-06-29T14:39:58.848253+0000","last_undegraded":"2024-06-29T14:39:58.848253+0000","last_fullsized":"2024-06-29T14:39:58.848253+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:47:07.408718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040050+0000","last_change":"2024-06-29T14:39:57.849818+0000","last_active":"2024-06-29T14:39:59.040050+0000","last_peered":"2024-06-29T14:39:59.040050+0000","last_clean":"2024-06-29T14:39:59.040050+0000","last_became_active":"2024-06-29T14:39:57.849423+0000","last_became_peered":"2024-06-29T14:39:57.849423+0000","last_unstale":"2024-06-29T14:39:59.040050+0000","last_undegraded":"2024-06-29T14:39:59.040050+0000","last_fullsized":"2024-06-29T14:39:59.040050+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:03.774187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842820+0000","last_change":"2024-06-29T14:39:57.845046+0000","last_active":"2024-06-29T14:39:58.842820+0000","last_peered":"2024-06-29T14:39:58.842820+0000","last_clean":"2024-06-29T14:39:58.842820+0000","last_became_active":"2024-06-29T14:39:57.844738+0000","last_became_peered":"2024-06-29T14:39:57.844738+0000","last_unstale":"2024-06-29T14:39:58.842820+0000","last_undegraded":"2024-06-29T14:39:58.842820+0000","last_fullsized":"2024-06-29T14:39:58.842820+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:43.646878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040319+0000","last_change":"2024-06-29T14:39:57.845938+0000","last_active":"2024-06-29T14:39:59.040319+0000","last_peered":"2024-06-29T14:39:59.040319+0000","last_clean":"2024-06-29T14:39:59.040319+0000","last_became_active":"2024-06-29T14:39:57.845305+0000","last_became_peered":"2024-06-29T14:39:57.845305+0000","last_unstale":"2024-06-29T14:39:59.040319+0000","last_undegraded":"2024-06-29T14:39:59.040319+0000","last_fullsized":"2024-06-29T14:39:59.040319+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:28.336101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842696+0000","last_change":"2024-06-29T14:39:57.839988+0000","last_active":"2024-06-29T14:39:58.842696+0000","last_peered":"2024-06-29T14:39:58.842696+0000","last_clean":"2024-06-29T14:39:58.842696+0000","last_became_active":"2024-06-29T14:39:57.839686+0000","last_became_peered":"2024-06-29T14:39:57.839686+0000","last_unstale":"2024-06-29T14:39:58.842696+0000","last_undegraded":"2024-06-29T14:39:58.842696+0000","last_fullsized":"2024-06-29T14:39:58.842696+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:54:28.240291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847815+0000","last_change":"2024-06-29T14:39:57.842347+0000","last_active":"2024-06-29T14:39:58.847815+0000","last_peered":"2024-06-29T14:39:58.847815+0000","last_clean":"2024-06-29T14:39:58.847815+0000","last_became_active":"2024-06-29T14:39:57.842018+0000","last_became_peered":"2024-06-29T14:39:57.842018+0000","last_unstale":"2024-06-29T14:39:58.847815+0000","last_undegraded":"2024-06-29T14:39:58.847815+0000","last_fullsized":"2024-06-29T14:39:58.847815+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:58.680752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847875+0000","last_change":"2024-06-29T14:39:57.839825+0000","last_active":"2024-06-29T14:39:58.847875+0000","last_peered":"2024-06-29T14:39:58.847875+0000","last_clean":"2024-06-29T14:39:58.847875+0000","last_became_active":"2024-06-29T14:39:57.839430+0000","last_became_peered":"2024-06-29T14:39:57.839430+0000","last_unstale":"2024-06-29T14:39:58.847875+0000","last_undegraded":"2024-06-29T14:39:58.847875+0000","last_fullsized":"2024-06-29T14:39:58.847875+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:44.564916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.844020+0000","last_change":"2024-06-29T14:39:57.844900+0000","last_active":"2024-06-29T14:39:58.844020+0000","last_peered":"2024-06-29T14:39:58.844020+0000","last_clean":"2024-06-29T14:39:58.844020+0000","last_became_active":"2024-06-29T14:39:57.844571+0000","last_became_peered":"2024-06-29T14:39:57.844571+0000","last_unstale":"2024-06-29T14:39:58.844020+0000","last_undegraded":"2024-06-29T14:39:58.844020+0000","last_fullsized":"2024-06-29T14:39:58.844020+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:48:31.337531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.844392+0000","last_change":"2024-06-29T14:39:57.851759+0000","last_active":"2024-06-29T14:39:58.844392+0000","last_peered":"2024-06-29T14:39:58.844392+0000","last_clean":"2024-06-29T14:39:58.844392+0000","last_became_active":"2024-06-29T14:39:57.850952+0000","last_became_peered":"2024-06-29T14:39:57.850952+0000","last_unstale":"2024-06-29T14:39:58.844392+0000","last_undegraded":"2024-06-29T14:39:58.844392+0000","last_fullsized":"2024-06-29T14:39:58.844392+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:51.814707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848318+0000","last_change":"2024-06-29T14:39:57.842268+0000","last_active":"2024-06-29T14:39:58.848318+0000","last_peered":"2024-06-29T14:39:58.848318+0000","last_clean":"2024-06-29T14:39:58.848318+0000","last_became_active":"2024-06-29T14:39:57.841932+0000","last_became_peered":"2024-06-29T14:39:57.841932+0000","last_unstale":"2024-06-29T14:39:58.848318+0000","last_undegraded":"2024-06-29T14:39:58.848318+0000","last_fullsized":"2024-06-29T14:39:58.848318+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:45.003516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843323+0000","last_change":"2024-06-29T14:39:57.843420+0000","last_active":"2024-06-29T14:39:58.843323+0000","last_peered":"2024-06-29T14:39:58.843323+0000","last_clean":"2024-06-29T14:39:58.843323+0000","last_became_active":"2024-06-29T14:39:57.842990+0000","last_became_peered":"2024-06-29T14:39:57.842990+0000","last_unstale":"2024-06-29T14:39:58.843323+0000","last_undegraded":"2024-06-29T14:39:58.843323+0000","last_fullsized":"2024-06-29T14:39:58.843323+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:58.995288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843623+0000","last_change":"2024-06-29T14:39:57.854634+0000","last_active":"2024-06-29T14:39:58.843623+0000","last_peered":"2024-06-29T14:39:58.843623+0000","last_clean":"2024-06-29T14:39:58.843623+0000","last_became_active":"2024-06-29T14:39:57.854305+0000","last_became_peered":"2024-06-29T14:39:57.854305+0000","last_unstale":"2024-06-29T14:39:58.843623+0000","last_undegraded":"2024-06-29T14:39:58.843623+0000","last_fullsized":"2024-06-29T14:39:58.843623+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:05.853092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847920+0000","last_change":"2024-06-29T14:39:57.841161+0000","last_active":"2024-06-29T14:39:58.847920+0000","last_peered":"2024-06-29T14:39:58.847920+0000","last_clean":"2024-06-29T14:39:58.847920+0000","last_became_active":"2024-06-29T14:39:57.840411+0000","last_became_peered":"2024-06-29T14:39:57.840411+0000","last_unstale":"2024-06-29T14:39:58.847920+0000","last_undegraded":"2024-06-29T14:39:58.847920+0000","last_fullsized":"2024-06-29T14:39:58.847920+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:31.166640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842849+0000","last_change":"2024-06-29T14:39:57.845802+0000","last_active":"2024-06-29T14:39:58.842849+0000","last_peered":"2024-06-29T14:39:58.842849+0000","last_clean":"2024-06-29T14:39:58.842849+0000","last_became_active":"2024-06-29T14:39:57.844664+0000","last_became_peered":"2024-06-29T14:39:57.844664+0000","last_unstale":"2024-06-29T14:39:58.842849+0000","last_undegraded":"2024-06-29T14:39:58.842849+0000","last_fullsized":"2024-06-29T14:39:58.842849+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:30.633816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039562+0000","last_change":"2024-06-29T14:39:57.851463+0000","last_active":"2024-06-29T14:39:59.039562+0000","last_peered":"2024-06-29T14:39:59.039562+0000","last_clean":"2024-06-29T14:39:59.039562+0000","last_became_active":"2024-06-29T14:39:57.851110+0000","last_became_peered":"2024-06-29T14:39:57.851110+0000","last_unstale":"2024-06-29T14:39:59.039562+0000","last_undegraded":"2024-06-29T14:39:59.039562+0000","last_fullsized":"2024-06-29T14:39:59.039562+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:30:41.597832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040243+0000","last_change":"2024-06-29T14:39:57.837670+0000","last_active":"2024-06-29T14:39:59.040243+0000","last_peered":"2024-06-29T14:39:59.040243+0000","last_clean":"2024-06-29T14:39:59.040243+0000","last_became_active":"2024-06-29T14:39:57.837383+0000","last_became_peered":"2024-06-29T14:39:57.837383+0000","last_unstale":"2024-06-29T14:39:59.040243+0000","last_undegraded":"2024-06-29T14:39:59.040243+0000","last_fullsized":"2024-06-29T14:39:59.040243+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:08:39.035371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843136+0000","last_change":"2024-06-29T14:39:57.840279+0000","last_active":"2024-06-29T14:39:58.843136+0000","last_peered":"2024-06-29T14:39:58.843136+0000","last_clean":"2024-06-29T14:39:58.843136+0000","last_became_active":"2024-06-29T14:39:57.839953+0000","last_became_peered":"2024-06-29T14:39:57.839953+0000","last_unstale":"2024-06-29T14:39:58.843136+0000","last_undegraded":"2024-06-29T14:39:58.843136+0000","last_fullsized":"2024-06-29T14:39:58.843136+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:03:03.049445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"118'4","reported_seq":18,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848249+0000","last_change":"2024-06-29T14:39:56.833353+0000","last_active":"2024-06-29T14:39:58.848249+0000","last_peered":"2024-06-29T14:39:58.848249+0000","last_clean":"2024-06-29T14:39:58.848249+0000","last_became_active":"2024-06-29T14:39:56.833045+0000","last_became_peered":"2024-06-29T14:39:56.833045+0000","last_unstale":"2024-06-29T14:39:58.848249+0000","last_undegraded":"2024-06-29T14:39:58.848249+0000","last_fullsized":"2024-06-29T14:39:58.848249+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:31.714143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040706+0000","last_change":"2024-06-29T14:39:57.854295+0000","last_active":"2024-06-29T14:39:59.040706+0000","last_peered":"2024-06-29T14:39:59.040706+0000","last_clean":"2024-06-29T14:39:59.040706+0000","last_became_active":"2024-06-29T14:39:57.853932+0000","last_became_peered":"2024-06-29T14:39:57.853932+0000","last_unstale":"2024-06-29T14:39:59.040706+0000","last_undegraded":"2024-06-29T14:39:59.040706+0000","last_fullsized":"2024-06-29T14:39:59.040706+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:27.096845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"118'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843526+0000","last_change":"2024-06-29T14:39:56.841525+0000","last_active":"2024-06-29T14:39:58.843526+0000","last_peered":"2024-06-29T14:39:58.843526+0000","last_clean":"2024-06-29T14:39:58.843526+0000","last_became_active":"2024-06-29T14:39:56.841278+0000","last_became_peered":"2024-06-29T14:39:56.841278+0000","last_unstale":"2024-06-29T14:39:58.843526+0000","last_undegraded":"2024-06-29T14:39:58.843526+0000","last_fullsized":"2024-06-29T14:39:58.843526+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:05:24.682860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039789+0000","last_change":"2024-06-29T14:39:57.843253+0000","last_active":"2024-06-29T14:39:59.039789+0000","last_peered":"2024-06-29T14:39:59.039789+0000","last_clean":"2024-06-29T14:39:59.039789+0000","last_became_active":"2024-06-29T14:39:57.838180+0000","last_became_peered":"2024-06-29T14:39:57.838180+0000","last_unstale":"2024-06-29T14:39:59.039789+0000","last_undegraded":"2024-06-29T14:39:59.039789+0000","last_fullsized":"2024-06-29T14:39:59.039789+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:18:24.334437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847976+0000","last_change":"2024-06-29T14:39:56.838793+0000","last_active":"2024-06-29T14:39:58.847976+0000","last_peered":"2024-06-29T14:39:58.847976+0000","last_clean":"2024-06-29T14:39:58.847976+0000","last_became_active":"2024-06-29T14:39:56.838461+0000","last_became_peered":"2024-06-29T14:39:56.838461+0000","last_unstale":"2024-06-29T14:39:58.847976+0000","last_undegraded":"2024-06-29T14:39:58.847976+0000","last_fullsized":"2024-06-29T14:39:58.847976+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:29.628766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040465+0000","last_change":"2024-06-29T14:39:57.854061+0000","last_active":"2024-06-29T14:39:59.040465+0000","last_peered":"2024-06-29T14:39:59.040465+0000","last_clean":"2024-06-29T14:39:59.040465+0000","last_became_active":"2024-06-29T14:39:57.853619+0000","last_became_peered":"2024-06-29T14:39:57.853619+0000","last_unstale":"2024-06-29T14:39:59.040465+0000","last_undegraded":"2024-06-29T14:39:59.040465+0000","last_fullsized":"2024-06-29T14:39:59.040465+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:18:55.538272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848003+0000","last_change":"2024-06-29T14:39:56.834954+0000","last_active":"2024-06-29T14:39:58.848003+0000","last_peered":"2024-06-29T14:39:58.848003+0000","last_clean":"2024-06-29T14:39:58.848003+0000","last_became_active":"2024-06-29T14:39:56.834655+0000","last_became_peered":"2024-06-29T14:39:56.834655+0000","last_unstale":"2024-06-29T14:39:58.848003+0000","last_undegraded":"2024-06-29T14:39:58.848003+0000","last_fullsized":"2024-06-29T14:39:58.848003+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:54:27.235240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040031+0000","last_change":"2024-06-29T14:39:57.836215+0000","last_active":"2024-06-29T14:39:59.040031+0000","last_peered":"2024-06-29T14:39:59.040031+0000","last_clean":"2024-06-29T14:39:59.040031+0000","last_became_active":"2024-06-29T14:39:57.835890+0000","last_became_peered":"2024-06-29T14:39:57.835890+0000","last_unstale":"2024-06-29T14:39:59.040031+0000","last_undegraded":"2024-06-29T14:39:59.040031+0000","last_fullsized":"2024-06-29T14:39:59.040031+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:44:20.130939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"118'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039650+0000","last_change":"2024-06-29T14:39:56.834107+0000","last_active":"2024-06-29T14:39:59.039650+0000","last_peered":"2024-06-29T14:39:59.039650+0000","last_clean":"2024-06-29T14:39:59.039650+0000","last_became_active":"2024-06-29T14:39:56.833797+0000","last_became_peered":"2024-06-29T14:39:56.833797+0000","last_unstale":"2024-06-29T14:39:59.039650+0000","last_undegraded":"2024-06-29T14:39:59.039650+0000","last_fullsized":"2024-06-29T14:39:59.039650+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:36:05.796567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843925+0000","last_change":"2024-06-29T14:39:57.848519+0000","last_active":"2024-06-29T14:39:58.843925+0000","last_peered":"2024-06-29T14:39:58.843925+0000","last_clean":"2024-06-29T14:39:58.843925+0000","last_became_active":"2024-06-29T14:39:57.848109+0000","last_became_peered":"2024-06-29T14:39:57.848109+0000","last_unstale":"2024-06-29T14:39:58.843925+0000","last_undegraded":"2024-06-29T14:39:58.843925+0000","last_fullsized":"2024-06-29T14:39:58.843925+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:25.657058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"118'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847774+0000","last_change":"2024-06-29T14:39:56.841597+0000","last_active":"2024-06-29T14:39:58.847774+0000","last_peered":"2024-06-29T14:39:58.847774+0000","last_clean":"2024-06-29T14:39:58.847774+0000","last_became_active":"2024-06-29T14:39:56.841265+0000","last_became_peered":"2024-06-29T14:39:56.841265+0000","last_unstale":"2024-06-29T14:39:58.847774+0000","last_undegraded":"2024-06-29T14:39:58.847774+0000","last_fullsized":"2024-06-29T14:39:58.847774+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:18:23.329386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039362+0000","last_change":"2024-06-29T14:39:57.840031+0000","last_active":"2024-06-29T14:39:59.039362+0000","last_peered":"2024-06-29T14:39:59.039362+0000","last_clean":"2024-06-29T14:39:59.039362+0000","last_became_active":"2024-06-29T14:39:57.839145+0000","last_became_peered":"2024-06-29T14:39:57.839145+0000","last_unstale":"2024-06-29T14:39:59.039362+0000","last_undegraded":"2024-06-29T14:39:59.039362+0000","last_fullsized":"2024-06-29T14:39:59.039362+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:05:25.687910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843483+0000","last_change":"2024-06-29T14:39:56.834936+0000","last_active":"2024-06-29T14:39:58.843483+0000","last_peered":"2024-06-29T14:39:58.843483+0000","last_clean":"2024-06-29T14:39:58.843483+0000","last_became_active":"2024-06-29T14:39:56.834591+0000","last_became_peered":"2024-06-29T14:39:56.834591+0000","last_unstale":"2024-06-29T14:39:58.843483+0000","last_undegraded":"2024-06-29T14:39:58.843483+0000","last_fullsized":"2024-06-29T14:39:58.843483+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:18:23.329386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039704+0000","last_change":"2024-06-29T14:39:57.845265+0000","last_active":"2024-06-29T14:39:59.039704+0000","last_peered":"2024-06-29T14:39:59.039704+0000","last_clean":"2024-06-29T14:39:59.039704+0000","last_became_active":"2024-06-29T14:39:57.844745+0000","last_became_peered":"2024-06-29T14:39:57.844745+0000","last_unstale":"2024-06-29T14:39:59.039704+0000","last_undegraded":"2024-06-29T14:39:59.039704+0000","last_fullsized":"2024-06-29T14:39:59.039704+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:22.409261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040252+0000","last_change":"2024-06-29T14:39:56.828457+0000","last_active":"2024-06-29T14:39:59.040252+0000","last_peered":"2024-06-29T14:39:59.040252+0000","last_clean":"2024-06-29T14:39:59.040252+0000","last_became_active":"2024-06-29T14:39:56.828148+0000","last_became_peered":"2024-06-29T14:39:56.828148+0000","last_unstale":"2024-06-29T14:39:59.040252+0000","last_undegraded":"2024-06-29T14:39:59.040252+0000","last_fullsized":"2024-06-29T14:39:59.040252+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:08:10.219706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848410+0000","last_change":"2024-06-29T14:39:57.846566+0000","last_active":"2024-06-29T14:39:58.848410+0000","last_peered":"2024-06-29T14:39:58.848410+0000","last_clean":"2024-06-29T14:39:58.848410+0000","last_became_active":"2024-06-29T14:39:57.846273+0000","last_became_peered":"2024-06-29T14:39:57.846273+0000","last_unstale":"2024-06-29T14:39:58.848410+0000","last_undegraded":"2024-06-29T14:39:58.848410+0000","last_fullsized":"2024-06-29T14:39:58.848410+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:03.847375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040147+0000","last_change":"2024-06-29T14:39:56.823558+0000","last_active":"2024-06-29T14:39:59.040147+0000","last_peered":"2024-06-29T14:39:59.040147+0000","last_clean":"2024-06-29T14:39:59.040147+0000","last_became_active":"2024-06-29T14:39:56.823239+0000","last_became_peered":"2024-06-29T14:39:56.823239+0000","last_unstale":"2024-06-29T14:39:59.040147+0000","last_undegraded":"2024-06-29T14:39:59.040147+0000","last_fullsized":"2024-06-29T14:39:59.040147+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:11.397261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040400+0000","last_change":"2024-06-29T14:39:57.843327+0000","last_active":"2024-06-29T14:39:59.040400+0000","last_peered":"2024-06-29T14:39:59.040400+0000","last_clean":"2024-06-29T14:39:59.040400+0000","last_became_active":"2024-06-29T14:39:57.842952+0000","last_became_peered":"2024-06-29T14:39:57.842952+0000","last_unstale":"2024-06-29T14:39:59.040400+0000","last_undegraded":"2024-06-29T14:39:59.040400+0000","last_fullsized":"2024-06-29T14:39:59.040400+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:06:44.796688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040045+0000","last_change":"2024-06-29T14:39:56.839993+0000","last_active":"2024-06-29T14:39:59.040045+0000","last_peered":"2024-06-29T14:39:59.040045+0000","last_clean":"2024-06-29T14:39:59.040045+0000","last_became_active":"2024-06-29T14:39:56.839705+0000","last_became_peered":"2024-06-29T14:39:56.839705+0000","last_unstale":"2024-06-29T14:39:59.040045+0000","last_undegraded":"2024-06-29T14:39:59.040045+0000","last_fullsized":"2024-06-29T14:39:59.040045+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:55:02.568976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843794+0000","last_change":"2024-06-29T14:39:57.851942+0000","last_active":"2024-06-29T14:39:58.843794+0000","last_peered":"2024-06-29T14:39:58.843794+0000","last_clean":"2024-06-29T14:39:58.843794+0000","last_became_active":"2024-06-29T14:39:57.850883+0000","last_became_peered":"2024-06-29T14:39:57.850883+0000","last_unstale":"2024-06-29T14:39:58.843794+0000","last_undegraded":"2024-06-29T14:39:58.843794+0000","last_fullsized":"2024-06-29T14:39:58.843794+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:34.571580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"118'3","reported_seq":17,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843886+0000","last_change":"2024-06-29T14:39:56.836238+0000","last_active":"2024-06-29T14:39:58.843886+0000","last_peered":"2024-06-29T14:39:58.843886+0000","last_clean":"2024-06-29T14:39:58.843886+0000","last_became_active":"2024-06-29T14:39:56.835911+0000","last_became_peered":"2024-06-29T14:39:56.835911+0000","last_unstale":"2024-06-29T14:39:58.843886+0000","last_undegraded":"2024-06-29T14:39:58.843886+0000","last_fullsized":"2024-06-29T14:39:58.843886+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:30:40.592782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843527+0000","last_change":"2024-06-29T14:39:57.841746+0000","last_active":"2024-06-29T14:39:58.843527+0000","last_peered":"2024-06-29T14:39:58.843527+0000","last_clean":"2024-06-29T14:39:58.843527+0000","last_became_active":"2024-06-29T14:39:57.841099+0000","last_became_peered":"2024-06-29T14:39:57.841099+0000","last_unstale":"2024-06-29T14:39:58.843527+0000","last_undegraded":"2024-06-29T14:39:58.843527+0000","last_fullsized":"2024-06-29T14:39:58.843527+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:51:03.360015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"118'6","reported_seq":20,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040391+0000","last_change":"2024-06-29T14:39:56.832661+0000","last_active":"2024-06-29T14:39:59.040391+0000","last_peered":"2024-06-29T14:39:59.040391+0000","last_clean":"2024-06-29T14:39:59.040391+0000","last_became_active":"2024-06-29T14:39:56.832319+0000","last_became_peered":"2024-06-29T14:39:56.832319+0000","last_unstale":"2024-06-29T14:39:59.040391+0000","last_undegraded":"2024-06-29T14:39:59.040391+0000","last_fullsized":"2024-06-29T14:39:59.040391+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:50.809657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039833+0000","last_change":"2024-06-29T14:39:57.839517+0000","last_active":"2024-06-29T14:39:59.039833+0000","last_peered":"2024-06-29T14:39:59.039833+0000","last_clean":"2024-06-29T14:39:59.039833+0000","last_became_active":"2024-06-29T14:39:57.838591+0000","last_became_peered":"2024-06-29T14:39:57.838591+0000","last_unstale":"2024-06-29T14:39:59.039833+0000","last_undegraded":"2024-06-29T14:39:59.039833+0000","last_fullsized":"2024-06-29T14:39:59.039833+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:07.228887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842800+0000","last_change":"2024-06-29T14:39:56.831998+0000","last_active":"2024-06-29T14:39:58.842800+0000","last_peered":"2024-06-29T14:39:58.842800+0000","last_clean":"2024-06-29T14:39:58.842800+0000","last_became_active":"2024-06-29T14:39:56.831734+0000","last_became_peered":"2024-06-29T14:39:56.831734+0000","last_unstale":"2024-06-29T14:39:58.842800+0000","last_undegraded":"2024-06-29T14:39:58.842800+0000","last_fullsized":"2024-06-29T14:39:58.842800+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:56:45.254013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039682+0000","last_change":"2024-06-29T14:39:57.836269+0000","last_active":"2024-06-29T14:39:59.039682+0000","last_peered":"2024-06-29T14:39:59.039682+0000","last_clean":"2024-06-29T14:39:59.039682+0000","last_became_active":"2024-06-29T14:39:57.835977+0000","last_became_peered":"2024-06-29T14:39:57.835977+0000","last_unstale":"2024-06-29T14:39:59.039682+0000","last_undegraded":"2024-06-29T14:39:59.039682+0000","last_fullsized":"2024-06-29T14:39:59.039682+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:43.646878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843124+0000","last_change":"2024-06-29T14:39:56.826218+0000","last_active":"2024-06-29T14:39:58.843124+0000","last_peered":"2024-06-29T14:39:58.843124+0000","last_clean":"2024-06-29T14:39:58.843124+0000","last_became_active":"2024-06-29T14:39:56.825903+0000","last_became_peered":"2024-06-29T14:39:56.825903+0000","last_unstale":"2024-06-29T14:39:58.843124+0000","last_undegraded":"2024-06-29T14:39:58.843124+0000","last_fullsized":"2024-06-29T14:39:58.843124+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:53.783050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039550+0000","last_change":"2024-06-29T14:39:57.845042+0000","last_active":"2024-06-29T14:39:59.039550+0000","last_peered":"2024-06-29T14:39:59.039550+0000","last_clean":"2024-06-29T14:39:59.039550+0000","last_became_active":"2024-06-29T14:39:57.844707+0000","last_became_peered":"2024-06-29T14:39:57.844707+0000","last_unstale":"2024-06-29T14:39:59.039550+0000","last_undegraded":"2024-06-29T14:39:59.039550+0000","last_fullsized":"2024-06-29T14:39:59.039550+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:26.658664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039837+0000","last_change":"2024-06-29T14:39:56.832545+0000","last_active":"2024-06-29T14:39:59.039837+0000","last_peered":"2024-06-29T14:39:59.039837+0000","last_clean":"2024-06-29T14:39:59.039837+0000","last_became_active":"2024-06-29T14:39:56.832210+0000","last_became_peered":"2024-06-29T14:39:56.832210+0000","last_unstale":"2024-06-29T14:39:59.039837+0000","last_undegraded":"2024-06-29T14:39:59.039837+0000","last_fullsized":"2024-06-29T14:39:59.039837+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:30.161590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842499+0000","last_change":"2024-06-29T14:39:57.847059+0000","last_active":"2024-06-29T14:39:58.842499+0000","last_peered":"2024-06-29T14:39:58.842499+0000","last_clean":"2024-06-29T14:39:58.842499+0000","last_became_active":"2024-06-29T14:39:57.846776+0000","last_became_peered":"2024-06-29T14:39:57.846776+0000","last_unstale":"2024-06-29T14:39:58.842499+0000","last_undegraded":"2024-06-29T14:39:58.842499+0000","last_fullsized":"2024-06-29T14:39:58.842499+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:37:53.246938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"118'8","reported_seq":22,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040312+0000","last_change":"2024-06-29T14:39:56.831629+0000","last_active":"2024-06-29T14:39:59.040312+0000","last_peered":"2024-06-29T14:39:59.040312+0000","last_clean":"2024-06-29T14:39:59.040312+0000","last_became_active":"2024-06-29T14:39:56.831322+0000","last_became_peered":"2024-06-29T14:39:56.831322+0000","last_unstale":"2024-06-29T14:39:59.040312+0000","last_undegraded":"2024-06-29T14:39:59.040312+0000","last_fullsized":"2024-06-29T14:39:59.040312+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:28.289098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.844433+0000","last_change":"2024-06-29T14:39:57.848885+0000","last_active":"2024-06-29T14:39:58.844433+0000","last_peered":"2024-06-29T14:39:58.844433+0000","last_clean":"2024-06-29T14:39:58.844433+0000","last_became_active":"2024-06-29T14:39:57.848548+0000","last_became_peered":"2024-06-29T14:39:57.848548+0000","last_unstale":"2024-06-29T14:39:58.844433+0000","last_undegraded":"2024-06-29T14:39:58.844433+0000","last_fullsized":"2024-06-29T14:39:58.844433+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:44.564916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039830+0000","last_change":"2024-06-29T14:39:56.832929+0000","last_active":"2024-06-29T14:39:59.039830+0000","last_peered":"2024-06-29T14:39:59.039830+0000","last_clean":"2024-06-29T14:39:59.039830+0000","last_became_active":"2024-06-29T14:39:56.831959+0000","last_became_peered":"2024-06-29T14:39:56.831959+0000","last_unstale":"2024-06-29T14:39:59.039830+0000","last_undegraded":"2024-06-29T14:39:59.039830+0000","last_fullsized":"2024-06-29T14:39:59.039830+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:43.559866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039977+0000","last_change":"2024-06-29T14:39:57.851950+0000","last_active":"2024-06-29T14:39:59.039977+0000","last_peered":"2024-06-29T14:39:59.039977+0000","last_clean":"2024-06-29T14:39:59.039977+0000","last_became_active":"2024-06-29T14:39:57.851629+0000","last_became_peered":"2024-06-29T14:39:57.851629+0000","last_unstale":"2024-06-29T14:39:59.039977+0000","last_undegraded":"2024-06-29T14:39:59.039977+0000","last_fullsized":"2024-06-29T14:39:59.039977+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:35:40.221749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"118'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843339+0000","last_change":"2024-06-29T14:39:56.828518+0000","last_active":"2024-06-29T14:39:58.843339+0000","last_peered":"2024-06-29T14:39:58.843339+0000","last_clean":"2024-06-29T14:39:58.843339+0000","last_became_active":"2024-06-29T14:39:56.828200+0000","last_became_peered":"2024-06-29T14:39:56.828200+0000","last_unstale":"2024-06-29T14:39:58.843339+0000","last_undegraded":"2024-06-29T14:39:58.843339+0000","last_fullsized":"2024-06-29T14:39:58.843339+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:29.628766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843772+0000","last_change":"2024-06-29T14:39:57.849829+0000","last_active":"2024-06-29T14:39:58.843772+0000","last_peered":"2024-06-29T14:39:58.843772+0000","last_clean":"2024-06-29T14:39:58.843772+0000","last_became_active":"2024-06-29T14:39:57.849474+0000","last_became_peered":"2024-06-29T14:39:57.849474+0000","last_unstale":"2024-06-29T14:39:58.843772+0000","last_undegraded":"2024-06-29T14:39:58.843772+0000","last_fullsized":"2024-06-29T14:39:58.843772+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:02:22.189268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848021+0000","last_change":"2024-06-29T14:39:56.835706+0000","last_active":"2024-06-29T14:39:58.848021+0000","last_peered":"2024-06-29T14:39:58.848021+0000","last_clean":"2024-06-29T14:39:58.848021+0000","last_became_active":"2024-06-29T14:39:56.835398+0000","last_became_peered":"2024-06-29T14:39:56.835398+0000","last_unstale":"2024-06-29T14:39:58.848021+0000","last_undegraded":"2024-06-29T14:39:58.848021+0000","last_fullsized":"2024-06-29T14:39:58.848021+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:42.641827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843006+0000","last_change":"2024-06-29T14:39:57.849930+0000","last_active":"2024-06-29T14:39:58.843006+0000","last_peered":"2024-06-29T14:39:58.843006+0000","last_clean":"2024-06-29T14:39:58.843006+0000","last_became_active":"2024-06-29T14:39:57.849622+0000","last_became_peered":"2024-06-29T14:39:57.849622+0000","last_unstale":"2024-06-29T14:39:58.843006+0000","last_undegraded":"2024-06-29T14:39:58.843006+0000","last_fullsized":"2024-06-29T14:39:58.843006+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:32.719194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847989+0000","last_change":"2024-06-29T14:39:56.838836+0000","last_active":"2024-06-29T14:39:58.847989+0000","last_peered":"2024-06-29T14:39:58.847989+0000","last_clean":"2024-06-29T14:39:58.847989+0000","last_became_active":"2024-06-29T14:39:56.838514+0000","last_became_peered":"2024-06-29T14:39:56.838514+0000","last_unstale":"2024-06-29T14:39:58.847989+0000","last_undegraded":"2024-06-29T14:39:58.847989+0000","last_fullsized":"2024-06-29T14:39:58.847989+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:03:02.044394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842826+0000","last_change":"2024-06-29T14:39:57.839242+0000","last_active":"2024-06-29T14:39:58.842826+0000","last_peered":"2024-06-29T14:39:58.842826+0000","last_clean":"2024-06-29T14:39:58.842826+0000","last_became_active":"2024-06-29T14:39:57.838968+0000","last_became_peered":"2024-06-29T14:39:57.838968+0000","last_unstale":"2024-06-29T14:39:58.842826+0000","last_undegraded":"2024-06-29T14:39:58.842826+0000","last_fullsized":"2024-06-29T14:39:58.842826+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:18:24.334437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040148+0000","last_change":"2024-06-29T14:39:56.825735+0000","last_active":"2024-06-29T14:39:59.040148+0000","last_peered":"2024-06-29T14:39:59.040148+0000","last_clean":"2024-06-29T14:39:59.040148+0000","last_became_active":"2024-06-29T14:39:56.825208+0000","last_became_peered":"2024-06-29T14:39:56.825208+0000","last_unstale":"2024-06-29T14:39:59.040148+0000","last_undegraded":"2024-06-29T14:39:59.040148+0000","last_fullsized":"2024-06-29T14:39:59.040148+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:16:43.529532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842501+0000","last_change":"2024-06-29T14:39:57.844218+0000","last_active":"2024-06-29T14:39:58.842501+0000","last_peered":"2024-06-29T14:39:58.842501+0000","last_clean":"2024-06-29T14:39:58.842501+0000","last_became_active":"2024-06-29T14:39:57.843944+0000","last_became_peered":"2024-06-29T14:39:57.843944+0000","last_unstale":"2024-06-29T14:39:58.842501+0000","last_undegraded":"2024-06-29T14:39:58.842501+0000","last_fullsized":"2024-06-29T14:39:58.842501+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:55.324776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039779+0000","last_change":"2024-06-29T14:39:56.826709+0000","last_active":"2024-06-29T14:39:59.039779+0000","last_peered":"2024-06-29T14:39:59.039779+0000","last_clean":"2024-06-29T14:39:59.039779+0000","last_became_active":"2024-06-29T14:39:56.826391+0000","last_became_peered":"2024-06-29T14:39:56.826391+0000","last_unstale":"2024-06-29T14:39:59.039779+0000","last_undegraded":"2024-06-29T14:39:59.039779+0000","last_fullsized":"2024-06-29T14:39:59.039779+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:00.808223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843178+0000","last_change":"2024-06-29T14:39:57.849821+0000","last_active":"2024-06-29T14:39:58.843178+0000","last_peered":"2024-06-29T14:39:58.843178+0000","last_clean":"2024-06-29T14:39:58.843178+0000","last_became_active":"2024-06-29T14:39:57.849481+0000","last_became_peered":"2024-06-29T14:39:57.849481+0000","last_unstale":"2024-06-29T14:39:58.843178+0000","last_undegraded":"2024-06-29T14:39:58.843178+0000","last_fullsized":"2024-06-29T14:39:58.843178+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:55:38.416779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848067+0000","last_change":"2024-06-29T14:39:56.838733+0000","last_active":"2024-06-29T14:39:58.848067+0000","last_peered":"2024-06-29T14:39:58.848067+0000","last_clean":"2024-06-29T14:39:58.848067+0000","last_became_active":"2024-06-29T14:39:56.838422+0000","last_became_peered":"2024-06-29T14:39:56.838422+0000","last_unstale":"2024-06-29T14:39:58.848067+0000","last_undegraded":"2024-06-29T14:39:58.848067+0000","last_fullsized":"2024-06-29T14:39:58.848067+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:32.469239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848468+0000","last_change":"2024-06-29T14:39:57.837764+0000","last_active":"2024-06-29T14:39:58.848468+0000","last_peered":"2024-06-29T14:39:58.848468+0000","last_clean":"2024-06-29T14:39:58.848468+0000","last_became_active":"2024-06-29T14:39:57.837449+0000","last_became_peered":"2024-06-29T14:39:57.837449+0000","last_unstale":"2024-06-29T14:39:58.848468+0000","last_undegraded":"2024-06-29T14:39:58.848468+0000","last_fullsized":"2024-06-29T14:39:58.848468+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:08:16.837391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040480+0000","last_change":"2024-06-29T14:39:56.830999+0000","last_active":"2024-06-29T14:39:59.040480+0000","last_peered":"2024-06-29T14:39:59.040480+0000","last_clean":"2024-06-29T14:39:59.040480+0000","last_became_active":"2024-06-29T14:39:56.827267+0000","last_became_peered":"2024-06-29T14:39:56.827267+0000","last_unstale":"2024-06-29T14:39:59.040480+0000","last_undegraded":"2024-06-29T14:39:59.040480+0000","last_fullsized":"2024-06-29T14:39:59.040480+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:25:37.344207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040026+0000","last_change":"2024-06-29T14:39:57.837088+0000","last_active":"2024-06-29T14:39:59.040026+0000","last_peered":"2024-06-29T14:39:59.040026+0000","last_clean":"2024-06-29T14:39:59.040026+0000","last_became_active":"2024-06-29T14:39:57.836740+0000","last_became_peered":"2024-06-29T14:39:57.836740+0000","last_unstale":"2024-06-29T14:39:59.040026+0000","last_undegraded":"2024-06-29T14:39:59.040026+0000","last_fullsized":"2024-06-29T14:39:59.040026+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:03:03.049445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"119'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.849498+0000","last_change":"2024-06-29T14:39:56.834829+0000","last_active":"2024-06-29T14:39:58.849498+0000","last_peered":"2024-06-29T14:39:58.849498+0000","last_clean":"2024-06-29T14:39:58.849498+0000","last_became_active":"2024-06-29T14:39:56.834544+0000","last_became_peered":"2024-06-29T14:39:56.834544+0000","last_unstale":"2024-06-29T14:39:58.849498+0000","last_undegraded":"2024-06-29T14:39:58.849498+0000","last_fullsized":"2024-06-29T14:39:58.849498+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:08:38.030321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039815+0000","last_change":"2024-06-29T14:39:57.848707+0000","last_active":"2024-06-29T14:39:59.039815+0000","last_peered":"2024-06-29T14:39:59.039815+0000","last_clean":"2024-06-29T14:39:59.039815+0000","last_became_active":"2024-06-29T14:39:57.848313+0000","last_became_peered":"2024-06-29T14:39:57.848313+0000","last_unstale":"2024-06-29T14:39:59.039815+0000","last_undegraded":"2024-06-29T14:39:59.039815+0000","last_fullsized":"2024-06-29T14:39:59.039815+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:32.719194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848074+0000","last_change":"2024-06-29T14:39:56.838772+0000","last_active":"2024-06-29T14:39:58.848074+0000","last_peered":"2024-06-29T14:39:58.848074+0000","last_clean":"2024-06-29T14:39:58.848074+0000","last_became_active":"2024-06-29T14:39:56.838409+0000","last_became_peered":"2024-06-29T14:39:56.838409+0000","last_unstale":"2024-06-29T14:39:58.848074+0000","last_undegraded":"2024-06-29T14:39:58.848074+0000","last_fullsized":"2024-06-29T14:39:58.848074+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:32.330042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843203+0000","last_change":"2024-06-29T14:39:57.857016+0000","last_active":"2024-06-29T14:39:58.843203+0000","last_peered":"2024-06-29T14:39:58.843203+0000","last_clean":"2024-06-29T14:39:58.843203+0000","last_became_active":"2024-06-29T14:39:57.856695+0000","last_became_peered":"2024-06-29T14:39:57.856695+0000","last_unstale":"2024-06-29T14:39:58.843203+0000","last_undegraded":"2024-06-29T14:39:58.843203+0000","last_fullsized":"2024-06-29T14:39:58.843203+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:15:02.147328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843921+0000","last_change":"2024-06-29T14:39:56.833503+0000","last_active":"2024-06-29T14:39:58.843921+0000","last_peered":"2024-06-29T14:39:58.843921+0000","last_clean":"2024-06-29T14:39:58.843921+0000","last_became_active":"2024-06-29T14:39:56.833014+0000","last_became_peered":"2024-06-29T14:39:56.833014+0000","last_unstale":"2024-06-29T14:39:58.843921+0000","last_undegraded":"2024-06-29T14:39:58.843921+0000","last_fullsized":"2024-06-29T14:39:58.843921+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:31.714143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843064+0000","last_change":"2024-06-29T14:39:57.841870+0000","last_active":"2024-06-29T14:39:58.843064+0000","last_peered":"2024-06-29T14:39:58.843064+0000","last_clean":"2024-06-29T14:39:58.843064+0000","last_became_active":"2024-06-29T14:39:57.841567+0000","last_became_peered":"2024-06-29T14:39:57.841567+0000","last_unstale":"2024-06-29T14:39:58.843064+0000","last_undegraded":"2024-06-29T14:39:58.843064+0000","last_fullsized":"2024-06-29T14:39:58.843064+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:33.335092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848043+0000","last_change":"2024-06-29T14:39:56.834107+0000","last_active":"2024-06-29T14:39:58.848043+0000","last_peered":"2024-06-29T14:39:58.848043+0000","last_clean":"2024-06-29T14:39:58.848043+0000","last_became_active":"2024-06-29T14:39:56.833794+0000","last_became_peered":"2024-06-29T14:39:56.833794+0000","last_unstale":"2024-06-29T14:39:58.848043+0000","last_undegraded":"2024-06-29T14:39:58.848043+0000","last_fullsized":"2024-06-29T14:39:58.848043+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:44:52.405450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039751+0000","last_change":"2024-06-29T14:39:57.849141+0000","last_active":"2024-06-29T14:39:59.039751+0000","last_peered":"2024-06-29T14:39:59.039751+0000","last_clean":"2024-06-29T14:39:59.039751+0000","last_became_active":"2024-06-29T14:39:57.848649+0000","last_became_peered":"2024-06-29T14:39:57.848649+0000","last_unstale":"2024-06-29T14:39:59.039751+0000","last_undegraded":"2024-06-29T14:39:59.039751+0000","last_fullsized":"2024-06-29T14:39:59.039751+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:22.094937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843572+0000","last_change":"2024-06-29T14:39:56.832250+0000","last_active":"2024-06-29T14:39:58.843572+0000","last_peered":"2024-06-29T14:39:58.843572+0000","last_clean":"2024-06-29T14:39:58.843572+0000","last_became_active":"2024-06-29T14:39:56.831909+0000","last_became_peered":"2024-06-29T14:39:56.831909+0000","last_unstale":"2024-06-29T14:39:58.843572+0000","last_undegraded":"2024-06-29T14:39:58.843572+0000","last_fullsized":"2024-06-29T14:39:58.843572+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:42.641827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842499+0000","last_change":"2024-06-29T14:39:57.841144+0000","last_active":"2024-06-29T14:39:58.842499+0000","last_peered":"2024-06-29T14:39:58.842499+0000","last_clean":"2024-06-29T14:39:58.842499+0000","last_became_active":"2024-06-29T14:39:57.840869+0000","last_became_peered":"2024-06-29T14:39:57.840869+0000","last_unstale":"2024-06-29T14:39:58.842499+0000","last_undegraded":"2024-06-29T14:39:58.842499+0000","last_fullsized":"2024-06-29T14:39:58.842499+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:51.624872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039397+0000","last_change":"2024-06-29T14:39:56.831213+0000","last_active":"2024-06-29T14:39:59.039397+0000","last_peered":"2024-06-29T14:39:59.039397+0000","last_clean":"2024-06-29T14:39:59.039397+0000","last_became_active":"2024-06-29T14:39:56.830908+0000","last_became_peered":"2024-06-29T14:39:56.830908+0000","last_unstale":"2024-06-29T14:39:59.039397+0000","last_undegraded":"2024-06-29T14:39:59.039397+0000","last_fullsized":"2024-06-29T14:39:59.039397+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:49:20.393779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843352+0000","last_change":"2024-06-29T14:39:57.854803+0000","last_active":"2024-06-29T14:39:58.843352+0000","last_peered":"2024-06-29T14:39:58.843352+0000","last_clean":"2024-06-29T14:39:58.843352+0000","last_became_active":"2024-06-29T14:39:57.854308+0000","last_became_peered":"2024-06-29T14:39:57.854308+0000","last_unstale":"2024-06-29T14:39:58.843352+0000","last_undegraded":"2024-06-29T14:39:58.843352+0000","last_fullsized":"2024-06-29T14:39:58.843352+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:51:00.345728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842898+0000","last_change":"2024-06-29T14:39:56.835319+0000","last_active":"2024-06-29T14:39:58.842898+0000","last_peered":"2024-06-29T14:39:58.842898+0000","last_clean":"2024-06-29T14:39:58.842898+0000","last_became_active":"2024-06-29T14:39:56.834975+0000","last_became_peered":"2024-06-29T14:39:56.834975+0000","last_unstale":"2024-06-29T14:39:58.842898+0000","last_undegraded":"2024-06-29T14:39:58.842898+0000","last_fullsized":"2024-06-29T14:39:58.842898+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:24:53.330687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843250+0000","last_change":"2024-06-29T14:39:57.843999+0000","last_active":"2024-06-29T14:39:58.843250+0000","last_peered":"2024-06-29T14:39:58.843250+0000","last_clean":"2024-06-29T14:39:58.843250+0000","last_became_active":"2024-06-29T14:39:57.843707+0000","last_became_peered":"2024-06-29T14:39:57.843707+0000","last_unstale":"2024-06-29T14:39:58.843250+0000","last_undegraded":"2024-06-29T14:39:58.843250+0000","last_fullsized":"2024-06-29T14:39:58.843250+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:08:39.035371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"118'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843566+0000","last_change":"2024-06-29T14:39:56.837957+0000","last_active":"2024-06-29T14:39:58.843566+0000","last_peered":"2024-06-29T14:39:58.843566+0000","last_clean":"2024-06-29T14:39:58.843566+0000","last_became_active":"2024-06-29T14:39:56.837577+0000","last_became_peered":"2024-06-29T14:39:56.837577+0000","last_unstale":"2024-06-29T14:39:58.843566+0000","last_undegraded":"2024-06-29T14:39:58.843566+0000","last_fullsized":"2024-06-29T14:39:58.843566+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:34.452884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"108'128","reported_seq":501,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.844015+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:39:58.844015+0000","last_peered":"2024-06-29T14:39:58.844015+0000","last_clean":"2024-06-29T14:39:58.844015+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:39:58.844015+0000","last_undegraded":"2024-06-29T14:39:58.844015+0000","last_fullsized":"2024-06-29T14:39:58.844015+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":197,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":197,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575060,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33208,"kb_used_data":2024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338632,"statfs":{"total":96636764160,"available":96602759168,"internally_reserved":0,"allocated":2072576,"data_stored":1295698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575061,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36788,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335052,"statfs":{"total":96636764160,"available":96599093248,"internally_reserved":0,"allocated":1478656,"data_stored":705870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575059,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36776,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335064,"statfs":{"total":96636764160,"available":96599105536,"internally_reserved":0,"allocated":1462272,"data_stored":698916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575060,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58992,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312848,"statfs":{"total":96636764160,"available":96576356352,"internally_reserved":0,"allocated":1478656,"data_stored":700582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575060,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":1474560,"data_stored":700480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575060,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40948,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330892,"statfs":{"total":96636764160,"available":96594833408,"internally_reserved":0,"allocated":1474560,"data_stored":705252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54812,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317028,"statfs":{"total":96636764160,"available":96580636672,"internally_reserved":0,"allocated":1458176,"data_stored":698846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575060,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":2068480,"data_stored":1296168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:40:01.877 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:40:02.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.029+0000 7f006688f640 1 -- 172.21.15.81:0/2256246715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0060074520 msgr2=0x7f006006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.029+0000 7f006688f640 1 --2- 172.21.15.81:0/2256246715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0060074520 0x7f006006f140 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f0050009830 tx=0x7f005002a510 comp rx=0 tx=0).stop 2024-06-29T14:40:02.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.029+0000 7f006688f640 1 -- 172.21.15.81:0/2256246715 shutdown_connections 2024-06-29T14:40:02.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.029+0000 7f006688f640 1 --2- 172.21.15.81:0/2256246715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f006006f680 0x7f006006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.029+0000 7f006688f640 1 --2- 172.21.15.81:0/2256246715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0060074520 0x7f006006f140 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.029+0000 7f006688f640 1 -- 172.21.15.81:0/2256246715 >> 172.21.15.81:0/2256246715 conn(0x7f0060072950 msgr2=0x7f00600b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.029+0000 7f006688f640 1 -- 172.21.15.81:0/2256246715 shutdown_connections 2024-06-29T14:40:02.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.029+0000 7f006688f640 1 -- 172.21.15.81:0/2256246715 wait complete. 2024-06-29T14:40:02.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.031+0000 7f006688f640 1 Processor -- start 2024-06-29T14:40:02.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.031+0000 7f006688f640 1 -- start start 2024-06-29T14:40:02.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f006688f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f006006f680 0x7f00601e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f006688f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f006006fa90 0x7f00600cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f006688f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0060074520 0x7f00600b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f006688f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f006015b010 con 0x7f006006f680 2024-06-29T14:40:02.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f006688f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f006006d150 con 0x7f0060074520 2024-06-29T14:40:02.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f006688f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f006015b190 con 0x7f006006fa90 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f005ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f006006f680 0x7f00601e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f005ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f006006f680 0x7f00601e4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46270/0 (socket says 172.21.15.81:46270) 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f005ffff640 1 -- 172.21.15.81:0/1133548522 learned_addr learned my addr 172.21.15.81:0/1133548522 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f005f7fe640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f006006fa90 0x7f00600cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.032+0000 7f0064e2c640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0060074520 0x7f00600b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f005ffff640 1 -- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f006006fa90 msgr2=0x7f00600cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f005ffff640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f006006fa90 0x7f00600cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f005ffff640 1 -- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0060074520 msgr2=0x7f00600b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f005ffff640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0060074520 0x7f00600b0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f005ffff640 1 -- 172.21.15.81:0/1133548522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00600cf6b0 con 0x7f006006f680 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f005ffff640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f006006f680 0x7f00601e4b00 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f0050009950 tx=0x7f0050033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f005cff9640 1 -- 172.21.15.81:0/1133548522 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050031d60 con 0x7f006006f680 2024-06-29T14:40:02.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f005cff9640 1 -- 172.21.15.81:0/1133548522 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00500330d0 con 0x7f006006f680 2024-06-29T14:40:02.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f006688f640 1 -- 172.21.15.81:0/1133548522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00600cf990 con 0x7f006006f680 2024-06-29T14:40:02.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f005cff9640 1 -- 172.21.15.81:0/1133548522 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050033520 con 0x7f006006f680 2024-06-29T14:40:02.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.033+0000 7f006688f640 1 -- 172.21.15.81:0/1133548522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00601e5040 con 0x7f006006f680 2024-06-29T14:40:02.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.034+0000 7f006688f640 1 -- 172.21.15.81:0/1133548522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f006006cc80 con 0x7f006006f680 2024-06-29T14:40:02.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.035+0000 7f005cff9640 1 -- 172.21.15.81:0/1133548522 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f00500035f0 con 0x7f006006f680 2024-06-29T14:40:02.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.035+0000 7f005cff9640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f004c076420 0x7f004c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.035+0000 7f005cff9640 1 -- 172.21.15.81:0/1133548522 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7f00500b8ec0 con 0x7f006006f680 2024-06-29T14:40:02.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.036+0000 7f005f7fe640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f004c076420 0x7f004c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.036+0000 7f005f7fe640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f004c076420 0x7f004c078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0060142da0 tx=0x7f0054008040 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.040+0000 7f005cff9640 1 -- 172.21.15.81:0/1133548522 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f006006cc80 con 0x7f006006f680 2024-06-29T14:40:02.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.217+0000 7f006688f640 1 -- 172.21.15.81:0/1133548522 --> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f00601582e0 con 0x7f004c076420 2024-06-29T14:40:02.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.222+0000 7f005cff9640 1 -- 172.21.15.81:0/1133548522 <== mgr.6533 v2:172.21.15.167:6832/473361950 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252488 (secure 0 0 0) 0x7f00601582e0 con 0x7f004c076420 2024-06-29T14:40:02.221 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:40:02.223 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 -- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f004c076420 msgr2=0x7f004c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f004c076420 0x7f004c078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0060142da0 tx=0x7f0054008040 comp rx=0 tx=0).stop 2024-06-29T14:40:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 -- 172.21.15.81:0/1133548522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f006006f680 msgr2=0x7f00601e4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f006006f680 0x7f00601e4b00 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f0050009950 tx=0x7f0050033d60 comp rx=0 tx=0).stop 2024-06-29T14:40:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 -- 172.21.15.81:0/1133548522 shutdown_connections 2024-06-29T14:40:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f004c076420 0x7f004c078930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0060074520 0x7f00600b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f006006fa90 0x7f00600cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 --2- 172.21.15.81:0/1133548522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f006006f680 0x7f00601e4b00 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 -- 172.21.15.81:0/1133548522 >> 172.21.15.81:0/1133548522 conn(0x7f0060072950 msgr2=0x7f00601597b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 -- 172.21.15.81:0/1133548522 shutdown_connections 2024-06-29T14:40:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.225+0000 7f006688f640 1 -- 172.21.15.81:0/1133548522 wait complete. 2024-06-29T14:40:02.233 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:40:01.405279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":242,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167,"ondisk_log_size":167,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331508,"kb_used_data":12664,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754643212,"statfs":{"total":773094113280,"available":772754649088,"internally_reserved":0,"allocated":12967936,"data_stored":6801812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":326356530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.596026"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039848+0000","last_change":"2024-06-29T14:39:57.835654+0000","last_active":"2024-06-29T14:39:59.039848+0000","last_peered":"2024-06-29T14:39:59.039848+0000","last_clean":"2024-06-29T14:39:59.039848+0000","last_became_active":"2024-06-29T14:39:57.835297+0000","last_became_peered":"2024-06-29T14:39:57.835297+0000","last_unstale":"2024-06-29T14:39:59.039848+0000","last_undegraded":"2024-06-29T14:39:59.039848+0000","last_fullsized":"2024-06-29T14:39:59.039848+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:33.335092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848259+0000","last_change":"2024-06-29T14:39:57.840717+0000","last_active":"2024-06-29T14:39:58.848259+0000","last_peered":"2024-06-29T14:39:58.848259+0000","last_clean":"2024-06-29T14:39:58.848259+0000","last_became_active":"2024-06-29T14:39:57.840233+0000","last_became_peered":"2024-06-29T14:39:57.840233+0000","last_unstale":"2024-06-29T14:39:58.848259+0000","last_undegraded":"2024-06-29T14:39:58.848259+0000","last_fullsized":"2024-06-29T14:39:58.848259+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:18.320916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039814+0000","last_change":"2024-06-29T14:39:57.835220+0000","last_active":"2024-06-29T14:39:59.039814+0000","last_peered":"2024-06-29T14:39:59.039814+0000","last_clean":"2024-06-29T14:39:59.039814+0000","last_became_active":"2024-06-29T14:39:57.834763+0000","last_became_peered":"2024-06-29T14:39:57.834763+0000","last_unstale":"2024-06-29T14:39:59.039814+0000","last_undegraded":"2024-06-29T14:39:59.039814+0000","last_fullsized":"2024-06-29T14:39:59.039814+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:30.633816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040140+0000","last_change":"2024-06-29T14:39:57.839122+0000","last_active":"2024-06-29T14:39:59.040140+0000","last_peered":"2024-06-29T14:39:59.040140+0000","last_clean":"2024-06-29T14:39:59.040140+0000","last_became_active":"2024-06-29T14:39:57.838817+0000","last_became_peered":"2024-06-29T14:39:57.838817+0000","last_unstale":"2024-06-29T14:39:59.040140+0000","last_undegraded":"2024-06-29T14:39:59.040140+0000","last_fullsized":"2024-06-29T14:39:59.040140+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:14:24.366124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.844165+0000","last_change":"2024-06-29T14:39:57.855460+0000","last_active":"2024-06-29T14:39:58.844165+0000","last_peered":"2024-06-29T14:39:58.844165+0000","last_clean":"2024-06-29T14:39:58.844165+0000","last_became_active":"2024-06-29T14:39:57.854498+0000","last_became_peered":"2024-06-29T14:39:57.854498+0000","last_unstale":"2024-06-29T14:39:58.844165+0000","last_undegraded":"2024-06-29T14:39:58.844165+0000","last_fullsized":"2024-06-29T14:39:58.844165+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:58.680752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843680+0000","last_change":"2024-06-29T14:39:57.838238+0000","last_active":"2024-06-29T14:39:58.843680+0000","last_peered":"2024-06-29T14:39:58.843680+0000","last_clean":"2024-06-29T14:39:58.843680+0000","last_became_active":"2024-06-29T14:39:57.837888+0000","last_became_peered":"2024-06-29T14:39:57.837888+0000","last_unstale":"2024-06-29T14:39:58.843680+0000","last_undegraded":"2024-06-29T14:39:58.843680+0000","last_fullsized":"2024-06-29T14:39:58.843680+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:02.523081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843051+0000","last_change":"2024-06-29T14:39:57.838453+0000","last_active":"2024-06-29T14:39:58.843051+0000","last_peered":"2024-06-29T14:39:58.843051+0000","last_clean":"2024-06-29T14:39:58.843051+0000","last_became_active":"2024-06-29T14:39:57.838010+0000","last_became_peered":"2024-06-29T14:39:57.838010+0000","last_unstale":"2024-06-29T14:39:58.843051+0000","last_undegraded":"2024-06-29T14:39:58.843051+0000","last_fullsized":"2024-06-29T14:39:58.843051+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:04:14.381186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847819+0000","last_change":"2024-06-29T14:39:57.841723+0000","last_active":"2024-06-29T14:39:58.847819+0000","last_peered":"2024-06-29T14:39:58.847819+0000","last_clean":"2024-06-29T14:39:58.847819+0000","last_became_active":"2024-06-29T14:39:57.841394+0000","last_became_peered":"2024-06-29T14:39:57.841394+0000","last_unstale":"2024-06-29T14:39:58.847819+0000","last_undegraded":"2024-06-29T14:39:58.847819+0000","last_fullsized":"2024-06-29T14:39:58.847819+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:51.814707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040109+0000","last_change":"2024-06-29T14:39:57.846519+0000","last_active":"2024-06-29T14:39:59.040109+0000","last_peered":"2024-06-29T14:39:59.040109+0000","last_clean":"2024-06-29T14:39:59.040109+0000","last_became_active":"2024-06-29T14:39:57.846214+0000","last_became_peered":"2024-06-29T14:39:57.846214+0000","last_unstale":"2024-06-29T14:39:59.040109+0000","last_undegraded":"2024-06-29T14:39:59.040109+0000","last_fullsized":"2024-06-29T14:39:59.040109+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:38:02.569597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843767+0000","last_change":"2024-06-29T14:39:57.850573+0000","last_active":"2024-06-29T14:39:58.843767+0000","last_peered":"2024-06-29T14:39:58.843767+0000","last_clean":"2024-06-29T14:39:58.843767+0000","last_became_active":"2024-06-29T14:39:57.850253+0000","last_became_peered":"2024-06-29T14:39:57.850253+0000","last_unstale":"2024-06-29T14:39:58.843767+0000","last_undegraded":"2024-06-29T14:39:58.843767+0000","last_fullsized":"2024-06-29T14:39:58.843767+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:37.398231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843144+0000","last_change":"2024-06-29T14:39:57.846625+0000","last_active":"2024-06-29T14:39:58.843144+0000","last_peered":"2024-06-29T14:39:58.843144+0000","last_clean":"2024-06-29T14:39:58.843144+0000","last_became_active":"2024-06-29T14:39:57.846093+0000","last_became_peered":"2024-06-29T14:39:57.846093+0000","last_unstale":"2024-06-29T14:39:58.843144+0000","last_undegraded":"2024-06-29T14:39:58.843144+0000","last_fullsized":"2024-06-29T14:39:58.843144+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:47:41.982090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040259+0000","last_change":"2024-06-29T14:39:57.846795+0000","last_active":"2024-06-29T14:39:59.040259+0000","last_peered":"2024-06-29T14:39:59.040259+0000","last_clean":"2024-06-29T14:39:59.040259+0000","last_became_active":"2024-06-29T14:39:57.845938+0000","last_became_peered":"2024-06-29T14:39:57.845938+0000","last_unstale":"2024-06-29T14:39:59.040259+0000","last_undegraded":"2024-06-29T14:39:59.040259+0000","last_fullsized":"2024-06-29T14:39:59.040259+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:13:58.936072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040191+0000","last_change":"2024-06-29T14:39:57.837125+0000","last_active":"2024-06-29T14:39:59.040191+0000","last_peered":"2024-06-29T14:39:59.040191+0000","last_clean":"2024-06-29T14:39:59.040191+0000","last_became_active":"2024-06-29T14:39:57.836819+0000","last_became_peered":"2024-06-29T14:39:57.836819+0000","last_unstale":"2024-06-29T14:39:59.040191+0000","last_undegraded":"2024-06-29T14:39:59.040191+0000","last_fullsized":"2024-06-29T14:39:59.040191+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:46:50.642335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842589+0000","last_change":"2024-06-29T14:39:57.848231+0000","last_active":"2024-06-29T14:39:58.842589+0000","last_peered":"2024-06-29T14:39:58.842589+0000","last_clean":"2024-06-29T14:39:58.842589+0000","last_became_active":"2024-06-29T14:39:57.847950+0000","last_became_peered":"2024-06-29T14:39:57.847950+0000","last_unstale":"2024-06-29T14:39:58.842589+0000","last_undegraded":"2024-06-29T14:39:58.842589+0000","last_fullsized":"2024-06-29T14:39:58.842589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:27:47.844050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039425+0000","last_change":"2024-06-29T14:39:57.837808+0000","last_active":"2024-06-29T14:39:59.039425+0000","last_peered":"2024-06-29T14:39:59.039425+0000","last_clean":"2024-06-29T14:39:59.039425+0000","last_became_active":"2024-06-29T14:39:57.837478+0000","last_became_peered":"2024-06-29T14:39:57.837478+0000","last_unstale":"2024-06-29T14:39:59.039425+0000","last_undegraded":"2024-06-29T14:39:59.039425+0000","last_fullsized":"2024-06-29T14:39:59.039425+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:35.457935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843626+0000","last_change":"2024-06-29T14:39:57.841970+0000","last_active":"2024-06-29T14:39:58.843626+0000","last_peered":"2024-06-29T14:39:58.843626+0000","last_clean":"2024-06-29T14:39:58.843626+0000","last_became_active":"2024-06-29T14:39:57.841656+0000","last_became_peered":"2024-06-29T14:39:57.841656+0000","last_unstale":"2024-06-29T14:39:58.843626+0000","last_undegraded":"2024-06-29T14:39:58.843626+0000","last_fullsized":"2024-06-29T14:39:58.843626+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:21.019105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848253+0000","last_change":"2024-06-29T14:39:57.842839+0000","last_active":"2024-06-29T14:39:58.848253+0000","last_peered":"2024-06-29T14:39:58.848253+0000","last_clean":"2024-06-29T14:39:58.848253+0000","last_became_active":"2024-06-29T14:39:57.842549+0000","last_became_peered":"2024-06-29T14:39:57.842549+0000","last_unstale":"2024-06-29T14:39:58.848253+0000","last_undegraded":"2024-06-29T14:39:58.848253+0000","last_fullsized":"2024-06-29T14:39:58.848253+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:47:07.408718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040050+0000","last_change":"2024-06-29T14:39:57.849818+0000","last_active":"2024-06-29T14:39:59.040050+0000","last_peered":"2024-06-29T14:39:59.040050+0000","last_clean":"2024-06-29T14:39:59.040050+0000","last_became_active":"2024-06-29T14:39:57.849423+0000","last_became_peered":"2024-06-29T14:39:57.849423+0000","last_unstale":"2024-06-29T14:39:59.040050+0000","last_undegraded":"2024-06-29T14:39:59.040050+0000","last_fullsized":"2024-06-29T14:39:59.040050+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:15:03.774187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842820+0000","last_change":"2024-06-29T14:39:57.845046+0000","last_active":"2024-06-29T14:39:58.842820+0000","last_peered":"2024-06-29T14:39:58.842820+0000","last_clean":"2024-06-29T14:39:58.842820+0000","last_became_active":"2024-06-29T14:39:57.844738+0000","last_became_peered":"2024-06-29T14:39:57.844738+0000","last_unstale":"2024-06-29T14:39:58.842820+0000","last_undegraded":"2024-06-29T14:39:58.842820+0000","last_fullsized":"2024-06-29T14:39:58.842820+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:43.646878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040319+0000","last_change":"2024-06-29T14:39:57.845938+0000","last_active":"2024-06-29T14:39:59.040319+0000","last_peered":"2024-06-29T14:39:59.040319+0000","last_clean":"2024-06-29T14:39:59.040319+0000","last_became_active":"2024-06-29T14:39:57.845305+0000","last_became_peered":"2024-06-29T14:39:57.845305+0000","last_unstale":"2024-06-29T14:39:59.040319+0000","last_undegraded":"2024-06-29T14:39:59.040319+0000","last_fullsized":"2024-06-29T14:39:59.040319+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:03:28.336101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842696+0000","last_change":"2024-06-29T14:39:57.839988+0000","last_active":"2024-06-29T14:39:58.842696+0000","last_peered":"2024-06-29T14:39:58.842696+0000","last_clean":"2024-06-29T14:39:58.842696+0000","last_became_active":"2024-06-29T14:39:57.839686+0000","last_became_peered":"2024-06-29T14:39:57.839686+0000","last_unstale":"2024-06-29T14:39:58.842696+0000","last_undegraded":"2024-06-29T14:39:58.842696+0000","last_fullsized":"2024-06-29T14:39:58.842696+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:54:28.240291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847815+0000","last_change":"2024-06-29T14:39:57.842347+0000","last_active":"2024-06-29T14:39:58.847815+0000","last_peered":"2024-06-29T14:39:58.847815+0000","last_clean":"2024-06-29T14:39:58.847815+0000","last_became_active":"2024-06-29T14:39:57.842018+0000","last_became_peered":"2024-06-29T14:39:57.842018+0000","last_unstale":"2024-06-29T14:39:58.847815+0000","last_undegraded":"2024-06-29T14:39:58.847815+0000","last_fullsized":"2024-06-29T14:39:58.847815+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:58.680752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847875+0000","last_change":"2024-06-29T14:39:57.839825+0000","last_active":"2024-06-29T14:39:58.847875+0000","last_peered":"2024-06-29T14:39:58.847875+0000","last_clean":"2024-06-29T14:39:58.847875+0000","last_became_active":"2024-06-29T14:39:57.839430+0000","last_became_peered":"2024-06-29T14:39:57.839430+0000","last_unstale":"2024-06-29T14:39:58.847875+0000","last_undegraded":"2024-06-29T14:39:58.847875+0000","last_fullsized":"2024-06-29T14:39:58.847875+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:44.564916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.844020+0000","last_change":"2024-06-29T14:39:57.844900+0000","last_active":"2024-06-29T14:39:58.844020+0000","last_peered":"2024-06-29T14:39:58.844020+0000","last_clean":"2024-06-29T14:39:58.844020+0000","last_became_active":"2024-06-29T14:39:57.844571+0000","last_became_peered":"2024-06-29T14:39:57.844571+0000","last_unstale":"2024-06-29T14:39:58.844020+0000","last_undegraded":"2024-06-29T14:39:58.844020+0000","last_fullsized":"2024-06-29T14:39:58.844020+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:48:31.337531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.844392+0000","last_change":"2024-06-29T14:39:57.851759+0000","last_active":"2024-06-29T14:39:58.844392+0000","last_peered":"2024-06-29T14:39:58.844392+0000","last_clean":"2024-06-29T14:39:58.844392+0000","last_became_active":"2024-06-29T14:39:57.850952+0000","last_became_peered":"2024-06-29T14:39:57.850952+0000","last_unstale":"2024-06-29T14:39:58.844392+0000","last_undegraded":"2024-06-29T14:39:58.844392+0000","last_fullsized":"2024-06-29T14:39:58.844392+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:51.814707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848318+0000","last_change":"2024-06-29T14:39:57.842268+0000","last_active":"2024-06-29T14:39:58.848318+0000","last_peered":"2024-06-29T14:39:58.848318+0000","last_clean":"2024-06-29T14:39:58.848318+0000","last_became_active":"2024-06-29T14:39:57.841932+0000","last_became_peered":"2024-06-29T14:39:57.841932+0000","last_unstale":"2024-06-29T14:39:58.848318+0000","last_undegraded":"2024-06-29T14:39:58.848318+0000","last_fullsized":"2024-06-29T14:39:58.848318+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:45.003516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843323+0000","last_change":"2024-06-29T14:39:57.843420+0000","last_active":"2024-06-29T14:39:58.843323+0000","last_peered":"2024-06-29T14:39:58.843323+0000","last_clean":"2024-06-29T14:39:58.843323+0000","last_became_active":"2024-06-29T14:39:57.842990+0000","last_became_peered":"2024-06-29T14:39:57.842990+0000","last_unstale":"2024-06-29T14:39:58.843323+0000","last_undegraded":"2024-06-29T14:39:58.843323+0000","last_fullsized":"2024-06-29T14:39:58.843323+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:58.995288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843623+0000","last_change":"2024-06-29T14:39:57.854634+0000","last_active":"2024-06-29T14:39:58.843623+0000","last_peered":"2024-06-29T14:39:58.843623+0000","last_clean":"2024-06-29T14:39:58.843623+0000","last_became_active":"2024-06-29T14:39:57.854305+0000","last_became_peered":"2024-06-29T14:39:57.854305+0000","last_unstale":"2024-06-29T14:39:58.843623+0000","last_undegraded":"2024-06-29T14:39:58.843623+0000","last_fullsized":"2024-06-29T14:39:58.843623+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:32:05.853092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847920+0000","last_change":"2024-06-29T14:39:57.841161+0000","last_active":"2024-06-29T14:39:58.847920+0000","last_peered":"2024-06-29T14:39:58.847920+0000","last_clean":"2024-06-29T14:39:58.847920+0000","last_became_active":"2024-06-29T14:39:57.840411+0000","last_became_peered":"2024-06-29T14:39:57.840411+0000","last_unstale":"2024-06-29T14:39:58.847920+0000","last_undegraded":"2024-06-29T14:39:58.847920+0000","last_fullsized":"2024-06-29T14:39:58.847920+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:31.166640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842849+0000","last_change":"2024-06-29T14:39:57.845802+0000","last_active":"2024-06-29T14:39:58.842849+0000","last_peered":"2024-06-29T14:39:58.842849+0000","last_clean":"2024-06-29T14:39:58.842849+0000","last_became_active":"2024-06-29T14:39:57.844664+0000","last_became_peered":"2024-06-29T14:39:57.844664+0000","last_unstale":"2024-06-29T14:39:58.842849+0000","last_undegraded":"2024-06-29T14:39:58.842849+0000","last_fullsized":"2024-06-29T14:39:58.842849+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:30.633816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039562+0000","last_change":"2024-06-29T14:39:57.851463+0000","last_active":"2024-06-29T14:39:59.039562+0000","last_peered":"2024-06-29T14:39:59.039562+0000","last_clean":"2024-06-29T14:39:59.039562+0000","last_became_active":"2024-06-29T14:39:57.851110+0000","last_became_peered":"2024-06-29T14:39:57.851110+0000","last_unstale":"2024-06-29T14:39:59.039562+0000","last_undegraded":"2024-06-29T14:39:59.039562+0000","last_fullsized":"2024-06-29T14:39:59.039562+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:30:41.597832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040243+0000","last_change":"2024-06-29T14:39:57.837670+0000","last_active":"2024-06-29T14:39:59.040243+0000","last_peered":"2024-06-29T14:39:59.040243+0000","last_clean":"2024-06-29T14:39:59.040243+0000","last_became_active":"2024-06-29T14:39:57.837383+0000","last_became_peered":"2024-06-29T14:39:57.837383+0000","last_unstale":"2024-06-29T14:39:59.040243+0000","last_undegraded":"2024-06-29T14:39:59.040243+0000","last_fullsized":"2024-06-29T14:39:59.040243+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:08:39.035371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843136+0000","last_change":"2024-06-29T14:39:57.840279+0000","last_active":"2024-06-29T14:39:58.843136+0000","last_peered":"2024-06-29T14:39:58.843136+0000","last_clean":"2024-06-29T14:39:58.843136+0000","last_became_active":"2024-06-29T14:39:57.839953+0000","last_became_peered":"2024-06-29T14:39:57.839953+0000","last_unstale":"2024-06-29T14:39:58.843136+0000","last_undegraded":"2024-06-29T14:39:58.843136+0000","last_fullsized":"2024-06-29T14:39:58.843136+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:03:03.049445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"118'4","reported_seq":18,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848249+0000","last_change":"2024-06-29T14:39:56.833353+0000","last_active":"2024-06-29T14:39:58.848249+0000","last_peered":"2024-06-29T14:39:58.848249+0000","last_clean":"2024-06-29T14:39:58.848249+0000","last_became_active":"2024-06-29T14:39:56.833045+0000","last_became_peered":"2024-06-29T14:39:56.833045+0000","last_unstale":"2024-06-29T14:39:58.848249+0000","last_undegraded":"2024-06-29T14:39:58.848249+0000","last_fullsized":"2024-06-29T14:39:58.848249+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:31.714143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040706+0000","last_change":"2024-06-29T14:39:57.854295+0000","last_active":"2024-06-29T14:39:59.040706+0000","last_peered":"2024-06-29T14:39:59.040706+0000","last_clean":"2024-06-29T14:39:59.040706+0000","last_became_active":"2024-06-29T14:39:57.853932+0000","last_became_peered":"2024-06-29T14:39:57.853932+0000","last_unstale":"2024-06-29T14:39:59.040706+0000","last_undegraded":"2024-06-29T14:39:59.040706+0000","last_fullsized":"2024-06-29T14:39:59.040706+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:27.096845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"118'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843526+0000","last_change":"2024-06-29T14:39:56.841525+0000","last_active":"2024-06-29T14:39:58.843526+0000","last_peered":"2024-06-29T14:39:58.843526+0000","last_clean":"2024-06-29T14:39:58.843526+0000","last_became_active":"2024-06-29T14:39:56.841278+0000","last_became_peered":"2024-06-29T14:39:56.841278+0000","last_unstale":"2024-06-29T14:39:58.843526+0000","last_undegraded":"2024-06-29T14:39:58.843526+0000","last_fullsized":"2024-06-29T14:39:58.843526+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:05:24.682860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039789+0000","last_change":"2024-06-29T14:39:57.843253+0000","last_active":"2024-06-29T14:39:59.039789+0000","last_peered":"2024-06-29T14:39:59.039789+0000","last_clean":"2024-06-29T14:39:59.039789+0000","last_became_active":"2024-06-29T14:39:57.838180+0000","last_became_peered":"2024-06-29T14:39:57.838180+0000","last_unstale":"2024-06-29T14:39:59.039789+0000","last_undegraded":"2024-06-29T14:39:59.039789+0000","last_fullsized":"2024-06-29T14:39:59.039789+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:18:24.334437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847976+0000","last_change":"2024-06-29T14:39:56.838793+0000","last_active":"2024-06-29T14:39:58.847976+0000","last_peered":"2024-06-29T14:39:58.847976+0000","last_clean":"2024-06-29T14:39:58.847976+0000","last_became_active":"2024-06-29T14:39:56.838461+0000","last_became_peered":"2024-06-29T14:39:56.838461+0000","last_unstale":"2024-06-29T14:39:58.847976+0000","last_undegraded":"2024-06-29T14:39:58.847976+0000","last_fullsized":"2024-06-29T14:39:58.847976+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:29.628766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040465+0000","last_change":"2024-06-29T14:39:57.854061+0000","last_active":"2024-06-29T14:39:59.040465+0000","last_peered":"2024-06-29T14:39:59.040465+0000","last_clean":"2024-06-29T14:39:59.040465+0000","last_became_active":"2024-06-29T14:39:57.853619+0000","last_became_peered":"2024-06-29T14:39:57.853619+0000","last_unstale":"2024-06-29T14:39:59.040465+0000","last_undegraded":"2024-06-29T14:39:59.040465+0000","last_fullsized":"2024-06-29T14:39:59.040465+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:18:55.538272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848003+0000","last_change":"2024-06-29T14:39:56.834954+0000","last_active":"2024-06-29T14:39:58.848003+0000","last_peered":"2024-06-29T14:39:58.848003+0000","last_clean":"2024-06-29T14:39:58.848003+0000","last_became_active":"2024-06-29T14:39:56.834655+0000","last_became_peered":"2024-06-29T14:39:56.834655+0000","last_unstale":"2024-06-29T14:39:58.848003+0000","last_undegraded":"2024-06-29T14:39:58.848003+0000","last_fullsized":"2024-06-29T14:39:58.848003+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:54:27.235240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040031+0000","last_change":"2024-06-29T14:39:57.836215+0000","last_active":"2024-06-29T14:39:59.040031+0000","last_peered":"2024-06-29T14:39:59.040031+0000","last_clean":"2024-06-29T14:39:59.040031+0000","last_became_active":"2024-06-29T14:39:57.835890+0000","last_became_peered":"2024-06-29T14:39:57.835890+0000","last_unstale":"2024-06-29T14:39:59.040031+0000","last_undegraded":"2024-06-29T14:39:59.040031+0000","last_fullsized":"2024-06-29T14:39:59.040031+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:44:20.130939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"118'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039650+0000","last_change":"2024-06-29T14:39:56.834107+0000","last_active":"2024-06-29T14:39:59.039650+0000","last_peered":"2024-06-29T14:39:59.039650+0000","last_clean":"2024-06-29T14:39:59.039650+0000","last_became_active":"2024-06-29T14:39:56.833797+0000","last_became_peered":"2024-06-29T14:39:56.833797+0000","last_unstale":"2024-06-29T14:39:59.039650+0000","last_undegraded":"2024-06-29T14:39:59.039650+0000","last_fullsized":"2024-06-29T14:39:59.039650+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:36:05.796567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843925+0000","last_change":"2024-06-29T14:39:57.848519+0000","last_active":"2024-06-29T14:39:58.843925+0000","last_peered":"2024-06-29T14:39:58.843925+0000","last_clean":"2024-06-29T14:39:58.843925+0000","last_became_active":"2024-06-29T14:39:57.848109+0000","last_became_peered":"2024-06-29T14:39:57.848109+0000","last_unstale":"2024-06-29T14:39:58.843925+0000","last_undegraded":"2024-06-29T14:39:58.843925+0000","last_fullsized":"2024-06-29T14:39:58.843925+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:25.657058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"118'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847774+0000","last_change":"2024-06-29T14:39:56.841597+0000","last_active":"2024-06-29T14:39:58.847774+0000","last_peered":"2024-06-29T14:39:58.847774+0000","last_clean":"2024-06-29T14:39:58.847774+0000","last_became_active":"2024-06-29T14:39:56.841265+0000","last_became_peered":"2024-06-29T14:39:56.841265+0000","last_unstale":"2024-06-29T14:39:58.847774+0000","last_undegraded":"2024-06-29T14:39:58.847774+0000","last_fullsized":"2024-06-29T14:39:58.847774+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:18:23.329386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039362+0000","last_change":"2024-06-29T14:39:57.840031+0000","last_active":"2024-06-29T14:39:59.039362+0000","last_peered":"2024-06-29T14:39:59.039362+0000","last_clean":"2024-06-29T14:39:59.039362+0000","last_became_active":"2024-06-29T14:39:57.839145+0000","last_became_peered":"2024-06-29T14:39:57.839145+0000","last_unstale":"2024-06-29T14:39:59.039362+0000","last_undegraded":"2024-06-29T14:39:59.039362+0000","last_fullsized":"2024-06-29T14:39:59.039362+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:05:25.687910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843483+0000","last_change":"2024-06-29T14:39:56.834936+0000","last_active":"2024-06-29T14:39:58.843483+0000","last_peered":"2024-06-29T14:39:58.843483+0000","last_clean":"2024-06-29T14:39:58.843483+0000","last_became_active":"2024-06-29T14:39:56.834591+0000","last_became_peered":"2024-06-29T14:39:56.834591+0000","last_unstale":"2024-06-29T14:39:58.843483+0000","last_undegraded":"2024-06-29T14:39:58.843483+0000","last_fullsized":"2024-06-29T14:39:58.843483+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:18:23.329386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039704+0000","last_change":"2024-06-29T14:39:57.845265+0000","last_active":"2024-06-29T14:39:59.039704+0000","last_peered":"2024-06-29T14:39:59.039704+0000","last_clean":"2024-06-29T14:39:59.039704+0000","last_became_active":"2024-06-29T14:39:57.844745+0000","last_became_peered":"2024-06-29T14:39:57.844745+0000","last_unstale":"2024-06-29T14:39:59.039704+0000","last_undegraded":"2024-06-29T14:39:59.039704+0000","last_fullsized":"2024-06-29T14:39:59.039704+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:22.409261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040252+0000","last_change":"2024-06-29T14:39:56.828457+0000","last_active":"2024-06-29T14:39:59.040252+0000","last_peered":"2024-06-29T14:39:59.040252+0000","last_clean":"2024-06-29T14:39:59.040252+0000","last_became_active":"2024-06-29T14:39:56.828148+0000","last_became_peered":"2024-06-29T14:39:56.828148+0000","last_unstale":"2024-06-29T14:39:59.040252+0000","last_undegraded":"2024-06-29T14:39:59.040252+0000","last_fullsized":"2024-06-29T14:39:59.040252+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:08:10.219706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848410+0000","last_change":"2024-06-29T14:39:57.846566+0000","last_active":"2024-06-29T14:39:58.848410+0000","last_peered":"2024-06-29T14:39:58.848410+0000","last_clean":"2024-06-29T14:39:58.848410+0000","last_became_active":"2024-06-29T14:39:57.846273+0000","last_became_peered":"2024-06-29T14:39:57.846273+0000","last_unstale":"2024-06-29T14:39:58.848410+0000","last_undegraded":"2024-06-29T14:39:58.848410+0000","last_fullsized":"2024-06-29T14:39:58.848410+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:03.847375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040147+0000","last_change":"2024-06-29T14:39:56.823558+0000","last_active":"2024-06-29T14:39:59.040147+0000","last_peered":"2024-06-29T14:39:59.040147+0000","last_clean":"2024-06-29T14:39:59.040147+0000","last_became_active":"2024-06-29T14:39:56.823239+0000","last_became_peered":"2024-06-29T14:39:56.823239+0000","last_unstale":"2024-06-29T14:39:59.040147+0000","last_undegraded":"2024-06-29T14:39:59.040147+0000","last_fullsized":"2024-06-29T14:39:59.040147+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:11.397261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040400+0000","last_change":"2024-06-29T14:39:57.843327+0000","last_active":"2024-06-29T14:39:59.040400+0000","last_peered":"2024-06-29T14:39:59.040400+0000","last_clean":"2024-06-29T14:39:59.040400+0000","last_became_active":"2024-06-29T14:39:57.842952+0000","last_became_peered":"2024-06-29T14:39:57.842952+0000","last_unstale":"2024-06-29T14:39:59.040400+0000","last_undegraded":"2024-06-29T14:39:59.040400+0000","last_fullsized":"2024-06-29T14:39:59.040400+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:06:44.796688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040045+0000","last_change":"2024-06-29T14:39:56.839993+0000","last_active":"2024-06-29T14:39:59.040045+0000","last_peered":"2024-06-29T14:39:59.040045+0000","last_clean":"2024-06-29T14:39:59.040045+0000","last_became_active":"2024-06-29T14:39:56.839705+0000","last_became_peered":"2024-06-29T14:39:56.839705+0000","last_unstale":"2024-06-29T14:39:59.040045+0000","last_undegraded":"2024-06-29T14:39:59.040045+0000","last_fullsized":"2024-06-29T14:39:59.040045+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:55:02.568976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843794+0000","last_change":"2024-06-29T14:39:57.851942+0000","last_active":"2024-06-29T14:39:58.843794+0000","last_peered":"2024-06-29T14:39:58.843794+0000","last_clean":"2024-06-29T14:39:58.843794+0000","last_became_active":"2024-06-29T14:39:57.850883+0000","last_became_peered":"2024-06-29T14:39:57.850883+0000","last_unstale":"2024-06-29T14:39:58.843794+0000","last_undegraded":"2024-06-29T14:39:58.843794+0000","last_fullsized":"2024-06-29T14:39:58.843794+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:34.571580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"118'3","reported_seq":17,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843886+0000","last_change":"2024-06-29T14:39:56.836238+0000","last_active":"2024-06-29T14:39:58.843886+0000","last_peered":"2024-06-29T14:39:58.843886+0000","last_clean":"2024-06-29T14:39:58.843886+0000","last_became_active":"2024-06-29T14:39:56.835911+0000","last_became_peered":"2024-06-29T14:39:56.835911+0000","last_unstale":"2024-06-29T14:39:58.843886+0000","last_undegraded":"2024-06-29T14:39:58.843886+0000","last_fullsized":"2024-06-29T14:39:58.843886+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:30:40.592782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843527+0000","last_change":"2024-06-29T14:39:57.841746+0000","last_active":"2024-06-29T14:39:58.843527+0000","last_peered":"2024-06-29T14:39:58.843527+0000","last_clean":"2024-06-29T14:39:58.843527+0000","last_became_active":"2024-06-29T14:39:57.841099+0000","last_became_peered":"2024-06-29T14:39:57.841099+0000","last_unstale":"2024-06-29T14:39:58.843527+0000","last_undegraded":"2024-06-29T14:39:58.843527+0000","last_fullsized":"2024-06-29T14:39:58.843527+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:51:03.360015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"118'6","reported_seq":20,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040391+0000","last_change":"2024-06-29T14:39:56.832661+0000","last_active":"2024-06-29T14:39:59.040391+0000","last_peered":"2024-06-29T14:39:59.040391+0000","last_clean":"2024-06-29T14:39:59.040391+0000","last_became_active":"2024-06-29T14:39:56.832319+0000","last_became_peered":"2024-06-29T14:39:56.832319+0000","last_unstale":"2024-06-29T14:39:59.040391+0000","last_undegraded":"2024-06-29T14:39:59.040391+0000","last_fullsized":"2024-06-29T14:39:59.040391+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:37:50.809657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039833+0000","last_change":"2024-06-29T14:39:57.839517+0000","last_active":"2024-06-29T14:39:59.039833+0000","last_peered":"2024-06-29T14:39:59.039833+0000","last_clean":"2024-06-29T14:39:59.039833+0000","last_became_active":"2024-06-29T14:39:57.838591+0000","last_became_peered":"2024-06-29T14:39:57.838591+0000","last_unstale":"2024-06-29T14:39:59.039833+0000","last_undegraded":"2024-06-29T14:39:59.039833+0000","last_fullsized":"2024-06-29T14:39:59.039833+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:51:07.228887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842800+0000","last_change":"2024-06-29T14:39:56.831998+0000","last_active":"2024-06-29T14:39:58.842800+0000","last_peered":"2024-06-29T14:39:58.842800+0000","last_clean":"2024-06-29T14:39:58.842800+0000","last_became_active":"2024-06-29T14:39:56.831734+0000","last_became_peered":"2024-06-29T14:39:56.831734+0000","last_unstale":"2024-06-29T14:39:58.842800+0000","last_undegraded":"2024-06-29T14:39:58.842800+0000","last_fullsized":"2024-06-29T14:39:58.842800+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:56:45.254013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039682+0000","last_change":"2024-06-29T14:39:57.836269+0000","last_active":"2024-06-29T14:39:59.039682+0000","last_peered":"2024-06-29T14:39:59.039682+0000","last_clean":"2024-06-29T14:39:59.039682+0000","last_became_active":"2024-06-29T14:39:57.835977+0000","last_became_peered":"2024-06-29T14:39:57.835977+0000","last_unstale":"2024-06-29T14:39:59.039682+0000","last_undegraded":"2024-06-29T14:39:59.039682+0000","last_fullsized":"2024-06-29T14:39:59.039682+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:43.646878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843124+0000","last_change":"2024-06-29T14:39:56.826218+0000","last_active":"2024-06-29T14:39:58.843124+0000","last_peered":"2024-06-29T14:39:58.843124+0000","last_clean":"2024-06-29T14:39:58.843124+0000","last_became_active":"2024-06-29T14:39:56.825903+0000","last_became_peered":"2024-06-29T14:39:56.825903+0000","last_unstale":"2024-06-29T14:39:58.843124+0000","last_undegraded":"2024-06-29T14:39:58.843124+0000","last_fullsized":"2024-06-29T14:39:58.843124+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:53.783050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039550+0000","last_change":"2024-06-29T14:39:57.845042+0000","last_active":"2024-06-29T14:39:59.039550+0000","last_peered":"2024-06-29T14:39:59.039550+0000","last_clean":"2024-06-29T14:39:59.039550+0000","last_became_active":"2024-06-29T14:39:57.844707+0000","last_became_peered":"2024-06-29T14:39:57.844707+0000","last_unstale":"2024-06-29T14:39:59.039550+0000","last_undegraded":"2024-06-29T14:39:59.039550+0000","last_fullsized":"2024-06-29T14:39:59.039550+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:26.658664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039837+0000","last_change":"2024-06-29T14:39:56.832545+0000","last_active":"2024-06-29T14:39:59.039837+0000","last_peered":"2024-06-29T14:39:59.039837+0000","last_clean":"2024-06-29T14:39:59.039837+0000","last_became_active":"2024-06-29T14:39:56.832210+0000","last_became_peered":"2024-06-29T14:39:56.832210+0000","last_unstale":"2024-06-29T14:39:59.039837+0000","last_undegraded":"2024-06-29T14:39:59.039837+0000","last_fullsized":"2024-06-29T14:39:59.039837+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:30.161590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842499+0000","last_change":"2024-06-29T14:39:57.847059+0000","last_active":"2024-06-29T14:39:58.842499+0000","last_peered":"2024-06-29T14:39:58.842499+0000","last_clean":"2024-06-29T14:39:58.842499+0000","last_became_active":"2024-06-29T14:39:57.846776+0000","last_became_peered":"2024-06-29T14:39:57.846776+0000","last_unstale":"2024-06-29T14:39:58.842499+0000","last_undegraded":"2024-06-29T14:39:58.842499+0000","last_fullsized":"2024-06-29T14:39:58.842499+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:37:53.246938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"118'8","reported_seq":22,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040312+0000","last_change":"2024-06-29T14:39:56.831629+0000","last_active":"2024-06-29T14:39:59.040312+0000","last_peered":"2024-06-29T14:39:59.040312+0000","last_clean":"2024-06-29T14:39:59.040312+0000","last_became_active":"2024-06-29T14:39:56.831322+0000","last_became_peered":"2024-06-29T14:39:56.831322+0000","last_unstale":"2024-06-29T14:39:59.040312+0000","last_undegraded":"2024-06-29T14:39:59.040312+0000","last_fullsized":"2024-06-29T14:39:59.040312+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:28.289098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.844433+0000","last_change":"2024-06-29T14:39:57.848885+0000","last_active":"2024-06-29T14:39:58.844433+0000","last_peered":"2024-06-29T14:39:58.844433+0000","last_clean":"2024-06-29T14:39:58.844433+0000","last_became_active":"2024-06-29T14:39:57.848548+0000","last_became_peered":"2024-06-29T14:39:57.848548+0000","last_unstale":"2024-06-29T14:39:58.844433+0000","last_undegraded":"2024-06-29T14:39:58.844433+0000","last_fullsized":"2024-06-29T14:39:58.844433+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:44.564916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039830+0000","last_change":"2024-06-29T14:39:56.832929+0000","last_active":"2024-06-29T14:39:59.039830+0000","last_peered":"2024-06-29T14:39:59.039830+0000","last_clean":"2024-06-29T14:39:59.039830+0000","last_became_active":"2024-06-29T14:39:56.831959+0000","last_became_peered":"2024-06-29T14:39:56.831959+0000","last_unstale":"2024-06-29T14:39:59.039830+0000","last_undegraded":"2024-06-29T14:39:59.039830+0000","last_fullsized":"2024-06-29T14:39:59.039830+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:43.559866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039977+0000","last_change":"2024-06-29T14:39:57.851950+0000","last_active":"2024-06-29T14:39:59.039977+0000","last_peered":"2024-06-29T14:39:59.039977+0000","last_clean":"2024-06-29T14:39:59.039977+0000","last_became_active":"2024-06-29T14:39:57.851629+0000","last_became_peered":"2024-06-29T14:39:57.851629+0000","last_unstale":"2024-06-29T14:39:59.039977+0000","last_undegraded":"2024-06-29T14:39:59.039977+0000","last_fullsized":"2024-06-29T14:39:59.039977+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:35:40.221749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"118'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843339+0000","last_change":"2024-06-29T14:39:56.828518+0000","last_active":"2024-06-29T14:39:58.843339+0000","last_peered":"2024-06-29T14:39:58.843339+0000","last_clean":"2024-06-29T14:39:58.843339+0000","last_became_active":"2024-06-29T14:39:56.828200+0000","last_became_peered":"2024-06-29T14:39:56.828200+0000","last_unstale":"2024-06-29T14:39:58.843339+0000","last_undegraded":"2024-06-29T14:39:58.843339+0000","last_fullsized":"2024-06-29T14:39:58.843339+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:29.628766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843772+0000","last_change":"2024-06-29T14:39:57.849829+0000","last_active":"2024-06-29T14:39:58.843772+0000","last_peered":"2024-06-29T14:39:58.843772+0000","last_clean":"2024-06-29T14:39:58.843772+0000","last_became_active":"2024-06-29T14:39:57.849474+0000","last_became_peered":"2024-06-29T14:39:57.849474+0000","last_unstale":"2024-06-29T14:39:58.843772+0000","last_undegraded":"2024-06-29T14:39:58.843772+0000","last_fullsized":"2024-06-29T14:39:58.843772+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:02:22.189268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848021+0000","last_change":"2024-06-29T14:39:56.835706+0000","last_active":"2024-06-29T14:39:58.848021+0000","last_peered":"2024-06-29T14:39:58.848021+0000","last_clean":"2024-06-29T14:39:58.848021+0000","last_became_active":"2024-06-29T14:39:56.835398+0000","last_became_peered":"2024-06-29T14:39:56.835398+0000","last_unstale":"2024-06-29T14:39:58.848021+0000","last_undegraded":"2024-06-29T14:39:58.848021+0000","last_fullsized":"2024-06-29T14:39:58.848021+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:42.641827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843006+0000","last_change":"2024-06-29T14:39:57.849930+0000","last_active":"2024-06-29T14:39:58.843006+0000","last_peered":"2024-06-29T14:39:58.843006+0000","last_clean":"2024-06-29T14:39:58.843006+0000","last_became_active":"2024-06-29T14:39:57.849622+0000","last_became_peered":"2024-06-29T14:39:57.849622+0000","last_unstale":"2024-06-29T14:39:58.843006+0000","last_undegraded":"2024-06-29T14:39:58.843006+0000","last_fullsized":"2024-06-29T14:39:58.843006+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:32.719194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.847989+0000","last_change":"2024-06-29T14:39:56.838836+0000","last_active":"2024-06-29T14:39:58.847989+0000","last_peered":"2024-06-29T14:39:58.847989+0000","last_clean":"2024-06-29T14:39:58.847989+0000","last_became_active":"2024-06-29T14:39:56.838514+0000","last_became_peered":"2024-06-29T14:39:56.838514+0000","last_unstale":"2024-06-29T14:39:58.847989+0000","last_undegraded":"2024-06-29T14:39:58.847989+0000","last_fullsized":"2024-06-29T14:39:58.847989+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:03:02.044394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842826+0000","last_change":"2024-06-29T14:39:57.839242+0000","last_active":"2024-06-29T14:39:58.842826+0000","last_peered":"2024-06-29T14:39:58.842826+0000","last_clean":"2024-06-29T14:39:58.842826+0000","last_became_active":"2024-06-29T14:39:57.838968+0000","last_became_peered":"2024-06-29T14:39:57.838968+0000","last_unstale":"2024-06-29T14:39:58.842826+0000","last_undegraded":"2024-06-29T14:39:58.842826+0000","last_fullsized":"2024-06-29T14:39:58.842826+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:18:24.334437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040148+0000","last_change":"2024-06-29T14:39:56.825735+0000","last_active":"2024-06-29T14:39:59.040148+0000","last_peered":"2024-06-29T14:39:59.040148+0000","last_clean":"2024-06-29T14:39:59.040148+0000","last_became_active":"2024-06-29T14:39:56.825208+0000","last_became_peered":"2024-06-29T14:39:56.825208+0000","last_unstale":"2024-06-29T14:39:59.040148+0000","last_undegraded":"2024-06-29T14:39:59.040148+0000","last_fullsized":"2024-06-29T14:39:59.040148+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:16:43.529532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842501+0000","last_change":"2024-06-29T14:39:57.844218+0000","last_active":"2024-06-29T14:39:58.842501+0000","last_peered":"2024-06-29T14:39:58.842501+0000","last_clean":"2024-06-29T14:39:58.842501+0000","last_became_active":"2024-06-29T14:39:57.843944+0000","last_became_peered":"2024-06-29T14:39:57.843944+0000","last_unstale":"2024-06-29T14:39:58.842501+0000","last_undegraded":"2024-06-29T14:39:58.842501+0000","last_fullsized":"2024-06-29T14:39:58.842501+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:55.324776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039779+0000","last_change":"2024-06-29T14:39:56.826709+0000","last_active":"2024-06-29T14:39:59.039779+0000","last_peered":"2024-06-29T14:39:59.039779+0000","last_clean":"2024-06-29T14:39:59.039779+0000","last_became_active":"2024-06-29T14:39:56.826391+0000","last_became_peered":"2024-06-29T14:39:56.826391+0000","last_unstale":"2024-06-29T14:39:59.039779+0000","last_undegraded":"2024-06-29T14:39:59.039779+0000","last_fullsized":"2024-06-29T14:39:59.039779+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:00.808223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843178+0000","last_change":"2024-06-29T14:39:57.849821+0000","last_active":"2024-06-29T14:39:58.843178+0000","last_peered":"2024-06-29T14:39:58.843178+0000","last_clean":"2024-06-29T14:39:58.843178+0000","last_became_active":"2024-06-29T14:39:57.849481+0000","last_became_peered":"2024-06-29T14:39:57.849481+0000","last_unstale":"2024-06-29T14:39:58.843178+0000","last_undegraded":"2024-06-29T14:39:58.843178+0000","last_fullsized":"2024-06-29T14:39:58.843178+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:55:38.416779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848067+0000","last_change":"2024-06-29T14:39:56.838733+0000","last_active":"2024-06-29T14:39:58.848067+0000","last_peered":"2024-06-29T14:39:58.848067+0000","last_clean":"2024-06-29T14:39:58.848067+0000","last_became_active":"2024-06-29T14:39:56.838422+0000","last_became_peered":"2024-06-29T14:39:56.838422+0000","last_unstale":"2024-06-29T14:39:58.848067+0000","last_undegraded":"2024-06-29T14:39:58.848067+0000","last_fullsized":"2024-06-29T14:39:58.848067+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:32.469239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848468+0000","last_change":"2024-06-29T14:39:57.837764+0000","last_active":"2024-06-29T14:39:58.848468+0000","last_peered":"2024-06-29T14:39:58.848468+0000","last_clean":"2024-06-29T14:39:58.848468+0000","last_became_active":"2024-06-29T14:39:57.837449+0000","last_became_peered":"2024-06-29T14:39:57.837449+0000","last_unstale":"2024-06-29T14:39:58.848468+0000","last_undegraded":"2024-06-29T14:39:58.848468+0000","last_fullsized":"2024-06-29T14:39:58.848468+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:08:16.837391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"118'2","reported_seq":16,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040480+0000","last_change":"2024-06-29T14:39:56.830999+0000","last_active":"2024-06-29T14:39:59.040480+0000","last_peered":"2024-06-29T14:39:59.040480+0000","last_clean":"2024-06-29T14:39:59.040480+0000","last_became_active":"2024-06-29T14:39:56.827267+0000","last_became_peered":"2024-06-29T14:39:56.827267+0000","last_unstale":"2024-06-29T14:39:59.040480+0000","last_undegraded":"2024-06-29T14:39:59.040480+0000","last_fullsized":"2024-06-29T14:39:59.040480+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:25:37.344207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.040026+0000","last_change":"2024-06-29T14:39:57.837088+0000","last_active":"2024-06-29T14:39:59.040026+0000","last_peered":"2024-06-29T14:39:59.040026+0000","last_clean":"2024-06-29T14:39:59.040026+0000","last_became_active":"2024-06-29T14:39:57.836740+0000","last_became_peered":"2024-06-29T14:39:57.836740+0000","last_unstale":"2024-06-29T14:39:59.040026+0000","last_undegraded":"2024-06-29T14:39:59.040026+0000","last_fullsized":"2024-06-29T14:39:59.040026+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:03:03.049445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"119'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.849498+0000","last_change":"2024-06-29T14:39:56.834829+0000","last_active":"2024-06-29T14:39:58.849498+0000","last_peered":"2024-06-29T14:39:58.849498+0000","last_clean":"2024-06-29T14:39:58.849498+0000","last_became_active":"2024-06-29T14:39:56.834544+0000","last_became_peered":"2024-06-29T14:39:56.834544+0000","last_unstale":"2024-06-29T14:39:58.849498+0000","last_undegraded":"2024-06-29T14:39:58.849498+0000","last_fullsized":"2024-06-29T14:39:58.849498+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:08:38.030321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039815+0000","last_change":"2024-06-29T14:39:57.848707+0000","last_active":"2024-06-29T14:39:59.039815+0000","last_peered":"2024-06-29T14:39:59.039815+0000","last_clean":"2024-06-29T14:39:59.039815+0000","last_became_active":"2024-06-29T14:39:57.848313+0000","last_became_peered":"2024-06-29T14:39:57.848313+0000","last_unstale":"2024-06-29T14:39:59.039815+0000","last_undegraded":"2024-06-29T14:39:59.039815+0000","last_fullsized":"2024-06-29T14:39:59.039815+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:32.719194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848074+0000","last_change":"2024-06-29T14:39:56.838772+0000","last_active":"2024-06-29T14:39:58.848074+0000","last_peered":"2024-06-29T14:39:58.848074+0000","last_clean":"2024-06-29T14:39:58.848074+0000","last_became_active":"2024-06-29T14:39:56.838409+0000","last_became_peered":"2024-06-29T14:39:56.838409+0000","last_unstale":"2024-06-29T14:39:58.848074+0000","last_undegraded":"2024-06-29T14:39:58.848074+0000","last_fullsized":"2024-06-29T14:39:58.848074+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:32.330042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843203+0000","last_change":"2024-06-29T14:39:57.857016+0000","last_active":"2024-06-29T14:39:58.843203+0000","last_peered":"2024-06-29T14:39:58.843203+0000","last_clean":"2024-06-29T14:39:58.843203+0000","last_became_active":"2024-06-29T14:39:57.856695+0000","last_became_peered":"2024-06-29T14:39:57.856695+0000","last_unstale":"2024-06-29T14:39:58.843203+0000","last_undegraded":"2024-06-29T14:39:58.843203+0000","last_fullsized":"2024-06-29T14:39:58.843203+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:15:02.147328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843921+0000","last_change":"2024-06-29T14:39:56.833503+0000","last_active":"2024-06-29T14:39:58.843921+0000","last_peered":"2024-06-29T14:39:58.843921+0000","last_clean":"2024-06-29T14:39:58.843921+0000","last_became_active":"2024-06-29T14:39:56.833014+0000","last_became_peered":"2024-06-29T14:39:56.833014+0000","last_unstale":"2024-06-29T14:39:58.843921+0000","last_undegraded":"2024-06-29T14:39:58.843921+0000","last_fullsized":"2024-06-29T14:39:58.843921+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:31.714143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843064+0000","last_change":"2024-06-29T14:39:57.841870+0000","last_active":"2024-06-29T14:39:58.843064+0000","last_peered":"2024-06-29T14:39:58.843064+0000","last_clean":"2024-06-29T14:39:58.843064+0000","last_became_active":"2024-06-29T14:39:57.841567+0000","last_became_peered":"2024-06-29T14:39:57.841567+0000","last_unstale":"2024-06-29T14:39:58.843064+0000","last_undegraded":"2024-06-29T14:39:58.843064+0000","last_fullsized":"2024-06-29T14:39:58.843064+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:33.335092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.848043+0000","last_change":"2024-06-29T14:39:56.834107+0000","last_active":"2024-06-29T14:39:58.848043+0000","last_peered":"2024-06-29T14:39:58.848043+0000","last_clean":"2024-06-29T14:39:58.848043+0000","last_became_active":"2024-06-29T14:39:56.833794+0000","last_became_peered":"2024-06-29T14:39:56.833794+0000","last_unstale":"2024-06-29T14:39:58.848043+0000","last_undegraded":"2024-06-29T14:39:58.848043+0000","last_fullsized":"2024-06-29T14:39:58.848043+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:44:52.405450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039751+0000","last_change":"2024-06-29T14:39:57.849141+0000","last_active":"2024-06-29T14:39:59.039751+0000","last_peered":"2024-06-29T14:39:59.039751+0000","last_clean":"2024-06-29T14:39:59.039751+0000","last_became_active":"2024-06-29T14:39:57.848649+0000","last_became_peered":"2024-06-29T14:39:57.848649+0000","last_unstale":"2024-06-29T14:39:59.039751+0000","last_undegraded":"2024-06-29T14:39:59.039751+0000","last_fullsized":"2024-06-29T14:39:59.039751+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:22.094937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843572+0000","last_change":"2024-06-29T14:39:56.832250+0000","last_active":"2024-06-29T14:39:58.843572+0000","last_peered":"2024-06-29T14:39:58.843572+0000","last_clean":"2024-06-29T14:39:58.843572+0000","last_became_active":"2024-06-29T14:39:56.831909+0000","last_became_peered":"2024-06-29T14:39:56.831909+0000","last_unstale":"2024-06-29T14:39:58.843572+0000","last_undegraded":"2024-06-29T14:39:58.843572+0000","last_fullsized":"2024-06-29T14:39:58.843572+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:42.641827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842499+0000","last_change":"2024-06-29T14:39:57.841144+0000","last_active":"2024-06-29T14:39:58.842499+0000","last_peered":"2024-06-29T14:39:58.842499+0000","last_clean":"2024-06-29T14:39:58.842499+0000","last_became_active":"2024-06-29T14:39:57.840869+0000","last_became_peered":"2024-06-29T14:39:57.840869+0000","last_unstale":"2024-06-29T14:39:58.842499+0000","last_undegraded":"2024-06-29T14:39:58.842499+0000","last_fullsized":"2024-06-29T14:39:58.842499+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:46:51.624872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:59.039397+0000","last_change":"2024-06-29T14:39:56.831213+0000","last_active":"2024-06-29T14:39:59.039397+0000","last_peered":"2024-06-29T14:39:59.039397+0000","last_clean":"2024-06-29T14:39:59.039397+0000","last_became_active":"2024-06-29T14:39:56.830908+0000","last_became_peered":"2024-06-29T14:39:56.830908+0000","last_unstale":"2024-06-29T14:39:59.039397+0000","last_undegraded":"2024-06-29T14:39:59.039397+0000","last_fullsized":"2024-06-29T14:39:59.039397+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:49:20.393779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843352+0000","last_change":"2024-06-29T14:39:57.854803+0000","last_active":"2024-06-29T14:39:58.843352+0000","last_peered":"2024-06-29T14:39:58.843352+0000","last_clean":"2024-06-29T14:39:58.843352+0000","last_became_active":"2024-06-29T14:39:57.854308+0000","last_became_peered":"2024-06-29T14:39:57.854308+0000","last_unstale":"2024-06-29T14:39:58.843352+0000","last_undegraded":"2024-06-29T14:39:58.843352+0000","last_fullsized":"2024-06-29T14:39:58.843352+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:51:00.345728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.842898+0000","last_change":"2024-06-29T14:39:56.835319+0000","last_active":"2024-06-29T14:39:58.842898+0000","last_peered":"2024-06-29T14:39:58.842898+0000","last_clean":"2024-06-29T14:39:58.842898+0000","last_became_active":"2024-06-29T14:39:56.834975+0000","last_became_peered":"2024-06-29T14:39:56.834975+0000","last_unstale":"2024-06-29T14:39:58.842898+0000","last_undegraded":"2024-06-29T14:39:58.842898+0000","last_fullsized":"2024-06-29T14:39:58.842898+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:24:53.330687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843250+0000","last_change":"2024-06-29T14:39:57.843999+0000","last_active":"2024-06-29T14:39:58.843250+0000","last_peered":"2024-06-29T14:39:58.843250+0000","last_clean":"2024-06-29T14:39:58.843250+0000","last_became_active":"2024-06-29T14:39:57.843707+0000","last_became_peered":"2024-06-29T14:39:57.843707+0000","last_unstale":"2024-06-29T14:39:58.843250+0000","last_undegraded":"2024-06-29T14:39:58.843250+0000","last_fullsized":"2024-06-29T14:39:58.843250+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:56.808281+0000","last_clean_scrub_stamp":"2024-06-29T14:39:56.808281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:08:39.035371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"118'1","reported_seq":15,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.843566+0000","last_change":"2024-06-29T14:39:56.837957+0000","last_active":"2024-06-29T14:39:58.843566+0000","last_peered":"2024-06-29T14:39:58.843566+0000","last_clean":"2024-06-29T14:39:58.843566+0000","last_became_active":"2024-06-29T14:39:56.837577+0000","last_became_peered":"2024-06-29T14:39:56.837577+0000","last_unstale":"2024-06-29T14:39:58.843566+0000","last_undegraded":"2024-06-29T14:39:58.843566+0000","last_fullsized":"2024-06-29T14:39:58.843566+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:39:55.803230+0000","last_clean_scrub_stamp":"2024-06-29T14:39:55.803230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:34.452884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"108'128","reported_seq":501,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-06-29T14:39:58.844015+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:39:58.844015+0000","last_peered":"2024-06-29T14:39:58.844015+0000","last_clean":"2024-06-29T14:39:58.844015+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:39:58.844015+0000","last_undegraded":"2024-06-29T14:39:58.844015+0000","last_fullsized":"2024-06-29T14:39:58.844015+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":197,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":546,"num_read_kb":1463,"num_write":197,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575060,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33208,"kb_used_data":2024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338632,"statfs":{"total":96636764160,"available":96602759168,"internally_reserved":0,"allocated":2072576,"data_stored":1295698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575061,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36788,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335052,"statfs":{"total":96636764160,"available":96599093248,"internally_reserved":0,"allocated":1478656,"data_stored":705870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575059,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36776,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335064,"statfs":{"total":96636764160,"available":96599105536,"internally_reserved":0,"allocated":1462272,"data_stored":698916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575060,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58992,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312848,"statfs":{"total":96636764160,"available":96576356352,"internally_reserved":0,"allocated":1478656,"data_stored":700582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575060,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":1474560,"data_stored":700480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575060,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40948,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330892,"statfs":{"total":96636764160,"available":96594833408,"internally_reserved":0,"allocated":1474560,"data_stored":705252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54812,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317028,"statfs":{"total":96636764160,"available":96580636672,"internally_reserved":0,"allocated":1458176,"data_stored":698846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575060,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":2068480,"data_stored":1296168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:40:02.237 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:40:02.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:40:02.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.397+0000 7fde127e7640 1 -- 172.21.15.81:0/733106515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde0c0b6eb0 msgr2=0x7fde0c0d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.397+0000 7fde127e7640 1 --2- 172.21.15.81:0/733106515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde0c0b6eb0 0x7fde0c0d02d0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009830 tx=0x7fddfc02a480 comp rx=0 tx=0).stop 2024-06-29T14:40:02.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.397+0000 7fde127e7640 1 -- 172.21.15.81:0/733106515 shutdown_connections 2024-06-29T14:40:02.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.397+0000 7fde127e7640 1 --2- 172.21.15.81:0/733106515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde0c0b6eb0 0x7fde0c0d02d0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.397+0000 7fde127e7640 1 -- 172.21.15.81:0/733106515 >> 172.21.15.81:0/733106515 conn(0x7fde0c0b3510 msgr2=0x7fde0c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.397+0000 7fde127e7640 1 -- 172.21.15.81:0/733106515 shutdown_connections 2024-06-29T14:40:02.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.397+0000 7fde127e7640 1 -- 172.21.15.81:0/733106515 wait complete. 2024-06-29T14:40:02.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.399+0000 7fde127e7640 1 Processor -- start 2024-06-29T14:40:02.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.399+0000 7fde127e7640 1 -- start start 2024-06-29T14:40:02.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde127e7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde0c0b6eb0 0x7fde0c09c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde127e7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde0c0ce510 0x7fde0c1d7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde127e7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde0c0ce920 0x7fde0c09f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde127e7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c0cd600 con 0x7fde0c0ce510 2024-06-29T14:40:02.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde127e7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c0cd480 con 0x7fde0c0ce920 2024-06-29T14:40:02.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde127e7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fde0c0cd780 con 0x7fde0c0b6eb0 2024-06-29T14:40:02.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde0b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde0c0ce510 0x7fde0c1d7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde0b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde0c0ce510 0x7fde0c1d7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46288/0 (socket says 172.21.15.81:46288) 2024-06-29T14:40:02.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde0b7fe640 1 -- 172.21.15.81:0/1914220657 learned_addr learned my addr 172.21.15.81:0/1914220657 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde0bfff640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde0c0b6eb0 0x7fde0c09c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.400+0000 7fde10d84640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde0c0ce920 0x7fde0c09f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde0b7fe640 1 -- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde0c0b6eb0 msgr2=0x7fde0c09c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde0b7fe640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde0c0b6eb0 0x7fde0c09c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde0b7fe640 1 -- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde0c0ce920 msgr2=0x7fde0c09f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde0b7fe640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde0c0ce920 0x7fde0c09f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde0b7fe640 1 -- 172.21.15.81:0/1914220657 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0c1d8040 con 0x7fde0c0ce510 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde10d84640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde0c0ce920 0x7fde0c09f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde0bfff640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde0c0b6eb0 0x7fde0c09c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde0b7fe640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde0c0ce510 0x7fde0c1d7b00 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fde0000b420 tx=0x7fde0000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde08ff9640 1 -- 172.21.15.81:0/1914220657 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00011830 con 0x7fde0c0ce510 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde08ff9640 1 -- 172.21.15.81:0/1914220657 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde00016070 con 0x7fde0c0ce510 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde08ff9640 1 -- 172.21.15.81:0/1914220657 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00011d60 con 0x7fde0c0ce510 2024-06-29T14:40:02.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde127e7640 1 -- 172.21.15.81:0/1914220657 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde0c09cca0 con 0x7fde0c0ce510 2024-06-29T14:40:02.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.401+0000 7fde127e7640 1 -- 172.21.15.81:0/1914220657 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde0c09e950 con 0x7fde0c0ce510 2024-06-29T14:40:02.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.402+0000 7fde127e7640 1 -- 172.21.15.81:0/1914220657 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde0c09ca10 con 0x7fde0c0ce510 2024-06-29T14:40:02.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.403+0000 7fde08ff9640 1 -- 172.21.15.81:0/1914220657 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fde0001b460 con 0x7fde0c0ce510 2024-06-29T14:40:02.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.403+0000 7fde08ff9640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fddf80763f0 0x7fddf8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.403+0000 7fde08ff9640 1 -- 172.21.15.81:0/1914220657 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7fde000974b0 con 0x7fde0c0ce510 2024-06-29T14:40:02.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.403+0000 7fde0bfff640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fddf80763f0 0x7fddf8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.404+0000 7fde0bfff640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fddf80763f0 0x7fddf8078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fddfc034000 tx=0x7fddfc004730 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.408+0000 7fde08ff9640 1 -- 172.21.15.81:0/1914220657 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fde00020020 con 0x7fde0c0ce510 2024-06-29T14:40:02.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.662+0000 7fde127e7640 1 -- 172.21.15.81:0/1914220657 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fde0c0b5210 con 0x7fde0c0ce510 2024-06-29T14:40:02.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.665+0000 7fde08ff9640 1 -- 172.21.15.81:0/1914220657 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v35) ==== 190+0+147 (secure 0 0 0) 0x7fde00060c20 con 0x7fde0c0ce510 2024-06-29T14:40:02.662 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T14:40:02.662 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T14:40:02.663 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T14:40:02.663 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T14:40:02.663 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:40:02.663 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T14:40:02.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.667+0000 7fde127e7640 1 -- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fddf80763f0 msgr2=0x7fddf8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.667+0000 7fde127e7640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fddf80763f0 0x7fddf8078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fddfc034000 tx=0x7fddfc004730 comp rx=0 tx=0).stop 2024-06-29T14:40:02.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.667+0000 7fde127e7640 1 -- 172.21.15.81:0/1914220657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde0c0ce510 msgr2=0x7fde0c1d7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.667+0000 7fde127e7640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde0c0ce510 0x7fde0c1d7b00 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fde0000b420 tx=0x7fde0000b8e0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.667+0000 7fde127e7640 1 -- 172.21.15.81:0/1914220657 shutdown_connections 2024-06-29T14:40:02.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.667+0000 7fde127e7640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fddf80763f0 0x7fddf8078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.667+0000 7fde127e7640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde0c0ce920 0x7fde0c09f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.667+0000 7fde127e7640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde0c0ce510 0x7fde0c1d7b00 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.667+0000 7fde127e7640 1 --2- 172.21.15.81:0/1914220657 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde0c0b6eb0 0x7fde0c09c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.667+0000 7fde127e7640 1 -- 172.21.15.81:0/1914220657 >> 172.21.15.81:0/1914220657 conn(0x7fde0c0b3510 msgr2=0x7fde0c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.668+0000 7fde127e7640 1 -- 172.21.15.81:0/1914220657 shutdown_connections 2024-06-29T14:40:02.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.668+0000 7fde127e7640 1 -- 172.21.15.81:0/1914220657 wait complete. 2024-06-29T14:40:02.676 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:40:02.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.791+0000 7ff58852a640 1 -- 172.21.15.81:0/3436996556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5800cd480 msgr2=0x7ff5800ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.791+0000 7ff58852a640 1 --2- 172.21.15.81:0/3436996556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5800cd480 0x7ff5800ce510 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7ff570009830 tx=0x7ff57002a480 comp rx=0 tx=0).stop 2024-06-29T14:40:02.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.791+0000 7ff58852a640 1 -- 172.21.15.81:0/3436996556 shutdown_connections 2024-06-29T14:40:02.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.791+0000 7ff58852a640 1 --2- 172.21.15.81:0/3436996556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5800cd480 0x7ff5800ce510 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.791+0000 7ff58852a640 1 -- 172.21.15.81:0/3436996556 >> 172.21.15.81:0/3436996556 conn(0x7ff5800b3510 msgr2=0x7ff5800b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:02.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.791+0000 7ff58852a640 1 -- 172.21.15.81:0/3436996556 shutdown_connections 2024-06-29T14:40:02.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.791+0000 7ff58852a640 1 -- 172.21.15.81:0/3436996556 wait complete. 2024-06-29T14:40:02.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.793+0000 7ff58852a640 1 Processor -- start 2024-06-29T14:40:02.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.793+0000 7ff58852a640 1 -- start start 2024-06-29T14:40:02.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff58852a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5800b6eb0 0x7ff5800b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff58852a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5800cd480 0x7ff580153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff58852a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5800cea50 0x7ff580165790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff58852a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff5800b52b0 con 0x7ff5800cea50 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff58852a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff5800b5130 con 0x7ff5800cd480 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff58852a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff580167660 con 0x7ff5800b6eb0 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff586ac7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5800cea50 0x7ff580165790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff586ac7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5800cea50 0x7ff580165790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46314/0 (socket says 172.21.15.81:46314) 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff586ac7640 1 -- 172.21.15.81:0/4273308655 learned_addr learned my addr 172.21.15.81:0/4273308655 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff585ac5640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5800cd480 0x7ff580153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.794+0000 7ff5862c6640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5800b6eb0 0x7ff5800b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff586ac7640 1 -- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5800b6eb0 msgr2=0x7ff5800b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff586ac7640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5800b6eb0 0x7ff5800b14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff586ac7640 1 -- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5800cd480 msgr2=0x7ff580153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:02.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff586ac7640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5800cd480 0x7ff580153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:02.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff586ac7640 1 -- 172.21.15.81:0/4273308655 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff580164f40 con 0x7ff5800cea50 2024-06-29T14:40:02.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff586ac7640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5800cea50 0x7ff580165790 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c3c0 tx=0x7ff57800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff576ffd640 1 -- 172.21.15.81:0/4273308655 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578011790 con 0x7ff5800cea50 2024-06-29T14:40:02.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff576ffd640 1 -- 172.21.15.81:0/4273308655 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff578017070 con 0x7ff5800cea50 2024-06-29T14:40:02.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff58852a640 1 -- 172.21.15.81:0/4273308655 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff580165220 con 0x7ff5800cea50 2024-06-29T14:40:02.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff576ffd640 1 -- 172.21.15.81:0/4273308655 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578011ca0 con 0x7ff5800cea50 2024-06-29T14:40:02.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.795+0000 7ff58852a640 1 -- 172.21.15.81:0/4273308655 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff580154720 con 0x7ff5800cea50 2024-06-29T14:40:02.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.796+0000 7ff58852a640 1 -- 172.21.15.81:0/4273308655 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5800ce510 con 0x7ff5800cea50 2024-06-29T14:40:02.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.797+0000 7ff576ffd640 1 -- 172.21.15.81:0/4273308655 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7ff57801c460 con 0x7ff5800cea50 2024-06-29T14:40:02.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.797+0000 7ff576ffd640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7ff5640763f0 0x7ff564078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:02.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.797+0000 7ff576ffd640 1 -- 172.21.15.81:0/4273308655 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7ff5780977c0 con 0x7ff5800cea50 2024-06-29T14:40:02.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.797+0000 7ff5862c6640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7ff5640763f0 0x7ff564078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:02.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.798+0000 7ff5862c6640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7ff5640763f0 0x7ff564078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009f20 tx=0x7ff57c0094d0 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:02.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:02.802+0000 7ff576ffd640 1 -- 172.21.15.81:0/4273308655 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff578060f30 con 0x7ff5800cea50 2024-06-29T14:40:03.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.022+0000 7ff58852a640 1 -- 172.21.15.81:0/4273308655 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5800cdf00 con 0x7ff5800cea50 2024-06-29T14:40:03.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.022+0000 7ff576ffd640 1 -- 172.21.15.81:0/4273308655 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5231 (secure 0 0 0) 0x7ff578001080 con 0x7ff5800cea50 2024-06-29T14:40:03.020 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:40:03.020 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":59,"btime":"2024-06-29T14:40:00:847465+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1563799167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1563799167},{"type":"v1","addr":"172.21.15.167:6837","nonce":1563799167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7354,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3062579579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3062579579},{"type":"v1","addr":"172.21.15.81:6835","nonce":3062579579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7363,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3427077146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3427077146},{"type":"v1","addr":"172.21.15.167:6836","nonce":3427077146}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.183979+0000","modified":"2024-06-29T14:40:00.217351+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/1049868964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1049868964},{"type":"v1","addr":"172.21.15.81:6833","nonce":1049868964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8022,"qdb_cluster":[8022]},"id":7}]} 2024-06-29T14:40:03.020 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 59 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 -- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7ff5640763f0 msgr2=0x7ff564078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7ff5640763f0 0x7ff564078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009f20 tx=0x7ff57c0094d0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 -- 172.21.15.81:0/4273308655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5800cea50 msgr2=0x7ff580165790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5800cea50 0x7ff580165790 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c3c0 tx=0x7ff57800c880 comp rx=0 tx=0).stop 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 -- 172.21.15.81:0/4273308655 shutdown_connections 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7ff5640763f0 0x7ff564078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5800cea50 0x7ff580165790 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5800cd480 0x7ff580153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 --2- 172.21.15.81:0/4273308655 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5800b6eb0 0x7ff5800b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 -- 172.21.15.81:0/4273308655 >> 172.21.15.81:0/4273308655 conn(0x7ff5800b3510 msgr2=0x7ff580163860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 -- 172.21.15.81:0/4273308655 shutdown_connections 2024-06-29T14:40:03.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.025+0000 7ff58852a640 1 -- 172.21.15.81:0/4273308655 wait complete. 2024-06-29T14:40:03.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:39:57.183979+0000', 'modified': '2024-06-29T14:40:00.217351+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8022}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8022': {'gid': 8022, 'name': 'a', 'rank': 0, 'incarnation': 56, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6833/1049868964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6832', 'nonce': 1049868964}, {'type': 'v1', 'addr': '172.21.15.81:6833', 'nonce': 1049868964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8022, 'qdb_cluster': [8022]} 2024-06-29T14:40:03.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:40:03.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T14:40:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.555+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/2973626470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ef00b4f40 msgr2=0x7f7ef00b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.555+0000 7f7ef7fd2640 1 --2- 172.21.15.81:0/2973626470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ef00b4f40 0x7f7ef00b5350 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0074b10 tx=0x7f7ee002a820 comp rx=0 tx=0).stop 2024-06-29T14:40:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.555+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/2973626470 shutdown_connections 2024-06-29T14:40:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.555+0000 7f7ef7fd2640 1 --2- 172.21.15.81:0/2973626470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ef00b4f40 0x7f7ef00b5350 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.555+0000 7f7ef7fd2640 1 --2- 172.21.15.81:0/2973626470 >> v2:172.21.15.167:3300/0 conn(0x7f7ef00d0230 0x7f7ef00d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.555+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/2973626470 >> 172.21.15.81:0/2973626470 conn(0x7f7ef00b3680 msgr2=0x7f7ef00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.555+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/2973626470 shutdown_connections 2024-06-29T14:40:03.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.555+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/2973626470 wait complete. 2024-06-29T14:40:03.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.557+0000 7f7ef7fd2640 1 Processor -- start 2024-06-29T14:40:03.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.557+0000 7f7ef7fd2640 1 -- start start 2024-06-29T14:40:03.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.558+0000 7f7ef7fd2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ef00b3200 0x7f7ef01f5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.558+0000 7f7ef7fd2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ef00b4f40 0x7f7ef01e1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.558+0000 7f7ef7fd2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ef00d0230 0x7f7ef014be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.558+0000 7f7ef7fd2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef00b5c20 con 0x7f7ef00b3200 2024-06-29T14:40:03.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.558+0000 7f7ef7fd2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef00b5aa0 con 0x7f7ef00d0230 2024-06-29T14:40:03.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.558+0000 7f7ef7fd2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef00b5da0 con 0x7f7ef00b4f40 2024-06-29T14:40:03.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.558+0000 7f7ef5d6e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ef00b3200 0x7f7ef01f5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.558+0000 7f7ef5d6e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ef00b3200 0x7f7ef01f5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46326/0 (socket says 172.21.15.81:46326) 2024-06-29T14:40:03.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.558+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/1375185743 learned_addr learned my addr 172.21.15.81:0/1375185743 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:03.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7ef556d640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ef00b4f40 0x7f7ef01e1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ef00b4f40 msgr2=0x7f7ef01e1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7ef656f640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ef00d0230 0x7f7ef014be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ef00b4f40 0x7f7ef01e1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ef00d0230 msgr2=0x7f7ef014be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ef00d0230 0x7f7ef014be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/1375185743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef01e2af0 con 0x7f7ef00b3200 2024-06-29T14:40:03.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ef00b3200 0x7f7ef01f5b10 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800c760 tx=0x7f7ee800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7eee7fc640 1 -- 172.21.15.81:0/1375185743 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8016020 con 0x7f7ef00b3200 2024-06-29T14:40:03.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7eee7fc640 1 -- 172.21.15.81:0/1375185743 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ee800a950 con 0x7f7ef00b3200 2024-06-29T14:40:03.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/1375185743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef01e2dd0 con 0x7f7ef00b3200 2024-06-29T14:40:03.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7eee7fc640 1 -- 172.21.15.81:0/1375185743 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee800ada0 con 0x7f7ef00b3200 2024-06-29T14:40:03.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.559+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/1375185743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef014a160 con 0x7f7ef00b3200 2024-06-29T14:40:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.561+0000 7f7eee7fc640 1 -- 172.21.15.81:0/1375185743 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f7ee8019020 con 0x7f7ef00b3200 2024-06-29T14:40:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.561+0000 7f7eee7fc640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7edc0763f0 0x7f7edc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.561+0000 7f7eee7fc640 1 -- 172.21.15.81:0/1375185743 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7f7ee8097890 con 0x7f7ef00b3200 2024-06-29T14:40:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.562+0000 7f7ef556d640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7edc0763f0 0x7f7edc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.562+0000 7f7ef556d640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7edc0763f0 0x7f7edc078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4005fd0 tx=0x7f7ee4005850 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.562+0000 7f7efef66740 1 Processor -- start 2024-06-29T14:40:03.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.563+0000 7f7efef66740 1 -- start start 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7efef66740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7efef66740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f163b0 0x55d5932e11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7efef66740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59330ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7efef66740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d5932a7bc0 con 0x55d592f3f270 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7efef66740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55d593256630 con 0x55d592f163b0 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7efef66740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55d593245bd0 con 0x55d592f1e590 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef5d6e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef5d6e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46342/0 (socket says 172.21.15.81:46342) 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/1027025726 learned_addr learned my addr 172.21.15.81:0/1027025726 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef556d640 1 --2- 172.21.15.81:0/1027025726 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f163b0 0x55d5932e11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef656f640 1 --2- 172.21.15.81:0/1027025726 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59330ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/1027025726 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 msgr2=0x55d59330ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/1027025726 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59330ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/1027025726 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f163b0 msgr2=0x55d5932e11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/1027025726 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f163b0 0x55d5932e11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/1027025726 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d593242ca0 con 0x55d592f3f270 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.564+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/1027025726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e38e0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f7ee805cf80 tx=0x7f7ee8007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d0a5fe01444bb1 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7ed77fe640 1 -- 172.21.15.81:0/1027025726 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee80690b0 con 0x55d592f3f270 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7ed77fe640 1 -- 172.21.15.81:0/1027025726 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ee8060990 con 0x55d592f3f270 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7ed77fe640 1 -- 172.21.15.81:0/1027025726 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8064d40 con 0x55d592f3f270 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7efef66740 1 -- 172.21.15.81:0/1027025726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 msgr2=0x55d5932e38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7efef66740 1 --2- 172.21.15.81:0/1027025726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e38e0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f7ee805cf80 tx=0x7f7ee8007e60 comp rx=0 tx=0).stop 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7efef66740 1 -- 172.21.15.81:0/1027025726 shutdown_connections 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7efef66740 1 --2- 172.21.15.81:0/1027025726 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59330ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7efef66740 1 --2- 172.21.15.81:0/1027025726 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f163b0 0x55d5932e11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7efef66740 1 --2- 172.21.15.81:0/1027025726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e38e0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7efef66740 1 -- 172.21.15.81:0/1027025726 >> 172.21.15.81:0/1027025726 conn(0x55d592f333c0 msgr2=0x55d5932e2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7efef66740 1 -- 172.21.15.81:0/1027025726 shutdown_connections 2024-06-29T14:40:03.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.565+0000 7f7efef66740 1 -- 172.21.15.81:0/1027025726 wait complete. 2024-06-29T14:40:03.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.566+0000 7f7efef66740 1 -- start start 2024-06-29T14:40:03.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.566+0000 7f7efef66740 1 Processor -- start 2024-06-29T14:40:03.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7efef66740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d59333e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7efef66740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59331a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7efef66740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 0x55d5932e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7efef66740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d5931e3ae0 con 0x55d592f163b0 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7efef66740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55d593249c10 con 0x55d592f3f270 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7efef66740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55d592fb9620 con 0x55d592f1e590 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7efef66740 10 client.0 _ll_register_callbacks cb 0x55d5931f0480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef5d6e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d59333e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef5d6e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d59333e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46344/0 (socket says 172.21.15.81:46344) 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/995249933 learned_addr learned my addr 172.21.15.81:0/995249933 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef656f640 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 0x55d5932e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef556d640 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59331a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/995249933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 msgr2=0x55d59331a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59331a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/995249933 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 msgr2=0x55d5932e4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:03.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 0x55d5932e4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/995249933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d592faa850 con 0x55d592f163b0 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef656f640 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 0x55d5932e4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d59333e740 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8061240 tx=0x7f7ee8095bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32aa37ffbfdce745 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.568+0000 7f7ed4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/995249933 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee80690b0 con 0x55d592f163b0 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/995249933 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7f7ee805fbf0 con 0x55d592f163b0 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7efef66740 1 -- 172.21.15.81:0/995249933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee8060990 con 0x55d592f163b0 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/995249933 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f7ee8069760 con 0x55d592f163b0 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7efef66740 10 client.8184 fetch_fsmap learned FSMap version 59 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7efef66740 1 -- 172.21.15.81:0/995249933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) -- 0x55d5930216b0 con 0x55d592f163b0 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/995249933 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ee8060990 con 0x55d592f163b0 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/995249933 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 59) ==== 3502+0+0 (secure 0 0 0) 0x7f7ee805c300 con 0x55d592f163b0 2024-06-29T14:40:03.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7efef66740 10 client.8184 fetch_fsmap finished waiting for FSMap version 59 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7efef66740 10 client.8184 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7efef66740 20 client.8184 populate_metadata read hostname 'smithi081' 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7efef66740 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.81:6832/1049868964,v1:172.21.15.81:6833/1049868964] conn(0x55d592efde40 0x55d5933196a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7efef66740 4 client.8184 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7efef66740 1 -- 172.21.15.81:0/995249933 --> [v2:172.21.15.81:6832/1049868964,v1:172.21.15.81:6833/1049868964] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d593258580 con 0x55d592efde40 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.569+0000 7f7ef556d640 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.81:6832/1049868964,v1:172.21.15.81:6833/1049868964] conn(0x55d592efde40 0x55d5933196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.570+0000 7f7ed4ff9640 10 client.8184 ms_handle_connect on v2:172.21.15.81:6832/1049868964 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.570+0000 7f7ef556d640 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.81:6832/1049868964,v1:172.21.15.81:6833/1049868964] conn(0x55d592efde40 0x55d5933196a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78fcb2d94892c3b8 server_cookie=84c5c30893fa0db1 in_seq=0 out_seq=0 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.570+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/995249933 <== mds.0 v2:172.21.15.81:6832/1049868964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d593258580 con 0x55d592efde40 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.570+0000 7f7ed4ff9640 10 client.8184 handle_command_reply: tid=0 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.570+0000 7f7efef66740 1 client.8184 shutdown 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.570+0000 7f7efef66740 1 -- 172.21.15.81:0/995249933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 msgr2=0x55d59333e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.570+0000 7f7efef66740 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d59333e740 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8061240 tx=0x7f7ee8095bc0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.570+0000 7f7efef66740 1 -- 172.21.15.81:0/995249933 shutdown_connections 2024-06-29T14:40:03.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.571+0000 7f7efef66740 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 0x55d5932e4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.571+0000 7f7efef66740 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59331a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.571+0000 7f7efef66740 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.81:6832/1049868964,v1:172.21.15.81:6833/1049868964] conn(0x55d592efde40 0x55d5933196a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.571+0000 7f7efef66740 1 --2- 172.21.15.81:0/995249933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d59333e740 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.571+0000 7f7ed4ff9640 0 client.8184 ms_handle_reset on v2:172.21.15.81:6832/1049868964 2024-06-29T14:40:03.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.571+0000 7f7efef66740 1 -- 172.21.15.81:0/995249933 >> 172.21.15.81:0/995249933 conn(0x55d592f333c0 msgr2=0x55d5932e6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.571+0000 7f7efef66740 1 -- 172.21.15.81:0/995249933 shutdown_connections 2024-06-29T14:40:03.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.571+0000 7f7efef66740 1 -- 172.21.15.81:0/995249933 wait complete. 2024-06-29T14:40:03.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.571+0000 7f7efef66740 20 client.8184 trim_cache size 0 max 16384 2024-06-29T14:40:03.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.589+0000 7f7efef66740 1 Processor -- start 2024-06-29T14:40:03.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.590+0000 7f7efef66740 1 -- start start 2024-06-29T14:40:03.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7efef66740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f163b0 0x55d593329a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7efef66740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f1e590 0x55d5932deb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7efef66740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7efef66740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef006e610 con 0x55d592f3f270 2024-06-29T14:40:03.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7efef66740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef01f6f40 con 0x55d592f1e590 2024-06-29T14:40:03.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7efef66740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef014b4e0 con 0x55d592f163b0 2024-06-29T14:40:03.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef656f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef656f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46356/0 (socket says 172.21.15.81:46356) 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef656f640 1 -- 172.21.15.81:0/2050783323 learned_addr learned my addr 172.21.15.81:0/2050783323 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef556d640 1 --2- 172.21.15.81:0/2050783323 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f1e590 0x55d5932deb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/2050783323 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f163b0 0x55d593329a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef656f640 1 -- 172.21.15.81:0/2050783323 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f163b0 msgr2=0x55d593329a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef656f640 1 --2- 172.21.15.81:0/2050783323 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f163b0 0x55d593329a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef656f640 1 -- 172.21.15.81:0/2050783323 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f1e590 msgr2=0x55d5932deb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef656f640 1 --2- 172.21.15.81:0/2050783323 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f1e590 0x55d5932deb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef656f640 1 -- 172.21.15.81:0/2050783323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d593351ce0 con 0x55d592f3f270 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/2050783323 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f163b0 0x55d593329a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ef656f640 1 --2- 172.21.15.81:0/2050783323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e1450 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009ca0 tx=0x7f7ee0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c793ebe3e27b391d server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ed6ffd640 1 -- 172.21.15.81:0/2050783323 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0043030 con 0x55d592f3f270 2024-06-29T14:40:03.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.591+0000 7f7ed6ffd640 1 -- 172.21.15.81:0/2050783323 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ee0038040 con 0x55d592f3f270 2024-06-29T14:40:03.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.592+0000 7f7ed6ffd640 1 -- 172.21.15.81:0/2050783323 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0037040 con 0x55d592f3f270 2024-06-29T14:40:03.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.592+0000 7f7efef66740 1 -- 172.21.15.81:0/2050783323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 msgr2=0x55d5932e1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.592+0000 7f7efef66740 1 --2- 172.21.15.81:0/2050783323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e1450 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009ca0 tx=0x7f7ee0004a40 comp rx=0 tx=0).stop 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.592+0000 7f7efef66740 1 -- 172.21.15.81:0/2050783323 shutdown_connections 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.592+0000 7f7efef66740 1 --2- 172.21.15.81:0/2050783323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f3f270 0x55d5932e1450 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.592+0000 7f7efef66740 1 --2- 172.21.15.81:0/2050783323 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f1e590 0x55d5932deb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.592+0000 7f7efef66740 1 --2- 172.21.15.81:0/2050783323 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f163b0 0x55d593329a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.592+0000 7f7efef66740 1 -- 172.21.15.81:0/2050783323 >> 172.21.15.81:0/2050783323 conn(0x55d592f333c0 msgr2=0x55d593347530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.592+0000 7f7efef66740 1 -- 172.21.15.81:0/2050783323 shutdown_connections 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.592+0000 7f7efef66740 1 -- 172.21.15.81:0/2050783323 wait complete. 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.593+0000 7f7efef66740 1 -- start start 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.593+0000 7f7efef66740 1 Processor -- start 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7efef66740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d593328ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7efef66740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59334dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7efef66740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 0x55d5933a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7efef66740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d5931e3ae0 con 0x55d592f163b0 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7efef66740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55d592fb9620 con 0x55d592f3f270 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7efef66740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55d593249c10 con 0x55d592f1e590 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7efef66740 10 client.0 _ll_register_callbacks cb 0x7f7ee4015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef5d6e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d593328ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef5d6e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d593328ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46364/0 (socket says 172.21.15.81:46364) 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/511914246 learned_addr learned my addr 172.21.15.81:0/511914246 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef556d640 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59334dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef656f640 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 0x55d5933a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/511914246 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 msgr2=0x55d59334dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:03.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59334dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/511914246 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 msgr2=0x55d5933a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 0x55d5933a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef5d6e640 1 -- 172.21.15.81:0/511914246 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d59330b8c0 con 0x55d592f163b0 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ef5d6e640 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d593328ff0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800c730 tx=0x7f7ee8064e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1225a01affce6d12 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ed4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/511914246 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee807c030 con 0x55d592f163b0 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/511914246 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7f7ee805fe00 con 0x55d592f163b0 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7efef66740 1 -- 172.21.15.81:0/511914246 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee0038040 con 0x55d592f163b0 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/511914246 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f7ee805c850 con 0x55d592f163b0 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7efef66740 10 client.8190 fetch_fsmap learned FSMap version 59 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.595+0000 7f7efef66740 1 -- 172.21.15.81:0/511914246 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=2+}) -- 0x55d59330b8c0 con 0x55d592f163b0 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/511914246 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ee0038040 con 0x55d592f163b0 2024-06-29T14:40:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/511914246 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 59) ==== 3502+0+0 (secure 0 0 0) 0x7f7ee805f660 con 0x55d592f163b0 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7efef66740 10 client.8190 fetch_fsmap finished waiting for FSMap version 59 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7efef66740 10 client.8190 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7efef66740 20 client.8190 populate_metadata read hostname 'smithi081' 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7efef66740 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.81:6832/1049868964,v1:172.21.15.81:6833/1049868964] conn(0x55d59332b1c0 0x55d593312530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7efef66740 4 client.8190 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7efef66740 1 -- 172.21.15.81:0/511914246 --> [v2:172.21.15.81:6832/1049868964,v1:172.21.15.81:6833/1049868964] -- command(tid 0: {"prefix": "status"}) -- 0x55d593254020 con 0x55d59332b1c0 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7ef556d640 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.81:6832/1049868964,v1:172.21.15.81:6833/1049868964] conn(0x55d59332b1c0 0x55d593312530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7ed4ff9640 10 client.8190 ms_handle_connect on v2:172.21.15.81:6832/1049868964 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7ef556d640 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.81:6832/1049868964,v1:172.21.15.81:6833/1049868964] conn(0x55d59332b1c0 0x55d593312530 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c6a0ae29a3ac46c server_cookie=fc953d75e19afb55 in_seq=0 out_seq=0 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7ed4ff9640 1 -- 172.21.15.81:0/511914246 <== mds.0 v2:172.21.15.81:6832/1049868964 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x55d593254020 con 0x55d59332b1c0 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7ed4ff9640 10 client.8190 handle_command_reply: tid=0 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.596+0000 7f7efef66740 1 client.8190 shutdown 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 1 -- 172.21.15.81:0/511914246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 msgr2=0x55d593328ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d593328ff0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800c730 tx=0x7f7ee8064e90 comp rx=0 tx=0).stop 2024-06-29T14:40:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 1 -- 172.21.15.81:0/511914246 shutdown_connections 2024-06-29T14:40:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55d592f3f270 0x55d5933a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55d592f1e590 0x55d59334dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.81:6832/1049868964,v1:172.21.15.81:6833/1049868964] conn(0x55d59332b1c0 0x55d593312530 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 1 --2- 172.21.15.81:0/511914246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d592f163b0 0x55d593328ff0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ed4ff9640 0 client.8190 ms_handle_reset on v2:172.21.15.81:6832/1049868964 2024-06-29T14:40:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 1 -- 172.21.15.81:0/511914246 >> 172.21.15.81:0/511914246 conn(0x55d592f333c0 msgr2=0x55d59330bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 1 -- 172.21.15.81:0/511914246 shutdown_connections 2024-06-29T14:40:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 1 -- 172.21.15.81:0/511914246 wait complete. 2024-06-29T14:40:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7efef66740 20 client.8190 trim_cache size 0 max 16384 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 8022, 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7615140650000001, 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 59, 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 119, 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 119, 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.447114430999999 2024-06-29T14:40:03.601 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:40:03.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7edc0763f0 msgr2=0x7f7edc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7edc0763f0 0x7f7edc078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4005fd0 tx=0x7f7ee4005850 comp rx=0 tx=0).stop 2024-06-29T14:40:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/1375185743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ef00b3200 msgr2=0x7f7ef01f5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ef00b3200 0x7f7ef01f5b10 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800c760 tx=0x7f7ee800cc20 comp rx=0 tx=0).stop 2024-06-29T14:40:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/1375185743 shutdown_connections 2024-06-29T14:40:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7edc0763f0 0x7f7edc078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ef00d0230 0x7f7ef014be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ef00b4f40 0x7f7ef01e1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 --2- 172.21.15.81:0/1375185743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ef00b3200 0x7f7ef01f5b10 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/1375185743 >> 172.21.15.81:0/1375185743 conn(0x7f7ef00b3680 msgr2=0x7f7ef0071050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/1375185743 shutdown_connections 2024-06-29T14:40:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.597+0000 7f7ef7fd2640 1 -- 172.21.15.81:0/1375185743 wait complete. 2024-06-29T14:40:03.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:40:03.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.721+0000 7f66b55d2640 1 -- 172.21.15.81:0/2073111841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b006f850 msgr2=0x7f66b00ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.721+0000 7f66b55d2640 1 --2- 172.21.15.81:0/2073111841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b006f850 0x7f66b00ae300 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f6698009830 tx=0x7f669802a510 comp rx=0 tx=0).stop 2024-06-29T14:40:03.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.721+0000 7f66b55d2640 1 -- 172.21.15.81:0/2073111841 shutdown_connections 2024-06-29T14:40:03.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.721+0000 7f66b55d2640 1 --2- 172.21.15.81:0/2073111841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66b00ae840 0x7f66b00aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.721+0000 7f66b55d2640 1 --2- 172.21.15.81:0/2073111841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b006f850 0x7f66b00ae300 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.721+0000 7f66b55d2640 1 -- 172.21.15.81:0/2073111841 >> 172.21.15.81:0/2073111841 conn(0x7f66b0072950 msgr2=0x7f66b00b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.721+0000 7f66b55d2640 1 -- 172.21.15.81:0/2073111841 shutdown_connections 2024-06-29T14:40:03.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.721+0000 7f66b55d2640 1 -- 172.21.15.81:0/2073111841 wait complete. 2024-06-29T14:40:03.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.723+0000 7f66b55d2640 1 Processor -- start 2024-06-29T14:40:03.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.723+0000 7f66b55d2640 1 -- start start 2024-06-29T14:40:03.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66b55d2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b006f850 0x7f66b0156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66b55d2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66b0074520 0x7f66b01f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66b55d2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66b00ae840 0x7f66b01550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66b55d2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f66b015b190 con 0x7f66b006f850 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66b55d2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f66b015b010 con 0x7f66b00ae840 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66b55d2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f66b00af190 con 0x7f66b0074520 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66aeffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b006f850 0x7f66b0156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66aeffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b006f850 0x7f66b0156db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46380/0 (socket says 172.21.15.81:46380) 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66aeffd640 1 -- 172.21.15.81:0/4266339171 learned_addr learned my addr 172.21.15.81:0/4266339171 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66a67fc640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66b0074520 0x7f66b01f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66af7fe640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66b00ae840 0x7f66b01550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66aeffd640 1 -- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66b0074520 msgr2=0x7f66b01f2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66aeffd640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66b0074520 0x7f66b01f2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66aeffd640 1 -- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66b00ae840 msgr2=0x7f66b01550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66aeffd640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66b00ae840 0x7f66b01550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66aeffd640 1 -- 172.21.15.81:0/4266339171 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b0154030 con 0x7f66b006f850 2024-06-29T14:40:03.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66a67fc640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66b0074520 0x7f66b01f2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:03.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66aeffd640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b006f850 0x7f66b0156db0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f66a000c3c0 tx=0x7f66a000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66a7fff640 1 -- 172.21.15.81:0/4266339171 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0017070 con 0x7f66b006f850 2024-06-29T14:40:03.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66a7fff640 1 -- 172.21.15.81:0/4266339171 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66a00035f0 con 0x7f66b006f850 2024-06-29T14:40:03.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66b55d2640 1 -- 172.21.15.81:0/4266339171 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b0154310 con 0x7f66b006f850 2024-06-29T14:40:03.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66a7fff640 1 -- 172.21.15.81:0/4266339171 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0003a80 con 0x7f66b006f850 2024-06-29T14:40:03.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.725+0000 7f66b55d2640 1 -- 172.21.15.81:0/4266339171 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b01548d0 con 0x7f66b006f850 2024-06-29T14:40:03.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.726+0000 7f66b55d2640 1 -- 172.21.15.81:0/4266339171 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b00ae300 con 0x7f66b006f850 2024-06-29T14:40:03.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.727+0000 7f66a7fff640 1 -- 172.21.15.81:0/4266339171 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f66a001a070 con 0x7f66b006f850 2024-06-29T14:40:03.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.728+0000 7f66a7fff640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f66900763f0 0x7f6690078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:03.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.728+0000 7f66a7fff640 1 -- 172.21.15.81:0/4266339171 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7f66a0096db0 con 0x7f66b006f850 2024-06-29T14:40:03.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.728+0000 7f66a67fc640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f66900763f0 0x7f6690078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:03.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.729+0000 7f66a67fc640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f66900763f0 0x7f6690078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6698034000 tx=0x7f6698004730 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:03.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.732+0000 7f66a7fff640 1 -- 172.21.15.81:0/4266339171 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f66a00604a0 con 0x7f66b006f850 2024-06-29T14:40:03.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.952+0000 7f66b55d2640 1 -- 172.21.15.81:0/4266339171 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f66b01582c0 con 0x7f66b006f850 2024-06-29T14:40:03.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.952+0000 7f66a7fff640 1 -- 172.21.15.81:0/4266339171 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v59) ==== 53+0+73 (secure 0 0 0) 0x7f66a005fda0 con 0x7f66b006f850 2024-06-29T14:40:03.950 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 -- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f66900763f0 msgr2=0x7f6690078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f66900763f0 0x7f6690078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6698034000 tx=0x7f6698004730 comp rx=0 tx=0).stop 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 -- 172.21.15.81:0/4266339171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b006f850 msgr2=0x7f66b0156db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b006f850 0x7f66b0156db0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f66a000c3c0 tx=0x7f66a000c880 comp rx=0 tx=0).stop 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 -- 172.21.15.81:0/4266339171 shutdown_connections 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f66900763f0 0x7f6690078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66b00ae840 0x7f66b01550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66b0074520 0x7f66b01f2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 --2- 172.21.15.81:0/4266339171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b006f850 0x7f66b0156db0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 -- 172.21.15.81:0/4266339171 >> 172.21.15.81:0/4266339171 conn(0x7f66b0072950 msgr2=0x7f66b00cedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 -- 172.21.15.81:0/4266339171 shutdown_connections 2024-06-29T14:40:03.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:03.954+0000 7f66b55d2640 1 -- 172.21.15.81:0/4266339171 wait complete. 2024-06-29T14:40:03.963 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:40:03.963 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:40:03.963 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T14:40:03.963 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:40:03.963 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:40:03.963 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:40:03.963 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:40:03.963 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:40:03.963 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:40:03.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:03.964 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T14:40:03.980 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:40:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:40:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:40:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:40:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:03.980 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:40:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:40:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T14:40:03.980 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 41716sec preferred_lft 41716sec 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:40:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T14:40:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:40:03.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:03.983 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:40:04.044 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T14:40:04.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:04.045 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T14:40:04.102 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T14:40:04.102 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:40:04.102 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:40:04.102 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:40:04.102 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:40:04.103 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:40:04.103 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:40:04.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:40:04.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:40:04.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:40:04.103 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:40:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:40:04.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:40:04.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:04.229 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:40:04.244 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:40:04.244 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:40:04.244 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:40:04.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:04.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T14:40:04.309 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:40:04.310 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:40:04.365 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:40:04.365 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:40:04.366 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:40:04.421 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T14:40:04.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:04.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:40:04.514 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:40:04.515 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:40:04.523 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:40:04.523 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:40:04.523 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:40:04.523 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:40:04.523 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:40:04.524 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:40:04.524 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:40:04.524 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:40:04.524 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:40:04.524 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:40:04.526 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:40:04.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:04.526 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:40:04.551 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:40:04.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:40:04.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:40:04.655 INFO:teuthology.orchestra.run.smithi081.stdout:client6749 2024-06-29T14:40:04.655 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:40:04.655 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:40:04.671 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:40:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.828+0000 7fbe3ceef640 1 -- 172.21.15.81:0/987752422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe380b5250 msgr2=0x7fbe380d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.828+0000 7fbe3ceef640 1 --2- 172.21.15.81:0/987752422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe380b5250 0x7fbe380d1fc0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbe30005fd0 tx=0x7fbe3002a940 comp rx=0 tx=0).stop 2024-06-29T14:40:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.828+0000 7fbe3ceef640 1 -- 172.21.15.81:0/987752422 shutdown_connections 2024-06-29T14:40:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.828+0000 7fbe3ceef640 1 --2- 172.21.15.81:0/987752422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe380b5250 0x7fbe380d1fc0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.828+0000 7fbe3ceef640 1 --2- 172.21.15.81:0/987752422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe380d0b90 0x7fbe380d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.828+0000 7fbe3ceef640 1 -- 172.21.15.81:0/987752422 >> 172.21.15.81:0/987752422 conn(0x7fbe3809cf30 msgr2=0x7fbe380d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.828+0000 7fbe3ceef640 1 -- 172.21.15.81:0/987752422 shutdown_connections 2024-06-29T14:40:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.828+0000 7fbe3ceef640 1 -- 172.21.15.81:0/987752422 wait complete. 2024-06-29T14:40:04.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.830+0000 7fbe3ceef640 1 Processor -- start 2024-06-29T14:40:04.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.830+0000 7fbe3ceef640 1 -- start start 2024-06-29T14:40:04.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3ceef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe380b5250 0x7fbe38151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3ceef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe380b9040 0x7fbe380b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3ceef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe380b0820 0x7fbe380b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3ceef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbe380d2680 con 0x7fbe380b5250 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3ceef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbe380d2500 con 0x7fbe380b0820 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3ceef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbe380d2800 con 0x7fbe380b9040 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe380b5250 0x7fbe38151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe380b5250 0x7fbe38151180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46402/0 (socket says 172.21.15.81:46402) 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3659c640 1 -- 172.21.15.81:0/539802740 learned_addr learned my addr 172.21.15.81:0/539802740 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe35d9b640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe380b9040 0x7fbe380b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe36d9d640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe380b0820 0x7fbe380b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3659c640 1 -- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe380b9040 msgr2=0x7fbe380b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3659c640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe380b9040 0x7fbe380b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3659c640 1 -- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe380b0820 msgr2=0x7fbe380b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3659c640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe380b0820 0x7fbe380b1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3659c640 1 -- 172.21.15.81:0/539802740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe380b0f90 con 0x7fbe380b5250 2024-06-29T14:40:04.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.832+0000 7fbe3659c640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe380b5250 0x7fbe38151180 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000c3d0 tx=0x7fbe2000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:04.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.833+0000 7fbe2effd640 1 -- 172.21.15.81:0/539802740 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20011790 con 0x7fbe380b5250 2024-06-29T14:40:04.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.833+0000 7fbe2effd640 1 -- 172.21.15.81:0/539802740 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe20017070 con 0x7fbe380b5250 2024-06-29T14:40:04.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.833+0000 7fbe2effd640 1 -- 172.21.15.81:0/539802740 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe20011cc0 con 0x7fbe380b5250 2024-06-29T14:40:04.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.833+0000 7fbe3ceef640 1 -- 172.21.15.81:0/539802740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe380c7f90 con 0x7fbe380b5250 2024-06-29T14:40:04.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.833+0000 7fbe3ceef640 1 -- 172.21.15.81:0/539802740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe380c8550 con 0x7fbe380b5250 2024-06-29T14:40:04.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.834+0000 7fbe3ceef640 1 -- 172.21.15.81:0/539802740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe380d2d90 con 0x7fbe380b5250 2024-06-29T14:40:04.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.835+0000 7fbe2effd640 1 -- 172.21.15.81:0/539802740 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7fbe2001c460 con 0x7fbe380b5250 2024-06-29T14:40:04.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.835+0000 7fbe2effd640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbe18076420 0x7fbe18078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:04.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.835+0000 7fbe2effd640 1 -- 172.21.15.81:0/539802740 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7fbe20097650 con 0x7fbe380b5250 2024-06-29T14:40:04.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.835+0000 7fbe35d9b640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbe18076420 0x7fbe18078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:04.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.836+0000 7fbe35d9b640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbe18076420 0x7fbe18078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbe38154b60 tx=0x7fbe28005f90 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:04.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:04.840+0000 7fbe2effd640 1 -- 172.21.15.81:0/539802740 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbe200606f0 con 0x7fbe380b5250 2024-06-29T14:40:05.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.045+0000 7fbe3ceef640 1 -- 172.21.15.81:0/539802740 --> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fbe380c72a0 con 0x7fbe18076420 2024-06-29T14:40:05.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.047+0000 7fbe2effd640 1 -- 172.21.15.81:0/539802740 <== mgr.6533 v2:172.21.15.167:6832/473361950 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fbe380c72a0 con 0x7fbe18076420 2024-06-29T14:40:05.044 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:40:05.044 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:40:05.044 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T14:40:05.044 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:40:05.044 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:40:05.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 -- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbe18076420 msgr2=0x7fbe18078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbe18076420 0x7fbe18078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbe38154b60 tx=0x7fbe28005f90 comp rx=0 tx=0).stop 2024-06-29T14:40:05.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 -- 172.21.15.81:0/539802740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe380b5250 msgr2=0x7fbe38151180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe380b5250 0x7fbe38151180 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fbe2000c3d0 tx=0x7fbe2000c890 comp rx=0 tx=0).stop 2024-06-29T14:40:05.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 -- 172.21.15.81:0/539802740 shutdown_connections 2024-06-29T14:40:05.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbe18076420 0x7fbe18078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe380b0820 0x7fbe380b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe380b9040 0x7fbe380b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 --2- 172.21.15.81:0/539802740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe380b5250 0x7fbe38151180 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 -- 172.21.15.81:0/539802740 >> 172.21.15.81:0/539802740 conn(0x7fbe3809cf30 msgr2=0x7fbe3815e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:05.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 -- 172.21.15.81:0/539802740 shutdown_connections 2024-06-29T14:40:05.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.049+0000 7fbe3ceef640 1 -- 172.21.15.81:0/539802740 wait complete. 2024-06-29T14:40:05.057 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:40:05.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.177+0000 7f7a41d9a640 1 -- 172.21.15.81:0/360611965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a280040b0 msgr2=0x7f7a28004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.177+0000 7f7a41d9a640 1 --2- 172.21.15.81:0/360611965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a280040b0 0x7f7a28004800 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f7a24001690 tx=0x7f7a28005830 comp rx=0 tx=0).stop 2024-06-29T14:40:05.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.177+0000 7f7a3bfff640 1 -- 172.21.15.81:0/360611965 reap_dead start 2024-06-29T14:40:05.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.177+0000 7f7a41d9a640 1 -- 172.21.15.81:0/360611965 shutdown_connections 2024-06-29T14:40:05.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.177+0000 7f7a41d9a640 1 -- 172.21.15.81:0/360611965 >> 172.21.15.81:0/360611965 conn(0x7f7a3c0b3680 msgr2=0x7f7a3c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:05.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.177+0000 7f7a41d9a640 1 -- 172.21.15.81:0/360611965 shutdown_connections 2024-06-29T14:40:05.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.177+0000 7f7a41d9a640 1 -- 172.21.15.81:0/360611965 wait complete. 2024-06-29T14:40:05.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.178+0000 7f7a41d9a640 1 Processor -- start 2024-06-29T14:40:05.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.179+0000 7f7a41d9a640 1 -- start start 2024-06-29T14:40:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a41d9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a280040b0 0x7f7a3c14d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a41d9a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a3c159310 0x7f7a3c159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a41d9a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a3c151d20 0x7f7a3c152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a41d9a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c164d00 con 0x7f7a280040b0 2024-06-29T14:40:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a41d9a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c164b80 con 0x7f7a3c151d20 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a41d9a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7a3c164e80 con 0x7f7a3c159310 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a280040b0 0x7f7a3c14d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a280040b0 0x7f7a3c14d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46420/0 (socket says 172.21.15.81:46420) 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3b7fe640 1 -- 172.21.15.81:0/3151597863 learned_addr learned my addr 172.21.15.81:0/3151597863 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3affd640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a3c159310 0x7f7a3c159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3bfff640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a3c151d20 0x7f7a3c152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3b7fe640 1 -- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a3c159310 msgr2=0x7f7a3c159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3b7fe640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a3c159310 0x7f7a3c159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3b7fe640 1 -- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a3c151d20 msgr2=0x7f7a3c152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3b7fe640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a3c151d20 0x7f7a3c152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3b7fe640 1 -- 172.21.15.81:0/3151597863 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3c152670 con 0x7f7a280040b0 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.180+0000 7f7a3bfff640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a3c151d20 0x7f7a3c152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:40:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.181+0000 7f7a3b7fe640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a280040b0 0x7f7a3c14d8d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00e630 tx=0x7f7a2c00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:05.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.181+0000 7f7a33fff640 1 -- 172.21.15.81:0/3151597863 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c015070 con 0x7f7a280040b0 2024-06-29T14:40:05.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.181+0000 7f7a33fff640 1 -- 172.21.15.81:0/3151597863 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a2c00ca90 con 0x7f7a280040b0 2024-06-29T14:40:05.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.181+0000 7f7a41d9a640 1 -- 172.21.15.81:0/3151597863 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a3c14de10 con 0x7f7a280040b0 2024-06-29T14:40:05.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.181+0000 7f7a33fff640 1 -- 172.21.15.81:0/3151597863 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c011460 con 0x7f7a280040b0 2024-06-29T14:40:05.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.181+0000 7f7a41d9a640 1 -- 172.21.15.81:0/3151597863 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a3c14e4c0 con 0x7f7a280040b0 2024-06-29T14:40:05.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.182+0000 7f7a41d9a640 1 -- 172.21.15.81:0/3151597863 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a3c0cc7b0 con 0x7f7a280040b0 2024-06-29T14:40:05.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.182+0000 7f7a33fff640 1 -- 172.21.15.81:0/3151597863 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 57) ==== 98303+0+0 (secure 0 0 0) 0x7f7a2c018070 con 0x7f7a280040b0 2024-06-29T14:40:05.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.183+0000 7f7a33fff640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7a1c0763f0 0x7f7a1c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:05.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.183+0000 7f7a33fff640 1 -- 172.21.15.81:0/3151597863 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7f7a2c097e10 con 0x7f7a280040b0 2024-06-29T14:40:05.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.183+0000 7f7a3affd640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7a1c0763f0 0x7f7a1c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:05.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.184+0000 7f7a3affd640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7a1c0763f0 0x7f7a1c078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a24032000 tx=0x7f7a24001190 comp rx=0 tx=0).ready entity=mgr.6533 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:05.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.187+0000 7f7a33fff640 1 -- 172.21.15.81:0/3151597863 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7a3c152670 con 0x7f7a280040b0 2024-06-29T14:40:05.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.420+0000 7f7a41d9a640 1 -- 172.21.15.81:0/3151597863 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f7a3c0c5370 con 0x7f7a280040b0 2024-06-29T14:40:05.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.874+0000 7f7a33fff640 1 -- 172.21.15.81:0/3151597863 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v58) ==== 92+0+0 (secure 0 0 0) 0x7f7a2c060e20 con 0x7f7a280040b0 2024-06-29T14:40:05.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.876+0000 7f7a41d9a640 1 -- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7a1c0763f0 msgr2=0x7f7a1c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.876+0000 7f7a41d9a640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7a1c0763f0 0x7f7a1c078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a24032000 tx=0x7f7a24001190 comp rx=0 tx=0).stop 2024-06-29T14:40:05.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.876+0000 7f7a41d9a640 1 -- 172.21.15.81:0/3151597863 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a280040b0 msgr2=0x7f7a3c14d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:05.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.876+0000 7f7a41d9a640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a280040b0 0x7f7a3c14d8d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00e630 tx=0x7f7a2c00eaf0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.877+0000 7f7a41d9a640 1 -- 172.21.15.81:0/3151597863 shutdown_connections 2024-06-29T14:40:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.877+0000 7f7a41d9a640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f7a1c0763f0 0x7f7a1c078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.877+0000 7f7a41d9a640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a3c151d20 0x7f7a3c152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.877+0000 7f7a41d9a640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a3c159310 0x7f7a3c159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.877+0000 7f7a41d9a640 1 --2- 172.21.15.81:0/3151597863 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a280040b0 0x7f7a3c14d8d0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.877+0000 7f7a41d9a640 1 -- 172.21.15.81:0/3151597863 >> 172.21.15.81:0/3151597863 conn(0x7f7a3c0b3680 msgr2=0x7f7a3c15b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.878+0000 7f7a41d9a640 1 -- 172.21.15.81:0/3151597863 shutdown_connections 2024-06-29T14:40:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:05.878+0000 7f7a41d9a640 1 -- 172.21.15.81:0/3151597863 wait complete. 2024-06-29T14:40:05.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:40:06.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.013+0000 7fd9705e0640 1 -- 172.21.15.81:0/679129543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9680b9040 msgr2=0x7fd9680d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.013+0000 7fd9705e0640 1 --2- 172.21.15.81:0/679129543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9680b9040 0x7fd9680d0480 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fd960005fd0 tx=0x7fd96002a670 comp rx=0 tx=0).stop 2024-06-29T14:40:06.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.013+0000 7fd9705e0640 1 -- 172.21.15.81:0/679129543 shutdown_connections 2024-06-29T14:40:06.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.013+0000 7fd9705e0640 1 --2- 172.21.15.81:0/679129543 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9680d2a90 0x7fd9680d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.013+0000 7fd9705e0640 1 --2- 172.21.15.81:0/679129543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9680b9040 0x7fd9680d0480 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.013+0000 7fd9705e0640 1 -- 172.21.15.81:0/679129543 >> 172.21.15.81:0/679129543 conn(0x7fd96809cf30 msgr2=0x7fd9680d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.013+0000 7fd9705e0640 1 -- 172.21.15.81:0/679129543 shutdown_connections 2024-06-29T14:40:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.013+0000 7fd9705e0640 1 -- 172.21.15.81:0/679129543 wait complete. 2024-06-29T14:40:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.015+0000 7fd9705e0640 1 Processor -- start 2024-06-29T14:40:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.015+0000 7fd9705e0640 1 -- start start 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.016+0000 7fd9705e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9680b5250 0x7fd96815e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.016+0000 7fd9705e0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9680b9040 0x7fd9680c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.016+0000 7fd9705e0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9680d2a90 0x7fd9680a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.016+0000 7fd9705e0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd9680b7d30 con 0x7fd9680b5250 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.016+0000 7fd9705e0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd9680b7bb0 con 0x7fd9680b9040 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.016+0000 7fd9705e0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd9680b7eb0 con 0x7fd9680d2a90 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96e37c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9680b5250 0x7fd96815e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96e37c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9680b5250 0x7fd96815e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46464/0 (socket says 172.21.15.81:46464) 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96e37c640 1 -- 172.21.15.81:0/1801786315 learned_addr learned my addr 172.21.15.81:0/1801786315 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96db7b640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9680b9040 0x7fd9680c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96eb7d640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9680d2a90 0x7fd9680a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96e37c640 1 -- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9680d2a90 msgr2=0x7fd9680a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96e37c640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9680d2a90 0x7fd9680a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96e37c640 1 -- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9680b9040 msgr2=0x7fd9680c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96e37c640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9680b9040 0x7fd9680c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96e37c640 1 -- 172.21.15.81:0/1801786315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9680b38b0 con 0x7fd9680b5250 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd96e37c640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9680b5250 0x7fd96815e2c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fd960000c00 tx=0x7fd9600038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd966ffd640 1 -- 172.21.15.81:0/1801786315 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96003e030 con 0x7fd9680b5250 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd966ffd640 1 -- 172.21.15.81:0/1801786315 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd960003db0 con 0x7fd9680b5250 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd9705e0640 1 -- 172.21.15.81:0/1801786315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9680b3b90 con 0x7fd9680b5250 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd966ffd640 1 -- 172.21.15.81:0/1801786315 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960035040 con 0x7fd9680b5250 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.017+0000 7fd9705e0640 1 -- 172.21.15.81:0/1801786315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9680b4150 con 0x7fd9680b5250 2024-06-29T14:40:06.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.018+0000 7fd9705e0640 1 -- 172.21.15.81:0/1801786315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9680b8440 con 0x7fd9680b5250 2024-06-29T14:40:06.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.019+0000 7fd966ffd640 1 -- 172.21.15.81:0/1801786315 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 58) ==== 98320+0+0 (secure 0 0 0) 0x7fd960041030 con 0x7fd9680b5250 2024-06-29T14:40:06.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.020+0000 7fd966ffd640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd954076470 0x7fd954078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.020+0000 7fd966ffd640 1 -- 172.21.15.81:0/1801786315 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7fd9600b7f00 con 0x7fd9680b5250 2024-06-29T14:40:06.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.020+0000 7fd96db7b640 1 -- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd954076470 msgr2=0x7fd954078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/473361950 2024-06-29T14:40:06.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.020+0000 7fd96db7b640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd954076470 0x7fd954078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:40:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.024+0000 7fd966ffd640 1 -- 172.21.15.81:0/1801786315 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd960081030 con 0x7fd9680b5250 2024-06-29T14:40:06.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.212+0000 7fd9705e0640 1 -- 172.21.15.81:0/1801786315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fd9680cdfd0 con 0x7fd9680b5250 2024-06-29T14:40:06.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.216+0000 7fd966ffd640 1 -- 172.21.15.81:0/1801786315 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v49)=0 v49) ==== 138+0+0 (secure 0 0 0) 0x7fd960085330 con 0x7fd9680b5250 2024-06-29T14:40:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 -- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd954076470 msgr2=0x7fd954078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:40:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd954076470 0x7fd954078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 -- 172.21.15.81:0/1801786315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9680b5250 msgr2=0x7fd96815e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9680b5250 0x7fd96815e2c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fd960000c00 tx=0x7fd9600038a0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 -- 172.21.15.81:0/1801786315 shutdown_connections 2024-06-29T14:40:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fd954076470 0x7fd954078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd9680d2a90 0x7fd9680a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd9680b9040 0x7fd9680c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 --2- 172.21.15.81:0/1801786315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9680b5250 0x7fd96815e2c0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 -- 172.21.15.81:0/1801786315 >> 172.21.15.81:0/1801786315 conn(0x7fd96809cf30 msgr2=0x7fd96815f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 -- 172.21.15.81:0/1801786315 shutdown_connections 2024-06-29T14:40:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.218+0000 7fd9705e0640 1 -- 172.21.15.81:0/1801786315 wait complete. 2024-06-29T14:40:06.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:40:06.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.349+0000 7f475c33c640 1 -- 172.21.15.81:0/1360605814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47540b3cd0 msgr2=0x7f47540d2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.349+0000 7f475c33c640 1 --2- 172.21.15.81:0/1360605814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47540b3cd0 0x7f47540d2240 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4750005fd0 tx=0x7f475002a540 comp rx=0 tx=0).stop 2024-06-29T14:40:06.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.349+0000 7f475c33c640 1 -- 172.21.15.81:0/1360605814 shutdown_connections 2024-06-29T14:40:06.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.349+0000 7f475c33c640 1 --2- 172.21.15.81:0/1360605814 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47540ceb80 0x7f47540d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.349+0000 7f475c33c640 1 --2- 172.21.15.81:0/1360605814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47540b3cd0 0x7f47540d2240 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.349+0000 7f475c33c640 1 -- 172.21.15.81:0/1360605814 >> 172.21.15.81:0/1360605814 conn(0x7f47540d3c60 msgr2=0x7f47540d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:06.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.350+0000 7f475c33c640 1 -- 172.21.15.81:0/1360605814 shutdown_connections 2024-06-29T14:40:06.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.350+0000 7f475c33c640 1 -- 172.21.15.81:0/1360605814 wait complete. 2024-06-29T14:40:06.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.351+0000 7f475c33c640 1 Processor -- start 2024-06-29T14:40:06.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.351+0000 7f475c33c640 1 -- start start 2024-06-29T14:40:06.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f475c33c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47540ce770 0x7f4754151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f475c33c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47540ceb80 0x7f47540cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f475c33c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47540cdd40 0x7f47541671f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f475c33c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f47540d15a0 con 0x7f47540ceb80 2024-06-29T14:40:06.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f475c33c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f47540d1420 con 0x7f47540ce770 2024-06-29T14:40:06.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f475c33c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f47540d4990 con 0x7f47540cdd40 2024-06-29T14:40:06.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f47598d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47540ceb80 0x7f47540cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f47598d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47540ceb80 0x7f47540cd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46466/0 (socket says 172.21.15.81:46466) 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f47598d7640 1 -- 172.21.15.81:0/287388328 learned_addr learned my addr 172.21.15.81:0/287388328 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f475a0d8640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47540ce770 0x7f4754151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f475a8d9640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47540cdd40 0x7f47541671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f47598d7640 1 -- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47540cdd40 msgr2=0x7f47541671f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f47598d7640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47540cdd40 0x7f47541671f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f47598d7640 1 -- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47540ce770 msgr2=0x7f4754151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f47598d7640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47540ce770 0x7f4754151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f47598d7640 1 -- 172.21.15.81:0/287388328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4754164390 con 0x7f47540ceb80 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f47598d7640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47540ceb80 0x7f47540cd800 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f4750034040 tx=0x7f47500038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f4746ffd640 1 -- 172.21.15.81:0/287388328 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475003e030 con 0x7f47540ceb80 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f4746ffd640 1 -- 172.21.15.81:0/287388328 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4750003d90 con 0x7f47540ceb80 2024-06-29T14:40:06.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.353+0000 7f475c33c640 1 -- 172.21.15.81:0/287388328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4754161550 con 0x7f47540ceb80 2024-06-29T14:40:06.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.354+0000 7f4746ffd640 1 -- 172.21.15.81:0/287388328 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750035040 con 0x7f47540ceb80 2024-06-29T14:40:06.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.354+0000 7f475c33c640 1 -- 172.21.15.81:0/287388328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47541f8840 con 0x7f47540ceb80 2024-06-29T14:40:06.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.354+0000 7f475c33c640 1 -- 172.21.15.81:0/287388328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47541f82d0 con 0x7f47540ceb80 2024-06-29T14:40:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.356+0000 7f4746ffd640 1 -- 172.21.15.81:0/287388328 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 58) ==== 98320+0+0 (secure 0 0 0) 0x7f4750041030 con 0x7f47540ceb80 2024-06-29T14:40:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.356+0000 7f4746ffd640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4738076440 0x7f4738078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.356+0000 7f4746ffd640 1 -- 172.21.15.81:0/287388328 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7f47500b8670 con 0x7f47540ceb80 2024-06-29T14:40:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.356+0000 7f475a0d8640 1 -- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4738076440 msgr2=0x7f4738078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/473361950 2024-06-29T14:40:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.356+0000 7f475a0d8640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4738076440 0x7f4738078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:40:06.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.360+0000 7f4746ffd640 1 -- 172.21.15.81:0/287388328 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f47500817f0 con 0x7f47540ceb80 2024-06-29T14:40:06.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.544+0000 7f475c33c640 1 -- 172.21.15.81:0/287388328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f47540d2240 con 0x7f47540ceb80 2024-06-29T14:40:06.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.548+0000 7f4746ffd640 1 -- 172.21.15.81:0/287388328 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v50)=0 v50) ==== 133+0+0 (secure 0 0 0) 0x7f4750085af0 con 0x7f47540ceb80 2024-06-29T14:40:06.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 -- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4738076440 msgr2=0x7f4738078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:40:06.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4738076440 0x7f4738078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 -- 172.21.15.81:0/287388328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47540ceb80 msgr2=0x7f47540cd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47540ceb80 0x7f47540cd800 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f4750034040 tx=0x7f47500038a0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 -- 172.21.15.81:0/287388328 shutdown_connections 2024-06-29T14:40:06.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7f4738076440 0x7f4738078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47540cdd40 0x7f47541671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47540ceb80 0x7f47540cd800 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 --2- 172.21.15.81:0/287388328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47540ce770 0x7f4754151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 -- 172.21.15.81:0/287388328 >> 172.21.15.81:0/287388328 conn(0x7f47540d3c60 msgr2=0x7f475406ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:06.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 -- 172.21.15.81:0/287388328 shutdown_connections 2024-06-29T14:40:06.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.550+0000 7f475c33c640 1 -- 172.21.15.81:0/287388328 wait complete. 2024-06-29T14:40:06.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:06.560 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T14:40:06.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-06-29T14:40:06.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.741+0000 7fbfaed03640 1 -- 172.21.15.81:0/3868020888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfa80d0230 msgr2=0x7fbfa80d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.741+0000 7fbfaed03640 1 --2- 172.21.15.81:0/3868020888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfa80d0230 0x7fbfa80d0640 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009830 tx=0x7fbf9402a530 comp rx=0 tx=0).stop 2024-06-29T14:40:06.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.741+0000 7fbfaed03640 1 -- 172.21.15.81:0/3868020888 shutdown_connections 2024-06-29T14:40:06.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.741+0000 7fbfaed03640 1 --2- 172.21.15.81:0/3868020888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfa80d0230 0x7fbfa80d0640 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.741+0000 7fbfaed03640 1 -- 172.21.15.81:0/3868020888 >> 172.21.15.81:0/3868020888 conn(0x7fbfa80b3680 msgr2=0x7fbfa80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:06.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.741+0000 7fbfaed03640 1 -- 172.21.15.81:0/3868020888 shutdown_connections 2024-06-29T14:40:06.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.741+0000 7fbfaed03640 1 -- 172.21.15.81:0/3868020888 wait complete. 2024-06-29T14:40:06.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.743+0000 7fbfaed03640 1 Processor -- start 2024-06-29T14:40:06.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.743+0000 7fbfaed03640 1 -- start start 2024-06-29T14:40:06.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.744+0000 7fbfaed03640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfa80b3200 0x7fbfa806eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.744+0000 7fbfaed03640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfa80b4f40 0x7fbfa8154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.744+0000 7fbfaed03640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfa80d0230 0x7fbfa8155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.744+0000 7fbfaed03640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa80b5e50 con 0x7fbfa80b3200 2024-06-29T14:40:06.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.744+0000 7fbfaed03640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa80b5cd0 con 0x7fbfa80d0230 2024-06-29T14:40:06.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.744+0000 7fbfaed03640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa80b5fd0 con 0x7fbfa80b4f40 2024-06-29T14:40:06.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaca9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfa80b3200 0x7fbfa806eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaca9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfa80b3200 0x7fbfa806eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46488/0 (socket says 172.21.15.81:46488) 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaca9f640 1 -- 172.21.15.81:0/2292530217 learned_addr learned my addr 172.21.15.81:0/2292530217 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfad2a0640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfa80d0230 0x7fbfa8155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfa7fff640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfa80b4f40 0x7fbfa8154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaca9f640 1 -- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfa80b4f40 msgr2=0x7fbfa8154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaca9f640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfa80b4f40 0x7fbfa8154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaca9f640 1 -- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfa80d0230 msgr2=0x7fbfa8155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaca9f640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfa80d0230 0x7fbfa8155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaca9f640 1 -- 172.21.15.81:0/2292530217 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa81f5150 con 0x7fbfa80b3200 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaca9f640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfa80b3200 0x7fbfa806eb80 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00c370 tx=0x7fbf9c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c017070 con 0x7fbfa80b3200 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf9c0035f0 con 0x7fbfa80b3200 2024-06-29T14:40:06.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaed03640 1 -- 172.21.15.81:0/2292530217 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa81f5430 con 0x7fbfa80b3200 2024-06-29T14:40:06.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c003a40 con 0x7fbfa80b3200 2024-06-29T14:40:06.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.745+0000 7fbfaed03640 1 -- 172.21.15.81:0/2292530217 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa81f59f0 con 0x7fbfa80b3200 2024-06-29T14:40:06.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.746+0000 7fbfaed03640 1 -- 172.21.15.81:0/2292530217 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa80d0640 con 0x7fbfa80b3200 2024-06-29T14:40:06.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.747+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 58) ==== 98320+0+0 (secure 0 0 0) 0x7fbf9c01a070 con 0x7fbfa80b3200 2024-06-29T14:40:06.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.748+0000 7fbfa57fa640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 0x7fbf8c078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:06.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.748+0000 7fbfa7fff640 1 -- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 msgr2=0x7fbf8c078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/473361950 2024-06-29T14:40:06.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.748+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5781+0+0 (secure 0 0 0) 0x7fbf9c096f80 con 0x7fbfa80b3200 2024-06-29T14:40:06.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.748+0000 7fbfa7fff640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 0x7fbf8c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:40:06.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.752+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbf9c021020 con 0x7fbfa80b3200 2024-06-29T14:40:06.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.950+0000 7fbfa7fff640 1 -- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 msgr2=0x7fbf8c078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/473361950 2024-06-29T14:40:06.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.950+0000 7fbfa7fff640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 0x7fbf8c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:40:06.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:06.965+0000 7fbfaed03640 1 -- 172.21.15.81:0/2292530217 --> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fbfa81f61f0 con 0x7fbf8c076470 2024-06-29T14:40:07.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:07.351+0000 7fbfa7fff640 1 -- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 msgr2=0x7fbf8c078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/473361950 2024-06-29T14:40:07.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:07.351+0000 7fbfa7fff640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 0x7fbf8c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:40:08.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:08.152+0000 7fbfa7fff640 1 -- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 msgr2=0x7fbf8c078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/473361950 2024-06-29T14:40:08.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:08.152+0000 7fbfa7fff640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 0x7fbf8c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:40:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:09.494+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 59) ==== 97929+0+0 (secure 0 0 0) 0x7fbf9c05c410 con 0x7fbfa80b3200 2024-06-29T14:40:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:09.494+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 msgr2=0x7fbf8c078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:40:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:09.494+0000 7fbfa57fa640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/473361950,v1:172.21.15.167:6833/473361950] conn(0x7fbf8c076470 0x7fbf8c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:09.656 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:40:09.658+0000 7f669a3d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:09.656 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:40:09.658+0000 7f669a3d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:09.656 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:40:09.658+0000 7f669a3d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:09.656 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:40:09.658+0000 7f669a3d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:09.656 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:40:09.658+0000 7f669a3d2640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:09.656 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:40:09.658+0000 7f669d3d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:09.656 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:40:09.658+0000 7f669d3d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:09.657 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:40:09.658+0000 7f669d3d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:09.657 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:40:09.658+0000 7f669d3d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:09.657 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:40:09.658+0000 7f669d3d8640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:40:10.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.497+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 60) ==== 98056+0+0 (secure 0 0 0) 0x7fbf9c01acf0 con 0x7fbfa80b3200 2024-06-29T14:40:10.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.497+0000 7fbfa57fa640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fbf8c03b050 0x7fbf8c080460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:10.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.497+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 --> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fbf9c00ee50 con 0x7fbf8c03b050 2024-06-29T14:40:10.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.498+0000 7fbfa7fff640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fbf8c03b050 0x7fbf8c080460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:10.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.503+0000 7fbfa7fff640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fbf8c03b050 0x7fbf8c080460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbf94034000 tx=0x7fbf94004730 comp rx=0 tx=0).ready entity=mgr.6770 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:10.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.575+0000 7fbfa57fa640 1 -- 172.21.15.81:0/2292530217 <== mgr.6770 v2:172.21.15.167:6832/1146990244 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbf9c00ee50 con 0x7fbf8c03b050 2024-06-29T14:40:10.573 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1 2024-06-29T14:40:10.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.577+0000 7fbfaed03640 1 -- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fbf8c03b050 msgr2=0x7fbf8c080460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:10.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.577+0000 7fbfaed03640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fbf8c03b050 0x7fbf8c080460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbf94034000 tx=0x7fbf94004730 comp rx=0 tx=0).stop 2024-06-29T14:40:10.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.577+0000 7fbfaed03640 1 -- 172.21.15.81:0/2292530217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfa80b3200 msgr2=0x7fbfa806eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:10.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.577+0000 7fbfaed03640 1 --2- 172.21.15.81:0/2292530217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfa80b3200 0x7fbfa806eb80 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c00c370 tx=0x7fbf9c00c830 comp rx=0 tx=0).stop 2024-06-29T14:40:10.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.577+0000 7fbfad2a0640 1 -- 172.21.15.81:0/2292530217 reap_dead start 2024-06-29T14:40:10.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.578+0000 7fbfaed03640 1 -- 172.21.15.81:0/2292530217 shutdown_connections 2024-06-29T14:40:10.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.578+0000 7fbfaed03640 1 -- 172.21.15.81:0/2292530217 >> 172.21.15.81:0/2292530217 conn(0x7fbfa80b3680 msgr2=0x7fbfa80cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:10.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.578+0000 7fbfaed03640 1 -- 172.21.15.81:0/2292530217 shutdown_connections 2024-06-29T14:40:10.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.578+0000 7fbfaed03640 1 -- 172.21.15.81:0/2292530217 wait complete. 2024-06-29T14:40:10.587 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T14:40:10.587 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-06-29-14_41_00 in ~50s... 2024-06-29T14:40:10.587 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-06-29T14:40:10.587 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-06-29T14:40:10.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.706+0000 7f33d47d4640 1 -- 172.21.15.81:0/3157738231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc0cd540 msgr2=0x7f33cc0b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:10.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.706+0000 7f33d47d4640 1 --2- 172.21.15.81:0/3157738231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc0cd540 0x7f33cc0b47b0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0d2600 tx=0x7f33c002a510 comp rx=0 tx=0).stop 2024-06-29T14:40:10.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.706+0000 7f33d47d4640 1 -- 172.21.15.81:0/3157738231 shutdown_connections 2024-06-29T14:40:10.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.706+0000 7f33d47d4640 1 --2- 172.21.15.81:0/3157738231 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33cc0b4cf0 0x7f33cc0d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:10.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.706+0000 7f33d47d4640 1 --2- 172.21.15.81:0/3157738231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc0cd540 0x7f33cc0b47b0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:10.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.706+0000 7f33d47d4640 1 -- 172.21.15.81:0/3157738231 >> 172.21.15.81:0/3157738231 conn(0x7f33cc0d1960 msgr2=0x7f33cc0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:10.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.706+0000 7f33d47d4640 1 -- 172.21.15.81:0/3157738231 shutdown_connections 2024-06-29T14:40:10.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.706+0000 7f33d47d4640 1 -- 172.21.15.81:0/3157738231 wait complete. 2024-06-29T14:40:10.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.708+0000 7f33d47d4640 1 Processor -- start 2024-06-29T14:40:10.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.708+0000 7f33d47d4640 1 -- start start 2024-06-29T14:40:10.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d47d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc0b4cf0 0x7f33cc15a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:10.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d47d4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33cc0b6d60 0x7f33cc159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:10.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d47d4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33cc0cd540 0x7f33cc15c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:10.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d47d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc0d07b0 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d47d4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc0d0630 con 0x7f33cc0b6d60 2024-06-29T14:40:10.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d47d4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f33cc0d0930 con 0x7f33cc0cd540 2024-06-29T14:40:10.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d2570640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc0b4cf0 0x7f33cc15a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:10.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d2570640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc0b4cf0 0x7f33cc15a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57674/0 (socket says 172.21.15.81:57674) 2024-06-29T14:40:10.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d2570640 1 -- 172.21.15.81:0/4147045650 learned_addr learned my addr 172.21.15.81:0/4147045650 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:40:10.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d2d71640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33cc0cd540 0x7f33cc15c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d1d6f640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33cc0b6d60 0x7f33cc159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d2570640 1 -- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33cc0cd540 msgr2=0x7f33cc15c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.709+0000 7f33d2570640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33cc0cd540 0x7f33cc15c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.710+0000 7f33d2570640 1 -- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33cc0b6d60 msgr2=0x7f33cc159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.710+0000 7f33d2570640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33cc0b6d60 0x7f33cc159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.710+0000 7f33d2570640 1 -- 172.21.15.81:0/4147045650 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc15d0e0 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.710+0000 7f33d2570640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc0b4cf0 0x7f33cc15a6c0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00c3a0 tx=0x7f33bc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.710+0000 7f33caffd640 1 -- 172.21.15.81:0/4147045650 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33bc011790 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.710+0000 7f33caffd640 1 -- 172.21.15.81:0/4147045650 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33bc017070 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.710+0000 7f33d47d4640 1 -- 172.21.15.81:0/4147045650 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33cc15d3c0 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.710+0000 7f33caffd640 1 -- 172.21.15.81:0/4147045650 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33bc011c80 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.710+0000 7f33d47d4640 1 -- 172.21.15.81:0/4147045650 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33cc15ac00 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.711+0000 7f33d47d4640 1 -- 172.21.15.81:0/4147045650 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33cc0d00f0 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.711+0000 7f33caffd640 1 -- 172.21.15.81:0/4147045650 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 60) ==== 98056+0+0 (secure 0 0 0) 0x7f33bc01c460 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.712+0000 7f33caffd640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f33b8076170 0x7f33b8078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:40:10.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.712+0000 7f33caffd640 1 -- 172.21.15.81:0/4147045650 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7f33bc097270 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.712+0000 7f33d1d6f640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f33b8076170 0x7f33b8078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:40:10.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.713+0000 7f33d1d6f640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f33b8076170 0x7f33b8078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f33c0034000 tx=0x7f33c0004730 comp rx=0 tx=0).ready entity=mgr.6770 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:40:10.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.717+0000 7f33caffd640 1 -- 172.21.15.81:0/4147045650 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f33bc060690 con 0x7f33cc0b4cf0 2024-06-29T14:40:10.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.917+0000 7f33d47d4640 1 -- 172.21.15.81:0/4147045650 --> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f33cc15d8e0 con 0x7f33b8076170 2024-06-29T14:40:10.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.924+0000 7f33caffd640 1 -- 172.21.15.81:0/4147045650 <== mgr.6770 v2:172.21.15.167:6832/1146990244 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+66 (secure 0 0 0) 0x7f33cc15d8e0 con 0x7f33b8076170 2024-06-29T14:40:10.922 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:40:10.922 INFO:teuthology.orchestra.run.smithi081.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-06-29T14:40:10.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.926+0000 7f33d47d4640 1 -- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f33b8076170 msgr2=0x7f33b8078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:10.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.926+0000 7f33d47d4640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f33b8076170 0x7f33b8078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f33c0034000 tx=0x7f33c0004730 comp rx=0 tx=0).stop 2024-06-29T14:40:10.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.926+0000 7f33d47d4640 1 -- 172.21.15.81:0/4147045650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc0b4cf0 msgr2=0x7f33cc15a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:40:10.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.926+0000 7f33d47d4640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc0b4cf0 0x7f33cc15a6c0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00c3a0 tx=0x7f33bc00c860 comp rx=0 tx=0).stop 2024-06-29T14:40:10.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.927+0000 7f33d47d4640 1 -- 172.21.15.81:0/4147045650 shutdown_connections 2024-06-29T14:40:10.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.927+0000 7f33d47d4640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f33b8076170 0x7f33b8078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:10.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.927+0000 7f33d47d4640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33cc0cd540 0x7f33cc15c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:10.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.927+0000 7f33d47d4640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33cc0b6d60 0x7f33cc159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:10.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.927+0000 7f33d47d4640 1 --2- 172.21.15.81:0/4147045650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc0b4cf0 0x7f33cc15a6c0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:40:10.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.927+0000 7f33d47d4640 1 -- 172.21.15.81:0/4147045650 >> 172.21.15.81:0/4147045650 conn(0x7f33cc0d1960 msgr2=0x7f33cc071860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:40:10.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.927+0000 7f33d47d4640 1 -- 172.21.15.81:0/4147045650 shutdown_connections 2024-06-29T14:40:10.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:40:10.927+0000 7f33d47d4640 1 -- 172.21.15.81:0/4147045650 wait complete. 2024-06-29T14:40:10.935 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2024-06-29T14:40:10.935 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2024-06-29T14:40:10.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:10.936 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:10.951 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:10.952 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:10.952 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:11.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:40:11.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:40:11.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:11.953 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:11.971 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:11.971 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:11.971 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:12.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:12.972 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:12.992 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:12.992 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:12.992 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:13.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:13.994 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:14.014 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:14.014 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:14.014 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:15.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:15.015 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:15.035 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:15.035 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:15.036 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:16.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:16.037 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:16.057 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:16.057 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:16.057 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:17.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:17.059 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:17.078 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:17.078 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:17.078 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:18.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:18.080 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:18.100 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:18.100 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:18.100 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:19.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:19.101 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:19.122 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:19.122 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:19.122 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:20.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:20.125 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:20.145 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:20.145 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:20.146 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:21.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:21.147 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:21.167 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:21.167 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:21.167 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:22.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:22.168 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:22.188 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:22.189 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:22.189 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:23.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:23.190 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:23.210 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:23.210 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:23.210 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:24.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:24.211 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:24.231 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:24.231 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:24.231 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:25.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:25.233 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:25.253 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:25.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:25.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:26.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:26.254 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:26.274 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:26.275 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:26.275 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:27.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:27.276 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:27.296 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:27.296 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:27.296 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:28.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:28.297 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:28.317 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:28.317 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:28.317 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:29.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:29.318 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:29.338 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:29.339 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:29.339 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:30.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:30.340 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:30.360 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:30.360 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:30.360 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:31.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:31.361 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:31.381 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:31.381 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:31.381 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:32.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:32.382 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:32.403 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:32.403 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:32.403 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:33.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:33.404 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:33.423 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:33.424 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:33.424 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:34.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:34.424 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:34.444 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:34.444 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:34.445 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:35.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:35.446 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:35.466 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:35.467 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:35.467 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:36.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:36.468 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:36.488 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:36.488 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:36.488 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:37.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:37.489 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:37.509 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:37.510 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:37.510 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:38.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:38.511 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:38.531 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:38.531 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:38.531 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:39.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:39.533 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:39.554 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:39.554 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:39.554 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:40.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:40.555 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:40.575 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:40.576 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:40.576 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:41.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:41.577 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:41.597 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:41.597 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:41.597 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:41.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:40:41.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:40:42.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:42.599 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:42.619 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:42.619 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:42.620 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:43.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:43.621 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:43.641 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:43.641 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:43.641 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:44.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:44.646 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:44.666 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:44.667 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:44.667 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:45.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:45.667 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:45.687 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:45.688 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:45.688 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:46.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:46.689 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:46.709 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:46.709 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:46.709 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:47.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:47.710 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:47.730 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:47.730 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:47.730 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:48.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:48.731 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:48.751 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:48.751 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:48.752 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:49.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:49.753 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:49.773 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:49.773 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:49.773 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:50.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:50.774 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:50.795 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:50.795 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:50.796 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:51.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:51.797 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:51.817 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:51.817 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:51.817 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:52.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:52.818 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:52.839 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:52.839 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:52.840 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:53.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:53.840 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:53.860 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:53.860 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:53.860 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:54.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:54.861 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:54.885 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:54.885 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:54.885 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:55.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:55.890 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:55.911 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:55.911 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:55.911 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:56.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:56.915 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:56.936 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:56.936 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:56.936 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:57.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:57.937 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:57.957 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:57.957 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:57.957 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:58.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:58.958 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:40:58.978 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:40:58.979 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:40:58.979 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:40:59.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:40:59.980 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:41:00.001 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T14:41:00.001 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T14:41:00.001 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:41:00.584 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:00.586+0000 7f668cbb7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:00.584 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:00.586+0000 7f668cbb7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:00.584 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:00.586+0000 7f668cbb7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:00.585 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:00.586+0000 7f668cbb7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:00.585 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:00.586+0000 7f668cbb7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:01.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:01.002 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:41:01.021 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_41_00_UTC 2024-06-29T14:41:01.021 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-14_41_00_UTC'} 2024-06-29T14:41:01.021 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-06-29-14_41_00_UTC'} 2024-06-29T14:41:01.022 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T14:41:01.022 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-06-29-14_41_00_UTC'] 2024-06-29T14:41:01.022 DEBUG:tasks.cephfs.test_snap_schedules:exec=1719672010.5875113, now = 1719672061.0222194, timo = 50 2024-06-29T14:41:02.025 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-06-29T14:41:02.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.157+0000 7f6499ab3640 1 -- 172.21.15.81:0/1506953066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6494071f20 msgr2=0x7f6494144a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.157+0000 7f6499ab3640 1 --2- 172.21.15.81:0/1506953066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6494071f20 0x7f6494144a20 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f6484005fd0 tx=0x7f648402a620 comp rx=0 tx=0).stop 2024-06-29T14:41:02.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.157+0000 7f6499ab3640 1 -- 172.21.15.81:0/1506953066 shutdown_connections 2024-06-29T14:41:02.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.157+0000 7f6499ab3640 1 --2- 172.21.15.81:0/1506953066 >> v2:172.21.15.167:3300/0 conn(0x7f649406f4e0 0x7f649415c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.157+0000 7f6499ab3640 1 --2- 172.21.15.81:0/1506953066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6494071f20 0x7f6494144a20 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.157+0000 7f6499ab3640 1 -- 172.21.15.81:0/1506953066 >> 172.21.15.81:0/1506953066 conn(0x7f649406ccc0 msgr2=0x7f64940b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:02.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.157+0000 7f6499ab3640 1 -- 172.21.15.81:0/1506953066 shutdown_connections 2024-06-29T14:41:02.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.157+0000 7f6499ab3640 1 -- 172.21.15.81:0/1506953066 wait complete. 2024-06-29T14:41:02.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.159+0000 7f6499ab3640 1 Processor -- start 2024-06-29T14:41:02.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.159+0000 7f6499ab3640 1 -- start start 2024-06-29T14:41:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.160+0000 7f6499ab3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6494071f20 0x7f64941f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.160+0000 7f6499ab3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6494145180 0x7f64940a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.160+0000 7f6499ab3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6494141ad0 0x7f6494141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.160+0000 7f6499ab3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f649415d1f0 con 0x7f6494071f20 2024-06-29T14:41:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.160+0000 7f6499ab3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f649415d070 con 0x7f6494141ad0 2024-06-29T14:41:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.160+0000 7f6499ab3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f649415d370 con 0x7f6494145180 2024-06-29T14:41:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.160+0000 7f64937fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6494071f20 0x7f64941f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.160+0000 7f64937fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6494071f20 0x7f64941f5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34874/0 (socket says 172.21.15.81:34874) 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.160+0000 7f64937fe640 1 -- 172.21.15.81:0/3079687498 learned_addr learned my addr 172.21.15.81:0/3079687498 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f6492ffd640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6494145180 0x7f64940a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f6493fff640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6494141ad0 0x7f6494141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f64937fe640 1 -- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6494145180 msgr2=0x7f64940a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f64937fe640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6494145180 0x7f64940a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f64937fe640 1 -- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6494141ad0 msgr2=0x7f6494141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f64937fe640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6494141ad0 0x7f6494141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f64937fe640 1 -- 172.21.15.81:0/3079687498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6494141600 con 0x7f6494071f20 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f6493fff640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6494141ad0 0x7f6494141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f64937fe640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6494071f20 0x7f64941f5980 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f6484034040 tx=0x7f648402ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f647bfff640 1 -- 172.21.15.81:0/3079687498 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f648403e030 con 0x7f6494071f20 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f647bfff640 1 -- 172.21.15.81:0/3079687498 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6484004b20 con 0x7f6494071f20 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f6499ab3640 1 -- 172.21.15.81:0/3079687498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64941f48e0 con 0x7f6494071f20 2024-06-29T14:41:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f647bfff640 1 -- 172.21.15.81:0/3079687498 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6484035040 con 0x7f6494071f20 2024-06-29T14:41:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.161+0000 7f6499ab3640 1 -- 172.21.15.81:0/3079687498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64941f4ea0 con 0x7f6494071f20 2024-06-29T14:41:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.162+0000 7f6499ab3640 1 -- 172.21.15.81:0/3079687498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f649415d890 con 0x7f6494071f20 2024-06-29T14:41:02.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.163+0000 7f647bfff640 1 -- 172.21.15.81:0/3079687498 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 63) ==== 98434+0+0 (secure 0 0 0) 0x7f6484041030 con 0x7f6494071f20 2024-06-29T14:41:02.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.163+0000 7f647bfff640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f6480076590 0x7f6480078aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.164+0000 7f647bfff640 1 -- 172.21.15.81:0/3079687498 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7f64840b8650 con 0x7f6494071f20 2024-06-29T14:41:02.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.164+0000 7f6492ffd640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f6480076590 0x7f6480078aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.164+0000 7f6492ffd640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f6480076590 0x7f6480078aa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f64941e7770 tx=0x7f6488009040 comp rx=0 tx=0).ready entity=mgr.6770 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:02.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.168+0000 7f647bfff640 1 -- 172.21.15.81:0/3079687498 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6484081310 con 0x7f6494071f20 2024-06-29T14:41:02.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.368+0000 7f6499ab3640 1 -- 172.21.15.81:0/3079687498 --> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f64940b1c20 con 0x7f6480076590 2024-06-29T14:41:02.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.451+0000 7f647bfff640 1 -- 172.21.15.81:0/3079687498 <== mgr.6770 v2:172.21.15.167:6832/1146990244 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f64940b1c20 con 0x7f6480076590 2024-06-29T14:41:02.449 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path snap_test_dir1 2024-06-29T14:41:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 -- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f6480076590 msgr2=0x7f6480078aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f6480076590 0x7f6480078aa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f64941e7770 tx=0x7f6488009040 comp rx=0 tx=0).stop 2024-06-29T14:41:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 -- 172.21.15.81:0/3079687498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6494071f20 msgr2=0x7f64941f5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6494071f20 0x7f64941f5980 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f6484034040 tx=0x7f648402ae70 comp rx=0 tx=0).stop 2024-06-29T14:41:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 -- 172.21.15.81:0/3079687498 shutdown_connections 2024-06-29T14:41:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f6480076590 0x7f6480078aa0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6494141ad0 0x7f6494141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6494145180 0x7f64940a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 --2- 172.21.15.81:0/3079687498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6494071f20 0x7f64941f5980 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 -- 172.21.15.81:0/3079687498 >> 172.21.15.81:0/3079687498 conn(0x7f649406ccc0 msgr2=0x7f649415e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:02.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 -- 172.21.15.81:0/3079687498 shutdown_connections 2024-06-29T14:41:02.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.453+0000 7f6499ab3640 1 -- 172.21.15.81:0/3079687498 wait complete. 2024-06-29T14:41:02.462 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-06-29T14:41:02.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:02.462 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T14:41:02.478 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_41_00_UTC 2024-06-29T14:41:02.478 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_41_00_UTC 2024-06-29T14:41:02.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:02.479 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_41_00_UTC) 2024-06-29T14:41:02.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:02.538 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T14:41:02.596 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:41:02.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.754+0000 7faa90e1f640 1 -- 172.21.15.81:0/1859678974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa78003fb0 msgr2=0x7faa78004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.754+0000 7faa90e1f640 1 --2- 172.21.15.81:0/1859678974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa78003fb0 0x7faa78004800 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0514c0 tx=0x7faa78005830 comp rx=0 tx=0).stop 2024-06-29T14:41:02.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.754+0000 7faa8ad9d640 1 -- 172.21.15.81:0/1859678974 reap_dead start 2024-06-29T14:41:02.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.755+0000 7faa90e1f640 1 -- 172.21.15.81:0/1859678974 shutdown_connections 2024-06-29T14:41:02.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.755+0000 7faa90e1f640 1 -- 172.21.15.81:0/1859678974 >> 172.21.15.81:0/1859678974 conn(0x7faa8c0d0930 msgr2=0x7faa8c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:02.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.755+0000 7faa90e1f640 1 -- 172.21.15.81:0/1859678974 shutdown_connections 2024-06-29T14:41:02.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.755+0000 7faa90e1f640 1 -- 172.21.15.81:0/1859678974 wait complete. 2024-06-29T14:41:02.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.757+0000 7faa90e1f640 1 Processor -- start 2024-06-29T14:41:02.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.757+0000 7faa90e1f640 1 -- start start 2024-06-29T14:41:02.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa90e1f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa78003fb0 0x7faa8c14ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa90e1f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa8c155820 0x7faa8c1516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa90e1f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faa8c06c2f0 0x7faa8c06c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa90e1f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c15b260 con 0x7faa78003fb0 2024-06-29T14:41:02.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa90e1f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c15b0e0 con 0x7faa8c06c2f0 2024-06-29T14:41:02.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa90e1f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faa8c15b3e0 con 0x7faa8c155820 2024-06-29T14:41:02.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa8a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa78003fb0 0x7faa8c14ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa8a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa78003fb0 0x7faa8c14ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34902/0 (socket says 172.21.15.81:34902) 2024-06-29T14:41:02.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa8a59c640 1 -- 172.21.15.81:0/2183862627 learned_addr learned my addr 172.21.15.81:0/2183862627 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:02.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa89d9b640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa8c155820 0x7faa8c1516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa8ad9d640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faa8c06c2f0 0x7faa8c06c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa8a59c640 1 -- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa8c155820 msgr2=0x7faa8c1516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa8a59c640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa8c155820 0x7faa8c1516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa8a59c640 1 -- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faa8c06c2f0 msgr2=0x7faa8c06c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa8a59c640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faa8c06c2f0 0x7faa8c06c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa8a59c640 1 -- 172.21.15.81:0/2183862627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8c06cc40 con 0x7faa78003fb0 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.758+0000 7faa8a59c640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa78003fb0 0x7faa8c14ee80 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7faa7c010c70 tx=0x7faa7c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.759+0000 7faa82ffd640 1 -- 172.21.15.81:0/2183862627 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c003b70 con 0x7faa78003fb0 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.759+0000 7faa82ffd640 1 -- 172.21.15.81:0/2183862627 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa7c003e00 con 0x7faa78003fb0 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.759+0000 7faa82ffd640 1 -- 172.21.15.81:0/2183862627 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7c0042b0 con 0x7faa78003fb0 2024-06-29T14:41:02.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.759+0000 7faa90e1f640 1 -- 172.21.15.81:0/2183862627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8c14f3c0 con 0x7faa78003fb0 2024-06-29T14:41:02.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.759+0000 7faa90e1f640 1 -- 172.21.15.81:0/2183862627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8c0c6190 con 0x7faa78003fb0 2024-06-29T14:41:02.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.760+0000 7faa90e1f640 1 -- 172.21.15.81:0/2183862627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa8c06e2a0 con 0x7faa78003fb0 2024-06-29T14:41:02.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.760+0000 7faa82ffd640 1 -- 172.21.15.81:0/2183862627 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 63) ==== 98434+0+0 (secure 0 0 0) 0x7faa7c004450 con 0x7faa78003fb0 2024-06-29T14:41:02.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.761+0000 7faa82ffd640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa6c076560 0x7faa6c078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:02.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.761+0000 7faa82ffd640 1 -- 172.21.15.81:0/2183862627 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7faa7c098fc0 con 0x7faa78003fb0 2024-06-29T14:41:02.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.761+0000 7faa89d9b640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa6c076560 0x7faa6c078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:02.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.762+0000 7faa89d9b640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa6c076560 0x7faa6c078a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faa74033000 tx=0x7faa8c0b50b0 comp rx=0 tx=0).ready entity=mgr.6770 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:02.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.765+0000 7faa82ffd640 1 -- 172.21.15.81:0/2183862627 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faa8c06e2a0 con 0x7faa78003fb0 2024-06-29T14:41:02.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:02.994+0000 7faa90e1f640 1 -- 172.21.15.81:0/2183862627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7faa8c0b3d70 con 0x7faa78003fb0 2024-06-29T14:41:03.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.603+0000 7faa82ffd640 1 -- 172.21.15.81:0/2183862627 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v64) ==== 93+0+0 (secure 0 0 0) 0x7faa8c06cc40 con 0x7faa78003fb0 2024-06-29T14:41:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.606+0000 7faa90e1f640 1 -- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa6c076560 msgr2=0x7faa6c078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.606+0000 7faa90e1f640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa6c076560 0x7faa6c078a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faa74033000 tx=0x7faa8c0b50b0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.606+0000 7faa90e1f640 1 -- 172.21.15.81:0/2183862627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa78003fb0 msgr2=0x7faa8c14ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.606+0000 7faa90e1f640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa78003fb0 0x7faa8c14ee80 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7faa7c010c70 tx=0x7faa7c00e720 comp rx=0 tx=0).stop 2024-06-29T14:41:03.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.607+0000 7faa90e1f640 1 -- 172.21.15.81:0/2183862627 shutdown_connections 2024-06-29T14:41:03.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.607+0000 7faa90e1f640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa6c076560 0x7faa6c078a70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.607+0000 7faa90e1f640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faa8c06c2f0 0x7faa8c06c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.607+0000 7faa90e1f640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa8c155820 0x7faa8c1516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.607+0000 7faa90e1f640 1 --2- 172.21.15.81:0/2183862627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa78003fb0 0x7faa8c14ee80 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.607+0000 7faa90e1f640 1 -- 172.21.15.81:0/2183862627 >> 172.21.15.81:0/2183862627 conn(0x7faa8c0d0930 msgr2=0x7faa8c0ce190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:03.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.607+0000 7faa90e1f640 1 -- 172.21.15.81:0/2183862627 shutdown_connections 2024-06-29T14:41:03.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.607+0000 7faa90e1f640 1 -- 172.21.15.81:0/2183862627 wait complete. 2024-06-29T14:41:03.618 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:41:03.618 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:03.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:03.725 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:41:03.725 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:03.725 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:03.726 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:03.741 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:03.741 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:03.799 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:03.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.957+0000 7f727a9f0640 1 -- 172.21.15.81:0/2043158706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f725c004800 msgr2=0x7f725c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.957+0000 7f727a9f0640 1 --2- 172.21.15.81:0/2043158706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f725c004800 0x7f725c004c10 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f7268009b20 tx=0x7f725c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:03.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.957+0000 7f727a9f0640 1 -- 172.21.15.81:0/2043158706 shutdown_connections 2024-06-29T14:41:03.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.957+0000 7f727a9f0640 1 --2- 172.21.15.81:0/2043158706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f725c004800 0x7f725c004c10 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.957+0000 7f727a9f0640 1 -- 172.21.15.81:0/2043158706 >> 172.21.15.81:0/2043158706 conn(0x7f72740b3680 msgr2=0x7f72740b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:03.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.957+0000 7f727a9f0640 1 -- 172.21.15.81:0/2043158706 shutdown_connections 2024-06-29T14:41:03.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.957+0000 7f727a9f0640 1 -- 172.21.15.81:0/2043158706 wait complete. 2024-06-29T14:41:03.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.959+0000 7f727a9f0640 1 Processor -- start 2024-06-29T14:41:03.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.959+0000 7f727a9f0640 1 -- start start 2024-06-29T14:41:03.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f727a9f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f725c004800 0x7f7274159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f727a9f0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7274157c10 0x7f7274150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f727a9f0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727409d0d0 0x7f727409d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f727a9f0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72740cc930 con 0x7f725c004800 2024-06-29T14:41:03.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f727a9f0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f72740cc7b0 con 0x7f7274157c10 2024-06-29T14:41:03.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f727a9f0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f72740ccab0 con 0x7f727409d0d0 2024-06-29T14:41:03.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f7273fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f725c004800 0x7f7274159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:03.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f7273fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f725c004800 0x7f7274159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34934/0 (socket says 172.21.15.81:34934) 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f7273fff640 1 -- 172.21.15.81:0/1641198623 learned_addr learned my addr 172.21.15.81:0/1641198623 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f7278f8d640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727409d0d0 0x7f727409d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.960+0000 7f72737fe640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7274157c10 0x7f7274150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f7273fff640 1 -- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727409d0d0 msgr2=0x7f727409d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f7273fff640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727409d0d0 0x7f727409d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f7273fff640 1 -- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7274157c10 msgr2=0x7f7274150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f7273fff640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7274157c10 0x7f7274150f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f7273fff640 1 -- 172.21.15.81:0/1641198623 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7274158380 con 0x7f725c004800 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f7273fff640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f725c004800 0x7f7274159d70 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f7264006e10 tx=0x7f72640049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f7270ff9640 1 -- 172.21.15.81:0/1641198623 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7264015020 con 0x7f725c004800 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f7270ff9640 1 -- 172.21.15.81:0/1641198623 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7264011960 con 0x7f725c004800 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f727a9f0640 1 -- 172.21.15.81:0/1641198623 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7274158660 con 0x7f725c004800 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f7270ff9640 1 -- 172.21.15.81:0/1641198623 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7264011d90 con 0x7f725c004800 2024-06-29T14:41:03.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.961+0000 7f727a9f0640 1 -- 172.21.15.81:0/1641198623 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7274158c20 con 0x7f725c004800 2024-06-29T14:41:03.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.962+0000 7f727a9f0640 1 -- 172.21.15.81:0/1641198623 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72740b7460 con 0x7f725c004800 2024-06-29T14:41:03.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.963+0000 7f7270ff9640 1 -- 172.21.15.81:0/1641198623 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7f7264018020 con 0x7f725c004800 2024-06-29T14:41:03.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.963+0000 7f7270ff9640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f7260076510 0x7f7260078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:03.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.964+0000 7f72737fe640 1 -- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f7260076510 msgr2=0x7f7260078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:03.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.964+0000 7f7270ff9640 1 -- 172.21.15.81:0/1641198623 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7f7264098820 con 0x7f725c004800 2024-06-29T14:41:03.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.964+0000 7f72737fe640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f7260076510 0x7f7260078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:03.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:03.968+0000 7f7270ff9640 1 -- 172.21.15.81:0/1641198623 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f72740b7460 con 0x7f725c004800 2024-06-29T14:41:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.153+0000 7f727a9f0640 1 -- 172.21.15.81:0/1641198623 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72740cd010 con 0x7f725c004800 2024-06-29T14:41:04.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.155+0000 7f7270ff9640 1 -- 172.21.15.81:0/1641198623 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) ==== 74+0+17225 (secure 0 0 0) 0x7f726401f020 con 0x7f725c004800 2024-06-29T14:41:04.152 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:04.152 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":120,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:40:09.486435+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:55.492112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.145353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/586922734":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/1014005913":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/705657234":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/2888901478":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/1794407759":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/839096394":"2024-06-30T14:40:09.486395+0000","172.21.15.167:6833/473361950":"2024-06-30T14:40:09.486395+0000","172.21.15.167:6832/473361950":"2024-06-30T14:40:09.486395+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:04.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 -- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f7260076510 msgr2=0x7f7260078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:04.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f7260076510 0x7f7260078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 -- 172.21.15.81:0/1641198623 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f725c004800 msgr2=0x7f7274159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f725c004800 0x7f7274159d70 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f7264006e10 tx=0x7f72640049d0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 -- 172.21.15.81:0/1641198623 shutdown_connections 2024-06-29T14:41:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f7260076510 0x7f7260078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727409d0d0 0x7f727409d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7274157c10 0x7f7274150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 --2- 172.21.15.81:0/1641198623 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f725c004800 0x7f7274159d70 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 -- 172.21.15.81:0/1641198623 >> 172.21.15.81:0/1641198623 conn(0x7f72740b3680 msgr2=0x7f72740b64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 -- 172.21.15.81:0/1641198623 shutdown_connections 2024-06-29T14:41:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.157+0000 7f727a9f0640 1 -- 172.21.15.81:0/1641198623 wait complete. 2024-06-29T14:41:04.167 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:39:55.492112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:39:56.145353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:04.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:04.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.289+0000 7faa48a4d640 1 -- 172.21.15.81:0/4044983216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa440d0940 msgr2=0x7faa440b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.289+0000 7faa48a4d640 1 --2- 172.21.15.81:0/4044983216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa440d0940 0x7faa440b4f40 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7faa3c005fd0 tx=0x7faa3c02aa90 comp rx=0 tx=0).stop 2024-06-29T14:41:04.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.289+0000 7faa48a4d640 1 -- 172.21.15.81:0/4044983216 shutdown_connections 2024-06-29T14:41:04.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.289+0000 7faa48a4d640 1 --2- 172.21.15.81:0/4044983216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa440d0940 0x7faa440b4f40 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.289+0000 7faa48a4d640 1 --2- 172.21.15.81:0/4044983216 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa440b3200 0x7faa440cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.289+0000 7faa48a4d640 1 -- 172.21.15.81:0/4044983216 >> 172.21.15.81:0/4044983216 conn(0x7faa440b3680 msgr2=0x7faa440b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.289+0000 7faa48a4d640 1 -- 172.21.15.81:0/4044983216 shutdown_connections 2024-06-29T14:41:04.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.289+0000 7faa48a4d640 1 -- 172.21.15.81:0/4044983216 wait complete. 2024-06-29T14:41:04.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.291+0000 7faa48a4d640 1 Processor -- start 2024-06-29T14:41:04.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.291+0000 7faa48a4d640 1 -- start start 2024-06-29T14:41:04.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa48a4d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa440b3200 0x7faa4409caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa48a4d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa440cccf0 0x7faa4409cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa48a4d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faa4409d520 0x7faa44156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa48a4d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faa440b5c00 con 0x7faa440b3200 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa48a4d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faa440b5a80 con 0x7faa4409d520 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa48a4d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faa440b5d80 con 0x7faa440cccf0 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa4259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa440b3200 0x7faa4409caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa4259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa440b3200 0x7faa4409caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34952/0 (socket says 172.21.15.81:34952) 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa4259c640 1 -- 172.21.15.81:0/4005275733 learned_addr learned my addr 172.21.15.81:0/4005275733 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa3bfff640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa440cccf0 0x7faa4409cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.292+0000 7faa42d9d640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faa4409d520 0x7faa44156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa4259c640 1 -- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa440cccf0 msgr2=0x7faa4409cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa4259c640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa440cccf0 0x7faa4409cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa4259c640 1 -- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faa4409d520 msgr2=0x7faa44156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa4259c640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faa4409d520 0x7faa44156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa4259c640 1 -- 172.21.15.81:0/4005275733 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa44159780 con 0x7faa440b3200 2024-06-29T14:41:04.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa4259c640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa440b3200 0x7faa4409caa0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7faa3400d770 tx=0x7faa3400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:04.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa3affd640 1 -- 172.21.15.81:0/4005275733 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34017070 con 0x7faa440b3200 2024-06-29T14:41:04.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa3affd640 1 -- 172.21.15.81:0/4005275733 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa34009d20 con 0x7faa440b3200 2024-06-29T14:41:04.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa48a4d640 1 -- 172.21.15.81:0/4005275733 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa441e1fc0 con 0x7faa440b3200 2024-06-29T14:41:04.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa3affd640 1 -- 172.21.15.81:0/4005275733 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa340106c0 con 0x7faa440b3200 2024-06-29T14:41:04.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.293+0000 7faa48a4d640 1 -- 172.21.15.81:0/4005275733 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa441e2580 con 0x7faa440b3200 2024-06-29T14:41:04.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.294+0000 7faa48a4d640 1 -- 172.21.15.81:0/4005275733 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa440b6ad0 con 0x7faa440b3200 2024-06-29T14:41:04.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.295+0000 7faa3affd640 1 -- 172.21.15.81:0/4005275733 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7faa3401a070 con 0x7faa440b3200 2024-06-29T14:41:04.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.295+0000 7faa3affd640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa24076540 0x7faa24078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.295+0000 7faa3bfff640 1 -- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa24076540 msgr2=0x7faa24078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:04.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.295+0000 7faa3affd640 1 -- 172.21.15.81:0/4005275733 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7faa34097e70 con 0x7faa440b3200 2024-06-29T14:41:04.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.295+0000 7faa3bfff640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa24076540 0x7faa24078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:04.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.304+0000 7faa3affd640 1 -- 172.21.15.81:0/4005275733 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faa440b6ad0 con 0x7faa440b3200 2024-06-29T14:41:04.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.491+0000 7faa48a4d640 1 -- 172.21.15.81:0/4005275733 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa440cc7b0 con 0x7faa440b3200 2024-06-29T14:41:04.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.491+0000 7faa3affd640 1 -- 172.21.15.81:0/4005275733 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) ==== 93+0+10 (secure 0 0 0) 0x7faa34060ae0 con 0x7faa440b3200 2024-06-29T14:41:04.489 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:41:04.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 -- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa24076540 msgr2=0x7faa24078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:04.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa24076540 0x7faa24078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 -- 172.21.15.81:0/4005275733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa440b3200 msgr2=0x7faa4409caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa440b3200 0x7faa4409caa0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7faa3400d770 tx=0x7faa3400dc30 comp rx=0 tx=0).stop 2024-06-29T14:41:04.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 -- 172.21.15.81:0/4005275733 shutdown_connections 2024-06-29T14:41:04.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faa24076540 0x7faa24078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faa4409d520 0x7faa44156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faa440cccf0 0x7faa4409cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 --2- 172.21.15.81:0/4005275733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa440b3200 0x7faa4409caa0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 -- 172.21.15.81:0/4005275733 >> 172.21.15.81:0/4005275733 conn(0x7faa440b3680 msgr2=0x7faa440cba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 -- 172.21.15.81:0/4005275733 shutdown_connections 2024-06-29T14:41:04.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.493+0000 7faa48a4d640 1 -- 172.21.15.81:0/4005275733 wait complete. 2024-06-29T14:41:04.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:41:04.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.625+0000 7f295c088640 1 -- 172.21.15.81:0/2844845094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29540b46a0 msgr2=0x7f2954072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.625+0000 7f295c088640 1 --2- 172.21.15.81:0/2844845094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29540b46a0 0x7f2954072f40 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f2954072d10 tx=0x7f294402a5f0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.625+0000 7f295c088640 1 -- 172.21.15.81:0/2844845094 shutdown_connections 2024-06-29T14:41:04.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.625+0000 7f295c088640 1 --2- 172.21.15.81:0/2844845094 >> v2:172.21.15.167:3300/0 conn(0x7f2954073480 0x7f295415b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.625+0000 7f295c088640 1 --2- 172.21.15.81:0/2844845094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29540b46a0 0x7f2954072f40 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.625+0000 7f295c088640 1 -- 172.21.15.81:0/2844845094 >> 172.21.15.81:0/2844845094 conn(0x7f29540d0b70 msgr2=0x7f29540d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.625+0000 7f295c088640 1 -- 172.21.15.81:0/2844845094 shutdown_connections 2024-06-29T14:41:04.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.625+0000 7f295c088640 1 -- 172.21.15.81:0/2844845094 wait complete. 2024-06-29T14:41:04.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.628+0000 7f295c088640 1 Processor -- start 2024-06-29T14:41:04.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.628+0000 7f295c088640 1 -- start start 2024-06-29T14:41:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295c088640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2954073480 0x7f295413e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295c088640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29540b46a0 0x7f2954150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295c088640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f295415b720 0x7f29540ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295c088640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f295406f640 con 0x7f295415b720 2024-06-29T14:41:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295c088640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f295406f4c0 con 0x7f2954073480 2024-06-29T14:41:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295c088640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f295406f7c0 con 0x7f29540b46a0 2024-06-29T14:41:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295a625640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f295415b720 0x7f29540ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295a625640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f295415b720 0x7f29540ad910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34968/0 (socket says 172.21.15.81:34968) 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295a625640 1 -- 172.21.15.81:0/2723771802 learned_addr learned my addr 172.21.15.81:0/2723771802 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f2959e24640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2954073480 0x7f295413e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f2959623640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29540b46a0 0x7f2954150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295a625640 1 -- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29540b46a0 msgr2=0x7f2954150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295a625640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29540b46a0 0x7f2954150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295a625640 1 -- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2954073480 msgr2=0x7f295413e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295a625640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2954073480 0x7f295413e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295a625640 1 -- 172.21.15.81:0/2723771802 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29540ac870 con 0x7f295415b720 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f295a625640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f295415b720 0x7f29540ad910 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f294c00c700 tx=0x7f294c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f294a7fc640 1 -- 172.21.15.81:0/2723771802 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c003b70 con 0x7f295415b720 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f294a7fc640 1 -- 172.21.15.81:0/2723771802 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f294c003d10 con 0x7f295415b720 2024-06-29T14:41:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.631+0000 7f294a7fc640 1 -- 172.21.15.81:0/2723771802 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c00aa50 con 0x7f295415b720 2024-06-29T14:41:04.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.632+0000 7f295c088640 1 -- 172.21.15.81:0/2723771802 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29540acb50 con 0x7f295415b720 2024-06-29T14:41:04.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.632+0000 7f295c088640 1 -- 172.21.15.81:0/2723771802 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29540ad110 con 0x7f295415b720 2024-06-29T14:41:04.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.633+0000 7f295c088640 1 -- 172.21.15.81:0/2723771802 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f295415b1e0 con 0x7f295415b720 2024-06-29T14:41:04.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.633+0000 7f294a7fc640 1 -- 172.21.15.81:0/2723771802 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7f294c00abf0 con 0x7f295415b720 2024-06-29T14:41:04.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.634+0000 7f294a7fc640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2938076540 0x7f2938078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.634+0000 7f2959e24640 1 -- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2938076540 msgr2=0x7f2938078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:04.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.634+0000 7f294a7fc640 1 -- 172.21.15.81:0/2723771802 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7f294c097a60 con 0x7f295415b720 2024-06-29T14:41:04.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.634+0000 7f2959e24640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2938076540 0x7f2938078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:04.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.638+0000 7f294a7fc640 1 -- 172.21.15.81:0/2723771802 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2954073890 con 0x7f295415b720 2024-06-29T14:41:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.826+0000 7f295c088640 1 -- 172.21.15.81:0/2723771802 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f29540ac590 con 0x7f295415b720 2024-06-29T14:41:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.826+0000 7f294a7fc640 1 -- 172.21.15.81:0/2723771802 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) ==== 104+0+11 (secure 0 0 0) 0x7f294c0607e0 con 0x7f295415b720 2024-06-29T14:41:04.824 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:41:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 -- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2938076540 msgr2=0x7f2938078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2938076540 0x7f2938078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 -- 172.21.15.81:0/2723771802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f295415b720 msgr2=0x7f29540ad910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f295415b720 0x7f29540ad910 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f294c00c700 tx=0x7f294c00cbc0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 -- 172.21.15.81:0/2723771802 shutdown_connections 2024-06-29T14:41:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2938076540 0x7f2938078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f295415b720 0x7f29540ad910 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29540b46a0 0x7f2954150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 --2- 172.21.15.81:0/2723771802 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2954073480 0x7f295413e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 -- 172.21.15.81:0/2723771802 >> 172.21.15.81:0/2723771802 conn(0x7f29540d0b70 msgr2=0x7f295406edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 -- 172.21.15.81:0/2723771802 shutdown_connections 2024-06-29T14:41:04.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.828+0000 7f295c088640 1 -- 172.21.15.81:0/2723771802 wait complete. 2024-06-29T14:41:04.837 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:41:04.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.958+0000 7f2e75b2a640 1 -- 172.21.15.81:0/1008604536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e700b9040 msgr2=0x7f2e700d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.958+0000 7f2e75b2a640 1 --2- 172.21.15.81:0/1008604536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e700b9040 0x7f2e700d0480 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009830 tx=0x7f2e6002a530 comp rx=0 tx=0).stop 2024-06-29T14:41:04.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.958+0000 7f2e75b2a640 1 -- 172.21.15.81:0/1008604536 shutdown_connections 2024-06-29T14:41:04.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.958+0000 7f2e75b2a640 1 --2- 172.21.15.81:0/1008604536 >> v2:172.21.15.167:3300/0 conn(0x7f2e700d2a90 0x7f2e700d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.958+0000 7f2e75b2a640 1 --2- 172.21.15.81:0/1008604536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e700b9040 0x7f2e700d0480 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.958+0000 7f2e75b2a640 1 -- 172.21.15.81:0/1008604536 >> 172.21.15.81:0/1008604536 conn(0x7f2e7009cf30 msgr2=0x7f2e700d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:04.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.959+0000 7f2e75b2a640 1 -- 172.21.15.81:0/1008604536 shutdown_connections 2024-06-29T14:41:04.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.959+0000 7f2e75b2a640 1 -- 172.21.15.81:0/1008604536 wait complete. 2024-06-29T14:41:04.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.960+0000 7f2e75b2a640 1 Processor -- start 2024-06-29T14:41:04.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.960+0000 7f2e75b2a640 1 -- start start 2024-06-29T14:41:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e75b2a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e700b5250 0x7f2e70154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e75b2a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e700b9040 0x7f2e700c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e75b2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e700d2a90 0x7f2e700c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e75b2a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2e700b7d30 con 0x7f2e700d2a90 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e75b2a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2e700b7bb0 con 0x7f2e700b5250 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e75b2a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2e700b7eb0 con 0x7f2e700b9040 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e700d2a90 0x7f2e700c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e700d2a90 0x7f2e700c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34984/0 (socket says 172.21.15.81:34984) 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6ffff640 1 -- 172.21.15.81:0/3513383471 learned_addr learned my addr 172.21.15.81:0/3513383471 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6f7fe640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e700b5250 0x7f2e70154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6effd640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e700b9040 0x7f2e700c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6ffff640 1 -- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e700b9040 msgr2=0x7f2e700c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6ffff640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e700b9040 0x7f2e700c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6ffff640 1 -- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e700b5250 msgr2=0x7f2e70154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:04.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6ffff640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e700b5250 0x7f2e70154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6ffff640 1 -- 172.21.15.81:0/3513383471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e700c7ef0 con 0x7f2e700d2a90 2024-06-29T14:41:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.962+0000 7f2e6ffff640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e700d2a90 0x7f2e700c8bf0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800ec20 tx=0x7f2e6800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.963+0000 7f2e57fff640 1 -- 172.21.15.81:0/3513383471 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68003b70 con 0x7f2e700d2a90 2024-06-29T14:41:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.963+0000 7f2e57fff640 1 -- 172.21.15.81:0/3513383471 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e68003d10 con 0x7f2e700d2a90 2024-06-29T14:41:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.963+0000 7f2e75b2a640 1 -- 172.21.15.81:0/3513383471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7006e260 con 0x7f2e700d2a90 2024-06-29T14:41:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.963+0000 7f2e57fff640 1 -- 172.21.15.81:0/3513383471 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68004280 con 0x7f2e700d2a90 2024-06-29T14:41:04.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.963+0000 7f2e75b2a640 1 -- 172.21.15.81:0/3513383471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e700c8190 con 0x7f2e700d2a90 2024-06-29T14:41:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.964+0000 7f2e75b2a640 1 -- 172.21.15.81:0/3513383471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e700b83b0 con 0x7f2e700d2a90 2024-06-29T14:41:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.964+0000 7f2e57fff640 1 -- 172.21.15.81:0/3513383471 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7f2e68004420 con 0x7f2e700d2a90 2024-06-29T14:41:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.965+0000 7f2e57fff640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2e5c076540 0x7f2e5c078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.965+0000 7f2e6f7fe640 1 -- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2e5c076540 msgr2=0x7f2e5c078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.965+0000 7f2e57fff640 1 -- 172.21.15.81:0/3513383471 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7f2e68098860 con 0x7f2e700d2a90 2024-06-29T14:41:04.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.965+0000 7f2e6f7fe640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2e5c076540 0x7f2e5c078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:04.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:04.969+0000 7f2e57fff640 1 -- 172.21.15.81:0/3513383471 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2e68061560 con 0x7f2e700d2a90 2024-06-29T14:41:05.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.157+0000 7f2e75b2a640 1 -- 172.21.15.81:0/3513383471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2e700b1450 con 0x7f2e700d2a90 2024-06-29T14:41:05.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.157+0000 7f2e57fff640 1 -- 172.21.15.81:0/3513383471 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v120) ==== 100+0+11 (secure 0 0 0) 0x7f2e68065860 con 0x7f2e700d2a90 2024-06-29T14:41:05.155 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 -- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2e5c076540 msgr2=0x7f2e5c078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2e5c076540 0x7f2e5c078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 -- 172.21.15.81:0/3513383471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e700d2a90 msgr2=0x7f2e700c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e700d2a90 0x7f2e700c8bf0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800ec20 tx=0x7f2e6800c6a0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 -- 172.21.15.81:0/3513383471 shutdown_connections 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f2e5c076540 0x7f2e5c078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e700d2a90 0x7f2e700c8bf0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e700b9040 0x7f2e700c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 --2- 172.21.15.81:0/3513383471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e700b5250 0x7f2e70154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 -- 172.21.15.81:0/3513383471 >> 172.21.15.81:0/3513383471 conn(0x7f2e7009cf30 msgr2=0x7f2e700d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:05.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 -- 172.21.15.81:0/3513383471 shutdown_connections 2024-06-29T14:41:05.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.159+0000 7f2e75b2a640 1 -- 172.21.15.81:0/3513383471 wait complete. 2024-06-29T14:41:05.169 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:41:05.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.287+0000 7f3c14d31640 1 -- 172.21.15.81:0/3060993335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c100b9040 msgr2=0x7f3c100d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.287+0000 7f3c14d31640 1 --2- 172.21.15.81:0/3060993335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c100b9040 0x7f3c100d0480 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009830 tx=0x7f3c0002a640 comp rx=0 tx=0).stop 2024-06-29T14:41:05.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.287+0000 7f3c14d31640 1 -- 172.21.15.81:0/3060993335 shutdown_connections 2024-06-29T14:41:05.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.287+0000 7f3c14d31640 1 --2- 172.21.15.81:0/3060993335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c100b9040 0x7f3c100d0480 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.287+0000 7f3c14d31640 1 -- 172.21.15.81:0/3060993335 >> 172.21.15.81:0/3060993335 conn(0x7f3c1009cf30 msgr2=0x7f3c100d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:05.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.287+0000 7f3c14d31640 1 -- 172.21.15.81:0/3060993335 shutdown_connections 2024-06-29T14:41:05.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.287+0000 7f3c14d31640 1 -- 172.21.15.81:0/3060993335 wait complete. 2024-06-29T14:41:05.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.289+0000 7f3c14d31640 1 Processor -- start 2024-06-29T14:41:05.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.289+0000 7f3c14d31640 1 -- start start 2024-06-29T14:41:05.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c14d31640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c100b5250 0x7f3c1015e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c14d31640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c100b9040 0x7f3c100b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c14d31640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c100d2a90 0x7f3c1015fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c14d31640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3c100d0d50 con 0x7f3c100b9040 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c14d31640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3c100d0bd0 con 0x7f3c100d2a90 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c14d31640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3c100b80b0 con 0x7f3c100b5250 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c100b9040 0x7f3c100b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c100b9040 0x7f3c100b3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34998/0 (socket says 172.21.15.81:34998) 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0dd9b640 1 -- 172.21.15.81:0/133703226 learned_addr learned my addr 172.21.15.81:0/133703226 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0e59c640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c100b5250 0x7f3c1015e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0ed9d640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c100d2a90 0x7f3c1015fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0dd9b640 1 -- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c100b5250 msgr2=0x7f3c1015e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0dd9b640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c100b5250 0x7f3c1015e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0dd9b640 1 -- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c100d2a90 msgr2=0x7f3c1015fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0dd9b640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c100d2a90 0x7f3c1015fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0dd9b640 1 -- 172.21.15.81:0/133703226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c101e7520 con 0x7f3c100b9040 2024-06-29T14:41:05.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c0dd9b640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c100b9040 0x7f3c100b3400 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800c400 tx=0x7f3bf800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:05.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c06ffd640 1 -- 172.21.15.81:0/133703226 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8011790 con 0x7f3c100b9040 2024-06-29T14:41:05.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c06ffd640 1 -- 172.21.15.81:0/133703226 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bf8017070 con 0x7f3c100b9040 2024-06-29T14:41:05.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.291+0000 7f3c06ffd640 1 -- 172.21.15.81:0/133703226 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8011ce0 con 0x7f3c100b9040 2024-06-29T14:41:05.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.292+0000 7f3c14d31640 1 -- 172.21.15.81:0/133703226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c101e7800 con 0x7f3c100b9040 2024-06-29T14:41:05.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.292+0000 7f3c14d31640 1 -- 172.21.15.81:0/133703226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c101e7dc0 con 0x7f3c100b9040 2024-06-29T14:41:05.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.293+0000 7f3c14d31640 1 -- 172.21.15.81:0/133703226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c100d0480 con 0x7f3c100b9040 2024-06-29T14:41:05.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.293+0000 7f3c06ffd640 1 -- 172.21.15.81:0/133703226 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7f3bf801c460 con 0x7f3c100b9040 2024-06-29T14:41:05.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.294+0000 7f3c06ffd640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f3bf0076510 0x7f3bf0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.294+0000 7f3c0e59c640 1 -- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f3bf0076510 msgr2=0x7f3bf0078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:05.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.294+0000 7f3c06ffd640 1 -- 172.21.15.81:0/133703226 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7f3bf8097a80 con 0x7f3c100b9040 2024-06-29T14:41:05.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.294+0000 7f3c0e59c640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f3bf0076510 0x7f3bf0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:05.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.298+0000 7f3c06ffd640 1 -- 172.21.15.81:0/133703226 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3bf80608d0 con 0x7f3c100b9040 2024-06-29T14:41:05.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.495+0000 7f3c0e59c640 1 -- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f3bf0076510 msgr2=0x7f3bf0078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:05.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.495+0000 7f3c0e59c640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f3bf0076510 0x7f3bf0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:05.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.541+0000 7f3c14d31640 1 -- 172.21.15.81:0/133703226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3c101e7210 con 0x7f3c100b9040 2024-06-29T14:41:05.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.541+0000 7f3c06ffd640 1 -- 172.21.15.81:0/133703226 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v120) ==== 80+0+473 (secure 0 0 0) 0x7f3bf8064bd0 con 0x7f3c100b9040 2024-06-29T14:41:05.539 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/586922734 2024-06-30T14:40:09.486395+0000 2024-06-29T14:41:05.539 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1014005913 2024-06-30T14:40:09.486395+0000 2024-06-29T14:41:05.539 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/705657234 2024-06-30T14:40:09.486395+0000 2024-06-29T14:41:05.539 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2888901478 2024-06-30T14:40:09.486395+0000 2024-06-29T14:41:05.539 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1794407759 2024-06-30T14:40:09.486395+0000 2024-06-29T14:41:05.539 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/839096394 2024-06-30T14:40:09.486395+0000 2024-06-29T14:41:05.539 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/473361950 2024-06-30T14:40:09.486395+0000 2024-06-29T14:41:05.539 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/473361950 2024-06-30T14:40:09.486395+0000 2024-06-29T14:41:05.539 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 -- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f3bf0076510 msgr2=0x7f3bf0078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f3bf0076510 0x7f3bf0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 -- 172.21.15.81:0/133703226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c100b9040 msgr2=0x7f3c100b3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c100b9040 0x7f3c100b3400 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800c400 tx=0x7f3bf800c8c0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 -- 172.21.15.81:0/133703226 shutdown_connections 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7f3bf0076510 0x7f3bf0078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c100d2a90 0x7f3c1015fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c100b9040 0x7f3c100b3400 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 --2- 172.21.15.81:0/133703226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c100b5250 0x7f3c1015e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 -- 172.21.15.81:0/133703226 >> 172.21.15.81:0/133703226 conn(0x7f3c1009cf30 msgr2=0x7f3c100685b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 -- 172.21.15.81:0/133703226 shutdown_connections 2024-06-29T14:41:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.543+0000 7f3c14d31640 1 -- 172.21.15.81:0/133703226 wait complete. 2024-06-29T14:41:05.552 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:41:05.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:05.552 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:05.582 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:41:05.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:05.582 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:41:05.638 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:05.639 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:41:05.639 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:05.654 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:05.654 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:05.655 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:05.655 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:05.695 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:05.695 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:05.695 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:41:05.695 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:05.713 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:05.713 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:05.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.853+0000 7fd80f7c7640 1 -- 172.21.15.81:0/1363917694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8080d0230 msgr2=0x7fd8080d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.853+0000 7fd80f7c7640 1 --2- 172.21.15.81:0/1363917694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8080d0230 0x7fd8080d0640 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fd8080c5590 tx=0x7fd7f802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.853+0000 7fd80f7c7640 1 -- 172.21.15.81:0/1363917694 shutdown_connections 2024-06-29T14:41:05.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.853+0000 7fd80f7c7640 1 --2- 172.21.15.81:0/1363917694 >> v2:172.21.15.167:3300/0 conn(0x7fd8080b4f40 0x7fd8080b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.853+0000 7fd80f7c7640 1 --2- 172.21.15.81:0/1363917694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8080d0230 0x7fd8080d0640 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.853+0000 7fd80f7c7640 1 -- 172.21.15.81:0/1363917694 >> 172.21.15.81:0/1363917694 conn(0x7fd8080b3680 msgr2=0x7fd8080b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:05.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.854+0000 7fd80f7c7640 1 -- 172.21.15.81:0/1363917694 shutdown_connections 2024-06-29T14:41:05.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.854+0000 7fd80f7c7640 1 -- 172.21.15.81:0/1363917694 wait complete. 2024-06-29T14:41:05.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.855+0000 7fd80f7c7640 1 Processor -- start 2024-06-29T14:41:05.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.855+0000 7fd80f7c7640 1 -- start start 2024-06-29T14:41:05.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80f7c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8080b3200 0x7fd80809c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80f7c7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8080b4f40 0x7fd8080ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80f7c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8080d0230 0x7fd808158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80f7c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd8080b5c20 con 0x7fd8080d0230 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80f7c7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd8080b5aa0 con 0x7fd8080b3200 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80f7c7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd8080b5da0 con 0x7fd8080b4f40 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80dd64640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8080d0230 0x7fd808158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80dd64640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8080d0230 0x7fd808158dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35014/0 (socket says 172.21.15.81:35014) 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80dd64640 1 -- 172.21.15.81:0/2862895251 learned_addr learned my addr 172.21.15.81:0/2862895251 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80d563640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8080b3200 0x7fd80809c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80cd62640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8080b4f40 0x7fd8080ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80dd64640 1 -- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8080b4f40 msgr2=0x7fd8080ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80dd64640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8080b4f40 0x7fd8080ae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80dd64640 1 -- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8080b3200 msgr2=0x7fd80809c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:05.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80dd64640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8080b3200 0x7fd80809c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:05.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80dd64640 1 -- 172.21.15.81:0/2862895251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd808147e50 con 0x7fd8080d0230 2024-06-29T14:41:05.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.857+0000 7fd80dd64640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8080d0230 0x7fd808158dc0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd80000ea80 tx=0x7fd80000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:05.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.858+0000 7fd7fdffb640 1 -- 172.21.15.81:0/2862895251 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800004b20 con 0x7fd8080d0230 2024-06-29T14:41:05.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.858+0000 7fd7fdffb640 1 -- 172.21.15.81:0/2862895251 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd800004ce0 con 0x7fd8080d0230 2024-06-29T14:41:05.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.858+0000 7fd80f7c7640 1 -- 172.21.15.81:0/2862895251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd80809d400 con 0x7fd8080d0230 2024-06-29T14:41:05.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.858+0000 7fd7fdffb640 1 -- 172.21.15.81:0/2862895251 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800005280 con 0x7fd8080d0230 2024-06-29T14:41:05.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.858+0000 7fd80f7c7640 1 -- 172.21.15.81:0/2862895251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd80809d120 con 0x7fd8080d0230 2024-06-29T14:41:05.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.859+0000 7fd80f7c7640 1 -- 172.21.15.81:0/2862895251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd80809cbb0 con 0x7fd8080d0230 2024-06-29T14:41:05.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.860+0000 7fd7fdffb640 1 -- 172.21.15.81:0/2862895251 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7fd800005420 con 0x7fd8080d0230 2024-06-29T14:41:05.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.861+0000 7fd7fdffb640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fd7ec076510 0x7fd7ec078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:05.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.861+0000 7fd7fdffb640 1 -- 172.21.15.81:0/2862895251 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7fd800098900 con 0x7fd8080d0230 2024-06-29T14:41:05.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.861+0000 7fd80d563640 1 -- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fd7ec076510 msgr2=0x7fd7ec078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:05.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.861+0000 7fd80d563640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fd7ec076510 0x7fd7ec078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:05.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:05.865+0000 7fd7fdffb640 1 -- 172.21.15.81:0/2862895251 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd80809d400 con 0x7fd8080d0230 2024-06-29T14:41:06.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.063+0000 7fd80d563640 1 -- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fd7ec076510 msgr2=0x7fd7ec078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:06.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.063+0000 7fd80d563640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fd7ec076510 0x7fd7ec078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:06.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.091+0000 7fd80f7c7640 1 -- 172.21.15.81:0/2862895251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8080b5350 con 0x7fd8080d0230 2024-06-29T14:41:06.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.092+0000 7fd7fdffb640 1 -- 172.21.15.81:0/2862895251 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5231 (secure 0 0 0) 0x7fd800061600 con 0x7fd8080d0230 2024-06-29T14:41:06.089 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:06.089 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":59,"btime":"2024-06-29T14:40:00:847465+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1563799167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1563799167},{"type":"v1","addr":"172.21.15.167:6837","nonce":1563799167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7354,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3062579579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3062579579},{"type":"v1","addr":"172.21.15.81:6835","nonce":3062579579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7363,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3427077146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3427077146},{"type":"v1","addr":"172.21.15.167:6836","nonce":3427077146}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.183979+0000","modified":"2024-06-29T14:40:00.217351+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/1049868964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1049868964},{"type":"v1","addr":"172.21.15.81:6833","nonce":1049868964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8022,"qdb_cluster":[8022]},"id":7}]} 2024-06-29T14:41:06.089 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 59 2024-06-29T14:41:06.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 -- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fd7ec076510 msgr2=0x7fd7ec078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:06.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fd7ec076510 0x7fd7ec078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 -- 172.21.15.81:0/2862895251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8080d0230 msgr2=0x7fd808158dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8080d0230 0x7fd808158dc0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd80000ea80 tx=0x7fd80000ef40 comp rx=0 tx=0).stop 2024-06-29T14:41:06.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 -- 172.21.15.81:0/2862895251 shutdown_connections 2024-06-29T14:41:06.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fd7ec076510 0x7fd7ec078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8080d0230 0x7fd808158dc0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8080b4f40 0x7fd8080ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 --2- 172.21.15.81:0/2862895251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8080b3200 0x7fd80809c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 -- 172.21.15.81:0/2862895251 >> 172.21.15.81:0/2862895251 conn(0x7fd8080b3680 msgr2=0x7fd8081e34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 -- 172.21.15.81:0/2862895251 shutdown_connections 2024-06-29T14:41:06.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.094+0000 7fd80f7c7640 1 -- 172.21.15.81:0/2862895251 wait complete. 2024-06-29T14:41:06.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:06.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.221+0000 7faf2d991640 1 -- 172.21.15.81:0/706261401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf280cea50 msgr2=0x7faf280b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.221+0000 7faf2d991640 1 --2- 172.21.15.81:0/706261401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf280cea50 0x7faf280b4810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faf20005fd0 tx=0x7faf2002aa10 comp rx=0 tx=0).stop 2024-06-29T14:41:06.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.221+0000 7faf2d991640 1 -- 172.21.15.81:0/706261401 shutdown_connections 2024-06-29T14:41:06.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.221+0000 7faf2d991640 1 --2- 172.21.15.81:0/706261401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf280cea50 0x7faf280b4810 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.221+0000 7faf2d991640 1 --2- 172.21.15.81:0/706261401 >> v2:172.21.15.167:3300/0 conn(0x7faf280cd480 0x7faf280ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.221+0000 7faf2d991640 1 -- 172.21.15.81:0/706261401 >> 172.21.15.81:0/706261401 conn(0x7faf280b3510 msgr2=0x7faf280b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.221+0000 7faf2d991640 1 -- 172.21.15.81:0/706261401 shutdown_connections 2024-06-29T14:41:06.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.221+0000 7faf2d991640 1 -- 172.21.15.81:0/706261401 wait complete. 2024-06-29T14:41:06.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.222+0000 7faf2d991640 1 Processor -- start 2024-06-29T14:41:06.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.223+0000 7faf2d991640 1 -- start start 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf2d991640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf280b6eb0 0x7faf280ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf2d991640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf280cd480 0x7faf2806de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf2d991640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf280cea50 0x7faf2815e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf2d991640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf280b50e0 con 0x7faf280cea50 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf2d991640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faf280b4f60 con 0x7faf280cd480 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf2d991640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faf280b5260 con 0x7faf280b6eb0 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf280cea50 0x7faf2815e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf280cea50 0x7faf2815e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35018/0 (socket says 172.21.15.81:35018) 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf277fe640 1 -- 172.21.15.81:0/1505204031 learned_addr learned my addr 172.21.15.81:0/1505204031 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf26ffd640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf280b6eb0 0x7faf280ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf267fc640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf280cd480 0x7faf2806de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf277fe640 1 -- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf280b6eb0 msgr2=0x7faf280ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf277fe640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf280b6eb0 0x7faf280ca860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf277fe640 1 -- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf280cd480 msgr2=0x7faf2806de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf277fe640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf280cd480 0x7faf2806de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf277fe640 1 -- 172.21.15.81:0/1505204031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2815b410 con 0x7faf280cea50 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.224+0000 7faf26ffd640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf280b6eb0 0x7faf280ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.225+0000 7faf277fe640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf280cea50 0x7faf2815e800 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7faf20006010 tx=0x7faf20003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.225+0000 7faf0ffff640 1 -- 172.21.15.81:0/1505204031 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20046070 con 0x7faf280cea50 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.225+0000 7faf0ffff640 1 -- 172.21.15.81:0/1505204031 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf20004540 con 0x7faf280cea50 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.225+0000 7faf2d991640 1 -- 172.21.15.81:0/1505204031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2815d100 con 0x7faf280cea50 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.225+0000 7faf0ffff640 1 -- 172.21.15.81:0/1505204031 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20033070 con 0x7faf280cea50 2024-06-29T14:41:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.225+0000 7faf2d991640 1 -- 172.21.15.81:0/1505204031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2806d610 con 0x7faf280cea50 2024-06-29T14:41:06.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.226+0000 7faf2d991640 1 -- 172.21.15.81:0/1505204031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf28068df0 con 0x7faf280cea50 2024-06-29T14:41:06.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.227+0000 7faf0ffff640 1 -- 172.21.15.81:0/1505204031 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7faf20049070 con 0x7faf280cea50 2024-06-29T14:41:06.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.227+0000 7faf0ffff640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faf14076540 0x7faf14078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.227+0000 7faf26ffd640 1 -- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faf14076540 msgr2=0x7faf14078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:06.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.227+0000 7faf0ffff640 1 -- 172.21.15.81:0/1505204031 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7faf200b8cd0 con 0x7faf280cea50 2024-06-29T14:41:06.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.227+0000 7faf26ffd640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faf14076540 0x7faf14078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:06.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.232+0000 7faf0ffff640 1 -- 172.21.15.81:0/1505204031 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faf28068df0 con 0x7faf280cea50 2024-06-29T14:41:06.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.420+0000 7faf2d991640 1 -- 172.21.15.81:0/1505204031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf2806d400 con 0x7faf280cea50 2024-06-29T14:41:06.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.421+0000 7faf0ffff640 1 -- 172.21.15.81:0/1505204031 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v120) ==== 74+0+17225 (secure 0 0 0) 0x7faf200afe20 con 0x7faf280cea50 2024-06-29T14:41:06.419 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:06.419 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":120,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:40:09.486435+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:55.492112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.145353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/586922734":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/1014005913":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/705657234":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/2888901478":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/1794407759":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/839096394":"2024-06-30T14:40:09.486395+0000","172.21.15.167:6833/473361950":"2024-06-30T14:40:09.486395+0000","172.21.15.167:6832/473361950":"2024-06-30T14:40:09.486395+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:06.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.423+0000 7faf2d991640 1 -- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faf14076540 msgr2=0x7faf14078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:06.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.423+0000 7faf2d991640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faf14076540 0x7faf14078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.423+0000 7faf2d991640 1 -- 172.21.15.81:0/1505204031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf280cea50 msgr2=0x7faf2815e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.423+0000 7faf2d991640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf280cea50 0x7faf2815e800 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7faf20006010 tx=0x7faf20003f10 comp rx=0 tx=0).stop 2024-06-29T14:41:06.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.423+0000 7faf2d991640 1 -- 172.21.15.81:0/1505204031 shutdown_connections 2024-06-29T14:41:06.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.423+0000 7faf2d991640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7faf14076540 0x7faf14078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.423+0000 7faf2d991640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf280cea50 0x7faf2815e800 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.423+0000 7faf2d991640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf280cd480 0x7faf2806de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.423+0000 7faf2d991640 1 --2- 172.21.15.81:0/1505204031 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf280b6eb0 0x7faf280ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.423+0000 7faf2d991640 1 -- 172.21.15.81:0/1505204031 >> 172.21.15.81:0/1505204031 conn(0x7faf280b3510 msgr2=0x7faf280b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.424+0000 7faf2d991640 1 -- 172.21.15.81:0/1505204031 shutdown_connections 2024-06-29T14:41:06.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.424+0000 7faf2d991640 1 -- 172.21.15.81:0/1505204031 wait complete. 2024-06-29T14:41:06.432 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:39:55.492112+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:39:56.145353+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:06.433 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:06.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.554+0000 7fb7dff82640 1 -- 172.21.15.81:0/1753897434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d80b46a0 msgr2=0x7fb7d8072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.554+0000 7fb7dff82640 1 --2- 172.21.15.81:0/1753897434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d80b46a0 0x7fb7d8072f40 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005fd0 tx=0x7fb7c802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.554+0000 7fb7dff82640 1 -- 172.21.15.81:0/1753897434 shutdown_connections 2024-06-29T14:41:06.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.554+0000 7fb7dff82640 1 --2- 172.21.15.81:0/1753897434 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d815b8f0 0x7fb7d806edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.554+0000 7fb7dff82640 1 --2- 172.21.15.81:0/1753897434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d80b46a0 0x7fb7d8072f40 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.554+0000 7fb7dff82640 1 -- 172.21.15.81:0/1753897434 >> 172.21.15.81:0/1753897434 conn(0x7fb7d80d0b70 msgr2=0x7fb7d80d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.554+0000 7fb7dff82640 1 -- 172.21.15.81:0/1753897434 shutdown_connections 2024-06-29T14:41:06.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.554+0000 7fb7dff82640 1 -- 172.21.15.81:0/1753897434 wait complete. 2024-06-29T14:41:06.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.556+0000 7fb7dff82640 1 Processor -- start 2024-06-29T14:41:06.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.556+0000 7fb7dff82640 1 -- start start 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.557+0000 7fb7dff82640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d8073480 0x7fb7d8155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.557+0000 7fb7dff82640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d80b46a0 0x7fb7d81f4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.557+0000 7fb7dff82640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d815b8f0 0x7fb7d806c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.557+0000 7fb7dff82640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d806f680 con 0x7fb7d8073480 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.557+0000 7fb7dff82640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d806f500 con 0x7fb7d815b8f0 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.557+0000 7fb7dff82640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb7d806f800 con 0x7fb7d80b46a0 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ddd1e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d8073480 0x7fb7d8155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ddd1e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d8073480 0x7fb7d8155f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35026/0 (socket says 172.21.15.81:35026) 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ddd1e640 1 -- 172.21.15.81:0/3489583270 learned_addr learned my addr 172.21.15.81:0/3489583270 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7dd51d640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d80b46a0 0x7fb7d81f4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7de51f640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d815b8f0 0x7fb7d806c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ddd1e640 1 -- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d80b46a0 msgr2=0x7fb7d81f4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ddd1e640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d80b46a0 0x7fb7d81f4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ddd1e640 1 -- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d815b8f0 msgr2=0x7fb7d806c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ddd1e640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d815b8f0 0x7fb7d806c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ddd1e640 1 -- 172.21.15.81:0/3489583270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d8155d40 con 0x7fb7d8073480 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ddd1e640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d8073480 0x7fb7d8155f70 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8034040 tx=0x7fb7c80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ce7fc640 1 -- 172.21.15.81:0/3489583270 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c803e030 con 0x7fb7d8073480 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ce7fc640 1 -- 172.21.15.81:0/3489583270 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7c8003b00 con 0x7fb7d8073480 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7dff82640 1 -- 172.21.15.81:0/3489583270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d81f8f10 con 0x7fb7d8073480 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7ce7fc640 1 -- 172.21.15.81:0/3489583270 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8035040 con 0x7fb7d8073480 2024-06-29T14:41:06.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.558+0000 7fb7dff82640 1 -- 172.21.15.81:0/3489583270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d81f94d0 con 0x7fb7d8073480 2024-06-29T14:41:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.559+0000 7fb7dff82640 1 -- 172.21.15.81:0/3489583270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7d8073200 con 0x7fb7d8073480 2024-06-29T14:41:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.560+0000 7fb7ce7fc640 1 -- 172.21.15.81:0/3489583270 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7fb7c8041030 con 0x7fb7d8073480 2024-06-29T14:41:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.561+0000 7fb7ce7fc640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fb7bc076510 0x7fb7bc078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.561+0000 7fb7dd51d640 1 -- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fb7bc076510 msgr2=0x7fb7bc078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:06.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.561+0000 7fb7ce7fc640 1 -- 172.21.15.81:0/3489583270 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7fb7c80b8020 con 0x7fb7d8073480 2024-06-29T14:41:06.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.561+0000 7fb7dd51d640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fb7bc076510 0x7fb7bc078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:06.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.565+0000 7fb7ce7fc640 1 -- 172.21.15.81:0/3489583270 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb7d8073200 con 0x7fb7d8073480 2024-06-29T14:41:06.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.753+0000 7fb7dff82640 1 -- 172.21.15.81:0/3489583270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb7d8072f40 con 0x7fb7d8073480 2024-06-29T14:41:06.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.753+0000 7fb7ce7fc640 1 -- 172.21.15.81:0/3489583270 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v120) ==== 93+0+10 (secure 0 0 0) 0x7fb7c8080df0 con 0x7fb7d8073480 2024-06-29T14:41:06.751 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:41:06.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 -- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fb7bc076510 msgr2=0x7fb7bc078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fb7bc076510 0x7fb7bc078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 -- 172.21.15.81:0/3489583270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d8073480 msgr2=0x7fb7d8155f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d8073480 0x7fb7d8155f70 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8034040 tx=0x7fb7c80035f0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 -- 172.21.15.81:0/3489583270 shutdown_connections 2024-06-29T14:41:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fb7bc076510 0x7fb7bc078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d815b8f0 0x7fb7d806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d80b46a0 0x7fb7d81f4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 --2- 172.21.15.81:0/3489583270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d8073480 0x7fb7d8155f70 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 -- 172.21.15.81:0/3489583270 >> 172.21.15.81:0/3489583270 conn(0x7fb7d80d0b70 msgr2=0x7fb7d80b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 -- 172.21.15.81:0/3489583270 shutdown_connections 2024-06-29T14:41:06.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.755+0000 7fb7dff82640 1 -- 172.21.15.81:0/3489583270 wait complete. 2024-06-29T14:41:06.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:41:06.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.886+0000 7fcec9f43640 1 -- 172.21.15.81:0/288536709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec40cebd0 msgr2=0x7fcec40b7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.886+0000 7fcec9f43640 1 --2- 172.21.15.81:0/288536709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec40cebd0 0x7fcec40b7190 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fceac009830 tx=0x7fceac02a5c0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.886+0000 7fcec9f43640 1 -- 172.21.15.81:0/288536709 shutdown_connections 2024-06-29T14:41:06.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.886+0000 7fcec9f43640 1 --2- 172.21.15.81:0/288536709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec40cebd0 0x7fcec40b7190 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.886+0000 7fcec9f43640 1 -- 172.21.15.81:0/288536709 >> 172.21.15.81:0/288536709 conn(0x7fcec40b3680 msgr2=0x7fcec40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:06.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.886+0000 7fcec9f43640 1 -- 172.21.15.81:0/288536709 shutdown_connections 2024-06-29T14:41:06.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.886+0000 7fcec9f43640 1 -- 172.21.15.81:0/288536709 wait complete. 2024-06-29T14:41:06.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.888+0000 7fcec9f43640 1 Processor -- start 2024-06-29T14:41:06.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.888+0000 7fcec9f43640 1 -- start start 2024-06-29T14:41:06.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcec9f43640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcec40b6610 0x7fcec40cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcec9f43640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec40cfb50 0x7fcec41588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcec9f43640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcec4147c30 0x7fcec4158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcec9f43640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40cd070 con 0x7fcec40cfb50 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcec9f43640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40ccef0 con 0x7fcec40b6610 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcec9f43640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcec40cd1f0 con 0x7fcec4147c30 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec40cfb50 0x7fcec41588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec40cfb50 0x7fcec41588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35046/0 (socket says 172.21.15.81:35046) 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcebbfff640 1 -- 172.21.15.81:0/3034365027 learned_addr learned my addr 172.21.15.81:0/3034365027 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcec37fe640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcec40b6610 0x7fcec40cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcec3fff640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcec4147c30 0x7fcec4158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcebbfff640 1 -- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcec4147c30 msgr2=0x7fcec4158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcebbfff640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcec4147c30 0x7fcec4158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcebbfff640 1 -- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcec40b6610 msgr2=0x7fcec40cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcebbfff640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcec40b6610 0x7fcec40cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:06.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.889+0000 7fcebbfff640 1 -- 172.21.15.81:0/3034365027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec415d120 con 0x7fcec40cfb50 2024-06-29T14:41:06.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.890+0000 7fcebbfff640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec40cfb50 0x7fcec41588a0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fceac02abb0 tx=0x7fceac033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:06.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.890+0000 7fcec0ff9640 1 -- 172.21.15.81:0/3034365027 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac031d60 con 0x7fcec40cfb50 2024-06-29T14:41:06.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.890+0000 7fcec0ff9640 1 -- 172.21.15.81:0/3034365027 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fceac005020 con 0x7fcec40cfb50 2024-06-29T14:41:06.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.890+0000 7fcec9f43640 1 -- 172.21.15.81:0/3034365027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec415a180 con 0x7fcec40cfb50 2024-06-29T14:41:06.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.890+0000 7fcec0ff9640 1 -- 172.21.15.81:0/3034365027 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac005470 con 0x7fcec40cfb50 2024-06-29T14:41:06.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.890+0000 7fcec9f43640 1 -- 172.21.15.81:0/3034365027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec4155f70 con 0x7fcec40cfb50 2024-06-29T14:41:06.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.891+0000 7fcec9f43640 1 -- 172.21.15.81:0/3034365027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec40cd710 con 0x7fcec40cfb50 2024-06-29T14:41:06.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.892+0000 7fcec0ff9640 1 -- 172.21.15.81:0/3034365027 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7fceac0047a0 con 0x7fcec40cfb50 2024-06-29T14:41:06.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.892+0000 7fcec0ff9640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fcea4076540 0x7fcea4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:06.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.892+0000 7fcec37fe640 1 -- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fcea4076540 msgr2=0x7fcea4078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:06.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.892+0000 7fcec0ff9640 1 -- 172.21.15.81:0/3034365027 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7fceac0b8f30 con 0x7fcec40cfb50 2024-06-29T14:41:06.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.892+0000 7fcec37fe640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fcea4076540 0x7fcea4078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:06.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:06.901+0000 7fcec0ff9640 1 -- 172.21.15.81:0/3034365027 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fceac03f030 con 0x7fcec40cfb50 2024-06-29T14:41:07.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.085+0000 7fcec9f43640 1 -- 172.21.15.81:0/3034365027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcec40716a0 con 0x7fcec40cfb50 2024-06-29T14:41:07.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.086+0000 7fcec0ff9640 1 -- 172.21.15.81:0/3034365027 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v120) ==== 104+0+11 (secure 0 0 0) 0x7fceac081c80 con 0x7fcec40cfb50 2024-06-29T14:41:07.083 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:41:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 -- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fcea4076540 msgr2=0x7fcea4078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fcea4076540 0x7fcea4078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 -- 172.21.15.81:0/3034365027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec40cfb50 msgr2=0x7fcec41588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec40cfb50 0x7fcec41588a0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fceac02abb0 tx=0x7fceac033ce0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 -- 172.21.15.81:0/3034365027 shutdown_connections 2024-06-29T14:41:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fcea4076540 0x7fcea4078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcec4147c30 0x7fcec4158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec40cfb50 0x7fcec41588a0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 --2- 172.21.15.81:0/3034365027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcec40b6610 0x7fcec40cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 -- 172.21.15.81:0/3034365027 >> 172.21.15.81:0/3034365027 conn(0x7fcec40b3680 msgr2=0x7fcec40c83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 -- 172.21.15.81:0/3034365027 shutdown_connections 2024-06-29T14:41:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.088+0000 7fcec9f43640 1 -- 172.21.15.81:0/3034365027 wait complete. 2024-06-29T14:41:07.097 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:41:07.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.218+0000 7fa1976b3640 1 -- 172.21.15.81:0/3782742586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1900b4f40 msgr2=0x7fa1900b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.218+0000 7fa1976b3640 1 --2- 172.21.15.81:0/3782742586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1900b4f40 0x7fa1900b5350 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fa190074b10 tx=0x7fa18802aa30 comp rx=0 tx=0).stop 2024-06-29T14:41:07.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.218+0000 7fa1976b3640 1 -- 172.21.15.81:0/3782742586 shutdown_connections 2024-06-29T14:41:07.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.218+0000 7fa1976b3640 1 --2- 172.21.15.81:0/3782742586 >> v2:172.21.15.167:3300/0 conn(0x7fa1900d0230 0x7fa1900d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.218+0000 7fa1976b3640 1 --2- 172.21.15.81:0/3782742586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1900b4f40 0x7fa1900b5350 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.218+0000 7fa1976b3640 1 -- 172.21.15.81:0/3782742586 >> 172.21.15.81:0/3782742586 conn(0x7fa1900b3680 msgr2=0x7fa1900b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.218+0000 7fa1976b3640 1 -- 172.21.15.81:0/3782742586 shutdown_connections 2024-06-29T14:41:07.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.218+0000 7fa1976b3640 1 -- 172.21.15.81:0/3782742586 wait complete. 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.220+0000 7fa1976b3640 1 Processor -- start 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.220+0000 7fa1976b3640 1 -- start start 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa1976b3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1900b3200 0x7fa19009d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa1976b3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1900b4f40 0x7fa1901f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa1976b3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1900d0230 0x7fa190148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa1976b3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa1900b5c20 con 0x7fa1900d0230 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa1976b3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa1900b5aa0 con 0x7fa1900b4f40 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa1976b3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa1900b5da0 con 0x7fa1900b3200 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa195c50640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1900d0230 0x7fa190148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa195c50640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1900d0230 0x7fa190148770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35062/0 (socket says 172.21.15.81:35062) 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa195c50640 1 -- 172.21.15.81:0/351638943 learned_addr learned my addr 172.21.15.81:0/351638943 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa194c4e640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1900b4f40 0x7fa1901f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa19544f640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1900b3200 0x7fa19009d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa195c50640 1 -- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1900b3200 msgr2=0x7fa19009d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa195c50640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1900b3200 0x7fa19009d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa195c50640 1 -- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1900b4f40 msgr2=0x7fa1901f7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa195c50640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1900b4f40 0x7fa1901f7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.221+0000 7fa195c50640 1 -- 172.21.15.81:0/351638943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa190156590 con 0x7fa1900d0230 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.222+0000 7fa195c50640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1900d0230 0x7fa190148770 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fa188005fd0 tx=0x7fa188003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.222+0000 7fa185ffb640 1 -- 172.21.15.81:0/351638943 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188033050 con 0x7fa1900d0230 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.222+0000 7fa185ffb640 1 -- 172.21.15.81:0/351638943 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa188004540 con 0x7fa1900d0230 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.222+0000 7fa185ffb640 1 -- 172.21.15.81:0/351638943 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188038450 con 0x7fa1900d0230 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.222+0000 7fa1976b3640 1 -- 172.21.15.81:0/351638943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1901476d0 con 0x7fa1900d0230 2024-06-29T14:41:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.222+0000 7fa1976b3640 1 -- 172.21.15.81:0/351638943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa190147c90 con 0x7fa1900d0230 2024-06-29T14:41:07.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.223+0000 7fa1976b3640 1 -- 172.21.15.81:0/351638943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1900b5350 con 0x7fa1900d0230 2024-06-29T14:41:07.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.224+0000 7fa185ffb640 1 -- 172.21.15.81:0/351638943 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 64) ==== 98417+0+0 (secure 0 0 0) 0x7fa188036070 con 0x7fa1900d0230 2024-06-29T14:41:07.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.224+0000 7fa185ffb640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fa174076510 0x7fa174078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.225+0000 7fa185ffb640 1 -- 172.21.15.81:0/351638943 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6125+0+0 (secure 0 0 0) 0x7fa1880b9380 con 0x7fa1900d0230 2024-06-29T14:41:07.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.225+0000 7fa19544f640 1 -- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fa174076510 msgr2=0x7fa174078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1146990244 2024-06-29T14:41:07.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.225+0000 7fa19544f640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fa174076510 0x7fa174078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.228+0000 7fa185ffb640 1 -- 172.21.15.81:0/351638943 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa188082080 con 0x7fa1900d0230 2024-06-29T14:41:07.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.261+0000 7fa185ffb640 1 -- 172.21.15.81:0/351638943 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 65) ==== 97912+0+0 (secure 0 0 0) 0x7fa18802d070 con 0x7fa1900d0230 2024-06-29T14:41:07.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.261+0000 7fa185ffb640 1 -- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fa174076510 msgr2=0x7fa174078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:07.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.261+0000 7fa185ffb640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fa174076510 0x7fa174078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.310 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:07.313+0000 7fb2cfa1c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.311 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:07.313+0000 7fb2cfa1c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.311 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:07.313+0000 7fb2cfa1c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.311 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:07.313+0000 7fb2cfa1c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.311 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:07.313+0000 7fb2cfa1c640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.311 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:07.313+0000 7fb2d2a22640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.311 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:07.313+0000 7fb2d2a22640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.311 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:07.313+0000 7fb2d2a22640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.311 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:07.314+0000 7fb2d2a22640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.312 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:07.314+0000 7fb2d2a22640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.426+0000 7fa1976b3640 1 -- 172.21.15.81:0/351638943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa1900cb960 con 0x7fa1900d0230 2024-06-29T14:41:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.427+0000 7fa185ffb640 1 -- 172.21.15.81:0/351638943 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v121) ==== 100+0+11 (secure 0 0 0) 0x7fa18809ea60 con 0x7fa1900d0230 2024-06-29T14:41:07.424 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:41:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.429+0000 7fa1976b3640 1 -- 172.21.15.81:0/351638943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1900d0230 msgr2=0x7fa190148770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.429+0000 7fa1976b3640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1900d0230 0x7fa190148770 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fa188005fd0 tx=0x7fa188003f90 comp rx=0 tx=0).stop 2024-06-29T14:41:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.429+0000 7fa1976b3640 1 -- 172.21.15.81:0/351638943 shutdown_connections 2024-06-29T14:41:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.429+0000 7fa1976b3640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:6832/1146990244,v1:172.21.15.167:6833/1146990244] conn(0x7fa174076510 0x7fa174078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.429+0000 7fa1976b3640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1900d0230 0x7fa190148770 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.429+0000 7fa1976b3640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1900b4f40 0x7fa1901f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.429+0000 7fa1976b3640 1 --2- 172.21.15.81:0/351638943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1900b3200 0x7fa19009d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.429+0000 7fa1976b3640 1 -- 172.21.15.81:0/351638943 >> 172.21.15.81:0/351638943 conn(0x7fa1900b3680 msgr2=0x7fa1900d1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.429+0000 7fa1976b3640 1 -- 172.21.15.81:0/351638943 shutdown_connections 2024-06-29T14:41:07.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.429+0000 7fa1976b3640 1 -- 172.21.15.81:0/351638943 wait complete. 2024-06-29T14:41:07.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:07.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.553+0000 7fbb7492e640 1 -- 172.21.15.81:0/781628990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb700d2a90 msgr2=0x7fbb700d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.553+0000 7fbb7492e640 1 --2- 172.21.15.81:0/781628990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb700d2a90 0x7fbb700d09c0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fbb7008e230 tx=0x7fbb6802ab40 comp rx=0 tx=0).stop 2024-06-29T14:41:07.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.553+0000 7fbb7492e640 1 -- 172.21.15.81:0/781628990 shutdown_connections 2024-06-29T14:41:07.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.553+0000 7fbb7492e640 1 --2- 172.21.15.81:0/781628990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb700d2a90 0x7fbb700d09c0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.553+0000 7fbb7492e640 1 --2- 172.21.15.81:0/781628990 >> v2:172.21.15.167:3300/0 conn(0x7fbb700b9040 0x7fbb700d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.553+0000 7fbb7492e640 1 -- 172.21.15.81:0/781628990 >> 172.21.15.81:0/781628990 conn(0x7fbb7009cf30 msgr2=0x7fbb700d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.553+0000 7fbb7492e640 1 -- 172.21.15.81:0/781628990 shutdown_connections 2024-06-29T14:41:07.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.553+0000 7fbb7492e640 1 -- 172.21.15.81:0/781628990 wait complete. 2024-06-29T14:41:07.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.555+0000 7fbb7492e640 1 Processor -- start 2024-06-29T14:41:07.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.555+0000 7fbb7492e640 1 -- start start 2024-06-29T14:41:07.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.556+0000 7fbb7492e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb700b5250 0x7fbb701f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.556+0000 7fbb7492e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb700b9040 0x7fbb700a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.556+0000 7fbb7492e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb700d2a90 0x7fbb700a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.556+0000 7fbb7492e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb700b7d30 con 0x7fbb700b9040 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.556+0000 7fbb7492e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbb700b7bb0 con 0x7fbb700b5250 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.556+0000 7fbb7492e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbb700b7eb0 con 0x7fbb700d2a90 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb67fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb700b9040 0x7fbb700a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb67fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb700b9040 0x7fbb700a0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35074/0 (socket says 172.21.15.81:35074) 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb67fff640 1 -- 172.21.15.81:0/1327208390 learned_addr learned my addr 172.21.15.81:0/1327208390 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb6e59c640 1 --2- 172.21.15.81:0/1327208390 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb700b5250 0x7fbb701f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb6ed9d640 1 --2- 172.21.15.81:0/1327208390 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb700d2a90 0x7fbb700a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb67fff640 1 -- 172.21.15.81:0/1327208390 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb700d2a90 msgr2=0x7fbb700a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb67fff640 1 --2- 172.21.15.81:0/1327208390 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb700d2a90 0x7fbb700a12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb67fff640 1 -- 172.21.15.81:0/1327208390 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb700b5250 msgr2=0x7fbb701f5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb67fff640 1 --2- 172.21.15.81:0/1327208390 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb700b5250 0x7fbb701f5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb67fff640 1 -- 172.21.15.81:0/1327208390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb700b1ac0 con 0x7fbb700b9040 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb6ed9d640 1 --2- 172.21.15.81:0/1327208390 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb700d2a90 0x7fbb700a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb67fff640 1 --2- 172.21.15.81:0/1327208390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb700b9040 0x7fbb700a0da0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800c3a0 tx=0x7fbb5800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb66ffd640 1 -- 172.21.15.81:0/1327208390 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb58017070 con 0x7fbb700b9040 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb66ffd640 1 -- 172.21.15.81:0/1327208390 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb580053c0 con 0x7fbb700b9040 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb7492e640 1 -- 172.21.15.81:0/1327208390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb700b1da0 con 0x7fbb700b9040 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb66ffd640 1 -- 172.21.15.81:0/1327208390 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb58003850 con 0x7fbb700b9040 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.557+0000 7fbb7492e640 1 -- 172.21.15.81:0/1327208390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb700b2360 con 0x7fbb700b9040 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.558+0000 7fbb7492e640 1 -- 172.21.15.81:0/1327208390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb700b8440 con 0x7fbb700b9040 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.559+0000 7fbb66ffd640 1 -- 172.21.15.81:0/1327208390 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 65) ==== 97912+0+0 (secure 0 0 0) 0x7fbb5801a070 con 0x7fbb700b9040 2024-06-29T14:41:07.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.559+0000 7fbb66ffd640 1 -- 172.21.15.81:0/1327208390 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6555+0+0 (secure 0 0 0) 0x7fbb58096850 con 0x7fbb700b9040 2024-06-29T14:41:07.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.564+0000 7fbb66ffd640 1 -- 172.21.15.81:0/1327208390 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb5805fc60 con 0x7fbb700b9040 2024-06-29T14:41:07.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.780+0000 7fbb7492e640 1 -- 172.21.15.81:0/1327208390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb7015a880 con 0x7fbb700b9040 2024-06-29T14:41:07.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.780+0000 7fbb66ffd640 1 -- 172.21.15.81:0/1327208390 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5231 (secure 0 0 0) 0x7fbb580682a0 con 0x7fbb700b9040 2024-06-29T14:41:07.778 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:07.778 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":59,"btime":"2024-06-29T14:40:00:847465+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1563799167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1563799167},{"type":"v1","addr":"172.21.15.167:6837","nonce":1563799167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7354,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3062579579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3062579579},{"type":"v1","addr":"172.21.15.81:6835","nonce":3062579579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7363,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3427077146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3427077146},{"type":"v1","addr":"172.21.15.167:6836","nonce":3427077146}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.183979+0000","modified":"2024-06-29T14:40:00.217351+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/1049868964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1049868964},{"type":"v1","addr":"172.21.15.81:6833","nonce":1049868964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8022,"qdb_cluster":[8022]},"id":7}]} 2024-06-29T14:41:07.778 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 59 2024-06-29T14:41:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.782+0000 7fbb7492e640 1 -- 172.21.15.81:0/1327208390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb700b9040 msgr2=0x7fbb700a0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.782+0000 7fbb7492e640 1 --2- 172.21.15.81:0/1327208390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb700b9040 0x7fbb700a0da0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800c3a0 tx=0x7fbb5800c860 comp rx=0 tx=0).stop 2024-06-29T14:41:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.782+0000 7fbb7492e640 1 -- 172.21.15.81:0/1327208390 shutdown_connections 2024-06-29T14:41:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.782+0000 7fbb7492e640 1 --2- 172.21.15.81:0/1327208390 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb700d2a90 0x7fbb700a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.782+0000 7fbb7492e640 1 --2- 172.21.15.81:0/1327208390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb700b9040 0x7fbb700a0da0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.782+0000 7fbb7492e640 1 --2- 172.21.15.81:0/1327208390 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb700b5250 0x7fbb701f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.782+0000 7fbb7492e640 1 -- 172.21.15.81:0/1327208390 >> 172.21.15.81:0/1327208390 conn(0x7fbb7009cf30 msgr2=0x7fbb700685d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.782+0000 7fbb7492e640 1 -- 172.21.15.81:0/1327208390 shutdown_connections 2024-06-29T14:41:07.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.782+0000 7fbb7492e640 1 -- 172.21.15.81:0/1327208390 wait complete. 2024-06-29T14:41:07.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:07.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.906+0000 7f817d08f640 1 -- 172.21.15.81:0/3066599563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81780b4430 msgr2=0x7f81780cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.906+0000 7f817d08f640 1 --2- 172.21.15.81:0/3066599563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81780b4430 0x7f81780cf130 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f8178070220 tx=0x7f816002a560 comp rx=0 tx=0).stop 2024-06-29T14:41:07.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.906+0000 7f817d08f640 1 -- 172.21.15.81:0/3066599563 shutdown_connections 2024-06-29T14:41:07.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.906+0000 7f817d08f640 1 --2- 172.21.15.81:0/3066599563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81780b4430 0x7f81780cf130 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.906+0000 7f817d08f640 1 --2- 172.21.15.81:0/3066599563 >> v2:172.21.15.167:3300/0 conn(0x7f81780b35b0 0x7f817815b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.906+0000 7f817d08f640 1 -- 172.21.15.81:0/3066599563 >> 172.21.15.81:0/3066599563 conn(0x7f817806ccc0 msgr2=0x7f81780b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:07.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.906+0000 7f817d08f640 1 -- 172.21.15.81:0/3066599563 shutdown_connections 2024-06-29T14:41:07.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.906+0000 7f817d08f640 1 -- 172.21.15.81:0/3066599563 wait complete. 2024-06-29T14:41:07.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.908+0000 7f817d08f640 1 Processor -- start 2024-06-29T14:41:07.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.908+0000 7f817d08f640 1 -- start start 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f817d08f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f817806f660 0x7f817813e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f817d08f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81780b35b0 0x7f81781f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f817d08f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81780b4430 0x7f81781f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f817d08f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f817806f160 con 0x7f81780b35b0 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f817d08f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f817806efe0 con 0x7f817806f660 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f817d08f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f817806f2e0 con 0x7f81780b4430 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f816e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81780b35b0 0x7f81781f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f816e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81780b35b0 0x7f81781f3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35092/0 (socket says 172.21.15.81:35092) 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f816e59c640 1 -- 172.21.15.81:0/1220706537 learned_addr learned my addr 172.21.15.81:0/1220706537 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f8176d9d640 1 --2- 172.21.15.81:0/1220706537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f817806f660 0x7f817813e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f817759e640 1 --2- 172.21.15.81:0/1220706537 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81780b4430 0x7f81781f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.909+0000 7f816e59c640 1 -- 172.21.15.81:0/1220706537 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81780b4430 msgr2=0x7f81781f44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.910+0000 7f816e59c640 1 --2- 172.21.15.81:0/1220706537 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81780b4430 0x7f81781f44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.910+0000 7f816e59c640 1 -- 172.21.15.81:0/1220706537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f817806f660 msgr2=0x7f817813e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:07.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.910+0000 7f816e59c640 1 --2- 172.21.15.81:0/1220706537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f817806f660 0x7f817813e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:07.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.910+0000 7f816e59c640 1 -- 172.21.15.81:0/1220706537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81781e5980 con 0x7f81780b35b0 2024-06-29T14:41:07.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.910+0000 7f816e59c640 1 --2- 172.21.15.81:0/1220706537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81780b35b0 0x7f81781f3fb0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f8160034040 tx=0x7f8160003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:07.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.910+0000 7f816ffff640 1 -- 172.21.15.81:0/1220706537 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160035040 con 0x7f81780b35b0 2024-06-29T14:41:07.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.910+0000 7f816ffff640 1 -- 172.21.15.81:0/1220706537 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81600047a0 con 0x7f81780b35b0 2024-06-29T14:41:07.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.910+0000 7f817d08f640 1 -- 172.21.15.81:0/1220706537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817814bc90 con 0x7f81780b35b0 2024-06-29T14:41:07.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.910+0000 7f816ffff640 1 -- 172.21.15.81:0/1220706537 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160004b70 con 0x7f81780b35b0 2024-06-29T14:41:07.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.910+0000 7f817d08f640 1 -- 172.21.15.81:0/1220706537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817814c210 con 0x7f81780b35b0 2024-06-29T14:41:07.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.911+0000 7f817d08f640 1 -- 172.21.15.81:0/1220706537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f817806fec0 con 0x7f81780b35b0 2024-06-29T14:41:07.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.912+0000 7f816ffff640 1 -- 172.21.15.81:0/1220706537 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 65) ==== 97912+0+0 (secure 0 0 0) 0x7f8160004d10 con 0x7f81780b35b0 2024-06-29T14:41:07.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.912+0000 7f816ffff640 1 -- 172.21.15.81:0/1220706537 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6555+0+0 (secure 0 0 0) 0x7f81600b9d10 con 0x7f81780b35b0 2024-06-29T14:41:07.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:07.917+0000 7f816ffff640 1 -- 172.21.15.81:0/1220706537 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f817806fec0 con 0x7f81780b35b0 2024-06-29T14:41:08.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.099+0000 7f817d08f640 1 -- 172.21.15.81:0/1220706537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f817815b5b0 con 0x7f81780b35b0 2024-06-29T14:41:08.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.100+0000 7f816ffff640 1 -- 172.21.15.81:0/1220706537 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) ==== 74+0+17859 (secure 0 0 0) 0x7f8160083120 con 0x7f81780b35b0 2024-06-29T14:41:08.098 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:08.098 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":121,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:07.251311+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:55.492112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.145353+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/1146990244":"2024-06-30T14:41:07.251243+0000","172.21.15.167:6832/1146990244":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/3014421868":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/3886955317":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/2019716400":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/421920145":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/977399066":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/586922734":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/2613865897":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/2148809735":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/1014005913":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/705657234":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/4108453499":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/2888901478":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/1794407759":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/839096394":"2024-06-30T14:40:09.486395+0000","172.21.15.167:6833/473361950":"2024-06-30T14:40:09.486395+0000","172.21.15.167:6832/473361950":"2024-06-30T14:40:09.486395+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:08.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.102+0000 7f817d08f640 1 -- 172.21.15.81:0/1220706537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81780b35b0 msgr2=0x7f81781f3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.102+0000 7f817d08f640 1 --2- 172.21.15.81:0/1220706537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81780b35b0 0x7f81781f3fb0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f8160034040 tx=0x7f8160003910 comp rx=0 tx=0).stop 2024-06-29T14:41:08.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.102+0000 7f817d08f640 1 -- 172.21.15.81:0/1220706537 shutdown_connections 2024-06-29T14:41:08.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.102+0000 7f817d08f640 1 --2- 172.21.15.81:0/1220706537 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81780b4430 0x7f81781f44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.102+0000 7f817d08f640 1 --2- 172.21.15.81:0/1220706537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81780b35b0 0x7f81781f3fb0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.102+0000 7f817d08f640 1 --2- 172.21.15.81:0/1220706537 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f817806f660 0x7f817813e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.102+0000 7f817d08f640 1 -- 172.21.15.81:0/1220706537 >> 172.21.15.81:0/1220706537 conn(0x7f817806ccc0 msgr2=0x7f817815bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.102+0000 7f817d08f640 1 -- 172.21.15.81:0/1220706537 shutdown_connections 2024-06-29T14:41:08.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.102+0000 7f817d08f640 1 -- 172.21.15.81:0/1220706537 wait complete. 2024-06-29T14:41:08.111 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:41:08.111 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:08.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.227+0000 7fa56401f640 1 -- 172.21.15.81:0/2496692719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa55c0b8fa0 msgr2=0x7fa55c0d0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.227+0000 7fa56401f640 1 --2- 172.21.15.81:0/2496692719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa55c0b8fa0 0x7fa55c0d0460 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0d27f0 tx=0x7fa55802a540 comp rx=0 tx=0).stop 2024-06-29T14:41:08.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.227+0000 7fa56401f640 1 -- 172.21.15.81:0/2496692719 shutdown_connections 2024-06-29T14:41:08.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.227+0000 7fa56401f640 1 --2- 172.21.15.81:0/2496692719 >> v2:172.21.15.167:3300/0 conn(0x7fa55c0d0db0 0x7fa55c0b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.227+0000 7fa56401f640 1 --2- 172.21.15.81:0/2496692719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa55c0d09a0 0x7fa55c0b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.227+0000 7fa56401f640 1 --2- 172.21.15.81:0/2496692719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa55c0b8fa0 0x7fa55c0d0460 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.227+0000 7fa56401f640 1 -- 172.21.15.81:0/2496692719 >> 172.21.15.81:0/2496692719 conn(0x7fa55c0d3c50 msgr2=0x7fa55c0d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.227+0000 7fa56401f640 1 -- 172.21.15.81:0/2496692719 shutdown_connections 2024-06-29T14:41:08.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.227+0000 7fa56401f640 1 -- 172.21.15.81:0/2496692719 wait complete. 2024-06-29T14:41:08.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.229+0000 7fa56401f640 1 Processor -- start 2024-06-29T14:41:08.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.229+0000 7fa56401f640 1 -- start start 2024-06-29T14:41:08.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.230+0000 7fa56401f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa55c0b8fa0 0x7fa55c0b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.230+0000 7fa56401f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa55c0d09a0 0x7fa55c0cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.230+0000 7fa56401f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa55c0d0db0 0x7fa55c0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.230+0000 7fa56401f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c0b41a0 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.230+0000 7fa56401f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c0b4020 con 0x7fa55c0d0db0 2024-06-29T14:41:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.230+0000 7fa56401f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa55c0ce7f0 con 0x7fa55c0d09a0 2024-06-29T14:41:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.230+0000 7fa561dbb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa55c0b8fa0 0x7fa55c0b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.230+0000 7fa561dbb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa55c0b8fa0 0x7fa55c0b7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35108/0 (socket says 172.21.15.81:35108) 2024-06-29T14:41:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.230+0000 7fa561dbb640 1 -- 172.21.15.81:0/2828549456 learned_addr learned my addr 172.21.15.81:0/2828549456 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa5625bc640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa55c0d0db0 0x7fa55c0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa5615ba640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa55c0d09a0 0x7fa55c0cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa561dbb640 1 -- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa55c0d09a0 msgr2=0x7fa55c0cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa561dbb640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa55c0d09a0 0x7fa55c0cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa561dbb640 1 -- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa55c0d0db0 msgr2=0x7fa55c0cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa561dbb640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa55c0d0db0 0x7fa55c0cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa561dbb640 1 -- 172.21.15.81:0/2828549456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55c0b7a80 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa561dbb640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa55c0b8fa0 0x7fa55c0b7540 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fa558034040 tx=0x7fa558003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa5527fc640 1 -- 172.21.15.81:0/2828549456 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55803e030 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa5527fc640 1 -- 172.21.15.81:0/2828549456 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa558003e00 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa56401f640 1 -- 172.21.15.81:0/2828549456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa55c0b6a80 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa5527fc640 1 -- 172.21.15.81:0/2828549456 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558035040 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.231+0000 7fa56401f640 1 -- 172.21.15.81:0/2828549456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa55c1e9220 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.232+0000 7fa56401f640 1 -- 172.21.15.81:0/2828549456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa55c0d0460 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.233+0000 7fa5527fc640 1 -- 172.21.15.81:0/2828549456 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 65) ==== 97912+0+0 (secure 0 0 0) 0x7fa558041030 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.234+0000 7fa5527fc640 1 -- 172.21.15.81:0/2828549456 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6555+0+0 (secure 0 0 0) 0x7fa5580b7d20 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.238+0000 7fa5527fc640 1 -- 172.21.15.81:0/2828549456 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa55c0b7a80 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.266+0000 7fa5527fc640 1 -- 172.21.15.81:0/2828549456 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 66) ==== 98039+0+0 (secure 0 0 0) 0x7fa55807ca10 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.266+0000 7fa5527fc640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fa54007bb80 0x7fa54007dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.266+0000 7fa5615ba640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fa54007bb80 0x7fa54007dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.267+0000 7fa5615ba640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fa54007bb80 0x7fa54007dfa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0b29b0 tx=0x7fa54c00b260 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.461+0000 7fa56401f640 1 -- 172.21.15.81:0/2828549456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa55c0b7c60 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.461+0000 7fa5527fc640 1 -- 172.21.15.81:0/2828549456 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5231 (secure 0 0 0) 0x7fa5580b1c00 con 0x7fa55c0b8fa0 2024-06-29T14:41:08.459 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:08.460 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":59,"btime":"2024-06-29T14:40:00:847465+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1563799167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1563799167},{"type":"v1","addr":"172.21.15.167:6837","nonce":1563799167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7354,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3062579579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3062579579},{"type":"v1","addr":"172.21.15.81:6835","nonce":3062579579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7363,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3427077146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3427077146},{"type":"v1","addr":"172.21.15.167:6836","nonce":3427077146}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.183979+0000","modified":"2024-06-29T14:40:00.217351+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/1049868964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1049868964},{"type":"v1","addr":"172.21.15.81:6833","nonce":1049868964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8022,"qdb_cluster":[8022]},"id":7}]} 2024-06-29T14:41:08.460 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 59 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 -- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fa54007bb80 msgr2=0x7fa54007dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fa54007bb80 0x7fa54007dfa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0b29b0 tx=0x7fa54c00b260 comp rx=0 tx=0).stop 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 -- 172.21.15.81:0/2828549456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa55c0b8fa0 msgr2=0x7fa55c0b7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa55c0b8fa0 0x7fa55c0b7540 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fa558034040 tx=0x7fa558003910 comp rx=0 tx=0).stop 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 -- 172.21.15.81:0/2828549456 shutdown_connections 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fa54007bb80 0x7fa54007dfa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa55c0d0db0 0x7fa55c0cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa55c0d09a0 0x7fa55c0cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 --2- 172.21.15.81:0/2828549456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa55c0b8fa0 0x7fa55c0b7540 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 -- 172.21.15.81:0/2828549456 >> 172.21.15.81:0/2828549456 conn(0x7fa55c0d3c50 msgr2=0x7fa55c1e6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 -- 172.21.15.81:0/2828549456 shutdown_connections 2024-06-29T14:41:08.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.463+0000 7fa56401f640 1 -- 172.21.15.81:0/2828549456 wait complete. 2024-06-29T14:41:08.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:08.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.592+0000 7f6728f55640 1 -- 172.21.15.81:0/894031491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67240b3cd0 msgr2=0x7f67240d2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.592+0000 7f6728f55640 1 --2- 172.21.15.81:0/894031491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67240b3cd0 0x7f67240d2240 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f6718005fd0 tx=0x7f671802a540 comp rx=0 tx=0).stop 2024-06-29T14:41:08.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.592+0000 7f6728f55640 1 -- 172.21.15.81:0/894031491 shutdown_connections 2024-06-29T14:41:08.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.592+0000 7f6728f55640 1 --2- 172.21.15.81:0/894031491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67240ceb80 0x7f67240d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.592+0000 7f6728f55640 1 --2- 172.21.15.81:0/894031491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67240b3cd0 0x7f67240d2240 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.592+0000 7f6728f55640 1 -- 172.21.15.81:0/894031491 >> 172.21.15.81:0/894031491 conn(0x7f67240d3c60 msgr2=0x7f67240d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.592+0000 7f6728f55640 1 -- 172.21.15.81:0/894031491 shutdown_connections 2024-06-29T14:41:08.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.592+0000 7f6728f55640 1 -- 172.21.15.81:0/894031491 wait complete. 2024-06-29T14:41:08.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.594+0000 7f6728f55640 1 Processor -- start 2024-06-29T14:41:08.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.594+0000 7f6728f55640 1 -- start start 2024-06-29T14:41:08.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6728f55640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67240ce770 0x7f67241521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6728f55640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67240ceb80 0x7f67240cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6728f55640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6724165a60 0x7f67241671f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6728f55640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f67240d15a0 con 0x7f6724165a60 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6728f55640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f67240d1420 con 0x7f67240ce770 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6728f55640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f67240d4990 con 0x7f67240ceb80 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6722d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6724165a60 0x7f67241671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6722d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6724165a60 0x7f67241671f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35126/0 (socket says 172.21.15.81:35126) 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6722d9d640 1 -- 172.21.15.81:0/19570137 learned_addr learned my addr 172.21.15.81:0/19570137 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6721d9b640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67240ceb80 0x7f67240cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f672259c640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67240ce770 0x7f67241521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6722d9d640 1 -- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67240ceb80 msgr2=0x7f67240cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6722d9d640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67240ceb80 0x7f67240cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6722d9d640 1 -- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67240ce770 msgr2=0x7f67241521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6722d9d640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67240ce770 0x7f67241521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6722d9d640 1 -- 172.21.15.81:0/19570137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6724152720 con 0x7f6724165a60 2024-06-29T14:41:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6722d9d640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6724165a60 0x7f67241671f0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f671c00eab0 tx=0x7f671c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6712ffd640 1 -- 172.21.15.81:0/19570137 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c00cea0 con 0x7f6724165a60 2024-06-29T14:41:08.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6712ffd640 1 -- 172.21.15.81:0/19570137 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f671c004b20 con 0x7f6724165a60 2024-06-29T14:41:08.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.596+0000 7f6728f55640 1 -- 172.21.15.81:0/19570137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67240cdfa0 con 0x7f6724165a60 2024-06-29T14:41:08.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.597+0000 7f6712ffd640 1 -- 172.21.15.81:0/19570137 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c005020 con 0x7f6724165a60 2024-06-29T14:41:08.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.597+0000 7f6728f55640 1 -- 172.21.15.81:0/19570137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6724164ca0 con 0x7f6724165a60 2024-06-29T14:41:08.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.598+0000 7f6728f55640 1 -- 172.21.15.81:0/19570137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67240d2240 con 0x7f6724165a60 2024-06-29T14:41:08.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.598+0000 7f6712ffd640 1 -- 172.21.15.81:0/19570137 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 66) ==== 98039+0+0 (secure 0 0 0) 0x7f671c0051c0 con 0x7f6724165a60 2024-06-29T14:41:08.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.599+0000 7f6712ffd640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6704076120 0x7f6704078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.599+0000 7f6712ffd640 1 -- 172.21.15.81:0/19570137 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6611+0+0 (secure 0 0 0) 0x7f671c0982a0 con 0x7f6724165a60 2024-06-29T14:41:08.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.599+0000 7f672259c640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6704076120 0x7f6704078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.599+0000 7f672259c640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6704076120 0x7f6704078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f67240d0120 tx=0x7f6714009040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.603+0000 7f6712ffd640 1 -- 172.21.15.81:0/19570137 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f671c0614e0 con 0x7f6724165a60 2024-06-29T14:41:08.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.822+0000 7f6728f55640 1 -- 172.21.15.81:0/19570137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6724168710 con 0x7f6724165a60 2024-06-29T14:41:08.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.823+0000 7f6712ffd640 1 -- 172.21.15.81:0/19570137 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5231 (secure 0 0 0) 0x7f671c0657e0 con 0x7f6724165a60 2024-06-29T14:41:08.820 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:08.821 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":59,"btime":"2024-06-29T14:40:00:847465+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1563799167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1563799167},{"type":"v1","addr":"172.21.15.167:6837","nonce":1563799167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7354,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3062579579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3062579579},{"type":"v1","addr":"172.21.15.81:6835","nonce":3062579579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7363,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3427077146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3427077146},{"type":"v1","addr":"172.21.15.167:6836","nonce":3427077146}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:39:57.183979+0000","modified":"2024-06-29T14:40:00.217351+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"a","rank":0,"incarnation":56,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/1049868964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1049868964},{"type":"v1","addr":"172.21.15.81:6833","nonce":1049868964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8022,"qdb_cluster":[8022]},"id":7}]} 2024-06-29T14:41:08.821 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 59 2024-06-29T14:41:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 -- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6704076120 msgr2=0x7f6704078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6704076120 0x7f6704078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f67240d0120 tx=0x7f6714009040 comp rx=0 tx=0).stop 2024-06-29T14:41:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 -- 172.21.15.81:0/19570137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6724165a60 msgr2=0x7f67241671f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6724165a60 0x7f67241671f0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f671c00eab0 tx=0x7f671c00ef70 comp rx=0 tx=0).stop 2024-06-29T14:41:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 -- 172.21.15.81:0/19570137 shutdown_connections 2024-06-29T14:41:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6704076120 0x7f6704078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6724165a60 0x7f67241671f0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67240ceb80 0x7f67240cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 --2- 172.21.15.81:0/19570137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67240ce770 0x7f67241521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 -- 172.21.15.81:0/19570137 >> 172.21.15.81:0/19570137 conn(0x7f67240d3c60 msgr2=0x7f672406ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 -- 172.21.15.81:0/19570137 shutdown_connections 2024-06-29T14:41:08.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.825+0000 7f6728f55640 1 -- 172.21.15.81:0/19570137 wait complete. 2024-06-29T14:41:08.834 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:08.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.950+0000 7f30f237a640 1 -- 172.21.15.81:0/1317665172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30d40053a0 msgr2=0x7f30d400ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.950+0000 7f30f237a640 1 --2- 172.21.15.81:0/1317665172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30d40053a0 0x7f30d400ba50 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f30e4001dc0 tx=0x7f30d4005830 comp rx=0 tx=0).stop 2024-06-29T14:41:08.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.950+0000 7f30f237a640 1 -- 172.21.15.81:0/1317665172 shutdown_connections 2024-06-29T14:41:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.950+0000 7f30f237a640 1 --2- 172.21.15.81:0/1317665172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30d40053a0 0x7f30d400ba50 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.950+0000 7f30f237a640 1 -- 172.21.15.81:0/1317665172 >> 172.21.15.81:0/1317665172 conn(0x7f30ec0b3680 msgr2=0x7f30ec0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.950+0000 7f30f237a640 1 -- 172.21.15.81:0/1317665172 shutdown_connections 2024-06-29T14:41:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.950+0000 7f30f237a640 1 -- 172.21.15.81:0/1317665172 wait complete. 2024-06-29T14:41:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.951+0000 7f30f237a640 1 Processor -- start 2024-06-29T14:41:08.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.952+0000 7f30f237a640 1 -- start start 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f237a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d40053a0 0x7f30ec156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f237a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30ec159ca0 0x7f30ec09edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f237a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30ec09f2f0 0x7f30ec156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f237a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec0cc930 con 0x7f30ec09f2f0 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f237a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec0cc7b0 con 0x7f30d40053a0 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f237a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f30ec0ccab0 con 0x7f30ec159ca0 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f0917640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30ec09f2f0 0x7f30ec156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f0917640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30ec09f2f0 0x7f30ec156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35142/0 (socket says 172.21.15.81:35142) 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f0917640 1 -- 172.21.15.81:0/2268187838 learned_addr learned my addr 172.21.15.81:0/2268187838 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30ebfff640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d40053a0 0x7f30ec156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30eb7fe640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30ec159ca0 0x7f30ec09edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f0917640 1 -- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30ec159ca0 msgr2=0x7f30ec09edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f0917640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30ec159ca0 0x7f30ec09edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f0917640 1 -- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d40053a0 msgr2=0x7f30ec156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f0917640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d40053a0 0x7f30ec156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30f0917640 1 -- 172.21.15.81:0/2268187838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec156980 con 0x7f30ec09f2f0 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30ebfff640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d40053a0 0x7f30ec156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.953+0000 7f30eb7fe640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30ec159ca0 0x7f30ec09edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.954+0000 7f30f0917640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30ec09f2f0 0x7f30ec156d10 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f30e4001eb0 tx=0x7f30e4006e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.954+0000 7f30e8ff9640 1 -- 172.21.15.81:0/2268187838 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e4003d80 con 0x7f30ec09f2f0 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.954+0000 7f30f237a640 1 -- 172.21.15.81:0/2268187838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ec158ee0 con 0x7f30ec09f2f0 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.954+0000 7f30e8ff9640 1 -- 172.21.15.81:0/2268187838 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30e40060d0 con 0x7f30ec09f2f0 2024-06-29T14:41:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.954+0000 7f30e8ff9640 1 -- 172.21.15.81:0/2268187838 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e40326d0 con 0x7f30ec09f2f0 2024-06-29T14:41:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.954+0000 7f30f237a640 1 -- 172.21.15.81:0/2268187838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ec1594a0 con 0x7f30ec09f2f0 2024-06-29T14:41:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.955+0000 7f30f237a640 1 -- 172.21.15.81:0/2268187838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30ec0ccfc0 con 0x7f30ec09f2f0 2024-06-29T14:41:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.956+0000 7f30e8ff9640 1 -- 172.21.15.81:0/2268187838 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 66) ==== 98039+0+0 (secure 0 0 0) 0x7f30e4032870 con 0x7f30ec09f2f0 2024-06-29T14:41:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.956+0000 7f30e8ff9640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f30d8076150 0x7f30d8078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.956+0000 7f30e8ff9640 1 -- 172.21.15.81:0/2268187838 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6611+0+0 (secure 0 0 0) 0x7f30e40b75f0 con 0x7f30ec09f2f0 2024-06-29T14:41:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.957+0000 7f30ebfff640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f30d8076150 0x7f30d8078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.957+0000 7f30ebfff640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f30d8076150 0x7f30d8078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f30dc001820 tx=0x7f30dc0012b0 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:08.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:08.961+0000 7f30e8ff9640 1 -- 172.21.15.81:0/2268187838 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f30e4045020 con 0x7f30ec09f2f0 2024-06-29T14:41:09.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.143+0000 7f30f237a640 1 -- 172.21.15.81:0/2268187838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f30ec09f700 con 0x7f30ec09f2f0 2024-06-29T14:41:09.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.144+0000 7f30e8ff9640 1 -- 172.21.15.81:0/2268187838 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) ==== 74+0+17971 (secure 0 0 0) 0x7f30ec156980 con 0x7f30ec09f2f0 2024-06-29T14:41:09.142 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:09.142 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":122,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:08.254551+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:39:55.492112+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-06-29T14:39:56.145353+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":122,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/1146990244":"2024-06-30T14:41:07.251243+0000","172.21.15.167:6832/1146990244":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/3014421868":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/3886955317":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/2019716400":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/421920145":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/977399066":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/586922734":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/2613865897":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/2148809735":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/1014005913":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/705657234":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/4108453499":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/2888901478":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/1794407759":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/839096394":"2024-06-30T14:40:09.486395+0000","172.21.15.167:6833/473361950":"2024-06-30T14:40:09.486395+0000","172.21.15.167:6832/473361950":"2024-06-30T14:40:09.486395+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 -- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f30d8076150 msgr2=0x7f30d8078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f30d8076150 0x7f30d8078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f30dc001820 tx=0x7f30dc0012b0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 -- 172.21.15.81:0/2268187838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30ec09f2f0 msgr2=0x7f30ec156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30ec09f2f0 0x7f30ec156d10 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f30e4001eb0 tx=0x7f30e4006e70 comp rx=0 tx=0).stop 2024-06-29T14:41:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 -- 172.21.15.81:0/2268187838 shutdown_connections 2024-06-29T14:41:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f30d8076150 0x7f30d8078660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30ec09f2f0 0x7f30ec156d10 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30ec159ca0 0x7f30ec09edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 --2- 172.21.15.81:0/2268187838 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f30d40053a0 0x7f30ec156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 -- 172.21.15.81:0/2268187838 >> 172.21.15.81:0/2268187838 conn(0x7f30ec0b3680 msgr2=0x7f30ec0b63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:09.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 -- 172.21.15.81:0/2268187838 shutdown_connections 2024-06-29T14:41:09.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.146+0000 7f30f237a640 1 -- 172.21.15.81:0/2268187838 wait complete. 2024-06-29T14:41:09.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:41:09.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.272+0000 7f40eac9e640 1 -- 172.21.15.81:0/2104912215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e406ec50 msgr2=0x7f40e40acfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.272+0000 7f40eac9e640 1 --2- 172.21.15.81:0/2104912215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e406ec50 0x7f40e40acfc0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f40e40cdd20 tx=0x7f40dc02a980 comp rx=0 tx=0).stop 2024-06-29T14:41:09.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.272+0000 7f40eac9e640 1 -- 172.21.15.81:0/2104912215 shutdown_connections 2024-06-29T14:41:09.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.272+0000 7f40eac9e640 1 --2- 172.21.15.81:0/2104912215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e406ec50 0x7f40e40acfc0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.272+0000 7f40eac9e640 1 --2- 172.21.15.81:0/2104912215 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40e40744f0 0x7f40e406e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.272+0000 7f40eac9e640 1 --2- 172.21.15.81:0/2104912215 >> v2:172.21.15.167:3300/0 conn(0x7f40e415afa0 0x7f40e40ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.272+0000 7f40eac9e640 1 -- 172.21.15.81:0/2104912215 >> 172.21.15.81:0/2104912215 conn(0x7f40e40b1ec0 msgr2=0x7f40e40b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:09.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.272+0000 7f40eac9e640 1 -- 172.21.15.81:0/2104912215 shutdown_connections 2024-06-29T14:41:09.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.272+0000 7f40eac9e640 1 -- 172.21.15.81:0/2104912215 wait complete. 2024-06-29T14:41:09.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.274+0000 7f40eac9e640 1 Processor -- start 2024-06-29T14:41:09.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.274+0000 7f40eac9e640 1 -- start start 2024-06-29T14:41:09.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.275+0000 7f40eac9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e406ec50 0x7f40e41455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.275+0000 7f40eac9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40e40744f0 0x7f40e41f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.275+0000 7f40eac9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40e415afa0 0x7f40e41f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.275+0000 7f40eac9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f40e4073790 con 0x7f40e406ec50 2024-06-29T14:41:09.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.275+0000 7f40eac9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f40e4073610 con 0x7f40e40744f0 2024-06-29T14:41:09.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.275+0000 7f40eac9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f40e4073910 con 0x7f40e415afa0 2024-06-29T14:41:09.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.275+0000 7f40e8a3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e406ec50 0x7f40e41455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.275+0000 7f40e8a3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e406ec50 0x7f40e41455c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35158/0 (socket says 172.21.15.81:35158) 2024-06-29T14:41:09.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.275+0000 7f40e8a3a640 1 -- 172.21.15.81:0/1924414665 learned_addr learned my addr 172.21.15.81:0/1924414665 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:09.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e923b640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40e415afa0 0x7f40e41f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e8a3a640 1 -- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40e415afa0 msgr2=0x7f40e41f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e8a3a640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40e415afa0 0x7f40e41f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e3fff640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40e40744f0 0x7f40e41f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e8a3a640 1 -- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40e40744f0 msgr2=0x7f40e41f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e8a3a640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40e40744f0 0x7f40e41f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e8a3a640 1 -- 172.21.15.81:0/1924414665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e41f2d80 con 0x7f40e406ec50 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e3fff640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40e40744f0 0x7f40e41f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e8a3a640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e406ec50 0x7f40e41455c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f40d800d9c0 tx=0x7f40d800de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e17fa640 1 -- 172.21.15.81:0/1924414665 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d801b070 con 0x7f40e406ec50 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e17fa640 1 -- 172.21.15.81:0/1924414665 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40d8016440 con 0x7f40e406ec50 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40eac9e640 1 -- 172.21.15.81:0/1924414665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e415c9a0 con 0x7f40e406ec50 2024-06-29T14:41:09.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40e17fa640 1 -- 172.21.15.81:0/1924414665 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8016890 con 0x7f40e406ec50 2024-06-29T14:41:09.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.276+0000 7f40eac9e640 1 -- 172.21.15.81:0/1924414665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e415cf60 con 0x7f40e406ec50 2024-06-29T14:41:09.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.277+0000 7f40eac9e640 1 -- 172.21.15.81:0/1924414665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40e409e2f0 con 0x7f40e406ec50 2024-06-29T14:41:09.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.278+0000 7f40e17fa640 1 -- 172.21.15.81:0/1924414665 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 66) ==== 98039+0+0 (secure 0 0 0) 0x7f40d801e070 con 0x7f40e406ec50 2024-06-29T14:41:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.278+0000 7f40e17fa640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f40c8076120 0x7f40c8078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.279+0000 7f40e17fa640 1 -- 172.21.15.81:0/1924414665 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6611+0+0 (secure 0 0 0) 0x7f40d8099690 con 0x7f40e406ec50 2024-06-29T14:41:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.279+0000 7f40e3fff640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f40c8076120 0x7f40c8078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.279+0000 7f40e3fff640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f40c8076120 0x7f40c8078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f40e41e3d40 tx=0x7f40d0008040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:09.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.283+0000 7f40e17fa640 1 -- 172.21.15.81:0/1924414665 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f40e409e2f0 con 0x7f40e406ec50 2024-06-29T14:41:09.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.501+0000 7f40eac9e640 1 -- 172.21.15.81:0/1924414665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f40e4073eb0 con 0x7f40e406ec50 2024-06-29T14:41:09.499 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T14:41:09.502+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:41:09.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.503+0000 7f40e17fa640 1 -- 172.21.15.81:0/1924414665 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v60) ==== 161+0+0 (secure 0 0 0) 0x7f40e41f2d80 con 0x7f40e406ec50 2024-06-29T14:41:09.501 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 -- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f40c8076120 msgr2=0x7f40c8078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f40c8076120 0x7f40c8078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f40e41e3d40 tx=0x7f40d0008040 comp rx=0 tx=0).stop 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 -- 172.21.15.81:0/1924414665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e406ec50 msgr2=0x7f40e41455c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e406ec50 0x7f40e41455c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f40d800d9c0 tx=0x7f40d800de80 comp rx=0 tx=0).stop 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 -- 172.21.15.81:0/1924414665 shutdown_connections 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f40c8076120 0x7f40c8078630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40e415afa0 0x7f40e41f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40e40744f0 0x7f40e41f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 --2- 172.21.15.81:0/1924414665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e406ec50 0x7f40e41455c0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 -- 172.21.15.81:0/1924414665 >> 172.21.15.81:0/1924414665 conn(0x7f40e40b1ec0 msgr2=0x7f40e40d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 -- 172.21.15.81:0/1924414665 shutdown_connections 2024-06-29T14:41:09.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.506+0000 7f40eac9e640 1 -- 172.21.15.81:0/1924414665 wait complete. 2024-06-29T14:41:09.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:41:09.563 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:41:09.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.638+0000 7f3057fc7640 1 -- 172.21.15.81:0/2847767077 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f303c0039e0 msgr2=0x7f303c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.638+0000 7f3057fc7640 1 --2- 172.21.15.81:0/2847767077 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f303c0039e0 0x7f303c003e30 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f304c00c310 tx=0x7f303c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:09.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.638+0000 7f3056564640 1 -- 172.21.15.81:0/2847767077 reap_dead start 2024-06-29T14:41:09.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.638+0000 7f3057fc7640 1 -- 172.21.15.81:0/2847767077 shutdown_connections 2024-06-29T14:41:09.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.638+0000 7f3057fc7640 1 -- 172.21.15.81:0/2847767077 >> 172.21.15.81:0/2847767077 conn(0x7f30500b34a0 msgr2=0x7f30500b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:09.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.638+0000 7f3057fc7640 1 -- 172.21.15.81:0/2847767077 shutdown_connections 2024-06-29T14:41:09.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.638+0000 7f3057fc7640 1 -- 172.21.15.81:0/2847767077 wait complete. 2024-06-29T14:41:09.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.640+0000 7f3057fc7640 1 Processor -- start 2024-06-29T14:41:09.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.640+0000 7f3057fc7640 1 -- start start 2024-06-29T14:41:09.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3057fc7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f303c0039e0 0x7f30501e1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3057fc7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30500b51a0 0x7f3050070730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3057fc7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30500b55b0 0x7f3050070c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3057fc7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f30500b1150 con 0x7f30500b55b0 2024-06-29T14:41:09.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3057fc7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f30500b0fd0 con 0x7f303c0039e0 2024-06-29T14:41:09.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3057fc7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f30500b12d0 con 0x7f30500b51a0 2024-06-29T14:41:09.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3056564640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30500b55b0 0x7f3050070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3056564640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30500b55b0 0x7f3050070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35188/0 (socket says 172.21.15.81:35188) 2024-06-29T14:41:09.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3056564640 1 -- 172.21.15.81:0/3716374447 learned_addr learned my addr 172.21.15.81:0/3716374447 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:09.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3055d63640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f303c0039e0 0x7f30501e1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.641+0000 7f3055562640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30500b51a0 0x7f3050070730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f3056564640 1 -- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30500b51a0 msgr2=0x7f3050070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f3056564640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30500b51a0 0x7f3050070730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f3056564640 1 -- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f303c0039e0 msgr2=0x7f30501e1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:09.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f3056564640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f303c0039e0 0x7f30501e1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:09.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f3056564640 1 -- 172.21.15.81:0/3716374447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30500b6400 con 0x7f30500b55b0 2024-06-29T14:41:09.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f3056564640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30500b55b0 0x7f3050070c70 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f30500b4580 tx=0x7f3048010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:09.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f30467fc640 1 -- 172.21.15.81:0/3716374447 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304800ee70 con 0x7f30500b55b0 2024-06-29T14:41:09.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f30467fc640 1 -- 172.21.15.81:0/3716374447 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3048018070 con 0x7f30500b55b0 2024-06-29T14:41:09.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f30467fc640 1 -- 172.21.15.81:0/3716374447 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048004d60 con 0x7f30500b55b0 2024-06-29T14:41:09.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f3057fc7640 1 -- 172.21.15.81:0/3716374447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f305009bc70 con 0x7f30500b55b0 2024-06-29T14:41:09.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.642+0000 7f3057fc7640 1 -- 172.21.15.81:0/3716374447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f305009c230 con 0x7f30500b55b0 2024-06-29T14:41:09.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.643+0000 7f3057fc7640 1 -- 172.21.15.81:0/3716374447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f305009b9e0 con 0x7f30500b55b0 2024-06-29T14:41:09.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.644+0000 7f30467fc640 1 -- 172.21.15.81:0/3716374447 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 67) ==== 98247+0+0 (secure 0 0 0) 0x7f3048003970 con 0x7f30500b55b0 2024-06-29T14:41:09.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.644+0000 7f30467fc640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f3034076390 0x7f30340788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:09.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.645+0000 7f30467fc640 1 -- 172.21.15.81:0/3716374447 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6669+0+0 (secure 0 0 0) 0x7f3048098550 con 0x7f30500b55b0 2024-06-29T14:41:09.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.645+0000 7f3055d63640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f3034076390 0x7f30340788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:09.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.645+0000 7f3055d63640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f3034076390 0x7f30340788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f304c033000 tx=0x7f304c038040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:09.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.649+0000 7f30467fc640 1 -- 172.21.15.81:0/3716374447 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3048061310 con 0x7f30500b55b0 2024-06-29T14:41:09.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:09.870+0000 7f3057fc7640 1 -- 172.21.15.81:0/3716374447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f30500d0400 con 0x7f30500b55b0 2024-06-29T14:41:10.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.505+0000 7f30467fc640 1 -- 172.21.15.81:0/3716374447 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v61) ==== 104+0+0 (secure 0 0 0) 0x7f3048065610 con 0x7f30500b55b0 2024-06-29T14:41:10.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.507+0000 7f3057fc7640 1 -- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f3034076390 msgr2=0x7f30340788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:10.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.507+0000 7f3057fc7640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f3034076390 0x7f30340788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f304c033000 tx=0x7f304c038040 comp rx=0 tx=0).stop 2024-06-29T14:41:10.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.507+0000 7f3057fc7640 1 -- 172.21.15.81:0/3716374447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30500b55b0 msgr2=0x7f3050070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:10.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.507+0000 7f3057fc7640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30500b55b0 0x7f3050070c70 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f30500b4580 tx=0x7f3048010f00 comp rx=0 tx=0).stop 2024-06-29T14:41:10.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.507+0000 7f3057fc7640 1 -- 172.21.15.81:0/3716374447 shutdown_connections 2024-06-29T14:41:10.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.507+0000 7f3057fc7640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f3034076390 0x7f30340788a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.507+0000 7f3057fc7640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30500b55b0 0x7f3050070c70 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.507+0000 7f3057fc7640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f30500b51a0 0x7f3050070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.507+0000 7f3057fc7640 1 --2- 172.21.15.81:0/3716374447 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f303c0039e0 0x7f30501e1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.507+0000 7f3057fc7640 1 -- 172.21.15.81:0/3716374447 >> 172.21.15.81:0/3716374447 conn(0x7f30500b34a0 msgr2=0x7f30500af020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:10.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.508+0000 7f3057fc7640 1 -- 172.21.15.81:0/3716374447 shutdown_connections 2024-06-29T14:41:10.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.508+0000 7f3057fc7640 1 -- 172.21.15.81:0/3716374447 wait complete. 2024-06-29T14:41:10.517 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:41:10.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.636+0000 7f1d8adb9640 1 -- 172.21.15.81:0/2388321580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d840cf8e0 msgr2=0x7f1d840d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:10.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.636+0000 7f1d8adb9640 1 --2- 172.21.15.81:0/2388321580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d840cf8e0 0x7f1d840d0680 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f1d8408c7b0 tx=0x7f1d7402a5f0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.636+0000 7f1d8adb9640 1 -- 172.21.15.81:0/2388321580 shutdown_connections 2024-06-29T14:41:10.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.636+0000 7f1d8adb9640 1 --2- 172.21.15.81:0/2388321580 >> v2:172.21.15.167:3300/0 conn(0x7f1d840d0bc0 0x7f1d840d0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.636+0000 7f1d8adb9640 1 --2- 172.21.15.81:0/2388321580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d840cf8e0 0x7f1d840d0680 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.636+0000 7f1d8adb9640 1 -- 172.21.15.81:0/2388321580 >> 172.21.15.81:0/2388321580 conn(0x7f1d840d3cf0 msgr2=0x7f1d840d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.636+0000 7f1d8adb9640 1 -- 172.21.15.81:0/2388321580 shutdown_connections 2024-06-29T14:41:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.636+0000 7f1d8adb9640 1 -- 172.21.15.81:0/2388321580 wait complete. 2024-06-29T14:41:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.638+0000 7f1d8adb9640 1 Processor -- start 2024-06-29T14:41:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.638+0000 7f1d8adb9640 1 -- start start 2024-06-29T14:41:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d8adb9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d840cf4d0 0x7f1d841e1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d8adb9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d840cf8e0 0x7f1d840a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d8adb9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d840d0bc0 0x7f1d840a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d8adb9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1d840d2150 con 0x7f1d840cf8e0 2024-06-29T14:41:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d8adb9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1d840d1fd0 con 0x7f1d840cf4d0 2024-06-29T14:41:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d8adb9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1d840b47f0 con 0x7f1d840d0bc0 2024-06-29T14:41:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d7bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d840cf8e0 0x7f1d840a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d7bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d840cf8e0 0x7f1d840a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60412/0 (socket says 172.21.15.81:60412) 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d7bfff640 1 -- 172.21.15.81:0/3017638885 learned_addr learned my addr 172.21.15.81:0/3017638885 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d89356640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d840d0bc0 0x7f1d840a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d88b55640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d840cf4d0 0x7f1d841e1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d7bfff640 1 -- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d840d0bc0 msgr2=0x7f1d840a1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d7bfff640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d840d0bc0 0x7f1d840a1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d7bfff640 1 -- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d840cf4d0 msgr2=0x7f1d841e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d7bfff640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d840cf4d0 0x7f1d841e1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d7bfff640 1 -- 172.21.15.81:0/3017638885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d840a1890 con 0x7f1d840cf8e0 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d89356640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d840d0bc0 0x7f1d840a1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.640+0000 7f1d7bfff640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d840cf8e0 0x7f1d840a0270 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f1d74034040 tx=0x7f1d740039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.641+0000 7f1d797fa640 1 -- 172.21.15.81:0/3017638885 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7403e030 con 0x7f1d840cf8e0 2024-06-29T14:41:10.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.641+0000 7f1d797fa640 1 -- 172.21.15.81:0/3017638885 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d740047a0 con 0x7f1d840cf8e0 2024-06-29T14:41:10.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.641+0000 7f1d8adb9640 1 -- 172.21.15.81:0/3017638885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d841e10d0 con 0x7f1d840cf8e0 2024-06-29T14:41:10.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.641+0000 7f1d797fa640 1 -- 172.21.15.81:0/3017638885 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74035040 con 0x7f1d840cf8e0 2024-06-29T14:41:10.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.641+0000 7f1d8adb9640 1 -- 172.21.15.81:0/3017638885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d841e1690 con 0x7f1d840cf8e0 2024-06-29T14:41:10.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.642+0000 7f1d8adb9640 1 -- 172.21.15.81:0/3017638885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d840d0680 con 0x7f1d840cf8e0 2024-06-29T14:41:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.642+0000 7f1d797fa640 1 -- 172.21.15.81:0/3017638885 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 67) ==== 98247+0+0 (secure 0 0 0) 0x7f1d74041030 con 0x7f1d840cf8e0 2024-06-29T14:41:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.643+0000 7f1d797fa640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f1d70076360 0x7f1d70078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.643+0000 7f1d797fa640 1 -- 172.21.15.81:0/3017638885 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6669+0+0 (secure 0 0 0) 0x7f1d740b8470 con 0x7f1d840cf8e0 2024-06-29T14:41:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.643+0000 7f1d88b55640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f1d70076360 0x7f1d70078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.643+0000 7f1d88b55640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f1d70076360 0x7f1d70078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1d80000f60 tx=0x7f1d80009040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:10.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.647+0000 7f1d797fa640 1 -- 172.21.15.81:0/3017638885 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1d74081230 con 0x7f1d840cf8e0 2024-06-29T14:41:10.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:10.826+0000 7f1d8adb9640 1 -- 172.21.15.81:0/3017638885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1d840d0250 con 0x7f1d840cf8e0 2024-06-29T14:41:11.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.632+0000 7f1d797fa640 1 -- 172.21.15.81:0/3017638885 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v124) ==== 181+0+0 (secure 0 0 0) 0x7f1d74085530 con 0x7f1d840cf8e0 2024-06-29T14:41:11.629 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:41:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 -- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f1d70076360 msgr2=0x7f1d70078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f1d70076360 0x7f1d70078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1d80000f60 tx=0x7f1d80009040 comp rx=0 tx=0).stop 2024-06-29T14:41:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 -- 172.21.15.81:0/3017638885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d840cf8e0 msgr2=0x7f1d840a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d840cf8e0 0x7f1d840a0270 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f1d74034040 tx=0x7f1d740039e0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 -- 172.21.15.81:0/3017638885 shutdown_connections 2024-06-29T14:41:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f1d70076360 0x7f1d70078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d840d0bc0 0x7f1d840a1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d840cf8e0 0x7f1d840a0270 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 --2- 172.21.15.81:0/3017638885 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d840cf4d0 0x7f1d841e1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 -- 172.21.15.81:0/3017638885 >> 172.21.15.81:0/3017638885 conn(0x7f1d840d3cf0 msgr2=0x7f1d841e6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:11.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 -- 172.21.15.81:0/3017638885 shutdown_connections 2024-06-29T14:41:11.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.634+0000 7f1d8adb9640 1 -- 172.21.15.81:0/3017638885 wait complete. 2024-06-29T14:41:11.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:41:11.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.775+0000 7f9dd5a65640 1 -- 172.21.15.81:0/2363229758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9db8004800 msgr2=0x7f9db8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:11.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.775+0000 7f9dd5a65640 1 --2- 172.21.15.81:0/2363229758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9db8004800 0x7f9db8004c10 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40016c0 tx=0x7f9db8005830 comp rx=0 tx=0).stop 2024-06-29T14:41:11.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.775+0000 7f9dcffff640 1 -- 172.21.15.81:0/2363229758 reap_dead start 2024-06-29T14:41:11.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.775+0000 7f9dd5a65640 1 -- 172.21.15.81:0/2363229758 shutdown_connections 2024-06-29T14:41:11.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.775+0000 7f9dd5a65640 1 -- 172.21.15.81:0/2363229758 >> 172.21.15.81:0/2363229758 conn(0x7f9dd00d08f0 msgr2=0x7f9dd00d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:11.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.775+0000 7f9dd5a65640 1 -- 172.21.15.81:0/2363229758 shutdown_connections 2024-06-29T14:41:11.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.775+0000 7f9dd5a65640 1 -- 172.21.15.81:0/2363229758 wait complete. 2024-06-29T14:41:11.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.777+0000 7f9dd5a65640 1 Processor -- start 2024-06-29T14:41:11.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.777+0000 7f9dd5a65640 1 -- start start 2024-06-29T14:41:11.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dd5a65640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9db8004800 0x7f9dd01f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:11.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dd5a65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd00aae70 0x7f9dd00ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:11.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dd5a65640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9dd0158530 0x7f9dd0158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:11.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dd5a65640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd006cb80 con 0x7f9dd00aae70 2024-06-29T14:41:11.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dd5a65640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd006ca00 con 0x7f9dd0158530 2024-06-29T14:41:11.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dd5a65640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9dd00cfa60 con 0x7f9db8004800 2024-06-29T14:41:11.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dceffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd00aae70 0x7f9dd00ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dceffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd00aae70 0x7f9dd00ab280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60434/0 (socket says 172.21.15.81:60434) 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dceffd640 1 -- 172.21.15.81:0/961352064 learned_addr learned my addr 172.21.15.81:0/961352064 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dcf7fe640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9db8004800 0x7f9dd01f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dcffff640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9dd0158530 0x7f9dd0158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dceffd640 1 -- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9db8004800 msgr2=0x7f9dd01f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dceffd640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9db8004800 0x7f9dd01f2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dceffd640 1 -- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9dd0158530 msgr2=0x7f9dd0158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dceffd640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9dd0158530 0x7f9dd0158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dceffd640 1 -- 172.21.15.81:0/961352064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd0158e80 con 0x7f9dd00aae70 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dceffd640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd00aae70 0x7f9dd00ab280 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f9dc402a930 tx=0x7f9dc4006310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9db7fff640 1 -- 172.21.15.81:0/961352064 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4041070 con 0x7f9dd00aae70 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9db7fff640 1 -- 172.21.15.81:0/961352064 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9dc4003a00 con 0x7f9dd00aae70 2024-06-29T14:41:11.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dd5a65640 1 -- 172.21.15.81:0/961352064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd00a9640 con 0x7f9dd00aae70 2024-06-29T14:41:11.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9db7fff640 1 -- 172.21.15.81:0/961352064 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4038070 con 0x7f9dd00aae70 2024-06-29T14:41:11.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.779+0000 7f9dd5a65640 1 -- 172.21.15.81:0/961352064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd00a9c00 con 0x7f9dd00aae70 2024-06-29T14:41:11.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.780+0000 7f9dd5a65640 1 -- 172.21.15.81:0/961352064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd0075c90 con 0x7f9dd00aae70 2024-06-29T14:41:11.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.781+0000 7f9db7fff640 1 -- 172.21.15.81:0/961352064 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f9dc4030070 con 0x7f9dd00aae70 2024-06-29T14:41:11.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.781+0000 7f9db7fff640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dbc0763f0 0x7f9dbc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:11.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.782+0000 7f9db7fff640 1 -- 172.21.15.81:0/961352064 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6231+0+0 (secure 0 0 0) 0x7f9dc40b7190 con 0x7f9dd00aae70 2024-06-29T14:41:11.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.782+0000 7f9dcf7fe640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dbc0763f0 0x7f9dbc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:11.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.782+0000 7f9dcf7fe640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dbc0763f0 0x7f9dbc078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0001860 tx=0x7f9dc0001110 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:11.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.786+0000 7f9db7fff640 1 -- 172.21.15.81:0/961352064 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9dd0075c90 con 0x7f9dd00aae70 2024-06-29T14:41:11.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:41:11.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:41:11.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:11.968+0000 7f9dd5a65640 1 -- 172.21.15.81:0/961352064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9dd006c2f0 con 0x7f9dd00aae70 2024-06-29T14:41:12.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.633+0000 7f9db7fff640 1 -- 172.21.15.81:0/961352064 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v125) ==== 169+0+0 (secure 0 0 0) 0x7f9dd0158e80 con 0x7f9dd00aae70 2024-06-29T14:41:12.631 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T14:41:12.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 -- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dbc0763f0 msgr2=0x7f9dbc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dbc0763f0 0x7f9dbc078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0001860 tx=0x7f9dc0001110 comp rx=0 tx=0).stop 2024-06-29T14:41:12.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 -- 172.21.15.81:0/961352064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd00aae70 msgr2=0x7f9dd00ab280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd00aae70 0x7f9dd00ab280 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f9dc402a930 tx=0x7f9dc4006310 comp rx=0 tx=0).stop 2024-06-29T14:41:12.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 -- 172.21.15.81:0/961352064 shutdown_connections 2024-06-29T14:41:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dbc0763f0 0x7f9dbc078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9dd0158530 0x7f9dd0158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd00aae70 0x7f9dd00ab280 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 --2- 172.21.15.81:0/961352064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9db8004800 0x7f9dd01f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 -- 172.21.15.81:0/961352064 >> 172.21.15.81:0/961352064 conn(0x7f9dd00d08f0 msgr2=0x7f9dd01559b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.636+0000 7f9dd5a65640 1 -- 172.21.15.81:0/961352064 shutdown_connections 2024-06-29T14:41:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.637+0000 7f9dd5a65640 1 -- 172.21.15.81:0/961352064 wait complete. 2024-06-29T14:41:12.648 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:41:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.806+0000 7f0ef732a640 1 -- 172.21.15.81:0/2809530510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ef015e960 msgr2=0x7f0ef015ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.806+0000 7f0ef732a640 1 --2- 172.21.15.81:0/2809530510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ef015e960 0x7f0ef015ed70 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009830 tx=0x7f0ee802a5c0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.806+0000 7f0ef732a640 1 -- 172.21.15.81:0/2809530510 shutdown_connections 2024-06-29T14:41:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.806+0000 7f0ef732a640 1 --2- 172.21.15.81:0/2809530510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ef015e960 0x7f0ef015ed70 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.806+0000 7f0ef732a640 1 -- 172.21.15.81:0/2809530510 >> 172.21.15.81:0/2809530510 conn(0x7f0ef006ccc0 msgr2=0x7f0ef00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.806+0000 7f0ef732a640 1 -- 172.21.15.81:0/2809530510 shutdown_connections 2024-06-29T14:41:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.806+0000 7f0ef732a640 1 -- 172.21.15.81:0/2809530510 wait complete. 2024-06-29T14:41:12.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.808+0000 7f0ef732a640 1 Processor -- start 2024-06-29T14:41:12.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.808+0000 7f0ef732a640 1 -- start start 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef732a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ef0073030 0x7f0ef0140250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef732a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ef01412f0 0x7f0ef01586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef732a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ef015e960 0x7f0ef00ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef732a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef00b3390 con 0x7f0ef01412f0 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef732a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef00b3210 con 0x7f0ef0073030 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef732a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef00b3510 con 0x7f0ef015e960 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef48c5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ef01412f0 0x7f0ef01586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef48c5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ef01412f0 0x7f0ef01586a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60464/0 (socket says 172.21.15.81:60464) 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef48c5640 1 -- 172.21.15.81:0/4157718969 learned_addr learned my addr 172.21.15.81:0/4157718969 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef48c5640 1 -- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ef015e960 msgr2=0x7f0ef00ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef58c7640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ef015e960 0x7f0ef00ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.809+0000 7f0ef50c6640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ef0073030 0x7f0ef0140250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.810+0000 7f0ef48c5640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ef015e960 0x7f0ef00ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.810+0000 7f0ef48c5640 1 -- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ef0073030 msgr2=0x7f0ef0140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:12.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.810+0000 7f0ef48c5640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ef0073030 0x7f0ef0140250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:12.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.810+0000 7f0ef48c5640 1 -- 172.21.15.81:0/4157718969 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef00a9710 con 0x7f0ef01412f0 2024-06-29T14:41:12.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.810+0000 7f0ef48c5640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ef01412f0 0x7f0ef01586a0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400c3a0 tx=0x7f0ee400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:12.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.810+0000 7f0eedffb640 1 -- 172.21.15.81:0/4157718969 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4017070 con 0x7f0ef01412f0 2024-06-29T14:41:12.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.810+0000 7f0eedffb640 1 -- 172.21.15.81:0/4157718969 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ee40053c0 con 0x7f0ef01412f0 2024-06-29T14:41:12.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.810+0000 7f0eedffb640 1 -- 172.21.15.81:0/4157718969 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4003850 con 0x7f0ef01412f0 2024-06-29T14:41:12.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.810+0000 7f0ef732a640 1 -- 172.21.15.81:0/4157718969 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef00a99f0 con 0x7f0ef01412f0 2024-06-29T14:41:12.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.810+0000 7f0ef732a640 1 -- 172.21.15.81:0/4157718969 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef00a9fb0 con 0x7f0ef01412f0 2024-06-29T14:41:12.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.811+0000 7f0ef732a640 1 -- 172.21.15.81:0/4157718969 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef00b3a30 con 0x7f0ef01412f0 2024-06-29T14:41:12.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.812+0000 7f0eedffb640 1 -- 172.21.15.81:0/4157718969 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f0ee401a070 con 0x7f0ef01412f0 2024-06-29T14:41:12.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.812+0000 7f0eedffb640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f0edc076420 0x7f0edc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:12.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.812+0000 7f0eedffb640 1 -- 172.21.15.81:0/4157718969 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5817+0+0 (secure 0 0 0) 0x7f0ee4096dd0 con 0x7f0ef01412f0 2024-06-29T14:41:12.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.812+0000 7f0ef50c6640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f0edc076420 0x7f0edc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:12.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.813+0000 7f0ef50c6640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f0edc076420 0x7f0edc078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0004a30 tx=0x7f0ee0009040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:12.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.816+0000 7f0eedffb640 1 -- 172.21.15.81:0/4157718969 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0ee4060520 con 0x7f0ef01412f0 2024-06-29T14:41:12.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:12.997+0000 7f0ef732a640 1 -- 172.21.15.81:0/4157718969 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f0ef015a1a0 con 0x7f0ef01412f0 2024-06-29T14:41:12.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.002+0000 7f0eedffb640 1 -- 172.21.15.81:0/4157718969 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v51) ==== 71+0+0 (secure 0 0 0) 0x7f0ee4098080 con 0x7f0ef01412f0 2024-06-29T14:41:13.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.004+0000 7f0ef732a640 1 -- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f0edc076420 msgr2=0x7f0edc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f0edc076420 0x7f0edc078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0004a30 tx=0x7f0ee0009040 comp rx=0 tx=0).stop 2024-06-29T14:41:13.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 -- 172.21.15.81:0/4157718969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ef01412f0 msgr2=0x7f0ef01586a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ef01412f0 0x7f0ef01586a0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400c3a0 tx=0x7f0ee400c860 comp rx=0 tx=0).stop 2024-06-29T14:41:13.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 -- 172.21.15.81:0/4157718969 shutdown_connections 2024-06-29T14:41:13.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f0edc076420 0x7f0edc078930 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ef015e960 0x7f0ef00ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ef01412f0 0x7f0ef01586a0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 --2- 172.21.15.81:0/4157718969 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ef0073030 0x7f0ef0140250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 -- 172.21.15.81:0/4157718969 >> 172.21.15.81:0/4157718969 conn(0x7f0ef006ccc0 msgr2=0x7f0ef0159620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:13.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 -- 172.21.15.81:0/4157718969 shutdown_connections 2024-06-29T14:41:13.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.005+0000 7f0ef732a640 1 -- 172.21.15.81:0/4157718969 wait complete. 2024-06-29T14:41:13.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2024-06-29T14:41:13.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.131+0000 7f00cd609640 1 -- 172.21.15.81:0/1043471280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f00c80b6eb0 msgr2=0x7f00c80d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.131+0000 7f00cd609640 1 --2- 172.21.15.81:0/1043471280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f00c80b6eb0 0x7f00c80d02d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f00c80b41b0 tx=0x7f00b802aa00 comp rx=0 tx=0).stop 2024-06-29T14:41:13.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.131+0000 7f00cd609640 1 -- 172.21.15.81:0/1043471280 shutdown_connections 2024-06-29T14:41:13.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.131+0000 7f00cd609640 1 --2- 172.21.15.81:0/1043471280 >> v2:172.21.15.167:3300/0 conn(0x7f00c80ce510 0x7f00c80ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.131+0000 7f00cd609640 1 --2- 172.21.15.81:0/1043471280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f00c80b6eb0 0x7f00c80d02d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.131+0000 7f00cd609640 1 -- 172.21.15.81:0/1043471280 >> 172.21.15.81:0/1043471280 conn(0x7f00c80b3510 msgr2=0x7f00c80b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:13.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.131+0000 7f00cd609640 1 -- 172.21.15.81:0/1043471280 shutdown_connections 2024-06-29T14:41:13.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.131+0000 7f00cd609640 1 -- 172.21.15.81:0/1043471280 wait complete. 2024-06-29T14:41:13.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.133+0000 7f00cd609640 1 Processor -- start 2024-06-29T14:41:13.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.133+0000 7f00cd609640 1 -- start start 2024-06-29T14:41:13.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.134+0000 7f00cd609640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f00c80b4810 0x7f00c806d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00cd609640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c80b6eb0 0x7f00c814ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00cd609640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f00c80ce510 0x7f00c806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00cd609640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f00c80cd480 con 0x7f00c80b6eb0 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00cd609640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f00c80b29c0 con 0x7f00c80b4810 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00cd609640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f00c80cd600 con 0x7f00c80ce510 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c80b6eb0 0x7f00c814ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c80b6eb0 0x7f00c814ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60468/0 (socket says 172.21.15.81:60468) 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c67fc640 1 -- 172.21.15.81:0/460534506 learned_addr learned my addr 172.21.15.81:0/460534506 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c6ffd640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f00c80b4810 0x7f00c806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c77fe640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f00c80ce510 0x7f00c806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c67fc640 1 -- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f00c80ce510 msgr2=0x7f00c806eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c67fc640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f00c80ce510 0x7f00c806eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c67fc640 1 -- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f00c80b4810 msgr2=0x7f00c806d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c67fc640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f00c80b4810 0x7f00c806d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c67fc640 1 -- 172.21.15.81:0/460534506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c806dd00 con 0x7f00c80b6eb0 2024-06-29T14:41:13.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00c67fc640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c80b6eb0 0x7f00c814ad20 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00ea20 tx=0x7f00bc00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:13.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00af7fe640 1 -- 172.21.15.81:0/460534506 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc00ce40 con 0x7f00c80b6eb0 2024-06-29T14:41:13.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00af7fe640 1 -- 172.21.15.81:0/460534506 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00bc004020 con 0x7f00c80b6eb0 2024-06-29T14:41:13.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00cd609640 1 -- 172.21.15.81:0/460534506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c806dfe0 con 0x7f00c80b6eb0 2024-06-29T14:41:13.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00af7fe640 1 -- 172.21.15.81:0/460534506 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc004470 con 0x7f00c80b6eb0 2024-06-29T14:41:13.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.135+0000 7f00cd609640 1 -- 172.21.15.81:0/460534506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c806e5a0 con 0x7f00c80b6eb0 2024-06-29T14:41:13.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.136+0000 7f00cd609640 1 -- 172.21.15.81:0/460534506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c80d02d0 con 0x7f00c80b6eb0 2024-06-29T14:41:13.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.137+0000 7f00af7fe640 1 -- 172.21.15.81:0/460534506 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f00bc0035f0 con 0x7f00c80b6eb0 2024-06-29T14:41:13.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.137+0000 7f00af7fe640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f00b40763f0 0x7f00b4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.138+0000 7f00af7fe640 1 -- 172.21.15.81:0/460534506 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5817+0+0 (secure 0 0 0) 0x7f00bc0983c0 con 0x7f00c80b6eb0 2024-06-29T14:41:13.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.138+0000 7f00c6ffd640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f00b40763f0 0x7f00b4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.138+0000 7f00c6ffd640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f00b40763f0 0x7f00b4078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f00b8035000 tx=0x7f00b803b040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:13.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.142+0000 7f00af7fe640 1 -- 172.21.15.81:0/460534506 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f00bc0613c0 con 0x7f00c80b6eb0 2024-06-29T14:41:13.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.406+0000 7f00cd609640 1 -- 172.21.15.81:0/460534506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) -- 0x7f00c806da90 con 0x7f00c80b6eb0 2024-06-29T14:41:13.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.642+0000 7f00af7fe640 1 -- 172.21.15.81:0/460534506 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v701) ==== 148+0+0 (secure 0 0 0) 0x7f00bc0656c0 con 0x7f00c80b6eb0 2024-06-29T14:41:13.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 -- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f00b40763f0 msgr2=0x7f00b4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f00b40763f0 0x7f00b4078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f00b8035000 tx=0x7f00b803b040 comp rx=0 tx=0).stop 2024-06-29T14:41:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 -- 172.21.15.81:0/460534506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c80b6eb0 msgr2=0x7f00c814ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c80b6eb0 0x7f00c814ad20 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00ea20 tx=0x7f00bc00eee0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 -- 172.21.15.81:0/460534506 shutdown_connections 2024-06-29T14:41:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f00b40763f0 0x7f00b4078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f00c80ce510 0x7f00c806eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c80b6eb0 0x7f00c814ad20 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 --2- 172.21.15.81:0/460534506 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f00c80b4810 0x7f00c806d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 -- 172.21.15.81:0/460534506 >> 172.21.15.81:0/460534506 conn(0x7f00c80b3510 msgr2=0x7f00c8070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 -- 172.21.15.81:0/460534506 shutdown_connections 2024-06-29T14:41:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.644+0000 7f00cd609640 1 -- 172.21.15.81:0/460534506 wait complete. 2024-06-29T14:41:13.653 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2024-06-29T14:41:13.653 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:41:13.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:13.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.768+0000 7fabbcc65640 1 -- 172.21.15.81:0/3436498887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabb80d0b90 msgr2=0x7fabb80d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.768+0000 7fabbcc65640 1 --2- 172.21.15.81:0/3436498887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabb80d0b90 0x7fabb80d1a80 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fabb808e330 tx=0x7fabac02a680 comp rx=0 tx=0).stop 2024-06-29T14:41:13.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.768+0000 7fabbcc65640 1 -- 172.21.15.81:0/3436498887 shutdown_connections 2024-06-29T14:41:13.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.768+0000 7fabbcc65640 1 --2- 172.21.15.81:0/3436498887 >> v2:172.21.15.167:3300/0 conn(0x7fabb80b5250 0x7fabb80d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.768+0000 7fabbcc65640 1 --2- 172.21.15.81:0/3436498887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabb80d0b90 0x7fabb80d1a80 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.768+0000 7fabbcc65640 1 -- 172.21.15.81:0/3436498887 >> 172.21.15.81:0/3436498887 conn(0x7fabb809cf30 msgr2=0x7fabb80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:13.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.768+0000 7fabbcc65640 1 -- 172.21.15.81:0/3436498887 shutdown_connections 2024-06-29T14:41:13.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.768+0000 7fabbcc65640 1 -- 172.21.15.81:0/3436498887 wait complete. 2024-06-29T14:41:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.770+0000 7fabbcc65640 1 Processor -- start 2024-06-29T14:41:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.770+0000 7fabbcc65640 1 -- start start 2024-06-29T14:41:13.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabbcc65640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fabb80b5250 0x7fabb815c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabbcc65640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabb80b9040 0x7fabb815ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabbcc65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabb81f5b70 0x7fabb81f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabbcc65640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fabb80d2500 con 0x7fabb81f5b70 2024-06-29T14:41:13.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabbcc65640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fabb80b8530 con 0x7fabb80b5250 2024-06-29T14:41:13.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabbcc65640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fabb80d2680 con 0x7fabb80b9040 2024-06-29T14:41:13.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabb6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabb81f5b70 0x7fabb81f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabb6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabb81f5b70 0x7fabb81f5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60482/0 (socket says 172.21.15.81:60482) 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabb6d9d640 1 -- 172.21.15.81:0/3103695469 learned_addr learned my addr 172.21.15.81:0/3103695469 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabb5d9b640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabb80b9040 0x7fabb815ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.771+0000 7fabb659c640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fabb80b5250 0x7fabb815c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7fabb6d9d640 1 -- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabb80b9040 msgr2=0x7fabb815ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7fabb6d9d640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabb80b9040 0x7fabb815ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7fabb6d9d640 1 -- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fabb80b5250 msgr2=0x7fabb815c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7fabb6d9d640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fabb80b5250 0x7fabb815c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7fabb6d9d640 1 -- 172.21.15.81:0/3103695469 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb815b180 con 0x7fabb81f5b70 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7fabb5d9b640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabb80b9040 0x7fabb815ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7fabb6d9d640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabb81f5b70 0x7fabb81f5f80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fabb000d5f0 tx=0x7fabb000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7faba6ffd640 1 -- 172.21.15.81:0/3103695469 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0017070 con 0x7fabb81f5b70 2024-06-29T14:41:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7faba6ffd640 1 -- 172.21.15.81:0/3103695469 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabb0005330 con 0x7fabb81f5b70 2024-06-29T14:41:13.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7fabbcc65640 1 -- 172.21.15.81:0/3103695469 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb81f58e0 con 0x7fabb81f5b70 2024-06-29T14:41:13.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7faba6ffd640 1 -- 172.21.15.81:0/3103695469 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb00047a0 con 0x7fabb81f5b70 2024-06-29T14:41:13.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.772+0000 7fabbcc65640 1 -- 172.21.15.81:0/3103695469 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb815b750 con 0x7fabb81f5b70 2024-06-29T14:41:13.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.773+0000 7fabbcc65640 1 -- 172.21.15.81:0/3103695469 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb81f5630 con 0x7fabb81f5b70 2024-06-29T14:41:13.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.774+0000 7faba6ffd640 1 -- 172.21.15.81:0/3103695469 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fabb0004940 con 0x7fabb81f5b70 2024-06-29T14:41:13.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.774+0000 7faba6ffd640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fab98076420 0x7fab98078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:13.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.774+0000 7faba6ffd640 1 -- 172.21.15.81:0/3103695469 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5817+0+0 (secure 0 0 0) 0x7fabb0097af0 con 0x7fabb81f5b70 2024-06-29T14:41:13.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.774+0000 7fabb659c640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fab98076420 0x7fab98078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:13.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.775+0000 7fabb659c640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fab98076420 0x7fab98078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faba8007fd0 tx=0x7faba8009040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:13.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.779+0000 7faba6ffd640 1 -- 172.21.15.81:0/3103695469 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fabb00610b0 con 0x7fabb81f5b70 2024-06-29T14:41:13.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.957+0000 7fabbcc65640 1 -- 172.21.15.81:0/3103695469 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fabb806deb0 con 0x7fabb81f5b70 2024-06-29T14:41:13.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.958+0000 7faba6ffd640 1 -- 172.21.15.81:0/3103695469 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) ==== 74+0+14316 (secure 0 0 0) 0x7fabb0069130 con 0x7fabb81f5b70 2024-06-29T14:41:13.955 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:13.955 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":125,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:12.632400+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6832/1049868964":"2024-06-30T14:41:09.502847+0000","172.21.15.167:6833/1146990244":"2024-06-30T14:41:07.251243+0000","172.21.15.167:6832/1146990244":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/3014421868":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/3886955317":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/2019716400":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/421920145":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/977399066":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/586922734":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/2613865897":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/2148809735":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/1014005913":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/705657234":"2024-06-30T14:40:09.486395+0000","172.21.15.81:6833/1049868964":"2024-06-30T14:41:09.502847+0000","172.21.15.167:0/4108453499":"2024-06-30T14:41:07.251243+0000","172.21.15.167:0/2888901478":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/1794407759":"2024-06-30T14:40:09.486395+0000","172.21.15.167:0/839096394":"2024-06-30T14:40:09.486395+0000","172.21.15.167:6833/473361950":"2024-06-30T14:40:09.486395+0000","172.21.15.167:6832/473361950":"2024-06-30T14:40:09.486395+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:13.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 -- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fab98076420 msgr2=0x7fab98078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fab98076420 0x7fab98078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faba8007fd0 tx=0x7faba8009040 comp rx=0 tx=0).stop 2024-06-29T14:41:13.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 -- 172.21.15.81:0/3103695469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabb81f5b70 msgr2=0x7fabb81f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:13.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabb81f5b70 0x7fabb81f5f80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fabb000d5f0 tx=0x7fabb000dab0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 -- 172.21.15.81:0/3103695469 shutdown_connections 2024-06-29T14:41:13.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fab98076420 0x7fab98078930 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabb81f5b70 0x7fabb81f5f80 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabb80b9040 0x7fabb815ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 --2- 172.21.15.81:0/3103695469 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fabb80b5250 0x7fabb815c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:13.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 -- 172.21.15.81:0/3103695469 >> 172.21.15.81:0/3103695469 conn(0x7fabb809cf30 msgr2=0x7fabb815f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:13.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 -- 172.21.15.81:0/3103695469 shutdown_connections 2024-06-29T14:41:13.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:13.960+0000 7fabbcc65640 1 -- 172.21.15.81:0/3103695469 wait complete. 2024-06-29T14:41:13.968 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:13.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:14.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.087+0000 7fdc77889640 1 -- 172.21.15.81:0/11779637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc700ae840 msgr2=0x7fdc700aec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.087+0000 7fdc77889640 1 --2- 172.21.15.81:0/11779637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc700ae840 0x7fdc700aec50 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fdc68005fd0 tx=0x7fdc6802a530 comp rx=0 tx=0).stop 2024-06-29T14:41:14.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.087+0000 7fdc77889640 1 -- 172.21.15.81:0/11779637 shutdown_connections 2024-06-29T14:41:14.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.087+0000 7fdc77889640 1 --2- 172.21.15.81:0/11779637 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7006f850 0x7fdc700ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.087+0000 7fdc77889640 1 --2- 172.21.15.81:0/11779637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc700ae840 0x7fdc700aec50 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.087+0000 7fdc77889640 1 -- 172.21.15.81:0/11779637 >> 172.21.15.81:0/11779637 conn(0x7fdc70072950 msgr2=0x7fdc700b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:14.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.087+0000 7fdc77889640 1 -- 172.21.15.81:0/11779637 shutdown_connections 2024-06-29T14:41:14.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.087+0000 7fdc77889640 1 -- 172.21.15.81:0/11779637 wait complete. 2024-06-29T14:41:14.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.089+0000 7fdc77889640 1 Processor -- start 2024-06-29T14:41:14.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.089+0000 7fdc77889640 1 -- start start 2024-06-29T14:41:14.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.090+0000 7fdc77889640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7006f850 0x7fdc7014e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.090+0000 7fdc77889640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70074520 0x7fdc7015ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.090+0000 7fdc77889640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc700ae840 0x7fdc7015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.090+0000 7fdc77889640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7015b190 con 0x7fdc70074520 2024-06-29T14:41:14.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.090+0000 7fdc77889640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7015b010 con 0x7fdc700ae840 2024-06-29T14:41:14.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.090+0000 7fdc77889640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdc700af190 con 0x7fdc7006f850 2024-06-29T14:41:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc74e24640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70074520 0x7fdc7015ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc74e24640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70074520 0x7fdc7015ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60496/0 (socket says 172.21.15.81:60496) 2024-06-29T14:41:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc74e24640 1 -- 172.21.15.81:0/3506194243 learned_addr learned my addr 172.21.15.81:0/3506194243 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc75e26640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc700ae840 0x7fdc7015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc75625640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7006f850 0x7fdc7014e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc74e24640 1 -- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7006f850 msgr2=0x7fdc7014e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc74e24640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7006f850 0x7fdc7014e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc74e24640 1 -- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc700ae840 msgr2=0x7fdc7015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc74e24640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc700ae840 0x7fdc7015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc74e24640 1 -- 172.21.15.81:0/3506194243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc701f2cf0 con 0x7fdc70074520 2024-06-29T14:41:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc75e26640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc700ae840 0x7fdc7015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc75625640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7006f850 0x7fdc7014e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc74e24640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70074520 0x7fdc7015ca50 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00c9a0 tx=0x7fdc6c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc61ffb640 1 -- 172.21.15.81:0/3506194243 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c007b50 con 0x7fdc70074520 2024-06-29T14:41:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc61ffb640 1 -- 172.21.15.81:0/3506194243 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc6c007de0 con 0x7fdc70074520 2024-06-29T14:41:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc77889640 1 -- 172.21.15.81:0/3506194243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc7015d510 con 0x7fdc70074520 2024-06-29T14:41:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc61ffb640 1 -- 172.21.15.81:0/3506194243 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c00f6b0 con 0x7fdc70074520 2024-06-29T14:41:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.091+0000 7fdc77889640 1 -- 172.21.15.81:0/3506194243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc701f32b0 con 0x7fdc70074520 2024-06-29T14:41:14.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.093+0000 7fdc77889640 1 -- 172.21.15.81:0/3506194243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc700ae300 con 0x7fdc70074520 2024-06-29T14:41:14.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.093+0000 7fdc61ffb640 1 -- 172.21.15.81:0/3506194243 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fdc6c00f850 con 0x7fdc70074520 2024-06-29T14:41:14.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.093+0000 7fdc61ffb640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fdc54076420 0x7fdc54078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.094+0000 7fdc61ffb640 1 -- 172.21.15.81:0/3506194243 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5817+0+0 (secure 0 0 0) 0x7fdc6c098150 con 0x7fdc70074520 2024-06-29T14:41:14.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.094+0000 7fdc75625640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fdc54076420 0x7fdc54078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:14.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.094+0000 7fdc75625640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fdc54076420 0x7fdc54078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009fc0 tx=0x7fdc64009550 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:14.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.098+0000 7fdc61ffb640 1 -- 172.21.15.81:0/3506194243 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc6c014030 con 0x7fdc70074520 2024-06-29T14:41:14.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.276+0000 7fdc77889640 1 -- 172.21.15.81:0/3506194243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdc700b5af0 con 0x7fdc70074520 2024-06-29T14:41:14.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.276+0000 7fdc61ffb640 1 -- 172.21.15.81:0/3506194243 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v125) ==== 93+0+10 (secure 0 0 0) 0x7fdc6c061820 con 0x7fdc70074520 2024-06-29T14:41:14.274 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 -- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fdc54076420 msgr2=0x7fdc54078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fdc54076420 0x7fdc54078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009fc0 tx=0x7fdc64009550 comp rx=0 tx=0).stop 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 -- 172.21.15.81:0/3506194243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70074520 msgr2=0x7fdc7015ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70074520 0x7fdc7015ca50 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00c9a0 tx=0x7fdc6c00ce60 comp rx=0 tx=0).stop 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 -- 172.21.15.81:0/3506194243 shutdown_connections 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fdc54076420 0x7fdc54078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc700ae840 0x7fdc7015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70074520 0x7fdc7015ca50 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 --2- 172.21.15.81:0/3506194243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc7006f850 0x7fdc7014e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 -- 172.21.15.81:0/3506194243 >> 172.21.15.81:0/3506194243 conn(0x7fdc70072950 msgr2=0x7fdc700b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 -- 172.21.15.81:0/3506194243 shutdown_connections 2024-06-29T14:41:14.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.278+0000 7fdc77889640 1 -- 172.21.15.81:0/3506194243 wait complete. 2024-06-29T14:41:14.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-06-29T14:41:14.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.402+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/744258778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3200 msgr2=0x7fd1e00cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.402+0000 7fd1e5fbd640 1 --2- 172.21.15.81:0/744258778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3200 0x7fd1e00cc7b0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0005fd0 tx=0x7fd1d002a530 comp rx=0 tx=0).stop 2024-06-29T14:41:14.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.402+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/744258778 shutdown_connections 2024-06-29T14:41:14.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.402+0000 7fd1e5fbd640 1 --2- 172.21.15.81:0/744258778 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e00cccf0 0x7fd1e00d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.402+0000 7fd1e5fbd640 1 --2- 172.21.15.81:0/744258778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3200 0x7fd1e00cc7b0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.402+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/744258778 >> 172.21.15.81:0/744258778 conn(0x7fd1e00b3680 msgr2=0x7fd1e00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:14.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.402+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/744258778 shutdown_connections 2024-06-29T14:41:14.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.402+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/744258778 wait complete. 2024-06-29T14:41:14.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.404+0000 7fd1e5fbd640 1 Processor -- start 2024-06-29T14:41:14.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.404+0000 7fd1e5fbd640 1 -- start start 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.405+0000 7fd1e5fbd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3200 0x7fd1e01e49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.405+0000 7fd1e5fbd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e00cccf0 0x7fd1e0155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.405+0000 7fd1e5fbd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e00d0770 0x7fd1e00cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.405+0000 7fd1e5fbd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00cedb0 con 0x7fd1e00b3200 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.405+0000 7fd1e5fbd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00cec30 con 0x7fd1e00cccf0 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.405+0000 7fd1e5fbd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e00cef30 con 0x7fd1e00d0770 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1df7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3200 0x7fd1e01e49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1df7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3200 0x7fd1e01e49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60516/0 (socket says 172.21.15.81:60516) 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1df7fe640 1 -- 172.21.15.81:0/362274474 learned_addr learned my addr 172.21.15.81:0/362274474 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1df7fe640 1 -- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e00d0770 msgr2=0x7fd1e00cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1df7fe640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e00d0770 0x7fd1e00cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1df7fe640 1 -- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e00cccf0 msgr2=0x7fd1e0155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:14.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1df7fe640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e00cccf0 0x7fd1e0155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:14.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1df7fe640 1 -- 172.21.15.81:0/362274474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e01e5380 con 0x7fd1e00b3200 2024-06-29T14:41:14.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1df7fe640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3200 0x7fd1e01e49a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0006010 tx=0x7fd1d00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:14.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1c7fff640 1 -- 172.21.15.81:0/362274474 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d003e030 con 0x7fd1e00b3200 2024-06-29T14:41:14.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1c7fff640 1 -- 172.21.15.81:0/362274474 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1d00047a0 con 0x7fd1e00b3200 2024-06-29T14:41:14.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/362274474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e01e5660 con 0x7fd1e00b3200 2024-06-29T14:41:14.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1c7fff640 1 -- 172.21.15.81:0/362274474 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0035040 con 0x7fd1e00b3200 2024-06-29T14:41:14.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.406+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/362274474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e01e5c20 con 0x7fd1e00b3200 2024-06-29T14:41:14.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.407+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/362274474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e00cc7b0 con 0x7fd1e00b3200 2024-06-29T14:41:14.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.408+0000 7fd1c7fff640 1 -- 172.21.15.81:0/362274474 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fd1d0041030 con 0x7fd1e00b3200 2024-06-29T14:41:14.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.409+0000 7fd1c7fff640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd1cc076420 0x7fd1cc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:14.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.409+0000 7fd1c7fff640 1 -- 172.21.15.81:0/362274474 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5817+0+0 (secure 0 0 0) 0x7fd1d00b82c0 con 0x7fd1e00b3200 2024-06-29T14:41:14.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.409+0000 7fd1deffd640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd1cc076420 0x7fd1cc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:14.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.410+0000 7fd1deffd640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd1cc076420 0x7fd1cc078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd1e009efe0 tx=0x7fd1d4006bf0 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:14.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.413+0000 7fd1c7fff640 1 -- 172.21.15.81:0/362274474 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd1e00d0b80 con 0x7fd1e00b3200 2024-06-29T14:41:14.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:14.667+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/362274474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) -- 0x7fd1e00d2270 con 0x7fd1e00b3200 2024-06-29T14:41:15.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.652+0000 7fd1c7fff640 1 -- 172.21.15.81:0/362274474 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v703) ==== 163+0+0 (secure 0 0 0) 0x7fd1d0081900 con 0x7fd1e00b3200 2024-06-29T14:41:15.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd1cc076420 msgr2=0x7fd1cc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd1cc076420 0x7fd1cc078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd1e009efe0 tx=0x7fd1d4006bf0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/362274474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3200 msgr2=0x7fd1e01e49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3200 0x7fd1e01e49a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0006010 tx=0x7fd1d00039e0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/362274474 shutdown_connections 2024-06-29T14:41:15.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd1cc076420 0x7fd1cc078930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e00d0770 0x7fd1e00cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e00cccf0 0x7fd1e0155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 --2- 172.21.15.81:0/362274474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3200 0x7fd1e01e49a0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/362274474 >> 172.21.15.81:0/362274474 conn(0x7fd1e00b3680 msgr2=0x7fd1e00c7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:15.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/362274474 shutdown_connections 2024-06-29T14:41:15.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.655+0000 7fd1e5fbd640 1 -- 172.21.15.81:0/362274474 wait complete. 2024-06-29T14:41:15.666 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:41:15.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.788+0000 7f79530e4640 1 -- 172.21.15.81:0/1047751180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f793c004800 msgr2=0x7f793c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.788+0000 7f79530e4640 1 --2- 172.21.15.81:0/1047751180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f793c004800 0x7f793c004c10 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7938006fd0 tx=0x7f793c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.788+0000 7f79530e4640 1 -- 172.21.15.81:0/1047751180 shutdown_connections 2024-06-29T14:41:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.788+0000 7f79530e4640 1 --2- 172.21.15.81:0/1047751180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f793c004800 0x7f793c004c10 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.788+0000 7f79530e4640 1 -- 172.21.15.81:0/1047751180 >> 172.21.15.81:0/1047751180 conn(0x7f794c0d3c60 msgr2=0x7f794c0d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.788+0000 7f79530e4640 1 -- 172.21.15.81:0/1047751180 shutdown_connections 2024-06-29T14:41:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.788+0000 7f79530e4640 1 -- 172.21.15.81:0/1047751180 wait complete. 2024-06-29T14:41:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.790+0000 7f79530e4640 1 Processor -- start 2024-06-29T14:41:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.790+0000 7f79530e4640 1 -- start start 2024-06-29T14:41:15.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.791+0000 7f79530e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f793c004800 0x7f794c1e2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.791+0000 7f79530e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f794c0a1710 0x7f794c0b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.791+0000 7f79530e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f794c0cad10 0x7f794c1e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.791+0000 7f79530e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f794c0d4b10 con 0x7f793c004800 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.791+0000 7f79530e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f794c0d4990 con 0x7f794c0cad10 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.791+0000 7f79530e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f794c0d2240 con 0x7f794c0a1710 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.791+0000 7f7950e80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f793c004800 0x7f794c1e2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.791+0000 7f7950e80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f793c004800 0x7f794c1e2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60542/0 (socket says 172.21.15.81:60542) 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.791+0000 7f7950e80640 1 -- 172.21.15.81:0/404845628 learned_addr learned my addr 172.21.15.81:0/404845628 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.791+0000 7f7951681640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f794c0cad10 0x7f794c1e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f7950e80640 1 -- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f794c0a1710 msgr2=0x7f794c0b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f7950e80640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f794c0a1710 0x7f794c0b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f7950e80640 1 -- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f794c0cad10 msgr2=0x7f794c1e79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f7950e80640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f794c0cad10 0x7f794c1e79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f7950e80640 1 -- 172.21.15.81:0/404845628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c0cb520 con 0x7f793c004800 2024-06-29T14:41:15.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f7950e80640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f793c004800 0x7f794c1e2130 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f7940011b10 tx=0x7f7940009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:15.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f79497fa640 1 -- 172.21.15.81:0/404845628 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794001e070 con 0x7f793c004800 2024-06-29T14:41:15.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f79497fa640 1 -- 172.21.15.81:0/404845628 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f794000fc40 con 0x7f793c004800 2024-06-29T14:41:15.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f79530e4640 1 -- 172.21.15.81:0/404845628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794c0cbc70 con 0x7f793c004800 2024-06-29T14:41:15.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f79497fa640 1 -- 172.21.15.81:0/404845628 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940019410 con 0x7f793c004800 2024-06-29T14:41:15.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.792+0000 7f79530e4640 1 -- 172.21.15.81:0/404845628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f794c0b1d70 con 0x7f793c004800 2024-06-29T14:41:15.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.793+0000 7f79530e4640 1 -- 172.21.15.81:0/404845628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f794c0b35c0 con 0x7f793c004800 2024-06-29T14:41:15.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.794+0000 7f79497fa640 1 -- 172.21.15.81:0/404845628 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f7940021070 con 0x7f793c004800 2024-06-29T14:41:15.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.794+0000 7f79497fa640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f7930076420 0x7f7930078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:15.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.794+0000 7f79497fa640 1 -- 172.21.15.81:0/404845628 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5817+0+0 (secure 0 0 0) 0x7f7940098260 con 0x7f793c004800 2024-06-29T14:41:15.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.794+0000 7f794bfff640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f7930076420 0x7f7930078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:15.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.795+0000 7f794bfff640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f7930076420 0x7f7930078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7938033000 tx=0x7f794c0b51c0 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:15.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.798+0000 7f79497fa640 1 -- 172.21.15.81:0/404845628 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7940061250 con 0x7f793c004800 2024-06-29T14:41:15.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.976+0000 7f79530e4640 1 -- 172.21.15.81:0/404845628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f794c0b8a30 con 0x7f793c004800 2024-06-29T14:41:15.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.977+0000 7f79497fa640 1 -- 172.21.15.81:0/404845628 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v51) ==== 59+0+1998 (secure 0 0 0) 0x7f7940065550 con 0x7f793c004800 2024-06-29T14:41:15.974 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:41:15.974 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:41:15.974 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T14:41:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:41:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T14:41:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T14:41:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:41:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:41:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T14:41:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:41:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:41:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:41:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T14:41:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T14:41:15.977 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T14:41:15.977 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:41:15.977 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:41:15.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 -- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f7930076420 msgr2=0x7f7930078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f7930076420 0x7f7930078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7938033000 tx=0x7f794c0b51c0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 -- 172.21.15.81:0/404845628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f793c004800 msgr2=0x7f794c1e2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:15.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f793c004800 0x7f794c1e2130 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f7940011b10 tx=0x7f7940009ed0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 -- 172.21.15.81:0/404845628 shutdown_connections 2024-06-29T14:41:15.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f7930076420 0x7f7930078930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f794c0cad10 0x7f794c1e79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f794c0a1710 0x7f794c0b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 --2- 172.21.15.81:0/404845628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f793c004800 0x7f794c1e2130 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:15.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 -- 172.21.15.81:0/404845628 >> 172.21.15.81:0/404845628 conn(0x7f794c0d3c60 msgr2=0x7f794c1e6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:15.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 -- 172.21.15.81:0/404845628 shutdown_connections 2024-06-29T14:41:15.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:15.979+0000 7f79530e4640 1 -- 172.21.15.81:0/404845628 wait complete. 2024-06-29T14:41:15.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:41:16.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.105+0000 7ff5db8e4640 1 -- 172.21.15.81:0/1137999324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5d415b5b0 msgr2=0x7ff5d40b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.105+0000 7ff5db8e4640 1 --2- 172.21.15.81:0/1137999324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5d415b5b0 0x7ff5d40b4430 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4005fd0 tx=0x7ff5c402a540 comp rx=0 tx=0).stop 2024-06-29T14:41:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.105+0000 7ff5db8e4640 1 -- 172.21.15.81:0/1137999324 shutdown_connections 2024-06-29T14:41:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.105+0000 7ff5db8e4640 1 --2- 172.21.15.81:0/1137999324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5d406f660 0x7ff5d406fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.105+0000 7ff5db8e4640 1 --2- 172.21.15.81:0/1137999324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5d415b5b0 0x7ff5d40b4430 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.105+0000 7ff5db8e4640 1 -- 172.21.15.81:0/1137999324 >> 172.21.15.81:0/1137999324 conn(0x7ff5d406ccc0 msgr2=0x7ff5d40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.105+0000 7ff5db8e4640 1 -- 172.21.15.81:0/1137999324 shutdown_connections 2024-06-29T14:41:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.105+0000 7ff5db8e4640 1 -- 172.21.15.81:0/1137999324 wait complete. 2024-06-29T14:41:16.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.107+0000 7ff5db8e4640 1 Processor -- start 2024-06-29T14:41:16.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.107+0000 7ff5db8e4640 1 -- start start 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5db8e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5d406f660 0x7ff5d41492e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5db8e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5d40b35b0 0x7ff5d4155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5db8e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5d415b5b0 0x7ff5d414dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5db8e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d406f210 con 0x7ff5d415b5b0 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5db8e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d406f090 con 0x7ff5d406f660 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5db8e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d406f390 con 0x7ff5d40b35b0 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9e81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5d415b5b0 0x7ff5d414dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9e81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5d415b5b0 0x7ff5d414dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60560/0 (socket says 172.21.15.81:60560) 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9e81640 1 -- 172.21.15.81:0/3440127170 learned_addr learned my addr 172.21.15.81:0/3440127170 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d8e7f640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5d40b35b0 0x7ff5d4155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9680640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5d406f660 0x7ff5d41492e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9e81640 1 -- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5d40b35b0 msgr2=0x7ff5d4155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9e81640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5d40b35b0 0x7ff5d4155b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9e81640 1 -- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5d406f660 msgr2=0x7ff5d41492e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9e81640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5d406f660 0x7ff5d41492e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9e81640 1 -- 172.21.15.81:0/3440127170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d41f3cd0 con 0x7ff5d415b5b0 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d8e7f640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5d40b35b0 0x7ff5d4155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9680640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5d406f660 0x7ff5d41492e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.108+0000 7ff5d9e81640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5d415b5b0 0x7ff5d414dde0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00ddf0 tx=0x7ff5cc00b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.109+0000 7ff5c9ffb640 1 -- 172.21.15.81:0/3440127170 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc004d20 con 0x7ff5d415b5b0 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.109+0000 7ff5c9ffb640 1 -- 172.21.15.81:0/3440127170 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5cc00be40 con 0x7ff5d415b5b0 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.109+0000 7ff5db8e4640 1 -- 172.21.15.81:0/3440127170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d4151c00 con 0x7ff5d415b5b0 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.109+0000 7ff5c9ffb640 1 -- 172.21.15.81:0/3440127170 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc00e8a0 con 0x7ff5d415b5b0 2024-06-29T14:41:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.109+0000 7ff5db8e4640 1 -- 172.21.15.81:0/3440127170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d41521c0 con 0x7ff5d415b5b0 2024-06-29T14:41:16.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.110+0000 7ff5db8e4640 1 -- 172.21.15.81:0/3440127170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d41563f0 con 0x7ff5d415b5b0 2024-06-29T14:41:16.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.111+0000 7ff5c9ffb640 1 -- 172.21.15.81:0/3440127170 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7ff5cc00ea40 con 0x7ff5d415b5b0 2024-06-29T14:41:16.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.111+0000 7ff5c9ffb640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff5b80763f0 0x7ff5b8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.111+0000 7ff5c9ffb640 1 -- 172.21.15.81:0/3440127170 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5817+0+0 (secure 0 0 0) 0x7ff5cc098550 con 0x7ff5d415b5b0 2024-06-29T14:41:16.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.112+0000 7ff5d9680640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff5b80763f0 0x7ff5b8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.112+0000 7ff5d9680640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff5b80763f0 0x7ff5b8078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0000f60 tx=0x7ff5d0009040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:16.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.115+0000 7ff5c9ffb640 1 -- 172.21.15.81:0/3440127170 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff5cc061d30 con 0x7ff5d415b5b0 2024-06-29T14:41:16.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.294+0000 7ff5db8e4640 1 -- 172.21.15.81:0/3440127170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff5d406fa70 con 0x7ff5d415b5b0 2024-06-29T14:41:16.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.298+0000 7ff5c9ffb640 1 -- 172.21.15.81:0/3440127170 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) ==== 122+0+0 (secure 0 0 0) 0x7ff5cc061630 con 0x7ff5d415b5b0 2024-06-29T14:41:16.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 -- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff5b80763f0 msgr2=0x7ff5b8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff5b80763f0 0x7ff5b8078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0000f60 tx=0x7ff5d0009040 comp rx=0 tx=0).stop 2024-06-29T14:41:16.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 -- 172.21.15.81:0/3440127170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5d415b5b0 msgr2=0x7ff5d414dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5d415b5b0 0x7ff5d414dde0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00ddf0 tx=0x7ff5cc00b730 comp rx=0 tx=0).stop 2024-06-29T14:41:16.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 -- 172.21.15.81:0/3440127170 shutdown_connections 2024-06-29T14:41:16.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff5b80763f0 0x7ff5b8078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5d415b5b0 0x7ff5d414dde0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5d40b35b0 0x7ff5d4155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 --2- 172.21.15.81:0/3440127170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5d406f660 0x7ff5d41492e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 -- 172.21.15.81:0/3440127170 >> 172.21.15.81:0/3440127170 conn(0x7ff5d406ccc0 msgr2=0x7ff5d414b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:16.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 -- 172.21.15.81:0/3440127170 shutdown_connections 2024-06-29T14:41:16.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.300+0000 7ff5db8e4640 1 -- 172.21.15.81:0/3440127170 wait complete. 2024-06-29T14:41:16.309 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:41:16.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:16.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:16.400 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:41:16.400 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:16.400 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:16.417 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:16.417 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:41:16.417 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:16.459 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:16.459 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:16.459 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:16.474 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:16.474 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:41:16.474 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:16.517 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:16.517 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:41:16.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:16.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.633+0000 7f67bc738640 1 -- 172.21.15.81:0/982887557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67b415bc20 msgr2=0x7f67b406ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.633+0000 7f67bc738640 1 --2- 172.21.15.81:0/982887557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67b415bc20 0x7f67b406ed90 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f67ac005fd0 tx=0x7f67ac02a970 comp rx=0 tx=0).stop 2024-06-29T14:41:16.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.633+0000 7f67bc738640 1 -- 172.21.15.81:0/982887557 shutdown_connections 2024-06-29T14:41:16.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.633+0000 7f67bc738640 1 --2- 172.21.15.81:0/982887557 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67b4073650 0x7f67b415b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.633+0000 7f67bc738640 1 --2- 172.21.15.81:0/982887557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67b415bc20 0x7f67b406ed90 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.633+0000 7f67bc738640 1 -- 172.21.15.81:0/982887557 >> 172.21.15.81:0/982887557 conn(0x7f67b40b2920 msgr2=0x7f67b40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:16.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.634+0000 7f67bc738640 1 -- 172.21.15.81:0/982887557 shutdown_connections 2024-06-29T14:41:16.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.634+0000 7f67bc738640 1 -- 172.21.15.81:0/982887557 wait complete. 2024-06-29T14:41:16.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.635+0000 7f67bc738640 1 Processor -- start 2024-06-29T14:41:16.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.635+0000 7f67bc738640 1 -- start start 2024-06-29T14:41:16.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.636+0000 7f67bc738640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67b40cd150 0x7f67b41493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67bc738640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67b415bc20 0x7f67b40cf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67bc738640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67b40cfe70 0x7f67b40b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67bc738640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f67b406f660 con 0x7f67b415bc20 2024-06-29T14:41:16.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67bc738640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f67b406f4e0 con 0x7f67b40cfe70 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67bc738640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f67b406f7e0 con 0x7f67b40cd150 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67b9cd3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67b415bc20 0x7f67b40cf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67b9cd3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67b415bc20 0x7f67b40cf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60582/0 (socket says 172.21.15.81:60582) 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67b9cd3640 1 -- 172.21.15.81:0/3505597118 learned_addr learned my addr 172.21.15.81:0/3505597118 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67bacd5640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67b40cfe70 0x7f67b40b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67ba4d4640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67b40cd150 0x7f67b41493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67b9cd3640 1 -- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67b40cd150 msgr2=0x7f67b41493c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67b9cd3640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67b40cd150 0x7f67b41493c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67b9cd3640 1 -- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67b40cfe70 msgr2=0x7f67b40b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67b9cd3640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67b40cfe70 0x7f67b40b0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67b9cd3640 1 -- 172.21.15.81:0/3505597118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b4149900 con 0x7f67b415bc20 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67b9cd3640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67b415bc20 0x7f67b40cf930 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f67a400b410 tx=0x7f67a400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67aaffd640 1 -- 172.21.15.81:0/3505597118 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4015020 con 0x7f67b415bc20 2024-06-29T14:41:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67aaffd640 1 -- 172.21.15.81:0/3505597118 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67a4011920 con 0x7f67b415bc20 2024-06-29T14:41:16.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67bc738640 1 -- 172.21.15.81:0/3505597118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b40d0580 con 0x7f67b415bc20 2024-06-29T14:41:16.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67aaffd640 1 -- 172.21.15.81:0/3505597118 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4011d70 con 0x7f67b415bc20 2024-06-29T14:41:16.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.637+0000 7f67bc738640 1 -- 172.21.15.81:0/3505597118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b40d0b40 con 0x7f67b415bc20 2024-06-29T14:41:16.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.638+0000 7f67bc738640 1 -- 172.21.15.81:0/3505597118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b40d02f0 con 0x7f67b415bc20 2024-06-29T14:41:16.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.639+0000 7f67aaffd640 1 -- 172.21.15.81:0/3505597118 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f67a4018020 con 0x7f67b415bc20 2024-06-29T14:41:16.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.639+0000 7f67aaffd640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6798076420 0x7f6798078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:16.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.640+0000 7f67aaffd640 1 -- 172.21.15.81:0/3505597118 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5817+0+0 (secure 0 0 0) 0x7f67a40978f0 con 0x7f67b415bc20 2024-06-29T14:41:16.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.640+0000 7f67ba4d4640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6798076420 0x7f6798078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:16.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.640+0000 7f67ba4d4640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6798076420 0x7f6798078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f67b0006fd0 tx=0x7f67b0008040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.644+0000 7f67aaffd640 1 -- 172.21.15.81:0/3505597118 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f67a401f020 con 0x7f67b415bc20 2024-06-29T14:41:16.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.860+0000 7f67bc738640 1 -- 172.21.15.81:0/3505597118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67b406fd10 con 0x7f67b415bc20 2024-06-29T14:41:16.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.861+0000 7f67aaffd640 1 -- 172.21.15.81:0/3505597118 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+3917 (secure 0 0 0) 0x7f67a4061050 con 0x7f67b415bc20 2024-06-29T14:41:16.858 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:16.858 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":61,"btime":"2024-06-29T14:41:10:504421+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6680,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1563799167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1563799167},{"type":"v1","addr":"172.21.15.167:6837","nonce":1563799167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7354,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3062579579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3062579579},{"type":"v1","addr":"172.21.15.81:6835","nonce":3062579579}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":7363,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3427077146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3427077146},{"type":"v1","addr":"172.21.15.167:6836","nonce":3427077146}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55},{"gid":8466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/2502752863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2502752863},{"type":"v1","addr":"172.21.15.81:6833","nonce":2502752863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[]} 2024-06-29T14:41:16.859 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 61 2024-06-29T14:41:16.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 -- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6798076420 msgr2=0x7f6798078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6798076420 0x7f6798078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f67b0006fd0 tx=0x7f67b0008040 comp rx=0 tx=0).stop 2024-06-29T14:41:16.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 -- 172.21.15.81:0/3505597118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67b415bc20 msgr2=0x7f67b40cf930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:16.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67b415bc20 0x7f67b40cf930 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f67a400b410 tx=0x7f67a400b8d0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 -- 172.21.15.81:0/3505597118 shutdown_connections 2024-06-29T14:41:16.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6798076420 0x7f6798078930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67b40cfe70 0x7f67b40b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67b415bc20 0x7f67b40cf930 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 --2- 172.21.15.81:0/3505597118 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67b40cd150 0x7f67b41493c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:16.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 -- 172.21.15.81:0/3505597118 >> 172.21.15.81:0/3505597118 conn(0x7f67b40b2920 msgr2=0x7f67b4152720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:16.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 -- 172.21.15.81:0/3505597118 shutdown_connections 2024-06-29T14:41:16.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:16.863+0000 7f67bc738640 1 -- 172.21.15.81:0/3505597118 wait complete. 2024-06-29T14:41:16.872 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:41:16.872 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:41:16.872 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:41:16.872 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:16.872 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:41:16.872 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:41:16.873 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:41:16.873 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:16.873 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:41:16.873 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:41:16.873 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:41:16.873 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:16.874 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:41:16.874 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:41:16.874 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:41:16.874 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:16.882 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:41:16.882 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:41:16.884 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:41:16.884 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:41:16.889 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:41:16.889 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:41:16.890 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:41:16.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:41:16.926 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:41:16.927 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:16.928 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:41:16.928 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:16.932 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:41:16.932 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:16.933 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:41:16.933 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:16.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:41:17.012 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T14:41:17.013 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:41:17.030 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:41:17.030 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T14:41:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.057+0000 7f621e681640 1 -- 172.21.15.81:0/628176010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6200003a00 msgr2=0x7f6200003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.057+0000 7f621e681640 1 --2- 172.21.15.81:0/628176010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6200003a00 0x7f6200003e70 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f620800bb20 tx=0x7f6200005830 comp rx=0 tx=0).stop 2024-06-29T14:41:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.057+0000 7f621cc1e640 1 -- 172.21.15.81:0/628176010 reap_dead start 2024-06-29T14:41:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.057+0000 7f621e681640 1 -- 172.21.15.81:0/628176010 shutdown_connections 2024-06-29T14:41:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.057+0000 7f621e681640 1 -- 172.21.15.81:0/628176010 >> 172.21.15.81:0/628176010 conn(0x7f621809cf30 msgr2=0x7f62180d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.057+0000 7f621e681640 1 -- 172.21.15.81:0/628176010 shutdown_connections 2024-06-29T14:41:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.057+0000 7f621e681640 1 -- 172.21.15.81:0/628176010 wait complete. 2024-06-29T14:41:17.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.059+0000 7f621e681640 1 Processor -- start 2024-06-29T14:41:17.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.059+0000 7f621e681640 1 -- start start 2024-06-29T14:41:17.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.060+0000 7f621e681640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6200003a00 0x7f621814eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.060+0000 7f621e681640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6218150af0 0x7f62180a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.060+0000 7f621e681640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62180a0f40 0x7f62180b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.060+0000 7f621e681640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62180d0d50 con 0x7f62180a0f40 2024-06-29T14:41:17.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.060+0000 7f621e681640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f62180d0bd0 con 0x7f6218150af0 2024-06-29T14:41:17.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.060+0000 7f621e681640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f62180b79a0 con 0x7f6200003a00 2024-06-29T14:41:17.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.060+0000 7f621cc1e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62180a0f40 0x7f62180b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.060+0000 7f621cc1e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62180a0f40 0x7f62180b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60630/0 (socket says 172.21.15.81:60630) 2024-06-29T14:41:17.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.060+0000 7f621cc1e640 1 -- 172.21.15.81:0/1831300847 learned_addr learned my addr 172.21.15.81:0/1831300847 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:17.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f6217fff640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6200003a00 0x7f621814eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f62177fe640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6218150af0 0x7f62180a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f621cc1e640 1 -- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6200003a00 msgr2=0x7f621814eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f621cc1e640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6200003a00 0x7f621814eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f621cc1e640 1 -- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6218150af0 msgr2=0x7f62180a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f621cc1e640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6218150af0 0x7f62180a0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f621cc1e640 1 -- 172.21.15.81:0/1831300847 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62180a1350 con 0x7f62180a0f40 2024-06-29T14:41:17.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f621cc1e640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62180a0f40 0x7f62180b0e70 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f62180d4460 tx=0x7f6210009a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:17.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f6214ff9640 1 -- 172.21.15.81:0/1831300847 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6210004b20 con 0x7f62180a0f40 2024-06-29T14:41:17.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f6214ff9640 1 -- 172.21.15.81:0/1831300847 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6210004db0 con 0x7f62180a0f40 2024-06-29T14:41:17.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f6214ff9640 1 -- 172.21.15.81:0/1831300847 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6210011970 con 0x7f62180a0f40 2024-06-29T14:41:17.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f621e681640 1 -- 172.21.15.81:0/1831300847 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6218151210 con 0x7f62180a0f40 2024-06-29T14:41:17.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.061+0000 7f621e681640 1 -- 172.21.15.81:0/1831300847 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62181517d0 con 0x7f62180a0f40 2024-06-29T14:41:17.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.062+0000 7f621e681640 1 -- 172.21.15.81:0/1831300847 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62180d0480 con 0x7f62180a0f40 2024-06-29T14:41:17.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.063+0000 7f6214ff9640 1 -- 172.21.15.81:0/1831300847 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f621001b020 con 0x7f62180a0f40 2024-06-29T14:41:17.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.063+0000 7f6214ff9640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6204076420 0x7f6204078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.063+0000 7f6214ff9640 1 -- 172.21.15.81:0/1831300847 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5817+0+0 (secure 0 0 0) 0x7f6210097bd0 con 0x7f62180a0f40 2024-06-29T14:41:17.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.064+0000 7f6217fff640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6204076420 0x7f6204078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.064+0000 7f6217fff640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6204076420 0x7f6204078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6208036000 tx=0x7f62080046c0 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:17.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.068+0000 7f6214ff9640 1 -- 172.21.15.81:0/1831300847 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f62180a1350 con 0x7f62180a0f40 2024-06-29T14:41:17.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.260+0000 7f621e681640 1 -- 172.21.15.81:0/1831300847 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f6218150f00 con 0x7f62180a0f40 2024-06-29T14:41:17.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.664+0000 7f6214ff9640 1 -- 172.21.15.81:0/1831300847 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v127) ==== 97+0+0 (secure 0 0 0) 0x7f62100612a0 con 0x7f62180a0f40 2024-06-29T14:41:17.662 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T14:41:17.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 -- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6204076420 msgr2=0x7f6204078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6204076420 0x7f6204078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6208036000 tx=0x7f62080046c0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 -- 172.21.15.81:0/1831300847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62180a0f40 msgr2=0x7f62180b0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62180a0f40 0x7f62180b0e70 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f62180d4460 tx=0x7f6210009a80 comp rx=0 tx=0).stop 2024-06-29T14:41:17.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 -- 172.21.15.81:0/1831300847 shutdown_connections 2024-06-29T14:41:17.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6204076420 0x7f6204078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62180a0f40 0x7f62180b0e70 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6218150af0 0x7f62180a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 --2- 172.21.15.81:0/1831300847 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6200003a00 0x7f621814eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 -- 172.21.15.81:0/1831300847 >> 172.21.15.81:0/1831300847 conn(0x7f621809cf30 msgr2=0x7f621815f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:17.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 -- 172.21.15.81:0/1831300847 shutdown_connections 2024-06-29T14:41:17.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.667+0000 7f621e681640 1 -- 172.21.15.81:0/1831300847 wait complete. 2024-06-29T14:41:17.676 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:41:17.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.793+0000 7f6cd963c640 1 -- 172.21.15.81:0/1250015687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6cbc003a00 msgr2=0x7f6cbc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.793+0000 7f6cd963c640 1 --2- 172.21.15.81:0/1250015687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6cbc003a00 0x7f6cbc003e70 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400c780 tx=0x7f6cbc005830 comp rx=0 tx=0).stop 2024-06-29T14:41:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.793+0000 7f6cd37fe640 1 -- 172.21.15.81:0/1250015687 reap_dead start 2024-06-29T14:41:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.793+0000 7f6cd963c640 1 -- 172.21.15.81:0/1250015687 shutdown_connections 2024-06-29T14:41:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.793+0000 7f6cd963c640 1 -- 172.21.15.81:0/1250015687 >> 172.21.15.81:0/1250015687 conn(0x7f6cd40b3680 msgr2=0x7f6cd40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.793+0000 7f6cd963c640 1 -- 172.21.15.81:0/1250015687 shutdown_connections 2024-06-29T14:41:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.793+0000 7f6cd963c640 1 -- 172.21.15.81:0/1250015687 wait complete. 2024-06-29T14:41:17.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.795+0000 7f6cd963c640 1 Processor -- start 2024-06-29T14:41:17.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.795+0000 7f6cd963c640 1 -- start start 2024-06-29T14:41:17.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd963c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6cbc003a00 0x7f6cd40cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd963c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6cd41e10a0 0x7f6cd41e73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd963c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6cd41e14b0 0x7f6cd41e7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd963c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd40d09d0 con 0x7f6cd41e10a0 2024-06-29T14:41:17.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd963c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd40d0850 con 0x7f6cd41e14b0 2024-06-29T14:41:17.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd963c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd40b5890 con 0x7f6cbc003a00 2024-06-29T14:41:17.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd27fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6cd41e10a0 0x7f6cd41e73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd27fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6cd41e10a0 0x7f6cd41e73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60652/0 (socket says 172.21.15.81:60652) 2024-06-29T14:41:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd27fc640 1 -- 172.21.15.81:0/2365692948 learned_addr learned my addr 172.21.15.81:0/2365692948 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd37fe640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6cd41e14b0 0x7f6cd41e7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd27fc640 1 -- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6cbc003a00 msgr2=0x7f6cd40cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd2ffd640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6cbc003a00 0x7f6cd40cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd27fc640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6cbc003a00 0x7f6cd40cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd27fc640 1 -- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6cd41e14b0 msgr2=0x7f6cd41e7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd27fc640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6cd41e14b0 0x7f6cd41e7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.796+0000 7f6cd27fc640 1 -- 172.21.15.81:0/2365692948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd40c9bb0 con 0x7f6cd41e10a0 2024-06-29T14:41:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.797+0000 7f6cd27fc640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6cd41e10a0 0x7f6cd41e73f0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8004650 tx=0x7f6cc800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:17.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.797+0000 7f6cbb7fe640 1 -- 172.21.15.81:0/2365692948 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8010850 con 0x7f6cd41e10a0 2024-06-29T14:41:17.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.797+0000 7f6cbb7fe640 1 -- 172.21.15.81:0/2365692948 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6cc80109f0 con 0x7f6cd41e10a0 2024-06-29T14:41:17.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.797+0000 7f6cd963c640 1 -- 172.21.15.81:0/2365692948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd40c9e90 con 0x7f6cd41e10a0 2024-06-29T14:41:17.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.797+0000 7f6cbb7fe640 1 -- 172.21.15.81:0/2365692948 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8010e20 con 0x7f6cd41e10a0 2024-06-29T14:41:17.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.797+0000 7f6cd963c640 1 -- 172.21.15.81:0/2365692948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd40ca450 con 0x7f6cd41e10a0 2024-06-29T14:41:17.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.798+0000 7f6cd963c640 1 -- 172.21.15.81:0/2365692948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd40b5350 con 0x7f6cd41e10a0 2024-06-29T14:41:17.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.798+0000 7f6cbb7fe640 1 -- 172.21.15.81:0/2365692948 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f6cc8023440 con 0x7f6cd41e10a0 2024-06-29T14:41:17.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.799+0000 7f6cbb7fe640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6cc00763f0 0x7f6cc0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:17.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.799+0000 7f6cbb7fe640 1 -- 172.21.15.81:0/2365692948 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 4957+0+0 (secure 0 0 0) 0x7f6cc80983d0 con 0x7f6cd41e10a0 2024-06-29T14:41:17.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.799+0000 7f6cd2ffd640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6cc00763f0 0x7f6cc0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:17.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.800+0000 7f6cd2ffd640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6cc00763f0 0x7f6cc0078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4033000 tx=0x7f6cc402d070 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:17.803+0000 7f6cbb7fe640 1 -- 172.21.15.81:0/2365692948 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6cc8061e90 con 0x7f6cd41e10a0 2024-06-29T14:41:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.056+0000 7f6cd963c640 1 -- 172.21.15.81:0/2365692948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f6cd41e4fd0 con 0x7f6cd41e10a0 2024-06-29T14:41:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.056+0000 7f6cbb7fe640 1 -- 172.21.15.81:0/2365692948 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) ==== 80+0+5304 (secure 0 0 0) 0x7f6cc8061790 con 0x7f6cd41e10a0 2024-06-29T14:41:18.054 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:18.054 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:41:18.054 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T14:41:18.054 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.054 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T14:41:18.055 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T14:41:18.055 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.055 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:18.055 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:41:18.055 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:41:18.055 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.055 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.056 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T14:41:18.057 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:18.058 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.059 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.060 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:18.061 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.062 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:18.063 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:41:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.065 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.065 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.065 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T14:41:18.065 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T14:41:18.065 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.065 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:18.065 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:41:18.065 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:41:18.066 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T14:41:18.067 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:41:18.068 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:18.069 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T14:41:18.070 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:41:18.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.058+0000 7f6cd963c640 1 -- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6cc00763f0 msgr2=0x7f6cc0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.058+0000 7f6cd963c640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6cc00763f0 0x7f6cc0078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4033000 tx=0x7f6cc402d070 comp rx=0 tx=0).stop 2024-06-29T14:41:18.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.058+0000 7f6cd963c640 1 -- 172.21.15.81:0/2365692948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6cd41e10a0 msgr2=0x7f6cd41e73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.058+0000 7f6cd963c640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6cd41e10a0 0x7f6cd41e73f0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8004650 tx=0x7f6cc800ba40 comp rx=0 tx=0).stop 2024-06-29T14:41:18.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.059+0000 7f6cd963c640 1 -- 172.21.15.81:0/2365692948 shutdown_connections 2024-06-29T14:41:18.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.059+0000 7f6cd963c640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f6cc00763f0 0x7f6cc0078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.059+0000 7f6cd963c640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6cd41e14b0 0x7f6cd41e7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.059+0000 7f6cd963c640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6cd41e10a0 0x7f6cd41e73f0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.059+0000 7f6cd963c640 1 --2- 172.21.15.81:0/2365692948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6cbc003a00 0x7f6cd40cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.059+0000 7f6cd963c640 1 -- 172.21.15.81:0/2365692948 >> 172.21.15.81:0/2365692948 conn(0x7f6cd40b3680 msgr2=0x7f6cd41e3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.059+0000 7f6cd963c640 1 -- 172.21.15.81:0/2365692948 shutdown_connections 2024-06-29T14:41:18.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.059+0000 7f6cd963c640 1 -- 172.21.15.81:0/2365692948 wait complete. 2024-06-29T14:41:18.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:18.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.188+0000 7ff61d711640 1 -- 172.21.15.81:0/1818308427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff600003a00 msgr2=0x7ff600003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.188+0000 7ff61d711640 1 --2- 172.21.15.81:0/1818308427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff600003a00 0x7ff600003e70 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7ff608009ed0 tx=0x7ff600005830 comp rx=0 tx=0).stop 2024-06-29T14:41:18.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.188+0000 7ff6177fe640 1 -- 172.21.15.81:0/1818308427 reap_dead start 2024-06-29T14:41:18.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.188+0000 7ff61d711640 1 -- 172.21.15.81:0/1818308427 shutdown_connections 2024-06-29T14:41:18.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.188+0000 7ff61d711640 1 -- 172.21.15.81:0/1818308427 >> 172.21.15.81:0/1818308427 conn(0x7ff6180d0930 msgr2=0x7ff6180d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.188+0000 7ff61d711640 1 -- 172.21.15.81:0/1818308427 shutdown_connections 2024-06-29T14:41:18.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.188+0000 7ff61d711640 1 -- 172.21.15.81:0/1818308427 wait complete. 2024-06-29T14:41:18.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.190+0000 7ff61d711640 1 Processor -- start 2024-06-29T14:41:18.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.190+0000 7ff61d711640 1 -- start start 2024-06-29T14:41:18.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.191+0000 7ff61d711640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff600003a00 0x7ff61814ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.191+0000 7ff61d711640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff61806cac0 0x7ff6180aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff61d711640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6180c6b10 0x7ff6180c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff61d711640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff61815aed0 con 0x7ff61806cac0 2024-06-29T14:41:18.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff61d711640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff6180b4bb0 con 0x7ff6180c6b10 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff61d711640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff61815b050 con 0x7ff600003a00 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6167fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff61806cac0 0x7ff6180aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6167fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff61806cac0 0x7ff6180aebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60658/0 (socket says 172.21.15.81:60658) 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6167fc640 1 -- 172.21.15.81:0/2694672240 learned_addr learned my addr 172.21.15.81:0/2694672240 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff616ffd640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff600003a00 0x7ff61814ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6177fe640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6180c6b10 0x7ff6180c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6167fc640 1 -- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff600003a00 msgr2=0x7ff61814ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6167fc640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff600003a00 0x7ff61814ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6167fc640 1 -- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6180c6b10 msgr2=0x7ff6180c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6167fc640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6180c6b10 0x7ff6180c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6167fc640 1 -- 172.21.15.81:0/2694672240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6180c5a70 con 0x7ff61806cac0 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6177fe640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6180c6b10 0x7ff6180c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:18.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff6167fc640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff61806cac0 0x7ff6180aebe0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7ff60c011880 tx=0x7ff60c011d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff5ff7fe640 1 -- 172.21.15.81:0/2694672240 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c01f070 con 0x7ff61806cac0 2024-06-29T14:41:18.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff5ff7fe640 1 -- 172.21.15.81:0/2694672240 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff60c00fe40 con 0x7ff61806cac0 2024-06-29T14:41:18.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff5ff7fe640 1 -- 172.21.15.81:0/2694672240 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c0126b0 con 0x7ff61806cac0 2024-06-29T14:41:18.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff61d711640 1 -- 172.21.15.81:0/2694672240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6180c5d50 con 0x7ff61806cac0 2024-06-29T14:41:18.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.192+0000 7ff61d711640 1 -- 172.21.15.81:0/2694672240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6180c6310 con 0x7ff61806cac0 2024-06-29T14:41:18.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.193+0000 7ff61d711640 1 -- 172.21.15.81:0/2694672240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff61806e2a0 con 0x7ff61806cac0 2024-06-29T14:41:18.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.194+0000 7ff5ff7fe640 1 -- 172.21.15.81:0/2694672240 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7ff60c022070 con 0x7ff61806cac0 2024-06-29T14:41:18.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.195+0000 7ff5ff7fe640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff6040763f0 0x7ff604078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.195+0000 7ff5ff7fe640 1 -- 172.21.15.81:0/2694672240 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 4957+0+0 (secure 0 0 0) 0x7ff60c0980e0 con 0x7ff61806cac0 2024-06-29T14:41:18.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.195+0000 7ff616ffd640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff6040763f0 0x7ff604078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.195+0000 7ff616ffd640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff6040763f0 0x7ff604078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff608009ed0 tx=0x7ff60802d040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.199+0000 7ff5ff7fe640 1 -- 172.21.15.81:0/2694672240 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff61806e2a0 con 0x7ff61806cac0 2024-06-29T14:41:18.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.381+0000 7ff61d711640 1 -- 172.21.15.81:0/2694672240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff61806f3a0 con 0x7ff61806cac0 2024-06-29T14:41:18.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.382+0000 7ff5ff7fe640 1 -- 172.21.15.81:0/2694672240 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+13048 (secure 0 0 0) 0x7ff60c0615e0 con 0x7ff61806cac0 2024-06-29T14:41:18.379 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:18.380 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":129,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:17.675684+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:18.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 -- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff6040763f0 msgr2=0x7ff604078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff6040763f0 0x7ff604078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff608009ed0 tx=0x7ff60802d040 comp rx=0 tx=0).stop 2024-06-29T14:41:18.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 -- 172.21.15.81:0/2694672240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff61806cac0 msgr2=0x7ff6180aebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff61806cac0 0x7ff6180aebe0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7ff60c011880 tx=0x7ff60c011d40 comp rx=0 tx=0).stop 2024-06-29T14:41:18.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 -- 172.21.15.81:0/2694672240 shutdown_connections 2024-06-29T14:41:18.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff6040763f0 0x7ff604078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6180c6b10 0x7ff6180c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff61806cac0 0x7ff6180aebe0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 --2- 172.21.15.81:0/2694672240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff600003a00 0x7ff61814ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 -- 172.21.15.81:0/2694672240 >> 172.21.15.81:0/2694672240 conn(0x7ff6180d0930 msgr2=0x7ff6180ce5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 -- 172.21.15.81:0/2694672240 shutdown_connections 2024-06-29T14:41:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.384+0000 7ff61d711640 1 -- 172.21.15.81:0/2694672240 wait complete. 2024-06-29T14:41:18.393 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:18.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:18.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.513+0000 7fd5ab238640 1 -- 172.21.15.81:0/2590276747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd58c003a00 msgr2=0x7fd58c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.513+0000 7fd5ab238640 1 --2- 172.21.15.81:0/2590276747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd58c003a00 0x7fd58c003e70 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fd594009b50 tx=0x7fd58c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:18.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.513+0000 7fd5a97d5640 1 -- 172.21.15.81:0/2590276747 reap_dead start 2024-06-29T14:41:18.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.513+0000 7fd5ab238640 1 -- 172.21.15.81:0/2590276747 shutdown_connections 2024-06-29T14:41:18.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.513+0000 7fd5ab238640 1 -- 172.21.15.81:0/2590276747 >> 172.21.15.81:0/2590276747 conn(0x7fd5a40d0930 msgr2=0x7fd5a40d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.513+0000 7fd5ab238640 1 -- 172.21.15.81:0/2590276747 shutdown_connections 2024-06-29T14:41:18.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.513+0000 7fd5ab238640 1 -- 172.21.15.81:0/2590276747 wait complete. 2024-06-29T14:41:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.514+0000 7fd5ab238640 1 Processor -- start 2024-06-29T14:41:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.515+0000 7fd5ab238640 1 -- start start 2024-06-29T14:41:18.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5ab238640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd58c003a00 0x7fd5a4150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5ab238640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a41503c0 0x7fd5a414fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5ab238640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a40c58d0 0x7fd5a40c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5ab238640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a415b260 con 0x7fd5a40c58d0 2024-06-29T14:41:18.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5ab238640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a415b0e0 con 0x7fd58c003a00 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5ab238640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a415b3e0 con 0x7fd5a41503c0 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a97d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a40c58d0 0x7fd5a40c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a97d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a40c58d0 0x7fd5a40c5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60672/0 (socket says 172.21.15.81:60672) 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a97d5640 1 -- 172.21.15.81:0/1411235494 learned_addr learned my addr 172.21.15.81:0/1411235494 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a8fd4640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd58c003a00 0x7fd5a4150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a3fff640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a41503c0 0x7fd5a414fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a97d5640 1 -- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a41503c0 msgr2=0x7fd5a414fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a97d5640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a41503c0 0x7fd5a414fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a97d5640 1 -- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd58c003a00 msgr2=0x7fd5a4150880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a97d5640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd58c003a00 0x7fd5a4150880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a97d5640 1 -- 172.21.15.81:0/1411235494 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a40ab570 con 0x7fd5a40c58d0 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.516+0000 7fd5a97d5640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a40c58d0 0x7fd5a40c5ce0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd59c010910 tx=0x7fd59c010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.517+0000 7fd5a17fa640 1 -- 172.21.15.81:0/1411235494 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c00ea00 con 0x7fd5a40c58d0 2024-06-29T14:41:18.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.517+0000 7fd5a17fa640 1 -- 172.21.15.81:0/1411235494 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd59c00ec90 con 0x7fd5a40c58d0 2024-06-29T14:41:18.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.517+0000 7fd5a17fa640 1 -- 172.21.15.81:0/1411235494 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c005280 con 0x7fd5a40c58d0 2024-06-29T14:41:18.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.517+0000 7fd5ab238640 1 -- 172.21.15.81:0/1411235494 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a40ab810 con 0x7fd5a40c58d0 2024-06-29T14:41:18.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.517+0000 7fd5ab238640 1 -- 172.21.15.81:0/1411235494 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a40abdd0 con 0x7fd5a40c58d0 2024-06-29T14:41:18.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.518+0000 7fd5ab238640 1 -- 172.21.15.81:0/1411235494 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a406e2a0 con 0x7fd5a40c58d0 2024-06-29T14:41:18.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.518+0000 7fd5a17fa640 1 -- 172.21.15.81:0/1411235494 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fd59c01d020 con 0x7fd5a40c58d0 2024-06-29T14:41:18.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.519+0000 7fd5a17fa640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd590076420 0x7fd590078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.519+0000 7fd5a17fa640 1 -- 172.21.15.81:0/1411235494 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 4957+0+0 (secure 0 0 0) 0x7fd59c097420 con 0x7fd5a40c58d0 2024-06-29T14:41:18.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.519+0000 7fd5a8fd4640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd590076420 0x7fd590078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.520+0000 7fd5a8fd4640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd590076420 0x7fd590078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd594009b50 tx=0x7fd59402d040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.524+0000 7fd5a17fa640 1 -- 172.21.15.81:0/1411235494 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd5a406e2a0 con 0x7fd5a40c58d0 2024-06-29T14:41:18.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.704+0000 7fd5ab238640 1 -- 172.21.15.81:0/1411235494 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd5a40b3d70 con 0x7fd5a40c58d0 2024-06-29T14:41:18.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.704+0000 7fd5a17fa640 1 -- 172.21.15.81:0/1411235494 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) ==== 93+0+10 (secure 0 0 0) 0x7fd59c00ee60 con 0x7fd5a40c58d0 2024-06-29T14:41:18.701 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:41:18.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 -- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd590076420 msgr2=0x7fd590078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd590076420 0x7fd590078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd594009b50 tx=0x7fd59402d040 comp rx=0 tx=0).stop 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 -- 172.21.15.81:0/1411235494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a40c58d0 msgr2=0x7fd5a40c5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a40c58d0 0x7fd5a40c5ce0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd59c010910 tx=0x7fd59c010dd0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 -- 172.21.15.81:0/1411235494 shutdown_connections 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd590076420 0x7fd590078930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a40c58d0 0x7fd5a40c5ce0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a41503c0 0x7fd5a414fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 --2- 172.21.15.81:0/1411235494 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd58c003a00 0x7fd5a4150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 -- 172.21.15.81:0/1411235494 >> 172.21.15.81:0/1411235494 conn(0x7fd5a40d0930 msgr2=0x7fd5a40cf490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 -- 172.21.15.81:0/1411235494 shutdown_connections 2024-06-29T14:41:18.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.706+0000 7fd5ab238640 1 -- 172.21.15.81:0/1411235494 wait complete. 2024-06-29T14:41:18.715 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:41:18.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.831+0000 7f3736065640 1 -- 172.21.15.81:0/3103775204 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37300cea50 msgr2=0x7f37300b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.831+0000 7f3736065640 1 --2- 172.21.15.81:0/3103775204 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37300cea50 0x7f37300b4810 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3728009830 tx=0x7f372802a480 comp rx=0 tx=0).stop 2024-06-29T14:41:18.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.831+0000 7f3736065640 1 -- 172.21.15.81:0/3103775204 shutdown_connections 2024-06-29T14:41:18.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.831+0000 7f3736065640 1 --2- 172.21.15.81:0/3103775204 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37300cea50 0x7f37300b4810 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.831+0000 7f3736065640 1 --2- 172.21.15.81:0/3103775204 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37300cd480 0x7f37300ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.831+0000 7f3736065640 1 -- 172.21.15.81:0/3103775204 >> 172.21.15.81:0/3103775204 conn(0x7f37300b3510 msgr2=0x7f37300b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:18.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.831+0000 7f3736065640 1 -- 172.21.15.81:0/3103775204 shutdown_connections 2024-06-29T14:41:18.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.831+0000 7f3736065640 1 -- 172.21.15.81:0/3103775204 wait complete. 2024-06-29T14:41:18.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.832+0000 7f3736065640 1 Processor -- start 2024-06-29T14:41:18.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.833+0000 7f3736065640 1 -- start start 2024-06-29T14:41:18.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f3736065640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37300b6eb0 0x7f3730165f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f3736065640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37300cd480 0x7f3730152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f3736065640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37300cea50 0x7f3730151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f3736065640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f37300b50e0 con 0x7f37300cea50 2024-06-29T14:41:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f3736065640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f37300b4f60 con 0x7f37300cd480 2024-06-29T14:41:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f3736065640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f37300b5260 con 0x7f37300b6eb0 2024-06-29T14:41:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37300cea50 0x7f3730151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37300cea50 0x7f3730151510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60674/0 (socket says 172.21.15.81:60674) 2024-06-29T14:41:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372ffff640 1 -- 172.21.15.81:0/3185409226 learned_addr learned my addr 172.21.15.81:0/3185409226 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372effd640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37300cd480 0x7f3730152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372f7fe640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37300b6eb0 0x7f3730165f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372ffff640 1 -- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37300b6eb0 msgr2=0x7f3730165f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372ffff640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37300b6eb0 0x7f3730165f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372ffff640 1 -- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37300cd480 msgr2=0x7f3730152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372ffff640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37300cd480 0x7f3730152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372ffff640 1 -- 172.21.15.81:0/3185409226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3730166460 con 0x7f37300cea50 2024-06-29T14:41:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.834+0000 7f372ffff640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37300cea50 0x7f3730151510 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f372802aa70 tx=0x7f3728033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.835+0000 7f3717fff640 1 -- 172.21.15.81:0/3185409226 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728031b50 con 0x7f37300cea50 2024-06-29T14:41:18.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.835+0000 7f3717fff640 1 -- 172.21.15.81:0/3185409226 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3728031de0 con 0x7f37300cea50 2024-06-29T14:41:18.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.835+0000 7f3717fff640 1 -- 172.21.15.81:0/3185409226 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728005280 con 0x7f37300cea50 2024-06-29T14:41:18.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.835+0000 7f3736065640 1 -- 172.21.15.81:0/3185409226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3730154490 con 0x7f37300cea50 2024-06-29T14:41:18.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.835+0000 7f3736065640 1 -- 172.21.15.81:0/3185409226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37300ca460 con 0x7f37300cea50 2024-06-29T14:41:18.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.836+0000 7f3736065640 1 -- 172.21.15.81:0/3185409226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37300ce510 con 0x7f37300cea50 2024-06-29T14:41:18.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.836+0000 7f3717fff640 1 -- 172.21.15.81:0/3185409226 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f3728005420 con 0x7f37300cea50 2024-06-29T14:41:18.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.837+0000 7f3717fff640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f371c076420 0x7f371c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:18.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.837+0000 7f3717fff640 1 -- 172.21.15.81:0/3185409226 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 4957+0+0 (secure 0 0 0) 0x7f37280b8a30 con 0x7f37300cea50 2024-06-29T14:41:18.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.837+0000 7f372f7fe640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f371c076420 0x7f371c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:18.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.837+0000 7f372f7fe640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f371c076420 0x7f371c078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f37300cdb60 tx=0x7f3720009040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:18.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:18.841+0000 7f3717fff640 1 -- 172.21.15.81:0/3185409226 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f372803f030 con 0x7f37300cea50 2024-06-29T14:41:19.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.023+0000 7f3736065640 1 -- 172.21.15.81:0/3185409226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3730160710 con 0x7f37300cea50 2024-06-29T14:41:19.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.023+0000 7f3717fff640 1 -- 172.21.15.81:0/3185409226 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v129) ==== 84+0+65 (secure 0 0 0) 0x7f3728031770 con 0x7f37300cea50 2024-06-29T14:41:19.020 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:19.021 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:41:19.021 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:19.021 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T14:41:19.021 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T14:41:19.021 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:19.021 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:41:19.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 -- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f371c076420 msgr2=0x7f371c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f371c076420 0x7f371c078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f37300cdb60 tx=0x7f3720009040 comp rx=0 tx=0).stop 2024-06-29T14:41:19.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 -- 172.21.15.81:0/3185409226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37300cea50 msgr2=0x7f3730151510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37300cea50 0x7f3730151510 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f372802aa70 tx=0x7f3728033ba0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 -- 172.21.15.81:0/3185409226 shutdown_connections 2024-06-29T14:41:19.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f371c076420 0x7f371c078930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37300cea50 0x7f3730151510 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37300cd480 0x7f3730152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 --2- 172.21.15.81:0/3185409226 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37300b6eb0 0x7f3730165f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 -- 172.21.15.81:0/3185409226 >> 172.21.15.81:0/3185409226 conn(0x7f37300b3510 msgr2=0x7f37300b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:19.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 -- 172.21.15.81:0/3185409226 shutdown_connections 2024-06-29T14:41:19.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.025+0000 7f3736065640 1 -- 172.21.15.81:0/3185409226 wait complete. 2024-06-29T14:41:19.033 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:41:19.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:41:19.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.154+0000 7f134f3be640 1 -- 172.21.15.81:0/2937332219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1334005250 msgr2=0x7f133400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.154+0000 7f134f3be640 1 --2- 172.21.15.81:0/2937332219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1334005250 0x7f133400b9c0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f1340001d90 tx=0x7f1334005830 comp rx=0 tx=0).stop 2024-06-29T14:41:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.154+0000 7f134f3be640 1 -- 172.21.15.81:0/2937332219 shutdown_connections 2024-06-29T14:41:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.154+0000 7f134f3be640 1 --2- 172.21.15.81:0/2937332219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1334005250 0x7f133400b9c0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.154+0000 7f134f3be640 1 -- 172.21.15.81:0/2937332219 >> 172.21.15.81:0/2937332219 conn(0x7f134806cd60 msgr2=0x7f13480b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:19.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.155+0000 7f134f3be640 1 -- 172.21.15.81:0/2937332219 shutdown_connections 2024-06-29T14:41:19.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.155+0000 7f134f3be640 1 -- 172.21.15.81:0/2937332219 wait complete. 2024-06-29T14:41:19.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.156+0000 7f134f3be640 1 Processor -- start 2024-06-29T14:41:19.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.156+0000 7f134f3be640 1 -- start start 2024-06-29T14:41:19.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134f3be640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1334005250 0x7f134815a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134f3be640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f134815dd20 0x7f134813c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134f3be640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1348139700 0x7f1348139b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134f3be640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f134815fd40 con 0x7f1334005250 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134f3be640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f134815fbc0 con 0x7f1348139700 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134f3be640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f134815fec0 con 0x7f134815dd20 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134d15a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1334005250 0x7f134815a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134d15a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1334005250 0x7f134815a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60694/0 (socket says 172.21.15.81:60694) 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134d15a640 1 -- 172.21.15.81:0/424589668 learned_addr learned my addr 172.21.15.81:0/424589668 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134c959640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f134815dd20 0x7f134813c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134d95b640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1348139700 0x7f1348139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134d15a640 1 -- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f134815dd20 msgr2=0x7f134813c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134d15a640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f134815dd20 0x7f134813c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134d15a640 1 -- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1348139700 msgr2=0x7f1348139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134d15a640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1348139700 0x7f1348139b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134d15a640 1 -- 172.21.15.81:0/424589668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134813a050 con 0x7f1334005250 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134d15a640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1334005250 0x7f134815a330 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f1344011820 tx=0x7f1344011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f133dffb640 1 -- 172.21.15.81:0/424589668 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134400fd50 con 0x7f1334005250 2024-06-29T14:41:19.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f133dffb640 1 -- 172.21.15.81:0/424589668 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1344016070 con 0x7f1334005250 2024-06-29T14:41:19.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f133dffb640 1 -- 172.21.15.81:0/424589668 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344012690 con 0x7f1334005250 2024-06-29T14:41:19.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.158+0000 7f134f3be640 1 -- 172.21.15.81:0/424589668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134815cc80 con 0x7f1334005250 2024-06-29T14:41:19.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.159+0000 7f134f3be640 1 -- 172.21.15.81:0/424589668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134815d240 con 0x7f1334005250 2024-06-29T14:41:19.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.160+0000 7f134f3be640 1 -- 172.21.15.81:0/424589668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1348160840 con 0x7f1334005250 2024-06-29T14:41:19.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.160+0000 7f133dffb640 1 -- 172.21.15.81:0/424589668 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f1344018030 con 0x7f1334005250 2024-06-29T14:41:19.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.161+0000 7f133dffb640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f132c0763f0 0x7f132c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.161+0000 7f133dffb640 1 -- 172.21.15.81:0/424589668 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 4957+0+0 (secure 0 0 0) 0x7f1344097ec0 con 0x7f1334005250 2024-06-29T14:41:19.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.161+0000 7f134c959640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f132c0763f0 0x7f132c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.162+0000 7f134c959640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f132c0763f0 0x7f132c078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f13481439e0 tx=0x7f13380067a0 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:19.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.165+0000 7f133dffb640 1 -- 172.21.15.81:0/424589668 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f134813a050 con 0x7f1334005250 2024-06-29T14:41:19.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.347+0000 7f134f3be640 1 -- 172.21.15.81:0/424589668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1348140720 con 0x7f1334005250 2024-06-29T14:41:19.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.678+0000 7f133dffb640 1 -- 172.21.15.81:0/424589668 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v130) ==== 138+0+0 (secure 0 0 0) 0x7f1344061970 con 0x7f1334005250 2024-06-29T14:41:19.676 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T14:41:19.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 -- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f132c0763f0 msgr2=0x7f132c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f132c0763f0 0x7f132c078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f13481439e0 tx=0x7f13380067a0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 -- 172.21.15.81:0/424589668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1334005250 msgr2=0x7f134815a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1334005250 0x7f134815a330 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f1344011820 tx=0x7f1344011ce0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 -- 172.21.15.81:0/424589668 shutdown_connections 2024-06-29T14:41:19.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f132c0763f0 0x7f132c078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1348139700 0x7f1348139b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f134815dd20 0x7f134813c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 --2- 172.21.15.81:0/424589668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1334005250 0x7f134815a330 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 -- 172.21.15.81:0/424589668 >> 172.21.15.81:0/424589668 conn(0x7f134806cd60 msgr2=0x7f1348144ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:19.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 -- 172.21.15.81:0/424589668 shutdown_connections 2024-06-29T14:41:19.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.680+0000 7f134f3be640 1 -- 172.21.15.81:0/424589668 wait complete. 2024-06-29T14:41:19.693 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:41:19.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.818+0000 7f9de7489640 1 -- 172.21.15.81:0/25998454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9de00b9040 msgr2=0x7f9de00d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.818+0000 7f9de7489640 1 --2- 172.21.15.81:0/25998454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9de00b9040 0x7f9de00d0480 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc005fd0 tx=0x7f9ddc02a940 comp rx=0 tx=0).stop 2024-06-29T14:41:19.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.818+0000 7f9de7489640 1 -- 172.21.15.81:0/25998454 shutdown_connections 2024-06-29T14:41:19.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.818+0000 7f9de7489640 1 --2- 172.21.15.81:0/25998454 >> v2:172.21.15.167:3300/0 conn(0x7f9de00d2a90 0x7f9de00d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.818+0000 7f9de7489640 1 --2- 172.21.15.81:0/25998454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9de00b9040 0x7f9de00d0480 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.818+0000 7f9de7489640 1 -- 172.21.15.81:0/25998454 >> 172.21.15.81:0/25998454 conn(0x7f9de009cf30 msgr2=0x7f9de00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:19.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.818+0000 7f9de7489640 1 -- 172.21.15.81:0/25998454 shutdown_connections 2024-06-29T14:41:19.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.818+0000 7f9de7489640 1 -- 172.21.15.81:0/25998454 wait complete. 2024-06-29T14:41:19.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.819+0000 7f9de7489640 1 Processor -- start 2024-06-29T14:41:19.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.820+0000 7f9de7489640 1 -- start start 2024-06-29T14:41:19.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de7489640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9de00b5250 0x7f9de00a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de7489640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9de00b9040 0x7f9de00cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de7489640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9de00d2a90 0x7f9de0160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de7489640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9de00b7d30 con 0x7f9de00b5250 2024-06-29T14:41:19.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de7489640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9de00b7bb0 con 0x7f9de00b9040 2024-06-29T14:41:19.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de7489640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9de00b7eb0 con 0x7f9de00d2a90 2024-06-29T14:41:19.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de5225640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9de00b5250 0x7f9de00a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de5225640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9de00b5250 0x7f9de00a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60700/0 (socket says 172.21.15.81:60700) 2024-06-29T14:41:19.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de5225640 1 -- 172.21.15.81:0/2981509528 learned_addr learned my addr 172.21.15.81:0/2981509528 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:19.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de5a26640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9de00d2a90 0x7f9de0160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de4a24640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9de00b9040 0x7f9de00cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de5225640 1 -- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9de00d2a90 msgr2=0x7f9de0160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de5225640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9de00d2a90 0x7f9de0160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de5225640 1 -- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9de00b9040 msgr2=0x7f9de00cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:19.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de5225640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9de00b9040 0x7f9de00cdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:19.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de5225640 1 -- 172.21.15.81:0/2981509528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de01f8ef0 con 0x7f9de00b5250 2024-06-29T14:41:19.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.821+0000 7f9de5225640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9de00b5250 0x7f9de00a1820 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc009cd0 tx=0x7f9ddc031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:19.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.822+0000 7f9dd5ffb640 1 -- 172.21.15.81:0/2981509528 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc03e070 con 0x7f9de00b5250 2024-06-29T14:41:19.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.822+0000 7f9dd5ffb640 1 -- 172.21.15.81:0/2981509528 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ddc004540 con 0x7f9de00b5250 2024-06-29T14:41:19.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.822+0000 7f9de7489640 1 -- 172.21.15.81:0/2981509528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de00ce170 con 0x7f9de00b5250 2024-06-29T14:41:19.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.822+0000 7f9dd5ffb640 1 -- 172.21.15.81:0/2981509528 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc043460 con 0x7f9de00b5250 2024-06-29T14:41:19.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.822+0000 7f9de7489640 1 -- 172.21.15.81:0/2981509528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de01f96d0 con 0x7f9de00b5250 2024-06-29T14:41:19.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.823+0000 7f9de7489640 1 -- 172.21.15.81:0/2981509528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9de00b83b0 con 0x7f9de00b5250 2024-06-29T14:41:19.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.824+0000 7f9dd5ffb640 1 -- 172.21.15.81:0/2981509528 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f9ddc03a440 con 0x7f9de00b5250 2024-06-29T14:41:19.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.824+0000 7f9dd5ffb640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dc40763f0 0x7f9dc4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:19.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.824+0000 7f9dd5ffb640 1 -- 172.21.15.81:0/2981509528 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5327+0+0 (secure 0 0 0) 0x7f9ddc0b9440 con 0x7f9de00b5250 2024-06-29T14:41:19.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.824+0000 7f9de4a24640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dc40763f0 0x7f9dc4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:19.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.825+0000 7f9de4a24640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dc40763f0 0x7f9dc4078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00046b0 tx=0x7f9dd0009040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:19.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:19.829+0000 7f9dd5ffb640 1 -- 172.21.15.81:0/2981509528 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9ddc082d10 con 0x7f9de00b5250 2024-06-29T14:41:20.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.009+0000 7f9de7489640 1 -- 172.21.15.81:0/2981509528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9de00b4e20 con 0x7f9de00b5250 2024-06-29T14:41:20.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.680+0000 7f9dd5ffb640 1 -- 172.21.15.81:0/2981509528 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v131) ==== 170+0+0 (secure 0 0 0) 0x7f9ddc082610 con 0x7f9de00b5250 2024-06-29T14:41:20.678 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T14:41:20.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.683+0000 7f9de7489640 1 -- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dc40763f0 msgr2=0x7f9dc4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:20.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.683+0000 7f9de7489640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dc40763f0 0x7f9dc4078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9dd00046b0 tx=0x7f9dd0009040 comp rx=0 tx=0).stop 2024-06-29T14:41:20.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.683+0000 7f9de7489640 1 -- 172.21.15.81:0/2981509528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9de00b5250 msgr2=0x7f9de00a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:20.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.684+0000 7f9de7489640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9de00b5250 0x7f9de00a1820 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc009cd0 tx=0x7f9ddc031ea0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.684+0000 7f9de7489640 1 -- 172.21.15.81:0/2981509528 shutdown_connections 2024-06-29T14:41:20.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.684+0000 7f9de7489640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f9dc40763f0 0x7f9dc4078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.684+0000 7f9de7489640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9de00d2a90 0x7f9de0160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.684+0000 7f9de7489640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9de00b9040 0x7f9de00cdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.684+0000 7f9de7489640 1 --2- 172.21.15.81:0/2981509528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9de00b5250 0x7f9de00a1820 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.684+0000 7f9de7489640 1 -- 172.21.15.81:0/2981509528 >> 172.21.15.81:0/2981509528 conn(0x7f9de009cf30 msgr2=0x7f9de00cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:20.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.684+0000 7f9de7489640 1 -- 172.21.15.81:0/2981509528 shutdown_connections 2024-06-29T14:41:20.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.684+0000 7f9de7489640 1 -- 172.21.15.81:0/2981509528 wait complete. 2024-06-29T14:41:20.709 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:41:20.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.832+0000 7fcc98b39640 1 -- 172.21.15.81:0/2474139105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9415b400 msgr2=0x7fcc9415b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:20.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.832+0000 7fcc98b39640 1 --2- 172.21.15.81:0/2474139105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9415b400 0x7fcc9415b810 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c005fd0 tx=0x7fcc8c02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.832+0000 7fcc98b39640 1 -- 172.21.15.81:0/2474139105 shutdown_connections 2024-06-29T14:41:20.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.832+0000 7fcc98b39640 1 --2- 172.21.15.81:0/2474139105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9415b400 0x7fcc9415b810 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.832+0000 7fcc98b39640 1 --2- 172.21.15.81:0/2474139105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc94073690 0x7fcc94144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.832+0000 7fcc98b39640 1 -- 172.21.15.81:0/2474139105 >> 172.21.15.81:0/2474139105 conn(0x7fcc940b2920 msgr2=0x7fcc940b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:20.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.832+0000 7fcc98b39640 1 -- 172.21.15.81:0/2474139105 shutdown_connections 2024-06-29T14:41:20.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.832+0000 7fcc98b39640 1 -- 172.21.15.81:0/2474139105 wait complete. 2024-06-29T14:41:20.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.834+0000 7fcc98b39640 1 Processor -- start 2024-06-29T14:41:20.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.834+0000 7fcc98b39640 1 -- start start 2024-06-29T14:41:20.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.835+0000 7fcc98b39640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc940b4800 0x7fcc941f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:20.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.835+0000 7fcc98b39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9415b400 0x7fcc94142640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:20.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.835+0000 7fcc98b39640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9415a840 0x7fcc941403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:20.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.835+0000 7fcc98b39640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9406f2e0 con 0x7fcc9415b400 2024-06-29T14:41:20.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.835+0000 7fcc98b39640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9406f160 con 0x7fcc9415a840 2024-06-29T14:41:20.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.835+0000 7fcc98b39640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcc9406f460 con 0x7fcc940b4800 2024-06-29T14:41:20.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.835+0000 7fcc91d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9415b400 0x7fcc94142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:20.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.835+0000 7fcc91d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9415b400 0x7fcc94142640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47968/0 (socket says 172.21.15.81:47968) 2024-06-29T14:41:20.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.835+0000 7fcc91d9b640 1 -- 172.21.15.81:0/1071961551 learned_addr learned my addr 172.21.15.81:0/1071961551 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc9259c640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc940b4800 0x7fcc941f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc92d9d640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9415a840 0x7fcc941403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc91d9b640 1 -- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc940b4800 msgr2=0x7fcc941f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc91d9b640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc940b4800 0x7fcc941f4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc91d9b640 1 -- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9415a840 msgr2=0x7fcc941403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc91d9b640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9415a840 0x7fcc941403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc91d9b640 1 -- 172.21.15.81:0/1071961551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc941f5360 con 0x7fcc9415b400 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc91d9b640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9415b400 0x7fcc94142640 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fcc8800ea20 tx=0x7fcc8800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc82ffd640 1 -- 172.21.15.81:0/1071961551 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8800cd50 con 0x7fcc9415b400 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc98b39640 1 -- 172.21.15.81:0/1071961551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc941f5640 con 0x7fcc9415b400 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc82ffd640 1 -- 172.21.15.81:0/1071961551 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc88004020 con 0x7fcc9415b400 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc82ffd640 1 -- 172.21.15.81:0/1071961551 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc88004450 con 0x7fcc9415b400 2024-06-29T14:41:20.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.836+0000 7fcc98b39640 1 -- 172.21.15.81:0/1071961551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc9415db20 con 0x7fcc9415b400 2024-06-29T14:41:20.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.837+0000 7fcc98b39640 1 -- 172.21.15.81:0/1071961551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc941f5b60 con 0x7fcc9415b400 2024-06-29T14:41:20.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.838+0000 7fcc82ffd640 1 -- 172.21.15.81:0/1071961551 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fcc880035f0 con 0x7fcc9415b400 2024-06-29T14:41:20.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.839+0000 7fcc82ffd640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fcc740763f0 0x7fcc74078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:20.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.839+0000 7fcc82ffd640 1 -- 172.21.15.81:0/1071961551 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 5709+0+0 (secure 0 0 0) 0x7fcc88098360 con 0x7fcc9415b400 2024-06-29T14:41:20.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.839+0000 7fcc9259c640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fcc740763f0 0x7fcc74078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:20.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.839+0000 7fcc9259c640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fcc740763f0 0x7fcc74078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc84001720 tx=0x7fcc84007e90 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:20.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:20.842+0000 7fcc82ffd640 1 -- 172.21.15.81:0/1071961551 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcc941f5b60 con 0x7fcc9415b400 2024-06-29T14:41:21.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.060+0000 7fcc98b39640 1 -- 172.21.15.81:0/1071961551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fcc941f50f0 con 0x7fcc9415b400 2024-06-29T14:41:21.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.708+0000 7fcc82ffd640 1 -- 172.21.15.81:0/1071961551 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v64) ==== 174+0+0 (secure 0 0 0) 0x7fcc88061450 con 0x7fcc9415b400 2024-06-29T14:41:21.706 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 16 and data pool 17 2024-06-29T14:41:21.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.713+0000 7fcc98b39640 1 -- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fcc740763f0 msgr2=0x7fcc74078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.713+0000 7fcc98b39640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fcc740763f0 0x7fcc74078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc84001720 tx=0x7fcc84007e90 comp rx=0 tx=0).stop 2024-06-29T14:41:21.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.713+0000 7fcc98b39640 1 -- 172.21.15.81:0/1071961551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9415b400 msgr2=0x7fcc94142640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.713+0000 7fcc98b39640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9415b400 0x7fcc94142640 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fcc8800ea20 tx=0x7fcc8800eee0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.721+0000 7fcc98b39640 1 -- 172.21.15.81:0/1071961551 shutdown_connections 2024-06-29T14:41:21.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.721+0000 7fcc98b39640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fcc740763f0 0x7fcc74078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.721+0000 7fcc98b39640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9415a840 0x7fcc941403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.721+0000 7fcc98b39640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9415b400 0x7fcc94142640 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.721+0000 7fcc98b39640 1 --2- 172.21.15.81:0/1071961551 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc940b4800 0x7fcc941f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.721+0000 7fcc98b39640 1 -- 172.21.15.81:0/1071961551 >> 172.21.15.81:0/1071961551 conn(0x7fcc940b2920 msgr2=0x7fcc940b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:21.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.722+0000 7fcc98b39640 1 -- 172.21.15.81:0/1071961551 shutdown_connections 2024-06-29T14:41:21.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.722+0000 7fcc98b39640 1 -- 172.21.15.81:0/1071961551 wait complete. 2024-06-29T14:41:21.737 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:21.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.853+0000 7efcab739640 1 -- 172.21.15.81:0/2082557170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc90005250 msgr2=0x7efc9000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.853+0000 7efcab739640 1 --2- 172.21.15.81:0/2082557170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc90005250 0x7efc9000b9c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7efc9c005fd0 tx=0x7efc90005830 comp rx=0 tx=0).stop 2024-06-29T14:41:21.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.853+0000 7efcab739640 1 -- 172.21.15.81:0/2082557170 shutdown_connections 2024-06-29T14:41:21.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.853+0000 7efcab739640 1 --2- 172.21.15.81:0/2082557170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc90005250 0x7efc9000b9c0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.853+0000 7efcab739640 1 -- 172.21.15.81:0/2082557170 >> 172.21.15.81:0/2082557170 conn(0x7efca40d00b0 msgr2=0x7efca40d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:21.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.853+0000 7efcab739640 1 -- 172.21.15.81:0/2082557170 shutdown_connections 2024-06-29T14:41:21.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.853+0000 7efcab739640 1 -- 172.21.15.81:0/2082557170 wait complete. 2024-06-29T14:41:21.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.855+0000 7efcab739640 1 Processor -- start 2024-06-29T14:41:21.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.855+0000 7efcab739640 1 -- start start 2024-06-29T14:41:21.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.856+0000 7efcab739640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc90005250 0x7efca413d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.856+0000 7efcab739640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efca41430c0 0x7efca4073830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.856+0000 7efcab739640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca40c98a0 0x7efca41f05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.856+0000 7efcab739640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efca40c5600 con 0x7efca40c98a0 2024-06-29T14:41:21.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.856+0000 7efcab739640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efca40c5480 con 0x7efca41430c0 2024-06-29T14:41:21.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.856+0000 7efcab739640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efca40c5780 con 0x7efc90005250 2024-06-29T14:41:21.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.856+0000 7efca9cd6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca40c98a0 0x7efca41f05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.856+0000 7efca9cd6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca40c98a0 0x7efca41f05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47982/0 (socket says 172.21.15.81:47982) 2024-06-29T14:41:21.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.856+0000 7efca9cd6640 1 -- 172.21.15.81:0/1919409414 learned_addr learned my addr 172.21.15.81:0/1919409414 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:21.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.856+0000 7efca94d5640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc90005250 0x7efca413d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efca9cd6640 1 -- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc90005250 msgr2=0x7efca413d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efca9cd6640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc90005250 0x7efca413d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efca9cd6640 1 -- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efca41430c0 msgr2=0x7efca4073830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:21.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efca9cd6640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efca41430c0 0x7efca4073830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:21.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efca9cd6640 1 -- 172.21.15.81:0/1919409414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca40b2fe0 con 0x7efca40c98a0 2024-06-29T14:41:21.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efca94d5640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc90005250 0x7efca413d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:21.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efca9cd6640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca40c98a0 0x7efca41f05d0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009e20 tx=0x7efc9c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:21.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efc99ffb640 1 -- 172.21.15.81:0/1919409414 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c033a70 con 0x7efca40c98a0 2024-06-29T14:41:21.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efc99ffb640 1 -- 172.21.15.81:0/1919409414 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc9c033d00 con 0x7efca40c98a0 2024-06-29T14:41:21.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efcab739640 1 -- 172.21.15.81:0/1919409414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca4073070 con 0x7efca40c98a0 2024-06-29T14:41:21.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efc99ffb640 1 -- 172.21.15.81:0/1919409414 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c043660 con 0x7efca40c98a0 2024-06-29T14:41:21.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.857+0000 7efcab739640 1 -- 172.21.15.81:0/1919409414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca4073310 con 0x7efca40c98a0 2024-06-29T14:41:21.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.858+0000 7efcab739640 1 -- 172.21.15.81:0/1919409414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca4072e80 con 0x7efca40c98a0 2024-06-29T14:41:21.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.859+0000 7efc99ffb640 1 -- 172.21.15.81:0/1919409414 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7efc9c043800 con 0x7efca40c98a0 2024-06-29T14:41:21.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.859+0000 7efc99ffb640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7efc88076420 0x7efc88078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:21.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.859+0000 7efc99ffb640 1 -- 172.21.15.81:0/1919409414 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5809+0+0 (secure 0 0 0) 0x7efc9c0ba4d0 con 0x7efca40c98a0 2024-06-29T14:41:21.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.860+0000 7efca94d5640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7efc88076420 0x7efc88078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:21.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.860+0000 7efca94d5640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7efc88076420 0x7efc88078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efca0006f00 tx=0x7efca00061b0 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:21.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:21.864+0000 7efc99ffb640 1 -- 172.21.15.81:0/1919409414 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efc9c083cb0 con 0x7efca40c98a0 2024-06-29T14:41:22.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.043+0000 7efcab739640 1 -- 172.21.15.81:0/1919409414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efca40ca750 con 0x7efca40c98a0 2024-06-29T14:41:22.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.044+0000 7efc99ffb640 1 -- 172.21.15.81:0/1919409414 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) ==== 74+0+16787 (secure 0 0 0) 0x7efc9c0835b0 con 0x7efca40c98a0 2024-06-29T14:41:22.042 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:22.042 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":132,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:21.061023+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.352751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:20.014553+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:22.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.046+0000 7efcab739640 1 -- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7efc88076420 msgr2=0x7efc88078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.046+0000 7efcab739640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7efc88076420 0x7efc88078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efca0006f00 tx=0x7efca00061b0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.046+0000 7efcab739640 1 -- 172.21.15.81:0/1919409414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca40c98a0 msgr2=0x7efca41f05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.046+0000 7efcab739640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca40c98a0 0x7efca41f05d0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009e20 tx=0x7efc9c033670 comp rx=0 tx=0).stop 2024-06-29T14:41:22.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.047+0000 7efcab739640 1 -- 172.21.15.81:0/1919409414 shutdown_connections 2024-06-29T14:41:22.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.047+0000 7efcab739640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7efc88076420 0x7efc88078930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.047+0000 7efcab739640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca40c98a0 0x7efca41f05d0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.047+0000 7efcab739640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efca41430c0 0x7efca4073830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.047+0000 7efcab739640 1 --2- 172.21.15.81:0/1919409414 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efc90005250 0x7efca413d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.047+0000 7efcab739640 1 -- 172.21.15.81:0/1919409414 >> 172.21.15.81:0/1919409414 conn(0x7efca40d00b0 msgr2=0x7efca4150c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:22.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.047+0000 7efcab739640 1 -- 172.21.15.81:0/1919409414 shutdown_connections 2024-06-29T14:41:22.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.047+0000 7efcab739640 1 -- 172.21.15.81:0/1919409414 wait complete. 2024-06-29T14:41:22.055 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.170+0000 7f5f440c1640 1 -- 172.21.15.81:0/1528374655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f28004800 msgr2=0x7f5f28004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.170+0000 7f5f440c1640 1 --2- 172.21.15.81:0/1528374655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f28004800 0x7f5f28004c10 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0016c0 tx=0x7f5f28005830 comp rx=0 tx=0).stop 2024-06-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.170+0000 7f5f440c1640 1 -- 172.21.15.81:0/1528374655 shutdown_connections 2024-06-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.170+0000 7f5f440c1640 1 --2- 172.21.15.81:0/1528374655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f28004800 0x7f5f28004c10 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.170+0000 7f5f440c1640 1 -- 172.21.15.81:0/1528374655 >> 172.21.15.81:0/1528374655 conn(0x7f5f3c0d00b0 msgr2=0x7f5f3c0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.170+0000 7f5f440c1640 1 -- 172.21.15.81:0/1528374655 shutdown_connections 2024-06-29T14:41:22.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.171+0000 7f5f440c1640 1 -- 172.21.15.81:0/1528374655 wait complete. 2024-06-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.172+0000 7f5f440c1640 1 Processor -- start 2024-06-29T14:41:22.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.172+0000 7f5f440c1640 1 -- start start 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.173+0000 7f5f440c1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f28004800 0x7f5f3c1f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.173+0000 7f5f440c1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f3c0b59a0 0x7f5f3c1e7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f440c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f3c1577a0 0x7f5f3c15e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f440c1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c06f620 con 0x7f5f3c1577a0 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f440c1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c06f4a0 con 0x7f5f28004800 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f440c1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5f3c06f7a0 con 0x7f5f3c0b59a0 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f4265e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f3c1577a0 0x7f5f3c15e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f4265e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f3c1577a0 0x7f5f3c15e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47998/0 (socket says 172.21.15.81:47998) 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f4265e640 1 -- 172.21.15.81:0/2297688387 learned_addr learned my addr 172.21.15.81:0/2297688387 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f4265e640 1 -- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f3c0b59a0 msgr2=0x7f5f3c1e7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f4265e640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f3c0b59a0 0x7f5f3c1e7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f4265e640 1 -- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f28004800 msgr2=0x7f5f3c1f3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f4265e640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f28004800 0x7f5f3c1f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f4265e640 1 -- 172.21.15.81:0/2297688387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3c1e83a0 con 0x7f5f3c1577a0 2024-06-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f4265e640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f3c1577a0 0x7f5f3c15e030 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f5f34005fd0 tx=0x7f5f34010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f327fc640 1 -- 172.21.15.81:0/2297688387 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3400eea0 con 0x7f5f3c1577a0 2024-06-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f327fc640 1 -- 172.21.15.81:0/2297688387 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f34015070 con 0x7f5f3c1577a0 2024-06-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f440c1640 1 -- 172.21.15.81:0/2297688387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3c15eef0 con 0x7f5f3c1577a0 2024-06-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f327fc640 1 -- 172.21.15.81:0/2297688387 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34003d00 con 0x7f5f3c1577a0 2024-06-29T14:41:22.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.174+0000 7f5f440c1640 1 -- 172.21.15.81:0/2297688387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f3c15d830 con 0x7f5f3c1577a0 2024-06-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.175+0000 7f5f440c1640 1 -- 172.21.15.81:0/2297688387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f3c06c3d0 con 0x7f5f3c1577a0 2024-06-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.176+0000 7f5f327fc640 1 -- 172.21.15.81:0/2297688387 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f5f34004030 con 0x7f5f3c1577a0 2024-06-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.176+0000 7f5f327fc640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f5f20076420 0x7f5f20078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.177+0000 7f5f327fc640 1 -- 172.21.15.81:0/2297688387 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5809+0+0 (secure 0 0 0) 0x7f5f340984f0 con 0x7f5f3c1577a0 2024-06-29T14:41:22.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.177+0000 7f5f41e5d640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f5f20076420 0x7f5f20078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:22.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.177+0000 7f5f41e5d640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f5f20076420 0x7f5f20078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f38001860 tx=0x7f5f38001c90 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:22.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.181+0000 7f5f327fc640 1 -- 172.21.15.81:0/2297688387 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5f34061d60 con 0x7f5f3c1577a0 2024-06-29T14:41:22.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.359+0000 7f5f440c1640 1 -- 172.21.15.81:0/2297688387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f3c159960 con 0x7f5f3c1577a0 2024-06-29T14:41:22.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.360+0000 7f5f327fc640 1 -- 172.21.15.81:0/2297688387 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) ==== 74+0+16787 (secure 0 0 0) 0x7f5f34061660 con 0x7f5f3c1577a0 2024-06-29T14:41:22.357 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:22.358 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":132,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:21.061023+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.352751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:20.014553+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:22.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 -- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f5f20076420 msgr2=0x7f5f20078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f5f20076420 0x7f5f20078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f38001860 tx=0x7f5f38001c90 comp rx=0 tx=0).stop 2024-06-29T14:41:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 -- 172.21.15.81:0/2297688387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f3c1577a0 msgr2=0x7f5f3c15e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f3c1577a0 0x7f5f3c15e030 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f5f34005fd0 tx=0x7f5f34010f40 comp rx=0 tx=0).stop 2024-06-29T14:41:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 -- 172.21.15.81:0/2297688387 shutdown_connections 2024-06-29T14:41:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f5f20076420 0x7f5f20078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f3c1577a0 0x7f5f3c15e030 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f3c0b59a0 0x7f5f3c1e7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 --2- 172.21.15.81:0/2297688387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f28004800 0x7f5f3c1f3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 -- 172.21.15.81:0/2297688387 >> 172.21.15.81:0/2297688387 conn(0x7f5f3c0d00b0 msgr2=0x7f5f3c159e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 -- 172.21.15.81:0/2297688387 shutdown_connections 2024-06-29T14:41:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.362+0000 7f5f440c1640 1 -- 172.21.15.81:0/2297688387 wait complete. 2024-06-29T14:41:22.371 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:41:22.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.491+0000 7f7921474640 1 -- 172.21.15.81:0/2315023456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f791c0d0b90 msgr2=0x7f791c0d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.491+0000 7f7921474640 1 --2- 172.21.15.81:0/2315023456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f791c0d0b90 0x7f791c0d1a80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7910009860 tx=0x7f791002a5c0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.491+0000 7f7921474640 1 -- 172.21.15.81:0/2315023456 shutdown_connections 2024-06-29T14:41:22.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.491+0000 7f7921474640 1 --2- 172.21.15.81:0/2315023456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f791c0d0b90 0x7f791c0d1a80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.491+0000 7f7921474640 1 -- 172.21.15.81:0/2315023456 >> 172.21.15.81:0/2315023456 conn(0x7f791c09cf30 msgr2=0x7f791c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:22.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.491+0000 7f7921474640 1 -- 172.21.15.81:0/2315023456 shutdown_connections 2024-06-29T14:41:22.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.491+0000 7f7921474640 1 -- 172.21.15.81:0/2315023456 wait complete. 2024-06-29T14:41:22.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.493+0000 7f7921474640 1 Processor -- start 2024-06-29T14:41:22.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.493+0000 7f7921474640 1 -- start start 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f7921474640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f791c0b5250 0x7f791c09e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f7921474640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f791c0b9040 0x7f791c1505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f7921474640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f791c150bb0 0x7f791c0a0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f7921474640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f791c0d2850 con 0x7f791c0b5250 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f7921474640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f791c0d26d0 con 0x7f791c150bb0 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f7921474640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f791c0d29d0 con 0x7f791c0b9040 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f791affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f791c0b5250 0x7f791c09e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f791affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f791c0b5250 0x7f791c09e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48004/0 (socket says 172.21.15.81:48004) 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f791affd640 1 -- 172.21.15.81:0/199980905 learned_addr learned my addr 172.21.15.81:0/199980905 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f791a7fc640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f791c0b9040 0x7f791c1505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.494+0000 7f791b7fe640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f791c150bb0 0x7f791c0a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:22.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f791affd640 1 -- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f791c0b9040 msgr2=0x7f791c1505a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f791affd640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f791c0b9040 0x7f791c1505a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f791affd640 1 -- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f791c150bb0 msgr2=0x7f791c0a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f791affd640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f791c150bb0 0x7f791c0a0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f791affd640 1 -- 172.21.15.81:0/199980905 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791c09ef60 con 0x7f791c0b5250 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f791b7fe640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f791c150bb0 0x7f791c0a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f791affd640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f791c0b5250 0x7f791c09e7a0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f790c0015a0 tx=0x7f790c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f79037fe640 1 -- 172.21.15.81:0/199980905 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c016020 con 0x7f791c0b5250 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f7921474640 1 -- 172.21.15.81:0/199980905 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791c09f240 con 0x7f791c0b5250 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f79037fe640 1 -- 172.21.15.81:0/199980905 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f790c00ce20 con 0x7f791c0b5250 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f79037fe640 1 -- 172.21.15.81:0/199980905 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790c012680 con 0x7f791c0b5250 2024-06-29T14:41:22.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.495+0000 7f7921474640 1 -- 172.21.15.81:0/199980905 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f791c09f800 con 0x7f791c0b5250 2024-06-29T14:41:22.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.496+0000 7f7921474640 1 -- 172.21.15.81:0/199980905 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f791c0d0480 con 0x7f791c0b5250 2024-06-29T14:41:22.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.497+0000 7f79037fe640 1 -- 172.21.15.81:0/199980905 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f790c019020 con 0x7f791c0b5250 2024-06-29T14:41:22.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.497+0000 7f79037fe640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f79080763f0 0x7f7908078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:22.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.497+0000 7f79037fe640 1 -- 172.21.15.81:0/199980905 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5809+0+0 (secure 0 0 0) 0x7f790c097ea0 con 0x7f791c0b5250 2024-06-29T14:41:22.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.497+0000 7f791a7fc640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f79080763f0 0x7f7908078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:22.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.498+0000 7f791a7fc640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f79080763f0 0x7f7908078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7910034000 tx=0x7f7910004730 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:22.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.501+0000 7f79037fe640 1 -- 172.21.15.81:0/199980905 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f790c020030 con 0x7f791c0b5250 2024-06-29T14:41:22.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:22.724+0000 7f7921474640 1 -- 172.21.15.81:0/199980905 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f791c068420 con 0x7f791c0b5250 2024-06-29T14:41:23.672 INFO:tasks.ceph.mds.a.smithi081.stderr:2024-06-29T14:41:23.674+0000 7f306022e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:41:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.726+0000 7f79037fe640 1 -- 172.21.15.81:0/199980905 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v67) ==== 118+0+0 (secure 0 0 0) 0x7f790c061600 con 0x7f791c0b5250 2024-06-29T14:41:23.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.728+0000 7f7921474640 1 -- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f79080763f0 msgr2=0x7f7908078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:23.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.728+0000 7f7921474640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f79080763f0 0x7f7908078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7910034000 tx=0x7f7910004730 comp rx=0 tx=0).stop 2024-06-29T14:41:23.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.729+0000 7f7921474640 1 -- 172.21.15.81:0/199980905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f791c0b5250 msgr2=0x7f791c09e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:23.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.729+0000 7f7921474640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f791c0b5250 0x7f791c09e7a0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f790c0015a0 tx=0x7f790c00a430 comp rx=0 tx=0).stop 2024-06-29T14:41:23.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.729+0000 7f7921474640 1 -- 172.21.15.81:0/199980905 shutdown_connections 2024-06-29T14:41:23.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.729+0000 7f7921474640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f79080763f0 0x7f7908078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.729+0000 7f7921474640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f791c150bb0 0x7f791c0a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.729+0000 7f7921474640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f791c0b9040 0x7f791c1505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.729+0000 7f7921474640 1 --2- 172.21.15.81:0/199980905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f791c0b5250 0x7f791c09e7a0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.729+0000 7f7921474640 1 -- 172.21.15.81:0/199980905 >> 172.21.15.81:0/199980905 conn(0x7f791c09cf30 msgr2=0x7f791c15ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:23.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.729+0000 7f7921474640 1 -- 172.21.15.81:0/199980905 shutdown_connections 2024-06-29T14:41:23.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.729+0000 7f7921474640 1 -- 172.21.15.81:0/199980905 wait complete. 2024-06-29T14:41:23.740 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:41:23.741 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:41:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.862+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/3757976541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe68074520 msgr2=0x7fbe6806f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.862+0000 7fbe6f9b2640 1 --2- 172.21.15.81:0/3757976541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe68074520 0x7fbe6806f140 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009830 tx=0x7fbe6402a510 comp rx=0 tx=0).stop 2024-06-29T14:41:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.862+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/3757976541 shutdown_connections 2024-06-29T14:41:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.862+0000 7fbe6f9b2640 1 --2- 172.21.15.81:0/3757976541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe68074520 0x7fbe6806f140 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.863+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/3757976541 >> 172.21.15.81:0/3757976541 conn(0x7fbe68072950 msgr2=0x7fbe680b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:23.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.863+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/3757976541 shutdown_connections 2024-06-29T14:41:23.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.863+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/3757976541 wait complete. 2024-06-29T14:41:23.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.864+0000 7fbe6f9b2640 1 Processor -- start 2024-06-29T14:41:23.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.864+0000 7fbe6f9b2640 1 -- start start 2024-06-29T14:41:23.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6f9b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe6806f680 0x7fbe680cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:23.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6f9b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe6806fa90 0x7fbe681470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:23.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6f9b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe68074520 0x7fbe681f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6f9b2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6815b190 con 0x7fbe6806fa90 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6f9b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6815b010 con 0x7fbe6806f680 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6f9b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbe680af310 con 0x7fbe68074520 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6cf4d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe6806fa90 0x7fbe681470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6cf4d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe6806fa90 0x7fbe681470f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48030/0 (socket says 172.21.15.81:48030) 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6cf4d640 1 -- 172.21.15.81:0/2091641222 learned_addr learned my addr 172.21.15.81:0/2091641222 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6d74e640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe6806f680 0x7fbe680cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6df4f640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe68074520 0x7fbe681f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6cf4d640 1 -- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe68074520 msgr2=0x7fbe681f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6cf4d640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe68074520 0x7fbe681f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6cf4d640 1 -- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe6806f680 msgr2=0x7fbe680cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6cf4d640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe6806f680 0x7fbe680cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6cf4d640 1 -- 172.21.15.81:0/2091641222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe680ce2b0 con 0x7fbe6806fa90 2024-06-29T14:41:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6df4f640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe68074520 0x7fbe681f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6d74e640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe6806f680 0x7fbe680cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6cf4d640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe6806fa90 0x7fbe681470f0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800d9f0 tx=0x7fbe5800deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe5dffb640 1 -- 172.21.15.81:0/2091641222 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800edc0 con 0x7fbe6806fa90 2024-06-29T14:41:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe5dffb640 1 -- 172.21.15.81:0/2091641222 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe58004020 con 0x7fbe6806fa90 2024-06-29T14:41:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/2091641222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe680ce590 con 0x7fbe6806fa90 2024-06-29T14:41:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.866+0000 7fbe5dffb640 1 -- 172.21.15.81:0/2091641222 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe58004470 con 0x7fbe6806fa90 2024-06-29T14:41:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.867+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/2091641222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe680ceb50 con 0x7fbe6806fa90 2024-06-29T14:41:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.868+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/2091641222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe680ce000 con 0x7fbe6806fa90 2024-06-29T14:41:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.868+0000 7fbe5dffb640 1 -- 172.21.15.81:0/2091641222 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fbe580035f0 con 0x7fbe6806fa90 2024-06-29T14:41:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.869+0000 7fbe5dffb640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe4c076420 0x7fbe4c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.869+0000 7fbe5dffb640 1 -- 172.21.15.81:0/2091641222 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7fbe58099910 con 0x7fbe6806fa90 2024-06-29T14:41:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.869+0000 7fbe6d74e640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe4c076420 0x7fbe4c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.870+0000 7fbe6d74e640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe4c076420 0x7fbe4c078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe64034000 tx=0x7fbe64004730 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:23.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:23.873+0000 7fbe5dffb640 1 -- 172.21.15.81:0/2091641222 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbe580630f0 con 0x7fbe6806fa90 2024-06-29T14:41:24.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.091+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/2091641222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fbe68153e10 con 0x7fbe6806fa90 2024-06-29T14:41:24.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.733+0000 7fbe5dffb640 1 -- 172.21.15.81:0/2091641222 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v68) ==== 122+0+0 (secure 0 0 0) 0x7fbe580629f0 con 0x7fbe6806fa90 2024-06-29T14:41:24.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe4c076420 msgr2=0x7fbe4c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe4c076420 0x7fbe4c078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe64034000 tx=0x7fbe64004730 comp rx=0 tx=0).stop 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/2091641222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe6806fa90 msgr2=0x7fbe681470f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe6806fa90 0x7fbe681470f0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800d9f0 tx=0x7fbe5800deb0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/2091641222 shutdown_connections 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe4c076420 0x7fbe4c078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe68074520 0x7fbe681f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe6806fa90 0x7fbe681470f0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 --2- 172.21.15.81:0/2091641222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe6806f680 0x7fbe680cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/2091641222 >> 172.21.15.81:0/2091641222 conn(0x7fbe68072950 msgr2=0x7fbe68068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/2091641222 shutdown_connections 2024-06-29T14:41:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.736+0000 7fbe6f9b2640 1 -- 172.21.15.81:0/2091641222 wait complete. 2024-06-29T14:41:24.746 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:41:24.746 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:24.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.862+0000 7f354cb4a640 1 -- 172.21.15.81:0/3443517727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3548073740 msgr2=0x7f354815b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.862+0000 7f354cb4a640 1 --2- 172.21.15.81:0/3443517727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3548073740 0x7f354815b570 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f3548140f10 tx=0x7f353002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.862+0000 7f354cb4a640 1 -- 172.21.15.81:0/3443517727 shutdown_connections 2024-06-29T14:41:24.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.862+0000 7f354cb4a640 1 --2- 172.21.15.81:0/3443517727 >> v2:172.21.15.167:3300/0 conn(0x7f354815bab0 0x7f35480ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.862+0000 7f354cb4a640 1 --2- 172.21.15.81:0/3443517727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3548073740 0x7f354815b570 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.862+0000 7f354cb4a640 1 -- 172.21.15.81:0/3443517727 >> 172.21.15.81:0/3443517727 conn(0x7f354806ccc0 msgr2=0x7f35480b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:24.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.862+0000 7f354cb4a640 1 -- 172.21.15.81:0/3443517727 shutdown_connections 2024-06-29T14:41:24.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.862+0000 7f354cb4a640 1 -- 172.21.15.81:0/3443517727 wait complete. 2024-06-29T14:41:24.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.864+0000 7f354cb4a640 1 Processor -- start 2024-06-29T14:41:24.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.864+0000 7f354cb4a640 1 -- start start 2024-06-29T14:41:24.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f354cb4a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f354813dd70 0x7f35480aec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f354cb4a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f354815bab0 0x7f3548131c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f354cb4a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3548158e80 0x7f3548158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f354cb4a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f35480cecd0 con 0x7f3548158e80 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f354cb4a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f35480ceb50 con 0x7f354813dd70 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f354cb4a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f35480cee50 con 0x7f354815bab0 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f3546d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3548158e80 0x7f3548158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f3546d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3548158e80 0x7f3548158230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48044/0 (socket says 172.21.15.81:48044) 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f3546d9d640 1 -- 172.21.15.81:0/2705966060 learned_addr learned my addr 172.21.15.81:0/2705966060 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f354659c640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f354813dd70 0x7f35480aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f353dd9b640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f354815bab0 0x7f3548131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f3546d9d640 1 -- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f354815bab0 msgr2=0x7f3548131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f3546d9d640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f354815bab0 0x7f3548131c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f3546d9d640 1 -- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f354813dd70 msgr2=0x7f35480aec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:24.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f3546d9d640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f354813dd70 0x7f35480aec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:24.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f3546d9d640 1 -- 172.21.15.81:0/2705966060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35481321a0 con 0x7f3548158e80 2024-06-29T14:41:24.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f3546d9d640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3548158e80 0x7f3548158230 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f354000e960 tx=0x7f354000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:24.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f353f7fe640 1 -- 172.21.15.81:0/2705966060 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3540004d20 con 0x7f3548158e80 2024-06-29T14:41:24.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f353f7fe640 1 -- 172.21.15.81:0/2705966060 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f354000cac0 con 0x7f3548158e80 2024-06-29T14:41:24.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f353f7fe640 1 -- 172.21.15.81:0/2705966060 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3540005020 con 0x7f3548158e80 2024-06-29T14:41:24.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.866+0000 7f354cb4a640 1 -- 172.21.15.81:0/2705966060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3548157190 con 0x7f3548158e80 2024-06-29T14:41:24.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.867+0000 7f354cb4a640 1 -- 172.21.15.81:0/2705966060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3548157750 con 0x7f3548158e80 2024-06-29T14:41:24.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.868+0000 7f354cb4a640 1 -- 172.21.15.81:0/2705966060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35480cf3e0 con 0x7f3548158e80 2024-06-29T14:41:24.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.868+0000 7f353f7fe640 1 -- 172.21.15.81:0/2705966060 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f35400051c0 con 0x7f3548158e80 2024-06-29T14:41:24.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.869+0000 7f353f7fe640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f35280763f0 0x7f3528078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:24.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.869+0000 7f353f7fe640 1 -- 172.21.15.81:0/2705966060 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7f35400982b0 con 0x7f3548158e80 2024-06-29T14:41:24.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.869+0000 7f354659c640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f35280763f0 0x7f3528078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:24.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.869+0000 7f354659c640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f35280763f0 0x7f3528078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3538001720 tx=0x7f3538009040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:24.873+0000 7f353f7fe640 1 -- 172.21.15.81:0/2705966060 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3540016030 con 0x7f3548158e80 2024-06-29T14:41:25.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.090+0000 7f354cb4a640 1 -- 172.21.15.81:0/2705966060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f354806fba0 con 0x7f3548158e80 2024-06-29T14:41:25.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.091+0000 7f353f7fe640 1 -- 172.21.15.81:0/2705966060 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5228 (secure 0 0 0) 0x7f35400980d0 con 0x7f3548158e80 2024-06-29T14:41:25.090 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:25.090 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":68,"btime":"2024-06-29T14:41:24:728024+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/4097483790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":4097483790},{"type":"v1","addr":"172.21.15.167:6837","nonce":4097483790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1957730707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1957730707},{"type":"v1","addr":"172.21.15.167:6836","nonce":1957730707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":8550,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3852254611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3852254611},{"type":"v1","addr":"172.21.15.81:6833","nonce":3852254611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:21.060976+0000","modified":"2024-06-29T14:41:24.093831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8553":{"gid":8553,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/942339953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":942339953},{"type":"v1","addr":"172.21.15.81:6835","nonce":942339953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8553,"qdb_cluster":[8553]},"id":8}]} 2024-06-29T14:41:25.090 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 68 2024-06-29T14:41:25.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 -- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f35280763f0 msgr2=0x7f3528078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f35280763f0 0x7f3528078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3538001720 tx=0x7f3538009040 comp rx=0 tx=0).stop 2024-06-29T14:41:25.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 -- 172.21.15.81:0/2705966060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3548158e80 msgr2=0x7f3548158230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3548158e80 0x7f3548158230 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f354000e960 tx=0x7f354000ee20 comp rx=0 tx=0).stop 2024-06-29T14:41:25.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 -- 172.21.15.81:0/2705966060 shutdown_connections 2024-06-29T14:41:25.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f35280763f0 0x7f3528078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3548158e80 0x7f3548158230 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f354815bab0 0x7f3548131c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 --2- 172.21.15.81:0/2705966060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f354813dd70 0x7f35480aec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 -- 172.21.15.81:0/2705966060 >> 172.21.15.81:0/2705966060 conn(0x7f354806ccc0 msgr2=0x7f354813e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 -- 172.21.15.81:0/2705966060 shutdown_connections 2024-06-29T14:41:25.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.093+0000 7f354cb4a640 1 -- 172.21.15.81:0/2705966060 wait complete. 2024-06-29T14:41:25.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:25.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.217+0000 7fbe79c3c640 1 -- 172.21.15.81:0/832917925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe7406f660 msgr2=0x7fbe7406fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.217+0000 7fbe79c3c640 1 --2- 172.21.15.81:0/832917925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe7406f660 0x7fbe7406fa70 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fbe740b2430 tx=0x7fbe6c02a4d0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.217+0000 7fbe79c3c640 1 -- 172.21.15.81:0/832917925 shutdown_connections 2024-06-29T14:41:25.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.217+0000 7fbe79c3c640 1 --2- 172.21.15.81:0/832917925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe7406f660 0x7fbe7406fa70 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.217+0000 7fbe79c3c640 1 --2- 172.21.15.81:0/832917925 >> v2:172.21.15.167:3300/0 conn(0x7fbe740b4430 0x7fbe740cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.217+0000 7fbe79c3c640 1 --2- 172.21.15.81:0/832917925 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe740b35b0 0x7fbe7415b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.217+0000 7fbe79c3c640 1 -- 172.21.15.81:0/832917925 >> 172.21.15.81:0/832917925 conn(0x7fbe7406ccc0 msgr2=0x7fbe740b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.217+0000 7fbe79c3c640 1 -- 172.21.15.81:0/832917925 shutdown_connections 2024-06-29T14:41:25.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.217+0000 7fbe79c3c640 1 -- 172.21.15.81:0/832917925 wait complete. 2024-06-29T14:41:25.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.219+0000 7fbe79c3c640 1 Processor -- start 2024-06-29T14:41:25.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.219+0000 7fbe79c3c640 1 -- start start 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe79c3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe7406f660 0x7fbe741403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe79c3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe740b35b0 0x7fbe74150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe79c3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe740b4430 0x7fbe741510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe79c3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7406f160 con 0x7fbe740b4430 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe79c3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7406efe0 con 0x7fbe7406f660 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe79c3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbe7406f2e0 con 0x7fbe740b35b0 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe73fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe740b4430 0x7fbe741510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe73fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe740b4430 0x7fbe741510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48068/0 (socket says 172.21.15.81:48068) 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe73fff640 1 -- 172.21.15.81:0/3849163151 learned_addr learned my addr 172.21.15.81:0/3849163151 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe72ffd640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe740b35b0 0x7fbe74150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe737fe640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe7406f660 0x7fbe741403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe73fff640 1 -- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe740b35b0 msgr2=0x7fbe74150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe73fff640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe740b35b0 0x7fbe74150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe73fff640 1 -- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe7406f660 msgr2=0x7fbe741403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.220+0000 7fbe73fff640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe7406f660 0x7fbe741403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.221+0000 7fbe73fff640 1 -- 172.21.15.81:0/3849163151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe74144a60 con 0x7fbe740b4430 2024-06-29T14:41:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.221+0000 7fbe72ffd640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe740b35b0 0x7fbe74150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.221+0000 7fbe73fff640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe740b4430 0x7fbe741510a0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009230 tx=0x7fbe6c003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.221+0000 7fbe5bfff640 1 -- 172.21.15.81:0/3849163151 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c03e030 con 0x7fbe740b4430 2024-06-29T14:41:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.221+0000 7fbe5bfff640 1 -- 172.21.15.81:0/3849163151 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe6c003e50 con 0x7fbe740b4430 2024-06-29T14:41:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.221+0000 7fbe79c3c640 1 -- 172.21.15.81:0/3849163151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe74144d40 con 0x7fbe740b4430 2024-06-29T14:41:25.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.221+0000 7fbe5bfff640 1 -- 172.21.15.81:0/3849163151 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c035040 con 0x7fbe740b4430 2024-06-29T14:41:25.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.221+0000 7fbe79c3c640 1 -- 172.21.15.81:0/3849163151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe74146830 con 0x7fbe740b4430 2024-06-29T14:41:25.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.222+0000 7fbe79c3c640 1 -- 172.21.15.81:0/3849163151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe741447c0 con 0x7fbe740b4430 2024-06-29T14:41:25.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.223+0000 7fbe5bfff640 1 -- 172.21.15.81:0/3849163151 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fbe6c041030 con 0x7fbe740b4430 2024-06-29T14:41:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.223+0000 7fbe5bfff640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe600763f0 0x7fbe60078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.223+0000 7fbe5bfff640 1 -- 172.21.15.81:0/3849163151 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7fbe6c0b81a0 con 0x7fbe740b4430 2024-06-29T14:41:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.223+0000 7fbe737fe640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe600763f0 0x7fbe60078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.224+0000 7fbe737fe640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe600763f0 0x7fbe60078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbe64005fd0 tx=0x7fbe64009300 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.228+0000 7fbe5bfff640 1 -- 172.21.15.81:0/3849163151 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbe6c081870 con 0x7fbe740b4430 2024-06-29T14:41:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.410+0000 7fbe79c3c640 1 -- 172.21.15.81:0/3849163151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe7415b5b0 con 0x7fbe740b4430 2024-06-29T14:41:25.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.411+0000 7fbe5bfff640 1 -- 172.21.15.81:0/3849163151 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) ==== 74+0+16774 (secure 0 0 0) 0x7fbe6c081170 con 0x7fbe740b4430 2024-06-29T14:41:25.410 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:25.410 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":133,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:22.695545+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.352751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:20.014553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:25.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.413+0000 7fbe79c3c640 1 -- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe600763f0 msgr2=0x7fbe60078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.413+0000 7fbe79c3c640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe600763f0 0x7fbe60078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbe64005fd0 tx=0x7fbe64009300 comp rx=0 tx=0).stop 2024-06-29T14:41:25.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.413+0000 7fbe79c3c640 1 -- 172.21.15.81:0/3849163151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe740b4430 msgr2=0x7fbe741510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.413+0000 7fbe79c3c640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe740b4430 0x7fbe741510a0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c009230 tx=0x7fbe6c003960 comp rx=0 tx=0).stop 2024-06-29T14:41:25.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.413+0000 7fbe79c3c640 1 -- 172.21.15.81:0/3849163151 shutdown_connections 2024-06-29T14:41:25.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.413+0000 7fbe79c3c640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fbe600763f0 0x7fbe60078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.413+0000 7fbe79c3c640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe740b4430 0x7fbe741510a0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.413+0000 7fbe79c3c640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe740b35b0 0x7fbe74150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.413+0000 7fbe79c3c640 1 --2- 172.21.15.81:0/3849163151 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe7406f660 0x7fbe741403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.413+0000 7fbe79c3c640 1 -- 172.21.15.81:0/3849163151 >> 172.21.15.81:0/3849163151 conn(0x7fbe7406ccc0 msgr2=0x7fbe74071eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.414+0000 7fbe79c3c640 1 -- 172.21.15.81:0/3849163151 shutdown_connections 2024-06-29T14:41:25.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.414+0000 7fbe79c3c640 1 -- 172.21.15.81:0/3849163151 wait complete. 2024-06-29T14:41:25.423 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:41:25.424 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:41:25.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.538+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/598571228 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0d80b35b0 msgr2=0x7fb0d815b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.538+0000 7fb0dd1fc640 1 --2- 172.21.15.81:0/598571228 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0d80b35b0 0x7fb0d815b5b0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8005fd0 tx=0x7fb0c802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.538+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/598571228 shutdown_connections 2024-06-29T14:41:25.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.538+0000 7fb0dd1fc640 1 --2- 172.21.15.81:0/598571228 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0d80b4430 0x7fb0d80cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.538+0000 7fb0dd1fc640 1 --2- 172.21.15.81:0/598571228 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0d80b35b0 0x7fb0d815b5b0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.538+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/598571228 >> 172.21.15.81:0/598571228 conn(0x7fb0d806ccc0 msgr2=0x7fb0d80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.538+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/598571228 shutdown_connections 2024-06-29T14:41:25.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.538+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/598571228 wait complete. 2024-06-29T14:41:25.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.540+0000 7fb0dd1fc640 1 Processor -- start 2024-06-29T14:41:25.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.541+0000 7fb0dd1fc640 1 -- start start 2024-06-29T14:41:25.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0dd1fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0d806f660 0x7fb0d8158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0dd1fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0d80b35b0 0x7fb0d81f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0dd1fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0d80b4430 0x7fb0d81f6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0dd1fc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d8072330 con 0x7fb0d80b35b0 2024-06-29T14:41:25.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0dd1fc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d80721b0 con 0x7fb0d80b4430 2024-06-29T14:41:25.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0dd1fc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d80724b0 con 0x7fb0d806f660 2024-06-29T14:41:25.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0ce59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0d80b35b0 0x7fb0d81f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0ce59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0d80b35b0 0x7fb0d81f61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48086/0 (socket says 172.21.15.81:48086) 2024-06-29T14:41:25.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0ce59c640 1 -- 172.21.15.81:0/3938509441 learned_addr learned my addr 172.21.15.81:0/3938509441 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:25.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0d6d9d640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0d806f660 0x7fb0d8158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0ce59c640 1 -- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0d806f660 msgr2=0x7fb0d8158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0d759e640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0d80b4430 0x7fb0d81f6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0ce59c640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0d806f660 0x7fb0d8158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0ce59c640 1 -- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0d80b4430 msgr2=0x7fb0d81f6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0ce59c640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0d80b4430 0x7fb0d81f6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0ce59c640 1 -- 172.21.15.81:0/3938509441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d80a9ed0 con 0x7fb0d80b35b0 2024-06-29T14:41:25.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.542+0000 7fb0d759e640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0d80b4430 0x7fb0d81f6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:25.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.543+0000 7fb0ce59c640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0d80b35b0 0x7fb0d81f61c0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000c970 tx=0x7fb0c000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:25.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.543+0000 7fb0cffff640 1 -- 172.21.15.81:0/3938509441 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0007bb0 con 0x7fb0d80b35b0 2024-06-29T14:41:25.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.543+0000 7fb0cffff640 1 -- 172.21.15.81:0/3938509441 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0c0005020 con 0x7fb0d80b35b0 2024-06-29T14:41:25.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.543+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/3938509441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d80aa340 con 0x7fb0d80b35b0 2024-06-29T14:41:25.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.543+0000 7fb0cffff640 1 -- 172.21.15.81:0/3938509441 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0007e60 con 0x7fb0d80b35b0 2024-06-29T14:41:25.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.543+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/3938509441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d80aa900 con 0x7fb0d80b35b0 2024-06-29T14:41:25.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.544+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/3938509441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d80729a0 con 0x7fb0d80b35b0 2024-06-29T14:41:25.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.545+0000 7fb0cffff640 1 -- 172.21.15.81:0/3938509441 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fb0c0005470 con 0x7fb0d80b35b0 2024-06-29T14:41:25.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.545+0000 7fb0cffff640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fb0b80763f0 0x7fb0b8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.545+0000 7fb0cffff640 1 -- 172.21.15.81:0/3938509441 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7fb0c0097ee0 con 0x7fb0d80b35b0 2024-06-29T14:41:25.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.545+0000 7fb0d6d9d640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fb0b80763f0 0x7fb0b8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.546+0000 7fb0d6d9d640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fb0b80763f0 0x7fb0b8078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8033000 tx=0x7fb0c8004730 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.550+0000 7fb0cffff640 1 -- 172.21.15.81:0/3938509441 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb0c0014030 con 0x7fb0d80b35b0 2024-06-29T14:41:25.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.728+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/3938509441 --> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb0d80aa0a0 con 0x7fb0b80763f0 2024-06-29T14:41:25.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.734+0000 7fb0cffff640 1 -- 172.21.15.81:0/3938509441 <== mgr.8328 v2:172.21.15.167:6832/971640072 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252456 (secure 0 0 0) 0x7fb0d80aa0a0 con 0x7fb0b80763f0 2024-06-29T14:41:25.735 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:25.737 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:41:25.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fb0b80763f0 msgr2=0x7fb0b8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fb0b80763f0 0x7fb0b8078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8033000 tx=0x7fb0c8004730 comp rx=0 tx=0).stop 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/3938509441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0d80b35b0 msgr2=0x7fb0d81f61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0d80b35b0 0x7fb0d81f61c0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000c970 tx=0x7fb0c000ce30 comp rx=0 tx=0).stop 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/3938509441 shutdown_connections 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fb0b80763f0 0x7fb0b8078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0d80b4430 0x7fb0d81f6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0d80b35b0 0x7fb0d81f61c0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 --2- 172.21.15.81:0/3938509441 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0d806f660 0x7fb0d8158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/3938509441 >> 172.21.15.81:0/3938509441 conn(0x7fb0d806ccc0 msgr2=0x7fb0d815b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/3938509441 shutdown_connections 2024-06-29T14:41:25.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.737+0000 7fb0dd1fc640 1 -- 172.21.15.81:0/3938509441 wait complete. 2024-06-29T14:41:25.748 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-06-29T14:41:25.272382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":266,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":332888,"kb_used_data":14048,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754641832,"statfs":{"total":773094113280,"available":772753235968,"internally_reserved":0,"allocated":14385152,"data_stored":7501780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":326356530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.592418"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.721167+0000","last_change":"2024-06-29T14:41:21.718413+0000","last_active":"2024-06-29T14:41:21.721167+0000","last_peered":"2024-06-29T14:41:21.721167+0000","last_clean":"2024-06-29T14:41:21.721167+0000","last_became_active":"2024-06-29T14:41:21.718094+0000","last_became_peered":"2024-06-29T14:41:21.718094+0000","last_unstale":"2024-06-29T14:41:21.721167+0000","last_undegraded":"2024-06-29T14:41:21.721167+0000","last_fullsized":"2024-06-29T14:41:21.721167+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:48:14.512858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703045+0000","last_change":"2024-06-29T14:41:21.715818+0000","last_active":"2024-06-29T14:41:22.703045+0000","last_peered":"2024-06-29T14:41:22.703045+0000","last_clean":"2024-06-29T14:41:22.703045+0000","last_became_active":"2024-06-29T14:41:21.712890+0000","last_became_peered":"2024-06-29T14:41:21.712890+0000","last_unstale":"2024-06-29T14:41:22.703045+0000","last_undegraded":"2024-06-29T14:41:22.703045+0000","last_fullsized":"2024-06-29T14:41:22.703045+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:44.889628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253462+0000","last_change":"2024-06-29T14:41:21.724779+0000","last_active":"2024-06-29T14:41:23.253462+0000","last_peered":"2024-06-29T14:41:23.253462+0000","last_clean":"2024-06-29T14:41:23.253462+0000","last_became_active":"2024-06-29T14:41:21.724463+0000","last_became_peered":"2024-06-29T14:41:21.724463+0000","last_unstale":"2024-06-29T14:41:23.253462+0000","last_undegraded":"2024-06-29T14:41:23.253462+0000","last_fullsized":"2024-06-29T14:41:23.253462+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:49.679475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.723276+0000","last_change":"2024-06-29T14:41:21.718741+0000","last_active":"2024-06-29T14:41:21.723276+0000","last_peered":"2024-06-29T14:41:21.723276+0000","last_clean":"2024-06-29T14:41:21.723276+0000","last_became_active":"2024-06-29T14:41:21.718227+0000","last_became_peered":"2024-06-29T14:41:21.718227+0000","last_unstale":"2024-06-29T14:41:21.723276+0000","last_undegraded":"2024-06-29T14:41:21.723276+0000","last_fullsized":"2024-06-29T14:41:21.723276+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:20:50.967368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.726522+0000","last_change":"2024-06-29T14:41:21.717137+0000","last_active":"2024-06-29T14:41:21.726522+0000","last_peered":"2024-06-29T14:41:21.726522+0000","last_clean":"2024-06-29T14:41:21.726522+0000","last_became_active":"2024-06-29T14:41:21.716835+0000","last_became_peered":"2024-06-29T14:41:21.716835+0000","last_unstale":"2024-06-29T14:41:21.726522+0000","last_undegraded":"2024-06-29T14:41:21.726522+0000","last_fullsized":"2024-06-29T14:41:21.726522+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:26.645847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699490+0000","last_change":"2024-06-29T14:41:21.712324+0000","last_active":"2024-06-29T14:41:22.699490+0000","last_peered":"2024-06-29T14:41:22.699490+0000","last_clean":"2024-06-29T14:41:22.699490+0000","last_became_active":"2024-06-29T14:41:21.712031+0000","last_became_peered":"2024-06-29T14:41:21.712031+0000","last_unstale":"2024-06-29T14:41:22.699490+0000","last_undegraded":"2024-06-29T14:41:22.699490+0000","last_fullsized":"2024-06-29T14:41:22.699490+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:38.450632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703274+0000","last_change":"2024-06-29T14:41:21.714258+0000","last_active":"2024-06-29T14:41:22.703274+0000","last_peered":"2024-06-29T14:41:22.703274+0000","last_clean":"2024-06-29T14:41:22.703274+0000","last_became_active":"2024-06-29T14:41:21.713821+0000","last_became_peered":"2024-06-29T14:41:21.713821+0000","last_unstale":"2024-06-29T14:41:22.703274+0000","last_undegraded":"2024-06-29T14:41:22.703274+0000","last_fullsized":"2024-06-29T14:41:22.703274+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:23.229276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253221+0000","last_change":"2024-06-29T14:41:21.726845+0000","last_active":"2024-06-29T14:41:23.253221+0000","last_peered":"2024-06-29T14:41:23.253221+0000","last_clean":"2024-06-29T14:41:23.253221+0000","last_became_active":"2024-06-29T14:41:21.726498+0000","last_became_peered":"2024-06-29T14:41:21.726498+0000","last_unstale":"2024-06-29T14:41:23.253221+0000","last_undegraded":"2024-06-29T14:41:23.253221+0000","last_fullsized":"2024-06-29T14:41:23.253221+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:36:23.413101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.729342+0000","last_change":"2024-06-29T14:41:21.724593+0000","last_active":"2024-06-29T14:41:21.729342+0000","last_peered":"2024-06-29T14:41:21.729342+0000","last_clean":"2024-06-29T14:41:21.729342+0000","last_became_active":"2024-06-29T14:41:21.724002+0000","last_became_peered":"2024-06-29T14:41:21.724002+0000","last_unstale":"2024-06-29T14:41:21.729342+0000","last_undegraded":"2024-06-29T14:41:21.729342+0000","last_fullsized":"2024-06-29T14:41:21.729342+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:38.251709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.721456+0000","last_change":"2024-06-29T14:41:21.716148+0000","last_active":"2024-06-29T14:41:21.721456+0000","last_peered":"2024-06-29T14:41:21.721456+0000","last_clean":"2024-06-29T14:41:21.721456+0000","last_became_active":"2024-06-29T14:41:21.715751+0000","last_became_peered":"2024-06-29T14:41:21.715751+0000","last_unstale":"2024-06-29T14:41:21.721456+0000","last_undegraded":"2024-06-29T14:41:21.721456+0000","last_fullsized":"2024-06-29T14:41:21.721456+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:48.236647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703218+0000","last_change":"2024-06-29T14:41:21.717671+0000","last_active":"2024-06-29T14:41:22.703218+0000","last_peered":"2024-06-29T14:41:22.703218+0000","last_clean":"2024-06-29T14:41:22.703218+0000","last_became_active":"2024-06-29T14:41:21.716553+0000","last_became_peered":"2024-06-29T14:41:21.716553+0000","last_unstale":"2024-06-29T14:41:22.703218+0000","last_undegraded":"2024-06-29T14:41:22.703218+0000","last_fullsized":"2024-06-29T14:41:22.703218+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:49.527582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.723301+0000","last_change":"2024-06-29T14:41:21.718686+0000","last_active":"2024-06-29T14:41:21.723301+0000","last_peered":"2024-06-29T14:41:21.723301+0000","last_clean":"2024-06-29T14:41:21.723301+0000","last_became_active":"2024-06-29T14:41:21.717887+0000","last_became_peered":"2024-06-29T14:41:21.717887+0000","last_unstale":"2024-06-29T14:41:21.723301+0000","last_undegraded":"2024-06-29T14:41:21.723301+0000","last_fullsized":"2024-06-29T14:41:21.723301+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:19:18.761595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253299+0000","last_change":"2024-06-29T14:41:21.728555+0000","last_active":"2024-06-29T14:41:23.253299+0000","last_peered":"2024-06-29T14:41:23.253299+0000","last_clean":"2024-06-29T14:41:23.253299+0000","last_became_active":"2024-06-29T14:41:21.727680+0000","last_became_peered":"2024-06-29T14:41:21.727680+0000","last_unstale":"2024-06-29T14:41:23.253299+0000","last_undegraded":"2024-06-29T14:41:23.253299+0000","last_fullsized":"2024-06-29T14:41:23.253299+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:07:14.151770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253191+0000","last_change":"2024-06-29T14:41:21.725694+0000","last_active":"2024-06-29T14:41:23.253191+0000","last_peered":"2024-06-29T14:41:23.253191+0000","last_clean":"2024-06-29T14:41:23.253191+0000","last_became_active":"2024-06-29T14:41:21.725380+0000","last_became_peered":"2024-06-29T14:41:21.725380+0000","last_unstale":"2024-06-29T14:41:23.253191+0000","last_undegraded":"2024-06-29T14:41:23.253191+0000","last_fullsized":"2024-06-29T14:41:23.253191+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:43:57.185593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699412+0000","last_change":"2024-06-29T14:41:21.713671+0000","last_active":"2024-06-29T14:41:22.699412+0000","last_peered":"2024-06-29T14:41:22.699412+0000","last_clean":"2024-06-29T14:41:22.699412+0000","last_became_active":"2024-06-29T14:41:21.713364+0000","last_became_peered":"2024-06-29T14:41:21.713364+0000","last_unstale":"2024-06-29T14:41:22.699412+0000","last_undegraded":"2024-06-29T14:41:22.699412+0000","last_fullsized":"2024-06-29T14:41:22.699412+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:51.203732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.729314+0000","last_change":"2024-06-29T14:41:21.724153+0000","last_active":"2024-06-29T14:41:21.729314+0000","last_peered":"2024-06-29T14:41:21.729314+0000","last_clean":"2024-06-29T14:41:21.729314+0000","last_became_active":"2024-06-29T14:41:21.723841+0000","last_became_peered":"2024-06-29T14:41:21.723841+0000","last_unstale":"2024-06-29T14:41:21.729314+0000","last_undegraded":"2024-06-29T14:41:21.729314+0000","last_fullsized":"2024-06-29T14:41:21.729314+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:24.890464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703637+0000","last_change":"2024-06-29T14:41:21.716298+0000","last_active":"2024-06-29T14:41:22.703637+0000","last_peered":"2024-06-29T14:41:22.703637+0000","last_clean":"2024-06-29T14:41:22.703637+0000","last_became_active":"2024-06-29T14:41:21.715651+0000","last_became_peered":"2024-06-29T14:41:21.715651+0000","last_unstale":"2024-06-29T14:41:22.703637+0000","last_undegraded":"2024-06-29T14:41:22.703637+0000","last_fullsized":"2024-06-29T14:41:22.703637+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:10.650320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717627+0000","last_change":"2024-06-29T14:41:21.712006+0000","last_active":"2024-06-29T14:41:22.717627+0000","last_peered":"2024-06-29T14:41:22.717627+0000","last_clean":"2024-06-29T14:41:22.717627+0000","last_became_active":"2024-06-29T14:41:21.711631+0000","last_became_peered":"2024-06-29T14:41:21.711631+0000","last_unstale":"2024-06-29T14:41:22.717627+0000","last_undegraded":"2024-06-29T14:41:22.717627+0000","last_fullsized":"2024-06-29T14:41:22.717627+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:41:21.021074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253682+0000","last_change":"2024-06-29T14:41:21.728181+0000","last_active":"2024-06-29T14:41:23.253682+0000","last_peered":"2024-06-29T14:41:23.253682+0000","last_clean":"2024-06-29T14:41:23.253682+0000","last_became_active":"2024-06-29T14:41:21.726987+0000","last_became_peered":"2024-06-29T14:41:21.726987+0000","last_unstale":"2024-06-29T14:41:23.253682+0000","last_undegraded":"2024-06-29T14:41:23.253682+0000","last_fullsized":"2024-06-29T14:41:23.253682+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:21.510978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703164+0000","last_change":"2024-06-29T14:41:21.715650+0000","last_active":"2024-06-29T14:41:22.703164+0000","last_peered":"2024-06-29T14:41:22.703164+0000","last_clean":"2024-06-29T14:41:22.703164+0000","last_became_active":"2024-06-29T14:41:21.715311+0000","last_became_peered":"2024-06-29T14:41:21.715311+0000","last_unstale":"2024-06-29T14:41:22.703164+0000","last_undegraded":"2024-06-29T14:41:22.703164+0000","last_fullsized":"2024-06-29T14:41:22.703164+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:26.393604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703167+0000","last_change":"2024-06-29T14:41:21.724101+0000","last_active":"2024-06-29T14:41:22.703167+0000","last_peered":"2024-06-29T14:41:22.703167+0000","last_clean":"2024-06-29T14:41:22.703167+0000","last_became_active":"2024-06-29T14:41:21.723706+0000","last_became_peered":"2024-06-29T14:41:21.723706+0000","last_unstale":"2024-06-29T14:41:22.703167+0000","last_undegraded":"2024-06-29T14:41:22.703167+0000","last_fullsized":"2024-06-29T14:41:22.703167+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:46.059791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.722357+0000","last_change":"2024-06-29T14:41:21.717148+0000","last_active":"2024-06-29T14:41:21.722357+0000","last_peered":"2024-06-29T14:41:21.722357+0000","last_clean":"2024-06-29T14:41:21.722357+0000","last_became_active":"2024-06-29T14:41:21.716847+0000","last_became_peered":"2024-06-29T14:41:21.716847+0000","last_unstale":"2024-06-29T14:41:21.722357+0000","last_undegraded":"2024-06-29T14:41:21.722357+0000","last_fullsized":"2024-06-29T14:41:21.722357+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:15:49.783155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703348+0000","last_change":"2024-06-29T14:41:21.712619+0000","last_active":"2024-06-29T14:41:22.703348+0000","last_peered":"2024-06-29T14:41:22.703348+0000","last_clean":"2024-06-29T14:41:22.703348+0000","last_became_active":"2024-06-29T14:41:21.712307+0000","last_became_peered":"2024-06-29T14:41:21.712307+0000","last_unstale":"2024-06-29T14:41:22.703348+0000","last_undegraded":"2024-06-29T14:41:22.703348+0000","last_fullsized":"2024-06-29T14:41:22.703348+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:49:55.208054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.726207+0000","last_change":"2024-06-29T14:41:21.721707+0000","last_active":"2024-06-29T14:41:21.726207+0000","last_peered":"2024-06-29T14:41:21.726207+0000","last_clean":"2024-06-29T14:41:21.726207+0000","last_became_active":"2024-06-29T14:41:21.721419+0000","last_became_peered":"2024-06-29T14:41:21.721419+0000","last_unstale":"2024-06-29T14:41:21.726207+0000","last_undegraded":"2024-06-29T14:41:21.726207+0000","last_fullsized":"2024-06-29T14:41:21.726207+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:22.389762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703838+0000","last_change":"2024-06-29T14:41:21.714731+0000","last_active":"2024-06-29T14:41:22.703838+0000","last_peered":"2024-06-29T14:41:22.703838+0000","last_clean":"2024-06-29T14:41:22.703838+0000","last_became_active":"2024-06-29T14:41:21.713275+0000","last_became_peered":"2024-06-29T14:41:21.713275+0000","last_unstale":"2024-06-29T14:41:22.703838+0000","last_undegraded":"2024-06-29T14:41:22.703838+0000","last_fullsized":"2024-06-29T14:41:22.703838+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:56.293327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.721236+0000","last_change":"2024-06-29T14:41:21.715999+0000","last_active":"2024-06-29T14:41:21.721236+0000","last_peered":"2024-06-29T14:41:21.721236+0000","last_clean":"2024-06-29T14:41:21.721236+0000","last_became_active":"2024-06-29T14:41:21.714846+0000","last_became_peered":"2024-06-29T14:41:21.714846+0000","last_unstale":"2024-06-29T14:41:21.721236+0000","last_undegraded":"2024-06-29T14:41:21.721236+0000","last_fullsized":"2024-06-29T14:41:21.721236+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:27.644710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699875+0000","last_change":"2024-06-29T14:41:21.715393+0000","last_active":"2024-06-29T14:41:22.699875+0000","last_peered":"2024-06-29T14:41:22.699875+0000","last_clean":"2024-06-29T14:41:22.699875+0000","last_became_active":"2024-06-29T14:41:21.715092+0000","last_became_peered":"2024-06-29T14:41:21.715092+0000","last_unstale":"2024-06-29T14:41:22.699875+0000","last_undegraded":"2024-06-29T14:41:22.699875+0000","last_fullsized":"2024-06-29T14:41:22.699875+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:26.645847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.729014+0000","last_change":"2024-06-29T14:41:21.725740+0000","last_active":"2024-06-29T14:41:21.729014+0000","last_peered":"2024-06-29T14:41:21.729014+0000","last_clean":"2024-06-29T14:41:21.729014+0000","last_became_active":"2024-06-29T14:41:21.725408+0000","last_became_peered":"2024-06-29T14:41:21.725408+0000","last_unstale":"2024-06-29T14:41:21.729014+0000","last_undegraded":"2024-06-29T14:41:21.729014+0000","last_fullsized":"2024-06-29T14:41:21.729014+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:43:28.973414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702773+0000","last_change":"2024-06-29T14:41:21.720366+0000","last_active":"2024-06-29T14:41:22.702773+0000","last_peered":"2024-06-29T14:41:22.702773+0000","last_clean":"2024-06-29T14:41:22.702773+0000","last_became_active":"2024-06-29T14:41:21.720014+0000","last_became_peered":"2024-06-29T14:41:21.720014+0000","last_unstale":"2024-06-29T14:41:22.702773+0000","last_undegraded":"2024-06-29T14:41:22.702773+0000","last_fullsized":"2024-06-29T14:41:22.702773+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:05.852614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703103+0000","last_change":"2024-06-29T14:41:21.713240+0000","last_active":"2024-06-29T14:41:22.703103+0000","last_peered":"2024-06-29T14:41:22.703103+0000","last_clean":"2024-06-29T14:41:22.703103+0000","last_became_active":"2024-06-29T14:41:21.712700+0000","last_became_peered":"2024-06-29T14:41:21.712700+0000","last_unstale":"2024-06-29T14:41:22.703103+0000","last_undegraded":"2024-06-29T14:41:22.703103+0000","last_fullsized":"2024-06-29T14:41:22.703103+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:56.293327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.714667+0000","last_change":"2024-06-29T14:41:21.712793+0000","last_active":"2024-06-29T14:41:21.714667+0000","last_peered":"2024-06-29T14:41:21.714667+0000","last_clean":"2024-06-29T14:41:21.714667+0000","last_became_active":"2024-06-29T14:41:21.712472+0000","last_became_peered":"2024-06-29T14:41:21.712472+0000","last_unstale":"2024-06-29T14:41:21.714667+0000","last_undegraded":"2024-06-29T14:41:21.714667+0000","last_fullsized":"2024-06-29T14:41:21.714667+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:12:00.463362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699383+0000","last_change":"2024-06-29T14:41:21.713048+0000","last_active":"2024-06-29T14:41:22.699383+0000","last_peered":"2024-06-29T14:41:22.699383+0000","last_clean":"2024-06-29T14:41:22.699383+0000","last_became_active":"2024-06-29T14:41:21.712768+0000","last_became_peered":"2024-06-29T14:41:21.712768+0000","last_unstale":"2024-06-29T14:41:22.699383+0000","last_undegraded":"2024-06-29T14:41:22.699383+0000","last_fullsized":"2024-06-29T14:41:22.699383+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:44.001462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699863+0000","last_change":"2024-06-29T14:41:21.708246+0000","last_active":"2024-06-29T14:41:22.699863+0000","last_peered":"2024-06-29T14:41:22.699863+0000","last_clean":"2024-06-29T14:41:22.699863+0000","last_became_active":"2024-06-29T14:41:21.707942+0000","last_became_peered":"2024-06-29T14:41:21.707942+0000","last_unstale":"2024-06-29T14:41:22.699863+0000","last_undegraded":"2024-06-29T14:41:22.699863+0000","last_fullsized":"2024-06-29T14:41:22.699863+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:06:12.716366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"132'4","reported_seq":18,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703452+0000","last_change":"2024-06-29T14:41:20.701718+0000","last_active":"2024-06-29T14:41:22.703452+0000","last_peered":"2024-06-29T14:41:22.703452+0000","last_clean":"2024-06-29T14:41:22.703452+0000","last_became_active":"2024-06-29T14:41:20.701395+0000","last_became_peered":"2024-06-29T14:41:20.701395+0000","last_unstale":"2024-06-29T14:41:22.703452+0000","last_undegraded":"2024-06-29T14:41:22.703452+0000","last_fullsized":"2024-06-29T14:41:22.703452+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:21.857684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717693+0000","last_change":"2024-06-29T14:41:21.717954+0000","last_active":"2024-06-29T14:41:22.717693+0000","last_peered":"2024-06-29T14:41:22.717693+0000","last_clean":"2024-06-29T14:41:22.717693+0000","last_became_active":"2024-06-29T14:41:21.717583+0000","last_became_peered":"2024-06-29T14:41:21.717583+0000","last_unstale":"2024-06-29T14:41:22.717693+0000","last_undegraded":"2024-06-29T14:41:22.717693+0000","last_fullsized":"2024-06-29T14:41:22.717693+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:35:26.174541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"132'1","reported_seq":15,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253628+0000","last_change":"2024-06-29T14:41:20.704760+0000","last_active":"2024-06-29T14:41:23.253628+0000","last_peered":"2024-06-29T14:41:23.253628+0000","last_clean":"2024-06-29T14:41:23.253628+0000","last_became_active":"2024-06-29T14:41:20.704443+0000","last_became_peered":"2024-06-29T14:41:20.704443+0000","last_unstale":"2024-06-29T14:41:23.253628+0000","last_undegraded":"2024-06-29T14:41:23.253628+0000","last_fullsized":"2024-06-29T14:41:23.253628+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:30:46.569231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699691+0000","last_change":"2024-06-29T14:41:21.713662+0000","last_active":"2024-06-29T14:41:22.699691+0000","last_peered":"2024-06-29T14:41:22.699691+0000","last_clean":"2024-06-29T14:41:22.699691+0000","last_became_active":"2024-06-29T14:41:21.713301+0000","last_became_peered":"2024-06-29T14:41:21.713301+0000","last_unstale":"2024-06-29T14:41:22.699691+0000","last_undegraded":"2024-06-29T14:41:22.699691+0000","last_fullsized":"2024-06-29T14:41:22.699691+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:54.601899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"132'2","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.730130+0000","last_change":"2024-06-29T14:41:20.702170+0000","last_active":"2024-06-29T14:41:21.730130+0000","last_peered":"2024-06-29T14:41:21.730130+0000","last_clean":"2024-06-29T14:41:21.730130+0000","last_became_active":"2024-06-29T14:41:20.701695+0000","last_became_peered":"2024-06-29T14:41:20.701695+0000","last_unstale":"2024-06-29T14:41:21.730130+0000","last_undegraded":"2024-06-29T14:41:21.730130+0000","last_fullsized":"2024-06-29T14:41:21.730130+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:53.593773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253694+0000","last_change":"2024-06-29T14:41:21.728453+0000","last_active":"2024-06-29T14:41:23.253694+0000","last_peered":"2024-06-29T14:41:23.253694+0000","last_clean":"2024-06-29T14:41:23.253694+0000","last_became_active":"2024-06-29T14:41:21.727640+0000","last_became_peered":"2024-06-29T14:41:21.727640+0000","last_unstale":"2024-06-29T14:41:23.253694+0000","last_undegraded":"2024-06-29T14:41:23.253694+0000","last_fullsized":"2024-06-29T14:41:23.253694+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:45:50.663167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"132'2","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.730202+0000","last_change":"2024-06-29T14:41:20.702028+0000","last_active":"2024-06-29T14:41:21.730202+0000","last_peered":"2024-06-29T14:41:21.730202+0000","last_clean":"2024-06-29T14:41:21.730202+0000","last_became_active":"2024-06-29T14:41:20.701649+0000","last_became_peered":"2024-06-29T14:41:20.701649+0000","last_unstale":"2024-06-29T14:41:21.730202+0000","last_undegraded":"2024-06-29T14:41:21.730202+0000","last_fullsized":"2024-06-29T14:41:21.730202+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:50.195605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.729282+0000","last_change":"2024-06-29T14:41:21.724822+0000","last_active":"2024-06-29T14:41:21.729282+0000","last_peered":"2024-06-29T14:41:21.729282+0000","last_clean":"2024-06-29T14:41:21.729282+0000","last_became_active":"2024-06-29T14:41:21.724519+0000","last_became_peered":"2024-06-29T14:41:21.724519+0000","last_unstale":"2024-06-29T14:41:21.729282+0000","last_undegraded":"2024-06-29T14:41:21.729282+0000","last_fullsized":"2024-06-29T14:41:21.729282+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:05.852614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"132'1","reported_seq":15,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703324+0000","last_change":"2024-06-29T14:41:20.696598+0000","last_active":"2024-06-29T14:41:22.703324+0000","last_peered":"2024-06-29T14:41:22.703324+0000","last_clean":"2024-06-29T14:41:22.703324+0000","last_became_active":"2024-06-29T14:41:20.696283+0000","last_became_peered":"2024-06-29T14:41:20.696283+0000","last_unstale":"2024-06-29T14:41:22.703324+0000","last_undegraded":"2024-06-29T14:41:22.703324+0000","last_fullsized":"2024-06-29T14:41:22.703324+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:37.243582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702872+0000","last_change":"2024-06-29T14:41:21.717143+0000","last_active":"2024-06-29T14:41:22.702872+0000","last_peered":"2024-06-29T14:41:22.702872+0000","last_clean":"2024-06-29T14:41:22.702872+0000","last_became_active":"2024-06-29T14:41:21.715351+0000","last_became_peered":"2024-06-29T14:41:21.715351+0000","last_unstale":"2024-06-29T14:41:22.702872+0000","last_undegraded":"2024-06-29T14:41:22.702872+0000","last_fullsized":"2024-06-29T14:41:22.702872+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:22.865811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"132'1","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.730588+0000","last_change":"2024-06-29T14:41:20.708482+0000","last_active":"2024-06-29T14:41:21.730588+0000","last_peered":"2024-06-29T14:41:21.730588+0000","last_clean":"2024-06-29T14:41:21.730588+0000","last_became_active":"2024-06-29T14:41:20.707516+0000","last_became_peered":"2024-06-29T14:41:20.707516+0000","last_unstale":"2024-06-29T14:41:21.730588+0000","last_undegraded":"2024-06-29T14:41:21.730588+0000","last_fullsized":"2024-06-29T14:41:21.730588+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:40.873222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.714211+0000","last_change":"2024-06-29T14:41:21.710694+0000","last_active":"2024-06-29T14:41:21.714211+0000","last_peered":"2024-06-29T14:41:21.714211+0000","last_clean":"2024-06-29T14:41:21.714211+0000","last_became_active":"2024-06-29T14:41:21.710379+0000","last_became_peered":"2024-06-29T14:41:21.710379+0000","last_unstale":"2024-06-29T14:41:21.714211+0000","last_undegraded":"2024-06-29T14:41:21.714211+0000","last_fullsized":"2024-06-29T14:41:21.714211+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:11.602177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702521+0000","last_change":"2024-06-29T14:41:20.688482+0000","last_active":"2024-06-29T14:41:22.702521+0000","last_peered":"2024-06-29T14:41:22.702521+0000","last_clean":"2024-06-29T14:41:22.702521+0000","last_became_active":"2024-06-29T14:41:20.688141+0000","last_became_peered":"2024-06-29T14:41:20.688141+0000","last_unstale":"2024-06-29T14:41:22.702521+0000","last_undegraded":"2024-06-29T14:41:22.702521+0000","last_fullsized":"2024-06-29T14:41:22.702521+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:48.550307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702563+0000","last_change":"2024-06-29T14:41:21.714560+0000","last_active":"2024-06-29T14:41:22.702563+0000","last_peered":"2024-06-29T14:41:22.702563+0000","last_clean":"2024-06-29T14:41:22.702563+0000","last_became_active":"2024-06-29T14:41:21.714245+0000","last_became_peered":"2024-06-29T14:41:21.714245+0000","last_unstale":"2024-06-29T14:41:22.702563+0000","last_undegraded":"2024-06-29T14:41:22.702563+0000","last_fullsized":"2024-06-29T14:41:22.702563+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:10.650320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.710030+0000","last_change":"2024-06-29T14:41:20.707672+0000","last_active":"2024-06-29T14:41:21.710030+0000","last_peered":"2024-06-29T14:41:21.710030+0000","last_clean":"2024-06-29T14:41:21.710030+0000","last_became_active":"2024-06-29T14:41:20.706986+0000","last_became_peered":"2024-06-29T14:41:20.706986+0000","last_unstale":"2024-06-29T14:41:21.710030+0000","last_undegraded":"2024-06-29T14:41:21.710030+0000","last_fullsized":"2024-06-29T14:41:21.710030+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:57.987710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703200+0000","last_change":"2024-06-29T14:41:21.709498+0000","last_active":"2024-06-29T14:41:22.703200+0000","last_peered":"2024-06-29T14:41:22.703200+0000","last_clean":"2024-06-29T14:41:22.703200+0000","last_became_active":"2024-06-29T14:41:21.709173+0000","last_became_peered":"2024-06-29T14:41:21.709173+0000","last_unstale":"2024-06-29T14:41:22.703200+0000","last_undegraded":"2024-06-29T14:41:22.703200+0000","last_fullsized":"2024-06-29T14:41:22.703200+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:55.550304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.701052+0000","last_change":"2024-06-29T14:41:20.701399+0000","last_active":"2024-06-29T14:41:21.701052+0000","last_peered":"2024-06-29T14:41:21.701052+0000","last_clean":"2024-06-29T14:41:21.701052+0000","last_became_active":"2024-06-29T14:41:20.701067+0000","last_became_peered":"2024-06-29T14:41:20.701067+0000","last_unstale":"2024-06-29T14:41:21.701052+0000","last_undegraded":"2024-06-29T14:41:21.701052+0000","last_fullsized":"2024-06-29T14:41:21.701052+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:37.442505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717687+0000","last_change":"2024-06-29T14:41:21.722165+0000","last_active":"2024-06-29T14:41:22.717687+0000","last_peered":"2024-06-29T14:41:22.717687+0000","last_clean":"2024-06-29T14:41:22.717687+0000","last_became_active":"2024-06-29T14:41:21.721619+0000","last_became_peered":"2024-06-29T14:41:21.721619+0000","last_unstale":"2024-06-29T14:41:22.717687+0000","last_undegraded":"2024-06-29T14:41:22.717687+0000","last_fullsized":"2024-06-29T14:41:22.717687+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:29.214902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.702461+0000","last_change":"2024-06-29T14:41:20.707863+0000","last_active":"2024-06-29T14:41:21.702461+0000","last_peered":"2024-06-29T14:41:21.702461+0000","last_clean":"2024-06-29T14:41:21.702461+0000","last_became_active":"2024-06-29T14:41:20.707545+0000","last_became_peered":"2024-06-29T14:41:20.707545+0000","last_unstale":"2024-06-29T14:41:21.702461+0000","last_undegraded":"2024-06-29T14:41:21.702461+0000","last_fullsized":"2024-06-29T14:41:21.702461+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:59.291183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.716256+0000","last_change":"2024-06-29T14:41:21.709146+0000","last_active":"2024-06-29T14:41:21.716256+0000","last_peered":"2024-06-29T14:41:21.716256+0000","last_clean":"2024-06-29T14:41:21.716256+0000","last_became_active":"2024-06-29T14:41:21.708860+0000","last_became_peered":"2024-06-29T14:41:21.708860+0000","last_unstale":"2024-06-29T14:41:21.716256+0000","last_undegraded":"2024-06-29T14:41:21.716256+0000","last_fullsized":"2024-06-29T14:41:21.716256+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:07.037259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"132'3","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.736364+0000","last_change":"2024-06-29T14:41:20.705246+0000","last_active":"2024-06-29T14:41:21.736364+0000","last_peered":"2024-06-29T14:41:21.736364+0000","last_clean":"2024-06-29T14:41:21.736364+0000","last_became_active":"2024-06-29T14:41:20.704766+0000","last_became_peered":"2024-06-29T14:41:20.704766+0000","last_unstale":"2024-06-29T14:41:21.736364+0000","last_undegraded":"2024-06-29T14:41:21.736364+0000","last_fullsized":"2024-06-29T14:41:21.736364+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:48.550307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703587+0000","last_change":"2024-06-29T14:41:21.719216+0000","last_active":"2024-06-29T14:41:22.703587+0000","last_peered":"2024-06-29T14:41:22.703587+0000","last_clean":"2024-06-29T14:41:22.703587+0000","last_became_active":"2024-06-29T14:41:21.718377+0000","last_became_peered":"2024-06-29T14:41:21.718377+0000","last_unstale":"2024-06-29T14:41:22.703587+0000","last_undegraded":"2024-06-29T14:41:22.703587+0000","last_fullsized":"2024-06-29T14:41:22.703587+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:01.789064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"132'6","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.738915+0000","last_change":"2024-06-29T14:41:20.701417+0000","last_active":"2024-06-29T14:41:21.738915+0000","last_peered":"2024-06-29T14:41:21.738915+0000","last_clean":"2024-06-29T14:41:21.738915+0000","last_became_active":"2024-06-29T14:41:20.701113+0000","last_became_peered":"2024-06-29T14:41:20.701113+0000","last_unstale":"2024-06-29T14:41:21.738915+0000","last_undegraded":"2024-06-29T14:41:21.738915+0000","last_fullsized":"2024-06-29T14:41:21.738915+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:35:25.166414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.716308+0000","last_change":"2024-06-29T14:41:21.706413+0000","last_active":"2024-06-29T14:41:21.716308+0000","last_peered":"2024-06-29T14:41:21.716308+0000","last_clean":"2024-06-29T14:41:21.716308+0000","last_became_active":"2024-06-29T14:41:21.706067+0000","last_became_peered":"2024-06-29T14:41:21.706067+0000","last_unstale":"2024-06-29T14:41:21.716308+0000","last_undegraded":"2024-06-29T14:41:21.716308+0000","last_fullsized":"2024-06-29T14:41:21.716308+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:43:44.068355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"132'2","reported_seq":16,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699634+0000","last_change":"2024-06-29T14:41:20.703226+0000","last_active":"2024-06-29T14:41:22.699634+0000","last_peered":"2024-06-29T14:41:22.699634+0000","last_clean":"2024-06-29T14:41:22.699634+0000","last_became_active":"2024-06-29T14:41:20.702875+0000","last_became_peered":"2024-06-29T14:41:20.702875+0000","last_unstale":"2024-06-29T14:41:22.699634+0000","last_undegraded":"2024-06-29T14:41:22.699634+0000","last_fullsized":"2024-06-29T14:41:22.699634+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:19:10.558384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.724209+0000","last_change":"2024-06-29T14:41:21.718234+0000","last_active":"2024-06-29T14:41:21.724209+0000","last_peered":"2024-06-29T14:41:21.724209+0000","last_clean":"2024-06-29T14:41:21.724209+0000","last_became_active":"2024-06-29T14:41:21.717729+0000","last_became_peered":"2024-06-29T14:41:21.717729+0000","last_unstale":"2024-06-29T14:41:21.724209+0000","last_undegraded":"2024-06-29T14:41:21.724209+0000","last_fullsized":"2024-06-29T14:41:21.724209+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:53.358891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717645+0000","last_change":"2024-06-29T14:41:20.706496+0000","last_active":"2024-06-29T14:41:22.717645+0000","last_peered":"2024-06-29T14:41:22.717645+0000","last_clean":"2024-06-29T14:41:22.717645+0000","last_became_active":"2024-06-29T14:41:20.706193+0000","last_became_peered":"2024-06-29T14:41:20.706193+0000","last_unstale":"2024-06-29T14:41:22.717645+0000","last_undegraded":"2024-06-29T14:41:22.717645+0000","last_fullsized":"2024-06-29T14:41:22.717645+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:00.632847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.721589+0000","last_change":"2024-06-29T14:41:21.717248+0000","last_active":"2024-06-29T14:41:21.721589+0000","last_peered":"2024-06-29T14:41:21.721589+0000","last_clean":"2024-06-29T14:41:21.721589+0000","last_became_active":"2024-06-29T14:41:21.716831+0000","last_became_peered":"2024-06-29T14:41:21.716831+0000","last_unstale":"2024-06-29T14:41:21.721589+0000","last_undegraded":"2024-06-29T14:41:21.721589+0000","last_fullsized":"2024-06-29T14:41:21.721589+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:07:32.418246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"132'8","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.731867+0000","last_change":"2024-06-29T14:41:20.708590+0000","last_active":"2024-06-29T14:41:21.731867+0000","last_peered":"2024-06-29T14:41:21.731867+0000","last_clean":"2024-06-29T14:41:21.731867+0000","last_became_active":"2024-06-29T14:41:20.708250+0000","last_became_peered":"2024-06-29T14:41:20.708250+0000","last_unstale":"2024-06-29T14:41:21.731867+0000","last_undegraded":"2024-06-29T14:41:21.731867+0000","last_fullsized":"2024-06-29T14:41:21.731867+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:37:03.084146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.726652+0000","last_change":"2024-06-29T14:41:21.718198+0000","last_active":"2024-06-29T14:41:21.726652+0000","last_peered":"2024-06-29T14:41:21.726652+0000","last_clean":"2024-06-29T14:41:21.726652+0000","last_became_active":"2024-06-29T14:41:21.717746+0000","last_became_peered":"2024-06-29T14:41:21.717746+0000","last_unstale":"2024-06-29T14:41:21.726652+0000","last_undegraded":"2024-06-29T14:41:21.726652+0000","last_fullsized":"2024-06-29T14:41:21.726652+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:01.640974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"132'1","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.731691+0000","last_change":"2024-06-29T14:41:20.706945+0000","last_active":"2024-06-29T14:41:21.731691+0000","last_peered":"2024-06-29T14:41:21.731691+0000","last_clean":"2024-06-29T14:41:21.731691+0000","last_became_active":"2024-06-29T14:41:20.704685+0000","last_became_peered":"2024-06-29T14:41:20.704685+0000","last_unstale":"2024-06-29T14:41:21.731691+0000","last_undegraded":"2024-06-29T14:41:21.731691+0000","last_fullsized":"2024-06-29T14:41:21.731691+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:04.460229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699905+0000","last_change":"2024-06-29T14:41:21.715301+0000","last_active":"2024-06-29T14:41:22.699905+0000","last_peered":"2024-06-29T14:41:22.699905+0000","last_clean":"2024-06-29T14:41:22.699905+0000","last_became_active":"2024-06-29T14:41:21.714990+0000","last_became_peered":"2024-06-29T14:41:21.714990+0000","last_unstale":"2024-06-29T14:41:22.699905+0000","last_undegraded":"2024-06-29T14:41:22.699905+0000","last_fullsized":"2024-06-29T14:41:22.699905+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:01.640974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"132'2","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.730342+0000","last_change":"2024-06-29T14:41:20.703672+0000","last_active":"2024-06-29T14:41:21.730342+0000","last_peered":"2024-06-29T14:41:21.730342+0000","last_clean":"2024-06-29T14:41:21.730342+0000","last_became_active":"2024-06-29T14:41:20.703321+0000","last_became_peered":"2024-06-29T14:41:20.703321+0000","last_unstale":"2024-06-29T14:41:21.730342+0000","last_undegraded":"2024-06-29T14:41:21.730342+0000","last_fullsized":"2024-06-29T14:41:21.730342+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:06:11.708240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253350+0000","last_change":"2024-06-29T14:41:21.732766+0000","last_active":"2024-06-29T14:41:23.253350+0000","last_peered":"2024-06-29T14:41:23.253350+0000","last_clean":"2024-06-29T14:41:23.253350+0000","last_became_active":"2024-06-29T14:41:21.732501+0000","last_became_peered":"2024-06-29T14:41:21.732501+0000","last_unstale":"2024-06-29T14:41:23.253350+0000","last_undegraded":"2024-06-29T14:41:23.253350+0000","last_fullsized":"2024-06-29T14:41:23.253350+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:20.575144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"132'1","reported_seq":15,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717558+0000","last_change":"2024-06-29T14:41:20.709668+0000","last_active":"2024-06-29T14:41:22.717558+0000","last_peered":"2024-06-29T14:41:22.717558+0000","last_clean":"2024-06-29T14:41:22.717558+0000","last_became_active":"2024-06-29T14:41:20.709261+0000","last_became_peered":"2024-06-29T14:41:20.709261+0000","last_unstale":"2024-06-29T14:41:22.717558+0000","last_undegraded":"2024-06-29T14:41:22.717558+0000","last_fullsized":"2024-06-29T14:41:22.717558+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:08.649527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253792+0000","last_change":"2024-06-29T14:41:21.722892+0000","last_active":"2024-06-29T14:41:23.253792+0000","last_peered":"2024-06-29T14:41:23.253792+0000","last_clean":"2024-06-29T14:41:23.253792+0000","last_became_active":"2024-06-29T14:41:21.722572+0000","last_became_peered":"2024-06-29T14:41:21.722572+0000","last_unstale":"2024-06-29T14:41:23.253792+0000","last_undegraded":"2024-06-29T14:41:23.253792+0000","last_fullsized":"2024-06-29T14:41:23.253792+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:53.313037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703448+0000","last_change":"2024-06-29T14:41:20.701698+0000","last_active":"2024-06-29T14:41:22.703448+0000","last_peered":"2024-06-29T14:41:22.703448+0000","last_clean":"2024-06-29T14:41:22.703448+0000","last_became_active":"2024-06-29T14:41:20.701346+0000","last_became_peered":"2024-06-29T14:41:20.701346+0000","last_unstale":"2024-06-29T14:41:22.703448+0000","last_undegraded":"2024-06-29T14:41:22.703448+0000","last_fullsized":"2024-06-29T14:41:22.703448+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:09.791011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253172+0000","last_change":"2024-06-29T14:41:21.722475+0000","last_active":"2024-06-29T14:41:23.253172+0000","last_peered":"2024-06-29T14:41:23.253172+0000","last_clean":"2024-06-29T14:41:23.253172+0000","last_became_active":"2024-06-29T14:41:21.722101+0000","last_became_peered":"2024-06-29T14:41:21.722101+0000","last_unstale":"2024-06-29T14:41:23.253172+0000","last_undegraded":"2024-06-29T14:41:23.253172+0000","last_fullsized":"2024-06-29T14:41:23.253172+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:34.531265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"132'2","reported_seq":16,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699850+0000","last_change":"2024-06-29T14:41:20.700389+0000","last_active":"2024-06-29T14:41:22.699850+0000","last_peered":"2024-06-29T14:41:22.699850+0000","last_clean":"2024-06-29T14:41:22.699850+0000","last_became_active":"2024-06-29T14:41:20.700035+0000","last_became_peered":"2024-06-29T14:41:20.700035+0000","last_unstale":"2024-06-29T14:41:22.699850+0000","last_undegraded":"2024-06-29T14:41:22.699850+0000","last_fullsized":"2024-06-29T14:41:22.699850+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:43:02.057931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.732078+0000","last_change":"2024-06-29T14:41:21.724536+0000","last_active":"2024-06-29T14:41:21.732078+0000","last_peered":"2024-06-29T14:41:21.732078+0000","last_clean":"2024-06-29T14:41:21.732078+0000","last_became_active":"2024-06-29T14:41:21.724211+0000","last_became_peered":"2024-06-29T14:41:21.724211+0000","last_unstale":"2024-06-29T14:41:21.732078+0000","last_undegraded":"2024-06-29T14:41:21.732078+0000","last_fullsized":"2024-06-29T14:41:21.732078+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:10.799138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.700588+0000","last_change":"2024-06-29T14:41:20.701456+0000","last_active":"2024-06-29T14:41:21.700588+0000","last_peered":"2024-06-29T14:41:21.700588+0000","last_clean":"2024-06-29T14:41:21.700588+0000","last_became_active":"2024-06-29T14:41:20.701136+0000","last_became_peered":"2024-06-29T14:41:20.701136+0000","last_unstale":"2024-06-29T14:41:21.700588+0000","last_undegraded":"2024-06-29T14:41:21.700588+0000","last_fullsized":"2024-06-29T14:41:21.700588+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:02.438990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253352+0000","last_change":"2024-06-29T14:41:21.728475+0000","last_active":"2024-06-29T14:41:23.253352+0000","last_peered":"2024-06-29T14:41:23.253352+0000","last_clean":"2024-06-29T14:41:23.253352+0000","last_became_active":"2024-06-29T14:41:21.727909+0000","last_became_peered":"2024-06-29T14:41:21.727909+0000","last_unstale":"2024-06-29T14:41:23.253352+0000","last_undegraded":"2024-06-29T14:41:23.253352+0000","last_fullsized":"2024-06-29T14:41:23.253352+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:09.771556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699523+0000","last_change":"2024-06-29T14:41:20.697326+0000","last_active":"2024-06-29T14:41:22.699523+0000","last_peered":"2024-06-29T14:41:22.699523+0000","last_clean":"2024-06-29T14:41:22.699523+0000","last_became_active":"2024-06-29T14:41:20.696991+0000","last_became_peered":"2024-06-29T14:41:20.696991+0000","last_unstale":"2024-06-29T14:41:22.699523+0000","last_undegraded":"2024-06-29T14:41:22.699523+0000","last_fullsized":"2024-06-29T14:41:22.699523+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:21.138223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.721564+0000","last_change":"2024-06-29T14:41:21.716801+0000","last_active":"2024-06-29T14:41:21.721564+0000","last_peered":"2024-06-29T14:41:21.721564+0000","last_clean":"2024-06-29T14:41:21.721564+0000","last_became_active":"2024-06-29T14:41:21.716380+0000","last_became_peered":"2024-06-29T14:41:21.716380+0000","last_unstale":"2024-06-29T14:41:21.721564+0000","last_undegraded":"2024-06-29T14:41:21.721564+0000","last_fullsized":"2024-06-29T14:41:21.721564+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:31:24.510113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.709884+0000","last_change":"2024-06-29T14:41:20.707303+0000","last_active":"2024-06-29T14:41:21.709884+0000","last_peered":"2024-06-29T14:41:21.709884+0000","last_clean":"2024-06-29T14:41:21.709884+0000","last_became_active":"2024-06-29T14:41:20.705129+0000","last_became_peered":"2024-06-29T14:41:20.705129+0000","last_unstale":"2024-06-29T14:41:21.709884+0000","last_undegraded":"2024-06-29T14:41:21.709884+0000","last_fullsized":"2024-06-29T14:41:21.709884+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:28.206776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717497+0000","last_change":"2024-06-29T14:41:21.722454+0000","last_active":"2024-06-29T14:41:22.717497+0000","last_peered":"2024-06-29T14:41:22.717497+0000","last_clean":"2024-06-29T14:41:22.717497+0000","last_became_active":"2024-06-29T14:41:21.721683+0000","last_became_peered":"2024-06-29T14:41:21.721683+0000","last_unstale":"2024-06-29T14:41:22.717497+0000","last_undegraded":"2024-06-29T14:41:22.717497+0000","last_fullsized":"2024-06-29T14:41:22.717497+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:56:27.444549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"132'2","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.732516+0000","last_change":"2024-06-29T14:41:20.709603+0000","last_active":"2024-06-29T14:41:21.732516+0000","last_peered":"2024-06-29T14:41:21.732516+0000","last_clean":"2024-06-29T14:41:21.732516+0000","last_became_active":"2024-06-29T14:41:20.709283+0000","last_became_peered":"2024-06-29T14:41:20.709283+0000","last_unstale":"2024-06-29T14:41:21.732516+0000","last_undegraded":"2024-06-29T14:41:21.732516+0000","last_fullsized":"2024-06-29T14:41:21.732516+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:14.117393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.717435+0000","last_change":"2024-06-29T14:41:21.714812+0000","last_active":"2024-06-29T14:41:21.717435+0000","last_peered":"2024-06-29T14:41:21.717435+0000","last_clean":"2024-06-29T14:41:21.717435+0000","last_became_active":"2024-06-29T14:41:21.714400+0000","last_became_peered":"2024-06-29T14:41:21.714400+0000","last_unstale":"2024-06-29T14:41:21.717435+0000","last_undegraded":"2024-06-29T14:41:21.717435+0000","last_fullsized":"2024-06-29T14:41:21.717435+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:31.099410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"133'1","reported_seq":15,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.720992+0000","last_change":"2024-06-29T14:41:20.705032+0000","last_active":"2024-06-29T14:41:22.720992+0000","last_peered":"2024-06-29T14:41:22.720992+0000","last_clean":"2024-06-29T14:41:22.720992+0000","last_became_active":"2024-06-29T14:41:20.704678+0000","last_became_peered":"2024-06-29T14:41:20.704678+0000","last_unstale":"2024-06-29T14:41:22.720992+0000","last_undegraded":"2024-06-29T14:41:22.720992+0000","last_fullsized":"2024-06-29T14:41:22.720992+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:31.737350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.723322+0000","last_change":"2024-06-29T14:41:21.718987+0000","last_active":"2024-06-29T14:41:21.723322+0000","last_peered":"2024-06-29T14:41:21.723322+0000","last_clean":"2024-06-29T14:41:21.723322+0000","last_became_active":"2024-06-29T14:41:21.717554+0000","last_became_peered":"2024-06-29T14:41:21.717554+0000","last_unstale":"2024-06-29T14:41:21.723322+0000","last_undegraded":"2024-06-29T14:41:21.723322+0000","last_fullsized":"2024-06-29T14:41:21.723322+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:46:32.284666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.713719+0000","last_change":"2024-06-29T14:41:20.704958+0000","last_active":"2024-06-29T14:41:21.713719+0000","last_peered":"2024-06-29T14:41:21.713719+0000","last_clean":"2024-06-29T14:41:21.713719+0000","last_became_active":"2024-06-29T14:41:20.704637+0000","last_became_peered":"2024-06-29T14:41:20.704637+0000","last_unstale":"2024-06-29T14:41:21.713719+0000","last_undegraded":"2024-06-29T14:41:21.713719+0000","last_fullsized":"2024-06-29T14:41:21.713719+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:09:58.320331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717636+0000","last_change":"2024-06-29T14:41:21.724392+0000","last_active":"2024-06-29T14:41:22.717636+0000","last_peered":"2024-06-29T14:41:22.717636+0000","last_clean":"2024-06-29T14:41:22.717636+0000","last_became_active":"2024-06-29T14:41:21.721839+0000","last_became_peered":"2024-06-29T14:41:21.721839+0000","last_unstale":"2024-06-29T14:41:22.717636+0000","last_undegraded":"2024-06-29T14:41:22.717636+0000","last_fullsized":"2024-06-29T14:41:22.717636+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:33.257760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.702276+0000","last_change":"2024-06-29T14:41:20.693255+0000","last_active":"2024-06-29T14:41:21.702276+0000","last_peered":"2024-06-29T14:41:21.702276+0000","last_clean":"2024-06-29T14:41:21.702276+0000","last_became_active":"2024-06-29T14:41:20.692912+0000","last_became_peered":"2024-06-29T14:41:20.692912+0000","last_unstale":"2024-06-29T14:41:21.702276+0000","last_undegraded":"2024-06-29T14:41:21.702276+0000","last_fullsized":"2024-06-29T14:41:21.702276+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:33:59.477398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702601+0000","last_change":"2024-06-29T14:41:21.709493+0000","last_active":"2024-06-29T14:41:22.702601+0000","last_peered":"2024-06-29T14:41:22.702601+0000","last_clean":"2024-06-29T14:41:22.702601+0000","last_became_active":"2024-06-29T14:41:21.709179+0000","last_became_peered":"2024-06-29T14:41:21.709179+0000","last_unstale":"2024-06-29T14:41:22.702601+0000","last_undegraded":"2024-06-29T14:41:22.702601+0000","last_fullsized":"2024-06-29T14:41:22.702601+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:38.251709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703559+0000","last_change":"2024-06-29T14:41:20.701417+0000","last_active":"2024-06-29T14:41:22.703559+0000","last_peered":"2024-06-29T14:41:22.703559+0000","last_clean":"2024-06-29T14:41:22.703559+0000","last_became_active":"2024-06-29T14:41:20.701105+0000","last_became_peered":"2024-06-29T14:41:20.701105+0000","last_unstale":"2024-06-29T14:41:22.703559+0000","last_undegraded":"2024-06-29T14:41:22.703559+0000","last_fullsized":"2024-06-29T14:41:22.703559+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:04.844487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702969+0000","last_change":"2024-06-29T14:41:21.711170+0000","last_active":"2024-06-29T14:41:22.702969+0000","last_peered":"2024-06-29T14:41:22.702969+0000","last_clean":"2024-06-29T14:41:22.702969+0000","last_became_active":"2024-06-29T14:41:21.710801+0000","last_became_peered":"2024-06-29T14:41:21.710801+0000","last_unstale":"2024-06-29T14:41:22.702969+0000","last_undegraded":"2024-06-29T14:41:22.702969+0000","last_fullsized":"2024-06-29T14:41:22.702969+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:52:27.230538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703519+0000","last_change":"2024-06-29T14:41:20.694502+0000","last_active":"2024-06-29T14:41:22.703519+0000","last_peered":"2024-06-29T14:41:22.703519+0000","last_clean":"2024-06-29T14:41:22.703519+0000","last_became_active":"2024-06-29T14:41:20.694242+0000","last_became_peered":"2024-06-29T14:41:20.694242+0000","last_unstale":"2024-06-29T14:41:22.703519+0000","last_undegraded":"2024-06-29T14:41:22.703519+0000","last_fullsized":"2024-06-29T14:41:22.703519+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:59.455236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253392+0000","last_change":"2024-06-29T14:41:21.725677+0000","last_active":"2024-06-29T14:41:23.253392+0000","last_peered":"2024-06-29T14:41:23.253392+0000","last_clean":"2024-06-29T14:41:23.253392+0000","last_became_active":"2024-06-29T14:41:21.725165+0000","last_became_peered":"2024-06-29T14:41:21.725165+0000","last_unstale":"2024-06-29T14:41:23.253392+0000","last_undegraded":"2024-06-29T14:41:23.253392+0000","last_fullsized":"2024-06-29T14:41:23.253392+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:17.983080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.700816+0000","last_change":"2024-06-29T14:41:20.696645+0000","last_active":"2024-06-29T14:41:21.700816+0000","last_peered":"2024-06-29T14:41:21.700816+0000","last_clean":"2024-06-29T14:41:21.700816+0000","last_became_active":"2024-06-29T14:41:20.696283+0000","last_became_peered":"2024-06-29T14:41:20.696283+0000","last_unstale":"2024-06-29T14:41:21.700816+0000","last_undegraded":"2024-06-29T14:41:21.700816+0000","last_fullsized":"2024-06-29T14:41:21.700816+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:42.993335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.713838+0000","last_change":"2024-06-29T14:41:21.708325+0000","last_active":"2024-06-29T14:41:21.713838+0000","last_peered":"2024-06-29T14:41:21.713838+0000","last_clean":"2024-06-29T14:41:21.713838+0000","last_became_active":"2024-06-29T14:41:21.707948+0000","last_became_peered":"2024-06-29T14:41:21.707948+0000","last_unstale":"2024-06-29T14:41:21.713838+0000","last_undegraded":"2024-06-29T14:41:21.713838+0000","last_fullsized":"2024-06-29T14:41:21.713838+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:33.257760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.710682+0000","last_change":"2024-06-29T14:41:20.707556+0000","last_active":"2024-06-29T14:41:21.710682+0000","last_peered":"2024-06-29T14:41:21.710682+0000","last_clean":"2024-06-29T14:41:21.710682+0000","last_became_active":"2024-06-29T14:41:20.705784+0000","last_became_peered":"2024-06-29T14:41:20.705784+0000","last_unstale":"2024-06-29T14:41:21.710682+0000","last_undegraded":"2024-06-29T14:41:21.710682+0000","last_fullsized":"2024-06-29T14:41:21.710682+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:16.054121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.726177+0000","last_change":"2024-06-29T14:41:21.720836+0000","last_active":"2024-06-29T14:41:21.726177+0000","last_peered":"2024-06-29T14:41:21.726177+0000","last_clean":"2024-06-29T14:41:21.726177+0000","last_became_active":"2024-06-29T14:41:21.720541+0000","last_became_peered":"2024-06-29T14:41:21.720541+0000","last_unstale":"2024-06-29T14:41:21.726177+0000","last_undegraded":"2024-06-29T14:41:21.726177+0000","last_fullsized":"2024-06-29T14:41:21.726177+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:24:24.312386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703403+0000","last_change":"2024-06-29T14:41:20.705572+0000","last_active":"2024-06-29T14:41:22.703403+0000","last_peered":"2024-06-29T14:41:22.703403+0000","last_clean":"2024-06-29T14:41:22.703403+0000","last_became_active":"2024-06-29T14:41:20.705255+0000","last_became_peered":"2024-06-29T14:41:20.705255+0000","last_unstale":"2024-06-29T14:41:22.703403+0000","last_undegraded":"2024-06-29T14:41:22.703403+0000","last_fullsized":"2024-06-29T14:41:22.703403+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:52:26.222412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"122'142","reported_seq":579,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253438+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:41:23.253438+0000","last_peered":"2024-06-29T14:41:23.253438+0000","last_clean":"2024-06-29T14:41:23.253438+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:41:23.253438+0000","last_undegraded":"2024-06-29T14:41:23.253438+0000","last_fullsized":"2024-06-29T14:41:23.253438+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":221,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":221,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142,"ondisk_log_size":142,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575076,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33368,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338472,"statfs":{"total":96636764160,"available":96602595328,"internally_reserved":0,"allocated":2240512,"data_stored":1378912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575077,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36960,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334880,"statfs":{"total":96636764160,"available":96598917120,"internally_reserved":0,"allocated":1654784,"data_stored":794398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575075,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36948,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334892,"statfs":{"total":96636764160,"available":96598929408,"internally_reserved":0,"allocated":1646592,"data_stored":788532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575076,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59164,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312676,"statfs":{"total":96636764160,"available":96576180224,"internally_reserved":0,"allocated":1650688,"data_stored":794328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575076,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36960,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334880,"statfs":{"total":96636764160,"available":96598917120,"internally_reserved":0,"allocated":1654784,"data_stored":789610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575076,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41104,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330736,"statfs":{"total":96636764160,"available":96594673664,"internally_reserved":0,"allocated":1638400,"data_stored":787928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575075,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54996,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316844,"statfs":{"total":96636764160,"available":96580448256,"internally_reserved":0,"allocated":1646592,"data_stored":788052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33388,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338452,"statfs":{"total":96636764160,"available":96602574848,"internally_reserved":0,"allocated":2252800,"data_stored":1380020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:41:25.752 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:41:25.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.906+0000 7ffba7563640 1 -- 172.21.15.81:0/535588781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb88004090 msgr2=0x7ffb88004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.906+0000 7ffba7563640 1 --2- 172.21.15.81:0/535588781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb88004090 0x7ffb88004800 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7ffb940016c0 tx=0x7ffb88005830 comp rx=0 tx=0).stop 2024-06-29T14:41:25.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.906+0000 7ffba5b00640 1 -- 172.21.15.81:0/535588781 reap_dead start 2024-06-29T14:41:25.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.906+0000 7ffba7563640 1 -- 172.21.15.81:0/535588781 shutdown_connections 2024-06-29T14:41:25.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.906+0000 7ffba7563640 1 -- 172.21.15.81:0/535588781 >> 172.21.15.81:0/535588781 conn(0x7ffba006ccc0 msgr2=0x7ffba00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:25.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.906+0000 7ffba7563640 1 -- 172.21.15.81:0/535588781 shutdown_connections 2024-06-29T14:41:25.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.906+0000 7ffba7563640 1 -- 172.21.15.81:0/535588781 wait complete. 2024-06-29T14:41:25.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.907+0000 7ffba7563640 1 Processor -- start 2024-06-29T14:41:25.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.908+0000 7ffba7563640 1 -- start start 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba7563640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb88004090 0x7ffba013fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba7563640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba00b0c20 0x7ffba00ae940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba7563640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffba00aee80 0x7ffba01502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba7563640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffba015eef0 con 0x7ffba00b0c20 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba7563640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ffba015ed70 con 0x7ffba00aee80 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba7563640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ffba015f070 con 0x7ffb88004090 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba4afe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba00b0c20 0x7ffba00ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba4afe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba00b0c20 0x7ffba00ae940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48104/0 (socket says 172.21.15.81:48104) 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba4afe640 1 -- 172.21.15.81:0/4027598858 learned_addr learned my addr 172.21.15.81:0/4027598858 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba52ff640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb88004090 0x7ffba013fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba5b00640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffba00aee80 0x7ffba01502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba4afe640 1 -- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb88004090 msgr2=0x7ffba013fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba4afe640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb88004090 0x7ffba013fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba4afe640 1 -- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffba00aee80 msgr2=0x7ffba01502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba4afe640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffba00aee80 0x7ffba01502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba4afe640 1 -- 172.21.15.81:0/4027598858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba0154a20 con 0x7ffba00b0c20 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba52ff640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb88004090 0x7ffba013fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffba4afe640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba00b0c20 0x7ffba00ae940 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7ffb940017e0 tx=0x7ffb94006360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.909+0000 7ffb9dffb640 1 -- 172.21.15.81:0/4027598858 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb94038070 con 0x7ffba00b0c20 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.910+0000 7ffb9dffb640 1 -- 172.21.15.81:0/4027598858 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb94003af0 con 0x7ffba00b0c20 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.910+0000 7ffba7563640 1 -- 172.21.15.81:0/4027598858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffba00af290 con 0x7ffba00b0c20 2024-06-29T14:41:25.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.910+0000 7ffb9dffb640 1 -- 172.21.15.81:0/4027598858 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb94033420 con 0x7ffba00b0c20 2024-06-29T14:41:25.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.910+0000 7ffba7563640 1 -- 172.21.15.81:0/4027598858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffba01526e0 con 0x7ffba00b0c20 2024-06-29T14:41:25.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.911+0000 7ffba7563640 1 -- 172.21.15.81:0/4027598858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffba00b1410 con 0x7ffba00b0c20 2024-06-29T14:41:25.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.911+0000 7ffb9dffb640 1 -- 172.21.15.81:0/4027598858 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7ffb94043020 con 0x7ffba00b0c20 2024-06-29T14:41:25.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.912+0000 7ffb9dffb640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ffb8c0763f0 0x7ffb8c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:25.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.912+0000 7ffb9dffb640 1 -- 172.21.15.81:0/4027598858 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7ffb940b6fc0 con 0x7ffba00b0c20 2024-06-29T14:41:25.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.912+0000 7ffba52ff640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ffb8c0763f0 0x7ffb8c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:25.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.913+0000 7ffba52ff640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ffb8c0763f0 0x7ffb8c078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffb90006fd0 tx=0x7ffb90006b20 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:25.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:25.916+0000 7ffb9dffb640 1 -- 172.21.15.81:0/4027598858 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffba00af290 con 0x7ffba00b0c20 2024-06-29T14:41:26.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.092+0000 7ffba7563640 1 -- 172.21.15.81:0/4027598858 --> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffba00b16a0 con 0x7ffb8c0763f0 2024-06-29T14:41:26.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.097+0000 7ffb9dffb640 1 -- 172.21.15.81:0/4027598858 <== mgr.8328 v2:172.21.15.167:6832/971640072 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252456 (secure 0 0 0) 0x7ffba00b16a0 con 0x7ffb8c0763f0 2024-06-29T14:41:26.097 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:26.099 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:41:26.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 -- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ffb8c0763f0 msgr2=0x7ffb8c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ffb8c0763f0 0x7ffb8c078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ffb90006fd0 tx=0x7ffb90006b20 comp rx=0 tx=0).stop 2024-06-29T14:41:26.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 -- 172.21.15.81:0/4027598858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba00b0c20 msgr2=0x7ffba00ae940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba00b0c20 0x7ffba00ae940 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7ffb940017e0 tx=0x7ffb94006360 comp rx=0 tx=0).stop 2024-06-29T14:41:26.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 -- 172.21.15.81:0/4027598858 shutdown_connections 2024-06-29T14:41:26.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ffb8c0763f0 0x7ffb8c078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffba00aee80 0x7ffba01502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba00b0c20 0x7ffba00ae940 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 --2- 172.21.15.81:0/4027598858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb88004090 0x7ffba013fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 -- 172.21.15.81:0/4027598858 >> 172.21.15.81:0/4027598858 conn(0x7ffba006ccc0 msgr2=0x7ffba006e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:26.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 -- 172.21.15.81:0/4027598858 shutdown_connections 2024-06-29T14:41:26.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.100+0000 7ffba7563640 1 -- 172.21.15.81:0/4027598858 wait complete. 2024-06-29T14:41:26.109 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-06-29T14:41:25.272382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":266,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":332888,"kb_used_data":14048,"kb_used_omap":12,"kb_used_meta":318707,"kb_avail":754641832,"statfs":{"total":773094113280,"available":772753235968,"internally_reserved":0,"allocated":14385152,"data_stored":7501780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":326356530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.592418"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.721167+0000","last_change":"2024-06-29T14:41:21.718413+0000","last_active":"2024-06-29T14:41:21.721167+0000","last_peered":"2024-06-29T14:41:21.721167+0000","last_clean":"2024-06-29T14:41:21.721167+0000","last_became_active":"2024-06-29T14:41:21.718094+0000","last_became_peered":"2024-06-29T14:41:21.718094+0000","last_unstale":"2024-06-29T14:41:21.721167+0000","last_undegraded":"2024-06-29T14:41:21.721167+0000","last_fullsized":"2024-06-29T14:41:21.721167+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:48:14.512858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703045+0000","last_change":"2024-06-29T14:41:21.715818+0000","last_active":"2024-06-29T14:41:22.703045+0000","last_peered":"2024-06-29T14:41:22.703045+0000","last_clean":"2024-06-29T14:41:22.703045+0000","last_became_active":"2024-06-29T14:41:21.712890+0000","last_became_peered":"2024-06-29T14:41:21.712890+0000","last_unstale":"2024-06-29T14:41:22.703045+0000","last_undegraded":"2024-06-29T14:41:22.703045+0000","last_fullsized":"2024-06-29T14:41:22.703045+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:44.889628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253462+0000","last_change":"2024-06-29T14:41:21.724779+0000","last_active":"2024-06-29T14:41:23.253462+0000","last_peered":"2024-06-29T14:41:23.253462+0000","last_clean":"2024-06-29T14:41:23.253462+0000","last_became_active":"2024-06-29T14:41:21.724463+0000","last_became_peered":"2024-06-29T14:41:21.724463+0000","last_unstale":"2024-06-29T14:41:23.253462+0000","last_undegraded":"2024-06-29T14:41:23.253462+0000","last_fullsized":"2024-06-29T14:41:23.253462+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:20:49.679475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.723276+0000","last_change":"2024-06-29T14:41:21.718741+0000","last_active":"2024-06-29T14:41:21.723276+0000","last_peered":"2024-06-29T14:41:21.723276+0000","last_clean":"2024-06-29T14:41:21.723276+0000","last_became_active":"2024-06-29T14:41:21.718227+0000","last_became_peered":"2024-06-29T14:41:21.718227+0000","last_unstale":"2024-06-29T14:41:21.723276+0000","last_undegraded":"2024-06-29T14:41:21.723276+0000","last_fullsized":"2024-06-29T14:41:21.723276+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:20:50.967368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.726522+0000","last_change":"2024-06-29T14:41:21.717137+0000","last_active":"2024-06-29T14:41:21.726522+0000","last_peered":"2024-06-29T14:41:21.726522+0000","last_clean":"2024-06-29T14:41:21.726522+0000","last_became_active":"2024-06-29T14:41:21.716835+0000","last_became_peered":"2024-06-29T14:41:21.716835+0000","last_unstale":"2024-06-29T14:41:21.726522+0000","last_undegraded":"2024-06-29T14:41:21.726522+0000","last_fullsized":"2024-06-29T14:41:21.726522+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:26.645847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699490+0000","last_change":"2024-06-29T14:41:21.712324+0000","last_active":"2024-06-29T14:41:22.699490+0000","last_peered":"2024-06-29T14:41:22.699490+0000","last_clean":"2024-06-29T14:41:22.699490+0000","last_became_active":"2024-06-29T14:41:21.712031+0000","last_became_peered":"2024-06-29T14:41:21.712031+0000","last_unstale":"2024-06-29T14:41:22.699490+0000","last_undegraded":"2024-06-29T14:41:22.699490+0000","last_fullsized":"2024-06-29T14:41:22.699490+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:38.450632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703274+0000","last_change":"2024-06-29T14:41:21.714258+0000","last_active":"2024-06-29T14:41:22.703274+0000","last_peered":"2024-06-29T14:41:22.703274+0000","last_clean":"2024-06-29T14:41:22.703274+0000","last_became_active":"2024-06-29T14:41:21.713821+0000","last_became_peered":"2024-06-29T14:41:21.713821+0000","last_unstale":"2024-06-29T14:41:22.703274+0000","last_undegraded":"2024-06-29T14:41:22.703274+0000","last_fullsized":"2024-06-29T14:41:22.703274+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:23.229276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253221+0000","last_change":"2024-06-29T14:41:21.726845+0000","last_active":"2024-06-29T14:41:23.253221+0000","last_peered":"2024-06-29T14:41:23.253221+0000","last_clean":"2024-06-29T14:41:23.253221+0000","last_became_active":"2024-06-29T14:41:21.726498+0000","last_became_peered":"2024-06-29T14:41:21.726498+0000","last_unstale":"2024-06-29T14:41:23.253221+0000","last_undegraded":"2024-06-29T14:41:23.253221+0000","last_fullsized":"2024-06-29T14:41:23.253221+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:36:23.413101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.729342+0000","last_change":"2024-06-29T14:41:21.724593+0000","last_active":"2024-06-29T14:41:21.729342+0000","last_peered":"2024-06-29T14:41:21.729342+0000","last_clean":"2024-06-29T14:41:21.729342+0000","last_became_active":"2024-06-29T14:41:21.724002+0000","last_became_peered":"2024-06-29T14:41:21.724002+0000","last_unstale":"2024-06-29T14:41:21.729342+0000","last_undegraded":"2024-06-29T14:41:21.729342+0000","last_fullsized":"2024-06-29T14:41:21.729342+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:38.251709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.721456+0000","last_change":"2024-06-29T14:41:21.716148+0000","last_active":"2024-06-29T14:41:21.721456+0000","last_peered":"2024-06-29T14:41:21.721456+0000","last_clean":"2024-06-29T14:41:21.721456+0000","last_became_active":"2024-06-29T14:41:21.715751+0000","last_became_peered":"2024-06-29T14:41:21.715751+0000","last_unstale":"2024-06-29T14:41:21.721456+0000","last_undegraded":"2024-06-29T14:41:21.721456+0000","last_fullsized":"2024-06-29T14:41:21.721456+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:15:48.236647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703218+0000","last_change":"2024-06-29T14:41:21.717671+0000","last_active":"2024-06-29T14:41:22.703218+0000","last_peered":"2024-06-29T14:41:22.703218+0000","last_clean":"2024-06-29T14:41:22.703218+0000","last_became_active":"2024-06-29T14:41:21.716553+0000","last_became_peered":"2024-06-29T14:41:21.716553+0000","last_unstale":"2024-06-29T14:41:22.703218+0000","last_undegraded":"2024-06-29T14:41:22.703218+0000","last_fullsized":"2024-06-29T14:41:22.703218+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:49.527582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.723301+0000","last_change":"2024-06-29T14:41:21.718686+0000","last_active":"2024-06-29T14:41:21.723301+0000","last_peered":"2024-06-29T14:41:21.723301+0000","last_clean":"2024-06-29T14:41:21.723301+0000","last_became_active":"2024-06-29T14:41:21.717887+0000","last_became_peered":"2024-06-29T14:41:21.717887+0000","last_unstale":"2024-06-29T14:41:21.723301+0000","last_undegraded":"2024-06-29T14:41:21.723301+0000","last_fullsized":"2024-06-29T14:41:21.723301+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:19:18.761595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253299+0000","last_change":"2024-06-29T14:41:21.728555+0000","last_active":"2024-06-29T14:41:23.253299+0000","last_peered":"2024-06-29T14:41:23.253299+0000","last_clean":"2024-06-29T14:41:23.253299+0000","last_became_active":"2024-06-29T14:41:21.727680+0000","last_became_peered":"2024-06-29T14:41:21.727680+0000","last_unstale":"2024-06-29T14:41:23.253299+0000","last_undegraded":"2024-06-29T14:41:23.253299+0000","last_fullsized":"2024-06-29T14:41:23.253299+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:07:14.151770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253191+0000","last_change":"2024-06-29T14:41:21.725694+0000","last_active":"2024-06-29T14:41:23.253191+0000","last_peered":"2024-06-29T14:41:23.253191+0000","last_clean":"2024-06-29T14:41:23.253191+0000","last_became_active":"2024-06-29T14:41:21.725380+0000","last_became_peered":"2024-06-29T14:41:21.725380+0000","last_unstale":"2024-06-29T14:41:23.253191+0000","last_undegraded":"2024-06-29T14:41:23.253191+0000","last_fullsized":"2024-06-29T14:41:23.253191+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:43:57.185593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699412+0000","last_change":"2024-06-29T14:41:21.713671+0000","last_active":"2024-06-29T14:41:22.699412+0000","last_peered":"2024-06-29T14:41:22.699412+0000","last_clean":"2024-06-29T14:41:22.699412+0000","last_became_active":"2024-06-29T14:41:21.713364+0000","last_became_peered":"2024-06-29T14:41:21.713364+0000","last_unstale":"2024-06-29T14:41:22.699412+0000","last_undegraded":"2024-06-29T14:41:22.699412+0000","last_fullsized":"2024-06-29T14:41:22.699412+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:51.203732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.729314+0000","last_change":"2024-06-29T14:41:21.724153+0000","last_active":"2024-06-29T14:41:21.729314+0000","last_peered":"2024-06-29T14:41:21.729314+0000","last_clean":"2024-06-29T14:41:21.729314+0000","last_became_active":"2024-06-29T14:41:21.723841+0000","last_became_peered":"2024-06-29T14:41:21.723841+0000","last_unstale":"2024-06-29T14:41:21.729314+0000","last_undegraded":"2024-06-29T14:41:21.729314+0000","last_fullsized":"2024-06-29T14:41:21.729314+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:24.890464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703637+0000","last_change":"2024-06-29T14:41:21.716298+0000","last_active":"2024-06-29T14:41:22.703637+0000","last_peered":"2024-06-29T14:41:22.703637+0000","last_clean":"2024-06-29T14:41:22.703637+0000","last_became_active":"2024-06-29T14:41:21.715651+0000","last_became_peered":"2024-06-29T14:41:21.715651+0000","last_unstale":"2024-06-29T14:41:22.703637+0000","last_undegraded":"2024-06-29T14:41:22.703637+0000","last_fullsized":"2024-06-29T14:41:22.703637+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:10.650320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717627+0000","last_change":"2024-06-29T14:41:21.712006+0000","last_active":"2024-06-29T14:41:22.717627+0000","last_peered":"2024-06-29T14:41:22.717627+0000","last_clean":"2024-06-29T14:41:22.717627+0000","last_became_active":"2024-06-29T14:41:21.711631+0000","last_became_peered":"2024-06-29T14:41:21.711631+0000","last_unstale":"2024-06-29T14:41:22.717627+0000","last_undegraded":"2024-06-29T14:41:22.717627+0000","last_fullsized":"2024-06-29T14:41:22.717627+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:41:21.021074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253682+0000","last_change":"2024-06-29T14:41:21.728181+0000","last_active":"2024-06-29T14:41:23.253682+0000","last_peered":"2024-06-29T14:41:23.253682+0000","last_clean":"2024-06-29T14:41:23.253682+0000","last_became_active":"2024-06-29T14:41:21.726987+0000","last_became_peered":"2024-06-29T14:41:21.726987+0000","last_unstale":"2024-06-29T14:41:23.253682+0000","last_undegraded":"2024-06-29T14:41:23.253682+0000","last_fullsized":"2024-06-29T14:41:23.253682+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:21.510978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703164+0000","last_change":"2024-06-29T14:41:21.715650+0000","last_active":"2024-06-29T14:41:22.703164+0000","last_peered":"2024-06-29T14:41:22.703164+0000","last_clean":"2024-06-29T14:41:22.703164+0000","last_became_active":"2024-06-29T14:41:21.715311+0000","last_became_peered":"2024-06-29T14:41:21.715311+0000","last_unstale":"2024-06-29T14:41:22.703164+0000","last_undegraded":"2024-06-29T14:41:22.703164+0000","last_fullsized":"2024-06-29T14:41:22.703164+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:26.393604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703167+0000","last_change":"2024-06-29T14:41:21.724101+0000","last_active":"2024-06-29T14:41:22.703167+0000","last_peered":"2024-06-29T14:41:22.703167+0000","last_clean":"2024-06-29T14:41:22.703167+0000","last_became_active":"2024-06-29T14:41:21.723706+0000","last_became_peered":"2024-06-29T14:41:21.723706+0000","last_unstale":"2024-06-29T14:41:22.703167+0000","last_undegraded":"2024-06-29T14:41:22.703167+0000","last_fullsized":"2024-06-29T14:41:22.703167+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:03:46.059791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.722357+0000","last_change":"2024-06-29T14:41:21.717148+0000","last_active":"2024-06-29T14:41:21.722357+0000","last_peered":"2024-06-29T14:41:21.722357+0000","last_clean":"2024-06-29T14:41:21.722357+0000","last_became_active":"2024-06-29T14:41:21.716847+0000","last_became_peered":"2024-06-29T14:41:21.716847+0000","last_unstale":"2024-06-29T14:41:21.722357+0000","last_undegraded":"2024-06-29T14:41:21.722357+0000","last_fullsized":"2024-06-29T14:41:21.722357+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:15:49.783155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703348+0000","last_change":"2024-06-29T14:41:21.712619+0000","last_active":"2024-06-29T14:41:22.703348+0000","last_peered":"2024-06-29T14:41:22.703348+0000","last_clean":"2024-06-29T14:41:22.703348+0000","last_became_active":"2024-06-29T14:41:21.712307+0000","last_became_peered":"2024-06-29T14:41:21.712307+0000","last_unstale":"2024-06-29T14:41:22.703348+0000","last_undegraded":"2024-06-29T14:41:22.703348+0000","last_fullsized":"2024-06-29T14:41:22.703348+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:49:55.208054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.726207+0000","last_change":"2024-06-29T14:41:21.721707+0000","last_active":"2024-06-29T14:41:21.726207+0000","last_peered":"2024-06-29T14:41:21.726207+0000","last_clean":"2024-06-29T14:41:21.726207+0000","last_became_active":"2024-06-29T14:41:21.721419+0000","last_became_peered":"2024-06-29T14:41:21.721419+0000","last_unstale":"2024-06-29T14:41:21.726207+0000","last_undegraded":"2024-06-29T14:41:21.726207+0000","last_fullsized":"2024-06-29T14:41:21.726207+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:22.389762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703838+0000","last_change":"2024-06-29T14:41:21.714731+0000","last_active":"2024-06-29T14:41:22.703838+0000","last_peered":"2024-06-29T14:41:22.703838+0000","last_clean":"2024-06-29T14:41:22.703838+0000","last_became_active":"2024-06-29T14:41:21.713275+0000","last_became_peered":"2024-06-29T14:41:21.713275+0000","last_unstale":"2024-06-29T14:41:22.703838+0000","last_undegraded":"2024-06-29T14:41:22.703838+0000","last_fullsized":"2024-06-29T14:41:22.703838+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:56.293327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.721236+0000","last_change":"2024-06-29T14:41:21.715999+0000","last_active":"2024-06-29T14:41:21.721236+0000","last_peered":"2024-06-29T14:41:21.721236+0000","last_clean":"2024-06-29T14:41:21.721236+0000","last_became_active":"2024-06-29T14:41:21.714846+0000","last_became_peered":"2024-06-29T14:41:21.714846+0000","last_unstale":"2024-06-29T14:41:21.721236+0000","last_undegraded":"2024-06-29T14:41:21.721236+0000","last_fullsized":"2024-06-29T14:41:21.721236+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:27.644710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699875+0000","last_change":"2024-06-29T14:41:21.715393+0000","last_active":"2024-06-29T14:41:22.699875+0000","last_peered":"2024-06-29T14:41:22.699875+0000","last_clean":"2024-06-29T14:41:22.699875+0000","last_became_active":"2024-06-29T14:41:21.715092+0000","last_became_peered":"2024-06-29T14:41:21.715092+0000","last_unstale":"2024-06-29T14:41:22.699875+0000","last_undegraded":"2024-06-29T14:41:22.699875+0000","last_fullsized":"2024-06-29T14:41:22.699875+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:26.645847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.729014+0000","last_change":"2024-06-29T14:41:21.725740+0000","last_active":"2024-06-29T14:41:21.729014+0000","last_peered":"2024-06-29T14:41:21.729014+0000","last_clean":"2024-06-29T14:41:21.729014+0000","last_became_active":"2024-06-29T14:41:21.725408+0000","last_became_peered":"2024-06-29T14:41:21.725408+0000","last_unstale":"2024-06-29T14:41:21.729014+0000","last_undegraded":"2024-06-29T14:41:21.729014+0000","last_fullsized":"2024-06-29T14:41:21.729014+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:43:28.973414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702773+0000","last_change":"2024-06-29T14:41:21.720366+0000","last_active":"2024-06-29T14:41:22.702773+0000","last_peered":"2024-06-29T14:41:22.702773+0000","last_clean":"2024-06-29T14:41:22.702773+0000","last_became_active":"2024-06-29T14:41:21.720014+0000","last_became_peered":"2024-06-29T14:41:21.720014+0000","last_unstale":"2024-06-29T14:41:22.702773+0000","last_undegraded":"2024-06-29T14:41:22.702773+0000","last_fullsized":"2024-06-29T14:41:22.702773+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:05.852614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703103+0000","last_change":"2024-06-29T14:41:21.713240+0000","last_active":"2024-06-29T14:41:22.703103+0000","last_peered":"2024-06-29T14:41:22.703103+0000","last_clean":"2024-06-29T14:41:22.703103+0000","last_became_active":"2024-06-29T14:41:21.712700+0000","last_became_peered":"2024-06-29T14:41:21.712700+0000","last_unstale":"2024-06-29T14:41:22.703103+0000","last_undegraded":"2024-06-29T14:41:22.703103+0000","last_fullsized":"2024-06-29T14:41:22.703103+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:54:56.293327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.714667+0000","last_change":"2024-06-29T14:41:21.712793+0000","last_active":"2024-06-29T14:41:21.714667+0000","last_peered":"2024-06-29T14:41:21.714667+0000","last_clean":"2024-06-29T14:41:21.714667+0000","last_became_active":"2024-06-29T14:41:21.712472+0000","last_became_peered":"2024-06-29T14:41:21.712472+0000","last_unstale":"2024-06-29T14:41:21.714667+0000","last_undegraded":"2024-06-29T14:41:21.714667+0000","last_fullsized":"2024-06-29T14:41:21.714667+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:12:00.463362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699383+0000","last_change":"2024-06-29T14:41:21.713048+0000","last_active":"2024-06-29T14:41:22.699383+0000","last_peered":"2024-06-29T14:41:22.699383+0000","last_clean":"2024-06-29T14:41:22.699383+0000","last_became_active":"2024-06-29T14:41:21.712768+0000","last_became_peered":"2024-06-29T14:41:21.712768+0000","last_unstale":"2024-06-29T14:41:22.699383+0000","last_undegraded":"2024-06-29T14:41:22.699383+0000","last_fullsized":"2024-06-29T14:41:22.699383+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:44.001462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699863+0000","last_change":"2024-06-29T14:41:21.708246+0000","last_active":"2024-06-29T14:41:22.699863+0000","last_peered":"2024-06-29T14:41:22.699863+0000","last_clean":"2024-06-29T14:41:22.699863+0000","last_became_active":"2024-06-29T14:41:21.707942+0000","last_became_peered":"2024-06-29T14:41:21.707942+0000","last_unstale":"2024-06-29T14:41:22.699863+0000","last_undegraded":"2024-06-29T14:41:22.699863+0000","last_fullsized":"2024-06-29T14:41:22.699863+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:06:12.716366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"132'4","reported_seq":18,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703452+0000","last_change":"2024-06-29T14:41:20.701718+0000","last_active":"2024-06-29T14:41:22.703452+0000","last_peered":"2024-06-29T14:41:22.703452+0000","last_clean":"2024-06-29T14:41:22.703452+0000","last_became_active":"2024-06-29T14:41:20.701395+0000","last_became_peered":"2024-06-29T14:41:20.701395+0000","last_unstale":"2024-06-29T14:41:22.703452+0000","last_undegraded":"2024-06-29T14:41:22.703452+0000","last_fullsized":"2024-06-29T14:41:22.703452+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:21.857684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717693+0000","last_change":"2024-06-29T14:41:21.717954+0000","last_active":"2024-06-29T14:41:22.717693+0000","last_peered":"2024-06-29T14:41:22.717693+0000","last_clean":"2024-06-29T14:41:22.717693+0000","last_became_active":"2024-06-29T14:41:21.717583+0000","last_became_peered":"2024-06-29T14:41:21.717583+0000","last_unstale":"2024-06-29T14:41:22.717693+0000","last_undegraded":"2024-06-29T14:41:22.717693+0000","last_fullsized":"2024-06-29T14:41:22.717693+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:35:26.174541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"132'1","reported_seq":15,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253628+0000","last_change":"2024-06-29T14:41:20.704760+0000","last_active":"2024-06-29T14:41:23.253628+0000","last_peered":"2024-06-29T14:41:23.253628+0000","last_clean":"2024-06-29T14:41:23.253628+0000","last_became_active":"2024-06-29T14:41:20.704443+0000","last_became_peered":"2024-06-29T14:41:20.704443+0000","last_unstale":"2024-06-29T14:41:23.253628+0000","last_undegraded":"2024-06-29T14:41:23.253628+0000","last_fullsized":"2024-06-29T14:41:23.253628+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:30:46.569231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699691+0000","last_change":"2024-06-29T14:41:21.713662+0000","last_active":"2024-06-29T14:41:22.699691+0000","last_peered":"2024-06-29T14:41:22.699691+0000","last_clean":"2024-06-29T14:41:22.699691+0000","last_became_active":"2024-06-29T14:41:21.713301+0000","last_became_peered":"2024-06-29T14:41:21.713301+0000","last_unstale":"2024-06-29T14:41:22.699691+0000","last_undegraded":"2024-06-29T14:41:22.699691+0000","last_fullsized":"2024-06-29T14:41:22.699691+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:54.601899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"132'2","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.730130+0000","last_change":"2024-06-29T14:41:20.702170+0000","last_active":"2024-06-29T14:41:21.730130+0000","last_peered":"2024-06-29T14:41:21.730130+0000","last_clean":"2024-06-29T14:41:21.730130+0000","last_became_active":"2024-06-29T14:41:20.701695+0000","last_became_peered":"2024-06-29T14:41:20.701695+0000","last_unstale":"2024-06-29T14:41:21.730130+0000","last_undegraded":"2024-06-29T14:41:21.730130+0000","last_fullsized":"2024-06-29T14:41:21.730130+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:52:53.593773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253694+0000","last_change":"2024-06-29T14:41:21.728453+0000","last_active":"2024-06-29T14:41:23.253694+0000","last_peered":"2024-06-29T14:41:23.253694+0000","last_clean":"2024-06-29T14:41:23.253694+0000","last_became_active":"2024-06-29T14:41:21.727640+0000","last_became_peered":"2024-06-29T14:41:21.727640+0000","last_unstale":"2024-06-29T14:41:23.253694+0000","last_undegraded":"2024-06-29T14:41:23.253694+0000","last_fullsized":"2024-06-29T14:41:23.253694+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:45:50.663167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"132'2","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.730202+0000","last_change":"2024-06-29T14:41:20.702028+0000","last_active":"2024-06-29T14:41:21.730202+0000","last_peered":"2024-06-29T14:41:21.730202+0000","last_clean":"2024-06-29T14:41:21.730202+0000","last_became_active":"2024-06-29T14:41:20.701649+0000","last_became_peered":"2024-06-29T14:41:20.701649+0000","last_unstale":"2024-06-29T14:41:21.730202+0000","last_undegraded":"2024-06-29T14:41:21.730202+0000","last_fullsized":"2024-06-29T14:41:21.730202+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:44:50.195605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.729282+0000","last_change":"2024-06-29T14:41:21.724822+0000","last_active":"2024-06-29T14:41:21.729282+0000","last_peered":"2024-06-29T14:41:21.729282+0000","last_clean":"2024-06-29T14:41:21.729282+0000","last_became_active":"2024-06-29T14:41:21.724519+0000","last_became_peered":"2024-06-29T14:41:21.724519+0000","last_unstale":"2024-06-29T14:41:21.729282+0000","last_undegraded":"2024-06-29T14:41:21.729282+0000","last_fullsized":"2024-06-29T14:41:21.729282+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:05.852614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"132'1","reported_seq":15,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703324+0000","last_change":"2024-06-29T14:41:20.696598+0000","last_active":"2024-06-29T14:41:22.703324+0000","last_peered":"2024-06-29T14:41:22.703324+0000","last_clean":"2024-06-29T14:41:22.703324+0000","last_became_active":"2024-06-29T14:41:20.696283+0000","last_became_peered":"2024-06-29T14:41:20.696283+0000","last_unstale":"2024-06-29T14:41:22.703324+0000","last_undegraded":"2024-06-29T14:41:22.703324+0000","last_fullsized":"2024-06-29T14:41:22.703324+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:37.243582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702872+0000","last_change":"2024-06-29T14:41:21.717143+0000","last_active":"2024-06-29T14:41:22.702872+0000","last_peered":"2024-06-29T14:41:22.702872+0000","last_clean":"2024-06-29T14:41:22.702872+0000","last_became_active":"2024-06-29T14:41:21.715351+0000","last_became_peered":"2024-06-29T14:41:21.715351+0000","last_unstale":"2024-06-29T14:41:22.702872+0000","last_undegraded":"2024-06-29T14:41:22.702872+0000","last_fullsized":"2024-06-29T14:41:22.702872+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:22.865811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"132'1","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.730588+0000","last_change":"2024-06-29T14:41:20.708482+0000","last_active":"2024-06-29T14:41:21.730588+0000","last_peered":"2024-06-29T14:41:21.730588+0000","last_clean":"2024-06-29T14:41:21.730588+0000","last_became_active":"2024-06-29T14:41:20.707516+0000","last_became_peered":"2024-06-29T14:41:20.707516+0000","last_unstale":"2024-06-29T14:41:21.730588+0000","last_undegraded":"2024-06-29T14:41:21.730588+0000","last_fullsized":"2024-06-29T14:41:21.730588+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:40.873222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.714211+0000","last_change":"2024-06-29T14:41:21.710694+0000","last_active":"2024-06-29T14:41:21.714211+0000","last_peered":"2024-06-29T14:41:21.714211+0000","last_clean":"2024-06-29T14:41:21.714211+0000","last_became_active":"2024-06-29T14:41:21.710379+0000","last_became_peered":"2024-06-29T14:41:21.710379+0000","last_unstale":"2024-06-29T14:41:21.714211+0000","last_undegraded":"2024-06-29T14:41:21.714211+0000","last_fullsized":"2024-06-29T14:41:21.714211+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:33:11.602177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702521+0000","last_change":"2024-06-29T14:41:20.688482+0000","last_active":"2024-06-29T14:41:22.702521+0000","last_peered":"2024-06-29T14:41:22.702521+0000","last_clean":"2024-06-29T14:41:22.702521+0000","last_became_active":"2024-06-29T14:41:20.688141+0000","last_became_peered":"2024-06-29T14:41:20.688141+0000","last_unstale":"2024-06-29T14:41:22.702521+0000","last_undegraded":"2024-06-29T14:41:22.702521+0000","last_fullsized":"2024-06-29T14:41:22.702521+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:48.550307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702563+0000","last_change":"2024-06-29T14:41:21.714560+0000","last_active":"2024-06-29T14:41:22.702563+0000","last_peered":"2024-06-29T14:41:22.702563+0000","last_clean":"2024-06-29T14:41:22.702563+0000","last_became_active":"2024-06-29T14:41:21.714245+0000","last_became_peered":"2024-06-29T14:41:21.714245+0000","last_unstale":"2024-06-29T14:41:22.702563+0000","last_undegraded":"2024-06-29T14:41:22.702563+0000","last_fullsized":"2024-06-29T14:41:22.702563+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:10.650320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.710030+0000","last_change":"2024-06-29T14:41:20.707672+0000","last_active":"2024-06-29T14:41:21.710030+0000","last_peered":"2024-06-29T14:41:21.710030+0000","last_clean":"2024-06-29T14:41:21.710030+0000","last_became_active":"2024-06-29T14:41:20.706986+0000","last_became_peered":"2024-06-29T14:41:20.706986+0000","last_unstale":"2024-06-29T14:41:21.710030+0000","last_undegraded":"2024-06-29T14:41:21.710030+0000","last_fullsized":"2024-06-29T14:41:21.710030+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:57.987710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703200+0000","last_change":"2024-06-29T14:41:21.709498+0000","last_active":"2024-06-29T14:41:22.703200+0000","last_peered":"2024-06-29T14:41:22.703200+0000","last_clean":"2024-06-29T14:41:22.703200+0000","last_became_active":"2024-06-29T14:41:21.709173+0000","last_became_peered":"2024-06-29T14:41:21.709173+0000","last_unstale":"2024-06-29T14:41:22.703200+0000","last_undegraded":"2024-06-29T14:41:22.703200+0000","last_fullsized":"2024-06-29T14:41:22.703200+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:55.550304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.701052+0000","last_change":"2024-06-29T14:41:20.701399+0000","last_active":"2024-06-29T14:41:21.701052+0000","last_peered":"2024-06-29T14:41:21.701052+0000","last_clean":"2024-06-29T14:41:21.701052+0000","last_became_active":"2024-06-29T14:41:20.701067+0000","last_became_peered":"2024-06-29T14:41:20.701067+0000","last_unstale":"2024-06-29T14:41:21.701052+0000","last_undegraded":"2024-06-29T14:41:21.701052+0000","last_fullsized":"2024-06-29T14:41:21.701052+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:37.442505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717687+0000","last_change":"2024-06-29T14:41:21.722165+0000","last_active":"2024-06-29T14:41:22.717687+0000","last_peered":"2024-06-29T14:41:22.717687+0000","last_clean":"2024-06-29T14:41:22.717687+0000","last_became_active":"2024-06-29T14:41:21.721619+0000","last_became_peered":"2024-06-29T14:41:21.721619+0000","last_unstale":"2024-06-29T14:41:22.717687+0000","last_undegraded":"2024-06-29T14:41:22.717687+0000","last_fullsized":"2024-06-29T14:41:22.717687+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:29.214902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.702461+0000","last_change":"2024-06-29T14:41:20.707863+0000","last_active":"2024-06-29T14:41:21.702461+0000","last_peered":"2024-06-29T14:41:21.702461+0000","last_clean":"2024-06-29T14:41:21.702461+0000","last_became_active":"2024-06-29T14:41:20.707545+0000","last_became_peered":"2024-06-29T14:41:20.707545+0000","last_unstale":"2024-06-29T14:41:21.702461+0000","last_undegraded":"2024-06-29T14:41:21.702461+0000","last_fullsized":"2024-06-29T14:41:21.702461+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:59.291183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.716256+0000","last_change":"2024-06-29T14:41:21.709146+0000","last_active":"2024-06-29T14:41:21.716256+0000","last_peered":"2024-06-29T14:41:21.716256+0000","last_clean":"2024-06-29T14:41:21.716256+0000","last_became_active":"2024-06-29T14:41:21.708860+0000","last_became_peered":"2024-06-29T14:41:21.708860+0000","last_unstale":"2024-06-29T14:41:21.716256+0000","last_undegraded":"2024-06-29T14:41:21.716256+0000","last_fullsized":"2024-06-29T14:41:21.716256+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:07.037259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"132'3","reported_seq":15,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.736364+0000","last_change":"2024-06-29T14:41:20.705246+0000","last_active":"2024-06-29T14:41:21.736364+0000","last_peered":"2024-06-29T14:41:21.736364+0000","last_clean":"2024-06-29T14:41:21.736364+0000","last_became_active":"2024-06-29T14:41:20.704766+0000","last_became_peered":"2024-06-29T14:41:20.704766+0000","last_unstale":"2024-06-29T14:41:21.736364+0000","last_undegraded":"2024-06-29T14:41:21.736364+0000","last_fullsized":"2024-06-29T14:41:21.736364+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:48.550307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703587+0000","last_change":"2024-06-29T14:41:21.719216+0000","last_active":"2024-06-29T14:41:22.703587+0000","last_peered":"2024-06-29T14:41:22.703587+0000","last_clean":"2024-06-29T14:41:22.703587+0000","last_became_active":"2024-06-29T14:41:21.718377+0000","last_became_peered":"2024-06-29T14:41:21.718377+0000","last_unstale":"2024-06-29T14:41:22.703587+0000","last_undegraded":"2024-06-29T14:41:22.703587+0000","last_fullsized":"2024-06-29T14:41:22.703587+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:01.789064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"132'6","reported_seq":18,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.738915+0000","last_change":"2024-06-29T14:41:20.701417+0000","last_active":"2024-06-29T14:41:21.738915+0000","last_peered":"2024-06-29T14:41:21.738915+0000","last_clean":"2024-06-29T14:41:21.738915+0000","last_became_active":"2024-06-29T14:41:20.701113+0000","last_became_peered":"2024-06-29T14:41:20.701113+0000","last_unstale":"2024-06-29T14:41:21.738915+0000","last_undegraded":"2024-06-29T14:41:21.738915+0000","last_fullsized":"2024-06-29T14:41:21.738915+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:35:25.166414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.716308+0000","last_change":"2024-06-29T14:41:21.706413+0000","last_active":"2024-06-29T14:41:21.716308+0000","last_peered":"2024-06-29T14:41:21.716308+0000","last_clean":"2024-06-29T14:41:21.716308+0000","last_became_active":"2024-06-29T14:41:21.706067+0000","last_became_peered":"2024-06-29T14:41:21.706067+0000","last_unstale":"2024-06-29T14:41:21.716308+0000","last_undegraded":"2024-06-29T14:41:21.716308+0000","last_fullsized":"2024-06-29T14:41:21.716308+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:43:44.068355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"132'2","reported_seq":16,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699634+0000","last_change":"2024-06-29T14:41:20.703226+0000","last_active":"2024-06-29T14:41:22.699634+0000","last_peered":"2024-06-29T14:41:22.699634+0000","last_clean":"2024-06-29T14:41:22.699634+0000","last_became_active":"2024-06-29T14:41:20.702875+0000","last_became_peered":"2024-06-29T14:41:20.702875+0000","last_unstale":"2024-06-29T14:41:22.699634+0000","last_undegraded":"2024-06-29T14:41:22.699634+0000","last_fullsized":"2024-06-29T14:41:22.699634+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:19:10.558384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.724209+0000","last_change":"2024-06-29T14:41:21.718234+0000","last_active":"2024-06-29T14:41:21.724209+0000","last_peered":"2024-06-29T14:41:21.724209+0000","last_clean":"2024-06-29T14:41:21.724209+0000","last_became_active":"2024-06-29T14:41:21.717729+0000","last_became_peered":"2024-06-29T14:41:21.717729+0000","last_unstale":"2024-06-29T14:41:21.724209+0000","last_undegraded":"2024-06-29T14:41:21.724209+0000","last_fullsized":"2024-06-29T14:41:21.724209+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:53.358891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717645+0000","last_change":"2024-06-29T14:41:20.706496+0000","last_active":"2024-06-29T14:41:22.717645+0000","last_peered":"2024-06-29T14:41:22.717645+0000","last_clean":"2024-06-29T14:41:22.717645+0000","last_became_active":"2024-06-29T14:41:20.706193+0000","last_became_peered":"2024-06-29T14:41:20.706193+0000","last_unstale":"2024-06-29T14:41:22.717645+0000","last_undegraded":"2024-06-29T14:41:22.717645+0000","last_fullsized":"2024-06-29T14:41:22.717645+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:00.632847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.721589+0000","last_change":"2024-06-29T14:41:21.717248+0000","last_active":"2024-06-29T14:41:21.721589+0000","last_peered":"2024-06-29T14:41:21.721589+0000","last_clean":"2024-06-29T14:41:21.721589+0000","last_became_active":"2024-06-29T14:41:21.716831+0000","last_became_peered":"2024-06-29T14:41:21.716831+0000","last_unstale":"2024-06-29T14:41:21.721589+0000","last_undegraded":"2024-06-29T14:41:21.721589+0000","last_fullsized":"2024-06-29T14:41:21.721589+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:07:32.418246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"132'8","reported_seq":20,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.731867+0000","last_change":"2024-06-29T14:41:20.708590+0000","last_active":"2024-06-29T14:41:21.731867+0000","last_peered":"2024-06-29T14:41:21.731867+0000","last_clean":"2024-06-29T14:41:21.731867+0000","last_became_active":"2024-06-29T14:41:20.708250+0000","last_became_peered":"2024-06-29T14:41:20.708250+0000","last_unstale":"2024-06-29T14:41:21.731867+0000","last_undegraded":"2024-06-29T14:41:21.731867+0000","last_fullsized":"2024-06-29T14:41:21.731867+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:37:03.084146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.726652+0000","last_change":"2024-06-29T14:41:21.718198+0000","last_active":"2024-06-29T14:41:21.726652+0000","last_peered":"2024-06-29T14:41:21.726652+0000","last_clean":"2024-06-29T14:41:21.726652+0000","last_became_active":"2024-06-29T14:41:21.717746+0000","last_became_peered":"2024-06-29T14:41:21.717746+0000","last_unstale":"2024-06-29T14:41:21.726652+0000","last_undegraded":"2024-06-29T14:41:21.726652+0000","last_fullsized":"2024-06-29T14:41:21.726652+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:01.640974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"132'1","reported_seq":13,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.731691+0000","last_change":"2024-06-29T14:41:20.706945+0000","last_active":"2024-06-29T14:41:21.731691+0000","last_peered":"2024-06-29T14:41:21.731691+0000","last_clean":"2024-06-29T14:41:21.731691+0000","last_became_active":"2024-06-29T14:41:20.704685+0000","last_became_peered":"2024-06-29T14:41:20.704685+0000","last_unstale":"2024-06-29T14:41:21.731691+0000","last_undegraded":"2024-06-29T14:41:21.731691+0000","last_fullsized":"2024-06-29T14:41:21.731691+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:04.460229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699905+0000","last_change":"2024-06-29T14:41:21.715301+0000","last_active":"2024-06-29T14:41:22.699905+0000","last_peered":"2024-06-29T14:41:22.699905+0000","last_clean":"2024-06-29T14:41:22.699905+0000","last_became_active":"2024-06-29T14:41:21.714990+0000","last_became_peered":"2024-06-29T14:41:21.714990+0000","last_unstale":"2024-06-29T14:41:22.699905+0000","last_undegraded":"2024-06-29T14:41:22.699905+0000","last_fullsized":"2024-06-29T14:41:22.699905+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:01.640974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"132'2","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.730342+0000","last_change":"2024-06-29T14:41:20.703672+0000","last_active":"2024-06-29T14:41:21.730342+0000","last_peered":"2024-06-29T14:41:21.730342+0000","last_clean":"2024-06-29T14:41:21.730342+0000","last_became_active":"2024-06-29T14:41:20.703321+0000","last_became_peered":"2024-06-29T14:41:20.703321+0000","last_unstale":"2024-06-29T14:41:21.730342+0000","last_undegraded":"2024-06-29T14:41:21.730342+0000","last_fullsized":"2024-06-29T14:41:21.730342+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:06:11.708240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253350+0000","last_change":"2024-06-29T14:41:21.732766+0000","last_active":"2024-06-29T14:41:23.253350+0000","last_peered":"2024-06-29T14:41:23.253350+0000","last_clean":"2024-06-29T14:41:23.253350+0000","last_became_active":"2024-06-29T14:41:21.732501+0000","last_became_peered":"2024-06-29T14:41:21.732501+0000","last_unstale":"2024-06-29T14:41:23.253350+0000","last_undegraded":"2024-06-29T14:41:23.253350+0000","last_fullsized":"2024-06-29T14:41:23.253350+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:46:20.575144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"132'1","reported_seq":15,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717558+0000","last_change":"2024-06-29T14:41:20.709668+0000","last_active":"2024-06-29T14:41:22.717558+0000","last_peered":"2024-06-29T14:41:22.717558+0000","last_clean":"2024-06-29T14:41:22.717558+0000","last_became_active":"2024-06-29T14:41:20.709261+0000","last_became_peered":"2024-06-29T14:41:20.709261+0000","last_unstale":"2024-06-29T14:41:22.717558+0000","last_undegraded":"2024-06-29T14:41:22.717558+0000","last_fullsized":"2024-06-29T14:41:22.717558+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:08.649527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253792+0000","last_change":"2024-06-29T14:41:21.722892+0000","last_active":"2024-06-29T14:41:23.253792+0000","last_peered":"2024-06-29T14:41:23.253792+0000","last_clean":"2024-06-29T14:41:23.253792+0000","last_became_active":"2024-06-29T14:41:21.722572+0000","last_became_peered":"2024-06-29T14:41:21.722572+0000","last_unstale":"2024-06-29T14:41:23.253792+0000","last_undegraded":"2024-06-29T14:41:23.253792+0000","last_fullsized":"2024-06-29T14:41:23.253792+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:03:53.313037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703448+0000","last_change":"2024-06-29T14:41:20.701698+0000","last_active":"2024-06-29T14:41:22.703448+0000","last_peered":"2024-06-29T14:41:22.703448+0000","last_clean":"2024-06-29T14:41:22.703448+0000","last_became_active":"2024-06-29T14:41:20.701346+0000","last_became_peered":"2024-06-29T14:41:20.701346+0000","last_unstale":"2024-06-29T14:41:22.703448+0000","last_undegraded":"2024-06-29T14:41:22.703448+0000","last_fullsized":"2024-06-29T14:41:22.703448+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:09.791011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253172+0000","last_change":"2024-06-29T14:41:21.722475+0000","last_active":"2024-06-29T14:41:23.253172+0000","last_peered":"2024-06-29T14:41:23.253172+0000","last_clean":"2024-06-29T14:41:23.253172+0000","last_became_active":"2024-06-29T14:41:21.722101+0000","last_became_peered":"2024-06-29T14:41:21.722101+0000","last_unstale":"2024-06-29T14:41:23.253172+0000","last_undegraded":"2024-06-29T14:41:23.253172+0000","last_fullsized":"2024-06-29T14:41:23.253172+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:34.531265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"132'2","reported_seq":16,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699850+0000","last_change":"2024-06-29T14:41:20.700389+0000","last_active":"2024-06-29T14:41:22.699850+0000","last_peered":"2024-06-29T14:41:22.699850+0000","last_clean":"2024-06-29T14:41:22.699850+0000","last_became_active":"2024-06-29T14:41:20.700035+0000","last_became_peered":"2024-06-29T14:41:20.700035+0000","last_unstale":"2024-06-29T14:41:22.699850+0000","last_undegraded":"2024-06-29T14:41:22.699850+0000","last_fullsized":"2024-06-29T14:41:22.699850+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:43:02.057931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.732078+0000","last_change":"2024-06-29T14:41:21.724536+0000","last_active":"2024-06-29T14:41:21.732078+0000","last_peered":"2024-06-29T14:41:21.732078+0000","last_clean":"2024-06-29T14:41:21.732078+0000","last_became_active":"2024-06-29T14:41:21.724211+0000","last_became_peered":"2024-06-29T14:41:21.724211+0000","last_unstale":"2024-06-29T14:41:21.732078+0000","last_undegraded":"2024-06-29T14:41:21.732078+0000","last_fullsized":"2024-06-29T14:41:21.732078+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:10.799138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.700588+0000","last_change":"2024-06-29T14:41:20.701456+0000","last_active":"2024-06-29T14:41:21.700588+0000","last_peered":"2024-06-29T14:41:21.700588+0000","last_clean":"2024-06-29T14:41:21.700588+0000","last_became_active":"2024-06-29T14:41:20.701136+0000","last_became_peered":"2024-06-29T14:41:20.701136+0000","last_unstale":"2024-06-29T14:41:21.700588+0000","last_undegraded":"2024-06-29T14:41:21.700588+0000","last_fullsized":"2024-06-29T14:41:21.700588+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:02.438990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253352+0000","last_change":"2024-06-29T14:41:21.728475+0000","last_active":"2024-06-29T14:41:23.253352+0000","last_peered":"2024-06-29T14:41:23.253352+0000","last_clean":"2024-06-29T14:41:23.253352+0000","last_became_active":"2024-06-29T14:41:21.727909+0000","last_became_peered":"2024-06-29T14:41:21.727909+0000","last_unstale":"2024-06-29T14:41:23.253352+0000","last_undegraded":"2024-06-29T14:41:23.253352+0000","last_fullsized":"2024-06-29T14:41:23.253352+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:09.771556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.699523+0000","last_change":"2024-06-29T14:41:20.697326+0000","last_active":"2024-06-29T14:41:22.699523+0000","last_peered":"2024-06-29T14:41:22.699523+0000","last_clean":"2024-06-29T14:41:22.699523+0000","last_became_active":"2024-06-29T14:41:20.696991+0000","last_became_peered":"2024-06-29T14:41:20.696991+0000","last_unstale":"2024-06-29T14:41:22.699523+0000","last_undegraded":"2024-06-29T14:41:22.699523+0000","last_fullsized":"2024-06-29T14:41:22.699523+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:21.138223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.721564+0000","last_change":"2024-06-29T14:41:21.716801+0000","last_active":"2024-06-29T14:41:21.721564+0000","last_peered":"2024-06-29T14:41:21.721564+0000","last_clean":"2024-06-29T14:41:21.721564+0000","last_became_active":"2024-06-29T14:41:21.716380+0000","last_became_peered":"2024-06-29T14:41:21.716380+0000","last_unstale":"2024-06-29T14:41:21.721564+0000","last_undegraded":"2024-06-29T14:41:21.721564+0000","last_fullsized":"2024-06-29T14:41:21.721564+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:31:24.510113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.709884+0000","last_change":"2024-06-29T14:41:20.707303+0000","last_active":"2024-06-29T14:41:21.709884+0000","last_peered":"2024-06-29T14:41:21.709884+0000","last_clean":"2024-06-29T14:41:21.709884+0000","last_became_active":"2024-06-29T14:41:20.705129+0000","last_became_peered":"2024-06-29T14:41:20.705129+0000","last_unstale":"2024-06-29T14:41:21.709884+0000","last_undegraded":"2024-06-29T14:41:21.709884+0000","last_fullsized":"2024-06-29T14:41:21.709884+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:28.206776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717497+0000","last_change":"2024-06-29T14:41:21.722454+0000","last_active":"2024-06-29T14:41:22.717497+0000","last_peered":"2024-06-29T14:41:22.717497+0000","last_clean":"2024-06-29T14:41:22.717497+0000","last_became_active":"2024-06-29T14:41:21.721683+0000","last_became_peered":"2024-06-29T14:41:21.721683+0000","last_unstale":"2024-06-29T14:41:22.717497+0000","last_undegraded":"2024-06-29T14:41:22.717497+0000","last_fullsized":"2024-06-29T14:41:22.717497+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:56:27.444549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"132'2","reported_seq":14,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.732516+0000","last_change":"2024-06-29T14:41:20.709603+0000","last_active":"2024-06-29T14:41:21.732516+0000","last_peered":"2024-06-29T14:41:21.732516+0000","last_clean":"2024-06-29T14:41:21.732516+0000","last_became_active":"2024-06-29T14:41:20.709283+0000","last_became_peered":"2024-06-29T14:41:20.709283+0000","last_unstale":"2024-06-29T14:41:21.732516+0000","last_undegraded":"2024-06-29T14:41:21.732516+0000","last_fullsized":"2024-06-29T14:41:21.732516+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:06:14.117393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.717435+0000","last_change":"2024-06-29T14:41:21.714812+0000","last_active":"2024-06-29T14:41:21.717435+0000","last_peered":"2024-06-29T14:41:21.717435+0000","last_clean":"2024-06-29T14:41:21.717435+0000","last_became_active":"2024-06-29T14:41:21.714400+0000","last_became_peered":"2024-06-29T14:41:21.714400+0000","last_unstale":"2024-06-29T14:41:21.717435+0000","last_undegraded":"2024-06-29T14:41:21.717435+0000","last_fullsized":"2024-06-29T14:41:21.717435+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:31.099410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"133'1","reported_seq":15,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.720992+0000","last_change":"2024-06-29T14:41:20.705032+0000","last_active":"2024-06-29T14:41:22.720992+0000","last_peered":"2024-06-29T14:41:22.720992+0000","last_clean":"2024-06-29T14:41:22.720992+0000","last_became_active":"2024-06-29T14:41:20.704678+0000","last_became_peered":"2024-06-29T14:41:20.704678+0000","last_unstale":"2024-06-29T14:41:22.720992+0000","last_undegraded":"2024-06-29T14:41:22.720992+0000","last_fullsized":"2024-06-29T14:41:22.720992+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:31.737350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.723322+0000","last_change":"2024-06-29T14:41:21.718987+0000","last_active":"2024-06-29T14:41:21.723322+0000","last_peered":"2024-06-29T14:41:21.723322+0000","last_clean":"2024-06-29T14:41:21.723322+0000","last_became_active":"2024-06-29T14:41:21.717554+0000","last_became_peered":"2024-06-29T14:41:21.717554+0000","last_unstale":"2024-06-29T14:41:21.723322+0000","last_undegraded":"2024-06-29T14:41:21.723322+0000","last_fullsized":"2024-06-29T14:41:21.723322+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:46:32.284666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.713719+0000","last_change":"2024-06-29T14:41:20.704958+0000","last_active":"2024-06-29T14:41:21.713719+0000","last_peered":"2024-06-29T14:41:21.713719+0000","last_clean":"2024-06-29T14:41:21.713719+0000","last_became_active":"2024-06-29T14:41:20.704637+0000","last_became_peered":"2024-06-29T14:41:20.704637+0000","last_unstale":"2024-06-29T14:41:21.713719+0000","last_undegraded":"2024-06-29T14:41:21.713719+0000","last_fullsized":"2024-06-29T14:41:21.713719+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:09:58.320331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.717636+0000","last_change":"2024-06-29T14:41:21.724392+0000","last_active":"2024-06-29T14:41:22.717636+0000","last_peered":"2024-06-29T14:41:22.717636+0000","last_clean":"2024-06-29T14:41:22.717636+0000","last_became_active":"2024-06-29T14:41:21.721839+0000","last_became_peered":"2024-06-29T14:41:21.721839+0000","last_unstale":"2024-06-29T14:41:22.717636+0000","last_undegraded":"2024-06-29T14:41:22.717636+0000","last_fullsized":"2024-06-29T14:41:22.717636+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:33.257760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.702276+0000","last_change":"2024-06-29T14:41:20.693255+0000","last_active":"2024-06-29T14:41:21.702276+0000","last_peered":"2024-06-29T14:41:21.702276+0000","last_clean":"2024-06-29T14:41:21.702276+0000","last_became_active":"2024-06-29T14:41:20.692912+0000","last_became_peered":"2024-06-29T14:41:20.692912+0000","last_unstale":"2024-06-29T14:41:21.702276+0000","last_undegraded":"2024-06-29T14:41:21.702276+0000","last_fullsized":"2024-06-29T14:41:21.702276+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:33:59.477398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702601+0000","last_change":"2024-06-29T14:41:21.709493+0000","last_active":"2024-06-29T14:41:22.702601+0000","last_peered":"2024-06-29T14:41:22.702601+0000","last_clean":"2024-06-29T14:41:22.702601+0000","last_became_active":"2024-06-29T14:41:21.709179+0000","last_became_peered":"2024-06-29T14:41:21.709179+0000","last_unstale":"2024-06-29T14:41:22.702601+0000","last_undegraded":"2024-06-29T14:41:22.702601+0000","last_fullsized":"2024-06-29T14:41:22.702601+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:38.251709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703559+0000","last_change":"2024-06-29T14:41:20.701417+0000","last_active":"2024-06-29T14:41:22.703559+0000","last_peered":"2024-06-29T14:41:22.703559+0000","last_clean":"2024-06-29T14:41:22.703559+0000","last_became_active":"2024-06-29T14:41:20.701105+0000","last_became_peered":"2024-06-29T14:41:20.701105+0000","last_unstale":"2024-06-29T14:41:22.703559+0000","last_undegraded":"2024-06-29T14:41:22.703559+0000","last_fullsized":"2024-06-29T14:41:22.703559+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:04.844487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.702969+0000","last_change":"2024-06-29T14:41:21.711170+0000","last_active":"2024-06-29T14:41:22.702969+0000","last_peered":"2024-06-29T14:41:22.702969+0000","last_clean":"2024-06-29T14:41:22.702969+0000","last_became_active":"2024-06-29T14:41:21.710801+0000","last_became_peered":"2024-06-29T14:41:21.710801+0000","last_unstale":"2024-06-29T14:41:22.702969+0000","last_undegraded":"2024-06-29T14:41:22.702969+0000","last_fullsized":"2024-06-29T14:41:22.702969+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:52:27.230538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703519+0000","last_change":"2024-06-29T14:41:20.694502+0000","last_active":"2024-06-29T14:41:22.703519+0000","last_peered":"2024-06-29T14:41:22.703519+0000","last_clean":"2024-06-29T14:41:22.703519+0000","last_became_active":"2024-06-29T14:41:20.694242+0000","last_became_peered":"2024-06-29T14:41:20.694242+0000","last_unstale":"2024-06-29T14:41:22.703519+0000","last_undegraded":"2024-06-29T14:41:22.703519+0000","last_fullsized":"2024-06-29T14:41:22.703519+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:59.455236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253392+0000","last_change":"2024-06-29T14:41:21.725677+0000","last_active":"2024-06-29T14:41:23.253392+0000","last_peered":"2024-06-29T14:41:23.253392+0000","last_clean":"2024-06-29T14:41:23.253392+0000","last_became_active":"2024-06-29T14:41:21.725165+0000","last_became_peered":"2024-06-29T14:41:21.725165+0000","last_unstale":"2024-06-29T14:41:23.253392+0000","last_undegraded":"2024-06-29T14:41:23.253392+0000","last_fullsized":"2024-06-29T14:41:23.253392+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:17.983080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.700816+0000","last_change":"2024-06-29T14:41:20.696645+0000","last_active":"2024-06-29T14:41:21.700816+0000","last_peered":"2024-06-29T14:41:21.700816+0000","last_clean":"2024-06-29T14:41:21.700816+0000","last_became_active":"2024-06-29T14:41:20.696283+0000","last_became_peered":"2024-06-29T14:41:20.696283+0000","last_unstale":"2024-06-29T14:41:21.700816+0000","last_undegraded":"2024-06-29T14:41:21.700816+0000","last_fullsized":"2024-06-29T14:41:21.700816+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:42.993335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.713838+0000","last_change":"2024-06-29T14:41:21.708325+0000","last_active":"2024-06-29T14:41:21.713838+0000","last_peered":"2024-06-29T14:41:21.713838+0000","last_clean":"2024-06-29T14:41:21.713838+0000","last_became_active":"2024-06-29T14:41:21.707948+0000","last_became_peered":"2024-06-29T14:41:21.707948+0000","last_unstale":"2024-06-29T14:41:21.713838+0000","last_undegraded":"2024-06-29T14:41:21.713838+0000","last_fullsized":"2024-06-29T14:41:21.713838+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:17:33.257760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.710682+0000","last_change":"2024-06-29T14:41:20.707556+0000","last_active":"2024-06-29T14:41:21.710682+0000","last_peered":"2024-06-29T14:41:21.710682+0000","last_clean":"2024-06-29T14:41:21.710682+0000","last_became_active":"2024-06-29T14:41:20.705784+0000","last_became_peered":"2024-06-29T14:41:20.705784+0000","last_unstale":"2024-06-29T14:41:21.710682+0000","last_undegraded":"2024-06-29T14:41:21.710682+0000","last_fullsized":"2024-06-29T14:41:21.710682+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:16.054121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-06-29T14:41:21.726177+0000","last_change":"2024-06-29T14:41:21.720836+0000","last_active":"2024-06-29T14:41:21.726177+0000","last_peered":"2024-06-29T14:41:21.726177+0000","last_clean":"2024-06-29T14:41:21.726177+0000","last_became_active":"2024-06-29T14:41:21.720541+0000","last_became_peered":"2024-06-29T14:41:21.720541+0000","last_unstale":"2024-06-29T14:41:21.726177+0000","last_undegraded":"2024-06-29T14:41:21.726177+0000","last_fullsized":"2024-06-29T14:41:21.726177+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":131,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:20.678804+0000","last_clean_scrub_stamp":"2024-06-29T14:41:20.678804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:24:24.312386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:22.703403+0000","last_change":"2024-06-29T14:41:20.705572+0000","last_active":"2024-06-29T14:41:22.703403+0000","last_peered":"2024-06-29T14:41:22.703403+0000","last_clean":"2024-06-29T14:41:22.703403+0000","last_became_active":"2024-06-29T14:41:20.705255+0000","last_became_peered":"2024-06-29T14:41:20.705255+0000","last_unstale":"2024-06-29T14:41:22.703403+0000","last_undegraded":"2024-06-29T14:41:22.703403+0000","last_fullsized":"2024-06-29T14:41:22.703403+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":130,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:19.670677+0000","last_clean_scrub_stamp":"2024-06-29T14:41:19.670677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:52:26.222412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"122'142","reported_seq":579,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-06-29T14:41:23.253438+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:41:23.253438+0000","last_peered":"2024-06-29T14:41:23.253438+0000","last_clean":"2024-06-29T14:41:23.253438+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:41:23.253438+0000","last_undegraded":"2024-06-29T14:41:23.253438+0000","last_fullsized":"2024-06-29T14:41:23.253438+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":221,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":666,"num_read_kb":1815,"num_write":221,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142,"ondisk_log_size":142,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575076,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33368,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338472,"statfs":{"total":96636764160,"available":96602595328,"internally_reserved":0,"allocated":2240512,"data_stored":1378912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575077,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36960,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334880,"statfs":{"total":96636764160,"available":96598917120,"internally_reserved":0,"allocated":1654784,"data_stored":794398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575075,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36948,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334892,"statfs":{"total":96636764160,"available":96598929408,"internally_reserved":0,"allocated":1646592,"data_stored":788532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575076,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59164,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312676,"statfs":{"total":96636764160,"available":96576180224,"internally_reserved":0,"allocated":1650688,"data_stored":794328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575076,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36960,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334880,"statfs":{"total":96636764160,"available":96598917120,"internally_reserved":0,"allocated":1654784,"data_stored":789610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575076,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41104,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330736,"statfs":{"total":96636764160,"available":96594673664,"internally_reserved":0,"allocated":1638400,"data_stored":787928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575075,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54996,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316844,"statfs":{"total":96636764160,"available":96580448256,"internally_reserved":0,"allocated":1646592,"data_stored":788052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575076,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33388,"kb_used_data":2200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338452,"statfs":{"total":96636764160,"available":96602574848,"internally_reserved":0,"allocated":2252800,"data_stored":1380020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:41:26.113 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:41:26.114 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:41:26.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.273+0000 7fe81b66d640 1 -- 172.21.15.81:0/1793329957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8140b35b0 msgr2=0x7fe81415b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.273+0000 7fe81b66d640 1 --2- 172.21.15.81:0/1793329957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8140b35b0 0x7fe81415b5b0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fe8140cc260 tx=0x7fe80402a4a0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.273+0000 7fe81b66d640 1 -- 172.21.15.81:0/1793329957 shutdown_connections 2024-06-29T14:41:26.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.273+0000 7fe81b66d640 1 --2- 172.21.15.81:0/1793329957 >> v2:172.21.15.167:3300/0 conn(0x7fe8140b4430 0x7fe8140cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.273+0000 7fe81b66d640 1 --2- 172.21.15.81:0/1793329957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8140b35b0 0x7fe81415b5b0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.273+0000 7fe81b66d640 1 -- 172.21.15.81:0/1793329957 >> 172.21.15.81:0/1793329957 conn(0x7fe81406ccc0 msgr2=0x7fe8140b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:26.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.273+0000 7fe81b66d640 1 -- 172.21.15.81:0/1793329957 shutdown_connections 2024-06-29T14:41:26.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.273+0000 7fe81b66d640 1 -- 172.21.15.81:0/1793329957 wait complete. 2024-06-29T14:41:26.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.275+0000 7fe81b66d640 1 Processor -- start 2024-06-29T14:41:26.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.275+0000 7fe81b66d640 1 -- start start 2024-06-29T14:41:26.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe81b66d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe81406f660 0x7fe8140b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe81b66d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8140b35b0 0x7fe8140aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe81b66d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8140b4430 0x7fe814159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe81b66d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe81406f1e0 con 0x7fe81406f660 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe81b66d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe81406f060 con 0x7fe8140b4430 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe81b66d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe81406f360 con 0x7fe8140b35b0 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe819409640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe81406f660 0x7fe8140b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe819409640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe81406f660 0x7fe8140b0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48126/0 (socket says 172.21.15.81:48126) 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe819409640 1 -- 172.21.15.81:0/1356569394 learned_addr learned my addr 172.21.15.81:0/1356569394 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe819c0a640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8140b4430 0x7fe814159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe818c08640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8140b35b0 0x7fe8140aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe819409640 1 -- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8140b35b0 msgr2=0x7fe8140aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe819409640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8140b35b0 0x7fe8140aea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe819409640 1 -- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8140b4430 msgr2=0x7fe814159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe819409640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8140b4430 0x7fe814159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.276+0000 7fe819409640 1 -- 172.21.15.81:0/1356569394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8140aef60 con 0x7fe81406f660 2024-06-29T14:41:26.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.277+0000 7fe819409640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe81406f660 0x7fe8140b0930 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fe8040098f0 tx=0x7fe804033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:26.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.277+0000 7fe811ffb640 1 -- 172.21.15.81:0/1356569394 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804031d60 con 0x7fe81406f660 2024-06-29T14:41:26.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.277+0000 7fe811ffb640 1 -- 172.21.15.81:0/1356569394 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe804033e10 con 0x7fe81406f660 2024-06-29T14:41:26.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.277+0000 7fe81b66d640 1 -- 172.21.15.81:0/1356569394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8141663c0 con 0x7fe81406f660 2024-06-29T14:41:26.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.277+0000 7fe811ffb640 1 -- 172.21.15.81:0/1356569394 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804005260 con 0x7fe81406f660 2024-06-29T14:41:26.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.277+0000 7fe81b66d640 1 -- 172.21.15.81:0/1356569394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe814166680 con 0x7fe81406f660 2024-06-29T14:41:26.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.278+0000 7fe81b66d640 1 -- 172.21.15.81:0/1356569394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe81415b5b0 con 0x7fe81406f660 2024-06-29T14:41:26.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.279+0000 7fe811ffb640 1 -- 172.21.15.81:0/1356569394 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fe804005400 con 0x7fe81406f660 2024-06-29T14:41:26.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.279+0000 7fe811ffb640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fe8000763f0 0x7fe800078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.279+0000 7fe811ffb640 1 -- 172.21.15.81:0/1356569394 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7fe8040b8d60 con 0x7fe81406f660 2024-06-29T14:41:26.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.279+0000 7fe818c08640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fe8000763f0 0x7fe800078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.280+0000 7fe818c08640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fe8000763f0 0x7fe800078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe8140cc0f0 tx=0x7fe808005be0 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:26.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.284+0000 7fe811ffb640 1 -- 172.21.15.81:0/1356569394 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe80403f030 con 0x7fe81406f660 2024-06-29T14:41:26.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.533+0000 7fe81b66d640 1 -- 172.21.15.81:0/1356569394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fe814149c10 con 0x7fe81406f660 2024-06-29T14:41:26.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.535+0000 7fe811ffb640 1 -- 172.21.15.81:0/1356569394 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v36) ==== 190+0+147 (secure 0 0 0) 0x7fe8040823a0 con 0x7fe81406f660 2024-06-29T14:41:26.534 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T14:41:26.534 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T14:41:26.534 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T14:41:26.534 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T14:41:26.534 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:41:26.534 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T14:41:26.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 -- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fe8000763f0 msgr2=0x7fe800078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fe8000763f0 0x7fe800078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe8140cc0f0 tx=0x7fe808005be0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 -- 172.21.15.81:0/1356569394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe81406f660 msgr2=0x7fe8140b0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe81406f660 0x7fe8140b0930 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fe8040098f0 tx=0x7fe804033c20 comp rx=0 tx=0).stop 2024-06-29T14:41:26.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 -- 172.21.15.81:0/1356569394 shutdown_connections 2024-06-29T14:41:26.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fe8000763f0 0x7fe800078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8140b4430 0x7fe814159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8140b35b0 0x7fe8140aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 --2- 172.21.15.81:0/1356569394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe81406f660 0x7fe8140b0930 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 -- 172.21.15.81:0/1356569394 >> 172.21.15.81:0/1356569394 conn(0x7fe81406ccc0 msgr2=0x7fe814149160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:26.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 -- 172.21.15.81:0/1356569394 shutdown_connections 2024-06-29T14:41:26.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.538+0000 7fe81b66d640 1 -- 172.21.15.81:0/1356569394 wait complete. 2024-06-29T14:41:26.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:26.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.662+0000 7f7171b3c640 1 -- 172.21.15.81:0/4022081634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7154003a30 msgr2=0x7f7154003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.662+0000 7f7171b3c640 1 --2- 172.21.15.81:0/4022081634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7154003a30 0x7f7154003ea0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f715c001dc0 tx=0x7f7154005830 comp rx=0 tx=0).stop 2024-06-29T14:41:26.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.662+0000 7f716bfff640 1 -- 172.21.15.81:0/4022081634 reap_dead start 2024-06-29T14:41:26.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.662+0000 7f7171b3c640 1 -- 172.21.15.81:0/4022081634 shutdown_connections 2024-06-29T14:41:26.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.662+0000 7f7171b3c640 1 -- 172.21.15.81:0/4022081634 >> 172.21.15.81:0/4022081634 conn(0x7f716c06ccc0 msgr2=0x7f716c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:26.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.662+0000 7f7171b3c640 1 -- 172.21.15.81:0/4022081634 shutdown_connections 2024-06-29T14:41:26.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.662+0000 7f7171b3c640 1 -- 172.21.15.81:0/4022081634 wait complete. 2024-06-29T14:41:26.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.664+0000 7f7171b3c640 1 Processor -- start 2024-06-29T14:41:26.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.664+0000 7f7171b3c640 1 -- start start 2024-06-29T14:41:26.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7171b3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7154003a30 0x7f716c159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7171b3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f716c13b100 0x7f716c140420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7171b3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f716c13b510 0x7f716c140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7171b3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f716c164c30 con 0x7f716c13b100 2024-06-29T14:41:26.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7171b3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f716c164ab0 con 0x7f716c13b510 2024-06-29T14:41:26.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7171b3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f716c164db0 con 0x7f7154003a30 2024-06-29T14:41:26.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f716c13b100 0x7f716c140420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f716c13b100 0x7f716c140420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48138/0 (socket says 172.21.15.81:48138) 2024-06-29T14:41:26.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716affd640 1 -- 172.21.15.81:0/1420205156 learned_addr learned my addr 172.21.15.81:0/1420205156 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:26.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716b7fe640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7154003a30 0x7f716c159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716bfff640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f716c13b510 0x7f716c140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716affd640 1 -- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7154003a30 msgr2=0x7f716c159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716affd640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7154003a30 0x7f716c159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716affd640 1 -- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f716c13b510 msgr2=0x7f716c140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716affd640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f716c13b510 0x7f716c140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716affd640 1 -- 172.21.15.81:0/1420205156 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716c156730 con 0x7f716c13b100 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716bfff640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f716c13b510 0x7f716c140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f716affd640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f716c13b100 0x7f716c140420 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f716000e590 tx=0x7f716000ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7153fff640 1 -- 172.21.15.81:0/1420205156 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7160015070 con 0x7f716c13b100 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7153fff640 1 -- 172.21.15.81:0/1420205156 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7160010980 con 0x7f716c13b100 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7171b3c640 1 -- 172.21.15.81:0/1420205156 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716c156170 con 0x7f716c13b100 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7153fff640 1 -- 172.21.15.81:0/1420205156 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7160010dd0 con 0x7f716c13b100 2024-06-29T14:41:26.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.666+0000 7f7171b3c640 1 -- 172.21.15.81:0/1420205156 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716c141b20 con 0x7f716c13b100 2024-06-29T14:41:26.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.668+0000 7f7171b3c640 1 -- 172.21.15.81:0/1420205156 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f716c155e90 con 0x7f716c13b100 2024-06-29T14:41:26.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.668+0000 7f7153fff640 1 -- 172.21.15.81:0/1420205156 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f7160018070 con 0x7f716c13b100 2024-06-29T14:41:26.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.669+0000 7f7153fff640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f71580763f0 0x7f7158078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:26.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.669+0000 7f7153fff640 1 -- 172.21.15.81:0/1420205156 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7f7160097da0 con 0x7f716c13b100 2024-06-29T14:41:26.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.669+0000 7f716b7fe640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f71580763f0 0x7f7158078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:26.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.669+0000 7f716b7fe640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f71580763f0 0x7f7158078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f715c032000 tx=0x7f715c001510 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:26.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.673+0000 7f7153fff640 1 -- 172.21.15.81:0/1420205156 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7160060da0 con 0x7f716c13b100 2024-06-29T14:41:26.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.890+0000 7f7171b3c640 1 -- 172.21.15.81:0/1420205156 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f716c15de00 con 0x7f716c13b100 2024-06-29T14:41:26.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.890+0000 7f7153fff640 1 -- 172.21.15.81:0/1420205156 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5228 (secure 0 0 0) 0x7f71600693e0 con 0x7f716c13b100 2024-06-29T14:41:26.889 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:26.889 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":68,"btime":"2024-06-29T14:41:24:728024+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/4097483790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":4097483790},{"type":"v1","addr":"172.21.15.167:6837","nonce":4097483790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1957730707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1957730707},{"type":"v1","addr":"172.21.15.167:6836","nonce":1957730707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":8550,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3852254611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3852254611},{"type":"v1","addr":"172.21.15.81:6833","nonce":3852254611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:21.060976+0000","modified":"2024-06-29T14:41:24.093831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8553":{"gid":8553,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/942339953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":942339953},{"type":"v1","addr":"172.21.15.81:6835","nonce":942339953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8553,"qdb_cluster":[8553]},"id":8}]} 2024-06-29T14:41:26.889 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 68 2024-06-29T14:41:26.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 -- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f71580763f0 msgr2=0x7f7158078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f71580763f0 0x7f7158078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f715c032000 tx=0x7f715c001510 comp rx=0 tx=0).stop 2024-06-29T14:41:26.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 -- 172.21.15.81:0/1420205156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f716c13b100 msgr2=0x7f716c140420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:26.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f716c13b100 0x7f716c140420 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f716000e590 tx=0x7f716000ea50 comp rx=0 tx=0).stop 2024-06-29T14:41:26.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 -- 172.21.15.81:0/1420205156 shutdown_connections 2024-06-29T14:41:26.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f71580763f0 0x7f7158078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f716c13b510 0x7f716c140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f716c13b100 0x7f716c140420 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 --2- 172.21.15.81:0/1420205156 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7154003a30 0x7f716c159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:26.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 -- 172.21.15.81:0/1420205156 >> 172.21.15.81:0/1420205156 conn(0x7f716c06ccc0 msgr2=0x7f716c15a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:26.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 -- 172.21.15.81:0/1420205156 shutdown_connections 2024-06-29T14:41:26.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:26.892+0000 7f7171b3c640 1 -- 172.21.15.81:0/1420205156 wait complete. 2024-06-29T14:41:27.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:41:21.060976+0000', 'modified': '2024-06-29T14:41:24.093831+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8553': {'gid': 8553, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6835/942339953', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 942339953}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 942339953}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8553, 'qdb_cluster': [8553]} 2024-06-29T14:41:27.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:41:27.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T14:41:27.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.423+0000 7f77128fd640 1 -- 172.21.15.81:0/3354261139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76f40039e0 msgr2=0x7f76f4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.423+0000 7f77128fd640 1 --2- 172.21.15.81:0/3354261139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76f40039e0 0x7f76f4003e10 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009b50 tx=0x7f76f4005830 comp rx=0 tx=0).stop 2024-06-29T14:41:27.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.423+0000 7f7710e9a640 1 -- 172.21.15.81:0/3354261139 reap_dead start 2024-06-29T14:41:27.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.423+0000 7f77128fd640 1 -- 172.21.15.81:0/3354261139 shutdown_connections 2024-06-29T14:41:27.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.423+0000 7f77128fd640 1 -- 172.21.15.81:0/3354261139 >> 172.21.15.81:0/3354261139 conn(0x7f770c072950 msgr2=0x7f770c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.423+0000 7f77128fd640 1 -- 172.21.15.81:0/3354261139 shutdown_connections 2024-06-29T14:41:27.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.423+0000 7f77128fd640 1 -- 172.21.15.81:0/3354261139 wait complete. 2024-06-29T14:41:27.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.425+0000 7f77128fd640 1 Processor -- start 2024-06-29T14:41:27.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.425+0000 7f77128fd640 1 -- start start 2024-06-29T14:41:27.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f77128fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76f40039e0 0x7f770c153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f77128fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f770c09ba10 0x7f770c0b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f77128fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f770c0b11e0 0x7f770c09c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f77128fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f770c15b190 con 0x7f770c09ba10 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f77128fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f770c15b010 con 0x7f770c0b11e0 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f77128fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f770c0af310 con 0x7f76f40039e0 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f770b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f770c09ba10 0x7f770c0b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f770b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f770c09ba10 0x7f770c0b0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48158/0 (socket says 172.21.15.81:48158) 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f770b7fe640 1 -- 172.21.15.81:0/748861034 learned_addr learned my addr 172.21.15.81:0/748861034 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f770bfff640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76f40039e0 0x7f770c153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f7710e9a640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f770c0b11e0 0x7f770c09c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f770b7fe640 1 -- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76f40039e0 msgr2=0x7f770c153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f770b7fe640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76f40039e0 0x7f770c153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f770b7fe640 1 -- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f770c0b11e0 msgr2=0x7f770c09c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f770b7fe640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f770c0b11e0 0x7f770c09c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f770b7fe640 1 -- 172.21.15.81:0/748861034 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c0b15f0 con 0x7f770c09ba10 2024-06-29T14:41:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f7710e9a640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f770c0b11e0 0x7f770c09c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f770b7fe640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f770c09ba10 0x7f770c0b0ca0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f77000066b0 tx=0x7f77000039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f7708ff9640 1 -- 172.21.15.81:0/748861034 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700015070 con 0x7f770c09ba10 2024-06-29T14:41:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f7708ff9640 1 -- 172.21.15.81:0/748861034 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7700011920 con 0x7f770c09ba10 2024-06-29T14:41:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.427+0000 7f7708ff9640 1 -- 172.21.15.81:0/748861034 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700011d70 con 0x7f770c09ba10 2024-06-29T14:41:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.428+0000 7f77128fd640 1 -- 172.21.15.81:0/748861034 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770c1e1bd0 con 0x7f770c09ba10 2024-06-29T14:41:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.428+0000 7f77128fd640 1 -- 172.21.15.81:0/748861034 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770c1e07e0 con 0x7f770c09ba10 2024-06-29T14:41:27.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.429+0000 7f7708ff9640 1 -- 172.21.15.81:0/748861034 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f7700018070 con 0x7f770c09ba10 2024-06-29T14:41:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.430+0000 7f7708ff9640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f76f8076420 0x7f76f8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.430+0000 7f7708ff9640 1 -- 172.21.15.81:0/748861034 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7f7700098570 con 0x7f770c09ba10 2024-06-29T14:41:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.430+0000 7f770bfff640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f76f8076420 0x7f76f8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.431+0000 7f77197bb740 1 Processor -- start 2024-06-29T14:41:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.431+0000 7f770bfff640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f76f8076420 0x7f76f8078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f76fc035000 tx=0x7f76fc02d040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.431+0000 7f77197bb740 1 -- start start 2024-06-29T14:41:27.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.432+0000 7f77197bb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf41c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.432+0000 7f77197bb740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf0073b0 0x55fbdf4003a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.432+0000 7f77197bb740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf00f590 0x55fbdf4008e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.432+0000 7f77197bb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf398bc0 con 0x55fbdf030270 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.432+0000 7f77197bb740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf347630 con 0x55fbdf00f590 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.432+0000 7f77197bb740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55fbdf336bd0 con 0x55fbdf0073b0 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.432+0000 7f770bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf41c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.432+0000 7f770bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf41c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48170/0 (socket says 172.21.15.81:48170) 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.432+0000 7f770bfff640 1 -- 172.21.15.81:0/4144474583 learned_addr learned my addr 172.21.15.81:0/4144474583 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f770b7fe640 1 --2- 172.21.15.81:0/4144474583 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf0073b0 0x55fbdf4003a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f770bfff640 1 -- 172.21.15.81:0/4144474583 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf0073b0 msgr2=0x55fbdf4003a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f770bfff640 1 --2- 172.21.15.81:0/4144474583 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf0073b0 0x55fbdf4003a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f7710e9a640 1 --2- 172.21.15.81:0/4144474583 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf00f590 0x55fbdf4008e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f770bfff640 1 -- 172.21.15.81:0/4144474583 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf00f590 msgr2=0x55fbdf4008e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f770bfff640 1 --2- 172.21.15.81:0/4144474583 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf00f590 0x55fbdf4008e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f770bfff640 1 -- 172.21.15.81:0/4144474583 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbdf333ca0 con 0x55fbdf030270 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f770bfff640 1 --2- 172.21.15.81:0/4144474583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf41c6e0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f76fc02b4d0 tx=0x7f76f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6233f09ef76466 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f76f1ffb640 1 -- 172.21.15.81:0/4144474583 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc038040 con 0x55fbdf030270 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f76f1ffb640 1 -- 172.21.15.81:0/4144474583 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76fc044070 con 0x55fbdf030270 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.433+0000 7f76f1ffb640 1 -- 172.21.15.81:0/4144474583 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc03ead0 con 0x55fbdf030270 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.434+0000 7f77197bb740 1 -- 172.21.15.81:0/4144474583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 msgr2=0x55fbdf41c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.434+0000 7f77197bb740 1 --2- 172.21.15.81:0/4144474583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf41c6e0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f76fc02b4d0 tx=0x7f76f4005830 comp rx=0 tx=0).stop 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.434+0000 7f77197bb740 1 -- 172.21.15.81:0/4144474583 shutdown_connections 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.434+0000 7f77197bb740 1 --2- 172.21.15.81:0/4144474583 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf00f590 0x55fbdf4008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.434+0000 7f77197bb740 1 --2- 172.21.15.81:0/4144474583 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf0073b0 0x55fbdf4003a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.434+0000 7f77197bb740 1 --2- 172.21.15.81:0/4144474583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf41c6e0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.434+0000 7f77197bb740 1 -- 172.21.15.81:0/4144474583 >> 172.21.15.81:0/4144474583 conn(0x55fbdf0243c0 msgr2=0x55fbdf3d3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.434+0000 7f77197bb740 1 -- 172.21.15.81:0/4144474583 shutdown_connections 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.434+0000 7f77197bb740 1 -- 172.21.15.81:0/4144474583 wait complete. 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.435+0000 7f77197bb740 1 -- start start 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.435+0000 7f77197bb740 1 Processor -- start 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f77197bb740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf0073b0 0x55fbdf3feb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f77197bb740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf00f590 0x55fbdf42b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f77197bb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf42b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f77197bb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf2d4ae0 con 0x55fbdf030270 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f77197bb740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf33ac10 con 0x55fbdf0073b0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f77197bb740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55fbdf0aa620 con 0x55fbdf00f590 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f77197bb740 10 client.0 _ll_register_callbacks cb 0x55fbdf2e1480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f7710e9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf42b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f7710e9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf42b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48176/0 (socket says 172.21.15.81:48176) 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f7710e9a640 1 -- 172.21.15.81:0/2851007297 learned_addr learned my addr 172.21.15.81:0/2851007297 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f770bfff640 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf0073b0 0x55fbdf3feb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.436+0000 7f770b7fe640 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf00f590 0x55fbdf42b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f7710e9a640 1 -- 172.21.15.81:0/2851007297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf00f590 msgr2=0x55fbdf42b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f7710e9a640 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf00f590 0x55fbdf42b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f7710e9a640 1 -- 172.21.15.81:0/2851007297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf0073b0 msgr2=0x55fbdf3feb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f7710e9a640 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf0073b0 0x55fbdf3feb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f7710e9a640 1 -- 172.21.15.81:0/2851007297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbdf09b850 con 0x55fbdf030270 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f7710e9a640 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf42b770 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f770c0737a0 tx=0x7f7704006870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae123ccda668d12 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f76eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f76eaffd640 1 -- 172.21.15.81:0/2851007297 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704016070 con 0x55fbdf030270 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f76eaffd640 1 -- 172.21.15.81:0/2851007297 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7f7704012440 con 0x55fbdf030270 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f77197bb740 1 -- 172.21.15.81:0/2851007297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76fc044070 con 0x55fbdf030270 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f76eaffd640 1 -- 172.21.15.81:0/2851007297 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f7704021050 con 0x55fbdf030270 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f77197bb740 10 client.8682 fetch_fsmap learned FSMap version 68 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f77197bb740 1 -- 172.21.15.81:0/2851007297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55fbdf1126b0 con 0x55fbdf030270 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f76eaffd640 1 -- 172.21.15.81:0/2851007297 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76fc044070 con 0x55fbdf030270 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f76eaffd640 1 -- 172.21.15.81:0/2851007297 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 68) ==== 3502+0+0 (secure 0 0 0) 0x7f7704024020 con 0x55fbdf030270 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f77197bb740 10 client.8682 fetch_fsmap finished waiting for FSMap version 68 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f77197bb740 10 client.8682 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f77197bb740 20 client.8682 populate_metadata read hostname 'smithi081' 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f77197bb740 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.81:6834/942339953,v1:172.21.15.81:6835/942339953] conn(0x55fbdefeee40 0x55fbdf4388f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f77197bb740 4 client.8682 mds_command: new command op to 8553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.437+0000 7f77197bb740 1 -- 172.21.15.81:0/2851007297 --> [v2:172.21.15.81:6834/942339953,v1:172.21.15.81:6835/942339953] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbdf349580 con 0x55fbdefeee40 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.438+0000 7f770bfff640 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.81:6834/942339953,v1:172.21.15.81:6835/942339953] conn(0x55fbdefeee40 0x55fbdf4388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.438+0000 7f76eaffd640 10 client.8682 ms_handle_connect on v2:172.21.15.81:6834/942339953 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.438+0000 7f770bfff640 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.81:6834/942339953,v1:172.21.15.81:6835/942339953] conn(0x55fbdefeee40 0x55fbdf4388f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0edefb8e676bfbf server_cookie=553a6eab89871e18 in_seq=0 out_seq=0 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f76eaffd640 1 -- 172.21.15.81:0/2851007297 <== mds.0 v2:172.21.15.81:6834/942339953 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55fbdf349580 con 0x55fbdefeee40 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f76eaffd640 10 client.8682 handle_command_reply: tid=0 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 client.8682 shutdown 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 -- 172.21.15.81:0/2851007297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 msgr2=0x55fbdf42b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf42b770 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f770c0737a0 tx=0x7f7704006870 comp rx=0 tx=0).stop 2024-06-29T14:41:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 -- 172.21.15.81:0/2851007297 shutdown_connections 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.81:6834/942339953,v1:172.21.15.81:6835/942339953] conn(0x55fbdefeee40 0x55fbdf4388f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf42b770 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f76eaffd640 0 client.8682 ms_handle_reset on v2:172.21.15.81:6834/942339953 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf00f590 0x55fbdf42b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 --2- 172.21.15.81:0/2851007297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf0073b0 0x55fbdf3feb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 -- 172.21.15.81:0/2851007297 >> 172.21.15.81:0/2851007297 conn(0x55fbdf0243c0 msgr2=0x55fbdf3fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 -- 172.21.15.81:0/2851007297 shutdown_connections 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 1 -- 172.21.15.81:0/2851007297 wait complete. 2024-06-29T14:41:27.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.439+0000 7f77197bb740 20 client.8682 trim_cache size 0 max 16384 2024-06-29T14:41:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.458+0000 7f77197bb740 1 Processor -- start 2024-06-29T14:41:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.458+0000 7f77197bb740 1 -- start start 2024-06-29T14:41:27.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.459+0000 7f77197bb740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf0073b0 0x55fbdf3fb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.459+0000 7f77197bb740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf00f590 0x55fbdf3fdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.459+0000 7f77197bb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf3fe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.459+0000 7f77197bb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1fead0 con 0x55fbdf030270 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.459+0000 7f77197bb740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1555b0 con 0x55fbdf0073b0 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.459+0000 7f77197bb740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f770c1fe480 con 0x55fbdf00f590 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.459+0000 7f7710e9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf3fe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f7710e9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf3fe1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48186/0 (socket says 172.21.15.81:48186) 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f7710e9a640 1 -- 172.21.15.81:0/3444963704 learned_addr learned my addr 172.21.15.81:0/3444963704 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f770bfff640 1 --2- 172.21.15.81:0/3444963704 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf0073b0 0x55fbdf3fb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f770b7fe640 1 --2- 172.21.15.81:0/3444963704 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf00f590 0x55fbdf3fdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f7710e9a640 1 -- 172.21.15.81:0/3444963704 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf00f590 msgr2=0x55fbdf3fdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f7710e9a640 1 --2- 172.21.15.81:0/3444963704 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf00f590 0x55fbdf3fdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f7710e9a640 1 -- 172.21.15.81:0/3444963704 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf0073b0 msgr2=0x55fbdf3fb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f7710e9a640 1 --2- 172.21.15.81:0/3444963704 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf0073b0 0x55fbdf3fb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f7710e9a640 1 -- 172.21.15.81:0/3444963704 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbdf3fd440 con 0x55fbdf030270 2024-06-29T14:41:27.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f7710e9a640 1 --2- 172.21.15.81:0/3444963704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf3fe1e0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f7704001350 tx=0x7f7704006090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb93cc807a623ff4 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f76f17fa640 1 -- 172.21.15.81:0/3444963704 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704016040 con 0x55fbdf030270 2024-06-29T14:41:27.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f76f17fa640 1 -- 172.21.15.81:0/3444963704 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7704012830 con 0x55fbdf030270 2024-06-29T14:41:27.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.460+0000 7f76f17fa640 1 -- 172.21.15.81:0/3444963704 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704012db0 con 0x55fbdf030270 2024-06-29T14:41:27.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.461+0000 7f77197bb740 1 -- 172.21.15.81:0/3444963704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 msgr2=0x55fbdf3fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.461+0000 7f77197bb740 1 --2- 172.21.15.81:0/3444963704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf3fe1e0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f7704001350 tx=0x7f7704006090 comp rx=0 tx=0).stop 2024-06-29T14:41:27.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.461+0000 7f77197bb740 1 -- 172.21.15.81:0/3444963704 shutdown_connections 2024-06-29T14:41:27.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.461+0000 7f77197bb740 1 --2- 172.21.15.81:0/3444963704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf3fe1e0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.461+0000 7f77197bb740 1 --2- 172.21.15.81:0/3444963704 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf00f590 0x55fbdf3fdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.461+0000 7f77197bb740 1 --2- 172.21.15.81:0/3444963704 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf0073b0 0x55fbdf3fb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.461+0000 7f77197bb740 1 -- 172.21.15.81:0/3444963704 >> 172.21.15.81:0/3444963704 conn(0x55fbdf0243c0 msgr2=0x55fbdf3cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.461+0000 7f77197bb740 1 -- 172.21.15.81:0/3444963704 shutdown_connections 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.461+0000 7f77197bb740 1 -- 172.21.15.81:0/3444963704 wait complete. 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.462+0000 7f77197bb740 1 -- start start 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.462+0000 7f77197bb740 1 Processor -- start 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.463+0000 7f77197bb740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf0073b0 0x55fbdf432650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.463+0000 7f77197bb740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf00f590 0x55fbdf497eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.463+0000 7f77197bb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf4983f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.463+0000 7f77197bb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf42bf80 con 0x55fbdf030270 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.463+0000 7f77197bb740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55fbdf41ba20 con 0x55fbdf00f590 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.463+0000 7f77197bb740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55fbdf434ad0 con 0x55fbdf0073b0 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.463+0000 7f77197bb740 10 client.0 _ll_register_callbacks cb 0x7f7704004aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:41:27.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.463+0000 7f7710e9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf4983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.463+0000 7f7710e9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf4983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48196/0 (socket says 172.21.15.81:48196) 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.463+0000 7f7710e9a640 1 -- 172.21.15.81:0/1073182157 learned_addr learned my addr 172.21.15.81:0/1073182157 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f770b7fe640 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf00f590 0x55fbdf497eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f7710e9a640 1 -- 172.21.15.81:0/1073182157 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf0073b0 msgr2=0x55fbdf432650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f7710e9a640 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf0073b0 0x55fbdf432650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f7710e9a640 1 -- 172.21.15.81:0/1073182157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf00f590 msgr2=0x55fbdf497eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f7710e9a640 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf00f590 0x55fbdf497eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f7710e9a640 1 -- 172.21.15.81:0/1073182157 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbdf3fe690 con 0x55fbdf030270 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f7710e9a640 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf4983f0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f7704012610 tx=0x7f7704014f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7955b2e96d54f8b3 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f76eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f76eaffd640 1 -- 172.21.15.81:0/1073182157 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704016070 con 0x55fbdf030270 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f76eaffd640 1 -- 172.21.15.81:0/1073182157 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7f7704012d60 con 0x55fbdf030270 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f77197bb740 1 -- 172.21.15.81:0/1073182157 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7704012830 con 0x55fbdf030270 2024-06-29T14:41:27.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f76eaffd640 1 -- 172.21.15.81:0/1073182157 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f7704014070 con 0x55fbdf030270 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f77197bb740 10 client.8688 fetch_fsmap learned FSMap version 68 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f77197bb740 1 -- 172.21.15.81:0/1073182157 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55fbdf49ce30 con 0x55fbdf030270 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f76eaffd640 1 -- 172.21.15.81:0/1073182157 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7704012830 con 0x55fbdf030270 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f76eaffd640 1 -- 172.21.15.81:0/1073182157 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 68) ==== 3502+0+0 (secure 0 0 0) 0x7f7704024070 con 0x55fbdf030270 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f77197bb740 10 client.8688 fetch_fsmap finished waiting for FSMap version 68 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f77197bb740 10 client.8688 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f77197bb740 20 client.8688 populate_metadata read hostname 'smithi081' 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f77197bb740 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.81:6834/942339953,v1:172.21.15.81:6835/942339953] conn(0x55fbdf49c710 0x55fbdf4307a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f77197bb740 4 client.8688 mds_command: new command op to 8553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f77197bb740 1 -- 172.21.15.81:0/1073182157 --> [v2:172.21.15.81:6834/942339953,v1:172.21.15.81:6835/942339953] -- command(tid 0: {"prefix": "status"}) -- 0x55fbdf345020 con 0x55fbdf49c710 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.464+0000 7f770bfff640 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.81:6834/942339953,v1:172.21.15.81:6835/942339953] conn(0x55fbdf49c710 0x55fbdf4307a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f76eaffd640 10 client.8688 ms_handle_connect on v2:172.21.15.81:6834/942339953 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f770bfff640 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.81:6834/942339953,v1:172.21.15.81:6835/942339953] conn(0x55fbdf49c710 0x55fbdf4307a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddfde3c896f7274d server_cookie=af5102b021cd97e5 in_seq=0 out_seq=0 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f76eaffd640 1 -- 172.21.15.81:0/1073182157 <== mds.0 v2:172.21.15.81:6834/942339953 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55fbdf345020 con 0x55fbdf49c710 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f76eaffd640 10 client.8688 handle_command_reply: tid=0 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 client.8688 shutdown 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 -- 172.21.15.81:0/1073182157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 msgr2=0x55fbdf4983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf4983f0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f7704012610 tx=0x7f7704014f50 comp rx=0 tx=0).stop 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 -- 172.21.15.81:0/1073182157 shutdown_connections 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.81:6834/942339953,v1:172.21.15.81:6835/942339953] conn(0x55fbdf49c710 0x55fbdf4307a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fbdf030270 0x55fbdf4983f0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55fbdf00f590 0x55fbdf497eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f76eaffd640 0 client.8688 ms_handle_reset on v2:172.21.15.81:6834/942339953 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 --2- 172.21.15.81:0/1073182157 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55fbdf0073b0 0x55fbdf432650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 -- 172.21.15.81:0/1073182157 >> 172.21.15.81:0/1073182157 conn(0x55fbdf0243c0 msgr2=0x55fbdf3fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 -- 172.21.15.81:0/1073182157 shutdown_connections 2024-06-29T14:41:27.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.465+0000 7f77197bb740 1 -- 172.21.15.81:0/1073182157 wait complete. 2024-06-29T14:41:27.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77197bb740 20 client.8688 trim_cache size 0 max 16384 2024-06-29T14:41:27.470 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:41:27.470 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T14:41:27.470 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T14:41:27.470 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 8553, 2024-06-29T14:41:27.470 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T14:41:27.470 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T14:41:27.470 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T14:41:27.471 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7578134390000004, 2024-06-29T14:41:27.471 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 68, 2024-06-29T14:41:27.471 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 133, 2024-06-29T14:41:27.471 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 133, 2024-06-29T14:41:27.471 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.448408397 2024-06-29T14:41:27.471 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:41:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 -- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f76f8076420 msgr2=0x7f76f8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f76f8076420 0x7f76f8078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f76fc035000 tx=0x7f76fc02d040 comp rx=0 tx=0).stop 2024-06-29T14:41:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 -- 172.21.15.81:0/748861034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f770c09ba10 msgr2=0x7f770c0b0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f770c09ba10 0x7f770c0b0ca0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f77000066b0 tx=0x7f77000039c0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 -- 172.21.15.81:0/748861034 shutdown_connections 2024-06-29T14:41:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f76f8076420 0x7f76f8078930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f770c0b11e0 0x7f770c09c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f770c09ba10 0x7f770c0b0ca0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 --2- 172.21.15.81:0/748861034 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76f40039e0 0x7f770c153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 -- 172.21.15.81:0/748861034 >> 172.21.15.81:0/748861034 conn(0x7f770c072950 msgr2=0x7f770c09a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 -- 172.21.15.81:0/748861034 shutdown_connections 2024-06-29T14:41:27.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.466+0000 7f77128fd640 1 -- 172.21.15.81:0/748861034 wait complete. 2024-06-29T14:41:27.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:41:27.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.591+0000 7fba44b2e640 1 -- 172.21.15.81:0/2728255501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba2c003a00 msgr2=0x7fba2c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.591+0000 7fba44b2e640 1 --2- 172.21.15.81:0/2728255501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba2c003a00 0x7fba2c003e30 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fba30009b50 tx=0x7fba2c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:27.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.591+0000 7fba44b2e640 1 -- 172.21.15.81:0/2728255501 shutdown_connections 2024-06-29T14:41:27.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.591+0000 7fba44b2e640 1 --2- 172.21.15.81:0/2728255501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba2c003a00 0x7fba2c003e30 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.591+0000 7fba44b2e640 1 -- 172.21.15.81:0/2728255501 >> 172.21.15.81:0/2728255501 conn(0x7fba40072950 msgr2=0x7fba400b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.591+0000 7fba44b2e640 1 -- 172.21.15.81:0/2728255501 shutdown_connections 2024-06-29T14:41:27.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.592+0000 7fba44b2e640 1 -- 172.21.15.81:0/2728255501 wait complete. 2024-06-29T14:41:27.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.594+0000 7fba44b2e640 1 Processor -- start 2024-06-29T14:41:27.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.594+0000 7fba44b2e640 1 -- start start 2024-06-29T14:41:27.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba44b2e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba2c003a00 0x7fba401f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba44b2e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fba4006fe20 0x7fba40150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba44b2e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba40070230 0x7fba40151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba44b2e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fba4015bdf0 con 0x7fba40070230 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba44b2e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fba40074460 con 0x7fba2c003a00 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba44b2e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fba4015bf70 con 0x7fba4006fe20 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba3ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba40070230 0x7fba40151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba3ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba40070230 0x7fba40151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48220/0 (socket says 172.21.15.81:48220) 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba3ed9d640 1 -- 172.21.15.81:0/2202717487 learned_addr learned my addr 172.21.15.81:0/2202717487 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba35d9b640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fba4006fe20 0x7fba40150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba3e59c640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba2c003a00 0x7fba401f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba3ed9d640 1 -- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fba4006fe20 msgr2=0x7fba40150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba3ed9d640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fba4006fe20 0x7fba40150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba3ed9d640 1 -- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba2c003a00 msgr2=0x7fba401f43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba3ed9d640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba2c003a00 0x7fba401f43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba3ed9d640 1 -- 172.21.15.81:0/2202717487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4006ec50 con 0x7fba40070230 2024-06-29T14:41:27.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.595+0000 7fba3ed9d640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba40070230 0x7fba40151010 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fba40072520 tx=0x7fba38001b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.596+0000 7fba377fe640 1 -- 172.21.15.81:0/2202717487 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba38010890 con 0x7fba40070230 2024-06-29T14:41:27.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.596+0000 7fba377fe640 1 -- 172.21.15.81:0/2202717487 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba38016070 con 0x7fba40070230 2024-06-29T14:41:27.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.596+0000 7fba377fe640 1 -- 172.21.15.81:0/2202717487 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba38010de0 con 0x7fba40070230 2024-06-29T14:41:27.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.596+0000 7fba44b2e640 1 -- 172.21.15.81:0/2202717487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4006ef30 con 0x7fba40070230 2024-06-29T14:41:27.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.596+0000 7fba44b2e640 1 -- 172.21.15.81:0/2202717487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4006f4f0 con 0x7fba40070230 2024-06-29T14:41:27.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.597+0000 7fba44b2e640 1 -- 172.21.15.81:0/2202717487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba400aea70 con 0x7fba40070230 2024-06-29T14:41:27.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.597+0000 7fba377fe640 1 -- 172.21.15.81:0/2202717487 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7fba3801b460 con 0x7fba40070230 2024-06-29T14:41:27.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.598+0000 7fba377fe640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fba200763f0 0x7fba20078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:27.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.598+0000 7fba377fe640 1 -- 172.21.15.81:0/2202717487 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7fba380981e0 con 0x7fba40070230 2024-06-29T14:41:27.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.598+0000 7fba3e59c640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fba200763f0 0x7fba20078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:27.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.599+0000 7fba3e59c640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fba200763f0 0x7fba20078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fba30035000 tx=0x7fba300046c0 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:27.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.603+0000 7fba377fe640 1 -- 172.21.15.81:0/2202717487 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fba38061260 con 0x7fba40070230 2024-06-29T14:41:27.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.822+0000 7fba44b2e640 1 -- 172.21.15.81:0/2202717487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fba4006e9e0 con 0x7fba40070230 2024-06-29T14:41:27.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.822+0000 7fba377fe640 1 -- 172.21.15.81:0/2202717487 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v68) ==== 53+0+73 (secure 0 0 0) 0x7fba38065560 con 0x7fba40070230 2024-06-29T14:41:27.822 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:41:27.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 -- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fba200763f0 msgr2=0x7fba20078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fba200763f0 0x7fba20078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fba30035000 tx=0x7fba300046c0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 -- 172.21.15.81:0/2202717487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba40070230 msgr2=0x7fba40151010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:27.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba40070230 0x7fba40151010 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fba40072520 tx=0x7fba38001b70 comp rx=0 tx=0).stop 2024-06-29T14:41:27.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 -- 172.21.15.81:0/2202717487 shutdown_connections 2024-06-29T14:41:27.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fba200763f0 0x7fba20078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba40070230 0x7fba40151010 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fba4006fe20 0x7fba40150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 --2- 172.21.15.81:0/2202717487 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fba2c003a00 0x7fba401f43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:27.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 -- 172.21.15.81:0/2202717487 >> 172.21.15.81:0/2202717487 conn(0x7fba40072950 msgr2=0x7fba400b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:27.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 -- 172.21.15.81:0/2202717487 shutdown_connections 2024-06-29T14:41:27.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:27.824+0000 7fba44b2e640 1 -- 172.21.15.81:0/2202717487 wait complete. 2024-06-29T14:41:27.834 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:41:27.835 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:41:27.835 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T14:41:27.835 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:41:27.835 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:27.835 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:41:27.835 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:41:27.835 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:41:27.835 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:41:27.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:27.835 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T14:41:27.851 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:41:27.851 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:41:27.851 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:41:27.851 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.851 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:41:27.852 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 41632sec preferred_lft 41632sec 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:41:27.853 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:41:27.854 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T14:41:27.854 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:41:27.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:27.854 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:41:27.914 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T14:41:27.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:27.916 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T14:41:27.972 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T14:41:27.972 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:41:27.973 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:41:27.973 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:41:27.973 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:41:27.973 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:41:27.973 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:41:27.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:41:27.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:41:27.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:41:27.973 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:41:28.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:41:28.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:41:28.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:28.099 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:41:28.115 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:41:28.115 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:41:28.115 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:41:28.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:28.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T14:41:28.180 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:41:28.180 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:28.236 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:41:28.236 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:41:28.236 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:28.291 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T14:41:28.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:28.292 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:41:28.385 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:41:28.385 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:41:28.393 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:41:28.393 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:41:28.393 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:41:28.393 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:41:28.393 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:41:28.394 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:41:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:41:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:41:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:41:28.394 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:41:28.395 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:41:28.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:28.396 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:28.421 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:41:28.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:41:28.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:41:28.521 INFO:teuthology.orchestra.run.smithi081.stdout:client7097 2024-06-29T14:41:28.522 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:28.522 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:28.537 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:41:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.694+0000 7ff0e82ef640 1 -- 172.21.15.81:0/728116048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0e00b5250 msgr2=0x7ff0e00d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.694+0000 7ff0e82ef640 1 --2- 172.21.15.81:0/728116048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0e00b5250 0x7ff0e00d1fc0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8005fd0 tx=0x7ff0d802a940 comp rx=0 tx=0).stop 2024-06-29T14:41:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.694+0000 7ff0e82ef640 1 -- 172.21.15.81:0/728116048 shutdown_connections 2024-06-29T14:41:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.694+0000 7ff0e82ef640 1 --2- 172.21.15.81:0/728116048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0e00b5250 0x7ff0e00d1fc0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.694+0000 7ff0e82ef640 1 --2- 172.21.15.81:0/728116048 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0e00d0b90 0x7ff0e00d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.695+0000 7ff0e82ef640 1 -- 172.21.15.81:0/728116048 >> 172.21.15.81:0/728116048 conn(0x7ff0e009cf30 msgr2=0x7ff0e00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:28.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.695+0000 7ff0e82ef640 1 -- 172.21.15.81:0/728116048 shutdown_connections 2024-06-29T14:41:28.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.695+0000 7ff0e82ef640 1 -- 172.21.15.81:0/728116048 wait complete. 2024-06-29T14:41:28.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.696+0000 7ff0e82ef640 1 Processor -- start 2024-06-29T14:41:28.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.696+0000 7ff0e82ef640 1 -- start start 2024-06-29T14:41:28.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e82ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0e00b5250 0x7ff0e00ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e82ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0e00b9040 0x7ff0e014a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e82ef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0e015c0b0 0x7ff0e00c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e82ef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e00d2680 con 0x7ff0e00b9040 2024-06-29T14:41:28.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e82ef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e00d2500 con 0x7ff0e015c0b0 2024-06-29T14:41:28.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e82ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff0e00d2800 con 0x7ff0e00b5250 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e588a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0e00b9040 0x7ff0e014a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e588a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0e00b9040 0x7ff0e014a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48242/0 (socket says 172.21.15.81:48242) 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e588a640 1 -- 172.21.15.81:0/3244380387 learned_addr learned my addr 172.21.15.81:0/3244380387 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e688c640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0e015c0b0 0x7ff0e00c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e608b640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0e00b5250 0x7ff0e00ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e588a640 1 -- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0e00b5250 msgr2=0x7ff0e00ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e588a640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0e00b5250 0x7ff0e00ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e588a640 1 -- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0e015c0b0 msgr2=0x7ff0e00c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e588a640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0e015c0b0 0x7ff0e00c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e588a640 1 -- 172.21.15.81:0/3244380387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e014b070 con 0x7ff0e00b9040 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0e588a640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0e00b9040 0x7ff0e014a730 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000d680 tx=0x7ff0d000db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.698+0000 7ff0d6ffd640 1 -- 172.21.15.81:0/3244380387 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0017070 con 0x7ff0e00b9040 2024-06-29T14:41:28.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.699+0000 7ff0d6ffd640 1 -- 172.21.15.81:0/3244380387 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0d0009b10 con 0x7ff0e00b9040 2024-06-29T14:41:28.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.699+0000 7ff0e82ef640 1 -- 172.21.15.81:0/3244380387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e014b350 con 0x7ff0e00b9040 2024-06-29T14:41:28.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.699+0000 7ff0d6ffd640 1 -- 172.21.15.81:0/3244380387 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0010460 con 0x7ff0e00b9040 2024-06-29T14:41:28.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.699+0000 7ff0e82ef640 1 -- 172.21.15.81:0/3244380387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e014b910 con 0x7ff0e00b9040 2024-06-29T14:41:28.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.700+0000 7ff0e82ef640 1 -- 172.21.15.81:0/3244380387 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0e015fbc0 con 0x7ff0e00b9040 2024-06-29T14:41:28.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.701+0000 7ff0d6ffd640 1 -- 172.21.15.81:0/3244380387 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7ff0d001a070 con 0x7ff0e00b9040 2024-06-29T14:41:28.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.701+0000 7ff0d6ffd640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff0c4076420 0x7ff0c4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:28.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.701+0000 7ff0d6ffd640 1 -- 172.21.15.81:0/3244380387 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7ff0d0097a50 con 0x7ff0e00b9040 2024-06-29T14:41:28.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.701+0000 7ff0e608b640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff0c4076420 0x7ff0c4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:28.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.702+0000 7ff0e608b640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff0c4076420 0x7ff0c4078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc001720 tx=0x7ff0dc007e90 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:28.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.706+0000 7ff0d6ffd640 1 -- 172.21.15.81:0/3244380387 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff0d0060a50 con 0x7ff0e00b9040 2024-06-29T14:41:28.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.910+0000 7ff0e82ef640 1 -- 172.21.15.81:0/3244380387 --> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7ff0e00ca930 con 0x7ff0c4076420 2024-06-29T14:41:28.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.912+0000 7ff0d6ffd640 1 -- 172.21.15.81:0/3244380387 <== mgr.8328 v2:172.21.15.167:6832/971640072 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7ff0e00ca930 con 0x7ff0c4076420 2024-06-29T14:41:28.910 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:41:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T14:41:28.910 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:28.910 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:41:28.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.913+0000 7ff0e82ef640 1 -- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff0c4076420 msgr2=0x7ff0c4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff0c4076420 0x7ff0c4078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc001720 tx=0x7ff0dc007e90 comp rx=0 tx=0).stop 2024-06-29T14:41:28.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 -- 172.21.15.81:0/3244380387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0e00b9040 msgr2=0x7ff0e014a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:28.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0e00b9040 0x7ff0e014a730 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000d680 tx=0x7ff0d000db40 comp rx=0 tx=0).stop 2024-06-29T14:41:28.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 -- 172.21.15.81:0/3244380387 shutdown_connections 2024-06-29T14:41:28.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7ff0c4076420 0x7ff0c4078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0e015c0b0 0x7ff0e00c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0e00b9040 0x7ff0e014a730 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 --2- 172.21.15.81:0/3244380387 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0e00b5250 0x7ff0e00ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:28.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 -- 172.21.15.81:0/3244380387 >> 172.21.15.81:0/3244380387 conn(0x7ff0e009cf30 msgr2=0x7ff0e006d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:28.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 -- 172.21.15.81:0/3244380387 shutdown_connections 2024-06-29T14:41:28.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:28.914+0000 7ff0e82ef640 1 -- 172.21.15.81:0/3244380387 wait complete. 2024-06-29T14:41:28.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:41:29.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.039+0000 7f33e46e4640 1 -- 172.21.15.81:0/697547558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33dc0d2a90 msgr2=0x7f33dc0d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.039+0000 7f33e46e4640 1 --2- 172.21.15.81:0/697547558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33dc0d2a90 0x7f33dc0d09c0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f33d4005fd0 tx=0x7f33d402a960 comp rx=0 tx=0).stop 2024-06-29T14:41:29.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.039+0000 7f33e46e4640 1 -- 172.21.15.81:0/697547558 shutdown_connections 2024-06-29T14:41:29.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.039+0000 7f33e46e4640 1 --2- 172.21.15.81:0/697547558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33dc0d2a90 0x7f33dc0d09c0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.039+0000 7f33e46e4640 1 --2- 172.21.15.81:0/697547558 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33dc0b9040 0x7f33dc0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.039+0000 7f33e46e4640 1 -- 172.21.15.81:0/697547558 >> 172.21.15.81:0/697547558 conn(0x7f33dc09cf30 msgr2=0x7f33dc0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:29.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.039+0000 7f33e46e4640 1 -- 172.21.15.81:0/697547558 shutdown_connections 2024-06-29T14:41:29.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.039+0000 7f33e46e4640 1 -- 172.21.15.81:0/697547558 wait complete. 2024-06-29T14:41:29.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.041+0000 7f33e46e4640 1 Processor -- start 2024-06-29T14:41:29.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.041+0000 7f33e46e4640 1 -- start start 2024-06-29T14:41:29.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.042+0000 7f33e46e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33dc0b5250 0x7f33dc15c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.042+0000 7f33e46e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33dc0b9040 0x7f33dc0b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.042+0000 7f33e46e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33dc0d2a90 0x7f33dc0b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.042+0000 7f33e46e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc0b7d30 con 0x7f33dc0b9040 2024-06-29T14:41:29.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.042+0000 7f33e46e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc0b7bb0 con 0x7f33dc0b5250 2024-06-29T14:41:29.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.042+0000 7f33e46e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f33dc0b7eb0 con 0x7f33dc0d2a90 2024-06-29T14:41:29.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.042+0000 7f33e1c7f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33dc0b9040 0x7f33dc0b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e1c7f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33dc0b9040 0x7f33dc0b25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48262/0 (socket says 172.21.15.81:48262) 2024-06-29T14:41:29.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e1c7f640 1 -- 172.21.15.81:0/3860612931 learned_addr learned my addr 172.21.15.81:0/3860612931 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e2c81640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33dc0d2a90 0x7f33dc0b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e2480640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33dc0b5250 0x7f33dc15c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e1c7f640 1 -- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33dc0d2a90 msgr2=0x7f33dc0b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e1c7f640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33dc0d2a90 0x7f33dc0b2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e1c7f640 1 -- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33dc0b5250 msgr2=0x7f33dc15c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e1c7f640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33dc0b5250 0x7f33dc15c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e1c7f640 1 -- 172.21.15.81:0/3860612931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc0b1c90 con 0x7f33dc0b9040 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e1c7f640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33dc0b9040 0x7f33dc0b25c0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00c340 tx=0x7f33cc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33d2ffd640 1 -- 172.21.15.81:0/3860612931 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc017070 con 0x7f33dc0b9040 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33d2ffd640 1 -- 172.21.15.81:0/3860612931 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33cc0035f0 con 0x7f33dc0b9040 2024-06-29T14:41:29.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e46e4640 1 -- 172.21.15.81:0/3860612931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33dc0cd8a0 con 0x7f33dc0b9040 2024-06-29T14:41:29.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33d2ffd640 1 -- 172.21.15.81:0/3860612931 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc003a40 con 0x7f33dc0b9040 2024-06-29T14:41:29.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.043+0000 7f33e46e4640 1 -- 172.21.15.81:0/3860612931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33dc0b1640 con 0x7f33dc0b9040 2024-06-29T14:41:29.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.044+0000 7f33e46e4640 1 -- 172.21.15.81:0/3860612931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33dc0b83b0 con 0x7f33dc0b9040 2024-06-29T14:41:29.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.045+0000 7f33d2ffd640 1 -- 172.21.15.81:0/3860612931 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 68) ==== 98303+0+0 (secure 0 0 0) 0x7f33cc01a070 con 0x7f33dc0b9040 2024-06-29T14:41:29.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.045+0000 7f33d2ffd640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f33c00763f0 0x7f33c0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.045+0000 7f33d2ffd640 1 -- 172.21.15.81:0/3860612931 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7f33cc096c70 con 0x7f33dc0b9040 2024-06-29T14:41:29.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.046+0000 7f33e2480640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f33c00763f0 0x7f33c0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.046+0000 7f33e2480640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f33c00763f0 0x7f33c0078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33d800a770 tx=0x7f33d8008040 comp rx=0 tx=0).ready entity=mgr.8328 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:29.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.050+0000 7f33d2ffd640 1 -- 172.21.15.81:0/3860612931 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f33cc060450 con 0x7f33dc0b9040 2024-06-29T14:41:29.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.278+0000 7f33e46e4640 1 -- 172.21.15.81:0/3860612931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f33dc0cacc0 con 0x7f33dc0b9040 2024-06-29T14:41:29.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.757+0000 7f33d2ffd640 1 -- 172.21.15.81:0/3860612931 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v69) ==== 92+0+0 (secure 0 0 0) 0x7f33cc05fd50 con 0x7f33dc0b9040 2024-06-29T14:41:29.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.760+0000 7f33e46e4640 1 -- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f33c00763f0 msgr2=0x7f33c0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.760+0000 7f33e46e4640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f33c00763f0 0x7f33c0078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33d800a770 tx=0x7f33d8008040 comp rx=0 tx=0).stop 2024-06-29T14:41:29.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.760+0000 7f33e46e4640 1 -- 172.21.15.81:0/3860612931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33dc0b9040 msgr2=0x7f33dc0b25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.760+0000 7f33e46e4640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33dc0b9040 0x7f33dc0b25c0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00c340 tx=0x7f33cc00c800 comp rx=0 tx=0).stop 2024-06-29T14:41:29.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.761+0000 7f33e46e4640 1 -- 172.21.15.81:0/3860612931 shutdown_connections 2024-06-29T14:41:29.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.761+0000 7f33e46e4640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f33c00763f0 0x7f33c0078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.761+0000 7f33e46e4640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f33dc0d2a90 0x7f33dc0b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.761+0000 7f33e46e4640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33dc0b9040 0x7f33dc0b25c0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.761+0000 7f33e46e4640 1 --2- 172.21.15.81:0/3860612931 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f33dc0b5250 0x7f33dc15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.761+0000 7f33e46e4640 1 -- 172.21.15.81:0/3860612931 >> 172.21.15.81:0/3860612931 conn(0x7f33dc09cf30 msgr2=0x7f33dc0cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:29.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.761+0000 7f33e46e4640 1 -- 172.21.15.81:0/3860612931 shutdown_connections 2024-06-29T14:41:29.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.761+0000 7f33e46e4640 1 -- 172.21.15.81:0/3860612931 wait complete. 2024-06-29T14:41:29.772 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:41:29.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.895+0000 7f72a03e7640 1 -- 172.21.15.81:0/2762604493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7284004d40 msgr2=0x7f7284005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.895+0000 7f72a03e7640 1 --2- 172.21.15.81:0/2762604493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7284004d40 0x7f7284005210 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f7294006f60 tx=0x7f7284005830 comp rx=0 tx=0).stop 2024-06-29T14:41:29.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.895+0000 7f729e984640 1 -- 172.21.15.81:0/2762604493 reap_dead start 2024-06-29T14:41:29.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.895+0000 7f72a03e7640 1 -- 172.21.15.81:0/2762604493 shutdown_connections 2024-06-29T14:41:29.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.895+0000 7f72a03e7640 1 -- 172.21.15.81:0/2762604493 >> 172.21.15.81:0/2762604493 conn(0x7f729806ccc0 msgr2=0x7f72980b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:29.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.895+0000 7f72a03e7640 1 -- 172.21.15.81:0/2762604493 shutdown_connections 2024-06-29T14:41:29.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.895+0000 7f72a03e7640 1 -- 172.21.15.81:0/2762604493 wait complete. 2024-06-29T14:41:29.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.898+0000 7f72a03e7640 1 Processor -- start 2024-06-29T14:41:29.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.898+0000 7f72a03e7640 1 -- start start 2024-06-29T14:41:29.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.900+0000 7f72a03e7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7284004d40 0x7f7298159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.900+0000 7f72a03e7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72981f3670 0x7f72980b0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.900+0000 7f72a03e7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72981f3a80 0x7f72980b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.900+0000 7f72a03e7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f729806ef50 con 0x7f7284004d40 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.900+0000 7f72a03e7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f729806edd0 con 0x7f72981f3670 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.900+0000 7f72a03e7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f729806f0d0 con 0x7f72981f3a80 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.900+0000 7f729e183640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7284004d40 0x7f7298159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f729e183640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7284004d40 0x7f7298159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48290/0 (socket says 172.21.15.81:48290) 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f729e183640 1 -- 172.21.15.81:0/1640452987 learned_addr learned my addr 172.21.15.81:0/1640452987 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f729e984640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72981f3a80 0x7f72980b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f729d982640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72981f3670 0x7f72980b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f729e183640 1 -- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72981f3a80 msgr2=0x7f72980b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f729e183640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72981f3a80 0x7f72980b0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f729e183640 1 -- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72981f3670 msgr2=0x7f72980b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f729e183640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72981f3670 0x7f72980b0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:29.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f729e183640 1 -- 172.21.15.81:0/1640452987 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7298157520 con 0x7f7284004d40 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f729e183640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7284004d40 0x7f7298159130 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f729000afc0 tx=0x7f729000d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f728effd640 1 -- 172.21.15.81:0/1640452987 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729001e070 con 0x7f7284004d40 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f728effd640 1 -- 172.21.15.81:0/1640452987 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7290003e40 con 0x7f7284004d40 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f728effd640 1 -- 172.21.15.81:0/1640452987 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72900042b0 con 0x7f7284004d40 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f72a03e7640 1 -- 172.21.15.81:0/1640452987 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7298157800 con 0x7f7284004d40 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.901+0000 7f72a03e7640 1 -- 172.21.15.81:0/1640452987 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7298157dc0 con 0x7f7284004d40 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.903+0000 7f72a03e7640 1 -- 172.21.15.81:0/1640452987 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72980cf130 con 0x7f7284004d40 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.903+0000 7f728effd640 1 -- 172.21.15.81:0/1640452987 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 69) ==== 98320+0+0 (secure 0 0 0) 0x7f7290021070 con 0x7f7284004d40 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.904+0000 7f728effd640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f727c076470 0x7f727c078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.904+0000 7f729d982640 1 -- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f727c076470 msgr2=0x7f727c078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/971640072 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.904+0000 7f728effd640 1 -- 172.21.15.81:0/1640452987 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7f7290099990 con 0x7f7284004d40 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.904+0000 7f729d982640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f727c076470 0x7f727c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:29.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:29.908+0000 7f728effd640 1 -- 172.21.15.81:0/1640452987 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7290062a10 con 0x7f7284004d40 2024-06-29T14:41:30.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.096+0000 7f72a03e7640 1 -- 172.21.15.81:0/1640452987 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f729815c9e0 con 0x7f7284004d40 2024-06-29T14:41:30.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.100+0000 7f728effd640 1 -- 172.21.15.81:0/1640452987 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) ==== 138+0+0 (secure 0 0 0) 0x7f7290066d10 con 0x7f7284004d40 2024-06-29T14:41:30.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 -- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f727c076470 msgr2=0x7f727c078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:30.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f727c076470 0x7f727c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 -- 172.21.15.81:0/1640452987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7284004d40 msgr2=0x7f7298159130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7284004d40 0x7f7298159130 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f729000afc0 tx=0x7f729000d870 comp rx=0 tx=0).stop 2024-06-29T14:41:30.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 -- 172.21.15.81:0/1640452987 shutdown_connections 2024-06-29T14:41:30.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f727c076470 0x7f727c078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72981f3a80 0x7f72980b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72981f3670 0x7f72980b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 --2- 172.21.15.81:0/1640452987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7284004d40 0x7f7298159130 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 -- 172.21.15.81:0/1640452987 >> 172.21.15.81:0/1640452987 conn(0x7f729806ccc0 msgr2=0x7f7298150b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:30.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 -- 172.21.15.81:0/1640452987 shutdown_connections 2024-06-29T14:41:30.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.102+0000 7f72a03e7640 1 -- 172.21.15.81:0/1640452987 wait complete. 2024-06-29T14:41:30.112 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:41:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.231+0000 7fd0d2881640 1 -- 172.21.15.81:0/2215339125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0cc0b3200 msgr2=0x7fd0cc0cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.231+0000 7fd0d2881640 1 --2- 172.21.15.81:0/2215339125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0cc0b3200 0x7fd0cc0cc7b0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc005fd0 tx=0x7fd0bc02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.231+0000 7fd0d2881640 1 -- 172.21.15.81:0/2215339125 shutdown_connections 2024-06-29T14:41:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.231+0000 7fd0d2881640 1 --2- 172.21.15.81:0/2215339125 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0cc0cccf0 0x7fd0cc0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.231+0000 7fd0d2881640 1 --2- 172.21.15.81:0/2215339125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0cc0b3200 0x7fd0cc0cc7b0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.231+0000 7fd0d2881640 1 -- 172.21.15.81:0/2215339125 >> 172.21.15.81:0/2215339125 conn(0x7fd0cc0b3680 msgr2=0x7fd0cc0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.231+0000 7fd0d2881640 1 -- 172.21.15.81:0/2215339125 shutdown_connections 2024-06-29T14:41:30.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.231+0000 7fd0d2881640 1 -- 172.21.15.81:0/2215339125 wait complete. 2024-06-29T14:41:30.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.233+0000 7fd0d2881640 1 Processor -- start 2024-06-29T14:41:30.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.233+0000 7fd0d2881640 1 -- start start 2024-06-29T14:41:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.234+0000 7fd0d2881640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0cc0b3200 0x7fd0cc09da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.234+0000 7fd0d2881640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0cc0cccf0 0x7fd0cc157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.234+0000 7fd0d2881640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0cc0d0770 0x7fd0cc1f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.234+0000 7fd0d2881640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc0cedb0 con 0x7fd0cc0b3200 2024-06-29T14:41:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.234+0000 7fd0d2881640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc0cec30 con 0x7fd0cc0cccf0 2024-06-29T14:41:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.234+0000 7fd0d2881640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd0cc0cef30 con 0x7fd0cc0d0770 2024-06-29T14:41:30.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.234+0000 7fd0cbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0cc0b3200 0x7fd0cc09da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.234+0000 7fd0cbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0cc0b3200 0x7fd0cc09da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37278/0 (socket says 172.21.15.81:37278) 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.234+0000 7fd0cbfff640 1 -- 172.21.15.81:0/2110564454 learned_addr learned my addr 172.21.15.81:0/2110564454 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0cb7fe640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0cc0cccf0 0x7fd0cc157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0d0e1e640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0cc0d0770 0x7fd0cc1f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0cbfff640 1 -- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0cc0d0770 msgr2=0x7fd0cc1f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0cbfff640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0cc0d0770 0x7fd0cc1f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0cbfff640 1 -- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0cc0cccf0 msgr2=0x7fd0cc157130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0cbfff640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0cc0cccf0 0x7fd0cc157130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0cbfff640 1 -- 172.21.15.81:0/2110564454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0cc157a40 con 0x7fd0cc0b3200 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0cb7fe640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0cc0cccf0 0x7fd0cc157130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0cbfff640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0cc0b3200 0x7fd0cc09da00 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc034040 tx=0x7fd0bc0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0c8ff9640 1 -- 172.21.15.81:0/2110564454 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc03e030 con 0x7fd0cc0b3200 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0c8ff9640 1 -- 172.21.15.81:0/2110564454 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0bc003ae0 con 0x7fd0cc0b3200 2024-06-29T14:41:30.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0d2881640 1 -- 172.21.15.81:0/2110564454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0cc157d20 con 0x7fd0cc0b3200 2024-06-29T14:41:30.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0c8ff9640 1 -- 172.21.15.81:0/2110564454 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc035040 con 0x7fd0cc0b3200 2024-06-29T14:41:30.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.235+0000 7fd0d2881640 1 -- 172.21.15.81:0/2110564454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0cc1582e0 con 0x7fd0cc0b3200 2024-06-29T14:41:30.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.236+0000 7fd0d2881640 1 -- 172.21.15.81:0/2110564454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0cc0b5c10 con 0x7fd0cc0b3200 2024-06-29T14:41:30.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.238+0000 7fd0c8ff9640 1 -- 172.21.15.81:0/2110564454 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 69) ==== 98320+0+0 (secure 0 0 0) 0x7fd0bc041030 con 0x7fd0cc0b3200 2024-06-29T14:41:30.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.238+0000 7fd0c8ff9640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd0ac076470 0x7fd0ac078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.238+0000 7fd0cb7fe640 1 -- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd0ac076470 msgr2=0x7fd0ac078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/971640072 2024-06-29T14:41:30.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.238+0000 7fd0c8ff9640 1 -- 172.21.15.81:0/2110564454 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7fd0bc0b8170 con 0x7fd0cc0b3200 2024-06-29T14:41:30.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.238+0000 7fd0cb7fe640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd0ac076470 0x7fd0ac078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:30.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.243+0000 7fd0c8ff9640 1 -- 172.21.15.81:0/2110564454 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd0bc0811f0 con 0x7fd0cc0b3200 2024-06-29T14:41:30.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.430+0000 7fd0d2881640 1 -- 172.21.15.81:0/2110564454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fd0cc15b0b0 con 0x7fd0cc0b3200 2024-06-29T14:41:30.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.434+0000 7fd0c8ff9640 1 -- 172.21.15.81:0/2110564454 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) ==== 133+0+0 (secure 0 0 0) 0x7fd0bc0854f0 con 0x7fd0cc0b3200 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 -- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd0ac076470 msgr2=0x7fd0ac078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd0ac076470 0x7fd0ac078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 -- 172.21.15.81:0/2110564454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0cc0b3200 msgr2=0x7fd0cc09da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0cc0b3200 0x7fd0cc09da00 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc034040 tx=0x7fd0bc0035f0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 -- 172.21.15.81:0/2110564454 shutdown_connections 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7fd0ac076470 0x7fd0ac078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0cc0d0770 0x7fd0cc1f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0cc0cccf0 0x7fd0cc157130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 --2- 172.21.15.81:0/2110564454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0cc0b3200 0x7fd0cc09da00 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 -- 172.21.15.81:0/2110564454 >> 172.21.15.81:0/2110564454 conn(0x7fd0cc0b3680 msgr2=0x7fd0cc070300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:30.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 -- 172.21.15.81:0/2110564454 shutdown_connections 2024-06-29T14:41:30.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.436+0000 7fd0d2881640 1 -- 172.21.15.81:0/2110564454 wait complete. 2024-06-29T14:41:30.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:30.446 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2024-06-29T14:41:30.466 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2024-06-29T14:41:30.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.626+0000 7f4dd1aac640 1 -- 172.21.15.81:0/1858064022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4dcc0b5250 msgr2=0x7f4dcc0d2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.626+0000 7f4dd1aac640 1 --2- 172.21.15.81:0/1858064022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4dcc0b5250 0x7f4dcc0d2190 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4005fd0 tx=0x7f4dc402a890 comp rx=0 tx=0).stop 2024-06-29T14:41:30.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.626+0000 7f4dd1aac640 1 -- 172.21.15.81:0/1858064022 shutdown_connections 2024-06-29T14:41:30.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.626+0000 7f4dd1aac640 1 --2- 172.21.15.81:0/1858064022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4dcc0b5250 0x7f4dcc0d2190 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.626+0000 7f4dd1aac640 1 -- 172.21.15.81:0/1858064022 >> 172.21.15.81:0/1858064022 conn(0x7f4dcc09cf30 msgr2=0x7f4dcc0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:30.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.627+0000 7f4dd1aac640 1 -- 172.21.15.81:0/1858064022 shutdown_connections 2024-06-29T14:41:30.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.627+0000 7f4dd1aac640 1 -- 172.21.15.81:0/1858064022 wait complete. 2024-06-29T14:41:30.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.628+0000 7f4dd1aac640 1 Processor -- start 2024-06-29T14:41:30.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.629+0000 7f4dd1aac640 1 -- start start 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dd1aac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4dcc0b5250 0x7f4dcc1f7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dd1aac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dcc0b9040 0x7f4dcc149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dd1aac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4dcc1f40f0 0x7f4dcc0b7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dd1aac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc0d26d0 con 0x7f4dcc0b9040 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dd1aac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc0b8560 con 0x7f4dcc0b5250 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dd1aac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4dcc0d2850 con 0x7f4dcc1f40f0 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcb7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4dcc0b5250 0x7f4dcc1f7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dcc0b9040 0x7f4dcc149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcb7fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4dcc0b5250 0x7f4dcc1f7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:50408/0 (socket says 172.21.15.81:50408) 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcb7fe640 1 -- 172.21.15.81:0/3105811302 learned_addr learned my addr 172.21.15.81:0/3105811302 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcbfff640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4dcc1f40f0 0x7f4dcc0b7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:30.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcb7fe640 1 -- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4dcc1f40f0 msgr2=0x7f4dcc0b7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcb7fe640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4dcc1f40f0 0x7f4dcc0b7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcb7fe640 1 -- 172.21.15.81:0/3105811302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dcc0b9040 msgr2=0x7f4dcc149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcb7fe640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dcc0b9040 0x7f4dcc149820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcb7fe640 1 -- 172.21.15.81:0/3105811302 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dcc149d60 con 0x7f4dcc0b5250 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.630+0000 7f4dcaffd640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dcc0b9040 0x7f4dcc149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.631+0000 7f4dcb7fe640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4dcc0b5250 0x7f4dcc1f7f50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c3d0 tx=0x7f4db400c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.631+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db4017070 con 0x7f4dcc0b5250 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.631+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4db40035f0 con 0x7f4dcc0b5250 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.631+0000 7f4dd1aac640 1 -- 172.21.15.81:0/3105811302 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dcc1f8930 con 0x7f4dcc0b5250 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.631+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db4003a40 con 0x7f4dcc0b5250 2024-06-29T14:41:30.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.631+0000 7f4dd1aac640 1 -- 172.21.15.81:0/3105811302 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dcc149020 con 0x7f4dcc0b5250 2024-06-29T14:41:30.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.632+0000 7f4dd1aac640 1 -- 172.21.15.81:0/3105811302 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dcc0d2de0 con 0x7f4dcc0b5250 2024-06-29T14:41:30.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.633+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 69) ==== 98320+0+0 (secure 0 0 0) 0x7f4db401a070 con 0x7f4dcc0b5250 2024-06-29T14:41:30.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.633+0000 7f4dc3fff640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 0x7f4dac078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:30.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.633+0000 7f4dcaffd640 1 -- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 msgr2=0x7f4dac078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/971640072 2024-06-29T14:41:30.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.633+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5809+0+0 (secure 0 0 0) 0x7f4db4096db0 con 0x7f4dcc0b5250 2024-06-29T14:41:30.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.633+0000 7f4dcaffd640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 0x7f4dac078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:30.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.638+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4db409b070 con 0x7f4dcc0b5250 2024-06-29T14:41:30.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.835+0000 7f4dcaffd640 1 -- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 msgr2=0x7f4dac078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/971640072 2024-06-29T14:41:30.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.835+0000 7f4dcaffd640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 0x7f4dac078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:30.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:30.849+0000 7f4dd1aac640 1 -- 172.21.15.81:0/3105811302 --> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4dcc15d490 con 0x7f4dac076440 2024-06-29T14:41:31.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:31.236+0000 7f4dcaffd640 1 -- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 msgr2=0x7f4dac078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/971640072 2024-06-29T14:41:31.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:31.236+0000 7f4dcaffd640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 0x7f4dac078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:41:32.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:32.037+0000 7f4dcaffd640 1 -- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 msgr2=0x7f4dac078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/971640072 2024-06-29T14:41:32.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:32.037+0000 7f4dcaffd640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 0x7f4dac078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:41:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:33.393+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mgrmap(e 70) ==== 97929+0+0 (secure 0 0 0) 0x7f4db405c130 con 0x7f4dcc0b5250 2024-06-29T14:41:33.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:33.393+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 msgr2=0x7f4dac078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:33.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:33.393+0000 7f4dc3fff640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/971640072,v1:172.21.15.167:6833/971640072] conn(0x7f4dac076440 0x7f4dac078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:33.543 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:33.543+0000 7f85e4e17640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.543 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:33.543+0000 7f85e4e17640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.543 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:33.543+0000 7f85e4e17640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.543 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:33.543+0000 7f85e4e17640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.543 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:33.543+0000 7f85e4e17640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.544 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:33.543+0000 7f85e1e11640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.544 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:33.543+0000 7f85e1e11640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.544 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:33.543+0000 7f85e1e11640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.544 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:33.543+0000 7f85e1e11640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:33.544 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:33.543+0000 7f85e1e11640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.400+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 <== mon.1 v2:172.21.15.167:3300/0 8 ==== mgrmap(e 71) ==== 98056+0+0 (secure 0 0 0) 0x7f4db406c1a0 con 0x7f4dcc0b5250 2024-06-29T14:41:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.400+0000 7f4dc3fff640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4dac03b070 0x7f4dac0804a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.400+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4db401feb0 con 0x7f4dac03b070 2024-06-29T14:41:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.401+0000 7f4dcaffd640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4dac03b070 0x7f4dac0804a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.403+0000 7f4dcaffd640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4dac03b070 0x7f4dac0804a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc007b60 tx=0x7f4dbc007ab0 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:34.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.471+0000 7f4dc3fff640 1 -- 172.21.15.81:0/3105811302 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+45 (secure 0 0 0) 0x7f4db401feb0 con 0x7f4dac03b070 2024-06-29T14:41:34.470 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1/minutes 2024-06-29T14:41:34.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.474+0000 7f4dd1aac640 1 -- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4dac03b070 msgr2=0x7f4dac0804a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.474+0000 7f4dd1aac640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4dac03b070 0x7f4dac0804a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc007b60 tx=0x7f4dbc007ab0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.474+0000 7f4dd1aac640 1 -- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4dcc0b5250 msgr2=0x7f4dcc1f7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.474+0000 7f4dd1aac640 1 --2- 172.21.15.81:0/3105811302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4dcc0b5250 0x7f4dcc1f7f50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c3d0 tx=0x7f4db400c890 comp rx=0 tx=0).stop 2024-06-29T14:41:34.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.474+0000 7f4dcbfff640 1 -- 172.21.15.81:0/3105811302 reap_dead start 2024-06-29T14:41:34.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.474+0000 7f4dd1aac640 1 -- 172.21.15.81:0/3105811302 shutdown_connections 2024-06-29T14:41:34.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.474+0000 7f4dd1aac640 1 -- 172.21.15.81:0/3105811302 >> 172.21.15.81:0/3105811302 conn(0x7f4dcc09cf30 msgr2=0x7f4dcc15e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:34.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.474+0000 7f4dd1aac640 1 -- 172.21.15.81:0/3105811302 shutdown_connections 2024-06-29T14:41:34.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.474+0000 7f4dd1aac640 1 -- 172.21.15.81:0/3105811302 wait complete. 2024-06-29T14:41:34.484 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2024-06-29T14:41:34.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:34.485 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2024-06-29T14:41:34.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2024-06-29T14:41:34.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.661+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/2367865839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd78c004d90 msgr2=0x7fd78c005240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.661+0000 7fd7a8d9a640 1 --2- 172.21.15.81:0/2367865839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd78c004d90 0x7fd78c005240 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4071690 tx=0x7fd78c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:34.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.662+0000 7fd7a2d9d640 1 -- 172.21.15.81:0/2367865839 reap_dead start 2024-06-29T14:41:34.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.662+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/2367865839 shutdown_connections 2024-06-29T14:41:34.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.662+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/2367865839 >> 172.21.15.81:0/2367865839 conn(0x7fd7a40d0930 msgr2=0x7fd7a40d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:34.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.662+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/2367865839 shutdown_connections 2024-06-29T14:41:34.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.662+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/2367865839 wait complete. 2024-06-29T14:41:34.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.664+0000 7fd7a8d9a640 1 Processor -- start 2024-06-29T14:41:34.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.664+0000 7fd7a8d9a640 1 -- start start 2024-06-29T14:41:34.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a8d9a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd78c004d90 0x7fd7a40a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a8d9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7a40b1f20 0x7fd7a4157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a8d9a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7a4157c90 0x7fd7a4151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a8d9a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a40d0570 con 0x7fd7a40b1f20 2024-06-29T14:41:34.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a8d9a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a40d03f0 con 0x7fd78c004d90 2024-06-29T14:41:34.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a8d9a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a406c2f0 con 0x7fd7a4157c90 2024-06-29T14:41:34.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a1d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7a40b1f20 0x7fd7a4157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a1d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7a40b1f20 0x7fd7a4157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37300/0 (socket says 172.21.15.81:37300) 2024-06-29T14:41:34.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a1d9b640 1 -- 172.21.15.81:0/1890964860 learned_addr learned my addr 172.21.15.81:0/1890964860 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:34.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a259c640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd78c004d90 0x7fd7a40a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a2d9d640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7a4157c90 0x7fd7a4151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a1d9b640 1 -- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7a4157c90 msgr2=0x7fd7a4151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a1d9b640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7a4157c90 0x7fd7a4151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a1d9b640 1 -- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd78c004d90 msgr2=0x7fd7a40a9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a1d9b640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd78c004d90 0x7fd7a40a9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a1d9b640 1 -- 172.21.15.81:0/1890964860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a40af6f0 con 0x7fd7a40b1f20 2024-06-29T14:41:34.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.665+0000 7fd7a1d9b640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7a40b1f20 0x7fd7a4157750 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fd798011ae0 tx=0x7fd798009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:34.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.666+0000 7fd792ffd640 1 -- 172.21.15.81:0/1890964860 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79801e070 con 0x7fd7a40b1f20 2024-06-29T14:41:34.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.666+0000 7fd792ffd640 1 -- 172.21.15.81:0/1890964860 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd79800fe70 con 0x7fd7a40b1f20 2024-06-29T14:41:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.666+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/1890964860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a40b2330 con 0x7fd7a40b1f20 2024-06-29T14:41:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.666+0000 7fd792ffd640 1 -- 172.21.15.81:0/1890964860 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd798004290 con 0x7fd7a40b1f20 2024-06-29T14:41:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.666+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/1890964860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a40b1440 con 0x7fd7a40b1f20 2024-06-29T14:41:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.667+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/1890964860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a406e2a0 con 0x7fd7a40b1f20 2024-06-29T14:41:34.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.667+0000 7fd792ffd640 1 -- 172.21.15.81:0/1890964860 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 71) ==== 98056+0+0 (secure 0 0 0) 0x7fd798021070 con 0x7fd7a40b1f20 2024-06-29T14:41:34.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.668+0000 7fd792ffd640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fd784076170 0x7fd784078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:34.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.668+0000 7fd792ffd640 1 -- 172.21.15.81:0/1890964860 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7fd798097fd0 con 0x7fd7a40b1f20 2024-06-29T14:41:34.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.668+0000 7fd7a259c640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fd784076170 0x7fd784078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:34.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.669+0000 7fd7a259c640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fd784076170 0x7fd784078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd794006fd0 tx=0x7fd794006aa0 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:34.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.672+0000 7fd792ffd640 1 -- 172.21.15.81:0/1890964860 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd7a406e2a0 con 0x7fd7a40b1f20 2024-06-29T14:41:34.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.873+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/1890964860 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) -- 0x7fd7a40af1a0 con 0x7fd784076170 2024-06-29T14:41:34.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.942+0000 7fd792ffd640 1 -- 172.21.15.81:0/1890964860 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+44 (secure 0 0 0) 0x7fd7a40af1a0 con 0x7fd784076170 2024-06-29T14:41:34.939 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1/hourly 2024-06-29T14:41:34.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fd784076170 msgr2=0x7fd784078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fd784076170 0x7fd784078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd794006fd0 tx=0x7fd794006aa0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/1890964860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7a40b1f20 msgr2=0x7fd7a4157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7a40b1f20 0x7fd7a4157750 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fd798011ae0 tx=0x7fd798009e90 comp rx=0 tx=0).stop 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/1890964860 shutdown_connections 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fd784076170 0x7fd784078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd7a4157c90 0x7fd7a4151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7a40b1f20 0x7fd7a4157750 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 --2- 172.21.15.81:0/1890964860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd78c004d90 0x7fd7a40a9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/1890964860 >> 172.21.15.81:0/1890964860 conn(0x7fd7a40d0930 msgr2=0x7fd7a406f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/1890964860 shutdown_connections 2024-06-29T14:41:34.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:34.943+0000 7fd7a8d9a640 1 -- 172.21.15.81:0/1890964860 wait complete. 2024-06-29T14:41:34.953 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2024-06-29T14:41:34.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:34.953 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2024-06-29T14:41:34.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2024-06-29T14:41:35.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.130+0000 7f4fd4065640 1 -- 172.21.15.81:0/2590517732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fb8005150 msgr2=0x7f4fb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.130+0000 7f4fd4065640 1 --2- 172.21.15.81:0/2590517732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fb8005150 0x7f4fb800b9c0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4005fd0 tx=0x7f4fb8005830 comp rx=0 tx=0).stop 2024-06-29T14:41:35.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.130+0000 7f4fd2602640 1 -- 172.21.15.81:0/2590517732 reap_dead start 2024-06-29T14:41:35.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.130+0000 7f4fd4065640 1 -- 172.21.15.81:0/2590517732 shutdown_connections 2024-06-29T14:41:35.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.130+0000 7f4fd4065640 1 -- 172.21.15.81:0/2590517732 >> 172.21.15.81:0/2590517732 conn(0x7f4fcc072950 msgr2=0x7f4fcc0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:35.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.130+0000 7f4fd4065640 1 -- 172.21.15.81:0/2590517732 shutdown_connections 2024-06-29T14:41:35.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.130+0000 7f4fd4065640 1 -- 172.21.15.81:0/2590517732 wait complete. 2024-06-29T14:41:35.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.132+0000 7f4fd4065640 1 Processor -- start 2024-06-29T14:41:35.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.132+0000 7f4fd4065640 1 -- start start 2024-06-29T14:41:35.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd4065640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4fb8005150 0x7f4fcc0c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd4065640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4fcc152550 0x7f4fcc14f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd4065640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc0b0a80 0x7f4fcc0b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd4065640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc06d0c0 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd4065640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc06cf40 con 0x7f4fb8005150 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd4065640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4fcc073f20 con 0x7f4fcc152550 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd2602640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc0b0a80 0x7f4fcc0b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd2602640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc0b0a80 0x7f4fcc0b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37322/0 (socket says 172.21.15.81:37322) 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd2602640 1 -- 172.21.15.81:0/1188352175 learned_addr learned my addr 172.21.15.81:0/1188352175 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd1600640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4fcc152550 0x7f4fcc14f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd1e01640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4fb8005150 0x7f4fcc0c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd2602640 1 -- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4fcc152550 msgr2=0x7f4fcc14f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd2602640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4fcc152550 0x7f4fcc14f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd2602640 1 -- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4fb8005150 msgr2=0x7f4fcc0c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd2602640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4fb8005150 0x7f4fcc0c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.133+0000 7f4fd2602640 1 -- 172.21.15.81:0/1188352175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fcc0b13d0 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.134+0000 7f4fd2602640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc0b0a80 0x7f4fcc0b0e90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400e380 tx=0x7f4fc4033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:35.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.134+0000 7f4fc27fc640 1 -- 172.21.15.81:0/1188352175 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4009c10 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.134+0000 7f4fc27fc640 1 -- 172.21.15.81:0/1188352175 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fc4044070 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.134+0000 7f4fd4065640 1 -- 172.21.15.81:0/1188352175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fcc0c9c90 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.134+0000 7f4fc27fc640 1 -- 172.21.15.81:0/1188352175 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc403d6b0 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.134+0000 7f4fd4065640 1 -- 172.21.15.81:0/1188352175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fcc0ca250 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.135+0000 7f4fd4065640 1 -- 172.21.15.81:0/1188352175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fcc074440 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.135+0000 7f4fc27fc640 1 -- 172.21.15.81:0/1188352175 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 71) ==== 98056+0+0 (secure 0 0 0) 0x7f4fc403d850 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.136+0000 7f4fc27fc640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4fb0076170 0x7f4fb0078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.136+0000 7f4fc27fc640 1 -- 172.21.15.81:0/1188352175 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f4fc40b8b90 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.137+0000 7f4fd1e01640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4fb0076170 0x7f4fb0078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.137+0000 7f4fd1e01640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4fb0076170 0x7f4fb0078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8006c70 tx=0x7f4fc80067a0 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:35.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.141+0000 7f4fc27fc640 1 -- 172.21.15.81:0/1188352175 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4fcc0b13d0 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.342+0000 7f4fd4065640 1 -- 172.21.15.81:0/1188352175 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) -- 0x7f4fcc06cc20 con 0x7f4fb0076170 2024-06-29T14:41:35.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.404+0000 7f4fc27fc640 1 -- 172.21.15.81:0/1188352175 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7f4fc407e220 con 0x7f4fcc0b0a80 2024-06-29T14:41:35.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.412+0000 7f4fc27fc640 1 -- 172.21.15.81:0/1188352175 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f4fcc06cc20 con 0x7f4fb0076170 2024-06-29T14:41:35.410 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1/daily 2024-06-29T14:41:35.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.414+0000 7f4fd4065640 1 -- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4fb0076170 msgr2=0x7f4fb0078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.414+0000 7f4fd4065640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4fb0076170 0x7f4fb0078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8006c70 tx=0x7f4fc80067a0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.414+0000 7f4fd4065640 1 -- 172.21.15.81:0/1188352175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc0b0a80 msgr2=0x7f4fcc0b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.414+0000 7f4fd4065640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc0b0a80 0x7f4fcc0b0e90 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400e380 tx=0x7f4fc4033ba0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.414+0000 7f4fd4065640 1 -- 172.21.15.81:0/1188352175 shutdown_connections 2024-06-29T14:41:35.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.414+0000 7f4fd4065640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4fb0076170 0x7f4fb0078680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.414+0000 7f4fd4065640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc0b0a80 0x7f4fcc0b0e90 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.414+0000 7f4fd4065640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4fcc152550 0x7f4fcc14f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.414+0000 7f4fd4065640 1 --2- 172.21.15.81:0/1188352175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4fb8005150 0x7f4fcc0c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.414+0000 7f4fd4065640 1 -- 172.21.15.81:0/1188352175 >> 172.21.15.81:0/1188352175 conn(0x7f4fcc072950 msgr2=0x7f4fcc068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:35.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.415+0000 7f4fd4065640 1 -- 172.21.15.81:0/1188352175 shutdown_connections 2024-06-29T14:41:35.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.415+0000 7f4fd4065640 1 -- 172.21.15.81:0/1188352175 wait complete. 2024-06-29T14:41:35.425 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2024-06-29T14:41:35.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:35.425 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2024-06-29T14:41:35.441 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2024-06-29T14:41:35.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.599+0000 7f47fe4cc640 1 -- 172.21.15.81:0/1672673982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47f806f850 msgr2=0x7f47f80ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.599+0000 7f47fe4cc640 1 --2- 172.21.15.81:0/1672673982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47f806f850 0x7f47f80ae300 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f47f8073780 tx=0x7f47ec02a560 comp rx=0 tx=0).stop 2024-06-29T14:41:35.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.599+0000 7f47fe4cc640 1 -- 172.21.15.81:0/1672673982 shutdown_connections 2024-06-29T14:41:35.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.599+0000 7f47fe4cc640 1 --2- 172.21.15.81:0/1672673982 >> v2:172.21.15.167:3300/0 conn(0x7f47f80ae840 0x7f47f80aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.599+0000 7f47fe4cc640 1 --2- 172.21.15.81:0/1672673982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47f806f850 0x7f47f80ae300 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.599+0000 7f47fe4cc640 1 -- 172.21.15.81:0/1672673982 >> 172.21.15.81:0/1672673982 conn(0x7f47f8072950 msgr2=0x7f47f80b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:35.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.599+0000 7f47fe4cc640 1 -- 172.21.15.81:0/1672673982 shutdown_connections 2024-06-29T14:41:35.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.599+0000 7f47fe4cc640 1 -- 172.21.15.81:0/1672673982 wait complete. 2024-06-29T14:41:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.601+0000 7f47fe4cc640 1 Processor -- start 2024-06-29T14:41:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.601+0000 7f47fe4cc640 1 -- start start 2024-06-29T14:41:35.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47fe4cc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47f806f850 0x7f47f80d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47fe4cc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47f8074520 0x7f47f80d0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47fe4cc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47f80ae840 0x7f47f815c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47fe4cc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f47f815b190 con 0x7f47f806f850 2024-06-29T14:41:35.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47fe4cc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f47f815b010 con 0x7f47f8074520 2024-06-29T14:41:35.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47fe4cc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f47f80af190 con 0x7f47f80ae840 2024-06-29T14:41:35.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47f7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47f806f850 0x7f47f80d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47f7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47f806f850 0x7f47f80d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37350/0 (socket says 172.21.15.81:37350) 2024-06-29T14:41:35.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47f7fff640 1 -- 172.21.15.81:0/3928914201 learned_addr learned my addr 172.21.15.81:0/3928914201 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47f77fe640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47f8074520 0x7f47f80d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.602+0000 7f47fca69640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47f80ae840 0x7f47f815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47f7fff640 1 -- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47f80ae840 msgr2=0x7f47f815c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47f7fff640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47f80ae840 0x7f47f815c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47f7fff640 1 -- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47f8074520 msgr2=0x7f47f80d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47f7fff640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47f8074520 0x7f47f80d0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47f7fff640 1 -- 172.21.15.81:0/3928914201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f81526c0 con 0x7f47f806f850 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47f7fff640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47f806f850 0x7f47f80d0030 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f47e800b3f0 tx=0x7f47e800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47f4ff9640 1 -- 172.21.15.81:0/3928914201 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8015020 con 0x7f47f806f850 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47f4ff9640 1 -- 172.21.15.81:0/3928914201 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47e8011920 con 0x7f47f806f850 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47fe4cc640 1 -- 172.21.15.81:0/3928914201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47f81529a0 con 0x7f47f806f850 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47f4ff9640 1 -- 172.21.15.81:0/3928914201 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8011d70 con 0x7f47f806f850 2024-06-29T14:41:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.603+0000 7f47fe4cc640 1 -- 172.21.15.81:0/3928914201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47f80c9460 con 0x7f47f806f850 2024-06-29T14:41:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.604+0000 7f47fe4cc640 1 -- 172.21.15.81:0/3928914201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47f80ae300 con 0x7f47f806f850 2024-06-29T14:41:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.605+0000 7f47f4ff9640 1 -- 172.21.15.81:0/3928914201 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7f47e8018020 con 0x7f47f806f850 2024-06-29T14:41:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.605+0000 7f47f4ff9640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f47e4076440 0x7f47e4078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.605+0000 7f47f4ff9640 1 -- 172.21.15.81:0/3928914201 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f47e8097980 con 0x7f47f806f850 2024-06-29T14:41:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.606+0000 7f47f77fe640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f47e4076440 0x7f47e4078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.606+0000 7f47f77fe640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f47e4076440 0x7f47e4078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f47ec033000 tx=0x7f47ec004730 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.610+0000 7f47f4ff9640 1 -- 172.21.15.81:0/3928914201 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f47e801f020 con 0x7f47f806f850 2024-06-29T14:41:35.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.811+0000 7f47fe4cc640 1 -- 172.21.15.81:0/3928914201 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) -- 0x7f47f8159a00 con 0x7f47e4076440 2024-06-29T14:41:35.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.869+0000 7f47f4ff9640 1 -- 172.21.15.81:0/3928914201 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+44 (secure 0 0 0) 0x7f47f8159a00 con 0x7f47e4076440 2024-06-29T14:41:35.867 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1/weekly 2024-06-29T14:41:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 -- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f47e4076440 msgr2=0x7f47e4078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f47e4076440 0x7f47e4078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f47ec033000 tx=0x7f47ec004730 comp rx=0 tx=0).stop 2024-06-29T14:41:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 -- 172.21.15.81:0/3928914201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47f806f850 msgr2=0x7f47f80d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47f806f850 0x7f47f80d0030 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f47e800b3f0 tx=0x7f47e800b8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 -- 172.21.15.81:0/3928914201 shutdown_connections 2024-06-29T14:41:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f47e4076440 0x7f47e4078950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47f80ae840 0x7f47f815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47f8074520 0x7f47f80d0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 --2- 172.21.15.81:0/3928914201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47f806f850 0x7f47f80d0030 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:35.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 -- 172.21.15.81:0/3928914201 >> 172.21.15.81:0/3928914201 conn(0x7f47f8072950 msgr2=0x7f47f81585d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:35.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 -- 172.21.15.81:0/3928914201 shutdown_connections 2024-06-29T14:41:35.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:35.871+0000 7f47fe4cc640 1 -- 172.21.15.81:0/3928914201 wait complete. 2024-06-29T14:41:35.881 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2024-06-29T14:41:35.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:35.881 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2024-06-29T14:41:35.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2024-06-29T14:41:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.058+0000 7f285e895640 1 -- 172.21.15.81:0/4076612821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2844003a50 msgr2=0x7f2844003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.058+0000 7f285e895640 1 --2- 172.21.15.81:0/4076612821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2844003a50 0x7f2844003ec0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f28480016c0 tx=0x7f2844005830 comp rx=0 tx=0).stop 2024-06-29T14:41:36.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.058+0000 7f285e895640 1 -- 172.21.15.81:0/4076612821 shutdown_connections 2024-06-29T14:41:36.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.058+0000 7f285e895640 1 --2- 172.21.15.81:0/4076612821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2844003a50 0x7f2844003ec0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.058+0000 7f285e895640 1 -- 172.21.15.81:0/4076612821 >> 172.21.15.81:0/4076612821 conn(0x7f2858072950 msgr2=0x7f28580b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:36.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.058+0000 7f285e895640 1 -- 172.21.15.81:0/4076612821 shutdown_connections 2024-06-29T14:41:36.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.058+0000 7f285e895640 1 -- 172.21.15.81:0/4076612821 wait complete. 2024-06-29T14:41:36.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.060+0000 7f285e895640 1 Processor -- start 2024-06-29T14:41:36.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.060+0000 7f285e895640 1 -- start start 2024-06-29T14:41:36.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f285e895640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2844003a50 0x7f285815da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f285e895640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f285814e650 0x7f28581464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f285e895640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28581f1b90 0x7f285815c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f285e895640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f285815b190 con 0x7f285814e650 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f285e895640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f285815b010 con 0x7f2844003a50 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f285e895640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f28580ae300 con 0x7f28581f1b90 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f284ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f285814e650 0x7f28581464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f284ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f285814e650 0x7f28581464a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37366/0 (socket says 172.21.15.81:37366) 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f284ffff640 1 -- 172.21.15.81:0/1680964942 learned_addr learned my addr 172.21.15.81:0/1680964942 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f285ce32640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28581f1b90 0x7f285815c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f2857fff640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2844003a50 0x7f285815da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f284ffff640 1 -- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28581f1b90 msgr2=0x7f285815c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f284ffff640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28581f1b90 0x7f285815c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f284ffff640 1 -- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2844003a50 msgr2=0x7f285815da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f284ffff640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2844003a50 0x7f285815da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f284ffff640 1 -- 172.21.15.81:0/1680964942 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28581507b0 con 0x7f285814e650 2024-06-29T14:41:36.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.061+0000 7f284ffff640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f285814e650 0x7f28581464a0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f284000e5e0 tx=0x7f284000eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:36.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.062+0000 7f28557fa640 1 -- 172.21.15.81:0/1680964942 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2840015020 con 0x7f285814e650 2024-06-29T14:41:36.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.062+0000 7f28557fa640 1 -- 172.21.15.81:0/1680964942 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f284000ca00 con 0x7f285814e650 2024-06-29T14:41:36.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.062+0000 7f28557fa640 1 -- 172.21.15.81:0/1680964942 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284000ce50 con 0x7f285814e650 2024-06-29T14:41:36.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.062+0000 7f285e895640 1 -- 172.21.15.81:0/1680964942 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285815d220 con 0x7f285814e650 2024-06-29T14:41:36.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.062+0000 7f285e895640 1 -- 172.21.15.81:0/1680964942 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28581469e0 con 0x7f285814e650 2024-06-29T14:41:36.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.063+0000 7f285e895640 1 -- 172.21.15.81:0/1680964942 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f285806f140 con 0x7f285814e650 2024-06-29T14:41:36.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.063+0000 7f28557fa640 1 -- 172.21.15.81:0/1680964942 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7f2840018020 con 0x7f285814e650 2024-06-29T14:41:36.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.064+0000 7f28557fa640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f2838076440 0x7f2838078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.064+0000 7f28557fa640 1 -- 172.21.15.81:0/1680964942 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f2840097fa0 con 0x7f285814e650 2024-06-29T14:41:36.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.064+0000 7f2857fff640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f2838076440 0x7f2838078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.064+0000 7f2857fff640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f2838076440 0x7f2838078950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2848032000 tx=0x7f2848001190 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:36.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.068+0000 7f28557fa640 1 -- 172.21.15.81:0/1680964942 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f284001f020 con 0x7f285814e650 2024-06-29T14:41:36.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.270+0000 7f285e895640 1 -- 172.21.15.81:0/1680964942 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f2858150a90 con 0x7f2838076440 2024-06-29T14:41:36.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.331+0000 7f28557fa640 1 -- 172.21.15.81:0/1680964942 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+45 (secure 0 0 0) 0x7f2858150a90 con 0x7f2838076440 2024-06-29T14:41:36.330 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1/monthly 2024-06-29T14:41:36.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.333+0000 7f285e895640 1 -- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f2838076440 msgr2=0x7f2838078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.333+0000 7f285e895640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f2838076440 0x7f2838078950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2848032000 tx=0x7f2848001190 comp rx=0 tx=0).stop 2024-06-29T14:41:36.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.333+0000 7f285e895640 1 -- 172.21.15.81:0/1680964942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f285814e650 msgr2=0x7f28581464a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.333+0000 7f285e895640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f285814e650 0x7f28581464a0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f284000e5e0 tx=0x7f284000eaa0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.333+0000 7f285e895640 1 -- 172.21.15.81:0/1680964942 shutdown_connections 2024-06-29T14:41:36.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.333+0000 7f285e895640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f2838076440 0x7f2838078950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.333+0000 7f285e895640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28581f1b90 0x7f285815c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.333+0000 7f285e895640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f285814e650 0x7f28581464a0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.333+0000 7f285e895640 1 --2- 172.21.15.81:0/1680964942 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2844003a50 0x7f285815da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.333+0000 7f285e895640 1 -- 172.21.15.81:0/1680964942 >> 172.21.15.81:0/1680964942 conn(0x7f2858072950 msgr2=0x7f2858158570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:36.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.334+0000 7f285e895640 1 -- 172.21.15.81:0/1680964942 shutdown_connections 2024-06-29T14:41:36.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.334+0000 7f285e895640 1 -- 172.21.15.81:0/1680964942 wait complete. 2024-06-29T14:41:36.343 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2024-06-29T14:41:36.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:36.343 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2024-06-29T14:41:36.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1y 2024-06-29T14:41:36.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.518+0000 7fec91b44640 1 -- 172.21.15.81:0/3517065344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8c0aea70 msgr2=0x7fec8c0aee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.518+0000 7fec91b44640 1 --2- 172.21.15.81:0/3517065344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8c0aea70 0x7fec8c0aee80 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0d0c70 tx=0x7fec8002a4a0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.518+0000 7fec91b44640 1 -- 172.21.15.81:0/3517065344 shutdown_connections 2024-06-29T14:41:36.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.518+0000 7fec91b44640 1 --2- 172.21.15.81:0/3517065344 >> v2:172.21.15.167:3300/0 conn(0x7fec8c0af3c0 0x7fec8c06c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.518+0000 7fec91b44640 1 --2- 172.21.15.81:0/3517065344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8c0aea70 0x7fec8c0aee80 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.518+0000 7fec91b44640 1 -- 172.21.15.81:0/3517065344 >> 172.21.15.81:0/3517065344 conn(0x7fec8c072950 msgr2=0x7fec8c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:36.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.518+0000 7fec91b44640 1 -- 172.21.15.81:0/3517065344 shutdown_connections 2024-06-29T14:41:36.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.518+0000 7fec91b44640 1 -- 172.21.15.81:0/3517065344 wait complete. 2024-06-29T14:41:36.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.519+0000 7fec91b44640 1 Processor -- start 2024-06-29T14:41:36.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.520+0000 7fec91b44640 1 -- start start 2024-06-29T14:41:36.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec91b44640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec8c0ad0c0 0x7fec8c099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec91b44640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec8c0aea70 0x7fec8c1f2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec91b44640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8c0af3c0 0x7fec8c14d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec91b44640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c06cf80 con 0x7fec8c0af3c0 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec91b44640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c06ce00 con 0x7fec8c0ad0c0 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec91b44640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fec8c06d100 con 0x7fec8c0aea70 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8c0af3c0 0x7fec8c14d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8c0af3c0 0x7fec8c14d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37384/0 (socket says 172.21.15.81:37384) 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8bfff640 1 -- 172.21.15.81:0/1933751995 learned_addr learned my addr 172.21.15.81:0/1933751995 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8affd640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec8c0aea70 0x7fec8c1f2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8b7fe640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec8c0ad0c0 0x7fec8c099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8bfff640 1 -- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec8c0aea70 msgr2=0x7fec8c1f2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8bfff640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec8c0aea70 0x7fec8c1f2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8bfff640 1 -- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec8c0ad0c0 msgr2=0x7fec8c099ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8bfff640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec8c0ad0c0 0x7fec8c099ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8bfff640 1 -- 172.21.15.81:0/1933751995 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8c153ec0 con 0x7fec8c0af3c0 2024-06-29T14:41:36.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.521+0000 7fec8bfff640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8c0af3c0 0x7fec8c14d610 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fec8400e960 tx=0x7fec8400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:36.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.522+0000 7fec73fff640 1 -- 172.21.15.81:0/1933751995 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84004d20 con 0x7fec8c0af3c0 2024-06-29T14:41:36.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.522+0000 7fec73fff640 1 -- 172.21.15.81:0/1933751995 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec8400cac0 con 0x7fec8c0af3c0 2024-06-29T14:41:36.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.522+0000 7fec73fff640 1 -- 172.21.15.81:0/1933751995 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84005020 con 0x7fec8c0af3c0 2024-06-29T14:41:36.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.522+0000 7fec91b44640 1 -- 172.21.15.81:0/1933751995 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8c153be0 con 0x7fec8c0af3c0 2024-06-29T14:41:36.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.522+0000 7fec91b44640 1 -- 172.21.15.81:0/1933751995 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec8c14ce10 con 0x7fec8c0af3c0 2024-06-29T14:41:36.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.523+0000 7fec91b44640 1 -- 172.21.15.81:0/1933751995 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec8c06d5b0 con 0x7fec8c0af3c0 2024-06-29T14:41:36.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.524+0000 7fec73fff640 1 -- 172.21.15.81:0/1933751995 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7fec840051c0 con 0x7fec8c0af3c0 2024-06-29T14:41:36.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.524+0000 7fec73fff640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fec78076440 0x7fec78078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.524+0000 7fec73fff640 1 -- 172.21.15.81:0/1933751995 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7fec84098780 con 0x7fec8c0af3c0 2024-06-29T14:41:36.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.524+0000 7fec8b7fe640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fec78076440 0x7fec78078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.525+0000 7fec8b7fe640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fec78076440 0x7fec78078950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0046b0 tx=0x7fec7c007eb0 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:36.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.528+0000 7fec73fff640 1 -- 172.21.15.81:0/1933751995 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fec8c153ec0 con 0x7fec8c0af3c0 2024-06-29T14:41:36.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.732+0000 7fec91b44640 1 -- 172.21.15.81:0/1933751995 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1y", "target": ["mon-mgr", ""]}) -- 0x7fec8c155bb0 con 0x7fec78076440 2024-06-29T14:41:36.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.787+0000 7fec73fff640 1 -- 172.21.15.81:0/1933751995 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+44 (secure 0 0 0) 0x7fec8c155bb0 con 0x7fec78076440 2024-06-29T14:41:36.786 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1/yearly 2024-06-29T14:41:36.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.789+0000 7fec91b44640 1 -- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fec78076440 msgr2=0x7fec78078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.789+0000 7fec91b44640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fec78076440 0x7fec78078950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0046b0 tx=0x7fec7c007eb0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.789+0000 7fec91b44640 1 -- 172.21.15.81:0/1933751995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8c0af3c0 msgr2=0x7fec8c14d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.789+0000 7fec91b44640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8c0af3c0 0x7fec8c14d610 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fec8400e960 tx=0x7fec8400ee20 comp rx=0 tx=0).stop 2024-06-29T14:41:36.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.789+0000 7fec91b44640 1 -- 172.21.15.81:0/1933751995 shutdown_connections 2024-06-29T14:41:36.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.789+0000 7fec91b44640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fec78076440 0x7fec78078950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.789+0000 7fec91b44640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8c0af3c0 0x7fec8c14d610 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.789+0000 7fec91b44640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec8c0aea70 0x7fec8c1f2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.790+0000 7fec91b44640 1 --2- 172.21.15.81:0/1933751995 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec8c0ad0c0 0x7fec8c099ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.790+0000 7fec91b44640 1 -- 172.21.15.81:0/1933751995 >> 172.21.15.81:0/1933751995 conn(0x7fec8c072950 msgr2=0x7fec8c151020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:36.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.790+0000 7fec91b44640 1 -- 172.21.15.81:0/1933751995 shutdown_connections 2024-06-29T14:41:36.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.790+0000 7fec91b44640 1 -- 172.21.15.81:0/1933751995 wait complete. 2024-06-29T14:41:36.799 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2024-06-29T14:41:36.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:36.800 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2024-06-29T14:41:36.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2024-06-29T14:41:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.974+0000 7fc6da4e4640 1 -- 172.21.15.81:0/8183849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6d41412f0 msgr2=0x7fc6d40cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.974+0000 7fc6da4e4640 1 --2- 172.21.15.81:0/8183849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6d41412f0 0x7fc6d40cef90 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4005fd0 tx=0x7fc6c402a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.974+0000 7fc6da4e4640 1 -- 172.21.15.81:0/8183849 shutdown_connections 2024-06-29T14:41:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.974+0000 7fc6da4e4640 1 --2- 172.21.15.81:0/8183849 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6d4073740 0x7fc6d415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.974+0000 7fc6da4e4640 1 --2- 172.21.15.81:0/8183849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6d41412f0 0x7fc6d40cef90 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.974+0000 7fc6da4e4640 1 -- 172.21.15.81:0/8183849 >> 172.21.15.81:0/8183849 conn(0x7fc6d406ccc0 msgr2=0x7fc6d40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:36.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.975+0000 7fc6da4e4640 1 -- 172.21.15.81:0/8183849 shutdown_connections 2024-06-29T14:41:36.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.975+0000 7fc6da4e4640 1 -- 172.21.15.81:0/8183849 wait complete. 2024-06-29T14:41:36.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.976+0000 7fc6da4e4640 1 Processor -- start 2024-06-29T14:41:36.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.976+0000 7fc6da4e4640 1 -- start start 2024-06-29T14:41:36.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6da4e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6d40cf4d0 0x7fc6d4134f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6da4e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6d41412f0 0x7fc6d4156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6da4e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6d415d4f0 0x7fc6d415d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6da4e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d40b3210 con 0x7fc6d41412f0 2024-06-29T14:41:36.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6da4e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d415f270 con 0x7fc6d40cf4d0 2024-06-29T14:41:36.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6da4e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d40b3390 con 0x7fc6d415d4f0 2024-06-29T14:41:36.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6d41412f0 0x7fc6d4156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6d40cf4d0 0x7fc6d4134f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6d41412f0 0x7fc6d4156390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37396/0 (socket says 172.21.15.81:37396) 2024-06-29T14:41:36.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d37fe640 1 -- 172.21.15.81:0/605517357 learned_addr learned my addr 172.21.15.81:0/605517357 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:36.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d8a81640 1 --2- 172.21.15.81:0/605517357 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6d415d4f0 0x7fc6d415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d37fe640 1 -- 172.21.15.81:0/605517357 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6d415d4f0 msgr2=0x7fc6d415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d37fe640 1 --2- 172.21.15.81:0/605517357 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6d415d4f0 0x7fc6d415d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d37fe640 1 -- 172.21.15.81:0/605517357 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6d40cf4d0 msgr2=0x7fc6d4134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d37fe640 1 --2- 172.21.15.81:0/605517357 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6d40cf4d0 0x7fc6d4134f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d37fe640 1 -- 172.21.15.81:0/605517357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d41f1b00 con 0x7fc6d41412f0 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d37fe640 1 --2- 172.21.15.81:0/605517357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6d41412f0 0x7fc6d4156390 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800c430 tx=0x7fc6c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.978+0000 7fc6d0ff9640 1 -- 172.21.15.81:0/605517357 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8017070 con 0x7fc6d41412f0 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.979+0000 7fc6d0ff9640 1 -- 172.21.15.81:0/605517357 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6c80053c0 con 0x7fc6d41412f0 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.979+0000 7fc6da4e4640 1 -- 172.21.15.81:0/605517357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d415de40 con 0x7fc6d41412f0 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.979+0000 7fc6d0ff9640 1 -- 172.21.15.81:0/605517357 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8003850 con 0x7fc6d41412f0 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.979+0000 7fc6da4e4640 1 -- 172.21.15.81:0/605517357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d4157160 con 0x7fc6d41412f0 2024-06-29T14:41:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.979+0000 7fc6da4e4640 1 -- 172.21.15.81:0/605517357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d40b38b0 con 0x7fc6d41412f0 2024-06-29T14:41:36.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.980+0000 7fc6d0ff9640 1 -- 172.21.15.81:0/605517357 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7fc6c801a070 con 0x7fc6d41412f0 2024-06-29T14:41:36.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.981+0000 7fc6d0ff9640 1 --2- 172.21.15.81:0/605517357 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fc6c0076470 0x7fc6c0078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:36.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.981+0000 7fc6d0ff9640 1 -- 172.21.15.81:0/605517357 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7fc6c8096f60 con 0x7fc6d41412f0 2024-06-29T14:41:36.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.981+0000 7fc6d3fff640 1 --2- 172.21.15.81:0/605517357 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fc6c0076470 0x7fc6c0078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:36.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.982+0000 7fc6d3fff640 1 --2- 172.21.15.81:0/605517357 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fc6c0076470 0x7fc6c0078980 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4033000 tx=0x7fc6c4004730 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:36.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:36.985+0000 7fc6d0ff9640 1 -- 172.21.15.81:0/605517357 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc6d415de40 con 0x7fc6d41412f0 2024-06-29T14:41:37.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.189+0000 7fc6da4e4640 1 -- 172.21.15.81:0/605517357 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) -- 0x7fc6d4156c10 con 0x7fc6c0076470 2024-06-29T14:41:37.189 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:37.191+0000 7f85ebe25640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,y 2024-06-29T14:41:37.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.191+0000 7fc6d0ff9640 1 -- 172.21.15.81:0/605517357 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,y) ==== 74+0+0 (secure 0 0 0) 0x7fc6d4156c10 con 0x7fc6c0076470 2024-06-29T14:41:37.190 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,y 2024-06-29T14:41:37.194 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:37.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2024-06-29T14:41:37.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.312+0000 7f1a1a77a640 1 -- 172.21.15.81:0/513450875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1415aed0 msgr2=0x7f1a140b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.312+0000 7f1a1a77a640 1 --2- 172.21.15.81:0/513450875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1415aed0 0x7f1a140b4070 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c005fd0 tx=0x7f1a0c02a940 comp rx=0 tx=0).stop 2024-06-29T14:41:37.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.312+0000 7f1a1a77a640 1 -- 172.21.15.81:0/513450875 shutdown_connections 2024-06-29T14:41:37.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.312+0000 7f1a1a77a640 1 --2- 172.21.15.81:0/513450875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1415aed0 0x7f1a140b4070 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.312+0000 7f1a1a77a640 1 -- 172.21.15.81:0/513450875 >> 172.21.15.81:0/513450875 conn(0x7f1a140d0930 msgr2=0x7f1a140d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:37.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.312+0000 7f1a1a77a640 1 -- 172.21.15.81:0/513450875 shutdown_connections 2024-06-29T14:41:37.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.312+0000 7f1a1a77a640 1 -- 172.21.15.81:0/513450875 wait complete. 2024-06-29T14:41:37.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.314+0000 7f1a1a77a640 1 Processor -- start 2024-06-29T14:41:37.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.314+0000 7f1a1a77a640 1 -- start start 2024-06-29T14:41:37.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a1a77a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1406f6a0 0x7f1a141e3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a1a77a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a140b57b0 0x7f1a141f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a1a77a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1415aed0 0x7f1a141e7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a1a77a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a140b4730 con 0x7f1a1406f6a0 2024-06-29T14:41:37.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a1a77a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a140b45b0 con 0x7f1a1415aed0 2024-06-29T14:41:37.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a1a77a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a140b48b0 con 0x7f1a140b57b0 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a13fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1406f6a0 0x7f1a141e3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a13fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1406f6a0 0x7f1a141e3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37408/0 (socket says 172.21.15.81:37408) 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a13fff640 1 -- 172.21.15.81:0/600529876 learned_addr learned my addr 172.21.15.81:0/600529876 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a18d17640 1 --2- 172.21.15.81:0/600529876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1415aed0 0x7f1a141e7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a137fe640 1 --2- 172.21.15.81:0/600529876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a140b57b0 0x7f1a141f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a13fff640 1 -- 172.21.15.81:0/600529876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a140b57b0 msgr2=0x7f1a141f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a13fff640 1 --2- 172.21.15.81:0/600529876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a140b57b0 0x7f1a141f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a13fff640 1 -- 172.21.15.81:0/600529876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1415aed0 msgr2=0x7f1a141e7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a13fff640 1 --2- 172.21.15.81:0/600529876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1415aed0 0x7f1a141e7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a13fff640 1 -- 172.21.15.81:0/600529876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a141f3f80 con 0x7f1a1406f6a0 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a13fff640 1 --2- 172.21.15.81:0/600529876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1406f6a0 0x7f1a141e3e90 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f1a0400c400 tx=0x7f1a0400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a10ff9640 1 -- 172.21.15.81:0/600529876 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a04011790 con 0x7f1a1406f6a0 2024-06-29T14:41:37.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a10ff9640 1 -- 172.21.15.81:0/600529876 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a04017070 con 0x7f1a1406f6a0 2024-06-29T14:41:37.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a1a77a640 1 -- 172.21.15.81:0/600529876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a141f4260 con 0x7f1a1406f6a0 2024-06-29T14:41:37.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.316+0000 7f1a10ff9640 1 -- 172.21.15.81:0/600529876 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a04011ce0 con 0x7f1a1406f6a0 2024-06-29T14:41:37.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.317+0000 7f1a1a77a640 1 -- 172.21.15.81:0/600529876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a141e3970 con 0x7f1a1406f6a0 2024-06-29T14:41:37.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.318+0000 7f1a1a77a640 1 -- 172.21.15.81:0/600529876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a1406e2a0 con 0x7f1a1406f6a0 2024-06-29T14:41:37.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.318+0000 7f1a10ff9640 1 -- 172.21.15.81:0/600529876 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 72) ==== 98321+0+0 (secure 0 0 0) 0x7f1a0401c460 con 0x7f1a1406f6a0 2024-06-29T14:41:37.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.319+0000 7f1a10ff9640 1 --2- 172.21.15.81:0/600529876 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1a00076440 0x7f1a00078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.319+0000 7f1a10ff9640 1 -- 172.21.15.81:0/600529876 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f1a040978f0 con 0x7f1a1406f6a0 2024-06-29T14:41:37.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.319+0000 7f1a137fe640 1 --2- 172.21.15.81:0/600529876 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1a00076440 0x7f1a00078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.319+0000 7f1a137fe640 1 --2- 172.21.15.81:0/600529876 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1a00076440 0x7f1a00078950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1a141e0b90 tx=0x7f1a08009040 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.323+0000 7f1a10ff9640 1 -- 172.21.15.81:0/600529876 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a1406e2a0 con 0x7f1a1406f6a0 2024-06-29T14:41:37.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.414+0000 7f1a10ff9640 1 -- 172.21.15.81:0/600529876 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7f1a04095c50 con 0x7f1a1406f6a0 2024-06-29T14:41:37.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.527+0000 7f1a1a77a640 1 -- 172.21.15.81:0/600529876 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) -- 0x7f1a141e3480 con 0x7f1a00076440 2024-06-29T14:41:37.527 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:37.530+0000 7f85ebe25640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:37.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.530+0000 7f1a10ff9640 1 -- 172.21.15.81:0/600529876 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7f1a141e3480 con 0x7f1a00076440 2024-06-29T14:41:37.528 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:37.534 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2024-06-29T14:41:37.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.652+0000 7f1ef46a7640 1 -- 172.21.15.81:0/2927027498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eec0b35b0 msgr2=0x7f1eec15b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.652+0000 7f1ef46a7640 1 --2- 172.21.15.81:0/2927027498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eec0b35b0 0x7f1eec15b5b0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0cd500 tx=0x7f1ee802a4a0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.652+0000 7f1ef46a7640 1 -- 172.21.15.81:0/2927027498 shutdown_connections 2024-06-29T14:41:37.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.652+0000 7f1ef46a7640 1 --2- 172.21.15.81:0/2927027498 >> v2:172.21.15.167:3300/0 conn(0x7f1eec06f660 0x7f1eec06fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.652+0000 7f1ef46a7640 1 --2- 172.21.15.81:0/2927027498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eec0b35b0 0x7f1eec15b5b0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.652+0000 7f1ef46a7640 1 -- 172.21.15.81:0/2927027498 >> 172.21.15.81:0/2927027498 conn(0x7f1eec06ccc0 msgr2=0x7f1eec0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:37.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.653+0000 7f1ef46a7640 1 -- 172.21.15.81:0/2927027498 shutdown_connections 2024-06-29T14:41:37.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.653+0000 7f1ef46a7640 1 -- 172.21.15.81:0/2927027498 wait complete. 2024-06-29T14:41:37.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.654+0000 7f1ef46a7640 1 Processor -- start 2024-06-29T14:41:37.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.654+0000 7f1ef46a7640 1 -- start start 2024-06-29T14:41:37.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef46a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eec06f660 0x7f1eec1f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef46a7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1eec0b35b0 0x7f1eec148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef46a7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1eec0b4430 0x7f1eec0cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef46a7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec0722b0 con 0x7f1eec06f660 2024-06-29T14:41:37.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef46a7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec072130 con 0x7f1eec0b4430 2024-06-29T14:41:37.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef46a7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1eec072430 con 0x7f1eec0b35b0 2024-06-29T14:41:37.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef2443640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eec06f660 0x7f1eec1f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef2443640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eec06f660 0x7f1eec1f39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37436/0 (socket says 172.21.15.81:37436) 2024-06-29T14:41:37.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef2443640 1 -- 172.21.15.81:0/1765685534 learned_addr learned my addr 172.21.15.81:0/1765685534 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:37.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef2c44640 1 --2- 172.21.15.81:0/1765685534 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1eec0b4430 0x7f1eec0cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef1c42640 1 --2- 172.21.15.81:0/1765685534 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1eec0b35b0 0x7f1eec148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef2443640 1 -- 172.21.15.81:0/1765685534 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1eec0b35b0 msgr2=0x7f1eec148e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef2443640 1 --2- 172.21.15.81:0/1765685534 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1eec0b35b0 0x7f1eec148e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef2443640 1 -- 172.21.15.81:0/1765685534 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1eec0b4430 msgr2=0x7f1eec0cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef2443640 1 --2- 172.21.15.81:0/1765685534 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1eec0b4430 0x7f1eec0cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef2443640 1 -- 172.21.15.81:0/1765685534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eec1f6500 con 0x7f1eec06f660 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef2443640 1 --2- 172.21.15.81:0/1765685534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eec06f660 0x7f1eec1f39d0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009920 tx=0x7f1ee8033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ee2ffd640 1 -- 172.21.15.81:0/1765685534 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8031e20 con 0x7f1eec06f660 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ee2ffd640 1 -- 172.21.15.81:0/1765685534 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ee8005110 con 0x7f1eec06f660 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ee2ffd640 1 -- 172.21.15.81:0/1765685534 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8005560 con 0x7f1eec06f660 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef46a7640 1 -- 172.21.15.81:0/1765685534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eec1f67a0 con 0x7f1eec06f660 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.656+0000 7f1ef46a7640 1 -- 172.21.15.81:0/1765685534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eec1f6d60 con 0x7f1eec06f660 2024-06-29T14:41:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.658+0000 7f1ef46a7640 1 -- 172.21.15.81:0/1765685534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eec15d880 con 0x7f1eec06f660 2024-06-29T14:41:37.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.658+0000 7f1ee2ffd640 1 -- 172.21.15.81:0/1765685534 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7f1ee80047a0 con 0x7f1eec06f660 2024-06-29T14:41:37.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.658+0000 7f1ee2ffd640 1 --2- 172.21.15.81:0/1765685534 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1ed0076500 0x7f1ed0078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.659+0000 7f1ee2ffd640 1 -- 172.21.15.81:0/1765685534 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f1ee80b9060 con 0x7f1eec06f660 2024-06-29T14:41:37.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.659+0000 7f1ef1c42640 1 --2- 172.21.15.81:0/1765685534 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1ed0076500 0x7f1ed0078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.659+0000 7f1ef1c42640 1 --2- 172.21.15.81:0/1765685534 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1ed0076500 0x7f1ed0078a10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1eec15db30 tx=0x7f1edc009040 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.663+0000 7f1ee2ffd640 1 -- 172.21.15.81:0/1765685534 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1ee803f030 con 0x7f1eec06f660 2024-06-29T14:41:37.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.868+0000 7f1ef46a7640 1 -- 172.21.15.81:0/1765685534 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) -- 0x7f1eec069550 con 0x7f1ed0076500 2024-06-29T14:41:37.868 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:37.871+0000 7f85ebe25640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:37.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.870+0000 7f1ee2ffd640 1 -- 172.21.15.81:0/1765685534 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7f1eec069550 con 0x7f1ed0076500 2024-06-29T14:41:37.868 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:37.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2024-06-29T14:41:37.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.993+0000 7fefe654a640 1 -- 172.21.15.81:0/1204985677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefc8004800 msgr2=0x7fefc8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.993+0000 7fefe654a640 1 --2- 172.21.15.81:0/1204985677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefc8004800 0x7fefc8004c10 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009b50 tx=0x7fefc8005830 comp rx=0 tx=0).stop 2024-06-29T14:41:37.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.993+0000 7fefe654a640 1 -- 172.21.15.81:0/1204985677 shutdown_connections 2024-06-29T14:41:37.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.993+0000 7fefe654a640 1 --2- 172.21.15.81:0/1204985677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefc8004800 0x7fefc8004c10 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.993+0000 7fefe654a640 1 -- 172.21.15.81:0/1204985677 >> 172.21.15.81:0/1204985677 conn(0x7fefe009cf30 msgr2=0x7fefe00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:37.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.993+0000 7fefe654a640 1 -- 172.21.15.81:0/1204985677 shutdown_connections 2024-06-29T14:41:37.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.993+0000 7fefe654a640 1 -- 172.21.15.81:0/1204985677 wait complete. 2024-06-29T14:41:37.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.994+0000 7fefe654a640 1 Processor -- start 2024-06-29T14:41:37.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.995+0000 7fefe654a640 1 -- start start 2024-06-29T14:41:37.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe654a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc8004800 0x7fefe014e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe654a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fefe00cc5a0 0x7fefe01e8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe654a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe01e8e60 0x7fefe00cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe654a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fefe00d0860 con 0x7fefe01e8e60 2024-06-29T14:41:37.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe654a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fefe00d06e0 con 0x7fefc8004800 2024-06-29T14:41:37.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe654a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fefe00d09e0 con 0x7fefe00cc5a0 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe4ae7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe01e8e60 0x7fefe00cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe4ae7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe01e8e60 0x7fefe00cacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37454/0 (socket says 172.21.15.81:37454) 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe4ae7640 1 -- 172.21.15.81:0/2276158795 learned_addr learned my addr 172.21.15.81:0/2276158795 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefdf7fe640 1 --2- 172.21.15.81:0/2276158795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fefe00cc5a0 0x7fefe01e8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe4ae7640 1 -- 172.21.15.81:0/2276158795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fefe00cc5a0 msgr2=0x7fefe01e8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe4ae7640 1 --2- 172.21.15.81:0/2276158795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fefe00cc5a0 0x7fefe01e8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe4ae7640 1 -- 172.21.15.81:0/2276158795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc8004800 msgr2=0x7fefe014e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefdffff640 1 --2- 172.21.15.81:0/2276158795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc8004800 0x7fefe014e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe4ae7640 1 --2- 172.21.15.81:0/2276158795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fefc8004800 0x7fefe014e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.996+0000 7fefe4ae7640 1 -- 172.21.15.81:0/2276158795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe01e9270 con 0x7fefe01e8e60 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.997+0000 7fefe4ae7640 1 --2- 172.21.15.81:0/2276158795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe01e8e60 0x7fefe00cacc0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fefd8001330 tx=0x7fefd80019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.997+0000 7fefdcff9640 1 -- 172.21.15.81:0/2276158795 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8011870 con 0x7fefe01e8e60 2024-06-29T14:41:37.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.997+0000 7fefdcff9640 1 -- 172.21.15.81:0/2276158795 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefd8017070 con 0x7fefe01e8e60 2024-06-29T14:41:37.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.997+0000 7fefe654a640 1 -- 172.21.15.81:0/2276158795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe014e570 con 0x7fefe01e8e60 2024-06-29T14:41:37.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.997+0000 7fefdcff9640 1 -- 172.21.15.81:0/2276158795 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8011dc0 con 0x7fefe01e8e60 2024-06-29T14:41:37.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.997+0000 7fefe654a640 1 -- 172.21.15.81:0/2276158795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe00cd610 con 0x7fefe01e8e60 2024-06-29T14:41:37.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.998+0000 7fefe654a640 1 -- 172.21.15.81:0/2276158795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe00b31d0 con 0x7fefe01e8e60 2024-06-29T14:41:37.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.999+0000 7fefdcff9640 1 -- 172.21.15.81:0/2276158795 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7fefd801c460 con 0x7fefe01e8e60 2024-06-29T14:41:37.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.999+0000 7fefdcff9640 1 --2- 172.21.15.81:0/2276158795 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fefcc0764d0 0x7fefcc0789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:37.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:37.999+0000 7fefdcff9640 1 -- 172.21.15.81:0/2276158795 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7fefd8098620 con 0x7fefe01e8e60 2024-06-29T14:41:37.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.000+0000 7fefdffff640 1 --2- 172.21.15.81:0/2276158795 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fefcc0764d0 0x7fefcc0789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:37.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.000+0000 7fefdffff640 1 --2- 172.21.15.81:0/2276158795 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fefcc0764d0 0x7fefcc0789e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fefd0001eb0 tx=0x7fefd00067a0 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:38.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.004+0000 7fefdcff9640 1 -- 172.21.15.81:0/2276158795 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fefe01e9270 con 0x7fefe01e8e60 2024-06-29T14:41:38.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.202+0000 7fefe654a640 1 -- 172.21.15.81:0/2276158795 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) -- 0x7fefe00cdf40 con 0x7fefcc0764d0 2024-06-29T14:41:38.202 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:38.204+0000 7f85ebe25640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:38.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.205+0000 7fefdcff9640 1 -- 172.21.15.81:0/2276158795 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7fefe00cdf40 con 0x7fefcc0764d0 2024-06-29T14:41:38.202 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:38.208 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:38.208 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2024-06-29T14:41:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.326+0000 7f9b71d63640 1 -- 172.21.15.81:0/197462878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b6c073570 msgr2=0x7f9b6c15e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.326+0000 7f9b71d63640 1 --2- 172.21.15.81:0/197462878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b6c073570 0x7f9b6c15e960 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c070e50 tx=0x7f9b6402a560 comp rx=0 tx=0).stop 2024-06-29T14:41:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.326+0000 7f9b71d63640 1 -- 172.21.15.81:0/197462878 shutdown_connections 2024-06-29T14:41:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.326+0000 7f9b71d63640 1 --2- 172.21.15.81:0/197462878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b6c073570 0x7f9b6c15e960 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.326+0000 7f9b71d63640 1 --2- 172.21.15.81:0/197462878 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b6c0cf4d0 0x7f9b6c073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.326+0000 7f9b71d63640 1 --2- 172.21.15.81:0/197462878 >> v2:172.21.15.167:3300/0 conn(0x7f9b6c1412f0 0x7f9b6c0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.326+0000 7f9b71d63640 1 -- 172.21.15.81:0/197462878 >> 172.21.15.81:0/197462878 conn(0x7f9b6c06ccc0 msgr2=0x7f9b6c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.326+0000 7f9b71d63640 1 -- 172.21.15.81:0/197462878 shutdown_connections 2024-06-29T14:41:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.326+0000 7f9b71d63640 1 -- 172.21.15.81:0/197462878 wait complete. 2024-06-29T14:41:38.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.328+0000 7f9b71d63640 1 Processor -- start 2024-06-29T14:41:38.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.328+0000 7f9b71d63640 1 -- start start 2024-06-29T14:41:38.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.329+0000 7f9b71d63640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b6c073570 0x7f9b6c13cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.329+0000 7f9b71d63640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b6c0cf4d0 0x7f9b6c0ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.329+0000 7f9b71d63640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b6c1412f0 0x7f9b6c0aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.329+0000 7f9b71d63640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c15f230 con 0x7f9b6c073570 2024-06-29T14:41:38.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.329+0000 7f9b71d63640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c15f0b0 con 0x7f9b6c1412f0 2024-06-29T14:41:38.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.329+0000 7f9b71d63640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6c0b3210 con 0x7f9b6c0cf4d0 2024-06-29T14:41:38.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b6c073570 0x7f9b6c13cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b6c073570 0x7f9b6c13cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37466/0 (socket says 172.21.15.81:37466) 2024-06-29T14:41:38.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6b7fe640 1 -- 172.21.15.81:0/2716724187 learned_addr learned my addr 172.21.15.81:0/2716724187 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:38.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6affd640 1 --2- 172.21.15.81:0/2716724187 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b6c0cf4d0 0x7f9b6c0ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6bfff640 1 --2- 172.21.15.81:0/2716724187 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b6c1412f0 0x7f9b6c0aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6b7fe640 1 -- 172.21.15.81:0/2716724187 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b6c0cf4d0 msgr2=0x7f9b6c0ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6b7fe640 1 --2- 172.21.15.81:0/2716724187 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b6c0cf4d0 0x7f9b6c0ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6b7fe640 1 -- 172.21.15.81:0/2716724187 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b6c1412f0 msgr2=0x7f9b6c0aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6b7fe640 1 --2- 172.21.15.81:0/2716724187 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b6c1412f0 0x7f9b6c0aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6b7fe640 1 -- 172.21.15.81:0/2716724187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c167ad0 con 0x7f9b6c073570 2024-06-29T14:41:38.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6bfff640 1 --2- 172.21.15.81:0/2716724187 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b6c1412f0 0x7f9b6c0aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:38.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b6b7fe640 1 --2- 172.21.15.81:0/2716724187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b6c073570 0x7f9b6c13cb30 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00cd10 tx=0x7f9b5c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:38.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b53fff640 1 -- 172.21.15.81:0/2716724187 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c00f450 con 0x7f9b6c073570 2024-06-29T14:41:38.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b53fff640 1 -- 172.21.15.81:0/2716724187 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b5c00f6e0 con 0x7f9b6c073570 2024-06-29T14:41:38.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b53fff640 1 -- 172.21.15.81:0/2716724187 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c00fb10 con 0x7f9b6c073570 2024-06-29T14:41:38.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b71d63640 1 -- 172.21.15.81:0/2716724187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6c1527b0 con 0x7f9b6c073570 2024-06-29T14:41:38.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.330+0000 7f9b71d63640 1 -- 172.21.15.81:0/2716724187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6c152d70 con 0x7f9b6c073570 2024-06-29T14:41:38.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.331+0000 7f9b71d63640 1 -- 172.21.15.81:0/2716724187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b6c15e960 con 0x7f9b6c073570 2024-06-29T14:41:38.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.332+0000 7f9b53fff640 1 -- 172.21.15.81:0/2716724187 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7f9b5c00fcb0 con 0x7f9b6c073570 2024-06-29T14:41:38.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.332+0000 7f9b53fff640 1 --2- 172.21.15.81:0/2716724187 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f9b580764d0 0x7f9b580789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.333+0000 7f9b53fff640 1 -- 172.21.15.81:0/2716724187 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f9b5c098890 con 0x7f9b6c073570 2024-06-29T14:41:38.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.333+0000 7f9b6affd640 1 --2- 172.21.15.81:0/2716724187 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f9b580764d0 0x7f9b580789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.333+0000 7f9b6affd640 1 --2- 172.21.15.81:0/2716724187 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f9b580764d0 0x7f9b580789e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c15d9f0 tx=0x7f9b60005d70 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:38.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.337+0000 7f9b53fff640 1 -- 172.21.15.81:0/2716724187 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9b5c014030 con 0x7f9b6c073570 2024-06-29T14:41:38.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.538+0000 7f9b71d63640 1 -- 172.21.15.81:0/2716724187 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) -- 0x7f9b6c06fa30 con 0x7f9b580764d0 2024-06-29T14:41:38.538 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:38.540+0000 7f85ebe25640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-06-29T14:41:38.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.541+0000 7f9b53fff640 1 -- 172.21.15.81:0/2716724187 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) ==== 96+0+0 (secure 0 0 0) 0x7f9b6c06fa30 con 0x7f9b580764d0 2024-06-29T14:41:38.539 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2024-06-29T14:41:38.543 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:38.544 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2024-06-29T14:41:38.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.662+0000 7f5d9219f640 1 -- 172.21.15.81:0/2307162867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d74004d40 msgr2=0x7f5d740051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.662+0000 7f5d9219f640 1 --2- 172.21.15.81:0/2307162867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d74004d40 0x7f5d740051f0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009b50 tx=0x7f5d74005830 comp rx=0 tx=0).stop 2024-06-29T14:41:38.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.662+0000 7f5d8bfff640 1 -- 172.21.15.81:0/2307162867 reap_dead start 2024-06-29T14:41:38.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.663+0000 7f5d9219f640 1 -- 172.21.15.81:0/2307162867 shutdown_connections 2024-06-29T14:41:38.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.663+0000 7f5d9219f640 1 -- 172.21.15.81:0/2307162867 >> 172.21.15.81:0/2307162867 conn(0x7f5d8c06ccc0 msgr2=0x7f5d8c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:38.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.663+0000 7f5d9219f640 1 -- 172.21.15.81:0/2307162867 shutdown_connections 2024-06-29T14:41:38.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.663+0000 7f5d9219f640 1 -- 172.21.15.81:0/2307162867 wait complete. 2024-06-29T14:41:38.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.664+0000 7f5d9219f640 1 Processor -- start 2024-06-29T14:41:38.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.665+0000 7f5d9219f640 1 -- start start 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d9219f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d74004d40 0x7f5d8c155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d9219f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d8c13d050 0x7f5d8c13ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d9219f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d8c13b060 0x7f5d8c141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d9219f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c164c30 con 0x7f5d74004d40 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d9219f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c164ab0 con 0x7f5d8c13d050 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d9219f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5d8c164db0 con 0x7f5d8c13b060 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d74004d40 0x7f5d8c155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d74004d40 0x7f5d8c155780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37484/0 (socket says 172.21.15.81:37484) 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8b7fe640 1 -- 172.21.15.81:0/2500300291 learned_addr learned my addr 172.21.15.81:0/2500300291 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8affd640 1 --2- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d8c13d050 0x7f5d8c13ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8bfff640 1 --2- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d8c13b060 0x7f5d8c141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8b7fe640 1 -- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d8c13b060 msgr2=0x7f5d8c141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8b7fe640 1 --2- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d8c13b060 0x7f5d8c141a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8b7fe640 1 -- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d8c13d050 msgr2=0x7f5d8c13ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8b7fe640 1 --2- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d8c13d050 0x7f5d8c13ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8b7fe640 1 -- 172.21.15.81:0/2500300291 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d8c13e250 con 0x7f5d74004d40 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8bfff640 1 --2- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d8c13b060 0x7f5d8c141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8affd640 1 --2- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d8c13d050 0x7f5d8c13ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d8b7fe640 1 --2- 172.21.15.81:0/2500300291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d74004d40 0x7f5d8c155780 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c006090 tx=0x7f5d7c006550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d73fff640 1 -- 172.21.15.81:0/2500300291 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c015070 con 0x7f5d74004d40 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d73fff640 1 -- 172.21.15.81:0/2500300291 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d7c00f9d0 con 0x7f5d74004d40 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d73fff640 1 -- 172.21.15.81:0/2500300291 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c00fe20 con 0x7f5d74004d40 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.666+0000 7f5d9219f640 1 -- 172.21.15.81:0/2500300291 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d8c13b470 con 0x7f5d74004d40 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.667+0000 7f5d9219f640 1 -- 172.21.15.81:0/2500300291 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d8c13dc90 con 0x7f5d74004d40 2024-06-29T14:41:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.668+0000 7f5d9219f640 1 -- 172.21.15.81:0/2500300291 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d8c13cad0 con 0x7f5d74004d40 2024-06-29T14:41:38.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.668+0000 7f5d73fff640 1 -- 172.21.15.81:0/2500300291 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7f5d7c011430 con 0x7f5d74004d40 2024-06-29T14:41:38.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.669+0000 7f5d73fff640 1 --2- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f5d780764d0 0x7f5d780789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:38.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.669+0000 7f5d73fff640 1 -- 172.21.15.81:0/2500300291 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f5d7c098c20 con 0x7f5d74004d40 2024-06-29T14:41:38.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.669+0000 7f5d8affd640 1 --2- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f5d780764d0 0x7f5d780789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:38.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.669+0000 7f5d8affd640 1 --2- 172.21.15.81:0/2500300291 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f5d780764d0 0x7f5d780789e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5d80001240 tx=0x7f5d80001110 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:38.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.673+0000 7f5d73fff640 1 -- 172.21.15.81:0/2500300291 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d8c13b470 con 0x7f5d74004d40 2024-06-29T14:41:38.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.873+0000 7f5d9219f640 1 -- 172.21.15.81:0/2500300291 --> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) -- 0x7f5d8c15c570 con 0x7f5d780764d0 2024-06-29T14:41:38.873 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:38.875+0000 7f85ebe25640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:38.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:38.875+0000 7f5d73fff640 1 -- 172.21.15.81:0/2500300291 <== mgr.7109 v2:172.21.15.167:6832/2082979226 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc.) ==== 138+0+0 (secure 0 0 0) 0x7f5d8c15c570 con 0x7f5d780764d0 2024-06-29T14:41:38.873 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,y e.g. 1h or 4d etc. 2024-06-29T14:41:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T14:41:38.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:38.882 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2024-06-29T14:41:38.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:38.899 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2024-06-29T14:41:38.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:38.955 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2024-06-29T14:41:39.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:39.012 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2024-06-29T14:41:39.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:39.069 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2024-06-29T14:41:39.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:39.126 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2024-06-29T14:41:39.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:39.183 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2024-06-29T14:41:39.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T14:41:39.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.396+0000 7fdf652e4640 1 -- 172.21.15.81:0/4171351773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf600b3200 msgr2=0x7fdf600cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:39.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.396+0000 7fdf652e4640 1 --2- 172.21.15.81:0/4171351773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf600b3200 0x7fdf600cc7b0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fdf60074b10 tx=0x7fdf5002a4d0 comp rx=0 tx=0).stop 2024-06-29T14:41:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.396+0000 7fdf652e4640 1 -- 172.21.15.81:0/4171351773 shutdown_connections 2024-06-29T14:41:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.396+0000 7fdf652e4640 1 --2- 172.21.15.81:0/4171351773 >> v2:172.21.15.167:3300/0 conn(0x7fdf600cccf0 0x7fdf600d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.396+0000 7fdf652e4640 1 --2- 172.21.15.81:0/4171351773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf600b3200 0x7fdf600cc7b0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.397+0000 7fdf652e4640 1 -- 172.21.15.81:0/4171351773 >> 172.21.15.81:0/4171351773 conn(0x7fdf600b3680 msgr2=0x7fdf600b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.397+0000 7fdf652e4640 1 -- 172.21.15.81:0/4171351773 shutdown_connections 2024-06-29T14:41:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.397+0000 7fdf652e4640 1 -- 172.21.15.81:0/4171351773 wait complete. 2024-06-29T14:41:39.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.398+0000 7fdf652e4640 1 Processor -- start 2024-06-29T14:41:39.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.398+0000 7fdf652e4640 1 -- start start 2024-06-29T14:41:39.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf652e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf600b3200 0x7fdf600c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf652e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf600cccf0 0x7fdf600c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf652e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf600d0770 0x7fdf6006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf652e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdf600cedb0 con 0x7fdf600b3200 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf652e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdf600cec30 con 0x7fdf600cccf0 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf652e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdf600cef30 con 0x7fdf600d0770 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf5effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf600b3200 0x7fdf600c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf5effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf600b3200 0x7fdf600c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37502/0 (socket says 172.21.15.81:37502) 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf5effd640 1 -- 172.21.15.81:0/3015210536 learned_addr learned my addr 172.21.15.81:0/3015210536 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf5f7fe640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf600d0770 0x7fdf6006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf57fff640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf600cccf0 0x7fdf600c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf5effd640 1 -- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf600d0770 msgr2=0x7fdf6006e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf5effd640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf600d0770 0x7fdf6006e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf5effd640 1 -- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf600cccf0 msgr2=0x7fdf600c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf5effd640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf600cccf0 0x7fdf600c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf5effd640 1 -- 172.21.15.81:0/3015210536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf601e44a0 con 0x7fdf600b3200 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf5effd640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf600b3200 0x7fdf600c7dc0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fdf500099c0 tx=0x7fdf500039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf577fe640 1 -- 172.21.15.81:0/3015210536 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5003e030 con 0x7fdf600b3200 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf577fe640 1 -- 172.21.15.81:0/3015210536 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf500047a0 con 0x7fdf600b3200 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf652e4640 1 -- 172.21.15.81:0/3015210536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf601e3ee0 con 0x7fdf600b3200 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.400+0000 7fdf577fe640 1 -- 172.21.15.81:0/3015210536 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50035040 con 0x7fdf600b3200 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.401+0000 7fdf652e4640 1 -- 172.21.15.81:0/3015210536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf601e1a30 con 0x7fdf600b3200 2024-06-29T14:41:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.401+0000 7fdf652e4640 1 -- 172.21.15.81:0/3015210536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf601e1c90 con 0x7fdf600b3200 2024-06-29T14:41:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.402+0000 7fdf577fe640 1 -- 172.21.15.81:0/3015210536 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 73) ==== 98377+0+0 (secure 0 0 0) 0x7fdf50041030 con 0x7fdf600b3200 2024-06-29T14:41:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.403+0000 7fdf577fe640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fdf40076500 0x7fdf40078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.403+0000 7fdf577fe640 1 -- 172.21.15.81:0/3015210536 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7fdf500b8560 con 0x7fdf600b3200 2024-06-29T14:41:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.403+0000 7fdf57fff640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fdf40076500 0x7fdf40078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.404+0000 7fdf57fff640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fdf40076500 0x7fdf40078a10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdf6006f0f0 tx=0x7fdf48005f60 comp rx=0 tx=0).ready entity=mgr.7109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:39.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.407+0000 7fdf577fe640 1 -- 172.21.15.81:0/3015210536 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdf600d0b80 con 0x7fdf600b3200 2024-06-29T14:41:39.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:39.635+0000 7fdf652e4640 1 -- 172.21.15.81:0/3015210536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7fdf600cc7b0 con 0x7fdf600b3200 2024-06-29T14:41:40.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.425+0000 7fdf577fe640 1 -- 172.21.15.81:0/3015210536 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v74) ==== 93+0+0 (secure 0 0 0) 0x7fdf50081340 con 0x7fdf600b3200 2024-06-29T14:41:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.429+0000 7fdf652e4640 1 -- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fdf40076500 msgr2=0x7fdf40078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.429+0000 7fdf652e4640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fdf40076500 0x7fdf40078a10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdf6006f0f0 tx=0x7fdf48005f60 comp rx=0 tx=0).stop 2024-06-29T14:41:40.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.429+0000 7fdf652e4640 1 -- 172.21.15.81:0/3015210536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf600b3200 msgr2=0x7fdf600c7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.429+0000 7fdf652e4640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf600b3200 0x7fdf600c7dc0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fdf500099c0 tx=0x7fdf500039e0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.429+0000 7fdf652e4640 1 -- 172.21.15.81:0/3015210536 shutdown_connections 2024-06-29T14:41:40.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.429+0000 7fdf652e4640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fdf40076500 0x7fdf40078a10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.429+0000 7fdf652e4640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf600d0770 0x7fdf6006e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.429+0000 7fdf652e4640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf600cccf0 0x7fdf600c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.429+0000 7fdf652e4640 1 --2- 172.21.15.81:0/3015210536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf600b3200 0x7fdf600c7dc0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.429+0000 7fdf652e4640 1 -- 172.21.15.81:0/3015210536 >> 172.21.15.81:0/3015210536 conn(0x7fdf600b3680 msgr2=0x7fdf60159860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:40.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.430+0000 7fdf652e4640 1 -- 172.21.15.81:0/3015210536 shutdown_connections 2024-06-29T14:41:40.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.430+0000 7fdf652e4640 1 -- 172.21.15.81:0/3015210536 wait complete. 2024-06-29T14:41:40.441 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:41:40.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:40.482 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:40.520 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:41:40.520 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:40.520 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:40.520 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:40.539 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:40.539 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:40.595 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:40.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.752+0000 7f06e2970640 1 -- 172.21.15.81:0/2023223231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06dc0b4800 msgr2=0x7f06dc144590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.752+0000 7f06e2970640 1 --2- 172.21.15.81:0/2023223231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06dc0b4800 0x7f06dc144590 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009830 tx=0x7f06c402a510 comp rx=0 tx=0).stop 2024-06-29T14:41:40.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.752+0000 7f06e2970640 1 -- 172.21.15.81:0/2023223231 shutdown_connections 2024-06-29T14:41:40.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.752+0000 7f06e2970640 1 --2- 172.21.15.81:0/2023223231 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06dc072f80 0x7f06dc073390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.752+0000 7f06e2970640 1 --2- 172.21.15.81:0/2023223231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06dc0b4800 0x7f06dc144590 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.752+0000 7f06e2970640 1 -- 172.21.15.81:0/2023223231 >> 172.21.15.81:0/2023223231 conn(0x7f06dc0b2920 msgr2=0x7f06dc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:40.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.752+0000 7f06e2970640 1 -- 172.21.15.81:0/2023223231 shutdown_connections 2024-06-29T14:41:40.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.752+0000 7f06e2970640 1 -- 172.21.15.81:0/2023223231 wait complete. 2024-06-29T14:41:40.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.754+0000 7f06e2970640 1 Processor -- start 2024-06-29T14:41:40.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.754+0000 7f06e2970640 1 -- start start 2024-06-29T14:41:40.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.755+0000 7f06e2970640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06dc072f80 0x7f06dc0aa960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:40.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.755+0000 7f06e2970640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06dc0b4800 0x7f06dc0abf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:40.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.755+0000 7f06e2970640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06dc15b400 0x7f06dc1f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:40.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.755+0000 7f06e2970640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc06f2e0 con 0x7f06dc072f80 2024-06-29T14:41:40.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.755+0000 7f06e2970640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc06f160 con 0x7f06dc0b4800 2024-06-29T14:41:40.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.755+0000 7f06e2970640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f06dc06f460 con 0x7f06dc15b400 2024-06-29T14:41:40.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.755+0000 7f06dbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06dc072f80 0x7f06dc0aa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:40.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.755+0000 7f06dbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06dc072f80 0x7f06dc0aa960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51440/0 (socket says 172.21.15.81:51440) 2024-06-29T14:41:40.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.755+0000 7f06dbfff640 1 -- 172.21.15.81:0/2191662963 learned_addr learned my addr 172.21.15.81:0/2191662963 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:40.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06db7fe640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06dc0b4800 0x7f06dc0abf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:40.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06e0f0d640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06dc15b400 0x7f06dc1f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:40.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06dbfff640 1 -- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06dc15b400 msgr2=0x7f06dc1f2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06dbfff640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06dc15b400 0x7f06dc1f2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06dbfff640 1 -- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06dc0b4800 msgr2=0x7f06dc0abf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06dbfff640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06dc0b4800 0x7f06dc0abf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06dbfff640 1 -- 172.21.15.81:0/2191662963 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06dc0ace10 con 0x7f06dc072f80 2024-06-29T14:41:40.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06dbfff640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06dc072f80 0x7f06dc0aa960 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009950 tx=0x7f06c4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:40.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06d8ff9640 1 -- 172.21.15.81:0/2191662963 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4031d60 con 0x7f06dc072f80 2024-06-29T14:41:40.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06d8ff9640 1 -- 172.21.15.81:0/2191662963 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06c4033e10 con 0x7f06dc072f80 2024-06-29T14:41:40.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06e2970640 1 -- 172.21.15.81:0/2191662963 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06dc0ad0f0 con 0x7f06dc072f80 2024-06-29T14:41:40.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06d8ff9640 1 -- 172.21.15.81:0/2191662963 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4033310 con 0x7f06dc072f80 2024-06-29T14:41:40.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.756+0000 7f06e2970640 1 -- 172.21.15.81:0/2191662963 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06dc0ad6b0 con 0x7f06dc072f80 2024-06-29T14:41:40.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.758+0000 7f06e2970640 1 -- 172.21.15.81:0/2191662963 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06dc15b090 con 0x7f06dc072f80 2024-06-29T14:41:40.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.758+0000 7f06d8ff9640 1 -- 172.21.15.81:0/2191662963 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f06c40334b0 con 0x7f06dc072f80 2024-06-29T14:41:40.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.759+0000 7f06d8ff9640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f06bc076480 0x7f06bc078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:40.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.759+0000 7f06d8ff9640 1 -- 172.21.15.81:0/2191662963 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f06c40b8fa0 con 0x7f06dc072f80 2024-06-29T14:41:40.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.759+0000 7f06db7fe640 1 -- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f06bc076480 msgr2=0x7f06bc078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:40.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.759+0000 7f06db7fe640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f06bc076480 0x7f06bc078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:40.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.763+0000 7f06d8ff9640 1 -- 172.21.15.81:0/2191662963 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f06dc15b090 con 0x7f06dc072f80 2024-06-29T14:41:40.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.952+0000 7f06e2970640 1 -- 172.21.15.81:0/2191662963 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06dc06f930 con 0x7f06dc072f80 2024-06-29T14:41:40.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.953+0000 7f06d8ff9640 1 -- 172.21.15.81:0/2191662963 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+17280 (secure 0 0 0) 0x7f06c403f030 con 0x7f06dc072f80 2024-06-29T14:41:40.952 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:40.952 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":134,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:33.386142+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.352751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:20.014553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/1304833179":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1979367025":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/639087269":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1135028328":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1876032601":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/2923077924":"2024-06-30T14:41:33.386103+0000","172.21.15.167:6833/971640072":"2024-06-30T14:41:33.386103+0000","172.21.15.167:6832/971640072":"2024-06-30T14:41:33.386103+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:40.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.955+0000 7f06e2970640 1 -- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f06bc076480 msgr2=0x7f06bc078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:40.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.955+0000 7f06e2970640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f06bc076480 0x7f06bc078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.955+0000 7f06e2970640 1 -- 172.21.15.81:0/2191662963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06dc072f80 msgr2=0x7f06dc0aa960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:40.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.955+0000 7f06e2970640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06dc072f80 0x7f06dc0aa960 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009950 tx=0x7f06c4033c20 comp rx=0 tx=0).stop 2024-06-29T14:41:40.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.955+0000 7f06e2970640 1 -- 172.21.15.81:0/2191662963 shutdown_connections 2024-06-29T14:41:40.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.955+0000 7f06e2970640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f06bc076480 0x7f06bc078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.955+0000 7f06e2970640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06dc15b400 0x7f06dc1f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.956+0000 7f06e2970640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06dc0b4800 0x7f06dc0abf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.956+0000 7f06e2970640 1 --2- 172.21.15.81:0/2191662963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06dc072f80 0x7f06dc0aa960 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:40.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.956+0000 7f06e2970640 1 -- 172.21.15.81:0/2191662963 >> 172.21.15.81:0/2191662963 conn(0x7f06dc0b2920 msgr2=0x7f06dc0cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:40.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.956+0000 7f06e2970640 1 -- 172.21.15.81:0/2191662963 shutdown_connections 2024-06-29T14:41:40.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:40.956+0000 7f06e2970640 1 -- 172.21.15.81:0/2191662963 wait complete. 2024-06-29T14:41:40.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:41:19.352751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:41:20.014553+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:40.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:41.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.088+0000 7f127d6c5640 1 -- 172.21.15.81:0/1344540113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12640052a0 msgr2=0x7f126400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.088+0000 7f127d6c5640 1 --2- 172.21.15.81:0/1344540113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12640052a0 0x7f126400b9c0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f1270005fd0 tx=0x7f1264005830 comp rx=0 tx=0).stop 2024-06-29T14:41:41.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.088+0000 7f127d6c5640 1 -- 172.21.15.81:0/1344540113 shutdown_connections 2024-06-29T14:41:41.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.088+0000 7f127d6c5640 1 --2- 172.21.15.81:0/1344540113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12640052a0 0x7f126400b9c0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.088+0000 7f127d6c5640 1 -- 172.21.15.81:0/1344540113 >> 172.21.15.81:0/1344540113 conn(0x7f12780d3c50 msgr2=0x7f12780d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.088+0000 7f127d6c5640 1 -- 172.21.15.81:0/1344540113 shutdown_connections 2024-06-29T14:41:41.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.088+0000 7f127d6c5640 1 -- 172.21.15.81:0/1344540113 wait complete. 2024-06-29T14:41:41.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.090+0000 7f127d6c5640 1 Processor -- start 2024-06-29T14:41:41.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.090+0000 7f127d6c5640 1 -- start start 2024-06-29T14:41:41.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f127d6c5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12640052a0 0x7f12781f7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f127d6c5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12780cb9c0 0x7f12780cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f127d6c5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12781e4690 0x7f12781e4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f127d6c5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f12780d10e0 con 0x7f12780cb9c0 2024-06-29T14:41:41.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f127d6c5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f12780d2d10 con 0x7f12640052a0 2024-06-29T14:41:41.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f127d6c5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f12780cf310 con 0x7f12781e4690 2024-06-29T14:41:41.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f12767fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12780cb9c0 0x7f12780cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f12767fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12780cb9c0 0x7f12780cbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51458/0 (socket says 172.21.15.81:51458) 2024-06-29T14:41:41.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f12767fc640 1 -- 172.21.15.81:0/3145295169 learned_addr learned my addr 172.21.15.81:0/3145295169 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:41.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f1276ffd640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12640052a0 0x7f12781f7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.091+0000 7f12777fe640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12781e4690 0x7f12781e4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f12767fc640 1 -- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12781e4690 msgr2=0x7f12781e4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f12767fc640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12781e4690 0x7f12781e4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f12767fc640 1 -- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12640052a0 msgr2=0x7f12781f7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f12767fc640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12640052a0 0x7f12781f7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f12767fc640 1 -- 172.21.15.81:0/3145295169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12781e4fe0 con 0x7f12780cb9c0 2024-06-29T14:41:41.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f1276ffd640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12640052a0 0x7f12781f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:41.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f12767fc640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12780cb9c0 0x7f12780cbdd0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f1260003dd0 tx=0x7f12600017f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:41.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f126f7fe640 1 -- 172.21.15.81:0/3145295169 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1260011640 con 0x7f12780cb9c0 2024-06-29T14:41:41.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f126f7fe640 1 -- 172.21.15.81:0/3145295169 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1260016020 con 0x7f12780cb9c0 2024-06-29T14:41:41.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f127d6c5640 1 -- 172.21.15.81:0/3145295169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12781e8b60 con 0x7f12780cb9c0 2024-06-29T14:41:41.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f126f7fe640 1 -- 172.21.15.81:0/3145295169 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1260011a80 con 0x7f12780cb9c0 2024-06-29T14:41:41.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.092+0000 7f127d6c5640 1 -- 172.21.15.81:0/3145295169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12781e9120 con 0x7f12780cb9c0 2024-06-29T14:41:41.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.093+0000 7f127d6c5640 1 -- 172.21.15.81:0/3145295169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12780d04b0 con 0x7f12780cb9c0 2024-06-29T14:41:41.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.094+0000 7f126f7fe640 1 -- 172.21.15.81:0/3145295169 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f1260011c20 con 0x7f12780cb9c0 2024-06-29T14:41:41.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.094+0000 7f126f7fe640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1258076480 0x7f1258078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.095+0000 7f1276ffd640 1 -- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1258076480 msgr2=0x7f1258078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:41.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.095+0000 7f126f7fe640 1 -- 172.21.15.81:0/3145295169 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f1260098d80 con 0x7f12780cb9c0 2024-06-29T14:41:41.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.095+0000 7f1276ffd640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1258076480 0x7f1258078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:41.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.098+0000 7f126f7fe640 1 -- 172.21.15.81:0/3145295169 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1260020020 con 0x7f12780cb9c0 2024-06-29T14:41:41.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.281+0000 7f127d6c5640 1 -- 172.21.15.81:0/3145295169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f12780d1290 con 0x7f12780cb9c0 2024-06-29T14:41:41.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.282+0000 7f126f7fe640 1 -- 172.21.15.81:0/3145295169 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) ==== 93+0+10 (secure 0 0 0) 0x7f12781e4fe0 con 0x7f12780cb9c0 2024-06-29T14:41:41.280 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:41:41.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 -- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1258076480 msgr2=0x7f1258078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1258076480 0x7f1258078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 -- 172.21.15.81:0/3145295169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12780cb9c0 msgr2=0x7f12780cbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12780cb9c0 0x7f12780cbdd0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f1260003dd0 tx=0x7f12600017f0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 -- 172.21.15.81:0/3145295169 shutdown_connections 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1258076480 0x7f1258078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12781e4690 0x7f12781e4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12780cb9c0 0x7f12780cbdd0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 --2- 172.21.15.81:0/3145295169 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12640052a0 0x7f12781f7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 -- 172.21.15.81:0/3145295169 >> 172.21.15.81:0/3145295169 conn(0x7f12780d3c50 msgr2=0x7f12781e6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 -- 172.21.15.81:0/3145295169 shutdown_connections 2024-06-29T14:41:41.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.284+0000 7f127d6c5640 1 -- 172.21.15.81:0/3145295169 wait complete. 2024-06-29T14:41:41.294 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:41:41.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.415+0000 7f47939a1640 1 -- 172.21.15.81:0/4103817183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f478c0d2a90 msgr2=0x7f478c0d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.415+0000 7f47939a1640 1 --2- 172.21.15.81:0/4103817183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f478c0d2a90 0x7f478c0d09c0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f4784005fd0 tx=0x7f478402a960 comp rx=0 tx=0).stop 2024-06-29T14:41:41.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.415+0000 7f47939a1640 1 -- 172.21.15.81:0/4103817183 shutdown_connections 2024-06-29T14:41:41.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.415+0000 7f47939a1640 1 --2- 172.21.15.81:0/4103817183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f478c0d2a90 0x7f478c0d09c0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.415+0000 7f47939a1640 1 --2- 172.21.15.81:0/4103817183 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f478c0b9040 0x7f478c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.415+0000 7f47939a1640 1 -- 172.21.15.81:0/4103817183 >> 172.21.15.81:0/4103817183 conn(0x7f478c09cf30 msgr2=0x7f478c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.415+0000 7f47939a1640 1 -- 172.21.15.81:0/4103817183 shutdown_connections 2024-06-29T14:41:41.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.416+0000 7f47939a1640 1 -- 172.21.15.81:0/4103817183 wait complete. 2024-06-29T14:41:41.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.417+0000 7f47939a1640 1 Processor -- start 2024-06-29T14:41:41.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.417+0000 7f47939a1640 1 -- start start 2024-06-29T14:41:41.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f47939a1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f478c0b5250 0x7f478c15e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f47939a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f478c0b9040 0x7f478c0b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f47939a1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f478c0d2a90 0x7f478c0b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f47939a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f478c0b7d30 con 0x7f478c0b9040 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f47939a1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f478c0b7bb0 con 0x7f478c0d2a90 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f47939a1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f478c0b7eb0 con 0x7f478c0b5250 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4790f3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f478c0b9040 0x7f478c0b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4790f3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f478c0b9040 0x7f478c0b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51480/0 (socket says 172.21.15.81:51480) 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4790f3c640 1 -- 172.21.15.81:0/1600704376 learned_addr learned my addr 172.21.15.81:0/1600704376 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f479173d640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f478c0b5250 0x7f478c15e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4791f3e640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f478c0d2a90 0x7f478c0b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4790f3c640 1 -- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f478c0b5250 msgr2=0x7f478c15e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4790f3c640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f478c0b5250 0x7f478c15e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4790f3c640 1 -- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f478c0d2a90 msgr2=0x7f478c0b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4790f3c640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f478c0d2a90 0x7f478c0b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4790f3c640 1 -- 172.21.15.81:0/1600704376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f478c0b4e40 con 0x7f478c0b9040 2024-06-29T14:41:41.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4790f3c640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f478c0b9040 0x7f478c0b26f0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f478800c340 tx=0x7f478800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:41.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4781ffb640 1 -- 172.21.15.81:0/1600704376 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788017070 con 0x7f478c0b9040 2024-06-29T14:41:41.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f47939a1640 1 -- 172.21.15.81:0/1600704376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f478c0b0950 con 0x7f478c0b9040 2024-06-29T14:41:41.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4781ffb640 1 -- 172.21.15.81:0/1600704376 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47880035f0 con 0x7f478c0b9040 2024-06-29T14:41:41.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.419+0000 7f4781ffb640 1 -- 172.21.15.81:0/1600704376 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788003a40 con 0x7f478c0b9040 2024-06-29T14:41:41.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.420+0000 7f47939a1640 1 -- 172.21.15.81:0/1600704376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f478c0b4880 con 0x7f478c0b9040 2024-06-29T14:41:41.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.421+0000 7f47939a1640 1 -- 172.21.15.81:0/1600704376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f478c0b83b0 con 0x7f478c0b9040 2024-06-29T14:41:41.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.421+0000 7f4781ffb640 1 -- 172.21.15.81:0/1600704376 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f478801a070 con 0x7f478c0b9040 2024-06-29T14:41:41.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.422+0000 7f4781ffb640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4770076480 0x7f4770078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.422+0000 7f479173d640 1 -- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4770076480 msgr2=0x7f4770078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:41.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.422+0000 7f4781ffb640 1 -- 172.21.15.81:0/1600704376 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f4788096e90 con 0x7f478c0b9040 2024-06-29T14:41:41.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.422+0000 7f479173d640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4770076480 0x7f4770078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:41.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.426+0000 7f4781ffb640 1 -- 172.21.15.81:0/1600704376 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f478805fd90 con 0x7f478c0b9040 2024-06-29T14:41:41.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.623+0000 7f479173d640 1 -- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4770076480 msgr2=0x7f4770078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:41.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.623+0000 7f479173d640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4770076480 0x7f4770078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:41.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.627+0000 7f47939a1640 1 -- 172.21.15.81:0/1600704376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f478c068860 con 0x7f478c0b9040 2024-06-29T14:41:41.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.628+0000 7f4781ffb640 1 -- 172.21.15.81:0/1600704376 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) ==== 104+0+11 (secure 0 0 0) 0x7f4788064090 con 0x7f478c0b9040 2024-06-29T14:41:41.626 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 -- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4770076480 msgr2=0x7f4770078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4770076480 0x7f4770078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 -- 172.21.15.81:0/1600704376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f478c0b9040 msgr2=0x7f478c0b26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f478c0b9040 0x7f478c0b26f0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f478800c340 tx=0x7f478800c800 comp rx=0 tx=0).stop 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 -- 172.21.15.81:0/1600704376 shutdown_connections 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4770076480 0x7f4770078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f478c0d2a90 0x7f478c0b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f478c0b9040 0x7f478c0b26f0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 --2- 172.21.15.81:0/1600704376 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f478c0b5250 0x7f478c15e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 -- 172.21.15.81:0/1600704376 >> 172.21.15.81:0/1600704376 conn(0x7f478c09cf30 msgr2=0x7f478c1612d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 -- 172.21.15.81:0/1600704376 shutdown_connections 2024-06-29T14:41:41.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.630+0000 7f47939a1640 1 -- 172.21.15.81:0/1600704376 wait complete. 2024-06-29T14:41:41.640 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:41:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.759+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/2563852442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8415bc20 msgr2=0x7f4b8406ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.759+0000 7f4b8b7ef640 1 --2- 172.21.15.81:0/2563852442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8415bc20 0x7f4b8406ed90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c005fd0 tx=0x7f4b7c02a820 comp rx=0 tx=0).stop 2024-06-29T14:41:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.759+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/2563852442 shutdown_connections 2024-06-29T14:41:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.759+0000 7f4b8b7ef640 1 --2- 172.21.15.81:0/2563852442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8415bc20 0x7f4b8406ed90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.759+0000 7f4b8b7ef640 1 --2- 172.21.15.81:0/2563852442 >> v2:172.21.15.167:3300/0 conn(0x7f4b84073650 0x7f4b8415b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.759+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/2563852442 >> 172.21.15.81:0/2563852442 conn(0x7f4b840b2920 msgr2=0x7f4b840b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.759+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/2563852442 shutdown_connections 2024-06-29T14:41:41.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.759+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/2563852442 wait complete. 2024-06-29T14:41:41.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.761+0000 7f4b8b7ef640 1 Processor -- start 2024-06-29T14:41:41.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.761+0000 7f4b8b7ef640 1 -- start start 2024-06-29T14:41:41.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.762+0000 7f4b8b7ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b840cd150 0x7f4b8414b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.762+0000 7f4b8b7ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b8415bc20 0x7f4b8415c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.762+0000 7f4b8b7ef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b8415ca40 0x7f4b840af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.762+0000 7f4b8b7ef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8406f660 con 0x7f4b8415bc20 2024-06-29T14:41:41.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.762+0000 7f4b8b7ef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8406f4e0 con 0x7f4b8415ca40 2024-06-29T14:41:41.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.762+0000 7f4b8b7ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4b8406f7e0 con 0x7f4b840cd150 2024-06-29T14:41:41.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b88d8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b8415bc20 0x7f4b8415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b88d8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b8415bc20 0x7f4b8415c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51486/0 (socket says 172.21.15.81:51486) 2024-06-29T14:41:41.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b88d8a640 1 -- 172.21.15.81:0/4122005170 learned_addr learned my addr 172.21.15.81:0/4122005170 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:41.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b8958b640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b840cd150 0x7f4b8414b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b89d8c640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b8415ca40 0x7f4b840af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b88d8a640 1 -- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b840cd150 msgr2=0x7f4b8414b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b88d8a640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b840cd150 0x7f4b8414b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b88d8a640 1 -- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b8415ca40 msgr2=0x7f4b840af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b88d8a640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b8415ca40 0x7f4b840af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b88d8a640 1 -- 172.21.15.81:0/4122005170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8415ce50 con 0x7f4b8415bc20 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b88d8a640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b8415bc20 0x7f4b8415c500 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000e9f0 tx=0x7f4b8000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b75ffb640 1 -- 172.21.15.81:0/4122005170 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8000cd20 con 0x7f4b8415bc20 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b75ffb640 1 -- 172.21.15.81:0/4122005170 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b80004020 con 0x7f4b8415bc20 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/4122005170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b840adbb0 con 0x7f4b8415bc20 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b75ffb640 1 -- 172.21.15.81:0/4122005170 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80004450 con 0x7f4b8415bc20 2024-06-29T14:41:41.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.763+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/4122005170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b840ade90 con 0x7f4b8415bc20 2024-06-29T14:41:41.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.765+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/4122005170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b8406fd00 con 0x7f4b8415bc20 2024-06-29T14:41:41.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.766+0000 7f4b75ffb640 1 -- 172.21.15.81:0/4122005170 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f4b800035f0 con 0x7f4b8415bc20 2024-06-29T14:41:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.766+0000 7f4b75ffb640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4b680764b0 0x7f4b680789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.766+0000 7f4b75ffb640 1 -- 172.21.15.81:0/4122005170 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f4b800984a0 con 0x7f4b8415bc20 2024-06-29T14:41:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.767+0000 7f4b8958b640 1 -- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4b680764b0 msgr2=0x7f4b680789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:41.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.767+0000 7f4b8958b640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4b680764b0 0x7f4b680789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:41.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.771+0000 7f4b75ffb640 1 -- 172.21.15.81:0/4122005170 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4b8415ce50 con 0x7f4b8415bc20 2024-06-29T14:41:41.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.957+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/4122005170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4b840b2e60 con 0x7f4b8415bc20 2024-06-29T14:41:41.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.958+0000 7f4b75ffb640 1 -- 172.21.15.81:0/4122005170 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v134) ==== 100+0+11 (secure 0 0 0) 0x7f4b80061260 con 0x7f4b8415bc20 2024-06-29T14:41:41.956 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4b680764b0 msgr2=0x7f4b680789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4b680764b0 0x7f4b680789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/4122005170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b8415bc20 msgr2=0x7f4b8415c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b8415bc20 0x7f4b8415c500 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000e9f0 tx=0x7f4b8000eeb0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/4122005170 shutdown_connections 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f4b680764b0 0x7f4b680789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b8415ca40 0x7f4b840af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b8415bc20 0x7f4b8415c500 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 --2- 172.21.15.81:0/4122005170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b840cd150 0x7f4b8414b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/4122005170 >> 172.21.15.81:0/4122005170 conn(0x7f4b840b2920 msgr2=0x7f4b84152750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/4122005170 shutdown_connections 2024-06-29T14:41:41.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:41.960+0000 7f4b8b7ef640 1 -- 172.21.15.81:0/4122005170 wait complete. 2024-06-29T14:41:41.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T14:41:42.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.089+0000 7f32e7046640 1 -- 172.21.15.81:0/2985950423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32e00b5250 msgr2=0x7f32e00d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.089+0000 7f32e7046640 1 --2- 172.21.15.81:0/2985950423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32e00b5250 0x7f32e00d09c0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009830 tx=0x7f32cc02a5a0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.089+0000 7f32e7046640 1 -- 172.21.15.81:0/2985950423 shutdown_connections 2024-06-29T14:41:42.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.089+0000 7f32e7046640 1 --2- 172.21.15.81:0/2985950423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32e00b5250 0x7f32e00d09c0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.089+0000 7f32e7046640 1 --2- 172.21.15.81:0/2985950423 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32e00b9040 0x7f32e00d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.089+0000 7f32e7046640 1 -- 172.21.15.81:0/2985950423 >> 172.21.15.81:0/2985950423 conn(0x7f32e009cf30 msgr2=0x7f32e00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:42.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.089+0000 7f32e7046640 1 -- 172.21.15.81:0/2985950423 shutdown_connections 2024-06-29T14:41:42.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.089+0000 7f32e7046640 1 -- 172.21.15.81:0/2985950423 wait complete. 2024-06-29T14:41:42.089 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:41:42.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.091+0000 7f32e7046640 1 Processor -- start 2024-06-29T14:41:42.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.091+0000 7f32e7046640 1 -- start start 2024-06-29T14:41:42.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.092+0000 7f32e7046640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32e00b5250 0x7f32e0159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.092+0000 7f32e7046640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32e00b7290 0x7f32e00a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.092+0000 7f32e7046640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32e00b9040 0x7f32e009ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.092+0000 7f32e7046640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f32e00b7db0 con 0x7f32e00b9040 2024-06-29T14:41:42.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.092+0000 7f32e7046640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f32e00d2d50 con 0x7f32e00b5250 2024-06-29T14:41:42.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.092+0000 7f32e7046640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f32e00b7f30 con 0x7f32e00b7290 2024-06-29T14:41:42.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e55e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32e00b9040 0x7f32e009ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e55e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32e00b9040 0x7f32e009ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51500/0 (socket says 172.21.15.81:51500) 2024-06-29T14:41:42.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e55e3640 1 -- 172.21.15.81:0/2400139849 learned_addr learned my addr 172.21.15.81:0/2400139849 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:42.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32dffff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32e00b7290 0x7f32e00a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e4de2640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32e00b5250 0x7f32e0159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:41:42.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e55e3640 1 -- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32e00b7290 msgr2=0x7f32e00a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e55e3640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32e00b7290 0x7f32e00a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e55e3640 1 -- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32e00b5250 msgr2=0x7f32e0159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e55e3640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32e00b5250 0x7f32e0159f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e55e3640 1 -- 172.21.15.81:0/2400139849 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e00a04a0 con 0x7f32e00b9040 2024-06-29T14:41:42.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e4de2640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32e00b5250 0x7f32e0159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32e55e3640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32e00b9040 0x7f32e009ef70 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f32d800ca50 tx=0x7f32d800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32dd7fa640 1 -- 172.21.15.81:0/2400139849 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8016020 con 0x7f32e00b9040 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.093+0000 7f32dd7fa640 1 -- 172.21.15.81:0/2400139849 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32d800aca0 con 0x7f32e00b9040 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.094+0000 7f32e7046640 1 -- 172.21.15.81:0/2400139849 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e00a0780 con 0x7f32e00b9040 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.094+0000 7f32dd7fa640 1 -- 172.21.15.81:0/2400139849 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8012680 con 0x7f32e00b9040 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.094+0000 7f32e7046640 1 -- 172.21.15.81:0/2400139849 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e00b3ac0 con 0x7f32e00b9040 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.095+0000 7f32e7046640 1 -- 172.21.15.81:0/2400139849 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e00b3d20 con 0x7f32e00b9040 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.095+0000 7f32dd7fa640 1 -- 172.21.15.81:0/2400139849 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f32d8019020 con 0x7f32e00b9040 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.096+0000 7f32dd7fa640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f32c40764b0 0x7f32c40789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.096+0000 7f32e4de2640 1 -- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f32c40764b0 msgr2=0x7f32c40789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.096+0000 7f32dd7fa640 1 -- 172.21.15.81:0/2400139849 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f32d8097e90 con 0x7f32e00b9040 2024-06-29T14:41:42.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.096+0000 7f32e4de2640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f32c40764b0 0x7f32c40789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:42.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.101+0000 7f32dd7fa640 1 -- 172.21.15.81:0/2400139849 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f32d8020030 con 0x7f32e00b9040 2024-06-29T14:41:42.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.297+0000 7f32e4de2640 1 -- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f32c40764b0 msgr2=0x7f32c40789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:42.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.297+0000 7f32e4de2640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f32c40764b0 0x7f32c40789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:42.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.300+0000 7f32e7046640 1 -- 172.21.15.81:0/2400139849 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f32e00b84c0 con 0x7f32e00b9040 2024-06-29T14:41:42.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.300+0000 7f32dd7fa640 1 -- 172.21.15.81:0/2400139849 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v134) ==== 80+0+475 (secure 0 0 0) 0x7f32d8060cb0 con 0x7f32e00b9040 2024-06-29T14:41:42.298 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1304833179 2024-06-30T14:41:33.386103+0000 2024-06-29T14:41:42.299 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1979367025 2024-06-30T14:41:33.386103+0000 2024-06-29T14:41:42.299 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/639087269 2024-06-30T14:41:33.386103+0000 2024-06-29T14:41:42.299 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1135028328 2024-06-30T14:41:33.386103+0000 2024-06-29T14:41:42.299 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1876032601 2024-06-30T14:41:33.386103+0000 2024-06-29T14:41:42.299 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2923077924 2024-06-30T14:41:33.386103+0000 2024-06-29T14:41:42.299 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/971640072 2024-06-30T14:41:33.386103+0000 2024-06-29T14:41:42.299 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/971640072 2024-06-30T14:41:33.386103+0000 2024-06-29T14:41:42.299 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T14:41:42.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 -- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f32c40764b0 msgr2=0x7f32c40789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:42.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f32c40764b0 0x7f32c40789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 -- 172.21.15.81:0/2400139849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32e00b9040 msgr2=0x7f32e009ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32e00b9040 0x7f32e009ef70 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f32d800ca50 tx=0x7f32d800cf10 comp rx=0 tx=0).stop 2024-06-29T14:41:42.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 -- 172.21.15.81:0/2400139849 shutdown_connections 2024-06-29T14:41:42.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f32c40764b0 0x7f32c40789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32e00b9040 0x7f32e009ef70 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32e00b7290 0x7f32e00a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 --2- 172.21.15.81:0/2400139849 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32e00b5250 0x7f32e0159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 -- 172.21.15.81:0/2400139849 >> 172.21.15.81:0/2400139849 conn(0x7f32e009cf30 msgr2=0x7f32e00d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:42.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 -- 172.21.15.81:0/2400139849 shutdown_connections 2024-06-29T14:41:42.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.302+0000 7f32e7046640 1 -- 172.21.15.81:0/2400139849 wait complete. 2024-06-29T14:41:42.312 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T14:41:42.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:42.312 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T14:41:42.341 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T14:41:42.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:41:42.341 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T14:41:42.396 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:42.397 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:41:42.397 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:42.412 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:42.413 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:42.413 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:42.413 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:42.453 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:42.453 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T14:41:42.453 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:41:42.453 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:42.469 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:42.469 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:42.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.610+0000 7f5632ff0640 1 -- 172.21.15.81:0/1153089951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c004d40 msgr2=0x7f561c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.610+0000 7f5632ff0640 1 --2- 172.21.15.81:0/1153089951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c004d40 0x7f561c005190 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f562400c740 tx=0x7f561c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:42.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.610+0000 7f563158d640 1 -- 172.21.15.81:0/1153089951 reap_dead start 2024-06-29T14:41:42.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.610+0000 7f5632ff0640 1 -- 172.21.15.81:0/1153089951 shutdown_connections 2024-06-29T14:41:42.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.610+0000 7f5632ff0640 1 -- 172.21.15.81:0/1153089951 >> 172.21.15.81:0/1153089951 conn(0x7f562c0b3680 msgr2=0x7f562c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:42.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.610+0000 7f5632ff0640 1 -- 172.21.15.81:0/1153089951 shutdown_connections 2024-06-29T14:41:42.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.610+0000 7f5632ff0640 1 -- 172.21.15.81:0/1153089951 wait complete. 2024-06-29T14:41:42.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.612+0000 7f5632ff0640 1 Processor -- start 2024-06-29T14:41:42.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.612+0000 7f5632ff0640 1 -- start start 2024-06-29T14:41:42.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5632ff0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c004d40 0x7f562c0b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5632ff0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f562c06d400 0x7f562c1f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5632ff0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f562c0ce5a0 0x7f562c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5632ff0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f562c0d09d0 con 0x7f561c004d40 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5632ff0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f562c0d0850 con 0x7f562c0ce5a0 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5632ff0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f562c0b5a60 con 0x7f562c06d400 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5630d8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c004d40 0x7f562c0b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5630d8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c004d40 0x7f562c0b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51516/0 (socket says 172.21.15.81:51516) 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5630d8c640 1 -- 172.21.15.81:0/3031340994 learned_addr learned my addr 172.21.15.81:0/3031340994 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f562bfff640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f562c06d400 0x7f562c1f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f563158d640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f562c0ce5a0 0x7f562c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5630d8c640 1 -- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f562c06d400 msgr2=0x7f562c1f4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5630d8c640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f562c06d400 0x7f562c1f4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5630d8c640 1 -- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f562c0ce5a0 msgr2=0x7f562c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5630d8c640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f562c0ce5a0 0x7f562c0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5630d8c640 1 -- 172.21.15.81:0/3031340994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562c0ceef0 con 0x7f561c004d40 2024-06-29T14:41:42.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.614+0000 7f5630d8c640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c004d40 0x7f562c0b22d0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f5620003e60 tx=0x7f5620001730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.615+0000 7f56297fa640 1 -- 172.21.15.81:0/3031340994 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5620011430 con 0x7f561c004d40 2024-06-29T14:41:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.615+0000 7f5632ff0640 1 -- 172.21.15.81:0/3031340994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f562c06dec0 con 0x7f561c004d40 2024-06-29T14:41:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.615+0000 7f56297fa640 1 -- 172.21.15.81:0/3031340994 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5620016020 con 0x7f561c004d40 2024-06-29T14:41:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.615+0000 7f56297fa640 1 -- 172.21.15.81:0/3031340994 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56200118b0 con 0x7f561c004d40 2024-06-29T14:41:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.615+0000 7f5632ff0640 1 -- 172.21.15.81:0/3031340994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f562c09bb30 con 0x7f561c004d40 2024-06-29T14:41:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.616+0000 7f5632ff0640 1 -- 172.21.15.81:0/3031340994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f562c0b5f70 con 0x7f561c004d40 2024-06-29T14:41:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.617+0000 7f56297fa640 1 -- 172.21.15.81:0/3031340994 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f5620011a50 con 0x7f561c004d40 2024-06-29T14:41:42.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.618+0000 7f56297fa640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f56100764b0 0x7f56100789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.618+0000 7f562bfff640 1 -- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f56100764b0 msgr2=0x7f56100789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:42.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.618+0000 7f56297fa640 1 -- 172.21.15.81:0/3031340994 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f5620098430 con 0x7f561c004d40 2024-06-29T14:41:42.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.618+0000 7f562bfff640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f56100764b0 0x7f56100789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:42.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.622+0000 7f56297fa640 1 -- 172.21.15.81:0/3031340994 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5620020030 con 0x7f561c004d40 2024-06-29T14:41:42.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.819+0000 7f562bfff640 1 -- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f56100764b0 msgr2=0x7f56100789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:42.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.820+0000 7f562bfff640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f56100764b0 0x7f56100789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:41:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.850+0000 7f5632ff0640 1 -- 172.21.15.81:0/3031340994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f562c09bfd0 con 0x7f561c004d40 2024-06-29T14:41:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.850+0000 7f56297fa640 1 -- 172.21.15.81:0/3031340994 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5228 (secure 0 0 0) 0x7f562c0ceef0 con 0x7f561c004d40 2024-06-29T14:41:42.849 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:42.849 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":68,"btime":"2024-06-29T14:41:24:728024+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/4097483790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":4097483790},{"type":"v1","addr":"172.21.15.167:6837","nonce":4097483790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1957730707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1957730707},{"type":"v1","addr":"172.21.15.167:6836","nonce":1957730707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":8550,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3852254611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3852254611},{"type":"v1","addr":"172.21.15.81:6833","nonce":3852254611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:21.060976+0000","modified":"2024-06-29T14:41:24.093831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8553":{"gid":8553,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/942339953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":942339953},{"type":"v1","addr":"172.21.15.81:6835","nonce":942339953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8553,"qdb_cluster":[8553]},"id":8}]} 2024-06-29T14:41:42.849 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 68 2024-06-29T14:41:42.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 -- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f56100764b0 msgr2=0x7f56100789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:42.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f56100764b0 0x7f56100789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 -- 172.21.15.81:0/3031340994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c004d40 msgr2=0x7f562c0b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c004d40 0x7f562c0b22d0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f5620003e60 tx=0x7f5620001730 comp rx=0 tx=0).stop 2024-06-29T14:41:42.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 -- 172.21.15.81:0/3031340994 shutdown_connections 2024-06-29T14:41:42.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f56100764b0 0x7f56100789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f562c0ce5a0 0x7f562c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f562c06d400 0x7f562c1f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 --2- 172.21.15.81:0/3031340994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c004d40 0x7f562c0b22d0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 -- 172.21.15.81:0/3031340994 >> 172.21.15.81:0/3031340994 conn(0x7f562c0b3680 msgr2=0x7f562c1e8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:42.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.852+0000 7f5632ff0640 1 -- 172.21.15.81:0/3031340994 shutdown_connections 2024-06-29T14:41:42.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.853+0000 7f5632ff0640 1 -- 172.21.15.81:0/3031340994 wait complete. 2024-06-29T14:41:42.862 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:42.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.980+0000 7fa466e1f640 1 -- 172.21.15.81:0/1449479485 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4600cd150 msgr2=0x7fa460072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.980+0000 7fa466e1f640 1 --2- 172.21.15.81:0/1449479485 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4600cd150 0x7fa460072f40 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009830 tx=0x7fa45c02a480 comp rx=0 tx=0).stop 2024-06-29T14:41:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.980+0000 7fa466e1f640 1 -- 172.21.15.81:0/1449479485 shutdown_connections 2024-06-29T14:41:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.980+0000 7fa466e1f640 1 --2- 172.21.15.81:0/1449479485 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa460073480 0x7fa46015b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.980+0000 7fa466e1f640 1 --2- 172.21.15.81:0/1449479485 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4600cd150 0x7fa460072f40 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.980+0000 7fa466e1f640 1 -- 172.21.15.81:0/1449479485 >> 172.21.15.81:0/1449479485 conn(0x7fa4600b2920 msgr2=0x7fa4600b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.980+0000 7fa466e1f640 1 -- 172.21.15.81:0/1449479485 shutdown_connections 2024-06-29T14:41:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.980+0000 7fa466e1f640 1 -- 172.21.15.81:0/1449479485 wait complete. 2024-06-29T14:41:42.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.982+0000 7fa466e1f640 1 Processor -- start 2024-06-29T14:41:42.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.982+0000 7fa466e1f640 1 -- start start 2024-06-29T14:41:42.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa466e1f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa460073480 0x7fa460153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa466e1f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4600cd150 0x7fa4601ec810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa466e1f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa46015bc20 0x7fa4601ecd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa466e1f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa46015c1b0 con 0x7fa460073480 2024-06-29T14:41:42.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa466e1f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa46015c030 con 0x7fa46015bc20 2024-06-29T14:41:42.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa466e1f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa46015f060 con 0x7fa4600cd150 2024-06-29T14:41:42.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa464bbb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa460073480 0x7fa460153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa464bbb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa460073480 0x7fa460153930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51526/0 (socket says 172.21.15.81:51526) 2024-06-29T14:41:42.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa464bbb640 1 -- 172.21.15.81:0/3698960297 learned_addr learned my addr 172.21.15.81:0/3698960297 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa457fff640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4600cd150 0x7fa4601ec810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.983+0000 7fa4653bc640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa46015bc20 0x7fa4601ecd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa464bbb640 1 -- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4600cd150 msgr2=0x7fa4601ec810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa464bbb640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4600cd150 0x7fa4601ec810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa464bbb640 1 -- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa46015bc20 msgr2=0x7fa4601ecd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa464bbb640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa46015bc20 0x7fa4601ecd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa464bbb640 1 -- 172.21.15.81:0/3698960297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4601f2c30 con 0x7fa460073480 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa464bbb640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa460073480 0x7fa460153930 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fa45c02aa70 tx=0x7fa45c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa4557fa640 1 -- 172.21.15.81:0/3698960297 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c005c90 con 0x7fa460073480 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa4557fa640 1 -- 172.21.15.81:0/3698960297 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa45c031e50 con 0x7fa460073480 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa466e1f640 1 -- 172.21.15.81:0/3698960297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46014fac0 con 0x7fa460073480 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa4557fa640 1 -- 172.21.15.81:0/3698960297 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c005260 con 0x7fa460073480 2024-06-29T14:41:42.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.984+0000 7fa466e1f640 1 -- 172.21.15.81:0/3698960297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa460150080 con 0x7fa460073480 2024-06-29T14:41:42.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.985+0000 7fa466e1f640 1 -- 172.21.15.81:0/3698960297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa46015f580 con 0x7fa460073480 2024-06-29T14:41:42.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.986+0000 7fa4557fa640 1 -- 172.21.15.81:0/3698960297 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7fa45c005400 con 0x7fa460073480 2024-06-29T14:41:42.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.986+0000 7fa4557fa640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fa44c076480 0x7fa44c078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:42.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.986+0000 7fa457fff640 1 -- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fa44c076480 msgr2=0x7fa44c078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:42.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.987+0000 7fa4557fa640 1 -- 172.21.15.81:0/3698960297 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7fa45c0b8c80 con 0x7fa460073480 2024-06-29T14:41:42.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.987+0000 7fa457fff640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fa44c076480 0x7fa44c078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:42.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:42.991+0000 7fa4557fa640 1 -- 172.21.15.81:0/3698960297 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa45c03f030 con 0x7fa460073480 2024-06-29T14:41:43.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.179+0000 7fa466e1f640 1 -- 172.21.15.81:0/3698960297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4601516a0 con 0x7fa460073480 2024-06-29T14:41:43.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.180+0000 7fa4557fa640 1 -- 172.21.15.81:0/3698960297 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+17280 (secure 0 0 0) 0x7fa45c0a0390 con 0x7fa460073480 2024-06-29T14:41:43.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:43.182 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":134,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:33.386142+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.352751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:20.014553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/1304833179":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1979367025":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/639087269":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1135028328":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1876032601":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/2923077924":"2024-06-30T14:41:33.386103+0000","172.21.15.167:6833/971640072":"2024-06-30T14:41:33.386103+0000","172.21.15.167:6832/971640072":"2024-06-30T14:41:33.386103+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:43.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.182+0000 7fa466e1f640 1 -- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fa44c076480 msgr2=0x7fa44c078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:43.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fa44c076480 0x7fa44c078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 -- 172.21.15.81:0/3698960297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa460073480 msgr2=0x7fa460153930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa460073480 0x7fa460153930 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fa45c02aa70 tx=0x7fa45c033c20 comp rx=0 tx=0).stop 2024-06-29T14:41:43.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 -- 172.21.15.81:0/3698960297 shutdown_connections 2024-06-29T14:41:43.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fa44c076480 0x7fa44c078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa46015bc20 0x7fa4601ecd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4600cd150 0x7fa4601ec810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 --2- 172.21.15.81:0/3698960297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa460073480 0x7fa460153930 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 -- 172.21.15.81:0/3698960297 >> 172.21.15.81:0/3698960297 conn(0x7fa4600b2920 msgr2=0x7fa4600709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 -- 172.21.15.81:0/3698960297 shutdown_connections 2024-06-29T14:41:43.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.183+0000 7fa466e1f640 1 -- 172.21.15.81:0/3698960297 wait complete. 2024-06-29T14:41:43.192 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:41:19.352751+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:41:20.014553+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:43.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:43.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.313+0000 7f1cde19e640 1 -- 172.21.15.81:0/252233952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd80b6610 msgr2=0x7f1cd80b7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.313+0000 7f1cde19e640 1 --2- 172.21.15.81:0/252233952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd80b6610 0x7f1cd80b7190 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0005fd0 tx=0x7f1cd002a940 comp rx=0 tx=0).stop 2024-06-29T14:41:43.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.313+0000 7f1cde19e640 1 -- 172.21.15.81:0/252233952 shutdown_connections 2024-06-29T14:41:43.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.313+0000 7f1cde19e640 1 --2- 172.21.15.81:0/252233952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd80b6610 0x7f1cd80b7190 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.313+0000 7f1cde19e640 1 -- 172.21.15.81:0/252233952 >> 172.21.15.81:0/252233952 conn(0x7f1cd80b3680 msgr2=0x7f1cd80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.313+0000 7f1cde19e640 1 -- 172.21.15.81:0/252233952 shutdown_connections 2024-06-29T14:41:43.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.313+0000 7f1cde19e640 1 -- 172.21.15.81:0/252233952 wait complete. 2024-06-29T14:41:43.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.315+0000 7f1cde19e640 1 Processor -- start 2024-06-29T14:41:43.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.315+0000 7f1cde19e640 1 -- start start 2024-06-29T14:41:43.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.316+0000 7f1cde19e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd80b6610 0x7f1cd81f5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.316+0000 7f1cde19e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd80cfb50 0x7f1cd8158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.316+0000 7f1cde19e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd8070850 0x7f1cd8070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.316+0000 7f1cde19e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd80cd070 con 0x7f1cd80b6610 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.316+0000 7f1cde19e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd80ccef0 con 0x7f1cd80cfb50 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.316+0000 7f1cde19e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1cd80cd1f0 con 0x7f1cd8070850 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.316+0000 7f1cd77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd80b6610 0x7f1cd81f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.316+0000 7f1cd77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd80b6610 0x7f1cd81f5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51542/0 (socket says 172.21.15.81:51542) 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.316+0000 7f1cd77fe640 1 -- 172.21.15.81:0/4275665794 learned_addr learned my addr 172.21.15.81:0/4275665794 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cd6ffd640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd80cfb50 0x7f1cd8158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cd7fff640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd8070850 0x7f1cd8070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cd77fe640 1 -- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd8070850 msgr2=0x7f1cd8070c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cd77fe640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd8070850 0x7f1cd8070c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cd77fe640 1 -- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd80cfb50 msgr2=0x7f1cd8158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cd77fe640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd80cfb50 0x7f1cd8158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cd77fe640 1 -- 172.21.15.81:0/4275665794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd80711a0 con 0x7f1cd80b6610 2024-06-29T14:41:43.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cd77fe640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd80b6610 0x7f1cd81f5780 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00c370 tx=0x7f1ccc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:43.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cbffff640 1 -- 172.21.15.81:0/4275665794 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc017070 con 0x7f1cd80b6610 2024-06-29T14:41:43.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cbffff640 1 -- 172.21.15.81:0/4275665794 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ccc0035f0 con 0x7f1cd80b6610 2024-06-29T14:41:43.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cde19e640 1 -- 172.21.15.81:0/4275665794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd80c9e00 con 0x7f1cd80b6610 2024-06-29T14:41:43.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cbffff640 1 -- 172.21.15.81:0/4275665794 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc003a40 con 0x7f1cd80b6610 2024-06-29T14:41:43.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.317+0000 7f1cde19e640 1 -- 172.21.15.81:0/4275665794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd80ca3c0 con 0x7f1cd80b6610 2024-06-29T14:41:43.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.318+0000 7f1cde19e640 1 -- 172.21.15.81:0/4275665794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cd80cd710 con 0x7f1cd80b6610 2024-06-29T14:41:43.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.319+0000 7f1cbffff640 1 -- 172.21.15.81:0/4275665794 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f1ccc01a070 con 0x7f1cd80b6610 2024-06-29T14:41:43.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.320+0000 7f1cbffff640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1cc4076480 0x7f1cc4078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.320+0000 7f1cd6ffd640 1 -- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1cc4076480 msgr2=0x7f1cc4078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:43.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.320+0000 7f1cbffff640 1 -- 172.21.15.81:0/4275665794 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f1ccc096f20 con 0x7f1cd80b6610 2024-06-29T14:41:43.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.320+0000 7f1cd6ffd640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1cc4076480 0x7f1cc4078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:43.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.324+0000 7f1cbffff640 1 -- 172.21.15.81:0/4275665794 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1cd80711a0 con 0x7f1cd80b6610 2024-06-29T14:41:43.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.512+0000 7f1cde19e640 1 -- 172.21.15.81:0/4275665794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1cd80c5310 con 0x7f1cd80b6610 2024-06-29T14:41:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.512+0000 7f1cbffff640 1 -- 172.21.15.81:0/4275665794 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) ==== 93+0+10 (secure 0 0 0) 0x7f1ccc05fd90 con 0x7f1cd80b6610 2024-06-29T14:41:43.511 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:41:43.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 -- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1cc4076480 msgr2=0x7f1cc4078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:43.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1cc4076480 0x7f1cc4078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 -- 172.21.15.81:0/4275665794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd80b6610 msgr2=0x7f1cd81f5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd80b6610 0x7f1cd81f5780 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00c370 tx=0x7f1ccc00c830 comp rx=0 tx=0).stop 2024-06-29T14:41:43.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 -- 172.21.15.81:0/4275665794 shutdown_connections 2024-06-29T14:41:43.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1cc4076480 0x7f1cc4078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd8070850 0x7f1cd8070c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd80cfb50 0x7f1cd8158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 --2- 172.21.15.81:0/4275665794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd80b6610 0x7f1cd81f5780 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 -- 172.21.15.81:0/4275665794 >> 172.21.15.81:0/4275665794 conn(0x7f1cd80b3680 msgr2=0x7f1cd81e7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 -- 172.21.15.81:0/4275665794 shutdown_connections 2024-06-29T14:41:43.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.515+0000 7f1cde19e640 1 -- 172.21.15.81:0/4275665794 wait complete. 2024-06-29T14:41:43.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T14:41:43.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.644+0000 7feddd8ac640 1 -- 172.21.15.81:0/3840671996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedd815bdf0 msgr2=0x7fedd8165ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.644+0000 7feddd8ac640 1 --2- 172.21.15.81:0/3840671996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedd815bdf0 0x7fedd8165ab0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009830 tx=0x7fedc802a530 comp rx=0 tx=0).stop 2024-06-29T14:41:43.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.644+0000 7feddd8ac640 1 -- 172.21.15.81:0/3840671996 shutdown_connections 2024-06-29T14:41:43.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.644+0000 7feddd8ac640 1 --2- 172.21.15.81:0/3840671996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedd815bdf0 0x7fedd8165ab0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.644+0000 7feddd8ac640 1 -- 172.21.15.81:0/3840671996 >> 172.21.15.81:0/3840671996 conn(0x7fedd80b2920 msgr2=0x7fedd80b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.644+0000 7feddd8ac640 1 -- 172.21.15.81:0/3840671996 shutdown_connections 2024-06-29T14:41:43.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.644+0000 7feddd8ac640 1 -- 172.21.15.81:0/3840671996 wait complete. 2024-06-29T14:41:43.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.646+0000 7feddd8ac640 1 Processor -- start 2024-06-29T14:41:43.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.646+0000 7feddd8ac640 1 -- start start 2024-06-29T14:41:43.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.647+0000 7feddd8ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedd80cd150 0x7fedd8150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.647+0000 7feddd8ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fedd815bdf0 0x7fedd814ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.647+0000 7feddd8ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fedd814b520 0x7fedd8158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.647+0000 7feddd8ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8166170 con 0x7fedd80cd150 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.647+0000 7feddd8ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8165ff0 con 0x7fedd815bdf0 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.647+0000 7feddd8ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fedd81662f0 con 0x7fedd814b520 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedd80cd150 0x7fedd8150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedd80cd150 0x7fedd8150e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51566/0 (socket says 172.21.15.81:51566) 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd6ffd640 1 -- 172.21.15.81:0/4210356608 learned_addr learned my addr 172.21.15.81:0/4210356608 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd67fc640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fedd815bdf0 0x7fedd814ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd77fe640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fedd814b520 0x7fedd8158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd6ffd640 1 -- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fedd814b520 msgr2=0x7fedd8158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd6ffd640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fedd814b520 0x7fedd8158f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd6ffd640 1 -- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fedd815bdf0 msgr2=0x7fedd814ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd6ffd640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fedd815bdf0 0x7fedd814ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd6ffd640 1 -- 172.21.15.81:0/4210356608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd80b17d0 con 0x7fedd80cd150 2024-06-29T14:41:43.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedd6ffd640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedd80cd150 0x7fedd8150e20 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fedd000c730 tx=0x7fedd000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:43.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedcffff640 1 -- 172.21.15.81:0/4210356608 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0003b70 con 0x7fedd80cd150 2024-06-29T14:41:43.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedcffff640 1 -- 172.21.15.81:0/4210356608 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fedd0003d30 con 0x7fedd80cd150 2024-06-29T14:41:43.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7feddd8ac640 1 -- 172.21.15.81:0/4210356608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd8150340 con 0x7fedd80cd150 2024-06-29T14:41:43.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7fedcffff640 1 -- 172.21.15.81:0/4210356608 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd000aab0 con 0x7fedd80cd150 2024-06-29T14:41:43.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.648+0000 7feddd8ac640 1 -- 172.21.15.81:0/4210356608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd8150900 con 0x7fedd80cd150 2024-06-29T14:41:43.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.649+0000 7feddd8ac640 1 -- 172.21.15.81:0/4210356608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedd815b6e0 con 0x7fedd80cd150 2024-06-29T14:41:43.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.650+0000 7fedcffff640 1 -- 172.21.15.81:0/4210356608 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7fedd000ac50 con 0x7fedd80cd150 2024-06-29T14:41:43.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.651+0000 7fedcffff640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fedb80764b0 0x7fedb80789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.651+0000 7fedcffff640 1 -- 172.21.15.81:0/4210356608 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7fedd0097970 con 0x7fedd80cd150 2024-06-29T14:41:43.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.651+0000 7fedd67fc640 1 -- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fedb80764b0 msgr2=0x7fedb80789c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:43.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.651+0000 7fedd67fc640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fedb80764b0 0x7fedb80789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:43.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.655+0000 7fedcffff640 1 -- 172.21.15.81:0/4210356608 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fedd0060840 con 0x7fedd80cd150 2024-06-29T14:41:43.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.843+0000 7feddd8ac640 1 -- 172.21.15.81:0/4210356608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fedd8152630 con 0x7fedd80cd150 2024-06-29T14:41:43.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.843+0000 7fedcffff640 1 -- 172.21.15.81:0/4210356608 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) ==== 104+0+11 (secure 0 0 0) 0x7fedd0064b40 con 0x7fedd80cd150 2024-06-29T14:41:43.841 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T14:41:43.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 -- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fedb80764b0 msgr2=0x7fedb80789c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:43.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fedb80764b0 0x7fedb80789c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 -- 172.21.15.81:0/4210356608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedd80cd150 msgr2=0x7fedd8150e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedd80cd150 0x7fedd8150e20 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fedd000c730 tx=0x7fedd000cbf0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 -- 172.21.15.81:0/4210356608 shutdown_connections 2024-06-29T14:41:43.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7fedb80764b0 0x7fedb80789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fedd814b520 0x7fedd8158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fedd815bdf0 0x7fedd814ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 --2- 172.21.15.81:0/4210356608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedd80cd150 0x7fedd8150e20 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 -- 172.21.15.81:0/4210356608 >> 172.21.15.81:0/4210356608 conn(0x7fedd80b2920 msgr2=0x7fedd815c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 -- 172.21.15.81:0/4210356608 shutdown_connections 2024-06-29T14:41:43.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.846+0000 7feddd8ac640 1 -- 172.21.15.81:0/4210356608 wait complete. 2024-06-29T14:41:43.855 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T14:41:43.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.975+0000 7f1e620e4640 1 -- 172.21.15.81:0/1532784414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e48005150 msgr2=0x7f1e4800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.975+0000 7f1e620e4640 1 --2- 172.21.15.81:0/1532784414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e48005150 0x7f1e4800b9c0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009fd0 tx=0x7f1e48005830 comp rx=0 tx=0).stop 2024-06-29T14:41:43.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.975+0000 7f1e5bfff640 1 -- 172.21.15.81:0/1532784414 reap_dead start 2024-06-29T14:41:43.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.975+0000 7f1e620e4640 1 -- 172.21.15.81:0/1532784414 shutdown_connections 2024-06-29T14:41:43.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.975+0000 7f1e620e4640 1 -- 172.21.15.81:0/1532784414 >> 172.21.15.81:0/1532784414 conn(0x7f1e5c0b3680 msgr2=0x7f1e5c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:43.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.975+0000 7f1e620e4640 1 -- 172.21.15.81:0/1532784414 shutdown_connections 2024-06-29T14:41:43.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.975+0000 7f1e620e4640 1 -- 172.21.15.81:0/1532784414 wait complete. 2024-06-29T14:41:43.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.977+0000 7f1e620e4640 1 Processor -- start 2024-06-29T14:41:43.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.977+0000 7f1e620e4640 1 -- start start 2024-06-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.978+0000 7f1e620e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e48005150 0x7f1e5c156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.978+0000 7f1e620e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e5c0c8bc0 0x7f1e5c06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.978+0000 7f1e620e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e5c06d940 0x7f1e5c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.978+0000 7f1e620e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c0d09d0 con 0x7f1e5c0c8bc0 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.978+0000 7f1e620e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c0d0850 con 0x7f1e48005150 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.978+0000 7f1e620e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1e5c0b5890 con 0x7f1e5c06d940 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.978+0000 7f1e53fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e5c0c8bc0 0x7f1e5c06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e53fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e5c0c8bc0 0x7f1e5c06d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51586/0 (socket says 172.21.15.81:51586) 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e53fff640 1 -- 172.21.15.81:0/2567418277 learned_addr learned my addr 172.21.15.81:0/2567418277 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e5bfff640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e5c06d940 0x7f1e5c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e5b7fe640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e48005150 0x7f1e5c156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e53fff640 1 -- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e5c06d940 msgr2=0x7f1e5c06dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e53fff640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e5c06d940 0x7f1e5c06dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e53fff640 1 -- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e48005150 msgr2=0x7f1e5c156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:43.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e53fff640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e48005150 0x7f1e5c156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:43.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e53fff640 1 -- 172.21.15.81:0/2567418277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5c0c4db0 con 0x7f1e5c0c8bc0 2024-06-29T14:41:43.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e53fff640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e5c0c8bc0 0x7f1e5c06d400 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f1e44011ab0 tx=0x7f1e44011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:43.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e58ff9640 1 -- 172.21.15.81:0/2567418277 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4401e070 con 0x7f1e5c0c8bc0 2024-06-29T14:41:43.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e58ff9640 1 -- 172.21.15.81:0/2567418277 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e4400fe20 con 0x7f1e5c0c8bc0 2024-06-29T14:41:43.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e58ff9640 1 -- 172.21.15.81:0/2567418277 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e44019650 con 0x7f1e5c0c8bc0 2024-06-29T14:41:43.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e620e4640 1 -- 172.21.15.81:0/2567418277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5c1e20b0 con 0x7f1e5c0c8bc0 2024-06-29T14:41:43.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.979+0000 7f1e620e4640 1 -- 172.21.15.81:0/2567418277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5c1e2670 con 0x7f1e5c0c8bc0 2024-06-29T14:41:43.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.980+0000 7f1e620e4640 1 -- 172.21.15.81:0/2567418277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e5c0b5350 con 0x7f1e5c0c8bc0 2024-06-29T14:41:43.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.981+0000 7f1e58ff9640 1 -- 172.21.15.81:0/2567418277 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 74) ==== 98360+0+0 (secure 0 0 0) 0x7f1e44021070 con 0x7f1e5c0c8bc0 2024-06-29T14:41:43.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.982+0000 7f1e58ff9640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1e3c076480 0x7f1e3c078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:43.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.982+0000 7f1e5b7fe640 1 -- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1e3c076480 msgr2=0x7f1e3c078990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2082979226 2024-06-29T14:41:43.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.982+0000 7f1e58ff9640 1 -- 172.21.15.81:0/2567418277 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 6153+0+0 (secure 0 0 0) 0x7f1e44098c20 con 0x7f1e5c0c8bc0 2024-06-29T14:41:43.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.982+0000 7f1e5b7fe640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1e3c076480 0x7f1e3c078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:41:43.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:43.985+0000 7f1e58ff9640 1 -- 172.21.15.81:0/2567418277 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1e44028070 con 0x7f1e5c0c8bc0 2024-06-29T14:41:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.053+0000 7f1e58ff9640 1 -- 172.21.15.81:0/2567418277 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 75) ==== 97912+0+0 (secure 0 0 0) 0x7f1e4405dc90 con 0x7f1e5c0c8bc0 2024-06-29T14:41:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.053+0000 7f1e58ff9640 1 -- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1e3c076480 msgr2=0x7f1e3c078990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:41:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.053+0000 7f1e58ff9640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1e3c076480 0x7f1e3c078990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.102 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:44.105+0000 7fb16c697640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.102 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:44.105+0000 7fb16c697640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:44.105+0000 7fb16c697640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:44.105+0000 7fb16c697640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:44.105+0000 7fb16c697640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:44.105+0000 7fb169691640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:44.105+0000 7fb169691640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:44.105+0000 7fb169691640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:44.105+0000 7fb169691640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:41:44.105+0000 7fb169691640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:41:44.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.183+0000 7f1e620e4640 1 -- 172.21.15.81:0/2567418277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1e5c0c4ad0 con 0x7f1e5c0c8bc0 2024-06-29T14:41:44.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.183+0000 7f1e58ff9640 1 -- 172.21.15.81:0/2567418277 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v135) ==== 100+0+11 (secure 0 0 0) 0x7f1e440432d0 con 0x7f1e5c0c8bc0 2024-06-29T14:41:44.181 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T14:41:44.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.185+0000 7f1e620e4640 1 -- 172.21.15.81:0/2567418277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e5c0c8bc0 msgr2=0x7f1e5c06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.185+0000 7f1e620e4640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e5c0c8bc0 0x7f1e5c06d400 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f1e44011ab0 tx=0x7f1e44011f70 comp rx=0 tx=0).stop 2024-06-29T14:41:44.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.185+0000 7f1e620e4640 1 -- 172.21.15.81:0/2567418277 shutdown_connections 2024-06-29T14:41:44.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.185+0000 7f1e620e4640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:6832/2082979226,v1:172.21.15.167:6833/2082979226] conn(0x7f1e3c076480 0x7f1e3c078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.185+0000 7f1e620e4640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e5c06d940 0x7f1e5c06dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.185+0000 7f1e620e4640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e5c0c8bc0 0x7f1e5c06d400 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.185+0000 7f1e620e4640 1 --2- 172.21.15.81:0/2567418277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e48005150 0x7f1e5c156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.185+0000 7f1e620e4640 1 -- 172.21.15.81:0/2567418277 >> 172.21.15.81:0/2567418277 conn(0x7f1e5c0b3680 msgr2=0x7f1e5c070d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.185+0000 7f1e620e4640 1 -- 172.21.15.81:0/2567418277 shutdown_connections 2024-06-29T14:41:44.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.185+0000 7f1e620e4640 1 -- 172.21.15.81:0/2567418277 wait complete. 2024-06-29T14:41:44.194 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:44.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.311+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1265350602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d00073030 msgr2=0x7f5d00073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.311+0000 7f5d07ee4640 1 --2- 172.21.15.81:0/1265350602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d00073030 0x7f5d00073440 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009860 tx=0x7f5cf002a530 comp rx=0 tx=0).stop 2024-06-29T14:41:44.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.311+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1265350602 shutdown_connections 2024-06-29T14:41:44.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.311+0000 7f5d07ee4640 1 --2- 172.21.15.81:0/1265350602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d00073030 0x7f5d00073440 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.311+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1265350602 >> 172.21.15.81:0/1265350602 conn(0x7f5d0006ccc0 msgr2=0x7f5d000b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.311+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1265350602 shutdown_connections 2024-06-29T14:41:44.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.311+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1265350602 wait complete. 2024-06-29T14:41:44.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.313+0000 7f5d07ee4640 1 Processor -- start 2024-06-29T14:41:44.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.313+0000 7f5d07ee4640 1 -- start start 2024-06-29T14:41:44.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.314+0000 7f5d07ee4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d00073030 0x7f5d0013f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.314+0000 7f5d07ee4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d001412f0 0x7f5d00071f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.314+0000 7f5d07ee4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d0015e960 0x7f5d00072460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.314+0000 7f5d07ee4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5d000b35a0 con 0x7f5d00073030 2024-06-29T14:41:44.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.314+0000 7f5d07ee4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5d000b3420 con 0x7f5d001412f0 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.314+0000 7f5d07ee4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5d000b3720 con 0x7f5d0015e960 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d05c80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d00073030 0x7f5d0013f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d05c80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d00073030 0x7f5d0013f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51602/0 (socket says 172.21.15.81:51602) 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d05c80640 1 -- 172.21.15.81:0/1675589841 learned_addr learned my addr 172.21.15.81:0/1675589841 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d0547f640 1 --2- 172.21.15.81:0/1675589841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d001412f0 0x7f5d00071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d06481640 1 --2- 172.21.15.81:0/1675589841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d0015e960 0x7f5d00072460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d05c80640 1 -- 172.21.15.81:0/1675589841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d0015e960 msgr2=0x7f5d00072460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d05c80640 1 --2- 172.21.15.81:0/1675589841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d0015e960 0x7f5d00072460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d05c80640 1 -- 172.21.15.81:0/1675589841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d001412f0 msgr2=0x7f5d00071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d05c80640 1 --2- 172.21.15.81:0/1675589841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d001412f0 0x7f5d00071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d05c80640 1 -- 172.21.15.81:0/1675589841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d000a9560 con 0x7f5d00073030 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d05c80640 1 --2- 172.21.15.81:0/1675589841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d00073030 0x7f5d0013f690 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c730 tx=0x7f5cfc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:44.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5cf67fc640 1 -- 172.21.15.81:0/1675589841 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc017070 con 0x7f5d00073030 2024-06-29T14:41:44.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5cf67fc640 1 -- 172.21.15.81:0/1675589841 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cfc005360 con 0x7f5d00073030 2024-06-29T14:41:44.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1675589841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d000a9840 con 0x7f5d00073030 2024-06-29T14:41:44.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5cf67fc640 1 -- 172.21.15.81:0/1675589841 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc00a8a0 con 0x7f5d00073030 2024-06-29T14:41:44.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.315+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1675589841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d000ac970 con 0x7f5d00073030 2024-06-29T14:41:44.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.316+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1675589841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d00073440 con 0x7f5d00073030 2024-06-29T14:41:44.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.317+0000 7f5cf67fc640 1 -- 172.21.15.81:0/1675589841 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 75) ==== 97912+0+0 (secure 0 0 0) 0x7f5cfc01a070 con 0x7f5d00073030 2024-06-29T14:41:44.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.317+0000 7f5cf67fc640 1 -- 172.21.15.81:0/1675589841 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6540+0+0 (secure 0 0 0) 0x7f5cfc096e60 con 0x7f5d00073030 2024-06-29T14:41:44.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.322+0000 7f5cf67fc640 1 -- 172.21.15.81:0/1675589841 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d00073980 con 0x7f5d00073030 2024-06-29T14:41:44.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.540+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1675589841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d0015aee0 con 0x7f5d00073030 2024-06-29T14:41:44.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.541+0000 7f5cf67fc640 1 -- 172.21.15.81:0/1675589841 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5228 (secure 0 0 0) 0x7f5cfc060170 con 0x7f5d00073030 2024-06-29T14:41:44.539 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:44.539 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":68,"btime":"2024-06-29T14:41:24:728024+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/4097483790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":4097483790},{"type":"v1","addr":"172.21.15.167:6837","nonce":4097483790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1957730707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1957730707},{"type":"v1","addr":"172.21.15.167:6836","nonce":1957730707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":8550,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3852254611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3852254611},{"type":"v1","addr":"172.21.15.81:6833","nonce":3852254611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:21.060976+0000","modified":"2024-06-29T14:41:24.093831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8553":{"gid":8553,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/942339953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":942339953},{"type":"v1","addr":"172.21.15.81:6835","nonce":942339953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8553,"qdb_cluster":[8553]},"id":8}]} 2024-06-29T14:41:44.539 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 68 2024-06-29T14:41:44.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.543+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1675589841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d00073030 msgr2=0x7f5d0013f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.543+0000 7f5d07ee4640 1 --2- 172.21.15.81:0/1675589841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d00073030 0x7f5d0013f690 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c730 tx=0x7f5cfc00cbf0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.543+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1675589841 shutdown_connections 2024-06-29T14:41:44.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.543+0000 7f5d07ee4640 1 --2- 172.21.15.81:0/1675589841 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d0015e960 0x7f5d00072460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.543+0000 7f5d07ee4640 1 --2- 172.21.15.81:0/1675589841 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d001412f0 0x7f5d00071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.543+0000 7f5d07ee4640 1 --2- 172.21.15.81:0/1675589841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d00073030 0x7f5d0013f690 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.543+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1675589841 >> 172.21.15.81:0/1675589841 conn(0x7f5d0006ccc0 msgr2=0x7f5d000af5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.543+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1675589841 shutdown_connections 2024-06-29T14:41:44.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.543+0000 7f5d07ee4640 1 -- 172.21.15.81:0/1675589841 wait complete. 2024-06-29T14:41:44.551 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:44.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.667+0000 7fb11abeb640 1 -- 172.21.15.81:0/780879188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb11406f6a0 msgr2=0x7fb11415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.667+0000 7fb11abeb640 1 --2- 172.21.15.81:0/780879188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb11406f6a0 0x7fb11415aed0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fb108005fd0 tx=0x7fb10802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.667+0000 7fb11abeb640 1 -- 172.21.15.81:0/780879188 shutdown_connections 2024-06-29T14:41:44.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.667+0000 7fb11abeb640 1 --2- 172.21.15.81:0/780879188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb11406f6a0 0x7fb11415aed0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.667+0000 7fb11abeb640 1 -- 172.21.15.81:0/780879188 >> 172.21.15.81:0/780879188 conn(0x7fb1140d0930 msgr2=0x7fb1140d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.667+0000 7fb11abeb640 1 -- 172.21.15.81:0/780879188 shutdown_connections 2024-06-29T14:41:44.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.667+0000 7fb11abeb640 1 -- 172.21.15.81:0/780879188 wait complete. 2024-06-29T14:41:44.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.668+0000 7fb11abeb640 1 Processor -- start 2024-06-29T14:41:44.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.668+0000 7fb11abeb640 1 -- start start 2024-06-29T14:41:44.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb11abeb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb11406f6a0 0x7fb114152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb11abeb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1140b3960 0x7fb114156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb11abeb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1140b57b0 0x7fb114157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb11abeb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb1140b4600 con 0x7fb11406f6a0 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb11abeb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb1140b4480 con 0x7fb1140b57b0 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb11abeb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb1140b4780 con 0x7fb1140b3960 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb118987640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb11406f6a0 0x7fb114152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb118987640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb11406f6a0 0x7fb114152f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51620/0 (socket says 172.21.15.81:51620) 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb118987640 1 -- 172.21.15.81:0/332642359 learned_addr learned my addr 172.21.15.81:0/332642359 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb119188640 1 --2- 172.21.15.81:0/332642359 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1140b57b0 0x7fb114157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb113fff640 1 --2- 172.21.15.81:0/332642359 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1140b3960 0x7fb114156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb118987640 1 -- 172.21.15.81:0/332642359 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1140b3960 msgr2=0x7fb114156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb118987640 1 --2- 172.21.15.81:0/332642359 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1140b3960 0x7fb114156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb118987640 1 -- 172.21.15.81:0/332642359 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1140b57b0 msgr2=0x7fb114157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb118987640 1 --2- 172.21.15.81:0/332642359 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1140b57b0 0x7fb114157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb118987640 1 -- 172.21.15.81:0/332642359 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1141e75f0 con 0x7fb11406f6a0 2024-06-29T14:41:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb113fff640 1 --2- 172.21.15.81:0/332642359 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1140b3960 0x7fb114156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.670+0000 7fb118987640 1 --2- 172.21.15.81:0/332642359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb11406f6a0 0x7fb114152f00 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fb10400c3d0 tx=0x7fb10400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.671+0000 7fb1117fa640 1 -- 172.21.15.81:0/332642359 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb104017070 con 0x7fb11406f6a0 2024-06-29T14:41:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.671+0000 7fb1117fa640 1 -- 172.21.15.81:0/332642359 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1040053c0 con 0x7fb11406f6a0 2024-06-29T14:41:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.671+0000 7fb11abeb640 1 -- 172.21.15.81:0/332642359 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb11413f440 con 0x7fb11406f6a0 2024-06-29T14:41:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.671+0000 7fb1117fa640 1 -- 172.21.15.81:0/332642359 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb104003850 con 0x7fb11406f6a0 2024-06-29T14:41:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.671+0000 7fb11abeb640 1 -- 172.21.15.81:0/332642359 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb11413fa00 con 0x7fb11406f6a0 2024-06-29T14:41:44.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.672+0000 7fb11abeb640 1 -- 172.21.15.81:0/332642359 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb11406e2a0 con 0x7fb11406f6a0 2024-06-29T14:41:44.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.672+0000 7fb1117fa640 1 -- 172.21.15.81:0/332642359 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 75) ==== 97912+0+0 (secure 0 0 0) 0x7fb10401a070 con 0x7fb11406f6a0 2024-06-29T14:41:44.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.673+0000 7fb1117fa640 1 -- 172.21.15.81:0/332642359 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6540+0+0 (secure 0 0 0) 0x7fb104096900 con 0x7fb11406f6a0 2024-06-29T14:41:44.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.677+0000 7fb1117fa640 1 -- 172.21.15.81:0/332642359 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb11406e2a0 con 0x7fb11406f6a0 2024-06-29T14:41:44.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.861+0000 7fb11abeb640 1 -- 172.21.15.81:0/332642359 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb1140b4bf0 con 0x7fb11406f6a0 2024-06-29T14:41:44.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.862+0000 7fb1117fa640 1 -- 172.21.15.81:0/332642359 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+17852 (secure 0 0 0) 0x7fb104068360 con 0x7fb11406f6a0 2024-06-29T14:41:44.860 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:44.860 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":135,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:44.046259+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.352751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:20.014553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3620322784":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/3205425880":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/348977351":"2024-06-30T14:41:44.046221+0000","172.21.15.167:6833/2082979226":"2024-06-30T14:41:44.046221+0000","172.21.15.167:6832/2082979226":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/1183708751":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/1998218314":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/1304833179":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1979367025":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/639087269":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1135028328":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1876032601":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/2923077924":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/4051047679":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/2119756783":"2024-06-30T14:41:44.046221+0000","172.21.15.167:6833/971640072":"2024-06-30T14:41:33.386103+0000","172.21.15.167:6832/971640072":"2024-06-30T14:41:33.386103+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:44.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.864+0000 7fb11abeb640 1 -- 172.21.15.81:0/332642359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb11406f6a0 msgr2=0x7fb114152f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.864+0000 7fb11abeb640 1 --2- 172.21.15.81:0/332642359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb11406f6a0 0x7fb114152f00 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fb10400c3d0 tx=0x7fb10400c890 comp rx=0 tx=0).stop 2024-06-29T14:41:44.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.864+0000 7fb11abeb640 1 -- 172.21.15.81:0/332642359 shutdown_connections 2024-06-29T14:41:44.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.864+0000 7fb11abeb640 1 --2- 172.21.15.81:0/332642359 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1140b57b0 0x7fb114157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.864+0000 7fb11abeb640 1 --2- 172.21.15.81:0/332642359 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1140b3960 0x7fb114156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.864+0000 7fb11abeb640 1 --2- 172.21.15.81:0/332642359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb11406f6a0 0x7fb114152f00 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.864+0000 7fb11abeb640 1 -- 172.21.15.81:0/332642359 >> 172.21.15.81:0/332642359 conn(0x7fb1140d0930 msgr2=0x7fb114159500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.864+0000 7fb11abeb640 1 -- 172.21.15.81:0/332642359 shutdown_connections 2024-06-29T14:41:44.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.864+0000 7fb11abeb640 1 -- 172.21.15.81:0/332642359 wait complete. 2024-06-29T14:41:44.873 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T14:41:44.873 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:44.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.988+0000 7f55d39fc640 1 -- 172.21.15.81:0/230987896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55b8003a30 msgr2=0x7f55b8003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.988+0000 7f55d39fc640 1 --2- 172.21.15.81:0/230987896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55b8003a30 0x7f55b8003ea0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c3f0 tx=0x7f55b8005830 comp rx=0 tx=0).stop 2024-06-29T14:41:44.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.988+0000 7f55d1f99640 1 -- 172.21.15.81:0/230987896 reap_dead start 2024-06-29T14:41:44.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.988+0000 7f55d39fc640 1 -- 172.21.15.81:0/230987896 shutdown_connections 2024-06-29T14:41:44.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.988+0000 7f55d39fc640 1 -- 172.21.15.81:0/230987896 >> 172.21.15.81:0/230987896 conn(0x7f55cc06ccc0 msgr2=0x7f55cc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:44.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.988+0000 7f55d39fc640 1 -- 172.21.15.81:0/230987896 shutdown_connections 2024-06-29T14:41:44.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.988+0000 7f55d39fc640 1 -- 172.21.15.81:0/230987896 wait complete. 2024-06-29T14:41:44.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.990+0000 7f55d39fc640 1 Processor -- start 2024-06-29T14:41:44.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.990+0000 7f55d39fc640 1 -- start start 2024-06-29T14:41:44.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.991+0000 7f55d39fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55b8003a30 0x7f55cc0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.991+0000 7f55d39fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55cc15b010 0x7f55cc0b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.991+0000 7f55d39fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55cc0b3700 0x7f55cc13b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.991+0000 7f55d39fc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc15d0a0 con 0x7f55b8003a30 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.991+0000 7f55d39fc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc15cf20 con 0x7f55cc0b3700 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.991+0000 7f55d39fc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f55cc15d220 con 0x7f55cc15b010 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.991+0000 7f55d1798640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55b8003a30 0x7f55cc0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.991+0000 7f55d1798640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55b8003a30 0x7f55cc0aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51648/0 (socket says 172.21.15.81:51648) 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.991+0000 7f55d1798640 1 -- 172.21.15.81:0/1770874084 learned_addr learned my addr 172.21.15.81:0/1770874084 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d1f99640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55cc0b3700 0x7f55cc13b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d0f97640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55cc15b010 0x7f55cc0b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d1798640 1 -- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55cc15b010 msgr2=0x7f55cc0b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d1798640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55cc15b010 0x7f55cc0b31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d1798640 1 -- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55cc0b3700 msgr2=0x7f55cc13b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d1798640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55cc0b3700 0x7f55cc13b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d1798640 1 -- 172.21.15.81:0/1770874084 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55cc13d930 con 0x7f55b8003a30 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d1f99640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55cc0b3700 0x7f55cc13b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d0f97640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55cc15b010 0x7f55cc0b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d1798640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55b8003a30 0x7f55cc0aedc0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c510 tx=0x7f55c8006060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55c1ffb640 1 -- 172.21.15.81:0/1770874084 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8043070 con 0x7f55b8003a30 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55c1ffb640 1 -- 172.21.15.81:0/1770874084 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55c803a070 con 0x7f55b8003a30 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d39fc640 1 -- 172.21.15.81:0/1770874084 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55cc13dc10 con 0x7f55b8003a30 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55c1ffb640 1 -- 172.21.15.81:0/1770874084 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8003850 con 0x7f55b8003a30 2024-06-29T14:41:44.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.992+0000 7f55d39fc640 1 -- 172.21.15.81:0/1770874084 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55cc0ae8a0 con 0x7f55b8003a30 2024-06-29T14:41:44.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.994+0000 7f55d39fc640 1 -- 172.21.15.81:0/1770874084 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55cc15d7b0 con 0x7f55b8003a30 2024-06-29T14:41:44.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.994+0000 7f55c1ffb640 1 -- 172.21.15.81:0/1770874084 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 75) ==== 97912+0+0 (secure 0 0 0) 0x7f55c8046070 con 0x7f55b8003a30 2024-06-29T14:41:44.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.995+0000 7f55c1ffb640 1 -- 172.21.15.81:0/1770874084 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6540+0+0 (secure 0 0 0) 0x7f55c80b71b0 con 0x7f55b8003a30 2024-06-29T14:41:44.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:44.999+0000 7f55c1ffb640 1 -- 172.21.15.81:0/1770874084 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f55cc15c350 con 0x7f55b8003a30 2024-06-29T14:41:45.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.060+0000 7f55c1ffb640 1 -- 172.21.15.81:0/1770874084 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 76) ==== 98039+0+0 (secure 0 0 0) 0x7f55c803c070 con 0x7f55b8003a30 2024-06-29T14:41:45.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.061+0000 7f55c1ffb640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f55b007bbb0 0x7f55b007dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.062+0000 7f55d0f97640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f55b007bbb0 0x7f55b007dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.063+0000 7f55d0f97640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f55b007bbb0 0x7f55b007dfd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0b1010 tx=0x7f55bc005f90 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.221+0000 7f55d39fc640 1 -- 172.21.15.81:0/1770874084 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55cc0b1040 con 0x7f55b8003a30 2024-06-29T14:41:45.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.222+0000 7f55c1ffb640 1 -- 172.21.15.81:0/1770874084 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5228 (secure 0 0 0) 0x7f55c8080c30 con 0x7f55b8003a30 2024-06-29T14:41:45.219 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:45.219 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":68,"btime":"2024-06-29T14:41:24:728024+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/4097483790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":4097483790},{"type":"v1","addr":"172.21.15.167:6837","nonce":4097483790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1957730707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1957730707},{"type":"v1","addr":"172.21.15.167:6836","nonce":1957730707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":8550,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3852254611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3852254611},{"type":"v1","addr":"172.21.15.81:6833","nonce":3852254611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:21.060976+0000","modified":"2024-06-29T14:41:24.093831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8553":{"gid":8553,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/942339953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":942339953},{"type":"v1","addr":"172.21.15.81:6835","nonce":942339953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8553,"qdb_cluster":[8553]},"id":8}]} 2024-06-29T14:41:45.220 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 68 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 -- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f55b007bbb0 msgr2=0x7f55b007dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f55b007bbb0 0x7f55b007dfd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0b1010 tx=0x7f55bc005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 -- 172.21.15.81:0/1770874084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55b8003a30 msgr2=0x7f55cc0aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55b8003a30 0x7f55cc0aedc0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c510 tx=0x7f55c8006060 comp rx=0 tx=0).stop 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 -- 172.21.15.81:0/1770874084 shutdown_connections 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f55b007bbb0 0x7f55b007dfd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f55cc0b3700 0x7f55cc13b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f55cc15b010 0x7f55cc0b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 --2- 172.21.15.81:0/1770874084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55b8003a30 0x7f55cc0aedc0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 -- 172.21.15.81:0/1770874084 >> 172.21.15.81:0/1770874084 conn(0x7f55cc06ccc0 msgr2=0x7f55cc0afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:45.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 -- 172.21.15.81:0/1770874084 shutdown_connections 2024-06-29T14:41:45.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.224+0000 7f55d39fc640 1 -- 172.21.15.81:0/1770874084 wait complete. 2024-06-29T14:41:45.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:45.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.348+0000 7ffac7beb640 1 -- 172.21.15.81:0/486157980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaac004b60 msgr2=0x7ffaac004fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.348+0000 7ffac7beb640 1 --2- 172.21.15.81:0/486157980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaac004b60 0x7ffaac004fd0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009f20 tx=0x7ffaac006940 comp rx=0 tx=0).stop 2024-06-29T14:41:45.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.348+0000 7ffac7beb640 1 -- 172.21.15.81:0/486157980 shutdown_connections 2024-06-29T14:41:45.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.348+0000 7ffac7beb640 1 --2- 172.21.15.81:0/486157980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaac004b60 0x7ffaac004fd0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.348+0000 7ffac7beb640 1 -- 172.21.15.81:0/486157980 >> 172.21.15.81:0/486157980 conn(0x7ffac006ccc0 msgr2=0x7ffac00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:45.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.348+0000 7ffac7beb640 1 -- 172.21.15.81:0/486157980 shutdown_connections 2024-06-29T14:41:45.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.348+0000 7ffac7beb640 1 -- 172.21.15.81:0/486157980 wait complete. 2024-06-29T14:41:45.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.350+0000 7ffac7beb640 1 Processor -- start 2024-06-29T14:41:45.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.350+0000 7ffac7beb640 1 -- start start 2024-06-29T14:41:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac7beb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffaac004b60 0x7ffac014c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac7beb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac00af070 0x7ffac0071ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac7beb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffac014ea00 0x7ffac0072420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac7beb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0151030 con 0x7ffac00af070 2024-06-29T14:41:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac7beb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0070220 con 0x7ffac014ea00 2024-06-29T14:41:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac7beb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ffac01511b0 con 0x7ffaac004b60 2024-06-29T14:41:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac5186640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac00af070 0x7ffac0071ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac5186640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac00af070 0x7ffac0071ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51658/0 (socket says 172.21.15.81:51658) 2024-06-29T14:41:45.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac5186640 1 -- 172.21.15.81:0/3203485741 learned_addr learned my addr 172.21.15.81:0/3203485741 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac5987640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffaac004b60 0x7ffac014c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.351+0000 7ffac6188640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffac014ea00 0x7ffac0072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffac5186640 1 -- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffaac004b60 msgr2=0x7ffac014c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffac5186640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffaac004b60 0x7ffac014c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffac5186640 1 -- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffac014ea00 msgr2=0x7ffac0072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffac5186640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffac014ea00 0x7ffac0072420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffac5186640 1 -- 172.21.15.81:0/3203485741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac014b750 con 0x7ffac00af070 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffac5186640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac00af070 0x7ffac0071ee0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7ffab0000c00 tx=0x7ffab00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffab67fc640 1 -- 172.21.15.81:0/3203485741 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0015070 con 0x7ffac00af070 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffab67fc640 1 -- 172.21.15.81:0/3203485741 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffab0011860 con 0x7ffac00af070 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffab67fc640 1 -- 172.21.15.81:0/3203485741 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0011cb0 con 0x7ffac00af070 2024-06-29T14:41:45.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffac7beb640 1 -- 172.21.15.81:0/3203485741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac015cd50 con 0x7ffac00af070 2024-06-29T14:41:45.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.352+0000 7ffac7beb640 1 -- 172.21.15.81:0/3203485741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac015d4a0 con 0x7ffac00af070 2024-06-29T14:41:45.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.353+0000 7ffac7beb640 1 -- 172.21.15.81:0/3203485741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffac006edd0 con 0x7ffac00af070 2024-06-29T14:41:45.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.354+0000 7ffab67fc640 1 -- 172.21.15.81:0/3203485741 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 76) ==== 98039+0+0 (secure 0 0 0) 0x7ffab0018070 con 0x7ffac00af070 2024-06-29T14:41:45.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.354+0000 7ffab67fc640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffaa4076150 0x7ffaa4078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.354+0000 7ffab67fc640 1 -- 172.21.15.81:0/3203485741 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6540+0+0 (secure 0 0 0) 0x7ffab00982d0 con 0x7ffac00af070 2024-06-29T14:41:45.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.354+0000 7ffac5987640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffaa4076150 0x7ffaa4078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.355+0000 7ffac5987640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffaa4076150 0x7ffaa4078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffabc036000 tx=0x7ffabc02d040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.359+0000 7ffab67fc640 1 -- 172.21.15.81:0/3203485741 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffab0061560 con 0x7ffac00af070 2024-06-29T14:41:45.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.579+0000 7ffac7beb640 1 -- 172.21.15.81:0/3203485741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffac015cf20 con 0x7ffac00af070 2024-06-29T14:41:45.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.579+0000 7ffab67fc640 1 -- 172.21.15.81:0/3203485741 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5228 (secure 0 0 0) 0x7ffab0065860 con 0x7ffac00af070 2024-06-29T14:41:45.577 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:45.577 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":68,"btime":"2024-06-29T14:41:24:728024+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/4097483790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":4097483790},{"type":"v1","addr":"172.21.15.167:6837","nonce":4097483790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1957730707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1957730707},{"type":"v1","addr":"172.21.15.167:6836","nonce":1957730707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":8550,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3852254611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3852254611},{"type":"v1","addr":"172.21.15.81:6833","nonce":3852254611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:21.060976+0000","modified":"2024-06-29T14:41:24.093831+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8553":{"gid":8553,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/942339953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":942339953},{"type":"v1","addr":"172.21.15.81:6835","nonce":942339953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8553,"qdb_cluster":[8553]},"id":8}]} 2024-06-29T14:41:45.577 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 68 2024-06-29T14:41:45.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.581+0000 7ffac7beb640 1 -- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffaa4076150 msgr2=0x7ffaa4078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.581+0000 7ffac7beb640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffaa4076150 0x7ffaa4078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffabc036000 tx=0x7ffabc02d040 comp rx=0 tx=0).stop 2024-06-29T14:41:45.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.581+0000 7ffac7beb640 1 -- 172.21.15.81:0/3203485741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac00af070 msgr2=0x7ffac0071ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.581+0000 7ffac7beb640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac00af070 0x7ffac0071ee0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7ffab0000c00 tx=0x7ffab00039c0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.581+0000 7ffac7beb640 1 -- 172.21.15.81:0/3203485741 shutdown_connections 2024-06-29T14:41:45.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.581+0000 7ffac7beb640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffaa4076150 0x7ffaa4078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.581+0000 7ffac7beb640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffac014ea00 0x7ffac0072420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.581+0000 7ffac7beb640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac00af070 0x7ffac0071ee0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.581+0000 7ffac7beb640 1 --2- 172.21.15.81:0/3203485741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffaac004b60 0x7ffac014c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.582+0000 7ffac7beb640 1 -- 172.21.15.81:0/3203485741 >> 172.21.15.81:0/3203485741 conn(0x7ffac006ccc0 msgr2=0x7ffac014fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:45.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.582+0000 7ffac7beb640 1 -- 172.21.15.81:0/3203485741 shutdown_connections 2024-06-29T14:41:45.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.582+0000 7ffac7beb640 1 -- 172.21.15.81:0/3203485741 wait complete. 2024-06-29T14:41:45.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:45.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.706+0000 7fe699617640 1 -- 172.21.15.81:0/2768963763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c005250 msgr2=0x7fe67c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.706+0000 7fe699617640 1 --2- 172.21.15.81:0/2768963763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c005250 0x7fe67c00b9c0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fe6940d43d0 tx=0x7fe67c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:45.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.706+0000 7fe699617640 1 -- 172.21.15.81:0/2768963763 shutdown_connections 2024-06-29T14:41:45.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.706+0000 7fe699617640 1 --2- 172.21.15.81:0/2768963763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c005250 0x7fe67c00b9c0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.706+0000 7fe699617640 1 -- 172.21.15.81:0/2768963763 >> 172.21.15.81:0/2768963763 conn(0x7fe69409cf30 msgr2=0x7fe6940d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:45.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.706+0000 7fe699617640 1 -- 172.21.15.81:0/2768963763 shutdown_connections 2024-06-29T14:41:45.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.706+0000 7fe699617640 1 -- 172.21.15.81:0/2768963763 wait complete. 2024-06-29T14:41:45.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.708+0000 7fe699617640 1 Processor -- start 2024-06-29T14:41:45.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.708+0000 7fe699617640 1 -- start start 2024-06-29T14:41:45.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe699617640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c005250 0x7fe694152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe699617640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6941f8070 0x7fe6941f8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe699617640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6940a06e0 0x7fe6940a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe699617640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6940d0810 con 0x7fe67c005250 2024-06-29T14:41:45.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe699617640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe6940d0690 con 0x7fe6941f8070 2024-06-29T14:41:45.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe699617640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe6940d0990 con 0x7fe6940a06e0 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe692ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c005250 0x7fe694152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe692ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c005250 0x7fe694152a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51662/0 (socket says 172.21.15.81:51662) 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe692ffd640 1 -- 172.21.15.81:0/2268182843 learned_addr learned my addr 172.21.15.81:0/2268182843 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe6927fc640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6941f8070 0x7fe6941f8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe6937fe640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6940a06e0 0x7fe6940a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe692ffd640 1 -- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6940a06e0 msgr2=0x7fe6940a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe692ffd640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6940a06e0 0x7fe6940a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe692ffd640 1 -- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6941f8070 msgr2=0x7fe6941f8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe692ffd640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6941f8070 0x7fe6941f8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe692ffd640 1 -- 172.21.15.81:0/2268182843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6941f89c0 con 0x7fe67c005250 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe692ffd640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c005250 0x7fe694152a10 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fe6840117d0 tx=0x7fe684011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe67b7fe640 1 -- 172.21.15.81:0/2268182843 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684004460 con 0x7fe67c005250 2024-06-29T14:41:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.710+0000 7fe67b7fe640 1 -- 172.21.15.81:0/2268182843 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe684016070 con 0x7fe67c005250 2024-06-29T14:41:45.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.711+0000 7fe699617640 1 -- 172.21.15.81:0/2268182843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6941f4a30 con 0x7fe67c005250 2024-06-29T14:41:45.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.711+0000 7fe67b7fe640 1 -- 172.21.15.81:0/2268182843 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68400fb60 con 0x7fe67c005250 2024-06-29T14:41:45.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.711+0000 7fe699617640 1 -- 172.21.15.81:0/2268182843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69409fc00 con 0x7fe67c005250 2024-06-29T14:41:45.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.712+0000 7fe699617640 1 -- 172.21.15.81:0/2268182843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6940b8210 con 0x7fe67c005250 2024-06-29T14:41:45.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.713+0000 7fe67b7fe640 1 -- 172.21.15.81:0/2268182843 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 76) ==== 98039+0+0 (secure 0 0 0) 0x7fe684018030 con 0x7fe67c005250 2024-06-29T14:41:45.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.713+0000 7fe67b7fe640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe680076150 0x7fe680078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:45.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.713+0000 7fe67b7fe640 1 -- 172.21.15.81:0/2268182843 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6540+0+0 (secure 0 0 0) 0x7fe684097e10 con 0x7fe67c005250 2024-06-29T14:41:45.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.713+0000 7fe6927fc640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe680076150 0x7fe680078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:45.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.714+0000 7fe6927fc640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe680076150 0x7fe680078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe69406cd30 tx=0x7fe688006040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:45.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.718+0000 7fe67b7fe640 1 -- 172.21.15.81:0/2268182843 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe6940b8210 con 0x7fe67c005250 2024-06-29T14:41:45.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.901+0000 7fe699617640 1 -- 172.21.15.81:0/2268182843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe6940d1a80 con 0x7fe67c005250 2024-06-29T14:41:45.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.902+0000 7fe67b7fe640 1 -- 172.21.15.81:0/2268182843 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+17852 (secure 0 0 0) 0x7fe6941f89c0 con 0x7fe67c005250 2024-06-29T14:41:45.900 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:45.900 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":135,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:44.046259+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:19.352751+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"132","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:20.014553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3620322784":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/3205425880":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/348977351":"2024-06-30T14:41:44.046221+0000","172.21.15.167:6833/2082979226":"2024-06-30T14:41:44.046221+0000","172.21.15.167:6832/2082979226":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/1183708751":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/1998218314":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/1304833179":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1979367025":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/639087269":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1135028328":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1876032601":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/2923077924":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/4051047679":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/2119756783":"2024-06-30T14:41:44.046221+0000","172.21.15.167:6833/971640072":"2024-06-30T14:41:33.386103+0000","172.21.15.167:6832/971640072":"2024-06-30T14:41:33.386103+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:45.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 -- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe680076150 msgr2=0x7fe680078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe680076150 0x7fe680078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe69406cd30 tx=0x7fe688006040 comp rx=0 tx=0).stop 2024-06-29T14:41:45.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 -- 172.21.15.81:0/2268182843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c005250 msgr2=0x7fe694152a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:45.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c005250 0x7fe694152a10 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fe6840117d0 tx=0x7fe684011c90 comp rx=0 tx=0).stop 2024-06-29T14:41:45.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 -- 172.21.15.81:0/2268182843 shutdown_connections 2024-06-29T14:41:45.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe680076150 0x7fe680078660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6940a06e0 0x7fe6940a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6941f8070 0x7fe6941f8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 --2- 172.21.15.81:0/2268182843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c005250 0x7fe694152a10 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:45.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 -- 172.21.15.81:0/2268182843 >> 172.21.15.81:0/2268182843 conn(0x7fe69409cf30 msgr2=0x7fe69415eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:45.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 -- 172.21.15.81:0/2268182843 shutdown_connections 2024-06-29T14:41:45.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:45.904+0000 7fe699617640 1 -- 172.21.15.81:0/2268182843 wait complete. 2024-06-29T14:41:45.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T14:41:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.030+0000 7f5268e09640 1 -- 172.21.15.81:0/3211281005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f524c004800 msgr2=0x7f524c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.030+0000 7f5268e09640 1 --2- 172.21.15.81:0/3211281005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f524c004800 0x7f524c004c10 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f5258006f80 tx=0x7f524c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.030+0000 7f5268e09640 1 -- 172.21.15.81:0/3211281005 shutdown_connections 2024-06-29T14:41:46.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.030+0000 7f5268e09640 1 --2- 172.21.15.81:0/3211281005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f524c004800 0x7f524c004c10 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.030+0000 7f5268e09640 1 -- 172.21.15.81:0/3211281005 >> 172.21.15.81:0/3211281005 conn(0x7f526406ccc0 msgr2=0x7f52640b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:46.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.031+0000 7f5268e09640 1 -- 172.21.15.81:0/3211281005 shutdown_connections 2024-06-29T14:41:46.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.031+0000 7f5268e09640 1 -- 172.21.15.81:0/3211281005 wait complete. 2024-06-29T14:41:46.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.032+0000 7f5268e09640 1 Processor -- start 2024-06-29T14:41:46.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.032+0000 7f5268e09640 1 -- start start 2024-06-29T14:41:46.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5268e09640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f524c004800 0x7f5264146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5268e09640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5264147850 0x7f5264153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5268e09640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f526414b680 0x7f5264154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5268e09640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f52640cf4c0 con 0x7f524c004800 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5268e09640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f52640cf340 con 0x7f5264147850 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5268e09640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f52640cf640 con 0x7f526414b680 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f526259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f524c004800 0x7f5264146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f526259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f524c004800 0x7f5264146fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51676/0 (socket says 172.21.15.81:51676) 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f526259c640 1 -- 172.21.15.81:0/756193322 learned_addr learned my addr 172.21.15.81:0/756193322 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f526259c640 1 -- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f526414b680 msgr2=0x7f5264154130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f526259c640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f526414b680 0x7f5264154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5261d9b640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5264147850 0x7f5264153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f526259c640 1 -- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5264147850 msgr2=0x7f5264153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f526259c640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5264147850 0x7f5264153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f526259c640 1 -- 172.21.15.81:0/756193322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52641548b0 con 0x7f524c004800 2024-06-29T14:41:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5261d9b640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5264147850 0x7f5264153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f526259c640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f524c004800 0x7f5264146fb0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f5254006900 tx=0x7f5254001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5252ffd640 1 -- 172.21.15.81:0/756193322 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254015070 con 0x7f524c004800 2024-06-29T14:41:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5252ffd640 1 -- 172.21.15.81:0/756193322 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5254010980 con 0x7f524c004800 2024-06-29T14:41:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5268e09640 1 -- 172.21.15.81:0/756193322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52641529b0 con 0x7f524c004800 2024-06-29T14:41:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.034+0000 7f5252ffd640 1 -- 172.21.15.81:0/756193322 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254010dd0 con 0x7f524c004800 2024-06-29T14:41:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.035+0000 7f5268e09640 1 -- 172.21.15.81:0/756193322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5264152f70 con 0x7f524c004800 2024-06-29T14:41:46.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.035+0000 7f5268e09640 1 -- 172.21.15.81:0/756193322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5264072000 con 0x7f524c004800 2024-06-29T14:41:46.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.036+0000 7f5252ffd640 1 -- 172.21.15.81:0/756193322 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 76) ==== 98039+0+0 (secure 0 0 0) 0x7f5254018070 con 0x7f524c004800 2024-06-29T14:41:46.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.037+0000 7f5252ffd640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5244076150 0x7f5244078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.037+0000 7f5252ffd640 1 -- 172.21.15.81:0/756193322 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 6540+0+0 (secure 0 0 0) 0x7f5254097e70 con 0x7f524c004800 2024-06-29T14:41:46.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.037+0000 7f5261d9b640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5244076150 0x7f5244078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.038+0000 7f5261d9b640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5244076150 0x7f5244078660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5258033000 tx=0x7f525802d000 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:46.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.041+0000 7f5252ffd640 1 -- 172.21.15.81:0/756193322 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f52540617d0 con 0x7f524c004800 2024-06-29T14:41:46.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.261+0000 7f5268e09640 1 -- 172.21.15.81:0/756193322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f526415ccc0 con 0x7f524c004800 2024-06-29T14:41:46.259 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T14:41:46.261+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T14:41:46.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.263+0000 7f5252ffd640 1 -- 172.21.15.81:0/756193322 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v69) ==== 161+0+0 (secure 0 0 0) 0x7f52540610d0 con 0x7f524c004800 2024-06-29T14:41:46.260 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T14:41:46.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 -- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5244076150 msgr2=0x7f5244078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5244076150 0x7f5244078660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5258033000 tx=0x7f525802d000 comp rx=0 tx=0).stop 2024-06-29T14:41:46.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 -- 172.21.15.81:0/756193322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f524c004800 msgr2=0x7f5264146fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f524c004800 0x7f5264146fb0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f5254006900 tx=0x7f5254001b80 comp rx=0 tx=0).stop 2024-06-29T14:41:46.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 -- 172.21.15.81:0/756193322 shutdown_connections 2024-06-29T14:41:46.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5244076150 0x7f5244078660 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f526414b680 0x7f5264154130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5264147850 0x7f5264153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 --2- 172.21.15.81:0/756193322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f524c004800 0x7f5264146fb0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 -- 172.21.15.81:0/756193322 >> 172.21.15.81:0/756193322 conn(0x7f526406ccc0 msgr2=0x7f526415bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:46.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 -- 172.21.15.81:0/756193322 shutdown_connections 2024-06-29T14:41:46.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.266+0000 7f5268e09640 1 -- 172.21.15.81:0/756193322 wait complete. 2024-06-29T14:41:46.266 INFO:tasks.ceph.mds.a.smithi081.stderr: -8979> 2024-06-29T14:41:23.674+0000 7f306022e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:41:46.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T14:41:46.323 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:41:46.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.396+0000 7f120ae5e640 1 -- 172.21.15.81:0/940967301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1204074240 msgr2=0x7f12041599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.396+0000 7f120ae5e640 1 --2- 172.21.15.81:0/940967301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1204074240 0x7f12041599f0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009830 tx=0x7f11f402a480 comp rx=0 tx=0).stop 2024-06-29T14:41:46.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.396+0000 7f120ae5e640 1 -- 172.21.15.81:0/940967301 shutdown_connections 2024-06-29T14:41:46.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.396+0000 7f120ae5e640 1 --2- 172.21.15.81:0/940967301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f120415d1a0 0x7f120415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.396+0000 7f120ae5e640 1 --2- 172.21.15.81:0/940967301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1204074240 0x7f12041599f0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.396+0000 7f120ae5e640 1 -- 172.21.15.81:0/940967301 >> 172.21.15.81:0/940967301 conn(0x7f1204072970 msgr2=0x7f12040b5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:46.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.396+0000 7f120ae5e640 1 -- 172.21.15.81:0/940967301 shutdown_connections 2024-06-29T14:41:46.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.396+0000 7f120ae5e640 1 -- 172.21.15.81:0/940967301 wait complete. 2024-06-29T14:41:46.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.398+0000 7f120ae5e640 1 Processor -- start 2024-06-29T14:41:46.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.398+0000 7f120ae5e640 1 -- start start 2024-06-29T14:41:46.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f120ae5e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12040ad160 0x7f1204153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f120ae5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f120415d1a0 0x7f120409d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f120ae5e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f120409da90 0x7f12040b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f120ae5e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f120415ffe0 con 0x7f120415d1a0 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f120ae5e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f120406d750 con 0x7f12040ad160 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f120ae5e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1204160160 con 0x7f120409da90 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f11fbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f120415d1a0 0x7f120409d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f11fbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f120415d1a0 0x7f120409d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51704/0 (socket says 172.21.15.81:51704) 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f11fbfff640 1 -- 172.21.15.81:0/1866311910 learned_addr learned my addr 172.21.15.81:0/1866311910 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f1208bfa640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12040ad160 0x7f1204153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f12093fb640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f120409da90 0x7f12040b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f11fbfff640 1 -- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f120409da90 msgr2=0x7f12040b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f11fbfff640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f120409da90 0x7f12040b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f11fbfff640 1 -- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12040ad160 msgr2=0x7f1204153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:46.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f11fbfff640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12040ad160 0x7f1204153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:46.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f11fbfff640 1 -- 172.21.15.81:0/1866311910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120409bda0 con 0x7f120415d1a0 2024-06-29T14:41:46.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.399+0000 7f1208bfa640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12040ad160 0x7f1204153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:46.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.400+0000 7f11fbfff640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f120415d1a0 0x7f120409d550 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009950 tx=0x7f11f4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:46.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.400+0000 7f11f97fa640 1 -- 172.21.15.81:0/1866311910 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4031d60 con 0x7f120415d1a0 2024-06-29T14:41:46.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.400+0000 7f11f97fa640 1 -- 172.21.15.81:0/1866311910 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11f4033e10 con 0x7f120415d1a0 2024-06-29T14:41:46.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.400+0000 7f120ae5e640 1 -- 172.21.15.81:0/1866311910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f120409c080 con 0x7f120415d1a0 2024-06-29T14:41:46.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.400+0000 7f11f97fa640 1 -- 172.21.15.81:0/1866311910 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f4005260 con 0x7f120415d1a0 2024-06-29T14:41:46.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.400+0000 7f120ae5e640 1 -- 172.21.15.81:0/1866311910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f120409c640 con 0x7f120415d1a0 2024-06-29T14:41:46.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.401+0000 7f120ae5e640 1 -- 172.21.15.81:0/1866311910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1204160610 con 0x7f120415d1a0 2024-06-29T14:41:46.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.401+0000 7f11f97fa640 1 -- 172.21.15.81:0/1866311910 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 76) ==== 98039+0+0 (secure 0 0 0) 0x7f11f4005400 con 0x7f120415d1a0 2024-06-29T14:41:46.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.402+0000 7f11f97fa640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f11f0076120 0x7f11f0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:46.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.402+0000 7f11f97fa640 1 -- 172.21.15.81:0/1866311910 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6626+0+0 (secure 0 0 0) 0x7f11f40b8940 con 0x7f120415d1a0 2024-06-29T14:41:46.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.402+0000 7f1208bfa640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f11f0076120 0x7f11f0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:46.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.403+0000 7f1208bfa640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f11f0076120 0x7f11f0078630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1200000f60 tx=0x7f1200009040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:46.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.406+0000 7f11f97fa640 1 -- 172.21.15.81:0/1866311910 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f11f403f030 con 0x7f120415d1a0 2024-06-29T14:41:46.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.442+0000 7f11f97fa640 1 -- 172.21.15.81:0/1866311910 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 77) ==== 98247+0+0 (secure 0 0 0) 0x7f11f407dd40 con 0x7f120415d1a0 2024-06-29T14:41:46.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:46.626+0000 7f120ae5e640 1 -- 172.21.15.81:0/1866311910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1204074950 con 0x7f120415d1a0 2024-06-29T14:41:47.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.264+0000 7f11f97fa640 1 -- 172.21.15.81:0/1866311910 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v70) ==== 104+0+0 (secure 0 0 0) 0x7f11f4005d00 con 0x7f120415d1a0 2024-06-29T14:41:47.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.266+0000 7f120ae5e640 1 -- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f11f0076120 msgr2=0x7f11f0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:47.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.266+0000 7f120ae5e640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f11f0076120 0x7f11f0078630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1200000f60 tx=0x7f1200009040 comp rx=0 tx=0).stop 2024-06-29T14:41:47.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.266+0000 7f120ae5e640 1 -- 172.21.15.81:0/1866311910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f120415d1a0 msgr2=0x7f120409d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:47.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.266+0000 7f120ae5e640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f120415d1a0 0x7f120409d550 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009950 tx=0x7f11f4033c20 comp rx=0 tx=0).stop 2024-06-29T14:41:47.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.267+0000 7f120ae5e640 1 -- 172.21.15.81:0/1866311910 shutdown_connections 2024-06-29T14:41:47.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.267+0000 7f120ae5e640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f11f0076120 0x7f11f0078630 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.267+0000 7f120ae5e640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f120409da90 0x7f12040b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.267+0000 7f120ae5e640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f120415d1a0 0x7f120409d550 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.267+0000 7f120ae5e640 1 --2- 172.21.15.81:0/1866311910 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12040ad160 0x7f1204153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.267+0000 7f120ae5e640 1 -- 172.21.15.81:0/1866311910 >> 172.21.15.81:0/1866311910 conn(0x7f1204072970 msgr2=0x7f1204155750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:47.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.267+0000 7f120ae5e640 1 -- 172.21.15.81:0/1866311910 shutdown_connections 2024-06-29T14:41:47.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.267+0000 7f120ae5e640 1 -- 172.21.15.81:0/1866311910 wait complete. 2024-06-29T14:41:47.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T14:41:47.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.398+0000 7f954ad2a640 1 -- 172.21.15.81:0/1740488822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952c003a50 msgr2=0x7f952c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:47.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.398+0000 7f954ad2a640 1 --2- 172.21.15.81:0/1740488822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952c003a50 0x7f952c003ec0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f95340016c0 tx=0x7f952c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:47.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.398+0000 7f954ad2a640 1 -- 172.21.15.81:0/1740488822 shutdown_connections 2024-06-29T14:41:47.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.398+0000 7f954ad2a640 1 --2- 172.21.15.81:0/1740488822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952c003a50 0x7f952c003ec0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.398+0000 7f954ad2a640 1 -- 172.21.15.81:0/1740488822 >> 172.21.15.81:0/1740488822 conn(0x7f9544072970 msgr2=0x7f95440b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:47.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.398+0000 7f954ad2a640 1 -- 172.21.15.81:0/1740488822 shutdown_connections 2024-06-29T14:41:47.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.398+0000 7f954ad2a640 1 -- 172.21.15.81:0/1740488822 wait complete. 2024-06-29T14:41:47.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.400+0000 7f954ad2a640 1 Processor -- start 2024-06-29T14:41:47.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.400+0000 7f954ad2a640 1 -- start start 2024-06-29T14:41:47.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.401+0000 7f954ad2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952c003a50 0x7f9544146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:47.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.401+0000 7f954ad2a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f954414d610 0x7f95440cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:47.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.401+0000 7f954ad2a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9544156000 0x7f954415b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.401+0000 7f954ad2a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9544159c30 con 0x7f952c003a50 2024-06-29T14:41:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.401+0000 7f954ad2a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9544159ab0 con 0x7f954414d610 2024-06-29T14:41:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.401+0000 7f954ad2a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f95440747b0 con 0x7f9544156000 2024-06-29T14:41:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f9548ac6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952c003a50 0x7f9544146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f9548ac6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952c003a50 0x7f9544146530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51712/0 (socket says 172.21.15.81:51712) 2024-06-29T14:41:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f9548ac6640 1 -- 172.21.15.81:0/3988624187 learned_addr learned my addr 172.21.15.81:0/3988624187 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f9543fff640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f954414d610 0x7f95440cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f95492c7640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9544156000 0x7f954415b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f9548ac6640 1 -- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9544156000 msgr2=0x7f954415b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f9548ac6640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9544156000 0x7f954415b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f9548ac6640 1 -- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f954414d610 msgr2=0x7f95440cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f9548ac6640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f954414d610 0x7f95440cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f9548ac6640 1 -- 172.21.15.81:0/3988624187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95441f4a10 con 0x7f952c003a50 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f9548ac6640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952c003a50 0x7f9544146530 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f95340017e0 tx=0x7f95340063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f95417fa640 1 -- 172.21.15.81:0/3988624187 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9534041070 con 0x7f952c003a50 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f95417fa640 1 -- 172.21.15.81:0/3988624187 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9534003af0 con 0x7f952c003a50 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f954ad2a640 1 -- 172.21.15.81:0/3988624187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f954415a5b0 con 0x7f952c003a50 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f95417fa640 1 -- 172.21.15.81:0/3988624187 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9534038070 con 0x7f952c003a50 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.402+0000 7f954ad2a640 1 -- 172.21.15.81:0/3988624187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f954415ab70 con 0x7f952c003a50 2024-06-29T14:41:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.403+0000 7f954ad2a640 1 -- 172.21.15.81:0/3988624187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95440740b0 con 0x7f952c003a50 2024-06-29T14:41:47.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.404+0000 7f95417fa640 1 -- 172.21.15.81:0/3988624187 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 77) ==== 98247+0+0 (secure 0 0 0) 0x7f9534030070 con 0x7f952c003a50 2024-06-29T14:41:47.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.405+0000 7f95417fa640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9530076360 0x7f9530078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:47.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.405+0000 7f95417fa640 1 -- 172.21.15.81:0/3988624187 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 6626+0+0 (secure 0 0 0) 0x7f95340b7080 con 0x7f952c003a50 2024-06-29T14:41:47.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.405+0000 7f9543fff640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9530076360 0x7f9530078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:47.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.405+0000 7f9543fff640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9530076360 0x7f9530078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9538005eb0 tx=0x7f9538005f90 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:47.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.409+0000 7f95417fa640 1 -- 172.21.15.81:0/3988624187 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9544074480 con 0x7f952c003a50 2024-06-29T14:41:47.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:47.592+0000 7f954ad2a640 1 -- 172.21.15.81:0/3988624187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f954406ce00 con 0x7f952c003a50 2024-06-29T14:41:48.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.457+0000 7f95417fa640 1 -- 172.21.15.81:0/3988624187 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v137) ==== 181+0+0 (secure 0 0 0) 0x7f9534080650 con 0x7f952c003a50 2024-06-29T14:41:48.454 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T14:41:48.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.463+0000 7f954ad2a640 1 -- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9530076360 msgr2=0x7f9530078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:48.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.463+0000 7f954ad2a640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9530076360 0x7f9530078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9538005eb0 tx=0x7f9538005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:48.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.463+0000 7f954ad2a640 1 -- 172.21.15.81:0/3988624187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952c003a50 msgr2=0x7f9544146530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:48.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.463+0000 7f954ad2a640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952c003a50 0x7f9544146530 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f95340017e0 tx=0x7f95340063a0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.464+0000 7f954ad2a640 1 -- 172.21.15.81:0/3988624187 shutdown_connections 2024-06-29T14:41:48.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.464+0000 7f954ad2a640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9530076360 0x7f9530078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.464+0000 7f954ad2a640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9544156000 0x7f954415b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.464+0000 7f954ad2a640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f954414d610 0x7f95440cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.464+0000 7f954ad2a640 1 --2- 172.21.15.81:0/3988624187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952c003a50 0x7f9544146530 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.464+0000 7f954ad2a640 1 -- 172.21.15.81:0/3988624187 >> 172.21.15.81:0/3988624187 conn(0x7f9544072970 msgr2=0x7f95440b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:48.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.465+0000 7f954ad2a640 1 -- 172.21.15.81:0/3988624187 shutdown_connections 2024-06-29T14:41:48.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.465+0000 7f954ad2a640 1 -- 172.21.15.81:0/3988624187 wait complete. 2024-06-29T14:41:48.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.603+0000 7fb7428a5640 1 -- 172.21.15.81:0/1255302864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb73c0d08b0 msgr2=0x7fb73c0b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.603+0000 7fb7428a5640 1 --2- 172.21.15.81:0/1255302864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb73c0d08b0 0x7fb73c0b13e0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0d2ae0 tx=0x7fb73402a8f0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.603+0000 7fb7428a5640 1 -- 172.21.15.81:0/1255302864 shutdown_connections 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.603+0000 7fb7428a5640 1 --2- 172.21.15.81:0/1255302864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb73c0d08b0 0x7fb73c0b13e0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.603+0000 7fb7428a5640 1 --2- 172.21.15.81:0/1255302864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb73c0d04a0 0x7fb73c0b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.603+0000 7fb7428a5640 1 --2- 172.21.15.81:0/1255302864 >> v2:172.21.15.167:3300/0 conn(0x7fb73c0b25b0 0x7fb73c0cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.603+0000 7fb7428a5640 1 -- 172.21.15.81:0/1255302864 >> 172.21.15.81:0/1255302864 conn(0x7fb73c0b35a0 msgr2=0x7fb73c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.603+0000 7fb7428a5640 1 -- 172.21.15.81:0/1255302864 shutdown_connections 2024-06-29T14:41:48.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.603+0000 7fb7428a5640 1 -- 172.21.15.81:0/1255302864 wait complete. 2024-06-29T14:41:48.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.605+0000 7fb7428a5640 1 Processor -- start 2024-06-29T14:41:48.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.605+0000 7fb7428a5640 1 -- start start 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb7428a5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb73c0b25b0 0x7fb73c06de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb7428a5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb73c0d04a0 0x7fb73c1d44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb7428a5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb73c0d08b0 0x7fb73c1e3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb7428a5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c0b1f80 con 0x7fb73c0b25b0 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb7428a5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c0b1e00 con 0x7fb73c0d08b0 2024-06-29T14:41:48.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb7428a5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb73c0cc620 con 0x7fb73c0d04a0 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb73bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb73c0b25b0 0x7fb73c06de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb73bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb73c0b25b0 0x7fb73c06de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51732/0 (socket says 172.21.15.81:51732) 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb73bfff640 1 -- 172.21.15.81:0/3588419720 learned_addr learned my addr 172.21.15.81:0/3588419720 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb740e42640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb73c0d08b0 0x7fb73c1e3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.606+0000 7fb733fff640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb73c0d04a0 0x7fb73c1d44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb73bfff640 1 -- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb73c0d04a0 msgr2=0x7fb73c1d44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb73bfff640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb73c0d04a0 0x7fb73c1d44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb73bfff640 1 -- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb73c0d08b0 msgr2=0x7fb73c1e3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb73bfff640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb73c0d08b0 0x7fb73c1e3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:48.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb73bfff640 1 -- 172.21.15.81:0/3588419720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73c1f8af0 con 0x7fb73c0b25b0 2024-06-29T14:41:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb740e42640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb73c0d08b0 0x7fb73c1e3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb733fff640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb73c0d04a0 0x7fb73c1d44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb73bfff640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb73c0b25b0 0x7fb73c06de10 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fb72c00b440 tx=0x7fb72c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb7397fa640 1 -- 172.21.15.81:0/3588419720 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c015020 con 0x7fb73c0b25b0 2024-06-29T14:41:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb7397fa640 1 -- 172.21.15.81:0/3588419720 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb72c011980 con 0x7fb73c0b25b0 2024-06-29T14:41:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb7397fa640 1 -- 172.21.15.81:0/3588419720 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72c011dd0 con 0x7fb73c0b25b0 2024-06-29T14:41:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb7428a5640 1 -- 172.21.15.81:0/3588419720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb73c1f8dd0 con 0x7fb73c0b25b0 2024-06-29T14:41:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.607+0000 7fb7428a5640 1 -- 172.21.15.81:0/3588419720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb73c1f9390 con 0x7fb73c0b25b0 2024-06-29T14:41:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.608+0000 7fb7428a5640 1 -- 172.21.15.81:0/3588419720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb73c0cff60 con 0x7fb73c0b25b0 2024-06-29T14:41:48.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.609+0000 7fb7397fa640 1 -- 172.21.15.81:0/3588419720 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fb72c018020 con 0x7fb73c0b25b0 2024-06-29T14:41:48.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.609+0000 7fb7397fa640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb71c0763f0 0x7fb71c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:48.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.609+0000 7fb7397fa640 1 -- 172.21.15.81:0/3588419720 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 6188+0+0 (secure 0 0 0) 0x7fb72c097c60 con 0x7fb73c0b25b0 2024-06-29T14:41:48.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.610+0000 7fb733fff640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb71c0763f0 0x7fb71c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:48.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.610+0000 7fb733fff640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb71c0763f0 0x7fb71c078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb73c1e8620 tx=0x7fb724008040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:48.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.614+0000 7fb7397fa640 1 -- 172.21.15.81:0/3588419720 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb72c01f020 con 0x7fb73c0b25b0 2024-06-29T14:41:48.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:48.803+0000 7fb7428a5640 1 -- 172.21.15.81:0/3588419720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb73c0b0c60 con 0x7fb73c0b25b0 2024-06-29T14:41:49.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.463+0000 7fb7397fa640 1 -- 172.21.15.81:0/3588419720 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v138) ==== 169+0+0 (secure 0 0 0) 0x7fb72c00d040 con 0x7fb73c0b25b0 2024-06-29T14:41:49.461 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T14:41:49.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.466+0000 7fb7428a5640 1 -- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb71c0763f0 msgr2=0x7fb71c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.466+0000 7fb7428a5640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb71c0763f0 0x7fb71c078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb73c1e8620 tx=0x7fb724008040 comp rx=0 tx=0).stop 2024-06-29T14:41:49.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.467+0000 7fb7428a5640 1 -- 172.21.15.81:0/3588419720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb73c0b25b0 msgr2=0x7fb73c06de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.467+0000 7fb7428a5640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb73c0b25b0 0x7fb73c06de10 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fb72c00b440 tx=0x7fb72c00b900 comp rx=0 tx=0).stop 2024-06-29T14:41:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.467+0000 7fb7428a5640 1 -- 172.21.15.81:0/3588419720 shutdown_connections 2024-06-29T14:41:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.467+0000 7fb7428a5640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb71c0763f0 0x7fb71c078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.467+0000 7fb7428a5640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb73c0d08b0 0x7fb73c1e3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.467+0000 7fb7428a5640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb73c0d04a0 0x7fb73c1d44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.467+0000 7fb7428a5640 1 --2- 172.21.15.81:0/3588419720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb73c0b25b0 0x7fb73c06de10 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.467+0000 7fb7428a5640 1 -- 172.21.15.81:0/3588419720 >> 172.21.15.81:0/3588419720 conn(0x7fb73c0b35a0 msgr2=0x7fb73c0b4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.467+0000 7fb7428a5640 1 -- 172.21.15.81:0/3588419720 shutdown_connections 2024-06-29T14:41:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.467+0000 7fb7428a5640 1 -- 172.21.15.81:0/3588419720 wait complete. 2024-06-29T14:41:49.477 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T14:41:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.604+0000 7fb08d9f0640 1 -- 172.21.15.81:0/1787635617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880b3200 msgr2=0x7fb0880cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.604+0000 7fb08d9f0640 1 --2- 172.21.15.81:0/1787635617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880b3200 0x7fb0880cc7b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb078005fd0 tx=0x7fb07802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.604+0000 7fb08d9f0640 1 -- 172.21.15.81:0/1787635617 shutdown_connections 2024-06-29T14:41:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.604+0000 7fb08d9f0640 1 --2- 172.21.15.81:0/1787635617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880cccf0 0x7fb0880d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.604+0000 7fb08d9f0640 1 --2- 172.21.15.81:0/1787635617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880b3200 0x7fb0880cc7b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.604+0000 7fb08d9f0640 1 -- 172.21.15.81:0/1787635617 >> 172.21.15.81:0/1787635617 conn(0x7fb0880b3680 msgr2=0x7fb0880b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.604+0000 7fb08d9f0640 1 -- 172.21.15.81:0/1787635617 shutdown_connections 2024-06-29T14:41:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.604+0000 7fb08d9f0640 1 -- 172.21.15.81:0/1787635617 wait complete. 2024-06-29T14:41:49.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.606+0000 7fb08d9f0640 1 Processor -- start 2024-06-29T14:41:49.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.606+0000 7fb08d9f0640 1 -- start start 2024-06-29T14:41:49.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb08d9f0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880b3200 0x7fb088155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb08d9f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb0880c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb08d9f0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880d0770 0x7fb088159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb08d9f0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0880cedb0 con 0x7fb0880cccf0 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb08d9f0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb0880cec30 con 0x7fb0880d0770 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb08d9f0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb0880cef30 con 0x7fb0880b3200 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0867fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb0880c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0867fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb0880c9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51746/0 (socket says 172.21.15.81:51746) 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0867fc640 1 -- 172.21.15.81:0/3805614852 learned_addr learned my addr 172.21.15.81:0/3805614852 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb086ffd640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880b3200 0x7fb088155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0877fe640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880d0770 0x7fb088159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0867fc640 1 -- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880b3200 msgr2=0x7fb088155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0867fc640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880b3200 0x7fb088155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0867fc640 1 -- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880d0770 msgr2=0x7fb088159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0867fc640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880d0770 0x7fb088159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0867fc640 1 -- 172.21.15.81:0/3805614852 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0880cb840 con 0x7fb0880cccf0 2024-06-29T14:41:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb086ffd640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880b3200 0x7fb088155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0877fe640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880d0770 0x7fb088159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb0867fc640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb0880c9610 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fb07c00ea20 tx=0x7fb07c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb06ffff640 1 -- 172.21.15.81:0/3805614852 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07c00ce40 con 0x7fb0880cccf0 2024-06-29T14:41:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb06ffff640 1 -- 172.21.15.81:0/3805614852 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb07c004020 con 0x7fb0880cccf0 2024-06-29T14:41:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb08d9f0640 1 -- 172.21.15.81:0/3805614852 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0880cbb20 con 0x7fb0880cccf0 2024-06-29T14:41:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.608+0000 7fb06ffff640 1 -- 172.21.15.81:0/3805614852 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07c004470 con 0x7fb0880cccf0 2024-06-29T14:41:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.609+0000 7fb08d9f0640 1 -- 172.21.15.81:0/3805614852 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0881e2b50 con 0x7fb0880cccf0 2024-06-29T14:41:49.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.610+0000 7fb08d9f0640 1 -- 172.21.15.81:0/3805614852 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0880cb5b0 con 0x7fb0880cccf0 2024-06-29T14:41:49.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.610+0000 7fb06ffff640 1 -- 172.21.15.81:0/3805614852 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fb07c0035f0 con 0x7fb0880cccf0 2024-06-29T14:41:49.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.611+0000 7fb06ffff640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb0740763f0 0x7fb074078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.611+0000 7fb06ffff640 1 -- 172.21.15.81:0/3805614852 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5774+0+0 (secure 0 0 0) 0x7fb07c098450 con 0x7fb0880cccf0 2024-06-29T14:41:49.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.611+0000 7fb086ffd640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb0740763f0 0x7fb074078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.612+0000 7fb086ffd640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb0740763f0 0x7fb074078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb078033000 tx=0x7fb078004730 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:49.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.615+0000 7fb06ffff640 1 -- 172.21.15.81:0/3805614852 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb07c061b60 con 0x7fb0880cccf0 2024-06-29T14:41:49.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.796+0000 7fb08d9f0640 1 -- 172.21.15.81:0/3805614852 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fb0880b7460 con 0x7fb0880cccf0 2024-06-29T14:41:49.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.800+0000 7fb06ffff640 1 -- 172.21.15.81:0/3805614852 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v55) ==== 71+0+0 (secure 0 0 0) 0x7fb0880b7460 con 0x7fb0880cccf0 2024-06-29T14:41:49.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 -- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb0740763f0 msgr2=0x7fb074078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb0740763f0 0x7fb074078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb078033000 tx=0x7fb078004730 comp rx=0 tx=0).stop 2024-06-29T14:41:49.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 -- 172.21.15.81:0/3805614852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 msgr2=0x7fb0880c9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb0880c9610 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fb07c00ea20 tx=0x7fb07c00eee0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 -- 172.21.15.81:0/3805614852 shutdown_connections 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb0740763f0 0x7fb074078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb0880d0770 0x7fb088159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0880cccf0 0x7fb0880c9610 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 --2- 172.21.15.81:0/3805614852 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb0880b3200 0x7fb088155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 -- 172.21.15.81:0/3805614852 >> 172.21.15.81:0/3805614852 conn(0x7fb0880b3680 msgr2=0x7fb0880d1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 -- 172.21.15.81:0/3805614852 shutdown_connections 2024-06-29T14:41:49.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.802+0000 7fb08d9f0640 1 -- 172.21.15.81:0/3805614852 wait complete. 2024-06-29T14:41:49.811 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2024-06-29T14:41:49.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.926+0000 7fb507afe640 1 -- 172.21.15.81:0/239475012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb50006f6a0 msgr2=0x7fb50015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.926+0000 7fb507afe640 1 --2- 172.21.15.81:0/239475012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb50006f6a0 0x7fb50015aed0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009830 tx=0x7fb4f002a510 comp rx=0 tx=0).stop 2024-06-29T14:41:49.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.926+0000 7fb507afe640 1 -- 172.21.15.81:0/239475012 shutdown_connections 2024-06-29T14:41:49.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.926+0000 7fb507afe640 1 --2- 172.21.15.81:0/239475012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5000b3960 0x7fb5000b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.926+0000 7fb507afe640 1 --2- 172.21.15.81:0/239475012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb50006f6a0 0x7fb50015aed0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.926+0000 7fb507afe640 1 -- 172.21.15.81:0/239475012 >> 172.21.15.81:0/239475012 conn(0x7fb5000d0930 msgr2=0x7fb5000d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:49.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.926+0000 7fb507afe640 1 -- 172.21.15.81:0/239475012 shutdown_connections 2024-06-29T14:41:49.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.926+0000 7fb507afe640 1 -- 172.21.15.81:0/239475012 wait complete. 2024-06-29T14:41:49.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.928+0000 7fb507afe640 1 Processor -- start 2024-06-29T14:41:49.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.928+0000 7fb507afe640 1 -- start start 2024-06-29T14:41:49.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.929+0000 7fb507afe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb50006f6a0 0x7fb5001f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.929+0000 7fb507afe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5000b3960 0x7fb500155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.929+0000 7fb507afe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5000b57b0 0x7fb5001506e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.929+0000 7fb507afe640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb5000b4430 con 0x7fb5000b3960 2024-06-29T14:41:49.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.929+0000 7fb507afe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb5000b42b0 con 0x7fb50006f6a0 2024-06-29T14:41:49.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.929+0000 7fb507afe640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb5000b45b0 con 0x7fb5000b57b0 2024-06-29T14:41:49.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb505099640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5000b3960 0x7fb500155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb505099640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5000b3960 0x7fb500155f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51750/0 (socket says 172.21.15.81:51750) 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb505099640 1 -- 172.21.15.81:0/94202010 learned_addr learned my addr 172.21.15.81:0/94202010 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb50609b640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5000b57b0 0x7fb5001506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb50589a640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb50006f6a0 0x7fb5001f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb505099640 1 -- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5000b57b0 msgr2=0x7fb5001506e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb505099640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5000b57b0 0x7fb5001506e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb505099640 1 -- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb50006f6a0 msgr2=0x7fb5001f4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb505099640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb50006f6a0 0x7fb5001f4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb505099640 1 -- 172.21.15.81:0/94202010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5001564c0 con 0x7fb5000b3960 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb505099640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5000b3960 0x7fb500155f80 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009950 tx=0x7fb4f0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb4f67fc640 1 -- 172.21.15.81:0/94202010 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0031d60 con 0x7fb5000b3960 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb4f67fc640 1 -- 172.21.15.81:0/94202010 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4f0033e10 con 0x7fb5000b3960 2024-06-29T14:41:49.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb507afe640 1 -- 172.21.15.81:0/94202010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5000adc20 con 0x7fb5000b3960 2024-06-29T14:41:49.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb4f67fc640 1 -- 172.21.15.81:0/94202010 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0005260 con 0x7fb5000b3960 2024-06-29T14:41:49.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.930+0000 7fb507afe640 1 -- 172.21.15.81:0/94202010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5000ae1e0 con 0x7fb5000b3960 2024-06-29T14:41:49.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.932+0000 7fb507afe640 1 -- 172.21.15.81:0/94202010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb50006f4c0 con 0x7fb5000b3960 2024-06-29T14:41:49.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.932+0000 7fb4f67fc640 1 -- 172.21.15.81:0/94202010 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fb4f0005400 con 0x7fb5000b3960 2024-06-29T14:41:49.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.933+0000 7fb4f67fc640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb4e40763f0 0x7fb4e4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:49.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.933+0000 7fb4f67fc640 1 -- 172.21.15.81:0/94202010 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5774+0+0 (secure 0 0 0) 0x7fb4f00b8b20 con 0x7fb5000b3960 2024-06-29T14:41:49.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.933+0000 7fb50589a640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb4e40763f0 0x7fb4e4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:49.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.933+0000 7fb50589a640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb4e40763f0 0x7fb4e4078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0015e0 tx=0x7fb4fc009040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:49.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:49.937+0000 7fb4f67fc640 1 -- 172.21.15.81:0/94202010 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb50006f4c0 con 0x7fb5000b3960 2024-06-29T14:41:50.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.186+0000 7fb507afe640 1 -- 172.21.15.81:0/94202010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) -- 0x7fb50006e2a0 con 0x7fb5000b3960 2024-06-29T14:41:50.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.464+0000 7fb4f67fc640 1 -- 172.21.15.81:0/94202010 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v735) ==== 160+0+0 (secure 0 0 0) 0x7fb50006e2a0 con 0x7fb5000b3960 2024-06-29T14:41:50.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 -- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb4e40763f0 msgr2=0x7fb4e4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb4e40763f0 0x7fb4e4078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0015e0 tx=0x7fb4fc009040 comp rx=0 tx=0).stop 2024-06-29T14:41:50.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 -- 172.21.15.81:0/94202010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5000b3960 msgr2=0x7fb500155f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5000b3960 0x7fb500155f80 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009950 tx=0x7fb4f0033c20 comp rx=0 tx=0).stop 2024-06-29T14:41:50.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 -- 172.21.15.81:0/94202010 shutdown_connections 2024-06-29T14:41:50.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb4e40763f0 0x7fb4e4078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5000b57b0 0x7fb5001506e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5000b3960 0x7fb500155f80 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 --2- 172.21.15.81:0/94202010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb50006f6a0 0x7fb5001f4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 -- 172.21.15.81:0/94202010 >> 172.21.15.81:0/94202010 conn(0x7fb5000d0930 msgr2=0x7fb50006c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:50.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 -- 172.21.15.81:0/94202010 shutdown_connections 2024-06-29T14:41:50.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.467+0000 7fb507afe640 1 -- 172.21.15.81:0/94202010 wait complete. 2024-06-29T14:41:50.475 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2024-06-29T14:41:50.475 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2024-06-29T14:41:50.476 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T14:41:50.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.592+0000 7f47e0864640 1 -- 172.21.15.81:0/1927914875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47c8005250 msgr2=0x7f47c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.592+0000 7f47e0864640 1 --2- 172.21.15.81:0/1927914875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47c8005250 0x7f47c800b9c0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f47dc070040 tx=0x7f47c8005830 comp rx=0 tx=0).stop 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.592+0000 7f47e0864640 1 -- 172.21.15.81:0/1927914875 shutdown_connections 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.592+0000 7f47e0864640 1 --2- 172.21.15.81:0/1927914875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47c8005250 0x7f47c800b9c0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.592+0000 7f47e0864640 1 -- 172.21.15.81:0/1927914875 >> 172.21.15.81:0/1927914875 conn(0x7f47dc06ccc0 msgr2=0x7f47dc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.592+0000 7f47e0864640 1 -- 172.21.15.81:0/1927914875 shutdown_connections 2024-06-29T14:41:50.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.592+0000 7f47e0864640 1 -- 172.21.15.81:0/1927914875 wait complete. 2024-06-29T14:41:50.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.594+0000 7f47e0864640 1 Processor -- start 2024-06-29T14:41:50.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.594+0000 7f47e0864640 1 -- start start 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.595+0000 7f47e0864640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47c8005250 0x7f47dc142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.595+0000 7f47e0864640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47dc13b070 0x7f47dc136460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.595+0000 7f47e0864640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47dc139190 0x7f47dc1395a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.595+0000 7f47e0864640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc06edd0 con 0x7f47dc139190 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.595+0000 7f47e0864640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc144ce0 con 0x7f47dc13b070 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.595+0000 7f47e0864640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f47dc06ef50 con 0x7f47c8005250 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47dad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47dc139190 0x7f47dc1395a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47dad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47dc139190 0x7f47dc1395a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34976/0 (socket says 172.21.15.81:34976) 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47dad9d640 1 -- 172.21.15.81:0/3769564550 learned_addr learned my addr 172.21.15.81:0/3769564550 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47d1d9b640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47dc13b070 0x7f47dc136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47da59c640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47c8005250 0x7f47dc142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47dad9d640 1 -- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47c8005250 msgr2=0x7f47dc142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47dad9d640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47c8005250 0x7f47dc142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47dad9d640 1 -- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47dc13b070 msgr2=0x7f47dc136460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47dad9d640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47dc13b070 0x7f47dc136460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47dad9d640 1 -- 172.21.15.81:0/3769564550 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47dc139ae0 con 0x7f47dc139190 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47da59c640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47c8005250 0x7f47dc142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47dad9d640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47dc139190 0x7f47dc1395a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f47d40012e0 tx=0x7f47d4001490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47d37fe640 1 -- 172.21.15.81:0/3769564550 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d4039070 con 0x7f47dc139190 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47d37fe640 1 -- 172.21.15.81:0/3769564550 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47d402be40 con 0x7f47dc139190 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47e0864640 1 -- 172.21.15.81:0/3769564550 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47dc1369a0 con 0x7f47dc139190 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47d37fe640 1 -- 172.21.15.81:0/3769564550 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d40346d0 con 0x7f47dc139190 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.596+0000 7f47e0864640 1 -- 172.21.15.81:0/3769564550 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47dc13c1d0 con 0x7f47dc139190 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.597+0000 7f47e0864640 1 -- 172.21.15.81:0/3769564550 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47dc06f4b0 con 0x7f47dc139190 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.598+0000 7f47d37fe640 1 -- 172.21.15.81:0/3769564550 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f47d4034870 con 0x7f47dc139190 2024-06-29T14:41:50.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.598+0000 7f47d37fe640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f47bc0763f0 0x7f47bc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.598+0000 7f47d37fe640 1 -- 172.21.15.81:0/3769564550 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5774+0+0 (secure 0 0 0) 0x7f47d40b7f40 con 0x7f47dc139190 2024-06-29T14:41:50.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.599+0000 7f47da59c640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f47bc0763f0 0x7f47bc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.599+0000 7f47da59c640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f47bc0763f0 0x7f47bc078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009460 tx=0x7f47cc005f90 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:50.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.603+0000 7f47d37fe640 1 -- 172.21.15.81:0/3769564550 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f47dc139ae0 con 0x7f47dc139190 2024-06-29T14:41:50.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.784+0000 7f47e0864640 1 -- 172.21.15.81:0/3769564550 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47dc142790 con 0x7f47dc139190 2024-06-29T14:41:50.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.785+0000 7f47d37fe640 1 -- 172.21.15.81:0/3769564550 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+14254 (secure 0 0 0) 0x7f47dc15c350 con 0x7f47dc139190 2024-06-29T14:41:50.783 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:50.783 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":138,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:49.457930+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3620322784":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/3205425880":"2024-06-30T14:41:44.046221+0000","172.21.15.81:6835/942339953":"2024-06-30T14:41:46.262139+0000","172.21.15.167:0/348977351":"2024-06-30T14:41:44.046221+0000","172.21.15.167:6833/2082979226":"2024-06-30T14:41:44.046221+0000","172.21.15.167:6832/2082979226":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/1183708751":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/1998218314":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/1304833179":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1979367025":"2024-06-30T14:41:33.386103+0000","172.21.15.81:6834/942339953":"2024-06-30T14:41:46.262139+0000","172.21.15.167:0/639087269":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1135028328":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/1876032601":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/2923077924":"2024-06-30T14:41:33.386103+0000","172.21.15.167:0/4051047679":"2024-06-30T14:41:44.046221+0000","172.21.15.167:0/2119756783":"2024-06-30T14:41:44.046221+0000","172.21.15.167:6833/971640072":"2024-06-30T14:41:33.386103+0000","172.21.15.167:6832/971640072":"2024-06-30T14:41:33.386103+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 -- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f47bc0763f0 msgr2=0x7f47bc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f47bc0763f0 0x7f47bc078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009460 tx=0x7f47cc005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 -- 172.21.15.81:0/3769564550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47dc139190 msgr2=0x7f47dc1395a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47dc139190 0x7f47dc1395a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f47d40012e0 tx=0x7f47d4001490 comp rx=0 tx=0).stop 2024-06-29T14:41:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 -- 172.21.15.81:0/3769564550 shutdown_connections 2024-06-29T14:41:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f47bc0763f0 0x7f47bc078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47dc139190 0x7f47dc1395a0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47dc13b070 0x7f47dc136460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 --2- 172.21.15.81:0/3769564550 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47c8005250 0x7f47dc142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 -- 172.21.15.81:0/3769564550 >> 172.21.15.81:0/3769564550 conn(0x7f47dc06ccc0 msgr2=0x7f47dc15b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 -- 172.21.15.81:0/3769564550 shutdown_connections 2024-06-29T14:41:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.787+0000 7f47e0864640 1 -- 172.21.15.81:0/3769564550 wait complete. 2024-06-29T14:41:50.796 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:50.796 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:50.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.916+0000 7f2000471640 1 -- 172.21.15.81:0/3706741973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fe4003a50 msgr2=0x7f1fe4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.916+0000 7f2000471640 1 --2- 172.21.15.81:0/3706741973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fe4003a50 0x7f1fe4003ec0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009b50 tx=0x7f1fe4005830 comp rx=0 tx=0).stop 2024-06-29T14:41:50.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.916+0000 7f2000471640 1 -- 172.21.15.81:0/3706741973 shutdown_connections 2024-06-29T14:41:50.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.916+0000 7f2000471640 1 --2- 172.21.15.81:0/3706741973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fe4003a50 0x7f1fe4003ec0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.916+0000 7f2000471640 1 -- 172.21.15.81:0/3706741973 >> 172.21.15.81:0/3706741973 conn(0x7f1ff80b3510 msgr2=0x7f1ff80b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:50.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.916+0000 7f2000471640 1 -- 172.21.15.81:0/3706741973 shutdown_connections 2024-06-29T14:41:50.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.916+0000 7f2000471640 1 -- 172.21.15.81:0/3706741973 wait complete. 2024-06-29T14:41:50.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.918+0000 7f2000471640 1 Processor -- start 2024-06-29T14:41:50.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.918+0000 7f2000471640 1 -- start start 2024-06-29T14:41:50.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.919+0000 7f2000471640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fe4003a50 0x7f1ff81df870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.919+0000 7f2000471640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff81f3b90 0x7f1ff80b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.919+0000 7f2000471640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff81f3450 0x7f1ff80b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.919+0000 7f2000471640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff80d0660 con 0x7f1ff81f3b90 2024-06-29T14:41:50.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.919+0000 7f2000471640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff80d04e0 con 0x7f1ff81f3450 2024-06-29T14:41:50.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.919+0000 7f2000471640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff80d07e0 con 0x7f1fe4003a50 2024-06-29T14:41:50.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffda0c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff81f3b90 0x7f1ff80b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffda0c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff81f3b90 0x7f1ff80b0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34990/0 (socket says 172.21.15.81:34990) 2024-06-29T14:41:50.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffda0c640 1 -- 172.21.15.81:0/3201596639 learned_addr learned my addr 172.21.15.81:0/3201596639 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffe20d640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fe4003a50 0x7f1ff81df870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffea0e640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff81f3450 0x7f1ff80b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffda0c640 1 -- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fe4003a50 msgr2=0x7f1ff81df870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffda0c640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fe4003a50 0x7f1ff81df870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffda0c640 1 -- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff81f3450 msgr2=0x7f1ff80b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffda0c640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff81f3450 0x7f1ff80b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffda0c640 1 -- 172.21.15.81:0/3201596639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff809f130 con 0x7f1ff81f3b90 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffea0e640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff81f3450 0x7f1ff80b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1ffda0c640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff81f3b90 0x7f1ff80b0430 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8004d90 tx=0x7f1fe8001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1feeffd640 1 -- 172.21.15.81:0/3201596639 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80105d0 con 0x7f1ff81f3b90 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1feeffd640 1 -- 172.21.15.81:0/3201596639 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fe8010860 con 0x7f1ff81f3b90 2024-06-29T14:41:50.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f1feeffd640 1 -- 172.21.15.81:0/3201596639 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8010c70 con 0x7f1ff81f3b90 2024-06-29T14:41:50.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f2000471640 1 -- 172.21.15.81:0/3201596639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff81dfdb0 con 0x7f1ff81f3b90 2024-06-29T14:41:50.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.920+0000 7f2000471640 1 -- 172.21.15.81:0/3201596639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff809f6b0 con 0x7f1ff81f3b90 2024-06-29T14:41:50.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.921+0000 7f2000471640 1 -- 172.21.15.81:0/3201596639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ff80b49d0 con 0x7f1ff81f3b90 2024-06-29T14:41:50.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.922+0000 7f1feeffd640 1 -- 172.21.15.81:0/3201596639 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f1fe801c470 con 0x7f1ff81f3b90 2024-06-29T14:41:50.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.922+0000 7f1feeffd640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f1fdc0763f0 0x7f1fdc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:50.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.922+0000 7f1feeffd640 1 -- 172.21.15.81:0/3201596639 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5774+0+0 (secure 0 0 0) 0x7f1fe80983e0 con 0x7f1ff81f3b90 2024-06-29T14:41:50.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.923+0000 7f1ffe20d640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f1fdc0763f0 0x7f1fdc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:50.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.923+0000 7f1ffe20d640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f1fdc0763f0 0x7f1fdc078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4035000 tx=0x7f1ff40046c0 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:50.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:50.927+0000 7f1feeffd640 1 -- 172.21.15.81:0/3201596639 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1fe8061b60 con 0x7f1ff81f3b90 2024-06-29T14:41:51.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.107+0000 7f2000471640 1 -- 172.21.15.81:0/3201596639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1ff80cdf00 con 0x7f1ff81f3b90 2024-06-29T14:41:51.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.108+0000 7f1feeffd640 1 -- 172.21.15.81:0/3201596639 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v138) ==== 93+0+10 (secure 0 0 0) 0x7f1fe8061460 con 0x7f1ff81f3b90 2024-06-29T14:41:51.105 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:41:51.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 -- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f1fdc0763f0 msgr2=0x7f1fdc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:51.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f1fdc0763f0 0x7f1fdc078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4035000 tx=0x7f1ff40046c0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 -- 172.21.15.81:0/3201596639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff81f3b90 msgr2=0x7f1ff80b0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:51.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff81f3b90 0x7f1ff80b0430 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8004d90 tx=0x7f1fe8001630 comp rx=0 tx=0).stop 2024-06-29T14:41:51.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 -- 172.21.15.81:0/3201596639 shutdown_connections 2024-06-29T14:41:51.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f1fdc0763f0 0x7f1fdc078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ff81f3450 0x7f1ff80b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff81f3b90 0x7f1ff80b0430 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 --2- 172.21.15.81:0/3201596639 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fe4003a50 0x7f1ff81df870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 -- 172.21.15.81:0/3201596639 >> 172.21.15.81:0/3201596639 conn(0x7f1ff80b3510 msgr2=0x7f1ff80b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:51.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 -- 172.21.15.81:0/3201596639 shutdown_connections 2024-06-29T14:41:51.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.110+0000 7f2000471640 1 -- 172.21.15.81:0/3201596639 wait complete. 2024-06-29T14:41:51.118 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-06-29T14:41:51.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.234+0000 7f390dff5640 1 -- 172.21.15.81:0/922898384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39080b9040 msgr2=0x7f39080d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:51.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.234+0000 7f390dff5640 1 --2- 172.21.15.81:0/922898384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39080b9040 0x7f39080d0480 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f390808e290 tx=0x7f38f802a5f0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.234+0000 7f390dff5640 1 -- 172.21.15.81:0/922898384 shutdown_connections 2024-06-29T14:41:51.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.234+0000 7f390dff5640 1 --2- 172.21.15.81:0/922898384 >> v2:172.21.15.167:3300/0 conn(0x7f39080b7290 0x7f39080b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.234+0000 7f390dff5640 1 --2- 172.21.15.81:0/922898384 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39080b5250 0x7f39080d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.234+0000 7f390dff5640 1 --2- 172.21.15.81:0/922898384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39080b9040 0x7f39080d0480 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.234+0000 7f390dff5640 1 -- 172.21.15.81:0/922898384 >> 172.21.15.81:0/922898384 conn(0x7f390809cf30 msgr2=0x7f39080d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:51.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.234+0000 7f390dff5640 1 -- 172.21.15.81:0/922898384 shutdown_connections 2024-06-29T14:41:51.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.234+0000 7f390dff5640 1 -- 172.21.15.81:0/922898384 wait complete. 2024-06-29T14:41:51.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.236+0000 7f390dff5640 1 Processor -- start 2024-06-29T14:41:51.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.236+0000 7f390dff5640 1 -- start start 2024-06-29T14:41:51.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.237+0000 7f390dff5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39080b5250 0x7f39080cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:51.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.237+0000 7f390dff5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39080b7290 0x7f390815d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:51.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f390dff5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39080b9040 0x7f39080b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:51.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f390dff5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39080b7ff0 con 0x7f39080b7290 2024-06-29T14:41:51.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f390dff5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f39080b7e70 con 0x7f39080b9040 2024-06-29T14:41:51.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f390dff5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f39080b8170 con 0x7f39080b5250 2024-06-29T14:41:51.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f3906ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39080b7290 0x7f390815d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:51.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f3906ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39080b7290 0x7f390815d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35006/0 (socket says 172.21.15.81:35006) 2024-06-29T14:41:51.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f3906ffd640 1 -- 172.21.15.81:0/1039246698 learned_addr learned my addr 172.21.15.81:0/1039246698 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f39077fe640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39080b5250 0x7f39080cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f3907fff640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39080b9040 0x7f39080b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f3906ffd640 1 -- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39080b5250 msgr2=0x7f39080cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f3906ffd640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39080b5250 0x7f39080cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f3906ffd640 1 -- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39080b9040 msgr2=0x7f39080b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f3906ffd640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39080b9040 0x7f39080b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f3906ffd640 1 -- 172.21.15.81:0/1039246698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39080cbc40 con 0x7f39080b7290 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f3906ffd640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39080b7290 0x7f390815d690 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00e680 tx=0x7f38fc00eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f38effff640 1 -- 172.21.15.81:0/1039246698 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00ca50 con 0x7f39080b7290 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f38effff640 1 -- 172.21.15.81:0/1039246698 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38fc00cbf0 con 0x7f39080b7290 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f390dff5640 1 -- 172.21.15.81:0/1039246698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39080cbf20 con 0x7f39080b7290 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.238+0000 7f38effff640 1 -- 172.21.15.81:0/1039246698 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc005020 con 0x7f39080b7290 2024-06-29T14:41:51.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.239+0000 7f390dff5640 1 -- 172.21.15.81:0/1039246698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39080cc4e0 con 0x7f39080b7290 2024-06-29T14:41:51.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.240+0000 7f390dff5640 1 -- 172.21.15.81:0/1039246698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39080b76a0 con 0x7f39080b7290 2024-06-29T14:41:51.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.240+0000 7f38effff640 1 -- 172.21.15.81:0/1039246698 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f38fc0051c0 con 0x7f39080b7290 2024-06-29T14:41:51.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.241+0000 7f38effff640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f38f40763f0 0x7f38f4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:51.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.241+0000 7f38effff640 1 -- 172.21.15.81:0/1039246698 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5774+0+0 (secure 0 0 0) 0x7f38fc098190 con 0x7f39080b7290 2024-06-29T14:41:51.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.241+0000 7f39077fe640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f38f40763f0 0x7f38f4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:51.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.241+0000 7f39077fe640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f38f40763f0 0x7f38f4078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f38f8033000 tx=0x7f38f8004730 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:51.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.245+0000 7f38effff640 1 -- 172.21.15.81:0/1039246698 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f38fc0611b0 con 0x7f39080b7290 2024-06-29T14:41:51.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:51.498+0000 7f390dff5640 1 -- 172.21.15.81:0/1039246698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) -- 0x7f39080d2ad0 con 0x7f39080b7290 2024-06-29T14:41:52.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.474+0000 7f38effff640 1 -- 172.21.15.81:0/1039246698 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v737) ==== 181+0+0 (secure 0 0 0) 0x7f38fc0654b0 con 0x7f39080b7290 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 -- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f38f40763f0 msgr2=0x7f38f4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f38f40763f0 0x7f38f4078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f38f8033000 tx=0x7f38f8004730 comp rx=0 tx=0).stop 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 -- 172.21.15.81:0/1039246698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39080b7290 msgr2=0x7f390815d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39080b7290 0x7f390815d690 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00e680 tx=0x7f38fc00eb40 comp rx=0 tx=0).stop 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 -- 172.21.15.81:0/1039246698 shutdown_connections 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f38f40763f0 0x7f38f4078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39080b9040 0x7f39080b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39080b7290 0x7f390815d690 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 --2- 172.21.15.81:0/1039246698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39080b5250 0x7f39080cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 -- 172.21.15.81:0/1039246698 >> 172.21.15.81:0/1039246698 conn(0x7f390809cf30 msgr2=0x7f39080d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 -- 172.21.15.81:0/1039246698 shutdown_connections 2024-06-29T14:41:52.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.477+0000 7f390dff5640 1 -- 172.21.15.81:0/1039246698 wait complete. 2024-06-29T14:41:52.488 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T14:41:52.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.610+0000 7fb46a103640 1 -- 172.21.15.81:0/1285323082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb44c003ff0 msgr2=0x7fb44c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.610+0000 7fb46a103640 1 --2- 172.21.15.81:0/1285323082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb44c003ff0 0x7fb44c004800 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fb458009b50 tx=0x7fb44c005830 comp rx=0 tx=0).stop 2024-06-29T14:41:52.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.610+0000 7fb463fff640 1 -- 172.21.15.81:0/1285323082 reap_dead start 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.610+0000 7fb46a103640 1 -- 172.21.15.81:0/1285323082 shutdown_connections 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.610+0000 7fb46a103640 1 -- 172.21.15.81:0/1285323082 >> 172.21.15.81:0/1285323082 conn(0x7fb464072950 msgr2=0x7fb4640b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.610+0000 7fb46a103640 1 -- 172.21.15.81:0/1285323082 shutdown_connections 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.610+0000 7fb46a103640 1 -- 172.21.15.81:0/1285323082 wait complete. 2024-06-29T14:41:52.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.612+0000 7fb46a103640 1 Processor -- start 2024-06-29T14:41:52.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.612+0000 7fb46a103640 1 -- start start 2024-06-29T14:41:52.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb46a103640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb44c003ff0 0x7fb46414f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb46a103640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4640cd320 0x7fb464154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb46a103640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb464153920 0x7fb464155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb46a103640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb46415b010 con 0x7fb4640cd320 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb46a103640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb46406d150 con 0x7fb464153920 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb46a103640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb46415b190 con 0x7fb44c003ff0 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb462ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4640cd320 0x7fb464154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb462ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4640cd320 0x7fb464154cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35022/0 (socket says 172.21.15.81:35022) 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb462ffd640 1 -- 172.21.15.81:0/2598639597 learned_addr learned my addr 172.21.15.81:0/2598639597 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb4637fe640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb44c003ff0 0x7fb46414f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb463fff640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb464153920 0x7fb464155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb462ffd640 1 -- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb44c003ff0 msgr2=0x7fb46414f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb462ffd640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb44c003ff0 0x7fb46414f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb462ffd640 1 -- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb464153920 msgr2=0x7fb464155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb462ffd640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb464153920 0x7fb464155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.613+0000 7fb462ffd640 1 -- 172.21.15.81:0/2598639597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46414fac0 con 0x7fb4640cd320 2024-06-29T14:41:52.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.614+0000 7fb462ffd640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4640cd320 0x7fb464154cf0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fb458036040 tx=0x7fb458033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:52.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.614+0000 7fb44bfff640 1 -- 172.21.15.81:0/2598639597 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb458038040 con 0x7fb4640cd320 2024-06-29T14:41:52.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.614+0000 7fb44bfff640 1 -- 172.21.15.81:0/2598639597 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb458041070 con 0x7fb4640cd320 2024-06-29T14:41:52.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.614+0000 7fb46a103640 1 -- 172.21.15.81:0/2598639597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb46414fda0 con 0x7fb4640cd320 2024-06-29T14:41:52.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.614+0000 7fb44bfff640 1 -- 172.21.15.81:0/2598639597 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb458033ca0 con 0x7fb4640cd320 2024-06-29T14:41:52.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.614+0000 7fb46a103640 1 -- 172.21.15.81:0/2598639597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb464150360 con 0x7fb4640cd320 2024-06-29T14:41:52.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.615+0000 7fb46a103640 1 -- 172.21.15.81:0/2598639597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb464153e70 con 0x7fb4640cd320 2024-06-29T14:41:52.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.615+0000 7fb44bfff640 1 -- 172.21.15.81:0/2598639597 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fb45802b850 con 0x7fb4640cd320 2024-06-29T14:41:52.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.616+0000 7fb44bfff640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb450076420 0x7fb450078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.616+0000 7fb44bfff640 1 -- 172.21.15.81:0/2598639597 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5774+0+0 (secure 0 0 0) 0x7fb4580b8140 con 0x7fb4640cd320 2024-06-29T14:41:52.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.616+0000 7fb4637fe640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb450076420 0x7fb450078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.617+0000 7fb4637fe640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb450076420 0x7fb450078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb454009cd0 tx=0x7fb454005f90 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:52.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.620+0000 7fb44bfff640 1 -- 172.21.15.81:0/2598639597 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb458081950 con 0x7fb4640cd320 2024-06-29T14:41:52.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.799+0000 7fb46a103640 1 -- 172.21.15.81:0/2598639597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fb464068ef0 con 0x7fb4640cd320 2024-06-29T14:41:52.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.800+0000 7fb44bfff640 1 -- 172.21.15.81:0/2598639597 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v55) ==== 59+0+1998 (secure 0 0 0) 0x7fb458081250 con 0x7fb4640cd320 2024-06-29T14:41:52.797 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T14:41:52.797 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T14:41:52.797 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T14:41:52.798 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T14:41:52.799 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T14:41:52.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 -- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb450076420 msgr2=0x7fb450078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb450076420 0x7fb450078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb454009cd0 tx=0x7fb454005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:52.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 -- 172.21.15.81:0/2598639597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4640cd320 msgr2=0x7fb464154cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4640cd320 0x7fb464154cf0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fb458036040 tx=0x7fb458033670 comp rx=0 tx=0).stop 2024-06-29T14:41:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 -- 172.21.15.81:0/2598639597 shutdown_connections 2024-06-29T14:41:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fb450076420 0x7fb450078930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb464153920 0x7fb464155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4640cd320 0x7fb464154cf0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 --2- 172.21.15.81:0/2598639597 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb44c003ff0 0x7fb46414f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 -- 172.21.15.81:0/2598639597 >> 172.21.15.81:0/2598639597 conn(0x7fb464072950 msgr2=0x7fb464158970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 -- 172.21.15.81:0/2598639597 shutdown_connections 2024-06-29T14:41:52.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.802+0000 7fb46a103640 1 -- 172.21.15.81:0/2598639597 wait complete. 2024-06-29T14:41:52.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T14:41:52.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.929+0000 7f7093d3c640 1 -- 172.21.15.81:0/316409632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7074003a00 msgr2=0x7f7074003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.929+0000 7f7093d3c640 1 --2- 172.21.15.81:0/316409632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7074003a00 0x7f7074003e70 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f708c06f150 tx=0x7f7074005830 comp rx=0 tx=0).stop 2024-06-29T14:41:52.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.929+0000 7f70922d9640 1 -- 172.21.15.81:0/316409632 reap_dead start 2024-06-29T14:41:52.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.929+0000 7f7093d3c640 1 -- 172.21.15.81:0/316409632 shutdown_connections 2024-06-29T14:41:52.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.929+0000 7f7093d3c640 1 -- 172.21.15.81:0/316409632 >> 172.21.15.81:0/316409632 conn(0x7f708c09cf30 msgr2=0x7f708c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:52.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.929+0000 7f7093d3c640 1 -- 172.21.15.81:0/316409632 shutdown_connections 2024-06-29T14:41:52.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.929+0000 7f7093d3c640 1 -- 172.21.15.81:0/316409632 wait complete. 2024-06-29T14:41:52.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.931+0000 7f7093d3c640 1 Processor -- start 2024-06-29T14:41:52.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.931+0000 7f7093d3c640 1 -- start start 2024-06-29T14:41:52.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f7093d3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7074003a00 0x7f708c0a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f7093d3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f708c09d9a0 0x7f708c0a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f7093d3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f708c15de30 0x7f708c1f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f7093d3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f708c0d0810 con 0x7f708c09d9a0 2024-06-29T14:41:52.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f7093d3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f708c0d0690 con 0x7f7074003a00 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f7093d3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f708c0b79a0 con 0x7f708c15de30 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f70912d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f708c09d9a0 0x7f708c0a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f70912d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f708c09d9a0 0x7f708c0a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35042/0 (socket says 172.21.15.81:35042) 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f70912d7640 1 -- 172.21.15.81:0/284593115 learned_addr learned my addr 172.21.15.81:0/284593115 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f7091ad8640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7074003a00 0x7f708c0a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f70922d9640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f708c15de30 0x7f708c1f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f70912d7640 1 -- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f708c15de30 msgr2=0x7f708c1f4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f70912d7640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f708c15de30 0x7f708c1f4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f70912d7640 1 -- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7074003a00 msgr2=0x7f708c0a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f70912d7640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7074003a00 0x7f708c0a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.932+0000 7f70912d7640 1 -- 172.21.15.81:0/284593115 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708c0b45d0 con 0x7f708c09d9a0 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.933+0000 7f70912d7640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f708c09d9a0 0x7f708c0a1180 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f7080000fb0 tx=0x7f708000ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.933+0000 7f708a7fc640 1 -- 172.21.15.81:0/284593115 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080015070 con 0x7f708c09d9a0 2024-06-29T14:41:52.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.933+0000 7f708a7fc640 1 -- 172.21.15.81:0/284593115 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7080010980 con 0x7f708c09d9a0 2024-06-29T14:41:52.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.933+0000 7f7093d3c640 1 -- 172.21.15.81:0/284593115 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f708c0b48b0 con 0x7f708c09d9a0 2024-06-29T14:41:52.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.933+0000 7f708a7fc640 1 -- 172.21.15.81:0/284593115 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080010db0 con 0x7f708c09d9a0 2024-06-29T14:41:52.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.933+0000 7f7093d3c640 1 -- 172.21.15.81:0/284593115 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f708c09dde0 con 0x7f708c09d9a0 2024-06-29T14:41:52.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.934+0000 7f7093d3c640 1 -- 172.21.15.81:0/284593115 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f708c0b4ad0 con 0x7f708c09d9a0 2024-06-29T14:41:52.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.935+0000 7f708a7fc640 1 -- 172.21.15.81:0/284593115 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f7080018070 con 0x7f708c09d9a0 2024-06-29T14:41:52.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.936+0000 7f708a7fc640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f7078076420 0x7f7078078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:52.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.936+0000 7f708a7fc640 1 -- 172.21.15.81:0/284593115 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5774+0+0 (secure 0 0 0) 0x7f7080098150 con 0x7f708c09d9a0 2024-06-29T14:41:52.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.936+0000 7f7091ad8640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f7078076420 0x7f7078078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:52.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.936+0000 7f7091ad8640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f7078076420 0x7f7078078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f707c034000 tx=0x7f707c02d040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:52.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:52.939+0000 7f708a7fc640 1 -- 172.21.15.81:0/284593115 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f70800618a0 con 0x7f708c09d9a0 2024-06-29T14:41:53.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.119+0000 7f7093d3c640 1 -- 172.21.15.81:0/284593115 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f708c0b4350 con 0x7f708c09d9a0 2024-06-29T14:41:53.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.122+0000 7f708a7fc640 1 -- 172.21.15.81:0/284593115 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v56)=0 v56) ==== 122+0+0 (secure 0 0 0) 0x7f708c0b4350 con 0x7f708c09d9a0 2024-06-29T14:41:53.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 -- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f7078076420 msgr2=0x7f7078078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f7078076420 0x7f7078078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f707c034000 tx=0x7f707c02d040 comp rx=0 tx=0).stop 2024-06-29T14:41:53.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 -- 172.21.15.81:0/284593115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f708c09d9a0 msgr2=0x7f708c0a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f708c09d9a0 0x7f708c0a1180 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f7080000fb0 tx=0x7f708000ea20 comp rx=0 tx=0).stop 2024-06-29T14:41:53.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 -- 172.21.15.81:0/284593115 shutdown_connections 2024-06-29T14:41:53.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f7078076420 0x7f7078078930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f708c15de30 0x7f708c1f4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f708c09d9a0 0x7f708c0a1180 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 --2- 172.21.15.81:0/284593115 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7074003a00 0x7f708c0a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 -- 172.21.15.81:0/284593115 >> 172.21.15.81:0/284593115 conn(0x7f708c09cf30 msgr2=0x7f708c068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:53.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 -- 172.21.15.81:0/284593115 shutdown_connections 2024-06-29T14:41:53.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.125+0000 7f7093d3c640 1 -- 172.21.15.81:0/284593115 wait complete. 2024-06-29T14:41:53.133 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T14:41:53.134 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:53.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T14:41:53.220 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T14:41:53.220 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:53.220 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:53.236 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:53.236 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:41:53.236 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:53.278 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:53.278 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:41:53.278 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:53.294 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T14:41:53.294 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T14:41:53.294 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:41:53.335 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T14:41:53.335 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T14:41:53.336 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:41:53.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.452+0000 7f42e60e4640 1 -- 172.21.15.81:0/4029913959 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42e00ce3a0 msgr2=0x7f42e00ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.452+0000 7f42e60e4640 1 --2- 172.21.15.81:0/4029913959 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42e00ce3a0 0x7f42e00ce7b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f42c8005fd0 tx=0x7f42c802a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.453+0000 7f42e60e4640 1 -- 172.21.15.81:0/4029913959 shutdown_connections 2024-06-29T14:41:53.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.453+0000 7f42e60e4640 1 --2- 172.21.15.81:0/4029913959 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42e00ce3a0 0x7f42e00ce7b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.453+0000 7f42e60e4640 1 --2- 172.21.15.81:0/4029913959 >> v2:172.21.15.167:3300/0 conn(0x7f42e00b7360 0x7f42e00cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.453+0000 7f42e60e4640 1 -- 172.21.15.81:0/4029913959 >> 172.21.15.81:0/4029913959 conn(0x7f42e00b35a0 msgr2=0x7f42e00b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:53.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.453+0000 7f42e60e4640 1 -- 172.21.15.81:0/4029913959 shutdown_connections 2024-06-29T14:41:53.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.453+0000 7f42e60e4640 1 -- 172.21.15.81:0/4029913959 wait complete. 2024-06-29T14:41:53.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.454+0000 7f42e60e4640 1 Processor -- start 2024-06-29T14:41:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.455+0000 7f42e60e4640 1 -- start start 2024-06-29T14:41:53.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42e60e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e00ce3a0 0x7f42e01d15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42e60e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f42e00cf600 0x7f42e01d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42e60e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42e009dea0 0x7f42e009c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42e60e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f42e00cd880 con 0x7f42e00ce3a0 2024-06-29T14:41:53.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42e60e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f42e00cd700 con 0x7f42e00cf600 2024-06-29T14:41:53.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42e60e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f42e00b6df0 con 0x7f42e009dea0 2024-06-29T14:41:53.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42df7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e00ce3a0 0x7f42e01d15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42df7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e00ce3a0 0x7f42e01d15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35052/0 (socket says 172.21.15.81:35052) 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42df7fe640 1 -- 172.21.15.81:0/463420792 learned_addr learned my addr 172.21.15.81:0/463420792 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42dffff640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42e009dea0 0x7f42e009c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42d6ffd640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f42e00cf600 0x7f42e01d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42df7fe640 1 -- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42e009dea0 msgr2=0x7f42e009c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42df7fe640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42e009dea0 0x7f42e009c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42df7fe640 1 -- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f42e00cf600 msgr2=0x7f42e01d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42df7fe640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f42e00cf600 0x7f42e01d0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42df7fe640 1 -- 172.21.15.81:0/463420792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e01d1b10 con 0x7f42e00ce3a0 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.456+0000 7f42df7fe640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e00ce3a0 0x7f42e01d15d0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f42d000e790 tx=0x7f42d000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.457+0000 7f42dcff9640 1 -- 172.21.15.81:0/463420792 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d000eea0 con 0x7f42e00ce3a0 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.457+0000 7f42e60e4640 1 -- 172.21.15.81:0/463420792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e01d12e0 con 0x7f42e00ce3a0 2024-06-29T14:41:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.457+0000 7f42dcff9640 1 -- 172.21.15.81:0/463420792 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42d000cdb0 con 0x7f42e00ce3a0 2024-06-29T14:41:53.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.457+0000 7f42dcff9640 1 -- 172.21.15.81:0/463420792 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d0010690 con 0x7f42e00ce3a0 2024-06-29T14:41:53.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.457+0000 7f42e60e4640 1 -- 172.21.15.81:0/463420792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e009d0e0 con 0x7f42e00ce3a0 2024-06-29T14:41:53.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.458+0000 7f42e60e4640 1 -- 172.21.15.81:0/463420792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42e00b7360 con 0x7f42e00ce3a0 2024-06-29T14:41:53.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.458+0000 7f42dcff9640 1 -- 172.21.15.81:0/463420792 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f42d0010830 con 0x7f42e00ce3a0 2024-06-29T14:41:53.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.459+0000 7f42dcff9640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f42c00763f0 0x7f42c0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.459+0000 7f42dcff9640 1 -- 172.21.15.81:0/463420792 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5774+0+0 (secure 0 0 0) 0x7f42d0098430 con 0x7f42e00ce3a0 2024-06-29T14:41:53.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.459+0000 7f42d6ffd640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f42c00763f0 0x7f42c0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.460+0000 7f42d6ffd640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f42c00763f0 0x7f42c0078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f42c8033000 tx=0x7f42c8004730 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:53.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.463+0000 7f42dcff9640 1 -- 172.21.15.81:0/463420792 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f42d0061ab0 con 0x7f42e00ce3a0 2024-06-29T14:41:53.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.684+0000 7f42e60e4640 1 -- 172.21.15.81:0/463420792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42e0071420 con 0x7f42e00ce3a0 2024-06-29T14:41:53.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.684+0000 7f42dcff9640 1 -- 172.21.15.81:0/463420792 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+3917 (secure 0 0 0) 0x7f42d00613b0 con 0x7f42e00ce3a0 2024-06-29T14:41:53.682 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:53.682 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":70,"btime":"2024-06-29T14:41:47:263795+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/4097483790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":4097483790},{"type":"v1","addr":"172.21.15.167:6837","nonce":4097483790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":7007,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1957730707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1957730707},{"type":"v1","addr":"172.21.15.167:6836","nonce":1957730707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":8550,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3852254611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3852254611},{"type":"v1","addr":"172.21.15.81:6833","nonce":3852254611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3739935504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3739935504},{"type":"v1","addr":"172.21.15.81:6835","nonce":3739935504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":70}],"filesystems":[]} 2024-06-29T14:41:53.682 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 70 2024-06-29T14:41:53.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.686+0000 7f42e60e4640 1 -- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f42c00763f0 msgr2=0x7f42c0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.686+0000 7f42e60e4640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f42c00763f0 0x7f42c0078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f42c8033000 tx=0x7f42c8004730 comp rx=0 tx=0).stop 2024-06-29T14:41:53.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.686+0000 7f42e60e4640 1 -- 172.21.15.81:0/463420792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e00ce3a0 msgr2=0x7f42e01d15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.686+0000 7f42e60e4640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e00ce3a0 0x7f42e01d15d0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f42d000e790 tx=0x7f42d000ec50 comp rx=0 tx=0).stop 2024-06-29T14:41:53.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.686+0000 7f42e60e4640 1 -- 172.21.15.81:0/463420792 shutdown_connections 2024-06-29T14:41:53.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.686+0000 7f42e60e4640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f42c00763f0 0x7f42c0078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.686+0000 7f42e60e4640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f42e009dea0 0x7f42e009c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.686+0000 7f42e60e4640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f42e00cf600 0x7f42e01d0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.686+0000 7f42e60e4640 1 --2- 172.21.15.81:0/463420792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e00ce3a0 0x7f42e01d15d0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.686+0000 7f42e60e4640 1 -- 172.21.15.81:0/463420792 >> 172.21.15.81:0/463420792 conn(0x7f42e00b35a0 msgr2=0x7f42e00b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:53.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.687+0000 7f42e60e4640 1 -- 172.21.15.81:0/463420792 shutdown_connections 2024-06-29T14:41:53.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.687+0000 7f42e60e4640 1 -- 172.21.15.81:0/463420792 wait complete. 2024-06-29T14:41:53.695 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T14:41:53.695 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T14:41:53.695 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T14:41:53.695 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:53.696 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T14:41:53.696 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T14:41:53.696 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T14:41:53.696 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:53.696 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T14:41:53.696 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T14:41:53.697 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T14:41:53.697 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:53.697 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T14:41:53.697 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T14:41:53.697 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T14:41:53.697 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T14:41:53.705 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T14:41:53.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T14:41:53.707 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T14:41:53.707 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T14:41:53.709 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T14:41:53.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T14:41:53.710 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T14:41:53.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T14:41:53.749 INFO:tasks.ceph.mds.c:Started 2024-06-29T14:41:53.749 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:53.751 INFO:tasks.ceph.mds.a:Started 2024-06-29T14:41:53.751 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:53.752 INFO:tasks.ceph.mds.d:Started 2024-06-29T14:41:53.752 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:53.752 INFO:tasks.ceph.mds.b:Started 2024-06-29T14:41:53.753 DEBUG:teuthology.parallel:result is None 2024-06-29T14:41:53.753 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T14:41:53.834 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T14:41:53.835 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T14:41:53.847 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T14:41:53.847 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T14:41:53.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.875+0000 7f63defee640 1 -- 172.21.15.81:0/2345341845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d80ce770 msgr2=0x7f63d80b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.875+0000 7f63defee640 1 --2- 172.21.15.81:0/2345341845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d80ce770 0x7f63d80b35c0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f63d4005fd0 tx=0x7f63d402a540 comp rx=0 tx=0).stop 2024-06-29T14:41:53.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.875+0000 7f63defee640 1 -- 172.21.15.81:0/2345341845 shutdown_connections 2024-06-29T14:41:53.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.875+0000 7f63defee640 1 --2- 172.21.15.81:0/2345341845 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63d80ceb80 0x7f63d80b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.875+0000 7f63defee640 1 --2- 172.21.15.81:0/2345341845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d80ce770 0x7f63d80b35c0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.875+0000 7f63defee640 1 -- 172.21.15.81:0/2345341845 >> 172.21.15.81:0/2345341845 conn(0x7f63d80d3c60 msgr2=0x7f63d80d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:53.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.875+0000 7f63defee640 1 -- 172.21.15.81:0/2345341845 shutdown_connections 2024-06-29T14:41:53.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.875+0000 7f63defee640 1 -- 172.21.15.81:0/2345341845 wait complete. 2024-06-29T14:41:53.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.877+0000 7f63defee640 1 Processor -- start 2024-06-29T14:41:53.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.877+0000 7f63defee640 1 -- start start 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63defee640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d80ceb80 0x7f63d8157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63defee640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63d80d2240 0x7f63d80b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63defee640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63d8157ca0 0x7f63d80b8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63defee640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f63d80d15a0 con 0x7f63d80ceb80 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63defee640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f63d80d1420 con 0x7f63d80d2240 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63defee640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f63d80d4990 con 0x7f63d8157ca0 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63dcd8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d80ceb80 0x7f63d8157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63dcd8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d80ceb80 0x7f63d8157080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35100/0 (socket says 172.21.15.81:35100) 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63dcd8a640 1 -- 172.21.15.81:0/771018631 learned_addr learned my addr 172.21.15.81:0/771018631 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63cffff640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63d80d2240 0x7f63d80b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.878+0000 7f63dd58b640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63d8157ca0 0x7f63d80b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63dcd8a640 1 -- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63d8157ca0 msgr2=0x7f63d80b8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63dcd8a640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63d8157ca0 0x7f63d80b8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63dcd8a640 1 -- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63d80d2240 msgr2=0x7f63d80b83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:53.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63dcd8a640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63d80d2240 0x7f63d80b83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:53.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63dcd8a640 1 -- 172.21.15.81:0/771018631 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d8161cc0 con 0x7f63d80ceb80 2024-06-29T14:41:53.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63cffff640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63d80d2240 0x7f63d80b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T14:41:53.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63dcd8a640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d80ceb80 0x7f63d8157080 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f63d4006010 tx=0x7f63d40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:53.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63cd7fa640 1 -- 172.21.15.81:0/771018631 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d403f070 con 0x7f63d80ceb80 2024-06-29T14:41:53.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63cd7fa640 1 -- 172.21.15.81:0/771018631 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63d40047a0 con 0x7f63d80ceb80 2024-06-29T14:41:53.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63defee640 1 -- 172.21.15.81:0/771018631 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63d815f670 con 0x7f63d80ceb80 2024-06-29T14:41:53.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63cd7fa640 1 -- 172.21.15.81:0/771018631 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4035030 con 0x7f63d80ceb80 2024-06-29T14:41:53.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.879+0000 7f63defee640 1 -- 172.21.15.81:0/771018631 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63d81614c0 con 0x7f63d80ceb80 2024-06-29T14:41:53.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.880+0000 7f63defee640 1 -- 172.21.15.81:0/771018631 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63d80d2650 con 0x7f63d80ceb80 2024-06-29T14:41:53.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.881+0000 7f63cd7fa640 1 -- 172.21.15.81:0/771018631 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f63d4042070 con 0x7f63d80ceb80 2024-06-29T14:41:53.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.881+0000 7f63cd7fa640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f63c40763f0 0x7f63c4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:53.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.881+0000 7f63cd7fa640 1 -- 172.21.15.81:0/771018631 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5774+0+0 (secure 0 0 0) 0x7f63d40b8840 con 0x7f63d80ceb80 2024-06-29T14:41:53.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.882+0000 7f63cffff640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f63c40763f0 0x7f63c4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:53.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.882+0000 7f63cffff640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f63c40763f0 0x7f63c4078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f63d80d0090 tx=0x7f63c800b2e0 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:53.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:53.886+0000 7f63cd7fa640 1 -- 172.21.15.81:0/771018631 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f63d4081860 con 0x7f63d80ceb80 2024-06-29T14:41:54.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.077+0000 7f63defee640 1 -- 172.21.15.81:0/771018631 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f63d81586b0 con 0x7f63d80ceb80 2024-06-29T14:41:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.486+0000 7f63cd7fa640 1 -- 172.21.15.81:0/771018631 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v140) ==== 97+0+0 (secure 0 0 0) 0x7f63d4085b60 con 0x7f63d80ceb80 2024-06-29T14:41:54.484 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T14:41:54.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 -- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f63c40763f0 msgr2=0x7f63c4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f63c40763f0 0x7f63c4078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f63d80d0090 tx=0x7f63c800b2e0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 -- 172.21.15.81:0/771018631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d80ceb80 msgr2=0x7f63d8157080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d80ceb80 0x7f63d8157080 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f63d4006010 tx=0x7f63d40039e0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 -- 172.21.15.81:0/771018631 shutdown_connections 2024-06-29T14:41:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f63c40763f0 0x7f63c4078900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63d8157ca0 0x7f63d80b8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f63d80d2240 0x7f63d80b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 --2- 172.21.15.81:0/771018631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d80ceb80 0x7f63d8157080 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 -- 172.21.15.81:0/771018631 >> 172.21.15.81:0/771018631 conn(0x7f63d80d3c60 msgr2=0x7f63d81588f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 -- 172.21.15.81:0/771018631 shutdown_connections 2024-06-29T14:41:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.489+0000 7f63defee640 1 -- 172.21.15.81:0/771018631 wait complete. 2024-06-29T14:41:54.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T14:41:54.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.618+0000 7fa4e277f640 1 -- 172.21.15.81:0/112344069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4dc0ad0c0 msgr2=0x7fa4dc15bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.618+0000 7fa4e277f640 1 --2- 172.21.15.81:0/112344069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4dc0ad0c0 0x7fa4dc15bdf0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0d0c70 tx=0x7fa4cc02a560 comp rx=0 tx=0).stop 2024-06-29T14:41:54.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.618+0000 7fa4e277f640 1 -- 172.21.15.81:0/112344069 shutdown_connections 2024-06-29T14:41:54.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.618+0000 7fa4e277f640 1 --2- 172.21.15.81:0/112344069 >> v2:172.21.15.167:3300/0 conn(0x7fa4dc074630 0x7fa4dc06c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.618+0000 7fa4e277f640 1 --2- 172.21.15.81:0/112344069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4dc0ad0c0 0x7fa4dc15bdf0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.618+0000 7fa4e277f640 1 -- 172.21.15.81:0/112344069 >> 172.21.15.81:0/112344069 conn(0x7fa4dc072950 msgr2=0x7fa4dc0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:54.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.618+0000 7fa4e277f640 1 -- 172.21.15.81:0/112344069 shutdown_connections 2024-06-29T14:41:54.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.618+0000 7fa4e277f640 1 -- 172.21.15.81:0/112344069 wait complete. 2024-06-29T14:41:54.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.620+0000 7fa4e277f640 1 Processor -- start 2024-06-29T14:41:54.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.620+0000 7fa4e277f640 1 -- start start 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.621+0000 7fa4e277f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4dc073b10 0x7fa4dc157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.621+0000 7fa4e277f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4dc074630 0x7fa4dc1f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.621+0000 7fa4e277f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4dc0ad0c0 0x7fa4dc1509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.621+0000 7fa4e277f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa4dc06cf80 con 0x7fa4dc074630 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.621+0000 7fa4e277f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa4dc06ce00 con 0x7fa4dc073b10 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.621+0000 7fa4e277f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa4dc06d100 con 0x7fa4dc0ad0c0 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.621+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4dc074630 0x7fa4dc1f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:54.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.621+0000 7fa4db7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4dc074630 0x7fa4dc1f30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35120/0 (socket says 172.21.15.81:35120) 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.621+0000 7fa4db7fe640 1 -- 172.21.15.81:0/4074620073 learned_addr learned my addr 172.21.15.81:0/4074620073 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4dbfff640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4dc073b10 0x7fa4dc157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4e0d1c640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4dc0ad0c0 0x7fa4dc1509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4db7fe640 1 -- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4dc0ad0c0 msgr2=0x7fa4dc1509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4db7fe640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4dc0ad0c0 0x7fa4dc1509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4db7fe640 1 -- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4dc073b10 msgr2=0x7fa4dc157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4db7fe640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4dc073b10 0x7fa4dc157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4db7fe640 1 -- 172.21.15.81:0/4074620073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4dc14edf0 con 0x7fa4dc074630 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4db7fe640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4dc074630 0x7fa4dc1f30b0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000c3c0 tx=0x7fa4d000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4d8ff9640 1 -- 172.21.15.81:0/4074620073 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0017070 con 0x7fa4dc074630 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4d8ff9640 1 -- 172.21.15.81:0/4074620073 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4d00053e0 con 0x7fa4dc074630 2024-06-29T14:41:54.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4e277f640 1 -- 172.21.15.81:0/4074620073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4dc14e480 con 0x7fa4dc074630 2024-06-29T14:41:54.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4d8ff9640 1 -- 172.21.15.81:0/4074620073 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0003870 con 0x7fa4dc074630 2024-06-29T14:41:54.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.622+0000 7fa4e277f640 1 -- 172.21.15.81:0/4074620073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4dc14f0b0 con 0x7fa4dc074630 2024-06-29T14:41:54.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.623+0000 7fa4e277f640 1 -- 172.21.15.81:0/4074620073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4dc06d5d0 con 0x7fa4dc074630 2024-06-29T14:41:54.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.624+0000 7fa4d8ff9640 1 -- 172.21.15.81:0/4074620073 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fa4d001a070 con 0x7fa4dc074630 2024-06-29T14:41:54.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.624+0000 7fa4d8ff9640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fa4c80763f0 0x7fa4c8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:54.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.624+0000 7fa4d8ff9640 1 -- 172.21.15.81:0/4074620073 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4957+0+0 (secure 0 0 0) 0x7fa4d00969e0 con 0x7fa4dc074630 2024-06-29T14:41:54.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.625+0000 7fa4dbfff640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fa4c80763f0 0x7fa4c8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:54.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.625+0000 7fa4dbfff640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fa4c80763f0 0x7fa4c8078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc033000 tx=0x7fa4cc004730 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:54.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.629+0000 7fa4d8ff9640 1 -- 172.21.15.81:0/4074620073 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa4d005fdc0 con 0x7fa4dc074630 2024-06-29T14:41:54.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.879+0000 7fa4e277f640 1 -- 172.21.15.81:0/4074620073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa4dc0cc510 con 0x7fa4dc074630 2024-06-29T14:41:54.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.880+0000 7fa4d8ff9640 1 -- 172.21.15.81:0/4074620073 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v36) ==== 80+0+5304 (secure 0 0 0) 0x7fa4d00640c0 con 0x7fa4dc074630 2024-06-29T14:41:54.877 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:54.877 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.878 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T14:41:54.879 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:54.880 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.881 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.882 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.883 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.884 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.885 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:54.886 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.887 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.888 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.889 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:54.890 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T14:41:54.891 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 -- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fa4c80763f0 msgr2=0x7fa4c8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fa4c80763f0 0x7fa4c8078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc033000 tx=0x7fa4cc004730 comp rx=0 tx=0).stop 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 -- 172.21.15.81:0/4074620073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4dc074630 msgr2=0x7fa4dc1f30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4dc074630 0x7fa4dc1f30b0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000c3c0 tx=0x7fa4d000c880 comp rx=0 tx=0).stop 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 -- 172.21.15.81:0/4074620073 shutdown_connections 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fa4c80763f0 0x7fa4c8078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4dc0ad0c0 0x7fa4dc1509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4dc074630 0x7fa4dc1f30b0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 --2- 172.21.15.81:0/4074620073 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4dc073b10 0x7fa4dc157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 -- 172.21.15.81:0/4074620073 >> 172.21.15.81:0/4074620073 conn(0x7fa4dc072950 msgr2=0x7fa4dc155160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 -- 172.21.15.81:0/4074620073 shutdown_connections 2024-06-29T14:41:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:54.882+0000 7fa4e277f640 1 -- 172.21.15.81:0/4074620073 wait complete. 2024-06-29T14:41:54.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:55.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.009+0000 7fe49ac1f640 1 -- 172.21.15.81:0/4087320264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4940cf4d0 msgr2=0x7fe494073030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.009+0000 7fe49ac1f640 1 --2- 172.21.15.81:0/4087320264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4940cf4d0 0x7fe494073030 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fe488005fd0 tx=0x7fe48802aa90 comp rx=0 tx=0).stop 2024-06-29T14:41:55.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.009+0000 7fe49ac1f640 1 -- 172.21.15.81:0/4087320264 shutdown_connections 2024-06-29T14:41:55.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.009+0000 7fe49ac1f640 1 --2- 172.21.15.81:0/4087320264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4940cf4d0 0x7fe494073030 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.009+0000 7fe49ac1f640 1 --2- 172.21.15.81:0/4087320264 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4941412f0 0x7fe4940cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.009+0000 7fe49ac1f640 1 -- 172.21.15.81:0/4087320264 >> 172.21.15.81:0/4087320264 conn(0x7fe49406ccc0 msgr2=0x7fe4940b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.009+0000 7fe49ac1f640 1 -- 172.21.15.81:0/4087320264 shutdown_connections 2024-06-29T14:41:55.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.009+0000 7fe49ac1f640 1 -- 172.21.15.81:0/4087320264 wait complete. 2024-06-29T14:41:55.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.011+0000 7fe49ac1f640 1 Processor -- start 2024-06-29T14:41:55.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.011+0000 7fe49ac1f640 1 -- start start 2024-06-29T14:41:55.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe49ac1f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe494073570 0x7fe4940ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe49ac1f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4940cf4d0 0x7fe4941f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe49ac1f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4941412f0 0x7fe4941f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe49ac1f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe4940b3210 con 0x7fe4941412f0 2024-06-29T14:41:55.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe49ac1f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe49415f280 con 0x7fe494073570 2024-06-29T14:41:55.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe49ac1f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe4940b3390 con 0x7fe4940cf4d0 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe4991bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4941412f0 0x7fe4941f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe4991bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4941412f0 0x7fe4941f6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35138/0 (socket says 172.21.15.81:35138) 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe4991bc640 1 -- 172.21.15.81:0/1942019114 learned_addr learned my addr 172.21.15.81:0/1942019114 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe4989bb640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe494073570 0x7fe4940ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.012+0000 7fe493fff640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4940cf4d0 0x7fe4941f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe4991bc640 1 -- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4940cf4d0 msgr2=0x7fe4941f6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe4991bc640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4940cf4d0 0x7fe4941f6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe4991bc640 1 -- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe494073570 msgr2=0x7fe4940ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe4991bc640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe494073570 0x7fe4940ae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe4991bc640 1 -- 172.21.15.81:0/1942019114 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe494156210 con 0x7fe4941412f0 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe4989bb640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe494073570 0x7fe4940ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe4991bc640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4941412f0 0x7fe4941f6d80 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fe48c00c3c0 tx=0x7fe48c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe4917fa640 1 -- 172.21.15.81:0/1942019114 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48c017070 con 0x7fe4941412f0 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe4917fa640 1 -- 172.21.15.81:0/1942019114 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe48c0035f0 con 0x7fe4941412f0 2024-06-29T14:41:55.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe49ac1f640 1 -- 172.21.15.81:0/1942019114 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4941564b0 con 0x7fe4941412f0 2024-06-29T14:41:55.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe4917fa640 1 -- 172.21.15.81:0/1942019114 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48c003a40 con 0x7fe4941412f0 2024-06-29T14:41:55.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.013+0000 7fe49ac1f640 1 -- 172.21.15.81:0/1942019114 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe494156a70 con 0x7fe4941412f0 2024-06-29T14:41:55.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.014+0000 7fe49ac1f640 1 -- 172.21.15.81:0/1942019114 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4940cef90 con 0x7fe4941412f0 2024-06-29T14:41:55.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.015+0000 7fe4917fa640 1 -- 172.21.15.81:0/1942019114 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fe48c01a070 con 0x7fe4941412f0 2024-06-29T14:41:55.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.015+0000 7fe4917fa640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe480076420 0x7fe480078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.015+0000 7fe4917fa640 1 -- 172.21.15.81:0/1942019114 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4957+0+0 (secure 0 0 0) 0x7fe48c0967a0 con 0x7fe4941412f0 2024-06-29T14:41:55.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.016+0000 7fe4989bb640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe480076420 0x7fe480078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.016+0000 7fe4989bb640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe480076420 0x7fe480078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe484005fd0 tx=0x7fe48400f360 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.020+0000 7fe4917fa640 1 -- 172.21.15.81:0/1942019114 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe494073980 con 0x7fe4941412f0 2024-06-29T14:41:55.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.202+0000 7fe49ac1f640 1 -- 172.21.15.81:0/1942019114 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4941597d0 con 0x7fe4941412f0 2024-06-29T14:41:55.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.202+0000 7fe4917fa640 1 -- 172.21.15.81:0/1942019114 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) ==== 74+0+13048 (secure 0 0 0) 0x7fe48c060250 con 0x7fe4941412f0 2024-06-29T14:41:55.200 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:55.200 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":142,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:54.496060+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":131,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:55.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 -- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe480076420 msgr2=0x7fe480078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe480076420 0x7fe480078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe484005fd0 tx=0x7fe48400f360 comp rx=0 tx=0).stop 2024-06-29T14:41:55.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 -- 172.21.15.81:0/1942019114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4941412f0 msgr2=0x7fe4941f6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4941412f0 0x7fe4941f6d80 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fe48c00c3c0 tx=0x7fe48c00c880 comp rx=0 tx=0).stop 2024-06-29T14:41:55.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 -- 172.21.15.81:0/1942019114 shutdown_connections 2024-06-29T14:41:55.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fe480076420 0x7fe480078930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4941412f0 0x7fe4941f6d80 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4940cf4d0 0x7fe4941f6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 --2- 172.21.15.81:0/1942019114 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe494073570 0x7fe4940ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 -- 172.21.15.81:0/1942019114 >> 172.21.15.81:0/1942019114 conn(0x7fe49406ccc0 msgr2=0x7fe494159cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 -- 172.21.15.81:0/1942019114 shutdown_connections 2024-06-29T14:41:55.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.205+0000 7fe49ac1f640 1 -- 172.21.15.81:0/1942019114 wait complete. 2024-06-29T14:41:55.214 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T14:41:55.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T14:41:55.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.332+0000 7f76f217a640 1 -- 172.21.15.81:0/1137539353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec073570 msgr2=0x7f76ec15e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.332+0000 7f76f217a640 1 --2- 172.21.15.81:0/1137539353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec073570 0x7f76ec15e960 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f76ec070e50 tx=0x7f76e402aa30 comp rx=0 tx=0).stop 2024-06-29T14:41:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.332+0000 7f76f217a640 1 -- 172.21.15.81:0/1137539353 shutdown_connections 2024-06-29T14:41:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.332+0000 7f76f217a640 1 --2- 172.21.15.81:0/1137539353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec073570 0x7f76ec15e960 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.332+0000 7f76f217a640 1 --2- 172.21.15.81:0/1137539353 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76ec0cf4d0 0x7f76ec073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.332+0000 7f76f217a640 1 --2- 172.21.15.81:0/1137539353 >> v2:172.21.15.167:3300/0 conn(0x7f76ec1412f0 0x7f76ec0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.332+0000 7f76f217a640 1 -- 172.21.15.81:0/1137539353 >> 172.21.15.81:0/1137539353 conn(0x7f76ec06ccc0 msgr2=0x7f76ec0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.332+0000 7f76f217a640 1 -- 172.21.15.81:0/1137539353 shutdown_connections 2024-06-29T14:41:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.332+0000 7f76f217a640 1 -- 172.21.15.81:0/1137539353 wait complete. 2024-06-29T14:41:55.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.333+0000 7f76f217a640 1 Processor -- start 2024-06-29T14:41:55.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.334+0000 7f76f217a640 1 -- start start 2024-06-29T14:41:55.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76f217a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec073570 0x7f76ec0ac8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76f217a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76ec0cf4d0 0x7f76ec159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76f217a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76ec1412f0 0x7f76ec0a96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76f217a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec15f230 con 0x7f76ec073570 2024-06-29T14:41:55.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76f217a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec15f0b0 con 0x7f76ec0cf4d0 2024-06-29T14:41:55.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76f217a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f76ec0b3210 con 0x7f76ec1412f0 2024-06-29T14:41:55.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76eb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec073570 0x7f76ec0ac8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76eb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec073570 0x7f76ec0ac8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35146/0 (socket says 172.21.15.81:35146) 2024-06-29T14:41:55.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76eb7fe640 1 -- 172.21.15.81:0/1668683741 learned_addr learned my addr 172.21.15.81:0/1668683741 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:55.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76eaffd640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76ec0cf4d0 0x7f76ec159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76ebfff640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76ec1412f0 0x7f76ec0a96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76eb7fe640 1 -- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76ec1412f0 msgr2=0x7f76ec0a96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76eb7fe640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76ec1412f0 0x7f76ec0a96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76eb7fe640 1 -- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76ec0cf4d0 msgr2=0x7f76ec159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76eb7fe640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76ec0cf4d0 0x7f76ec159f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76eb7fe640 1 -- 172.21.15.81:0/1668683741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76ec140bc0 con 0x7f76ec073570 2024-06-29T14:41:55.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76ebfff640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76ec1412f0 0x7f76ec0a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:55.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.335+0000 7f76eb7fe640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec073570 0x7f76ec0ac8f0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f76dc00e790 tx=0x7f76dc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.336+0000 7f76d3fff640 1 -- 172.21.15.81:0/1668683741 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc00eea0 con 0x7f76ec073570 2024-06-29T14:41:55.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.336+0000 7f76d3fff640 1 -- 172.21.15.81:0/1668683741 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76dc00cdb0 con 0x7f76ec073570 2024-06-29T14:41:55.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.336+0000 7f76d3fff640 1 -- 172.21.15.81:0/1668683741 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc010690 con 0x7f76ec073570 2024-06-29T14:41:55.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.336+0000 7f76f217a640 1 -- 172.21.15.81:0/1668683741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76ec158c50 con 0x7f76ec073570 2024-06-29T14:41:55.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.336+0000 7f76f217a640 1 -- 172.21.15.81:0/1668683741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76ec159210 con 0x7f76ec073570 2024-06-29T14:41:55.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.337+0000 7f76f217a640 1 -- 172.21.15.81:0/1668683741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76ec15e960 con 0x7f76ec073570 2024-06-29T14:41:55.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.337+0000 7f76d3fff640 1 -- 172.21.15.81:0/1668683741 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f76dc010830 con 0x7f76ec073570 2024-06-29T14:41:55.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.338+0000 7f76d3fff640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f76d80763f0 0x7f76d8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.338+0000 7f76d3fff640 1 -- 172.21.15.81:0/1668683741 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4957+0+0 (secure 0 0 0) 0x7f76dc0980b0 con 0x7f76ec073570 2024-06-29T14:41:55.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.338+0000 7f76eaffd640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f76d80763f0 0x7f76d8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.339+0000 7f76eaffd640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f76d80763f0 0x7f76d8078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f76ec15c780 tx=0x7f76e0005d70 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.342+0000 7f76d3fff640 1 -- 172.21.15.81:0/1668683741 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76dc061ae0 con 0x7f76ec073570 2024-06-29T14:41:55.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.524+0000 7f76f217a640 1 -- 172.21.15.81:0/1668683741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f76ec06fb70 con 0x7f76ec073570 2024-06-29T14:41:55.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.524+0000 7f76d3fff640 1 -- 172.21.15.81:0/1668683741 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v142) ==== 93+0+10 (secure 0 0 0) 0x7f76dc0613e0 con 0x7f76ec073570 2024-06-29T14:41:55.522 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 -- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f76d80763f0 msgr2=0x7f76d8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f76d80763f0 0x7f76d8078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f76ec15c780 tx=0x7f76e0005d70 comp rx=0 tx=0).stop 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 -- 172.21.15.81:0/1668683741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec073570 msgr2=0x7f76ec0ac8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec073570 0x7f76ec0ac8f0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f76dc00e790 tx=0x7f76dc00ec50 comp rx=0 tx=0).stop 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 -- 172.21.15.81:0/1668683741 shutdown_connections 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f76d80763f0 0x7f76d8078900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76ec1412f0 0x7f76ec0a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76ec0cf4d0 0x7f76ec159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 --2- 172.21.15.81:0/1668683741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec073570 0x7f76ec0ac8f0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 -- 172.21.15.81:0/1668683741 >> 172.21.15.81:0/1668683741 conn(0x7f76ec06ccc0 msgr2=0x7f76ec071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 -- 172.21.15.81:0/1668683741 shutdown_connections 2024-06-29T14:41:55.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.526+0000 7f76f217a640 1 -- 172.21.15.81:0/1668683741 wait complete. 2024-06-29T14:41:55.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T14:41:55.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.651+0000 7f171425b640 1 -- 172.21.15.81:0/3499625698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f170c06ca00 msgr2=0x7f170c0af8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.651+0000 7f171425b640 1 --2- 172.21.15.81:0/3499625698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f170c06ca00 0x7f170c0af8f0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f1704005fd0 tx=0x7f170402a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.651+0000 7f171425b640 1 -- 172.21.15.81:0/3499625698 shutdown_connections 2024-06-29T14:41:55.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.651+0000 7f171425b640 1 --2- 172.21.15.81:0/3499625698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f170c06ca00 0x7f170c0af8f0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.651+0000 7f171425b640 1 -- 172.21.15.81:0/3499625698 >> 172.21.15.81:0/3499625698 conn(0x7f170c0d0930 msgr2=0x7f170c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.651+0000 7f171425b640 1 -- 172.21.15.81:0/3499625698 shutdown_connections 2024-06-29T14:41:55.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.651+0000 7f171425b640 1 -- 172.21.15.81:0/3499625698 wait complete. 2024-06-29T14:41:55.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.653+0000 7f171425b640 1 Processor -- start 2024-06-29T14:41:55.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.653+0000 7f171425b640 1 -- start start 2024-06-29T14:41:55.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f171425b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f170c154fc0 0x7f170c1f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f171425b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f170c15e710 0x7f170c152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f171425b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f170c15d190 0x7f170c15b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f171425b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f170c0affb0 con 0x7f170c15e710 2024-06-29T14:41:55.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f171425b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f170c0afe30 con 0x7f170c15d190 2024-06-29T14:41:55.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f171425b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f170c0b0130 con 0x7f170c154fc0 2024-06-29T14:41:55.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f17117f6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f170c15e710 0x7f170c152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f17117f6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f170c15e710 0x7f170c152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35166/0 (socket says 172.21.15.81:35166) 2024-06-29T14:41:55.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f17117f6640 1 -- 172.21.15.81:0/3265965170 learned_addr learned my addr 172.21.15.81:0/3265965170 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f1711ff7640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f170c154fc0 0x7f170c1f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f17127f8640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f170c15d190 0x7f170c15b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f17117f6640 1 -- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f170c154fc0 msgr2=0x7f170c1f5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f17117f6640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f170c154fc0 0x7f170c1f5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f17117f6640 1 -- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f170c15d190 msgr2=0x7f170c15b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f17117f6640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f170c15d190 0x7f170c15b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f17117f6640 1 -- 172.21.15.81:0/3265965170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170c15d930 con 0x7f170c15e710 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.654+0000 7f17117f6640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f170c15e710 0x7f170c152dc0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00c3a0 tx=0x7f16fc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.655+0000 7f17027fc640 1 -- 172.21.15.81:0/3265965170 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc017070 con 0x7f170c15e710 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.655+0000 7f17027fc640 1 -- 172.21.15.81:0/3265965170 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16fc0053c0 con 0x7f170c15e710 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.655+0000 7f171425b640 1 -- 172.21.15.81:0/3265965170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170c15dc10 con 0x7f170c15e710 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.655+0000 7f17027fc640 1 -- 172.21.15.81:0/3265965170 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc003850 con 0x7f170c15e710 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.655+0000 7f171425b640 1 -- 172.21.15.81:0/3265965170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170c159270 con 0x7f170c15e710 2024-06-29T14:41:55.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.656+0000 7f171425b640 1 -- 172.21.15.81:0/3265965170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f170c15d6a0 con 0x7f170c15e710 2024-06-29T14:41:55.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.656+0000 7f17027fc640 1 -- 172.21.15.81:0/3265965170 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f16fc01a070 con 0x7f170c15e710 2024-06-29T14:41:55.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.657+0000 7f17027fc640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f16f0076420 0x7f16f0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.657+0000 7f17027fc640 1 -- 172.21.15.81:0/3265965170 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4957+0+0 (secure 0 0 0) 0x7f16fc0969e0 con 0x7f170c15e710 2024-06-29T14:41:55.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.657+0000 7f1711ff7640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f16f0076420 0x7f16f0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.658+0000 7f1711ff7640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f16f0076420 0x7f16f0078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f17080046b0 tx=0x7f1708009040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.661+0000 7f17027fc640 1 -- 172.21.15.81:0/3265965170 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f16fc060520 con 0x7f170c15e710 2024-06-29T14:41:55.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.842+0000 7f171425b640 1 -- 172.21.15.81:0/3265965170 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f170c06e2a0 con 0x7f170c15e710 2024-06-29T14:41:55.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.842+0000 7f17027fc640 1 -- 172.21.15.81:0/3265965170 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v142) ==== 84+0+65 (secure 0 0 0) 0x7f170c06e2a0 con 0x7f170c15e710 2024-06-29T14:41:55.839 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:55.840 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:41:55.840 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:41:55.840 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T14:41:55.840 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T14:41:55.840 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:41:55.840 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 -- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f16f0076420 msgr2=0x7f16f0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f16f0076420 0x7f16f0078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f17080046b0 tx=0x7f1708009040 comp rx=0 tx=0).stop 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 -- 172.21.15.81:0/3265965170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f170c15e710 msgr2=0x7f170c152dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f170c15e710 0x7f170c152dc0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00c3a0 tx=0x7f16fc00c860 comp rx=0 tx=0).stop 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 -- 172.21.15.81:0/3265965170 shutdown_connections 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f16f0076420 0x7f16f0078930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f170c15d190 0x7f170c15b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f170c15e710 0x7f170c152dc0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 --2- 172.21.15.81:0/3265965170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f170c154fc0 0x7f170c1f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 -- 172.21.15.81:0/3265965170 >> 172.21.15.81:0/3265965170 conn(0x7f170c0d0930 msgr2=0x7f170c0ad9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 -- 172.21.15.81:0/3265965170 shutdown_connections 2024-06-29T14:41:55.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.844+0000 7f171425b640 1 -- 172.21.15.81:0/3265965170 wait complete. 2024-06-29T14:41:55.852 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T14:41:55.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T14:41:55.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.973+0000 7efe79991640 1 -- 172.21.15.81:0/1133830148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe60004800 msgr2=0x7efe60004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.973+0000 7efe79991640 1 --2- 172.21.15.81:0/1133830148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe60004800 0x7efe60004c10 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7efe5c001050 tx=0x7efe60005830 comp rx=0 tx=0).stop 2024-06-29T14:41:55.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.973+0000 7efe79991640 1 -- 172.21.15.81:0/1133830148 shutdown_connections 2024-06-29T14:41:55.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.973+0000 7efe79991640 1 --2- 172.21.15.81:0/1133830148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe60004800 0x7efe60004c10 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.973+0000 7efe79991640 1 -- 172.21.15.81:0/1133830148 >> 172.21.15.81:0/1133830148 conn(0x7efe740b5700 msgr2=0x7efe740b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:55.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.973+0000 7efe79991640 1 -- 172.21.15.81:0/1133830148 shutdown_connections 2024-06-29T14:41:55.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.973+0000 7efe79991640 1 -- 172.21.15.81:0/1133830148 wait complete. 2024-06-29T14:41:55.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.975+0000 7efe79991640 1 Processor -- start 2024-06-29T14:41:55.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.975+0000 7efe79991640 1 -- start start 2024-06-29T14:41:55.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.976+0000 7efe79991640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe60004800 0x7efe740cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.976+0000 7efe79991640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe7409ddc0 0x7efe740cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.976+0000 7efe79991640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe74161180 0x7efe7409fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.976+0000 7efe79991640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe7408c5a0 con 0x7efe7409ddc0 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.976+0000 7efe79991640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efe7408c420 con 0x7efe74161180 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.976+0000 7efe79991640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efe740d17e0 con 0x7efe60004800 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.976+0000 7efe6a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe7409ddc0 0x7efe740cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe6a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe7409ddc0 0x7efe740cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35178/0 (socket says 172.21.15.81:35178) 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe6a7fc640 1 -- 172.21.15.81:0/1680989144 learned_addr learned my addr 172.21.15.81:0/1680989144 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe6a7fc640 1 -- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe60004800 msgr2=0x7efe740cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe737fe640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe74161180 0x7efe7409fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe72ffd640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe60004800 0x7efe740cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe6a7fc640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe60004800 0x7efe740cb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe6a7fc640 1 -- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe74161180 msgr2=0x7efe7409fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe6a7fc640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe74161180 0x7efe7409fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe6a7fc640 1 -- 172.21.15.81:0/1680989144 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe7415f700 con 0x7efe7409ddc0 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe737fe640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe74161180 0x7efe7409fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:41:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe6a7fc640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe7409ddc0 0x7efe740cbb30 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00c350 tx=0x7efe5c006d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe70ff9640 1 -- 172.21.15.81:0/1680989144 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c003a00 con 0x7efe7409ddc0 2024-06-29T14:41:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe70ff9640 1 -- 172.21.15.81:0/1680989144 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe5c003c90 con 0x7efe7409ddc0 2024-06-29T14:41:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe79991640 1 -- 172.21.15.81:0/1680989144 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe741603c0 con 0x7efe7409ddc0 2024-06-29T14:41:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe70ff9640 1 -- 172.21.15.81:0/1680989144 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c02e6e0 con 0x7efe7409ddc0 2024-06-29T14:41:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.977+0000 7efe79991640 1 -- 172.21.15.81:0/1680989144 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe74160980 con 0x7efe7409ddc0 2024-06-29T14:41:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.978+0000 7efe79991640 1 -- 172.21.15.81:0/1680989144 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe740cebc0 con 0x7efe7409ddc0 2024-06-29T14:41:55.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.979+0000 7efe70ff9640 1 -- 172.21.15.81:0/1680989144 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7efe5c035020 con 0x7efe7409ddc0 2024-06-29T14:41:55.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.979+0000 7efe70ff9640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7efe540763f0 0x7efe54078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:55.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.980+0000 7efe70ff9640 1 -- 172.21.15.81:0/1680989144 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 4957+0+0 (secure 0 0 0) 0x7efe5c0b7c30 con 0x7efe7409ddc0 2024-06-29T14:41:55.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.980+0000 7efe72ffd640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7efe540763f0 0x7efe54078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:55.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.980+0000 7efe72ffd640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7efe540763f0 0x7efe54078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efe64006fd0 tx=0x7efe64006bc0 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:55.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:55.984+0000 7efe70ff9640 1 -- 172.21.15.81:0/1680989144 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efe5c047030 con 0x7efe7409ddc0 2024-06-29T14:41:56.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.165+0000 7efe79991640 1 -- 172.21.15.81:0/1680989144 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7efe740b64d0 con 0x7efe7409ddc0 2024-06-29T14:41:56.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.495+0000 7efe70ff9640 1 -- 172.21.15.81:0/1680989144 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v143) ==== 138+0+0 (secure 0 0 0) 0x7efe5c03e070 con 0x7efe7409ddc0 2024-06-29T14:41:56.492 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T14:41:56.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 -- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7efe540763f0 msgr2=0x7efe54078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7efe540763f0 0x7efe54078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efe64006fd0 tx=0x7efe64006bc0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 -- 172.21.15.81:0/1680989144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe7409ddc0 msgr2=0x7efe740cbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe7409ddc0 0x7efe740cbb30 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00c350 tx=0x7efe5c006d00 comp rx=0 tx=0).stop 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 -- 172.21.15.81:0/1680989144 shutdown_connections 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7efe540763f0 0x7efe54078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efe74161180 0x7efe7409fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe7409ddc0 0x7efe740cbb30 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 --2- 172.21.15.81:0/1680989144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efe60004800 0x7efe740cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 -- 172.21.15.81:0/1680989144 >> 172.21.15.81:0/1680989144 conn(0x7efe740b5700 msgr2=0x7efe7406e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 -- 172.21.15.81:0/1680989144 shutdown_connections 2024-06-29T14:41:56.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.497+0000 7efe79991640 1 -- 172.21.15.81:0/1680989144 wait complete. 2024-06-29T14:41:56.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T14:41:56.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.634+0000 7f9f562fe640 1 -- 172.21.15.81:0/2090596039 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f500b4f40 msgr2=0x7f9f500b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.634+0000 7f9f562fe640 1 --2- 172.21.15.81:0/2090596039 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f500b4f40 0x7f9f500b5350 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9f48005fd0 tx=0x7f9f4802a8b0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.634+0000 7f9f562fe640 1 -- 172.21.15.81:0/2090596039 shutdown_connections 2024-06-29T14:41:56.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.634+0000 7f9f562fe640 1 --2- 172.21.15.81:0/2090596039 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f500b4f40 0x7f9f500b5350 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.634+0000 7f9f562fe640 1 --2- 172.21.15.81:0/2090596039 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f500d0230 0x7f9f500d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.634+0000 7f9f562fe640 1 -- 172.21.15.81:0/2090596039 >> 172.21.15.81:0/2090596039 conn(0x7f9f500b3680 msgr2=0x7f9f500b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:56.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.634+0000 7f9f562fe640 1 -- 172.21.15.81:0/2090596039 shutdown_connections 2024-06-29T14:41:56.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.634+0000 7f9f562fe640 1 -- 172.21.15.81:0/2090596039 wait complete. 2024-06-29T14:41:56.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.635+0000 7f9f562fe640 1 Processor -- start 2024-06-29T14:41:56.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.636+0000 7f9f562fe640 1 -- start start 2024-06-29T14:41:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f562fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f500b3200 0x7f9f501e0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f562fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f500b4f40 0x7f9f501e3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f562fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f500d0230 0x7f9f501e3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f562fe640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f500b5c20 con 0x7f9f500b4f40 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f562fe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9f500b5aa0 con 0x7f9f500b3200 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f562fe640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9f500b5da0 con 0x7f9f500d0230 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f4f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f500b4f40 0x7f9f501e3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f4f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f500b4f40 0x7f9f501e3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35190/0 (socket says 172.21.15.81:35190) 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f4f7fe640 1 -- 172.21.15.81:0/631736912 learned_addr learned my addr 172.21.15.81:0/631736912 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f4ffff640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f500b3200 0x7f9f501e0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f5489b640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f500d0230 0x7f9f501e3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f4f7fe640 1 -- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f500d0230 msgr2=0x7f9f501e3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f4f7fe640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f500d0230 0x7f9f501e3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f4f7fe640 1 -- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f500b3200 msgr2=0x7f9f501e0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f4f7fe640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f500b3200 0x7f9f501e0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.637+0000 7f9f4f7fe640 1 -- 172.21.15.81:0/631736912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f501e3220 con 0x7f9f500b4f40 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.638+0000 7f9f4f7fe640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f500b4f40 0x7f9f501e3800 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400e790 tx=0x7f9f4400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.638+0000 7f9f4cff9640 1 -- 172.21.15.81:0/631736912 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4400eea0 con 0x7f9f500b4f40 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.638+0000 7f9f4cff9640 1 -- 172.21.15.81:0/631736912 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f4400cd50 con 0x7f9f500b4f40 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.638+0000 7f9f4cff9640 1 -- 172.21.15.81:0/631736912 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f44010690 con 0x7f9f500b4f40 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.638+0000 7f9f562fe640 1 -- 172.21.15.81:0/631736912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f501e2460 con 0x7f9f500b4f40 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.638+0000 7f9f562fe640 1 -- 172.21.15.81:0/631736912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f501e2a20 con 0x7f9f500b4f40 2024-06-29T14:41:56.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.639+0000 7f9f562fe640 1 -- 172.21.15.81:0/631736912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f501e4310 con 0x7f9f500b4f40 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.640+0000 7f9f4cff9640 1 -- 172.21.15.81:0/631736912 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f9f44010830 con 0x7f9f500b4f40 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.640+0000 7f9f4cff9640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9f3c0763f0 0x7f9f3c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.640+0000 7f9f4cff9640 1 -- 172.21.15.81:0/631736912 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5327+0+0 (secure 0 0 0) 0x7f9f44098170 con 0x7f9f500b4f40 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.640+0000 7f9f4ffff640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9f3c0763f0 0x7f9f3c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:56.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.641+0000 7f9f4ffff640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9f3c0763f0 0x7f9f3c078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f400015e0 tx=0x7f9f40009040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:56.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.645+0000 7f9f4cff9640 1 -- 172.21.15.81:0/631736912 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9f44061a30 con 0x7f9f500b4f40 2024-06-29T14:41:56.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:56.824+0000 7f9f562fe640 1 -- 172.21.15.81:0/631736912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9f500b5350 con 0x7f9f500b4f40 2024-06-29T14:41:57.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.497+0000 7f9f4cff9640 1 -- 172.21.15.81:0/631736912 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v144) ==== 170+0+0 (secure 0 0 0) 0x7f9f44061330 con 0x7f9f500b4f40 2024-06-29T14:41:57.495 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 -- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9f3c0763f0 msgr2=0x7f9f3c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9f3c0763f0 0x7f9f3c078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9f400015e0 tx=0x7f9f40009040 comp rx=0 tx=0).stop 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 -- 172.21.15.81:0/631736912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f500b4f40 msgr2=0x7f9f501e3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f500b4f40 0x7f9f501e3800 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400e790 tx=0x7f9f4400ec50 comp rx=0 tx=0).stop 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 -- 172.21.15.81:0/631736912 shutdown_connections 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9f3c0763f0 0x7f9f3c078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f500d0230 0x7f9f501e3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f500b4f40 0x7f9f501e3800 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 --2- 172.21.15.81:0/631736912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f500b3200 0x7f9f501e0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 -- 172.21.15.81:0/631736912 >> 172.21.15.81:0/631736912 conn(0x7f9f500b3680 msgr2=0x7f9f500714a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 -- 172.21.15.81:0/631736912 shutdown_connections 2024-06-29T14:41:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.500+0000 7f9f562fe640 1 -- 172.21.15.81:0/631736912 wait complete. 2024-06-29T14:41:57.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T14:41:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.644+0000 7fd19ece4640 1 -- 172.21.15.81:0/2026435697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1980b3960 msgr2=0x7fd1980b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.644+0000 7fd19ece4640 1 --2- 172.21.15.81:0/2026435697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1980b3960 0x7fd1980b3d70 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fd190005fd0 tx=0x7fd19002aa90 comp rx=0 tx=0).stop 2024-06-29T14:41:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.644+0000 7fd19ece4640 1 -- 172.21.15.81:0/2026435697 shutdown_connections 2024-06-29T14:41:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.644+0000 7fd19ece4640 1 --2- 172.21.15.81:0/2026435697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1980b3960 0x7fd1980b3d70 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.644+0000 7fd19ece4640 1 --2- 172.21.15.81:0/2026435697 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1980b57b0 0x7fd19815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.644+0000 7fd19ece4640 1 -- 172.21.15.81:0/2026435697 >> 172.21.15.81:0/2026435697 conn(0x7fd1980d0930 msgr2=0x7fd1980d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.644+0000 7fd19ece4640 1 -- 172.21.15.81:0/2026435697 shutdown_connections 2024-06-29T14:41:57.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.644+0000 7fd19ece4640 1 -- 172.21.15.81:0/2026435697 wait complete. 2024-06-29T14:41:57.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.646+0000 7fd19ece4640 1 Processor -- start 2024-06-29T14:41:57.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.646+0000 7fd19ece4640 1 -- start start 2024-06-29T14:41:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.647+0000 7fd19ece4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd19806ef90 0x7fd198150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.647+0000 7fd19ece4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1980b3960 0x7fd1980aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.647+0000 7fd19ece4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1980b57b0 0x7fd198153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.647+0000 7fd19ece4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1980b4430 con 0x7fd1980b3960 2024-06-29T14:41:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.647+0000 7fd19ece4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd1980b42b0 con 0x7fd1980b57b0 2024-06-29T14:41:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.647+0000 7fd19ece4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd1980b45b0 con 0x7fd19806ef90 2024-06-29T14:41:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.647+0000 7fd18ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1980b3960 0x7fd1980aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.647+0000 7fd18ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1980b3960 0x7fd1980aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35212/0 (socket says 172.21.15.81:35212) 2024-06-29T14:41:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.647+0000 7fd18ffff640 1 -- 172.21.15.81:0/3113775004 learned_addr learned my addr 172.21.15.81:0/3113775004 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd19ca80640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd19806ef90 0x7fd198150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd19d281640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1980b57b0 0x7fd198153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd18ffff640 1 -- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd19806ef90 msgr2=0x7fd198150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd18ffff640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd19806ef90 0x7fd198150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd18ffff640 1 -- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1980b57b0 msgr2=0x7fd198153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd18ffff640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1980b57b0 0x7fd198153490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd18ffff640 1 -- 172.21.15.81:0/3113775004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1981f2dc0 con 0x7fd1980b3960 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd18ffff640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1980b3960 0x7fd1980aeb90 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c3a0 tx=0x7fd18800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd18d7fa640 1 -- 172.21.15.81:0/3113775004 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd188017070 con 0x7fd1980b3960 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd18d7fa640 1 -- 172.21.15.81:0/3113775004 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1880053c0 con 0x7fd1980b3960 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd18d7fa640 1 -- 172.21.15.81:0/3113775004 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd188003850 con 0x7fd1980b3960 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd19ece4640 1 -- 172.21.15.81:0/3113775004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1981f30a0 con 0x7fd1980b3960 2024-06-29T14:41:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.648+0000 7fd19ece4640 1 -- 172.21.15.81:0/3113775004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1981f3660 con 0x7fd1980b3960 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.649+0000 7fd19ece4640 1 -- 172.21.15.81:0/3113775004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd19806e2a0 con 0x7fd1980b3960 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.650+0000 7fd18d7fa640 1 -- 172.21.15.81:0/3113775004 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fd18801a070 con 0x7fd1980b3960 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.651+0000 7fd18d7fa640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fd1840763f0 0x7fd184078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:57.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.651+0000 7fd18d7fa640 1 -- 172.21.15.81:0/3113775004 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 5709+0+0 (secure 0 0 0) 0x7fd188096cf0 con 0x7fd1980b3960 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.651+0000 7fd19ca80640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fd1840763f0 0x7fd184078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.652+0000 7fd19ca80640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fd1840763f0 0x7fd184078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd194004630 tx=0x7fd19400b2e0 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:57.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.655+0000 7fd18d7fa640 1 -- 172.21.15.81:0/3113775004 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd19806e2a0 con 0x7fd1980b3960 2024-06-29T14:41:57.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:57.874+0000 7fd19ece4640 1 -- 172.21.15.81:0/3113775004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd1980b4a60 con 0x7fd1980b3960 2024-06-29T14:41:58.527 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 18 and data pool 19 2024-06-29T14:41:58.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.529+0000 7fd18d7fa640 1 -- 172.21.15.81:0/3113775004 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v73) ==== 174+0+0 (secure 0 0 0) 0x7fd18809b070 con 0x7fd1980b3960 2024-06-29T14:41:58.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.534+0000 7fd19ece4640 1 -- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fd1840763f0 msgr2=0x7fd184078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.534+0000 7fd19ece4640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fd1840763f0 0x7fd184078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd194004630 tx=0x7fd19400b2e0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.534+0000 7fd19ece4640 1 -- 172.21.15.81:0/3113775004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1980b3960 msgr2=0x7fd1980aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.534+0000 7fd19ece4640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1980b3960 0x7fd1980aeb90 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c3a0 tx=0x7fd18800c860 comp rx=0 tx=0).stop 2024-06-29T14:41:58.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.535+0000 7fd19ece4640 1 -- 172.21.15.81:0/3113775004 shutdown_connections 2024-06-29T14:41:58.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.535+0000 7fd19ece4640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fd1840763f0 0x7fd184078900 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd194004630 tx=0x7fd19400b2e0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.535+0000 7fd19ece4640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1980b57b0 0x7fd198153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.535+0000 7fd19ece4640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1980b3960 0x7fd1980aeb90 secure :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c3a0 tx=0x7fd18800c860 comp rx=0 tx=0).stop 2024-06-29T14:41:58.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.535+0000 7fd19ece4640 1 --2- 172.21.15.81:0/3113775004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd19806ef90 0x7fd198150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.535+0000 7fd19ece4640 1 -- 172.21.15.81:0/3113775004 >> 172.21.15.81:0/3113775004 conn(0x7fd1980d0930 msgr2=0x7fd1980ce190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:58.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.535+0000 7fd19ece4640 1 -- 172.21.15.81:0/3113775004 shutdown_connections 2024-06-29T14:41:58.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.535+0000 7fd19ece4640 1 -- 172.21.15.81:0/3113775004 wait complete. 2024-06-29T14:41:58.552 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:58.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.668+0000 7fcac6143640 1 -- 172.21.15.81:0/2822544126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcaa8003a00 msgr2=0x7fcaa8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.668+0000 7fcac6143640 1 --2- 172.21.15.81:0/2822544126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcaa8003a00 0x7fcaa8003e70 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fcab000c640 tx=0x7fcaa8005830 comp rx=0 tx=0).stop 2024-06-29T14:41:58.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.668+0000 7fcabffff640 1 -- 172.21.15.81:0/2822544126 reap_dead start 2024-06-29T14:41:58.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.668+0000 7fcac6143640 1 -- 172.21.15.81:0/2822544126 shutdown_connections 2024-06-29T14:41:58.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.668+0000 7fcac6143640 1 -- 172.21.15.81:0/2822544126 >> 172.21.15.81:0/2822544126 conn(0x7fcac00d0930 msgr2=0x7fcac00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:58.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.668+0000 7fcac6143640 1 -- 172.21.15.81:0/2822544126 shutdown_connections 2024-06-29T14:41:58.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.668+0000 7fcac6143640 1 -- 172.21.15.81:0/2822544126 wait complete. 2024-06-29T14:41:58.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.670+0000 7fcac6143640 1 Processor -- start 2024-06-29T14:41:58.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.670+0000 7fcac6143640 1 -- start start 2024-06-29T14:41:58.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcac6143640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcaa8003a00 0x7fcac00aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcac6143640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcac01e2150 0x7fcac01e0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcac6143640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcac01545b0 0x7fcac0153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcac6143640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcac00b3ef0 con 0x7fcaa8003a00 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcac6143640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcac00b3d70 con 0x7fcac01545b0 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcac6143640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcac00b4070 con 0x7fcac01e2150 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcaa8003a00 0x7fcac00aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcaa8003a00 0x7fcac00aeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35222/0 (socket says 172.21.15.81:35222) 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabf7fe640 1 -- 172.21.15.81:0/911112711 learned_addr learned my addr 172.21.15.81:0/911112711 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabeffd640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcac01e2150 0x7fcac01e0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabffff640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcac01545b0 0x7fcac0153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabf7fe640 1 -- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcac01e2150 msgr2=0x7fcac01e0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabf7fe640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcac01e2150 0x7fcac01e0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabf7fe640 1 -- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcac01545b0 msgr2=0x7fcac0153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabf7fe640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcac01545b0 0x7fcac0153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabf7fe640 1 -- 172.21.15.81:0/911112711 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac01e0330 con 0x7fcaa8003a00 2024-06-29T14:41:58.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcabf7fe640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcaa8003a00 0x7fcac00aeed0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fcab002b490 tx=0x7fcab0004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:58.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.672+0000 7fcaa7fff640 1 -- 172.21.15.81:0/911112711 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab002b9e0 con 0x7fcaa8003a00 2024-06-29T14:41:58.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.673+0000 7fcaa7fff640 1 -- 172.21.15.81:0/911112711 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcab002bb80 con 0x7fcaa8003a00 2024-06-29T14:41:58.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.673+0000 7fcac6143640 1 -- 172.21.15.81:0/911112711 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac00af720 con 0x7fcaa8003a00 2024-06-29T14:41:58.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.673+0000 7fcaa7fff640 1 -- 172.21.15.81:0/911112711 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0006070 con 0x7fcaa8003a00 2024-06-29T14:41:58.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.673+0000 7fcac6143640 1 -- 172.21.15.81:0/911112711 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac01e08b0 con 0x7fcaa8003a00 2024-06-29T14:41:58.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.673+0000 7fcac6143640 1 -- 172.21.15.81:0/911112711 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac006a5d0 con 0x7fcaa8003a00 2024-06-29T14:41:58.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.674+0000 7fcaa7fff640 1 -- 172.21.15.81:0/911112711 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7fcab0006210 con 0x7fcaa8003a00 2024-06-29T14:41:58.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.675+0000 7fcaa7fff640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fcaac0763f0 0x7fcaac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.675+0000 7fcaa7fff640 1 -- 172.21.15.81:0/911112711 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 5809+0+0 (secure 0 0 0) 0x7fcab00b8370 con 0x7fcaa8003a00 2024-06-29T14:41:58.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.675+0000 7fcabeffd640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fcaac0763f0 0x7fcaac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.675+0000 7fcabeffd640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fcaac0763f0 0x7fcaac078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcac01e5850 tx=0x7fcab4005f90 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:58.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.679+0000 7fcaa7fff640 1 -- 172.21.15.81:0/911112711 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcac006a5d0 con 0x7fcaa8003a00 2024-06-29T14:41:58.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.862+0000 7fcac6143640 1 -- 172.21.15.81:0/911112711 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcac006e2a0 con 0x7fcaa8003a00 2024-06-29T14:41:58.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.863+0000 7fcaa7fff640 1 -- 172.21.15.81:0/911112711 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) ==== 74+0+16839 (secure 0 0 0) 0x7fcac006e2a0 con 0x7fcaa8003a00 2024-06-29T14:41:58.861 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:58.861 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":145,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:57.875501+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.170598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.829654+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:58.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 -- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fcaac0763f0 msgr2=0x7fcaac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fcaac0763f0 0x7fcaac078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcac01e5850 tx=0x7fcab4005f90 comp rx=0 tx=0).stop 2024-06-29T14:41:58.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 -- 172.21.15.81:0/911112711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcaa8003a00 msgr2=0x7fcac00aeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcaa8003a00 0x7fcac00aeed0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fcab002b490 tx=0x7fcab0004bb0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 -- 172.21.15.81:0/911112711 shutdown_connections 2024-06-29T14:41:58.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fcaac0763f0 0x7fcaac078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcac01545b0 0x7fcac0153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcac01e2150 0x7fcac01e0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 --2- 172.21.15.81:0/911112711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcaa8003a00 0x7fcac00aeed0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 -- 172.21.15.81:0/911112711 >> 172.21.15.81:0/911112711 conn(0x7fcac00d0930 msgr2=0x7fcac01583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:58.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 -- 172.21.15.81:0/911112711 shutdown_connections 2024-06-29T14:41:58.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.865+0000 7fcac6143640 1 -- 172.21.15.81:0/911112711 wait complete. 2024-06-29T14:41:58.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:41:58.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.990+0000 7f97d5cef640 1 -- 172.21.15.81:0/1332546145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d015e890 msgr2=0x7f97d015b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.990+0000 7f97d5cef640 1 --2- 172.21.15.81:0/1332546145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d015e890 0x7f97d015b9d0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f97c0005fd0 tx=0x7f97c002aa90 comp rx=0 tx=0).stop 2024-06-29T14:41:58.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.990+0000 7f97d5cef640 1 -- 172.21.15.81:0/1332546145 shutdown_connections 2024-06-29T14:41:58.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.990+0000 7f97d5cef640 1 --2- 172.21.15.81:0/1332546145 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97d00739b0 0x7f97d0140600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.990+0000 7f97d5cef640 1 --2- 172.21.15.81:0/1332546145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d015e890 0x7f97d015b9d0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.990+0000 7f97d5cef640 1 -- 172.21.15.81:0/1332546145 >> 172.21.15.81:0/1332546145 conn(0x7f97d00b1ce0 msgr2=0x7f97d00b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:58.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.990+0000 7f97d5cef640 1 -- 172.21.15.81:0/1332546145 shutdown_connections 2024-06-29T14:41:58.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.990+0000 7f97d5cef640 1 -- 172.21.15.81:0/1332546145 wait complete. 2024-06-29T14:41:58.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.992+0000 7f97d5cef640 1 Processor -- start 2024-06-29T14:41:58.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.992+0000 7f97d5cef640 1 -- start start 2024-06-29T14:41:58.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97d5cef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97d015bf10 0x7f97d0159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97d5cef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97d015e890 0x7f97d00c9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97d5cef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d00c9a80 0x7f97d00b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97d5cef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f97d0141290 con 0x7f97d00c9a80 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97d5cef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f97d0141110 con 0x7f97d015bf10 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97d5cef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f97d00ce550 con 0x7f97d015e890 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97cffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d00c9a80 0x7f97d00b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97cffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d00c9a80 0x7f97d00b2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35238/0 (socket says 172.21.15.81:35238) 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97cffff640 1 -- 172.21.15.81:0/3551937860 learned_addr learned my addr 172.21.15.81:0/3551937860 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97cf7fe640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97d015bf10 0x7f97d0159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.993+0000 7f97ceffd640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97d015e890 0x7f97d00c9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97cffff640 1 -- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97d015e890 msgr2=0x7f97d00c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97cffff640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97d015e890 0x7f97d00c9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97cffff640 1 -- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97d015bf10 msgr2=0x7f97d0159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:58.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97cffff640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97d015bf10 0x7f97d0159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:58.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97cffff640 1 -- 172.21.15.81:0/3551937860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d00ca290 con 0x7f97d00c9a80 2024-06-29T14:41:58.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97cffff640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d00c9a80 0x7f97d00b2f00 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f97c800d660 tx=0x7f97c800db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:58.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97b7fff640 1 -- 172.21.15.81:0/3551937860 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8017070 con 0x7f97d00c9a80 2024-06-29T14:41:58.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97b7fff640 1 -- 172.21.15.81:0/3551937860 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97c8005310 con 0x7f97d00c9a80 2024-06-29T14:41:58.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97d5cef640 1 -- 172.21.15.81:0/3551937860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d00ca570 con 0x7f97d00c9a80 2024-06-29T14:41:58.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97b7fff640 1 -- 172.21.15.81:0/3551937860 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8009820 con 0x7f97d00c9a80 2024-06-29T14:41:58.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.994+0000 7f97d5cef640 1 -- 172.21.15.81:0/3551937860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d00cab30 con 0x7f97d00c9a80 2024-06-29T14:41:58.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.995+0000 7f97d5cef640 1 -- 172.21.15.81:0/3551937860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d00c9fe0 con 0x7f97d00c9a80 2024-06-29T14:41:58.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.996+0000 7f97b7fff640 1 -- 172.21.15.81:0/3551937860 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f97c80099c0 con 0x7f97d00c9a80 2024-06-29T14:41:58.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.996+0000 7f97b7fff640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f97bc076420 0x7f97bc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:58.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.996+0000 7f97b7fff640 1 -- 172.21.15.81:0/3551937860 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 5809+0+0 (secure 0 0 0) 0x7f97c80979c0 con 0x7f97d00c9a80 2024-06-29T14:41:58.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.996+0000 7f97cf7fe640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f97bc076420 0x7f97bc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:58.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:58.997+0000 7f97cf7fe640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f97bc076420 0x7f97bc078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97c0035000 tx=0x7f97c0038040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:58.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.001+0000 7f97b7fff640 1 -- 172.21.15.81:0/3551937860 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f97c8099e20 con 0x7f97d00c9a80 2024-06-29T14:41:59.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.184+0000 7f97d5cef640 1 -- 172.21.15.81:0/3551937860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97d0140600 con 0x7f97d00c9a80 2024-06-29T14:41:59.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.185+0000 7f97b7fff640 1 -- 172.21.15.81:0/3551937860 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) ==== 74+0+16839 (secure 0 0 0) 0x7f97c8069080 con 0x7f97d00c9a80 2024-06-29T14:41:59.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:41:59.182 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":145,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:57.875501+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.170598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.829654+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:41:59.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 -- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f97bc076420 msgr2=0x7f97bc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:59.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f97bc076420 0x7f97bc078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97c0035000 tx=0x7f97c0038040 comp rx=0 tx=0).stop 2024-06-29T14:41:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 -- 172.21.15.81:0/3551937860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d00c9a80 msgr2=0x7f97d00b2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d00c9a80 0x7f97d00b2f00 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f97c800d660 tx=0x7f97c800db20 comp rx=0 tx=0).stop 2024-06-29T14:41:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 -- 172.21.15.81:0/3551937860 shutdown_connections 2024-06-29T14:41:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f97bc076420 0x7f97bc078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d00c9a80 0x7f97d00b2f00 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97d015e890 0x7f97d00c9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 --2- 172.21.15.81:0/3551937860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97d015bf10 0x7f97d0159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 -- 172.21.15.81:0/3551937860 >> 172.21.15.81:0/3551937860 conn(0x7f97d00b1ce0 msgr2=0x7f97d006f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 -- 172.21.15.81:0/3551937860 shutdown_connections 2024-06-29T14:41:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.187+0000 7f97d5cef640 1 -- 172.21.15.81:0/3551937860 wait complete. 2024-06-29T14:41:59.196 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T14:41:59.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.315+0000 7f82d76ce640 1 -- 172.21.15.81:0/3269114046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d00b57b0 msgr2=0x7f82d015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:59.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.315+0000 7f82d76ce640 1 --2- 172.21.15.81:0/3269114046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d00b57b0 0x7f82d015aed0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f82c0005fd0 tx=0x7f82c002a5d0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.315+0000 7f82d76ce640 1 -- 172.21.15.81:0/3269114046 shutdown_connections 2024-06-29T14:41:59.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.315+0000 7f82d76ce640 1 --2- 172.21.15.81:0/3269114046 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82d006ef90 0x7f82d006f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.315+0000 7f82d76ce640 1 --2- 172.21.15.81:0/3269114046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d00b57b0 0x7f82d015aed0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.315+0000 7f82d76ce640 1 -- 172.21.15.81:0/3269114046 >> 172.21.15.81:0/3269114046 conn(0x7f82d00d0930 msgr2=0x7f82d00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:41:59.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.315+0000 7f82d76ce640 1 -- 172.21.15.81:0/3269114046 shutdown_connections 2024-06-29T14:41:59.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.315+0000 7f82d76ce640 1 -- 172.21.15.81:0/3269114046 wait complete. 2024-06-29T14:41:59.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.316+0000 7f82d76ce640 1 Processor -- start 2024-06-29T14:41:59.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.316+0000 7f82d76ce640 1 -- start start 2024-06-29T14:41:59.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d76ce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d006ef90 0x7f82d0155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:59.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d76ce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82d00b57b0 0x7f82d0156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d76ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82d00ce1d0 0x7f82d01566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d76ce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82d00cecf0 con 0x7f82d006ef90 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d76ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f82d00b4b80 con 0x7f82d00ce1d0 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d76ce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f82d00cee70 con 0x7f82d00b57b0 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d546a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d006ef90 0x7f82d0155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d546a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d006ef90 0x7f82d0155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35260/0 (socket says 172.21.15.81:35260) 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d546a640 1 -- 172.21.15.81:0/1998133731 learned_addr learned my addr 172.21.15.81:0/1998133731 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d4c69640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82d00b57b0 0x7f82d0156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d5c6b640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82d00ce1d0 0x7f82d01566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d546a640 1 -- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82d00b57b0 msgr2=0x7f82d0156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d546a640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82d00b57b0 0x7f82d0156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d546a640 1 -- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82d00ce1d0 msgr2=0x7f82d01566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:41:59.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d546a640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82d00ce1d0 0x7f82d01566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:41:59.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d546a640 1 -- 172.21.15.81:0/1998133731 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d00afac0 con 0x7f82d006ef90 2024-06-29T14:41:59.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d546a640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d006ef90 0x7f82d0155410 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f82c0034040 tx=0x7f82c00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:59.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82cdffb640 1 -- 172.21.15.81:0/1998133731 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c003e030 con 0x7f82d006ef90 2024-06-29T14:41:59.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82cdffb640 1 -- 172.21.15.81:0/1998133731 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82c0003ca0 con 0x7f82d006ef90 2024-06-29T14:41:59.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.318+0000 7f82d76ce640 1 -- 172.21.15.81:0/1998133731 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d01f24f0 con 0x7f82d006ef90 2024-06-29T14:41:59.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.319+0000 7f82cdffb640 1 -- 172.21.15.81:0/1998133731 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c0035040 con 0x7f82d006ef90 2024-06-29T14:41:59.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.319+0000 7f82d76ce640 1 -- 172.21.15.81:0/1998133731 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d01f2ab0 con 0x7f82d006ef90 2024-06-29T14:41:59.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.320+0000 7f82d76ce640 1 -- 172.21.15.81:0/1998133731 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d006e2a0 con 0x7f82d006ef90 2024-06-29T14:41:59.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.320+0000 7f82cdffb640 1 -- 172.21.15.81:0/1998133731 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f82c0041030 con 0x7f82d006ef90 2024-06-29T14:41:59.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.321+0000 7f82cdffb640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f82bc0763f0 0x7f82bc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:41:59.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.321+0000 7f82cdffb640 1 -- 172.21.15.81:0/1998133731 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 5809+0+0 (secure 0 0 0) 0x7f82c00b7fa0 con 0x7f82d006ef90 2024-06-29T14:41:59.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.321+0000 7f82d4c69640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f82bc0763f0 0x7f82bc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:41:59.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.322+0000 7f82d4c69640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f82bc0763f0 0x7f82bc078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f82d01592e0 tx=0x7f82c400b2e0 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:41:59.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.325+0000 7f82cdffb640 1 -- 172.21.15.81:0/1998133731 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f82d006e2a0 con 0x7f82d006ef90 2024-06-29T14:41:59.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:41:59.546+0000 7f82d76ce640 1 -- 172.21.15.81:0/1998133731 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f82d006f3a0 con 0x7f82d006ef90 2024-06-29T14:42:00.490 INFO:tasks.ceph.mds.b.smithi167.stderr:2024-06-29T14:42:00.493+0000 7ff9d1d40640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T14:42:00.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.548+0000 7f82cdffb640 1 -- 172.21.15.81:0/1998133731 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) ==== 118+0+0 (secure 0 0 0) 0x7f82c0080f90 con 0x7f82d006ef90 2024-06-29T14:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 -- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f82bc0763f0 msgr2=0x7f82bc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f82bc0763f0 0x7f82bc078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f82d01592e0 tx=0x7f82c400b2e0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 -- 172.21.15.81:0/1998133731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d006ef90 msgr2=0x7f82d0155410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d006ef90 0x7f82d0155410 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f82c0034040 tx=0x7f82c00038a0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 -- 172.21.15.81:0/1998133731 shutdown_connections 2024-06-29T14:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f82bc0763f0 0x7f82bc078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82d00ce1d0 0x7f82d01566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82d00b57b0 0x7f82d0156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 --2- 172.21.15.81:0/1998133731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d006ef90 0x7f82d0155410 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 -- 172.21.15.81:0/1998133731 >> 172.21.15.81:0/1998133731 conn(0x7f82d00d0930 msgr2=0x7f82d01583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.551+0000 7f82d76ce640 1 -- 172.21.15.81:0/1998133731 shutdown_connections 2024-06-29T14:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.552+0000 7f82d76ce640 1 -- 172.21.15.81:0/1998133731 wait complete. 2024-06-29T14:42:00.563 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T14:42:00.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T14:42:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.688+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/2645322827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e080cd150 msgr2=0x7f0e08072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.688+0000 7f0e0ccf1640 1 --2- 172.21.15.81:0/2645322827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e080cd150 0x7f0e08072f40 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f0df8005fd0 tx=0x7f0df802aa00 comp rx=0 tx=0).stop 2024-06-29T14:42:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.688+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/2645322827 shutdown_connections 2024-06-29T14:42:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.688+0000 7f0e0ccf1640 1 --2- 172.21.15.81:0/2645322827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e08073480 0x7f0e0815b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.688+0000 7f0e0ccf1640 1 --2- 172.21.15.81:0/2645322827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e080cd150 0x7f0e08072f40 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.688+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/2645322827 >> 172.21.15.81:0/2645322827 conn(0x7f0e080b2920 msgr2=0x7f0e080b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.688+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/2645322827 shutdown_connections 2024-06-29T14:42:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.688+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/2645322827 wait complete. 2024-06-29T14:42:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.690+0000 7f0e0ccf1640 1 Processor -- start 2024-06-29T14:42:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.690+0000 7f0e0ccf1640 1 -- start start 2024-06-29T14:42:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e0ccf1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e08073480 0x7f0e0814eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e0ccf1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e080cd150 0x7f0e080d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e0ccf1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e0815bc20 0x7f0e081528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e0ccf1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0815c1b0 con 0x7f0e0815bc20 2024-06-29T14:42:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e0ccf1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0815c030 con 0x7f0e080cd150 2024-06-29T14:42:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e0ccf1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0e0815f060 con 0x7f0e08073480 2024-06-29T14:42:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e06d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e0815bc20 0x7f0e081528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e06d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e0815bc20 0x7f0e081528a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55654/0 (socket says 172.21.15.81:55654) 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e06d9d640 1 -- 172.21.15.81:0/140757355 learned_addr learned my addr 172.21.15.81:0/140757355 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e0659c640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e08073480 0x7f0e0814eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e05d9b640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e080cd150 0x7f0e080d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e06d9d640 1 -- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e08073480 msgr2=0x7f0e0814eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e06d9d640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e08073480 0x7f0e0814eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e06d9d640 1 -- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e080cd150 msgr2=0x7f0e080d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e06d9d640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e080cd150 0x7f0e080d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e06d9d640 1 -- 172.21.15.81:0/140757355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e080a9890 con 0x7f0e0815bc20 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e06d9d640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e0815bc20 0x7f0e081528a0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000c3c0 tx=0x7f0e0000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0df6ffd640 1 -- 172.21.15.81:0/140757355 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00017070 con 0x7f0e0815bc20 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0df6ffd640 1 -- 172.21.15.81:0/140757355 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e000053c0 con 0x7f0e0815bc20 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/140757355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e08150f00 con 0x7f0e0815bc20 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0df6ffd640 1 -- 172.21.15.81:0/140757355 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00003850 con 0x7f0e0815bc20 2024-06-29T14:42:00.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.692+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/140757355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e081514c0 con 0x7f0e0815bc20 2024-06-29T14:42:00.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.694+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/140757355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e080714d0 con 0x7f0e0815bc20 2024-06-29T14:42:00.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.694+0000 7f0df6ffd640 1 -- 172.21.15.81:0/140757355 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f0e0001a070 con 0x7f0e0815bc20 2024-06-29T14:42:00.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.695+0000 7f0df6ffd640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0de8076420 0x7f0de8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:00.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.695+0000 7f0df6ffd640 1 -- 172.21.15.81:0/140757355 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f0e00096b60 con 0x7f0e0815bc20 2024-06-29T14:42:00.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.695+0000 7f0e0659c640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0de8076420 0x7f0de8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:00.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.696+0000 7f0e0659c640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0de8076420 0x7f0de8078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0df8035000 tx=0x7f0df803b040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.699+0000 7f0df6ffd640 1 -- 172.21.15.81:0/140757355 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0e080714d0 con 0x7f0e0815bc20 2024-06-29T14:42:00.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:00.918+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/140757355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0e0815f580 con 0x7f0e0815bc20 2024-06-29T14:42:01.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.549+0000 7f0df6ffd640 1 -- 172.21.15.81:0/140757355 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v77) ==== 122+0+0 (secure 0 0 0) 0x7f0e08073890 con 0x7f0e0815bc20 2024-06-29T14:42:01.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0de8076420 msgr2=0x7f0de8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0de8076420 0x7f0de8078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0df8035000 tx=0x7f0df803b040 comp rx=0 tx=0).stop 2024-06-29T14:42:01.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/140757355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e0815bc20 msgr2=0x7f0e081528a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e0815bc20 0x7f0e081528a0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000c3c0 tx=0x7f0e0000c880 comp rx=0 tx=0).stop 2024-06-29T14:42:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/140757355 shutdown_connections 2024-06-29T14:42:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0de8076420 0x7f0de8078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e0815bc20 0x7f0e081528a0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e080cd150 0x7f0e080d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 --2- 172.21.15.81:0/140757355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e08073480 0x7f0e0814eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/140757355 >> 172.21.15.81:0/140757355 conn(0x7f0e080b2920 msgr2=0x7f0e0806ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/140757355 shutdown_connections 2024-06-29T14:42:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.553+0000 7f0e0ccf1640 1 -- 172.21.15.81:0/140757355 wait complete. 2024-06-29T14:42:01.563 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T14:42:01.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.680+0000 7f9038768640 1 -- 172.21.15.81:0/192075678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90180053a0 msgr2=0x7f901800ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.680+0000 7f9038768640 1 --2- 172.21.15.81:0/192075678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90180053a0 0x7f901800ba50 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f9028009e20 tx=0x7f9018005830 comp rx=0 tx=0).stop 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.680+0000 7f9038768640 1 -- 172.21.15.81:0/192075678 shutdown_connections 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.680+0000 7f9038768640 1 --2- 172.21.15.81:0/192075678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90180053a0 0x7f901800ba50 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.680+0000 7f9038768640 1 -- 172.21.15.81:0/192075678 >> 172.21.15.81:0/192075678 conn(0x7f90300d0930 msgr2=0x7f90300d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.680+0000 7f9038768640 1 -- 172.21.15.81:0/192075678 shutdown_connections 2024-06-29T14:42:01.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.680+0000 7f9038768640 1 -- 172.21.15.81:0/192075678 wait complete. 2024-06-29T14:42:01.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.682+0000 7f9038768640 1 Processor -- start 2024-06-29T14:42:01.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.682+0000 7f9038768640 1 -- start start 2024-06-29T14:42:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.683+0000 7f9038768640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90180053a0 0x7f90301483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.683+0000 7f9038768640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9030151700 0x7f9030153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9038768640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90301582a0 0x7f90301586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9038768640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f90300afce0 con 0x7f9030151700 2024-06-29T14:42:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9038768640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f90300afb60 con 0x7f90180053a0 2024-06-29T14:42:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9038768640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f90300afe60 con 0x7f90301582a0 2024-06-29T14:42:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9035d03640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9030151700 0x7f9030153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9035d03640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9030151700 0x7f9030153aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55664/0 (socket says 172.21.15.81:55664) 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9035d03640 1 -- 172.21.15.81:0/2634832782 learned_addr learned my addr 172.21.15.81:0/2634832782 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9036504640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90180053a0 0x7f90301483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9036d05640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90301582a0 0x7f90301586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9035d03640 1 -- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90301582a0 msgr2=0x7f90301586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9035d03640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90301582a0 0x7f90301586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9035d03640 1 -- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90180053a0 msgr2=0x7f90301483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9035d03640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90180053a0 0x7f90301483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9035d03640 1 -- 172.21.15.81:0/2634832782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9030148910 con 0x7f9030151700 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9036504640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90180053a0 0x7f90301483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9035d03640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9030151700 0x7f9030153aa0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f9024003e10 tx=0x7f9024001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f902effd640 1 -- 172.21.15.81:0/2634832782 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9024015070 con 0x7f9030151700 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f902effd640 1 -- 172.21.15.81:0/2634832782 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9024011840 con 0x7f9030151700 2024-06-29T14:42:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9038768640 1 -- 172.21.15.81:0/2634832782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9030153fe0 con 0x7f9030151700 2024-06-29T14:42:01.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f902effd640 1 -- 172.21.15.81:0/2634832782 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9024011c70 con 0x7f9030151700 2024-06-29T14:42:01.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.684+0000 7f9038768640 1 -- 172.21.15.81:0/2634832782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90301574e0 con 0x7f9030151700 2024-06-29T14:42:01.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.685+0000 7f9038768640 1 -- 172.21.15.81:0/2634832782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90300b2360 con 0x7f9030151700 2024-06-29T14:42:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.686+0000 7f902effd640 1 -- 172.21.15.81:0/2634832782 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f9024018070 con 0x7f9030151700 2024-06-29T14:42:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.686+0000 7f902effd640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f901c0763f0 0x7f901c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.687+0000 7f902effd640 1 -- 172.21.15.81:0/2634832782 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f90240985b0 con 0x7f9030151700 2024-06-29T14:42:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.687+0000 7f9036504640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f901c0763f0 0x7f901c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.687+0000 7f9036504640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f901c0763f0 0x7f901c078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9020000fb0 tx=0x7f9020001d10 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:01.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.691+0000 7f902effd640 1 -- 172.21.15.81:0/2634832782 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f90300b2360 con 0x7f9030151700 2024-06-29T14:42:01.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.910+0000 7f9038768640 1 -- 172.21.15.81:0/2634832782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9030156f90 con 0x7f9030151700 2024-06-29T14:42:01.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.910+0000 7f902effd640 1 -- 172.21.15.81:0/2634832782 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5231 (secure 0 0 0) 0x7f9030148910 con 0x7f9030151700 2024-06-29T14:42:01.908 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:42:01.908 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":77,"btime":"2024-06-29T14:42:01:549259+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/4068340499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":4068340499},{"type":"v1","addr":"172.21.15.81:6834","nonce":4068340499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1903134398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1903134398},{"type":"v1","addr":"172.21.15.167:6837","nonce":1903134398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1126258483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1126258483},{"type":"v1","addr":"172.21.15.81:6835","nonce":1126258483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.875457+0000","modified":"2024-06-29T14:42:00.919213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9111":{"gid":9111,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/4254103476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":4254103476},{"type":"v1","addr":"172.21.15.167:6836","nonce":4254103476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9111,"qdb_cluster":[9111]},"id":9}]} 2024-06-29T14:42:01.908 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 77 2024-06-29T14:42:01.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 -- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f901c0763f0 msgr2=0x7f901c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f901c0763f0 0x7f901c078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9020000fb0 tx=0x7f9020001d10 comp rx=0 tx=0).stop 2024-06-29T14:42:01.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 -- 172.21.15.81:0/2634832782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9030151700 msgr2=0x7f9030153aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:01.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9030151700 0x7f9030153aa0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f9024003e10 tx=0x7f9024001630 comp rx=0 tx=0).stop 2024-06-29T14:42:01.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 -- 172.21.15.81:0/2634832782 shutdown_connections 2024-06-29T14:42:01.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f901c0763f0 0x7f901c078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90301582a0 0x7f90301586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9030151700 0x7f9030153aa0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 --2- 172.21.15.81:0/2634832782 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f90180053a0 0x7f90301483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:01.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 -- 172.21.15.81:0/2634832782 >> 172.21.15.81:0/2634832782 conn(0x7f90300d0930 msgr2=0x7f9030152520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:01.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 -- 172.21.15.81:0/2634832782 shutdown_connections 2024-06-29T14:42:01.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:01.912+0000 7f9038768640 1 -- 172.21.15.81:0/2634832782 wait complete. 2024-06-29T14:42:01.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T14:42:02.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.036+0000 7f3ad63eb640 1 -- 172.21.15.81:0/3633965781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ad00cd480 msgr2=0x7f3ad00ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.036+0000 7f3ad63eb640 1 --2- 172.21.15.81:0/3633965781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ad00cd480 0x7f3ad00ce510 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4005fd0 tx=0x7f3ac402a540 comp rx=0 tx=0).stop 2024-06-29T14:42:02.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.037+0000 7f3ad63eb640 1 -- 172.21.15.81:0/3633965781 shutdown_connections 2024-06-29T14:42:02.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.037+0000 7f3ad63eb640 1 --2- 172.21.15.81:0/3633965781 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ad00cea50 0x7f3ad00b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.037+0000 7f3ad63eb640 1 --2- 172.21.15.81:0/3633965781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ad00cd480 0x7f3ad00ce510 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.037+0000 7f3ad63eb640 1 -- 172.21.15.81:0/3633965781 >> 172.21.15.81:0/3633965781 conn(0x7f3ad00b3510 msgr2=0x7f3ad00b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.037+0000 7f3ad63eb640 1 -- 172.21.15.81:0/3633965781 shutdown_connections 2024-06-29T14:42:02.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.037+0000 7f3ad63eb640 1 -- 172.21.15.81:0/3633965781 wait complete. 2024-06-29T14:42:02.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.038+0000 7f3ad63eb640 1 Processor -- start 2024-06-29T14:42:02.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.038+0000 7f3ad63eb640 1 -- start start 2024-06-29T14:42:02.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3ad63eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ad00b6eb0 0x7f3ad015f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3ad63eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ad00cd480 0x7f3ad009bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3ad63eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ad00cea50 0x7f3ad0159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3ad63eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad00b50e0 con 0x7f3ad00cd480 2024-06-29T14:42:02.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3ad63eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad00b4f60 con 0x7f3ad00b6eb0 2024-06-29T14:42:02.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3ad63eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3ad00b5260 con 0x7f3ad00cea50 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3acf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ad00cd480 0x7f3ad009bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3acf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ad00cd480 0x7f3ad009bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55678/0 (socket says 172.21.15.81:55678) 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3acf7fe640 1 -- 172.21.15.81:0/996756101 learned_addr learned my addr 172.21.15.81:0/996756101 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3acffff640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ad00b6eb0 0x7f3ad015f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3ad4988640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ad00cea50 0x7f3ad0159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3acf7fe640 1 -- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ad00cea50 msgr2=0x7f3ad0159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3acf7fe640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ad00cea50 0x7f3ad0159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3acf7fe640 1 -- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ad00b6eb0 msgr2=0x7f3ad015f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3acf7fe640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ad00b6eb0 0x7f3ad015f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3acf7fe640 1 -- 172.21.15.81:0/996756101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad00aedd0 con 0x7f3ad00cd480 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3acf7fe640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ad00cd480 0x7f3ad009bc70 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4006010 tx=0x7f3ac40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.040+0000 7f3accff9640 1 -- 172.21.15.81:0/996756101 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac403e030 con 0x7f3ad00cd480 2024-06-29T14:42:02.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.041+0000 7f3accff9640 1 -- 172.21.15.81:0/996756101 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ac4003d90 con 0x7f3ad00cd480 2024-06-29T14:42:02.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.041+0000 7f3ad63eb640 1 -- 172.21.15.81:0/996756101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad00af0b0 con 0x7f3ad00cd480 2024-06-29T14:42:02.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.041+0000 7f3accff9640 1 -- 172.21.15.81:0/996756101 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4035040 con 0x7f3ad00cd480 2024-06-29T14:42:02.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.041+0000 7f3ad63eb640 1 -- 172.21.15.81:0/996756101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad00b0880 con 0x7f3ad00cd480 2024-06-29T14:42:02.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.041+0000 7f3ad63eb640 1 -- 172.21.15.81:0/996756101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ad00af5d0 con 0x7f3ad00cd480 2024-06-29T14:42:02.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.042+0000 7f3accff9640 1 -- 172.21.15.81:0/996756101 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f3ac4041030 con 0x7f3ad00cd480 2024-06-29T14:42:02.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.043+0000 7f3accff9640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f3abc0763f0 0x7f3abc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.043+0000 7f3accff9640 1 -- 172.21.15.81:0/996756101 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f3ac40b8080 con 0x7f3ad00cd480 2024-06-29T14:42:02.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.043+0000 7f3acffff640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f3abc0763f0 0x7f3abc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.043+0000 7f3acffff640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f3abc0763f0 0x7f3abc078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ad00cdb60 tx=0x7f3ac0009040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.047+0000 7f3accff9640 1 -- 172.21.15.81:0/996756101 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3ad00af5d0 con 0x7f3ad00cd480 2024-06-29T14:42:02.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.228+0000 7f3ad63eb640 1 -- 172.21.15.81:0/996756101 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ad00ce510 con 0x7f3ad00cd480 2024-06-29T14:42:02.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.229+0000 7f3accff9640 1 -- 172.21.15.81:0/996756101 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) ==== 74+0+16826 (secure 0 0 0) 0x7f3ac4081100 con 0x7f3ad00cd480 2024-06-29T14:42:02.227 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:42:02.227 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":146,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T14:41:59.514922+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.170598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.829654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 -- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f3abc0763f0 msgr2=0x7f3abc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f3abc0763f0 0x7f3abc078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ad00cdb60 tx=0x7f3ac0009040 comp rx=0 tx=0).stop 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 -- 172.21.15.81:0/996756101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ad00cd480 msgr2=0x7f3ad009bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ad00cd480 0x7f3ad009bc70 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4006010 tx=0x7f3ac40038a0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 -- 172.21.15.81:0/996756101 shutdown_connections 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f3abc0763f0 0x7f3abc078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ad00cea50 0x7f3ad0159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ad00cd480 0x7f3ad009bc70 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 --2- 172.21.15.81:0/996756101 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ad00b6eb0 0x7f3ad015f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 -- 172.21.15.81:0/996756101 >> 172.21.15.81:0/996756101 conn(0x7f3ad00b3510 msgr2=0x7f3ad0070570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 -- 172.21.15.81:0/996756101 shutdown_connections 2024-06-29T14:42:02.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.231+0000 7f3ad63eb640 1 -- 172.21.15.81:0/996756101 wait complete. 2024-06-29T14:42:02.240 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T14:42:02.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:42:02.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.356+0000 7f0ec598c640 1 -- 172.21.15.81:0/2113780750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec00b5250 msgr2=0x7f0ec00d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.356+0000 7f0ec598c640 1 --2- 172.21.15.81:0/2113780750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec00b5250 0x7f0ec00d1fc0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8005fd0 tx=0x7f0eb802a960 comp rx=0 tx=0).stop 2024-06-29T14:42:02.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.356+0000 7f0ec598c640 1 -- 172.21.15.81:0/2113780750 shutdown_connections 2024-06-29T14:42:02.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.356+0000 7f0ec598c640 1 --2- 172.21.15.81:0/2113780750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec00b5250 0x7f0ec00d1fc0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.356+0000 7f0ec598c640 1 --2- 172.21.15.81:0/2113780750 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ec00d0b90 0x7f0ec00d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.356+0000 7f0ec598c640 1 -- 172.21.15.81:0/2113780750 >> 172.21.15.81:0/2113780750 conn(0x7f0ec009cf30 msgr2=0x7f0ec00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.356+0000 7f0ec598c640 1 -- 172.21.15.81:0/2113780750 shutdown_connections 2024-06-29T14:42:02.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.356+0000 7f0ec598c640 1 -- 172.21.15.81:0/2113780750 wait complete. 2024-06-29T14:42:02.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.358+0000 7f0ec598c640 1 Processor -- start 2024-06-29T14:42:02.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.358+0000 7f0ec598c640 1 -- start start 2024-06-29T14:42:02.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.359+0000 7f0ec598c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ec00b5250 0x7f0ec00cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.359+0000 7f0ec598c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ec00b9040 0x7f0ec00a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.359+0000 7f0ec598c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec0154580 0x7f0ec00a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ec598c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec00d2680 con 0x7f0ec0154580 2024-06-29T14:42:02.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ec598c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec00d2500 con 0x7f0ec00b9040 2024-06-29T14:42:02.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ec598c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec00d2800 con 0x7f0ec00b5250 2024-06-29T14:42:02.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec0154580 0x7f0ec00a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec0154580 0x7f0ec00a0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55692/0 (socket says 172.21.15.81:55692) 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebf7fe640 1 -- 172.21.15.81:0/2833237774 learned_addr learned my addr 172.21.15.81:0/2833237774 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebeffd640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ec00b5250 0x7f0ec00cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebe7fc640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ec00b9040 0x7f0ec00a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebf7fe640 1 -- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ec00b5250 msgr2=0x7f0ec00cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebf7fe640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ec00b5250 0x7f0ec00cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebf7fe640 1 -- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ec00b9040 msgr2=0x7f0ec00a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebf7fe640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ec00b9040 0x7f0ec00a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebf7fe640 1 -- 172.21.15.81:0/2833237774 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec0151a50 con 0x7f0ec0154580 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ebf7fe640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec0154580 0x7f0ec00a0f50 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009cd0 tx=0x7f0eb8032bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ea7fff640 1 -- 172.21.15.81:0/2833237774 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8030ae0 con 0x7f0ec0154580 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ea7fff640 1 -- 172.21.15.81:0/2833237774 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0eb803e070 con 0x7f0ec0154580 2024-06-29T14:42:02.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ec598c640 1 -- 172.21.15.81:0/2833237774 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec0151d30 con 0x7f0ec0154580 2024-06-29T14:42:02.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ea7fff640 1 -- 172.21.15.81:0/2833237774 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8032070 con 0x7f0ec0154580 2024-06-29T14:42:02.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.360+0000 7f0ec598c640 1 -- 172.21.15.81:0/2833237774 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec01522f0 con 0x7f0ec0154580 2024-06-29T14:42:02.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.362+0000 7f0ec598c640 1 -- 172.21.15.81:0/2833237774 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ec00a1890 con 0x7f0ec0154580 2024-06-29T14:42:02.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.362+0000 7f0ea7fff640 1 -- 172.21.15.81:0/2833237774 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f0eb8032210 con 0x7f0ec0154580 2024-06-29T14:42:02.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.362+0000 7f0ea7fff640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0eac076420 0x7f0eac078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.363+0000 7f0ea7fff640 1 -- 172.21.15.81:0/2833237774 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f0eb80b93a0 con 0x7f0ec0154580 2024-06-29T14:42:02.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.363+0000 7f0ebeffd640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0eac076420 0x7f0eac078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.363+0000 7f0ebeffd640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0eac076420 0x7f0eac078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0007fd0 tx=0x7f0eb0009040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.367+0000 7f0ea7fff640 1 -- 172.21.15.81:0/2833237774 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0eb8082b80 con 0x7f0ec0154580 2024-06-29T14:42:02.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.549+0000 7f0ec598c640 1 -- 172.21.15.81:0/2833237774 --> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ec00d2d90 con 0x7f0eac076420 2024-06-29T14:42:02.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.554+0000 7f0ea7fff640 1 -- 172.21.15.81:0/2833237774 <== mgr.7223 v2:172.21.15.167:6832/3201324170 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252478 (secure 0 0 0) 0x7f0ec00d2d90 con 0x7f0eac076420 2024-06-29T14:42:02.555 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:42:02.556 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:42:02.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.557+0000 7f0ec598c640 1 -- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0eac076420 msgr2=0x7f0eac078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.557+0000 7f0ec598c640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0eac076420 0x7f0eac078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0007fd0 tx=0x7f0eb0009040 comp rx=0 tx=0).stop 2024-06-29T14:42:02.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.557+0000 7f0ec598c640 1 -- 172.21.15.81:0/2833237774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec0154580 msgr2=0x7f0ec00a0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.557+0000 7f0ec598c640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec0154580 0x7f0ec00a0f50 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009cd0 tx=0x7f0eb8032bb0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.557+0000 7f0ec598c640 1 -- 172.21.15.81:0/2833237774 shutdown_connections 2024-06-29T14:42:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.557+0000 7f0ec598c640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0eac076420 0x7f0eac078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.557+0000 7f0ec598c640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec0154580 0x7f0ec00a0f50 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.557+0000 7f0ec598c640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ec00b9040 0x7f0ec00a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.557+0000 7f0ec598c640 1 --2- 172.21.15.81:0/2833237774 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ec00b5250 0x7f0ec00cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.557+0000 7f0ec598c640 1 -- 172.21.15.81:0/2833237774 >> 172.21.15.81:0/2833237774 conn(0x7f0ec009cf30 msgr2=0x7f0ec006d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.558+0000 7f0ec598c640 1 -- 172.21.15.81:0/2833237774 shutdown_connections 2024-06-29T14:42:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.558+0000 7f0ec598c640 1 -- 172.21.15.81:0/2833237774 wait complete. 2024-06-29T14:42:02.566 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:42:02.066783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":289,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":352256,"kb_used_data":15360,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754622464,"statfs":{"total":773094113280,"available":772733403136,"internally_reserved":0,"allocated":15728640,"data_stored":8154924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":344837682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.567702"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518915+0000","last_change":"2024-06-29T14:41:58.546002+0000","last_active":"2024-06-29T14:41:59.518915+0000","last_peered":"2024-06-29T14:41:59.518915+0000","last_clean":"2024-06-29T14:41:59.518915+0000","last_became_active":"2024-06-29T14:41:58.545684+0000","last_became_peered":"2024-06-29T14:41:58.545684+0000","last_unstale":"2024-06-29T14:41:59.518915+0000","last_undegraded":"2024-06-29T14:41:59.518915+0000","last_fullsized":"2024-06-29T14:41:59.518915+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:26.344464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882035+0000","last_change":"2024-06-29T14:41:58.541620+0000","last_active":"2024-06-29T14:41:59.882035+0000","last_peered":"2024-06-29T14:41:59.882035+0000","last_clean":"2024-06-29T14:41:59.882035+0000","last_became_active":"2024-06-29T14:41:58.541312+0000","last_became_peered":"2024-06-29T14:41:58.541312+0000","last_unstale":"2024-06-29T14:41:59.882035+0000","last_undegraded":"2024-06-29T14:41:59.882035+0000","last_fullsized":"2024-06-29T14:41:59.882035+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:03.210487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883481+0000","last_change":"2024-06-29T14:41:58.535498+0000","last_active":"2024-06-29T14:41:59.883481+0000","last_peered":"2024-06-29T14:41:59.883481+0000","last_clean":"2024-06-29T14:41:59.883481+0000","last_became_active":"2024-06-29T14:41:58.534977+0000","last_became_peered":"2024-06-29T14:41:58.534977+0000","last_unstale":"2024-06-29T14:41:59.883481+0000","last_undegraded":"2024-06-29T14:41:59.883481+0000","last_fullsized":"2024-06-29T14:41:59.883481+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:38.605946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518547+0000","last_change":"2024-06-29T14:41:58.538521+0000","last_active":"2024-06-29T14:41:59.518547+0000","last_peered":"2024-06-29T14:41:59.518547+0000","last_clean":"2024-06-29T14:41:59.518547+0000","last_became_active":"2024-06-29T14:41:58.538166+0000","last_became_peered":"2024-06-29T14:41:58.538166+0000","last_unstale":"2024-06-29T14:41:59.518547+0000","last_undegraded":"2024-06-29T14:41:59.518547+0000","last_fullsized":"2024-06-29T14:41:59.518547+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:42.669496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883054+0000","last_change":"2024-06-29T14:41:58.545829+0000","last_active":"2024-06-29T14:41:59.883054+0000","last_peered":"2024-06-29T14:41:59.883054+0000","last_clean":"2024-06-29T14:41:59.883054+0000","last_became_active":"2024-06-29T14:41:58.545408+0000","last_became_peered":"2024-06-29T14:41:58.545408+0000","last_unstale":"2024-06-29T14:41:59.883054+0000","last_undegraded":"2024-06-29T14:41:59.883054+0000","last_fullsized":"2024-06-29T14:41:59.883054+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:53:05.168790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520450+0000","last_change":"2024-06-29T14:41:58.535159+0000","last_active":"2024-06-29T14:41:59.520450+0000","last_peered":"2024-06-29T14:41:59.520450+0000","last_clean":"2024-06-29T14:41:59.520450+0000","last_became_active":"2024-06-29T14:41:58.534835+0000","last_became_peered":"2024-06-29T14:41:58.534835+0000","last_unstale":"2024-06-29T14:41:59.520450+0000","last_undegraded":"2024-06-29T14:41:59.520450+0000","last_fullsized":"2024-06-29T14:41:59.520450+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:05.790297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.519900+0000","last_change":"2024-06-29T14:41:58.545975+0000","last_active":"2024-06-29T14:41:59.519900+0000","last_peered":"2024-06-29T14:41:59.519900+0000","last_clean":"2024-06-29T14:41:59.519900+0000","last_became_active":"2024-06-29T14:41:58.545554+0000","last_became_peered":"2024-06-29T14:41:58.545554+0000","last_unstale":"2024-06-29T14:41:59.519900+0000","last_undegraded":"2024-06-29T14:41:59.519900+0000","last_fullsized":"2024-06-29T14:41:59.519900+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:30.175774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882512+0000","last_change":"2024-06-29T14:41:58.533534+0000","last_active":"2024-06-29T14:41:59.882512+0000","last_peered":"2024-06-29T14:41:59.882512+0000","last_clean":"2024-06-29T14:41:59.882512+0000","last_became_active":"2024-06-29T14:41:58.533125+0000","last_became_peered":"2024-06-29T14:41:58.533125+0000","last_unstale":"2024-06-29T14:41:59.882512+0000","last_undegraded":"2024-06-29T14:41:59.882512+0000","last_fullsized":"2024-06-29T14:41:59.882512+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:32.024937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883570+0000","last_change":"2024-06-29T14:41:58.537489+0000","last_active":"2024-06-29T14:41:59.883570+0000","last_peered":"2024-06-29T14:41:59.883570+0000","last_clean":"2024-06-29T14:41:59.883570+0000","last_became_active":"2024-06-29T14:41:58.537202+0000","last_became_peered":"2024-06-29T14:41:58.537202+0000","last_unstale":"2024-06-29T14:41:59.883570+0000","last_undegraded":"2024-06-29T14:41:59.883570+0000","last_fullsized":"2024-06-29T14:41:59.883570+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:40.903285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882132+0000","last_change":"2024-06-29T14:41:58.541482+0000","last_active":"2024-06-29T14:41:59.882132+0000","last_peered":"2024-06-29T14:41:59.882132+0000","last_clean":"2024-06-29T14:41:59.882132+0000","last_became_active":"2024-06-29T14:41:58.541169+0000","last_became_peered":"2024-06-29T14:41:58.541169+0000","last_unstale":"2024-06-29T14:41:59.882132+0000","last_undegraded":"2024-06-29T14:41:59.882132+0000","last_fullsized":"2024-06-29T14:41:59.882132+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:04:22.876673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882309+0000","last_change":"2024-06-29T14:41:58.534856+0000","last_active":"2024-06-29T14:41:59.882309+0000","last_peered":"2024-06-29T14:41:59.882309+0000","last_clean":"2024-06-29T14:41:59.882309+0000","last_became_active":"2024-06-29T14:41:58.534523+0000","last_became_peered":"2024-06-29T14:41:58.534523+0000","last_unstale":"2024-06-29T14:41:59.882309+0000","last_undegraded":"2024-06-29T14:41:59.882309+0000","last_fullsized":"2024-06-29T14:41:59.882309+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:46.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883320+0000","last_change":"2024-06-29T14:41:58.539286+0000","last_active":"2024-06-29T14:41:59.883320+0000","last_peered":"2024-06-29T14:41:59.883320+0000","last_clean":"2024-06-29T14:41:59.883320+0000","last_became_active":"2024-06-29T14:41:58.538908+0000","last_became_peered":"2024-06-29T14:41:58.538908+0000","last_unstale":"2024-06-29T14:41:59.883320+0000","last_undegraded":"2024-06-29T14:41:59.883320+0000","last_fullsized":"2024-06-29T14:41:59.883320+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:32.367186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518569+0000","last_change":"2024-06-29T14:41:58.546260+0000","last_active":"2024-06-29T14:41:59.518569+0000","last_peered":"2024-06-29T14:41:59.518569+0000","last_clean":"2024-06-29T14:41:59.518569+0000","last_became_active":"2024-06-29T14:41:58.545761+0000","last_became_peered":"2024-06-29T14:41:58.545761+0000","last_unstale":"2024-06-29T14:41:59.518569+0000","last_undegraded":"2024-06-29T14:41:59.518569+0000","last_fullsized":"2024-06-29T14:41:59.518569+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:59.682693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882464+0000","last_change":"2024-06-29T14:41:58.526825+0000","last_active":"2024-06-29T14:41:59.882464+0000","last_peered":"2024-06-29T14:41:59.882464+0000","last_clean":"2024-06-29T14:41:59.882464+0000","last_became_active":"2024-06-29T14:41:58.525395+0000","last_became_peered":"2024-06-29T14:41:58.525395+0000","last_unstale":"2024-06-29T14:41:59.882464+0000","last_undegraded":"2024-06-29T14:41:59.882464+0000","last_fullsized":"2024-06-29T14:41:59.882464+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:47.467202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518227+0000","last_change":"2024-06-29T14:41:58.534385+0000","last_active":"2024-06-29T14:41:59.518227+0000","last_peered":"2024-06-29T14:41:59.518227+0000","last_clean":"2024-06-29T14:41:59.518227+0000","last_became_active":"2024-06-29T14:41:58.534050+0000","last_became_peered":"2024-06-29T14:41:58.534050+0000","last_unstale":"2024-06-29T14:41:59.518227+0000","last_undegraded":"2024-06-29T14:41:59.518227+0000","last_fullsized":"2024-06-29T14:41:59.518227+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:47:09.101548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882938+0000","last_change":"2024-06-29T14:41:58.527469+0000","last_active":"2024-06-29T14:41:59.882938+0000","last_peered":"2024-06-29T14:41:59.882938+0000","last_clean":"2024-06-29T14:41:59.882938+0000","last_became_active":"2024-06-29T14:41:58.527150+0000","last_became_peered":"2024-06-29T14:41:58.527150+0000","last_unstale":"2024-06-29T14:41:59.882938+0000","last_undegraded":"2024-06-29T14:41:59.882938+0000","last_fullsized":"2024-06-29T14:41:59.882938+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:05.186180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520031+0000","last_change":"2024-06-29T14:41:58.540443+0000","last_active":"2024-06-29T14:41:59.520031+0000","last_peered":"2024-06-29T14:41:59.520031+0000","last_clean":"2024-06-29T14:41:59.520031+0000","last_became_active":"2024-06-29T14:41:58.540085+0000","last_became_peered":"2024-06-29T14:41:58.540085+0000","last_unstale":"2024-06-29T14:41:59.520031+0000","last_undegraded":"2024-06-29T14:41:59.520031+0000","last_fullsized":"2024-06-29T14:41:59.520031+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:01.129268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882745+0000","last_change":"2024-06-29T14:41:58.540558+0000","last_active":"2024-06-29T14:41:59.882745+0000","last_peered":"2024-06-29T14:41:59.882745+0000","last_clean":"2024-06-29T14:41:59.882745+0000","last_became_active":"2024-06-29T14:41:58.540258+0000","last_became_peered":"2024-06-29T14:41:58.540258+0000","last_unstale":"2024-06-29T14:41:59.882745+0000","last_undegraded":"2024-06-29T14:41:59.882745+0000","last_fullsized":"2024-06-29T14:41:59.882745+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:06.432760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882473+0000","last_change":"2024-06-29T14:41:58.544350+0000","last_active":"2024-06-29T14:41:59.882473+0000","last_peered":"2024-06-29T14:41:59.882473+0000","last_clean":"2024-06-29T14:41:59.882473+0000","last_became_active":"2024-06-29T14:41:58.544036+0000","last_became_peered":"2024-06-29T14:41:58.544036+0000","last_unstale":"2024-06-29T14:41:59.882473+0000","last_undegraded":"2024-06-29T14:41:59.882473+0000","last_fullsized":"2024-06-29T14:41:59.882473+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:19.408347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882362+0000","last_change":"2024-06-29T14:41:58.528180+0000","last_active":"2024-06-29T14:41:59.882362+0000","last_peered":"2024-06-29T14:41:59.882362+0000","last_clean":"2024-06-29T14:41:59.882362+0000","last_became_active":"2024-06-29T14:41:58.527870+0000","last_became_peered":"2024-06-29T14:41:58.527870+0000","last_unstale":"2024-06-29T14:41:59.882362+0000","last_undegraded":"2024-06-29T14:41:59.882362+0000","last_fullsized":"2024-06-29T14:41:59.882362+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:25.349931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883369+0000","last_change":"2024-06-29T14:41:58.544672+0000","last_active":"2024-06-29T14:41:59.883369+0000","last_peered":"2024-06-29T14:41:59.883369+0000","last_clean":"2024-06-29T14:41:59.883369+0000","last_became_active":"2024-06-29T14:41:58.544367+0000","last_became_peered":"2024-06-29T14:41:58.544367+0000","last_unstale":"2024-06-29T14:41:59.883369+0000","last_undegraded":"2024-06-29T14:41:59.883369+0000","last_fullsized":"2024-06-29T14:41:59.883369+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:45:32.977469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882143+0000","last_change":"2024-06-29T14:41:58.536979+0000","last_active":"2024-06-29T14:41:59.882143+0000","last_peered":"2024-06-29T14:41:59.882143+0000","last_clean":"2024-06-29T14:41:59.882143+0000","last_became_active":"2024-06-29T14:41:58.536678+0000","last_became_peered":"2024-06-29T14:41:58.536678+0000","last_unstale":"2024-06-29T14:41:59.882143+0000","last_undegraded":"2024-06-29T14:41:59.882143+0000","last_fullsized":"2024-06-29T14:41:59.882143+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:01.757137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883378+0000","last_change":"2024-06-29T14:41:58.536594+0000","last_active":"2024-06-29T14:41:59.883378+0000","last_peered":"2024-06-29T14:41:59.883378+0000","last_clean":"2024-06-29T14:41:59.883378+0000","last_became_active":"2024-06-29T14:41:58.536247+0000","last_became_peered":"2024-06-29T14:41:58.536247+0000","last_unstale":"2024-06-29T14:41:59.883378+0000","last_undegraded":"2024-06-29T14:41:59.883378+0000","last_fullsized":"2024-06-29T14:41:59.883378+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:00.046159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882675+0000","last_change":"2024-06-29T14:41:58.539607+0000","last_active":"2024-06-29T14:41:59.882675+0000","last_peered":"2024-06-29T14:41:59.882675+0000","last_clean":"2024-06-29T14:41:59.882675+0000","last_became_active":"2024-06-29T14:41:58.539306+0000","last_became_peered":"2024-06-29T14:41:58.539306+0000","last_unstale":"2024-06-29T14:41:59.882675+0000","last_undegraded":"2024-06-29T14:41:59.882675+0000","last_fullsized":"2024-06-29T14:41:59.882675+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:17:42.624622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882211+0000","last_change":"2024-06-29T14:41:58.539688+0000","last_active":"2024-06-29T14:41:59.882211+0000","last_peered":"2024-06-29T14:41:59.882211+0000","last_clean":"2024-06-29T14:41:59.882211+0000","last_became_active":"2024-06-29T14:41:58.539379+0000","last_became_peered":"2024-06-29T14:41:58.539379+0000","last_unstale":"2024-06-29T14:41:59.882211+0000","last_undegraded":"2024-06-29T14:41:59.882211+0000","last_fullsized":"2024-06-29T14:41:59.882211+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:26.344464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882932+0000","last_change":"2024-06-29T14:41:58.528088+0000","last_active":"2024-06-29T14:41:59.882932+0000","last_peered":"2024-06-29T14:41:59.882932+0000","last_clean":"2024-06-29T14:41:59.882932+0000","last_became_active":"2024-06-29T14:41:58.527798+0000","last_became_peered":"2024-06-29T14:41:58.527798+0000","last_unstale":"2024-06-29T14:41:59.882932+0000","last_undegraded":"2024-06-29T14:41:59.882932+0000","last_fullsized":"2024-06-29T14:41:59.882932+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:53:15.415717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518388+0000","last_change":"2024-06-29T14:41:58.546508+0000","last_active":"2024-06-29T14:41:59.518388+0000","last_peered":"2024-06-29T14:41:59.518388+0000","last_clean":"2024-06-29T14:41:59.518388+0000","last_became_active":"2024-06-29T14:41:58.546065+0000","last_became_peered":"2024-06-29T14:41:58.546065+0000","last_unstale":"2024-06-29T14:41:59.518388+0000","last_undegraded":"2024-06-29T14:41:59.518388+0000","last_fullsized":"2024-06-29T14:41:59.518388+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:06:15.068591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.519965+0000","last_change":"2024-06-29T14:41:58.527897+0000","last_active":"2024-06-29T14:41:59.519965+0000","last_peered":"2024-06-29T14:41:59.519965+0000","last_clean":"2024-06-29T14:41:59.519965+0000","last_became_active":"2024-06-29T14:41:58.527625+0000","last_became_peered":"2024-06-29T14:41:58.527625+0000","last_unstale":"2024-06-29T14:41:59.519965+0000","last_undegraded":"2024-06-29T14:41:59.519965+0000","last_fullsized":"2024-06-29T14:41:59.519965+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:46.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518589+0000","last_change":"2024-06-29T14:41:58.533164+0000","last_active":"2024-06-29T14:41:59.518589+0000","last_peered":"2024-06-29T14:41:59.518589+0000","last_clean":"2024-06-29T14:41:59.518589+0000","last_became_active":"2024-06-29T14:41:58.532865+0000","last_became_peered":"2024-06-29T14:41:58.532865+0000","last_unstale":"2024-06-29T14:41:59.518589+0000","last_undegraded":"2024-06-29T14:41:59.518589+0000","last_fullsized":"2024-06-29T14:41:59.518589+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:53:04.047421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518236+0000","last_change":"2024-06-29T14:41:58.540722+0000","last_active":"2024-06-29T14:41:59.518236+0000","last_peered":"2024-06-29T14:41:59.518236+0000","last_clean":"2024-06-29T14:41:59.518236+0000","last_became_active":"2024-06-29T14:41:58.540393+0000","last_became_peered":"2024-06-29T14:41:58.540393+0000","last_unstale":"2024-06-29T14:41:59.518236+0000","last_undegraded":"2024-06-29T14:41:59.518236+0000","last_fullsized":"2024-06-29T14:41:59.518236+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:01.707346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882530+0000","last_change":"2024-06-29T14:41:58.544502+0000","last_active":"2024-06-29T14:41:59.882530+0000","last_peered":"2024-06-29T14:41:59.882530+0000","last_clean":"2024-06-29T14:41:59.882530+0000","last_became_active":"2024-06-29T14:41:58.543891+0000","last_became_peered":"2024-06-29T14:41:58.543891+0000","last_unstale":"2024-06-29T14:41:59.882530+0000","last_undegraded":"2024-06-29T14:41:59.882530+0000","last_fullsized":"2024-06-29T14:41:59.882530+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:34.239368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.519730+0000","last_change":"2024-06-29T14:41:58.536757+0000","last_active":"2024-06-29T14:41:59.519730+0000","last_peered":"2024-06-29T14:41:59.519730+0000","last_clean":"2024-06-29T14:41:59.519730+0000","last_became_active":"2024-06-29T14:41:58.536424+0000","last_became_peered":"2024-06-29T14:41:58.536424+0000","last_unstale":"2024-06-29T14:41:59.519730+0000","last_undegraded":"2024-06-29T14:41:59.519730+0000","last_fullsized":"2024-06-29T14:41:59.519730+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:08:09.235128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883232+0000","last_change":"2024-06-29T14:41:58.535130+0000","last_active":"2024-06-29T14:41:59.883232+0000","last_peered":"2024-06-29T14:41:59.883232+0000","last_clean":"2024-06-29T14:41:59.883232+0000","last_became_active":"2024-06-29T14:41:58.534791+0000","last_became_peered":"2024-06-29T14:41:58.534791+0000","last_unstale":"2024-06-29T14:41:59.883232+0000","last_undegraded":"2024-06-29T14:41:59.883232+0000","last_fullsized":"2024-06-29T14:41:59.883232+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:16.109457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"145'4","reported_seq":18,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883055+0000","last_change":"2024-06-29T14:41:57.526059+0000","last_active":"2024-06-29T14:41:59.883055+0000","last_peered":"2024-06-29T14:41:59.883055+0000","last_clean":"2024-06-29T14:41:59.883055+0000","last_became_active":"2024-06-29T14:41:57.525685+0000","last_became_peered":"2024-06-29T14:41:57.525685+0000","last_unstale":"2024-06-29T14:41:59.883055+0000","last_undegraded":"2024-06-29T14:41:59.883055+0000","last_fullsized":"2024-06-29T14:41:59.883055+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:23.584362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883661+0000","last_change":"2024-06-29T14:41:58.532823+0000","last_active":"2024-06-29T14:41:59.883661+0000","last_peered":"2024-06-29T14:41:59.883661+0000","last_clean":"2024-06-29T14:41:59.883661+0000","last_became_active":"2024-06-29T14:41:58.532491+0000","last_became_peered":"2024-06-29T14:41:58.532491+0000","last_unstale":"2024-06-29T14:41:59.883661+0000","last_undegraded":"2024-06-29T14:41:59.883661+0000","last_fullsized":"2024-06-29T14:41:59.883661+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:38.363172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"145'1","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520799+0000","last_change":"2024-06-29T14:41:57.519722+0000","last_active":"2024-06-29T14:41:59.520799+0000","last_peered":"2024-06-29T14:41:59.520799+0000","last_clean":"2024-06-29T14:41:59.520799+0000","last_became_active":"2024-06-29T14:41:57.518971+0000","last_became_peered":"2024-06-29T14:41:57.518971+0000","last_unstale":"2024-06-29T14:41:59.520799+0000","last_undegraded":"2024-06-29T14:41:59.520799+0000","last_fullsized":"2024-06-29T14:41:59.520799+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:42.846558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883445+0000","last_change":"2024-06-29T14:41:58.545635+0000","last_active":"2024-06-29T14:41:59.883445+0000","last_peered":"2024-06-29T14:41:59.883445+0000","last_clean":"2024-06-29T14:41:59.883445+0000","last_became_active":"2024-06-29T14:41:58.545048+0000","last_became_peered":"2024-06-29T14:41:58.545048+0000","last_unstale":"2024-06-29T14:41:59.883445+0000","last_undegraded":"2024-06-29T14:41:59.883445+0000","last_fullsized":"2024-06-29T14:41:59.883445+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:59.751188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882485+0000","last_change":"2024-06-29T14:41:57.520872+0000","last_active":"2024-06-29T14:41:59.882485+0000","last_peered":"2024-06-29T14:41:59.882485+0000","last_clean":"2024-06-29T14:41:59.882485+0000","last_became_active":"2024-06-29T14:41:57.520576+0000","last_became_peered":"2024-06-29T14:41:57.520576+0000","last_unstale":"2024-06-29T14:41:59.882485+0000","last_undegraded":"2024-06-29T14:41:59.882485+0000","last_fullsized":"2024-06-29T14:41:59.882485+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:00.319413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882963+0000","last_change":"2024-06-29T14:41:58.546541+0000","last_active":"2024-06-29T14:41:59.882963+0000","last_peered":"2024-06-29T14:41:59.882963+0000","last_clean":"2024-06-29T14:41:59.882963+0000","last_became_active":"2024-06-29T14:41:58.546220+0000","last_became_peered":"2024-06-29T14:41:58.546220+0000","last_unstale":"2024-06-29T14:41:59.882963+0000","last_undegraded":"2024-06-29T14:41:59.882963+0000","last_fullsized":"2024-06-29T14:41:59.882963+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:41.043970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883116+0000","last_change":"2024-06-29T14:41:57.518675+0000","last_active":"2024-06-29T14:41:59.883116+0000","last_peered":"2024-06-29T14:41:59.883116+0000","last_clean":"2024-06-29T14:41:59.883116+0000","last_became_active":"2024-06-29T14:41:57.518371+0000","last_became_peered":"2024-06-29T14:41:57.518371+0000","last_unstale":"2024-06-29T14:41:59.883116+0000","last_undegraded":"2024-06-29T14:41:59.883116+0000","last_fullsized":"2024-06-29T14:41:59.883116+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:07:25.367733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883239+0000","last_change":"2024-06-29T14:41:58.541060+0000","last_active":"2024-06-29T14:41:59.883239+0000","last_peered":"2024-06-29T14:41:59.883239+0000","last_clean":"2024-06-29T14:41:59.883239+0000","last_became_active":"2024-06-29T14:41:58.540759+0000","last_became_peered":"2024-06-29T14:41:58.540759+0000","last_unstale":"2024-06-29T14:41:59.883239+0000","last_undegraded":"2024-06-29T14:41:59.883239+0000","last_fullsized":"2024-06-29T14:41:59.883239+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:24.591945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"145'1","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882286+0000","last_change":"2024-06-29T14:41:57.510953+0000","last_active":"2024-06-29T14:41:59.882286+0000","last_peered":"2024-06-29T14:41:59.882286+0000","last_clean":"2024-06-29T14:41:59.882286+0000","last_became_active":"2024-06-29T14:41:57.510084+0000","last_became_peered":"2024-06-29T14:41:57.510084+0000","last_unstale":"2024-06-29T14:41:59.882286+0000","last_undegraded":"2024-06-29T14:41:59.882286+0000","last_fullsized":"2024-06-29T14:41:59.882286+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:10:35.137757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520441+0000","last_change":"2024-06-29T14:41:58.542805+0000","last_active":"2024-06-29T14:41:59.520441+0000","last_peered":"2024-06-29T14:41:59.520441+0000","last_clean":"2024-06-29T14:41:59.520441+0000","last_became_active":"2024-06-29T14:41:58.542469+0000","last_became_peered":"2024-06-29T14:41:58.542469+0000","last_unstale":"2024-06-29T14:41:59.520441+0000","last_undegraded":"2024-06-29T14:41:59.520441+0000","last_fullsized":"2024-06-29T14:41:59.520441+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:59.206644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"145'1","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883355+0000","last_change":"2024-06-29T14:41:57.513970+0000","last_active":"2024-06-29T14:41:59.883355+0000","last_peered":"2024-06-29T14:41:59.883355+0000","last_clean":"2024-06-29T14:41:59.883355+0000","last_became_active":"2024-06-29T14:41:57.513369+0000","last_became_peered":"2024-06-29T14:41:57.513369+0000","last_unstale":"2024-06-29T14:41:59.883355+0000","last_undegraded":"2024-06-29T14:41:59.883355+0000","last_fullsized":"2024-06-29T14:41:59.883355+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:57.320277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883189+0000","last_change":"2024-06-29T14:41:58.528742+0000","last_active":"2024-06-29T14:41:59.883189+0000","last_peered":"2024-06-29T14:41:59.883189+0000","last_clean":"2024-06-29T14:41:59.883189+0000","last_became_active":"2024-06-29T14:41:58.528491+0000","last_became_peered":"2024-06-29T14:41:58.528491+0000","last_unstale":"2024-06-29T14:41:59.883189+0000","last_undegraded":"2024-06-29T14:41:59.883189+0000","last_fullsized":"2024-06-29T14:41:59.883189+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:55:33.110209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520124+0000","last_change":"2024-06-29T14:41:57.524848+0000","last_active":"2024-06-29T14:41:59.520124+0000","last_peered":"2024-06-29T14:41:59.520124+0000","last_clean":"2024-06-29T14:41:59.520124+0000","last_became_active":"2024-06-29T14:41:57.524534+0000","last_became_peered":"2024-06-29T14:41:57.524534+0000","last_unstale":"2024-06-29T14:41:59.520124+0000","last_undegraded":"2024-06-29T14:41:59.520124+0000","last_fullsized":"2024-06-29T14:41:59.520124+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:27.570333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882499+0000","last_change":"2024-06-29T14:41:58.525953+0000","last_active":"2024-06-29T14:41:59.882499+0000","last_peered":"2024-06-29T14:41:59.882499+0000","last_clean":"2024-06-29T14:41:59.882499+0000","last_became_active":"2024-06-29T14:41:58.525654+0000","last_became_peered":"2024-06-29T14:41:58.525654+0000","last_unstale":"2024-06-29T14:41:59.882499+0000","last_undegraded":"2024-06-29T14:41:59.882499+0000","last_fullsized":"2024-06-29T14:41:59.882499+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:55:33.110209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882012+0000","last_change":"2024-06-29T14:41:57.512123+0000","last_active":"2024-06-29T14:41:59.882012+0000","last_peered":"2024-06-29T14:41:59.882012+0000","last_clean":"2024-06-29T14:41:59.882012+0000","last_became_active":"2024-06-29T14:41:57.511804+0000","last_became_peered":"2024-06-29T14:41:57.511804+0000","last_unstale":"2024-06-29T14:41:59.882012+0000","last_undegraded":"2024-06-29T14:41:59.882012+0000","last_fullsized":"2024-06-29T14:41:59.882012+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:06.908709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518762+0000","last_change":"2024-06-29T14:41:58.532335+0000","last_active":"2024-06-29T14:41:59.518762+0000","last_peered":"2024-06-29T14:41:59.518762+0000","last_clean":"2024-06-29T14:41:59.518762+0000","last_became_active":"2024-06-29T14:41:58.531650+0000","last_became_peered":"2024-06-29T14:41:58.531650+0000","last_unstale":"2024-06-29T14:41:59.518762+0000","last_undegraded":"2024-06-29T14:41:59.518762+0000","last_fullsized":"2024-06-29T14:41:59.518762+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:03.210487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520233+0000","last_change":"2024-06-29T14:41:57.517632+0000","last_active":"2024-06-29T14:41:59.520233+0000","last_peered":"2024-06-29T14:41:59.520233+0000","last_clean":"2024-06-29T14:41:59.520233+0000","last_became_active":"2024-06-29T14:41:57.517317+0000","last_became_peered":"2024-06-29T14:41:57.517317+0000","last_unstale":"2024-06-29T14:41:59.520233+0000","last_undegraded":"2024-06-29T14:41:59.520233+0000","last_fullsized":"2024-06-29T14:41:59.520233+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:08:08.227545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520514+0000","last_change":"2024-06-29T14:41:58.529284+0000","last_active":"2024-06-29T14:41:59.520514+0000","last_peered":"2024-06-29T14:41:59.520514+0000","last_clean":"2024-06-29T14:41:59.520514+0000","last_became_active":"2024-06-29T14:41:58.528902+0000","last_became_peered":"2024-06-29T14:41:58.528902+0000","last_unstale":"2024-06-29T14:41:59.520514+0000","last_undegraded":"2024-06-29T14:41:59.520514+0000","last_fullsized":"2024-06-29T14:41:59.520514+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:31:24.394240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883345+0000","last_change":"2024-06-29T14:41:57.518573+0000","last_active":"2024-06-29T14:41:59.883345+0000","last_peered":"2024-06-29T14:41:59.883345+0000","last_clean":"2024-06-29T14:41:59.883345+0000","last_became_active":"2024-06-29T14:41:57.518295+0000","last_became_peered":"2024-06-29T14:41:57.518295+0000","last_unstale":"2024-06-29T14:41:59.883345+0000","last_undegraded":"2024-06-29T14:41:59.883345+0000","last_fullsized":"2024-06-29T14:41:59.883345+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:34.767077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883501+0000","last_change":"2024-06-29T14:41:58.547434+0000","last_active":"2024-06-29T14:41:59.883501+0000","last_peered":"2024-06-29T14:41:59.883501+0000","last_clean":"2024-06-29T14:41:59.883501+0000","last_became_active":"2024-06-29T14:41:58.547127+0000","last_became_peered":"2024-06-29T14:41:58.547127+0000","last_unstale":"2024-06-29T14:41:59.883501+0000","last_undegraded":"2024-06-29T14:41:59.883501+0000","last_fullsized":"2024-06-29T14:41:59.883501+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:52:57.483931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"145'3","reported_seq":17,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.521126+0000","last_change":"2024-06-29T14:41:57.515647+0000","last_active":"2024-06-29T14:41:59.521126+0000","last_peered":"2024-06-29T14:41:59.521126+0000","last_clean":"2024-06-29T14:41:59.521126+0000","last_became_active":"2024-06-29T14:41:57.515330+0000","last_became_peered":"2024-06-29T14:41:57.515330+0000","last_unstale":"2024-06-29T14:41:59.521126+0000","last_undegraded":"2024-06-29T14:41:59.521126+0000","last_fullsized":"2024-06-29T14:41:59.521126+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:19.877654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883135+0000","last_change":"2024-06-29T14:41:58.550259+0000","last_active":"2024-06-29T14:41:59.883135+0000","last_peered":"2024-06-29T14:41:59.883135+0000","last_clean":"2024-06-29T14:41:59.883135+0000","last_became_active":"2024-06-29T14:41:58.549955+0000","last_became_peered":"2024-06-29T14:41:58.549955+0000","last_unstale":"2024-06-29T14:41:59.883135+0000","last_undegraded":"2024-06-29T14:41:59.883135+0000","last_fullsized":"2024-06-29T14:41:59.883135+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:07:28.188343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"145'6","reported_seq":20,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883036+0000","last_change":"2024-06-29T14:41:57.524442+0000","last_active":"2024-06-29T14:41:59.883036+0000","last_peered":"2024-06-29T14:41:59.883036+0000","last_clean":"2024-06-29T14:41:59.883036+0000","last_became_active":"2024-06-29T14:41:57.524028+0000","last_became_peered":"2024-06-29T14:41:57.524028+0000","last_unstale":"2024-06-29T14:41:59.883036+0000","last_undegraded":"2024-06-29T14:41:59.883036+0000","last_fullsized":"2024-06-29T14:41:59.883036+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:47:08.093965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518722+0000","last_change":"2024-06-29T14:41:58.532438+0000","last_active":"2024-06-29T14:41:59.518722+0000","last_peered":"2024-06-29T14:41:59.518722+0000","last_clean":"2024-06-29T14:41:59.518722+0000","last_became_active":"2024-06-29T14:41:58.532097+0000","last_became_peered":"2024-06-29T14:41:58.532097+0000","last_unstale":"2024-06-29T14:41:59.518722+0000","last_undegraded":"2024-06-29T14:41:59.518722+0000","last_fullsized":"2024-06-29T14:41:59.518722+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:21.706510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518336+0000","last_change":"2024-06-29T14:41:57.515771+0000","last_active":"2024-06-29T14:41:59.518336+0000","last_peered":"2024-06-29T14:41:59.518336+0000","last_clean":"2024-06-29T14:41:59.518336+0000","last_became_active":"2024-06-29T14:41:57.515422+0000","last_became_peered":"2024-06-29T14:41:57.515422+0000","last_unstale":"2024-06-29T14:41:59.518336+0000","last_undegraded":"2024-06-29T14:41:59.518336+0000","last_fullsized":"2024-06-29T14:41:59.518336+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:52.871547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.881892+0000","last_change":"2024-06-29T14:41:58.540270+0000","last_active":"2024-06-29T14:41:59.881892+0000","last_peered":"2024-06-29T14:41:59.881892+0000","last_clean":"2024-06-29T14:41:59.881892+0000","last_became_active":"2024-06-29T14:41:58.539956+0000","last_became_peered":"2024-06-29T14:41:58.539956+0000","last_unstale":"2024-06-29T14:41:59.881892+0000","last_undegraded":"2024-06-29T14:41:59.881892+0000","last_fullsized":"2024-06-29T14:41:59.881892+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:21.706510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882809+0000","last_change":"2024-06-29T14:41:57.522710+0000","last_active":"2024-06-29T14:41:59.882809+0000","last_peered":"2024-06-29T14:41:59.882809+0000","last_clean":"2024-06-29T14:41:59.882809+0000","last_became_active":"2024-06-29T14:41:57.522434+0000","last_became_peered":"2024-06-29T14:41:57.522434+0000","last_unstale":"2024-06-29T14:41:59.882809+0000","last_undegraded":"2024-06-29T14:41:59.882809+0000","last_fullsized":"2024-06-29T14:41:59.882809+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:41.277655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883150+0000","last_change":"2024-06-29T14:41:58.538420+0000","last_active":"2024-06-29T14:41:59.883150+0000","last_peered":"2024-06-29T14:41:59.883150+0000","last_clean":"2024-06-29T14:41:59.883150+0000","last_became_active":"2024-06-29T14:41:58.537911+0000","last_became_peered":"2024-06-29T14:41:58.537911+0000","last_unstale":"2024-06-29T14:41:59.883150+0000","last_undegraded":"2024-06-29T14:41:59.883150+0000","last_fullsized":"2024-06-29T14:41:59.883150+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:47.467202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"145'8","reported_seq":22,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882751+0000","last_change":"2024-06-29T14:41:57.517845+0000","last_active":"2024-06-29T14:41:59.882751+0000","last_peered":"2024-06-29T14:41:59.882751+0000","last_clean":"2024-06-29T14:41:59.882751+0000","last_became_active":"2024-06-29T14:41:57.517533+0000","last_became_peered":"2024-06-29T14:41:57.517533+0000","last_unstale":"2024-06-29T14:41:59.882751+0000","last_undegraded":"2024-06-29T14:41:59.882751+0000","last_fullsized":"2024-06-29T14:41:59.882751+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:00.699763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883715+0000","last_change":"2024-06-29T14:41:58.532290+0000","last_active":"2024-06-29T14:41:59.883715+0000","last_peered":"2024-06-29T14:41:59.883715+0000","last_clean":"2024-06-29T14:41:59.883715+0000","last_became_active":"2024-06-29T14:41:58.531945+0000","last_became_peered":"2024-06-29T14:41:58.531945+0000","last_unstale":"2024-06-29T14:41:59.883715+0000","last_undegraded":"2024-06-29T14:41:59.883715+0000","last_fullsized":"2024-06-29T14:41:59.883715+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:21:34.989370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"145'1","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520369+0000","last_change":"2024-06-29T14:41:57.521869+0000","last_active":"2024-06-29T14:41:59.520369+0000","last_peered":"2024-06-29T14:41:59.520369+0000","last_clean":"2024-06-29T14:41:59.520369+0000","last_became_active":"2024-06-29T14:41:57.521295+0000","last_became_peered":"2024-06-29T14:41:57.521295+0000","last_unstale":"2024-06-29T14:41:59.520369+0000","last_undegraded":"2024-06-29T14:41:59.520369+0000","last_fullsized":"2024-06-29T14:41:59.520369+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:29.168190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520088+0000","last_change":"2024-06-29T14:41:58.540935+0000","last_active":"2024-06-29T14:41:59.520088+0000","last_peered":"2024-06-29T14:41:59.520088+0000","last_clean":"2024-06-29T14:41:59.520088+0000","last_became_active":"2024-06-29T14:41:58.535673+0000","last_became_peered":"2024-06-29T14:41:58.535673+0000","last_unstale":"2024-06-29T14:41:59.520088+0000","last_undegraded":"2024-06-29T14:41:59.520088+0000","last_fullsized":"2024-06-29T14:41:59.520088+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:19.408347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883318+0000","last_change":"2024-06-29T14:41:57.521214+0000","last_active":"2024-06-29T14:41:59.883318+0000","last_peered":"2024-06-29T14:41:59.883318+0000","last_clean":"2024-06-29T14:41:59.883318+0000","last_became_active":"2024-06-29T14:41:57.520868+0000","last_became_peered":"2024-06-29T14:41:57.520868+0000","last_unstale":"2024-06-29T14:41:59.883318+0000","last_undegraded":"2024-06-29T14:41:59.883318+0000","last_fullsized":"2024-06-29T14:41:59.883318+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:25.592455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882539+0000","last_change":"2024-06-29T14:41:58.530559+0000","last_active":"2024-06-29T14:41:59.882539+0000","last_peered":"2024-06-29T14:41:59.882539+0000","last_clean":"2024-06-29T14:41:59.882539+0000","last_became_active":"2024-06-29T14:41:58.530236+0000","last_became_peered":"2024-06-29T14:41:58.530236+0000","last_unstale":"2024-06-29T14:41:59.882539+0000","last_undegraded":"2024-06-29T14:41:59.882539+0000","last_fullsized":"2024-06-29T14:41:59.882539+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:27:36.015471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"145'1","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.521316+0000","last_change":"2024-06-29T14:41:57.517241+0000","last_active":"2024-06-29T14:41:59.521316+0000","last_peered":"2024-06-29T14:41:59.521316+0000","last_clean":"2024-06-29T14:41:59.521316+0000","last_became_active":"2024-06-29T14:41:57.516919+0000","last_became_peered":"2024-06-29T14:41:57.516919+0000","last_unstale":"2024-06-29T14:41:59.521316+0000","last_undegraded":"2024-06-29T14:41:59.521316+0000","last_fullsized":"2024-06-29T14:41:59.521316+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:18.721939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882336+0000","last_change":"2024-06-29T14:41:58.529352+0000","last_active":"2024-06-29T14:41:59.882336+0000","last_peered":"2024-06-29T14:41:59.882336+0000","last_clean":"2024-06-29T14:41:59.882336+0000","last_became_active":"2024-06-29T14:41:58.529058+0000","last_became_peered":"2024-06-29T14:41:58.529058+0000","last_unstale":"2024-06-29T14:41:59.882336+0000","last_undegraded":"2024-06-29T14:41:59.882336+0000","last_fullsized":"2024-06-29T14:41:59.882336+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:32.024937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520080+0000","last_change":"2024-06-29T14:41:57.518748+0000","last_active":"2024-06-29T14:41:59.520080+0000","last_peered":"2024-06-29T14:41:59.520080+0000","last_clean":"2024-06-29T14:41:59.520080+0000","last_became_active":"2024-06-29T14:41:57.518413+0000","last_became_peered":"2024-06-29T14:41:57.518413+0000","last_unstale":"2024-06-29T14:41:59.520080+0000","last_undegraded":"2024-06-29T14:41:59.520080+0000","last_fullsized":"2024-06-29T14:41:59.520080+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:00.319413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518287+0000","last_change":"2024-06-29T14:41:58.545198+0000","last_active":"2024-06-29T14:41:59.518287+0000","last_peered":"2024-06-29T14:41:59.518287+0000","last_clean":"2024-06-29T14:41:59.518287+0000","last_became_active":"2024-06-29T14:41:58.544883+0000","last_became_peered":"2024-06-29T14:41:58.544883+0000","last_unstale":"2024-06-29T14:41:59.518287+0000","last_undegraded":"2024-06-29T14:41:59.518287+0000","last_fullsized":"2024-06-29T14:41:59.518287+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:47.616020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520607+0000","last_change":"2024-06-29T14:41:57.514665+0000","last_active":"2024-06-29T14:41:59.520607+0000","last_peered":"2024-06-29T14:41:59.520607+0000","last_clean":"2024-06-29T14:41:59.520607+0000","last_became_active":"2024-06-29T14:41:57.514340+0000","last_became_peered":"2024-06-29T14:41:57.514340+0000","last_unstale":"2024-06-29T14:41:59.520607+0000","last_undegraded":"2024-06-29T14:41:59.520607+0000","last_fullsized":"2024-06-29T14:41:59.520607+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:25.592455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520639+0000","last_change":"2024-06-29T14:41:58.545545+0000","last_active":"2024-06-29T14:41:59.520639+0000","last_peered":"2024-06-29T14:41:59.520639+0000","last_clean":"2024-06-29T14:41:59.520639+0000","last_became_active":"2024-06-29T14:41:58.545128+0000","last_became_peered":"2024-06-29T14:41:58.545128+0000","last_unstale":"2024-06-29T14:41:59.520639+0000","last_undegraded":"2024-06-29T14:41:59.520639+0000","last_fullsized":"2024-06-29T14:41:59.520639+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:33:36.000748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883425+0000","last_change":"2024-06-29T14:41:57.514209+0000","last_active":"2024-06-29T14:41:59.883425+0000","last_peered":"2024-06-29T14:41:59.883425+0000","last_clean":"2024-06-29T14:41:59.883425+0000","last_became_active":"2024-06-29T14:41:57.512897+0000","last_became_peered":"2024-06-29T14:41:57.512897+0000","last_unstale":"2024-06-29T14:41:59.883425+0000","last_undegraded":"2024-06-29T14:41:59.883425+0000","last_fullsized":"2024-06-29T14:41:59.883425+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:04:29.122336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.881888+0000","last_change":"2024-06-29T14:41:58.531912+0000","last_active":"2024-06-29T14:41:59.881888+0000","last_peered":"2024-06-29T14:41:59.881888+0000","last_clean":"2024-06-29T14:41:59.881888+0000","last_became_active":"2024-06-29T14:41:58.531566+0000","last_became_peered":"2024-06-29T14:41:58.531566+0000","last_unstale":"2024-06-29T14:41:59.881888+0000","last_undegraded":"2024-06-29T14:41:59.881888+0000","last_fullsized":"2024-06-29T14:41:59.881888+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:53:04.047421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882705+0000","last_change":"2024-06-29T14:41:57.515401+0000","last_active":"2024-06-29T14:41:59.882705+0000","last_peered":"2024-06-29T14:41:59.882705+0000","last_clean":"2024-06-29T14:41:59.882705+0000","last_became_active":"2024-06-29T14:41:57.514805+0000","last_became_peered":"2024-06-29T14:41:57.514805+0000","last_unstale":"2024-06-29T14:41:59.882705+0000","last_undegraded":"2024-06-29T14:41:59.882705+0000","last_fullsized":"2024-06-29T14:41:59.882705+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:46:26.472467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883306+0000","last_change":"2024-06-29T14:41:58.544238+0000","last_active":"2024-06-29T14:41:59.883306+0000","last_peered":"2024-06-29T14:41:59.883306+0000","last_clean":"2024-06-29T14:41:59.883306+0000","last_became_active":"2024-06-29T14:41:58.543507+0000","last_became_peered":"2024-06-29T14:41:58.543507+0000","last_unstale":"2024-06-29T14:41:59.883306+0000","last_undegraded":"2024-06-29T14:41:59.883306+0000","last_fullsized":"2024-06-29T14:41:59.883306+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:48:43.997072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882783+0000","last_change":"2024-06-29T14:41:57.520705+0000","last_active":"2024-06-29T14:41:59.882783+0000","last_peered":"2024-06-29T14:41:59.882783+0000","last_clean":"2024-06-29T14:41:59.882783+0000","last_became_active":"2024-06-29T14:41:57.520343+0000","last_became_peered":"2024-06-29T14:41:57.520343+0000","last_unstale":"2024-06-29T14:41:59.882783+0000","last_undegraded":"2024-06-29T14:41:59.882783+0000","last_fullsized":"2024-06-29T14:41:59.882783+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:08:08.227545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.521045+0000","last_change":"2024-06-29T14:41:58.541990+0000","last_active":"2024-06-29T14:41:59.521045+0000","last_peered":"2024-06-29T14:41:59.521045+0000","last_clean":"2024-06-29T14:41:59.521045+0000","last_became_active":"2024-06-29T14:41:58.541653+0000","last_became_peered":"2024-06-29T14:41:58.541653+0000","last_unstale":"2024-06-29T14:41:59.521045+0000","last_undegraded":"2024-06-29T14:41:59.521045+0000","last_fullsized":"2024-06-29T14:41:59.521045+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:01.757137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518965+0000","last_change":"2024-06-29T14:41:57.516375+0000","last_active":"2024-06-29T14:41:59.518965+0000","last_peered":"2024-06-29T14:41:59.518965+0000","last_clean":"2024-06-29T14:41:59.518965+0000","last_became_active":"2024-06-29T14:41:57.516069+0000","last_became_peered":"2024-06-29T14:41:57.516069+0000","last_unstale":"2024-06-29T14:41:59.518965+0000","last_undegraded":"2024-06-29T14:41:59.518965+0000","last_fullsized":"2024-06-29T14:41:59.518965+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:03:34.805136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518854+0000","last_change":"2024-06-29T14:41:58.533836+0000","last_active":"2024-06-29T14:41:59.518854+0000","last_peered":"2024-06-29T14:41:59.518854+0000","last_clean":"2024-06-29T14:41:59.518854+0000","last_became_active":"2024-06-29T14:41:58.533541+0000","last_became_peered":"2024-06-29T14:41:58.533541+0000","last_unstale":"2024-06-29T14:41:59.518854+0000","last_undegraded":"2024-06-29T14:41:59.518854+0000","last_fullsized":"2024-06-29T14:41:59.518854+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:04:22.876673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883245+0000","last_change":"2024-06-29T14:41:57.525920+0000","last_active":"2024-06-29T14:41:59.883245+0000","last_peered":"2024-06-29T14:41:59.883245+0000","last_clean":"2024-06-29T14:41:59.883245+0000","last_became_active":"2024-06-29T14:41:57.525591+0000","last_became_peered":"2024-06-29T14:41:57.525591+0000","last_unstale":"2024-06-29T14:41:59.883245+0000","last_undegraded":"2024-06-29T14:41:59.883245+0000","last_fullsized":"2024-06-29T14:41:59.883245+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:15.101874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883264+0000","last_change":"2024-06-29T14:41:58.539751+0000","last_active":"2024-06-29T14:41:59.883264+0000","last_peered":"2024-06-29T14:41:59.883264+0000","last_clean":"2024-06-29T14:41:59.883264+0000","last_became_active":"2024-06-29T14:41:58.539467+0000","last_became_peered":"2024-06-29T14:41:58.539467+0000","last_unstale":"2024-06-29T14:41:59.883264+0000","last_undegraded":"2024-06-29T14:41:59.883264+0000","last_fullsized":"2024-06-29T14:41:59.883264+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:00.046159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882411+0000","last_change":"2024-06-29T14:41:57.511991+0000","last_active":"2024-06-29T14:41:59.882411+0000","last_peered":"2024-06-29T14:41:59.882411+0000","last_clean":"2024-06-29T14:41:59.882411+0000","last_became_active":"2024-06-29T14:41:57.511642+0000","last_became_peered":"2024-06-29T14:41:57.511642+0000","last_unstale":"2024-06-29T14:41:59.882411+0000","last_undegraded":"2024-06-29T14:41:59.882411+0000","last_fullsized":"2024-06-29T14:41:59.882411+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:03.454009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.521291+0000","last_change":"2024-06-29T14:41:58.543244+0000","last_active":"2024-06-29T14:41:59.521291+0000","last_peered":"2024-06-29T14:41:59.521291+0000","last_clean":"2024-06-29T14:41:59.521291+0000","last_became_active":"2024-06-29T14:41:58.542928+0000","last_became_peered":"2024-06-29T14:41:58.542928+0000","last_unstale":"2024-06-29T14:41:59.521291+0000","last_undegraded":"2024-06-29T14:41:59.521291+0000","last_fullsized":"2024-06-29T14:41:59.521291+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:46.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883306+0000","last_change":"2024-06-29T14:41:57.520297+0000","last_active":"2024-06-29T14:41:59.883306+0000","last_peered":"2024-06-29T14:41:59.883306+0000","last_clean":"2024-06-29T14:41:59.883306+0000","last_became_active":"2024-06-29T14:41:57.519802+0000","last_became_peered":"2024-06-29T14:41:57.519802+0000","last_unstale":"2024-06-29T14:41:59.883306+0000","last_undegraded":"2024-06-29T14:41:59.883306+0000","last_fullsized":"2024-06-29T14:41:59.883306+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:42.846558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882608+0000","last_change":"2024-06-29T14:41:58.546419+0000","last_active":"2024-06-29T14:41:59.882608+0000","last_peered":"2024-06-29T14:41:59.882608+0000","last_clean":"2024-06-29T14:41:59.882608+0000","last_became_active":"2024-06-29T14:41:58.546130+0000","last_became_peered":"2024-06-29T14:41:58.546130+0000","last_unstale":"2024-06-29T14:41:59.882608+0000","last_undegraded":"2024-06-29T14:41:59.882608+0000","last_fullsized":"2024-06-29T14:41:59.882608+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:37.484966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882393+0000","last_change":"2024-06-29T14:41:57.523618+0000","last_active":"2024-06-29T14:41:59.882393+0000","last_peered":"2024-06-29T14:41:59.882393+0000","last_clean":"2024-06-29T14:41:59.882393+0000","last_became_active":"2024-06-29T14:41:57.523323+0000","last_became_peered":"2024-06-29T14:41:57.523323+0000","last_unstale":"2024-06-29T14:41:59.882393+0000","last_undegraded":"2024-06-29T14:41:59.882393+0000","last_fullsized":"2024-06-29T14:41:59.882393+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:12:36.272661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520468+0000","last_change":"2024-06-29T14:41:58.527521+0000","last_active":"2024-06-29T14:41:59.520468+0000","last_peered":"2024-06-29T14:41:59.520468+0000","last_clean":"2024-06-29T14:41:59.520468+0000","last_became_active":"2024-06-29T14:41:58.527196+0000","last_became_peered":"2024-06-29T14:41:58.527196+0000","last_unstale":"2024-06-29T14:41:59.520468+0000","last_undegraded":"2024-06-29T14:41:59.520468+0000","last_fullsized":"2024-06-29T14:41:59.520468+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:17:42.624622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882745+0000","last_change":"2024-06-29T14:41:57.527887+0000","last_active":"2024-06-29T14:41:59.882745+0000","last_peered":"2024-06-29T14:41:59.882745+0000","last_clean":"2024-06-29T14:41:59.882745+0000","last_became_active":"2024-06-29T14:41:57.527563+0000","last_became_peered":"2024-06-29T14:41:57.527563+0000","last_unstale":"2024-06-29T14:41:59.882745+0000","last_undegraded":"2024-06-29T14:41:59.882745+0000","last_fullsized":"2024-06-29T14:41:59.882745+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:04.951455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883160+0000","last_change":"2024-06-29T14:41:58.539956+0000","last_active":"2024-06-29T14:41:59.883160+0000","last_peered":"2024-06-29T14:41:59.883160+0000","last_clean":"2024-06-29T14:41:59.883160+0000","last_became_active":"2024-06-29T14:41:58.539658+0000","last_became_peered":"2024-06-29T14:41:58.539658+0000","last_unstale":"2024-06-29T14:41:59.883160+0000","last_undegraded":"2024-06-29T14:41:59.883160+0000","last_fullsized":"2024-06-29T14:41:59.883160+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:50.654794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882809+0000","last_change":"2024-06-29T14:41:57.513673+0000","last_active":"2024-06-29T14:41:59.882809+0000","last_peered":"2024-06-29T14:41:59.882809+0000","last_clean":"2024-06-29T14:41:59.882809+0000","last_became_active":"2024-06-29T14:41:57.513343+0000","last_became_peered":"2024-06-29T14:41:57.513343+0000","last_unstale":"2024-06-29T14:41:59.882809+0000","last_undegraded":"2024-06-29T14:41:59.882809+0000","last_fullsized":"2024-06-29T14:41:59.882809+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:59.031255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882677+0000","last_change":"2024-06-29T14:41:58.533051+0000","last_active":"2024-06-29T14:41:59.882677+0000","last_peered":"2024-06-29T14:41:59.882677+0000","last_clean":"2024-06-29T14:41:59.882677+0000","last_became_active":"2024-06-29T14:41:58.532682+0000","last_became_peered":"2024-06-29T14:41:58.532682+0000","last_unstale":"2024-06-29T14:41:59.882677+0000","last_undegraded":"2024-06-29T14:41:59.882677+0000","last_fullsized":"2024-06-29T14:41:59.882677+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:32.367186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882883+0000","last_change":"2024-06-29T14:41:57.521672+0000","last_active":"2024-06-29T14:41:59.882883+0000","last_peered":"2024-06-29T14:41:59.882883+0000","last_clean":"2024-06-29T14:41:59.882883+0000","last_became_active":"2024-06-29T14:41:57.521363+0000","last_became_peered":"2024-06-29T14:41:57.521363+0000","last_unstale":"2024-06-29T14:41:59.882883+0000","last_undegraded":"2024-06-29T14:41:59.882883+0000","last_fullsized":"2024-06-29T14:41:59.882883+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:29.168190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882643+0000","last_change":"2024-06-29T14:41:58.542973+0000","last_active":"2024-06-29T14:41:59.882643+0000","last_peered":"2024-06-29T14:41:59.882643+0000","last_clean":"2024-06-29T14:41:59.882643+0000","last_became_active":"2024-06-29T14:41:58.542587+0000","last_became_peered":"2024-06-29T14:41:58.542587+0000","last_unstale":"2024-06-29T14:41:59.882643+0000","last_undegraded":"2024-06-29T14:41:59.882643+0000","last_fullsized":"2024-06-29T14:41:59.882643+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:02:55.136388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883171+0000","last_change":"2024-06-29T14:41:57.525834+0000","last_active":"2024-06-29T14:41:59.883171+0000","last_peered":"2024-06-29T14:41:59.883171+0000","last_clean":"2024-06-29T14:41:59.883171+0000","last_became_active":"2024-06-29T14:41:57.525328+0000","last_became_peered":"2024-06-29T14:41:57.525328+0000","last_unstale":"2024-06-29T14:41:59.883171+0000","last_undegraded":"2024-06-29T14:41:59.883171+0000","last_fullsized":"2024-06-29T14:41:59.883171+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:38.029430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"135'156","reported_seq":653,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882410+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:41:59.882410+0000","last_peered":"2024-06-29T14:41:59.882410+0000","last_clean":"2024-06-29T14:41:59.882410+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:41:59.882410+0000","last_undegraded":"2024-06-29T14:41:59.882410+0000","last_fullsized":"2024-06-29T14:41:59.882410+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":245,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":245,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575083,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33568,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338272,"statfs":{"total":96636764160,"available":96602390528,"internally_reserved":0,"allocated":2437120,"data_stored":1468589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575084,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37116,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334724,"statfs":{"total":96636764160,"available":96598757376,"internally_reserved":0,"allocated":1818624,"data_stored":871183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575082,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37116,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334724,"statfs":{"total":96636764160,"available":96598757376,"internally_reserved":0,"allocated":1814528,"data_stored":869683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59336,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312504,"statfs":{"total":96636764160,"available":96576004096,"internally_reserved":0,"allocated":1822720,"data_stored":876041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575083,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37112,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334728,"statfs":{"total":96636764160,"available":96598761472,"internally_reserved":0,"allocated":1814528,"data_stored":869731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575083,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59328,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312512,"statfs":{"total":96636764160,"available":96576012288,"internally_reserved":0,"allocated":1814528,"data_stored":870187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55156,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316684,"statfs":{"total":96636764160,"available":96580284416,"internally_reserved":0,"allocated":1806336,"data_stored":869571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575083,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33524,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338316,"statfs":{"total":96636764160,"available":96602435584,"internally_reserved":0,"allocated":2400256,"data_stored":1459939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:42:02.571 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T14:42:02.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.723+0000 7f0abf01f640 1 -- 172.21.15.81:0/3285600405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ab80d2a90 msgr2=0x7f0ab80d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.723+0000 7f0abf01f640 1 --2- 172.21.15.81:0/3285600405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ab80d2a90 0x7f0ab80d09c0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0005fd0 tx=0x7f0ab002aa90 comp rx=0 tx=0).stop 2024-06-29T14:42:02.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.723+0000 7f0abf01f640 1 -- 172.21.15.81:0/3285600405 shutdown_connections 2024-06-29T14:42:02.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.723+0000 7f0abf01f640 1 --2- 172.21.15.81:0/3285600405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ab80d2a90 0x7f0ab80d09c0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.723+0000 7f0abf01f640 1 --2- 172.21.15.81:0/3285600405 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ab80b9040 0x7f0ab80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.723+0000 7f0abf01f640 1 -- 172.21.15.81:0/3285600405 >> 172.21.15.81:0/3285600405 conn(0x7f0ab809cf30 msgr2=0x7f0ab80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.723+0000 7f0abf01f640 1 -- 172.21.15.81:0/3285600405 shutdown_connections 2024-06-29T14:42:02.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.723+0000 7f0abf01f640 1 -- 172.21.15.81:0/3285600405 wait complete. 2024-06-29T14:42:02.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.725+0000 7f0abf01f640 1 Processor -- start 2024-06-29T14:42:02.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.725+0000 7f0abf01f640 1 -- start start 2024-06-29T14:42:02.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.726+0000 7f0abf01f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ab80b5250 0x7f0ab809d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.726+0000 7f0abf01f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ab80b9040 0x7f0ab809dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.726+0000 7f0abf01f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ab80d2a90 0x7f0ab80c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.726+0000 7f0abf01f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab80b7d30 con 0x7f0ab80d2a90 2024-06-29T14:42:02.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.726+0000 7f0abf01f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab80b7bb0 con 0x7f0ab80b9040 2024-06-29T14:42:02.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.726+0000 7f0abf01f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0ab80b7eb0 con 0x7f0ab80b5250 2024-06-29T14:42:02.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abd5bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ab80d2a90 0x7f0ab80c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abd5bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ab80d2a90 0x7f0ab80c9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55724/0 (socket says 172.21.15.81:55724) 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abd5bc640 1 -- 172.21.15.81:0/2818568252 learned_addr learned my addr 172.21.15.81:0/2818568252 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0ab7fff640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ab80b9040 0x7f0ab809dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abd5bc640 1 -- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ab80b5250 msgr2=0x7f0ab809d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abcdbb640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ab80b5250 0x7f0ab809d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abd5bc640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ab80b5250 0x7f0ab809d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abd5bc640 1 -- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ab80b9040 msgr2=0x7f0ab809dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abd5bc640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ab80b9040 0x7f0ab809dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abd5bc640 1 -- 172.21.15.81:0/2818568252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab80c7be0 con 0x7f0ab80d2a90 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0ab7fff640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ab80b9040 0x7f0ab809dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abcdbb640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ab80b5250 0x7f0ab809d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abd5bc640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ab80d2a90 0x7f0ab80c9140 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0006010 tx=0x7f0ab0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0ab57fa640 1 -- 172.21.15.81:0/2818568252 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0044070 con 0x7f0ab80d2a90 2024-06-29T14:42:02.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0ab57fa640 1 -- 172.21.15.81:0/2818568252 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ab0004250 con 0x7f0ab80d2a90 2024-06-29T14:42:02.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abf01f640 1 -- 172.21.15.81:0/2818568252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ab80c7ec0 con 0x7f0ab80d2a90 2024-06-29T14:42:02.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0ab57fa640 1 -- 172.21.15.81:0/2818568252 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0004680 con 0x7f0ab80d2a90 2024-06-29T14:42:02.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.727+0000 7f0abf01f640 1 -- 172.21.15.81:0/2818568252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ab80c8480 con 0x7f0ab80d2a90 2024-06-29T14:42:02.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.728+0000 7f0abf01f640 1 -- 172.21.15.81:0/2818568252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ab80b8440 con 0x7f0ab80d2a90 2024-06-29T14:42:02.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.729+0000 7f0ab57fa640 1 -- 172.21.15.81:0/2818568252 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f0ab0047070 con 0x7f0ab80d2a90 2024-06-29T14:42:02.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.730+0000 7f0ab57fa640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0a9c076420 0x7f0a9c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:02.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.730+0000 7f0ab57fa640 1 -- 172.21.15.81:0/2818568252 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f0ab00b8ab0 con 0x7f0ab80d2a90 2024-06-29T14:42:02.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.730+0000 7f0abcdbb640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0a9c076420 0x7f0a9c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:02.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.730+0000 7f0abcdbb640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0a9c076420 0x7f0a9c078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00a870 tx=0x7f0aac008040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:02.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.734+0000 7f0ab57fa640 1 -- 172.21.15.81:0/2818568252 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0ab0032030 con 0x7f0ab80d2a90 2024-06-29T14:42:02.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.914+0000 7f0abf01f640 1 -- 172.21.15.81:0/2818568252 --> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ab80c7970 con 0x7f0a9c076420 2024-06-29T14:42:02.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.919+0000 7f0ab57fa640 1 -- 172.21.15.81:0/2818568252 <== mgr.7223 v2:172.21.15.167:6832/3201324170 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252478 (secure 0 0 0) 0x7f0ab80c7970 con 0x7f0a9c076420 2024-06-29T14:42:02.919 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:42:02.920 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T14:42:02.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.922+0000 7f0abf01f640 1 -- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0a9c076420 msgr2=0x7f0a9c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.922+0000 7f0abf01f640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0a9c076420 0x7f0a9c078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00a870 tx=0x7f0aac008040 comp rx=0 tx=0).stop 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.922+0000 7f0abf01f640 1 -- 172.21.15.81:0/2818568252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ab80d2a90 msgr2=0x7f0ab80c9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.922+0000 7f0abf01f640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ab80d2a90 0x7f0ab80c9140 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0006010 tx=0x7f0ab0003f10 comp rx=0 tx=0).stop 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.923+0000 7f0abf01f640 1 -- 172.21.15.81:0/2818568252 shutdown_connections 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.923+0000 7f0abf01f640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0a9c076420 0x7f0a9c078930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.923+0000 7f0abf01f640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ab80d2a90 0x7f0ab80c9140 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.923+0000 7f0abf01f640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ab80b9040 0x7f0ab809dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.923+0000 7f0abf01f640 1 --2- 172.21.15.81:0/2818568252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ab80b5250 0x7f0ab809d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.923+0000 7f0abf01f640 1 -- 172.21.15.81:0/2818568252 >> 172.21.15.81:0/2818568252 conn(0x7f0ab809cf30 msgr2=0x7f0ab806dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.923+0000 7f0abf01f640 1 -- 172.21.15.81:0/2818568252 shutdown_connections 2024-06-29T14:42:02.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:02.923+0000 7f0abf01f640 1 -- 172.21.15.81:0/2818568252 wait complete. 2024-06-29T14:42:02.931 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T14:42:02.066783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":289,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":352256,"kb_used_data":15360,"kb_used_omap":12,"kb_used_meta":336755,"kb_avail":754622464,"statfs":{"total":773094113280,"available":772733403136,"internally_reserved":0,"allocated":15728640,"data_stored":8154924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":344837682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.567702"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518915+0000","last_change":"2024-06-29T14:41:58.546002+0000","last_active":"2024-06-29T14:41:59.518915+0000","last_peered":"2024-06-29T14:41:59.518915+0000","last_clean":"2024-06-29T14:41:59.518915+0000","last_became_active":"2024-06-29T14:41:58.545684+0000","last_became_peered":"2024-06-29T14:41:58.545684+0000","last_unstale":"2024-06-29T14:41:59.518915+0000","last_undegraded":"2024-06-29T14:41:59.518915+0000","last_fullsized":"2024-06-29T14:41:59.518915+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:26.344464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882035+0000","last_change":"2024-06-29T14:41:58.541620+0000","last_active":"2024-06-29T14:41:59.882035+0000","last_peered":"2024-06-29T14:41:59.882035+0000","last_clean":"2024-06-29T14:41:59.882035+0000","last_became_active":"2024-06-29T14:41:58.541312+0000","last_became_peered":"2024-06-29T14:41:58.541312+0000","last_unstale":"2024-06-29T14:41:59.882035+0000","last_undegraded":"2024-06-29T14:41:59.882035+0000","last_fullsized":"2024-06-29T14:41:59.882035+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:03.210487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883481+0000","last_change":"2024-06-29T14:41:58.535498+0000","last_active":"2024-06-29T14:41:59.883481+0000","last_peered":"2024-06-29T14:41:59.883481+0000","last_clean":"2024-06-29T14:41:59.883481+0000","last_became_active":"2024-06-29T14:41:58.534977+0000","last_became_peered":"2024-06-29T14:41:58.534977+0000","last_unstale":"2024-06-29T14:41:59.883481+0000","last_undegraded":"2024-06-29T14:41:59.883481+0000","last_fullsized":"2024-06-29T14:41:59.883481+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:49:38.605946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518547+0000","last_change":"2024-06-29T14:41:58.538521+0000","last_active":"2024-06-29T14:41:59.518547+0000","last_peered":"2024-06-29T14:41:59.518547+0000","last_clean":"2024-06-29T14:41:59.518547+0000","last_became_active":"2024-06-29T14:41:58.538166+0000","last_became_peered":"2024-06-29T14:41:58.538166+0000","last_unstale":"2024-06-29T14:41:59.518547+0000","last_undegraded":"2024-06-29T14:41:59.518547+0000","last_fullsized":"2024-06-29T14:41:59.518547+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:49:42.669496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883054+0000","last_change":"2024-06-29T14:41:58.545829+0000","last_active":"2024-06-29T14:41:59.883054+0000","last_peered":"2024-06-29T14:41:59.883054+0000","last_clean":"2024-06-29T14:41:59.883054+0000","last_became_active":"2024-06-29T14:41:58.545408+0000","last_became_peered":"2024-06-29T14:41:58.545408+0000","last_unstale":"2024-06-29T14:41:59.883054+0000","last_undegraded":"2024-06-29T14:41:59.883054+0000","last_fullsized":"2024-06-29T14:41:59.883054+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:53:05.168790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520450+0000","last_change":"2024-06-29T14:41:58.535159+0000","last_active":"2024-06-29T14:41:59.520450+0000","last_peered":"2024-06-29T14:41:59.520450+0000","last_clean":"2024-06-29T14:41:59.520450+0000","last_became_active":"2024-06-29T14:41:58.534835+0000","last_became_peered":"2024-06-29T14:41:58.534835+0000","last_unstale":"2024-06-29T14:41:59.520450+0000","last_undegraded":"2024-06-29T14:41:59.520450+0000","last_fullsized":"2024-06-29T14:41:59.520450+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:05.790297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.519900+0000","last_change":"2024-06-29T14:41:58.545975+0000","last_active":"2024-06-29T14:41:59.519900+0000","last_peered":"2024-06-29T14:41:59.519900+0000","last_clean":"2024-06-29T14:41:59.519900+0000","last_became_active":"2024-06-29T14:41:58.545554+0000","last_became_peered":"2024-06-29T14:41:58.545554+0000","last_unstale":"2024-06-29T14:41:59.519900+0000","last_undegraded":"2024-06-29T14:41:59.519900+0000","last_fullsized":"2024-06-29T14:41:59.519900+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:30.175774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882512+0000","last_change":"2024-06-29T14:41:58.533534+0000","last_active":"2024-06-29T14:41:59.882512+0000","last_peered":"2024-06-29T14:41:59.882512+0000","last_clean":"2024-06-29T14:41:59.882512+0000","last_became_active":"2024-06-29T14:41:58.533125+0000","last_became_peered":"2024-06-29T14:41:58.533125+0000","last_unstale":"2024-06-29T14:41:59.882512+0000","last_undegraded":"2024-06-29T14:41:59.882512+0000","last_fullsized":"2024-06-29T14:41:59.882512+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:32.024937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883570+0000","last_change":"2024-06-29T14:41:58.537489+0000","last_active":"2024-06-29T14:41:59.883570+0000","last_peered":"2024-06-29T14:41:59.883570+0000","last_clean":"2024-06-29T14:41:59.883570+0000","last_became_active":"2024-06-29T14:41:58.537202+0000","last_became_peered":"2024-06-29T14:41:58.537202+0000","last_unstale":"2024-06-29T14:41:59.883570+0000","last_undegraded":"2024-06-29T14:41:59.883570+0000","last_fullsized":"2024-06-29T14:41:59.883570+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:40.903285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882132+0000","last_change":"2024-06-29T14:41:58.541482+0000","last_active":"2024-06-29T14:41:59.882132+0000","last_peered":"2024-06-29T14:41:59.882132+0000","last_clean":"2024-06-29T14:41:59.882132+0000","last_became_active":"2024-06-29T14:41:58.541169+0000","last_became_peered":"2024-06-29T14:41:58.541169+0000","last_unstale":"2024-06-29T14:41:59.882132+0000","last_undegraded":"2024-06-29T14:41:59.882132+0000","last_fullsized":"2024-06-29T14:41:59.882132+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:04:22.876673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882309+0000","last_change":"2024-06-29T14:41:58.534856+0000","last_active":"2024-06-29T14:41:59.882309+0000","last_peered":"2024-06-29T14:41:59.882309+0000","last_clean":"2024-06-29T14:41:59.882309+0000","last_became_active":"2024-06-29T14:41:58.534523+0000","last_became_peered":"2024-06-29T14:41:58.534523+0000","last_unstale":"2024-06-29T14:41:59.882309+0000","last_undegraded":"2024-06-29T14:41:59.882309+0000","last_fullsized":"2024-06-29T14:41:59.882309+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:46.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883320+0000","last_change":"2024-06-29T14:41:58.539286+0000","last_active":"2024-06-29T14:41:59.883320+0000","last_peered":"2024-06-29T14:41:59.883320+0000","last_clean":"2024-06-29T14:41:59.883320+0000","last_became_active":"2024-06-29T14:41:58.538908+0000","last_became_peered":"2024-06-29T14:41:58.538908+0000","last_unstale":"2024-06-29T14:41:59.883320+0000","last_undegraded":"2024-06-29T14:41:59.883320+0000","last_fullsized":"2024-06-29T14:41:59.883320+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:32.367186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518569+0000","last_change":"2024-06-29T14:41:58.546260+0000","last_active":"2024-06-29T14:41:59.518569+0000","last_peered":"2024-06-29T14:41:59.518569+0000","last_clean":"2024-06-29T14:41:59.518569+0000","last_became_active":"2024-06-29T14:41:58.545761+0000","last_became_peered":"2024-06-29T14:41:58.545761+0000","last_unstale":"2024-06-29T14:41:59.518569+0000","last_undegraded":"2024-06-29T14:41:59.518569+0000","last_fullsized":"2024-06-29T14:41:59.518569+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:59.682693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882464+0000","last_change":"2024-06-29T14:41:58.526825+0000","last_active":"2024-06-29T14:41:59.882464+0000","last_peered":"2024-06-29T14:41:59.882464+0000","last_clean":"2024-06-29T14:41:59.882464+0000","last_became_active":"2024-06-29T14:41:58.525395+0000","last_became_peered":"2024-06-29T14:41:58.525395+0000","last_unstale":"2024-06-29T14:41:59.882464+0000","last_undegraded":"2024-06-29T14:41:59.882464+0000","last_fullsized":"2024-06-29T14:41:59.882464+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:47.467202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518227+0000","last_change":"2024-06-29T14:41:58.534385+0000","last_active":"2024-06-29T14:41:59.518227+0000","last_peered":"2024-06-29T14:41:59.518227+0000","last_clean":"2024-06-29T14:41:59.518227+0000","last_became_active":"2024-06-29T14:41:58.534050+0000","last_became_peered":"2024-06-29T14:41:58.534050+0000","last_unstale":"2024-06-29T14:41:59.518227+0000","last_undegraded":"2024-06-29T14:41:59.518227+0000","last_fullsized":"2024-06-29T14:41:59.518227+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:47:09.101548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882938+0000","last_change":"2024-06-29T14:41:58.527469+0000","last_active":"2024-06-29T14:41:59.882938+0000","last_peered":"2024-06-29T14:41:59.882938+0000","last_clean":"2024-06-29T14:41:59.882938+0000","last_became_active":"2024-06-29T14:41:58.527150+0000","last_became_peered":"2024-06-29T14:41:58.527150+0000","last_unstale":"2024-06-29T14:41:59.882938+0000","last_undegraded":"2024-06-29T14:41:59.882938+0000","last_fullsized":"2024-06-29T14:41:59.882938+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:22:05.186180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520031+0000","last_change":"2024-06-29T14:41:58.540443+0000","last_active":"2024-06-29T14:41:59.520031+0000","last_peered":"2024-06-29T14:41:59.520031+0000","last_clean":"2024-06-29T14:41:59.520031+0000","last_became_active":"2024-06-29T14:41:58.540085+0000","last_became_peered":"2024-06-29T14:41:58.540085+0000","last_unstale":"2024-06-29T14:41:59.520031+0000","last_undegraded":"2024-06-29T14:41:59.520031+0000","last_fullsized":"2024-06-29T14:41:59.520031+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:25:01.129268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882745+0000","last_change":"2024-06-29T14:41:58.540558+0000","last_active":"2024-06-29T14:41:59.882745+0000","last_peered":"2024-06-29T14:41:59.882745+0000","last_clean":"2024-06-29T14:41:59.882745+0000","last_became_active":"2024-06-29T14:41:58.540258+0000","last_became_peered":"2024-06-29T14:41:58.540258+0000","last_unstale":"2024-06-29T14:41:59.882745+0000","last_undegraded":"2024-06-29T14:41:59.882745+0000","last_fullsized":"2024-06-29T14:41:59.882745+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:06.432760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882473+0000","last_change":"2024-06-29T14:41:58.544350+0000","last_active":"2024-06-29T14:41:59.882473+0000","last_peered":"2024-06-29T14:41:59.882473+0000","last_clean":"2024-06-29T14:41:59.882473+0000","last_became_active":"2024-06-29T14:41:58.544036+0000","last_became_peered":"2024-06-29T14:41:58.544036+0000","last_unstale":"2024-06-29T14:41:59.882473+0000","last_undegraded":"2024-06-29T14:41:59.882473+0000","last_fullsized":"2024-06-29T14:41:59.882473+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:19.408347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882362+0000","last_change":"2024-06-29T14:41:58.528180+0000","last_active":"2024-06-29T14:41:59.882362+0000","last_peered":"2024-06-29T14:41:59.882362+0000","last_clean":"2024-06-29T14:41:59.882362+0000","last_became_active":"2024-06-29T14:41:58.527870+0000","last_became_peered":"2024-06-29T14:41:58.527870+0000","last_unstale":"2024-06-29T14:41:59.882362+0000","last_undegraded":"2024-06-29T14:41:59.882362+0000","last_fullsized":"2024-06-29T14:41:59.882362+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:42:25.349931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883369+0000","last_change":"2024-06-29T14:41:58.544672+0000","last_active":"2024-06-29T14:41:59.883369+0000","last_peered":"2024-06-29T14:41:59.883369+0000","last_clean":"2024-06-29T14:41:59.883369+0000","last_became_active":"2024-06-29T14:41:58.544367+0000","last_became_peered":"2024-06-29T14:41:58.544367+0000","last_unstale":"2024-06-29T14:41:59.883369+0000","last_undegraded":"2024-06-29T14:41:59.883369+0000","last_fullsized":"2024-06-29T14:41:59.883369+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:45:32.977469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882143+0000","last_change":"2024-06-29T14:41:58.536979+0000","last_active":"2024-06-29T14:41:59.882143+0000","last_peered":"2024-06-29T14:41:59.882143+0000","last_clean":"2024-06-29T14:41:59.882143+0000","last_became_active":"2024-06-29T14:41:58.536678+0000","last_became_peered":"2024-06-29T14:41:58.536678+0000","last_unstale":"2024-06-29T14:41:59.882143+0000","last_undegraded":"2024-06-29T14:41:59.882143+0000","last_fullsized":"2024-06-29T14:41:59.882143+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:01.757137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883378+0000","last_change":"2024-06-29T14:41:58.536594+0000","last_active":"2024-06-29T14:41:59.883378+0000","last_peered":"2024-06-29T14:41:59.883378+0000","last_clean":"2024-06-29T14:41:59.883378+0000","last_became_active":"2024-06-29T14:41:58.536247+0000","last_became_peered":"2024-06-29T14:41:58.536247+0000","last_unstale":"2024-06-29T14:41:59.883378+0000","last_undegraded":"2024-06-29T14:41:59.883378+0000","last_fullsized":"2024-06-29T14:41:59.883378+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:00.046159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882675+0000","last_change":"2024-06-29T14:41:58.539607+0000","last_active":"2024-06-29T14:41:59.882675+0000","last_peered":"2024-06-29T14:41:59.882675+0000","last_clean":"2024-06-29T14:41:59.882675+0000","last_became_active":"2024-06-29T14:41:58.539306+0000","last_became_peered":"2024-06-29T14:41:58.539306+0000","last_unstale":"2024-06-29T14:41:59.882675+0000","last_undegraded":"2024-06-29T14:41:59.882675+0000","last_fullsized":"2024-06-29T14:41:59.882675+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:17:42.624622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882211+0000","last_change":"2024-06-29T14:41:58.539688+0000","last_active":"2024-06-29T14:41:59.882211+0000","last_peered":"2024-06-29T14:41:59.882211+0000","last_clean":"2024-06-29T14:41:59.882211+0000","last_became_active":"2024-06-29T14:41:58.539379+0000","last_became_peered":"2024-06-29T14:41:58.539379+0000","last_unstale":"2024-06-29T14:41:59.882211+0000","last_undegraded":"2024-06-29T14:41:59.882211+0000","last_fullsized":"2024-06-29T14:41:59.882211+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:26.344464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882932+0000","last_change":"2024-06-29T14:41:58.528088+0000","last_active":"2024-06-29T14:41:59.882932+0000","last_peered":"2024-06-29T14:41:59.882932+0000","last_clean":"2024-06-29T14:41:59.882932+0000","last_became_active":"2024-06-29T14:41:58.527798+0000","last_became_peered":"2024-06-29T14:41:58.527798+0000","last_unstale":"2024-06-29T14:41:59.882932+0000","last_undegraded":"2024-06-29T14:41:59.882932+0000","last_fullsized":"2024-06-29T14:41:59.882932+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:53:15.415717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518388+0000","last_change":"2024-06-29T14:41:58.546508+0000","last_active":"2024-06-29T14:41:59.518388+0000","last_peered":"2024-06-29T14:41:59.518388+0000","last_clean":"2024-06-29T14:41:59.518388+0000","last_became_active":"2024-06-29T14:41:58.546065+0000","last_became_peered":"2024-06-29T14:41:58.546065+0000","last_unstale":"2024-06-29T14:41:59.518388+0000","last_undegraded":"2024-06-29T14:41:59.518388+0000","last_fullsized":"2024-06-29T14:41:59.518388+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:06:15.068591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.519965+0000","last_change":"2024-06-29T14:41:58.527897+0000","last_active":"2024-06-29T14:41:59.519965+0000","last_peered":"2024-06-29T14:41:59.519965+0000","last_clean":"2024-06-29T14:41:59.519965+0000","last_became_active":"2024-06-29T14:41:58.527625+0000","last_became_peered":"2024-06-29T14:41:58.527625+0000","last_unstale":"2024-06-29T14:41:59.519965+0000","last_undegraded":"2024-06-29T14:41:59.519965+0000","last_fullsized":"2024-06-29T14:41:59.519965+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:46.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518589+0000","last_change":"2024-06-29T14:41:58.533164+0000","last_active":"2024-06-29T14:41:59.518589+0000","last_peered":"2024-06-29T14:41:59.518589+0000","last_clean":"2024-06-29T14:41:59.518589+0000","last_became_active":"2024-06-29T14:41:58.532865+0000","last_became_peered":"2024-06-29T14:41:58.532865+0000","last_unstale":"2024-06-29T14:41:59.518589+0000","last_undegraded":"2024-06-29T14:41:59.518589+0000","last_fullsized":"2024-06-29T14:41:59.518589+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:53:04.047421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518236+0000","last_change":"2024-06-29T14:41:58.540722+0000","last_active":"2024-06-29T14:41:59.518236+0000","last_peered":"2024-06-29T14:41:59.518236+0000","last_clean":"2024-06-29T14:41:59.518236+0000","last_became_active":"2024-06-29T14:41:58.540393+0000","last_became_peered":"2024-06-29T14:41:58.540393+0000","last_unstale":"2024-06-29T14:41:59.518236+0000","last_undegraded":"2024-06-29T14:41:59.518236+0000","last_fullsized":"2024-06-29T14:41:59.518236+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:01.707346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882530+0000","last_change":"2024-06-29T14:41:58.544502+0000","last_active":"2024-06-29T14:41:59.882530+0000","last_peered":"2024-06-29T14:41:59.882530+0000","last_clean":"2024-06-29T14:41:59.882530+0000","last_became_active":"2024-06-29T14:41:58.543891+0000","last_became_peered":"2024-06-29T14:41:58.543891+0000","last_unstale":"2024-06-29T14:41:59.882530+0000","last_undegraded":"2024-06-29T14:41:59.882530+0000","last_fullsized":"2024-06-29T14:41:59.882530+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:34.239368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.519730+0000","last_change":"2024-06-29T14:41:58.536757+0000","last_active":"2024-06-29T14:41:59.519730+0000","last_peered":"2024-06-29T14:41:59.519730+0000","last_clean":"2024-06-29T14:41:59.519730+0000","last_became_active":"2024-06-29T14:41:58.536424+0000","last_became_peered":"2024-06-29T14:41:58.536424+0000","last_unstale":"2024-06-29T14:41:59.519730+0000","last_undegraded":"2024-06-29T14:41:59.519730+0000","last_fullsized":"2024-06-29T14:41:59.519730+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:08:09.235128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883232+0000","last_change":"2024-06-29T14:41:58.535130+0000","last_active":"2024-06-29T14:41:59.883232+0000","last_peered":"2024-06-29T14:41:59.883232+0000","last_clean":"2024-06-29T14:41:59.883232+0000","last_became_active":"2024-06-29T14:41:58.534791+0000","last_became_peered":"2024-06-29T14:41:58.534791+0000","last_unstale":"2024-06-29T14:41:59.883232+0000","last_undegraded":"2024-06-29T14:41:59.883232+0000","last_fullsized":"2024-06-29T14:41:59.883232+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:16.109457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"145'4","reported_seq":18,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883055+0000","last_change":"2024-06-29T14:41:57.526059+0000","last_active":"2024-06-29T14:41:59.883055+0000","last_peered":"2024-06-29T14:41:59.883055+0000","last_clean":"2024-06-29T14:41:59.883055+0000","last_became_active":"2024-06-29T14:41:57.525685+0000","last_became_peered":"2024-06-29T14:41:57.525685+0000","last_unstale":"2024-06-29T14:41:59.883055+0000","last_undegraded":"2024-06-29T14:41:59.883055+0000","last_fullsized":"2024-06-29T14:41:59.883055+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:23.584362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883661+0000","last_change":"2024-06-29T14:41:58.532823+0000","last_active":"2024-06-29T14:41:59.883661+0000","last_peered":"2024-06-29T14:41:59.883661+0000","last_clean":"2024-06-29T14:41:59.883661+0000","last_became_active":"2024-06-29T14:41:58.532491+0000","last_became_peered":"2024-06-29T14:41:58.532491+0000","last_unstale":"2024-06-29T14:41:59.883661+0000","last_undegraded":"2024-06-29T14:41:59.883661+0000","last_fullsized":"2024-06-29T14:41:59.883661+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:11:38.363172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"145'1","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520799+0000","last_change":"2024-06-29T14:41:57.519722+0000","last_active":"2024-06-29T14:41:59.520799+0000","last_peered":"2024-06-29T14:41:59.520799+0000","last_clean":"2024-06-29T14:41:59.520799+0000","last_became_active":"2024-06-29T14:41:57.518971+0000","last_became_peered":"2024-06-29T14:41:57.518971+0000","last_unstale":"2024-06-29T14:41:59.520799+0000","last_undegraded":"2024-06-29T14:41:59.520799+0000","last_fullsized":"2024-06-29T14:41:59.520799+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:42.846558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883445+0000","last_change":"2024-06-29T14:41:58.545635+0000","last_active":"2024-06-29T14:41:59.883445+0000","last_peered":"2024-06-29T14:41:59.883445+0000","last_clean":"2024-06-29T14:41:59.883445+0000","last_became_active":"2024-06-29T14:41:58.545048+0000","last_became_peered":"2024-06-29T14:41:58.545048+0000","last_unstale":"2024-06-29T14:41:59.883445+0000","last_undegraded":"2024-06-29T14:41:59.883445+0000","last_fullsized":"2024-06-29T14:41:59.883445+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:59.751188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882485+0000","last_change":"2024-06-29T14:41:57.520872+0000","last_active":"2024-06-29T14:41:59.882485+0000","last_peered":"2024-06-29T14:41:59.882485+0000","last_clean":"2024-06-29T14:41:59.882485+0000","last_became_active":"2024-06-29T14:41:57.520576+0000","last_became_peered":"2024-06-29T14:41:57.520576+0000","last_unstale":"2024-06-29T14:41:59.882485+0000","last_undegraded":"2024-06-29T14:41:59.882485+0000","last_fullsized":"2024-06-29T14:41:59.882485+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:00.319413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882963+0000","last_change":"2024-06-29T14:41:58.546541+0000","last_active":"2024-06-29T14:41:59.882963+0000","last_peered":"2024-06-29T14:41:59.882963+0000","last_clean":"2024-06-29T14:41:59.882963+0000","last_became_active":"2024-06-29T14:41:58.546220+0000","last_became_peered":"2024-06-29T14:41:58.546220+0000","last_unstale":"2024-06-29T14:41:59.882963+0000","last_undegraded":"2024-06-29T14:41:59.882963+0000","last_fullsized":"2024-06-29T14:41:59.882963+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:41.043970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883116+0000","last_change":"2024-06-29T14:41:57.518675+0000","last_active":"2024-06-29T14:41:59.883116+0000","last_peered":"2024-06-29T14:41:59.883116+0000","last_clean":"2024-06-29T14:41:59.883116+0000","last_became_active":"2024-06-29T14:41:57.518371+0000","last_became_peered":"2024-06-29T14:41:57.518371+0000","last_unstale":"2024-06-29T14:41:59.883116+0000","last_undegraded":"2024-06-29T14:41:59.883116+0000","last_fullsized":"2024-06-29T14:41:59.883116+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:07:25.367733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883239+0000","last_change":"2024-06-29T14:41:58.541060+0000","last_active":"2024-06-29T14:41:59.883239+0000","last_peered":"2024-06-29T14:41:59.883239+0000","last_clean":"2024-06-29T14:41:59.883239+0000","last_became_active":"2024-06-29T14:41:58.540759+0000","last_became_peered":"2024-06-29T14:41:58.540759+0000","last_unstale":"2024-06-29T14:41:59.883239+0000","last_undegraded":"2024-06-29T14:41:59.883239+0000","last_fullsized":"2024-06-29T14:41:59.883239+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:24.591945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"145'1","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882286+0000","last_change":"2024-06-29T14:41:57.510953+0000","last_active":"2024-06-29T14:41:59.882286+0000","last_peered":"2024-06-29T14:41:59.882286+0000","last_clean":"2024-06-29T14:41:59.882286+0000","last_became_active":"2024-06-29T14:41:57.510084+0000","last_became_peered":"2024-06-29T14:41:57.510084+0000","last_unstale":"2024-06-29T14:41:59.882286+0000","last_undegraded":"2024-06-29T14:41:59.882286+0000","last_fullsized":"2024-06-29T14:41:59.882286+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:10:35.137757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520441+0000","last_change":"2024-06-29T14:41:58.542805+0000","last_active":"2024-06-29T14:41:59.520441+0000","last_peered":"2024-06-29T14:41:59.520441+0000","last_clean":"2024-06-29T14:41:59.520441+0000","last_became_active":"2024-06-29T14:41:58.542469+0000","last_became_peered":"2024-06-29T14:41:58.542469+0000","last_unstale":"2024-06-29T14:41:59.520441+0000","last_undegraded":"2024-06-29T14:41:59.520441+0000","last_fullsized":"2024-06-29T14:41:59.520441+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:59.206644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"145'1","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883355+0000","last_change":"2024-06-29T14:41:57.513970+0000","last_active":"2024-06-29T14:41:59.883355+0000","last_peered":"2024-06-29T14:41:59.883355+0000","last_clean":"2024-06-29T14:41:59.883355+0000","last_became_active":"2024-06-29T14:41:57.513369+0000","last_became_peered":"2024-06-29T14:41:57.513369+0000","last_unstale":"2024-06-29T14:41:59.883355+0000","last_undegraded":"2024-06-29T14:41:59.883355+0000","last_fullsized":"2024-06-29T14:41:59.883355+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:06:57.320277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883189+0000","last_change":"2024-06-29T14:41:58.528742+0000","last_active":"2024-06-29T14:41:59.883189+0000","last_peered":"2024-06-29T14:41:59.883189+0000","last_clean":"2024-06-29T14:41:59.883189+0000","last_became_active":"2024-06-29T14:41:58.528491+0000","last_became_peered":"2024-06-29T14:41:58.528491+0000","last_unstale":"2024-06-29T14:41:59.883189+0000","last_undegraded":"2024-06-29T14:41:59.883189+0000","last_fullsized":"2024-06-29T14:41:59.883189+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:55:33.110209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520124+0000","last_change":"2024-06-29T14:41:57.524848+0000","last_active":"2024-06-29T14:41:59.520124+0000","last_peered":"2024-06-29T14:41:59.520124+0000","last_clean":"2024-06-29T14:41:59.520124+0000","last_became_active":"2024-06-29T14:41:57.524534+0000","last_became_peered":"2024-06-29T14:41:57.524534+0000","last_unstale":"2024-06-29T14:41:59.520124+0000","last_undegraded":"2024-06-29T14:41:59.520124+0000","last_fullsized":"2024-06-29T14:41:59.520124+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:27.570333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882499+0000","last_change":"2024-06-29T14:41:58.525953+0000","last_active":"2024-06-29T14:41:59.882499+0000","last_peered":"2024-06-29T14:41:59.882499+0000","last_clean":"2024-06-29T14:41:59.882499+0000","last_became_active":"2024-06-29T14:41:58.525654+0000","last_became_peered":"2024-06-29T14:41:58.525654+0000","last_unstale":"2024-06-29T14:41:59.882499+0000","last_undegraded":"2024-06-29T14:41:59.882499+0000","last_fullsized":"2024-06-29T14:41:59.882499+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:55:33.110209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882012+0000","last_change":"2024-06-29T14:41:57.512123+0000","last_active":"2024-06-29T14:41:59.882012+0000","last_peered":"2024-06-29T14:41:59.882012+0000","last_clean":"2024-06-29T14:41:59.882012+0000","last_became_active":"2024-06-29T14:41:57.511804+0000","last_became_peered":"2024-06-29T14:41:57.511804+0000","last_unstale":"2024-06-29T14:41:59.882012+0000","last_undegraded":"2024-06-29T14:41:59.882012+0000","last_fullsized":"2024-06-29T14:41:59.882012+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:06.908709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518762+0000","last_change":"2024-06-29T14:41:58.532335+0000","last_active":"2024-06-29T14:41:59.518762+0000","last_peered":"2024-06-29T14:41:59.518762+0000","last_clean":"2024-06-29T14:41:59.518762+0000","last_became_active":"2024-06-29T14:41:58.531650+0000","last_became_peered":"2024-06-29T14:41:58.531650+0000","last_unstale":"2024-06-29T14:41:59.518762+0000","last_undegraded":"2024-06-29T14:41:59.518762+0000","last_fullsized":"2024-06-29T14:41:59.518762+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:03.210487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520233+0000","last_change":"2024-06-29T14:41:57.517632+0000","last_active":"2024-06-29T14:41:59.520233+0000","last_peered":"2024-06-29T14:41:59.520233+0000","last_clean":"2024-06-29T14:41:59.520233+0000","last_became_active":"2024-06-29T14:41:57.517317+0000","last_became_peered":"2024-06-29T14:41:57.517317+0000","last_unstale":"2024-06-29T14:41:59.520233+0000","last_undegraded":"2024-06-29T14:41:59.520233+0000","last_fullsized":"2024-06-29T14:41:59.520233+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:08:08.227545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520514+0000","last_change":"2024-06-29T14:41:58.529284+0000","last_active":"2024-06-29T14:41:59.520514+0000","last_peered":"2024-06-29T14:41:59.520514+0000","last_clean":"2024-06-29T14:41:59.520514+0000","last_became_active":"2024-06-29T14:41:58.528902+0000","last_became_peered":"2024-06-29T14:41:58.528902+0000","last_unstale":"2024-06-29T14:41:59.520514+0000","last_undegraded":"2024-06-29T14:41:59.520514+0000","last_fullsized":"2024-06-29T14:41:59.520514+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:31:24.394240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883345+0000","last_change":"2024-06-29T14:41:57.518573+0000","last_active":"2024-06-29T14:41:59.883345+0000","last_peered":"2024-06-29T14:41:59.883345+0000","last_clean":"2024-06-29T14:41:59.883345+0000","last_became_active":"2024-06-29T14:41:57.518295+0000","last_became_peered":"2024-06-29T14:41:57.518295+0000","last_unstale":"2024-06-29T14:41:59.883345+0000","last_undegraded":"2024-06-29T14:41:59.883345+0000","last_fullsized":"2024-06-29T14:41:59.883345+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:34.767077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883501+0000","last_change":"2024-06-29T14:41:58.547434+0000","last_active":"2024-06-29T14:41:59.883501+0000","last_peered":"2024-06-29T14:41:59.883501+0000","last_clean":"2024-06-29T14:41:59.883501+0000","last_became_active":"2024-06-29T14:41:58.547127+0000","last_became_peered":"2024-06-29T14:41:58.547127+0000","last_unstale":"2024-06-29T14:41:59.883501+0000","last_undegraded":"2024-06-29T14:41:59.883501+0000","last_fullsized":"2024-06-29T14:41:59.883501+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:52:57.483931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"145'3","reported_seq":17,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.521126+0000","last_change":"2024-06-29T14:41:57.515647+0000","last_active":"2024-06-29T14:41:59.521126+0000","last_peered":"2024-06-29T14:41:59.521126+0000","last_clean":"2024-06-29T14:41:59.521126+0000","last_became_active":"2024-06-29T14:41:57.515330+0000","last_became_peered":"2024-06-29T14:41:57.515330+0000","last_unstale":"2024-06-29T14:41:59.521126+0000","last_undegraded":"2024-06-29T14:41:59.521126+0000","last_fullsized":"2024-06-29T14:41:59.521126+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:19.877654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883135+0000","last_change":"2024-06-29T14:41:58.550259+0000","last_active":"2024-06-29T14:41:59.883135+0000","last_peered":"2024-06-29T14:41:59.883135+0000","last_clean":"2024-06-29T14:41:59.883135+0000","last_became_active":"2024-06-29T14:41:58.549955+0000","last_became_peered":"2024-06-29T14:41:58.549955+0000","last_unstale":"2024-06-29T14:41:59.883135+0000","last_undegraded":"2024-06-29T14:41:59.883135+0000","last_fullsized":"2024-06-29T14:41:59.883135+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:07:28.188343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"145'6","reported_seq":20,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883036+0000","last_change":"2024-06-29T14:41:57.524442+0000","last_active":"2024-06-29T14:41:59.883036+0000","last_peered":"2024-06-29T14:41:59.883036+0000","last_clean":"2024-06-29T14:41:59.883036+0000","last_became_active":"2024-06-29T14:41:57.524028+0000","last_became_peered":"2024-06-29T14:41:57.524028+0000","last_unstale":"2024-06-29T14:41:59.883036+0000","last_undegraded":"2024-06-29T14:41:59.883036+0000","last_fullsized":"2024-06-29T14:41:59.883036+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:47:08.093965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518722+0000","last_change":"2024-06-29T14:41:58.532438+0000","last_active":"2024-06-29T14:41:59.518722+0000","last_peered":"2024-06-29T14:41:59.518722+0000","last_clean":"2024-06-29T14:41:59.518722+0000","last_became_active":"2024-06-29T14:41:58.532097+0000","last_became_peered":"2024-06-29T14:41:58.532097+0000","last_unstale":"2024-06-29T14:41:59.518722+0000","last_undegraded":"2024-06-29T14:41:59.518722+0000","last_fullsized":"2024-06-29T14:41:59.518722+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:21.706510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518336+0000","last_change":"2024-06-29T14:41:57.515771+0000","last_active":"2024-06-29T14:41:59.518336+0000","last_peered":"2024-06-29T14:41:59.518336+0000","last_clean":"2024-06-29T14:41:59.518336+0000","last_became_active":"2024-06-29T14:41:57.515422+0000","last_became_peered":"2024-06-29T14:41:57.515422+0000","last_unstale":"2024-06-29T14:41:59.518336+0000","last_undegraded":"2024-06-29T14:41:59.518336+0000","last_fullsized":"2024-06-29T14:41:59.518336+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:13:52.871547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.881892+0000","last_change":"2024-06-29T14:41:58.540270+0000","last_active":"2024-06-29T14:41:59.881892+0000","last_peered":"2024-06-29T14:41:59.881892+0000","last_clean":"2024-06-29T14:41:59.881892+0000","last_became_active":"2024-06-29T14:41:58.539956+0000","last_became_peered":"2024-06-29T14:41:58.539956+0000","last_unstale":"2024-06-29T14:41:59.881892+0000","last_undegraded":"2024-06-29T14:41:59.881892+0000","last_fullsized":"2024-06-29T14:41:59.881892+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:21.706510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882809+0000","last_change":"2024-06-29T14:41:57.522710+0000","last_active":"2024-06-29T14:41:59.882809+0000","last_peered":"2024-06-29T14:41:59.882809+0000","last_clean":"2024-06-29T14:41:59.882809+0000","last_became_active":"2024-06-29T14:41:57.522434+0000","last_became_peered":"2024-06-29T14:41:57.522434+0000","last_unstale":"2024-06-29T14:41:59.882809+0000","last_undegraded":"2024-06-29T14:41:59.882809+0000","last_fullsized":"2024-06-29T14:41:59.882809+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:41.277655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883150+0000","last_change":"2024-06-29T14:41:58.538420+0000","last_active":"2024-06-29T14:41:59.883150+0000","last_peered":"2024-06-29T14:41:59.883150+0000","last_clean":"2024-06-29T14:41:59.883150+0000","last_became_active":"2024-06-29T14:41:58.537911+0000","last_became_peered":"2024-06-29T14:41:58.537911+0000","last_unstale":"2024-06-29T14:41:59.883150+0000","last_undegraded":"2024-06-29T14:41:59.883150+0000","last_fullsized":"2024-06-29T14:41:59.883150+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:13:47.467202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"145'8","reported_seq":22,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882751+0000","last_change":"2024-06-29T14:41:57.517845+0000","last_active":"2024-06-29T14:41:59.882751+0000","last_peered":"2024-06-29T14:41:59.882751+0000","last_clean":"2024-06-29T14:41:59.882751+0000","last_became_active":"2024-06-29T14:41:57.517533+0000","last_became_peered":"2024-06-29T14:41:57.517533+0000","last_unstale":"2024-06-29T14:41:59.882751+0000","last_undegraded":"2024-06-29T14:41:59.882751+0000","last_fullsized":"2024-06-29T14:41:59.882751+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:33:00.699763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883715+0000","last_change":"2024-06-29T14:41:58.532290+0000","last_active":"2024-06-29T14:41:59.883715+0000","last_peered":"2024-06-29T14:41:59.883715+0000","last_clean":"2024-06-29T14:41:59.883715+0000","last_became_active":"2024-06-29T14:41:58.531945+0000","last_became_peered":"2024-06-29T14:41:58.531945+0000","last_unstale":"2024-06-29T14:41:59.883715+0000","last_undegraded":"2024-06-29T14:41:59.883715+0000","last_fullsized":"2024-06-29T14:41:59.883715+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:21:34.989370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"145'1","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520369+0000","last_change":"2024-06-29T14:41:57.521869+0000","last_active":"2024-06-29T14:41:59.520369+0000","last_peered":"2024-06-29T14:41:59.520369+0000","last_clean":"2024-06-29T14:41:59.520369+0000","last_became_active":"2024-06-29T14:41:57.521295+0000","last_became_peered":"2024-06-29T14:41:57.521295+0000","last_unstale":"2024-06-29T14:41:59.520369+0000","last_undegraded":"2024-06-29T14:41:59.520369+0000","last_fullsized":"2024-06-29T14:41:59.520369+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:29.168190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520088+0000","last_change":"2024-06-29T14:41:58.540935+0000","last_active":"2024-06-29T14:41:59.520088+0000","last_peered":"2024-06-29T14:41:59.520088+0000","last_clean":"2024-06-29T14:41:59.520088+0000","last_became_active":"2024-06-29T14:41:58.535673+0000","last_became_peered":"2024-06-29T14:41:58.535673+0000","last_unstale":"2024-06-29T14:41:59.520088+0000","last_undegraded":"2024-06-29T14:41:59.520088+0000","last_fullsized":"2024-06-29T14:41:59.520088+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:32:19.408347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883318+0000","last_change":"2024-06-29T14:41:57.521214+0000","last_active":"2024-06-29T14:41:59.883318+0000","last_peered":"2024-06-29T14:41:59.883318+0000","last_clean":"2024-06-29T14:41:59.883318+0000","last_became_active":"2024-06-29T14:41:57.520868+0000","last_became_peered":"2024-06-29T14:41:57.520868+0000","last_unstale":"2024-06-29T14:41:59.883318+0000","last_undegraded":"2024-06-29T14:41:59.883318+0000","last_fullsized":"2024-06-29T14:41:59.883318+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:25.592455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882539+0000","last_change":"2024-06-29T14:41:58.530559+0000","last_active":"2024-06-29T14:41:59.882539+0000","last_peered":"2024-06-29T14:41:59.882539+0000","last_clean":"2024-06-29T14:41:59.882539+0000","last_became_active":"2024-06-29T14:41:58.530236+0000","last_became_peered":"2024-06-29T14:41:58.530236+0000","last_unstale":"2024-06-29T14:41:59.882539+0000","last_undegraded":"2024-06-29T14:41:59.882539+0000","last_fullsized":"2024-06-29T14:41:59.882539+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:27:36.015471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"145'1","reported_seq":15,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.521316+0000","last_change":"2024-06-29T14:41:57.517241+0000","last_active":"2024-06-29T14:41:59.521316+0000","last_peered":"2024-06-29T14:41:59.521316+0000","last_clean":"2024-06-29T14:41:59.521316+0000","last_became_active":"2024-06-29T14:41:57.516919+0000","last_became_peered":"2024-06-29T14:41:57.516919+0000","last_unstale":"2024-06-29T14:41:59.521316+0000","last_undegraded":"2024-06-29T14:41:59.521316+0000","last_fullsized":"2024-06-29T14:41:59.521316+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:18.721939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882336+0000","last_change":"2024-06-29T14:41:58.529352+0000","last_active":"2024-06-29T14:41:59.882336+0000","last_peered":"2024-06-29T14:41:59.882336+0000","last_clean":"2024-06-29T14:41:59.882336+0000","last_became_active":"2024-06-29T14:41:58.529058+0000","last_became_peered":"2024-06-29T14:41:58.529058+0000","last_unstale":"2024-06-29T14:41:59.882336+0000","last_undegraded":"2024-06-29T14:41:59.882336+0000","last_fullsized":"2024-06-29T14:41:59.882336+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:50:32.024937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520080+0000","last_change":"2024-06-29T14:41:57.518748+0000","last_active":"2024-06-29T14:41:59.520080+0000","last_peered":"2024-06-29T14:41:59.520080+0000","last_clean":"2024-06-29T14:41:59.520080+0000","last_became_active":"2024-06-29T14:41:57.518413+0000","last_became_peered":"2024-06-29T14:41:57.518413+0000","last_unstale":"2024-06-29T14:41:59.520080+0000","last_undegraded":"2024-06-29T14:41:59.520080+0000","last_fullsized":"2024-06-29T14:41:59.520080+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:00.319413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518287+0000","last_change":"2024-06-29T14:41:58.545198+0000","last_active":"2024-06-29T14:41:59.518287+0000","last_peered":"2024-06-29T14:41:59.518287+0000","last_clean":"2024-06-29T14:41:59.518287+0000","last_became_active":"2024-06-29T14:41:58.544883+0000","last_became_peered":"2024-06-29T14:41:58.544883+0000","last_unstale":"2024-06-29T14:41:59.518287+0000","last_undegraded":"2024-06-29T14:41:59.518287+0000","last_fullsized":"2024-06-29T14:41:59.518287+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:44:47.616020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520607+0000","last_change":"2024-06-29T14:41:57.514665+0000","last_active":"2024-06-29T14:41:59.520607+0000","last_peered":"2024-06-29T14:41:59.520607+0000","last_clean":"2024-06-29T14:41:59.520607+0000","last_became_active":"2024-06-29T14:41:57.514340+0000","last_became_peered":"2024-06-29T14:41:57.514340+0000","last_unstale":"2024-06-29T14:41:59.520607+0000","last_undegraded":"2024-06-29T14:41:59.520607+0000","last_fullsized":"2024-06-29T14:41:59.520607+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:16:25.592455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520639+0000","last_change":"2024-06-29T14:41:58.545545+0000","last_active":"2024-06-29T14:41:59.520639+0000","last_peered":"2024-06-29T14:41:59.520639+0000","last_clean":"2024-06-29T14:41:59.520639+0000","last_became_active":"2024-06-29T14:41:58.545128+0000","last_became_peered":"2024-06-29T14:41:58.545128+0000","last_unstale":"2024-06-29T14:41:59.520639+0000","last_undegraded":"2024-06-29T14:41:59.520639+0000","last_fullsized":"2024-06-29T14:41:59.520639+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:33:36.000748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883425+0000","last_change":"2024-06-29T14:41:57.514209+0000","last_active":"2024-06-29T14:41:59.883425+0000","last_peered":"2024-06-29T14:41:59.883425+0000","last_clean":"2024-06-29T14:41:59.883425+0000","last_became_active":"2024-06-29T14:41:57.512897+0000","last_became_peered":"2024-06-29T14:41:57.512897+0000","last_unstale":"2024-06-29T14:41:59.883425+0000","last_undegraded":"2024-06-29T14:41:59.883425+0000","last_fullsized":"2024-06-29T14:41:59.883425+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:04:29.122336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.881888+0000","last_change":"2024-06-29T14:41:58.531912+0000","last_active":"2024-06-29T14:41:59.881888+0000","last_peered":"2024-06-29T14:41:59.881888+0000","last_clean":"2024-06-29T14:41:59.881888+0000","last_became_active":"2024-06-29T14:41:58.531566+0000","last_became_peered":"2024-06-29T14:41:58.531566+0000","last_unstale":"2024-06-29T14:41:59.881888+0000","last_undegraded":"2024-06-29T14:41:59.881888+0000","last_fullsized":"2024-06-29T14:41:59.881888+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:53:04.047421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882705+0000","last_change":"2024-06-29T14:41:57.515401+0000","last_active":"2024-06-29T14:41:59.882705+0000","last_peered":"2024-06-29T14:41:59.882705+0000","last_clean":"2024-06-29T14:41:59.882705+0000","last_became_active":"2024-06-29T14:41:57.514805+0000","last_became_peered":"2024-06-29T14:41:57.514805+0000","last_unstale":"2024-06-29T14:41:59.882705+0000","last_undegraded":"2024-06-29T14:41:59.882705+0000","last_fullsized":"2024-06-29T14:41:59.882705+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:46:26.472467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883306+0000","last_change":"2024-06-29T14:41:58.544238+0000","last_active":"2024-06-29T14:41:59.883306+0000","last_peered":"2024-06-29T14:41:59.883306+0000","last_clean":"2024-06-29T14:41:59.883306+0000","last_became_active":"2024-06-29T14:41:58.543507+0000","last_became_peered":"2024-06-29T14:41:58.543507+0000","last_unstale":"2024-06-29T14:41:59.883306+0000","last_undegraded":"2024-06-29T14:41:59.883306+0000","last_fullsized":"2024-06-29T14:41:59.883306+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:48:43.997072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882783+0000","last_change":"2024-06-29T14:41:57.520705+0000","last_active":"2024-06-29T14:41:59.882783+0000","last_peered":"2024-06-29T14:41:59.882783+0000","last_clean":"2024-06-29T14:41:59.882783+0000","last_became_active":"2024-06-29T14:41:57.520343+0000","last_became_peered":"2024-06-29T14:41:57.520343+0000","last_unstale":"2024-06-29T14:41:59.882783+0000","last_undegraded":"2024-06-29T14:41:59.882783+0000","last_fullsized":"2024-06-29T14:41:59.882783+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:08:08.227545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.521045+0000","last_change":"2024-06-29T14:41:58.541990+0000","last_active":"2024-06-29T14:41:59.521045+0000","last_peered":"2024-06-29T14:41:59.521045+0000","last_clean":"2024-06-29T14:41:59.521045+0000","last_became_active":"2024-06-29T14:41:58.541653+0000","last_became_peered":"2024-06-29T14:41:58.541653+0000","last_unstale":"2024-06-29T14:41:59.521045+0000","last_undegraded":"2024-06-29T14:41:59.521045+0000","last_fullsized":"2024-06-29T14:41:59.521045+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:48:01.757137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518965+0000","last_change":"2024-06-29T14:41:57.516375+0000","last_active":"2024-06-29T14:41:59.518965+0000","last_peered":"2024-06-29T14:41:59.518965+0000","last_clean":"2024-06-29T14:41:59.518965+0000","last_became_active":"2024-06-29T14:41:57.516069+0000","last_became_peered":"2024-06-29T14:41:57.516069+0000","last_unstale":"2024-06-29T14:41:59.518965+0000","last_undegraded":"2024-06-29T14:41:59.518965+0000","last_fullsized":"2024-06-29T14:41:59.518965+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:03:34.805136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.518854+0000","last_change":"2024-06-29T14:41:58.533836+0000","last_active":"2024-06-29T14:41:59.518854+0000","last_peered":"2024-06-29T14:41:59.518854+0000","last_clean":"2024-06-29T14:41:59.518854+0000","last_became_active":"2024-06-29T14:41:58.533541+0000","last_became_peered":"2024-06-29T14:41:58.533541+0000","last_unstale":"2024-06-29T14:41:59.518854+0000","last_undegraded":"2024-06-29T14:41:59.518854+0000","last_fullsized":"2024-06-29T14:41:59.518854+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:04:22.876673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883245+0000","last_change":"2024-06-29T14:41:57.525920+0000","last_active":"2024-06-29T14:41:59.883245+0000","last_peered":"2024-06-29T14:41:59.883245+0000","last_clean":"2024-06-29T14:41:59.883245+0000","last_became_active":"2024-06-29T14:41:57.525591+0000","last_became_peered":"2024-06-29T14:41:57.525591+0000","last_unstale":"2024-06-29T14:41:59.883245+0000","last_undegraded":"2024-06-29T14:41:59.883245+0000","last_fullsized":"2024-06-29T14:41:59.883245+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:27:15.101874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883264+0000","last_change":"2024-06-29T14:41:58.539751+0000","last_active":"2024-06-29T14:41:59.883264+0000","last_peered":"2024-06-29T14:41:59.883264+0000","last_clean":"2024-06-29T14:41:59.883264+0000","last_became_active":"2024-06-29T14:41:58.539467+0000","last_became_peered":"2024-06-29T14:41:58.539467+0000","last_unstale":"2024-06-29T14:41:59.883264+0000","last_undegraded":"2024-06-29T14:41:59.883264+0000","last_fullsized":"2024-06-29T14:41:59.883264+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:00.046159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882411+0000","last_change":"2024-06-29T14:41:57.511991+0000","last_active":"2024-06-29T14:41:59.882411+0000","last_peered":"2024-06-29T14:41:59.882411+0000","last_clean":"2024-06-29T14:41:59.882411+0000","last_became_active":"2024-06-29T14:41:57.511642+0000","last_became_peered":"2024-06-29T14:41:57.511642+0000","last_unstale":"2024-06-29T14:41:59.882411+0000","last_undegraded":"2024-06-29T14:41:59.882411+0000","last_fullsized":"2024-06-29T14:41:59.882411+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:17:03.454009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.521291+0000","last_change":"2024-06-29T14:41:58.543244+0000","last_active":"2024-06-29T14:41:59.521291+0000","last_peered":"2024-06-29T14:41:59.521291+0000","last_clean":"2024-06-29T14:41:59.521291+0000","last_became_active":"2024-06-29T14:41:58.542928+0000","last_became_peered":"2024-06-29T14:41:58.542928+0000","last_unstale":"2024-06-29T14:41:59.521291+0000","last_undegraded":"2024-06-29T14:41:59.521291+0000","last_fullsized":"2024-06-29T14:41:59.521291+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T14:51:46.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883306+0000","last_change":"2024-06-29T14:41:57.520297+0000","last_active":"2024-06-29T14:41:59.883306+0000","last_peered":"2024-06-29T14:41:59.883306+0000","last_clean":"2024-06-29T14:41:59.883306+0000","last_became_active":"2024-06-29T14:41:57.519802+0000","last_became_peered":"2024-06-29T14:41:57.519802+0000","last_unstale":"2024-06-29T14:41:59.883306+0000","last_undegraded":"2024-06-29T14:41:59.883306+0000","last_fullsized":"2024-06-29T14:41:59.883306+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:42.846558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882608+0000","last_change":"2024-06-29T14:41:58.546419+0000","last_active":"2024-06-29T14:41:59.882608+0000","last_peered":"2024-06-29T14:41:59.882608+0000","last_clean":"2024-06-29T14:41:59.882608+0000","last_became_active":"2024-06-29T14:41:58.546130+0000","last_became_peered":"2024-06-29T14:41:58.546130+0000","last_unstale":"2024-06-29T14:41:59.882608+0000","last_undegraded":"2024-06-29T14:41:59.882608+0000","last_fullsized":"2024-06-29T14:41:59.882608+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:37.484966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882393+0000","last_change":"2024-06-29T14:41:57.523618+0000","last_active":"2024-06-29T14:41:59.882393+0000","last_peered":"2024-06-29T14:41:59.882393+0000","last_clean":"2024-06-29T14:41:59.882393+0000","last_became_active":"2024-06-29T14:41:57.523323+0000","last_became_peered":"2024-06-29T14:41:57.523323+0000","last_unstale":"2024-06-29T14:41:59.882393+0000","last_undegraded":"2024-06-29T14:41:59.882393+0000","last_fullsized":"2024-06-29T14:41:59.882393+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:12:36.272661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.520468+0000","last_change":"2024-06-29T14:41:58.527521+0000","last_active":"2024-06-29T14:41:59.520468+0000","last_peered":"2024-06-29T14:41:59.520468+0000","last_clean":"2024-06-29T14:41:59.520468+0000","last_became_active":"2024-06-29T14:41:58.527196+0000","last_became_peered":"2024-06-29T14:41:58.527196+0000","last_unstale":"2024-06-29T14:41:59.520468+0000","last_undegraded":"2024-06-29T14:41:59.520468+0000","last_fullsized":"2024-06-29T14:41:59.520468+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:17:42.624622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882745+0000","last_change":"2024-06-29T14:41:57.527887+0000","last_active":"2024-06-29T14:41:59.882745+0000","last_peered":"2024-06-29T14:41:59.882745+0000","last_clean":"2024-06-29T14:41:59.882745+0000","last_became_active":"2024-06-29T14:41:57.527563+0000","last_became_peered":"2024-06-29T14:41:57.527563+0000","last_unstale":"2024-06-29T14:41:59.882745+0000","last_undegraded":"2024-06-29T14:41:59.882745+0000","last_fullsized":"2024-06-29T14:41:59.882745+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:04.951455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883160+0000","last_change":"2024-06-29T14:41:58.539956+0000","last_active":"2024-06-29T14:41:59.883160+0000","last_peered":"2024-06-29T14:41:59.883160+0000","last_clean":"2024-06-29T14:41:59.883160+0000","last_became_active":"2024-06-29T14:41:58.539658+0000","last_became_peered":"2024-06-29T14:41:58.539658+0000","last_unstale":"2024-06-29T14:41:59.883160+0000","last_undegraded":"2024-06-29T14:41:59.883160+0000","last_fullsized":"2024-06-29T14:41:59.883160+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:42:50.654794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882809+0000","last_change":"2024-06-29T14:41:57.513673+0000","last_active":"2024-06-29T14:41:59.882809+0000","last_peered":"2024-06-29T14:41:59.882809+0000","last_clean":"2024-06-29T14:41:59.882809+0000","last_became_active":"2024-06-29T14:41:57.513343+0000","last_became_peered":"2024-06-29T14:41:57.513343+0000","last_unstale":"2024-06-29T14:41:59.882809+0000","last_undegraded":"2024-06-29T14:41:59.882809+0000","last_fullsized":"2024-06-29T14:41:59.882809+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:59.031255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882677+0000","last_change":"2024-06-29T14:41:58.533051+0000","last_active":"2024-06-29T14:41:59.882677+0000","last_peered":"2024-06-29T14:41:59.882677+0000","last_clean":"2024-06-29T14:41:59.882677+0000","last_became_active":"2024-06-29T14:41:58.532682+0000","last_became_peered":"2024-06-29T14:41:58.532682+0000","last_unstale":"2024-06-29T14:41:59.882677+0000","last_undegraded":"2024-06-29T14:41:59.882677+0000","last_fullsized":"2024-06-29T14:41:59.882677+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:32.367186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882883+0000","last_change":"2024-06-29T14:41:57.521672+0000","last_active":"2024-06-29T14:41:59.882883+0000","last_peered":"2024-06-29T14:41:59.882883+0000","last_clean":"2024-06-29T14:41:59.882883+0000","last_became_active":"2024-06-29T14:41:57.521363+0000","last_became_peered":"2024-06-29T14:41:57.521363+0000","last_unstale":"2024-06-29T14:41:59.882883+0000","last_undegraded":"2024-06-29T14:41:59.882883+0000","last_fullsized":"2024-06-29T14:41:59.882883+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:30:29.168190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882643+0000","last_change":"2024-06-29T14:41:58.542973+0000","last_active":"2024-06-29T14:41:59.882643+0000","last_peered":"2024-06-29T14:41:59.882643+0000","last_clean":"2024-06-29T14:41:59.882643+0000","last_became_active":"2024-06-29T14:41:58.542587+0000","last_became_peered":"2024-06-29T14:41:58.542587+0000","last_unstale":"2024-06-29T14:41:59.882643+0000","last_undegraded":"2024-06-29T14:41:59.882643+0000","last_fullsized":"2024-06-29T14:41:59.882643+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:57.495686+0000","last_clean_scrub_stamp":"2024-06-29T14:41:57.495686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:02:55.136388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"145'2","reported_seq":16,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.883171+0000","last_change":"2024-06-29T14:41:57.525834+0000","last_active":"2024-06-29T14:41:59.883171+0000","last_peered":"2024-06-29T14:41:59.883171+0000","last_clean":"2024-06-29T14:41:59.883171+0000","last_became_active":"2024-06-29T14:41:57.525328+0000","last_became_peered":"2024-06-29T14:41:57.525328+0000","last_unstale":"2024-06-29T14:41:59.883171+0000","last_undegraded":"2024-06-29T14:41:59.883171+0000","last_fullsized":"2024-06-29T14:41:59.883171+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:41:56.488103+0000","last_clean_scrub_stamp":"2024-06-29T14:41:56.488103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:29:38.029430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"135'156","reported_seq":653,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-06-29T14:41:59.882410+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T14:41:59.882410+0000","last_peered":"2024-06-29T14:41:59.882410+0000","last_clean":"2024-06-29T14:41:59.882410+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T14:41:59.882410+0000","last_undegraded":"2024-06-29T14:41:59.882410+0000","last_fullsized":"2024-06-29T14:41:59.882410+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":245,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":786,"num_read_kb":2167,"num_write":245,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575083,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33568,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338272,"statfs":{"total":96636764160,"available":96602390528,"internally_reserved":0,"allocated":2437120,"data_stored":1468589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575084,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37116,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334724,"statfs":{"total":96636764160,"available":96598757376,"internally_reserved":0,"allocated":1818624,"data_stored":871183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575082,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37116,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334724,"statfs":{"total":96636764160,"available":96598757376,"internally_reserved":0,"allocated":1814528,"data_stored":869683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575083,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59336,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312504,"statfs":{"total":96636764160,"available":96576004096,"internally_reserved":0,"allocated":1822720,"data_stored":876041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575083,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37112,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334728,"statfs":{"total":96636764160,"available":96598761472,"internally_reserved":0,"allocated":1814528,"data_stored":869731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575083,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59328,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312512,"statfs":{"total":96636764160,"available":96576012288,"internally_reserved":0,"allocated":1814528,"data_stored":870187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55156,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316684,"statfs":{"total":96636764160,"available":96580284416,"internally_reserved":0,"allocated":1806336,"data_stored":869571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575083,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33524,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338316,"statfs":{"total":96636764160,"available":96602435584,"internally_reserved":0,"allocated":2400256,"data_stored":1459939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T14:42:02.935 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T14:42:02.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T14:42:03.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.094+0000 7f0d4a026640 1 -- 172.21.15.81:0/3404128677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d4406f6a0 msgr2=0x7f0d4415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.094+0000 7f0d4a026640 1 --2- 172.21.15.81:0/3404128677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d4406f6a0 0x7f0d4415aed0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c005fd0 tx=0x7f0d2c02a530 comp rx=0 tx=0).stop 2024-06-29T14:42:03.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.094+0000 7f0d4a026640 1 -- 172.21.15.81:0/3404128677 shutdown_connections 2024-06-29T14:42:03.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.094+0000 7f0d4a026640 1 --2- 172.21.15.81:0/3404128677 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d440b3960 0x7f0d440b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.094+0000 7f0d4a026640 1 --2- 172.21.15.81:0/3404128677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d4406f6a0 0x7f0d4415aed0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.094+0000 7f0d4a026640 1 -- 172.21.15.81:0/3404128677 >> 172.21.15.81:0/3404128677 conn(0x7f0d440d0930 msgr2=0x7f0d440d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:03.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.094+0000 7f0d4a026640 1 -- 172.21.15.81:0/3404128677 shutdown_connections 2024-06-29T14:42:03.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.094+0000 7f0d4a026640 1 -- 172.21.15.81:0/3404128677 wait complete. 2024-06-29T14:42:03.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.096+0000 7f0d4a026640 1 Processor -- start 2024-06-29T14:42:03.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.096+0000 7f0d4a026640 1 -- start start 2024-06-29T14:42:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.097+0000 7f0d4a026640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d4406f6a0 0x7f0d440ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.097+0000 7f0d4a026640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d440b3960 0x7f0d4414c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.097+0000 7f0d4a026640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d440b57b0 0x7f0d441f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.097+0000 7f0d4a026640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0d440b4430 con 0x7f0d4406f6a0 2024-06-29T14:42:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.097+0000 7f0d4a026640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0d440b42b0 con 0x7f0d440b3960 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.097+0000 7f0d4a026640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0d440b45b0 con 0x7f0d440b57b0 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.097+0000 7f0d437fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d4406f6a0 0x7f0d440ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d437fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d4406f6a0 0x7f0d440ae500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55728/0 (socket says 172.21.15.81:55728) 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d437fe640 1 -- 172.21.15.81:0/950327671 learned_addr learned my addr 172.21.15.81:0/950327671 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d43fff640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d440b57b0 0x7f0d441f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d3affd640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d440b3960 0x7f0d4414c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d437fe640 1 -- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d440b57b0 msgr2=0x7f0d441f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d437fe640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d440b57b0 0x7f0d441f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d437fe640 1 -- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d440b3960 msgr2=0x7f0d4414c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d437fe640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d440b3960 0x7f0d4414c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d437fe640 1 -- 172.21.15.81:0/950327671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4414c9c0 con 0x7f0d4406f6a0 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d43fff640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d440b57b0 0x7f0d441f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:42:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d3affd640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d440b3960 0x7f0d4414c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T14:42:03.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d437fe640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d4406f6a0 0x7f0d440ae500 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400b420 tx=0x7f0d3400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:03.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d40ff9640 1 -- 172.21.15.81:0/950327671 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d34011830 con 0x7f0d4406f6a0 2024-06-29T14:42:03.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d40ff9640 1 -- 172.21.15.81:0/950327671 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d34016070 con 0x7f0d4406f6a0 2024-06-29T14:42:03.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d40ff9640 1 -- 172.21.15.81:0/950327671 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d34011d60 con 0x7f0d4406f6a0 2024-06-29T14:42:03.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d4a026640 1 -- 172.21.15.81:0/950327671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d440aee00 con 0x7f0d4406f6a0 2024-06-29T14:42:03.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.098+0000 7f0d4a026640 1 -- 172.21.15.81:0/950327671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d440af3c0 con 0x7f0d4406f6a0 2024-06-29T14:42:03.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.099+0000 7f0d4a026640 1 -- 172.21.15.81:0/950327671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d4415a640 con 0x7f0d4406f6a0 2024-06-29T14:42:03.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.100+0000 7f0d40ff9640 1 -- 172.21.15.81:0/950327671 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f0d3401b460 con 0x7f0d4406f6a0 2024-06-29T14:42:03.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.101+0000 7f0d40ff9640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0d240763f0 0x7f0d24078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.101+0000 7f0d40ff9640 1 -- 172.21.15.81:0/950327671 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f0d34097580 con 0x7f0d4406f6a0 2024-06-29T14:42:03.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.101+0000 7f0d3affd640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0d240763f0 0x7f0d24078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.101+0000 7f0d3affd640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0d240763f0 0x7f0d24078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c033000 tx=0x7f0d2c004730 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:03.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.105+0000 7f0d40ff9640 1 -- 172.21.15.81:0/950327671 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0d4415a640 con 0x7f0d4406f6a0 2024-06-29T14:42:03.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.353+0000 7f0d4a026640 1 -- 172.21.15.81:0/950327671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f0d4406e2a0 con 0x7f0d4406f6a0 2024-06-29T14:42:03.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.355+0000 7f0d40ff9640 1 -- 172.21.15.81:0/950327671 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v37) ==== 190+0+147 (secure 0 0 0) 0x7f0d4406e2a0 con 0x7f0d4406f6a0 2024-06-29T14:42:03.353 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T14:42:03.353 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T14:42:03.353 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T14:42:03.353 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T14:42:03.353 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T14:42:03.353 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T14:42:03.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.357+0000 7f0d4a026640 1 -- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0d240763f0 msgr2=0x7f0d24078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.357+0000 7f0d4a026640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0d240763f0 0x7f0d24078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c033000 tx=0x7f0d2c004730 comp rx=0 tx=0).stop 2024-06-29T14:42:03.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.358+0000 7f0d4a026640 1 -- 172.21.15.81:0/950327671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d4406f6a0 msgr2=0x7f0d440ae500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.358+0000 7f0d4a026640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d4406f6a0 0x7f0d440ae500 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400b420 tx=0x7f0d3400b8e0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.358+0000 7f0d4a026640 1 -- 172.21.15.81:0/950327671 shutdown_connections 2024-06-29T14:42:03.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.358+0000 7f0d4a026640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f0d240763f0 0x7f0d24078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.358+0000 7f0d4a026640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0d440b57b0 0x7f0d441f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.358+0000 7f0d4a026640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0d440b3960 0x7f0d4414c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.358+0000 7f0d4a026640 1 --2- 172.21.15.81:0/950327671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d4406f6a0 0x7f0d440ae500 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.358+0000 7f0d4a026640 1 -- 172.21.15.81:0/950327671 >> 172.21.15.81:0/950327671 conn(0x7f0d440d0930 msgr2=0x7f0d441583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:03.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.358+0000 7f0d4a026640 1 -- 172.21.15.81:0/950327671 shutdown_connections 2024-06-29T14:42:03.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.358+0000 7f0d4a026640 1 -- 172.21.15.81:0/950327671 wait complete. 2024-06-29T14:42:03.366 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T14:42:03.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.483+0000 7f844023f640 1 -- 172.21.15.81:0/3105956492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84380b6eb0 msgr2=0x7f84380d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.483+0000 7f844023f640 1 --2- 172.21.15.81:0/3105956492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84380b6eb0 0x7f84380d02d0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f84380b41b0 tx=0x7f842802a9f0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.483+0000 7f844023f640 1 -- 172.21.15.81:0/3105956492 shutdown_connections 2024-06-29T14:42:03.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.483+0000 7f844023f640 1 --2- 172.21.15.81:0/3105956492 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84380ce510 0x7f84380ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.483+0000 7f844023f640 1 --2- 172.21.15.81:0/3105956492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84380b6eb0 0x7f84380d02d0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.483+0000 7f844023f640 1 -- 172.21.15.81:0/3105956492 >> 172.21.15.81:0/3105956492 conn(0x7f84380b3510 msgr2=0x7f84380b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:03.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.483+0000 7f844023f640 1 -- 172.21.15.81:0/3105956492 shutdown_connections 2024-06-29T14:42:03.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.483+0000 7f844023f640 1 -- 172.21.15.81:0/3105956492 wait complete. 2024-06-29T14:42:03.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.485+0000 7f844023f640 1 Processor -- start 2024-06-29T14:42:03.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.485+0000 7f844023f640 1 -- start start 2024-06-29T14:42:03.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f844023f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84380b4810 0x7f843806efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f844023f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84380b6eb0 0x7f84380cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f844023f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84380ce510 0x7f8438165910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f844023f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f84380cd480 con 0x7f84380b6eb0 2024-06-29T14:42:03.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f844023f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f84380b29c0 con 0x7f84380b4810 2024-06-29T14:42:03.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f844023f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f84380cd600 con 0x7f84380ce510 2024-06-29T14:42:03.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843d7da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84380b6eb0 0x7f84380cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843d7da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84380b6eb0 0x7f84380cabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55748/0 (socket says 172.21.15.81:55748) 2024-06-29T14:42:03.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843d7da640 1 -- 172.21.15.81:0/607610386 learned_addr learned my addr 172.21.15.81:0/607610386 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:03.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843dfdb640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84380b4810 0x7f843806efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843d7da640 1 -- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84380ce510 msgr2=0x7f8438165910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843e7dc640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84380ce510 0x7f8438165910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843d7da640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84380ce510 0x7f8438165910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843d7da640 1 -- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84380b4810 msgr2=0x7f843806efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843d7da640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84380b4810 0x7f843806efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843d7da640 1 -- 172.21.15.81:0/607610386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84381f8bb0 con 0x7f84380b6eb0 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.486+0000 7f843dfdb640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84380b4810 0x7f843806efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.487+0000 7f843d7da640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84380b6eb0 0x7f84380cabf0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f842c00e790 tx=0x7f842c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.487+0000 7f84367fc640 1 -- 172.21.15.81:0/607610386 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c00eea0 con 0x7f84380b6eb0 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.487+0000 7f84367fc640 1 -- 172.21.15.81:0/607610386 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f842c00cdb0 con 0x7f84380b6eb0 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.487+0000 7f844023f640 1 -- 172.21.15.81:0/607610386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84381f8e90 con 0x7f84380b6eb0 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.487+0000 7f84367fc640 1 -- 172.21.15.81:0/607610386 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c010690 con 0x7f84380b6eb0 2024-06-29T14:42:03.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.487+0000 7f844023f640 1 -- 172.21.15.81:0/607610386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f843815eac0 con 0x7f84380b6eb0 2024-06-29T14:42:03.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.488+0000 7f844023f640 1 -- 172.21.15.81:0/607610386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84380d02d0 con 0x7f84380b6eb0 2024-06-29T14:42:03.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.488+0000 7f84367fc640 1 -- 172.21.15.81:0/607610386 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f842c010830 con 0x7f84380b6eb0 2024-06-29T14:42:03.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.489+0000 7f84367fc640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f84240763f0 0x7f8424078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:03.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.489+0000 7f84367fc640 1 -- 172.21.15.81:0/607610386 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f842c098430 con 0x7f84380b6eb0 2024-06-29T14:42:03.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.489+0000 7f843dfdb640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f84240763f0 0x7f8424078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:03.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.490+0000 7f843dfdb640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f84240763f0 0x7f8424078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8428035000 tx=0x7f8428004ab0 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:03.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.493+0000 7f84367fc640 1 -- 172.21.15.81:0/607610386 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f842c061b00 con 0x7f84380b6eb0 2024-06-29T14:42:03.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.710+0000 7f844023f640 1 -- 172.21.15.81:0/607610386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84381f8940 con 0x7f84380b6eb0 2024-06-29T14:42:03.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.710+0000 7f84367fc640 1 -- 172.21.15.81:0/607610386 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5231 (secure 0 0 0) 0x7f842c061400 con 0x7f84380b6eb0 2024-06-29T14:42:03.708 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:42:03.708 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":77,"btime":"2024-06-29T14:42:01:549259+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/4068340499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":4068340499},{"type":"v1","addr":"172.21.15.81:6834","nonce":4068340499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1903134398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1903134398},{"type":"v1","addr":"172.21.15.167:6837","nonce":1903134398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1126258483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1126258483},{"type":"v1","addr":"172.21.15.81:6835","nonce":1126258483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.875457+0000","modified":"2024-06-29T14:42:00.919213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9111":{"gid":9111,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/4254103476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":4254103476},{"type":"v1","addr":"172.21.15.167:6836","nonce":4254103476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9111,"qdb_cluster":[9111]},"id":9}]} 2024-06-29T14:42:03.708 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 77 2024-06-29T14:42:03.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.712+0000 7f844023f640 1 -- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f84240763f0 msgr2=0x7f8424078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.712+0000 7f844023f640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f84240763f0 0x7f8424078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8428035000 tx=0x7f8428004ab0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.712+0000 7f844023f640 1 -- 172.21.15.81:0/607610386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84380b6eb0 msgr2=0x7f84380cabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:03.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.712+0000 7f844023f640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84380b6eb0 0x7f84380cabf0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f842c00e790 tx=0x7f842c00ec50 comp rx=0 tx=0).stop 2024-06-29T14:42:03.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.712+0000 7f844023f640 1 -- 172.21.15.81:0/607610386 shutdown_connections 2024-06-29T14:42:03.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.712+0000 7f844023f640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f84240763f0 0x7f8424078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.712+0000 7f844023f640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84380ce510 0x7f8438165910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.712+0000 7f844023f640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84380b6eb0 0x7f84380cabf0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.712+0000 7f844023f640 1 --2- 172.21.15.81:0/607610386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84380b4810 0x7f843806efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:03.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.712+0000 7f844023f640 1 -- 172.21.15.81:0/607610386 >> 172.21.15.81:0/607610386 conn(0x7f84380b3510 msgr2=0x7f84380cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:03.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.713+0000 7f844023f640 1 -- 172.21.15.81:0/607610386 shutdown_connections 2024-06-29T14:42:03.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:03.713+0000 7f844023f640 1 -- 172.21.15.81:0/607610386 wait complete. 2024-06-29T14:42:04.124 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T14:41:57.875457+0000', 'modified': '2024-06-29T14:42:00.919213+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9111}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9111': {'gid': 9111, 'name': 'b', 'rank': 0, 'incarnation': 73, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.167:6836/4254103476', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6834', 'nonce': 4254103476}, {'type': 'v1', 'addr': '172.21.15.167:6836', 'nonce': 4254103476}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9111, 'qdb_cluster': [9111]} 2024-06-29T14:42:04.125 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T14:42:04.125 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-06-29T14:42:04.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.246+0000 7f2f6badf640 1 -- 172.21.15.81:0/3109584229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f640b9040 msgr2=0x7f2f640d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.246+0000 7f2f6badf640 1 --2- 172.21.15.81:0/3109584229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f640b9040 0x7f2f640d0480 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f2f640a1b70 tx=0x7f2f6002a5f0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.246+0000 7f2f6badf640 1 -- 172.21.15.81:0/3109584229 shutdown_connections 2024-06-29T14:42:04.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.246+0000 7f2f6badf640 1 --2- 172.21.15.81:0/3109584229 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f640b7290 0x7f2f640b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.246+0000 7f2f6badf640 1 --2- 172.21.15.81:0/3109584229 >> v2:172.21.15.167:3300/0 conn(0x7f2f640b5250 0x7f2f640d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.246+0000 7f2f6badf640 1 --2- 172.21.15.81:0/3109584229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f640b9040 0x7f2f640d0480 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.246+0000 7f2f6badf640 1 -- 172.21.15.81:0/3109584229 >> 172.21.15.81:0/3109584229 conn(0x7f2f6409cf30 msgr2=0x7f2f640d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.247+0000 7f2f6badf640 1 -- 172.21.15.81:0/3109584229 shutdown_connections 2024-06-29T14:42:04.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.247+0000 7f2f6badf640 1 -- 172.21.15.81:0/3109584229 wait complete. 2024-06-29T14:42:04.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.248+0000 7f2f6badf640 1 Processor -- start 2024-06-29T14:42:04.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.248+0000 7f2f6badf640 1 -- start start 2024-06-29T14:42:04.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6badf640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f640b5250 0x7f2f640c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6badf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f640b7290 0x7f2f6415ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6badf640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f640b9040 0x7f2f64160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6badf640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2f640b7f70 con 0x7f2f640b7290 2024-06-29T14:42:04.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6badf640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2f640b7df0 con 0x7f2f640b9040 2024-06-29T14:42:04.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6badf640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2f640b80f0 con 0x7f2f640b5250 2024-06-29T14:42:04.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6907a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f640b7290 0x7f2f6415ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6907a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f640b7290 0x7f2f6415ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55772/0 (socket says 172.21.15.81:55772) 2024-06-29T14:42:04.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6907a640 1 -- 172.21.15.81:0/696516082 learned_addr learned my addr 172.21.15.81:0/696516082 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:04.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6a07c640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f640b9040 0x7f2f64160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6987b640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f640b5250 0x7f2f640c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6907a640 1 -- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f640b5250 msgr2=0x7f2f640c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6907a640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f640b5250 0x7f2f640c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6907a640 1 -- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f640b9040 msgr2=0x7f2f64160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6907a640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f640b9040 0x7f2f64160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6907a640 1 -- 172.21.15.81:0/696516082 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f640c7fa0 con 0x7f2f640b7290 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6907a640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f640b7290 0x7f2f6415ff30 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400d740 tx=0x7f2f5400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f5a7fc640 1 -- 172.21.15.81:0/696516082 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f54017070 con 0x7f2f640b7290 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f5a7fc640 1 -- 172.21.15.81:0/696516082 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f54009cc0 con 0x7f2f640b7290 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6badf640 1 -- 172.21.15.81:0/696516082 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f640c8280 con 0x7f2f640b7290 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f5a7fc640 1 -- 172.21.15.81:0/696516082 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f540106c0 con 0x7f2f640b7290 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.250+0000 7f2f6badf640 1 -- 172.21.15.81:0/696516082 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f640c8840 con 0x7f2f640b7290 2024-06-29T14:42:04.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.252+0000 7f2f5a7fc640 1 -- 172.21.15.81:0/696516082 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f2f5401a070 con 0x7f2f640b7290 2024-06-29T14:42:04.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.253+0000 7f2f5a7fc640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f2f480763f0 0x7f2f48078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.253+0000 7f2f5a7fc640 1 -- 172.21.15.81:0/696516082 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f2f540979b0 con 0x7f2f640b7290 2024-06-29T14:42:04.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.253+0000 7f2f6987b640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f2f480763f0 0x7f2f48078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.254+0000 7f2f72999740 1 Processor -- start 2024-06-29T14:42:04.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.254+0000 7f2f6987b640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f2f480763f0 0x7f2f48078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2f60033000 tx=0x7f2f60004730 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.254+0000 7f2f72999740 1 -- start start 2024-06-29T14:42:04.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f72999740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520587270 0x5635209736e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f72999740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f72999740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x5635209578e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f72999740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5635208efbc0 con 0x56352055e3b0 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f72999740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x56352089e630 con 0x563520587270 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f72999740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x56352088dbd0 con 0x563520566590 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f6907a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f6907a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55788/0 (socket says 172.21.15.81:55788) 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f6907a640 1 -- 172.21.15.81:0/3234242713 learned_addr learned my addr 172.21.15.81:0/3234242713 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f6987b640 1 --2- 172.21.15.81:0/3234242713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520587270 0x5635209736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.255+0000 7f2f6a07c640 1 --2- 172.21.15.81:0/3234242713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x5635209578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.256+0000 7f2f6907a640 1 -- 172.21.15.81:0/3234242713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 msgr2=0x5635209578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.256+0000 7f2f6907a640 1 --2- 172.21.15.81:0/3234242713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x5635209578e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.256+0000 7f2f6907a640 1 -- 172.21.15.81:0/3234242713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520587270 msgr2=0x5635209736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.256+0000 7f2f6907a640 1 --2- 172.21.15.81:0/3234242713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520587270 0x5635209736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.256+0000 7f2f6907a640 1 -- 172.21.15.81:0/3234242713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56352088aca0 con 0x56352055e3b0 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.256+0000 7f2f6907a640 1 --2- 172.21.15.81:0/3234242713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209573a0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f2f54095cd0 tx=0x7f2f540605c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303377c9c52db322 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.256+0000 7f2f477fe640 1 -- 172.21.15.81:0/3234242713 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f54021040 con 0x56352055e3b0 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.256+0000 7f2f477fe640 1 -- 172.21.15.81:0/3234242713 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f54060b40 con 0x56352055e3b0 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.256+0000 7f2f477fe640 1 -- 172.21.15.81:0/3234242713 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f540672b0 con 0x56352055e3b0 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.257+0000 7f2f72999740 1 -- 172.21.15.81:0/3234242713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 msgr2=0x5635209573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.257+0000 7f2f72999740 1 --2- 172.21.15.81:0/3234242713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209573a0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f2f54095cd0 tx=0x7f2f540605c0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.257+0000 7f2f72999740 1 -- 172.21.15.81:0/3234242713 shutdown_connections 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.257+0000 7f2f72999740 1 --2- 172.21.15.81:0/3234242713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x5635209578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.257+0000 7f2f72999740 1 --2- 172.21.15.81:0/3234242713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209573a0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.257+0000 7f2f72999740 1 --2- 172.21.15.81:0/3234242713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520587270 0x5635209736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.257+0000 7f2f72999740 1 -- 172.21.15.81:0/3234242713 >> 172.21.15.81:0/3234242713 conn(0x56352057b3c0 msgr2=0x56352092af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.257+0000 7f2f72999740 1 -- 172.21.15.81:0/3234242713 shutdown_connections 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.257+0000 7f2f72999740 1 -- 172.21.15.81:0/3234242713 wait complete. 2024-06-29T14:42:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.258+0000 7f2f72999740 1 -- start start 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.258+0000 7f2f72999740 1 Processor -- start 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.259+0000 7f2f72999740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56352055e3b0 0x56352094e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.259+0000 7f2f72999740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520566590 0x563520985290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.259+0000 7f2f72999740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x5635209857d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.259+0000 7f2f72999740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56352082bae0 con 0x563520587270 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.259+0000 7f2f72999740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x563520891c10 con 0x563520566590 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.259+0000 7f2f72999740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x563520601620 con 0x56352055e3b0 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.259+0000 7f2f72999740 10 client.0 _ll_register_callbacks cb 0x563520838480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.259+0000 7f2f6a07c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x5635209857d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.259+0000 7f2f6a07c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x5635209857d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55794/0 (socket says 172.21.15.81:55794) 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.259+0000 7f2f6a07c640 1 -- 172.21.15.81:0/361766820 learned_addr learned my addr 172.21.15.81:0/361766820 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f6a07c640 1 -- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56352055e3b0 msgr2=0x56352094e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f6907a640 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520566590 0x563520985290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f6a07c640 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56352055e3b0 0x56352094e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f6a07c640 1 -- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520566590 msgr2=0x563520985290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f6a07c640 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520566590 0x563520985290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f6a07c640 1 -- 172.21.15.81:0/361766820 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635205f2850 con 0x563520587270 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f6a07c640 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x5635209857d0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0139f0 tx=0x7f2f5c013eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963d49644a480a34 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f44ff9640 1 -- 172.21.15.81:0/361766820 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c003970 con 0x563520587270 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f44ff9640 1 -- 172.21.15.81:0/361766820 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f2f5c003b10 con 0x563520587270 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f72999740 1 -- 172.21.15.81:0/361766820 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f54060b40 con 0x563520587270 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f44ff9640 1 -- 172.21.15.81:0/361766820 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f2f5c024050 con 0x563520587270 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f72999740 10 client.9240 fetch_fsmap learned FSMap version 77 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f72999740 1 -- 172.21.15.81:0/361766820 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) -- 0x5635206696b0 con 0x563520587270 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f44ff9640 1 -- 172.21.15.81:0/361766820 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f54060b40 con 0x563520587270 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f44ff9640 1 -- 172.21.15.81:0/361766820 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 77) ==== 3502+0+0 (secure 0 0 0) 0x7f2f5c027020 con 0x563520587270 2024-06-29T14:42:04.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f72999740 10 client.9240 fetch_fsmap finished waiting for FSMap version 77 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f72999740 10 client.9240 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f72999740 20 client.9240 populate_metadata read hostname 'smithi081' 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f72999740 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:6834/4254103476,v1:172.21.15.167:6836/4254103476] conn(0x563520545e40 0x563520954b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f72999740 4 client.9240 mds_command: new command op to 9111 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.260+0000 7f2f72999740 1 -- 172.21.15.81:0/361766820 --> [v2:172.21.15.167:6834/4254103476,v1:172.21.15.167:6836/4254103476] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635208a0580 con 0x563520545e40 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.261+0000 7f2f6987b640 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:6834/4254103476,v1:172.21.15.167:6836/4254103476] conn(0x563520545e40 0x563520954b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.261+0000 7f2f44ff9640 10 client.9240 ms_handle_connect on v2:172.21.15.167:6834/4254103476 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.261+0000 7f2f6987b640 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:6834/4254103476,v1:172.21.15.167:6836/4254103476] conn(0x563520545e40 0x563520954b10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=135360d8c0a95f34 server_cookie=806257de6f664f67 in_seq=0 out_seq=0 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f44ff9640 1 -- 172.21.15.81:0/361766820 <== mds.0 v2:172.21.15.167:6834/4254103476 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5635208a0580 con 0x563520545e40 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f44ff9640 10 client.9240 handle_command_reply: tid=0 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 client.9240 shutdown 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 -- 172.21.15.81:0/361766820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 msgr2=0x5635209857d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x5635209857d0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0139f0 tx=0x7f2f5c013eb0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 -- 172.21.15.81:0/361766820 shutdown_connections 2024-06-29T14:42:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x5635209857d0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520566590 0x563520985290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:6834/4254103476,v1:172.21.15.167:6836/4254103476] conn(0x563520545e40 0x563520954b10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 --2- 172.21.15.81:0/361766820 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56352055e3b0 0x56352094e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f44ff9640 0 client.9240 ms_handle_reset on v2:172.21.15.167:6834/4254103476 2024-06-29T14:42:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 -- 172.21.15.81:0/361766820 >> 172.21.15.81:0/361766820 conn(0x56352057b3c0 msgr2=0x563520926b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 -- 172.21.15.81:0/361766820 shutdown_connections 2024-06-29T14:42:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 1 -- 172.21.15.81:0/361766820 wait complete. 2024-06-29T14:42:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.262+0000 7f2f72999740 20 client.9240 trim_cache size 0 max 16384 2024-06-29T14:42:04.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.281+0000 7f2f72999740 1 Processor -- start 2024-06-29T14:42:04.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.281+0000 7f2f72999740 1 -- start start 2024-06-29T14:42:04.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.282+0000 7f2f72999740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56352055e3b0 0x563520954c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.282+0000 7f2f72999740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x563520928b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.282+0000 7f2f72999740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x563520929090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.282+0000 7f2f72999740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2f642009c0 con 0x563520587270 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.282+0000 7f2f72999740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2f640b1cd0 con 0x56352055e3b0 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.282+0000 7f2f72999740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2f640b2320 con 0x563520566590 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6a07c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x563520929090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6a07c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x563520929090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55800/0 (socket says 172.21.15.81:55800) 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6a07c640 1 -- 172.21.15.81:0/159691064 learned_addr learned my addr 172.21.15.81:0/159691064 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6907a640 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x563520928b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6987b640 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56352055e3b0 0x563520954c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6a07c640 1 -- 172.21.15.81:0/159691064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 msgr2=0x563520928b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6a07c640 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x563520928b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6a07c640 1 -- 172.21.15.81:0/159691064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56352055e3b0 msgr2=0x563520954c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6a07c640 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56352055e3b0 0x563520954c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6a07c640 1 -- 172.21.15.81:0/159691064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563520974490 con 0x563520587270 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6907a640 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x563520928b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6987b640 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56352055e3b0 0x563520954c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f6a07c640 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x563520929090 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c011850 tx=0x7f2f5c013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b95cf72a28969e server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f46ffd640 1 -- 172.21.15.81:0/159691064 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c02c030 con 0x563520587270 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f46ffd640 1 -- 172.21.15.81:0/159691064 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f5c013260 con 0x563520587270 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.283+0000 7f2f46ffd640 1 -- 172.21.15.81:0/159691064 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c021040 con 0x563520587270 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.284+0000 7f2f72999740 1 -- 172.21.15.81:0/159691064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 msgr2=0x563520929090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.284+0000 7f2f72999740 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x563520929090 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c011850 tx=0x7f2f5c013040 comp rx=0 tx=0).stop 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.284+0000 7f2f72999740 1 -- 172.21.15.81:0/159691064 shutdown_connections 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.284+0000 7f2f72999740 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563520587270 0x563520929090 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.284+0000 7f2f72999740 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x563520928b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.284+0000 7f2f72999740 1 --2- 172.21.15.81:0/159691064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56352055e3b0 0x563520954c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.284+0000 7f2f72999740 1 -- 172.21.15.81:0/159691064 >> 172.21.15.81:0/159691064 conn(0x56352057b3c0 msgr2=0x563520995bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.284+0000 7f2f72999740 1 -- 172.21.15.81:0/159691064 shutdown_connections 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.284+0000 7f2f72999740 1 -- 172.21.15.81:0/159691064 wait complete. 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.285+0000 7f2f72999740 1 -- start start 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.285+0000 7f2f72999740 1 Processor -- start 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.286+0000 7f2f72999740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209efd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.286+0000 7f2f72999740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x563520983da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.286+0000 7f2f72999740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520587270 0x5635209842e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f72999740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563520985db0 con 0x56352055e3b0 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f72999740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x563520971b50 con 0x563520587270 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f72999740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x563520970cb0 con 0x563520566590 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f72999740 10 client.0 _ll_register_callbacks cb 0x7f2f5c009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6987b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209efd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6987b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209efd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55808/0 (socket says 172.21.15.81:55808) 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6987b640 1 -- 172.21.15.81:0/2383239449 learned_addr learned my addr 172.21.15.81:0/2383239449 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6a07c640 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520587270 0x5635209842e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6907a640 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x563520983da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6987b640 1 -- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 msgr2=0x563520983da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6987b640 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x563520983da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6987b640 1 -- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520587270 msgr2=0x5635209842e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6987b640 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520587270 0x5635209842e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6987b640 1 -- 172.21.15.81:0/2383239449 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563520954440 con 0x56352055e3b0 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6907a640 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x563520983da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f6987b640 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209efd10 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f2f60034040 tx=0x7f2f60009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf1270ae4b09b94 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f44ff9640 1 -- 172.21.15.81:0/2383239449 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f60035040 con 0x56352055e3b0 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f44ff9640 1 -- 172.21.15.81:0/2383239449 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f2f60003770 con 0x56352055e3b0 2024-06-29T14:42:04.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f72999740 1 -- 172.21.15.81:0/2383239449 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f5c013260 con 0x56352055e3b0 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f44ff9640 1 -- 172.21.15.81:0/2383239449 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f2f60061050 con 0x56352055e3b0 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f72999740 10 client.9246 fetch_fsmap learned FSMap version 77 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.287+0000 7f2f72999740 1 -- 172.21.15.81:0/2383239449 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) -- 0x563520955530 con 0x56352055e3b0 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f44ff9640 1 -- 172.21.15.81:0/2383239449 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f5c013260 con 0x56352055e3b0 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f44ff9640 1 -- 172.21.15.81:0/2383239449 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 77) ==== 3502+0+0 (secure 0 0 0) 0x7f2f60003a60 con 0x56352055e3b0 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f72999740 10 client.9246 fetch_fsmap finished waiting for FSMap version 77 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f72999740 10 client.9246 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f72999740 20 client.9246 populate_metadata read hostname 'smithi081' 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f72999740 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:6834/4254103476,v1:172.21.15.167:6836/4254103476] conn(0x56352096dff0 0x56352096e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f72999740 4 client.9246 mds_command: new command op to 9111 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f72999740 1 -- 172.21.15.81:0/2383239449 --> [v2:172.21.15.167:6834/4254103476,v1:172.21.15.167:6836/4254103476] -- command(tid 0: {"prefix": "status"}) -- 0x56352089c020 con 0x56352096dff0 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f6907a640 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:6834/4254103476,v1:172.21.15.167:6836/4254103476] conn(0x56352096dff0 0x56352096e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f44ff9640 10 client.9246 ms_handle_connect on v2:172.21.15.167:6834/4254103476 2024-06-29T14:42:04.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.288+0000 7f2f6907a640 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:6834/4254103476,v1:172.21.15.167:6836/4254103476] conn(0x56352096dff0 0x56352096e400 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96a5b392d6cc8790 server_cookie=299c7c059ab35812 in_seq=0 out_seq=0 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f44ff9640 1 -- 172.21.15.81:0/2383239449 <== mds.0 v2:172.21.15.167:6834/4254103476 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x56352089c020 con 0x56352096dff0 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f44ff9640 10 client.9246 handle_command_reply: tid=0 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 client.9246 shutdown 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 -- 172.21.15.81:0/2383239449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 msgr2=0x5635209efd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209efd10 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f2f60034040 tx=0x7f2f60009260 comp rx=0 tx=0).stop 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 -- 172.21.15.81:0/2383239449 shutdown_connections 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x563520587270 0x5635209842e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:6834/4254103476,v1:172.21.15.167:6836/4254103476] conn(0x56352096dff0 0x56352096e400 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x563520566590 0x563520983da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 --2- 172.21.15.81:0/2383239449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56352055e3b0 0x5635209efd10 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f44ff9640 0 client.9246 ms_handle_reset on v2:172.21.15.167:6834/4254103476 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 -- 172.21.15.81:0/2383239449 >> 172.21.15.81:0/2383239449 conn(0x56352057b3c0 msgr2=0x5635209684a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 -- 172.21.15.81:0/2383239449 shutdown_connections 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 1 -- 172.21.15.81:0/2383239449 wait complete. 2024-06-29T14:42:04.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f72999740 20 client.9246 trim_cache size 0 max 16384 2024-06-29T14:42:04.301 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T14:42:04.301 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T14:42:04.301 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T14:42:04.301 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 9111, 2024-06-29T14:42:04.301 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T14:42:04.301 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T14:42:04.301 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T14:42:04.301 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7575295659999997, 2024-06-29T14:42:04.301 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 77, 2024-06-29T14:42:04.301 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 146, 2024-06-29T14:42:04.302 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 146, 2024-06-29T14:42:04.302 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.437694774000001 2024-06-29T14:42:04.302 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T14:42:04.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f6badf640 1 -- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f2f480763f0 msgr2=0x7f2f48078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f6badf640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f2f480763f0 0x7f2f48078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2f60033000 tx=0x7f2f60004730 comp rx=0 tx=0).stop 2024-06-29T14:42:04.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f6badf640 1 -- 172.21.15.81:0/696516082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f640b7290 msgr2=0x7f2f6415ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f6badf640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f640b7290 0x7f2f6415ff30 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400d740 tx=0x7f2f5400dc00 comp rx=0 tx=0).stop 2024-06-29T14:42:04.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f6badf640 1 -- 172.21.15.81:0/696516082 shutdown_connections 2024-06-29T14:42:04.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f6badf640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f2f480763f0 0x7f2f48078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f6badf640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f640b9040 0x7f2f64160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f6badf640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f640b7290 0x7f2f6415ff30 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.289+0000 7f2f6badf640 1 --2- 172.21.15.81:0/696516082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f640b5250 0x7f2f640c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.290+0000 7f2f6badf640 1 -- 172.21.15.81:0/696516082 >> 172.21.15.81:0/696516082 conn(0x7f2f6409cf30 msgr2=0x7f2f6406d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.290+0000 7f2f6badf640 1 -- 172.21.15.81:0/696516082 shutdown_connections 2024-06-29T14:42:04.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.290+0000 7f2f6badf640 1 -- 172.21.15.81:0/696516082 wait complete. 2024-06-29T14:42:04.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T14:42:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.420+0000 7f9a69511640 1 -- 172.21.15.81:0/1260014917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a640cd540 msgr2=0x7f9a640b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.420+0000 7f9a69511640 1 --2- 172.21.15.81:0/1260014917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a640cd540 0x7f9a640b47b0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f9a640d2600 tx=0x7f9a4c02a5d0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.420+0000 7f9a69511640 1 -- 172.21.15.81:0/1260014917 shutdown_connections 2024-06-29T14:42:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.420+0000 7f9a69511640 1 --2- 172.21.15.81:0/1260014917 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a640b4cf0 0x7f9a640d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.420+0000 7f9a69511640 1 --2- 172.21.15.81:0/1260014917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a640cd540 0x7f9a640b47b0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.420+0000 7f9a69511640 1 -- 172.21.15.81:0/1260014917 >> 172.21.15.81:0/1260014917 conn(0x7f9a640d1960 msgr2=0x7f9a640d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.420+0000 7f9a69511640 1 -- 172.21.15.81:0/1260014917 shutdown_connections 2024-06-29T14:42:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.420+0000 7f9a69511640 1 -- 172.21.15.81:0/1260014917 wait complete. 2024-06-29T14:42:04.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.422+0000 7f9a69511640 1 Processor -- start 2024-06-29T14:42:04.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.422+0000 7f9a69511640 1 -- start start 2024-06-29T14:42:04.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a69511640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a640b4cf0 0x7f9a64158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a69511640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a640b6d60 0x7f9a641658f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a69511640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a640cd540 0x7f9a64165e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a69511640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a640d07b0 con 0x7f9a640cd540 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a69511640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9a640d0630 con 0x7f9a640b4cf0 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a69511640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9a640d0930 con 0x7f9a640b6d60 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a637fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a640cd540 0x7f9a64165e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a637fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a640cd540 0x7f9a64165e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55830/0 (socket says 172.21.15.81:55830) 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a637fe640 1 -- 172.21.15.81:0/1614603932 learned_addr learned my addr 172.21.15.81:0/1614603932 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a62ffd640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a640b4cf0 0x7f9a64158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.423+0000 7f9a5a7fc640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a640b6d60 0x7f9a641658f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a637fe640 1 -- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a640b6d60 msgr2=0x7f9a641658f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a637fe640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a640b6d60 0x7f9a641658f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a637fe640 1 -- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a640b4cf0 msgr2=0x7f9a64158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a637fe640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a640b4cf0 0x7f9a64158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a637fe640 1 -- 172.21.15.81:0/1614603932 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a641594a0 con 0x7f9a640cd540 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a637fe640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a640cd540 0x7f9a64165e30 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00e9c0 tx=0x7f9a5c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a5bfff640 1 -- 172.21.15.81:0/1614603932 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c004d20 con 0x7f9a640cd540 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a5bfff640 1 -- 172.21.15.81:0/1614603932 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a5c00cac0 con 0x7f9a640cd540 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a69511640 1 -- 172.21.15.81:0/1614603932 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a640c7d30 con 0x7f9a640cd540 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a5bfff640 1 -- 172.21.15.81:0/1614603932 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c005020 con 0x7f9a640cd540 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.424+0000 7f9a69511640 1 -- 172.21.15.81:0/1614603932 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6415a380 con 0x7f9a640cd540 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.425+0000 7f9a69511640 1 -- 172.21.15.81:0/1614603932 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a640d00f0 con 0x7f9a640cd540 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.426+0000 7f9a5bfff640 1 -- 172.21.15.81:0/1614603932 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f9a5c0051c0 con 0x7f9a640cd540 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.426+0000 7f9a5bfff640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9a44076420 0x7f9a44078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.426+0000 7f9a5bfff640 1 -- 172.21.15.81:0/1614603932 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f9a5c098240 con 0x7f9a640cd540 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.426+0000 7f9a62ffd640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9a44076420 0x7f9a44078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:04.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.427+0000 7f9a62ffd640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9a44076420 0x7f9a44078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a640c5220 tx=0x7f9a54009040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:04.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.431+0000 7f9a5bfff640 1 -- 172.21.15.81:0/1614603932 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9a5c016030 con 0x7f9a640cd540 2024-06-29T14:42:04.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.648+0000 7f9a69511640 1 -- 172.21.15.81:0/1614603932 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9a640cdec0 con 0x7f9a640cd540 2024-06-29T14:42:04.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.649+0000 7f9a5bfff640 1 -- 172.21.15.81:0/1614603932 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v77) ==== 53+0+73 (secure 0 0 0) 0x7f9a5c0611b0 con 0x7f9a640cd540 2024-06-29T14:42:04.646 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T14:42:04.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 -- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9a44076420 msgr2=0x7f9a44078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9a44076420 0x7f9a44078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a640c5220 tx=0x7f9a54009040 comp rx=0 tx=0).stop 2024-06-29T14:42:04.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 -- 172.21.15.81:0/1614603932 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a640cd540 msgr2=0x7f9a64165e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:04.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a640cd540 0x7f9a64165e30 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00e9c0 tx=0x7f9a5c00ee80 comp rx=0 tx=0).stop 2024-06-29T14:42:04.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 -- 172.21.15.81:0/1614603932 shutdown_connections 2024-06-29T14:42:04.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f9a44076420 0x7f9a44078930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a640cd540 0x7f9a64165e30 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a640b6d60 0x7f9a641658f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 --2- 172.21.15.81:0/1614603932 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a640b4cf0 0x7f9a64158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:04.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 -- 172.21.15.81:0/1614603932 >> 172.21.15.81:0/1614603932 conn(0x7f9a640d1960 msgr2=0x7f9a640717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:04.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 -- 172.21.15.81:0/1614603932 shutdown_connections 2024-06-29T14:42:04.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04.651+0000 7f9a69511640 1 -- 172.21.15.81:0/1614603932 wait complete. 2024-06-29T14:42:04.659 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T14:42:04.659 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T14:42:04.659 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T14:42:04.660 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T14:42:04.660 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T14:42:04.660 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T14:42:04.660 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T14:42:04.660 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T14:42:04.660 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:42:04.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:04.660 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T14:42:04.676 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T14:42:04.676 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T14:42:04.676 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T14:42:04.676 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.676 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T14:42:04.676 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.676 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:42:04.676 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 41595sec preferred_lft 41595sec 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.677 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T14:42:04.678 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T14:42:04.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:04.679 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T14:42:04.739 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T14:42:04.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:04.740 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T14:42:04.797 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T14:42:04.797 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T14:42:04.797 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T14:42:04.797 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T14:42:04.797 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T14:42:04.798 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T14:42:04.798 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T14:42:04.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T14:42:04.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T14:42:04.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T14:42:04.798 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T14:42:04.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T14:42:04.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:42:04.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:04.923 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T14:42:04.938 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T14:42:04.939 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T14:42:04.939 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:42:04.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:04.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T14:42:05.004 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:42:05.004 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T14:42:05.059 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T14:42:05.060 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T14:42:05.060 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:42:05.118 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T14:42:05.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:05.118 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:42:05.212 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:42:05.212 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:42:05.221 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T14:42:05.222 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T14:42:05.222 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T14:42:05.222 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T14:42:05.222 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T14:42:05.223 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T14:42:05.223 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T14:42:05.223 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T14:42:05.223 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T14:42:05.223 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T14:42:05.229 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T14:42:05.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:05.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T14:42:05.255 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T14:42:05.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T14:42:05.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T14:42:05.358 INFO:teuthology.orchestra.run.smithi081.stdout:client9255 2024-06-29T14:42:05.358 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T14:42:05.358 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T14:42:05.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T14:42:05.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.533+0000 7f1681911640 1 -- 172.21.15.81:0/685615256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f167c08ccd0 msgr2=0x7f167c0b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.533+0000 7f1681911640 1 --2- 172.21.15.81:0/685615256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f167c08ccd0 0x7f167c0b3240 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f1664009830 tx=0x7f166402a3f0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.534+0000 7f1681911640 1 -- 172.21.15.81:0/685615256 shutdown_connections 2024-06-29T14:42:05.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.534+0000 7f1681911640 1 --2- 172.21.15.81:0/685615256 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167c0cf040 0x7f167c0b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.534+0000 7f1681911640 1 --2- 172.21.15.81:0/685615256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f167c08ccd0 0x7f167c0b3240 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.534+0000 7f1681911640 1 -- 172.21.15.81:0/685615256 >> 172.21.15.81:0/685615256 conn(0x7f167c0b56d0 msgr2=0x7f167c0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:05.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.534+0000 7f1681911640 1 -- 172.21.15.81:0/685615256 shutdown_connections 2024-06-29T14:42:05.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.534+0000 7f1681911640 1 -- 172.21.15.81:0/685615256 wait complete. 2024-06-29T14:42:05.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.535+0000 7f1681911640 1 Processor -- start 2024-06-29T14:42:05.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.536+0000 7f1681911640 1 -- start start 2024-06-29T14:42:05.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f1681911640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f167c08ccd0 0x7f167c1dbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f1681911640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167c0cf040 0x7f167c0a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f1681911640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f167c06d840 0x7f167c06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f1681911640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f167c0b4210 con 0x7f167c06d840 2024-06-29T14:42:05.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f1681911640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f167c0b4090 con 0x7f167c08ccd0 2024-06-29T14:42:05.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f1681911640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f167c0b4390 con 0x7f167c0cf040 2024-06-29T14:42:05.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f167c06d840 0x7f167c06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f167c06d840 0x7f167c06dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55838/0 (socket says 172.21.15.81:55838) 2024-06-29T14:42:05.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167b7fe640 1 -- 172.21.15.81:0/2864721592 learned_addr learned my addr 172.21.15.81:0/2864721592 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:05.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167a7fc640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167c0cf040 0x7f167c0a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167affd640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f167c08ccd0 0x7f167c1dbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167b7fe640 1 -- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167c0cf040 msgr2=0x7f167c0a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167b7fe640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167c0cf040 0x7f167c0a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167b7fe640 1 -- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f167c08ccd0 msgr2=0x7f167c1dbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167b7fe640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f167c08ccd0 0x7f167c1dbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167b7fe640 1 -- 172.21.15.81:0/2864721592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c06e190 con 0x7f167c06d840 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.537+0000 7f167b7fe640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f167c06d840 0x7f167c06dc50 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f166c00c740 tx=0x7f166c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.538+0000 7f1673fff640 1 -- 172.21.15.81:0/2864721592 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c007bb0 con 0x7f167c06d840 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.538+0000 7f1673fff640 1 -- 172.21.15.81:0/2864721592 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f166c00ce50 con 0x7f167c06d840 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.538+0000 7f1681911640 1 -- 172.21.15.81:0/2864721592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167c0c79e0 con 0x7f167c06d840 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.538+0000 7f1673fff640 1 -- 172.21.15.81:0/2864721592 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c005280 con 0x7f167c06d840 2024-06-29T14:42:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.538+0000 7f1681911640 1 -- 172.21.15.81:0/2864721592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167c0c7fa0 con 0x7f167c06d840 2024-06-29T14:42:05.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.539+0000 7f1681911640 1 -- 172.21.15.81:0/2864721592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167c0c7750 con 0x7f167c06d840 2024-06-29T14:42:05.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.539+0000 7f1673fff640 1 -- 172.21.15.81:0/2864721592 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f166c005420 con 0x7f167c06d840 2024-06-29T14:42:05.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.540+0000 7f1673fff640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f165c0763f0 0x7f165c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.540+0000 7f1673fff640 1 -- 172.21.15.81:0/2864721592 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f166c097c00 con 0x7f167c06d840 2024-06-29T14:42:05.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.540+0000 7f167affd640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f165c0763f0 0x7f165c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.540+0000 7f167affd640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f165c0763f0 0x7f165c078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f16740012e0 tx=0x7f1674009040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.544+0000 7f1673fff640 1 -- 172.21.15.81:0/2864721592 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f166c09e080 con 0x7f167c06d840 2024-06-29T14:42:05.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.752+0000 7f1681911640 1 -- 172.21.15.81:0/2864721592 --> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f167c08c270 con 0x7f165c0763f0 2024-06-29T14:42:05.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.754+0000 7f1673fff640 1 -- 172.21.15.81:0/2864721592 <== mgr.7223 v2:172.21.15.167:6832/3201324170 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f167c08c270 con 0x7f165c0763f0 2024-06-29T14:42:05.865 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T14:42:05.865 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T14:42:05.865 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T14:42:05.865 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T14:42:05.866 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T14:42:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 -- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f165c0763f0 msgr2=0x7f165c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f165c0763f0 0x7f165c078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f16740012e0 tx=0x7f1674009040 comp rx=0 tx=0).stop 2024-06-29T14:42:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 -- 172.21.15.81:0/2864721592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f167c06d840 msgr2=0x7f167c06dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f167c06d840 0x7f167c06dc50 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f166c00c740 tx=0x7f166c00cc00 comp rx=0 tx=0).stop 2024-06-29T14:42:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 -- 172.21.15.81:0/2864721592 shutdown_connections 2024-06-29T14:42:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f165c0763f0 0x7f165c078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f167c06d840 0x7f167c06dc50 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f167c0cf040 0x7f167c0a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 --2- 172.21.15.81:0/2864721592 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f167c08ccd0 0x7f167c1dbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 -- 172.21.15.81:0/2864721592 >> 172.21.15.81:0/2864721592 conn(0x7f167c0b56d0 msgr2=0x7f167c0b8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 -- 172.21.15.81:0/2864721592 shutdown_connections 2024-06-29T14:42:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.756+0000 7f1681911640 1 -- 172.21.15.81:0/2864721592 wait complete. 2024-06-29T14:42:05.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T14:42:05.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.986+0000 7f5ba8426640 1 -- 172.21.15.81:0/3678837972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ba015d750 msgr2=0x7f5ba00afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.986+0000 7f5ba8426640 1 --2- 172.21.15.81:0/3678837972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ba015d750 0x7f5ba00afb60 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0074e30 tx=0x7f5b9c02a530 comp rx=0 tx=0).stop 2024-06-29T14:42:05.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.986+0000 7f5ba8426640 1 -- 172.21.15.81:0/3678837972 shutdown_connections 2024-06-29T14:42:05.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.986+0000 7f5ba8426640 1 --2- 172.21.15.81:0/3678837972 >> v2:172.21.15.167:3300/0 conn(0x7f5ba006ca00 0x7f5ba00cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.986+0000 7f5ba8426640 1 --2- 172.21.15.81:0/3678837972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ba015d750 0x7f5ba00afb60 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.986+0000 7f5ba8426640 1 -- 172.21.15.81:0/3678837972 >> 172.21.15.81:0/3678837972 conn(0x7f5ba00d0930 msgr2=0x7f5ba00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:05.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.986+0000 7f5ba8426640 1 -- 172.21.15.81:0/3678837972 shutdown_connections 2024-06-29T14:42:05.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.986+0000 7f5ba8426640 1 -- 172.21.15.81:0/3678837972 wait complete. 2024-06-29T14:42:05.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.988+0000 7f5ba8426640 1 Processor -- start 2024-06-29T14:42:05.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.988+0000 7f5ba8426640 1 -- start start 2024-06-29T14:42:05.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.989+0000 7f5ba8426640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ba015c030 0x7f5ba01f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.989+0000 7f5ba8426640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5ba015d750 0x7f5ba014d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.989+0000 7f5ba8426640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5ba014dd20 0x7f5ba015e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.989+0000 7f5ba8426640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba00cfce0 con 0x7f5ba015c030 2024-06-29T14:42:05.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.989+0000 7f5ba8426640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba00cfe60 con 0x7f5ba014dd20 2024-06-29T14:42:05.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.989+0000 7f5ba8426640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5ba00cffe0 con 0x7f5ba015d750 2024-06-29T14:42:05.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.989+0000 7f5ba61c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ba015c030 0x7f5ba01f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.989+0000 7f5ba61c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ba015c030 0x7f5ba01f3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55854/0 (socket says 172.21.15.81:55854) 2024-06-29T14:42:05.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.989+0000 7f5ba61c2640 1 -- 172.21.15.81:0/4083987272 learned_addr learned my addr 172.21.15.81:0/4083987272 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:05.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5ba69c3640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5ba014dd20 0x7f5ba015e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5ba59c1640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5ba015d750 0x7f5ba014d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5ba61c2640 1 -- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5ba015d750 msgr2=0x7f5ba014d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5ba61c2640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5ba015d750 0x7f5ba014d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5ba61c2640 1 -- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5ba014dd20 msgr2=0x7f5ba015e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5ba61c2640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5ba014dd20 0x7f5ba015e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5ba61c2640 1 -- 172.21.15.81:0/4083987272 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba00ac6c0 con 0x7f5ba015c030 2024-06-29T14:42:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5ba61c2640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ba015c030 0x7f5ba01f3170 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0098f0 tx=0x7f5b9c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5b96ffd640 1 -- 172.21.15.81:0/4083987272 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c031d60 con 0x7f5ba015c030 2024-06-29T14:42:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5b96ffd640 1 -- 172.21.15.81:0/4083987272 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b9c033e10 con 0x7f5ba015c030 2024-06-29T14:42:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5ba8426640 1 -- 172.21.15.81:0/4083987272 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ba00ac9a0 con 0x7f5ba015c030 2024-06-29T14:42:05.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5b96ffd640 1 -- 172.21.15.81:0/4083987272 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c005260 con 0x7f5ba015c030 2024-06-29T14:42:05.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.990+0000 7f5ba8426640 1 -- 172.21.15.81:0/4083987272 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ba00acf60 con 0x7f5ba015c030 2024-06-29T14:42:05.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.991+0000 7f5ba8426640 1 -- 172.21.15.81:0/4083987272 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ba006e2a0 con 0x7f5ba015c030 2024-06-29T14:42:05.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.992+0000 7f5b96ffd640 1 -- 172.21.15.81:0/4083987272 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 78) ==== 98303+0+0 (secure 0 0 0) 0x7f5b9c005400 con 0x7f5ba015c030 2024-06-29T14:42:05.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.992+0000 7f5b96ffd640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5b840763f0 0x7f5b84078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:05.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.992+0000 7f5b96ffd640 1 -- 172.21.15.81:0/4083987272 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7f5b9c0b8c10 con 0x7f5ba015c030 2024-06-29T14:42:05.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.993+0000 7f5ba59c1640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5b840763f0 0x7f5b84078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:05.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.993+0000 7f5ba59c1640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5b840763f0 0x7f5b84078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5b90000f60 tx=0x7f5b90009040 comp rx=0 tx=0).ready entity=mgr.7223 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:05.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:05.997+0000 7f5b96ffd640 1 -- 172.21.15.81:0/4083987272 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5ba006e2a0 con 0x7f5ba015c030 2024-06-29T14:42:06.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.222+0000 7f5ba8426640 1 -- 172.21.15.81:0/4083987272 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f5ba00d0490 con 0x7f5ba015c030 2024-06-29T14:42:06.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.574+0000 7f5b96ffd640 1 -- 172.21.15.81:0/4083987272 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v79) ==== 92+0+0 (secure 0 0 0) 0x7f5b9c03f030 con 0x7f5ba015c030 2024-06-29T14:42:06.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 -- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5b840763f0 msgr2=0x7f5b84078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5b840763f0 0x7f5b84078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5b90000f60 tx=0x7f5b90009040 comp rx=0 tx=0).stop 2024-06-29T14:42:06.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 -- 172.21.15.81:0/4083987272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ba015c030 msgr2=0x7f5ba01f3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ba015c030 0x7f5ba01f3170 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0098f0 tx=0x7f5b9c033c20 comp rx=0 tx=0).stop 2024-06-29T14:42:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 -- 172.21.15.81:0/4083987272 shutdown_connections 2024-06-29T14:42:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7f5b840763f0 0x7f5b84078900 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5b90000f60 tx=0x7f5b90009040 comp rx=0 tx=0).stop 2024-06-29T14:42:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5ba014dd20 0x7f5ba015e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5ba015d750 0x7f5ba014d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 --2- 172.21.15.81:0/4083987272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ba015c030 0x7f5ba01f3170 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 -- 172.21.15.81:0/4083987272 >> 172.21.15.81:0/4083987272 conn(0x7f5ba00d0930 msgr2=0x7f5ba00ccb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 -- 172.21.15.81:0/4083987272 shutdown_connections 2024-06-29T14:42:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.577+0000 7f5ba8426640 1 -- 172.21.15.81:0/4083987272 wait complete. 2024-06-29T14:42:06.587 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T14:42:06.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.711+0000 7fc2e41a9640 1 -- 172.21.15.81:0/498029457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2dc0b3200 msgr2=0x7fc2dc0cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.711+0000 7fc2e41a9640 1 --2- 172.21.15.81:0/498029457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2dc0b3200 0x7fc2dc0cc7b0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0c5590 tx=0x7fc2d402a4a0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.711+0000 7fc2e41a9640 1 -- 172.21.15.81:0/498029457 shutdown_connections 2024-06-29T14:42:06.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.711+0000 7fc2e41a9640 1 --2- 172.21.15.81:0/498029457 >> v2:172.21.15.167:3300/0 conn(0x7fc2dc0d0940 0x7fc2dc0b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.711+0000 7fc2e41a9640 1 --2- 172.21.15.81:0/498029457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2dc0b3200 0x7fc2dc0cc7b0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.711+0000 7fc2e41a9640 1 -- 172.21.15.81:0/498029457 >> 172.21.15.81:0/498029457 conn(0x7fc2dc0b3680 msgr2=0x7fc2dc0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:06.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.711+0000 7fc2e41a9640 1 -- 172.21.15.81:0/498029457 shutdown_connections 2024-06-29T14:42:06.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.711+0000 7fc2e41a9640 1 -- 172.21.15.81:0/498029457 wait complete. 2024-06-29T14:42:06.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.713+0000 7fc2e41a9640 1 Processor -- start 2024-06-29T14:42:06.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.713+0000 7fc2e41a9640 1 -- start start 2024-06-29T14:42:06.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e41a9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2dc0b3200 0x7fc2dc1e6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e41a9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2dc0cccf0 0x7fc2dc06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e41a9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2dc06d940 0x7fc2dc1dccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e41a9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc0b5bb0 con 0x7fc2dc0b3200 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e41a9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc0b5a30 con 0x7fc2dc06d940 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e41a9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc0b5d30 con 0x7fc2dc0cccf0 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e1f45640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2dc0b3200 0x7fc2dc1e6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e1f45640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2dc0b3200 0x7fc2dc1e6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55878/0 (socket says 172.21.15.81:55878) 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e1f45640 1 -- 172.21.15.81:0/3882418095 learned_addr learned my addr 172.21.15.81:0/3882418095 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e2746640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2dc06d940 0x7fc2dc1dccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e1744640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2dc0cccf0 0x7fc2dc06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e1f45640 1 -- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2dc0cccf0 msgr2=0x7fc2dc06d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e1f45640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2dc0cccf0 0x7fc2dc06d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e1f45640 1 -- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2dc06d940 msgr2=0x7fc2dc1dccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e1f45640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2dc06d940 0x7fc2dc1dccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e1f45640 1 -- 172.21.15.81:0/3882418095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc1e7d00 con 0x7fc2dc0b3200 2024-06-29T14:42:06.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e1f45640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2dc0b3200 0x7fc2dc1e6e00 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009780 tx=0x7fc2d4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:06.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2da7fc640 1 -- 172.21.15.81:0/3882418095 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4031d60 con 0x7fc2dc0b3200 2024-06-29T14:42:06.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2da7fc640 1 -- 172.21.15.81:0/3882418095 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2d4033e10 con 0x7fc2dc0b3200 2024-06-29T14:42:06.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2da7fc640 1 -- 172.21.15.81:0/3882418095 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4005260 con 0x7fc2dc0b3200 2024-06-29T14:42:06.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.715+0000 7fc2e41a9640 1 -- 172.21.15.81:0/3882418095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2dc1e7fe0 con 0x7fc2dc0b3200 2024-06-29T14:42:06.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.716+0000 7fc2e41a9640 1 -- 172.21.15.81:0/3882418095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2dc1e68e0 con 0x7fc2dc0b3200 2024-06-29T14:42:06.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.717+0000 7fc2e41a9640 1 -- 172.21.15.81:0/3882418095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2dc0b7400 con 0x7fc2dc0b3200 2024-06-29T14:42:06.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.717+0000 7fc2da7fc640 1 -- 172.21.15.81:0/3882418095 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 79) ==== 98320+0+0 (secure 0 0 0) 0x7fc2d4005400 con 0x7fc2dc0b3200 2024-06-29T14:42:06.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.718+0000 7fc2da7fc640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fc2c8076470 0x7fc2c8078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:06.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.718+0000 7fc2e1744640 1 -- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fc2c8076470 msgr2=0x7fc2c8078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3201324170 2024-06-29T14:42:06.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.718+0000 7fc2da7fc640 1 -- 172.21.15.81:0/3882418095 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7fc2d40b8a10 con 0x7fc2dc0b3200 2024-06-29T14:42:06.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.718+0000 7fc2e1744640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fc2c8076470 0x7fc2c8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:42:06.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.722+0000 7fc2da7fc640 1 -- 172.21.15.81:0/3882418095 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc2d403f030 con 0x7fc2dc0b3200 2024-06-29T14:42:06.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.909+0000 7fc2e41a9640 1 -- 172.21.15.81:0/3882418095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fc2dc1e63f0 con 0x7fc2dc0b3200 2024-06-29T14:42:06.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.912+0000 7fc2da7fc640 1 -- 172.21.15.81:0/3882418095 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v57)=0 v57) ==== 138+0+0 (secure 0 0 0) 0x7fc2d408a0d0 con 0x7fc2dc0b3200 2024-06-29T14:42:06.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 -- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fc2c8076470 msgr2=0x7fc2c8078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:42:06.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fc2c8076470 0x7fc2c8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 -- 172.21.15.81:0/3882418095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2dc0b3200 msgr2=0x7fc2dc1e6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2dc0b3200 0x7fc2dc1e6e00 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009780 tx=0x7fc2d4033c20 comp rx=0 tx=0).stop 2024-06-29T14:42:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 -- 172.21.15.81:0/3882418095 shutdown_connections 2024-06-29T14:42:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7fc2c8076470 0x7fc2c8078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2dc06d940 0x7fc2dc1dccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2dc0cccf0 0x7fc2dc06d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 --2- 172.21.15.81:0/3882418095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2dc0b3200 0x7fc2dc1e6e00 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 -- 172.21.15.81:0/3882418095 >> 172.21.15.81:0/3882418095 conn(0x7fc2dc0b3680 msgr2=0x7fc2dc0cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 -- 172.21.15.81:0/3882418095 shutdown_connections 2024-06-29T14:42:06.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:06.915+0000 7fc2e41a9640 1 -- 172.21.15.81:0/3882418095 wait complete. 2024-06-29T14:42:06.924 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T14:42:07.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.045+0000 7ffb706ef640 1 -- 172.21.15.81:0/342106114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb6806c830 msgr2=0x7ffb6806f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.045+0000 7ffb706ef640 1 --2- 172.21.15.81:0/342106114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb6806c830 0x7ffb6806f9a0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7ffb68074090 tx=0x7ffb6002aa30 comp rx=0 tx=0).stop 2024-06-29T14:42:07.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.045+0000 7ffb706ef640 1 -- 172.21.15.81:0/342106114 shutdown_connections 2024-06-29T14:42:07.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.045+0000 7ffb706ef640 1 --2- 172.21.15.81:0/342106114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb6806c830 0x7ffb6806f9a0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.045+0000 7ffb706ef640 1 --2- 172.21.15.81:0/342106114 >> v2:172.21.15.167:3300/0 conn(0x7ffb680d0550 0x7ffb6806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.045+0000 7ffb706ef640 1 --2- 172.21.15.81:0/342106114 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb68159bd0 0x7ffb680d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.045+0000 7ffb706ef640 1 -- 172.21.15.81:0/342106114 >> 172.21.15.81:0/342106114 conn(0x7ffb680b2920 msgr2=0x7ffb680b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:07.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.045+0000 7ffb706ef640 1 -- 172.21.15.81:0/342106114 shutdown_connections 2024-06-29T14:42:07.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.045+0000 7ffb706ef640 1 -- 172.21.15.81:0/342106114 wait complete. 2024-06-29T14:42:07.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.047+0000 7ffb706ef640 1 Processor -- start 2024-06-29T14:42:07.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.047+0000 7ffb706ef640 1 -- start start 2024-06-29T14:42:07.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.048+0000 7ffb706ef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb6806c830 0x7ffb680afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.048+0000 7ffb706ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb680d0550 0x7ffb68132ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.048+0000 7ffb706ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb68159bd0 0x7ffb680a9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.048+0000 7ffb706ef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6815fa90 con 0x7ffb68159bd0 2024-06-29T14:42:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.048+0000 7ffb706ef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ffb680702e0 con 0x7ffb6806c830 2024-06-29T14:42:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.048+0000 7ffb706ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6815fc10 con 0x7ffb680d0550 2024-06-29T14:42:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6ec8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb68159bd0 0x7ffb680a9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6ec8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb68159bd0 0x7ffb680a9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55888/0 (socket says 172.21.15.81:55888) 2024-06-29T14:42:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6ec8c640 1 -- 172.21.15.81:0/3732875154 learned_addr learned my addr 172.21.15.81:0/3732875154 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6e48b640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb6806c830 0x7ffb680afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6dc8a640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb680d0550 0x7ffb68132ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6ec8c640 1 -- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb680d0550 msgr2=0x7ffb68132ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6ec8c640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb680d0550 0x7ffb68132ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6ec8c640 1 -- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb6806c830 msgr2=0x7ffb680afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6ec8c640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb6806c830 0x7ffb680afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6ec8c640 1 -- 172.21.15.81:0/3732875154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb68135a10 con 0x7ffb68159bd0 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb6ec8c640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb68159bd0 0x7ffb680a9f10 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005fd0 tx=0x7ffb60003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb5effd640 1 -- 172.21.15.81:0/3732875154 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60033050 con 0x7ffb68159bd0 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb5effd640 1 -- 172.21.15.81:0/3732875154 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb60004540 con 0x7ffb68159bd0 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb5effd640 1 -- 172.21.15.81:0/3732875154 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60038450 con 0x7ffb68159bd0 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb706ef640 1 -- 172.21.15.81:0/3732875154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb68135cf0 con 0x7ffb68159bd0 2024-06-29T14:42:07.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.049+0000 7ffb706ef640 1 -- 172.21.15.81:0/3732875154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb68139230 con 0x7ffb68159bd0 2024-06-29T14:42:07.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.050+0000 7ffb706ef640 1 -- 172.21.15.81:0/3732875154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb6806c2f0 con 0x7ffb68159bd0 2024-06-29T14:42:07.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.051+0000 7ffb5effd640 1 -- 172.21.15.81:0/3732875154 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 79) ==== 98320+0+0 (secure 0 0 0) 0x7ffb60036070 con 0x7ffb68159bd0 2024-06-29T14:42:07.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.052+0000 7ffb5effd640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffb4c076440 0x7ffb4c078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.052+0000 7ffb6e48b640 1 -- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffb4c076440 msgr2=0x7ffb4c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3201324170 2024-06-29T14:42:07.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.052+0000 7ffb5effd640 1 -- 172.21.15.81:0/3732875154 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7ffb600b8c50 con 0x7ffb68159bd0 2024-06-29T14:42:07.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.052+0000 7ffb6e48b640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffb4c076440 0x7ffb4c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:42:07.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.056+0000 7ffb5effd640 1 -- 172.21.15.81:0/3732875154 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffb60081e40 con 0x7ffb68159bd0 2024-06-29T14:42:07.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.245+0000 7ffb706ef640 1 -- 172.21.15.81:0/3732875154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7ffb68135ec0 con 0x7ffb68159bd0 2024-06-29T14:42:07.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.251+0000 7ffb5effd640 1 -- 172.21.15.81:0/3732875154 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v58)=0 v58) ==== 133+0+0 (secure 0 0 0) 0x7ffb60086140 con 0x7ffb68159bd0 2024-06-29T14:42:07.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 -- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffb4c076440 msgr2=0x7ffb4c078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:42:07.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffb4c076440 0x7ffb4c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 -- 172.21.15.81:0/3732875154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb68159bd0 msgr2=0x7ffb680a9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb68159bd0 0x7ffb680a9f10 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7ffb60005fd0 tx=0x7ffb60003f90 comp rx=0 tx=0).stop 2024-06-29T14:42:07.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 -- 172.21.15.81:0/3732875154 shutdown_connections 2024-06-29T14:42:07.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ffb4c076440 0x7ffb4c078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb68159bd0 0x7ffb680a9f10 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb680d0550 0x7ffb68132ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 --2- 172.21.15.81:0/3732875154 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb6806c830 0x7ffb680afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 -- 172.21.15.81:0/3732875154 >> 172.21.15.81:0/3732875154 conn(0x7ffb680b2920 msgr2=0x7ffb68072ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:07.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 -- 172.21.15.81:0/3732875154 shutdown_connections 2024-06-29T14:42:07.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.253+0000 7ffb706ef640 1 -- 172.21.15.81:0/3732875154 wait complete. 2024-06-29T14:42:07.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T14:42:07.262 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T14:42:07.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2024-06-29T14:42:07.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.443+0000 7ff2ebce4640 1 -- 172.21.15.81:0/498741264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2e415bdf0 msgr2=0x7ff2e4165ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.443+0000 7ff2ebce4640 1 --2- 172.21.15.81:0/498741264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2e415bdf0 0x7ff2e4165ab0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc005fd0 tx=0x7ff2dc02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.443+0000 7ff2ebce4640 1 -- 172.21.15.81:0/498741264 shutdown_connections 2024-06-29T14:42:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.443+0000 7ff2ebce4640 1 --2- 172.21.15.81:0/498741264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2e415bdf0 0x7ff2e4165ab0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.443+0000 7ff2ebce4640 1 -- 172.21.15.81:0/498741264 >> 172.21.15.81:0/498741264 conn(0x7ff2e40b2920 msgr2=0x7ff2e40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.443+0000 7ff2ebce4640 1 -- 172.21.15.81:0/498741264 shutdown_connections 2024-06-29T14:42:07.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.443+0000 7ff2ebce4640 1 -- 172.21.15.81:0/498741264 wait complete. 2024-06-29T14:42:07.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.444+0000 7ff2ebce4640 1 Processor -- start 2024-06-29T14:42:07.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.445+0000 7ff2ebce4640 1 -- start start 2024-06-29T14:42:07.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ebce4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2e40cd150 0x7ff2e41418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ebce4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2e415bdf0 0x7ff2e415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ebce4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2e41436c0 0x7ff2e415aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ebce4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e4166170 con 0x7ff2e41436c0 2024-06-29T14:42:07.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ebce4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e4165ff0 con 0x7ff2e415bdf0 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ebce4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff2e41662f0 con 0x7ff2e40cd150 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ea281640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2e41436c0 0x7ff2e415aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ea281640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2e41436c0 0x7ff2e415aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55894/0 (socket says 172.21.15.81:55894) 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ea281640 1 -- 172.21.15.81:0/2073325644 learned_addr learned my addr 172.21.15.81:0/2073325644 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2e927f640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2e415bdf0 0x7ff2e415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2e9a80640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2e40cd150 0x7ff2e41418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ea281640 1 -- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2e40cd150 msgr2=0x7ff2e41418b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ea281640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2e40cd150 0x7ff2e41418b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ea281640 1 -- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2e415bdf0 msgr2=0x7ff2e415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ea281640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff2e415bdf0 0x7ff2e415a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.446+0000 7ff2ea281640 1 -- 172.21.15.81:0/2073325644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e41424d0 con 0x7ff2e41436c0 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.447+0000 7ff2ea281640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2e41436c0 0x7ff2e415aa20 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc009cd0 tx=0x7ff2dc003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:07.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.447+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc004450 con 0x7ff2e41436c0 2024-06-29T14:42:07.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.447+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2dc004610 con 0x7ff2e41436c0 2024-06-29T14:42:07.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.447+0000 7ff2ebce4640 1 -- 172.21.15.81:0/2073325644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2e41427b0 con 0x7ff2e41436c0 2024-06-29T14:42:07.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.447+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc003e10 con 0x7ff2e41436c0 2024-06-29T14:42:07.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.447+0000 7ff2ebce4640 1 -- 172.21.15.81:0/2073325644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2e4142d70 con 0x7ff2e41436c0 2024-06-29T14:42:07.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.448+0000 7ff2ebce4640 1 -- 172.21.15.81:0/2073325644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2e415b6e0 con 0x7ff2e41436c0 2024-06-29T14:42:07.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.448+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 79) ==== 98320+0+0 (secure 0 0 0) 0x7ff2dc03d430 con 0x7ff2e41436c0 2024-06-29T14:42:07.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.449+0000 7ff2da7fc640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 0x7ff2c8078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:07.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.449+0000 7ff2e9a80640 1 -- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 msgr2=0x7ff2c8078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3201324170 2024-06-29T14:42:07.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.449+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 5809+0+0 (secure 0 0 0) 0x7ff2dc0bb6e0 con 0x7ff2e41436c0 2024-06-29T14:42:07.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.449+0000 7ff2e9a80640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 0x7ff2c8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T14:42:07.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.453+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff2dc041030 con 0x7ff2e41436c0 2024-06-29T14:42:07.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.650+0000 7ff2e9a80640 1 -- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 msgr2=0x7ff2c8078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3201324170 2024-06-29T14:42:07.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.650+0000 7ff2e9a80640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 0x7ff2c8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T14:42:07.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:07.667+0000 7ff2ebce4640 1 -- 172.21.15.81:0/2073325644 --> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7ff2e41421f0 con 0x7ff2c8076440 2024-06-29T14:42:08.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:08.051+0000 7ff2e9a80640 1 -- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 msgr2=0x7ff2c8078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3201324170 2024-06-29T14:42:08.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:08.051+0000 7ff2e9a80640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 0x7ff2c8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T14:42:08.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:08.852+0000 7ff2e9a80640 1 -- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 msgr2=0x7ff2c8078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3201324170 2024-06-29T14:42:08.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:08.852+0000 7ff2e9a80640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 0x7ff2c8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T14:42:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:10.222+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 80) ==== 97929+0+0 (secure 0 0 0) 0x7ff2dc0b9b90 con 0x7ff2e41436c0 2024-06-29T14:42:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:10.222+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 msgr2=0x7ff2c8078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T14:42:10.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:10.222+0000 7ff2da7fc640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/3201324170,v1:172.21.15.167:6833/3201324170] conn(0x7ff2c8076440 0x7ff2c8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:10.365 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:42:10.367+0000 7f5e6dd31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.365 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:42:10.367+0000 7f5e6dd31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.365 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:42:10.367+0000 7f5e6dd31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.365 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:42:10.367+0000 7f5e6dd31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.365 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:42:10.367+0000 7f5e6dd31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.366 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:42:10.368+0000 7f5e70d37640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.366 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:42:10.368+0000 7f5e70d37640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.366 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:42:10.368+0000 7f5e70d37640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.366 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:42:10.368+0000 7f5e70d37640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:10.366 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:42:10.368+0000 7f5e70d37640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:42:11.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.222+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 81) ==== 98056+0+0 (secure 0 0 0) 0x7ff2dc0809f0 con 0x7ff2e41436c0 2024-06-29T14:42:11.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.222+0000 7ff2da7fc640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7ff2c803afb0 0x7ff2c8080490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.223+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 --> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7ff2dc04cec0 con 0x7ff2c803afb0 2024-06-29T14:42:11.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.225+0000 7ff2e9a80640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7ff2c803afb0 0x7ff2c8080490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.230+0000 7ff2e9a80640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7ff2c803afb0 0x7ff2c8080490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0000f60 tx=0x7ff2e0007df0 comp rx=0 tx=0).ready entity=mgr.7502 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:11.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.291+0000 7ff2da7fc640 1 -- 172.21.15.81:0/2073325644 <== mgr.7502 v2:172.21.15.167:6832/828186388 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff2dc04cec0 con 0x7ff2c803afb0 2024-06-29T14:42:11.289 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1 2024-06-29T14:42:11.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.293+0000 7ff2ebce4640 1 -- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7ff2c803afb0 msgr2=0x7ff2c8080490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.293+0000 7ff2ebce4640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7ff2c803afb0 0x7ff2c8080490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0000f60 tx=0x7ff2e0007df0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.294+0000 7ff2ebce4640 1 -- 172.21.15.81:0/2073325644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2e41436c0 msgr2=0x7ff2e415aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.294+0000 7ff2ebce4640 1 --2- 172.21.15.81:0/2073325644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2e41436c0 0x7ff2e415aa20 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc009cd0 tx=0x7ff2dc003c20 comp rx=0 tx=0).stop 2024-06-29T14:42:11.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.294+0000 7ff2ea281640 1 -- 172.21.15.81:0/2073325644 reap_dead start 2024-06-29T14:42:11.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.294+0000 7ff2ebce4640 1 -- 172.21.15.81:0/2073325644 shutdown_connections 2024-06-29T14:42:11.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.294+0000 7ff2ebce4640 1 -- 172.21.15.81:0/2073325644 >> 172.21.15.81:0/2073325644 conn(0x7ff2e40b2920 msgr2=0x7ff2e414d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:11.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.294+0000 7ff2ebce4640 1 -- 172.21.15.81:0/2073325644 shutdown_connections 2024-06-29T14:42:11.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.294+0000 7ff2ebce4640 1 -- 172.21.15.81:0/2073325644 wait complete. 2024-06-29T14:42:11.303 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2024-06-29T14:42:11.303 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2024-06-29T14:42:11.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.424+0000 7f397339e640 1 -- 172.21.15.81:0/2037865251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f396c1555e0 msgr2=0x7f396c15bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.424+0000 7f397339e640 1 --2- 172.21.15.81:0/2037865251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f396c1555e0 0x7f396c15bea0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f395c005fd0 tx=0x7f395c02a8b0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.424+0000 7f397339e640 1 -- 172.21.15.81:0/2037865251 shutdown_connections 2024-06-29T14:42:11.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.424+0000 7f397339e640 1 --2- 172.21.15.81:0/2037865251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f396c0708f0 0x7f396c0b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.424+0000 7f397339e640 1 --2- 172.21.15.81:0/2037865251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f396c1555e0 0x7f396c15bea0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.424+0000 7f397339e640 1 -- 172.21.15.81:0/2037865251 >> 172.21.15.81:0/2037865251 conn(0x7f396c0d08f0 msgr2=0x7f396c0d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:11.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.424+0000 7f397339e640 1 -- 172.21.15.81:0/2037865251 shutdown_connections 2024-06-29T14:42:11.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.424+0000 7f397339e640 1 -- 172.21.15.81:0/2037865251 wait complete. 2024-06-29T14:42:11.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.426+0000 7f397339e640 1 Processor -- start 2024-06-29T14:42:11.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.426+0000 7f397339e640 1 -- start start 2024-06-29T14:42:11.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.427+0000 7f397339e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f396c0708f0 0x7f396c157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.427+0000 7f397339e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f396c0af730 0x7f396c14b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.427+0000 7f397339e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f396c1555e0 0x7f396c1f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.427+0000 7f397339e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f396c06ff10 con 0x7f396c0708f0 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.427+0000 7f397339e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f396c06fd90 con 0x7f396c0af730 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.427+0000 7f397339e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f396c070090 con 0x7f396c1555e0 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397113a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f396c0708f0 0x7f396c157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397113a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f396c0708f0 0x7f396c157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50302/0 (socket says 172.21.15.81:50302) 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397113a640 1 -- 172.21.15.81:0/2658235635 learned_addr learned my addr 172.21.15.81:0/2658235635 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f3970939640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f396c0af730 0x7f396c14b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397193b640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f396c1555e0 0x7f396c1f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397113a640 1 -- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f396c1555e0 msgr2=0x7f396c1f3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397113a640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f396c1555e0 0x7f396c1f3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397113a640 1 -- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f396c0af730 msgr2=0x7f396c14b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397113a640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f396c0af730 0x7f396c14b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397113a640 1 -- 172.21.15.81:0/2658235635 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c14d1b0 con 0x7f396c0708f0 2024-06-29T14:42:11.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397113a640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f396c0708f0 0x7f396c157c00 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f395c000f80 tx=0x7f395c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:11.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f3969ffb640 1 -- 172.21.15.81:0/2658235635 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c031990 con 0x7f396c0708f0 2024-06-29T14:42:11.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f397339e640 1 -- 172.21.15.81:0/2658235635 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c14d490 con 0x7f396c0708f0 2024-06-29T14:42:11.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f3969ffb640 1 -- 172.21.15.81:0/2658235635 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f395c031c20 con 0x7f396c0708f0 2024-06-29T14:42:11.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.428+0000 7f3969ffb640 1 -- 172.21.15.81:0/2658235635 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c0330d0 con 0x7f396c0708f0 2024-06-29T14:42:11.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.429+0000 7f397339e640 1 -- 172.21.15.81:0/2658235635 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c14da50 con 0x7f396c0708f0 2024-06-29T14:42:11.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.429+0000 7f397339e640 1 -- 172.21.15.81:0/2658235635 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396c075c90 con 0x7f396c0708f0 2024-06-29T14:42:11.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.430+0000 7f3969ffb640 1 -- 172.21.15.81:0/2658235635 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 81) ==== 98056+0+0 (secure 0 0 0) 0x7f395c033270 con 0x7f396c0708f0 2024-06-29T14:42:11.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.431+0000 7f3969ffb640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f39580761a0 0x7f39580786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.431+0000 7f3969ffb640 1 -- 172.21.15.81:0/2658235635 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 6153+0+0 (secure 0 0 0) 0x7f395c0b89e0 con 0x7f396c0708f0 2024-06-29T14:42:11.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.431+0000 7f3970939640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f39580761a0 0x7f39580786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.431+0000 7f3970939640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f39580761a0 0x7f39580786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f39600096c0 tx=0x7f3960009260 comp rx=0 tx=0).ready entity=mgr.7502 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:11.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.435+0000 7f3969ffb640 1 -- 172.21.15.81:0/2658235635 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f396c075c90 con 0x7f396c0708f0 2024-06-29T14:42:11.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.639+0000 7f397339e640 1 -- 172.21.15.81:0/2658235635 --> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) -- 0x7f396c0705b0 con 0x7f39580761a0 2024-06-29T14:42:11.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.695+0000 7f3969ffb640 1 -- 172.21.15.81:0/2658235635 <== mgr.7502 v2:172.21.15.167:6832/828186388 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+40 (secure 0 0 0) 0x7f396c0705b0 con 0x7f39580761a0 2024-06-29T14:42:11.692 INFO:teuthology.orchestra.run.smithi081.stdout:Retention added to path /snap_test_dir1 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 -- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f39580761a0 msgr2=0x7f39580786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f39580761a0 0x7f39580786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f39600096c0 tx=0x7f3960009260 comp rx=0 tx=0).stop 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 -- 172.21.15.81:0/2658235635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f396c0708f0 msgr2=0x7f396c157c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f396c0708f0 0x7f396c157c00 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f395c000f80 tx=0x7f395c003970 comp rx=0 tx=0).stop 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 -- 172.21.15.81:0/2658235635 shutdown_connections 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f39580761a0 0x7f39580786b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f396c1555e0 0x7f396c1f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f396c0af730 0x7f396c14b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 --2- 172.21.15.81:0/2658235635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f396c0708f0 0x7f396c157c00 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 -- 172.21.15.81:0/2658235635 >> 172.21.15.81:0/2658235635 conn(0x7f396c0d08f0 msgr2=0x7f396c154cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 -- 172.21.15.81:0/2658235635 shutdown_connections 2024-06-29T14:42:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.697+0000 7f397339e640 1 -- 172.21.15.81:0/2658235635 wait complete. 2024-06-29T14:42:11.706 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2024-06-29T14:42:11.706 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2024-06-29T14:42:11.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2024-06-29T14:42:11.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.824+0000 7fbb4e281640 1 -- 172.21.15.81:0/1231614854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb480b3200 msgr2=0x7fbb480cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.824+0000 7fbb4e281640 1 --2- 172.21.15.81:0/1231614854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb480b3200 0x7fbb480cc7b0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009830 tx=0x7fbb3802a510 comp rx=0 tx=0).stop 2024-06-29T14:42:11.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.824+0000 7fbb4e281640 1 -- 172.21.15.81:0/1231614854 shutdown_connections 2024-06-29T14:42:11.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.824+0000 7fbb4e281640 1 --2- 172.21.15.81:0/1231614854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb480cccf0 0x7fbb480d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.824+0000 7fbb4e281640 1 --2- 172.21.15.81:0/1231614854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb480b3200 0x7fbb480cc7b0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.824+0000 7fbb4e281640 1 -- 172.21.15.81:0/1231614854 >> 172.21.15.81:0/1231614854 conn(0x7fbb480b3680 msgr2=0x7fbb480b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:11.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.824+0000 7fbb4e281640 1 -- 172.21.15.81:0/1231614854 shutdown_connections 2024-06-29T14:42:11.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.824+0000 7fbb4e281640 1 -- 172.21.15.81:0/1231614854 wait complete. 2024-06-29T14:42:11.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.826+0000 7fbb4e281640 1 Processor -- start 2024-06-29T14:42:11.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.826+0000 7fbb4e281640 1 -- start start 2024-06-29T14:42:11.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.827+0000 7fbb4e281640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb480b3200 0x7fbb481e2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.827+0000 7fbb4e281640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb480cccf0 0x7fbb481e2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.827+0000 7fbb4e281640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb480d0770 0x7fbb481e2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.827+0000 7fbb4e281640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb480cedb0 con 0x7fbb480cccf0 2024-06-29T14:42:11.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.827+0000 7fbb4e281640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbb480cec30 con 0x7fbb480b3200 2024-06-29T14:42:11.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.827+0000 7fbb4e281640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbb480cef30 con 0x7fbb480d0770 2024-06-29T14:42:11.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb477fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb480cccf0 0x7fbb481e2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb477fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb480cccf0 0x7fbb481e2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50320/0 (socket says 172.21.15.81:50320) 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb477fe640 1 -- 172.21.15.81:0/665811940 learned_addr learned my addr 172.21.15.81:0/665811940 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb47fff640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb480b3200 0x7fbb481e2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb4c81e640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb480d0770 0x7fbb481e2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb477fe640 1 -- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb480d0770 msgr2=0x7fbb481e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb477fe640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb480d0770 0x7fbb481e2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb477fe640 1 -- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb480b3200 msgr2=0x7fbb481e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb477fe640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb480b3200 0x7fbb481e2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb477fe640 1 -- 172.21.15.81:0/665811940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4806e400 con 0x7fbb480cccf0 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb477fe640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb480cccf0 0x7fbb481e2a50 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00d740 tx=0x7fbb3c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb44ff9640 1 -- 172.21.15.81:0/665811940 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c017070 con 0x7fbb480cccf0 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb44ff9640 1 -- 172.21.15.81:0/665811940 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb3c009ab0 con 0x7fbb480cccf0 2024-06-29T14:42:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb4e281640 1 -- 172.21.15.81:0/665811940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb4806e6e0 con 0x7fbb480cccf0 2024-06-29T14:42:11.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb44ff9640 1 -- 172.21.15.81:0/665811940 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c010460 con 0x7fbb480cccf0 2024-06-29T14:42:11.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.828+0000 7fbb4e281640 1 -- 172.21.15.81:0/665811940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb48156ff0 con 0x7fbb480cccf0 2024-06-29T14:42:11.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.829+0000 7fbb4e281640 1 -- 172.21.15.81:0/665811940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb480cc7b0 con 0x7fbb480cccf0 2024-06-29T14:42:11.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.830+0000 7fbb44ff9640 1 -- 172.21.15.81:0/665811940 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 81) ==== 98056+0+0 (secure 0 0 0) 0x7fbb3c01a070 con 0x7fbb480cccf0 2024-06-29T14:42:11.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.830+0000 7fbb44ff9640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fbb340761a0 0x7fbb340786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T14:42:11.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.831+0000 7fbb44ff9640 1 -- 172.21.15.81:0/665811940 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 6153+0+0 (secure 0 0 0) 0x7fbb3c0975e0 con 0x7fbb480cccf0 2024-06-29T14:42:11.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.831+0000 7fbb47fff640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fbb340761a0 0x7fbb340786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T14:42:11.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.831+0000 7fbb47fff640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fbb340761a0 0x7fbb340786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbb38034000 tx=0x7fbb38004730 comp rx=0 tx=0).ready entity=mgr.7502 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T14:42:11.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:11.835+0000 7fbb44ff9640 1 -- 172.21.15.81:0/665811940 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb3c0609f0 con 0x7fbb480cccf0 2024-06-29T14:42:12.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.038+0000 7fbb4e281640 1 -- 172.21.15.81:0/665811940 --> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb4815a4e0 con 0x7fbb340761a0 2024-06-29T14:42:12.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.045+0000 7fbb44ff9640 1 -- 172.21.15.81:0/665811940 <== mgr.7502 v2:172.21.15.167:6832/828186388 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+74 (secure 0 0 0) 0x7fbb4815a4e0 con 0x7fbb340761a0 2024-06-29T14:42:12.042 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T14:42:12.042 INFO:teuthology.orchestra.run.smithi081.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-06-29T14:42:12.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 -- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fbb340761a0 msgr2=0x7fbb340786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:12.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fbb340761a0 0x7fbb340786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbb38034000 tx=0x7fbb38004730 comp rx=0 tx=0).stop 2024-06-29T14:42:12.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 -- 172.21.15.81:0/665811940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb480cccf0 msgr2=0x7fbb481e2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T14:42:12.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb480cccf0 0x7fbb481e2a50 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00d740 tx=0x7fbb3c00dc00 comp rx=0 tx=0).stop 2024-06-29T14:42:12.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 -- 172.21.15.81:0/665811940 shutdown_connections 2024-06-29T14:42:12.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fbb340761a0 0x7fbb340786b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:12.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb480d0770 0x7fbb481e2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:12.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb480cccf0 0x7fbb481e2a50 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:12.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 --2- 172.21.15.81:0/665811940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb480b3200 0x7fbb481e2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T14:42:12.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 -- 172.21.15.81:0/665811940 >> 172.21.15.81:0/665811940 conn(0x7fbb480b3680 msgr2=0x7fbb48070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T14:42:12.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 -- 172.21.15.81:0/665811940 shutdown_connections 2024-06-29T14:42:12.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T14:42:12.047+0000 7fbb4e281640 1 -- 172.21.15.81:0/665811940 wait complete. 2024-06-29T14:42:12.056 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2024-06-29T14:42:12.056 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2024-06-29T14:42:12.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:42:12.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:42:42.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:42:42.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:43:00.703 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:43:00.705+0000 7f5e5fd15640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:43:00.703 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:43:00.705+0000 7f5e5fd15640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:43:00.703 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:43:00.705+0000 7f5e5fd15640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:43:00.703 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:43:00.705+0000 7f5e5fd15640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:43:00.704 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T14:43:00.705+0000 7f5e5fd15640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T14:43:12.482 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:43:12.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:43:42.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:43:42.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:44:12.773 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:44:12.776 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:44:42.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:44:42.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:45:13.026 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:45:13.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:45:43.180 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:45:43.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:46:13.344 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:46:13.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:46:43.417 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:46:43.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:47:13.569 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:47:13.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:47:43.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:47:43.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:48:13.845 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:48:13.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:48:44.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:48:44.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:49:14.081 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:49:14.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:49:44.203 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:49:44.207 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:50:14.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:50:14.362 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:50:44.503 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:50:44.507 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:51:14.634 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:51:14.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:51:44.732 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:51:44.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:52:14.867 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:52:14.871 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:52:45.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:52:45.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:53:15.252 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:53:15.256 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:53:45.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:53:45.339 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:54:15.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:54:15.452 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:54:45.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:54:45.584 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:55:15.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:55:15.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:55:45.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:55:45.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:56:16.121 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:56:16.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:56:46.243 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:56:46.247 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:57:16.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:57:16.389 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:57:46.522 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:57:46.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:58:16.670 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:58:16.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:58:46.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:58:46.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:59:16.943 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:59:16.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:59:47.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T14:59:47.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:00:17.220 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:00:17.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:00:47.367 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:00:47.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:01:17.507 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:01:17.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:01:47.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:01:47.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:02:17.712 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:02:17.716 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:02:47.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:02:47.870 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:03:18.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:03:18.028 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:03:48.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:03:48.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:04:18.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:04:18.230 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:04:48.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:04:48.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:05:18.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:05:18.442 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:05:48.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:05:48.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:06:18.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:06:18.705 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:06:48.985 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:06:48.988 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:07:19.132 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:07:19.137 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:07:49.247 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:07:49.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:08:19.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:08:19.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:08:49.723 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:08:49.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:09:19.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:09:19.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:09:50.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:09:50.078 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:10:20.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:10:20.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:10:50.343 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:10:50.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:11:20.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:11:20.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:11:50.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:11:50.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:12:20.790 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:12:20.793 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:12:50.954 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:12:50.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:13:21.031 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:13:21.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:13:51.162 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:13:51.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:14:21.312 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:14:21.316 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:14:51.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:14:51.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:15:21.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:15:21.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:15:51.695 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:15:51.699 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:16:21.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:16:21.813 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:16:51.965 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:16:51.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:17:22.038 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:17:22.042 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:17:52.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:17:52.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:18:22.243 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:18:22.247 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:18:52.379 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:18:52.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:19:22.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:19:22.522 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:19:52.781 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:19:52.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:20:22.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:20:22.972 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:20:53.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:20:53.146 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:21:23.290 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:21:23.294 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:21:53.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:21:53.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:22:23.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:22:23.566 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:22:53.709 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:22:53.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:23:23.830 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:23:23.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:23:54.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:23:54.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:24:24.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:24:24.171 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:24:54.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:24:54.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:25:24.379 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:25:24.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:25:54.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:25:54.504 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:26:24.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:26:24.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:26:54.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:26:54.795 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:27:24.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:27:24.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:27:55.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:27:55.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:28:25.177 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:28:25.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:28:55.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:28:55.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:29:25.432 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:29:25.436 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:29:55.646 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:29:55.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:30:25.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:30:25.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:30:55.883 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:30:55.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:31:26.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:31:26.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:31:56.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:31:56.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:32:26.310 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:32:26.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:32:56.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:32:56.412 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:33:26.496 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:33:26.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:33:56.619 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:33:56.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:34:26.732 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:34:26.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:34:56.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:34:56.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:35:26.944 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:35:26.948 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:35:57.261 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:35:57.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:36:27.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:36:27.463 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:36:57.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:36:57.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:37:27.847 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:37:27.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:37:57.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:37:57.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:38:16.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:16.058 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:38:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_44_00_UTC 2024-06-29T15:38:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_45_00_UTC 2024-06-29T15:38:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_46_00_UTC 2024-06-29T15:38:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_47_00_UTC 2024-06-29T15:38:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_48_00_UTC 2024-06-29T15:38:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_49_00_UTC 2024-06-29T15:38:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_50_00_UTC 2024-06-29T15:38:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_51_00_UTC 2024-06-29T15:38:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_52_00_UTC 2024-06-29T15:38:16.084 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_53_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_54_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_55_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_56_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_57_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_58_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_59_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_00_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_01_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_02_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_03_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_04_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_05_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_06_00_UTC 2024-06-29T15:38:16.085 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_07_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_08_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_09_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_10_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_11_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_12_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_13_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_14_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_15_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_16_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_17_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_18_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_19_00_UTC 2024-06-29T15:38:16.086 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_20_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_21_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_22_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_23_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_24_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_25_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_26_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_27_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_28_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_29_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_30_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_31_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_32_00_UTC 2024-06-29T15:38:16.087 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_33_00_UTC 2024-06-29T15:38:16.088 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_34_00_UTC 2024-06-29T15:38:16.088 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_35_00_UTC 2024-06-29T15:38:16.088 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_36_00_UTC 2024-06-29T15:38:16.088 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_37_00_UTC 2024-06-29T15:38:16.088 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_38_00_UTC 2024-06-29T15:38:16.088 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-14_44_00_UTC', 'scheduled-2024-06-29-14_45_00_UTC', 'scheduled-2024-06-29-14_46_00_UTC', 'scheduled-2024-06-29-14_47_00_UTC', 'scheduled-2024-06-29-14_48_00_UTC', 'scheduled-2024-06-29-14_49_00_UTC', 'scheduled-2024-06-29-14_50_00_UTC', 'scheduled-2024-06-29-14_51_00_UTC', 'scheduled-2024-06-29-14_52_00_UTC', 'scheduled-2024-06-29-14_53_00_UTC', 'scheduled-2024-06-29-14_54_00_UTC', 'scheduled-2024-06-29-14_55_00_UTC', 'scheduled-2024-06-29-14_56_00_UTC', 'scheduled-2024-06-29-14_57_00_UTC', 'scheduled-2024-06-29-14_58_00_UTC', 'scheduled-2024-06-29-14_59_00_UTC', 'scheduled-2024-06-29-15_00_00_UTC', 'scheduled-2024-06-29-15_01_00_UTC', 'scheduled-2024-06-29-15_02_00_UTC', 'scheduled-2024-06-29-15_03_00_UTC', 'scheduled-2024-06-29-15_04_00_UTC', 'scheduled-2024-06-29-15_05_00_UTC', 'scheduled-2024-06-29-15_06_00_UTC', 'scheduled-2024-06-29-15_07_00_UTC', 'scheduled-2024-06-29-15_08_00_UTC', 'scheduled-2024-06-29-15_09_00_UTC', 'scheduled-2024-06-29-15_10_00_UTC', 'scheduled-2024-06-29-15_11_00_UTC', 'scheduled-2024-06-29-15_12_00_UTC', 'scheduled-2024-06-29-15_13_00_UTC', 'scheduled-2024-06-29-15_14_00_UTC', 'scheduled-2024-06-29-15_15_00_UTC', 'scheduled-2024-06-29-15_16_00_UTC', 'scheduled-2024-06-29-15_17_00_UTC', 'scheduled-2024-06-29-15_18_00_UTC', 'scheduled-2024-06-29-15_19_00_UTC', 'scheduled-2024-06-29-15_20_00_UTC', 'scheduled-2024-06-29-15_21_00_UTC', 'scheduled-2024-06-29-15_22_00_UTC', 'scheduled-2024-06-29-15_23_00_UTC', 'scheduled-2024-06-29-15_24_00_UTC', 'scheduled-2024-06-29-15_25_00_UTC', 'scheduled-2024-06-29-15_26_00_UTC', 'scheduled-2024-06-29-15_27_00_UTC', 'scheduled-2024-06-29-15_28_00_UTC', 'scheduled-2024-06-29-15_29_00_UTC', 'scheduled-2024-06-29-15_30_00_UTC', 'scheduled-2024-06-29-15_31_00_UTC', 'scheduled-2024-06-29-15_32_00_UTC', 'scheduled-2024-06-29-15_33_00_UTC', 'scheduled-2024-06-29-15_34_00_UTC', 'scheduled-2024-06-29-15_35_00_UTC', 'scheduled-2024-06-29-15_36_00_UTC', 'scheduled-2024-06-29-15_37_00_UTC', 'scheduled-2024-06-29-15_38_00_UTC'] 2024-06-29T15:38:16.088 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2024-06-29T15:38:16.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.251+0000 7f9b0eb19640 1 -- 172.21.15.81:0/1157304973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b0806f4e0 msgr2=0x7f9b08166aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.251+0000 7f9b0eb19640 1 --2- 172.21.15.81:0/1157304973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b0806f4e0 0x7f9b08166aa0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f9b00005fd0 tx=0x7f9b0002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.251+0000 7f9b0eb19640 1 -- 172.21.15.81:0/1157304973 shutdown_connections 2024-06-29T15:38:16.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.251+0000 7f9b0eb19640 1 --2- 172.21.15.81:0/1157304973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b0806f4e0 0x7f9b08166aa0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.251+0000 7f9b0eb19640 1 -- 172.21.15.81:0/1157304973 >> 172.21.15.81:0/1157304973 conn(0x7f9b0806ccc0 msgr2=0x7f9b080b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:16.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.251+0000 7f9b0eb19640 1 -- 172.21.15.81:0/1157304973 shutdown_connections 2024-06-29T15:38:16.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.251+0000 7f9b0eb19640 1 -- 172.21.15.81:0/1157304973 wait complete. 2024-06-29T15:38:16.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.253+0000 7f9b0eb19640 1 Processor -- start 2024-06-29T15:38:16.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.253+0000 7f9b0eb19640 1 -- start start 2024-06-29T15:38:16.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.254+0000 7f9b0eb19640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b08071f20 0x7f9b081593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.254+0000 7f9b0eb19640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b08144a20 0x7f9b08159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.254+0000 7f9b0eb19640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b08159ff0 0x7f9b081f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.254+0000 7f9b0eb19640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9b08167370 con 0x7f9b08144a20 2024-06-29T15:38:16.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.254+0000 7f9b0eb19640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9b081671f0 con 0x7f9b08071f20 2024-06-29T15:38:16.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.254+0000 7f9b0eb19640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9b081674f0 con 0x7f9b08159ff0 2024-06-29T15:38:16.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.254+0000 7f9b07fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b08144a20 0x7f9b08159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.254+0000 7f9b07fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b08144a20 0x7f9b08159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44750/0 (socket says 172.21.15.81:44750) 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.254+0000 7f9b07fff640 1 -- 172.21.15.81:0/3388403379 learned_addr learned my addr 172.21.15.81:0/3388403379 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b0c8b5640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b08071f20 0x7f9b081593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b0d0b6640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b08159ff0 0x7f9b081f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b07fff640 1 -- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b08159ff0 msgr2=0x7f9b081f5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b07fff640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b08159ff0 0x7f9b081f5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b07fff640 1 -- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b08071f20 msgr2=0x7f9b081593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b07fff640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b08071f20 0x7f9b081593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b07fff640 1 -- 172.21.15.81:0/3388403379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b08137520 con 0x7f9b08144a20 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b07fff640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b08144a20 0x7f9b08159ab0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00c3c0 tx=0x7f9afc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b057fa640 1 -- 172.21.15.81:0/3388403379 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9afc017070 con 0x7f9b08144a20 2024-06-29T15:38:16.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b057fa640 1 -- 172.21.15.81:0/3388403379 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9afc0035f0 con 0x7f9b08144a20 2024-06-29T15:38:16.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b0eb19640 1 -- 172.21.15.81:0/3388403379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b081413c0 con 0x7f9b08144a20 2024-06-29T15:38:16.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b057fa640 1 -- 172.21.15.81:0/3388403379 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9afc003a80 con 0x7f9b08144a20 2024-06-29T15:38:16.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.255+0000 7f9b0eb19640 1 -- 172.21.15.81:0/3388403379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b08137aa0 con 0x7f9b08144a20 2024-06-29T15:38:16.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.256+0000 7f9b0eb19640 1 -- 172.21.15.81:0/3388403379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b08145180 con 0x7f9b08144a20 2024-06-29T15:38:16.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.257+0000 7f9b057fa640 1 -- 172.21.15.81:0/3388403379 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 85) ==== 98332+0+0 (secure 0 0 0) 0x7f9afc01a070 con 0x7f9b08144a20 2024-06-29T15:38:16.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.257+0000 7f9b057fa640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9af4076440 0x7f9af4078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.257+0000 7f9b057fa640 1 -- 172.21.15.81:0/3388403379 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 6255+0+0 (secure 0 0 0) 0x7f9afc096f90 con 0x7f9b08144a20 2024-06-29T15:38:16.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.257+0000 7f9b0c8b5640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9af4076440 0x7f9af4078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.258+0000 7f9b0c8b5640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9af4076440 0x7f9af4078950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9af8006fd0 tx=0x7f9af8008040 comp rx=0 tx=0).ready entity=mgr.7502 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:16.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.262+0000 7f9b057fa640 1 -- 172.21.15.81:0/3388403379 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9b0815c350 con 0x7f9b08144a20 2024-06-29T15:38:16.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.461+0000 7f9b0eb19640 1 -- 172.21.15.81:0/3388403379 --> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9b080cf700 con 0x7f9af4076440 2024-06-29T15:38:16.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.471+0000 7f9b057fa640 1 -- 172.21.15.81:0/3388403379 <== mgr.7502 v2:172.21.15.167:6832/828186388 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+373 (secure 0 0 0) 0x7f9b080cf700 con 0x7f9af4076440 2024-06-29T15:38:16.471 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:16.471 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:42:11", "first": "2024-06-29T14:43:00", "last": "2024-06-29T15:38:00", "last_pruned": "2024-06-29T15:38:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-06-29T15:38:16.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 -- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9af4076440 msgr2=0x7f9af4078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9af4076440 0x7f9af4078950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9af8006fd0 tx=0x7f9af8008040 comp rx=0 tx=0).stop 2024-06-29T15:38:16.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 -- 172.21.15.81:0/3388403379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b08144a20 msgr2=0x7f9b08159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b08144a20 0x7f9b08159ab0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00c3c0 tx=0x7f9afc00c880 comp rx=0 tx=0).stop 2024-06-29T15:38:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 -- 172.21.15.81:0/3388403379 shutdown_connections 2024-06-29T15:38:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9af4076440 0x7f9af4078950 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b08159ff0 0x7f9b081f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b08144a20 0x7f9b08159ab0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 --2- 172.21.15.81:0/3388403379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b08071f20 0x7f9b081593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 -- 172.21.15.81:0/3388403379 >> 172.21.15.81:0/3388403379 conn(0x7f9b0806ccc0 msgr2=0x7f9b0815e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 -- 172.21.15.81:0/3388403379 shutdown_connections 2024-06-29T15:38:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.473+0000 7f9b0eb19640 1 -- 172.21.15.81:0/3388403379 wait complete. 2024-06-29T15:38:16.484 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T14:42:11", "first": "2024-06-29T14:43:00", "last": "2024-06-29T15:38:00", "last_pruned": "2024-06-29T15:38:00", "created_count": 56, "pruned_count": 1, "active": true}] 2024-06-29T15:38:16.484 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T14:42:11', 'first': '2024-06-29T14:43:00', 'last': '2024-06-29T15:38:00', 'last_pruned': '2024-06-29T15:38:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2024-06-29T15:38:16.484 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2024-06-29T15:38:16.484 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2024-06-29T15:38:16.485 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2024-06-29T15:38:16.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.600+0000 7fa697ac1640 1 -- 172.21.15.81:0/624385738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa69015b5b0 msgr2=0x7fa6900b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.600+0000 7fa697ac1640 1 --2- 172.21.15.81:0/624385738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa69015b5b0 0x7fa6900b4430 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fa6900ccf70 tx=0x7fa68002a4a0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.600+0000 7fa697ac1640 1 -- 172.21.15.81:0/624385738 shutdown_connections 2024-06-29T15:38:16.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.600+0000 7fa697ac1640 1 --2- 172.21.15.81:0/624385738 >> v2:172.21.15.167:3300/0 conn(0x7fa69006f660 0x7fa69006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.600+0000 7fa697ac1640 1 --2- 172.21.15.81:0/624385738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa69015b5b0 0x7fa6900b4430 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.600+0000 7fa697ac1640 1 -- 172.21.15.81:0/624385738 >> 172.21.15.81:0/624385738 conn(0x7fa69006ccc0 msgr2=0x7fa6900b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:16.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.600+0000 7fa697ac1640 1 -- 172.21.15.81:0/624385738 shutdown_connections 2024-06-29T15:38:16.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.600+0000 7fa697ac1640 1 -- 172.21.15.81:0/624385738 wait complete. 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.602+0000 7fa697ac1640 1 Processor -- start 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.602+0000 7fa697ac1640 1 -- start start 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa697ac1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa69006f660 0x7fa6900b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa697ac1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6900b35b0 0x7fa6900b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa697ac1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa69015b5b0 0x7fa690146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa697ac1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa690072360 con 0x7fa69006f660 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa697ac1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa6900721e0 con 0x7fa69015b5b0 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa697ac1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa6900724e0 con 0x7fa6900b35b0 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69585d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa69006f660 0x7fa6900b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69585d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa69006f660 0x7fa6900b0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44758/0 (socket says 172.21.15.81:44758) 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69585d640 1 -- 172.21.15.81:0/2484636634 learned_addr learned my addr 172.21.15.81:0/2484636634 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:16.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69605e640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa69015b5b0 0x7fa690146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69505c640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6900b35b0 0x7fa6900b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69585d640 1 -- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6900b35b0 msgr2=0x7fa6900b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69585d640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6900b35b0 0x7fa6900b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69585d640 1 -- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa69015b5b0 msgr2=0x7fa690146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69585d640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa69015b5b0 0x7fa690146110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69585d640 1 -- 172.21.15.81:0/2484636634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6901f1bb0 con 0x7fa69006f660 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69605e640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa69015b5b0 0x7fa690146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.603+0000 7fa69585d640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa69006f660 0x7fa6900b0f00 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00e710 tx=0x7fa68c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.604+0000 7fa6867fc640 1 -- 172.21.15.81:0/2484636634 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c001a70 con 0x7fa69006f660 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.604+0000 7fa6867fc640 1 -- 172.21.15.81:0/2484636634 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa68c001d00 con 0x7fa69006f660 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.604+0000 7fa697ac1640 1 -- 172.21.15.81:0/2484636634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6901f1e90 con 0x7fa69006f660 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.604+0000 7fa6867fc640 1 -- 172.21.15.81:0/2484636634 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c0116b0 con 0x7fa69006f660 2024-06-29T15:38:16.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.604+0000 7fa697ac1640 1 -- 172.21.15.81:0/2484636634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6901f2450 con 0x7fa69006f660 2024-06-29T15:38:16.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.605+0000 7fa697ac1640 1 -- 172.21.15.81:0/2484636634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa690072a00 con 0x7fa69006f660 2024-06-29T15:38:16.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.605+0000 7fa6867fc640 1 -- 172.21.15.81:0/2484636634 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 85) ==== 98332+0+0 (secure 0 0 0) 0x7fa68c011850 con 0x7fa69006f660 2024-06-29T15:38:16.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.606+0000 7fa6867fc640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa674076440 0x7fa674078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:16.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.606+0000 7fa6867fc640 1 -- 172.21.15.81:0/2484636634 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 6255+0+0 (secure 0 0 0) 0x7fa68c09a290 con 0x7fa69006f660 2024-06-29T15:38:16.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.606+0000 7fa69505c640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa674076440 0x7fa674078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:16.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.607+0000 7fa69505c640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa674076440 0x7fa674078950 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa680034000 tx=0x7fa680004730 comp rx=0 tx=0).ready entity=mgr.7502 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:16.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.610+0000 7fa6867fc640 1 -- 172.21.15.81:0/2484636634 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa68c016030 con 0x7fa69006f660 2024-06-29T15:38:16.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.814+0000 7fa697ac1640 1 -- 172.21.15.81:0/2484636634 --> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7fa690069550 con 0x7fa674076440 2024-06-29T15:38:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.899+0000 7fa6867fc640 1 -- 172.21.15.81:0/2484636634 <== mgr.7502 v2:172.21.15.167:6832/828186388 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+42 (secure 0 0 0) 0x7fa690069550 con 0x7fa674076440 2024-06-29T15:38:16.900 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1 2024-06-29T15:38:16.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.901+0000 7fa697ac1640 1 -- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa674076440 msgr2=0x7fa674078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.901+0000 7fa697ac1640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa674076440 0x7fa674078950 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa680034000 tx=0x7fa680004730 comp rx=0 tx=0).stop 2024-06-29T15:38:16.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.901+0000 7fa697ac1640 1 -- 172.21.15.81:0/2484636634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa69006f660 msgr2=0x7fa6900b0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:16.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.901+0000 7fa697ac1640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa69006f660 0x7fa6900b0f00 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00e710 tx=0x7fa68c00ebd0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.902+0000 7fa697ac1640 1 -- 172.21.15.81:0/2484636634 shutdown_connections 2024-06-29T15:38:16.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.902+0000 7fa697ac1640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa674076440 0x7fa674078950 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.902+0000 7fa697ac1640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa69015b5b0 0x7fa690146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.902+0000 7fa697ac1640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa6900b35b0 0x7fa6900b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.902+0000 7fa697ac1640 1 --2- 172.21.15.81:0/2484636634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa69006f660 0x7fa6900b0f00 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.902+0000 7fa697ac1640 1 -- 172.21.15.81:0/2484636634 >> 172.21.15.81:0/2484636634 conn(0x7fa69006ccc0 msgr2=0x7fa6900b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.902+0000 7fa697ac1640 1 -- 172.21.15.81:0/2484636634 shutdown_connections 2024-06-29T15:38:16.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:16.902+0000 7fa697ac1640 1 -- 172.21.15.81:0/2484636634 wait complete. 2024-06-29T15:38:16.913 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2024-06-29T15:38:16.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:16.913 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:38:16.932 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_44_00_UTC 2024-06-29T15:38:16.932 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_45_00_UTC 2024-06-29T15:38:16.932 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_46_00_UTC 2024-06-29T15:38:16.933 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_47_00_UTC 2024-06-29T15:38:16.933 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_48_00_UTC 2024-06-29T15:38:16.933 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_49_00_UTC 2024-06-29T15:38:16.933 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_50_00_UTC 2024-06-29T15:38:16.933 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_51_00_UTC 2024-06-29T15:38:16.933 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_52_00_UTC 2024-06-29T15:38:16.933 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_53_00_UTC 2024-06-29T15:38:16.933 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_54_00_UTC 2024-06-29T15:38:16.933 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_55_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_56_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_57_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_58_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-14_59_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_00_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_01_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_02_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_03_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_04_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_05_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_06_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_07_00_UTC 2024-06-29T15:38:16.934 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_08_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_09_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_10_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_11_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_12_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_13_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_14_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_15_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_16_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_17_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_18_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_19_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_20_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_21_00_UTC 2024-06-29T15:38:16.935 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_22_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_23_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_24_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_25_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_26_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_27_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_28_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_29_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_30_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_31_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_32_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_33_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_34_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_35_00_UTC 2024-06-29T15:38:16.936 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_36_00_UTC 2024-06-29T15:38:16.937 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_37_00_UTC 2024-06-29T15:38:16.937 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_38_00_UTC 2024-06-29T15:38:16.937 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_44_00_UTC 2024-06-29T15:38:16.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:16.937 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_44_00_UTC) 2024-06-29T15:38:16.994 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_45_00_UTC 2024-06-29T15:38:16.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:16.994 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_45_00_UTC) 2024-06-29T15:38:17.055 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_46_00_UTC 2024-06-29T15:38:17.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.055 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_46_00_UTC) 2024-06-29T15:38:17.116 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_47_00_UTC 2024-06-29T15:38:17.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.116 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_47_00_UTC) 2024-06-29T15:38:17.176 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_48_00_UTC 2024-06-29T15:38:17.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.177 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_48_00_UTC) 2024-06-29T15:38:17.238 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_49_00_UTC 2024-06-29T15:38:17.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.238 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_49_00_UTC) 2024-06-29T15:38:17.300 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_50_00_UTC 2024-06-29T15:38:17.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.300 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_50_00_UTC) 2024-06-29T15:38:17.360 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_51_00_UTC 2024-06-29T15:38:17.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.361 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_51_00_UTC) 2024-06-29T15:38:17.421 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_52_00_UTC 2024-06-29T15:38:17.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.422 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_52_00_UTC) 2024-06-29T15:38:17.482 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_53_00_UTC 2024-06-29T15:38:17.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.482 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_53_00_UTC) 2024-06-29T15:38:17.543 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_54_00_UTC 2024-06-29T15:38:17.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.543 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_54_00_UTC) 2024-06-29T15:38:17.603 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_55_00_UTC 2024-06-29T15:38:17.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.604 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_55_00_UTC) 2024-06-29T15:38:17.664 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_56_00_UTC 2024-06-29T15:38:17.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.664 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_56_00_UTC) 2024-06-29T15:38:17.725 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_57_00_UTC 2024-06-29T15:38:17.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.726 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_57_00_UTC) 2024-06-29T15:38:17.786 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_58_00_UTC 2024-06-29T15:38:17.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.786 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_58_00_UTC) 2024-06-29T15:38:17.847 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-14_59_00_UTC 2024-06-29T15:38:17.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.847 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-14_59_00_UTC) 2024-06-29T15:38:17.908 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_00_00_UTC 2024-06-29T15:38:17.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.908 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_00_00_UTC) 2024-06-29T15:38:17.969 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_01_00_UTC 2024-06-29T15:38:17.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:17.969 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_01_00_UTC) 2024-06-29T15:38:18.030 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_02_00_UTC 2024-06-29T15:38:18.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.030 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_02_00_UTC) 2024-06-29T15:38:18.091 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_03_00_UTC 2024-06-29T15:38:18.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.091 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_03_00_UTC) 2024-06-29T15:38:18.152 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_04_00_UTC 2024-06-29T15:38:18.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.153 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_04_00_UTC) 2024-06-29T15:38:18.213 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_05_00_UTC 2024-06-29T15:38:18.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.213 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_05_00_UTC) 2024-06-29T15:38:18.274 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_06_00_UTC 2024-06-29T15:38:18.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.275 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_06_00_UTC) 2024-06-29T15:38:18.336 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_07_00_UTC 2024-06-29T15:38:18.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.337 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_07_00_UTC) 2024-06-29T15:38:18.398 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_08_00_UTC 2024-06-29T15:38:18.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.399 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_08_00_UTC) 2024-06-29T15:38:18.459 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_09_00_UTC 2024-06-29T15:38:18.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.460 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_09_00_UTC) 2024-06-29T15:38:18.519 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_10_00_UTC 2024-06-29T15:38:18.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.520 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_10_00_UTC) 2024-06-29T15:38:18.580 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_11_00_UTC 2024-06-29T15:38:18.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.581 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_11_00_UTC) 2024-06-29T15:38:18.642 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_12_00_UTC 2024-06-29T15:38:18.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.642 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_12_00_UTC) 2024-06-29T15:38:18.702 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_13_00_UTC 2024-06-29T15:38:18.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.703 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_13_00_UTC) 2024-06-29T15:38:18.763 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_14_00_UTC 2024-06-29T15:38:18.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.764 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_14_00_UTC) 2024-06-29T15:38:18.824 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_15_00_UTC 2024-06-29T15:38:18.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.825 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_15_00_UTC) 2024-06-29T15:38:18.885 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_16_00_UTC 2024-06-29T15:38:18.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.886 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_16_00_UTC) 2024-06-29T15:38:18.946 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_17_00_UTC 2024-06-29T15:38:18.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:18.946 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_17_00_UTC) 2024-06-29T15:38:19.007 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_18_00_UTC 2024-06-29T15:38:19.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.008 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_18_00_UTC) 2024-06-29T15:38:19.069 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_19_00_UTC 2024-06-29T15:38:19.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.069 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_19_00_UTC) 2024-06-29T15:38:19.131 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_20_00_UTC 2024-06-29T15:38:19.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.131 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_20_00_UTC) 2024-06-29T15:38:19.192 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_21_00_UTC 2024-06-29T15:38:19.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.193 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_21_00_UTC) 2024-06-29T15:38:19.254 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_22_00_UTC 2024-06-29T15:38:19.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.254 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_22_00_UTC) 2024-06-29T15:38:19.340 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_23_00_UTC 2024-06-29T15:38:19.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.340 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_23_00_UTC) 2024-06-29T15:38:19.377 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_24_00_UTC 2024-06-29T15:38:19.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.377 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_24_00_UTC) 2024-06-29T15:38:19.447 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_25_00_UTC 2024-06-29T15:38:19.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.447 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_25_00_UTC) 2024-06-29T15:38:19.508 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_26_00_UTC 2024-06-29T15:38:19.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.508 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_26_00_UTC) 2024-06-29T15:38:19.569 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_27_00_UTC 2024-06-29T15:38:19.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.570 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_27_00_UTC) 2024-06-29T15:38:19.631 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_28_00_UTC 2024-06-29T15:38:19.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.631 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_28_00_UTC) 2024-06-29T15:38:19.692 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_29_00_UTC 2024-06-29T15:38:19.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.692 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_29_00_UTC) 2024-06-29T15:38:19.753 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_30_00_UTC 2024-06-29T15:38:19.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.754 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_30_00_UTC) 2024-06-29T15:38:19.816 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_31_00_UTC 2024-06-29T15:38:19.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.817 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_31_00_UTC) 2024-06-29T15:38:19.877 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_32_00_UTC 2024-06-29T15:38:19.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.877 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_32_00_UTC) 2024-06-29T15:38:19.943 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_33_00_UTC 2024-06-29T15:38:19.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:19.944 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_33_00_UTC) 2024-06-29T15:38:20.003 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_34_00_UTC 2024-06-29T15:38:20.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.004 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_34_00_UTC) 2024-06-29T15:38:20.065 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_35_00_UTC 2024-06-29T15:38:20.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.065 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_35_00_UTC) 2024-06-29T15:38:20.126 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_36_00_UTC 2024-06-29T15:38:20.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.126 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_36_00_UTC) 2024-06-29T15:38:20.186 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_37_00_UTC 2024-06-29T15:38:20.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.187 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_37_00_UTC) 2024-06-29T15:38:20.248 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_38_00_UTC 2024-06-29T15:38:20.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.248 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_38_00_UTC) 2024-06-29T15:38:20.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:20.314 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T15:38:20.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:38:20.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.527+0000 7f470cec1640 1 -- 172.21.15.81:0/3597700095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47080d2a90 msgr2=0x7f47080d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:20.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.527+0000 7f470cec1640 1 --2- 172.21.15.81:0/3597700095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47080d2a90 0x7f47080d09c0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f470808e230 tx=0x7f470002a940 comp rx=0 tx=0).stop 2024-06-29T15:38:20.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.527+0000 7f470cec1640 1 -- 172.21.15.81:0/3597700095 shutdown_connections 2024-06-29T15:38:20.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.527+0000 7f470cec1640 1 --2- 172.21.15.81:0/3597700095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47080d2a90 0x7f47080d09c0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:20.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.527+0000 7f470cec1640 1 --2- 172.21.15.81:0/3597700095 >> v2:172.21.15.167:3300/0 conn(0x7f47080b9040 0x7f47080d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:20.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.527+0000 7f470cec1640 1 -- 172.21.15.81:0/3597700095 >> 172.21.15.81:0/3597700095 conn(0x7f470809cf30 msgr2=0x7f47080d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:20.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.527+0000 7f470cec1640 1 -- 172.21.15.81:0/3597700095 shutdown_connections 2024-06-29T15:38:20.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.527+0000 7f470cec1640 1 -- 172.21.15.81:0/3597700095 wait complete. 2024-06-29T15:38:20.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.529+0000 7f470cec1640 1 Processor -- start 2024-06-29T15:38:20.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.529+0000 7f470cec1640 1 -- start start 2024-06-29T15:38:20.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.530+0000 7f470cec1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47080b5250 0x7f470814fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:20.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.530+0000 7f470cec1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47080b9040 0x7f470814e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:20.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.530+0000 7f470cec1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47080d2a90 0x7f470814d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:20.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.530+0000 7f470cec1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f47080b7d30 con 0x7f47080b9040 2024-06-29T15:38:20.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.530+0000 7f470cec1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f47080b7bb0 con 0x7f47080b5250 2024-06-29T15:38:20.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.530+0000 7f470cec1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f47080b7eb0 con 0x7f47080d2a90 2024-06-29T15:38:20.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46fdd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47080b9040 0x7f470814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46fdd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47080b9040 0x7f470814e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47848/0 (socket says 172.21.15.81:47848) 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46fdd9b640 1 -- 172.21.15.81:0/759976211 learned_addr learned my addr 172.21.15.81:0/759976211 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f4706d9d640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47080d2a90 0x7f470814d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f470659c640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47080b5250 0x7f470814fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46fdd9b640 1 -- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47080d2a90 msgr2=0x7f470814d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46fdd9b640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47080d2a90 0x7f470814d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46fdd9b640 1 -- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47080b5250 msgr2=0x7f470814fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46fdd9b640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47080b5250 0x7f470814fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46fdd9b640 1 -- 172.21.15.81:0/759976211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470814de80 con 0x7f47080b9040 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f4706d9d640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47080d2a90 0x7f470814d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f470659c640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47080b5250 0x7f470814fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46fdd9b640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47080b9040 0x7f470814e190 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f46f000c3a0 tx=0x7f46f000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46ff7fe640 1 -- 172.21.15.81:0/759976211 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0011790 con 0x7f47080b9040 2024-06-29T15:38:20.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46ff7fe640 1 -- 172.21.15.81:0/759976211 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46f0017070 con 0x7f47080b9040 2024-06-29T15:38:20.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f470cec1640 1 -- 172.21.15.81:0/759976211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470814eac0 con 0x7f47080b9040 2024-06-29T15:38:20.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f46ff7fe640 1 -- 172.21.15.81:0/759976211 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0011cc0 con 0x7f47080b9040 2024-06-29T15:38:20.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.531+0000 7f470cec1640 1 -- 172.21.15.81:0/759976211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470814f080 con 0x7f47080b9040 2024-06-29T15:38:20.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.532+0000 7f470cec1640 1 -- 172.21.15.81:0/759976211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47080b8440 con 0x7f47080b9040 2024-06-29T15:38:20.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.533+0000 7f46ff7fe640 1 -- 172.21.15.81:0/759976211 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 85) ==== 98332+0+0 (secure 0 0 0) 0x7f46f001c460 con 0x7f47080b9040 2024-06-29T15:38:20.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.533+0000 7f46ff7fe640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f46e8076440 0x7f46e8078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:20.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.533+0000 7f46ff7fe640 1 -- 172.21.15.81:0/759976211 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 6299+0+0 (secure 0 0 0) 0x7f46f0097b50 con 0x7f47080b9040 2024-06-29T15:38:20.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.534+0000 7f470659c640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f46e8076440 0x7f46e8078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:20.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.534+0000 7f470659c640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f46e8076440 0x7f46e8078950 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009ab0 tx=0x7f46f8005f90 comp rx=0 tx=0).ready entity=mgr.7502 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.538+0000 7f46ff7fe640 1 -- 172.21.15.81:0/759976211 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f46f0061040 con 0x7f47080b9040 2024-06-29T15:38:20.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:20.765+0000 7f470cec1640 1 -- 172.21.15.81:0/759976211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f47080688c0 con 0x7f47080b9040 2024-06-29T15:38:21.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.267+0000 7f46ff7fe640 1 -- 172.21.15.81:0/759976211 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v86) ==== 93+0+0 (secure 0 0 0) 0x7f46f0060940 con 0x7f47080b9040 2024-06-29T15:38:21.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.273+0000 7f46ff7fe640 1 -- 172.21.15.81:0/759976211 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f46f0095e20 con 0x7f47080b9040 2024-06-29T15:38:21.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.274+0000 7f470cec1640 1 -- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f46e8076440 msgr2=0x7f46e8078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.274+0000 7f470cec1640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f46e8076440 0x7f46e8078950 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f46f8009ab0 tx=0x7f46f8005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:21.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.274+0000 7f470cec1640 1 -- 172.21.15.81:0/759976211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47080b9040 msgr2=0x7f470814e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.274+0000 7f470cec1640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47080b9040 0x7f470814e190 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f46f000c3a0 tx=0x7f46f000c860 comp rx=0 tx=0).stop 2024-06-29T15:38:21.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.275+0000 7f470cec1640 1 -- 172.21.15.81:0/759976211 shutdown_connections 2024-06-29T15:38:21.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.275+0000 7f470cec1640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f46e8076440 0x7f46e8078950 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.275+0000 7f470cec1640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47080d2a90 0x7f470814d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.275+0000 7f470cec1640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47080b9040 0x7f470814e190 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.275+0000 7f470cec1640 1 --2- 172.21.15.81:0/759976211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47080b5250 0x7f470814fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.275+0000 7f470cec1640 1 -- 172.21.15.81:0/759976211 >> 172.21.15.81:0/759976211 conn(0x7f470809cf30 msgr2=0x7f47081613b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:21.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.275+0000 7f470cec1640 1 -- 172.21.15.81:0/759976211 shutdown_connections 2024-06-29T15:38:21.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.275+0000 7f470cec1640 1 -- 172.21.15.81:0/759976211 wait complete. 2024-06-29T15:38:21.296 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:38:21.296 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:21.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:21.399 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:38:21.399 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:21.399 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:38:21.399 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:21.416 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:38:21.416 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:21.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:21.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.628+0000 7fa523f7a640 1 -- 172.21.15.81:0/872327556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0cccf0 msgr2=0x7fa51c0d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.628+0000 7fa523f7a640 1 --2- 172.21.15.81:0/872327556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0cccf0 0x7fa51c0d0230 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fa51c073550 tx=0x7fa51802a560 comp rx=0 tx=0).stop 2024-06-29T15:38:21.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.628+0000 7fa523f7a640 1 -- 172.21.15.81:0/872327556 shutdown_connections 2024-06-29T15:38:21.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.628+0000 7fa523f7a640 1 --2- 172.21.15.81:0/872327556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0cccf0 0x7fa51c0d0230 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.628+0000 7fa523f7a640 1 --2- 172.21.15.81:0/872327556 >> v2:172.21.15.167:3300/0 conn(0x7fa51c0b3200 0x7fa51c0cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.628+0000 7fa523f7a640 1 -- 172.21.15.81:0/872327556 >> 172.21.15.81:0/872327556 conn(0x7fa51c0b3680 msgr2=0x7fa51c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:21.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.628+0000 7fa523f7a640 1 -- 172.21.15.81:0/872327556 shutdown_connections 2024-06-29T15:38:21.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.628+0000 7fa523f7a640 1 -- 172.21.15.81:0/872327556 wait complete. 2024-06-29T15:38:21.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.631+0000 7fa523f7a640 1 Processor -- start 2024-06-29T15:38:21.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.631+0000 7fa523f7a640 1 -- start start 2024-06-29T15:38:21.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.633+0000 7fa523f7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0b3200 0x7fa51c0b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.633+0000 7fa523f7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa51c0cccf0 0x7fa51c09d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.633+0000 7fa523f7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa51c0d0770 0x7fa51c1f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.633+0000 7fa523f7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c0cedb0 con 0x7fa51c0b3200 2024-06-29T15:38:21.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.633+0000 7fa523f7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c0cec30 con 0x7fa51c0cccf0 2024-06-29T15:38:21.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.633+0000 7fa523f7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa51c0cef30 con 0x7fa51c0d0770 2024-06-29T15:38:21.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.633+0000 7fa521d16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0b3200 0x7fa51c0b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:21.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa521d16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0b3200 0x7fa51c0b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47882/0 (socket says 172.21.15.81:47882) 2024-06-29T15:38:21.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa521d16640 1 -- 172.21.15.81:0/452126893 learned_addr learned my addr 172.21.15.81:0/452126893 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:21.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa521515640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa51c0cccf0 0x7fa51c09d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:21.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa522517640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa51c0d0770 0x7fa51c1f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:21.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa521d16640 1 -- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa51c0d0770 msgr2=0x7fa51c1f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa521d16640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa51c0d0770 0x7fa51c1f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa521d16640 1 -- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa51c0cccf0 msgr2=0x7fa51c09d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa521d16640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa51c0cccf0 0x7fa51c09d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa521d16640 1 -- 172.21.15.81:0/452126893 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51c1e2480 con 0x7fa51c0b3200 2024-06-29T15:38:21.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa521515640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa51c0cccf0 0x7fa51c09d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:38:21.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa521d16640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0b3200 0x7fa51c0b0ac0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa50c00e7c0 tx=0x7fa50c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:21.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa5127fc640 1 -- 172.21.15.81:0/452126893 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c00ccc0 con 0x7fa51c0b3200 2024-06-29T15:38:21.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa5127fc640 1 -- 172.21.15.81:0/452126893 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa50c010430 con 0x7fa51c0b3200 2024-06-29T15:38:21.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa523f7a640 1 -- 172.21.15.81:0/452126893 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51c0afa10 con 0x7fa51c0b3200 2024-06-29T15:38:21.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa5127fc640 1 -- 172.21.15.81:0/452126893 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa50c010860 con 0x7fa51c0b3200 2024-06-29T15:38:21.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.634+0000 7fa523f7a640 1 -- 172.21.15.81:0/452126893 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa51c0affd0 con 0x7fa51c0b3200 2024-06-29T15:38:21.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.635+0000 7fa523f7a640 1 -- 172.21.15.81:0/452126893 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa51c0cc7b0 con 0x7fa51c0b3200 2024-06-29T15:38:21.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.636+0000 7fa5127fc640 1 -- 172.21.15.81:0/452126893 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7fa50c010a00 con 0x7fa51c0b3200 2024-06-29T15:38:21.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.637+0000 7fa5127fc640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa500076420 0x7fa500078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:21.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.637+0000 7fa5127fc640 1 -- 172.21.15.81:0/452126893 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 6299+0+0 (secure 0 0 0) 0x7fa50c098de0 con 0x7fa51c0b3200 2024-06-29T15:38:21.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.637+0000 7fa521515640 1 -- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa500076420 msgr2=0x7fa500078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:21.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.637+0000 7fa521515640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa500076420 0x7fa500078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:21.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.641+0000 7fa5127fc640 1 -- 172.21.15.81:0/452126893 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa50c061cd0 con 0x7fa51c0b3200 2024-06-29T15:38:21.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.825+0000 7fa523f7a640 1 -- 172.21.15.81:0/452126893 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa51c0b5690 con 0x7fa51c0b3200 2024-06-29T15:38:21.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.827+0000 7fa5127fc640 1 -- 172.21.15.81:0/452126893 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) ==== 74+0+17543 (secure 0 0 0) 0x7fa50c065fd0 con 0x7fa51c0b3200 2024-06-29T15:38:21.828 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:21.828 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":154,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:21.262072+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.170598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.829654+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":111,"snap_epoch":154,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.25","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/1730106798":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/2472315005":"2024-06-30T14:42:10.211800+0000","172.21.15.167:6833/3201324170":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/2674867295":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/588266296":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3014230071":"2024-06-30T14:42:10.211800+0000","172.21.15.167:6832/3201324170":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3312315666":"2024-06-30T14:42:10.211800+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":53},{"begin":59,"length":53}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":40,"length":16},{"begin":96,"length":16}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:21.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 -- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa500076420 msgr2=0x7fa500078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:21.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa500076420 0x7fa500078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 -- 172.21.15.81:0/452126893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0b3200 msgr2=0x7fa51c0b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0b3200 0x7fa51c0b0ac0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa50c00e7c0 tx=0x7fa50c00ec80 comp rx=0 tx=0).stop 2024-06-29T15:38:21.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 -- 172.21.15.81:0/452126893 shutdown_connections 2024-06-29T15:38:21.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fa500076420 0x7fa500078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa51c0d0770 0x7fa51c1f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa51c0cccf0 0x7fa51c09d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 --2- 172.21.15.81:0/452126893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0b3200 0x7fa51c0b0ac0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 -- 172.21.15.81:0/452126893 >> 172.21.15.81:0/452126893 conn(0x7fa51c0b3680 msgr2=0x7fa51c1e3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:21.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 -- 172.21.15.81:0/452126893 shutdown_connections 2024-06-29T15:38:21.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.829+0000 7fa523f7a640 1 -- 172.21.15.81:0/452126893 wait complete. 2024-06-29T15:38:21.842 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:41:56.170598+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:41:56.829654+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '154', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 111, 'snap_epoch': 154, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:21.842 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.994+0000 7f1d113f0640 1 -- 172.21.15.81:0/486023365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c0cf860 msgr2=0x7f1d0c165810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.994+0000 7f1d113f0640 1 --2- 172.21.15.81:0/486023365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c0cf860 0x7f1d0c165810 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f1d04005fd0 tx=0x7f1d0402abc0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.994+0000 7f1d113f0640 1 -- 172.21.15.81:0/486023365 shutdown_connections 2024-06-29T15:38:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.994+0000 7f1d113f0640 1 --2- 172.21.15.81:0/486023365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c0cf860 0x7f1d0c165810 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.994+0000 7f1d113f0640 1 -- 172.21.15.81:0/486023365 >> 172.21.15.81:0/486023365 conn(0x7f1d0c0d00b0 msgr2=0x7f1d0c0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.994+0000 7f1d113f0640 1 -- 172.21.15.81:0/486023365 shutdown_connections 2024-06-29T15:38:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.994+0000 7f1d113f0640 1 -- 172.21.15.81:0/486023365 wait complete. 2024-06-29T15:38:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.996+0000 7f1d113f0640 1 Processor -- start 2024-06-29T15:38:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.996+0000 7f1d113f0640 1 -- start start 2024-06-29T15:38:22.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d113f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c0cf860 0x7f1d0c148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d113f0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d0c15ae90 0x7f1d0c0b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d113f0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d0c148e90 0x7f1d0c0b1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d113f0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c1660e0 con 0x7f1d0c0cf860 2024-06-29T15:38:22.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d113f0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c165f60 con 0x7f1d0c15ae90 2024-06-29T15:38:22.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d113f0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1d0c166260 con 0x7f1d0c148e90 2024-06-29T15:38:22.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c0cf860 0x7f1d0c148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c0cf860 0x7f1d0c148670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47900/0 (socket says 172.21.15.81:47900) 2024-06-29T15:38:22.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0affd640 1 -- 172.21.15.81:0/3183150697 learned_addr learned my addr 172.21.15.81:0/3183150697 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:22.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0a7fc640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d0c15ae90 0x7f1d0c0b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0b7fe640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d0c148e90 0x7f1d0c0b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0affd640 1 -- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d0c148e90 msgr2=0x7f1d0c0b1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0affd640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d0c148e90 0x7f1d0c0b1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0affd640 1 -- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d0c15ae90 msgr2=0x7f1d0c0b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0affd640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d0c15ae90 0x7f1d0c0b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0affd640 1 -- 172.21.15.81:0/3183150697 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0c0acfc0 con 0x7f1d0c0cf860 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1d0affd640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c0cf860 0x7f1d0c148670 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00c430 tx=0x7f1cfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1cf37fe640 1 -- 172.21.15.81:0/3183150697 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc017070 con 0x7f1d0c0cf860 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1cf37fe640 1 -- 172.21.15.81:0/3183150697 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1cfc0053c0 con 0x7f1d0c0cf860 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.998+0000 7f1cf37fe640 1 -- 172.21.15.81:0/3183150697 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc003850 con 0x7f1d0c0cf860 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.999+0000 7f1d113f0640 1 -- 172.21.15.81:0/3183150697 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d0c0ad4c0 con 0x7f1d0c0cf860 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:21.999+0000 7f1d113f0640 1 -- 172.21.15.81:0/3183150697 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d0c0ada80 con 0x7f1d0c0cf860 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.000+0000 7f1d113f0640 1 -- 172.21.15.81:0/3183150697 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d0c0c5160 con 0x7f1d0c0cf860 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.001+0000 7f1cf37fe640 1 -- 172.21.15.81:0/3183150697 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f1cfc01a070 con 0x7f1d0c0cf860 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.001+0000 7f1cf37fe640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f1cf80763f0 0x7f1cf8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.001+0000 7f1d0a7fc640 1 -- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f1cf80763f0 msgr2=0x7f1cf8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:22.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.001+0000 7f1cf37fe640 1 -- 172.21.15.81:0/3183150697 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 6299+0+0 (secure 0 0 0) 0x7f1cfc096ec0 con 0x7f1d0c0cf860 2024-06-29T15:38:22.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.001+0000 7f1d0a7fc640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f1cf80763f0 0x7f1cf8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:22.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.006+0000 7f1cf37fe640 1 -- 172.21.15.81:0/3183150697 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1cfc05fe70 con 0x7f1d0c0cf860 2024-06-29T15:38:22.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.202+0000 7f1d0a7fc640 1 -- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f1cf80763f0 msgr2=0x7f1cf8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:22.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.202+0000 7f1d0a7fc640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f1cf80763f0 0x7f1cf8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:22.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.227+0000 7f1d113f0640 1 -- 172.21.15.81:0/3183150697 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1d0c0cd9b0 con 0x7f1d0c0cf860 2024-06-29T15:38:22.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.228+0000 7f1cf37fe640 1 -- 172.21.15.81:0/3183150697 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v154) ==== 93+0+10 (secure 0 0 0) 0x7f1cfc064170 con 0x7f1d0c0cf860 2024-06-29T15:38:22.228 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:38:22.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 -- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f1cf80763f0 msgr2=0x7f1cf8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:22.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f1cf80763f0 0x7f1cf8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 -- 172.21.15.81:0/3183150697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c0cf860 msgr2=0x7f1d0c148670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c0cf860 0x7f1d0c148670 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00c430 tx=0x7f1cfc00c8f0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 -- 172.21.15.81:0/3183150697 shutdown_connections 2024-06-29T15:38:22.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f1cf80763f0 0x7f1cf8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d0c148e90 0x7f1d0c0b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d0c15ae90 0x7f1d0c0b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 --2- 172.21.15.81:0/3183150697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c0cf860 0x7f1d0c148670 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 -- 172.21.15.81:0/3183150697 >> 172.21.15.81:0/3183150697 conn(0x7f1d0c0d00b0 msgr2=0x7f1d0c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:22.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 -- 172.21.15.81:0/3183150697 shutdown_connections 2024-06-29T15:38:22.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.230+0000 7f1d113f0640 1 -- 172.21.15.81:0/3183150697 wait complete. 2024-06-29T15:38:22.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:38:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.364+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/162409902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180b6a20 msgr2=0x7f3b180d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.364+0000 7f3b1d6e4640 1 --2- 172.21.15.81:0/162409902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180b6a20 0x7f3b180d0410 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f3b180d0f10 tx=0x7f3b0002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.364+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/162409902 shutdown_connections 2024-06-29T15:38:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.364+0000 7f3b1d6e4640 1 --2- 172.21.15.81:0/162409902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b180d0950 0x7f3b180cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.364+0000 7f3b1d6e4640 1 --2- 172.21.15.81:0/162409902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180b6a20 0x7f3b180d0410 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.364+0000 7f3b1d6e4640 1 --2- 172.21.15.81:0/162409902 >> v2:172.21.15.167:3300/0 conn(0x7f3b180b8ba0 0x7f3b180d46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.364+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/162409902 >> 172.21.15.81:0/162409902 conn(0x7f3b180d3bd0 msgr2=0x7f3b180d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.364+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/162409902 shutdown_connections 2024-06-29T15:38:22.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.364+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/162409902 wait complete. 2024-06-29T15:38:22.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.366+0000 7f3b1d6e4640 1 Processor -- start 2024-06-29T15:38:22.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.366+0000 7f3b1d6e4640 1 -- start start 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b1d6e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b180b6a20 0x7f3b180ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b1d6e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180b8ba0 0x7f3b18154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b1d6e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b180d0950 0x7f3b18154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b1d6e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3b180b9290 con 0x7f3b180b8ba0 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b1d6e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3b180b7040 con 0x7f3b180b6a20 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b1d6e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3b180b9410 con 0x7f3b180d0950 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b0ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180b8ba0 0x7f3b18154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b0ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180b8ba0 0x7f3b18154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47916/0 (socket says 172.21.15.81:47916) 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b0ffff640 1 -- 172.21.15.81:0/3854178752 learned_addr learned my addr 172.21.15.81:0/3854178752 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b16ffd640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b180b6a20 0x7f3b180ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.367+0000 7f3b177fe640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b180d0950 0x7f3b18154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b0ffff640 1 -- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b180d0950 msgr2=0x7f3b18154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b0ffff640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b180d0950 0x7f3b18154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b0ffff640 1 -- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b180b6a20 msgr2=0x7f3b180ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b0ffff640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b180b6a20 0x7f3b180ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b0ffff640 1 -- 172.21.15.81:0/3854178752 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b181547b0 con 0x7f3b180b8ba0 2024-06-29T15:38:22.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b0ffff640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180b8ba0 0x7f3b18154270 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f3b00005fd0 tx=0x7f3b00003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:22.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b0f7fe640 1 -- 172.21.15.81:0/3854178752 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0003e030 con 0x7f3b180b8ba0 2024-06-29T15:38:22.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b0f7fe640 1 -- 172.21.15.81:0/3854178752 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b00003c20 con 0x7f3b180b8ba0 2024-06-29T15:38:22.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/3854178752 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b18155840 con 0x7f3b180b8ba0 2024-06-29T15:38:22.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b0f7fe640 1 -- 172.21.15.81:0/3854178752 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b00035040 con 0x7f3b180b8ba0 2024-06-29T15:38:22.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.368+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/3854178752 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b18155e00 con 0x7f3b180b8ba0 2024-06-29T15:38:22.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.369+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/3854178752 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b180d46f0 con 0x7f3b180b8ba0 2024-06-29T15:38:22.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.371+0000 7f3b0f7fe640 1 -- 172.21.15.81:0/3854178752 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f3b00041030 con 0x7f3b180b8ba0 2024-06-29T15:38:22.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.371+0000 7f3b0f7fe640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f3af8076420 0x7f3af8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.371+0000 7f3b16ffd640 1 -- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f3af8076420 msgr2=0x7f3af8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:22.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.371+0000 7f3b0f7fe640 1 -- 172.21.15.81:0/3854178752 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6299+0+0 (secure 0 0 0) 0x7f3b000b7f60 con 0x7f3b180b8ba0 2024-06-29T15:38:22.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.371+0000 7f3b16ffd640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f3af8076420 0x7f3af8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:22.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.376+0000 7f3b0f7fe640 1 -- 172.21.15.81:0/3854178752 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3b181547b0 con 0x7f3b180b8ba0 2024-06-29T15:38:22.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.565+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/3854178752 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3b180b9700 con 0x7f3b180b8ba0 2024-06-29T15:38:22.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.565+0000 7f3b0f7fe640 1 -- 172.21.15.81:0/3854178752 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v155) ==== 104+0+11 (secure 0 0 0) 0x7f3b00080e00 con 0x7f3b180b8ba0 2024-06-29T15:38:22.566 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:38:22.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f3af8076420 msgr2=0x7f3af8078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:22.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f3af8076420 0x7f3af8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/3854178752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180b8ba0 msgr2=0x7f3b18154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180b8ba0 0x7f3b18154270 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f3b00005fd0 tx=0x7f3b00003820 comp rx=0 tx=0).stop 2024-06-29T15:38:22.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/3854178752 shutdown_connections 2024-06-29T15:38:22.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f3af8076420 0x7f3af8078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3b180d0950 0x7f3b18154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180b8ba0 0x7f3b18154270 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 --2- 172.21.15.81:0/3854178752 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3b180b6a20 0x7f3b180ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/3854178752 >> 172.21.15.81:0/3854178752 conn(0x7f3b180d3bd0 msgr2=0x7f3b18161a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:22.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/3854178752 shutdown_connections 2024-06-29T15:38:22.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.567+0000 7f3b1d6e4640 1 -- 172.21.15.81:0/3854178752 wait complete. 2024-06-29T15:38:22.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:38:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.699+0000 7fc2fff7a640 1 -- 172.21.15.81:0/2545558318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2f815b5b0 msgr2=0x7fc2f80b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.699+0000 7fc2fff7a640 1 --2- 172.21.15.81:0/2545558318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2f815b5b0 0x7fc2f80b4430 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4009830 tx=0x7fc2f402a410 comp rx=0 tx=0).stop 2024-06-29T15:38:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.699+0000 7fc2fff7a640 1 -- 172.21.15.81:0/2545558318 shutdown_connections 2024-06-29T15:38:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.699+0000 7fc2fff7a640 1 --2- 172.21.15.81:0/2545558318 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2f806f660 0x7fc2f806fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.699+0000 7fc2fff7a640 1 --2- 172.21.15.81:0/2545558318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2f815b5b0 0x7fc2f80b4430 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.699+0000 7fc2fff7a640 1 -- 172.21.15.81:0/2545558318 >> 172.21.15.81:0/2545558318 conn(0x7fc2f806ccc0 msgr2=0x7fc2f80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.699+0000 7fc2fff7a640 1 -- 172.21.15.81:0/2545558318 shutdown_connections 2024-06-29T15:38:22.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.699+0000 7fc2fff7a640 1 -- 172.21.15.81:0/2545558318 wait complete. 2024-06-29T15:38:22.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.701+0000 7fc2fff7a640 1 Processor -- start 2024-06-29T15:38:22.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.701+0000 7fc2fff7a640 1 -- start start 2024-06-29T15:38:22.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.702+0000 7fc2fff7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2f806f660 0x7fc2f8159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.702+0000 7fc2fff7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2f80b35b0 0x7fc2f8153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.702+0000 7fc2fff7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2f815b5b0 0x7fc2f814e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.702+0000 7fc2fff7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f806f160 con 0x7fc2f806f660 2024-06-29T15:38:22.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.702+0000 7fc2fff7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f806efe0 con 0x7fc2f80b35b0 2024-06-29T15:38:22.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.702+0000 7fc2fff7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f806f2e0 con 0x7fc2f815b5b0 2024-06-29T15:38:22.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.702+0000 7fc2fdd16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2f806f660 0x7fc2f8159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.702+0000 7fc2fdd16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2f806f660 0x7fc2f8159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47932/0 (socket says 172.21.15.81:47932) 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.702+0000 7fc2fdd16640 1 -- 172.21.15.81:0/3280122809 learned_addr learned my addr 172.21.15.81:0/3280122809 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2fd515640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2f80b35b0 0x7fc2f8153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2fe517640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2f815b5b0 0x7fc2f814e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2fdd16640 1 -- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2f815b5b0 msgr2=0x7fc2f814e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2fdd16640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2f815b5b0 0x7fc2f814e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2fdd16640 1 -- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2f80b35b0 msgr2=0x7fc2f8153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2fdd16640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2f80b35b0 0x7fc2f8153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2fdd16640 1 -- 172.21.15.81:0/3280122809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f814be00 con 0x7fc2f806f660 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2fdd16640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2f806f660 0x7fc2f8159b20 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c500 tx=0x7fc2e800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2ee7fc640 1 -- 172.21.15.81:0/3280122809 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8011790 con 0x7fc2f806f660 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2fff7a640 1 -- 172.21.15.81:0/3280122809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f814c070 con 0x7fc2f806f660 2024-06-29T15:38:22.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2ee7fc640 1 -- 172.21.15.81:0/3280122809 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2e8017070 con 0x7fc2f806f660 2024-06-29T15:38:22.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2ee7fc640 1 -- 172.21.15.81:0/3280122809 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8011cc0 con 0x7fc2f806f660 2024-06-29T15:38:22.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.703+0000 7fc2fff7a640 1 -- 172.21.15.81:0/3280122809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f814c630 con 0x7fc2f806f660 2024-06-29T15:38:22.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.704+0000 7fc2fff7a640 1 -- 172.21.15.81:0/3280122809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f806fa70 con 0x7fc2f806f660 2024-06-29T15:38:22.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.705+0000 7fc2ee7fc640 1 -- 172.21.15.81:0/3280122809 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7fc2e801c460 con 0x7fc2f806f660 2024-06-29T15:38:22.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.706+0000 7fc2ee7fc640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fc2dc0763f0 0x7fc2dc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:22.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.706+0000 7fc2ee7fc640 1 -- 172.21.15.81:0/3280122809 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6299+0+0 (secure 0 0 0) 0x7fc2e8097a70 con 0x7fc2f806f660 2024-06-29T15:38:22.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.706+0000 7fc2fd515640 1 -- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fc2dc0763f0 msgr2=0x7fc2dc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:22.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.706+0000 7fc2fd515640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fc2dc0763f0 0x7fc2dc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:22.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.710+0000 7fc2ee7fc640 1 -- 172.21.15.81:0/3280122809 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc2f806ffb0 con 0x7fc2f806f660 2024-06-29T15:38:22.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.900+0000 7fc2fff7a640 1 -- 172.21.15.81:0/3280122809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc2f80b1c20 con 0x7fc2f806f660 2024-06-29T15:38:22.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.900+0000 7fc2ee7fc640 1 -- 172.21.15.81:0/3280122809 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v155) ==== 100+0+11 (secure 0 0 0) 0x7fc2e8060f60 con 0x7fc2f806f660 2024-06-29T15:38:22.901 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:38:22.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 -- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fc2dc0763f0 msgr2=0x7fc2dc078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:22.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fc2dc0763f0 0x7fc2dc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 -- 172.21.15.81:0/3280122809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2f806f660 msgr2=0x7fc2f8159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:22.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2f806f660 0x7fc2f8159b20 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c500 tx=0x7fc2e800c900 comp rx=0 tx=0).stop 2024-06-29T15:38:22.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 -- 172.21.15.81:0/3280122809 shutdown_connections 2024-06-29T15:38:22.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fc2dc0763f0 0x7fc2dc078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc2f815b5b0 0x7fc2f814e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc2f80b35b0 0x7fc2f8153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 --2- 172.21.15.81:0/3280122809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2f806f660 0x7fc2f8159b20 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:22.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 -- 172.21.15.81:0/3280122809 >> 172.21.15.81:0/3280122809 conn(0x7fc2f806ccc0 msgr2=0x7fc2f814b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:22.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.902+0000 7fc2fff7a640 1 -- 172.21.15.81:0/3280122809 shutdown_connections 2024-06-29T15:38:22.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:22.903+0000 7fc2fff7a640 1 -- 172.21.15.81:0/3280122809 wait complete. 2024-06-29T15:38:22.916 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:38:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.032+0000 7f4f8f012640 1 -- 172.21.15.81:0/2471132059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f880cd150 msgr2=0x7f4f88072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.032+0000 7f4f8f012640 1 --2- 172.21.15.81:0/2471132059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f880cd150 0x7f4f88072f40 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f4f88074310 tx=0x7f4f7c02a560 comp rx=0 tx=0).stop 2024-06-29T15:38:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.032+0000 7f4f8f012640 1 -- 172.21.15.81:0/2471132059 shutdown_connections 2024-06-29T15:38:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.032+0000 7f4f8f012640 1 --2- 172.21.15.81:0/2471132059 >> v2:172.21.15.167:3300/0 conn(0x7f4f88073480 0x7f4f8815b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.032+0000 7f4f8f012640 1 --2- 172.21.15.81:0/2471132059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f880cd150 0x7f4f88072f40 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.032+0000 7f4f8f012640 1 -- 172.21.15.81:0/2471132059 >> 172.21.15.81:0/2471132059 conn(0x7f4f880b2920 msgr2=0x7f4f880b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.032+0000 7f4f8f012640 1 -- 172.21.15.81:0/2471132059 shutdown_connections 2024-06-29T15:38:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.032+0000 7f4f8f012640 1 -- 172.21.15.81:0/2471132059 wait complete. 2024-06-29T15:38:23.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.034+0000 7f4f8f012640 1 Processor -- start 2024-06-29T15:38:23.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.034+0000 7f4f8f012640 1 -- start start 2024-06-29T15:38:23.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8f012640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f88073480 0x7f4f88142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8f012640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f880cd150 0x7f4f8814f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8f012640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f8815bc20 0x7f4f88157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8f012640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8815c1b0 con 0x7f4f88073480 2024-06-29T15:38:23.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8f012640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8815c030 con 0x7f4f880cd150 2024-06-29T15:38:23.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8f012640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8815f060 con 0x7f4f8815bc20 2024-06-29T15:38:23.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8cdae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f88073480 0x7f4f88142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8cdae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f88073480 0x7f4f88142600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47950/0 (socket says 172.21.15.81:47950) 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8cdae640 1 -- 172.21.15.81:0/249022673 learned_addr learned my addr 172.21.15.81:0/249022673 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f87fff640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f880cd150 0x7f4f8814f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8d5af640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f8815bc20 0x7f4f88157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8cdae640 1 -- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f8815bc20 msgr2=0x7f4f88157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.035+0000 7f4f8cdae640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f8815bc20 0x7f4f88157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.036+0000 7f4f8cdae640 1 -- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f880cd150 msgr2=0x7f4f8814f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.036+0000 7f4f8cdae640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f880cd150 0x7f4f8814f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.036+0000 7f4f8cdae640 1 -- 172.21.15.81:0/249022673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f880cf2e0 con 0x7f4f88073480 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.036+0000 7f4f8cdae640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f88073480 0x7f4f88142600 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c034040 tx=0x7f4f7c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.036+0000 7f4f857fa640 1 -- 172.21.15.81:0/249022673 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c03e030 con 0x7f4f88073480 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.036+0000 7f4f857fa640 1 -- 172.21.15.81:0/249022673 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f7c0047a0 con 0x7f4f88073480 2024-06-29T15:38:23.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.036+0000 7f4f8f012640 1 -- 172.21.15.81:0/249022673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f880cf5c0 con 0x7f4f88073480 2024-06-29T15:38:23.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.036+0000 7f4f857fa640 1 -- 172.21.15.81:0/249022673 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c035040 con 0x7f4f88073480 2024-06-29T15:38:23.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.036+0000 7f4f8f012640 1 -- 172.21.15.81:0/249022673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f880cfb80 con 0x7f4f88073480 2024-06-29T15:38:23.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.037+0000 7f4f8f012640 1 -- 172.21.15.81:0/249022673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8815f4c0 con 0x7f4f88073480 2024-06-29T15:38:23.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.038+0000 7f4f857fa640 1 -- 172.21.15.81:0/249022673 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f4f7c041030 con 0x7f4f88073480 2024-06-29T15:38:23.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.038+0000 7f4f857fa640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f4f6c076420 0x7f4f6c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.038+0000 7f4f87fff640 1 -- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f4f6c076420 msgr2=0x7f4f6c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:23.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.038+0000 7f4f857fa640 1 -- 172.21.15.81:0/249022673 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6299+0+0 (secure 0 0 0) 0x7f4f7c0b8200 con 0x7f4f88073480 2024-06-29T15:38:23.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.038+0000 7f4f87fff640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f4f6c076420 0x7f4f6c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:23.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.043+0000 7f4f857fa640 1 -- 172.21.15.81:0/249022673 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f7c081130 con 0x7f4f88073480 2024-06-29T15:38:23.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.239+0000 7f4f87fff640 1 -- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f4f6c076420 msgr2=0x7f4f6c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:23.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.239+0000 7f4f87fff640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f4f6c076420 0x7f4f6c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:23.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.242+0000 7f4f8f012640 1 -- 172.21.15.81:0/249022673 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4f88155210 con 0x7f4f88073480 2024-06-29T15:38:23.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.243+0000 7f4f857fa640 1 -- 172.21.15.81:0/249022673 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v155) ==== 80+0+477 (secure 0 0 0) 0x7f4f7c085430 con 0x7f4f88073480 2024-06-29T15:38:23.243 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1730106798 2024-06-30T14:42:10.211800+0000 2024-06-29T15:38:23.244 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2472315005 2024-06-30T14:42:10.211800+0000 2024-06-29T15:38:23.244 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/3201324170 2024-06-30T14:42:10.211800+0000 2024-06-29T15:38:23.244 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2674867295 2024-06-30T14:42:10.211800+0000 2024-06-29T15:38:23.244 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/588266296 2024-06-30T14:42:10.211800+0000 2024-06-29T15:38:23.244 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3014230071 2024-06-30T14:42:10.211800+0000 2024-06-29T15:38:23.244 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/3201324170 2024-06-30T14:42:10.211800+0000 2024-06-29T15:38:23.244 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3312315666 2024-06-30T14:42:10.211800+0000 2024-06-29T15:38:23.244 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T15:38:23.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 -- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f4f6c076420 msgr2=0x7f4f6c078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:23.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f4f6c076420 0x7f4f6c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 -- 172.21.15.81:0/249022673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f88073480 msgr2=0x7f4f88142600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f88073480 0x7f4f88142600 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c034040 tx=0x7f4f7c0039e0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 -- 172.21.15.81:0/249022673 shutdown_connections 2024-06-29T15:38:23.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f4f6c076420 0x7f4f6c078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f8815bc20 0x7f4f88157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f880cd150 0x7f4f8814f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 --2- 172.21.15.81:0/249022673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f88073480 0x7f4f88142600 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 -- 172.21.15.81:0/249022673 >> 172.21.15.81:0/249022673 conn(0x7f4f880b2920 msgr2=0x7f4f880b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:23.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 -- 172.21.15.81:0/249022673 shutdown_connections 2024-06-29T15:38:23.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.245+0000 7f4f8f012640 1 -- 172.21.15.81:0/249022673 wait complete. 2024-06-29T15:38:23.257 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:38:23.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:23.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:23.289 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T15:38:23.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:23.289 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:38:23.346 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:23.346 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:38:23.346 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:23.363 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:23.363 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:23.363 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:38:23.363 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:23.402 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:38:23.402 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:23.402 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:38:23.402 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:23.419 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:23.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:23.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.558+0000 7f08930b3640 1 -- 172.21.15.81:0/2944541569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f087c003c30 msgr2=0x7f087c004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.558+0000 7f08930b3640 1 --2- 172.21.15.81:0/2944541569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f087c003c30 0x7f087c004080 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f0880009d00 tx=0x7f087c005830 comp rx=0 tx=0).stop 2024-06-29T15:38:23.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.558+0000 7f08930b3640 1 -- 172.21.15.81:0/2944541569 shutdown_connections 2024-06-29T15:38:23.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.558+0000 7f08930b3640 1 --2- 172.21.15.81:0/2944541569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f087c003c30 0x7f087c004080 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.558+0000 7f08930b3640 1 -- 172.21.15.81:0/2944541569 >> 172.21.15.81:0/2944541569 conn(0x7f088c0b3680 msgr2=0x7f088c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:23.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.558+0000 7f08930b3640 1 -- 172.21.15.81:0/2944541569 shutdown_connections 2024-06-29T15:38:23.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.558+0000 7f08930b3640 1 -- 172.21.15.81:0/2944541569 wait complete. 2024-06-29T15:38:23.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.560+0000 7f08930b3640 1 Processor -- start 2024-06-29T15:38:23.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.560+0000 7f08930b3640 1 -- start start 2024-06-29T15:38:23.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f08930b3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f087c003c30 0x7f088c06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f08930b3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f088c0c8450 0x7f088c0c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f08930b3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f088c0c61c0 0x7f088c1f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f08930b3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f088c0d03b0 con 0x7f088c0c8450 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f08930b3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f088c0d0230 con 0x7f088c0c61c0 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f08930b3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f088c0d0530 con 0x7f087c003c30 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f088bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f088c0c8450 0x7f088c0c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f088bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f088c0c8450 0x7f088c0c5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47968/0 (socket says 172.21.15.81:47968) 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f088bfff640 1 -- 172.21.15.81:0/3113213527 learned_addr learned my addr 172.21.15.81:0/3113213527 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f0890e4f640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f087c003c30 0x7f088c06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f0891650640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f088c0c61c0 0x7f088c1f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f088bfff640 1 -- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f087c003c30 msgr2=0x7f088c06ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f088bfff640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f087c003c30 0x7f088c06ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f088bfff640 1 -- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f088c0c61c0 msgr2=0x7f088c1f7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f088bfff640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f088c0c61c0 0x7f088c1f7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f088bfff640 1 -- 172.21.15.81:0/3113213527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f088c06d9e0 con 0x7f088c0c8450 2024-06-29T15:38:23.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f088bfff640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f088c0c8450 0x7f088c0c5c80 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f08780063d0 tx=0x7f08780039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:23.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.562+0000 7f08897fa640 1 -- 172.21.15.81:0/3113213527 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878015070 con 0x7f088c0c8450 2024-06-29T15:38:23.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.563+0000 7f08897fa640 1 -- 172.21.15.81:0/3113213527 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0878011840 con 0x7f088c0c8450 2024-06-29T15:38:23.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.563+0000 7f08897fa640 1 -- 172.21.15.81:0/3113213527 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878011c90 con 0x7f088c0c8450 2024-06-29T15:38:23.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.563+0000 7f08930b3640 1 -- 172.21.15.81:0/3113213527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f088c06dcc0 con 0x7f088c0c8450 2024-06-29T15:38:23.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.563+0000 7f08930b3640 1 -- 172.21.15.81:0/3113213527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f088c06e280 con 0x7f088c0c8450 2024-06-29T15:38:23.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.564+0000 7f08930b3640 1 -- 172.21.15.81:0/3113213527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f088c06d750 con 0x7f088c0c8450 2024-06-29T15:38:23.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.564+0000 7f08897fa640 1 -- 172.21.15.81:0/3113213527 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f0878018070 con 0x7f088c0c8450 2024-06-29T15:38:23.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.565+0000 7f08897fa640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f08700763f0 0x7f0870078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.565+0000 7f0890e4f640 1 -- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f08700763f0 msgr2=0x7f0870078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:23.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.565+0000 7f08897fa640 1 -- 172.21.15.81:0/3113213527 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6299+0+0 (secure 0 0 0) 0x7f0878098780 con 0x7f088c0c8450 2024-06-29T15:38:23.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.565+0000 7f0890e4f640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f08700763f0 0x7f0870078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:23.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.569+0000 7f08897fa640 1 -- 172.21.15.81:0/3113213527 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0878061590 con 0x7f088c0c8450 2024-06-29T15:38:23.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.766+0000 7f0890e4f640 1 -- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f08700763f0 msgr2=0x7f0870078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:23.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.766+0000 7f0890e4f640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f08700763f0 0x7f0870078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:23.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.794+0000 7f08930b3640 1 -- 172.21.15.81:0/3113213527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f088c0d0a90 con 0x7f088c0c8450 2024-06-29T15:38:23.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.794+0000 7f08897fa640 1 -- 172.21.15.81:0/3113213527 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5231 (secure 0 0 0) 0x7f0878065890 con 0x7f088c0c8450 2024-06-29T15:38:23.795 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:23.795 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":77,"btime":"2024-06-29T14:42:01:549259+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/4068340499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":4068340499},{"type":"v1","addr":"172.21.15.81:6834","nonce":4068340499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1903134398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1903134398},{"type":"v1","addr":"172.21.15.167:6837","nonce":1903134398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1126258483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1126258483},{"type":"v1","addr":"172.21.15.81:6835","nonce":1126258483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.875457+0000","modified":"2024-06-29T14:42:00.919213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9111":{"gid":9111,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/4254103476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":4254103476},{"type":"v1","addr":"172.21.15.167:6836","nonce":4254103476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9111,"qdb_cluster":[9111]},"id":9}]} 2024-06-29T15:38:23.795 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 77 2024-06-29T15:38:23.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 -- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f08700763f0 msgr2=0x7f0870078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:23.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f08700763f0 0x7f0870078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 -- 172.21.15.81:0/3113213527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f088c0c8450 msgr2=0x7f088c0c5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f088c0c8450 0x7f088c0c5c80 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f08780063d0 tx=0x7f08780039c0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 -- 172.21.15.81:0/3113213527 shutdown_connections 2024-06-29T15:38:23.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f08700763f0 0x7f0870078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f088c0c61c0 0x7f088c1f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f088c0c8450 0x7f088c0c5c80 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 --2- 172.21.15.81:0/3113213527 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f087c003c30 0x7f088c06ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 -- 172.21.15.81:0/3113213527 >> 172.21.15.81:0/3113213527 conn(0x7f088c0b3680 msgr2=0x7f088c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:23.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 -- 172.21.15.81:0/3113213527 shutdown_connections 2024-06-29T15:38:23.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.796+0000 7f08930b3640 1 -- 172.21.15.81:0/3113213527 wait complete. 2024-06-29T15:38:23.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:23.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.927+0000 7f74bd410640 1 -- 172.21.15.81:0/3399491261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74b808c270 msgr2=0x7f74b80ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.927+0000 7f74bd410640 1 --2- 172.21.15.81:0/3399491261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74b808c270 0x7f74b80ce930 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009830 tx=0x7f74a002a3f0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.927+0000 7f74bd410640 1 -- 172.21.15.81:0/3399491261 shutdown_connections 2024-06-29T15:38:23.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.927+0000 7f74bd410640 1 --2- 172.21.15.81:0/3399491261 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74b808ccd0 0x7f74b80cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.927+0000 7f74bd410640 1 --2- 172.21.15.81:0/3399491261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74b808c270 0x7f74b80ce930 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.927+0000 7f74bd410640 1 -- 172.21.15.81:0/3399491261 >> 172.21.15.81:0/3399491261 conn(0x7f74b80b56d0 msgr2=0x7f74b80b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:23.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.927+0000 7f74bd410640 1 -- 172.21.15.81:0/3399491261 shutdown_connections 2024-06-29T15:38:23.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.927+0000 7f74bd410640 1 -- 172.21.15.81:0/3399491261 wait complete. 2024-06-29T15:38:23.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.929+0000 7f74bd410640 1 Processor -- start 2024-06-29T15:38:23.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.929+0000 7f74bd410640 1 -- start start 2024-06-29T15:38:23.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.930+0000 7f74bd410640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74b808ccd0 0x7f74b8165f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.930+0000 7f74bd410640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74b80d17b0 0x7f74b815f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.930+0000 7f74bd410640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74b8161f10 0x7f74b80a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.930+0000 7f74bd410640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74b80d2450 con 0x7f74b808ccd0 2024-06-29T15:38:23.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74bd410640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f74b80d22d0 con 0x7f74b8161f10 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74bd410640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f74b80d25d0 con 0x7f74b80d17b0 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74b808ccd0 0x7f74b8165f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74b808ccd0 0x7f74b8165f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47976/0 (socket says 172.21.15.81:47976) 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b6ffd640 1 -- 172.21.15.81:0/3172734528 learned_addr learned my addr 172.21.15.81:0/3172734528 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b67fc640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74b80d17b0 0x7f74b815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b77fe640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74b8161f10 0x7f74b80a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b6ffd640 1 -- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74b80d17b0 msgr2=0x7f74b815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b6ffd640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74b80d17b0 0x7f74b815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b6ffd640 1 -- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74b8161f10 msgr2=0x7f74b80a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b6ffd640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74b8161f10 0x7f74b80a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b6ffd640 1 -- 172.21.15.81:0/3172734528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b81664d0 con 0x7f74b808ccd0 2024-06-29T15:38:23.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74b6ffd640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74b808ccd0 0x7f74b8165f90 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f74a0000c00 tx=0x7f74a0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:23.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.931+0000 7f74af7fe640 1 -- 172.21.15.81:0/3172734528 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0031d60 con 0x7f74b808ccd0 2024-06-29T15:38:23.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.932+0000 7f74af7fe640 1 -- 172.21.15.81:0/3172734528 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74a0033e10 con 0x7f74b808ccd0 2024-06-29T15:38:23.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.932+0000 7f74af7fe640 1 -- 172.21.15.81:0/3172734528 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0033310 con 0x7f74b808ccd0 2024-06-29T15:38:23.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.932+0000 7f74bd410640 1 -- 172.21.15.81:0/3172734528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b80a0650 con 0x7f74b808ccd0 2024-06-29T15:38:23.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.932+0000 7f74bd410640 1 -- 172.21.15.81:0/3172734528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b80a0c10 con 0x7f74b808ccd0 2024-06-29T15:38:23.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.933+0000 7f74bd410640 1 -- 172.21.15.81:0/3172734528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74b80a03d0 con 0x7f74b808ccd0 2024-06-29T15:38:23.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.934+0000 7f74af7fe640 1 -- 172.21.15.81:0/3172734528 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f74a00334b0 con 0x7f74b808ccd0 2024-06-29T15:38:23.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.935+0000 7f74af7fe640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f7498076420 0x7f7498078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:23.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.935+0000 7f74b67fc640 1 -- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f7498076420 msgr2=0x7f7498078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:23.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.935+0000 7f74af7fe640 1 -- 172.21.15.81:0/3172734528 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6299+0+0 (secure 0 0 0) 0x7f74a00b8f30 con 0x7f74b808ccd0 2024-06-29T15:38:23.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.935+0000 7f74b67fc640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f7498076420 0x7f7498078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:23.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:23.939+0000 7f74af7fe640 1 -- 172.21.15.81:0/3172734528 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f74b80a03d0 con 0x7f74b808ccd0 2024-06-29T15:38:24.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.127+0000 7f74bd410640 1 -- 172.21.15.81:0/3172734528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74b808c270 con 0x7f74b808ccd0 2024-06-29T15:38:24.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.128+0000 7f74af7fe640 1 -- 172.21.15.81:0/3172734528 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+17542 (secure 0 0 0) 0x7f74a003f030 con 0x7f74b808ccd0 2024-06-29T15:38:24.129 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:24.129 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":155,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:22.267057+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.170598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.829654+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":155,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.25","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/1730106798":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/2472315005":"2024-06-30T14:42:10.211800+0000","172.21.15.167:6833/3201324170":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/2674867295":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/588266296":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3014230071":"2024-06-30T14:42:10.211800+0000","172.21.15.167:6832/3201324170":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3312315666":"2024-06-30T14:42:10.211800+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":56,"length":1},{"begin":112,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:24.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 -- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f7498076420 msgr2=0x7f7498078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:24.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f7498076420 0x7f7498078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 -- 172.21.15.81:0/3172734528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74b808ccd0 msgr2=0x7f74b8165f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74b808ccd0 0x7f74b8165f90 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f74a0000c00 tx=0x7f74a0033c20 comp rx=0 tx=0).stop 2024-06-29T15:38:24.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 -- 172.21.15.81:0/3172734528 shutdown_connections 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f7498076420 0x7f7498078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74b8161f10 0x7f74b80a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74b80d17b0 0x7f74b815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 --2- 172.21.15.81:0/3172734528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74b808ccd0 0x7f74b8165f90 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 -- 172.21.15.81:0/3172734528 >> 172.21.15.81:0/3172734528 conn(0x7f74b80b56d0 msgr2=0x7f74b8068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 -- 172.21.15.81:0/3172734528 shutdown_connections 2024-06-29T15:38:24.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.131+0000 7f74bd410640 1 -- 172.21.15.81:0/3172734528 wait complete. 2024-06-29T15:38:24.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T14:41:56.170598+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T14:41:56.829654+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 155, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:24.143 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:24.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.261+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1761386442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8c073650 msgr2=0x7f9a8c15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.261+0000 7f9a92f9e640 1 --2- 172.21.15.81:0/1761386442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8c073650 0x7f9a8c15b6e0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c005fd0 tx=0x7f9a7c02a540 comp rx=0 tx=0).stop 2024-06-29T15:38:24.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.261+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1761386442 shutdown_connections 2024-06-29T15:38:24.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.261+0000 7f9a92f9e640 1 --2- 172.21.15.81:0/1761386442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a8c15bc20 0x7f9a8c06ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.261+0000 7f9a92f9e640 1 --2- 172.21.15.81:0/1761386442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8c073650 0x7f9a8c15b6e0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.261+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1761386442 >> 172.21.15.81:0/1761386442 conn(0x7f9a8c0b2920 msgr2=0x7f9a8c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.262+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1761386442 shutdown_connections 2024-06-29T15:38:24.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.262+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1761386442 wait complete. 2024-06-29T15:38:24.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.263+0000 7f9a92f9e640 1 Processor -- start 2024-06-29T15:38:24.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.263+0000 7f9a92f9e640 1 -- start start 2024-06-29T15:38:24.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a92f9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8c0cd150 0x7f9a8c15ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a92f9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a8c15bc20 0x7f9a8c14a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a92f9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a8c06c2f0 0x7f9a8c06c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a92f9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c06f660 con 0x7f9a8c0cd150 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a92f9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c06f4e0 con 0x7f9a8c06c2f0 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a92f9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8c06f7e0 con 0x7f9a8c15bc20 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a90d3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8c0cd150 0x7f9a8c15ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a90d3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8c0cd150 0x7f9a8c15ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48000/0 (socket says 172.21.15.81:48000) 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a90d3a640 1 -- 172.21.15.81:0/1038826479 learned_addr learned my addr 172.21.15.81:0/1038826479 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a83fff640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a8c15bc20 0x7f9a8c14a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a9153b640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a8c06c2f0 0x7f9a8c06c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a90d3a640 1 -- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a8c15bc20 msgr2=0x7f9a8c14a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a90d3a640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a8c15bc20 0x7f9a8c14a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a90d3a640 1 -- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a8c06c2f0 msgr2=0x7f9a8c06c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a90d3a640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a8c06c2f0 0x7f9a8c06c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a90d3a640 1 -- 172.21.15.81:0/1038826479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c06cc40 con 0x7f9a8c0cd150 2024-06-29T15:38:24.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.265+0000 7f9a90d3a640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8c0cd150 0x7f9a8c15ad00 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800c3a0 tx=0x7f9a8800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:24.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.266+0000 7f9a817fa640 1 -- 172.21.15.81:0/1038826479 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88017070 con 0x7f9a8c0cd150 2024-06-29T15:38:24.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.266+0000 7f9a817fa640 1 -- 172.21.15.81:0/1038826479 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a880053c0 con 0x7f9a8c0cd150 2024-06-29T15:38:24.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.266+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1038826479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8c14f540 con 0x7f9a8c0cd150 2024-06-29T15:38:24.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.266+0000 7f9a817fa640 1 -- 172.21.15.81:0/1038826479 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88003850 con 0x7f9a8c0cd150 2024-06-29T15:38:24.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.266+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1038826479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8c151200 con 0x7f9a8c0cd150 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.267+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1038826479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a8c14fa60 con 0x7f9a8c0cd150 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.268+0000 7f9a817fa640 1 -- 172.21.15.81:0/1038826479 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f9a8801a070 con 0x7f9a8c0cd150 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.268+0000 7f9a817fa640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9a780763f0 0x7f9a78078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.268+0000 7f9a83fff640 1 -- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9a780763f0 msgr2=0x7f9a78078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.268+0000 7f9a817fa640 1 -- 172.21.15.81:0/1038826479 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6299+0+0 (secure 0 0 0) 0x7f9a88096f20 con 0x7f9a8c0cd150 2024-06-29T15:38:24.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.268+0000 7f9a83fff640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9a780763f0 0x7f9a78078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:24.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.277+0000 7f9a817fa640 1 -- 172.21.15.81:0/1038826479 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9a8c06cc40 con 0x7f9a8c0cd150 2024-06-29T15:38:24.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.463+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1038826479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9a8c0b2f80 con 0x7f9a8c0cd150 2024-06-29T15:38:24.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.463+0000 7f9a817fa640 1 -- 172.21.15.81:0/1038826479 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v155) ==== 93+0+10 (secure 0 0 0) 0x7f9a8805fe50 con 0x7f9a8c0cd150 2024-06-29T15:38:24.466 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:38:24.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.465+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9a780763f0 msgr2=0x7f9a78078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:24.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.465+0000 7f9a92f9e640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9a780763f0 0x7f9a78078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.465+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1038826479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8c0cd150 msgr2=0x7f9a8c15ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.465+0000 7f9a92f9e640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8c0cd150 0x7f9a8c15ad00 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800c3a0 tx=0x7f9a8800c860 comp rx=0 tx=0).stop 2024-06-29T15:38:24.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.466+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1038826479 shutdown_connections 2024-06-29T15:38:24.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.466+0000 7f9a92f9e640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f9a780763f0 0x7f9a78078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.466+0000 7f9a92f9e640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a8c06c2f0 0x7f9a8c06c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.466+0000 7f9a92f9e640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a8c15bc20 0x7f9a8c14a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.466+0000 7f9a92f9e640 1 --2- 172.21.15.81:0/1038826479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8c0cd150 0x7f9a8c15ad00 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.466+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1038826479 >> 172.21.15.81:0/1038826479 conn(0x7f9a8c0b2920 msgr2=0x7f9a8c0d1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.466+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1038826479 shutdown_connections 2024-06-29T15:38:24.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.466+0000 7f9a92f9e640 1 -- 172.21.15.81:0/1038826479 wait complete. 2024-06-29T15:38:24.478 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:38:24.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.596+0000 7fea17368640 1 -- 172.21.15.81:0/2782127857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea100aed70 msgr2=0x7fea100b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.596+0000 7fea17368640 1 --2- 172.21.15.81:0/2782127857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea100aed70 0x7fea100b4980 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fea08005fd0 tx=0x7fea0802a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.596+0000 7fea17368640 1 -- 172.21.15.81:0/2782127857 shutdown_connections 2024-06-29T15:38:24.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.596+0000 7fea17368640 1 --2- 172.21.15.81:0/2782127857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea100aed70 0x7fea100b4980 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.596+0000 7fea17368640 1 --2- 172.21.15.81:0/2782127857 >> v2:172.21.15.167:3300/0 conn(0x7fea1006e840 0x7fea100b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.596+0000 7fea17368640 1 -- 172.21.15.81:0/2782127857 >> 172.21.15.81:0/2782127857 conn(0x7fea100b3680 msgr2=0x7fea100b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.596+0000 7fea17368640 1 -- 172.21.15.81:0/2782127857 shutdown_connections 2024-06-29T15:38:24.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.596+0000 7fea17368640 1 -- 172.21.15.81:0/2782127857 wait complete. 2024-06-29T15:38:24.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.598+0000 7fea17368640 1 Processor -- start 2024-06-29T15:38:24.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.598+0000 7fea17368640 1 -- start start 2024-06-29T15:38:24.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea17368640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fea1006e840 0x7fea1015f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea17368640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea100ae960 0x7fea10149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea17368640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea100aed70 0x7fea100ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea17368640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fea100b5250 con 0x7fea100aed70 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea17368640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fea100b50d0 con 0x7fea1006e840 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea17368640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fea100b53d0 con 0x7fea100ae960 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea15905640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea100aed70 0x7fea100ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea15905640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea100aed70 0x7fea100ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48016/0 (socket says 172.21.15.81:48016) 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea15905640 1 -- 172.21.15.81:0/3097063332 learned_addr learned my addr 172.21.15.81:0/3097063332 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea15104640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fea1006e840 0x7fea1015f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea14903640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea100ae960 0x7fea10149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea15905640 1 -- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea100ae960 msgr2=0x7fea10149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea15905640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea100ae960 0x7fea10149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea15905640 1 -- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fea1006e840 msgr2=0x7fea1015f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea15905640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fea1006e840 0x7fea1015f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.599+0000 7fea15905640 1 -- 172.21.15.81:0/3097063332 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea10159fb0 con 0x7fea100aed70 2024-06-29T15:38:24.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.600+0000 7fea15905640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea100aed70 0x7fea100ce130 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fea08009cd0 tx=0x7fea080316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:24.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.600+0000 7fea05ffb640 1 -- 172.21.15.81:0/3097063332 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08031da0 con 0x7fea100aed70 2024-06-29T15:38:24.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.600+0000 7fea05ffb640 1 -- 172.21.15.81:0/3097063332 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea08003a60 con 0x7fea100aed70 2024-06-29T15:38:24.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.600+0000 7fea05ffb640 1 -- 172.21.15.81:0/3097063332 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea080330d0 con 0x7fea100aed70 2024-06-29T15:38:24.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.600+0000 7fea17368640 1 -- 172.21.15.81:0/3097063332 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1015fc40 con 0x7fea100aed70 2024-06-29T15:38:24.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.600+0000 7fea17368640 1 -- 172.21.15.81:0/3097063332 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea10160200 con 0x7fea100aed70 2024-06-29T15:38:24.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.601+0000 7fea17368640 1 -- 172.21.15.81:0/3097063332 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea10159d20 con 0x7fea100aed70 2024-06-29T15:38:24.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.602+0000 7fea05ffb640 1 -- 172.21.15.81:0/3097063332 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7fea08033270 con 0x7fea100aed70 2024-06-29T15:38:24.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.602+0000 7fea05ffb640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fe9f40763f0 0x7fe9f4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.602+0000 7fea15104640 1 -- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fe9f40763f0 msgr2=0x7fe9f4078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:24.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.602+0000 7fea05ffb640 1 -- 172.21.15.81:0/3097063332 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 6299+0+0 (secure 0 0 0) 0x7fea080b8c40 con 0x7fea100aed70 2024-06-29T15:38:24.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.602+0000 7fea15104640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fe9f40763f0 0x7fe9f4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:24.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.607+0000 7fea05ffb640 1 -- 172.21.15.81:0/3097063332 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fea0804a030 con 0x7fea100aed70 2024-06-29T15:38:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.795+0000 7fea17368640 1 -- 172.21.15.81:0/3097063332 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fea100b0350 con 0x7fea100aed70 2024-06-29T15:38:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.795+0000 7fea05ffb640 1 -- 172.21.15.81:0/3097063332 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v155) ==== 104+0+11 (secure 0 0 0) 0x7fea0803f030 con 0x7fea100aed70 2024-06-29T15:38:24.796 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:38:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 -- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fe9f40763f0 msgr2=0x7fe9f4078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fe9f40763f0 0x7fe9f4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 -- 172.21.15.81:0/3097063332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea100aed70 msgr2=0x7fea100ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea100aed70 0x7fea100ce130 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fea08009cd0 tx=0x7fea080316f0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 -- 172.21.15.81:0/3097063332 shutdown_connections 2024-06-29T15:38:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7fe9f40763f0 0x7fe9f4078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea100aed70 0x7fea100ce130 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fea100ae960 0x7fea10149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 --2- 172.21.15.81:0/3097063332 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fea1006e840 0x7fea1015f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 -- 172.21.15.81:0/3097063332 >> 172.21.15.81:0/3097063332 conn(0x7fea100b3680 msgr2=0x7fea1015c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 -- 172.21.15.81:0/3097063332 shutdown_connections 2024-06-29T15:38:24.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.797+0000 7fea17368640 1 -- 172.21.15.81:0/3097063332 wait complete. 2024-06-29T15:38:24.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:38:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.928+0000 7f2f45ad6640 1 -- 172.21.15.81:0/980071129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f400ad570 msgr2=0x7f2f400cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.928+0000 7f2f45ad6640 1 --2- 172.21.15.81:0/980071129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f400ad570 0x7f2f400cec60 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f2f40074940 tx=0x7f2f3802a8f0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.928+0000 7f2f45ad6640 1 -- 172.21.15.81:0/980071129 shutdown_connections 2024-06-29T15:38:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.928+0000 7f2f45ad6640 1 --2- 172.21.15.81:0/980071129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f400ad570 0x7f2f400cec60 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.928+0000 7f2f45ad6640 1 --2- 172.21.15.81:0/980071129 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f4006ee60 0x7f2f400ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.928+0000 7f2f45ad6640 1 --2- 172.21.15.81:0/980071129 >> v2:172.21.15.167:3300/0 conn(0x7f2f4006c2f0 0x7f2f4006e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.928+0000 7f2f45ad6640 1 -- 172.21.15.81:0/980071129 >> 172.21.15.81:0/980071129 conn(0x7f2f400b1ec0 msgr2=0x7f2f400b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.928+0000 7f2f45ad6640 1 -- 172.21.15.81:0/980071129 shutdown_connections 2024-06-29T15:38:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.928+0000 7f2f45ad6640 1 -- 172.21.15.81:0/980071129 wait complete. 2024-06-29T15:38:24.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.930+0000 7f2f45ad6640 1 Processor -- start 2024-06-29T15:38:24.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.930+0000 7f2f45ad6640 1 -- start start 2024-06-29T15:38:24.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f45ad6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f4006c2f0 0x7f2f40151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f45ad6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f4006ee60 0x7f2f4015ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f45ad6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f400ad570 0x7f2f4014ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f45ad6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2f400cf530 con 0x7f2f4006c2f0 2024-06-29T15:38:24.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f45ad6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2f400cf3b0 con 0x7f2f400ad570 2024-06-29T15:38:24.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f45ad6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2f400cf6b0 con 0x7f2f4006ee60 2024-06-29T15:38:24.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f3f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f4006c2f0 0x7f2f40151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f3f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f4006c2f0 0x7f2f40151d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48034/0 (socket says 172.21.15.81:48034) 2024-06-29T15:38:24.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f3f7fe640 1 -- 172.21.15.81:0/2008496374 learned_addr learned my addr 172.21.15.81:0/2008496374 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:24.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f37fff640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f4006ee60 0x7f2f4015ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f3ffff640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f400ad570 0x7f2f4014ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f3f7fe640 1 -- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f4006ee60 msgr2=0x7f2f4015ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f3f7fe640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f4006ee60 0x7f2f4015ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f3f7fe640 1 -- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f400ad570 msgr2=0x7f2f4014ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f3f7fe640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f400ad570 0x7f2f4014ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.931+0000 7f2f3f7fe640 1 -- 172.21.15.81:0/2008496374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f40152270 con 0x7f2f4006c2f0 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.932+0000 7f2f3f7fe640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f4006c2f0 0x7f2f40151d30 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000e710 tx=0x7f2f3000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.932+0000 7f2f3cff9640 1 -- 172.21.15.81:0/2008496374 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3000ca50 con 0x7f2f4006c2f0 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.932+0000 7f2f3cff9640 1 -- 172.21.15.81:0/2008496374 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f3000cce0 con 0x7f2f4006c2f0 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.932+0000 7f2f45ad6640 1 -- 172.21.15.81:0/2008496374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4015d380 con 0x7f2f4006c2f0 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.932+0000 7f2f3cff9640 1 -- 172.21.15.81:0/2008496374 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30010690 con 0x7f2f4006c2f0 2024-06-29T15:38:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.932+0000 7f2f45ad6640 1 -- 172.21.15.81:0/2008496374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4015c360 con 0x7f2f4006c2f0 2024-06-29T15:38:24.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.933+0000 7f2f45ad6640 1 -- 172.21.15.81:0/2008496374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f400ad030 con 0x7f2f4006c2f0 2024-06-29T15:38:24.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.934+0000 7f2f3cff9640 1 -- 172.21.15.81:0/2008496374 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 86) ==== 98315+0+0 (secure 0 0 0) 0x7f2f30010830 con 0x7f2f4006c2f0 2024-06-29T15:38:24.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.934+0000 7f2f3cff9640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f2f200763f0 0x7f2f20078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:24.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.934+0000 7f2f37fff640 1 -- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f2f200763f0 msgr2=0x7f2f20078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/828186388 2024-06-29T15:38:24.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.934+0000 7f2f3cff9640 1 -- 172.21.15.81:0/2008496374 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 6299+0+0 (secure 0 0 0) 0x7f2f30098500 con 0x7f2f4006c2f0 2024-06-29T15:38:24.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.934+0000 7f2f37fff640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f2f200763f0 0x7f2f20078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:24.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:24.938+0000 7f2f3cff9640 1 -- 172.21.15.81:0/2008496374 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2f300613a0 con 0x7f2f4006c2f0 2024-06-29T15:38:25.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.052+0000 7f2f3cff9640 1 -- 172.21.15.81:0/2008496374 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 87) ==== 97912+0+0 (secure 0 0 0) 0x7f2f300967d0 con 0x7f2f4006c2f0 2024-06-29T15:38:25.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.052+0000 7f2f3cff9640 1 -- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f2f200763f0 msgr2=0x7f2f20078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:25.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.052+0000 7f2f3cff9640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f2f200763f0 0x7f2f20078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:25.103+0000 7f25b92be640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:25.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:25.103+0000 7f25b92be640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:25.104 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:25.103+0000 7f25b92be640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:25.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:25.103+0000 7f25b92be640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:25.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:25.103+0000 7f25b92be640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:25.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:25.104+0000 7f25bcac5640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:25.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:25.104+0000 7f25bcac5640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:25.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:25.104+0000 7f25bcac5640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:25.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:25.104+0000 7f25bcac5640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:25.105 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:25.104+0000 7f25bcac5640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:25.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.137+0000 7f2f45ad6640 1 -- 172.21.15.81:0/2008496374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2f40159000 con 0x7f2f4006c2f0 2024-06-29T15:38:25.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.137+0000 7f2f3cff9640 1 -- 172.21.15.81:0/2008496374 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v157) ==== 100+0+11 (secure 0 0 0) 0x7f2f3007aef0 con 0x7f2f4006c2f0 2024-06-29T15:38:25.138 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:38:25.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.140+0000 7f2f45ad6640 1 -- 172.21.15.81:0/2008496374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f4006c2f0 msgr2=0x7f2f40151d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.140+0000 7f2f45ad6640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f4006c2f0 0x7f2f40151d30 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000e710 tx=0x7f2f3000ebd0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.140+0000 7f2f45ad6640 1 -- 172.21.15.81:0/2008496374 shutdown_connections 2024-06-29T15:38:25.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.140+0000 7f2f45ad6640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:6832/828186388,v1:172.21.15.167:6833/828186388] conn(0x7f2f200763f0 0x7f2f20078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.140+0000 7f2f45ad6640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2f400ad570 0x7f2f4014ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.140+0000 7f2f45ad6640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2f4006ee60 0x7f2f4015ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.140+0000 7f2f45ad6640 1 --2- 172.21.15.81:0/2008496374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f4006c2f0 0x7f2f40151d30 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.140+0000 7f2f45ad6640 1 -- 172.21.15.81:0/2008496374 >> 172.21.15.81:0/2008496374 conn(0x7f2f400b1ec0 msgr2=0x7f2f400afb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.140+0000 7f2f45ad6640 1 -- 172.21.15.81:0/2008496374 shutdown_connections 2024-06-29T15:38:25.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.140+0000 7f2f45ad6640 1 -- 172.21.15.81:0/2008496374 wait complete. 2024-06-29T15:38:25.151 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:25.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.266+0000 7f855457f640 1 -- 172.21.15.81:0/4116475824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c0d2a90 msgr2=0x7f854c0d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.266+0000 7f855457f640 1 --2- 172.21.15.81:0/4116475824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c0d2a90 0x7f854c0d09c0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f8544005fd0 tx=0x7f854402a960 comp rx=0 tx=0).stop 2024-06-29T15:38:25.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.266+0000 7f855457f640 1 -- 172.21.15.81:0/4116475824 shutdown_connections 2024-06-29T15:38:25.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.266+0000 7f855457f640 1 --2- 172.21.15.81:0/4116475824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c0d2a90 0x7f854c0d09c0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.266+0000 7f855457f640 1 --2- 172.21.15.81:0/4116475824 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f854c0b9040 0x7f854c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.266+0000 7f855457f640 1 -- 172.21.15.81:0/4116475824 >> 172.21.15.81:0/4116475824 conn(0x7f854c09cf30 msgr2=0x7f854c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.266+0000 7f855457f640 1 -- 172.21.15.81:0/4116475824 shutdown_connections 2024-06-29T15:38:25.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.266+0000 7f855457f640 1 -- 172.21.15.81:0/4116475824 wait complete. 2024-06-29T15:38:25.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.268+0000 7f855457f640 1 Processor -- start 2024-06-29T15:38:25.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.268+0000 7f855457f640 1 -- start start 2024-06-29T15:38:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.269+0000 7f855457f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c0b5250 0x7f854c0a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.269+0000 7f855457f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f854c0b9040 0x7f854c151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.269+0000 7f855457f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f854c0d2a90 0x7f854c0cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.269+0000 7f855457f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f854c0b7d30 con 0x7f854c0b5250 2024-06-29T15:38:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.269+0000 7f855457f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f854c0b7bb0 con 0x7f854c0d2a90 2024-06-29T15:38:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.269+0000 7f855457f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f854c0b7eb0 con 0x7f854c0b9040 2024-06-29T15:38:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855231b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c0b5250 0x7f854c0a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855231b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c0b5250 0x7f854c0a0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48058/0 (socket says 172.21.15.81:48058) 2024-06-29T15:38:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855231b640 1 -- 172.21.15.81:0/1063850456 learned_addr learned my addr 172.21.15.81:0/1063850456 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f8551b1a640 1 --2- 172.21.15.81:0/1063850456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f854c0b9040 0x7f854c151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f8552b1c640 1 --2- 172.21.15.81:0/1063850456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f854c0d2a90 0x7f854c0cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855231b640 1 -- 172.21.15.81:0/1063850456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f854c0b9040 msgr2=0x7f854c151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855231b640 1 --2- 172.21.15.81:0/1063850456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f854c0b9040 0x7f854c151f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855231b640 1 -- 172.21.15.81:0/1063850456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f854c0d2a90 msgr2=0x7f854c0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855231b640 1 --2- 172.21.15.81:0/1063850456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f854c0d2a90 0x7f854c0cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855231b640 1 -- 172.21.15.81:0/1063850456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c0a1060 con 0x7f854c0b5250 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f8552b1c640 1 --2- 172.21.15.81:0/1063850456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f854c0d2a90 0x7f854c0cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855231b640 1 --2- 172.21.15.81:0/1063850456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c0b5250 0x7f854c0a0a10 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f854800c9a0 tx=0x7f854800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f8542ffd640 1 -- 172.21.15.81:0/1063850456 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548007bb0 con 0x7f854c0b5250 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f8542ffd640 1 -- 172.21.15.81:0/1063850456 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8548007e40 con 0x7f854c0b5250 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855457f640 1 -- 172.21.15.81:0/1063850456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854c0a1340 con 0x7f854c0b5250 2024-06-29T15:38:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f8542ffd640 1 -- 172.21.15.81:0/1063850456 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854800f690 con 0x7f854c0b5250 2024-06-29T15:38:25.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.270+0000 7f855457f640 1 -- 172.21.15.81:0/1063850456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f854c0a1900 con 0x7f854c0b5250 2024-06-29T15:38:25.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.271+0000 7f855457f640 1 -- 172.21.15.81:0/1063850456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f854c0b83b0 con 0x7f854c0b5250 2024-06-29T15:38:25.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.272+0000 7f8542ffd640 1 -- 172.21.15.81:0/1063850456 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 87) ==== 97912+0+0 (secure 0 0 0) 0x7f854800f830 con 0x7f854c0b5250 2024-06-29T15:38:25.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.273+0000 7f8542ffd640 1 -- 172.21.15.81:0/1063850456 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 6599+0+0 (secure 0 0 0) 0x7f8548097ca0 con 0x7f854c0b5250 2024-06-29T15:38:25.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.277+0000 7f8542ffd640 1 -- 172.21.15.81:0/1063850456 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f854809c070 con 0x7f854c0b5250 2024-06-29T15:38:25.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.496+0000 7f855457f640 1 -- 172.21.15.81:0/1063850456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f854c1617c0 con 0x7f854c0b5250 2024-06-29T15:38:25.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.497+0000 7f8542ffd640 1 -- 172.21.15.81:0/1063850456 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5231 (secure 0 0 0) 0x7f8548014030 con 0x7f854c0b5250 2024-06-29T15:38:25.497 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:25.497 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":77,"btime":"2024-06-29T14:42:01:549259+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/4068340499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":4068340499},{"type":"v1","addr":"172.21.15.81:6834","nonce":4068340499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1903134398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1903134398},{"type":"v1","addr":"172.21.15.167:6837","nonce":1903134398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1126258483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1126258483},{"type":"v1","addr":"172.21.15.81:6835","nonce":1126258483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.875457+0000","modified":"2024-06-29T14:42:00.919213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9111":{"gid":9111,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/4254103476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":4254103476},{"type":"v1","addr":"172.21.15.167:6836","nonce":4254103476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9111,"qdb_cluster":[9111]},"id":9}]} 2024-06-29T15:38:25.498 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 77 2024-06-29T15:38:25.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.498+0000 7f855457f640 1 -- 172.21.15.81:0/1063850456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c0b5250 msgr2=0x7f854c0a0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.498+0000 7f855457f640 1 --2- 172.21.15.81:0/1063850456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c0b5250 0x7f854c0a0a10 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f854800c9a0 tx=0x7f854800ce60 comp rx=0 tx=0).stop 2024-06-29T15:38:25.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.499+0000 7f855457f640 1 -- 172.21.15.81:0/1063850456 shutdown_connections 2024-06-29T15:38:25.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.499+0000 7f855457f640 1 --2- 172.21.15.81:0/1063850456 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f854c0d2a90 0x7f854c0cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.499+0000 7f855457f640 1 --2- 172.21.15.81:0/1063850456 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f854c0b9040 0x7f854c151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.499+0000 7f855457f640 1 --2- 172.21.15.81:0/1063850456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c0b5250 0x7f854c0a0a10 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.499+0000 7f855457f640 1 -- 172.21.15.81:0/1063850456 >> 172.21.15.81:0/1063850456 conn(0x7f854c09cf30 msgr2=0x7f854c06d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.499+0000 7f855457f640 1 -- 172.21.15.81:0/1063850456 shutdown_connections 2024-06-29T15:38:25.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.499+0000 7f855457f640 1 -- 172.21.15.81:0/1063850456 wait complete. 2024-06-29T15:38:25.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:25.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.622+0000 7f51b5495640 1 -- 172.21.15.81:0/635229507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51b00d0b90 msgr2=0x7f51b00d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.622+0000 7f51b5495640 1 --2- 172.21.15.81:0/635229507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51b00d0b90 0x7f51b00d1a80 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f5198009830 tx=0x7f519802a640 comp rx=0 tx=0).stop 2024-06-29T15:38:25.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.622+0000 7f51b5495640 1 -- 172.21.15.81:0/635229507 shutdown_connections 2024-06-29T15:38:25.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.622+0000 7f51b5495640 1 --2- 172.21.15.81:0/635229507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51b00d0b90 0x7f51b00d1a80 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.622+0000 7f51b5495640 1 -- 172.21.15.81:0/635229507 >> 172.21.15.81:0/635229507 conn(0x7f51b009cf30 msgr2=0x7f51b00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.622+0000 7f51b5495640 1 -- 172.21.15.81:0/635229507 shutdown_connections 2024-06-29T15:38:25.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.622+0000 7f51b5495640 1 -- 172.21.15.81:0/635229507 wait complete. 2024-06-29T15:38:25.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.623+0000 7f51b5495640 1 Processor -- start 2024-06-29T15:38:25.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.624+0000 7f51b5495640 1 -- start start 2024-06-29T15:38:25.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51b5495640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51b00b5250 0x7f51b0154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51b5495640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51b00b9040 0x7f51b015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51b5495640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b0155040 0x7f51b015a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51b5495640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f51b00d26d0 con 0x7f51b00b9040 2024-06-29T15:38:25.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51b5495640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f51b00b85c0 con 0x7f51b0155040 2024-06-29T15:38:25.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51b5495640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f51b00d2850 con 0x7f51b00b5250 2024-06-29T15:38:25.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51a67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51b00b9040 0x7f51b015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51a67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51b00b9040 0x7f51b015a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48078/0 (socket says 172.21.15.81:48078) 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51a67fc640 1 -- 172.21.15.81:0/1559186100 learned_addr learned my addr 172.21.15.81:0/1559186100 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51af7fe640 1 --2- 172.21.15.81:0/1559186100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b0155040 0x7f51b015a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51aeffd640 1 --2- 172.21.15.81:0/1559186100 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51b00b5250 0x7f51b0154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51a67fc640 1 -- 172.21.15.81:0/1559186100 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51b00b5250 msgr2=0x7f51b0154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51a67fc640 1 --2- 172.21.15.81:0/1559186100 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51b00b5250 0x7f51b0154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51a67fc640 1 -- 172.21.15.81:0/1559186100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b0155040 msgr2=0x7f51b015a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51a67fc640 1 --2- 172.21.15.81:0/1559186100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b0155040 0x7f51b015a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.625+0000 7f51a67fc640 1 -- 172.21.15.81:0/1559186100 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b0155ac0 con 0x7f51b00b9040 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.626+0000 7f51a67fc640 1 --2- 172.21.15.81:0/1559186100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51b00b9040 0x7f51b015a260 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f5198009920 tx=0x7f5198033df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.626+0000 7f51a7fff640 1 -- 172.21.15.81:0/1559186100 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51980332e0 con 0x7f51b00b9040 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.626+0000 7f51a7fff640 1 -- 172.21.15.81:0/1559186100 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5198033480 con 0x7f51b00b9040 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.626+0000 7f51b5495640 1 -- 172.21.15.81:0/1559186100 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b0155d30 con 0x7f51b00b9040 2024-06-29T15:38:25.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.626+0000 7f51a7fff640 1 -- 172.21.15.81:0/1559186100 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198005280 con 0x7f51b00b9040 2024-06-29T15:38:25.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.626+0000 7f51b5495640 1 -- 172.21.15.81:0/1559186100 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b0157890 con 0x7f51b00b9040 2024-06-29T15:38:25.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.627+0000 7f51b5495640 1 -- 172.21.15.81:0/1559186100 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51b00d2de0 con 0x7f51b00b9040 2024-06-29T15:38:25.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.628+0000 7f51a7fff640 1 -- 172.21.15.81:0/1559186100 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 87) ==== 97912+0+0 (secure 0 0 0) 0x7f5198005420 con 0x7f51b00b9040 2024-06-29T15:38:25.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.628+0000 7f51a7fff640 1 -- 172.21.15.81:0/1559186100 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 6599+0+0 (secure 0 0 0) 0x7f51980b8490 con 0x7f51b00b9040 2024-06-29T15:38:25.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.632+0000 7f51a7fff640 1 -- 172.21.15.81:0/1559186100 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f519803f030 con 0x7f51b00b9040 2024-06-29T15:38:25.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.813+0000 7f51b5495640 1 -- 172.21.15.81:0/1559186100 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51b01e84f0 con 0x7f51b00b9040 2024-06-29T15:38:25.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.814+0000 7f51a7fff640 1 -- 172.21.15.81:0/1559186100 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) ==== 74+0+17979 (secure 0 0 0) 0x7f5198081870 con 0x7f51b00b9040 2024-06-29T15:38:25.817 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:25.817 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":157,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:25.044482+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.170598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.829654+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":156,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.25","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3122315648":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/3189551239":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/4281804658":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/2936900238":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/386189888":"2024-06-30T15:38:25.044444+0000","172.21.15.167:6833/828186388":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/1730106798":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/2472315005":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3327921477":"2024-06-30T15:38:25.044444+0000","172.21.15.167:6833/3201324170":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/2674867295":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/588266296":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3014230071":"2024-06-30T14:42:10.211800+0000","172.21.15.167:6832/828186388":"2024-06-30T15:38:25.044444+0000","172.21.15.167:6832/3201324170":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3312315666":"2024-06-30T14:42:10.211800+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:25.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.816+0000 7f51b5495640 1 -- 172.21.15.81:0/1559186100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51b00b9040 msgr2=0x7f51b015a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.816+0000 7f51b5495640 1 --2- 172.21.15.81:0/1559186100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51b00b9040 0x7f51b015a260 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f5198009920 tx=0x7f5198033df0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.816+0000 7f51b5495640 1 -- 172.21.15.81:0/1559186100 shutdown_connections 2024-06-29T15:38:25.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.816+0000 7f51b5495640 1 --2- 172.21.15.81:0/1559186100 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51b0155040 0x7f51b015a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.816+0000 7f51b5495640 1 --2- 172.21.15.81:0/1559186100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51b00b9040 0x7f51b015a260 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.816+0000 7f51b5495640 1 --2- 172.21.15.81:0/1559186100 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51b00b5250 0x7f51b0154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.816+0000 7f51b5495640 1 -- 172.21.15.81:0/1559186100 >> 172.21.15.81:0/1559186100 conn(0x7f51b009cf30 msgr2=0x7f51b00b5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.816+0000 7f51b5495640 1 -- 172.21.15.81:0/1559186100 shutdown_connections 2024-06-29T15:38:25.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.816+0000 7f51b5495640 1 -- 172.21.15.81:0/1559186100 wait complete. 2024-06-29T15:38:25.829 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:38:25.830 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:25.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.942+0000 7fac660df640 1 -- 172.21.15.81:0/3429611297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac4c003a00 msgr2=0x7fac4c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.942+0000 7fac660df640 1 --2- 172.21.15.81:0/3429611297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac4c003a00 0x7fac4c003e50 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fac4800c710 tx=0x7fac4c005830 comp rx=0 tx=0).stop 2024-06-29T15:38:25.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.942+0000 7fac660df640 1 -- 172.21.15.81:0/3429611297 shutdown_connections 2024-06-29T15:38:25.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.942+0000 7fac660df640 1 --2- 172.21.15.81:0/3429611297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac4c003a00 0x7fac4c003e50 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.942+0000 7fac660df640 1 -- 172.21.15.81:0/3429611297 >> 172.21.15.81:0/3429611297 conn(0x7fac600b3680 msgr2=0x7fac600b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:25.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.942+0000 7fac660df640 1 -- 172.21.15.81:0/3429611297 shutdown_connections 2024-06-29T15:38:25.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.942+0000 7fac660df640 1 -- 172.21.15.81:0/3429611297 wait complete. 2024-06-29T15:38:25.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.943+0000 7fac660df640 1 Processor -- start 2024-06-29T15:38:25.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.944+0000 7fac660df640 1 -- start start 2024-06-29T15:38:25.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac660df640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac4c003a00 0x7fac60155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac660df640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fac600c8f30 0x7fac600ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac660df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac600caee0 0x7fac601e28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:25.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac660df640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac600cc930 con 0x7fac600caee0 2024-06-29T15:38:25.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac660df640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fac600cc7b0 con 0x7fac4c003a00 2024-06-29T15:38:25.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac660df640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fac600ccab0 con 0x7fac600c8f30 2024-06-29T15:38:25.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac600caee0 0x7fac601e28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac600caee0 0x7fac601e28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48102/0 (socket says 172.21.15.81:48102) 2024-06-29T15:38:25.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5ffff640 1 -- 172.21.15.81:0/13200438 learned_addr learned my addr 172.21.15.81:0/13200438 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:25.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5f7fe640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac4c003a00 0x7fac60155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5effd640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fac600c8f30 0x7fac600ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:25.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5ffff640 1 -- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fac600c8f30 msgr2=0x7fac600ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5ffff640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fac600c8f30 0x7fac600ca9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5ffff640 1 -- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac4c003a00 msgr2=0x7fac60155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:25.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5ffff640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac4c003a00 0x7fac60155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:25.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5ffff640 1 -- 172.21.15.81:0/13200438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac600cb2f0 con 0x7fac600caee0 2024-06-29T15:38:25.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.945+0000 7fac5ffff640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac600caee0 0x7fac601e28d0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fac600d0f70 tx=0x7fac580064f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:25.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.946+0000 7fac57fff640 1 -- 172.21.15.81:0/13200438 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac580119c0 con 0x7fac600caee0 2024-06-29T15:38:25.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.946+0000 7fac57fff640 1 -- 172.21.15.81:0/13200438 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac5801a070 con 0x7fac600caee0 2024-06-29T15:38:25.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.946+0000 7fac57fff640 1 -- 172.21.15.81:0/13200438 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58011e40 con 0x7fac600caee0 2024-06-29T15:38:25.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.946+0000 7fac660df640 1 -- 172.21.15.81:0/13200438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac601e7050 con 0x7fac600caee0 2024-06-29T15:38:25.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.946+0000 7fac660df640 1 -- 172.21.15.81:0/13200438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac601e20d0 con 0x7fac600caee0 2024-06-29T15:38:25.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.947+0000 7fac660df640 1 -- 172.21.15.81:0/13200438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac600cd010 con 0x7fac600caee0 2024-06-29T15:38:25.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.947+0000 7fac57fff640 1 -- 172.21.15.81:0/13200438 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 87) ==== 97912+0+0 (secure 0 0 0) 0x7fac5801f460 con 0x7fac600caee0 2024-06-29T15:38:25.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.948+0000 7fac57fff640 1 -- 172.21.15.81:0/13200438 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 6599+0+0 (secure 0 0 0) 0x7fac58097620 con 0x7fac600caee0 2024-06-29T15:38:25.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:25.953+0000 7fac57fff640 1 -- 172.21.15.81:0/13200438 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fac600cb2f0 con 0x7fac600caee0 2024-06-29T15:38:26.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.053+0000 7fac57fff640 1 -- 172.21.15.81:0/13200438 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 88) ==== 98039+0+0 (secure 0 0 0) 0x7fac5805c370 con 0x7fac600caee0 2024-06-29T15:38:26.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.054+0000 7fac57fff640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fac4007bca0 0x7fac4007e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.054+0000 7fac5f7fe640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fac4007bca0 0x7fac4007e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.057+0000 7fac5f7fe640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fac4007bca0 0x7fac4007e0c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fac48032000 tx=0x7fac4802c040 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.171+0000 7fac660df640 1 -- 172.21.15.81:0/13200438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac6015abb0 con 0x7fac600caee0 2024-06-29T15:38:26.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.172+0000 7fac57fff640 1 -- 172.21.15.81:0/13200438 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5231 (secure 0 0 0) 0x7fac600d0b80 con 0x7fac600caee0 2024-06-29T15:38:26.174 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:26.174 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":77,"btime":"2024-06-29T14:42:01:549259+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/4068340499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":4068340499},{"type":"v1","addr":"172.21.15.81:6834","nonce":4068340499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1903134398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1903134398},{"type":"v1","addr":"172.21.15.167:6837","nonce":1903134398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1126258483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1126258483},{"type":"v1","addr":"172.21.15.81:6835","nonce":1126258483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.875457+0000","modified":"2024-06-29T14:42:00.919213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9111":{"gid":9111,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/4254103476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":4254103476},{"type":"v1","addr":"172.21.15.167:6836","nonce":4254103476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9111,"qdb_cluster":[9111]},"id":9}]} 2024-06-29T15:38:26.174 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 77 2024-06-29T15:38:26.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 -- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fac4007bca0 msgr2=0x7fac4007e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fac4007bca0 0x7fac4007e0c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fac48032000 tx=0x7fac4802c040 comp rx=0 tx=0).stop 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 -- 172.21.15.81:0/13200438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac600caee0 msgr2=0x7fac601e28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac600caee0 0x7fac601e28d0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fac600d0f70 tx=0x7fac580064f0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 -- 172.21.15.81:0/13200438 shutdown_connections 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fac4007bca0 0x7fac4007e0c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac600caee0 0x7fac601e28d0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fac600c8f30 0x7fac600ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 --2- 172.21.15.81:0/13200438 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac4c003a00 0x7fac60155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 -- 172.21.15.81:0/13200438 >> 172.21.15.81:0/13200438 conn(0x7fac600b3680 msgr2=0x7fac600b5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 -- 172.21.15.81:0/13200438 shutdown_connections 2024-06-29T15:38:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.174+0000 7fac660df640 1 -- 172.21.15.81:0/13200438 wait complete. 2024-06-29T15:38:26.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:26.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.299+0000 7f223b409640 1 -- 172.21.15.81:0/3731347619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f223406f6a0 msgr2=0x7f223415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.299+0000 7f223b409640 1 --2- 172.21.15.81:0/3731347619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f223406f6a0 0x7f223415aed0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f22340743b0 tx=0x7f222402a560 comp rx=0 tx=0).stop 2024-06-29T15:38:26.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.299+0000 7f223b409640 1 -- 172.21.15.81:0/3731347619 shutdown_connections 2024-06-29T15:38:26.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.299+0000 7f223b409640 1 --2- 172.21.15.81:0/3731347619 >> v2:172.21.15.167:3300/0 conn(0x7f22340b3960 0x7f22340b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.299+0000 7f223b409640 1 --2- 172.21.15.81:0/3731347619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f223406f6a0 0x7f223415aed0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.299+0000 7f223b409640 1 -- 172.21.15.81:0/3731347619 >> 172.21.15.81:0/3731347619 conn(0x7f22340d0930 msgr2=0x7f22340d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.299+0000 7f223b409640 1 -- 172.21.15.81:0/3731347619 shutdown_connections 2024-06-29T15:38:26.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.299+0000 7f223b409640 1 -- 172.21.15.81:0/3731347619 wait complete. 2024-06-29T15:38:26.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.301+0000 7f223b409640 1 Processor -- start 2024-06-29T15:38:26.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.301+0000 7f223b409640 1 -- start start 2024-06-29T15:38:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f223b409640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f223406f6a0 0x7f223414ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f223b409640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22340b3960 0x7f223414c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f223b409640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22340b57b0 0x7f223414a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f223b409640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f22340b4430 con 0x7f22340b57b0 2024-06-29T15:38:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f223b409640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f22340b42b0 con 0x7f22340b3960 2024-06-29T15:38:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f223b409640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f22340b45b0 con 0x7f223406f6a0 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f22399a6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22340b57b0 0x7f223414a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f22399a6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22340b57b0 0x7f223414a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48118/0 (socket says 172.21.15.81:48118) 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f22399a6640 1 -- 172.21.15.81:0/2015769319 learned_addr learned my addr 172.21.15.81:0/2015769319 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f22389a4640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22340b3960 0x7f223414c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f22391a5640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f223406f6a0 0x7f223414ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f22399a6640 1 -- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f223406f6a0 msgr2=0x7f223414ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f22399a6640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f223406f6a0 0x7f223414ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f22399a6640 1 -- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22340b3960 msgr2=0x7f223414c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f22399a6640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22340b3960 0x7f223414c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.302+0000 7f22399a6640 1 -- 172.21.15.81:0/2015769319 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223414ac60 con 0x7f22340b57b0 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.303+0000 7f22399a6640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22340b57b0 0x7f223414a720 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f222c00ba10 tx=0x7f222c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.303+0000 7f2229ffb640 1 -- 172.21.15.81:0/2015769319 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c019070 con 0x7f22340b57b0 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.303+0000 7f2229ffb640 1 -- 172.21.15.81:0/2015769319 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f222c00cc50 con 0x7f22340b57b0 2024-06-29T15:38:26.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.303+0000 7f223b409640 1 -- 172.21.15.81:0/2015769319 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223414f600 con 0x7f22340b57b0 2024-06-29T15:38:26.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.303+0000 7f2229ffb640 1 -- 172.21.15.81:0/2015769319 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c0146a0 con 0x7f22340b57b0 2024-06-29T15:38:26.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.303+0000 7f223b409640 1 -- 172.21.15.81:0/2015769319 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223414fbc0 con 0x7f22340b57b0 2024-06-29T15:38:26.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.304+0000 7f223b409640 1 -- 172.21.15.81:0/2015769319 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f223406e2a0 con 0x7f22340b57b0 2024-06-29T15:38:26.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.305+0000 7f2229ffb640 1 -- 172.21.15.81:0/2015769319 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 88) ==== 98039+0+0 (secure 0 0 0) 0x7f222c01c070 con 0x7f22340b57b0 2024-06-29T15:38:26.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.305+0000 7f2229ffb640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f2218076120 0x7f2218078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.305+0000 7f2229ffb640 1 -- 172.21.15.81:0/2015769319 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 6599+0+0 (secure 0 0 0) 0x7f222c0973d0 con 0x7f22340b57b0 2024-06-29T15:38:26.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.305+0000 7f22391a5640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f2218076120 0x7f2218078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.306+0000 7f22391a5640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f2218076120 0x7f2218078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2230006fd0 tx=0x7f2230009060 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.310+0000 7f2229ffb640 1 -- 172.21.15.81:0/2015769319 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f223406e2a0 con 0x7f22340b57b0 2024-06-29T15:38:26.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.525+0000 7f223b409640 1 -- 172.21.15.81:0/2015769319 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f223414f390 con 0x7f22340b57b0 2024-06-29T15:38:26.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.525+0000 7f2229ffb640 1 -- 172.21.15.81:0/2015769319 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5231 (secure 0 0 0) 0x7f222c0606a0 con 0x7f22340b57b0 2024-06-29T15:38:26.527 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:26.527 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":77,"btime":"2024-06-29T14:42:01:549259+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/4068340499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":4068340499},{"type":"v1","addr":"172.21.15.81:6834","nonce":4068340499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1903134398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1903134398},{"type":"v1","addr":"172.21.15.167:6837","nonce":1903134398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1126258483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1126258483},{"type":"v1","addr":"172.21.15.81:6835","nonce":1126258483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T14:41:57.875457+0000","modified":"2024-06-29T14:42:00.919213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9111},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9111":{"gid":9111,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6836/4254103476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":4254103476},{"type":"v1","addr":"172.21.15.167:6836","nonce":4254103476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9111,"qdb_cluster":[9111]},"id":9}]} 2024-06-29T15:38:26.527 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 77 2024-06-29T15:38:26.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.527+0000 7f223b409640 1 -- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f2218076120 msgr2=0x7f2218078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.527+0000 7f223b409640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f2218076120 0x7f2218078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2230006fd0 tx=0x7f2230009060 comp rx=0 tx=0).stop 2024-06-29T15:38:26.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.527+0000 7f223b409640 1 -- 172.21.15.81:0/2015769319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22340b57b0 msgr2=0x7f223414a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.527+0000 7f223b409640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22340b57b0 0x7f223414a720 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f222c00ba10 tx=0x7f222c00bed0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.527+0000 7f223b409640 1 -- 172.21.15.81:0/2015769319 shutdown_connections 2024-06-29T15:38:26.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.528+0000 7f223b409640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f2218076120 0x7f2218078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.528+0000 7f223b409640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22340b57b0 0x7f223414a720 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.528+0000 7f223b409640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22340b3960 0x7f223414c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.528+0000 7f223b409640 1 --2- 172.21.15.81:0/2015769319 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f223406f6a0 0x7f223414ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.528+0000 7f223b409640 1 -- 172.21.15.81:0/2015769319 >> 172.21.15.81:0/2015769319 conn(0x7f22340d0930 msgr2=0x7f2234159a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.528+0000 7f223b409640 1 -- 172.21.15.81:0/2015769319 shutdown_connections 2024-06-29T15:38:26.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.528+0000 7f223b409640 1 -- 172.21.15.81:0/2015769319 wait complete. 2024-06-29T15:38:26.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:26.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.652+0000 7fe684474640 1 -- 172.21.15.81:0/3991462240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe67c08c270 msgr2=0x7fe67c0ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.652+0000 7fe684474640 1 --2- 172.21.15.81:0/3991462240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe67c08c270 0x7fe67c0ce930 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe678005fd0 tx=0x7fe67802a4b0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.652+0000 7fe684474640 1 -- 172.21.15.81:0/3991462240 shutdown_connections 2024-06-29T15:38:26.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.652+0000 7fe684474640 1 --2- 172.21.15.81:0/3991462240 >> v2:172.21.15.167:3300/0 conn(0x7fe67c08ccd0 0x7fe67c0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.652+0000 7fe684474640 1 --2- 172.21.15.81:0/3991462240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe67c08c270 0x7fe67c0ce930 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.652+0000 7fe684474640 1 -- 172.21.15.81:0/3991462240 >> 172.21.15.81:0/3991462240 conn(0x7fe67c0b56d0 msgr2=0x7fe67c0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.652+0000 7fe684474640 1 -- 172.21.15.81:0/3991462240 shutdown_connections 2024-06-29T15:38:26.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.652+0000 7fe684474640 1 -- 172.21.15.81:0/3991462240 wait complete. 2024-06-29T15:38:26.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.653+0000 7fe684474640 1 Processor -- start 2024-06-29T15:38:26.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.654+0000 7fe684474640 1 -- start start 2024-06-29T15:38:26.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe684474640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe67c08ccd0 0x7fe67c09d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe684474640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe67c0d17b0 0x7fe67c160de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe684474640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c167620 0x7fe67c1665e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe684474640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c0d2450 con 0x7fe67c167620 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe684474640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c0d22d0 con 0x7fe67c0d17b0 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe684474640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe67c0d25d0 con 0x7fe67c08ccd0 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe682a11640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c167620 0x7fe67c1665e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe682a11640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c167620 0x7fe67c1665e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48122/0 (socket says 172.21.15.81:48122) 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe682a11640 1 -- 172.21.15.81:0/1326485792 learned_addr learned my addr 172.21.15.81:0/1326485792 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe682210640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe67c08ccd0 0x7fe67c09d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe681a0f640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe67c0d17b0 0x7fe67c160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe682a11640 1 -- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe67c08ccd0 msgr2=0x7fe67c09d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe682a11640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe67c08ccd0 0x7fe67c09d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe682a11640 1 -- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe67c0d17b0 msgr2=0x7fe67c160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe682a11640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe67c0d17b0 0x7fe67c160de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.655+0000 7fe682a11640 1 -- 172.21.15.81:0/1326485792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe67c0a1bc0 con 0x7fe67c167620 2024-06-29T15:38:26.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.656+0000 7fe682a11640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c167620 0x7fe67c1665e0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fe67400c700 tx=0x7fe67400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.656+0000 7fe672ffd640 1 -- 172.21.15.81:0/1326485792 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674003b70 con 0x7fe67c167620 2024-06-29T15:38:26.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.656+0000 7fe672ffd640 1 -- 172.21.15.81:0/1326485792 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe674003d10 con 0x7fe67c167620 2024-06-29T15:38:26.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.656+0000 7fe672ffd640 1 -- 172.21.15.81:0/1326485792 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67400aab0 con 0x7fe67c167620 2024-06-29T15:38:26.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.656+0000 7fe684474640 1 -- 172.21.15.81:0/1326485792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe67c0a1320 con 0x7fe67c167620 2024-06-29T15:38:26.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.656+0000 7fe684474640 1 -- 172.21.15.81:0/1326485792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe67c0a18e0 con 0x7fe67c167620 2024-06-29T15:38:26.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.657+0000 7fe684474640 1 -- 172.21.15.81:0/1326485792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe67c08c270 con 0x7fe67c167620 2024-06-29T15:38:26.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.657+0000 7fe672ffd640 1 -- 172.21.15.81:0/1326485792 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 88) ==== 98039+0+0 (secure 0 0 0) 0x7fe67400ac50 con 0x7fe67c167620 2024-06-29T15:38:26.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.658+0000 7fe672ffd640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe660076120 0x7fe660078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.658+0000 7fe672ffd640 1 -- 172.21.15.81:0/1326485792 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 6599+0+0 (secure 0 0 0) 0x7fe6740973b0 con 0x7fe67c167620 2024-06-29T15:38:26.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.658+0000 7fe682210640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe660076120 0x7fe660078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.659+0000 7fe682210640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe660076120 0x7fe660078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe678033000 tx=0x7fe678004730 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.662+0000 7fe672ffd640 1 -- 172.21.15.81:0/1326485792 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe674060680 con 0x7fe67c167620 2024-06-29T15:38:26.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.843+0000 7fe684474640 1 -- 172.21.15.81:0/1326485792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe67c165550 con 0x7fe67c167620 2024-06-29T15:38:26.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.845+0000 7fe672ffd640 1 -- 172.21.15.81:0/1326485792 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) ==== 74+0+17979 (secure 0 0 0) 0x7fe674064980 con 0x7fe67c167620 2024-06-29T15:38:26.846 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:26.846 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":157,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:25.044482+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-06-29T14:41:56.170598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"145","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-06-29T14:41:56.829654+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":156,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.25","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3122315648":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/3189551239":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/4281804658":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/2936900238":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/386189888":"2024-06-30T15:38:25.044444+0000","172.21.15.167:6833/828186388":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/1730106798":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/2472315005":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3327921477":"2024-06-30T15:38:25.044444+0000","172.21.15.167:6833/3201324170":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/2674867295":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/588266296":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3014230071":"2024-06-30T14:42:10.211800+0000","172.21.15.167:6832/828186388":"2024-06-30T15:38:25.044444+0000","172.21.15.167:6832/3201324170":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3312315666":"2024-06-30T14:42:10.211800+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:26.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 -- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe660076120 msgr2=0x7fe660078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe660076120 0x7fe660078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe678033000 tx=0x7fe678004730 comp rx=0 tx=0).stop 2024-06-29T15:38:26.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 -- 172.21.15.81:0/1326485792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c167620 msgr2=0x7fe67c1665e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c167620 0x7fe67c1665e0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fe67400c700 tx=0x7fe67400cbc0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 -- 172.21.15.81:0/1326485792 shutdown_connections 2024-06-29T15:38:26.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe660076120 0x7fe660078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c167620 0x7fe67c1665e0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe67c0d17b0 0x7fe67c160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 --2- 172.21.15.81:0/1326485792 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe67c08ccd0 0x7fe67c09d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 -- 172.21.15.81:0/1326485792 >> 172.21.15.81:0/1326485792 conn(0x7fe67c0b56d0 msgr2=0x7fe67c068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 -- 172.21.15.81:0/1326485792 shutdown_connections 2024-06-29T15:38:26.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.847+0000 7fe684474640 1 -- 172.21.15.81:0/1326485792 wait complete. 2024-06-29T15:38:26.860 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:38:26.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.973+0000 7f6a2b52b640 1 -- 172.21.15.81:0/1921174973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a240b35b0 msgr2=0x7f6a2415b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.973+0000 7f6a2b52b640 1 --2- 172.21.15.81:0/1921174973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a240b35b0 0x7f6a2415b5b0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f6a240cc260 tx=0x7f6a2002a4a0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.973+0000 7f6a2b52b640 1 -- 172.21.15.81:0/1921174973 shutdown_connections 2024-06-29T15:38:26.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.973+0000 7f6a2b52b640 1 --2- 172.21.15.81:0/1921174973 >> v2:172.21.15.167:3300/0 conn(0x7f6a240b4430 0x7f6a240cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.973+0000 7f6a2b52b640 1 --2- 172.21.15.81:0/1921174973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a240b35b0 0x7f6a2415b5b0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.973+0000 7f6a2b52b640 1 -- 172.21.15.81:0/1921174973 >> 172.21.15.81:0/1921174973 conn(0x7f6a2406ccc0 msgr2=0x7f6a240b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:26.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.973+0000 7f6a2b52b640 1 -- 172.21.15.81:0/1921174973 shutdown_connections 2024-06-29T15:38:26.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.973+0000 7f6a2b52b640 1 -- 172.21.15.81:0/1921174973 wait complete. 2024-06-29T15:38:26.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.975+0000 7f6a2b52b640 1 Processor -- start 2024-06-29T15:38:26.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.975+0000 7f6a2b52b640 1 -- start start 2024-06-29T15:38:26.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a2b52b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a2406f660 0x7f6a24133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a2b52b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a240b35b0 0x7f6a24166420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a2b52b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6a240b4430 0x7f6a24144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a2b52b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2406f1e0 con 0x7f6a2406f660 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a2b52b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2406f060 con 0x7f6a240b4430 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a2b52b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6a2406f360 con 0x7f6a240b35b0 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a292c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a2406f660 0x7f6a24133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a292c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a2406f660 0x7f6a24133760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48144/0 (socket says 172.21.15.81:48144) 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a292c7640 1 -- 172.21.15.81:0/2051896182 learned_addr learned my addr 172.21.15.81:0/2051896182 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a28ac6640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a240b35b0 0x7f6a24166420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a29ac8640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6a240b4430 0x7f6a24144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a292c7640 1 -- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a240b35b0 msgr2=0x7f6a24166420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a292c7640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a240b35b0 0x7f6a24166420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a292c7640 1 -- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6a240b4430 msgr2=0x7f6a24144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:26.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a292c7640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6a240b4430 0x7f6a24144100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:26.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.976+0000 7f6a292c7640 1 -- 172.21.15.81:0/2051896182 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a24145420 con 0x7f6a2406f660 2024-06-29T15:38:26.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.977+0000 7f6a292c7640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a2406f660 0x7f6a24133760 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f6a2002aa90 tx=0x7f6a20033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.977+0000 7f6a19ffb640 1 -- 172.21.15.81:0/2051896182 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20005c90 con 0x7f6a2406f660 2024-06-29T15:38:26.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.977+0000 7f6a19ffb640 1 -- 172.21.15.81:0/2051896182 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a20031e50 con 0x7f6a2406f660 2024-06-29T15:38:26.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.977+0000 7f6a19ffb640 1 -- 172.21.15.81:0/2051896182 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20005260 con 0x7f6a2406f660 2024-06-29T15:38:26.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.977+0000 7f6a2b52b640 1 -- 172.21.15.81:0/2051896182 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a24145700 con 0x7f6a2406f660 2024-06-29T15:38:26.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.977+0000 7f6a2b52b640 1 -- 172.21.15.81:0/2051896182 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a24145cc0 con 0x7f6a2406f660 2024-06-29T15:38:26.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.978+0000 7f6a2b52b640 1 -- 172.21.15.81:0/2051896182 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a240af520 con 0x7f6a2406f660 2024-06-29T15:38:26.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.979+0000 7f6a19ffb640 1 -- 172.21.15.81:0/2051896182 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 88) ==== 98039+0+0 (secure 0 0 0) 0x7f6a20005400 con 0x7f6a2406f660 2024-06-29T15:38:26.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.979+0000 7f6a19ffb640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6a08076120 0x7f6a08078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:26.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.979+0000 7f6a19ffb640 1 -- 172.21.15.81:0/2051896182 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 6599+0+0 (secure 0 0 0) 0x7f6a200b8730 con 0x7f6a2406f660 2024-06-29T15:38:26.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.980+0000 7f6a28ac6640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6a08076120 0x7f6a08078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:26.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.980+0000 7f6a28ac6640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6a08076120 0x7f6a08078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6a241f65a0 tx=0x7f6a14005f90 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:26.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:26.984+0000 7f6a19ffb640 1 -- 172.21.15.81:0/2051896182 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6a2003f030 con 0x7f6a2406f660 2024-06-29T15:38:27.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.206+0000 7f6a2b52b640 1 -- 172.21.15.81:0/2051896182 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6a24130c50 con 0x7f6a2406f660 2024-06-29T15:38:27.208 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T15:38:27.206+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:38:27.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.208+0000 7f6a19ffb640 1 -- 172.21.15.81:0/2051896182 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v78) ==== 161+0+0 (secure 0 0 0) 0x7f6a20031770 con 0x7f6a2406f660 2024-06-29T15:38:27.209 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 -- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6a08076120 msgr2=0x7f6a08078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6a08076120 0x7f6a08078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6a241f65a0 tx=0x7f6a14005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 -- 172.21.15.81:0/2051896182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a2406f660 msgr2=0x7f6a24133760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a2406f660 0x7f6a24133760 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f6a2002aa90 tx=0x7f6a20033c20 comp rx=0 tx=0).stop 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 -- 172.21.15.81:0/2051896182 shutdown_connections 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6a08076120 0x7f6a08078630 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6a241f65a0 tx=0x7f6a14005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6a240b4430 0x7f6a24144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6a240b35b0 0x7f6a24166420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 --2- 172.21.15.81:0/2051896182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a2406f660 0x7f6a24133760 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 -- 172.21.15.81:0/2051896182 >> 172.21.15.81:0/2051896182 conn(0x7f6a2406ccc0 msgr2=0x7f6a24149060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 -- 172.21.15.81:0/2051896182 shutdown_connections 2024-06-29T15:38:27.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.211+0000 7f6a2b52b640 1 -- 172.21.15.81:0/2051896182 wait complete. 2024-06-29T15:38:27.224 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:38:27.276 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T15:38:27.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.341+0000 7fcbabfd4640 1 -- 172.21.15.81:0/2129054189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba4072bd0 msgr2=0x7fcba4072fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.341+0000 7fcbabfd4640 1 --2- 172.21.15.81:0/2129054189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba4072bd0 0x7fcba4072fe0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fcba40af630 tx=0x7fcb9402aa30 comp rx=0 tx=0).stop 2024-06-29T15:38:27.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.341+0000 7fcbabfd4640 1 -- 172.21.15.81:0/2129054189 shutdown_connections 2024-06-29T15:38:27.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.341+0000 7fcbabfd4640 1 --2- 172.21.15.81:0/2129054189 >> v2:172.21.15.167:3300/0 conn(0x7fcba40b4690 0x7fcba41641e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.341+0000 7fcbabfd4640 1 --2- 172.21.15.81:0/2129054189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba4072bd0 0x7fcba4072fe0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.341+0000 7fcbabfd4640 1 -- 172.21.15.81:0/2129054189 >> 172.21.15.81:0/2129054189 conn(0x7fcba40b1730 msgr2=0x7fcba40b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:27.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.341+0000 7fcbabfd4640 1 -- 172.21.15.81:0/2129054189 shutdown_connections 2024-06-29T15:38:27.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.341+0000 7fcbabfd4640 1 -- 172.21.15.81:0/2129054189 wait complete. 2024-06-29T15:38:27.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.343+0000 7fcbabfd4640 1 Processor -- start 2024-06-29T15:38:27.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.343+0000 7fcbabfd4640 1 -- start start 2024-06-29T15:38:27.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcbabfd4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcba4072bd0 0x7fcba41529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcbabfd4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcba40b3b70 0x7fcba40aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcbabfd4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba40b4690 0x7fcba414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcbabfd4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcba4164720 con 0x7fcba40b4690 2024-06-29T15:38:27.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcbabfd4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcba40cdcf0 con 0x7fcba40b3b70 2024-06-29T15:38:27.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcbabfd4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcba41648a0 con 0x7fcba4072bd0 2024-06-29T15:38:27.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcbaa571640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba40b4690 0x7fcba414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcbaa571640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba40b4690 0x7fcba414ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48156/0 (socket says 172.21.15.81:48156) 2024-06-29T15:38:27.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcbaa571640 1 -- 172.21.15.81:0/393449502 learned_addr learned my addr 172.21.15.81:0/393449502 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:27.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcba956f640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcba40b3b70 0x7fcba40aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.344+0000 7fcba9d70640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcba4072bd0 0x7fcba41529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcbaa571640 1 -- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcba4072bd0 msgr2=0x7fcba41529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcbaa571640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcba4072bd0 0x7fcba41529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcbaa571640 1 -- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcba40b3b70 msgr2=0x7fcba40aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcbaa571640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcba40b3b70 0x7fcba40aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcbaa571640 1 -- 172.21.15.81:0/393449502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba4150510 con 0x7fcba40b4690 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcba956f640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcba40b3b70 0x7fcba40aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcbaa571640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba40b4690 0x7fcba414ffd0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00d620 tx=0x7fcb9c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcba27fc640 1 -- 172.21.15.81:0/393449502 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c017070 con 0x7fcba40b4690 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcba27fc640 1 -- 172.21.15.81:0/393449502 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb9c005310 con 0x7fcba40b4690 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcbabfd4640 1 -- 172.21.15.81:0/393449502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba4151950 con 0x7fcba40b4690 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcba27fc640 1 -- 172.21.15.81:0/393449502 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c0047a0 con 0x7fcba40b4690 2024-06-29T15:38:27.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.345+0000 7fcbabfd4640 1 -- 172.21.15.81:0/393449502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba4151f10 con 0x7fcba40b4690 2024-06-29T15:38:27.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.346+0000 7fcbabfd4640 1 -- 172.21.15.81:0/393449502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcba4072fe0 con 0x7fcba40b4690 2024-06-29T15:38:27.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.347+0000 7fcba27fc640 1 -- 172.21.15.81:0/393449502 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 89) ==== 98247+0+0 (secure 0 0 0) 0x7fcb9c004940 con 0x7fcba40b4690 2024-06-29T15:38:27.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.347+0000 7fcba27fc640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fcb90076360 0x7fcb90078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:27.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.348+0000 7fcba27fc640 1 -- 172.21.15.81:0/393449502 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 6685+0+0 (secure 0 0 0) 0x7fcb9c097c80 con 0x7fcba40b4690 2024-06-29T15:38:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.348+0000 7fcba9d70640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fcb90076360 0x7fcb90078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.348+0000 7fcba9d70640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fcb90076360 0x7fcb90078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcb94035000 tx=0x7fcb94004ab0 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:27.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.352+0000 7fcba27fc640 1 -- 172.21.15.81:0/393449502 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcb9c061080 con 0x7fcba40b4690 2024-06-29T15:38:27.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:27.566+0000 7fcbabfd4640 1 -- 172.21.15.81:0/393449502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fcba41516e0 con 0x7fcba40b4690 2024-06-29T15:38:28.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.212+0000 7fcba27fc640 1 -- 172.21.15.81:0/393449502 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v79) ==== 104+0+0 (secure 0 0 0) 0x7fcb9c060980 con 0x7fcba40b4690 2024-06-29T15:38:28.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:38:28.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 -- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fcb90076360 msgr2=0x7fcb90078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fcb90076360 0x7fcb90078870 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcb94035000 tx=0x7fcb94004ab0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 -- 172.21.15.81:0/393449502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba40b4690 msgr2=0x7fcba414ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba40b4690 0x7fcba414ffd0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00d620 tx=0x7fcb9c00dae0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 -- 172.21.15.81:0/393449502 shutdown_connections 2024-06-29T15:38:28.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fcb90076360 0x7fcb90078870 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba40b4690 0x7fcba414ffd0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcba40b3b70 0x7fcba40aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 --2- 172.21.15.81:0/393449502 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcba4072bd0 0x7fcba41529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 -- 172.21.15.81:0/393449502 >> 172.21.15.81:0/393449502 conn(0x7fcba40b1730 msgr2=0x7fcba40b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:28.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 -- 172.21.15.81:0/393449502 shutdown_connections 2024-06-29T15:38:28.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.214+0000 7fcbabfd4640 1 -- 172.21.15.81:0/393449502 wait complete. 2024-06-29T15:38:28.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:38:28.230 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:38:28.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.392+0000 7f6b8a074640 1 -- 172.21.15.81:0/3567679962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c004800 msgr2=0x7f6b6c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.392+0000 7f6b8a074640 1 --2- 172.21.15.81:0/3567679962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c004800 0x7f6b6c004c10 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009b50 tx=0x7f6b6c005830 comp rx=0 tx=0).stop 2024-06-29T15:38:28.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.392+0000 7f6b83fff640 1 -- 172.21.15.81:0/3567679962 reap_dead start 2024-06-29T15:38:28.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.392+0000 7f6b8a074640 1 -- 172.21.15.81:0/3567679962 shutdown_connections 2024-06-29T15:38:28.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.392+0000 7f6b8a074640 1 -- 172.21.15.81:0/3567679962 >> 172.21.15.81:0/3567679962 conn(0x7f6b84072970 msgr2=0x7f6b840b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:28.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.392+0000 7f6b8a074640 1 -- 172.21.15.81:0/3567679962 shutdown_connections 2024-06-29T15:38:28.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.392+0000 7f6b8a074640 1 -- 172.21.15.81:0/3567679962 wait complete. 2024-06-29T15:38:28.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.394+0000 7f6b8a074640 1 Processor -- start 2024-06-29T15:38:28.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.394+0000 7f6b8a074640 1 -- start start 2024-06-29T15:38:28.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b8a074640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c004800 0x7f6b840999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:28.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b8a074640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b840cf3a0 0x7f6b8415d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:28.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b8a074640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b840cf7b0 0x7f6b8415dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b8a074640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6b840af6f0 con 0x7f6b6c004800 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b8a074640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6b840af570 con 0x7f6b840cf7b0 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b8a074640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6b840af870 con 0x7f6b840cf3a0 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b837fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c004800 0x7f6b840999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b837fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c004800 0x7f6b840999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48178/0 (socket says 172.21.15.81:48178) 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b837fe640 1 -- 172.21.15.81:0/184874178 learned_addr learned my addr 172.21.15.81:0/184874178 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b83fff640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b840cf7b0 0x7f6b8415dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b82ffd640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b840cf3a0 0x7f6b8415d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b837fe640 1 -- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b840cf3a0 msgr2=0x7f6b8415d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b837fe640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b840cf3a0 0x7f6b8415d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b837fe640 1 -- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b840cf7b0 msgr2=0x7f6b8415dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b837fe640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b840cf7b0 0x7f6b8415dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b837fe640 1 -- 172.21.15.81:0/184874178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b840cdb50 con 0x7f6b6c004800 2024-06-29T15:38:28.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b837fe640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c004800 0x7f6b840999b0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f6b74011a80 tx=0x7f6b74011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:28.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b6bfff640 1 -- 172.21.15.81:0/184874178 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7400fd50 con 0x7f6b6c004800 2024-06-29T15:38:28.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.396+0000 7f6b6bfff640 1 -- 172.21.15.81:0/184874178 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b74016070 con 0x7f6b6c004800 2024-06-29T15:38:28.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.397+0000 7f6b8a074640 1 -- 172.21.15.81:0/184874178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b84099ef0 con 0x7f6b6c004800 2024-06-29T15:38:28.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.397+0000 7f6b6bfff640 1 -- 172.21.15.81:0/184874178 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b74012690 con 0x7f6b6c004800 2024-06-29T15:38:28.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.397+0000 7f6b8a074640 1 -- 172.21.15.81:0/184874178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b840cff80 con 0x7f6b6c004800 2024-06-29T15:38:28.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.397+0000 7f6b8a074640 1 -- 172.21.15.81:0/184874178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b8406d850 con 0x7f6b6c004800 2024-06-29T15:38:28.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.398+0000 7f6b6bfff640 1 -- 172.21.15.81:0/184874178 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 89) ==== 98247+0+0 (secure 0 0 0) 0x7f6b74018030 con 0x7f6b6c004800 2024-06-29T15:38:28.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.399+0000 7f6b6bfff640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6b70076360 0x7f6b70078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:28.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.399+0000 7f6b6bfff640 1 -- 172.21.15.81:0/184874178 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 6685+0+0 (secure 0 0 0) 0x7f6b74098280 con 0x7f6b6c004800 2024-06-29T15:38:28.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.399+0000 7f6b82ffd640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6b70076360 0x7f6b70078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:28.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.400+0000 7f6b82ffd640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6b70076360 0x7f6b70078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6b78035000 tx=0x7f6b7802d040 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:28.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.403+0000 7f6b6bfff640 1 -- 172.21.15.81:0/184874178 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6b84099ef0 con 0x7f6b6c004800 2024-06-29T15:38:28.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.583+0000 7f6b8a074640 1 -- 172.21.15.81:0/184874178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6b840b0080 con 0x7f6b6c004800 2024-06-29T15:38:28.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.914+0000 7f6b6bfff640 1 -- 172.21.15.81:0/184874178 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v159) ==== 181+0+0 (secure 0 0 0) 0x7f6b840cdb50 con 0x7f6b6c004800 2024-06-29T15:38:28.916 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:38:28.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 -- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6b70076360 msgr2=0x7f6b70078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6b70076360 0x7f6b70078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6b78035000 tx=0x7f6b7802d040 comp rx=0 tx=0).stop 2024-06-29T15:38:28.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 -- 172.21.15.81:0/184874178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c004800 msgr2=0x7f6b840999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:28.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c004800 0x7f6b840999b0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f6b74011a80 tx=0x7f6b74011e80 comp rx=0 tx=0).stop 2024-06-29T15:38:28.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 -- 172.21.15.81:0/184874178 shutdown_connections 2024-06-29T15:38:28.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6b70076360 0x7f6b70078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b840cf7b0 0x7f6b8415dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b840cf3a0 0x7f6b8415d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 --2- 172.21.15.81:0/184874178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c004800 0x7f6b840999b0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:28.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 -- 172.21.15.81:0/184874178 >> 172.21.15.81:0/184874178 conn(0x7f6b84072970 msgr2=0x7f6b8406e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:28.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 -- 172.21.15.81:0/184874178 shutdown_connections 2024-06-29T15:38:28.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:28.918+0000 7f6b8a074640 1 -- 172.21.15.81:0/184874178 wait complete. 2024-06-29T15:38:28.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:38:29.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.052+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2328753255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63406c830 msgr2=0x7ff6340cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.052+0000 7ff63a7b2640 1 --2- 172.21.15.81:0/2328753255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63406c830 0x7ff6340cfa60 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7ff62c005fd0 tx=0x7ff62c02aa90 comp rx=0 tx=0).stop 2024-06-29T15:38:29.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.052+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2328753255 shutdown_connections 2024-06-29T15:38:29.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.052+0000 7ff63a7b2640 1 --2- 172.21.15.81:0/2328753255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63406c830 0x7ff6340cfa60 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.052+0000 7ff63a7b2640 1 --2- 172.21.15.81:0/2328753255 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff63406f000 0x7ff63406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.052+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2328753255 >> 172.21.15.81:0/2328753255 conn(0x7ff6340d08f0 msgr2=0x7ff6340d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.052+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2328753255 shutdown_connections 2024-06-29T15:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.052+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2328753255 wait complete. 2024-06-29T15:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.054+0000 7ff63a7b2640 1 Processor -- start 2024-06-29T15:38:29.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.054+0000 7ff63a7b2640 1 -- start start 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.055+0000 7ff63a7b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63406c830 0x7ff634158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.055+0000 7ff63a7b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff63406f000 0x7ff634143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.055+0000 7ff63a7b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6340d1410 0x7ff634154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.055+0000 7ff63a7b2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff6340cffa0 con 0x7ff63406c830 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.055+0000 7ff63a7b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff634159e80 con 0x7ff6340d1410 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.055+0000 7ff63a7b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff6340d0120 con 0x7ff63406f000 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.055+0000 7ff633fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63406c830 0x7ff634158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.055+0000 7ff633fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63406c830 0x7ff634158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48200/0 (socket says 172.21.15.81:48200) 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.055+0000 7ff633fff640 1 -- 172.21.15.81:0/2101383322 learned_addr learned my addr 172.21.15.81:0/2101383322 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff633fff640 1 -- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff63406f000 msgr2=0x7ff634143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff633fff640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff63406f000 0x7ff634143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff633fff640 1 -- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6340d1410 msgr2=0x7ff634154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff633fff640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6340d1410 0x7ff634154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff633fff640 1 -- 172.21.15.81:0/2101383322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6341435b0 con 0x7ff63406c830 2024-06-29T15:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff633fff640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63406c830 0x7ff634158130 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7ff62400c3a0 tx=0x7ff62400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff630ff9640 1 -- 172.21.15.81:0/2101383322 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624017070 con 0x7ff63406c830 2024-06-29T15:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff630ff9640 1 -- 172.21.15.81:0/2101383322 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6240053c0 con 0x7ff63406c830 2024-06-29T15:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2101383322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff634158aa0 con 0x7ff63406c830 2024-06-29T15:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff630ff9640 1 -- 172.21.15.81:0/2101383322 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624003850 con 0x7ff63406c830 2024-06-29T15:38:29.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.056+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2101383322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff634159060 con 0x7ff63406c830 2024-06-29T15:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.057+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2101383322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff634075c90 con 0x7ff63406c830 2024-06-29T15:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.058+0000 7ff630ff9640 1 -- 172.21.15.81:0/2101383322 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 89) ==== 98247+0+0 (secure 0 0 0) 0x7ff62401a070 con 0x7ff63406c830 2024-06-29T15:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.058+0000 7ff630ff9640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff620076360 0x7ff620078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:29.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.058+0000 7ff630ff9640 1 -- 172.21.15.81:0/2101383322 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 6247+0+0 (secure 0 0 0) 0x7ff624096de0 con 0x7ff63406c830 2024-06-29T15:38:29.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.059+0000 7ff6337fe640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff620076360 0x7ff620078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:29.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.059+0000 7ff6337fe640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff620076360 0x7ff620078870 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff6280058c0 tx=0x7ff628005850 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:29.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.063+0000 7ff630ff9640 1 -- 172.21.15.81:0/2101383322 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff634075c90 con 0x7ff63406c830 2024-06-29T15:38:29.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.245+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2101383322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff6340d0640 con 0x7ff63406c830 2024-06-29T15:38:29.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.923+0000 7ff630ff9640 1 -- 172.21.15.81:0/2101383322 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v160) ==== 169+0+0 (secure 0 0 0) 0x7ff63406cc40 con 0x7ff63406c830 2024-06-29T15:38:29.925 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T15:38:29.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.926+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff620076360 msgr2=0x7ff620078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.926+0000 7ff63a7b2640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff620076360 0x7ff620078870 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff6280058c0 tx=0x7ff628005850 comp rx=0 tx=0).stop 2024-06-29T15:38:29.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.926+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2101383322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63406c830 msgr2=0x7ff634158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:29.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.926+0000 7ff63a7b2640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63406c830 0x7ff634158130 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7ff62400c3a0 tx=0x7ff62400c860 comp rx=0 tx=0).stop 2024-06-29T15:38:29.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.927+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2101383322 shutdown_connections 2024-06-29T15:38:29.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.927+0000 7ff63a7b2640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff620076360 0x7ff620078870 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.927+0000 7ff63a7b2640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6340d1410 0x7ff634154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.927+0000 7ff63a7b2640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff63406f000 0x7ff634143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:29.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.927+0000 7ff63a7b2640 1 --2- 172.21.15.81:0/2101383322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63406c830 0x7ff634158130 secure :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7ff62400c3a0 tx=0x7ff62400c860 comp rx=0 tx=0).stop 2024-06-29T15:38:29.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.928+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2101383322 >> 172.21.15.81:0/2101383322 conn(0x7ff6340d08f0 msgr2=0x7ff63415d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:29.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.929+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2101383322 shutdown_connections 2024-06-29T15:38:29.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:29.929+0000 7ff63a7b2640 1 -- 172.21.15.81:0/2101383322 wait complete. 2024-06-29T15:38:29.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:38:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.065+0000 7f0904a09640 1 -- 172.21.15.81:0/891373816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0900074520 msgr2=0x7f090006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.065+0000 7f0904a09640 1 --2- 172.21.15.81:0/891373816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0900074520 0x7f090006f140 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f0900073780 tx=0x7f08f002aac0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.065+0000 7f0904a09640 1 -- 172.21.15.81:0/891373816 shutdown_connections 2024-06-29T15:38:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.065+0000 7f0904a09640 1 --2- 172.21.15.81:0/891373816 >> v2:172.21.15.167:3300/0 conn(0x7f090006fa90 0x7f090006ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.065+0000 7f0904a09640 1 --2- 172.21.15.81:0/891373816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0900074520 0x7f090006f140 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.065+0000 7f0904a09640 1 -- 172.21.15.81:0/891373816 >> 172.21.15.81:0/891373816 conn(0x7f0900072950 msgr2=0x7f09000b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.065+0000 7f0904a09640 1 -- 172.21.15.81:0/891373816 shutdown_connections 2024-06-29T15:38:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.065+0000 7f0904a09640 1 -- 172.21.15.81:0/891373816 wait complete. 2024-06-29T15:38:30.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.067+0000 7f0904a09640 1 Processor -- start 2024-06-29T15:38:30.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.067+0000 7f0904a09640 1 -- start start 2024-06-29T15:38:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f0904a09640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f090006f680 0x7f090009dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f0904a09640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f090006fa90 0x7f0900156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f0904a09640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0900074520 0x7f09000ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f0904a09640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f090015b010 con 0x7f090006f680 2024-06-29T15:38:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f0904a09640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f090006d150 con 0x7f0900074520 2024-06-29T15:38:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f0904a09640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f090015b190 con 0x7f090006fa90 2024-06-29T15:38:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fe59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f090006f680 0x7f090009dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fe59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f090006f680 0x7f090009dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35010/0 (socket says 172.21.15.81:35010) 2024-06-29T15:38:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fe59c640 1 -- 172.21.15.81:0/1887301075 learned_addr learned my addr 172.21.15.81:0/1887301075 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:30.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fed9d640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0900074520 0x7f09000ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fdd9b640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f090006fa90 0x7f0900156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fe59c640 1 -- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f090006fa90 msgr2=0x7f0900156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fe59c640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f090006fa90 0x7f0900156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fe59c640 1 -- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0900074520 msgr2=0x7f09000ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fe59c640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0900074520 0x7f09000ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fe59c640 1 -- 172.21.15.81:0/1887301075 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090009cb40 con 0x7f090006f680 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.068+0000 7f08fdd9b640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f090006fa90 0x7f0900156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.069+0000 7f08fe59c640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f090006f680 0x7f090009dbe0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f08f0000f80 tx=0x7f08f0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.069+0000 7f08eeffd640 1 -- 172.21.15.81:0/1887301075 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f0044070 con 0x7f090006f680 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.069+0000 7f08eeffd640 1 -- 172.21.15.81:0/1887301075 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08f0004310 con 0x7f090006f680 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.069+0000 7f0904a09640 1 -- 172.21.15.81:0/1887301075 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090009ce20 con 0x7f090006f680 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.069+0000 7f08eeffd640 1 -- 172.21.15.81:0/1887301075 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f0004740 con 0x7f090006f680 2024-06-29T15:38:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.069+0000 7f0904a09640 1 -- 172.21.15.81:0/1887301075 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090009d3e0 con 0x7f090006f680 2024-06-29T15:38:30.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.070+0000 7f0904a09640 1 -- 172.21.15.81:0/1887301075 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09000af480 con 0x7f090006f680 2024-06-29T15:38:30.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.071+0000 7f08eeffd640 1 -- 172.21.15.81:0/1887301075 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f08f0036400 con 0x7f090006f680 2024-06-29T15:38:30.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.071+0000 7f08eeffd640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f08e00763f0 0x7f08e0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.071+0000 7f08eeffd640 1 -- 172.21.15.81:0/1887301075 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5731+0+0 (secure 0 0 0) 0x7f08f00b82d0 con 0x7f090006f680 2024-06-29T15:38:30.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.071+0000 7f08fdd9b640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f08e00763f0 0x7f08e0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.072+0000 7f08fdd9b640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f08e00763f0 0x7f08e0078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f09001542f0 tx=0x7f08f4009040 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:30.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.076+0000 7f08eeffd640 1 -- 172.21.15.81:0/1887301075 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f08f0032030 con 0x7f090006f680 2024-06-29T15:38:30.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.258+0000 7f0904a09640 1 -- 172.21.15.81:0/1887301075 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f09000d0b70 con 0x7f090006f680 2024-06-29T15:38:30.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.262+0000 7f08eeffd640 1 -- 172.21.15.81:0/1887301075 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v59) ==== 71+0+0 (secure 0 0 0) 0x7f08f00818e0 con 0x7f090006f680 2024-06-29T15:38:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.264+0000 7f0904a09640 1 -- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f08e00763f0 msgr2=0x7f08e0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.264+0000 7f0904a09640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f08e00763f0 0x7f08e0078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f09001542f0 tx=0x7f08f4009040 comp rx=0 tx=0).stop 2024-06-29T15:38:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.264+0000 7f0904a09640 1 -- 172.21.15.81:0/1887301075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f090006f680 msgr2=0x7f090009dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.264+0000 7f0904a09640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f090006f680 0x7f090009dbe0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f08f0000f80 tx=0x7f08f0003f10 comp rx=0 tx=0).stop 2024-06-29T15:38:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.264+0000 7f0904a09640 1 -- 172.21.15.81:0/1887301075 shutdown_connections 2024-06-29T15:38:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.264+0000 7f0904a09640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f08e00763f0 0x7f08e0078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.264+0000 7f0904a09640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0900074520 0x7f09000ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.264+0000 7f0904a09640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f090006fa90 0x7f0900156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.264+0000 7f0904a09640 1 --2- 172.21.15.81:0/1887301075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f090006f680 0x7f090009dbe0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.264+0000 7f0904a09640 1 -- 172.21.15.81:0/1887301075 >> 172.21.15.81:0/1887301075 conn(0x7f0900072950 msgr2=0x7f09000b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:30.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.265+0000 7f0904a09640 1 -- 172.21.15.81:0/1887301075 shutdown_connections 2024-06-29T15:38:30.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.265+0000 7f0904a09640 1 -- 172.21.15.81:0/1887301075 wait complete. 2024-06-29T15:38:30.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2024-06-29T15:38:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.389+0000 7f29c648f640 1 -- 172.21.15.81:0/3219572877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29c00b4430 msgr2=0x7f29c00cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.389+0000 7f29c648f640 1 --2- 172.21.15.81:0/3219572877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29c00b4430 0x7f29c00cf130 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f29c015aac0 tx=0x7f29b402a4d0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.389+0000 7f29c648f640 1 -- 172.21.15.81:0/3219572877 shutdown_connections 2024-06-29T15:38:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.389+0000 7f29c648f640 1 --2- 172.21.15.81:0/3219572877 >> v2:172.21.15.167:3300/0 conn(0x7f29c006f660 0x7f29c006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.389+0000 7f29c648f640 1 --2- 172.21.15.81:0/3219572877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29c00b4430 0x7f29c00cf130 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.389+0000 7f29c648f640 1 --2- 172.21.15.81:0/3219572877 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29c00b35b0 0x7f29c015b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.389+0000 7f29c648f640 1 -- 172.21.15.81:0/3219572877 >> 172.21.15.81:0/3219572877 conn(0x7f29c006ccc0 msgr2=0x7f29c00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:30.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.389+0000 7f29c648f640 1 -- 172.21.15.81:0/3219572877 shutdown_connections 2024-06-29T15:38:30.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.390+0000 7f29c648f640 1 -- 172.21.15.81:0/3219572877 wait complete. 2024-06-29T15:38:30.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.391+0000 7f29c648f640 1 Processor -- start 2024-06-29T15:38:30.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.391+0000 7f29c648f640 1 -- start start 2024-06-29T15:38:30.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29c648f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29c006f660 0x7f29c01f8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29c648f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29c00b35b0 0x7f29c014e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29c648f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29c00b4430 0x7f29c015ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29c648f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f29c00722b0 con 0x7f29c006f660 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29c648f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f29c0072130 con 0x7f29c00b35b0 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29c648f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f29c0072430 con 0x7f29c00b4430 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29c006f660 0x7f29c01f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29c006f660 0x7f29c01f8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35030/0 (socket says 172.21.15.81:35030) 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bffff640 1 -- 172.21.15.81:0/4151585920 learned_addr learned my addr 172.21.15.81:0/4151585920 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bf7fe640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29c00b35b0 0x7f29c014e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29c4a2c640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29c00b4430 0x7f29c015ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bffff640 1 -- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29c00b4430 msgr2=0x7f29c015ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bffff640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29c00b4430 0x7f29c015ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bffff640 1 -- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29c00b35b0 msgr2=0x7f29c014e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:30.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bffff640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29c00b35b0 0x7f29c014e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:30.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bffff640 1 -- 172.21.15.81:0/4151585920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c014f310 con 0x7f29c006f660 2024-06-29T15:38:30.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bffff640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29c006f660 0x7f29c01f8260 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f29b000b410 tx=0x7f29b000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:30.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bcff9640 1 -- 172.21.15.81:0/4151585920 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0015020 con 0x7f29c006f660 2024-06-29T15:38:30.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.393+0000 7f29bcff9640 1 -- 172.21.15.81:0/4151585920 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29b0011940 con 0x7f29c006f660 2024-06-29T15:38:30.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.394+0000 7f29c648f640 1 -- 172.21.15.81:0/4151585920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c014f5f0 con 0x7f29c006f660 2024-06-29T15:38:30.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.394+0000 7f29bcff9640 1 -- 172.21.15.81:0/4151585920 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0011d90 con 0x7f29c006f660 2024-06-29T15:38:30.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.394+0000 7f29c648f640 1 -- 172.21.15.81:0/4151585920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c014fbb0 con 0x7f29c006f660 2024-06-29T15:38:30.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.394+0000 7f29c648f640 1 -- 172.21.15.81:0/4151585920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c0072a40 con 0x7f29c006f660 2024-06-29T15:38:30.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.395+0000 7f29bcff9640 1 -- 172.21.15.81:0/4151585920 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f29b0018020 con 0x7f29c006f660 2024-06-29T15:38:30.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.396+0000 7f29bcff9640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f29ac076420 0x7f29ac078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:30.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.396+0000 7f29bcff9640 1 -- 172.21.15.81:0/4151585920 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5731+0+0 (secure 0 0 0) 0x7f29b00978c0 con 0x7f29c006f660 2024-06-29T15:38:30.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.396+0000 7f29bf7fe640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f29ac076420 0x7f29ac078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:30.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.397+0000 7f29bf7fe640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f29ac076420 0x7f29ac078930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f29b4033000 tx=0x7f29b4004730 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:30.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.400+0000 7f29bcff9640 1 -- 172.21.15.81:0/4151585920 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f29b001f020 con 0x7f29c006f660 2024-06-29T15:38:30.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:30.647+0000 7f29c648f640 1 -- 172.21.15.81:0/4151585920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) -- 0x7f29c006a5b0 con 0x7f29c006f660 2024-06-29T15:38:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.303+0000 7f29bcff9640 1 -- 172.21.15.81:0/4151585920 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2456) ==== 178+0+0 (secure 0 0 0) 0x7f29b0060990 con 0x7f29c006f660 2024-06-29T15:38:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 -- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f29ac076420 msgr2=0x7f29ac078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f29ac076420 0x7f29ac078930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f29b4033000 tx=0x7f29b4004730 comp rx=0 tx=0).stop 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 -- 172.21.15.81:0/4151585920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29c006f660 msgr2=0x7f29c01f8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29c006f660 0x7f29c01f8260 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f29b000b410 tx=0x7f29b000b8d0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 -- 172.21.15.81:0/4151585920 shutdown_connections 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f29ac076420 0x7f29ac078930 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29c00b4430 0x7f29c015ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29c00b35b0 0x7f29c014e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 --2- 172.21.15.81:0/4151585920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29c006f660 0x7f29c01f8260 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 -- 172.21.15.81:0/4151585920 >> 172.21.15.81:0/4151585920 conn(0x7f29c006ccc0 msgr2=0x7f29c014c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 -- 172.21.15.81:0/4151585920 shutdown_connections 2024-06-29T15:38:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.305+0000 7f29c648f640 1 -- 172.21.15.81:0/4151585920 wait complete. 2024-06-29T15:38:31.319 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2024-06-29T15:38:31.319 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2024-06-29T15:38:31.320 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T15:38:31.320 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:31.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.435+0000 7f574b9d4640 1 -- 172.21.15.81:0/3085803451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5730003a50 msgr2=0x7f5730003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.435+0000 7f574b9d4640 1 --2- 172.21.15.81:0/3085803451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5730003a50 0x7f5730003ec0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f573c009ed0 tx=0x7f5730005830 comp rx=0 tx=0).stop 2024-06-29T15:38:31.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.435+0000 7f574b9d4640 1 -- 172.21.15.81:0/3085803451 shutdown_connections 2024-06-29T15:38:31.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.435+0000 7f574b9d4640 1 --2- 172.21.15.81:0/3085803451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5730003a50 0x7f5730003ec0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.435+0000 7f574b9d4640 1 -- 172.21.15.81:0/3085803451 >> 172.21.15.81:0/3085803451 conn(0x7f574409cf30 msgr2=0x7f57440d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.435+0000 7f574b9d4640 1 -- 172.21.15.81:0/3085803451 shutdown_connections 2024-06-29T15:38:31.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.435+0000 7f574b9d4640 1 -- 172.21.15.81:0/3085803451 wait complete. 2024-06-29T15:38:31.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.437+0000 7f574b9d4640 1 Processor -- start 2024-06-29T15:38:31.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.437+0000 7f574b9d4640 1 -- start start 2024-06-29T15:38:31.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f574b9d4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5730003a50 0x7f57440a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f574b9d4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57440b0f10 0x7f57440d1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f574b9d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57440d21d0 0x7f57440b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f574b9d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f57440d0810 con 0x7f57440d21d0 2024-06-29T15:38:31.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f574b9d4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f57440d0690 con 0x7f5730003a50 2024-06-29T15:38:31.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f574b9d4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f57440d0990 con 0x7f57440b0f10 2024-06-29T15:38:31.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f5749f71640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57440d21d0 0x7f57440b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f5749f71640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57440d21d0 0x7f57440b2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35042/0 (socket says 172.21.15.81:35042) 2024-06-29T15:38:31.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f5749f71640 1 -- 172.21.15.81:0/1195256037 learned_addr learned my addr 172.21.15.81:0/1195256037 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:31.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f5749770640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5730003a50 0x7f57440a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.438+0000 7f5748f6f640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57440b0f10 0x7f57440d1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f5749f71640 1 -- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57440b0f10 msgr2=0x7f57440d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f5749f71640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57440b0f10 0x7f57440d1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f5749f71640 1 -- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5730003a50 msgr2=0x7f57440a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f5749f71640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5730003a50 0x7f57440a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f5749f71640 1 -- 172.21.15.81:0/1195256037 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57440d25e0 con 0x7f57440d21d0 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f5749f71640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57440d21d0 0x7f57440b2ef0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f57440d4460 tx=0x7f574000ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f5739ffb640 1 -- 172.21.15.81:0/1195256037 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740010890 con 0x7f57440d21d0 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f5739ffb640 1 -- 172.21.15.81:0/1195256037 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5740016070 con 0x7f57440d21d0 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f574b9d4640 1 -- 172.21.15.81:0/1195256037 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57441f4ad0 con 0x7f57440d21d0 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f5739ffb640 1 -- 172.21.15.81:0/1195256037 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740010de0 con 0x7f57440d21d0 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.439+0000 7f574b9d4640 1 -- 172.21.15.81:0/1195256037 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57441f5090 con 0x7f57440d21d0 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.440+0000 7f574b9d4640 1 -- 172.21.15.81:0/1195256037 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57440d1a80 con 0x7f57440d21d0 2024-06-29T15:38:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.441+0000 7f5739ffb640 1 -- 172.21.15.81:0/1195256037 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f574001b460 con 0x7f57440d21d0 2024-06-29T15:38:31.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.441+0000 7f5739ffb640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f57280763f0 0x7f5728078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.441+0000 7f5739ffb640 1 -- 172.21.15.81:0/1195256037 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5731+0+0 (secure 0 0 0) 0x7f5740097df0 con 0x7f57440d21d0 2024-06-29T15:38:31.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.442+0000 7f5749770640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f57280763f0 0x7f5728078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.442+0000 7f5749770640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f57280763f0 0x7f5728078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f573c035000 tx=0x7f573c004a40 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:31.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.446+0000 7f5739ffb640 1 -- 172.21.15.81:0/1195256037 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f57440d25e0 con 0x7f57440d21d0 2024-06-29T15:38:31.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.628+0000 7f574b9d4640 1 -- 172.21.15.81:0/1195256037 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5744069230 con 0x7f57440d21d0 2024-06-29T15:38:31.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.628+0000 7f5739ffb640 1 -- 172.21.15.81:0/1195256037 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) ==== 74+0+14195 (secure 0 0 0) 0x7f5740061510 con 0x7f57440d21d0 2024-06-29T15:38:31.630 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:31.630 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":160,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:29.916489+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6836/4254103476":"2024-06-30T15:38:27.208048+0000","172.21.15.167:6834/4254103476":"2024-06-30T15:38:27.208048+0000","172.21.15.167:0/3122315648":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/3189551239":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/4281804658":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/2936900238":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/386189888":"2024-06-30T15:38:25.044444+0000","172.21.15.167:6833/828186388":"2024-06-30T15:38:25.044444+0000","172.21.15.167:0/1730106798":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/2472315005":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3327921477":"2024-06-30T15:38:25.044444+0000","172.21.15.167:6833/3201324170":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/2674867295":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/588266296":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3014230071":"2024-06-30T14:42:10.211800+0000","172.21.15.167:6832/828186388":"2024-06-30T15:38:25.044444+0000","172.21.15.167:6832/3201324170":"2024-06-30T14:42:10.211800+0000","172.21.15.167:0/3312315666":"2024-06-30T14:42:10.211800+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:31.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 -- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f57280763f0 msgr2=0x7f5728078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f57280763f0 0x7f5728078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f573c035000 tx=0x7f573c004a40 comp rx=0 tx=0).stop 2024-06-29T15:38:31.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 -- 172.21.15.81:0/1195256037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57440d21d0 msgr2=0x7f57440b2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57440d21d0 0x7f57440b2ef0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f57440d4460 tx=0x7f574000ea20 comp rx=0 tx=0).stop 2024-06-29T15:38:31.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 -- 172.21.15.81:0/1195256037 shutdown_connections 2024-06-29T15:38:31.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f57280763f0 0x7f5728078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57440d21d0 0x7f57440b2ef0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57440b0f10 0x7f57440d1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 --2- 172.21.15.81:0/1195256037 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5730003a50 0x7f57440a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 -- 172.21.15.81:0/1195256037 >> 172.21.15.81:0/1195256037 conn(0x7f574409cf30 msgr2=0x7f5744068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 -- 172.21.15.81:0/1195256037 shutdown_connections 2024-06-29T15:38:31.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.631+0000 7f574b9d4640 1 -- 172.21.15.81:0/1195256037 wait complete. 2024-06-29T15:38:31.644 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:31.644 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:31.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.759+0000 7f185a4c1640 1 -- 172.21.15.81:0/3983830853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1854145530 msgr2=0x7f1854073280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.759+0000 7f185a4c1640 1 --2- 172.21.15.81:0/3983830853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1854145530 0x7f1854073280 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f1844005fd0 tx=0x7f184402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.759+0000 7f185a4c1640 1 -- 172.21.15.81:0/3983830853 shutdown_connections 2024-06-29T15:38:31.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.759+0000 7f185a4c1640 1 --2- 172.21.15.81:0/3983830853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18540737c0 0x7f185415c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.759+0000 7f185a4c1640 1 --2- 172.21.15.81:0/3983830853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1854145530 0x7f1854073280 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.759+0000 7f185a4c1640 1 -- 172.21.15.81:0/3983830853 >> 172.21.15.81:0/3983830853 conn(0x7f185406ccc0 msgr2=0x7f18540b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.759+0000 7f185a4c1640 1 -- 172.21.15.81:0/3983830853 shutdown_connections 2024-06-29T15:38:31.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.759+0000 7f185a4c1640 1 -- 172.21.15.81:0/3983830853 wait complete. 2024-06-29T15:38:31.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.761+0000 7f185a4c1640 1 Processor -- start 2024-06-29T15:38:31.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.761+0000 7f185a4c1640 1 -- start start 2024-06-29T15:38:31.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.762+0000 7f185a4c1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18540737c0 0x7f1854155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.762+0000 7f185a4c1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1854145530 0x7f1854157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.762+0000 7f185a4c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f185415cb30 0x7f1854158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.762+0000 7f185a4c1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f185415d0c0 con 0x7f185415cb30 2024-06-29T15:38:31.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.762+0000 7f185a4c1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f185415cf40 con 0x7f1854145530 2024-06-29T15:38:31.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.762+0000 7f185a4c1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1854160050 con 0x7f18540737c0 2024-06-29T15:38:31.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.762+0000 7f1858a5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f185415cb30 0x7f1854158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.762+0000 7f1858a5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f185415cb30 0x7f1854158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35058/0 (socket says 172.21.15.81:35058) 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.762+0000 7f1858a5e640 1 -- 172.21.15.81:0/1970314584 learned_addr learned my addr 172.21.15.81:0/1970314584 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1853fff640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18540737c0 0x7f1854155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f18537fe640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1854145530 0x7f1854157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1858a5e640 1 -- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18540737c0 msgr2=0x7f1854155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1858a5e640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18540737c0 0x7f1854155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1858a5e640 1 -- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1854145530 msgr2=0x7f1854157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1858a5e640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1854145530 0x7f1854157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1858a5e640 1 -- 172.21.15.81:0/1970314584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185413e2f0 con 0x7f185415cb30 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1853fff640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18540737c0 0x7f1854155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:31.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1858a5e640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f185415cb30 0x7f1854158030 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f184c00c700 tx=0x7f184c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:31.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1850ff9640 1 -- 172.21.15.81:0/1970314584 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c003b70 con 0x7f185415cb30 2024-06-29T15:38:31.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1850ff9640 1 -- 172.21.15.81:0/1970314584 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f184c003d10 con 0x7f185415cb30 2024-06-29T15:38:31.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f1850ff9640 1 -- 172.21.15.81:0/1970314584 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c00aab0 con 0x7f185415cb30 2024-06-29T15:38:31.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f185a4c1640 1 -- 172.21.15.81:0/1970314584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1854156d30 con 0x7f185415cb30 2024-06-29T15:38:31.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.763+0000 7f185a4c1640 1 -- 172.21.15.81:0/1970314584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18541572f0 con 0x7f185415cb30 2024-06-29T15:38:31.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.764+0000 7f185a4c1640 1 -- 172.21.15.81:0/1970314584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1854160500 con 0x7f185415cb30 2024-06-29T15:38:31.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.765+0000 7f1850ff9640 1 -- 172.21.15.81:0/1970314584 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f184c00ac50 con 0x7f185415cb30 2024-06-29T15:38:31.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.765+0000 7f1850ff9640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1840076420 0x7f1840078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:31.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.765+0000 7f1850ff9640 1 -- 172.21.15.81:0/1970314584 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5731+0+0 (secure 0 0 0) 0x7f184c0976a0 con 0x7f185415cb30 2024-06-29T15:38:31.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.766+0000 7f1853fff640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1840076420 0x7f1840078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:31.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.766+0000 7f1853fff640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1840076420 0x7f1840078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1844033000 tx=0x7f1844004730 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:31.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.770+0000 7f1850ff9640 1 -- 172.21.15.81:0/1970314584 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1854073bd0 con 0x7f185415cb30 2024-06-29T15:38:31.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.951+0000 7f185a4c1640 1 -- 172.21.15.81:0/1970314584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f18540affc0 con 0x7f185415cb30 2024-06-29T15:38:31.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.952+0000 7f1850ff9640 1 -- 172.21.15.81:0/1970314584 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v160) ==== 93+0+10 (secure 0 0 0) 0x7f184c060e40 con 0x7f185415cb30 2024-06-29T15:38:31.953 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:38:31.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 -- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1840076420 msgr2=0x7f1840078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1840076420 0x7f1840078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1844033000 tx=0x7f1844004730 comp rx=0 tx=0).stop 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 -- 172.21.15.81:0/1970314584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f185415cb30 msgr2=0x7f1854158030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f185415cb30 0x7f1854158030 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f184c00c700 tx=0x7f184c00cbc0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 -- 172.21.15.81:0/1970314584 shutdown_connections 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1840076420 0x7f1840078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f185415cb30 0x7f1854158030 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1854145530 0x7f1854157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 --2- 172.21.15.81:0/1970314584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18540737c0 0x7f1854155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 -- 172.21.15.81:0/1970314584 >> 172.21.15.81:0/1970314584 conn(0x7f185406ccc0 msgr2=0x7f18540b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 -- 172.21.15.81:0/1970314584 shutdown_connections 2024-06-29T15:38:31.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:31.954+0000 7f185a4c1640 1 -- 172.21.15.81:0/1970314584 wait complete. 2024-06-29T15:38:31.967 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-06-29T15:38:32.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.079+0000 7fdd92768640 1 -- 172.21.15.81:0/1714272349 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd8c073480 msgr2=0x7fdd8c15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:32.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.079+0000 7fdd92768640 1 --2- 172.21.15.81:0/1714272349 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd8c073480 0x7fdd8c15b6e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fdd80005fd0 tx=0x7fdd8002a540 comp rx=0 tx=0).stop 2024-06-29T15:38:32.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.079+0000 7fdd92768640 1 -- 172.21.15.81:0/1714272349 shutdown_connections 2024-06-29T15:38:32.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.079+0000 7fdd92768640 1 --2- 172.21.15.81:0/1714272349 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd8c073480 0x7fdd8c15b6e0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:32.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.079+0000 7fdd92768640 1 --2- 172.21.15.81:0/1714272349 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd8c0cd150 0x7fdd8c072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:32.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.079+0000 7fdd92768640 1 -- 172.21.15.81:0/1714272349 >> 172.21.15.81:0/1714272349 conn(0x7fdd8c0b2920 msgr2=0x7fdd8c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:32.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.079+0000 7fdd92768640 1 -- 172.21.15.81:0/1714272349 shutdown_connections 2024-06-29T15:38:32.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.079+0000 7fdd92768640 1 -- 172.21.15.81:0/1714272349 wait complete. 2024-06-29T15:38:32.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.080+0000 7fdd92768640 1 Processor -- start 2024-06-29T15:38:32.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.081+0000 7fdd92768640 1 -- start start 2024-06-29T15:38:32.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd92768640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd8c073480 0x7fdd8c14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:32.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd92768640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd8c0cd150 0x7fdd8c14b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:32.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd92768640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd8c15bc20 0x7fdd8c0cf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:32.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd92768640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8c15c1b0 con 0x7fdd8c073480 2024-06-29T15:38:32.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd92768640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8c15c030 con 0x7fdd8c15bc20 2024-06-29T15:38:32.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd92768640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8c15f060 con 0x7fdd8c0cd150 2024-06-29T15:38:32.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd8bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd8c073480 0x7fdd8c14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:32.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd8bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd8c073480 0x7fdd8c14a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35066/0 (socket says 172.21.15.81:35066) 2024-06-29T15:38:32.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd8bfff640 1 -- 172.21.15.81:0/2233658364 learned_addr learned my addr 172.21.15.81:0/2233658364 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:32.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd8b7fe640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd8c0cd150 0x7fdd8c14b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd90d05640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd8c15bc20 0x7fdd8c0cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd8bfff640 1 -- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd8c0cd150 msgr2=0x7fdd8c14b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd8bfff640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd8c0cd150 0x7fdd8c14b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd8bfff640 1 -- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd8c15bc20 msgr2=0x7fdd8c0cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.082+0000 7fdd8bfff640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd8c15bc20 0x7fdd8c0cf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.083+0000 7fdd8bfff640 1 -- 172.21.15.81:0/2233658364 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c0cffa0 con 0x7fdd8c073480 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.083+0000 7fdd8bfff640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd8c073480 0x7fdd8c14a780 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c00e9f0 tx=0x7fdd7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.083+0000 7fdd88ff9640 1 -- 172.21.15.81:0/2233658364 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c00cd20 con 0x7fdd8c073480 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.083+0000 7fdd88ff9640 1 -- 172.21.15.81:0/2233658364 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd7c004020 con 0x7fdd8c073480 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.083+0000 7fdd88ff9640 1 -- 172.21.15.81:0/2233658364 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c004470 con 0x7fdd8c073480 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.083+0000 7fdd92768640 1 -- 172.21.15.81:0/2233658364 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8c158820 con 0x7fdd8c073480 2024-06-29T15:38:32.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.083+0000 7fdd92768640 1 -- 172.21.15.81:0/2233658364 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd8c14bb90 con 0x7fdd8c073480 2024-06-29T15:38:32.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.084+0000 7fdd92768640 1 -- 172.21.15.81:0/2233658364 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd8c14b9c0 con 0x7fdd8c073480 2024-06-29T15:38:32.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.085+0000 7fdd88ff9640 1 -- 172.21.15.81:0/2233658364 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fdd7c0035f0 con 0x7fdd8c073480 2024-06-29T15:38:32.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.085+0000 7fdd88ff9640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fdd780763f0 0x7fdd78078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:32.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.085+0000 7fdd88ff9640 1 -- 172.21.15.81:0/2233658364 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5731+0+0 (secure 0 0 0) 0x7fdd7c0981f0 con 0x7fdd8c073480 2024-06-29T15:38:32.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.086+0000 7fdd8b7fe640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fdd780763f0 0x7fdd78078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:32.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.086+0000 7fdd8b7fe640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fdd780763f0 0x7fdd78078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd80033000 tx=0x7fdd80004730 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:32.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.090+0000 7fdd88ff9640 1 -- 172.21.15.81:0/2233658364 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdd7c061230 con 0x7fdd8c073480 2024-06-29T15:38:32.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:32.340+0000 7fdd92768640 1 -- 172.21.15.81:0/2233658364 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) -- 0x7fdd8c15f580 con 0x7fdd8c073480 2024-06-29T15:38:33.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.313+0000 7fdd88ff9640 1 -- 172.21.15.81:0/2233658364 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2458) ==== 177+0+0 (secure 0 0 0) 0x7fdd7c065530 con 0x7fdd8c073480 2024-06-29T15:38:33.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.316+0000 7fdd92768640 1 -- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fdd780763f0 msgr2=0x7fdd78078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.316+0000 7fdd92768640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fdd780763f0 0x7fdd78078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd80033000 tx=0x7fdd80004730 comp rx=0 tx=0).stop 2024-06-29T15:38:33.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.316+0000 7fdd92768640 1 -- 172.21.15.81:0/2233658364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd8c073480 msgr2=0x7fdd8c14a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.316+0000 7fdd92768640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd8c073480 0x7fdd8c14a780 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c00e9f0 tx=0x7fdd7c00eeb0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.317+0000 7fdd92768640 1 -- 172.21.15.81:0/2233658364 shutdown_connections 2024-06-29T15:38:33.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.317+0000 7fdd92768640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fdd780763f0 0x7fdd78078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.317+0000 7fdd92768640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd8c15bc20 0x7fdd8c0cf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.317+0000 7fdd92768640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd8c0cd150 0x7fdd8c14b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.317+0000 7fdd92768640 1 --2- 172.21.15.81:0/2233658364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd8c073480 0x7fdd8c14a780 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.317+0000 7fdd92768640 1 -- 172.21.15.81:0/2233658364 >> 172.21.15.81:0/2233658364 conn(0x7fdd8c0b2920 msgr2=0x7fdd8c0709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:33.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.317+0000 7fdd92768640 1 -- 172.21.15.81:0/2233658364 shutdown_connections 2024-06-29T15:38:33.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.317+0000 7fdd92768640 1 -- 172.21.15.81:0/2233658364 wait complete. 2024-06-29T15:38:33.332 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:38:33.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.450+0000 7f8c74d11640 1 -- 172.21.15.81:0/3173566606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c58003a30 msgr2=0x7f8c58003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.450+0000 7f8c74d11640 1 --2- 172.21.15.81:0/3173566606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c58003a30 0x7f8c58003ea0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f8c600016c0 tx=0x7f8c58005830 comp rx=0 tx=0).stop 2024-06-29T15:38:33.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.450+0000 7f8c6ed9d640 1 -- 172.21.15.81:0/3173566606 reap_dead start 2024-06-29T15:38:33.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.450+0000 7f8c74d11640 1 -- 172.21.15.81:0/3173566606 shutdown_connections 2024-06-29T15:38:33.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.450+0000 7f8c74d11640 1 -- 172.21.15.81:0/3173566606 >> 172.21.15.81:0/3173566606 conn(0x7f8c700b3510 msgr2=0x7f8c700b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:33.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.450+0000 7f8c74d11640 1 -- 172.21.15.81:0/3173566606 shutdown_connections 2024-06-29T15:38:33.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.450+0000 7f8c74d11640 1 -- 172.21.15.81:0/3173566606 wait complete. 2024-06-29T15:38:33.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.452+0000 7f8c74d11640 1 Processor -- start 2024-06-29T15:38:33.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.452+0000 7f8c74d11640 1 -- start start 2024-06-29T15:38:33.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c74d11640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c58003a30 0x7f8c701f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c74d11640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c701f38c0 0x7f8c701dfc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c74d11640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c7009c0f0 0x7f8c7009ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c74d11640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c700ce8a0 con 0x7f8c701f38c0 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c74d11640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8c700ce720 con 0x7f8c7009c0f0 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c74d11640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8c700b4f20 con 0x7f8c58003a30 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c701f38c0 0x7f8c701dfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c701f38c0 0x7f8c701dfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35088/0 (socket says 172.21.15.81:35088) 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6dd9b640 1 -- 172.21.15.81:0/924761298 learned_addr learned my addr 172.21.15.81:0/924761298 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6e59c640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c58003a30 0x7f8c701f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6ed9d640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c7009c0f0 0x7f8c7009ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6dd9b640 1 -- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c58003a30 msgr2=0x7f8c701f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6dd9b640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c58003a30 0x7f8c701f4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6dd9b640 1 -- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c7009c0f0 msgr2=0x7f8c7009ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6dd9b640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c7009c0f0 0x7f8c7009ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6dd9b640 1 -- 172.21.15.81:0/924761298 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7009dbf0 con 0x7f8c701f38c0 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6ed9d640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c7009c0f0 0x7f8c7009ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c6dd9b640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c701f38c0 0x7f8c701dfc30 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f8c64011a80 tx=0x7f8c64011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:33.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c5effd640 1 -- 172.21.15.81:0/924761298 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6400fd50 con 0x7f8c701f38c0 2024-06-29T15:38:33.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c5effd640 1 -- 172.21.15.81:0/924761298 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c64016070 con 0x7f8c701f38c0 2024-06-29T15:38:33.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c74d11640 1 -- 172.21.15.81:0/924761298 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7009ded0 con 0x7f8c701f38c0 2024-06-29T15:38:33.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c5effd640 1 -- 172.21.15.81:0/924761298 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c640126b0 con 0x7f8c701f38c0 2024-06-29T15:38:33.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.454+0000 7f8c74d11640 1 -- 172.21.15.81:0/924761298 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c7009e490 con 0x7f8c701f38c0 2024-06-29T15:38:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.456+0000 7f8c74d11640 1 -- 172.21.15.81:0/924761298 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c701f3da0 con 0x7f8c701f38c0 2024-06-29T15:38:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.456+0000 7f8c5effd640 1 -- 172.21.15.81:0/924761298 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f8c64018030 con 0x7f8c701f38c0 2024-06-29T15:38:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.457+0000 7f8c5effd640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8c500763f0 0x7f8c50078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.457+0000 7f8c5effd640 1 -- 172.21.15.81:0/924761298 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5731+0+0 (secure 0 0 0) 0x7f8c64098130 con 0x7f8c701f38c0 2024-06-29T15:38:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.457+0000 7f8c6e59c640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8c500763f0 0x7f8c50078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.457+0000 7f8c6e59c640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8c500763f0 0x7f8c50078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8c60032000 tx=0x7f8c60001190 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:33.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.461+0000 7f8c5effd640 1 -- 172.21.15.81:0/924761298 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8c64061310 con 0x7f8c701f38c0 2024-06-29T15:38:33.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.642+0000 7f8c74d11640 1 -- 172.21.15.81:0/924761298 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f8c700736c0 con 0x7f8c701f38c0 2024-06-29T15:38:33.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.642+0000 7f8c5effd640 1 -- 172.21.15.81:0/924761298 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v59) ==== 59+0+1998 (secure 0 0 0) 0x7f8c700736c0 con 0x7f8c701f38c0 2024-06-29T15:38:33.644 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:38:33.644 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:38:33.644 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:38:33.644 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:38:33.644 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T15:38:33.644 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T15:38:33.644 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T15:38:33.644 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:38:33.645 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:38:33.646 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:38:33.646 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T15:38:33.646 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T15:38:33.646 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T15:38:33.646 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:38:33.646 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 -- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8c500763f0 msgr2=0x7f8c50078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8c500763f0 0x7f8c50078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8c60032000 tx=0x7f8c60001190 comp rx=0 tx=0).stop 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 -- 172.21.15.81:0/924761298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c701f38c0 msgr2=0x7f8c701dfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c701f38c0 0x7f8c701dfc30 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f8c64011a80 tx=0x7f8c64011f40 comp rx=0 tx=0).stop 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 -- 172.21.15.81:0/924761298 shutdown_connections 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8c500763f0 0x7f8c50078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c7009c0f0 0x7f8c7009ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c701f38c0 0x7f8c701dfc30 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 --2- 172.21.15.81:0/924761298 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c58003a30 0x7f8c701f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 -- 172.21.15.81:0/924761298 >> 172.21.15.81:0/924761298 conn(0x7f8c700b3510 msgr2=0x7f8c700717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:33.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.644+0000 7f8c74d11640 1 -- 172.21.15.81:0/924761298 shutdown_connections 2024-06-29T15:38:33.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.645+0000 7f8c74d11640 1 -- 172.21.15.81:0/924761298 wait complete. 2024-06-29T15:38:33.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:38:33.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.772+0000 7f732e25b640 1 -- 172.21.15.81:0/94966836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f732806fa90 msgr2=0x7f732806ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.772+0000 7f732e25b640 1 --2- 172.21.15.81:0/94966836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f732806fa90 0x7f732806ca00 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f7320005fd0 tx=0x7f732002aa90 comp rx=0 tx=0).stop 2024-06-29T15:38:33.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.772+0000 7f732e25b640 1 -- 172.21.15.81:0/94966836 shutdown_connections 2024-06-29T15:38:33.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.772+0000 7f732e25b640 1 --2- 172.21.15.81:0/94966836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f732806fa90 0x7f732806ca00 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.772+0000 7f732e25b640 1 --2- 172.21.15.81:0/94966836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7328074520 0x7f732806f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.772+0000 7f732e25b640 1 -- 172.21.15.81:0/94966836 >> 172.21.15.81:0/94966836 conn(0x7f7328072950 msgr2=0x7f73280b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:33.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.772+0000 7f732e25b640 1 -- 172.21.15.81:0/94966836 shutdown_connections 2024-06-29T15:38:33.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.772+0000 7f732e25b640 1 -- 172.21.15.81:0/94966836 wait complete. 2024-06-29T15:38:33.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.773+0000 7f732e25b640 1 Processor -- start 2024-06-29T15:38:33.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.774+0000 7f732e25b640 1 -- start start 2024-06-29T15:38:33.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f732e25b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f732806f680 0x7f732814e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f732e25b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f732806fa90 0x7f732814f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f732e25b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7328074520 0x7f73280d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f732e25b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f732815b010 con 0x7f7328074520 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f732e25b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f732806d150 con 0x7f732806fa90 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f732e25b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f732815b190 con 0x7f732806f680 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f7327fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7328074520 0x7f73280d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f7327fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7328074520 0x7f73280d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35112/0 (socket says 172.21.15.81:35112) 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f7327fff640 1 -- 172.21.15.81:0/2582160617 learned_addr learned my addr 172.21.15.81:0/2582160617 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f73277fe640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f732806f680 0x7f732814e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f7326ffd640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f732806fa90 0x7f732814f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f7327fff640 1 -- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f732806f680 msgr2=0x7f732814e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f7327fff640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f732806f680 0x7f732814e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f7327fff640 1 -- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f732806fa90 msgr2=0x7f732814f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f7327fff640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f732806fa90 0x7f732814f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.775+0000 7f7327fff640 1 -- 172.21.15.81:0/2582160617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7328150450 con 0x7f7328074520 2024-06-29T15:38:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.776+0000 7f7327fff640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7328074520 0x7f73280d0030 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f7320009d90 tx=0x7f7320003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.776+0000 7f730ffff640 1 -- 172.21.15.81:0/2582160617 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320044070 con 0x7f7328074520 2024-06-29T15:38:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.776+0000 7f730ffff640 1 -- 172.21.15.81:0/2582160617 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7320004250 con 0x7f7328074520 2024-06-29T15:38:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.776+0000 7f732e25b640 1 -- 172.21.15.81:0/2582160617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7328150730 con 0x7f7328074520 2024-06-29T15:38:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.776+0000 7f730ffff640 1 -- 172.21.15.81:0/2582160617 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320004680 con 0x7f7328074520 2024-06-29T15:38:33.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.776+0000 7f732e25b640 1 -- 172.21.15.81:0/2582160617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7328150cf0 con 0x7f7328074520 2024-06-29T15:38:33.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.777+0000 7f732e25b640 1 -- 172.21.15.81:0/2582160617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73281586c0 con 0x7f7328074520 2024-06-29T15:38:33.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.778+0000 7f730ffff640 1 -- 172.21.15.81:0/2582160617 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f7320047070 con 0x7f7328074520 2024-06-29T15:38:33.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.778+0000 7f730ffff640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f73140763f0 0x7f7314078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:33.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.778+0000 7f730ffff640 1 -- 172.21.15.81:0/2582160617 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5731+0+0 (secure 0 0 0) 0x7f73200b8aa0 con 0x7f7328074520 2024-06-29T15:38:33.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.778+0000 7f73277fe640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f73140763f0 0x7f7314078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:33.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.779+0000 7f73277fe640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f73140763f0 0x7f7314078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7318005d70 tx=0x7f7318005d00 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:33.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.782+0000 7f730ffff640 1 -- 172.21.15.81:0/2582160617 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f73281586c0 con 0x7f7328074520 2024-06-29T15:38:33.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.962+0000 7f732e25b640 1 -- 172.21.15.81:0/2582160617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f732806f140 con 0x7f7328074520 2024-06-29T15:38:33.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.966+0000 7f730ffff640 1 -- 172.21.15.81:0/2582160617 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v60)=0 v60) ==== 122+0+0 (secure 0 0 0) 0x7f7320032030 con 0x7f7328074520 2024-06-29T15:38:33.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 -- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f73140763f0 msgr2=0x7f7314078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f73140763f0 0x7f7314078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7318005d70 tx=0x7f7318005d00 comp rx=0 tx=0).stop 2024-06-29T15:38:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 -- 172.21.15.81:0/2582160617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7328074520 msgr2=0x7f73280d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7328074520 0x7f73280d0030 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f7320009d90 tx=0x7f7320003f10 comp rx=0 tx=0).stop 2024-06-29T15:38:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 -- 172.21.15.81:0/2582160617 shutdown_connections 2024-06-29T15:38:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f73140763f0 0x7f7314078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7328074520 0x7f73280d0030 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f732806fa90 0x7f732814f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 --2- 172.21.15.81:0/2582160617 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f732806f680 0x7f732814e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 -- 172.21.15.81:0/2582160617 >> 172.21.15.81:0/2582160617 conn(0x7f7328072950 msgr2=0x7f73280b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 -- 172.21.15.81:0/2582160617 shutdown_connections 2024-06-29T15:38:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:33.968+0000 7f732e25b640 1 -- 172.21.15.81:0/2582160617 wait complete. 2024-06-29T15:38:33.981 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:38:33.981 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:34.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:34.063 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:38:34.063 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:38:34.063 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:34.082 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:38:34.082 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:38:34.082 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:34.122 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:34.122 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:38:34.122 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:34.137 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:38:34.138 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:38:34.138 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:34.181 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:34.181 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:38:34.181 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:34.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.293+0000 7f5d48703640 1 -- 172.21.15.81:0/3025359319 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d400af3c0 msgr2=0x7f5d4006c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.293+0000 7f5d48703640 1 --2- 172.21.15.81:0/3025359319 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d400af3c0 0x7f5d4006c6b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5d38005fd0 tx=0x7f5d3802aa10 comp rx=0 tx=0).stop 2024-06-29T15:38:34.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.293+0000 7f5d48703640 1 -- 172.21.15.81:0/3025359319 shutdown_connections 2024-06-29T15:38:34.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.293+0000 7f5d48703640 1 --2- 172.21.15.81:0/3025359319 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d400af3c0 0x7f5d4006c6b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.293+0000 7f5d48703640 1 --2- 172.21.15.81:0/3025359319 >> v2:172.21.15.167:3300/0 conn(0x7f5d400aea70 0x7f5d400aee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.293+0000 7f5d48703640 1 -- 172.21.15.81:0/3025359319 >> 172.21.15.81:0/3025359319 conn(0x7f5d40072950 msgr2=0x7f5d400b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:34.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.293+0000 7f5d48703640 1 -- 172.21.15.81:0/3025359319 shutdown_connections 2024-06-29T15:38:34.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.293+0000 7f5d48703640 1 -- 172.21.15.81:0/3025359319 wait complete. 2024-06-29T15:38:34.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.295+0000 7f5d48703640 1 Processor -- start 2024-06-29T15:38:34.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.295+0000 7f5d48703640 1 -- start start 2024-06-29T15:38:34.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.296+0000 7f5d48703640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d400ad0c0 0x7f5d40151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.296+0000 7f5d48703640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d400aea70 0x7f5d400b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.296+0000 7f5d48703640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d400af3c0 0x7f5d4014bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.296+0000 7f5d48703640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4006cf80 con 0x7f5d400af3c0 2024-06-29T15:38:34.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.296+0000 7f5d48703640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4006ce00 con 0x7f5d400aea70 2024-06-29T15:38:34.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.296+0000 7f5d48703640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5d4006d100 con 0x7f5d400ad0c0 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.296+0000 7f5d46ca0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d400af3c0 0x7f5d4014bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.296+0000 7f5d46ca0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d400af3c0 0x7f5d4014bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35126/0 (socket says 172.21.15.81:35126) 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.296+0000 7f5d46ca0640 1 -- 172.21.15.81:0/4174490741 learned_addr learned my addr 172.21.15.81:0/4174490741 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d45c9e640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d400aea70 0x7f5d400b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d4649f640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d400ad0c0 0x7f5d40151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d46ca0640 1 -- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d400ad0c0 msgr2=0x7f5d40151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d46ca0640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d400ad0c0 0x7f5d40151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d46ca0640 1 -- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d400aea70 msgr2=0x7f5d400b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d46ca0640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d400aea70 0x7f5d400b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d46ca0640 1 -- 172.21.15.81:0/4174490741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4013dc80 con 0x7f5d400af3c0 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d46ca0640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d400af3c0 0x7f5d4014bbc0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009d90 tx=0x7f5d38003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d36ffd640 1 -- 172.21.15.81:0/4174490741 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38046070 con 0x7f5d400af3c0 2024-06-29T15:38:34.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d36ffd640 1 -- 172.21.15.81:0/4174490741 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d38004540 con 0x7f5d400af3c0 2024-06-29T15:38:34.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d48703640 1 -- 172.21.15.81:0/4174490741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d40151c60 con 0x7f5d400af3c0 2024-06-29T15:38:34.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d36ffd640 1 -- 172.21.15.81:0/4174490741 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38033070 con 0x7f5d400af3c0 2024-06-29T15:38:34.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.297+0000 7f5d48703640 1 -- 172.21.15.81:0/4174490741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d401549a0 con 0x7f5d400af3c0 2024-06-29T15:38:34.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.298+0000 7f5d48703640 1 -- 172.21.15.81:0/4174490741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d4006d620 con 0x7f5d400af3c0 2024-06-29T15:38:34.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.299+0000 7f5d36ffd640 1 -- 172.21.15.81:0/4174490741 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f5d38049070 con 0x7f5d400af3c0 2024-06-29T15:38:34.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.299+0000 7f5d36ffd640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5d240763f0 0x7f5d24078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.299+0000 7f5d36ffd640 1 -- 172.21.15.81:0/4174490741 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5731+0+0 (secure 0 0 0) 0x7f5d380b8b20 con 0x7f5d400af3c0 2024-06-29T15:38:34.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.299+0000 7f5d4649f640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5d240763f0 0x7f5d24078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.300+0000 7f5d4649f640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5d240763f0 0x7f5d24078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5d40075640 tx=0x7f5d3c009040 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:34.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.304+0000 7f5d36ffd640 1 -- 172.21.15.81:0/4174490741 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d3803e070 con 0x7f5d400af3c0 2024-06-29T15:38:34.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.525+0000 7f5d48703640 1 -- 172.21.15.81:0/4174490741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d400b47c0 con 0x7f5d400af3c0 2024-06-29T15:38:34.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.526+0000 7f5d36ffd640 1 -- 172.21.15.81:0/4174490741 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+3914 (secure 0 0 0) 0x7f5d38082130 con 0x7f5d400af3c0 2024-06-29T15:38:34.527 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:34.528 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":79,"btime":"2024-06-29T15:38:28:210027+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7388,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/4068340499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":4068340499},{"type":"v1","addr":"172.21.15.81:6834","nonce":4068340499}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":7394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1903134398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1903134398},{"type":"v1","addr":"172.21.15.167:6837","nonce":1903134398}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":8662,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/897057810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":897057810},{"type":"v1","addr":"172.21.15.167:6836","nonce":897057810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":79},{"gid":9099,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1126258483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1126258483},{"type":"v1","addr":"172.21.15.81:6835","nonce":1126258483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[]} 2024-06-29T15:38:34.528 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 79 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 -- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5d240763f0 msgr2=0x7f5d24078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5d240763f0 0x7f5d24078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5d40075640 tx=0x7f5d3c009040 comp rx=0 tx=0).stop 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 -- 172.21.15.81:0/4174490741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d400af3c0 msgr2=0x7f5d4014bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d400af3c0 0x7f5d4014bbc0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009d90 tx=0x7f5d38003f20 comp rx=0 tx=0).stop 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 -- 172.21.15.81:0/4174490741 shutdown_connections 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5d240763f0 0x7f5d24078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d400af3c0 0x7f5d4014bbc0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d400aea70 0x7f5d400b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 --2- 172.21.15.81:0/4174490741 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d400ad0c0 0x7f5d40151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 -- 172.21.15.81:0/4174490741 >> 172.21.15.81:0/4174490741 conn(0x7f5d40072950 msgr2=0x7f5d401570f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:34.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 -- 172.21.15.81:0/4174490741 shutdown_connections 2024-06-29T15:38:34.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.528+0000 7f5d48703640 1 -- 172.21.15.81:0/4174490741 wait complete. 2024-06-29T15:38:34.541 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:38:34.541 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:38:34.541 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:38:34.541 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:38:34.542 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:38:34.542 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:38:34.542 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:38:34.542 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:38:34.542 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:38:34.542 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:38:34.542 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:38:34.543 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:38:34.543 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:38:34.543 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:38:34.543 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:38:34.543 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:38:34.551 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:38:34.552 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:38:34.553 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:38:34.553 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:38:34.558 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:38:34.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:38:34.560 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:38:34.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:38:34.595 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:38:34.595 DEBUG:teuthology.parallel:result is None 2024-06-29T15:38:34.596 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:38:34.596 DEBUG:teuthology.parallel:result is None 2024-06-29T15:38:34.601 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:38:34.601 DEBUG:teuthology.parallel:result is None 2024-06-29T15:38:34.601 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:38:34.602 DEBUG:teuthology.parallel:result is None 2024-06-29T15:38:34.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:38:34.682 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T15:38:34.683 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T15:38:34.697 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:38:34.698 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T15:38:34.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.721+0000 7f76eda9d640 1 -- 172.21.15.81:0/1773862169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76e80b3200 msgr2=0x7f76e80cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.721+0000 7f76eda9d640 1 --2- 172.21.15.81:0/1773862169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76e80b3200 0x7f76e80cc7b0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f76d8009830 tx=0x7f76d802a510 comp rx=0 tx=0).stop 2024-06-29T15:38:34.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.722+0000 7f76eda9d640 1 -- 172.21.15.81:0/1773862169 shutdown_connections 2024-06-29T15:38:34.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.722+0000 7f76eda9d640 1 --2- 172.21.15.81:0/1773862169 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76e80cccf0 0x7f76e80d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.722+0000 7f76eda9d640 1 --2- 172.21.15.81:0/1773862169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76e80b3200 0x7f76e80cc7b0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.722+0000 7f76eda9d640 1 -- 172.21.15.81:0/1773862169 >> 172.21.15.81:0/1773862169 conn(0x7f76e80b3680 msgr2=0x7f76e80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:34.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.722+0000 7f76eda9d640 1 -- 172.21.15.81:0/1773862169 shutdown_connections 2024-06-29T15:38:34.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.722+0000 7f76eda9d640 1 -- 172.21.15.81:0/1773862169 wait complete. 2024-06-29T15:38:34.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.723+0000 7f76eda9d640 1 Processor -- start 2024-06-29T15:38:34.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.724+0000 7f76eda9d640 1 -- start start 2024-06-29T15:38:34.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76eda9d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76e80b3200 0x7f76e814af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76eda9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76e80cccf0 0x7f76e8157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76eda9d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76e80d0770 0x7f76e81583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76eda9d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76e80cedb0 con 0x7f76e80cccf0 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76eda9d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f76e80cec30 con 0x7f76e80b3200 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76eda9d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f76e80cef30 con 0x7f76e80d0770 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76e80cccf0 0x7f76e8157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76e80cccf0 0x7f76e8157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35166/0 (socket says 172.21.15.81:35166) 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e6ffd640 1 -- 172.21.15.81:0/2583838484 learned_addr learned my addr 172.21.15.81:0/2583838484 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e7fff640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76e80d0770 0x7f76e81583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e77fe640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76e80b3200 0x7f76e814af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e6ffd640 1 -- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76e80d0770 msgr2=0x7f76e81583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e6ffd640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76e80d0770 0x7f76e81583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e6ffd640 1 -- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76e80b3200 msgr2=0x7f76e814af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e6ffd640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76e80b3200 0x7f76e814af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e6ffd640 1 -- 172.21.15.81:0/2583838484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e814b4a0 con 0x7f76e80cccf0 2024-06-29T15:38:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.725+0000 7f76e6ffd640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76e80cccf0 0x7f76e8157eb0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f76dc00d700 tx=0x7f76dc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:34.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.726+0000 7f76cffff640 1 -- 172.21.15.81:0/2583838484 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc017070 con 0x7f76e80cccf0 2024-06-29T15:38:34.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.726+0000 7f76cffff640 1 -- 172.21.15.81:0/2583838484 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76dc005330 con 0x7f76e80cccf0 2024-06-29T15:38:34.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.726+0000 7f76eda9d640 1 -- 172.21.15.81:0/2583838484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76e81e6390 con 0x7f76e80cccf0 2024-06-29T15:38:34.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.726+0000 7f76cffff640 1 -- 172.21.15.81:0/2583838484 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc009820 con 0x7f76e80cccf0 2024-06-29T15:38:34.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.726+0000 7f76eda9d640 1 -- 172.21.15.81:0/2583838484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76e81e6b70 con 0x7f76e80cccf0 2024-06-29T15:38:34.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.727+0000 7f76eda9d640 1 -- 172.21.15.81:0/2583838484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76e80cc7b0 con 0x7f76e80cccf0 2024-06-29T15:38:34.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.727+0000 7f76cffff640 1 -- 172.21.15.81:0/2583838484 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f76dc0099c0 con 0x7f76e80cccf0 2024-06-29T15:38:34.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.728+0000 7f76cffff640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f76d40763f0 0x7f76d4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:34.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.728+0000 7f76cffff640 1 -- 172.21.15.81:0/2583838484 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5731+0+0 (secure 0 0 0) 0x7f76dc097b40 con 0x7f76e80cccf0 2024-06-29T15:38:34.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.728+0000 7f76e77fe640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f76d40763f0 0x7f76d4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:34.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.729+0000 7f76e77fe640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f76d40763f0 0x7f76d4078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f76d8034000 tx=0x7f76d8004730 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:34.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.732+0000 7f76cffff640 1 -- 172.21.15.81:0/2583838484 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76e814b4a0 con 0x7f76e80cccf0 2024-06-29T15:38:34.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:34.926+0000 7f76eda9d640 1 -- 172.21.15.81:0/2583838484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f76e815ae70 con 0x7f76e80cccf0 2024-06-29T15:38:35.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.334+0000 7f76cffff640 1 -- 172.21.15.81:0/2583838484 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v162) ==== 97+0+0 (secure 0 0 0) 0x7f76dc0691d0 con 0x7f76e80cccf0 2024-06-29T15:38:35.336 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 -- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f76d40763f0 msgr2=0x7f76d4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f76d40763f0 0x7f76d4078900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f76d8034000 tx=0x7f76d8004730 comp rx=0 tx=0).stop 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 -- 172.21.15.81:0/2583838484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76e80cccf0 msgr2=0x7f76e8157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76e80cccf0 0x7f76e8157eb0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f76dc00d700 tx=0x7f76dc00dbc0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 -- 172.21.15.81:0/2583838484 shutdown_connections 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f76d40763f0 0x7f76d4078900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76e80d0770 0x7f76e81583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76e80cccf0 0x7f76e8157eb0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 --2- 172.21.15.81:0/2583838484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76e80b3200 0x7f76e814af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 -- 172.21.15.81:0/2583838484 >> 172.21.15.81:0/2583838484 conn(0x7f76e80b3680 msgr2=0x7f76e80d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 -- 172.21.15.81:0/2583838484 shutdown_connections 2024-06-29T15:38:35.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.337+0000 7f76eda9d640 1 -- 172.21.15.81:0/2583838484 wait complete. 2024-06-29T15:38:35.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:38:35.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.463+0000 7fd5afb8c640 1 -- 172.21.15.81:0/3180396338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a815e960 msgr2=0x7fd5a815ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.463+0000 7fd5afb8c640 1 --2- 172.21.15.81:0/3180396338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a815e960 0x7fd5a815ed70 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0005fd0 tx=0x7fd5a002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.463+0000 7fd5afb8c640 1 -- 172.21.15.81:0/3180396338 shutdown_connections 2024-06-29T15:38:35.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.463+0000 7fd5afb8c640 1 --2- 172.21.15.81:0/3180396338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a815e960 0x7fd5a815ed70 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.463+0000 7fd5afb8c640 1 --2- 172.21.15.81:0/3180396338 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a8073030 0x7fd5a8073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.463+0000 7fd5afb8c640 1 -- 172.21.15.81:0/3180396338 >> 172.21.15.81:0/3180396338 conn(0x7fd5a806ccc0 msgr2=0x7fd5a80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:35.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.464+0000 7fd5afb8c640 1 -- 172.21.15.81:0/3180396338 shutdown_connections 2024-06-29T15:38:35.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.464+0000 7fd5afb8c640 1 -- 172.21.15.81:0/3180396338 wait complete. 2024-06-29T15:38:35.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.465+0000 7fd5afb8c640 1 Processor -- start 2024-06-29T15:38:35.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.465+0000 7fd5afb8c640 1 -- start start 2024-06-29T15:38:35.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5afb8c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5a8073030 0x7fd5a8140320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5afb8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a81412f0 0x7fd5a813fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5afb8c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a815e960 0x7fd5a8153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5afb8c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a80b3390 con 0x7fd5a81412f0 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5afb8c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a80b3210 con 0x7fd5a8073030 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5afb8c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a80b3510 con 0x7fd5a815e960 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ad127640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a81412f0 0x7fd5a813fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ad127640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a81412f0 0x7fd5a813fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35190/0 (socket says 172.21.15.81:35190) 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ad127640 1 -- 172.21.15.81:0/844297255 learned_addr learned my addr 172.21.15.81:0/844297255 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ad928640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5a8073030 0x7fd5a8140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ae129640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a815e960 0x7fd5a8153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ad127640 1 -- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a815e960 msgr2=0x7fd5a8153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ad127640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a815e960 0x7fd5a8153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ad127640 1 -- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5a8073030 msgr2=0x7fd5a8140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ad127640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5a8073030 0x7fd5a8140320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ad127640 1 -- 172.21.15.81:0/844297255 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a80a94c0 con 0x7fd5a81412f0 2024-06-29T15:38:35.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5ad127640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a81412f0 0x7fd5a813fad0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fd59800c9a0 tx=0x7fd59800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:35.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd59e7fc640 1 -- 172.21.15.81:0/844297255 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598007bb0 con 0x7fd5a81412f0 2024-06-29T15:38:35.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd59e7fc640 1 -- 172.21.15.81:0/844297255 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd598007e40 con 0x7fd5a81412f0 2024-06-29T15:38:35.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd59e7fc640 1 -- 172.21.15.81:0/844297255 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598005280 con 0x7fd5a81412f0 2024-06-29T15:38:35.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.467+0000 7fd5afb8c640 1 -- 172.21.15.81:0/844297255 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a80a97a0 con 0x7fd5a81412f0 2024-06-29T15:38:35.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.468+0000 7fd5afb8c640 1 -- 172.21.15.81:0/844297255 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a80a9d60 con 0x7fd5a81412f0 2024-06-29T15:38:35.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.468+0000 7fd5afb8c640 1 -- 172.21.15.81:0/844297255 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a8140990 con 0x7fd5a81412f0 2024-06-29T15:38:35.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.469+0000 7fd59e7fc640 1 -- 172.21.15.81:0/844297255 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fd598005420 con 0x7fd5a81412f0 2024-06-29T15:38:35.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.470+0000 7fd59e7fc640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd58c076420 0x7fd58c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.470+0000 7fd59e7fc640 1 -- 172.21.15.81:0/844297255 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 4957+0+0 (secure 0 0 0) 0x7fd598097a80 con 0x7fd5a81412f0 2024-06-29T15:38:35.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.470+0000 7fd5ad928640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd58c076420 0x7fd58c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.471+0000 7fd5ad928640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd58c076420 0x7fd58c078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4007d60 tx=0x7fd5a4007cf0 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:35.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.474+0000 7fd59e7fc640 1 -- 172.21.15.81:0/844297255 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd598014030 con 0x7fd5a81412f0 2024-06-29T15:38:35.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.720+0000 7fd5afb8c640 1 -- 172.21.15.81:0/844297255 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fd5a80b3a10 con 0x7fd5a81412f0 2024-06-29T15:38:35.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.720+0000 7fd59e7fc640 1 -- 172.21.15.81:0/844297255 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v57) ==== 80+0+5304 (secure 0 0 0) 0x7fd598060e90 con 0x7fd5a81412f0 2024-06-29T15:38:35.722 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:35.722 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:38:35.722 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T15:38:35.722 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.722 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T15:38:35.723 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T15:38:35.723 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:38:35.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:38:35.723 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:38:35.723 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.723 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.725 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.725 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.725 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T15:38:35.725 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.726 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:38:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.729 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.730 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.731 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.732 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.733 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T15:38:35.734 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T15:38:35.735 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:38:35.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:38:35.735 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:38:35.735 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:38:35.735 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.735 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.735 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.735 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T15:38:35.736 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:38:35.737 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T15:38:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T15:38:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:38:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.739 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:35.740 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T15:38:35.740 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:38:35.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 -- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd58c076420 msgr2=0x7fd58c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd58c076420 0x7fd58c078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4007d60 tx=0x7fd5a4007cf0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 -- 172.21.15.81:0/844297255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a81412f0 msgr2=0x7fd5a813fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a81412f0 0x7fd5a813fad0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fd59800c9a0 tx=0x7fd59800ce60 comp rx=0 tx=0).stop 2024-06-29T15:38:35.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 -- 172.21.15.81:0/844297255 shutdown_connections 2024-06-29T15:38:35.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd58c076420 0x7fd58c078930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5a815e960 0x7fd5a8153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5a81412f0 0x7fd5a813fad0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 --2- 172.21.15.81:0/844297255 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5a8073030 0x7fd5a8140320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 -- 172.21.15.81:0/844297255 >> 172.21.15.81:0/844297255 conn(0x7fd5a806ccc0 msgr2=0x7fd5a806f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:35.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 -- 172.21.15.81:0/844297255 shutdown_connections 2024-06-29T15:38:35.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.722+0000 7fd5afb8c640 1 -- 172.21.15.81:0/844297255 wait complete. 2024-06-29T15:38:35.743 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:35.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.856+0000 7fc66fbf0640 1 -- 172.21.15.81:0/4130035029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc66806f660 msgr2=0x7fc66806fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.856+0000 7fc66fbf0640 1 --2- 172.21.15.81:0/4130035029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc66806f660 0x7fc66806fa70 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fc668070220 tx=0x7fc66002a980 comp rx=0 tx=0).stop 2024-06-29T15:38:35.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.856+0000 7fc66fbf0640 1 -- 172.21.15.81:0/4130035029 shutdown_connections 2024-06-29T15:38:35.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.856+0000 7fc66fbf0640 1 --2- 172.21.15.81:0/4130035029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc66806f660 0x7fc66806fa70 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.856+0000 7fc66fbf0640 1 --2- 172.21.15.81:0/4130035029 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6680b4430 0x7fc6680cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.856+0000 7fc66fbf0640 1 --2- 172.21.15.81:0/4130035029 >> v2:172.21.15.167:3300/0 conn(0x7fc6680b35b0 0x7fc66815b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.856+0000 7fc66fbf0640 1 -- 172.21.15.81:0/4130035029 >> 172.21.15.81:0/4130035029 conn(0x7fc66806ccc0 msgr2=0x7fc6680b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:35.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.856+0000 7fc66fbf0640 1 -- 172.21.15.81:0/4130035029 shutdown_connections 2024-06-29T15:38:35.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.856+0000 7fc66fbf0640 1 -- 172.21.15.81:0/4130035029 wait complete. 2024-06-29T15:38:35.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.858+0000 7fc66fbf0640 1 Processor -- start 2024-06-29T15:38:35.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.858+0000 7fc66fbf0640 1 -- start start 2024-06-29T15:38:35.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66fbf0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66806f660 0x7fc66814fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66fbf0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6680b35b0 0x7fc6680aaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66fbf0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6680b4430 0x7fc6680ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66fbf0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc6680722b0 con 0x7fc6680b35b0 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66fbf0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc668072130 con 0x7fc66806f660 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66fbf0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc668072430 con 0x7fc6680b4430 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66d18b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6680b35b0 0x7fc6680aaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66d18b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6680b35b0 0x7fc6680aaf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35206/0 (socket says 172.21.15.81:35206) 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66d18b640 1 -- 172.21.15.81:0/1593211921 learned_addr learned my addr 172.21.15.81:0/1593211921 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66d98c640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66806f660 0x7fc66814fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66e18d640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6680b4430 0x7fc6680ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66d18b640 1 -- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6680b4430 msgr2=0x7fc6680ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66d18b640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6680b4430 0x7fc6680ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66d18b640 1 -- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66806f660 msgr2=0x7fc66814fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.859+0000 7fc66d18b640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66806f660 0x7fc66814fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.860+0000 7fc66d18b640 1 -- 172.21.15.81:0/1593211921 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6680ad490 con 0x7fc6680b35b0 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.860+0000 7fc66d98c640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66806f660 0x7fc66814fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.860+0000 7fc66d18b640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6680b35b0 0x7fc6680aaf40 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00b410 tx=0x7fc65c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.860+0000 7fc6667fc640 1 -- 172.21.15.81:0/1593211921 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c015020 con 0x7fc6680b35b0 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.860+0000 7fc6667fc640 1 -- 172.21.15.81:0/1593211921 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc65c011940 con 0x7fc6680b35b0 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.860+0000 7fc6667fc640 1 -- 172.21.15.81:0/1593211921 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c011d90 con 0x7fc6680b35b0 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.860+0000 7fc66fbf0640 1 -- 172.21.15.81:0/1593211921 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6680aa120 con 0x7fc6680b35b0 2024-06-29T15:38:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.860+0000 7fc66fbf0640 1 -- 172.21.15.81:0/1593211921 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6680aa6e0 con 0x7fc6680b35b0 2024-06-29T15:38:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.861+0000 7fc66fbf0640 1 -- 172.21.15.81:0/1593211921 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc66815a520 con 0x7fc6680b35b0 2024-06-29T15:38:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.862+0000 7fc6667fc640 1 -- 172.21.15.81:0/1593211921 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fc65c018020 con 0x7fc6680b35b0 2024-06-29T15:38:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.862+0000 7fc6667fc640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc6540763f0 0x7fc654078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.862+0000 7fc6667fc640 1 -- 172.21.15.81:0/1593211921 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 4957+0+0 (secure 0 0 0) 0x7fc65c097590 con 0x7fc6680b35b0 2024-06-29T15:38:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.862+0000 7fc66d98c640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc6540763f0 0x7fc654078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.863+0000 7fc66d98c640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc6540763f0 0x7fc654078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc658009890 tx=0x7fc658008040 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:35.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:35.866+0000 7fc6667fc640 1 -- 172.21.15.81:0/1593211921 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc65c01f020 con 0x7fc6680b35b0 2024-06-29T15:38:36.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.051+0000 7fc66fbf0640 1 -- 172.21.15.81:0/1593211921 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc668072a40 con 0x7fc6680b35b0 2024-06-29T15:38:36.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.051+0000 7fc6667fc640 1 -- 172.21.15.81:0/1593211921 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) ==== 74+0+13048 (secure 0 0 0) 0x7fc65c075e20 con 0x7fc6680b35b0 2024-06-29T15:38:36.053 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:36.053 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":164,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:35.345195+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:36.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 -- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc6540763f0 msgr2=0x7fc654078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc6540763f0 0x7fc654078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc658009890 tx=0x7fc658008040 comp rx=0 tx=0).stop 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 -- 172.21.15.81:0/1593211921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6680b35b0 msgr2=0x7fc6680aaf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6680b35b0 0x7fc6680aaf40 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00b410 tx=0x7fc65c00b8d0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 -- 172.21.15.81:0/1593211921 shutdown_connections 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc6540763f0 0x7fc654078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6680b4430 0x7fc6680ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6680b35b0 0x7fc6680aaf40 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 --2- 172.21.15.81:0/1593211921 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66806f660 0x7fc66814fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 -- 172.21.15.81:0/1593211921 >> 172.21.15.81:0/1593211921 conn(0x7fc66806ccc0 msgr2=0x7fc668152080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 -- 172.21.15.81:0/1593211921 shutdown_connections 2024-06-29T15:38:36.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.054+0000 7fc66fbf0640 1 -- 172.21.15.81:0/1593211921 wait complete. 2024-06-29T15:38:36.067 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:36.067 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:36.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.183+0000 7f853ffac640 1 -- 172.21.15.81:0/2726684556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8524004800 msgr2=0x7f8524004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.183+0000 7f853ffac640 1 --2- 172.21.15.81:0/2726684556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8524004800 0x7f8524004c10 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f853815f8c0 tx=0x7f8524005830 comp rx=0 tx=0).stop 2024-06-29T15:38:36.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.183+0000 7f853e549640 1 -- 172.21.15.81:0/2726684556 reap_dead start 2024-06-29T15:38:36.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.183+0000 7f853ffac640 1 -- 172.21.15.81:0/2726684556 shutdown_connections 2024-06-29T15:38:36.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.183+0000 7f853ffac640 1 -- 172.21.15.81:0/2726684556 >> 172.21.15.81:0/2726684556 conn(0x7f8538072950 msgr2=0x7f85380b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:36.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.183+0000 7f853ffac640 1 -- 172.21.15.81:0/2726684556 shutdown_connections 2024-06-29T15:38:36.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.183+0000 7f853ffac640 1 -- 172.21.15.81:0/2726684556 wait complete. 2024-06-29T15:38:36.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.185+0000 7f853ffac640 1 Processor -- start 2024-06-29T15:38:36.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.185+0000 7f853ffac640 1 -- start start 2024-06-29T15:38:36.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.186+0000 7f853ffac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8524004800 0x7f85381543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.186+0000 7f853ffac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f853813c980 0x7f853813c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.186+0000 7f853ffac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f853815f6b0 0x7f853815fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.186+0000 7f853ffac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f85380740a0 con 0x7f8524004800 2024-06-29T15:38:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.186+0000 7f853ffac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8538073f20 con 0x7f853815f6b0 2024-06-29T15:38:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.186+0000 7f853ffac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8538074220 con 0x7f853813c980 2024-06-29T15:38:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853dd48640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8524004800 0x7f85381543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853dd48640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8524004800 0x7f85381543b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35222/0 (socket says 172.21.15.81:35222) 2024-06-29T15:38:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853dd48640 1 -- 172.21.15.81:0/319851748 learned_addr learned my addr 172.21.15.81:0/319851748 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853d547640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f853813c980 0x7f853813c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853e549640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f853815f6b0 0x7f853815fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853dd48640 1 -- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f853813c980 msgr2=0x7f853813c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853dd48640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f853813c980 0x7f853813c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853dd48640 1 -- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f853815f6b0 msgr2=0x7f853815fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853dd48640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f853815f6b0 0x7f853815fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853dd48640 1 -- 172.21.15.81:0/319851748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8538151b80 con 0x7f8524004800 2024-06-29T15:38:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853dd48640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8524004800 0x7f85381543b0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f8534004410 tx=0x7f85340048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f852e7fc640 1 -- 172.21.15.81:0/319851748 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534016070 con 0x7f8524004800 2024-06-29T15:38:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f852e7fc640 1 -- 172.21.15.81:0/319851748 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8534010dc0 con 0x7f8524004800 2024-06-29T15:38:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f852e7fc640 1 -- 172.21.15.81:0/319851748 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85340126b0 con 0x7f8524004800 2024-06-29T15:38:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853ffac640 1 -- 172.21.15.81:0/319851748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8538160000 con 0x7f8524004800 2024-06-29T15:38:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.187+0000 7f853ffac640 1 -- 172.21.15.81:0/319851748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8538151df0 con 0x7f8524004800 2024-06-29T15:38:36.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.188+0000 7f853ffac640 1 -- 172.21.15.81:0/319851748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85381594b0 con 0x7f8524004800 2024-06-29T15:38:36.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.189+0000 7f852e7fc640 1 -- 172.21.15.81:0/319851748 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f8534012850 con 0x7f8524004800 2024-06-29T15:38:36.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.189+0000 7f852e7fc640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f851c076420 0x7f851c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.189+0000 7f852e7fc640 1 -- 172.21.15.81:0/319851748 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 4957+0+0 (secure 0 0 0) 0x7f8534098a50 con 0x7f8524004800 2024-06-29T15:38:36.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.190+0000 7f853d547640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f851c076420 0x7f851c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.190+0000 7f853d547640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f851c076420 0x7f851c078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8528036000 tx=0x7f8528035040 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.194+0000 7f852e7fc640 1 -- 172.21.15.81:0/319851748 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f85381594b0 con 0x7f8524004800 2024-06-29T15:38:36.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.378+0000 7f853ffac640 1 -- 172.21.15.81:0/319851748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f853815bdf0 con 0x7f8524004800 2024-06-29T15:38:36.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.378+0000 7f852e7fc640 1 -- 172.21.15.81:0/319851748 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) ==== 93+0+10 (secure 0 0 0) 0x7f8534081e20 con 0x7f8524004800 2024-06-29T15:38:36.380 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:38:36.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.380+0000 7f853ffac640 1 -- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f851c076420 msgr2=0x7f851c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.380+0000 7f853ffac640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f851c076420 0x7f851c078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8528036000 tx=0x7f8528035040 comp rx=0 tx=0).stop 2024-06-29T15:38:36.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.380+0000 7f853ffac640 1 -- 172.21.15.81:0/319851748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8524004800 msgr2=0x7f85381543b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.380+0000 7f853ffac640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8524004800 0x7f85381543b0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f8534004410 tx=0x7f85340048d0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.380+0000 7f853ffac640 1 -- 172.21.15.81:0/319851748 shutdown_connections 2024-06-29T15:38:36.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.380+0000 7f853ffac640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f851c076420 0x7f851c078930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.380+0000 7f853ffac640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f853815f6b0 0x7f853815fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.380+0000 7f853ffac640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f853813c980 0x7f853813c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.380+0000 7f853ffac640 1 --2- 172.21.15.81:0/319851748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8524004800 0x7f85381543b0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.380+0000 7f853ffac640 1 -- 172.21.15.81:0/319851748 >> 172.21.15.81:0/319851748 conn(0x7f8538072950 msgr2=0x7f85380d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:36.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.381+0000 7f853ffac640 1 -- 172.21.15.81:0/319851748 shutdown_connections 2024-06-29T15:38:36.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.381+0000 7f853ffac640 1 -- 172.21.15.81:0/319851748 wait complete. 2024-06-29T15:38:36.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:38:36.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.506+0000 7f38426ba640 1 -- 172.21.15.81:0/1612549722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3824004800 msgr2=0x7f3824004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.506+0000 7f38426ba640 1 --2- 172.21.15.81:0/1612549722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3824004800 0x7f3824004c10 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f383000bb50 tx=0x7f3824005830 comp rx=0 tx=0).stop 2024-06-29T15:38:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.506+0000 7f3840c57640 1 -- 172.21.15.81:0/1612549722 reap_dead start 2024-06-29T15:38:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.506+0000 7f38426ba640 1 -- 172.21.15.81:0/1612549722 shutdown_connections 2024-06-29T15:38:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.506+0000 7f38426ba640 1 -- 172.21.15.81:0/1612549722 >> 172.21.15.81:0/1612549722 conn(0x7f383c0b3510 msgr2=0x7f383c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.506+0000 7f38426ba640 1 -- 172.21.15.81:0/1612549722 shutdown_connections 2024-06-29T15:38:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.506+0000 7f38426ba640 1 -- 172.21.15.81:0/1612549722 wait complete. 2024-06-29T15:38:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.508+0000 7f38426ba640 1 Processor -- start 2024-06-29T15:38:36.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.508+0000 7f38426ba640 1 -- start start 2024-06-29T15:38:36.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f38426ba640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3824004800 0x7f383c1e1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f38426ba640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f383c09f270 0x7f383c0caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f38426ba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f383c1e22e0 0x7f383c0caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f38426ba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f383c0ce8a0 con 0x7f383c1e22e0 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f38426ba640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f383c0ce720 con 0x7f3824004800 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f38426ba640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f383c0b4f20 con 0x7f383c09f270 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f3840c57640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f383c1e22e0 0x7f383c0caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f3840c57640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f383c1e22e0 0x7f383c0caf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35250/0 (socket says 172.21.15.81:35250) 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f3840c57640 1 -- 172.21.15.81:0/2407243611 learned_addr learned my addr 172.21.15.81:0/2407243611 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f383b7fe640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f383c09f270 0x7f383c0caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f383bfff640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3824004800 0x7f383c1e1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f3840c57640 1 -- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f383c09f270 msgr2=0x7f383c0caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f3840c57640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f383c09f270 0x7f383c0caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f3840c57640 1 -- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3824004800 msgr2=0x7f383c1e1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f3840c57640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3824004800 0x7f383c1e1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f3840c57640 1 -- 172.21.15.81:0/2407243611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c0b1230 con 0x7f383c1e22e0 2024-06-29T15:38:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f3840c57640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f383c1e22e0 0x7f383c0caf80 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f383c0cdb30 tx=0x7f38340019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.510+0000 7f3838ff9640 1 -- 172.21.15.81:0/2407243611 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834017070 con 0x7f383c1e22e0 2024-06-29T15:38:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.511+0000 7f3838ff9640 1 -- 172.21.15.81:0/2407243611 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38340108c0 con 0x7f383c1e22e0 2024-06-29T15:38:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.511+0000 7f3838ff9640 1 -- 172.21.15.81:0/2407243611 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834010d10 con 0x7f383c1e22e0 2024-06-29T15:38:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.511+0000 7f38426ba640 1 -- 172.21.15.81:0/2407243611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c0b1510 con 0x7f383c1e22e0 2024-06-29T15:38:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.511+0000 7f38426ba640 1 -- 172.21.15.81:0/2407243611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c1e55f0 con 0x7f383c1e22e0 2024-06-29T15:38:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.512+0000 7f38426ba640 1 -- 172.21.15.81:0/2407243611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383c0d02d0 con 0x7f383c1e22e0 2024-06-29T15:38:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.512+0000 7f3838ff9640 1 -- 172.21.15.81:0/2407243611 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f383401a070 con 0x7f383c1e22e0 2024-06-29T15:38:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.513+0000 7f3838ff9640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f38280763f0 0x7f3828078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.513+0000 7f3838ff9640 1 -- 172.21.15.81:0/2407243611 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 4957+0+0 (secure 0 0 0) 0x7f3834097360 con 0x7f383c1e22e0 2024-06-29T15:38:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.513+0000 7f383bfff640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f38280763f0 0x7f3828078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.514+0000 7f383bfff640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f38280763f0 0x7f3828078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f382c000ff0 tx=0x7f382c0067a0 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.517+0000 7f3838ff9640 1 -- 172.21.15.81:0/2407243611 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3834060740 con 0x7f383c1e22e0 2024-06-29T15:38:36.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.698+0000 7f38426ba640 1 -- 172.21.15.81:0/2407243611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f383c09f770 con 0x7f383c1e22e0 2024-06-29T15:38:36.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.699+0000 7f3838ff9640 1 -- 172.21.15.81:0/2407243611 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v164) ==== 84+0+65 (secure 0 0 0) 0x7f3834064a40 con 0x7f383c1e22e0 2024-06-29T15:38:36.700 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:36.701 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:38:36.701 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:36.701 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T15:38:36.701 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T15:38:36.701 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:36.701 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:38:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 -- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f38280763f0 msgr2=0x7f3828078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f38280763f0 0x7f3828078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f382c000ff0 tx=0x7f382c0067a0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 -- 172.21.15.81:0/2407243611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f383c1e22e0 msgr2=0x7f383c0caf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f383c1e22e0 0x7f383c0caf80 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f383c0cdb30 tx=0x7f38340019b0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 -- 172.21.15.81:0/2407243611 shutdown_connections 2024-06-29T15:38:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f38280763f0 0x7f3828078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f383c1e22e0 0x7f383c0caf80 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f383c09f270 0x7f383c0caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 --2- 172.21.15.81:0/2407243611 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3824004800 0x7f383c1e1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 -- 172.21.15.81:0/2407243611 >> 172.21.15.81:0/2407243611 conn(0x7f383c0b3510 msgr2=0x7f383c071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 -- 172.21.15.81:0/2407243611 shutdown_connections 2024-06-29T15:38:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.701+0000 7f38426ba640 1 -- 172.21.15.81:0/2407243611 wait complete. 2024-06-29T15:38:36.714 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:38:36.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:38:36.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.830+0000 7fe600fac640 1 -- 172.21.15.81:0/3050753752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5fc06f760 msgr2=0x7fe5fc06fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.830+0000 7fe600fac640 1 --2- 172.21.15.81:0/3050753752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5fc06f760 0x7fe5fc06fb70 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc071130 tx=0x7fe5f402aa30 comp rx=0 tx=0).stop 2024-06-29T15:38:36.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.830+0000 7fe600fac640 1 -- 172.21.15.81:0/3050753752 shutdown_connections 2024-06-29T15:38:36.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.830+0000 7fe600fac640 1 --2- 172.21.15.81:0/3050753752 >> v2:172.21.15.167:3300/0 conn(0x7fe5fc144630 0x7fe5fc072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.830+0000 7fe600fac640 1 --2- 172.21.15.81:0/3050753752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5fc06f760 0x7fe5fc06fb70 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.830+0000 7fe600fac640 1 -- 172.21.15.81:0/3050753752 >> 172.21.15.81:0/3050753752 conn(0x7fe5fc06cd60 msgr2=0x7fe5fc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:36.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.830+0000 7fe600fac640 1 -- 172.21.15.81:0/3050753752 shutdown_connections 2024-06-29T15:38:36.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.830+0000 7fe600fac640 1 -- 172.21.15.81:0/3050753752 wait complete. 2024-06-29T15:38:36.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.832+0000 7fe600fac640 1 Processor -- start 2024-06-29T15:38:36.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.832+0000 7fe600fac640 1 -- start start 2024-06-29T15:38:36.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe600fac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5fc06f760 0x7fe5fc15a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe600fac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5fc0b4130 0x7fe5fc0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe600fac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5fc144630 0x7fe5fc0cea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe600fac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc0b2a30 con 0x7fe5fc144630 2024-06-29T15:38:36.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe600fac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc072eb0 con 0x7fe5fc0b4130 2024-06-29T15:38:36.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe600fac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe5fc0b2bb0 con 0x7fe5fc06f760 2024-06-29T15:38:36.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe5fad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5fc144630 0x7fe5fc0cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe5fad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5fc144630 0x7fe5fc0cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35264/0 (socket says 172.21.15.81:35264) 2024-06-29T15:38:36.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe5fad9d640 1 -- 172.21.15.81:0/298912508 learned_addr learned my addr 172.21.15.81:0/298912508 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe5fa59c640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5fc06f760 0x7fe5fc15a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.833+0000 7fe5f9d9b640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5fc0b4130 0x7fe5fc0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5fad9d640 1 -- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5fc06f760 msgr2=0x7fe5fc15a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5fad9d640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5fc06f760 0x7fe5fc15a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5fad9d640 1 -- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5fc0b4130 msgr2=0x7fe5fc0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5fad9d640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5fc0b4130 0x7fe5fc0ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5fad9d640 1 -- 172.21.15.81:0/298912508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5fc15ad80 con 0x7fe5fc144630 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5f9d9b640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5fc0b4130 0x7fe5fc0ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5fa59c640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5fc06f760 0x7fe5fc15a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5fad9d640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5fc144630 0x7fe5fc0cea50 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009d60 tx=0x7fe5f4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5eaffd640 1 -- 172.21.15.81:0/298912508 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4033050 con 0x7fe5fc144630 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5eaffd640 1 -- 172.21.15.81:0/298912508 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5f4004550 con 0x7fe5fc144630 2024-06-29T15:38:36.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe600fac640 1 -- 172.21.15.81:0/298912508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5fc0cdb50 con 0x7fe5fc144630 2024-06-29T15:38:36.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe5eaffd640 1 -- 172.21.15.81:0/298912508 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4038450 con 0x7fe5fc144630 2024-06-29T15:38:36.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.834+0000 7fe600fac640 1 -- 172.21.15.81:0/298912508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5fc0ce110 con 0x7fe5fc144630 2024-06-29T15:38:36.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.835+0000 7fe600fac640 1 -- 172.21.15.81:0/298912508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5fc0cc610 con 0x7fe5fc144630 2024-06-29T15:38:36.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.836+0000 7fe5eaffd640 1 -- 172.21.15.81:0/298912508 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fe5f4036070 con 0x7fe5fc144630 2024-06-29T15:38:36.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.836+0000 7fe5eaffd640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe5dc0763f0 0x7fe5dc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:36.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.836+0000 7fe5eaffd640 1 -- 172.21.15.81:0/298912508 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 4957+0+0 (secure 0 0 0) 0x7fe5f40b8d20 con 0x7fe5fc144630 2024-06-29T15:38:36.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.837+0000 7fe5fa59c640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe5dc0763f0 0x7fe5dc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:36.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.837+0000 7fe5fa59c640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe5dc0763f0 0x7fe5dc078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec006fd0 tx=0x7fe5ec008040 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:36.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:36.841+0000 7fe5eaffd640 1 -- 172.21.15.81:0/298912508 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe5fc0cc610 con 0x7fe5fc144630 2024-06-29T15:38:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.023+0000 7fe600fac640 1 -- 172.21.15.81:0/298912508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe5fc06fb70 con 0x7fe5fc144630 2024-06-29T15:38:37.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.339+0000 7fe5eaffd640 1 -- 172.21.15.81:0/298912508 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v165) ==== 138+0+0 (secure 0 0 0) 0x7fe5fc0700b0 con 0x7fe5fc144630 2024-06-29T15:38:37.341 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T15:38:37.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 -- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe5dc0763f0 msgr2=0x7fe5dc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:37.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe5dc0763f0 0x7fe5dc078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec006fd0 tx=0x7fe5ec008040 comp rx=0 tx=0).stop 2024-06-29T15:38:37.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 -- 172.21.15.81:0/298912508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5fc144630 msgr2=0x7fe5fc0cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:37.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5fc144630 0x7fe5fc0cea50 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009d60 tx=0x7fe5f4003f10 comp rx=0 tx=0).stop 2024-06-29T15:38:37.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 -- 172.21.15.81:0/298912508 shutdown_connections 2024-06-29T15:38:37.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe5dc0763f0 0x7fe5dc078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5fc144630 0x7fe5fc0cea50 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5fc0b4130 0x7fe5fc0ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 --2- 172.21.15.81:0/298912508 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5fc06f760 0x7fe5fc15a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 -- 172.21.15.81:0/298912508 >> 172.21.15.81:0/298912508 conn(0x7fe5fc06cd60 msgr2=0x7fe5fc140250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:37.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 -- 172.21.15.81:0/298912508 shutdown_connections 2024-06-29T15:38:37.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.342+0000 7fe600fac640 1 -- 172.21.15.81:0/298912508 wait complete. 2024-06-29T15:38:37.358 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:38:37.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.477+0000 7f7280ccc640 1 -- 172.21.15.81:0/3449882769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727c145530 msgr2=0x7f727c073280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:37.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.477+0000 7f7280ccc640 1 --2- 172.21.15.81:0/3449882769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727c145530 0x7f727c073280 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f726c005fd0 tx=0x7f726c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.478+0000 7f7280ccc640 1 -- 172.21.15.81:0/3449882769 shutdown_connections 2024-06-29T15:38:37.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.478+0000 7f7280ccc640 1 --2- 172.21.15.81:0/3449882769 >> v2:172.21.15.167:3300/0 conn(0x7f727c15cd00 0x7f727c15d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.478+0000 7f7280ccc640 1 --2- 172.21.15.81:0/3449882769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727c145530 0x7f727c073280 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.478+0000 7f7280ccc640 1 -- 172.21.15.81:0/3449882769 >> 172.21.15.81:0/3449882769 conn(0x7f727c06ccc0 msgr2=0x7f727c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:37.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.478+0000 7f7280ccc640 1 -- 172.21.15.81:0/3449882769 shutdown_connections 2024-06-29T15:38:37.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.478+0000 7f7280ccc640 1 -- 172.21.15.81:0/3449882769 wait complete. 2024-06-29T15:38:37.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.479+0000 7f7280ccc640 1 Processor -- start 2024-06-29T15:38:37.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.480+0000 7f7280ccc640 1 -- start start 2024-06-29T15:38:37.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7280ccc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727c0737c0 0x7f727c13cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:37.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7280ccc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f727c145530 0x7f727c139390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:37.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7280ccc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727c15cd00 0x7f727c139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7280ccc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f727c15dbd0 con 0x7f727c145530 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7280ccc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f727c15da50 con 0x7f727c0737c0 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7280ccc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f727c15dd50 con 0x7f727c15cd00 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7279d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f727c145530 0x7f727c139390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7279d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f727c145530 0x7f727c139390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35270/0 (socket says 172.21.15.81:35270) 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7279d9b640 1 -- 172.21.15.81:0/296512786 learned_addr learned my addr 172.21.15.81:0/296512786 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f727ad9d640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727c15cd00 0x7f727c139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f727a59c640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727c0737c0 0x7f727c13cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7279d9b640 1 -- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727c15cd00 msgr2=0x7f727c139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7279d9b640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727c15cd00 0x7f727c139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7279d9b640 1 -- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727c0737c0 msgr2=0x7f727c13cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7279d9b640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727c0737c0 0x7f727c13cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:37.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7279d9b640 1 -- 172.21.15.81:0/296512786 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727c1398d0 con 0x7f727c145530 2024-06-29T15:38:37.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f727a59c640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727c0737c0 0x7f727c13cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:37.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.481+0000 7f7279d9b640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f727c145530 0x7f727c139390 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f727000c430 tx=0x7f727000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:37.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.482+0000 7f726affd640 1 -- 172.21.15.81:0/296512786 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270011790 con 0x7f727c145530 2024-06-29T15:38:37.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.482+0000 7f726affd640 1 -- 172.21.15.81:0/296512786 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7270017070 con 0x7f727c145530 2024-06-29T15:38:37.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.482+0000 7f726affd640 1 -- 172.21.15.81:0/296512786 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270011ca0 con 0x7f727c145530 2024-06-29T15:38:37.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.482+0000 7f7280ccc640 1 -- 172.21.15.81:0/296512786 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f727c157180 con 0x7f727c145530 2024-06-29T15:38:37.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.482+0000 7f7280ccc640 1 -- 172.21.15.81:0/296512786 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f727c157740 con 0x7f727c145530 2024-06-29T15:38:37.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.483+0000 7f7280ccc640 1 -- 172.21.15.81:0/296512786 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f727c15e290 con 0x7f727c145530 2024-06-29T15:38:37.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.483+0000 7f726affd640 1 -- 172.21.15.81:0/296512786 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f727001c460 con 0x7f727c145530 2024-06-29T15:38:37.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.484+0000 7f726affd640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f725c0763f0 0x7f725c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:37.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.484+0000 7f726affd640 1 -- 172.21.15.81:0/296512786 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 5327+0+0 (secure 0 0 0) 0x7f7270098010 con 0x7f727c145530 2024-06-29T15:38:37.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.484+0000 7f727a59c640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f725c0763f0 0x7f725c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:37.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.485+0000 7f727a59c640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f725c0763f0 0x7f725c078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f726c034000 tx=0x7f726c004ab0 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:37.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.488+0000 7f726affd640 1 -- 172.21.15.81:0/296512786 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f727c073bd0 con 0x7f727c145530 2024-06-29T15:38:37.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:37.668+0000 7f7280ccc640 1 -- 172.21.15.81:0/296512786 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f727c156f10 con 0x7f727c145530 2024-06-29T15:38:38.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.350+0000 7f726affd640 1 -- 172.21.15.81:0/296512786 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v166) ==== 170+0+0 (secure 0 0 0) 0x7f72700619e0 con 0x7f727c145530 2024-06-29T15:38:38.352 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 -- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f725c0763f0 msgr2=0x7f725c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f725c0763f0 0x7f725c078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f726c034000 tx=0x7f726c004ab0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 -- 172.21.15.81:0/296512786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f727c145530 msgr2=0x7f727c139390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f727c145530 0x7f727c139390 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f727000c430 tx=0x7f727000c830 comp rx=0 tx=0).stop 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 -- 172.21.15.81:0/296512786 shutdown_connections 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f725c0763f0 0x7f725c078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f727c15cd00 0x7f727c139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f727c145530 0x7f727c139390 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 --2- 172.21.15.81:0/296512786 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f727c0737c0 0x7f727c13cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 -- 172.21.15.81:0/296512786 >> 172.21.15.81:0/296512786 conn(0x7f727c06ccc0 msgr2=0x7f727c0b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 -- 172.21.15.81:0/296512786 shutdown_connections 2024-06-29T15:38:38.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.353+0000 7f7280ccc640 1 -- 172.21.15.81:0/296512786 wait complete. 2024-06-29T15:38:38.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:38:38.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.501+0000 7f1adb2a4640 1 -- 172.21.15.81:0/2764863277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc003a50 msgr2=0x7f1abc003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.501+0000 7f1adb2a4640 1 --2- 172.21.15.81:0/2764863277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc003a50 0x7f1abc003ec0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40013d0 tx=0x7f1abc005830 comp rx=0 tx=0).stop 2024-06-29T15:38:38.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.501+0000 7f1adb2a4640 1 -- 172.21.15.81:0/2764863277 shutdown_connections 2024-06-29T15:38:38.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.501+0000 7f1adb2a4640 1 --2- 172.21.15.81:0/2764863277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc003a50 0x7f1abc003ec0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.501+0000 7f1adb2a4640 1 -- 172.21.15.81:0/2764863277 >> 172.21.15.81:0/2764863277 conn(0x7f1ad409cf30 msgr2=0x7f1ad40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:38.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.501+0000 7f1adb2a4640 1 -- 172.21.15.81:0/2764863277 shutdown_connections 2024-06-29T15:38:38.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.501+0000 7f1adb2a4640 1 -- 172.21.15.81:0/2764863277 wait complete. 2024-06-29T15:38:38.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.503+0000 7f1adb2a4640 1 Processor -- start 2024-06-29T15:38:38.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.503+0000 7f1adb2a4640 1 -- start start 2024-06-29T15:38:38.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1adb2a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc003a50 0x7f1ad40a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:38.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1adb2a4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ad409ebe0 0x7f1ad409eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:38.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1adb2a4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ad41f4b30 0x7f1ad41f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:38.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1adb2a4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad40d0860 con 0x7f1abc003a50 2024-06-29T15:38:38.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1adb2a4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad40d06e0 con 0x7f1ad409ebe0 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1adb2a4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad40d09e0 con 0x7f1ad41f4b30 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1ad9040640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc003a50 0x7f1ad40a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1ad9040640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc003a50 0x7f1ad40a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35274/0 (socket says 172.21.15.81:35274) 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1ad9040640 1 -- 172.21.15.81:0/3574112637 learned_addr learned my addr 172.21.15.81:0/3574112637 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1ad883f640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ad409ebe0 0x7f1ad409eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1ad9841640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ad41f4b30 0x7f1ad41f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1ad9040640 1 -- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ad41f4b30 msgr2=0x7f1ad41f4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1ad9040640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ad41f4b30 0x7f1ad41f4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1ad9040640 1 -- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ad409ebe0 msgr2=0x7f1ad409eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1ad9040640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ad409ebe0 0x7f1ad409eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.504+0000 7f1ad9040640 1 -- 172.21.15.81:0/3574112637 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad40b0820 con 0x7f1abc003a50 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.505+0000 7f1ad9040640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc003a50 0x7f1ad40a0a50 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f1ac400c700 tx=0x7f1ac4004c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.505+0000 7f1ad1ffb640 1 -- 172.21.15.81:0/3574112637 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4039070 con 0x7f1abc003a50 2024-06-29T15:38:38.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.505+0000 7f1ad1ffb640 1 -- 172.21.15.81:0/3574112637 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ac402bc60 con 0x7f1abc003a50 2024-06-29T15:38:38.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.505+0000 7f1adb2a4640 1 -- 172.21.15.81:0/3574112637 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad40a1350 con 0x7f1abc003a50 2024-06-29T15:38:38.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.505+0000 7f1ad1ffb640 1 -- 172.21.15.81:0/3574112637 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac40062b0 con 0x7f1abc003a50 2024-06-29T15:38:38.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.505+0000 7f1adb2a4640 1 -- 172.21.15.81:0/3574112637 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad40a1910 con 0x7f1abc003a50 2024-06-29T15:38:38.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.506+0000 7f1adb2a4640 1 -- 172.21.15.81:0/3574112637 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad40b09f0 con 0x7f1abc003a50 2024-06-29T15:38:38.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.507+0000 7f1ad1ffb640 1 -- 172.21.15.81:0/3574112637 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f1ac4006450 con 0x7f1abc003a50 2024-06-29T15:38:38.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.507+0000 7f1ad1ffb640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1ac00763f0 0x7f1ac0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:38.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.507+0000 7f1ad1ffb640 1 -- 172.21.15.81:0/3574112637 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 5709+0+0 (secure 0 0 0) 0x7f1ac40b7d40 con 0x7f1abc003a50 2024-06-29T15:38:38.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.508+0000 7f1ad883f640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1ac00763f0 0x7f1ac0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:38.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.508+0000 7f1ad883f640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1ac00763f0 0x7f1ac0078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1ad40cac60 tx=0x7f1ac8006e10 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:38.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.511+0000 7f1ad1ffb640 1 -- 172.21.15.81:0/3574112637 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1ac4004e20 con 0x7f1abc003a50 2024-06-29T15:38:38.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:38.729+0000 7f1adb2a4640 1 -- 172.21.15.81:0/3574112637 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f1ad415e590 con 0x7f1abc003a50 2024-06-29T15:38:39.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.367+0000 7f1ad1ffb640 1 -- 172.21.15.81:0/3574112637 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v82) ==== 174+0+0 (secure 0 0 0) 0x7f1ac4081700 con 0x7f1abc003a50 2024-06-29T15:38:39.369 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 20 and data pool 21 2024-06-29T15:38:39.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.372+0000 7f1adb2a4640 1 -- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1ac00763f0 msgr2=0x7f1ac0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.372+0000 7f1adb2a4640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1ac00763f0 0x7f1ac0078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1ad40cac60 tx=0x7f1ac8006e10 comp rx=0 tx=0).stop 2024-06-29T15:38:39.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.372+0000 7f1adb2a4640 1 -- 172.21.15.81:0/3574112637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc003a50 msgr2=0x7f1ad40a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.372+0000 7f1adb2a4640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc003a50 0x7f1ad40a0a50 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f1ac400c700 tx=0x7f1ac4004c30 comp rx=0 tx=0).stop 2024-06-29T15:38:39.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.374+0000 7f1adb2a4640 1 -- 172.21.15.81:0/3574112637 shutdown_connections 2024-06-29T15:38:39.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.374+0000 7f1adb2a4640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1ac00763f0 0x7f1ac0078900 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1ad40cac60 tx=0x7f1ac8006e10 comp rx=0 tx=0).stop 2024-06-29T15:38:39.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.374+0000 7f1adb2a4640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ad41f4b30 0x7f1ad41f4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.374+0000 7f1adb2a4640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ad409ebe0 0x7f1ad409eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.374+0000 7f1adb2a4640 1 --2- 172.21.15.81:0/3574112637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc003a50 0x7f1ad40a0a50 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.374+0000 7f1adb2a4640 1 -- 172.21.15.81:0/3574112637 >> 172.21.15.81:0/3574112637 conn(0x7f1ad409cf30 msgr2=0x7f1ad415f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:39.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.375+0000 7f1adb2a4640 1 -- 172.21.15.81:0/3574112637 shutdown_connections 2024-06-29T15:38:39.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.375+0000 7f1adb2a4640 1 -- 172.21.15.81:0/3574112637 wait complete. 2024-06-29T15:38:39.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:39.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.506+0000 7f66ec23c640 1 -- 172.21.15.81:0/3582073759 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66e40b7290 msgr2=0x7f66e40b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.506+0000 7f66ec23c640 1 --2- 172.21.15.81:0/3582073759 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66e40b7290 0x7f66e40b76a0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f66e408e230 tx=0x7f66dc02a5f0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.506+0000 7f66ec23c640 1 -- 172.21.15.81:0/3582073759 shutdown_connections 2024-06-29T15:38:39.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.506+0000 7f66ec23c640 1 --2- 172.21.15.81:0/3582073759 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66e40b7290 0x7f66e40b76a0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.506+0000 7f66ec23c640 1 --2- 172.21.15.81:0/3582073759 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66e40b5250 0x7f66e40d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.506+0000 7f66ec23c640 1 --2- 172.21.15.81:0/3582073759 >> v2:172.21.15.167:3300/0 conn(0x7f66e40b9040 0x7f66e40d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.506+0000 7f66ec23c640 1 -- 172.21.15.81:0/3582073759 >> 172.21.15.81:0/3582073759 conn(0x7f66e409cf30 msgr2=0x7f66e40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:39.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.506+0000 7f66ec23c640 1 -- 172.21.15.81:0/3582073759 shutdown_connections 2024-06-29T15:38:39.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.506+0000 7f66ec23c640 1 -- 172.21.15.81:0/3582073759 wait complete. 2024-06-29T15:38:39.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.508+0000 7f66ec23c640 1 Processor -- start 2024-06-29T15:38:39.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.508+0000 7f66ec23c640 1 -- start start 2024-06-29T15:38:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.509+0000 7f66ec23c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66e40b5250 0x7f66e40a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.509+0000 7f66ec23c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66e40b7290 0x7f66e414f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.509+0000 7f66ec23c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66e40b9040 0x7f66e41f5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.509+0000 7f66ec23c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f66e40b7ff0 con 0x7f66e40b7290 2024-06-29T15:38:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.509+0000 7f66ec23c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f66e40b7e70 con 0x7f66e40b5250 2024-06-29T15:38:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.509+0000 7f66ec23c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f66e40b8170 con 0x7f66e40b9040 2024-06-29T15:38:39.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.509+0000 7f66e97d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66e40b7290 0x7f66e414f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.509+0000 7f66e97d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66e40b7290 0x7f66e414f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35300/0 (socket says 172.21.15.81:35300) 2024-06-29T15:38:39.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e97d7640 1 -- 172.21.15.81:0/3663357249 learned_addr learned my addr 172.21.15.81:0/3663357249 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:39.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e9fd8640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66e40b5250 0x7f66e40a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66ea7d9640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66e40b9040 0x7f66e41f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e97d7640 1 -- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66e40b9040 msgr2=0x7f66e41f5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e97d7640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66e40b9040 0x7f66e41f5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e97d7640 1 -- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66e40b5250 msgr2=0x7f66e40a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e97d7640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66e40b5250 0x7f66e40a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e97d7640 1 -- 172.21.15.81:0/3663357249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e409f970 con 0x7f66e40b7290 2024-06-29T15:38:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e9fd8640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66e40b5250 0x7f66e40a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e97d7640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66e40b7290 0x7f66e414f960 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f66d800d650 tx=0x7f66d800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:39.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e27fc640 1 -- 172.21.15.81:0/3663357249 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8017070 con 0x7f66e40b7290 2024-06-29T15:38:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e27fc640 1 -- 172.21.15.81:0/3663357249 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66d8009b10 con 0x7f66e40b7290 2024-06-29T15:38:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66ec23c640 1 -- 172.21.15.81:0/3663357249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e409fc50 con 0x7f66e40b7290 2024-06-29T15:38:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66e27fc640 1 -- 172.21.15.81:0/3663357249 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8010460 con 0x7f66e40b7290 2024-06-29T15:38:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.510+0000 7f66ec23c640 1 -- 172.21.15.81:0/3663357249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e40a0210 con 0x7f66e40b7290 2024-06-29T15:38:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.511+0000 7f66ec23c640 1 -- 172.21.15.81:0/3663357249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66e409f6e0 con 0x7f66e40b7290 2024-06-29T15:38:39.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.512+0000 7f66e27fc640 1 -- 172.21.15.81:0/3663357249 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f66d801a070 con 0x7f66e40b7290 2024-06-29T15:38:39.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.513+0000 7f66e27fc640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f66d0076420 0x7f66d0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.513+0000 7f66e27fc640 1 -- 172.21.15.81:0/3663357249 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 5809+0+0 (secure 0 0 0) 0x7f66d8097e00 con 0x7f66e40b7290 2024-06-29T15:38:39.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.513+0000 7f66e9fd8640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f66d0076420 0x7f66d0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.513+0000 7f66e9fd8640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f66d0076420 0x7f66d0078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f66d4005c40 tx=0x7f66d4005bd0 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:39.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.516+0000 7f66e27fc640 1 -- 172.21.15.81:0/3663357249 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f66d8060f60 con 0x7f66e40b7290 2024-06-29T15:38:39.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.696+0000 7f66ec23c640 1 -- 172.21.15.81:0/3663357249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66e40b76a0 con 0x7f66e40b7290 2024-06-29T15:38:39.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.697+0000 7f66e27fc640 1 -- 172.21.15.81:0/3663357249 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) ==== 74+0+16839 (secure 0 0 0) 0x7f66d8065260 con 0x7f66e40b7290 2024-06-29T15:38:39.699 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:39.699 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":167,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:38.731738+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:37.031836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.676381+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:39.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 -- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f66d0076420 msgr2=0x7f66d0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f66d0076420 0x7f66d0078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f66d4005c40 tx=0x7f66d4005bd0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 -- 172.21.15.81:0/3663357249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66e40b7290 msgr2=0x7f66e414f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66e40b7290 0x7f66e414f960 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f66d800d650 tx=0x7f66d800db10 comp rx=0 tx=0).stop 2024-06-29T15:38:39.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 -- 172.21.15.81:0/3663357249 shutdown_connections 2024-06-29T15:38:39.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f66d0076420 0x7f66d0078930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f66e40b9040 0x7f66e41f5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66e40b7290 0x7f66e414f960 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 --2- 172.21.15.81:0/3663357249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f66e40b5250 0x7f66e40a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 -- 172.21.15.81:0/3663357249 >> 172.21.15.81:0/3663357249 conn(0x7f66e409cf30 msgr2=0x7f66e40cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:39.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 -- 172.21.15.81:0/3663357249 shutdown_connections 2024-06-29T15:38:39.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.699+0000 7f66ec23c640 1 -- 172.21.15.81:0/3663357249 wait complete. 2024-06-29T15:38:39.712 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:39.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.824+0000 7f9b586fe640 1 -- 172.21.15.81:0/1023050953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b500d0b90 msgr2=0x7f9b500d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.824+0000 7f9b586fe640 1 --2- 172.21.15.81:0/1023050953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b500d0b90 0x7f9b500d1a80 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f9b5008e330 tx=0x7f9b4002a680 comp rx=0 tx=0).stop 2024-06-29T15:38:39.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.824+0000 7f9b586fe640 1 -- 172.21.15.81:0/1023050953 shutdown_connections 2024-06-29T15:38:39.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.824+0000 7f9b586fe640 1 --2- 172.21.15.81:0/1023050953 >> v2:172.21.15.167:3300/0 conn(0x7f9b500b5250 0x7f9b500d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.824+0000 7f9b586fe640 1 --2- 172.21.15.81:0/1023050953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b500d0b90 0x7f9b500d1a80 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.824+0000 7f9b586fe640 1 -- 172.21.15.81:0/1023050953 >> 172.21.15.81:0/1023050953 conn(0x7f9b5009cf30 msgr2=0x7f9b500d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:39.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.824+0000 7f9b586fe640 1 -- 172.21.15.81:0/1023050953 shutdown_connections 2024-06-29T15:38:39.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.824+0000 7f9b586fe640 1 -- 172.21.15.81:0/1023050953 wait complete. 2024-06-29T15:38:39.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.826+0000 7f9b586fe640 1 Processor -- start 2024-06-29T15:38:39.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.826+0000 7f9b586fe640 1 -- start start 2024-06-29T15:38:39.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b586fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b500b5250 0x7f9b500b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b586fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b500b9040 0x7f9b501e9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b586fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b500b4750 0x7f9b501e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b586fe640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9b500d2500 con 0x7f9b500b5250 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b586fe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9b500b8530 con 0x7f9b500b4750 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b586fe640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9b500d2680 con 0x7f9b500b9040 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b5649a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b500b5250 0x7f9b500b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b5649a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b500b5250 0x7f9b500b1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35312/0 (socket says 172.21.15.81:35312) 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b5649a640 1 -- 172.21.15.81:0/3118769360 learned_addr learned my addr 172.21.15.81:0/3118769360 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b55c99640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b500b9040 0x7f9b501e9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.827+0000 7f9b56c9b640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b500b4750 0x7f9b501e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b5649a640 1 -- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b500b9040 msgr2=0x7f9b501e9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b5649a640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b500b9040 0x7f9b501e9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b5649a640 1 -- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b500b4750 msgr2=0x7f9b501e9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b5649a640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b500b4750 0x7f9b501e9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b5649a640 1 -- 172.21.15.81:0/3118769360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b500a0030 con 0x7f9b500b5250 2024-06-29T15:38:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b5649a640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b500b5250 0x7f9b500b1200 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00c3a0 tx=0x7f9b4c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b46ffd640 1 -- 172.21.15.81:0/3118769360 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c017070 con 0x7f9b500b5250 2024-06-29T15:38:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b46ffd640 1 -- 172.21.15.81:0/3118769360 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b4c0053c0 con 0x7f9b500b5250 2024-06-29T15:38:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b46ffd640 1 -- 172.21.15.81:0/3118769360 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c003850 con 0x7f9b500b5250 2024-06-29T15:38:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b586fe640 1 -- 172.21.15.81:0/3118769360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b500a0310 con 0x7f9b500b5250 2024-06-29T15:38:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.828+0000 7f9b586fe640 1 -- 172.21.15.81:0/3118769360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b5009fab0 con 0x7f9b500b5250 2024-06-29T15:38:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.829+0000 7f9b586fe640 1 -- 172.21.15.81:0/3118769360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b500a04e0 con 0x7f9b500b5250 2024-06-29T15:38:39.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.830+0000 7f9b46ffd640 1 -- 172.21.15.81:0/3118769360 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f9b4c01a070 con 0x7f9b500b5250 2024-06-29T15:38:39.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.830+0000 7f9b46ffd640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9b34076420 0x7f9b34078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:39.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.830+0000 7f9b46ffd640 1 -- 172.21.15.81:0/3118769360 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 5809+0+0 (secure 0 0 0) 0x7f9b4c096c90 con 0x7f9b500b5250 2024-06-29T15:38:39.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.831+0000 7f9b55c99640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9b34076420 0x7f9b34078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:39.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.831+0000 7f9b55c99640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9b34076420 0x7f9b34078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b40033000 tx=0x7f9b40004730 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:39.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:39.835+0000 7f9b46ffd640 1 -- 172.21.15.81:0/3118769360 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9b4c05fe20 con 0x7f9b500b5250 2024-06-29T15:38:40.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.015+0000 7f9b586fe640 1 -- 172.21.15.81:0/3118769360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b500ce240 con 0x7f9b500b5250 2024-06-29T15:38:40.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.016+0000 7f9b46ffd640 1 -- 172.21.15.81:0/3118769360 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) ==== 74+0+16839 (secure 0 0 0) 0x7f9b500ce240 con 0x7f9b500b5250 2024-06-29T15:38:40.018 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:40.018 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":167,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:38.731738+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:37.031836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.676381+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 -- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9b34076420 msgr2=0x7f9b34078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9b34076420 0x7f9b34078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b40033000 tx=0x7f9b40004730 comp rx=0 tx=0).stop 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 -- 172.21.15.81:0/3118769360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b500b5250 msgr2=0x7f9b500b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b500b5250 0x7f9b500b1200 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c00c3a0 tx=0x7f9b4c00c860 comp rx=0 tx=0).stop 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 -- 172.21.15.81:0/3118769360 shutdown_connections 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9b34076420 0x7f9b34078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9b500b4750 0x7f9b501e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9b500b9040 0x7f9b501e9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 --2- 172.21.15.81:0/3118769360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b500b5250 0x7f9b500b1200 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 -- 172.21.15.81:0/3118769360 >> 172.21.15.81:0/3118769360 conn(0x7f9b5009cf30 msgr2=0x7f9b501610e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:40.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 -- 172.21.15.81:0/3118769360 shutdown_connections 2024-06-29T15:38:40.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.018+0000 7f9b586fe640 1 -- 172.21.15.81:0/3118769360 wait complete. 2024-06-29T15:38:40.031 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:38:40.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.148+0000 7f0fe1209640 1 -- 172.21.15.81:0/1577988010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fdc06f6a0 msgr2=0x7f0fdc15aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:40.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.148+0000 7f0fe1209640 1 --2- 172.21.15.81:0/1577988010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fdc06f6a0 0x7f0fdc15aed0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009830 tx=0x7f0fc402a510 comp rx=0 tx=0).stop 2024-06-29T15:38:40.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.148+0000 7f0fe1209640 1 -- 172.21.15.81:0/1577988010 shutdown_connections 2024-06-29T15:38:40.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.148+0000 7f0fe1209640 1 --2- 172.21.15.81:0/1577988010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fdc06f6a0 0x7f0fdc15aed0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:40.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.148+0000 7f0fe1209640 1 -- 172.21.15.81:0/1577988010 >> 172.21.15.81:0/1577988010 conn(0x7f0fdc0d0930 msgr2=0x7f0fdc0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:40.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.148+0000 7f0fe1209640 1 -- 172.21.15.81:0/1577988010 shutdown_connections 2024-06-29T15:38:40.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.148+0000 7f0fe1209640 1 -- 172.21.15.81:0/1577988010 wait complete. 2024-06-29T15:38:40.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.150+0000 7f0fe1209640 1 Processor -- start 2024-06-29T15:38:40.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.150+0000 7f0fe1209640 1 -- start start 2024-06-29T15:38:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fe1209640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fdc06f6a0 0x7f0fdc1e73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fe1209640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fdc0b3960 0x7f0fdc153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fe1209640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0b57b0 0x7f0fdc1e24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fe1209640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc0b4600 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fe1209640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc0b4480 con 0x7f0fdc0b57b0 2024-06-29T15:38:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fe1209640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0fdc0b4780 con 0x7f0fdc0b3960 2024-06-29T15:38:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fdad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fdc06f6a0 0x7f0fdc1e73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fdad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fdc06f6a0 0x7f0fdc1e73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53696/0 (socket says 172.21.15.81:53696) 2024-06-29T15:38:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fdad9d640 1 -- 172.21.15.81:0/295224907 learned_addr learned my addr 172.21.15.81:0/295224907 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fd259c640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fdc0b3960 0x7f0fdc153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fdb59e640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0b57b0 0x7f0fdc1e24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fdad9d640 1 -- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fdc0b3960 msgr2=0x7f0fdc153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fdad9d640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fdc0b3960 0x7f0fdc153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fdad9d640 1 -- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0b57b0 msgr2=0x7f0fdc1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fdad9d640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0b57b0 0x7f0fdc1e24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fdad9d640 1 -- 172.21.15.81:0/295224907 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fdc1f40a0 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.151+0000 7f0fdb59e640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0b57b0 0x7f0fdc1e24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.152+0000 7f0fdad9d640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fdc06f6a0 0x7f0fdc1e73c0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00b420 tx=0x7f0fcc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.152+0000 7f0fd3fff640 1 -- 172.21.15.81:0/295224907 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc011830 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.152+0000 7f0fd3fff640 1 -- 172.21.15.81:0/295224907 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0fcc016070 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.152+0000 7f0fe1209640 1 -- 172.21.15.81:0/295224907 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fdc1e6600 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.152+0000 7f0fd3fff640 1 -- 172.21.15.81:0/295224907 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc011d60 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.152+0000 7f0fe1209640 1 -- 172.21.15.81:0/295224907 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fdc1e6bc0 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.153+0000 7f0fe1209640 1 -- 172.21.15.81:0/295224907 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fdc0aff90 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.154+0000 7f0fd3fff640 1 -- 172.21.15.81:0/295224907 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f0fcc01b460 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.154+0000 7f0fd3fff640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f0fbc076420 0x7f0fbc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.154+0000 7f0fd3fff640 1 -- 172.21.15.81:0/295224907 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(167..167 src has 1..167) ==== 5809+0+0 (secure 0 0 0) 0x7f0fcc0974c0 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.155+0000 7f0fd259c640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f0fbc076420 0x7f0fbc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.155+0000 7f0fd259c640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f0fbc076420 0x7f0fbc078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4034000 tx=0x7f0fc4004730 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:40.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.159+0000 7f0fd3fff640 1 -- 172.21.15.81:0/295224907 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0fdc0aff90 con 0x7f0fdc06f6a0 2024-06-29T15:38:40.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:40.380+0000 7f0fe1209640 1 -- 172.21.15.81:0/295224907 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0fdc06e2a0 con 0x7f0fdc06f6a0 2024-06-29T15:38:41.338 INFO:tasks.ceph.mds.c.smithi081.stderr:2024-06-29T15:38:41.337+0000 7f0285804640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:38:41.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.384+0000 7f0fd3fff640 1 -- 172.21.15.81:0/295224907 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v85) ==== 118+0+0 (secure 0 0 0) 0x7f0fdc06e2a0 con 0x7f0fdc06f6a0 2024-06-29T15:38:41.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 -- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f0fbc076420 msgr2=0x7f0fbc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:41.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f0fbc076420 0x7f0fbc078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4034000 tx=0x7f0fc4004730 comp rx=0 tx=0).stop 2024-06-29T15:38:41.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 -- 172.21.15.81:0/295224907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fdc06f6a0 msgr2=0x7f0fdc1e73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:41.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fdc06f6a0 0x7f0fdc1e73c0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00b420 tx=0x7f0fcc00b8e0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 -- 172.21.15.81:0/295224907 shutdown_connections 2024-06-29T15:38:41.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f0fbc076420 0x7f0fbc078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0fdc0b57b0 0x7f0fdc1e24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0fdc0b3960 0x7f0fdc153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 --2- 172.21.15.81:0/295224907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fdc06f6a0 0x7f0fdc1e73c0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 -- 172.21.15.81:0/295224907 >> 172.21.15.81:0/295224907 conn(0x7f0fdc0d0930 msgr2=0x7f0fdc072f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:41.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 -- 172.21.15.81:0/295224907 shutdown_connections 2024-06-29T15:38:41.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.387+0000 7f0fe1209640 1 -- 172.21.15.81:0/295224907 wait complete. 2024-06-29T15:38:41.402 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:38:41.402 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:38:41.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.521+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/1252479305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9815c030 msgr2=0x7f7f9806c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:41.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.521+0000 7f7f9e0b2640 1 --2- 172.21.15.81:0/1252479305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9815c030 0x7f7f9806c2f0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f7f98074e30 tx=0x7f7f8c02a560 comp rx=0 tx=0).stop 2024-06-29T15:38:41.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.521+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/1252479305 shutdown_connections 2024-06-29T15:38:41.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.521+0000 7f7f9e0b2640 1 --2- 172.21.15.81:0/1252479305 >> v2:172.21.15.167:3300/0 conn(0x7f7f9815c440 0x7f7f9806c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.521+0000 7f7f9e0b2640 1 --2- 172.21.15.81:0/1252479305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9815c030 0x7f7f9806c2f0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.521+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/1252479305 >> 172.21.15.81:0/1252479305 conn(0x7f7f980d0930 msgr2=0x7f7f980d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:41.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.521+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/1252479305 shutdown_connections 2024-06-29T15:38:41.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.521+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/1252479305 wait complete. 2024-06-29T15:38:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.523+0000 7f7f9e0b2640 1 Processor -- start 2024-06-29T15:38:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.523+0000 7f7f9e0b2640 1 -- start start 2024-06-29T15:38:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.524+0000 7f7f9e0b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f9815c030 0x7f7f981474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.524+0000 7f7f9e0b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9815c440 0x7f7f98159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.524+0000 7f7f9e0b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f9815d750 0x7f7f980a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.524+0000 7f7f9e0b2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7f980cfca0 con 0x7f7f9815c440 2024-06-29T15:38:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.524+0000 7f7f9e0b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7f980cfb20 con 0x7f7f9815c030 2024-06-29T15:38:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.524+0000 7f7f9e0b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7f980cfe20 con 0x7f7f9815d750 2024-06-29T15:38:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.524+0000 7f7f96ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9815c440 0x7f7f98159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.524+0000 7f7f96ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9815c440 0x7f7f98159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53718/0 (socket says 172.21.15.81:53718) 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.524+0000 7f7f96ffd640 1 -- 172.21.15.81:0/614020659 learned_addr learned my addr 172.21.15.81:0/614020659 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f97fff640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f9815d750 0x7f7f980a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f96ffd640 1 -- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f9815d750 msgr2=0x7f7f980a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f96ffd640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f9815d750 0x7f7f980a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f96ffd640 1 -- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f9815c030 msgr2=0x7f7f981474c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f977fe640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f9815c030 0x7f7f981474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f96ffd640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f9815c030 0x7f7f981474c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f96ffd640 1 -- 172.21.15.81:0/614020659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f98158490 con 0x7f7f9815c440 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f96ffd640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9815c440 0x7f7f98159530 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0099c0 tx=0x7f7f8c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f7ffff640 1 -- 172.21.15.81:0/614020659 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c03e030 con 0x7f7f9815c440 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f7ffff640 1 -- 172.21.15.81:0/614020659 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f8c0047a0 con 0x7f7f9815c440 2024-06-29T15:38:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/614020659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f98158770 con 0x7f7f9815c440 2024-06-29T15:38:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f7ffff640 1 -- 172.21.15.81:0/614020659 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c035040 con 0x7f7f9815c440 2024-06-29T15:38:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.525+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/614020659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f98158d30 con 0x7f7f9815c440 2024-06-29T15:38:41.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.526+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/614020659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f9806e2a0 con 0x7f7f9815c440 2024-06-29T15:38:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.527+0000 7f7f7ffff640 1 -- 172.21.15.81:0/614020659 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f7f8c041030 con 0x7f7f9815c440 2024-06-29T15:38:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.527+0000 7f7f7ffff640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f7f840763f0 0x7f7f84078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.527+0000 7f7f7ffff640 1 -- 172.21.15.81:0/614020659 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f7f8c0b8340 con 0x7f7f9815c440 2024-06-29T15:38:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.528+0000 7f7f977fe640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f7f840763f0 0x7f7f84078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.528+0000 7f7f977fe640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f7f840763f0 0x7f7f84078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7f88007dc0 tx=0x7f7f880072f0 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.532+0000 7f7f7ffff640 1 -- 172.21.15.81:0/614020659 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7f9806e2a0 con 0x7f7f9815c440 2024-06-29T15:38:41.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:41.747+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/614020659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f7f980d02f0 con 0x7f7f9815c440 2024-06-29T15:38:42.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.385+0000 7f7f7ffff640 1 -- 172.21.15.81:0/614020659 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v86) ==== 122+0+0 (secure 0 0 0) 0x7f7f8c081980 con 0x7f7f9815c440 2024-06-29T15:38:42.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f7f840763f0 msgr2=0x7f7f84078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f7f840763f0 0x7f7f84078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7f88007dc0 tx=0x7f7f880072f0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/614020659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9815c440 msgr2=0x7f7f98159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9815c440 0x7f7f98159530 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c0099c0 tx=0x7f7f8c0039e0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/614020659 shutdown_connections 2024-06-29T15:38:42.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f7f840763f0 0x7f7f84078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f9815d750 0x7f7f980a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9815c440 0x7f7f98159530 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 --2- 172.21.15.81:0/614020659 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f9815c030 0x7f7f981474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/614020659 >> 172.21.15.81:0/614020659 conn(0x7f7f980d0930 msgr2=0x7f7f9806f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:42.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/614020659 shutdown_connections 2024-06-29T15:38:42.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.388+0000 7f7f9e0b2640 1 -- 172.21.15.81:0/614020659 wait complete. 2024-06-29T15:38:42.402 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:38:42.402 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:42.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.517+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/4031207592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e98073570 msgr2=0x7f1e9815e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.517+0000 7f1e9cbf5640 1 --2- 172.21.15.81:0/4031207592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e98073570 0x7f1e9815e960 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f1e980b2700 tx=0x7f1e9002aa30 comp rx=0 tx=0).stop 2024-06-29T15:38:42.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.517+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/4031207592 shutdown_connections 2024-06-29T15:38:42.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.517+0000 7f1e9cbf5640 1 --2- 172.21.15.81:0/4031207592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e98073570 0x7f1e9815e960 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.517+0000 7f1e9cbf5640 1 --2- 172.21.15.81:0/4031207592 >> v2:172.21.15.167:3300/0 conn(0x7f1e980cf4d0 0x7f1e98073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.517+0000 7f1e9cbf5640 1 --2- 172.21.15.81:0/4031207592 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e981412f0 0x7f1e980cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.517+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/4031207592 >> 172.21.15.81:0/4031207592 conn(0x7f1e9806ccc0 msgr2=0x7f1e980b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:42.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.517+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/4031207592 shutdown_connections 2024-06-29T15:38:42.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.517+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/4031207592 wait complete. 2024-06-29T15:38:42.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.519+0000 7f1e9cbf5640 1 Processor -- start 2024-06-29T15:38:42.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.519+0000 7f1e9cbf5640 1 -- start start 2024-06-29T15:38:42.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e9cbf5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e98073570 0x7f1e980ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e9cbf5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e980cf4d0 0x7f1e980ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e9cbf5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e981412f0 0x7f1e98071f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e9cbf5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9815f230 con 0x7f1e980cf4d0 2024-06-29T15:38:42.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e9cbf5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9815f0b0 con 0x7f1e98073570 2024-06-29T15:38:42.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e9cbf5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1e980b3210 con 0x7f1e981412f0 2024-06-29T15:38:42.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e8dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e980cf4d0 0x7f1e980ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e8dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e980cf4d0 0x7f1e980ad750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53738/0 (socket says 172.21.15.81:53738) 2024-06-29T15:38:42.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e8dd9b640 1 -- 172.21.15.81:0/2348775979 learned_addr learned my addr 172.21.15.81:0/2348775979 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e9659c640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e98073570 0x7f1e980ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.520+0000 7f1e96d9d640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e981412f0 0x7f1e98071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e8dd9b640 1 -- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e981412f0 msgr2=0x7f1e98071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e8dd9b640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e981412f0 0x7f1e98071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e8dd9b640 1 -- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e98073570 msgr2=0x7f1e980ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e8dd9b640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e98073570 0x7f1e980ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e8dd9b640 1 -- 172.21.15.81:0/2348775979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e980af330 con 0x7f1e980cf4d0 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e8dd9b640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e980cf4d0 0x7f1e980ad750 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000d750 tx=0x7f1e8000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e8f7fe640 1 -- 172.21.15.81:0/2348775979 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80017070 con 0x7f1e980cf4d0 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e8f7fe640 1 -- 172.21.15.81:0/2348775979 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e80009cc0 con 0x7f1e980cf4d0 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/2348775979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e981f3850 con 0x7f1e980cf4d0 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e8f7fe640 1 -- 172.21.15.81:0/2348775979 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e800106c0 con 0x7f1e980cf4d0 2024-06-29T15:38:42.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.521+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/2348775979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e981f3e10 con 0x7f1e980cf4d0 2024-06-29T15:38:42.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.522+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/2348775979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e98073030 con 0x7f1e980cf4d0 2024-06-29T15:38:42.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.523+0000 7f1e8f7fe640 1 -- 172.21.15.81:0/2348775979 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f1e8001a070 con 0x7f1e980cf4d0 2024-06-29T15:38:42.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.523+0000 7f1e8f7fe640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1e780763f0 0x7f1e78078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.524+0000 7f1e8f7fe640 1 -- 172.21.15.81:0/2348775979 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f1e80097a50 con 0x7f1e980cf4d0 2024-06-29T15:38:42.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.524+0000 7f1e9659c640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1e780763f0 0x7f1e78078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.524+0000 7f1e9659c640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1e780763f0 0x7f1e78078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1e88004630 tx=0x7f1e8800b2e0 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:42.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.528+0000 7f1e8f7fe640 1 -- 172.21.15.81:0/2348775979 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1e8000de60 con 0x7f1e980cf4d0 2024-06-29T15:38:42.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.746+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/2348775979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e980aee80 con 0x7f1e980cf4d0 2024-06-29T15:38:42.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.747+0000 7f1e8f7fe640 1 -- 172.21.15.81:0/2348775979 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5233 (secure 0 0 0) 0x7f1e80060a40 con 0x7f1e980cf4d0 2024-06-29T15:38:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:42.749 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":86,"btime":"2024-06-29T15:38:42:386002+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7754,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3261744821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3261744821},{"type":"v1","addr":"172.21.15.167:6836","nonce":3261744821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7757,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1617521107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1617521107},{"type":"v1","addr":"172.21.15.167:6837","nonce":1617521107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":9636,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3224768380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3224768380},{"type":"v1","addr":"172.21.15.81:6833","nonce":3224768380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.731697+0000","modified":"2024-06-29T15:38:41.749109+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9639":{"gid":9639,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/1634290596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1634290596},{"type":"v1","addr":"172.21.15.81:6835","nonce":1634290596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9639,"qdb_cluster":[9639]},"id":10}]} 2024-06-29T15:38:42.749 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 86 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1e780763f0 msgr2=0x7f1e78078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1e780763f0 0x7f1e78078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1e88004630 tx=0x7f1e8800b2e0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/2348775979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e980cf4d0 msgr2=0x7f1e980ad750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e980cf4d0 0x7f1e980ad750 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000d750 tx=0x7f1e8000dc10 comp rx=0 tx=0).stop 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/2348775979 shutdown_connections 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1e780763f0 0x7f1e78078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e981412f0 0x7f1e98071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e980cf4d0 0x7f1e980ad750 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 --2- 172.21.15.81:0/2348775979 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e98073570 0x7f1e980ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/2348775979 >> 172.21.15.81:0/2348775979 conn(0x7f1e9806ccc0 msgr2=0x7f1e980af5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/2348775979 shutdown_connections 2024-06-29T15:38:42.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.749+0000 7f1e9cbf5640 1 -- 172.21.15.81:0/2348775979 wait complete. 2024-06-29T15:38:42.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.874+0000 7ff7a509d640 1 -- 172.21.15.81:0/786863633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a0074520 msgr2=0x7ff7a006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.874+0000 7ff7a509d640 1 --2- 172.21.15.81:0/786863633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a0074520 0x7ff7a006f140 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff790005fd0 tx=0x7ff79002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.874+0000 7ff7a509d640 1 -- 172.21.15.81:0/786863633 shutdown_connections 2024-06-29T15:38:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.874+0000 7ff7a509d640 1 --2- 172.21.15.81:0/786863633 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a006fa90 0x7ff7a006ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.874+0000 7ff7a509d640 1 --2- 172.21.15.81:0/786863633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a0074520 0x7ff7a006f140 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.874+0000 7ff7a509d640 1 -- 172.21.15.81:0/786863633 >> 172.21.15.81:0/786863633 conn(0x7ff7a0072950 msgr2=0x7ff7a00b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.874+0000 7ff7a509d640 1 -- 172.21.15.81:0/786863633 shutdown_connections 2024-06-29T15:38:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.874+0000 7ff7a509d640 1 -- 172.21.15.81:0/786863633 wait complete. 2024-06-29T15:38:42.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.876+0000 7ff7a509d640 1 Processor -- start 2024-06-29T15:38:42.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.876+0000 7ff7a509d640 1 -- start start 2024-06-29T15:38:42.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff7a509d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a006f680 0x7ff7a0155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff7a509d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7a006fa90 0x7ff7a00992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff7a509d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a0074520 0x7ff7a0159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff7a509d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a015b010 con 0x7ff7a006fa90 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff7a509d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a006d150 con 0x7ff7a006f680 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff7a509d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a015b190 con 0x7ff7a0074520 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff79659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7a006fa90 0x7ff7a00992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff79659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7a006fa90 0x7ff7a00992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53750/0 (socket says 172.21.15.81:53750) 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff79659c640 1 -- 172.21.15.81:0/292211279 learned_addr learned my addr 172.21.15.81:0/292211279 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff79f59e640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a0074520 0x7ff7a0159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.877+0000 7ff79ed9d640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a006f680 0x7ff7a0155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff79659c640 1 -- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a0074520 msgr2=0x7ff7a0159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff79659c640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a0074520 0x7ff7a0159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff79659c640 1 -- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a006f680 msgr2=0x7ff7a0155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff79659c640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a006f680 0x7ff7a0155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff79659c640 1 -- 172.21.15.81:0/292211279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a01513d0 con 0x7ff7a006fa90 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff79659c640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7a006fa90 0x7ff7a00992f0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7ff78800c400 tx=0x7ff78800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff797fff640 1 -- 172.21.15.81:0/292211279 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff788017070 con 0x7ff7a006fa90 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff797fff640 1 -- 172.21.15.81:0/292211279 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7880053c0 con 0x7ff7a006fa90 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff7a509d640 1 -- 172.21.15.81:0/292211279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a01516b0 con 0x7ff7a006fa90 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff797fff640 1 -- 172.21.15.81:0/292211279 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff788003850 con 0x7ff7a006fa90 2024-06-29T15:38:42.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.878+0000 7ff7a509d640 1 -- 172.21.15.81:0/292211279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a0151c70 con 0x7ff7a006fa90 2024-06-29T15:38:42.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.879+0000 7ff7a509d640 1 -- 172.21.15.81:0/292211279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7a0152050 con 0x7ff7a006fa90 2024-06-29T15:38:42.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.880+0000 7ff797fff640 1 -- 172.21.15.81:0/292211279 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7ff78801a070 con 0x7ff7a006fa90 2024-06-29T15:38:42.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.880+0000 7ff797fff640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff780076420 0x7ff780078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:42.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.880+0000 7ff797fff640 1 -- 172.21.15.81:0/292211279 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7ff788096d10 con 0x7ff7a006fa90 2024-06-29T15:38:42.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.880+0000 7ff79ed9d640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff780076420 0x7ff780078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:42.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.881+0000 7ff79ed9d640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff780076420 0x7ff780078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff790033000 tx=0x7ff790004730 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:42.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:42.885+0000 7ff797fff640 1 -- 172.21.15.81:0/292211279 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff78805fe20 con 0x7ff7a006fa90 2024-06-29T15:38:43.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.065+0000 7ff7a509d640 1 -- 172.21.15.81:0/292211279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7a00af4d0 con 0x7ff7a006fa90 2024-06-29T15:38:43.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.066+0000 7ff797fff640 1 -- 172.21.15.81:0/292211279 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) ==== 74+0+16826 (secure 0 0 0) 0x7ff788064120 con 0x7ff7a006fa90 2024-06-29T15:38:43.067 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:43.068 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":168,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:40.353217+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:37.031836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.676381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:43.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 -- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff780076420 msgr2=0x7ff780078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff780076420 0x7ff780078930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff790033000 tx=0x7ff790004730 comp rx=0 tx=0).stop 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 -- 172.21.15.81:0/292211279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7a006fa90 msgr2=0x7ff7a00992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7a006fa90 0x7ff7a00992f0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7ff78800c400 tx=0x7ff78800c8c0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 -- 172.21.15.81:0/292211279 shutdown_connections 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff780076420 0x7ff780078930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a0074520 0x7ff7a0159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7a006fa90 0x7ff7a00992f0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 --2- 172.21.15.81:0/292211279 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a006f680 0x7ff7a0155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 -- 172.21.15.81:0/292211279 >> 172.21.15.81:0/292211279 conn(0x7ff7a0072950 msgr2=0x7ff7a00b0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 -- 172.21.15.81:0/292211279 shutdown_connections 2024-06-29T15:38:43.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.068+0000 7ff7a509d640 1 -- 172.21.15.81:0/292211279 wait complete. 2024-06-29T15:38:43.081 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:38:43.081 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:38:43.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.194+0000 7f21da7eb640 1 -- 172.21.15.81:0/1351961572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21bc004800 msgr2=0x7f21bc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.194+0000 7f21da7eb640 1 --2- 172.21.15.81:0/1351961572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21bc004800 0x7f21bc004c10 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f21c800baf0 tx=0x7f21bc005830 comp rx=0 tx=0).stop 2024-06-29T15:38:43.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.194+0000 7f21d8d88640 1 -- 172.21.15.81:0/1351961572 reap_dead start 2024-06-29T15:38:43.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.194+0000 7f21da7eb640 1 -- 172.21.15.81:0/1351961572 shutdown_connections 2024-06-29T15:38:43.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.194+0000 7f21da7eb640 1 -- 172.21.15.81:0/1351961572 >> 172.21.15.81:0/1351961572 conn(0x7f21d40b2920 msgr2=0x7f21d40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.194+0000 7f21da7eb640 1 -- 172.21.15.81:0/1351961572 shutdown_connections 2024-06-29T15:38:43.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.194+0000 7f21da7eb640 1 -- 172.21.15.81:0/1351961572 wait complete. 2024-06-29T15:38:43.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.196+0000 7f21da7eb640 1 Processor -- start 2024-06-29T15:38:43.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.196+0000 7f21da7eb640 1 -- start start 2024-06-29T15:38:43.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.197+0000 7f21da7eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21bc004800 0x7f21d414ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.197+0000 7f21da7eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21d4149000 0x7f21d4153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.197+0000 7f21da7eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21d4154100 0x7f21d40cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.197+0000 7f21da7eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f21d415c1b0 con 0x7f21d4149000 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.197+0000 7f21da7eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f21d415c030 con 0x7f21bc004800 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.197+0000 7f21da7eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f21d4072f40 con 0x7f21d4154100 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.197+0000 7f21d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21d4149000 0x7f21d4153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.197+0000 7f21d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21d4149000 0x7f21d4153bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53772/0 (socket says 172.21.15.81:53772) 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.197+0000 7f21d37fe640 1 -- 172.21.15.81:0/3640313285 learned_addr learned my addr 172.21.15.81:0/3640313285 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d8d88640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21d4154100 0x7f21d40cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d3fff640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21bc004800 0x7f21d414ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d37fe640 1 -- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21d4154100 msgr2=0x7f21d40cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d37fe640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21d4154100 0x7f21d40cf6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d37fe640 1 -- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21bc004800 msgr2=0x7f21d414ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d37fe640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21bc004800 0x7f21d414ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d37fe640 1 -- 172.21.15.81:0/3640313285 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d4154510 con 0x7f21d4149000 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d8d88640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21d4154100 0x7f21d40cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d3fff640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21bc004800 0x7f21d414ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d37fe640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21d4149000 0x7f21d4153bc0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f21c802b000 tx=0x7f21c8005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d0ff9640 1 -- 172.21.15.81:0/3640313285 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8040070 con 0x7f21d4149000 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d0ff9640 1 -- 172.21.15.81:0/3640313285 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21c803a720 con 0x7f21d4149000 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21d0ff9640 1 -- 172.21.15.81:0/3640313285 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8035040 con 0x7f21d4149000 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21da7eb640 1 -- 172.21.15.81:0/3640313285 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d414d030 con 0x7f21d4149000 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.198+0000 7f21da7eb640 1 -- 172.21.15.81:0/3640313285 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d414d5f0 con 0x7f21d4149000 2024-06-29T15:38:43.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.199+0000 7f21da7eb640 1 -- 172.21.15.81:0/3640313285 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d40d1180 con 0x7f21d4149000 2024-06-29T15:38:43.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.200+0000 7f21d0ff9640 1 -- 172.21.15.81:0/3640313285 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f21c8038020 con 0x7f21d4149000 2024-06-29T15:38:43.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.200+0000 7f21d0ff9640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f21c0076420 0x7f21c0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.201+0000 7f21d0ff9640 1 -- 172.21.15.81:0/3640313285 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f21c80b8970 con 0x7f21d4149000 2024-06-29T15:38:43.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.201+0000 7f21d3fff640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f21c0076420 0x7f21c0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.201+0000 7f21d3fff640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f21c0076420 0x7f21c0078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21c40045e0 tx=0x7f21c400b260 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.205+0000 7f21d0ff9640 1 -- 172.21.15.81:0/3640313285 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f21d4154510 con 0x7f21d4149000 2024-06-29T15:38:43.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.382+0000 7f21da7eb640 1 -- 172.21.15.81:0/3640313285 --> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f21d415b6e0 con 0x7f21c0076420 2024-06-29T15:38:43.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.388+0000 7f21d0ff9640 1 -- 172.21.15.81:0/3640313285 <== mgr.9414 v2:172.21.15.167:6832/150801987 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252493 (secure 0 0 0) 0x7f21d415b6e0 con 0x7f21c0076420 2024-06-29T15:38:43.392 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:43.393 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:38:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 -- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f21c0076420 msgr2=0x7f21c0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f21c0076420 0x7f21c0078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21c40045e0 tx=0x7f21c400b260 comp rx=0 tx=0).stop 2024-06-29T15:38:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 -- 172.21.15.81:0/3640313285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21d4149000 msgr2=0x7f21d4153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21d4149000 0x7f21d4153bc0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f21c802b000 tx=0x7f21c8005890 comp rx=0 tx=0).stop 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 -- 172.21.15.81:0/3640313285 shutdown_connections 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f21c0076420 0x7f21c0078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f21d4154100 0x7f21d40cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21d4149000 0x7f21d4153bc0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 --2- 172.21.15.81:0/3640313285 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f21bc004800 0x7f21d414ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 -- 172.21.15.81:0/3640313285 >> 172.21.15.81:0/3640313285 conn(0x7f21d40b2920 msgr2=0x7f21d406c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 -- 172.21.15.81:0/3640313285 shutdown_connections 2024-06-29T15:38:43.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.391+0000 7f21da7eb640 1 -- 172.21.15.81:0/3640313285 wait complete. 2024-06-29T15:38:43.404 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:38:43.065296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":313,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":437736,"kb_used_data":17572,"kb_used_omap":12,"kb_used_meta":420019,"kb_avail":754536984,"statfs":{"total":773094113280,"available":772645871616,"internally_reserved":0,"allocated":17993728,"data_stored":9315988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":430100018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.722779"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.519030+0000","last_change":"2024-06-29T15:38:39.382691+0000","last_active":"2024-06-29T15:38:40.519030+0000","last_peered":"2024-06-29T15:38:40.519030+0000","last_clean":"2024-06-29T15:38:40.519030+0000","last_became_active":"2024-06-29T15:38:39.382328+0000","last_became_peered":"2024-06-29T15:38:39.382328+0000","last_unstale":"2024-06-29T15:38:40.519030+0000","last_undegraded":"2024-06-29T15:38:40.519030+0000","last_fullsized":"2024-06-29T15:38:40.519030+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:21:41.534117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692224+0000","last_change":"2024-06-29T15:38:39.368599+0000","last_active":"2024-06-29T15:38:40.692224+0000","last_peered":"2024-06-29T15:38:40.692224+0000","last_clean":"2024-06-29T15:38:40.692224+0000","last_became_active":"2024-06-29T15:38:39.368309+0000","last_became_peered":"2024-06-29T15:38:39.368309+0000","last_unstale":"2024-06-29T15:38:40.692224+0000","last_undegraded":"2024-06-29T15:38:40.692224+0000","last_fullsized":"2024-06-29T15:38:40.692224+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:21.876609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.359141+0000","last_change":"2024-06-29T15:38:39.368614+0000","last_active":"2024-06-29T15:38:40.359141+0000","last_peered":"2024-06-29T15:38:40.359141+0000","last_clean":"2024-06-29T15:38:40.359141+0000","last_became_active":"2024-06-29T15:38:39.368057+0000","last_became_peered":"2024-06-29T15:38:39.368057+0000","last_unstale":"2024-06-29T15:38:40.359141+0000","last_undegraded":"2024-06-29T15:38:40.359141+0000","last_fullsized":"2024-06-29T15:38:40.359141+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:34:52.641610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693098+0000","last_change":"2024-06-29T15:38:39.375177+0000","last_active":"2024-06-29T15:38:40.693098+0000","last_peered":"2024-06-29T15:38:40.693098+0000","last_clean":"2024-06-29T15:38:40.693098+0000","last_became_active":"2024-06-29T15:38:39.374875+0000","last_became_peered":"2024-06-29T15:38:39.374875+0000","last_unstale":"2024-06-29T15:38:40.693098+0000","last_undegraded":"2024-06-29T15:38:40.693098+0000","last_fullsized":"2024-06-29T15:38:40.693098+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:06.075398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692461+0000","last_change":"2024-06-29T15:38:39.380152+0000","last_active":"2024-06-29T15:38:40.692461+0000","last_peered":"2024-06-29T15:38:40.692461+0000","last_clean":"2024-06-29T15:38:40.692461+0000","last_became_active":"2024-06-29T15:38:39.379773+0000","last_became_peered":"2024-06-29T15:38:39.379773+0000","last_unstale":"2024-06-29T15:38:40.692461+0000","last_undegraded":"2024-06-29T15:38:40.692461+0000","last_fullsized":"2024-06-29T15:38:40.692461+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:35:51.885387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518881+0000","last_change":"2024-06-29T15:38:39.365466+0000","last_active":"2024-06-29T15:38:40.518881+0000","last_peered":"2024-06-29T15:38:40.518881+0000","last_clean":"2024-06-29T15:38:40.518881+0000","last_became_active":"2024-06-29T15:38:39.363169+0000","last_became_peered":"2024-06-29T15:38:39.363169+0000","last_unstale":"2024-06-29T15:38:40.518881+0000","last_undegraded":"2024-06-29T15:38:40.518881+0000","last_fullsized":"2024-06-29T15:38:40.518881+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:31.663745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691759+0000","last_change":"2024-06-29T15:38:39.371745+0000","last_active":"2024-06-29T15:38:40.691759+0000","last_peered":"2024-06-29T15:38:40.691759+0000","last_clean":"2024-06-29T15:38:40.691759+0000","last_became_active":"2024-06-29T15:38:39.371453+0000","last_became_peered":"2024-06-29T15:38:39.371453+0000","last_unstale":"2024-06-29T15:38:40.691759+0000","last_undegraded":"2024-06-29T15:38:40.691759+0000","last_fullsized":"2024-06-29T15:38:40.691759+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:01.154044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518622+0000","last_change":"2024-06-29T15:38:39.363138+0000","last_active":"2024-06-29T15:38:40.518622+0000","last_peered":"2024-06-29T15:38:40.518622+0000","last_clean":"2024-06-29T15:38:40.518622+0000","last_became_active":"2024-06-29T15:38:39.362832+0000","last_became_peered":"2024-06-29T15:38:39.362832+0000","last_unstale":"2024-06-29T15:38:40.518622+0000","last_undegraded":"2024-06-29T15:38:40.518622+0000","last_fullsized":"2024-06-29T15:38:40.518622+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:48.908922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691989+0000","last_change":"2024-06-29T15:38:39.363738+0000","last_active":"2024-06-29T15:38:40.691989+0000","last_peered":"2024-06-29T15:38:40.691989+0000","last_clean":"2024-06-29T15:38:40.691989+0000","last_became_active":"2024-06-29T15:38:39.363436+0000","last_became_peered":"2024-06-29T15:38:39.363436+0000","last_unstale":"2024-06-29T15:38:40.691989+0000","last_undegraded":"2024-06-29T15:38:40.691989+0000","last_fullsized":"2024-06-29T15:38:40.691989+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:54.895627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358450+0000","last_change":"2024-06-29T15:38:39.388372+0000","last_active":"2024-06-29T15:38:40.358450+0000","last_peered":"2024-06-29T15:38:40.358450+0000","last_clean":"2024-06-29T15:38:40.358450+0000","last_became_active":"2024-06-29T15:38:39.387788+0000","last_became_peered":"2024-06-29T15:38:39.387788+0000","last_unstale":"2024-06-29T15:38:40.358450+0000","last_undegraded":"2024-06-29T15:38:40.358450+0000","last_fullsized":"2024-06-29T15:38:40.358450+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:38.995705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518901+0000","last_change":"2024-06-29T15:38:39.371096+0000","last_active":"2024-06-29T15:38:40.518901+0000","last_peered":"2024-06-29T15:38:40.518901+0000","last_clean":"2024-06-29T15:38:40.518901+0000","last_became_active":"2024-06-29T15:38:39.370547+0000","last_became_peered":"2024-06-29T15:38:39.370547+0000","last_unstale":"2024-06-29T15:38:40.518901+0000","last_undegraded":"2024-06-29T15:38:40.518901+0000","last_fullsized":"2024-06-29T15:38:40.518901+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:43.076403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358311+0000","last_change":"2024-06-29T15:38:39.381688+0000","last_active":"2024-06-29T15:38:40.358311+0000","last_peered":"2024-06-29T15:38:40.358311+0000","last_clean":"2024-06-29T15:38:40.358311+0000","last_became_active":"2024-06-29T15:38:39.381364+0000","last_became_peered":"2024-06-29T15:38:39.381364+0000","last_unstale":"2024-06-29T15:38:40.358311+0000","last_undegraded":"2024-06-29T15:38:40.358311+0000","last_fullsized":"2024-06-29T15:38:40.358311+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:43:53.819101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692998+0000","last_change":"2024-06-29T15:38:39.378616+0000","last_active":"2024-06-29T15:38:40.692998+0000","last_peered":"2024-06-29T15:38:40.692998+0000","last_clean":"2024-06-29T15:38:40.692998+0000","last_became_active":"2024-06-29T15:38:39.378301+0000","last_became_peered":"2024-06-29T15:38:39.378301+0000","last_unstale":"2024-06-29T15:38:40.692998+0000","last_undegraded":"2024-06-29T15:38:40.692998+0000","last_fullsized":"2024-06-29T15:38:40.692998+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:44.674713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692553+0000","last_change":"2024-06-29T15:38:39.382903+0000","last_active":"2024-06-29T15:38:40.692553+0000","last_peered":"2024-06-29T15:38:40.692553+0000","last_clean":"2024-06-29T15:38:40.692553+0000","last_became_active":"2024-06-29T15:38:39.382535+0000","last_became_peered":"2024-06-29T15:38:39.382535+0000","last_unstale":"2024-06-29T15:38:40.692553+0000","last_undegraded":"2024-06-29T15:38:40.692553+0000","last_fullsized":"2024-06-29T15:38:40.692553+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:05:55.638711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357367+0000","last_change":"2024-06-29T15:38:39.379309+0000","last_active":"2024-06-29T15:38:40.357367+0000","last_peered":"2024-06-29T15:38:40.357367+0000","last_clean":"2024-06-29T15:38:40.357367+0000","last_became_active":"2024-06-29T15:38:39.379021+0000","last_became_peered":"2024-06-29T15:38:39.379021+0000","last_unstale":"2024-06-29T15:38:40.357367+0000","last_undegraded":"2024-06-29T15:38:40.357367+0000","last_fullsized":"2024-06-29T15:38:40.357367+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:41.525177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.356992+0000","last_change":"2024-06-29T15:38:39.363595+0000","last_active":"2024-06-29T15:38:40.356992+0000","last_peered":"2024-06-29T15:38:40.356992+0000","last_clean":"2024-06-29T15:38:40.356992+0000","last_became_active":"2024-06-29T15:38:39.363307+0000","last_became_peered":"2024-06-29T15:38:39.363307+0000","last_unstale":"2024-06-29T15:38:40.356992+0000","last_undegraded":"2024-06-29T15:38:40.356992+0000","last_fullsized":"2024-06-29T15:38:40.356992+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:00.534645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.359055+0000","last_change":"2024-06-29T15:38:39.386449+0000","last_active":"2024-06-29T15:38:40.359055+0000","last_peered":"2024-06-29T15:38:40.359055+0000","last_clean":"2024-06-29T15:38:40.359055+0000","last_became_active":"2024-06-29T15:38:39.385697+0000","last_became_peered":"2024-06-29T15:38:39.385697+0000","last_unstale":"2024-06-29T15:38:40.359055+0000","last_undegraded":"2024-06-29T15:38:40.359055+0000","last_fullsized":"2024-06-29T15:38:40.359055+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:10.064127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358769+0000","last_change":"2024-06-29T15:38:39.365403+0000","last_active":"2024-06-29T15:38:40.358769+0000","last_peered":"2024-06-29T15:38:40.358769+0000","last_clean":"2024-06-29T15:38:40.358769+0000","last_became_active":"2024-06-29T15:38:39.365029+0000","last_became_peered":"2024-06-29T15:38:39.365029+0000","last_unstale":"2024-06-29T15:38:40.358769+0000","last_undegraded":"2024-06-29T15:38:40.358769+0000","last_fullsized":"2024-06-29T15:38:40.358769+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:03.058907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.519052+0000","last_change":"2024-06-29T15:38:39.366364+0000","last_active":"2024-06-29T15:38:40.519052+0000","last_peered":"2024-06-29T15:38:40.519052+0000","last_clean":"2024-06-29T15:38:40.519052+0000","last_became_active":"2024-06-29T15:38:39.365547+0000","last_became_peered":"2024-06-29T15:38:39.365547+0000","last_unstale":"2024-06-29T15:38:40.519052+0000","last_undegraded":"2024-06-29T15:38:40.519052+0000","last_fullsized":"2024-06-29T15:38:40.519052+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:29.795879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357254+0000","last_change":"2024-06-29T15:38:39.366096+0000","last_active":"2024-06-29T15:38:40.357254+0000","last_peered":"2024-06-29T15:38:40.357254+0000","last_clean":"2024-06-29T15:38:40.357254+0000","last_became_active":"2024-06-29T15:38:39.365775+0000","last_became_peered":"2024-06-29T15:38:39.365775+0000","last_unstale":"2024-06-29T15:38:40.357254+0000","last_undegraded":"2024-06-29T15:38:40.357254+0000","last_fullsized":"2024-06-29T15:38:40.357254+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:03.058907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.359312+0000","last_change":"2024-06-29T15:38:39.388084+0000","last_active":"2024-06-29T15:38:40.359312+0000","last_peered":"2024-06-29T15:38:40.359312+0000","last_clean":"2024-06-29T15:38:40.359312+0000","last_became_active":"2024-06-29T15:38:39.387765+0000","last_became_peered":"2024-06-29T15:38:39.387765+0000","last_unstale":"2024-06-29T15:38:40.359312+0000","last_undegraded":"2024-06-29T15:38:40.359312+0000","last_fullsized":"2024-06-29T15:38:40.359312+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:25:27.774536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692765+0000","last_change":"2024-06-29T15:38:39.371056+0000","last_active":"2024-06-29T15:38:40.692765+0000","last_peered":"2024-06-29T15:38:40.692765+0000","last_clean":"2024-06-29T15:38:40.692765+0000","last_became_active":"2024-06-29T15:38:39.370715+0000","last_became_peered":"2024-06-29T15:38:39.370715+0000","last_unstale":"2024-06-29T15:38:40.692765+0000","last_undegraded":"2024-06-29T15:38:40.692765+0000","last_fullsized":"2024-06-29T15:38:40.692765+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:35:45.940249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693279+0000","last_change":"2024-06-29T15:38:39.368931+0000","last_active":"2024-06-29T15:38:40.693279+0000","last_peered":"2024-06-29T15:38:40.693279+0000","last_clean":"2024-06-29T15:38:40.693279+0000","last_became_active":"2024-06-29T15:38:39.368545+0000","last_became_peered":"2024-06-29T15:38:39.368545+0000","last_unstale":"2024-06-29T15:38:40.693279+0000","last_undegraded":"2024-06-29T15:38:40.693279+0000","last_fullsized":"2024-06-29T15:38:40.693279+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:35:31.375256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357785+0000","last_change":"2024-06-29T15:38:39.388352+0000","last_active":"2024-06-29T15:38:40.357785+0000","last_peered":"2024-06-29T15:38:40.357785+0000","last_clean":"2024-06-29T15:38:40.357785+0000","last_became_active":"2024-06-29T15:38:39.388036+0000","last_became_peered":"2024-06-29T15:38:39.388036+0000","last_unstale":"2024-06-29T15:38:40.357785+0000","last_undegraded":"2024-06-29T15:38:40.357785+0000","last_fullsized":"2024-06-29T15:38:40.357785+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:50:25.514462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357170+0000","last_change":"2024-06-29T15:38:39.363840+0000","last_active":"2024-06-29T15:38:40.357170+0000","last_peered":"2024-06-29T15:38:40.357170+0000","last_clean":"2024-06-29T15:38:40.357170+0000","last_became_active":"2024-06-29T15:38:39.363327+0000","last_became_peered":"2024-06-29T15:38:39.363327+0000","last_unstale":"2024-06-29T15:38:40.357170+0000","last_undegraded":"2024-06-29T15:38:40.357170+0000","last_fullsized":"2024-06-29T15:38:40.357170+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:45:04.155625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518843+0000","last_change":"2024-06-29T15:38:39.364207+0000","last_active":"2024-06-29T15:38:40.518843+0000","last_peered":"2024-06-29T15:38:40.518843+0000","last_clean":"2024-06-29T15:38:40.518843+0000","last_became_active":"2024-06-29T15:38:39.363896+0000","last_became_peered":"2024-06-29T15:38:39.363896+0000","last_unstale":"2024-06-29T15:38:40.518843+0000","last_undegraded":"2024-06-29T15:38:40.518843+0000","last_fullsized":"2024-06-29T15:38:40.518843+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:09.992782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357378+0000","last_change":"2024-06-29T15:38:39.371562+0000","last_active":"2024-06-29T15:38:40.357378+0000","last_peered":"2024-06-29T15:38:40.357378+0000","last_clean":"2024-06-29T15:38:40.357378+0000","last_became_active":"2024-06-29T15:38:39.371266+0000","last_became_peered":"2024-06-29T15:38:39.371266+0000","last_unstale":"2024-06-29T15:38:40.357378+0000","last_undegraded":"2024-06-29T15:38:40.357378+0000","last_fullsized":"2024-06-29T15:38:40.357378+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:34:52.641610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691701+0000","last_change":"2024-06-29T15:38:39.374845+0000","last_active":"2024-06-29T15:38:40.691701+0000","last_peered":"2024-06-29T15:38:40.691701+0000","last_clean":"2024-06-29T15:38:40.691701+0000","last_became_active":"2024-06-29T15:38:39.374537+0000","last_became_peered":"2024-06-29T15:38:39.374537+0000","last_unstale":"2024-06-29T15:38:40.691701+0000","last_undegraded":"2024-06-29T15:38:40.691701+0000","last_fullsized":"2024-06-29T15:38:40.691701+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:48.908922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518661+0000","last_change":"2024-06-29T15:38:39.365275+0000","last_active":"2024-06-29T15:38:40.518661+0000","last_peered":"2024-06-29T15:38:40.518661+0000","last_clean":"2024-06-29T15:38:40.518661+0000","last_became_active":"2024-06-29T15:38:39.364775+0000","last_became_peered":"2024-06-29T15:38:39.364775+0000","last_unstale":"2024-06-29T15:38:40.518661+0000","last_undegraded":"2024-06-29T15:38:40.518661+0000","last_fullsized":"2024-06-29T15:38:40.518661+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:56.070386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692809+0000","last_change":"2024-06-29T15:38:39.374266+0000","last_active":"2024-06-29T15:38:40.692809+0000","last_peered":"2024-06-29T15:38:40.692809+0000","last_clean":"2024-06-29T15:38:40.692809+0000","last_became_active":"2024-06-29T15:38:39.373923+0000","last_became_peered":"2024-06-29T15:38:39.373923+0000","last_unstale":"2024-06-29T15:38:40.692809+0000","last_undegraded":"2024-06-29T15:38:40.692809+0000","last_fullsized":"2024-06-29T15:38:40.692809+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:19:08.383472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692146+0000","last_change":"2024-06-29T15:38:39.374954+0000","last_active":"2024-06-29T15:38:40.692146+0000","last_peered":"2024-06-29T15:38:40.692146+0000","last_clean":"2024-06-29T15:38:40.692146+0000","last_became_active":"2024-06-29T15:38:39.374637+0000","last_became_peered":"2024-06-29T15:38:39.374637+0000","last_unstale":"2024-06-29T15:38:40.692146+0000","last_undegraded":"2024-06-29T15:38:40.692146+0000","last_fullsized":"2024-06-29T15:38:40.692146+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:26.406721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357404+0000","last_change":"2024-06-29T15:38:39.363765+0000","last_active":"2024-06-29T15:38:40.357404+0000","last_peered":"2024-06-29T15:38:40.357404+0000","last_clean":"2024-06-29T15:38:40.357404+0000","last_became_active":"2024-06-29T15:38:39.363488+0000","last_became_peered":"2024-06-29T15:38:39.363488+0000","last_unstale":"2024-06-29T15:38:40.357404+0000","last_undegraded":"2024-06-29T15:38:40.357404+0000","last_fullsized":"2024-06-29T15:38:40.357404+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:48.908922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358480+0000","last_change":"2024-06-29T15:38:39.384650+0000","last_active":"2024-06-29T15:38:40.358480+0000","last_peered":"2024-06-29T15:38:40.358480+0000","last_clean":"2024-06-29T15:38:40.358480+0000","last_became_active":"2024-06-29T15:38:39.384319+0000","last_became_peered":"2024-06-29T15:38:39.384319+0000","last_unstale":"2024-06-29T15:38:40.358480+0000","last_undegraded":"2024-06-29T15:38:40.358480+0000","last_fullsized":"2024-06-29T15:38:40.358480+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:32.414242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"167'4","reported_seq":18,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358863+0000","last_change":"2024-06-29T15:38:38.375263+0000","last_active":"2024-06-29T15:38:40.358863+0000","last_peered":"2024-06-29T15:38:40.358863+0000","last_clean":"2024-06-29T15:38:40.358863+0000","last_became_active":"2024-06-29T15:38:38.374826+0000","last_became_peered":"2024-06-29T15:38:38.374826+0000","last_unstale":"2024-06-29T15:38:40.358863+0000","last_undegraded":"2024-06-29T15:38:40.358863+0000","last_fullsized":"2024-06-29T15:38:40.358863+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:56.527020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692022+0000","last_change":"2024-06-29T15:38:39.367987+0000","last_active":"2024-06-29T15:38:40.692022+0000","last_peered":"2024-06-29T15:38:40.692022+0000","last_clean":"2024-06-29T15:38:40.692022+0000","last_became_active":"2024-06-29T15:38:39.367679+0000","last_became_peered":"2024-06-29T15:38:39.367679+0000","last_unstale":"2024-06-29T15:38:40.692022+0000","last_undegraded":"2024-06-29T15:38:40.692022+0000","last_fullsized":"2024-06-29T15:38:40.692022+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:55.685876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"167'1","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692251+0000","last_change":"2024-06-29T15:38:38.372700+0000","last_active":"2024-06-29T15:38:40.692251+0000","last_peered":"2024-06-29T15:38:40.692251+0000","last_clean":"2024-06-29T15:38:40.692251+0000","last_became_active":"2024-06-29T15:38:38.372184+0000","last_became_peered":"2024-06-29T15:38:38.372184+0000","last_unstale":"2024-06-29T15:38:40.692251+0000","last_undegraded":"2024-06-29T15:38:40.692251+0000","last_fullsized":"2024-06-29T15:38:40.692251+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:15.462869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518701+0000","last_change":"2024-06-29T15:38:39.369624+0000","last_active":"2024-06-29T15:38:40.518701+0000","last_peered":"2024-06-29T15:38:40.518701+0000","last_clean":"2024-06-29T15:38:40.518701+0000","last_became_active":"2024-06-29T15:38:39.369309+0000","last_became_peered":"2024-06-29T15:38:39.369309+0000","last_unstale":"2024-06-29T15:38:40.518701+0000","last_undegraded":"2024-06-29T15:38:40.518701+0000","last_fullsized":"2024-06-29T15:38:40.518701+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:01.154044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693110+0000","last_change":"2024-06-29T15:38:38.370062+0000","last_active":"2024-06-29T15:38:40.693110+0000","last_peered":"2024-06-29T15:38:40.693110+0000","last_clean":"2024-06-29T15:38:40.693110+0000","last_became_active":"2024-06-29T15:38:38.368922+0000","last_became_peered":"2024-06-29T15:38:38.368922+0000","last_unstale":"2024-06-29T15:38:40.693110+0000","last_undegraded":"2024-06-29T15:38:40.693110+0000","last_fullsized":"2024-06-29T15:38:40.693110+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:06.829962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358773+0000","last_change":"2024-06-29T15:38:39.378397+0000","last_active":"2024-06-29T15:38:40.358773+0000","last_peered":"2024-06-29T15:38:40.358773+0000","last_clean":"2024-06-29T15:38:40.358773+0000","last_became_active":"2024-06-29T15:38:39.378104+0000","last_became_peered":"2024-06-29T15:38:39.378104+0000","last_unstale":"2024-06-29T15:38:40.358773+0000","last_undegraded":"2024-06-29T15:38:40.358773+0000","last_fullsized":"2024-06-29T15:38:40.358773+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:41.573791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692967+0000","last_change":"2024-06-29T15:38:38.369367+0000","last_active":"2024-06-29T15:38:40.692967+0000","last_peered":"2024-06-29T15:38:40.692967+0000","last_clean":"2024-06-29T15:38:40.692967+0000","last_became_active":"2024-06-29T15:38:38.369022+0000","last_became_peered":"2024-06-29T15:38:38.369022+0000","last_unstale":"2024-06-29T15:38:40.692967+0000","last_undegraded":"2024-06-29T15:38:40.692967+0000","last_fullsized":"2024-06-29T15:38:40.692967+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:21:40.525440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692428+0000","last_change":"2024-06-29T15:38:39.366918+0000","last_active":"2024-06-29T15:38:40.692428+0000","last_peered":"2024-06-29T15:38:40.692428+0000","last_clean":"2024-06-29T15:38:40.692428+0000","last_became_active":"2024-06-29T15:38:39.366617+0000","last_became_peered":"2024-06-29T15:38:39.366617+0000","last_unstale":"2024-06-29T15:38:40.692428+0000","last_undegraded":"2024-06-29T15:38:40.692428+0000","last_fullsized":"2024-06-29T15:38:40.692428+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:33.770741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"167'1","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357422+0000","last_change":"2024-06-29T15:38:38.364862+0000","last_active":"2024-06-29T15:38:40.357422+0000","last_peered":"2024-06-29T15:38:40.357422+0000","last_clean":"2024-06-29T15:38:40.357422+0000","last_became_active":"2024-06-29T15:38:38.364529+0000","last_became_peered":"2024-06-29T15:38:38.364529+0000","last_unstale":"2024-06-29T15:38:40.357422+0000","last_undegraded":"2024-06-29T15:38:40.357422+0000","last_fullsized":"2024-06-29T15:38:40.357422+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:21.086645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693151+0000","last_change":"2024-06-29T15:38:39.373959+0000","last_active":"2024-06-29T15:38:40.693151+0000","last_peered":"2024-06-29T15:38:40.693151+0000","last_clean":"2024-06-29T15:38:40.693151+0000","last_became_active":"2024-06-29T15:38:39.373646+0000","last_became_peered":"2024-06-29T15:38:39.373646+0000","last_unstale":"2024-06-29T15:38:40.693151+0000","last_undegraded":"2024-06-29T15:38:40.693151+0000","last_fullsized":"2024-06-29T15:38:40.693151+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:21:11.326784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"167'1","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692557+0000","last_change":"2024-06-29T15:38:38.363928+0000","last_active":"2024-06-29T15:38:40.692557+0000","last_peered":"2024-06-29T15:38:40.692557+0000","last_clean":"2024-06-29T15:38:40.692557+0000","last_became_active":"2024-06-29T15:38:38.363553+0000","last_became_peered":"2024-06-29T15:38:38.363553+0000","last_unstale":"2024-06-29T15:38:40.692557+0000","last_undegraded":"2024-06-29T15:38:40.692557+0000","last_fullsized":"2024-06-29T15:38:40.692557+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:46.451406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692237+0000","last_change":"2024-06-29T15:38:39.371897+0000","last_active":"2024-06-29T15:38:40.692237+0000","last_peered":"2024-06-29T15:38:40.692237+0000","last_clean":"2024-06-29T15:38:40.692237+0000","last_became_active":"2024-06-29T15:38:39.371582+0000","last_became_peered":"2024-06-29T15:38:39.371582+0000","last_unstale":"2024-06-29T15:38:40.692237+0000","last_undegraded":"2024-06-29T15:38:40.692237+0000","last_fullsized":"2024-06-29T15:38:40.692237+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:47:52.311087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357145+0000","last_change":"2024-06-29T15:38:38.367400+0000","last_active":"2024-06-29T15:38:40.357145+0000","last_peered":"2024-06-29T15:38:40.357145+0000","last_clean":"2024-06-29T15:38:40.357145+0000","last_became_active":"2024-06-29T15:38:38.367054+0000","last_became_peered":"2024-06-29T15:38:38.367054+0000","last_unstale":"2024-06-29T15:38:40.357145+0000","last_undegraded":"2024-06-29T15:38:40.357145+0000","last_fullsized":"2024-06-29T15:38:40.357145+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:35:38.344362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692388+0000","last_change":"2024-06-29T15:38:39.367679+0000","last_active":"2024-06-29T15:38:40.692388+0000","last_peered":"2024-06-29T15:38:40.692388+0000","last_clean":"2024-06-29T15:38:40.692388+0000","last_became_active":"2024-06-29T15:38:39.366673+0000","last_became_peered":"2024-06-29T15:38:39.366673+0000","last_unstale":"2024-06-29T15:38:40.692388+0000","last_undegraded":"2024-06-29T15:38:40.692388+0000","last_fullsized":"2024-06-29T15:38:40.692388+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:24.789828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692250+0000","last_change":"2024-06-29T15:38:38.373514+0000","last_active":"2024-06-29T15:38:40.692250+0000","last_peered":"2024-06-29T15:38:40.692250+0000","last_clean":"2024-06-29T15:38:40.692250+0000","last_became_active":"2024-06-29T15:38:38.373176+0000","last_became_peered":"2024-06-29T15:38:38.373176+0000","last_unstale":"2024-06-29T15:38:40.692250+0000","last_undegraded":"2024-06-29T15:38:40.692250+0000","last_fullsized":"2024-06-29T15:38:40.692250+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:42.067726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692986+0000","last_change":"2024-06-29T15:38:39.366698+0000","last_active":"2024-06-29T15:38:40.692986+0000","last_peered":"2024-06-29T15:38:40.692986+0000","last_clean":"2024-06-29T15:38:40.692986+0000","last_became_active":"2024-06-29T15:38:39.366358+0000","last_became_peered":"2024-06-29T15:38:39.366358+0000","last_unstale":"2024-06-29T15:38:40.692986+0000","last_undegraded":"2024-06-29T15:38:40.692986+0000","last_fullsized":"2024-06-29T15:38:40.692986+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:00.534645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692121+0000","last_change":"2024-06-29T15:38:38.366411+0000","last_active":"2024-06-29T15:38:40.692121+0000","last_peered":"2024-06-29T15:38:40.692121+0000","last_clean":"2024-06-29T15:38:40.692121+0000","last_became_active":"2024-06-29T15:38:38.366076+0000","last_became_peered":"2024-06-29T15:38:38.366076+0000","last_unstale":"2024-06-29T15:38:40.692121+0000","last_undegraded":"2024-06-29T15:38:40.692121+0000","last_fullsized":"2024-06-29T15:38:40.692121+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:20.817965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358724+0000","last_change":"2024-06-29T15:38:39.386156+0000","last_active":"2024-06-29T15:38:40.358724+0000","last_peered":"2024-06-29T15:38:40.358724+0000","last_clean":"2024-06-29T15:38:40.358724+0000","last_became_active":"2024-06-29T15:38:39.385596+0000","last_became_peered":"2024-06-29T15:38:39.385596+0000","last_unstale":"2024-06-29T15:38:40.358724+0000","last_undegraded":"2024-06-29T15:38:40.358724+0000","last_fullsized":"2024-06-29T15:38:40.358724+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:44:11.976563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358612+0000","last_change":"2024-06-29T15:38:38.375628+0000","last_active":"2024-06-29T15:38:40.358612+0000","last_peered":"2024-06-29T15:38:40.358612+0000","last_clean":"2024-06-29T15:38:40.358612+0000","last_became_active":"2024-06-29T15:38:38.375252+0000","last_became_peered":"2024-06-29T15:38:38.375252+0000","last_unstale":"2024-06-29T15:38:40.358612+0000","last_undegraded":"2024-06-29T15:38:40.358612+0000","last_fullsized":"2024-06-29T15:38:40.358612+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:44.846550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357247+0000","last_change":"2024-06-29T15:38:39.366315+0000","last_active":"2024-06-29T15:38:40.357247+0000","last_peered":"2024-06-29T15:38:40.357247+0000","last_clean":"2024-06-29T15:38:40.357247+0000","last_became_active":"2024-06-29T15:38:39.365982+0000","last_became_peered":"2024-06-29T15:38:39.365982+0000","last_unstale":"2024-06-29T15:38:40.357247+0000","last_undegraded":"2024-06-29T15:38:40.357247+0000","last_fullsized":"2024-06-29T15:38:40.357247+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:11.499263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"167'6","reported_seq":20,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691857+0000","last_change":"2024-06-29T15:38:38.373640+0000","last_active":"2024-06-29T15:38:40.691857+0000","last_peered":"2024-06-29T15:38:40.691857+0000","last_clean":"2024-06-29T15:38:40.691857+0000","last_became_active":"2024-06-29T15:38:38.373288+0000","last_became_peered":"2024-06-29T15:38:38.373288+0000","last_unstale":"2024-06-29T15:38:40.691857+0000","last_undegraded":"2024-06-29T15:38:40.691857+0000","last_fullsized":"2024-06-29T15:38:40.691857+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:40.276368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357635+0000","last_change":"2024-06-29T15:38:39.373107+0000","last_active":"2024-06-29T15:38:40.357635+0000","last_peered":"2024-06-29T15:38:40.357635+0000","last_clean":"2024-06-29T15:38:40.357635+0000","last_became_active":"2024-06-29T15:38:39.372780+0000","last_became_peered":"2024-06-29T15:38:39.372780+0000","last_unstale":"2024-06-29T15:38:40.357635+0000","last_undegraded":"2024-06-29T15:38:40.357635+0000","last_fullsized":"2024-06-29T15:38:40.357635+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:44.674713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358255+0000","last_change":"2024-06-29T15:38:38.370647+0000","last_active":"2024-06-29T15:38:40.358255+0000","last_peered":"2024-06-29T15:38:40.358255+0000","last_clean":"2024-06-29T15:38:40.358255+0000","last_became_active":"2024-06-29T15:38:38.370337+0000","last_became_peered":"2024-06-29T15:38:38.370337+0000","last_unstale":"2024-06-29T15:38:40.358255+0000","last_undegraded":"2024-06-29T15:38:40.358255+0000","last_fullsized":"2024-06-29T15:38:40.358255+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:43:11.097338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691923+0000","last_change":"2024-06-29T15:38:39.368162+0000","last_active":"2024-06-29T15:38:40.691923+0000","last_peered":"2024-06-29T15:38:40.691923+0000","last_clean":"2024-06-29T15:38:40.691923+0000","last_became_active":"2024-06-29T15:38:39.367819+0000","last_became_peered":"2024-06-29T15:38:39.367819+0000","last_unstale":"2024-06-29T15:38:40.691923+0000","last_undegraded":"2024-06-29T15:38:40.691923+0000","last_fullsized":"2024-06-29T15:38:40.691923+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:52.382697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357588+0000","last_change":"2024-06-29T15:38:38.369584+0000","last_active":"2024-06-29T15:38:40.357588+0000","last_peered":"2024-06-29T15:38:40.357588+0000","last_clean":"2024-06-29T15:38:40.357588+0000","last_became_active":"2024-06-29T15:38:38.369067+0000","last_became_peered":"2024-06-29T15:38:38.369067+0000","last_unstale":"2024-06-29T15:38:40.357588+0000","last_undegraded":"2024-06-29T15:38:40.357588+0000","last_fullsized":"2024-06-29T15:38:40.357588+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:34.851686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692760+0000","last_change":"2024-06-29T15:38:39.382892+0000","last_active":"2024-06-29T15:38:40.692760+0000","last_peered":"2024-06-29T15:38:40.692760+0000","last_clean":"2024-06-29T15:38:40.692760+0000","last_became_active":"2024-06-29T15:38:39.382576+0000","last_became_peered":"2024-06-29T15:38:39.382576+0000","last_unstale":"2024-06-29T15:38:40.692760+0000","last_undegraded":"2024-06-29T15:38:40.692760+0000","last_fullsized":"2024-06-29T15:38:40.692760+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:41.285045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518549+0000","last_change":"2024-06-29T15:38:38.369546+0000","last_active":"2024-06-29T15:38:40.518549+0000","last_peered":"2024-06-29T15:38:40.518549+0000","last_clean":"2024-06-29T15:38:40.518549+0000","last_became_active":"2024-06-29T15:38:38.369167+0000","last_became_peered":"2024-06-29T15:38:38.369167+0000","last_unstale":"2024-06-29T15:38:40.518549+0000","last_undegraded":"2024-06-29T15:38:40.518549+0000","last_fullsized":"2024-06-29T15:38:40.518549+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:14.166568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357791+0000","last_change":"2024-06-29T15:38:39.367649+0000","last_active":"2024-06-29T15:38:40.357791+0000","last_peered":"2024-06-29T15:38:40.357791+0000","last_clean":"2024-06-29T15:38:40.357791+0000","last_became_active":"2024-06-29T15:38:39.367354+0000","last_became_peered":"2024-06-29T15:38:39.367354+0000","last_unstale":"2024-06-29T15:38:40.357791+0000","last_undegraded":"2024-06-29T15:38:40.357791+0000","last_fullsized":"2024-06-29T15:38:40.357791+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:45:04.155625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"167'8","reported_seq":22,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357446+0000","last_change":"2024-06-29T15:38:38.372405+0000","last_active":"2024-06-29T15:38:40.357446+0000","last_peered":"2024-06-29T15:38:40.357446+0000","last_clean":"2024-06-29T15:38:40.357446+0000","last_became_active":"2024-06-29T15:38:38.372012+0000","last_became_peered":"2024-06-29T15:38:38.372012+0000","last_unstale":"2024-06-29T15:38:40.357446+0000","last_undegraded":"2024-06-29T15:38:40.357446+0000","last_fullsized":"2024-06-29T15:38:40.357446+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:03.888658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358789+0000","last_change":"2024-06-29T15:38:39.386031+0000","last_active":"2024-06-29T15:38:40.358789+0000","last_peered":"2024-06-29T15:38:40.358789+0000","last_clean":"2024-06-29T15:38:40.358789+0000","last_became_active":"2024-06-29T15:38:39.385661+0000","last_became_peered":"2024-06-29T15:38:39.385661+0000","last_unstale":"2024-06-29T15:38:40.358789+0000","last_undegraded":"2024-06-29T15:38:40.358789+0000","last_fullsized":"2024-06-29T15:38:40.358789+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:46:46.673043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"167'1","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692649+0000","last_change":"2024-06-29T15:38:38.371901+0000","last_active":"2024-06-29T15:38:40.692649+0000","last_peered":"2024-06-29T15:38:40.692649+0000","last_clean":"2024-06-29T15:38:40.692649+0000","last_became_active":"2024-06-29T15:38:38.371581+0000","last_became_peered":"2024-06-29T15:38:38.371581+0000","last_unstale":"2024-06-29T15:38:40.692649+0000","last_undegraded":"2024-06-29T15:38:40.692649+0000","last_fullsized":"2024-06-29T15:38:40.692649+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:09.205714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357352+0000","last_change":"2024-06-29T15:38:39.381194+0000","last_active":"2024-06-29T15:38:40.357352+0000","last_peered":"2024-06-29T15:38:40.357352+0000","last_clean":"2024-06-29T15:38:40.357352+0000","last_became_active":"2024-06-29T15:38:39.380871+0000","last_became_peered":"2024-06-29T15:38:39.380871+0000","last_unstale":"2024-06-29T15:38:40.357352+0000","last_undegraded":"2024-06-29T15:38:40.357352+0000","last_fullsized":"2024-06-29T15:38:40.357352+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:17.144132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692309+0000","last_change":"2024-06-29T15:38:38.372454+0000","last_active":"2024-06-29T15:38:40.692309+0000","last_peered":"2024-06-29T15:38:40.692309+0000","last_clean":"2024-06-29T15:38:40.692309+0000","last_became_active":"2024-06-29T15:38:38.371789+0000","last_became_peered":"2024-06-29T15:38:38.371789+0000","last_unstale":"2024-06-29T15:38:40.692309+0000","last_undegraded":"2024-06-29T15:38:40.692309+0000","last_fullsized":"2024-06-29T15:38:40.692309+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:49.171782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358472+0000","last_change":"2024-06-29T15:38:39.372125+0000","last_active":"2024-06-29T15:38:40.358472+0000","last_peered":"2024-06-29T15:38:40.358472+0000","last_clean":"2024-06-29T15:38:40.358472+0000","last_became_active":"2024-06-29T15:38:39.371831+0000","last_became_peered":"2024-06-29T15:38:39.371831+0000","last_unstale":"2024-06-29T15:38:40.358472+0000","last_undegraded":"2024-06-29T15:38:40.358472+0000","last_fullsized":"2024-06-29T15:38:40.358472+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:11.499263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"167'1","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357367+0000","last_change":"2024-06-29T15:38:38.374464+0000","last_active":"2024-06-29T15:38:40.357367+0000","last_peered":"2024-06-29T15:38:40.357367+0000","last_clean":"2024-06-29T15:38:40.357367+0000","last_became_active":"2024-06-29T15:38:38.374116+0000","last_became_peered":"2024-06-29T15:38:38.374116+0000","last_unstale":"2024-06-29T15:38:40.357367+0000","last_undegraded":"2024-06-29T15:38:40.357367+0000","last_fullsized":"2024-06-29T15:38:40.357367+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:43:11.097338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518791+0000","last_change":"2024-06-29T15:38:39.365185+0000","last_active":"2024-06-29T15:38:40.518791+0000","last_peered":"2024-06-29T15:38:40.518791+0000","last_clean":"2024-06-29T15:38:40.518791+0000","last_became_active":"2024-06-29T15:38:39.364832+0000","last_became_peered":"2024-06-29T15:38:39.364832+0000","last_unstale":"2024-06-29T15:38:40.518791+0000","last_undegraded":"2024-06-29T15:38:40.518791+0000","last_fullsized":"2024-06-29T15:38:40.518791+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:30.305412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692306+0000","last_change":"2024-06-29T15:38:38.367800+0000","last_active":"2024-06-29T15:38:40.692306+0000","last_peered":"2024-06-29T15:38:40.692306+0000","last_clean":"2024-06-29T15:38:40.692306+0000","last_became_active":"2024-06-29T15:38:38.367489+0000","last_became_peered":"2024-06-29T15:38:38.367489+0000","last_unstale":"2024-06-29T15:38:40.692306+0000","last_undegraded":"2024-06-29T15:38:40.692306+0000","last_fullsized":"2024-06-29T15:38:40.692306+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:25:51.658576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693007+0000","last_change":"2024-06-29T15:38:39.372396+0000","last_active":"2024-06-29T15:38:40.693007+0000","last_peered":"2024-06-29T15:38:40.693007+0000","last_clean":"2024-06-29T15:38:40.693007+0000","last_became_active":"2024-06-29T15:38:39.372083+0000","last_became_peered":"2024-06-29T15:38:39.372083+0000","last_unstale":"2024-06-29T15:38:40.693007+0000","last_undegraded":"2024-06-29T15:38:40.693007+0000","last_fullsized":"2024-06-29T15:38:40.693007+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:03:09.182345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691802+0000","last_change":"2024-06-29T15:38:38.371666+0000","last_active":"2024-06-29T15:38:40.691802+0000","last_peered":"2024-06-29T15:38:40.691802+0000","last_clean":"2024-06-29T15:38:40.691802+0000","last_became_active":"2024-06-29T15:38:38.371347+0000","last_became_peered":"2024-06-29T15:38:38.371347+0000","last_unstale":"2024-06-29T15:38:40.691802+0000","last_undegraded":"2024-06-29T15:38:40.691802+0000","last_fullsized":"2024-06-29T15:38:40.691802+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:57.468225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691962+0000","last_change":"2024-06-29T15:38:39.373311+0000","last_active":"2024-06-29T15:38:40.691962+0000","last_peered":"2024-06-29T15:38:40.691962+0000","last_clean":"2024-06-29T15:38:40.691962+0000","last_became_active":"2024-06-29T15:38:39.372719+0000","last_became_peered":"2024-06-29T15:38:39.372719+0000","last_unstale":"2024-06-29T15:38:40.691962+0000","last_undegraded":"2024-06-29T15:38:40.691962+0000","last_fullsized":"2024-06-29T15:38:40.691962+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:30.305412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358075+0000","last_change":"2024-06-29T15:38:38.372354+0000","last_active":"2024-06-29T15:38:40.358075+0000","last_peered":"2024-06-29T15:38:40.358075+0000","last_clean":"2024-06-29T15:38:40.358075+0000","last_became_active":"2024-06-29T15:38:38.372054+0000","last_became_peered":"2024-06-29T15:38:38.372054+0000","last_unstale":"2024-06-29T15:38:40.358075+0000","last_undegraded":"2024-06-29T15:38:40.358075+0000","last_fullsized":"2024-06-29T15:38:40.358075+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:02.199608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692221+0000","last_change":"2024-06-29T15:38:39.368528+0000","last_active":"2024-06-29T15:38:40.692221+0000","last_peered":"2024-06-29T15:38:40.692221+0000","last_clean":"2024-06-29T15:38:40.692221+0000","last_became_active":"2024-06-29T15:38:39.368046+0000","last_became_peered":"2024-06-29T15:38:39.368046+0000","last_unstale":"2024-06-29T15:38:40.692221+0000","last_undegraded":"2024-06-29T15:38:40.692221+0000","last_fullsized":"2024-06-29T15:38:40.692221+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:21.935398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357863+0000","last_change":"2024-06-29T15:38:38.376220+0000","last_active":"2024-06-29T15:38:40.357863+0000","last_peered":"2024-06-29T15:38:40.357863+0000","last_clean":"2024-06-29T15:38:40.357863+0000","last_became_active":"2024-06-29T15:38:38.375771+0000","last_became_peered":"2024-06-29T15:38:38.375771+0000","last_unstale":"2024-06-29T15:38:40.357863+0000","last_undegraded":"2024-06-29T15:38:40.357863+0000","last_fullsized":"2024-06-29T15:38:40.357863+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:39.182877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691663+0000","last_change":"2024-06-29T15:38:39.373875+0000","last_active":"2024-06-29T15:38:40.691663+0000","last_peered":"2024-06-29T15:38:40.691663+0000","last_clean":"2024-06-29T15:38:40.691663+0000","last_became_active":"2024-06-29T15:38:39.372892+0000","last_became_peered":"2024-06-29T15:38:39.372892+0000","last_unstale":"2024-06-29T15:38:40.691663+0000","last_undegraded":"2024-06-29T15:38:40.691663+0000","last_fullsized":"2024-06-29T15:38:40.691663+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:34:52.641610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357303+0000","last_change":"2024-06-29T15:38:38.370774+0000","last_active":"2024-06-29T15:38:40.357303+0000","last_peered":"2024-06-29T15:38:40.357303+0000","last_clean":"2024-06-29T15:38:40.357303+0000","last_became_active":"2024-06-29T15:38:38.370393+0000","last_became_peered":"2024-06-29T15:38:38.370393+0000","last_unstale":"2024-06-29T15:38:40.357303+0000","last_undegraded":"2024-06-29T15:38:40.357303+0000","last_fullsized":"2024-06-29T15:38:40.357303+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:14.166568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693173+0000","last_change":"2024-06-29T15:38:39.384002+0000","last_active":"2024-06-29T15:38:40.693173+0000","last_peered":"2024-06-29T15:38:40.693173+0000","last_clean":"2024-06-29T15:38:40.693173+0000","last_became_active":"2024-06-29T15:38:39.383618+0000","last_became_peered":"2024-06-29T15:38:39.383618+0000","last_unstale":"2024-06-29T15:38:40.693173+0000","last_undegraded":"2024-06-29T15:38:40.693173+0000","last_fullsized":"2024-06-29T15:38:40.693173+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:45:04.155625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.359173+0000","last_change":"2024-06-29T15:38:38.377903+0000","last_active":"2024-06-29T15:38:40.359173+0000","last_peered":"2024-06-29T15:38:40.359173+0000","last_clean":"2024-06-29T15:38:40.359173+0000","last_became_active":"2024-06-29T15:38:38.377547+0000","last_became_peered":"2024-06-29T15:38:38.377547+0000","last_unstale":"2024-06-29T15:38:40.359173+0000","last_undegraded":"2024-06-29T15:38:40.359173+0000","last_fullsized":"2024-06-29T15:38:40.359173+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:54:10.496353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358884+0000","last_change":"2024-06-29T15:38:39.370921+0000","last_active":"2024-06-29T15:38:40.358884+0000","last_peered":"2024-06-29T15:38:40.358884+0000","last_clean":"2024-06-29T15:38:40.358884+0000","last_became_active":"2024-06-29T15:38:39.370534+0000","last_became_peered":"2024-06-29T15:38:39.370534+0000","last_unstale":"2024-06-29T15:38:40.358884+0000","last_undegraded":"2024-06-29T15:38:40.358884+0000","last_fullsized":"2024-06-29T15:38:40.358884+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:17.144132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518873+0000","last_change":"2024-06-29T15:38:38.372716+0000","last_active":"2024-06-29T15:38:40.518873+0000","last_peered":"2024-06-29T15:38:40.518873+0000","last_clean":"2024-06-29T15:38:40.518873+0000","last_became_active":"2024-06-29T15:38:38.372409+0000","last_became_peered":"2024-06-29T15:38:38.372409+0000","last_unstale":"2024-06-29T15:38:40.518873+0000","last_undegraded":"2024-06-29T15:38:40.518873+0000","last_fullsized":"2024-06-29T15:38:40.518873+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:34.851686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692201+0000","last_change":"2024-06-29T15:38:39.369633+0000","last_active":"2024-06-29T15:38:40.692201+0000","last_peered":"2024-06-29T15:38:40.692201+0000","last_clean":"2024-06-29T15:38:40.692201+0000","last_became_active":"2024-06-29T15:38:39.369333+0000","last_became_peered":"2024-06-29T15:38:39.369333+0000","last_unstale":"2024-06-29T15:38:40.692201+0000","last_undegraded":"2024-06-29T15:38:40.692201+0000","last_fullsized":"2024-06-29T15:38:40.692201+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:06:27.721406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.519011+0000","last_change":"2024-06-29T15:38:38.366146+0000","last_active":"2024-06-29T15:38:40.519011+0000","last_peered":"2024-06-29T15:38:40.519011+0000","last_clean":"2024-06-29T15:38:40.519011+0000","last_became_active":"2024-06-29T15:38:38.365478+0000","last_became_peered":"2024-06-29T15:38:38.365478+0000","last_unstale":"2024-06-29T15:38:40.519011+0000","last_undegraded":"2024-06-29T15:38:40.519011+0000","last_fullsized":"2024-06-29T15:38:40.519011+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:40.172894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518952+0000","last_change":"2024-06-29T15:38:39.371069+0000","last_active":"2024-06-29T15:38:40.518952+0000","last_peered":"2024-06-29T15:38:40.518952+0000","last_clean":"2024-06-29T15:38:40.518952+0000","last_became_active":"2024-06-29T15:38:39.370167+0000","last_became_peered":"2024-06-29T15:38:39.370167+0000","last_unstale":"2024-06-29T15:38:40.518952+0000","last_undegraded":"2024-06-29T15:38:40.518952+0000","last_fullsized":"2024-06-29T15:38:40.518952+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:07.838639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518728+0000","last_change":"2024-06-29T15:38:38.365433+0000","last_active":"2024-06-29T15:38:40.518728+0000","last_peered":"2024-06-29T15:38:40.518728+0000","last_clean":"2024-06-29T15:38:40.518728+0000","last_became_active":"2024-06-29T15:38:38.365109+0000","last_became_peered":"2024-06-29T15:38:38.365109+0000","last_unstale":"2024-06-29T15:38:40.518728+0000","last_undegraded":"2024-06-29T15:38:40.518728+0000","last_fullsized":"2024-06-29T15:38:40.518728+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:47:18.411014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357070+0000","last_change":"2024-06-29T15:38:39.375287+0000","last_active":"2024-06-29T15:38:40.357070+0000","last_peered":"2024-06-29T15:38:40.357070+0000","last_clean":"2024-06-29T15:38:40.357070+0000","last_became_active":"2024-06-29T15:38:39.374964+0000","last_became_peered":"2024-06-29T15:38:39.374964+0000","last_unstale":"2024-06-29T15:38:40.357070+0000","last_undegraded":"2024-06-29T15:38:40.357070+0000","last_fullsized":"2024-06-29T15:38:40.357070+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:44.674713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691928+0000","last_change":"2024-06-29T15:38:38.370558+0000","last_active":"2024-06-29T15:38:40.691928+0000","last_peered":"2024-06-29T15:38:40.691928+0000","last_clean":"2024-06-29T15:38:40.691928+0000","last_became_active":"2024-06-29T15:38:38.370212+0000","last_became_peered":"2024-06-29T15:38:38.370212+0000","last_unstale":"2024-06-29T15:38:40.691928+0000","last_undegraded":"2024-06-29T15:38:40.691928+0000","last_fullsized":"2024-06-29T15:38:40.691928+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:10:59.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693196+0000","last_change":"2024-06-29T15:38:39.374950+0000","last_active":"2024-06-29T15:38:40.693196+0000","last_peered":"2024-06-29T15:38:40.693196+0000","last_clean":"2024-06-29T15:38:40.693196+0000","last_became_active":"2024-06-29T15:38:39.374565+0000","last_became_peered":"2024-06-29T15:38:39.374565+0000","last_unstale":"2024-06-29T15:38:40.693196+0000","last_undegraded":"2024-06-29T15:38:40.693196+0000","last_fullsized":"2024-06-29T15:38:40.693196+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:11.499263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358117+0000","last_change":"2024-06-29T15:38:38.377849+0000","last_active":"2024-06-29T15:38:40.358117+0000","last_peered":"2024-06-29T15:38:40.358117+0000","last_clean":"2024-06-29T15:38:40.358117+0000","last_became_active":"2024-06-29T15:38:38.377552+0000","last_became_peered":"2024-06-29T15:38:38.377552+0000","last_unstale":"2024-06-29T15:38:40.358117+0000","last_undegraded":"2024-06-29T15:38:40.358117+0000","last_fullsized":"2024-06-29T15:38:40.358117+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:01.311674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357499+0000","last_change":"2024-06-29T15:38:39.383189+0000","last_active":"2024-06-29T15:38:40.357499+0000","last_peered":"2024-06-29T15:38:40.357499+0000","last_clean":"2024-06-29T15:38:40.357499+0000","last_became_active":"2024-06-29T15:38:39.382725+0000","last_became_peered":"2024-06-29T15:38:39.382725+0000","last_unstale":"2024-06-29T15:38:40.357499+0000","last_undegraded":"2024-06-29T15:38:40.357499+0000","last_fullsized":"2024-06-29T15:38:40.357499+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:21.241394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357765+0000","last_change":"2024-06-29T15:38:38.371168+0000","last_active":"2024-06-29T15:38:40.357765+0000","last_peered":"2024-06-29T15:38:40.357765+0000","last_clean":"2024-06-29T15:38:40.357765+0000","last_became_active":"2024-06-29T15:38:38.370536+0000","last_became_peered":"2024-06-29T15:38:38.370536+0000","last_unstale":"2024-06-29T15:38:40.357765+0000","last_undegraded":"2024-06-29T15:38:40.357765+0000","last_fullsized":"2024-06-29T15:38:40.357765+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:21.086645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692502+0000","last_change":"2024-06-29T15:38:39.367619+0000","last_active":"2024-06-29T15:38:40.692502+0000","last_peered":"2024-06-29T15:38:40.692502+0000","last_clean":"2024-06-29T15:38:40.692502+0000","last_became_active":"2024-06-29T15:38:39.367078+0000","last_became_peered":"2024-06-29T15:38:39.367078+0000","last_unstale":"2024-06-29T15:38:40.692502+0000","last_undegraded":"2024-06-29T15:38:40.692502+0000","last_fullsized":"2024-06-29T15:38:40.692502+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:58.476902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692180+0000","last_change":"2024-06-29T15:38:38.372070+0000","last_active":"2024-06-29T15:38:40.692180+0000","last_peered":"2024-06-29T15:38:40.692180+0000","last_clean":"2024-06-29T15:38:40.692180+0000","last_became_active":"2024-06-29T15:38:38.371723+0000","last_became_peered":"2024-06-29T15:38:38.371723+0000","last_unstale":"2024-06-29T15:38:40.692180+0000","last_undegraded":"2024-06-29T15:38:40.692180+0000","last_fullsized":"2024-06-29T15:38:40.692180+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:06:19.130043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691749+0000","last_change":"2024-06-29T15:38:39.373425+0000","last_active":"2024-06-29T15:38:40.691749+0000","last_peered":"2024-06-29T15:38:40.691749+0000","last_clean":"2024-06-29T15:38:40.691749+0000","last_became_active":"2024-06-29T15:38:39.373136+0000","last_became_peered":"2024-06-29T15:38:39.373136+0000","last_unstale":"2024-06-29T15:38:40.691749+0000","last_undegraded":"2024-06-29T15:38:40.691749+0000","last_fullsized":"2024-06-29T15:38:40.691749+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:56.070386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"167'3","reported_seq":17,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.359230+0000","last_change":"2024-06-29T15:38:38.369778+0000","last_active":"2024-06-29T15:38:40.359230+0000","last_peered":"2024-06-29T15:38:40.359230+0000","last_clean":"2024-06-29T15:38:40.359230+0000","last_became_active":"2024-06-29T15:38:38.369402+0000","last_became_peered":"2024-06-29T15:38:38.369402+0000","last_unstale":"2024-06-29T15:38:40.359230+0000","last_undegraded":"2024-06-29T15:38:40.359230+0000","last_fullsized":"2024-06-29T15:38:40.359230+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:03.888658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"157'170","reported_seq":773,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357638+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:38:40.357638+0000","last_peered":"2024-06-29T15:38:40.357638+0000","last_clean":"2024-06-29T15:38:40.357638+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:38:40.357638+0000","last_undegraded":"2024-06-29T15:38:40.357638+0000","last_fullsized":"2024-06-29T15:38:40.357638+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":269,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":269,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575763,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60184,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311656,"statfs":{"total":96636764160,"available":96575135744,"internally_reserved":0,"allocated":2699264,"data_stored":1609724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575764,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41568,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330272,"statfs":{"total":96636764160,"available":96594198528,"internally_reserved":0,"allocated":2109440,"data_stored":1018850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575762,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41560,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330280,"statfs":{"total":96636764160,"available":96594206720,"internally_reserved":0,"allocated":2101248,"data_stored":1018726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55444,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316396,"statfs":{"total":96636764160,"available":96579989504,"internally_reserved":0,"allocated":2097152,"data_stored":1018704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59604,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312236,"statfs":{"total":96636764160,"available":96575729664,"internally_reserved":0,"allocated":2101248,"data_stored":1018794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575763,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63768,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308072,"statfs":{"total":96636764160,"available":96571465728,"internally_reserved":0,"allocated":2101248,"data_stored":1013986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575762,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59588,"kb_used_data":2040,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312252,"statfs":{"total":96636764160,"available":96575746048,"internally_reserved":0,"allocated":2088960,"data_stored":1012886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575763,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56020,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315820,"statfs":{"total":96636764160,"available":96579399680,"internally_reserved":0,"allocated":2695168,"data_stored":1604318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:38:43.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:38:43.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.556+0000 7ff3cda4d640 1 -- 172.21.15.81:0/1153211788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c80b7290 msgr2=0x7ff3c80b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.556+0000 7ff3cda4d640 1 --2- 172.21.15.81:0/1153211788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c80b7290 0x7ff3c80b76a0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80b9b80 tx=0x7ff3c002a610 comp rx=0 tx=0).stop 2024-06-29T15:38:43.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.556+0000 7ff3cda4d640 1 -- 172.21.15.81:0/1153211788 shutdown_connections 2024-06-29T15:38:43.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.556+0000 7ff3cda4d640 1 --2- 172.21.15.81:0/1153211788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c80b7290 0x7ff3c80b76a0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.556+0000 7ff3cda4d640 1 --2- 172.21.15.81:0/1153211788 >> v2:172.21.15.167:3300/0 conn(0x7ff3c80b5250 0x7ff3c80d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.556+0000 7ff3cda4d640 1 --2- 172.21.15.81:0/1153211788 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3c80b9040 0x7ff3c80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.556+0000 7ff3cda4d640 1 -- 172.21.15.81:0/1153211788 >> 172.21.15.81:0/1153211788 conn(0x7ff3c809cf30 msgr2=0x7ff3c80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.556+0000 7ff3cda4d640 1 -- 172.21.15.81:0/1153211788 shutdown_connections 2024-06-29T15:38:43.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.556+0000 7ff3cda4d640 1 -- 172.21.15.81:0/1153211788 wait complete. 2024-06-29T15:38:43.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.557+0000 7ff3cda4d640 1 Processor -- start 2024-06-29T15:38:43.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.558+0000 7ff3cda4d640 1 -- start start 2024-06-29T15:38:43.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3cda4d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3c80b5250 0x7ff3c81561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3cda4d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c80b7290 0x7ff3c80a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3cda4d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3c80b9040 0x7ff3c815fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3cda4d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c80b7ff0 con 0x7ff3c80b7290 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3cda4d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c80b7e70 con 0x7ff3c80b9040 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3cda4d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff3c80b8170 con 0x7ff3c80b5250 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c80b7290 0x7ff3c80a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c80b7290 0x7ff3c80a0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53786/0 (socket says 172.21.15.81:53786) 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c67fc640 1 -- 172.21.15.81:0/3930759351 learned_addr learned my addr 172.21.15.81:0/3930759351 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c6ffd640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3c80b5250 0x7ff3c81561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c77fe640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3c80b9040 0x7ff3c815fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c67fc640 1 -- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3c80b5250 msgr2=0x7ff3c81561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c67fc640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3c80b5250 0x7ff3c81561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c67fc640 1 -- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3c80b9040 msgr2=0x7ff3c815fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c67fc640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3c80b9040 0x7ff3c815fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c67fc640 1 -- 172.21.15.81:0/3930759351 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c809f180 con 0x7ff3c80b7290 2024-06-29T15:38:43.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c77fe640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3c80b9040 0x7ff3c815fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:43.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.559+0000 7ff3c67fc640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c80b7290 0x7ff3c80a0df0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00b3c0 tx=0x7ff3bc00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.560+0000 7ff3affff640 1 -- 172.21.15.81:0/3930759351 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc011830 con 0x7ff3c80b7290 2024-06-29T15:38:43.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.560+0000 7ff3affff640 1 -- 172.21.15.81:0/3930759351 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3bc016070 con 0x7ff3c80b7290 2024-06-29T15:38:43.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.560+0000 7ff3cda4d640 1 -- 172.21.15.81:0/3930759351 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c809f460 con 0x7ff3c80b7290 2024-06-29T15:38:43.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.560+0000 7ff3affff640 1 -- 172.21.15.81:0/3930759351 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc011d40 con 0x7ff3c80b7290 2024-06-29T15:38:43.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.560+0000 7ff3cda4d640 1 -- 172.21.15.81:0/3930759351 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c809fa20 con 0x7ff3c80b7290 2024-06-29T15:38:43.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.561+0000 7ff3cda4d640 1 -- 172.21.15.81:0/3930759351 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c80b76a0 con 0x7ff3c80b7290 2024-06-29T15:38:43.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.561+0000 7ff3affff640 1 -- 172.21.15.81:0/3930759351 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7ff3bc01b460 con 0x7ff3c80b7290 2024-06-29T15:38:43.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.562+0000 7ff3affff640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff3b4076420 0x7ff3b4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.562+0000 7ff3affff640 1 -- 172.21.15.81:0/3930759351 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7ff3bc097600 con 0x7ff3c80b7290 2024-06-29T15:38:43.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.562+0000 7ff3c6ffd640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff3b4076420 0x7ff3b4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.562+0000 7ff3c6ffd640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff3b4076420 0x7ff3b4078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800a400 tx=0x7ff3b800d040 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.566+0000 7ff3affff640 1 -- 172.21.15.81:0/3930759351 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff3bc060d50 con 0x7ff3c80b7290 2024-06-29T15:38:43.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.744+0000 7ff3cda4d640 1 -- 172.21.15.81:0/3930759351 --> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff3c806a2b0 con 0x7ff3b4076420 2024-06-29T15:38:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.749+0000 7ff3affff640 1 -- 172.21.15.81:0/3930759351 <== mgr.9414 v2:172.21.15.167:6832/150801987 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252493 (secure 0 0 0) 0x7ff3c806a2b0 con 0x7ff3b4076420 2024-06-29T15:38:43.753 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:43.754 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:38:43.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 -- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff3b4076420 msgr2=0x7ff3b4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff3b4076420 0x7ff3b4078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800a400 tx=0x7ff3b800d040 comp rx=0 tx=0).stop 2024-06-29T15:38:43.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 -- 172.21.15.81:0/3930759351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c80b7290 msgr2=0x7ff3c80a0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c80b7290 0x7ff3c80a0df0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00b3c0 tx=0x7ff3bc00b880 comp rx=0 tx=0).stop 2024-06-29T15:38:43.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 -- 172.21.15.81:0/3930759351 shutdown_connections 2024-06-29T15:38:43.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7ff3b4076420 0x7ff3b4078930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff3c80b9040 0x7ff3c815fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c80b7290 0x7ff3c80a0df0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 --2- 172.21.15.81:0/3930759351 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff3c80b5250 0x7ff3c81561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 -- 172.21.15.81:0/3930759351 >> 172.21.15.81:0/3930759351 conn(0x7ff3c809cf30 msgr2=0x7ff3c80d2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 -- 172.21.15.81:0/3930759351 shutdown_connections 2024-06-29T15:38:43.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.752+0000 7ff3cda4d640 1 -- 172.21.15.81:0/3930759351 wait complete. 2024-06-29T15:38:43.765 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:38:43.065296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":313,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":437736,"kb_used_data":17572,"kb_used_omap":12,"kb_used_meta":420019,"kb_avail":754536984,"statfs":{"total":773094113280,"available":772645871616,"internally_reserved":0,"allocated":17993728,"data_stored":9315988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":430100018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.722779"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.519030+0000","last_change":"2024-06-29T15:38:39.382691+0000","last_active":"2024-06-29T15:38:40.519030+0000","last_peered":"2024-06-29T15:38:40.519030+0000","last_clean":"2024-06-29T15:38:40.519030+0000","last_became_active":"2024-06-29T15:38:39.382328+0000","last_became_peered":"2024-06-29T15:38:39.382328+0000","last_unstale":"2024-06-29T15:38:40.519030+0000","last_undegraded":"2024-06-29T15:38:40.519030+0000","last_fullsized":"2024-06-29T15:38:40.519030+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:21:41.534117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692224+0000","last_change":"2024-06-29T15:38:39.368599+0000","last_active":"2024-06-29T15:38:40.692224+0000","last_peered":"2024-06-29T15:38:40.692224+0000","last_clean":"2024-06-29T15:38:40.692224+0000","last_became_active":"2024-06-29T15:38:39.368309+0000","last_became_peered":"2024-06-29T15:38:39.368309+0000","last_unstale":"2024-06-29T15:38:40.692224+0000","last_undegraded":"2024-06-29T15:38:40.692224+0000","last_fullsized":"2024-06-29T15:38:40.692224+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:30:21.876609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.359141+0000","last_change":"2024-06-29T15:38:39.368614+0000","last_active":"2024-06-29T15:38:40.359141+0000","last_peered":"2024-06-29T15:38:40.359141+0000","last_clean":"2024-06-29T15:38:40.359141+0000","last_became_active":"2024-06-29T15:38:39.368057+0000","last_became_peered":"2024-06-29T15:38:39.368057+0000","last_unstale":"2024-06-29T15:38:40.359141+0000","last_undegraded":"2024-06-29T15:38:40.359141+0000","last_fullsized":"2024-06-29T15:38:40.359141+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:34:52.641610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693098+0000","last_change":"2024-06-29T15:38:39.375177+0000","last_active":"2024-06-29T15:38:40.693098+0000","last_peered":"2024-06-29T15:38:40.693098+0000","last_clean":"2024-06-29T15:38:40.693098+0000","last_became_active":"2024-06-29T15:38:39.374875+0000","last_became_peered":"2024-06-29T15:38:39.374875+0000","last_unstale":"2024-06-29T15:38:40.693098+0000","last_undegraded":"2024-06-29T15:38:40.693098+0000","last_fullsized":"2024-06-29T15:38:40.693098+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:36:06.075398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692461+0000","last_change":"2024-06-29T15:38:39.380152+0000","last_active":"2024-06-29T15:38:40.692461+0000","last_peered":"2024-06-29T15:38:40.692461+0000","last_clean":"2024-06-29T15:38:40.692461+0000","last_became_active":"2024-06-29T15:38:39.379773+0000","last_became_peered":"2024-06-29T15:38:39.379773+0000","last_unstale":"2024-06-29T15:38:40.692461+0000","last_undegraded":"2024-06-29T15:38:40.692461+0000","last_fullsized":"2024-06-29T15:38:40.692461+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:35:51.885387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518881+0000","last_change":"2024-06-29T15:38:39.365466+0000","last_active":"2024-06-29T15:38:40.518881+0000","last_peered":"2024-06-29T15:38:40.518881+0000","last_clean":"2024-06-29T15:38:40.518881+0000","last_became_active":"2024-06-29T15:38:39.363169+0000","last_became_peered":"2024-06-29T15:38:39.363169+0000","last_unstale":"2024-06-29T15:38:40.518881+0000","last_undegraded":"2024-06-29T15:38:40.518881+0000","last_fullsized":"2024-06-29T15:38:40.518881+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:16:31.663745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691759+0000","last_change":"2024-06-29T15:38:39.371745+0000","last_active":"2024-06-29T15:38:40.691759+0000","last_peered":"2024-06-29T15:38:40.691759+0000","last_clean":"2024-06-29T15:38:40.691759+0000","last_became_active":"2024-06-29T15:38:39.371453+0000","last_became_peered":"2024-06-29T15:38:39.371453+0000","last_unstale":"2024-06-29T15:38:40.691759+0000","last_undegraded":"2024-06-29T15:38:40.691759+0000","last_fullsized":"2024-06-29T15:38:40.691759+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:01.154044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518622+0000","last_change":"2024-06-29T15:38:39.363138+0000","last_active":"2024-06-29T15:38:40.518622+0000","last_peered":"2024-06-29T15:38:40.518622+0000","last_clean":"2024-06-29T15:38:40.518622+0000","last_became_active":"2024-06-29T15:38:39.362832+0000","last_became_peered":"2024-06-29T15:38:39.362832+0000","last_unstale":"2024-06-29T15:38:40.518622+0000","last_undegraded":"2024-06-29T15:38:40.518622+0000","last_fullsized":"2024-06-29T15:38:40.518622+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:48.908922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691989+0000","last_change":"2024-06-29T15:38:39.363738+0000","last_active":"2024-06-29T15:38:40.691989+0000","last_peered":"2024-06-29T15:38:40.691989+0000","last_clean":"2024-06-29T15:38:40.691989+0000","last_became_active":"2024-06-29T15:38:39.363436+0000","last_became_peered":"2024-06-29T15:38:39.363436+0000","last_unstale":"2024-06-29T15:38:40.691989+0000","last_undegraded":"2024-06-29T15:38:40.691989+0000","last_fullsized":"2024-06-29T15:38:40.691989+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:30:54.895627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358450+0000","last_change":"2024-06-29T15:38:39.388372+0000","last_active":"2024-06-29T15:38:40.358450+0000","last_peered":"2024-06-29T15:38:40.358450+0000","last_clean":"2024-06-29T15:38:40.358450+0000","last_became_active":"2024-06-29T15:38:39.387788+0000","last_became_peered":"2024-06-29T15:38:39.387788+0000","last_unstale":"2024-06-29T15:38:40.358450+0000","last_undegraded":"2024-06-29T15:38:40.358450+0000","last_fullsized":"2024-06-29T15:38:40.358450+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:38.995705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518901+0000","last_change":"2024-06-29T15:38:39.371096+0000","last_active":"2024-06-29T15:38:40.518901+0000","last_peered":"2024-06-29T15:38:40.518901+0000","last_clean":"2024-06-29T15:38:40.518901+0000","last_became_active":"2024-06-29T15:38:39.370547+0000","last_became_peered":"2024-06-29T15:38:39.370547+0000","last_unstale":"2024-06-29T15:38:40.518901+0000","last_undegraded":"2024-06-29T15:38:40.518901+0000","last_fullsized":"2024-06-29T15:38:40.518901+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:43.076403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358311+0000","last_change":"2024-06-29T15:38:39.381688+0000","last_active":"2024-06-29T15:38:40.358311+0000","last_peered":"2024-06-29T15:38:40.358311+0000","last_clean":"2024-06-29T15:38:40.358311+0000","last_became_active":"2024-06-29T15:38:39.381364+0000","last_became_peered":"2024-06-29T15:38:39.381364+0000","last_unstale":"2024-06-29T15:38:40.358311+0000","last_undegraded":"2024-06-29T15:38:40.358311+0000","last_fullsized":"2024-06-29T15:38:40.358311+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:43:53.819101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692998+0000","last_change":"2024-06-29T15:38:39.378616+0000","last_active":"2024-06-29T15:38:40.692998+0000","last_peered":"2024-06-29T15:38:40.692998+0000","last_clean":"2024-06-29T15:38:40.692998+0000","last_became_active":"2024-06-29T15:38:39.378301+0000","last_became_peered":"2024-06-29T15:38:39.378301+0000","last_unstale":"2024-06-29T15:38:40.692998+0000","last_undegraded":"2024-06-29T15:38:40.692998+0000","last_fullsized":"2024-06-29T15:38:40.692998+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:44.674713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692553+0000","last_change":"2024-06-29T15:38:39.382903+0000","last_active":"2024-06-29T15:38:40.692553+0000","last_peered":"2024-06-29T15:38:40.692553+0000","last_clean":"2024-06-29T15:38:40.692553+0000","last_became_active":"2024-06-29T15:38:39.382535+0000","last_became_peered":"2024-06-29T15:38:39.382535+0000","last_unstale":"2024-06-29T15:38:40.692553+0000","last_undegraded":"2024-06-29T15:38:40.692553+0000","last_fullsized":"2024-06-29T15:38:40.692553+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:05:55.638711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357367+0000","last_change":"2024-06-29T15:38:39.379309+0000","last_active":"2024-06-29T15:38:40.357367+0000","last_peered":"2024-06-29T15:38:40.357367+0000","last_clean":"2024-06-29T15:38:40.357367+0000","last_became_active":"2024-06-29T15:38:39.379021+0000","last_became_peered":"2024-06-29T15:38:39.379021+0000","last_unstale":"2024-06-29T15:38:40.357367+0000","last_undegraded":"2024-06-29T15:38:40.357367+0000","last_fullsized":"2024-06-29T15:38:40.357367+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:41.525177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.356992+0000","last_change":"2024-06-29T15:38:39.363595+0000","last_active":"2024-06-29T15:38:40.356992+0000","last_peered":"2024-06-29T15:38:40.356992+0000","last_clean":"2024-06-29T15:38:40.356992+0000","last_became_active":"2024-06-29T15:38:39.363307+0000","last_became_peered":"2024-06-29T15:38:39.363307+0000","last_unstale":"2024-06-29T15:38:40.356992+0000","last_undegraded":"2024-06-29T15:38:40.356992+0000","last_fullsized":"2024-06-29T15:38:40.356992+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:00.534645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.359055+0000","last_change":"2024-06-29T15:38:39.386449+0000","last_active":"2024-06-29T15:38:40.359055+0000","last_peered":"2024-06-29T15:38:40.359055+0000","last_clean":"2024-06-29T15:38:40.359055+0000","last_became_active":"2024-06-29T15:38:39.385697+0000","last_became_peered":"2024-06-29T15:38:39.385697+0000","last_unstale":"2024-06-29T15:38:40.359055+0000","last_undegraded":"2024-06-29T15:38:40.359055+0000","last_fullsized":"2024-06-29T15:38:40.359055+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:10.064127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358769+0000","last_change":"2024-06-29T15:38:39.365403+0000","last_active":"2024-06-29T15:38:40.358769+0000","last_peered":"2024-06-29T15:38:40.358769+0000","last_clean":"2024-06-29T15:38:40.358769+0000","last_became_active":"2024-06-29T15:38:39.365029+0000","last_became_peered":"2024-06-29T15:38:39.365029+0000","last_unstale":"2024-06-29T15:38:40.358769+0000","last_undegraded":"2024-06-29T15:38:40.358769+0000","last_fullsized":"2024-06-29T15:38:40.358769+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:03.058907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.519052+0000","last_change":"2024-06-29T15:38:39.366364+0000","last_active":"2024-06-29T15:38:40.519052+0000","last_peered":"2024-06-29T15:38:40.519052+0000","last_clean":"2024-06-29T15:38:40.519052+0000","last_became_active":"2024-06-29T15:38:39.365547+0000","last_became_peered":"2024-06-29T15:38:39.365547+0000","last_unstale":"2024-06-29T15:38:40.519052+0000","last_undegraded":"2024-06-29T15:38:40.519052+0000","last_fullsized":"2024-06-29T15:38:40.519052+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:29.795879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357254+0000","last_change":"2024-06-29T15:38:39.366096+0000","last_active":"2024-06-29T15:38:40.357254+0000","last_peered":"2024-06-29T15:38:40.357254+0000","last_clean":"2024-06-29T15:38:40.357254+0000","last_became_active":"2024-06-29T15:38:39.365775+0000","last_became_peered":"2024-06-29T15:38:39.365775+0000","last_unstale":"2024-06-29T15:38:40.357254+0000","last_undegraded":"2024-06-29T15:38:40.357254+0000","last_fullsized":"2024-06-29T15:38:40.357254+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:03.058907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.359312+0000","last_change":"2024-06-29T15:38:39.388084+0000","last_active":"2024-06-29T15:38:40.359312+0000","last_peered":"2024-06-29T15:38:40.359312+0000","last_clean":"2024-06-29T15:38:40.359312+0000","last_became_active":"2024-06-29T15:38:39.387765+0000","last_became_peered":"2024-06-29T15:38:39.387765+0000","last_unstale":"2024-06-29T15:38:40.359312+0000","last_undegraded":"2024-06-29T15:38:40.359312+0000","last_fullsized":"2024-06-29T15:38:40.359312+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:25:27.774536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692765+0000","last_change":"2024-06-29T15:38:39.371056+0000","last_active":"2024-06-29T15:38:40.692765+0000","last_peered":"2024-06-29T15:38:40.692765+0000","last_clean":"2024-06-29T15:38:40.692765+0000","last_became_active":"2024-06-29T15:38:39.370715+0000","last_became_peered":"2024-06-29T15:38:39.370715+0000","last_unstale":"2024-06-29T15:38:40.692765+0000","last_undegraded":"2024-06-29T15:38:40.692765+0000","last_fullsized":"2024-06-29T15:38:40.692765+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:35:45.940249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693279+0000","last_change":"2024-06-29T15:38:39.368931+0000","last_active":"2024-06-29T15:38:40.693279+0000","last_peered":"2024-06-29T15:38:40.693279+0000","last_clean":"2024-06-29T15:38:40.693279+0000","last_became_active":"2024-06-29T15:38:39.368545+0000","last_became_peered":"2024-06-29T15:38:39.368545+0000","last_unstale":"2024-06-29T15:38:40.693279+0000","last_undegraded":"2024-06-29T15:38:40.693279+0000","last_fullsized":"2024-06-29T15:38:40.693279+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:35:31.375256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357785+0000","last_change":"2024-06-29T15:38:39.388352+0000","last_active":"2024-06-29T15:38:40.357785+0000","last_peered":"2024-06-29T15:38:40.357785+0000","last_clean":"2024-06-29T15:38:40.357785+0000","last_became_active":"2024-06-29T15:38:39.388036+0000","last_became_peered":"2024-06-29T15:38:39.388036+0000","last_unstale":"2024-06-29T15:38:40.357785+0000","last_undegraded":"2024-06-29T15:38:40.357785+0000","last_fullsized":"2024-06-29T15:38:40.357785+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:50:25.514462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357170+0000","last_change":"2024-06-29T15:38:39.363840+0000","last_active":"2024-06-29T15:38:40.357170+0000","last_peered":"2024-06-29T15:38:40.357170+0000","last_clean":"2024-06-29T15:38:40.357170+0000","last_became_active":"2024-06-29T15:38:39.363327+0000","last_became_peered":"2024-06-29T15:38:39.363327+0000","last_unstale":"2024-06-29T15:38:40.357170+0000","last_undegraded":"2024-06-29T15:38:40.357170+0000","last_fullsized":"2024-06-29T15:38:40.357170+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:45:04.155625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518843+0000","last_change":"2024-06-29T15:38:39.364207+0000","last_active":"2024-06-29T15:38:40.518843+0000","last_peered":"2024-06-29T15:38:40.518843+0000","last_clean":"2024-06-29T15:38:40.518843+0000","last_became_active":"2024-06-29T15:38:39.363896+0000","last_became_peered":"2024-06-29T15:38:39.363896+0000","last_unstale":"2024-06-29T15:38:40.518843+0000","last_undegraded":"2024-06-29T15:38:40.518843+0000","last_fullsized":"2024-06-29T15:38:40.518843+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:09.992782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357378+0000","last_change":"2024-06-29T15:38:39.371562+0000","last_active":"2024-06-29T15:38:40.357378+0000","last_peered":"2024-06-29T15:38:40.357378+0000","last_clean":"2024-06-29T15:38:40.357378+0000","last_became_active":"2024-06-29T15:38:39.371266+0000","last_became_peered":"2024-06-29T15:38:39.371266+0000","last_unstale":"2024-06-29T15:38:40.357378+0000","last_undegraded":"2024-06-29T15:38:40.357378+0000","last_fullsized":"2024-06-29T15:38:40.357378+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:34:52.641610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691701+0000","last_change":"2024-06-29T15:38:39.374845+0000","last_active":"2024-06-29T15:38:40.691701+0000","last_peered":"2024-06-29T15:38:40.691701+0000","last_clean":"2024-06-29T15:38:40.691701+0000","last_became_active":"2024-06-29T15:38:39.374537+0000","last_became_peered":"2024-06-29T15:38:39.374537+0000","last_unstale":"2024-06-29T15:38:40.691701+0000","last_undegraded":"2024-06-29T15:38:40.691701+0000","last_fullsized":"2024-06-29T15:38:40.691701+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:48.908922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518661+0000","last_change":"2024-06-29T15:38:39.365275+0000","last_active":"2024-06-29T15:38:40.518661+0000","last_peered":"2024-06-29T15:38:40.518661+0000","last_clean":"2024-06-29T15:38:40.518661+0000","last_became_active":"2024-06-29T15:38:39.364775+0000","last_became_peered":"2024-06-29T15:38:39.364775+0000","last_unstale":"2024-06-29T15:38:40.518661+0000","last_undegraded":"2024-06-29T15:38:40.518661+0000","last_fullsized":"2024-06-29T15:38:40.518661+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:56.070386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692809+0000","last_change":"2024-06-29T15:38:39.374266+0000","last_active":"2024-06-29T15:38:40.692809+0000","last_peered":"2024-06-29T15:38:40.692809+0000","last_clean":"2024-06-29T15:38:40.692809+0000","last_became_active":"2024-06-29T15:38:39.373923+0000","last_became_peered":"2024-06-29T15:38:39.373923+0000","last_unstale":"2024-06-29T15:38:40.692809+0000","last_undegraded":"2024-06-29T15:38:40.692809+0000","last_fullsized":"2024-06-29T15:38:40.692809+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:19:08.383472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692146+0000","last_change":"2024-06-29T15:38:39.374954+0000","last_active":"2024-06-29T15:38:40.692146+0000","last_peered":"2024-06-29T15:38:40.692146+0000","last_clean":"2024-06-29T15:38:40.692146+0000","last_became_active":"2024-06-29T15:38:39.374637+0000","last_became_peered":"2024-06-29T15:38:39.374637+0000","last_unstale":"2024-06-29T15:38:40.692146+0000","last_undegraded":"2024-06-29T15:38:40.692146+0000","last_fullsized":"2024-06-29T15:38:40.692146+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:33:26.406721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357404+0000","last_change":"2024-06-29T15:38:39.363765+0000","last_active":"2024-06-29T15:38:40.357404+0000","last_peered":"2024-06-29T15:38:40.357404+0000","last_clean":"2024-06-29T15:38:40.357404+0000","last_became_active":"2024-06-29T15:38:39.363488+0000","last_became_peered":"2024-06-29T15:38:39.363488+0000","last_unstale":"2024-06-29T15:38:40.357404+0000","last_undegraded":"2024-06-29T15:38:40.357404+0000","last_fullsized":"2024-06-29T15:38:40.357404+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:48.908922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358480+0000","last_change":"2024-06-29T15:38:39.384650+0000","last_active":"2024-06-29T15:38:40.358480+0000","last_peered":"2024-06-29T15:38:40.358480+0000","last_clean":"2024-06-29T15:38:40.358480+0000","last_became_active":"2024-06-29T15:38:39.384319+0000","last_became_peered":"2024-06-29T15:38:39.384319+0000","last_unstale":"2024-06-29T15:38:40.358480+0000","last_undegraded":"2024-06-29T15:38:40.358480+0000","last_fullsized":"2024-06-29T15:38:40.358480+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:26:32.414242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"167'4","reported_seq":18,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358863+0000","last_change":"2024-06-29T15:38:38.375263+0000","last_active":"2024-06-29T15:38:40.358863+0000","last_peered":"2024-06-29T15:38:40.358863+0000","last_clean":"2024-06-29T15:38:40.358863+0000","last_became_active":"2024-06-29T15:38:38.374826+0000","last_became_peered":"2024-06-29T15:38:38.374826+0000","last_unstale":"2024-06-29T15:38:40.358863+0000","last_undegraded":"2024-06-29T15:38:40.358863+0000","last_fullsized":"2024-06-29T15:38:40.358863+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:56.527020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692022+0000","last_change":"2024-06-29T15:38:39.367987+0000","last_active":"2024-06-29T15:38:40.692022+0000","last_peered":"2024-06-29T15:38:40.692022+0000","last_clean":"2024-06-29T15:38:40.692022+0000","last_became_active":"2024-06-29T15:38:39.367679+0000","last_became_peered":"2024-06-29T15:38:39.367679+0000","last_unstale":"2024-06-29T15:38:40.692022+0000","last_undegraded":"2024-06-29T15:38:40.692022+0000","last_fullsized":"2024-06-29T15:38:40.692022+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:10:55.685876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"167'1","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692251+0000","last_change":"2024-06-29T15:38:38.372700+0000","last_active":"2024-06-29T15:38:40.692251+0000","last_peered":"2024-06-29T15:38:40.692251+0000","last_clean":"2024-06-29T15:38:40.692251+0000","last_became_active":"2024-06-29T15:38:38.372184+0000","last_became_peered":"2024-06-29T15:38:38.372184+0000","last_unstale":"2024-06-29T15:38:40.692251+0000","last_undegraded":"2024-06-29T15:38:40.692251+0000","last_fullsized":"2024-06-29T15:38:40.692251+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:15.462869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518701+0000","last_change":"2024-06-29T15:38:39.369624+0000","last_active":"2024-06-29T15:38:40.518701+0000","last_peered":"2024-06-29T15:38:40.518701+0000","last_clean":"2024-06-29T15:38:40.518701+0000","last_became_active":"2024-06-29T15:38:39.369309+0000","last_became_peered":"2024-06-29T15:38:39.369309+0000","last_unstale":"2024-06-29T15:38:40.518701+0000","last_undegraded":"2024-06-29T15:38:40.518701+0000","last_fullsized":"2024-06-29T15:38:40.518701+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:01.154044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693110+0000","last_change":"2024-06-29T15:38:38.370062+0000","last_active":"2024-06-29T15:38:40.693110+0000","last_peered":"2024-06-29T15:38:40.693110+0000","last_clean":"2024-06-29T15:38:40.693110+0000","last_became_active":"2024-06-29T15:38:38.368922+0000","last_became_peered":"2024-06-29T15:38:38.368922+0000","last_unstale":"2024-06-29T15:38:40.693110+0000","last_undegraded":"2024-06-29T15:38:40.693110+0000","last_fullsized":"2024-06-29T15:38:40.693110+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:06.829962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358773+0000","last_change":"2024-06-29T15:38:39.378397+0000","last_active":"2024-06-29T15:38:40.358773+0000","last_peered":"2024-06-29T15:38:40.358773+0000","last_clean":"2024-06-29T15:38:40.358773+0000","last_became_active":"2024-06-29T15:38:39.378104+0000","last_became_peered":"2024-06-29T15:38:39.378104+0000","last_unstale":"2024-06-29T15:38:40.358773+0000","last_undegraded":"2024-06-29T15:38:40.358773+0000","last_fullsized":"2024-06-29T15:38:40.358773+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:07:41.573791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692967+0000","last_change":"2024-06-29T15:38:38.369367+0000","last_active":"2024-06-29T15:38:40.692967+0000","last_peered":"2024-06-29T15:38:40.692967+0000","last_clean":"2024-06-29T15:38:40.692967+0000","last_became_active":"2024-06-29T15:38:38.369022+0000","last_became_peered":"2024-06-29T15:38:38.369022+0000","last_unstale":"2024-06-29T15:38:40.692967+0000","last_undegraded":"2024-06-29T15:38:40.692967+0000","last_fullsized":"2024-06-29T15:38:40.692967+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:21:40.525440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692428+0000","last_change":"2024-06-29T15:38:39.366918+0000","last_active":"2024-06-29T15:38:40.692428+0000","last_peered":"2024-06-29T15:38:40.692428+0000","last_clean":"2024-06-29T15:38:40.692428+0000","last_became_active":"2024-06-29T15:38:39.366617+0000","last_became_peered":"2024-06-29T15:38:39.366617+0000","last_unstale":"2024-06-29T15:38:40.692428+0000","last_undegraded":"2024-06-29T15:38:40.692428+0000","last_fullsized":"2024-06-29T15:38:40.692428+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:33.770741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"167'1","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357422+0000","last_change":"2024-06-29T15:38:38.364862+0000","last_active":"2024-06-29T15:38:40.357422+0000","last_peered":"2024-06-29T15:38:40.357422+0000","last_clean":"2024-06-29T15:38:40.357422+0000","last_became_active":"2024-06-29T15:38:38.364529+0000","last_became_peered":"2024-06-29T15:38:38.364529+0000","last_unstale":"2024-06-29T15:38:40.357422+0000","last_undegraded":"2024-06-29T15:38:40.357422+0000","last_fullsized":"2024-06-29T15:38:40.357422+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:21.086645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693151+0000","last_change":"2024-06-29T15:38:39.373959+0000","last_active":"2024-06-29T15:38:40.693151+0000","last_peered":"2024-06-29T15:38:40.693151+0000","last_clean":"2024-06-29T15:38:40.693151+0000","last_became_active":"2024-06-29T15:38:39.373646+0000","last_became_peered":"2024-06-29T15:38:39.373646+0000","last_unstale":"2024-06-29T15:38:40.693151+0000","last_undegraded":"2024-06-29T15:38:40.693151+0000","last_fullsized":"2024-06-29T15:38:40.693151+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:21:11.326784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"167'1","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692557+0000","last_change":"2024-06-29T15:38:38.363928+0000","last_active":"2024-06-29T15:38:40.692557+0000","last_peered":"2024-06-29T15:38:40.692557+0000","last_clean":"2024-06-29T15:38:40.692557+0000","last_became_active":"2024-06-29T15:38:38.363553+0000","last_became_peered":"2024-06-29T15:38:38.363553+0000","last_unstale":"2024-06-29T15:38:40.692557+0000","last_undegraded":"2024-06-29T15:38:40.692557+0000","last_fullsized":"2024-06-29T15:38:40.692557+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:46.451406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692237+0000","last_change":"2024-06-29T15:38:39.371897+0000","last_active":"2024-06-29T15:38:40.692237+0000","last_peered":"2024-06-29T15:38:40.692237+0000","last_clean":"2024-06-29T15:38:40.692237+0000","last_became_active":"2024-06-29T15:38:39.371582+0000","last_became_peered":"2024-06-29T15:38:39.371582+0000","last_unstale":"2024-06-29T15:38:40.692237+0000","last_undegraded":"2024-06-29T15:38:40.692237+0000","last_fullsized":"2024-06-29T15:38:40.692237+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:47:52.311087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357145+0000","last_change":"2024-06-29T15:38:38.367400+0000","last_active":"2024-06-29T15:38:40.357145+0000","last_peered":"2024-06-29T15:38:40.357145+0000","last_clean":"2024-06-29T15:38:40.357145+0000","last_became_active":"2024-06-29T15:38:38.367054+0000","last_became_peered":"2024-06-29T15:38:38.367054+0000","last_unstale":"2024-06-29T15:38:40.357145+0000","last_undegraded":"2024-06-29T15:38:40.357145+0000","last_fullsized":"2024-06-29T15:38:40.357145+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:35:38.344362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692388+0000","last_change":"2024-06-29T15:38:39.367679+0000","last_active":"2024-06-29T15:38:40.692388+0000","last_peered":"2024-06-29T15:38:40.692388+0000","last_clean":"2024-06-29T15:38:40.692388+0000","last_became_active":"2024-06-29T15:38:39.366673+0000","last_became_peered":"2024-06-29T15:38:39.366673+0000","last_unstale":"2024-06-29T15:38:40.692388+0000","last_undegraded":"2024-06-29T15:38:40.692388+0000","last_fullsized":"2024-06-29T15:38:40.692388+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:24.789828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692250+0000","last_change":"2024-06-29T15:38:38.373514+0000","last_active":"2024-06-29T15:38:40.692250+0000","last_peered":"2024-06-29T15:38:40.692250+0000","last_clean":"2024-06-29T15:38:40.692250+0000","last_became_active":"2024-06-29T15:38:38.373176+0000","last_became_peered":"2024-06-29T15:38:38.373176+0000","last_unstale":"2024-06-29T15:38:40.692250+0000","last_undegraded":"2024-06-29T15:38:40.692250+0000","last_fullsized":"2024-06-29T15:38:40.692250+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:42.067726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692986+0000","last_change":"2024-06-29T15:38:39.366698+0000","last_active":"2024-06-29T15:38:40.692986+0000","last_peered":"2024-06-29T15:38:40.692986+0000","last_clean":"2024-06-29T15:38:40.692986+0000","last_became_active":"2024-06-29T15:38:39.366358+0000","last_became_peered":"2024-06-29T15:38:39.366358+0000","last_unstale":"2024-06-29T15:38:40.692986+0000","last_undegraded":"2024-06-29T15:38:40.692986+0000","last_fullsized":"2024-06-29T15:38:40.692986+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:00.534645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692121+0000","last_change":"2024-06-29T15:38:38.366411+0000","last_active":"2024-06-29T15:38:40.692121+0000","last_peered":"2024-06-29T15:38:40.692121+0000","last_clean":"2024-06-29T15:38:40.692121+0000","last_became_active":"2024-06-29T15:38:38.366076+0000","last_became_peered":"2024-06-29T15:38:38.366076+0000","last_unstale":"2024-06-29T15:38:40.692121+0000","last_undegraded":"2024-06-29T15:38:40.692121+0000","last_fullsized":"2024-06-29T15:38:40.692121+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:20.817965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358724+0000","last_change":"2024-06-29T15:38:39.386156+0000","last_active":"2024-06-29T15:38:40.358724+0000","last_peered":"2024-06-29T15:38:40.358724+0000","last_clean":"2024-06-29T15:38:40.358724+0000","last_became_active":"2024-06-29T15:38:39.385596+0000","last_became_peered":"2024-06-29T15:38:39.385596+0000","last_unstale":"2024-06-29T15:38:40.358724+0000","last_undegraded":"2024-06-29T15:38:40.358724+0000","last_fullsized":"2024-06-29T15:38:40.358724+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:44:11.976563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358612+0000","last_change":"2024-06-29T15:38:38.375628+0000","last_active":"2024-06-29T15:38:40.358612+0000","last_peered":"2024-06-29T15:38:40.358612+0000","last_clean":"2024-06-29T15:38:40.358612+0000","last_became_active":"2024-06-29T15:38:38.375252+0000","last_became_peered":"2024-06-29T15:38:38.375252+0000","last_unstale":"2024-06-29T15:38:40.358612+0000","last_undegraded":"2024-06-29T15:38:40.358612+0000","last_fullsized":"2024-06-29T15:38:40.358612+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:57:44.846550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357247+0000","last_change":"2024-06-29T15:38:39.366315+0000","last_active":"2024-06-29T15:38:40.357247+0000","last_peered":"2024-06-29T15:38:40.357247+0000","last_clean":"2024-06-29T15:38:40.357247+0000","last_became_active":"2024-06-29T15:38:39.365982+0000","last_became_peered":"2024-06-29T15:38:39.365982+0000","last_unstale":"2024-06-29T15:38:40.357247+0000","last_undegraded":"2024-06-29T15:38:40.357247+0000","last_fullsized":"2024-06-29T15:38:40.357247+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:11.499263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"167'6","reported_seq":20,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691857+0000","last_change":"2024-06-29T15:38:38.373640+0000","last_active":"2024-06-29T15:38:40.691857+0000","last_peered":"2024-06-29T15:38:40.691857+0000","last_clean":"2024-06-29T15:38:40.691857+0000","last_became_active":"2024-06-29T15:38:38.373288+0000","last_became_peered":"2024-06-29T15:38:38.373288+0000","last_unstale":"2024-06-29T15:38:40.691857+0000","last_undegraded":"2024-06-29T15:38:40.691857+0000","last_fullsized":"2024-06-29T15:38:40.691857+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:40.276368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357635+0000","last_change":"2024-06-29T15:38:39.373107+0000","last_active":"2024-06-29T15:38:40.357635+0000","last_peered":"2024-06-29T15:38:40.357635+0000","last_clean":"2024-06-29T15:38:40.357635+0000","last_became_active":"2024-06-29T15:38:39.372780+0000","last_became_peered":"2024-06-29T15:38:39.372780+0000","last_unstale":"2024-06-29T15:38:40.357635+0000","last_undegraded":"2024-06-29T15:38:40.357635+0000","last_fullsized":"2024-06-29T15:38:40.357635+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:44.674713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358255+0000","last_change":"2024-06-29T15:38:38.370647+0000","last_active":"2024-06-29T15:38:40.358255+0000","last_peered":"2024-06-29T15:38:40.358255+0000","last_clean":"2024-06-29T15:38:40.358255+0000","last_became_active":"2024-06-29T15:38:38.370337+0000","last_became_peered":"2024-06-29T15:38:38.370337+0000","last_unstale":"2024-06-29T15:38:40.358255+0000","last_undegraded":"2024-06-29T15:38:40.358255+0000","last_fullsized":"2024-06-29T15:38:40.358255+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:43:11.097338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691923+0000","last_change":"2024-06-29T15:38:39.368162+0000","last_active":"2024-06-29T15:38:40.691923+0000","last_peered":"2024-06-29T15:38:40.691923+0000","last_clean":"2024-06-29T15:38:40.691923+0000","last_became_active":"2024-06-29T15:38:39.367819+0000","last_became_peered":"2024-06-29T15:38:39.367819+0000","last_unstale":"2024-06-29T15:38:40.691923+0000","last_undegraded":"2024-06-29T15:38:40.691923+0000","last_fullsized":"2024-06-29T15:38:40.691923+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:52.382697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357588+0000","last_change":"2024-06-29T15:38:38.369584+0000","last_active":"2024-06-29T15:38:40.357588+0000","last_peered":"2024-06-29T15:38:40.357588+0000","last_clean":"2024-06-29T15:38:40.357588+0000","last_became_active":"2024-06-29T15:38:38.369067+0000","last_became_peered":"2024-06-29T15:38:38.369067+0000","last_unstale":"2024-06-29T15:38:40.357588+0000","last_undegraded":"2024-06-29T15:38:40.357588+0000","last_fullsized":"2024-06-29T15:38:40.357588+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:34.851686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692760+0000","last_change":"2024-06-29T15:38:39.382892+0000","last_active":"2024-06-29T15:38:40.692760+0000","last_peered":"2024-06-29T15:38:40.692760+0000","last_clean":"2024-06-29T15:38:40.692760+0000","last_became_active":"2024-06-29T15:38:39.382576+0000","last_became_peered":"2024-06-29T15:38:39.382576+0000","last_unstale":"2024-06-29T15:38:40.692760+0000","last_undegraded":"2024-06-29T15:38:40.692760+0000","last_fullsized":"2024-06-29T15:38:40.692760+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:40:41.285045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518549+0000","last_change":"2024-06-29T15:38:38.369546+0000","last_active":"2024-06-29T15:38:40.518549+0000","last_peered":"2024-06-29T15:38:40.518549+0000","last_clean":"2024-06-29T15:38:40.518549+0000","last_became_active":"2024-06-29T15:38:38.369167+0000","last_became_peered":"2024-06-29T15:38:38.369167+0000","last_unstale":"2024-06-29T15:38:40.518549+0000","last_undegraded":"2024-06-29T15:38:40.518549+0000","last_fullsized":"2024-06-29T15:38:40.518549+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:14.166568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357791+0000","last_change":"2024-06-29T15:38:39.367649+0000","last_active":"2024-06-29T15:38:40.357791+0000","last_peered":"2024-06-29T15:38:40.357791+0000","last_clean":"2024-06-29T15:38:40.357791+0000","last_became_active":"2024-06-29T15:38:39.367354+0000","last_became_peered":"2024-06-29T15:38:39.367354+0000","last_unstale":"2024-06-29T15:38:40.357791+0000","last_undegraded":"2024-06-29T15:38:40.357791+0000","last_fullsized":"2024-06-29T15:38:40.357791+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:45:04.155625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"167'8","reported_seq":22,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357446+0000","last_change":"2024-06-29T15:38:38.372405+0000","last_active":"2024-06-29T15:38:40.357446+0000","last_peered":"2024-06-29T15:38:40.357446+0000","last_clean":"2024-06-29T15:38:40.357446+0000","last_became_active":"2024-06-29T15:38:38.372012+0000","last_became_peered":"2024-06-29T15:38:38.372012+0000","last_unstale":"2024-06-29T15:38:40.357446+0000","last_undegraded":"2024-06-29T15:38:40.357446+0000","last_fullsized":"2024-06-29T15:38:40.357446+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:03.888658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358789+0000","last_change":"2024-06-29T15:38:39.386031+0000","last_active":"2024-06-29T15:38:40.358789+0000","last_peered":"2024-06-29T15:38:40.358789+0000","last_clean":"2024-06-29T15:38:40.358789+0000","last_became_active":"2024-06-29T15:38:39.385661+0000","last_became_peered":"2024-06-29T15:38:39.385661+0000","last_unstale":"2024-06-29T15:38:40.358789+0000","last_undegraded":"2024-06-29T15:38:40.358789+0000","last_fullsized":"2024-06-29T15:38:40.358789+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:46:46.673043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"167'1","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692649+0000","last_change":"2024-06-29T15:38:38.371901+0000","last_active":"2024-06-29T15:38:40.692649+0000","last_peered":"2024-06-29T15:38:40.692649+0000","last_clean":"2024-06-29T15:38:40.692649+0000","last_became_active":"2024-06-29T15:38:38.371581+0000","last_became_peered":"2024-06-29T15:38:38.371581+0000","last_unstale":"2024-06-29T15:38:40.692649+0000","last_undegraded":"2024-06-29T15:38:40.692649+0000","last_fullsized":"2024-06-29T15:38:40.692649+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:09.205714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357352+0000","last_change":"2024-06-29T15:38:39.381194+0000","last_active":"2024-06-29T15:38:40.357352+0000","last_peered":"2024-06-29T15:38:40.357352+0000","last_clean":"2024-06-29T15:38:40.357352+0000","last_became_active":"2024-06-29T15:38:39.380871+0000","last_became_peered":"2024-06-29T15:38:39.380871+0000","last_unstale":"2024-06-29T15:38:40.357352+0000","last_undegraded":"2024-06-29T15:38:40.357352+0000","last_fullsized":"2024-06-29T15:38:40.357352+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:17.144132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692309+0000","last_change":"2024-06-29T15:38:38.372454+0000","last_active":"2024-06-29T15:38:40.692309+0000","last_peered":"2024-06-29T15:38:40.692309+0000","last_clean":"2024-06-29T15:38:40.692309+0000","last_became_active":"2024-06-29T15:38:38.371789+0000","last_became_peered":"2024-06-29T15:38:38.371789+0000","last_unstale":"2024-06-29T15:38:40.692309+0000","last_undegraded":"2024-06-29T15:38:40.692309+0000","last_fullsized":"2024-06-29T15:38:40.692309+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:49.171782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358472+0000","last_change":"2024-06-29T15:38:39.372125+0000","last_active":"2024-06-29T15:38:40.358472+0000","last_peered":"2024-06-29T15:38:40.358472+0000","last_clean":"2024-06-29T15:38:40.358472+0000","last_became_active":"2024-06-29T15:38:39.371831+0000","last_became_peered":"2024-06-29T15:38:39.371831+0000","last_unstale":"2024-06-29T15:38:40.358472+0000","last_undegraded":"2024-06-29T15:38:40.358472+0000","last_fullsized":"2024-06-29T15:38:40.358472+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:11.499263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"167'1","reported_seq":15,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357367+0000","last_change":"2024-06-29T15:38:38.374464+0000","last_active":"2024-06-29T15:38:40.357367+0000","last_peered":"2024-06-29T15:38:40.357367+0000","last_clean":"2024-06-29T15:38:40.357367+0000","last_became_active":"2024-06-29T15:38:38.374116+0000","last_became_peered":"2024-06-29T15:38:38.374116+0000","last_unstale":"2024-06-29T15:38:40.357367+0000","last_undegraded":"2024-06-29T15:38:40.357367+0000","last_fullsized":"2024-06-29T15:38:40.357367+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:43:11.097338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518791+0000","last_change":"2024-06-29T15:38:39.365185+0000","last_active":"2024-06-29T15:38:40.518791+0000","last_peered":"2024-06-29T15:38:40.518791+0000","last_clean":"2024-06-29T15:38:40.518791+0000","last_became_active":"2024-06-29T15:38:39.364832+0000","last_became_peered":"2024-06-29T15:38:39.364832+0000","last_unstale":"2024-06-29T15:38:40.518791+0000","last_undegraded":"2024-06-29T15:38:40.518791+0000","last_fullsized":"2024-06-29T15:38:40.518791+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:30.305412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692306+0000","last_change":"2024-06-29T15:38:38.367800+0000","last_active":"2024-06-29T15:38:40.692306+0000","last_peered":"2024-06-29T15:38:40.692306+0000","last_clean":"2024-06-29T15:38:40.692306+0000","last_became_active":"2024-06-29T15:38:38.367489+0000","last_became_peered":"2024-06-29T15:38:38.367489+0000","last_unstale":"2024-06-29T15:38:40.692306+0000","last_undegraded":"2024-06-29T15:38:40.692306+0000","last_fullsized":"2024-06-29T15:38:40.692306+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:25:51.658576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693007+0000","last_change":"2024-06-29T15:38:39.372396+0000","last_active":"2024-06-29T15:38:40.693007+0000","last_peered":"2024-06-29T15:38:40.693007+0000","last_clean":"2024-06-29T15:38:40.693007+0000","last_became_active":"2024-06-29T15:38:39.372083+0000","last_became_peered":"2024-06-29T15:38:39.372083+0000","last_unstale":"2024-06-29T15:38:40.693007+0000","last_undegraded":"2024-06-29T15:38:40.693007+0000","last_fullsized":"2024-06-29T15:38:40.693007+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:03:09.182345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691802+0000","last_change":"2024-06-29T15:38:38.371666+0000","last_active":"2024-06-29T15:38:40.691802+0000","last_peered":"2024-06-29T15:38:40.691802+0000","last_clean":"2024-06-29T15:38:40.691802+0000","last_became_active":"2024-06-29T15:38:38.371347+0000","last_became_peered":"2024-06-29T15:38:38.371347+0000","last_unstale":"2024-06-29T15:38:40.691802+0000","last_undegraded":"2024-06-29T15:38:40.691802+0000","last_fullsized":"2024-06-29T15:38:40.691802+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:57.468225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691962+0000","last_change":"2024-06-29T15:38:39.373311+0000","last_active":"2024-06-29T15:38:40.691962+0000","last_peered":"2024-06-29T15:38:40.691962+0000","last_clean":"2024-06-29T15:38:40.691962+0000","last_became_active":"2024-06-29T15:38:39.372719+0000","last_became_peered":"2024-06-29T15:38:39.372719+0000","last_unstale":"2024-06-29T15:38:40.691962+0000","last_undegraded":"2024-06-29T15:38:40.691962+0000","last_fullsized":"2024-06-29T15:38:40.691962+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:26:30.305412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358075+0000","last_change":"2024-06-29T15:38:38.372354+0000","last_active":"2024-06-29T15:38:40.358075+0000","last_peered":"2024-06-29T15:38:40.358075+0000","last_clean":"2024-06-29T15:38:40.358075+0000","last_became_active":"2024-06-29T15:38:38.372054+0000","last_became_peered":"2024-06-29T15:38:38.372054+0000","last_unstale":"2024-06-29T15:38:40.358075+0000","last_undegraded":"2024-06-29T15:38:40.358075+0000","last_fullsized":"2024-06-29T15:38:40.358075+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:02.199608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692221+0000","last_change":"2024-06-29T15:38:39.368528+0000","last_active":"2024-06-29T15:38:40.692221+0000","last_peered":"2024-06-29T15:38:40.692221+0000","last_clean":"2024-06-29T15:38:40.692221+0000","last_became_active":"2024-06-29T15:38:39.368046+0000","last_became_peered":"2024-06-29T15:38:39.368046+0000","last_unstale":"2024-06-29T15:38:40.692221+0000","last_undegraded":"2024-06-29T15:38:40.692221+0000","last_fullsized":"2024-06-29T15:38:40.692221+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:21.935398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357863+0000","last_change":"2024-06-29T15:38:38.376220+0000","last_active":"2024-06-29T15:38:40.357863+0000","last_peered":"2024-06-29T15:38:40.357863+0000","last_clean":"2024-06-29T15:38:40.357863+0000","last_became_active":"2024-06-29T15:38:38.375771+0000","last_became_peered":"2024-06-29T15:38:38.375771+0000","last_unstale":"2024-06-29T15:38:40.357863+0000","last_undegraded":"2024-06-29T15:38:40.357863+0000","last_fullsized":"2024-06-29T15:38:40.357863+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:12:39.182877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691663+0000","last_change":"2024-06-29T15:38:39.373875+0000","last_active":"2024-06-29T15:38:40.691663+0000","last_peered":"2024-06-29T15:38:40.691663+0000","last_clean":"2024-06-29T15:38:40.691663+0000","last_became_active":"2024-06-29T15:38:39.372892+0000","last_became_peered":"2024-06-29T15:38:39.372892+0000","last_unstale":"2024-06-29T15:38:40.691663+0000","last_undegraded":"2024-06-29T15:38:40.691663+0000","last_fullsized":"2024-06-29T15:38:40.691663+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:34:52.641610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357303+0000","last_change":"2024-06-29T15:38:38.370774+0000","last_active":"2024-06-29T15:38:40.357303+0000","last_peered":"2024-06-29T15:38:40.357303+0000","last_clean":"2024-06-29T15:38:40.357303+0000","last_became_active":"2024-06-29T15:38:38.370393+0000","last_became_peered":"2024-06-29T15:38:38.370393+0000","last_unstale":"2024-06-29T15:38:40.357303+0000","last_undegraded":"2024-06-29T15:38:40.357303+0000","last_fullsized":"2024-06-29T15:38:40.357303+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:14.166568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693173+0000","last_change":"2024-06-29T15:38:39.384002+0000","last_active":"2024-06-29T15:38:40.693173+0000","last_peered":"2024-06-29T15:38:40.693173+0000","last_clean":"2024-06-29T15:38:40.693173+0000","last_became_active":"2024-06-29T15:38:39.383618+0000","last_became_peered":"2024-06-29T15:38:39.383618+0000","last_unstale":"2024-06-29T15:38:40.693173+0000","last_undegraded":"2024-06-29T15:38:40.693173+0000","last_fullsized":"2024-06-29T15:38:40.693173+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:45:04.155625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"167'2","reported_seq":16,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.359173+0000","last_change":"2024-06-29T15:38:38.377903+0000","last_active":"2024-06-29T15:38:40.359173+0000","last_peered":"2024-06-29T15:38:40.359173+0000","last_clean":"2024-06-29T15:38:40.359173+0000","last_became_active":"2024-06-29T15:38:38.377547+0000","last_became_peered":"2024-06-29T15:38:38.377547+0000","last_unstale":"2024-06-29T15:38:40.359173+0000","last_undegraded":"2024-06-29T15:38:40.359173+0000","last_fullsized":"2024-06-29T15:38:40.359173+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:54:10.496353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358884+0000","last_change":"2024-06-29T15:38:39.370921+0000","last_active":"2024-06-29T15:38:40.358884+0000","last_peered":"2024-06-29T15:38:40.358884+0000","last_clean":"2024-06-29T15:38:40.358884+0000","last_became_active":"2024-06-29T15:38:39.370534+0000","last_became_peered":"2024-06-29T15:38:39.370534+0000","last_unstale":"2024-06-29T15:38:40.358884+0000","last_undegraded":"2024-06-29T15:38:40.358884+0000","last_fullsized":"2024-06-29T15:38:40.358884+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:17.144132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518873+0000","last_change":"2024-06-29T15:38:38.372716+0000","last_active":"2024-06-29T15:38:40.518873+0000","last_peered":"2024-06-29T15:38:40.518873+0000","last_clean":"2024-06-29T15:38:40.518873+0000","last_became_active":"2024-06-29T15:38:38.372409+0000","last_became_peered":"2024-06-29T15:38:38.372409+0000","last_unstale":"2024-06-29T15:38:40.518873+0000","last_undegraded":"2024-06-29T15:38:40.518873+0000","last_fullsized":"2024-06-29T15:38:40.518873+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:39:34.851686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692201+0000","last_change":"2024-06-29T15:38:39.369633+0000","last_active":"2024-06-29T15:38:40.692201+0000","last_peered":"2024-06-29T15:38:40.692201+0000","last_clean":"2024-06-29T15:38:40.692201+0000","last_became_active":"2024-06-29T15:38:39.369333+0000","last_became_peered":"2024-06-29T15:38:39.369333+0000","last_unstale":"2024-06-29T15:38:40.692201+0000","last_undegraded":"2024-06-29T15:38:40.692201+0000","last_fullsized":"2024-06-29T15:38:40.692201+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:06:27.721406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.519011+0000","last_change":"2024-06-29T15:38:38.366146+0000","last_active":"2024-06-29T15:38:40.519011+0000","last_peered":"2024-06-29T15:38:40.519011+0000","last_clean":"2024-06-29T15:38:40.519011+0000","last_became_active":"2024-06-29T15:38:38.365478+0000","last_became_peered":"2024-06-29T15:38:38.365478+0000","last_unstale":"2024-06-29T15:38:40.519011+0000","last_undegraded":"2024-06-29T15:38:40.519011+0000","last_fullsized":"2024-06-29T15:38:40.519011+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:02:40.172894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518952+0000","last_change":"2024-06-29T15:38:39.371069+0000","last_active":"2024-06-29T15:38:40.518952+0000","last_peered":"2024-06-29T15:38:40.518952+0000","last_clean":"2024-06-29T15:38:40.518952+0000","last_became_active":"2024-06-29T15:38:39.370167+0000","last_became_peered":"2024-06-29T15:38:39.370167+0000","last_unstale":"2024-06-29T15:38:40.518952+0000","last_undegraded":"2024-06-29T15:38:40.518952+0000","last_fullsized":"2024-06-29T15:38:40.518952+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:24:07.838639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.518728+0000","last_change":"2024-06-29T15:38:38.365433+0000","last_active":"2024-06-29T15:38:40.518728+0000","last_peered":"2024-06-29T15:38:40.518728+0000","last_clean":"2024-06-29T15:38:40.518728+0000","last_became_active":"2024-06-29T15:38:38.365109+0000","last_became_peered":"2024-06-29T15:38:38.365109+0000","last_unstale":"2024-06-29T15:38:40.518728+0000","last_undegraded":"2024-06-29T15:38:40.518728+0000","last_fullsized":"2024-06-29T15:38:40.518728+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:47:18.411014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357070+0000","last_change":"2024-06-29T15:38:39.375287+0000","last_active":"2024-06-29T15:38:40.357070+0000","last_peered":"2024-06-29T15:38:40.357070+0000","last_clean":"2024-06-29T15:38:40.357070+0000","last_became_active":"2024-06-29T15:38:39.374964+0000","last_became_peered":"2024-06-29T15:38:39.374964+0000","last_unstale":"2024-06-29T15:38:40.357070+0000","last_undegraded":"2024-06-29T15:38:40.357070+0000","last_fullsized":"2024-06-29T15:38:40.357070+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:40:44.674713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691928+0000","last_change":"2024-06-29T15:38:38.370558+0000","last_active":"2024-06-29T15:38:40.691928+0000","last_peered":"2024-06-29T15:38:40.691928+0000","last_clean":"2024-06-29T15:38:40.691928+0000","last_became_active":"2024-06-29T15:38:38.370212+0000","last_became_peered":"2024-06-29T15:38:38.370212+0000","last_unstale":"2024-06-29T15:38:40.691928+0000","last_undegraded":"2024-06-29T15:38:40.691928+0000","last_fullsized":"2024-06-29T15:38:40.691928+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:10:59.525967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.693196+0000","last_change":"2024-06-29T15:38:39.374950+0000","last_active":"2024-06-29T15:38:40.693196+0000","last_peered":"2024-06-29T15:38:40.693196+0000","last_clean":"2024-06-29T15:38:40.693196+0000","last_became_active":"2024-06-29T15:38:39.374565+0000","last_became_peered":"2024-06-29T15:38:39.374565+0000","last_unstale":"2024-06-29T15:38:40.693196+0000","last_undegraded":"2024-06-29T15:38:40.693196+0000","last_fullsized":"2024-06-29T15:38:40.693196+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:11.499263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.358117+0000","last_change":"2024-06-29T15:38:38.377849+0000","last_active":"2024-06-29T15:38:40.358117+0000","last_peered":"2024-06-29T15:38:40.358117+0000","last_clean":"2024-06-29T15:38:40.358117+0000","last_became_active":"2024-06-29T15:38:38.377552+0000","last_became_peered":"2024-06-29T15:38:38.377552+0000","last_unstale":"2024-06-29T15:38:40.358117+0000","last_undegraded":"2024-06-29T15:38:40.358117+0000","last_fullsized":"2024-06-29T15:38:40.358117+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:01.311674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357499+0000","last_change":"2024-06-29T15:38:39.383189+0000","last_active":"2024-06-29T15:38:40.357499+0000","last_peered":"2024-06-29T15:38:40.357499+0000","last_clean":"2024-06-29T15:38:40.357499+0000","last_became_active":"2024-06-29T15:38:39.382725+0000","last_became_peered":"2024-06-29T15:38:39.382725+0000","last_unstale":"2024-06-29T15:38:40.357499+0000","last_undegraded":"2024-06-29T15:38:40.357499+0000","last_fullsized":"2024-06-29T15:38:40.357499+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:21.241394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357765+0000","last_change":"2024-06-29T15:38:38.371168+0000","last_active":"2024-06-29T15:38:40.357765+0000","last_peered":"2024-06-29T15:38:40.357765+0000","last_clean":"2024-06-29T15:38:40.357765+0000","last_became_active":"2024-06-29T15:38:38.370536+0000","last_became_peered":"2024-06-29T15:38:38.370536+0000","last_unstale":"2024-06-29T15:38:40.357765+0000","last_undegraded":"2024-06-29T15:38:40.357765+0000","last_fullsized":"2024-06-29T15:38:40.357765+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:21.086645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692502+0000","last_change":"2024-06-29T15:38:39.367619+0000","last_active":"2024-06-29T15:38:40.692502+0000","last_peered":"2024-06-29T15:38:40.692502+0000","last_clean":"2024-06-29T15:38:40.692502+0000","last_became_active":"2024-06-29T15:38:39.367078+0000","last_became_peered":"2024-06-29T15:38:39.367078+0000","last_unstale":"2024-06-29T15:38:40.692502+0000","last_undegraded":"2024-06-29T15:38:40.692502+0000","last_fullsized":"2024-06-29T15:38:40.692502+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:58.476902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":14,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.692180+0000","last_change":"2024-06-29T15:38:38.372070+0000","last_active":"2024-06-29T15:38:40.692180+0000","last_peered":"2024-06-29T15:38:40.692180+0000","last_clean":"2024-06-29T15:38:40.692180+0000","last_became_active":"2024-06-29T15:38:38.371723+0000","last_became_peered":"2024-06-29T15:38:38.371723+0000","last_unstale":"2024-06-29T15:38:40.692180+0000","last_undegraded":"2024-06-29T15:38:40.692180+0000","last_fullsized":"2024-06-29T15:38:40.692180+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:06:19.130043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":12,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.691749+0000","last_change":"2024-06-29T15:38:39.373425+0000","last_active":"2024-06-29T15:38:40.691749+0000","last_peered":"2024-06-29T15:38:40.691749+0000","last_clean":"2024-06-29T15:38:40.691749+0000","last_became_active":"2024-06-29T15:38:39.373136+0000","last_became_peered":"2024-06-29T15:38:39.373136+0000","last_unstale":"2024-06-29T15:38:40.691749+0000","last_undegraded":"2024-06-29T15:38:40.691749+0000","last_fullsized":"2024-06-29T15:38:40.691749+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":166,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:38.341733+0000","last_clean_scrub_stamp":"2024-06-29T15:38:38.341733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:56.070386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"167'3","reported_seq":17,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.359230+0000","last_change":"2024-06-29T15:38:38.369778+0000","last_active":"2024-06-29T15:38:40.359230+0000","last_peered":"2024-06-29T15:38:40.359230+0000","last_clean":"2024-06-29T15:38:40.359230+0000","last_became_active":"2024-06-29T15:38:38.369402+0000","last_became_peered":"2024-06-29T15:38:38.369402+0000","last_unstale":"2024-06-29T15:38:40.359230+0000","last_undegraded":"2024-06-29T15:38:40.359230+0000","last_fullsized":"2024-06-29T15:38:40.359230+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:38:37.333056+0000","last_clean_scrub_stamp":"2024-06-29T15:38:37.333056+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:03.888658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"157'170","reported_seq":773,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-06-29T15:38:40.357638+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:38:40.357638+0000","last_peered":"2024-06-29T15:38:40.357638+0000","last_clean":"2024-06-29T15:38:40.357638+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:38:40.357638+0000","last_undegraded":"2024-06-29T15:38:40.357638+0000","last_fullsized":"2024-06-29T15:38:40.357638+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":269,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":906,"num_read_kb":2519,"num_write":269,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575763,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60184,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311656,"statfs":{"total":96636764160,"available":96575135744,"internally_reserved":0,"allocated":2699264,"data_stored":1609724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575764,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41568,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330272,"statfs":{"total":96636764160,"available":96594198528,"internally_reserved":0,"allocated":2109440,"data_stored":1018850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575762,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41560,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330280,"statfs":{"total":96636764160,"available":96594206720,"internally_reserved":0,"allocated":2101248,"data_stored":1018726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55444,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316396,"statfs":{"total":96636764160,"available":96579989504,"internally_reserved":0,"allocated":2097152,"data_stored":1018704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59604,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312236,"statfs":{"total":96636764160,"available":96575729664,"internally_reserved":0,"allocated":2101248,"data_stored":1018794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575763,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63768,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308072,"statfs":{"total":96636764160,"available":96571465728,"internally_reserved":0,"allocated":2101248,"data_stored":1013986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575762,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59588,"kb_used_data":2040,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312252,"statfs":{"total":96636764160,"available":96575746048,"internally_reserved":0,"allocated":2088960,"data_stored":1012886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575763,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56020,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315820,"statfs":{"total":96636764160,"available":96579399680,"internally_reserved":0,"allocated":2695168,"data_stored":1604318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:38:43.769 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:38:43.769 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:38:43.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.924+0000 7f5899ace640 1 -- 172.21.15.81:0/1540599643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58940b6eb0 msgr2=0x7f58940d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.924+0000 7f5899ace640 1 --2- 172.21.15.81:0/1540599643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58940b6eb0 0x7f58940d02d0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5884009830 tx=0x7f588402a470 comp rx=0 tx=0).stop 2024-06-29T15:38:43.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.924+0000 7f5899ace640 1 -- 172.21.15.81:0/1540599643 shutdown_connections 2024-06-29T15:38:43.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.924+0000 7f5899ace640 1 --2- 172.21.15.81:0/1540599643 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58940ce920 0x7f58940cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.924+0000 7f5899ace640 1 --2- 172.21.15.81:0/1540599643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58940b6eb0 0x7f58940d02d0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.924+0000 7f5899ace640 1 -- 172.21.15.81:0/1540599643 >> 172.21.15.81:0/1540599643 conn(0x7f58940b3510 msgr2=0x7f58940b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:43.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.924+0000 7f5899ace640 1 -- 172.21.15.81:0/1540599643 shutdown_connections 2024-06-29T15:38:43.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.924+0000 7f5899ace640 1 -- 172.21.15.81:0/1540599643 wait complete. 2024-06-29T15:38:43.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.925+0000 7f5899ace640 1 Processor -- start 2024-06-29T15:38:43.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.926+0000 7f5899ace640 1 -- start start 2024-06-29T15:38:43.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f5899ace640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58940b6eb0 0x7f58941651d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f5899ace640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58940ce510 0x7f5894165710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f5899ace640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58940ce920 0x7f58941f87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f5899ace640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f58940b50a0 con 0x7f58940b6eb0 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f5899ace640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f58940b4f20 con 0x7f58940ce510 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f5899ace640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f58940b5220 con 0x7f58940ce920 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f58937fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58940b6eb0 0x7f58941651d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f58937fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58940b6eb0 0x7f58941651d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53816/0 (socket says 172.21.15.81:53816) 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f58937fe640 1 -- 172.21.15.81:0/2242174062 learned_addr learned my addr 172.21.15.81:0/2242174062 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f5893fff640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58940ce920 0x7f58941f87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f58937fe640 1 -- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58940ce920 msgr2=0x7f58941f87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f58937fe640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58940ce920 0x7f58941f87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f58937fe640 1 -- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58940ce510 msgr2=0x7f5894165710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:43.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f5892ffd640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58940ce510 0x7f5894165710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f58937fe640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58940ce510 0x7f5894165710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:43.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.927+0000 7f58937fe640 1 -- 172.21.15.81:0/2242174062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5894160d80 con 0x7f58940b6eb0 2024-06-29T15:38:43.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.928+0000 7f58937fe640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58940b6eb0 0x7f58941651d0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f5884009950 tx=0x7f5884033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.928+0000 7f588bfff640 1 -- 172.21.15.81:0/2242174062 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884031d60 con 0x7f58940b6eb0 2024-06-29T15:38:43.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.928+0000 7f588bfff640 1 -- 172.21.15.81:0/2242174062 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5884033e10 con 0x7f58940b6eb0 2024-06-29T15:38:43.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.928+0000 7f5899ace640 1 -- 172.21.15.81:0/2242174062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5894161060 con 0x7f58940b6eb0 2024-06-29T15:38:43.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.928+0000 7f588bfff640 1 -- 172.21.15.81:0/2242174062 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884005260 con 0x7f58940b6eb0 2024-06-29T15:38:43.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.928+0000 7f5899ace640 1 -- 172.21.15.81:0/2242174062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5894161620 con 0x7f58940b6eb0 2024-06-29T15:38:43.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.929+0000 7f5899ace640 1 -- 172.21.15.81:0/2242174062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58940d02d0 con 0x7f58940b6eb0 2024-06-29T15:38:43.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.930+0000 7f588bfff640 1 -- 172.21.15.81:0/2242174062 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f5884005400 con 0x7f58940b6eb0 2024-06-29T15:38:43.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.930+0000 7f588bfff640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5874076420 0x7f5874078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:43.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.930+0000 7f588bfff640 1 -- 172.21.15.81:0/2242174062 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f58840b8b40 con 0x7f58940b6eb0 2024-06-29T15:38:43.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.930+0000 7f5892ffd640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5874076420 0x7f5874078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:43.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.931+0000 7f5892ffd640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5874076420 0x7f5874078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f587c000f60 tx=0x7f587c009040 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:43.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:43.935+0000 7f588bfff640 1 -- 172.21.15.81:0/2242174062 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f588403f030 con 0x7f58940b6eb0 2024-06-29T15:38:44.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.183+0000 7f5899ace640 1 -- 172.21.15.81:0/2242174062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f58940cdf00 con 0x7f58940b6eb0 2024-06-29T15:38:44.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.187+0000 7f588bfff640 1 -- 172.21.15.81:0/2242174062 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v58) ==== 190+0+147 (secure 0 0 0) 0x7f588408a020 con 0x7f58940b6eb0 2024-06-29T15:38:44.188 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T15:38:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T15:38:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T15:38:44.188 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T15:38:44.189 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:38:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T15:38:44.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.188+0000 7f5899ace640 1 -- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5874076420 msgr2=0x7f5874078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.188+0000 7f5899ace640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5874076420 0x7f5874078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f587c000f60 tx=0x7f587c009040 comp rx=0 tx=0).stop 2024-06-29T15:38:44.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.188+0000 7f5899ace640 1 -- 172.21.15.81:0/2242174062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58940b6eb0 msgr2=0x7f58941651d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.189+0000 7f5899ace640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58940b6eb0 0x7f58941651d0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f5884009950 tx=0x7f5884033c20 comp rx=0 tx=0).stop 2024-06-29T15:38:44.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.189+0000 7f5899ace640 1 -- 172.21.15.81:0/2242174062 shutdown_connections 2024-06-29T15:38:44.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.189+0000 7f5899ace640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f5874076420 0x7f5874078930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.189+0000 7f5899ace640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f58940ce920 0x7f58941f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.189+0000 7f5899ace640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f58940ce510 0x7f5894165710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.189+0000 7f5899ace640 1 --2- 172.21.15.81:0/2242174062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58940b6eb0 0x7f58941651d0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.189+0000 7f5899ace640 1 -- 172.21.15.81:0/2242174062 >> 172.21.15.81:0/2242174062 conn(0x7f58940b3510 msgr2=0x7f58940cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:44.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.189+0000 7f5899ace640 1 -- 172.21.15.81:0/2242174062 shutdown_connections 2024-06-29T15:38:44.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.189+0000 7f5899ace640 1 -- 172.21.15.81:0/2242174062 wait complete. 2024-06-29T15:38:44.202 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:44.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.315+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/3142807270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1db80ce510 msgr2=0x7f1db80ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.315+0000 7f1dbfd2b640 1 --2- 172.21.15.81:0/3142807270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1db80ce510 0x7f1db80ce920 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f1db80b41b0 tx=0x7f1da802a470 comp rx=0 tx=0).stop 2024-06-29T15:38:44.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.315+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/3142807270 shutdown_connections 2024-06-29T15:38:44.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.315+0000 7f1dbfd2b640 1 --2- 172.21.15.81:0/3142807270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1db80ce510 0x7f1db80ce920 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.315+0000 7f1dbfd2b640 1 --2- 172.21.15.81:0/3142807270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1db80b6eb0 0x7f1db80d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.315+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/3142807270 >> 172.21.15.81:0/3142807270 conn(0x7f1db80b3510 msgr2=0x7f1db80b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:44.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.315+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/3142807270 shutdown_connections 2024-06-29T15:38:44.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.315+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/3142807270 wait complete. 2024-06-29T15:38:44.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.316+0000 7f1dbfd2b640 1 Processor -- start 2024-06-29T15:38:44.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.317+0000 7f1dbfd2b640 1 -- start start 2024-06-29T15:38:44.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbfd2b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1db80b4810 0x7f1db815ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbfd2b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1db80b6eb0 0x7f1db81620b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbfd2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1db80ce510 0x7f1db815d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbfd2b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1db80cd480 con 0x7f1db80ce510 2024-06-29T15:38:44.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbfd2b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1db80b29c0 con 0x7f1db80b6eb0 2024-06-29T15:38:44.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbfd2b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1db80cd600 con 0x7f1db80b4810 2024-06-29T15:38:44.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbe2c8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1db80ce510 0x7f1db815d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbe2c8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1db80ce510 0x7f1db815d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53832/0 (socket says 172.21.15.81:53832) 2024-06-29T15:38:44.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbe2c8640 1 -- 172.21.15.81:0/1194012795 learned_addr learned my addr 172.21.15.81:0/1194012795 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbdac7640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1db80b4810 0x7f1db815ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbd2c6640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1db80b6eb0 0x7f1db81620b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbe2c8640 1 -- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1db80b4810 msgr2=0x7f1db815ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbe2c8640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1db80b4810 0x7f1db815ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbe2c8640 1 -- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1db80b6eb0 msgr2=0x7f1db81620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbe2c8640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1db80b6eb0 0x7f1db81620b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbe2c8640 1 -- 172.21.15.81:0/1194012795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db80c8f80 con 0x7f1db80ce510 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dbe2c8640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1db80ce510 0x7f1db815d7d0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f1db000c760 tx=0x7f1db000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.318+0000 7f1dae7fc640 1 -- 172.21.15.81:0/1194012795 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db0003b70 con 0x7f1db80ce510 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.319+0000 7f1dae7fc640 1 -- 172.21.15.81:0/1194012795 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1db0003d10 con 0x7f1db80ce510 2024-06-29T15:38:44.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.319+0000 7f1dae7fc640 1 -- 172.21.15.81:0/1194012795 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db00052d0 con 0x7f1db80ce510 2024-06-29T15:38:44.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.319+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/1194012795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1db80c9260 con 0x7f1db80ce510 2024-06-29T15:38:44.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.319+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/1194012795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1db80c9820 con 0x7f1db80ce510 2024-06-29T15:38:44.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.319+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/1194012795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1db80d02d0 con 0x7f1db80ce510 2024-06-29T15:38:44.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.320+0000 7f1dae7fc640 1 -- 172.21.15.81:0/1194012795 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f1db0005470 con 0x7f1db80ce510 2024-06-29T15:38:44.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.321+0000 7f1dae7fc640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1d9c0763f0 0x7f1d9c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:44.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.321+0000 7f1dae7fc640 1 -- 172.21.15.81:0/1194012795 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f1db00975e0 con 0x7f1db80ce510 2024-06-29T15:38:44.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.321+0000 7f1dbdac7640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1d9c0763f0 0x7f1d9c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:44.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.322+0000 7f1dbdac7640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1d9c0763f0 0x7f1d9c078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1db400bb50 tx=0x7f1db400b2e0 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:44.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.326+0000 7f1dae7fc640 1 -- 172.21.15.81:0/1194012795 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1db0060770 con 0x7f1db80ce510 2024-06-29T15:38:44.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.544+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/1194012795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1db80c5250 con 0x7f1db80ce510 2024-06-29T15:38:44.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.544+0000 7f1dae7fc640 1 -- 172.21.15.81:0/1194012795 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5233 (secure 0 0 0) 0x7f1db0064a70 con 0x7f1db80ce510 2024-06-29T15:38:44.546 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:44.546 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":86,"btime":"2024-06-29T15:38:42:386002+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7754,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3261744821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3261744821},{"type":"v1","addr":"172.21.15.167:6836","nonce":3261744821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7757,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1617521107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1617521107},{"type":"v1","addr":"172.21.15.167:6837","nonce":1617521107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":9636,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3224768380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3224768380},{"type":"v1","addr":"172.21.15.81:6833","nonce":3224768380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.731697+0000","modified":"2024-06-29T15:38:41.749109+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9639":{"gid":9639,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/1634290596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1634290596},{"type":"v1","addr":"172.21.15.81:6835","nonce":1634290596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9639,"qdb_cluster":[9639]},"id":10}]} 2024-06-29T15:38:44.546 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 86 2024-06-29T15:38:44.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1d9c0763f0 msgr2=0x7f1d9c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1d9c0763f0 0x7f1d9c078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1db400bb50 tx=0x7f1db400b2e0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/1194012795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1db80ce510 msgr2=0x7f1db815d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:44.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1db80ce510 0x7f1db815d7d0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f1db000c760 tx=0x7f1db000cb60 comp rx=0 tx=0).stop 2024-06-29T15:38:44.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/1194012795 shutdown_connections 2024-06-29T15:38:44.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f1d9c0763f0 0x7f1d9c078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1db80ce510 0x7f1db815d7d0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1db80b6eb0 0x7f1db81620b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 --2- 172.21.15.81:0/1194012795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1db80b4810 0x7f1db815ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:44.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/1194012795 >> 172.21.15.81:0/1194012795 conn(0x7f1db80b3510 msgr2=0x7f1db80cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:44.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/1194012795 shutdown_connections 2024-06-29T15:38:44.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:44.546+0000 7f1dbfd2b640 1 -- 172.21.15.81:0/1194012795 wait complete. 2024-06-29T15:38:44.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:38:38.731697+0000', 'modified': '2024-06-29T15:38:41.749109+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9639}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9639': {'gid': 9639, 'name': 'c', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6835/1634290596', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1634290596}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1634290596}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9639, 'qdb_cluster': [9639]} 2024-06-29T15:38:44.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:38:44.962 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-06-29T15:38:45.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.077+0000 7f87b36a7640 1 -- 172.21.15.81:0/1913073081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87ac06f850 msgr2=0x7f87ac0ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.077+0000 7f87b36a7640 1 --2- 172.21.15.81:0/1913073081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87ac06f850 0x7f87ac0ae300 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f879c009830 tx=0x7f879c02a510 comp rx=0 tx=0).stop 2024-06-29T15:38:45.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.077+0000 7f87b36a7640 1 -- 172.21.15.81:0/1913073081 shutdown_connections 2024-06-29T15:38:45.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.077+0000 7f87b36a7640 1 --2- 172.21.15.81:0/1913073081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87ac06f850 0x7f87ac0ae300 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.077+0000 7f87b36a7640 1 -- 172.21.15.81:0/1913073081 >> 172.21.15.81:0/1913073081 conn(0x7f87ac072950 msgr2=0x7f87ac0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.077+0000 7f87b36a7640 1 -- 172.21.15.81:0/1913073081 shutdown_connections 2024-06-29T15:38:45.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.077+0000 7f87b36a7640 1 -- 172.21.15.81:0/1913073081 wait complete. 2024-06-29T15:38:45.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.079+0000 7f87b36a7640 1 Processor -- start 2024-06-29T15:38:45.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.079+0000 7f87b36a7640 1 -- start start 2024-06-29T15:38:45.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b36a7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87ac06f850 0x7f87ac09d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b36a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87ac074520 0x7f87ac14e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b36a7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87ac0ae840 0x7f87ac14ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b36a7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac15b190 con 0x7f87ac074520 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b36a7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac15b010 con 0x7f87ac06f850 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b36a7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f87ac0af360 con 0x7f87ac0ae840 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b0c42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87ac074520 0x7f87ac14e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b0c42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87ac074520 0x7f87ac14e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53858/0 (socket says 172.21.15.81:53858) 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b0c42640 1 -- 172.21.15.81:0/572138103 learned_addr learned my addr 172.21.15.81:0/572138103 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b0c42640 1 -- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87ac0ae840 msgr2=0x7f87ac14ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b0c42640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87ac0ae840 0x7f87ac14ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b0c42640 1 -- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87ac06f850 msgr2=0x7f87ac09d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b0c42640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87ac06f850 0x7f87ac09d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b0c42640 1 -- 172.21.15.81:0/572138103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ac154dd0 con 0x7f87ac074520 2024-06-29T15:38:45.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.080+0000 7f87b0c42640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87ac074520 0x7f87ac14e4b0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f879c02ab00 tx=0x7f879c033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.081+0000 7f87a1ffb640 1 -- 172.21.15.81:0/572138103 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c031d60 con 0x7f87ac074520 2024-06-29T15:38:45.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.081+0000 7f87a1ffb640 1 -- 172.21.15.81:0/572138103 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f879c005020 con 0x7f87ac074520 2024-06-29T15:38:45.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.081+0000 7f87b36a7640 1 -- 172.21.15.81:0/572138103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87ac155090 con 0x7f87ac074520 2024-06-29T15:38:45.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.081+0000 7f87a1ffb640 1 -- 172.21.15.81:0/572138103 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c005470 con 0x7f87ac074520 2024-06-29T15:38:45.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.081+0000 7f87b36a7640 1 -- 172.21.15.81:0/572138103 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87ac155650 con 0x7f87ac074520 2024-06-29T15:38:45.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.083+0000 7f87a1ffb640 1 -- 172.21.15.81:0/572138103 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f879c0047a0 con 0x7f87ac074520 2024-06-29T15:38:45.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.083+0000 7f87a1ffb640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8790076420 0x7f8790078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.084+0000 7f87a1ffb640 1 -- 172.21.15.81:0/572138103 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f879c0b8bf0 con 0x7f87ac074520 2024-06-29T15:38:45.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.084+0000 7f87b1443640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8790076420 0x7f8790078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.084+0000 7f87b1443640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8790076420 0x7f8790078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009650 tx=0x7f87a8006ba0 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.084+0000 7f87ba1fd740 1 Processor -- start 2024-06-29T15:38:45.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.084+0000 7f87ba1fd740 1 -- start start 2024-06-29T15:38:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aae16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6cc3b0 0x55e67aac53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6d4590 0x55e67aac58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e67aa5dbc0 con 0x55e67a6d4590 2024-06-29T15:38:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55e67aa0c630 con 0x55e67a6cc3b0 2024-06-29T15:38:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55e67a9fbbd0 con 0x55e67a6f5270 2024-06-29T15:38:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b1c44640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6d4590 0x55e67aac58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b1c44640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6d4590 0x55e67aac58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53860/0 (socket says 172.21.15.81:53860) 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b1c44640 1 -- 172.21.15.81:0/3407906303 learned_addr learned my addr 172.21.15.81:0/3407906303 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b0c42640 1 --2- 172.21.15.81:0/3407906303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6cc3b0 0x55e67aac53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b1443640 1 --2- 172.21.15.81:0/3407906303 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aae16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b1c44640 1 -- 172.21.15.81:0/3407906303 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 msgr2=0x55e67aae16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b1c44640 1 --2- 172.21.15.81:0/3407906303 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aae16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b1c44640 1 -- 172.21.15.81:0/3407906303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6cc3b0 msgr2=0x55e67aac53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b1c44640 1 --2- 172.21.15.81:0/3407906303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6cc3b0 0x55e67aac53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b1c44640 1 -- 172.21.15.81:0/3407906303 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e67a9f8ca0 con 0x55e67a6d4590 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b0c42640 1 --2- 172.21.15.81:0/3407906303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6cc3b0 0x55e67aac53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f87b1c44640 1 --2- 172.21.15.81:0/3407906303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6d4590 0x55e67aac58e0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f87a4002090 tx=0x7f87a400c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7229879c40680616 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f8796ffd640 1 -- 172.21.15.81:0/3407906303 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a4012620 con 0x55e67a6d4590 2024-06-29T15:38:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f8796ffd640 1 -- 172.21.15.81:0/3407906303 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87a4017070 con 0x55e67a6d4590 2024-06-29T15:38:45.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.086+0000 7f8796ffd640 1 -- 172.21.15.81:0/3407906303 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a4012a80 con 0x55e67a6d4590 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.087+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3407906303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6d4590 msgr2=0x55e67aac58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.087+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3407906303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6d4590 0x55e67aac58e0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f87a4002090 tx=0x7f87a400c4b0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.087+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3407906303 shutdown_connections 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.087+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3407906303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6d4590 0x55e67aac58e0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.087+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3407906303 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6cc3b0 0x55e67aac53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.087+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3407906303 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aae16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.087+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3407906303 >> 172.21.15.81:0/3407906303 conn(0x55e67a6e93c0 msgr2=0x55e67aa98f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.087+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3407906303 shutdown_connections 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.087+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3407906303 wait complete. 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.088+0000 7f87ba1fd740 1 -- start start 2024-06-29T15:38:45.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.088+0000 7f87ba1fd740 1 Processor -- start 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aad78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6d4590 0x55e67aabd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aaf3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e67a999ae0 con 0x55e67a6cc3b0 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55e67a9ffc10 con 0x55e67a6d4590 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55e67a76f620 con 0x55e67a6f5270 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87ba1fd740 10 client.0 _ll_register_callbacks cb 0x55e67a9a6480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b1443640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aad78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b1443640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aad78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53870/0 (socket says 172.21.15.81:53870) 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b1443640 1 -- 172.21.15.81:0/3387846346 learned_addr learned my addr 172.21.15.81:0/3387846346 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:45.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b1c44640 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aaf3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b0c42640 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6d4590 0x55e67aabd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b1443640 1 -- 172.21.15.81:0/3387846346 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 msgr2=0x55e67aaf3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b1443640 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aaf3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b1443640 1 -- 172.21.15.81:0/3387846346 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6d4590 msgr2=0x55e67aabd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b1443640 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6d4590 0x55e67aabd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b1443640 1 -- 172.21.15.81:0/3387846346 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e67a760850 con 0x55e67a6cc3b0 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87b1443640 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aad78b0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f87a800b690 tx=0x7f87a80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6492fdb84df857 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f878bfff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f878bfff640 1 -- 172.21.15.81:0/3387846346 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8023020 con 0x55e67a6cc3b0 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f878bfff640 1 -- 172.21.15.81:0/3387846346 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f87a801f850 con 0x55e67a6cc3b0 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.090+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3387846346 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87a4017070 con 0x55e67a6cc3b0 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f878bfff640 1 -- 172.21.15.81:0/3387846346 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f87a802d050 con 0x55e67a6cc3b0 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f87ba1fd740 10 client.9786 fetch_fsmap learned FSMap version 86 2024-06-29T15:38:45.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3387846346 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) -- 0x55e67a7d76b0 con 0x55e67a6cc3b0 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f878bfff640 1 -- 172.21.15.81:0/3387846346 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87a4017070 con 0x55e67a6cc3b0 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f878bfff640 1 -- 172.21.15.81:0/3387846346 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 86) ==== 3502+0+0 (secure 0 0 0) 0x7f87a8030020 con 0x55e67a6cc3b0 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f87ba1fd740 10 client.9786 fetch_fsmap finished waiting for FSMap version 86 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f87ba1fd740 10 client.9786 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f87ba1fd740 20 client.9786 populate_metadata read hostname 'smithi081' 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.81:6834/1634290596,v1:172.21.15.81:6835/1634290596] conn(0x55e67a6b3e40 0x55e67aafd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f87ba1fd740 4 client.9786 mds_command: new command op to 9639 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3387846346 --> [v2:172.21.15.81:6834/1634290596,v1:172.21.15.81:6835/1634290596] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e67aa0e580 con 0x55e67a6b3e40 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f87b0c42640 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.81:6834/1634290596,v1:172.21.15.81:6835/1634290596] conn(0x55e67a6b3e40 0x55e67aafd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f878bfff640 10 client.9786 ms_handle_connect on v2:172.21.15.81:6834/1634290596 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.091+0000 7f87b0c42640 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.81:6834/1634290596,v1:172.21.15.81:6835/1634290596] conn(0x55e67a6b3e40 0x55e67aafd510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68856587f985a4c5 server_cookie=ed4f2fe09e677df3 in_seq=0 out_seq=0 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f878bfff640 1 -- 172.21.15.81:0/3387846346 <== mds.0 v2:172.21.15.81:6834/1634290596 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55e67aa0e580 con 0x55e67a6b3e40 2024-06-29T15:38:45.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f878bfff640 10 client.9786 handle_command_reply: tid=0 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 client.9786 shutdown 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3387846346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 msgr2=0x55e67aad78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aad78b0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f87a800b690 tx=0x7f87a80046b0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3387846346 shutdown_connections 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.81:6834/1634290596,v1:172.21.15.81:6835/1634290596] conn(0x55e67a6b3e40 0x55e67aafd510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aaf3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6d4590 0x55e67aabd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3387846346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aad78b0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3387846346 >> 172.21.15.81:0/3387846346 conn(0x55e67a6e93c0 msgr2=0x55e67aac1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f878bfff640 0 client.9786 ms_handle_reset on v2:172.21.15.81:6834/1634290596 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3387846346 shutdown_connections 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.092+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3387846346 wait complete. 2024-06-29T15:38:45.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.093+0000 7f87ba1fd740 20 client.9786 trim_cache size 0 max 16384 2024-06-29T15:38:45.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.111+0000 7f87ba1fd740 1 Processor -- start 2024-06-29T15:38:45.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.111+0000 7f87ba1fd740 1 -- start start 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aa99060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6d4590 0x55e67aa9aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6f5270 0x55e67aa9b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac200bd0 con 0x55e67a6cc3b0 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac0d0df0 con 0x55e67a6f5270 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f87ac0cd2f0 con 0x55e67a6d4590 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87b1443640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aa99060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87b1443640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aa99060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53886/0 (socket says 172.21.15.81:53886) 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87b1443640 1 -- 172.21.15.81:0/1609178545 learned_addr learned my addr 172.21.15.81:0/1609178545 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87b1443640 1 -- 172.21.15.81:0/1609178545 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6d4590 msgr2=0x55e67aa9aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87b1443640 1 --2- 172.21.15.81:0/1609178545 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6d4590 0x55e67aa9aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87b1443640 1 -- 172.21.15.81:0/1609178545 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6f5270 msgr2=0x55e67aa9b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87b1443640 1 --2- 172.21.15.81:0/1609178545 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6f5270 0x55e67aa9b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87b1443640 1 -- 172.21.15.81:0/1609178545 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e67aac5dd0 con 0x55e67a6cc3b0 2024-06-29T15:38:45.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87b1443640 1 --2- 172.21.15.81:0/1609178545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aa99060 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f87a80136b0 tx=0x7f87a801fdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d27f5d5a05b5a7c server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87967fc640 1 -- 172.21.15.81:0/1609178545 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8036070 con 0x55e67a6cc3b0 2024-06-29T15:38:45.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87967fc640 1 -- 172.21.15.81:0/1609178545 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87a8005940 con 0x55e67a6cc3b0 2024-06-29T15:38:45.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.113+0000 7f87967fc640 1 -- 172.21.15.81:0/1609178545 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8023070 con 0x55e67a6cc3b0 2024-06-29T15:38:45.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.114+0000 7f87ba1fd740 1 -- 172.21.15.81:0/1609178545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 msgr2=0x55e67aa99060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.114+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/1609178545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aa99060 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f87a80136b0 tx=0x7f87a801fdb0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.114+0000 7f87ba1fd740 1 -- 172.21.15.81:0/1609178545 shutdown_connections 2024-06-29T15:38:45.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.114+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/1609178545 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6f5270 0x55e67aa9b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.114+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/1609178545 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6d4590 0x55e67aa9aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.114+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/1609178545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aa99060 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.114+0000 7f87ba1fd740 1 -- 172.21.15.81:0/1609178545 >> 172.21.15.81:0/1609178545 conn(0x55e67a6e93c0 msgr2=0x55e67aac41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.114+0000 7f87ba1fd740 1 -- 172.21.15.81:0/1609178545 shutdown_connections 2024-06-29T15:38:45.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.114+0000 7f87ba1fd740 1 -- 172.21.15.81:0/1609178545 wait complete. 2024-06-29T15:38:45.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.115+0000 7f87ba1fd740 1 -- start start 2024-06-29T15:38:45.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.115+0000 7f87ba1fd740 1 Processor -- start 2024-06-29T15:38:45.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.116+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aaf8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.116+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6d4590 0x55e67aaf9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.116+0000 7f87ba1fd740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aa99b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.116+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e67aad81d0 con 0x55e67a6cc3b0 2024-06-29T15:38:45.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.116+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55e67aaf3ff0 con 0x55e67a6d4590 2024-06-29T15:38:45.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.116+0000 7f87ba1fd740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55e67aac3410 con 0x55e67a6f5270 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.116+0000 7f87ba1fd740 10 client.0 _ll_register_callbacks cb 0x7f87a8013510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.116+0000 7f87b1443640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aaf8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.116+0000 7f87b1443640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aaf8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53898/0 (socket says 172.21.15.81:53898) 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.116+0000 7f87b1443640 1 -- 172.21.15.81:0/3644803335 learned_addr learned my addr 172.21.15.81:0/3644803335 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f87b1443640 1 -- 172.21.15.81:0/3644803335 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 msgr2=0x55e67aa99b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f87b1443640 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aa99b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f87b1443640 1 -- 172.21.15.81:0/3644803335 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6d4590 msgr2=0x55e67aaf9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f87b1443640 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6d4590 0x55e67aaf9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f87b1443640 1 -- 172.21.15.81:0/3644803335 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e67aa91520 con 0x55e67a6cc3b0 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f87b1443640 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aaf8bd0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009430 tx=0x7f87a8028b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfebd5b9488d19e3 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f878bfff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f878bfff640 1 -- 172.21.15.81:0/3644803335 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8036070 con 0x55e67a6cc3b0 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f878bfff640 1 -- 172.21.15.81:0/3644803335 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f87a803db60 con 0x55e67a6cc3b0 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3644803335 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87a8005940 con 0x55e67a6cc3b0 2024-06-29T15:38:45.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f878bfff640 1 -- 172.21.15.81:0/3644803335 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f87a8023040 con 0x55e67a6cc3b0 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f87ba1fd740 10 client.9792 fetch_fsmap learned FSMap version 86 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3644803335 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) -- 0x55e67aa9b7d0 con 0x55e67a6cc3b0 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.117+0000 7f878bfff640 1 -- 172.21.15.81:0/3644803335 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87a8005940 con 0x55e67a6cc3b0 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f878bfff640 1 -- 172.21.15.81:0/3644803335 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 86) ==== 3502+0+0 (secure 0 0 0) 0x7f87a8030030 con 0x55e67a6cc3b0 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 10 client.9792 fetch_fsmap finished waiting for FSMap version 86 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 10 client.9792 resolve_mds: resolved name 'c' to daemon mds.c 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 20 client.9792 populate_metadata read hostname 'smithi081' 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.81:6834/1634290596,v1:172.21.15.81:6835/1634290596] conn(0x55e67aa9a4b0 0x55e67aa9a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 4 client.9792 mds_command: new command op to 9639 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3644803335 --> [v2:172.21.15.81:6834/1634290596,v1:172.21.15.81:6835/1634290596] -- command(tid 0: {"prefix": "status"}) -- 0x55e67aa0a020 con 0x55e67aa9a4b0 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87b0c42640 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.81:6834/1634290596,v1:172.21.15.81:6835/1634290596] conn(0x55e67aa9a4b0 0x55e67aa9a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f878bfff640 10 client.9792 ms_handle_connect on v2:172.21.15.81:6834/1634290596 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87b0c42640 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.81:6834/1634290596,v1:172.21.15.81:6835/1634290596] conn(0x55e67aa9a4b0 0x55e67aa9a8c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=208588afbb8bf5e3 server_cookie=8d080d8a634425be in_seq=0 out_seq=0 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f878bfff640 1 -- 172.21.15.81:0/3644803335 <== mds.0 v2:172.21.15.81:6834/1634290596 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55e67aa0a020 con 0x55e67aa9a4b0 2024-06-29T15:38:45.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f878bfff640 10 client.9792 handle_command_reply: tid=0 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 client.9792 shutdown 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3644803335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 msgr2=0x55e67aaf8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aaf8bd0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009430 tx=0x7f87a8028b70 comp rx=0 tx=0).stop 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3644803335 shutdown_connections 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.81:6834/1634290596,v1:172.21.15.81:6835/1634290596] conn(0x55e67aa9a4b0 0x55e67aa9a8c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55e67a6f5270 0x55e67aa99b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f878bfff640 0 client.9792 ms_handle_reset on v2:172.21.15.81:6834/1634290596 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55e67a6d4590 0x55e67aaf9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 --2- 172.21.15.81:0/3644803335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e67a6cc3b0 0x55e67aaf8bd0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.118+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3644803335 >> 172.21.15.81:0/3644803335 conn(0x55e67a6e93c0 msgr2=0x55e67aadaae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3644803335 shutdown_connections 2024-06-29T15:38:45.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87ba1fd740 1 -- 172.21.15.81:0/3644803335 wait complete. 2024-06-29T15:38:45.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87ba1fd740 20 client.9792 trim_cache size 0 max 16384 2024-06-29T15:38:45.126 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:38:45.126 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T15:38:45.126 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T15:38:45.126 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 9639, 2024-06-29T15:38:45.126 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T15:38:45.126 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T15:38:45.126 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T15:38:45.127 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7538908280000003, 2024-06-29T15:38:45.127 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 86, 2024-06-29T15:38:45.127 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 168, 2024-06-29T15:38:45.127 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 168, 2024-06-29T15:38:45.127 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.43726038 2024-06-29T15:38:45.127 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 -- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8790076420 msgr2=0x7f8790078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8790076420 0x7f8790078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f87a8009650 tx=0x7f87a8006ba0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 -- 172.21.15.81:0/572138103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87ac074520 msgr2=0x7f87ac14e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87ac074520 0x7f87ac14e4b0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f879c02ab00 tx=0x7f879c033ce0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 -- 172.21.15.81:0/572138103 shutdown_connections 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f8790076420 0x7f8790078930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87ac0ae840 0x7f87ac14ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87ac074520 0x7f87ac14e4b0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 --2- 172.21.15.81:0/572138103 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f87ac06f850 0x7f87ac09d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 -- 172.21.15.81:0/572138103 >> 172.21.15.81:0/572138103 conn(0x7f87ac072950 msgr2=0x7f87ac156f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 -- 172.21.15.81:0/572138103 shutdown_connections 2024-06-29T15:38:45.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.119+0000 7f87b36a7640 1 -- 172.21.15.81:0/572138103 wait complete. 2024-06-29T15:38:45.132 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:38:45.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.243+0000 7fd16fba1640 1 -- 172.21.15.81:0/4192612086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1680b57b0 msgr2=0x7fd16815aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.243+0000 7fd16fba1640 1 --2- 172.21.15.81:0/4192612086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1680b57b0 0x7fd16815aed0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fd164009830 tx=0x7fd16402a510 comp rx=0 tx=0).stop 2024-06-29T15:38:45.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.243+0000 7fd16fba1640 1 -- 172.21.15.81:0/4192612086 shutdown_connections 2024-06-29T15:38:45.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.243+0000 7fd16fba1640 1 --2- 172.21.15.81:0/4192612086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1680b57b0 0x7fd16815aed0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.243+0000 7fd16fba1640 1 -- 172.21.15.81:0/4192612086 >> 172.21.15.81:0/4192612086 conn(0x7fd1680d0930 msgr2=0x7fd1680d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.243+0000 7fd16fba1640 1 -- 172.21.15.81:0/4192612086 shutdown_connections 2024-06-29T15:38:45.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.243+0000 7fd16fba1640 1 -- 172.21.15.81:0/4192612086 wait complete. 2024-06-29T15:38:45.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.245+0000 7fd16fba1640 1 Processor -- start 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.245+0000 7fd16fba1640 1 -- start start 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.246+0000 7fd16fba1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16806ef90 0x7fd168151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.246+0000 7fd16fba1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1680b3960 0x7fd168150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.246+0000 7fd16fba1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1680b57b0 0x7fd1680c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.246+0000 7fd16fba1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1680b4600 con 0x7fd1680b57b0 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.246+0000 7fd16fba1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd1680b4480 con 0x7fd1680b3960 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.246+0000 7fd16fba1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd1680b4780 con 0x7fd16806ef90 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16e13e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1680b57b0 0x7fd1680c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16e13e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1680b57b0 0x7fd1680c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53924/0 (socket says 172.21.15.81:53924) 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16e13e640 1 -- 172.21.15.81:0/1699911769 learned_addr learned my addr 172.21.15.81:0/1699911769 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16d13c640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1680b3960 0x7fd168150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16d93d640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16806ef90 0x7fd168151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16e13e640 1 -- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16806ef90 msgr2=0x7fd168151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16e13e640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16806ef90 0x7fd168151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16e13e640 1 -- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1680b3960 msgr2=0x7fd168150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16e13e640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1680b3960 0x7fd168150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16e13e640 1 -- 172.21.15.81:0/1699911769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd168150970 con 0x7fd1680b57b0 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16e13e640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1680b57b0 0x7fd1680c5fc0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fd16000c730 tx=0x7fd16000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd15e7fc640 1 -- 172.21.15.81:0/1699911769 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd160003b70 con 0x7fd1680b57b0 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd15e7fc640 1 -- 172.21.15.81:0/1699911769 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd160003d30 con 0x7fd1680b57b0 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd15e7fc640 1 -- 172.21.15.81:0/1699911769 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1600052d0 con 0x7fd1680b57b0 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16fba1640 1 -- 172.21.15.81:0/1699911769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1680c6810 con 0x7fd1680b57b0 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.247+0000 7fd16fba1640 1 -- 172.21.15.81:0/1699911769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd168151200 con 0x7fd1680b57b0 2024-06-29T15:38:45.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.248+0000 7fd16fba1640 1 -- 172.21.15.81:0/1699911769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16806e2a0 con 0x7fd1680b57b0 2024-06-29T15:38:45.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.249+0000 7fd15e7fc640 1 -- 172.21.15.81:0/1699911769 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7fd160005680 con 0x7fd1680b57b0 2024-06-29T15:38:45.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.249+0000 7fd15e7fc640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd14c0763f0 0x7fd14c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:45.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.250+0000 7fd15e7fc640 1 -- 172.21.15.81:0/1699911769 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7fd160097930 con 0x7fd1680b57b0 2024-06-29T15:38:45.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.250+0000 7fd16d93d640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd14c0763f0 0x7fd14c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:45.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.250+0000 7fd16d93d640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd14c0763f0 0x7fd14c078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd164034000 tx=0x7fd164004730 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:45.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.254+0000 7fd15e7fc640 1 -- 172.21.15.81:0/1699911769 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd16806e2a0 con 0x7fd1680b57b0 2024-06-29T15:38:45.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.476+0000 7fd16fba1640 1 -- 172.21.15.81:0/1699911769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd16815aed0 con 0x7fd1680b57b0 2024-06-29T15:38:45.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.476+0000 7fd15e7fc640 1 -- 172.21.15.81:0/1699911769 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v86) ==== 53+0+73 (secure 0 0 0) 0x7fd160061110 con 0x7fd1680b57b0 2024-06-29T15:38:45.477 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:38:45.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 -- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd14c0763f0 msgr2=0x7fd14c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd14c0763f0 0x7fd14c078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd164034000 tx=0x7fd164004730 comp rx=0 tx=0).stop 2024-06-29T15:38:45.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 -- 172.21.15.81:0/1699911769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1680b57b0 msgr2=0x7fd1680c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1680b57b0 0x7fd1680c5fc0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fd16000c730 tx=0x7fd16000cbf0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 -- 172.21.15.81:0/1699911769 shutdown_connections 2024-06-29T15:38:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fd14c0763f0 0x7fd14c078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1680b57b0 0x7fd1680c5fc0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1680b3960 0x7fd168150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 --2- 172.21.15.81:0/1699911769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16806ef90 0x7fd168151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 -- 172.21.15.81:0/1699911769 >> 172.21.15.81:0/1699911769 conn(0x7fd1680d0930 msgr2=0x7fd1680cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 -- 172.21.15.81:0/1699911769 shutdown_connections 2024-06-29T15:38:45.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45.479+0000 7fd16fba1640 1 -- 172.21.15.81:0/1699911769 wait complete. 2024-06-29T15:38:45.491 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:38:45.491 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:38:45.491 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T15:38:45.491 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:38:45.491 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:45.491 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:38:45.491 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:38:45.491 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:38:45.491 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:38:45.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.492 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T15:38:45.507 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:38:45.507 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:38:45.507 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:38:45.507 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:38:45.508 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 38194sec preferred_lft 38194sec 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:38:45.509 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T15:38:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.510 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:38:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:38:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T15:38:45.510 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:38:45.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.510 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:38:45.571 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T15:38:45.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.573 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T15:38:45.630 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T15:38:45.630 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:38:45.630 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T15:38:45.630 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T15:38:45.630 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:38:45.630 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:38:45.631 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T15:38:45.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T15:38:45.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T15:38:45.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T15:38:45.631 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T15:38:45.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:38:45.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:38:45.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.756 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T15:38:45.771 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:38:45.771 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:38:45.772 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:38:45.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.772 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T15:38:45.836 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:38:45.836 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:45.891 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:38:45.892 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:38:45.892 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:45.947 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T15:38:45.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:45.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:38:46.040 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:38:46.041 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:38:46.049 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:38:46.049 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:38:46.049 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T15:38:46.049 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T15:38:46.049 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T15:38:46.050 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:38:46.050 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:38:46.050 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:38:46.050 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:38:46.050 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:38:46.052 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:38:46.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:46.052 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:46.077 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:38:46.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:38:46.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:38:46.179 INFO:teuthology.orchestra.run.smithi081.stdout:client7835 2024-06-29T15:38:46.179 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:38:46.179 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:46.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:38:46.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.349+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/844618292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50005150 msgr2=0x7f6f5000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.349+0000 7f6f6c4f6640 1 --2- 172.21.15.81:0/844618292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50005150 0x7f6f5000b9c0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c00c610 tx=0x7f6f50005830 comp rx=0 tx=0).stop 2024-06-29T15:38:46.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.349+0000 7f6f6aa93640 1 -- 172.21.15.81:0/844618292 reap_dead start 2024-06-29T15:38:46.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.349+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/844618292 shutdown_connections 2024-06-29T15:38:46.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.349+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/844618292 >> 172.21.15.81:0/844618292 conn(0x7f6f6409cf30 msgr2=0x7f6f640d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:46.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.349+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/844618292 shutdown_connections 2024-06-29T15:38:46.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.349+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/844618292 wait complete. 2024-06-29T15:38:46.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.351+0000 7f6f6c4f6640 1 Processor -- start 2024-06-29T15:38:46.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.351+0000 7f6f6c4f6640 1 -- start start 2024-06-29T15:38:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6c4f6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50005150 0x7f6f6415ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6c4f6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f6415d610 0x7f6f640c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6c4f6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f640b2090 0x7f6f640b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6c4f6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f640b8420 con 0x7f6f50005150 2024-06-29T15:38:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6c4f6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6f640b82a0 con 0x7f6f6415d610 2024-06-29T15:38:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6c4f6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6f640b85a0 con 0x7f6f640b2090 2024-06-29T15:38:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6a292640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50005150 0x7f6f6415ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6a292640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50005150 0x7f6f6415ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53944/0 (socket says 172.21.15.81:53944) 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6a292640 1 -- 172.21.15.81:0/3941941315 learned_addr learned my addr 172.21.15.81:0/3941941315 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f69a91640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f6415d610 0x7f6f640c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6aa93640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f640b2090 0x7f6f640b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6a292640 1 -- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f640b2090 msgr2=0x7f6f640b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6a292640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f640b2090 0x7f6f640b24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6a292640 1 -- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f6415d610 msgr2=0x7f6f640c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6a292640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f6415d610 0x7f6f640c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.352+0000 7f6f6a292640 1 -- 172.21.15.81:0/3941941315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f64160a10 con 0x7f6f50005150 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.353+0000 7f6f6a292640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50005150 0x7f6f6415ba20 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f6f60003ec0 tx=0x7f6f60001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.353+0000 7f6f5affd640 1 -- 172.21.15.81:0/3941941315 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60015070 con 0x7f6f50005150 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.353+0000 7f6f5affd640 1 -- 172.21.15.81:0/3941941315 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f60011860 con 0x7f6f50005150 2024-06-29T15:38:46.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.353+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/3941941315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f64160f10 con 0x7f6f50005150 2024-06-29T15:38:46.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.353+0000 7f6f5affd640 1 -- 172.21.15.81:0/3941941315 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60011cb0 con 0x7f6f50005150 2024-06-29T15:38:46.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.353+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/3941941315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f640b15b0 con 0x7f6f50005150 2024-06-29T15:38:46.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.354+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/3941941315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f640d0480 con 0x7f6f50005150 2024-06-29T15:38:46.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.354+0000 7f6f5affd640 1 -- 172.21.15.81:0/3941941315 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f6f60018070 con 0x7f6f50005150 2024-06-29T15:38:46.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.355+0000 7f6f5affd640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6f48076420 0x7f6f48078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.355+0000 7f6f5affd640 1 -- 172.21.15.81:0/3941941315 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f6f60098670 con 0x7f6f50005150 2024-06-29T15:38:46.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.355+0000 7f6f69a91640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6f48076420 0x7f6f48078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.356+0000 7f6f69a91640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6f48076420 0x7f6f48078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f640b4930 tx=0x7f6f54005f90 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:46.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.359+0000 7f6f5affd640 1 -- 172.21.15.81:0/3941941315 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6f60061cc0 con 0x7f6f50005150 2024-06-29T15:38:46.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.564+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/3941941315 --> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f6f64069230 con 0x7f6f48076420 2024-06-29T15:38:46.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.566+0000 7f6f5affd640 1 -- 172.21.15.81:0/3941941315 <== mgr.9414 v2:172.21.15.167:6832/150801987 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f6f64069230 con 0x7f6f48076420 2024-06-29T15:38:46.567 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:38:46.567 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:38:46.567 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T15:38:46.567 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:38:46.567 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:38:46.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.568+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6f48076420 msgr2=0x7f6f48078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.568+0000 7f6f6c4f6640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6f48076420 0x7f6f48078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f640b4930 tx=0x7f6f54005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:46.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.568+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/3941941315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50005150 msgr2=0x7f6f6415ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.568+0000 7f6f6c4f6640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50005150 0x7f6f6415ba20 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f6f60003ec0 tx=0x7f6f60001710 comp rx=0 tx=0).stop 2024-06-29T15:38:46.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.569+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/3941941315 shutdown_connections 2024-06-29T15:38:46.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.569+0000 7f6f6c4f6640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f6f48076420 0x7f6f48078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.569+0000 7f6f6c4f6640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f640b2090 0x7f6f640b24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.569+0000 7f6f6c4f6640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f6415d610 0x7f6f640c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.569+0000 7f6f6c4f6640 1 --2- 172.21.15.81:0/3941941315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50005150 0x7f6f6415ba20 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.569+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/3941941315 >> 172.21.15.81:0/3941941315 conn(0x7f6f6409cf30 msgr2=0x7f6f6415dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:46.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.569+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/3941941315 shutdown_connections 2024-06-29T15:38:46.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.569+0000 7f6f6c4f6640 1 -- 172.21.15.81:0/3941941315 wait complete. 2024-06-29T15:38:46.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:38:46.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.698+0000 7f9f34b35640 1 -- 172.21.15.81:0/2075391148 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f3006f9a0 msgr2=0x7f9f3006fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.698+0000 7f9f34b35640 1 --2- 172.21.15.81:0/2075391148 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f3006f9a0 0x7f9f3006fdb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9f28005fd0 tx=0x7f9f2802a5d0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.698+0000 7f9f34b35640 1 -- 172.21.15.81:0/2075391148 shutdown_connections 2024-06-29T15:38:46.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.698+0000 7f9f34b35640 1 --2- 172.21.15.81:0/2075391148 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f3006f9a0 0x7f9f3006fdb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.698+0000 7f9f34b35640 1 --2- 172.21.15.81:0/2075391148 >> v2:172.21.15.167:3300/0 conn(0x7f9f3006c2f0 0x7f9f3006c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.698+0000 7f9f34b35640 1 -- 172.21.15.81:0/2075391148 >> 172.21.15.81:0/2075391148 conn(0x7f9f300b2920 msgr2=0x7f9f300b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:46.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.698+0000 7f9f34b35640 1 -- 172.21.15.81:0/2075391148 shutdown_connections 2024-06-29T15:38:46.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.698+0000 7f9f34b35640 1 -- 172.21.15.81:0/2075391148 wait complete. 2024-06-29T15:38:46.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.700+0000 7f9f34b35640 1 Processor -- start 2024-06-29T15:38:46.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.700+0000 7f9f34b35640 1 -- start start 2024-06-29T15:38:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f34b35640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f3006c2f0 0x7f9f300ab400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f34b35640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f3006f9a0 0x7f9f30158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f34b35640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f30159bd0 0x7f9f300b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f34b35640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3015fc10 con 0x7f9f3006c2f0 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f34b35640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3015fa90 con 0x7f9f3006f9a0 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f34b35640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9f3015fd90 con 0x7f9f30159bd0 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f2e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f3006c2f0 0x7f9f300ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f2e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f3006c2f0 0x7f9f300ab400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53954/0 (socket says 172.21.15.81:53954) 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f2e59c640 1 -- 172.21.15.81:0/4104846118 learned_addr learned my addr 172.21.15.81:0/4104846118 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f2dd9b640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f3006f9a0 0x7f9f30158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f2ed9d640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f30159bd0 0x7f9f300b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f2e59c640 1 -- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f30159bd0 msgr2=0x7f9f300b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f2e59c640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f30159bd0 0x7f9f300b0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f2e59c640 1 -- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f3006f9a0 msgr2=0x7f9f30158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:46.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.701+0000 7f9f2e59c640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f3006f9a0 0x7f9f30158690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:46.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.702+0000 7f9f2e59c640 1 -- 172.21.15.81:0/4104846118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f300ab940 con 0x7f9f3006c2f0 2024-06-29T15:38:46.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.702+0000 7f9f2e59c640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f3006c2f0 0x7f9f300ab400 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000c7a0 tx=0x7f9f2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:46.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.702+0000 7f9f1effd640 1 -- 172.21.15.81:0/4104846118 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f20003b70 con 0x7f9f3006c2f0 2024-06-29T15:38:46.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.702+0000 7f9f1effd640 1 -- 172.21.15.81:0/4104846118 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f20003d30 con 0x7f9f3006c2f0 2024-06-29T15:38:46.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.702+0000 7f9f34b35640 1 -- 172.21.15.81:0/4104846118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f301f6db0 con 0x7f9f3006c2f0 2024-06-29T15:38:46.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.702+0000 7f9f1effd640 1 -- 172.21.15.81:0/4104846118 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2000aab0 con 0x7f9f3006c2f0 2024-06-29T15:38:46.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.702+0000 7f9f34b35640 1 -- 172.21.15.81:0/4104846118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f301f7370 con 0x7f9f3006c2f0 2024-06-29T15:38:46.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.703+0000 7f9f34b35640 1 -- 172.21.15.81:0/4104846118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f301602e0 con 0x7f9f3006c2f0 2024-06-29T15:38:46.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.704+0000 7f9f1effd640 1 -- 172.21.15.81:0/4104846118 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 90) ==== 98303+0+0 (secure 0 0 0) 0x7f9f2000ac50 con 0x7f9f3006c2f0 2024-06-29T15:38:46.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.704+0000 7f9f1effd640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9f100763f0 0x7f9f10078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:46.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.704+0000 7f9f1effd640 1 -- 172.21.15.81:0/4104846118 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f9f20097750 con 0x7f9f3006c2f0 2024-06-29T15:38:46.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.705+0000 7f9f2dd9b640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9f100763f0 0x7f9f10078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:46.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.705+0000 7f9f2dd9b640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9f100763f0 0x7f9f10078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9f300d14a0 tx=0x7f9f24005f90 comp rx=0 tx=0).ready entity=mgr.9414 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:46.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.709+0000 7f9f1effd640 1 -- 172.21.15.81:0/4104846118 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9f3006cc40 con 0x7f9f3006c2f0 2024-06-29T15:38:46.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:46.933+0000 7f9f34b35640 1 -- 172.21.15.81:0/4104846118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f9f3006f330 con 0x7f9f3006c2f0 2024-06-29T15:38:47.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.411+0000 7f9f1effd640 1 -- 172.21.15.81:0/4104846118 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v91) ==== 92+0+0 (secure 0 0 0) 0x7f9f200607d0 con 0x7f9f3006c2f0 2024-06-29T15:38:47.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.415+0000 7f9f34b35640 1 -- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9f100763f0 msgr2=0x7f9f10078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.415+0000 7f9f34b35640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9f100763f0 0x7f9f10078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9f300d14a0 tx=0x7f9f24005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:47.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.415+0000 7f9f34b35640 1 -- 172.21.15.81:0/4104846118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f3006c2f0 msgr2=0x7f9f300ab400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.415+0000 7f9f34b35640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f3006c2f0 0x7f9f300ab400 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000c7a0 tx=0x7f9f2000cc60 comp rx=0 tx=0).stop 2024-06-29T15:38:47.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.415+0000 7f9f34b35640 1 -- 172.21.15.81:0/4104846118 shutdown_connections 2024-06-29T15:38:47.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.415+0000 7f9f34b35640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f9f100763f0 0x7f9f10078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.415+0000 7f9f34b35640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f30159bd0 0x7f9f300b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.415+0000 7f9f34b35640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f3006f9a0 0x7f9f30158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.415+0000 7f9f34b35640 1 --2- 172.21.15.81:0/4104846118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f3006c2f0 0x7f9f300ab400 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.415+0000 7f9f34b35640 1 -- 172.21.15.81:0/4104846118 >> 172.21.15.81:0/4104846118 conn(0x7f9f300b2920 msgr2=0x7f9f300b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:47.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.416+0000 7f9f34b35640 1 -- 172.21.15.81:0/4104846118 shutdown_connections 2024-06-29T15:38:47.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.416+0000 7f9f34b35640 1 -- 172.21.15.81:0/4104846118 wait complete. 2024-06-29T15:38:47.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:38:47.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.550+0000 7fe835ab3640 1 -- 172.21.15.81:0/2306266007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300b5250 msgr2=0x7fe8300d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.550+0000 7fe835ab3640 1 --2- 172.21.15.81:0/2306266007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300b5250 0x7fe8300d1fc0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fe828009830 tx=0x7fe82802a5a0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.550+0000 7fe835ab3640 1 -- 172.21.15.81:0/2306266007 shutdown_connections 2024-06-29T15:38:47.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.550+0000 7fe835ab3640 1 --2- 172.21.15.81:0/2306266007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300b5250 0x7fe8300d1fc0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.550+0000 7fe835ab3640 1 --2- 172.21.15.81:0/2306266007 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8300d0b90 0x7fe8300d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.550+0000 7fe835ab3640 1 -- 172.21.15.81:0/2306266007 >> 172.21.15.81:0/2306266007 conn(0x7fe83009cf30 msgr2=0x7fe8300d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:47.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.550+0000 7fe835ab3640 1 -- 172.21.15.81:0/2306266007 shutdown_connections 2024-06-29T15:38:47.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.550+0000 7fe835ab3640 1 -- 172.21.15.81:0/2306266007 wait complete. 2024-06-29T15:38:47.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.552+0000 7fe835ab3640 1 Processor -- start 2024-06-29T15:38:47.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.552+0000 7fe835ab3640 1 -- start start 2024-06-29T15:38:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.553+0000 7fe835ab3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8300b5250 0x7fe83015c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.553+0000 7fe835ab3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8300b9040 0x7fe8300c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.553+0000 7fe835ab3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300ccf10 0x7fe8300cd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.553+0000 7fe835ab3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe8300d2680 con 0x7fe8300ccf10 2024-06-29T15:38:47.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.553+0000 7fe835ab3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe8300d2500 con 0x7fe8300b9040 2024-06-29T15:38:47.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.553+0000 7fe835ab3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe8300d2800 con 0x7fe8300b5250 2024-06-29T15:38:47.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300ccf10 0x7fe8300cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300ccf10 0x7fe8300cd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53972/0 (socket says 172.21.15.81:53972) 2024-06-29T15:38:47.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82ffff640 1 -- 172.21.15.81:0/2190158000 learned_addr learned my addr 172.21.15.81:0/2190158000 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:47.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82effd640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8300b9040 0x7fe8300c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82f7fe640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8300b5250 0x7fe83015c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82ffff640 1 -- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8300b5250 msgr2=0x7fe83015c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82ffff640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8300b5250 0x7fe83015c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82ffff640 1 -- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8300b9040 msgr2=0x7fe8300c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82ffff640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8300b9040 0x7fe8300c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82ffff640 1 -- 172.21.15.81:0/2190158000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8300cd860 con 0x7fe8300ccf10 2024-06-29T15:38:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe82ffff640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300ccf10 0x7fe8300cd320 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fe82802ab90 tx=0x7fe828033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe827fff640 1 -- 172.21.15.81:0/2190158000 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe828005c90 con 0x7fe8300ccf10 2024-06-29T15:38:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe827fff640 1 -- 172.21.15.81:0/2190158000 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe828031b60 con 0x7fe8300ccf10 2024-06-29T15:38:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe835ab3640 1 -- 172.21.15.81:0/2190158000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8301f4cd0 con 0x7fe8300ccf10 2024-06-29T15:38:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe827fff640 1 -- 172.21.15.81:0/2190158000 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe828005020 con 0x7fe8300ccf10 2024-06-29T15:38:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.554+0000 7fe835ab3640 1 -- 172.21.15.81:0/2190158000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8301f54b0 con 0x7fe8300ccf10 2024-06-29T15:38:47.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.556+0000 7fe835ab3640 1 -- 172.21.15.81:0/2190158000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8301f4f40 con 0x7fe8300ccf10 2024-06-29T15:38:47.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.556+0000 7fe827fff640 1 -- 172.21.15.81:0/2190158000 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 91) ==== 98320+0+0 (secure 0 0 0) 0x7fe828045020 con 0x7fe8300ccf10 2024-06-29T15:38:47.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.557+0000 7fe827fff640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe810076440 0x7fe810078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.557+0000 7fe82f7fe640 1 -- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe810076440 msgr2=0x7fe810078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/150801987 2024-06-29T15:38:47.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.557+0000 7fe827fff640 1 -- 172.21.15.81:0/2190158000 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7fe8280b8250 con 0x7fe8300ccf10 2024-06-29T15:38:47.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.557+0000 7fe82f7fe640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe810076440 0x7fe810078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:47.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.561+0000 7fe827fff640 1 -- 172.21.15.81:0/2190158000 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe8300cd860 con 0x7fe8300ccf10 2024-06-29T15:38:47.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.758+0000 7fe82f7fe640 1 -- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe810076440 msgr2=0x7fe810078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/150801987 2024-06-29T15:38:47.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.758+0000 7fe82f7fe640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe810076440 0x7fe810078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:47.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.760+0000 7fe835ab3640 1 -- 172.21.15.81:0/2190158000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fe8300d2d90 con 0x7fe8300ccf10 2024-06-29T15:38:47.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.763+0000 7fe827fff640 1 -- 172.21.15.81:0/2190158000 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v61)=0 v61) ==== 138+0+0 (secure 0 0 0) 0x7fe828031770 con 0x7fe8300ccf10 2024-06-29T15:38:47.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 -- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe810076440 msgr2=0x7fe810078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:47.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe810076440 0x7fe810078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 -- 172.21.15.81:0/2190158000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300ccf10 msgr2=0x7fe8300cd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300ccf10 0x7fe8300cd320 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fe82802ab90 tx=0x7fe828033c20 comp rx=0 tx=0).stop 2024-06-29T15:38:47.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 -- 172.21.15.81:0/2190158000 shutdown_connections 2024-06-29T15:38:47.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fe810076440 0x7fe810078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8300ccf10 0x7fe8300cd320 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe8300b9040 0x7fe8300c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 --2- 172.21.15.81:0/2190158000 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe8300b5250 0x7fe83015c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 -- 172.21.15.81:0/2190158000 >> 172.21.15.81:0/2190158000 conn(0x7fe83009cf30 msgr2=0x7fe83006e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:47.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 -- 172.21.15.81:0/2190158000 shutdown_connections 2024-06-29T15:38:47.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.767+0000 7fe835ab3640 1 -- 172.21.15.81:0/2190158000 wait complete. 2024-06-29T15:38:47.783 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:38:47.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.902+0000 7fc666ca5640 1 -- 172.21.15.81:0/1306023872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6600cff30 msgr2=0x7fc6600d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.902+0000 7fc666ca5640 1 --2- 172.21.15.81:0/1306023872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6600cff30 0x7fc6600d0340 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009830 tx=0x7fc64c02a440 comp rx=0 tx=0).stop 2024-06-29T15:38:47.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.902+0000 7fc666ca5640 1 -- 172.21.15.81:0/1306023872 shutdown_connections 2024-06-29T15:38:47.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.902+0000 7fc666ca5640 1 --2- 172.21.15.81:0/1306023872 >> v2:172.21.15.167:3300/0 conn(0x7fc66006c2f0 0x7fc66006c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.902+0000 7fc666ca5640 1 --2- 172.21.15.81:0/1306023872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6600cff30 0x7fc6600d0340 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.902+0000 7fc666ca5640 1 -- 172.21.15.81:0/1306023872 >> 172.21.15.81:0/1306023872 conn(0x7fc6600b2920 msgr2=0x7fc6600b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:47.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.902+0000 7fc666ca5640 1 -- 172.21.15.81:0/1306023872 shutdown_connections 2024-06-29T15:38:47.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.902+0000 7fc666ca5640 1 -- 172.21.15.81:0/1306023872 wait complete. 2024-06-29T15:38:47.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.903+0000 7fc666ca5640 1 Processor -- start 2024-06-29T15:38:47.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.904+0000 7fc666ca5640 1 -- start start 2024-06-29T15:38:47.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc666ca5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66006c2f0 0x7fc6601f3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc666ca5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660070d60 0x7fc66015b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc666ca5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6600cff30 0x7fc660157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc666ca5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc66015fcb0 con 0x7fc660070d60 2024-06-29T15:38:47.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc666ca5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc66015fb30 con 0x7fc66006c2f0 2024-06-29T15:38:47.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc666ca5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc66015fe30 con 0x7fc6600cff30 2024-06-29T15:38:47.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc65ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660070d60 0x7fc66015b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc65ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660070d60 0x7fc66015b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53984/0 (socket says 172.21.15.81:53984) 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc65ffff640 1 -- 172.21.15.81:0/2808323594 learned_addr learned my addr 172.21.15.81:0/2808323594 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc664a41640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66006c2f0 0x7fc6601f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc665242640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6600cff30 0x7fc660157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc65ffff640 1 -- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6600cff30 msgr2=0x7fc660157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc65ffff640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6600cff30 0x7fc660157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc65ffff640 1 -- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66006c2f0 msgr2=0x7fc6601f3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc65ffff640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66006c2f0 0x7fc6601f3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.905+0000 7fc65ffff640 1 -- 172.21.15.81:0/2808323594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc660156ca0 con 0x7fc660070d60 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.906+0000 7fc65ffff640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660070d60 0x7fc66015b6a0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009950 tx=0x7fc64c033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.906+0000 7fc65d7fa640 1 -- 172.21.15.81:0/2808323594 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c005c90 con 0x7fc660070d60 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.906+0000 7fc666ca5640 1 -- 172.21.15.81:0/2808323594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc660156f80 con 0x7fc660070d60 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.906+0000 7fc65d7fa640 1 -- 172.21.15.81:0/2808323594 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc64c031e50 con 0x7fc660070d60 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.906+0000 7fc65d7fa640 1 -- 172.21.15.81:0/2808323594 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c005260 con 0x7fc660070d60 2024-06-29T15:38:47.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.906+0000 7fc666ca5640 1 -- 172.21.15.81:0/2808323594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc660157540 con 0x7fc660070d60 2024-06-29T15:38:47.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.907+0000 7fc666ca5640 1 -- 172.21.15.81:0/2808323594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6600ceac0 con 0x7fc660070d60 2024-06-29T15:38:47.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.908+0000 7fc65d7fa640 1 -- 172.21.15.81:0/2808323594 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 91) ==== 98320+0+0 (secure 0 0 0) 0x7fc64c005400 con 0x7fc660070d60 2024-06-29T15:38:47.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.909+0000 7fc65d7fa640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc644076440 0x7fc644078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:47.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.909+0000 7fc664a41640 1 -- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc644076440 msgr2=0x7fc644078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/150801987 2024-06-29T15:38:47.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.909+0000 7fc65d7fa640 1 -- 172.21.15.81:0/2808323594 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7fc64c0b8b70 con 0x7fc660070d60 2024-06-29T15:38:47.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.909+0000 7fc664a41640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc644076440 0x7fc644078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:47.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:47.914+0000 7fc65d7fa640 1 -- 172.21.15.81:0/2808323594 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc64c03f030 con 0x7fc660070d60 2024-06-29T15:38:48.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.103+0000 7fc666ca5640 1 -- 172.21.15.81:0/2808323594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fc6601602f0 con 0x7fc660070d60 2024-06-29T15:38:48.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.107+0000 7fc65d7fa640 1 -- 172.21.15.81:0/2808323594 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v62)=0 v62) ==== 133+0+0 (secure 0 0 0) 0x7fc64c08a1b0 con 0x7fc660070d60 2024-06-29T15:38:48.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.109+0000 7fc666ca5640 1 -- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc644076440 msgr2=0x7fc644078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:48.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc644076440 0x7fc644078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 -- 172.21.15.81:0/2808323594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660070d60 msgr2=0x7fc66015b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:48.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660070d60 0x7fc66015b6a0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009950 tx=0x7fc64c033c90 comp rx=0 tx=0).stop 2024-06-29T15:38:48.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 -- 172.21.15.81:0/2808323594 shutdown_connections 2024-06-29T15:38:48.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7fc644076440 0x7fc644078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6600cff30 0x7fc660157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660070d60 0x7fc66015b6a0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 --2- 172.21.15.81:0/2808323594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc66006c2f0 0x7fc6601f3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 -- 172.21.15.81:0/2808323594 >> 172.21.15.81:0/2808323594 conn(0x7fc6600b2920 msgr2=0x7fc6600b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:48.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 -- 172.21.15.81:0/2808323594 shutdown_connections 2024-06-29T15:38:48.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.110+0000 7fc666ca5640 1 -- 172.21.15.81:0/2808323594 wait complete. 2024-06-29T15:38:48.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:48.122 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T15:38:48.141 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2024-06-29T15:38:48.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.297+0000 7f866eee7640 1 -- 172.21.15.81:0/3128371624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86680b4f40 msgr2=0x7f86680b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:48.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.297+0000 7f866eee7640 1 --2- 172.21.15.81:0/3128371624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86680b4f40 0x7f86680b5350 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f8660005fd0 tx=0x7f866002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.297+0000 7f866eee7640 1 -- 172.21.15.81:0/3128371624 shutdown_connections 2024-06-29T15:38:48.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.297+0000 7f866eee7640 1 --2- 172.21.15.81:0/3128371624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86680b4f40 0x7f86680b5350 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.297+0000 7f866eee7640 1 --2- 172.21.15.81:0/3128371624 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86680d0230 0x7f86680d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.297+0000 7f866eee7640 1 -- 172.21.15.81:0/3128371624 >> 172.21.15.81:0/3128371624 conn(0x7f86680b3680 msgr2=0x7f86680b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:48.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.297+0000 7f866eee7640 1 -- 172.21.15.81:0/3128371624 shutdown_connections 2024-06-29T15:38:48.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.297+0000 7f866eee7640 1 -- 172.21.15.81:0/3128371624 wait complete. 2024-06-29T15:38:48.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.299+0000 7f866eee7640 1 Processor -- start 2024-06-29T15:38:48.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.299+0000 7f866eee7640 1 -- start start 2024-06-29T15:38:48.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f866eee7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86680b3200 0x7f86681e0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:48.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f866eee7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86680b4f40 0x7f86680c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f866eee7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86680d0230 0x7f86681e3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f866eee7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f86680b5c20 con 0x7f86680b4f40 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f866eee7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f86680b5aa0 con 0x7f86680b3200 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f866eee7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f86680b5da0 con 0x7f86680d0230 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f8667fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86680b4f40 0x7f86680c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f8667fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86680b4f40 0x7f86680c7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53996/0 (socket says 172.21.15.81:53996) 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f8667fff640 1 -- 172.21.15.81:0/3705682367 learned_addr learned my addr 172.21.15.81:0/3705682367 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f866d484640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86680d0230 0x7f86681e3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f866cc83640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86680b3200 0x7f86681e0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f8667fff640 1 -- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86680d0230 msgr2=0x7f86681e3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f8667fff640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86680d0230 0x7f86681e3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f8667fff640 1 -- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86680b3200 msgr2=0x7f86681e0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:48.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f8667fff640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86680b3200 0x7f86681e0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:48.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f8667fff640 1 -- 172.21.15.81:0/3705682367 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86680ca090 con 0x7f86680b4f40 2024-06-29T15:38:48.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f8667fff640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86680b4f40 0x7f86680c7720 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f865400e790 tx=0x7f865400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:48.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.301+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865400eea0 con 0x7f86680b4f40 2024-06-29T15:38:48.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.302+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f865400cdb0 con 0x7f86680b4f40 2024-06-29T15:38:48.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.302+0000 7f866eee7640 1 -- 172.21.15.81:0/3705682367 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866806d7a0 con 0x7f86680b4f40 2024-06-29T15:38:48.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.302+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8654010690 con 0x7f86680b4f40 2024-06-29T15:38:48.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.302+0000 7f866eee7640 1 -- 172.21.15.81:0/3705682367 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866806dd60 con 0x7f86680b4f40 2024-06-29T15:38:48.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.303+0000 7f866eee7640 1 -- 172.21.15.81:0/3705682367 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86680b5350 con 0x7f86680b4f40 2024-06-29T15:38:48.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.303+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 91) ==== 98320+0+0 (secure 0 0 0) 0x7f8654010830 con 0x7f86680b4f40 2024-06-29T15:38:48.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.304+0000 7f86657fa640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 0x7f864c078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:48.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.304+0000 7f866cc83640 1 -- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 msgr2=0x7f864c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/150801987 2024-06-29T15:38:48.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.304+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 5809+0+0 (secure 0 0 0) 0x7f8654098340 con 0x7f86680b4f40 2024-06-29T15:38:48.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.304+0000 7f866cc83640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 0x7f864c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:48.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.309+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f86680ca090 con 0x7f86680b4f40 2024-06-29T15:38:48.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.506+0000 7f866cc83640 1 -- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 msgr2=0x7f864c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/150801987 2024-06-29T15:38:48.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.506+0000 7f866cc83640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 0x7f864c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:48.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.522+0000 7f866eee7640 1 -- 172.21.15.81:0/3705682367 --> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) -- 0x7f86680d1f10 con 0x7f864c076440 2024-06-29T15:38:48.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.907+0000 7f866cc83640 1 -- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 msgr2=0x7f864c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/150801987 2024-06-29T15:38:48.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:48.907+0000 7f866cc83640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 0x7f864c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:38:49.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:49.708+0000 7f866cc83640 1 -- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 msgr2=0x7f864c078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/150801987 2024-06-29T15:38:49.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:49.708+0000 7f866cc83640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 0x7f864c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:38:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:51.013+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 92) ==== 97929+0+0 (secure 0 0 0) 0x7f86540967f0 con 0x7f86680b4f40 2024-06-29T15:38:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:51.013+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 msgr2=0x7f864c078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:51.013+0000 7f86657fa640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/150801987,v1:172.21.15.167:6833/150801987] conn(0x7f864c076440 0x7f864c078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:51.160 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:51.159+0000 7f1ec0536640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:51.160 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:51.159+0000 7f1ec0536640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:51.160 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:51.159+0000 7f1ec0536640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:51.160 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:51.159+0000 7f1ec0536640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:51.160 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:51.159+0000 7f1ebdd31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:51.160 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:51.159+0000 7f1ec0536640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:51.160 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:51.159+0000 7f1ebdd31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:51.160 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:51.159+0000 7f1ebdd31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:51.160 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:51.159+0000 7f1ebdd31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:51.160 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:51.159+0000 7f1ebdd31640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:52.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.020+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 93) ==== 98056+0+0 (secure 0 0 0) 0x7f865405d650 con 0x7f86680b4f40 2024-06-29T15:38:52.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.020+0000 7f86657fa640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f864c07e480 0x7f864c0808a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.020+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 --> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) -- 0x7f8654062430 con 0x7f864c07e480 2024-06-29T15:38:52.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.022+0000 7f866cc83640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f864c07e480 0x7f864c0808a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.028+0000 7f866cc83640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f864c07e480 0x7f864c0808a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f865c0011c0 tx=0x7f865c007e90 comp rx=0 tx=0).ready entity=mgr.8848 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.096+0000 7f86657fa640 1 -- 172.21.15.81:0/3705682367 <== mgr.8848 v2:172.21.15.167:6832/4280924958 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8654062430 con 0x7f864c07e480 2024-06-29T15:38:52.097 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1 2024-06-29T15:38:52.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.098+0000 7f866eee7640 1 -- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f864c07e480 msgr2=0x7f864c0808a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.098+0000 7f866eee7640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f864c07e480 0x7f864c0808a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f865c0011c0 tx=0x7f865c007e90 comp rx=0 tx=0).stop 2024-06-29T15:38:52.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.099+0000 7f866eee7640 1 -- 172.21.15.81:0/3705682367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86680b4f40 msgr2=0x7f86680c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.099+0000 7f866eee7640 1 --2- 172.21.15.81:0/3705682367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86680b4f40 0x7f86680c7720 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f865400e790 tx=0x7f865400ec50 comp rx=0 tx=0).stop 2024-06-29T15:38:52.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.099+0000 7f866d484640 1 -- 172.21.15.81:0/3705682367 reap_dead start 2024-06-29T15:38:52.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.099+0000 7f866eee7640 1 -- 172.21.15.81:0/3705682367 shutdown_connections 2024-06-29T15:38:52.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.099+0000 7f866eee7640 1 -- 172.21.15.81:0/3705682367 >> 172.21.15.81:0/3705682367 conn(0x7f86680b3680 msgr2=0x7f86680c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:52.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.099+0000 7f866eee7640 1 -- 172.21.15.81:0/3705682367 shutdown_connections 2024-06-29T15:38:52.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.099+0000 7f866eee7640 1 -- 172.21.15.81:0/3705682367 wait complete. 2024-06-29T15:38:52.111 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T15:38:52.111 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-06-29T15:38:52.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.227+0000 7f81e6c26640 1 -- 172.21.15.81:0/3204837917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81e006f850 msgr2=0x7f81e00ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.227+0000 7f81e6c26640 1 --2- 172.21.15.81:0/3204837917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81e006f850 0x7f81e00ae300 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f81d4005fd0 tx=0x7f81d402a530 comp rx=0 tx=0).stop 2024-06-29T15:38:52.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.227+0000 7f81e6c26640 1 -- 172.21.15.81:0/3204837917 shutdown_connections 2024-06-29T15:38:52.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.227+0000 7f81e6c26640 1 --2- 172.21.15.81:0/3204837917 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81e00ae840 0x7f81e00aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.227+0000 7f81e6c26640 1 --2- 172.21.15.81:0/3204837917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81e006f850 0x7f81e00ae300 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.227+0000 7f81e6c26640 1 -- 172.21.15.81:0/3204837917 >> 172.21.15.81:0/3204837917 conn(0x7f81e0072950 msgr2=0x7f81e00b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:52.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.227+0000 7f81e6c26640 1 -- 172.21.15.81:0/3204837917 shutdown_connections 2024-06-29T15:38:52.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.227+0000 7f81e6c26640 1 -- 172.21.15.81:0/3204837917 wait complete. 2024-06-29T15:38:52.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.229+0000 7f81e6c26640 1 Processor -- start 2024-06-29T15:38:52.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.229+0000 7f81e6c26640 1 -- start start 2024-06-29T15:38:52.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.230+0000 7f81e6c26640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81e006f850 0x7f81e01521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.230+0000 7f81e6c26640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81e0074520 0x7f81e0154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.230+0000 7f81e6c26640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81e00ae840 0x7f81e0155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.230+0000 7f81e6c26640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f81e015b190 con 0x7f81e00ae840 2024-06-29T15:38:52.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.230+0000 7f81e6c26640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f81e015b010 con 0x7f81e0074520 2024-06-29T15:38:52.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.230+0000 7f81e6c26640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f81e00af190 con 0x7f81e006f850 2024-06-29T15:38:52.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.230+0000 7f81e51c3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81e00ae840 0x7f81e0155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.230+0000 7f81e51c3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81e00ae840 0x7f81e0155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60332/0 (socket says 172.21.15.81:60332) 2024-06-29T15:38:52.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.230+0000 7f81e51c3640 1 -- 172.21.15.81:0/695810959 learned_addr learned my addr 172.21.15.81:0/695810959 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:52.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81dffff640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81e0074520 0x7f81e0154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81e49c2640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81e006f850 0x7f81e01521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81e51c3640 1 -- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81e006f850 msgr2=0x7f81e01521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81e51c3640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81e006f850 0x7f81e01521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81e51c3640 1 -- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81e0074520 msgr2=0x7f81e0154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81e51c3640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81e0074520 0x7f81e0154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81e51c3640 1 -- 172.21.15.81:0/695810959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e0152d20 con 0x7f81e00ae840 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81dffff640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81e0074520 0x7f81e0154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81e51c3640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81e00ae840 0x7f81e0155330 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f81d800cab0 tx=0x7f81d800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81dd7fa640 1 -- 172.21.15.81:0/695810959 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8007d90 con 0x7f81e00ae840 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81dd7fa640 1 -- 172.21.15.81:0/695810959 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81d800f560 con 0x7f81e00ae840 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81e6c26640 1 -- 172.21.15.81:0/695810959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e015c2e0 con 0x7f81e00ae840 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81dd7fa640 1 -- 172.21.15.81:0/695810959 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d800f9b0 con 0x7f81e00ae840 2024-06-29T15:38:52.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.231+0000 7f81e6c26640 1 -- 172.21.15.81:0/695810959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e015c8a0 con 0x7f81e00ae840 2024-06-29T15:38:52.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.232+0000 7f81e6c26640 1 -- 172.21.15.81:0/695810959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e00ae300 con 0x7f81e00ae840 2024-06-29T15:38:52.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.233+0000 7f81dd7fa640 1 -- 172.21.15.81:0/695810959 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 93) ==== 98056+0+0 (secure 0 0 0) 0x7f81d800fb50 con 0x7f81e00ae840 2024-06-29T15:38:52.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.233+0000 7f81dd7fa640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f81cc0761a0 0x7f81cc0786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.233+0000 7f81dd7fa640 1 -- 172.21.15.81:0/695810959 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7f81d8097f60 con 0x7f81e00ae840 2024-06-29T15:38:52.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.234+0000 7f81e49c2640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f81cc0761a0 0x7f81cc0786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.234+0000 7f81e49c2640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f81cc0761a0 0x7f81cc0786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f81d0006fd0 tx=0x7f81d0008040 comp rx=0 tx=0).ready entity=mgr.8848 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.238+0000 7f81dd7fa640 1 -- 172.21.15.81:0/695810959 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f81d8014070 con 0x7f81e00ae840 2024-06-29T15:38:52.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.440+0000 7f81e6c26640 1 -- 172.21.15.81:0/695810959 --> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f81e006f520 con 0x7f81cc0761a0 2024-06-29T15:38:52.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.510+0000 7f81dd7fa640 1 -- 172.21.15.81:0/695810959 <== mgr.8848 v2:172.21.15.167:6832/4280924958 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f81e006f520 con 0x7f81cc0761a0 2024-06-29T15:38:52.511 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path snap_test_dir1 2024-06-29T15:38:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 -- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f81cc0761a0 msgr2=0x7f81cc0786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f81cc0761a0 0x7f81cc0786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f81d0006fd0 tx=0x7f81d0008040 comp rx=0 tx=0).stop 2024-06-29T15:38:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 -- 172.21.15.81:0/695810959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81e00ae840 msgr2=0x7f81e0155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81e00ae840 0x7f81e0155330 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f81d800cab0 tx=0x7f81d800cf70 comp rx=0 tx=0).stop 2024-06-29T15:38:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 -- 172.21.15.81:0/695810959 shutdown_connections 2024-06-29T15:38:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f81cc0761a0 0x7f81cc0786b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81e00ae840 0x7f81e0155330 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f81e0074520 0x7f81e0154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 --2- 172.21.15.81:0/695810959 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f81e006f850 0x7f81e01521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 -- 172.21.15.81:0/695810959 >> 172.21.15.81:0/695810959 conn(0x7f81e0072950 msgr2=0x7f81e01564b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:52.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 -- 172.21.15.81:0/695810959 shutdown_connections 2024-06-29T15:38:52.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.512+0000 7f81e6c26640 1 -- 172.21.15.81:0/695810959 wait complete. 2024-06-29T15:38:52.524 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-06-29T15:38:52.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2024-06-29T15:38:52.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.640+0000 7f3e47fb2640 1 -- 172.21.15.81:0/322235139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c003c30 msgr2=0x7f3e2c004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.640+0000 7f3e47fb2640 1 --2- 172.21.15.81:0/322235139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c003c30 0x7f3e2c004080 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f3e38006fb0 tx=0x7f3e2c005830 comp rx=0 tx=0).stop 2024-06-29T15:38:52.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.640+0000 7f3e47fb2640 1 -- 172.21.15.81:0/322235139 shutdown_connections 2024-06-29T15:38:52.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.640+0000 7f3e47fb2640 1 --2- 172.21.15.81:0/322235139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c003c30 0x7f3e2c004080 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.640+0000 7f3e47fb2640 1 -- 172.21.15.81:0/322235139 >> 172.21.15.81:0/322235139 conn(0x7f3e400b3510 msgr2=0x7f3e400b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:52.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.640+0000 7f3e47fb2640 1 -- 172.21.15.81:0/322235139 shutdown_connections 2024-06-29T15:38:52.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.640+0000 7f3e47fb2640 1 -- 172.21.15.81:0/322235139 wait complete. 2024-06-29T15:38:52.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.642+0000 7f3e47fb2640 1 Processor -- start 2024-06-29T15:38:52.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.642+0000 7f3e47fb2640 1 -- start start 2024-06-29T15:38:52.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.643+0000 7f3e47fb2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c003c30 0x7f3e401e6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.643+0000 7f3e47fb2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e401da7d0 0x7f3e401da120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.643+0000 7f3e47fb2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e400c5620 0x7f3e400c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.643+0000 7f3e47fb2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3e400ce690 con 0x7f3e2c003c30 2024-06-29T15:38:52.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.643+0000 7f3e47fb2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3e400ce510 con 0x7f3e401da7d0 2024-06-29T15:38:52.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.643+0000 7f3e47fb2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3e400ce810 con 0x7f3e400c5620 2024-06-29T15:38:52.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.643+0000 7f3e45d4e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c003c30 0x7f3e401e6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e45d4e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c003c30 0x7f3e401e6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60362/0 (socket says 172.21.15.81:60362) 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e45d4e640 1 -- 172.21.15.81:0/3158096105 learned_addr learned my addr 172.21.15.81:0/3158096105 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e4554d640 1 --2- 172.21.15.81:0/3158096105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e401da7d0 0x7f3e401da120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e4654f640 1 --2- 172.21.15.81:0/3158096105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e400c5620 0x7f3e400c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e45d4e640 1 -- 172.21.15.81:0/3158096105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e400c5620 msgr2=0x7f3e400c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e45d4e640 1 --2- 172.21.15.81:0/3158096105 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e400c5620 0x7f3e400c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e45d4e640 1 -- 172.21.15.81:0/3158096105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e401da7d0 msgr2=0x7f3e401da120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e45d4e640 1 --2- 172.21.15.81:0/3158096105 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e401da7d0 0x7f3e401da120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e45d4e640 1 -- 172.21.15.81:0/3158096105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e401e5a70 con 0x7f3e2c003c30 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e45d4e640 1 --2- 172.21.15.81:0/3158096105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c003c30 0x7f3e401e6510 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f3e38006fb0 tx=0x7f3e38006bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e327fc640 1 -- 172.21.15.81:0/3158096105 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e380037f0 con 0x7f3e2c003c30 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e327fc640 1 -- 172.21.15.81:0/3158096105 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e38003990 con 0x7f3e2c003c30 2024-06-29T15:38:52.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e47fb2640 1 -- 172.21.15.81:0/3158096105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e401e1500 con 0x7f3e2c003c30 2024-06-29T15:38:52.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e327fc640 1 -- 172.21.15.81:0/3158096105 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e38003dc0 con 0x7f3e2c003c30 2024-06-29T15:38:52.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.644+0000 7f3e47fb2640 1 -- 172.21.15.81:0/3158096105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e401e0f90 con 0x7f3e2c003c30 2024-06-29T15:38:52.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.645+0000 7f3e47fb2640 1 -- 172.21.15.81:0/3158096105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e400d0320 con 0x7f3e2c003c30 2024-06-29T15:38:52.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.646+0000 7f3e327fc640 1 -- 172.21.15.81:0/3158096105 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 93) ==== 98056+0+0 (secure 0 0 0) 0x7f3e38035020 con 0x7f3e2c003c30 2024-06-29T15:38:52.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.646+0000 7f3e327fc640 1 --2- 172.21.15.81:0/3158096105 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f3e240761a0 0x7f3e240786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:52.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.646+0000 7f3e327fc640 1 -- 172.21.15.81:0/3158096105 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7f3e380b7e60 con 0x7f3e2c003c30 2024-06-29T15:38:52.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.647+0000 7f3e4554d640 1 --2- 172.21.15.81:0/3158096105 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f3e240761a0 0x7f3e240786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:52.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.647+0000 7f3e4554d640 1 --2- 172.21.15.81:0/3158096105 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f3e240761a0 0x7f3e240786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3e401e6dc0 tx=0x7f3e3c006b10 comp rx=0 tx=0).ready entity=mgr.8848 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:52.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.651+0000 7f3e327fc640 1 -- 172.21.15.81:0/3158096105 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3e38081940 con 0x7f3e2c003c30 2024-06-29T15:38:52.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.856+0000 7f3e47fb2640 1 -- 172.21.15.81:0/3158096105 --> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f3e400cdf00 con 0x7f3e240761a0 2024-06-29T15:38:52.864 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:52.863+0000 7f1ec7d45640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2024-06-29T15:38:52.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:52.863+0000 7f3e327fc640 1 -- 172.21.15.81:0/3158096105 <== mgr.8848 v2:172.21.15.167:6832/4280924958 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) ==== 49+0+0 (secure 0 0 0) 0x7f3e400cdf00 con 0x7f3e240761a0 2024-06-29T15:38:52.864 INFO:teuthology.orchestra.run.smithi081.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2024-06-29T15:38:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-06-29T15:38:52.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:52.870 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T15:38:52.887 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:38:53.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.043+0000 7f52a3323640 1 -- 172.21.15.81:0/2603165584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f529c145180 msgr2=0x7f529c06edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:53.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.043+0000 7f52a3323640 1 --2- 172.21.15.81:0/2603165584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f529c145180 0x7f529c06edd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f528c009830 tx=0x7f528c02a510 comp rx=0 tx=0).stop 2024-06-29T15:38:53.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.043+0000 7f52a3323640 1 -- 172.21.15.81:0/2603165584 shutdown_connections 2024-06-29T15:38:53.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.043+0000 7f52a3323640 1 --2- 172.21.15.81:0/2603165584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f529c145180 0x7f529c06edd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:53.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.043+0000 7f52a3323640 1 --2- 172.21.15.81:0/2603165584 >> v2:172.21.15.167:3300/0 conn(0x7f529c071f20 0x7f529c144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:53.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.043+0000 7f52a3323640 1 -- 172.21.15.81:0/2603165584 >> 172.21.15.81:0/2603165584 conn(0x7f529c06ccc0 msgr2=0x7f529c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:53.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.043+0000 7f52a3323640 1 -- 172.21.15.81:0/2603165584 shutdown_connections 2024-06-29T15:38:53.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.043+0000 7f52a3323640 1 -- 172.21.15.81:0/2603165584 wait complete. 2024-06-29T15:38:53.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.045+0000 7f52a3323640 1 Processor -- start 2024-06-29T15:38:53.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.045+0000 7f52a3323640 1 -- start start 2024-06-29T15:38:53.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.046+0000 7f52a3323640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c071f20 0x7f529c13a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:53.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.046+0000 7f52a3323640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f529c145180 0x7f529c0b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:53.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.046+0000 7f52a3323640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f529c145590 0x7f529c0aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:53.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a3323640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f529c160050 con 0x7f529c071f20 2024-06-29T15:38:53.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a3323640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f529c06f660 con 0x7f529c145180 2024-06-29T15:38:53.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a3323640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f529c1601d0 con 0x7f529c145590 2024-06-29T15:38:53.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a10bf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c071f20 0x7f529c13a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a10bf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c071f20 0x7f529c13a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60364/0 (socket says 172.21.15.81:60364) 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a10bf640 1 -- 172.21.15.81:0/2773925897 learned_addr learned my addr 172.21.15.81:0/2773925897 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a08be640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f529c145180 0x7f529c0b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a18c0640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f529c145590 0x7f529c0aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a10bf640 1 -- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f529c145590 msgr2=0x7f529c0aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a10bf640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f529c145590 0x7f529c0aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a10bf640 1 -- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f529c145180 msgr2=0x7f529c0b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a10bf640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f529c145180 0x7f529c0b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a10bf640 1 -- 172.21.15.81:0/2773925897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529c1f6380 con 0x7f529c071f20 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a10bf640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c071f20 0x7f529c13a670 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f529800e790 tx=0x7f529800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f5291ffb640 1 -- 172.21.15.81:0/2773925897 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529800eea0 con 0x7f529c071f20 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f5291ffb640 1 -- 172.21.15.81:0/2773925897 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f529800cdb0 con 0x7f529c071f20 2024-06-29T15:38:53.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f52a3323640 1 -- 172.21.15.81:0/2773925897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529c0adae0 con 0x7f529c071f20 2024-06-29T15:38:53.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.047+0000 7f5291ffb640 1 -- 172.21.15.81:0/2773925897 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298010690 con 0x7f529c071f20 2024-06-29T15:38:53.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.048+0000 7f52a3323640 1 -- 172.21.15.81:0/2773925897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f529c0ae0a0 con 0x7f529c071f20 2024-06-29T15:38:53.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.049+0000 7f52a3323640 1 -- 172.21.15.81:0/2773925897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f529c0cf7a0 con 0x7f529c071f20 2024-06-29T15:38:53.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.049+0000 7f5291ffb640 1 -- 172.21.15.81:0/2773925897 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 93) ==== 98056+0+0 (secure 0 0 0) 0x7f5298010830 con 0x7f529c071f20 2024-06-29T15:38:53.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.050+0000 7f5291ffb640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5280076170 0x7f5280078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:53.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.050+0000 7f5291ffb640 1 -- 172.21.15.81:0/2773925897 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7f5298097f40 con 0x7f529c071f20 2024-06-29T15:38:53.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.050+0000 7f52a08be640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5280076170 0x7f5280078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:53.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.050+0000 7f52a08be640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5280076170 0x7f5280078680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f528c034000 tx=0x7f528c004730 comp rx=0 tx=0).ready entity=mgr.8848 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:53.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.054+0000 7f5291ffb640 1 -- 172.21.15.81:0/2773925897 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f529c0cf7a0 con 0x7f529c071f20 2024-06-29T15:38:53.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.068+0000 7f5291ffb640 1 -- 172.21.15.81:0/2773925897 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 94) ==== 98321+0+0 (secure 0 0 0) 0x7f529805d5e0 con 0x7f529c071f20 2024-06-29T15:38:53.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:53.284+0000 7f52a3323640 1 -- 172.21.15.81:0/2773925897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f529c1606f0 con 0x7f529c071f20 2024-06-29T15:38:54.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.075+0000 7f5291ffb640 1 -- 172.21.15.81:0/2773925897 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v95) ==== 93+0+0 (secure 0 0 0) 0x7f5298058890 con 0x7f529c071f20 2024-06-29T15:38:54.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.080+0000 7f52a3323640 1 -- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5280076170 msgr2=0x7f5280078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.080+0000 7f52a3323640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5280076170 0x7f5280078680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f528c034000 tx=0x7f528c004730 comp rx=0 tx=0).stop 2024-06-29T15:38:54.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.080+0000 7f52a3323640 1 -- 172.21.15.81:0/2773925897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c071f20 msgr2=0x7f529c13a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.080+0000 7f52a3323640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c071f20 0x7f529c13a670 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f529800e790 tx=0x7f529800ec50 comp rx=0 tx=0).stop 2024-06-29T15:38:54.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.081+0000 7f52a3323640 1 -- 172.21.15.81:0/2773925897 shutdown_connections 2024-06-29T15:38:54.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.081+0000 7f52a3323640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f529c145590 0x7f529c0aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.081+0000 7f52a3323640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f529c145180 0x7f529c0b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.081+0000 7f52a3323640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5280076170 0x7f5280078680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.081+0000 7f52a3323640 1 --2- 172.21.15.81:0/2773925897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c071f20 0x7f529c13a670 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.081+0000 7f52a3323640 1 -- 172.21.15.81:0/2773925897 >> 172.21.15.81:0/2773925897 conn(0x7f529c06ccc0 msgr2=0x7f529c15efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:54.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.081+0000 7f52a3323640 1 -- 172.21.15.81:0/2773925897 shutdown_connections 2024-06-29T15:38:54.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.081+0000 7f52a3323640 1 -- 172.21.15.81:0/2773925897 wait complete. 2024-06-29T15:38:54.099 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:38:54.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:54.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:38:54.200 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:38:54.200 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:54.200 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:38:54.201 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:54.217 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:38:54.217 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:54.273 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:54.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.429+0000 7fa4829fb640 1 -- 172.21.15.81:0/1813511309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa47c0b9040 msgr2=0x7fa47c0d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.429+0000 7fa4829fb640 1 --2- 172.21.15.81:0/1813511309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa47c0b9040 0x7fa47c0d0480 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009860 tx=0x7fa46c02a620 comp rx=0 tx=0).stop 2024-06-29T15:38:54.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.429+0000 7fa4829fb640 1 -- 172.21.15.81:0/1813511309 shutdown_connections 2024-06-29T15:38:54.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.429+0000 7fa4829fb640 1 --2- 172.21.15.81:0/1813511309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa47c0b9040 0x7fa47c0d0480 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.429+0000 7fa4829fb640 1 -- 172.21.15.81:0/1813511309 >> 172.21.15.81:0/1813511309 conn(0x7fa47c09cf30 msgr2=0x7fa47c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:54.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.430+0000 7fa4829fb640 1 -- 172.21.15.81:0/1813511309 shutdown_connections 2024-06-29T15:38:54.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.430+0000 7fa4829fb640 1 -- 172.21.15.81:0/1813511309 wait complete. 2024-06-29T15:38:54.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.431+0000 7fa4829fb640 1 Processor -- start 2024-06-29T15:38:54.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.431+0000 7fa4829fb640 1 -- start start 2024-06-29T15:38:54.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa4829fb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa47c0b5250 0x7fa47c15a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa4829fb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa47c0b9040 0x7fa47c15cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa4829fb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa47c0d2a90 0x7fa47c15d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa4829fb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c0d0e00 con 0x7fa47c0b5250 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa4829fb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c0d0c80 con 0x7fa47c0d2a90 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa4829fb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa47c0b80b0 con 0x7fa47c0b9040 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa47bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa47c0b5250 0x7fa47c15a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa47bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa47c0b5250 0x7fa47c15a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60374/0 (socket says 172.21.15.81:60374) 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa47bfff640 1 -- 172.21.15.81:0/3547737922 learned_addr learned my addr 172.21.15.81:0/3547737922 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa47b7fe640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa47c0b9040 0x7fa47c15cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa480f98640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa47c0d2a90 0x7fa47c15d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa47bfff640 1 -- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa47c0b9040 msgr2=0x7fa47c15cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa47bfff640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa47c0b9040 0x7fa47c15cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa47bfff640 1 -- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa47c0d2a90 msgr2=0x7fa47c15d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa47bfff640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa47c0d2a90 0x7fa47c15d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa47bfff640 1 -- 172.21.15.81:0/3547737922 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47c15b530 con 0x7fa47c0b5250 2024-06-29T15:38:54.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.433+0000 7fa47bfff640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa47c0b5250 0x7fa47c15a1a0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fa46c000c00 tx=0x7fa46c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:54.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.434+0000 7fa478ff9640 1 -- 172.21.15.81:0/3547737922 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c0044e0 con 0x7fa47c0b5250 2024-06-29T15:38:54.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.434+0000 7fa478ff9640 1 -- 172.21.15.81:0/3547737922 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa46c005330 con 0x7fa47c0b5250 2024-06-29T15:38:54.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.434+0000 7fa478ff9640 1 -- 172.21.15.81:0/3547737922 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c003800 con 0x7fa47c0b5250 2024-06-29T15:38:54.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.434+0000 7fa4829fb640 1 -- 172.21.15.81:0/3547737922 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47c15ba30 con 0x7fa47c0b5250 2024-06-29T15:38:54.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.434+0000 7fa4829fb640 1 -- 172.21.15.81:0/3547737922 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa47c15bff0 con 0x7fa47c0b5250 2024-06-29T15:38:54.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.435+0000 7fa4829fb640 1 -- 172.21.15.81:0/3547737922 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa47c0d0480 con 0x7fa47c0b5250 2024-06-29T15:38:54.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.436+0000 7fa478ff9640 1 -- 172.21.15.81:0/3547737922 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7fa46c0039a0 con 0x7fa47c0b5250 2024-06-29T15:38:54.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.437+0000 7fa478ff9640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fa45c076420 0x7fa45c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.437+0000 7fa47b7fe640 1 -- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fa45c076420 msgr2=0x7fa45c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:54.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.437+0000 7fa478ff9640 1 -- 172.21.15.81:0/3547737922 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7fa46c0b8c90 con 0x7fa47c0b5250 2024-06-29T15:38:54.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.437+0000 7fa47b7fe640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fa45c076420 0x7fa45c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:54.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.441+0000 7fa478ff9640 1 -- 172.21.15.81:0/3547737922 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa46c03f030 con 0x7fa47c0b5250 2024-06-29T15:38:54.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.625+0000 7fa4829fb640 1 -- 172.21.15.81:0/3547737922 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa47c0b8db0 con 0x7fa47c0b5250 2024-06-29T15:38:54.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.626+0000 7fa478ff9640 1 -- 172.21.15.81:0/3547737922 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) ==== 74+0+17333 (secure 0 0 0) 0x7fa46c031770 con 0x7fa47c0b5250 2024-06-29T15:38:54.627 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:54.627 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":169,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:51.005408+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:37.031836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.676381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/1753956564":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/1949204156":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/4139817389":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/3283653249":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/2505002611":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/4205626971":"2024-06-30T15:38:51.005371+0000","172.21.15.167:6833/150801987":"2024-06-30T15:38:51.005371+0000","172.21.15.167:6832/150801987":"2024-06-30T15:38:51.005371+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:54.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 -- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fa45c076420 msgr2=0x7fa45c078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:54.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fa45c076420 0x7fa45c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 -- 172.21.15.81:0/3547737922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa47c0b5250 msgr2=0x7fa47c15a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa47c0b5250 0x7fa47c15a1a0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fa46c000c00 tx=0x7fa46c0042f0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 -- 172.21.15.81:0/3547737922 shutdown_connections 2024-06-29T15:38:54.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa47c0d2a90 0x7fa47c15d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa47c0b9040 0x7fa47c15cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fa45c076420 0x7fa45c078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 --2- 172.21.15.81:0/3547737922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa47c0b5250 0x7fa47c15a1a0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 -- 172.21.15.81:0/3547737922 >> 172.21.15.81:0/3547737922 conn(0x7fa47c09cf30 msgr2=0x7fa47c06dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:54.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 -- 172.21.15.81:0/3547737922 shutdown_connections 2024-06-29T15:38:54.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.628+0000 7fa4829fb640 1 -- 172.21.15.81:0/3547737922 wait complete. 2024-06-29T15:38:54.641 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:38:37.031836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '167', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:38:37.676381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '168', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:54.641 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:54.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.760+0000 7f5e333eb640 1 -- 172.21.15.81:0/1287813906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2c0b3200 msgr2=0x7f5e2c0cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.760+0000 7f5e333eb640 1 --2- 172.21.15.81:0/1287813906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2c0b3200 0x7f5e2c0cc7b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c009830 tx=0x7f5e1c02a510 comp rx=0 tx=0).stop 2024-06-29T15:38:54.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.760+0000 7f5e333eb640 1 -- 172.21.15.81:0/1287813906 shutdown_connections 2024-06-29T15:38:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.760+0000 7f5e333eb640 1 --2- 172.21.15.81:0/1287813906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e2c0cccf0 0x7f5e2c0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.760+0000 7f5e333eb640 1 --2- 172.21.15.81:0/1287813906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2c0b3200 0x7f5e2c0cc7b0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.760+0000 7f5e333eb640 1 -- 172.21.15.81:0/1287813906 >> 172.21.15.81:0/1287813906 conn(0x7f5e2c0b3680 msgr2=0x7f5e2c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.760+0000 7f5e333eb640 1 -- 172.21.15.81:0/1287813906 shutdown_connections 2024-06-29T15:38:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.760+0000 7f5e333eb640 1 -- 172.21.15.81:0/1287813906 wait complete. 2024-06-29T15:38:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.762+0000 7f5e333eb640 1 Processor -- start 2024-06-29T15:38:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.762+0000 7f5e333eb640 1 -- start start 2024-06-29T15:38:54.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.763+0000 7f5e333eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e2c0b3200 0x7f5e2c1f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.763+0000 7f5e333eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e2c0cccf0 0x7f5e2c1f8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e333eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2c0d0770 0x7f5e2c06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e333eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c0cedb0 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e333eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c0cec30 con 0x7f5e2c0d0770 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e333eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5e2c0cef30 con 0x7f5e2c0b3200 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e30986640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e2c0cccf0 0x7f5e2c1f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e30986640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e2c0cccf0 0x7f5e2c1f8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60388/0 (socket says 172.21.15.81:60388) 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e30986640 1 -- 172.21.15.81:0/1700591717 learned_addr learned my addr 172.21.15.81:0/1700591717 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e31187640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e2c0b3200 0x7f5e2c1f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e31988640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2c0d0770 0x7f5e2c06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e30986640 1 -- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e2c0b3200 msgr2=0x7f5e2c1f56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e30986640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e2c0b3200 0x7f5e2c1f56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e30986640 1 -- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2c0d0770 msgr2=0x7f5e2c06d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e30986640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2c0d0770 0x7f5e2c06d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e30986640 1 -- 172.21.15.81:0/1700591717 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2c1f5c10 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e30986640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e2c0cccf0 0x7f5e2c1f8570 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800e9f0 tx=0x7f5e2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e21ffb640 1 -- 172.21.15.81:0/1700591717 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2800cd80 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e21ffb640 1 -- 172.21.15.81:0/1700591717 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e28004020 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e333eb640 1 -- 172.21.15.81:0/1700591717 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2c1f8f30 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.764+0000 7f5e21ffb640 1 -- 172.21.15.81:0/1700591717 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28004470 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.765+0000 7f5e333eb640 1 -- 172.21.15.81:0/1700591717 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e2c1f51d0 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.766+0000 7f5e333eb640 1 -- 172.21.15.81:0/1700591717 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e2c0cc7b0 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.766+0000 7f5e21ffb640 1 -- 172.21.15.81:0/1700591717 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7f5e280035f0 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.767+0000 7f5e21ffb640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e10076420 0x7f5e10078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:54.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.767+0000 7f5e21ffb640 1 -- 172.21.15.81:0/1700591717 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7f5e280984d0 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.767+0000 7f5e31187640 1 -- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e10076420 msgr2=0x7f5e10078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:54.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.767+0000 7f5e31187640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e10076420 0x7f5e10078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:54.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.771+0000 7f5e21ffb640 1 -- 172.21.15.81:0/1700591717 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5e2c1f5c10 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.951+0000 7f5e333eb640 1 -- 172.21.15.81:0/1700591717 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5e2c0d25a0 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.952+0000 7f5e21ffb640 1 -- 172.21.15.81:0/1700591717 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v169) ==== 93+0+10 (secure 0 0 0) 0x7f5e28061370 con 0x7f5e2c0cccf0 2024-06-29T15:38:54.953 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:38:54.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 -- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e10076420 msgr2=0x7f5e10078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:54.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e10076420 0x7f5e10078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 -- 172.21.15.81:0/1700591717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e2c0cccf0 msgr2=0x7f5e2c1f8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:54.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e2c0cccf0 0x7f5e2c1f8570 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800e9f0 tx=0x7f5e2800eeb0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 -- 172.21.15.81:0/1700591717 shutdown_connections 2024-06-29T15:38:54.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2c0d0770 0x7f5e2c06d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e10076420 0x7f5e10078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e2c0cccf0 0x7f5e2c1f8570 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 --2- 172.21.15.81:0/1700591717 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e2c0b3200 0x7f5e2c1f56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:54.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 -- 172.21.15.81:0/1700591717 >> 172.21.15.81:0/1700591717 conn(0x7f5e2c0b3680 msgr2=0x7f5e2c070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:54.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 -- 172.21.15.81:0/1700591717 shutdown_connections 2024-06-29T15:38:54.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:54.954+0000 7f5e333eb640 1 -- 172.21.15.81:0/1700591717 wait complete. 2024-06-29T15:38:54.966 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:38:55.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.087+0000 7f1cd69eb640 1 -- 172.21.15.81:0/1251781753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd015b8f0 msgr2=0x7f1cd006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.087+0000 7f1cd69eb640 1 --2- 172.21.15.81:0/1251781753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd015b8f0 0x7f1cd006edb0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009ca0 tx=0x7f1cc802aac0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.087+0000 7f1cd69eb640 1 -- 172.21.15.81:0/1251781753 shutdown_connections 2024-06-29T15:38:55.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.087+0000 7f1cd69eb640 1 --2- 172.21.15.81:0/1251781753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd015b8f0 0x7f1cd006edb0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.087+0000 7f1cd69eb640 1 --2- 172.21.15.81:0/1251781753 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b46a0 0x7f1cd0072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.087+0000 7f1cd69eb640 1 -- 172.21.15.81:0/1251781753 >> 172.21.15.81:0/1251781753 conn(0x7f1cd00d0b70 msgr2=0x7f1cd00d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.087+0000 7f1cd69eb640 1 -- 172.21.15.81:0/1251781753 shutdown_connections 2024-06-29T15:38:55.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.087+0000 7f1cd69eb640 1 -- 172.21.15.81:0/1251781753 wait complete. 2024-06-29T15:38:55.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.089+0000 7f1cd69eb640 1 Processor -- start 2024-06-29T15:38:55.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.089+0000 7f1cd69eb640 1 -- start start 2024-06-29T15:38:55.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1cd69eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd0073480 0x7f1cd01f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1cd69eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b46a0 0x7f1cd00a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1cd69eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd015b8f0 0x7f1cd006c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1cd69eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd006f680 con 0x7f1cd0073480 2024-06-29T15:38:55.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1cd69eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd006f500 con 0x7f1cd015b8f0 2024-06-29T15:38:55.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1cd69eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1cd006f800 con 0x7f1cd00b46a0 2024-06-29T15:38:55.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1ccffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd0073480 0x7f1cd01f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1ccffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd0073480 0x7f1cd01f5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60408/0 (socket says 172.21.15.81:60408) 2024-06-29T15:38:55.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1ccffff640 1 -- 172.21.15.81:0/3498198004 learned_addr learned my addr 172.21.15.81:0/3498198004 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:55.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1cd4f88640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd015b8f0 0x7f1cd006c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1ccf7fe640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b46a0 0x7f1cd00a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1ccffff640 1 -- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b46a0 msgr2=0x7f1cd00a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1ccffff640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b46a0 0x7f1cd00a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1ccffff640 1 -- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd015b8f0 msgr2=0x7f1cd006c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1ccffff640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd015b8f0 0x7f1cd006c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1ccffff640 1 -- 172.21.15.81:0/3498198004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd00a9ea0 con 0x7f1cd0073480 2024-06-29T15:38:55.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.091+0000 7f1ccffff640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd0073480 0x7f1cd01f5fd0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000e7c0 tx=0x7f1cc000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:55.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.092+0000 7f1cccff9640 1 -- 172.21.15.81:0/3498198004 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc000ccc0 con 0x7f1cd0073480 2024-06-29T15:38:55.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.092+0000 7f1cccff9640 1 -- 172.21.15.81:0/3498198004 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1cc0010430 con 0x7f1cd0073480 2024-06-29T15:38:55.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.092+0000 7f1cd69eb640 1 -- 172.21.15.81:0/3498198004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd00aa180 con 0x7f1cd0073480 2024-06-29T15:38:55.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.092+0000 7f1cccff9640 1 -- 172.21.15.81:0/3498198004 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0010860 con 0x7f1cd0073480 2024-06-29T15:38:55.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.092+0000 7f1cd69eb640 1 -- 172.21.15.81:0/3498198004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd00aa740 con 0x7f1cd0073480 2024-06-29T15:38:55.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.093+0000 7f1cd69eb640 1 -- 172.21.15.81:0/3498198004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cd00cd670 con 0x7f1cd0073480 2024-06-29T15:38:55.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.093+0000 7f1cccff9640 1 -- 172.21.15.81:0/3498198004 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7f1cc0010a00 con 0x7f1cd0073480 2024-06-29T15:38:55.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.094+0000 7f1cccff9640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f1cbc0763f0 0x7f1cbc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.094+0000 7f1ccf7fe640 1 -- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f1cbc0763f0 msgr2=0x7f1cbc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:55.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.094+0000 7f1cccff9640 1 -- 172.21.15.81:0/3498198004 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7f1cc0098450 con 0x7f1cd0073480 2024-06-29T15:38:55.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.094+0000 7f1ccf7fe640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f1cbc0763f0 0x7f1cbc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.099+0000 7f1cccff9640 1 -- 172.21.15.81:0/3498198004 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1cd00cd670 con 0x7f1cd0073480 2024-06-29T15:38:55.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.286+0000 7f1cd69eb640 1 -- 172.21.15.81:0/3498198004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1cd006f040 con 0x7f1cd0073480 2024-06-29T15:38:55.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.286+0000 7f1cccff9640 1 -- 172.21.15.81:0/3498198004 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v169) ==== 104+0+11 (secure 0 0 0) 0x7f1cd006f040 con 0x7f1cd0073480 2024-06-29T15:38:55.287 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:38:55.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 -- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f1cbc0763f0 msgr2=0x7f1cbc078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:55.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f1cbc0763f0 0x7f1cbc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 -- 172.21.15.81:0/3498198004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd0073480 msgr2=0x7f1cd01f5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd0073480 0x7f1cd01f5fd0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f1cc000e7c0 tx=0x7f1cc000ec80 comp rx=0 tx=0).stop 2024-06-29T15:38:55.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 -- 172.21.15.81:0/3498198004 shutdown_connections 2024-06-29T15:38:55.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd015b8f0 0x7f1cd006c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b46a0 0x7f1cd00a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f1cbc0763f0 0x7f1cbc078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 --2- 172.21.15.81:0/3498198004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd0073480 0x7f1cd01f5fd0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 -- 172.21.15.81:0/3498198004 >> 172.21.15.81:0/3498198004 conn(0x7f1cd00d0b70 msgr2=0x7f1cd015c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 -- 172.21.15.81:0/3498198004 shutdown_connections 2024-06-29T15:38:55.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.288+0000 7f1cd69eb640 1 -- 172.21.15.81:0/3498198004 wait complete. 2024-06-29T15:38:55.300 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:38:55.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.419+0000 7f54a7438640 1 -- 172.21.15.81:0/903450248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54a00b7290 msgr2=0x7f54a00b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.419+0000 7f54a7438640 1 --2- 172.21.15.81:0/903450248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54a00b7290 0x7f54a00b76a0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f54a008e230 tx=0x7f549802aac0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.419+0000 7f54a7438640 1 -- 172.21.15.81:0/903450248 shutdown_connections 2024-06-29T15:38:55.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.419+0000 7f54a7438640 1 --2- 172.21.15.81:0/903450248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54a00b7290 0x7f54a00b76a0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.419+0000 7f54a7438640 1 --2- 172.21.15.81:0/903450248 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54a00b5250 0x7f54a00d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.419+0000 7f54a7438640 1 --2- 172.21.15.81:0/903450248 >> v2:172.21.15.167:3300/0 conn(0x7f54a00b9040 0x7f54a00d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.419+0000 7f54a7438640 1 -- 172.21.15.81:0/903450248 >> 172.21.15.81:0/903450248 conn(0x7f54a009cf30 msgr2=0x7f54a00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.419+0000 7f54a7438640 1 -- 172.21.15.81:0/903450248 shutdown_connections 2024-06-29T15:38:55.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.419+0000 7f54a7438640 1 -- 172.21.15.81:0/903450248 wait complete. 2024-06-29T15:38:55.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.421+0000 7f54a7438640 1 Processor -- start 2024-06-29T15:38:55.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.421+0000 7f54a7438640 1 -- start start 2024-06-29T15:38:55.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a7438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54a00b5250 0x7f54a01f7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a7438640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54a00b7290 0x7f54a009f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a7438640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54a00b9040 0x7f54a009feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a7438640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f54a00b7f70 con 0x7f54a00b5250 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a7438640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f54a00b7df0 con 0x7f54a00b9040 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a7438640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f54a00b80f0 con 0x7f54a00b7290 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a51d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54a00b5250 0x7f54a01f7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a51d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54a00b5250 0x7f54a01f7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60430/0 (socket says 172.21.15.81:60430) 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a51d4640 1 -- 172.21.15.81:0/463882311 learned_addr learned my addr 172.21.15.81:0/463882311 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a49d3640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54a00b7290 0x7f54a009f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a59d5640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54a00b9040 0x7f54a009feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a51d4640 1 -- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54a00b7290 msgr2=0x7f54a009f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a51d4640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54a00b7290 0x7f54a009f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a51d4640 1 -- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54a00b9040 msgr2=0x7f54a009feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a51d4640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54a00b9040 0x7f54a009feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a51d4640 1 -- 172.21.15.81:0/463882311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a00c7ef0 con 0x7f54a00b5250 2024-06-29T15:38:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a51d4640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54a00b5250 0x7f54a01f7f00 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f549000e790 tx=0x7f549000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:55.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f5495ffb640 1 -- 172.21.15.81:0/463882311 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549000eea0 con 0x7f54a00b5250 2024-06-29T15:38:55.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f5495ffb640 1 -- 172.21.15.81:0/463882311 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f549000cdb0 con 0x7f54a00b5250 2024-06-29T15:38:55.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f54a7438640 1 -- 172.21.15.81:0/463882311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a00c81b0 con 0x7f54a00b5250 2024-06-29T15:38:55.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.423+0000 7f5495ffb640 1 -- 172.21.15.81:0/463882311 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490010690 con 0x7f54a00b5250 2024-06-29T15:38:55.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.424+0000 7f54a7438640 1 -- 172.21.15.81:0/463882311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a00c8770 con 0x7f54a00b5250 2024-06-29T15:38:55.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.425+0000 7f54a7438640 1 -- 172.21.15.81:0/463882311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a00b76a0 con 0x7f54a00b5250 2024-06-29T15:38:55.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.425+0000 7f5495ffb640 1 -- 172.21.15.81:0/463882311 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7f5490010830 con 0x7f54a00b5250 2024-06-29T15:38:55.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.426+0000 7f5495ffb640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f54840763f0 0x7f5484078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.426+0000 7f54a49d3640 1 -- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f54840763f0 msgr2=0x7f5484078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:55.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.426+0000 7f5495ffb640 1 -- 172.21.15.81:0/463882311 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7f5490098b60 con 0x7f54a00b5250 2024-06-29T15:38:55.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.426+0000 7f54a49d3640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f54840763f0 0x7f5484078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.434+0000 7f5495ffb640 1 -- 172.21.15.81:0/463882311 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5490061b70 con 0x7f54a00b5250 2024-06-29T15:38:55.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.616+0000 7f54a7438640 1 -- 172.21.15.81:0/463882311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f54a00ca7c0 con 0x7f54a00b5250 2024-06-29T15:38:55.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.616+0000 7f5495ffb640 1 -- 172.21.15.81:0/463882311 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v169) ==== 100+0+11 (secure 0 0 0) 0x7f5490065e70 con 0x7f54a00b5250 2024-06-29T15:38:55.617 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:38:55.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 -- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f54840763f0 msgr2=0x7f5484078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:55.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f54840763f0 0x7f5484078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 -- 172.21.15.81:0/463882311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54a00b5250 msgr2=0x7f54a01f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54a00b5250 0x7f54a01f7f00 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f549000e790 tx=0x7f549000ec50 comp rx=0 tx=0).stop 2024-06-29T15:38:55.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 -- 172.21.15.81:0/463882311 shutdown_connections 2024-06-29T15:38:55.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54a00b9040 0x7f54a009feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54a00b7290 0x7f54a009f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f54840763f0 0x7f5484078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 --2- 172.21.15.81:0/463882311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54a00b5250 0x7f54a01f7f00 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 -- 172.21.15.81:0/463882311 >> 172.21.15.81:0/463882311 conn(0x7f54a009cf30 msgr2=0x7f54a015f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 -- 172.21.15.81:0/463882311 shutdown_connections 2024-06-29T15:38:55.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.618+0000 7f54a7438640 1 -- 172.21.15.81:0/463882311 wait complete. 2024-06-29T15:38:55.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:38:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.746+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/4026779387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f180b3200 msgr2=0x7f8f180cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.746+0000 7f8f1cfe0640 1 --2- 172.21.15.81:0/4026779387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f180b3200 0x7f8f180cc7b0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f8f08005fd0 tx=0x7f8f0802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.747+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/4026779387 shutdown_connections 2024-06-29T15:38:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.747+0000 7f8f1cfe0640 1 --2- 172.21.15.81:0/4026779387 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f180b3200 0x7f8f180cc7b0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.747+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/4026779387 >> 172.21.15.81:0/4026779387 conn(0x7f8f180b3680 msgr2=0x7f8f180b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.747+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/4026779387 shutdown_connections 2024-06-29T15:38:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.747+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/4026779387 wait complete. 2024-06-29T15:38:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.750+0000 7f8f1cfe0640 1 Processor -- start 2024-06-29T15:38:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.750+0000 7f8f1cfe0640 1 -- start start 2024-06-29T15:38:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1cfe0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f180b3200 0x7f8f181562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1cfe0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8f180cccf0 0x7f8f181664a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1cfe0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f180c6b40 0x7f8f180c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1cfe0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8f180b5e50 con 0x7f8f180b3200 2024-06-29T15:38:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1cfe0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8f180b5cd0 con 0x7f8f180cccf0 2024-06-29T15:38:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1cfe0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8f180b5fd0 con 0x7f8f180c6b40 2024-06-29T15:38:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f180b3200 0x7f8f181562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f180b3200 0x7f8f181562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60438/0 (socket says 172.21.15.81:60438) 2024-06-29T15:38:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1659c640 1 -- 172.21.15.81:0/3936801680 learned_addr learned my addr 172.21.15.81:0/3936801680 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f15d9b640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8f180cccf0 0x7f8f181664a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f16d9d640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f180c6b40 0x7f8f180c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1659c640 1 -- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f180c6b40 msgr2=0x7f8f180c6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1659c640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f180c6b40 0x7f8f180c6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1659c640 1 -- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8f180cccf0 msgr2=0x7f8f181664a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1659c640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8f180cccf0 0x7f8f181664a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f1659c640 1 -- 172.21.15.81:0/3936801680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f18165400 con 0x7f8f180b3200 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.752+0000 7f8f16d9d640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f180c6b40 0x7f8f180c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.753+0000 7f8f1659c640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f180b3200 0x7f8f181562a0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f8f08034040 tx=0x7f8f08003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.753+0000 7f8f0effd640 1 -- 172.21.15.81:0/3936801680 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f0803e030 con 0x7f8f180b3200 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.753+0000 7f8f0effd640 1 -- 172.21.15.81:0/3936801680 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f080047a0 con 0x7f8f180b3200 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.753+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/3936801680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f181656e0 con 0x7f8f180b3200 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.753+0000 7f8f0effd640 1 -- 172.21.15.81:0/3936801680 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08035040 con 0x7f8f180b3200 2024-06-29T15:38:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.753+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/3936801680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f18165ca0 con 0x7f8f180b3200 2024-06-29T15:38:55.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.754+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/3936801680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f180cc7b0 con 0x7f8f180b3200 2024-06-29T15:38:55.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.755+0000 7f8f0effd640 1 -- 172.21.15.81:0/3936801680 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7f8f08041030 con 0x7f8f180b3200 2024-06-29T15:38:55.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.755+0000 7f8f0effd640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f8ef80763f0 0x7f8ef8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:55.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.755+0000 7f8f15d9b640 1 -- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f8ef80763f0 msgr2=0x7f8ef8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:55.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.755+0000 7f8f0effd640 1 -- 172.21.15.81:0/3936801680 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7f8f080b85a0 con 0x7f8f180b3200 2024-06-29T15:38:55.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.755+0000 7f8f15d9b640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f8ef80763f0 0x7f8ef8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:55.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.763+0000 7f8f0effd640 1 -- 172.21.15.81:0/3936801680 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8f08081640 con 0x7f8f180b3200 2024-06-29T15:38:55.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.956+0000 7f8f15d9b640 1 -- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f8ef80763f0 msgr2=0x7f8ef8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:55.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.956+0000 7f8f15d9b640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f8ef80763f0 0x7f8ef8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:55.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.958+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/3936801680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8f18163990 con 0x7f8f180b3200 2024-06-29T15:38:55.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.958+0000 7f8f0effd640 1 -- 172.21.15.81:0/3936801680 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v169) ==== 80+0+476 (secure 0 0 0) 0x7f8f08085940 con 0x7f8f180b3200 2024-06-29T15:38:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1753956564 2024-06-30T15:38:51.005371+0000 2024-06-29T15:38:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1949204156 2024-06-30T15:38:51.005371+0000 2024-06-29T15:38:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/4139817389 2024-06-30T15:38:51.005371+0000 2024-06-29T15:38:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3283653249 2024-06-30T15:38:51.005371+0000 2024-06-29T15:38:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2505002611 2024-06-30T15:38:51.005371+0000 2024-06-29T15:38:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/4205626971 2024-06-30T15:38:51.005371+0000 2024-06-29T15:38:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/150801987 2024-06-30T15:38:51.005371+0000 2024-06-29T15:38:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/150801987 2024-06-30T15:38:51.005371+0000 2024-06-29T15:38:55.960 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T15:38:55.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.960+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f8ef80763f0 msgr2=0x7f8ef8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:55.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.960+0000 7f8f1cfe0640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f8ef80763f0 0x7f8ef8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.960+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/3936801680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f180b3200 msgr2=0x7f8f181562a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:55.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.960+0000 7f8f1cfe0640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f180b3200 0x7f8f181562a0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f8f08034040 tx=0x7f8f08003970 comp rx=0 tx=0).stop 2024-06-29T15:38:55.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.960+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/3936801680 shutdown_connections 2024-06-29T15:38:55.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.960+0000 7f8f1cfe0640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f180c6b40 0x7f8f180c6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.960+0000 7f8f1cfe0640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8f180cccf0 0x7f8f181664a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.960+0000 7f8f1cfe0640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f8ef80763f0 0x7f8ef8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.960+0000 7f8f1cfe0640 1 --2- 172.21.15.81:0/3936801680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f180b3200 0x7f8f181562a0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:55.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.961+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/3936801680 >> 172.21.15.81:0/3936801680 conn(0x7f8f180b3680 msgr2=0x7f8f180ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:55.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.961+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/3936801680 shutdown_connections 2024-06-29T15:38:55.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:55.961+0000 7f8f1cfe0640 1 -- 172.21.15.81:0/3936801680 wait complete. 2024-06-29T15:38:55.972 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:38:55.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:55.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:38:56.002 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T15:38:56.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:38:56.002 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:38:56.057 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:56.057 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:38:56.058 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:56.073 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:56.073 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:56.073 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:38:56.073 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:56.113 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:38:56.113 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:38:56.113 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:38:56.113 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:38:56.129 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:38:56.129 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:56.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.268+0000 7fe37ba38640 1 -- 172.21.15.81:0/2308747054 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3740b3200 msgr2=0x7fe3740cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.268+0000 7fe37ba38640 1 --2- 172.21.15.81:0/2308747054 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3740b3200 0x7fe3740cc7b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe370009830 tx=0x7fe37002a4b0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.268+0000 7fe37ba38640 1 -- 172.21.15.81:0/2308747054 shutdown_connections 2024-06-29T15:38:56.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.268+0000 7fe37ba38640 1 --2- 172.21.15.81:0/2308747054 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3740b3200 0x7fe3740cc7b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.268+0000 7fe37ba38640 1 -- 172.21.15.81:0/2308747054 >> 172.21.15.81:0/2308747054 conn(0x7fe3740b3680 msgr2=0x7fe3740b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:56.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.268+0000 7fe37ba38640 1 -- 172.21.15.81:0/2308747054 shutdown_connections 2024-06-29T15:38:56.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.268+0000 7fe37ba38640 1 -- 172.21.15.81:0/2308747054 wait complete. 2024-06-29T15:38:56.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.270+0000 7fe37ba38640 1 Processor -- start 2024-06-29T15:38:56.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.270+0000 7fe37ba38640 1 -- start start 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe37ba38640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe3740b3200 0x7fe37415c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe37ba38640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3740cccf0 0x7fe374161850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe37ba38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe374166fa0 0x7fe374161d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe37ba38640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3740b5e50 con 0x7fe374166fa0 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe37ba38640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe3740b5cd0 con 0x7fe3740cccf0 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe37ba38640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe3740b5fd0 con 0x7fe3740b3200 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe379fd5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe374166fa0 0x7fe374161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe379fd5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe374166fa0 0x7fe374161d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60450/0 (socket says 172.21.15.81:60450) 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe379fd5640 1 -- 172.21.15.81:0/152196135 learned_addr learned my addr 172.21.15.81:0/152196135 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe3797d4640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe3740b3200 0x7fe37415c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe378fd3640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3740cccf0 0x7fe374161850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe379fd5640 1 -- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe3740b3200 msgr2=0x7fe37415c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe379fd5640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe3740b3200 0x7fe37415c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.271+0000 7fe379fd5640 1 -- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3740cccf0 msgr2=0x7fe374161850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.272+0000 7fe379fd5640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3740cccf0 0x7fe374161850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.272+0000 7fe379fd5640 1 -- 172.21.15.81:0/152196135 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37415c5b0 con 0x7fe374166fa0 2024-06-29T15:38:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.272+0000 7fe379fd5640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe374166fa0 0x7fe374161d90 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00c700 tx=0x7fe36c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.272+0000 7fe369ffb640 1 -- 172.21.15.81:0/152196135 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c003b70 con 0x7fe374166fa0 2024-06-29T15:38:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.272+0000 7fe37ba38640 1 -- 172.21.15.81:0/152196135 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37409dff0 con 0x7fe374166fa0 2024-06-29T15:38:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.272+0000 7fe369ffb640 1 -- 172.21.15.81:0/152196135 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe36c003d10 con 0x7fe374166fa0 2024-06-29T15:38:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.272+0000 7fe369ffb640 1 -- 172.21.15.81:0/152196135 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c005330 con 0x7fe374166fa0 2024-06-29T15:38:56.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.272+0000 7fe37ba38640 1 -- 172.21.15.81:0/152196135 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe37409e5b0 con 0x7fe374166fa0 2024-06-29T15:38:56.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.273+0000 7fe37ba38640 1 -- 172.21.15.81:0/152196135 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3740cc7b0 con 0x7fe374166fa0 2024-06-29T15:38:56.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.274+0000 7fe369ffb640 1 -- 172.21.15.81:0/152196135 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7fe36c0054d0 con 0x7fe374166fa0 2024-06-29T15:38:56.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.274+0000 7fe369ffb640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fe3580763f0 0x7fe358078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.274+0000 7fe3797d4640 1 -- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fe3580763f0 msgr2=0x7fe358078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:56.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.274+0000 7fe369ffb640 1 -- 172.21.15.81:0/152196135 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7fe36c097800 con 0x7fe374166fa0 2024-06-29T15:38:56.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.274+0000 7fe3797d4640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fe3580763f0 0x7fe358078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:56.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.278+0000 7fe369ffb640 1 -- 172.21.15.81:0/152196135 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe36c0607b0 con 0x7fe374166fa0 2024-06-29T15:38:56.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.476+0000 7fe3797d4640 1 -- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fe3580763f0 msgr2=0x7fe358078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:56.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.476+0000 7fe3797d4640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fe3580763f0 0x7fe358078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:38:56.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.506+0000 7fe37ba38640 1 -- 172.21.15.81:0/152196135 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3740705b0 con 0x7fe374166fa0 2024-06-29T15:38:56.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.506+0000 7fe369ffb640 1 -- 172.21.15.81:0/152196135 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5233 (secure 0 0 0) 0x7fe36c064ab0 con 0x7fe374166fa0 2024-06-29T15:38:56.507 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:56.507 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":86,"btime":"2024-06-29T15:38:42:386002+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7754,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3261744821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3261744821},{"type":"v1","addr":"172.21.15.167:6836","nonce":3261744821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7757,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1617521107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1617521107},{"type":"v1","addr":"172.21.15.167:6837","nonce":1617521107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":9636,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3224768380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3224768380},{"type":"v1","addr":"172.21.15.81:6833","nonce":3224768380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.731697+0000","modified":"2024-06-29T15:38:41.749109+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9639":{"gid":9639,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/1634290596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1634290596},{"type":"v1","addr":"172.21.15.81:6835","nonce":1634290596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9639,"qdb_cluster":[9639]},"id":10}]} 2024-06-29T15:38:56.507 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 86 2024-06-29T15:38:56.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 -- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fe3580763f0 msgr2=0x7fe358078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fe3580763f0 0x7fe358078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 -- 172.21.15.81:0/152196135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe374166fa0 msgr2=0x7fe374161d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe374166fa0 0x7fe374161d90 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fe36c00c700 tx=0x7fe36c00cbc0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 -- 172.21.15.81:0/152196135 shutdown_connections 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fe3580763f0 0x7fe358078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe374166fa0 0x7fe374161d90 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3740cccf0 0x7fe374161850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 --2- 172.21.15.81:0/152196135 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe3740b3200 0x7fe37415c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 -- 172.21.15.81:0/152196135 >> 172.21.15.81:0/152196135 conn(0x7fe3740b3680 msgr2=0x7fe3740d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 -- 172.21.15.81:0/152196135 shutdown_connections 2024-06-29T15:38:56.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.509+0000 7fe37ba38640 1 -- 172.21.15.81:0/152196135 wait complete. 2024-06-29T15:38:56.521 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.634+0000 7f5ea0703640 1 -- 172.21.15.81:0/935774422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e980b3200 msgr2=0x7f5e980cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.634+0000 7f5ea0703640 1 --2- 172.21.15.81:0/935774422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e980b3200 0x7f5e980cc7b0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f5e980c5590 tx=0x7f5e8c02a4a0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.634+0000 7f5ea0703640 1 -- 172.21.15.81:0/935774422 shutdown_connections 2024-06-29T15:38:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.634+0000 7f5ea0703640 1 --2- 172.21.15.81:0/935774422 >> v2:172.21.15.167:3300/0 conn(0x7f5e980d0940 0x7f5e980b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.634+0000 7f5ea0703640 1 --2- 172.21.15.81:0/935774422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e980b3200 0x7f5e980cc7b0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.634+0000 7f5ea0703640 1 -- 172.21.15.81:0/935774422 >> 172.21.15.81:0/935774422 conn(0x7f5e980b3680 msgr2=0x7f5e980b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.634+0000 7f5ea0703640 1 -- 172.21.15.81:0/935774422 shutdown_connections 2024-06-29T15:38:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.634+0000 7f5ea0703640 1 -- 172.21.15.81:0/935774422 wait complete. 2024-06-29T15:38:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.636+0000 7f5ea0703640 1 Processor -- start 2024-06-29T15:38:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.636+0000 7f5ea0703640 1 -- start start 2024-06-29T15:38:56.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5ea0703640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e980b3200 0x7f5e981f8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5ea0703640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e980cccf0 0x7f5e981f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5ea0703640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e981d6230 0x7f5e980ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5ea0703640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5e980b5bb0 con 0x7f5e980cccf0 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5ea0703640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5e980b5a30 con 0x7f5e981d6230 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5ea0703640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5e980b5d30 con 0x7f5e980b3200 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5e9dc9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e980cccf0 0x7f5e981f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5e9dc9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e980cccf0 0x7f5e981f4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60470/0 (socket says 172.21.15.81:60470) 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5e9dc9e640 1 -- 172.21.15.81:0/222369906 learned_addr learned my addr 172.21.15.81:0/222369906 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5e9e49f640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e980b3200 0x7f5e981f8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5e9eca0640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e981d6230 0x7f5e980ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5e9dc9e640 1 -- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e980b3200 msgr2=0x7f5e981f8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5e9dc9e640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e980b3200 0x7f5e981f8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5e9dc9e640 1 -- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e981d6230 msgr2=0x7f5e980ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.637+0000 7f5e9dc9e640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e981d6230 0x7f5e980ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.638+0000 7f5e9dc9e640 1 -- 172.21.15.81:0/222369906 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e980c8150 con 0x7f5e980cccf0 2024-06-29T15:38:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.638+0000 7f5e9dc9e640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e980cccf0 0x7f5e981f4320 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400c3a0 tx=0x7f5e9400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.638+0000 7f5e8affd640 1 -- 172.21.15.81:0/222369906 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94011790 con 0x7f5e980cccf0 2024-06-29T15:38:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.638+0000 7f5ea0703640 1 -- 172.21.15.81:0/222369906 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e980c8430 con 0x7f5e980cccf0 2024-06-29T15:38:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.638+0000 7f5e8affd640 1 -- 172.21.15.81:0/222369906 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e94017070 con 0x7f5e980cccf0 2024-06-29T15:38:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.638+0000 7f5e8affd640 1 -- 172.21.15.81:0/222369906 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94011c80 con 0x7f5e980cccf0 2024-06-29T15:38:56.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.638+0000 7f5ea0703640 1 -- 172.21.15.81:0/222369906 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e981d27f0 con 0x7f5e980cccf0 2024-06-29T15:38:56.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.639+0000 7f5ea0703640 1 -- 172.21.15.81:0/222369906 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e980cc7b0 con 0x7f5e980cccf0 2024-06-29T15:38:56.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.640+0000 7f5e8affd640 1 -- 172.21.15.81:0/222369906 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7f5e9401c460 con 0x7f5e980cccf0 2024-06-29T15:38:56.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.641+0000 7f5e8affd640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e7c076420 0x7f5e7c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.641+0000 7f5e9e49f640 1 -- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e7c076420 msgr2=0x7f5e7c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:56.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.641+0000 7f5e8affd640 1 -- 172.21.15.81:0/222369906 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7f5e94097780 con 0x7f5e980cccf0 2024-06-29T15:38:56.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.641+0000 7f5e9e49f640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e7c076420 0x7f5e7c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:56.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.649+0000 7f5e8affd640 1 -- 172.21.15.81:0/222369906 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5e94060740 con 0x7f5e980cccf0 2024-06-29T15:38:56.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.833+0000 7f5ea0703640 1 -- 172.21.15.81:0/222369906 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e980cf8a0 con 0x7f5e980cccf0 2024-06-29T15:38:56.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.834+0000 7f5e8affd640 1 -- 172.21.15.81:0/222369906 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) ==== 74+0+17333 (secure 0 0 0) 0x7f5e94064a40 con 0x7f5e980cccf0 2024-06-29T15:38:56.838 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:56.839 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":169,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:51.005408+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:37.031836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.676381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/1753956564":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/1949204156":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/4139817389":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/3283653249":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/2505002611":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/4205626971":"2024-06-30T15:38:51.005371+0000","172.21.15.167:6833/150801987":"2024-06-30T15:38:51.005371+0000","172.21.15.167:6832/150801987":"2024-06-30T15:38:51.005371+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:56.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 -- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e7c076420 msgr2=0x7f5e7c078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:56.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e7c076420 0x7f5e7c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 -- 172.21.15.81:0/222369906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e980cccf0 msgr2=0x7f5e981f4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e980cccf0 0x7f5e981f4320 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400c3a0 tx=0x7f5e9400c860 comp rx=0 tx=0).stop 2024-06-29T15:38:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 -- 172.21.15.81:0/222369906 shutdown_connections 2024-06-29T15:38:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e981d6230 0x7f5e980ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f5e7c076420 0x7f5e7c078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e980cccf0 0x7f5e981f4320 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 --2- 172.21.15.81:0/222369906 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e980b3200 0x7f5e981f8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 -- 172.21.15.81:0/222369906 >> 172.21.15.81:0/222369906 conn(0x7f5e980b3680 msgr2=0x7f5e98070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 -- 172.21.15.81:0/222369906 shutdown_connections 2024-06-29T15:38:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.837+0000 7f5ea0703640 1 -- 172.21.15.81:0/222369906 wait complete. 2024-06-29T15:38:56.849 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:38:37.031836+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '167', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:38:37.676381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '168', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:38:56.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:38:56.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.966+0000 7fd1f1047640 1 -- 172.21.15.81:0/4154205518 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1ec0d0230 msgr2=0x7fd1ec0d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.966+0000 7fd1f1047640 1 --2- 172.21.15.81:0/4154205518 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1ec0d0230 0x7fd1ec0d0640 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4009830 tx=0x7fd1d402a530 comp rx=0 tx=0).stop 2024-06-29T15:38:56.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.966+0000 7fd1f1047640 1 -- 172.21.15.81:0/4154205518 shutdown_connections 2024-06-29T15:38:56.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.966+0000 7fd1f1047640 1 --2- 172.21.15.81:0/4154205518 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1ec0d0230 0x7fd1ec0d0640 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.966+0000 7fd1f1047640 1 -- 172.21.15.81:0/4154205518 >> 172.21.15.81:0/4154205518 conn(0x7fd1ec0b3680 msgr2=0x7fd1ec0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:56.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.966+0000 7fd1f1047640 1 -- 172.21.15.81:0/4154205518 shutdown_connections 2024-06-29T15:38:56.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.966+0000 7fd1f1047640 1 -- 172.21.15.81:0/4154205518 wait complete. 2024-06-29T15:38:56.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.968+0000 7fd1f1047640 1 Processor -- start 2024-06-29T15:38:56.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.968+0000 7fd1f1047640 1 -- start start 2024-06-29T15:38:56.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1f1047640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1ec0b3200 0x7fd1ec14b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1f1047640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1ec0b4f40 0x7fd1ec14e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1f1047640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1ec0d0230 0x7fd1ec09e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1f1047640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec0b5e50 con 0x7fd1ec0d0230 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1f1047640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec0b5cd0 con 0x7fd1ec0b4f40 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1f1047640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd1ec0b5fd0 con 0x7fd1ec0b3200 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1eb59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1ec0d0230 0x7fd1ec09e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1ea59c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1ec0b4f40 0x7fd1ec14e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1ead9d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1ec0b3200 0x7fd1ec14b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1eb59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1ec0d0230 0x7fd1ec09e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60474/0 (socket says 172.21.15.81:60474) 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1eb59e640 1 -- 172.21.15.81:0/3968404461 learned_addr learned my addr 172.21.15.81:0/3968404461 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1ea59c640 1 -- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1ec0b3200 msgr2=0x7fd1ec14b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1ea59c640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1ec0b3200 0x7fd1ec14b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1ea59c640 1 -- 172.21.15.81:0/3968404461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1ec0d0230 msgr2=0x7fd1ec09e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1ea59c640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1ec0d0230 0x7fd1ec09e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1ea59c640 1 -- 172.21.15.81:0/3968404461 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ec14f050 con 0x7fd1ec0b4f40 2024-06-29T15:38:56.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.970+0000 7fd1ea59c640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1ec0b4f40 0x7fd1ec14e5f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4031000 tx=0x7fd1d4033ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:56.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.971+0000 7fd1e37fe640 1 -- 172.21.15.81:0/3968404461 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4031b50 con 0x7fd1ec0b4f40 2024-06-29T15:38:56.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.971+0000 7fd1e37fe640 1 -- 172.21.15.81:0/3968404461 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1d4031de0 con 0x7fd1ec0b4f40 2024-06-29T15:38:56.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.971+0000 7fd1f1047640 1 -- 172.21.15.81:0/3968404461 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1ec14c820 con 0x7fd1ec0b4f40 2024-06-29T15:38:56.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.971+0000 7fd1e37fe640 1 -- 172.21.15.81:0/3968404461 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4005280 con 0x7fd1ec0b4f40 2024-06-29T15:38:56.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.971+0000 7fd1f1047640 1 -- 172.21.15.81:0/3968404461 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1ec14cde0 con 0x7fd1ec0b4f40 2024-06-29T15:38:56.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.972+0000 7fd1f1047640 1 -- 172.21.15.81:0/3968404461 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1ec0d0640 con 0x7fd1ec0b4f40 2024-06-29T15:38:56.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.973+0000 7fd1e37fe640 1 -- 172.21.15.81:0/3968404461 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7fd1d4005420 con 0x7fd1ec0b4f40 2024-06-29T15:38:56.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.973+0000 7fd1e37fe640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fd1cc0763f0 0x7fd1cc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:56.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.974+0000 7fd1ead9d640 1 -- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fd1cc0763f0 msgr2=0x7fd1cc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:56.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.974+0000 7fd1e37fe640 1 -- 172.21.15.81:0/3968404461 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7fd1d40b8eb0 con 0x7fd1ec0b4f40 2024-06-29T15:38:56.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.974+0000 7fd1ead9d640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fd1cc0763f0 0x7fd1cc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:56.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:56.978+0000 7fd1e37fe640 1 -- 172.21.15.81:0/3968404461 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd1d403f030 con 0x7fd1ec0b4f40 2024-06-29T15:38:57.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.164+0000 7fd1f1047640 1 -- 172.21.15.81:0/3968404461 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1ec0cb5f0 con 0x7fd1ec0b4f40 2024-06-29T15:38:57.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.164+0000 7fd1e37fe640 1 -- 172.21.15.81:0/3968404461 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v169) ==== 93+0+10 (secure 0 0 0) 0x7fd1d4031770 con 0x7fd1ec0b4f40 2024-06-29T15:38:57.165 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:38:57.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 -- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fd1cc0763f0 msgr2=0x7fd1cc078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:57.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fd1cc0763f0 0x7fd1cc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 -- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1ec0b4f40 msgr2=0x7fd1ec14e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1ec0b4f40 0x7fd1ec14e5f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4031000 tx=0x7fd1d4033ba0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 -- 172.21.15.81:0/3968404461 shutdown_connections 2024-06-29T15:38:57.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fd1cc0763f0 0x7fd1cc078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1ec0d0230 0x7fd1ec09e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1ec0b4f40 0x7fd1ec14e5f0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 --2- 172.21.15.81:0/3968404461 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1ec0b3200 0x7fd1ec14b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 -- 172.21.15.81:0/3968404461 >> 172.21.15.81:0/3968404461 conn(0x7fd1ec0b3680 msgr2=0x7fd1ec0cb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 -- 172.21.15.81:0/3968404461 shutdown_connections 2024-06-29T15:38:57.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.166+0000 7fd1f1047640 1 -- 172.21.15.81:0/3968404461 wait complete. 2024-06-29T15:38:57.178 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:38:57.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.297+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/125059620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd140d0940 msgr2=0x7fbd140b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.297+0000 7fbd1c1c7640 1 --2- 172.21.15.81:0/125059620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd140d0940 0x7fbd140b4f40 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c005fd0 tx=0x7fbd0c02aa90 comp rx=0 tx=0).stop 2024-06-29T15:38:57.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.297+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/125059620 shutdown_connections 2024-06-29T15:38:57.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.297+0000 7fbd1c1c7640 1 --2- 172.21.15.81:0/125059620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd140d0940 0x7fbd140b4f40 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.297+0000 7fbd1c1c7640 1 --2- 172.21.15.81:0/125059620 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd140b3200 0x7fbd140cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.297+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/125059620 >> 172.21.15.81:0/125059620 conn(0x7fbd140b3680 msgr2=0x7fbd140b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.297+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/125059620 shutdown_connections 2024-06-29T15:38:57.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.297+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/125059620 wait complete. 2024-06-29T15:38:57.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.299+0000 7fbd1c1c7640 1 Processor -- start 2024-06-29T15:38:57.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.299+0000 7fbd1c1c7640 1 -- start start 2024-06-29T15:38:57.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd1c1c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd140b3200 0x7fbd140af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd1c1c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd140cccf0 0x7fbd141540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd1c1c7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd141f6b30 0x7fbd141f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd1c1c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbd140b5cb0 con 0x7fbd140cccf0 2024-06-29T15:38:57.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd1c1c7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbd140b5b30 con 0x7fbd140b3200 2024-06-29T15:38:57.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd1c1c7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbd140b5e30 con 0x7fbd141f6b30 2024-06-29T15:38:57.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd19762640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd140cccf0 0x7fbd141540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd19762640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd140cccf0 0x7fbd141540f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60490/0 (socket says 172.21.15.81:60490) 2024-06-29T15:38:57.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd19762640 1 -- 172.21.15.81:0/2863114493 learned_addr learned my addr 172.21.15.81:0/2863114493 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:57.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd19f63640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd140b3200 0x7fbd140af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd1a764640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd141f6b30 0x7fbd141f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd19762640 1 -- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd141f6b30 msgr2=0x7fbd141f5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd19762640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd141f6b30 0x7fbd141f5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd19762640 1 -- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd140b3200 msgr2=0x7fbd140af370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd19762640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd140b3200 0x7fbd140af370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd19762640 1 -- 172.21.15.81:0/2863114493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd140b1af0 con 0x7fbd140cccf0 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.300+0000 7fbd19762640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd140cccf0 0x7fbd141540f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000c3a0 tx=0x7fbd1000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.301+0000 7fbd067fc640 1 -- 172.21.15.81:0/2863114493 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd10017070 con 0x7fbd140cccf0 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.301+0000 7fbd067fc640 1 -- 172.21.15.81:0/2863114493 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd100053c0 con 0x7fbd140cccf0 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.301+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/2863114493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd140b1dd0 con 0x7fbd140cccf0 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.301+0000 7fbd067fc640 1 -- 172.21.15.81:0/2863114493 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd10003850 con 0x7fbd140cccf0 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.301+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/2863114493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd140b2390 con 0x7fbd140cccf0 2024-06-29T15:38:57.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.302+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/2863114493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd140cc7b0 con 0x7fbd140cccf0 2024-06-29T15:38:57.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.303+0000 7fbd067fc640 1 -- 172.21.15.81:0/2863114493 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7fbd1001a070 con 0x7fbd140cccf0 2024-06-29T15:38:57.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.303+0000 7fbd067fc640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fbcf8076420 0x7fbcf8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.303+0000 7fbd067fc640 1 -- 172.21.15.81:0/2863114493 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7fbd10097210 con 0x7fbd140cccf0 2024-06-29T15:38:57.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.303+0000 7fbd19f63640 1 -- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fbcf8076420 msgr2=0x7fbcf8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:57.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.303+0000 7fbd19f63640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fbcf8076420 0x7fbcf8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:57.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.311+0000 7fbd067fc640 1 -- 172.21.15.81:0/2863114493 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbd10021020 con 0x7fbd140cccf0 2024-06-29T15:38:57.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.494+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/2863114493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbd140b1800 con 0x7fbd140cccf0 2024-06-29T15:38:57.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.494+0000 7fbd067fc640 1 -- 172.21.15.81:0/2863114493 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v169) ==== 104+0+11 (secure 0 0 0) 0x7fbd100601c0 con 0x7fbd140cccf0 2024-06-29T15:38:57.495 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:38:57.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fbcf8076420 msgr2=0x7fbcf8078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:57.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fbcf8076420 0x7fbcf8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/2863114493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd140cccf0 msgr2=0x7fbd141540f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd140cccf0 0x7fbd141540f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000c3a0 tx=0x7fbd1000c860 comp rx=0 tx=0).stop 2024-06-29T15:38:57.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/2863114493 shutdown_connections 2024-06-29T15:38:57.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd141f6b30 0x7fbd141f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7fbcf8076420 0x7fbcf8078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd140cccf0 0x7fbd141540f0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 --2- 172.21.15.81:0/2863114493 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd140b3200 0x7fbd140af370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/2863114493 >> 172.21.15.81:0/2863114493 conn(0x7fbd140b3680 msgr2=0x7fbd14155e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/2863114493 shutdown_connections 2024-06-29T15:38:57.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.496+0000 7fbd1c1c7640 1 -- 172.21.15.81:0/2863114493 wait complete. 2024-06-29T15:38:57.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:38:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.624+0000 7f7903ae4640 1 -- 172.21.15.81:0/4192935562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78fc073480 msgr2=0x7f78fc15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.624+0000 7f7903ae4640 1 --2- 172.21.15.81:0/4192935562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78fc073480 0x7f78fc15b6e0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f78fc072e80 tx=0x7f78f802a4d0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.625+0000 7f7903ae4640 1 -- 172.21.15.81:0/4192935562 shutdown_connections 2024-06-29T15:38:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.625+0000 7f7903ae4640 1 --2- 172.21.15.81:0/4192935562 >> v2:172.21.15.167:3300/0 conn(0x7f78fc15bc20 0x7f78fc15e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.625+0000 7f7903ae4640 1 --2- 172.21.15.81:0/4192935562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78fc073480 0x7f78fc15b6e0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.625+0000 7f7903ae4640 1 --2- 172.21.15.81:0/4192935562 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78fc0cd150 0x7f78fc072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.625+0000 7f7903ae4640 1 -- 172.21.15.81:0/4192935562 >> 172.21.15.81:0/4192935562 conn(0x7f78fc0b2920 msgr2=0x7f78fc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.625+0000 7f7903ae4640 1 -- 172.21.15.81:0/4192935562 shutdown_connections 2024-06-29T15:38:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.625+0000 7f7903ae4640 1 -- 172.21.15.81:0/4192935562 wait complete. 2024-06-29T15:38:57.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.627+0000 7f7903ae4640 1 Processor -- start 2024-06-29T15:38:57.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.628+0000 7f7903ae4640 1 -- start start 2024-06-29T15:38:57.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7903ae4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78fc073480 0x7f78fc0ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7903ae4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78fc0cd150 0x7f78fc0d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7903ae4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78fc15bc20 0x7f78fc1511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7903ae4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f78fc15f010 con 0x7f78fc073480 2024-06-29T15:38:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7903ae4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f78fc15ee90 con 0x7f78fc15bc20 2024-06-29T15:38:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7903ae4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f78fc15f190 con 0x7f78fc0cd150 2024-06-29T15:38:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7901880640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78fc073480 0x7f78fc0ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7901880640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78fc073480 0x7f78fc0ce6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60508/0 (socket says 172.21.15.81:60508) 2024-06-29T15:38:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7901880640 1 -- 172.21.15.81:0/1443276918 learned_addr learned my addr 172.21.15.81:0/1443276918 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f790107f640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78fc0cd150 0x7f78fc0d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7902081640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78fc15bc20 0x7f78fc1511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7901880640 1 -- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78fc0cd150 msgr2=0x7f78fc0d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7901880640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78fc0cd150 0x7f78fc0d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7901880640 1 -- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78fc15bc20 msgr2=0x7f78fc1511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7901880640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78fc15bc20 0x7f78fc1511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7901880640 1 -- 172.21.15.81:0/1443276918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78fc142520 con 0x7f78fc073480 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.630+0000 7f7901880640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78fc073480 0x7f78fc0ce6d0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f78ec00e7c0 tx=0x7f78ec00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.631+0000 7f78f27fc640 1 -- 172.21.15.81:0/1443276918 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec00ccc0 con 0x7f78fc073480 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.631+0000 7f78f27fc640 1 -- 172.21.15.81:0/1443276918 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78ec00ce60 con 0x7f78fc073480 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.631+0000 7f7903ae4640 1 -- 172.21.15.81:0/1443276918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78fc142a20 con 0x7f78fc073480 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.631+0000 7f78f27fc640 1 -- 172.21.15.81:0/1443276918 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec005280 con 0x7f78fc073480 2024-06-29T15:38:57.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.631+0000 7f7903ae4640 1 -- 172.21.15.81:0/1443276918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78fc142fe0 con 0x7f78fc073480 2024-06-29T15:38:57.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.632+0000 7f7903ae4640 1 -- 172.21.15.81:0/1443276918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78fc15f6b0 con 0x7f78fc073480 2024-06-29T15:38:57.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.632+0000 7f78f27fc640 1 -- 172.21.15.81:0/1443276918 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 95) ==== 98304+0+0 (secure 0 0 0) 0x7f78ec005420 con 0x7f78fc073480 2024-06-29T15:38:57.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.633+0000 7f78f27fc640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f78e0076420 0x7f78e0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.633+0000 7f790107f640 1 -- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f78e0076420 msgr2=0x7f78e0078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/4280924958 2024-06-29T15:38:57.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.633+0000 7f78f27fc640 1 -- 172.21.15.81:0/1443276918 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 6153+0+0 (secure 0 0 0) 0x7f78ec098370 con 0x7f78fc073480 2024-06-29T15:38:57.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.633+0000 7f790107f640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f78e0076420 0x7f78e0078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:38:57.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.637+0000 7f78f27fc640 1 -- 172.21.15.81:0/1443276918 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f78ec061280 con 0x7f78fc073480 2024-06-29T15:38:57.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.705+0000 7f78f27fc640 1 -- 172.21.15.81:0/1443276918 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 96) ==== 97912+0+0 (secure 0 0 0) 0x7f78ec05d510 con 0x7f78fc073480 2024-06-29T15:38:57.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.706+0000 7f78f27fc640 1 -- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f78e0076420 msgr2=0x7f78e0078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:38:57.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.706+0000 7f78f27fc640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f78e0076420 0x7f78e0078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.755 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:57.753+0000 7f445684b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.755 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:57.753+0000 7f445684b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.755 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:57.754+0000 7f445684b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.755 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:57.754+0000 7f445684b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.756 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:57.754+0000 7f445684b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.756 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:57.754+0000 7f4454046640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.756 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:57.754+0000 7f4454046640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.756 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:57.754+0000 7f4454046640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.756 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:57.754+0000 7f4454046640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.756 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:38:57.754+0000 7f4454046640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:38:57.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.829+0000 7f7903ae4640 1 -- 172.21.15.81:0/1443276918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78fc06fee0 con 0x7f78fc073480 2024-06-29T15:38:57.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.830+0000 7f78f27fc640 1 -- 172.21.15.81:0/1443276918 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v170) ==== 100+0+11 (secure 0 0 0) 0x7f78ec06e190 con 0x7f78fc073480 2024-06-29T15:38:57.830 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:38:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.832+0000 7f7903ae4640 1 -- 172.21.15.81:0/1443276918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78fc073480 msgr2=0x7f78fc0ce6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.832+0000 7f7903ae4640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78fc073480 0x7f78fc0ce6d0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f78ec00e7c0 tx=0x7f78ec00ec80 comp rx=0 tx=0).stop 2024-06-29T15:38:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.832+0000 7f7903ae4640 1 -- 172.21.15.81:0/1443276918 shutdown_connections 2024-06-29T15:38:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.832+0000 7f7903ae4640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78fc15bc20 0x7f78fc1511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.832+0000 7f7903ae4640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78fc0cd150 0x7f78fc0d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.832+0000 7f7903ae4640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.167:6832/4280924958,v1:172.21.15.167:6833/4280924958] conn(0x7f78e0076420 0x7f78e0078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.832+0000 7f7903ae4640 1 --2- 172.21.15.81:0/1443276918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78fc073480 0x7f78fc0ce6d0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.832+0000 7f7903ae4640 1 -- 172.21.15.81:0/1443276918 >> 172.21.15.81:0/1443276918 conn(0x7f78fc0b2920 msgr2=0x7f78fc0b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.832+0000 7f7903ae4640 1 -- 172.21.15.81:0/1443276918 shutdown_connections 2024-06-29T15:38:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.832+0000 7f7903ae4640 1 -- 172.21.15.81:0/1443276918 wait complete. 2024-06-29T15:38:57.844 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:57.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.958+0000 7ff978e81640 1 -- 172.21.15.81:0/3609883853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff97415ba10 msgr2=0x7ff9740ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.958+0000 7ff978e81640 1 --2- 172.21.15.81:0/3609883853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff97415ba10 0x7ff9740ce1b0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7ff96c005fd0 tx=0x7ff96c02aa90 comp rx=0 tx=0).stop 2024-06-29T15:38:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.958+0000 7ff978e81640 1 -- 172.21.15.81:0/3609883853 shutdown_connections 2024-06-29T15:38:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.958+0000 7ff978e81640 1 --2- 172.21.15.81:0/3609883853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff97415ba10 0x7ff9740ce1b0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.958+0000 7ff978e81640 1 --2- 172.21.15.81:0/3609883853 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff974073740 0x7ff97415b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.958+0000 7ff978e81640 1 -- 172.21.15.81:0/3609883853 >> 172.21.15.81:0/3609883853 conn(0x7ff97406ccc0 msgr2=0x7ff9740b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.958+0000 7ff978e81640 1 -- 172.21.15.81:0/3609883853 shutdown_connections 2024-06-29T15:38:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.958+0000 7ff978e81640 1 -- 172.21.15.81:0/3609883853 wait complete. 2024-06-29T15:38:57.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.960+0000 7ff978e81640 1 Processor -- start 2024-06-29T15:38:57.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.960+0000 7ff978e81640 1 -- start start 2024-06-29T15:38:57.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff978e81640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff97413dd70 0x7ff97413c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff978e81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff97415ba10 0x7ff9740ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff978e81640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9741f6870 0x7ff9741f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:57.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff978e81640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff9740ceb90 con 0x7ff97415ba10 2024-06-29T15:38:57.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff978e81640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff9740cea10 con 0x7ff9741f6870 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff978e81640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff9740ced10 con 0x7ff97413dd70 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff969d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff97415ba10 0x7ff9740ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff969d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff97415ba10 0x7ff9740ad510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60524/0 (socket says 172.21.15.81:60524) 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff969d9b640 1 -- 172.21.15.81:0/1691938776 learned_addr learned my addr 172.21.15.81:0/1691938776 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff97259c640 1 --2- 172.21.15.81:0/1691938776 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff97413dd70 0x7ff97413c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.961+0000 7ff972d9d640 1 --2- 172.21.15.81:0/1691938776 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9741f6870 0x7ff9741f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff969d9b640 1 -- 172.21.15.81:0/1691938776 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff97413dd70 msgr2=0x7ff97413c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff969d9b640 1 --2- 172.21.15.81:0/1691938776 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff97413dd70 0x7ff97413c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff969d9b640 1 -- 172.21.15.81:0/1691938776 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9741f6870 msgr2=0x7ff9741f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff969d9b640 1 --2- 172.21.15.81:0/1691938776 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9741f6870 0x7ff9741f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff969d9b640 1 -- 172.21.15.81:0/1691938776 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff97413cca0 con 0x7ff97415ba10 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff969d9b640 1 --2- 172.21.15.81:0/1691938776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff97415ba10 0x7ff9740ad510 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00e790 tx=0x7ff95c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff96b7fe640 1 -- 172.21.15.81:0/1691938776 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c00eea0 con 0x7ff97415ba10 2024-06-29T15:38:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff96b7fe640 1 -- 172.21.15.81:0/1691938776 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff95c00cd50 con 0x7ff97415ba10 2024-06-29T15:38:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff96b7fe640 1 -- 172.21.15.81:0/1691938776 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c010690 con 0x7ff97415ba10 2024-06-29T15:38:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff978e81640 1 -- 172.21.15.81:0/1691938776 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9740b15a0 con 0x7ff97415ba10 2024-06-29T15:38:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.962+0000 7ff978e81640 1 -- 172.21.15.81:0/1691938776 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9741f6070 con 0x7ff97415ba10 2024-06-29T15:38:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.963+0000 7ff978e81640 1 -- 172.21.15.81:0/1691938776 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9740cf230 con 0x7ff97415ba10 2024-06-29T15:38:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.964+0000 7ff96b7fe640 1 -- 172.21.15.81:0/1691938776 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 96) ==== 97912+0+0 (secure 0 0 0) 0x7ff95c010830 con 0x7ff97415ba10 2024-06-29T15:38:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.964+0000 7ff96b7fe640 1 -- 172.21.15.81:0/1691938776 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 6497+0+0 (secure 0 0 0) 0x7ff95c097da0 con 0x7ff97415ba10 2024-06-29T15:38:57.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:57.968+0000 7ff96b7fe640 1 -- 172.21.15.81:0/1691938776 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff97413cca0 con 0x7ff97415ba10 2024-06-29T15:38:58.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.189+0000 7ff978e81640 1 -- 172.21.15.81:0/1691938776 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff97415aaa0 con 0x7ff97415ba10 2024-06-29T15:38:58.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.190+0000 7ff96b7fe640 1 -- 172.21.15.81:0/1691938776 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5233 (secure 0 0 0) 0x7ff95c061150 con 0x7ff97415ba10 2024-06-29T15:38:58.191 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:58.191 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":86,"btime":"2024-06-29T15:38:42:386002+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7754,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3261744821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3261744821},{"type":"v1","addr":"172.21.15.167:6836","nonce":3261744821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7757,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1617521107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1617521107},{"type":"v1","addr":"172.21.15.167:6837","nonce":1617521107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":9636,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3224768380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3224768380},{"type":"v1","addr":"172.21.15.81:6833","nonce":3224768380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.731697+0000","modified":"2024-06-29T15:38:41.749109+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9639":{"gid":9639,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/1634290596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1634290596},{"type":"v1","addr":"172.21.15.81:6835","nonce":1634290596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9639,"qdb_cluster":[9639]},"id":10}]} 2024-06-29T15:38:58.191 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 86 2024-06-29T15:38:58.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.192+0000 7ff978e81640 1 -- 172.21.15.81:0/1691938776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff97415ba10 msgr2=0x7ff9740ad510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.192+0000 7ff978e81640 1 --2- 172.21.15.81:0/1691938776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff97415ba10 0x7ff9740ad510 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00e790 tx=0x7ff95c00ec50 comp rx=0 tx=0).stop 2024-06-29T15:38:58.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.192+0000 7ff978e81640 1 -- 172.21.15.81:0/1691938776 shutdown_connections 2024-06-29T15:38:58.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.192+0000 7ff978e81640 1 --2- 172.21.15.81:0/1691938776 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff9741f6870 0x7ff9741f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.192+0000 7ff978e81640 1 --2- 172.21.15.81:0/1691938776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff97415ba10 0x7ff9740ad510 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.192+0000 7ff978e81640 1 --2- 172.21.15.81:0/1691938776 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff97413dd70 0x7ff97413c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.192+0000 7ff978e81640 1 -- 172.21.15.81:0/1691938776 >> 172.21.15.81:0/1691938776 conn(0x7ff97406ccc0 msgr2=0x7ff97415e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:58.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.192+0000 7ff978e81640 1 -- 172.21.15.81:0/1691938776 shutdown_connections 2024-06-29T15:38:58.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.192+0000 7ff978e81640 1 -- 172.21.15.81:0/1691938776 wait complete. 2024-06-29T15:38:58.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:58.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.316+0000 7f20c39d2640 1 -- 172.21.15.81:0/1607804625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20bc06f660 msgr2=0x7f20bc06fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.316+0000 7f20c39d2640 1 --2- 172.21.15.81:0/1607804625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20bc06f660 0x7f20bc06fa70 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f20bc0cc260 tx=0x7f20b402a980 comp rx=0 tx=0).stop 2024-06-29T15:38:58.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.316+0000 7f20c39d2640 1 -- 172.21.15.81:0/1607804625 shutdown_connections 2024-06-29T15:38:58.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.316+0000 7f20c39d2640 1 --2- 172.21.15.81:0/1607804625 >> v2:172.21.15.167:3300/0 conn(0x7f20bc15b5b0 0x7f20bc0b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.316+0000 7f20c39d2640 1 --2- 172.21.15.81:0/1607804625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20bc06f660 0x7f20bc06fa70 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.316+0000 7f20c39d2640 1 -- 172.21.15.81:0/1607804625 >> 172.21.15.81:0/1607804625 conn(0x7f20bc06ccc0 msgr2=0x7f20bc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:58.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.316+0000 7f20c39d2640 1 -- 172.21.15.81:0/1607804625 shutdown_connections 2024-06-29T15:38:58.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.316+0000 7f20c39d2640 1 -- 172.21.15.81:0/1607804625 wait complete. 2024-06-29T15:38:58.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.318+0000 7f20c39d2640 1 Processor -- start 2024-06-29T15:38:58.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.318+0000 7f20c39d2640 1 -- start start 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c39d2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20bc06f660 0x7f20bc14aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c39d2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20bc0b35b0 0x7f20bc1463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c39d2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20bc15b5b0 0x7f20bc159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c39d2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f20bc072360 con 0x7f20bc15b5b0 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c39d2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f20bc0721e0 con 0x7f20bc06f660 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c39d2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f20bc0724e0 con 0x7f20bc0b35b0 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c1f6f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20bc15b5b0 0x7f20bc159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c1f6f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20bc15b5b0 0x7f20bc159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60550/0 (socket says 172.21.15.81:60550) 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c1f6f640 1 -- 172.21.15.81:0/2504304473 learned_addr learned my addr 172.21.15.81:0/2504304473 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c176e640 1 --2- 172.21.15.81:0/2504304473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20bc06f660 0x7f20bc14aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c0f6d640 1 --2- 172.21.15.81:0/2504304473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20bc0b35b0 0x7f20bc1463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c1f6f640 1 -- 172.21.15.81:0/2504304473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20bc0b35b0 msgr2=0x7f20bc1463e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c1f6f640 1 --2- 172.21.15.81:0/2504304473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20bc0b35b0 0x7f20bc1463e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c1f6f640 1 -- 172.21.15.81:0/2504304473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20bc06f660 msgr2=0x7f20bc14aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c1f6f640 1 --2- 172.21.15.81:0/2504304473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20bc06f660 0x7f20bc14aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.319+0000 7f20c1f6f640 1 -- 172.21.15.81:0/2504304473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20bc14b2b0 con 0x7f20bc15b5b0 2024-06-29T15:38:58.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.320+0000 7f20c1f6f640 1 --2- 172.21.15.81:0/2504304473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20bc15b5b0 0x7f20bc159530 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009d60 tx=0x7f20b4003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:58.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.320+0000 7f20b1ffb640 1 -- 172.21.15.81:0/2504304473 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b4034040 con 0x7f20bc15b5b0 2024-06-29T15:38:58.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.320+0000 7f20b1ffb640 1 -- 172.21.15.81:0/2504304473 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20b40046e0 con 0x7f20bc15b5b0 2024-06-29T15:38:58.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.320+0000 7f20c39d2640 1 -- 172.21.15.81:0/2504304473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20bc14b590 con 0x7f20bc15b5b0 2024-06-29T15:38:58.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.320+0000 7f20b1ffb640 1 -- 172.21.15.81:0/2504304473 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b40456e0 con 0x7f20bc15b5b0 2024-06-29T15:38:58.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.320+0000 7f20c39d2640 1 -- 172.21.15.81:0/2504304473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20bc14bb50 con 0x7f20bc15b5b0 2024-06-29T15:38:58.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.321+0000 7f20c39d2640 1 -- 172.21.15.81:0/2504304473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20bc072a60 con 0x7f20bc15b5b0 2024-06-29T15:38:58.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.322+0000 7f20b1ffb640 1 -- 172.21.15.81:0/2504304473 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 96) ==== 97912+0+0 (secure 0 0 0) 0x7f20b4037020 con 0x7f20bc15b5b0 2024-06-29T15:38:58.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.322+0000 7f20b1ffb640 1 -- 172.21.15.81:0/2504304473 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 6497+0+0 (secure 0 0 0) 0x7f20b40b9010 con 0x7f20bc15b5b0 2024-06-29T15:38:58.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.326+0000 7f20b1ffb640 1 -- 172.21.15.81:0/2504304473 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f20bc06ffb0 con 0x7f20bc15b5b0 2024-06-29T15:38:58.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:38:58.362 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:38:58.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.510+0000 7f20c39d2640 1 -- 172.21.15.81:0/2504304473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20bc14afc0 con 0x7f20bc15b5b0 2024-06-29T15:38:58.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.511+0000 7f20b1ffb640 1 -- 172.21.15.81:0/2504304473 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) ==== 74+0+17841 (secure 0 0 0) 0x7f20b40823c0 con 0x7f20bc15b5b0 2024-06-29T15:38:58.512 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:58.512 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":170,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:57.694419+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:37.031836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.676381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/678307808":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/3068206957":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/3438796129":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/353121370":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/1995307863":"2024-06-30T15:38:57.694381+0000","172.21.15.167:6833/4280924958":"2024-06-30T15:38:57.694381+0000","172.21.15.167:6832/4280924958":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/1753956564":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/1949204156":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/4139817389":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/3283653249":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/3855027753":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/2505002611":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/4205626971":"2024-06-30T15:38:51.005371+0000","172.21.15.167:6833/150801987":"2024-06-30T15:38:51.005371+0000","172.21.15.167:6832/150801987":"2024-06-30T15:38:51.005371+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:58.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.513+0000 7f20c39d2640 1 -- 172.21.15.81:0/2504304473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20bc15b5b0 msgr2=0x7f20bc159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.513+0000 7f20c39d2640 1 --2- 172.21.15.81:0/2504304473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20bc15b5b0 0x7f20bc159530 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009d60 tx=0x7f20b4003f70 comp rx=0 tx=0).stop 2024-06-29T15:38:58.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.513+0000 7f20c39d2640 1 -- 172.21.15.81:0/2504304473 shutdown_connections 2024-06-29T15:38:58.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.513+0000 7f20c39d2640 1 --2- 172.21.15.81:0/2504304473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20bc15b5b0 0x7f20bc159530 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.513+0000 7f20c39d2640 1 --2- 172.21.15.81:0/2504304473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20bc0b35b0 0x7f20bc1463e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.513+0000 7f20c39d2640 1 --2- 172.21.15.81:0/2504304473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20bc06f660 0x7f20bc14aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.513+0000 7f20c39d2640 1 -- 172.21.15.81:0/2504304473 >> 172.21.15.81:0/2504304473 conn(0x7f20bc06ccc0 msgr2=0x7f20bc15bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:58.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.513+0000 7f20c39d2640 1 -- 172.21.15.81:0/2504304473 shutdown_connections 2024-06-29T15:38:58.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.513+0000 7f20c39d2640 1 -- 172.21.15.81:0/2504304473 wait complete. 2024-06-29T15:38:58.525 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:38:58.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:58.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.637+0000 7f181cd0a640 1 -- 172.21.15.81:0/2043274938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18180b3640 msgr2=0x7f181815ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.637+0000 7f181cd0a640 1 --2- 172.21.15.81:0/2043274938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18180b3640 0x7f181815ab20 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f18180b2790 tx=0x7f180802a980 comp rx=0 tx=0).stop 2024-06-29T15:38:58.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.637+0000 7f181cd0a640 1 -- 172.21.15.81:0/2043274938 shutdown_connections 2024-06-29T15:38:58.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.637+0000 7f181cd0a640 1 --2- 172.21.15.81:0/2043274938 >> v2:172.21.15.167:3300/0 conn(0x7f181815b060 0x7f181806f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.637+0000 7f181cd0a640 1 --2- 172.21.15.81:0/2043274938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18180b3640 0x7f181815ab20 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.637+0000 7f181cd0a640 1 -- 172.21.15.81:0/2043274938 >> 172.21.15.81:0/2043274938 conn(0x7f181806ccc0 msgr2=0x7f18180b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:58.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.637+0000 7f181cd0a640 1 -- 172.21.15.81:0/2043274938 shutdown_connections 2024-06-29T15:38:58.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.637+0000 7f181cd0a640 1 -- 172.21.15.81:0/2043274938 wait complete. 2024-06-29T15:38:58.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.639+0000 7f181cd0a640 1 Processor -- start 2024-06-29T15:38:58.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.639+0000 7f181cd0a640 1 -- start start 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.640+0000 7f181cd0a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181806fba0 0x7f1818150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.640+0000 7f181cd0a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18180b3640 0x7f18181573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.640+0000 7f181cd0a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181815b060 0x7f1818157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.640+0000 7f181cd0a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f18180cf740 con 0x7f181815b060 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.640+0000 7f181cd0a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f18180cf5c0 con 0x7f18180b3640 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.640+0000 7f181cd0a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1818072630 con 0x7f181806fba0 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1816d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181815b060 0x7f1818157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1816d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181815b060 0x7f1818157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60558/0 (socket says 172.21.15.81:60558) 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1816d9d640 1 -- 172.21.15.81:0/2801335132 learned_addr learned my addr 172.21.15.81:0/2801335132 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1815d9b640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18180b3640 0x7f18181573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f181659c640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181806fba0 0x7f1818150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1816d9d640 1 -- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181806fba0 msgr2=0x7f1818150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1816d9d640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181806fba0 0x7f1818150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1816d9d640 1 -- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18180b3640 msgr2=0x7f18181573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1816d9d640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18180b3640 0x7f18181573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1816d9d640 1 -- 172.21.15.81:0/2801335132 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1818155c00 con 0x7f181815b060 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f181659c640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181806fba0 0x7f1818150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1816d9d640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181815b060 0x7f1818157900 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f181000b410 tx=0x7f181000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1806ffd640 1 -- 172.21.15.81:0/2801335132 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1810015030 con 0x7f181815b060 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1806ffd640 1 -- 172.21.15.81:0/2801335132 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1810004110 con 0x7f181815b060 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f1806ffd640 1 -- 172.21.15.81:0/2801335132 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1810011630 con 0x7f181815b060 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f181cd0a640 1 -- 172.21.15.81:0/2801335132 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1818155ea0 con 0x7f181815b060 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.641+0000 7f181cd0a640 1 -- 172.21.15.81:0/2801335132 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1818156460 con 0x7f181815b060 2024-06-29T15:38:58.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.642+0000 7f181cd0a640 1 -- 172.21.15.81:0/2801335132 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1818072b90 con 0x7f181815b060 2024-06-29T15:38:58.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.643+0000 7f1806ffd640 1 -- 172.21.15.81:0/2801335132 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 96) ==== 97912+0+0 (secure 0 0 0) 0x7f1810018070 con 0x7f181815b060 2024-06-29T15:38:58.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.644+0000 7f1806ffd640 1 -- 172.21.15.81:0/2801335132 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 6497+0+0 (secure 0 0 0) 0x7f1810097040 con 0x7f181815b060 2024-06-29T15:38:58.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.648+0000 7f1806ffd640 1 -- 172.21.15.81:0/2801335132 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1810060ad0 con 0x7f181815b060 2024-06-29T15:38:58.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.704+0000 7f1806ffd640 1 -- 172.21.15.81:0/2801335132 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 97) ==== 98039+0+0 (secure 0 0 0) 0x7f181005bd40 con 0x7f181815b060 2024-06-29T15:38:58.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.704+0000 7f1806ffd640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f17f807bb40 0x7f17f807df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:58.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.709+0000 7f181659c640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f17f807bb40 0x7f17f807df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.711+0000 7f181659c640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f17f807bb40 0x7f17f807df60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1808035000 tx=0x7f1808004ab0 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.874+0000 7f181cd0a640 1 -- 172.21.15.81:0/2801335132 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18180af680 con 0x7f181815b060 2024-06-29T15:38:58.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.874+0000 7f1806ffd640 1 -- 172.21.15.81:0/2801335132 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5233 (secure 0 0 0) 0x7f1810010380 con 0x7f181815b060 2024-06-29T15:38:58.876 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:58.876 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":86,"btime":"2024-06-29T15:38:42:386002+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7754,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3261744821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3261744821},{"type":"v1","addr":"172.21.15.167:6836","nonce":3261744821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7757,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1617521107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1617521107},{"type":"v1","addr":"172.21.15.167:6837","nonce":1617521107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":9636,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3224768380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3224768380},{"type":"v1","addr":"172.21.15.81:6833","nonce":3224768380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.731697+0000","modified":"2024-06-29T15:38:41.749109+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9639":{"gid":9639,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/1634290596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1634290596},{"type":"v1","addr":"172.21.15.81:6835","nonce":1634290596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9639,"qdb_cluster":[9639]},"id":10}]} 2024-06-29T15:38:58.876 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 86 2024-06-29T15:38:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 -- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f17f807bb40 msgr2=0x7f17f807df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f17f807bb40 0x7f17f807df60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1808035000 tx=0x7f1808004ab0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 -- 172.21.15.81:0/2801335132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181815b060 msgr2=0x7f1818157900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181815b060 0x7f1818157900 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f181000b410 tx=0x7f181000b8d0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 -- 172.21.15.81:0/2801335132 shutdown_connections 2024-06-29T15:38:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f17f807bb40 0x7f17f807df60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181815b060 0x7f1818157900 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18180b3640 0x7f18181573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 --2- 172.21.15.81:0/2801335132 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f181806fba0 0x7f1818150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 -- 172.21.15.81:0/2801335132 >> 172.21.15.81:0/2801335132 conn(0x7f181806ccc0 msgr2=0x7f18180afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 -- 172.21.15.81:0/2801335132 shutdown_connections 2024-06-29T15:38:58.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:58.876+0000 7f181cd0a640 1 -- 172.21.15.81:0/2801335132 wait complete. 2024-06-29T15:38:58.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:38:59.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.001+0000 7f0ba83bd640 1 -- 172.21.15.81:0/669915721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ba00b4250 msgr2=0x7f0ba00cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.001+0000 7f0ba83bd640 1 --2- 172.21.15.81:0/669915721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ba00b4250 0x7f0ba00cee70 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f0ba00d4c50 tx=0x7f0b9802a440 comp rx=0 tx=0).stop 2024-06-29T15:38:59.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.001+0000 7f0ba83bd640 1 -- 172.21.15.81:0/669915721 shutdown_connections 2024-06-29T15:38:59.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.001+0000 7f0ba83bd640 1 --2- 172.21.15.81:0/669915721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ba00b4250 0x7f0ba00cee70 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.001+0000 7f0ba83bd640 1 --2- 172.21.15.81:0/669915721 >> v2:172.21.15.167:3300/0 conn(0x7f0ba008c270 0x7f0ba00ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.001+0000 7f0ba83bd640 1 -- 172.21.15.81:0/669915721 >> 172.21.15.81:0/669915721 conn(0x7f0ba00b56d0 msgr2=0x7f0ba00b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.001+0000 7f0ba83bd640 1 -- 172.21.15.81:0/669915721 shutdown_connections 2024-06-29T15:38:59.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.001+0000 7f0ba83bd640 1 -- 172.21.15.81:0/669915721 wait complete. 2024-06-29T15:38:59.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.003+0000 7f0ba83bd640 1 Processor -- start 2024-06-29T15:38:59.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.003+0000 7f0ba83bd640 1 -- start start 2024-06-29T15:38:59.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba83bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba008ccd0 0x7f0ba01e75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba83bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ba00b4250 0x7f0ba01d5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba83bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba01e92f0 0x7f0ba01e9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba83bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba00d2250 con 0x7f0ba00b4250 2024-06-29T15:38:59.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba83bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba00d20d0 con 0x7f0ba01e92f0 2024-06-29T15:38:59.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba83bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba00d23d0 con 0x7f0ba008ccd0 2024-06-29T15:38:59.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba5958640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ba00b4250 0x7f0ba01d5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba5958640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ba00b4250 0x7f0ba01d5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60568/0 (socket says 172.21.15.81:60568) 2024-06-29T15:38:59.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba5958640 1 -- 172.21.15.81:0/975210265 learned_addr learned my addr 172.21.15.81:0/975210265 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba695a640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba01e92f0 0x7f0ba01e9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba6159640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba008ccd0 0x7f0ba01e75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba5958640 1 -- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba008ccd0 msgr2=0x7f0ba01e75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba5958640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba008ccd0 0x7f0ba01e75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba5958640 1 -- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba01e92f0 msgr2=0x7f0ba01e9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba5958640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba01e92f0 0x7f0ba01e9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba5958640 1 -- 172.21.15.81:0/975210265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba01e8b60 con 0x7f0ba00b4250 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.004+0000 7f0ba6159640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba008ccd0 0x7f0ba01e75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.005+0000 7f0ba5958640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ba00b4250 0x7f0ba01d5c40 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400b420 tx=0x7f0b9400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.005+0000 7f0b9effd640 1 -- 172.21.15.81:0/975210265 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94011830 con 0x7f0ba00b4250 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.005+0000 7f0b9effd640 1 -- 172.21.15.81:0/975210265 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b94016070 con 0x7f0ba00b4250 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.005+0000 7f0b9effd640 1 -- 172.21.15.81:0/975210265 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94011d60 con 0x7f0ba00b4250 2024-06-29T15:38:59.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.005+0000 7f0ba83bd640 1 -- 172.21.15.81:0/975210265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba01e9c40 con 0x7f0ba00b4250 2024-06-29T15:38:59.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.005+0000 7f0ba83bd640 1 -- 172.21.15.81:0/975210265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba01e8230 con 0x7f0ba00b4250 2024-06-29T15:38:59.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.006+0000 7f0ba83bd640 1 -- 172.21.15.81:0/975210265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ba008c270 con 0x7f0ba00b4250 2024-06-29T15:38:59.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.006+0000 7f0b9effd640 1 -- 172.21.15.81:0/975210265 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 97) ==== 98039+0+0 (secure 0 0 0) 0x7f0b9401b460 con 0x7f0ba00b4250 2024-06-29T15:38:59.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.007+0000 7f0b9effd640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f0b8c076150 0x7f0b8c078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.007+0000 7f0b9effd640 1 -- 172.21.15.81:0/975210265 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 6497+0+0 (secure 0 0 0) 0x7f0b94097240 con 0x7f0ba00b4250 2024-06-29T15:38:59.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.007+0000 7f0ba6159640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f0b8c076150 0x7f0b8c078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.008+0000 7f0ba6159640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f0b8c076150 0x7f0b8c078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0b90005c40 tx=0x7f0b90005f90 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.012+0000 7f0b9effd640 1 -- 172.21.15.81:0/975210265 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0b94020020 con 0x7f0ba00b4250 2024-06-29T15:38:59.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.229+0000 7f0ba83bd640 1 -- 172.21.15.81:0/975210265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ba01e7ce0 con 0x7f0ba00b4250 2024-06-29T15:38:59.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.230+0000 7f0b9effd640 1 -- 172.21.15.81:0/975210265 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5233 (secure 0 0 0) 0x7f0ba01e9c40 con 0x7f0ba00b4250 2024-06-29T15:38:59.230 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:59.231 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":86,"btime":"2024-06-29T15:38:42:386002+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7754,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3261744821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3261744821},{"type":"v1","addr":"172.21.15.167:6836","nonce":3261744821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7757,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1617521107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1617521107},{"type":"v1","addr":"172.21.15.167:6837","nonce":1617521107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":9636,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3224768380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3224768380},{"type":"v1","addr":"172.21.15.81:6833","nonce":3224768380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:38:38.731697+0000","modified":"2024-06-29T15:38:41.749109+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9639},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9639":{"gid":9639,"name":"c","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6835/1634290596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1634290596},{"type":"v1","addr":"172.21.15.81:6835","nonce":1634290596}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9639,"qdb_cluster":[9639]},"id":10}]} 2024-06-29T15:38:59.231 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 86 2024-06-29T15:38:59.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 -- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f0b8c076150 msgr2=0x7f0b8c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f0b8c076150 0x7f0b8c078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0b90005c40 tx=0x7f0b90005f90 comp rx=0 tx=0).stop 2024-06-29T15:38:59.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 -- 172.21.15.81:0/975210265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ba00b4250 msgr2=0x7f0ba01d5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ba00b4250 0x7f0ba01d5c40 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400b420 tx=0x7f0b9400b8e0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 -- 172.21.15.81:0/975210265 shutdown_connections 2024-06-29T15:38:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f0b8c076150 0x7f0b8c078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba01e92f0 0x7f0ba01e9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ba00b4250 0x7f0ba01d5c40 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 --2- 172.21.15.81:0/975210265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba008ccd0 0x7f0ba01e75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 -- 172.21.15.81:0/975210265 >> 172.21.15.81:0/975210265 conn(0x7f0ba00b56d0 msgr2=0x7f0ba01e3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 -- 172.21.15.81:0/975210265 shutdown_connections 2024-06-29T15:38:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.232+0000 7f0ba83bd640 1 -- 172.21.15.81:0/975210265 wait complete. 2024-06-29T15:38:59.243 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:38:59.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.355+0000 7fa1ed72a640 1 -- 172.21.15.81:0/744694799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e80b3200 msgr2=0x7fa1e80cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.355+0000 7fa1ed72a640 1 --2- 172.21.15.81:0/744694799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e80b3200 0x7fa1e80cc7b0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8074b10 tx=0x7fa1d802aa30 comp rx=0 tx=0).stop 2024-06-29T15:38:59.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.355+0000 7fa1ed72a640 1 -- 172.21.15.81:0/744694799 shutdown_connections 2024-06-29T15:38:59.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.355+0000 7fa1ed72a640 1 --2- 172.21.15.81:0/744694799 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1e80d0770 0x7fa1e80ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.355+0000 7fa1ed72a640 1 --2- 172.21.15.81:0/744694799 >> v2:172.21.15.167:3300/0 conn(0x7fa1e80cccf0 0x7fa1e80d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.355+0000 7fa1ed72a640 1 --2- 172.21.15.81:0/744694799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e80b3200 0x7fa1e80cc7b0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.355+0000 7fa1ed72a640 1 -- 172.21.15.81:0/744694799 >> 172.21.15.81:0/744694799 conn(0x7fa1e80b3680 msgr2=0x7fa1e80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.355+0000 7fa1ed72a640 1 -- 172.21.15.81:0/744694799 shutdown_connections 2024-06-29T15:38:59.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.355+0000 7fa1ed72a640 1 -- 172.21.15.81:0/744694799 wait complete. 2024-06-29T15:38:59.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.357+0000 7fa1ed72a640 1 Processor -- start 2024-06-29T15:38:59.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.357+0000 7fa1ed72a640 1 -- start start 2024-06-29T15:38:59.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.358+0000 7fa1ed72a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1e80b3200 0x7fa1e81562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.358+0000 7fa1ed72a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e80cccf0 0x7fa1e815b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.358+0000 7fa1ed72a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1e80d0770 0x7fa1e814b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.358+0000 7fa1ed72a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e80cefe0 con 0x7fa1e80cccf0 2024-06-29T15:38:59.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.358+0000 7fa1ed72a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e80cee60 con 0x7fa1e80d0770 2024-06-29T15:38:59.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.358+0000 7fa1ed72a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa1e80cf160 con 0x7fa1e80b3200 2024-06-29T15:38:59.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e80cccf0 0x7fa1e815b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e80cccf0 0x7fa1e815b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60578/0 (socket says 172.21.15.81:60578) 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e67fc640 1 -- 172.21.15.81:0/3677547107 learned_addr learned my addr 172.21.15.81:0/3677547107 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e6ffd640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1e80b3200 0x7fa1e81562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e77fe640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1e80d0770 0x7fa1e814b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e67fc640 1 -- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1e80b3200 msgr2=0x7fa1e81562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e67fc640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1e80b3200 0x7fa1e81562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e67fc640 1 -- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1e80d0770 msgr2=0x7fa1e814b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e67fc640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1e80d0770 0x7fa1e814b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e67fc640 1 -- 172.21.15.81:0/3677547107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e806df40 con 0x7fa1e80cccf0 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1e67fc640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e80cccf0 0x7fa1e815b3c0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c970 tx=0x7fa1dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1cf7fe640 1 -- 172.21.15.81:0/3677547107 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc007bb0 con 0x7fa1e80cccf0 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1cf7fe640 1 -- 172.21.15.81:0/3677547107 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1dc007d50 con 0x7fa1e80cccf0 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1cf7fe640 1 -- 172.21.15.81:0/3677547107 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc005280 con 0x7fa1e80cccf0 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.359+0000 7fa1ed72a640 1 -- 172.21.15.81:0/3677547107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1e806e220 con 0x7fa1e80cccf0 2024-06-29T15:38:59.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.360+0000 7fa1ed72a640 1 -- 172.21.15.81:0/3677547107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1e806f1d0 con 0x7fa1e80cccf0 2024-06-29T15:38:59.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.360+0000 7fa1ed72a640 1 -- 172.21.15.81:0/3677547107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1e806e460 con 0x7fa1e80cccf0 2024-06-29T15:38:59.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.361+0000 7fa1cf7fe640 1 -- 172.21.15.81:0/3677547107 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 97) ==== 98039+0+0 (secure 0 0 0) 0x7fa1dc005420 con 0x7fa1e80cccf0 2024-06-29T15:38:59.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.362+0000 7fa1cf7fe640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa1d4076120 0x7fa1d4078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.362+0000 7fa1cf7fe640 1 -- 172.21.15.81:0/3677547107 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 6497+0+0 (secure 0 0 0) 0x7fa1dc097a20 con 0x7fa1e80cccf0 2024-06-29T15:38:59.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.362+0000 7fa1e6ffd640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa1d4076120 0x7fa1d4078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.362+0000 7fa1e6ffd640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa1d4076120 0x7fa1d4078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8035000 tx=0x7fa1d803b040 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.366+0000 7fa1cf7fe640 1 -- 172.21.15.81:0/3677547107 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa1e806e460 con 0x7fa1e80cccf0 2024-06-29T15:38:59.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.546+0000 7fa1ed72a640 1 -- 172.21.15.81:0/3677547107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1e80d0230 con 0x7fa1e80cccf0 2024-06-29T15:38:59.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.548+0000 7fa1cf7fe640 1 -- 172.21.15.81:0/3677547107 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) ==== 74+0+17841 (secure 0 0 0) 0x7fa1dc014030 con 0x7fa1e80cccf0 2024-06-29T15:38:59.549 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:38:59.550 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":170,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:38:57.694419+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:38:37.031836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-06-29T15:38:37.676381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"168","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/678307808":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/3068206957":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/3438796129":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/353121370":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/1995307863":"2024-06-30T15:38:57.694381+0000","172.21.15.167:6833/4280924958":"2024-06-30T15:38:57.694381+0000","172.21.15.167:6832/4280924958":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/1753956564":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/1949204156":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/4139817389":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/3283653249":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/3855027753":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/2505002611":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/4205626971":"2024-06-30T15:38:51.005371+0000","172.21.15.167:6833/150801987":"2024-06-30T15:38:51.005371+0000","172.21.15.167:6832/150801987":"2024-06-30T15:38:51.005371+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 -- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa1d4076120 msgr2=0x7fa1d4078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa1d4076120 0x7fa1d4078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8035000 tx=0x7fa1d803b040 comp rx=0 tx=0).stop 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 -- 172.21.15.81:0/3677547107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e80cccf0 msgr2=0x7fa1e815b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e80cccf0 0x7fa1e815b3c0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c970 tx=0x7fa1dc00ce30 comp rx=0 tx=0).stop 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 -- 172.21.15.81:0/3677547107 shutdown_connections 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa1d4076120 0x7fa1d4078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1e80d0770 0x7fa1e814b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e80cccf0 0x7fa1e815b3c0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 --2- 172.21.15.81:0/3677547107 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1e80b3200 0x7fa1e81562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 -- 172.21.15.81:0/3677547107 >> 172.21.15.81:0/3677547107 conn(0x7fa1e80b3680 msgr2=0x7fa1e80c8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 -- 172.21.15.81:0/3677547107 shutdown_connections 2024-06-29T15:38:59.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.550+0000 7fa1ed72a640 1 -- 172.21.15.81:0/3677547107 wait complete. 2024-06-29T15:38:59.562 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:38:59.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.675+0000 7f7ee9b24640 1 -- 172.21.15.81:0/2153144348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ed0004d40 msgr2=0x7f7ed00051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.675+0000 7f7ee9b24640 1 --2- 172.21.15.81:0/2153144348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ed0004d40 0x7f7ed00051f0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009ed0 tx=0x7f7ed0005830 comp rx=0 tx=0).stop 2024-06-29T15:38:59.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.675+0000 7f7ee3fff640 1 -- 172.21.15.81:0/2153144348 reap_dead start 2024-06-29T15:38:59.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.675+0000 7f7ee9b24640 1 -- 172.21.15.81:0/2153144348 shutdown_connections 2024-06-29T15:38:59.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.675+0000 7f7ee9b24640 1 -- 172.21.15.81:0/2153144348 >> 172.21.15.81:0/2153144348 conn(0x7f7ee406ccc0 msgr2=0x7f7ee40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.676+0000 7f7ee9b24640 1 -- 172.21.15.81:0/2153144348 shutdown_connections 2024-06-29T15:38:59.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.676+0000 7f7ee9b24640 1 -- 172.21.15.81:0/2153144348 wait complete. 2024-06-29T15:38:59.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.677+0000 7f7ee9b24640 1 Processor -- start 2024-06-29T15:38:59.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.677+0000 7f7ee9b24640 1 -- start start 2024-06-29T15:38:59.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7ee9b24640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ed0004d40 0x7f7ee413adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7ee9b24640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee4138d20 0x7f7ee4142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7ee9b24640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ee41430f0 0x7f7ee415b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7ee9b24640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee4164c30 con 0x7f7ee4138d20 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7ee9b24640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee4164ab0 con 0x7f7ed0004d40 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7ee9b24640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee4164db0 con 0x7f7ee41430f0 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee4138d20 0x7f7ee4142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7edaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee4138d20 0x7f7ee4142bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60602/0 (socket says 172.21.15.81:60602) 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7edaffd640 1 -- 172.21.15.81:0/3250683569 learned_addr learned my addr 172.21.15.81:0/3250683569 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7ee37fe640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ed0004d40 0x7f7ee413adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7ee3fff640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ee41430f0 0x7f7ee415b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7edaffd640 1 -- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ee41430f0 msgr2=0x7f7ee415b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7edaffd640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ee41430f0 0x7f7ee415b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7edaffd640 1 -- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ed0004d40 msgr2=0x7f7ee413adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7edaffd640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ed0004d40 0x7f7ee413adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7edaffd640 1 -- 172.21.15.81:0/3250683569 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee4139d50 con 0x7f7ee4138d20 2024-06-29T15:38:59.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7edaffd640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee4138d20 0x7f7ee4142bb0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc003fb0 tx=0x7f7ecc0016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7ee0ff9640 1 -- 172.21.15.81:0/3250683569 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc015020 con 0x7f7ee4138d20 2024-06-29T15:38:59.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.679+0000 7f7ee0ff9640 1 -- 172.21.15.81:0/3250683569 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ecc011860 con 0x7f7ee4138d20 2024-06-29T15:38:59.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.680+0000 7f7ee9b24640 1 -- 172.21.15.81:0/3250683569 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee413a030 con 0x7f7ee4138d20 2024-06-29T15:38:59.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.680+0000 7f7ee0ff9640 1 -- 172.21.15.81:0/3250683569 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc011cb0 con 0x7f7ee4138d20 2024-06-29T15:38:59.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.680+0000 7f7ee9b24640 1 -- 172.21.15.81:0/3250683569 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee413a5f0 con 0x7f7ee4138d20 2024-06-29T15:38:59.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.680+0000 7f7ee9b24640 1 -- 172.21.15.81:0/3250683569 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee4139ab0 con 0x7f7ee4138d20 2024-06-29T15:38:59.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.681+0000 7f7ee0ff9640 1 -- 172.21.15.81:0/3250683569 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 97) ==== 98039+0+0 (secure 0 0 0) 0x7f7ecc018020 con 0x7f7ee4138d20 2024-06-29T15:38:59.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.682+0000 7f7ee0ff9640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7ec4076150 0x7f7ec4078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:38:59.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.682+0000 7f7ee0ff9640 1 -- 172.21.15.81:0/3250683569 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 6497+0+0 (secure 0 0 0) 0x7f7ecc098090 con 0x7f7ee4138d20 2024-06-29T15:38:59.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.682+0000 7f7ee37fe640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7ec4076150 0x7f7ec4078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:38:59.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.682+0000 7f7ee37fe640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7ec4076150 0x7f7ec4078660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4001240 tx=0x7f7ed4006040 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:38:59.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.687+0000 7f7ee0ff9640 1 -- 172.21.15.81:0/3250683569 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7ee4139ab0 con 0x7f7ee4138d20 2024-06-29T15:38:59.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.911+0000 7f7ee9b24640 1 -- 172.21.15.81:0/3250683569 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7ee413be00 con 0x7f7ee4138d20 2024-06-29T15:38:59.912 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T15:38:59.911+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:38:59.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.913+0000 7f7ee0ff9640 1 -- 172.21.15.81:0/3250683569 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v87) ==== 161+0+0 (secure 0 0 0) 0x7f7ecc01f020 con 0x7f7ee4138d20 2024-06-29T15:38:59.913 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:38:59.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.916+0000 7f7ee9b24640 1 -- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7ec4076150 msgr2=0x7f7ec4078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.916+0000 7f7ee9b24640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7ec4076150 0x7f7ec4078660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4001240 tx=0x7f7ed4006040 comp rx=0 tx=0).stop 2024-06-29T15:38:59.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.916+0000 7f7ee9b24640 1 -- 172.21.15.81:0/3250683569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee4138d20 msgr2=0x7f7ee4142bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:38:59.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.916+0000 7f7ee9b24640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee4138d20 0x7f7ee4142bb0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc003fb0 tx=0x7f7ecc0016f0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.916+0000 7f7ee9b24640 1 -- 172.21.15.81:0/3250683569 shutdown_connections 2024-06-29T15:38:59.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.916+0000 7f7ee9b24640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7ec4076150 0x7f7ec4078660 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4001240 tx=0x7f7ed4006040 comp rx=0 tx=0).stop 2024-06-29T15:38:59.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.916+0000 7f7ee9b24640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ee41430f0 0x7f7ee415b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.916+0000 7f7ee9b24640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee4138d20 0x7f7ee4142bb0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.916+0000 7f7ee9b24640 1 --2- 172.21.15.81:0/3250683569 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ed0004d40 0x7f7ee413adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:38:59.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.916+0000 7f7ee9b24640 1 -- 172.21.15.81:0/3250683569 >> 172.21.15.81:0/3250683569 conn(0x7f7ee406ccc0 msgr2=0x7f7ee413c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:38:59.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.917+0000 7f7ee9b24640 1 -- 172.21.15.81:0/3250683569 shutdown_connections 2024-06-29T15:38:59.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:38:59.917+0000 7f7ee9b24640 1 -- 172.21.15.81:0/3250683569 wait complete. 2024-06-29T15:38:59.919 INFO:tasks.ceph.mds.c.smithi081.stderr: -4307> 2024-06-29T15:38:41.337+0000 7f0285804640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:38:59.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:38:59.972 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T15:39:00.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.048+0000 7f7d11943640 1 -- 172.21.15.81:0/882314219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d0c0d2270 msgr2=0x7f7d0c0d2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:00.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.048+0000 7f7d11943640 1 --2- 172.21.15.81:0/882314219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d0c0d2270 0x7f7d0c0d2680 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c08c8b0 tx=0x7f7cf402a4b0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.048+0000 7f7d11943640 1 -- 172.21.15.81:0/882314219 shutdown_connections 2024-06-29T15:39:00.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.048+0000 7f7d11943640 1 --2- 172.21.15.81:0/882314219 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0c161c60 0x7f7d0c162070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.048+0000 7f7d11943640 1 --2- 172.21.15.81:0/882314219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d0c0d2270 0x7f7d0c0d2680 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.048+0000 7f7d11943640 1 --2- 172.21.15.81:0/882314219 >> v2:172.21.15.167:3300/0 conn(0x7f7d0c0d0480 0x7f7d0c0b8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.048+0000 7f7d11943640 1 -- 172.21.15.81:0/882314219 >> 172.21.15.81:0/882314219 conn(0x7f7d0c0d3b70 msgr2=0x7f7d0c0d3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:00.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.049+0000 7f7d11943640 1 -- 172.21.15.81:0/882314219 shutdown_connections 2024-06-29T15:39:00.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.049+0000 7f7d11943640 1 -- 172.21.15.81:0/882314219 wait complete. 2024-06-29T15:39:00.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.050+0000 7f7d11943640 1 Processor -- start 2024-06-29T15:39:00.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.050+0000 7f7d11943640 1 -- start start 2024-06-29T15:39:00.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.051+0000 7f7d11943640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0c0d0480 0x7f7d0c150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:00.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.051+0000 7f7d11943640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d0c0d2270 0x7f7d0c14d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:00.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.051+0000 7f7d11943640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d0c161c60 0x7f7d0c15e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:00.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d11943640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c0b0920 con 0x7f7d0c0d2270 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d11943640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c0b07a0 con 0x7f7d0c161c60 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d11943640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0c1625b0 con 0x7f7d0c0d0480 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d03fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d0c0d2270 0x7f7d0c14d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d03fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d0c0d2270 0x7f7d0c14d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60200/0 (socket says 172.21.15.81:60200) 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d03fff640 1 -- 172.21.15.81:0/314133055 learned_addr learned my addr 172.21.15.81:0/314133055 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d0affd640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0c0d0480 0x7f7d0c150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d03fff640 1 -- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0c0d0480 msgr2=0x7f7d0c150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d03fff640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0c0d0480 0x7f7d0c150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d03fff640 1 -- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d0c161c60 msgr2=0x7f7d0c15e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d03fff640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d0c161c60 0x7f7d0c15e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d03fff640 1 -- 172.21.15.81:0/314133055 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0c1511a0 con 0x7f7d0c0d2270 2024-06-29T15:39:00.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d03fff640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d0c0d2270 0x7f7d0c14d880 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40099a0 tx=0x7f7cf4004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:00.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d08ff9640 1 -- 172.21.15.81:0/314133055 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4031d60 con 0x7f7d0c0d2270 2024-06-29T15:39:00.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d08ff9640 1 -- 172.21.15.81:0/314133055 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cf4033070 con 0x7f7d0c0d2270 2024-06-29T15:39:00.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d08ff9640 1 -- 172.21.15.81:0/314133055 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4033480 con 0x7f7d0c0d2270 2024-06-29T15:39:00.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d11943640 1 -- 172.21.15.81:0/314133055 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d0c0cce70 con 0x7f7d0c0d2270 2024-06-29T15:39:00.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.052+0000 7f7d11943640 1 -- 172.21.15.81:0/314133055 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d0c0cd430 con 0x7f7d0c0d2270 2024-06-29T15:39:00.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.053+0000 7f7d11943640 1 -- 172.21.15.81:0/314133055 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d0c0d2680 con 0x7f7d0c0d2270 2024-06-29T15:39:00.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.054+0000 7f7d08ff9640 1 -- 172.21.15.81:0/314133055 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 97) ==== 98039+0+0 (secure 0 0 0) 0x7f7cf4033620 con 0x7f7d0c0d2270 2024-06-29T15:39:00.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.054+0000 7f7d08ff9640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cec076150 0x7f7cec078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:00.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.054+0000 7f7d08ff9640 1 -- 172.21.15.81:0/314133055 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(171..171 src has 1..171) ==== 6583+0+0 (secure 0 0 0) 0x7f7cf40b8b60 con 0x7f7d0c0d2270 2024-06-29T15:39:00.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.055+0000 7f7d0affd640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cec076150 0x7f7cec078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:00.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.055+0000 7f7d0affd640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cec076150 0x7f7cec078660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00b920 tx=0x7f7cfc005ba0 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:00.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.059+0000 7f7d08ff9640 1 -- 172.21.15.81:0/314133055 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7cf403f030 con 0x7f7d0c0d2270 2024-06-29T15:39:00.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.093+0000 7f7d08ff9640 1 -- 172.21.15.81:0/314133055 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 98) ==== 98247+0+0 (secure 0 0 0) 0x7f7cf407df90 con 0x7f7d0c0d2270 2024-06-29T15:39:00.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.284+0000 7f7d11943640 1 -- 172.21.15.81:0/314133055 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f7d0c0b9a30 con 0x7f7d0c0d2270 2024-06-29T15:39:00.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.915+0000 7f7d08ff9640 1 -- 172.21.15.81:0/314133055 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v88) ==== 104+0+0 (secure 0 0 0) 0x7f7cf4082400 con 0x7f7d0c0d2270 2024-06-29T15:39:00.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 -- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cec076150 msgr2=0x7f7cec078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:00.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cec076150 0x7f7cec078660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00b920 tx=0x7f7cfc005ba0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 -- 172.21.15.81:0/314133055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d0c0d2270 msgr2=0x7f7d0c14d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:00.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d0c0d2270 0x7f7d0c14d880 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40099a0 tx=0x7f7cf4004ac0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 -- 172.21.15.81:0/314133055 shutdown_connections 2024-06-29T15:39:00.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cec076150 0x7f7cec078660 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d0c161c60 0x7f7d0c15e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d0c0d2270 0x7f7d0c14d880 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 --2- 172.21.15.81:0/314133055 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d0c0d0480 0x7f7d0c150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:00.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 -- 172.21.15.81:0/314133055 >> 172.21.15.81:0/314133055 conn(0x7f7d0c0d3b70 msgr2=0x7f7d0c0cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:00.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 -- 172.21.15.81:0/314133055 shutdown_connections 2024-06-29T15:39:00.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:00.917+0000 7f7d11943640 1 -- 172.21.15.81:0/314133055 wait complete. 2024-06-29T15:39:00.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:39:01.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.098+0000 7f317e847640 1 -- 172.21.15.81:0/3535200739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31780cf3a0 msgr2=0x7f31780b7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:01.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.098+0000 7f317e847640 1 --2- 172.21.15.81:0/3535200739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31780cf3a0 0x7f31780b7470 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f317806f540 tx=0x7f317002aa30 comp rx=0 tx=0).stop 2024-06-29T15:39:01.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.098+0000 7f317e847640 1 -- 172.21.15.81:0/3535200739 shutdown_connections 2024-06-29T15:39:01.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.098+0000 7f317e847640 1 --2- 172.21.15.81:0/3535200739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31780cf3a0 0x7f31780b7470 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.098+0000 7f317e847640 1 --2- 172.21.15.81:0/3535200739 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31780cea50 0x7f31780cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.098+0000 7f317e847640 1 --2- 172.21.15.81:0/3535200739 >> v2:172.21.15.167:3300/0 conn(0x7f31780d1e80 0x7f31780d2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.098+0000 7f317e847640 1 -- 172.21.15.81:0/3535200739 >> 172.21.15.81:0/3535200739 conn(0x7f31780b5700 msgr2=0x7f31780b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:01.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.098+0000 7f317e847640 1 -- 172.21.15.81:0/3535200739 shutdown_connections 2024-06-29T15:39:01.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.098+0000 7f317e847640 1 -- 172.21.15.81:0/3535200739 wait complete. 2024-06-29T15:39:01.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.100+0000 7f317e847640 1 Processor -- start 2024-06-29T15:39:01.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.100+0000 7f317e847640 1 -- start start 2024-06-29T15:39:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f317e847640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31780cea50 0x7f31781dadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f317e847640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31780cf3a0 0x7f317809e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f317e847640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31780d1e80 0x7f31780b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f317e847640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f31780d1960 con 0x7f31780cea50 2024-06-29T15:39:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f317e847640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f31780d17e0 con 0x7f31780cf3a0 2024-06-29T15:39:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f317e847640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f31780cf7b0 con 0x7f31780d1e80 2024-06-29T15:39:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f3177fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31780cea50 0x7f31781dadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f3177fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31780cea50 0x7f31781dadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60222/0 (socket says 172.21.15.81:60222) 2024-06-29T15:39:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f3177fff640 1 -- 172.21.15.81:0/3900232679 learned_addr learned my addr 172.21.15.81:0/3900232679 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f31777fe640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31780cf3a0 0x7f317809e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.101+0000 7f317cde4640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31780d1e80 0x7f31780b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f3177fff640 1 -- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31780d1e80 msgr2=0x7f31780b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f3177fff640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31780d1e80 0x7f31780b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f3177fff640 1 -- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31780cf3a0 msgr2=0x7f317809e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f3177fff640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31780cf3a0 0x7f317809e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f3177fff640 1 -- 172.21.15.81:0/3900232679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317809f5b0 con 0x7f31780cea50 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f31777fe640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31780cf3a0 0x7f317809e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f3177fff640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31780cea50 0x7f31781dadc0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f316800e7f0 tx=0x7f316800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f3174ff9640 1 -- 172.21.15.81:0/3900232679 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316800ccc0 con 0x7f31780cea50 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f3174ff9640 1 -- 172.21.15.81:0/3900232679 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3168010430 con 0x7f31780cea50 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f3174ff9640 1 -- 172.21.15.81:0/3900232679 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168010840 con 0x7f31780cea50 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f317e847640 1 -- 172.21.15.81:0/3900232679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31780b08b0 con 0x7f31780cea50 2024-06-29T15:39:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.102+0000 7f317e847640 1 -- 172.21.15.81:0/3900232679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f317809f2d0 con 0x7f31780cea50 2024-06-29T15:39:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.103+0000 7f317e847640 1 -- 172.21.15.81:0/3900232679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31780d2290 con 0x7f31780cea50 2024-06-29T15:39:01.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.104+0000 7f3174ff9640 1 -- 172.21.15.81:0/3900232679 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 98) ==== 98247+0+0 (secure 0 0 0) 0x7f31680109e0 con 0x7f31780cea50 2024-06-29T15:39:01.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.104+0000 7f3174ff9640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3158076360 0x7f3158078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:01.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.104+0000 7f3174ff9640 1 -- 172.21.15.81:0/3900232679 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(171..171 src has 1..171) ==== 6583+0+0 (secure 0 0 0) 0x7f31680986e0 con 0x7f31780cea50 2024-06-29T15:39:01.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.104+0000 7f31777fe640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3158076360 0x7f3158078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:01.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.105+0000 7f31777fe640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3158076360 0x7f3158078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f317806e5e0 tx=0x7f3160005f90 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.108+0000 7f3174ff9640 1 -- 172.21.15.81:0/3900232679 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f31780d2290 con 0x7f31780cea50 2024-06-29T15:39:01.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:01.287+0000 7f317e847640 1 -- 172.21.15.81:0/3900232679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f31780cee60 con 0x7f31780cea50 2024-06-29T15:39:02.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.100+0000 7f3174ff9640 1 -- 172.21.15.81:0/3900232679 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v172) ==== 181+0+0 (secure 0 0 0) 0x7f3168061b50 con 0x7f31780cea50 2024-06-29T15:39:02.101 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:39:02.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 -- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3158076360 msgr2=0x7f3158078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:02.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3158076360 0x7f3158078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f317806e5e0 tx=0x7f3160005f90 comp rx=0 tx=0).stop 2024-06-29T15:39:02.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 -- 172.21.15.81:0/3900232679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31780cea50 msgr2=0x7f31781dadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:02.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31780cea50 0x7f31781dadc0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f316800e7f0 tx=0x7f316800ecb0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 -- 172.21.15.81:0/3900232679 shutdown_connections 2024-06-29T15:39:02.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3158076360 0x7f3158078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f31780d1e80 0x7f31780b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f31780cf3a0 0x7f317809e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 --2- 172.21.15.81:0/3900232679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31780cea50 0x7f31781dadc0 secure :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f316800e7f0 tx=0x7f316800ecb0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 -- 172.21.15.81:0/3900232679 >> 172.21.15.81:0/3900232679 conn(0x7f31780b5700 msgr2=0x7f31780cdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:02.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 -- 172.21.15.81:0/3900232679 shutdown_connections 2024-06-29T15:39:02.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.105+0000 7f317e847640 1 -- 172.21.15.81:0/3900232679 wait complete. 2024-06-29T15:39:02.123 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:39:02.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.247+0000 7fd6041eb640 1 -- 172.21.15.81:0/1712063598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5fc1565e0 msgr2=0x7fd5fc15ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:02.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.247+0000 7fd6041eb640 1 --2- 172.21.15.81:0/1712063598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5fc1565e0 0x7fd5fc15ce20 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc145bd0 tx=0x7fd5f402a560 comp rx=0 tx=0).stop 2024-06-29T15:39:02.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.247+0000 7fd6041eb640 1 -- 172.21.15.81:0/1712063598 shutdown_connections 2024-06-29T15:39:02.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.247+0000 7fd6041eb640 1 --2- 172.21.15.81:0/1712063598 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5fc06c830 0x7fd5fc072ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.247+0000 7fd6041eb640 1 --2- 172.21.15.81:0/1712063598 >> v2:172.21.15.167:3300/0 conn(0x7fd5fc15f9c0 0x7fd5fc06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.247+0000 7fd6041eb640 1 --2- 172.21.15.81:0/1712063598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5fc1565e0 0x7fd5fc15ce20 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.247+0000 7fd6041eb640 1 -- 172.21.15.81:0/1712063598 >> 172.21.15.81:0/1712063598 conn(0x7fd5fc0b2580 msgr2=0x7fd5fc0b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:02.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.247+0000 7fd6041eb640 1 -- 172.21.15.81:0/1712063598 shutdown_connections 2024-06-29T15:39:02.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.247+0000 7fd6041eb640 1 -- 172.21.15.81:0/1712063598 wait complete. 2024-06-29T15:39:02.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.249+0000 7fd6041eb640 1 Processor -- start 2024-06-29T15:39:02.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.249+0000 7fd6041eb640 1 -- start start 2024-06-29T15:39:02.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd6041eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5fc06c830 0x7fd5fc15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd6041eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5fc15f9c0 0x7fd5fc1f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd6041eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5fc155580 0x7fd5fc159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd6041eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc0735b0 con 0x7fd5fc06c830 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd6041eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd5fc073430 con 0x7fd5fc15f9c0 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd6041eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd5fc073730 con 0x7fd5fc155580 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd601f87640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5fc06c830 0x7fd5fc15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd601f87640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5fc06c830 0x7fd5fc15ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60246/0 (socket says 172.21.15.81:60246) 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd601f87640 1 -- 172.21.15.81:0/14395301 learned_addr learned my addr 172.21.15.81:0/14395301 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd601786640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5fc15f9c0 0x7fd5fc1f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd602788640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5fc155580 0x7fd5fc159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd601f87640 1 -- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5fc155580 msgr2=0x7fd5fc159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:02.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd601f87640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5fc155580 0x7fd5fc159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd601f87640 1 -- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5fc15f9c0 msgr2=0x7fd5fc1f2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd601f87640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5fc15f9c0 0x7fd5fc1f2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.250+0000 7fd601f87640 1 -- 172.21.15.81:0/14395301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5fc0aa350 con 0x7fd5fc06c830 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.251+0000 7fd601f87640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5fc06c830 0x7fd5fc15ae00 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4034040 tx=0x7fd5f40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.251+0000 7fd5fa7fc640 1 -- 172.21.15.81:0/14395301 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f403e030 con 0x7fd5fc06c830 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.251+0000 7fd5fa7fc640 1 -- 172.21.15.81:0/14395301 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5f40047a0 con 0x7fd5fc06c830 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.251+0000 7fd5fa7fc640 1 -- 172.21.15.81:0/14395301 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f4035040 con 0x7fd5fc06c830 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.251+0000 7fd6041eb640 1 -- 172.21.15.81:0/14395301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5fc0aa630 con 0x7fd5fc06c830 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.251+0000 7fd6041eb640 1 -- 172.21.15.81:0/14395301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5fc1f33c0 con 0x7fd5fc06c830 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.252+0000 7fd6041eb640 1 -- 172.21.15.81:0/14395301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5fc073c70 con 0x7fd5fc06c830 2024-06-29T15:39:02.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.252+0000 7fd5fa7fc640 1 -- 172.21.15.81:0/14395301 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fd5f4041030 con 0x7fd5fc06c830 2024-06-29T15:39:02.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.253+0000 7fd5fa7fc640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5e80763f0 0x7fd5e8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:02.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.253+0000 7fd5fa7fc640 1 -- 172.21.15.81:0/14395301 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 6145+0+0 (secure 0 0 0) 0x7fd5f40b8460 con 0x7fd5fc06c830 2024-06-29T15:39:02.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.253+0000 7fd601786640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5e80763f0 0x7fd5e8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:02.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.254+0000 7fd601786640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5e80763f0 0x7fd5e8078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000a2e0 tx=0x7fd5f000a270 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:02.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.258+0000 7fd5fa7fc640 1 -- 172.21.15.81:0/14395301 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd5f4081950 con 0x7fd5fc06c830 2024-06-29T15:39:02.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:02.438+0000 7fd6041eb640 1 -- 172.21.15.81:0/14395301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd5fc0af9c0 con 0x7fd5fc06c830 2024-06-29T15:39:03.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.108+0000 7fd5fa7fc640 1 -- 172.21.15.81:0/14395301 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v173) ==== 169+0+0 (secure 0 0 0) 0x7fd5f4081250 con 0x7fd5fc06c830 2024-06-29T15:39:03.109 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T15:39:03.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.112+0000 7fd6041eb640 1 -- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5e80763f0 msgr2=0x7fd5e8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.112+0000 7fd6041eb640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5e80763f0 0x7fd5e8078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd5f000a2e0 tx=0x7fd5f000a270 comp rx=0 tx=0).stop 2024-06-29T15:39:03.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.112+0000 7fd6041eb640 1 -- 172.21.15.81:0/14395301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5fc06c830 msgr2=0x7fd5fc15ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.112+0000 7fd6041eb640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5fc06c830 0x7fd5fc15ae00 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4034040 tx=0x7fd5f40039e0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.112+0000 7fd6041eb640 1 -- 172.21.15.81:0/14395301 shutdown_connections 2024-06-29T15:39:03.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.112+0000 7fd6041eb640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5e80763f0 0x7fd5e8078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.113+0000 7fd6041eb640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd5fc155580 0x7fd5fc159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.113+0000 7fd6041eb640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd5fc15f9c0 0x7fd5fc1f2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.113+0000 7fd6041eb640 1 --2- 172.21.15.81:0/14395301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5fc06c830 0x7fd5fc15ae00 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.113+0000 7fd6041eb640 1 -- 172.21.15.81:0/14395301 >> 172.21.15.81:0/14395301 conn(0x7fd5fc0b2580 msgr2=0x7fd5fc15e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:03.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.113+0000 7fd6041eb640 1 -- 172.21.15.81:0/14395301 shutdown_connections 2024-06-29T15:39:03.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.113+0000 7fd6041eb640 1 -- 172.21.15.81:0/14395301 wait complete. 2024-06-29T15:39:03.127 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:39:03.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.250+0000 7faedfc95640 1 -- 172.21.15.81:0/2715860777 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faed806f850 msgr2=0x7faed80ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.250+0000 7faedfc95640 1 --2- 172.21.15.81:0/2715860777 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faed806f850 0x7faed80ae300 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faec8009830 tx=0x7faec802a510 comp rx=0 tx=0).stop 2024-06-29T15:39:03.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.250+0000 7faedfc95640 1 -- 172.21.15.81:0/2715860777 shutdown_connections 2024-06-29T15:39:03.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.250+0000 7faedfc95640 1 --2- 172.21.15.81:0/2715860777 >> v2:172.21.15.167:3300/0 conn(0x7faed80ae840 0x7faed80aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.250+0000 7faedfc95640 1 --2- 172.21.15.81:0/2715860777 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faed806f850 0x7faed80ae300 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.250+0000 7faedfc95640 1 -- 172.21.15.81:0/2715860777 >> 172.21.15.81:0/2715860777 conn(0x7faed8072950 msgr2=0x7faed80b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:03.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.250+0000 7faedfc95640 1 -- 172.21.15.81:0/2715860777 shutdown_connections 2024-06-29T15:39:03.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.250+0000 7faedfc95640 1 -- 172.21.15.81:0/2715860777 wait complete. 2024-06-29T15:39:03.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.252+0000 7faedfc95640 1 Processor -- start 2024-06-29T15:39:03.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.252+0000 7faedfc95640 1 -- start start 2024-06-29T15:39:03.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.253+0000 7faedfc95640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed806f850 0x7faed814ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.253+0000 7faedfc95640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faed8074520 0x7faed8152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.253+0000 7faedfc95640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faed80ae840 0x7faed8152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.253+0000 7faedfc95640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faed815b190 con 0x7faed806f850 2024-06-29T15:39:03.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.253+0000 7faedfc95640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faed815b010 con 0x7faed80ae840 2024-06-29T15:39:03.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.253+0000 7faedfc95640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faed80af190 con 0x7faed8074520 2024-06-29T15:39:03.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.253+0000 7faedda31640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed806f850 0x7faed814ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedda31640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed806f850 0x7faed814ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60250/0 (socket says 172.21.15.81:60250) 2024-06-29T15:39:03.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedda31640 1 -- 172.21.15.81:0/3216753902 learned_addr learned my addr 172.21.15.81:0/3216753902 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:03.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedd230640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faed8074520 0x7faed8152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faede232640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faed80ae840 0x7faed8152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedda31640 1 -- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faed8074520 msgr2=0x7faed8152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedda31640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faed8074520 0x7faed8152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedda31640 1 -- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faed80ae840 msgr2=0x7faed8152780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedda31640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faed80ae840 0x7faed8152780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedda31640 1 -- 172.21.15.81:0/3216753902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed809d0d0 con 0x7faed806f850 2024-06-29T15:39:03.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedda31640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed806f850 0x7faed814ea00 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7faed400c3d0 tx=0x7faed400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:03.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faece7fc640 1 -- 172.21.15.81:0/3216753902 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed4017070 con 0x7faed806f850 2024-06-29T15:39:03.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faece7fc640 1 -- 172.21.15.81:0/3216753902 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faed40053c0 con 0x7faed806f850 2024-06-29T15:39:03.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faece7fc640 1 -- 172.21.15.81:0/3216753902 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed4003850 con 0x7faed806f850 2024-06-29T15:39:03.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedfc95640 1 -- 172.21.15.81:0/3216753902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed809d3b0 con 0x7faed806f850 2024-06-29T15:39:03.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.254+0000 7faedfc95640 1 -- 172.21.15.81:0/3216753902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed809d970 con 0x7faed806f850 2024-06-29T15:39:03.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.255+0000 7faedfc95640 1 -- 172.21.15.81:0/3216753902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed809ce30 con 0x7faed806f850 2024-06-29T15:39:03.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.256+0000 7faece7fc640 1 -- 172.21.15.81:0/3216753902 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7faed401a070 con 0x7faed806f850 2024-06-29T15:39:03.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.257+0000 7faece7fc640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7faebc076420 0x7faebc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.257+0000 7faece7fc640 1 -- 172.21.15.81:0/3216753902 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5731+0+0 (secure 0 0 0) 0x7faed4096c30 con 0x7faed806f850 2024-06-29T15:39:03.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.257+0000 7faedd230640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7faebc076420 0x7faebc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.258+0000 7faedd230640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7faebc076420 0x7faebc078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faec8034000 tx=0x7faec8004730 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:03.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.261+0000 7faece7fc640 1 -- 172.21.15.81:0/3216753902 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faed405fd90 con 0x7faed806f850 2024-06-29T15:39:03.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.444+0000 7faedfc95640 1 -- 172.21.15.81:0/3216753902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7faed80ae300 con 0x7faed806f850 2024-06-29T15:39:03.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.448+0000 7faece7fc640 1 -- 172.21.15.81:0/3216753902 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v63) ==== 71+0+0 (secure 0 0 0) 0x7faed40683d0 con 0x7faed806f850 2024-06-29T15:39:03.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 -- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7faebc076420 msgr2=0x7faebc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7faebc076420 0x7faebc078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faec8034000 tx=0x7faec8004730 comp rx=0 tx=0).stop 2024-06-29T15:39:03.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 -- 172.21.15.81:0/3216753902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed806f850 msgr2=0x7faed814ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed806f850 0x7faed814ea00 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7faed400c3d0 tx=0x7faed400c890 comp rx=0 tx=0).stop 2024-06-29T15:39:03.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 -- 172.21.15.81:0/3216753902 shutdown_connections 2024-06-29T15:39:03.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7faebc076420 0x7faebc078930 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faed80ae840 0x7faed8152780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faed8074520 0x7faed8152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 --2- 172.21.15.81:0/3216753902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed806f850 0x7faed814ea00 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 -- 172.21.15.81:0/3216753902 >> 172.21.15.81:0/3216753902 conn(0x7faed8072950 msgr2=0x7faed806d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 -- 172.21.15.81:0/3216753902 shutdown_connections 2024-06-29T15:39:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.450+0000 7faedfc95640 1 -- 172.21.15.81:0/3216753902 wait complete. 2024-06-29T15:39:03.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2024-06-29T15:39:03.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.575+0000 7fe5d36ce640 1 -- 172.21.15.81:0/966101866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5cc0b3640 msgr2=0x7fe5cc15ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.575+0000 7fe5d36ce640 1 --2- 172.21.15.81:0/966101866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5cc0b3640 0x7fe5cc15ab20 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0703b0 tx=0x7fe5bc02a980 comp rx=0 tx=0).stop 2024-06-29T15:39:03.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.575+0000 7fe5d36ce640 1 -- 172.21.15.81:0/966101866 shutdown_connections 2024-06-29T15:39:03.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.575+0000 7fe5d36ce640 1 --2- 172.21.15.81:0/966101866 >> v2:172.21.15.167:3300/0 conn(0x7fe5cc06fd70 0x7fe5cc071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.575+0000 7fe5d36ce640 1 --2- 172.21.15.81:0/966101866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5cc0b3640 0x7fe5cc15ab20 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.575+0000 7fe5d36ce640 1 -- 172.21.15.81:0/966101866 >> 172.21.15.81:0/966101866 conn(0x7fe5cc06ccc0 msgr2=0x7fe5cc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:03.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.575+0000 7fe5d36ce640 1 -- 172.21.15.81:0/966101866 shutdown_connections 2024-06-29T15:39:03.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.575+0000 7fe5d36ce640 1 -- 172.21.15.81:0/966101866 wait complete. 2024-06-29T15:39:03.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.577+0000 7fe5d36ce640 1 Processor -- start 2024-06-29T15:39:03.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.577+0000 7fe5d36ce640 1 -- start start 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d36ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5cc0b3640 0x7fe5cc13ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d36ce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5cc15b060 0x7fe5cc14e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d36ce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5cc140dc0 0x7fe5cc14ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d36ce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc06edd0 con 0x7fe5cc15b060 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d36ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc0cf7d0 con 0x7fe5cc0b3640 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d36ce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe5cc06ef50 con 0x7fe5cc140dc0 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d0c69640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5cc15b060 0x7fe5cc14e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d0c69640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5cc15b060 0x7fe5cc14e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60268/0 (socket says 172.21.15.81:60268) 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d0c69640 1 -- 172.21.15.81:0/903004580 learned_addr learned my addr 172.21.15.81:0/903004580 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d146a640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5cc0b3640 0x7fe5cc13ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d1c6b640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5cc140dc0 0x7fe5cc14ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d0c69640 1 -- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5cc140dc0 msgr2=0x7fe5cc14ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.578+0000 7fe5d0c69640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5cc140dc0 0x7fe5cc14ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.579+0000 7fe5d0c69640 1 -- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5cc0b3640 msgr2=0x7fe5cc13ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.579+0000 7fe5d0c69640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5cc0b3640 0x7fe5cc13ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.579+0000 7fe5d0c69640 1 -- 172.21.15.81:0/903004580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc151490 con 0x7fe5cc15b060 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.579+0000 7fe5d1c6b640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5cc140dc0 0x7fe5cc14ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.579+0000 7fe5d0c69640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5cc15b060 0x7fe5cc14e8c0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000b410 tx=0x7fe5c000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.579+0000 7fe5c9ffb640 1 -- 172.21.15.81:0/903004580 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0015020 con 0x7fe5cc15b060 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.579+0000 7fe5c9ffb640 1 -- 172.21.15.81:0/903004580 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5c0011920 con 0x7fe5cc15b060 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.579+0000 7fe5d36ce640 1 -- 172.21.15.81:0/903004580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5cc151710 con 0x7fe5cc15b060 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.579+0000 7fe5c9ffb640 1 -- 172.21.15.81:0/903004580 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0011d70 con 0x7fe5cc15b060 2024-06-29T15:39:03.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.579+0000 7fe5d36ce640 1 -- 172.21.15.81:0/903004580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5cc1482f0 con 0x7fe5cc15b060 2024-06-29T15:39:03.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.580+0000 7fe5d36ce640 1 -- 172.21.15.81:0/903004580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5cc140a20 con 0x7fe5cc15b060 2024-06-29T15:39:03.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.581+0000 7fe5c9ffb640 1 -- 172.21.15.81:0/903004580 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fe5c0018020 con 0x7fe5cc15b060 2024-06-29T15:39:03.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.581+0000 7fe5c9ffb640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe5b80763f0 0x7fe5b8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:03.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.581+0000 7fe5c9ffb640 1 -- 172.21.15.81:0/903004580 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5731+0+0 (secure 0 0 0) 0x7fe5c00978f0 con 0x7fe5cc15b060 2024-06-29T15:39:03.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.582+0000 7fe5d146a640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe5b80763f0 0x7fe5b8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:03.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.582+0000 7fe5d146a640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe5b80763f0 0x7fe5b8078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc035000 tx=0x7fe5bc004ab0 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:03.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.586+0000 7fe5c9ffb640 1 -- 172.21.15.81:0/903004580 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe5c001f020 con 0x7fe5cc15b060 2024-06-29T15:39:03.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:03.836+0000 7fe5d36ce640 1 -- 172.21.15.81:0/903004580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) -- 0x7fe5cc072280 con 0x7fe5cc15b060 2024-06-29T15:39:04.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.110+0000 7fe5c9ffb640 1 -- 172.21.15.81:0/903004580 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2487) ==== 174+0+0 (secure 0 0 0) 0x7fe5cc151490 con 0x7fe5cc15b060 2024-06-29T15:39:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 -- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe5b80763f0 msgr2=0x7fe5b8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe5b80763f0 0x7fe5b8078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc035000 tx=0x7fe5bc004ab0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 -- 172.21.15.81:0/903004580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5cc15b060 msgr2=0x7fe5cc14e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5cc15b060 0x7fe5cc14e8c0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000b410 tx=0x7fe5c000b8d0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 -- 172.21.15.81:0/903004580 shutdown_connections 2024-06-29T15:39:04.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe5b80763f0 0x7fe5b8078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe5cc140dc0 0x7fe5cc14ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5cc15b060 0x7fe5cc14e8c0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 --2- 172.21.15.81:0/903004580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe5cc0b3640 0x7fe5cc13ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 -- 172.21.15.81:0/903004580 >> 172.21.15.81:0/903004580 conn(0x7fe5cc06ccc0 msgr2=0x7fe5cc15e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 -- 172.21.15.81:0/903004580 shutdown_connections 2024-06-29T15:39:04.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.112+0000 7fe5d36ce640 1 -- 172.21.15.81:0/903004580 wait complete. 2024-06-29T15:39:04.147 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2024-06-29T15:39:04.147 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2024-06-29T15:39:04.147 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T15:39:04.147 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:39:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.261+0000 7f95d49ee640 1 -- 172.21.15.81:0/754783664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95b8003a50 msgr2=0x7f95b8003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.261+0000 7f95d49ee640 1 --2- 172.21.15.81:0/754783664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95b8003a50 0x7f95b8003ec0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f95c00016c0 tx=0x7f95b8005830 comp rx=0 tx=0).stop 2024-06-29T15:39:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.261+0000 7f95d49ee640 1 -- 172.21.15.81:0/754783664 shutdown_connections 2024-06-29T15:39:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.261+0000 7f95d49ee640 1 --2- 172.21.15.81:0/754783664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95b8003a50 0x7f95b8003ec0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.261+0000 7f95d49ee640 1 -- 172.21.15.81:0/754783664 >> 172.21.15.81:0/754783664 conn(0x7f95d00b3510 msgr2=0x7f95d00b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.261+0000 7f95d49ee640 1 -- 172.21.15.81:0/754783664 shutdown_connections 2024-06-29T15:39:04.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.261+0000 7f95d49ee640 1 -- 172.21.15.81:0/754783664 wait complete. 2024-06-29T15:39:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.263+0000 7f95d49ee640 1 Processor -- start 2024-06-29T15:39:04.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.263+0000 7f95d49ee640 1 -- start start 2024-06-29T15:39:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.264+0000 7f95d49ee640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95b8003a50 0x7f95d00c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.264+0000 7f95d49ee640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d00af920 0x7f95d01ddda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.264+0000 7f95d49ee640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d01e6a60 0x7f95d01e6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.264+0000 7f95d49ee640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f95d00ce8a0 con 0x7f95b8003a50 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.264+0000 7f95d49ee640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f95d00ce720 con 0x7f95d01e6a60 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.264+0000 7f95d49ee640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f95d00d02d0 con 0x7f95d00af920 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.264+0000 7f95ce59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95b8003a50 0x7f95d00c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.264+0000 7f95ce59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95b8003a50 0x7f95d00c8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60282/0 (socket says 172.21.15.81:60282) 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.264+0000 7f95ce59c640 1 -- 172.21.15.81:0/146489482 learned_addr learned my addr 172.21.15.81:0/146489482 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95cdd9b640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d00af920 0x7f95d01ddda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95ced9d640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d01e6a60 0x7f95d01e6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95ce59c640 1 -- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d00af920 msgr2=0x7f95d01ddda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95ce59c640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d00af920 0x7f95d01ddda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95ce59c640 1 -- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d01e6a60 msgr2=0x7f95d01e6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95ce59c640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d01e6a60 0x7f95d01e6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95ce59c640 1 -- 172.21.15.81:0/146489482 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d01e73b0 con 0x7f95b8003a50 2024-06-29T15:39:04.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95ce59c640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95b8003a50 0x7f95d00c8aa0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f95c002a910 tx=0x7f95c00063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95beffd640 1 -- 172.21.15.81:0/146489482 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0003a00 con 0x7f95b8003a50 2024-06-29T15:39:04.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95beffd640 1 -- 172.21.15.81:0/146489482 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95c0039070 con 0x7f95b8003a50 2024-06-29T15:39:04.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95beffd640 1 -- 172.21.15.81:0/146489482 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0033420 con 0x7f95b8003a50 2024-06-29T15:39:04.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95d49ee640 1 -- 172.21.15.81:0/146489482 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95d00cbd60 con 0x7f95b8003a50 2024-06-29T15:39:04.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.265+0000 7f95d49ee640 1 -- 172.21.15.81:0/146489482 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95d00cb710 con 0x7f95b8003a50 2024-06-29T15:39:04.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.266+0000 7f95d49ee640 1 -- 172.21.15.81:0/146489482 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95d00d0830 con 0x7f95b8003a50 2024-06-29T15:39:04.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.267+0000 7f95beffd640 1 -- 172.21.15.81:0/146489482 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f95c0044020 con 0x7f95b8003a50 2024-06-29T15:39:04.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.267+0000 7f95beffd640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f95b00763f0 0x7f95b0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.267+0000 7f95beffd640 1 -- 172.21.15.81:0/146489482 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5731+0+0 (secure 0 0 0) 0x7f95c00b7220 con 0x7f95b8003a50 2024-06-29T15:39:04.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.268+0000 7f95cdd9b640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f95b00763f0 0x7f95b0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.268+0000 7f95cdd9b640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f95b00763f0 0x7f95b0078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009a80 tx=0x7f95c4005f90 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.272+0000 7f95beffd640 1 -- 172.21.15.81:0/146489482 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f95d01e73b0 con 0x7f95b8003a50 2024-06-29T15:39:04.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.453+0000 7f95d49ee640 1 -- 172.21.15.81:0/146489482 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f95d00cdf00 con 0x7f95b8003a50 2024-06-29T15:39:04.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.454+0000 7f95beffd640 1 -- 172.21.15.81:0/146489482 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) ==== 74+0+14193 (secure 0 0 0) 0x7f95c0080a50 con 0x7f95b8003a50 2024-06-29T15:39:04.455 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:04.455 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":173,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:39:03.101604+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/678307808":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/3068206957":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/3438796129":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/353121370":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/1995307863":"2024-06-30T15:38:57.694381+0000","172.21.15.167:6833/4280924958":"2024-06-30T15:38:57.694381+0000","172.21.15.167:6832/4280924958":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/1753956564":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/1949204156":"2024-06-30T15:38:51.005371+0000","172.21.15.81:6834/1634290596":"2024-06-30T15:38:59.912209+0000","172.21.15.167:0/4139817389":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/3283653249":"2024-06-30T15:38:51.005371+0000","172.21.15.81:6835/1634290596":"2024-06-30T15:38:59.912209+0000","172.21.15.167:0/3855027753":"2024-06-30T15:38:57.694381+0000","172.21.15.167:0/2505002611":"2024-06-30T15:38:51.005371+0000","172.21.15.167:0/4205626971":"2024-06-30T15:38:51.005371+0000","172.21.15.167:6833/150801987":"2024-06-30T15:38:51.005371+0000","172.21.15.167:6832/150801987":"2024-06-30T15:38:51.005371+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:39:04.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.456+0000 7f95d49ee640 1 -- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f95b00763f0 msgr2=0x7f95b0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.456+0000 7f95d49ee640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f95b00763f0 0x7f95b0078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009a80 tx=0x7f95c4005f90 comp rx=0 tx=0).stop 2024-06-29T15:39:04.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.456+0000 7f95d49ee640 1 -- 172.21.15.81:0/146489482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95b8003a50 msgr2=0x7f95d00c8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.456+0000 7f95d49ee640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95b8003a50 0x7f95d00c8aa0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f95c002a910 tx=0x7f95c00063a0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.457+0000 7f95d49ee640 1 -- 172.21.15.81:0/146489482 shutdown_connections 2024-06-29T15:39:04.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.457+0000 7f95d49ee640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f95b00763f0 0x7f95b0078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.457+0000 7f95d49ee640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95d01e6a60 0x7f95d01e6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.457+0000 7f95d49ee640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95d00af920 0x7f95d01ddda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.457+0000 7f95d49ee640 1 --2- 172.21.15.81:0/146489482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95b8003a50 0x7f95d00c8aa0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.457+0000 7f95d49ee640 1 -- 172.21.15.81:0/146489482 >> 172.21.15.81:0/146489482 conn(0x7f95d00b3510 msgr2=0x7f95d00b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.457+0000 7f95d49ee640 1 -- 172.21.15.81:0/146489482 shutdown_connections 2024-06-29T15:39:04.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.457+0000 7f95d49ee640 1 -- 172.21.15.81:0/146489482 wait complete. 2024-06-29T15:39:04.469 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:39:04.470 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:39:04.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.586+0000 7f7dfb89d640 1 -- 172.21.15.81:0/1134036522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7df41412f0 msgr2=0x7f7df40cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.586+0000 7f7dfb89d640 1 --2- 172.21.15.81:0/1134036522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7df41412f0 0x7f7df40cef90 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f7df4071d20 tx=0x7f7de402a8b0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.586+0000 7f7dfb89d640 1 -- 172.21.15.81:0/1134036522 shutdown_connections 2024-06-29T15:39:04.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.586+0000 7f7dfb89d640 1 --2- 172.21.15.81:0/1134036522 >> v2:172.21.15.167:3300/0 conn(0x7f7df4073740 0x7f7df415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.586+0000 7f7dfb89d640 1 --2- 172.21.15.81:0/1134036522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7df41412f0 0x7f7df40cef90 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.586+0000 7f7dfb89d640 1 -- 172.21.15.81:0/1134036522 >> 172.21.15.81:0/1134036522 conn(0x7f7df406ccc0 msgr2=0x7f7df40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.586+0000 7f7dfb89d640 1 -- 172.21.15.81:0/1134036522 shutdown_connections 2024-06-29T15:39:04.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.586+0000 7f7dfb89d640 1 -- 172.21.15.81:0/1134036522 wait complete. 2024-06-29T15:39:04.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.588+0000 7f7dfb89d640 1 Processor -- start 2024-06-29T15:39:04.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.588+0000 7f7dfb89d640 1 -- start start 2024-06-29T15:39:04.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7dfb89d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7df40cf4d0 0x7f7df413feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7dfb89d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7df41412f0 0x7f7df4158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7dfb89d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7df4155f10 0x7f7df4156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7dfb89d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7df40b3210 con 0x7f7df4155f10 2024-06-29T15:39:04.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7dfb89d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7df415f1f0 con 0x7f7df41412f0 2024-06-29T15:39:04.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7dfb89d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7df40b3390 con 0x7f7df40cf4d0 2024-06-29T15:39:04.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df9e3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7df4155f10 0x7f7df4156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df9e3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7df4155f10 0x7f7df4156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60310/0 (socket says 172.21.15.81:60310) 2024-06-29T15:39:04.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df9e3a640 1 -- 172.21.15.81:0/3279849857 learned_addr learned my addr 172.21.15.81:0/3279849857 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df9639640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7df40cf4d0 0x7f7df413feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df8e38640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7df41412f0 0x7f7df4158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df9e3a640 1 -- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7df40cf4d0 msgr2=0x7f7df413feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df9e3a640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7df40cf4d0 0x7f7df413feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df9e3a640 1 -- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7df41412f0 msgr2=0x7f7df4158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df9e3a640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7df41412f0 0x7f7df4158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df9e3a640 1 -- 172.21.15.81:0/3279849857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df4156860 con 0x7f7df4155f10 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.589+0000 7f7df9639640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7df40cf4d0 0x7f7df413feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.590+0000 7f7df9e3a640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7df4155f10 0x7f7df4156320 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00b440 tx=0x7f7dec00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.590+0000 7f7df1ffb640 1 -- 172.21.15.81:0/3279849857 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec015020 con 0x7f7df4155f10 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.590+0000 7f7df1ffb640 1 -- 172.21.15.81:0/3279849857 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7dec011920 con 0x7f7df4155f10 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.590+0000 7f7dfb89d640 1 -- 172.21.15.81:0/3279849857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df40af2c0 con 0x7f7df4155f10 2024-06-29T15:39:04.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.590+0000 7f7df1ffb640 1 -- 172.21.15.81:0/3279849857 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec011d70 con 0x7f7df4155f10 2024-06-29T15:39:04.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.590+0000 7f7dfb89d640 1 -- 172.21.15.81:0/3279849857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df4154a00 con 0x7f7df4155f10 2024-06-29T15:39:04.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.591+0000 7f7dfb89d640 1 -- 172.21.15.81:0/3279849857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df40b3910 con 0x7f7df4155f10 2024-06-29T15:39:04.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.592+0000 7f7df1ffb640 1 -- 172.21.15.81:0/3279849857 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f7dec018020 con 0x7f7df4155f10 2024-06-29T15:39:04.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.592+0000 7f7df1ffb640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7de00763f0 0x7f7de0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.592+0000 7f7df1ffb640 1 -- 172.21.15.81:0/3279849857 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5731+0+0 (secure 0 0 0) 0x7f7dec097840 con 0x7f7df4155f10 2024-06-29T15:39:04.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.592+0000 7f7df9639640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7de00763f0 0x7f7de0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.593+0000 7f7df9639640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7de00763f0 0x7f7de0078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7de4034000 tx=0x7f7de4004ab0 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.596+0000 7f7df1ffb640 1 -- 172.21.15.81:0/3279849857 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7dec01f020 con 0x7f7df4155f10 2024-06-29T15:39:04.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.774+0000 7f7dfb89d640 1 -- 172.21.15.81:0/3279849857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7df4159820 con 0x7f7df4155f10 2024-06-29T15:39:04.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.774+0000 7f7df1ffb640 1 -- 172.21.15.81:0/3279849857 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v173) ==== 93+0+10 (secure 0 0 0) 0x7f7df4156860 con 0x7f7df4155f10 2024-06-29T15:39:04.775 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:39:04.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.776+0000 7f7dfb89d640 1 -- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7de00763f0 msgr2=0x7f7de0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.776+0000 7f7dfb89d640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7de00763f0 0x7f7de0078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7de4034000 tx=0x7f7de4004ab0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.776+0000 7f7dfb89d640 1 -- 172.21.15.81:0/3279849857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7df4155f10 msgr2=0x7f7df4156320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.776+0000 7f7dfb89d640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7df4155f10 0x7f7df4156320 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00b440 tx=0x7f7dec00b840 comp rx=0 tx=0).stop 2024-06-29T15:39:04.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.777+0000 7f7dfb89d640 1 -- 172.21.15.81:0/3279849857 shutdown_connections 2024-06-29T15:39:04.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.777+0000 7f7dfb89d640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7de00763f0 0x7f7de0078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.777+0000 7f7dfb89d640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7df4155f10 0x7f7df4156320 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.777+0000 7f7dfb89d640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7df41412f0 0x7f7df4158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.777+0000 7f7dfb89d640 1 --2- 172.21.15.81:0/3279849857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7df40cf4d0 0x7f7df413feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.777+0000 7f7dfb89d640 1 -- 172.21.15.81:0/3279849857 >> 172.21.15.81:0/3279849857 conn(0x7f7df406ccc0 msgr2=0x7f7df40b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.777+0000 7f7dfb89d640 1 -- 172.21.15.81:0/3279849857 shutdown_connections 2024-06-29T15:39:04.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.777+0000 7f7dfb89d640 1 -- 172.21.15.81:0/3279849857 wait complete. 2024-06-29T15:39:04.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-06-29T15:39:04.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.901+0000 7f4a27e65640 1 -- 172.21.15.81:0/2095293388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a20154fc0 msgr2=0x7f4a2006c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.901+0000 7f4a27e65640 1 --2- 172.21.15.81:0/2095293388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a20154fc0 0x7f4a2006c2f0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f4a20072700 tx=0x7f4a1402a4d0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.901+0000 7f4a27e65640 1 -- 172.21.15.81:0/2095293388 shutdown_connections 2024-06-29T15:39:04.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.901+0000 7f4a27e65640 1 --2- 172.21.15.81:0/2095293388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a2006c830 0x7f4a200af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.901+0000 7f4a27e65640 1 --2- 172.21.15.81:0/2095293388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a20154fc0 0x7f4a2006c2f0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.901+0000 7f4a27e65640 1 --2- 172.21.15.81:0/2095293388 >> v2:172.21.15.167:3300/0 conn(0x7f4a2015e710 0x7f4a2006f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.902+0000 7f4a27e65640 1 -- 172.21.15.81:0/2095293388 >> 172.21.15.81:0/2095293388 conn(0x7f4a200d0930 msgr2=0x7f4a200d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:04.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.902+0000 7f4a27e65640 1 -- 172.21.15.81:0/2095293388 shutdown_connections 2024-06-29T15:39:04.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.902+0000 7f4a27e65640 1 -- 172.21.15.81:0/2095293388 wait complete. 2024-06-29T15:39:04.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.903+0000 7f4a27e65640 1 Processor -- start 2024-06-29T15:39:04.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.903+0000 7f4a27e65640 1 -- start start 2024-06-29T15:39:04.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a27e65640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a2006c830 0x7f4a20151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a27e65640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a20154fc0 0x7f4a2014d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a27e65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2015e710 0x7f4a200cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a27e65640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4a200affb0 con 0x7f4a2015e710 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a27e65640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4a200afe30 con 0x7f4a2006c830 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a27e65640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4a200b0130 con 0x7f4a20154fc0 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a26402640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2015e710 0x7f4a200cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a26402640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2015e710 0x7f4a200cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60334/0 (socket says 172.21.15.81:60334) 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a26402640 1 -- 172.21.15.81:0/2193981237 learned_addr learned my addr 172.21.15.81:0/2193981237 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a25400640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a20154fc0 0x7f4a2014d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a25c01640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a2006c830 0x7f4a20151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a26402640 1 -- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a20154fc0 msgr2=0x7f4a2014d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a26402640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a20154fc0 0x7f4a2014d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a26402640 1 -- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a2006c830 msgr2=0x7f4a20151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a26402640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a2006c830 0x7f4a20151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:04.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a26402640 1 -- 172.21.15.81:0/2193981237 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2014f540 con 0x7f4a2015e710 2024-06-29T15:39:04.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a25c01640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a2006c830 0x7f4a20151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:04.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a26402640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2015e710 0x7f4a200cf510 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800cab0 tx=0x7f4a1800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a127fc640 1 -- 172.21.15.81:0/2193981237 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a18007df0 con 0x7f4a2015e710 2024-06-29T15:39:04.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a127fc640 1 -- 172.21.15.81:0/2193981237 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a1800f560 con 0x7f4a2015e710 2024-06-29T15:39:04.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a27e65640 1 -- 172.21.15.81:0/2193981237 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2014f820 con 0x7f4a2015e710 2024-06-29T15:39:04.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.905+0000 7f4a127fc640 1 -- 172.21.15.81:0/2193981237 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1800f9b0 con 0x7f4a2015e710 2024-06-29T15:39:04.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.906+0000 7f4a27e65640 1 -- 172.21.15.81:0/2193981237 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2014efd0 con 0x7f4a2015e710 2024-06-29T15:39:04.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.906+0000 7f4a27e65640 1 -- 172.21.15.81:0/2193981237 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a2006e2a0 con 0x7f4a2015e710 2024-06-29T15:39:04.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.907+0000 7f4a127fc640 1 -- 172.21.15.81:0/2193981237 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f4a1800fb50 con 0x7f4a2015e710 2024-06-29T15:39:04.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.908+0000 7f4a127fc640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4a04076420 0x7f4a04078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:04.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.908+0000 7f4a127fc640 1 -- 172.21.15.81:0/2193981237 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5731+0+0 (secure 0 0 0) 0x7f4a180983a0 con 0x7f4a2015e710 2024-06-29T15:39:04.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.908+0000 7f4a25c01640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4a04076420 0x7f4a04078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:04.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.909+0000 7f4a25c01640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4a04076420 0x7f4a04078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c005fd0 tx=0x7f4a1c00f360 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:04.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:04.912+0000 7f4a127fc640 1 -- 172.21.15.81:0/2193981237 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4a2006e2a0 con 0x7f4a2015e710 2024-06-29T15:39:05.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:05.163+0000 7f4a27e65640 1 -- 172.21.15.81:0/2193981237 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) -- 0x7f4a2006c2f0 con 0x7f4a2015e710 2024-06-29T15:39:06.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.120+0000 7f4a127fc640 1 -- 172.21.15.81:0/2193981237 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2489) ==== 168+0+0 (secure 0 0 0) 0x7f4a18014070 con 0x7f4a2015e710 2024-06-29T15:39:06.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.122+0000 7f4a27e65640 1 -- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4a04076420 msgr2=0x7f4a04078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.122+0000 7f4a27e65640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4a04076420 0x7f4a04078930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c005fd0 tx=0x7f4a1c00f360 comp rx=0 tx=0).stop 2024-06-29T15:39:06.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.122+0000 7f4a27e65640 1 -- 172.21.15.81:0/2193981237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2015e710 msgr2=0x7f4a200cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.122+0000 7f4a27e65640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2015e710 0x7f4a200cf510 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800cab0 tx=0x7f4a1800cf70 comp rx=0 tx=0).stop 2024-06-29T15:39:06.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.122+0000 7f4a27e65640 1 -- 172.21.15.81:0/2193981237 shutdown_connections 2024-06-29T15:39:06.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.122+0000 7f4a27e65640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4a04076420 0x7f4a04078930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.122+0000 7f4a27e65640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2015e710 0x7f4a200cf510 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.122+0000 7f4a27e65640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4a20154fc0 0x7f4a2014d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.122+0000 7f4a27e65640 1 --2- 172.21.15.81:0/2193981237 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4a2006c830 0x7f4a20151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.122+0000 7f4a27e65640 1 -- 172.21.15.81:0/2193981237 >> 172.21.15.81:0/2193981237 conn(0x7f4a200d0930 msgr2=0x7f4a200adb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:06.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.123+0000 7f4a27e65640 1 -- 172.21.15.81:0/2193981237 shutdown_connections 2024-06-29T15:39:06.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.123+0000 7f4a27e65640 1 -- 172.21.15.81:0/2193981237 wait complete. 2024-06-29T15:39:06.136 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:39:06.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.251+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1239712430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b815e960 msgr2=0x7fb3b815ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.251+0000 7fb3be5aa640 1 --2- 172.21.15.81:0/1239712430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b815e960 0x7fb3b815ed70 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0005fd0 tx=0x7fb3b002aa00 comp rx=0 tx=0).stop 2024-06-29T15:39:06.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.251+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1239712430 shutdown_connections 2024-06-29T15:39:06.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.251+0000 7fb3be5aa640 1 --2- 172.21.15.81:0/1239712430 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3b8073030 0x7fb3b8073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.251+0000 7fb3be5aa640 1 --2- 172.21.15.81:0/1239712430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b815e960 0x7fb3b815ed70 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.251+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1239712430 >> 172.21.15.81:0/1239712430 conn(0x7fb3b806ccc0 msgr2=0x7fb3b80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:06.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.251+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1239712430 shutdown_connections 2024-06-29T15:39:06.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.251+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1239712430 wait complete. 2024-06-29T15:39:06.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.252+0000 7fb3be5aa640 1 Processor -- start 2024-06-29T15:39:06.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.253+0000 7fb3be5aa640 1 -- start start 2024-06-29T15:39:06.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3be5aa640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3b8073030 0x7fb3b80aa810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3be5aa640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3b81412f0 0x7fb3b81f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3be5aa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b815e960 0x7fb3b80ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3be5aa640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b80b3210 con 0x7fb3b815e960 2024-06-29T15:39:06.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3be5aa640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b80cf720 con 0x7fb3b8073030 2024-06-29T15:39:06.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3be5aa640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb3b80b3390 con 0x7fb3b81412f0 2024-06-29T15:39:06.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3bcb47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b815e960 0x7fb3b80ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3bcb47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b815e960 0x7fb3b80ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60342/0 (socket says 172.21.15.81:60342) 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3bcb47640 1 -- 172.21.15.81:0/1118544929 learned_addr learned my addr 172.21.15.81:0/1118544929 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3b7fff640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3b8073030 0x7fb3b80aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3b77fe640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3b81412f0 0x7fb3b81f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3bcb47640 1 -- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3b81412f0 msgr2=0x7fb3b81f3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3bcb47640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3b81412f0 0x7fb3b81f3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3bcb47640 1 -- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3b8073030 msgr2=0x7fb3b80aa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3bcb47640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3b8073030 0x7fb3b80aa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.254+0000 7fb3bcb47640 1 -- 172.21.15.81:0/1118544929 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b8158ed0 con 0x7fb3b815e960 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.255+0000 7fb3bcb47640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b815e960 0x7fb3b80ae040 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0006010 tx=0x7fb3b0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.255+0000 7fb3b4ff9640 1 -- 172.21.15.81:0/1118544929 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0046070 con 0x7fb3b815e960 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.255+0000 7fb3b4ff9640 1 -- 172.21.15.81:0/1118544929 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3b0004540 con 0x7fb3b815e960 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.255+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1118544929 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3b81f4140 con 0x7fb3b815e960 2024-06-29T15:39:06.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.255+0000 7fb3b4ff9640 1 -- 172.21.15.81:0/1118544929 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0033070 con 0x7fb3b815e960 2024-06-29T15:39:06.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.255+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1118544929 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3b80ad560 con 0x7fb3b815e960 2024-06-29T15:39:06.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.256+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1118544929 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3b8070230 con 0x7fb3b815e960 2024-06-29T15:39:06.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.256+0000 7fb3b4ff9640 1 -- 172.21.15.81:0/1118544929 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fb3b0049070 con 0x7fb3b815e960 2024-06-29T15:39:06.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.257+0000 7fb3b4ff9640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fb3a40763f0 0x7fb3a4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.257+0000 7fb3b4ff9640 1 -- 172.21.15.81:0/1118544929 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5731+0+0 (secure 0 0 0) 0x7fb3b00b8980 con 0x7fb3b815e960 2024-06-29T15:39:06.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.257+0000 7fb3b7fff640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fb3a40763f0 0x7fb3a4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.258+0000 7fb3b7fff640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fb3a40763f0 0x7fb3a4078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8006fd0 tx=0x7fb3a8009060 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:06.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.262+0000 7fb3b4ff9640 1 -- 172.21.15.81:0/1118544929 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb3b8070230 con 0x7fb3b815e960 2024-06-29T15:39:06.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.441+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1118544929 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fb3b80b3890 con 0x7fb3b815e960 2024-06-29T15:39:06.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.441+0000 7fb3b4ff9640 1 -- 172.21.15.81:0/1118544929 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v63) ==== 59+0+1998 (secure 0 0 0) 0x7fb3b8158ed0 con 0x7fb3b815e960 2024-06-29T15:39:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:39:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:39:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:39:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:39:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T15:39:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T15:39:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T15:39:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:39:06.443 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:39:06.444 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T15:39:06.444 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T15:39:06.444 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T15:39:06.444 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:39:06.444 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fb3a40763f0 msgr2=0x7fb3a4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fb3a40763f0 0x7fb3a4078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8006fd0 tx=0x7fb3a8009060 comp rx=0 tx=0).stop 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1118544929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b815e960 msgr2=0x7fb3b80ae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b815e960 0x7fb3b80ae040 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0006010 tx=0x7fb3b0003f10 comp rx=0 tx=0).stop 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1118544929 shutdown_connections 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fb3a40763f0 0x7fb3a4078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b815e960 0x7fb3b80ae040 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3b81412f0 0x7fb3b81f3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 --2- 172.21.15.81:0/1118544929 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3b8073030 0x7fb3b80aa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1118544929 >> 172.21.15.81:0/1118544929 conn(0x7fb3b806ccc0 msgr2=0x7fb3b806dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1118544929 shutdown_connections 2024-06-29T15:39:06.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.443+0000 7fb3be5aa640 1 -- 172.21.15.81:0/1118544929 wait complete. 2024-06-29T15:39:06.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:39:06.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.574+0000 7f49885e0640 1 -- 172.21.15.81:0/3448608638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f498015d750 msgr2=0x7f49800afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.574+0000 7f49885e0640 1 --2- 172.21.15.81:0/3448608638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f498015d750 0x7f49800afb60 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f49800cd380 tx=0x7f497c02a580 comp rx=0 tx=0).stop 2024-06-29T15:39:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.574+0000 7f49885e0640 1 -- 172.21.15.81:0/3448608638 shutdown_connections 2024-06-29T15:39:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.574+0000 7f49885e0640 1 --2- 172.21.15.81:0/3448608638 >> v2:172.21.15.167:3300/0 conn(0x7f498015c440 0x7f498006c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.574+0000 7f49885e0640 1 --2- 172.21.15.81:0/3448608638 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f498015c030 0x7f498006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.574+0000 7f49885e0640 1 --2- 172.21.15.81:0/3448608638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f498015d750 0x7f49800afb60 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.574+0000 7f49885e0640 1 -- 172.21.15.81:0/3448608638 >> 172.21.15.81:0/3448608638 conn(0x7f49800d0930 msgr2=0x7f49800d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.574+0000 7f49885e0640 1 -- 172.21.15.81:0/3448608638 shutdown_connections 2024-06-29T15:39:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.575+0000 7f49885e0640 1 -- 172.21.15.81:0/3448608638 wait complete. 2024-06-29T15:39:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.576+0000 7f49885e0640 1 Processor -- start 2024-06-29T15:39:06.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.576+0000 7f49885e0640 1 -- start start 2024-06-29T15:39:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f49885e0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f498015c030 0x7f4980155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f49885e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f498015c440 0x7f4980156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f49885e0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f498015d750 0x7f49801f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f49885e0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f498015fff0 con 0x7f498015c440 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f49885e0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f498015fe70 con 0x7f498015d750 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f49885e0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4980160170 con 0x7f498015c030 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4985b7b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f498015c440 0x7f4980156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4985b7b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f498015c440 0x7f4980156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60360/0 (socket says 172.21.15.81:60360) 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4985b7b640 1 -- 172.21.15.81:0/3139814800 learned_addr learned my addr 172.21.15.81:0/3139814800 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f498637c640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f498015c030 0x7f4980155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4986b7d640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f498015d750 0x7f49801f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4985b7b640 1 -- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f498015c030 msgr2=0x7f4980155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4985b7b640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f498015c030 0x7f4980155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4985b7b640 1 -- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f498015d750 msgr2=0x7f49801f6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4985b7b640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f498015d750 0x7f49801f6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4985b7b640 1 -- 172.21.15.81:0/3139814800 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49801f3880 con 0x7f498015c440 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4985b7b640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f498015c440 0x7f4980156d70 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f497000b410 tx=0x7f497000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4976ffd640 1 -- 172.21.15.81:0/3139814800 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970015020 con 0x7f498015c440 2024-06-29T15:39:06.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4976ffd640 1 -- 172.21.15.81:0/3139814800 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4970011940 con 0x7f498015c440 2024-06-29T15:39:06.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f49885e0640 1 -- 172.21.15.81:0/3139814800 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49801f3b60 con 0x7f498015c440 2024-06-29T15:39:06.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f4976ffd640 1 -- 172.21.15.81:0/3139814800 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970011d90 con 0x7f498015c440 2024-06-29T15:39:06.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.578+0000 7f49885e0640 1 -- 172.21.15.81:0/3139814800 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49801f6930 con 0x7f498015c440 2024-06-29T15:39:06.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.579+0000 7f49885e0640 1 -- 172.21.15.81:0/3139814800 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49801f35f0 con 0x7f498015c440 2024-06-29T15:39:06.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.580+0000 7f4976ffd640 1 -- 172.21.15.81:0/3139814800 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f4970018020 con 0x7f498015c440 2024-06-29T15:39:06.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.581+0000 7f4976ffd640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4964076420 0x7f4964078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:06.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.581+0000 7f4976ffd640 1 -- 172.21.15.81:0/3139814800 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5731+0+0 (secure 0 0 0) 0x7f49700978e0 con 0x7f498015c440 2024-06-29T15:39:06.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.581+0000 7f498637c640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4964076420 0x7f4964078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:06.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.582+0000 7f498637c640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4964076420 0x7f4964078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f497c033000 tx=0x7f497c004730 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:06.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.585+0000 7f4976ffd640 1 -- 172.21.15.81:0/3139814800 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f497001f020 con 0x7f498015c440 2024-06-29T15:39:06.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.767+0000 7f49885e0640 1 -- 172.21.15.81:0/3139814800 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f49800716c0 con 0x7f498015c440 2024-06-29T15:39:06.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.771+0000 7f4976ffd640 1 -- 172.21.15.81:0/3139814800 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) ==== 122+0+0 (secure 0 0 0) 0x7f49800716c0 con 0x7f498015c440 2024-06-29T15:39:06.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.773+0000 7f49885e0640 1 -- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4964076420 msgr2=0x7f4964078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.773+0000 7f49885e0640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4964076420 0x7f4964078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f497c033000 tx=0x7f497c004730 comp rx=0 tx=0).stop 2024-06-29T15:39:06.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.773+0000 7f49885e0640 1 -- 172.21.15.81:0/3139814800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f498015c440 msgr2=0x7f4980156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:06.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.773+0000 7f49885e0640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f498015c440 0x7f4980156d70 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f497000b410 tx=0x7f497000b8d0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.774+0000 7f49885e0640 1 -- 172.21.15.81:0/3139814800 shutdown_connections 2024-06-29T15:39:06.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.774+0000 7f49885e0640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4964076420 0x7f4964078930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.774+0000 7f49885e0640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f498015d750 0x7f49801f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.774+0000 7f49885e0640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f498015c440 0x7f4980156d70 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.774+0000 7f49885e0640 1 --2- 172.21.15.81:0/3139814800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f498015c030 0x7f4980155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:06.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.774+0000 7f49885e0640 1 -- 172.21.15.81:0/3139814800 >> 172.21.15.81:0/3139814800 conn(0x7f49800d0930 msgr2=0x7f49800b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:06.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.774+0000 7f49885e0640 1 -- 172.21.15.81:0/3139814800 shutdown_connections 2024-06-29T15:39:06.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:06.774+0000 7f49885e0640 1 -- 172.21.15.81:0/3139814800 wait complete. 2024-06-29T15:39:06.785 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:39:06.785 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:39:06.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:39:06.877 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:39:06.877 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:39:06.877 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:39:06.893 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:39:06.893 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:39:06.894 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:39:06.936 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:39:06.936 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:39:06.936 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:39:06.952 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:39:06.952 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:39:06.952 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:39:06.994 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:39:06.995 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:39:06.995 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:39:07.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.107+0000 7fa29963d640 1 -- 172.21.15.81:0/662805444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2940d46f0 msgr2=0x7fa2940d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.107+0000 7fa29963d640 1 --2- 172.21.15.81:0/662805444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2940d46f0 0x7fa2940d0410 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fa288009830 tx=0x7fa28802a5a0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.107+0000 7fa29963d640 1 -- 172.21.15.81:0/662805444 shutdown_connections 2024-06-29T15:39:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.107+0000 7fa29963d640 1 --2- 172.21.15.81:0/662805444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2940d46f0 0x7fa2940d0410 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.107+0000 7fa29963d640 1 -- 172.21.15.81:0/662805444 >> 172.21.15.81:0/662805444 conn(0x7fa2940d3bd0 msgr2=0x7fa2940d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.107+0000 7fa29963d640 1 -- 172.21.15.81:0/662805444 shutdown_connections 2024-06-29T15:39:07.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.107+0000 7fa29963d640 1 -- 172.21.15.81:0/662805444 wait complete. 2024-06-29T15:39:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.109+0000 7fa29963d640 1 Processor -- start 2024-06-29T15:39:07.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.109+0000 7fa29963d640 1 -- start start 2024-06-29T15:39:07.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa29963d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2940b8ba0 0x7fa2940b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa29963d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2940d0950 0x7fa2940cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa29963d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2940d46f0 0x7fa2940a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa29963d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa2940b9410 con 0x7fa2940d46f0 2024-06-29T15:39:07.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa29963d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa2940b9290 con 0x7fa2940b8ba0 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa29963d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa2940cfcd0 con 0x7fa2940d0950 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa2937fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2940d46f0 0x7fa2940a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa2937fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2940d46f0 0x7fa2940a0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60372/0 (socket says 172.21.15.81:60372) 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa2937fe640 1 -- 172.21.15.81:0/3243105832 learned_addr learned my addr 172.21.15.81:0/3243105832 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa292ffd640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2940b8ba0 0x7fa2940b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa2927fc640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2940d0950 0x7fa2940cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa2937fe640 1 -- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2940d0950 msgr2=0x7fa2940cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa2937fe640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2940d0950 0x7fa2940cdb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa2937fe640 1 -- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2940b8ba0 msgr2=0x7fa2940b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa2937fe640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2940b8ba0 0x7fa2940b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa2937fe640 1 -- 172.21.15.81:0/3243105832 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2940b0fa0 con 0x7fa2940d46f0 2024-06-29T15:39:07.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.110+0000 7fa292ffd640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2940b8ba0 0x7fa2940b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:07.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.111+0000 7fa2937fe640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2940d46f0 0x7fa2940a0d30 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00b410 tx=0x7fa28c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:07.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.111+0000 7fa27b7fe640 1 -- 172.21.15.81:0/3243105832 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c015020 con 0x7fa2940d46f0 2024-06-29T15:39:07.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.111+0000 7fa27b7fe640 1 -- 172.21.15.81:0/3243105832 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa28c011960 con 0x7fa2940d46f0 2024-06-29T15:39:07.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.111+0000 7fa27b7fe640 1 -- 172.21.15.81:0/3243105832 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c011db0 con 0x7fa2940d46f0 2024-06-29T15:39:07.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.111+0000 7fa29963d640 1 -- 172.21.15.81:0/3243105832 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2940b1280 con 0x7fa2940d46f0 2024-06-29T15:39:07.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.111+0000 7fa29963d640 1 -- 172.21.15.81:0/3243105832 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2940b1840 con 0x7fa2940d46f0 2024-06-29T15:39:07.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.112+0000 7fa29963d640 1 -- 172.21.15.81:0/3243105832 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2940ce050 con 0x7fa2940d46f0 2024-06-29T15:39:07.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.113+0000 7fa27b7fe640 1 -- 172.21.15.81:0/3243105832 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fa28c018020 con 0x7fa2940d46f0 2024-06-29T15:39:07.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.113+0000 7fa27b7fe640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa280076420 0x7fa280078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.113+0000 7fa27b7fe640 1 -- 172.21.15.81:0/3243105832 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5731+0+0 (secure 0 0 0) 0x7fa28c097500 con 0x7fa2940d46f0 2024-06-29T15:39:07.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.113+0000 7fa292ffd640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa280076420 0x7fa280078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.114+0000 7fa292ffd640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa280076420 0x7fa280078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa284007fd0 tx=0x7fa284009040 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:07.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.117+0000 7fa27b7fe640 1 -- 172.21.15.81:0/3243105832 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa28c01f030 con 0x7fa2940d46f0 2024-06-29T15:39:07.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.336+0000 7fa29963d640 1 -- 172.21.15.81:0/3243105832 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2940d0410 con 0x7fa2940d46f0 2024-06-29T15:39:07.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.336+0000 7fa27b7fe640 1 -- 172.21.15.81:0/3243105832 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+3918 (secure 0 0 0) 0x7fa28c060cb0 con 0x7fa2940d46f0 2024-06-29T15:39:07.337 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:07.337 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":88,"btime":"2024-06-29T15:39:00:914154+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7754,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3261744821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3261744821},{"type":"v1","addr":"172.21.15.167:6836","nonce":3261744821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":7757,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1617521107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1617521107},{"type":"v1","addr":"172.21.15.167:6837","nonce":1617521107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81},{"gid":9636,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/3224768380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3224768380},{"type":"v1","addr":"172.21.15.81:6833","nonce":3224768380}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10074,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2703362264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2703362264},{"type":"v1","addr":"172.21.15.81:6835","nonce":2703362264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":88}],"filesystems":[]} 2024-06-29T15:39:07.337 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 88 2024-06-29T15:39:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 -- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa280076420 msgr2=0x7fa280078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa280076420 0x7fa280078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa284007fd0 tx=0x7fa284009040 comp rx=0 tx=0).stop 2024-06-29T15:39:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 -- 172.21.15.81:0/3243105832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2940d46f0 msgr2=0x7fa2940a0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2940d46f0 0x7fa2940a0d30 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00b410 tx=0x7fa28c00b8d0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 -- 172.21.15.81:0/3243105832 shutdown_connections 2024-06-29T15:39:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fa280076420 0x7fa280078930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2940d46f0 0x7fa2940a0d30 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2940d0950 0x7fa2940cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 --2- 172.21.15.81:0/3243105832 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2940b8ba0 0x7fa2940b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 -- 172.21.15.81:0/3243105832 >> 172.21.15.81:0/3243105832 conn(0x7fa2940d3bd0 msgr2=0x7fa29406d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:07.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 -- 172.21.15.81:0/3243105832 shutdown_connections 2024-06-29T15:39:07.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.338+0000 7fa29963d640 1 -- 172.21.15.81:0/3243105832 wait complete. 2024-06-29T15:39:07.350 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:39:07.350 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:39:07.351 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:39:07.351 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:39:07.351 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:39:07.351 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:39:07.351 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:39:07.351 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:39:07.352 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:39:07.352 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:39:07.352 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:39:07.352 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:39:07.352 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:39:07.352 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:39:07.353 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:39:07.353 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:39:07.360 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:39:07.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:39:07.363 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:39:07.363 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:39:07.365 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:39:07.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:39:07.366 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:39:07.366 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:39:07.405 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:39:07.406 DEBUG:teuthology.parallel:result is None 2024-06-29T15:39:07.407 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:39:07.407 DEBUG:teuthology.parallel:result is None 2024-06-29T15:39:07.408 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:39:07.408 DEBUG:teuthology.parallel:result is None 2024-06-29T15:39:07.409 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:39:07.409 DEBUG:teuthology.parallel:result is None 2024-06-29T15:39:07.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:39:07.492 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T15:39:07.493 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T15:39:07.506 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:39:07.506 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T15:39:07.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.529+0000 7f2db3b63640 1 -- 172.21.15.81:0/303609416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dac0b9040 msgr2=0x7f2dac0d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.529+0000 7f2db3b63640 1 --2- 172.21.15.81:0/303609416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dac0b9040 0x7f2dac0d0480 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f2da8005fd0 tx=0x7f2da802a5c0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.529+0000 7f2db3b63640 1 -- 172.21.15.81:0/303609416 shutdown_connections 2024-06-29T15:39:07.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.529+0000 7f2db3b63640 1 --2- 172.21.15.81:0/303609416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dac0b5250 0x7f2dac0d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.529+0000 7f2db3b63640 1 --2- 172.21.15.81:0/303609416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dac0b9040 0x7f2dac0d0480 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.529+0000 7f2db3b63640 1 -- 172.21.15.81:0/303609416 >> 172.21.15.81:0/303609416 conn(0x7f2dac09cf30 msgr2=0x7f2dac0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:07.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.529+0000 7f2db3b63640 1 -- 172.21.15.81:0/303609416 shutdown_connections 2024-06-29T15:39:07.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.530+0000 7f2db3b63640 1 -- 172.21.15.81:0/303609416 wait complete. 2024-06-29T15:39:07.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.531+0000 7f2db3b63640 1 Processor -- start 2024-06-29T15:39:07.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.531+0000 7f2db3b63640 1 -- start start 2024-06-29T15:39:07.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.532+0000 7f2db3b63640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dac0b5250 0x7f2dac0ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.532+0000 7f2db3b63640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2dac0b7290 0x7f2dac151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.532+0000 7f2db3b63640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dac0b9040 0x7f2dac151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.532+0000 7f2db3b63640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac0b7db0 con 0x7f2dac0b5250 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.532+0000 7f2db3b63640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac0d2d50 con 0x7f2dac0b7290 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.532+0000 7f2db3b63640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2dac0b7f30 con 0x7f2dac0b9040 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db18ff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dac0b5250 0x7f2dac0ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db18ff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dac0b5250 0x7f2dac0ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60426/0 (socket says 172.21.15.81:60426) 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db18ff640 1 -- 172.21.15.81:0/1713000421 learned_addr learned my addr 172.21.15.81:0/1713000421 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db2100640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dac0b9040 0x7f2dac151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db10fe640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2dac0b7290 0x7f2dac151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db18ff640 1 -- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dac0b9040 msgr2=0x7f2dac151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db18ff640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dac0b9040 0x7f2dac151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db18ff640 1 -- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2dac0b7290 msgr2=0x7f2dac151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:07.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db18ff640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2dac0b7290 0x7f2dac151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:07.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db18ff640 1 -- 172.21.15.81:0/1713000421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac0c8950 con 0x7f2dac0b5250 2024-06-29T15:39:07.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db18ff640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dac0b5250 0x7f2dac0ca6c0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f2da8000c00 tx=0x7f2da8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:07.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2da27fc640 1 -- 172.21.15.81:0/1713000421 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da803e030 con 0x7f2dac0b5250 2024-06-29T15:39:07.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2da27fc640 1 -- 172.21.15.81:0/1713000421 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2da8003e00 con 0x7f2dac0b5250 2024-06-29T15:39:07.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2da27fc640 1 -- 172.21.15.81:0/1713000421 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da8035040 con 0x7f2dac0b5250 2024-06-29T15:39:07.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db3b63640 1 -- 172.21.15.81:0/1713000421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dac0c8c30 con 0x7f2dac0b5250 2024-06-29T15:39:07.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.533+0000 7f2db3b63640 1 -- 172.21.15.81:0/1713000421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dac1f7e80 con 0x7f2dac0b5250 2024-06-29T15:39:07.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.534+0000 7f2db3b63640 1 -- 172.21.15.81:0/1713000421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dac0b84c0 con 0x7f2dac0b5250 2024-06-29T15:39:07.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.535+0000 7f2da27fc640 1 -- 172.21.15.81:0/1713000421 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f2da8041030 con 0x7f2dac0b5250 2024-06-29T15:39:07.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.536+0000 7f2da27fc640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2d90076420 0x7f2d90078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:07.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.536+0000 7f2da27fc640 1 -- 172.21.15.81:0/1713000421 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5731+0+0 (secure 0 0 0) 0x7f2da80b8400 con 0x7f2dac0b5250 2024-06-29T15:39:07.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.536+0000 7f2db10fe640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2d90076420 0x7f2d90078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:07.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.536+0000 7f2db10fe640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2d90076420 0x7f2d90078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2dac06dda0 tx=0x7f2d9c00b2e0 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:07.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.539+0000 7f2da27fc640 1 -- 172.21.15.81:0/1713000421 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2da80814d0 con 0x7f2dac0b5250 2024-06-29T15:39:07.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:07.730+0000 7f2db3b63640 1 -- 172.21.15.81:0/1713000421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f2dac1f80b0 con 0x7f2dac0b5250 2024-06-29T15:39:08.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.140+0000 7f2da27fc640 1 -- 172.21.15.81:0/1713000421 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v175) ==== 97+0+0 (secure 0 0 0) 0x7f2da80857d0 con 0x7f2dac0b5250 2024-06-29T15:39:08.141 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T15:39:08.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 -- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2d90076420 msgr2=0x7f2d90078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2d90076420 0x7f2d90078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2dac06dda0 tx=0x7f2d9c00b2e0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 -- 172.21.15.81:0/1713000421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dac0b5250 msgr2=0x7f2dac0ca6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dac0b5250 0x7f2dac0ca6c0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f2da8000c00 tx=0x7f2da8003910 comp rx=0 tx=0).stop 2024-06-29T15:39:08.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 -- 172.21.15.81:0/1713000421 shutdown_connections 2024-06-29T15:39:08.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2d90076420 0x7f2d90078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2dac0b9040 0x7f2dac151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2dac0b7290 0x7f2dac151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 --2- 172.21.15.81:0/1713000421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dac0b5250 0x7f2dac0ca6c0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 -- 172.21.15.81:0/1713000421 >> 172.21.15.81:0/1713000421 conn(0x7f2dac09cf30 msgr2=0x7f2dac0b86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 -- 172.21.15.81:0/1713000421 shutdown_connections 2024-06-29T15:39:08.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.143+0000 7f2db3b63640 1 -- 172.21.15.81:0/1713000421 wait complete. 2024-06-29T15:39:08.156 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:39:08.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.269+0000 7f28b717a640 1 -- 172.21.15.81:0/2039360455 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28b01457a0 msgr2=0x7f28b0142390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.269+0000 7f28b717a640 1 --2- 172.21.15.81:0/2039360455 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28b01457a0 0x7f28b0142390 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f28a0009830 tx=0x7f28a002a510 comp rx=0 tx=0).stop 2024-06-29T15:39:08.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.269+0000 7f28b717a640 1 -- 172.21.15.81:0/2039360455 shutdown_connections 2024-06-29T15:39:08.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.269+0000 7f28b717a640 1 --2- 172.21.15.81:0/2039360455 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28b00cf310 0x7f28b0073430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.269+0000 7f28b717a640 1 --2- 172.21.15.81:0/2039360455 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28b01457a0 0x7f28b0142390 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.269+0000 7f28b717a640 1 -- 172.21.15.81:0/2039360455 >> 172.21.15.81:0/2039360455 conn(0x7f28b006ccc0 msgr2=0x7f28b00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.269+0000 7f28b717a640 1 -- 172.21.15.81:0/2039360455 shutdown_connections 2024-06-29T15:39:08.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.269+0000 7f28b717a640 1 -- 172.21.15.81:0/2039360455 wait complete. 2024-06-29T15:39:08.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.271+0000 7f28b717a640 1 Processor -- start 2024-06-29T15:39:08.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.271+0000 7f28b717a640 1 -- start start 2024-06-29T15:39:08.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28b717a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28b0073970 0x7f28b00aaf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28b717a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28b00cf310 0x7f28b00ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28b717a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28b01457a0 0x7f28b00ac290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28b717a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28b01600f0 con 0x7f28b00cf310 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28b717a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f28b0143f90 con 0x7f28b01457a0 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28b717a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f28b0160270 con 0x7f28b0073970 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28affff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28b00cf310 0x7f28b00ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28affff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28b00cf310 0x7f28b00ab4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60442/0 (socket says 172.21.15.81:60442) 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28affff640 1 -- 172.21.15.81:0/2444808417 learned_addr learned my addr 172.21.15.81:0/2444808417 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28b4f16640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28b0073970 0x7f28b00aaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28b5717640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28b01457a0 0x7f28b00ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28affff640 1 -- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28b0073970 msgr2=0x7f28b00aaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28affff640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28b0073970 0x7f28b00aaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28affff640 1 -- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28b01457a0 msgr2=0x7f28b00ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28affff640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28b01457a0 0x7f28b00ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28affff640 1 -- 172.21.15.81:0/2444808417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b015b220 con 0x7f28b00cf310 2024-06-29T15:39:08.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28affff640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28b00cf310 0x7f28b00ab4a0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f28a400ee00 tx=0x7f28a400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.273+0000 7f28ad7fa640 1 -- 172.21.15.81:0/2444808417 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a401b070 con 0x7f28b00cf310 2024-06-29T15:39:08.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.274+0000 7f28ad7fa640 1 -- 172.21.15.81:0/2444808417 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28a400b860 con 0x7f28b00cf310 2024-06-29T15:39:08.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.274+0000 7f28b717a640 1 -- 172.21.15.81:0/2444808417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b00aa480 con 0x7f28b00cf310 2024-06-29T15:39:08.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.274+0000 7f28ad7fa640 1 -- 172.21.15.81:0/2444808417 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a400bcb0 con 0x7f28b00cf310 2024-06-29T15:39:08.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.274+0000 7f28b717a640 1 -- 172.21.15.81:0/2444808417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b013b720 con 0x7f28b00cf310 2024-06-29T15:39:08.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.274+0000 7f28b717a640 1 -- 172.21.15.81:0/2444808417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b015af90 con 0x7f28b00cf310 2024-06-29T15:39:08.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.275+0000 7f28ad7fa640 1 -- 172.21.15.81:0/2444808417 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f28a401e070 con 0x7f28b00cf310 2024-06-29T15:39:08.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.276+0000 7f28ad7fa640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f289c0763f0 0x7f289c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.276+0000 7f28ad7fa640 1 -- 172.21.15.81:0/2444808417 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4957+0+0 (secure 0 0 0) 0x7f28a4097eb0 con 0x7f28b00cf310 2024-06-29T15:39:08.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.276+0000 7f28b4f16640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f289c0763f0 0x7f289c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.277+0000 7f28b4f16640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f289c0763f0 0x7f289c078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f28a0034000 tx=0x7f28a0004730 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.280+0000 7f28ad7fa640 1 -- 172.21.15.81:0/2444808417 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f28a4061290 con 0x7f28b00cf310 2024-06-29T15:39:08.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.528+0000 7f28b717a640 1 -- 172.21.15.81:0/2444808417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f28b0160790 con 0x7f28b00cf310 2024-06-29T15:39:08.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.528+0000 7f28ad7fa640 1 -- 172.21.15.81:0/2444808417 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v58) ==== 80+0+5304 (secure 0 0 0) 0x7f28a4065590 con 0x7f28b00cf310 2024-06-29T15:39:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:08.529 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:39:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T15:39:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T15:39:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T15:39:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.529 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:39:08.530 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.531 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.532 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T15:39:08.533 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.534 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T15:39:08.536 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.537 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.538 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T15:39:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T15:39:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:39:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T15:39:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:39:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T15:39:08.539 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:39:08.540 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:39:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:39:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:39:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T15:39:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T15:39:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.541 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:39:08.542 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:39:08.543 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T15:39:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T15:39:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:39:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:39:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:39:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:39:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T15:39:08.544 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:39:08.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 -- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f289c0763f0 msgr2=0x7f289c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f289c0763f0 0x7f289c078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f28a0034000 tx=0x7f28a0004730 comp rx=0 tx=0).stop 2024-06-29T15:39:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 -- 172.21.15.81:0/2444808417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28b00cf310 msgr2=0x7f28b00ab4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28b00cf310 0x7f28b00ab4a0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f28a400ee00 tx=0x7f28a400c740 comp rx=0 tx=0).stop 2024-06-29T15:39:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 -- 172.21.15.81:0/2444808417 shutdown_connections 2024-06-29T15:39:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f289c0763f0 0x7f289c078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f28b01457a0 0x7f28b00ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28b00cf310 0x7f28b00ab4a0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 --2- 172.21.15.81:0/2444808417 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f28b0073970 0x7f28b00aaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 -- 172.21.15.81:0/2444808417 >> 172.21.15.81:0/2444808417 conn(0x7f28b006ccc0 msgr2=0x7f28b0140f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 -- 172.21.15.81:0/2444808417 shutdown_connections 2024-06-29T15:39:08.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.530+0000 7f28b717a640 1 -- 172.21.15.81:0/2444808417 wait complete. 2024-06-29T15:39:08.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:39:08.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.659+0000 7fd61297f640 1 -- 172.21.15.81:0/2467914110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd60c074520 msgr2=0x7fd60c06f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.659+0000 7fd61297f640 1 --2- 172.21.15.81:0/2467914110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd60c074520 0x7fd60c06f140 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fd60c06e210 tx=0x7fd5fc02a530 comp rx=0 tx=0).stop 2024-06-29T15:39:08.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.659+0000 7fd61297f640 1 -- 172.21.15.81:0/2467914110 shutdown_connections 2024-06-29T15:39:08.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.659+0000 7fd61297f640 1 --2- 172.21.15.81:0/2467914110 >> v2:172.21.15.167:3300/0 conn(0x7fd60c06f680 0x7fd60c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.659+0000 7fd61297f640 1 --2- 172.21.15.81:0/2467914110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd60c074520 0x7fd60c06f140 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.659+0000 7fd61297f640 1 -- 172.21.15.81:0/2467914110 >> 172.21.15.81:0/2467914110 conn(0x7fd60c072950 msgr2=0x7fd60c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.659+0000 7fd61297f640 1 -- 172.21.15.81:0/2467914110 shutdown_connections 2024-06-29T15:39:08.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.659+0000 7fd61297f640 1 -- 172.21.15.81:0/2467914110 wait complete. 2024-06-29T15:39:08.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.661+0000 7fd61297f640 1 Processor -- start 2024-06-29T15:39:08.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.661+0000 7fd61297f640 1 -- start start 2024-06-29T15:39:08.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.662+0000 7fd61297f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd60c06f680 0x7fd60c0b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.662+0000 7fd61297f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd60c06fa90 0x7fd60c0cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.662+0000 7fd61297f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd60c074520 0x7fd60c0cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.662+0000 7fd61297f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c15b010 con 0x7fd60c06f680 2024-06-29T15:39:08.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.662+0000 7fd61297f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c06d150 con 0x7fd60c074520 2024-06-29T15:39:08.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.662+0000 7fd61297f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd60c15b190 con 0x7fd60c06fa90 2024-06-29T15:39:08.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.662+0000 7fd60bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd60c06f680 0x7fd60c0b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.662+0000 7fd60bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd60c06f680 0x7fd60c0b0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60472/0 (socket says 172.21.15.81:60472) 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd60bfff640 1 -- 172.21.15.81:0/1412494836 learned_addr learned my addr 172.21.15.81:0/1412494836 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd60b7fe640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd60c06fa90 0x7fd60c0cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd610f1c640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd60c074520 0x7fd60c0cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd60bfff640 1 -- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd60c06fa90 msgr2=0x7fd60c0cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd60bfff640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd60c06fa90 0x7fd60c0cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd60bfff640 1 -- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd60c074520 msgr2=0x7fd60c0cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd60bfff640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd60c074520 0x7fd60c0cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd60bfff640 1 -- 172.21.15.81:0/1412494836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60c1f5320 con 0x7fd60c06f680 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd60bfff640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd60c06f680 0x7fd60c0b0080 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009780 tx=0x7fd5fc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd608ff9640 1 -- 172.21.15.81:0/1412494836 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc031d60 con 0x7fd60c06f680 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd608ff9640 1 -- 172.21.15.81:0/1412494836 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5fc033e10 con 0x7fd60c06f680 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd61297f640 1 -- 172.21.15.81:0/1412494836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60c1f5600 con 0x7fd60c06f680 2024-06-29T15:39:08.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd608ff9640 1 -- 172.21.15.81:0/1412494836 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc005260 con 0x7fd60c06f680 2024-06-29T15:39:08.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.663+0000 7fd61297f640 1 -- 172.21.15.81:0/1412494836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60c1f5bc0 con 0x7fd60c06f680 2024-06-29T15:39:08.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.664+0000 7fd61297f640 1 -- 172.21.15.81:0/1412494836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd60c0af4d0 con 0x7fd60c06f680 2024-06-29T15:39:08.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.665+0000 7fd608ff9640 1 -- 172.21.15.81:0/1412494836 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fd5fc005400 con 0x7fd60c06f680 2024-06-29T15:39:08.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.666+0000 7fd608ff9640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5f80763f0 0x7fd5f8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.666+0000 7fd608ff9640 1 -- 172.21.15.81:0/1412494836 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4957+0+0 (secure 0 0 0) 0x7fd5fc0b8890 con 0x7fd60c06f680 2024-06-29T15:39:08.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.666+0000 7fd60b7fe640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5f80763f0 0x7fd5f8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.666+0000 7fd60b7fe640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5f80763f0 0x7fd5f8078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd6000058c0 tx=0x7fd600005850 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.670+0000 7fd608ff9640 1 -- 172.21.15.81:0/1412494836 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd5fc03f030 con 0x7fd60c06f680 2024-06-29T15:39:08.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.849+0000 7fd61297f640 1 -- 172.21.15.81:0/1412494836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd60c156280 con 0x7fd60c06f680 2024-06-29T15:39:08.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.850+0000 7fd608ff9640 1 -- 172.21.15.81:0/1412494836 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) ==== 74+0+13048 (secure 0 0 0) 0x7fd5fc0822c0 con 0x7fd60c06f680 2024-06-29T15:39:08.850 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:08.850 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":177,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:39:08.152003+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":166,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:39:08.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.851+0000 7fd61297f640 1 -- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5f80763f0 msgr2=0x7fd5f8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.851+0000 7fd61297f640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5f80763f0 0x7fd5f8078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd6000058c0 tx=0x7fd600005850 comp rx=0 tx=0).stop 2024-06-29T15:39:08.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.852+0000 7fd61297f640 1 -- 172.21.15.81:0/1412494836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd60c06f680 msgr2=0x7fd60c0b0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.852+0000 7fd61297f640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd60c06f680 0x7fd60c0b0080 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009780 tx=0x7fd5fc033c20 comp rx=0 tx=0).stop 2024-06-29T15:39:08.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.852+0000 7fd61297f640 1 -- 172.21.15.81:0/1412494836 shutdown_connections 2024-06-29T15:39:08.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.852+0000 7fd61297f640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fd5f80763f0 0x7fd5f8078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.852+0000 7fd61297f640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd60c074520 0x7fd60c0cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.852+0000 7fd61297f640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd60c06fa90 0x7fd60c0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.852+0000 7fd61297f640 1 --2- 172.21.15.81:0/1412494836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd60c06f680 0x7fd60c0b0080 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.852+0000 7fd61297f640 1 -- 172.21.15.81:0/1412494836 >> 172.21.15.81:0/1412494836 conn(0x7fd60c072950 msgr2=0x7fd60c155880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.852+0000 7fd61297f640 1 -- 172.21.15.81:0/1412494836 shutdown_connections 2024-06-29T15:39:08.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.852+0000 7fd61297f640 1 -- 172.21.15.81:0/1412494836 wait complete. 2024-06-29T15:39:08.864 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:39:08.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:39:08.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.980+0000 7fe1b6426640 1 -- 172.21.15.81:0/2208494353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b0074520 msgr2=0x7fe1b006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.980+0000 7fe1b6426640 1 --2- 172.21.15.81:0/2208494353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b0074520 0x7fe1b006f140 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0005fd0 tx=0x7fe1a002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.980+0000 7fe1b6426640 1 -- 172.21.15.81:0/2208494353 shutdown_connections 2024-06-29T15:39:08.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.980+0000 7fe1b6426640 1 --2- 172.21.15.81:0/2208494353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b0074520 0x7fe1b006f140 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.980+0000 7fe1b6426640 1 -- 172.21.15.81:0/2208494353 >> 172.21.15.81:0/2208494353 conn(0x7fe1b0072950 msgr2=0x7fe1b00b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:08.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.980+0000 7fe1b6426640 1 -- 172.21.15.81:0/2208494353 shutdown_connections 2024-06-29T15:39:08.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.980+0000 7fe1b6426640 1 -- 172.21.15.81:0/2208494353 wait complete. 2024-06-29T15:39:08.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.982+0000 7fe1b6426640 1 Processor -- start 2024-06-29T15:39:08.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.982+0000 7fe1b6426640 1 -- start start 2024-06-29T15:39:08.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1b6426640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1b006f680 0x7fe1b00cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1b6426640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe1b006fa90 0x7fe1b0147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1b6426640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b0074520 0x7fe1b014a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1b6426640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b015b190 con 0x7fe1b0074520 2024-06-29T15:39:08.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1b6426640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b015b010 con 0x7fe1b006f680 2024-06-29T15:39:08.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1b6426640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b00af310 con 0x7fe1b006fa90 2024-06-29T15:39:08.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1b49c3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b0074520 0x7fe1b014a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1b49c3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b0074520 0x7fe1b014a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60496/0 (socket says 172.21.15.81:60496) 2024-06-29T15:39:08.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1b49c3640 1 -- 172.21.15.81:0/1879561858 learned_addr learned my addr 172.21.15.81:0/1879561858 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:08.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1affff640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1b006f680 0x7fe1b00cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.983+0000 7fe1af7fe640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe1b006fa90 0x7fe1b0147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1b49c3640 1 -- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe1b006fa90 msgr2=0x7fe1b0147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1b49c3640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe1b006fa90 0x7fe1b0147660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1b49c3640 1 -- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1b006f680 msgr2=0x7fe1b00cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1b49c3640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1b006f680 0x7fe1b00cbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1b49c3640 1 -- 172.21.15.81:0/1879561858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b00ca0c0 con 0x7fe1b0074520 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1b49c3640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b0074520 0x7fe1b014a010 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800b9a0 tx=0x7fe1a800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1acff9640 1 -- 172.21.15.81:0/1879561858 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8019070 con 0x7fe1b0074520 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1acff9640 1 -- 172.21.15.81:0/1879561858 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1a800cc90 con 0x7fe1b0074520 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1b6426640 1 -- 172.21.15.81:0/1879561858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b00ca3a0 con 0x7fe1b0074520 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1acff9640 1 -- 172.21.15.81:0/1879561858 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a80146a0 con 0x7fe1b0074520 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.984+0000 7fe1b6426640 1 -- 172.21.15.81:0/1879561858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b0147ba0 con 0x7fe1b0074520 2024-06-29T15:39:08.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.985+0000 7fe1b6426640 1 -- 172.21.15.81:0/1879561858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b006f140 con 0x7fe1b0074520 2024-06-29T15:39:08.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.986+0000 7fe1acff9640 1 -- 172.21.15.81:0/1879561858 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fe1a801c070 con 0x7fe1b0074520 2024-06-29T15:39:08.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.986+0000 7fe1acff9640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe19c076420 0x7fe19c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:08.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.986+0000 7fe1acff9640 1 -- 172.21.15.81:0/1879561858 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4957+0+0 (secure 0 0 0) 0x7fe1a8097590 con 0x7fe1b0074520 2024-06-29T15:39:08.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.987+0000 7fe1affff640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe19c076420 0x7fe19c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:08.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.987+0000 7fe1affff640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe19c076420 0x7fe19c078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0033000 tx=0x7fe1a0004730 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:08.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:08.990+0000 7fe1acff9640 1 -- 172.21.15.81:0/1879561858 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe1a8023080 con 0x7fe1b0074520 2024-06-29T15:39:09.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.168+0000 7fe1b6426640 1 -- 172.21.15.81:0/1879561858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe1b00b47e0 con 0x7fe1b0074520 2024-06-29T15:39:09.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.169+0000 7fe1acff9640 1 -- 172.21.15.81:0/1879561858 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v177) ==== 93+0+10 (secure 0 0 0) 0x7fe1a80611b0 con 0x7fe1b0074520 2024-06-29T15:39:09.170 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:39:09.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 -- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe19c076420 msgr2=0x7fe19c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe19c076420 0x7fe19c078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0033000 tx=0x7fe1a0004730 comp rx=0 tx=0).stop 2024-06-29T15:39:09.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 -- 172.21.15.81:0/1879561858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b0074520 msgr2=0x7fe1b014a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b0074520 0x7fe1b014a010 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800b9a0 tx=0x7fe1a800be60 comp rx=0 tx=0).stop 2024-06-29T15:39:09.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 -- 172.21.15.81:0/1879561858 shutdown_connections 2024-06-29T15:39:09.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fe19c076420 0x7fe19c078930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b0074520 0x7fe1b014a010 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe1b006fa90 0x7fe1b0147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 --2- 172.21.15.81:0/1879561858 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe1b006f680 0x7fe1b00cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 -- 172.21.15.81:0/1879561858 >> 172.21.15.81:0/1879561858 conn(0x7fe1b0072950 msgr2=0x7fe1b0158640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:09.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 -- 172.21.15.81:0/1879561858 shutdown_connections 2024-06-29T15:39:09.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.171+0000 7fe1b6426640 1 -- 172.21.15.81:0/1879561858 wait complete. 2024-06-29T15:39:09.184 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:39:09.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.296+0000 7fc9fe33c640 1 -- 172.21.15.81:0/3241683993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9f8074520 msgr2=0x7fc9f806f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.296+0000 7fc9fe33c640 1 --2- 172.21.15.81:0/3241683993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9f8074520 0x7fc9f806f140 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8005fd0 tx=0x7fc9e802a530 comp rx=0 tx=0).stop 2024-06-29T15:39:09.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.296+0000 7fc9fe33c640 1 -- 172.21.15.81:0/3241683993 shutdown_connections 2024-06-29T15:39:09.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.296+0000 7fc9fe33c640 1 --2- 172.21.15.81:0/3241683993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc9f806fa90 0x7fc9f806ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.296+0000 7fc9fe33c640 1 --2- 172.21.15.81:0/3241683993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9f8074520 0x7fc9f806f140 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.296+0000 7fc9fe33c640 1 -- 172.21.15.81:0/3241683993 >> 172.21.15.81:0/3241683993 conn(0x7fc9f8072950 msgr2=0x7fc9f80b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:09.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.296+0000 7fc9fe33c640 1 -- 172.21.15.81:0/3241683993 shutdown_connections 2024-06-29T15:39:09.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.296+0000 7fc9fe33c640 1 -- 172.21.15.81:0/3241683993 wait complete. 2024-06-29T15:39:09.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.298+0000 7fc9fe33c640 1 Processor -- start 2024-06-29T15:39:09.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.298+0000 7fc9fe33c640 1 -- start start 2024-06-29T15:39:09.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.299+0000 7fc9fe33c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc9f806f680 0x7fc9f80c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.299+0000 7fc9fe33c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9f806fa90 0x7fc9f814f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.299+0000 7fc9fe33c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc9f8074520 0x7fc9f80cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.299+0000 7fc9fe33c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f815b190 con 0x7fc9f806fa90 2024-06-29T15:39:09.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.299+0000 7fc9fe33c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f815b010 con 0x7fc9f8074520 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.299+0000 7fc9fe33c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f80af310 con 0x7fc9f806f680 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.299+0000 7fc9f77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9f806fa90 0x7fc9f814f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.299+0000 7fc9f77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9f806fa90 0x7fc9f814f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60510/0 (socket says 172.21.15.81:60510) 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.299+0000 7fc9f77fe640 1 -- 172.21.15.81:0/1494669201 learned_addr learned my addr 172.21.15.81:0/1494669201 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f7fff640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc9f806f680 0x7fc9f80c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9fc8d9640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc9f8074520 0x7fc9f80cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f77fe640 1 -- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc9f806f680 msgr2=0x7fc9f80c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f77fe640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc9f806f680 0x7fc9f80c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f77fe640 1 -- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc9f8074520 msgr2=0x7fc9f80cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f77fe640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc9f8074520 0x7fc9f80cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f77fe640 1 -- 172.21.15.81:0/1494669201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f80cb270 con 0x7fc9f806fa90 2024-06-29T15:39:09.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f7fff640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc9f806f680 0x7fc9f80c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:09.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f77fe640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9f806fa90 0x7fc9f814f5c0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00b440 tx=0x7fc9ec00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:09.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f4ff9640 1 -- 172.21.15.81:0/1494669201 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec015020 con 0x7fc9f806fa90 2024-06-29T15:39:09.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f4ff9640 1 -- 172.21.15.81:0/1494669201 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9ec0119a0 con 0x7fc9f806fa90 2024-06-29T15:39:09.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9f4ff9640 1 -- 172.21.15.81:0/1494669201 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec011df0 con 0x7fc9f806fa90 2024-06-29T15:39:09.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9fe33c640 1 -- 172.21.15.81:0/1494669201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f80c9c90 con 0x7fc9f806fa90 2024-06-29T15:39:09.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.300+0000 7fc9fe33c640 1 -- 172.21.15.81:0/1494669201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f80ca250 con 0x7fc9f806fa90 2024-06-29T15:39:09.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.301+0000 7fc9fe33c640 1 -- 172.21.15.81:0/1494669201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f806f140 con 0x7fc9f806fa90 2024-06-29T15:39:09.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.302+0000 7fc9f4ff9640 1 -- 172.21.15.81:0/1494669201 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fc9ec018020 con 0x7fc9f806fa90 2024-06-29T15:39:09.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.302+0000 7fc9f4ff9640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fc9e4076420 0x7fc9e4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.302+0000 7fc9f4ff9640 1 -- 172.21.15.81:0/1494669201 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4957+0+0 (secure 0 0 0) 0x7fc9ec097790 con 0x7fc9f806fa90 2024-06-29T15:39:09.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.303+0000 7fc9f7fff640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fc9e4076420 0x7fc9e4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.303+0000 7fc9f7fff640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fc9e4076420 0x7fc9e4078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8033000 tx=0x7fc9e8004730 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:09.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.307+0000 7fc9f4ff9640 1 -- 172.21.15.81:0/1494669201 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc9ec01f020 con 0x7fc9f806fa90 2024-06-29T15:39:09.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.487+0000 7fc9fe33c640 1 -- 172.21.15.81:0/1494669201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc9f80cc520 con 0x7fc9f806fa90 2024-06-29T15:39:09.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.487+0000 7fc9f4ff9640 1 -- 172.21.15.81:0/1494669201 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v177) ==== 84+0+65 (secure 0 0 0) 0x7fc9ec00d040 con 0x7fc9f806fa90 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:09.488 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 -- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fc9e4076420 msgr2=0x7fc9e4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fc9e4076420 0x7fc9e4078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8033000 tx=0x7fc9e8004730 comp rx=0 tx=0).stop 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 -- 172.21.15.81:0/1494669201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9f806fa90 msgr2=0x7fc9f814f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9f806fa90 0x7fc9f814f5c0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00b440 tx=0x7fc9ec00b900 comp rx=0 tx=0).stop 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 -- 172.21.15.81:0/1494669201 shutdown_connections 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fc9e4076420 0x7fc9e4078930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc9f8074520 0x7fc9f80cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9f806fa90 0x7fc9f814f5c0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 --2- 172.21.15.81:0/1494669201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc9f806f680 0x7fc9f80c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 -- 172.21.15.81:0/1494669201 >> 172.21.15.81:0/1494669201 conn(0x7fc9f8072950 msgr2=0x7fc9f80afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 -- 172.21.15.81:0/1494669201 shutdown_connections 2024-06-29T15:39:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.489+0000 7fc9fe33c640 1 -- 172.21.15.81:0/1494669201 wait complete. 2024-06-29T15:39:09.501 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:39:09.501 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:39:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.619+0000 7f7a4e163640 1 -- 172.21.15.81:0/521602907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a30003a50 msgr2=0x7f7a30003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.619+0000 7f7a4e163640 1 --2- 172.21.15.81:0/521602907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a30003a50 0x7f7a30003ec0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009b50 tx=0x7f7a30005830 comp rx=0 tx=0).stop 2024-06-29T15:39:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.619+0000 7f7a47fff640 1 -- 172.21.15.81:0/521602907 reap_dead start 2024-06-29T15:39:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.619+0000 7f7a4e163640 1 -- 172.21.15.81:0/521602907 shutdown_connections 2024-06-29T15:39:09.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.619+0000 7f7a4e163640 1 -- 172.21.15.81:0/521602907 >> 172.21.15.81:0/521602907 conn(0x7f7a480b35a0 msgr2=0x7f7a480b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.619+0000 7f7a4e163640 1 -- 172.21.15.81:0/521602907 shutdown_connections 2024-06-29T15:39:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.619+0000 7f7a4e163640 1 -- 172.21.15.81:0/521602907 wait complete. 2024-06-29T15:39:09.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.621+0000 7f7a4e163640 1 Processor -- start 2024-06-29T15:39:09.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.621+0000 7f7a4e163640 1 -- start start 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a4e163640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a30003a50 0x7f7a4809c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a4e163640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a4809b990 0x7f7a481e5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a4e163640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a481e1e40 0x7f7a481e5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a4e163640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7a480b1730 con 0x7f7a481e1e40 2024-06-29T15:39:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a4e163640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7a480b15b0 con 0x7f7a30003a50 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a4e163640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7a480b18b0 con 0x7f7a4809b990 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a47fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a481e1e40 0x7f7a481e5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a47fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a481e1e40 0x7f7a481e5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60522/0 (socket says 172.21.15.81:60522) 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a47fff640 1 -- 172.21.15.81:0/49663661 learned_addr learned my addr 172.21.15.81:0/49663661 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a477fe640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a30003a50 0x7f7a4809c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a46ffd640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a4809b990 0x7f7a481e5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a47fff640 1 -- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a4809b990 msgr2=0x7f7a481e5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a47fff640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a4809b990 0x7f7a481e5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a47fff640 1 -- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a30003a50 msgr2=0x7f7a4809c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a47fff640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a30003a50 0x7f7a4809c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a47fff640 1 -- 172.21.15.81:0/49663661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4809d100 con 0x7f7a481e1e40 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a477fe640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a30003a50 0x7f7a4809c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:09.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a47fff640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a481e1e40 0x7f7a481e5d70 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f7a40005fd0 tx=0x7f7a4000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:09.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a2ffff640 1 -- 172.21.15.81:0/49663661 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40003970 con 0x7f7a481e1e40 2024-06-29T15:39:09.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a2ffff640 1 -- 172.21.15.81:0/49663661 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a40003b10 con 0x7f7a481e1e40 2024-06-29T15:39:09.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a4e163640 1 -- 172.21.15.81:0/49663661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a481e7a00 con 0x7f7a481e1e40 2024-06-29T15:39:09.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.623+0000 7f7a2ffff640 1 -- 172.21.15.81:0/49663661 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40004030 con 0x7f7a481e1e40 2024-06-29T15:39:09.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.624+0000 7f7a4e163640 1 -- 172.21.15.81:0/49663661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a481e7fc0 con 0x7f7a481e1e40 2024-06-29T15:39:09.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.625+0000 7f7a4e163640 1 -- 172.21.15.81:0/49663661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a480b1db0 con 0x7f7a481e1e40 2024-06-29T15:39:09.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.625+0000 7f7a2ffff640 1 -- 172.21.15.81:0/49663661 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f7a400041d0 con 0x7f7a481e1e40 2024-06-29T15:39:09.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.626+0000 7f7a2ffff640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7a340763f0 0x7f7a34078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:09.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.626+0000 7f7a2ffff640 1 -- 172.21.15.81:0/49663661 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4957+0+0 (secure 0 0 0) 0x7f7a40097b70 con 0x7f7a481e1e40 2024-06-29T15:39:09.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.626+0000 7f7a477fe640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7a340763f0 0x7f7a34078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:09.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.626+0000 7f7a477fe640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7a340763f0 0x7f7a34078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7a38035000 tx=0x7f7a3802d040 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:09.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.630+0000 7f7a2ffff640 1 -- 172.21.15.81:0/49663661 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7a40061510 con 0x7f7a481e1e40 2024-06-29T15:39:09.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:09.814+0000 7f7a4e163640 1 -- 172.21.15.81:0/49663661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f7a480b0c60 con 0x7f7a481e1e40 2024-06-29T15:39:10.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.146+0000 7f7a2ffff640 1 -- 172.21.15.81:0/49663661 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v178) ==== 138+0+0 (secure 0 0 0) 0x7f7a40060e10 con 0x7f7a481e1e40 2024-06-29T15:39:10.147 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T15:39:10.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.149+0000 7f7a4e163640 1 -- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7a340763f0 msgr2=0x7f7a34078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:10.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.149+0000 7f7a4e163640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7a340763f0 0x7f7a34078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7a38035000 tx=0x7f7a3802d040 comp rx=0 tx=0).stop 2024-06-29T15:39:10.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.149+0000 7f7a4e163640 1 -- 172.21.15.81:0/49663661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a481e1e40 msgr2=0x7f7a481e5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:10.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.149+0000 7f7a4e163640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a481e1e40 0x7f7a481e5d70 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f7a40005fd0 tx=0x7f7a4000f3c0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.149+0000 7f7a4e163640 1 -- 172.21.15.81:0/49663661 shutdown_connections 2024-06-29T15:39:10.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.149+0000 7f7a4e163640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7a340763f0 0x7f7a34078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.149+0000 7f7a4e163640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a481e1e40 0x7f7a481e5d70 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.149+0000 7f7a4e163640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a4809b990 0x7f7a481e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.149+0000 7f7a4e163640 1 --2- 172.21.15.81:0/49663661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a30003a50 0x7f7a4809c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.149+0000 7f7a4e163640 1 -- 172.21.15.81:0/49663661 >> 172.21.15.81:0/49663661 conn(0x7f7a480b35a0 msgr2=0x7f7a480b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:10.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.150+0000 7f7a4e163640 1 -- 172.21.15.81:0/49663661 shutdown_connections 2024-06-29T15:39:10.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.150+0000 7f7a4e163640 1 -- 172.21.15.81:0/49663661 wait complete. 2024-06-29T15:39:10.165 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:39:10.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.286+0000 7ff618cef640 1 -- 172.21.15.81:0/904767029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5fc004d40 msgr2=0x7ff5fc005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:10.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.286+0000 7ff618cef640 1 --2- 172.21.15.81:0/904767029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5fc004d40 0x7ff5fc005150 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7ff60c00c750 tx=0x7ff5fc005830 comp rx=0 tx=0).stop 2024-06-29T15:39:10.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.286+0000 7ff612d9d640 1 -- 172.21.15.81:0/904767029 reap_dead start 2024-06-29T15:39:10.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.286+0000 7ff618cef640 1 -- 172.21.15.81:0/904767029 shutdown_connections 2024-06-29T15:39:10.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.286+0000 7ff618cef640 1 -- 172.21.15.81:0/904767029 >> 172.21.15.81:0/904767029 conn(0x7ff61406ccc0 msgr2=0x7ff6140b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:10.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.286+0000 7ff618cef640 1 -- 172.21.15.81:0/904767029 shutdown_connections 2024-06-29T15:39:10.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.286+0000 7ff618cef640 1 -- 172.21.15.81:0/904767029 wait complete. 2024-06-29T15:39:10.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.288+0000 7ff618cef640 1 Processor -- start 2024-06-29T15:39:10.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.288+0000 7ff618cef640 1 -- start start 2024-06-29T15:39:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.289+0000 7ff618cef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5fc004d40 0x7ff6140b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.289+0000 7ff618cef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6140cc6c0 0x7ff6140b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.289+0000 7ff618cef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6140b0ab0 0x7ff6140ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.289+0000 7ff618cef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff614073030 con 0x7ff5fc004d40 2024-06-29T15:39:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.289+0000 7ff618cef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff61415f280 con 0x7ff6140b0ab0 2024-06-29T15:39:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.289+0000 7ff618cef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff6140731b0 con 0x7ff6140cc6c0 2024-06-29T15:39:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.289+0000 7ff61259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5fc004d40 0x7ff6140b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.289+0000 7ff61259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5fc004d40 0x7ff6140b0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56066/0 (socket says 172.21.15.81:56066) 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.289+0000 7ff61259c640 1 -- 172.21.15.81:0/3513538333 learned_addr learned my addr 172.21.15.81:0/3513538333 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff611d9b640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6140cc6c0 0x7ff6140b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff612d9d640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6140b0ab0 0x7ff6140ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff61259c640 1 -- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6140cc6c0 msgr2=0x7ff6140b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff61259c640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6140cc6c0 0x7ff6140b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff61259c640 1 -- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6140b0ab0 msgr2=0x7ff6140ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff61259c640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6140b0ab0 0x7ff6140ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff61259c640 1 -- 172.21.15.81:0/3513538333 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6140b0ec0 con 0x7ff5fc004d40 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff61259c640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5fc004d40 0x7ff6140b0030 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7ff604011ae0 tx=0x7ff604009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff602ffd640 1 -- 172.21.15.81:0/3513538333 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60401e070 con 0x7ff5fc004d40 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff602ffd640 1 -- 172.21.15.81:0/3513538333 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff60400fc70 con 0x7ff5fc004d40 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff602ffd640 1 -- 172.21.15.81:0/3513538333 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604019410 con 0x7ff5fc004d40 2024-06-29T15:39:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff618cef640 1 -- 172.21.15.81:0/3513538333 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6140aede0 con 0x7ff5fc004d40 2024-06-29T15:39:10.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.290+0000 7ff618cef640 1 -- 172.21.15.81:0/3513538333 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6140af830 con 0x7ff5fc004d40 2024-06-29T15:39:10.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.291+0000 7ff618cef640 1 -- 172.21.15.81:0/3513538333 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6140cce40 con 0x7ff5fc004d40 2024-06-29T15:39:10.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.292+0000 7ff602ffd640 1 -- 172.21.15.81:0/3513538333 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7ff604021070 con 0x7ff5fc004d40 2024-06-29T15:39:10.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.292+0000 7ff602ffd640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff5f40763f0 0x7ff5f4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:10.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.293+0000 7ff602ffd640 1 -- 172.21.15.81:0/3513538333 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 5327+0+0 (secure 0 0 0) 0x7ff6040984a0 con 0x7ff5fc004d40 2024-06-29T15:39:10.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.293+0000 7ff611d9b640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff5f40763f0 0x7ff5f4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:10.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.293+0000 7ff611d9b640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff5f40763f0 0x7ff5f4078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff6140b1600 tx=0x7ff608005f90 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:10.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.296+0000 7ff602ffd640 1 -- 172.21.15.81:0/3513538333 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff6140cce40 con 0x7ff5fc004d40 2024-06-29T15:39:10.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:10.477+0000 7ff618cef640 1 -- 172.21.15.81:0/3513538333 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff6140cef90 con 0x7ff5fc004d40 2024-06-29T15:39:11.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.149+0000 7ff602ffd640 1 -- 172.21.15.81:0/3513538333 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v179) ==== 170+0+0 (secure 0 0 0) 0x7ff604061760 con 0x7ff5fc004d40 2024-06-29T15:39:11.150 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T15:39:11.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 -- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff5f40763f0 msgr2=0x7ff5f4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:11.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff5f40763f0 0x7ff5f4078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff6140b1600 tx=0x7ff608005f90 comp rx=0 tx=0).stop 2024-06-29T15:39:11.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 -- 172.21.15.81:0/3513538333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5fc004d40 msgr2=0x7ff6140b0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:11.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5fc004d40 0x7ff6140b0030 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7ff604011ae0 tx=0x7ff604009eb0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 -- 172.21.15.81:0/3513538333 shutdown_connections 2024-06-29T15:39:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff5f40763f0 0x7ff5f4078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff6140b0ab0 0x7ff6140ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6140cc6c0 0x7ff6140b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 --2- 172.21.15.81:0/3513538333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5fc004d40 0x7ff6140b0030 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 -- 172.21.15.81:0/3513538333 >> 172.21.15.81:0/3513538333 conn(0x7ff61406ccc0 msgr2=0x7ff61406efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 -- 172.21.15.81:0/3513538333 shutdown_connections 2024-06-29T15:39:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.152+0000 7ff618cef640 1 -- 172.21.15.81:0/3513538333 wait complete. 2024-06-29T15:39:11.178 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:39:11.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.296+0000 7f4aca124640 1 -- 172.21.15.81:0/2805082905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ac4074520 msgr2=0x7f4ac406f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:11.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.296+0000 7f4aca124640 1 --2- 172.21.15.81:0/2805082905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ac4074520 0x7f4ac406f140 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f4ac406e210 tx=0x7f4ab402a560 comp rx=0 tx=0).stop 2024-06-29T15:39:11.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.296+0000 7f4aca124640 1 -- 172.21.15.81:0/2805082905 shutdown_connections 2024-06-29T15:39:11.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.296+0000 7f4aca124640 1 --2- 172.21.15.81:0/2805082905 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ac406fa90 0x7f4ac406c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.296+0000 7f4aca124640 1 --2- 172.21.15.81:0/2805082905 >> v2:172.21.15.167:3300/0 conn(0x7f4ac406f680 0x7f4ac406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.296+0000 7f4aca124640 1 --2- 172.21.15.81:0/2805082905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ac4074520 0x7f4ac406f140 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.296+0000 7f4aca124640 1 -- 172.21.15.81:0/2805082905 >> 172.21.15.81:0/2805082905 conn(0x7f4ac4072950 msgr2=0x7f4ac40b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:11.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.296+0000 7f4aca124640 1 -- 172.21.15.81:0/2805082905 shutdown_connections 2024-06-29T15:39:11.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.296+0000 7f4aca124640 1 -- 172.21.15.81:0/2805082905 wait complete. 2024-06-29T15:39:11.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.298+0000 7f4aca124640 1 Processor -- start 2024-06-29T15:39:11.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.298+0000 7f4aca124640 1 -- start start 2024-06-29T15:39:11.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.299+0000 7f4aca124640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ac406f680 0x7f4ac413f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:11.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.299+0000 7f4aca124640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ac406fa90 0x7f4ac414deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.299+0000 7f4aca124640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ac4074520 0x7f4ac413d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.299+0000 7f4aca124640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac406cd70 con 0x7f4ac4074520 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.299+0000 7f4aca124640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac415b220 con 0x7f4ac406fa90 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.299+0000 7f4aca124640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac406cef0 con 0x7f4ac406f680 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.299+0000 7f4ac3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ac4074520 0x7f4ac413d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.299+0000 7f4ac3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ac4074520 0x7f4ac413d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56078/0 (socket says 172.21.15.81:56078) 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.299+0000 7f4ac3fff640 1 -- 172.21.15.81:0/4186495354 learned_addr learned my addr 172.21.15.81:0/4186495354 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4ac37fe640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ac406f680 0x7f4ac413f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4ac2ffd640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ac406fa90 0x7f4ac414deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4ac3fff640 1 -- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ac406f680 msgr2=0x7f4ac413f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4ac3fff640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ac406f680 0x7f4ac413f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4ac3fff640 1 -- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ac406fa90 msgr2=0x7f4ac414deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:11.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4ac3fff640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ac406fa90 0x7f4ac414deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:11.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4ac3fff640 1 -- 172.21.15.81:0/4186495354 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac413c580 con 0x7f4ac4074520 2024-06-29T15:39:11.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4ac3fff640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ac4074520 0x7f4ac413d620 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00e9f0 tx=0x7f4abc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:11.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4aabfff640 1 -- 172.21.15.81:0/4186495354 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc004d20 con 0x7f4ac4074520 2024-06-29T15:39:11.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4aabfff640 1 -- 172.21.15.81:0/4186495354 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4abc00cb20 con 0x7f4ac4074520 2024-06-29T15:39:11.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4aca124640 1 -- 172.21.15.81:0/4186495354 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac413c860 con 0x7f4ac4074520 2024-06-29T15:39:11.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4aabfff640 1 -- 172.21.15.81:0/4186495354 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc005020 con 0x7f4ac4074520 2024-06-29T15:39:11.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.300+0000 7f4aca124640 1 -- 172.21.15.81:0/4186495354 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac413ce20 con 0x7f4ac4074520 2024-06-29T15:39:11.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.301+0000 7f4aca124640 1 -- 172.21.15.81:0/4186495354 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac413c2f0 con 0x7f4ac4074520 2024-06-29T15:39:11.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.302+0000 7f4aabfff640 1 -- 172.21.15.81:0/4186495354 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f4abc0051c0 con 0x7f4ac4074520 2024-06-29T15:39:11.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.303+0000 7f4aabfff640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4ab00763f0 0x7f4ab0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:11.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.303+0000 7f4aabfff640 1 -- 172.21.15.81:0/4186495354 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 5709+0+0 (secure 0 0 0) 0x7f4abc098660 con 0x7f4ac4074520 2024-06-29T15:39:11.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.303+0000 7f4ac37fe640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4ab00763f0 0x7f4ab0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:11.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.304+0000 7f4ac37fe640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4ab00763f0 0x7f4ab0078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4033000 tx=0x7f4ab4004730 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:11.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.307+0000 7f4aabfff640 1 -- 172.21.15.81:0/4186495354 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4abc016030 con 0x7f4ac4074520 2024-06-29T15:39:11.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:11.525+0000 7f4aca124640 1 -- 172.21.15.81:0/4186495354 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4ac4158ff0 con 0x7f4ac4074520 2024-06-29T15:39:12.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.171+0000 7f4aabfff640 1 -- 172.21.15.81:0/4186495354 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v91) ==== 174+0+0 (secure 0 0 0) 0x7f4abc068e20 con 0x7f4ac4074520 2024-06-29T15:39:12.172 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 22 and data pool 23 2024-06-29T15:39:12.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.176+0000 7f4aca124640 1 -- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4ab00763f0 msgr2=0x7f4ab0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.176+0000 7f4aca124640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4ab00763f0 0x7f4ab0078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4033000 tx=0x7f4ab4004730 comp rx=0 tx=0).stop 2024-06-29T15:39:12.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.176+0000 7f4aca124640 1 -- 172.21.15.81:0/4186495354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ac4074520 msgr2=0x7f4ac413d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.176+0000 7f4aca124640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ac4074520 0x7f4ac413d620 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00e9f0 tx=0x7f4abc00eeb0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.177+0000 7f4aca124640 1 -- 172.21.15.81:0/4186495354 shutdown_connections 2024-06-29T15:39:12.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.177+0000 7f4aca124640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f4ab00763f0 0x7f4ab0078900 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4033000 tx=0x7f4ab4004730 comp rx=0 tx=0).stop 2024-06-29T15:39:12.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.177+0000 7f4aca124640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ac4074520 0x7f4ac413d620 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.177+0000 7f4aca124640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ac406fa90 0x7f4ac414deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.177+0000 7f4aca124640 1 --2- 172.21.15.81:0/4186495354 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ac406f680 0x7f4ac413f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.177+0000 7f4aca124640 1 -- 172.21.15.81:0/4186495354 >> 172.21.15.81:0/4186495354 conn(0x7f4ac4072950 msgr2=0x7f4ac415a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.177+0000 7f4aca124640 1 -- 172.21.15.81:0/4186495354 shutdown_connections 2024-06-29T15:39:12.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.178+0000 7f4aca124640 1 -- 172.21.15.81:0/4186495354 wait complete. 2024-06-29T15:39:12.200 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:39:12.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.312+0000 7fecad0a4640 1 -- 172.21.15.81:0/1147864550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec90003c30 msgr2=0x7fec90004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.312+0000 7fecad0a4640 1 --2- 172.21.15.81:0/1147864550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec90003c30 0x7fec90004080 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fec98009b50 tx=0x7fec90005830 comp rx=0 tx=0).stop 2024-06-29T15:39:12.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.312+0000 7fecad0a4640 1 -- 172.21.15.81:0/1147864550 shutdown_connections 2024-06-29T15:39:12.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.312+0000 7fecad0a4640 1 --2- 172.21.15.81:0/1147864550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec90003c30 0x7fec90004080 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.312+0000 7fecad0a4640 1 -- 172.21.15.81:0/1147864550 >> 172.21.15.81:0/1147864550 conn(0x7feca80b56d0 msgr2=0x7feca80b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.312+0000 7fecad0a4640 1 -- 172.21.15.81:0/1147864550 shutdown_connections 2024-06-29T15:39:12.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.312+0000 7fecad0a4640 1 -- 172.21.15.81:0/1147864550 wait complete. 2024-06-29T15:39:12.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.314+0000 7fecad0a4640 1 Processor -- start 2024-06-29T15:39:12.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.314+0000 7fecad0a4640 1 -- start start 2024-06-29T15:39:12.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fecad0a4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec90003c30 0x7feca81dce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fecad0a4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feca81db660 0x7feca81f5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fecad0a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feca81f4320 0x7feca80b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fecad0a4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feca80ceab0 con 0x7feca81f4320 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fecad0a4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7feca80ce930 con 0x7feca81db660 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fecad0a4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7feca80cec30 con 0x7fec90003c30 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca759e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feca81f4320 0x7feca80b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca759e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feca81f4320 0x7feca80b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56098/0 (socket says 172.21.15.81:56098) 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca759e640 1 -- 172.21.15.81:0/3399366993 learned_addr learned my addr 172.21.15.81:0/3399366993 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca6d9d640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec90003c30 0x7feca81dce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca659c640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feca81db660 0x7feca81f5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca759e640 1 -- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec90003c30 msgr2=0x7feca81dce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca759e640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec90003c30 0x7feca81dce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca759e640 1 -- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feca81db660 msgr2=0x7feca81f5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca759e640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feca81db660 0x7feca81f5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca759e640 1 -- 172.21.15.81:0/3399366993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca81dbd90 con 0x7feca81f4320 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca659c640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feca81db660 0x7feca81f5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7feca759e640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feca81f4320 0x7feca80b6a10 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7feca00014a0 tx=0x7feca000eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fec8f7fe640 1 -- 172.21.15.81:0/3399366993 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca000c970 con 0x7feca81f4320 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fec8f7fe640 1 -- 172.21.15.81:0/3399366993 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feca0018070 con 0x7feca81f4320 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fecad0a4640 1 -- 172.21.15.81:0/3399366993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca81dc070 con 0x7feca81f4320 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fec8f7fe640 1 -- 172.21.15.81:0/3399366993 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca000cea0 con 0x7feca81f4320 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.316+0000 7fecad0a4640 1 -- 172.21.15.81:0/3399366993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca81dc630 con 0x7feca81f4320 2024-06-29T15:39:12.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.317+0000 7fecad0a4640 1 -- 172.21.15.81:0/3399366993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca80cf140 con 0x7feca81f4320 2024-06-29T15:39:12.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.318+0000 7fec8f7fe640 1 -- 172.21.15.81:0/3399366993 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7feca001d460 con 0x7feca81f4320 2024-06-29T15:39:12.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.319+0000 7fec8f7fe640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fec940763f0 0x7fec94078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.319+0000 7fec8f7fe640 1 -- 172.21.15.81:0/3399366993 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(180..180 src has 1..180) ==== 5809+0+0 (secure 0 0 0) 0x7feca0097f20 con 0x7feca81f4320 2024-06-29T15:39:12.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.319+0000 7feca6d9d640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fec940763f0 0x7fec94078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.319+0000 7feca6d9d640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fec940763f0 0x7fec94078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fec98035000 tx=0x7fec980046c0 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.323+0000 7fec8f7fe640 1 -- 172.21.15.81:0/3399366993 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feca0061560 con 0x7feca81f4320 2024-06-29T15:39:12.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.504+0000 7fecad0a4640 1 -- 172.21.15.81:0/3399366993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feca80d2310 con 0x7feca81f4320 2024-06-29T15:39:12.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.505+0000 7fec8f7fe640 1 -- 172.21.15.81:0/3399366993 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) ==== 74+0+16831 (secure 0 0 0) 0x7feca0060e60 con 0x7feca81f4320 2024-06-29T15:39:12.506 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:12.506 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":180,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:39:11.526799+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.821386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.484738+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:39:12.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.507+0000 7fecad0a4640 1 -- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fec940763f0 msgr2=0x7fec94078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.507+0000 7fecad0a4640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fec940763f0 0x7fec94078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fec98035000 tx=0x7fec980046c0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.507+0000 7fecad0a4640 1 -- 172.21.15.81:0/3399366993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feca81f4320 msgr2=0x7feca80b6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.507+0000 7fecad0a4640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feca81f4320 0x7feca80b6a10 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7feca00014a0 tx=0x7feca000eb00 comp rx=0 tx=0).stop 2024-06-29T15:39:12.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.507+0000 7fecad0a4640 1 -- 172.21.15.81:0/3399366993 shutdown_connections 2024-06-29T15:39:12.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.507+0000 7fecad0a4640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fec940763f0 0x7fec94078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.507+0000 7fecad0a4640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feca81f4320 0x7feca80b6a10 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.507+0000 7fecad0a4640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feca81db660 0x7feca81f5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.507+0000 7fecad0a4640 1 --2- 172.21.15.81:0/3399366993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec90003c30 0x7feca81dce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.507+0000 7fecad0a4640 1 -- 172.21.15.81:0/3399366993 >> 172.21.15.81:0/3399366993 conn(0x7feca80b56d0 msgr2=0x7feca80cdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.508+0000 7fecad0a4640 1 -- 172.21.15.81:0/3399366993 shutdown_connections 2024-06-29T15:39:12.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.508+0000 7fecad0a4640 1 -- 172.21.15.81:0/3399366993 wait complete. 2024-06-29T15:39:12.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:39:12.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.633+0000 7f611bfa1640 1 -- 172.21.15.81:0/3039273153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f611406ca00 msgr2=0x7f61140af8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.633+0000 7f611bfa1640 1 --2- 172.21.15.81:0/3039273153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f611406ca00 0x7f61140af8f0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f610c005fd0 tx=0x7f610c02a9a0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.633+0000 7f611bfa1640 1 -- 172.21.15.81:0/3039273153 shutdown_connections 2024-06-29T15:39:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.633+0000 7f611bfa1640 1 --2- 172.21.15.81:0/3039273153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f611406ca00 0x7f61140af8f0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.633+0000 7f611bfa1640 1 -- 172.21.15.81:0/3039273153 >> 172.21.15.81:0/3039273153 conn(0x7f61140d0930 msgr2=0x7f61140d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.633+0000 7f611bfa1640 1 -- 172.21.15.81:0/3039273153 shutdown_connections 2024-06-29T15:39:12.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.633+0000 7f611bfa1640 1 -- 172.21.15.81:0/3039273153 wait complete. 2024-06-29T15:39:12.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.634+0000 7f611bfa1640 1 Processor -- start 2024-06-29T15:39:12.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.635+0000 7f611bfa1640 1 -- start start 2024-06-29T15:39:12.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611bfa1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6114154fc0 0x7f6114156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611bfa1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f611415e710 0x7f6114156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611bfa1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6114158440 0x7f61141f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611bfa1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61140affb0 con 0x7f6114158440 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611bfa1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f61140afe30 con 0x7f6114154fc0 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611bfa1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f61140b0130 con 0x7f611415e710 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611a53e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6114158440 0x7f61141f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611a53e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6114158440 0x7f61141f49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56114/0 (socket says 172.21.15.81:56114) 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611a53e640 1 -- 172.21.15.81:0/3221373381 learned_addr learned my addr 172.21.15.81:0/3221373381 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f6119d3d640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6114154fc0 0x7f6114156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611953c640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f611415e710 0x7f6114156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611a53e640 1 -- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f611415e710 msgr2=0x7f6114156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611a53e640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f611415e710 0x7f6114156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611a53e640 1 -- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6114154fc0 msgr2=0x7f6114156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611a53e640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6114154fc0 0x7f6114156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.636+0000 7f611a53e640 1 -- 172.21.15.81:0/3221373381 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61141f5690 con 0x7f6114158440 2024-06-29T15:39:12.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.637+0000 7f611a53e640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6114158440 0x7f61141f49c0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f610c009d90 tx=0x7f610c030ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.637+0000 7f610a7fc640 1 -- 172.21.15.81:0/3221373381 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610c004040 con 0x7f6114158440 2024-06-29T15:39:12.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.637+0000 7f610a7fc640 1 -- 172.21.15.81:0/3221373381 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f610c0041e0 con 0x7f6114158440 2024-06-29T15:39:12.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.637+0000 7f611bfa1640 1 -- 172.21.15.81:0/3221373381 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61141f5970 con 0x7f6114158440 2024-06-29T15:39:12.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.637+0000 7f610a7fc640 1 -- 172.21.15.81:0/3221373381 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610c004630 con 0x7f6114158440 2024-06-29T15:39:12.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.637+0000 7f611bfa1640 1 -- 172.21.15.81:0/3221373381 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6114155b20 con 0x7f6114158440 2024-06-29T15:39:12.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.638+0000 7f611bfa1640 1 -- 172.21.15.81:0/3221373381 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f611406e2a0 con 0x7f6114158440 2024-06-29T15:39:12.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.638+0000 7f610a7fc640 1 -- 172.21.15.81:0/3221373381 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f610c031470 con 0x7f6114158440 2024-06-29T15:39:12.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.639+0000 7f610a7fc640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f60f8076420 0x7f60f8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.639+0000 7f610a7fc640 1 -- 172.21.15.81:0/3221373381 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(180..180 src has 1..180) ==== 5809+0+0 (secure 0 0 0) 0x7f610c0b9070 con 0x7f6114158440 2024-06-29T15:39:12.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.639+0000 7f6119d3d640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f60f8076420 0x7f60f8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.640+0000 7f6119d3d640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f60f8076420 0x7f60f8078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6110001720 tx=0x7f6110009040 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.643+0000 7f610a7fc640 1 -- 172.21.15.81:0/3221373381 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f611406e2a0 con 0x7f6114158440 2024-06-29T15:39:12.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.823+0000 7f611bfa1640 1 -- 172.21.15.81:0/3221373381 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61140af8f0 con 0x7f6114158440 2024-06-29T15:39:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.824+0000 7f610a7fc640 1 -- 172.21.15.81:0/3221373381 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) ==== 74+0+16831 (secure 0 0 0) 0x7f610c082180 con 0x7f6114158440 2024-06-29T15:39:12.825 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:12.825 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":180,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:39:11.526799+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.821386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.484738+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:39:12.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 -- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f60f8076420 msgr2=0x7f60f8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f60f8076420 0x7f60f8078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6110001720 tx=0x7f6110009040 comp rx=0 tx=0).stop 2024-06-29T15:39:12.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 -- 172.21.15.81:0/3221373381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6114158440 msgr2=0x7f61141f49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6114158440 0x7f61141f49c0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f610c009d90 tx=0x7f610c030ef0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 -- 172.21.15.81:0/3221373381 shutdown_connections 2024-06-29T15:39:12.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f60f8076420 0x7f60f8078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6114158440 0x7f61141f49c0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f611415e710 0x7f6114156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 --2- 172.21.15.81:0/3221373381 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6114154fc0 0x7f6114156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 -- 172.21.15.81:0/3221373381 >> 172.21.15.81:0/3221373381 conn(0x7f61140d0930 msgr2=0x7f61140b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 -- 172.21.15.81:0/3221373381 shutdown_connections 2024-06-29T15:39:12.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.826+0000 7f611bfa1640 1 -- 172.21.15.81:0/3221373381 wait complete. 2024-06-29T15:39:12.838 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:39:12.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.953+0000 7f6eeba26640 1 -- 172.21.15.81:0/2328025814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ee406fa90 msgr2=0x7f6ee406c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.953+0000 7f6eeba26640 1 --2- 172.21.15.81:0/2328025814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ee406fa90 0x7f6ee406c830 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4072d60 tx=0x7f6edc02aa30 comp rx=0 tx=0).stop 2024-06-29T15:39:12.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.953+0000 7f6eeba26640 1 -- 172.21.15.81:0/2328025814 shutdown_connections 2024-06-29T15:39:12.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.953+0000 7f6eeba26640 1 --2- 172.21.15.81:0/2328025814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ee406fa90 0x7f6ee406c830 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.953+0000 7f6eeba26640 1 --2- 172.21.15.81:0/2328025814 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ee406f680 0x7f6ee406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.953+0000 7f6eeba26640 1 --2- 172.21.15.81:0/2328025814 >> v2:172.21.15.167:3300/0 conn(0x7f6ee4074520 0x7f6ee406f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.953+0000 7f6eeba26640 1 -- 172.21.15.81:0/2328025814 >> 172.21.15.81:0/2328025814 conn(0x7f6ee4072950 msgr2=0x7f6ee40b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:12.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.953+0000 7f6eeba26640 1 -- 172.21.15.81:0/2328025814 shutdown_connections 2024-06-29T15:39:12.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.953+0000 7f6eeba26640 1 -- 172.21.15.81:0/2328025814 wait complete. 2024-06-29T15:39:12.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.955+0000 7f6eeba26640 1 Processor -- start 2024-06-29T15:39:12.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.955+0000 7f6eeba26640 1 -- start start 2024-06-29T15:39:12.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6eeba26640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ee406f680 0x7f6ee40c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6eeba26640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ee406fa90 0x7f6ee414a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6eeba26640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ee4074520 0x7f6ee40d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6eeba26640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee406cd70 con 0x7f6ee406f680 2024-06-29T15:39:12.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6eeba26640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee415b220 con 0x7f6ee4074520 2024-06-29T15:39:12.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6eeba26640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6ee406cef0 con 0x7f6ee406fa90 2024-06-29T15:39:12.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee97c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ee406f680 0x7f6ee40c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee97c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ee406f680 0x7f6ee40c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56134/0 (socket says 172.21.15.81:56134) 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee97c2640 1 -- 172.21.15.81:0/2049055783 learned_addr learned my addr 172.21.15.81:0/2049055783 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee8fc1640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ee406fa90 0x7f6ee414a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee97c2640 1 -- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ee406fa90 msgr2=0x7f6ee414a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee97c2640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ee406fa90 0x7f6ee414a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee97c2640 1 -- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ee4074520 msgr2=0x7f6ee40d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee9fc3640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ee4074520 0x7f6ee40d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee97c2640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ee4074520 0x7f6ee40d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee97c2640 1 -- 172.21.15.81:0/2049055783 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee40c9f00 con 0x7f6ee406f680 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.956+0000 7f6ee9fc3640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ee4074520 0x7f6ee40d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.957+0000 7f6ee97c2640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ee406f680 0x7f6ee40c9460 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000e790 tx=0x7f6ee000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.957+0000 7f6ed9ffb640 1 -- 172.21.15.81:0/2049055783 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee000eea0 con 0x7f6ee406f680 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.957+0000 7f6ed9ffb640 1 -- 172.21.15.81:0/2049055783 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ee000cdb0 con 0x7f6ee406f680 2024-06-29T15:39:12.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.957+0000 7f6ed9ffb640 1 -- 172.21.15.81:0/2049055783 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0010690 con 0x7f6ee406f680 2024-06-29T15:39:12.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.957+0000 7f6eeba26640 1 -- 172.21.15.81:0/2049055783 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ee40ca1e0 con 0x7f6ee406f680 2024-06-29T15:39:12.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.957+0000 7f6eeba26640 1 -- 172.21.15.81:0/2049055783 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ee40ccf80 con 0x7f6ee406f680 2024-06-29T15:39:12.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.958+0000 7f6eeba26640 1 -- 172.21.15.81:0/2049055783 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ee406f140 con 0x7f6ee406f680 2024-06-29T15:39:12.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.959+0000 7f6ed9ffb640 1 -- 172.21.15.81:0/2049055783 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f6ee0010830 con 0x7f6ee406f680 2024-06-29T15:39:12.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.959+0000 7f6ed9ffb640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f6ec80763f0 0x7f6ec8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:12.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.959+0000 7f6ed9ffb640 1 -- 172.21.15.81:0/2049055783 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(180..180 src has 1..180) ==== 5809+0+0 (secure 0 0 0) 0x7f6ee00983c0 con 0x7f6ee406f680 2024-06-29T15:39:12.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.960+0000 7f6ee8fc1640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f6ec80763f0 0x7f6ec8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:12.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.960+0000 7f6ee8fc1640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f6ec80763f0 0x7f6ec8078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6ee40d1110 tx=0x7f6ed4005c90 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:12.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:12.964+0000 7f6ed9ffb640 1 -- 172.21.15.81:0/2049055783 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6ee0061a90 con 0x7f6ee406f680 2024-06-29T15:39:13.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:13.181+0000 7f6eeba26640 1 -- 172.21.15.81:0/2049055783 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6ee415e4f0 con 0x7f6ee406f680 2024-06-29T15:39:14.144 INFO:tasks.ceph.mds.b.smithi167.stderr:2024-06-29T15:39:14.144+0000 7f2ff4fc6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:39:14.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.191+0000 7f6ed9ffb640 1 -- 172.21.15.81:0/2049055783 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v94) ==== 118+0+0 (secure 0 0 0) 0x7f6ee0061390 con 0x7f6ee406f680 2024-06-29T15:39:14.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 -- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f6ec80763f0 msgr2=0x7f6ec8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:14.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f6ec80763f0 0x7f6ec8078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6ee40d1110 tx=0x7f6ed4005c90 comp rx=0 tx=0).stop 2024-06-29T15:39:14.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 -- 172.21.15.81:0/2049055783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ee406f680 msgr2=0x7f6ee40c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:14.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ee406f680 0x7f6ee40c9460 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000e790 tx=0x7f6ee000ec50 comp rx=0 tx=0).stop 2024-06-29T15:39:14.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 -- 172.21.15.81:0/2049055783 shutdown_connections 2024-06-29T15:39:14.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f6ec80763f0 0x7f6ec8078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6ee4074520 0x7f6ee40d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6ee406fa90 0x7f6ee414a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 --2- 172.21.15.81:0/2049055783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ee406f680 0x7f6ee40c9460 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 -- 172.21.15.81:0/2049055783 >> 172.21.15.81:0/2049055783 conn(0x7f6ee4072950 msgr2=0x7f6ee4157db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:14.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 -- 172.21.15.81:0/2049055783 shutdown_connections 2024-06-29T15:39:14.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.195+0000 7f6eeba26640 1 -- 172.21.15.81:0/2049055783 wait complete. 2024-06-29T15:39:14.209 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:39:14.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:39:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.332+0000 7f3883838640 1 -- 172.21.15.81:0/607461786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f387c15bc20 msgr2=0x7f387c06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.332+0000 7f3883838640 1 --2- 172.21.15.81:0/607461786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f387c15bc20 0x7f387c06ed90 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f3874009830 tx=0x7f387402a480 comp rx=0 tx=0).stop 2024-06-29T15:39:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.332+0000 7f3883838640 1 -- 172.21.15.81:0/607461786 shutdown_connections 2024-06-29T15:39:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.332+0000 7f3883838640 1 --2- 172.21.15.81:0/607461786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f387c15bc20 0x7f387c06ed90 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.332+0000 7f3883838640 1 --2- 172.21.15.81:0/607461786 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f387c073650 0x7f387c15b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.332+0000 7f3883838640 1 -- 172.21.15.81:0/607461786 >> 172.21.15.81:0/607461786 conn(0x7f387c0b2920 msgr2=0x7f387c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.332+0000 7f3883838640 1 -- 172.21.15.81:0/607461786 shutdown_connections 2024-06-29T15:39:14.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.332+0000 7f3883838640 1 -- 172.21.15.81:0/607461786 wait complete. 2024-06-29T15:39:14.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.334+0000 7f3883838640 1 Processor -- start 2024-06-29T15:39:14.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.334+0000 7f3883838640 1 -- start start 2024-06-29T15:39:14.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.335+0000 7f3883838640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f387c0cd150 0x7f387c156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:14.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.335+0000 7f3883838640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f387c15bc20 0x7f387c0d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:14.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.335+0000 7f3883838640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f387c0d0bf0 0x7f387c1583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:14.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.335+0000 7f3883838640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f387c06f450 con 0x7f387c15bc20 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.335+0000 7f3883838640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f387c06f2d0 con 0x7f387c0d0bf0 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.335+0000 7f3883838640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f387c06f5d0 con 0x7f387c0cd150 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3880dd3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f387c15bc20 0x7f387c0d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3880dd3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f387c15bc20 0x7f387c0d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56144/0 (socket says 172.21.15.81:56144) 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3880dd3640 1 -- 172.21.15.81:0/1522203274 learned_addr learned my addr 172.21.15.81:0/1522203274 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f38815d4640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f387c0cd150 0x7f387c156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3881dd5640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f387c0d0bf0 0x7f387c1583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3880dd3640 1 -- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f387c0cd150 msgr2=0x7f387c156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3880dd3640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f387c0cd150 0x7f387c156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3880dd3640 1 -- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f387c0d0bf0 msgr2=0x7f387c1583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3880dd3640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f387c0d0bf0 0x7f387c1583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3880dd3640 1 -- 172.21.15.81:0/1522203274 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387c1f4ac0 con 0x7f387c15bc20 2024-06-29T15:39:14.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3881dd5640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f387c0d0bf0 0x7f387c1583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:14.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3880dd3640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f387c15bc20 0x7f387c0d06b0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f387000c970 tx=0x7f387000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:14.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3879ffb640 1 -- 172.21.15.81:0/1522203274 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870007bb0 con 0x7f387c15bc20 2024-06-29T15:39:14.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3879ffb640 1 -- 172.21.15.81:0/1522203274 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3870007e40 con 0x7f387c15bc20 2024-06-29T15:39:14.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3883838640 1 -- 172.21.15.81:0/1522203274 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f387c1f4fc0 con 0x7f387c15bc20 2024-06-29T15:39:14.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3879ffb640 1 -- 172.21.15.81:0/1522203274 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387000f6b0 con 0x7f387c15bc20 2024-06-29T15:39:14.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.336+0000 7f3883838640 1 -- 172.21.15.81:0/1522203274 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f387c1f5580 con 0x7f387c15bc20 2024-06-29T15:39:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.337+0000 7f3883838640 1 -- 172.21.15.81:0/1522203274 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f387c1f4d30 con 0x7f387c15bc20 2024-06-29T15:39:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.338+0000 7f3879ffb640 1 -- 172.21.15.81:0/1522203274 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f387000f850 con 0x7f387c15bc20 2024-06-29T15:39:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.339+0000 7f3879ffb640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f38680763f0 0x7f3868078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.339+0000 7f3879ffb640 1 -- 172.21.15.81:0/1522203274 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f38700981e0 con 0x7f387c15bc20 2024-06-29T15:39:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.339+0000 7f38815d4640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f38680763f0 0x7f3868078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.339+0000 7f38815d4640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f38680763f0 0x7f3868078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f386c0012e0 tx=0x7f386c009040 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:14.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.343+0000 7f3879ffb640 1 -- 172.21.15.81:0/1522203274 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3870014030 con 0x7f387c15bc20 2024-06-29T15:39:14.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:14.561+0000 7f3883838640 1 -- 172.21.15.81:0/1522203274 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f387c073900 con 0x7f387c15bc20 2024-06-29T15:39:15.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.193+0000 7f3879ffb640 1 -- 172.21.15.81:0/1522203274 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v95) ==== 122+0+0 (secure 0 0 0) 0x7f387c073900 con 0x7f387c15bc20 2024-06-29T15:39:15.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.195+0000 7f3883838640 1 -- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f38680763f0 msgr2=0x7f3868078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.195+0000 7f3883838640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f38680763f0 0x7f3868078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f386c0012e0 tx=0x7f386c009040 comp rx=0 tx=0).stop 2024-06-29T15:39:15.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.195+0000 7f3883838640 1 -- 172.21.15.81:0/1522203274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f387c15bc20 msgr2=0x7f387c0d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.195+0000 7f3883838640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f387c15bc20 0x7f387c0d06b0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f387000c970 tx=0x7f387000ce30 comp rx=0 tx=0).stop 2024-06-29T15:39:15.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.196+0000 7f3883838640 1 -- 172.21.15.81:0/1522203274 shutdown_connections 2024-06-29T15:39:15.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.196+0000 7f3883838640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f38680763f0 0x7f3868078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.196+0000 7f3883838640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f387c0d0bf0 0x7f387c1583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.196+0000 7f3883838640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f387c15bc20 0x7f387c0d06b0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.196+0000 7f3883838640 1 --2- 172.21.15.81:0/1522203274 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f387c0cd150 0x7f387c156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.196+0000 7f3883838640 1 -- 172.21.15.81:0/1522203274 >> 172.21.15.81:0/1522203274 conn(0x7f387c0b2920 msgr2=0x7f387c14a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.196+0000 7f3883838640 1 -- 172.21.15.81:0/1522203274 shutdown_connections 2024-06-29T15:39:15.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.196+0000 7f3883838640 1 -- 172.21.15.81:0/1522203274 wait complete. 2024-06-29T15:39:15.210 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:39:15.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:39:15.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.330+0000 7f9a476ac640 1 -- 172.21.15.81:0/3495676866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c0039e0 msgr2=0x7f9a2c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.330+0000 7f9a476ac640 1 --2- 172.21.15.81:0/3495676866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c0039e0 0x7f9a2c003e30 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0043e0 tx=0x7f9a2c005830 comp rx=0 tx=0).stop 2024-06-29T15:39:15.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.330+0000 7f9a45c49640 1 -- 172.21.15.81:0/3495676866 reap_dead start 2024-06-29T15:39:15.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.330+0000 7f9a476ac640 1 -- 172.21.15.81:0/3495676866 shutdown_connections 2024-06-29T15:39:15.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.330+0000 7f9a476ac640 1 -- 172.21.15.81:0/3495676866 >> 172.21.15.81:0/3495676866 conn(0x7f9a4006ccc0 msgr2=0x7f9a400b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.330+0000 7f9a476ac640 1 -- 172.21.15.81:0/3495676866 shutdown_connections 2024-06-29T15:39:15.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.330+0000 7f9a476ac640 1 -- 172.21.15.81:0/3495676866 wait complete. 2024-06-29T15:39:15.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.332+0000 7f9a476ac640 1 Processor -- start 2024-06-29T15:39:15.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.332+0000 7f9a476ac640 1 -- start start 2024-06-29T15:39:15.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a476ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c0039e0 0x7f9a400ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a476ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a40151360 0x7f9a40072100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a476ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a40072640 0x7f9a401e5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a476ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a400737d0 con 0x7f9a2c0039e0 2024-06-29T15:39:15.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a476ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9a40073650 con 0x7f9a40072640 2024-06-29T15:39:15.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a476ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9a401417e0 con 0x7f9a40151360 2024-06-29T15:39:15.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a45448640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c0039e0 0x7f9a400ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a45448640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c0039e0 0x7f9a400ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56154/0 (socket says 172.21.15.81:56154) 2024-06-29T15:39:15.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a45448640 1 -- 172.21.15.81:0/766303828 learned_addr learned my addr 172.21.15.81:0/766303828 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:15.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a44c47640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a40151360 0x7f9a40072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a45c49640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a40072640 0x7f9a401e5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a45448640 1 -- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a40151360 msgr2=0x7f9a40072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a45448640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a40151360 0x7f9a40072100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a45448640 1 -- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a40072640 msgr2=0x7f9a401e5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a45448640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a40072640 0x7f9a401e5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a45448640 1 -- 172.21.15.81:0/766303828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a40151ab0 con 0x7f9a2c0039e0 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.333+0000 7f9a45448640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c0039e0 0x7f9a400ae7e0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c004500 tx=0x7f9a3c004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.334+0000 7f9a35ffb640 1 -- 172.21.15.81:0/766303828 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c037040 con 0x7f9a2c0039e0 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.334+0000 7f9a35ffb640 1 -- 172.21.15.81:0/766303828 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a3c0062c0 con 0x7f9a2c0039e0 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.334+0000 7f9a476ac640 1 -- 172.21.15.81:0/766303828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a40151d90 con 0x7f9a2c0039e0 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.334+0000 7f9a35ffb640 1 -- 172.21.15.81:0/766303828 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0066d0 con 0x7f9a2c0039e0 2024-06-29T15:39:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.334+0000 7f9a476ac640 1 -- 172.21.15.81:0/766303828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a40152350 con 0x7f9a2c0039e0 2024-06-29T15:39:15.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.335+0000 7f9a476ac640 1 -- 172.21.15.81:0/766303828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a400cef90 con 0x7f9a2c0039e0 2024-06-29T15:39:15.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.336+0000 7f9a35ffb640 1 -- 172.21.15.81:0/766303828 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f9a3c045450 con 0x7f9a2c0039e0 2024-06-29T15:39:15.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.336+0000 7f9a35ffb640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f9a24076420 0x7f9a24078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.336+0000 7f9a35ffb640 1 -- 172.21.15.81:0/766303828 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f9a3c0b7f90 con 0x7f9a2c0039e0 2024-06-29T15:39:15.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.336+0000 7f9a44c47640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f9a24076420 0x7f9a24078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.337+0000 7f9a44c47640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f9a24076420 0x7f9a24078930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9a401e6790 tx=0x7f9a3000b260 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:15.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.341+0000 7f9a35ffb640 1 -- 172.21.15.81:0/766303828 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9a3c02ae20 con 0x7f9a2c0039e0 2024-06-29T15:39:15.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.556+0000 7f9a476ac640 1 -- 172.21.15.81:0/766303828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a400b0020 con 0x7f9a2c0039e0 2024-06-29T15:39:15.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.557+0000 7f9a35ffb640 1 -- 172.21.15.81:0/766303828 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+5237 (secure 0 0 0) 0x7f9a3c081740 con 0x7f9a2c0039e0 2024-06-29T15:39:15.558 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:15.558 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":95,"btime":"2024-06-29T15:39:15:192034+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2580108037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2580108037},{"type":"v1","addr":"172.21.15.167:6836","nonce":2580108037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":10155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3892298237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3892298237},{"type":"v1","addr":"172.21.15.81:6834","nonce":3892298237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":10158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2457574359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2457574359},{"type":"v1","addr":"172.21.15.81:6835","nonce":2457574359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.526747+0000","modified":"2024-06-29T15:39:14.562127+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10170},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10170":{"gid":10170,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/446857148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":446857148},{"type":"v1","addr":"172.21.15.167:6837","nonce":446857148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10170,"qdb_cluster":[10170]},"id":11}]} 2024-06-29T15:39:15.558 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 95 2024-06-29T15:39:15.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 -- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f9a24076420 msgr2=0x7f9a24078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f9a24076420 0x7f9a24078930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9a401e6790 tx=0x7f9a3000b260 comp rx=0 tx=0).stop 2024-06-29T15:39:15.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 -- 172.21.15.81:0/766303828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c0039e0 msgr2=0x7f9a400ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c0039e0 0x7f9a400ae7e0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c004500 tx=0x7f9a3c004a50 comp rx=0 tx=0).stop 2024-06-29T15:39:15.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 -- 172.21.15.81:0/766303828 shutdown_connections 2024-06-29T15:39:15.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f9a24076420 0x7f9a24078930 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a40072640 0x7f9a401e5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a40151360 0x7f9a40072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 --2- 172.21.15.81:0/766303828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c0039e0 0x7f9a400ae7e0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 -- 172.21.15.81:0/766303828 >> 172.21.15.81:0/766303828 conn(0x7f9a4006ccc0 msgr2=0x7f9a4015ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 -- 172.21.15.81:0/766303828 shutdown_connections 2024-06-29T15:39:15.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.559+0000 7f9a476ac640 1 -- 172.21.15.81:0/766303828 wait complete. 2024-06-29T15:39:15.571 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:39:15.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.684+0000 7fdecc426640 1 -- 172.21.15.81:0/2761854815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdec41412f0 msgr2=0x7fdec40cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.684+0000 7fdecc426640 1 --2- 172.21.15.81:0/2761854815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdec41412f0 0x7fdec40cef90 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009830 tx=0x7fdec002a510 comp rx=0 tx=0).stop 2024-06-29T15:39:15.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.684+0000 7fdecc426640 1 -- 172.21.15.81:0/2761854815 shutdown_connections 2024-06-29T15:39:15.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.684+0000 7fdecc426640 1 --2- 172.21.15.81:0/2761854815 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdec40cf4d0 0x7fdec4073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.684+0000 7fdecc426640 1 --2- 172.21.15.81:0/2761854815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdec41412f0 0x7fdec40cef90 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.684+0000 7fdecc426640 1 -- 172.21.15.81:0/2761854815 >> 172.21.15.81:0/2761854815 conn(0x7fdec406ccc0 msgr2=0x7fdec40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.684+0000 7fdecc426640 1 -- 172.21.15.81:0/2761854815 shutdown_connections 2024-06-29T15:39:15.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.684+0000 7fdecc426640 1 -- 172.21.15.81:0/2761854815 wait complete. 2024-06-29T15:39:15.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.686+0000 7fdecc426640 1 Processor -- start 2024-06-29T15:39:15.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.686+0000 7fdecc426640 1 -- start start 2024-06-29T15:39:15.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.687+0000 7fdecc426640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdec4073570 0x7fdec40aa120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.687+0000 7fdecc426640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdec40cf4d0 0x7fdec41542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.687+0000 7fdecc426640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdec41412f0 0x7fdec4072100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.687+0000 7fdecc426640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdec40b3210 con 0x7fdec40cf4d0 2024-06-29T15:39:15.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.687+0000 7fdecc426640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdec415f280 con 0x7fdec4073570 2024-06-29T15:39:15.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.687+0000 7fdecc426640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdec40b3390 con 0x7fdec41412f0 2024-06-29T15:39:15.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.687+0000 7fdec99c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdec40cf4d0 0x7fdec41542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.687+0000 7fdec99c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdec40cf4d0 0x7fdec41542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56176/0 (socket says 172.21.15.81:56176) 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.687+0000 7fdec99c1640 1 -- 172.21.15.81:0/3984684318 learned_addr learned my addr 172.21.15.81:0/3984684318 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdec99c1640 1 -- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdec41412f0 msgr2=0x7fdec4072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdeca9c3640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdec41412f0 0x7fdec4072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdec99c1640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdec41412f0 0x7fdec4072100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdec99c1640 1 -- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdec4073570 msgr2=0x7fdec40aa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdeca1c2640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdec4073570 0x7fdec40aa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdec99c1640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdec4073570 0x7fdec40aa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdec99c1640 1 -- 172.21.15.81:0/3984684318 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec4154820 con 0x7fdec40cf4d0 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdeca9c3640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdec41412f0 0x7fdec4072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdeca1c2640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdec4073570 0x7fdec40aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:15.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdec99c1640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdec40cf4d0 0x7fdec41542e0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400e7c0 tx=0x7fdeb400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:15.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdebaffd640 1 -- 172.21.15.81:0/3984684318 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb400ccc0 con 0x7fdec40cf4d0 2024-06-29T15:39:15.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdebaffd640 1 -- 172.21.15.81:0/3984684318 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdeb4010430 con 0x7fdec40cf4d0 2024-06-29T15:39:15.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdecc426640 1 -- 172.21.15.81:0/3984684318 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec4153820 con 0x7fdec40cf4d0 2024-06-29T15:39:15.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdebaffd640 1 -- 172.21.15.81:0/3984684318 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4010860 con 0x7fdec40cf4d0 2024-06-29T15:39:15.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.688+0000 7fdecc426640 1 -- 172.21.15.81:0/3984684318 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec40a9920 con 0x7fdec40cf4d0 2024-06-29T15:39:15.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.689+0000 7fdecc426640 1 -- 172.21.15.81:0/3984684318 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec40cd5e0 con 0x7fdec40cf4d0 2024-06-29T15:39:15.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.690+0000 7fdebaffd640 1 -- 172.21.15.81:0/3984684318 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fdeb4010a00 con 0x7fdec40cf4d0 2024-06-29T15:39:15.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.690+0000 7fdebaffd640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fdea80763f0 0x7fdea8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:15.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.690+0000 7fdebaffd640 1 -- 172.21.15.81:0/3984684318 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7fdeb40984c0 con 0x7fdec40cf4d0 2024-06-29T15:39:15.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.690+0000 7fdeca1c2640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fdea80763f0 0x7fdea8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:15.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.691+0000 7fdeca1c2640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fdea80763f0 0x7fdea8078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdec0034000 tx=0x7fdec0004730 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:15.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.694+0000 7fdebaffd640 1 -- 172.21.15.81:0/3984684318 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdec40cd5e0 con 0x7fdec40cf4d0 2024-06-29T15:39:15.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.876+0000 7fdecc426640 1 -- 172.21.15.81:0/3984684318 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdec4068f50 con 0x7fdec40cf4d0 2024-06-29T15:39:15.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.877+0000 7fdebaffd640 1 -- 172.21.15.81:0/3984684318 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) ==== 74+0+16818 (secure 0 0 0) 0x7fdec4068f50 con 0x7fdec40cf4d0 2024-06-29T15:39:15.878 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:15.878 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":181,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:39:13.171150+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.821386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.484738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 -- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fdea80763f0 msgr2=0x7fdea8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fdea80763f0 0x7fdea8078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdec0034000 tx=0x7fdec0004730 comp rx=0 tx=0).stop 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 -- 172.21.15.81:0/3984684318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdec40cf4d0 msgr2=0x7fdec41542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdec40cf4d0 0x7fdec41542e0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400e7c0 tx=0x7fdeb400ec80 comp rx=0 tx=0).stop 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 -- 172.21.15.81:0/3984684318 shutdown_connections 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fdea80763f0 0x7fdea8078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdec41412f0 0x7fdec4072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdec40cf4d0 0x7fdec41542e0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 --2- 172.21.15.81:0/3984684318 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdec4073570 0x7fdec40aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 -- 172.21.15.81:0/3984684318 >> 172.21.15.81:0/3984684318 conn(0x7fdec406ccc0 msgr2=0x7fdec406f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 -- 172.21.15.81:0/3984684318 shutdown_connections 2024-06-29T15:39:15.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:15.879+0000 7fdecc426640 1 -- 172.21.15.81:0/3984684318 wait complete. 2024-06-29T15:39:15.892 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:39:15.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:39:16.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.006+0000 7f7cd61fc640 1 -- 172.21.15.81:0/2129782073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb8004800 msgr2=0x7f7cb8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.006+0000 7f7cd61fc640 1 --2- 172.21.15.81:0/2129782073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb8004800 0x7f7cb8004c10 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40016c0 tx=0x7f7cb8005830 comp rx=0 tx=0).stop 2024-06-29T15:39:16.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.006+0000 7f7cd61fc640 1 -- 172.21.15.81:0/2129782073 shutdown_connections 2024-06-29T15:39:16.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.006+0000 7f7cd61fc640 1 --2- 172.21.15.81:0/2129782073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb8004800 0x7f7cb8004c10 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.006+0000 7f7cd61fc640 1 -- 172.21.15.81:0/2129782073 >> 172.21.15.81:0/2129782073 conn(0x7f7cd009cf30 msgr2=0x7f7cd00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.006+0000 7f7cd61fc640 1 -- 172.21.15.81:0/2129782073 shutdown_connections 2024-06-29T15:39:16.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.006+0000 7f7cd61fc640 1 -- 172.21.15.81:0/2129782073 wait complete. 2024-06-29T15:39:16.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.008+0000 7f7cd61fc640 1 Processor -- start 2024-06-29T15:39:16.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.008+0000 7f7cd61fc640 1 -- start start 2024-06-29T15:39:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7cd61fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb8004800 0x7f7cd0155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7cd61fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7cd0155920 0x7f7cd00b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7cd61fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7cd00b4120 0x7f7cd00b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7cd61fc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd00d0810 con 0x7f7cb8004800 2024-06-29T15:39:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7cd61fc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd00d0690 con 0x7f7cd0155920 2024-06-29T15:39:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7cd61fc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7cd00d0990 con 0x7f7cd00b4120 2024-06-29T15:39:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7ccf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb8004800 0x7f7cd0155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7ccf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb8004800 0x7f7cd0155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56204/0 (socket says 172.21.15.81:56204) 2024-06-29T15:39:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7ccf7fe640 1 -- 172.21.15.81:0/1923922583 learned_addr learned my addr 172.21.15.81:0/1923922583 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7cceffd640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7cd0155920 0x7f7cd00b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7ccffff640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7cd00b4120 0x7f7cd00b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7ccf7fe640 1 -- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7cd00b4120 msgr2=0x7f7cd00b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7ccf7fe640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7cd00b4120 0x7f7cd00b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7ccf7fe640 1 -- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7cd0155920 msgr2=0x7f7cd00b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7ccf7fe640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7cd0155920 0x7f7cd00b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.009+0000 7f7ccf7fe640 1 -- 172.21.15.81:0/1923922583 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd009f290 con 0x7f7cb8004800 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.010+0000 7f7ccf7fe640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb8004800 0x7f7cd0155130 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0010c70 tx=0x7f7cc000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.010+0000 7f7cb7fff640 1 -- 172.21.15.81:0/1923922583 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0017070 con 0x7f7cb8004800 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.010+0000 7f7cb7fff640 1 -- 172.21.15.81:0/1923922583 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cc0011640 con 0x7f7cb8004800 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.010+0000 7f7cb7fff640 1 -- 172.21.15.81:0/1923922583 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0011a90 con 0x7f7cb8004800 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.010+0000 7f7cd61fc640 1 -- 172.21.15.81:0/1923922583 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd0155670 con 0x7f7cb8004800 2024-06-29T15:39:16.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.010+0000 7f7cd61fc640 1 -- 172.21.15.81:0/1923922583 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd009f830 con 0x7f7cb8004800 2024-06-29T15:39:16.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.011+0000 7f7cd61fc640 1 -- 172.21.15.81:0/1923922583 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd00d1a80 con 0x7f7cb8004800 2024-06-29T15:39:16.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.012+0000 7f7cb7fff640 1 -- 172.21.15.81:0/1923922583 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f7cc0011c30 con 0x7f7cb8004800 2024-06-29T15:39:16.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.012+0000 7f7cb7fff640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cbc076420 0x7f7cbc078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.012+0000 7f7cb7fff640 1 -- 172.21.15.81:0/1923922583 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f7cc0098800 con 0x7f7cb8004800 2024-06-29T15:39:16.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.012+0000 7f7cceffd640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cbc076420 0x7f7cbc078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.013+0000 7f7cceffd640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cbc076420 0x7f7cbc078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4032000 tx=0x7f7cc402c040 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:16.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.017+0000 7f7cb7fff640 1 -- 172.21.15.81:0/1923922583 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7cc0061fe0 con 0x7f7cb8004800 2024-06-29T15:39:16.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.192+0000 7f7cd61fc640 1 -- 172.21.15.81:0/1923922583 --> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7cd00ca650 con 0x7f7cbc076420 2024-06-29T15:39:16.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.198+0000 7f7cb7fff640 1 -- 172.21.15.81:0/1923922583 <== mgr.7925 v2:172.21.15.167:6832/2546364927 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252494 (secure 0 0 0) 0x7f7cd00ca650 con 0x7f7cbc076420 2024-06-29T15:39:16.201 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:16.203 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:39:16.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 -- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cbc076420 msgr2=0x7f7cbc078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cbc076420 0x7f7cbc078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4032000 tx=0x7f7cc402c040 comp rx=0 tx=0).stop 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 -- 172.21.15.81:0/1923922583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb8004800 msgr2=0x7f7cd0155130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb8004800 0x7f7cd0155130 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0010c70 tx=0x7f7cc000e720 comp rx=0 tx=0).stop 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 -- 172.21.15.81:0/1923922583 shutdown_connections 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7cbc076420 0x7f7cbc078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7cd00b4120 0x7f7cd00b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7cd0155920 0x7f7cd00b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 --2- 172.21.15.81:0/1923922583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb8004800 0x7f7cd0155130 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 -- 172.21.15.81:0/1923922583 >> 172.21.15.81:0/1923922583 conn(0x7f7cd009cf30 msgr2=0x7f7cd006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 -- 172.21.15.81:0/1923922583 shutdown_connections 2024-06-29T15:39:16.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.201+0000 7f7cd61fc640 1 -- 172.21.15.81:0/1923922583 wait complete. 2024-06-29T15:39:16.213 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:39:15.714966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":338,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":426516,"kb_used_data":18860,"kb_used_omap":12,"kb_used_meta":407539,"kb_avail":754548204,"statfs":{"total":773094113280,"available":772657360896,"internally_reserved":0,"allocated":19312640,"data_stored":9953988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":417320498},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.566646"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322812+0000","last_change":"2024-06-29T15:39:12.190245+0000","last_active":"2024-06-29T15:39:13.322812+0000","last_peered":"2024-06-29T15:39:13.322812+0000","last_clean":"2024-06-29T15:39:13.322812+0000","last_became_active":"2024-06-29T15:39:12.189930+0000","last_became_peered":"2024-06-29T15:39:12.189930+0000","last_unstale":"2024-06-29T15:39:13.322812+0000","last_undegraded":"2024-06-29T15:39:13.322812+0000","last_fullsized":"2024-06-29T15:39:13.322812+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:32.419833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323201+0000","last_change":"2024-06-29T15:39:12.181762+0000","last_active":"2024-06-29T15:39:13.323201+0000","last_peered":"2024-06-29T15:39:13.323201+0000","last_clean":"2024-06-29T15:39:13.323201+0000","last_became_active":"2024-06-29T15:39:12.180266+0000","last_became_peered":"2024-06-29T15:39:12.180266+0000","last_unstale":"2024-06-29T15:39:13.323201+0000","last_undegraded":"2024-06-29T15:39:13.323201+0000","last_fullsized":"2024-06-29T15:39:13.323201+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:51.638294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176900+0000","last_change":"2024-06-29T15:39:12.189952+0000","last_active":"2024-06-29T15:39:13.176900+0000","last_peered":"2024-06-29T15:39:13.176900+0000","last_clean":"2024-06-29T15:39:13.176900+0000","last_became_active":"2024-06-29T15:39:12.189651+0000","last_became_peered":"2024-06-29T15:39:12.189651+0000","last_unstale":"2024-06-29T15:39:13.176900+0000","last_undegraded":"2024-06-29T15:39:13.176900+0000","last_fullsized":"2024-06-29T15:39:13.176900+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:07.772575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174680+0000","last_change":"2024-06-29T15:39:12.175835+0000","last_active":"2024-06-29T15:39:13.174680+0000","last_peered":"2024-06-29T15:39:13.174680+0000","last_clean":"2024-06-29T15:39:13.174680+0000","last_became_active":"2024-06-29T15:39:12.175503+0000","last_became_peered":"2024-06-29T15:39:12.175503+0000","last_unstale":"2024-06-29T15:39:13.174680+0000","last_undegraded":"2024-06-29T15:39:13.174680+0000","last_fullsized":"2024-06-29T15:39:13.174680+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:01.116296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323325+0000","last_change":"2024-06-29T15:39:12.181677+0000","last_active":"2024-06-29T15:39:13.323325+0000","last_peered":"2024-06-29T15:39:13.323325+0000","last_clean":"2024-06-29T15:39:13.323325+0000","last_became_active":"2024-06-29T15:39:12.181323+0000","last_became_peered":"2024-06-29T15:39:12.181323+0000","last_unstale":"2024-06-29T15:39:13.323325+0000","last_undegraded":"2024-06-29T15:39:13.323325+0000","last_fullsized":"2024-06-29T15:39:13.323325+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:06.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323284+0000","last_change":"2024-06-29T15:39:12.188162+0000","last_active":"2024-06-29T15:39:13.323284+0000","last_peered":"2024-06-29T15:39:13.323284+0000","last_clean":"2024-06-29T15:39:13.323284+0000","last_became_active":"2024-06-29T15:39:12.187829+0000","last_became_peered":"2024-06-29T15:39:12.187829+0000","last_unstale":"2024-06-29T15:39:13.323284+0000","last_undegraded":"2024-06-29T15:39:13.323284+0000","last_fullsized":"2024-06-29T15:39:13.323284+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:14.450584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322577+0000","last_change":"2024-06-29T15:39:12.192892+0000","last_active":"2024-06-29T15:39:13.322577+0000","last_peered":"2024-06-29T15:39:13.322577+0000","last_clean":"2024-06-29T15:39:13.322577+0000","last_became_active":"2024-06-29T15:39:12.192629+0000","last_became_peered":"2024-06-29T15:39:12.192629+0000","last_unstale":"2024-06-29T15:39:13.322577+0000","last_undegraded":"2024-06-29T15:39:13.322577+0000","last_fullsized":"2024-06-29T15:39:13.322577+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:38.128367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323406+0000","last_change":"2024-06-29T15:39:12.180704+0000","last_active":"2024-06-29T15:39:13.323406+0000","last_peered":"2024-06-29T15:39:13.323406+0000","last_clean":"2024-06-29T15:39:13.323406+0000","last_became_active":"2024-06-29T15:39:12.180397+0000","last_became_peered":"2024-06-29T15:39:12.180397+0000","last_unstale":"2024-06-29T15:39:13.323406+0000","last_undegraded":"2024-06-29T15:39:13.323406+0000","last_fullsized":"2024-06-29T15:39:13.323406+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:14.450584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323610+0000","last_change":"2024-06-29T15:39:12.182865+0000","last_active":"2024-06-29T15:39:13.323610+0000","last_peered":"2024-06-29T15:39:13.323610+0000","last_clean":"2024-06-29T15:39:13.323610+0000","last_became_active":"2024-06-29T15:39:12.182554+0000","last_became_peered":"2024-06-29T15:39:12.182554+0000","last_unstale":"2024-06-29T15:39:13.323610+0000","last_undegraded":"2024-06-29T15:39:13.323610+0000","last_fullsized":"2024-06-29T15:39:13.323610+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:07.772575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323436+0000","last_change":"2024-06-29T15:39:12.176705+0000","last_active":"2024-06-29T15:39:13.323436+0000","last_peered":"2024-06-29T15:39:13.323436+0000","last_clean":"2024-06-29T15:39:13.323436+0000","last_became_active":"2024-06-29T15:39:12.176429+0000","last_became_peered":"2024-06-29T15:39:12.176429+0000","last_unstale":"2024-06-29T15:39:13.323436+0000","last_undegraded":"2024-06-29T15:39:13.323436+0000","last_fullsized":"2024-06-29T15:39:13.323436+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:12.462070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323289+0000","last_change":"2024-06-29T15:39:12.183433+0000","last_active":"2024-06-29T15:39:13.323289+0000","last_peered":"2024-06-29T15:39:13.323289+0000","last_clean":"2024-06-29T15:39:13.323289+0000","last_became_active":"2024-06-29T15:39:12.183112+0000","last_became_peered":"2024-06-29T15:39:12.183112+0000","last_unstale":"2024-06-29T15:39:13.323289+0000","last_undegraded":"2024-06-29T15:39:13.323289+0000","last_fullsized":"2024-06-29T15:39:13.323289+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:20.498391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323076+0000","last_change":"2024-06-29T15:39:12.180345+0000","last_active":"2024-06-29T15:39:13.323076+0000","last_peered":"2024-06-29T15:39:13.323076+0000","last_clean":"2024-06-29T15:39:13.323076+0000","last_became_active":"2024-06-29T15:39:12.180044+0000","last_became_peered":"2024-06-29T15:39:12.180044+0000","last_unstale":"2024-06-29T15:39:13.323076+0000","last_undegraded":"2024-06-29T15:39:13.323076+0000","last_fullsized":"2024-06-29T15:39:13.323076+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:09.931223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176410+0000","last_change":"2024-06-29T15:39:12.177026+0000","last_active":"2024-06-29T15:39:13.176410+0000","last_peered":"2024-06-29T15:39:13.176410+0000","last_clean":"2024-06-29T15:39:13.176410+0000","last_became_active":"2024-06-29T15:39:12.176677+0000","last_became_peered":"2024-06-29T15:39:12.176677+0000","last_unstale":"2024-06-29T15:39:13.176410+0000","last_undegraded":"2024-06-29T15:39:13.176410+0000","last_fullsized":"2024-06-29T15:39:13.176410+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:06.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176778+0000","last_change":"2024-06-29T15:39:12.184481+0000","last_active":"2024-06-29T15:39:13.176778+0000","last_peered":"2024-06-29T15:39:13.176778+0000","last_clean":"2024-06-29T15:39:13.176778+0000","last_became_active":"2024-06-29T15:39:12.184173+0000","last_became_peered":"2024-06-29T15:39:12.184173+0000","last_unstale":"2024-06-29T15:39:13.176778+0000","last_undegraded":"2024-06-29T15:39:13.176778+0000","last_fullsized":"2024-06-29T15:39:13.176778+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:38.128367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322344+0000","last_change":"2024-06-29T15:39:12.182667+0000","last_active":"2024-06-29T15:39:13.322344+0000","last_peered":"2024-06-29T15:39:13.322344+0000","last_clean":"2024-06-29T15:39:13.322344+0000","last_became_active":"2024-06-29T15:39:12.182290+0000","last_became_peered":"2024-06-29T15:39:12.182290+0000","last_unstale":"2024-06-29T15:39:13.322344+0000","last_undegraded":"2024-06-29T15:39:13.322344+0000","last_fullsized":"2024-06-29T15:39:13.322344+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:40.089740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.321795+0000","last_change":"2024-06-29T15:39:12.188237+0000","last_active":"2024-06-29T15:39:13.321795+0000","last_peered":"2024-06-29T15:39:13.321795+0000","last_clean":"2024-06-29T15:39:13.321795+0000","last_became_active":"2024-06-29T15:39:12.187952+0000","last_became_peered":"2024-06-29T15:39:12.187952+0000","last_unstale":"2024-06-29T15:39:13.321795+0000","last_undegraded":"2024-06-29T15:39:13.321795+0000","last_fullsized":"2024-06-29T15:39:13.321795+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:09.931223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323398+0000","last_change":"2024-06-29T15:39:12.174114+0000","last_active":"2024-06-29T15:39:13.323398+0000","last_peered":"2024-06-29T15:39:13.323398+0000","last_clean":"2024-06-29T15:39:13.323398+0000","last_became_active":"2024-06-29T15:39:12.173799+0000","last_became_peered":"2024-06-29T15:39:12.173799+0000","last_unstale":"2024-06-29T15:39:13.323398+0000","last_undegraded":"2024-06-29T15:39:13.323398+0000","last_fullsized":"2024-06-29T15:39:13.323398+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:38.128367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322490+0000","last_change":"2024-06-29T15:39:12.180032+0000","last_active":"2024-06-29T15:39:13.322490+0000","last_peered":"2024-06-29T15:39:13.322490+0000","last_clean":"2024-06-29T15:39:13.322490+0000","last_became_active":"2024-06-29T15:39:12.179723+0000","last_became_peered":"2024-06-29T15:39:12.179723+0000","last_unstale":"2024-06-29T15:39:13.322490+0000","last_undegraded":"2024-06-29T15:39:13.322490+0000","last_fullsized":"2024-06-29T15:39:13.322490+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:30.341348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323330+0000","last_change":"2024-06-29T15:39:12.188051+0000","last_active":"2024-06-29T15:39:13.323330+0000","last_peered":"2024-06-29T15:39:13.323330+0000","last_clean":"2024-06-29T15:39:13.323330+0000","last_became_active":"2024-06-29T15:39:12.187588+0000","last_became_peered":"2024-06-29T15:39:12.187588+0000","last_unstale":"2024-06-29T15:39:13.323330+0000","last_undegraded":"2024-06-29T15:39:13.323330+0000","last_fullsized":"2024-06-29T15:39:13.323330+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:12.462070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176074+0000","last_change":"2024-06-29T15:39:12.179779+0000","last_active":"2024-06-29T15:39:13.176074+0000","last_peered":"2024-06-29T15:39:13.176074+0000","last_clean":"2024-06-29T15:39:13.176074+0000","last_became_active":"2024-06-29T15:39:12.176792+0000","last_became_peered":"2024-06-29T15:39:12.176792+0000","last_unstale":"2024-06-29T15:39:13.176074+0000","last_undegraded":"2024-06-29T15:39:13.176074+0000","last_fullsized":"2024-06-29T15:39:13.176074+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:20.498391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323149+0000","last_change":"2024-06-29T15:39:12.175294+0000","last_active":"2024-06-29T15:39:13.323149+0000","last_peered":"2024-06-29T15:39:13.323149+0000","last_clean":"2024-06-29T15:39:13.323149+0000","last_became_active":"2024-06-29T15:39:12.174680+0000","last_became_peered":"2024-06-29T15:39:12.174680+0000","last_unstale":"2024-06-29T15:39:13.323149+0000","last_undegraded":"2024-06-29T15:39:13.323149+0000","last_fullsized":"2024-06-29T15:39:13.323149+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:48.748686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323104+0000","last_change":"2024-06-29T15:39:12.186535+0000","last_active":"2024-06-29T15:39:13.323104+0000","last_peered":"2024-06-29T15:39:13.323104+0000","last_clean":"2024-06-29T15:39:13.323104+0000","last_became_active":"2024-06-29T15:39:12.186228+0000","last_became_peered":"2024-06-29T15:39:12.186228+0000","last_unstale":"2024-06-29T15:39:13.323104+0000","last_undegraded":"2024-06-29T15:39:13.323104+0000","last_fullsized":"2024-06-29T15:39:13.323104+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:48.748686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323082+0000","last_change":"2024-06-29T15:39:12.171265+0000","last_active":"2024-06-29T15:39:13.323082+0000","last_peered":"2024-06-29T15:39:13.323082+0000","last_clean":"2024-06-29T15:39:13.323082+0000","last_became_active":"2024-06-29T15:39:12.170958+0000","last_became_peered":"2024-06-29T15:39:12.170958+0000","last_unstale":"2024-06-29T15:39:13.323082+0000","last_undegraded":"2024-06-29T15:39:13.323082+0000","last_fullsized":"2024-06-29T15:39:13.323082+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:37.338437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323536+0000","last_change":"2024-06-29T15:39:12.177761+0000","last_active":"2024-06-29T15:39:13.323536+0000","last_peered":"2024-06-29T15:39:13.323536+0000","last_clean":"2024-06-29T15:39:13.323536+0000","last_became_active":"2024-06-29T15:39:12.177441+0000","last_became_peered":"2024-06-29T15:39:12.177441+0000","last_unstale":"2024-06-29T15:39:13.323536+0000","last_undegraded":"2024-06-29T15:39:13.323536+0000","last_fullsized":"2024-06-29T15:39:13.323536+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:38.128367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.177012+0000","last_change":"2024-06-29T15:39:12.186256+0000","last_active":"2024-06-29T15:39:13.177012+0000","last_peered":"2024-06-29T15:39:13.177012+0000","last_clean":"2024-06-29T15:39:13.177012+0000","last_became_active":"2024-06-29T15:39:12.185948+0000","last_became_peered":"2024-06-29T15:39:12.185948+0000","last_unstale":"2024-06-29T15:39:13.177012+0000","last_undegraded":"2024-06-29T15:39:13.177012+0000","last_fullsized":"2024-06-29T15:39:13.177012+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:08.650885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322670+0000","last_change":"2024-06-29T15:39:12.185241+0000","last_active":"2024-06-29T15:39:13.322670+0000","last_peered":"2024-06-29T15:39:13.322670+0000","last_clean":"2024-06-29T15:39:13.322670+0000","last_became_active":"2024-06-29T15:39:12.184855+0000","last_became_peered":"2024-06-29T15:39:12.184855+0000","last_unstale":"2024-06-29T15:39:13.322670+0000","last_undegraded":"2024-06-29T15:39:13.322670+0000","last_fullsized":"2024-06-29T15:39:13.322670+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:07.772575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.177011+0000","last_change":"2024-06-29T15:39:12.175252+0000","last_active":"2024-06-29T15:39:13.177011+0000","last_peered":"2024-06-29T15:39:13.177011+0000","last_clean":"2024-06-29T15:39:13.177011+0000","last_became_active":"2024-06-29T15:39:12.174927+0000","last_became_peered":"2024-06-29T15:39:12.174927+0000","last_unstale":"2024-06-29T15:39:13.177011+0000","last_undegraded":"2024-06-29T15:39:13.177011+0000","last_fullsized":"2024-06-29T15:39:13.177011+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:10.964979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174689+0000","last_change":"2024-06-29T15:39:12.178781+0000","last_active":"2024-06-29T15:39:13.174689+0000","last_peered":"2024-06-29T15:39:13.174689+0000","last_clean":"2024-06-29T15:39:13.174689+0000","last_became_active":"2024-06-29T15:39:12.178470+0000","last_became_peered":"2024-06-29T15:39:12.178470+0000","last_unstale":"2024-06-29T15:39:13.174689+0000","last_undegraded":"2024-06-29T15:39:13.174689+0000","last_fullsized":"2024-06-29T15:39:13.174689+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:59:36.920023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176633+0000","last_change":"2024-06-29T15:39:12.172473+0000","last_active":"2024-06-29T15:39:13.176633+0000","last_peered":"2024-06-29T15:39:13.176633+0000","last_clean":"2024-06-29T15:39:13.176633+0000","last_became_active":"2024-06-29T15:39:12.172041+0000","last_became_peered":"2024-06-29T15:39:12.172041+0000","last_unstale":"2024-06-29T15:39:13.176633+0000","last_undegraded":"2024-06-29T15:39:13.176633+0000","last_fullsized":"2024-06-29T15:39:13.176633+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:14.450584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.175688+0000","last_change":"2024-06-29T15:39:12.183922+0000","last_active":"2024-06-29T15:39:13.175688+0000","last_peered":"2024-06-29T15:39:13.175688+0000","last_clean":"2024-06-29T15:39:13.175688+0000","last_became_active":"2024-06-29T15:39:12.183621+0000","last_became_peered":"2024-06-29T15:39:12.183621+0000","last_unstale":"2024-06-29T15:39:13.175688+0000","last_undegraded":"2024-06-29T15:39:13.175688+0000","last_fullsized":"2024-06-29T15:39:13.175688+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:23:18.506427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176038+0000","last_change":"2024-06-29T15:39:12.188270+0000","last_active":"2024-06-29T15:39:13.176038+0000","last_peered":"2024-06-29T15:39:13.176038+0000","last_clean":"2024-06-29T15:39:13.176038+0000","last_became_active":"2024-06-29T15:39:12.187954+0000","last_became_peered":"2024-06-29T15:39:12.187954+0000","last_unstale":"2024-06-29T15:39:13.176038+0000","last_undegraded":"2024-06-29T15:39:13.176038+0000","last_fullsized":"2024-06-29T15:39:13.176038+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:26:51.077527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.321837+0000","last_change":"2024-06-29T15:39:12.179912+0000","last_active":"2024-06-29T15:39:13.321837+0000","last_peered":"2024-06-29T15:39:13.321837+0000","last_clean":"2024-06-29T15:39:13.321837+0000","last_became_active":"2024-06-29T15:39:12.179562+0000","last_became_peered":"2024-06-29T15:39:12.179562+0000","last_unstale":"2024-06-29T15:39:13.321837+0000","last_undegraded":"2024-06-29T15:39:13.321837+0000","last_fullsized":"2024-06-29T15:39:13.321837+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:35.126003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322745+0000","last_change":"2024-06-29T15:39:12.185725+0000","last_active":"2024-06-29T15:39:13.322745+0000","last_peered":"2024-06-29T15:39:13.322745+0000","last_clean":"2024-06-29T15:39:13.322745+0000","last_became_active":"2024-06-29T15:39:12.185357+0000","last_became_peered":"2024-06-29T15:39:12.185357+0000","last_unstale":"2024-06-29T15:39:13.322745+0000","last_undegraded":"2024-06-29T15:39:13.322745+0000","last_fullsized":"2024-06-29T15:39:13.322745+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:23:17.196740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"180'4","reported_seq":18,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322933+0000","last_change":"2024-06-29T15:39:11.176535+0000","last_active":"2024-06-29T15:39:13.322933+0000","last_peered":"2024-06-29T15:39:13.322933+0000","last_clean":"2024-06-29T15:39:13.322933+0000","last_became_active":"2024-06-29T15:39:11.176193+0000","last_became_peered":"2024-06-29T15:39:11.176193+0000","last_unstale":"2024-06-29T15:39:13.322933+0000","last_undegraded":"2024-06-29T15:39:13.322933+0000","last_fullsized":"2024-06-29T15:39:13.322933+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:50.168689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176076+0000","last_change":"2024-06-29T15:39:12.179493+0000","last_active":"2024-06-29T15:39:13.176076+0000","last_peered":"2024-06-29T15:39:13.176076+0000","last_clean":"2024-06-29T15:39:13.176076+0000","last_became_active":"2024-06-29T15:39:12.179087+0000","last_became_peered":"2024-06-29T15:39:12.179087+0000","last_unstale":"2024-06-29T15:39:13.176076+0000","last_undegraded":"2024-06-29T15:39:13.176076+0000","last_fullsized":"2024-06-29T15:39:13.176076+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:18:54.770043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"180'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.175638+0000","last_change":"2024-06-29T15:39:11.169170+0000","last_active":"2024-06-29T15:39:13.175638+0000","last_peered":"2024-06-29T15:39:13.175638+0000","last_clean":"2024-06-29T15:39:13.175638+0000","last_became_active":"2024-06-29T15:39:11.168784+0000","last_became_peered":"2024-06-29T15:39:11.168784+0000","last_unstale":"2024-06-29T15:39:13.175638+0000","last_undegraded":"2024-06-29T15:39:13.175638+0000","last_fullsized":"2024-06-29T15:39:13.175638+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:40.672795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323182+0000","last_change":"2024-06-29T15:39:12.179979+0000","last_active":"2024-06-29T15:39:13.323182+0000","last_peered":"2024-06-29T15:39:13.323182+0000","last_clean":"2024-06-29T15:39:13.323182+0000","last_became_active":"2024-06-29T15:39:12.179669+0000","last_became_peered":"2024-06-29T15:39:12.179669+0000","last_unstale":"2024-06-29T15:39:13.323182+0000","last_undegraded":"2024-06-29T15:39:13.323182+0000","last_fullsized":"2024-06-29T15:39:13.323182+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:51.638294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174731+0000","last_change":"2024-06-29T15:39:11.160720+0000","last_active":"2024-06-29T15:39:13.174731+0000","last_peered":"2024-06-29T15:39:13.174731+0000","last_clean":"2024-06-29T15:39:13.174731+0000","last_became_active":"2024-06-29T15:39:11.160361+0000","last_became_peered":"2024-06-29T15:39:11.160361+0000","last_unstale":"2024-06-29T15:39:13.174731+0000","last_undegraded":"2024-06-29T15:39:13.174731+0000","last_fullsized":"2024-06-29T15:39:13.174731+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:47.742779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.177091+0000","last_change":"2024-06-29T15:39:12.183965+0000","last_active":"2024-06-29T15:39:13.177091+0000","last_peered":"2024-06-29T15:39:13.177091+0000","last_clean":"2024-06-29T15:39:13.177091+0000","last_became_active":"2024-06-29T15:39:12.183015+0000","last_became_peered":"2024-06-29T15:39:12.183015+0000","last_unstale":"2024-06-29T15:39:13.177091+0000","last_undegraded":"2024-06-29T15:39:13.177091+0000","last_fullsized":"2024-06-29T15:39:13.177091+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:08.994857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174864+0000","last_change":"2024-06-29T15:39:11.169812+0000","last_active":"2024-06-29T15:39:13.174864+0000","last_peered":"2024-06-29T15:39:13.174864+0000","last_clean":"2024-06-29T15:39:13.174864+0000","last_became_active":"2024-06-29T15:39:11.169501+0000","last_became_peered":"2024-06-29T15:39:11.169501+0000","last_unstale":"2024-06-29T15:39:13.174864+0000","last_undegraded":"2024-06-29T15:39:13.174864+0000","last_fullsized":"2024-06-29T15:39:13.174864+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:19.492485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322366+0000","last_change":"2024-06-29T15:39:12.185582+0000","last_active":"2024-06-29T15:39:13.322366+0000","last_peered":"2024-06-29T15:39:13.322366+0000","last_clean":"2024-06-29T15:39:13.322366+0000","last_became_active":"2024-06-29T15:39:12.185270+0000","last_became_peered":"2024-06-29T15:39:12.185270+0000","last_unstale":"2024-06-29T15:39:13.322366+0000","last_undegraded":"2024-06-29T15:39:13.322366+0000","last_fullsized":"2024-06-29T15:39:13.322366+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:14.450584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"180'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176767+0000","last_change":"2024-06-29T15:39:11.171496+0000","last_active":"2024-06-29T15:39:13.176767+0000","last_peered":"2024-06-29T15:39:13.176767+0000","last_clean":"2024-06-29T15:39:13.176767+0000","last_became_active":"2024-06-29T15:39:11.171216+0000","last_became_peered":"2024-06-29T15:39:11.171216+0000","last_unstale":"2024-06-29T15:39:13.176767+0000","last_undegraded":"2024-06-29T15:39:13.176767+0000","last_fullsized":"2024-06-29T15:39:13.176767+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:12:14.900156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174517+0000","last_change":"2024-06-29T15:39:12.186561+0000","last_active":"2024-06-29T15:39:13.174517+0000","last_peered":"2024-06-29T15:39:13.174517+0000","last_clean":"2024-06-29T15:39:13.174517+0000","last_became_active":"2024-06-29T15:39:12.186190+0000","last_became_peered":"2024-06-29T15:39:12.186190+0000","last_unstale":"2024-06-29T15:39:13.174517+0000","last_undegraded":"2024-06-29T15:39:13.174517+0000","last_fullsized":"2024-06-29T15:39:13.174517+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:38:23.275712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"180'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174313+0000","last_change":"2024-06-29T15:39:11.170320+0000","last_active":"2024-06-29T15:39:13.174313+0000","last_peered":"2024-06-29T15:39:13.174313+0000","last_clean":"2024-06-29T15:39:13.174313+0000","last_became_active":"2024-06-29T15:39:11.169334+0000","last_became_peered":"2024-06-29T15:39:11.169334+0000","last_unstale":"2024-06-29T15:39:13.174313+0000","last_undegraded":"2024-06-29T15:39:13.174313+0000","last_fullsized":"2024-06-29T15:39:13.174313+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:38:44.208143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174282+0000","last_change":"2024-06-29T15:39:12.176979+0000","last_active":"2024-06-29T15:39:13.174282+0000","last_peered":"2024-06-29T15:39:13.174282+0000","last_clean":"2024-06-29T15:39:13.174282+0000","last_became_active":"2024-06-29T15:39:12.176669+0000","last_became_peered":"2024-06-29T15:39:12.176669+0000","last_unstale":"2024-06-29T15:39:13.174282+0000","last_undegraded":"2024-06-29T15:39:13.174282+0000","last_fullsized":"2024-06-29T15:39:13.174282+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:08.994857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323064+0000","last_change":"2024-06-29T15:39:11.164365+0000","last_active":"2024-06-29T15:39:13.323064+0000","last_peered":"2024-06-29T15:39:13.323064+0000","last_clean":"2024-06-29T15:39:13.323064+0000","last_became_active":"2024-06-29T15:39:11.163313+0000","last_became_peered":"2024-06-29T15:39:11.163313+0000","last_unstale":"2024-06-29T15:39:13.323064+0000","last_undegraded":"2024-06-29T15:39:13.323064+0000","last_fullsized":"2024-06-29T15:39:13.323064+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:53.735143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323143+0000","last_change":"2024-06-29T15:39:12.187203+0000","last_active":"2024-06-29T15:39:13.323143+0000","last_peered":"2024-06-29T15:39:13.323143+0000","last_clean":"2024-06-29T15:39:13.323143+0000","last_became_active":"2024-06-29T15:39:12.186847+0000","last_became_peered":"2024-06-29T15:39:12.186847+0000","last_unstale":"2024-06-29T15:39:13.323143+0000","last_undegraded":"2024-06-29T15:39:13.323143+0000","last_fullsized":"2024-06-29T15:39:13.323143+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:09.931223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.321908+0000","last_change":"2024-06-29T15:39:11.167992+0000","last_active":"2024-06-29T15:39:13.321908+0000","last_peered":"2024-06-29T15:39:13.321908+0000","last_clean":"2024-06-29T15:39:13.321908+0000","last_became_active":"2024-06-29T15:39:11.167642+0000","last_became_peered":"2024-06-29T15:39:11.167642+0000","last_unstale":"2024-06-29T15:39:13.321908+0000","last_undegraded":"2024-06-29T15:39:13.321908+0000","last_fullsized":"2024-06-29T15:39:13.321908+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:53.735143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322796+0000","last_change":"2024-06-29T15:39:12.174189+0000","last_active":"2024-06-29T15:39:13.322796+0000","last_peered":"2024-06-29T15:39:13.322796+0000","last_clean":"2024-06-29T15:39:13.322796+0000","last_became_active":"2024-06-29T15:39:12.173878+0000","last_became_peered":"2024-06-29T15:39:12.173878+0000","last_unstale":"2024-06-29T15:39:13.322796+0000","last_undegraded":"2024-06-29T15:39:13.322796+0000","last_fullsized":"2024-06-29T15:39:13.322796+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:05:13.755084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323339+0000","last_change":"2024-06-29T15:39:11.174542+0000","last_active":"2024-06-29T15:39:13.323339+0000","last_peered":"2024-06-29T15:39:13.323339+0000","last_clean":"2024-06-29T15:39:13.323339+0000","last_became_active":"2024-06-29T15:39:11.174299+0000","last_became_peered":"2024-06-29T15:39:11.174299+0000","last_unstale":"2024-06-29T15:39:13.323339+0000","last_undegraded":"2024-06-29T15:39:13.323339+0000","last_fullsized":"2024-06-29T15:39:13.323339+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:12:14.900156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322479+0000","last_change":"2024-06-29T15:39:12.175133+0000","last_active":"2024-06-29T15:39:13.322479+0000","last_peered":"2024-06-29T15:39:13.322479+0000","last_clean":"2024-06-29T15:39:13.322479+0000","last_became_active":"2024-06-29T15:39:12.174837+0000","last_became_peered":"2024-06-29T15:39:12.174837+0000","last_unstale":"2024-06-29T15:39:13.322479+0000","last_undegraded":"2024-06-29T15:39:13.322479+0000","last_fullsized":"2024-06-29T15:39:13.322479+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:58.097609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"180'3","reported_seq":17,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323326+0000","last_change":"2024-06-29T15:39:11.170472+0000","last_active":"2024-06-29T15:39:13.323326+0000","last_peered":"2024-06-29T15:39:13.323326+0000","last_clean":"2024-06-29T15:39:13.323326+0000","last_became_active":"2024-06-29T15:39:11.170146+0000","last_became_peered":"2024-06-29T15:39:11.170146+0000","last_unstale":"2024-06-29T15:39:13.323326+0000","last_undegraded":"2024-06-29T15:39:13.323326+0000","last_fullsized":"2024-06-29T15:39:13.323326+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:53.735143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176349+0000","last_change":"2024-06-29T15:39:12.191198+0000","last_active":"2024-06-29T15:39:13.176349+0000","last_peered":"2024-06-29T15:39:13.176349+0000","last_clean":"2024-06-29T15:39:13.176349+0000","last_became_active":"2024-06-29T15:39:12.190836+0000","last_became_peered":"2024-06-29T15:39:12.190836+0000","last_unstale":"2024-06-29T15:39:13.176349+0000","last_undegraded":"2024-06-29T15:39:13.176349+0000","last_fullsized":"2024-06-29T15:39:13.176349+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:48:49.861345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"180'6","reported_seq":20,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.175943+0000","last_change":"2024-06-29T15:39:11.171125+0000","last_active":"2024-06-29T15:39:13.175943+0000","last_peered":"2024-06-29T15:39:13.175943+0000","last_clean":"2024-06-29T15:39:13.175943+0000","last_became_active":"2024-06-29T15:39:11.170808+0000","last_became_peered":"2024-06-29T15:39:11.170808+0000","last_unstale":"2024-06-29T15:39:13.175943+0000","last_undegraded":"2024-06-29T15:39:13.175943+0000","last_fullsized":"2024-06-29T15:39:13.175943+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:36.329410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322140+0000","last_change":"2024-06-29T15:39:12.173531+0000","last_active":"2024-06-29T15:39:13.322140+0000","last_peered":"2024-06-29T15:39:13.322140+0000","last_clean":"2024-06-29T15:39:13.322140+0000","last_became_active":"2024-06-29T15:39:12.173136+0000","last_became_peered":"2024-06-29T15:39:12.173136+0000","last_unstale":"2024-06-29T15:39:13.322140+0000","last_undegraded":"2024-06-29T15:39:13.322140+0000","last_fullsized":"2024-06-29T15:39:13.322140+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:18.660879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174506+0000","last_change":"2024-06-29T15:39:11.168493+0000","last_active":"2024-06-29T15:39:13.174506+0000","last_peered":"2024-06-29T15:39:13.174506+0000","last_clean":"2024-06-29T15:39:13.174506+0000","last_became_active":"2024-06-29T15:39:11.167944+0000","last_became_peered":"2024-06-29T15:39:11.167944+0000","last_unstale":"2024-06-29T15:39:13.174506+0000","last_undegraded":"2024-06-29T15:39:13.174506+0000","last_fullsized":"2024-06-29T15:39:13.174506+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:50.632387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322076+0000","last_change":"2024-06-29T15:39:12.189959+0000","last_active":"2024-06-29T15:39:13.322076+0000","last_peered":"2024-06-29T15:39:13.322076+0000","last_clean":"2024-06-29T15:39:13.322076+0000","last_became_active":"2024-06-29T15:39:12.189684+0000","last_became_peered":"2024-06-29T15:39:12.189684+0000","last_unstale":"2024-06-29T15:39:13.322076+0000","last_undegraded":"2024-06-29T15:39:13.322076+0000","last_fullsized":"2024-06-29T15:39:13.322076+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:48.748686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322462+0000","last_change":"2024-06-29T15:39:11.172945+0000","last_active":"2024-06-29T15:39:13.322462+0000","last_peered":"2024-06-29T15:39:13.322462+0000","last_clean":"2024-06-29T15:39:13.322462+0000","last_became_active":"2024-06-29T15:39:11.172668+0000","last_became_peered":"2024-06-29T15:39:11.172668+0000","last_unstale":"2024-06-29T15:39:13.322462+0000","last_undegraded":"2024-06-29T15:39:13.322462+0000","last_fullsized":"2024-06-29T15:39:13.322462+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:51:22.435423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174281+0000","last_change":"2024-06-29T15:39:12.174140+0000","last_active":"2024-06-29T15:39:13.174281+0000","last_peered":"2024-06-29T15:39:13.174281+0000","last_clean":"2024-06-29T15:39:13.174281+0000","last_became_active":"2024-06-29T15:39:12.173860+0000","last_became_peered":"2024-06-29T15:39:12.173860+0000","last_unstale":"2024-06-29T15:39:13.174281+0000","last_undegraded":"2024-06-29T15:39:13.174281+0000","last_fullsized":"2024-06-29T15:39:13.174281+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:08.650885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174291+0000","last_change":"2024-06-29T15:39:11.163889+0000","last_active":"2024-06-29T15:39:13.174291+0000","last_peered":"2024-06-29T15:39:13.174291+0000","last_clean":"2024-06-29T15:39:13.174291+0000","last_became_active":"2024-06-29T15:39:11.163491+0000","last_became_peered":"2024-06-29T15:39:11.163491+0000","last_unstale":"2024-06-29T15:39:13.174291+0000","last_undegraded":"2024-06-29T15:39:13.174291+0000","last_fullsized":"2024-06-29T15:39:13.174291+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:08.940401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176640+0000","last_change":"2024-06-29T15:39:12.187444+0000","last_active":"2024-06-29T15:39:13.176640+0000","last_peered":"2024-06-29T15:39:13.176640+0000","last_clean":"2024-06-29T15:39:13.176640+0000","last_became_active":"2024-06-29T15:39:12.186675+0000","last_became_peered":"2024-06-29T15:39:12.186675+0000","last_unstale":"2024-06-29T15:39:13.176640+0000","last_undegraded":"2024-06-29T15:39:13.176640+0000","last_fullsized":"2024-06-29T15:39:13.176640+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:12.462070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"180'8","reported_seq":22,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.175424+0000","last_change":"2024-06-29T15:39:11.176083+0000","last_active":"2024-06-29T15:39:13.175424+0000","last_peered":"2024-06-29T15:39:13.175424+0000","last_clean":"2024-06-29T15:39:13.175424+0000","last_became_active":"2024-06-29T15:39:11.174416+0000","last_became_peered":"2024-06-29T15:39:11.174416+0000","last_unstale":"2024-06-29T15:39:13.175424+0000","last_undegraded":"2024-06-29T15:39:13.175424+0000","last_fullsized":"2024-06-29T15:39:13.175424+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:00.110390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174916+0000","last_change":"2024-06-29T15:39:12.173357+0000","last_active":"2024-06-29T15:39:13.174916+0000","last_peered":"2024-06-29T15:39:13.174916+0000","last_clean":"2024-06-29T15:39:13.174916+0000","last_became_active":"2024-06-29T15:39:12.173046+0000","last_became_peered":"2024-06-29T15:39:12.173046+0000","last_unstale":"2024-06-29T15:39:13.174916+0000","last_undegraded":"2024-06-29T15:39:13.174916+0000","last_fullsized":"2024-06-29T15:39:13.174916+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:37.335316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"180'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.177076+0000","last_change":"2024-06-29T15:39:11.166313+0000","last_active":"2024-06-29T15:39:13.177076+0000","last_peered":"2024-06-29T15:39:13.177076+0000","last_clean":"2024-06-29T15:39:13.177076+0000","last_became_active":"2024-06-29T15:39:11.166004+0000","last_became_peered":"2024-06-29T15:39:11.166004+0000","last_unstale":"2024-06-29T15:39:13.177076+0000","last_undegraded":"2024-06-29T15:39:13.177076+0000","last_fullsized":"2024-06-29T15:39:13.177076+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:36:17.739994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322867+0000","last_change":"2024-06-29T15:39:12.183328+0000","last_active":"2024-06-29T15:39:13.322867+0000","last_peered":"2024-06-29T15:39:13.322867+0000","last_clean":"2024-06-29T15:39:13.322867+0000","last_became_active":"2024-06-29T15:39:12.182868+0000","last_became_peered":"2024-06-29T15:39:12.182868+0000","last_unstale":"2024-06-29T15:39:13.322867+0000","last_undegraded":"2024-06-29T15:39:13.322867+0000","last_fullsized":"2024-06-29T15:39:13.322867+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:36.533840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176307+0000","last_change":"2024-06-29T15:39:11.173418+0000","last_active":"2024-06-29T15:39:13.176307+0000","last_peered":"2024-06-29T15:39:13.176307+0000","last_clean":"2024-06-29T15:39:13.176307+0000","last_became_active":"2024-06-29T15:39:11.173055+0000","last_became_peered":"2024-06-29T15:39:11.173055+0000","last_unstale":"2024-06-29T15:39:13.176307+0000","last_undegraded":"2024-06-29T15:39:13.176307+0000","last_fullsized":"2024-06-29T15:39:13.176307+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:53.735143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174861+0000","last_change":"2024-06-29T15:39:12.185065+0000","last_active":"2024-06-29T15:39:13.174861+0000","last_peered":"2024-06-29T15:39:13.174861+0000","last_clean":"2024-06-29T15:39:13.174861+0000","last_became_active":"2024-06-29T15:39:12.184743+0000","last_became_peered":"2024-06-29T15:39:12.184743+0000","last_unstale":"2024-06-29T15:39:13.174861+0000","last_undegraded":"2024-06-29T15:39:13.174861+0000","last_fullsized":"2024-06-29T15:39:13.174861+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:09.350727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"180'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322769+0000","last_change":"2024-06-29T15:39:11.175278+0000","last_active":"2024-06-29T15:39:13.322769+0000","last_peered":"2024-06-29T15:39:13.322769+0000","last_clean":"2024-06-29T15:39:13.322769+0000","last_became_active":"2024-06-29T15:39:11.174970+0000","last_became_peered":"2024-06-29T15:39:11.174970+0000","last_unstale":"2024-06-29T15:39:13.322769+0000","last_undegraded":"2024-06-29T15:39:13.322769+0000","last_fullsized":"2024-06-29T15:39:13.322769+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:20.423023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322160+0000","last_change":"2024-06-29T15:39:12.189151+0000","last_active":"2024-06-29T15:39:13.322160+0000","last_peered":"2024-06-29T15:39:13.322160+0000","last_clean":"2024-06-29T15:39:13.322160+0000","last_became_active":"2024-06-29T15:39:12.188817+0000","last_became_peered":"2024-06-29T15:39:12.188817+0000","last_unstale":"2024-06-29T15:39:13.322160+0000","last_undegraded":"2024-06-29T15:39:13.322160+0000","last_fullsized":"2024-06-29T15:39:13.322160+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:51.638294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323025+0000","last_change":"2024-06-29T15:39:11.176437+0000","last_active":"2024-06-29T15:39:13.323025+0000","last_peered":"2024-06-29T15:39:13.323025+0000","last_clean":"2024-06-29T15:39:13.323025+0000","last_became_active":"2024-06-29T15:39:11.176114+0000","last_became_peered":"2024-06-29T15:39:11.176114+0000","last_unstale":"2024-06-29T15:39:13.323025+0000","last_undegraded":"2024-06-29T15:39:13.323025+0000","last_fullsized":"2024-06-29T15:39:13.323025+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:37:55.570200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176950+0000","last_change":"2024-06-29T15:39:12.178656+0000","last_active":"2024-06-29T15:39:13.176950+0000","last_peered":"2024-06-29T15:39:13.176950+0000","last_clean":"2024-06-29T15:39:13.176950+0000","last_became_active":"2024-06-29T15:39:12.178339+0000","last_became_peered":"2024-06-29T15:39:12.178339+0000","last_unstale":"2024-06-29T15:39:13.176950+0000","last_undegraded":"2024-06-29T15:39:13.176950+0000","last_fullsized":"2024-06-29T15:39:13.176950+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:23:17.196740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174604+0000","last_change":"2024-06-29T15:39:11.159932+0000","last_active":"2024-06-29T15:39:13.174604+0000","last_peered":"2024-06-29T15:39:13.174604+0000","last_clean":"2024-06-29T15:39:13.174604+0000","last_became_active":"2024-06-29T15:39:11.159624+0000","last_became_peered":"2024-06-29T15:39:11.159624+0000","last_unstale":"2024-06-29T15:39:13.174604+0000","last_undegraded":"2024-06-29T15:39:13.174604+0000","last_fullsized":"2024-06-29T15:39:13.174604+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:05.471412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322244+0000","last_change":"2024-06-29T15:39:12.186851+0000","last_active":"2024-06-29T15:39:13.322244+0000","last_peered":"2024-06-29T15:39:13.322244+0000","last_clean":"2024-06-29T15:39:13.322244+0000","last_became_active":"2024-06-29T15:39:12.185795+0000","last_became_peered":"2024-06-29T15:39:12.185795+0000","last_unstale":"2024-06-29T15:39:13.322244+0000","last_undegraded":"2024-06-29T15:39:13.322244+0000","last_fullsized":"2024-06-29T15:39:13.322244+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:20.498391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322550+0000","last_change":"2024-06-29T15:39:11.175248+0000","last_active":"2024-06-29T15:39:13.322550+0000","last_peered":"2024-06-29T15:39:13.322550+0000","last_clean":"2024-06-29T15:39:13.322550+0000","last_became_active":"2024-06-29T15:39:11.174927+0000","last_became_peered":"2024-06-29T15:39:11.174927+0000","last_unstale":"2024-06-29T15:39:13.322550+0000","last_undegraded":"2024-06-29T15:39:13.322550+0000","last_fullsized":"2024-06-29T15:39:13.322550+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:00.304411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322425+0000","last_change":"2024-06-29T15:39:12.186070+0000","last_active":"2024-06-29T15:39:13.322425+0000","last_peered":"2024-06-29T15:39:13.322425+0000","last_clean":"2024-06-29T15:39:13.322425+0000","last_became_active":"2024-06-29T15:39:12.185772+0000","last_became_peered":"2024-06-29T15:39:12.185772+0000","last_unstale":"2024-06-29T15:39:13.322425+0000","last_undegraded":"2024-06-29T15:39:13.322425+0000","last_fullsized":"2024-06-29T15:39:13.322425+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:21.848026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174310+0000","last_change":"2024-06-29T15:39:11.173238+0000","last_active":"2024-06-29T15:39:13.174310+0000","last_peered":"2024-06-29T15:39:13.174310+0000","last_clean":"2024-06-29T15:39:13.174310+0000","last_became_active":"2024-06-29T15:39:11.172942+0000","last_became_peered":"2024-06-29T15:39:11.172942+0000","last_unstale":"2024-06-29T15:39:13.174310+0000","last_undegraded":"2024-06-29T15:39:13.174310+0000","last_fullsized":"2024-06-29T15:39:13.174310+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:36.332531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323271+0000","last_change":"2024-06-29T15:39:12.172033+0000","last_active":"2024-06-29T15:39:13.323271+0000","last_peered":"2024-06-29T15:39:13.323271+0000","last_clean":"2024-06-29T15:39:13.323271+0000","last_became_active":"2024-06-29T15:39:12.171740+0000","last_became_peered":"2024-06-29T15:39:12.171740+0000","last_unstale":"2024-06-29T15:39:13.323271+0000","last_undegraded":"2024-06-29T15:39:13.323271+0000","last_fullsized":"2024-06-29T15:39:13.323271+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:06.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322510+0000","last_change":"2024-06-29T15:39:11.175891+0000","last_active":"2024-06-29T15:39:13.322510+0000","last_peered":"2024-06-29T15:39:13.322510+0000","last_clean":"2024-06-29T15:39:13.322510+0000","last_became_active":"2024-06-29T15:39:11.175508+0000","last_became_peered":"2024-06-29T15:39:11.175508+0000","last_unstale":"2024-06-29T15:39:13.322510+0000","last_undegraded":"2024-06-29T15:39:13.322510+0000","last_fullsized":"2024-06-29T15:39:13.322510+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:06:37.288488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322251+0000","last_change":"2024-06-29T15:39:12.175449+0000","last_active":"2024-06-29T15:39:13.322251+0000","last_peered":"2024-06-29T15:39:13.322251+0000","last_clean":"2024-06-29T15:39:13.322251+0000","last_became_active":"2024-06-29T15:39:12.174840+0000","last_became_peered":"2024-06-29T15:39:12.174840+0000","last_unstale":"2024-06-29T15:39:13.322251+0000","last_undegraded":"2024-06-29T15:39:13.322251+0000","last_fullsized":"2024-06-29T15:39:13.322251+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:20.822596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322907+0000","last_change":"2024-06-29T15:39:11.175356+0000","last_active":"2024-06-29T15:39:13.322907+0000","last_peered":"2024-06-29T15:39:13.322907+0000","last_clean":"2024-06-29T15:39:13.322907+0000","last_became_active":"2024-06-29T15:39:11.175073+0000","last_became_peered":"2024-06-29T15:39:11.175073+0000","last_unstale":"2024-06-29T15:39:13.322907+0000","last_undegraded":"2024-06-29T15:39:13.322907+0000","last_fullsized":"2024-06-29T15:39:13.322907+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:27.895788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322480+0000","last_change":"2024-06-29T15:39:12.191110+0000","last_active":"2024-06-29T15:39:13.322480+0000","last_peered":"2024-06-29T15:39:13.322480+0000","last_clean":"2024-06-29T15:39:13.322480+0000","last_became_active":"2024-06-29T15:39:12.190799+0000","last_became_peered":"2024-06-29T15:39:12.190799+0000","last_unstale":"2024-06-29T15:39:13.322480+0000","last_undegraded":"2024-06-29T15:39:13.322480+0000","last_fullsized":"2024-06-29T15:39:13.322480+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:12.462070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"181'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.188696+0000","last_change":"2024-06-29T15:39:11.171344+0000","last_active":"2024-06-29T15:39:13.188696+0000","last_peered":"2024-06-29T15:39:13.188696+0000","last_clean":"2024-06-29T15:39:13.188696+0000","last_became_active":"2024-06-29T15:39:11.171015+0000","last_became_peered":"2024-06-29T15:39:11.171015+0000","last_unstale":"2024-06-29T15:39:13.188696+0000","last_undegraded":"2024-06-29T15:39:13.188696+0000","last_fullsized":"2024-06-29T15:39:13.188696+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:27.438455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323198+0000","last_change":"2024-06-29T15:39:12.172568+0000","last_active":"2024-06-29T15:39:13.323198+0000","last_peered":"2024-06-29T15:39:13.323198+0000","last_clean":"2024-06-29T15:39:13.323198+0000","last_became_active":"2024-06-29T15:39:12.172262+0000","last_became_peered":"2024-06-29T15:39:12.172262+0000","last_unstale":"2024-06-29T15:39:13.323198+0000","last_undegraded":"2024-06-29T15:39:13.323198+0000","last_fullsized":"2024-06-29T15:39:13.323198+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:20.498391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323472+0000","last_change":"2024-06-29T15:39:11.170181+0000","last_active":"2024-06-29T15:39:13.323472+0000","last_peered":"2024-06-29T15:39:13.323472+0000","last_clean":"2024-06-29T15:39:13.323472+0000","last_became_active":"2024-06-29T15:39:11.169824+0000","last_became_peered":"2024-06-29T15:39:11.169824+0000","last_unstale":"2024-06-29T15:39:13.323472+0000","last_undegraded":"2024-06-29T15:39:13.323472+0000","last_fullsized":"2024-06-29T15:39:13.323472+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:12:14.900156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322210+0000","last_change":"2024-06-29T15:39:12.180757+0000","last_active":"2024-06-29T15:39:13.322210+0000","last_peered":"2024-06-29T15:39:13.322210+0000","last_clean":"2024-06-29T15:39:13.322210+0000","last_became_active":"2024-06-29T15:39:12.180409+0000","last_became_peered":"2024-06-29T15:39:12.180409+0000","last_unstale":"2024-06-29T15:39:13.322210+0000","last_undegraded":"2024-06-29T15:39:13.322210+0000","last_fullsized":"2024-06-29T15:39:13.322210+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:54:44.310682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322586+0000","last_change":"2024-06-29T15:39:11.173447+0000","last_active":"2024-06-29T15:39:13.322586+0000","last_peered":"2024-06-29T15:39:13.322586+0000","last_clean":"2024-06-29T15:39:13.322586+0000","last_became_active":"2024-06-29T15:39:11.173132+0000","last_became_peered":"2024-06-29T15:39:11.173132+0000","last_unstale":"2024-06-29T15:39:13.322586+0000","last_undegraded":"2024-06-29T15:39:13.322586+0000","last_fullsized":"2024-06-29T15:39:13.322586+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:15:17.041258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322375+0000","last_change":"2024-06-29T15:39:12.180363+0000","last_active":"2024-06-29T15:39:13.322375+0000","last_peered":"2024-06-29T15:39:13.322375+0000","last_clean":"2024-06-29T15:39:13.322375+0000","last_became_active":"2024-06-29T15:39:12.179785+0000","last_became_peered":"2024-06-29T15:39:12.179785+0000","last_unstale":"2024-06-29T15:39:13.322375+0000","last_undegraded":"2024-06-29T15:39:13.322375+0000","last_fullsized":"2024-06-29T15:39:13.322375+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:58.673623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176394+0000","last_change":"2024-06-29T15:39:11.172515+0000","last_active":"2024-06-29T15:39:13.176394+0000","last_peered":"2024-06-29T15:39:13.176394+0000","last_clean":"2024-06-29T15:39:13.176394+0000","last_became_active":"2024-06-29T15:39:11.172186+0000","last_became_peered":"2024-06-29T15:39:11.172186+0000","last_unstale":"2024-06-29T15:39:13.176394+0000","last_undegraded":"2024-06-29T15:39:13.176394+0000","last_fullsized":"2024-06-29T15:39:13.176394+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:08.344821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176231+0000","last_change":"2024-06-29T15:39:12.191410+0000","last_active":"2024-06-29T15:39:13.176231+0000","last_peered":"2024-06-29T15:39:13.176231+0000","last_clean":"2024-06-29T15:39:13.176231+0000","last_became_active":"2024-06-29T15:39:12.188271+0000","last_became_peered":"2024-06-29T15:39:12.188271+0000","last_unstale":"2024-06-29T15:39:13.176231+0000","last_undegraded":"2024-06-29T15:39:13.176231+0000","last_fullsized":"2024-06-29T15:39:13.176231+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:19:44.697460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174403+0000","last_change":"2024-06-29T15:39:11.171394+0000","last_active":"2024-06-29T15:39:13.174403+0000","last_peered":"2024-06-29T15:39:13.174403+0000","last_clean":"2024-06-29T15:39:13.174403+0000","last_became_active":"2024-06-29T15:39:11.171075+0000","last_became_peered":"2024-06-29T15:39:11.171075+0000","last_unstale":"2024-06-29T15:39:13.174403+0000","last_undegraded":"2024-06-29T15:39:13.174403+0000","last_fullsized":"2024-06-29T15:39:13.174403+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:08.925317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322349+0000","last_change":"2024-06-29T15:39:12.187396+0000","last_active":"2024-06-29T15:39:13.322349+0000","last_peered":"2024-06-29T15:39:13.322349+0000","last_clean":"2024-06-29T15:39:13.322349+0000","last_became_active":"2024-06-29T15:39:12.187082+0000","last_became_peered":"2024-06-29T15:39:12.187082+0000","last_unstale":"2024-06-29T15:39:13.322349+0000","last_undegraded":"2024-06-29T15:39:13.322349+0000","last_fullsized":"2024-06-29T15:39:13.322349+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:06.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323304+0000","last_change":"2024-06-29T15:39:11.168054+0000","last_active":"2024-06-29T15:39:13.323304+0000","last_peered":"2024-06-29T15:39:13.323304+0000","last_clean":"2024-06-29T15:39:13.323304+0000","last_became_active":"2024-06-29T15:39:11.167717+0000","last_became_peered":"2024-06-29T15:39:11.167717+0000","last_unstale":"2024-06-29T15:39:13.323304+0000","last_undegraded":"2024-06-29T15:39:13.323304+0000","last_fullsized":"2024-06-29T15:39:13.323304+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:27.438455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176965+0000","last_change":"2024-06-29T15:39:12.182211+0000","last_active":"2024-06-29T15:39:13.176965+0000","last_peered":"2024-06-29T15:39:13.176965+0000","last_clean":"2024-06-29T15:39:13.176965+0000","last_became_active":"2024-06-29T15:39:12.181892+0000","last_became_peered":"2024-06-29T15:39:12.181892+0000","last_unstale":"2024-06-29T15:39:13.176965+0000","last_undegraded":"2024-06-29T15:39:13.176965+0000","last_fullsized":"2024-06-29T15:39:13.176965+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:07.625273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.175440+0000","last_change":"2024-06-29T15:39:11.170040+0000","last_active":"2024-06-29T15:39:13.175440+0000","last_peered":"2024-06-29T15:39:13.175440+0000","last_clean":"2024-06-29T15:39:13.175440+0000","last_became_active":"2024-06-29T15:39:11.169708+0000","last_became_peered":"2024-06-29T15:39:11.169708+0000","last_unstale":"2024-06-29T15:39:13.175440+0000","last_undegraded":"2024-06-29T15:39:13.175440+0000","last_fullsized":"2024-06-29T15:39:13.175440+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:59:35.914117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322085+0000","last_change":"2024-06-29T15:39:12.176363+0000","last_active":"2024-06-29T15:39:13.322085+0000","last_peered":"2024-06-29T15:39:13.322085+0000","last_clean":"2024-06-29T15:39:13.322085+0000","last_became_active":"2024-06-29T15:39:12.175188+0000","last_became_peered":"2024-06-29T15:39:12.175188+0000","last_unstale":"2024-06-29T15:39:13.322085+0000","last_undegraded":"2024-06-29T15:39:13.322085+0000","last_fullsized":"2024-06-29T15:39:13.322085+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:25.393589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176202+0000","last_change":"2024-06-29T15:39:11.174722+0000","last_active":"2024-06-29T15:39:13.176202+0000","last_peered":"2024-06-29T15:39:13.176202+0000","last_clean":"2024-06-29T15:39:13.176202+0000","last_became_active":"2024-06-29T15:39:11.174385+0000","last_became_peered":"2024-06-29T15:39:11.174385+0000","last_unstale":"2024-06-29T15:39:13.176202+0000","last_undegraded":"2024-06-29T15:39:13.176202+0000","last_fullsized":"2024-06-29T15:39:13.176202+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:53.626387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"170'184","reported_seq":847,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322504+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:39:13.322504+0000","last_peered":"2024-06-29T15:39:13.322504+0000","last_clean":"2024-06-29T15:39:13.322504+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:39:13.322504+0000","last_undegraded":"2024-06-29T15:39:13.322504+0000","last_fullsized":"2024-06-29T15:39:13.322504+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":293,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":293,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47872,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323968,"statfs":{"total":96636764160,"available":96587743232,"internally_reserved":0,"allocated":2871296,"data_stored":1686376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575770,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41704,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330136,"statfs":{"total":96636764160,"available":96594059264,"internally_reserved":0,"allocated":2256896,"data_stored":1093676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575768,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41724,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330116,"statfs":{"total":96636764160,"available":96594038784,"internally_reserved":0,"allocated":2273280,"data_stored":1100658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575769,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55620,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316220,"statfs":{"total":96636764160,"available":96579809280,"internally_reserved":0,"allocated":2281472,"data_stored":1101736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59752,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312088,"statfs":{"total":96636764160,"available":96575578112,"internally_reserved":0,"allocated":2256896,"data_stored":1093688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575769,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63912,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307928,"statfs":{"total":96636764160,"available":96571318272,"internally_reserved":0,"allocated":2256896,"data_stored":1093744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575768,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59752,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312088,"statfs":{"total":96636764160,"available":96575578112,"internally_reserved":0,"allocated":2256896,"data_stored":1093688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575769,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56180,"kb_used_data":2792,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315660,"statfs":{"total":96636764160,"available":96579235840,"internally_reserved":0,"allocated":2859008,"data_stored":1690422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:39:16.217 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:39:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.366+0000 7f0833e47640 1 -- 172.21.15.81:0/4131375986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082c0ce500 msgr2=0x7f082c0ce910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.366+0000 7f0833e47640 1 --2- 172.21.15.81:0/4131375986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082c0ce500 0x7f082c0ce910 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f081c005fd0 tx=0x7f081c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.367+0000 7f0833e47640 1 -- 172.21.15.81:0/4131375986 shutdown_connections 2024-06-29T15:39:16.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.367+0000 7f0833e47640 1 --2- 172.21.15.81:0/4131375986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082c0ce500 0x7f082c0ce910 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.367+0000 7f0833e47640 1 --2- 172.21.15.81:0/4131375986 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f082c0b7040 0x7f082c0d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.367+0000 7f0833e47640 1 -- 172.21.15.81:0/4131375986 >> 172.21.15.81:0/4131375986 conn(0x7f082c0b3510 msgr2=0x7f082c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.367+0000 7f0833e47640 1 -- 172.21.15.81:0/4131375986 shutdown_connections 2024-06-29T15:39:16.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.367+0000 7f0833e47640 1 -- 172.21.15.81:0/4131375986 wait complete. 2024-06-29T15:39:16.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.368+0000 7f0833e47640 1 Processor -- start 2024-06-29T15:39:16.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.368+0000 7f0833e47640 1 -- start start 2024-06-29T15:39:16.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f0833e47640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f082c0b7040 0x7f082c1f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f0833e47640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f082c0cbfd0 0x7f082c0b6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f0833e47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082c0ce500 0x7f082c1ea550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f0833e47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f082c0d17b0 con 0x7f082c0ce500 2024-06-29T15:39:16.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f0833e47640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f082c0b4240 con 0x7f082c0cbfd0 2024-06-29T15:39:16.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f0833e47640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f082c0ccaf0 con 0x7f082c0b7040 2024-06-29T15:39:16.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08323e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082c0ce500 0x7f082c1ea550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08323e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082c0ce500 0x7f082c1ea550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56228/0 (socket says 172.21.15.81:56228) 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08323e4640 1 -- 172.21.15.81:0/1500435112 learned_addr learned my addr 172.21.15.81:0/1500435112 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08313e2640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f082c0cbfd0 0x7f082c0b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f0831be3640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f082c0b7040 0x7f082c1f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08323e4640 1 -- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f082c0b7040 msgr2=0x7f082c1f7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08323e4640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f082c0b7040 0x7f082c1f7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08323e4640 1 -- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f082c0cbfd0 msgr2=0x7f082c0b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08323e4640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f082c0cbfd0 0x7f082c0b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08323e4640 1 -- 172.21.15.81:0/1500435112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082c0aeef0 con 0x7f082c0ce500 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08323e4640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082c0ce500 0x7f082c1ea550 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f082400c700 tx=0x7f082400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.370+0000 7f08227fc640 1 -- 172.21.15.81:0/1500435112 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824003b70 con 0x7f082c0ce500 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.371+0000 7f08227fc640 1 -- 172.21.15.81:0/1500435112 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0824003d10 con 0x7f082c0ce500 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.371+0000 7f0833e47640 1 -- 172.21.15.81:0/1500435112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f082c0af1d0 con 0x7f082c0ce500 2024-06-29T15:39:16.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.371+0000 7f08227fc640 1 -- 172.21.15.81:0/1500435112 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824005330 con 0x7f082c0ce500 2024-06-29T15:39:16.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.371+0000 7f0833e47640 1 -- 172.21.15.81:0/1500435112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f082c0af790 con 0x7f082c0ce500 2024-06-29T15:39:16.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.371+0000 7f0833e47640 1 -- 172.21.15.81:0/1500435112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f082c0d0d00 con 0x7f082c0ce500 2024-06-29T15:39:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.372+0000 7f08227fc640 1 -- 172.21.15.81:0/1500435112 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f08240054d0 con 0x7f082c0ce500 2024-06-29T15:39:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.373+0000 7f08227fc640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f08100763f0 0x7f0810078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.373+0000 7f08227fc640 1 -- 172.21.15.81:0/1500435112 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f0824097520 con 0x7f082c0ce500 2024-06-29T15:39:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.373+0000 7f0831be3640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f08100763f0 0x7f0810078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.373+0000 7f0831be3640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f08100763f0 0x7f0810078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f082c073b10 tx=0x7f082800b2e0 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:16.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.377+0000 7f08227fc640 1 -- 172.21.15.81:0/1500435112 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0824060740 con 0x7f082c0ce500 2024-06-29T15:39:16.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.555+0000 7f0833e47640 1 -- 172.21.15.81:0/1500435112 --> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f082c06eed0 con 0x7f08100763f0 2024-06-29T15:39:16.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.560+0000 7f08227fc640 1 -- 172.21.15.81:0/1500435112 <== mgr.7925 v2:172.21.15.167:6832/2546364927 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252494 (secure 0 0 0) 0x7f082c06eed0 con 0x7f08100763f0 2024-06-29T15:39:16.563 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:16.564 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:39:16.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 -- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f08100763f0 msgr2=0x7f0810078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f08100763f0 0x7f0810078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f082c073b10 tx=0x7f082800b2e0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 -- 172.21.15.81:0/1500435112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082c0ce500 msgr2=0x7f082c1ea550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082c0ce500 0x7f082c1ea550 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f082400c700 tx=0x7f082400cbc0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 -- 172.21.15.81:0/1500435112 shutdown_connections 2024-06-29T15:39:16.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f08100763f0 0x7f0810078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082c0ce500 0x7f082c1ea550 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f082c0cbfd0 0x7f082c0b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 --2- 172.21.15.81:0/1500435112 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f082c0b7040 0x7f082c1f7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 -- 172.21.15.81:0/1500435112 >> 172.21.15.81:0/1500435112 conn(0x7f082c0b3510 msgr2=0x7f082c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 -- 172.21.15.81:0/1500435112 shutdown_connections 2024-06-29T15:39:16.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.563+0000 7f0833e47640 1 -- 172.21.15.81:0/1500435112 wait complete. 2024-06-29T15:39:16.575 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:39:15.714966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":338,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":426516,"kb_used_data":18860,"kb_used_omap":12,"kb_used_meta":407539,"kb_avail":754548204,"statfs":{"total":773094113280,"available":772657360896,"internally_reserved":0,"allocated":19312640,"data_stored":9953988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":417320498},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.566646"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322812+0000","last_change":"2024-06-29T15:39:12.190245+0000","last_active":"2024-06-29T15:39:13.322812+0000","last_peered":"2024-06-29T15:39:13.322812+0000","last_clean":"2024-06-29T15:39:13.322812+0000","last_became_active":"2024-06-29T15:39:12.189930+0000","last_became_peered":"2024-06-29T15:39:12.189930+0000","last_unstale":"2024-06-29T15:39:13.322812+0000","last_undegraded":"2024-06-29T15:39:13.322812+0000","last_fullsized":"2024-06-29T15:39:13.322812+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:00:32.419833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323201+0000","last_change":"2024-06-29T15:39:12.181762+0000","last_active":"2024-06-29T15:39:13.323201+0000","last_peered":"2024-06-29T15:39:13.323201+0000","last_clean":"2024-06-29T15:39:13.323201+0000","last_became_active":"2024-06-29T15:39:12.180266+0000","last_became_peered":"2024-06-29T15:39:12.180266+0000","last_unstale":"2024-06-29T15:39:13.323201+0000","last_undegraded":"2024-06-29T15:39:13.323201+0000","last_fullsized":"2024-06-29T15:39:13.323201+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:51.638294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176900+0000","last_change":"2024-06-29T15:39:12.189952+0000","last_active":"2024-06-29T15:39:13.176900+0000","last_peered":"2024-06-29T15:39:13.176900+0000","last_clean":"2024-06-29T15:39:13.176900+0000","last_became_active":"2024-06-29T15:39:12.189651+0000","last_became_peered":"2024-06-29T15:39:12.189651+0000","last_unstale":"2024-06-29T15:39:13.176900+0000","last_undegraded":"2024-06-29T15:39:13.176900+0000","last_fullsized":"2024-06-29T15:39:13.176900+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:07.772575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174680+0000","last_change":"2024-06-29T15:39:12.175835+0000","last_active":"2024-06-29T15:39:13.174680+0000","last_peered":"2024-06-29T15:39:13.174680+0000","last_clean":"2024-06-29T15:39:13.174680+0000","last_became_active":"2024-06-29T15:39:12.175503+0000","last_became_peered":"2024-06-29T15:39:12.175503+0000","last_unstale":"2024-06-29T15:39:13.174680+0000","last_undegraded":"2024-06-29T15:39:13.174680+0000","last_fullsized":"2024-06-29T15:39:13.174680+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:01.116296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323325+0000","last_change":"2024-06-29T15:39:12.181677+0000","last_active":"2024-06-29T15:39:13.323325+0000","last_peered":"2024-06-29T15:39:13.323325+0000","last_clean":"2024-06-29T15:39:13.323325+0000","last_became_active":"2024-06-29T15:39:12.181323+0000","last_became_peered":"2024-06-29T15:39:12.181323+0000","last_unstale":"2024-06-29T15:39:13.323325+0000","last_undegraded":"2024-06-29T15:39:13.323325+0000","last_fullsized":"2024-06-29T15:39:13.323325+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:06.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323284+0000","last_change":"2024-06-29T15:39:12.188162+0000","last_active":"2024-06-29T15:39:13.323284+0000","last_peered":"2024-06-29T15:39:13.323284+0000","last_clean":"2024-06-29T15:39:13.323284+0000","last_became_active":"2024-06-29T15:39:12.187829+0000","last_became_peered":"2024-06-29T15:39:12.187829+0000","last_unstale":"2024-06-29T15:39:13.323284+0000","last_undegraded":"2024-06-29T15:39:13.323284+0000","last_fullsized":"2024-06-29T15:39:13.323284+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:14.450584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322577+0000","last_change":"2024-06-29T15:39:12.192892+0000","last_active":"2024-06-29T15:39:13.322577+0000","last_peered":"2024-06-29T15:39:13.322577+0000","last_clean":"2024-06-29T15:39:13.322577+0000","last_became_active":"2024-06-29T15:39:12.192629+0000","last_became_peered":"2024-06-29T15:39:12.192629+0000","last_unstale":"2024-06-29T15:39:13.322577+0000","last_undegraded":"2024-06-29T15:39:13.322577+0000","last_fullsized":"2024-06-29T15:39:13.322577+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:38.128367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323406+0000","last_change":"2024-06-29T15:39:12.180704+0000","last_active":"2024-06-29T15:39:13.323406+0000","last_peered":"2024-06-29T15:39:13.323406+0000","last_clean":"2024-06-29T15:39:13.323406+0000","last_became_active":"2024-06-29T15:39:12.180397+0000","last_became_peered":"2024-06-29T15:39:12.180397+0000","last_unstale":"2024-06-29T15:39:13.323406+0000","last_undegraded":"2024-06-29T15:39:13.323406+0000","last_fullsized":"2024-06-29T15:39:13.323406+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:14.450584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323610+0000","last_change":"2024-06-29T15:39:12.182865+0000","last_active":"2024-06-29T15:39:13.323610+0000","last_peered":"2024-06-29T15:39:13.323610+0000","last_clean":"2024-06-29T15:39:13.323610+0000","last_became_active":"2024-06-29T15:39:12.182554+0000","last_became_peered":"2024-06-29T15:39:12.182554+0000","last_unstale":"2024-06-29T15:39:13.323610+0000","last_undegraded":"2024-06-29T15:39:13.323610+0000","last_fullsized":"2024-06-29T15:39:13.323610+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:07.772575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323436+0000","last_change":"2024-06-29T15:39:12.176705+0000","last_active":"2024-06-29T15:39:13.323436+0000","last_peered":"2024-06-29T15:39:13.323436+0000","last_clean":"2024-06-29T15:39:13.323436+0000","last_became_active":"2024-06-29T15:39:12.176429+0000","last_became_peered":"2024-06-29T15:39:12.176429+0000","last_unstale":"2024-06-29T15:39:13.323436+0000","last_undegraded":"2024-06-29T15:39:13.323436+0000","last_fullsized":"2024-06-29T15:39:13.323436+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:12.462070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323289+0000","last_change":"2024-06-29T15:39:12.183433+0000","last_active":"2024-06-29T15:39:13.323289+0000","last_peered":"2024-06-29T15:39:13.323289+0000","last_clean":"2024-06-29T15:39:13.323289+0000","last_became_active":"2024-06-29T15:39:12.183112+0000","last_became_peered":"2024-06-29T15:39:12.183112+0000","last_unstale":"2024-06-29T15:39:13.323289+0000","last_undegraded":"2024-06-29T15:39:13.323289+0000","last_fullsized":"2024-06-29T15:39:13.323289+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:20.498391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323076+0000","last_change":"2024-06-29T15:39:12.180345+0000","last_active":"2024-06-29T15:39:13.323076+0000","last_peered":"2024-06-29T15:39:13.323076+0000","last_clean":"2024-06-29T15:39:13.323076+0000","last_became_active":"2024-06-29T15:39:12.180044+0000","last_became_peered":"2024-06-29T15:39:12.180044+0000","last_unstale":"2024-06-29T15:39:13.323076+0000","last_undegraded":"2024-06-29T15:39:13.323076+0000","last_fullsized":"2024-06-29T15:39:13.323076+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:09.931223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176410+0000","last_change":"2024-06-29T15:39:12.177026+0000","last_active":"2024-06-29T15:39:13.176410+0000","last_peered":"2024-06-29T15:39:13.176410+0000","last_clean":"2024-06-29T15:39:13.176410+0000","last_became_active":"2024-06-29T15:39:12.176677+0000","last_became_peered":"2024-06-29T15:39:12.176677+0000","last_unstale":"2024-06-29T15:39:13.176410+0000","last_undegraded":"2024-06-29T15:39:13.176410+0000","last_fullsized":"2024-06-29T15:39:13.176410+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:06.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176778+0000","last_change":"2024-06-29T15:39:12.184481+0000","last_active":"2024-06-29T15:39:13.176778+0000","last_peered":"2024-06-29T15:39:13.176778+0000","last_clean":"2024-06-29T15:39:13.176778+0000","last_became_active":"2024-06-29T15:39:12.184173+0000","last_became_peered":"2024-06-29T15:39:12.184173+0000","last_unstale":"2024-06-29T15:39:13.176778+0000","last_undegraded":"2024-06-29T15:39:13.176778+0000","last_fullsized":"2024-06-29T15:39:13.176778+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:38.128367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322344+0000","last_change":"2024-06-29T15:39:12.182667+0000","last_active":"2024-06-29T15:39:13.322344+0000","last_peered":"2024-06-29T15:39:13.322344+0000","last_clean":"2024-06-29T15:39:13.322344+0000","last_became_active":"2024-06-29T15:39:12.182290+0000","last_became_peered":"2024-06-29T15:39:12.182290+0000","last_unstale":"2024-06-29T15:39:13.322344+0000","last_undegraded":"2024-06-29T15:39:13.322344+0000","last_fullsized":"2024-06-29T15:39:13.322344+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:40.089740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.321795+0000","last_change":"2024-06-29T15:39:12.188237+0000","last_active":"2024-06-29T15:39:13.321795+0000","last_peered":"2024-06-29T15:39:13.321795+0000","last_clean":"2024-06-29T15:39:13.321795+0000","last_became_active":"2024-06-29T15:39:12.187952+0000","last_became_peered":"2024-06-29T15:39:12.187952+0000","last_unstale":"2024-06-29T15:39:13.321795+0000","last_undegraded":"2024-06-29T15:39:13.321795+0000","last_fullsized":"2024-06-29T15:39:13.321795+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:09.931223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323398+0000","last_change":"2024-06-29T15:39:12.174114+0000","last_active":"2024-06-29T15:39:13.323398+0000","last_peered":"2024-06-29T15:39:13.323398+0000","last_clean":"2024-06-29T15:39:13.323398+0000","last_became_active":"2024-06-29T15:39:12.173799+0000","last_became_peered":"2024-06-29T15:39:12.173799+0000","last_unstale":"2024-06-29T15:39:13.323398+0000","last_undegraded":"2024-06-29T15:39:13.323398+0000","last_fullsized":"2024-06-29T15:39:13.323398+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:38.128367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322490+0000","last_change":"2024-06-29T15:39:12.180032+0000","last_active":"2024-06-29T15:39:13.322490+0000","last_peered":"2024-06-29T15:39:13.322490+0000","last_clean":"2024-06-29T15:39:13.322490+0000","last_became_active":"2024-06-29T15:39:12.179723+0000","last_became_peered":"2024-06-29T15:39:12.179723+0000","last_unstale":"2024-06-29T15:39:13.322490+0000","last_undegraded":"2024-06-29T15:39:13.322490+0000","last_fullsized":"2024-06-29T15:39:13.322490+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:45:30.341348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323330+0000","last_change":"2024-06-29T15:39:12.188051+0000","last_active":"2024-06-29T15:39:13.323330+0000","last_peered":"2024-06-29T15:39:13.323330+0000","last_clean":"2024-06-29T15:39:13.323330+0000","last_became_active":"2024-06-29T15:39:12.187588+0000","last_became_peered":"2024-06-29T15:39:12.187588+0000","last_unstale":"2024-06-29T15:39:13.323330+0000","last_undegraded":"2024-06-29T15:39:13.323330+0000","last_fullsized":"2024-06-29T15:39:13.323330+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:12.462070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176074+0000","last_change":"2024-06-29T15:39:12.179779+0000","last_active":"2024-06-29T15:39:13.176074+0000","last_peered":"2024-06-29T15:39:13.176074+0000","last_clean":"2024-06-29T15:39:13.176074+0000","last_became_active":"2024-06-29T15:39:12.176792+0000","last_became_peered":"2024-06-29T15:39:12.176792+0000","last_unstale":"2024-06-29T15:39:13.176074+0000","last_undegraded":"2024-06-29T15:39:13.176074+0000","last_fullsized":"2024-06-29T15:39:13.176074+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:20.498391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323149+0000","last_change":"2024-06-29T15:39:12.175294+0000","last_active":"2024-06-29T15:39:13.323149+0000","last_peered":"2024-06-29T15:39:13.323149+0000","last_clean":"2024-06-29T15:39:13.323149+0000","last_became_active":"2024-06-29T15:39:12.174680+0000","last_became_peered":"2024-06-29T15:39:12.174680+0000","last_unstale":"2024-06-29T15:39:13.323149+0000","last_undegraded":"2024-06-29T15:39:13.323149+0000","last_fullsized":"2024-06-29T15:39:13.323149+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:48.748686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323104+0000","last_change":"2024-06-29T15:39:12.186535+0000","last_active":"2024-06-29T15:39:13.323104+0000","last_peered":"2024-06-29T15:39:13.323104+0000","last_clean":"2024-06-29T15:39:13.323104+0000","last_became_active":"2024-06-29T15:39:12.186228+0000","last_became_peered":"2024-06-29T15:39:12.186228+0000","last_unstale":"2024-06-29T15:39:13.323104+0000","last_undegraded":"2024-06-29T15:39:13.323104+0000","last_fullsized":"2024-06-29T15:39:13.323104+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:48.748686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323082+0000","last_change":"2024-06-29T15:39:12.171265+0000","last_active":"2024-06-29T15:39:13.323082+0000","last_peered":"2024-06-29T15:39:13.323082+0000","last_clean":"2024-06-29T15:39:13.323082+0000","last_became_active":"2024-06-29T15:39:12.170958+0000","last_became_peered":"2024-06-29T15:39:12.170958+0000","last_unstale":"2024-06-29T15:39:13.323082+0000","last_undegraded":"2024-06-29T15:39:13.323082+0000","last_fullsized":"2024-06-29T15:39:13.323082+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:37.338437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323536+0000","last_change":"2024-06-29T15:39:12.177761+0000","last_active":"2024-06-29T15:39:13.323536+0000","last_peered":"2024-06-29T15:39:13.323536+0000","last_clean":"2024-06-29T15:39:13.323536+0000","last_became_active":"2024-06-29T15:39:12.177441+0000","last_became_peered":"2024-06-29T15:39:12.177441+0000","last_unstale":"2024-06-29T15:39:13.323536+0000","last_undegraded":"2024-06-29T15:39:13.323536+0000","last_fullsized":"2024-06-29T15:39:13.323536+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:38.128367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.177012+0000","last_change":"2024-06-29T15:39:12.186256+0000","last_active":"2024-06-29T15:39:13.177012+0000","last_peered":"2024-06-29T15:39:13.177012+0000","last_clean":"2024-06-29T15:39:13.177012+0000","last_became_active":"2024-06-29T15:39:12.185948+0000","last_became_peered":"2024-06-29T15:39:12.185948+0000","last_unstale":"2024-06-29T15:39:13.177012+0000","last_undegraded":"2024-06-29T15:39:13.177012+0000","last_fullsized":"2024-06-29T15:39:13.177012+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:08.650885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322670+0000","last_change":"2024-06-29T15:39:12.185241+0000","last_active":"2024-06-29T15:39:13.322670+0000","last_peered":"2024-06-29T15:39:13.322670+0000","last_clean":"2024-06-29T15:39:13.322670+0000","last_became_active":"2024-06-29T15:39:12.184855+0000","last_became_peered":"2024-06-29T15:39:12.184855+0000","last_unstale":"2024-06-29T15:39:13.322670+0000","last_undegraded":"2024-06-29T15:39:13.322670+0000","last_fullsized":"2024-06-29T15:39:13.322670+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:16:07.772575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.177011+0000","last_change":"2024-06-29T15:39:12.175252+0000","last_active":"2024-06-29T15:39:13.177011+0000","last_peered":"2024-06-29T15:39:13.177011+0000","last_clean":"2024-06-29T15:39:13.177011+0000","last_became_active":"2024-06-29T15:39:12.174927+0000","last_became_peered":"2024-06-29T15:39:12.174927+0000","last_unstale":"2024-06-29T15:39:13.177011+0000","last_undegraded":"2024-06-29T15:39:13.177011+0000","last_fullsized":"2024-06-29T15:39:13.177011+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:59:10.964979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174689+0000","last_change":"2024-06-29T15:39:12.178781+0000","last_active":"2024-06-29T15:39:13.174689+0000","last_peered":"2024-06-29T15:39:13.174689+0000","last_clean":"2024-06-29T15:39:13.174689+0000","last_became_active":"2024-06-29T15:39:12.178470+0000","last_became_peered":"2024-06-29T15:39:12.178470+0000","last_unstale":"2024-06-29T15:39:13.174689+0000","last_undegraded":"2024-06-29T15:39:13.174689+0000","last_fullsized":"2024-06-29T15:39:13.174689+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:59:36.920023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176633+0000","last_change":"2024-06-29T15:39:12.172473+0000","last_active":"2024-06-29T15:39:13.176633+0000","last_peered":"2024-06-29T15:39:13.176633+0000","last_clean":"2024-06-29T15:39:13.176633+0000","last_became_active":"2024-06-29T15:39:12.172041+0000","last_became_peered":"2024-06-29T15:39:12.172041+0000","last_unstale":"2024-06-29T15:39:13.176633+0000","last_undegraded":"2024-06-29T15:39:13.176633+0000","last_fullsized":"2024-06-29T15:39:13.176633+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:14.450584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.175688+0000","last_change":"2024-06-29T15:39:12.183922+0000","last_active":"2024-06-29T15:39:13.175688+0000","last_peered":"2024-06-29T15:39:13.175688+0000","last_clean":"2024-06-29T15:39:13.175688+0000","last_became_active":"2024-06-29T15:39:12.183621+0000","last_became_peered":"2024-06-29T15:39:12.183621+0000","last_unstale":"2024-06-29T15:39:13.175688+0000","last_undegraded":"2024-06-29T15:39:13.175688+0000","last_fullsized":"2024-06-29T15:39:13.175688+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:23:18.506427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176038+0000","last_change":"2024-06-29T15:39:12.188270+0000","last_active":"2024-06-29T15:39:13.176038+0000","last_peered":"2024-06-29T15:39:13.176038+0000","last_clean":"2024-06-29T15:39:13.176038+0000","last_became_active":"2024-06-29T15:39:12.187954+0000","last_became_peered":"2024-06-29T15:39:12.187954+0000","last_unstale":"2024-06-29T15:39:13.176038+0000","last_undegraded":"2024-06-29T15:39:13.176038+0000","last_fullsized":"2024-06-29T15:39:13.176038+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:26:51.077527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.321837+0000","last_change":"2024-06-29T15:39:12.179912+0000","last_active":"2024-06-29T15:39:13.321837+0000","last_peered":"2024-06-29T15:39:13.321837+0000","last_clean":"2024-06-29T15:39:13.321837+0000","last_became_active":"2024-06-29T15:39:12.179562+0000","last_became_peered":"2024-06-29T15:39:12.179562+0000","last_unstale":"2024-06-29T15:39:13.321837+0000","last_undegraded":"2024-06-29T15:39:13.321837+0000","last_fullsized":"2024-06-29T15:39:13.321837+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:06:35.126003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322745+0000","last_change":"2024-06-29T15:39:12.185725+0000","last_active":"2024-06-29T15:39:13.322745+0000","last_peered":"2024-06-29T15:39:13.322745+0000","last_clean":"2024-06-29T15:39:13.322745+0000","last_became_active":"2024-06-29T15:39:12.185357+0000","last_became_peered":"2024-06-29T15:39:12.185357+0000","last_unstale":"2024-06-29T15:39:13.322745+0000","last_undegraded":"2024-06-29T15:39:13.322745+0000","last_fullsized":"2024-06-29T15:39:13.322745+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:23:17.196740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"180'4","reported_seq":18,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322933+0000","last_change":"2024-06-29T15:39:11.176535+0000","last_active":"2024-06-29T15:39:13.322933+0000","last_peered":"2024-06-29T15:39:13.322933+0000","last_clean":"2024-06-29T15:39:13.322933+0000","last_became_active":"2024-06-29T15:39:11.176193+0000","last_became_peered":"2024-06-29T15:39:11.176193+0000","last_unstale":"2024-06-29T15:39:13.322933+0000","last_undegraded":"2024-06-29T15:39:13.322933+0000","last_fullsized":"2024-06-29T15:39:13.322933+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:50.168689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176076+0000","last_change":"2024-06-29T15:39:12.179493+0000","last_active":"2024-06-29T15:39:13.176076+0000","last_peered":"2024-06-29T15:39:13.176076+0000","last_clean":"2024-06-29T15:39:13.176076+0000","last_became_active":"2024-06-29T15:39:12.179087+0000","last_became_peered":"2024-06-29T15:39:12.179087+0000","last_unstale":"2024-06-29T15:39:13.176076+0000","last_undegraded":"2024-06-29T15:39:13.176076+0000","last_fullsized":"2024-06-29T15:39:13.176076+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:18:54.770043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"180'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.175638+0000","last_change":"2024-06-29T15:39:11.169170+0000","last_active":"2024-06-29T15:39:13.175638+0000","last_peered":"2024-06-29T15:39:13.175638+0000","last_clean":"2024-06-29T15:39:13.175638+0000","last_became_active":"2024-06-29T15:39:11.168784+0000","last_became_peered":"2024-06-29T15:39:11.168784+0000","last_unstale":"2024-06-29T15:39:13.175638+0000","last_undegraded":"2024-06-29T15:39:13.175638+0000","last_fullsized":"2024-06-29T15:39:13.175638+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:32:40.672795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323182+0000","last_change":"2024-06-29T15:39:12.179979+0000","last_active":"2024-06-29T15:39:13.323182+0000","last_peered":"2024-06-29T15:39:13.323182+0000","last_clean":"2024-06-29T15:39:13.323182+0000","last_became_active":"2024-06-29T15:39:12.179669+0000","last_became_peered":"2024-06-29T15:39:12.179669+0000","last_unstale":"2024-06-29T15:39:13.323182+0000","last_undegraded":"2024-06-29T15:39:13.323182+0000","last_fullsized":"2024-06-29T15:39:13.323182+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:51.638294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174731+0000","last_change":"2024-06-29T15:39:11.160720+0000","last_active":"2024-06-29T15:39:13.174731+0000","last_peered":"2024-06-29T15:39:13.174731+0000","last_clean":"2024-06-29T15:39:13.174731+0000","last_became_active":"2024-06-29T15:39:11.160361+0000","last_became_peered":"2024-06-29T15:39:11.160361+0000","last_unstale":"2024-06-29T15:39:13.174731+0000","last_undegraded":"2024-06-29T15:39:13.174731+0000","last_fullsized":"2024-06-29T15:39:13.174731+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:47.742779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.177091+0000","last_change":"2024-06-29T15:39:12.183965+0000","last_active":"2024-06-29T15:39:13.177091+0000","last_peered":"2024-06-29T15:39:13.177091+0000","last_clean":"2024-06-29T15:39:13.177091+0000","last_became_active":"2024-06-29T15:39:12.183015+0000","last_became_peered":"2024-06-29T15:39:12.183015+0000","last_unstale":"2024-06-29T15:39:13.177091+0000","last_undegraded":"2024-06-29T15:39:13.177091+0000","last_fullsized":"2024-06-29T15:39:13.177091+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:08.994857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174864+0000","last_change":"2024-06-29T15:39:11.169812+0000","last_active":"2024-06-29T15:39:13.174864+0000","last_peered":"2024-06-29T15:39:13.174864+0000","last_clean":"2024-06-29T15:39:13.174864+0000","last_became_active":"2024-06-29T15:39:11.169501+0000","last_became_peered":"2024-06-29T15:39:11.169501+0000","last_unstale":"2024-06-29T15:39:13.174864+0000","last_undegraded":"2024-06-29T15:39:13.174864+0000","last_fullsized":"2024-06-29T15:39:13.174864+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:19.492485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322366+0000","last_change":"2024-06-29T15:39:12.185582+0000","last_active":"2024-06-29T15:39:13.322366+0000","last_peered":"2024-06-29T15:39:13.322366+0000","last_clean":"2024-06-29T15:39:13.322366+0000","last_became_active":"2024-06-29T15:39:12.185270+0000","last_became_peered":"2024-06-29T15:39:12.185270+0000","last_unstale":"2024-06-29T15:39:13.322366+0000","last_undegraded":"2024-06-29T15:39:13.322366+0000","last_fullsized":"2024-06-29T15:39:13.322366+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:38:14.450584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"180'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176767+0000","last_change":"2024-06-29T15:39:11.171496+0000","last_active":"2024-06-29T15:39:13.176767+0000","last_peered":"2024-06-29T15:39:13.176767+0000","last_clean":"2024-06-29T15:39:13.176767+0000","last_became_active":"2024-06-29T15:39:11.171216+0000","last_became_peered":"2024-06-29T15:39:11.171216+0000","last_unstale":"2024-06-29T15:39:13.176767+0000","last_undegraded":"2024-06-29T15:39:13.176767+0000","last_fullsized":"2024-06-29T15:39:13.176767+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:12:14.900156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174517+0000","last_change":"2024-06-29T15:39:12.186561+0000","last_active":"2024-06-29T15:39:13.174517+0000","last_peered":"2024-06-29T15:39:13.174517+0000","last_clean":"2024-06-29T15:39:13.174517+0000","last_became_active":"2024-06-29T15:39:12.186190+0000","last_became_peered":"2024-06-29T15:39:12.186190+0000","last_unstale":"2024-06-29T15:39:13.174517+0000","last_undegraded":"2024-06-29T15:39:13.174517+0000","last_fullsized":"2024-06-29T15:39:13.174517+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:38:23.275712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"180'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174313+0000","last_change":"2024-06-29T15:39:11.170320+0000","last_active":"2024-06-29T15:39:13.174313+0000","last_peered":"2024-06-29T15:39:13.174313+0000","last_clean":"2024-06-29T15:39:13.174313+0000","last_became_active":"2024-06-29T15:39:11.169334+0000","last_became_peered":"2024-06-29T15:39:11.169334+0000","last_unstale":"2024-06-29T15:39:13.174313+0000","last_undegraded":"2024-06-29T15:39:13.174313+0000","last_fullsized":"2024-06-29T15:39:13.174313+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:38:44.208143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174282+0000","last_change":"2024-06-29T15:39:12.176979+0000","last_active":"2024-06-29T15:39:13.174282+0000","last_peered":"2024-06-29T15:39:13.174282+0000","last_clean":"2024-06-29T15:39:13.174282+0000","last_became_active":"2024-06-29T15:39:12.176669+0000","last_became_peered":"2024-06-29T15:39:12.176669+0000","last_unstale":"2024-06-29T15:39:13.174282+0000","last_undegraded":"2024-06-29T15:39:13.174282+0000","last_fullsized":"2024-06-29T15:39:13.174282+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:10:08.994857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323064+0000","last_change":"2024-06-29T15:39:11.164365+0000","last_active":"2024-06-29T15:39:13.323064+0000","last_peered":"2024-06-29T15:39:13.323064+0000","last_clean":"2024-06-29T15:39:13.323064+0000","last_became_active":"2024-06-29T15:39:11.163313+0000","last_became_peered":"2024-06-29T15:39:11.163313+0000","last_unstale":"2024-06-29T15:39:13.323064+0000","last_undegraded":"2024-06-29T15:39:13.323064+0000","last_fullsized":"2024-06-29T15:39:13.323064+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:53.735143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323143+0000","last_change":"2024-06-29T15:39:12.187203+0000","last_active":"2024-06-29T15:39:13.323143+0000","last_peered":"2024-06-29T15:39:13.323143+0000","last_clean":"2024-06-29T15:39:13.323143+0000","last_became_active":"2024-06-29T15:39:12.186847+0000","last_became_peered":"2024-06-29T15:39:12.186847+0000","last_unstale":"2024-06-29T15:39:13.323143+0000","last_undegraded":"2024-06-29T15:39:13.323143+0000","last_fullsized":"2024-06-29T15:39:13.323143+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:09.931223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.321908+0000","last_change":"2024-06-29T15:39:11.167992+0000","last_active":"2024-06-29T15:39:13.321908+0000","last_peered":"2024-06-29T15:39:13.321908+0000","last_clean":"2024-06-29T15:39:13.321908+0000","last_became_active":"2024-06-29T15:39:11.167642+0000","last_became_peered":"2024-06-29T15:39:11.167642+0000","last_unstale":"2024-06-29T15:39:13.321908+0000","last_undegraded":"2024-06-29T15:39:13.321908+0000","last_fullsized":"2024-06-29T15:39:13.321908+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:53.735143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322796+0000","last_change":"2024-06-29T15:39:12.174189+0000","last_active":"2024-06-29T15:39:13.322796+0000","last_peered":"2024-06-29T15:39:13.322796+0000","last_clean":"2024-06-29T15:39:13.322796+0000","last_became_active":"2024-06-29T15:39:12.173878+0000","last_became_peered":"2024-06-29T15:39:12.173878+0000","last_unstale":"2024-06-29T15:39:13.322796+0000","last_undegraded":"2024-06-29T15:39:13.322796+0000","last_fullsized":"2024-06-29T15:39:13.322796+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:05:13.755084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323339+0000","last_change":"2024-06-29T15:39:11.174542+0000","last_active":"2024-06-29T15:39:13.323339+0000","last_peered":"2024-06-29T15:39:13.323339+0000","last_clean":"2024-06-29T15:39:13.323339+0000","last_became_active":"2024-06-29T15:39:11.174299+0000","last_became_peered":"2024-06-29T15:39:11.174299+0000","last_unstale":"2024-06-29T15:39:13.323339+0000","last_undegraded":"2024-06-29T15:39:13.323339+0000","last_fullsized":"2024-06-29T15:39:13.323339+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:12:14.900156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322479+0000","last_change":"2024-06-29T15:39:12.175133+0000","last_active":"2024-06-29T15:39:13.322479+0000","last_peered":"2024-06-29T15:39:13.322479+0000","last_clean":"2024-06-29T15:39:13.322479+0000","last_became_active":"2024-06-29T15:39:12.174837+0000","last_became_peered":"2024-06-29T15:39:12.174837+0000","last_unstale":"2024-06-29T15:39:13.322479+0000","last_undegraded":"2024-06-29T15:39:13.322479+0000","last_fullsized":"2024-06-29T15:39:13.322479+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:23:58.097609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"180'3","reported_seq":17,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323326+0000","last_change":"2024-06-29T15:39:11.170472+0000","last_active":"2024-06-29T15:39:13.323326+0000","last_peered":"2024-06-29T15:39:13.323326+0000","last_clean":"2024-06-29T15:39:13.323326+0000","last_became_active":"2024-06-29T15:39:11.170146+0000","last_became_peered":"2024-06-29T15:39:11.170146+0000","last_unstale":"2024-06-29T15:39:13.323326+0000","last_undegraded":"2024-06-29T15:39:13.323326+0000","last_fullsized":"2024-06-29T15:39:13.323326+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:53.735143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176349+0000","last_change":"2024-06-29T15:39:12.191198+0000","last_active":"2024-06-29T15:39:13.176349+0000","last_peered":"2024-06-29T15:39:13.176349+0000","last_clean":"2024-06-29T15:39:13.176349+0000","last_became_active":"2024-06-29T15:39:12.190836+0000","last_became_peered":"2024-06-29T15:39:12.190836+0000","last_unstale":"2024-06-29T15:39:13.176349+0000","last_undegraded":"2024-06-29T15:39:13.176349+0000","last_fullsized":"2024-06-29T15:39:13.176349+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T15:48:49.861345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"180'6","reported_seq":20,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.175943+0000","last_change":"2024-06-29T15:39:11.171125+0000","last_active":"2024-06-29T15:39:13.175943+0000","last_peered":"2024-06-29T15:39:13.175943+0000","last_clean":"2024-06-29T15:39:13.175943+0000","last_became_active":"2024-06-29T15:39:11.170808+0000","last_became_peered":"2024-06-29T15:39:11.170808+0000","last_unstale":"2024-06-29T15:39:13.175943+0000","last_undegraded":"2024-06-29T15:39:13.175943+0000","last_fullsized":"2024-06-29T15:39:13.175943+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:36.329410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322140+0000","last_change":"2024-06-29T15:39:12.173531+0000","last_active":"2024-06-29T15:39:13.322140+0000","last_peered":"2024-06-29T15:39:13.322140+0000","last_clean":"2024-06-29T15:39:13.322140+0000","last_became_active":"2024-06-29T15:39:12.173136+0000","last_became_peered":"2024-06-29T15:39:12.173136+0000","last_unstale":"2024-06-29T15:39:13.322140+0000","last_undegraded":"2024-06-29T15:39:13.322140+0000","last_fullsized":"2024-06-29T15:39:13.322140+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:18.660879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174506+0000","last_change":"2024-06-29T15:39:11.168493+0000","last_active":"2024-06-29T15:39:13.174506+0000","last_peered":"2024-06-29T15:39:13.174506+0000","last_clean":"2024-06-29T15:39:13.174506+0000","last_became_active":"2024-06-29T15:39:11.167944+0000","last_became_peered":"2024-06-29T15:39:11.167944+0000","last_unstale":"2024-06-29T15:39:13.174506+0000","last_undegraded":"2024-06-29T15:39:13.174506+0000","last_fullsized":"2024-06-29T15:39:13.174506+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:50.632387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322076+0000","last_change":"2024-06-29T15:39:12.189959+0000","last_active":"2024-06-29T15:39:13.322076+0000","last_peered":"2024-06-29T15:39:13.322076+0000","last_clean":"2024-06-29T15:39:13.322076+0000","last_became_active":"2024-06-29T15:39:12.189684+0000","last_became_peered":"2024-06-29T15:39:12.189684+0000","last_unstale":"2024-06-29T15:39:13.322076+0000","last_undegraded":"2024-06-29T15:39:13.322076+0000","last_fullsized":"2024-06-29T15:39:13.322076+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:16:48.748686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322462+0000","last_change":"2024-06-29T15:39:11.172945+0000","last_active":"2024-06-29T15:39:13.322462+0000","last_peered":"2024-06-29T15:39:13.322462+0000","last_clean":"2024-06-29T15:39:13.322462+0000","last_became_active":"2024-06-29T15:39:11.172668+0000","last_became_peered":"2024-06-29T15:39:11.172668+0000","last_unstale":"2024-06-29T15:39:13.322462+0000","last_undegraded":"2024-06-29T15:39:13.322462+0000","last_fullsized":"2024-06-29T15:39:13.322462+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:51:22.435423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174281+0000","last_change":"2024-06-29T15:39:12.174140+0000","last_active":"2024-06-29T15:39:13.174281+0000","last_peered":"2024-06-29T15:39:13.174281+0000","last_clean":"2024-06-29T15:39:13.174281+0000","last_became_active":"2024-06-29T15:39:12.173860+0000","last_became_peered":"2024-06-29T15:39:12.173860+0000","last_unstale":"2024-06-29T15:39:13.174281+0000","last_undegraded":"2024-06-29T15:39:13.174281+0000","last_fullsized":"2024-06-29T15:39:13.174281+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:21:08.650885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174291+0000","last_change":"2024-06-29T15:39:11.163889+0000","last_active":"2024-06-29T15:39:13.174291+0000","last_peered":"2024-06-29T15:39:13.174291+0000","last_clean":"2024-06-29T15:39:13.174291+0000","last_became_active":"2024-06-29T15:39:11.163491+0000","last_became_peered":"2024-06-29T15:39:11.163491+0000","last_unstale":"2024-06-29T15:39:13.174291+0000","last_undegraded":"2024-06-29T15:39:13.174291+0000","last_fullsized":"2024-06-29T15:39:13.174291+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:08.940401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176640+0000","last_change":"2024-06-29T15:39:12.187444+0000","last_active":"2024-06-29T15:39:13.176640+0000","last_peered":"2024-06-29T15:39:13.176640+0000","last_clean":"2024-06-29T15:39:13.176640+0000","last_became_active":"2024-06-29T15:39:12.186675+0000","last_became_peered":"2024-06-29T15:39:12.186675+0000","last_unstale":"2024-06-29T15:39:13.176640+0000","last_undegraded":"2024-06-29T15:39:13.176640+0000","last_fullsized":"2024-06-29T15:39:13.176640+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:12.462070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"180'8","reported_seq":22,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.175424+0000","last_change":"2024-06-29T15:39:11.176083+0000","last_active":"2024-06-29T15:39:13.175424+0000","last_peered":"2024-06-29T15:39:13.175424+0000","last_clean":"2024-06-29T15:39:13.175424+0000","last_became_active":"2024-06-29T15:39:11.174416+0000","last_became_peered":"2024-06-29T15:39:11.174416+0000","last_unstale":"2024-06-29T15:39:13.175424+0000","last_undegraded":"2024-06-29T15:39:13.175424+0000","last_fullsized":"2024-06-29T15:39:13.175424+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:16:00.110390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174916+0000","last_change":"2024-06-29T15:39:12.173357+0000","last_active":"2024-06-29T15:39:13.174916+0000","last_peered":"2024-06-29T15:39:13.174916+0000","last_clean":"2024-06-29T15:39:13.174916+0000","last_became_active":"2024-06-29T15:39:12.173046+0000","last_became_peered":"2024-06-29T15:39:12.173046+0000","last_unstale":"2024-06-29T15:39:13.174916+0000","last_undegraded":"2024-06-29T15:39:13.174916+0000","last_fullsized":"2024-06-29T15:39:13.174916+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:37.335316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"180'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.177076+0000","last_change":"2024-06-29T15:39:11.166313+0000","last_active":"2024-06-29T15:39:13.177076+0000","last_peered":"2024-06-29T15:39:13.177076+0000","last_clean":"2024-06-29T15:39:13.177076+0000","last_became_active":"2024-06-29T15:39:11.166004+0000","last_became_peered":"2024-06-29T15:39:11.166004+0000","last_unstale":"2024-06-29T15:39:13.177076+0000","last_undegraded":"2024-06-29T15:39:13.177076+0000","last_fullsized":"2024-06-29T15:39:13.177076+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:36:17.739994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322867+0000","last_change":"2024-06-29T15:39:12.183328+0000","last_active":"2024-06-29T15:39:13.322867+0000","last_peered":"2024-06-29T15:39:13.322867+0000","last_clean":"2024-06-29T15:39:13.322867+0000","last_became_active":"2024-06-29T15:39:12.182868+0000","last_became_peered":"2024-06-29T15:39:12.182868+0000","last_unstale":"2024-06-29T15:39:13.322867+0000","last_undegraded":"2024-06-29T15:39:13.322867+0000","last_fullsized":"2024-06-29T15:39:13.322867+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:36.533840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176307+0000","last_change":"2024-06-29T15:39:11.173418+0000","last_active":"2024-06-29T15:39:13.176307+0000","last_peered":"2024-06-29T15:39:13.176307+0000","last_clean":"2024-06-29T15:39:13.176307+0000","last_became_active":"2024-06-29T15:39:11.173055+0000","last_became_peered":"2024-06-29T15:39:11.173055+0000","last_unstale":"2024-06-29T15:39:13.176307+0000","last_undegraded":"2024-06-29T15:39:13.176307+0000","last_fullsized":"2024-06-29T15:39:13.176307+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:23:53.735143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174861+0000","last_change":"2024-06-29T15:39:12.185065+0000","last_active":"2024-06-29T15:39:13.174861+0000","last_peered":"2024-06-29T15:39:13.174861+0000","last_clean":"2024-06-29T15:39:13.174861+0000","last_became_active":"2024-06-29T15:39:12.184743+0000","last_became_peered":"2024-06-29T15:39:12.184743+0000","last_unstale":"2024-06-29T15:39:13.174861+0000","last_undegraded":"2024-06-29T15:39:13.174861+0000","last_fullsized":"2024-06-29T15:39:13.174861+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:09.350727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"180'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322769+0000","last_change":"2024-06-29T15:39:11.175278+0000","last_active":"2024-06-29T15:39:13.322769+0000","last_peered":"2024-06-29T15:39:13.322769+0000","last_clean":"2024-06-29T15:39:13.322769+0000","last_became_active":"2024-06-29T15:39:11.174970+0000","last_became_peered":"2024-06-29T15:39:11.174970+0000","last_unstale":"2024-06-29T15:39:13.322769+0000","last_undegraded":"2024-06-29T15:39:13.322769+0000","last_fullsized":"2024-06-29T15:39:13.322769+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:18:20.423023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322160+0000","last_change":"2024-06-29T15:39:12.189151+0000","last_active":"2024-06-29T15:39:13.322160+0000","last_peered":"2024-06-29T15:39:13.322160+0000","last_clean":"2024-06-29T15:39:13.322160+0000","last_became_active":"2024-06-29T15:39:12.188817+0000","last_became_peered":"2024-06-29T15:39:12.188817+0000","last_unstale":"2024-06-29T15:39:13.322160+0000","last_undegraded":"2024-06-29T15:39:13.322160+0000","last_fullsized":"2024-06-29T15:39:13.322160+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:31:51.638294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323025+0000","last_change":"2024-06-29T15:39:11.176437+0000","last_active":"2024-06-29T15:39:13.323025+0000","last_peered":"2024-06-29T15:39:13.323025+0000","last_clean":"2024-06-29T15:39:13.323025+0000","last_became_active":"2024-06-29T15:39:11.176114+0000","last_became_peered":"2024-06-29T15:39:11.176114+0000","last_unstale":"2024-06-29T15:39:13.323025+0000","last_undegraded":"2024-06-29T15:39:13.323025+0000","last_fullsized":"2024-06-29T15:39:13.323025+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:37:55.570200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176950+0000","last_change":"2024-06-29T15:39:12.178656+0000","last_active":"2024-06-29T15:39:13.176950+0000","last_peered":"2024-06-29T15:39:13.176950+0000","last_clean":"2024-06-29T15:39:13.176950+0000","last_became_active":"2024-06-29T15:39:12.178339+0000","last_became_peered":"2024-06-29T15:39:12.178339+0000","last_unstale":"2024-06-29T15:39:13.176950+0000","last_undegraded":"2024-06-29T15:39:13.176950+0000","last_fullsized":"2024-06-29T15:39:13.176950+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:23:17.196740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174604+0000","last_change":"2024-06-29T15:39:11.159932+0000","last_active":"2024-06-29T15:39:13.174604+0000","last_peered":"2024-06-29T15:39:13.174604+0000","last_clean":"2024-06-29T15:39:13.174604+0000","last_became_active":"2024-06-29T15:39:11.159624+0000","last_became_peered":"2024-06-29T15:39:11.159624+0000","last_unstale":"2024-06-29T15:39:13.174604+0000","last_undegraded":"2024-06-29T15:39:13.174604+0000","last_fullsized":"2024-06-29T15:39:13.174604+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:05.471412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322244+0000","last_change":"2024-06-29T15:39:12.186851+0000","last_active":"2024-06-29T15:39:13.322244+0000","last_peered":"2024-06-29T15:39:13.322244+0000","last_clean":"2024-06-29T15:39:13.322244+0000","last_became_active":"2024-06-29T15:39:12.185795+0000","last_became_peered":"2024-06-29T15:39:12.185795+0000","last_unstale":"2024-06-29T15:39:13.322244+0000","last_undegraded":"2024-06-29T15:39:13.322244+0000","last_fullsized":"2024-06-29T15:39:13.322244+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:20.498391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322550+0000","last_change":"2024-06-29T15:39:11.175248+0000","last_active":"2024-06-29T15:39:13.322550+0000","last_peered":"2024-06-29T15:39:13.322550+0000","last_clean":"2024-06-29T15:39:13.322550+0000","last_became_active":"2024-06-29T15:39:11.174927+0000","last_became_peered":"2024-06-29T15:39:11.174927+0000","last_unstale":"2024-06-29T15:39:13.322550+0000","last_undegraded":"2024-06-29T15:39:13.322550+0000","last_fullsized":"2024-06-29T15:39:13.322550+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:42:00.304411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322425+0000","last_change":"2024-06-29T15:39:12.186070+0000","last_active":"2024-06-29T15:39:13.322425+0000","last_peered":"2024-06-29T15:39:13.322425+0000","last_clean":"2024-06-29T15:39:13.322425+0000","last_became_active":"2024-06-29T15:39:12.185772+0000","last_became_peered":"2024-06-29T15:39:12.185772+0000","last_unstale":"2024-06-29T15:39:13.322425+0000","last_undegraded":"2024-06-29T15:39:13.322425+0000","last_fullsized":"2024-06-29T15:39:13.322425+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:01:21.848026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174310+0000","last_change":"2024-06-29T15:39:11.173238+0000","last_active":"2024-06-29T15:39:13.174310+0000","last_peered":"2024-06-29T15:39:13.174310+0000","last_clean":"2024-06-29T15:39:13.174310+0000","last_became_active":"2024-06-29T15:39:11.172942+0000","last_became_peered":"2024-06-29T15:39:11.172942+0000","last_unstale":"2024-06-29T15:39:13.174310+0000","last_undegraded":"2024-06-29T15:39:13.174310+0000","last_fullsized":"2024-06-29T15:39:13.174310+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:35:36.332531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323271+0000","last_change":"2024-06-29T15:39:12.172033+0000","last_active":"2024-06-29T15:39:13.323271+0000","last_peered":"2024-06-29T15:39:13.323271+0000","last_clean":"2024-06-29T15:39:13.323271+0000","last_became_active":"2024-06-29T15:39:12.171740+0000","last_became_peered":"2024-06-29T15:39:12.171740+0000","last_unstale":"2024-06-29T15:39:13.323271+0000","last_undegraded":"2024-06-29T15:39:13.323271+0000","last_fullsized":"2024-06-29T15:39:13.323271+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:06.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322510+0000","last_change":"2024-06-29T15:39:11.175891+0000","last_active":"2024-06-29T15:39:13.322510+0000","last_peered":"2024-06-29T15:39:13.322510+0000","last_clean":"2024-06-29T15:39:13.322510+0000","last_became_active":"2024-06-29T15:39:11.175508+0000","last_became_peered":"2024-06-29T15:39:11.175508+0000","last_unstale":"2024-06-29T15:39:13.322510+0000","last_undegraded":"2024-06-29T15:39:13.322510+0000","last_fullsized":"2024-06-29T15:39:13.322510+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:06:37.288488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322251+0000","last_change":"2024-06-29T15:39:12.175449+0000","last_active":"2024-06-29T15:39:13.322251+0000","last_peered":"2024-06-29T15:39:13.322251+0000","last_clean":"2024-06-29T15:39:13.322251+0000","last_became_active":"2024-06-29T15:39:12.174840+0000","last_became_peered":"2024-06-29T15:39:12.174840+0000","last_unstale":"2024-06-29T15:39:13.322251+0000","last_undegraded":"2024-06-29T15:39:13.322251+0000","last_fullsized":"2024-06-29T15:39:13.322251+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:20.822596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"180'2","reported_seq":16,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322907+0000","last_change":"2024-06-29T15:39:11.175356+0000","last_active":"2024-06-29T15:39:13.322907+0000","last_peered":"2024-06-29T15:39:13.322907+0000","last_clean":"2024-06-29T15:39:13.322907+0000","last_became_active":"2024-06-29T15:39:11.175073+0000","last_became_peered":"2024-06-29T15:39:11.175073+0000","last_unstale":"2024-06-29T15:39:13.322907+0000","last_undegraded":"2024-06-29T15:39:13.322907+0000","last_fullsized":"2024-06-29T15:39:13.322907+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:26:27.895788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322480+0000","last_change":"2024-06-29T15:39:12.191110+0000","last_active":"2024-06-29T15:39:13.322480+0000","last_peered":"2024-06-29T15:39:13.322480+0000","last_clean":"2024-06-29T15:39:13.322480+0000","last_became_active":"2024-06-29T15:39:12.190799+0000","last_became_peered":"2024-06-29T15:39:12.190799+0000","last_unstale":"2024-06-29T15:39:13.322480+0000","last_undegraded":"2024-06-29T15:39:13.322480+0000","last_fullsized":"2024-06-29T15:39:13.322480+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:12:12.462070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"181'1","reported_seq":15,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.188696+0000","last_change":"2024-06-29T15:39:11.171344+0000","last_active":"2024-06-29T15:39:13.188696+0000","last_peered":"2024-06-29T15:39:13.188696+0000","last_clean":"2024-06-29T15:39:13.188696+0000","last_became_active":"2024-06-29T15:39:11.171015+0000","last_became_peered":"2024-06-29T15:39:11.171015+0000","last_unstale":"2024-06-29T15:39:13.188696+0000","last_undegraded":"2024-06-29T15:39:13.188696+0000","last_fullsized":"2024-06-29T15:39:13.188696+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:27.438455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323198+0000","last_change":"2024-06-29T15:39:12.172568+0000","last_active":"2024-06-29T15:39:13.323198+0000","last_peered":"2024-06-29T15:39:13.323198+0000","last_clean":"2024-06-29T15:39:13.323198+0000","last_became_active":"2024-06-29T15:39:12.172262+0000","last_became_peered":"2024-06-29T15:39:12.172262+0000","last_unstale":"2024-06-29T15:39:13.323198+0000","last_undegraded":"2024-06-29T15:39:13.323198+0000","last_fullsized":"2024-06-29T15:39:13.323198+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:24:20.498391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323472+0000","last_change":"2024-06-29T15:39:11.170181+0000","last_active":"2024-06-29T15:39:13.323472+0000","last_peered":"2024-06-29T15:39:13.323472+0000","last_clean":"2024-06-29T15:39:13.323472+0000","last_became_active":"2024-06-29T15:39:11.169824+0000","last_became_peered":"2024-06-29T15:39:11.169824+0000","last_unstale":"2024-06-29T15:39:13.323472+0000","last_undegraded":"2024-06-29T15:39:13.323472+0000","last_fullsized":"2024-06-29T15:39:13.323472+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:12:14.900156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322210+0000","last_change":"2024-06-29T15:39:12.180757+0000","last_active":"2024-06-29T15:39:13.322210+0000","last_peered":"2024-06-29T15:39:13.322210+0000","last_clean":"2024-06-29T15:39:13.322210+0000","last_became_active":"2024-06-29T15:39:12.180409+0000","last_became_peered":"2024-06-29T15:39:12.180409+0000","last_unstale":"2024-06-29T15:39:13.322210+0000","last_undegraded":"2024-06-29T15:39:13.322210+0000","last_fullsized":"2024-06-29T15:39:13.322210+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:54:44.310682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322586+0000","last_change":"2024-06-29T15:39:11.173447+0000","last_active":"2024-06-29T15:39:13.322586+0000","last_peered":"2024-06-29T15:39:13.322586+0000","last_clean":"2024-06-29T15:39:13.322586+0000","last_became_active":"2024-06-29T15:39:11.173132+0000","last_became_peered":"2024-06-29T15:39:11.173132+0000","last_unstale":"2024-06-29T15:39:13.322586+0000","last_undegraded":"2024-06-29T15:39:13.322586+0000","last_fullsized":"2024-06-29T15:39:13.322586+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:15:17.041258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322375+0000","last_change":"2024-06-29T15:39:12.180363+0000","last_active":"2024-06-29T15:39:13.322375+0000","last_peered":"2024-06-29T15:39:13.322375+0000","last_clean":"2024-06-29T15:39:13.322375+0000","last_became_active":"2024-06-29T15:39:12.179785+0000","last_became_peered":"2024-06-29T15:39:12.179785+0000","last_unstale":"2024-06-29T15:39:13.322375+0000","last_undegraded":"2024-06-29T15:39:13.322375+0000","last_fullsized":"2024-06-29T15:39:13.322375+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:34:58.673623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176394+0000","last_change":"2024-06-29T15:39:11.172515+0000","last_active":"2024-06-29T15:39:13.176394+0000","last_peered":"2024-06-29T15:39:13.176394+0000","last_clean":"2024-06-29T15:39:13.176394+0000","last_became_active":"2024-06-29T15:39:11.172186+0000","last_became_peered":"2024-06-29T15:39:11.172186+0000","last_unstale":"2024-06-29T15:39:13.176394+0000","last_undegraded":"2024-06-29T15:39:13.176394+0000","last_fullsized":"2024-06-29T15:39:13.176394+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:08.344821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176231+0000","last_change":"2024-06-29T15:39:12.191410+0000","last_active":"2024-06-29T15:39:13.176231+0000","last_peered":"2024-06-29T15:39:13.176231+0000","last_clean":"2024-06-29T15:39:13.176231+0000","last_became_active":"2024-06-29T15:39:12.188271+0000","last_became_peered":"2024-06-29T15:39:12.188271+0000","last_unstale":"2024-06-29T15:39:13.176231+0000","last_undegraded":"2024-06-29T15:39:13.176231+0000","last_fullsized":"2024-06-29T15:39:13.176231+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:19:44.697460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.174403+0000","last_change":"2024-06-29T15:39:11.171394+0000","last_active":"2024-06-29T15:39:13.174403+0000","last_peered":"2024-06-29T15:39:13.174403+0000","last_clean":"2024-06-29T15:39:13.174403+0000","last_became_active":"2024-06-29T15:39:11.171075+0000","last_became_peered":"2024-06-29T15:39:11.171075+0000","last_unstale":"2024-06-29T15:39:13.174403+0000","last_undegraded":"2024-06-29T15:39:13.174403+0000","last_fullsized":"2024-06-29T15:39:13.174403+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:08.925317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322349+0000","last_change":"2024-06-29T15:39:12.187396+0000","last_active":"2024-06-29T15:39:13.322349+0000","last_peered":"2024-06-29T15:39:13.322349+0000","last_clean":"2024-06-29T15:39:13.322349+0000","last_became_active":"2024-06-29T15:39:12.187082+0000","last_became_peered":"2024-06-29T15:39:12.187082+0000","last_unstale":"2024-06-29T15:39:13.322349+0000","last_undegraded":"2024-06-29T15:39:13.322349+0000","last_fullsized":"2024-06-29T15:39:13.322349+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:06.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.323304+0000","last_change":"2024-06-29T15:39:11.168054+0000","last_active":"2024-06-29T15:39:13.323304+0000","last_peered":"2024-06-29T15:39:13.323304+0000","last_clean":"2024-06-29T15:39:13.323304+0000","last_became_active":"2024-06-29T15:39:11.167717+0000","last_became_peered":"2024-06-29T15:39:11.167717+0000","last_unstale":"2024-06-29T15:39:13.323304+0000","last_undegraded":"2024-06-29T15:39:13.323304+0000","last_fullsized":"2024-06-29T15:39:13.323304+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:27.438455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176965+0000","last_change":"2024-06-29T15:39:12.182211+0000","last_active":"2024-06-29T15:39:13.176965+0000","last_peered":"2024-06-29T15:39:13.176965+0000","last_clean":"2024-06-29T15:39:13.176965+0000","last_became_active":"2024-06-29T15:39:12.181892+0000","last_became_peered":"2024-06-29T15:39:12.181892+0000","last_unstale":"2024-06-29T15:39:13.176965+0000","last_undegraded":"2024-06-29T15:39:13.176965+0000","last_fullsized":"2024-06-29T15:39:13.176965+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:07.625273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.175440+0000","last_change":"2024-06-29T15:39:11.170040+0000","last_active":"2024-06-29T15:39:13.175440+0000","last_peered":"2024-06-29T15:39:13.175440+0000","last_clean":"2024-06-29T15:39:13.175440+0000","last_became_active":"2024-06-29T15:39:11.169708+0000","last_became_peered":"2024-06-29T15:39:11.169708+0000","last_unstale":"2024-06-29T15:39:13.175440+0000","last_undegraded":"2024-06-29T15:39:13.175440+0000","last_fullsized":"2024-06-29T15:39:13.175440+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:59:35.914117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":12,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322085+0000","last_change":"2024-06-29T15:39:12.176363+0000","last_active":"2024-06-29T15:39:13.322085+0000","last_peered":"2024-06-29T15:39:13.322085+0000","last_clean":"2024-06-29T15:39:13.322085+0000","last_became_active":"2024-06-29T15:39:12.175188+0000","last_became_peered":"2024-06-29T15:39:12.175188+0000","last_unstale":"2024-06-29T15:39:13.322085+0000","last_undegraded":"2024-06-29T15:39:13.322085+0000","last_fullsized":"2024-06-29T15:39:13.322085+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":179,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:11.147385+0000","last_clean_scrub_stamp":"2024-06-29T15:39:11.147385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:25.393589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":14,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.176202+0000","last_change":"2024-06-29T15:39:11.174722+0000","last_active":"2024-06-29T15:39:13.176202+0000","last_peered":"2024-06-29T15:39:13.176202+0000","last_clean":"2024-06-29T15:39:13.176202+0000","last_became_active":"2024-06-29T15:39:11.174385+0000","last_became_peered":"2024-06-29T15:39:11.174385+0000","last_unstale":"2024-06-29T15:39:13.176202+0000","last_undegraded":"2024-06-29T15:39:13.176202+0000","last_fullsized":"2024-06-29T15:39:13.176202+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:39:10.141478+0000","last_clean_scrub_stamp":"2024-06-29T15:39:10.141478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:53.626387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"170'184","reported_seq":847,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-06-29T15:39:13.322504+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:39:13.322504+0000","last_peered":"2024-06-29T15:39:13.322504+0000","last_clean":"2024-06-29T15:39:13.322504+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:39:13.322504+0000","last_undegraded":"2024-06-29T15:39:13.322504+0000","last_fullsized":"2024-06-29T15:39:13.322504+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":293,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1026,"num_read_kb":2871,"num_write":293,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575769,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47872,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323968,"statfs":{"total":96636764160,"available":96587743232,"internally_reserved":0,"allocated":2871296,"data_stored":1686376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575770,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41704,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330136,"statfs":{"total":96636764160,"available":96594059264,"internally_reserved":0,"allocated":2256896,"data_stored":1093676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575768,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41724,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330116,"statfs":{"total":96636764160,"available":96594038784,"internally_reserved":0,"allocated":2273280,"data_stored":1100658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575769,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55620,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316220,"statfs":{"total":96636764160,"available":96579809280,"internally_reserved":0,"allocated":2281472,"data_stored":1101736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59752,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312088,"statfs":{"total":96636764160,"available":96575578112,"internally_reserved":0,"allocated":2256896,"data_stored":1093688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575769,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63912,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307928,"statfs":{"total":96636764160,"available":96571318272,"internally_reserved":0,"allocated":2256896,"data_stored":1093744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575768,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59752,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312088,"statfs":{"total":96636764160,"available":96575578112,"internally_reserved":0,"allocated":2256896,"data_stored":1093688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575769,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56180,"kb_used_data":2792,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315660,"statfs":{"total":96636764160,"available":96579235840,"internally_reserved":0,"allocated":2859008,"data_stored":1690422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:39:16.579 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:39:16.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:39:16.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.734+0000 7ff074622640 1 -- 172.21.15.81:0/215085793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff06c06f6a0 msgr2=0x7ff06c15aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.734+0000 7ff074622640 1 --2- 172.21.15.81:0/215085793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff06c06f6a0 0x7ff06c15aed0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7ff05c005fd0 tx=0x7ff05c02a530 comp rx=0 tx=0).stop 2024-06-29T15:39:16.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.735+0000 7ff074622640 1 -- 172.21.15.81:0/215085793 shutdown_connections 2024-06-29T15:39:16.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.735+0000 7ff074622640 1 --2- 172.21.15.81:0/215085793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff06c0b3960 0x7ff06c0b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.735+0000 7ff074622640 1 --2- 172.21.15.81:0/215085793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff06c06f6a0 0x7ff06c15aed0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.735+0000 7ff074622640 1 -- 172.21.15.81:0/215085793 >> 172.21.15.81:0/215085793 conn(0x7ff06c0d0930 msgr2=0x7ff06c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.735+0000 7ff074622640 1 -- 172.21.15.81:0/215085793 shutdown_connections 2024-06-29T15:39:16.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.735+0000 7ff074622640 1 -- 172.21.15.81:0/215085793 wait complete. 2024-06-29T15:39:16.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.736+0000 7ff074622640 1 Processor -- start 2024-06-29T15:39:16.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.736+0000 7ff074622640 1 -- start start 2024-06-29T15:39:16.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff074622640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff06c06f6a0 0x7ff06c152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff074622640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff06c0b3960 0x7ff06c0aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff074622640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff06c0b57b0 0x7ff06c0aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff074622640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c0b4430 con 0x7ff06c0b3960 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff074622640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c0b42b0 con 0x7ff06c0b57b0 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff074622640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff06c0b45b0 con 0x7ff06c06f6a0 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff071bbd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff06c0b3960 0x7ff06c0aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff071bbd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff06c0b3960 0x7ff06c0aea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56252/0 (socket says 172.21.15.81:56252) 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff071bbd640 1 -- 172.21.15.81:0/732001399 learned_addr learned my addr 172.21.15.81:0/732001399 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff0723be640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff06c06f6a0 0x7ff06c152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff072bbf640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff06c0b57b0 0x7ff06c0aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff071bbd640 1 -- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff06c06f6a0 msgr2=0x7ff06c152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff071bbd640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff06c06f6a0 0x7ff06c152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff071bbd640 1 -- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff06c0b57b0 msgr2=0x7ff06c0aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff071bbd640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff06c0b57b0 0x7ff06c0aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff071bbd640 1 -- 172.21.15.81:0/732001399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06c153bc0 con 0x7ff06c0b3960 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff0723be640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff06c06f6a0 0x7ff06c152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:16.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff072bbf640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff06c0b57b0 0x7ff06c0aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff071bbd640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff06c0b3960 0x7ff06c0aea00 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7ff05c034040 tx=0x7ff05c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff062ffd640 1 -- 172.21.15.81:0/732001399 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c03e030 con 0x7ff06c0b3960 2024-06-29T15:39:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff062ffd640 1 -- 172.21.15.81:0/732001399 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff05c003d90 con 0x7ff06c0b3960 2024-06-29T15:39:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff074622640 1 -- 172.21.15.81:0/732001399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff06c153ea0 con 0x7ff06c0b3960 2024-06-29T15:39:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff062ffd640 1 -- 172.21.15.81:0/732001399 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c035040 con 0x7ff06c0b3960 2024-06-29T15:39:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.738+0000 7ff074622640 1 -- 172.21.15.81:0/732001399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff06c154460 con 0x7ff06c0b3960 2024-06-29T15:39:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.740+0000 7ff074622640 1 -- 172.21.15.81:0/732001399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff06c153930 con 0x7ff06c0b3960 2024-06-29T15:39:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.740+0000 7ff062ffd640 1 -- 172.21.15.81:0/732001399 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7ff05c041030 con 0x7ff06c0b3960 2024-06-29T15:39:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.741+0000 7ff062ffd640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff0500763f0 0x7ff050078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.741+0000 7ff062ffd640 1 -- 172.21.15.81:0/732001399 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7ff05c0b8220 con 0x7ff06c0b3960 2024-06-29T15:39:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.741+0000 7ff0723be640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff0500763f0 0x7ff050078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.742+0000 7ff0723be640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff0500763f0 0x7ff050078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff068006fd0 tx=0x7ff068008040 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.745+0000 7ff062ffd640 1 -- 172.21.15.81:0/732001399 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff05c0818f0 con 0x7ff06c0b3960 2024-06-29T15:39:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.992+0000 7ff074622640 1 -- 172.21.15.81:0/732001399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ff06c06e2a0 con 0x7ff06c0b3960 2024-06-29T15:39:16.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.995+0000 7ff062ffd640 1 -- 172.21.15.81:0/732001399 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v59) ==== 190+0+147 (secure 0 0 0) 0x7ff06c06e2a0 con 0x7ff06c0b3960 2024-06-29T15:39:16.996 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T15:39:16.996 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T15:39:16.996 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T15:39:16.997 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T15:39:16.997 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:39:16.997 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T15:39:16.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 -- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff0500763f0 msgr2=0x7ff050078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff0500763f0 0x7ff050078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff068006fd0 tx=0x7ff068008040 comp rx=0 tx=0).stop 2024-06-29T15:39:16.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 -- 172.21.15.81:0/732001399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff06c0b3960 msgr2=0x7ff06c0aea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:16.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff06c0b3960 0x7ff06c0aea00 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7ff05c034040 tx=0x7ff05c0038a0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 -- 172.21.15.81:0/732001399 shutdown_connections 2024-06-29T15:39:16.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff0500763f0 0x7ff050078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff06c0b57b0 0x7ff06c0aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff06c0b3960 0x7ff06c0aea00 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 --2- 172.21.15.81:0/732001399 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff06c06f6a0 0x7ff06c152ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:16.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 -- 172.21.15.81:0/732001399 >> 172.21.15.81:0/732001399 conn(0x7ff06c0d0930 msgr2=0x7ff06c1583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:16.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 -- 172.21.15.81:0/732001399 shutdown_connections 2024-06-29T15:39:16.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:16.997+0000 7ff074622640 1 -- 172.21.15.81:0/732001399 wait complete. 2024-06-29T15:39:17.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:39:17.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.122+0000 7f3aa04ac640 1 -- 172.21.15.81:0/3844696993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a980b6eb0 msgr2=0x7f3a980d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.122+0000 7f3aa04ac640 1 --2- 172.21.15.81:0/3844696993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a980b6eb0 0x7f3a980d02d0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f3a98073550 tx=0x7f3a9402a4d0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.122+0000 7f3aa04ac640 1 -- 172.21.15.81:0/3844696993 shutdown_connections 2024-06-29T15:39:17.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.122+0000 7f3aa04ac640 1 --2- 172.21.15.81:0/3844696993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a980b4810 0x7f3a980b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.122+0000 7f3aa04ac640 1 --2- 172.21.15.81:0/3844696993 >> v2:172.21.15.167:3300/0 conn(0x7f3a980ce510 0x7f3a980ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.122+0000 7f3aa04ac640 1 --2- 172.21.15.81:0/3844696993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a980b6eb0 0x7f3a980d02d0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.122+0000 7f3aa04ac640 1 -- 172.21.15.81:0/3844696993 >> 172.21.15.81:0/3844696993 conn(0x7f3a980b3510 msgr2=0x7f3a980b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.122+0000 7f3aa04ac640 1 -- 172.21.15.81:0/3844696993 shutdown_connections 2024-06-29T15:39:17.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.122+0000 7f3aa04ac640 1 -- 172.21.15.81:0/3844696993 wait complete. 2024-06-29T15:39:17.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.124+0000 7f3aa04ac640 1 Processor -- start 2024-06-29T15:39:17.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.124+0000 7f3aa04ac640 1 -- start start 2024-06-29T15:39:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3aa04ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a980b4810 0x7f3a981df390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3aa04ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a980b6eb0 0x7f3a980b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3aa04ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a980ce510 0x7f3a980b5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3aa04ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3a980b52e0 con 0x7f3a980b6eb0 2024-06-29T15:39:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3aa04ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3a980b5160 con 0x7f3a980ce510 2024-06-29T15:39:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3aa04ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3a980cc470 con 0x7f3a980b4810 2024-06-29T15:39:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3a9da47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a980b6eb0 0x7f3a980b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3a9da47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a980b6eb0 0x7f3a980b1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56268/0 (socket says 172.21.15.81:56268) 2024-06-29T15:39:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3a9da47640 1 -- 172.21.15.81:0/200140822 learned_addr learned my addr 172.21.15.81:0/200140822 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3a9e248640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a980b4810 0x7f3a981df390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3a9da47640 1 -- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a980b4810 msgr2=0x7f3a981df390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3a9da47640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a980b4810 0x7f3a981df390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3a9da47640 1 -- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a980ce510 msgr2=0x7f3a980b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3a9ea49640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a980ce510 0x7f3a980b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3a9da47640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a980ce510 0x7f3a980b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.125+0000 7f3a9da47640 1 -- 172.21.15.81:0/200140822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a980c6860 con 0x7f3a980b6eb0 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.126+0000 7f3a9da47640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a980b6eb0 0x7f3a980b1190 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800d770 tx=0x7f3a8800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.126+0000 7f3a8effd640 1 -- 172.21.15.81:0/200140822 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88017070 con 0x7f3a980b6eb0 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.126+0000 7f3a8effd640 1 -- 172.21.15.81:0/200140822 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a88009d20 con 0x7f3a980b6eb0 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.126+0000 7f3aa04ac640 1 -- 172.21.15.81:0/200140822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a980c6b40 con 0x7f3a980b6eb0 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.126+0000 7f3a8effd640 1 -- 172.21.15.81:0/200140822 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a880106c0 con 0x7f3a980b6eb0 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.126+0000 7f3aa04ac640 1 -- 172.21.15.81:0/200140822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a980c7100 con 0x7f3a980b6eb0 2024-06-29T15:39:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.127+0000 7f3aa04ac640 1 -- 172.21.15.81:0/200140822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a980d02d0 con 0x7f3a980b6eb0 2024-06-29T15:39:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.128+0000 7f3a8effd640 1 -- 172.21.15.81:0/200140822 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f3a8801a070 con 0x7f3a980b6eb0 2024-06-29T15:39:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.128+0000 7f3a8effd640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3a7c0763f0 0x7f3a7c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.128+0000 7f3a8effd640 1 -- 172.21.15.81:0/200140822 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f3a88098080 con 0x7f3a980b6eb0 2024-06-29T15:39:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.128+0000 7f3a9e248640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3a7c0763f0 0x7f3a7c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.129+0000 7f3a9e248640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3a7c0763f0 0x7f3a7c078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3a94033000 tx=0x7f3a94004730 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.132+0000 7f3a8effd640 1 -- 172.21.15.81:0/200140822 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3a88061750 con 0x7f3a980b6eb0 2024-06-29T15:39:17.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.352+0000 7f3aa04ac640 1 -- 172.21.15.81:0/200140822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a980cdf00 con 0x7f3a980b6eb0 2024-06-29T15:39:17.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.353+0000 7f3a8effd640 1 -- 172.21.15.81:0/200140822 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+5237 (secure 0 0 0) 0x7f3a88061050 con 0x7f3a980b6eb0 2024-06-29T15:39:17.354 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:39:17.354 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":95,"btime":"2024-06-29T15:39:15:192034+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2580108037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2580108037},{"type":"v1","addr":"172.21.15.167:6836","nonce":2580108037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":10155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3892298237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3892298237},{"type":"v1","addr":"172.21.15.81:6834","nonce":3892298237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":10158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2457574359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2457574359},{"type":"v1","addr":"172.21.15.81:6835","nonce":2457574359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.526747+0000","modified":"2024-06-29T15:39:14.562127+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10170},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10170":{"gid":10170,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/446857148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":446857148},{"type":"v1","addr":"172.21.15.167:6837","nonce":446857148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10170,"qdb_cluster":[10170]},"id":11}]} 2024-06-29T15:39:17.355 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 95 2024-06-29T15:39:17.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 -- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3a7c0763f0 msgr2=0x7f3a7c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3a7c0763f0 0x7f3a7c078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3a94033000 tx=0x7f3a94004730 comp rx=0 tx=0).stop 2024-06-29T15:39:17.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 -- 172.21.15.81:0/200140822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a980b6eb0 msgr2=0x7f3a980b1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a980b6eb0 0x7f3a980b1190 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800d770 tx=0x7f3a8800dc30 comp rx=0 tx=0).stop 2024-06-29T15:39:17.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 -- 172.21.15.81:0/200140822 shutdown_connections 2024-06-29T15:39:17.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f3a7c0763f0 0x7f3a7c078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a980ce510 0x7f3a980b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a980b6eb0 0x7f3a980b1190 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 --2- 172.21.15.81:0/200140822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a980b4810 0x7f3a981df390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 -- 172.21.15.81:0/200140822 >> 172.21.15.81:0/200140822 conn(0x7f3a980b3510 msgr2=0x7f3a980ce920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 -- 172.21.15.81:0/200140822 shutdown_connections 2024-06-29T15:39:17.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.355+0000 7f3aa04ac640 1 -- 172.21.15.81:0/200140822 wait complete. 2024-06-29T15:39:17.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:39:11.526747+0000', 'modified': '2024-06-29T15:39:14.562127+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10170}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10170': {'gid': 10170, 'name': 'b', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.167:6837/446857148', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6835', 'nonce': 446857148}, {'type': 'v1', 'addr': '172.21.15.167:6837', 'nonce': 446857148}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10170, 'qdb_cluster': [10170]} 2024-06-29T15:39:17.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:39:17.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-06-29T15:39:17.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.885+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4154764473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e1c0b7290 msgr2=0x7f7e1c0b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.885+0000 7f7e23a4e640 1 --2- 172.21.15.81:0/4154764473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e1c0b7290 0x7f7e1c0b76a0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0b9b80 tx=0x7f7e1402a5f0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.885+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4154764473 shutdown_connections 2024-06-29T15:39:17.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.885+0000 7f7e23a4e640 1 --2- 172.21.15.81:0/4154764473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e1c0b7290 0x7f7e1c0b76a0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.885+0000 7f7e23a4e640 1 --2- 172.21.15.81:0/4154764473 >> v2:172.21.15.167:3300/0 conn(0x7f7e1c0b5250 0x7f7e1c0d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.885+0000 7f7e23a4e640 1 --2- 172.21.15.81:0/4154764473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e1c0b9040 0x7f7e1c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.885+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4154764473 >> 172.21.15.81:0/4154764473 conn(0x7f7e1c09cf30 msgr2=0x7f7e1c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.885+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4154764473 shutdown_connections 2024-06-29T15:39:17.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.885+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4154764473 wait complete. 2024-06-29T15:39:17.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.887+0000 7f7e23a4e640 1 Processor -- start 2024-06-29T15:39:17.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.887+0000 7f7e23a4e640 1 -- start start 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.888+0000 7f7e23a4e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e1c0b5250 0x7f7e1c0b1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.888+0000 7f7e23a4e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0b7290 0x7f7e1c153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.888+0000 7f7e23a4e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e1c0b9040 0x7f7e1c0cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.888+0000 7f7e23a4e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c0b7ff0 con 0x7f7e1c0b5250 2024-06-29T15:39:17.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e23a4e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c0b7e70 con 0x7f7e1c0b7290 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e23a4e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7e1c0b8170 con 0x7f7e1c0b9040 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e217ea640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e1c0b5250 0x7f7e1c0b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e217ea640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e1c0b5250 0x7f7e1c0b1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56286/0 (socket says 172.21.15.81:56286) 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e217ea640 1 -- 172.21.15.81:0/4292447153 learned_addr learned my addr 172.21.15.81:0/4292447153 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e21feb640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e1c0b9040 0x7f7e1c0cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0b7290 0x7f7e1c153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e217ea640 1 -- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e1c0b9040 msgr2=0x7f7e1c0cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e217ea640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e1c0b9040 0x7f7e1c0cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e217ea640 1 -- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0b7290 msgr2=0x7f7e1c153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e217ea640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0b7290 0x7f7e1c153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e217ea640 1 -- 172.21.15.81:0/4292447153 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c0c9160 con 0x7f7e1c0b5250 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e217ea640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e1c0b5250 0x7f7e1c0b1700 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800cd10 tx=0x7f7e18007610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e11ffb640 1 -- 172.21.15.81:0/4292447153 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18004b20 con 0x7f7e1c0b5250 2024-06-29T15:39:17.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e11ffb640 1 -- 172.21.15.81:0/4292447153 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e18004db0 con 0x7f7e1c0b5250 2024-06-29T15:39:17.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4292447153 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1c0c9440 con 0x7f7e1c0b5250 2024-06-29T15:39:17.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e11ffb640 1 -- 172.21.15.81:0/4292447153 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18005280 con 0x7f7e1c0b5250 2024-06-29T15:39:17.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.889+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4292447153 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e1c0c9a00 con 0x7f7e1c0b5250 2024-06-29T15:39:17.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.891+0000 7f7e11ffb640 1 -- 172.21.15.81:0/4292447153 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f7e18005420 con 0x7f7e1c0b5250 2024-06-29T15:39:17.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.891+0000 7f7e11ffb640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7e00076420 0x7f7e00078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.892+0000 7f7e11ffb640 1 -- 172.21.15.81:0/4292447153 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f7e180980f0 con 0x7f7e1c0b5250 2024-06-29T15:39:17.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.892+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7e00076420 0x7f7e00078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.892+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7e00076420 0x7f7e00078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c005fd0 tx=0x7f7e0c005da0 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.893+0000 7f7e2a9ee740 1 Processor -- start 2024-06-29T15:39:17.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.893+0000 7f7e2a9ee740 1 -- start start 2024-06-29T15:39:17.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030fb4270 0x5650313a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030f8b3b0 0x5650313843a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 0x5650313848e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56503131cbc0 con 0x565030fb4270 2024-06-29T15:39:17.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x5650312cb630 con 0x565030f8b3b0 2024-06-29T15:39:17.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x5650312babd0 con 0x565030f93590 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e217ea640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030fb4270 0x5650313a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e217ea640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030fb4270 0x5650313a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56298/0 (socket says 172.21.15.81:56298) 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e217ea640 1 -- 172.21.15.81:0/3419958620 learned_addr learned my addr 172.21.15.81:0/3419958620 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/3419958620 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030f8b3b0 0x5650313843a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e21feb640 1 --2- 172.21.15.81:0/3419958620 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 0x5650313848e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e217ea640 1 -- 172.21.15.81:0/3419958620 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 msgr2=0x5650313848e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e217ea640 1 --2- 172.21.15.81:0/3419958620 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 0x5650313848e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e217ea640 1 -- 172.21.15.81:0/3419958620 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030f8b3b0 msgr2=0x5650313843a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e217ea640 1 --2- 172.21.15.81:0/3419958620 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030f8b3b0 0x5650313843a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e217ea640 1 -- 172.21.15.81:0/3419958620 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650312b7ca0 con 0x565030fb4270 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.894+0000 7f7e217ea640 1 --2- 172.21.15.81:0/3419958620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030fb4270 0x5650313a06e0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f7e1805d1d0 tx=0x7f7e18060c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8367339ecb4159a5 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e06ffd640 1 -- 172.21.15.81:0/3419958620 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1801d040 con 0x565030fb4270 2024-06-29T15:39:17.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e06ffd640 1 -- 172.21.15.81:0/3419958620 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e18065500 con 0x565030fb4270 2024-06-29T15:39:17.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e06ffd640 1 -- 172.21.15.81:0/3419958620 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18069f20 con 0x565030fb4270 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/3419958620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030fb4270 msgr2=0x5650313a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/3419958620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030fb4270 0x5650313a06e0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f7e1805d1d0 tx=0x7f7e18060c60 comp rx=0 tx=0).stop 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/3419958620 shutdown_connections 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/3419958620 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 0x5650313848e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/3419958620 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030f8b3b0 0x5650313843a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/3419958620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030fb4270 0x5650313a06e0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/3419958620 >> 172.21.15.81:0/3419958620 conn(0x565030fa83c0 msgr2=0x565031357f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/3419958620 shutdown_connections 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.895+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/3419958620 wait complete. 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.896+0000 7f7e2a9ee740 1 -- start start 2024-06-29T15:39:17.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.896+0000 7f7e2a9ee740 1 Processor -- start 2024-06-29T15:39:17.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030f8b3b0 0x56503137e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313b2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030fb4270 0x5650313b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x565031258ae0 con 0x565030f93590 2024-06-29T15:39:17.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x5650312bec10 con 0x565030f8b3b0 2024-06-29T15:39:17.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x56503102e620 con 0x565030fb4270 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e2a9ee740 10 client.0 _ll_register_callbacks cb 0x565031265480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e20fe9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e20fe9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313b2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56314/0 (socket says 172.21.15.81:56314) 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e20fe9640 1 -- 172.21.15.81:0/2010817614 learned_addr learned my addr 172.21.15.81:0/2010817614 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e217ea640 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030f8b3b0 0x56503137e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e21feb640 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030fb4270 0x5650313b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e20fe9640 1 -- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030fb4270 msgr2=0x5650313b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030fb4270 0x5650313b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e20fe9640 1 -- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030f8b3b0 msgr2=0x56503137e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030f8b3b0 0x56503137e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.898+0000 7f7e20fe9640 1 -- 172.21.15.81:0/2010817614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56503101f850 con 0x565030f93590 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313b2080 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c08c480 tx=0x7f7e0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e02f60d30eced5 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7dfbfff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2010817614 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c01dd50 con 0x565030f93590 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2010817614 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f7e0c01e430 con 0x565030f93590 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2010817614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e18065500 con 0x565030f93590 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2010817614 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f7e0c030050 con 0x565030f93590 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e2a9ee740 10 client.10302 fetch_fsmap learned FSMap version 95 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2010817614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) -- 0x5650310966b0 con 0x565030f93590 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2010817614 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e18065500 con 0x565030f93590 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2010817614 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 95) ==== 3502+0+0 (secure 0 0 0) 0x7f7e0c033020 con 0x565030f93590 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e2a9ee740 10 client.10302 fetch_fsmap finished waiting for FSMap version 95 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e2a9ee740 10 client.10302 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e2a9ee740 20 client.10302 populate_metadata read hostname 'smithi081' 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:6835/446857148,v1:172.21.15.167:6837/446857148] conn(0x565030f72e40 0x565031382ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e2a9ee740 4 client.10302 mds_command: new command op to 10170 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2010817614 --> [v2:172.21.15.167:6835/446857148,v1:172.21.15.167:6837/446857148] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650312cd580 con 0x565030f72e40 2024-06-29T15:39:17.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.899+0000 7f7e217ea640 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:6835/446857148,v1:172.21.15.167:6837/446857148] conn(0x565030f72e40 0x565031382ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.900+0000 7f7dfbfff640 10 client.10302 ms_handle_connect on v2:172.21.15.167:6835/446857148 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.900+0000 7f7e217ea640 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:6835/446857148,v1:172.21.15.167:6837/446857148] conn(0x565030f72e40 0x565031382ee0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d529e7448f785271 server_cookie=cedacf76c36d006c in_seq=0 out_seq=0 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2010817614 <== mds.0 v2:172.21.15.167:6835/446857148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5650312cd580 con 0x565030f72e40 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7dfbfff640 10 client.10302 handle_command_reply: tid=0 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 client.10302 shutdown 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2010817614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 msgr2=0x5650313b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313b2080 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c08c480 tx=0x7f7e0c0046c0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2010817614 shutdown_connections 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:6835/446857148,v1:172.21.15.167:6837/446857148] conn(0x565030f72e40 0x565031382ee0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030fb4270 0x5650313b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313b2080 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7dfbfff640 0 client.10302 ms_handle_reset on v2:172.21.15.167:6835/446857148 2024-06-29T15:39:17.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2010817614 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030f8b3b0 0x56503137e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2010817614 >> 172.21.15.81:0/2010817614 conn(0x565030fa83c0 msgr2=0x565031380c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2010817614 shutdown_connections 2024-06-29T15:39:17.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2010817614 wait complete. 2024-06-29T15:39:17.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.901+0000 7f7e2a9ee740 20 client.10302 trim_cache size 0 max 16384 2024-06-29T15:39:17.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.920+0000 7f7e2a9ee740 1 Processor -- start 2024-06-29T15:39:17.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.920+0000 7f7e2a9ee740 1 -- start start 2024-06-29T15:39:17.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.921+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f8b3b0 0x5650313844b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.921+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313ada50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.921+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 0x5650313adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.921+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c200a20 con 0x565030f93590 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.921+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c0b3ca0 con 0x565030fb4270 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.921+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7e1c0b42f0 con 0x565030f8b3b0 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.921+0000 7f7e20fe9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.921+0000 7f7e20fe9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313ada50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56316/0 (socket says 172.21.15.81:56316) 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.921+0000 7f7e20fe9640 1 -- 172.21.15.81:0/3940851516 learned_addr learned my addr 172.21.15.81:0/3940851516 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e217ea640 1 --2- 172.21.15.81:0/3940851516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f8b3b0 0x5650313844b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e21feb640 1 --2- 172.21.15.81:0/3940851516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 0x5650313adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e20fe9640 1 -- 172.21.15.81:0/3940851516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f8b3b0 msgr2=0x5650313844b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/3940851516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f8b3b0 0x5650313844b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e20fe9640 1 -- 172.21.15.81:0/3940851516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 msgr2=0x5650313adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/3940851516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 0x5650313adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e20fe9640 1 -- 172.21.15.81:0/3940851516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565031381440 con 0x565030f93590 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e21feb640 1 --2- 172.21.15.81:0/3940851516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 0x5650313adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/3940851516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313ada50 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0058c0 tx=0x7f7e0c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0dbebb029541bf server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e067fc640 1 -- 172.21.15.81:0/3940851516 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c01dd50 con 0x565030f93590 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e067fc640 1 -- 172.21.15.81:0/3940851516 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e0c038030 con 0x565030f93590 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.922+0000 7f7e067fc640 1 -- 172.21.15.81:0/3940851516 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c01e690 con 0x565030f93590 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.923+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/3940851516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 msgr2=0x5650313ada50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.923+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/3940851516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313ada50 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0058c0 tx=0x7f7e0c013000 comp rx=0 tx=0).stop 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.923+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/3940851516 shutdown_connections 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.923+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/3940851516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 0x5650313adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.923+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/3940851516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f93590 0x5650313ada50 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.923+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/3940851516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f8b3b0 0x5650313844b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.923+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/3940851516 >> 172.21.15.81:0/3940851516 conn(0x565030fa83c0 msgr2=0x5650313553c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.923+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/3940851516 shutdown_connections 2024-06-29T15:39:17.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.923+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/3940851516 wait complete. 2024-06-29T15:39:17.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.924+0000 7f7e2a9ee740 1 -- start start 2024-06-29T15:39:17.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.924+0000 7f7e2a9ee740 1 Processor -- start 2024-06-29T15:39:17.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.925+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f8b3b0 0x56503141b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.925+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 0x5650313ada50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.925+0000 7f7e2a9ee740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 0x5650313adf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.925+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x565031384de0 con 0x565030f8b3b0 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.925+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x565031383c20 con 0x565030fb4270 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.925+0000 7f7e2a9ee740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x565031371d20 con 0x565030f93590 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.925+0000 7f7e2a9ee740 10 client.0 _ll_register_callbacks cb 0x7f7e0c00ea60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.925+0000 7f7e217ea640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f8b3b0 0x56503141b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.925+0000 7f7e217ea640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f8b3b0 0x56503141b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56322/0 (socket says 172.21.15.81:56322) 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.925+0000 7f7e217ea640 1 -- 172.21.15.81:0/2058909609 learned_addr learned my addr 172.21.15.81:0/2058909609 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e21feb640 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 0x5650313adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 0x5650313ada50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e217ea640 1 -- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 msgr2=0x5650313ada50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e217ea640 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 0x5650313ada50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e217ea640 1 -- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 msgr2=0x5650313adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e217ea640 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 0x5650313adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e217ea640 1 -- 172.21.15.81:0/2058909609 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56503137e2b0 con 0x565030f8b3b0 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e21feb640 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 0x5650313adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 0x5650313ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e217ea640 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f8b3b0 0x56503141b950 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f7e1805d740 tx=0x7f7e1800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220bd0fc635c0310 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7dfbfff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2058909609 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1801d040 con 0x565030f8b3b0 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2058909609 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f7e18005710 con 0x565030f8b3b0 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2058909609 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e0c038030 con 0x565030f8b3b0 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2058909609 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f7e18082050 con 0x565030f8b3b0 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e2a9ee740 10 client.10308 fetch_fsmap learned FSMap version 95 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.926+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2058909609 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=2+}) -- 0x56503141be90 con 0x565030f8b3b0 2024-06-29T15:39:17.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2058909609 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e0c038030 con 0x565030f8b3b0 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2058909609 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 95) ==== 3502+0+0 (secure 0 0 0) 0x7f7e180603b0 con 0x565030f8b3b0 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7e2a9ee740 10 client.10308 fetch_fsmap finished waiting for FSMap version 95 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7e2a9ee740 10 client.10308 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7e2a9ee740 20 client.10308 populate_metadata read hostname 'smithi081' 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:6835/446857148,v1:172.21.15.167:6837/446857148] conn(0x565031383770 0x56503139fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7e2a9ee740 4 client.10308 mds_command: new command op to 10170 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2058909609 --> [v2:172.21.15.167:6835/446857148,v1:172.21.15.167:6837/446857148] -- command(tid 0: {"prefix": "status"}) -- 0x5650312c9020 con 0x565031383770 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:6835/446857148,v1:172.21.15.167:6837/446857148] conn(0x565031383770 0x56503139fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7e20fe9640 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:6835/446857148,v1:172.21.15.167:6837/446857148] conn(0x565031383770 0x56503139fb10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=112ab8f85300f498 server_cookie=5876497419c8f8de in_seq=0 out_seq=0 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.927+0000 7f7dfbfff640 10 client.10308 ms_handle_connect on v2:172.21.15.167:6835/446857148 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7dfbfff640 1 -- 172.21.15.81:0/2058909609 <== mds.0 v2:172.21.15.167:6835/446857148 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x5650312c9020 con 0x565031383770 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7dfbfff640 10 client.10308 handle_command_reply: tid=0 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 client.10308 shutdown 2024-06-29T15:39:17.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2058909609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f8b3b0 msgr2=0x56503141b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f8b3b0 0x56503141b950 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f7e1805d740 tx=0x7f7e1800cc30 comp rx=0 tx=0).stop 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2058909609 shutdown_connections 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:6835/446857148,v1:172.21.15.167:6837/446857148] conn(0x565031383770 0x56503139fb10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x565030fb4270 0x5650313adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x565030f93590 0x5650313ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 --2- 172.21.15.81:0/2058909609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565030f8b3b0 0x56503141b950 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7dfbfff640 0 client.10308 ms_handle_reset on v2:172.21.15.167:6835/446857148 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2058909609 >> 172.21.15.81:0/2058909609 conn(0x565030fa83c0 msgr2=0x56503135b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2058909609 shutdown_connections 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 1 -- 172.21.15.81:0/2058909609 wait complete. 2024-06-29T15:39:17.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e2a9ee740 20 client.10308 trim_cache size 0 max 16384 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 10170, 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7556976410000003, 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 95, 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 181, 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 181, 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.421487977 2024-06-29T15:39:17.935 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:39:17.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7e00076420 msgr2=0x7f7e00078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e23a4e640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7e00076420 0x7f7e00078930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c005fd0 tx=0x7f7e0c005da0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4292447153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e1c0b5250 msgr2=0x7f7e1c0b1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:17.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.928+0000 7f7e23a4e640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e1c0b5250 0x7f7e1c0b1700 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800cd10 tx=0x7f7e18007610 comp rx=0 tx=0).stop 2024-06-29T15:39:17.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.929+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4292447153 shutdown_connections 2024-06-29T15:39:17.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.929+0000 7f7e23a4e640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f7e00076420 0x7f7e00078930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.929+0000 7f7e23a4e640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e1c0b9040 0x7f7e1c0cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.929+0000 7f7e23a4e640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e1c0b7290 0x7f7e1c153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.929+0000 7f7e23a4e640 1 --2- 172.21.15.81:0/4292447153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e1c0b5250 0x7f7e1c0b1700 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:17.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.929+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4292447153 >> 172.21.15.81:0/4292447153 conn(0x7f7e1c09cf30 msgr2=0x7f7e1c0d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:17.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.929+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4292447153 shutdown_connections 2024-06-29T15:39:17.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:17.929+0000 7f7e23a4e640 1 -- 172.21.15.81:0/4292447153 wait complete. 2024-06-29T15:39:17.941 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:39:18.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.052+0000 7fde41cac640 1 -- 172.21.15.81:0/3756638375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde3c0ad0c0 msgr2=0x7fde3c15bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:18.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.052+0000 7fde41cac640 1 --2- 172.21.15.81:0/3756638375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde3c0ad0c0 0x7fde3c15bdf0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0d0c70 tx=0x7fde2c02a560 comp rx=0 tx=0).stop 2024-06-29T15:39:18.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.052+0000 7fde41cac640 1 -- 172.21.15.81:0/3756638375 shutdown_connections 2024-06-29T15:39:18.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.052+0000 7fde41cac640 1 --2- 172.21.15.81:0/3756638375 >> v2:172.21.15.167:3300/0 conn(0x7fde3c074630 0x7fde3c06c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.052+0000 7fde41cac640 1 --2- 172.21.15.81:0/3756638375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde3c0ad0c0 0x7fde3c15bdf0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.052+0000 7fde41cac640 1 -- 172.21.15.81:0/3756638375 >> 172.21.15.81:0/3756638375 conn(0x7fde3c072950 msgr2=0x7fde3c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:18.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.052+0000 7fde41cac640 1 -- 172.21.15.81:0/3756638375 shutdown_connections 2024-06-29T15:39:18.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.052+0000 7fde41cac640 1 -- 172.21.15.81:0/3756638375 wait complete. 2024-06-29T15:39:18.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.054+0000 7fde41cac640 1 Processor -- start 2024-06-29T15:39:18.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.054+0000 7fde41cac640 1 -- start start 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde41cac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde3c073b10 0x7fde3c145520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde41cac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde3c074630 0x7fde3c154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde41cac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde3c0ad0c0 0x7fde3c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde41cac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c06cf80 con 0x7fde3c074630 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde41cac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c06ce00 con 0x7fde3c0ad0c0 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde41cac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fde3c06d100 con 0x7fde3c073b10 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde3c074630 0x7fde3c154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde3c074630 0x7fde3c154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56354/0 (socket says 172.21.15.81:56354) 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3affd640 1 -- 172.21.15.81:0/3628559671 learned_addr learned my addr 172.21.15.81:0/3628559671 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3bfff640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde3c0ad0c0 0x7fde3c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:18.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3b7fe640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde3c073b10 0x7fde3c145520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3affd640 1 -- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde3c073b10 msgr2=0x7fde3c145520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3affd640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde3c073b10 0x7fde3c145520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3affd640 1 -- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde3c0ad0c0 msgr2=0x7fde3c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3affd640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde3c0ad0c0 0x7fde3c15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3affd640 1 -- 172.21.15.81:0/3628559671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c145a60 con 0x7fde3c074630 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3bfff640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde3c0ad0c0 0x7fde3c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde3affd640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde3c074630 0x7fde3c154440 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c340 tx=0x7fde2400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde33fff640 1 -- 172.21.15.81:0/3628559671 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24017070 con 0x7fde3c074630 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde33fff640 1 -- 172.21.15.81:0/3628559671 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde240053c0 con 0x7fde3c074630 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde41cac640 1 -- 172.21.15.81:0/3628559671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3c15d610 con 0x7fde3c074630 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde33fff640 1 -- 172.21.15.81:0/3628559671 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24003870 con 0x7fde3c074630 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.056+0000 7fde41cac640 1 -- 172.21.15.81:0/3628559671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3c151ad0 con 0x7fde3c074630 2024-06-29T15:39:18.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.057+0000 7fde41cac640 1 -- 172.21.15.81:0/3628559671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde3c06dfc0 con 0x7fde3c074630 2024-06-29T15:39:18.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.058+0000 7fde33fff640 1 -- 172.21.15.81:0/3628559671 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7fde2401a070 con 0x7fde3c074630 2024-06-29T15:39:18.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.059+0000 7fde33fff640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fde1c0763f0 0x7fde1c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:18.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.059+0000 7fde33fff640 1 -- 172.21.15.81:0/3628559671 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7fde24096e20 con 0x7fde3c074630 2024-06-29T15:39:18.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.059+0000 7fde3b7fe640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fde1c0763f0 0x7fde1c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:18.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.059+0000 7fde3b7fe640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fde1c0763f0 0x7fde1c078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fde2c033000 tx=0x7fde2c004730 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:18.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.063+0000 7fde33fff640 1 -- 172.21.15.81:0/3628559671 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fde3c06dfc0 con 0x7fde3c074630 2024-06-29T15:39:18.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.283+0000 7fde41cac640 1 -- 172.21.15.81:0/3628559671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fde3c068ef0 con 0x7fde3c074630 2024-06-29T15:39:18.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.284+0000 7fde33fff640 1 -- 172.21.15.81:0/3628559671 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v95) ==== 53+0+73 (secure 0 0 0) 0x7fde3c068ef0 con 0x7fde3c074630 2024-06-29T15:39:18.285 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 -- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fde1c0763f0 msgr2=0x7fde1c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fde1c0763f0 0x7fde1c078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fde2c033000 tx=0x7fde2c004730 comp rx=0 tx=0).stop 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 -- 172.21.15.81:0/3628559671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde3c074630 msgr2=0x7fde3c154440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde3c074630 0x7fde3c154440 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c340 tx=0x7fde2400c800 comp rx=0 tx=0).stop 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 -- 172.21.15.81:0/3628559671 shutdown_connections 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fde1c0763f0 0x7fde1c078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde3c0ad0c0 0x7fde3c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde3c074630 0x7fde3c154440 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 --2- 172.21.15.81:0/3628559671 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde3c073b10 0x7fde3c145520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 -- 172.21.15.81:0/3628559671 >> 172.21.15.81:0/3628559671 conn(0x7fde3c072950 msgr2=0x7fde3c073f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 -- 172.21.15.81:0/3628559671 shutdown_connections 2024-06-29T15:39:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18.286+0000 7fde41cac640 1 -- 172.21.15.81:0/3628559671 wait complete. 2024-06-29T15:39:18.298 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:39:18.298 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:39:18.299 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T15:39:18.299 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:39:18.299 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:39:18.299 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:39:18.299 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:39:18.299 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:39:18.299 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:39:18.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.299 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T15:39:18.315 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:39:18.315 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:39:18.315 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:39:18.315 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.315 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T15:39:18.315 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.315 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:39:18.315 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 38161sec preferred_lft 38161sec 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:39:18.316 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:39:18.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.318 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:39:18.378 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T15:39:18.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.380 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T15:39:18.436 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T15:39:18.436 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:39:18.436 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T15:39:18.436 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T15:39:18.436 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:39:18.437 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:39:18.437 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T15:39:18.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T15:39:18.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T15:39:18.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T15:39:18.437 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T15:39:18.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:39:18.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:39:18.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.562 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T15:39:18.578 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:39:18.578 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:39:18.578 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:39:18.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T15:39:18.643 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:39:18.644 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:39:18.699 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:39:18.699 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:39:18.699 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:39:18.754 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T15:39:18.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:39:18.848 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:39:18.848 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:39:18.856 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:39:18.856 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:39:18.856 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T15:39:18.856 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T15:39:18.856 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T15:39:18.857 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:39:18.857 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:39:18.857 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:39:18.857 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:39:18.857 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:39:18.858 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:39:18.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:18.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:39:18.884 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:39:18.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:39:18.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:39:18.987 INFO:teuthology.orchestra.run.smithi081.stdout:client9235 2024-06-29T15:39:18.987 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:39:18.987 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:39:19.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:39:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.155+0000 7f2ceabeb640 1 -- 172.21.15.81:0/611692759 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce41412f0 msgr2=0x7f2ce40cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.155+0000 7f2ceabeb640 1 --2- 172.21.15.81:0/611692759 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce41412f0 0x7f2ce40cef90 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4071d20 tx=0x7f2cd402a530 comp rx=0 tx=0).stop 2024-06-29T15:39:19.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.156+0000 7f2ceabeb640 1 -- 172.21.15.81:0/611692759 shutdown_connections 2024-06-29T15:39:19.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.156+0000 7f2ceabeb640 1 --2- 172.21.15.81:0/611692759 >> v2:172.21.15.167:3300/0 conn(0x7f2ce4073740 0x7f2ce415e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.156+0000 7f2ceabeb640 1 --2- 172.21.15.81:0/611692759 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce41412f0 0x7f2ce40cef90 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.156+0000 7f2ceabeb640 1 -- 172.21.15.81:0/611692759 >> 172.21.15.81:0/611692759 conn(0x7f2ce406ccc0 msgr2=0x7f2ce40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:19.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.156+0000 7f2ceabeb640 1 -- 172.21.15.81:0/611692759 shutdown_connections 2024-06-29T15:39:19.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.156+0000 7f2ceabeb640 1 -- 172.21.15.81:0/611692759 wait complete. 2024-06-29T15:39:19.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.157+0000 7f2ceabeb640 1 Processor -- start 2024-06-29T15:39:19.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.157+0000 7f2ceabeb640 1 -- start start 2024-06-29T15:39:19.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ceabeb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ce40cf4d0 0x7f2ce41562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ceabeb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ce41412f0 0x7f2ce4152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ceabeb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce41401f0 0x7f2ce4152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ceabeb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce40b3210 con 0x7f2ce41401f0 2024-06-29T15:39:19.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ceabeb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce415f1f0 con 0x7f2ce41412f0 2024-06-29T15:39:19.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ceabeb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2ce40b3390 con 0x7f2ce40cf4d0 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ce9188640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce41401f0 0x7f2ce4152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ce9188640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce41401f0 0x7f2ce4152740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56366/0 (socket says 172.21.15.81:56366) 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ce9188640 1 -- 172.21.15.81:0/2492915813 learned_addr learned my addr 172.21.15.81:0/2492915813 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ce3fff640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ce41412f0 0x7f2ce4152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ce8987640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ce40cf4d0 0x7f2ce41562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ce9188640 1 -- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ce40cf4d0 msgr2=0x7f2ce41562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ce9188640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ce40cf4d0 0x7f2ce41562b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ce9188640 1 -- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ce41412f0 msgr2=0x7f2ce4152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ce9188640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ce41412f0 0x7f2ce4152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.159+0000 7f2ce9188640 1 -- 172.21.15.81:0/2492915813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce41511a0 con 0x7f2ce41401f0 2024-06-29T15:39:19.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.160+0000 7f2ce9188640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce41401f0 0x7f2ce4152740 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009a40 tx=0x7f2cdc009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:19.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.160+0000 7f2ce17fa640 1 -- 172.21.15.81:0/2492915813 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc00e2e0 con 0x7f2ce41401f0 2024-06-29T15:39:19.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.160+0000 7f2ce17fa640 1 -- 172.21.15.81:0/2492915813 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cdc004b20 con 0x7f2ce41401f0 2024-06-29T15:39:19.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.160+0000 7f2ce17fa640 1 -- 172.21.15.81:0/2492915813 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc0078c0 con 0x7f2ce41401f0 2024-06-29T15:39:19.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.160+0000 7f2ceabeb640 1 -- 172.21.15.81:0/2492915813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce4151440 con 0x7f2ce41401f0 2024-06-29T15:39:19.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.160+0000 7f2ceabeb640 1 -- 172.21.15.81:0/2492915813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ce4151a00 con 0x7f2ce41401f0 2024-06-29T15:39:19.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.161+0000 7f2ceabeb640 1 -- 172.21.15.81:0/2492915813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ce40b1c60 con 0x7f2ce41401f0 2024-06-29T15:39:19.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.162+0000 7f2ce17fa640 1 -- 172.21.15.81:0/2492915813 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f2cdc007a60 con 0x7f2ce41401f0 2024-06-29T15:39:19.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.162+0000 7f2ce17fa640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2cd0076420 0x7f2cd0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.162+0000 7f2ce17fa640 1 -- 172.21.15.81:0/2492915813 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f2cdc0992e0 con 0x7f2ce41401f0 2024-06-29T15:39:19.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.162+0000 7f2ce8987640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2cd0076420 0x7f2cd0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.163+0000 7f2ce8987640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2cd0076420 0x7f2cd0078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4034000 tx=0x7f2cd4004730 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:19.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.167+0000 7f2ce17fa640 1 -- 172.21.15.81:0/2492915813 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2ce40b1c60 con 0x7f2ce41401f0 2024-06-29T15:39:19.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.374+0000 7f2ceabeb640 1 -- 172.21.15.81:0/2492915813 --> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f2ce4068f50 con 0x7f2cd0076420 2024-06-29T15:39:19.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.376+0000 7f2ce17fa640 1 -- 172.21.15.81:0/2492915813 <== mgr.7925 v2:172.21.15.167:6832/2546364927 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f2ce4068f50 con 0x7f2cd0076420 2024-06-29T15:39:19.377 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:39:19.378 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:39:19.378 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T15:39:19.378 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:39:19.378 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 -- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2cd0076420 msgr2=0x7f2cd0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2cd0076420 0x7f2cd0078930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4034000 tx=0x7f2cd4004730 comp rx=0 tx=0).stop 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 -- 172.21.15.81:0/2492915813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce41401f0 msgr2=0x7f2ce4152740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce41401f0 0x7f2ce4152740 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009a40 tx=0x7f2cdc009f00 comp rx=0 tx=0).stop 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 -- 172.21.15.81:0/2492915813 shutdown_connections 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f2cd0076420 0x7f2cd0078930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce41401f0 0x7f2ce4152740 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ce41412f0 0x7f2ce4152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 --2- 172.21.15.81:0/2492915813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ce40cf4d0 0x7f2ce41562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 -- 172.21.15.81:0/2492915813 >> 172.21.15.81:0/2492915813 conn(0x7f2ce406ccc0 msgr2=0x7f2ce40b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 -- 172.21.15.81:0/2492915813 shutdown_connections 2024-06-29T15:39:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.378+0000 7f2ceabeb640 1 -- 172.21.15.81:0/2492915813 wait complete. 2024-06-29T15:39:19.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:39:19.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.504+0000 7f52150fe640 1 -- 172.21.15.81:0/1258868446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51f8003a50 msgr2=0x7f51f8003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.504+0000 7f52150fe640 1 --2- 172.21.15.81:0/1258868446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51f8003a50 0x7f51f8003ec0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f5200001dc0 tx=0x7f51f8005830 comp rx=0 tx=0).stop 2024-06-29T15:39:19.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.504+0000 7f52150fe640 1 -- 172.21.15.81:0/1258868446 shutdown_connections 2024-06-29T15:39:19.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.504+0000 7f52150fe640 1 --2- 172.21.15.81:0/1258868446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51f8003a50 0x7f51f8003ec0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.504+0000 7f52150fe640 1 -- 172.21.15.81:0/1258868446 >> 172.21.15.81:0/1258868446 conn(0x7f52100b1ce0 msgr2=0x7f52100b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:19.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.504+0000 7f52150fe640 1 -- 172.21.15.81:0/1258868446 shutdown_connections 2024-06-29T15:39:19.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.504+0000 7f52150fe640 1 -- 172.21.15.81:0/1258868446 wait complete. 2024-06-29T15:39:19.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.506+0000 7f52150fe640 1 Processor -- start 2024-06-29T15:39:19.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.506+0000 7f52150fe640 1 -- start start 2024-06-29T15:39:19.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.507+0000 7f52150fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51f8003a50 0x7f52101f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f52150fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52101f5910 0x7f5210157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f52150fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5210157cb0 0x7f5210133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f52150fe640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f521015bb50 con 0x7f51f8003a50 2024-06-29T15:39:19.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f52150fe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f521015b9d0 con 0x7f52101f5910 2024-06-29T15:39:19.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f52150fe640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f521015bcd0 con 0x7f5210157cb0 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51f8003a50 0x7f52101f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51f8003a50 0x7f52101f53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56382/0 (socket says 172.21.15.81:56382) 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520ed9d640 1 -- 172.21.15.81:0/1018282684 learned_addr learned my addr 172.21.15.81:0/1018282684 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520e59c640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52101f5910 0x7f5210157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520f59e640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5210157cb0 0x7f5210133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520ed9d640 1 -- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5210157cb0 msgr2=0x7f5210133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520ed9d640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5210157cb0 0x7f5210133c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520ed9d640 1 -- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52101f5910 msgr2=0x7f5210157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520ed9d640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52101f5910 0x7f5210157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520ed9d640 1 -- 172.21.15.81:0/1018282684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52101f5d20 con 0x7f51f8003a50 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f520ed9d640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51f8003a50 0x7f52101f53d0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f5200000f80 tx=0x7f5200003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f51f77fe640 1 -- 172.21.15.81:0/1018282684 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5200006460 con 0x7f51f8003a50 2024-06-29T15:39:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f51f77fe640 1 -- 172.21.15.81:0/1018282684 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52000066f0 con 0x7f51f8003a50 2024-06-29T15:39:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f51f77fe640 1 -- 172.21.15.81:0/1018282684 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52000336e0 con 0x7f51f8003a50 2024-06-29T15:39:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f52150fe640 1 -- 172.21.15.81:0/1018282684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52101569b0 con 0x7f51f8003a50 2024-06-29T15:39:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.508+0000 7f52150fe640 1 -- 172.21.15.81:0/1018282684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5210156f70 con 0x7f51f8003a50 2024-06-29T15:39:19.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.510+0000 7f52150fe640 1 -- 172.21.15.81:0/1018282684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f521015c230 con 0x7f51f8003a50 2024-06-29T15:39:19.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.510+0000 7f51f77fe640 1 -- 172.21.15.81:0/1018282684 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 99) ==== 98303+0+0 (secure 0 0 0) 0x7f520003a040 con 0x7f51f8003a50 2024-06-29T15:39:19.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.511+0000 7f51f77fe640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f51fc0763f0 0x7f51fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:19.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.511+0000 7f51f77fe640 1 -- 172.21.15.81:0/1018282684 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f52000ba710 con 0x7f51f8003a50 2024-06-29T15:39:19.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.511+0000 7f520e59c640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f51fc0763f0 0x7f51fc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:19.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.511+0000 7f520e59c640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f51fc0763f0 0x7f51fc078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f52101436b0 tx=0x7f520400b260 comp rx=0 tx=0).ready entity=mgr.7925 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:19.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.515+0000 7f51f77fe640 1 -- 172.21.15.81:0/1018282684 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f52101f5d20 con 0x7f51f8003a50 2024-06-29T15:39:19.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:19.743+0000 7f52150fe640 1 -- 172.21.15.81:0/1018282684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f52100cd010 con 0x7f51f8003a50 2024-06-29T15:39:20.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.213+0000 7f51f77fe640 1 -- 172.21.15.81:0/1018282684 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v100) ==== 92+0+0 (secure 0 0 0) 0x7f5200083e70 con 0x7f51f8003a50 2024-06-29T15:39:20.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.216+0000 7f52150fe640 1 -- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f51fc0763f0 msgr2=0x7f51fc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.216+0000 7f52150fe640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f51fc0763f0 0x7f51fc078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f52101436b0 tx=0x7f520400b260 comp rx=0 tx=0).stop 2024-06-29T15:39:20.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.216+0000 7f52150fe640 1 -- 172.21.15.81:0/1018282684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51f8003a50 msgr2=0x7f52101f53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.216+0000 7f52150fe640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51f8003a50 0x7f52101f53d0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f5200000f80 tx=0x7f5200003aa0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.216+0000 7f52150fe640 1 -- 172.21.15.81:0/1018282684 shutdown_connections 2024-06-29T15:39:20.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.216+0000 7f52150fe640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f51fc0763f0 0x7f51fc078900 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f52101436b0 tx=0x7f520400b260 comp rx=0 tx=0).stop 2024-06-29T15:39:20.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.216+0000 7f52150fe640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5210157cb0 0x7f5210133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.216+0000 7f52150fe640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52101f5910 0x7f5210157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.216+0000 7f52150fe640 1 --2- 172.21.15.81:0/1018282684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51f8003a50 0x7f52101f53d0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.216+0000 7f52150fe640 1 -- 172.21.15.81:0/1018282684 >> 172.21.15.81:0/1018282684 conn(0x7f52100b1ce0 msgr2=0x7f52100b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:20.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.217+0000 7f52150fe640 1 -- 172.21.15.81:0/1018282684 shutdown_connections 2024-06-29T15:39:20.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.217+0000 7f52150fe640 1 -- 172.21.15.81:0/1018282684 wait complete. 2024-06-29T15:39:20.232 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:39:20.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.352+0000 7fee17903640 1 -- 172.21.15.81:0/1963543033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee100ce770 msgr2=0x7fee100b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.352+0000 7fee17903640 1 --2- 172.21.15.81:0/1963543033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee100ce770 0x7fee100b35c0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fee100c7400 tx=0x7fee0002a4d0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.352+0000 7fee17903640 1 -- 172.21.15.81:0/1963543033 shutdown_connections 2024-06-29T15:39:20.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.352+0000 7fee17903640 1 --2- 172.21.15.81:0/1963543033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee100d2240 0x7fee100d2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.352+0000 7fee17903640 1 --2- 172.21.15.81:0/1963543033 >> v2:172.21.15.167:3300/0 conn(0x7fee100ceb80 0x7fee100b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.352+0000 7fee17903640 1 --2- 172.21.15.81:0/1963543033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee100ce770 0x7fee100b35c0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.352+0000 7fee17903640 1 -- 172.21.15.81:0/1963543033 >> 172.21.15.81:0/1963543033 conn(0x7fee100d3c60 msgr2=0x7fee100d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:20.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.352+0000 7fee17903640 1 -- 172.21.15.81:0/1963543033 shutdown_connections 2024-06-29T15:39:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.352+0000 7fee17903640 1 -- 172.21.15.81:0/1963543033 wait complete. 2024-06-29T15:39:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.354+0000 7fee17903640 1 Processor -- start 2024-06-29T15:39:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.354+0000 7fee17903640 1 -- start start 2024-06-29T15:39:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee17903640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee100ce770 0x7fee101dd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee17903640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee100ceb80 0x7fee101e3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee17903640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee100d2240 0x7fee101e4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee17903640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee100d4990 con 0x7fee100ceb80 2024-06-29T15:39:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee17903640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fee100d4b10 con 0x7fee100ce770 2024-06-29T15:39:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee17903640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fee100d0410 con 0x7fee100d2240 2024-06-29T15:39:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee14e9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee100ceb80 0x7fee101e3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee14e9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee100ceb80 0x7fee101e3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35834/0 (socket says 172.21.15.81:35834) 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee14e9e640 1 -- 172.21.15.81:0/3398537010 learned_addr learned my addr 172.21.15.81:0/3398537010 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee15ea0640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee100d2240 0x7fee101e4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.355+0000 7fee1569f640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee100ce770 0x7fee101dd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee14e9e640 1 -- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee100d2240 msgr2=0x7fee101e4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee14e9e640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee100d2240 0x7fee101e4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee14e9e640 1 -- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee100ce770 msgr2=0x7fee101dd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee14e9e640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee100ce770 0x7fee101dd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee14e9e640 1 -- 172.21.15.81:0/3398537010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee101dd690 con 0x7fee100ceb80 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee14e9e640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee100ceb80 0x7fee101e3e40 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00e740 tx=0x7fee0c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee05ffb640 1 -- 172.21.15.81:0/3398537010 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c00ccc0 con 0x7fee100ceb80 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee05ffb640 1 -- 172.21.15.81:0/3398537010 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee0c00ee50 con 0x7fee100ceb80 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee17903640 1 -- 172.21.15.81:0/3398537010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee100cacf0 con 0x7fee100ceb80 2024-06-29T15:39:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee05ffb640 1 -- 172.21.15.81:0/3398537010 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c010670 con 0x7fee100ceb80 2024-06-29T15:39:20.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.356+0000 7fee17903640 1 -- 172.21.15.81:0/3398537010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee1009dba0 con 0x7fee100ceb80 2024-06-29T15:39:20.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.357+0000 7fee17903640 1 -- 172.21.15.81:0/3398537010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee100d2650 con 0x7fee100ceb80 2024-06-29T15:39:20.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.358+0000 7fee05ffb640 1 -- 172.21.15.81:0/3398537010 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 100) ==== 98320+0+0 (secure 0 0 0) 0x7fee0c010810 con 0x7fee100ceb80 2024-06-29T15:39:20.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.359+0000 7fee05ffb640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fedf4076440 0x7fedf4078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.359+0000 7fee1569f640 1 -- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fedf4076440 msgr2=0x7fedf4078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2546364927 2024-06-29T15:39:20.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.359+0000 7fee05ffb640 1 -- 172.21.15.81:0/3398537010 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7fee0c098a40 con 0x7fee100ceb80 2024-06-29T15:39:20.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.359+0000 7fee1569f640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fedf4076440 0x7fedf4078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:39:20.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.366+0000 7fee05ffb640 1 -- 172.21.15.81:0/3398537010 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fee101dd690 con 0x7fee100ceb80 2024-06-29T15:39:20.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.557+0000 7fee17903640 1 -- 172.21.15.81:0/3398537010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fee100cfd90 con 0x7fee100ceb80 2024-06-29T15:39:20.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.560+0000 7fee1569f640 1 -- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fedf4076440 msgr2=0x7fedf4078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2546364927 2024-06-29T15:39:20.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.560+0000 7fee1569f640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fedf4076440 0x7fedf4078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:39:20.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.560+0000 7fee05ffb640 1 -- 172.21.15.81:0/3398537010 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) ==== 138+0+0 (secure 0 0 0) 0x7fee0c061c30 con 0x7fee100ceb80 2024-06-29T15:39:20.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.562+0000 7fee17903640 1 -- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fedf4076440 msgr2=0x7fedf4078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.562+0000 7fee17903640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fedf4076440 0x7fedf4078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.562+0000 7fee17903640 1 -- 172.21.15.81:0/3398537010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee100ceb80 msgr2=0x7fee101e3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.562+0000 7fee17903640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee100ceb80 0x7fee101e3e40 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00e740 tx=0x7fee0c00ec00 comp rx=0 tx=0).stop 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.562+0000 7fee17903640 1 -- 172.21.15.81:0/3398537010 shutdown_connections 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.562+0000 7fee17903640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7fedf4076440 0x7fedf4078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.562+0000 7fee17903640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fee100d2240 0x7fee101e4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.562+0000 7fee17903640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee100ceb80 0x7fee101e3e40 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.562+0000 7fee17903640 1 --2- 172.21.15.81:0/3398537010 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fee100ce770 0x7fee101dd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.562+0000 7fee17903640 1 -- 172.21.15.81:0/3398537010 >> 172.21.15.81:0/3398537010 conn(0x7fee100d3c60 msgr2=0x7fee100b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.563+0000 7fee17903640 1 -- 172.21.15.81:0/3398537010 shutdown_connections 2024-06-29T15:39:20.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.563+0000 7fee17903640 1 -- 172.21.15.81:0/3398537010 wait complete. 2024-06-29T15:39:20.576 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:39:20.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.694+0000 7ff71e066640 1 -- 172.21.15.81:0/1785838303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff700004800 msgr2=0x7ff700004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.694+0000 7ff71e066640 1 --2- 172.21.15.81:0/1785838303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff700004800 0x7ff700004c10 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7ff70c001f00 tx=0x7ff700005830 comp rx=0 tx=0).stop 2024-06-29T15:39:20.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.694+0000 7ff71e066640 1 -- 172.21.15.81:0/1785838303 shutdown_connections 2024-06-29T15:39:20.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.694+0000 7ff71e066640 1 --2- 172.21.15.81:0/1785838303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff700004800 0x7ff700004c10 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.694+0000 7ff71e066640 1 -- 172.21.15.81:0/1785838303 >> 172.21.15.81:0/1785838303 conn(0x7ff7180b3680 msgr2=0x7ff7180b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:20.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.694+0000 7ff71e066640 1 -- 172.21.15.81:0/1785838303 shutdown_connections 2024-06-29T15:39:20.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.694+0000 7ff71e066640 1 -- 172.21.15.81:0/1785838303 wait complete. 2024-06-29T15:39:20.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.696+0000 7ff71e066640 1 Processor -- start 2024-06-29T15:39:20.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.696+0000 7ff71e066640 1 -- start start 2024-06-29T15:39:20.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff71e066640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff700004800 0x7ff7181537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff71e066640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7180caf80 0x7ff7180b5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff71e066640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7180cb390 0x7ff7180b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff71e066640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff7180d05c0 con 0x7ff700004800 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff71e066640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff7180d0440 con 0x7ff7180cb390 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff71e066640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff7180d0740 con 0x7ff7180caf80 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff7177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff700004800 0x7ff7181537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff7177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff700004800 0x7ff7181537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35850/0 (socket says 172.21.15.81:35850) 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff7177fe640 1 -- 172.21.15.81:0/1750920155 learned_addr learned my addr 172.21.15.81:0/1750920155 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff716ffd640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7180caf80 0x7ff7180b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff717fff640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7180cb390 0x7ff7180b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff7177fe640 1 -- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7180caf80 msgr2=0x7ff7180b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff7177fe640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7180caf80 0x7ff7180b5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff7177fe640 1 -- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7180cb390 msgr2=0x7ff7180b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff7177fe640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7180cb390 0x7ff7180b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff7177fe640 1 -- 172.21.15.81:0/1750920155 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff718154040 con 0x7ff700004800 2024-06-29T15:39:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff7177fe640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff700004800 0x7ff7181537c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7ff708011780 tx=0x7ff708011b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:20.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.698+0000 7ff6fffff640 1 -- 172.21.15.81:0/1750920155 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff708004230 con 0x7ff700004800 2024-06-29T15:39:20.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.699+0000 7ff6fffff640 1 -- 172.21.15.81:0/1750920155 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff70800f8c0 con 0x7ff700004800 2024-06-29T15:39:20.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.699+0000 7ff6fffff640 1 -- 172.21.15.81:0/1750920155 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7080044c0 con 0x7ff700004800 2024-06-29T15:39:20.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.699+0000 7ff71e066640 1 -- 172.21.15.81:0/1750920155 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7180c8d80 con 0x7ff700004800 2024-06-29T15:39:20.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.699+0000 7ff71e066640 1 -- 172.21.15.81:0/1750920155 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7180c9340 con 0x7ff700004800 2024-06-29T15:39:20.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.700+0000 7ff71e066640 1 -- 172.21.15.81:0/1750920155 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7180ce520 con 0x7ff700004800 2024-06-29T15:39:20.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.700+0000 7ff6fffff640 1 -- 172.21.15.81:0/1750920155 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 100) ==== 98320+0+0 (secure 0 0 0) 0x7ff70800fd30 con 0x7ff700004800 2024-06-29T15:39:20.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.701+0000 7ff6fffff640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff704076440 0x7ff704078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:20.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.701+0000 7ff716ffd640 1 -- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff704076440 msgr2=0x7ff704078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2546364927 2024-06-29T15:39:20.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.701+0000 7ff6fffff640 1 -- 172.21.15.81:0/1750920155 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7ff7080983a0 con 0x7ff700004800 2024-06-29T15:39:20.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.701+0000 7ff716ffd640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff704076440 0x7ff704078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:39:20.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.705+0000 7ff6fffff640 1 -- 172.21.15.81:0/1750920155 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff7080615c0 con 0x7ff700004800 2024-06-29T15:39:20.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.890+0000 7ff71e066640 1 -- 172.21.15.81:0/1750920155 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7ff718159e50 con 0x7ff700004800 2024-06-29T15:39:20.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.894+0000 7ff6fffff640 1 -- 172.21.15.81:0/1750920155 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) ==== 133+0+0 (secure 0 0 0) 0x7ff7080658c0 con 0x7ff700004800 2024-06-29T15:39:20.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 -- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff704076440 msgr2=0x7ff704078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:39:20.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff704076440 0x7ff704078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 -- 172.21.15.81:0/1750920155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff700004800 msgr2=0x7ff7181537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:20.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff700004800 0x7ff7181537c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7ff708011780 tx=0x7ff708011b80 comp rx=0 tx=0).stop 2024-06-29T15:39:20.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 -- 172.21.15.81:0/1750920155 shutdown_connections 2024-06-29T15:39:20.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7ff704076440 0x7ff704078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7180cb390 0x7ff7180b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7180caf80 0x7ff7180b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 --2- 172.21.15.81:0/1750920155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff700004800 0x7ff7181537c0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:20.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 -- 172.21.15.81:0/1750920155 >> 172.21.15.81:0/1750920155 conn(0x7ff7180b3680 msgr2=0x7ff7180708f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:20.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 -- 172.21.15.81:0/1750920155 shutdown_connections 2024-06-29T15:39:20.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:20.896+0000 7ff71e066640 1 -- 172.21.15.81:0/1750920155 wait complete. 2024-06-29T15:39:20.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:20.909 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T15:39:20.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:39:20.929 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2024-06-29T15:39:20.986 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2024-06-29T15:39:21.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.144+0000 7f1e8d773640 1 -- 172.21.15.81:0/3041789935 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e88073650 msgr2=0x7f1e8815b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:21.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.144+0000 7f1e8d773640 1 --2- 172.21.15.81:0/3041789935 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e88073650 0x7f1e8815b1e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c009830 tx=0x7f1e7c02a5c0 comp rx=0 tx=0).stop 2024-06-29T15:39:21.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.144+0000 7f1e8d773640 1 -- 172.21.15.81:0/3041789935 shutdown_connections 2024-06-29T15:39:21.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.144+0000 7f1e8d773640 1 --2- 172.21.15.81:0/3041789935 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e88073650 0x7f1e8815b1e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:21.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.144+0000 7f1e8d773640 1 -- 172.21.15.81:0/3041789935 >> 172.21.15.81:0/3041789935 conn(0x7f1e880d0b70 msgr2=0x7f1e880d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:21.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.144+0000 7f1e8d773640 1 -- 172.21.15.81:0/3041789935 shutdown_connections 2024-06-29T15:39:21.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.145+0000 7f1e8d773640 1 -- 172.21.15.81:0/3041789935 wait complete. 2024-06-29T15:39:21.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.146+0000 7f1e8d773640 1 Processor -- start 2024-06-29T15:39:21.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.146+0000 7f1e8d773640 1 -- start start 2024-06-29T15:39:21.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e8d773640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e880b46a0 0x7f1e8806c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:21.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e8d773640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e8815b720 0x7f1e8813df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e8d773640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e8806ca10 0x7f1e8813e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e8d773640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8806f850 con 0x7f1e8806ca10 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e8d773640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8806f6d0 con 0x7f1e8815b720 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e8d773640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1e8806f9d0 con 0x7f1e880b46a0 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e877fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e8806ca10 0x7f1e8813e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e867fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e8815b720 0x7f1e8813df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e877fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e8806ca10 0x7f1e8813e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35858/0 (socket says 172.21.15.81:35858) 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e877fe640 1 -- 172.21.15.81:0/3427484672 learned_addr learned my addr 172.21.15.81:0/3427484672 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e86ffd640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e880b46a0 0x7f1e8806c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e877fe640 1 -- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e880b46a0 msgr2=0x7f1e8806c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e877fe640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1e880b46a0 0x7f1e8806c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e877fe640 1 -- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e8815b720 msgr2=0x7f1e8813df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e877fe640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1e8815b720 0x7f1e8813df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:21.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.148+0000 7f1e877fe640 1 -- 172.21.15.81:0/3427484672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e8813d140 con 0x7f1e8806ca10 2024-06-29T15:39:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.149+0000 7f1e877fe640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e8806ca10 0x7f1e8813e440 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000cae0 tx=0x7f1e8000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.149+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80016020 con 0x7f1e8806ca10 2024-06-29T15:39:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.149+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e8000ad30 con 0x7f1e8806ca10 2024-06-29T15:39:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.149+0000 7f1e8d773640 1 -- 172.21.15.81:0/3427484672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e8813d420 con 0x7f1e8806ca10 2024-06-29T15:39:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.149+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e80012680 con 0x7f1e8806ca10 2024-06-29T15:39:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.149+0000 7f1e8d773640 1 -- 172.21.15.81:0/3427484672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e8815a360 con 0x7f1e8806ca10 2024-06-29T15:39:21.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.150+0000 7f1e8d773640 1 -- 172.21.15.81:0/3427484672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e8815b1e0 con 0x7f1e8806ca10 2024-06-29T15:39:21.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.151+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 100) ==== 98320+0+0 (secure 0 0 0) 0x7f1e80019020 con 0x7f1e8806ca10 2024-06-29T15:39:21.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.151+0000 7f1e6f7fe640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 0x7f1e74078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:21.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.151+0000 7f1e86ffd640 1 -- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 msgr2=0x7f1e74078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2546364927 2024-06-29T15:39:21.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.151+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 5809+0+0 (secure 0 0 0) 0x7f1e80097ee0 con 0x7f1e8806ca10 2024-06-29T15:39:21.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.151+0000 7f1e86ffd640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 0x7f1e74078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:39:21.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.156+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1e80020020 con 0x7f1e8806ca10 2024-06-29T15:39:21.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.353+0000 7f1e86ffd640 1 -- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 msgr2=0x7f1e74078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2546364927 2024-06-29T15:39:21.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.353+0000 7f1e86ffd640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 0x7f1e74078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:39:21.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.369+0000 7f1e8d773640 1 -- 172.21.15.81:0/3427484672 --> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f1e8815a690 con 0x7f1e74076440 2024-06-29T15:39:21.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.754+0000 7f1e86ffd640 1 -- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 msgr2=0x7f1e74078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2546364927 2024-06-29T15:39:21.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:21.754+0000 7f1e86ffd640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 0x7f1e74078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:39:22.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:22.555+0000 7f1e86ffd640 1 -- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 msgr2=0x7f1e74078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2546364927 2024-06-29T15:39:22.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:22.555+0000 7f1e86ffd640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 0x7f1e74078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:39:23.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:23.852+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 101) ==== 97929+0+0 (secure 0 0 0) 0x7f1e80096390 con 0x7f1e8806ca10 2024-06-29T15:39:23.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:23.852+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 msgr2=0x7f1e74078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:39:23.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:23.852+0000 7f1e6f7fe640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2546364927,v1:172.21.15.167:6833/2546364927] conn(0x7f1e74076440 0x7f1e74078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:23.997 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:39:23.996+0000 7f0166e7d640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.997 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:39:23.996+0000 7f0166e7d640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.997 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:39:23.996+0000 7f0166e7d640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.997 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:39:23.996+0000 7f0166e7d640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.997 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:39:23.996+0000 7f0166e7d640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.997 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:39:23.996+0000 7f0163e77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.997 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:39:23.996+0000 7f0163e77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.997 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:39:23.996+0000 7f0163e77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.998 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:39:23.996+0000 7f0163e77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:23.998 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:39:23.996+0000 7f0163e77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:39:24.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.856+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 102) ==== 98056+0+0 (secure 0 0 0) 0x7f1e8005d300 con 0x7f1e8806ca10 2024-06-29T15:39:24.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.856+0000 7f1e6f7fe640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1e7403ad30 0x7f1e740804a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:24.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.856+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 --> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f1e80095fb0 con 0x7f1e7403ad30 2024-06-29T15:39:24.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.858+0000 7f1e86ffd640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1e7403ad30 0x7f1e740804a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:24.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.861+0000 7f1e86ffd640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1e7403ad30 0x7f1e740804a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1e78000f60 tx=0x7f1e78007db0 comp rx=0 tx=0).ready entity=mgr.9265 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.924+0000 7f1e6f7fe640 1 -- 172.21.15.81:0/3427484672 <== mgr.9265 v2:172.21.15.167:6832/2288011293 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f1e80095fb0 con 0x7f1e7403ad30 2024-06-29T15:39:24.926 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /snap_test_dir1/test_restart 2024-06-29T15:39:24.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.927+0000 7f1e8d773640 1 -- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1e7403ad30 msgr2=0x7f1e740804a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:24.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.927+0000 7f1e8d773640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1e7403ad30 0x7f1e740804a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1e78000f60 tx=0x7f1e78007db0 comp rx=0 tx=0).stop 2024-06-29T15:39:24.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.927+0000 7f1e8d773640 1 -- 172.21.15.81:0/3427484672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e8806ca10 msgr2=0x7f1e8813e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.927+0000 7f1e8d773640 1 --2- 172.21.15.81:0/3427484672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e8806ca10 0x7f1e8813e440 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000cae0 tx=0x7f1e8000a060 comp rx=0 tx=0).stop 2024-06-29T15:39:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.927+0000 7f1e877fe640 1 -- 172.21.15.81:0/3427484672 reap_dead start 2024-06-29T15:39:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.927+0000 7f1e8d773640 1 -- 172.21.15.81:0/3427484672 shutdown_connections 2024-06-29T15:39:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.927+0000 7f1e8d773640 1 -- 172.21.15.81:0/3427484672 >> 172.21.15.81:0/3427484672 conn(0x7f1e880d0b70 msgr2=0x7f1e8815c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.927+0000 7f1e8d773640 1 -- 172.21.15.81:0/3427484672 shutdown_connections 2024-06-29T15:39:24.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:24.927+0000 7f1e8d773640 1 -- 172.21.15.81:0/3427484672 wait complete. 2024-06-29T15:39:24.941 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2024-06-29T15:39:24.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2024-06-29T15:39:25.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.059+0000 7f1fb5168640 1 -- 172.21.15.81:0/1903948578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f9c003a50 msgr2=0x7f1f9c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:25.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.059+0000 7f1fb5168640 1 --2- 172.21.15.81:0/1903948578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f9c003a50 0x7f1f9c003ec0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009b50 tx=0x7f1f9c005830 comp rx=0 tx=0).stop 2024-06-29T15:39:25.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.059+0000 7f1fb5168640 1 -- 172.21.15.81:0/1903948578 shutdown_connections 2024-06-29T15:39:25.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.059+0000 7f1fb5168640 1 --2- 172.21.15.81:0/1903948578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f9c003a50 0x7f1f9c003ec0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.059+0000 7f1fb5168640 1 -- 172.21.15.81:0/1903948578 >> 172.21.15.81:0/1903948578 conn(0x7f1fb00b3510 msgr2=0x7f1fb00b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:25.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.060+0000 7f1fb5168640 1 -- 172.21.15.81:0/1903948578 shutdown_connections 2024-06-29T15:39:25.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.060+0000 7f1fb5168640 1 -- 172.21.15.81:0/1903948578 wait complete. 2024-06-29T15:39:25.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.061+0000 7f1fb5168640 1 Processor -- start 2024-06-29T15:39:25.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.062+0000 7f1fb5168640 1 -- start start 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1fb5168640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f9c003a50 0x7f1fb01dfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1fb5168640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fb00c8b50 0x7f1fb00c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1fb5168640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fb00c9ac0 0x7f1fb01f83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1fb5168640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb00d0660 con 0x7f1f9c003a50 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1fb5168640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb00d04e0 con 0x7f1fb00c8b50 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1fb5168640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1fb00d07e0 con 0x7f1fb00c9ac0 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1faed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f9c003a50 0x7f1fb01dfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1faed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f9c003a50 0x7f1fb01dfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35878/0 (socket says 172.21.15.81:35878) 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1faed9d640 1 -- 172.21.15.81:0/405516691 learned_addr learned my addr 172.21.15.81:0/405516691 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1faf59e640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fb00c9ac0 0x7f1fb01f83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1fa659c640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fb00c8b50 0x7f1fb00c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1faed9d640 1 -- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fb00c9ac0 msgr2=0x7f1fb01f83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1faed9d640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fb00c9ac0 0x7f1fb01f83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1faed9d640 1 -- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fb00c8b50 msgr2=0x7f1fb00c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1faed9d640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fb00c8b50 0x7f1fb00c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.063+0000 7f1faed9d640 1 -- 172.21.15.81:0/405516691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb00c9ed0 con 0x7f1f9c003a50 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.064+0000 7f1faed9d640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f9c003a50 0x7f1fb01dfdc0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009a70 tx=0x7f1fa00336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.064+0000 7f1fa7fff640 1 -- 172.21.15.81:0/405516691 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0040020 con 0x7f1f9c003a50 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.064+0000 7f1fa7fff640 1 -- 172.21.15.81:0/405516691 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fa0004310 con 0x7f1f9c003a50 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.064+0000 7f1fb5168640 1 -- 172.21.15.81:0/405516691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb00c79f0 con 0x7f1f9c003a50 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.064+0000 7f1fa7fff640 1 -- 172.21.15.81:0/405516691 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0038040 con 0x7f1f9c003a50 2024-06-29T15:39:25.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.064+0000 7f1fb5168640 1 -- 172.21.15.81:0/405516691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb00b0cf0 con 0x7f1f9c003a50 2024-06-29T15:39:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.065+0000 7f1fb5168640 1 -- 172.21.15.81:0/405516691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fb0071110 con 0x7f1f9c003a50 2024-06-29T15:39:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.066+0000 7f1fa7fff640 1 -- 172.21.15.81:0/405516691 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 102) ==== 98056+0+0 (secure 0 0 0) 0x7f1fa002b8a0 con 0x7f1f9c003a50 2024-06-29T15:39:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.066+0000 7f1fa7fff640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1f90076170 0x7f1f90078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:39:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.066+0000 7f1fa7fff640 1 -- 172.21.15.81:0/405516691 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 6153+0+0 (secure 0 0 0) 0x7f1fa00b7cd0 con 0x7f1f9c003a50 2024-06-29T15:39:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.066+0000 7f1fa659c640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1f90076170 0x7f1f90078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:39:25.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.067+0000 7f1fa659c640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1f90076170 0x7f1f90078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f98001d30 tx=0x7f1f980067a0 comp rx=0 tx=0).ready entity=mgr.9265 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:39:25.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.071+0000 7f1fa7fff640 1 -- 172.21.15.81:0/405516691 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1fb0071110 con 0x7f1f9c003a50 2024-06-29T15:39:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.271+0000 7f1fb5168640 1 -- 172.21.15.81:0/405516691 --> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f1fb00b49d0 con 0x7f1f90076170 2024-06-29T15:39:25.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.316+0000 7f1fa7fff640 1 -- 172.21.15.81:0/405516691 <== mgr.9265 v2:172.21.15.167:6832/2288011293 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+57 (secure 0 0 0) 0x7f1fb00b49d0 con 0x7f1f90076170 2024-06-29T15:39:25.317 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path /snap_test_dir1/test_restart 2024-06-29T15:39:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 -- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1f90076170 msgr2=0x7f1f90078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1f90076170 0x7f1f90078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f98001d30 tx=0x7f1f980067a0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 -- 172.21.15.81:0/405516691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f9c003a50 msgr2=0x7f1fb01dfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:39:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f9c003a50 0x7f1fb01dfdc0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009a70 tx=0x7f1fa00336f0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 -- 172.21.15.81:0/405516691 shutdown_connections 2024-06-29T15:39:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f1f90076170 0x7f1f90078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1fb00c9ac0 0x7f1fb01f83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1fb00c8b50 0x7f1fb00c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 --2- 172.21.15.81:0/405516691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f9c003a50 0x7f1fb01dfdc0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:39:25.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 -- 172.21.15.81:0/405516691 >> 172.21.15.81:0/405516691 conn(0x7f1fb00b3510 msgr2=0x7f1fb00b6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:39:25.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 -- 172.21.15.81:0/405516691 shutdown_connections 2024-06-29T15:39:25.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:39:25.318+0000 7f1fb5168640 1 -- 172.21.15.81:0/405516691 wait complete. 2024-06-29T15:39:25.331 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2024-06-29T15:39:28.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:39:28.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:39:58.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:39:58.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:40:00.327 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:40:00.326+0000 7f0155e5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:40:00.328 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:40:00.326+0000 7f0155e5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:40:00.328 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:40:00.326+0000 7f0155e5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:40:00.328 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:40:00.326+0000 7f0155e5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:40:00.328 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:40:00.326+0000 7f0155e5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:40:28.711 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:40:28.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:40:58.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:40:58.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:41:28.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:41:28.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:41:59.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:41:59.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:42:29.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:42:29.413 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:42:59.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:42:59.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:43:29.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:43:29.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:43:59.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:43:59.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:44:30.049 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:44:30.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:45:00.197 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:45:00.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:45:30.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:45:30.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:46:00.475 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:46:00.479 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:46:30.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:46:30.544 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:47:00.693 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:47:00.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:47:30.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:47:30.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:48:00.950 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:48:00.954 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:48:31.091 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:48:31.096 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:49:01.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:49:01.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:49:31.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:49:31.389 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:50:01.561 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:50:01.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:50:16.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:50:16.332 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-06-29T15:50:16.354 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_40_00_UTC 2024-06-29T15:50:16.354 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_41_00_UTC 2024-06-29T15:50:16.354 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_42_00_UTC 2024-06-29T15:50:16.354 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_43_00_UTC 2024-06-29T15:50:16.354 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_44_00_UTC 2024-06-29T15:50:16.354 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_45_00_UTC 2024-06-29T15:50:16.354 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_46_00_UTC 2024-06-29T15:50:16.355 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_47_00_UTC 2024-06-29T15:50:16.355 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_48_00_UTC 2024-06-29T15:50:16.355 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_49_00_UTC 2024-06-29T15:50:16.355 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_50_00_UTC 2024-06-29T15:50:16.356 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-15_40_00_UTC', 'scheduled-2024-06-29-15_41_00_UTC', 'scheduled-2024-06-29-15_42_00_UTC', 'scheduled-2024-06-29-15_43_00_UTC', 'scheduled-2024-06-29-15_44_00_UTC', 'scheduled-2024-06-29-15_45_00_UTC', 'scheduled-2024-06-29-15_46_00_UTC', 'scheduled-2024-06-29-15_47_00_UTC', 'scheduled-2024-06-29-15_48_00_UTC', 'scheduled-2024-06-29-15_49_00_UTC', 'scheduled-2024-06-29-15_50_00_UTC'] 2024-06-29T15:50:16.356 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-06-29T15:50:16.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.524+0000 7f12251bd640 1 -- 172.21.15.81:0/4209429537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f122015e890 msgr2=0x7f122015b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.524+0000 7f12251bd640 1 --2- 172.21.15.81:0/4209429537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f122015e890 0x7f122015b9d0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f12201449e0 tx=0x7f121002aa30 comp rx=0 tx=0).stop 2024-06-29T15:50:16.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.524+0000 7f12251bd640 1 -- 172.21.15.81:0/4209429537 shutdown_connections 2024-06-29T15:50:16.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.524+0000 7f12251bd640 1 --2- 172.21.15.81:0/4209429537 >> v2:172.21.15.167:3300/0 conn(0x7f12200737e0 0x7f1220140600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.524+0000 7f12251bd640 1 --2- 172.21.15.81:0/4209429537 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f122015bf10 0x7f12200732a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.524+0000 7f12251bd640 1 --2- 172.21.15.81:0/4209429537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f122015e890 0x7f122015b9d0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.524+0000 7f12251bd640 1 -- 172.21.15.81:0/4209429537 >> 172.21.15.81:0/4209429537 conn(0x7f12200b1ce0 msgr2=0x7f12200b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:50:16.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.524+0000 7f12251bd640 1 -- 172.21.15.81:0/4209429537 shutdown_connections 2024-06-29T15:50:16.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.524+0000 7f12251bd640 1 -- 172.21.15.81:0/4209429537 wait complete. 2024-06-29T15:50:16.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.526+0000 7f12251bd640 1 Processor -- start 2024-06-29T15:50:16.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.526+0000 7f12251bd640 1 -- start start 2024-06-29T15:50:16.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.527+0000 7f12251bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12200737e0 0x7f122013d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.527+0000 7f12251bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f122015bf10 0x7f12200b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.527+0000 7f12251bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f122015e890 0x7f12200b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.527+0000 7f12251bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1220140fe0 con 0x7f122015bf10 2024-06-29T15:50:16.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.527+0000 7f12251bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1220140e60 con 0x7f12200737e0 2024-06-29T15:50:16.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.527+0000 7f12251bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1220141160 con 0x7f122015e890 2024-06-29T15:50:16.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.527+0000 7f121e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f122015bf10 0x7f12200b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.527+0000 7f121e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f122015bf10 0x7f12200b2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38602/0 (socket says 172.21.15.81:38602) 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.527+0000 7f121e59c640 1 -- 172.21.15.81:0/1110609827 learned_addr learned my addr 172.21.15.81:0/1110609827 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f121f59e640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f122015e890 0x7f12200b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f121e59c640 1 -- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f122015e890 msgr2=0x7f12200b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f121ed9d640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12200737e0 0x7f122013d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f121e59c640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f122015e890 0x7f12200b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f121e59c640 1 -- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12200737e0 msgr2=0x7f122013d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f121e59c640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12200737e0 0x7f122013d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f121e59c640 1 -- 172.21.15.81:0/1110609827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1220159790 con 0x7f122015bf10 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f121f59e640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f122015e890 0x7f12200b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f121e59c640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f122015bf10 0x7f12200b2f00 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f121400d700 tx=0x7f121400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f12077fe640 1 -- 172.21.15.81:0/1110609827 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214017070 con 0x7f122015bf10 2024-06-29T15:50:16.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f12077fe640 1 -- 172.21.15.81:0/1110609827 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1214005310 con 0x7f122015bf10 2024-06-29T15:50:16.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f12251bd640 1 -- 172.21.15.81:0/1110609827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1220159a70 con 0x7f122015bf10 2024-06-29T15:50:16.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f12077fe640 1 -- 172.21.15.81:0/1110609827 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214009820 con 0x7f122015bf10 2024-06-29T15:50:16.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.528+0000 7f12251bd640 1 -- 172.21.15.81:0/1110609827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12200ca850 con 0x7f122015bf10 2024-06-29T15:50:16.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.529+0000 7f12251bd640 1 -- 172.21.15.81:0/1110609827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f122015b940 con 0x7f122015bf10 2024-06-29T15:50:16.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.530+0000 7f12077fe640 1 -- 172.21.15.81:0/1110609827 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 106) ==== 98332+0+0 (secure 0 0 0) 0x7f12140099c0 con 0x7f122015bf10 2024-06-29T15:50:16.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.530+0000 7f12077fe640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f120c076470 0x7f120c078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.531+0000 7f12077fe640 1 -- 172.21.15.81:0/1110609827 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 6153+0+0 (secure 0 0 0) 0x7f1214097c60 con 0x7f122015bf10 2024-06-29T15:50:16.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.531+0000 7f121ed9d640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f120c076470 0x7f120c078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.531+0000 7f121ed9d640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f120c076470 0x7f120c078980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1210035000 tx=0x7f1210004ab0 comp rx=0 tx=0).ready entity=mgr.9265 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:50:16.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.535+0000 7f12077fe640 1 -- 172.21.15.81:0/1110609827 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f122015b940 con 0x7f122015bf10 2024-06-29T15:50:16.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.732+0000 7f12251bd640 1 -- 172.21.15.81:0/1110609827 --> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1220159520 con 0x7f120c076470 2024-06-29T15:50:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.741+0000 7f12077fe640 1 -- 172.21.15.81:0/1110609827 <== mgr.9265 v2:172.21.15.167:6832/2288011293 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+375 (secure 0 0 0) 0x7f1220159520 con 0x7f120c076470 2024-06-29T15:50:16.742 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:50:16.743 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:39:24", "first": "2024-06-29T15:40:00", "last": "2024-06-29T15:50:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-06-29T15:50:16.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 -- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f120c076470 msgr2=0x7f120c078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f120c076470 0x7f120c078980 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1210035000 tx=0x7f1210004ab0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 -- 172.21.15.81:0/1110609827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f122015bf10 msgr2=0x7f12200b2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f122015bf10 0x7f12200b2f00 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f121400d700 tx=0x7f121400dbc0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 -- 172.21.15.81:0/1110609827 shutdown_connections 2024-06-29T15:50:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f120c076470 0x7f120c078980 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f122015e890 0x7f12200b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f122015bf10 0x7f12200b2f00 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 --2- 172.21.15.81:0/1110609827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12200737e0 0x7f122013d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 -- 172.21.15.81:0/1110609827 >> 172.21.15.81:0/1110609827 conn(0x7f12200b1ce0 msgr2=0x7f12201543d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:50:16.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 -- 172.21.15.81:0/1110609827 shutdown_connections 2024-06-29T15:50:16.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.743+0000 7f12251bd640 1 -- 172.21.15.81:0/1110609827 wait complete. 2024-06-29T15:50:16.755 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:39:24", "first": "2024-06-29T15:40:00", "last": "2024-06-29T15:50:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2024-06-29T15:50:16.756 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T15:39:24', 'first': '2024-06-29T15:40:00', 'last': '2024-06-29T15:50:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2024-06-29T15:50:16.756 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2024-06-29T15:50:16.756 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2024-06-29T15:50:16.756 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-06-29T15:50:16.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.869+0000 7f496c5c1640 1 -- 172.21.15.81:0/1828095507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49640b35b0 msgr2=0x7f496415b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.869+0000 7f496c5c1640 1 --2- 172.21.15.81:0/1828095507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49640b35b0 0x7f496415b5b0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f49640cc260 tx=0x7f496002a980 comp rx=0 tx=0).stop 2024-06-29T15:50:16.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.869+0000 7f496c5c1640 1 -- 172.21.15.81:0/1828095507 shutdown_connections 2024-06-29T15:50:16.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.869+0000 7f496c5c1640 1 --2- 172.21.15.81:0/1828095507 >> v2:172.21.15.167:3300/0 conn(0x7f49640b4430 0x7f49640cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.869+0000 7f496c5c1640 1 --2- 172.21.15.81:0/1828095507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49640b35b0 0x7f496415b5b0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.869+0000 7f496c5c1640 1 -- 172.21.15.81:0/1828095507 >> 172.21.15.81:0/1828095507 conn(0x7f496406ccc0 msgr2=0x7f49640b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:50:16.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.869+0000 7f496c5c1640 1 -- 172.21.15.81:0/1828095507 shutdown_connections 2024-06-29T15:50:16.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.869+0000 7f496c5c1640 1 -- 172.21.15.81:0/1828095507 wait complete. 2024-06-29T15:50:16.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.871+0000 7f496c5c1640 1 Processor -- start 2024-06-29T15:50:16.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.871+0000 7f496c5c1640 1 -- start start 2024-06-29T15:50:16.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.872+0000 7f496c5c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496406f660 0x7f49640ad810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.872+0000 7f496c5c1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49640b35b0 0x7f4964144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.872+0000 7f496c5c1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f49640b4430 0x7f4964159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496c5c1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f496406f1e0 con 0x7f496406f660 2024-06-29T15:50:16.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496c5c1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f496406f060 con 0x7f49640b35b0 2024-06-29T15:50:16.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496c5c1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f496406f360 con 0x7f49640b4430 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496a35d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496406f660 0x7f49640ad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496a35d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496406f660 0x7f49640ad810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38620/0 (socket says 172.21.15.81:38620) 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496a35d640 1 -- 172.21.15.81:0/2750476375 learned_addr learned my addr 172.21.15.81:0/2750476375 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f4969b5c640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49640b35b0 0x7f4964144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496ab5e640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f49640b4430 0x7f4964159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496a35d640 1 -- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f49640b4430 msgr2=0x7f4964159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496a35d640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f49640b4430 0x7f4964159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496a35d640 1 -- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49640b35b0 msgr2=0x7f4964144d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496a35d640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49640b35b0 0x7f4964144d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496a35d640 1 -- 172.21.15.81:0/2750476375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49641f95b0 con 0x7f496406f660 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496a35d640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496406f660 0x7f49640ad810 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f4960009d60 tx=0x7f4960003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f495affd640 1 -- 172.21.15.81:0/2750476375 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960034040 con 0x7f496406f660 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f495affd640 1 -- 172.21.15.81:0/2750476375 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4960004600 con 0x7f496406f660 2024-06-29T15:50:16.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f495affd640 1 -- 172.21.15.81:0/2750476375 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960045480 con 0x7f496406f660 2024-06-29T15:50:16.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496c5c1640 1 -- 172.21.15.81:0/2750476375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4964144240 con 0x7f496406f660 2024-06-29T15:50:16.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.873+0000 7f496c5c1640 1 -- 172.21.15.81:0/2750476375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4964144800 con 0x7f496406f660 2024-06-29T15:50:16.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.874+0000 7f496c5c1640 1 -- 172.21.15.81:0/2750476375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f496415b5b0 con 0x7f496406f660 2024-06-29T15:50:16.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.875+0000 7f495affd640 1 -- 172.21.15.81:0/2750476375 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 106) ==== 98332+0+0 (secure 0 0 0) 0x7f4960037020 con 0x7f496406f660 2024-06-29T15:50:16.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.876+0000 7f495affd640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f4948076440 0x7f4948078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:50:16.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.876+0000 7f495affd640 1 -- 172.21.15.81:0/2750476375 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 6153+0+0 (secure 0 0 0) 0x7f49600b9330 con 0x7f496406f660 2024-06-29T15:50:16.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.876+0000 7f4969b5c640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f4948076440 0x7f4948078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:50:16.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.876+0000 7f4969b5c640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f4948076440 0x7f4948078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f49640af450 tx=0x7f495400b2e0 comp rx=0 tx=0).ready entity=mgr.9265 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:50:16.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:16.880+0000 7f495affd640 1 -- 172.21.15.81:0/2750476375 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4960082380 con 0x7f496406f660 2024-06-29T15:50:17.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.107+0000 7f496c5c1640 1 -- 172.21.15.81:0/2750476375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f49640ce170 con 0x7f496406f660 2024-06-29T15:50:17.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.110+0000 7f495affd640 1 -- 172.21.15.81:0/2750476375 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v106) ==== 74+0+189738 (secure 0 0 0) 0x7f4960086680 con 0x7f496406f660 2024-06-29T15:50:17.113 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:50:17.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 -- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f4948076440 msgr2=0x7f4948078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:17.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f4948076440 0x7f4948078950 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f49640af450 tx=0x7f495400b2e0 comp rx=0 tx=0).stop 2024-06-29T15:50:17.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 -- 172.21.15.81:0/2750476375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496406f660 msgr2=0x7f49640ad810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:50:17.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496406f660 0x7f49640ad810 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f4960009d60 tx=0x7f4960003f90 comp rx=0 tx=0).stop 2024-06-29T15:50:17.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 -- 172.21.15.81:0/2750476375 shutdown_connections 2024-06-29T15:50:17.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:6832/2288011293,v1:172.21.15.167:6833/2288011293] conn(0x7f4948076440 0x7f4948078950 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:17.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f49640b4430 0x7f4964159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:17.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f49640b35b0 0x7f4964144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:17.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 --2- 172.21.15.81:0/2750476375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496406f660 0x7f49640ad810 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:50:17.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 -- 172.21.15.81:0/2750476375 >> 172.21.15.81:0/2750476375 conn(0x7f496406ccc0 msgr2=0x7f49640ce690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:50:17.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 -- 172.21.15.81:0/2750476375 shutdown_connections 2024-06-29T15:50:17.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:50:17.113+0000 7f496c5c1640 1 -- 172.21.15.81:0/2750476375 wait complete. 2024-06-29T15:50:17.124 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":106,"flags":0,"active_gid":9265,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6832","nonce":2288011293},{"type":"v1","addr":"172.21.15.167:6833","nonce":2288011293}]},"active_addr":"172.21.15.167:6833/2288011293","active_change":"2024-06-29T15:39:23.843955+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":10338,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":182,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":1637259453}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3737424986}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":578369376}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":1870993879}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3941545005}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3006573425}]}]} 2024-06-29T15:50:17.178 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: x 2024-06-29T15:50:17.178 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-06-29T15:50:17.178 INFO:tasks.ceph.mgr.x:Stopping old one... 2024-06-29T15:50:17.179 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-06-29T15:50:17.179 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:50:17.219 INFO:tasks.ceph.mgr.x:Stopped 2024-06-29T15:50:17.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-06-29T15:50:17.262 INFO:tasks.ceph.mgr.x:Started 2024-06-29T15:50:21.060 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:50:21.059+0000 7f3af2f49640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:21.060 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:50:21.059+0000 7f3af2f49640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:21.060 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:50:21.059+0000 7f3af2f49640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:21.061 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:50:21.059+0000 7f3af2f49640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:21.061 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:50:21.059+0000 7f3af2f49640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:21.061 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:50:21.059+0000 7f3af5f4f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:21.061 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:50:21.059+0000 7f3af5f4f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:21.061 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:50:21.059+0000 7f3af5f4f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:21.061 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:50:21.059+0000 7f3af5f4f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:21.061 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:50:21.059+0000 7f3af5f4f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:50:31.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:50:31.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:51:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:51:00.056+0000 7f3afdf5f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:51:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:51:00.056+0000 7f3afdf5f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:51:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:51:00.056+0000 7f3afdf5f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:51:00.058 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:51:00.056+0000 7f3afdf5f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:51:00.059 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:51:00.056+0000 7f3afdf5f640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:51:01.831 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:51:01.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:51:31.979 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:51:31.983 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:52:02.067 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:52:02.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:52:32.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:52:32.213 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:53:02.338 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:53:02.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:53:32.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:53:32.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:54:02.565 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:54:02.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:54:32.663 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:54:32.667 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:55:02.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:55:02.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:55:17.263 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2024-06-29T15:55:17.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.393+0000 7f268974a640 1 -- 172.21.15.81:0/739595786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26840b5250 msgr2=0x7f26840d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.393+0000 7f268974a640 1 --2- 172.21.15.81:0/739595786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26840b5250 0x7f26840d09c0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f2678005fd0 tx=0x7f267802a5c0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.393+0000 7f268974a640 1 -- 172.21.15.81:0/739595786 shutdown_connections 2024-06-29T15:55:17.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.393+0000 7f268974a640 1 --2- 172.21.15.81:0/739595786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26840b5250 0x7f26840d09c0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.393+0000 7f268974a640 1 --2- 172.21.15.81:0/739595786 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26840b9040 0x7f26840d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.393+0000 7f268974a640 1 -- 172.21.15.81:0/739595786 >> 172.21.15.81:0/739595786 conn(0x7f268409cf30 msgr2=0x7f26840d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:17.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.393+0000 7f268974a640 1 -- 172.21.15.81:0/739595786 shutdown_connections 2024-06-29T15:55:17.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.393+0000 7f268974a640 1 -- 172.21.15.81:0/739595786 wait complete. 2024-06-29T15:55:17.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.395+0000 7f268974a640 1 Processor -- start 2024-06-29T15:55:17.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.395+0000 7f268974a640 1 -- start start 2024-06-29T15:55:17.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.396+0000 7f268974a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26840b5250 0x7f268414e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.396+0000 7f268974a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26840b7290 0x7f2684150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.396+0000 7f268974a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26840b9040 0x7f26841505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.396+0000 7f268974a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f26840b7db0 con 0x7f26840b5250 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.396+0000 7f268974a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f26840d2d50 con 0x7f26840b7290 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.396+0000 7f268974a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f26840b7f30 con 0x7f26840b9040 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.396+0000 7f2682ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26840b5250 0x7f268414e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.396+0000 7f2682ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26840b5250 0x7f268414e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54114/0 (socket says 172.21.15.81:54114) 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.396+0000 7f2682ffd640 1 -- 172.21.15.81:0/1949491851 learned_addr learned my addr 172.21.15.81:0/1949491851 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f26827fc640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26840b7290 0x7f2684150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f26837fe640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26840b9040 0x7f26841505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f2682ffd640 1 -- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26840b9040 msgr2=0x7f26841505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f2682ffd640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26840b9040 0x7f26841505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f2682ffd640 1 -- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26840b7290 msgr2=0x7f2684150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f2682ffd640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26840b7290 0x7f2684150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f2682ffd640 1 -- 172.21.15.81:0/1949491851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268414ede0 con 0x7f26840b5250 2024-06-29T15:55:17.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f26827fc640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26840b7290 0x7f2684150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:17.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f26837fe640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26840b9040 0x7f26841505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:17.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f2682ffd640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26840b5250 0x7f268414e070 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f267400c9a0 tx=0x7f267400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:17.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f266b7fe640 1 -- 172.21.15.81:0/1949491851 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2674007b50 con 0x7f26840b5250 2024-06-29T15:55:17.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f266b7fe640 1 -- 172.21.15.81:0/1949491851 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2674007e00 con 0x7f26840b5250 2024-06-29T15:55:17.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f268974a640 1 -- 172.21.15.81:0/1949491851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268414f0c0 con 0x7f26840b5250 2024-06-29T15:55:17.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f266b7fe640 1 -- 172.21.15.81:0/1949491851 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2674005280 con 0x7f26840b5250 2024-06-29T15:55:17.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.397+0000 7f268974a640 1 -- 172.21.15.81:0/1949491851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268414f680 con 0x7f26840b5250 2024-06-29T15:55:17.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.398+0000 7f268974a640 1 -- 172.21.15.81:0/1949491851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26840b84c0 con 0x7f26840b5250 2024-06-29T15:55:17.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.399+0000 7f266b7fe640 1 -- 172.21.15.81:0/1949491851 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 112) ==== 98332+0+0 (secure 0 0 0) 0x7f2674005420 con 0x7f26840b5250 2024-06-29T15:55:17.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.399+0000 7f266b7fe640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f2670076440 0x7f2670078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.400+0000 7f266b7fe640 1 -- 172.21.15.81:0/1949491851 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7f26740981f0 con 0x7f26840b5250 2024-06-29T15:55:17.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.400+0000 7f26827fc640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f2670076440 0x7f2670078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.400+0000 7f26827fc640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f2670076440 0x7f2670078950 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2678033000 tx=0x7f2678004730 comp rx=0 tx=0).ready entity=mgr.8261 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:17.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.404+0000 7f266b7fe640 1 -- 172.21.15.81:0/1949491851 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2674014030 con 0x7f26840b5250 2024-06-29T15:55:17.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.607+0000 7f268974a640 1 -- 172.21.15.81:0/1949491851 --> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f268414dd40 con 0x7f2670076440 2024-06-29T15:55:17.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.675+0000 7f266b7fe640 1 -- 172.21.15.81:0/1949491851 <== mgr.8261 v2:172.21.15.167:6832/1431883924 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f2678004050 con 0x7f2670076440 2024-06-29T15:55:17.676 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2024-06-29T15:55:17.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 -- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f2670076440 msgr2=0x7f2670078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f2670076440 0x7f2670078950 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2678033000 tx=0x7f2678004730 comp rx=0 tx=0).stop 2024-06-29T15:55:17.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 -- 172.21.15.81:0/1949491851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26840b5250 msgr2=0x7f268414e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26840b5250 0x7f268414e070 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f267400c9a0 tx=0x7f267400ce60 comp rx=0 tx=0).stop 2024-06-29T15:55:17.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 -- 172.21.15.81:0/1949491851 shutdown_connections 2024-06-29T15:55:17.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f2670076440 0x7f2670078950 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26840b9040 0x7f26841505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26840b7290 0x7f2684150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 --2- 172.21.15.81:0/1949491851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26840b5250 0x7f268414e070 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 -- 172.21.15.81:0/1949491851 >> 172.21.15.81:0/1949491851 conn(0x7f268409cf30 msgr2=0x7f268406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:17.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 -- 172.21.15.81:0/1949491851 shutdown_connections 2024-06-29T15:55:17.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.677+0000 7f268974a640 1 -- 172.21.15.81:0/1949491851 wait complete. 2024-06-29T15:55:17.689 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2024-06-29T15:55:17.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:17.689 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-06-29T15:55:17.705 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_40_00_UTC 2024-06-29T15:55:17.705 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_41_00_UTC 2024-06-29T15:55:17.705 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_42_00_UTC 2024-06-29T15:55:17.705 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_43_00_UTC 2024-06-29T15:55:17.705 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_44_00_UTC 2024-06-29T15:55:17.705 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_45_00_UTC 2024-06-29T15:55:17.705 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_46_00_UTC 2024-06-29T15:55:17.705 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_47_00_UTC 2024-06-29T15:55:17.706 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_48_00_UTC 2024-06-29T15:55:17.706 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_49_00_UTC 2024-06-29T15:55:17.706 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_50_00_UTC 2024-06-29T15:55:17.706 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_51_00_UTC 2024-06-29T15:55:17.706 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_52_00_UTC 2024-06-29T15:55:17.706 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_53_00_UTC 2024-06-29T15:55:17.706 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_54_00_UTC 2024-06-29T15:55:17.706 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_55_00_UTC 2024-06-29T15:55:17.706 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2024-06-29-15_40_00_UTC', 'scheduled-2024-06-29-15_41_00_UTC', 'scheduled-2024-06-29-15_42_00_UTC', 'scheduled-2024-06-29-15_43_00_UTC', 'scheduled-2024-06-29-15_44_00_UTC', 'scheduled-2024-06-29-15_45_00_UTC', 'scheduled-2024-06-29-15_46_00_UTC', 'scheduled-2024-06-29-15_47_00_UTC', 'scheduled-2024-06-29-15_48_00_UTC', 'scheduled-2024-06-29-15_49_00_UTC', 'scheduled-2024-06-29-15_50_00_UTC', 'scheduled-2024-06-29-15_51_00_UTC', 'scheduled-2024-06-29-15_52_00_UTC', 'scheduled-2024-06-29-15_53_00_UTC', 'scheduled-2024-06-29-15_54_00_UTC', 'scheduled-2024-06-29-15_55_00_UTC'] 2024-06-29T15:55:17.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2024-06-29T15:55:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.860+0000 7ff6a73eb640 1 -- 172.21.15.81:0/1627728680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff68c005e50 msgr2=0x7ff68c006300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.860+0000 7ff6a73eb640 1 --2- 172.21.15.81:0/1627728680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff68c005e50 0x7ff68c006300 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00cc280 tx=0x7ff68c006940 comp rx=0 tx=0).stop 2024-06-29T15:55:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.860+0000 7ff6a5988640 1 -- 172.21.15.81:0/1627728680 reap_dead start 2024-06-29T15:55:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.860+0000 7ff6a73eb640 1 -- 172.21.15.81:0/1627728680 shutdown_connections 2024-06-29T15:55:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.860+0000 7ff6a73eb640 1 -- 172.21.15.81:0/1627728680 >> 172.21.15.81:0/1627728680 conn(0x7ff6a006ccc0 msgr2=0x7ff6a00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.860+0000 7ff6a73eb640 1 -- 172.21.15.81:0/1627728680 shutdown_connections 2024-06-29T15:55:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.860+0000 7ff6a73eb640 1 -- 172.21.15.81:0/1627728680 wait complete. 2024-06-29T15:55:17.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.862+0000 7ff6a73eb640 1 Processor -- start 2024-06-29T15:55:17.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.862+0000 7ff6a73eb640 1 -- start start 2024-06-29T15:55:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.863+0000 7ff6a73eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff68c005e50 0x7ff6a0153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.863+0000 7ff6a73eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6a0156670 0x7ff6a0158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.863+0000 7ff6a73eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6a01587c0 0x7ff6a01550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.863+0000 7ff6a73eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a006ef50 con 0x7ff6a0156670 2024-06-29T15:55:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.863+0000 7ff6a73eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a006edd0 con 0x7ff68c005e50 2024-06-29T15:55:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.863+0000 7ff6a73eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a006f0d0 con 0x7ff6a01587c0 2024-06-29T15:55:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.863+0000 7ff6a4986640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6a0156670 0x7ff6a0158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.863+0000 7ff6a4986640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6a0156670 0x7ff6a0158280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54126/0 (socket says 172.21.15.81:54126) 2024-06-29T15:55:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.863+0000 7ff6a4986640 1 -- 172.21.15.81:0/3072362278 learned_addr learned my addr 172.21.15.81:0/3072362278 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff6a5187640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff68c005e50 0x7ff6a0153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff6a4986640 1 -- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6a01587c0 msgr2=0x7ff6a01550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff6a5988640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6a01587c0 0x7ff6a01550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff6a4986640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6a01587c0 0x7ff6a01550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff6a4986640 1 -- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff68c005e50 msgr2=0x7ff6a0153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff6a4986640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff68c005e50 0x7ff6a0153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff6a4986640 1 -- 172.21.15.81:0/3072362278 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a0152060 con 0x7ff6a0156670 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff6a4986640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6a0156670 0x7ff6a0158280 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7ff690004e20 tx=0x7ff690001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff695ffb640 1 -- 172.21.15.81:0/3072362278 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff690015020 con 0x7ff6a0156670 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff695ffb640 1 -- 172.21.15.81:0/3072362278 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff690011960 con 0x7ff6a0156670 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff6a73eb640 1 -- 172.21.15.81:0/3072362278 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a0152340 con 0x7ff6a0156670 2024-06-29T15:55:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff695ffb640 1 -- 172.21.15.81:0/3072362278 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff690011d70 con 0x7ff6a0156670 2024-06-29T15:55:17.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.864+0000 7ff6a73eb640 1 -- 172.21.15.81:0/3072362278 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a0152900 con 0x7ff6a0156670 2024-06-29T15:55:17.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.865+0000 7ff6a73eb640 1 -- 172.21.15.81:0/3072362278 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a00cf130 con 0x7ff6a0156670 2024-06-29T15:55:17.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.866+0000 7ff695ffb640 1 -- 172.21.15.81:0/3072362278 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 112) ==== 98332+0+0 (secure 0 0 0) 0x7ff690018020 con 0x7ff6a0156670 2024-06-29T15:55:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.867+0000 7ff695ffb640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ff684076470 0x7ff684078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.867+0000 7ff695ffb640 1 -- 172.21.15.81:0/3072362278 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7ff6900986b0 con 0x7ff6a0156670 2024-06-29T15:55:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.867+0000 7ff6a5187640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ff684076470 0x7ff684078980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.867+0000 7ff6a5187640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ff684076470 0x7ff684078980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009a90 tx=0x7ff69c005f90 comp rx=0 tx=0).ready entity=mgr.8261 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:17.871+0000 7ff695ffb640 1 -- 172.21.15.81:0/3072362278 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff69001f020 con 0x7ff6a0156670 2024-06-29T15:55:18.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.069+0000 7ff6a73eb640 1 -- 172.21.15.81:0/3072362278 --> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff6a00b0be0 con 0x7ff684076470 2024-06-29T15:55:18.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.077+0000 7ff695ffb640 1 -- 172.21.15.81:0/3072362278 <== mgr.8261 v2:172.21.15.167:6832/1431883924 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+376 (secure 0 0 0) 0x7ff6a00b0be0 con 0x7ff684076470 2024-06-29T15:55:18.077 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:18.077 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:39:24", "first": "2024-06-29T15:40:00", "last": "2024-06-29T15:55:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-06-29T15:55:18.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 -- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ff684076470 msgr2=0x7ff684078980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ff684076470 0x7ff684078980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009a90 tx=0x7ff69c005f90 comp rx=0 tx=0).stop 2024-06-29T15:55:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 -- 172.21.15.81:0/3072362278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6a0156670 msgr2=0x7ff6a0158280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6a0156670 0x7ff6a0158280 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7ff690004e20 tx=0x7ff690001630 comp rx=0 tx=0).stop 2024-06-29T15:55:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 -- 172.21.15.81:0/3072362278 shutdown_connections 2024-06-29T15:55:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ff684076470 0x7ff684078980 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff6a01587c0 0x7ff6a01550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6a0156670 0x7ff6a0158280 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 --2- 172.21.15.81:0/3072362278 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff68c005e50 0x7ff6a0153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 -- 172.21.15.81:0/3072362278 >> 172.21.15.81:0/3072362278 conn(0x7ff6a006ccc0 msgr2=0x7ff6a00b1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 -- 172.21.15.81:0/3072362278 shutdown_connections 2024-06-29T15:55:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.079+0000 7ff6a73eb640 1 -- 172.21.15.81:0/3072362278 wait complete. 2024-06-29T15:55:18.090 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:39:24", "first": "2024-06-29T15:40:00", "last": "2024-06-29T15:55:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2024-06-29T15:55:18.090 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2024-06-29T00:00:00', 'created': '2024-06-29T15:39:24', 'first': '2024-06-29T15:40:00', 'last': '2024-06-29T15:55:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2024-06-29T15:55:18.090 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2024-06-29T15:55:18.091 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2024-06-29T15:55:18.091 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2024-06-29T15:55:18.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.207+0000 7ffa5a503640 1 -- 172.21.15.81:0/3831013267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa540cd540 msgr2=0x7ffa540d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.207+0000 7ffa5a503640 1 --2- 172.21.15.81:0/3831013267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa540cd540 0x7ffa540d00f0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7ffa540b2c10 tx=0x7ffa4c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.207+0000 7ffa5a503640 1 -- 172.21.15.81:0/3831013267 shutdown_connections 2024-06-29T15:55:18.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.207+0000 7ffa5a503640 1 --2- 172.21.15.81:0/3831013267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa540cd540 0x7ffa540d00f0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.207+0000 7ffa5a503640 1 --2- 172.21.15.81:0/3831013267 >> v2:172.21.15.167:3300/0 conn(0x7ffa540b6d60 0x7ffa540ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.207+0000 7ffa5a503640 1 -- 172.21.15.81:0/3831013267 >> 172.21.15.81:0/3831013267 conn(0x7ffa540d1960 msgr2=0x7ffa540d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:18.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.207+0000 7ffa5a503640 1 -- 172.21.15.81:0/3831013267 shutdown_connections 2024-06-29T15:55:18.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.207+0000 7ffa5a503640 1 -- 172.21.15.81:0/3831013267 wait complete. 2024-06-29T15:55:18.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.209+0000 7ffa5a503640 1 Processor -- start 2024-06-29T15:55:18.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.209+0000 7ffa5a503640 1 -- start start 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.210+0000 7ffa5a503640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa540b6d60 0x7ffa541e4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.210+0000 7ffa5a503640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa540cd540 0x7ffa540cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.210+0000 7ffa5a503640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa540cea10 0x7ffa540c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.210+0000 7ffa5a503640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffa540d0630 con 0x7ffa540cea10 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.210+0000 7ffa5a503640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ffa540b2790 con 0x7ffa540b6d60 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.210+0000 7ffa5a503640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ffa540d07b0 con 0x7ffa540cd540 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.210+0000 7ffa58aa0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa540cea10 0x7ffa540c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.210+0000 7ffa58aa0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa540cea10 0x7ffa540c89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54144/0 (socket says 172.21.15.81:54144) 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.210+0000 7ffa58aa0640 1 -- 172.21.15.81:0/732568715 learned_addr learned my addr 172.21.15.81:0/732568715 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa58aa0640 1 -- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa540cd540 msgr2=0x7ffa540cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa58aa0640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa540cd540 0x7ffa540cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa58aa0640 1 -- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa540b6d60 msgr2=0x7ffa541e4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa58aa0640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa540b6d60 0x7ffa541e4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa58aa0640 1 -- 172.21.15.81:0/732568715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa541e80e0 con 0x7ffa540cea10 2024-06-29T15:55:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa58aa0640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa540cea10 0x7ffa540c89e0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c02aea0 tx=0x7ffa4c0039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa50ff9640 1 -- 172.21.15.81:0/732568715 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c031990 con 0x7ffa540cea10 2024-06-29T15:55:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa50ff9640 1 -- 172.21.15.81:0/732568715 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa4c031c20 con 0x7ffa540cea10 2024-06-29T15:55:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa5a503640 1 -- 172.21.15.81:0/732568715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa541e83c0 con 0x7ffa540cea10 2024-06-29T15:55:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa50ff9640 1 -- 172.21.15.81:0/732568715 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c0330d0 con 0x7ffa540cea10 2024-06-29T15:55:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.211+0000 7ffa5a503640 1 -- 172.21.15.81:0/732568715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa540cbe20 con 0x7ffa540cea10 2024-06-29T15:55:18.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.212+0000 7ffa5a503640 1 -- 172.21.15.81:0/732568715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa541e88e0 con 0x7ffa540cea10 2024-06-29T15:55:18.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.213+0000 7ffa50ff9640 1 -- 172.21.15.81:0/732568715 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 112) ==== 98332+0+0 (secure 0 0 0) 0x7ffa4c033270 con 0x7ffa540cea10 2024-06-29T15:55:18.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.213+0000 7ffa50ff9640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ffa40076440 0x7ffa40078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:18.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.213+0000 7ffa50ff9640 1 -- 172.21.15.81:0/732568715 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7ffa4c0b9330 con 0x7ffa540cea10 2024-06-29T15:55:18.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.214+0000 7ffa53fff640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ffa40076440 0x7ffa40078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:18.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.214+0000 7ffa53fff640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ffa40076440 0x7ffa40078950 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffa540c5220 tx=0x7ffa44005b70 comp rx=0 tx=0).ready entity=mgr.8261 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:18.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.218+0000 7ffa50ff9640 1 -- 172.21.15.81:0/732568715 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa541e88e0 con 0x7ffa540cea10 2024-06-29T15:55:18.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.418+0000 7ffa5a503640 1 -- 172.21.15.81:0/732568715 --> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7ffa540ce4d0 con 0x7ffa40076440 2024-06-29T15:55:18.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.492+0000 7ffa50ff9640 1 -- 172.21.15.81:0/732568715 <== mgr.8261 v2:172.21.15.167:6832/1431883924 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+55 (secure 0 0 0) 0x7ffa540ce4d0 con 0x7ffa40076440 2024-06-29T15:55:18.492 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /snap_test_dir1/test_restart 2024-06-29T15:55:18.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 -- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ffa40076440 msgr2=0x7ffa40078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ffa40076440 0x7ffa40078950 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffa540c5220 tx=0x7ffa44005b70 comp rx=0 tx=0).stop 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 -- 172.21.15.81:0/732568715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa540cea10 msgr2=0x7ffa540c89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa540cea10 0x7ffa540c89e0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c02aea0 tx=0x7ffa4c0039f0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 -- 172.21.15.81:0/732568715 shutdown_connections 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7ffa40076440 0x7ffa40078950 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa540cea10 0x7ffa540c89e0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa540cd540 0x7ffa540cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 --2- 172.21.15.81:0/732568715 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa540b6d60 0x7ffa541e4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 -- 172.21.15.81:0/732568715 >> 172.21.15.81:0/732568715 conn(0x7ffa540d1960 msgr2=0x7ffa54071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 -- 172.21.15.81:0/732568715 shutdown_connections 2024-06-29T15:55:18.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:18.494+0000 7ffa5a503640 1 -- 172.21.15.81:0/732568715 wait complete. 2024-06-29T15:55:18.505 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2024-06-29T15:55:18.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.506 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_40_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_41_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_42_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_43_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_44_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_45_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_46_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_47_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_48_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_49_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_50_00_UTC 2024-06-29T15:55:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_51_00_UTC 2024-06-29T15:55:18.524 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_52_00_UTC 2024-06-29T15:55:18.524 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_53_00_UTC 2024-06-29T15:55:18.524 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_54_00_UTC 2024-06-29T15:55:18.524 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_55_00_UTC 2024-06-29T15:55:18.524 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_40_00_UTC 2024-06-29T15:55:18.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.524 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_40_00_UTC) 2024-06-29T15:55:18.584 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_41_00_UTC 2024-06-29T15:55:18.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.584 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_41_00_UTC) 2024-06-29T15:55:18.644 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_42_00_UTC 2024-06-29T15:55:18.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.644 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_42_00_UTC) 2024-06-29T15:55:18.704 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_43_00_UTC 2024-06-29T15:55:18.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.704 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_43_00_UTC) 2024-06-29T15:55:18.764 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_44_00_UTC 2024-06-29T15:55:18.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.765 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_44_00_UTC) 2024-06-29T15:55:18.825 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_45_00_UTC 2024-06-29T15:55:18.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.826 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_45_00_UTC) 2024-06-29T15:55:18.885 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_46_00_UTC 2024-06-29T15:55:18.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.885 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_46_00_UTC) 2024-06-29T15:55:18.945 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_47_00_UTC 2024-06-29T15:55:18.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:18.945 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_47_00_UTC) 2024-06-29T15:55:19.005 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_48_00_UTC 2024-06-29T15:55:19.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.005 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_48_00_UTC) 2024-06-29T15:55:19.066 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_49_00_UTC 2024-06-29T15:55:19.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.066 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_49_00_UTC) 2024-06-29T15:55:19.126 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_50_00_UTC 2024-06-29T15:55:19.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.126 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_50_00_UTC) 2024-06-29T15:55:19.186 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_51_00_UTC 2024-06-29T15:55:19.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.187 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_51_00_UTC) 2024-06-29T15:55:19.247 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_52_00_UTC 2024-06-29T15:55:19.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.247 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_52_00_UTC) 2024-06-29T15:55:19.308 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_53_00_UTC 2024-06-29T15:55:19.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.308 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_53_00_UTC) 2024-06-29T15:55:19.369 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_54_00_UTC 2024-06-29T15:55:19.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.369 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_54_00_UTC) 2024-06-29T15:55:19.429 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_55_00_UTC 2024-06-29T15:55:19.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.430 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2024-06-29-15_55_00_UTC) 2024-06-29T15:55:19.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:19.491 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2024-06-29T15:55:19.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:55:19.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.703+0000 7f0f96b7f640 1 -- 172.21.15.81:0/939379709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f901412f0 msgr2=0x7f0f900cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:19.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.703+0000 7f0f96b7f640 1 --2- 172.21.15.81:0/939379709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f901412f0 0x7f0f900cef90 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f0f9006ff00 tx=0x7f0f8402a560 comp rx=0 tx=0).stop 2024-06-29T15:55:19.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.703+0000 7f0f96b7f640 1 -- 172.21.15.81:0/939379709 shutdown_connections 2024-06-29T15:55:19.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.703+0000 7f0f96b7f640 1 --2- 172.21.15.81:0/939379709 >> v2:172.21.15.167:3300/0 conn(0x7f0f90073570 0x7f0f9015e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.703+0000 7f0f96b7f640 1 --2- 172.21.15.81:0/939379709 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f900cf4d0 0x7f0f90073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.703+0000 7f0f96b7f640 1 --2- 172.21.15.81:0/939379709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f901412f0 0x7f0f900cef90 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.703+0000 7f0f96b7f640 1 -- 172.21.15.81:0/939379709 >> 172.21.15.81:0/939379709 conn(0x7f0f9006ccc0 msgr2=0x7f0f900b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:19.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.703+0000 7f0f96b7f640 1 -- 172.21.15.81:0/939379709 shutdown_connections 2024-06-29T15:55:19.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.703+0000 7f0f96b7f640 1 -- 172.21.15.81:0/939379709 wait complete. 2024-06-29T15:55:19.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.705+0000 7f0f96b7f640 1 Processor -- start 2024-06-29T15:55:19.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.705+0000 7f0f96b7f640 1 -- start start 2024-06-29T15:55:19.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f96b7f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f90073570 0x7f0f90155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:19.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f96b7f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f900cf4d0 0x7f0f9013ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:19.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f96b7f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f901412f0 0x7f0f901591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:19.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f96b7f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9015f230 con 0x7f0f901412f0 2024-06-29T15:55:19.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f96b7f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9015f0b0 con 0x7f0f900cf4d0 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f96b7f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0f900b3210 con 0x7f0f90073570 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f9511c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f901412f0 0x7f0f901591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f9511c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f901412f0 0x7f0f901591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54164/0 (socket says 172.21.15.81:54164) 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f9511c640 1 -- 172.21.15.81:0/4089657009 learned_addr learned my addr 172.21.15.81:0/4089657009 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f9491b640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f90073570 0x7f0f90155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f8ffff640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f900cf4d0 0x7f0f9013ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f9511c640 1 -- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f90073570 msgr2=0x7f0f90155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f9511c640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f90073570 0x7f0f90155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f9511c640 1 -- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f900cf4d0 msgr2=0x7f0f9013ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f9511c640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f900cf4d0 0x7f0f9013ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f9511c640 1 -- 172.21.15.81:0/4089657009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f901596e0 con 0x7f0f901412f0 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f9511c640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f901412f0 0x7f0f901591a0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800e8e0 tx=0x7f0f8800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f8d7fa640 1 -- 172.21.15.81:0/4089657009 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f88004d20 con 0x7f0f901412f0 2024-06-29T15:55:19.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f8d7fa640 1 -- 172.21.15.81:0/4089657009 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f8800ca00 con 0x7f0f901412f0 2024-06-29T15:55:19.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f96b7f640 1 -- 172.21.15.81:0/4089657009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f90159990 con 0x7f0f901412f0 2024-06-29T15:55:19.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f8d7fa640 1 -- 172.21.15.81:0/4089657009 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8800ce50 con 0x7f0f901412f0 2024-06-29T15:55:19.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.707+0000 7f0f96b7f640 1 -- 172.21.15.81:0/4089657009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f90157c80 con 0x7f0f901412f0 2024-06-29T15:55:19.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.708+0000 7f0f96b7f640 1 -- 172.21.15.81:0/4089657009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f9015e960 con 0x7f0f901412f0 2024-06-29T15:55:19.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.709+0000 7f0f8d7fa640 1 -- 172.21.15.81:0/4089657009 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 112) ==== 98332+0+0 (secure 0 0 0) 0x7f0f88011470 con 0x7f0f901412f0 2024-06-29T15:55:19.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.710+0000 7f0f8d7fa640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f0f740765e0 0x7f0f74078af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:19.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.710+0000 7f0f8d7fa640 1 -- 172.21.15.81:0/4089657009 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7f0f88098480 con 0x7f0f901412f0 2024-06-29T15:55:19.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.710+0000 7f0f9491b640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f0f740765e0 0x7f0f74078af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:19.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.711+0000 7f0f9491b640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f0f740765e0 0x7f0f74078af0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0f84033000 tx=0x7f0f84004730 comp rx=0 tx=0).ready entity=mgr.8261 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:19.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.714+0000 7f0f8d7fa640 1 -- 172.21.15.81:0/4089657009 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0f88016030 con 0x7f0f901412f0 2024-06-29T15:55:19.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:19.939+0000 7f0f96b7f640 1 -- 172.21.15.81:0/4089657009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f0f9006e240 con 0x7f0f901412f0 2024-06-29T15:55:20.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.198+0000 7f0f8d7fa640 1 -- 172.21.15.81:0/4089657009 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v113) ==== 93+0+0 (secure 0 0 0) 0x7f0f880611c0 con 0x7f0f901412f0 2024-06-29T15:55:20.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.201+0000 7f0f96b7f640 1 -- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f0f740765e0 msgr2=0x7f0f74078af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.201+0000 7f0f96b7f640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f0f740765e0 0x7f0f74078af0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0f84033000 tx=0x7f0f84004730 comp rx=0 tx=0).stop 2024-06-29T15:55:20.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.201+0000 7f0f96b7f640 1 -- 172.21.15.81:0/4089657009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f901412f0 msgr2=0x7f0f901591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.201+0000 7f0f96b7f640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f901412f0 0x7f0f901591a0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800e8e0 tx=0x7f0f8800eda0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.202+0000 7f0f96b7f640 1 -- 172.21.15.81:0/4089657009 shutdown_connections 2024-06-29T15:55:20.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.202+0000 7f0f96b7f640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f0f740765e0 0x7f0f74078af0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.202+0000 7f0f96b7f640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f901412f0 0x7f0f901591a0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.202+0000 7f0f96b7f640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0f900cf4d0 0x7f0f9013ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.202+0000 7f0f96b7f640 1 --2- 172.21.15.81:0/4089657009 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0f90073570 0x7f0f90155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.202+0000 7f0f96b7f640 1 -- 172.21.15.81:0/4089657009 >> 172.21.15.81:0/4089657009 conn(0x7f0f9006ccc0 msgr2=0x7f0f900b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:20.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.202+0000 7f0f96b7f640 1 -- 172.21.15.81:0/4089657009 shutdown_connections 2024-06-29T15:55:20.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.202+0000 7f0f96b7f640 1 -- 172.21.15.81:0/4089657009 wait complete. 2024-06-29T15:55:20.217 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:55:20.217 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:55:20.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:55:20.305 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:55:20.305 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:55:20.306 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:55:20.306 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:20.322 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:55:20.322 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:20.379 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.532+0000 7f1cd79eb640 1 -- 172.21.15.81:0/752366888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b5250 msgr2=0x7f1cd00d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.532+0000 7f1cd79eb640 1 --2- 172.21.15.81:0/752366888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b5250 0x7f1cd00d1fc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8005fd0 tx=0x7f1cc802ab20 comp rx=0 tx=0).stop 2024-06-29T15:55:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.532+0000 7f1cd79eb640 1 -- 172.21.15.81:0/752366888 shutdown_connections 2024-06-29T15:55:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.532+0000 7f1cd79eb640 1 --2- 172.21.15.81:0/752366888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b5250 0x7f1cd00d1fc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.532+0000 7f1cd79eb640 1 --2- 172.21.15.81:0/752366888 >> v2:172.21.15.167:3300/0 conn(0x7f1cd00d0b90 0x7f1cd00d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.532+0000 7f1cd79eb640 1 -- 172.21.15.81:0/752366888 >> 172.21.15.81:0/752366888 conn(0x7f1cd009cf30 msgr2=0x7f1cd00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.532+0000 7f1cd79eb640 1 -- 172.21.15.81:0/752366888 shutdown_connections 2024-06-29T15:55:20.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.532+0000 7f1cd79eb640 1 -- 172.21.15.81:0/752366888 wait complete. 2024-06-29T15:55:20.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.534+0000 7f1cd79eb640 1 Processor -- start 2024-06-29T15:55:20.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.534+0000 7f1cd79eb640 1 -- start start 2024-06-29T15:55:20.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd79eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd00b5250 0x7f1cd00a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd79eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd00b9040 0x7f1cd00b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd79eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b1f50 0x7f1cd00b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd79eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd00d2500 con 0x7f1cd00b9040 2024-06-29T15:55:20.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd79eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd00b8590 con 0x7f1cd00b5250 2024-06-29T15:55:20.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd79eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1cd00d2680 con 0x7f1cd00b1f50 2024-06-29T15:55:20.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd4f86640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd00b9040 0x7f1cd00b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd4f86640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd00b9040 0x7f1cd00b1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53378/0 (socket says 172.21.15.81:53378) 2024-06-29T15:55:20.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd4f86640 1 -- 172.21.15.81:0/81793074 learned_addr learned my addr 172.21.15.81:0/81793074 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:20.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd5787640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd00b5250 0x7f1cd00a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.535+0000 7f1cd5f88640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b1f50 0x7f1cd00b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cd4f86640 1 -- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b1f50 msgr2=0x7f1cd00b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cd4f86640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b1f50 0x7f1cd00b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cd4f86640 1 -- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd00b5250 msgr2=0x7f1cd00a1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cd4f86640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd00b5250 0x7f1cd00a1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cd4f86640 1 -- 172.21.15.81:0/81793074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd009dd80 con 0x7f1cd00b9040 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cd5787640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd00b5250 0x7f1cd00a1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cd5f88640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b1f50 0x7f1cd00b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cd4f86640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd00b9040 0x7f1cd00b1a10 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00e9f0 tx=0x7f1ccc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cc5ffb640 1 -- 172.21.15.81:0/81793074 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc00cd80 con 0x7f1cd00b9040 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cc5ffb640 1 -- 172.21.15.81:0/81793074 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ccc004020 con 0x7f1cd00b9040 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cd79eb640 1 -- 172.21.15.81:0/81793074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd009e060 con 0x7f1cd00b9040 2024-06-29T15:55:20.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cc5ffb640 1 -- 172.21.15.81:0/81793074 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc004470 con 0x7f1cd00b9040 2024-06-29T15:55:20.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.536+0000 7f1cd79eb640 1 -- 172.21.15.81:0/81793074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd009e620 con 0x7f1cd00b9040 2024-06-29T15:55:20.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.537+0000 7f1cd79eb640 1 -- 172.21.15.81:0/81793074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cd00d2bc0 con 0x7f1cd00b9040 2024-06-29T15:55:20.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.538+0000 7f1cc5ffb640 1 -- 172.21.15.81:0/81793074 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f1ccc0035f0 con 0x7f1cd00b9040 2024-06-29T15:55:20.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.538+0000 7f1cc5ffb640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f1cb40763f0 0x7f1cb4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.538+0000 7f1cd5787640 1 -- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f1cb40763f0 msgr2=0x7f1cb4078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:20.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.539+0000 7f1cc5ffb640 1 -- 172.21.15.81:0/81793074 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7f1ccc0986f0 con 0x7f1cd00b9040 2024-06-29T15:55:20.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.539+0000 7f1cd5787640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f1cb40763f0 0x7f1cb4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:20.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.543+0000 7f1cc5ffb640 1 -- 172.21.15.81:0/81793074 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1ccc061440 con 0x7f1cd00b9040 2024-06-29T15:55:20.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.728+0000 7f1cd79eb640 1 -- 172.21.15.81:0/81793074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1cd009daa0 con 0x7f1cd00b9040 2024-06-29T15:55:20.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.730+0000 7f1cc5ffb640 1 -- 172.21.15.81:0/81793074 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) ==== 74+0+17834 (secure 0 0 0) 0x7f1ccc065740 con 0x7f1cd00b9040 2024-06-29T15:55:20.730 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:20.731 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":183,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:50:20.905133+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.821386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.484738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/1637259453":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3006573425":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/578369376":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6833/2288011293":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6832/2288011293":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6833/2546364927":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/3941545005":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3712811511":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/955369459":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/153744272":"2024-06-30T15:39:23.843781+0000","172.21.15.167:6832/2546364927":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/2062891067":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/3310646322":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1870993879":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3737424986":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/1345841177":"2024-06-30T15:39:23.843781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:20.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 -- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f1cb40763f0 msgr2=0x7f1cb4078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:20.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f1cb40763f0 0x7f1cb4078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 -- 172.21.15.81:0/81793074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd00b9040 msgr2=0x7f1cd00b1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd00b9040 0x7f1cd00b1a10 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00e9f0 tx=0x7f1ccc00eeb0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 -- 172.21.15.81:0/81793074 shutdown_connections 2024-06-29T15:55:20.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f1cb40763f0 0x7f1cb4078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1cd00b1f50 0x7f1cd00b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd00b9040 0x7f1cd00b1a10 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 --2- 172.21.15.81:0/81793074 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1cd00b5250 0x7f1cd00a1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 -- 172.21.15.81:0/81793074 >> 172.21.15.81:0/81793074 conn(0x7f1cd009cf30 msgr2=0x7f1cd015e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 -- 172.21.15.81:0/81793074 shutdown_connections 2024-06-29T15:55:20.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.732+0000 7f1cd79eb640 1 -- 172.21.15.81:0/81793074 wait complete. 2024-06-29T15:55:20.744 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:39:09.821386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:39:10.484738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '181', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:55:20.745 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:55:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.862+0000 7f3d5a15b640 1 -- 172.21.15.81:0/969830954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d540d0940 msgr2=0x7f3d540b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.862+0000 7f3d5a15b640 1 --2- 172.21.15.81:0/969830954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d540d0940 0x7f3d540b4f40 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c005fd0 tx=0x7f3d4c02aa90 comp rx=0 tx=0).stop 2024-06-29T15:55:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.862+0000 7f3d5a15b640 1 -- 172.21.15.81:0/969830954 shutdown_connections 2024-06-29T15:55:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.862+0000 7f3d5a15b640 1 --2- 172.21.15.81:0/969830954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d540d0940 0x7f3d540b4f40 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.862+0000 7f3d5a15b640 1 --2- 172.21.15.81:0/969830954 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3d540b3200 0x7f3d540cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.862+0000 7f3d5a15b640 1 -- 172.21.15.81:0/969830954 >> 172.21.15.81:0/969830954 conn(0x7f3d540b3680 msgr2=0x7f3d540b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.862+0000 7f3d5a15b640 1 -- 172.21.15.81:0/969830954 shutdown_connections 2024-06-29T15:55:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.862+0000 7f3d5a15b640 1 -- 172.21.15.81:0/969830954 wait complete. 2024-06-29T15:55:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.865+0000 7f3d5a15b640 1 Processor -- start 2024-06-29T15:55:20.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.866+0000 7f3d5a15b640 1 -- start start 2024-06-29T15:55:20.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d5a15b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3d540b3200 0x7f3d5409cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d5a15b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3d540cccf0 0x7f3d54156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d5a15b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d54157460 0x7f3d54157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d5a15b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3d540b5cb0 con 0x7f3d54157460 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d5a15b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3d540b5b30 con 0x7f3d540cccf0 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d5a15b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3d540b5e30 con 0x7f3d540b3200 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d53fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d54157460 0x7f3d54157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d53fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d54157460 0x7f3d54157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53402/0 (socket says 172.21.15.81:53402) 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d53fff640 1 -- 172.21.15.81:0/3589386962 learned_addr learned my addr 172.21.15.81:0/3589386962 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d537fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3d540b3200 0x7f3d5409cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d52ffd640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3d540cccf0 0x7f3d54156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d53fff640 1 -- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3d540b3200 msgr2=0x7f3d5409cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d53fff640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3d540b3200 0x7f3d5409cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d53fff640 1 -- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3d540cccf0 msgr2=0x7f3d54156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d53fff640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3d540cccf0 0x7f3d54156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d53fff640 1 -- 172.21.15.81:0/3589386962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d54157f40 con 0x7f3d54157460 2024-06-29T15:55:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.868+0000 7f3d537fe640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3d540b3200 0x7f3d5409cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.869+0000 7f3d53fff640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d54157460 0x7f3d54157870 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009d90 tx=0x7f3d4c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.869+0000 7f3d3bfff640 1 -- 172.21.15.81:0/3589386962 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c044070 con 0x7f3d54157460 2024-06-29T15:55:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.869+0000 7f3d3bfff640 1 -- 172.21.15.81:0/3589386962 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d4c004040 con 0x7f3d54157460 2024-06-29T15:55:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.869+0000 7f3d3bfff640 1 -- 172.21.15.81:0/3589386962 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c004430 con 0x7f3d54157460 2024-06-29T15:55:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.869+0000 7f3d5a15b640 1 -- 172.21.15.81:0/3589386962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d541e46f0 con 0x7f3d54157460 2024-06-29T15:55:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.869+0000 7f3d5a15b640 1 -- 172.21.15.81:0/3589386962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d541e4c70 con 0x7f3d54157460 2024-06-29T15:55:20.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.870+0000 7f3d5a15b640 1 -- 172.21.15.81:0/3589386962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d540cc030 con 0x7f3d54157460 2024-06-29T15:55:20.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.871+0000 7f3d3bfff640 1 -- 172.21.15.81:0/3589386962 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f3d4c047070 con 0x7f3d54157460 2024-06-29T15:55:20.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.871+0000 7f3d3bfff640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f3d400763f0 0x7f3d40078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:20.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.872+0000 7f3d537fe640 1 -- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f3d400763f0 msgr2=0x7f3d40078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:20.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.872+0000 7f3d3bfff640 1 -- 172.21.15.81:0/3589386962 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7f3d4c0b8a30 con 0x7f3d54157460 2024-06-29T15:55:20.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.872+0000 7f3d537fe640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f3d400763f0 0x7f3d40078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:20.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:20.876+0000 7f3d3bfff640 1 -- 172.21.15.81:0/3589386962 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3d540cc030 con 0x7f3d54157460 2024-06-29T15:55:21.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.073+0000 7f3d537fe640 1 -- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f3d400763f0 msgr2=0x7f3d40078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:21.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.073+0000 7f3d537fe640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f3d400763f0 0x7f3d40078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:21.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.084+0000 7f3d5a15b640 1 -- 172.21.15.81:0/3589386962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3d540cc7b0 con 0x7f3d54157460 2024-06-29T15:55:21.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.084+0000 7f3d3bfff640 1 -- 172.21.15.81:0/3589386962 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v183) ==== 93+0+10 (secure 0 0 0) 0x7f3d541e46f0 con 0x7f3d54157460 2024-06-29T15:55:21.085 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:55:21.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.086+0000 7f3d5a15b640 1 -- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f3d400763f0 msgr2=0x7f3d40078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:21.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.086+0000 7f3d5a15b640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f3d400763f0 0x7f3d40078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.086+0000 7f3d5a15b640 1 -- 172.21.15.81:0/3589386962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d54157460 msgr2=0x7f3d54157870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.086+0000 7f3d5a15b640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d54157460 0x7f3d54157870 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009d90 tx=0x7f3d4c003ce0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.086+0000 7f3d5a15b640 1 -- 172.21.15.81:0/3589386962 shutdown_connections 2024-06-29T15:55:21.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.087+0000 7f3d5a15b640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f3d400763f0 0x7f3d40078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.087+0000 7f3d5a15b640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d54157460 0x7f3d54157870 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.087+0000 7f3d5a15b640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3d540cccf0 0x7f3d54156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.087+0000 7f3d5a15b640 1 --2- 172.21.15.81:0/3589386962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3d540b3200 0x7f3d5409cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.087+0000 7f3d5a15b640 1 -- 172.21.15.81:0/3589386962 >> 172.21.15.81:0/3589386962 conn(0x7f3d540b3680 msgr2=0x7f3d540cb920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.087+0000 7f3d5a15b640 1 -- 172.21.15.81:0/3589386962 shutdown_connections 2024-06-29T15:55:21.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.087+0000 7f3d5a15b640 1 -- 172.21.15.81:0/3589386962 wait complete. 2024-06-29T15:55:21.099 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:55:21.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.220+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/2318087960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6b80b57f0 msgr2=0x7fd6b8072f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.220+0000 7fd6bc8ef640 1 --2- 172.21.15.81:0/2318087960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6b80b57f0 0x7fd6b8072f10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009830 tx=0x7fd6ac02a480 comp rx=0 tx=0).stop 2024-06-29T15:55:21.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.220+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/2318087960 shutdown_connections 2024-06-29T15:55:21.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.220+0000 7fd6bc8ef640 1 --2- 172.21.15.81:0/2318087960 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6b80b57f0 0x7fd6b8072f10 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.220+0000 7fd6bc8ef640 1 --2- 172.21.15.81:0/2318087960 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6b8143860 0x7fd6b80b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.220+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/2318087960 >> 172.21.15.81:0/2318087960 conn(0x7fd6b80d0b70 msgr2=0x7fd6b80d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.220+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/2318087960 shutdown_connections 2024-06-29T15:55:21.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.220+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/2318087960 wait complete. 2024-06-29T15:55:21.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.222+0000 7fd6bc8ef640 1 Processor -- start 2024-06-29T15:55:21.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.222+0000 7fd6bc8ef640 1 -- start start 2024-06-29T15:55:21.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6bc8ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8073450 0x7fd6b813f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6bc8ef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6b80b57f0 0x7fd6b80b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6bc8ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6b8143860 0x7fd6b815c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6bc8ef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b815eef0 con 0x7fd6b8073450 2024-06-29T15:55:21.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6bc8ef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd6b80b48a0 con 0x7fd6b80b57f0 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6bc8ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd6b815f070 con 0x7fd6b8143860 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8073450 0x7fd6b813f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8073450 0x7fd6b813f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53408/0 (socket says 172.21.15.81:53408) 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b659c640 1 -- 172.21.15.81:0/1343060605 learned_addr learned my addr 172.21.15.81:0/1343060605 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b5d9b640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6b80b57f0 0x7fd6b80b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b6d9d640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6b8143860 0x7fd6b815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b659c640 1 -- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6b8143860 msgr2=0x7fd6b815c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b659c640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6b8143860 0x7fd6b815c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b659c640 1 -- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6b80b57f0 msgr2=0x7fd6b80b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b659c640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6b80b57f0 0x7fd6b80b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b659c640 1 -- 172.21.15.81:0/1343060605 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6b815cdb0 con 0x7fd6b8073450 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b5d9b640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6b80b57f0 0x7fd6b80b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:21.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6b659c640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8073450 0x7fd6b813f680 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800e9f0 tx=0x7fd6a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:21.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.224+0000 7fd6a6ffd640 1 -- 172.21.15.81:0/1343060605 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a800cd80 con 0x7fd6b8073450 2024-06-29T15:55:21.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.225+0000 7fd6a6ffd640 1 -- 172.21.15.81:0/1343060605 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6a8004020 con 0x7fd6b8073450 2024-06-29T15:55:21.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.225+0000 7fd6a6ffd640 1 -- 172.21.15.81:0/1343060605 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8004470 con 0x7fd6b8073450 2024-06-29T15:55:21.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.225+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/1343060605 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6b815b7d0 con 0x7fd6b8073450 2024-06-29T15:55:21.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.225+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/1343060605 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6b815bd90 con 0x7fd6b8073450 2024-06-29T15:55:21.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.226+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/1343060605 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6b815f5d0 con 0x7fd6b8073450 2024-06-29T15:55:21.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.226+0000 7fd6a6ffd640 1 -- 172.21.15.81:0/1343060605 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7fd6a80035f0 con 0x7fd6b8073450 2024-06-29T15:55:21.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.227+0000 7fd6a6ffd640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fd698076420 0x7fd698078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.227+0000 7fd6b5d9b640 1 -- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fd698076420 msgr2=0x7fd698078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:21.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.227+0000 7fd6a6ffd640 1 -- 172.21.15.81:0/1343060605 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7fd6a8098690 con 0x7fd6b8073450 2024-06-29T15:55:21.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.227+0000 7fd6b5d9b640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fd698076420 0x7fd698078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:21.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.231+0000 7fd6a6ffd640 1 -- 172.21.15.81:0/1343060605 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd6a80613e0 con 0x7fd6b8073450 2024-06-29T15:55:21.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.418+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/1343060605 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd6b80d1100 con 0x7fd6b8073450 2024-06-29T15:55:21.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.418+0000 7fd6a6ffd640 1 -- 172.21.15.81:0/1343060605 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v183) ==== 104+0+11 (secure 0 0 0) 0x7fd6a80656e0 con 0x7fd6b8073450 2024-06-29T15:55:21.419 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:55:21.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fd698076420 msgr2=0x7fd698078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:21.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fd698076420 0x7fd698078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/1343060605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8073450 msgr2=0x7fd6b813f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8073450 0x7fd6b813f680 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fd6a800e9f0 tx=0x7fd6a800eeb0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/1343060605 shutdown_connections 2024-06-29T15:55:21.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fd698076420 0x7fd698078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6b8143860 0x7fd6b815c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6b80b57f0 0x7fd6b80b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 --2- 172.21.15.81:0/1343060605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8073450 0x7fd6b813f680 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/1343060605 >> 172.21.15.81:0/1343060605 conn(0x7fd6b80d0b70 msgr2=0x7fd6b806edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/1343060605 shutdown_connections 2024-06-29T15:55:21.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.421+0000 7fd6bc8ef640 1 -- 172.21.15.81:0/1343060605 wait complete. 2024-06-29T15:55:21.433 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:55:21.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.551+0000 7f6509dee640 1 -- 172.21.15.81:0/1559379296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65040d0940 msgr2=0x7f65040b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.551+0000 7f6509dee640 1 --2- 172.21.15.81:0/1559379296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65040d0940 0x7f65040b4f40 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f6504073550 tx=0x7f64fc02aa30 comp rx=0 tx=0).stop 2024-06-29T15:55:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.552+0000 7f6509dee640 1 -- 172.21.15.81:0/1559379296 shutdown_connections 2024-06-29T15:55:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.552+0000 7f6509dee640 1 --2- 172.21.15.81:0/1559379296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65040d0940 0x7f65040b4f40 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.552+0000 7f6509dee640 1 --2- 172.21.15.81:0/1559379296 >> v2:172.21.15.167:3300/0 conn(0x7f65040b3200 0x7f65040cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.552+0000 7f6509dee640 1 -- 172.21.15.81:0/1559379296 >> 172.21.15.81:0/1559379296 conn(0x7f65040b3680 msgr2=0x7f65040b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.552+0000 7f6509dee640 1 -- 172.21.15.81:0/1559379296 shutdown_connections 2024-06-29T15:55:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.552+0000 7f6509dee640 1 -- 172.21.15.81:0/1559379296 wait complete. 2024-06-29T15:55:21.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.553+0000 7f6509dee640 1 Processor -- start 2024-06-29T15:55:21.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.553+0000 7f6509dee640 1 -- start start 2024-06-29T15:55:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6509dee640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65040b3200 0x7f65041d6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6509dee640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65040cccf0 0x7f65041e5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6509dee640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65041e2ed0 0x7f65041e6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6509dee640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f65040b5bb0 con 0x7f65041e2ed0 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6509dee640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f65040b5a30 con 0x7f65040cccf0 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6509dee640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f65040b5d30 con 0x7f65040b3200 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6503fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65041e2ed0 0x7f65041e6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6503fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65041e2ed0 0x7f65041e6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53436/0 (socket says 172.21.15.81:53436) 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6503fff640 1 -- 172.21.15.81:0/1814005551 learned_addr learned my addr 172.21.15.81:0/1814005551 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f65037fe640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65040b3200 0x7f65041d6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f64fbfff640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65040cccf0 0x7f65041e5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6503fff640 1 -- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65040b3200 msgr2=0x7f65041d6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6503fff640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65040b3200 0x7f65041d6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6503fff640 1 -- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65040cccf0 msgr2=0x7f65041e5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6503fff640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65040cccf0 0x7f65041e5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f6503fff640 1 -- 172.21.15.81:0/1814005551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65041e3770 con 0x7f65041e2ed0 2024-06-29T15:55:21.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.555+0000 7f64fbfff640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65040cccf0 0x7f65041e5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:21.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.556+0000 7f6503fff640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65041e2ed0 0x7f65041e6250 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f64fc009d60 tx=0x7f64fc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:21.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.556+0000 7f6500ff9640 1 -- 172.21.15.81:0/1814005551 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc033050 con 0x7f65041e2ed0 2024-06-29T15:55:21.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.556+0000 7f6500ff9640 1 -- 172.21.15.81:0/1814005551 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64fc004450 con 0x7f65041e2ed0 2024-06-29T15:55:21.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.556+0000 7f6509dee640 1 -- 172.21.15.81:0/1814005551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f650409ca10 con 0x7f65041e2ed0 2024-06-29T15:55:21.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.556+0000 7f6500ff9640 1 -- 172.21.15.81:0/1814005551 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc038450 con 0x7f65041e2ed0 2024-06-29T15:55:21.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.556+0000 7f6509dee640 1 -- 172.21.15.81:0/1814005551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f650409ee50 con 0x7f65041e2ed0 2024-06-29T15:55:21.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.557+0000 7f6509dee640 1 -- 172.21.15.81:0/1814005551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65040cc7b0 con 0x7f65041e2ed0 2024-06-29T15:55:21.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.558+0000 7f6500ff9640 1 -- 172.21.15.81:0/1814005551 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f64fc036070 con 0x7f65041e2ed0 2024-06-29T15:55:21.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.559+0000 7f6500ff9640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f64e4076420 0x7f64e4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.559+0000 7f65037fe640 1 -- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f64e4076420 msgr2=0x7f64e4078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:21.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.559+0000 7f6500ff9640 1 -- 172.21.15.81:0/1814005551 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7f64fc0b9060 con 0x7f65041e2ed0 2024-06-29T15:55:21.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.559+0000 7f65037fe640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f64e4076420 0x7f64e4078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:21.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.563+0000 7f6500ff9640 1 -- 172.21.15.81:0/1814005551 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f65041e3770 con 0x7f65041e2ed0 2024-06-29T15:55:21.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.747+0000 7f6509dee640 1 -- 172.21.15.81:0/1814005551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f65040b6ff0 con 0x7f65041e2ed0 2024-06-29T15:55:21.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.748+0000 7f6500ff9640 1 -- 172.21.15.81:0/1814005551 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v183) ==== 100+0+11 (secure 0 0 0) 0x7f64fc0a1e20 con 0x7f65041e2ed0 2024-06-29T15:55:21.748 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:55:21.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 -- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f64e4076420 msgr2=0x7f64e4078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:21.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f64e4076420 0x7f64e4078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 -- 172.21.15.81:0/1814005551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65041e2ed0 msgr2=0x7f65041e6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65041e2ed0 0x7f65041e6250 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f64fc009d60 tx=0x7f64fc003f10 comp rx=0 tx=0).stop 2024-06-29T15:55:21.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 -- 172.21.15.81:0/1814005551 shutdown_connections 2024-06-29T15:55:21.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f64e4076420 0x7f64e4078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65041e2ed0 0x7f65041e6250 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65040cccf0 0x7f65041e5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 --2- 172.21.15.81:0/1814005551 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65040b3200 0x7f65041d6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 -- 172.21.15.81:0/1814005551 >> 172.21.15.81:0/1814005551 conn(0x7f65040b3680 msgr2=0x7f6504070d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 -- 172.21.15.81:0/1814005551 shutdown_connections 2024-06-29T15:55:21.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.750+0000 7f6509dee640 1 -- 172.21.15.81:0/1814005551 wait complete. 2024-06-29T15:55:21.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:55:21.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.879+0000 7f7d450e4640 1 -- 172.21.15.81:0/1903188052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d400cf200 msgr2=0x7f7d400b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.879+0000 7f7d450e4640 1 --2- 172.21.15.81:0/1903188052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d400cf200 0x7f7d400b5180 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7d38005fd0 tx=0x7f7d3802aa90 comp rx=0 tx=0).stop 2024-06-29T15:55:21.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.879+0000 7f7d450e4640 1 -- 172.21.15.81:0/1903188052 shutdown_connections 2024-06-29T15:55:21.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.879+0000 7f7d450e4640 1 --2- 172.21.15.81:0/1903188052 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d400cf200 0x7f7d400b5180 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.879+0000 7f7d450e4640 1 --2- 172.21.15.81:0/1903188052 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d400cd760 0x7f7d400cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.879+0000 7f7d450e4640 1 -- 172.21.15.81:0/1903188052 >> 172.21.15.81:0/1903188052 conn(0x7f7d400b3680 msgr2=0x7f7d400b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:21.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.879+0000 7f7d450e4640 1 -- 172.21.15.81:0/1903188052 shutdown_connections 2024-06-29T15:55:21.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.879+0000 7f7d450e4640 1 -- 172.21.15.81:0/1903188052 wait complete. 2024-06-29T15:55:21.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.881+0000 7f7d450e4640 1 Processor -- start 2024-06-29T15:55:21.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.881+0000 7f7d450e4640 1 -- start start 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d450e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d400cd760 0x7f7d40156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d450e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d400cf200 0x7f7d40151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d450e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d4015f670 0x7f7d40159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d450e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7d400b5a50 con 0x7f7d4015f670 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d450e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7d400b58d0 con 0x7f7d400cf200 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d450e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7d400b5bd0 con 0x7f7d400cd760 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d3f59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d4015f670 0x7f7d40159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d3ed9d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d400cd760 0x7f7d40156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d3ed9d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d400cd760 0x7f7d40156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:41572/0 (socket says 172.21.15.81:41572) 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d3ed9d640 1 -- 172.21.15.81:0/4069734540 learned_addr learned my addr 172.21.15.81:0/4069734540 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d3e59c640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d400cf200 0x7f7d40151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d3f59e640 1 -- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d400cd760 msgr2=0x7f7d40156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d3f59e640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d400cd760 0x7f7d40156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d3f59e640 1 -- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d400cf200 msgr2=0x7f7d40151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:21.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d3f59e640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d400cf200 0x7f7d40151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:21.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.883+0000 7f7d3f59e640 1 -- 172.21.15.81:0/4069734540 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d40156f60 con 0x7f7d4015f670 2024-06-29T15:55:21.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.884+0000 7f7d3f59e640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d4015f670 0x7f7d40159e50 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f7d38032000 tx=0x7f7d38003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:21.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.884+0000 7f7d377fe640 1 -- 172.21.15.81:0/4069734540 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d38044070 con 0x7f7d4015f670 2024-06-29T15:55:21.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.884+0000 7f7d377fe640 1 -- 172.21.15.81:0/4069734540 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d38003e00 con 0x7f7d4015f670 2024-06-29T15:55:21.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.884+0000 7f7d450e4640 1 -- 172.21.15.81:0/4069734540 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d40154710 con 0x7f7d4015f670 2024-06-29T15:55:21.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.884+0000 7f7d377fe640 1 -- 172.21.15.81:0/4069734540 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d380044b0 con 0x7f7d4015f670 2024-06-29T15:55:21.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.884+0000 7f7d450e4640 1 -- 172.21.15.81:0/4069734540 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d40154cd0 con 0x7f7d4015f670 2024-06-29T15:55:21.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.885+0000 7f7d450e4640 1 -- 172.21.15.81:0/4069734540 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d40156ca0 con 0x7f7d4015f670 2024-06-29T15:55:21.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.886+0000 7f7d377fe640 1 -- 172.21.15.81:0/4069734540 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f7d38047070 con 0x7f7d4015f670 2024-06-29T15:55:21.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.886+0000 7f7d377fe640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f7d20076420 0x7f7d20078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:21.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.886+0000 7f7d3ed9d640 1 -- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f7d20076420 msgr2=0x7f7d20078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:21.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.886+0000 7f7d377fe640 1 -- 172.21.15.81:0/4069734540 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7f7d380b8a30 con 0x7f7d4015f670 2024-06-29T15:55:21.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.886+0000 7f7d3ed9d640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f7d20076420 0x7f7d20078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:21.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:21.890+0000 7f7d377fe640 1 -- 172.21.15.81:0/4069734540 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7d38032030 con 0x7f7d4015f670 2024-06-29T15:55:22.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.088+0000 7f7d3ed9d640 1 -- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f7d20076420 msgr2=0x7f7d20078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:22.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.088+0000 7f7d3ed9d640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f7d20076420 0x7f7d20078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:22.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.090+0000 7f7d450e4640 1 -- 172.21.15.81:0/4069734540 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7d400b5180 con 0x7f7d4015f670 2024-06-29T15:55:22.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.091+0000 7f7d377fe640 1 -- 172.21.15.81:0/4069734540 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v183) ==== 81+0+953 (secure 0 0 0) 0x7f7d38081800 con 0x7f7d4015f670 2024-06-29T15:55:22.091 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1637259453 2024-06-30T15:50:20.905071+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3006573425 2024-06-30T15:50:20.905071+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/578369376 2024-06-30T15:50:20.905071+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/2288011293 2024-06-30T15:50:20.905071+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/2288011293 2024-06-30T15:50:20.905071+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/2546364927 2024-06-30T15:39:23.843781+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3941545005 2024-06-30T15:50:20.905071+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3712811511 2024-06-30T15:39:23.843781+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/955369459 2024-06-30T15:39:23.843781+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/153744272 2024-06-30T15:39:23.843781+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/2546364927 2024-06-30T15:39:23.843781+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2062891067 2024-06-30T15:39:23.843781+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3310646322 2024-06-30T15:39:23.843781+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1870993879 2024-06-30T15:50:20.905071+0000 2024-06-29T15:55:22.092 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3737424986 2024-06-30T15:50:20.905071+0000 2024-06-29T15:55:22.093 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1345841177 2024-06-30T15:39:23.843781+0000 2024-06-29T15:55:22.093 INFO:teuthology.orchestra.run.smithi081.stderr:listed 16 entries 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 -- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f7d20076420 msgr2=0x7f7d20078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f7d20076420 0x7f7d20078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 -- 172.21.15.81:0/4069734540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d4015f670 msgr2=0x7f7d40159e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d4015f670 0x7f7d40159e50 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f7d38032000 tx=0x7f7d38003c10 comp rx=0 tx=0).stop 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 -- 172.21.15.81:0/4069734540 shutdown_connections 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f7d20076420 0x7f7d20078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d4015f670 0x7f7d40159e50 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d400cf200 0x7f7d40151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 --2- 172.21.15.81:0/4069734540 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d400cd760 0x7f7d40156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 -- 172.21.15.81:0/4069734540 >> 172.21.15.81:0/4069734540 conn(0x7f7d400b3680 msgr2=0x7f7d400d1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 -- 172.21.15.81:0/4069734540 shutdown_connections 2024-06-29T15:55:22.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.093+0000 7f7d450e4640 1 -- 172.21.15.81:0/4069734540 wait complete. 2024-06-29T15:55:22.106 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:55:22.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:22.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:55:22.138 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T15:55:22.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:22.138 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:55:22.194 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:55:22.194 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:55:22.194 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:22.210 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:55:22.210 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:55:22.210 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:55:22.210 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:22.250 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:55:22.250 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:55:22.250 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:55:22.250 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:22.265 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:55:22.266 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.406+0000 7fc0bf247640 1 -- 172.21.15.81:0/2957101687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b80b5f50 msgr2=0x7fc0b8166870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.406+0000 7fc0bf247640 1 --2- 172.21.15.81:0/2957101687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b80b5f50 0x7fc0b8166870 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0005fd0 tx=0x7fc0b002aaa0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.406+0000 7fc0bf247640 1 -- 172.21.15.81:0/2957101687 shutdown_connections 2024-06-29T15:55:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.406+0000 7fc0bf247640 1 --2- 172.21.15.81:0/2957101687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b80b5f50 0x7fc0b8166870 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.406+0000 7fc0bf247640 1 -- 172.21.15.81:0/2957101687 >> 172.21.15.81:0/2957101687 conn(0x7fc0b80b3680 msgr2=0x7fc0b80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.406+0000 7fc0bf247640 1 -- 172.21.15.81:0/2957101687 shutdown_connections 2024-06-29T15:55:22.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.406+0000 7fc0bf247640 1 -- 172.21.15.81:0/2957101687 wait complete. 2024-06-29T15:55:22.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.408+0000 7fc0bf247640 1 Processor -- start 2024-06-29T15:55:22.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.408+0000 7fc0bf247640 1 -- start start 2024-06-29T15:55:22.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bf247640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0b80b3200 0x7fc0b80ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bf247640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0b80b5f50 0x7fc0b80cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bf247640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b80d0230 0x7fc0b81f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bf247640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b8167390 con 0x7fc0b80d0230 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bf247640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b8167210 con 0x7fc0b80b5f50 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bf247640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc0b8167510 con 0x7fc0b80b3200 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bd7e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b80d0230 0x7fc0b81f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bd7e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b80d0230 0x7fc0b81f4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53458/0 (socket says 172.21.15.81:53458) 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bd7e4640 1 -- 172.21.15.81:0/3655651396 learned_addr learned my addr 172.21.15.81:0/3655651396 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0b7fff640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0b80b5f50 0x7fc0b80cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bcfe3640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0b80b3200 0x7fc0b80ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bd7e4640 1 -- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0b80b3200 msgr2=0x7fc0b80ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bd7e4640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0b80b3200 0x7fc0b80ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bd7e4640 1 -- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0b80b5f50 msgr2=0x7fc0b80cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bd7e4640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0b80b5f50 0x7fc0b80cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bd7e4640 1 -- 172.21.15.81:0/3655651396 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b81540d0 con 0x7fc0b80d0230 2024-06-29T15:55:22.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0b7fff640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0b80b5f50 0x7fc0b80cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:22.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.410+0000 7fc0bd7e4640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b80d0230 0x7fc0b81f4870 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009d90 tx=0x7fc0b0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:22.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.411+0000 7fc0b57fa640 1 -- 172.21.15.81:0/3655651396 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0044070 con 0x7fc0b80d0230 2024-06-29T15:55:22.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.411+0000 7fc0b57fa640 1 -- 172.21.15.81:0/3655651396 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0b0004250 con 0x7fc0b80d0230 2024-06-29T15:55:22.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.411+0000 7fc0b57fa640 1 -- 172.21.15.81:0/3655651396 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0004680 con 0x7fc0b80d0230 2024-06-29T15:55:22.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.411+0000 7fc0bf247640 1 -- 172.21.15.81:0/3655651396 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0b81543b0 con 0x7fc0b80d0230 2024-06-29T15:55:22.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.411+0000 7fc0bf247640 1 -- 172.21.15.81:0/3655651396 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0b814a810 con 0x7fc0b80d0230 2024-06-29T15:55:22.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.412+0000 7fc0bf247640 1 -- 172.21.15.81:0/3655651396 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0b814a320 con 0x7fc0b80d0230 2024-06-29T15:55:22.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.413+0000 7fc0b57fa640 1 -- 172.21.15.81:0/3655651396 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7fc0b0047070 con 0x7fc0b80d0230 2024-06-29T15:55:22.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.413+0000 7fc0b57fa640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fc09c0763f0 0x7fc09c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.413+0000 7fc0bcfe3640 1 -- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fc09c0763f0 msgr2=0x7fc09c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:22.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.413+0000 7fc0b57fa640 1 -- 172.21.15.81:0/3655651396 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7fc0b00b8bf0 con 0x7fc0b80d0230 2024-06-29T15:55:22.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.413+0000 7fc0bcfe3640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fc09c0763f0 0x7fc09c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:22.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.417+0000 7fc0b57fa640 1 -- 172.21.15.81:0/3655651396 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0b0032030 con 0x7fc0b80d0230 2024-06-29T15:55:22.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.615+0000 7fc0bcfe3640 1 -- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fc09c0763f0 msgr2=0x7fc09c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:22.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.615+0000 7fc0bcfe3640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fc09c0763f0 0x7fc09c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:22.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.636+0000 7fc0bf247640 1 -- 172.21.15.81:0/3655651396 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0b80cc7b0 con 0x7fc0b80d0230 2024-06-29T15:55:22.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.636+0000 7fc0b57fa640 1 -- 172.21.15.81:0/3655651396 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+5237 (secure 0 0 0) 0x7fc0b0004820 con 0x7fc0b80d0230 2024-06-29T15:55:22.637 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:22.637 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":95,"btime":"2024-06-29T15:39:15:192034+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2580108037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2580108037},{"type":"v1","addr":"172.21.15.167:6836","nonce":2580108037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":10155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3892298237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3892298237},{"type":"v1","addr":"172.21.15.81:6834","nonce":3892298237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":10158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2457574359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2457574359},{"type":"v1","addr":"172.21.15.81:6835","nonce":2457574359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.526747+0000","modified":"2024-06-29T15:39:14.562127+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10170},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10170":{"gid":10170,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/446857148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":446857148},{"type":"v1","addr":"172.21.15.167:6837","nonce":446857148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10170,"qdb_cluster":[10170]},"id":11}]} 2024-06-29T15:55:22.637 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 95 2024-06-29T15:55:22.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.638+0000 7fc0bf247640 1 -- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fc09c0763f0 msgr2=0x7fc09c078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:22.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.638+0000 7fc0bf247640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fc09c0763f0 0x7fc09c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.638+0000 7fc0bf247640 1 -- 172.21.15.81:0/3655651396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b80d0230 msgr2=0x7fc0b81f4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.638+0000 7fc0bf247640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b80d0230 0x7fc0b81f4870 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009d90 tx=0x7fc0b0003f10 comp rx=0 tx=0).stop 2024-06-29T15:55:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.638+0000 7fc0bf247640 1 -- 172.21.15.81:0/3655651396 shutdown_connections 2024-06-29T15:55:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.639+0000 7fc0bf247640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fc09c0763f0 0x7fc09c078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.639+0000 7fc0bf247640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b80d0230 0x7fc0b81f4870 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.639+0000 7fc0bf247640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc0b80b5f50 0x7fc0b80cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.639+0000 7fc0bf247640 1 --2- 172.21.15.81:0/3655651396 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc0b80b3200 0x7fc0b80ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.639+0000 7fc0bf247640 1 -- 172.21.15.81:0/3655651396 >> 172.21.15.81:0/3655651396 conn(0x7fc0b80b3680 msgr2=0x7fc0b80d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.639+0000 7fc0bf247640 1 -- 172.21.15.81:0/3655651396 shutdown_connections 2024-06-29T15:55:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.639+0000 7fc0bf247640 1 -- 172.21.15.81:0/3655651396 wait complete. 2024-06-29T15:55:22.651 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:22.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.764+0000 7f068f81f640 1 -- 172.21.15.81:0/1971946543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06880b3200 msgr2=0x7f06880cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.764+0000 7f068f81f640 1 --2- 172.21.15.81:0/1971946543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06880b3200 0x7f06880cc7b0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f06880c5590 tx=0x7f068402a560 comp rx=0 tx=0).stop 2024-06-29T15:55:22.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.764+0000 7f068f81f640 1 -- 172.21.15.81:0/1971946543 shutdown_connections 2024-06-29T15:55:22.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.764+0000 7f068f81f640 1 --2- 172.21.15.81:0/1971946543 >> v2:172.21.15.167:3300/0 conn(0x7f06880d0940 0x7f06880b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.764+0000 7f068f81f640 1 --2- 172.21.15.81:0/1971946543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06880b3200 0x7f06880cc7b0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.764+0000 7f068f81f640 1 -- 172.21.15.81:0/1971946543 >> 172.21.15.81:0/1971946543 conn(0x7f06880b3680 msgr2=0x7f06880b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:22.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.764+0000 7f068f81f640 1 -- 172.21.15.81:0/1971946543 shutdown_connections 2024-06-29T15:55:22.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.764+0000 7f068f81f640 1 -- 172.21.15.81:0/1971946543 wait complete. 2024-06-29T15:55:22.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.766+0000 7f068f81f640 1 Processor -- start 2024-06-29T15:55:22.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.766+0000 7f068f81f640 1 -- start start 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068f81f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06880b3200 0x7f068809c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068f81f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06880cccf0 0x7f06880cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068f81f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06881e2b50 0x7f06880d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068f81f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06880b5bb0 con 0x7f06880b3200 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068f81f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f06880b5a30 con 0x7f06881e2b50 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068f81f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f06880b5d30 con 0x7f06880cccf0 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068d5bb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06880b3200 0x7f068809c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068d5bb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06880b3200 0x7f068809c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53476/0 (socket says 172.21.15.81:53476) 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068d5bb640 1 -- 172.21.15.81:0/3523627918 learned_addr learned my addr 172.21.15.81:0/3523627918 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068ddbc640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06881e2b50 0x7f06880d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068cdba640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06880cccf0 0x7f06880cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068d5bb640 1 -- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06880cccf0 msgr2=0x7f06880cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068d5bb640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06880cccf0 0x7f06880cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068d5bb640 1 -- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06881e2b50 msgr2=0x7f06880d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068d5bb640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06881e2b50 0x7f06880d04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068d5bb640 1 -- 172.21.15.81:0/3523627918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068809f090 con 0x7f06880b3200 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.767+0000 7f068cdba640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06880cccf0 0x7f06880cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.768+0000 7f068d5bb640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06880b3200 0x7f068809c590 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f06840099c0 tx=0x7f06840038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.768+0000 7f067dffb640 1 -- 172.21.15.81:0/3523627918 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068403e030 con 0x7f06880b3200 2024-06-29T15:55:22.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.768+0000 7f067dffb640 1 -- 172.21.15.81:0/3523627918 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0684003db0 con 0x7f06880b3200 2024-06-29T15:55:22.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.768+0000 7f068f81f640 1 -- 172.21.15.81:0/3523627918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068809f370 con 0x7f06880b3200 2024-06-29T15:55:22.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.768+0000 7f067dffb640 1 -- 172.21.15.81:0/3523627918 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684035040 con 0x7f06880b3200 2024-06-29T15:55:22.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.768+0000 7f068f81f640 1 -- 172.21.15.81:0/3523627918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06880ca1b0 con 0x7f06880b3200 2024-06-29T15:55:22.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.769+0000 7f068f81f640 1 -- 172.21.15.81:0/3523627918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06880b52a0 con 0x7f06880b3200 2024-06-29T15:55:22.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.770+0000 7f067dffb640 1 -- 172.21.15.81:0/3523627918 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f0684041030 con 0x7f06880b3200 2024-06-29T15:55:22.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.770+0000 7f067dffb640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f066c0763f0 0x7f066c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:22.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.771+0000 7f068cdba640 1 -- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f066c0763f0 msgr2=0x7f066c078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:22.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.771+0000 7f067dffb640 1 -- 172.21.15.81:0/3523627918 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7f06840b8850 con 0x7f06880b3200 2024-06-29T15:55:22.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.771+0000 7f068cdba640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f066c0763f0 0x7f066c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:22.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.774+0000 7f067dffb640 1 -- 172.21.15.81:0/3523627918 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f06880b52a0 con 0x7f06880b3200 2024-06-29T15:55:22.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.957+0000 7f068f81f640 1 -- 172.21.15.81:0/3523627918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06880cc7b0 con 0x7f06880b3200 2024-06-29T15:55:22.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.958+0000 7f067dffb640 1 -- 172.21.15.81:0/3523627918 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) ==== 74+0+17834 (secure 0 0 0) 0x7f0684081700 con 0x7f06880b3200 2024-06-29T15:55:22.959 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:22.959 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":183,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:50:20.905133+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.821386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.484738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"181","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/1637259453":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3006573425":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/578369376":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6833/2288011293":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6832/2288011293":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6833/2546364927":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/3941545005":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3712811511":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/955369459":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/153744272":"2024-06-30T15:39:23.843781+0000","172.21.15.167:6832/2546364927":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/2062891067":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/3310646322":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1870993879":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3737424986":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/1345841177":"2024-06-30T15:39:23.843781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 -- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f066c0763f0 msgr2=0x7f066c078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f066c0763f0 0x7f066c078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 -- 172.21.15.81:0/3523627918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06880b3200 msgr2=0x7f068809c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06880b3200 0x7f068809c590 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f06840099c0 tx=0x7f06840038a0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 -- 172.21.15.81:0/3523627918 shutdown_connections 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f066c0763f0 0x7f066c078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06881e2b50 0x7f06880d04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06880cccf0 0x7f06880cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 --2- 172.21.15.81:0/3523627918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06880b3200 0x7f068809c590 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 -- 172.21.15.81:0/3523627918 >> 172.21.15.81:0/3523627918 conn(0x7f06880b3680 msgr2=0x7f06881e81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 -- 172.21.15.81:0/3523627918 shutdown_connections 2024-06-29T15:55:22.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:22.960+0000 7f068f81f640 1 -- 172.21.15.81:0/3523627918 wait complete. 2024-06-29T15:55:22.972 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:39:09.821386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '180', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:39:10.484738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '181', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:55:22.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:55:23.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.090+0000 7fbb41c5b640 1 -- 172.21.15.81:0/298015503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb3c0ce770 msgr2=0x7fbb3c0b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.090+0000 7fbb41c5b640 1 --2- 172.21.15.81:0/298015503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb3c0ce770 0x7fbb3c0b35c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbb24005fd0 tx=0x7fbb2402a540 comp rx=0 tx=0).stop 2024-06-29T15:55:23.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.090+0000 7fbb41c5b640 1 -- 172.21.15.81:0/298015503 shutdown_connections 2024-06-29T15:55:23.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.090+0000 7fbb41c5b640 1 --2- 172.21.15.81:0/298015503 >> v2:172.21.15.167:3300/0 conn(0x7fbb3c0ceb80 0x7fbb3c0b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.090+0000 7fbb41c5b640 1 --2- 172.21.15.81:0/298015503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb3c0ce770 0x7fbb3c0b35c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.090+0000 7fbb41c5b640 1 -- 172.21.15.81:0/298015503 >> 172.21.15.81:0/298015503 conn(0x7fbb3c0d3c60 msgr2=0x7fbb3c0d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:23.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.091+0000 7fbb41c5b640 1 -- 172.21.15.81:0/298015503 shutdown_connections 2024-06-29T15:55:23.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.091+0000 7fbb41c5b640 1 -- 172.21.15.81:0/298015503 wait complete. 2024-06-29T15:55:23.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.092+0000 7fbb41c5b640 1 Processor -- start 2024-06-29T15:55:23.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.092+0000 7fbb41c5b640 1 -- start start 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb41c5b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb3c0ce770 0x7fbb3c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb41c5b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb3c0ceb80 0x7fbb3c159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb41c5b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb3c0d2240 0x7fbb3c0b5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb41c5b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3c0d15a0 con 0x7fbb3c0d2240 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb41c5b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3c0d1420 con 0x7fbb3c0ceb80 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb41c5b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3c0d4990 con 0x7fbb3c0ce770 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3affd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb3c0ceb80 0x7fbb3c159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3b7fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb3c0ce770 0x7fbb3c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb3c0d2240 0x7fbb3c0b5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3affd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb3c0ceb80 0x7fbb3c159170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:54340/0 (socket says 172.21.15.81:54340) 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3affd640 1 -- 172.21.15.81:0/1304331471 learned_addr learned my addr 172.21.15.81:0/1304331471 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:23.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3b7fe640 1 -- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb3c0ceb80 msgr2=0x7fbb3c159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3b7fe640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb3c0ceb80 0x7fbb3c159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3b7fe640 1 -- 172.21.15.81:0/1304331471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb3c0d2240 msgr2=0x7fbb3c0b5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3b7fe640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb3c0d2240 0x7fbb3c0b5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3b7fe640 1 -- 172.21.15.81:0/1304331471 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3c15b8d0 con 0x7fbb3c0ce770 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3bfff640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb3c0d2240 0x7fbb3c0b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.094+0000 7fbb3affd640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb3c0ceb80 0x7fbb3c159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.095+0000 7fbb3b7fe640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb3c0ce770 0x7fbb3c158ae0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c0b4ee0 tx=0x7fbb240038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.095+0000 7fbb33fff640 1 -- 172.21.15.81:0/1304331471 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2403e030 con 0x7fbb3c0ce770 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.095+0000 7fbb33fff640 1 -- 172.21.15.81:0/1304331471 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb24003db0 con 0x7fbb3c0ce770 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.095+0000 7fbb33fff640 1 -- 172.21.15.81:0/1304331471 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24035040 con 0x7fbb3c0ce770 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.095+0000 7fbb41c5b640 1 -- 172.21.15.81:0/1304331471 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3c15bbb0 con 0x7fbb3c0ce770 2024-06-29T15:55:23.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.095+0000 7fbb41c5b640 1 -- 172.21.15.81:0/1304331471 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb3c15c170 con 0x7fbb3c0ce770 2024-06-29T15:55:23.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.096+0000 7fbb41c5b640 1 -- 172.21.15.81:0/1304331471 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3c074bb0 con 0x7fbb3c0ce770 2024-06-29T15:55:23.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.097+0000 7fbb33fff640 1 -- 172.21.15.81:0/1304331471 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7fbb24041030 con 0x7fbb3c0ce770 2024-06-29T15:55:23.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.097+0000 7fbb33fff640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fbb1c076420 0x7fbb1c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.097+0000 7fbb3affd640 1 -- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fbb1c076420 msgr2=0x7fbb1c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:23.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.097+0000 7fbb33fff640 1 -- 172.21.15.81:0/1304331471 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(183..183 src has 1..183) ==== 6497+0+0 (secure 0 0 0) 0x7fbb240b83a0 con 0x7fbb3c0ce770 2024-06-29T15:55:23.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.097+0000 7fbb3affd640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fbb1c076420 0x7fbb1c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:23.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.105+0000 7fbb33fff640 1 -- 172.21.15.81:0/1304331471 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb3c074bb0 con 0x7fbb3c0ce770 2024-06-29T15:55:23.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.299+0000 7fbb3affd640 1 -- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fbb1c076420 msgr2=0x7fbb1c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:23.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.299+0000 7fbb3affd640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fbb1c076420 0x7fbb1c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:23.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.350+0000 7fbb41c5b640 1 -- 172.21.15.81:0/1304331471 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbb3c0d2650 con 0x7fbb3c0ce770 2024-06-29T15:55:23.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.350+0000 7fbb33fff640 1 -- 172.21.15.81:0/1304331471 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) ==== 93+0+10 (secure 0 0 0) 0x7fbb240811c0 con 0x7fbb3c0ce770 2024-06-29T15:55:23.351 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:55:23.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.352+0000 7fbb41c5b640 1 -- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fbb1c076420 msgr2=0x7fbb1c078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:23.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.352+0000 7fbb41c5b640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fbb1c076420 0x7fbb1c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.352+0000 7fbb41c5b640 1 -- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb3c0ce770 msgr2=0x7fbb3c158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.352+0000 7fbb41c5b640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb3c0ce770 0x7fbb3c158ae0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c0b4ee0 tx=0x7fbb240038a0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.352+0000 7fbb41c5b640 1 -- 172.21.15.81:0/1304331471 shutdown_connections 2024-06-29T15:55:23.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.352+0000 7fbb41c5b640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fbb1c076420 0x7fbb1c078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.352+0000 7fbb41c5b640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb3c0d2240 0x7fbb3c0b5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.352+0000 7fbb41c5b640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbb3c0ceb80 0x7fbb3c159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.352+0000 7fbb41c5b640 1 --2- 172.21.15.81:0/1304331471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbb3c0ce770 0x7fbb3c158ae0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.352+0000 7fbb41c5b640 1 -- 172.21.15.81:0/1304331471 >> 172.21.15.81:0/1304331471 conn(0x7fbb3c0d3c60 msgr2=0x7fbb3c166000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:23.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.353+0000 7fbb41c5b640 1 -- 172.21.15.81:0/1304331471 shutdown_connections 2024-06-29T15:55:23.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.353+0000 7fbb41c5b640 1 -- 172.21.15.81:0/1304331471 wait complete. 2024-06-29T15:55:23.364 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:55:23.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.515+0000 7fb6bffff640 1 -- 172.21.15.81:0/3818011522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6b8004b40 msgr2=0x7fb6b8007490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.515+0000 7fb6bffff640 1 --2- 172.21.15.81:0/3818011522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6b8004b40 0x7fb6b8007490 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb6c00516e0 tx=0x7fb6c00a54f0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.516+0000 7fb6bffff640 1 -- 172.21.15.81:0/3818011522 shutdown_connections 2024-06-29T15:55:23.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.516+0000 7fb6bffff640 1 --2- 172.21.15.81:0/3818011522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6b80533a0 0x7fb6b80537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.516+0000 7fb6bffff640 1 --2- 172.21.15.81:0/3818011522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6b8004b40 0x7fb6b8007490 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.516+0000 7fb6bffff640 1 -- 172.21.15.81:0/3818011522 >> 172.21.15.81:0/3818011522 conn(0x7fb6b801dfb0 msgr2=0x7fb6b8039a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:23.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.516+0000 7fb6bffff640 1 -- 172.21.15.81:0/3818011522 shutdown_connections 2024-06-29T15:55:23.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.517+0000 7fb6bffff640 1 -- 172.21.15.81:0/3818011522 wait complete. 2024-06-29T15:55:23.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.520+0000 7fb6bffff640 1 Processor -- start 2024-06-29T15:55:23.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.521+0000 7fb6bffff640 1 -- start start 2024-06-29T15:55:23.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.523+0000 7fb6bffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6b8004b40 0x7fb6b80d5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.524+0000 7fb6beffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6b8004b40 0x7fb6b80d5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.524+0000 7fb6beffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6b8004b40 0x7fb6b80d5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53504/0 (socket says 172.21.15.81:53504) 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.524+0000 7fb6beffd640 1 -- 172.21.15.81:0/435376339 learned_addr learned my addr 172.21.15.81:0/435376339 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.524+0000 7fb6bffff640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6b80366a0 0x7fb6b801f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.524+0000 7fb6bffff640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6b80533a0 0x7fb6b80f0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.524+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b8037520 con 0x7fb6b8004b40 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.524+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b80373a0 con 0x7fb6b80366a0 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.524+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb6b8053cf0 con 0x7fb6b80533a0 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.524+0000 7fb6bf7fe640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6b80533a0 0x7fb6b80f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.524+0000 7fb6be7fc640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6b80366a0 0x7fb6b801f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6bf7fe640 1 -- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6b80366a0 msgr2=0x7fb6b801f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6bf7fe640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6b80366a0 0x7fb6b801f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6bf7fe640 1 -- 172.21.15.81:0/435376339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6b8004b40 msgr2=0x7fb6b80d5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6bf7fe640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6b8004b40 0x7fb6b80d5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6bf7fe640 1 -- 172.21.15.81:0/435376339 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b8022510 con 0x7fb6b80533a0 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6beffd640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6b8004b40 0x7fb6b80d5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6bf7fe640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6b80533a0 0x7fb6b80f0730 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000e670 tx=0x7fb6b000eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6a77fe640 1 -- 172.21.15.81:0/435376339 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0004220 con 0x7fb6b80533a0 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6a77fe640 1 -- 172.21.15.81:0/435376339 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6b000c8b0 con 0x7fb6b80533a0 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6a77fe640 1 -- 172.21.15.81:0/435376339 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b000cd00 con 0x7fb6b80533a0 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6b801f650 con 0x7fb6b80533a0 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.525+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6b8021c70 con 0x7fb6b80533a0 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.527+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6b801f880 con 0x7fb6b80533a0 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.527+0000 7fb6a77fe640 1 -- 172.21.15.81:0/435376339 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7fb6b00035f0 con 0x7fb6b80533a0 2024-06-29T15:55:23.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.528+0000 7fb6a77fe640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fb6ac0763f0 0x7fb6ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.528+0000 7fb6a77fe640 1 -- 172.21.15.81:0/435376339 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(184..184 src has 1..184) ==== 6553+0+0 (secure 0 0 0) 0x7fb6b0098120 con 0x7fb6b80533a0 2024-06-29T15:55:23.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.528+0000 7fb6beffd640 1 -- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fb6ac0763f0 msgr2=0x7fb6ac078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:23.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.528+0000 7fb6beffd640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fb6ac0763f0 0x7fb6ac078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:23.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.533+0000 7fb6a77fe640 1 -- 172.21.15.81:0/435376339 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb6b801f880 con 0x7fb6b80533a0 2024-06-29T15:55:23.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.720+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb6b80537b0 con 0x7fb6b80533a0 2024-06-29T15:55:23.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.721+0000 7fb6a77fe640 1 -- 172.21.15.81:0/435376339 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v184) ==== 104+0+11 (secure 0 0 0) 0x7fb6b0060ef0 con 0x7fb6b80533a0 2024-06-29T15:55:23.721 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:55:23.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fb6ac0763f0 msgr2=0x7fb6ac078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:23.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fb6ac0763f0 0x7fb6ac078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6b80533a0 msgr2=0x7fb6b80f0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6b80533a0 0x7fb6b80f0730 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000e670 tx=0x7fb6b000eb30 comp rx=0 tx=0).stop 2024-06-29T15:55:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 shutdown_connections 2024-06-29T15:55:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7fb6ac0763f0 0x7fb6ac078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb6b80533a0 0x7fb6b80f0730 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb6b80366a0 0x7fb6b801f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 --2- 172.21.15.81:0/435376339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6b8004b40 0x7fb6b80d5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 >> 172.21.15.81:0/435376339 conn(0x7fb6b801dfb0 msgr2=0x7fb6b804b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 shutdown_connections 2024-06-29T15:55:23.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.723+0000 7fb6bffff640 1 -- 172.21.15.81:0/435376339 wait complete. 2024-06-29T15:55:23.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:55:23.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.852+0000 7f40defc7640 1 -- 172.21.15.81:0/1912077785 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40d80b5250 msgr2=0x7f40d80d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.852+0000 7f40defc7640 1 --2- 172.21.15.81:0/1912077785 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40d80b5250 0x7f40d80d09c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f40c4005fd0 tx=0x7f40c402a670 comp rx=0 tx=0).stop 2024-06-29T15:55:23.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.852+0000 7f40defc7640 1 -- 172.21.15.81:0/1912077785 shutdown_connections 2024-06-29T15:55:23.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.852+0000 7f40defc7640 1 --2- 172.21.15.81:0/1912077785 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40d80b5250 0x7f40d80d09c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.852+0000 7f40defc7640 1 --2- 172.21.15.81:0/1912077785 >> v2:172.21.15.167:3300/0 conn(0x7f40d80b9040 0x7f40d80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.852+0000 7f40defc7640 1 -- 172.21.15.81:0/1912077785 >> 172.21.15.81:0/1912077785 conn(0x7f40d809cf30 msgr2=0x7f40d80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:23.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.852+0000 7f40defc7640 1 -- 172.21.15.81:0/1912077785 shutdown_connections 2024-06-29T15:55:23.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.852+0000 7f40defc7640 1 -- 172.21.15.81:0/1912077785 wait complete. 2024-06-29T15:55:23.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.854+0000 7f40defc7640 1 Processor -- start 2024-06-29T15:55:23.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.854+0000 7f40defc7640 1 -- start start 2024-06-29T15:55:23.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.855+0000 7f40defc7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40d80b5250 0x7f40d815d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.855+0000 7f40defc7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40d80b7290 0x7f40d81f50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.855+0000 7f40defc7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40d80b9040 0x7f40d81f5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40defc7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f40d80b7db0 con 0x7f40d80b7290 2024-06-29T15:55:23.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40defc7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f40d80d2d50 con 0x7f40d80b9040 2024-06-29T15:55:23.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40defc7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f40d80b7f30 con 0x7f40d80b5250 2024-06-29T15:55:23.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40d80b7290 0x7f40d81f50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40d80b7290 0x7f40d81f50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53516/0 (socket says 172.21.15.81:53516) 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d7fff640 1 -- 172.21.15.81:0/2616528356 learned_addr learned my addr 172.21.15.81:0/2616528356 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40dcd63640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40d80b5250 0x7f40d815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40dd564640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40d80b9040 0x7f40d81f5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d7fff640 1 -- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40d80b5250 msgr2=0x7f40d815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d7fff640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40d80b5250 0x7f40d815d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d7fff640 1 -- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40d80b9040 msgr2=0x7f40d81f5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d7fff640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40d80b9040 0x7f40d81f5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d7fff640 1 -- 172.21.15.81:0/2616528356 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d81e7240 con 0x7f40d80b7290 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d7fff640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40d80b7290 0x7f40d81f50c0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f40c4000c00 tx=0x7f40c40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d57fa640 1 -- 172.21.15.81:0/2616528356 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c403e030 con 0x7f40d80b7290 2024-06-29T15:55:23.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d57fa640 1 -- 172.21.15.81:0/2616528356 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40c4003db0 con 0x7f40d80b7290 2024-06-29T15:55:23.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40defc7640 1 -- 172.21.15.81:0/2616528356 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d815dab0 con 0x7f40d80b7290 2024-06-29T15:55:23.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.856+0000 7f40d57fa640 1 -- 172.21.15.81:0/2616528356 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c4035040 con 0x7f40d80b7290 2024-06-29T15:55:23.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.857+0000 7f40defc7640 1 -- 172.21.15.81:0/2616528356 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d80b3e00 con 0x7f40d80b7290 2024-06-29T15:55:23.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.858+0000 7f40defc7640 1 -- 172.21.15.81:0/2616528356 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d80b84c0 con 0x7f40d80b7290 2024-06-29T15:55:23.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.858+0000 7f40d57fa640 1 -- 172.21.15.81:0/2616528356 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 113) ==== 98315+0+0 (secure 0 0 0) 0x7f40c4041030 con 0x7f40d80b7290 2024-06-29T15:55:23.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.858+0000 7f40d57fa640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f40bc0763f0 0x7f40bc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:23.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.859+0000 7f40dcd63640 1 -- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f40bc0763f0 msgr2=0x7f40bc078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1431883924 2024-06-29T15:55:23.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.859+0000 7f40d57fa640 1 -- 172.21.15.81:0/2616528356 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(184..184 src has 1..184) ==== 6553+0+0 (secure 0 0 0) 0x7f40c40b84a0 con 0x7f40d80b7290 2024-06-29T15:55:23.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.859+0000 7f40dcd63640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f40bc0763f0 0x7f40bc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:23.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.863+0000 7f40d57fa640 1 -- 172.21.15.81:0/2616528356 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f40c4081130 con 0x7f40d80b7290 2024-06-29T15:55:23.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.912+0000 7f40d57fa640 1 -- 172.21.15.81:0/2616528356 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 114) ==== 97912+0+0 (secure 0 0 0) 0x7f40c40b6670 con 0x7f40d80b7290 2024-06-29T15:55:23.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.912+0000 7f40d57fa640 1 -- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f40bc0763f0 msgr2=0x7f40bc078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:23.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:23.912+0000 7f40d57fa640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f40bc0763f0 0x7f40bc078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:23.966 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:23.964+0000 7fb536b95640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.966 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:23.964+0000 7fb536b95640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.966 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:23.964+0000 7fb536b95640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.966 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:23.964+0000 7fb536b95640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:23.964+0000 7fb536b95640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:23.964+0000 7fb539b9b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:23.964+0000 7fb539b9b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:23.964+0000 7fb539b9b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:23.964+0000 7fb539b9b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:23.967 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:23.964+0000 7fb539b9b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:24.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.055+0000 7f40defc7640 1 -- 172.21.15.81:0/2616528356 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f40d80b3b20 con 0x7f40d80b7290 2024-06-29T15:55:24.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.055+0000 7f40d57fa640 1 -- 172.21.15.81:0/2616528356 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v185) ==== 100+0+11 (secure 0 0 0) 0x7f40c409b900 con 0x7f40d80b7290 2024-06-29T15:55:24.056 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:55:24.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.057+0000 7f40defc7640 1 -- 172.21.15.81:0/2616528356 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40d80b7290 msgr2=0x7f40d81f50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.057+0000 7f40defc7640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40d80b7290 0x7f40d81f50c0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f40c4000c00 tx=0x7f40c40038a0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.057+0000 7f40defc7640 1 -- 172.21.15.81:0/2616528356 shutdown_connections 2024-06-29T15:55:24.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.057+0000 7f40defc7640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:6832/1431883924,v1:172.21.15.167:6833/1431883924] conn(0x7f40bc0763f0 0x7f40bc078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.057+0000 7f40defc7640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40d80b9040 0x7f40d81f5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.057+0000 7f40defc7640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40d80b7290 0x7f40d81f50c0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.057+0000 7f40defc7640 1 --2- 172.21.15.81:0/2616528356 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40d80b5250 0x7f40d815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.057+0000 7f40defc7640 1 -- 172.21.15.81:0/2616528356 >> 172.21.15.81:0/2616528356 conn(0x7f40d809cf30 msgr2=0x7f40d80b8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.057+0000 7f40defc7640 1 -- 172.21.15.81:0/2616528356 shutdown_connections 2024-06-29T15:55:24.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.057+0000 7f40defc7640 1 -- 172.21.15.81:0/2616528356 wait complete. 2024-06-29T15:55:24.069 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:24.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.183+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/352864605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec074520 msgr2=0x7fb8ec06f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.183+0000 7fb8f3a7f640 1 --2- 172.21.15.81:0/352864605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec074520 0x7fb8ec06f140 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec06e210 tx=0x7fb8e802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.183+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/352864605 shutdown_connections 2024-06-29T15:55:24.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.183+0000 7fb8f3a7f640 1 --2- 172.21.15.81:0/352864605 >> v2:172.21.15.167:3300/0 conn(0x7fb8ec06f680 0x7fb8ec06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.183+0000 7fb8f3a7f640 1 --2- 172.21.15.81:0/352864605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec074520 0x7fb8ec06f140 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.183+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/352864605 >> 172.21.15.81:0/352864605 conn(0x7fb8ec072950 msgr2=0x7fb8ec0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.183+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/352864605 shutdown_connections 2024-06-29T15:55:24.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.183+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/352864605 wait complete. 2024-06-29T15:55:24.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.185+0000 7fb8f3a7f640 1 Processor -- start 2024-06-29T15:55:24.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.185+0000 7fb8f3a7f640 1 -- start start 2024-06-29T15:55:24.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f3a7f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8ec06f680 0x7fb8ec0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f3a7f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8ec06fa90 0x7fb8ec0cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f3a7f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec074520 0x7fb8ec0d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f3a7f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec15b010 con 0x7fb8ec074520 2024-06-29T15:55:24.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f3a7f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec06d150 con 0x7fb8ec06fa90 2024-06-29T15:55:24.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f3a7f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb8ec15b190 con 0x7fb8ec06f680 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f201c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec074520 0x7fb8ec0d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f201c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec074520 0x7fb8ec0d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53534/0 (socket says 172.21.15.81:53534) 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f201c640 1 -- 172.21.15.81:0/1548017548 learned_addr learned my addr 172.21.15.81:0/1548017548 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f101a640 1 --2- 172.21.15.81:0/1548017548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8ec06fa90 0x7fb8ec0cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f181b640 1 --2- 172.21.15.81:0/1548017548 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8ec06f680 0x7fb8ec0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f201c640 1 -- 172.21.15.81:0/1548017548 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8ec06f680 msgr2=0x7fb8ec0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f201c640 1 --2- 172.21.15.81:0/1548017548 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8ec06f680 0x7fb8ec0cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f201c640 1 -- 172.21.15.81:0/1548017548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8ec06fa90 msgr2=0x7fb8ec0cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f201c640 1 --2- 172.21.15.81:0/1548017548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8ec06fa90 0x7fb8ec0cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f201c640 1 -- 172.21.15.81:0/1548017548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec1f5350 con 0x7fb8ec074520 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.186+0000 7fb8f181b640 1 --2- 172.21.15.81:0/1548017548 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8ec06f680 0x7fb8ec0cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.187+0000 7fb8f201c640 1 --2- 172.21.15.81:0/1548017548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec074520 0x7fb8ec0d0030 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400c700 tx=0x7fb8e400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.187+0000 7fb8e27fc640 1 -- 172.21.15.81:0/1548017548 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4003b70 con 0x7fb8ec074520 2024-06-29T15:55:24.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.187+0000 7fb8e27fc640 1 -- 172.21.15.81:0/1548017548 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8e4003d10 con 0x7fb8ec074520 2024-06-29T15:55:24.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.187+0000 7fb8e27fc640 1 -- 172.21.15.81:0/1548017548 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e400aab0 con 0x7fb8ec074520 2024-06-29T15:55:24.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.187+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/1548017548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ec0cbb50 con 0x7fb8ec074520 2024-06-29T15:55:24.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.187+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/1548017548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ec0ca080 con 0x7fb8ec074520 2024-06-29T15:55:24.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.188+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/1548017548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8ec0af4d0 con 0x7fb8ec074520 2024-06-29T15:55:24.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.189+0000 7fb8e27fc640 1 -- 172.21.15.81:0/1548017548 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 114) ==== 97912+0+0 (secure 0 0 0) 0x7fb8e400ac50 con 0x7fb8ec074520 2024-06-29T15:55:24.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.189+0000 7fb8e27fc640 1 -- 172.21.15.81:0/1548017548 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 6869+0+0 (secure 0 0 0) 0x7fb8e4097540 con 0x7fb8ec074520 2024-06-29T15:55:24.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.194+0000 7fb8e27fc640 1 -- 172.21.15.81:0/1548017548 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb8ec1f5350 con 0x7fb8ec074520 2024-06-29T15:55:24.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.409+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/1548017548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8ec0cfd20 con 0x7fb8ec074520 2024-06-29T15:55:24.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.410+0000 7fb8e27fc640 1 -- 172.21.15.81:0/1548017548 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+5237 (secure 0 0 0) 0x7fb8e4060dd0 con 0x7fb8ec074520 2024-06-29T15:55:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:24.411 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":95,"btime":"2024-06-29T15:39:15:192034+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2580108037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2580108037},{"type":"v1","addr":"172.21.15.167:6836","nonce":2580108037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":10155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3892298237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3892298237},{"type":"v1","addr":"172.21.15.81:6834","nonce":3892298237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":10158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2457574359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2457574359},{"type":"v1","addr":"172.21.15.81:6835","nonce":2457574359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.526747+0000","modified":"2024-06-29T15:39:14.562127+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10170},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10170":{"gid":10170,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/446857148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":446857148},{"type":"v1","addr":"172.21.15.167:6837","nonce":446857148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10170,"qdb_cluster":[10170]},"id":11}]} 2024-06-29T15:55:24.411 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 95 2024-06-29T15:55:24.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.412+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/1548017548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec074520 msgr2=0x7fb8ec0d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.412+0000 7fb8f3a7f640 1 --2- 172.21.15.81:0/1548017548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec074520 0x7fb8ec0d0030 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400c700 tx=0x7fb8e400cbc0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.412+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/1548017548 shutdown_connections 2024-06-29T15:55:24.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.412+0000 7fb8f3a7f640 1 --2- 172.21.15.81:0/1548017548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec074520 0x7fb8ec0d0030 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.412+0000 7fb8f3a7f640 1 --2- 172.21.15.81:0/1548017548 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb8ec06fa90 0x7fb8ec0cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.412+0000 7fb8f3a7f640 1 --2- 172.21.15.81:0/1548017548 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8ec06f680 0x7fb8ec0cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.412+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/1548017548 >> 172.21.15.81:0/1548017548 conn(0x7fb8ec072950 msgr2=0x7fb8ec156fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.412+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/1548017548 shutdown_connections 2024-06-29T15:55:24.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.412+0000 7fb8f3a7f640 1 -- 172.21.15.81:0/1548017548 wait complete. 2024-06-29T15:55:24.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:24.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.536+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2582445670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c15e960 msgr2=0x7f5c2c15ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.536+0000 7f5c32d1c640 1 --2- 172.21.15.81:0/2582445670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c15e960 0x7f5c2c15ed70 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f5c24005fd0 tx=0x7f5c2402a670 comp rx=0 tx=0).stop 2024-06-29T15:55:24.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.536+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2582445670 shutdown_connections 2024-06-29T15:55:24.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.536+0000 7f5c32d1c640 1 --2- 172.21.15.81:0/2582445670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c15e960 0x7f5c2c15ed70 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.536+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2582445670 >> 172.21.15.81:0/2582445670 conn(0x7f5c2c06ccc0 msgr2=0x7f5c2c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.536+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2582445670 shutdown_connections 2024-06-29T15:55:24.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.536+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2582445670 wait complete. 2024-06-29T15:55:24.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.538+0000 7f5c32d1c640 1 Processor -- start 2024-06-29T15:55:24.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.538+0000 7f5c32d1c640 1 -- start start 2024-06-29T15:55:24.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.539+0000 7f5c32d1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c073030 0x7f5c2c1f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.539+0000 7f5c32d1c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c2c1412f0 0x7f5c2c154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.539+0000 7f5c32d1c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c2c15e960 0x7f5c2c140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.539+0000 7f5c32d1c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c0b3390 con 0x7f5c2c073030 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.539+0000 7f5c32d1c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c0b3210 con 0x7f5c2c1412f0 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.539+0000 7f5c32d1c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5c2c0b3510 con 0x7f5c2c15e960 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.539+0000 7f5c30ab8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c073030 0x7f5c2c1f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.539+0000 7f5c30ab8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c073030 0x7f5c2c1f5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53556/0 (socket says 172.21.15.81:53556) 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.539+0000 7f5c30ab8640 1 -- 172.21.15.81:0/2094795634 learned_addr learned my addr 172.21.15.81:0/2094795634 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c312b9640 1 --2- 172.21.15.81:0/2094795634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c2c15e960 0x7f5c2c140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c2bfff640 1 --2- 172.21.15.81:0/2094795634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c2c1412f0 0x7f5c2c154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c30ab8640 1 -- 172.21.15.81:0/2094795634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c2c15e960 msgr2=0x7f5c2c140370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c30ab8640 1 --2- 172.21.15.81:0/2094795634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c2c15e960 0x7f5c2c140370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c30ab8640 1 -- 172.21.15.81:0/2094795634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c2c1412f0 msgr2=0x7f5c2c154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c30ab8640 1 --2- 172.21.15.81:0/2094795634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c2c1412f0 0x7f5c2c154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c30ab8640 1 -- 172.21.15.81:0/2094795634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2c1552a0 con 0x7f5c2c073030 2024-06-29T15:55:24.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c312b9640 1 --2- 172.21.15.81:0/2094795634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c2c15e960 0x7f5c2c140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:24.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c30ab8640 1 --2- 172.21.15.81:0/2094795634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c073030 0x7f5c2c1f5f70 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00c3a0 tx=0x7f5c1c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:24.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c297fa640 1 -- 172.21.15.81:0/2094795634 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c017070 con 0x7f5c2c073030 2024-06-29T15:55:24.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c297fa640 1 -- 172.21.15.81:0/2094795634 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c1c0053c0 con 0x7f5c2c073030 2024-06-29T15:55:24.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2094795634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c2c157b50 con 0x7f5c2c073030 2024-06-29T15:55:24.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c297fa640 1 -- 172.21.15.81:0/2094795634 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c003850 con 0x7f5c2c073030 2024-06-29T15:55:24.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.540+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2094795634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c2c1f5770 con 0x7f5c2c073030 2024-06-29T15:55:24.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.541+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2094795634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c2c0b3a30 con 0x7f5c2c073030 2024-06-29T15:55:24.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.542+0000 7f5c297fa640 1 -- 172.21.15.81:0/2094795634 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 114) ==== 97912+0+0 (secure 0 0 0) 0x7f5c1c01a070 con 0x7f5c2c073030 2024-06-29T15:55:24.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.542+0000 7f5c297fa640 1 -- 172.21.15.81:0/2094795634 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 6869+0+0 (secure 0 0 0) 0x7f5c1c096a90 con 0x7f5c2c073030 2024-06-29T15:55:24.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.546+0000 7f5c297fa640 1 -- 172.21.15.81:0/2094795634 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5c1c0603b0 con 0x7f5c2c073030 2024-06-29T15:55:24.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.730+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2094795634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c2c068c30 con 0x7f5c2c073030 2024-06-29T15:55:24.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.732+0000 7f5c297fa640 1 -- 172.21.15.81:0/2094795634 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) ==== 74+0+18413 (secure 0 0 0) 0x7f5c1c0683a0 con 0x7f5c2c073030 2024-06-29T15:55:24.732 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:24.733 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":185,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:55:23.904119+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.821386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.484738+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":184,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3903269557":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/3685877654":"2024-06-30T15:55:23.904081+0000","172.21.15.167:6833/1431883924":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/1938597900":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/321095347":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/1637259453":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3006573425":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/578369376":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6833/2288011293":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6832/2288011293":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6833/2546364927":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/3941545005":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3712811511":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1119662655":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/955369459":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1088263349":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/153744272":"2024-06-30T15:39:23.843781+0000","172.21.15.167:6832/1431883924":"2024-06-30T15:55:23.904081+0000","172.21.15.167:6832/2546364927":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/2062891067":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/3310646322":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1870993879":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3737424986":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/1345841177":"2024-06-30T15:39:23.843781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:24.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.734+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2094795634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c073030 msgr2=0x7f5c2c1f5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.734+0000 7f5c32d1c640 1 --2- 172.21.15.81:0/2094795634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c073030 0x7f5c2c1f5f70 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00c3a0 tx=0x7f5c1c00c860 comp rx=0 tx=0).stop 2024-06-29T15:55:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.734+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2094795634 shutdown_connections 2024-06-29T15:55:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.734+0000 7f5c32d1c640 1 --2- 172.21.15.81:0/2094795634 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c2c15e960 0x7f5c2c140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.734+0000 7f5c32d1c640 1 --2- 172.21.15.81:0/2094795634 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c2c1412f0 0x7f5c2c154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.734+0000 7f5c32d1c640 1 --2- 172.21.15.81:0/2094795634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c2c073030 0x7f5c2c1f5f70 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.734+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2094795634 >> 172.21.15.81:0/2094795634 conn(0x7f5c2c06ccc0 msgr2=0x7f5c2c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.734+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2094795634 shutdown_connections 2024-06-29T15:55:24.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.734+0000 7f5c32d1c640 1 -- 172.21.15.81:0/2094795634 wait complete. 2024-06-29T15:55:24.746 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:55:24.746 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:24.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.858+0000 7f74f30e4640 1 -- 172.21.15.81:0/4069720771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74ec15e960 msgr2=0x7f74ec15ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.858+0000 7f74f30e4640 1 --2- 172.21.15.81:0/4069720771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74ec15e960 0x7f74ec15ed70 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f74e4005fd0 tx=0x7f74e402ab40 comp rx=0 tx=0).stop 2024-06-29T15:55:24.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.858+0000 7f74f30e4640 1 -- 172.21.15.81:0/4069720771 shutdown_connections 2024-06-29T15:55:24.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.858+0000 7f74f30e4640 1 --2- 172.21.15.81:0/4069720771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74ec15e960 0x7f74ec15ed70 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.858+0000 7f74f30e4640 1 -- 172.21.15.81:0/4069720771 >> 172.21.15.81:0/4069720771 conn(0x7f74ec06ccc0 msgr2=0x7f74ec0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:24.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.858+0000 7f74f30e4640 1 -- 172.21.15.81:0/4069720771 shutdown_connections 2024-06-29T15:55:24.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.858+0000 7f74f30e4640 1 -- 172.21.15.81:0/4069720771 wait complete. 2024-06-29T15:55:24.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.860+0000 7f74f30e4640 1 Processor -- start 2024-06-29T15:55:24.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.860+0000 7f74f30e4640 1 -- start start 2024-06-29T15:55:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.861+0000 7f74f30e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74ec073030 0x7f74ec1f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.861+0000 7f74f30e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74ec1412f0 0x7f74ec0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.861+0000 7f74f30e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74ec15e960 0x7f74ec0ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.861+0000 7f74f30e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74ec0b35a0 con 0x7f74ec1412f0 2024-06-29T15:55:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.861+0000 7f74f30e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f74ec0b3420 con 0x7f74ec073030 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.861+0000 7f74f30e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f74ec0b3720 con 0x7f74ec15e960 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74ec1412f0 0x7f74ec0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74ec1412f0 0x7f74ec0ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53576/0 (socket says 172.21.15.81:53576) 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e3fff640 1 -- 172.21.15.81:0/3608454503 learned_addr learned my addr 172.21.15.81:0/3608454503 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74f0e80640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74ec073030 0x7f74ec1f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74f1681640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74ec15e960 0x7f74ec0ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e3fff640 1 -- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74ec15e960 msgr2=0x7f74ec0ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e3fff640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74ec15e960 0x7f74ec0ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e3fff640 1 -- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74ec073030 msgr2=0x7f74ec1f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e3fff640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74ec073030 0x7f74ec1f52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e3fff640 1 -- 172.21.15.81:0/3608454503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74ec1f6460 con 0x7f74ec1412f0 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e3fff640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74ec1412f0 0x7f74ec0ad8d0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f74dc00c430 tx=0x7f74dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:24.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e17fa640 1 -- 172.21.15.81:0/3608454503 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc017070 con 0x7f74ec1412f0 2024-06-29T15:55:24.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e17fa640 1 -- 172.21.15.81:0/3608454503 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74dc0053c0 con 0x7f74ec1412f0 2024-06-29T15:55:24.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74f30e4640 1 -- 172.21.15.81:0/3608454503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74ec158120 con 0x7f74ec1412f0 2024-06-29T15:55:24.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74e17fa640 1 -- 172.21.15.81:0/3608454503 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc003850 con 0x7f74ec1412f0 2024-06-29T15:55:24.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.862+0000 7f74f30e4640 1 -- 172.21.15.81:0/3608454503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74ec1586e0 con 0x7f74ec1412f0 2024-06-29T15:55:24.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.863+0000 7f74f30e4640 1 -- 172.21.15.81:0/3608454503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74ec15ed70 con 0x7f74ec1412f0 2024-06-29T15:55:24.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.864+0000 7f74e17fa640 1 -- 172.21.15.81:0/3608454503 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 114) ==== 97912+0+0 (secure 0 0 0) 0x7f74dc01a070 con 0x7f74ec1412f0 2024-06-29T15:55:24.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.865+0000 7f74e17fa640 1 -- 172.21.15.81:0/3608454503 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 6869+0+0 (secure 0 0 0) 0x7f74dc096ac0 con 0x7f74ec1412f0 2024-06-29T15:55:24.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.869+0000 7f74e17fa640 1 -- 172.21.15.81:0/3608454503 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f74dc05fd00 con 0x7f74ec1412f0 2024-06-29T15:55:24.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.914+0000 7f74e17fa640 1 -- 172.21.15.81:0/3608454503 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 115) ==== 98039+0+0 (secure 0 0 0) 0x7f74dc05b670 con 0x7f74ec1412f0 2024-06-29T15:55:24.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.914+0000 7f74e17fa640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f74d807be20 0x7f74d807e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:24.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.918+0000 7f74f0e80640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f74d807be20 0x7f74d807e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:24.919+0000 7f74f0e80640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f74d807be20 0x7f74d807e240 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f74e8001720 tx=0x7f74e8009040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.091+0000 7f74f30e4640 1 -- 172.21.15.81:0/3608454503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74ec157e70 con 0x7f74ec1412f0 2024-06-29T15:55:25.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.091+0000 7f74e17fa640 1 -- 172.21.15.81:0/3608454503 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+5237 (secure 0 0 0) 0x7f74dc06d570 con 0x7f74ec1412f0 2024-06-29T15:55:25.092 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:25.092 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":95,"btime":"2024-06-29T15:39:15:192034+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2580108037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2580108037},{"type":"v1","addr":"172.21.15.167:6836","nonce":2580108037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":10155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3892298237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3892298237},{"type":"v1","addr":"172.21.15.81:6834","nonce":3892298237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":10158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2457574359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2457574359},{"type":"v1","addr":"172.21.15.81:6835","nonce":2457574359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.526747+0000","modified":"2024-06-29T15:39:14.562127+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10170},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10170":{"gid":10170,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/446857148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":446857148},{"type":"v1","addr":"172.21.15.167:6837","nonce":446857148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10170,"qdb_cluster":[10170]},"id":11}]} 2024-06-29T15:55:25.092 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 95 2024-06-29T15:55:25.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 -- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f74d807be20 msgr2=0x7f74d807e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f74d807be20 0x7f74d807e240 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f74e8001720 tx=0x7f74e8009040 comp rx=0 tx=0).stop 2024-06-29T15:55:25.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 -- 172.21.15.81:0/3608454503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74ec1412f0 msgr2=0x7f74ec0ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74ec1412f0 0x7f74ec0ad8d0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f74dc00c430 tx=0x7f74dc00c8f0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 -- 172.21.15.81:0/3608454503 shutdown_connections 2024-06-29T15:55:25.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f74d807be20 0x7f74d807e240 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74ec15e960 0x7f74ec0ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74ec1412f0 0x7f74ec0ad8d0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 --2- 172.21.15.81:0/3608454503 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74ec073030 0x7f74ec1f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 -- 172.21.15.81:0/3608454503 >> 172.21.15.81:0/3608454503 conn(0x7f74ec06ccc0 msgr2=0x7f74ec06dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 -- 172.21.15.81:0/3608454503 shutdown_connections 2024-06-29T15:55:25.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.093+0000 7f74f30e4640 1 -- 172.21.15.81:0/3608454503 wait complete. 2024-06-29T15:55:25.105 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.218+0000 7f883087f640 1 -- 172.21.15.81:0/487012251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8818003c30 msgr2=0x7f8818004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.218+0000 7f883087f640 1 --2- 172.21.15.81:0/487012251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8818003c30 0x7f8818004080 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f881c009b50 tx=0x7f8818005830 comp rx=0 tx=0).stop 2024-06-29T15:55:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.218+0000 7f883087f640 1 -- 172.21.15.81:0/487012251 shutdown_connections 2024-06-29T15:55:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.218+0000 7f883087f640 1 --2- 172.21.15.81:0/487012251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8818003c30 0x7f8818004080 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.218+0000 7f883087f640 1 -- 172.21.15.81:0/487012251 >> 172.21.15.81:0/487012251 conn(0x7f882c09cf30 msgr2=0x7f882c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.218+0000 7f883087f640 1 -- 172.21.15.81:0/487012251 shutdown_connections 2024-06-29T15:55:25.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.218+0000 7f883087f640 1 -- 172.21.15.81:0/487012251 wait complete. 2024-06-29T15:55:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.221+0000 7f883087f640 1 Processor -- start 2024-06-29T15:55:25.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.221+0000 7f883087f640 1 -- start start 2024-06-29T15:55:25.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f883087f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8818003c30 0x7f882c0a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f883087f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f882c0b36d0 0x7f882c0b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f883087f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f882c0b4020 0x7f882c1f4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f883087f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f882c0d0810 con 0x7f8818003c30 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f883087f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f882c0d0690 con 0x7f882c0b36d0 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f883087f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f882c0d0990 con 0x7f882c0b4020 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f882a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8818003c30 0x7f882c0a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f882a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8818003c30 0x7f882c0a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53584/0 (socket says 172.21.15.81:53584) 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f882a59c640 1 -- 172.21.15.81:0/635980432 learned_addr learned my addr 172.21.15.81:0/635980432 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f8821d9b640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f882c0b36d0 0x7f882c0b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f882ad9d640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f882c0b4020 0x7f882c1f4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f882a59c640 1 -- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f882c0b4020 msgr2=0x7f882c1f4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f882a59c640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f882c0b4020 0x7f882c1f4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f882a59c640 1 -- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f882c0b36d0 msgr2=0x7f882c0b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f882a59c640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f882c0b36d0 0x7f882c0b3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f882a59c640 1 -- 172.21.15.81:0/635980432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882c0a0960 con 0x7f8818003c30 2024-06-29T15:55:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.222+0000 7f8821d9b640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f882c0b36d0 0x7f882c0b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.223+0000 7f882a59c640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8818003c30 0x7f882c0a0420 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f881c009a70 tx=0x7f881c0336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.223+0000 7f88237fe640 1 -- 172.21.15.81:0/635980432 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881c038040 con 0x7f8818003c30 2024-06-29T15:55:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.223+0000 7f88237fe640 1 -- 172.21.15.81:0/635980432 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f881c0339d0 con 0x7f8818003c30 2024-06-29T15:55:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.223+0000 7f883087f640 1 -- 172.21.15.81:0/635980432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f882c0a0e30 con 0x7f8818003c30 2024-06-29T15:55:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.223+0000 7f88237fe640 1 -- 172.21.15.81:0/635980432 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881c033e20 con 0x7f8818003c30 2024-06-29T15:55:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.223+0000 7f883087f640 1 -- 172.21.15.81:0/635980432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f882c0a13f0 con 0x7f8818003c30 2024-06-29T15:55:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.224+0000 7f883087f640 1 -- 172.21.15.81:0/635980432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f882c0d1a80 con 0x7f8818003c30 2024-06-29T15:55:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.225+0000 7f88237fe640 1 -- 172.21.15.81:0/635980432 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 115) ==== 98039+0+0 (secure 0 0 0) 0x7f881c043020 con 0x7f8818003c30 2024-06-29T15:55:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.225+0000 7f88237fe640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f880c076120 0x7f880c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.225+0000 7f88237fe640 1 -- 172.21.15.81:0/635980432 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 6869+0+0 (secure 0 0 0) 0x7f881c0b7ac0 con 0x7f8818003c30 2024-06-29T15:55:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.225+0000 7f8821d9b640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f880c076120 0x7f880c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.226+0000 7f8821d9b640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f880c076120 0x7f880c078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f882c0a0220 tx=0x7f8814006b00 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.229+0000 7f88237fe640 1 -- 172.21.15.81:0/635980432 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f881c04a020 con 0x7f8818003c30 2024-06-29T15:55:25.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.445+0000 7f883087f640 1 -- 172.21.15.81:0/635980432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f882c0686b0 con 0x7f8818003c30 2024-06-29T15:55:25.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.446+0000 7f88237fe640 1 -- 172.21.15.81:0/635980432 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+5237 (secure 0 0 0) 0x7f881c0892c0 con 0x7f8818003c30 2024-06-29T15:55:25.447 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:25.447 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":95,"btime":"2024-06-29T15:39:15:192034+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2580108037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2580108037},{"type":"v1","addr":"172.21.15.167:6836","nonce":2580108037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":10155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3892298237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3892298237},{"type":"v1","addr":"172.21.15.81:6834","nonce":3892298237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":10158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2457574359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2457574359},{"type":"v1","addr":"172.21.15.81:6835","nonce":2457574359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:39:11.526747+0000","modified":"2024-06-29T15:39:14.562127+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10170},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10170":{"gid":10170,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/446857148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":446857148},{"type":"v1","addr":"172.21.15.167:6837","nonce":446857148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10170,"qdb_cluster":[10170]},"id":11}]} 2024-06-29T15:55:25.447 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 95 2024-06-29T15:55:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 -- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f880c076120 msgr2=0x7f880c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f880c076120 0x7f880c078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f882c0a0220 tx=0x7f8814006b00 comp rx=0 tx=0).stop 2024-06-29T15:55:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 -- 172.21.15.81:0/635980432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8818003c30 msgr2=0x7f882c0a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8818003c30 0x7f882c0a0420 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f881c009a70 tx=0x7f881c0336f0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 -- 172.21.15.81:0/635980432 shutdown_connections 2024-06-29T15:55:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f880c076120 0x7f880c078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f882c0b4020 0x7f882c1f4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f882c0b36d0 0x7f882c0b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 --2- 172.21.15.81:0/635980432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8818003c30 0x7f882c0a0420 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 -- 172.21.15.81:0/635980432 >> 172.21.15.81:0/635980432 conn(0x7f882c09cf30 msgr2=0x7f882c0ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 -- 172.21.15.81:0/635980432 shutdown_connections 2024-06-29T15:55:25.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.448+0000 7f883087f640 1 -- 172.21.15.81:0/635980432 wait complete. 2024-06-29T15:55:25.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:25.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.572+0000 7f9048139640 1 -- 172.21.15.81:0/3333899986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90400cefc0 msgr2=0x7f9040151030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.572+0000 7f9048139640 1 --2- 172.21.15.81:0/3333899986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90400cefc0 0x7f9040151030 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f9040158130 tx=0x7f903c02a5f0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.572+0000 7f9048139640 1 -- 172.21.15.81:0/3333899986 shutdown_connections 2024-06-29T15:55:25.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.572+0000 7f9048139640 1 --2- 172.21.15.81:0/3333899986 >> v2:172.21.15.167:3300/0 conn(0x7f904015e5f0 0x7f904006edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.572+0000 7f9048139640 1 --2- 172.21.15.81:0/3333899986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90400cefc0 0x7f9040151030 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.572+0000 7f9048139640 1 -- 172.21.15.81:0/3333899986 >> 172.21.15.81:0/3333899986 conn(0x7f904006ccc0 msgr2=0x7f90400b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.572+0000 7f9048139640 1 -- 172.21.15.81:0/3333899986 shutdown_connections 2024-06-29T15:55:25.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.572+0000 7f9048139640 1 -- 172.21.15.81:0/3333899986 wait complete. 2024-06-29T15:55:25.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.573+0000 7f9048139640 1 Processor -- start 2024-06-29T15:55:25.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.573+0000 7f9048139640 1 -- start start 2024-06-29T15:55:25.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f9048139640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f904006f310 0x7f904014ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f9048139640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90400cefc0 0x7f90400a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f9048139640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f904015e5f0 0x7f90400a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f9048139640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f904015b5a0 con 0x7f904015e5f0 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f9048139640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f904015b420 con 0x7f904006f310 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f9048139640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9040070010 con 0x7f90400cefc0 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90466d6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f904015e5f0 0x7f90400a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90466d6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f904015e5f0 0x7f90400a9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53606/0 (socket says 172.21.15.81:53606) 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90466d6640 1 -- 172.21.15.81:0/4290375152 learned_addr learned my addr 172.21.15.81:0/4290375152 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f9045ed5640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f904006f310 0x7f904014ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90456d4640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90400cefc0 0x7f90400a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90466d6640 1 -- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90400cefc0 msgr2=0x7f90400a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90466d6640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90400cefc0 0x7f90400a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90466d6640 1 -- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f904006f310 msgr2=0x7f904014ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90466d6640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f904006f310 0x7f904014ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90466d6640 1 -- 172.21.15.81:0/4290375152 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9040158850 con 0x7f904015e5f0 2024-06-29T15:55:25.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90466d6640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f904015e5f0 0x7f90400a9a00 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f9038002230 tx=0x7f903800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.575+0000 7f90367fc640 1 -- 172.21.15.81:0/4290375152 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9038003b70 con 0x7f904015e5f0 2024-06-29T15:55:25.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.576+0000 7f90367fc640 1 -- 172.21.15.81:0/4290375152 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9038003d10 con 0x7f904015e5f0 2024-06-29T15:55:25.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.576+0000 7f9048139640 1 -- 172.21.15.81:0/4290375152 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9040158b30 con 0x7f904015e5f0 2024-06-29T15:55:25.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.576+0000 7f90367fc640 1 -- 172.21.15.81:0/4290375152 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9038005330 con 0x7f904015e5f0 2024-06-29T15:55:25.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.576+0000 7f9048139640 1 -- 172.21.15.81:0/4290375152 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90401590f0 con 0x7f904015e5f0 2024-06-29T15:55:25.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.576+0000 7f9048139640 1 -- 172.21.15.81:0/4290375152 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9040151030 con 0x7f904015e5f0 2024-06-29T15:55:25.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.577+0000 7f90367fc640 1 -- 172.21.15.81:0/4290375152 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 115) ==== 98039+0+0 (secure 0 0 0) 0x7f90380054d0 con 0x7f904015e5f0 2024-06-29T15:55:25.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.578+0000 7f90367fc640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9024076150 0x7f9024078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.578+0000 7f90367fc640 1 -- 172.21.15.81:0/4290375152 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 6869+0+0 (secure 0 0 0) 0x7f9038097490 con 0x7f904015e5f0 2024-06-29T15:55:25.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.578+0000 7f9045ed5640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9024076150 0x7f9024078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.579+0000 7f9045ed5640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9024076150 0x7f9024078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f903c033000 tx=0x7f903c004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.582+0000 7f90367fc640 1 -- 172.21.15.81:0/4290375152 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f904006f720 con 0x7f904015e5f0 2024-06-29T15:55:25.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.764+0000 7f9048139640 1 -- 172.21.15.81:0/4290375152 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90400aff30 con 0x7f904015e5f0 2024-06-29T15:55:25.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.766+0000 7f90367fc640 1 -- 172.21.15.81:0/4290375152 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) ==== 74+0+18413 (secure 0 0 0) 0x7f9038060650 con 0x7f904015e5f0 2024-06-29T15:55:25.766 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:25.767 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":185,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:55:23.904119+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:39:09.821386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"180","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-06-29T15:39:10.484738+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"184","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":184,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3903269557":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/3685877654":"2024-06-30T15:55:23.904081+0000","172.21.15.167:6833/1431883924":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/1938597900":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/321095347":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/1637259453":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3006573425":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/578369376":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6833/2288011293":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6832/2288011293":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6833/2546364927":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/3941545005":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3712811511":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1119662655":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/955369459":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1088263349":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/153744272":"2024-06-30T15:39:23.843781+0000","172.21.15.167:6832/1431883924":"2024-06-30T15:55:23.904081+0000","172.21.15.167:6832/2546364927":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/2062891067":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/3310646322":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1870993879":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3737424986":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/1345841177":"2024-06-30T15:39:23.843781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 -- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9024076150 msgr2=0x7f9024078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9024076150 0x7f9024078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f903c033000 tx=0x7f903c004730 comp rx=0 tx=0).stop 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 -- 172.21.15.81:0/4290375152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f904015e5f0 msgr2=0x7f90400a9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f904015e5f0 0x7f90400a9a00 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f9038002230 tx=0x7f903800cb90 comp rx=0 tx=0).stop 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 -- 172.21.15.81:0/4290375152 shutdown_connections 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9024076150 0x7f9024078660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f904015e5f0 0x7f90400a9a00 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f90400cefc0 0x7f90400a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 --2- 172.21.15.81:0/4290375152 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f904006f310 0x7f904014ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 -- 172.21.15.81:0/4290375152 >> 172.21.15.81:0/4290375152 conn(0x7f904006ccc0 msgr2=0x7f90400b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 -- 172.21.15.81:0/4290375152 shutdown_connections 2024-06-29T15:55:25.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.768+0000 7f9048139640 1 -- 172.21.15.81:0/4290375152 wait complete. 2024-06-29T15:55:25.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:55:25.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.894+0000 7ff7e3290640 1 -- 172.21.15.81:0/1278817399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7c8005250 msgr2=0x7ff7c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.894+0000 7ff7e3290640 1 --2- 172.21.15.81:0/1278817399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7c8005250 0x7ff7c800b9c0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4001d90 tx=0x7ff7c8005830 comp rx=0 tx=0).stop 2024-06-29T15:55:25.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.895+0000 7ff7e3290640 1 -- 172.21.15.81:0/1278817399 shutdown_connections 2024-06-29T15:55:25.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.895+0000 7ff7e3290640 1 --2- 172.21.15.81:0/1278817399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7c8005250 0x7ff7c800b9c0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.895+0000 7ff7e3290640 1 -- 172.21.15.81:0/1278817399 >> 172.21.15.81:0/1278817399 conn(0x7ff7dc06ccc0 msgr2=0x7ff7dc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:25.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.895+0000 7ff7e3290640 1 -- 172.21.15.81:0/1278817399 shutdown_connections 2024-06-29T15:55:25.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.895+0000 7ff7e3290640 1 -- 172.21.15.81:0/1278817399 wait complete. 2024-06-29T15:55:25.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.896+0000 7ff7e3290640 1 Processor -- start 2024-06-29T15:55:25.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.896+0000 7ff7e3290640 1 -- start start 2024-06-29T15:55:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e3290640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7c8005250 0x7ff7dc0b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e3290640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7dc13e260 0x7ff7dc138040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e3290640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7dc1f4d00 0x7ff7dc1f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e3290640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc0731b0 con 0x7ff7dc13e260 2024-06-29T15:55:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e3290640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc073030 con 0x7ff7c8005250 2024-06-29T15:55:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e3290640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff7dc073330 con 0x7ff7dc1f4d00 2024-06-29T15:55:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e082b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7dc13e260 0x7ff7dc138040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e082b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7dc13e260 0x7ff7dc138040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53620/0 (socket says 172.21.15.81:53620) 2024-06-29T15:55:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e082b640 1 -- 172.21.15.81:0/1096874757 learned_addr learned my addr 172.21.15.81:0/1096874757 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e102c640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7c8005250 0x7ff7dc0b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e182d640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7dc1f4d00 0x7ff7dc1f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e082b640 1 -- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7dc1f4d00 msgr2=0x7ff7dc1f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e082b640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7dc1f4d00 0x7ff7dc1f5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e082b640 1 -- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7c8005250 msgr2=0x7ff7dc0b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e082b640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7c8005250 0x7ff7dc0b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e082b640 1 -- 172.21.15.81:0/1096874757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7dc0b1600 con 0x7ff7dc13e260 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e102c640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7c8005250 0x7ff7dc0b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e082b640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7dc13e260 0x7ff7dc138040 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc006680 tx=0x7ff7cc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7d1ffb640 1 -- 172.21.15.81:0/1096874757 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc015070 con 0x7ff7dc13e260 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7d1ffb640 1 -- 172.21.15.81:0/1096874757 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7cc011920 con 0x7ff7dc13e260 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7d1ffb640 1 -- 172.21.15.81:0/1096874757 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc011d70 con 0x7ff7dc13e260 2024-06-29T15:55:25.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.898+0000 7ff7e3290640 1 -- 172.21.15.81:0/1096874757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7dc0b0020 con 0x7ff7dc13e260 2024-06-29T15:55:25.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.899+0000 7ff7e3290640 1 -- 172.21.15.81:0/1096874757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7dc0b05e0 con 0x7ff7dc13e260 2024-06-29T15:55:25.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.900+0000 7ff7e3290640 1 -- 172.21.15.81:0/1096874757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7dc073890 con 0x7ff7dc13e260 2024-06-29T15:55:25.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.900+0000 7ff7d1ffb640 1 -- 172.21.15.81:0/1096874757 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 115) ==== 98039+0+0 (secure 0 0 0) 0x7ff7cc018070 con 0x7ff7dc13e260 2024-06-29T15:55:25.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.901+0000 7ff7d1ffb640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7ff7c0076120 0x7ff7c0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:25.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.901+0000 7ff7d1ffb640 1 -- 172.21.15.81:0/1096874757 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 6869+0+0 (secure 0 0 0) 0x7ff7cc098570 con 0x7ff7dc13e260 2024-06-29T15:55:25.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.901+0000 7ff7e102c640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7ff7c0076120 0x7ff7c0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:25.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.902+0000 7ff7e102c640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7ff7c0076120 0x7ff7c0078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8009a20 tx=0x7ff7d8005f90 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:25.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:25.905+0000 7ff7d1ffb640 1 -- 172.21.15.81:0/1096874757 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff7cc061cf0 con 0x7ff7dc13e260 2024-06-29T15:55:26.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.125+0000 7ff7e3290640 1 -- 172.21.15.81:0/1096874757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff7dc06f960 con 0x7ff7dc13e260 2024-06-29T15:55:26.126 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T15:55:26.126+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:55:26.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.127+0000 7ff7d1ffb640 1 -- 172.21.15.81:0/1096874757 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v96) ==== 161+0+0 (secure 0 0 0) 0x7ff7cc0615f0 con 0x7ff7dc13e260 2024-06-29T15:55:26.127 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:55:26.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 -- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7ff7c0076120 msgr2=0x7ff7c0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:26.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7ff7c0076120 0x7ff7c0078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8009a20 tx=0x7ff7d8005f90 comp rx=0 tx=0).stop 2024-06-29T15:55:26.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 -- 172.21.15.81:0/1096874757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7dc13e260 msgr2=0x7ff7dc138040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:26.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7dc13e260 0x7ff7dc138040 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc006680 tx=0x7ff7cc0039c0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 -- 172.21.15.81:0/1096874757 shutdown_connections 2024-06-29T15:55:26.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7ff7c0076120 0x7ff7c0078630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7dc1f4d00 0x7ff7dc1f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7dc13e260 0x7ff7dc138040 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 --2- 172.21.15.81:0/1096874757 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7c8005250 0x7ff7dc0b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 -- 172.21.15.81:0/1096874757 >> 172.21.15.81:0/1096874757 conn(0x7ff7dc06ccc0 msgr2=0x7ff7dc15d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:26.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 -- 172.21.15.81:0/1096874757 shutdown_connections 2024-06-29T15:55:26.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.129+0000 7ff7e3290640 1 -- 172.21.15.81:0/1096874757 wait complete. 2024-06-29T15:55:26.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:55:26.195 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T15:55:26.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.260+0000 7faf36ee4640 1 -- 172.21.15.81:0/2494651626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf18004800 msgr2=0x7faf18004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:26.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.260+0000 7faf36ee4640 1 --2- 172.21.15.81:0/2494651626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf18004800 0x7faf18004c10 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7faf3006ffe0 tx=0x7faf18005830 comp rx=0 tx=0).stop 2024-06-29T15:55:26.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.260+0000 7faf35481640 1 -- 172.21.15.81:0/2494651626 reap_dead start 2024-06-29T15:55:26.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.260+0000 7faf36ee4640 1 -- 172.21.15.81:0/2494651626 shutdown_connections 2024-06-29T15:55:26.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.260+0000 7faf36ee4640 1 -- 172.21.15.81:0/2494651626 >> 172.21.15.81:0/2494651626 conn(0x7faf300b3680 msgr2=0x7faf300b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:26.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.260+0000 7faf36ee4640 1 -- 172.21.15.81:0/2494651626 shutdown_connections 2024-06-29T15:55:26.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.260+0000 7faf36ee4640 1 -- 172.21.15.81:0/2494651626 wait complete. 2024-06-29T15:55:26.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.262+0000 7faf36ee4640 1 Processor -- start 2024-06-29T15:55:26.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.262+0000 7faf36ee4640 1 -- start start 2024-06-29T15:55:26.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf36ee4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf18004800 0x7faf3015b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:26.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf36ee4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf300b25b0 0x7faf300c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:26.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf36ee4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf300c7760 0x7faf300c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf36ee4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf300cc950 con 0x7faf300b25b0 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf36ee4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faf300cc7d0 con 0x7faf18004800 2024-06-29T15:55:26.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf36ee4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faf300b5890 con 0x7faf300c7760 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf27fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf300b25b0 0x7faf300c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf27fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf300b25b0 0x7faf300c7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53638/0 (socket says 172.21.15.81:53638) 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf27fff640 1 -- 172.21.15.81:0/1918116414 learned_addr learned my addr 172.21.15.81:0/1918116414 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf34c80640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf18004800 0x7faf3015b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.263+0000 7faf35481640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf300c7760 0x7faf300c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf27fff640 1 -- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf300c7760 msgr2=0x7faf300c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:26.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf27fff640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf300c7760 0x7faf300c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf27fff640 1 -- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf18004800 msgr2=0x7faf3015b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf27fff640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf18004800 0x7faf3015b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf27fff640 1 -- 172.21.15.81:0/1918116414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf300c7b70 con 0x7faf300b25b0 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf27fff640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf300b25b0 0x7faf300c7220 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7faf2002b020 tx=0x7faf200011e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf257fa640 1 -- 172.21.15.81:0/1918116414 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20032070 con 0x7faf300b25b0 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf257fa640 1 -- 172.21.15.81:0/1918116414 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf200038e0 con 0x7faf300b25b0 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf36ee4640 1 -- 172.21.15.81:0/1918116414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf300cb9a0 con 0x7faf300b25b0 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf257fa640 1 -- 172.21.15.81:0/1918116414 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20003d10 con 0x7faf300b25b0 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.264+0000 7faf36ee4640 1 -- 172.21.15.81:0/1918116414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf3009bfa0 con 0x7faf300b25b0 2024-06-29T15:55:26.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.265+0000 7faf36ee4640 1 -- 172.21.15.81:0/1918116414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf300b5180 con 0x7faf300b25b0 2024-06-29T15:55:26.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.266+0000 7faf257fa640 1 -- 172.21.15.81:0/1918116414 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 116) ==== 98247+0+0 (secure 0 0 0) 0x7faf20035020 con 0x7faf300b25b0 2024-06-29T15:55:26.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.266+0000 7faf257fa640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7faf1c076390 0x7faf1c0788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:26.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.267+0000 7faf257fa640 1 -- 172.21.15.81:0/1918116414 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 6955+0+0 (secure 0 0 0) 0x7faf200b8090 con 0x7faf300b25b0 2024-06-29T15:55:26.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.267+0000 7faf34c80640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7faf1c076390 0x7faf1c0788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:26.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.267+0000 7faf34c80640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7faf1c076390 0x7faf1c0788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009bb0 tx=0x7faf2c005f90 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:26.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.271+0000 7faf257fa640 1 -- 172.21.15.81:0/1918116414 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faf20046060 con 0x7faf300b25b0 2024-06-29T15:55:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:26.486+0000 7faf36ee4640 1 -- 172.21.15.81:0/1918116414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7faf3015f240 con 0x7faf300b25b0 2024-06-29T15:55:27.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.131+0000 7faf257fa640 1 -- 172.21.15.81:0/1918116414 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v97) ==== 104+0+0 (secure 0 0 0) 0x7faf300c7b70 con 0x7faf300b25b0 2024-06-29T15:55:27.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 -- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7faf1c076390 msgr2=0x7faf1c0788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:27.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7faf1c076390 0x7faf1c0788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009bb0 tx=0x7faf2c005f90 comp rx=0 tx=0).stop 2024-06-29T15:55:27.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 -- 172.21.15.81:0/1918116414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf300b25b0 msgr2=0x7faf300c7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:27.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf300b25b0 0x7faf300c7220 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7faf2002b020 tx=0x7faf200011e0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 -- 172.21.15.81:0/1918116414 shutdown_connections 2024-06-29T15:55:27.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7faf1c076390 0x7faf1c0788a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf300c7760 0x7faf300c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf300b25b0 0x7faf300c7220 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 --2- 172.21.15.81:0/1918116414 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf18004800 0x7faf3015b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 -- 172.21.15.81:0/1918116414 >> 172.21.15.81:0/1918116414 conn(0x7faf300b3680 msgr2=0x7faf30070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:27.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 -- 172.21.15.81:0/1918116414 shutdown_connections 2024-06-29T15:55:27.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.133+0000 7faf36ee4640 1 -- 172.21.15.81:0/1918116414 wait complete. 2024-06-29T15:55:27.147 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:55:27.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.270+0000 7fd347e95640 1 -- 172.21.15.81:0/3120721028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd34015abb0 msgr2=0x7fd340072ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:27.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.270+0000 7fd347e95640 1 --2- 172.21.15.81:0/3120721028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd34015abb0 0x7fd340072ea0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fd33c005fd0 tx=0x7fd33c02a980 comp rx=0 tx=0).stop 2024-06-29T15:55:27.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.270+0000 7fd347e95640 1 -- 172.21.15.81:0/3120721028 shutdown_connections 2024-06-29T15:55:27.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.270+0000 7fd347e95640 1 --2- 172.21.15.81:0/3120721028 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3400cefe0 0x7fd3400b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.270+0000 7fd347e95640 1 --2- 172.21.15.81:0/3120721028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd34015abb0 0x7fd340072ea0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.270+0000 7fd347e95640 1 -- 172.21.15.81:0/3120721028 >> 172.21.15.81:0/3120721028 conn(0x7fd34006cd60 msgr2=0x7fd3400b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:27.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.270+0000 7fd347e95640 1 -- 172.21.15.81:0/3120721028 shutdown_connections 2024-06-29T15:55:27.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.270+0000 7fd347e95640 1 -- 172.21.15.81:0/3120721028 wait complete. 2024-06-29T15:55:27.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.272+0000 7fd347e95640 1 Processor -- start 2024-06-29T15:55:27.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.272+0000 7fd347e95640 1 -- start start 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.273+0000 7fd347e95640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3400b3860 0x7fd3401f5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.273+0000 7fd347e95640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3400cefe0 0x7fd3401f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.273+0000 7fd347e95640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd34015abb0 0x7fd340144cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.273+0000 7fd347e95640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd34006fb50 con 0x7fd3400cefe0 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.273+0000 7fd347e95640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd34006f9d0 con 0x7fd34015abb0 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.273+0000 7fd347e95640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd34006fcd0 con 0x7fd3400b3860 2024-06-29T15:55:27.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.273+0000 7fd345430640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3400cefe0 0x7fd3401f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd345430640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3400cefe0 0x7fd3401f5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53654/0 (socket says 172.21.15.81:53654) 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd345430640 1 -- 172.21.15.81:0/1236820326 learned_addr learned my addr 172.21.15.81:0/1236820326 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd345c31640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3400b3860 0x7fd3401f5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd346432640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd34015abb0 0x7fd340144cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd345430640 1 -- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3400b3860 msgr2=0x7fd3401f5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd345430640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3400b3860 0x7fd3401f5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd345430640 1 -- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd34015abb0 msgr2=0x7fd340144cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd345430640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd34015abb0 0x7fd340144cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd345430640 1 -- 172.21.15.81:0/1236820326 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd340152980 con 0x7fd3400cefe0 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd345430640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3400cefe0 0x7fd3401f5a50 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fd33000e7c0 tx=0x7fd33000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd3367fc640 1 -- 172.21.15.81:0/1236820326 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33000ccc0 con 0x7fd3400cefe0 2024-06-29T15:55:27.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd3367fc640 1 -- 172.21.15.81:0/1236820326 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd330010430 con 0x7fd3400cefe0 2024-06-29T15:55:27.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd3367fc640 1 -- 172.21.15.81:0/1236820326 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd330010860 con 0x7fd3400cefe0 2024-06-29T15:55:27.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd347e95640 1 -- 172.21.15.81:0/1236820326 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3400a95d0 con 0x7fd3400cefe0 2024-06-29T15:55:27.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.274+0000 7fd347e95640 1 -- 172.21.15.81:0/1236820326 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3401541c0 con 0x7fd3400cefe0 2024-06-29T15:55:27.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.275+0000 7fd347e95640 1 -- 172.21.15.81:0/1236820326 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3400b3320 con 0x7fd3400cefe0 2024-06-29T15:55:27.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.276+0000 7fd3367fc640 1 -- 172.21.15.81:0/1236820326 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 116) ==== 98247+0+0 (secure 0 0 0) 0x7fd330010a00 con 0x7fd3400cefe0 2024-06-29T15:55:27.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.276+0000 7fd3367fc640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fd324076360 0x7fd324078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:27.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.276+0000 7fd3367fc640 1 -- 172.21.15.81:0/1236820326 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 6955+0+0 (secure 0 0 0) 0x7fd3300986a0 con 0x7fd3400cefe0 2024-06-29T15:55:27.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.276+0000 7fd345c31640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fd324076360 0x7fd324078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:27.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.277+0000 7fd345c31640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fd324076360 0x7fd324078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd33c035000 tx=0x7fd33c004ab0 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:27.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.281+0000 7fd3367fc640 1 -- 172.21.15.81:0/1236820326 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd3300613d0 con 0x7fd3400cefe0 2024-06-29T15:55:27.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:27.462+0000 7fd347e95640 1 -- 172.21.15.81:0/1236820326 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd340145ae0 con 0x7fd3400cefe0 2024-06-29T15:55:28.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.239+0000 7fd3367fc640 1 -- 172.21.15.81:0/1236820326 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v187) ==== 181+0+0 (secure 0 0 0) 0x7fd3300656d0 con 0x7fd3400cefe0 2024-06-29T15:55:28.240 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:55:28.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.245+0000 7fd3367fc640 1 -- 172.21.15.81:0/1236820326 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fd330010cf0 con 0x7fd3400cefe0 2024-06-29T15:55:28.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.247+0000 7fd347e95640 1 -- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fd324076360 msgr2=0x7fd324078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:28.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.247+0000 7fd347e95640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fd324076360 0x7fd324078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd33c035000 tx=0x7fd33c004ab0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.248+0000 7fd347e95640 1 -- 172.21.15.81:0/1236820326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3400cefe0 msgr2=0x7fd3401f5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.248+0000 7fd347e95640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3400cefe0 0x7fd3401f5a50 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fd33000e7c0 tx=0x7fd33000ec80 comp rx=0 tx=0).stop 2024-06-29T15:55:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.248+0000 7fd347e95640 1 -- 172.21.15.81:0/1236820326 shutdown_connections 2024-06-29T15:55:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.248+0000 7fd347e95640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fd324076360 0x7fd324078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.248+0000 7fd347e95640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd34015abb0 0x7fd340144cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.248+0000 7fd347e95640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3400cefe0 0x7fd3401f5a50 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.248+0000 7fd347e95640 1 --2- 172.21.15.81:0/1236820326 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd3400b3860 0x7fd3401f5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.248+0000 7fd347e95640 1 -- 172.21.15.81:0/1236820326 >> 172.21.15.81:0/1236820326 conn(0x7fd34006cd60 msgr2=0x7fd340159b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.248+0000 7fd347e95640 1 -- 172.21.15.81:0/1236820326 shutdown_connections 2024-06-29T15:55:28.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.248+0000 7fd347e95640 1 -- 172.21.15.81:0/1236820326 wait complete. 2024-06-29T15:55:28.262 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:55:28.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.381+0000 7efce64c3640 1 -- 172.21.15.81:0/746739193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc8004e60 msgr2=0x7efcc80052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:28.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.381+0000 7efce64c3640 1 --2- 172.21.15.81:0/746739193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc8004e60 0x7efcc80052b0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c610 tx=0x7efcc8005830 comp rx=0 tx=0).stop 2024-06-29T15:55:28.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.381+0000 7efce4a60640 1 -- 172.21.15.81:0/746739193 reap_dead start 2024-06-29T15:55:28.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.381+0000 7efce64c3640 1 -- 172.21.15.81:0/746739193 shutdown_connections 2024-06-29T15:55:28.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.381+0000 7efce64c3640 1 -- 172.21.15.81:0/746739193 >> 172.21.15.81:0/746739193 conn(0x7efce009aa60 msgr2=0x7efce009ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:28.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.381+0000 7efce64c3640 1 -- 172.21.15.81:0/746739193 shutdown_connections 2024-06-29T15:55:28.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.381+0000 7efce64c3640 1 -- 172.21.15.81:0/746739193 wait complete. 2024-06-29T15:55:28.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.382+0000 7efce64c3640 1 Processor -- start 2024-06-29T15:55:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.383+0000 7efce64c3640 1 -- start start 2024-06-29T15:55:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efce64c3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc8004e60 0x7efce01f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efce64c3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efce0151d50 0x7efce013fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efce64c3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efce00b5c50 0x7efce0141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efce64c3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efce00d1830 con 0x7efcc8004e60 2024-06-29T15:55:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efce64c3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efce00d16b0 con 0x7efce0151d50 2024-06-29T15:55:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efce64c3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efce00d19b0 con 0x7efce00b5c50 2024-06-29T15:55:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efcdffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc8004e60 0x7efce01f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efcdffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc8004e60 0x7efce01f2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53680/0 (socket says 172.21.15.81:53680) 2024-06-29T15:55:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efcdffff640 1 -- 172.21.15.81:0/2648248591 learned_addr learned my addr 172.21.15.81:0/2648248591 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efce4a60640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efce00b5c50 0x7efce0141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efcdf7fe640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efce0151d50 0x7efce013fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efcdffff640 1 -- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efce00b5c50 msgr2=0x7efce0141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efcdffff640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efce00b5c50 0x7efce0141f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efcdffff640 1 -- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efce0151d50 msgr2=0x7efce013fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efcdffff640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efce0151d50 0x7efce013fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efcdffff640 1 -- 172.21.15.81:0/2648248591 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce0154b40 con 0x7efcc8004e60 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efcdf7fe640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efce0151d50 0x7efce013fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.384+0000 7efce4a60640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efce00b5c50 0x7efce0141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.385+0000 7efcdffff640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc8004e60 0x7efce01f2370 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7efcd0004650 tx=0x7efcd000ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.385+0000 7efcdcff9640 1 -- 172.21.15.81:0/2648248591 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0010850 con 0x7efcc8004e60 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.385+0000 7efcdcff9640 1 -- 172.21.15.81:0/2648248591 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcd00109f0 con 0x7efcc8004e60 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.385+0000 7efce64c3640 1 -- 172.21.15.81:0/2648248591 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce0154e20 con 0x7efcc8004e60 2024-06-29T15:55:28.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.385+0000 7efcdcff9640 1 -- 172.21.15.81:0/2648248591 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd0010e20 con 0x7efcc8004e60 2024-06-29T15:55:28.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.385+0000 7efce64c3640 1 -- 172.21.15.81:0/2648248591 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce00b5730 con 0x7efcc8004e60 2024-06-29T15:55:28.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.386+0000 7efce64c3640 1 -- 172.21.15.81:0/2648248591 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce015fb80 con 0x7efcc8004e60 2024-06-29T15:55:28.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.387+0000 7efcdcff9640 1 -- 172.21.15.81:0/2648248591 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7efcd0023440 con 0x7efcc8004e60 2024-06-29T15:55:28.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.387+0000 7efcdcff9640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7efccc0763f0 0x7efccc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:28.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.387+0000 7efcdcff9640 1 -- 172.21.15.81:0/2648248591 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(187..187 src has 1..187) ==== 6517+0+0 (secure 0 0 0) 0x7efcd0098910 con 0x7efcc8004e60 2024-06-29T15:55:28.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.387+0000 7efcdf7fe640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7efccc0763f0 0x7efccc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:28.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.388+0000 7efcdf7fe640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7efccc0763f0 0x7efccc078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efce0160240 tx=0x7efcd4006b70 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:28.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.391+0000 7efcdcff9640 1 -- 172.21.15.81:0/2648248591 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efcd0061e40 con 0x7efcc8004e60 2024-06-29T15:55:28.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:28.570+0000 7efce64c3640 1 -- 172.21.15.81:0/2648248591 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7efce0071120 con 0x7efcc8004e60 2024-06-29T15:55:29.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.119+0000 7efcdcff9640 1 -- 172.21.15.81:0/2648248591 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v188) ==== 169+0+0 (secure 0 0 0) 0x7efcd0061740 con 0x7efcc8004e60 2024-06-29T15:55:29.120 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T15:55:29.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 -- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7efccc0763f0 msgr2=0x7efccc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7efccc0763f0 0x7efccc078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efce0160240 tx=0x7efcd4006b70 comp rx=0 tx=0).stop 2024-06-29T15:55:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 -- 172.21.15.81:0/2648248591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc8004e60 msgr2=0x7efce01f2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc8004e60 0x7efce01f2370 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7efcd0004650 tx=0x7efcd000ba40 comp rx=0 tx=0).stop 2024-06-29T15:55:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 -- 172.21.15.81:0/2648248591 shutdown_connections 2024-06-29T15:55:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7efccc0763f0 0x7efccc078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efce00b5c50 0x7efce0141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efce0151d50 0x7efce013fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 --2- 172.21.15.81:0/2648248591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc8004e60 0x7efce01f2370 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 -- 172.21.15.81:0/2648248591 >> 172.21.15.81:0/2648248591 conn(0x7efce009aa60 msgr2=0x7efce0071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 -- 172.21.15.81:0/2648248591 shutdown_connections 2024-06-29T15:55:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.123+0000 7efce64c3640 1 -- 172.21.15.81:0/2648248591 wait complete. 2024-06-29T15:55:29.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:55:29.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.256+0000 7f515f703640 1 -- 172.21.15.81:0/552284796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5140003a50 msgr2=0x7f5140003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.256+0000 7f515f703640 1 --2- 172.21.15.81:0/552284796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5140003a50 0x7f5140003ec0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f5148009b50 tx=0x7f5140005830 comp rx=0 tx=0).stop 2024-06-29T15:55:29.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.256+0000 7f515f703640 1 -- 172.21.15.81:0/552284796 shutdown_connections 2024-06-29T15:55:29.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.256+0000 7f515f703640 1 --2- 172.21.15.81:0/552284796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5140003a50 0x7f5140003ec0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.256+0000 7f515f703640 1 -- 172.21.15.81:0/552284796 >> 172.21.15.81:0/552284796 conn(0x7f515806ccc0 msgr2=0x7f51580b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:29.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.256+0000 7f515f703640 1 -- 172.21.15.81:0/552284796 shutdown_connections 2024-06-29T15:55:29.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.257+0000 7f515f703640 1 -- 172.21.15.81:0/552284796 wait complete. 2024-06-29T15:55:29.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.258+0000 7f515f703640 1 Processor -- start 2024-06-29T15:55:29.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.258+0000 7f515f703640 1 -- start start 2024-06-29T15:55:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515f703640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5140003a50 0x7f515814d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515f703640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f515814a720 0x7f5158159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515f703640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f515814dc70 0x7f5158159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515f703640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f515806fbf0 con 0x7f515814a720 2024-06-29T15:55:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515f703640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f515806fa70 con 0x7f515814dc70 2024-06-29T15:55:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515f703640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f515806fd70 con 0x7f5140003a50 2024-06-29T15:55:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515cc9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f515814a720 0x7f5158159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515cc9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f515814a720 0x7f5158159500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53704/0 (socket says 172.21.15.81:53704) 2024-06-29T15:55:29.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515cc9e640 1 -- 172.21.15.81:0/1008013532 learned_addr learned my addr 172.21.15.81:0/1008013532 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515d49f640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5140003a50 0x7f515814d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515dca0640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f515814dc70 0x7f5158159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515cc9e640 1 -- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5140003a50 msgr2=0x7f515814d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515cc9e640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5140003a50 0x7f515814d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515cc9e640 1 -- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f515814dc70 msgr2=0x7f5158159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515cc9e640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f515814dc70 0x7f5158159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515cc9e640 1 -- 172.21.15.81:0/1008013532 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51581f5090 con 0x7f515814a720 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515dca0640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f515814dc70 0x7f5158159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515d49f640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5140003a50 0x7f515814d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515cc9e640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f515814a720 0x7f5158159500 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f514c00ea00 tx=0x7f514c00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f5155ffb640 1 -- 172.21.15.81:0/1008013532 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c00c940 con 0x7f515814a720 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f5155ffb640 1 -- 172.21.15.81:0/1008013532 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f514c00cbd0 con 0x7f515814a720 2024-06-29T15:55:29.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f5155ffb640 1 -- 172.21.15.81:0/1008013532 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c0035f0 con 0x7f515814a720 2024-06-29T15:55:29.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.260+0000 7f515f703640 1 -- 172.21.15.81:0/1008013532 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51581f5370 con 0x7f515814a720 2024-06-29T15:55:29.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.261+0000 7f515f703640 1 -- 172.21.15.81:0/1008013532 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51581f5930 con 0x7f515814a720 2024-06-29T15:55:29.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.262+0000 7f515f703640 1 -- 172.21.15.81:0/1008013532 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51580cf2a0 con 0x7f515814a720 2024-06-29T15:55:29.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.262+0000 7f5155ffb640 1 -- 172.21.15.81:0/1008013532 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f514c003790 con 0x7f515814a720 2024-06-29T15:55:29.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.262+0000 7f5155ffb640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f51440763f0 0x7f5144078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.263+0000 7f5155ffb640 1 -- 172.21.15.81:0/1008013532 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6103+0+0 (secure 0 0 0) 0x7f514c098760 con 0x7f515814a720 2024-06-29T15:55:29.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.263+0000 7f515d49f640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f51440763f0 0x7f5144078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.263+0000 7f515d49f640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f51440763f0 0x7f5144078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5148035000 tx=0x7f51480046c0 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:29.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.267+0000 7f5155ffb640 1 -- 172.21.15.81:0/1008013532 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f514c01e020 con 0x7f515814a720 2024-06-29T15:55:29.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.454+0000 7f515f703640 1 -- 172.21.15.81:0/1008013532 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f515806eb50 con 0x7f515814a720 2024-06-29T15:55:29.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.458+0000 7f5155ffb640 1 -- 172.21.15.81:0/1008013532 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v67) ==== 71+0+0 (secure 0 0 0) 0x7f514c061e20 con 0x7f515814a720 2024-06-29T15:55:29.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 -- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f51440763f0 msgr2=0x7f5144078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f51440763f0 0x7f5144078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5148035000 tx=0x7f51480046c0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 -- 172.21.15.81:0/1008013532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f515814a720 msgr2=0x7f5158159500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f515814a720 0x7f5158159500 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f514c00ea00 tx=0x7f514c00eec0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 -- 172.21.15.81:0/1008013532 shutdown_connections 2024-06-29T15:55:29.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f51440763f0 0x7f5144078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f515814dc70 0x7f5158159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f515814a720 0x7f5158159500 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 --2- 172.21.15.81:0/1008013532 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5140003a50 0x7f515814d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 -- 172.21.15.81:0/1008013532 >> 172.21.15.81:0/1008013532 conn(0x7f515806ccc0 msgr2=0x7f515806ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:29.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 -- 172.21.15.81:0/1008013532 shutdown_connections 2024-06-29T15:55:29.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.460+0000 7f515f703640 1 -- 172.21.15.81:0/1008013532 wait complete. 2024-06-29T15:55:29.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2024-06-29T15:55:29.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.585+0000 7fe67aee3640 1 -- 172.21.15.81:0/374859198 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67406f310 msgr2=0x7fe67415c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.585+0000 7fe67aee3640 1 --2- 172.21.15.81:0/374859198 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67406f310 0x7fe67415c810 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fe674071410 tx=0x7fe66c02a900 comp rx=0 tx=0).stop 2024-06-29T15:55:29.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.585+0000 7fe67aee3640 1 -- 172.21.15.81:0/374859198 shutdown_connections 2024-06-29T15:55:29.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.585+0000 7fe67aee3640 1 --2- 172.21.15.81:0/374859198 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67406f310 0x7fe67415c810 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.585+0000 7fe67aee3640 1 --2- 172.21.15.81:0/374859198 >> v2:172.21.15.167:3300/0 conn(0x7fe674144a20 0x7fe67406edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.585+0000 7fe67aee3640 1 -- 172.21.15.81:0/374859198 >> 172.21.15.81:0/374859198 conn(0x7fe67406ccc0 msgr2=0x7fe6740b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:29.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.585+0000 7fe67aee3640 1 -- 172.21.15.81:0/374859198 shutdown_connections 2024-06-29T15:55:29.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.585+0000 7fe67aee3640 1 -- 172.21.15.81:0/374859198 wait complete. 2024-06-29T15:55:29.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.587+0000 7fe67aee3640 1 Processor -- start 2024-06-29T15:55:29.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.587+0000 7fe67aee3640 1 -- start start 2024-06-29T15:55:29.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe67aee3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe67406f310 0x7fe67415b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe67aee3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe674071f20 0x7fe6740a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe67aee3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe674144a20 0x7fe6740a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe67aee3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe67415d190 con 0x7fe674071f20 2024-06-29T15:55:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe67aee3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe67415d010 con 0x7fe67406f310 2024-06-29T15:55:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe67aee3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe67415d310 con 0x7fe674144a20 2024-06-29T15:55:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe673fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe674071f20 0x7fe6740a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe673fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe674071f20 0x7fe6740a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53720/0 (socket says 172.21.15.81:53720) 2024-06-29T15:55:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe673fff640 1 -- 172.21.15.81:0/3230002773 learned_addr learned my addr 172.21.15.81:0/3230002773 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe678c7f640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe67406f310 0x7fe67415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe679480640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe674144a20 0x7fe6740a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe673fff640 1 -- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe674144a20 msgr2=0x7fe6740a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.588+0000 7fe673fff640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe674144a20 0x7fe6740a9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.589+0000 7fe673fff640 1 -- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe67406f310 msgr2=0x7fe67415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.589+0000 7fe673fff640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe67406f310 0x7fe67415b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.589+0000 7fe673fff640 1 -- 172.21.15.81:0/3230002773 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe674132020 con 0x7fe674071f20 2024-06-29T15:55:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.589+0000 7fe679480640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe674144a20 0x7fe6740a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:29.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.589+0000 7fe673fff640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe674071f20 0x7fe6740a94c0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00c970 tx=0x7fe65c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:29.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.589+0000 7fe6717fa640 1 -- 172.21.15.81:0/3230002773 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65c007bb0 con 0x7fe674071f20 2024-06-29T15:55:29.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.589+0000 7fe6717fa640 1 -- 172.21.15.81:0/3230002773 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe65c007e40 con 0x7fe674071f20 2024-06-29T15:55:29.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.589+0000 7fe67aee3640 1 -- 172.21.15.81:0/3230002773 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe674132300 con 0x7fe674071f20 2024-06-29T15:55:29.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.589+0000 7fe6717fa640 1 -- 172.21.15.81:0/3230002773 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65c00f690 con 0x7fe674071f20 2024-06-29T15:55:29.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.589+0000 7fe67aee3640 1 -- 172.21.15.81:0/3230002773 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6741328c0 con 0x7fe674071f20 2024-06-29T15:55:29.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.590+0000 7fe67aee3640 1 -- 172.21.15.81:0/3230002773 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe67415d8a0 con 0x7fe674071f20 2024-06-29T15:55:29.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.591+0000 7fe6717fa640 1 -- 172.21.15.81:0/3230002773 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fe65c018400 con 0x7fe674071f20 2024-06-29T15:55:29.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.591+0000 7fe6717fa640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fe6540763f0 0x7fe654078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:29.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.591+0000 7fe6717fa640 1 -- 172.21.15.81:0/3230002773 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6103+0+0 (secure 0 0 0) 0x7fe65c0982c0 con 0x7fe674071f20 2024-06-29T15:55:29.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.592+0000 7fe678c7f640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fe6540763f0 0x7fe654078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:29.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.592+0000 7fe678c7f640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fe6540763f0 0x7fe654078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe664007fd0 tx=0x7fe664009040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:29.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.596+0000 7fe6717fa640 1 -- 172.21.15.81:0/3230002773 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe65c014030 con 0x7fe674071f20 2024-06-29T15:55:29.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:29.848+0000 7fe67aee3640 1 -- 172.21.15.81:0/3230002773 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) -- 0x7fe674131db0 con 0x7fe674071f20 2024-06-29T15:55:30.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.241+0000 7fe6717fa640 1 -- 172.21.15.81:0/3230002773 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2996) ==== 165+0+0 (secure 0 0 0) 0x7fe65c061870 con 0x7fe674071f20 2024-06-29T15:55:30.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 -- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fe6540763f0 msgr2=0x7fe654078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fe6540763f0 0x7fe654078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe664007fd0 tx=0x7fe664009040 comp rx=0 tx=0).stop 2024-06-29T15:55:30.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 -- 172.21.15.81:0/3230002773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe674071f20 msgr2=0x7fe6740a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe674071f20 0x7fe6740a94c0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00c970 tx=0x7fe65c00ce30 comp rx=0 tx=0).stop 2024-06-29T15:55:30.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 -- 172.21.15.81:0/3230002773 shutdown_connections 2024-06-29T15:55:30.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fe6540763f0 0x7fe654078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe674144a20 0x7fe6740a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe674071f20 0x7fe6740a94c0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 --2- 172.21.15.81:0/3230002773 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe67406f310 0x7fe67415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 -- 172.21.15.81:0/3230002773 >> 172.21.15.81:0/3230002773 conn(0x7fe67406ccc0 msgr2=0x7fe67415fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 -- 172.21.15.81:0/3230002773 shutdown_connections 2024-06-29T15:55:30.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.244+0000 7fe67aee3640 1 -- 172.21.15.81:0/3230002773 wait complete. 2024-06-29T15:55:30.256 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2024-06-29T15:55:30.257 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2024-06-29T15:55:30.257 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2024-06-29T15:55:30.257 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:30.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.371+0000 7f06558e3640 1 -- 172.21.15.81:0/383143643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06500cccf0 msgr2=0x7f06500d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.371+0000 7f06558e3640 1 --2- 172.21.15.81:0/383143643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06500cccf0 0x7f06500d0230 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f0650073550 tx=0x7f064402a560 comp rx=0 tx=0).stop 2024-06-29T15:55:30.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.371+0000 7f06558e3640 1 -- 172.21.15.81:0/383143643 shutdown_connections 2024-06-29T15:55:30.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.371+0000 7f06558e3640 1 --2- 172.21.15.81:0/383143643 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500d0770 0x7f06500ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.371+0000 7f06558e3640 1 --2- 172.21.15.81:0/383143643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06500cccf0 0x7f06500d0230 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.371+0000 7f06558e3640 1 --2- 172.21.15.81:0/383143643 >> v2:172.21.15.167:3300/0 conn(0x7f06500b3200 0x7f06500cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.371+0000 7f06558e3640 1 -- 172.21.15.81:0/383143643 >> 172.21.15.81:0/383143643 conn(0x7f06500b3680 msgr2=0x7f06500b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.371+0000 7f06558e3640 1 -- 172.21.15.81:0/383143643 shutdown_connections 2024-06-29T15:55:30.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.371+0000 7f06558e3640 1 -- 172.21.15.81:0/383143643 wait complete. 2024-06-29T15:55:30.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.373+0000 7f06558e3640 1 Processor -- start 2024-06-29T15:55:30.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.373+0000 7f06558e3640 1 -- start start 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.374+0000 7f06558e3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b3200 0x7f065009ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.374+0000 7f06558e3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06500cccf0 0x7f0650150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.374+0000 7f06558e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06500d0770 0x7f06501e6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.374+0000 7f06558e3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06500cefe0 con 0x7f06500d0770 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.374+0000 7f06558e3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f06500cee60 con 0x7f06500cccf0 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.374+0000 7f06558e3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f06500cf160 con 0x7f06500b3200 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06500d0770 0x7f06501e6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06500d0770 0x7f06501e6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33052/0 (socket says 172.21.15.81:33052) 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064f7fe640 1 -- 172.21.15.81:0/81533892 learned_addr learned my addr 172.21.15.81:0/81533892 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064e7fc640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06500cccf0 0x7f0650150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064effd640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b3200 0x7f065009ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064f7fe640 1 -- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b3200 msgr2=0x7f065009ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064f7fe640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b3200 0x7f065009ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064f7fe640 1 -- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06500cccf0 msgr2=0x7f0650150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064f7fe640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06500cccf0 0x7f0650150e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064f7fe640 1 -- 172.21.15.81:0/81533892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06500af0b0 con 0x7f06500d0770 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f064f7fe640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06500d0770 0x7f06501e6940 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f064800b440 tx=0x7f064800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f0637fff640 1 -- 172.21.15.81:0/81533892 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648015020 con 0x7f06500d0770 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f0637fff640 1 -- 172.21.15.81:0/81533892 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0648011940 con 0x7f06500d0770 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f06558e3640 1 -- 172.21.15.81:0/81533892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06500af390 con 0x7f06500d0770 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f0637fff640 1 -- 172.21.15.81:0/81533892 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648011d90 con 0x7f06500d0770 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.375+0000 7f06558e3640 1 -- 172.21.15.81:0/81533892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06500af950 con 0x7f06500d0770 2024-06-29T15:55:30.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.376+0000 7f06558e3640 1 -- 172.21.15.81:0/81533892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06500d0230 con 0x7f06500d0770 2024-06-29T15:55:30.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.377+0000 7f0637fff640 1 -- 172.21.15.81:0/81533892 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f0648018020 con 0x7f06500d0770 2024-06-29T15:55:30.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.378+0000 7f0637fff640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f063c0763f0 0x7f063c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.378+0000 7f0637fff640 1 -- 172.21.15.81:0/81533892 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6103+0+0 (secure 0 0 0) 0x7f0648097a80 con 0x7f06500d0770 2024-06-29T15:55:30.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.378+0000 7f064effd640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f063c0763f0 0x7f063c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.378+0000 7f064effd640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f063c0763f0 0x7f063c078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0640005c40 tx=0x7f064000a370 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:30.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.382+0000 7f0637fff640 1 -- 172.21.15.81:0/81533892 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f064801f020 con 0x7f06500d0770 2024-06-29T15:55:30.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.565+0000 7f06558e3640 1 -- 172.21.15.81:0/81533892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06501e2130 con 0x7f06500d0770 2024-06-29T15:55:30.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.566+0000 7f0637fff640 1 -- 172.21.15.81:0/81533892 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) ==== 74+0+14748 (secure 0 0 0) 0x7f0648061140 con 0x7f06500d0770 2024-06-29T15:55:30.567 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:30.567 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":188,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:55:29.118523+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6837/446857148":"2024-06-30T15:55:26.126358+0000","172.21.15.167:6835/446857148":"2024-06-30T15:55:26.126358+0000","172.21.15.167:0/3903269557":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/3685877654":"2024-06-30T15:55:23.904081+0000","172.21.15.167:6833/1431883924":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/1938597900":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/321095347":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/1637259453":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3006573425":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/578369376":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6833/2288011293":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6832/2288011293":"2024-06-30T15:50:20.905071+0000","172.21.15.167:6833/2546364927":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/3941545005":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3712811511":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1119662655":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/955369459":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1088263349":"2024-06-30T15:55:23.904081+0000","172.21.15.167:0/153744272":"2024-06-30T15:39:23.843781+0000","172.21.15.167:6832/1431883924":"2024-06-30T15:55:23.904081+0000","172.21.15.167:6832/2546364927":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/2062891067":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/3310646322":"2024-06-30T15:39:23.843781+0000","172.21.15.167:0/1870993879":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/3737424986":"2024-06-30T15:50:20.905071+0000","172.21.15.167:0/1345841177":"2024-06-30T15:39:23.843781+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":23,"snaps":[{"begin":2,"length":32}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 -- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f063c0763f0 msgr2=0x7f063c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f063c0763f0 0x7f063c078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0640005c40 tx=0x7f064000a370 comp rx=0 tx=0).stop 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 -- 172.21.15.81:0/81533892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06500d0770 msgr2=0x7f06501e6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06500d0770 0x7f06501e6940 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f064800b440 tx=0x7f064800b840 comp rx=0 tx=0).stop 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 -- 172.21.15.81:0/81533892 shutdown_connections 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f063c0763f0 0x7f063c078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06500d0770 0x7f06501e6940 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06500cccf0 0x7f0650150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 --2- 172.21.15.81:0/81533892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06500b3200 0x7f065009ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 -- 172.21.15.81:0/81533892 >> 172.21.15.81:0/81533892 conn(0x7f06500b3680 msgr2=0x7f06501e4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 -- 172.21.15.81:0/81533892 shutdown_connections 2024-06-29T15:55:30.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.568+0000 7f06558e3640 1 -- 172.21.15.81:0/81533892 wait complete. 2024-06-29T15:55:30.580 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:55:30.580 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:55:30.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.695+0000 7f713f638640 1 -- 172.21.15.81:0/3039055479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71381412f0 msgr2=0x7f71380cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.695+0000 7f713f638640 1 --2- 172.21.15.81:0/3039055479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71381412f0 0x7f71380cef90 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f7128005fd0 tx=0x7f712802a8b0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.695+0000 7f713f638640 1 -- 172.21.15.81:0/3039055479 shutdown_connections 2024-06-29T15:55:30.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.695+0000 7f713f638640 1 --2- 172.21.15.81:0/3039055479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7138073740 0x7f713815e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.695+0000 7f713f638640 1 --2- 172.21.15.81:0/3039055479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71381412f0 0x7f71380cef90 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.695+0000 7f713f638640 1 -- 172.21.15.81:0/3039055479 >> 172.21.15.81:0/3039055479 conn(0x7f713806ccc0 msgr2=0x7f71380b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.695+0000 7f713f638640 1 -- 172.21.15.81:0/3039055479 shutdown_connections 2024-06-29T15:55:30.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.695+0000 7f713f638640 1 -- 172.21.15.81:0/3039055479 wait complete. 2024-06-29T15:55:30.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.696+0000 7f713f638640 1 Processor -- start 2024-06-29T15:55:30.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.696+0000 7f713f638640 1 -- start start 2024-06-29T15:55:30.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713f638640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71380cf4d0 0x7f7138155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713f638640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71381412f0 0x7f71381f5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713f638640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7138156380 0x7f71381f1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713f638640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f71380b3210 con 0x7f71380cf4d0 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713f638640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f713815f270 con 0x7f71381412f0 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713f638640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f71380b3390 con 0x7f7138156380 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713d3d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71380cf4d0 0x7f7138155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713d3d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71380cf4d0 0x7f7138155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33070/0 (socket says 172.21.15.81:33070) 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713d3d4640 1 -- 172.21.15.81:0/95125377 learned_addr learned my addr 172.21.15.81:0/95125377 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713dbd5640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7138156380 0x7f71381f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713cbd3640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71381412f0 0x7f71381f5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713d3d4640 1 -- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7138156380 msgr2=0x7f71381f1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713d3d4640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7138156380 0x7f71381f1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713d3d4640 1 -- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71381412f0 msgr2=0x7f71381f5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713d3d4640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71381412f0 0x7f71381f5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713d3d4640 1 -- 172.21.15.81:0/95125377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7138151260 con 0x7f71380cf4d0 2024-06-29T15:55:30.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f713d3d4640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71380cf4d0 0x7f7138155e40 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f7128000f80 tx=0x7f7128031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:30.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.698+0000 7f7135ffb640 1 -- 172.21.15.81:0/95125377 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7128004240 con 0x7f71380cf4d0 2024-06-29T15:55:30.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.699+0000 7f7135ffb640 1 -- 172.21.15.81:0/95125377 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71280044d0 con 0x7f71380cf4d0 2024-06-29T15:55:30.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.699+0000 7f713f638640 1 -- 172.21.15.81:0/95125377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71381f6190 con 0x7f71380cf4d0 2024-06-29T15:55:30.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.699+0000 7f7135ffb640 1 -- 172.21.15.81:0/95125377 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712803d430 con 0x7f71380cf4d0 2024-06-29T15:55:30.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.699+0000 7f713f638640 1 -- 172.21.15.81:0/95125377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7138151800 con 0x7f71380cf4d0 2024-06-29T15:55:30.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.700+0000 7f713f638640 1 -- 172.21.15.81:0/95125377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71380b38b0 con 0x7f71380cf4d0 2024-06-29T15:55:30.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.701+0000 7f7135ffb640 1 -- 172.21.15.81:0/95125377 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f712803d5d0 con 0x7f71380cf4d0 2024-06-29T15:55:30.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.701+0000 7f7135ffb640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f71240763f0 0x7f7124078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:30.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.701+0000 7f7135ffb640 1 -- 172.21.15.81:0/95125377 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6103+0+0 (secure 0 0 0) 0x7f71280bb840 con 0x7f71380cf4d0 2024-06-29T15:55:30.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.701+0000 7f713cbd3640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f71240763f0 0x7f7124078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:30.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.702+0000 7f713cbd3640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f71240763f0 0x7f7124078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7138157b90 tx=0x7f712c007e90 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:30.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.706+0000 7f7135ffb640 1 -- 172.21.15.81:0/95125377 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7128041030 con 0x7f71380cf4d0 2024-06-29T15:55:30.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.883+0000 7f713f638640 1 -- 172.21.15.81:0/95125377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f713815b4f0 con 0x7f71380cf4d0 2024-06-29T15:55:30.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.883+0000 7f7135ffb640 1 -- 172.21.15.81:0/95125377 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) ==== 93+0+10 (secure 0 0 0) 0x7f7128084710 con 0x7f71380cf4d0 2024-06-29T15:55:30.884 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:55:30.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 -- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f71240763f0 msgr2=0x7f7124078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f71240763f0 0x7f7124078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7138157b90 tx=0x7f712c007e90 comp rx=0 tx=0).stop 2024-06-29T15:55:30.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 -- 172.21.15.81:0/95125377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71380cf4d0 msgr2=0x7f7138155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:30.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71380cf4d0 0x7f7138155e40 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f7128000f80 tx=0x7f7128031770 comp rx=0 tx=0).stop 2024-06-29T15:55:30.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 -- 172.21.15.81:0/95125377 shutdown_connections 2024-06-29T15:55:30.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f71240763f0 0x7f7124078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7138156380 0x7f71381f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71381412f0 0x7f71381f5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 --2- 172.21.15.81:0/95125377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71380cf4d0 0x7f7138155e40 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:30.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 -- 172.21.15.81:0/95125377 >> 172.21.15.81:0/95125377 conn(0x7f713806ccc0 msgr2=0x7f71380b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:30.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 -- 172.21.15.81:0/95125377 shutdown_connections 2024-06-29T15:55:30.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:30.885+0000 7f713f638640 1 -- 172.21.15.81:0/95125377 wait complete. 2024-06-29T15:55:30.897 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-06-29T15:55:31.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.008+0000 7fde94c5e640 1 -- 172.21.15.81:0/3036244552 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde901412f0 msgr2=0x7fde900cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:31.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.008+0000 7fde94c5e640 1 --2- 172.21.15.81:0/3036244552 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde901412f0 0x7fde900cef90 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fde90071d20 tx=0x7fde8002a560 comp rx=0 tx=0).stop 2024-06-29T15:55:31.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.009+0000 7fde94c5e640 1 -- 172.21.15.81:0/3036244552 shutdown_connections 2024-06-29T15:55:31.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.009+0000 7fde94c5e640 1 --2- 172.21.15.81:0/3036244552 >> v2:172.21.15.167:3300/0 conn(0x7fde90073740 0x7fde9015e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:31.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.009+0000 7fde94c5e640 1 --2- 172.21.15.81:0/3036244552 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde901412f0 0x7fde900cef90 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:31.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.009+0000 7fde94c5e640 1 -- 172.21.15.81:0/3036244552 >> 172.21.15.81:0/3036244552 conn(0x7fde9006ccc0 msgr2=0x7fde900b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:31.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.009+0000 7fde94c5e640 1 -- 172.21.15.81:0/3036244552 shutdown_connections 2024-06-29T15:55:31.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.009+0000 7fde94c5e640 1 -- 172.21.15.81:0/3036244552 wait complete. 2024-06-29T15:55:31.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.011+0000 7fde94c5e640 1 Processor -- start 2024-06-29T15:55:31.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.011+0000 7fde94c5e640 1 -- start start 2024-06-29T15:55:31.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde94c5e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde900cf4d0 0x7fde90154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde94c5e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde901412f0 0x7fde9013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde94c5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde90154780 0x7fde9006e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde94c5e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fde900b3210 con 0x7fde90154780 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde94c5e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fde9015f220 con 0x7fde901412f0 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde94c5e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fde900b3390 con 0x7fde900cf4d0 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde8ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde90154780 0x7fde9006e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde8ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde90154780 0x7fde9006e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33102/0 (socket says 172.21.15.81:33102) 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde8ed9d640 1 -- 172.21.15.81:0/1000005125 learned_addr learned my addr 172.21.15.81:0/1000005125 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde8e59c640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde900cf4d0 0x7fde90154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde8dd9b640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde901412f0 0x7fde9013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde8ed9d640 1 -- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde900cf4d0 msgr2=0x7fde90154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde8ed9d640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde900cf4d0 0x7fde90154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:31.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde8ed9d640 1 -- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde901412f0 msgr2=0x7fde9013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:31.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde8ed9d640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde901412f0 0x7fde9013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:31.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.012+0000 7fde8ed9d640 1 -- 172.21.15.81:0/1000005125 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde90155a70 con 0x7fde90154780 2024-06-29T15:55:31.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.013+0000 7fde8ed9d640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde90154780 0x7fde9006e2a0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fde8800e9c0 tx=0x7fde8800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:31.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.013+0000 7fde7effd640 1 -- 172.21.15.81:0/1000005125 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde88004d20 con 0x7fde90154780 2024-06-29T15:55:31.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.013+0000 7fde7effd640 1 -- 172.21.15.81:0/1000005125 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde8800cb20 con 0x7fde90154780 2024-06-29T15:55:31.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.013+0000 7fde7effd640 1 -- 172.21.15.81:0/1000005125 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde88005020 con 0x7fde90154780 2024-06-29T15:55:31.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.013+0000 7fde94c5e640 1 -- 172.21.15.81:0/1000005125 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde90154ef0 con 0x7fde90154780 2024-06-29T15:55:31.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.013+0000 7fde94c5e640 1 -- 172.21.15.81:0/1000005125 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde901554b0 con 0x7fde90154780 2024-06-29T15:55:31.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.014+0000 7fde94c5e640 1 -- 172.21.15.81:0/1000005125 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde90154c60 con 0x7fde90154780 2024-06-29T15:55:31.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.015+0000 7fde7effd640 1 -- 172.21.15.81:0/1000005125 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fde880051c0 con 0x7fde90154780 2024-06-29T15:55:31.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.015+0000 7fde7effd640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fde700763f0 0x7fde70078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:31.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.015+0000 7fde7effd640 1 -- 172.21.15.81:0/1000005125 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6103+0+0 (secure 0 0 0) 0x7fde88098340 con 0x7fde90154780 2024-06-29T15:55:31.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.015+0000 7fde8e59c640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fde700763f0 0x7fde70078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:31.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.016+0000 7fde8e59c640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fde700763f0 0x7fde70078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fde80033000 tx=0x7fde80004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:31.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.020+0000 7fde7effd640 1 -- 172.21.15.81:0/1000005125 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fde88016030 con 0x7fde90154780 2024-06-29T15:55:31.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:31.270+0000 7fde94c5e640 1 -- 172.21.15.81:0/1000005125 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) -- 0x7fde900b38b0 con 0x7fde90154780 2024-06-29T15:55:32.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.249+0000 7fde7effd640 1 -- 172.21.15.81:0/1000005125 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v2998) ==== 166+0+0 (secure 0 0 0) 0x7fde88061210 con 0x7fde90154780 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 -- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fde700763f0 msgr2=0x7fde70078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fde700763f0 0x7fde70078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fde80033000 tx=0x7fde80004730 comp rx=0 tx=0).stop 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 -- 172.21.15.81:0/1000005125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde90154780 msgr2=0x7fde9006e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde90154780 0x7fde9006e2a0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fde8800e9c0 tx=0x7fde8800ee80 comp rx=0 tx=0).stop 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 -- 172.21.15.81:0/1000005125 shutdown_connections 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fde700763f0 0x7fde70078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde90154780 0x7fde9006e2a0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde901412f0 0x7fde9013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 --2- 172.21.15.81:0/1000005125 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde900cf4d0 0x7fde90154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 -- 172.21.15.81:0/1000005125 >> 172.21.15.81:0/1000005125 conn(0x7fde9006ccc0 msgr2=0x7fde9015e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 -- 172.21.15.81:0/1000005125 shutdown_connections 2024-06-29T15:55:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.252+0000 7fde94c5e640 1 -- 172.21.15.81:0/1000005125 wait complete. 2024-06-29T15:55:32.266 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:55:32.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.382+0000 7fcd64cce640 1 -- 172.21.15.81:0/3578489456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd6015c030 msgr2=0x7fcd6006c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.382+0000 7fcd64cce640 1 --2- 172.21.15.81:0/3578489456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd6015c030 0x7fcd6006c2f0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fcd60074e30 tx=0x7fcd4802a530 comp rx=0 tx=0).stop 2024-06-29T15:55:32.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.382+0000 7fcd64cce640 1 -- 172.21.15.81:0/3578489456 shutdown_connections 2024-06-29T15:55:32.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.382+0000 7fcd64cce640 1 --2- 172.21.15.81:0/3578489456 >> v2:172.21.15.167:3300/0 conn(0x7fcd6015c440 0x7fcd6006c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.382+0000 7fcd64cce640 1 --2- 172.21.15.81:0/3578489456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd6015c030 0x7fcd6006c2f0 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.382+0000 7fcd64cce640 1 -- 172.21.15.81:0/3578489456 >> 172.21.15.81:0/3578489456 conn(0x7fcd600d0930 msgr2=0x7fcd600d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:32.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.383+0000 7fcd64cce640 1 -- 172.21.15.81:0/3578489456 shutdown_connections 2024-06-29T15:55:32.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.383+0000 7fcd64cce640 1 -- 172.21.15.81:0/3578489456 wait complete. 2024-06-29T15:55:32.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.385+0000 7fcd64cce640 1 Processor -- start 2024-06-29T15:55:32.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.385+0000 7fcd64cce640 1 -- start start 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd64cce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcd6015c030 0x7fcd600ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd64cce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcd6015c440 0x7fcd600b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd64cce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd6015d750 0x7fcd601f8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd64cce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcd600cfca0 con 0x7fcd6015d750 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd64cce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcd600cfb20 con 0x7fcd6015c030 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd64cce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcd600cfe20 con 0x7fcd6015c440 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd5ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd6015d750 0x7fcd601f8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd5ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd6015d750 0x7fcd601f8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33130/0 (socket says 172.21.15.81:33130) 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd5ed9d640 1 -- 172.21.15.81:0/4174646769 learned_addr learned my addr 172.21.15.81:0/4174646769 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd55d9b640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcd6015c440 0x7fcd600b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd5e59c640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcd6015c030 0x7fcd600ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd5ed9d640 1 -- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcd6015c440 msgr2=0x7fcd600b46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd5ed9d640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcd6015c440 0x7fcd600b46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd5ed9d640 1 -- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcd6015c030 msgr2=0x7fcd600ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd5ed9d640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcd6015c030 0x7fcd600ccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.386+0000 7fcd5ed9d640 1 -- 172.21.15.81:0/4174646769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd601f9260 con 0x7fcd6015d750 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.387+0000 7fcd5ed9d640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd6015d750 0x7fcd601f8490 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800cb00 tx=0x7fcd5800a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.387+0000 7fcd577fe640 1 -- 172.21.15.81:0/4174646769 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5800f450 con 0x7fcd6015d750 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.387+0000 7fcd577fe640 1 -- 172.21.15.81:0/4174646769 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd5800f700 con 0x7fcd6015d750 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.387+0000 7fcd64cce640 1 -- 172.21.15.81:0/4174646769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd60154820 con 0x7fcd6015d750 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.387+0000 7fcd577fe640 1 -- 172.21.15.81:0/4174646769 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5800fb50 con 0x7fcd6015d750 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.387+0000 7fcd64cce640 1 -- 172.21.15.81:0/4174646769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd60154de0 con 0x7fcd6015d750 2024-06-29T15:55:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.388+0000 7fcd64cce640 1 -- 172.21.15.81:0/4174646769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd6006e2a0 con 0x7fcd6015d750 2024-06-29T15:55:32.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.388+0000 7fcd577fe640 1 -- 172.21.15.81:0/4174646769 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fcd5800fcf0 con 0x7fcd6015d750 2024-06-29T15:55:32.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.389+0000 7fcd577fe640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fcd40076420 0x7fcd40078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.389+0000 7fcd577fe640 1 -- 172.21.15.81:0/4174646769 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6103+0+0 (secure 0 0 0) 0x7fcd580986b0 con 0x7fcd6015d750 2024-06-29T15:55:32.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.389+0000 7fcd5e59c640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fcd40076420 0x7fcd40078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.390+0000 7fcd5e59c640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fcd40076420 0x7fcd40078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcd50007fd0 tx=0x7fcd50009040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:32.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.393+0000 7fcd577fe640 1 -- 172.21.15.81:0/4174646769 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcd6006e2a0 con 0x7fcd6015d750 2024-06-29T15:55:32.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.575+0000 7fcd64cce640 1 -- 172.21.15.81:0/4174646769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fcd600d02d0 con 0x7fcd6015d750 2024-06-29T15:55:32.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.575+0000 7fcd577fe640 1 -- 172.21.15.81:0/4174646769 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v67) ==== 59+0+1998 (secure 0 0 0) 0x7fcd58014030 con 0x7fcd6015d750 2024-06-29T15:55:32.576 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:55:32.576 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:55:32.576 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:55:32.576 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:55:32.576 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T15:55:32.576 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T15:55:32.576 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:55:32.577 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T15:55:32.578 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:55:32.578 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:55:32.578 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:55:32.578 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T15:55:32.578 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T15:55:32.578 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T15:55:32.578 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:55:32.578 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:55:32.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 -- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fcd40076420 msgr2=0x7fcd40078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fcd40076420 0x7fcd40078930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcd50007fd0 tx=0x7fcd50009040 comp rx=0 tx=0).stop 2024-06-29T15:55:32.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 -- 172.21.15.81:0/4174646769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd6015d750 msgr2=0x7fcd601f8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd6015d750 0x7fcd601f8490 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800cb00 tx=0x7fcd5800a250 comp rx=0 tx=0).stop 2024-06-29T15:55:32.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 -- 172.21.15.81:0/4174646769 shutdown_connections 2024-06-29T15:55:32.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fcd40076420 0x7fcd40078930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd6015d750 0x7fcd601f8490 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcd6015c440 0x7fcd600b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 --2- 172.21.15.81:0/4174646769 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcd6015c030 0x7fcd600ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 -- 172.21.15.81:0/4174646769 >> 172.21.15.81:0/4174646769 conn(0x7fcd600d0930 msgr2=0x7fcd601523f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:32.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 -- 172.21.15.81:0/4174646769 shutdown_connections 2024-06-29T15:55:32.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.577+0000 7fcd64cce640 1 -- 172.21.15.81:0/4174646769 wait complete. 2024-06-29T15:55:32.589 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:55:32.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.704+0000 7f70ebe3d640 1 -- 172.21.15.81:0/4004919299 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d0004b60 msgr2=0x7f70d0004fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.704+0000 7f70ebe3d640 1 --2- 172.21.15.81:0/4004919299 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d0004b60 0x7f70d0004fd0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f70e00016c0 tx=0x7f70d0006940 comp rx=0 tx=0).stop 2024-06-29T15:55:32.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.704+0000 7f70ebe3d640 1 -- 172.21.15.81:0/4004919299 shutdown_connections 2024-06-29T15:55:32.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.704+0000 7f70ebe3d640 1 --2- 172.21.15.81:0/4004919299 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d0004b60 0x7f70d0004fd0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.704+0000 7f70ebe3d640 1 -- 172.21.15.81:0/4004919299 >> 172.21.15.81:0/4004919299 conn(0x7f70e406ccc0 msgr2=0x7f70e40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:32.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.704+0000 7f70ebe3d640 1 -- 172.21.15.81:0/4004919299 shutdown_connections 2024-06-29T15:55:32.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.704+0000 7f70ebe3d640 1 -- 172.21.15.81:0/4004919299 wait complete. 2024-06-29T15:55:32.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.706+0000 7f70ebe3d640 1 Processor -- start 2024-06-29T15:55:32.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.706+0000 7f70ebe3d640 1 -- start start 2024-06-29T15:55:32.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.707+0000 7f70ebe3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d0004b60 0x7f70e4154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.707+0000 7f70ebe3d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70e4149480 0x7f70e41f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.707+0000 7f70ebe3d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70e4145710 0x7f70e40af780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.707+0000 7f70ebe3d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f70e4151030 con 0x7f70d0004b60 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.707+0000 7f70ebe3d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f70e4070220 con 0x7f70e4145710 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.707+0000 7f70ebe3d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f70e41511b0 con 0x7f70e4149480 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e9bd9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d0004b60 0x7f70e4154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e9bd9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d0004b60 0x7f70e4154430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33146/0 (socket says 172.21.15.81:33146) 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e9bd9640 1 -- 172.21.15.81:0/41200873 learned_addr learned my addr 172.21.15.81:0/41200873 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e93d8640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70e4149480 0x7f70e41f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70ea3da640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70e4145710 0x7f70e40af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e9bd9640 1 -- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70e4149480 msgr2=0x7f70e41f6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e9bd9640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70e4149480 0x7f70e41f6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e9bd9640 1 -- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70e4145710 msgr2=0x7f70e40af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e9bd9640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70e4145710 0x7f70e40af780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e9bd9640 1 -- 172.21.15.81:0/41200873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e41568f0 con 0x7f70d0004b60 2024-06-29T15:55:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70ea3da640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70e4145710 0x7f70e40af780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e93d8640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70e4149480 0x7f70e41f6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70e9bd9640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d0004b60 0x7f70e4154430 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f70e002a910 tx=0x7f70e00063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70da7fc640 1 -- 172.21.15.81:0/41200873 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e0003a00 con 0x7f70d0004b60 2024-06-29T15:55:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70da7fc640 1 -- 172.21.15.81:0/41200873 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70e0039070 con 0x7f70d0004b60 2024-06-29T15:55:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70da7fc640 1 -- 172.21.15.81:0/41200873 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e0033420 con 0x7f70d0004b60 2024-06-29T15:55:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70ebe3d640 1 -- 172.21.15.81:0/41200873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e4156bd0 con 0x7f70d0004b60 2024-06-29T15:55:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.708+0000 7f70ebe3d640 1 -- 172.21.15.81:0/41200873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e4145b20 con 0x7f70d0004b60 2024-06-29T15:55:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.709+0000 7f70ebe3d640 1 -- 172.21.15.81:0/41200873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e406edd0 con 0x7f70d0004b60 2024-06-29T15:55:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.710+0000 7f70da7fc640 1 -- 172.21.15.81:0/41200873 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f70e0044020 con 0x7f70d0004b60 2024-06-29T15:55:32.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.710+0000 7f70da7fc640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f70c8076420 0x7f70c8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:32.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.711+0000 7f70da7fc640 1 -- 172.21.15.81:0/41200873 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6103+0+0 (secure 0 0 0) 0x7f70e00b73a0 con 0x7f70d0004b60 2024-06-29T15:55:32.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.711+0000 7f70e93d8640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f70c8076420 0x7f70c8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:32.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.711+0000 7f70e93d8640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f70c8076420 0x7f70c8078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f70d4005d00 tx=0x7f70d4005c70 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:32.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.715+0000 7f70da7fc640 1 -- 172.21.15.81:0/41200873 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f70e0080a60 con 0x7f70d0004b60 2024-06-29T15:55:32.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.897+0000 7f70ebe3d640 1 -- 172.21.15.81:0/41200873 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f70e415a330 con 0x7f70d0004b60 2024-06-29T15:55:32.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.901+0000 7f70da7fc640 1 -- 172.21.15.81:0/41200873 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v68)=0 v68) ==== 122+0+0 (secure 0 0 0) 0x7f70e0080360 con 0x7f70d0004b60 2024-06-29T15:55:32.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 -- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f70c8076420 msgr2=0x7f70c8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f70c8076420 0x7f70c8078930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f70d4005d00 tx=0x7f70d4005c70 comp rx=0 tx=0).stop 2024-06-29T15:55:32.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 -- 172.21.15.81:0/41200873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d0004b60 msgr2=0x7f70e4154430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d0004b60 0x7f70e4154430 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f70e002a910 tx=0x7f70e00063a0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 -- 172.21.15.81:0/41200873 shutdown_connections 2024-06-29T15:55:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f70c8076420 0x7f70c8078930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70e4145710 0x7f70e40af780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70e4149480 0x7f70e41f6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 --2- 172.21.15.81:0/41200873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d0004b60 0x7f70e4154430 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 -- 172.21.15.81:0/41200873 >> 172.21.15.81:0/41200873 conn(0x7f70e406ccc0 msgr2=0x7f70e414fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 -- 172.21.15.81:0/41200873 shutdown_connections 2024-06-29T15:55:32.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:32.903+0000 7f70ebe3d640 1 -- 172.21.15.81:0/41200873 wait complete. 2024-06-29T15:55:32.914 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:55:32.914 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:55:32.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:55:32.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:55:32.967 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:55:33.047 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:55:33.047 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:55:33.047 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:33.063 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:55:33.063 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:55:33.063 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:33.136 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:55:33.136 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:55:33.137 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:33.152 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:55:33.152 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:55:33.152 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:33.193 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:55:33.194 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:55:33.194 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:33.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.307+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1621275669 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf6415e890 msgr2=0x7fdf6415b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.307+0000 7fdf6c5fc640 1 --2- 172.21.15.81:0/1621275669 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf6415e890 0x7fdf6415b9d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009830 tx=0x7fdf6002a510 comp rx=0 tx=0).stop 2024-06-29T15:55:33.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.307+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1621275669 shutdown_connections 2024-06-29T15:55:33.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.307+0000 7fdf6c5fc640 1 --2- 172.21.15.81:0/1621275669 >> v2:172.21.15.167:3300/0 conn(0x7fdf6415bf10 0x7fdf640732a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.307+0000 7fdf6c5fc640 1 --2- 172.21.15.81:0/1621275669 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf6415e890 0x7fdf6415b9d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.307+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1621275669 >> 172.21.15.81:0/1621275669 conn(0x7fdf640b1ce0 msgr2=0x7fdf640b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:33.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.307+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1621275669 shutdown_connections 2024-06-29T15:55:33.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.307+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1621275669 wait complete. 2024-06-29T15:55:33.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.309+0000 7fdf6c5fc640 1 Processor -- start 2024-06-29T15:55:33.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.309+0000 7fdf6c5fc640 1 -- start start 2024-06-29T15:55:33.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.310+0000 7fdf6c5fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640737e0 0x7fdf6413ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.310+0000 7fdf6c5fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6415bf10 0x7fdf6413f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.310+0000 7fdf6c5fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf6415e890 0x7fdf641569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.310+0000 7fdf6c5fc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdf641412c0 con 0x7fdf6415bf10 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.310+0000 7fdf6c5fc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdf64141140 con 0x7fdf640737e0 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.310+0000 7fdf6c5fc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdf640ce550 con 0x7fdf6415e890 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf69b97640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6415bf10 0x7fdf6413f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf69b97640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6415bf10 0x7fdf6413f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33160/0 (socket says 172.21.15.81:33160) 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf69b97640 1 -- 172.21.15.81:0/1424293919 learned_addr learned my addr 172.21.15.81:0/1424293919 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf6a398640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640737e0 0x7fdf6413ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf6ab99640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf6415e890 0x7fdf641569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf69b97640 1 -- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf6415e890 msgr2=0x7fdf641569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf69b97640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf6415e890 0x7fdf641569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf69b97640 1 -- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640737e0 msgr2=0x7fdf6413ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf69b97640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640737e0 0x7fdf6413ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf69b97640 1 -- 172.21.15.81:0/1424293919 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf640ca8c0 con 0x7fdf6415bf10 2024-06-29T15:55:33.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf69b97640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6415bf10 0x7fdf6413f010 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400ea20 tx=0x7fdf5400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:33.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf5affd640 1 -- 172.21.15.81:0/1424293919 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5400ce40 con 0x7fdf6415bf10 2024-06-29T15:55:33.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf5affd640 1 -- 172.21.15.81:0/1424293919 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf54004020 con 0x7fdf6415bf10 2024-06-29T15:55:33.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1424293919 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf64138b50 con 0x7fdf6415bf10 2024-06-29T15:55:33.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf5affd640 1 -- 172.21.15.81:0/1424293919 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54004470 con 0x7fdf6415bf10 2024-06-29T15:55:33.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.311+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1424293919 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf64139b70 con 0x7fdf6415bf10 2024-06-29T15:55:33.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.313+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1424293919 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf640ca630 con 0x7fdf6415bf10 2024-06-29T15:55:33.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.313+0000 7fdf5affd640 1 -- 172.21.15.81:0/1424293919 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fdf540035f0 con 0x7fdf6415bf10 2024-06-29T15:55:33.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.314+0000 7fdf5affd640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf48076420 0x7fdf48078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.314+0000 7fdf5affd640 1 -- 172.21.15.81:0/1424293919 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6103+0+0 (secure 0 0 0) 0x7fdf54098460 con 0x7fdf6415bf10 2024-06-29T15:55:33.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.314+0000 7fdf6a398640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf48076420 0x7fdf48078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.315+0000 7fdf6a398640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf48076420 0x7fdf48078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf60034000 tx=0x7fdf60004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:33.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.318+0000 7fdf5affd640 1 -- 172.21.15.81:0/1424293919 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdf540613c0 con 0x7fdf6415bf10 2024-06-29T15:55:33.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.533+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1424293919 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf640732a0 con 0x7fdf6415bf10 2024-06-29T15:55:33.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.534+0000 7fdf5affd640 1 -- 172.21.15.81:0/1424293919 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+3919 (secure 0 0 0) 0x7fdf540656c0 con 0x7fdf6415bf10 2024-06-29T15:55:33.534 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:33.535 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":97,"btime":"2024-06-29T15:55:27:128143+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9139,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2580108037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2580108037},{"type":"v1","addr":"172.21.15.167:6836","nonce":2580108037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":9577,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/3761513092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":3761513092},{"type":"v1","addr":"172.21.15.167:6837","nonce":3761513092}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97},{"gid":10155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/3892298237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3892298237},{"type":"v1","addr":"172.21.15.81:6834","nonce":3892298237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89},{"gid":10158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2457574359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2457574359},{"type":"v1","addr":"172.21.15.81:6835","nonce":2457574359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90}],"filesystems":[]} 2024-06-29T15:55:33.535 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 97 2024-06-29T15:55:33.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf48076420 msgr2=0x7fdf48078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf48076420 0x7fdf48078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf60034000 tx=0x7fdf60004730 comp rx=0 tx=0).stop 2024-06-29T15:55:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1424293919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6415bf10 msgr2=0x7fdf6413f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6415bf10 0x7fdf6413f010 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400ea20 tx=0x7fdf5400eee0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1424293919 shutdown_connections 2024-06-29T15:55:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf48076420 0x7fdf48078930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf6415e890 0x7fdf641569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6415bf10 0x7fdf6413f010 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 --2- 172.21.15.81:0/1424293919 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf640737e0 0x7fdf6413ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1424293919 >> 172.21.15.81:0/1424293919 conn(0x7fdf640b1ce0 msgr2=0x7fdf640b38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1424293919 shutdown_connections 2024-06-29T15:55:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.536+0000 7fdf6c5fc640 1 -- 172.21.15.81:0/1424293919 wait complete. 2024-06-29T15:55:33.548 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:55:33.548 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:55:33.548 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:55:33.548 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:55:33.548 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:55:33.548 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:55:33.549 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:55:33.549 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:55:33.549 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:55:33.549 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:55:33.549 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:55:33.549 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:55:33.549 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:55:33.550 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:55:33.550 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:55:33.550 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:55:33.558 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:55:33.558 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:55:33.559 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:55:33.559 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:55:33.563 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:55:33.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:55:33.564 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:55:33.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:55:33.601 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:55:33.601 DEBUG:teuthology.parallel:result is None 2024-06-29T15:55:33.601 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:55:33.602 DEBUG:teuthology.parallel:result is None 2024-06-29T15:55:33.606 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:55:33.607 DEBUG:teuthology.parallel:result is None 2024-06-29T15:55:33.608 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:55:33.608 DEBUG:teuthology.parallel:result is None 2024-06-29T15:55:33.608 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:55:33.686 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T15:55:33.687 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T15:55:33.700 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:55:33.700 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T15:55:33.701 INFO:tasks.ceph.mds.d.smithi167.stderr:2024-06-29T15:55:33.700+0000 7fb275aa2640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-06-29T15:55:33.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.728+0000 7fb13897a640 1 -- 172.21.15.81:0/4201646591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1340cd100 msgr2=0x7fb1340b46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.728+0000 7fb13897a640 1 --2- 172.21.15.81:0/4201646591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1340cd100 0x7fb1340b46b0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fb134073b70 tx=0x7fb12402a560 comp rx=0 tx=0).stop 2024-06-29T15:55:33.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.728+0000 7fb13897a640 1 -- 172.21.15.81:0/4201646591 shutdown_connections 2024-06-29T15:55:33.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.728+0000 7fb13897a640 1 --2- 172.21.15.81:0/4201646591 >> v2:172.21.15.167:3300/0 conn(0x7fb134073690 0x7fb1340b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.728+0000 7fb13897a640 1 --2- 172.21.15.81:0/4201646591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1340cd100 0x7fb1340b46b0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.728+0000 7fb13897a640 1 -- 172.21.15.81:0/4201646591 >> 172.21.15.81:0/4201646591 conn(0x7fb1340b1910 msgr2=0x7fb1340b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:33.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.728+0000 7fb13897a640 1 -- 172.21.15.81:0/4201646591 shutdown_connections 2024-06-29T15:55:33.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.728+0000 7fb13897a640 1 -- 172.21.15.81:0/4201646591 wait complete. 2024-06-29T15:55:33.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.730+0000 7fb13897a640 1 Processor -- start 2024-06-29T15:55:33.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.730+0000 7fb13897a640 1 -- start start 2024-06-29T15:55:33.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.731+0000 7fb13897a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1340cd100 0x7fb1340ac660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.731+0000 7fb13897a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13416ac80 0x7fb134204e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.731+0000 7fb13897a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1342053c0 0x7fb1342057d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.731+0000 7fb13897a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb13406e7d0 con 0x7fb1342053c0 2024-06-29T15:55:33.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.731+0000 7fb13897a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb13406e650 con 0x7fb13416ac80 2024-06-29T15:55:33.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.731+0000 7fb13897a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb13406e950 con 0x7fb1340cd100 2024-06-29T15:55:33.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.731+0000 7fb132d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1342053c0 0x7fb1342057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.731+0000 7fb132d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1342053c0 0x7fb1342057d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33216/0 (socket says 172.21.15.81:33216) 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.731+0000 7fb132d9d640 1 -- 172.21.15.81:0/2448040748 learned_addr learned my addr 172.21.15.81:0/2448040748 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb131d9b640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13416ac80 0x7fb134204e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb13259c640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1340cd100 0x7fb1340ac660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb132d9d640 1 -- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1340cd100 msgr2=0x7fb1340ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb132d9d640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1340cd100 0x7fb1340ac660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb132d9d640 1 -- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13416ac80 msgr2=0x7fb134204e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb132d9d640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13416ac80 0x7fb134204e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb132d9d640 1 -- 172.21.15.81:0/2448040748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1340ae560 con 0x7fb1342053c0 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb131d9b640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13416ac80 0x7fb134204e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb132d9d640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1342053c0 0x7fb1342057d0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00ca20 tx=0x7fb12c00a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb122ffd640 1 -- 172.21.15.81:0/2448040748 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c018070 con 0x7fb1342053c0 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb122ffd640 1 -- 172.21.15.81:0/2448040748 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb12c004b20 con 0x7fb1342053c0 2024-06-29T15:55:33.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb122ffd640 1 -- 172.21.15.81:0/2448040748 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c014650 con 0x7fb1342053c0 2024-06-29T15:55:33.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb13897a640 1 -- 172.21.15.81:0/2448040748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1340ae840 con 0x7fb1342053c0 2024-06-29T15:55:33.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.732+0000 7fb13897a640 1 -- 172.21.15.81:0/2448040748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1340aee00 con 0x7fb1342053c0 2024-06-29T15:55:33.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.733+0000 7fb13897a640 1 -- 172.21.15.81:0/2448040748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb13406ee70 con 0x7fb1342053c0 2024-06-29T15:55:33.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.734+0000 7fb122ffd640 1 -- 172.21.15.81:0/2448040748 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fb12c0147f0 con 0x7fb1342053c0 2024-06-29T15:55:33.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.734+0000 7fb122ffd640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fb114076420 0x7fb114078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:33.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.734+0000 7fb122ffd640 1 -- 172.21.15.81:0/2448040748 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 6103+0+0 (secure 0 0 0) 0x7fb12c09aa60 con 0x7fb1342053c0 2024-06-29T15:55:33.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.735+0000 7fb13259c640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fb114076420 0x7fb114078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:33.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.735+0000 7fb13259c640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fb114076420 0x7fb114078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb124033000 tx=0x7fb124004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:33.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.739+0000 7fb122ffd640 1 -- 172.21.15.81:0/2448040748 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb12c063930 con 0x7fb1342053c0 2024-06-29T15:55:33.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:33.929+0000 7fb13897a640 1 -- 172.21.15.81:0/2448040748 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fb1340b21a0 con 0x7fb1342053c0 2024-06-29T15:55:34.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.264+0000 7fb122ffd640 1 -- 172.21.15.81:0/2448040748 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v190) ==== 97+0+0 (secure 0 0 0) 0x7fb12c067c30 con 0x7fb1342053c0 2024-06-29T15:55:34.264 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T15:55:34.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 -- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fb114076420 msgr2=0x7fb114078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fb114076420 0x7fb114078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb124033000 tx=0x7fb124004730 comp rx=0 tx=0).stop 2024-06-29T15:55:34.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 -- 172.21.15.81:0/2448040748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1342053c0 msgr2=0x7fb1342057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1342053c0 0x7fb1342057d0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00ca20 tx=0x7fb12c00a6c0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 -- 172.21.15.81:0/2448040748 shutdown_connections 2024-06-29T15:55:34.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fb114076420 0x7fb114078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1342053c0 0x7fb1342057d0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb13416ac80 0x7fb134204e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 --2- 172.21.15.81:0/2448040748 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1340cd100 0x7fb1340ac660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 -- 172.21.15.81:0/2448040748 >> 172.21.15.81:0/2448040748 conn(0x7fb1340b1910 msgr2=0x7fb134166470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:34.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 -- 172.21.15.81:0/2448040748 shutdown_connections 2024-06-29T15:55:34.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.266+0000 7fb13897a640 1 -- 172.21.15.81:0/2448040748 wait complete. 2024-06-29T15:55:34.280 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:55:34.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.394+0000 7f38963a9640 1 -- 172.21.15.81:0/646760924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f389006ef90 msgr2=0x7f389006f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.394+0000 7f38963a9640 1 --2- 172.21.15.81:0/646760924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f389006ef90 0x7f389006f3a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3884005fd0 tx=0x7f388402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.394+0000 7f38963a9640 1 -- 172.21.15.81:0/646760924 shutdown_connections 2024-06-29T15:55:34.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.394+0000 7f38963a9640 1 --2- 172.21.15.81:0/646760924 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f389006ef90 0x7f389006f3a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.394+0000 7f38963a9640 1 --2- 172.21.15.81:0/646760924 >> v2:172.21.15.167:3300/0 conn(0x7f38900b57b0 0x7f389015aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.394+0000 7f38963a9640 1 -- 172.21.15.81:0/646760924 >> 172.21.15.81:0/646760924 conn(0x7f38900d0930 msgr2=0x7f38900d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:34.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.394+0000 7f38963a9640 1 -- 172.21.15.81:0/646760924 shutdown_connections 2024-06-29T15:55:34.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.394+0000 7f38963a9640 1 -- 172.21.15.81:0/646760924 wait complete. 2024-06-29T15:55:34.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.396+0000 7f38963a9640 1 Processor -- start 2024-06-29T15:55:34.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.396+0000 7f38963a9640 1 -- start start 2024-06-29T15:55:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.397+0000 7f38963a9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f389006ef90 0x7f38901f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.397+0000 7f38963a9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38900b57b0 0x7f38900aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.397+0000 7f38963a9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38900ce1d0 0x7f389014d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.397+0000 7f38963a9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f38900cecf0 con 0x7f389006ef90 2024-06-29T15:55:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.397+0000 7f38963a9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f38900b4b80 con 0x7f38900ce1d0 2024-06-29T15:55:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.397+0000 7f38963a9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f38900cee70 con 0x7f38900b57b0 2024-06-29T15:55:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.397+0000 7f388ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f389006ef90 0x7f38901f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.397+0000 7f388ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f389006ef90 0x7f38901f2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33222/0 (socket says 172.21.15.81:33222) 2024-06-29T15:55:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.397+0000 7f388ffff640 1 -- 172.21.15.81:0/3353788562 learned_addr learned my addr 172.21.15.81:0/3353788562 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:34.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f388f7fe640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38900b57b0 0x7f38900aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f3894946640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38900ce1d0 0x7f389014d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f388ffff640 1 -- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38900b57b0 msgr2=0x7f38900aaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f388ffff640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38900b57b0 0x7f38900aaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f388ffff640 1 -- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38900ce1d0 msgr2=0x7f389014d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f388ffff640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38900ce1d0 0x7f389014d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f388ffff640 1 -- 172.21.15.81:0/3353788562 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38900aaf50 con 0x7f389006ef90 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f388ffff640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f389006ef90 0x7f38901f2fd0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f388000ea20 tx=0x7f388000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f388cff9640 1 -- 172.21.15.81:0/3353788562 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388000ce40 con 0x7f389006ef90 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f388cff9640 1 -- 172.21.15.81:0/3353788562 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3880004020 con 0x7f389006ef90 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f38963a9640 1 -- 172.21.15.81:0/3353788562 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38901f39b0 con 0x7f389006ef90 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f388cff9640 1 -- 172.21.15.81:0/3353788562 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880004470 con 0x7f389006ef90 2024-06-29T15:55:34.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.398+0000 7f38963a9640 1 -- 172.21.15.81:0/3353788562 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38900a9660 con 0x7f389006ef90 2024-06-29T15:55:34.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.399+0000 7f38963a9640 1 -- 172.21.15.81:0/3353788562 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f389006e2a0 con 0x7f389006ef90 2024-06-29T15:55:34.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.400+0000 7f388cff9640 1 -- 172.21.15.81:0/3353788562 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f38800035f0 con 0x7f389006ef90 2024-06-29T15:55:34.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.400+0000 7f388cff9640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f387c0763f0 0x7f387c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.400+0000 7f388cff9640 1 -- 172.21.15.81:0/3353788562 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 4957+0+0 (secure 0 0 0) 0x7f3880097fb0 con 0x7f389006ef90 2024-06-29T15:55:34.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.400+0000 7f388f7fe640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f387c0763f0 0x7f387c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.401+0000 7f388f7fe640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f387c0763f0 0x7f387c078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3884033000 tx=0x7f3884004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:34.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.405+0000 7f388cff9640 1 -- 172.21.15.81:0/3353788562 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f389006e2a0 con 0x7f389006ef90 2024-06-29T15:55:34.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.651+0000 7f38963a9640 1 -- 172.21.15.81:0/3353788562 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f389006f3a0 con 0x7f389006ef90 2024-06-29T15:55:34.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.652+0000 7f388cff9640 1 -- 172.21.15.81:0/3353788562 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v65) ==== 80+0+5304 (secure 0 0 0) 0x7f38800619e0 con 0x7f389006ef90 2024-06-29T15:55:34.652 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:55:34.652 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T15:55:34.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.653 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:55:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.655 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.656 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.657 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.658 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:55:34.660 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.661 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.662 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.664 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T15:55:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:55:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.653+0000 7f38963a9640 1 -- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f387c0763f0 msgr2=0x7f387c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.653+0000 7f38963a9640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f387c0763f0 0x7f387c078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3884033000 tx=0x7f3884004730 comp rx=0 tx=0).stop 2024-06-29T15:55:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.653+0000 7f38963a9640 1 -- 172.21.15.81:0/3353788562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f389006ef90 msgr2=0x7f38901f2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.653+0000 7f38963a9640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f389006ef90 0x7f38901f2fd0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f388000ea20 tx=0x7f388000eee0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.654+0000 7f38963a9640 1 -- 172.21.15.81:0/3353788562 shutdown_connections 2024-06-29T15:55:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.654+0000 7f38963a9640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f387c0763f0 0x7f387c078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.654+0000 7f38963a9640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38900ce1d0 0x7f389014d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.654+0000 7f38963a9640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38900b57b0 0x7f38900aaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.654+0000 7f38963a9640 1 --2- 172.21.15.81:0/3353788562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f389006ef90 0x7f38901f2fd0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.654+0000 7f38963a9640 1 -- 172.21.15.81:0/3353788562 >> 172.21.15.81:0/3353788562 conn(0x7f38900d0930 msgr2=0x7f38901583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:34.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.654+0000 7f38963a9640 1 -- 172.21.15.81:0/3353788562 shutdown_connections 2024-06-29T15:55:34.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.654+0000 7f38963a9640 1 -- 172.21.15.81:0/3353788562 wait complete. 2024-06-29T15:55:34.668 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:34.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.780+0000 7f848047f640 1 -- 172.21.15.81:0/3817376869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8478074520 msgr2=0x7f847806f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.780+0000 7f848047f640 1 --2- 172.21.15.81:0/3817376869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8478074520 0x7f847806f140 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8474009860 tx=0x7f847402a5a0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.780+0000 7f848047f640 1 -- 172.21.15.81:0/3817376869 shutdown_connections 2024-06-29T15:55:34.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.780+0000 7f848047f640 1 --2- 172.21.15.81:0/3817376869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8478074520 0x7f847806f140 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.780+0000 7f848047f640 1 -- 172.21.15.81:0/3817376869 >> 172.21.15.81:0/3817376869 conn(0x7f8478072950 msgr2=0x7f84780b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.780+0000 7f848047f640 1 -- 172.21.15.81:0/3817376869 shutdown_connections 2024-06-29T15:55:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.780+0000 7f848047f640 1 -- 172.21.15.81:0/3817376869 wait complete. 2024-06-29T15:55:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.782+0000 7f848047f640 1 Processor -- start 2024-06-29T15:55:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.782+0000 7f848047f640 1 -- start start 2024-06-29T15:55:34.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.783+0000 7f848047f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f847806f680 0x7f84781475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.783+0000 7f848047f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f847806fa90 0x7f8478149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f848047f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8478074520 0x7f847814a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f848047f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8478074a80 con 0x7f847806f680 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f848047f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f847815b220 con 0x7f8478074520 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f848047f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f84780af310 con 0x7f847806fa90 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847e21b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f847806f680 0x7f84781475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847e21b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f847806f680 0x7f84781475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33230/0 (socket says 172.21.15.81:33230) 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847e21b640 1 -- 172.21.15.81:0/4285882524 learned_addr learned my addr 172.21.15.81:0/4285882524 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847da1a640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f847806fa90 0x7f8478149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847ea1c640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8478074520 0x7f847814a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847e21b640 1 -- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f847806fa90 msgr2=0x7f8478149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847e21b640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f847806fa90 0x7f8478149e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847e21b640 1 -- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8478074520 msgr2=0x7f847814a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847e21b640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8478074520 0x7f847814a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847e21b640 1 -- 172.21.15.81:0/4285882524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847814b690 con 0x7f847806f680 2024-06-29T15:55:34.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f847e21b640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f847806f680 0x7f84781475d0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f847402ab90 tx=0x7f8474033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:34.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f846effd640 1 -- 172.21.15.81:0/4285882524 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8474031b50 con 0x7f847806f680 2024-06-29T15:55:34.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f846effd640 1 -- 172.21.15.81:0/4285882524 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8474031de0 con 0x7f847806f680 2024-06-29T15:55:34.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f846effd640 1 -- 172.21.15.81:0/4285882524 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84740332d0 con 0x7f847806f680 2024-06-29T15:55:34.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.784+0000 7f848047f640 1 -- 172.21.15.81:0/4285882524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84780cfd00 con 0x7f847806f680 2024-06-29T15:55:34.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.785+0000 7f848047f640 1 -- 172.21.15.81:0/4285882524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84780afcf0 con 0x7f847806f680 2024-06-29T15:55:34.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.785+0000 7f848047f640 1 -- 172.21.15.81:0/4285882524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f847806f140 con 0x7f847806f680 2024-06-29T15:55:34.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.786+0000 7f846effd640 1 -- 172.21.15.81:0/4285882524 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f8474033470 con 0x7f847806f680 2024-06-29T15:55:34.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.787+0000 7f846effd640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f845c076420 0x7f845c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:34.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.787+0000 7f846effd640 1 -- 172.21.15.81:0/4285882524 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 4957+0+0 (secure 0 0 0) 0x7f84740b8850 con 0x7f847806f680 2024-06-29T15:55:34.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.787+0000 7f847da1a640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f845c076420 0x7f845c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:34.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.787+0000 7f847da1a640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f845c076420 0x7f845c078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f847814afc0 tx=0x7f8468009040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:34.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.791+0000 7f846effd640 1 -- 172.21.15.81:0/4285882524 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f847403f030 con 0x7f847806f680 2024-06-29T15:55:34.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.971+0000 7f848047f640 1 -- 172.21.15.81:0/4285882524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8478159be0 con 0x7f847806f680 2024-06-29T15:55:34.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.971+0000 7f846effd640 1 -- 172.21.15.81:0/4285882524 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) ==== 74+0+13048 (secure 0 0 0) 0x7f8474031770 con 0x7f847806f680 2024-06-29T15:55:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:34.972 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":192,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:55:34.272730+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:34.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.973+0000 7f848047f640 1 -- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f845c076420 msgr2=0x7f845c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.973+0000 7f848047f640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f845c076420 0x7f845c078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f847814afc0 tx=0x7f8468009040 comp rx=0 tx=0).stop 2024-06-29T15:55:34.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.973+0000 7f848047f640 1 -- 172.21.15.81:0/4285882524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f847806f680 msgr2=0x7f84781475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:34.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.973+0000 7f848047f640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f847806f680 0x7f84781475d0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f847402ab90 tx=0x7f8474033ba0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.973+0000 7f848047f640 1 -- 172.21.15.81:0/4285882524 shutdown_connections 2024-06-29T15:55:34.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.973+0000 7f848047f640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f845c076420 0x7f845c078930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.973+0000 7f848047f640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8478074520 0x7f847814a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.973+0000 7f848047f640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f847806fa90 0x7f8478149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.973+0000 7f848047f640 1 --2- 172.21.15.81:0/4285882524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f847806f680 0x7f84781475d0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:34.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.974+0000 7f848047f640 1 -- 172.21.15.81:0/4285882524 >> 172.21.15.81:0/4285882524 conn(0x7f8478072950 msgr2=0x7f8478150c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:34.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.974+0000 7f848047f640 1 -- 172.21.15.81:0/4285882524 shutdown_connections 2024-06-29T15:55:34.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:34.974+0000 7f848047f640 1 -- 172.21.15.81:0/4285882524 wait complete. 2024-06-29T15:55:34.985 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:55:34.986 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:55:35.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.101+0000 7f5e0679e640 1 -- 172.21.15.81:0/3576007730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e000cccf0 msgr2=0x7f5e000d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.101+0000 7f5e0679e640 1 --2- 172.21.15.81:0/3576007730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e000cccf0 0x7f5e000d0230 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f5e00073550 tx=0x7f5df402a4a0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.101+0000 7f5e0679e640 1 -- 172.21.15.81:0/3576007730 shutdown_connections 2024-06-29T15:55:35.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.101+0000 7f5e0679e640 1 --2- 172.21.15.81:0/3576007730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e000cccf0 0x7f5e000d0230 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.101+0000 7f5e0679e640 1 --2- 172.21.15.81:0/3576007730 >> v2:172.21.15.167:3300/0 conn(0x7f5e000b3200 0x7f5e000cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.101+0000 7f5e0679e640 1 -- 172.21.15.81:0/3576007730 >> 172.21.15.81:0/3576007730 conn(0x7f5e000b3680 msgr2=0x7f5e000b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:35.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.101+0000 7f5e0679e640 1 -- 172.21.15.81:0/3576007730 shutdown_connections 2024-06-29T15:55:35.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.101+0000 7f5e0679e640 1 -- 172.21.15.81:0/3576007730 wait complete. 2024-06-29T15:55:35.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.103+0000 7f5e0679e640 1 Processor -- start 2024-06-29T15:55:35.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.103+0000 7f5e0679e640 1 -- start start 2024-06-29T15:55:35.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5e0679e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e000b3200 0x7f5e0009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5e0679e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e000cccf0 0x7f5e001d03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5e0679e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e000d0770 0x7f5e001e2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5e0679e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5e000cedb0 con 0x7f5e000b3200 2024-06-29T15:55:35.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5e0679e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5e000cec30 con 0x7f5e000d0770 2024-06-29T15:55:35.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5e0679e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5e000cef30 con 0x7f5e000cccf0 2024-06-29T15:55:35.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e000b3200 0x7f5e0009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e000b3200 0x7f5e0009c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33254/0 (socket says 172.21.15.81:33254) 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfffff640 1 -- 172.21.15.81:0/2759846708 learned_addr learned my addr 172.21.15.81:0/2759846708 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dff7fe640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e000cccf0 0x7f5e001d03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5e04d3b640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e000d0770 0x7f5e001e2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfffff640 1 -- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e000cccf0 msgr2=0x7f5e001d03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfffff640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e000cccf0 0x7f5e001d03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfffff640 1 -- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e000d0770 msgr2=0x7f5e001e2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfffff640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e000d0770 0x7f5e001e2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfffff640 1 -- 172.21.15.81:0/2759846708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e001e0fa0 con 0x7f5e000b3200 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dff7fe640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e000cccf0 0x7f5e001d03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfffff640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e000b3200 0x7f5e0009c0c0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f5df000c970 tx=0x7f5df000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfcff9640 1 -- 172.21.15.81:0/2759846708 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df0007b50 con 0x7f5e000b3200 2024-06-29T15:55:35.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfcff9640 1 -- 172.21.15.81:0/2759846708 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5df0007de0 con 0x7f5e000b3200 2024-06-29T15:55:35.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5e0679e640 1 -- 172.21.15.81:0/2759846708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e000c9eb0 con 0x7f5e000b3200 2024-06-29T15:55:35.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5dfcff9640 1 -- 172.21.15.81:0/2759846708 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df000f6b0 con 0x7f5e000b3200 2024-06-29T15:55:35.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.105+0000 7f5e0679e640 1 -- 172.21.15.81:0/2759846708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e000ca470 con 0x7f5e000b3200 2024-06-29T15:55:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.106+0000 7f5e0679e640 1 -- 172.21.15.81:0/2759846708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e000cc7b0 con 0x7f5e000b3200 2024-06-29T15:55:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.107+0000 7f5dfcff9640 1 -- 172.21.15.81:0/2759846708 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f5df000f850 con 0x7f5e000b3200 2024-06-29T15:55:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.107+0000 7f5dfcff9640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f5dec076420 0x7f5dec078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.108+0000 7f5dfcff9640 1 -- 172.21.15.81:0/2759846708 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 4957+0+0 (secure 0 0 0) 0x7f5df0097da0 con 0x7f5e000b3200 2024-06-29T15:55:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.108+0000 7f5dff7fe640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f5dec076420 0x7f5dec078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.108+0000 7f5dff7fe640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f5dec076420 0x7f5dec078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5df4034000 tx=0x7f5df4004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.112+0000 7f5dfcff9640 1 -- 172.21.15.81:0/2759846708 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5e001e0fa0 con 0x7f5e000b3200 2024-06-29T15:55:35.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.293+0000 7f5e0679e640 1 -- 172.21.15.81:0/2759846708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5e001e8700 con 0x7f5e000b3200 2024-06-29T15:55:35.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.293+0000 7f5dfcff9640 1 -- 172.21.15.81:0/2759846708 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v192) ==== 93+0+10 (secure 0 0 0) 0x7f5df0014030 con 0x7f5e000b3200 2024-06-29T15:55:35.294 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:55:35.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 -- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f5dec076420 msgr2=0x7f5dec078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f5dec076420 0x7f5dec078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5df4034000 tx=0x7f5df4004730 comp rx=0 tx=0).stop 2024-06-29T15:55:35.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 -- 172.21.15.81:0/2759846708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e000b3200 msgr2=0x7f5e0009c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e000b3200 0x7f5e0009c0c0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f5df000c970 tx=0x7f5df000ce30 comp rx=0 tx=0).stop 2024-06-29T15:55:35.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 -- 172.21.15.81:0/2759846708 shutdown_connections 2024-06-29T15:55:35.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f5dec076420 0x7f5dec078930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e000d0770 0x7f5e001e2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e000cccf0 0x7f5e001d03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 --2- 172.21.15.81:0/2759846708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e000b3200 0x7f5e0009c0c0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 -- 172.21.15.81:0/2759846708 >> 172.21.15.81:0/2759846708 conn(0x7f5e000b3680 msgr2=0x7f5e000d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:35.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 -- 172.21.15.81:0/2759846708 shutdown_connections 2024-06-29T15:55:35.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.295+0000 7f5e0679e640 1 -- 172.21.15.81:0/2759846708 wait complete. 2024-06-29T15:55:35.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:55:35.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.419+0000 7f292e93c640 1 -- 172.21.15.81:0/658331817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2910003a50 msgr2=0x7f2910003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.419+0000 7f292e93c640 1 --2- 172.21.15.81:0/658331817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2910003a50 0x7f2910003ec0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f291800bb20 tx=0x7f2910005830 comp rx=0 tx=0).stop 2024-06-29T15:55:35.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.419+0000 7f292e93c640 1 -- 172.21.15.81:0/658331817 shutdown_connections 2024-06-29T15:55:35.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.419+0000 7f292e93c640 1 --2- 172.21.15.81:0/658331817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2910003a50 0x7f2910003ec0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.419+0000 7f292e93c640 1 -- 172.21.15.81:0/658331817 >> 172.21.15.81:0/658331817 conn(0x7f292806ccc0 msgr2=0x7f29280b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:35.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.419+0000 7f292e93c640 1 -- 172.21.15.81:0/658331817 shutdown_connections 2024-06-29T15:55:35.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.419+0000 7f292e93c640 1 -- 172.21.15.81:0/658331817 wait complete. 2024-06-29T15:55:35.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.421+0000 7f292e93c640 1 Processor -- start 2024-06-29T15:55:35.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.421+0000 7f292e93c640 1 -- start start 2024-06-29T15:55:35.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f292e93c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2910003a50 0x7f2928141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f292e93c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f292813bdd0 0x7f29281f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f292e93c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29281f6a00 0x7f29281f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f292e93c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2928073400 con 0x7f292813bdd0 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f292e93c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2928073280 con 0x7f2910003a50 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f292e93c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2928073580 con 0x7f29281f6a00 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f29277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f292813bdd0 0x7f29281f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f29277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f292813bdd0 0x7f29281f6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33270/0 (socket says 172.21.15.81:33270) 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f29277fe640 1 -- 172.21.15.81:0/4262502543 learned_addr learned my addr 172.21.15.81:0/4262502543 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f2927fff640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2910003a50 0x7f2928141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f292ced9640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29281f6a00 0x7f29281f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f29277fe640 1 -- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29281f6a00 msgr2=0x7f29281f6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f29277fe640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29281f6a00 0x7f29281f6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f29277fe640 1 -- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2910003a50 msgr2=0x7f2928141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f29277fe640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2910003a50 0x7f2928141a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f29277fe640 1 -- 172.21.15.81:0/4262502543 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29281422b0 con 0x7f292813bdd0 2024-06-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f29277fe640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f292813bdd0 0x7f29281f6270 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f291c004c90 tx=0x7f291c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f2924ff9640 1 -- 172.21.15.81:0/4262502543 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291c015020 con 0x7f292813bdd0 2024-06-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f2924ff9640 1 -- 172.21.15.81:0/4262502543 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f291c011750 con 0x7f292813bdd0 2024-06-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f292e93c640 1 -- 172.21.15.81:0/4262502543 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29281f67b0 con 0x7f292813bdd0 2024-06-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.423+0000 7f2924ff9640 1 -- 172.21.15.81:0/4262502543 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291c011b20 con 0x7f292813bdd0 2024-06-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.424+0000 7f292e93c640 1 -- 172.21.15.81:0/4262502543 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29281443a0 con 0x7f292813bdd0 2024-06-29T15:55:35.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.424+0000 7f292e93c640 1 -- 172.21.15.81:0/4262502543 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2928142020 con 0x7f292813bdd0 2024-06-29T15:55:35.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.425+0000 7f2924ff9640 1 -- 172.21.15.81:0/4262502543 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f291c018020 con 0x7f292813bdd0 2024-06-29T15:55:35.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.426+0000 7f2924ff9640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2914076420 0x7f2914078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.426+0000 7f2924ff9640 1 -- 172.21.15.81:0/4262502543 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 4957+0+0 (secure 0 0 0) 0x7f291c097a40 con 0x7f292813bdd0 2024-06-29T15:55:35.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.426+0000 7f2927fff640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2914076420 0x7f2914078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.426+0000 7f2927fff640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2914076420 0x7f2914078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2918036000 tx=0x7f291803c040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.430+0000 7f2924ff9640 1 -- 172.21.15.81:0/4262502543 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f291c01f030 con 0x7f292813bdd0 2024-06-29T15:55:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.612+0000 7f292e93c640 1 -- 172.21.15.81:0/4262502543 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f292815c920 con 0x7f292813bdd0 2024-06-29T15:55:35.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.612+0000 7f2924ff9640 1 -- 172.21.15.81:0/4262502543 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v192) ==== 84+0+65 (secure 0 0 0) 0x7f291c097e20 con 0x7f292813bdd0 2024-06-29T15:55:35.614 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:35.614 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:55:35.614 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:35.614 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T15:55:35.614 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T15:55:35.614 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:35.618 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:55:35.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 -- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2914076420 msgr2=0x7f2914078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2914076420 0x7f2914078930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2918036000 tx=0x7f291803c040 comp rx=0 tx=0).stop 2024-06-29T15:55:35.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 -- 172.21.15.81:0/4262502543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f292813bdd0 msgr2=0x7f29281f6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f292813bdd0 0x7f29281f6270 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f291c004c90 tx=0x7f291c001630 comp rx=0 tx=0).stop 2024-06-29T15:55:35.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 -- 172.21.15.81:0/4262502543 shutdown_connections 2024-06-29T15:55:35.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2914076420 0x7f2914078930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29281f6a00 0x7f29281f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f292813bdd0 0x7f29281f6270 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 --2- 172.21.15.81:0/4262502543 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2910003a50 0x7f2928141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 -- 172.21.15.81:0/4262502543 >> 172.21.15.81:0/4262502543 conn(0x7f292806ccc0 msgr2=0x7f29280b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:35.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 -- 172.21.15.81:0/4262502543 shutdown_connections 2024-06-29T15:55:35.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.614+0000 7f292e93c640 1 -- 172.21.15.81:0/4262502543 wait complete. 2024-06-29T15:55:35.626 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:55:35.627 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:55:35.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.744+0000 7f76dce95640 1 -- 172.21.15.81:0/2677513695 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76d80b21a0 msgr2=0x7f76d80d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.744+0000 7f76dce95640 1 --2- 172.21.15.81:0/2677513695 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76d80b21a0 0x7f76d80d16b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f76d0005fd0 tx=0x7f76d002aa90 comp rx=0 tx=0).stop 2024-06-29T15:55:35.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.744+0000 7f76dce95640 1 -- 172.21.15.81:0/2677513695 shutdown_connections 2024-06-29T15:55:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.744+0000 7f76dce95640 1 --2- 172.21.15.81:0/2677513695 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76d80b21a0 0x7f76d80d16b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.744+0000 7f76dce95640 1 --2- 172.21.15.81:0/2677513695 >> v2:172.21.15.167:3300/0 conn(0x7f76d80ae8f0 0x7f76d815c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.744+0000 7f76dce95640 1 -- 172.21.15.81:0/2677513695 >> 172.21.15.81:0/2677513695 conn(0x7f76d809aa60 msgr2=0x7f76d809ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.744+0000 7f76dce95640 1 -- 172.21.15.81:0/2677513695 shutdown_connections 2024-06-29T15:55:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.744+0000 7f76dce95640 1 -- 172.21.15.81:0/2677513695 wait complete. 2024-06-29T15:55:35.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.746+0000 7f76dce95640 1 Processor -- start 2024-06-29T15:55:35.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.746+0000 7f76dce95640 1 -- start start 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76dce95640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ae8f0 0x7f76d8150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76dce95640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76d80b2cb0 0x7f76d815b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76dce95640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76d815e810 0x7f76d815ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76dce95640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8160020 con 0x7f76d80ae8f0 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76dce95640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f76d815fea0 con 0x7f76d815e810 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76dce95640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f76d81601a0 con 0x7f76d80b2cb0 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76d659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ae8f0 0x7f76d8150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76d659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ae8f0 0x7f76d8150770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33276/0 (socket says 172.21.15.81:33276) 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76d659c640 1 -- 172.21.15.81:0/3609658626 learned_addr learned my addr 172.21.15.81:0/3609658626 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76cdd9b640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76d80b2cb0 0x7f76d815b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76d6d9d640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76d815e810 0x7f76d815ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76d659c640 1 -- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76d80b2cb0 msgr2=0x7f76d815b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76d659c640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76d80b2cb0 0x7f76d815b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76d659c640 1 -- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76d815e810 msgr2=0x7f76d815ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76d659c640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76d815e810 0x7f76d815ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.747+0000 7f76d659c640 1 -- 172.21.15.81:0/3609658626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d815f160 con 0x7f76d80ae8f0 2024-06-29T15:55:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.748+0000 7f76d659c640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ae8f0 0x7f76d8150770 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f76c800e7c0 tx=0x7f76c800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.748+0000 7f76cf7fe640 1 -- 172.21.15.81:0/3609658626 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c800cc60 con 0x7f76d80ae8f0 2024-06-29T15:55:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.748+0000 7f76cf7fe640 1 -- 172.21.15.81:0/3609658626 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76c8010430 con 0x7f76d80ae8f0 2024-06-29T15:55:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.748+0000 7f76cf7fe640 1 -- 172.21.15.81:0/3609658626 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8010860 con 0x7f76d80ae8f0 2024-06-29T15:55:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.748+0000 7f76dce95640 1 -- 172.21.15.81:0/3609658626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d8153eb0 con 0x7f76d80ae8f0 2024-06-29T15:55:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.748+0000 7f76dce95640 1 -- 172.21.15.81:0/3609658626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d8154470 con 0x7f76d80ae8f0 2024-06-29T15:55:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.749+0000 7f76dce95640 1 -- 172.21.15.81:0/3609658626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d80d16b0 con 0x7f76d80ae8f0 2024-06-29T15:55:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.750+0000 7f76cf7fe640 1 -- 172.21.15.81:0/3609658626 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f76c8010a00 con 0x7f76d80ae8f0 2024-06-29T15:55:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.750+0000 7f76cf7fe640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f76b80763f0 0x7f76b8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.750+0000 7f76cf7fe640 1 -- 172.21.15.81:0/3609658626 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 4957+0+0 (secure 0 0 0) 0x7f76c8098070 con 0x7f76d80ae8f0 2024-06-29T15:55:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.750+0000 7f76cdd9b640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f76b80763f0 0x7f76b8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:35.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.751+0000 7f76cdd9b640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f76b80763f0 0x7f76b8078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f76d8070b40 tx=0x7f76c0009040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.754+0000 7f76cf7fe640 1 -- 172.21.15.81:0/3609658626 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76d815f160 con 0x7f76d80ae8f0 2024-06-29T15:55:35.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:35.938+0000 7f76dce95640 1 -- 172.21.15.81:0/3609658626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f76d80716d0 con 0x7f76d80ae8f0 2024-06-29T15:55:36.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.268+0000 7f76cf7fe640 1 -- 172.21.15.81:0/3609658626 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v193) ==== 138+0+0 (secure 0 0 0) 0x7f76c80613c0 con 0x7f76d80ae8f0 2024-06-29T15:55:36.269 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T15:55:36.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 -- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f76b80763f0 msgr2=0x7f76b8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:36.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f76b80763f0 0x7f76b8078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f76d8070b40 tx=0x7f76c0009040 comp rx=0 tx=0).stop 2024-06-29T15:55:36.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 -- 172.21.15.81:0/3609658626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ae8f0 msgr2=0x7f76d8150770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:36.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ae8f0 0x7f76d8150770 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f76c800e7c0 tx=0x7f76c800ec80 comp rx=0 tx=0).stop 2024-06-29T15:55:36.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 -- 172.21.15.81:0/3609658626 shutdown_connections 2024-06-29T15:55:36.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f76b80763f0 0x7f76b8078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76d815e810 0x7f76d815ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76d80b2cb0 0x7f76d815b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 --2- 172.21.15.81:0/3609658626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ae8f0 0x7f76d8150770 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 -- 172.21.15.81:0/3609658626 >> 172.21.15.81:0/3609658626 conn(0x7f76d809aa60 msgr2=0x7f76d809ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:36.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 -- 172.21.15.81:0/3609658626 shutdown_connections 2024-06-29T15:55:36.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.270+0000 7f76dce95640 1 -- 172.21.15.81:0/3609658626 wait complete. 2024-06-29T15:55:36.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:55:36.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.407+0000 7f596289e640 1 -- 172.21.15.81:0/3659786767 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f595c074520 msgr2=0x7f595c06f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:36.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.407+0000 7f596289e640 1 --2- 172.21.15.81:0/3659786767 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f595c074520 0x7f595c06f140 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f594c005fd0 tx=0x7f594c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.407+0000 7f596289e640 1 -- 172.21.15.81:0/3659786767 shutdown_connections 2024-06-29T15:55:36.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.407+0000 7f596289e640 1 --2- 172.21.15.81:0/3659786767 >> v2:172.21.15.167:3300/0 conn(0x7f595c06fa90 0x7f595c06ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.407+0000 7f596289e640 1 --2- 172.21.15.81:0/3659786767 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f595c074520 0x7f595c06f140 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.407+0000 7f596289e640 1 -- 172.21.15.81:0/3659786767 >> 172.21.15.81:0/3659786767 conn(0x7f595c072950 msgr2=0x7f595c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:36.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.407+0000 7f596289e640 1 -- 172.21.15.81:0/3659786767 shutdown_connections 2024-06-29T15:55:36.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.407+0000 7f596289e640 1 -- 172.21.15.81:0/3659786767 wait complete. 2024-06-29T15:55:36.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.409+0000 7f596289e640 1 Processor -- start 2024-06-29T15:55:36.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.409+0000 7f596289e640 1 -- start start 2024-06-29T15:55:36.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.410+0000 7f596289e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f595c06f680 0x7f595c151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:36.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.410+0000 7f596289e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f595c06fa90 0x7f595c14a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:36.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.410+0000 7f596289e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f595c074520 0x7f595c0cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:36.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.410+0000 7f596289e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f595c15b010 con 0x7f595c06fa90 2024-06-29T15:55:36.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.410+0000 7f596289e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f595c06d150 con 0x7f595c074520 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.410+0000 7f596289e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f595c15b190 con 0x7f595c06f680 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f595b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f595c06fa90 0x7f595c14a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f595b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f595c06fa90 0x7f595c14a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33278/0 (socket says 172.21.15.81:33278) 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f595b7fe640 1 -- 172.21.15.81:0/379868678 learned_addr learned my addr 172.21.15.81:0/379868678 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f595bfff640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f595c06f680 0x7f595c151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f5960e3b640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f595c074520 0x7f595c0cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f595b7fe640 1 -- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f595c06f680 msgr2=0x7f595c151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f595b7fe640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f595c06f680 0x7f595c151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f595b7fe640 1 -- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f595c074520 msgr2=0x7f595c0cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f595b7fe640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f595c074520 0x7f595c0cf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f595b7fe640 1 -- 172.21.15.81:0/379868678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595c0ccbf0 con 0x7f595c06fa90 2024-06-29T15:55:36.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f595b7fe640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f595c06fa90 0x7f595c14a9c0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f595000c3a0 tx=0x7f595000c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:36.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f5958ff9640 1 -- 172.21.15.81:0/379868678 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950017070 con 0x7f595c06fa90 2024-06-29T15:55:36.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f5958ff9640 1 -- 172.21.15.81:0/379868678 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59500053c0 con 0x7f595c06fa90 2024-06-29T15:55:36.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f596289e640 1 -- 172.21.15.81:0/379868678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f595c0cced0 con 0x7f595c06fa90 2024-06-29T15:55:36.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f5958ff9640 1 -- 172.21.15.81:0/379868678 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950003850 con 0x7f595c06fa90 2024-06-29T15:55:36.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.411+0000 7f596289e640 1 -- 172.21.15.81:0/379868678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f595c0cd490 con 0x7f595c06fa90 2024-06-29T15:55:36.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.412+0000 7f596289e640 1 -- 172.21.15.81:0/379868678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f595c151ed0 con 0x7f595c06fa90 2024-06-29T15:55:36.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.413+0000 7f5958ff9640 1 -- 172.21.15.81:0/379868678 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f595001a070 con 0x7f595c06fa90 2024-06-29T15:55:36.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.414+0000 7f5958ff9640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f59480763f0 0x7f5948078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:36.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.414+0000 7f5958ff9640 1 -- 172.21.15.81:0/379868678 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 5327+0+0 (secure 0 0 0) 0x7f5950096c50 con 0x7f595c06fa90 2024-06-29T15:55:36.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.414+0000 7f595bfff640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f59480763f0 0x7f5948078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:36.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.415+0000 7f595bfff640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f59480763f0 0x7f5948078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f594c033000 tx=0x7f594c004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:36.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.418+0000 7f5958ff9640 1 -- 172.21.15.81:0/379868678 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5950021080 con 0x7f595c06fa90 2024-06-29T15:55:36.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:36.597+0000 7f596289e640 1 -- 172.21.15.81:0/379868678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f595c068ef0 con 0x7f595c06fa90 2024-06-29T15:55:37.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.270+0000 7f5958ff9640 1 -- 172.21.15.81:0/379868678 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v194) ==== 170+0+0 (secure 0 0 0) 0x7f595005ffd0 con 0x7f595c06fa90 2024-06-29T15:55:37.272 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T15:55:37.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.275+0000 7f596289e640 1 -- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f59480763f0 msgr2=0x7f5948078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:37.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.275+0000 7f596289e640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f59480763f0 0x7f5948078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f594c033000 tx=0x7f594c004730 comp rx=0 tx=0).stop 2024-06-29T15:55:37.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.275+0000 7f596289e640 1 -- 172.21.15.81:0/379868678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f595c06fa90 msgr2=0x7f595c14a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:37.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.275+0000 7f596289e640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f595c06fa90 0x7f595c14a9c0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f595000c3a0 tx=0x7f595000c7a0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.276+0000 7f596289e640 1 -- 172.21.15.81:0/379868678 shutdown_connections 2024-06-29T15:55:37.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.276+0000 7f596289e640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f59480763f0 0x7f5948078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.276+0000 7f596289e640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f595c074520 0x7f595c0cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.276+0000 7f596289e640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f595c06fa90 0x7f595c14a9c0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.276+0000 7f596289e640 1 --2- 172.21.15.81:0/379868678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f595c06f680 0x7f595c151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.276+0000 7f596289e640 1 -- 172.21.15.81:0/379868678 >> 172.21.15.81:0/379868678 conn(0x7f595c072950 msgr2=0x7f595c0b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:37.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.280+0000 7f596289e640 1 -- 172.21.15.81:0/379868678 shutdown_connections 2024-06-29T15:55:37.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.282+0000 7f596289e640 1 -- 172.21.15.81:0/379868678 wait complete. 2024-06-29T15:55:37.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:55:37.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.419+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/3757939856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00b6eb0 msgr2=0x7fa4a00d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:37.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.419+0000 7fa4a7a5e640 1 --2- 172.21.15.81:0/3757939856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00b6eb0 0x7fa4a00d02d0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fa4a00b41b0 tx=0x7fa49c02a470 comp rx=0 tx=0).stop 2024-06-29T15:55:37.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.419+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/3757939856 shutdown_connections 2024-06-29T15:55:37.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.419+0000 7fa4a7a5e640 1 --2- 172.21.15.81:0/3757939856 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a00ce510 0x7fa4a00ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.419+0000 7fa4a7a5e640 1 --2- 172.21.15.81:0/3757939856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00b6eb0 0x7fa4a00d02d0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.419+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/3757939856 >> 172.21.15.81:0/3757939856 conn(0x7fa4a00b3510 msgr2=0x7fa4a00b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:37.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.419+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/3757939856 shutdown_connections 2024-06-29T15:55:37.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.419+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/3757939856 wait complete. 2024-06-29T15:55:37.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.421+0000 7fa4a7a5e640 1 Processor -- start 2024-06-29T15:55:37.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.421+0000 7fa4a7a5e640 1 -- start start 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a7a5e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a00b4810 0x7fa4a015bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a7a5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00b6eb0 0x7fa4a006e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a7a5e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a00ce510 0x7fa4a015e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a7a5e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a00cd480 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a7a5e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a00b29c0 con 0x7fa4a00b4810 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a7a5e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a00cd600 con 0x7fa4a00ce510 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a4ff9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00b6eb0 0x7fa4a006e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a4ff9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00b6eb0 0x7fa4a006e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33298/0 (socket says 172.21.15.81:33298) 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a4ff9640 1 -- 172.21.15.81:0/1417372454 learned_addr learned my addr 172.21.15.81:0/1417372454 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a57fa640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a00b4810 0x7fa4a015bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.422+0000 7fa4a5ffb640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a00ce510 0x7fa4a015e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa4a4ff9640 1 -- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a00ce510 msgr2=0x7fa4a015e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:37.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa4a4ff9640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a00ce510 0x7fa4a015e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa4a4ff9640 1 -- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a00b4810 msgr2=0x7fa4a015bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa4a4ff9640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a00b4810 0x7fa4a015bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa4a4ff9640 1 -- 172.21.15.81:0/1417372454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a0162190 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa4a57fa640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a00b4810 0x7fa4a015bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa4a5ffb640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a00ce510 0x7fa4a015e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa4a4ff9640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00b6eb0 0x7fa4a006e670 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fa49000c9d0 tx=0x7fa49000ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa495ffb640 1 -- 172.21.15.81:0/1417372454 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490007c70 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa495ffb640 1 -- 172.21.15.81:0/1417372454 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa49000f450 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/1417372454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a0162470 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa495ffb640 1 -- 172.21.15.81:0/1417372454 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49000f8a0 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.423+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/1417372454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a00b0930 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.424+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/1417372454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a006f110 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.425+0000 7fa495ffb640 1 -- 172.21.15.81:0/1417372454 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fa490018400 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.426+0000 7fa495ffb640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa484076420 0x7fa484078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:37.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.426+0000 7fa495ffb640 1 -- 172.21.15.81:0/1417372454 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(194..194 src has 1..194) ==== 5709+0+0 (secure 0 0 0) 0x7fa490098650 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.426+0000 7fa4a57fa640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa484076420 0x7fa484078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:37.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.426+0000 7fa4a57fa640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa484076420 0x7fa484078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa49c034000 tx=0x7fa49c004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:37.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.429+0000 7fa495ffb640 1 -- 172.21.15.81:0/1417372454 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa490026020 con 0x7fa4a00b6eb0 2024-06-29T15:55:37.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:37.647+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/1417372454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa4a00d02d0 con 0x7fa4a00b6eb0 2024-06-29T15:55:38.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.289+0000 7fa495ffb640 1 -- 172.21.15.81:0/1417372454 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v100) ==== 174+0+0 (secure 0 0 0) 0x7fa490014030 con 0x7fa4a00b6eb0 2024-06-29T15:55:38.291 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 24 and data pool 25 2024-06-29T15:55:38.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.301+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa484076420 msgr2=0x7fa484078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.301+0000 7fa4a7a5e640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa484076420 0x7fa484078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa49c034000 tx=0x7fa49c004730 comp rx=0 tx=0).stop 2024-06-29T15:55:38.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.301+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/1417372454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00b6eb0 msgr2=0x7fa4a006e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.301+0000 7fa4a7a5e640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00b6eb0 0x7fa4a006e670 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fa49000c9d0 tx=0x7fa49000ce90 comp rx=0 tx=0).stop 2024-06-29T15:55:38.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.303+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/1417372454 shutdown_connections 2024-06-29T15:55:38.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.303+0000 7fa4a7a5e640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa484076420 0x7fa484078930 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.303+0000 7fa4a7a5e640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a00ce510 0x7fa4a015e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.303+0000 7fa4a7a5e640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00b6eb0 0x7fa4a006e670 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.303+0000 7fa4a7a5e640 1 --2- 172.21.15.81:0/1417372454 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a00b4810 0x7fa4a015bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.303+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/1417372454 >> 172.21.15.81:0/1417372454 conn(0x7fa4a00b3510 msgr2=0x7fa4a015fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:38.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.307+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/1417372454 shutdown_connections 2024-06-29T15:55:38.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.308+0000 7fa4a7a5e640 1 -- 172.21.15.81:0/1417372454 wait complete. 2024-06-29T15:55:38.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:38.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.439+0000 7f9383cc0640 1 -- 172.21.15.81:0/1404686204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93680053a0 msgr2=0x7f936800ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.439+0000 7f9383cc0640 1 --2- 172.21.15.81:0/1404686204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93680053a0 0x7f936800ba50 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f9374005fd0 tx=0x7f9368005830 comp rx=0 tx=0).stop 2024-06-29T15:55:38.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.439+0000 7f9383cc0640 1 -- 172.21.15.81:0/1404686204 shutdown_connections 2024-06-29T15:55:38.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.439+0000 7f9383cc0640 1 --2- 172.21.15.81:0/1404686204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93680053a0 0x7f936800ba50 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.439+0000 7f9383cc0640 1 -- 172.21.15.81:0/1404686204 >> 172.21.15.81:0/1404686204 conn(0x7f937c0b56d0 msgr2=0x7f937c0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:38.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.440+0000 7f9383cc0640 1 -- 172.21.15.81:0/1404686204 shutdown_connections 2024-06-29T15:55:38.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.440+0000 7f9383cc0640 1 -- 172.21.15.81:0/1404686204 wait complete. 2024-06-29T15:55:38.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.441+0000 7f9383cc0640 1 Processor -- start 2024-06-29T15:55:38.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.441+0000 7f9383cc0640 1 -- start start 2024-06-29T15:55:38.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f9383cc0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93680053a0 0x7f937c1e84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f9383cc0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f937c1f86c0 0x7f937c1f8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f9383cc0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f937c1f9010 0x7f937c1de890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f9383cc0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f937c0cecc0 con 0x7f937c1f86c0 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f9383cc0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f937c0ceb40 con 0x7f937c1f9010 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f9383cc0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f937c0cee40 con 0x7f93680053a0 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f938125b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f937c1f86c0 0x7f937c1f8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f938125b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f937c1f86c0 0x7f937c1f8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33328/0 (socket says 172.21.15.81:33328) 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f938125b640 1 -- 172.21.15.81:0/430108668 learned_addr learned my addr 172.21.15.81:0/430108668 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f9381a5c640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93680053a0 0x7f937c1e84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f938225d640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f937c1f9010 0x7f937c1de890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f938125b640 1 -- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93680053a0 msgr2=0x7f937c1e84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f938125b640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93680053a0 0x7f937c1e84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f938125b640 1 -- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f937c1f9010 msgr2=0x7f937c1de890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f938125b640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f937c1f9010 0x7f937c1de890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f938125b640 1 -- 172.21.15.81:0/430108668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f937c1e89f0 con 0x7f937c1f86c0 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f938125b640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f937c1f86c0 0x7f937c1f8ad0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f936c00e5c0 tx=0x7f936c00e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:38.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f93727fc640 1 -- 172.21.15.81:0/430108668 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c015020 con 0x7f937c1f86c0 2024-06-29T15:55:38.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f93727fc640 1 -- 172.21.15.81:0/430108668 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f936c010980 con 0x7f937c1f86c0 2024-06-29T15:55:38.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f9383cc0640 1 -- 172.21.15.81:0/430108668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f937c0b7ab0 con 0x7f937c1f86c0 2024-06-29T15:55:38.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.443+0000 7f93727fc640 1 -- 172.21.15.81:0/430108668 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c010db0 con 0x7f937c1f86c0 2024-06-29T15:55:38.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.444+0000 7f9383cc0640 1 -- 172.21.15.81:0/430108668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f937c0b8070 con 0x7f937c1f86c0 2024-06-29T15:55:38.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.444+0000 7f9383cc0640 1 -- 172.21.15.81:0/430108668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f937c0b7820 con 0x7f937c1f86c0 2024-06-29T15:55:38.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.445+0000 7f93727fc640 1 -- 172.21.15.81:0/430108668 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f936c018020 con 0x7f937c1f86c0 2024-06-29T15:55:38.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.446+0000 7f93727fc640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f93600763f0 0x7f9360078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.446+0000 7f93727fc640 1 -- 172.21.15.81:0/430108668 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5809+0+0 (secure 0 0 0) 0x7f936c098050 con 0x7f937c1f86c0 2024-06-29T15:55:38.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.446+0000 7f9381a5c640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f93600763f0 0x7f9360078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.447+0000 7f9381a5c640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f93600763f0 0x7f9360078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9378000c00 tx=0x7f9378006b70 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:38.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.450+0000 7f93727fc640 1 -- 172.21.15.81:0/430108668 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f936c01f020 con 0x7f937c1f86c0 2024-06-29T15:55:38.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.631+0000 7f9383cc0640 1 -- 172.21.15.81:0/430108668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f937c0d19e0 con 0x7f937c1f86c0 2024-06-29T15:55:38.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.632+0000 7f93727fc640 1 -- 172.21.15.81:0/430108668 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) ==== 74+0+16839 (secure 0 0 0) 0x7f936c061730 con 0x7f937c1f86c0 2024-06-29T15:55:38.634 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:38.634 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":195,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:55:37.649098+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.947285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.605625+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:38.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.634+0000 7f9383cc0640 1 -- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f93600763f0 msgr2=0x7f9360078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.634+0000 7f9383cc0640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f93600763f0 0x7f9360078900 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9378000c00 tx=0x7f9378006b70 comp rx=0 tx=0).stop 2024-06-29T15:55:38.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.634+0000 7f9383cc0640 1 -- 172.21.15.81:0/430108668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f937c1f86c0 msgr2=0x7f937c1f8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.634+0000 7f9383cc0640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f937c1f86c0 0x7f937c1f8ad0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f936c00e5c0 tx=0x7f936c00e9c0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.635+0000 7f9383cc0640 1 -- 172.21.15.81:0/430108668 shutdown_connections 2024-06-29T15:55:38.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.635+0000 7f9383cc0640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f93600763f0 0x7f9360078900 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.635+0000 7f9383cc0640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f937c1f9010 0x7f937c1de890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.635+0000 7f9383cc0640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f937c1f86c0 0x7f937c1f8ad0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.635+0000 7f9383cc0640 1 --2- 172.21.15.81:0/430108668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f93680053a0 0x7f937c1e84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.635+0000 7f9383cc0640 1 -- 172.21.15.81:0/430108668 >> 172.21.15.81:0/430108668 conn(0x7f937c0b56d0 msgr2=0x7f937c1e95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:38.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.635+0000 7f9383cc0640 1 -- 172.21.15.81:0/430108668 shutdown_connections 2024-06-29T15:55:38.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.635+0000 7f9383cc0640 1 -- 172.21.15.81:0/430108668 wait complete. 2024-06-29T15:55:38.648 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.760+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1798942022 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50a8005150 msgr2=0x7f50a800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.760+0000 7f50c5c5e640 1 --2- 172.21.15.81:0/1798942022 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50a8005150 0x7f50a800b9c0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f50b8009ed0 tx=0x7f50a8005830 comp rx=0 tx=0).stop 2024-06-29T15:55:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.760+0000 7f50bffff640 1 -- 172.21.15.81:0/1798942022 reap_dead start 2024-06-29T15:55:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.760+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1798942022 shutdown_connections 2024-06-29T15:55:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.760+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1798942022 >> 172.21.15.81:0/1798942022 conn(0x7f50c009cf30 msgr2=0x7f50c00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.760+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1798942022 shutdown_connections 2024-06-29T15:55:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.760+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1798942022 wait complete. 2024-06-29T15:55:38.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.762+0000 7f50c5c5e640 1 Processor -- start 2024-06-29T15:55:38.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.762+0000 7f50c5c5e640 1 -- start start 2024-06-29T15:55:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.763+0000 7f50c5c5e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00a1640 0x7f50c00b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.763+0000 7f50c5c5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b0df0 0x7f50c01f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.763+0000 7f50c5c5e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50c01f7e80 0x7f50c01f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.763+0000 7f50c5c5e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f50c00b8420 con 0x7f50c00b0df0 2024-06-29T15:55:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.763+0000 7f50c5c5e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f50c00b82a0 con 0x7f50c01f7e80 2024-06-29T15:55:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.763+0000 7f50c5c5e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f50c00b85a0 con 0x7f50c00a1640 2024-06-29T15:55:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50beffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b0df0 0x7f50c01f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50beffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b0df0 0x7f50c01f7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33350/0 (socket says 172.21.15.81:33350) 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50beffd640 1 -- 172.21.15.81:0/1738204879 learned_addr learned my addr 172.21.15.81:0/1738204879 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50bf7fe640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00a1640 0x7f50c00b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50bffff640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50c01f7e80 0x7f50c01f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50beffd640 1 -- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00a1640 msgr2=0x7f50c00b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50beffd640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00a1640 0x7f50c00b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50beffd640 1 -- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50c01f7e80 msgr2=0x7f50c01f4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50beffd640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50c01f7e80 0x7f50c01f4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50beffd640 1 -- 172.21.15.81:0/1738204879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c01f8290 con 0x7f50c00b0df0 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50beffd640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b0df0 0x7f50c01f7940 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f50b40118b0 tx=0x7f50b4011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50a7fff640 1 -- 172.21.15.81:0/1738204879 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b400f9a0 con 0x7f50c00b0df0 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50a7fff640 1 -- 172.21.15.81:0/1738204879 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f50b400fb40 con 0x7f50c00b0df0 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50a7fff640 1 -- 172.21.15.81:0/1738204879 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4005020 con 0x7f50c00b0df0 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1738204879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c00b3890 con 0x7f50c00b0df0 2024-06-29T15:55:38.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.764+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1738204879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c00b3e50 con 0x7f50c00b0df0 2024-06-29T15:55:38.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.765+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1738204879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50c00d0480 con 0x7f50c00b0df0 2024-06-29T15:55:38.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.766+0000 7f50a7fff640 1 -- 172.21.15.81:0/1738204879 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f50b40051c0 con 0x7f50c00b0df0 2024-06-29T15:55:38.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.766+0000 7f50a7fff640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f50ac0763f0 0x7f50ac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:38.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.766+0000 7f50a7fff640 1 -- 172.21.15.81:0/1738204879 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5809+0+0 (secure 0 0 0) 0x7f50b40986a0 con 0x7f50c00b0df0 2024-06-29T15:55:38.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.767+0000 7f50bf7fe640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f50ac0763f0 0x7f50ac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:38.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.767+0000 7f50bf7fe640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f50ac0763f0 0x7f50ac078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f50b0003670 tx=0x7f50b0001ce0 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:38.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.771+0000 7f50a7fff640 1 -- 172.21.15.81:0/1738204879 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f50c01f8290 con 0x7f50c00b0df0 2024-06-29T15:55:38.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.955+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1738204879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50c00b1200 con 0x7f50c00b0df0 2024-06-29T15:55:38.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.956+0000 7f50a7fff640 1 -- 172.21.15.81:0/1738204879 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) ==== 74+0+16839 (secure 0 0 0) 0x7f50b4061d70 con 0x7f50c00b0df0 2024-06-29T15:55:38.958 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:38.958 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":195,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:55:37.649098+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.947285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.605625+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:38.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f50ac0763f0 msgr2=0x7f50ac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f50ac0763f0 0x7f50ac078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f50b0003670 tx=0x7f50b0001ce0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1738204879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b0df0 msgr2=0x7f50c01f7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:38.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b0df0 0x7f50c01f7940 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f50b40118b0 tx=0x7f50b4011d70 comp rx=0 tx=0).stop 2024-06-29T15:55:38.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1738204879 shutdown_connections 2024-06-29T15:55:38.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f50ac0763f0 0x7f50ac078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50c01f7e80 0x7f50c01f4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50c00b0df0 0x7f50c01f7940 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 --2- 172.21.15.81:0/1738204879 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50c00a1640 0x7f50c00b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:38.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1738204879 >> 172.21.15.81:0/1738204879 conn(0x7f50c009cf30 msgr2=0x7f50c0160ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:38.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1738204879 shutdown_connections 2024-06-29T15:55:38.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:38.958+0000 7f50c5c5e640 1 -- 172.21.15.81:0/1738204879 wait complete. 2024-06-29T15:55:38.971 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:55:39.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.086+0000 7fa09faf1640 1 -- 172.21.15.81:0/4068620981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa09815d750 msgr2=0x7fa0980afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:39.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.086+0000 7fa09faf1640 1 --2- 172.21.15.81:0/4068620981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa09815d750 0x7fa0980afb60 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fa088005fd0 tx=0x7fa08802a540 comp rx=0 tx=0).stop 2024-06-29T15:55:39.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.086+0000 7fa09faf1640 1 -- 172.21.15.81:0/4068620981 shutdown_connections 2024-06-29T15:55:39.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.086+0000 7fa09faf1640 1 --2- 172.21.15.81:0/4068620981 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa09815c030 0x7fa09806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:39.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.086+0000 7fa09faf1640 1 --2- 172.21.15.81:0/4068620981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa09815d750 0x7fa0980afb60 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:39.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.086+0000 7fa09faf1640 1 -- 172.21.15.81:0/4068620981 >> 172.21.15.81:0/4068620981 conn(0x7fa0980d0930 msgr2=0x7fa0980d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:39.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.086+0000 7fa09faf1640 1 -- 172.21.15.81:0/4068620981 shutdown_connections 2024-06-29T15:55:39.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.086+0000 7fa09faf1640 1 -- 172.21.15.81:0/4068620981 wait complete. 2024-06-29T15:55:39.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.088+0000 7fa09faf1640 1 Processor -- start 2024-06-29T15:55:39.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.088+0000 7fa09faf1640 1 -- start start 2024-06-29T15:55:39.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09faf1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa09815c030 0x7fa0980accf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09faf1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa09815c440 0x7fa0981f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09faf1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa09815d750 0x7fa098159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09faf1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa098160290 con 0x7fa09815d750 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09faf1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa09806cb50 con 0x7fa09815c440 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09faf1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa098160410 con 0x7fa09815c030 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09e08e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa09815d750 0x7fa098159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09e08e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa09815d750 0x7fa098159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33370/0 (socket says 172.21.15.81:33370) 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09e08e640 1 -- 172.21.15.81:0/1560719222 learned_addr learned my addr 172.21.15.81:0/1560719222 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09d88d640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa09815c030 0x7fa0980accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09d08c640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa09815c440 0x7fa0981f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09e08e640 1 -- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa09815c030 msgr2=0x7fa0980accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09e08e640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa09815c030 0x7fa0980accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:39.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09e08e640 1 -- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa09815c440 msgr2=0x7fa0981f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:39.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09e08e640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa09815c440 0x7fa0981f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:39.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.089+0000 7fa09e08e640 1 -- 172.21.15.81:0/1560719222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0980abc50 con 0x7fa09815d750 2024-06-29T15:55:39.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.090+0000 7fa09e08e640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa09815d750 0x7fa098159dc0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fa09000c760 tx=0x7fa09000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:39.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.090+0000 7fa0967fc640 1 -- 172.21.15.81:0/1560719222 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa090003b90 con 0x7fa09815d750 2024-06-29T15:55:39.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.090+0000 7fa0967fc640 1 -- 172.21.15.81:0/1560719222 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa090003d50 con 0x7fa09815d750 2024-06-29T15:55:39.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.090+0000 7fa09faf1640 1 -- 172.21.15.81:0/1560719222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0980abf30 con 0x7fa09815d750 2024-06-29T15:55:39.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.090+0000 7fa0967fc640 1 -- 172.21.15.81:0/1560719222 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09000aab0 con 0x7fa09815d750 2024-06-29T15:55:39.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.090+0000 7fa09faf1640 1 -- 172.21.15.81:0/1560719222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0980ac4f0 con 0x7fa09815d750 2024-06-29T15:55:39.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.091+0000 7fa09faf1640 1 -- 172.21.15.81:0/1560719222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa09806c650 con 0x7fa09815d750 2024-06-29T15:55:39.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.092+0000 7fa0967fc640 1 -- 172.21.15.81:0/1560719222 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fa09000ac50 con 0x7fa09815d750 2024-06-29T15:55:39.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.092+0000 7fa0967fc640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa0840763f0 0x7fa084078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:39.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.092+0000 7fa0967fc640 1 -- 172.21.15.81:0/1560719222 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5809+0+0 (secure 0 0 0) 0x7fa090098080 con 0x7fa09815d750 2024-06-29T15:55:39.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.092+0000 7fa09d88d640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa0840763f0 0x7fa084078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:39.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.093+0000 7fa09d88d640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa0840763f0 0x7fa084078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa088033000 tx=0x7fa088004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:39.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.096+0000 7fa0967fc640 1 -- 172.21.15.81:0/1560719222 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa09806c650 con 0x7fa09815d750 2024-06-29T15:55:39.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:39.320+0000 7fa09faf1640 1 -- 172.21.15.81:0/1560719222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa09806e2a0 con 0x7fa09815d750 2024-06-29T15:55:40.267 INFO:tasks.ceph.mds.d.smithi167.stderr:2024-06-29T15:55:40.267+0000 7fb26fa96640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:55:40.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.301+0000 7fa0967fc640 1 -- 172.21.15.81:0/1560719222 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v102) ==== 118+0+0 (secure 0 0 0) 0x7fa09806e2a0 con 0x7fa09815d750 2024-06-29T15:55:40.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.303+0000 7fa09faf1640 1 -- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa0840763f0 msgr2=0x7fa084078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:40.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.303+0000 7fa09faf1640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa0840763f0 0x7fa084078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa088033000 tx=0x7fa088004730 comp rx=0 tx=0).stop 2024-06-29T15:55:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.303+0000 7fa09faf1640 1 -- 172.21.15.81:0/1560719222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa09815d750 msgr2=0x7fa098159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.303+0000 7fa09faf1640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa09815d750 0x7fa098159dc0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7fa09000c760 tx=0x7fa09000cc20 comp rx=0 tx=0).stop 2024-06-29T15:55:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.304+0000 7fa09faf1640 1 -- 172.21.15.81:0/1560719222 shutdown_connections 2024-06-29T15:55:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.304+0000 7fa09faf1640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa0840763f0 0x7fa084078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.304+0000 7fa09faf1640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa09815d750 0x7fa098159dc0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.304+0000 7fa09faf1640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa09815c440 0x7fa0981f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.304+0000 7fa09faf1640 1 --2- 172.21.15.81:0/1560719222 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa09815c030 0x7fa0980accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.304+0000 7fa09faf1640 1 -- 172.21.15.81:0/1560719222 >> 172.21.15.81:0/1560719222 conn(0x7fa0980d0930 msgr2=0x7fa0980b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.304+0000 7fa09faf1640 1 -- 172.21.15.81:0/1560719222 shutdown_connections 2024-06-29T15:55:40.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.304+0000 7fa09faf1640 1 -- 172.21.15.81:0/1560719222 wait complete. 2024-06-29T15:55:40.319 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:55:40.320 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:55:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.441+0000 7fdff5081640 1 -- 172.21.15.81:0/1487690829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff00cea10 msgr2=0x7fdff00b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.441+0000 7fdff5081640 1 --2- 172.21.15.81:0/1487690829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff00cea10 0x7fdff00b47b0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fdff00b2c10 tx=0x7fdfd802a530 comp rx=0 tx=0).stop 2024-06-29T15:55:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.441+0000 7fdff5081640 1 -- 172.21.15.81:0/1487690829 shutdown_connections 2024-06-29T15:55:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.441+0000 7fdff5081640 1 --2- 172.21.15.81:0/1487690829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff00cea10 0x7fdff00b47b0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.441+0000 7fdff5081640 1 --2- 172.21.15.81:0/1487690829 >> v2:172.21.15.167:3300/0 conn(0x7fdff00b6d60 0x7fdff00ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.441+0000 7fdff5081640 1 -- 172.21.15.81:0/1487690829 >> 172.21.15.81:0/1487690829 conn(0x7fdff00d1960 msgr2=0x7fdff00d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.441+0000 7fdff5081640 1 -- 172.21.15.81:0/1487690829 shutdown_connections 2024-06-29T15:55:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.441+0000 7fdff5081640 1 -- 172.21.15.81:0/1487690829 wait complete. 2024-06-29T15:55:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.443+0000 7fdff5081640 1 Processor -- start 2024-06-29T15:55:40.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.443+0000 7fdff5081640 1 -- start start 2024-06-29T15:55:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.444+0000 7fdff5081640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdff00b4cf0 0x7fdff00ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.444+0000 7fdff5081640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff00b6d60 0x7fdff00c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:40.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.444+0000 7fdff5081640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff00cea10 0x7fdff00c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.444+0000 7fdff5081640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdff00cd540 con 0x7fdff00b6d60 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.444+0000 7fdff5081640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdff00d0a10 con 0x7fdff00b4cf0 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.444+0000 7fdff5081640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdff00cd6c0 con 0x7fdff00cea10 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff00b6d60 0x7fdff00c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff00b6d60 0x7fdff00c78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56238/0 (socket says 172.21.15.81:56238) 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe659c640 1 -- 172.21.15.81:0/2899686812 learned_addr learned my addr 172.21.15.81:0/2899686812 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfeed9d640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdff00b4cf0 0x7fdff00ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfef59e640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff00cea10 0x7fdff00c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe659c640 1 -- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff00cea10 msgr2=0x7fdff00c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe659c640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff00cea10 0x7fdff00c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe659c640 1 -- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdff00b4cf0 msgr2=0x7fdff00ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:40.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe659c640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdff00b4cf0 0x7fdff00ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:40.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe659c640 1 -- 172.21.15.81:0/2899686812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff00af080 con 0x7fdff00b6d60 2024-06-29T15:55:40.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe659c640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff00b6d60 0x7fdff00c78c0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fdfd802ab20 tx=0x7fdfd8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:40.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe7fff640 1 -- 172.21.15.81:0/2899686812 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8031d60 con 0x7fdff00b6d60 2024-06-29T15:55:40.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe7fff640 1 -- 172.21.15.81:0/2899686812 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdfd8033e10 con 0x7fdff00b6d60 2024-06-29T15:55:40.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdff5081640 1 -- 172.21.15.81:0/2899686812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff00af360 con 0x7fdff00b6d60 2024-06-29T15:55:40.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdfe7fff640 1 -- 172.21.15.81:0/2899686812 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8005260 con 0x7fdff00b6d60 2024-06-29T15:55:40.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.445+0000 7fdff5081640 1 -- 172.21.15.81:0/2899686812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff00af920 con 0x7fdff00b6d60 2024-06-29T15:55:40.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.446+0000 7fdff5081640 1 -- 172.21.15.81:0/2899686812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdff00ce4d0 con 0x7fdff00b6d60 2024-06-29T15:55:40.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.447+0000 7fdfe7fff640 1 -- 172.21.15.81:0/2899686812 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fdfd8005400 con 0x7fdff00b6d60 2024-06-29T15:55:40.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.448+0000 7fdfe7fff640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdfd00763f0 0x7fdfd0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:40.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.448+0000 7fdfe7fff640 1 -- 172.21.15.81:0/2899686812 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7fdfd80b8b30 con 0x7fdff00b6d60 2024-06-29T15:55:40.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.448+0000 7fdfeed9d640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdfd00763f0 0x7fdfd0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:40.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.449+0000 7fdfeed9d640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdfd00763f0 0x7fdfd0078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdff00c5220 tx=0x7fdfe000b2e0 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:40.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.452+0000 7fdfe7fff640 1 -- 172.21.15.81:0/2899686812 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdfd803f030 con 0x7fdff00b6d60 2024-06-29T15:55:40.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:40.669+0000 7fdff5081640 1 -- 172.21.15.81:0/2899686812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fdff00cdec0 con 0x7fdff00b6d60 2024-06-29T15:55:41.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.306+0000 7fdfe7fff640 1 -- 172.21.15.81:0/2899686812 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v103) ==== 122+0+0 (secure 0 0 0) 0x7fdfd8005d00 con 0x7fdff00b6d60 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 -- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdfd00763f0 msgr2=0x7fdfd0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdfd00763f0 0x7fdfd0078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdff00c5220 tx=0x7fdfe000b2e0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 -- 172.21.15.81:0/2899686812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff00b6d60 msgr2=0x7fdff00c78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff00b6d60 0x7fdff00c78c0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fdfd802ab20 tx=0x7fdfd8033c20 comp rx=0 tx=0).stop 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 -- 172.21.15.81:0/2899686812 shutdown_connections 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdfd00763f0 0x7fdfd0078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff00cea10 0x7fdff00c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff00b6d60 0x7fdff00c78c0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 --2- 172.21.15.81:0/2899686812 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdff00b4cf0 0x7fdff00ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 -- 172.21.15.81:0/2899686812 >> 172.21.15.81:0/2899686812 conn(0x7fdff00d1960 msgr2=0x7fdff0068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 -- 172.21.15.81:0/2899686812 shutdown_connections 2024-06-29T15:55:41.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.309+0000 7fdff5081640 1 -- 172.21.15.81:0/2899686812 wait complete. 2024-06-29T15:55:41.325 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:55:41.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:41.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.441+0000 7f027bb7a640 1 -- 172.21.15.81:0/1573263410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02740b57b0 msgr2=0x7f027415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.441+0000 7f027bb7a640 1 --2- 172.21.15.81:0/1573263410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02740b57b0 0x7f027415aed0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f0270009830 tx=0x7f027002a5c0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.441+0000 7f027bb7a640 1 -- 172.21.15.81:0/1573263410 shutdown_connections 2024-06-29T15:55:41.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.441+0000 7f027bb7a640 1 --2- 172.21.15.81:0/1573263410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02740b57b0 0x7f027415aed0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.441+0000 7f027bb7a640 1 -- 172.21.15.81:0/1573263410 >> 172.21.15.81:0/1573263410 conn(0x7f02740d0930 msgr2=0x7f02740d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:41.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.441+0000 7f027bb7a640 1 -- 172.21.15.81:0/1573263410 shutdown_connections 2024-06-29T15:55:41.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.441+0000 7f027bb7a640 1 -- 172.21.15.81:0/1573263410 wait complete. 2024-06-29T15:55:41.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.443+0000 7f027bb7a640 1 Processor -- start 2024-06-29T15:55:41.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.443+0000 7f027bb7a640 1 -- start start 2024-06-29T15:55:41.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.444+0000 7f027bb7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f027406ef90 0x7f0274154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.444+0000 7f027bb7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02740b3960 0x7f02741e2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.444+0000 7f027bb7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02740b57b0 0x7f02741e4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.444+0000 7f027bb7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f02740b4600 con 0x7f027406ef90 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.444+0000 7f027bb7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f02740b4480 con 0x7f02740b3960 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.444+0000 7f027bb7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f02740b4780 con 0x7f02740b57b0 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.444+0000 7f0279916640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f027406ef90 0x7f0274154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.444+0000 7f0279916640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f027406ef90 0x7f0274154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56246/0 (socket says 172.21.15.81:56246) 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.444+0000 7f0279916640 1 -- 172.21.15.81:0/2265812884 learned_addr learned my addr 172.21.15.81:0/2265812884 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f027a117640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02740b57b0 0x7f02741e4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f0279115640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02740b3960 0x7f02741e2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f0279916640 1 -- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02740b57b0 msgr2=0x7f02741e4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f0279916640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02740b57b0 0x7f02741e4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f0279916640 1 -- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02740b3960 msgr2=0x7f02741e2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f0279916640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02740b3960 0x7f02741e2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f0279916640 1 -- 172.21.15.81:0/2265812884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02741f1be0 con 0x7f027406ef90 2024-06-29T15:55:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f0279115640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02740b3960 0x7f02741e2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f0279916640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f027406ef90 0x7f0274154470 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f027002abb0 tx=0x7f0270033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f026a7fc640 1 -- 172.21.15.81:0/2265812884 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270031b60 con 0x7f027406ef90 2024-06-29T15:55:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f026a7fc640 1 -- 172.21.15.81:0/2265812884 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0270031df0 con 0x7f027406ef90 2024-06-29T15:55:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f026a7fc640 1 -- 172.21.15.81:0/2265812884 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270005280 con 0x7f027406ef90 2024-06-29T15:55:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f027bb7a640 1 -- 172.21.15.81:0/2265812884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02741e5e70 con 0x7f027406ef90 2024-06-29T15:55:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.445+0000 7f027bb7a640 1 -- 172.21.15.81:0/2265812884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02741e6430 con 0x7f027406ef90 2024-06-29T15:55:41.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.446+0000 7f027bb7a640 1 -- 172.21.15.81:0/2265812884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02741549b0 con 0x7f027406ef90 2024-06-29T15:55:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.447+0000 7f026a7fc640 1 -- 172.21.15.81:0/2265812884 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f0270005420 con 0x7f027406ef90 2024-06-29T15:55:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.447+0000 7f026a7fc640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f02580763f0 0x7f0258078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.447+0000 7f026a7fc640 1 -- 172.21.15.81:0/2265812884 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f02700b8b90 con 0x7f027406ef90 2024-06-29T15:55:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.447+0000 7f0279115640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f02580763f0 0x7f0258078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.448+0000 7f0279115640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f02580763f0 0x7f0258078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0274154d20 tx=0x7f0264009040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:41.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.452+0000 7f026a7fc640 1 -- 172.21.15.81:0/2265812884 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f027003f030 con 0x7f027406ef90 2024-06-29T15:55:41.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.668+0000 7f027bb7a640 1 -- 172.21.15.81:0/2265812884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f027406e2a0 con 0x7f027406ef90 2024-06-29T15:55:41.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.668+0000 7f026a7fc640 1 -- 172.21.15.81:0/2265812884 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5240 (secure 0 0 0) 0x7f027406e2a0 con 0x7f027406ef90 2024-06-29T15:55:41.670 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:41.670 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":103,"btime":"2024-06-29T15:55:41:302970+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1764785504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1764785504},{"type":"v1","addr":"172.21.15.167:6836","nonce":1764785504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99},{"gid":10713,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/1699508727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1699508727},{"type":"v1","addr":"172.21.15.81:6834","nonce":1699508727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/265635924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":265635924},{"type":"v1","addr":"172.21.15.81:6835","nonce":265635924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.649055+0000","modified":"2024-06-29T15:55:40.671516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/1700137710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1700137710},{"type":"v1","addr":"172.21.15.167:6837","nonce":1700137710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10731,"qdb_cluster":[10731]},"id":12}]} 2024-06-29T15:55:41.670 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 103 2024-06-29T15:55:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.670+0000 7f027bb7a640 1 -- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f02580763f0 msgr2=0x7f0258078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.670+0000 7f027bb7a640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f02580763f0 0x7f0258078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0274154d20 tx=0x7f0264009040 comp rx=0 tx=0).stop 2024-06-29T15:55:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.670+0000 7f027bb7a640 1 -- 172.21.15.81:0/2265812884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f027406ef90 msgr2=0x7f0274154470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.670+0000 7f027bb7a640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f027406ef90 0x7f0274154470 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f027002abb0 tx=0x7f0270033c10 comp rx=0 tx=0).stop 2024-06-29T15:55:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.670+0000 7f027bb7a640 1 -- 172.21.15.81:0/2265812884 shutdown_connections 2024-06-29T15:55:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.670+0000 7f027bb7a640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f02580763f0 0x7f0258078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.670+0000 7f027bb7a640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02740b57b0 0x7f02741e4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.670+0000 7f027bb7a640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02740b3960 0x7f02741e2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.670+0000 7f027bb7a640 1 --2- 172.21.15.81:0/2265812884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f027406ef90 0x7f0274154470 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.671+0000 7f027bb7a640 1 -- 172.21.15.81:0/2265812884 >> 172.21.15.81:0/2265812884 conn(0x7f02740d0930 msgr2=0x7f0274155670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.671+0000 7f027bb7a640 1 -- 172.21.15.81:0/2265812884 shutdown_connections 2024-06-29T15:55:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.671+0000 7f027bb7a640 1 -- 172.21.15.81:0/2265812884 wait complete. 2024-06-29T15:55:41.684 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:55:41.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.795+0000 7f4f9faba640 1 -- 172.21.15.81:0/200500090 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f980b9040 msgr2=0x7f4f980d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.795+0000 7f4f9faba640 1 --2- 172.21.15.81:0/200500090 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f980b9040 0x7f4f980d0480 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009830 tx=0x7f4f9402a5a0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.795+0000 7f4f9faba640 1 -- 172.21.15.81:0/200500090 shutdown_connections 2024-06-29T15:55:41.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.795+0000 7f4f9faba640 1 --2- 172.21.15.81:0/200500090 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f980b5250 0x7f4f980d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.795+0000 7f4f9faba640 1 --2- 172.21.15.81:0/200500090 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f980b9040 0x7f4f980d0480 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.795+0000 7f4f9faba640 1 -- 172.21.15.81:0/200500090 >> 172.21.15.81:0/200500090 conn(0x7f4f9809cf30 msgr2=0x7f4f980d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:41.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.795+0000 7f4f9faba640 1 -- 172.21.15.81:0/200500090 shutdown_connections 2024-06-29T15:55:41.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.795+0000 7f4f9faba640 1 -- 172.21.15.81:0/200500090 wait complete. 2024-06-29T15:55:41.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.797+0000 7f4f9faba640 1 Processor -- start 2024-06-29T15:55:41.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.797+0000 7f4f9faba640 1 -- start start 2024-06-29T15:55:41.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.798+0000 7f4f9faba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f980b5250 0x7f4f980a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.798+0000 7f4f9faba640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f980b7290 0x7f4f981f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.798+0000 7f4f9faba640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f980b9040 0x7f4f981f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.798+0000 7f4f9faba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4f980b7db0 con 0x7f4f980b5250 2024-06-29T15:55:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.798+0000 7f4f9faba640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4f980d2d50 con 0x7f4f980b9040 2024-06-29T15:55:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.798+0000 7f4f9faba640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4f980b7f30 con 0x7f4f980b7290 2024-06-29T15:55:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.798+0000 7f4f9d856640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f980b5250 0x7f4f980a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.798+0000 7f4f9d856640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f980b5250 0x7f4f980a11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56260/0 (socket says 172.21.15.81:56260) 2024-06-29T15:55:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.798+0000 7f4f9d856640 1 -- 172.21.15.81:0/653125499 learned_addr learned my addr 172.21.15.81:0/653125499 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f9d856640 1 -- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f980b7290 msgr2=0x7f4f981f7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f9d856640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f980b7290 0x7f4f981f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f9d856640 1 -- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f980b9040 msgr2=0x7f4f981f7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f9d856640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f980b9040 0x7f4f981f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f9d856640 1 -- 172.21.15.81:0/653125499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f980b4e10 con 0x7f4f980b5250 2024-06-29T15:55:41.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f9d856640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f980b5250 0x7f4f980a11b0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f4f9402ab90 tx=0x7f4f94033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:41.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f8e7fc640 1 -- 172.21.15.81:0/653125499 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f94005c90 con 0x7f4f980b5250 2024-06-29T15:55:41.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f8e7fc640 1 -- 172.21.15.81:0/653125499 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f94031e50 con 0x7f4f980b5250 2024-06-29T15:55:41.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f9faba640 1 -- 172.21.15.81:0/653125499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f980b4b30 con 0x7f4f980b5250 2024-06-29T15:55:41.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f8e7fc640 1 -- 172.21.15.81:0/653125499 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f94005260 con 0x7f4f980b5250 2024-06-29T15:55:41.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.799+0000 7f4f9faba640 1 -- 172.21.15.81:0/653125499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f981f6c50 con 0x7f4f980b5250 2024-06-29T15:55:41.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.800+0000 7f4f9faba640 1 -- 172.21.15.81:0/653125499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f980b84c0 con 0x7f4f980b5250 2024-06-29T15:55:41.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.801+0000 7f4f8e7fc640 1 -- 172.21.15.81:0/653125499 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f4f94005400 con 0x7f4f980b5250 2024-06-29T15:55:41.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.801+0000 7f4f8e7fc640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f4f7c0763f0 0x7f4f7c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:41.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.802+0000 7f4f8e7fc640 1 -- 172.21.15.81:0/653125499 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f4f940b8a20 con 0x7f4f980b5250 2024-06-29T15:55:41.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.802+0000 7f4f9d055640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f4f7c0763f0 0x7f4f7c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:41.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.802+0000 7f4f9d055640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f4f7c0763f0 0x7f4f7c078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f880058c0 tx=0x7f4f88005850 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:41.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.806+0000 7f4f8e7fc640 1 -- 172.21.15.81:0/653125499 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f980b4e10 con 0x7f4f980b5250 2024-06-29T15:55:41.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.987+0000 7f4f9faba640 1 -- 172.21.15.81:0/653125499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f9815f4d0 con 0x7f4f980b5250 2024-06-29T15:55:41.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.988+0000 7f4f8e7fc640 1 -- 172.21.15.81:0/653125499 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) ==== 74+0+16826 (secure 0 0 0) 0x7f4f9403f030 con 0x7f4f980b5250 2024-06-29T15:55:41.990 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:41.990 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":196,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:55:39.289651+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.947285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.605625+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:55:41.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 -- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f4f7c0763f0 msgr2=0x7f4f7c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f4f7c0763f0 0x7f4f7c078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f880058c0 tx=0x7f4f88005850 comp rx=0 tx=0).stop 2024-06-29T15:55:41.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 -- 172.21.15.81:0/653125499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f980b5250 msgr2=0x7f4f980a11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:41.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f980b5250 0x7f4f980a11b0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f4f9402ab90 tx=0x7f4f94033c20 comp rx=0 tx=0).stop 2024-06-29T15:55:41.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 -- 172.21.15.81:0/653125499 shutdown_connections 2024-06-29T15:55:41.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f4f7c0763f0 0x7f4f7c078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f980b9040 0x7f4f981f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f980b7290 0x7f4f981f7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 --2- 172.21.15.81:0/653125499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f980b5250 0x7f4f980a11b0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 -- 172.21.15.81:0/653125499 >> 172.21.15.81:0/653125499 conn(0x7f4f9809cf30 msgr2=0x7f4f980d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 -- 172.21.15.81:0/653125499 shutdown_connections 2024-06-29T15:55:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:41.990+0000 7f4f9faba640 1 -- 172.21.15.81:0/653125499 wait complete. 2024-06-29T15:55:42.003 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:55:42.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:55:42.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.115+0000 7fbfbe32a640 1 -- 172.21.15.81:0/2071157997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb80cf4d0 msgr2=0x7fbfb8073030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.115+0000 7fbfbe32a640 1 --2- 172.21.15.81:0/2071157997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb80cf4d0 0x7fbfb8073030 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8070e50 tx=0x7fbfac02a560 comp rx=0 tx=0).stop 2024-06-29T15:55:42.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.115+0000 7fbfbe32a640 1 -- 172.21.15.81:0/2071157997 shutdown_connections 2024-06-29T15:55:42.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.115+0000 7fbfbe32a640 1 --2- 172.21.15.81:0/2071157997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb80cf4d0 0x7fbfb8073030 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.115+0000 7fbfbe32a640 1 --2- 172.21.15.81:0/2071157997 >> v2:172.21.15.167:3300/0 conn(0x7fbfb81412f0 0x7fbfb80cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.115+0000 7fbfbe32a640 1 -- 172.21.15.81:0/2071157997 >> 172.21.15.81:0/2071157997 conn(0x7fbfb806ccc0 msgr2=0x7fbfb80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:42.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.115+0000 7fbfbe32a640 1 -- 172.21.15.81:0/2071157997 shutdown_connections 2024-06-29T15:55:42.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.115+0000 7fbfbe32a640 1 -- 172.21.15.81:0/2071157997 wait complete. 2024-06-29T15:55:42.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.117+0000 7fbfbe32a640 1 Processor -- start 2024-06-29T15:55:42.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.117+0000 7fbfbe32a640 1 -- start start 2024-06-29T15:55:42.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.118+0000 7fbfbe32a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb8073570 0x7fbfb8140590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.118+0000 7fbfbe32a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb80cf4d0 0x7fbfb80ab600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.118+0000 7fbfbe32a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb81412f0 0x7fbfb80aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.118+0000 7fbfbe32a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb80b3210 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.118+0000 7fbfbe32a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb815f280 con 0x7fbfb81412f0 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.118+0000 7fbfbe32a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb80b3390 con 0x7fbfb8073570 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.118+0000 7fbfb77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb80cf4d0 0x7fbfb80ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.118+0000 7fbfb77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb80cf4d0 0x7fbfb80ab600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56276/0 (socket says 172.21.15.81:56276) 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.118+0000 7fbfb77fe640 1 -- 172.21.15.81:0/3966553793 learned_addr learned my addr 172.21.15.81:0/3966553793 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfb7fff640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb8073570 0x7fbfb8140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfbc8c7640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb81412f0 0x7fbfb80aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfb77fe640 1 -- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb8073570 msgr2=0x7fbfb8140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfb77fe640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb8073570 0x7fbfb8140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfb77fe640 1 -- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb81412f0 msgr2=0x7fbfb80aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfb77fe640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb81412f0 0x7fbfb80aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfb77fe640 1 -- 172.21.15.81:0/3966553793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb80a95f0 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfbc8c7640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb81412f0 0x7fbfb80aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:42.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfb77fe640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb80cf4d0 0x7fbfb80ab600 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fbfac034040 tx=0x7fbfac0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/3966553793 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac03e030 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/3966553793 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfac0047a0 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/3966553793 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac035040 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfbe32a640 1 -- 172.21.15.81:0/3966553793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb80a98d0 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.119+0000 7fbfbe32a640 1 -- 172.21.15.81:0/3966553793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb80adf60 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.120+0000 7fbfbe32a640 1 -- 172.21.15.81:0/3966553793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfb80b38b0 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.121+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/3966553793 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fbfac041030 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.121+0000 7fbfb4ff9640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fbfa40763f0 0x7fbfa4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.121+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/3966553793 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7fbfac0b8110 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.121+0000 7fbfb7fff640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fbfa40763f0 0x7fbfa4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.122+0000 7fbfb7fff640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fbfa40763f0 0x7fbfa4078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8005fd0 tx=0x7fbfa800a250 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.126+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/3966553793 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbfb8073980 con 0x7fbfb80cf4d0 2024-06-29T15:55:42.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.300+0000 7fbfbe32a640 1 -- 172.21.15.81:0/3966553793 --> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbfb80abf00 con 0x7fbfa40763f0 2024-06-29T15:55:42.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.306+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/3966553793 <== mgr.8327 v2:172.21.15.167:6832/803000291 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252478 (secure 0 0 0) 0x7fbfb80abf00 con 0x7fbfa40763f0 2024-06-29T15:55:42.310 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:42.312 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 -- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fbfa40763f0 msgr2=0x7fbfa4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fbfa40763f0 0x7fbfa4078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8005fd0 tx=0x7fbfa800a250 comp rx=0 tx=0).stop 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 -- 172.21.15.81:0/3966553793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb80cf4d0 msgr2=0x7fbfb80ab600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb80cf4d0 0x7fbfb80ab600 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fbfac034040 tx=0x7fbfac0039e0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 -- 172.21.15.81:0/3966553793 shutdown_connections 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fbfa40763f0 0x7fbfa4078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb81412f0 0x7fbfb80aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb80cf4d0 0x7fbfb80ab600 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 --2- 172.21.15.81:0/3966553793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb8073570 0x7fbfb8140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 -- 172.21.15.81:0/3966553793 >> 172.21.15.81:0/3966553793 conn(0x7fbfb806ccc0 msgr2=0x7fbfb8159e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:42.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 -- 172.21.15.81:0/3966553793 shutdown_connections 2024-06-29T15:55:42.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.309+0000 7fbfbe32a640 1 -- 172.21.15.81:0/3966553793 wait complete. 2024-06-29T15:55:42.322 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:55:41.928357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1206,"num_read_kb":3399,"num_write":374,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":244,"ondisk_log_size":244,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":419708,"kb_used_data":20352,"kb_used_omap":12,"kb_used_meta":399219,"kb_avail":754555012,"statfs":{"total":773094113280,"available":772664332288,"internally_reserved":0,"allocated":20840448,"data_stored":10729212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":408800818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.657611"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.311412+0000","last_change":"2024-06-29T15:55:38.309447+0000","last_active":"2024-06-29T15:55:38.311412+0000","last_peered":"2024-06-29T15:55:38.311412+0000","last_clean":"2024-06-29T15:55:38.311412+0000","last_became_active":"2024-06-29T15:55:38.308332+0000","last_became_peered":"2024-06-29T15:55:38.308332+0000","last_unstale":"2024-06-29T15:55:38.311412+0000","last_undegraded":"2024-06-29T15:55:38.311412+0000","last_fullsized":"2024-06-29T15:55:38.311412+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:27.768147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.299455+0000","last_change":"2024-06-29T15:55:38.294177+0000","last_active":"2024-06-29T15:55:38.299455+0000","last_peered":"2024-06-29T15:55:38.299455+0000","last_clean":"2024-06-29T15:55:38.299455+0000","last_became_active":"2024-06-29T15:55:38.293872+0000","last_became_peered":"2024-06-29T15:55:38.293872+0000","last_unstale":"2024-06-29T15:55:38.299455+0000","last_undegraded":"2024-06-29T15:55:38.299455+0000","last_fullsized":"2024-06-29T15:55:38.299455+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:51.981474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304763+0000","last_change":"2024-06-29T15:55:38.292289+0000","last_active":"2024-06-29T15:55:39.304763+0000","last_peered":"2024-06-29T15:55:39.304763+0000","last_clean":"2024-06-29T15:55:39.304763+0000","last_became_active":"2024-06-29T15:55:38.291932+0000","last_became_peered":"2024-06-29T15:55:38.291932+0000","last_unstale":"2024-06-29T15:55:39.304763+0000","last_undegraded":"2024-06-29T15:55:39.304763+0000","last_fullsized":"2024-06-29T15:55:39.304763+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:00.578618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.304465+0000","last_change":"2024-06-29T15:55:38.297822+0000","last_active":"2024-06-29T15:55:38.304465+0000","last_peered":"2024-06-29T15:55:38.304465+0000","last_clean":"2024-06-29T15:55:38.304465+0000","last_became_active":"2024-06-29T15:55:38.297513+0000","last_became_peered":"2024-06-29T15:55:38.297513+0000","last_unstale":"2024-06-29T15:55:38.304465+0000","last_undegraded":"2024-06-29T15:55:38.304465+0000","last_fullsized":"2024-06-29T15:55:38.304465+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:32.249157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388276+0000","last_change":"2024-06-29T15:55:38.299546+0000","last_active":"2024-06-29T15:55:39.388276+0000","last_peered":"2024-06-29T15:55:39.388276+0000","last_clean":"2024-06-29T15:55:39.388276+0000","last_became_active":"2024-06-29T15:55:38.299251+0000","last_became_peered":"2024-06-29T15:55:38.299251+0000","last_unstale":"2024-06-29T15:55:39.388276+0000","last_undegraded":"2024-06-29T15:55:39.388276+0000","last_fullsized":"2024-06-29T15:55:39.388276+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:01.270814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304532+0000","last_change":"2024-06-29T15:55:38.310596+0000","last_active":"2024-06-29T15:55:39.304532+0000","last_peered":"2024-06-29T15:55:39.304532+0000","last_clean":"2024-06-29T15:55:39.304532+0000","last_became_active":"2024-06-29T15:55:38.310280+0000","last_became_peered":"2024-06-29T15:55:38.310280+0000","last_unstale":"2024-06-29T15:55:39.304532+0000","last_undegraded":"2024-06-29T15:55:39.304532+0000","last_fullsized":"2024-06-29T15:55:39.304532+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:22.351537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.294798+0000","last_change":"2024-06-29T15:55:38.302191+0000","last_active":"2024-06-29T15:55:39.294798+0000","last_peered":"2024-06-29T15:55:39.294798+0000","last_clean":"2024-06-29T15:55:39.294798+0000","last_became_active":"2024-06-29T15:55:38.301867+0000","last_became_peered":"2024-06-29T15:55:38.301867+0000","last_unstale":"2024-06-29T15:55:39.294798+0000","last_undegraded":"2024-06-29T15:55:39.294798+0000","last_fullsized":"2024-06-29T15:55:39.294798+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:00.031732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293600+0000","last_change":"2024-06-29T15:55:38.304111+0000","last_active":"2024-06-29T15:55:39.293600+0000","last_peered":"2024-06-29T15:55:39.293600+0000","last_clean":"2024-06-29T15:55:39.293600+0000","last_became_active":"2024-06-29T15:55:38.303784+0000","last_became_peered":"2024-06-29T15:55:38.303784+0000","last_unstale":"2024-06-29T15:55:39.293600+0000","last_undegraded":"2024-06-29T15:55:39.293600+0000","last_fullsized":"2024-06-29T15:55:39.293600+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:39:42.302401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295997+0000","last_change":"2024-06-29T15:55:38.298956+0000","last_active":"2024-06-29T15:55:39.295997+0000","last_peered":"2024-06-29T15:55:39.295997+0000","last_clean":"2024-06-29T15:55:39.295997+0000","last_became_active":"2024-06-29T15:55:38.298579+0000","last_became_peered":"2024-06-29T15:55:38.298579+0000","last_unstale":"2024-06-29T15:55:39.295997+0000","last_undegraded":"2024-06-29T15:55:39.295997+0000","last_fullsized":"2024-06-29T15:55:39.295997+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:05.156497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302455+0000","last_change":"2024-06-29T15:55:38.305201+0000","last_active":"2024-06-29T15:55:39.302455+0000","last_peered":"2024-06-29T15:55:39.302455+0000","last_clean":"2024-06-29T15:55:39.302455+0000","last_became_active":"2024-06-29T15:55:38.304803+0000","last_became_peered":"2024-06-29T15:55:38.304803+0000","last_unstale":"2024-06-29T15:55:39.302455+0000","last_undegraded":"2024-06-29T15:55:39.302455+0000","last_fullsized":"2024-06-29T15:55:39.302455+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:01.270814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.294984+0000","last_change":"2024-06-29T15:55:38.311369+0000","last_active":"2024-06-29T15:55:39.294984+0000","last_peered":"2024-06-29T15:55:39.294984+0000","last_clean":"2024-06-29T15:55:39.294984+0000","last_became_active":"2024-06-29T15:55:38.311037+0000","last_became_peered":"2024-06-29T15:55:38.311037+0000","last_unstale":"2024-06-29T15:55:39.294984+0000","last_undegraded":"2024-06-29T15:55:39.294984+0000","last_fullsized":"2024-06-29T15:55:39.294984+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:26.976767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.316051+0000","last_change":"2024-06-29T15:55:38.312500+0000","last_active":"2024-06-29T15:55:38.316051+0000","last_peered":"2024-06-29T15:55:38.316051+0000","last_clean":"2024-06-29T15:55:38.316051+0000","last_became_active":"2024-06-29T15:55:38.312189+0000","last_became_peered":"2024-06-29T15:55:38.312189+0000","last_unstale":"2024-06-29T15:55:38.316051+0000","last_undegraded":"2024-06-29T15:55:38.316051+0000","last_fullsized":"2024-06-29T15:55:38.316051+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:40:58.792475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.307373+0000","last_change":"2024-06-29T15:55:38.301849+0000","last_active":"2024-06-29T15:55:38.307373+0000","last_peered":"2024-06-29T15:55:38.307373+0000","last_clean":"2024-06-29T15:55:38.307373+0000","last_became_active":"2024-06-29T15:55:38.301540+0000","last_became_peered":"2024-06-29T15:55:38.301540+0000","last_unstale":"2024-06-29T15:55:38.307373+0000","last_undegraded":"2024-06-29T15:55:38.307373+0000","last_fullsized":"2024-06-29T15:55:38.307373+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:18.576678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295361+0000","last_change":"2024-06-29T15:55:38.306411+0000","last_active":"2024-06-29T15:55:39.295361+0000","last_peered":"2024-06-29T15:55:39.295361+0000","last_clean":"2024-06-29T15:55:39.295361+0000","last_became_active":"2024-06-29T15:55:38.306106+0000","last_became_peered":"2024-06-29T15:55:38.306106+0000","last_unstale":"2024-06-29T15:55:39.295361+0000","last_undegraded":"2024-06-29T15:55:39.295361+0000","last_fullsized":"2024-06-29T15:55:39.295361+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:01.270814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.311358+0000","last_change":"2024-06-29T15:55:38.305309+0000","last_active":"2024-06-29T15:55:38.311358+0000","last_peered":"2024-06-29T15:55:38.311358+0000","last_clean":"2024-06-29T15:55:38.311358+0000","last_became_active":"2024-06-29T15:55:38.305018+0000","last_became_peered":"2024-06-29T15:55:38.305018+0000","last_unstale":"2024-06-29T15:55:38.311358+0000","last_undegraded":"2024-06-29T15:55:38.311358+0000","last_fullsized":"2024-06-29T15:55:38.311358+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:37:41.103307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293398+0000","last_change":"2024-06-29T15:55:38.297095+0000","last_active":"2024-06-29T15:55:39.293398+0000","last_peered":"2024-06-29T15:55:39.293398+0000","last_clean":"2024-06-29T15:55:39.293398+0000","last_became_active":"2024-06-29T15:55:38.296761+0000","last_became_peered":"2024-06-29T15:55:38.296761+0000","last_unstale":"2024-06-29T15:55:39.293398+0000","last_undegraded":"2024-06-29T15:55:39.293398+0000","last_fullsized":"2024-06-29T15:55:39.293398+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:14.098331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388481+0000","last_change":"2024-06-29T15:55:38.300952+0000","last_active":"2024-06-29T15:55:39.388481+0000","last_peered":"2024-06-29T15:55:39.388481+0000","last_clean":"2024-06-29T15:55:39.388481+0000","last_became_active":"2024-06-29T15:55:38.300195+0000","last_became_peered":"2024-06-29T15:55:38.300195+0000","last_unstale":"2024-06-29T15:55:39.388481+0000","last_undegraded":"2024-06-29T15:55:39.388481+0000","last_fullsized":"2024-06-29T15:55:39.388481+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:05.156497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.311176+0000","last_change":"2024-06-29T15:55:38.309099+0000","last_active":"2024-06-29T15:55:38.311176+0000","last_peered":"2024-06-29T15:55:38.311176+0000","last_clean":"2024-06-29T15:55:38.311176+0000","last_became_active":"2024-06-29T15:55:38.308742+0000","last_became_peered":"2024-06-29T15:55:38.308742+0000","last_unstale":"2024-06-29T15:55:38.311176+0000","last_undegraded":"2024-06-29T15:55:38.311176+0000","last_fullsized":"2024-06-29T15:55:38.311176+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:17:46.621428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302309+0000","last_change":"2024-06-29T15:55:38.305172+0000","last_active":"2024-06-29T15:55:39.302309+0000","last_peered":"2024-06-29T15:55:39.302309+0000","last_clean":"2024-06-29T15:55:39.302309+0000","last_became_active":"2024-06-29T15:55:38.304841+0000","last_became_peered":"2024-06-29T15:55:38.304841+0000","last_unstale":"2024-06-29T15:55:39.302309+0000","last_undegraded":"2024-06-29T15:55:39.302309+0000","last_fullsized":"2024-06-29T15:55:39.302309+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:34.409026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388351+0000","last_change":"2024-06-29T15:55:38.300941+0000","last_active":"2024-06-29T15:55:39.388351+0000","last_peered":"2024-06-29T15:55:39.388351+0000","last_clean":"2024-06-29T15:55:39.388351+0000","last_became_active":"2024-06-29T15:55:38.300620+0000","last_became_peered":"2024-06-29T15:55:38.300620+0000","last_unstale":"2024-06-29T15:55:39.388351+0000","last_undegraded":"2024-06-29T15:55:39.388351+0000","last_fullsized":"2024-06-29T15:55:39.388351+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:45.708017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388342+0000","last_change":"2024-06-29T15:55:38.298833+0000","last_active":"2024-06-29T15:55:39.388342+0000","last_peered":"2024-06-29T15:55:39.388342+0000","last_clean":"2024-06-29T15:55:39.388342+0000","last_became_active":"2024-06-29T15:55:38.298530+0000","last_became_peered":"2024-06-29T15:55:38.298530+0000","last_unstale":"2024-06-29T15:55:39.388342+0000","last_undegraded":"2024-06-29T15:55:39.388342+0000","last_fullsized":"2024-06-29T15:55:39.388342+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:22.432293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302344+0000","last_change":"2024-06-29T15:55:38.307415+0000","last_active":"2024-06-29T15:55:39.302344+0000","last_peered":"2024-06-29T15:55:39.302344+0000","last_clean":"2024-06-29T15:55:39.302344+0000","last_became_active":"2024-06-29T15:55:38.306851+0000","last_became_peered":"2024-06-29T15:55:38.306851+0000","last_unstale":"2024-06-29T15:55:39.302344+0000","last_undegraded":"2024-06-29T15:55:39.302344+0000","last_fullsized":"2024-06-29T15:55:39.302344+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:44.137473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304519+0000","last_change":"2024-06-29T15:55:38.300825+0000","last_active":"2024-06-29T15:55:39.304519+0000","last_peered":"2024-06-29T15:55:39.304519+0000","last_clean":"2024-06-29T15:55:39.304519+0000","last_became_active":"2024-06-29T15:55:38.300518+0000","last_became_peered":"2024-06-29T15:55:38.300518+0000","last_unstale":"2024-06-29T15:55:39.304519+0000","last_undegraded":"2024-06-29T15:55:39.304519+0000","last_fullsized":"2024-06-29T15:55:39.304519+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:45.708017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293558+0000","last_change":"2024-06-29T15:55:38.291520+0000","last_active":"2024-06-29T15:55:39.293558+0000","last_peered":"2024-06-29T15:55:39.293558+0000","last_clean":"2024-06-29T15:55:39.293558+0000","last_became_active":"2024-06-29T15:55:38.291204+0000","last_became_peered":"2024-06-29T15:55:38.291204+0000","last_unstale":"2024-06-29T15:55:39.293558+0000","last_undegraded":"2024-06-29T15:55:39.293558+0000","last_fullsized":"2024-06-29T15:55:39.293558+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:51.009195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.306442+0000","last_change":"2024-06-29T15:55:38.303486+0000","last_active":"2024-06-29T15:55:38.306442+0000","last_peered":"2024-06-29T15:55:38.306442+0000","last_clean":"2024-06-29T15:55:38.306442+0000","last_became_active":"2024-06-29T15:55:38.303092+0000","last_became_peered":"2024-06-29T15:55:38.303092+0000","last_unstale":"2024-06-29T15:55:38.306442+0000","last_undegraded":"2024-06-29T15:55:38.306442+0000","last_fullsized":"2024-06-29T15:55:38.306442+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:52.615803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295296+0000","last_change":"2024-06-29T15:55:38.301657+0000","last_active":"2024-06-29T15:55:39.295296+0000","last_peered":"2024-06-29T15:55:39.295296+0000","last_clean":"2024-06-29T15:55:39.295296+0000","last_became_active":"2024-06-29T15:55:38.301189+0000","last_became_peered":"2024-06-29T15:55:38.301189+0000","last_unstale":"2024-06-29T15:55:39.295296+0000","last_undegraded":"2024-06-29T15:55:39.295296+0000","last_fullsized":"2024-06-29T15:55:39.295296+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:39.593663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304771+0000","last_change":"2024-06-29T15:55:38.305981+0000","last_active":"2024-06-29T15:55:39.304771+0000","last_peered":"2024-06-29T15:55:39.304771+0000","last_clean":"2024-06-29T15:55:39.304771+0000","last_became_active":"2024-06-29T15:55:38.305676+0000","last_became_peered":"2024-06-29T15:55:38.305676+0000","last_unstale":"2024-06-29T15:55:39.304771+0000","last_undegraded":"2024-06-29T15:55:39.304771+0000","last_fullsized":"2024-06-29T15:55:39.304771+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:00.031732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388373+0000","last_change":"2024-06-29T15:55:38.304260+0000","last_active":"2024-06-29T15:55:39.388373+0000","last_peered":"2024-06-29T15:55:39.388373+0000","last_clean":"2024-06-29T15:55:39.388373+0000","last_became_active":"2024-06-29T15:55:38.303942+0000","last_became_peered":"2024-06-29T15:55:38.303942+0000","last_unstale":"2024-06-29T15:55:39.388373+0000","last_undegraded":"2024-06-29T15:55:39.388373+0000","last_fullsized":"2024-06-29T15:55:39.388373+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:23.140306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.310472+0000","last_change":"2024-06-29T15:55:38.307208+0000","last_active":"2024-06-29T15:55:38.310472+0000","last_peered":"2024-06-29T15:55:38.310472+0000","last_clean":"2024-06-29T15:55:38.310472+0000","last_became_active":"2024-06-29T15:55:38.306879+0000","last_became_peered":"2024-06-29T15:55:38.306879+0000","last_unstale":"2024-06-29T15:55:38.310472+0000","last_undegraded":"2024-06-29T15:55:38.310472+0000","last_fullsized":"2024-06-29T15:55:38.310472+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:38.899326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295837+0000","last_change":"2024-06-29T15:55:38.305518+0000","last_active":"2024-06-29T15:55:39.295837+0000","last_peered":"2024-06-29T15:55:39.295837+0000","last_clean":"2024-06-29T15:55:39.295837+0000","last_became_active":"2024-06-29T15:55:38.305181+0000","last_became_peered":"2024-06-29T15:55:38.305181+0000","last_unstale":"2024-06-29T15:55:39.295837+0000","last_undegraded":"2024-06-29T15:55:39.295837+0000","last_fullsized":"2024-06-29T15:55:39.295837+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:22.432293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304525+0000","last_change":"2024-06-29T15:55:38.298004+0000","last_active":"2024-06-29T15:55:39.304525+0000","last_peered":"2024-06-29T15:55:39.304525+0000","last_clean":"2024-06-29T15:55:39.304525+0000","last_became_active":"2024-06-29T15:55:38.297688+0000","last_became_peered":"2024-06-29T15:55:38.297688+0000","last_unstale":"2024-06-29T15:55:39.304525+0000","last_undegraded":"2024-06-29T15:55:39.304525+0000","last_fullsized":"2024-06-29T15:55:39.304525+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:23.140306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388114+0000","last_change":"2024-06-29T15:55:38.301945+0000","last_active":"2024-06-29T15:55:39.388114+0000","last_peered":"2024-06-29T15:55:39.388114+0000","last_clean":"2024-06-29T15:55:39.388114+0000","last_became_active":"2024-06-29T15:55:38.301664+0000","last_became_peered":"2024-06-29T15:55:38.301664+0000","last_unstale":"2024-06-29T15:55:39.388114+0000","last_undegraded":"2024-06-29T15:55:39.388114+0000","last_fullsized":"2024-06-29T15:55:39.388114+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:13.760465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293766+0000","last_change":"2024-06-29T15:55:38.302209+0000","last_active":"2024-06-29T15:55:39.293766+0000","last_peered":"2024-06-29T15:55:39.293766+0000","last_clean":"2024-06-29T15:55:39.293766+0000","last_became_active":"2024-06-29T15:55:38.301825+0000","last_became_peered":"2024-06-29T15:55:38.301825+0000","last_unstale":"2024-06-29T15:55:39.293766+0000","last_undegraded":"2024-06-29T15:55:39.293766+0000","last_fullsized":"2024-06-29T15:55:39.293766+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:24.659163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"195'4","reported_seq":18,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388499+0000","last_change":"2024-06-29T15:55:37.292070+0000","last_active":"2024-06-29T15:55:39.388499+0000","last_peered":"2024-06-29T15:55:39.388499+0000","last_clean":"2024-06-29T15:55:39.388499+0000","last_became_active":"2024-06-29T15:55:37.291781+0000","last_became_peered":"2024-06-29T15:55:37.291781+0000","last_unstale":"2024-06-29T15:55:39.388499+0000","last_undegraded":"2024-06-29T15:55:39.388499+0000","last_fullsized":"2024-06-29T15:55:39.388499+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:50.976402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302782+0000","last_change":"2024-06-29T15:55:38.300811+0000","last_active":"2024-06-29T15:55:39.302782+0000","last_peered":"2024-06-29T15:55:39.302782+0000","last_clean":"2024-06-29T15:55:39.302782+0000","last_became_active":"2024-06-29T15:55:38.300500+0000","last_became_peered":"2024-06-29T15:55:38.300500+0000","last_unstale":"2024-06-29T15:55:39.302782+0000","last_undegraded":"2024-06-29T15:55:39.302782+0000","last_fullsized":"2024-06-29T15:55:39.302782+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:22.351537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295785+0000","last_change":"2024-06-29T15:55:37.286770+0000","last_active":"2024-06-29T15:55:39.295785+0000","last_peered":"2024-06-29T15:55:39.295785+0000","last_clean":"2024-06-29T15:55:39.295785+0000","last_became_active":"2024-06-29T15:55:37.286422+0000","last_became_peered":"2024-06-29T15:55:37.286422+0000","last_unstale":"2024-06-29T15:55:39.295785+0000","last_undegraded":"2024-06-29T15:55:39.295785+0000","last_fullsized":"2024-06-29T15:55:39.295785+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:49:25.704312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295155+0000","last_change":"2024-06-29T15:55:38.302639+0000","last_active":"2024-06-29T15:55:39.295155+0000","last_peered":"2024-06-29T15:55:39.295155+0000","last_clean":"2024-06-29T15:55:39.295155+0000","last_became_active":"2024-06-29T15:55:38.302331+0000","last_became_peered":"2024-06-29T15:55:38.302331+0000","last_unstale":"2024-06-29T15:55:39.295155+0000","last_undegraded":"2024-06-29T15:55:39.295155+0000","last_fullsized":"2024-06-29T15:55:39.295155+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:11.235966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"195'2","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304863+0000","last_change":"2024-06-29T15:55:37.286517+0000","last_active":"2024-06-29T15:55:39.304863+0000","last_peered":"2024-06-29T15:55:39.304863+0000","last_clean":"2024-06-29T15:55:39.304863+0000","last_became_active":"2024-06-29T15:55:37.286230+0000","last_became_peered":"2024-06-29T15:55:37.286230+0000","last_unstale":"2024-06-29T15:55:39.304863+0000","last_undegraded":"2024-06-29T15:55:39.304863+0000","last_fullsized":"2024-06-29T15:55:39.304863+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:13:45.897605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295891+0000","last_change":"2024-06-29T15:55:38.301698+0000","last_active":"2024-06-29T15:55:39.295891+0000","last_peered":"2024-06-29T15:55:39.295891+0000","last_clean":"2024-06-29T15:55:39.295891+0000","last_became_active":"2024-06-29T15:55:38.301382+0000","last_became_peered":"2024-06-29T15:55:38.301382+0000","last_unstale":"2024-06-29T15:55:39.295891+0000","last_undegraded":"2024-06-29T15:55:39.295891+0000","last_fullsized":"2024-06-29T15:55:39.295891+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:22.351537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"195'2","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.316232+0000","last_change":"2024-06-29T15:55:37.291631+0000","last_active":"2024-06-29T15:55:38.316232+0000","last_peered":"2024-06-29T15:55:38.316232+0000","last_clean":"2024-06-29T15:55:38.316232+0000","last_became_active":"2024-06-29T15:55:37.291253+0000","last_became_peered":"2024-06-29T15:55:37.291253+0000","last_unstale":"2024-06-29T15:55:38.316232+0000","last_undegraded":"2024-06-29T15:55:38.316232+0000","last_fullsized":"2024-06-29T15:55:38.316232+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:28:41.667396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302841+0000","last_change":"2024-06-29T15:55:38.307143+0000","last_active":"2024-06-29T15:55:39.302841+0000","last_peered":"2024-06-29T15:55:39.302841+0000","last_clean":"2024-06-29T15:55:39.302841+0000","last_became_active":"2024-06-29T15:55:38.306457+0000","last_became_peered":"2024-06-29T15:55:38.306457+0000","last_unstale":"2024-06-29T15:55:39.302841+0000","last_undegraded":"2024-06-29T15:55:39.302841+0000","last_fullsized":"2024-06-29T15:55:39.302841+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:00.578618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388467+0000","last_change":"2024-06-29T15:55:37.287909+0000","last_active":"2024-06-29T15:55:39.388467+0000","last_peered":"2024-06-29T15:55:39.388467+0000","last_clean":"2024-06-29T15:55:39.388467+0000","last_became_active":"2024-06-29T15:55:37.287633+0000","last_became_peered":"2024-06-29T15:55:37.287633+0000","last_unstale":"2024-06-29T15:55:39.388467+0000","last_undegraded":"2024-06-29T15:55:39.388467+0000","last_fullsized":"2024-06-29T15:55:39.388467+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:17.571606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293702+0000","last_change":"2024-06-29T15:55:38.293041+0000","last_active":"2024-06-29T15:55:39.293702+0000","last_peered":"2024-06-29T15:55:39.293702+0000","last_clean":"2024-06-29T15:55:39.293702+0000","last_became_active":"2024-06-29T15:55:38.292149+0000","last_became_peered":"2024-06-29T15:55:38.292149+0000","last_unstale":"2024-06-29T15:55:39.293702+0000","last_undegraded":"2024-06-29T15:55:39.293702+0000","last_fullsized":"2024-06-29T15:55:39.293702+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:09.629606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293514+0000","last_change":"2024-06-29T15:55:37.287897+0000","last_active":"2024-06-29T15:55:39.293514+0000","last_peered":"2024-06-29T15:55:39.293514+0000","last_clean":"2024-06-29T15:55:39.293514+0000","last_became_active":"2024-06-29T15:55:37.287597+0000","last_became_peered":"2024-06-29T15:55:37.287597+0000","last_unstale":"2024-06-29T15:55:39.293514+0000","last_undegraded":"2024-06-29T15:55:39.293514+0000","last_fullsized":"2024-06-29T15:55:39.293514+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:14.661757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.294893+0000","last_change":"2024-06-29T15:55:38.311576+0000","last_active":"2024-06-29T15:55:39.294893+0000","last_peered":"2024-06-29T15:55:39.294893+0000","last_clean":"2024-06-29T15:55:39.294893+0000","last_became_active":"2024-06-29T15:55:38.311168+0000","last_became_peered":"2024-06-29T15:55:38.311168+0000","last_unstale":"2024-06-29T15:55:39.294893+0000","last_undegraded":"2024-06-29T15:55:39.294893+0000","last_fullsized":"2024-06-29T15:55:39.294893+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:16.113894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.295188+0000","last_change":"2024-06-29T15:55:37.293524+0000","last_active":"2024-06-29T15:55:38.295188+0000","last_peered":"2024-06-29T15:55:38.295188+0000","last_clean":"2024-06-29T15:55:38.295188+0000","last_became_active":"2024-06-29T15:55:37.292267+0000","last_became_peered":"2024-06-29T15:55:37.292267+0000","last_unstale":"2024-06-29T15:55:38.295188+0000","last_undegraded":"2024-06-29T15:55:38.295188+0000","last_fullsized":"2024-06-29T15:55:38.295188+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:36:41.989152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304805+0000","last_change":"2024-06-29T15:55:38.293301+0000","last_active":"2024-06-29T15:55:39.304805+0000","last_peered":"2024-06-29T15:55:39.304805+0000","last_clean":"2024-06-29T15:55:39.304805+0000","last_became_active":"2024-06-29T15:55:38.293013+0000","last_became_peered":"2024-06-29T15:55:38.293013+0000","last_unstale":"2024-06-29T15:55:39.304805+0000","last_undegraded":"2024-06-29T15:55:39.304805+0000","last_fullsized":"2024-06-29T15:55:39.304805+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:16.113894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.294959+0000","last_change":"2024-06-29T15:55:37.295164+0000","last_active":"2024-06-29T15:55:39.294959+0000","last_peered":"2024-06-29T15:55:39.294959+0000","last_clean":"2024-06-29T15:55:39.294959+0000","last_became_active":"2024-06-29T15:55:37.294705+0000","last_became_peered":"2024-06-29T15:55:37.294705+0000","last_unstale":"2024-06-29T15:55:39.294959+0000","last_undegraded":"2024-06-29T15:55:39.294959+0000","last_fullsized":"2024-06-29T15:55:39.294959+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:18.836280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293568+0000","last_change":"2024-06-29T15:55:38.306390+0000","last_active":"2024-06-29T15:55:39.293568+0000","last_peered":"2024-06-29T15:55:39.293568+0000","last_clean":"2024-06-29T15:55:39.293568+0000","last_became_active":"2024-06-29T15:55:38.306066+0000","last_became_peered":"2024-06-29T15:55:38.306066+0000","last_unstale":"2024-06-29T15:55:39.293568+0000","last_undegraded":"2024-06-29T15:55:39.293568+0000","last_fullsized":"2024-06-29T15:55:39.293568+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:52.705253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388328+0000","last_change":"2024-06-29T15:55:37.295140+0000","last_active":"2024-06-29T15:55:39.388328+0000","last_peered":"2024-06-29T15:55:39.388328+0000","last_clean":"2024-06-29T15:55:39.388328+0000","last_became_active":"2024-06-29T15:55:37.294791+0000","last_became_peered":"2024-06-29T15:55:37.294791+0000","last_unstale":"2024-06-29T15:55:39.388328+0000","last_undegraded":"2024-06-29T15:55:39.388328+0000","last_fullsized":"2024-06-29T15:55:39.388328+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:51.610731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295456+0000","last_change":"2024-06-29T15:55:38.303655+0000","last_active":"2024-06-29T15:55:39.295456+0000","last_peered":"2024-06-29T15:55:39.295456+0000","last_clean":"2024-06-29T15:55:39.295456+0000","last_became_active":"2024-06-29T15:55:38.303256+0000","last_became_peered":"2024-06-29T15:55:38.303256+0000","last_unstale":"2024-06-29T15:55:39.295456+0000","last_undegraded":"2024-06-29T15:55:39.295456+0000","last_fullsized":"2024-06-29T15:55:39.295456+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:20.146322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"195'3","reported_seq":17,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388485+0000","last_change":"2024-06-29T15:55:37.294378+0000","last_active":"2024-06-29T15:55:39.388485+0000","last_peered":"2024-06-29T15:55:39.388485+0000","last_clean":"2024-06-29T15:55:39.388485+0000","last_became_active":"2024-06-29T15:55:37.294065+0000","last_became_peered":"2024-06-29T15:55:37.294065+0000","last_unstale":"2024-06-29T15:55:39.388485+0000","last_undegraded":"2024-06-29T15:55:39.388485+0000","last_fullsized":"2024-06-29T15:55:39.388485+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:21:25.949676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302514+0000","last_change":"2024-06-29T15:55:38.305875+0000","last_active":"2024-06-29T15:55:39.302514+0000","last_peered":"2024-06-29T15:55:39.302514+0000","last_clean":"2024-06-29T15:55:39.302514+0000","last_became_active":"2024-06-29T15:55:38.305569+0000","last_became_peered":"2024-06-29T15:55:38.305569+0000","last_unstale":"2024-06-29T15:55:39.302514+0000","last_undegraded":"2024-06-29T15:55:39.302514+0000","last_fullsized":"2024-06-29T15:55:39.302514+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:45.708017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"195'6","reported_seq":20,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304690+0000","last_change":"2024-06-29T15:55:37.293162+0000","last_active":"2024-06-29T15:55:39.304690+0000","last_peered":"2024-06-29T15:55:39.304690+0000","last_clean":"2024-06-29T15:55:39.304690+0000","last_became_active":"2024-06-29T15:55:37.292787+0000","last_became_peered":"2024-06-29T15:55:37.292787+0000","last_unstale":"2024-06-29T15:55:39.304690+0000","last_undegraded":"2024-06-29T15:55:39.304690+0000","last_fullsized":"2024-06-29T15:55:39.304690+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:40:57.787403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304597+0000","last_change":"2024-06-29T15:55:38.307081+0000","last_active":"2024-06-29T15:55:39.304597+0000","last_peered":"2024-06-29T15:55:39.304597+0000","last_clean":"2024-06-29T15:55:39.304597+0000","last_became_active":"2024-06-29T15:55:38.306779+0000","last_became_peered":"2024-06-29T15:55:38.306779+0000","last_unstale":"2024-06-29T15:55:39.304597+0000","last_undegraded":"2024-06-29T15:55:39.304597+0000","last_fullsized":"2024-06-29T15:55:39.304597+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:05.156497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"195'2","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.313932+0000","last_change":"2024-06-29T15:55:37.290515+0000","last_active":"2024-06-29T15:55:38.313932+0000","last_peered":"2024-06-29T15:55:38.313932+0000","last_clean":"2024-06-29T15:55:38.313932+0000","last_became_active":"2024-06-29T15:55:37.290217+0000","last_became_peered":"2024-06-29T15:55:37.290217+0000","last_unstale":"2024-06-29T15:55:38.313932+0000","last_undegraded":"2024-06-29T15:55:38.313932+0000","last_fullsized":"2024-06-29T15:55:38.313932+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:56.177707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.294734+0000","last_change":"2024-06-29T15:55:38.306162+0000","last_active":"2024-06-29T15:55:39.294734+0000","last_peered":"2024-06-29T15:55:39.294734+0000","last_clean":"2024-06-29T15:55:39.294734+0000","last_became_active":"2024-06-29T15:55:38.305801+0000","last_became_peered":"2024-06-29T15:55:38.305801+0000","last_unstale":"2024-06-29T15:55:39.294734+0000","last_undegraded":"2024-06-29T15:55:39.294734+0000","last_fullsized":"2024-06-29T15:55:39.294734+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:00.578618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302564+0000","last_change":"2024-06-29T15:55:37.285367+0000","last_active":"2024-06-29T15:55:39.302564+0000","last_peered":"2024-06-29T15:55:39.302564+0000","last_clean":"2024-06-29T15:55:39.302564+0000","last_became_active":"2024-06-29T15:55:37.285086+0000","last_became_peered":"2024-06-29T15:55:37.285086+0000","last_unstale":"2024-06-29T15:55:39.302564+0000","last_undegraded":"2024-06-29T15:55:39.302564+0000","last_fullsized":"2024-06-29T15:55:39.302564+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:21:25.949676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295663+0000","last_change":"2024-06-29T15:55:38.301104+0000","last_active":"2024-06-29T15:55:39.295663+0000","last_peered":"2024-06-29T15:55:39.295663+0000","last_clean":"2024-06-29T15:55:39.295663+0000","last_became_active":"2024-06-29T15:55:38.300625+0000","last_became_peered":"2024-06-29T15:55:38.300625+0000","last_unstale":"2024-06-29T15:55:39.295663+0000","last_undegraded":"2024-06-29T15:55:39.295663+0000","last_fullsized":"2024-06-29T15:55:39.295663+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:45.708017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293327+0000","last_change":"2024-06-29T15:55:37.288565+0000","last_active":"2024-06-29T15:55:39.293327+0000","last_peered":"2024-06-29T15:55:39.293327+0000","last_clean":"2024-06-29T15:55:39.293327+0000","last_became_active":"2024-06-29T15:55:37.288263+0000","last_became_peered":"2024-06-29T15:55:37.288263+0000","last_unstale":"2024-06-29T15:55:39.293327+0000","last_undegraded":"2024-06-29T15:55:39.293327+0000","last_fullsized":"2024-06-29T15:55:39.293327+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:17.890026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302300+0000","last_change":"2024-06-29T15:55:38.306970+0000","last_active":"2024-06-29T15:55:39.302300+0000","last_peered":"2024-06-29T15:55:39.302300+0000","last_clean":"2024-06-29T15:55:39.302300+0000","last_became_active":"2024-06-29T15:55:38.306585+0000","last_became_peered":"2024-06-29T15:55:38.306585+0000","last_unstale":"2024-06-29T15:55:39.302300+0000","last_undegraded":"2024-06-29T15:55:39.302300+0000","last_fullsized":"2024-06-29T15:55:39.302300+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:13.760465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"195'8","reported_seq":20,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.316803+0000","last_change":"2024-06-29T15:55:37.291650+0000","last_active":"2024-06-29T15:55:38.316803+0000","last_peered":"2024-06-29T15:55:38.316803+0000","last_clean":"2024-06-29T15:55:38.316803+0000","last_became_active":"2024-06-29T15:55:37.290970+0000","last_became_peered":"2024-06-29T15:55:37.290970+0000","last_unstale":"2024-06-29T15:55:38.316803+0000","last_undegraded":"2024-06-29T15:55:38.316803+0000","last_fullsized":"2024-06-29T15:55:38.316803+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:49.615716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302679+0000","last_change":"2024-06-29T15:55:38.307465+0000","last_active":"2024-06-29T15:55:39.302679+0000","last_peered":"2024-06-29T15:55:39.302679+0000","last_clean":"2024-06-29T15:55:39.302679+0000","last_became_active":"2024-06-29T15:55:38.307194+0000","last_became_peered":"2024-06-29T15:55:38.307194+0000","last_unstale":"2024-06-29T15:55:39.302679+0000","last_undegraded":"2024-06-29T15:55:39.302679+0000","last_fullsized":"2024-06-29T15:55:39.302679+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:23.140306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388542+0000","last_change":"2024-06-29T15:55:37.288147+0000","last_active":"2024-06-29T15:55:39.388542+0000","last_peered":"2024-06-29T15:55:39.388542+0000","last_clean":"2024-06-29T15:55:39.388542+0000","last_became_active":"2024-06-29T15:55:37.287740+0000","last_became_peered":"2024-06-29T15:55:37.287740+0000","last_unstale":"2024-06-29T15:55:39.388542+0000","last_undegraded":"2024-06-29T15:55:39.388542+0000","last_fullsized":"2024-06-29T15:55:39.388542+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:31.244085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.296104+0000","last_change":"2024-06-29T15:55:38.302325+0000","last_active":"2024-06-29T15:55:39.296104+0000","last_peered":"2024-06-29T15:55:39.296104+0000","last_clean":"2024-06-29T15:55:39.296104+0000","last_became_active":"2024-06-29T15:55:38.302011+0000","last_became_peered":"2024-06-29T15:55:38.302011+0000","last_unstale":"2024-06-29T15:55:39.296104+0000","last_undegraded":"2024-06-29T15:55:39.296104+0000","last_fullsized":"2024-06-29T15:55:39.296104+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:00.578618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"195'2","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293876+0000","last_change":"2024-06-29T15:55:37.291817+0000","last_active":"2024-06-29T15:55:39.293876+0000","last_peered":"2024-06-29T15:55:39.293876+0000","last_clean":"2024-06-29T15:55:39.293876+0000","last_became_active":"2024-06-29T15:55:37.291507+0000","last_became_peered":"2024-06-29T15:55:37.291507+0000","last_unstale":"2024-06-29T15:55:39.293876+0000","last_undegraded":"2024-06-29T15:55:39.293876+0000","last_fullsized":"2024-06-29T15:55:39.293876+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:19:07.597147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295735+0000","last_change":"2024-06-29T15:55:38.311691+0000","last_active":"2024-06-29T15:55:39.295735+0000","last_peered":"2024-06-29T15:55:39.295735+0000","last_clean":"2024-06-29T15:55:39.295735+0000","last_became_active":"2024-06-29T15:55:38.311393+0000","last_became_peered":"2024-06-29T15:55:38.311393+0000","last_unstale":"2024-06-29T15:55:39.295735+0000","last_undegraded":"2024-06-29T15:55:39.295735+0000","last_fullsized":"2024-06-29T15:55:39.295735+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:23.140306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"195'1","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.314119+0000","last_change":"2024-06-29T15:55:37.288592+0000","last_active":"2024-06-29T15:55:38.314119+0000","last_peered":"2024-06-29T15:55:38.314119+0000","last_clean":"2024-06-29T15:55:38.314119+0000","last_became_active":"2024-06-29T15:55:37.288278+0000","last_became_peered":"2024-06-29T15:55:37.288278+0000","last_unstale":"2024-06-29T15:55:38.314119+0000","last_undegraded":"2024-06-29T15:55:38.314119+0000","last_fullsized":"2024-06-29T15:55:38.314119+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:39.184728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.299327+0000","last_change":"2024-06-29T15:55:38.293400+0000","last_active":"2024-06-29T15:55:38.299327+0000","last_peered":"2024-06-29T15:55:38.299327+0000","last_clean":"2024-06-29T15:55:38.299327+0000","last_became_active":"2024-06-29T15:55:38.293089+0000","last_became_peered":"2024-06-29T15:55:38.293089+0000","last_unstale":"2024-06-29T15:55:38.299327+0000","last_undegraded":"2024-06-29T15:55:38.299327+0000","last_fullsized":"2024-06-29T15:55:38.299327+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:45:49.886541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295354+0000","last_change":"2024-06-29T15:55:37.288444+0000","last_active":"2024-06-29T15:55:39.295354+0000","last_peered":"2024-06-29T15:55:39.295354+0000","last_clean":"2024-06-29T15:55:39.295354+0000","last_became_active":"2024-06-29T15:55:37.287047+0000","last_became_peered":"2024-06-29T15:55:37.287047+0000","last_unstale":"2024-06-29T15:55:39.295354+0000","last_undegraded":"2024-06-29T15:55:39.295354+0000","last_fullsized":"2024-06-29T15:55:39.295354+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:12.755393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302504+0000","last_change":"2024-06-29T15:55:38.304546+0000","last_active":"2024-06-29T15:55:39.302504+0000","last_peered":"2024-06-29T15:55:39.302504+0000","last_clean":"2024-06-29T15:55:39.302504+0000","last_became_active":"2024-06-29T15:55:38.304181+0000","last_became_peered":"2024-06-29T15:55:38.304181+0000","last_unstale":"2024-06-29T15:55:39.302504+0000","last_undegraded":"2024-06-29T15:55:39.302504+0000","last_fullsized":"2024-06-29T15:55:39.302504+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:22.432293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"195'2","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304672+0000","last_change":"2024-06-29T15:55:37.284720+0000","last_active":"2024-06-29T15:55:39.304672+0000","last_peered":"2024-06-29T15:55:39.304672+0000","last_clean":"2024-06-29T15:55:39.304672+0000","last_became_active":"2024-06-29T15:55:37.284415+0000","last_became_peered":"2024-06-29T15:55:37.284415+0000","last_unstale":"2024-06-29T15:55:39.304672+0000","last_undegraded":"2024-06-29T15:55:39.304672+0000","last_fullsized":"2024-06-29T15:55:39.304672+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:37.894254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295037+0000","last_change":"2024-06-29T15:55:38.301563+0000","last_active":"2024-06-29T15:55:39.295037+0000","last_peered":"2024-06-29T15:55:39.295037+0000","last_clean":"2024-06-29T15:55:39.295037+0000","last_became_active":"2024-06-29T15:55:38.301166+0000","last_became_peered":"2024-06-29T15:55:38.301166+0000","last_unstale":"2024-06-29T15:55:39.295037+0000","last_undegraded":"2024-06-29T15:55:39.295037+0000","last_fullsized":"2024-06-29T15:55:39.295037+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:05.434514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388230+0000","last_change":"2024-06-29T15:55:37.293839+0000","last_active":"2024-06-29T15:55:39.388230+0000","last_peered":"2024-06-29T15:55:39.388230+0000","last_clean":"2024-06-29T15:55:39.388230+0000","last_became_active":"2024-06-29T15:55:37.293489+0000","last_became_peered":"2024-06-29T15:55:37.293489+0000","last_unstale":"2024-06-29T15:55:39.388230+0000","last_undegraded":"2024-06-29T15:55:39.388230+0000","last_fullsized":"2024-06-29T15:55:39.388230+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:45:48.881469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.309937+0000","last_change":"2024-06-29T15:55:38.306080+0000","last_active":"2024-06-29T15:55:38.309937+0000","last_peered":"2024-06-29T15:55:38.309937+0000","last_clean":"2024-06-29T15:55:38.309937+0000","last_became_active":"2024-06-29T15:55:38.305768+0000","last_became_peered":"2024-06-29T15:55:38.305768+0000","last_unstale":"2024-06-29T15:55:38.309937+0000","last_undegraded":"2024-06-29T15:55:38.309937+0000","last_fullsized":"2024-06-29T15:55:38.309937+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:15:28.214095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304636+0000","last_change":"2024-06-29T15:55:37.288276+0000","last_active":"2024-06-29T15:55:39.304636+0000","last_peered":"2024-06-29T15:55:39.304636+0000","last_clean":"2024-06-29T15:55:39.304636+0000","last_became_active":"2024-06-29T15:55:37.287931+0000","last_became_peered":"2024-06-29T15:55:37.287931+0000","last_unstale":"2024-06-29T15:55:39.304636+0000","last_undegraded":"2024-06-29T15:55:39.304636+0000","last_fullsized":"2024-06-29T15:55:39.304636+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:26.095060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388120+0000","last_change":"2024-06-29T15:55:38.300711+0000","last_active":"2024-06-29T15:55:39.388120+0000","last_peered":"2024-06-29T15:55:39.388120+0000","last_clean":"2024-06-29T15:55:39.388120+0000","last_became_active":"2024-06-29T15:55:38.300266+0000","last_became_peered":"2024-06-29T15:55:38.300266+0000","last_unstale":"2024-06-29T15:55:39.388120+0000","last_undegraded":"2024-06-29T15:55:39.388120+0000","last_fullsized":"2024-06-29T15:55:39.388120+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:34.409026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295653+0000","last_change":"2024-06-29T15:55:37.289034+0000","last_active":"2024-06-29T15:55:39.295653+0000","last_peered":"2024-06-29T15:55:39.295653+0000","last_clean":"2024-06-29T15:55:39.295653+0000","last_became_active":"2024-06-29T15:55:37.288737+0000","last_became_peered":"2024-06-29T15:55:37.288737+0000","last_unstale":"2024-06-29T15:55:39.295653+0000","last_undegraded":"2024-06-29T15:55:39.295653+0000","last_fullsized":"2024-06-29T15:55:39.295653+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:26.095060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388470+0000","last_change":"2024-06-29T15:55:38.302229+0000","last_active":"2024-06-29T15:55:39.388470+0000","last_peered":"2024-06-29T15:55:39.388470+0000","last_clean":"2024-06-29T15:55:39.388470+0000","last_became_active":"2024-06-29T15:55:38.301322+0000","last_became_peered":"2024-06-29T15:55:38.301322+0000","last_unstale":"2024-06-29T15:55:39.388470+0000","last_undegraded":"2024-06-29T15:55:39.388470+0000","last_fullsized":"2024-06-29T15:55:39.388470+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:22.351537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"195'2","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.314892+0000","last_change":"2024-06-29T15:55:37.293359+0000","last_active":"2024-06-29T15:55:38.314892+0000","last_peered":"2024-06-29T15:55:38.314892+0000","last_clean":"2024-06-29T15:55:38.314892+0000","last_became_active":"2024-06-29T15:55:37.293034+0000","last_became_peered":"2024-06-29T15:55:37.293034+0000","last_unstale":"2024-06-29T15:55:38.314892+0000","last_undegraded":"2024-06-29T15:55:38.314892+0000","last_fullsized":"2024-06-29T15:55:38.314892+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:09.475576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302834+0000","last_change":"2024-06-29T15:55:38.312682+0000","last_active":"2024-06-29T15:55:39.302834+0000","last_peered":"2024-06-29T15:55:39.302834+0000","last_clean":"2024-06-29T15:55:39.302834+0000","last_became_active":"2024-06-29T15:55:38.312377+0000","last_became_peered":"2024-06-29T15:55:38.312377+0000","last_unstale":"2024-06-29T15:55:39.302834+0000","last_undegraded":"2024-06-29T15:55:39.302834+0000","last_fullsized":"2024-06-29T15:55:39.302834+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:05.156497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"196'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.303817+0000","last_change":"2024-06-29T15:55:37.283739+0000","last_active":"2024-06-29T15:55:39.303817+0000","last_peered":"2024-06-29T15:55:39.303817+0000","last_clean":"2024-06-29T15:55:39.303817+0000","last_became_active":"2024-06-29T15:55:37.282478+0000","last_became_peered":"2024-06-29T15:55:37.282478+0000","last_unstale":"2024-06-29T15:55:39.303817+0000","last_undegraded":"2024-06-29T15:55:39.303817+0000","last_fullsized":"2024-06-29T15:55:39.303817+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:50.976402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293623+0000","last_change":"2024-06-29T15:55:38.293246+0000","last_active":"2024-06-29T15:55:39.293623+0000","last_peered":"2024-06-29T15:55:39.293623+0000","last_clean":"2024-06-29T15:55:39.293623+0000","last_became_active":"2024-06-29T15:55:38.292921+0000","last_became_peered":"2024-06-29T15:55:38.292921+0000","last_unstale":"2024-06-29T15:55:39.293623+0000","last_undegraded":"2024-06-29T15:55:39.293623+0000","last_fullsized":"2024-06-29T15:55:39.293623+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:58:52.393383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302468+0000","last_change":"2024-06-29T15:55:37.282836+0000","last_active":"2024-06-29T15:55:39.302468+0000","last_peered":"2024-06-29T15:55:39.302468+0000","last_clean":"2024-06-29T15:55:39.302468+0000","last_became_active":"2024-06-29T15:55:37.282505+0000","last_became_peered":"2024-06-29T15:55:37.282505+0000","last_unstale":"2024-06-29T15:55:39.302468+0000","last_undegraded":"2024-06-29T15:55:39.302468+0000","last_fullsized":"2024-06-29T15:55:39.302468+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:31.244085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293897+0000","last_change":"2024-06-29T15:55:38.297535+0000","last_active":"2024-06-29T15:55:39.293897+0000","last_peered":"2024-06-29T15:55:39.293897+0000","last_clean":"2024-06-29T15:55:39.293897+0000","last_became_active":"2024-06-29T15:55:38.297224+0000","last_became_peered":"2024-06-29T15:55:38.297224+0000","last_unstale":"2024-06-29T15:55:39.293897+0000","last_undegraded":"2024-06-29T15:55:39.293897+0000","last_fullsized":"2024-06-29T15:55:39.293897+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:27:40.644641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293716+0000","last_change":"2024-06-29T15:55:37.290803+0000","last_active":"2024-06-29T15:55:39.293716+0000","last_peered":"2024-06-29T15:55:39.293716+0000","last_clean":"2024-06-29T15:55:39.293716+0000","last_became_active":"2024-06-29T15:55:37.290493+0000","last_became_peered":"2024-06-29T15:55:37.290493+0000","last_unstale":"2024-06-29T15:55:39.293716+0000","last_undegraded":"2024-06-29T15:55:39.293716+0000","last_fullsized":"2024-06-29T15:55:39.293716+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:11:59.493868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.311080+0000","last_change":"2024-06-29T15:55:38.309132+0000","last_active":"2024-06-29T15:55:38.311080+0000","last_peered":"2024-06-29T15:55:38.311080+0000","last_clean":"2024-06-29T15:55:38.311080+0000","last_became_active":"2024-06-29T15:55:38.308811+0000","last_became_peered":"2024-06-29T15:55:38.308811+0000","last_unstale":"2024-06-29T15:55:38.311080+0000","last_undegraded":"2024-06-29T15:55:38.311080+0000","last_fullsized":"2024-06-29T15:55:38.311080+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:26:06.331146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295991+0000","last_change":"2024-06-29T15:55:37.280018+0000","last_active":"2024-06-29T15:55:39.295991+0000","last_peered":"2024-06-29T15:55:39.295991+0000","last_clean":"2024-06-29T15:55:39.295991+0000","last_became_active":"2024-06-29T15:55:37.279627+0000","last_became_peered":"2024-06-29T15:55:37.279627+0000","last_unstale":"2024-06-29T15:55:39.295991+0000","last_undegraded":"2024-06-29T15:55:39.295991+0000","last_fullsized":"2024-06-29T15:55:39.295991+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:40:57.787403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.313996+0000","last_change":"2024-06-29T15:55:38.310978+0000","last_active":"2024-06-29T15:55:38.313996+0000","last_peered":"2024-06-29T15:55:38.313996+0000","last_clean":"2024-06-29T15:55:38.313996+0000","last_became_active":"2024-06-29T15:55:38.310670+0000","last_became_peered":"2024-06-29T15:55:38.310670+0000","last_unstale":"2024-06-29T15:55:38.313996+0000","last_undegraded":"2024-06-29T15:55:38.313996+0000","last_fullsized":"2024-06-29T15:55:38.313996+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:37:34.282870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295616+0000","last_change":"2024-06-29T15:55:37.292036+0000","last_active":"2024-06-29T15:55:39.295616+0000","last_peered":"2024-06-29T15:55:39.295616+0000","last_clean":"2024-06-29T15:55:39.295616+0000","last_became_active":"2024-06-29T15:55:37.291651+0000","last_became_peered":"2024-06-29T15:55:37.291651+0000","last_unstale":"2024-06-29T15:55:39.295616+0000","last_undegraded":"2024-06-29T15:55:39.295616+0000","last_fullsized":"2024-06-29T15:55:39.295616+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:37.894254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.306419+0000","last_change":"2024-06-29T15:55:38.303838+0000","last_active":"2024-06-29T15:55:38.306419+0000","last_peered":"2024-06-29T15:55:38.306419+0000","last_clean":"2024-06-29T15:55:38.306419+0000","last_became_active":"2024-06-29T15:55:38.303497+0000","last_became_peered":"2024-06-29T15:55:38.303497+0000","last_unstale":"2024-06-29T15:55:38.306419+0000","last_undegraded":"2024-06-29T15:55:38.306419+0000","last_fullsized":"2024-06-29T15:55:38.306419+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:21:26.954748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.289345+0000","last_change":"2024-06-29T15:55:37.287764+0000","last_active":"2024-06-29T15:55:38.289345+0000","last_peered":"2024-06-29T15:55:38.289345+0000","last_clean":"2024-06-29T15:55:38.289345+0000","last_became_active":"2024-06-29T15:55:37.287253+0000","last_became_peered":"2024-06-29T15:55:37.287253+0000","last_unstale":"2024-06-29T15:55:38.289345+0000","last_undegraded":"2024-06-29T15:55:38.289345+0000","last_fullsized":"2024-06-29T15:55:38.289345+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:31.386686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388152+0000","last_change":"2024-06-29T15:55:38.303162+0000","last_active":"2024-06-29T15:55:39.388152+0000","last_peered":"2024-06-29T15:55:39.388152+0000","last_clean":"2024-06-29T15:55:39.388152+0000","last_became_active":"2024-06-29T15:55:38.302861+0000","last_became_peered":"2024-06-29T15:55:38.302861+0000","last_unstale":"2024-06-29T15:55:39.388152+0000","last_undegraded":"2024-06-29T15:55:39.388152+0000","last_fullsized":"2024-06-29T15:55:39.388152+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:44.137473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304443+0000","last_change":"2024-06-29T15:55:37.284806+0000","last_active":"2024-06-29T15:55:39.304443+0000","last_peered":"2024-06-29T15:55:39.304443+0000","last_clean":"2024-06-29T15:55:39.304443+0000","last_became_active":"2024-06-29T15:55:37.281664+0000","last_became_peered":"2024-06-29T15:55:37.281664+0000","last_unstale":"2024-06-29T15:55:39.304443+0000","last_undegraded":"2024-06-29T15:55:39.304443+0000","last_fullsized":"2024-06-29T15:55:39.304443+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:49:25.704312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302188+0000","last_change":"2024-06-29T15:55:38.300752+0000","last_active":"2024-06-29T15:55:39.302188+0000","last_peered":"2024-06-29T15:55:39.302188+0000","last_clean":"2024-06-29T15:55:39.302188+0000","last_became_active":"2024-06-29T15:55:38.300397+0000","last_became_peered":"2024-06-29T15:55:38.300397+0000","last_unstale":"2024-06-29T15:55:39.302188+0000","last_undegraded":"2024-06-29T15:55:39.302188+0000","last_fullsized":"2024-06-29T15:55:39.302188+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:19.841352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.295603+0000","last_change":"2024-06-29T15:55:37.292429+0000","last_active":"2024-06-29T15:55:38.295603+0000","last_peered":"2024-06-29T15:55:38.295603+0000","last_clean":"2024-06-29T15:55:38.295603+0000","last_became_active":"2024-06-29T15:55:37.292107+0000","last_became_peered":"2024-06-29T15:55:37.292107+0000","last_unstale":"2024-06-29T15:55:38.295603+0000","last_undegraded":"2024-06-29T15:55:38.295603+0000","last_fullsized":"2024-06-29T15:55:38.295603+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:35.819045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"185'205","reported_seq":955,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.295044+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:55:38.295044+0000","last_peered":"2024-06-29T15:55:38.295044+0000","last_clean":"2024-06-29T15:55:38.295044+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:55:38.295044+0000","last_undegraded":"2024-06-29T15:55:38.295044+0000","last_fullsized":"2024-06-29T15:55:38.295044+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1206,"num_read_kb":3399,"num_write":329,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1206,"num_read_kb":3399,"num_write":329,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48044,"kb_used_data":2976,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323796,"statfs":{"total":96636764160,"available":96587567104,"internally_reserved":0,"allocated":3047424,"data_stored":1781829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41896,"kb_used_data":2388,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329944,"statfs":{"total":96636764160,"available":96593862656,"internally_reserved":0,"allocated":2445312,"data_stored":1191357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41896,"kb_used_data":2388,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329944,"statfs":{"total":96636764160,"available":96593862656,"internally_reserved":0,"allocated":2445312,"data_stored":1191357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575965,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59952,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311888,"statfs":{"total":96636764160,"available":96575373312,"internally_reserved":0,"allocated":2461696,"data_stored":1193551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575965,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64120,"kb_used_data":2412,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307720,"statfs":{"total":96636764160,"available":96571105280,"internally_reserved":0,"allocated":2469888,"data_stored":1197951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575965,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47476,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324364,"statfs":{"total":96636764160,"available":96588148736,"internally_reserved":0,"allocated":2457600,"data_stored":1192051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575964,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59960,"kb_used_data":2412,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311880,"statfs":{"total":96636764160,"available":96575365120,"internally_reserved":0,"allocated":2469888,"data_stored":1199369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575965,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56364,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315476,"statfs":{"total":96636764160,"available":96579047424,"internally_reserved":0,"allocated":3043328,"data_stored":1781747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:55:42.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:55:42.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.474+0000 7f19eca5e640 1 -- 172.21.15.81:0/1392955712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19e8073030 msgr2=0x7f19e8073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.474+0000 7f19eca5e640 1 --2- 172.21.15.81:0/1392955712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19e8073030 0x7f19e8073440 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009830 tx=0x7f19d002a5c0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.474+0000 7f19eca5e640 1 -- 172.21.15.81:0/1392955712 shutdown_connections 2024-06-29T15:55:42.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.474+0000 7f19eca5e640 1 --2- 172.21.15.81:0/1392955712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19e8073030 0x7f19e8073440 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.474+0000 7f19eca5e640 1 -- 172.21.15.81:0/1392955712 >> 172.21.15.81:0/1392955712 conn(0x7f19e806ccc0 msgr2=0x7f19e80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:42.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.474+0000 7f19eca5e640 1 -- 172.21.15.81:0/1392955712 shutdown_connections 2024-06-29T15:55:42.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.474+0000 7f19eca5e640 1 -- 172.21.15.81:0/1392955712 wait complete. 2024-06-29T15:55:42.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.476+0000 7f19eca5e640 1 Processor -- start 2024-06-29T15:55:42.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.476+0000 7f19eca5e640 1 -- start start 2024-06-29T15:55:42.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.477+0000 7f19eca5e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19e8073030 0x7f19e81f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.477+0000 7f19eca5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19e81412f0 0x7f19e8155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.477+0000 7f19eca5e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19e815e960 0x7f19e8154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.477+0000 7f19eca5e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f19e80b35a0 con 0x7f19e81412f0 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.477+0000 7f19eca5e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f19e80b3420 con 0x7f19e815e960 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.477+0000 7f19eca5e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f19e80b3720 con 0x7f19e8073030 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.477+0000 7f19e5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19e81412f0 0x7f19e8155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.477+0000 7f19e5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19e81412f0 0x7f19e8155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56296/0 (socket says 172.21.15.81:56296) 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.477+0000 7f19e5d9b640 1 -- 172.21.15.81:0/3281854642 learned_addr learned my addr 172.21.15.81:0/3281854642 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19e6d9d640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19e815e960 0x7f19e8154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19e659c640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19e8073030 0x7f19e81f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19e5d9b640 1 -- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19e8073030 msgr2=0x7f19e81f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19e5d9b640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19e8073030 0x7f19e81f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19e5d9b640 1 -- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19e815e960 msgr2=0x7f19e8154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19e5d9b640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19e815e960 0x7f19e8154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19e5d9b640 1 -- 172.21.15.81:0/3281854642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e81f6e90 con 0x7f19e81412f0 2024-06-29T15:55:42.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19e6d9d640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19e815e960 0x7f19e8154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:42.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19e659c640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19e8073030 0x7f19e81f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:42.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19e5d9b640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19e81412f0 0x7f19e8155870 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f19d002abb0 tx=0x7f19d0033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19deffd640 1 -- 172.21.15.81:0/3281854642 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0031d60 con 0x7f19e81412f0 2024-06-29T15:55:42.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19deffd640 1 -- 172.21.15.81:0/3281854642 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19d0005020 con 0x7f19e81412f0 2024-06-29T15:55:42.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19deffd640 1 -- 172.21.15.81:0/3281854642 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0005470 con 0x7f19e81412f0 2024-06-29T15:55:42.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19eca5e640 1 -- 172.21.15.81:0/3281854642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e81530e0 con 0x7f19e81412f0 2024-06-29T15:55:42.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.478+0000 7f19eca5e640 1 -- 172.21.15.81:0/3281854642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e81536a0 con 0x7f19e81412f0 2024-06-29T15:55:42.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.479+0000 7f19eca5e640 1 -- 172.21.15.81:0/3281854642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e8073440 con 0x7f19e81412f0 2024-06-29T15:55:42.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.480+0000 7f19deffd640 1 -- 172.21.15.81:0/3281854642 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f19d00047a0 con 0x7f19e81412f0 2024-06-29T15:55:42.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.480+0000 7f19deffd640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f19c80763f0 0x7f19c8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.480+0000 7f19deffd640 1 -- 172.21.15.81:0/3281854642 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f19d00b8d10 con 0x7f19e81412f0 2024-06-29T15:55:42.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.481+0000 7f19e659c640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f19c80763f0 0x7f19c8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.481+0000 7f19e659c640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f19c80763f0 0x7f19c8078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f19e0001da0 tx=0x7f19e0009040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.485+0000 7f19deffd640 1 -- 172.21.15.81:0/3281854642 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f19d003f030 con 0x7f19e81412f0 2024-06-29T15:55:42.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.661+0000 7f19eca5e640 1 -- 172.21.15.81:0/3281854642 --> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f19e815b400 con 0x7f19c80763f0 2024-06-29T15:55:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.667+0000 7f19deffd640 1 -- 172.21.15.81:0/3281854642 <== mgr.8327 v2:172.21.15.167:6832/803000291 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252478 (secure 0 0 0) 0x7f19e815b400 con 0x7f19c80763f0 2024-06-29T15:55:42.671 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:42.672 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:55:42.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.669+0000 7f19eca5e640 1 -- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f19c80763f0 msgr2=0x7f19c8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.669+0000 7f19eca5e640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f19c80763f0 0x7f19c8078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f19e0001da0 tx=0x7f19e0009040 comp rx=0 tx=0).stop 2024-06-29T15:55:42.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.669+0000 7f19eca5e640 1 -- 172.21.15.81:0/3281854642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19e81412f0 msgr2=0x7f19e8155870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.669+0000 7f19eca5e640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19e81412f0 0x7f19e8155870 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f19d002abb0 tx=0x7f19d0033ce0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.669+0000 7f19eca5e640 1 -- 172.21.15.81:0/3281854642 shutdown_connections 2024-06-29T15:55:42.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.669+0000 7f19eca5e640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f19c80763f0 0x7f19c8078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.669+0000 7f19eca5e640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f19e815e960 0x7f19e8154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.669+0000 7f19eca5e640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19e81412f0 0x7f19e8155870 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.669+0000 7f19eca5e640 1 --2- 172.21.15.81:0/3281854642 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f19e8073030 0x7f19e81f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.669+0000 7f19eca5e640 1 -- 172.21.15.81:0/3281854642 >> 172.21.15.81:0/3281854642 conn(0x7f19e806ccc0 msgr2=0x7f19e815ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:42.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.670+0000 7f19eca5e640 1 -- 172.21.15.81:0/3281854642 shutdown_connections 2024-06-29T15:55:42.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.670+0000 7f19eca5e640 1 -- 172.21.15.81:0/3281854642 wait complete. 2024-06-29T15:55:42.682 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:55:41.928357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1206,"num_read_kb":3399,"num_write":374,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":244,"ondisk_log_size":244,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":419708,"kb_used_data":20352,"kb_used_omap":12,"kb_used_meta":399219,"kb_avail":754555012,"statfs":{"total":773094113280,"available":772664332288,"internally_reserved":0,"allocated":20840448,"data_stored":10729212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":408800818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.657611"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.311412+0000","last_change":"2024-06-29T15:55:38.309447+0000","last_active":"2024-06-29T15:55:38.311412+0000","last_peered":"2024-06-29T15:55:38.311412+0000","last_clean":"2024-06-29T15:55:38.311412+0000","last_became_active":"2024-06-29T15:55:38.308332+0000","last_became_peered":"2024-06-29T15:55:38.308332+0000","last_unstale":"2024-06-29T15:55:38.311412+0000","last_undegraded":"2024-06-29T15:55:38.311412+0000","last_fullsized":"2024-06-29T15:55:38.311412+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:25:27.768147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.299455+0000","last_change":"2024-06-29T15:55:38.294177+0000","last_active":"2024-06-29T15:55:38.299455+0000","last_peered":"2024-06-29T15:55:38.299455+0000","last_clean":"2024-06-29T15:55:38.299455+0000","last_became_active":"2024-06-29T15:55:38.293872+0000","last_became_peered":"2024-06-29T15:55:38.293872+0000","last_unstale":"2024-06-29T15:55:38.299455+0000","last_undegraded":"2024-06-29T15:55:38.299455+0000","last_fullsized":"2024-06-29T15:55:38.299455+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:51.981474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304763+0000","last_change":"2024-06-29T15:55:38.292289+0000","last_active":"2024-06-29T15:55:39.304763+0000","last_peered":"2024-06-29T15:55:39.304763+0000","last_clean":"2024-06-29T15:55:39.304763+0000","last_became_active":"2024-06-29T15:55:38.291932+0000","last_became_peered":"2024-06-29T15:55:38.291932+0000","last_unstale":"2024-06-29T15:55:39.304763+0000","last_undegraded":"2024-06-29T15:55:39.304763+0000","last_fullsized":"2024-06-29T15:55:39.304763+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:00.578618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.304465+0000","last_change":"2024-06-29T15:55:38.297822+0000","last_active":"2024-06-29T15:55:38.304465+0000","last_peered":"2024-06-29T15:55:38.304465+0000","last_clean":"2024-06-29T15:55:38.304465+0000","last_became_active":"2024-06-29T15:55:38.297513+0000","last_became_peered":"2024-06-29T15:55:38.297513+0000","last_unstale":"2024-06-29T15:55:38.304465+0000","last_undegraded":"2024-06-29T15:55:38.304465+0000","last_fullsized":"2024-06-29T15:55:38.304465+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:32.249157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388276+0000","last_change":"2024-06-29T15:55:38.299546+0000","last_active":"2024-06-29T15:55:39.388276+0000","last_peered":"2024-06-29T15:55:39.388276+0000","last_clean":"2024-06-29T15:55:39.388276+0000","last_became_active":"2024-06-29T15:55:38.299251+0000","last_became_peered":"2024-06-29T15:55:38.299251+0000","last_unstale":"2024-06-29T15:55:39.388276+0000","last_undegraded":"2024-06-29T15:55:39.388276+0000","last_fullsized":"2024-06-29T15:55:39.388276+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:01.270814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304532+0000","last_change":"2024-06-29T15:55:38.310596+0000","last_active":"2024-06-29T15:55:39.304532+0000","last_peered":"2024-06-29T15:55:39.304532+0000","last_clean":"2024-06-29T15:55:39.304532+0000","last_became_active":"2024-06-29T15:55:38.310280+0000","last_became_peered":"2024-06-29T15:55:38.310280+0000","last_unstale":"2024-06-29T15:55:39.304532+0000","last_undegraded":"2024-06-29T15:55:39.304532+0000","last_fullsized":"2024-06-29T15:55:39.304532+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:22.351537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.294798+0000","last_change":"2024-06-29T15:55:38.302191+0000","last_active":"2024-06-29T15:55:39.294798+0000","last_peered":"2024-06-29T15:55:39.294798+0000","last_clean":"2024-06-29T15:55:39.294798+0000","last_became_active":"2024-06-29T15:55:38.301867+0000","last_became_peered":"2024-06-29T15:55:38.301867+0000","last_unstale":"2024-06-29T15:55:39.294798+0000","last_undegraded":"2024-06-29T15:55:39.294798+0000","last_fullsized":"2024-06-29T15:55:39.294798+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:00.031732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293600+0000","last_change":"2024-06-29T15:55:38.304111+0000","last_active":"2024-06-29T15:55:39.293600+0000","last_peered":"2024-06-29T15:55:39.293600+0000","last_clean":"2024-06-29T15:55:39.293600+0000","last_became_active":"2024-06-29T15:55:38.303784+0000","last_became_peered":"2024-06-29T15:55:38.303784+0000","last_unstale":"2024-06-29T15:55:39.293600+0000","last_undegraded":"2024-06-29T15:55:39.293600+0000","last_fullsized":"2024-06-29T15:55:39.293600+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:39:42.302401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295997+0000","last_change":"2024-06-29T15:55:38.298956+0000","last_active":"2024-06-29T15:55:39.295997+0000","last_peered":"2024-06-29T15:55:39.295997+0000","last_clean":"2024-06-29T15:55:39.295997+0000","last_became_active":"2024-06-29T15:55:38.298579+0000","last_became_peered":"2024-06-29T15:55:38.298579+0000","last_unstale":"2024-06-29T15:55:39.295997+0000","last_undegraded":"2024-06-29T15:55:39.295997+0000","last_fullsized":"2024-06-29T15:55:39.295997+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:05.156497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302455+0000","last_change":"2024-06-29T15:55:38.305201+0000","last_active":"2024-06-29T15:55:39.302455+0000","last_peered":"2024-06-29T15:55:39.302455+0000","last_clean":"2024-06-29T15:55:39.302455+0000","last_became_active":"2024-06-29T15:55:38.304803+0000","last_became_peered":"2024-06-29T15:55:38.304803+0000","last_unstale":"2024-06-29T15:55:39.302455+0000","last_undegraded":"2024-06-29T15:55:39.302455+0000","last_fullsized":"2024-06-29T15:55:39.302455+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:01.270814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.294984+0000","last_change":"2024-06-29T15:55:38.311369+0000","last_active":"2024-06-29T15:55:39.294984+0000","last_peered":"2024-06-29T15:55:39.294984+0000","last_clean":"2024-06-29T15:55:39.294984+0000","last_became_active":"2024-06-29T15:55:38.311037+0000","last_became_peered":"2024-06-29T15:55:38.311037+0000","last_unstale":"2024-06-29T15:55:39.294984+0000","last_undegraded":"2024-06-29T15:55:39.294984+0000","last_fullsized":"2024-06-29T15:55:39.294984+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:12:26.976767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.316051+0000","last_change":"2024-06-29T15:55:38.312500+0000","last_active":"2024-06-29T15:55:38.316051+0000","last_peered":"2024-06-29T15:55:38.316051+0000","last_clean":"2024-06-29T15:55:38.316051+0000","last_became_active":"2024-06-29T15:55:38.312189+0000","last_became_peered":"2024-06-29T15:55:38.312189+0000","last_unstale":"2024-06-29T15:55:38.316051+0000","last_undegraded":"2024-06-29T15:55:38.316051+0000","last_fullsized":"2024-06-29T15:55:38.316051+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:40:58.792475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.307373+0000","last_change":"2024-06-29T15:55:38.301849+0000","last_active":"2024-06-29T15:55:38.307373+0000","last_peered":"2024-06-29T15:55:38.307373+0000","last_clean":"2024-06-29T15:55:38.307373+0000","last_became_active":"2024-06-29T15:55:38.301540+0000","last_became_peered":"2024-06-29T15:55:38.301540+0000","last_unstale":"2024-06-29T15:55:38.307373+0000","last_undegraded":"2024-06-29T15:55:38.307373+0000","last_fullsized":"2024-06-29T15:55:38.307373+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:18.576678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295361+0000","last_change":"2024-06-29T15:55:38.306411+0000","last_active":"2024-06-29T15:55:39.295361+0000","last_peered":"2024-06-29T15:55:39.295361+0000","last_clean":"2024-06-29T15:55:39.295361+0000","last_became_active":"2024-06-29T15:55:38.306106+0000","last_became_peered":"2024-06-29T15:55:38.306106+0000","last_unstale":"2024-06-29T15:55:39.295361+0000","last_undegraded":"2024-06-29T15:55:39.295361+0000","last_fullsized":"2024-06-29T15:55:39.295361+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:40:01.270814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.311358+0000","last_change":"2024-06-29T15:55:38.305309+0000","last_active":"2024-06-29T15:55:38.311358+0000","last_peered":"2024-06-29T15:55:38.311358+0000","last_clean":"2024-06-29T15:55:38.311358+0000","last_became_active":"2024-06-29T15:55:38.305018+0000","last_became_peered":"2024-06-29T15:55:38.305018+0000","last_unstale":"2024-06-29T15:55:38.311358+0000","last_undegraded":"2024-06-29T15:55:38.311358+0000","last_fullsized":"2024-06-29T15:55:38.311358+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:37:41.103307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293398+0000","last_change":"2024-06-29T15:55:38.297095+0000","last_active":"2024-06-29T15:55:39.293398+0000","last_peered":"2024-06-29T15:55:39.293398+0000","last_clean":"2024-06-29T15:55:39.293398+0000","last_became_active":"2024-06-29T15:55:38.296761+0000","last_became_peered":"2024-06-29T15:55:38.296761+0000","last_unstale":"2024-06-29T15:55:39.293398+0000","last_undegraded":"2024-06-29T15:55:39.293398+0000","last_fullsized":"2024-06-29T15:55:39.293398+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:14.098331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388481+0000","last_change":"2024-06-29T15:55:38.300952+0000","last_active":"2024-06-29T15:55:39.388481+0000","last_peered":"2024-06-29T15:55:39.388481+0000","last_clean":"2024-06-29T15:55:39.388481+0000","last_became_active":"2024-06-29T15:55:38.300195+0000","last_became_peered":"2024-06-29T15:55:38.300195+0000","last_unstale":"2024-06-29T15:55:39.388481+0000","last_undegraded":"2024-06-29T15:55:39.388481+0000","last_fullsized":"2024-06-29T15:55:39.388481+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:05.156497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.311176+0000","last_change":"2024-06-29T15:55:38.309099+0000","last_active":"2024-06-29T15:55:38.311176+0000","last_peered":"2024-06-29T15:55:38.311176+0000","last_clean":"2024-06-29T15:55:38.311176+0000","last_became_active":"2024-06-29T15:55:38.308742+0000","last_became_peered":"2024-06-29T15:55:38.308742+0000","last_unstale":"2024-06-29T15:55:38.311176+0000","last_undegraded":"2024-06-29T15:55:38.311176+0000","last_fullsized":"2024-06-29T15:55:38.311176+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:17:46.621428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302309+0000","last_change":"2024-06-29T15:55:38.305172+0000","last_active":"2024-06-29T15:55:39.302309+0000","last_peered":"2024-06-29T15:55:39.302309+0000","last_clean":"2024-06-29T15:55:39.302309+0000","last_became_active":"2024-06-29T15:55:38.304841+0000","last_became_peered":"2024-06-29T15:55:38.304841+0000","last_unstale":"2024-06-29T15:55:39.302309+0000","last_undegraded":"2024-06-29T15:55:39.302309+0000","last_fullsized":"2024-06-29T15:55:39.302309+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:34.409026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388351+0000","last_change":"2024-06-29T15:55:38.300941+0000","last_active":"2024-06-29T15:55:39.388351+0000","last_peered":"2024-06-29T15:55:39.388351+0000","last_clean":"2024-06-29T15:55:39.388351+0000","last_became_active":"2024-06-29T15:55:38.300620+0000","last_became_peered":"2024-06-29T15:55:38.300620+0000","last_unstale":"2024-06-29T15:55:39.388351+0000","last_undegraded":"2024-06-29T15:55:39.388351+0000","last_fullsized":"2024-06-29T15:55:39.388351+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:45.708017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388342+0000","last_change":"2024-06-29T15:55:38.298833+0000","last_active":"2024-06-29T15:55:39.388342+0000","last_peered":"2024-06-29T15:55:39.388342+0000","last_clean":"2024-06-29T15:55:39.388342+0000","last_became_active":"2024-06-29T15:55:38.298530+0000","last_became_peered":"2024-06-29T15:55:38.298530+0000","last_unstale":"2024-06-29T15:55:39.388342+0000","last_undegraded":"2024-06-29T15:55:39.388342+0000","last_fullsized":"2024-06-29T15:55:39.388342+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:22.432293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302344+0000","last_change":"2024-06-29T15:55:38.307415+0000","last_active":"2024-06-29T15:55:39.302344+0000","last_peered":"2024-06-29T15:55:39.302344+0000","last_clean":"2024-06-29T15:55:39.302344+0000","last_became_active":"2024-06-29T15:55:38.306851+0000","last_became_peered":"2024-06-29T15:55:38.306851+0000","last_unstale":"2024-06-29T15:55:39.302344+0000","last_undegraded":"2024-06-29T15:55:39.302344+0000","last_fullsized":"2024-06-29T15:55:39.302344+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:44.137473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304519+0000","last_change":"2024-06-29T15:55:38.300825+0000","last_active":"2024-06-29T15:55:39.304519+0000","last_peered":"2024-06-29T15:55:39.304519+0000","last_clean":"2024-06-29T15:55:39.304519+0000","last_became_active":"2024-06-29T15:55:38.300518+0000","last_became_peered":"2024-06-29T15:55:38.300518+0000","last_unstale":"2024-06-29T15:55:39.304519+0000","last_undegraded":"2024-06-29T15:55:39.304519+0000","last_fullsized":"2024-06-29T15:55:39.304519+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:45.708017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293558+0000","last_change":"2024-06-29T15:55:38.291520+0000","last_active":"2024-06-29T15:55:39.293558+0000","last_peered":"2024-06-29T15:55:39.293558+0000","last_clean":"2024-06-29T15:55:39.293558+0000","last_became_active":"2024-06-29T15:55:38.291204+0000","last_became_peered":"2024-06-29T15:55:38.291204+0000","last_unstale":"2024-06-29T15:55:39.293558+0000","last_undegraded":"2024-06-29T15:55:39.293558+0000","last_fullsized":"2024-06-29T15:55:39.293558+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:51.009195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.306442+0000","last_change":"2024-06-29T15:55:38.303486+0000","last_active":"2024-06-29T15:55:38.306442+0000","last_peered":"2024-06-29T15:55:38.306442+0000","last_clean":"2024-06-29T15:55:38.306442+0000","last_became_active":"2024-06-29T15:55:38.303092+0000","last_became_peered":"2024-06-29T15:55:38.303092+0000","last_unstale":"2024-06-29T15:55:38.306442+0000","last_undegraded":"2024-06-29T15:55:38.306442+0000","last_fullsized":"2024-06-29T15:55:38.306442+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:52.615803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295296+0000","last_change":"2024-06-29T15:55:38.301657+0000","last_active":"2024-06-29T15:55:39.295296+0000","last_peered":"2024-06-29T15:55:39.295296+0000","last_clean":"2024-06-29T15:55:39.295296+0000","last_became_active":"2024-06-29T15:55:38.301189+0000","last_became_peered":"2024-06-29T15:55:38.301189+0000","last_unstale":"2024-06-29T15:55:39.295296+0000","last_undegraded":"2024-06-29T15:55:39.295296+0000","last_fullsized":"2024-06-29T15:55:39.295296+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:39.593663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304771+0000","last_change":"2024-06-29T15:55:38.305981+0000","last_active":"2024-06-29T15:55:39.304771+0000","last_peered":"2024-06-29T15:55:39.304771+0000","last_clean":"2024-06-29T15:55:39.304771+0000","last_became_active":"2024-06-29T15:55:38.305676+0000","last_became_peered":"2024-06-29T15:55:38.305676+0000","last_unstale":"2024-06-29T15:55:39.304771+0000","last_undegraded":"2024-06-29T15:55:39.304771+0000","last_fullsized":"2024-06-29T15:55:39.304771+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:00.031732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388373+0000","last_change":"2024-06-29T15:55:38.304260+0000","last_active":"2024-06-29T15:55:39.388373+0000","last_peered":"2024-06-29T15:55:39.388373+0000","last_clean":"2024-06-29T15:55:39.388373+0000","last_became_active":"2024-06-29T15:55:38.303942+0000","last_became_peered":"2024-06-29T15:55:38.303942+0000","last_unstale":"2024-06-29T15:55:39.388373+0000","last_undegraded":"2024-06-29T15:55:39.388373+0000","last_fullsized":"2024-06-29T15:55:39.388373+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:23.140306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.310472+0000","last_change":"2024-06-29T15:55:38.307208+0000","last_active":"2024-06-29T15:55:38.310472+0000","last_peered":"2024-06-29T15:55:38.310472+0000","last_clean":"2024-06-29T15:55:38.310472+0000","last_became_active":"2024-06-29T15:55:38.306879+0000","last_became_peered":"2024-06-29T15:55:38.306879+0000","last_unstale":"2024-06-29T15:55:38.310472+0000","last_undegraded":"2024-06-29T15:55:38.310472+0000","last_fullsized":"2024-06-29T15:55:38.310472+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:38.899326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295837+0000","last_change":"2024-06-29T15:55:38.305518+0000","last_active":"2024-06-29T15:55:39.295837+0000","last_peered":"2024-06-29T15:55:39.295837+0000","last_clean":"2024-06-29T15:55:39.295837+0000","last_became_active":"2024-06-29T15:55:38.305181+0000","last_became_peered":"2024-06-29T15:55:38.305181+0000","last_unstale":"2024-06-29T15:55:39.295837+0000","last_undegraded":"2024-06-29T15:55:39.295837+0000","last_fullsized":"2024-06-29T15:55:39.295837+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:22.432293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304525+0000","last_change":"2024-06-29T15:55:38.298004+0000","last_active":"2024-06-29T15:55:39.304525+0000","last_peered":"2024-06-29T15:55:39.304525+0000","last_clean":"2024-06-29T15:55:39.304525+0000","last_became_active":"2024-06-29T15:55:38.297688+0000","last_became_peered":"2024-06-29T15:55:38.297688+0000","last_unstale":"2024-06-29T15:55:39.304525+0000","last_undegraded":"2024-06-29T15:55:39.304525+0000","last_fullsized":"2024-06-29T15:55:39.304525+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:23.140306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388114+0000","last_change":"2024-06-29T15:55:38.301945+0000","last_active":"2024-06-29T15:55:39.388114+0000","last_peered":"2024-06-29T15:55:39.388114+0000","last_clean":"2024-06-29T15:55:39.388114+0000","last_became_active":"2024-06-29T15:55:38.301664+0000","last_became_peered":"2024-06-29T15:55:38.301664+0000","last_unstale":"2024-06-29T15:55:39.388114+0000","last_undegraded":"2024-06-29T15:55:39.388114+0000","last_fullsized":"2024-06-29T15:55:39.388114+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:13.760465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293766+0000","last_change":"2024-06-29T15:55:38.302209+0000","last_active":"2024-06-29T15:55:39.293766+0000","last_peered":"2024-06-29T15:55:39.293766+0000","last_clean":"2024-06-29T15:55:39.293766+0000","last_became_active":"2024-06-29T15:55:38.301825+0000","last_became_peered":"2024-06-29T15:55:38.301825+0000","last_unstale":"2024-06-29T15:55:39.293766+0000","last_undegraded":"2024-06-29T15:55:39.293766+0000","last_fullsized":"2024-06-29T15:55:39.293766+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:24.659163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"195'4","reported_seq":18,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388499+0000","last_change":"2024-06-29T15:55:37.292070+0000","last_active":"2024-06-29T15:55:39.388499+0000","last_peered":"2024-06-29T15:55:39.388499+0000","last_clean":"2024-06-29T15:55:39.388499+0000","last_became_active":"2024-06-29T15:55:37.291781+0000","last_became_peered":"2024-06-29T15:55:37.291781+0000","last_unstale":"2024-06-29T15:55:39.388499+0000","last_undegraded":"2024-06-29T15:55:39.388499+0000","last_fullsized":"2024-06-29T15:55:39.388499+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:50.976402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302782+0000","last_change":"2024-06-29T15:55:38.300811+0000","last_active":"2024-06-29T15:55:39.302782+0000","last_peered":"2024-06-29T15:55:39.302782+0000","last_clean":"2024-06-29T15:55:39.302782+0000","last_became_active":"2024-06-29T15:55:38.300500+0000","last_became_peered":"2024-06-29T15:55:38.300500+0000","last_unstale":"2024-06-29T15:55:39.302782+0000","last_undegraded":"2024-06-29T15:55:39.302782+0000","last_fullsized":"2024-06-29T15:55:39.302782+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:22.351537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295785+0000","last_change":"2024-06-29T15:55:37.286770+0000","last_active":"2024-06-29T15:55:39.295785+0000","last_peered":"2024-06-29T15:55:39.295785+0000","last_clean":"2024-06-29T15:55:39.295785+0000","last_became_active":"2024-06-29T15:55:37.286422+0000","last_became_peered":"2024-06-29T15:55:37.286422+0000","last_unstale":"2024-06-29T15:55:39.295785+0000","last_undegraded":"2024-06-29T15:55:39.295785+0000","last_fullsized":"2024-06-29T15:55:39.295785+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:49:25.704312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295155+0000","last_change":"2024-06-29T15:55:38.302639+0000","last_active":"2024-06-29T15:55:39.295155+0000","last_peered":"2024-06-29T15:55:39.295155+0000","last_clean":"2024-06-29T15:55:39.295155+0000","last_became_active":"2024-06-29T15:55:38.302331+0000","last_became_peered":"2024-06-29T15:55:38.302331+0000","last_unstale":"2024-06-29T15:55:39.295155+0000","last_undegraded":"2024-06-29T15:55:39.295155+0000","last_fullsized":"2024-06-29T15:55:39.295155+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:01:11.235966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"195'2","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304863+0000","last_change":"2024-06-29T15:55:37.286517+0000","last_active":"2024-06-29T15:55:39.304863+0000","last_peered":"2024-06-29T15:55:39.304863+0000","last_clean":"2024-06-29T15:55:39.304863+0000","last_became_active":"2024-06-29T15:55:37.286230+0000","last_became_peered":"2024-06-29T15:55:37.286230+0000","last_unstale":"2024-06-29T15:55:39.304863+0000","last_undegraded":"2024-06-29T15:55:39.304863+0000","last_fullsized":"2024-06-29T15:55:39.304863+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:13:45.897605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295891+0000","last_change":"2024-06-29T15:55:38.301698+0000","last_active":"2024-06-29T15:55:39.295891+0000","last_peered":"2024-06-29T15:55:39.295891+0000","last_clean":"2024-06-29T15:55:39.295891+0000","last_became_active":"2024-06-29T15:55:38.301382+0000","last_became_peered":"2024-06-29T15:55:38.301382+0000","last_unstale":"2024-06-29T15:55:39.295891+0000","last_undegraded":"2024-06-29T15:55:39.295891+0000","last_fullsized":"2024-06-29T15:55:39.295891+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:22.351537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"195'2","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.316232+0000","last_change":"2024-06-29T15:55:37.291631+0000","last_active":"2024-06-29T15:55:38.316232+0000","last_peered":"2024-06-29T15:55:38.316232+0000","last_clean":"2024-06-29T15:55:38.316232+0000","last_became_active":"2024-06-29T15:55:37.291253+0000","last_became_peered":"2024-06-29T15:55:37.291253+0000","last_unstale":"2024-06-29T15:55:38.316232+0000","last_undegraded":"2024-06-29T15:55:38.316232+0000","last_fullsized":"2024-06-29T15:55:38.316232+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:28:41.667396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302841+0000","last_change":"2024-06-29T15:55:38.307143+0000","last_active":"2024-06-29T15:55:39.302841+0000","last_peered":"2024-06-29T15:55:39.302841+0000","last_clean":"2024-06-29T15:55:39.302841+0000","last_became_active":"2024-06-29T15:55:38.306457+0000","last_became_peered":"2024-06-29T15:55:38.306457+0000","last_unstale":"2024-06-29T15:55:39.302841+0000","last_undegraded":"2024-06-29T15:55:39.302841+0000","last_fullsized":"2024-06-29T15:55:39.302841+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:00.578618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388467+0000","last_change":"2024-06-29T15:55:37.287909+0000","last_active":"2024-06-29T15:55:39.388467+0000","last_peered":"2024-06-29T15:55:39.388467+0000","last_clean":"2024-06-29T15:55:39.388467+0000","last_became_active":"2024-06-29T15:55:37.287633+0000","last_became_peered":"2024-06-29T15:55:37.287633+0000","last_unstale":"2024-06-29T15:55:39.388467+0000","last_undegraded":"2024-06-29T15:55:39.388467+0000","last_fullsized":"2024-06-29T15:55:39.388467+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:17.571606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293702+0000","last_change":"2024-06-29T15:55:38.293041+0000","last_active":"2024-06-29T15:55:39.293702+0000","last_peered":"2024-06-29T15:55:39.293702+0000","last_clean":"2024-06-29T15:55:39.293702+0000","last_became_active":"2024-06-29T15:55:38.292149+0000","last_became_peered":"2024-06-29T15:55:38.292149+0000","last_unstale":"2024-06-29T15:55:39.293702+0000","last_undegraded":"2024-06-29T15:55:39.293702+0000","last_fullsized":"2024-06-29T15:55:39.293702+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:09.629606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293514+0000","last_change":"2024-06-29T15:55:37.287897+0000","last_active":"2024-06-29T15:55:39.293514+0000","last_peered":"2024-06-29T15:55:39.293514+0000","last_clean":"2024-06-29T15:55:39.293514+0000","last_became_active":"2024-06-29T15:55:37.287597+0000","last_became_peered":"2024-06-29T15:55:37.287597+0000","last_unstale":"2024-06-29T15:55:39.293514+0000","last_undegraded":"2024-06-29T15:55:39.293514+0000","last_fullsized":"2024-06-29T15:55:39.293514+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:11:14.661757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.294893+0000","last_change":"2024-06-29T15:55:38.311576+0000","last_active":"2024-06-29T15:55:39.294893+0000","last_peered":"2024-06-29T15:55:39.294893+0000","last_clean":"2024-06-29T15:55:39.294893+0000","last_became_active":"2024-06-29T15:55:38.311168+0000","last_became_peered":"2024-06-29T15:55:38.311168+0000","last_unstale":"2024-06-29T15:55:39.294893+0000","last_undegraded":"2024-06-29T15:55:39.294893+0000","last_fullsized":"2024-06-29T15:55:39.294893+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:16.113894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.295188+0000","last_change":"2024-06-29T15:55:37.293524+0000","last_active":"2024-06-29T15:55:38.295188+0000","last_peered":"2024-06-29T15:55:38.295188+0000","last_clean":"2024-06-29T15:55:38.295188+0000","last_became_active":"2024-06-29T15:55:37.292267+0000","last_became_peered":"2024-06-29T15:55:37.292267+0000","last_unstale":"2024-06-29T15:55:38.295188+0000","last_undegraded":"2024-06-29T15:55:38.295188+0000","last_fullsized":"2024-06-29T15:55:38.295188+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:36:41.989152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304805+0000","last_change":"2024-06-29T15:55:38.293301+0000","last_active":"2024-06-29T15:55:39.304805+0000","last_peered":"2024-06-29T15:55:39.304805+0000","last_clean":"2024-06-29T15:55:39.304805+0000","last_became_active":"2024-06-29T15:55:38.293013+0000","last_became_peered":"2024-06-29T15:55:38.293013+0000","last_unstale":"2024-06-29T15:55:39.304805+0000","last_undegraded":"2024-06-29T15:55:39.304805+0000","last_fullsized":"2024-06-29T15:55:39.304805+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:16.113894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.294959+0000","last_change":"2024-06-29T15:55:37.295164+0000","last_active":"2024-06-29T15:55:39.294959+0000","last_peered":"2024-06-29T15:55:39.294959+0000","last_clean":"2024-06-29T15:55:39.294959+0000","last_became_active":"2024-06-29T15:55:37.294705+0000","last_became_peered":"2024-06-29T15:55:37.294705+0000","last_unstale":"2024-06-29T15:55:39.294959+0000","last_undegraded":"2024-06-29T15:55:39.294959+0000","last_fullsized":"2024-06-29T15:55:39.294959+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:18.836280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293568+0000","last_change":"2024-06-29T15:55:38.306390+0000","last_active":"2024-06-29T15:55:39.293568+0000","last_peered":"2024-06-29T15:55:39.293568+0000","last_clean":"2024-06-29T15:55:39.293568+0000","last_became_active":"2024-06-29T15:55:38.306066+0000","last_became_peered":"2024-06-29T15:55:38.306066+0000","last_unstale":"2024-06-29T15:55:39.293568+0000","last_undegraded":"2024-06-29T15:55:39.293568+0000","last_fullsized":"2024-06-29T15:55:39.293568+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:52.705253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388328+0000","last_change":"2024-06-29T15:55:37.295140+0000","last_active":"2024-06-29T15:55:39.388328+0000","last_peered":"2024-06-29T15:55:39.388328+0000","last_clean":"2024-06-29T15:55:39.388328+0000","last_became_active":"2024-06-29T15:55:37.294791+0000","last_became_peered":"2024-06-29T15:55:37.294791+0000","last_unstale":"2024-06-29T15:55:39.388328+0000","last_undegraded":"2024-06-29T15:55:39.388328+0000","last_fullsized":"2024-06-29T15:55:39.388328+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:51.610731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295456+0000","last_change":"2024-06-29T15:55:38.303655+0000","last_active":"2024-06-29T15:55:39.295456+0000","last_peered":"2024-06-29T15:55:39.295456+0000","last_clean":"2024-06-29T15:55:39.295456+0000","last_became_active":"2024-06-29T15:55:38.303256+0000","last_became_peered":"2024-06-29T15:55:38.303256+0000","last_unstale":"2024-06-29T15:55:39.295456+0000","last_undegraded":"2024-06-29T15:55:39.295456+0000","last_fullsized":"2024-06-29T15:55:39.295456+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:20.146322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"195'3","reported_seq":17,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388485+0000","last_change":"2024-06-29T15:55:37.294378+0000","last_active":"2024-06-29T15:55:39.388485+0000","last_peered":"2024-06-29T15:55:39.388485+0000","last_clean":"2024-06-29T15:55:39.388485+0000","last_became_active":"2024-06-29T15:55:37.294065+0000","last_became_peered":"2024-06-29T15:55:37.294065+0000","last_unstale":"2024-06-29T15:55:39.388485+0000","last_undegraded":"2024-06-29T15:55:39.388485+0000","last_fullsized":"2024-06-29T15:55:39.388485+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:21:25.949676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302514+0000","last_change":"2024-06-29T15:55:38.305875+0000","last_active":"2024-06-29T15:55:39.302514+0000","last_peered":"2024-06-29T15:55:39.302514+0000","last_clean":"2024-06-29T15:55:39.302514+0000","last_became_active":"2024-06-29T15:55:38.305569+0000","last_became_peered":"2024-06-29T15:55:38.305569+0000","last_unstale":"2024-06-29T15:55:39.302514+0000","last_undegraded":"2024-06-29T15:55:39.302514+0000","last_fullsized":"2024-06-29T15:55:39.302514+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:45.708017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"195'6","reported_seq":20,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304690+0000","last_change":"2024-06-29T15:55:37.293162+0000","last_active":"2024-06-29T15:55:39.304690+0000","last_peered":"2024-06-29T15:55:39.304690+0000","last_clean":"2024-06-29T15:55:39.304690+0000","last_became_active":"2024-06-29T15:55:37.292787+0000","last_became_peered":"2024-06-29T15:55:37.292787+0000","last_unstale":"2024-06-29T15:55:39.304690+0000","last_undegraded":"2024-06-29T15:55:39.304690+0000","last_fullsized":"2024-06-29T15:55:39.304690+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:40:57.787403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304597+0000","last_change":"2024-06-29T15:55:38.307081+0000","last_active":"2024-06-29T15:55:39.304597+0000","last_peered":"2024-06-29T15:55:39.304597+0000","last_clean":"2024-06-29T15:55:39.304597+0000","last_became_active":"2024-06-29T15:55:38.306779+0000","last_became_peered":"2024-06-29T15:55:38.306779+0000","last_unstale":"2024-06-29T15:55:39.304597+0000","last_undegraded":"2024-06-29T15:55:39.304597+0000","last_fullsized":"2024-06-29T15:55:39.304597+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:05.156497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"195'2","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.313932+0000","last_change":"2024-06-29T15:55:37.290515+0000","last_active":"2024-06-29T15:55:38.313932+0000","last_peered":"2024-06-29T15:55:38.313932+0000","last_clean":"2024-06-29T15:55:38.313932+0000","last_became_active":"2024-06-29T15:55:37.290217+0000","last_became_peered":"2024-06-29T15:55:37.290217+0000","last_unstale":"2024-06-29T15:55:38.313932+0000","last_undegraded":"2024-06-29T15:55:38.313932+0000","last_fullsized":"2024-06-29T15:55:38.313932+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:29:56.177707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.294734+0000","last_change":"2024-06-29T15:55:38.306162+0000","last_active":"2024-06-29T15:55:39.294734+0000","last_peered":"2024-06-29T15:55:39.294734+0000","last_clean":"2024-06-29T15:55:39.294734+0000","last_became_active":"2024-06-29T15:55:38.305801+0000","last_became_peered":"2024-06-29T15:55:38.305801+0000","last_unstale":"2024-06-29T15:55:39.294734+0000","last_undegraded":"2024-06-29T15:55:39.294734+0000","last_fullsized":"2024-06-29T15:55:39.294734+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:00.578618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302564+0000","last_change":"2024-06-29T15:55:37.285367+0000","last_active":"2024-06-29T15:55:39.302564+0000","last_peered":"2024-06-29T15:55:39.302564+0000","last_clean":"2024-06-29T15:55:39.302564+0000","last_became_active":"2024-06-29T15:55:37.285086+0000","last_became_peered":"2024-06-29T15:55:37.285086+0000","last_unstale":"2024-06-29T15:55:39.302564+0000","last_undegraded":"2024-06-29T15:55:39.302564+0000","last_fullsized":"2024-06-29T15:55:39.302564+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:21:25.949676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295663+0000","last_change":"2024-06-29T15:55:38.301104+0000","last_active":"2024-06-29T15:55:39.295663+0000","last_peered":"2024-06-29T15:55:39.295663+0000","last_clean":"2024-06-29T15:55:39.295663+0000","last_became_active":"2024-06-29T15:55:38.300625+0000","last_became_peered":"2024-06-29T15:55:38.300625+0000","last_unstale":"2024-06-29T15:55:39.295663+0000","last_undegraded":"2024-06-29T15:55:39.295663+0000","last_fullsized":"2024-06-29T15:55:39.295663+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:45.708017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293327+0000","last_change":"2024-06-29T15:55:37.288565+0000","last_active":"2024-06-29T15:55:39.293327+0000","last_peered":"2024-06-29T15:55:39.293327+0000","last_clean":"2024-06-29T15:55:39.293327+0000","last_became_active":"2024-06-29T15:55:37.288263+0000","last_became_peered":"2024-06-29T15:55:37.288263+0000","last_unstale":"2024-06-29T15:55:39.293327+0000","last_undegraded":"2024-06-29T15:55:39.293327+0000","last_fullsized":"2024-06-29T15:55:39.293327+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:17.890026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302300+0000","last_change":"2024-06-29T15:55:38.306970+0000","last_active":"2024-06-29T15:55:39.302300+0000","last_peered":"2024-06-29T15:55:39.302300+0000","last_clean":"2024-06-29T15:55:39.302300+0000","last_became_active":"2024-06-29T15:55:38.306585+0000","last_became_peered":"2024-06-29T15:55:38.306585+0000","last_unstale":"2024-06-29T15:55:39.302300+0000","last_undegraded":"2024-06-29T15:55:39.302300+0000","last_fullsized":"2024-06-29T15:55:39.302300+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:13.760465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"195'8","reported_seq":20,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.316803+0000","last_change":"2024-06-29T15:55:37.291650+0000","last_active":"2024-06-29T15:55:38.316803+0000","last_peered":"2024-06-29T15:55:38.316803+0000","last_clean":"2024-06-29T15:55:38.316803+0000","last_became_active":"2024-06-29T15:55:37.290970+0000","last_became_peered":"2024-06-29T15:55:37.290970+0000","last_unstale":"2024-06-29T15:55:38.316803+0000","last_undegraded":"2024-06-29T15:55:38.316803+0000","last_fullsized":"2024-06-29T15:55:38.316803+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:34:49.615716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302679+0000","last_change":"2024-06-29T15:55:38.307465+0000","last_active":"2024-06-29T15:55:39.302679+0000","last_peered":"2024-06-29T15:55:39.302679+0000","last_clean":"2024-06-29T15:55:39.302679+0000","last_became_active":"2024-06-29T15:55:38.307194+0000","last_became_peered":"2024-06-29T15:55:38.307194+0000","last_unstale":"2024-06-29T15:55:39.302679+0000","last_undegraded":"2024-06-29T15:55:39.302679+0000","last_fullsized":"2024-06-29T15:55:39.302679+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:23.140306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"195'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388542+0000","last_change":"2024-06-29T15:55:37.288147+0000","last_active":"2024-06-29T15:55:39.388542+0000","last_peered":"2024-06-29T15:55:39.388542+0000","last_clean":"2024-06-29T15:55:39.388542+0000","last_became_active":"2024-06-29T15:55:37.287740+0000","last_became_peered":"2024-06-29T15:55:37.287740+0000","last_unstale":"2024-06-29T15:55:39.388542+0000","last_undegraded":"2024-06-29T15:55:39.388542+0000","last_fullsized":"2024-06-29T15:55:39.388542+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:31.244085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.296104+0000","last_change":"2024-06-29T15:55:38.302325+0000","last_active":"2024-06-29T15:55:39.296104+0000","last_peered":"2024-06-29T15:55:39.296104+0000","last_clean":"2024-06-29T15:55:39.296104+0000","last_became_active":"2024-06-29T15:55:38.302011+0000","last_became_peered":"2024-06-29T15:55:38.302011+0000","last_unstale":"2024-06-29T15:55:39.296104+0000","last_undegraded":"2024-06-29T15:55:39.296104+0000","last_fullsized":"2024-06-29T15:55:39.296104+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:00.578618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"195'2","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293876+0000","last_change":"2024-06-29T15:55:37.291817+0000","last_active":"2024-06-29T15:55:39.293876+0000","last_peered":"2024-06-29T15:55:39.293876+0000","last_clean":"2024-06-29T15:55:39.293876+0000","last_became_active":"2024-06-29T15:55:37.291507+0000","last_became_peered":"2024-06-29T15:55:37.291507+0000","last_unstale":"2024-06-29T15:55:39.293876+0000","last_undegraded":"2024-06-29T15:55:39.293876+0000","last_fullsized":"2024-06-29T15:55:39.293876+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:19:07.597147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295735+0000","last_change":"2024-06-29T15:55:38.311691+0000","last_active":"2024-06-29T15:55:39.295735+0000","last_peered":"2024-06-29T15:55:39.295735+0000","last_clean":"2024-06-29T15:55:39.295735+0000","last_became_active":"2024-06-29T15:55:38.311393+0000","last_became_peered":"2024-06-29T15:55:38.311393+0000","last_unstale":"2024-06-29T15:55:39.295735+0000","last_undegraded":"2024-06-29T15:55:39.295735+0000","last_fullsized":"2024-06-29T15:55:39.295735+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:05:23.140306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"195'1","reported_seq":13,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.314119+0000","last_change":"2024-06-29T15:55:37.288592+0000","last_active":"2024-06-29T15:55:38.314119+0000","last_peered":"2024-06-29T15:55:38.314119+0000","last_clean":"2024-06-29T15:55:38.314119+0000","last_became_active":"2024-06-29T15:55:37.288278+0000","last_became_peered":"2024-06-29T15:55:37.288278+0000","last_unstale":"2024-06-29T15:55:38.314119+0000","last_undegraded":"2024-06-29T15:55:38.314119+0000","last_fullsized":"2024-06-29T15:55:38.314119+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:58:39.184728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.299327+0000","last_change":"2024-06-29T15:55:38.293400+0000","last_active":"2024-06-29T15:55:38.299327+0000","last_peered":"2024-06-29T15:55:38.299327+0000","last_clean":"2024-06-29T15:55:38.299327+0000","last_became_active":"2024-06-29T15:55:38.293089+0000","last_became_peered":"2024-06-29T15:55:38.293089+0000","last_unstale":"2024-06-29T15:55:38.299327+0000","last_undegraded":"2024-06-29T15:55:38.299327+0000","last_fullsized":"2024-06-29T15:55:38.299327+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:45:49.886541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295354+0000","last_change":"2024-06-29T15:55:37.288444+0000","last_active":"2024-06-29T15:55:39.295354+0000","last_peered":"2024-06-29T15:55:39.295354+0000","last_clean":"2024-06-29T15:55:39.295354+0000","last_became_active":"2024-06-29T15:55:37.287047+0000","last_became_peered":"2024-06-29T15:55:37.287047+0000","last_unstale":"2024-06-29T15:55:39.295354+0000","last_undegraded":"2024-06-29T15:55:39.295354+0000","last_fullsized":"2024-06-29T15:55:39.295354+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:12.755393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302504+0000","last_change":"2024-06-29T15:55:38.304546+0000","last_active":"2024-06-29T15:55:39.302504+0000","last_peered":"2024-06-29T15:55:39.302504+0000","last_clean":"2024-06-29T15:55:39.302504+0000","last_became_active":"2024-06-29T15:55:38.304181+0000","last_became_peered":"2024-06-29T15:55:38.304181+0000","last_unstale":"2024-06-29T15:55:39.302504+0000","last_undegraded":"2024-06-29T15:55:39.302504+0000","last_fullsized":"2024-06-29T15:55:39.302504+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:26:22.432293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"195'2","reported_seq":16,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304672+0000","last_change":"2024-06-29T15:55:37.284720+0000","last_active":"2024-06-29T15:55:39.304672+0000","last_peered":"2024-06-29T15:55:39.304672+0000","last_clean":"2024-06-29T15:55:39.304672+0000","last_became_active":"2024-06-29T15:55:37.284415+0000","last_became_peered":"2024-06-29T15:55:37.284415+0000","last_unstale":"2024-06-29T15:55:39.304672+0000","last_undegraded":"2024-06-29T15:55:39.304672+0000","last_fullsized":"2024-06-29T15:55:39.304672+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:37.894254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295037+0000","last_change":"2024-06-29T15:55:38.301563+0000","last_active":"2024-06-29T15:55:39.295037+0000","last_peered":"2024-06-29T15:55:39.295037+0000","last_clean":"2024-06-29T15:55:39.295037+0000","last_became_active":"2024-06-29T15:55:38.301166+0000","last_became_peered":"2024-06-29T15:55:38.301166+0000","last_unstale":"2024-06-29T15:55:39.295037+0000","last_undegraded":"2024-06-29T15:55:39.295037+0000","last_fullsized":"2024-06-29T15:55:39.295037+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:35:05.434514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388230+0000","last_change":"2024-06-29T15:55:37.293839+0000","last_active":"2024-06-29T15:55:39.388230+0000","last_peered":"2024-06-29T15:55:39.388230+0000","last_clean":"2024-06-29T15:55:39.388230+0000","last_became_active":"2024-06-29T15:55:37.293489+0000","last_became_peered":"2024-06-29T15:55:37.293489+0000","last_unstale":"2024-06-29T15:55:39.388230+0000","last_undegraded":"2024-06-29T15:55:39.388230+0000","last_fullsized":"2024-06-29T15:55:39.388230+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:45:48.881469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.309937+0000","last_change":"2024-06-29T15:55:38.306080+0000","last_active":"2024-06-29T15:55:38.309937+0000","last_peered":"2024-06-29T15:55:38.309937+0000","last_clean":"2024-06-29T15:55:38.309937+0000","last_became_active":"2024-06-29T15:55:38.305768+0000","last_became_peered":"2024-06-29T15:55:38.305768+0000","last_unstale":"2024-06-29T15:55:38.309937+0000","last_undegraded":"2024-06-29T15:55:38.309937+0000","last_fullsized":"2024-06-29T15:55:38.309937+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:15:28.214095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304636+0000","last_change":"2024-06-29T15:55:37.288276+0000","last_active":"2024-06-29T15:55:39.304636+0000","last_peered":"2024-06-29T15:55:39.304636+0000","last_clean":"2024-06-29T15:55:39.304636+0000","last_became_active":"2024-06-29T15:55:37.287931+0000","last_became_peered":"2024-06-29T15:55:37.287931+0000","last_unstale":"2024-06-29T15:55:39.304636+0000","last_undegraded":"2024-06-29T15:55:39.304636+0000","last_fullsized":"2024-06-29T15:55:39.304636+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:26.095060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388120+0000","last_change":"2024-06-29T15:55:38.300711+0000","last_active":"2024-06-29T15:55:39.388120+0000","last_peered":"2024-06-29T15:55:39.388120+0000","last_clean":"2024-06-29T15:55:39.388120+0000","last_became_active":"2024-06-29T15:55:38.300266+0000","last_became_peered":"2024-06-29T15:55:38.300266+0000","last_unstale":"2024-06-29T15:55:39.388120+0000","last_undegraded":"2024-06-29T15:55:39.388120+0000","last_fullsized":"2024-06-29T15:55:39.388120+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:02:34.409026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295653+0000","last_change":"2024-06-29T15:55:37.289034+0000","last_active":"2024-06-29T15:55:39.295653+0000","last_peered":"2024-06-29T15:55:39.295653+0000","last_clean":"2024-06-29T15:55:39.295653+0000","last_became_active":"2024-06-29T15:55:37.288737+0000","last_became_peered":"2024-06-29T15:55:37.288737+0000","last_unstale":"2024-06-29T15:55:39.295653+0000","last_undegraded":"2024-06-29T15:55:39.295653+0000","last_fullsized":"2024-06-29T15:55:39.295653+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:26.095060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388470+0000","last_change":"2024-06-29T15:55:38.302229+0000","last_active":"2024-06-29T15:55:39.388470+0000","last_peered":"2024-06-29T15:55:39.388470+0000","last_clean":"2024-06-29T15:55:39.388470+0000","last_became_active":"2024-06-29T15:55:38.301322+0000","last_became_peered":"2024-06-29T15:55:38.301322+0000","last_unstale":"2024-06-29T15:55:39.388470+0000","last_undegraded":"2024-06-29T15:55:39.388470+0000","last_fullsized":"2024-06-29T15:55:39.388470+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:58:22.351537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"195'2","reported_seq":14,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.314892+0000","last_change":"2024-06-29T15:55:37.293359+0000","last_active":"2024-06-29T15:55:38.314892+0000","last_peered":"2024-06-29T15:55:38.314892+0000","last_clean":"2024-06-29T15:55:38.314892+0000","last_became_active":"2024-06-29T15:55:37.293034+0000","last_became_peered":"2024-06-29T15:55:37.293034+0000","last_unstale":"2024-06-29T15:55:38.314892+0000","last_undegraded":"2024-06-29T15:55:38.314892+0000","last_fullsized":"2024-06-29T15:55:38.314892+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:09.475576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302834+0000","last_change":"2024-06-29T15:55:38.312682+0000","last_active":"2024-06-29T15:55:39.302834+0000","last_peered":"2024-06-29T15:55:39.302834+0000","last_clean":"2024-06-29T15:55:39.302834+0000","last_became_active":"2024-06-29T15:55:38.312377+0000","last_became_peered":"2024-06-29T15:55:38.312377+0000","last_unstale":"2024-06-29T15:55:39.302834+0000","last_undegraded":"2024-06-29T15:55:39.302834+0000","last_fullsized":"2024-06-29T15:55:39.302834+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:21:05.156497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"196'1","reported_seq":15,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.303817+0000","last_change":"2024-06-29T15:55:37.283739+0000","last_active":"2024-06-29T15:55:39.303817+0000","last_peered":"2024-06-29T15:55:39.303817+0000","last_clean":"2024-06-29T15:55:39.303817+0000","last_became_active":"2024-06-29T15:55:37.282478+0000","last_became_peered":"2024-06-29T15:55:37.282478+0000","last_unstale":"2024-06-29T15:55:39.303817+0000","last_undegraded":"2024-06-29T15:55:39.303817+0000","last_fullsized":"2024-06-29T15:55:39.303817+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:08:50.976402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293623+0000","last_change":"2024-06-29T15:55:38.293246+0000","last_active":"2024-06-29T15:55:39.293623+0000","last_peered":"2024-06-29T15:55:39.293623+0000","last_clean":"2024-06-29T15:55:39.293623+0000","last_became_active":"2024-06-29T15:55:38.292921+0000","last_became_peered":"2024-06-29T15:55:38.292921+0000","last_unstale":"2024-06-29T15:55:39.293623+0000","last_undegraded":"2024-06-29T15:55:39.293623+0000","last_fullsized":"2024-06-29T15:55:39.293623+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:58:52.393383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302468+0000","last_change":"2024-06-29T15:55:37.282836+0000","last_active":"2024-06-29T15:55:39.302468+0000","last_peered":"2024-06-29T15:55:39.302468+0000","last_clean":"2024-06-29T15:55:39.302468+0000","last_became_active":"2024-06-29T15:55:37.282505+0000","last_became_peered":"2024-06-29T15:55:37.282505+0000","last_unstale":"2024-06-29T15:55:39.302468+0000","last_undegraded":"2024-06-29T15:55:39.302468+0000","last_fullsized":"2024-06-29T15:55:39.302468+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:31.244085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293897+0000","last_change":"2024-06-29T15:55:38.297535+0000","last_active":"2024-06-29T15:55:39.293897+0000","last_peered":"2024-06-29T15:55:39.293897+0000","last_clean":"2024-06-29T15:55:39.293897+0000","last_became_active":"2024-06-29T15:55:38.297224+0000","last_became_peered":"2024-06-29T15:55:38.297224+0000","last_unstale":"2024-06-29T15:55:39.293897+0000","last_undegraded":"2024-06-29T15:55:39.293897+0000","last_fullsized":"2024-06-29T15:55:39.293897+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:27:40.644641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.293716+0000","last_change":"2024-06-29T15:55:37.290803+0000","last_active":"2024-06-29T15:55:39.293716+0000","last_peered":"2024-06-29T15:55:39.293716+0000","last_clean":"2024-06-29T15:55:39.293716+0000","last_became_active":"2024-06-29T15:55:37.290493+0000","last_became_peered":"2024-06-29T15:55:37.290493+0000","last_unstale":"2024-06-29T15:55:39.293716+0000","last_undegraded":"2024-06-29T15:55:39.293716+0000","last_fullsized":"2024-06-29T15:55:39.293716+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:11:59.493868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.311080+0000","last_change":"2024-06-29T15:55:38.309132+0000","last_active":"2024-06-29T15:55:38.311080+0000","last_peered":"2024-06-29T15:55:38.311080+0000","last_clean":"2024-06-29T15:55:38.311080+0000","last_became_active":"2024-06-29T15:55:38.308811+0000","last_became_peered":"2024-06-29T15:55:38.308811+0000","last_unstale":"2024-06-29T15:55:38.311080+0000","last_undegraded":"2024-06-29T15:55:38.311080+0000","last_fullsized":"2024-06-29T15:55:38.311080+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:26:06.331146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295991+0000","last_change":"2024-06-29T15:55:37.280018+0000","last_active":"2024-06-29T15:55:39.295991+0000","last_peered":"2024-06-29T15:55:39.295991+0000","last_clean":"2024-06-29T15:55:39.295991+0000","last_became_active":"2024-06-29T15:55:37.279627+0000","last_became_peered":"2024-06-29T15:55:37.279627+0000","last_unstale":"2024-06-29T15:55:39.295991+0000","last_undegraded":"2024-06-29T15:55:39.295991+0000","last_fullsized":"2024-06-29T15:55:39.295991+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:40:57.787403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.313996+0000","last_change":"2024-06-29T15:55:38.310978+0000","last_active":"2024-06-29T15:55:38.313996+0000","last_peered":"2024-06-29T15:55:38.313996+0000","last_clean":"2024-06-29T15:55:38.313996+0000","last_became_active":"2024-06-29T15:55:38.310670+0000","last_became_peered":"2024-06-29T15:55:38.310670+0000","last_unstale":"2024-06-29T15:55:38.313996+0000","last_undegraded":"2024-06-29T15:55:38.313996+0000","last_fullsized":"2024-06-29T15:55:38.313996+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:37:34.282870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.295616+0000","last_change":"2024-06-29T15:55:37.292036+0000","last_active":"2024-06-29T15:55:39.295616+0000","last_peered":"2024-06-29T15:55:39.295616+0000","last_clean":"2024-06-29T15:55:39.295616+0000","last_became_active":"2024-06-29T15:55:37.291651+0000","last_became_peered":"2024-06-29T15:55:37.291651+0000","last_unstale":"2024-06-29T15:55:39.295616+0000","last_undegraded":"2024-06-29T15:55:39.295616+0000","last_fullsized":"2024-06-29T15:55:39.295616+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:09:37.894254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.306419+0000","last_change":"2024-06-29T15:55:38.303838+0000","last_active":"2024-06-29T15:55:38.306419+0000","last_peered":"2024-06-29T15:55:38.306419+0000","last_clean":"2024-06-29T15:55:38.306419+0000","last_became_active":"2024-06-29T15:55:38.303497+0000","last_became_peered":"2024-06-29T15:55:38.303497+0000","last_unstale":"2024-06-29T15:55:38.306419+0000","last_undegraded":"2024-06-29T15:55:38.306419+0000","last_fullsized":"2024-06-29T15:55:38.306419+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:21:26.954748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.289345+0000","last_change":"2024-06-29T15:55:37.287764+0000","last_active":"2024-06-29T15:55:38.289345+0000","last_peered":"2024-06-29T15:55:38.289345+0000","last_clean":"2024-06-29T15:55:38.289345+0000","last_became_active":"2024-06-29T15:55:37.287253+0000","last_became_peered":"2024-06-29T15:55:37.287253+0000","last_unstale":"2024-06-29T15:55:38.289345+0000","last_undegraded":"2024-06-29T15:55:38.289345+0000","last_fullsized":"2024-06-29T15:55:38.289345+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:48:31.386686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.388152+0000","last_change":"2024-06-29T15:55:38.303162+0000","last_active":"2024-06-29T15:55:39.388152+0000","last_peered":"2024-06-29T15:55:39.388152+0000","last_clean":"2024-06-29T15:55:39.388152+0000","last_became_active":"2024-06-29T15:55:38.302861+0000","last_became_peered":"2024-06-29T15:55:38.302861+0000","last_unstale":"2024-06-29T15:55:39.388152+0000","last_undegraded":"2024-06-29T15:55:39.388152+0000","last_fullsized":"2024-06-29T15:55:39.388152+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:44.137473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":14,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.304443+0000","last_change":"2024-06-29T15:55:37.284806+0000","last_active":"2024-06-29T15:55:39.304443+0000","last_peered":"2024-06-29T15:55:39.304443+0000","last_clean":"2024-06-29T15:55:39.304443+0000","last_became_active":"2024-06-29T15:55:37.281664+0000","last_became_peered":"2024-06-29T15:55:37.281664+0000","last_unstale":"2024-06-29T15:55:39.304443+0000","last_undegraded":"2024-06-29T15:55:39.304443+0000","last_fullsized":"2024-06-29T15:55:39.304443+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:49:25.704312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":12,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-06-29T15:55:39.302188+0000","last_change":"2024-06-29T15:55:38.300752+0000","last_active":"2024-06-29T15:55:39.302188+0000","last_peered":"2024-06-29T15:55:39.302188+0000","last_clean":"2024-06-29T15:55:39.302188+0000","last_became_active":"2024-06-29T15:55:38.300397+0000","last_became_peered":"2024-06-29T15:55:38.300397+0000","last_unstale":"2024-06-29T15:55:39.302188+0000","last_undegraded":"2024-06-29T15:55:39.302188+0000","last_fullsized":"2024-06-29T15:55:39.302188+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:37.269666+0000","last_clean_scrub_stamp":"2024-06-29T15:55:37.269666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:19.841352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":12,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.295603+0000","last_change":"2024-06-29T15:55:37.292429+0000","last_active":"2024-06-29T15:55:38.295603+0000","last_peered":"2024-06-29T15:55:38.295603+0000","last_clean":"2024-06-29T15:55:38.295603+0000","last_became_active":"2024-06-29T15:55:37.292107+0000","last_became_peered":"2024-06-29T15:55:37.292107+0000","last_unstale":"2024-06-29T15:55:38.295603+0000","last_undegraded":"2024-06-29T15:55:38.295603+0000","last_fullsized":"2024-06-29T15:55:38.295603+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:55:36.264594+0000","last_clean_scrub_stamp":"2024-06-29T15:55:36.264594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:35.819045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"185'205","reported_seq":955,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-06-29T15:55:38.295044+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:55:38.295044+0000","last_peered":"2024-06-29T15:55:38.295044+0000","last_clean":"2024-06-29T15:55:38.295044+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:55:38.295044+0000","last_undegraded":"2024-06-29T15:55:38.295044+0000","last_fullsized":"2024-06-29T15:55:38.295044+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1206,"num_read_kb":3399,"num_write":329,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1206,"num_read_kb":3399,"num_write":329,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48044,"kb_used_data":2976,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323796,"statfs":{"total":96636764160,"available":96587567104,"internally_reserved":0,"allocated":3047424,"data_stored":1781829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41896,"kb_used_data":2388,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329944,"statfs":{"total":96636764160,"available":96593862656,"internally_reserved":0,"allocated":2445312,"data_stored":1191357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41896,"kb_used_data":2388,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329944,"statfs":{"total":96636764160,"available":96593862656,"internally_reserved":0,"allocated":2445312,"data_stored":1191357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575965,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59952,"kb_used_data":2404,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311888,"statfs":{"total":96636764160,"available":96575373312,"internally_reserved":0,"allocated":2461696,"data_stored":1193551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575965,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64120,"kb_used_data":2412,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307720,"statfs":{"total":96636764160,"available":96571105280,"internally_reserved":0,"allocated":2469888,"data_stored":1197951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575965,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47476,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324364,"statfs":{"total":96636764160,"available":96588148736,"internally_reserved":0,"allocated":2457600,"data_stored":1192051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575964,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59960,"kb_used_data":2412,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311880,"statfs":{"total":96636764160,"available":96575365120,"internally_reserved":0,"allocated":2469888,"data_stored":1199369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575965,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56364,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315476,"statfs":{"total":96636764160,"available":96579047424,"internally_reserved":0,"allocated":3043328,"data_stored":1781747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:55:42.686 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:55:42.687 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:55:42.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.841+0000 7fdf91495640 1 -- 172.21.15.81:0/3170778566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf8c0b5250 msgr2=0x7fdf8c0d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.841+0000 7fdf91495640 1 --2- 172.21.15.81:0/3170778566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf8c0b5250 0x7fdf8c0d1fc0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fdf84005fd0 tx=0x7fdf8402a670 comp rx=0 tx=0).stop 2024-06-29T15:55:42.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.841+0000 7fdf91495640 1 -- 172.21.15.81:0/3170778566 shutdown_connections 2024-06-29T15:55:42.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.841+0000 7fdf91495640 1 --2- 172.21.15.81:0/3170778566 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf8c0d0b90 0x7fdf8c0d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.841+0000 7fdf91495640 1 --2- 172.21.15.81:0/3170778566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf8c0b5250 0x7fdf8c0d1fc0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.841+0000 7fdf91495640 1 -- 172.21.15.81:0/3170778566 >> 172.21.15.81:0/3170778566 conn(0x7fdf8c09cf30 msgr2=0x7fdf8c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:42.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.841+0000 7fdf91495640 1 -- 172.21.15.81:0/3170778566 shutdown_connections 2024-06-29T15:55:42.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.841+0000 7fdf91495640 1 -- 172.21.15.81:0/3170778566 wait complete. 2024-06-29T15:55:42.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.843+0000 7fdf91495640 1 Processor -- start 2024-06-29T15:55:42.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.843+0000 7fdf91495640 1 -- start start 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf91495640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf8c0b5250 0x7fdf8c0b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf91495640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf8c0b9040 0x7fdf8c09e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf91495640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf8c09ea80 0x7fdf8c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf91495640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8c0d2680 con 0x7fdf8c09ea80 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf91495640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8c0d2500 con 0x7fdf8c0b5250 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf91495640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdf8c0d2800 con 0x7fdf8c0b9040 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf8b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf8c09ea80 0x7fdf8c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf8b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf8c09ea80 0x7fdf8c0ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56306/0 (socket says 172.21.15.81:56306) 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf8b7fe640 1 -- 172.21.15.81:0/414678808 learned_addr learned my addr 172.21.15.81:0/414678808 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf8a7fc640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf8c0b9040 0x7fdf8c09e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.844+0000 7fdf8affd640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf8c0b5250 0x7fdf8c0b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf8b7fe640 1 -- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf8c0b9040 msgr2=0x7fdf8c09e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf8b7fe640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf8c0b9040 0x7fdf8c09e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf8b7fe640 1 -- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf8c0b5250 msgr2=0x7fdf8c0b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf8b7fe640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf8c0b5250 0x7fdf8c0b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf8b7fe640 1 -- 172.21.15.81:0/414678808 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8c0b4560 con 0x7fdf8c09ea80 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf8b7fe640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf8c09ea80 0x7fdf8c0ccf30 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fdf84006010 tx=0x7fdf84003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf837fe640 1 -- 172.21.15.81:0/414678808 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8403e030 con 0x7fdf8c09ea80 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf837fe640 1 -- 172.21.15.81:0/414678808 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf84003d80 con 0x7fdf8c09ea80 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf91495640 1 -- 172.21.15.81:0/414678808 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf8c0a0a30 con 0x7fdf8c09ea80 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf837fe640 1 -- 172.21.15.81:0/414678808 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84035040 con 0x7fdf8c09ea80 2024-06-29T15:55:42.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.845+0000 7fdf91495640 1 -- 172.21.15.81:0/414678808 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf8c0a0ff0 con 0x7fdf8c09ea80 2024-06-29T15:55:42.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.846+0000 7fdf91495640 1 -- 172.21.15.81:0/414678808 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf8c0d46a0 con 0x7fdf8c09ea80 2024-06-29T15:55:42.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.847+0000 7fdf837fe640 1 -- 172.21.15.81:0/414678808 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fdf84041030 con 0x7fdf8c09ea80 2024-06-29T15:55:42.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.847+0000 7fdf837fe640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf6c0763f0 0x7fdf6c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:42.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.847+0000 7fdf837fe640 1 -- 172.21.15.81:0/414678808 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7fdf840b8340 con 0x7fdf8c09ea80 2024-06-29T15:55:42.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.848+0000 7fdf8affd640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf6c0763f0 0x7fdf6c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:42.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.848+0000 7fdf8affd640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf6c0763f0 0x7fdf6c078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c001e20 tx=0x7fdf7c009040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:42.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:42.852+0000 7fdf837fe640 1 -- 172.21.15.81:0/414678808 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdf8c0d46a0 con 0x7fdf8c09ea80 2024-06-29T15:55:43.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.103+0000 7fdf91495640 1 -- 172.21.15.81:0/414678808 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fdf8c0d2d90 con 0x7fdf8c09ea80 2024-06-29T15:55:43.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.106+0000 7fdf837fe640 1 -- 172.21.15.81:0/414678808 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v66) ==== 190+0+147 (secure 0 0 0) 0x7fdf8c0b4560 con 0x7fdf8c09ea80 2024-06-29T15:55:43.107 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T15:55:43.107 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T15:55:43.108 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T15:55:43.108 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T15:55:43.108 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:55:43.108 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T15:55:43.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 -- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf6c0763f0 msgr2=0x7fdf6c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf6c0763f0 0x7fdf6c078900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c001e20 tx=0x7fdf7c009040 comp rx=0 tx=0).stop 2024-06-29T15:55:43.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 -- 172.21.15.81:0/414678808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf8c09ea80 msgr2=0x7fdf8c0ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf8c09ea80 0x7fdf8c0ccf30 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fdf84006010 tx=0x7fdf84003890 comp rx=0 tx=0).stop 2024-06-29T15:55:43.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 -- 172.21.15.81:0/414678808 shutdown_connections 2024-06-29T15:55:43.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fdf6c0763f0 0x7fdf6c078900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf8c09ea80 0x7fdf8c0ccf30 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf8c0b9040 0x7fdf8c09e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 --2- 172.21.15.81:0/414678808 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf8c0b5250 0x7fdf8c0b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 -- 172.21.15.81:0/414678808 >> 172.21.15.81:0/414678808 conn(0x7fdf8c09cf30 msgr2=0x7fdf8c06de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:43.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 -- 172.21.15.81:0/414678808 shutdown_connections 2024-06-29T15:55:43.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.108+0000 7fdf91495640 1 -- 172.21.15.81:0/414678808 wait complete. 2024-06-29T15:55:43.121 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:55:43.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.234+0000 7f6fd13be640 1 -- 172.21.15.81:0/3955380984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fcc08c270 msgr2=0x7f6fcc0ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.234+0000 7f6fd13be640 1 --2- 172.21.15.81:0/3955380984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fcc08c270 0x7f6fcc0ce930 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0b55d0 tx=0x7f6fbc02a440 comp rx=0 tx=0).stop 2024-06-29T15:55:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.234+0000 7f6fd13be640 1 -- 172.21.15.81:0/3955380984 shutdown_connections 2024-06-29T15:55:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.234+0000 7f6fd13be640 1 --2- 172.21.15.81:0/3955380984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6fcc0d17b0 0x7f6fcc0d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.234+0000 7f6fd13be640 1 --2- 172.21.15.81:0/3955380984 >> v2:172.21.15.167:3300/0 conn(0x7f6fcc08ccd0 0x7f6fcc0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.234+0000 7f6fd13be640 1 --2- 172.21.15.81:0/3955380984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fcc08c270 0x7f6fcc0ce930 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.234+0000 7f6fd13be640 1 -- 172.21.15.81:0/3955380984 >> 172.21.15.81:0/3955380984 conn(0x7f6fcc0b56d0 msgr2=0x7f6fcc0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:43.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.234+0000 7f6fd13be640 1 -- 172.21.15.81:0/3955380984 shutdown_connections 2024-06-29T15:55:43.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.234+0000 7f6fd13be640 1 -- 172.21.15.81:0/3955380984 wait complete. 2024-06-29T15:55:43.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.235+0000 7f6fd13be640 1 Processor -- start 2024-06-29T15:55:43.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.236+0000 7f6fd13be640 1 -- start start 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fd13be640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fcc08ccd0 0x7f6fcc1e4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fd13be640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6fcc0d17b0 0x7f6fcc09ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fd13be640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6fcc1f5ac0 0x7f6fcc0a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fd13be640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc0d26b0 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fd13be640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc0d2530 con 0x7f6fcc1f5ac0 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fd13be640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6fcc0b3240 con 0x7f6fcc0d17b0 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fcaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fcc08ccd0 0x7f6fcc1e4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fcaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fcc08ccd0 0x7f6fcc1e4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56322/0 (socket says 172.21.15.81:56322) 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fcaffd640 1 -- 172.21.15.81:0/1042276334 learned_addr learned my addr 172.21.15.81:0/1042276334 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fc27fc640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6fcc0d17b0 0x7f6fcc09ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fcaffd640 1 -- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6fcc0d17b0 msgr2=0x7f6fcc09ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fcb7fe640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6fcc1f5ac0 0x7f6fcc0a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:43.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fcaffd640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6fcc0d17b0 0x7f6fcc09ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fcaffd640 1 -- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6fcc1f5ac0 msgr2=0x7f6fcc0a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fcaffd640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6fcc1f5ac0 0x7f6fcc0a0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.237+0000 7f6fcaffd640 1 -- 172.21.15.81:0/1042276334 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc1e6fc0 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.238+0000 7f6fcaffd640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fcc08ccd0 0x7f6fcc1e4140 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc034040 tx=0x7f6fbc0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:43.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.238+0000 7f6fc3fff640 1 -- 172.21.15.81:0/1042276334 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc03e030 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.238+0000 7f6fc3fff640 1 -- 172.21.15.81:0/1042276334 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6fbc0047a0 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.238+0000 7f6fd13be640 1 -- 172.21.15.81:0/1042276334 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fcc1e72a0 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.238+0000 7f6fc3fff640 1 -- 172.21.15.81:0/1042276334 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc035040 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.238+0000 7f6fd13be640 1 -- 172.21.15.81:0/1042276334 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fcc1e7860 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.239+0000 7f6fd13be640 1 -- 172.21.15.81:0/1042276334 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fcc08c270 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.239+0000 7f6fc3fff640 1 -- 172.21.15.81:0/1042276334 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f6fbc041030 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.240+0000 7f6fc3fff640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f6fac0763f0 0x7f6fac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:43.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.240+0000 7f6fc3fff640 1 -- 172.21.15.81:0/1042276334 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f6fbc0b8300 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.240+0000 7f6fc27fc640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f6fac0763f0 0x7f6fac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:43.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.241+0000 7f6fc27fc640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f6fac0763f0 0x7f6fac078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0b51c0 tx=0x7f6fb4008040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:43.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.244+0000 7f6fc3fff640 1 -- 172.21.15.81:0/1042276334 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6fbc081950 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.467+0000 7f6fd13be640 1 -- 172.21.15.81:0/1042276334 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fcc09f280 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.468+0000 7f6fc3fff640 1 -- 172.21.15.81:0/1042276334 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5240 (secure 0 0 0) 0x7f6fbc081250 con 0x7f6fcc08ccd0 2024-06-29T15:55:43.470 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:43.470 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":103,"btime":"2024-06-29T15:55:41:302970+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1764785504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1764785504},{"type":"v1","addr":"172.21.15.167:6836","nonce":1764785504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99},{"gid":10713,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/1699508727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1699508727},{"type":"v1","addr":"172.21.15.81:6834","nonce":1699508727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/265635924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":265635924},{"type":"v1","addr":"172.21.15.81:6835","nonce":265635924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.649055+0000","modified":"2024-06-29T15:55:40.671516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/1700137710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1700137710},{"type":"v1","addr":"172.21.15.167:6837","nonce":1700137710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10731,"qdb_cluster":[10731]},"id":12}]} 2024-06-29T15:55:43.470 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 103 2024-06-29T15:55:43.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 -- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f6fac0763f0 msgr2=0x7f6fac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f6fac0763f0 0x7f6fac078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0b51c0 tx=0x7f6fb4008040 comp rx=0 tx=0).stop 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 -- 172.21.15.81:0/1042276334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fcc08ccd0 msgr2=0x7f6fcc1e4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fcc08ccd0 0x7f6fcc1e4140 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc034040 tx=0x7f6fbc0039e0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 -- 172.21.15.81:0/1042276334 shutdown_connections 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f6fac0763f0 0x7f6fac078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6fcc1f5ac0 0x7f6fcc0a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6fcc0d17b0 0x7f6fcc09ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 --2- 172.21.15.81:0/1042276334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fcc08ccd0 0x7f6fcc1e4140 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 -- 172.21.15.81:0/1042276334 >> 172.21.15.81:0/1042276334 conn(0x7f6fcc0b56d0 msgr2=0x7f6fcc0b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 -- 172.21.15.81:0/1042276334 shutdown_connections 2024-06-29T15:55:43.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:43.470+0000 7f6fd13be640 1 -- 172.21.15.81:0/1042276334 wait complete. 2024-06-29T15:55:43.885 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:55:37.649055+0000', 'modified': '2024-06-29T15:55:40.671516+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10731}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10731': {'gid': 10731, 'name': 'd', 'rank': 0, 'incarnation': 100, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.167:6837/1700137710', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6835', 'nonce': 1700137710}, {'type': 'v1', 'addr': '172.21.15.167:6837', 'nonce': 1700137710}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10731, 'qdb_cluster': [10731]} 2024-06-29T15:55:43.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:55:43.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-06-29T15:55:44.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.002+0000 7f2e874df640 1 -- 172.21.15.81:0/1843342753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e800cefc0 msgr2=0x7f2e80151030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.002+0000 7f2e874df640 1 --2- 172.21.15.81:0/1843342753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e800cefc0 0x7f2e80151030 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f2e80071d20 tx=0x7f2e7002b700 comp rx=0 tx=0).stop 2024-06-29T15:55:44.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.002+0000 7f2e874df640 1 -- 172.21.15.81:0/1843342753 shutdown_connections 2024-06-29T15:55:44.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.002+0000 7f2e874df640 1 --2- 172.21.15.81:0/1843342753 >> v2:172.21.15.167:3300/0 conn(0x7f2e8006f4e0 0x7f2e800cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.002+0000 7f2e874df640 1 --2- 172.21.15.81:0/1843342753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e800cefc0 0x7f2e80151030 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.002+0000 7f2e874df640 1 -- 172.21.15.81:0/1843342753 >> 172.21.15.81:0/1843342753 conn(0x7f2e8006ccc0 msgr2=0x7f2e800b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.002+0000 7f2e874df640 1 -- 172.21.15.81:0/1843342753 shutdown_connections 2024-06-29T15:55:44.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.002+0000 7f2e874df640 1 -- 172.21.15.81:0/1843342753 wait complete. 2024-06-29T15:55:44.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.003+0000 7f2e874df640 1 Processor -- start 2024-06-29T15:55:44.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.004+0000 7f2e874df640 1 -- start start 2024-06-29T15:55:44.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e874df640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e800cefc0 0x7f2e80157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e874df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8015e5f0 0x7f2e800ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e874df640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e80143dd0 0x7f2e800ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e874df640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2e800cdb00 con 0x7f2e8015e5f0 2024-06-29T15:55:44.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e874df640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2e80070220 con 0x7f2e800cefc0 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e874df640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2e800cdc80 con 0x7f2e80143dd0 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e84a7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8015e5f0 0x7f2e800ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e84a7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8015e5f0 0x7f2e800ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56342/0 (socket says 172.21.15.81:56342) 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e84a7a640 1 -- 172.21.15.81:0/3188553678 learned_addr learned my addr 172.21.15.81:0/3188553678 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e8527b640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e800cefc0 0x7f2e80157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e85a7c640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e80143dd0 0x7f2e800ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e84a7a640 1 -- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e80143dd0 msgr2=0x7f2e800ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e84a7a640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e80143dd0 0x7f2e800ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e84a7a640 1 -- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e800cefc0 msgr2=0x7f2e80157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e84a7a640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e800cefc0 0x7f2e80157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e84a7a640 1 -- 172.21.15.81:0/3188553678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e80156970 con 0x7f2e8015e5f0 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e84a7a640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8015e5f0 0x7f2e800ad8d0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400c370 tx=0x7f2e7400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.005+0000 7f2e7dffb640 1 -- 172.21.15.81:0/3188553678 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74017070 con 0x7f2e8015e5f0 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.006+0000 7f2e7dffb640 1 -- 172.21.15.81:0/3188553678 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e740053c0 con 0x7f2e8015e5f0 2024-06-29T15:55:44.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.006+0000 7f2e874df640 1 -- 172.21.15.81:0/3188553678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e80156c50 con 0x7f2e8015e5f0 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.006+0000 7f2e7dffb640 1 -- 172.21.15.81:0/3188553678 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74003850 con 0x7f2e8015e5f0 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.006+0000 7f2e874df640 1 -- 172.21.15.81:0/3188553678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e80157210 con 0x7f2e8015e5f0 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.007+0000 7f2e7dffb640 1 -- 172.21.15.81:0/3188553678 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f2e7401a070 con 0x7f2e8015e5f0 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.008+0000 7f2e7dffb640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2e6c0763f0 0x7f2e6c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.008+0000 7f2e7dffb640 1 -- 172.21.15.81:0/3188553678 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f2e74096cc0 con 0x7f2e8015e5f0 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.008+0000 7f2e8527b640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2e6c0763f0 0x7f2e6c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.009+0000 7f2e8527b640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2e6c0763f0 0x7f2e6c078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e70034000 tx=0x7f2e70004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.009+0000 7f2e8e396740 1 Processor -- start 2024-06-29T15:55:44.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.009+0000 7f2e8e396740 1 -- start start 2024-06-29T15:55:44.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5e6270 0x56473e9d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473e9b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e8e396740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56473e94ebc0 con 0x56473e5bd3b0 2024-06-29T15:55:44.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e8e396740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x56473e8fd630 con 0x56473e5e6270 2024-06-29T15:55:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e8e396740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x56473e8ecbd0 con 0x56473e5c5590 2024-06-29T15:55:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e84a7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473e9b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e84a7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473e9b63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56354/0 (socket says 172.21.15.81:56354) 2024-06-29T15:55:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e84a7a640 1 -- 172.21.15.81:0/4242781816 learned_addr learned my addr 172.21.15.81:0/4242781816 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e8527b640 1 --2- 172.21.15.81:0/4242781816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5e6270 0x56473e9d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e85a7c640 1 --2- 172.21.15.81:0/4242781816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.010+0000 7f2e84a7a640 1 -- 172.21.15.81:0/4242781816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 msgr2=0x56473e9b68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.011+0000 7f2e84a7a640 1 --2- 172.21.15.81:0/4242781816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9b68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.011+0000 7f2e84a7a640 1 -- 172.21.15.81:0/4242781816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5e6270 msgr2=0x56473e9d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.011+0000 7f2e84a7a640 1 --2- 172.21.15.81:0/4242781816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5e6270 0x56473e9d26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.011+0000 7f2e84a7a640 1 -- 172.21.15.81:0/4242781816 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56473e8e9ca0 con 0x56473e5bd3b0 2024-06-29T15:55:44.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.011+0000 7f2e85a7c640 1 --2- 172.21.15.81:0/4242781816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9b68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:44.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.011+0000 7f2e84a7a640 1 --2- 172.21.15.81:0/4242781816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473e9b63a0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400cd30 tx=0x7f2e7405f8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73efa15522d4e0bc server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.011+0000 7f2e66ffd640 1 -- 172.21.15.81:0/4242781816 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74068b70 con 0x56473e5bd3b0 2024-06-29T15:55:44.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.011+0000 7f2e66ffd640 1 -- 172.21.15.81:0/4242781816 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e7405f020 con 0x56473e5bd3b0 2024-06-29T15:55:44.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.011+0000 7f2e66ffd640 1 -- 172.21.15.81:0/4242781816 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7405b730 con 0x56473e5bd3b0 2024-06-29T15:55:44.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.012+0000 7f2e8e396740 1 -- 172.21.15.81:0/4242781816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 msgr2=0x56473e9b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.012+0000 7f2e8e396740 1 --2- 172.21.15.81:0/4242781816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473e9b63a0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400cd30 tx=0x7f2e7405f8d0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.012+0000 7f2e8e396740 1 -- 172.21.15.81:0/4242781816 shutdown_connections 2024-06-29T15:55:44.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.012+0000 7f2e8e396740 1 --2- 172.21.15.81:0/4242781816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9b68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.012+0000 7f2e8e396740 1 --2- 172.21.15.81:0/4242781816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473e9b63a0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.012+0000 7f2e8e396740 1 --2- 172.21.15.81:0/4242781816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5e6270 0x56473e9d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.012+0000 7f2e8e396740 1 -- 172.21.15.81:0/4242781816 >> 172.21.15.81:0/4242781816 conn(0x56473e5da3c0 msgr2=0x56473e989f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.012+0000 7f2e8e396740 1 -- 172.21.15.81:0/4242781816 shutdown_connections 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.012+0000 7f2e8e396740 1 -- 172.21.15.81:0/4242781816 wait complete. 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.012+0000 7f2e8e396740 1 -- start start 2024-06-29T15:55:44.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.013+0000 7f2e8e396740 1 Processor -- start 2024-06-29T15:55:44.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5bd3b0 0x56473e9ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5e6270 0x56473e9cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e8e396740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56473e88aae0 con 0x56473e5e6270 2024-06-29T15:55:44.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e8e396740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x56473e8f0c10 con 0x56473e5bd3b0 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e8e396740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x56473e660620 con 0x56473e5c5590 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e8e396740 10 client.0 _ll_register_callbacks cb 0x56473e897480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e85a7c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5e6270 0x56473e9cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e85a7c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5e6270 0x56473e9cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56366/0 (socket says 172.21.15.81:56366) 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e85a7c640 1 -- 172.21.15.81:0/3801582151 learned_addr learned my addr 172.21.15.81:0/3801582151 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e84a7a640 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e8527b640 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5bd3b0 0x56473e9ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.014+0000 7f2e85a7c640 1 -- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 msgr2=0x56473e9d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e85a7c640 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e85a7c640 1 -- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5bd3b0 msgr2=0x56473e9ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e85a7c640 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5bd3b0 0x56473e9ae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e85a7c640 1 -- 172.21.15.81:0/3801582151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56473e651850 con 0x56473e5e6270 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e85a7c640 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5e6270 0x56473e9cdee0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f2e78010740 tx=0x7f2e78010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7917f7b2484854 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e5ffff640 1 -- 172.21.15.81:0/3801582151 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7800ece0 con 0x56473e5e6270 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e5ffff640 1 -- 172.21.15.81:0/3801582151 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f2e7801bb50 con 0x56473e5e6270 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e8e396740 1 -- 172.21.15.81:0/3801582151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e7405f020 con 0x56473e5e6270 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e5ffff640 1 -- 172.21.15.81:0/3801582151 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f2e7800ee80 con 0x56473e5e6270 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e8e396740 10 client.10857 fetch_fsmap learned FSMap version 103 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e8e396740 1 -- 172.21.15.81:0/3801582151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) -- 0x56473e6c86b0 con 0x56473e5e6270 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e5ffff640 1 -- 172.21.15.81:0/3801582151 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e7405f020 con 0x56473e5e6270 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e5ffff640 1 -- 172.21.15.81:0/3801582151 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 103) ==== 3502+0+0 (secure 0 0 0) 0x7f2e78022050 con 0x56473e5e6270 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e8e396740 10 client.10857 fetch_fsmap finished waiting for FSMap version 103 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e8e396740 10 client.10857 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e8e396740 20 client.10857 populate_metadata read hostname 'smithi081' 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e8e396740 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:6835/1700137710,v1:172.21.15.167:6837/1700137710] conn(0x56473e5a4e40 0x56473e9b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e8e396740 4 client.10857 mds_command: new command op to 10731 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.015+0000 7f2e8e396740 1 -- 172.21.15.81:0/3801582151 --> [v2:172.21.15.167:6835/1700137710,v1:172.21.15.167:6837/1700137710] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56473e8ff580 con 0x56473e5a4e40 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.016+0000 7f2e8527b640 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:6835/1700137710,v1:172.21.15.167:6837/1700137710] conn(0x56473e5a4e40 0x56473e9b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.016+0000 7f2e5ffff640 10 client.10857 ms_handle_connect on v2:172.21.15.167:6835/1700137710 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.016+0000 7f2e8527b640 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:6835/1700137710,v1:172.21.15.167:6837/1700137710] conn(0x56473e5a4e40 0x56473e9b3b10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a8a67cc7f2210ad server_cookie=a4294ce2964ad68d in_seq=0 out_seq=0 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e5ffff640 1 -- 172.21.15.81:0/3801582151 <== mds.0 v2:172.21.15.167:6835/1700137710 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56473e8ff580 con 0x56473e5a4e40 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e5ffff640 10 client.10857 handle_command_reply: tid=0 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 client.10857 shutdown 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 -- 172.21.15.81:0/3801582151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5e6270 msgr2=0x56473e9cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5e6270 0x56473e9cdee0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f2e78010740 tx=0x7f2e78010c00 comp rx=0 tx=0).stop 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 -- 172.21.15.81:0/3801582151 shutdown_connections 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:6835/1700137710,v1:172.21.15.167:6837/1700137710] conn(0x56473e5a4e40 0x56473e9b3b10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5e6270 0x56473e9cdee0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e5ffff640 0 client.10857 ms_handle_reset on v2:172.21.15.167:6835/1700137710 2024-06-29T15:55:44.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 --2- 172.21.15.81:0/3801582151 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5bd3b0 0x56473e9ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 -- 172.21.15.81:0/3801582151 >> 172.21.15.81:0/3801582151 conn(0x56473e5da3c0 msgr2=0x56473e9b2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 -- 172.21.15.81:0/3801582151 shutdown_connections 2024-06-29T15:55:44.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 1 -- 172.21.15.81:0/3801582151 wait complete. 2024-06-29T15:55:44.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.017+0000 7f2e8e396740 20 client.10857 trim_cache size 0 max 16384 2024-06-29T15:55:44.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.036+0000 7f2e8e396740 1 Processor -- start 2024-06-29T15:55:44.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.036+0000 7f2e8e396740 1 -- start start 2024-06-29T15:55:44.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.037+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5a4e40 0x56473e9b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.037+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5bd3b0 0x56473e9ce620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8e396740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2e80201970 con 0x56473e5a4e40 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8e396740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2e80146af0 con 0x56473e5bd3b0 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8e396740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2e801f8ba0 con 0x56473e5c5590 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8527b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5a4e40 0x56473e9b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8527b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5a4e40 0x56473e9b4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56382/0 (socket says 172.21.15.81:56382) 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8527b640 1 -- 172.21.15.81:0/1697427970 learned_addr learned my addr 172.21.15.81:0/1697427970 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8527b640 1 -- 172.21.15.81:0/1697427970 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 msgr2=0x56473e9ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8527b640 1 --2- 172.21.15.81:0/1697427970 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8527b640 1 -- 172.21.15.81:0/1697427970 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5bd3b0 msgr2=0x56473e9ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8527b640 1 --2- 172.21.15.81:0/1697427970 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5bd3b0 0x56473e9ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8527b640 1 -- 172.21.15.81:0/1697427970 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56473e9b21b0 con 0x56473e5a4e40 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e8527b640 1 --2- 172.21.15.81:0/1697427970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5a4e40 0x56473e9b4a90 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009230 tx=0x7f2e7003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f2d5cbf99a92ce server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e667fc640 1 -- 172.21.15.81:0/1697427970 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70036040 con 0x56473e5a4e40 2024-06-29T15:55:44.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e667fc640 1 -- 172.21.15.81:0/1697427970 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e70047040 con 0x56473e5a4e40 2024-06-29T15:55:44.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.038+0000 7f2e667fc640 1 -- 172.21.15.81:0/1697427970 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e700049e0 con 0x56473e5a4e40 2024-06-29T15:55:44.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.039+0000 7f2e8e396740 1 -- 172.21.15.81:0/1697427970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5a4e40 msgr2=0x56473e9b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.039+0000 7f2e8e396740 1 --2- 172.21.15.81:0/1697427970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5a4e40 0x56473e9b4a90 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009230 tx=0x7f2e7003c000 comp rx=0 tx=0).stop 2024-06-29T15:55:44.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.039+0000 7f2e8e396740 1 -- 172.21.15.81:0/1697427970 shutdown_connections 2024-06-29T15:55:44.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.039+0000 7f2e8e396740 1 --2- 172.21.15.81:0/1697427970 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.039+0000 7f2e8e396740 1 --2- 172.21.15.81:0/1697427970 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5bd3b0 0x56473e9ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.039+0000 7f2e8e396740 1 --2- 172.21.15.81:0/1697427970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5a4e40 0x56473e9b4a90 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.039+0000 7f2e8e396740 1 -- 172.21.15.81:0/1697427970 >> 172.21.15.81:0/1697427970 conn(0x56473e5da3c0 msgr2=0x56473e9d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.039+0000 7f2e8e396740 1 -- 172.21.15.81:0/1697427970 shutdown_connections 2024-06-29T15:55:44.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.039+0000 7f2e8e396740 1 -- 172.21.15.81:0/1697427970 wait complete. 2024-06-29T15:55:44.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.040+0000 7f2e8e396740 1 -- start start 2024-06-29T15:55:44.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.040+0000 7f2e8e396740 1 Processor -- start 2024-06-29T15:55:44.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.041+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5a4e40 0x56473ea4d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.041+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473ea4e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.041+0000 7f2e8e396740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9cb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.041+0000 7f2e8e396740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56473e9e1080 con 0x56473e5bd3b0 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.041+0000 7f2e8e396740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2e801464a0 con 0x56473e5a4e40 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.041+0000 7f2e8e396740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x56473e9e0bd0 con 0x56473e5c5590 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.041+0000 7f2e8e396740 10 client.0 _ll_register_callbacks cb 0x7f2e78009630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.041+0000 7f2e84a7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473ea4e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.041+0000 7f2e84a7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473ea4e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56390/0 (socket says 172.21.15.81:56390) 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.041+0000 7f2e84a7a640 1 -- 172.21.15.81:0/1901540816 learned_addr learned my addr 172.21.15.81:0/1901540816 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e84a7a640 1 -- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 msgr2=0x56473e9cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e85a7c640 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e84a7a640 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e8527b640 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5a4e40 0x56473ea4d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e84a7a640 1 -- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5a4e40 msgr2=0x56473ea4d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e84a7a640 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5a4e40 0x56473ea4d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e84a7a640 1 -- 172.21.15.81:0/1901540816 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56473e9ea7f0 con 0x56473e5bd3b0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e84a7a640 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473ea4e710 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400cd30 tx=0x7f2e7405c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678ef31a95ba87a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e5ffff640 1 -- 172.21.15.81:0/1901540816 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74068b70 con 0x56473e5bd3b0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e5ffff640 1 -- 172.21.15.81:0/1901540816 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f2e7405b730 con 0x56473e5bd3b0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e8e396740 1 -- 172.21.15.81:0/1901540816 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e70047040 con 0x56473e5bd3b0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e5ffff640 1 -- 172.21.15.81:0/1901540816 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) ==== 24+0+0 (secure 0 0 0) 0x7f2e7405b930 con 0x56473e5bd3b0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e8e396740 10 client.10863 fetch_fsmap learned FSMap version 103 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e8e396740 1 -- 172.21.15.81:0/1901540816 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) -- 0x56473e9b6300 con 0x56473e5bd3b0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e5ffff640 1 -- 172.21.15.81:0/1901540816 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e70047040 con 0x56473e5bd3b0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e5ffff640 1 -- 172.21.15.81:0/1901540816 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 103) ==== 3502+0+0 (secure 0 0 0) 0x7f2e74081200 con 0x56473e5bd3b0 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e8e396740 10 client.10863 fetch_fsmap finished waiting for FSMap version 103 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e8e396740 10 client.10863 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:55:44.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e8e396740 20 client.10863 populate_metadata read hostname 'smithi081' 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e8e396740 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:6835/1700137710,v1:172.21.15.167:6837/1700137710] conn(0x56473e57a280 0x56473e9ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e8e396740 4 client.10863 mds_command: new command op to 10731 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.042+0000 7f2e8e396740 1 -- 172.21.15.81:0/1901540816 --> [v2:172.21.15.167:6835/1700137710,v1:172.21.15.167:6837/1700137710] -- command(tid 0: {"prefix": "status"}) -- 0x56473e8fb020 con 0x56473e57a280 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.043+0000 7f2e8527b640 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:6835/1700137710,v1:172.21.15.167:6837/1700137710] conn(0x56473e57a280 0x56473e9ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.043+0000 7f2e5ffff640 10 client.10863 ms_handle_connect on v2:172.21.15.167:6835/1700137710 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.043+0000 7f2e8527b640 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:6835/1700137710,v1:172.21.15.167:6837/1700137710] conn(0x56473e57a280 0x56473e9ce7c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6f5673cd6c78c07 server_cookie=cb7d17c355554a3b in_seq=0 out_seq=0 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.043+0000 7f2e5ffff640 1 -- 172.21.15.81:0/1901540816 <== mds.0 v2:172.21.15.167:6835/1700137710 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x56473e8fb020 con 0x56473e57a280 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.043+0000 7f2e5ffff640 10 client.10863 handle_command_reply: tid=0 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.043+0000 7f2e8e396740 1 client.10863 shutdown 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.043+0000 7f2e8e396740 1 -- 172.21.15.81:0/1901540816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 msgr2=0x56473ea4e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.043+0000 7f2e8e396740 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473ea4e710 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400cd30 tx=0x7f2e7405c280 comp rx=0 tx=0).stop 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e8e396740 1 -- 172.21.15.81:0/1901540816 shutdown_connections 2024-06-29T15:55:44.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e8e396740 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:6835/1700137710,v1:172.21.15.167:6837/1700137710] conn(0x56473e57a280 0x56473e9ce7c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e8e396740 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x56473e5c5590 0x56473e9cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e8e396740 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56473e5bd3b0 0x56473ea4e710 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e5ffff640 0 client.10863 ms_handle_reset on v2:172.21.15.167:6835/1700137710 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e8e396740 1 --2- 172.21.15.81:0/1901540816 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x56473e5a4e40 0x56473ea4d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e8e396740 1 -- 172.21.15.81:0/1901540816 >> 172.21.15.81:0/1901540816 conn(0x56473e5da3c0 msgr2=0x56473e9d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e8e396740 1 -- 172.21.15.81:0/1901540816 shutdown_connections 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e8e396740 1 -- 172.21.15.81:0/1901540816 wait complete. 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e8e396740 20 client.10863 trim_cache size 0 max 16384 2024-06-29T15:55:44.053 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 10731, 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.754427089, 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 103, 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 196, 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 196, 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.343285743999999 2024-06-29T15:55:44.054 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 -- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2e6c0763f0 msgr2=0x7f2e6c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2e6c0763f0 0x7f2e6c078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e70034000 tx=0x7f2e70004730 comp rx=0 tx=0).stop 2024-06-29T15:55:44.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 -- 172.21.15.81:0/3188553678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8015e5f0 msgr2=0x7f2e800ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8015e5f0 0x7f2e800ad8d0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400c370 tx=0x7f2e7400c830 comp rx=0 tx=0).stop 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 -- 172.21.15.81:0/3188553678 shutdown_connections 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f2e6c0763f0 0x7f2e6c078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2e80143dd0 0x7f2e800ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8015e5f0 0x7f2e800ad8d0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 --2- 172.21.15.81:0/3188553678 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2e800cefc0 0x7f2e80157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 -- 172.21.15.81:0/3188553678 >> 172.21.15.81:0/3188553678 conn(0x7f2e8006ccc0 msgr2=0x7f2e800b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 -- 172.21.15.81:0/3188553678 shutdown_connections 2024-06-29T15:55:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.044+0000 7f2e874df640 1 -- 172.21.15.81:0/3188553678 wait complete. 2024-06-29T15:55:44.057 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:55:44.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.170+0000 7f375f266640 1 -- 172.21.15.81:0/1664707297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37580d0b90 msgr2=0x7f37580d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.170+0000 7f375f266640 1 --2- 172.21.15.81:0/1664707297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37580d0b90 0x7f37580d1a80 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f3748009830 tx=0x7f374802a640 comp rx=0 tx=0).stop 2024-06-29T15:55:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.170+0000 7f375f266640 1 -- 172.21.15.81:0/1664707297 shutdown_connections 2024-06-29T15:55:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.170+0000 7f375f266640 1 --2- 172.21.15.81:0/1664707297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37580d0b90 0x7f37580d1a80 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.170+0000 7f375f266640 1 -- 172.21.15.81:0/1664707297 >> 172.21.15.81:0/1664707297 conn(0x7f375809cf30 msgr2=0x7f37580d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.170+0000 7f375f266640 1 -- 172.21.15.81:0/1664707297 shutdown_connections 2024-06-29T15:55:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.170+0000 7f375f266640 1 -- 172.21.15.81:0/1664707297 wait complete. 2024-06-29T15:55:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.172+0000 7f375f266640 1 Processor -- start 2024-06-29T15:55:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.172+0000 7f375f266640 1 -- start start 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375f266640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37580b5250 0x7f375809e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375f266640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37580b9040 0x7f37581f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375f266640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37581f5820 0x7f3758158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375f266640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f37580d26d0 con 0x7f37580b5250 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375f266640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f37580b85c0 con 0x7f37581f5820 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375f266640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f37580d2850 con 0x7f37580b9040 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375d002640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37580b5250 0x7f375809e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375d002640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37580b5250 0x7f375809e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56408/0 (socket says 172.21.15.81:56408) 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375d002640 1 -- 172.21.15.81:0/2000667416 learned_addr learned my addr 172.21.15.81:0/2000667416 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375c801640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37580b9040 0x7f37581f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375d803640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37581f5820 0x7f3758158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375d002640 1 -- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37580b9040 msgr2=0x7f37581f52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375d002640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37580b9040 0x7f37581f52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375d002640 1 -- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37581f5820 msgr2=0x7f3758158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375d002640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37581f5820 0x7f3758158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375d002640 1 -- 172.21.15.81:0/2000667416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37581592b0 con 0x7f37580b5250 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.173+0000 7f375d002640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37580b5250 0x7f375809e5a0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f375400c400 tx=0x7f375400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.174+0000 7f374dffb640 1 -- 172.21.15.81:0/2000667416 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754011790 con 0x7f37580b5250 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.174+0000 7f374dffb640 1 -- 172.21.15.81:0/2000667416 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3754017070 con 0x7f37580b5250 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.174+0000 7f375f266640 1 -- 172.21.15.81:0/2000667416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3758159590 con 0x7f37580b5250 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.174+0000 7f374dffb640 1 -- 172.21.15.81:0/2000667416 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754011ce0 con 0x7f37580b5250 2024-06-29T15:55:44.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.174+0000 7f375f266640 1 -- 172.21.15.81:0/2000667416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3758159b50 con 0x7f37580b5250 2024-06-29T15:55:44.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.175+0000 7f375f266640 1 -- 172.21.15.81:0/2000667416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37580d46a0 con 0x7f37580b5250 2024-06-29T15:55:44.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.175+0000 7f374dffb640 1 -- 172.21.15.81:0/2000667416 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f375401c460 con 0x7f37580b5250 2024-06-29T15:55:44.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.176+0000 7f374dffb640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f373c0763f0 0x7f373c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:44.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.176+0000 7f374dffb640 1 -- 172.21.15.81:0/2000667416 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f3754097870 con 0x7f37580b5250 2024-06-29T15:55:44.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.176+0000 7f375c801640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f373c0763f0 0x7f373c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:44.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.177+0000 7f375c801640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f373c0763f0 0x7f373c078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3748034000 tx=0x7f3748004730 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:44.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.180+0000 7f374dffb640 1 -- 172.21.15.81:0/2000667416 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f37580d46a0 con 0x7f37580b5250 2024-06-29T15:55:44.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.399+0000 7f375f266640 1 -- 172.21.15.81:0/2000667416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f37580d2de0 con 0x7f37580b5250 2024-06-29T15:55:44.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.400+0000 7f374dffb640 1 -- 172.21.15.81:0/2000667416 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v103) ==== 53+0+73 (secure 0 0 0) 0x7f3754060fc0 con 0x7f37580b5250 2024-06-29T15:55:44.401 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 -- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f373c0763f0 msgr2=0x7f373c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f373c0763f0 0x7f373c078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3748034000 tx=0x7f3748004730 comp rx=0 tx=0).stop 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 -- 172.21.15.81:0/2000667416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37580b5250 msgr2=0x7f375809e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37580b5250 0x7f375809e5a0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f375400c400 tx=0x7f375400c8c0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 -- 172.21.15.81:0/2000667416 shutdown_connections 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f373c0763f0 0x7f373c078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f37581f5820 0x7f3758158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f37580b9040 0x7f37581f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 --2- 172.21.15.81:0/2000667416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37580b5250 0x7f375809e5a0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 -- 172.21.15.81:0/2000667416 >> 172.21.15.81:0/2000667416 conn(0x7f375809cf30 msgr2=0x7f375815d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 -- 172.21.15.81:0/2000667416 shutdown_connections 2024-06-29T15:55:44.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44.402+0000 7f375f266640 1 -- 172.21.15.81:0/2000667416 wait complete. 2024-06-29T15:55:44.415 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:55:44.415 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:55:44.415 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T15:55:44.415 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:55:44.415 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:55:44.415 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:55:44.415 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:55:44.415 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:55:44.416 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:55:44.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.416 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T15:55:44.432 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:55:44.432 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:55:44.432 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:55:44.432 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.432 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T15:55:44.432 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.432 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:55:44.432 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 37175sec preferred_lft 37175sec 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.433 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:55:44.434 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:55:44.434 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:55:44.434 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.434 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T15:55:44.434 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.434 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:55:44.434 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:55:44.434 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T15:55:44.434 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.435 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:55:44.435 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:55:44.435 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T15:55:44.435 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:55:44.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.435 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:55:44.496 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T15:55:44.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.497 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T15:55:44.554 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T15:55:44.554 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:55:44.554 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T15:55:44.554 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T15:55:44.554 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:55:44.554 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:55:44.555 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T15:55:44.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T15:55:44.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T15:55:44.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T15:55:44.555 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T15:55:44.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:55:44.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:55:44.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.680 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T15:55:44.695 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:55:44.695 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:55:44.696 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:55:44.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.696 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T15:55:44.761 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:55:44.761 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:55:44.816 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:55:44.817 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:55:44.817 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:55:44.873 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T15:55:44.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.873 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:55:44.967 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:55:44.968 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:55:44.976 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:55:44.976 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:55:44.976 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T15:55:44.976 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T15:55:44.976 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T15:55:44.976 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:55:44.976 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:55:44.977 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:55:44.977 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:55:44.977 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:55:44.978 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:55:44.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:44.979 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:55:45.003 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:55:45.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:55:45.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:55:45.108 INFO:teuthology.orchestra.run.smithi081.stdout:client10875 2024-06-29T15:55:45.108 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:55:45.108 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:55:45.124 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:55:45.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.277+0000 7f84685ac640 1 -- 172.21.15.81:0/3617777251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f844c004d40 msgr2=0x7f844c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.277+0000 7f84685ac640 1 --2- 172.21.15.81:0/3617777251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f844c004d40 0x7f844c005150 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f8458009eb0 tx=0x7f844c005830 comp rx=0 tx=0).stop 2024-06-29T15:55:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.277+0000 7f8466b49640 1 -- 172.21.15.81:0/3617777251 reap_dead start 2024-06-29T15:55:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.277+0000 7f84685ac640 1 -- 172.21.15.81:0/3617777251 shutdown_connections 2024-06-29T15:55:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.277+0000 7f84685ac640 1 -- 172.21.15.81:0/3617777251 >> 172.21.15.81:0/3617777251 conn(0x7f84600b3510 msgr2=0x7f84600b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.277+0000 7f84685ac640 1 -- 172.21.15.81:0/3617777251 shutdown_connections 2024-06-29T15:55:45.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.277+0000 7f84685ac640 1 -- 172.21.15.81:0/3617777251 wait complete. 2024-06-29T15:55:45.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.279+0000 7f84685ac640 1 Processor -- start 2024-06-29T15:55:45.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.279+0000 7f84685ac640 1 -- start start 2024-06-29T15:55:45.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.280+0000 7f84685ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f844c004d40 0x7f84601e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.280+0000 7f84685ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84601e5a00 0x7f846009f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.280+0000 7f84685ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84601f2a60 0x7f84601f2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.280+0000 7f84685ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f84600b4ba0 con 0x7f84601e5a00 2024-06-29T15:55:45.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.280+0000 7f84685ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f84600b4a20 con 0x7f844c004d40 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.280+0000 7f84685ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f84600b4d20 con 0x7f84601f2a60 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.280+0000 7f8465b47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84601e5a00 0x7f846009f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8465b47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84601e5a00 0x7f846009f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56426/0 (socket says 172.21.15.81:56426) 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8465b47640 1 -- 172.21.15.81:0/1562127520 learned_addr learned my addr 172.21.15.81:0/1562127520 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8466348640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f844c004d40 0x7f84601e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8466b49640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84601f2a60 0x7f84601f2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8465b47640 1 -- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84601f2a60 msgr2=0x7f84601f2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8465b47640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84601f2a60 0x7f84601f2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8465b47640 1 -- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f844c004d40 msgr2=0x7f84601e73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8465b47640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f844c004d40 0x7f84601e73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8465b47640 1 -- 172.21.15.81:0/1562127520 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84601e5e10 con 0x7f84601e5a00 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8466b49640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84601f2a60 0x7f84601f2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:45.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8466348640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f844c004d40 0x7f84601e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:45.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8465b47640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84601e5a00 0x7f846009f0c0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f8450011a80 tx=0x7f8450011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:45.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8456ffd640 1 -- 172.21.15.81:0/1562127520 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845000fe10 con 0x7f84601e5a00 2024-06-29T15:55:45.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8456ffd640 1 -- 172.21.15.81:0/1562127520 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8450016070 con 0x7f84601e5a00 2024-06-29T15:55:45.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f84685ac640 1 -- 172.21.15.81:0/1562127520 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84601e7bf0 con 0x7f84601e5a00 2024-06-29T15:55:45.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f8456ffd640 1 -- 172.21.15.81:0/1562127520 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8450012780 con 0x7f84601e5a00 2024-06-29T15:55:45.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.281+0000 7f84685ac640 1 -- 172.21.15.81:0/1562127520 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84601e81b0 con 0x7f84601e5a00 2024-06-29T15:55:45.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.282+0000 7f84685ac640 1 -- 172.21.15.81:0/1562127520 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84600b52b0 con 0x7f84601e5a00 2024-06-29T15:55:45.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.283+0000 7f8456ffd640 1 -- 172.21.15.81:0/1562127520 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7f8450018030 con 0x7f84601e5a00 2024-06-29T15:55:45.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.283+0000 7f8456ffd640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f84440763f0 0x7f8444078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.284+0000 7f8456ffd640 1 -- 172.21.15.81:0/1562127520 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f8450098320 con 0x7f84601e5a00 2024-06-29T15:55:45.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.284+0000 7f8466348640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f84440763f0 0x7f8444078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.284+0000 7f8466348640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f84440763f0 0x7f8444078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f845c000ff0 tx=0x7f845c001e30 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:45.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.288+0000 7f8456ffd640 1 -- 172.21.15.81:0/1562127520 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8450061a70 con 0x7f84601e5a00 2024-06-29T15:55:45.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.492+0000 7f84685ac640 1 -- 172.21.15.81:0/1562127520 --> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f84600cdf00 con 0x7f84440763f0 2024-06-29T15:55:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.493+0000 7f8456ffd640 1 -- 172.21.15.81:0/1562127520 <== mgr.8327 v2:172.21.15.167:6832/803000291 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f84600cdf00 con 0x7f84440763f0 2024-06-29T15:55:45.495 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:55:45.495 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:55:45.495 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T15:55:45.495 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:55:45.495 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:55:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.495+0000 7f84685ac640 1 -- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f84440763f0 msgr2=0x7f8444078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.495+0000 7f84685ac640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f84440763f0 0x7f8444078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f845c000ff0 tx=0x7f845c001e30 comp rx=0 tx=0).stop 2024-06-29T15:55:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.495+0000 7f84685ac640 1 -- 172.21.15.81:0/1562127520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84601e5a00 msgr2=0x7f846009f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.495+0000 7f84685ac640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84601e5a00 0x7f846009f0c0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f8450011a80 tx=0x7f8450011f40 comp rx=0 tx=0).stop 2024-06-29T15:55:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.496+0000 7f84685ac640 1 -- 172.21.15.81:0/1562127520 shutdown_connections 2024-06-29T15:55:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.496+0000 7f84685ac640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f84440763f0 0x7f8444078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.496+0000 7f84685ac640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84601f2a60 0x7f84601f2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.496+0000 7f84685ac640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84601e5a00 0x7f846009f0c0 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.496+0000 7f84685ac640 1 --2- 172.21.15.81:0/1562127520 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f844c004d40 0x7f84601e73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.496+0000 7f84685ac640 1 -- 172.21.15.81:0/1562127520 >> 172.21.15.81:0/1562127520 conn(0x7f84600b3510 msgr2=0x7f84600b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.496+0000 7f84685ac640 1 -- 172.21.15.81:0/1562127520 shutdown_connections 2024-06-29T15:55:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.496+0000 7f84685ac640 1 -- 172.21.15.81:0/1562127520 wait complete. 2024-06-29T15:55:45.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:55:45.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.622+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/2338411856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e806fd70 msgr2=0x7fa9e8071f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.622+0000 7fa9ee0ce640 1 --2- 172.21.15.81:0/2338411856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e806fd70 0x7fa9e8071f20 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0005fd0 tx=0x7fa9e002a820 comp rx=0 tx=0).stop 2024-06-29T15:55:45.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.622+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/2338411856 shutdown_connections 2024-06-29T15:55:45.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.622+0000 7fa9ee0ce640 1 --2- 172.21.15.81:0/2338411856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e806fd70 0x7fa9e8071f20 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.622+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/2338411856 >> 172.21.15.81:0/2338411856 conn(0x7fa9e806ccc0 msgr2=0x7fa9e80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:45.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.622+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/2338411856 shutdown_connections 2024-06-29T15:55:45.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.622+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/2338411856 wait complete. 2024-06-29T15:55:45.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.623+0000 7fa9ee0ce640 1 Processor -- start 2024-06-29T15:55:45.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.624+0000 7fa9ee0ce640 1 -- start start 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9ee0ce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e80b3640 0x7fa9e8152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9ee0ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9e815b230 0x7fa9e81f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9ee0ce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa9e81f3140 0x7fa9e81f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9ee0ce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e80725e0 con 0x7fa9e80b3640 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9ee0ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8072460 con 0x7fa9e815b230 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9ee0ce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e8072760 con 0x7fa9e81f3140 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e80b3640 0x7fa9e8152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e80b3640 0x7fa9e8152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56434/0 (socket says 172.21.15.81:56434) 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e77fe640 1 -- 172.21.15.81:0/1263425594 learned_addr learned my addr 172.21.15.81:0/1263425594 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e6ffd640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9e815b230 0x7fa9e81f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e7fff640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa9e81f3140 0x7fa9e81f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e77fe640 1 -- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa9e81f3140 msgr2=0x7fa9e81f3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e77fe640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa9e81f3140 0x7fa9e81f3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e77fe640 1 -- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9e815b230 msgr2=0x7fa9e81f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e77fe640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9e815b230 0x7fa9e81f2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e77fe640 1 -- 172.21.15.81:0/1263425594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e81545a0 con 0x7fa9e80b3640 2024-06-29T15:55:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e6ffd640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9e815b230 0x7fa9e81f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:55:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.625+0000 7fa9e77fe640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e80b3640 0x7fa9e8152b90 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800c3c0 tx=0x7fa9d800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.626+0000 7fa9cffff640 1 -- 172.21.15.81:0/1263425594 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8017070 con 0x7fa9e80b3640 2024-06-29T15:55:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.626+0000 7fa9cffff640 1 -- 172.21.15.81:0/1263425594 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9d80035f0 con 0x7fa9e80b3640 2024-06-29T15:55:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.626+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/1263425594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e8153500 con 0x7fa9e80b3640 2024-06-29T15:55:45.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.626+0000 7fa9cffff640 1 -- 172.21.15.81:0/1263425594 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8003a80 con 0x7fa9e80b3640 2024-06-29T15:55:45.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.626+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/1263425594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e8153ac0 con 0x7fa9e80b3640 2024-06-29T15:55:45.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.627+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/1263425594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e8071f20 con 0x7fa9e80b3640 2024-06-29T15:55:45.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.627+0000 7fa9cffff640 1 -- 172.21.15.81:0/1263425594 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 117) ==== 98303+0+0 (secure 0 0 0) 0x7fa9d801a070 con 0x7fa9e80b3640 2024-06-29T15:55:45.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.628+0000 7fa9cffff640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa9d40763f0 0x7fa9d4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:45.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.628+0000 7fa9cffff640 1 -- 172.21.15.81:0/1263425594 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7fa9d8096bd0 con 0x7fa9e80b3640 2024-06-29T15:55:45.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.628+0000 7fa9e6ffd640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa9d40763f0 0x7fa9d4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:45.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.629+0000 7fa9e6ffd640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa9d40763f0 0x7fa9d4078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc006fd0 tx=0x7fa9dc008040 comp rx=0 tx=0).ready entity=mgr.8327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:45.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.632+0000 7fa9cffff640 1 -- 172.21.15.81:0/1263425594 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa9d805fd60 con 0x7fa9e80b3640 2024-06-29T15:55:45.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:45.860+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/1263425594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fa9e8072190 con 0x7fa9e80b3640 2024-06-29T15:55:46.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.326+0000 7fa9cffff640 1 -- 172.21.15.81:0/1263425594 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v118) ==== 92+0+0 (secure 0 0 0) 0x7fa9d80683a0 con 0x7fa9e80b3640 2024-06-29T15:55:46.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.328+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa9d40763f0 msgr2=0x7fa9d4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.328+0000 7fa9ee0ce640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa9d40763f0 0x7fa9d4078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc006fd0 tx=0x7fa9dc008040 comp rx=0 tx=0).stop 2024-06-29T15:55:46.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.328+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/1263425594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e80b3640 msgr2=0x7fa9e8152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.328+0000 7fa9ee0ce640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e80b3640 0x7fa9e8152b90 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800c3c0 tx=0x7fa9d800c880 comp rx=0 tx=0).stop 2024-06-29T15:55:46.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.329+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/1263425594 shutdown_connections 2024-06-29T15:55:46.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.329+0000 7fa9ee0ce640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7fa9d40763f0 0x7fa9d4078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.329+0000 7fa9ee0ce640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa9e81f3140 0x7fa9e81f3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.329+0000 7fa9ee0ce640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa9e815b230 0x7fa9e81f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.329+0000 7fa9ee0ce640 1 --2- 172.21.15.81:0/1263425594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e80b3640 0x7fa9e8152b90 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.329+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/1263425594 >> 172.21.15.81:0/1263425594 conn(0x7fa9e806ccc0 msgr2=0x7fa9e80cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.330+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/1263425594 shutdown_connections 2024-06-29T15:55:46.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.330+0000 7fa9ee0ce640 1 -- 172.21.15.81:0/1263425594 wait complete. 2024-06-29T15:55:46.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:55:46.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.466+0000 7f8d9593c640 1 -- 172.21.15.81:0/1696024407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d900b3200 msgr2=0x7f8d900cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.466+0000 7f8d9593c640 1 --2- 172.21.15.81:0/1696024407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d900b3200 0x7f8d900cc7b0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f8d90074b10 tx=0x7f8d8802a4d0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.466+0000 7f8d9593c640 1 -- 172.21.15.81:0/1696024407 shutdown_connections 2024-06-29T15:55:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.466+0000 7f8d9593c640 1 --2- 172.21.15.81:0/1696024407 >> v2:172.21.15.167:3300/0 conn(0x7f8d900cccf0 0x7f8d900d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.466+0000 7f8d9593c640 1 --2- 172.21.15.81:0/1696024407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d900b3200 0x7f8d900cc7b0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.466+0000 7f8d9593c640 1 -- 172.21.15.81:0/1696024407 >> 172.21.15.81:0/1696024407 conn(0x7f8d900b3680 msgr2=0x7f8d900b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.466+0000 7f8d9593c640 1 -- 172.21.15.81:0/1696024407 shutdown_connections 2024-06-29T15:55:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.466+0000 7f8d9593c640 1 -- 172.21.15.81:0/1696024407 wait complete. 2024-06-29T15:55:46.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.468+0000 7f8d9593c640 1 Processor -- start 2024-06-29T15:55:46.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.468+0000 7f8d9593c640 1 -- start start 2024-06-29T15:55:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d9593c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8d900b3200 0x7f8d901f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d9593c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8d900cccf0 0x7f8d9006d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d9593c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d900d0770 0x7f8d900cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d9593c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8d900cedb0 con 0x7f8d900d0770 2024-06-29T15:55:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d9593c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8d900cec30 con 0x7f8d900b3200 2024-06-29T15:55:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d9593c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8d900cef30 con 0x7f8d900cccf0 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d8f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d900d0770 0x7f8d900cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d8f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d900d0770 0x7f8d900cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56470/0 (socket says 172.21.15.81:56470) 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d8f7fe640 1 -- 172.21.15.81:0/4274919522 learned_addr learned my addr 172.21.15.81:0/4274919522 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d8effd640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8d900b3200 0x7f8d901f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.469+0000 7f8d8e7fc640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8d900cccf0 0x7f8d9006d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d8f7fe640 1 -- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8d900cccf0 msgr2=0x7f8d9006d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d8f7fe640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8d900cccf0 0x7f8d9006d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d8f7fe640 1 -- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8d900b3200 msgr2=0x7f8d901f5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d8f7fe640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8d900b3200 0x7f8d901f5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d8f7fe640 1 -- 172.21.15.81:0/4274919522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9006dec0 con 0x7f8d900d0770 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d8f7fe640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d900d0770 0x7f8d900cb070 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000b440 tx=0x7f8d8000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d77fff640 1 -- 172.21.15.81:0/4274919522 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80015020 con 0x7f8d900d0770 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d77fff640 1 -- 172.21.15.81:0/4274919522 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d80011920 con 0x7f8d900d0770 2024-06-29T15:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d77fff640 1 -- 172.21.15.81:0/4274919522 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80011d70 con 0x7f8d900d0770 2024-06-29T15:55:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d9593c640 1 -- 172.21.15.81:0/4274919522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9006e1a0 con 0x7f8d900d0770 2024-06-29T15:55:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.470+0000 7f8d9593c640 1 -- 172.21.15.81:0/4274919522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d9006e760 con 0x7f8d900d0770 2024-06-29T15:55:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.471+0000 7f8d9593c640 1 -- 172.21.15.81:0/4274919522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d900cc7b0 con 0x7f8d900d0770 2024-06-29T15:55:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.472+0000 7f8d77fff640 1 -- 172.21.15.81:0/4274919522 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 118) ==== 98320+0+0 (secure 0 0 0) 0x7f8d80018020 con 0x7f8d900d0770 2024-06-29T15:55:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.472+0000 7f8d77fff640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f8d7c076470 0x7f8d7c078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.472+0000 7f8d8effd640 1 -- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f8d7c076470 msgr2=0x7f8d7c078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/803000291 2024-06-29T15:55:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.472+0000 7f8d77fff640 1 -- 172.21.15.81:0/4274919522 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f8d80097540 con 0x7f8d900d0770 2024-06-29T15:55:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.472+0000 7f8d8effd640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f8d7c076470 0x7f8d7c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:46.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.477+0000 7f8d77fff640 1 -- 172.21.15.81:0/4274919522 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8d8001f030 con 0x7f8d900d0770 2024-06-29T15:55:46.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.673+0000 7f8d8effd640 1 -- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f8d7c076470 msgr2=0x7f8d7c078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/803000291 2024-06-29T15:55:46.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.673+0000 7f8d8effd640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f8d7c076470 0x7f8d7c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:46.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.677+0000 7f8d9593c640 1 -- 172.21.15.81:0/4274919522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f8d9006dbe0 con 0x7f8d900d0770 2024-06-29T15:55:46.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.681+0000 7f8d77fff640 1 -- 172.21.15.81:0/4274919522 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v69)=0 v69) ==== 138+0+0 (secure 0 0 0) 0x7f8d80060640 con 0x7f8d900d0770 2024-06-29T15:55:46.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 -- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f8d7c076470 msgr2=0x7f8d7c078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:46.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f8d7c076470 0x7f8d7c078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 -- 172.21.15.81:0/4274919522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d900d0770 msgr2=0x7f8d900cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d900d0770 0x7f8d900cb070 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000b440 tx=0x7f8d8000b900 comp rx=0 tx=0).stop 2024-06-29T15:55:46.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 -- 172.21.15.81:0/4274919522 shutdown_connections 2024-06-29T15:55:46.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f8d7c076470 0x7f8d7c078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d900d0770 0x7f8d900cb070 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8d900cccf0 0x7f8d9006d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 --2- 172.21.15.81:0/4274919522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8d900b3200 0x7f8d901f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 -- 172.21.15.81:0/4274919522 >> 172.21.15.81:0/4274919522 conn(0x7f8d900b3680 msgr2=0x7f8d900cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:46.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 -- 172.21.15.81:0/4274919522 shutdown_connections 2024-06-29T15:55:46.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.683+0000 7f8d9593c640 1 -- 172.21.15.81:0/4274919522 wait complete. 2024-06-29T15:55:46.696 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:55:46.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.814+0000 7f1d744fd640 1 -- 172.21.15.81:0/2321246415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d6c0cc670 msgr2=0x7f1d6c072900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.814+0000 7f1d744fd640 1 --2- 172.21.15.81:0/2321246415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d6c0cc670 0x7f1d6c072900 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0725b0 tx=0x7f1d6802a4d0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.814+0000 7f1d744fd640 1 -- 172.21.15.81:0/2321246415 shutdown_connections 2024-06-29T15:55:46.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.814+0000 7f1d744fd640 1 --2- 172.21.15.81:0/2321246415 >> v2:172.21.15.167:3300/0 conn(0x7f1d6c06d9b0 0x7f1d6c072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.814+0000 7f1d744fd640 1 --2- 172.21.15.81:0/2321246415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d6c0cc670 0x7f1d6c072900 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.814+0000 7f1d744fd640 1 --2- 172.21.15.81:0/2321246415 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d6c06d5a0 0x7f1d6c0cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.814+0000 7f1d744fd640 1 -- 172.21.15.81:0/2321246415 >> 172.21.15.81:0/2321246415 conn(0x7f1d6c0b1c00 msgr2=0x7f1d6c0b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:46.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.814+0000 7f1d744fd640 1 -- 172.21.15.81:0/2321246415 shutdown_connections 2024-06-29T15:55:46.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.814+0000 7f1d744fd640 1 -- 172.21.15.81:0/2321246415 wait complete. 2024-06-29T15:55:46.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.816+0000 7f1d744fd640 1 Processor -- start 2024-06-29T15:55:46.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.816+0000 7f1d744fd640 1 -- start start 2024-06-29T15:55:46.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.817+0000 7f1d744fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d6c06d5a0 0x7f1d6c0cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.817+0000 7f1d744fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d6c06d9b0 0x7f1d6c0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.817+0000 7f1d744fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d6c0cc670 0x7f1d6c0acee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.817+0000 7f1d744fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6c15b150 con 0x7f1d6c06d9b0 2024-06-29T15:55:46.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.817+0000 7f1d744fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6c15afd0 con 0x7f1d6c06d5a0 2024-06-29T15:55:46.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.817+0000 7f1d744fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1d6c15b2d0 con 0x7f1d6c0cc670 2024-06-29T15:55:46.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d71a98640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d6c06d9b0 0x7f1d6c0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d71a98640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d6c06d9b0 0x7f1d6c0cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56482/0 (socket says 172.21.15.81:56482) 2024-06-29T15:55:46.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d71a98640 1 -- 172.21.15.81:0/71564557 learned_addr learned my addr 172.21.15.81:0/71564557 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:46.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d72a9a640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d6c0cc670 0x7f1d6c0acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d72299640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d6c06d5a0 0x7f1d6c0cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:46.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d71a98640 1 -- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d6c0cc670 msgr2=0x7f1d6c0acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d71a98640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d6c0cc670 0x7f1d6c0acee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d71a98640 1 -- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d6c06d5a0 msgr2=0x7f1d6c0cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:46.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d71a98640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d6c06d5a0 0x7f1d6c0cb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:46.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d71a98640 1 -- 172.21.15.81:0/71564557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6c0cbd80 con 0x7f1d6c06d9b0 2024-06-29T15:55:46.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d71a98640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d6c06d9b0 0x7f1d6c0cef20 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f1d680099c0 tx=0x7f1d680039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:46.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d5effd640 1 -- 172.21.15.81:0/71564557 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d6803e030 con 0x7f1d6c06d9b0 2024-06-29T15:55:46.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d5effd640 1 -- 172.21.15.81:0/71564557 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d680047a0 con 0x7f1d6c06d9b0 2024-06-29T15:55:46.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d744fd640 1 -- 172.21.15.81:0/71564557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d6c0b3110 con 0x7f1d6c06d9b0 2024-06-29T15:55:46.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d5effd640 1 -- 172.21.15.81:0/71564557 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d68035040 con 0x7f1d6c06d9b0 2024-06-29T15:55:46.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.818+0000 7f1d744fd640 1 -- 172.21.15.81:0/71564557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d6c0cf460 con 0x7f1d6c06d9b0 2024-06-29T15:55:46.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.819+0000 7f1d744fd640 1 -- 172.21.15.81:0/71564557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d6c0cbaf0 con 0x7f1d6c06d9b0 2024-06-29T15:55:46.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.820+0000 7f1d5effd640 1 -- 172.21.15.81:0/71564557 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 118) ==== 98320+0+0 (secure 0 0 0) 0x7f1d68041030 con 0x7f1d6c06d9b0 2024-06-29T15:55:46.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.821+0000 7f1d5effd640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f1d50076470 0x7f1d50078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:46.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.821+0000 7f1d72299640 1 -- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f1d50076470 msgr2=0x7f1d50078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/803000291 2024-06-29T15:55:46.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.821+0000 7f1d5effd640 1 -- 172.21.15.81:0/71564557 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f1d680b84e0 con 0x7f1d6c06d9b0 2024-06-29T15:55:46.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.821+0000 7f1d72299640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f1d50076470 0x7f1d50078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:46.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:46.829+0000 7f1d5effd640 1 -- 172.21.15.81:0/71564557 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1d680815b0 con 0x7f1d6c06d9b0 2024-06-29T15:55:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.013+0000 7f1d744fd640 1 -- 172.21.15.81:0/71564557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f1d6c0b2f00 con 0x7f1d6c06d9b0 2024-06-29T15:55:47.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.017+0000 7f1d5effd640 1 -- 172.21.15.81:0/71564557 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v70)=0 v70) ==== 133+0+0 (secure 0 0 0) 0x7f1d680858b0 con 0x7f1d6c06d9b0 2024-06-29T15:55:47.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 -- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f1d50076470 msgr2=0x7f1d50078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:47.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f1d50076470 0x7f1d50078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 -- 172.21.15.81:0/71564557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d6c06d9b0 msgr2=0x7f1d6c0cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:47.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d6c06d9b0 0x7f1d6c0cef20 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f1d680099c0 tx=0x7f1d680039e0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 -- 172.21.15.81:0/71564557 shutdown_connections 2024-06-29T15:55:47.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f1d50076470 0x7f1d50078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1d6c0cc670 0x7f1d6c0acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d6c06d9b0 0x7f1d6c0cef20 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 --2- 172.21.15.81:0/71564557 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1d6c06d5a0 0x7f1d6c0cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 -- 172.21.15.81:0/71564557 >> 172.21.15.81:0/71564557 conn(0x7f1d6c0b1c00 msgr2=0x7f1d6c1511a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:47.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 -- 172.21.15.81:0/71564557 shutdown_connections 2024-06-29T15:55:47.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.019+0000 7f1d744fd640 1 -- 172.21.15.81:0/71564557 wait complete. 2024-06-29T15:55:47.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:47.032 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T15:55:47.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-06-29T15:55:47.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.208+0000 7f954763c640 1 -- 172.21.15.81:0/2744940790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95400b4f40 msgr2=0x7f95400b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:47.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.208+0000 7f954763c640 1 --2- 172.21.15.81:0/2744940790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95400b4f40 0x7f95400b5350 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f9540074b10 tx=0x7f953802aa30 comp rx=0 tx=0).stop 2024-06-29T15:55:47.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.208+0000 7f954763c640 1 -- 172.21.15.81:0/2744940790 shutdown_connections 2024-06-29T15:55:47.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.208+0000 7f954763c640 1 --2- 172.21.15.81:0/2744940790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95400b4f40 0x7f95400b5350 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.208+0000 7f954763c640 1 --2- 172.21.15.81:0/2744940790 >> v2:172.21.15.167:3300/0 conn(0x7f95400d0230 0x7f95400d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.208+0000 7f954763c640 1 -- 172.21.15.81:0/2744940790 >> 172.21.15.81:0/2744940790 conn(0x7f95400b3680 msgr2=0x7f95400b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:47.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.208+0000 7f954763c640 1 -- 172.21.15.81:0/2744940790 shutdown_connections 2024-06-29T15:55:47.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.208+0000 7f954763c640 1 -- 172.21.15.81:0/2744940790 wait complete. 2024-06-29T15:55:47.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.210+0000 7f954763c640 1 Processor -- start 2024-06-29T15:55:47.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.210+0000 7f954763c640 1 -- start start 2024-06-29T15:55:47.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.211+0000 7f954763c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95400b3200 0x7f95400c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:47.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.211+0000 7f954763c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95400b4f40 0x7f9540148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:47.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.211+0000 7f954763c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95400d0230 0x7f95401e34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:47.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.211+0000 7f954763c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f95400b5c20 con 0x7f95400d0230 2024-06-29T15:55:47.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.211+0000 7f954763c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f95400b5aa0 con 0x7f95400b3200 2024-06-29T15:55:47.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.211+0000 7f954763c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f95400b5da0 con 0x7f95400b4f40 2024-06-29T15:55:47.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.211+0000 7f9545bd9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95400d0230 0x7f95401e34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:47.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.211+0000 7f9545bd9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95400d0230 0x7f95401e34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56502/0 (socket says 172.21.15.81:56502) 2024-06-29T15:55:47.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.211+0000 7f9545bd9640 1 -- 172.21.15.81:0/3651064099 learned_addr learned my addr 172.21.15.81:0/3651064099 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f9544bd7640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95400b4f40 0x7f9540148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f95453d8640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95400b3200 0x7f95400c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f9545bd9640 1 -- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95400b4f40 msgr2=0x7f9540148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f9545bd9640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95400b4f40 0x7f9540148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f9545bd9640 1 -- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95400b3200 msgr2=0x7f95400c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f9545bd9640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95400b3200 0x7f95400c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f9545bd9640 1 -- 172.21.15.81:0/3651064099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95401e4a90 con 0x7f95400d0230 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f9545bd9640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95400d0230 0x7f95401e34e0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f953802b020 tx=0x7f9538003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9538033050 con 0x7f95400d0230 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9538004450 con 0x7f95400d0230 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f954763c640 1 -- 172.21.15.81:0/3651064099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95401e4d30 con 0x7f95400d0230 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9538038450 con 0x7f95400d0230 2024-06-29T15:55:47.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.212+0000 7f954763c640 1 -- 172.21.15.81:0/3651064099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95401e52f0 con 0x7f95400d0230 2024-06-29T15:55:47.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.214+0000 7f954763c640 1 -- 172.21.15.81:0/3651064099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95400cc2b0 con 0x7f95400d0230 2024-06-29T15:55:47.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.214+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 118) ==== 98320+0+0 (secure 0 0 0) 0x7f9538036070 con 0x7f95400d0230 2024-06-29T15:55:47.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.215+0000 7f9535ffb640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 0x7f9524078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:47.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.215+0000 7f95453d8640 1 -- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 msgr2=0x7f9524078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/803000291 2024-06-29T15:55:47.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.215+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 5809+0+0 (secure 0 0 0) 0x7f95380b9010 con 0x7f95400d0230 2024-06-29T15:55:47.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.215+0000 7f95453d8640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 0x7f9524078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:55:47.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.223+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f95400cc2b0 con 0x7f95400d0230 2024-06-29T15:55:47.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.416+0000 7f95453d8640 1 -- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 msgr2=0x7f9524078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/803000291 2024-06-29T15:55:47.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.416+0000 7f95453d8640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 0x7f9524078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:55:47.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.432+0000 7f954763c640 1 -- 172.21.15.81:0/3651064099 --> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f95400ca180 con 0x7f9524076440 2024-06-29T15:55:47.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.817+0000 7f95453d8640 1 -- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 msgr2=0x7f9524078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/803000291 2024-06-29T15:55:47.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:47.817+0000 7f95453d8640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 0x7f9524078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:55:48.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:48.618+0000 7f95453d8640 1 -- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 msgr2=0x7f9524078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/803000291 2024-06-29T15:55:48.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:48.618+0000 7f95453d8640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 0x7f9524078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:55:49.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:49.966+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 119) ==== 97929+0+0 (secure 0 0 0) 0x7f953802d070 con 0x7f95400d0230 2024-06-29T15:55:49.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:49.967+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 msgr2=0x7f9524078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:55:49.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:49.967+0000 7f9535ffb640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/803000291,v1:172.21.15.167:6833/803000291] conn(0x7f9524076440 0x7f9524078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:50.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:50.109+0000 7fcf31b77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:50.109+0000 7fcf31b77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:50.109+0000 7fcf31b77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:50.109+0000 7fcf31b77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:50.109+0000 7fcf31b77640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:50.109+0000 7fcf2eb71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:50.109+0000 7fcf2eb71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:50.109+0000 7fcf2eb71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:50.109+0000 7fcf2eb71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.110 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:55:50.109+0000 7fcf2eb71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:55:50.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:50.971+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 120) ==== 98056+0+0 (secure 0 0 0) 0x7f95380b74c0 con 0x7f95400d0230 2024-06-29T15:55:50.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:50.971+0000 7f9535ffb640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f952403afd0 0x7f9524080490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:50.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:50.971+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 --> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f95380020b0 con 0x7f952403afd0 2024-06-29T15:55:50.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:50.972+0000 7f95453d8640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f952403afd0 0x7f9524080490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:50.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:50.978+0000 7f95453d8640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f952403afd0 0x7f9524080490 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9530000f60 tx=0x7f9530007db0 comp rx=0 tx=0).ready entity=mgr.9787 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:51.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.045+0000 7f9535ffb640 1 -- 172.21.15.81:0/3651064099 <== mgr.9787 v2:172.21.15.167:6832/23077685 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f95380020b0 con 0x7f952403afd0 2024-06-29T15:55:51.047 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.048+0000 7f954763c640 1 -- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f952403afd0 msgr2=0x7f9524080490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.048+0000 7f954763c640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f952403afd0 0x7f9524080490 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9530000f60 tx=0x7f9530007db0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.048+0000 7f954763c640 1 -- 172.21.15.81:0/3651064099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95400d0230 msgr2=0x7f95401e34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.048+0000 7f954763c640 1 --2- 172.21.15.81:0/3651064099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95400d0230 0x7f95401e34e0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f953802b020 tx=0x7f9538003f90 comp rx=0 tx=0).stop 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.048+0000 7f9545bd9640 1 -- 172.21.15.81:0/3651064099 reap_dead start 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.048+0000 7f954763c640 1 -- 172.21.15.81:0/3651064099 shutdown_connections 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.048+0000 7f954763c640 1 -- 172.21.15.81:0/3651064099 >> 172.21.15.81:0/3651064099 conn(0x7f95400b3680 msgr2=0x7f95400cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.048+0000 7f954763c640 1 -- 172.21.15.81:0/3651064099 shutdown_connections 2024-06-29T15:55:51.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.048+0000 7f954763c640 1 -- 172.21.15.81:0/3651064099 wait complete. 2024-06-29T15:55:51.061 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T15:55:51.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-06-29T15:55:51.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.177+0000 7fb53cdd4640 1 -- 172.21.15.81:0/2110850166 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb53806f760 msgr2=0x7fb53806fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.177+0000 7fb53cdd4640 1 --2- 172.21.15.81:0/2110850166 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb53806f760 0x7fb53806fb70 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb520009830 tx=0x7fb52002a4b0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.177+0000 7fb53cdd4640 1 -- 172.21.15.81:0/2110850166 shutdown_connections 2024-06-29T15:55:51.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.177+0000 7fb53cdd4640 1 --2- 172.21.15.81:0/2110850166 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb53806f760 0x7fb53806fb70 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.177+0000 7fb53cdd4640 1 -- 172.21.15.81:0/2110850166 >> 172.21.15.81:0/2110850166 conn(0x7fb53806cd60 msgr2=0x7fb5380b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:51.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.177+0000 7fb53cdd4640 1 -- 172.21.15.81:0/2110850166 shutdown_connections 2024-06-29T15:55:51.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.177+0000 7fb53cdd4640 1 -- 172.21.15.81:0/2110850166 wait complete. 2024-06-29T15:55:51.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.179+0000 7fb53cdd4640 1 Processor -- start 2024-06-29T15:55:51.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.179+0000 7fb53cdd4640 1 -- start start 2024-06-29T15:55:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53cdd4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53806f760 0x7fb53813c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53cdd4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5380cdbf0 0x7fb53815da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53cdd4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5381415e0 0x7fb53815dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53cdd4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb5380ce890 con 0x7fb53806f760 2024-06-29T15:55:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53cdd4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb5380ce710 con 0x7fb5380cdbf0 2024-06-29T15:55:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53cdd4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb5380cea10 con 0x7fb5381415e0 2024-06-29T15:55:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53806f760 0x7fb53813c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53806f760 0x7fb53813c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52158/0 (socket says 172.21.15.81:52158) 2024-06-29T15:55:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53659c640 1 -- 172.21.15.81:0/3168200180 learned_addr learned my addr 172.21.15.81:0/3168200180 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb52dd9b640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5380cdbf0 0x7fb53815da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb536d9d640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5381415e0 0x7fb53815dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53659c640 1 -- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5381415e0 msgr2=0x7fb53815dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53659c640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5381415e0 0x7fb53815dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53659c640 1 -- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5380cdbf0 msgr2=0x7fb53815da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.180+0000 7fb53659c640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5380cdbf0 0x7fb53815da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.181+0000 7fb53659c640 1 -- 172.21.15.81:0/3168200180 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53813e1a0 con 0x7fb53806f760 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.181+0000 7fb536d9d640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5381415e0 0x7fb53815dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.181+0000 7fb53659c640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53806f760 0x7fb53813c7e0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e6a0 tx=0x7fb52800eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.181+0000 7fb52f7fe640 1 -- 172.21.15.81:0/3168200180 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5280052d0 con 0x7fb53806f760 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.181+0000 7fb52f7fe640 1 -- 172.21.15.81:0/3168200180 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb52800c850 con 0x7fb53806f760 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.181+0000 7fb53cdd4640 1 -- 172.21.15.81:0/3168200180 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5380a9560 con 0x7fb53806f760 2024-06-29T15:55:51.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.181+0000 7fb52f7fe640 1 -- 172.21.15.81:0/3168200180 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52800cca0 con 0x7fb53806f760 2024-06-29T15:55:51.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.181+0000 7fb53cdd4640 1 -- 172.21.15.81:0/3168200180 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5380a9b20 con 0x7fb53806f760 2024-06-29T15:55:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.182+0000 7fb53cdd4640 1 -- 172.21.15.81:0/3168200180 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb53806fb70 con 0x7fb53806f760 2024-06-29T15:55:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.183+0000 7fb52f7fe640 1 -- 172.21.15.81:0/3168200180 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 120) ==== 98056+0+0 (secure 0 0 0) 0x7fb5280035f0 con 0x7fb53806f760 2024-06-29T15:55:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.183+0000 7fb52f7fe640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fb518076170 0x7fb518078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.183+0000 7fb52f7fe640 1 -- 172.21.15.81:0/3168200180 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 6153+0+0 (secure 0 0 0) 0x7fb528099970 con 0x7fb53806f760 2024-06-29T15:55:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.183+0000 7fb52dd9b640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fb518076170 0x7fb518078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.184+0000 7fb52dd9b640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fb518076170 0x7fb518078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb520034000 tx=0x7fb520004730 comp rx=0 tx=0).ready entity=mgr.9787 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:51.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.188+0000 7fb52f7fe640 1 -- 172.21.15.81:0/3168200180 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb528016020 con 0x7fb53806f760 2024-06-29T15:55:51.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.390+0000 7fb53cdd4640 1 -- 172.21.15.81:0/3168200180 --> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb5380b1c20 con 0x7fb518076170 2024-06-29T15:55:51.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.448+0000 7fb52f7fe640 1 -- 172.21.15.81:0/3168200180 <== mgr.9787 v2:172.21.15.167:6832/23077685 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+39 (secure 0 0 0) 0x7fb5380b1c20 con 0x7fb518076170 2024-06-29T15:55:51.450 INFO:teuthology.orchestra.run.smithi081.stdout:Retention added to path snap_test_dir1 2024-06-29T15:55:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 -- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fb518076170 msgr2=0x7fb518078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fb518076170 0x7fb518078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb520034000 tx=0x7fb520004730 comp rx=0 tx=0).stop 2024-06-29T15:55:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 -- 172.21.15.81:0/3168200180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53806f760 msgr2=0x7fb53813c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53806f760 0x7fb53813c7e0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fb52800e6a0 tx=0x7fb52800eb60 comp rx=0 tx=0).stop 2024-06-29T15:55:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 -- 172.21.15.81:0/3168200180 shutdown_connections 2024-06-29T15:55:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fb518076170 0x7fb518078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5381415e0 0x7fb53815dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5380cdbf0 0x7fb53815da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 --2- 172.21.15.81:0/3168200180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53806f760 0x7fb53813c7e0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 -- 172.21.15.81:0/3168200180 >> 172.21.15.81:0/3168200180 conn(0x7fb53806cd60 msgr2=0x7fb5380ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 -- 172.21.15.81:0/3168200180 shutdown_connections 2024-06-29T15:55:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.451+0000 7fb53cdd4640 1 -- 172.21.15.81:0/3168200180 wait complete. 2024-06-29T15:55:51.464 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-06-29T15:55:51.464 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-06-29-15_56_00 in ~9s... 2024-06-29T15:55:51.464 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-06-29T15:55:51.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-06-29T15:55:51.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.577+0000 7f2627c64640 1 -- 172.21.15.81:0/3401396446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f262006ef90 msgr2=0x7f262006f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.577+0000 7f2627c64640 1 --2- 172.21.15.81:0/3401396446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f262006ef90 0x7f262006f3a0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f26200b53d0 tx=0x7f261002a540 comp rx=0 tx=0).stop 2024-06-29T15:55:51.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.577+0000 7f2627c64640 1 -- 172.21.15.81:0/3401396446 shutdown_connections 2024-06-29T15:55:51.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.577+0000 7f2627c64640 1 --2- 172.21.15.81:0/3401396446 >> v2:172.21.15.167:3300/0 conn(0x7f26200ce1d0 0x7f26200ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.577+0000 7f2627c64640 1 --2- 172.21.15.81:0/3401396446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f262006ef90 0x7f262006f3a0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.577+0000 7f2627c64640 1 --2- 172.21.15.81:0/3401396446 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26200b57b0 0x7f262015aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.577+0000 7f2627c64640 1 -- 172.21.15.81:0/3401396446 >> 172.21.15.81:0/3401396446 conn(0x7f26200d0930 msgr2=0x7f26200d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:51.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.577+0000 7f2627c64640 1 -- 172.21.15.81:0/3401396446 shutdown_connections 2024-06-29T15:55:51.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.577+0000 7f2627c64640 1 -- 172.21.15.81:0/3401396446 wait complete. 2024-06-29T15:55:51.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.579+0000 7f2627c64640 1 Processor -- start 2024-06-29T15:55:51.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.579+0000 7f2627c64640 1 -- start start 2024-06-29T15:55:51.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.580+0000 7f2627c64640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f262006ef90 0x7f26200ab6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.580+0000 7f2627c64640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26200b57b0 0x7f26200c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.580+0000 7f2627c64640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26200ce1d0 0x7f262006c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.580+0000 7f2627c64640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f26200ceeb0 con 0x7f262006ef90 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.580+0000 7f2627c64640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f26200ced30 con 0x7f26200b57b0 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.580+0000 7f2627c64640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f26200cf030 con 0x7f26200ce1d0 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2625a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f262006ef90 0x7f26200ab6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2625a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f262006ef90 0x7f26200ab6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52172/0 (socket says 172.21.15.81:52172) 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2625a00640 1 -- 172.21.15.81:0/2325889352 learned_addr learned my addr 172.21.15.81:0/2325889352 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f26251ff640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26200b57b0 0x7f26200c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2626201640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26200ce1d0 0x7f262006c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2625a00640 1 -- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26200ce1d0 msgr2=0x7f262006c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2625a00640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26200ce1d0 0x7f262006c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2625a00640 1 -- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26200b57b0 msgr2=0x7f26200c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2625a00640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26200b57b0 0x7f26200c5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2625a00640 1 -- 172.21.15.81:0/2325889352 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26200abfd0 con 0x7f262006ef90 2024-06-29T15:55:51.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2625a00640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f262006ef90 0x7f26200ab6d0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f261c00cce0 tx=0x7f261c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:51.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f26167fc640 1 -- 172.21.15.81:0/2325889352 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c00f450 con 0x7f262006ef90 2024-06-29T15:55:51.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f26167fc640 1 -- 172.21.15.81:0/2325889352 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f261c00f6e0 con 0x7f262006ef90 2024-06-29T15:55:51.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2627c64640 1 -- 172.21.15.81:0/2325889352 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26200ac2b0 con 0x7f262006ef90 2024-06-29T15:55:51.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f26167fc640 1 -- 172.21.15.81:0/2325889352 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c00fb30 con 0x7f262006ef90 2024-06-29T15:55:51.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.581+0000 7f2627c64640 1 -- 172.21.15.81:0/2325889352 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26200acd70 con 0x7f262006ef90 2024-06-29T15:55:51.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.582+0000 7f2627c64640 1 -- 172.21.15.81:0/2325889352 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f262006e2a0 con 0x7f262006ef90 2024-06-29T15:55:51.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.583+0000 7f26167fc640 1 -- 172.21.15.81:0/2325889352 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 120) ==== 98056+0+0 (secure 0 0 0) 0x7f261c00fcd0 con 0x7f262006ef90 2024-06-29T15:55:51.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.583+0000 7f26167fc640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f2604076170 0x7f2604078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:55:51.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.584+0000 7f26167fc640 1 -- 172.21.15.81:0/2325889352 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(197..197 src has 1..197) ==== 6153+0+0 (secure 0 0 0) 0x7f261c098130 con 0x7f262006ef90 2024-06-29T15:55:51.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.584+0000 7f26251ff640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f2604076170 0x7f2604078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:55:51.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.584+0000 7f26251ff640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f2604076170 0x7f2604078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2610033000 tx=0x7f2610004730 comp rx=0 tx=0).ready entity=mgr.9787 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:55:51.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.588+0000 7f26167fc640 1 -- 172.21.15.81:0/2325889352 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f262006e2a0 con 0x7f262006ef90 2024-06-29T15:55:51.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.796+0000 7f2627c64640 1 -- 172.21.15.81:0/2325889352 --> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f262015aed0 con 0x7f2604076170 2024-06-29T15:55:51.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.803+0000 7f26167fc640 1 -- 172.21.15.81:0/2325889352 <== mgr.9787 v2:172.21.15.167:6832/23077685 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+72 (secure 0 0 0) 0x7f262015aed0 con 0x7f2604076170 2024-06-29T15:55:51.805 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:55:51.805 INFO:teuthology.orchestra.run.smithi081.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-06-29T15:55:51.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 -- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f2604076170 msgr2=0x7f2604078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f2604076170 0x7f2604078680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2610033000 tx=0x7f2610004730 comp rx=0 tx=0).stop 2024-06-29T15:55:51.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 -- 172.21.15.81:0/2325889352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f262006ef90 msgr2=0x7f26200ab6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:55:51.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f262006ef90 0x7f26200ab6d0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f261c00cce0 tx=0x7f261c00a620 comp rx=0 tx=0).stop 2024-06-29T15:55:51.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 -- 172.21.15.81:0/2325889352 shutdown_connections 2024-06-29T15:55:51.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f2604076170 0x7f2604078680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f26200ce1d0 0x7f262006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f26200b57b0 0x7f26200c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 --2- 172.21.15.81:0/2325889352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f262006ef90 0x7f26200ab6d0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:55:51.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 -- 172.21.15.81:0/2325889352 >> 172.21.15.81:0/2325889352 conn(0x7f26200d0930 msgr2=0x7f26200b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:55:51.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 -- 172.21.15.81:0/2325889352 shutdown_connections 2024-06-29T15:55:51.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:55:51.805+0000 7f2627c64640 1 -- 172.21.15.81:0/2325889352 wait complete. 2024-06-29T15:55:51.818 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-06-29T15:55:51.819 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-06-29T15:55:51.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:51.819 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:51.835 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:51.835 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:51.836 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:52.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:52.836 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:52.857 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:52.857 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:52.857 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:53.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:53.858 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:53.878 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:53.879 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:53.879 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:54.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:54.880 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:54.900 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:54.900 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:54.900 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:55.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:55.901 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:55.922 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:55.922 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:55.922 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:56.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:56.923 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:56.943 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:56.943 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:56.943 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:57.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:57.944 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:57.965 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:57.965 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:57.965 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:58.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:58.966 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:55:58.986 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:55:58.986 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:55:58.986 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:55:59.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:55:59.988 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:00.008 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2024-06-29T15:56:00.008 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:00.008 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:00.461 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:56:00.461+0000 7fcf20b55640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:56:00.462 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:56:00.461+0000 7fcf20b55640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:56:00.462 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:56:00.461+0000 7fcf20b55640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:56:00.462 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:56:00.461+0000 7fcf20b55640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:56:00.462 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:56:00.461+0000 7fcf20b55640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:56:01.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:01.009 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:01.030 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:01.030 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:01.030 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:01.030 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:01.030 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2024-06-29-15_56_00_UTC'] 2024-06-29T15:56:01.030 DEBUG:tasks.cephfs.test_snap_schedules:exec=1719676551.4642165, now = 1719676561.030872, timo = 9 2024-06-29T15:56:02.031 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2024-06-29-15_56_00 in ~69s... 2024-06-29T15:56:02.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:02.032 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:02.049 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:02.049 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:02.049 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:02.050 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:03.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:03.051 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:03.071 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:03.071 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:03.071 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:03.071 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:03.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:56:03.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:56:04.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:04.072 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:04.091 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:04.092 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:04.092 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:04.092 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:05.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:05.094 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:05.114 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:05.114 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:05.114 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:05.114 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:06.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:06.115 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:06.134 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:06.135 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:06.135 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:06.135 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:07.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:07.136 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:07.156 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:07.157 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:07.157 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:07.157 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:08.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:08.158 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:08.178 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:08.179 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:08.179 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:08.179 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:09.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:09.180 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:09.200 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:09.200 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:09.200 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:09.200 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:10.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:10.202 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:10.223 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:10.223 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:10.223 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:10.223 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:11.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:11.224 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:11.244 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:11.245 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:11.245 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:11.245 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:12.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:12.247 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:12.266 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:12.267 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:12.267 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:12.267 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:13.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:13.268 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:13.289 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:13.289 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:13.289 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:13.289 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:14.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:14.291 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:14.310 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:14.311 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:14.311 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:14.311 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:15.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:15.312 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:15.332 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:15.333 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:15.333 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:15.333 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:16.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:16.335 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:16.354 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:16.355 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:16.355 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:16.355 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:17.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:17.356 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:17.375 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:17.376 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:17.376 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:17.376 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:18.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:18.377 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:18.397 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:18.398 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:18.398 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:18.398 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:19.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:19.399 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:19.418 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:19.419 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:19.420 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:19.420 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:20.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:20.421 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:20.440 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:20.441 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:20.442 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:20.442 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:21.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:21.443 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:21.463 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:21.464 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:21.464 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:21.464 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:22.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:22.465 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:22.486 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:22.486 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:22.486 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:22.486 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:23.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:23.487 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:23.506 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:23.506 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:23.506 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:23.506 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:24.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:24.507 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:24.526 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:24.527 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:24.527 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:24.527 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:25.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:25.529 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:25.549 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:25.550 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:25.550 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:25.550 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:26.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:26.552 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:26.571 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:26.572 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:26.572 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:26.572 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:27.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:27.574 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:27.593 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:27.594 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:27.594 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:27.595 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:28.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:28.595 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:28.615 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:28.616 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:28.616 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:28.616 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:29.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:29.618 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:29.637 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:29.638 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:29.638 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:29.638 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:30.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:30.639 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:30.660 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:30.660 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:30.660 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:30.660 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:31.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:31.662 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:31.682 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:31.683 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:31.683 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:31.683 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:32.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:32.684 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:32.705 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:32.706 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:32.706 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:32.706 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:33.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:56:33.447 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:56:33.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:33.707 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:33.727 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:33.727 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:33.727 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:33.727 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:34.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:34.729 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:34.749 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:34.749 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:34.749 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:34.749 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:35.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:35.750 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:35.771 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:35.772 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:35.772 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:35.772 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:36.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:36.773 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:36.793 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:36.794 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:36.794 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:36.794 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:37.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:37.795 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:37.815 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:37.816 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:37.816 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:37.816 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:38.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:38.817 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:38.836 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:38.837 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:38.837 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:38.837 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:39.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:39.838 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:39.857 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:39.858 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:39.858 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:39.858 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:40.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:40.859 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:40.879 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:40.880 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:40.880 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:40.880 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:41.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:41.881 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:41.901 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:41.902 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:41.902 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:41.902 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:42.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:42.904 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:42.924 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:42.924 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:42.925 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:42.925 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:43.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:43.925 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:43.946 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:43.946 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:43.946 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:43.946 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:44.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:44.947 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:44.968 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:44.968 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:44.968 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:44.968 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:45.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:45.970 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:45.990 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:45.991 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:45.991 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:45.991 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:46.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:46.992 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:47.012 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:47.013 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:47.013 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:47.013 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:48.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:48.014 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:48.034 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:48.035 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:48.035 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:48.035 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:49.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:49.036 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:49.056 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:49.057 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:49.057 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:49.057 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:50.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:50.058 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:50.079 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:50.079 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:50.079 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:50.079 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:51.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:51.080 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:51.101 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:51.102 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:51.102 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:51.103 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:52.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:52.104 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:52.124 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:52.124 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:52.124 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:52.124 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:53.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:53.125 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:53.144 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:53.145 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:53.145 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:53.145 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:54.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:54.146 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:54.165 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:54.166 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:54.166 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:54.166 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:55.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:55.167 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:55.187 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:55.188 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:55.189 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:55.189 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:56.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:56.190 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:56.209 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:56.210 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:56.211 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:56.211 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:57.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:57.212 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:57.232 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:57.233 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:57.233 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:57.233 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:58.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:58.234 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:58.253 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:58.254 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:58.254 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:58.254 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:56:59.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:56:59.256 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:56:59.275 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:56:59.276 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:56:59.276 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:56:59.276 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:57:00.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:00.277 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:57:00.297 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_56_00_UTC 2024-06-29T15:57:00.297 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:57:00.297 INFO:tasks.cephfs.test_snap_schedules:added: set() 2024-06-29T15:57:00.298 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2024-06-29T15:57:01.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:01.299 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:57:01.319 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_57_00_UTC 2024-06-29T15:57:01.319 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2024-06-29-15_57_00_UTC'} 2024-06-29T15:57:01.319 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2024-06-29-15_57_00_UTC'} 2024-06-29T15:57:01.319 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2024-06-29-15_56_00_UTC'} 2024-06-29T15:57:01.319 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2024-06-29-15_56_00_UTC'] 2024-06-29T15:57:01.320 DEBUG:tasks.cephfs.test_snap_schedules:exec=1719676551.4642165, now = 1719676621.3200345, timo = 69 2024-06-29T15:57:02.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-06-29T15:57:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.450+0000 7f01741eb640 1 -- 172.21.15.81:0/3639825368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158003a00 msgr2=0x7f0158003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.450+0000 7f01741eb640 1 --2- 172.21.15.81:0/3639825368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158003a00 0x7f0158003e50 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f016800baf0 tx=0x7f0158005830 comp rx=0 tx=0).stop 2024-06-29T15:57:02.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.450+0000 7f01741eb640 1 -- 172.21.15.81:0/3639825368 shutdown_connections 2024-06-29T15:57:02.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.450+0000 7f01741eb640 1 --2- 172.21.15.81:0/3639825368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158003a00 0x7f0158003e50 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.450+0000 7f01741eb640 1 -- 172.21.15.81:0/3639825368 >> 172.21.15.81:0/3639825368 conn(0x7f016c06ccc0 msgr2=0x7f016c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:02.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.451+0000 7f01741eb640 1 -- 172.21.15.81:0/3639825368 shutdown_connections 2024-06-29T15:57:02.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.451+0000 7f01741eb640 1 -- 172.21.15.81:0/3639825368 wait complete. 2024-06-29T15:57:02.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.452+0000 7f01741eb640 1 Processor -- start 2024-06-29T15:57:02.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.452+0000 7f01741eb640 1 -- start start 2024-06-29T15:57:02.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f01741eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158003a00 0x7f016c0cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:02.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f01741eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f016c1f5720 0x7f016c1f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:02.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f01741eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f016c1f1a30 0x7f016c143dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:02.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f01741eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f016c0731b0 con 0x7f0158003a00 2024-06-29T15:57:02.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f01741eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f016c073030 con 0x7f016c1f1a30 2024-06-29T15:57:02.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f01741eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f016c073330 con 0x7f016c1f5720 2024-06-29T15:57:02.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0171f87640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158003a00 0x7f016c0cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:02.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0171f87640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158003a00 0x7f016c0cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43590/0 (socket says 172.21.15.81:43590) 2024-06-29T15:57:02.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0171f87640 1 -- 172.21.15.81:0/2979984571 learned_addr learned my addr 172.21.15.81:0/2979984571 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0171786640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f016c1f5720 0x7f016c1f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0172788640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f016c1f1a30 0x7f016c143dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0171f87640 1 -- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f016c1f5720 msgr2=0x7f016c1f14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0171f87640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f016c1f5720 0x7f016c1f14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0171f87640 1 -- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f016c1f1a30 msgr2=0x7f016c143dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0171f87640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f016c1f1a30 0x7f016c143dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0171f87640 1 -- 172.21.15.81:0/2979984571 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f016c1f4c60 con 0x7f0158003a00 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f0171f87640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158003a00 0x7f016c0cc930 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f016800ba40 tx=0x7f0168032770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.454+0000 7f01627fc640 1 -- 172.21.15.81:0/2979984571 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168032d70 con 0x7f0158003a00 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.455+0000 7f01627fc640 1 -- 172.21.15.81:0/2979984571 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0168004030 con 0x7f0158003a00 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.455+0000 7f01741eb640 1 -- 172.21.15.81:0/2979984571 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f016c1f2200 con 0x7f0158003a00 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.455+0000 7f01627fc640 1 -- 172.21.15.81:0/2979984571 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168004460 con 0x7f0158003a00 2024-06-29T15:57:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.455+0000 7f01741eb640 1 -- 172.21.15.81:0/2979984571 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f016c0abf60 con 0x7f0158003a00 2024-06-29T15:57:02.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.456+0000 7f01741eb640 1 -- 172.21.15.81:0/2979984571 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f016c073890 con 0x7f0158003a00 2024-06-29T15:57:02.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.456+0000 7f01627fc640 1 -- 172.21.15.81:0/2979984571 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 123) ==== 98434+0+0 (secure 0 0 0) 0x7f01680035f0 con 0x7f0158003a00 2024-06-29T15:57:02.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.457+0000 7f01627fc640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f0150076560 0x7f0150078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:02.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.457+0000 7f01627fc640 1 -- 172.21.15.81:0/2979984571 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(199..199 src has 1..199) ==== 6211+0+0 (secure 0 0 0) 0x7f01680b8d60 con 0x7f0158003a00 2024-06-29T15:57:02.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.457+0000 7f0171786640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f0150076560 0x7f0150078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:02.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.458+0000 7f0171786640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f0150076560 0x7f0150078a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f016c0c4ce0 tx=0x7f015c006e00 comp rx=0 tx=0).ready entity=mgr.9787 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:02.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.461+0000 7f01627fc640 1 -- 172.21.15.81:0/2979984571 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f016804b060 con 0x7f0158003a00 2024-06-29T15:57:02.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.666+0000 7f01741eb640 1 -- 172.21.15.81:0/2979984571 --> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f016c0cce70 con 0x7f0150076560 2024-06-29T15:57:02.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.743+0000 7f01627fc640 1 -- 172.21.15.81:0/2979984571 <== mgr.9787 v2:172.21.15.167:6832/23077685 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f016c0cce70 con 0x7f0150076560 2024-06-29T15:57:02.743 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path snap_test_dir1 2024-06-29T15:57:02.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 -- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f0150076560 msgr2=0x7f0150078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f0150076560 0x7f0150078a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f016c0c4ce0 tx=0x7f015c006e00 comp rx=0 tx=0).stop 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 -- 172.21.15.81:0/2979984571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158003a00 msgr2=0x7f016c0cc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158003a00 0x7f016c0cc930 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f016800ba40 tx=0x7f0168032770 comp rx=0 tx=0).stop 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 -- 172.21.15.81:0/2979984571 shutdown_connections 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f016c1f1a30 0x7f016c143dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f0150076560 0x7f0150078a70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f016c1f5720 0x7f016c1f14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 --2- 172.21.15.81:0/2979984571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158003a00 0x7f016c0cc930 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 -- 172.21.15.81:0/2979984571 >> 172.21.15.81:0/2979984571 conn(0x7f016c06ccc0 msgr2=0x7f016c0b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 -- 172.21.15.81:0/2979984571 shutdown_connections 2024-06-29T15:57:02.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:02.745+0000 7f01741eb640 1 -- 172.21.15.81:0/2979984571 wait complete. 2024-06-29T15:57:02.756 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-06-29T15:57:02.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:02.756 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:57:02.773 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-15_57_00_UTC 2024-06-29T15:57:02.773 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2024-06-29-15_57_00_UTC 2024-06-29T15:57:02.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:02.773 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2024-06-29-15_57_00_UTC) 2024-06-29T15:57:02.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:02.834 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T15:57:02.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:57:03.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.049+0000 7f295081f640 1 -- 172.21.15.81:0/643172836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f294c15cd00 msgr2=0x7f294c15d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:03.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.049+0000 7f295081f640 1 --2- 172.21.15.81:0/643172836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f294c15cd00 0x7f294c15d510 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2944005fd0 tx=0x7f294402a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:03.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.049+0000 7f295081f640 1 -- 172.21.15.81:0/643172836 shutdown_connections 2024-06-29T15:57:03.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.049+0000 7f295081f640 1 --2- 172.21.15.81:0/643172836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f294c15cd00 0x7f294c15d510 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:03.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.049+0000 7f295081f640 1 --2- 172.21.15.81:0/643172836 >> v2:172.21.15.167:3300/0 conn(0x7f294c145530 0x7f294c073280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:03.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.049+0000 7f295081f640 1 -- 172.21.15.81:0/643172836 >> 172.21.15.81:0/643172836 conn(0x7f294c06ccc0 msgr2=0x7f294c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:03.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.049+0000 7f295081f640 1 -- 172.21.15.81:0/643172836 shutdown_connections 2024-06-29T15:57:03.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.049+0000 7f295081f640 1 -- 172.21.15.81:0/643172836 wait complete. 2024-06-29T15:57:03.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.051+0000 7f295081f640 1 Processor -- start 2024-06-29T15:57:03.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.051+0000 7f295081f640 1 -- start start 2024-06-29T15:57:03.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f295081f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0737c0 0x7f294c13eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f295081f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f294c145530 0x7f294c13fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f295081f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f294c15cd00 0x7f294c13b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f295081f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f294c15dde0 con 0x7f294c0737c0 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f295081f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f294c15dc60 con 0x7f294c15cd00 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f295081f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f294c15df60 con 0x7f294c145530 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f294a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0737c0 0x7f294c13eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f294a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0737c0 0x7f294c13eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43598/0 (socket says 172.21.15.81:43598) 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f294a59c640 1 -- 172.21.15.81:0/1612249923 learned_addr learned my addr 172.21.15.81:0/1612249923 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f294ad9d640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f294c15cd00 0x7f294c13b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.052+0000 7f2943fff640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f294c145530 0x7f294c13fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f294a59c640 1 -- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f294c145530 msgr2=0x7f294c13fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f294a59c640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f294c145530 0x7f294c13fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:03.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f294a59c640 1 -- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f294c15cd00 msgr2=0x7f294c13b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:03.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f294a59c640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f294c15cd00 0x7f294c13b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:03.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f294a59c640 1 -- 172.21.15.81:0/1612249923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c13c920 con 0x7f294c0737c0 2024-06-29T15:57:03.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f2943fff640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f294c145530 0x7f294c13fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:03.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f294a59c640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0737c0 0x7f294c13eb50 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f293c00e9c0 tx=0x7f293c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:03.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f2942ffd640 1 -- 172.21.15.81:0/1612249923 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c00cd80 con 0x7f294c0737c0 2024-06-29T15:57:03.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f2942ffd640 1 -- 172.21.15.81:0/1612249923 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f293c004020 con 0x7f294c0737c0 2024-06-29T15:57:03.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f295081f640 1 -- 172.21.15.81:0/1612249923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294c1577a0 con 0x7f294c0737c0 2024-06-29T15:57:03.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f2942ffd640 1 -- 172.21.15.81:0/1612249923 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c004450 con 0x7f294c0737c0 2024-06-29T15:57:03.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.053+0000 7f295081f640 1 -- 172.21.15.81:0/1612249923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c157a60 con 0x7f294c0737c0 2024-06-29T15:57:03.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.054+0000 7f295081f640 1 -- 172.21.15.81:0/1612249923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f294c15e4f0 con 0x7f294c0737c0 2024-06-29T15:57:03.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.055+0000 7f2942ffd640 1 -- 172.21.15.81:0/1612249923 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 123) ==== 98434+0+0 (secure 0 0 0) 0x7f293c0035f0 con 0x7f294c0737c0 2024-06-29T15:57:03.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.055+0000 7f2942ffd640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f292c076560 0x7f292c078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:03.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.055+0000 7f2942ffd640 1 -- 172.21.15.81:0/1612249923 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(199..199 src has 1..199) ==== 6211+0+0 (secure 0 0 0) 0x7f293c098840 con 0x7f294c0737c0 2024-06-29T15:57:03.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.055+0000 7f2943fff640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f292c076560 0x7f292c078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:03.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.056+0000 7f2943fff640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f292c076560 0x7f292c078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f294c157730 tx=0x7f2934009040 comp rx=0 tx=0).ready entity=mgr.9787 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:03.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.060+0000 7f2942ffd640 1 -- 172.21.15.81:0/1612249923 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f293c061ab0 con 0x7f294c0737c0 2024-06-29T15:57:03.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:03.287+0000 7f295081f640 1 -- 172.21.15.81:0/1612249923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f294c06a3e0 con 0x7f294c0737c0 2024-06-29T15:57:03.588 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:57:03.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:57:04.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.182+0000 7f2942ffd640 1 -- 172.21.15.81:0/1612249923 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v124) ==== 93+0+0 (secure 0 0 0) 0x7f293c0613b0 con 0x7f294c0737c0 2024-06-29T15:57:04.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.186+0000 7f295081f640 1 -- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f292c076560 msgr2=0x7f292c078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.186+0000 7f295081f640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f292c076560 0x7f292c078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f294c157730 tx=0x7f2934009040 comp rx=0 tx=0).stop 2024-06-29T15:57:04.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.186+0000 7f295081f640 1 -- 172.21.15.81:0/1612249923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0737c0 msgr2=0x7f294c13eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.186+0000 7f295081f640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0737c0 0x7f294c13eb50 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f293c00e9c0 tx=0x7f293c00ee80 comp rx=0 tx=0).stop 2024-06-29T15:57:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.186+0000 7f295081f640 1 -- 172.21.15.81:0/1612249923 shutdown_connections 2024-06-29T15:57:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.186+0000 7f295081f640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f294c15cd00 0x7f294c13b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.186+0000 7f295081f640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f292c076560 0x7f292c078a70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.186+0000 7f295081f640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f294c145530 0x7f294c13fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.186+0000 7f295081f640 1 --2- 172.21.15.81:0/1612249923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0737c0 0x7f294c13eb50 secure :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f293c00e9c0 tx=0x7f293c00ee80 comp rx=0 tx=0).stop 2024-06-29T15:57:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.186+0000 7f295081f640 1 -- 172.21.15.81:0/1612249923 >> 172.21.15.81:0/1612249923 conn(0x7f294c06ccc0 msgr2=0x7f294c0b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.190+0000 7f295081f640 1 -- 172.21.15.81:0/1612249923 shutdown_connections 2024-06-29T15:57:04.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.191+0000 7f295081f640 1 -- 172.21.15.81:0/1612249923 wait complete. 2024-06-29T15:57:04.204 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:57:04.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:04.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:04.297 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:57:04.297 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:04.297 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:57:04.297 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:04.315 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:57:04.315 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:04.371 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:04.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.525+0000 7fd850622640 1 -- 172.21.15.81:0/3679314355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8480b5f50 msgr2=0x7fd848166870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.525+0000 7fd850622640 1 --2- 172.21.15.81:0/3679314355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8480b5f50 0x7fd848166870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd840009be0 tx=0x7fd84002a920 comp rx=0 tx=0).stop 2024-06-29T15:57:04.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.526+0000 7fd850622640 1 -- 172.21.15.81:0/3679314355 shutdown_connections 2024-06-29T15:57:04.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.526+0000 7fd850622640 1 --2- 172.21.15.81:0/3679314355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8480b5f50 0x7fd848166870 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.526+0000 7fd850622640 1 -- 172.21.15.81:0/3679314355 >> 172.21.15.81:0/3679314355 conn(0x7fd8480b3680 msgr2=0x7fd8480b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:04.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.526+0000 7fd850622640 1 -- 172.21.15.81:0/3679314355 shutdown_connections 2024-06-29T15:57:04.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.526+0000 7fd850622640 1 -- 172.21.15.81:0/3679314355 wait complete. 2024-06-29T15:57:04.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.527+0000 7fd850622640 1 Processor -- start 2024-06-29T15:57:04.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.527+0000 7fd850622640 1 -- start start 2024-06-29T15:57:04.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd850622640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8480b3200 0x7fd848150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd850622640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8480b5f50 0x7fd848155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd850622640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8480d0230 0x7fd8481557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd850622640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd848167360 con 0x7fd8480d0230 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd850622640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd8481671e0 con 0x7fd8480b5f50 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd850622640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd8481674e0 con 0x7fd8480b3200 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84ebbf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8480d0230 0x7fd8481557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84e3be640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8480b3200 0x7fd848150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84dbbd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8480b5f50 0x7fd848155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84ebbf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8480d0230 0x7fd8481557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43608/0 (socket says 172.21.15.81:43608) 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84ebbf640 1 -- 172.21.15.81:0/285516398 learned_addr learned my addr 172.21.15.81:0/285516398 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84dbbd640 1 -- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8480b3200 msgr2=0x7fd848150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84dbbd640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8480b3200 0x7fd848150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84dbbd640 1 -- 172.21.15.81:0/285516398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8480d0230 msgr2=0x7fd8481557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84dbbd640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8480d0230 0x7fd8481557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84dbbd640 1 -- 172.21.15.81:0/285516398 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8480c9840 con 0x7fd8480b5f50 2024-06-29T15:57:04.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84e3be640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8480b3200 0x7fd848150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:04.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.529+0000 7fd84ebbf640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8480d0230 0x7fd8481557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:04.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.530+0000 7fd84dbbd640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8480b5f50 0x7fd848155280 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd84400c3f0 tx=0x7fd84400c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:04.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.530+0000 7fd83effd640 1 -- 172.21.15.81:0/285516398 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd844017070 con 0x7fd8480b5f50 2024-06-29T15:57:04.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.530+0000 7fd83effd640 1 -- 172.21.15.81:0/285516398 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd844005400 con 0x7fd8480b5f50 2024-06-29T15:57:04.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.530+0000 7fd83effd640 1 -- 172.21.15.81:0/285516398 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd844003850 con 0x7fd8480b5f50 2024-06-29T15:57:04.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.530+0000 7fd850622640 1 -- 172.21.15.81:0/285516398 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8480c9b20 con 0x7fd8480b5f50 2024-06-29T15:57:04.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.530+0000 7fd850622640 1 -- 172.21.15.81:0/285516398 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd848150b40 con 0x7fd8480b5f50 2024-06-29T15:57:04.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.531+0000 7fd850622640 1 -- 172.21.15.81:0/285516398 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8480cc7b0 con 0x7fd8480b5f50 2024-06-29T15:57:04.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.531+0000 7fd83effd640 1 -- 172.21.15.81:0/285516398 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7fd84401a070 con 0x7fd8480b5f50 2024-06-29T15:57:04.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.532+0000 7fd83effd640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fd82c076510 0x7fd82c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.532+0000 7fd84e3be640 1 -- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fd82c076510 msgr2=0x7fd82c078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:04.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.532+0000 7fd83effd640 1 -- 172.21.15.81:0/285516398 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 6299+0+0 (secure 0 0 0) 0x7fd8440970d0 con 0x7fd8480b5f50 2024-06-29T15:57:04.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.532+0000 7fd84e3be640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fd82c076510 0x7fd82c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:04.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.537+0000 7fd83effd640 1 -- 172.21.15.81:0/285516398 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd84405fdd0 con 0x7fd8480b5f50 2024-06-29T15:57:04.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.725+0000 7fd850622640 1 -- 172.21.15.81:0/285516398 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd84806ebf0 con 0x7fd8480b5f50 2024-06-29T15:57:04.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.727+0000 7fd83effd640 1 -- 172.21.15.81:0/285516398 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v200) ==== 74+0+17533 (secure 0 0 0) 0x7fd8440640d0 con 0x7fd8480b5f50 2024-06-29T15:57:04.727 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:04.727 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":200,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:04.172443+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.947285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.605625+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"200","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":200,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.1a","mappings":[{"from":0,"to":2}]},{"pgid":"25.3f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/803000291":"2024-06-30T15:55:49.959292+0000","172.21.15.167:6832/803000291":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/1482340295":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/2216474049":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/3153968292":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/1095374241":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/3639047578":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/2467432753":"2024-06-30T15:55:49.959292+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 -- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fd82c076510 msgr2=0x7fd82c078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fd82c076510 0x7fd82c078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 -- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8480b5f50 msgr2=0x7fd848155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8480b5f50 0x7fd848155280 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd84400c3f0 tx=0x7fd84400c8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 -- 172.21.15.81:0/285516398 shutdown_connections 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8480d0230 0x7fd8481557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8480b5f50 0x7fd848155280 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fd82c076510 0x7fd82c078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 --2- 172.21.15.81:0/285516398 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8480b3200 0x7fd848150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 -- 172.21.15.81:0/285516398 >> 172.21.15.81:0/285516398 conn(0x7fd8480b3680 msgr2=0x7fd848070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:04.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 -- 172.21.15.81:0/285516398 shutdown_connections 2024-06-29T15:57:04.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.729+0000 7fd850622640 1 -- 172.21.15.81:0/285516398 wait complete. 2024-06-29T15:57:04.742 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:55:35.947285+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:55:36.605625+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '200', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 200, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:04.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.860+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/1219079868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a0159bd0 msgr2=0x7fa4a00d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.860+0000 7fa4a6bcb640 1 --2- 172.21.15.81:0/1219079868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a0159bd0 0x7fa4a00d0010 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fa494005fd0 tx=0x7fa49402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.860+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/1219079868 shutdown_connections 2024-06-29T15:57:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.860+0000 7fa4a6bcb640 1 --2- 172.21.15.81:0/1219079868 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a00d0550 0x7fa4a006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.861+0000 7fa4a6bcb640 1 --2- 172.21.15.81:0/1219079868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a0159bd0 0x7fa4a00d0010 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.861+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/1219079868 >> 172.21.15.81:0/1219079868 conn(0x7fa4a00b2920 msgr2=0x7fa4a00b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.861+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/1219079868 shutdown_connections 2024-06-29T15:57:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.861+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/1219079868 wait complete. 2024-06-29T15:57:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.862+0000 7fa4a6bcb640 1 Processor -- start 2024-06-29T15:57:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.862+0000 7fa4a6bcb640 1 -- start start 2024-06-29T15:57:04.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a6bcb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a006c830 0x7fa4a015b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a6bcb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00d0550 0x7fa4a013b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a6bcb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a0159bd0 0x7fa4a013b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a6bcb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a015fc10 con 0x7fa4a00d0550 2024-06-29T15:57:04.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a6bcb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a015fa90 con 0x7fa4a006c830 2024-06-29T15:57:04.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a6bcb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a015fd90 con 0x7fa4a0159bd0 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa49ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00d0550 0x7fa4a013b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa49ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00d0550 0x7fa4a013b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43622/0 (socket says 172.21.15.81:43622) 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a4967640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a006c830 0x7fa4a015b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa49ffff640 1 -- 172.21.15.81:0/3620685901 learned_addr learned my addr 172.21.15.81:0/3620685901 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a5168640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a0159bd0 0x7fa4a013b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a4967640 1 -- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a0159bd0 msgr2=0x7fa4a013b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a4967640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a0159bd0 0x7fa4a013b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a4967640 1 -- 172.21.15.81:0/3620685901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00d0550 msgr2=0x7fa4a013b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a4967640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00d0550 0x7fa4a013b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a4967640 1 -- 172.21.15.81:0/3620685901 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a0141990 con 0x7fa4a006c830 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa49ffff640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00d0550 0x7fa4a013b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.864+0000 7fa4a4967640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a006c830 0x7fa4a015b260 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa494031000 tx=0x7fa494005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:04.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.865+0000 7fa49d7fa640 1 -- 172.21.15.81:0/3620685901 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49403e030 con 0x7fa4a006c830 2024-06-29T15:57:04.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.865+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/3620685901 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a0141c70 con 0x7fa4a006c830 2024-06-29T15:57:04.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.865+0000 7fa49d7fa640 1 -- 172.21.15.81:0/3620685901 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa494003a10 con 0x7fa4a006c830 2024-06-29T15:57:04.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.865+0000 7fa49d7fa640 1 -- 172.21.15.81:0/3620685901 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa494035040 con 0x7fa4a006c830 2024-06-29T15:57:04.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.865+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/3620685901 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a0142230 con 0x7fa4a006c830 2024-06-29T15:57:04.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.866+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/3620685901 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a0160240 con 0x7fa4a006c830 2024-06-29T15:57:04.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.867+0000 7fa49d7fa640 1 -- 172.21.15.81:0/3620685901 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7fa494041030 con 0x7fa4a006c830 2024-06-29T15:57:04.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.867+0000 7fa49d7fa640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fa484076510 0x7fa484078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:04.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.867+0000 7fa49ffff640 1 -- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fa484076510 msgr2=0x7fa484078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:04.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.867+0000 7fa49d7fa640 1 -- 172.21.15.81:0/3620685901 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 6299+0+0 (secure 0 0 0) 0x7fa4940b82d0 con 0x7fa4a006c830 2024-06-29T15:57:04.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.867+0000 7fa49ffff640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fa484076510 0x7fa484078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:04.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:04.871+0000 7fa49d7fa640 1 -- 172.21.15.81:0/3620685901 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa494080fe0 con 0x7fa4a006c830 2024-06-29T15:57:05.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.058+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/3620685901 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa4a006f330 con 0x7fa4a006c830 2024-06-29T15:57:05.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.058+0000 7fa49d7fa640 1 -- 172.21.15.81:0/3620685901 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v200) ==== 93+0+10 (secure 0 0 0) 0x7fa4940852e0 con 0x7fa4a006c830 2024-06-29T15:57:05.059 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:57:05.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fa484076510 msgr2=0x7fa484078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:05.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fa484076510 0x7fa484078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a006c830 msgr2=0x7fa4a015b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a006c830 0x7fa4a015b260 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa494031000 tx=0x7fa494005770 comp rx=0 tx=0).stop 2024-06-29T15:57:05.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/3620685901 shutdown_connections 2024-06-29T15:57:05.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fa484076510 0x7fa484078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4a0159bd0 0x7fa4a013b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4a00d0550 0x7fa4a013b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 --2- 172.21.15.81:0/3620685901 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4a006c830 0x7fa4a015b260 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/3620685901 >> 172.21.15.81:0/3620685901 conn(0x7fa4a00b2920 msgr2=0x7fa4a006f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/3620685901 shutdown_connections 2024-06-29T15:57:05.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.061+0000 7fa4a6bcb640 1 -- 172.21.15.81:0/3620685901 wait complete. 2024-06-29T15:57:05.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:57:05.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.196+0000 7f3ee53f5640 1 -- 172.21.15.81:0/3787573748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ecc004800 msgr2=0x7f3ecc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.196+0000 7f3ee53f5640 1 --2- 172.21.15.81:0/3787573748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ecc004800 0x7f3ecc004c10 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8006fa0 tx=0x7f3ecc005830 comp rx=0 tx=0).stop 2024-06-29T15:57:05.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.197+0000 7f3ee53f5640 1 -- 172.21.15.81:0/3787573748 shutdown_connections 2024-06-29T15:57:05.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.197+0000 7f3ee53f5640 1 --2- 172.21.15.81:0/3787573748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ecc004800 0x7f3ecc004c10 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.197+0000 7f3ee53f5640 1 -- 172.21.15.81:0/3787573748 >> 172.21.15.81:0/3787573748 conn(0x7f3ee00b1ec0 msgr2=0x7f3ee00b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.197+0000 7f3ee53f5640 1 -- 172.21.15.81:0/3787573748 shutdown_connections 2024-06-29T15:57:05.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.197+0000 7f3ee53f5640 1 -- 172.21.15.81:0/3787573748 wait complete. 2024-06-29T15:57:05.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.198+0000 7f3ee53f5640 1 Processor -- start 2024-06-29T15:57:05.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.198+0000 7f3ee53f5640 1 -- start start 2024-06-29T15:57:05.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3ee53f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ecc004800 0x7f3ee014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3ee53f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ee015d3e0 0x7f3ee0159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3ee53f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ee015a100 0x7f3ee015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3ee53f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee00ad3d0 con 0x7f3ecc004800 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3ee53f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee00ae470 con 0x7f3ee015a100 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3ee53f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3ee00ad550 con 0x7f3ee015d3e0 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3edeffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ecc004800 0x7f3ee014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3edeffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ecc004800 0x7f3ee014eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43634/0 (socket says 172.21.15.81:43634) 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3edeffd640 1 -- 172.21.15.81:0/4092571984 learned_addr learned my addr 172.21.15.81:0/4092571984 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3edf7fe640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ee015a100 0x7f3ee015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3ed7fff640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ee015d3e0 0x7f3ee0159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3edeffd640 1 -- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ee015d3e0 msgr2=0x7f3ee0159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3edeffd640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ee015d3e0 0x7f3ee0159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3edeffd640 1 -- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ee015a100 msgr2=0x7f3ee015b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3edeffd640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ee015a100 0x7f3ee015b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3edeffd640 1 -- 172.21.15.81:0/4092571984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee015e420 con 0x7f3ecc004800 2024-06-29T15:57:05.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3edeffd640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ecc004800 0x7f3ee014eb30 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0011b10 tx=0x7f3ed0009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:05.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3ed77fe640 1 -- 172.21.15.81:0/4092571984 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed000fc80 con 0x7f3ecc004800 2024-06-29T15:57:05.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.200+0000 7f3ed77fe640 1 -- 172.21.15.81:0/4092571984 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ed000fe20 con 0x7f3ecc004800 2024-06-29T15:57:05.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.201+0000 7f3ed77fe640 1 -- 172.21.15.81:0/4092571984 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0004280 con 0x7f3ecc004800 2024-06-29T15:57:05.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.201+0000 7f3ee53f5640 1 -- 172.21.15.81:0/4092571984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee015a510 con 0x7f3ecc004800 2024-06-29T15:57:05.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.201+0000 7f3ee53f5640 1 -- 172.21.15.81:0/4092571984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee0152fc0 con 0x7f3ecc004800 2024-06-29T15:57:05.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.201+0000 7f3ee53f5640 1 -- 172.21.15.81:0/4092571984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ee00afbf0 con 0x7f3ecc004800 2024-06-29T15:57:05.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.203+0000 7f3ed77fe640 1 -- 172.21.15.81:0/4092571984 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f3ed00035f0 con 0x7f3ecc004800 2024-06-29T15:57:05.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.203+0000 7f3ed77fe640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3ec0076510 0x7f3ec0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.203+0000 7f3ed77fe640 1 -- 172.21.15.81:0/4092571984 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 6283+0+0 (secure 0 0 0) 0x7f3ed0099070 con 0x7f3ecc004800 2024-06-29T15:57:05.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.203+0000 7f3ed7fff640 1 -- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3ec0076510 msgr2=0x7f3ec0078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:05.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.203+0000 7f3ed7fff640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3ec0076510 0x7f3ec0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:05.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.211+0000 7f3ed77fe640 1 -- 172.21.15.81:0/4092571984 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3ed0061db0 con 0x7f3ecc004800 2024-06-29T15:57:05.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.402+0000 7f3ee53f5640 1 -- 172.21.15.81:0/4092571984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3ee014f070 con 0x7f3ecc004800 2024-06-29T15:57:05.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.402+0000 7f3ed77fe640 1 -- 172.21.15.81:0/4092571984 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v201) ==== 104+0+11 (secure 0 0 0) 0x7f3ed006a3f0 con 0x7f3ecc004800 2024-06-29T15:57:05.403 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:57:05.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.404+0000 7f3ee53f5640 1 -- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3ec0076510 msgr2=0x7f3ec0078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:05.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.404+0000 7f3ee53f5640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3ec0076510 0x7f3ec0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.404+0000 7f3ee53f5640 1 -- 172.21.15.81:0/4092571984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ecc004800 msgr2=0x7f3ee014eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.404+0000 7f3ee53f5640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ecc004800 0x7f3ee014eb30 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0011b10 tx=0x7f3ed0009f10 comp rx=0 tx=0).stop 2024-06-29T15:57:05.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.405+0000 7f3ee53f5640 1 -- 172.21.15.81:0/4092571984 shutdown_connections 2024-06-29T15:57:05.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.405+0000 7f3ee53f5640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ee015a100 0x7f3ee015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.405+0000 7f3ee53f5640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3ec0076510 0x7f3ec0078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.405+0000 7f3ee53f5640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ee015d3e0 0x7f3ee0159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.405+0000 7f3ee53f5640 1 --2- 172.21.15.81:0/4092571984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ecc004800 0x7f3ee014eb30 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.405+0000 7f3ee53f5640 1 -- 172.21.15.81:0/4092571984 >> 172.21.15.81:0/4092571984 conn(0x7f3ee00b1ec0 msgr2=0x7f3ee00d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.405+0000 7f3ee53f5640 1 -- 172.21.15.81:0/4092571984 shutdown_connections 2024-06-29T15:57:05.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.405+0000 7f3ee53f5640 1 -- 172.21.15.81:0/4092571984 wait complete. 2024-06-29T15:57:05.417 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:57:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.536+0000 7f4616089640 1 -- 172.21.15.81:0/2929891022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46100b5250 msgr2=0x7f46100d2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.536+0000 7f4616089640 1 --2- 172.21.15.81:0/2929891022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46100b5250 0x7f46100d2190 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f4608005fd0 tx=0x7f460802a9d0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.536+0000 7f4616089640 1 -- 172.21.15.81:0/2929891022 shutdown_connections 2024-06-29T15:57:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.536+0000 7f4616089640 1 --2- 172.21.15.81:0/2929891022 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46100b5250 0x7f46100d2190 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.536+0000 7f4616089640 1 -- 172.21.15.81:0/2929891022 >> 172.21.15.81:0/2929891022 conn(0x7f461009cf30 msgr2=0x7f46100d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.536+0000 7f4616089640 1 -- 172.21.15.81:0/2929891022 shutdown_connections 2024-06-29T15:57:05.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.536+0000 7f4616089640 1 -- 172.21.15.81:0/2929891022 wait complete. 2024-06-29T15:57:05.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.538+0000 7f4616089640 1 Processor -- start 2024-06-29T15:57:05.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.538+0000 7f4616089640 1 -- start start 2024-06-29T15:57:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.539+0000 7f4616089640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100b5250 0x7f461015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.539+0000 7f4616089640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46100b9040 0x7f46100b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.539+0000 7f4616089640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46100b0820 0x7f4610148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.539+0000 7f4616089640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46100d26d0 con 0x7f46100b5250 2024-06-29T15:57:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f4616089640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f46100b85c0 con 0x7f46100b9040 2024-06-29T15:57:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f4616089640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f46100d2850 con 0x7f46100b0820 2024-06-29T15:57:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100b5250 0x7f461015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100b5250 0x7f461015b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43650/0 (socket says 172.21.15.81:43650) 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460f7fe640 1 -- 172.21.15.81:0/2028500927 learned_addr learned my addr 172.21.15.81:0/2028500927 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460effd640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46100b9040 0x7f46100b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460ffff640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46100b0820 0x7f4610148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460f7fe640 1 -- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46100b0820 msgr2=0x7f4610148730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460f7fe640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46100b0820 0x7f4610148730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460f7fe640 1 -- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46100b9040 msgr2=0x7f46100b2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460f7fe640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46100b9040 0x7f46100b2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460f7fe640 1 -- 172.21.15.81:0/2028500927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46101f4dd0 con 0x7f46100b5250 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f460f7fe640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100b5250 0x7f461015b2f0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f45f800c460 tx=0x7f45f800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f4607fff640 1 -- 172.21.15.81:0/2028500927 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f80052d0 con 0x7f46100b5250 2024-06-29T15:57:05.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f4607fff640 1 -- 172.21.15.81:0/2028500927 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45f8005470 con 0x7f46100b5250 2024-06-29T15:57:05.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f4616089640 1 -- 172.21.15.81:0/2028500927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461015b830 con 0x7f46100b5250 2024-06-29T15:57:05.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f4607fff640 1 -- 172.21.15.81:0/2028500927 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f8012890 con 0x7f46100b5250 2024-06-29T15:57:05.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.540+0000 7f4616089640 1 -- 172.21.15.81:0/2028500927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4610147970 con 0x7f46100b5250 2024-06-29T15:57:05.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.541+0000 7f4616089640 1 -- 172.21.15.81:0/2028500927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46100d2de0 con 0x7f46100b5250 2024-06-29T15:57:05.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.542+0000 7f4607fff640 1 -- 172.21.15.81:0/2028500927 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f45f80035f0 con 0x7f46100b5250 2024-06-29T15:57:05.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.543+0000 7f4607fff640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f45f0076540 0x7f45f0078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.543+0000 7f460effd640 1 -- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f45f0076540 msgr2=0x7f45f0078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:05.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.543+0000 7f4607fff640 1 -- 172.21.15.81:0/2028500927 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 6283+0+0 (secure 0 0 0) 0x7f45f80978e0 con 0x7f46100b5250 2024-06-29T15:57:05.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.543+0000 7f460effd640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f45f0076540 0x7f45f0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:05.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.547+0000 7f4607fff640 1 -- 172.21.15.81:0/2028500927 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f45f80605d0 con 0x7f46100b5250 2024-06-29T15:57:05.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.739+0000 7f4616089640 1 -- 172.21.15.81:0/2028500927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f46100b8de0 con 0x7f46100b5250 2024-06-29T15:57:05.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.739+0000 7f4607fff640 1 -- 172.21.15.81:0/2028500927 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v201) ==== 100+0+11 (secure 0 0 0) 0x7f45f80648d0 con 0x7f46100b5250 2024-06-29T15:57:05.739 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:57:05.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.741+0000 7f4616089640 1 -- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f45f0076540 msgr2=0x7f45f0078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:05.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.741+0000 7f4616089640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f45f0076540 0x7f45f0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.741+0000 7f4616089640 1 -- 172.21.15.81:0/2028500927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100b5250 msgr2=0x7f461015b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.741+0000 7f4616089640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100b5250 0x7f461015b2f0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f45f800c460 tx=0x7f45f800c920 comp rx=0 tx=0).stop 2024-06-29T15:57:05.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.741+0000 7f4616089640 1 -- 172.21.15.81:0/2028500927 shutdown_connections 2024-06-29T15:57:05.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.741+0000 7f4616089640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f45f0076540 0x7f45f0078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.741+0000 7f4616089640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46100b0820 0x7f4610148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.741+0000 7f4616089640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46100b9040 0x7f46100b2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.741+0000 7f4616089640 1 --2- 172.21.15.81:0/2028500927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46100b5250 0x7f461015b2f0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.741+0000 7f4616089640 1 -- 172.21.15.81:0/2028500927 >> 172.21.15.81:0/2028500927 conn(0x7f461009cf30 msgr2=0x7f46100cd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.742+0000 7f4616089640 1 -- 172.21.15.81:0/2028500927 shutdown_connections 2024-06-29T15:57:05.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.742+0000 7f4616089640 1 -- 172.21.15.81:0/2028500927 wait complete. 2024-06-29T15:57:05.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:57:05.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.870+0000 7f5128dac640 1 -- 172.21.15.81:0/1592862258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51240b5250 msgr2=0x7f51240d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.870+0000 7f5128dac640 1 --2- 172.21.15.81:0/1592862258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51240b5250 0x7f51240d09c0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f512408e230 tx=0x7f510c02a680 comp rx=0 tx=0).stop 2024-06-29T15:57:05.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.870+0000 7f5128dac640 1 -- 172.21.15.81:0/1592862258 shutdown_connections 2024-06-29T15:57:05.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.870+0000 7f5128dac640 1 --2- 172.21.15.81:0/1592862258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51240b5250 0x7f51240d09c0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.870+0000 7f5128dac640 1 --2- 172.21.15.81:0/1592862258 >> v2:172.21.15.167:3300/0 conn(0x7f51240b9040 0x7f51240d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.870+0000 7f5128dac640 1 -- 172.21.15.81:0/1592862258 >> 172.21.15.81:0/1592862258 conn(0x7f512409cf30 msgr2=0x7f51240d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:05.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.870+0000 7f5128dac640 1 -- 172.21.15.81:0/1592862258 shutdown_connections 2024-06-29T15:57:05.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.870+0000 7f5128dac640 1 -- 172.21.15.81:0/1592862258 wait complete. 2024-06-29T15:57:05.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.872+0000 7f5128dac640 1 Processor -- start 2024-06-29T15:57:05.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.872+0000 7f5128dac640 1 -- start start 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5128dac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51240b5250 0x7f51240b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5128dac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51240b7290 0x7f51240a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5128dac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51240b9040 0x7f51240a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5128dac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f51240d2e20 con 0x7f51240b9040 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5128dac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f51240d2ca0 con 0x7f51240b7290 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5128dac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f51240b7db0 con 0x7f51240b5250 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5122d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51240b9040 0x7f51240a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5122d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51240b9040 0x7f51240a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43672/0 (socket says 172.21.15.81:43672) 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5122d9d640 1 -- 172.21.15.81:0/2961725460 learned_addr learned my addr 172.21.15.81:0/2961725460 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f512259c640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51240b5250 0x7f51240b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f511bfff640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51240b7290 0x7f51240a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:05.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5122d9d640 1 -- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51240b5250 msgr2=0x7f51240b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5122d9d640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51240b5250 0x7f51240b0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5122d9d640 1 -- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51240b7290 msgr2=0x7f51240a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5122d9d640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51240b7290 0x7f51240a00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f5122d9d640 1 -- 172.21.15.81:0/2961725460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51240b3990 con 0x7f51240b9040 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.873+0000 7f512259c640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51240b5250 0x7f51240b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.874+0000 7f5122d9d640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51240b9040 0x7f51240a05f0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f511c00c6e0 tx=0x7f511c00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.874+0000 7f511affd640 1 -- 172.21.15.81:0/2961725460 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c003b70 con 0x7f51240b9040 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.874+0000 7f511affd640 1 -- 172.21.15.81:0/2961725460 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f511c003d10 con 0x7f51240b9040 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.874+0000 7f5128dac640 1 -- 172.21.15.81:0/2961725460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51240b3c70 con 0x7f51240b9040 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.874+0000 7f511affd640 1 -- 172.21.15.81:0/2961725460 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c005330 con 0x7f51240b9040 2024-06-29T15:57:05.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.874+0000 7f5128dac640 1 -- 172.21.15.81:0/2961725460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51240b4230 con 0x7f51240b9040 2024-06-29T15:57:05.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.875+0000 7f5128dac640 1 -- 172.21.15.81:0/2961725460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51240b34b0 con 0x7f51240b9040 2024-06-29T15:57:05.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.876+0000 7f511affd640 1 -- 172.21.15.81:0/2961725460 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f511c0054d0 con 0x7f51240b9040 2024-06-29T15:57:05.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.876+0000 7f511affd640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5104076510 0x7f5104078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:05.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.876+0000 7f512259c640 1 -- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5104076510 msgr2=0x7f5104078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:05.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.876+0000 7f511affd640 1 -- 172.21.15.81:0/2961725460 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 6283+0+0 (secure 0 0 0) 0x7f511c097a50 con 0x7f51240b9040 2024-06-29T15:57:05.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.876+0000 7f512259c640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5104076510 0x7f5104078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:05.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:05.880+0000 7f511affd640 1 -- 172.21.15.81:0/2961725460 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f51240b34b0 con 0x7f51240b9040 2024-06-29T15:57:06.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.078+0000 7f512259c640 1 -- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5104076510 msgr2=0x7f5104078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:06.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.078+0000 7f512259c640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5104076510 0x7f5104078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:06.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.082+0000 7f5128dac640 1 -- 172.21.15.81:0/2961725460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f51240b36b0 con 0x7f51240b9040 2024-06-29T15:57:06.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.083+0000 7f511affd640 1 -- 172.21.15.81:0/2961725460 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v201) ==== 80+0+476 (secure 0 0 0) 0x7f511c0607d0 con 0x7f51240b9040 2024-06-29T15:57:06.083 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/803000291 2024-06-30T15:55:49.959292+0000 2024-06-29T15:57:06.083 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/803000291 2024-06-30T15:55:49.959292+0000 2024-06-29T15:57:06.084 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1482340295 2024-06-30T15:55:49.959292+0000 2024-06-29T15:57:06.084 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2216474049 2024-06-30T15:55:49.959292+0000 2024-06-29T15:57:06.084 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3153968292 2024-06-30T15:55:49.959292+0000 2024-06-29T15:57:06.084 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1095374241 2024-06-30T15:55:49.959292+0000 2024-06-29T15:57:06.084 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3639047578 2024-06-30T15:55:49.959292+0000 2024-06-29T15:57:06.084 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2467432753 2024-06-30T15:55:49.959292+0000 2024-06-29T15:57:06.084 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T15:57:06.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 -- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5104076510 msgr2=0x7f5104078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:06.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5104076510 0x7f5104078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 -- 172.21.15.81:0/2961725460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51240b9040 msgr2=0x7f51240a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51240b9040 0x7f51240a05f0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f511c00c6e0 tx=0x7f511c00cba0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 -- 172.21.15.81:0/2961725460 shutdown_connections 2024-06-29T15:57:06.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51240b9040 0x7f51240a05f0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51240b7290 0x7f51240a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5104076510 0x7f5104078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 --2- 172.21.15.81:0/2961725460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51240b5250 0x7f51240b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 -- 172.21.15.81:0/2961725460 >> 172.21.15.81:0/2961725460 conn(0x7f512409cf30 msgr2=0x7f51240d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:06.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 -- 172.21.15.81:0/2961725460 shutdown_connections 2024-06-29T15:57:06.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.085+0000 7f5128dac640 1 -- 172.21.15.81:0/2961725460 wait complete. 2024-06-29T15:57:06.098 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:57:06.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:06.098 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:06.126 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T15:57:06.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:06.126 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:57:06.182 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:06.182 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:57:06.182 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:06.198 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:06.198 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:06.199 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:57:06.199 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:06.238 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:57:06.238 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:06.239 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:57:06.239 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:06.255 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:06.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:06.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.395+0000 7f46e33a1640 1 -- 172.21.15.81:0/114241870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc0cccf0 msgr2=0x7f46dc0d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.395+0000 7f46e33a1640 1 --2- 172.21.15.81:0/114241870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc0cccf0 0x7f46dc0d0230 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f46dc073550 tx=0x7f46d802a4a0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.395+0000 7f46e33a1640 1 -- 172.21.15.81:0/114241870 shutdown_connections 2024-06-29T15:57:06.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.395+0000 7f46e33a1640 1 --2- 172.21.15.81:0/114241870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc0cccf0 0x7f46dc0d0230 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.395+0000 7f46e33a1640 1 --2- 172.21.15.81:0/114241870 >> v2:172.21.15.167:3300/0 conn(0x7f46dc0b3200 0x7f46dc0cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.395+0000 7f46e33a1640 1 -- 172.21.15.81:0/114241870 >> 172.21.15.81:0/114241870 conn(0x7f46dc0b3680 msgr2=0x7f46dc0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:06.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.395+0000 7f46e33a1640 1 -- 172.21.15.81:0/114241870 shutdown_connections 2024-06-29T15:57:06.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.395+0000 7f46e33a1640 1 -- 172.21.15.81:0/114241870 wait complete. 2024-06-29T15:57:06.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.397+0000 7f46e33a1640 1 Processor -- start 2024-06-29T15:57:06.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.397+0000 7f46e33a1640 1 -- start start 2024-06-29T15:57:06.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e33a1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46dc0b3200 0x7f46dc1de650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e33a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc0cccf0 0x7f46dc09df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e33a1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46dc0d0770 0x7f46dc1e15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e33a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc0cedb0 con 0x7f46dc0cccf0 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e33a1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc0cec30 con 0x7f46dc0b3200 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e33a1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f46dc0cef30 con 0x7f46dc0d0770 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e093c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc0cccf0 0x7f46dc09df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e093c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc0cccf0 0x7f46dc09df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43688/0 (socket says 172.21.15.81:43688) 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e093c640 1 -- 172.21.15.81:0/3893095771 learned_addr learned my addr 172.21.15.81:0/3893095771 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e113d640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46dc0b3200 0x7f46dc1de650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e193e640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46dc0d0770 0x7f46dc1e15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e093c640 1 -- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46dc0d0770 msgr2=0x7f46dc1e15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e093c640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46dc0d0770 0x7f46dc1e15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e093c640 1 -- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46dc0b3200 msgr2=0x7f46dc1de650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e093c640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46dc0b3200 0x7f46dc1de650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e093c640 1 -- 172.21.15.81:0/3893095771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46dc1deb90 con 0x7f46dc0cccf0 2024-06-29T15:57:06.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e093c640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc0cccf0 0x7f46dc09df40 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f46d802aa90 tx=0x7f46d8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:06.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46d1ffb640 1 -- 172.21.15.81:0/3893095771 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8031d60 con 0x7f46dc0cccf0 2024-06-29T15:57:06.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46d1ffb640 1 -- 172.21.15.81:0/3893095771 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46d8033e10 con 0x7f46dc0cccf0 2024-06-29T15:57:06.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.399+0000 7f46e33a1640 1 -- 172.21.15.81:0/3893095771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46dc1dfdb0 con 0x7f46dc0cccf0 2024-06-29T15:57:06.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.400+0000 7f46d1ffb640 1 -- 172.21.15.81:0/3893095771 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d8005260 con 0x7f46dc0cccf0 2024-06-29T15:57:06.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.400+0000 7f46e33a1640 1 -- 172.21.15.81:0/3893095771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46dc1e0370 con 0x7f46dc0cccf0 2024-06-29T15:57:06.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.401+0000 7f46e33a1640 1 -- 172.21.15.81:0/3893095771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46dc0cc7b0 con 0x7f46dc0cccf0 2024-06-29T15:57:06.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.401+0000 7f46d1ffb640 1 -- 172.21.15.81:0/3893095771 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f46d8005400 con 0x7f46dc0cccf0 2024-06-29T15:57:06.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.402+0000 7f46d1ffb640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f46c0076540 0x7f46c0078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.402+0000 7f46e113d640 1 -- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f46c0076540 msgr2=0x7f46c0078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:06.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.402+0000 7f46d1ffb640 1 -- 172.21.15.81:0/3893095771 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 6283+0+0 (secure 0 0 0) 0x7f46d80b8ed0 con 0x7f46dc0cccf0 2024-06-29T15:57:06.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.402+0000 7f46e113d640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f46c0076540 0x7f46c0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:06.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.406+0000 7f46d1ffb640 1 -- 172.21.15.81:0/3893095771 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f46d803f030 con 0x7f46dc0cccf0 2024-06-29T15:57:06.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.603+0000 7f46e113d640 1 -- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f46c0076540 msgr2=0x7f46c0078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:06.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.603+0000 7f46e113d640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f46c0076540 0x7f46c0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:06.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.632+0000 7f46e33a1640 1 -- 172.21.15.81:0/3893095771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46dc1dfad0 con 0x7f46dc0cccf0 2024-06-29T15:57:06.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.633+0000 7f46d1ffb640 1 -- 172.21.15.81:0/3893095771 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5240 (secure 0 0 0) 0x7f46d8005d00 con 0x7f46dc0cccf0 2024-06-29T15:57:06.633 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:06.633 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":103,"btime":"2024-06-29T15:55:41:302970+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1764785504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1764785504},{"type":"v1","addr":"172.21.15.167:6836","nonce":1764785504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99},{"gid":10713,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/1699508727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1699508727},{"type":"v1","addr":"172.21.15.81:6834","nonce":1699508727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/265635924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":265635924},{"type":"v1","addr":"172.21.15.81:6835","nonce":265635924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.649055+0000","modified":"2024-06-29T15:55:40.671516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/1700137710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1700137710},{"type":"v1","addr":"172.21.15.167:6837","nonce":1700137710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10731,"qdb_cluster":[10731]},"id":12}]} 2024-06-29T15:57:06.633 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 103 2024-06-29T15:57:06.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 -- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f46c0076540 msgr2=0x7f46c0078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:06.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f46c0076540 0x7f46c0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 -- 172.21.15.81:0/3893095771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc0cccf0 msgr2=0x7f46dc09df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc0cccf0 0x7f46dc09df40 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f46d802aa90 tx=0x7f46d8033c20 comp rx=0 tx=0).stop 2024-06-29T15:57:06.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 -- 172.21.15.81:0/3893095771 shutdown_connections 2024-06-29T15:57:06.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f46c0076540 0x7f46c0078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46dc0d0770 0x7f46dc1e15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc0cccf0 0x7f46dc09df40 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 --2- 172.21.15.81:0/3893095771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46dc0b3200 0x7f46dc1de650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 -- 172.21.15.81:0/3893095771 >> 172.21.15.81:0/3893095771 conn(0x7f46dc0b3680 msgr2=0x7f46dc0d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:06.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 -- 172.21.15.81:0/3893095771 shutdown_connections 2024-06-29T15:57:06.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.635+0000 7f46e33a1640 1 -- 172.21.15.81:0/3893095771 wait complete. 2024-06-29T15:57:06.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:06.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.760+0000 7fc319355640 1 -- 172.21.15.81:0/974798359 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3140b3200 msgr2=0x7fc3140cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.760+0000 7fc319355640 1 --2- 172.21.15.81:0/974798359 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3140b3200 0x7fc3140cc7b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fc304009830 tx=0x7fc30402a530 comp rx=0 tx=0).stop 2024-06-29T15:57:06.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.760+0000 7fc319355640 1 -- 172.21.15.81:0/974798359 shutdown_connections 2024-06-29T15:57:06.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.761+0000 7fc319355640 1 --2- 172.21.15.81:0/974798359 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3140b3200 0x7fc3140cc7b0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.761+0000 7fc319355640 1 -- 172.21.15.81:0/974798359 >> 172.21.15.81:0/974798359 conn(0x7fc3140b3680 msgr2=0x7fc3140b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:06.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.761+0000 7fc319355640 1 -- 172.21.15.81:0/974798359 shutdown_connections 2024-06-29T15:57:06.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.761+0000 7fc319355640 1 -- 172.21.15.81:0/974798359 wait complete. 2024-06-29T15:57:06.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.762+0000 7fc319355640 1 Processor -- start 2024-06-29T15:57:06.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.762+0000 7fc319355640 1 -- start start 2024-06-29T15:57:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc319355640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3140b3200 0x7fc314162e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc319355640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc3140cccf0 0x7fc314156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc319355640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31415cb70 0x7fc3141568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc319355640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc3140b5e50 con 0x7fc31415cb70 2024-06-29T15:57:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc319355640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc3140b5cd0 con 0x7fc3140b3200 2024-06-29T15:57:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc319355640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc3140b5fd0 con 0x7fc3140cccf0 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc3137fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31415cb70 0x7fc3141568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc3137fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31415cb70 0x7fc3141568c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43700/0 (socket says 172.21.15.81:43700) 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc3137fe640 1 -- 172.21.15.81:0/1323223650 learned_addr learned my addr 172.21.15.81:0/1323223650 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc312ffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3140b3200 0x7fc314162e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc3127fc640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc3140cccf0 0x7fc314156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc312ffd640 1 -- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc3140cccf0 msgr2=0x7fc314156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc312ffd640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc3140cccf0 0x7fc314156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc312ffd640 1 -- 172.21.15.81:0/1323223650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31415cb70 msgr2=0x7fc3141568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc312ffd640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31415cb70 0x7fc3141568c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc312ffd640 1 -- 172.21.15.81:0/1323223650 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc314161d60 con 0x7fc3140b3200 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.764+0000 7fc312ffd640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3140b3200 0x7fc314162e00 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc304031000 tx=0x7fc304033ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.765+0000 7fc2fb7fe640 1 -- 172.21.15.81:0/1323223650 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304031b50 con 0x7fc3140b3200 2024-06-29T15:57:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.765+0000 7fc2fb7fe640 1 -- 172.21.15.81:0/1323223650 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc304031de0 con 0x7fc3140b3200 2024-06-29T15:57:06.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.765+0000 7fc319355640 1 -- 172.21.15.81:0/1323223650 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc314162040 con 0x7fc3140b3200 2024-06-29T15:57:06.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.765+0000 7fc2fb7fe640 1 -- 172.21.15.81:0/1323223650 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc304005280 con 0x7fc3140b3200 2024-06-29T15:57:06.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.765+0000 7fc319355640 1 -- 172.21.15.81:0/1323223650 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc314162600 con 0x7fc3140b3200 2024-06-29T15:57:06.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.766+0000 7fc319355640 1 -- 172.21.15.81:0/1323223650 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3140cc7b0 con 0x7fc3140b3200 2024-06-29T15:57:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.767+0000 7fc2fb7fe640 1 -- 172.21.15.81:0/1323223650 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7fc304005420 con 0x7fc3140b3200 2024-06-29T15:57:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.767+0000 7fc2fb7fe640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fc300076540 0x7fc300078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.767+0000 7fc3127fc640 1 -- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fc300076540 msgr2=0x7fc300078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.767+0000 7fc2fb7fe640 1 -- 172.21.15.81:0/1323223650 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 6283+0+0 (secure 0 0 0) 0x7fc3040b9490 con 0x7fc3140b3200 2024-06-29T15:57:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.767+0000 7fc3127fc640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fc300076540 0x7fc300078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:06.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.775+0000 7fc2fb7fe640 1 -- 172.21.15.81:0/1323223650 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc30403f030 con 0x7fc3140b3200 2024-06-29T15:57:06.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.959+0000 7fc319355640 1 -- 172.21.15.81:0/1323223650 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3140cea20 con 0x7fc3140b3200 2024-06-29T15:57:06.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.961+0000 7fc2fb7fe640 1 -- 172.21.15.81:0/1323223650 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) ==== 74+0+17510 (secure 0 0 0) 0x7fc304031770 con 0x7fc3140b3200 2024-06-29T15:57:06.961 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:06.962 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":201,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:05.179744+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.947285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.605625+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.1a","mappings":[{"from":0,"to":2}]},{"pgid":"25.3f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/803000291":"2024-06-30T15:55:49.959292+0000","172.21.15.167:6832/803000291":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/1482340295":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/2216474049":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/3153968292":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/1095374241":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/3639047578":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/2467432753":"2024-06-30T15:55:49.959292+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 -- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fc300076540 msgr2=0x7fc300078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fc300076540 0x7fc300078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 -- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3140b3200 msgr2=0x7fc314162e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3140b3200 0x7fc314162e00 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc304031000 tx=0x7fc304033ba0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 -- 172.21.15.81:0/1323223650 shutdown_connections 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31415cb70 0x7fc3141568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7fc300076540 0x7fc300078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc3140cccf0 0x7fc314156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 --2- 172.21.15.81:0/1323223650 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc3140b3200 0x7fc314162e00 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 -- 172.21.15.81:0/1323223650 >> 172.21.15.81:0/1323223650 conn(0x7fc3140b3680 msgr2=0x7fc3140b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 -- 172.21.15.81:0/1323223650 shutdown_connections 2024-06-29T15:57:06.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:06.963+0000 7fc319355640 1 -- 172.21.15.81:0/1323223650 wait complete. 2024-06-29T15:57:06.975 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:55:35.947285+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:55:36.605625+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '201', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 201, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:06.975 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:07.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.092+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2539313822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e140b3200 msgr2=0x7f5e140cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.092+0000 7f5e1bf9f640 1 --2- 172.21.15.81:0/2539313822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e140b3200 0x7f5e140cc7b0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f5e1415c790 tx=0x7f5e0402a560 comp rx=0 tx=0).stop 2024-06-29T15:57:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.092+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2539313822 shutdown_connections 2024-06-29T15:57:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.092+0000 7f5e1bf9f640 1 --2- 172.21.15.81:0/2539313822 >> v2:172.21.15.167:3300/0 conn(0x7f5e140d0770 0x7f5e140ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.092+0000 7f5e1bf9f640 1 --2- 172.21.15.81:0/2539313822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e140cccf0 0x7f5e140d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.092+0000 7f5e1bf9f640 1 --2- 172.21.15.81:0/2539313822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e140b3200 0x7f5e140cc7b0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.092+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2539313822 >> 172.21.15.81:0/2539313822 conn(0x7f5e140b3680 msgr2=0x7f5e140b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.092+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2539313822 shutdown_connections 2024-06-29T15:57:07.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.092+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2539313822 wait complete. 2024-06-29T15:57:07.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.094+0000 7f5e1bf9f640 1 Processor -- start 2024-06-29T15:57:07.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.094+0000 7f5e1bf9f640 1 -- start start 2024-06-29T15:57:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e1bf9f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e140b3200 0x7f5e1406d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e1bf9f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e140cccf0 0x7f5e1409ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e1bf9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e140d0770 0x7f5e1409c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e1bf9f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5e140cef90 con 0x7f5e140d0770 2024-06-29T15:57:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e1bf9f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5e140cee10 con 0x7f5e140b3200 2024-06-29T15:57:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e1bf9f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5e140cf110 con 0x7f5e140cccf0 2024-06-29T15:57:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e1a53c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e140d0770 0x7f5e1409c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e1a53c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e140d0770 0x7f5e1409c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43722/0 (socket says 172.21.15.81:43722) 2024-06-29T15:57:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e1a53c640 1 -- 172.21.15.81:0/2823839450 learned_addr learned my addr 172.21.15.81:0/2823839450 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:07.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e1953a640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e140cccf0 0x7f5e1409ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.095+0000 7f5e19d3b640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e140b3200 0x7f5e1406d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e1a53c640 1 -- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e140cccf0 msgr2=0x7f5e1409ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e1a53c640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e140cccf0 0x7f5e1409ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e1a53c640 1 -- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e140b3200 msgr2=0x7f5e1406d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e1a53c640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e140b3200 0x7f5e1406d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e1a53c640 1 -- 172.21.15.81:0/2823839450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1406f2e0 con 0x7f5e140d0770 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e1a53c640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e140d0770 0x7f5e1409c520 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00eab0 tx=0x7f5e0c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e0a7fc640 1 -- 172.21.15.81:0/2823839450 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c00cea0 con 0x7f5e140d0770 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e0a7fc640 1 -- 172.21.15.81:0/2823839450 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e0c004b20 con 0x7f5e140d0770 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e0a7fc640 1 -- 172.21.15.81:0/2823839450 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c005020 con 0x7f5e140d0770 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2823839450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e1409e190 con 0x7f5e140d0770 2024-06-29T15:57:07.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.096+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2823839450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e1409e750 con 0x7f5e140d0770 2024-06-29T15:57:07.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.097+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2823839450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e1406f050 con 0x7f5e140d0770 2024-06-29T15:57:07.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.098+0000 7f5e0a7fc640 1 -- 172.21.15.81:0/2823839450 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f5e0c0051c0 con 0x7f5e140d0770 2024-06-29T15:57:07.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.098+0000 7f5e0a7fc640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5df8076510 0x7f5df8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.098+0000 7f5e0a7fc640 1 -- 172.21.15.81:0/2823839450 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 6283+0+0 (secure 0 0 0) 0x7f5e0c098a60 con 0x7f5e140d0770 2024-06-29T15:57:07.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.098+0000 7f5e19d3b640 1 -- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5df8076510 msgr2=0x7f5df8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:07.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.098+0000 7f5e19d3b640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5df8076510 0x7f5df8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:07.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.103+0000 7f5e0a7fc640 1 -- 172.21.15.81:0/2823839450 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5e0c061630 con 0x7f5e140d0770 2024-06-29T15:57:07.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.288+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2823839450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5e140d0230 con 0x7f5e140d0770 2024-06-29T15:57:07.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.288+0000 7f5e0a7fc640 1 -- 172.21.15.81:0/2823839450 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v201) ==== 93+0+10 (secure 0 0 0) 0x7f5e0c065930 con 0x7f5e140d0770 2024-06-29T15:57:07.289 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:57:07.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5df8076510 msgr2=0x7f5df8078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:07.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5df8076510 0x7f5df8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2823839450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e140d0770 msgr2=0x7f5e1409c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e140d0770 0x7f5e1409c520 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00eab0 tx=0x7f5e0c00ef70 comp rx=0 tx=0).stop 2024-06-29T15:57:07.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2823839450 shutdown_connections 2024-06-29T15:57:07.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e140d0770 0x7f5e1409c520 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f5df8076510 0x7f5df8078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e140cccf0 0x7f5e1409ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 --2- 172.21.15.81:0/2823839450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e140b3200 0x7f5e1406d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2823839450 >> 172.21.15.81:0/2823839450 conn(0x7f5e140b3680 msgr2=0x7f5e141e4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2823839450 shutdown_connections 2024-06-29T15:57:07.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.290+0000 7f5e1bf9f640 1 -- 172.21.15.81:0/2823839450 wait complete. 2024-06-29T15:57:07.302 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:57:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.420+0000 7f3dcc690640 1 -- 172.21.15.81:0/1398829560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dc40b4800 msgr2=0x7f3dc4144590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.420+0000 7f3dcc690640 1 --2- 172.21.15.81:0/1398829560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dc40b4800 0x7f3dc4144590 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40d0a00 tx=0x7f3dc002a560 comp rx=0 tx=0).stop 2024-06-29T15:57:07.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.420+0000 7f3dcc690640 1 -- 172.21.15.81:0/1398829560 shutdown_connections 2024-06-29T15:57:07.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.420+0000 7f3dcc690640 1 --2- 172.21.15.81:0/1398829560 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dc415b400 0x7f3dc415b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.420+0000 7f3dcc690640 1 --2- 172.21.15.81:0/1398829560 >> v2:172.21.15.167:3300/0 conn(0x7f3dc4072f80 0x7f3dc4073390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.420+0000 7f3dcc690640 1 --2- 172.21.15.81:0/1398829560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dc40b4800 0x7f3dc4144590 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.420+0000 7f3dcc690640 1 -- 172.21.15.81:0/1398829560 >> 172.21.15.81:0/1398829560 conn(0x7f3dc40b2920 msgr2=0x7f3dc40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.420+0000 7f3dcc690640 1 -- 172.21.15.81:0/1398829560 shutdown_connections 2024-06-29T15:57:07.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.420+0000 7f3dcc690640 1 -- 172.21.15.81:0/1398829560 wait complete. 2024-06-29T15:57:07.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.421+0000 7f3dcc690640 1 Processor -- start 2024-06-29T15:57:07.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.422+0000 7f3dcc690640 1 -- start start 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dcc690640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dc4072f80 0x7f3dc41f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dcc690640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dc40b4800 0x7f3dc41eca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dcc690640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dc415b400 0x7f3dc40ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dcc690640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc406f140 con 0x7f3dc4072f80 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dcc690640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc406efc0 con 0x7f3dc415b400 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dcc690640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc406f2c0 con 0x7f3dc40b4800 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dca42c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dc4072f80 0x7f3dc41f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dca42c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dc4072f80 0x7f3dc41f3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43750/0 (socket says 172.21.15.81:43750) 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dca42c640 1 -- 172.21.15.81:0/1207550565 learned_addr learned my addr 172.21.15.81:0/1207550565 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dc9c2b640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dc40b4800 0x7f3dc41eca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dcac2d640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dc415b400 0x7f3dc40ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dca42c640 1 -- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dc40b4800 msgr2=0x7f3dc41eca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dca42c640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dc40b4800 0x7f3dc41eca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dca42c640 1 -- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dc415b400 msgr2=0x7f3dc40ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dca42c640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dc415b400 0x7f3dc40ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dca42c640 1 -- 172.21.15.81:0/1207550565 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc40a9c50 con 0x7f3dc4072f80 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dc9c2b640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dc40b4800 0x7f3dc41eca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.423+0000 7f3dcac2d640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dc415b400 0x7f3dc40ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.424+0000 7f3dca42c640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dc4072f80 0x7f3dc41f3550 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0000c00 tx=0x7f3dc00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.424+0000 7f3dbaffd640 1 -- 172.21.15.81:0/1207550565 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc003e030 con 0x7f3dc4072f80 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.424+0000 7f3dbaffd640 1 -- 172.21.15.81:0/1207550565 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dc0003ca0 con 0x7f3dc4072f80 2024-06-29T15:57:07.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.424+0000 7f3dcc690640 1 -- 172.21.15.81:0/1207550565 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc41f3a90 con 0x7f3dc4072f80 2024-06-29T15:57:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.424+0000 7f3dbaffd640 1 -- 172.21.15.81:0/1207550565 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0035040 con 0x7f3dc4072f80 2024-06-29T15:57:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.424+0000 7f3dcc690640 1 -- 172.21.15.81:0/1207550565 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc40aa460 con 0x7f3dc4072f80 2024-06-29T15:57:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.425+0000 7f3dcc690640 1 -- 172.21.15.81:0/1207550565 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc40aae60 con 0x7f3dc4072f80 2024-06-29T15:57:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.426+0000 7f3dbaffd640 1 -- 172.21.15.81:0/1207550565 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f3dc0041030 con 0x7f3dc4072f80 2024-06-29T15:57:07.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.426+0000 7f3dbaffd640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3da8076510 0x7f3da8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.426+0000 7f3dbaffd640 1 -- 172.21.15.81:0/1207550565 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 6283+0+0 (secure 0 0 0) 0x7f3dc00b8390 con 0x7f3dc4072f80 2024-06-29T15:57:07.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.426+0000 7f3dc9c2b640 1 -- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3da8076510 msgr2=0x7f3da8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:07.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.426+0000 7f3dc9c2b640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3da8076510 0x7f3da8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:07.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.434+0000 7f3dbaffd640 1 -- 172.21.15.81:0/1207550565 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3dc0081150 con 0x7f3dc4072f80 2024-06-29T15:57:07.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.617+0000 7f3dcc690640 1 -- 172.21.15.81:0/1207550565 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3dc4144590 con 0x7f3dc4072f80 2024-06-29T15:57:07.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.617+0000 7f3dbaffd640 1 -- 172.21.15.81:0/1207550565 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v201) ==== 104+0+11 (secure 0 0 0) 0x7f3dc0085450 con 0x7f3dc4072f80 2024-06-29T15:57:07.618 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:57:07.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.619+0000 7f3dcc690640 1 -- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3da8076510 msgr2=0x7f3da8078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:07.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.619+0000 7f3dcc690640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3da8076510 0x7f3da8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.619+0000 7f3dcc690640 1 -- 172.21.15.81:0/1207550565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dc4072f80 msgr2=0x7f3dc41f3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.619+0000 7f3dcc690640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dc4072f80 0x7f3dc41f3550 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0000c00 tx=0x7f3dc00038a0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.620+0000 7f3dcc690640 1 -- 172.21.15.81:0/1207550565 shutdown_connections 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.620+0000 7f3dcc690640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3dc415b400 0x7f3dc40ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.620+0000 7f3dcc690640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f3da8076510 0x7f3da8078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.620+0000 7f3dcc690640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3dc40b4800 0x7f3dc41eca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.620+0000 7f3dcc690640 1 --2- 172.21.15.81:0/1207550565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3dc4072f80 0x7f3dc41f3550 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.620+0000 7f3dcc690640 1 -- 172.21.15.81:0/1207550565 >> 172.21.15.81:0/1207550565 conn(0x7f3dc40b2920 msgr2=0x7f3dc4068960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.620+0000 7f3dcc690640 1 -- 172.21.15.81:0/1207550565 shutdown_connections 2024-06-29T15:57:07.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.620+0000 7f3dcc690640 1 -- 172.21.15.81:0/1207550565 wait complete. 2024-06-29T15:57:07.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:57:07.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.749+0000 7f982abf5640 1 -- 172.21.15.81:0/1877280628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98240b4f40 msgr2=0x7f98240b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.749+0000 7f982abf5640 1 --2- 172.21.15.81:0/1877280628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98240b4f40 0x7f98240b5350 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f981c005fd0 tx=0x7f981c02aa90 comp rx=0 tx=0).stop 2024-06-29T15:57:07.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.749+0000 7f982abf5640 1 -- 172.21.15.81:0/1877280628 shutdown_connections 2024-06-29T15:57:07.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.749+0000 7f982abf5640 1 --2- 172.21.15.81:0/1877280628 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98240d0230 0x7f98240d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.749+0000 7f982abf5640 1 --2- 172.21.15.81:0/1877280628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98240b4f40 0x7f98240b5350 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.749+0000 7f982abf5640 1 -- 172.21.15.81:0/1877280628 >> 172.21.15.81:0/1877280628 conn(0x7f98240b3680 msgr2=0x7f98240b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.749+0000 7f982abf5640 1 -- 172.21.15.81:0/1877280628 shutdown_connections 2024-06-29T15:57:07.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.749+0000 7f982abf5640 1 -- 172.21.15.81:0/1877280628 wait complete. 2024-06-29T15:57:07.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.751+0000 7f982abf5640 1 Processor -- start 2024-06-29T15:57:07.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.751+0000 7f982abf5640 1 -- start start 2024-06-29T15:57:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f982abf5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98240b3200 0x7f98241e0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f982abf5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98240b4f40 0x7f98241e5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f982abf5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98240d0230 0x7f98241e6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f982abf5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f98240b5c20 con 0x7f98240b3200 2024-06-29T15:57:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f982abf5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f98240b5aa0 con 0x7f98240d0230 2024-06-29T15:57:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f982abf5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f98240b5da0 con 0x7f98240b4f40 2024-06-29T15:57:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9828991640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98240b3200 0x7f98241e0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9828991640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98240b3200 0x7f98241e0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43754/0 (socket says 172.21.15.81:43754) 2024-06-29T15:57:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9828991640 1 -- 172.21.15.81:0/3132697120 learned_addr learned my addr 172.21.15.81:0/3132697120 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9823fff640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98240b4f40 0x7f98241e5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9829192640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98240d0230 0x7f98241e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9828991640 1 -- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98240b4f40 msgr2=0x7f98241e5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9828991640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98240b4f40 0x7f98241e5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9828991640 1 -- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98240d0230 msgr2=0x7f98241e6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9828991640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98240d0230 0x7f98241e6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9828991640 1 -- 172.21.15.81:0/3132697120 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98240cb480 con 0x7f98240b3200 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f9828991640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98240b3200 0x7f98241e0330 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f981800c3a0 tx=0x7f981800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.753+0000 7f98217fa640 1 -- 172.21.15.81:0/3132697120 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9818017070 con 0x7f98240b3200 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.754+0000 7f98217fa640 1 -- 172.21.15.81:0/3132697120 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98180053c0 con 0x7f98240b3200 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.754+0000 7f982abf5640 1 -- 172.21.15.81:0/3132697120 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98240cb760 con 0x7f98240b3200 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.754+0000 7f98217fa640 1 -- 172.21.15.81:0/3132697120 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9818003850 con 0x7f98240b3200 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.754+0000 7f982abf5640 1 -- 172.21.15.81:0/3132697120 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98240c8c60 con 0x7f98240b3200 2024-06-29T15:57:07.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.755+0000 7f982abf5640 1 -- 172.21.15.81:0/3132697120 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98240b5350 con 0x7f98240b3200 2024-06-29T15:57:07.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.755+0000 7f98217fa640 1 -- 172.21.15.81:0/3132697120 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 124) ==== 98417+0+0 (secure 0 0 0) 0x7f981801a070 con 0x7f98240b3200 2024-06-29T15:57:07.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.756+0000 7f98217fa640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f9808076510 0x7f9808078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:07.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.756+0000 7f9823fff640 1 -- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f9808076510 msgr2=0x7f9808078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/23077685 2024-06-29T15:57:07.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.756+0000 7f98217fa640 1 -- 172.21.15.81:0/3132697120 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 6283+0+0 (secure 0 0 0) 0x7f98180970a0 con 0x7f98240b3200 2024-06-29T15:57:07.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.756+0000 7f9823fff640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f9808076510 0x7f9808078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:07.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.760+0000 7f98217fa640 1 -- 172.21.15.81:0/3132697120 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f98240d0b80 con 0x7f98240b3200 2024-06-29T15:57:07.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.861+0000 7f98217fa640 1 -- 172.21.15.81:0/3132697120 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 125) ==== 97912+0+0 (secure 0 0 0) 0x7f981805c0d0 con 0x7f98240b3200 2024-06-29T15:57:07.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.861+0000 7f98217fa640 1 -- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f9808076510 msgr2=0x7f9808078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:07.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.861+0000 7f98217fa640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f9808076510 0x7f9808078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.911 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:07.910+0000 7f792cc70640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.911 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:07.910+0000 7f792cc70640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:07.910+0000 7f792cc70640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:07.910+0000 7f792cc70640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:07.910+0000 7f792cc70640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:07.910+0000 7f792fc76640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:07.910+0000 7f792fc76640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:07.910+0000 7f792fc76640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:07.910+0000 7f792fc76640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.912 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:07.910+0000 7f792fc76640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:07.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.957+0000 7f982abf5640 1 -- 172.21.15.81:0/3132697120 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f98240c5c90 con 0x7f98240b3200 2024-06-29T15:57:07.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.957+0000 7f98217fa640 1 -- 172.21.15.81:0/3132697120 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v202) ==== 100+0+11 (secure 0 0 0) 0x7f9818064140 con 0x7f98240b3200 2024-06-29T15:57:07.958 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:57:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.959+0000 7f982abf5640 1 -- 172.21.15.81:0/3132697120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98240b3200 msgr2=0x7f98241e0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.959+0000 7f982abf5640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98240b3200 0x7f98241e0330 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f981800c3a0 tx=0x7f981800c860 comp rx=0 tx=0).stop 2024-06-29T15:57:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.959+0000 7f982abf5640 1 -- 172.21.15.81:0/3132697120 shutdown_connections 2024-06-29T15:57:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.959+0000 7f982abf5640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98240d0230 0x7f98241e6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.959+0000 7f982abf5640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:6832/23077685,v1:172.21.15.167:6833/23077685] conn(0x7f9808076510 0x7f9808078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.959+0000 7f982abf5640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98240b4f40 0x7f98241e5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.959+0000 7f982abf5640 1 --2- 172.21.15.81:0/3132697120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98240b3200 0x7f98241e0330 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.959+0000 7f982abf5640 1 -- 172.21.15.81:0/3132697120 >> 172.21.15.81:0/3132697120 conn(0x7f98240b3680 msgr2=0x7f98240701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.960+0000 7f982abf5640 1 -- 172.21.15.81:0/3132697120 shutdown_connections 2024-06-29T15:57:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:07.960+0000 7f982abf5640 1 -- 172.21.15.81:0/3132697120 wait complete. 2024-06-29T15:57:07.971 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:08.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.086+0000 7fd167e26640 1 -- 172.21.15.81:0/2848285771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c003a50 msgr2=0x7fd14c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.086+0000 7fd167e26640 1 --2- 172.21.15.81:0/2848285771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c003a50 0x7fd14c003ec0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009b50 tx=0x7fd14c005830 comp rx=0 tx=0).stop 2024-06-29T15:57:08.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.086+0000 7fd167e26640 1 -- 172.21.15.81:0/2848285771 shutdown_connections 2024-06-29T15:57:08.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.086+0000 7fd167e26640 1 --2- 172.21.15.81:0/2848285771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c003a50 0x7fd14c003ec0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.086+0000 7fd167e26640 1 -- 172.21.15.81:0/2848285771 >> 172.21.15.81:0/2848285771 conn(0x7fd160072950 msgr2=0x7fd1600b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:08.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.086+0000 7fd167e26640 1 -- 172.21.15.81:0/2848285771 shutdown_connections 2024-06-29T15:57:08.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.086+0000 7fd167e26640 1 -- 172.21.15.81:0/2848285771 wait complete. 2024-06-29T15:57:08.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.088+0000 7fd167e26640 1 Processor -- start 2024-06-29T15:57:08.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.088+0000 7fd167e26640 1 -- start start 2024-06-29T15:57:08.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd167e26640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c003a50 0x7fd16015d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd167e26640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd160143d30 0x7fd160156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd167e26640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd160156ba0 0x7fd1601f1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd167e26640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd16015b190 con 0x7fd14c003a50 2024-06-29T15:57:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd167e26640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd16015b010 con 0x7fd160156ba0 2024-06-29T15:57:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd167e26640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd16006f140 con 0x7fd160143d30 2024-06-29T15:57:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd165bc2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c003a50 0x7fd16015d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd165bc2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c003a50 0x7fd16015d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43768/0 (socket says 172.21.15.81:43768) 2024-06-29T15:57:08.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd165bc2640 1 -- 172.21.15.81:0/309088199 learned_addr learned my addr 172.21.15.81:0/309088199 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:08.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd1663c3640 1 --2- 172.21.15.81:0/309088199 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd160156ba0 0x7fd1601f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd1653c1640 1 --2- 172.21.15.81:0/309088199 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd160143d30 0x7fd160156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd165bc2640 1 -- 172.21.15.81:0/309088199 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd160143d30 msgr2=0x7fd160156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd165bc2640 1 --2- 172.21.15.81:0/309088199 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd160143d30 0x7fd160156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd165bc2640 1 -- 172.21.15.81:0/309088199 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd160156ba0 msgr2=0x7fd1601f1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd165bc2640 1 --2- 172.21.15.81:0/309088199 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd160156ba0 0x7fd1601f1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd165bc2640 1 -- 172.21.15.81:0/309088199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16015dc40 con 0x7fd14c003a50 2024-06-29T15:57:08.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd165bc2640 1 --2- 172.21.15.81:0/309088199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c003a50 0x7fd16015d700 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009a70 tx=0x7fd15c0336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:08.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd1567fc640 1 -- 172.21.15.81:0/309088199 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c040020 con 0x7fd14c003a50 2024-06-29T15:57:08.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd1567fc640 1 -- 172.21.15.81:0/309088199 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd15c004110 con 0x7fd14c003a50 2024-06-29T15:57:08.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd167e26640 1 -- 172.21.15.81:0/309088199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd160152ee0 con 0x7fd14c003a50 2024-06-29T15:57:08.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd1567fc640 1 -- 172.21.15.81:0/309088199 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c038040 con 0x7fd14c003a50 2024-06-29T15:57:08.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.090+0000 7fd167e26640 1 -- 172.21.15.81:0/309088199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1601534a0 con 0x7fd14c003a50 2024-06-29T15:57:08.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.091+0000 7fd167e26640 1 -- 172.21.15.81:0/309088199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16006cab0 con 0x7fd14c003a50 2024-06-29T15:57:08.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.092+0000 7fd1567fc640 1 -- 172.21.15.81:0/309088199 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 125) ==== 97912+0+0 (secure 0 0 0) 0x7fd15c0338e0 con 0x7fd14c003a50 2024-06-29T15:57:08.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.093+0000 7fd1567fc640 1 -- 172.21.15.81:0/309088199 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 6669+0+0 (secure 0 0 0) 0x7fd15c0b7a50 con 0x7fd14c003a50 2024-06-29T15:57:08.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.097+0000 7fd1567fc640 1 -- 172.21.15.81:0/309088199 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd15c04a020 con 0x7fd14c003a50 2024-06-29T15:57:08.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.317+0000 7fd167e26640 1 -- 172.21.15.81:0/309088199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1600aebb0 con 0x7fd14c003a50 2024-06-29T15:57:08.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.317+0000 7fd1567fc640 1 -- 172.21.15.81:0/309088199 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5240 (secure 0 0 0) 0x7fd15c089430 con 0x7fd14c003a50 2024-06-29T15:57:08.318 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:08.318 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":103,"btime":"2024-06-29T15:55:41:302970+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1764785504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1764785504},{"type":"v1","addr":"172.21.15.167:6836","nonce":1764785504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99},{"gid":10713,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/1699508727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1699508727},{"type":"v1","addr":"172.21.15.81:6834","nonce":1699508727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/265635924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":265635924},{"type":"v1","addr":"172.21.15.81:6835","nonce":265635924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.649055+0000","modified":"2024-06-29T15:55:40.671516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/1700137710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1700137710},{"type":"v1","addr":"172.21.15.167:6837","nonce":1700137710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10731,"qdb_cluster":[10731]},"id":12}]} 2024-06-29T15:57:08.318 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 103 2024-06-29T15:57:08.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.319+0000 7fd167e26640 1 -- 172.21.15.81:0/309088199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c003a50 msgr2=0x7fd16015d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.319+0000 7fd167e26640 1 --2- 172.21.15.81:0/309088199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c003a50 0x7fd16015d700 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009a70 tx=0x7fd15c0336f0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.319+0000 7fd167e26640 1 -- 172.21.15.81:0/309088199 shutdown_connections 2024-06-29T15:57:08.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.319+0000 7fd167e26640 1 --2- 172.21.15.81:0/309088199 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd160156ba0 0x7fd1601f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.319+0000 7fd167e26640 1 --2- 172.21.15.81:0/309088199 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd160143d30 0x7fd160156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.319+0000 7fd167e26640 1 --2- 172.21.15.81:0/309088199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c003a50 0x7fd16015d700 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.319+0000 7fd167e26640 1 -- 172.21.15.81:0/309088199 >> 172.21.15.81:0/309088199 conn(0x7fd160072950 msgr2=0x7fd160157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:08.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.320+0000 7fd167e26640 1 -- 172.21.15.81:0/309088199 shutdown_connections 2024-06-29T15:57:08.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.320+0000 7fd167e26640 1 -- 172.21.15.81:0/309088199 wait complete. 2024-06-29T15:57:08.331 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:08.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.444+0000 7f9908647640 1 -- 172.21.15.81:0/853766486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99000b4430 msgr2=0x7f99000cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.444+0000 7f9908647640 1 --2- 172.21.15.81:0/853766486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99000b4430 0x7f99000cf130 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f990015aac0 tx=0x7f98f002a4d0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.444+0000 7f9908647640 1 -- 172.21.15.81:0/853766486 shutdown_connections 2024-06-29T15:57:08.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.444+0000 7f9908647640 1 --2- 172.21.15.81:0/853766486 >> v2:172.21.15.167:3300/0 conn(0x7f990006f660 0x7f990006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.444+0000 7f9908647640 1 --2- 172.21.15.81:0/853766486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99000b4430 0x7f99000cf130 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.444+0000 7f9908647640 1 --2- 172.21.15.81:0/853766486 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99000b35b0 0x7f990015b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.444+0000 7f9908647640 1 -- 172.21.15.81:0/853766486 >> 172.21.15.81:0/853766486 conn(0x7f990006ccc0 msgr2=0x7f99000b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:08.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.444+0000 7f9908647640 1 -- 172.21.15.81:0/853766486 shutdown_connections 2024-06-29T15:57:08.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.444+0000 7f9908647640 1 -- 172.21.15.81:0/853766486 wait complete. 2024-06-29T15:57:08.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.446+0000 7f9908647640 1 Processor -- start 2024-06-29T15:57:08.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.446+0000 7f9908647640 1 -- start start 2024-06-29T15:57:08.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9908647640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f990006f660 0x7f9900143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9908647640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99000b35b0 0x7f9900148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9908647640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99000b4430 0x7f99001486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9908647640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f990006f160 con 0x7f99000b4430 2024-06-29T15:57:08.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9908647640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f990006efe0 con 0x7f990006f660 2024-06-29T15:57:08.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9908647640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f990006f2e0 con 0x7f99000b35b0 2024-06-29T15:57:08.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9906be4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99000b4430 0x7f99001486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9906be4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99000b4430 0x7f99001486b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43786/0 (socket says 172.21.15.81:43786) 2024-06-29T15:57:08.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9906be4640 1 -- 172.21.15.81:0/4199027864 learned_addr learned my addr 172.21.15.81:0/4199027864 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:08.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f99063e3640 1 --2- 172.21.15.81:0/4199027864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f990006f660 0x7f9900143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9905be2640 1 --2- 172.21.15.81:0/4199027864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99000b35b0 0x7f9900148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9906be4640 1 -- 172.21.15.81:0/4199027864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99000b35b0 msgr2=0x7f9900148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9906be4640 1 --2- 172.21.15.81:0/4199027864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99000b35b0 0x7f9900148170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9906be4640 1 -- 172.21.15.81:0/4199027864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f990006f660 msgr2=0x7f9900143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9906be4640 1 --2- 172.21.15.81:0/4199027864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f990006f660 0x7f9900143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9906be4640 1 -- 172.21.15.81:0/4199027864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9900143b40 con 0x7f99000b4430 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.447+0000 7f9905be2640 1 --2- 172.21.15.81:0/4199027864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99000b35b0 0x7f9900148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.448+0000 7f9906be4640 1 --2- 172.21.15.81:0/4199027864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99000b4430 0x7f99001486b0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f98f800e990 tx=0x7f98f800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.448+0000 7f98f6ffd640 1 -- 172.21.15.81:0/4199027864 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8004d20 con 0x7f99000b4430 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.448+0000 7f98f6ffd640 1 -- 172.21.15.81:0/4199027864 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98f800ca40 con 0x7f99000b4430 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.448+0000 7f9908647640 1 -- 172.21.15.81:0/4199027864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9900149220 con 0x7f99000b4430 2024-06-29T15:57:08.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.448+0000 7f98f6ffd640 1 -- 172.21.15.81:0/4199027864 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8005020 con 0x7f99000b4430 2024-06-29T15:57:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.448+0000 7f9908647640 1 -- 172.21.15.81:0/4199027864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99001494c0 con 0x7f99000b4430 2024-06-29T15:57:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.449+0000 7f9908647640 1 -- 172.21.15.81:0/4199027864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99000cd2c0 con 0x7f99000b4430 2024-06-29T15:57:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.450+0000 7f98f6ffd640 1 -- 172.21.15.81:0/4199027864 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 125) ==== 97912+0+0 (secure 0 0 0) 0x7f98f80051c0 con 0x7f99000b4430 2024-06-29T15:57:08.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.450+0000 7f98f6ffd640 1 -- 172.21.15.81:0/4199027864 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 6669+0+0 (secure 0 0 0) 0x7f98f8097d40 con 0x7f99000b4430 2024-06-29T15:57:08.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.455+0000 7f98f6ffd640 1 -- 172.21.15.81:0/4199027864 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f99000cd2c0 con 0x7f99000b4430 2024-06-29T15:57:08.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.639+0000 7f9908647640 1 -- 172.21.15.81:0/4199027864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f990015b5b0 con 0x7f99000b4430 2024-06-29T15:57:08.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.640+0000 7f98f6ffd640 1 -- 172.21.15.81:0/4199027864 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) ==== 74+0+18072 (secure 0 0 0) 0x7f98f8016030 con 0x7f99000b4430 2024-06-29T15:57:08.640 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:08.641 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":202,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:07.850685+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.947285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.605625+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.1a","mappings":[{"from":0,"to":2}]},{"pgid":"25.3f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2766539241":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/649722843":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2657542019":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2947012481":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/3123996030":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/339582014":"2024-06-30T15:57:07.850629+0000","172.21.15.167:6832/23077685":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2019373615":"2024-06-30T15:57:07.850629+0000","172.21.15.167:6833/803000291":"2024-06-30T15:55:49.959292+0000","172.21.15.167:6833/23077685":"2024-06-30T15:57:07.850629+0000","172.21.15.167:6832/803000291":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/1482340295":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/2216474049":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/3153968292":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/1095374241":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/3639047578":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/481601843":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2467432753":"2024-06-30T15:55:49.959292+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:08.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.642+0000 7f9908647640 1 -- 172.21.15.81:0/4199027864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99000b4430 msgr2=0x7f99001486b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.642+0000 7f9908647640 1 --2- 172.21.15.81:0/4199027864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99000b4430 0x7f99001486b0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f98f800e990 tx=0x7f98f800ee50 comp rx=0 tx=0).stop 2024-06-29T15:57:08.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.642+0000 7f9908647640 1 -- 172.21.15.81:0/4199027864 shutdown_connections 2024-06-29T15:57:08.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.642+0000 7f9908647640 1 --2- 172.21.15.81:0/4199027864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99000b4430 0x7f99001486b0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.642+0000 7f9908647640 1 --2- 172.21.15.81:0/4199027864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99000b35b0 0x7f9900148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.642+0000 7f9908647640 1 --2- 172.21.15.81:0/4199027864 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f990006f660 0x7f9900143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.642+0000 7f9908647640 1 -- 172.21.15.81:0/4199027864 >> 172.21.15.81:0/4199027864 conn(0x7f990006ccc0 msgr2=0x7f990015ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:08.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.642+0000 7f9908647640 1 -- 172.21.15.81:0/4199027864 shutdown_connections 2024-06-29T15:57:08.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.642+0000 7f9908647640 1 -- 172.21.15.81:0/4199027864 wait complete. 2024-06-29T15:57:08.654 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:57:08.654 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:08.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.769+0000 7f8695210640 1 -- 172.21.15.81:0/708866272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f867c003ff0 msgr2=0x7f867c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.769+0000 7f8695210640 1 --2- 172.21.15.81:0/708866272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f867c003ff0 0x7f867c004800 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f8678009b50 tx=0x7f867c005830 comp rx=0 tx=0).stop 2024-06-29T15:57:08.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.769+0000 7f868f59e640 1 -- 172.21.15.81:0/708866272 reap_dead start 2024-06-29T15:57:08.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.769+0000 7f8695210640 1 -- 172.21.15.81:0/708866272 shutdown_connections 2024-06-29T15:57:08.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.769+0000 7f8695210640 1 -- 172.21.15.81:0/708866272 >> 172.21.15.81:0/708866272 conn(0x7f869006ccc0 msgr2=0x7f86900b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:08.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.769+0000 7f8695210640 1 -- 172.21.15.81:0/708866272 shutdown_connections 2024-06-29T15:57:08.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.769+0000 7f8695210640 1 -- 172.21.15.81:0/708866272 wait complete. 2024-06-29T15:57:08.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.771+0000 7f8695210640 1 Processor -- start 2024-06-29T15:57:08.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.771+0000 7f8695210640 1 -- start start 2024-06-29T15:57:08.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.772+0000 7f8695210640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f867c003ff0 0x7f8690130ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.772+0000 7f8695210640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8690140960 0x7f8690131000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.772+0000 7f8695210640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f869015a660 0x7f8690159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.772+0000 7f8695210640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8690073440 con 0x7f869015a660 2024-06-29T15:57:08.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.772+0000 7f8695210640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f86900cf6f0 con 0x7f867c003ff0 2024-06-29T15:57:08.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.772+0000 7f8695210640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f86900735c0 con 0x7f8690140960 2024-06-29T15:57:08.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.772+0000 7f868f59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f869015a660 0x7f8690159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.772+0000 7f868f59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f869015a660 0x7f8690159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43808/0 (socket says 172.21.15.81:43808) 2024-06-29T15:57:08.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.772+0000 7f868f59e640 1 -- 172.21.15.81:0/2725752788 learned_addr learned my addr 172.21.15.81:0/2725752788 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:08.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f8687fff640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8690140960 0x7f8690131000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f868ed9d640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f867c003ff0 0x7f8690130ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f868f59e640 1 -- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8690140960 msgr2=0x7f8690131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f868f59e640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8690140960 0x7f8690131000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f868f59e640 1 -- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f867c003ff0 msgr2=0x7f8690130ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:08.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f868f59e640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f867c003ff0 0x7f8690130ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:08.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f868f59e640 1 -- 172.21.15.81:0/2725752788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f869012f130 con 0x7f869015a660 2024-06-29T15:57:08.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f868ed9d640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f867c003ff0 0x7f8690130ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:08.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f868f59e640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f869015a660 0x7f8690159a40 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f8690071130 tx=0x7f868800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:08.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f86877fe640 1 -- 172.21.15.81:0/2725752788 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688004b20 con 0x7f869015a660 2024-06-29T15:57:08.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f86877fe640 1 -- 172.21.15.81:0/2725752788 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8688004cc0 con 0x7f869015a660 2024-06-29T15:57:08.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f8695210640 1 -- 172.21.15.81:0/2725752788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8690131620 con 0x7f869015a660 2024-06-29T15:57:08.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f86877fe640 1 -- 172.21.15.81:0/2725752788 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688003bd0 con 0x7f869015a660 2024-06-29T15:57:08.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.773+0000 7f8695210640 1 -- 172.21.15.81:0/2725752788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8690142e20 con 0x7f869015a660 2024-06-29T15:57:08.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.774+0000 7f8695210640 1 -- 172.21.15.81:0/2725752788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f869015d770 con 0x7f869015a660 2024-06-29T15:57:08.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.775+0000 7f86877fe640 1 -- 172.21.15.81:0/2725752788 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 125) ==== 97912+0+0 (secure 0 0 0) 0x7f8688004030 con 0x7f869015a660 2024-06-29T15:57:08.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.775+0000 7f86877fe640 1 -- 172.21.15.81:0/2725752788 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 6669+0+0 (secure 0 0 0) 0x7f8688097e20 con 0x7f869015a660 2024-06-29T15:57:08.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.779+0000 7f86877fe640 1 -- 172.21.15.81:0/2725752788 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f869015d770 con 0x7f869015a660 2024-06-29T15:57:08.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.860+0000 7f86877fe640 1 -- 172.21.15.81:0/2725752788 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 126) ==== 98039+0+0 (secure 0 0 0) 0x7f868805ca80 con 0x7f869015a660 2024-06-29T15:57:08.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.861+0000 7f86877fe640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f867007bd70 0x7f867007e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:08.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.864+0000 7f868ed9d640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f867007bd70 0x7f867007e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:08.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.866+0000 7f868ed9d640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f867007bd70 0x7f867007e190 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8680000fb0 tx=0x7f8680001e10 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:08.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:08.999+0000 7f8695210640 1 -- 172.21.15.81:0/2725752788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8690143080 con 0x7f869015a660 2024-06-29T15:57:09.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.000+0000 7f86877fe640 1 -- 172.21.15.81:0/2725752788 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5240 (secure 0 0 0) 0x7f8688061810 con 0x7f869015a660 2024-06-29T15:57:09.000 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:09.000 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":103,"btime":"2024-06-29T15:55:41:302970+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1764785504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1764785504},{"type":"v1","addr":"172.21.15.167:6836","nonce":1764785504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99},{"gid":10713,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/1699508727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1699508727},{"type":"v1","addr":"172.21.15.81:6834","nonce":1699508727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/265635924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":265635924},{"type":"v1","addr":"172.21.15.81:6835","nonce":265635924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.649055+0000","modified":"2024-06-29T15:55:40.671516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/1700137710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1700137710},{"type":"v1","addr":"172.21.15.167:6837","nonce":1700137710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10731,"qdb_cluster":[10731]},"id":12}]} 2024-06-29T15:57:09.000 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 103 2024-06-29T15:57:09.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 -- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f867007bd70 msgr2=0x7f867007e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f867007bd70 0x7f867007e190 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8680000fb0 tx=0x7f8680001e10 comp rx=0 tx=0).stop 2024-06-29T15:57:09.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 -- 172.21.15.81:0/2725752788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f869015a660 msgr2=0x7f8690159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f869015a660 0x7f8690159a40 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f8690071130 tx=0x7f868800f3c0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 -- 172.21.15.81:0/2725752788 shutdown_connections 2024-06-29T15:57:09.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f867007bd70 0x7f867007e190 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f869015a660 0x7f8690159a40 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8690140960 0x7f8690131000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 --2- 172.21.15.81:0/2725752788 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f867c003ff0 0x7f8690130ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 -- 172.21.15.81:0/2725752788 >> 172.21.15.81:0/2725752788 conn(0x7f869006ccc0 msgr2=0x7f869015c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 -- 172.21.15.81:0/2725752788 shutdown_connections 2024-06-29T15:57:09.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.002+0000 7f8695210640 1 -- 172.21.15.81:0/2725752788 wait complete. 2024-06-29T15:57:09.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:09.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.126+0000 7f930094a640 1 -- 172.21.15.81:0/402789158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc06f660 msgr2=0x7f92fc06fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.126+0000 7f930094a640 1 --2- 172.21.15.81:0/402789158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc06f660 0x7f92fc06fa70 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f92fc070220 tx=0x7f92f402b880 comp rx=0 tx=0).stop 2024-06-29T15:57:09.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.126+0000 7f930094a640 1 -- 172.21.15.81:0/402789158 shutdown_connections 2024-06-29T15:57:09.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.126+0000 7f930094a640 1 --2- 172.21.15.81:0/402789158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc06f660 0x7f92fc06fa70 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.126+0000 7f930094a640 1 --2- 172.21.15.81:0/402789158 >> v2:172.21.15.167:3300/0 conn(0x7f92fc0b35b0 0x7f92fc15b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.126+0000 7f930094a640 1 -- 172.21.15.81:0/402789158 >> 172.21.15.81:0/402789158 conn(0x7f92fc06ccc0 msgr2=0x7f92fc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.126+0000 7f930094a640 1 -- 172.21.15.81:0/402789158 shutdown_connections 2024-06-29T15:57:09.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.126+0000 7f930094a640 1 -- 172.21.15.81:0/402789158 wait complete. 2024-06-29T15:57:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.128+0000 7f930094a640 1 Processor -- start 2024-06-29T15:57:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.128+0000 7f930094a640 1 -- start start 2024-06-29T15:57:09.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f930094a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc06f660 0x7f92fc14f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f930094a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92fc0b35b0 0x7f92fc139300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f930094a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92fc0b4430 0x7f92fc139ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f930094a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc06f160 con 0x7f92fc06f660 2024-06-29T15:57:09.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f930094a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc06efe0 con 0x7f92fc0b35b0 2024-06-29T15:57:09.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f930094a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f92fc06f2e0 con 0x7f92fc0b4430 2024-06-29T15:57:09.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f92fa59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc06f660 0x7f92fc14f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f92fa59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc06f660 0x7f92fc14f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43834/0 (socket says 172.21.15.81:43834) 2024-06-29T15:57:09.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f92fa59c640 1 -- 172.21.15.81:0/1994543379 learned_addr learned my addr 172.21.15.81:0/1994543379 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f92f1d9b640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92fc0b35b0 0x7f92fc139300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.129+0000 7f92fad9d640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92fc0b4430 0x7f92fc139ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f92fa59c640 1 -- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92fc0b4430 msgr2=0x7f92fc139ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f92fa59c640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92fc0b4430 0x7f92fc139ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f92fa59c640 1 -- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92fc0b35b0 msgr2=0x7f92fc139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f92fa59c640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92fc0b35b0 0x7f92fc139300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f92fa59c640 1 -- 172.21.15.81:0/1994543379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92fc14fcd0 con 0x7f92fc06f660 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f92fa59c640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc06f660 0x7f92fc14f790 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00c9a0 tx=0x7f92ec00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f92f37fe640 1 -- 172.21.15.81:0/1994543379 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec007b50 con 0x7f92fc06f660 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f92f37fe640 1 -- 172.21.15.81:0/1994543379 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92ec007e00 con 0x7f92fc06f660 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f930094a640 1 -- 172.21.15.81:0/1994543379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92fc14e6f0 con 0x7f92fc06f660 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f92f37fe640 1 -- 172.21.15.81:0/1994543379 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec005280 con 0x7f92fc06f660 2024-06-29T15:57:09.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.130+0000 7f930094a640 1 -- 172.21.15.81:0/1994543379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92fc14ecb0 con 0x7f92fc06f660 2024-06-29T15:57:09.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.131+0000 7f930094a640 1 -- 172.21.15.81:0/1994543379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92fc06fa70 con 0x7f92fc06f660 2024-06-29T15:57:09.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.132+0000 7f92f37fe640 1 -- 172.21.15.81:0/1994543379 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 126) ==== 98039+0+0 (secure 0 0 0) 0x7f92ec005420 con 0x7f92fc06f660 2024-06-29T15:57:09.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.132+0000 7f92f37fe640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f92dc076120 0x7f92dc078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.132+0000 7f92f37fe640 1 -- 172.21.15.81:0/1994543379 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 6669+0+0 (secure 0 0 0) 0x7f92ec097a80 con 0x7f92fc06f660 2024-06-29T15:57:09.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.132+0000 7f92f1d9b640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f92dc076120 0x7f92dc078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.133+0000 7f92f1d9b640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f92dc076120 0x7f92dc078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f92fc14ffc0 tx=0x7f92e4009060 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.137+0000 7f92f37fe640 1 -- 172.21.15.81:0/1994543379 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f92ec014030 con 0x7f92fc06f660 2024-06-29T15:57:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.356+0000 7f930094a640 1 -- 172.21.15.81:0/1994543379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92fc072c90 con 0x7f92fc06f660 2024-06-29T15:57:09.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.357+0000 7f92f37fe640 1 -- 172.21.15.81:0/1994543379 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5240 (secure 0 0 0) 0x7f92ec069350 con 0x7f92fc06f660 2024-06-29T15:57:09.357 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:09.357 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":103,"btime":"2024-06-29T15:55:41:302970+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1764785504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1764785504},{"type":"v1","addr":"172.21.15.167:6836","nonce":1764785504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99},{"gid":10713,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/1699508727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1699508727},{"type":"v1","addr":"172.21.15.81:6834","nonce":1699508727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/265635924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":265635924},{"type":"v1","addr":"172.21.15.81:6835","nonce":265635924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:55:37.649055+0000","modified":"2024-06-29T15:55:40.671516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10731},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10731":{"gid":10731,"name":"d","rank":0,"incarnation":100,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/1700137710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1700137710},{"type":"v1","addr":"172.21.15.167:6837","nonce":1700137710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10731,"qdb_cluster":[10731]},"id":12}]} 2024-06-29T15:57:09.357 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 103 2024-06-29T15:57:09.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 -- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f92dc076120 msgr2=0x7f92dc078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f92dc076120 0x7f92dc078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f92fc14ffc0 tx=0x7f92e4009060 comp rx=0 tx=0).stop 2024-06-29T15:57:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 -- 172.21.15.81:0/1994543379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc06f660 msgr2=0x7f92fc14f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc06f660 0x7f92fc14f790 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00c9a0 tx=0x7f92ec00ce60 comp rx=0 tx=0).stop 2024-06-29T15:57:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 -- 172.21.15.81:0/1994543379 shutdown_connections 2024-06-29T15:57:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f92dc076120 0x7f92dc078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92fc0b4430 0x7f92fc139ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92fc0b35b0 0x7f92fc139300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 --2- 172.21.15.81:0/1994543379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc06f660 0x7f92fc14f790 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 -- 172.21.15.81:0/1994543379 >> 172.21.15.81:0/1994543379 conn(0x7f92fc06ccc0 msgr2=0x7f92fc14d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 -- 172.21.15.81:0/1994543379 shutdown_connections 2024-06-29T15:57:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.359+0000 7f930094a640 1 -- 172.21.15.81:0/1994543379 wait complete. 2024-06-29T15:57:09.370 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:09.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.484+0000 7fb20a50a640 1 -- 172.21.15.81:0/2048943383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1ec0053a0 msgr2=0x7fb1ec00ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.484+0000 7fb20a50a640 1 --2- 172.21.15.81:0/2048943383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1ec0053a0 0x7fb1ec00ba50 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc001dc0 tx=0x7fb1ec005830 comp rx=0 tx=0).stop 2024-06-29T15:57:09.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.484+0000 7fb20a50a640 1 -- 172.21.15.81:0/2048943383 shutdown_connections 2024-06-29T15:57:09.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.484+0000 7fb20a50a640 1 --2- 172.21.15.81:0/2048943383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1ec0053a0 0x7fb1ec00ba50 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.484+0000 7fb20a50a640 1 -- 172.21.15.81:0/2048943383 >> 172.21.15.81:0/2048943383 conn(0x7fb2040b3680 msgr2=0x7fb2040b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.484+0000 7fb20a50a640 1 -- 172.21.15.81:0/2048943383 shutdown_connections 2024-06-29T15:57:09.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.484+0000 7fb20a50a640 1 -- 172.21.15.81:0/2048943383 wait complete. 2024-06-29T15:57:09.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.486+0000 7fb20a50a640 1 Processor -- start 2024-06-29T15:57:09.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.486+0000 7fb20a50a640 1 -- start start 2024-06-29T15:57:09.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.487+0000 7fb20a50a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1ec0053a0 0x7fb20409d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.487+0000 7fb20a50a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2041e3290 0x7fb2040ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.487+0000 7fb20a50a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2040ca8e0 0x7fb2041d9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.487+0000 7fb20a50a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2040d05c0 con 0x7fb2041e3290 2024-06-29T15:57:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.487+0000 7fb20a50a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb2040d0440 con 0x7fb2040ca8e0 2024-06-29T15:57:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.487+0000 7fb20a50a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb2040d0740 con 0x7fb1ec0053a0 2024-06-29T15:57:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.487+0000 7fb2037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2041e3290 0x7fb2040ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb2037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2041e3290 0x7fb2040ca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43852/0 (socket says 172.21.15.81:43852) 2024-06-29T15:57:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb2037fe640 1 -- 172.21.15.81:0/379143522 learned_addr learned my addr 172.21.15.81:0/379143522 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:09.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb2037fe640 1 -- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1ec0053a0 msgr2=0x7fb20409d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb2037fe640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1ec0053a0 0x7fb20409d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb208aa7640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2040ca8e0 0x7fb2041d9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb2037fe640 1 -- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2040ca8e0 msgr2=0x7fb2041d9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb2037fe640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2040ca8e0 0x7fb2041d9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb2037fe640 1 -- 172.21.15.81:0/379143522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2040c9300 con 0x7fb2041e3290 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb2037fe640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2041e3290 0x7fb2040ca3a0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800e5c0 tx=0x7fb1f800ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb200ff9640 1 -- 172.21.15.81:0/379143522 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8015070 con 0x7fb2041e3290 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb200ff9640 1 -- 172.21.15.81:0/379143522 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1f800ca60 con 0x7fb2041e3290 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb20a50a640 1 -- 172.21.15.81:0/379143522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2040c95e0 con 0x7fb2041e3290 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb200ff9640 1 -- 172.21.15.81:0/379143522 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800ce90 con 0x7fb2041e3290 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.488+0000 7fb20a50a640 1 -- 172.21.15.81:0/379143522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2040c9ba0 con 0x7fb2041e3290 2024-06-29T15:57:09.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.489+0000 7fb20a50a640 1 -- 172.21.15.81:0/379143522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2041e4000 con 0x7fb2041e3290 2024-06-29T15:57:09.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.490+0000 7fb200ff9640 1 -- 172.21.15.81:0/379143522 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 126) ==== 98039+0+0 (secure 0 0 0) 0x7fb1f8018070 con 0x7fb2041e3290 2024-06-29T15:57:09.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.490+0000 7fb200ff9640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb1f00762c0 0x7fb1f00787d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.491+0000 7fb200ff9640 1 -- 172.21.15.81:0/379143522 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 6669+0+0 (secure 0 0 0) 0x7fb1f8097af0 con 0x7fb2041e3290 2024-06-29T15:57:09.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.491+0000 7fb203fff640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb1f00762c0 0x7fb1f00787d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.491+0000 7fb203fff640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb1f00762c0 0x7fb1f00787d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400d9b0 tx=0x7fb1f400d370 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.495+0000 7fb200ff9640 1 -- 172.21.15.81:0/379143522 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb2041e4000 con 0x7fb2041e3290 2024-06-29T15:57:09.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.677+0000 7fb20a50a640 1 -- 172.21.15.81:0/379143522 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb2040ce520 con 0x7fb2041e3290 2024-06-29T15:57:09.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.679+0000 7fb200ff9640 1 -- 172.21.15.81:0/379143522 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) ==== 74+0+18072 (secure 0 0 0) 0x7fb1f8061340 con 0x7fb2041e3290 2024-06-29T15:57:09.679 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:09.679 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":202,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:07.850685+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:55:35.947285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-06-29T15:55:36.605625+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"201","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":201,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.1a","mappings":[{"from":0,"to":2}]},{"pgid":"25.3f","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2766539241":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/649722843":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2657542019":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2947012481":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/3123996030":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/339582014":"2024-06-30T15:57:07.850629+0000","172.21.15.167:6832/23077685":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2019373615":"2024-06-30T15:57:07.850629+0000","172.21.15.167:6833/803000291":"2024-06-30T15:55:49.959292+0000","172.21.15.167:6833/23077685":"2024-06-30T15:57:07.850629+0000","172.21.15.167:6832/803000291":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/1482340295":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/2216474049":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/3153968292":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/1095374241":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/3639047578":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/481601843":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2467432753":"2024-06-30T15:55:49.959292+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:09.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.680+0000 7fb20a50a640 1 -- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb1f00762c0 msgr2=0x7fb1f00787d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb1f00762c0 0x7fb1f00787d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400d9b0 tx=0x7fb1f400d370 comp rx=0 tx=0).stop 2024-06-29T15:57:09.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 -- 172.21.15.81:0/379143522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2041e3290 msgr2=0x7fb2040ca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2041e3290 0x7fb2040ca3a0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800e5c0 tx=0x7fb1f800ea80 comp rx=0 tx=0).stop 2024-06-29T15:57:09.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 -- 172.21.15.81:0/379143522 shutdown_connections 2024-06-29T15:57:09.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb1f00762c0 0x7fb1f00787d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2040ca8e0 0x7fb2041d9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2041e3290 0x7fb2040ca3a0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 --2- 172.21.15.81:0/379143522 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1ec0053a0 0x7fb20409d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 -- 172.21.15.81:0/379143522 >> 172.21.15.81:0/379143522 conn(0x7fb2040b3680 msgr2=0x7fb2041e38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 -- 172.21.15.81:0/379143522 shutdown_connections 2024-06-29T15:57:09.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.681+0000 7fb20a50a640 1 -- 172.21.15.81:0/379143522 wait complete. 2024-06-29T15:57:09.693 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:57:09.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.807+0000 7f72b3143640 1 -- 172.21.15.81:0/98384852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72940051a0 msgr2=0x7f729400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.807+0000 7f72b3143640 1 --2- 172.21.15.81:0/98384852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72940051a0 0x7f729400b9c0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f72ac08cf40 tx=0x7f7294005830 comp rx=0 tx=0).stop 2024-06-29T15:57:09.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.807+0000 7f72b16e0640 1 -- 172.21.15.81:0/98384852 reap_dead start 2024-06-29T15:57:09.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.807+0000 7f72b3143640 1 -- 172.21.15.81:0/98384852 shutdown_connections 2024-06-29T15:57:09.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.807+0000 7f72b3143640 1 -- 172.21.15.81:0/98384852 >> 172.21.15.81:0/98384852 conn(0x7f72ac09cf30 msgr2=0x7f72ac0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:09.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.807+0000 7f72b3143640 1 -- 172.21.15.81:0/98384852 shutdown_connections 2024-06-29T15:57:09.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.807+0000 7f72b3143640 1 -- 172.21.15.81:0/98384852 wait complete. 2024-06-29T15:57:09.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.809+0000 7f72b3143640 1 Processor -- start 2024-06-29T15:57:09.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.809+0000 7f72b3143640 1 -- start start 2024-06-29T15:57:09.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.810+0000 7f72b3143640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72ac0b3520 0x7f72ac0b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.810+0000 7f72b3143640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72ac153fb0 0x7f72ac0cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.810+0000 7f72b3143640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72ac1543c0 0x7f72ac0cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.810+0000 7f72b3143640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac0b8420 con 0x7f72ac153fb0 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.810+0000 7f72b3143640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac0b82a0 con 0x7f72ac1543c0 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.810+0000 7f72b3143640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f72ac0b85a0 con 0x7f72ac0b3520 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.810+0000 7f72a3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72ac153fb0 0x7f72ac0cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.810+0000 7f72a3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72ac153fb0 0x7f72ac0cce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43860/0 (socket says 172.21.15.81:43860) 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.810+0000 7f72a3fff640 1 -- 172.21.15.81:0/580846869 learned_addr learned my addr 172.21.15.81:0/580846869 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72b0edf640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72ac0b3520 0x7f72ac0b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72b16e0640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72ac1543c0 0x7f72ac0cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72a3fff640 1 -- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72ac0b3520 msgr2=0x7f72ac0b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72a3fff640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72ac0b3520 0x7f72ac0b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72a3fff640 1 -- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72ac1543c0 msgr2=0x7f72ac0cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:09.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72a3fff640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72ac1543c0 0x7f72ac0cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:09.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72a3fff640 1 -- 172.21.15.81:0/580846869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ac0b3b20 con 0x7f72ac153fb0 2024-06-29T15:57:09.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72a3fff640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72ac153fb0 0x7f72ac0cce90 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f729c0118e0 tx=0x7f729c011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72a17fa640 1 -- 172.21.15.81:0/580846869 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c00f9a0 con 0x7f72ac153fb0 2024-06-29T15:57:09.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72a17fa640 1 -- 172.21.15.81:0/580846869 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f729c00fb40 con 0x7f72ac153fb0 2024-06-29T15:57:09.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72b3143640 1 -- 172.21.15.81:0/580846869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72ac0b3e00 con 0x7f72ac153fb0 2024-06-29T15:57:09.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72a17fa640 1 -- 172.21.15.81:0/580846869 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c005020 con 0x7f72ac153fb0 2024-06-29T15:57:09.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.811+0000 7f72b3143640 1 -- 172.21.15.81:0/580846869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72ac0b43c0 con 0x7f72ac153fb0 2024-06-29T15:57:09.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.812+0000 7f72b3143640 1 -- 172.21.15.81:0/580846869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72ac0d0480 con 0x7f72ac153fb0 2024-06-29T15:57:09.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.813+0000 7f72a17fa640 1 -- 172.21.15.81:0/580846869 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 126) ==== 98039+0+0 (secure 0 0 0) 0x7f729c0051c0 con 0x7f72ac153fb0 2024-06-29T15:57:09.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.813+0000 7f72a17fa640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f7298076120 0x7f7298078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:09.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.813+0000 7f72a17fa640 1 -- 172.21.15.81:0/580846869 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 6669+0+0 (secure 0 0 0) 0x7f729c098400 con 0x7f72ac153fb0 2024-06-29T15:57:09.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.813+0000 7f72b0edf640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f7298076120 0x7f7298078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:09.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.814+0000 7f72b0edf640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f7298076120 0x7f7298078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f72a8000fd0 tx=0x7f72a80035f0 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:09.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:09.818+0000 7f72a17fa640 1 -- 172.21.15.81:0/580846869 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f729c061ce0 con 0x7f72ac153fb0 2024-06-29T15:57:10.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.041+0000 7f72b3143640 1 -- 172.21.15.81:0/580846869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f72ac06cda0 con 0x7f72ac153fb0 2024-06-29T15:57:10.041 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T15:57:10.041+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:57:10.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.045+0000 7f72a17fa640 1 -- 172.21.15.81:0/580846869 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v104) ==== 161+0+0 (secure 0 0 0) 0x7f729c0615e0 con 0x7f72ac153fb0 2024-06-29T15:57:10.045 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:57:10.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 -- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f7298076120 msgr2=0x7f7298078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:10.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f7298076120 0x7f7298078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f72a8000fd0 tx=0x7f72a80035f0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 -- 172.21.15.81:0/580846869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72ac153fb0 msgr2=0x7f72ac0cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:10.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72ac153fb0 0x7f72ac0cce90 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f729c0118e0 tx=0x7f729c011da0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 -- 172.21.15.81:0/580846869 shutdown_connections 2024-06-29T15:57:10.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f7298076120 0x7f7298078630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72ac1543c0 0x7f72ac0cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72ac153fb0 0x7f72ac0cce90 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 --2- 172.21.15.81:0/580846869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72ac0b3520 0x7f72ac0b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 -- 172.21.15.81:0/580846869 >> 172.21.15.81:0/580846869 conn(0x7f72ac09cf30 msgr2=0x7f72ac161390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:10.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 -- 172.21.15.81:0/580846869 shutdown_connections 2024-06-29T15:57:10.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.047+0000 7f72b3143640 1 -- 172.21.15.81:0/580846869 wait complete. 2024-06-29T15:57:10.059 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:57:10.111 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:57:10.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.174+0000 7f97a9681640 1 -- 172.21.15.81:0/4055648772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a415f670 msgr2=0x7f97a40ccb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:10.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.174+0000 7f97a9681640 1 --2- 172.21.15.81:0/4055648772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a415f670 0x7f97a40ccb00 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f978c005fd0 tx=0x7f978c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.174+0000 7f97a9681640 1 -- 172.21.15.81:0/4055648772 shutdown_connections 2024-06-29T15:57:10.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.174+0000 7f97a9681640 1 --2- 172.21.15.81:0/4055648772 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a415f670 0x7f97a40ccb00 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.174+0000 7f97a9681640 1 --2- 172.21.15.81:0/4055648772 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97a40cd760 0x7f97a40cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.174+0000 7f97a9681640 1 -- 172.21.15.81:0/4055648772 >> 172.21.15.81:0/4055648772 conn(0x7f97a40b3680 msgr2=0x7f97a40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:10.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.175+0000 7f97a9681640 1 -- 172.21.15.81:0/4055648772 shutdown_connections 2024-06-29T15:57:10.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.175+0000 7f97a9681640 1 -- 172.21.15.81:0/4055648772 wait complete. 2024-06-29T15:57:10.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.176+0000 7f97a9681640 1 Processor -- start 2024-06-29T15:57:10.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.176+0000 7f97a9681640 1 -- start start 2024-06-29T15:57:10.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f97a9681640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97a40cd040 0x7f97a4158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f97a9681640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97a40cd760 0x7f97a4156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f97a9681640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a415f670 0x7f97a41451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f97a9681640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f97a40cec90 con 0x7f97a40cd760 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f97a9681640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f97a40ceb10 con 0x7f97a415f670 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f97a9681640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f97a40cee10 con 0x7f97a40cd040 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f979a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97a40cd760 0x7f97a4156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f979a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97a40cd760 0x7f97a4156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42536/0 (socket says 172.21.15.81:42536) 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f979a7fc640 1 -- 172.21.15.81:0/117101355 learned_addr learned my addr 172.21.15.81:0/117101355 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f97a2ffd640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97a40cd040 0x7f97a4158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f97a37fe640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a415f670 0x7f97a41451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f979a7fc640 1 -- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97a40cd040 msgr2=0x7f97a4158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f979a7fc640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97a40cd040 0x7f97a4158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f979a7fc640 1 -- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a415f670 msgr2=0x7f97a41451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:10.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f979a7fc640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a415f670 0x7f97a41451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:10.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f979a7fc640 1 -- 172.21.15.81:0/117101355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a4158230 con 0x7f97a40cd760 2024-06-29T15:57:10.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f97a37fe640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a415f670 0x7f97a41451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:10.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f979a7fc640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97a40cd760 0x7f97a4156ff0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f978c000c00 tx=0x7f978c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:10.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.178+0000 7f979bfff640 1 -- 172.21.15.81:0/117101355 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978c03e030 con 0x7f97a40cd760 2024-06-29T15:57:10.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.179+0000 7f97a9681640 1 -- 172.21.15.81:0/117101355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a40c8e80 con 0x7f97a40cd760 2024-06-29T15:57:10.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.179+0000 7f979bfff640 1 -- 172.21.15.81:0/117101355 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f978c003ae0 con 0x7f97a40cd760 2024-06-29T15:57:10.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.179+0000 7f979bfff640 1 -- 172.21.15.81:0/117101355 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978c035040 con 0x7f97a40cd760 2024-06-29T15:57:10.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.179+0000 7f97a9681640 1 -- 172.21.15.81:0/117101355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a41587b0 con 0x7f97a40cd760 2024-06-29T15:57:10.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.179+0000 7f97a9681640 1 -- 172.21.15.81:0/117101355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a40d0bd0 con 0x7f97a40cd760 2024-06-29T15:57:10.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.180+0000 7f979bfff640 1 -- 172.21.15.81:0/117101355 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 126) ==== 98039+0+0 (secure 0 0 0) 0x7f978c041030 con 0x7f97a40cd760 2024-06-29T15:57:10.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.181+0000 7f979bfff640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9784076150 0x7f9784078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:10.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.181+0000 7f979bfff640 1 -- 172.21.15.81:0/117101355 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 6755+0+0 (secure 0 0 0) 0x7f978c0b7c90 con 0x7f97a40cd760 2024-06-29T15:57:10.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.181+0000 7f97a2ffd640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9784076150 0x7f9784078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:10.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.182+0000 7f97a2ffd640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9784076150 0x7f9784078660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97940095f0 tx=0x7f9794005f90 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:10.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.186+0000 7f979bfff640 1 -- 172.21.15.81:0/117101355 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f978c080db0 con 0x7f97a40cd760 2024-06-29T15:57:10.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.199+0000 7f979bfff640 1 -- 172.21.15.81:0/117101355 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 127) ==== 98247+0+0 (secure 0 0 0) 0x7f978c07d040 con 0x7f97a40cd760 2024-06-29T15:57:10.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:10.405+0000 7f97a9681640 1 -- 172.21.15.81:0/117101355 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f97a40c8c10 con 0x7f97a40cd760 2024-06-29T15:57:11.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.047+0000 7f979bfff640 1 -- 172.21.15.81:0/117101355 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v105) ==== 104+0+0 (secure 0 0 0) 0x7f978c089480 con 0x7f97a40cd760 2024-06-29T15:57:11.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.049+0000 7f97a9681640 1 -- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9784076150 msgr2=0x7f9784078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:11.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.049+0000 7f97a9681640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9784076150 0x7f9784078660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97940095f0 tx=0x7f9794005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:11.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.049+0000 7f97a9681640 1 -- 172.21.15.81:0/117101355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97a40cd760 msgr2=0x7f97a4156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:11.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.049+0000 7f97a9681640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97a40cd760 0x7f97a4156ff0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f978c000c00 tx=0x7f978c0035f0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.050+0000 7f97a9681640 1 -- 172.21.15.81:0/117101355 shutdown_connections 2024-06-29T15:57:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.050+0000 7f97a9681640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9784076150 0x7f9784078660 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.050+0000 7f97a9681640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f97a415f670 0x7f97a41451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.050+0000 7f97a9681640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97a40cd760 0x7f97a4156ff0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.050+0000 7f97a9681640 1 --2- 172.21.15.81:0/117101355 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f97a40cd040 0x7f97a4158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.050+0000 7f97a9681640 1 -- 172.21.15.81:0/117101355 >> 172.21.15.81:0/117101355 conn(0x7f97a40b3680 msgr2=0x7f97a40c82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.050+0000 7f97a9681640 1 -- 172.21.15.81:0/117101355 shutdown_connections 2024-06-29T15:57:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.050+0000 7f97a9681640 1 -- 172.21.15.81:0/117101355 wait complete. 2024-06-29T15:57:11.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:57:11.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.184+0000 7f6f18973640 1 -- 172.21.15.81:0/3250046801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f00003c10 msgr2=0x7f6f00004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:11.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.184+0000 7f6f18973640 1 --2- 172.21.15.81:0/3250046801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f00003c10 0x7f6f00004060 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400c760 tx=0x7f6f00005830 comp rx=0 tx=0).stop 2024-06-29T15:57:11.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.184+0000 7f6f12d9d640 1 -- 172.21.15.81:0/3250046801 reap_dead start 2024-06-29T15:57:11.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.184+0000 7f6f18973640 1 -- 172.21.15.81:0/3250046801 shutdown_connections 2024-06-29T15:57:11.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.184+0000 7f6f18973640 1 -- 172.21.15.81:0/3250046801 >> 172.21.15.81:0/3250046801 conn(0x7f6f1406cd60 msgr2=0x7f6f140b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:11.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.184+0000 7f6f18973640 1 -- 172.21.15.81:0/3250046801 shutdown_connections 2024-06-29T15:57:11.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.184+0000 7f6f18973640 1 -- 172.21.15.81:0/3250046801 wait complete. 2024-06-29T15:57:11.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.186+0000 7f6f18973640 1 Processor -- start 2024-06-29T15:57:11.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.186+0000 7f6f18973640 1 -- start start 2024-06-29T15:57:11.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f18973640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f00003c10 0x7f6f14136c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:11.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f18973640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f140aed20 0x7f6f14155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:11.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f18973640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f141f4730 0x7f6f141f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:11.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f18973640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1406fcf0 con 0x7f6f140aed20 2024-06-29T15:57:11.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f18973640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1406fb70 con 0x7f6f141f4730 2024-06-29T15:57:11.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f18973640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1406fe70 con 0x7f6f00003c10 2024-06-29T15:57:11.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f11d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f140aed20 0x7f6f14155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:11.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f11d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f140aed20 0x7f6f14155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42564/0 (socket says 172.21.15.81:42564) 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f11d9b640 1 -- 172.21.15.81:0/4046851785 learned_addr learned my addr 172.21.15.81:0/4046851785 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f12d9d640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f141f4730 0x7f6f141f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.187+0000 7f6f1259c640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f00003c10 0x7f6f14136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f11d9b640 1 -- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f00003c10 msgr2=0x7f6f14136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f11d9b640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f00003c10 0x7f6f14136c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f11d9b640 1 -- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f141f4730 msgr2=0x7f6f141f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f11d9b640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f141f4730 0x7f6f141f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f11d9b640 1 -- 172.21.15.81:0/4046851785 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f141542c0 con 0x7f6f140aed20 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f11d9b640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f140aed20 0x7f6f14155360 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f6efc011a80 tx=0x7f6efc011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f0affd640 1 -- 172.21.15.81:0/4046851785 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc00fcf0 con 0x7f6f140aed20 2024-06-29T15:57:11.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f0affd640 1 -- 172.21.15.81:0/4046851785 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6efc016070 con 0x7f6f140aed20 2024-06-29T15:57:11.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f18973640 1 -- 172.21.15.81:0/4046851785 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f141545a0 con 0x7f6f140aed20 2024-06-29T15:57:11.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f0affd640 1 -- 172.21.15.81:0/4046851785 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc0126b0 con 0x7f6f140aed20 2024-06-29T15:57:11.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.188+0000 7f6f18973640 1 -- 172.21.15.81:0/4046851785 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f14154b60 con 0x7f6f140aed20 2024-06-29T15:57:11.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.189+0000 7f6f18973640 1 -- 172.21.15.81:0/4046851785 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f14154010 con 0x7f6f140aed20 2024-06-29T15:57:11.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.190+0000 7f6f0affd640 1 -- 172.21.15.81:0/4046851785 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 127) ==== 98247+0+0 (secure 0 0 0) 0x7f6efc018030 con 0x7f6f140aed20 2024-06-29T15:57:11.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.190+0000 7f6f0affd640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f6ef4076390 0x7f6ef40788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:11.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.190+0000 7f6f0affd640 1 -- 172.21.15.81:0/4046851785 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 6755+0+0 (secure 0 0 0) 0x7f6efc098380 con 0x7f6f140aed20 2024-06-29T15:57:11.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.191+0000 7f6f1259c640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f6ef4076390 0x7f6ef40788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:11.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.191+0000 7f6f1259c640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f6ef4076390 0x7f6ef40788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6f04033000 tx=0x7f6f0402d070 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:11.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.196+0000 7f6f0affd640 1 -- 172.21.15.81:0/4046851785 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6efc0611f0 con 0x7f6f140aed20 2024-06-29T15:57:11.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:11.378+0000 7f6f18973640 1 -- 172.21.15.81:0/4046851785 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6f140cea00 con 0x7f6f140aed20 2024-06-29T15:57:12.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.215+0000 7f6f0affd640 1 -- 172.21.15.81:0/4046851785 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v204) ==== 181+0+0 (secure 0 0 0) 0x7f6efc0654f0 con 0x7f6f140aed20 2024-06-29T15:57:12.216 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:57:12.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.222+0000 7f6f18973640 1 -- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f6ef4076390 msgr2=0x7f6ef40788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:12.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.222+0000 7f6f18973640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f6ef4076390 0x7f6ef40788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6f04033000 tx=0x7f6f0402d070 comp rx=0 tx=0).stop 2024-06-29T15:57:12.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.222+0000 7f6f18973640 1 -- 172.21.15.81:0/4046851785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f140aed20 msgr2=0x7f6f14155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:12.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.222+0000 7f6f18973640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f140aed20 0x7f6f14155360 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f6efc011a80 tx=0x7f6efc011f40 comp rx=0 tx=0).stop 2024-06-29T15:57:12.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.223+0000 7f6f18973640 1 -- 172.21.15.81:0/4046851785 shutdown_connections 2024-06-29T15:57:12.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.223+0000 7f6f18973640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f6ef4076390 0x7f6ef40788a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.223+0000 7f6f18973640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6f141f4730 0x7f6f141f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.223+0000 7f6f18973640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f140aed20 0x7f6f14155360 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.223+0000 7f6f18973640 1 --2- 172.21.15.81:0/4046851785 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6f00003c10 0x7f6f14136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.224+0000 7f6f18973640 1 -- 172.21.15.81:0/4046851785 >> 172.21.15.81:0/4046851785 conn(0x7f6f1406cd60 msgr2=0x7f6f140cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:12.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.224+0000 7f6f18973640 1 -- 172.21.15.81:0/4046851785 shutdown_connections 2024-06-29T15:57:12.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.224+0000 7f6f18973640 1 -- 172.21.15.81:0/4046851785 wait complete. 2024-06-29T15:57:12.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:57:12.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.355+0000 7f083588f640 1 -- 172.21.15.81:0/3392039763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08300b2ea0 msgr2=0x7f08300cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:12.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.355+0000 7f083588f640 1 --2- 172.21.15.81:0/3392039763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08300b2ea0 0x7f08300cf510 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f08300ce5f0 tx=0x7f082402a4d0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.356+0000 7f083588f640 1 -- 172.21.15.81:0/3392039763 shutdown_connections 2024-06-29T15:57:12.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.356+0000 7f083588f640 1 --2- 172.21.15.81:0/3392039763 >> v2:172.21.15.167:3300/0 conn(0x7f08300d03b0 0x7f08300d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.356+0000 7f083588f640 1 --2- 172.21.15.81:0/3392039763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08300b2ea0 0x7f08300cf510 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.356+0000 7f083588f640 1 --2- 172.21.15.81:0/3392039763 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08300d2440 0x7f08300d4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.356+0000 7f083588f640 1 -- 172.21.15.81:0/3392039763 >> 172.21.15.81:0/3392039763 conn(0x7f08300d3c50 msgr2=0x7f08300d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:12.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.356+0000 7f083588f640 1 -- 172.21.15.81:0/3392039763 shutdown_connections 2024-06-29T15:57:12.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.356+0000 7f083588f640 1 -- 172.21.15.81:0/3392039763 wait complete. 2024-06-29T15:57:12.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.357+0000 7f083588f640 1 Processor -- start 2024-06-29T15:57:12.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.358+0000 7f083588f640 1 -- start start 2024-06-29T15:57:12.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f083588f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08300b2ea0 0x7f083015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:12.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f083588f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08300d03b0 0x7f0830160ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:12.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f083588f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08300d2440 0x7f0830161020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:12.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f083588f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f08300cfbd0 con 0x7f08300d03b0 2024-06-29T15:57:12.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f083588f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f08300cfa50 con 0x7f08300d2440 2024-06-29T15:57:12.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f083588f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f083006cb10 con 0x7f08300b2ea0 2024-06-29T15:57:12.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08300d03b0 0x7f0830160ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:12.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08300d03b0 0x7f0830160ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42578/0 (socket says 172.21.15.81:42578) 2024-06-29T15:57:12.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082e7fc640 1 -- 172.21.15.81:0/2861591484 learned_addr learned my addr 172.21.15.81:0/2861591484 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:12.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082effd640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08300b2ea0 0x7f083015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:12.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082f7fe640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08300d2440 0x7f0830161020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:12.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082e7fc640 1 -- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08300b2ea0 msgr2=0x7f083015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:12.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082e7fc640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08300b2ea0 0x7f083015a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082e7fc640 1 -- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08300d2440 msgr2=0x7f0830161020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:12.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082e7fc640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08300d2440 0x7f0830161020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:12.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082e7fc640 1 -- 172.21.15.81:0/2861591484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083015b0e0 con 0x7f08300d03b0 2024-06-29T15:57:12.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.359+0000 7f082e7fc640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08300d03b0 0x7f0830160ae0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f0824034040 tx=0x7f08240039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:12.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.360+0000 7f0817fff640 1 -- 172.21.15.81:0/2861591484 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082403e030 con 0x7f08300d03b0 2024-06-29T15:57:12.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.360+0000 7f0817fff640 1 -- 172.21.15.81:0/2861591484 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08240047a0 con 0x7f08300d03b0 2024-06-29T15:57:12.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.360+0000 7f0817fff640 1 -- 172.21.15.81:0/2861591484 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0824035040 con 0x7f08300d03b0 2024-06-29T15:57:12.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.360+0000 7f083588f640 1 -- 172.21.15.81:0/2861591484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083015fd20 con 0x7f08300d03b0 2024-06-29T15:57:12.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.360+0000 7f083588f640 1 -- 172.21.15.81:0/2861591484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08301602e0 con 0x7f08300d03b0 2024-06-29T15:57:12.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.361+0000 7f083588f640 1 -- 172.21.15.81:0/2861591484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0830068ad0 con 0x7f08300d03b0 2024-06-29T15:57:12.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.361+0000 7f0817fff640 1 -- 172.21.15.81:0/2861591484 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f0824041030 con 0x7f08300d03b0 2024-06-29T15:57:12.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.362+0000 7f0817fff640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f081c0763f0 0x7f081c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:12.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.362+0000 7f0817fff640 1 -- 172.21.15.81:0/2861591484 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 6317+0+0 (secure 0 0 0) 0x7f08240b8340 con 0x7f08300d03b0 2024-06-29T15:57:12.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.362+0000 7f082effd640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f081c0763f0 0x7f081c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:12.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.363+0000 7f082effd640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f081c0763f0 0x7f081c078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0820005fd0 tx=0x7f08200095e0 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:12.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.367+0000 7f0817fff640 1 -- 172.21.15.81:0/2861591484 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0830068ad0 con 0x7f08300d03b0 2024-06-29T15:57:12.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:12.547+0000 7f083588f640 1 -- 172.21.15.81:0/2861591484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f08300d34a0 con 0x7f08300d03b0 2024-06-29T15:57:13.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.223+0000 7f0817fff640 1 -- 172.21.15.81:0/2861591484 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v205) ==== 169+0+0 (secure 0 0 0) 0x7f08300d34a0 con 0x7f08300d03b0 2024-06-29T15:57:13.224 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T15:57:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.226+0000 7f083588f640 1 -- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f081c0763f0 msgr2=0x7f081c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.226+0000 7f083588f640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f081c0763f0 0x7f081c078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0820005fd0 tx=0x7f08200095e0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.226+0000 7f083588f640 1 -- 172.21.15.81:0/2861591484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08300d03b0 msgr2=0x7f0830160ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.226+0000 7f083588f640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08300d03b0 0x7f0830160ae0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f0824034040 tx=0x7f08240039e0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.227+0000 7f083588f640 1 -- 172.21.15.81:0/2861591484 shutdown_connections 2024-06-29T15:57:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.227+0000 7f083588f640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f081c0763f0 0x7f081c078900 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0820005fd0 tx=0x7f08200095e0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.227+0000 7f083588f640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08300d2440 0x7f0830161020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.227+0000 7f083588f640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08300d03b0 0x7f0830160ae0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.227+0000 7f083588f640 1 --2- 172.21.15.81:0/2861591484 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08300b2ea0 0x7f083015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.227+0000 7f083588f640 1 -- 172.21.15.81:0/2861591484 >> 172.21.15.81:0/2861591484 conn(0x7f08300d3c50 msgr2=0x7f08300b7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.228+0000 7f083588f640 1 -- 172.21.15.81:0/2861591484 shutdown_connections 2024-06-29T15:57:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.228+0000 7f083588f640 1 -- 172.21.15.81:0/2861591484 wait complete. 2024-06-29T15:57:13.243 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:57:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.365+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1349729849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfa015cd00 msgr2=0x7fcfa015d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.365+0000 7fcfa5ee4640 1 --2- 172.21.15.81:0/1349729849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfa015cd00 0x7fcfa015d510 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fcf98005fd0 tx=0x7fcf9802a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.365+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1349729849 shutdown_connections 2024-06-29T15:57:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.365+0000 7fcfa5ee4640 1 --2- 172.21.15.81:0/1349729849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfa015cd00 0x7fcfa015d510 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.365+0000 7fcfa5ee4640 1 --2- 172.21.15.81:0/1349729849 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfa0145530 0x7fcfa0073280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.365+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1349729849 >> 172.21.15.81:0/1349729849 conn(0x7fcfa006ccc0 msgr2=0x7fcfa00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.365+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1349729849 shutdown_connections 2024-06-29T15:57:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.365+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1349729849 wait complete. 2024-06-29T15:57:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.366+0000 7fcfa5ee4640 1 Processor -- start 2024-06-29T15:57:13.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.367+0000 7fcfa5ee4640 1 -- start start 2024-06-29T15:57:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcfa5ee4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfa00737c0 0x7fcfa01e5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcfa5ee4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfa0145530 0x7fcfa01f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcfa5ee4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcfa015cd00 0x7fcfa01f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcfa5ee4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa015dbd0 con 0x7fcfa00737c0 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcfa5ee4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa015da50 con 0x7fcfa015cd00 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcfa5ee4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa015dd50 con 0x7fcfa0145530 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfa00737c0 0x7fcfa01e5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfa00737c0 0x7fcfa01e5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42604/0 (socket says 172.21.15.81:42604) 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9f7fe640 1 -- 172.21.15.81:0/1801609598 learned_addr learned my addr 172.21.15.81:0/1801609598 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9effd640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfa0145530 0x7fcfa01f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9ffff640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcfa015cd00 0x7fcfa01f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9f7fe640 1 -- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfa0145530 msgr2=0x7fcfa01f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9f7fe640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfa0145530 0x7fcfa01f2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9f7fe640 1 -- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcfa015cd00 msgr2=0x7fcfa01f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9f7fe640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcfa015cd00 0x7fcfa01f26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9f7fe640 1 -- 172.21.15.81:0/1801609598 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa01e6160 con 0x7fcfa00737c0 2024-06-29T15:57:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9effd640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfa0145530 0x7fcfa01f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf9f7fe640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfa00737c0 0x7fcfa01e5780 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000c970 tx=0x7fcf9000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.368+0000 7fcf87fff640 1 -- 172.21.15.81:0/1801609598 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90007bb0 con 0x7fcfa00737c0 2024-06-29T15:57:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.369+0000 7fcf87fff640 1 -- 172.21.15.81:0/1801609598 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf90007e40 con 0x7fcfa00737c0 2024-06-29T15:57:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.369+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1801609598 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa01e6440 con 0x7fcfa00737c0 2024-06-29T15:57:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.369+0000 7fcf87fff640 1 -- 172.21.15.81:0/1801609598 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90005280 con 0x7fcfa00737c0 2024-06-29T15:57:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.369+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1801609598 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa01e6a00 con 0x7fcfa00737c0 2024-06-29T15:57:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.369+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1801609598 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa00ceef0 con 0x7fcfa00737c0 2024-06-29T15:57:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.370+0000 7fcf87fff640 1 -- 172.21.15.81:0/1801609598 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fcf90005420 con 0x7fcfa00737c0 2024-06-29T15:57:13.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.371+0000 7fcf87fff640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcf8c0763f0 0x7fcf8c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.371+0000 7fcf87fff640 1 -- 172.21.15.81:0/1801609598 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5845+0+0 (secure 0 0 0) 0x7fcf90097f40 con 0x7fcfa00737c0 2024-06-29T15:57:13.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.371+0000 7fcf9effd640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcf8c0763f0 0x7fcf8c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.372+0000 7fcf9effd640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcf8c0763f0 0x7fcf8c078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcfa00b1c80 tx=0x7fcf94009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:13.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.375+0000 7fcf87fff640 1 -- 172.21.15.81:0/1801609598 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcfa00ceef0 con 0x7fcfa00737c0 2024-06-29T15:57:13.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.557+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1801609598 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fcfa015e290 con 0x7fcfa00737c0 2024-06-29T15:57:13.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.561+0000 7fcf87fff640 1 -- 172.21.15.81:0/1801609598 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v71) ==== 71+0+0 (secure 0 0 0) 0x7fcf90014030 con 0x7fcfa00737c0 2024-06-29T15:57:13.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcf8c0763f0 msgr2=0x7fcf8c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcf8c0763f0 0x7fcf8c078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcfa00b1c80 tx=0x7fcf94009040 comp rx=0 tx=0).stop 2024-06-29T15:57:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1801609598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfa00737c0 msgr2=0x7fcfa01e5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfa00737c0 0x7fcfa01e5780 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000c970 tx=0x7fcf9000ce30 comp rx=0 tx=0).stop 2024-06-29T15:57:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1801609598 shutdown_connections 2024-06-29T15:57:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcf8c0763f0 0x7fcf8c078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcfa015cd00 0x7fcfa01f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcfa0145530 0x7fcfa01f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 --2- 172.21.15.81:0/1801609598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfa00737c0 0x7fcfa01e5780 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1801609598 >> 172.21.15.81:0/1801609598 conn(0x7fcfa006ccc0 msgr2=0x7fcfa015c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1801609598 shutdown_connections 2024-06-29T15:57:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.563+0000 7fcfa5ee4640 1 -- 172.21.15.81:0/1801609598 wait complete. 2024-06-29T15:57:13.574 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2024-06-29T15:57:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.688+0000 7f600f66d640 1 -- 172.21.15.81:0/1457802540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6008073030 msgr2=0x7f6008073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.688+0000 7f600f66d640 1 --2- 172.21.15.81:0/1457802540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6008073030 0x7f6008073440 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f6008071d20 tx=0x7f5ff802a560 comp rx=0 tx=0).stop 2024-06-29T15:57:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.688+0000 7f600f66d640 1 -- 172.21.15.81:0/1457802540 shutdown_connections 2024-06-29T15:57:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.688+0000 7f600f66d640 1 --2- 172.21.15.81:0/1457802540 >> v2:172.21.15.167:3300/0 conn(0x7f600815e960 0x7f600815ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.688+0000 7f600f66d640 1 --2- 172.21.15.81:0/1457802540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6008073030 0x7f6008073440 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.688+0000 7f600f66d640 1 -- 172.21.15.81:0/1457802540 >> 172.21.15.81:0/1457802540 conn(0x7f600806ccc0 msgr2=0x7f60080b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.688+0000 7f600f66d640 1 -- 172.21.15.81:0/1457802540 shutdown_connections 2024-06-29T15:57:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.688+0000 7f600f66d640 1 -- 172.21.15.81:0/1457802540 wait complete. 2024-06-29T15:57:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.690+0000 7f600f66d640 1 Processor -- start 2024-06-29T15:57:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.690+0000 7f600f66d640 1 -- start start 2024-06-29T15:57:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.691+0000 7f600f66d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6008073030 0x7f60081402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.691+0000 7f600f66d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60081412f0 0x7f60081f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.691+0000 7f600f66d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f600815e960 0x7f60080b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.691+0000 7f600f66d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f60080b3210 con 0x7f60081412f0 2024-06-29T15:57:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.691+0000 7f600f66d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6008141740 con 0x7f6008073030 2024-06-29T15:57:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.691+0000 7f600f66d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f60080b3390 con 0x7f600815e960 2024-06-29T15:57:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.691+0000 7f600cc08640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60081412f0 0x7f60081f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.691+0000 7f600cc08640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60081412f0 0x7f60081f54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42612/0 (socket says 172.21.15.81:42612) 2024-06-29T15:57:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.691+0000 7f600cc08640 1 -- 172.21.15.81:0/1287111468 learned_addr learned my addr 172.21.15.81:0/1287111468 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.691+0000 7f600d409640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6008073030 0x7f60081402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f600dc0a640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f600815e960 0x7f60080b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f600cc08640 1 -- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f600815e960 msgr2=0x7f60080b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f600cc08640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f600815e960 0x7f60080b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f600cc08640 1 -- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6008073030 msgr2=0x7f60081402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f600cc08640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6008073030 0x7f60081402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f600cc08640 1 -- 172.21.15.81:0/1287111468 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60080afdf0 con 0x7f60081412f0 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f600cc08640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60081412f0 0x7f60081f54f0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80099c0 tx=0x7f5ff80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f5ffdffb640 1 -- 172.21.15.81:0/1287111468 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff803e030 con 0x7f60081412f0 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f5ffdffb640 1 -- 172.21.15.81:0/1287111468 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ff80047a0 con 0x7f60081412f0 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f600f66d640 1 -- 172.21.15.81:0/1287111468 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60080b00d0 con 0x7f60081412f0 2024-06-29T15:57:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f5ffdffb640 1 -- 172.21.15.81:0/1287111468 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8035040 con 0x7f60081412f0 2024-06-29T15:57:13.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.692+0000 7f600f66d640 1 -- 172.21.15.81:0/1287111468 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60080b0690 con 0x7f60081412f0 2024-06-29T15:57:13.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.693+0000 7f600f66d640 1 -- 172.21.15.81:0/1287111468 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60080afb60 con 0x7f60081412f0 2024-06-29T15:57:13.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.694+0000 7f5ffdffb640 1 -- 172.21.15.81:0/1287111468 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f5ff8041030 con 0x7f60081412f0 2024-06-29T15:57:13.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.694+0000 7f5ffdffb640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5fec0763f0 0x7f5fec078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:13.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.695+0000 7f5ffdffb640 1 -- 172.21.15.81:0/1287111468 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5845+0+0 (secure 0 0 0) 0x7f5ff80b82b0 con 0x7f60081412f0 2024-06-29T15:57:13.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.695+0000 7f600d409640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5fec0763f0 0x7f5fec078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:13.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.695+0000 7f600d409640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5fec0763f0 0x7f5fec078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6004009f60 tx=0x7f60040094f0 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:13.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.699+0000 7f5ffdffb640 1 -- 172.21.15.81:0/1287111468 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6008073980 con 0x7f60081412f0 2024-06-29T15:57:13.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:13.950+0000 7f600f66d640 1 -- 172.21.15.81:0/1287111468 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) -- 0x7f60080b3970 con 0x7f60081412f0 2024-06-29T15:57:14.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.225+0000 7f5ffdffb640 1 -- 172.21.15.81:0/1287111468 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3065) ==== 163+0+0 (secure 0 0 0) 0x7f5ff80818e0 con 0x7f60081412f0 2024-06-29T15:57:14.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.227+0000 7f600f66d640 1 -- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5fec0763f0 msgr2=0x7f5fec078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.227+0000 7f600f66d640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5fec0763f0 0x7f5fec078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6004009f60 tx=0x7f60040094f0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.228+0000 7f600f66d640 1 -- 172.21.15.81:0/1287111468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60081412f0 msgr2=0x7f60081f54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.228+0000 7f600f66d640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60081412f0 0x7f60081f54f0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80099c0 tx=0x7f5ff80039e0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.228+0000 7f600f66d640 1 -- 172.21.15.81:0/1287111468 shutdown_connections 2024-06-29T15:57:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.228+0000 7f600f66d640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5fec0763f0 0x7f5fec078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.228+0000 7f600f66d640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f600815e960 0x7f60080b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.228+0000 7f600f66d640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60081412f0 0x7f60081f54f0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.228+0000 7f600f66d640 1 --2- 172.21.15.81:0/1287111468 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6008073030 0x7f60081402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.228+0000 7f600f66d640 1 -- 172.21.15.81:0/1287111468 >> 172.21.15.81:0/1287111468 conn(0x7f600806ccc0 msgr2=0x7f600806e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.228+0000 7f600f66d640 1 -- 172.21.15.81:0/1287111468 shutdown_connections 2024-06-29T15:57:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.228+0000 7f600f66d640 1 -- 172.21.15.81:0/1287111468 wait complete. 2024-06-29T15:57:14.239 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2024-06-29T15:57:14.240 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2024-06-29T15:57:14.240 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-06-29T15:57:14.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:14.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.354+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1997346462 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02f4074240 msgr2=0x7f02f406fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.355+0000 7f02fb4fd640 1 --2- 172.21.15.81:0/1997346462 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02f4074240 0x7f02f406fc00 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f02e800a940 tx=0x7f02e802b7b0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.355+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1997346462 shutdown_connections 2024-06-29T15:57:14.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.355+0000 7f02fb4fd640 1 --2- 172.21.15.81:0/1997346462 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02f415b3c0 0x7f02f406eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.355+0000 7f02fb4fd640 1 --2- 172.21.15.81:0/1997346462 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02f4074240 0x7f02f406fc00 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.355+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1997346462 >> 172.21.15.81:0/1997346462 conn(0x7f02f40b1ce0 msgr2=0x7f02f40b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.355+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1997346462 shutdown_connections 2024-06-29T15:57:14.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.355+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1997346462 wait complete. 2024-06-29T15:57:14.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.356+0000 7f02fb4fd640 1 Processor -- start 2024-06-29T15:57:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.356+0000 7f02fb4fd640 1 -- start start 2024-06-29T15:57:14.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02fb4fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02f40708e0 0x7f02f40b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02fb4fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02f415b3c0 0x7f02f41e2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02fb4fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02f40aeec0 0x7f02f41e3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02fb4fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f02f4073420 con 0x7f02f40aeec0 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02fb4fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f02f40732a0 con 0x7f02f415b3c0 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02fb4fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f02f40735a0 con 0x7f02f40708e0 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f9a9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02f40aeec0 0x7f02f41e3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f9a9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02f40aeec0 0x7f02f41e3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42620/0 (socket says 172.21.15.81:42620) 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f9a9a640 1 -- 172.21.15.81:0/1332599207 learned_addr learned my addr 172.21.15.81:0/1332599207 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f8a98640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02f415b3c0 0x7f02f41e2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f9299640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02f40708e0 0x7f02f40b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f9a9a640 1 -- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02f40708e0 msgr2=0x7f02f40b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f9a9a640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02f40708e0 0x7f02f40b13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f9a9a640 1 -- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02f415b3c0 msgr2=0x7f02f41e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f9a9a640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02f415b3c0 0x7f02f41e2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f9a9a640 1 -- 172.21.15.81:0/1332599207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f40ade20 con 0x7f02f40aeec0 2024-06-29T15:57:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f9a9a640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02f40aeec0 0x7f02f41e3110 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e940 tx=0x7f02ec00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:14.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f1ffb640 1 -- 172.21.15.81:0/1332599207 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec00cd80 con 0x7f02f40aeec0 2024-06-29T15:57:14.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02f1ffb640 1 -- 172.21.15.81:0/1332599207 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02ec004020 con 0x7f02f40aeec0 2024-06-29T15:57:14.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.358+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1332599207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f40ae100 con 0x7f02f40aeec0 2024-06-29T15:57:14.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.359+0000 7f02f1ffb640 1 -- 172.21.15.81:0/1332599207 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec004470 con 0x7f02f40aeec0 2024-06-29T15:57:14.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.359+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1332599207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f40ae6c0 con 0x7f02f40aeec0 2024-06-29T15:57:14.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.360+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1332599207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f40cfab0 con 0x7f02f40aeec0 2024-06-29T15:57:14.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.360+0000 7f02f1ffb640 1 -- 172.21.15.81:0/1332599207 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f02ec0035f0 con 0x7f02f40aeec0 2024-06-29T15:57:14.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.361+0000 7f02f1ffb640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f02e00763f0 0x7f02e0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.361+0000 7f02f1ffb640 1 -- 172.21.15.81:0/1332599207 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5845+0+0 (secure 0 0 0) 0x7f02ec0982f0 con 0x7f02f40aeec0 2024-06-29T15:57:14.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.361+0000 7f02f9299640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f02e00763f0 0x7f02e0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.362+0000 7f02f9299640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f02e00763f0 0x7f02e0078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f02e40017a0 tx=0x7f02e4001730 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:14.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.365+0000 7f02f1ffb640 1 -- 172.21.15.81:0/1332599207 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f02f40cfab0 con 0x7f02f40aeec0 2024-06-29T15:57:14.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.548+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1332599207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02f40704d0 con 0x7f02f40aeec0 2024-06-29T15:57:14.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.549+0000 7f02f1ffb640 1 -- 172.21.15.81:0/1332599207 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) ==== 74+0+14361 (secure 0 0 0) 0x7f02f40704d0 con 0x7f02f40aeec0 2024-06-29T15:57:14.549 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:14.549 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":205,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:13.216559+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6835/1700137710":"2024-06-30T15:57:10.042097+0000","172.21.15.167:6837/1700137710":"2024-06-30T15:57:10.042097+0000","172.21.15.167:0/2766539241":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/649722843":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2657542019":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2947012481":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/3123996030":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/339582014":"2024-06-30T15:57:07.850629+0000","172.21.15.167:6832/23077685":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2019373615":"2024-06-30T15:57:07.850629+0000","172.21.15.167:6833/803000291":"2024-06-30T15:55:49.959292+0000","172.21.15.167:6833/23077685":"2024-06-30T15:57:07.850629+0000","172.21.15.167:6832/803000291":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/1482340295":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/2216474049":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/3153968292":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/1095374241":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/3639047578":"2024-06-30T15:55:49.959292+0000","172.21.15.167:0/481601843":"2024-06-30T15:57:07.850629+0000","172.21.15.167:0/2467432753":"2024-06-30T15:55:49.959292+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:14.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f02e00763f0 msgr2=0x7f02e0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f02e00763f0 0x7f02e0078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f02e40017a0 tx=0x7f02e4001730 comp rx=0 tx=0).stop 2024-06-29T15:57:14.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1332599207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02f40aeec0 msgr2=0x7f02f41e3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02f40aeec0 0x7f02f41e3110 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e940 tx=0x7f02ec00ee00 comp rx=0 tx=0).stop 2024-06-29T15:57:14.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1332599207 shutdown_connections 2024-06-29T15:57:14.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f02e00763f0 0x7f02e0078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02f40aeec0 0x7f02f41e3110 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02f415b3c0 0x7f02f41e2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 --2- 172.21.15.81:0/1332599207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02f40708e0 0x7f02f40b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1332599207 >> 172.21.15.81:0/1332599207 conn(0x7f02f40b1ce0 msgr2=0x7f02f40cdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1332599207 shutdown_connections 2024-06-29T15:57:14.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.551+0000 7f02fb4fd640 1 -- 172.21.15.81:0/1332599207 wait complete. 2024-06-29T15:57:14.563 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:14.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:14.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.679+0000 7f8c51a10640 1 -- 172.21.15.81:0/480277553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c0b8fa0 msgr2=0x7f8c4c0d0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.679+0000 7f8c51a10640 1 --2- 172.21.15.81:0/480277553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c0b8fa0 0x7f8c4c0d0460 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0ce560 tx=0x7f8c3c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.679+0000 7f8c51a10640 1 -- 172.21.15.81:0/480277553 shutdown_connections 2024-06-29T15:57:14.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.679+0000 7f8c51a10640 1 --2- 172.21.15.81:0/480277553 >> v2:172.21.15.167:3300/0 conn(0x7f8c4c0d09a0 0x7f8c4c0b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.679+0000 7f8c51a10640 1 --2- 172.21.15.81:0/480277553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c0b8fa0 0x7f8c4c0d0460 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.679+0000 7f8c51a10640 1 -- 172.21.15.81:0/480277553 >> 172.21.15.81:0/480277553 conn(0x7f8c4c0d3c50 msgr2=0x7f8c4c0d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.679+0000 7f8c51a10640 1 -- 172.21.15.81:0/480277553 shutdown_connections 2024-06-29T15:57:14.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.679+0000 7f8c51a10640 1 -- 172.21.15.81:0/480277553 wait complete. 2024-06-29T15:57:14.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.681+0000 7f8c51a10640 1 Processor -- start 2024-06-29T15:57:14.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.681+0000 7f8c51a10640 1 -- start start 2024-06-29T15:57:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.682+0000 7f8c51a10640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c0b8fa0 0x7f8c4c15cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.682+0000 7f8c51a10640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c4c0d09a0 0x7f8c4c160240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.682+0000 7f8c51a10640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c4c0d0db0 0x7f8c4c1614c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.682+0000 7f8c51a10640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c0b4160 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.682+0000 7f8c51a10640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c0b3fe0 con 0x7f8c4c0d09a0 2024-06-29T15:57:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.682+0000 7f8c51a10640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8c4c15efe0 con 0x7f8c4c0d0db0 2024-06-29T15:57:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.682+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c0b8fa0 0x7f8c4c15cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.682+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c0b8fa0 0x7f8c4c15cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42644/0 (socket says 172.21.15.81:42644) 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.682+0000 7f8c4affd640 1 -- 172.21.15.81:0/3434111197 learned_addr learned my addr 172.21.15.81:0/3434111197 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c4affd640 1 -- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c4c0d0db0 msgr2=0x7f8c4c1614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c4a7fc640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c4c0d09a0 0x7f8c4c160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c4b7fe640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c4c0d0db0 0x7f8c4c1614c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c4affd640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c4c0d0db0 0x7f8c4c1614c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c4affd640 1 -- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c4c0d09a0 msgr2=0x7f8c4c160240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c4affd640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c4c0d09a0 0x7f8c4c160240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c4affd640 1 -- 172.21.15.81:0/3434111197 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c15d780 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c4a7fc640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c4c0d09a0 0x7f8c4c160240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c4b7fe640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c4c0d0db0 0x7f8c4c1614c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c4affd640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c0b8fa0 0x7f8c4c15cec0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c005fd0 tx=0x7f8c3c003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c33fff640 1 -- 172.21.15.81:0/3434111197 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c031b90 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c33fff640 1 -- 172.21.15.81:0/3434111197 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c3c031e20 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c33fff640 1 -- 172.21.15.81:0/3434111197 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c033310 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c51a10640 1 -- 172.21.15.81:0/3434111197 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c15ead0 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.683+0000 7f8c51a10640 1 -- 172.21.15.81:0/3434111197 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c4c167bb0 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.684+0000 7f8c51a10640 1 -- 172.21.15.81:0/3434111197 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c4c0d0460 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.685+0000 7f8c33fff640 1 -- 172.21.15.81:0/3434111197 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f8c3c0334b0 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.685+0000 7f8c33fff640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8c380763f0 0x7f8c38078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:14.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.685+0000 7f8c33fff640 1 -- 172.21.15.81:0/3434111197 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5845+0+0 (secure 0 0 0) 0x7f8c3c0b9130 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.686+0000 7f8c4a7fc640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8c380763f0 0x7f8c38078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:14.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.686+0000 7f8c4a7fc640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8c380763f0 0x7f8c38078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c1f8270 tx=0x7f8c40008040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:14.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.690+0000 7f8c33fff640 1 -- 172.21.15.81:0/3434111197 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8c3c03f030 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.870+0000 7f8c51a10640 1 -- 172.21.15.81:0/3434111197 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8c4c0d1290 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.870+0000 7f8c33fff640 1 -- 172.21.15.81:0/3434111197 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v205) ==== 93+0+10 (secure 0 0 0) 0x7f8c3c082750 con 0x7f8c4c0b8fa0 2024-06-29T15:57:14.871 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:57:14.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.872+0000 7f8c51a10640 1 -- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8c380763f0 msgr2=0x7f8c38078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.872+0000 7f8c51a10640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8c380763f0 0x7f8c38078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c1f8270 tx=0x7f8c40008040 comp rx=0 tx=0).stop 2024-06-29T15:57:14.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.872+0000 7f8c51a10640 1 -- 172.21.15.81:0/3434111197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c0b8fa0 msgr2=0x7f8c4c15cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.872+0000 7f8c51a10640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c0b8fa0 0x7f8c4c15cec0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c005fd0 tx=0x7f8c3c003a10 comp rx=0 tx=0).stop 2024-06-29T15:57:14.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.872+0000 7f8c51a10640 1 -- 172.21.15.81:0/3434111197 shutdown_connections 2024-06-29T15:57:14.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.872+0000 7f8c51a10640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8c380763f0 0x7f8c38078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.872+0000 7f8c51a10640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8c4c0d0db0 0x7f8c4c1614c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.872+0000 7f8c51a10640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8c4c0d09a0 0x7f8c4c160240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.872+0000 7f8c51a10640 1 --2- 172.21.15.81:0/3434111197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c0b8fa0 0x7f8c4c15cec0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.872+0000 7f8c51a10640 1 -- 172.21.15.81:0/3434111197 >> 172.21.15.81:0/3434111197 conn(0x7f8c4c0d3c50 msgr2=0x7f8c4c06d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.873+0000 7f8c51a10640 1 -- 172.21.15.81:0/3434111197 shutdown_connections 2024-06-29T15:57:14.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.873+0000 7f8c51a10640 1 -- 172.21.15.81:0/3434111197 wait complete. 2024-06-29T15:57:14.884 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-06-29T15:57:14.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.997+0000 7f7774154640 1 -- 172.21.15.81:0/4026299553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f776c145530 msgr2=0x7f776c073280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:14.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.997+0000 7f7774154640 1 --2- 172.21.15.81:0/4026299553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f776c145530 0x7f776c073280 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f7764005fd0 tx=0x7f776402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.997+0000 7f7774154640 1 -- 172.21.15.81:0/4026299553 shutdown_connections 2024-06-29T15:57:14.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.997+0000 7f7774154640 1 --2- 172.21.15.81:0/4026299553 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f776c0737c0 0x7f776c15c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.997+0000 7f7774154640 1 --2- 172.21.15.81:0/4026299553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f776c145530 0x7f776c073280 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:14.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.997+0000 7f7774154640 1 -- 172.21.15.81:0/4026299553 >> 172.21.15.81:0/4026299553 conn(0x7f776c06ccc0 msgr2=0x7f776c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:14.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.997+0000 7f7774154640 1 -- 172.21.15.81:0/4026299553 shutdown_connections 2024-06-29T15:57:14.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.997+0000 7f7774154640 1 -- 172.21.15.81:0/4026299553 wait complete. 2024-06-29T15:57:14.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.999+0000 7f7774154640 1 Processor -- start 2024-06-29T15:57:14.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:14.999+0000 7f7774154640 1 -- start start 2024-06-29T15:57:15.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f7774154640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f776c0737c0 0x7f776c156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:15.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f7774154640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f776c145530 0x7f776c139270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f7774154640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f776c15cb30 0x7f776c1f4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f7774154640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f776c15d0c0 con 0x7f776c15cb30 2024-06-29T15:57:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f7774154640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f776c15cf40 con 0x7f776c0737c0 2024-06-29T15:57:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f7774154640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f776c160050 con 0x7f776c145530 2024-06-29T15:57:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f77726f1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f776c15cb30 0x7f776c1f4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f77726f1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f776c15cb30 0x7f776c1f4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42654/0 (socket says 172.21.15.81:42654) 2024-06-29T15:57:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f77726f1640 1 -- 172.21.15.81:0/4146832589 learned_addr learned my addr 172.21.15.81:0/4146832589 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f77716ef640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f776c145530 0x7f776c139270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.000+0000 7f7771ef0640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f776c0737c0 0x7f776c156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:15.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f77726f1640 1 -- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f776c145530 msgr2=0x7f776c139270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:15.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f77726f1640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f776c145530 0x7f776c139270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:15.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f77726f1640 1 -- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f776c0737c0 msgr2=0x7f776c156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:15.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f77726f1640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f776c0737c0 0x7f776c156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:15.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f77726f1640 1 -- 172.21.15.81:0/4146832589 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776c0b4420 con 0x7f776c15cb30 2024-06-29T15:57:15.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f77716ef640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f776c145530 0x7f776c139270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:15.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f77726f1640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f776c15cb30 0x7f776c1f4d40 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f775c00c700 tx=0x7f775c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:15.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f776a7fc640 1 -- 172.21.15.81:0/4146832589 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c003b70 con 0x7f776c15cb30 2024-06-29T15:57:15.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f776a7fc640 1 -- 172.21.15.81:0/4146832589 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f775c003d10 con 0x7f776c15cb30 2024-06-29T15:57:15.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f7774154640 1 -- 172.21.15.81:0/4146832589 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f776c0b4700 con 0x7f776c15cb30 2024-06-29T15:57:15.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f776a7fc640 1 -- 172.21.15.81:0/4146832589 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c00aa50 con 0x7f776c15cb30 2024-06-29T15:57:15.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.001+0000 7f7774154640 1 -- 172.21.15.81:0/4146832589 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f776c0ae530 con 0x7f776c15cb30 2024-06-29T15:57:15.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.002+0000 7f7774154640 1 -- 172.21.15.81:0/4146832589 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f776c160500 con 0x7f776c15cb30 2024-06-29T15:57:15.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.003+0000 7f776a7fc640 1 -- 172.21.15.81:0/4146832589 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f775c00abf0 con 0x7f776c15cb30 2024-06-29T15:57:15.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.003+0000 7f776a7fc640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77580763f0 0x7f7758078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:15.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.004+0000 7f776a7fc640 1 -- 172.21.15.81:0/4146832589 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5845+0+0 (secure 0 0 0) 0x7f775c097740 con 0x7f776c15cb30 2024-06-29T15:57:15.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.004+0000 7f7771ef0640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77580763f0 0x7f7758078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:15.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.004+0000 7f7771ef0640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77580763f0 0x7f7758078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7764033000 tx=0x7f7764004730 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:15.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.008+0000 7f776a7fc640 1 -- 172.21.15.81:0/4146832589 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f776c073bd0 con 0x7f776c15cb30 2024-06-29T15:57:15.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:15.256+0000 7f7774154640 1 -- 172.21.15.81:0/4146832589 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) -- 0x7f776c156f80 con 0x7f776c15cb30 2024-06-29T15:57:16.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.232+0000 7f776a7fc640 1 -- 172.21.15.81:0/4146832589 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3067) ==== 189+0+0 (secure 0 0 0) 0x7f775c060f00 con 0x7f776c15cb30 2024-06-29T15:57:16.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 -- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77580763f0 msgr2=0x7f7758078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77580763f0 0x7f7758078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7764033000 tx=0x7f7764004730 comp rx=0 tx=0).stop 2024-06-29T15:57:16.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 -- 172.21.15.81:0/4146832589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f776c15cb30 msgr2=0x7f776c1f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f776c15cb30 0x7f776c1f4d40 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f775c00c700 tx=0x7f775c00cbc0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 -- 172.21.15.81:0/4146832589 shutdown_connections 2024-06-29T15:57:16.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77580763f0 0x7f7758078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f776c15cb30 0x7f776c1f4d40 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f776c145530 0x7f776c139270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 --2- 172.21.15.81:0/4146832589 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f776c0737c0 0x7f776c156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 -- 172.21.15.81:0/4146832589 >> 172.21.15.81:0/4146832589 conn(0x7f776c06ccc0 msgr2=0x7f776c141170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:16.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 -- 172.21.15.81:0/4146832589 shutdown_connections 2024-06-29T15:57:16.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.235+0000 7f7774154640 1 -- 172.21.15.81:0/4146832589 wait complete. 2024-06-29T15:57:16.249 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:57:16.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.368+0000 7f759dd9a640 1 -- 172.21.15.81:0/528863600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75980b57b0 msgr2=0x7f759815aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.368+0000 7f759dd9a640 1 --2- 172.21.15.81:0/528863600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75980b57b0 0x7f759815aed0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f7588005fd0 tx=0x7f758802a670 comp rx=0 tx=0).stop 2024-06-29T15:57:16.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.369+0000 7f759dd9a640 1 -- 172.21.15.81:0/528863600 shutdown_connections 2024-06-29T15:57:16.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.369+0000 7f759dd9a640 1 --2- 172.21.15.81:0/528863600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75980b57b0 0x7f759815aed0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.369+0000 7f759dd9a640 1 -- 172.21.15.81:0/528863600 >> 172.21.15.81:0/528863600 conn(0x7f75980d0930 msgr2=0x7f75980d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:16.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.369+0000 7f759dd9a640 1 -- 172.21.15.81:0/528863600 shutdown_connections 2024-06-29T15:57:16.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.369+0000 7f759dd9a640 1 -- 172.21.15.81:0/528863600 wait complete. 2024-06-29T15:57:16.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.370+0000 7f759dd9a640 1 Processor -- start 2024-06-29T15:57:16.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.371+0000 7f759dd9a640 1 -- start start 2024-06-29T15:57:16.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f759dd9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759806ef90 0x7f759814dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f759dd9a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75980b3960 0x7f75981e5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f759dd9a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75980b57b0 0x7f75981e6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f759dd9a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f75980b4600 con 0x7f759806ef90 2024-06-29T15:57:16.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f759dd9a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f75980b4480 con 0x7f75980b3960 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f759dd9a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f75980b4780 con 0x7f75980b57b0 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f75977fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759806ef90 0x7f759814dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f75977fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759806ef90 0x7f759814dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42672/0 (socket says 172.21.15.81:42672) 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f75977fe640 1 -- 172.21.15.81:0/3169725201 learned_addr learned my addr 172.21.15.81:0/3169725201 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f7596ffd640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75980b3960 0x7f75981e5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f7597fff640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75980b57b0 0x7f75981e6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f75977fe640 1 -- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75980b57b0 msgr2=0x7f75981e6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f75977fe640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75980b57b0 0x7f75981e6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f75977fe640 1 -- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75980b3960 msgr2=0x7f75981e5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f75977fe640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75980b3960 0x7f75981e5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f75977fe640 1 -- 172.21.15.81:0/3169725201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759814e260 con 0x7f759806ef90 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f7596ffd640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75980b3960 0x7f75981e5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f75977fe640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759806ef90 0x7f759814dd20 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f7588034040 tx=0x7f7588003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:16.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.372+0000 7f757ffff640 1 -- 172.21.15.81:0/3169725201 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758803e030 con 0x7f759806ef90 2024-06-29T15:57:16.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.373+0000 7f757ffff640 1 -- 172.21.15.81:0/3169725201 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7588003e00 con 0x7f759806ef90 2024-06-29T15:57:16.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.373+0000 7f759dd9a640 1 -- 172.21.15.81:0/3169725201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75981e4480 con 0x7f759806ef90 2024-06-29T15:57:16.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.373+0000 7f757ffff640 1 -- 172.21.15.81:0/3169725201 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7588035040 con 0x7f759806ef90 2024-06-29T15:57:16.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.373+0000 7f759dd9a640 1 -- 172.21.15.81:0/3169725201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7598152690 con 0x7f759806ef90 2024-06-29T15:57:16.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.374+0000 7f759dd9a640 1 -- 172.21.15.81:0/3169725201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75981e49a0 con 0x7f759806ef90 2024-06-29T15:57:16.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.374+0000 7f757ffff640 1 -- 172.21.15.81:0/3169725201 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f7588041030 con 0x7f759806ef90 2024-06-29T15:57:16.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.375+0000 7f757ffff640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f75840763f0 0x7f7584078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.375+0000 7f757ffff640 1 -- 172.21.15.81:0/3169725201 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5845+0+0 (secure 0 0 0) 0x7f75880b8340 con 0x7f759806ef90 2024-06-29T15:57:16.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.375+0000 7f7596ffd640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f75840763f0 0x7f7584078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.375+0000 7f7596ffd640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f75840763f0 0x7f7584078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f758c007fd0 tx=0x7f758c009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:16.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.379+0000 7f757ffff640 1 -- 172.21.15.81:0/3169725201 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7588081960 con 0x7f759806ef90 2024-06-29T15:57:16.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.559+0000 7f759dd9a640 1 -- 172.21.15.81:0/3169725201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f759806e2a0 con 0x7f759806ef90 2024-06-29T15:57:16.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.559+0000 7f757ffff640 1 -- 172.21.15.81:0/3169725201 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v71) ==== 59+0+1998 (secure 0 0 0) 0x7f759806e2a0 con 0x7f759806ef90 2024-06-29T15:57:16.560 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:57:16.560 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:57:16.560 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:57:16.560 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:57:16.560 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T15:57:16.560 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T15:57:16.560 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T15:57:16.561 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:57:16.562 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:57:16.562 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:57:16.562 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T15:57:16.562 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T15:57:16.562 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T15:57:16.562 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:57:16.562 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:57:16.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.561+0000 7f759dd9a640 1 -- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f75840763f0 msgr2=0x7f7584078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.561+0000 7f759dd9a640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f75840763f0 0x7f7584078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f758c007fd0 tx=0x7f758c009040 comp rx=0 tx=0).stop 2024-06-29T15:57:16.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.561+0000 7f759dd9a640 1 -- 172.21.15.81:0/3169725201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759806ef90 msgr2=0x7f759814dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.561+0000 7f759dd9a640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759806ef90 0x7f759814dd20 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f7588034040 tx=0x7f7588003910 comp rx=0 tx=0).stop 2024-06-29T15:57:16.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.562+0000 7f759dd9a640 1 -- 172.21.15.81:0/3169725201 shutdown_connections 2024-06-29T15:57:16.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.562+0000 7f759dd9a640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f75840763f0 0x7f7584078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.562+0000 7f759dd9a640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75980b57b0 0x7f75981e6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.562+0000 7f759dd9a640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75980b3960 0x7f75981e5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.562+0000 7f759dd9a640 1 --2- 172.21.15.81:0/3169725201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759806ef90 0x7f759814dd20 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.562+0000 7f759dd9a640 1 -- 172.21.15.81:0/3169725201 >> 172.21.15.81:0/3169725201 conn(0x7f75980d0930 msgr2=0x7f7598155670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:16.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.562+0000 7f759dd9a640 1 -- 172.21.15.81:0/3169725201 shutdown_connections 2024-06-29T15:57:16.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.562+0000 7f759dd9a640 1 -- 172.21.15.81:0/3169725201 wait complete. 2024-06-29T15:57:16.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:57:16.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.688+0000 7fe98f01f640 1 -- 172.21.15.81:0/355070476 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9880cf600 msgr2=0x7fe9880cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.688+0000 7fe98f01f640 1 --2- 172.21.15.81:0/355070476 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9880cf600 0x7fe9880cfa10 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe980005fd0 tx=0x7fe98002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.689+0000 7fe98f01f640 1 -- 172.21.15.81:0/355070476 shutdown_connections 2024-06-29T15:57:16.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.689+0000 7fe98f01f640 1 --2- 172.21.15.81:0/355070476 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9880cf600 0x7fe9880cfa10 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.689+0000 7fe98f01f640 1 --2- 172.21.15.81:0/355070476 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9880ce3a0 0x7fe9880ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.689+0000 7fe98f01f640 1 -- 172.21.15.81:0/355070476 >> 172.21.15.81:0/355070476 conn(0x7fe9880b35a0 msgr2=0x7fe9880b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:16.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.689+0000 7fe98f01f640 1 -- 172.21.15.81:0/355070476 shutdown_connections 2024-06-29T15:57:16.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.689+0000 7fe98f01f640 1 -- 172.21.15.81:0/355070476 wait complete. 2024-06-29T15:57:16.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.690+0000 7fe98f01f640 1 Processor -- start 2024-06-29T15:57:16.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.690+0000 7fe98f01f640 1 -- start start 2024-06-29T15:57:16.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98f01f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9880ce3a0 0x7fe98809b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98f01f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9880cf600 0x7fe98809dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98f01f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9881d2bf0 0x7fe9881e5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98f01f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe9880b2830 con 0x7fe9881d2bf0 2024-06-29T15:57:16.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98f01f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe9880b26b0 con 0x7fe9880cf600 2024-06-29T15:57:16.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98f01f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe9880b6df0 con 0x7fe9880ce3a0 2024-06-29T15:57:16.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98d5bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9881d2bf0 0x7fe9881e5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98d5bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9881d2bf0 0x7fe9881e5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42686/0 (socket says 172.21.15.81:42686) 2024-06-29T15:57:16.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98d5bc640 1 -- 172.21.15.81:0/3477798047 learned_addr learned my addr 172.21.15.81:0/3477798047 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:16.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98cdbb640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9880ce3a0 0x7fe98809b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe987fff640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9880cf600 0x7fe98809dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98d5bc640 1 -- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9880ce3a0 msgr2=0x7fe98809b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98d5bc640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9880ce3a0 0x7fe98809b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98d5bc640 1 -- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9880cf600 msgr2=0x7fe98809dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98d5bc640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9880cf600 0x7fe98809dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.692+0000 7fe98d5bc640 1 -- 172.21.15.81:0/3477798047 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98809e0f0 con 0x7fe9881d2bf0 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.693+0000 7fe98d5bc640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9881d2bf0 0x7fe9881e5790 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fe980009cd0 tx=0x7fe980033f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.693+0000 7fe9857fa640 1 -- 172.21.15.81:0/3477798047 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe980031990 con 0x7fe9881d2bf0 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.693+0000 7fe9857fa640 1 -- 172.21.15.81:0/3477798047 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe980031c20 con 0x7fe9881d2bf0 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.693+0000 7fe9857fa640 1 -- 172.21.15.81:0/3477798047 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9800330d0 con 0x7fe9881d2bf0 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.693+0000 7fe98f01f640 1 -- 172.21.15.81:0/3477798047 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9881e46f0 con 0x7fe9881d2bf0 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.693+0000 7fe98f01f640 1 -- 172.21.15.81:0/3477798047 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9881e4cb0 con 0x7fe9881d2bf0 2024-06-29T15:57:16.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.694+0000 7fe98f01f640 1 -- 172.21.15.81:0/3477798047 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9880cfd40 con 0x7fe9881d2bf0 2024-06-29T15:57:16.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.695+0000 7fe9857fa640 1 -- 172.21.15.81:0/3477798047 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fe980033270 con 0x7fe9881d2bf0 2024-06-29T15:57:16.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.695+0000 7fe9857fa640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe9680763f0 0x7fe968078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:16.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.695+0000 7fe9857fa640 1 -- 172.21.15.81:0/3477798047 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5845+0+0 (secure 0 0 0) 0x7fe9800b87f0 con 0x7fe9881d2bf0 2024-06-29T15:57:16.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.695+0000 7fe98cdbb640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe9680763f0 0x7fe968078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:16.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.696+0000 7fe98cdbb640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe9680763f0 0x7fe968078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe970001720 tx=0x7fe970007e90 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:16.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.699+0000 7fe9857fa640 1 -- 172.21.15.81:0/3477798047 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe9880cfd40 con 0x7fe9881d2bf0 2024-06-29T15:57:16.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.878+0000 7fe98f01f640 1 -- 172.21.15.81:0/3477798047 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe98809e360 con 0x7fe9881d2bf0 2024-06-29T15:57:16.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.882+0000 7fe9857fa640 1 -- 172.21.15.81:0/3477798047 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v72)=0 v72) ==== 122+0+0 (secure 0 0 0) 0x7fe98004a070 con 0x7fe9881d2bf0 2024-06-29T15:57:16.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 -- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe9680763f0 msgr2=0x7fe968078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe9680763f0 0x7fe968078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe970001720 tx=0x7fe970007e90 comp rx=0 tx=0).stop 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 -- 172.21.15.81:0/3477798047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9881d2bf0 msgr2=0x7fe9881e5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9881d2bf0 0x7fe9881e5790 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fe980009cd0 tx=0x7fe980033f70 comp rx=0 tx=0).stop 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 -- 172.21.15.81:0/3477798047 shutdown_connections 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe9680763f0 0x7fe968078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9881d2bf0 0x7fe9881e5790 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9880cf600 0x7fe98809dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 --2- 172.21.15.81:0/3477798047 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9880ce3a0 0x7fe98809b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 -- 172.21.15.81:0/3477798047 >> 172.21.15.81:0/3477798047 conn(0x7fe9880b35a0 msgr2=0x7fe9880b18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 -- 172.21.15.81:0/3477798047 shutdown_connections 2024-06-29T15:57:16.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:16.884+0000 7fe98f01f640 1 -- 172.21.15.81:0/3477798047 wait complete. 2024-06-29T15:57:16.895 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:57:16.896 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:16.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:16.975 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:57:16.975 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:57:16.976 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:16.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:57:16.993 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:57:16.993 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:17.033 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:17.034 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:57:17.034 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:17.049 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:57:17.049 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:57:17.049 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:17.090 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:17.091 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:57:17.091 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:17.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.204+0000 7f2d5bd38640 1 -- 172.21.15.81:0/3747162834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540cea10 msgr2=0x7f2d540b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.204+0000 7f2d5bd38640 1 --2- 172.21.15.81:0/3747162834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540cea10 0x7f2d540b47b0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f2d540b2c10 tx=0x7f2d4802a540 comp rx=0 tx=0).stop 2024-06-29T15:57:17.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.204+0000 7f2d5bd38640 1 -- 172.21.15.81:0/3747162834 shutdown_connections 2024-06-29T15:57:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.204+0000 7f2d5bd38640 1 --2- 172.21.15.81:0/3747162834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540cea10 0x7f2d540b47b0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.204+0000 7f2d5bd38640 1 --2- 172.21.15.81:0/3747162834 >> v2:172.21.15.167:3300/0 conn(0x7f2d540b6d60 0x7f2d540ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.204+0000 7f2d5bd38640 1 -- 172.21.15.81:0/3747162834 >> 172.21.15.81:0/3747162834 conn(0x7f2d540d1960 msgr2=0x7f2d540d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.204+0000 7f2d5bd38640 1 -- 172.21.15.81:0/3747162834 shutdown_connections 2024-06-29T15:57:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.204+0000 7f2d5bd38640 1 -- 172.21.15.81:0/3747162834 wait complete. 2024-06-29T15:57:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.206+0000 7f2d5bd38640 1 Processor -- start 2024-06-29T15:57:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.206+0000 7f2d5bd38640 1 -- start start 2024-06-29T15:57:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d5bd38640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b4cf0 0x7f2d541dbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d5bd38640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b6d60 0x7f2d540c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d5bd38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540cea10 0x7f2d541e6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d5bd38640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2d540cd540 con 0x7f2d540cea10 2024-06-29T15:57:17.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d5bd38640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2d540d0a10 con 0x7f2d540b4cf0 2024-06-29T15:57:17.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d5bd38640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2d540cd6c0 con 0x7f2d540b6d60 2024-06-29T15:57:17.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d5a2d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540cea10 0x7f2d541e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d5a2d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540cea10 0x7f2d541e6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42706/0 (socket says 172.21.15.81:42706) 2024-06-29T15:57:17.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d5a2d5640 1 -- 172.21.15.81:0/456526160 learned_addr learned my addr 172.21.15.81:0/456526160 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:17.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d59ad4640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b4cf0 0x7f2d541dbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.207+0000 7f2d592d3640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b6d60 0x7f2d540c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d5a2d5640 1 -- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b6d60 msgr2=0x7f2d540c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d5a2d5640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b6d60 0x7f2d540c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d5a2d5640 1 -- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b4cf0 msgr2=0x7f2d541dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d5a2d5640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b4cf0 0x7f2d541dbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d5a2d5640 1 -- 172.21.15.81:0/456526160 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d540ae710 con 0x7f2d540cea10 2024-06-29T15:57:17.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d59ad4640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b4cf0 0x7f2d541dbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:17.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d5a2d5640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540cea10 0x7f2d541e6840 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400b9a0 tx=0x7f2d4400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:17.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d527fc640 1 -- 172.21.15.81:0/456526160 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44019070 con 0x7f2d540cea10 2024-06-29T15:57:17.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d527fc640 1 -- 172.21.15.81:0/456526160 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d4400cc30 con 0x7f2d540cea10 2024-06-29T15:57:17.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d5bd38640 1 -- 172.21.15.81:0/456526160 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d540ae9f0 con 0x7f2d540cea10 2024-06-29T15:57:17.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d527fc640 1 -- 172.21.15.81:0/456526160 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d440146a0 con 0x7f2d540cea10 2024-06-29T15:57:17.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.208+0000 7f2d5bd38640 1 -- 172.21.15.81:0/456526160 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d540aefb0 con 0x7f2d540cea10 2024-06-29T15:57:17.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.209+0000 7f2d5bd38640 1 -- 172.21.15.81:0/456526160 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d540ce4d0 con 0x7f2d540cea10 2024-06-29T15:57:17.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.210+0000 7f2d527fc640 1 -- 172.21.15.81:0/456526160 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f2d4401c070 con 0x7f2d540cea10 2024-06-29T15:57:17.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.210+0000 7f2d527fc640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f2d40076420 0x7f2d40078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.210+0000 7f2d527fc640 1 -- 172.21.15.81:0/456526160 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5845+0+0 (secure 0 0 0) 0x7f2d44097690 con 0x7f2d540cea10 2024-06-29T15:57:17.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.211+0000 7f2d59ad4640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f2d40076420 0x7f2d40078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.211+0000 7f2d59ad4640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f2d40076420 0x7f2d40078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c005fd0 tx=0x7f2d4c00f360 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:17.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.215+0000 7f2d527fc640 1 -- 172.21.15.81:0/456526160 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2d4400ce20 con 0x7f2d540cea10 2024-06-29T15:57:17.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.434+0000 7f2d5bd38640 1 -- 172.21.15.81:0/456526160 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d5406a5d0 con 0x7f2d540cea10 2024-06-29T15:57:17.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.435+0000 7f2d527fc640 1 -- 172.21.15.81:0/456526160 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3919 (secure 0 0 0) 0x7f2d44060dc0 con 0x7f2d540cea10 2024-06-29T15:57:17.435 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:17.436 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":105,"btime":"2024-06-29T15:57:11:045717+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/1764785504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":1764785504},{"type":"v1","addr":"172.21.15.167:6836","nonce":1764785504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99},{"gid":10036,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1355941618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1355941618},{"type":"v1","addr":"172.21.15.167:6837","nonce":1355941618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":105},{"gid":10713,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/1699508727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1699508727},{"type":"v1","addr":"172.21.15.81:6834","nonce":1699508727}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":98},{"gid":10716,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/265635924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":265635924},{"type":"v1","addr":"172.21.15.81:6835","nonce":265635924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":99}],"filesystems":[]} 2024-06-29T15:57:17.436 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 105 2024-06-29T15:57:17.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 -- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f2d40076420 msgr2=0x7f2d40078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f2d40076420 0x7f2d40078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c005fd0 tx=0x7f2d4c00f360 comp rx=0 tx=0).stop 2024-06-29T15:57:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 -- 172.21.15.81:0/456526160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540cea10 msgr2=0x7f2d541e6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540cea10 0x7f2d541e6840 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400b9a0 tx=0x7f2d4400be60 comp rx=0 tx=0).stop 2024-06-29T15:57:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 -- 172.21.15.81:0/456526160 shutdown_connections 2024-06-29T15:57:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f2d40076420 0x7f2d40078930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540cea10 0x7f2d541e6840 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b6d60 0x7f2d540c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 --2- 172.21.15.81:0/456526160 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b4cf0 0x7f2d541dbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 -- 172.21.15.81:0/456526160 >> 172.21.15.81:0/456526160 conn(0x7f2d540d1960 msgr2=0x7f2d540b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 -- 172.21.15.81:0/456526160 shutdown_connections 2024-06-29T15:57:17.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.437+0000 7f2d5bd38640 1 -- 172.21.15.81:0/456526160 wait complete. 2024-06-29T15:57:17.449 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:57:17.449 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:57:17.449 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:57:17.449 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:57:17.449 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:57:17.449 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:57:17.450 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:57:17.450 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:57:17.450 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:57:17.450 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:57:17.450 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:57:17.450 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:57:17.451 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:57:17.451 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:57:17.451 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:57:17.451 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:57:17.459 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:57:17.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:57:17.460 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:57:17.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:57:17.463 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:57:17.463 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:57:17.465 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:57:17.465 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:57:17.502 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:57:17.502 DEBUG:teuthology.parallel:result is None 2024-06-29T15:57:17.503 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:57:17.503 DEBUG:teuthology.parallel:result is None 2024-06-29T15:57:17.507 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:57:17.508 DEBUG:teuthology.parallel:result is None 2024-06-29T15:57:17.509 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:57:17.510 DEBUG:teuthology.parallel:result is None 2024-06-29T15:57:17.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:57:17.587 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T15:57:17.587 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T15:57:17.601 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:57:17.601 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T15:57:17.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.630+0000 7ffa837bd640 1 -- 172.21.15.81:0/2392222501 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa7c0d0230 msgr2=0x7ffa7c0d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.630+0000 7ffa837bd640 1 --2- 172.21.15.81:0/2392222501 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa7c0d0230 0x7ffa7c0d0640 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009860 tx=0x7ffa7002a530 comp rx=0 tx=0).stop 2024-06-29T15:57:17.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.630+0000 7ffa837bd640 1 -- 172.21.15.81:0/2392222501 shutdown_connections 2024-06-29T15:57:17.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.630+0000 7ffa837bd640 1 --2- 172.21.15.81:0/2392222501 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa7c0d0230 0x7ffa7c0d0640 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.630+0000 7ffa837bd640 1 -- 172.21.15.81:0/2392222501 >> 172.21.15.81:0/2392222501 conn(0x7ffa7c0b3680 msgr2=0x7ffa7c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:17.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.630+0000 7ffa837bd640 1 -- 172.21.15.81:0/2392222501 shutdown_connections 2024-06-29T15:57:17.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.631+0000 7ffa837bd640 1 -- 172.21.15.81:0/2392222501 wait complete. 2024-06-29T15:57:17.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.632+0000 7ffa837bd640 1 Processor -- start 2024-06-29T15:57:17.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.632+0000 7ffa837bd640 1 -- start start 2024-06-29T15:57:17.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.633+0000 7ffa837bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa7c0b3200 0x7ffa7c156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa837bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa7c0b4f40 0x7ffa7c1501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa837bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa7c0d0230 0x7ffa7c1530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa837bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c0b5e20 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa837bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c0b5ca0 con 0x7ffa7c0b3200 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa837bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ffa7c0b5fa0 con 0x7ffa7c0d0230 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa80d58640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa7c0b4f40 0x7ffa7c1501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa80d58640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa7c0b4f40 0x7ffa7c1501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42746/0 (socket says 172.21.15.81:42746) 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa80d58640 1 -- 172.21.15.81:0/2451334686 learned_addr learned my addr 172.21.15.81:0/2451334686 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa81559640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa7c0b3200 0x7ffa7c156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa81d5a640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa7c0d0230 0x7ffa7c1530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa80d58640 1 -- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa7c0d0230 msgr2=0x7ffa7c1530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa80d58640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa7c0d0230 0x7ffa7c1530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa80d58640 1 -- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa7c0b3200 msgr2=0x7ffa7c156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa80d58640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa7c0b3200 0x7ffa7c156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:17.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa80d58640 1 -- 172.21.15.81:0/2451334686 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7c158fc0 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa80d58640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa7c0b4f40 0x7ffa7c1501b0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7ffa7002ab20 tx=0x7ffa70003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa79ffb640 1 -- 172.21.15.81:0/2451334686 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa70005c90 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa837bd640 1 -- 172.21.15.81:0/2451334686 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa7c150e70 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa79ffb640 1 -- 172.21.15.81:0/2451334686 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa70005020 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.634+0000 7ffa79ffb640 1 -- 172.21.15.81:0/2451334686 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa70005c90 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.635+0000 7ffa837bd640 1 -- 172.21.15.81:0/2451334686 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa7c0ca560 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.636+0000 7ffa837bd640 1 -- 172.21.15.81:0/2451334686 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa7c0d0640 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.636+0000 7ffa79ffb640 1 -- 172.21.15.81:0/2451334686 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7ffa700051d0 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.637+0000 7ffa79ffb640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ffa680765f0 0x7ffa68078b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.637+0000 7ffa79ffb640 1 -- 172.21.15.81:0/2451334686 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(205..205 src has 1..205) ==== 5845+0+0 (secure 0 0 0) 0x7ffa700b8690 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.637+0000 7ffa81559640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ffa680765f0 0x7ffa68078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:17.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.637+0000 7ffa81559640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ffa680765f0 0x7ffa68078b00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0014a0 tx=0x7ffa6c009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:17.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.641+0000 7ffa79ffb640 1 -- 172.21.15.81:0/2451334686 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa7003f030 con 0x7ffa7c0b4f40 2024-06-29T15:57:17.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:17.831+0000 7ffa837bd640 1 -- 172.21.15.81:0/2451334686 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ffa7c150c00 con 0x7ffa7c0b4f40 2024-06-29T15:57:18.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.249+0000 7ffa79ffb640 1 -- 172.21.15.81:0/2451334686 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v207) ==== 97+0+0 (secure 0 0 0) 0x7ffa700815e0 con 0x7ffa7c0b4f40 2024-06-29T15:57:18.250 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 -- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ffa680765f0 msgr2=0x7ffa68078b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ffa680765f0 0x7ffa68078b00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0014a0 tx=0x7ffa6c009040 comp rx=0 tx=0).stop 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 -- 172.21.15.81:0/2451334686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa7c0b4f40 msgr2=0x7ffa7c1501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa7c0b4f40 0x7ffa7c1501b0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7ffa7002ab20 tx=0x7ffa70003c10 comp rx=0 tx=0).stop 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 -- 172.21.15.81:0/2451334686 shutdown_connections 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ffa680765f0 0x7ffa68078b00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa7c0d0230 0x7ffa7c1530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa7c0b4f40 0x7ffa7c1501b0 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 --2- 172.21.15.81:0/2451334686 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa7c0b3200 0x7ffa7c156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 -- 172.21.15.81:0/2451334686 >> 172.21.15.81:0/2451334686 conn(0x7ffa7c0b3680 msgr2=0x7ffa7c0c7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 -- 172.21.15.81:0/2451334686 shutdown_connections 2024-06-29T15:57:18.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.252+0000 7ffa837bd640 1 -- 172.21.15.81:0/2451334686 wait complete. 2024-06-29T15:57:18.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:57:18.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.377+0000 7fa259c47640 1 -- 172.21.15.81:0/3386270636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa23c003f90 msgr2=0x7fa23c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.377+0000 7fa259c47640 1 --2- 172.21.15.81:0/3386270636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa23c003f90 0x7fa23c004800 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fa248001740 tx=0x7fa23c005830 comp rx=0 tx=0).stop 2024-06-29T15:57:18.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.377+0000 7fa253fff640 1 -- 172.21.15.81:0/3386270636 reap_dead start 2024-06-29T15:57:18.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.377+0000 7fa259c47640 1 -- 172.21.15.81:0/3386270636 shutdown_connections 2024-06-29T15:57:18.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.377+0000 7fa259c47640 1 -- 172.21.15.81:0/3386270636 >> 172.21.15.81:0/3386270636 conn(0x7fa25406ccc0 msgr2=0x7fa2540b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:18.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.377+0000 7fa259c47640 1 -- 172.21.15.81:0/3386270636 shutdown_connections 2024-06-29T15:57:18.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.377+0000 7fa259c47640 1 -- 172.21.15.81:0/3386270636 wait complete. 2024-06-29T15:57:18.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.379+0000 7fa259c47640 1 Processor -- start 2024-06-29T15:57:18.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.379+0000 7fa259c47640 1 -- start start 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa259c47640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa23c003f90 0x7fa2540ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa259c47640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2540a9610 0x7fa25413b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa259c47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa25413ba90 0x7fa25414a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa259c47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa25413e3e0 con 0x7fa25413ba90 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa259c47640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa2540cf710 con 0x7fa23c003f90 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa259c47640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa25413e560 con 0x7fa2540a9610 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa253fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa25413ba90 0x7fa25414a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa253fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa25413ba90 0x7fa25414a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42772/0 (socket says 172.21.15.81:42772) 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa253fff640 1 -- 172.21.15.81:0/3623973407 learned_addr learned my addr 172.21.15.81:0/3623973407 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa252ffd640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2540a9610 0x7fa25413b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa2537fe640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa23c003f90 0x7fa2540ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa253fff640 1 -- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2540a9610 msgr2=0x7fa25413b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa253fff640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2540a9610 0x7fa25413b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa253fff640 1 -- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa23c003f90 msgr2=0x7fa2540ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa253fff640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa23c003f90 0x7fa2540ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa253fff640 1 -- 172.21.15.81:0/3623973407 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25413bea0 con 0x7fa25413ba90 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa2537fe640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa23c003f90 0x7fa2540ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa253fff640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa25413ba90 0x7fa25414a6f0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fa24c001350 tx=0x7fa24c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa23bfff640 1 -- 172.21.15.81:0/3623973407 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c00c9a0 con 0x7fa25413ba90 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa259c47640 1 -- 172.21.15.81:0/3623973407 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2540aee40 con 0x7fa25413ba90 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa23bfff640 1 -- 172.21.15.81:0/3623973407 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa24c018070 con 0x7fa25413ba90 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.381+0000 7fa23bfff640 1 -- 172.21.15.81:0/3623973407 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c00ce70 con 0x7fa25413ba90 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.382+0000 7fa259c47640 1 -- 172.21.15.81:0/3623973407 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa254157d90 con 0x7fa25413ba90 2024-06-29T15:57:18.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.383+0000 7fa259c47640 1 -- 172.21.15.81:0/3623973407 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa254157820 con 0x7fa25413ba90 2024-06-29T15:57:18.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.383+0000 7fa23bfff640 1 -- 172.21.15.81:0/3623973407 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fa24c01d460 con 0x7fa25413ba90 2024-06-29T15:57:18.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.384+0000 7fa23bfff640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa2400763f0 0x7fa240078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.384+0000 7fa23bfff640 1 -- 172.21.15.81:0/3623973407 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 4985+0+0 (secure 0 0 0) 0x7fa24c097ab0 con 0x7fa25413ba90 2024-06-29T15:57:18.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.384+0000 7fa2537fe640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa2400763f0 0x7fa240078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.384+0000 7fa2537fe640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa2400763f0 0x7fa240078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa244009c50 tx=0x7fa244005f90 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:18.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.388+0000 7fa23bfff640 1 -- 172.21.15.81:0/3623973407 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa25413bea0 con 0x7fa25413ba90 2024-06-29T15:57:18.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.638+0000 7fa259c47640 1 -- 172.21.15.81:0/3623973407 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa25413eac0 con 0x7fa25413ba90 2024-06-29T15:57:18.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.639+0000 7fa23bfff640 1 -- 172.21.15.81:0/3623973407 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v66) ==== 80+0+5304 (secure 0 0 0) 0x7fa24c061440 con 0x7fa25413ba90 2024-06-29T15:57:18.639 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:18.639 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.640 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:57:18.641 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.642 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T15:57:18.643 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.644 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T15:57:18.645 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.646 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T15:57:18.647 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:57:18.648 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T15:57:18.649 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:57:18.650 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.651 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T15:57:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:57:18.653 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:57:18.653 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:57:18.653 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:57:18.653 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.653 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:18.653 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T15:57:18.653 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 -- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa2400763f0 msgr2=0x7fa240078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa2400763f0 0x7fa240078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa244009c50 tx=0x7fa244005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 -- 172.21.15.81:0/3623973407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa25413ba90 msgr2=0x7fa25414a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa25413ba90 0x7fa25414a6f0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fa24c001350 tx=0x7fa24c00ed10 comp rx=0 tx=0).stop 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 -- 172.21.15.81:0/3623973407 shutdown_connections 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa2400763f0 0x7fa240078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa25413ba90 0x7fa25414a6f0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2540a9610 0x7fa25413b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 --2- 172.21.15.81:0/3623973407 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa23c003f90 0x7fa2540ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 -- 172.21.15.81:0/3623973407 >> 172.21.15.81:0/3623973407 conn(0x7fa25406ccc0 msgr2=0x7fa254158f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:18.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 -- 172.21.15.81:0/3623973407 shutdown_connections 2024-06-29T15:57:18.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.641+0000 7fa259c47640 1 -- 172.21.15.81:0/3623973407 wait complete. 2024-06-29T15:57:18.656 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:18.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.769+0000 7fe66c3fc640 1 -- 172.21.15.81:0/1715121312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6640b9040 msgr2=0x7fe6640d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.769+0000 7fe66c3fc640 1 --2- 172.21.15.81:0/1715121312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6640b9040 0x7fe6640d0480 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fe660005fd0 tx=0x7fe66002a5c0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.769+0000 7fe66c3fc640 1 -- 172.21.15.81:0/1715121312 shutdown_connections 2024-06-29T15:57:18.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.769+0000 7fe66c3fc640 1 --2- 172.21.15.81:0/1715121312 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6640b5250 0x7fe6640d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.769+0000 7fe66c3fc640 1 --2- 172.21.15.81:0/1715121312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6640b9040 0x7fe6640d0480 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.769+0000 7fe66c3fc640 1 -- 172.21.15.81:0/1715121312 >> 172.21.15.81:0/1715121312 conn(0x7fe66409cf30 msgr2=0x7fe6640d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:18.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.769+0000 7fe66c3fc640 1 -- 172.21.15.81:0/1715121312 shutdown_connections 2024-06-29T15:57:18.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.769+0000 7fe66c3fc640 1 -- 172.21.15.81:0/1715121312 wait complete. 2024-06-29T15:57:18.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.771+0000 7fe66c3fc640 1 Processor -- start 2024-06-29T15:57:18.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.771+0000 7fe66c3fc640 1 -- start start 2024-06-29T15:57:18.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.772+0000 7fe66c3fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6640b5250 0x7fe664152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.772+0000 7fe66c3fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6640b7290 0x7fe664152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.772+0000 7fe66c3fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6640b9040 0x7fe664153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.772+0000 7fe66c3fc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6640b7db0 con 0x7fe6640b5250 2024-06-29T15:57:18.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.772+0000 7fe66c3fc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe6640d2d50 con 0x7fe6640b9040 2024-06-29T15:57:18.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.772+0000 7fe66c3fc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe6640b7f30 con 0x7fe6640b7290 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66a198640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6640b5250 0x7fe664152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66a198640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6640b5250 0x7fe664152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42800/0 (socket says 172.21.15.81:42800) 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66a198640 1 -- 172.21.15.81:0/99036513 learned_addr learned my addr 172.21.15.81:0/99036513 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66a198640 1 -- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6640b7290 msgr2=0x7fe664152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66a198640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6640b7290 0x7fe664152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66a198640 1 -- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6640b9040 msgr2=0x7fe664153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66a198640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6640b9040 0x7fe664153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66a198640 1 -- 172.21.15.81:0/99036513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6641525d0 con 0x7fe6640b5250 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66a198640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6640b5250 0x7fe664152090 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fe660000c00 tx=0x7fe660003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe65affd640 1 -- 172.21.15.81:0/99036513 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66003e030 con 0x7fe6640b5250 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe65affd640 1 -- 172.21.15.81:0/99036513 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe660003e00 con 0x7fe6640b5250 2024-06-29T15:57:18.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66c3fc640 1 -- 172.21.15.81:0/99036513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe66409e720 con 0x7fe6640b5250 2024-06-29T15:57:18.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe65affd640 1 -- 172.21.15.81:0/99036513 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe660035040 con 0x7fe6640b5250 2024-06-29T15:57:18.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.773+0000 7fe66c3fc640 1 -- 172.21.15.81:0/99036513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe66409ece0 con 0x7fe6640b5250 2024-06-29T15:57:18.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.774+0000 7fe66c3fc640 1 -- 172.21.15.81:0/99036513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6640b84c0 con 0x7fe6640b5250 2024-06-29T15:57:18.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.775+0000 7fe65affd640 1 -- 172.21.15.81:0/99036513 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fe660041030 con 0x7fe6640b5250 2024-06-29T15:57:18.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.775+0000 7fe65affd640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe6480763f0 0x7fe648078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:18.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.776+0000 7fe65affd640 1 -- 172.21.15.81:0/99036513 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 4985+0+0 (secure 0 0 0) 0x7fe6600b7de0 con 0x7fe6640b5250 2024-06-29T15:57:18.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.776+0000 7fe669997640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe6480763f0 0x7fe648078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:18.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.776+0000 7fe669997640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe6480763f0 0x7fe648078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe6540058c0 tx=0x7fe654005850 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:18.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.780+0000 7fe65affd640 1 -- 172.21.15.81:0/99036513 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe6641525d0 con 0x7fe6640b5250 2024-06-29T15:57:18.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.961+0000 7fe66c3fc640 1 -- 172.21.15.81:0/99036513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe6640d24f0 con 0x7fe6640b5250 2024-06-29T15:57:18.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.962+0000 7fe65affd640 1 -- 172.21.15.81:0/99036513 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) ==== 74+0+13093 (secure 0 0 0) 0x7fe6600817f0 con 0x7fe6640b5250 2024-06-29T15:57:18.962 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:18.962 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":209,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:18.262316+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:18.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 -- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe6480763f0 msgr2=0x7fe648078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe6480763f0 0x7fe648078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe6540058c0 tx=0x7fe654005850 comp rx=0 tx=0).stop 2024-06-29T15:57:18.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 -- 172.21.15.81:0/99036513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6640b5250 msgr2=0x7fe664152090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:18.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6640b5250 0x7fe664152090 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fe660000c00 tx=0x7fe660003910 comp rx=0 tx=0).stop 2024-06-29T15:57:18.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 -- 172.21.15.81:0/99036513 shutdown_connections 2024-06-29T15:57:18.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe6480763f0 0x7fe648078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe6640b9040 0x7fe664153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe6640b7290 0x7fe664152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 --2- 172.21.15.81:0/99036513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6640b5250 0x7fe664152090 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:18.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 -- 172.21.15.81:0/99036513 >> 172.21.15.81:0/99036513 conn(0x7fe66409cf30 msgr2=0x7fe66415e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:18.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 -- 172.21.15.81:0/99036513 shutdown_connections 2024-06-29T15:57:18.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:18.964+0000 7fe66c3fc640 1 -- 172.21.15.81:0/99036513 wait complete. 2024-06-29T15:57:18.975 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:18.975 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:19.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.091+0000 7f406ac8f640 1 -- 172.21.15.81:0/3619493939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40640d0b90 msgr2=0x7f40640d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.091+0000 7f406ac8f640 1 --2- 172.21.15.81:0/3619493939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40640d0b90 0x7f40640d1a80 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f4054009830 tx=0x7f405402a5a0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.091+0000 7f406ac8f640 1 -- 172.21.15.81:0/3619493939 shutdown_connections 2024-06-29T15:57:19.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.091+0000 7f406ac8f640 1 --2- 172.21.15.81:0/3619493939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40640d0b90 0x7f40640d1a80 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.091+0000 7f406ac8f640 1 -- 172.21.15.81:0/3619493939 >> 172.21.15.81:0/3619493939 conn(0x7f406409cf30 msgr2=0x7f40640d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:19.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.091+0000 7f406ac8f640 1 -- 172.21.15.81:0/3619493939 shutdown_connections 2024-06-29T15:57:19.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.091+0000 7f406ac8f640 1 -- 172.21.15.81:0/3619493939 wait complete. 2024-06-29T15:57:19.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.093+0000 7f406ac8f640 1 Processor -- start 2024-06-29T15:57:19.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.093+0000 7f406ac8f640 1 -- start start 2024-06-29T15:57:19.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406ac8f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40640b5250 0x7f40640b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406ac8f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40640b9040 0x7f40641f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406ac8f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40640c7550 0x7f40640a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406ac8f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f40640d26d0 con 0x7f40640c7550 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406ac8f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f40640b85c0 con 0x7f40640b5250 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406ac8f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f40640d2850 con 0x7f40640b9040 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406922c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40640c7550 0x7f40640a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406922c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40640c7550 0x7f40640a0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42828/0 (socket says 172.21.15.81:42828) 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406922c640 1 -- 172.21.15.81:0/1545354964 learned_addr learned my addr 172.21.15.81:0/1545354964 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f4068a2b640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40640b5250 0x7f40640b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f405bfff640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40640b9040 0x7f40641f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406922c640 1 -- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40640b9040 msgr2=0x7f40641f7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406922c640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40640b9040 0x7f40641f7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406922c640 1 -- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40640b5250 msgr2=0x7f40640b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406922c640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40640b5250 0x7f40640b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406922c640 1 -- 172.21.15.81:0/1545354964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40640b2030 con 0x7f40640c7550 2024-06-29T15:57:19.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f405bfff640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40640b9040 0x7f40641f7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:19.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406922c640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40640c7550 0x7f40640a0a00 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f405c00b410 tx=0x7f405c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f40597fa640 1 -- 172.21.15.81:0/1545354964 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c015030 con 0x7f40640c7550 2024-06-29T15:57:19.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f40597fa640 1 -- 172.21.15.81:0/1545354964 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f405c004130 con 0x7f40640c7550 2024-06-29T15:57:19.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406ac8f640 1 -- 172.21.15.81:0/1545354964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40640b2310 con 0x7f40640c7550 2024-06-29T15:57:19.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f40597fa640 1 -- 172.21.15.81:0/1545354964 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c011630 con 0x7f40640c7550 2024-06-29T15:57:19.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.095+0000 7f406ac8f640 1 -- 172.21.15.81:0/1545354964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40640b28d0 con 0x7f40640c7550 2024-06-29T15:57:19.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.096+0000 7f406ac8f640 1 -- 172.21.15.81:0/1545354964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40640d2de0 con 0x7f40640c7550 2024-06-29T15:57:19.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.097+0000 7f40597fa640 1 -- 172.21.15.81:0/1545354964 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f405c018070 con 0x7f40640c7550 2024-06-29T15:57:19.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.098+0000 7f40597fa640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4050076420 0x7f4050078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.098+0000 7f40597fa640 1 -- 172.21.15.81:0/1545354964 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 4985+0+0 (secure 0 0 0) 0x7f405c097250 con 0x7f40640c7550 2024-06-29T15:57:19.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.098+0000 7f4068a2b640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4050076420 0x7f4050078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.098+0000 7f4068a2b640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4050076420 0x7f4050078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4060007fd0 tx=0x7f4060009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.102+0000 7f40597fa640 1 -- 172.21.15.81:0/1545354964 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f405c060c60 con 0x7f40640c7550 2024-06-29T15:57:19.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.285+0000 7f406ac8f640 1 -- 172.21.15.81:0/1545354964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f40640c7100 con 0x7f40640c7550 2024-06-29T15:57:19.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.286+0000 7f40597fa640 1 -- 172.21.15.81:0/1545354964 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v209) ==== 93+0+10 (secure 0 0 0) 0x7f405c004580 con 0x7f40640c7550 2024-06-29T15:57:19.287 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:57:19.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 -- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4050076420 msgr2=0x7f4050078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4050076420 0x7f4050078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4060007fd0 tx=0x7f4060009040 comp rx=0 tx=0).stop 2024-06-29T15:57:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 -- 172.21.15.81:0/1545354964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40640c7550 msgr2=0x7f40640a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40640c7550 0x7f40640a0a00 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f405c00b410 tx=0x7f405c00b8d0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 -- 172.21.15.81:0/1545354964 shutdown_connections 2024-06-29T15:57:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4050076420 0x7f4050078930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40640c7550 0x7f40640a0a00 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40640b9040 0x7f40641f7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 --2- 172.21.15.81:0/1545354964 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40640b5250 0x7f40640b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 -- 172.21.15.81:0/1545354964 >> 172.21.15.81:0/1545354964 conn(0x7f406409cf30 msgr2=0x7f406406e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 -- 172.21.15.81:0/1545354964 shutdown_connections 2024-06-29T15:57:19.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.288+0000 7f406ac8f640 1 -- 172.21.15.81:0/1545354964 wait complete. 2024-06-29T15:57:19.299 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:57:19.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.411+0000 7fd92121f640 1 -- 172.21.15.81:0/3808814901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd91c06f6a0 msgr2=0x7fd91c15aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.411+0000 7fd92121f640 1 --2- 172.21.15.81:0/3808814901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd91c06f6a0 0x7fd91c15aed0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fd904005fd0 tx=0x7fd90402a530 comp rx=0 tx=0).stop 2024-06-29T15:57:19.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.411+0000 7fd92121f640 1 -- 172.21.15.81:0/3808814901 shutdown_connections 2024-06-29T15:57:19.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.411+0000 7fd92121f640 1 --2- 172.21.15.81:0/3808814901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd91c0b3960 0x7fd91c0b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.411+0000 7fd92121f640 1 --2- 172.21.15.81:0/3808814901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd91c06f6a0 0x7fd91c15aed0 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.411+0000 7fd92121f640 1 -- 172.21.15.81:0/3808814901 >> 172.21.15.81:0/3808814901 conn(0x7fd91c0d0930 msgr2=0x7fd91c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:19.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.411+0000 7fd92121f640 1 -- 172.21.15.81:0/3808814901 shutdown_connections 2024-06-29T15:57:19.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.411+0000 7fd92121f640 1 -- 172.21.15.81:0/3808814901 wait complete. 2024-06-29T15:57:19.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.413+0000 7fd92121f640 1 Processor -- start 2024-06-29T15:57:19.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.413+0000 7fd92121f640 1 -- start start 2024-06-29T15:57:19.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd92121f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd91c06f6a0 0x7fd91c14ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd92121f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd91c0b3960 0x7fd91c0af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd92121f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd91c0b57b0 0x7fd91c06c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd92121f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c0b4430 con 0x7fd91c0b3960 2024-06-29T15:57:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd92121f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c0b42b0 con 0x7fd91c06f6a0 2024-06-29T15:57:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd92121f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd91c0b45b0 con 0x7fd91c0b57b0 2024-06-29T15:57:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd913fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd91c0b3960 0x7fd91c0af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd913fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd91c0b3960 0x7fd91c0af3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42838/0 (socket says 172.21.15.81:42838) 2024-06-29T15:57:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd913fff640 1 -- 172.21.15.81:0/299982086 learned_addr learned my addr 172.21.15.81:0/299982086 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd91b59e640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd91c0b57b0 0x7fd91c06c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd91ad9d640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd91c06f6a0 0x7fd91c14ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd913fff640 1 -- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd91c0b57b0 msgr2=0x7fd91c06c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd913fff640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd91c0b57b0 0x7fd91c06c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd913fff640 1 -- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd91c06f6a0 msgr2=0x7fd91c14ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd913fff640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd91c06f6a0 0x7fd91c14ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd913fff640 1 -- 172.21.15.81:0/299982086 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91c0ada50 con 0x7fd91c0b3960 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd91ad9d640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd91c06f6a0 0x7fd91c14ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd913fff640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd91c0b3960 0x7fd91c0af3a0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fd904000c00 tx=0x7fd904003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd9137fe640 1 -- 172.21.15.81:0/299982086 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90403e030 con 0x7fd91c0b3960 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd92121f640 1 -- 172.21.15.81:0/299982086 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd91c0add30 con 0x7fd91c0b3960 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.415+0000 7fd9137fe640 1 -- 172.21.15.81:0/299982086 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd904003e60 con 0x7fd91c0b3960 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.416+0000 7fd9137fe640 1 -- 172.21.15.81:0/299982086 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904035040 con 0x7fd91c0b3960 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.416+0000 7fd92121f640 1 -- 172.21.15.81:0/299982086 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd91c0ae2f0 con 0x7fd91c0b3960 2024-06-29T15:57:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.416+0000 7fd92121f640 1 -- 172.21.15.81:0/299982086 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd91c06e2a0 con 0x7fd91c0b3960 2024-06-29T15:57:19.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.417+0000 7fd9137fe640 1 -- 172.21.15.81:0/299982086 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fd904041030 con 0x7fd91c0b3960 2024-06-29T15:57:19.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.418+0000 7fd9137fe640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd8fc0763f0 0x7fd8fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.418+0000 7fd9137fe640 1 -- 172.21.15.81:0/299982086 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 4985+0+0 (secure 0 0 0) 0x7fd9040b8060 con 0x7fd91c0b3960 2024-06-29T15:57:19.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.418+0000 7fd91ad9d640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd8fc0763f0 0x7fd8fc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.418+0000 7fd91ad9d640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd8fc0763f0 0x7fd8fc078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd90c006fd0 tx=0x7fd90c008040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.422+0000 7fd9137fe640 1 -- 172.21.15.81:0/299982086 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd91c06e2a0 con 0x7fd91c0b3960 2024-06-29T15:57:19.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.605+0000 7fd92121f640 1 -- 172.21.15.81:0/299982086 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd91c0ad7e0 con 0x7fd91c0b3960 2024-06-29T15:57:19.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.606+0000 7fd9137fe640 1 -- 172.21.15.81:0/299982086 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v209) ==== 84+0+65 (secure 0 0 0) 0x7fd904081960 con 0x7fd91c0b3960 2024-06-29T15:57:19.606 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:19.607 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:57:19.607 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:19.607 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T15:57:19.607 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T15:57:19.607 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:19.607 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 -- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd8fc0763f0 msgr2=0x7fd8fc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd8fc0763f0 0x7fd8fc078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd90c006fd0 tx=0x7fd90c008040 comp rx=0 tx=0).stop 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 -- 172.21.15.81:0/299982086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd91c0b3960 msgr2=0x7fd91c0af3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd91c0b3960 0x7fd91c0af3a0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fd904000c00 tx=0x7fd904003970 comp rx=0 tx=0).stop 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 -- 172.21.15.81:0/299982086 shutdown_connections 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd8fc0763f0 0x7fd8fc078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd91c0b57b0 0x7fd91c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd91c0b3960 0x7fd91c0af3a0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 --2- 172.21.15.81:0/299982086 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd91c06f6a0 0x7fd91c14ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 -- 172.21.15.81:0/299982086 >> 172.21.15.81:0/299982086 conn(0x7fd91c0d0930 msgr2=0x7fd91c158bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:19.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 -- 172.21.15.81:0/299982086 shutdown_connections 2024-06-29T15:57:19.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.608+0000 7fd92121f640 1 -- 172.21.15.81:0/299982086 wait complete. 2024-06-29T15:57:19.620 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:57:19.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:57:19.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.737+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/694845538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b240ae8f0 msgr2=0x7f0b2415c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.737+0000 7f0b2c7d4640 1 --2- 172.21.15.81:0/694845538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b240ae8f0 0x7f0b2415c0e0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f0b20005fd0 tx=0x7f0b2002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.737+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/694845538 shutdown_connections 2024-06-29T15:57:19.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.737+0000 7f0b2c7d4640 1 --2- 172.21.15.81:0/694845538 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b240b2cb0 0x7f0b240d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.737+0000 7f0b2c7d4640 1 --2- 172.21.15.81:0/694845538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b240ae8f0 0x7f0b2415c0e0 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.737+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/694845538 >> 172.21.15.81:0/694845538 conn(0x7f0b2409aa60 msgr2=0x7f0b2409ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:19.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.737+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/694845538 shutdown_connections 2024-06-29T15:57:19.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.737+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/694845538 wait complete. 2024-06-29T15:57:19.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.739+0000 7f0b2c7d4640 1 Processor -- start 2024-06-29T15:57:19.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.739+0000 7f0b2c7d4640 1 -- start start 2024-06-29T15:57:19.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2c7d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b240ae8f0 0x7f0b24143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2c7d4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b240b1a90 0x7f0b24142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2c7d4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b240b2cb0 0x7f0b2415ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2c7d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0b24160050 con 0x7f0b240ae8f0 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2c7d4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0b2415fed0 con 0x7f0b240b1a90 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2c7d4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0b241601d0 con 0x7f0b240b2cb0 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2a570640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b240ae8f0 0x7f0b24143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2a570640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b240ae8f0 0x7f0b24143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42860/0 (socket says 172.21.15.81:42860) 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2a570640 1 -- 172.21.15.81:0/2373099491 learned_addr learned my addr 172.21.15.81:0/2373099491 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2ad71640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b240b2cb0 0x7f0b2415ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b29d6f640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b240b1a90 0x7f0b24142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2a570640 1 -- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b240b2cb0 msgr2=0x7f0b2415ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2a570640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b240b2cb0 0x7f0b2415ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2a570640 1 -- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b240b1a90 msgr2=0x7f0b24142360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:19.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2a570640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b240b1a90 0x7f0b24142360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:19.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b2a570640 1 -- 172.21.15.81:0/2373099491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b24142c90 con 0x7f0b240ae8f0 2024-06-29T15:57:19.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.740+0000 7f0b29d6f640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b240b1a90 0x7f0b24142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:19.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.741+0000 7f0b2a570640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b240ae8f0 0x7f0b24143d30 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f0b2002aea0 tx=0x7f0b20003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.741+0000 7f0b1affd640 1 -- 172.21.15.81:0/2373099491 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b20031b90 con 0x7f0b240ae8f0 2024-06-29T15:57:19.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.741+0000 7f0b1affd640 1 -- 172.21.15.81:0/2373099491 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b20031e20 con 0x7f0b240ae8f0 2024-06-29T15:57:19.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.741+0000 7f0b1affd640 1 -- 172.21.15.81:0/2373099491 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b20033330 con 0x7f0b240ae8f0 2024-06-29T15:57:19.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.741+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/2373099491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b24142f70 con 0x7f0b240ae8f0 2024-06-29T15:57:19.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.741+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/2373099491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b24143530 con 0x7f0b240ae8f0 2024-06-29T15:57:19.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.742+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/2373099491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b240d06d0 con 0x7f0b240ae8f0 2024-06-29T15:57:19.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.742+0000 7f0b1affd640 1 -- 172.21.15.81:0/2373099491 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f0b200334d0 con 0x7f0b240ae8f0 2024-06-29T15:57:19.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.743+0000 7f0b1affd640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b08076420 0x7f0b08078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:19.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.743+0000 7f0b1affd640 1 -- 172.21.15.81:0/2373099491 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 4985+0+0 (secure 0 0 0) 0x7f0b200b8c30 con 0x7f0b240ae8f0 2024-06-29T15:57:19.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.743+0000 7f0b29d6f640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b08076420 0x7f0b08078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:19.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.744+0000 7f0b29d6f640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b08076420 0x7f0b08078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b240b6dc0 tx=0x7f0b14005ef0 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:19.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.747+0000 7f0b1affd640 1 -- 172.21.15.81:0/2373099491 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0b240d06d0 con 0x7f0b240ae8f0 2024-06-29T15:57:19.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:19.929+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/2373099491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0b240b6e60 con 0x7f0b240ae8f0 2024-06-29T15:57:20.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.258+0000 7f0b1affd640 1 -- 172.21.15.81:0/2373099491 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v210) ==== 138+0+0 (secure 0 0 0) 0x7f0b2003f030 con 0x7f0b240ae8f0 2024-06-29T15:57:20.259 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T15:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b08076420 msgr2=0x7f0b08078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b08076420 0x7f0b08078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0b240b6dc0 tx=0x7f0b14005ef0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/2373099491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b240ae8f0 msgr2=0x7f0b24143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b240ae8f0 0x7f0b24143d30 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f0b2002aea0 tx=0x7f0b20003970 comp rx=0 tx=0).stop 2024-06-29T15:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/2373099491 shutdown_connections 2024-06-29T15:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b08076420 0x7f0b08078930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0b240b2cb0 0x7f0b2415ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0b240b1a90 0x7f0b24142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 --2- 172.21.15.81:0/2373099491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b240ae8f0 0x7f0b24143d30 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/2373099491 >> 172.21.15.81:0/2373099491 conn(0x7f0b2409aa60 msgr2=0x7f0b240706b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/2373099491 shutdown_connections 2024-06-29T15:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.261+0000 7f0b2c7d4640 1 -- 172.21.15.81:0/2373099491 wait complete. 2024-06-29T15:57:20.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:57:20.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.398+0000 7f9d836ef640 1 -- 172.21.15.81:0/1634611190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7c15d750 msgr2=0x7f9d7c0afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:20.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.398+0000 7f9d836ef640 1 --2- 172.21.15.81:0/1634611190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7c15d750 0x7f9d7c0afb60 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f9d78005fd0 tx=0x7f9d7802a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.398+0000 7f9d836ef640 1 -- 172.21.15.81:0/1634611190 shutdown_connections 2024-06-29T15:57:20.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.398+0000 7f9d836ef640 1 --2- 172.21.15.81:0/1634611190 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d7c06ca00 0x7f9d7c0cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.398+0000 7f9d836ef640 1 --2- 172.21.15.81:0/1634611190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7c15d750 0x7f9d7c0afb60 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.398+0000 7f9d836ef640 1 -- 172.21.15.81:0/1634611190 >> 172.21.15.81:0/1634611190 conn(0x7f9d7c0d0930 msgr2=0x7f9d7c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:20.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.398+0000 7f9d836ef640 1 -- 172.21.15.81:0/1634611190 shutdown_connections 2024-06-29T15:57:20.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.398+0000 7f9d836ef640 1 -- 172.21.15.81:0/1634611190 wait complete. 2024-06-29T15:57:20.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.399+0000 7f9d836ef640 1 Processor -- start 2024-06-29T15:57:20.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.399+0000 7f9d836ef640 1 -- start start 2024-06-29T15:57:20.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d836ef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d7c15c030 0x7f9d7c14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:20.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d836ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7c15d750 0x7f9d7c1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d836ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d7c156800 0x7f9d7c1f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d836ef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c0cfe60 con 0x7f9d7c15d750 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d836ef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c0cfce0 con 0x7f9d7c15c030 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d836ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9d7c0cffe0 con 0x7f9d7c156800 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d80c8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7c15d750 0x7f9d7c1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d80c8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7c15d750 0x7f9d7c1562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59742/0 (socket says 172.21.15.81:59742) 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d80c8a640 1 -- 172.21.15.81:0/3423774894 learned_addr learned my addr 172.21.15.81:0/3423774894 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d81c8c640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d7c156800 0x7f9d7c1f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d8148b640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d7c15c030 0x7f9d7c14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d80c8a640 1 -- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d7c156800 msgr2=0x7f9d7c1f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d80c8a640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d7c156800 0x7f9d7c1f2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d80c8a640 1 -- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d7c15c030 msgr2=0x7f9d7c14ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:20.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d80c8a640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d7c15c030 0x7f9d7c14ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:20.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d80c8a640 1 -- 172.21.15.81:0/3423774894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7c156f20 con 0x7f9d7c15d750 2024-06-29T15:57:20.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d8148b640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d7c15c030 0x7f9d7c14ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:20.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d80c8a640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7c15d750 0x7f9d7c1562c0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c400 tx=0x7f9d6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:20.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.401+0000 7f9d71ffb640 1 -- 172.21.15.81:0/3423774894 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c017070 con 0x7f9d7c15d750 2024-06-29T15:57:20.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.402+0000 7f9d71ffb640 1 -- 172.21.15.81:0/3423774894 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d6c0053c0 con 0x7f9d7c15d750 2024-06-29T15:57:20.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.402+0000 7f9d836ef640 1 -- 172.21.15.81:0/3423774894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d7c157200 con 0x7f9d7c15d750 2024-06-29T15:57:20.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.402+0000 7f9d71ffb640 1 -- 172.21.15.81:0/3423774894 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c003850 con 0x7f9d7c15d750 2024-06-29T15:57:20.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.402+0000 7f9d836ef640 1 -- 172.21.15.81:0/3423774894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d7c1504b0 con 0x7f9d7c15d750 2024-06-29T15:57:20.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.403+0000 7f9d836ef640 1 -- 172.21.15.81:0/3423774894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d7c06e2a0 con 0x7f9d7c15d750 2024-06-29T15:57:20.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.404+0000 7f9d71ffb640 1 -- 172.21.15.81:0/3423774894 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f9d6c01a070 con 0x7f9d7c15d750 2024-06-29T15:57:20.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.404+0000 7f9d71ffb640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9d600763f0 0x7f9d60078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:20.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.404+0000 7f9d71ffb640 1 -- 172.21.15.81:0/3423774894 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 5355+0+0 (secure 0 0 0) 0x7f9d6c096e20 con 0x7f9d7c15d750 2024-06-29T15:57:20.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.404+0000 7f9d8148b640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9d600763f0 0x7f9d60078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:20.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.405+0000 7f9d8148b640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9d600763f0 0x7f9d60078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d78033000 tx=0x7f9d78004730 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:20.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.408+0000 7f9d71ffb640 1 -- 172.21.15.81:0/3423774894 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9d7c06e2a0 con 0x7f9d7c15d750 2024-06-29T15:57:20.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:20.589+0000 7f9d836ef640 1 -- 172.21.15.81:0/3423774894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9d7c0d0500 con 0x7f9d7c15d750 2024-06-29T15:57:21.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.261+0000 7f9d71ffb640 1 -- 172.21.15.81:0/3423774894 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v211) ==== 170+0+0 (secure 0 0 0) 0x7f9d6c021020 con 0x7f9d7c15d750 2024-06-29T15:57:21.262 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T15:57:21.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.264+0000 7f9d836ef640 1 -- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9d600763f0 msgr2=0x7f9d60078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:21.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.264+0000 7f9d836ef640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9d600763f0 0x7f9d60078900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d78033000 tx=0x7f9d78004730 comp rx=0 tx=0).stop 2024-06-29T15:57:21.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.264+0000 7f9d836ef640 1 -- 172.21.15.81:0/3423774894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7c15d750 msgr2=0x7f9d7c1562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:21.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.264+0000 7f9d836ef640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7c15d750 0x7f9d7c1562c0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00c400 tx=0x7f9d6c00c8c0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.264+0000 7f9d836ef640 1 -- 172.21.15.81:0/3423774894 shutdown_connections 2024-06-29T15:57:21.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.264+0000 7f9d836ef640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f9d600763f0 0x7f9d60078900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.264+0000 7f9d836ef640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9d7c156800 0x7f9d7c1f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.264+0000 7f9d836ef640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7c15d750 0x7f9d7c1562c0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.264+0000 7f9d836ef640 1 --2- 172.21.15.81:0/3423774894 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9d7c15c030 0x7f9d7c14ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.264+0000 7f9d836ef640 1 -- 172.21.15.81:0/3423774894 >> 172.21.15.81:0/3423774894 conn(0x7f9d7c0d0930 msgr2=0x7f9d7c1523f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:21.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.265+0000 7f9d836ef640 1 -- 172.21.15.81:0/3423774894 shutdown_connections 2024-06-29T15:57:21.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.265+0000 7f9d836ef640 1 -- 172.21.15.81:0/3423774894 wait complete. 2024-06-29T15:57:21.289 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:57:21.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.406+0000 7f1efdc3c640 1 -- 172.21.15.81:0/3455218135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ee0003a50 msgr2=0x7f1ee0003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:21.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.406+0000 7f1efdc3c640 1 --2- 172.21.15.81:0/3455218135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ee0003a50 0x7f1ee0003ec0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009b50 tx=0x7f1ee0005830 comp rx=0 tx=0).stop 2024-06-29T15:57:21.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.406+0000 7f1efdc3c640 1 -- 172.21.15.81:0/3455218135 shutdown_connections 2024-06-29T15:57:21.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.406+0000 7f1efdc3c640 1 --2- 172.21.15.81:0/3455218135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ee0003a50 0x7f1ee0003ec0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.406+0000 7f1efdc3c640 1 -- 172.21.15.81:0/3455218135 >> 172.21.15.81:0/3455218135 conn(0x7f1ef806ccc0 msgr2=0x7f1ef80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:21.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.406+0000 7f1efdc3c640 1 -- 172.21.15.81:0/3455218135 shutdown_connections 2024-06-29T15:57:21.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.406+0000 7f1efdc3c640 1 -- 172.21.15.81:0/3455218135 wait complete. 2024-06-29T15:57:21.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.408+0000 7f1efdc3c640 1 Processor -- start 2024-06-29T15:57:21.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.408+0000 7f1efdc3c640 1 -- start start 2024-06-29T15:57:21.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.409+0000 7f1efdc3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ee0003a50 0x7f1ef81507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:21.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.409+0000 7f1efdc3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ef81430f0 0x7f1ef8138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:21.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.409+0000 7f1efdc3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ef8139510 0x7f1ef8141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:21.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.409+0000 7f1efdc3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef80731b0 con 0x7f1ee0003a50 2024-06-29T15:57:21.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.409+0000 7f1efdc3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef8073030 con 0x7f1ef8139510 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.409+0000 7f1efdc3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1ef8073330 con 0x7f1ef81430f0 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.409+0000 7f1ef77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ee0003a50 0x7f1ef81507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ee0003a50 0x7f1ef81507f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59758/0 (socket says 172.21.15.81:59758) 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef77fe640 1 -- 172.21.15.81:0/2278719940 learned_addr learned my addr 172.21.15.81:0/2278719940 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef7fff640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ef8139510 0x7f1ef8141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef6ffd640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ef81430f0 0x7f1ef8138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef77fe640 1 -- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ef81430f0 msgr2=0x7f1ef8138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef77fe640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ef81430f0 0x7f1ef8138fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef77fe640 1 -- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ef8139510 msgr2=0x7f1ef8141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef77fe640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ef8139510 0x7f1ef8141c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef77fe640 1 -- 172.21.15.81:0/2278719940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef8139920 con 0x7f1ee0003a50 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef7fff640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ef8139510 0x7f1ef8141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:21.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef6ffd640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ef81430f0 0x7f1ef8138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:21.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1ef77fe640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ee0003a50 0x7f1ef81507f0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009a70 tx=0x7f1ee8033c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:21.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1edffff640 1 -- 172.21.15.81:0/2278719940 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8042020 con 0x7f1ee0003a50 2024-06-29T15:57:21.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1edffff640 1 -- 172.21.15.81:0/2278719940 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ee803e470 con 0x7f1ee0003a50 2024-06-29T15:57:21.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1efdc3c640 1 -- 172.21.15.81:0/2278719940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ef81379e0 con 0x7f1ee0003a50 2024-06-29T15:57:21.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1edffff640 1 -- 172.21.15.81:0/2278719940 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8038040 con 0x7f1ee0003a50 2024-06-29T15:57:21.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.410+0000 7f1efdc3c640 1 -- 172.21.15.81:0/2278719940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ef8137fa0 con 0x7f1ee0003a50 2024-06-29T15:57:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.411+0000 7f1efdc3c640 1 -- 172.21.15.81:0/2278719940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ef8073890 con 0x7f1ee0003a50 2024-06-29T15:57:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.412+0000 7f1edffff640 1 -- 172.21.15.81:0/2278719940 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f1ee8045020 con 0x7f1ee0003a50 2024-06-29T15:57:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.413+0000 7f1edffff640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f1ee4076420 0x7f1ee4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.413+0000 7f1edffff640 1 -- 172.21.15.81:0/2278719940 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5737+0+0 (secure 0 0 0) 0x7f1ee80bac00 con 0x7f1ee0003a50 2024-06-29T15:57:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.413+0000 7f1ef6ffd640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f1ee4076420 0x7f1ee4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:21.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.414+0000 7f1ef6ffd640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f1ee4076420 0x7f1ee4078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1ef806f0e0 tx=0x7f1eec0016e0 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:21.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.417+0000 7f1edffff640 1 -- 172.21.15.81:0/2278719940 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1ee80c1080 con 0x7f1ee0003a50 2024-06-29T15:57:21.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:21.636+0000 7f1efdc3c640 1 -- 172.21.15.81:0/2278719940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f1ef80b4840 con 0x7f1ee0003a50 2024-06-29T15:57:22.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.284+0000 7f1edffff640 1 -- 172.21.15.81:0/2278719940 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v108) ==== 174+0+0 (secure 0 0 0) 0x7f1ee804c030 con 0x7f1ee0003a50 2024-06-29T15:57:22.289 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 26 and data pool 27 2024-06-29T15:57:22.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.297+0000 7f1efdc3c640 1 -- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f1ee4076420 msgr2=0x7f1ee4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.297+0000 7f1efdc3c640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f1ee4076420 0x7f1ee4078930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1ef806f0e0 tx=0x7f1eec0016e0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.297+0000 7f1efdc3c640 1 -- 172.21.15.81:0/2278719940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ee0003a50 msgr2=0x7f1ef81507f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.297+0000 7f1efdc3c640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ee0003a50 0x7f1ef81507f0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009a70 tx=0x7f1ee8033c30 comp rx=0 tx=0).stop 2024-06-29T15:57:22.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.297+0000 7f1efdc3c640 1 -- 172.21.15.81:0/2278719940 shutdown_connections 2024-06-29T15:57:22.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.297+0000 7f1efdc3c640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f1ee4076420 0x7f1ee4078930 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1ef806f0e0 tx=0x7f1eec0016e0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.297+0000 7f1efdc3c640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1ef8139510 0x7f1ef8141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.297+0000 7f1efdc3c640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1ef81430f0 0x7f1ef8138fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.297+0000 7f1efdc3c640 1 --2- 172.21.15.81:0/2278719940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ee0003a50 0x7f1ef81507f0 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.297+0000 7f1efdc3c640 1 -- 172.21.15.81:0/2278719940 >> 172.21.15.81:0/2278719940 conn(0x7f1ef806ccc0 msgr2=0x7f1ef80b4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:22.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.299+0000 7f1efdc3c640 1 -- 172.21.15.81:0/2278719940 shutdown_connections 2024-06-29T15:57:22.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.299+0000 7f1efdc3c640 1 -- 172.21.15.81:0/2278719940 wait complete. 2024-06-29T15:57:22.320 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:22.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.433+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3831230792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d40003a30 msgr2=0x7f5d40003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.433+0000 7f5d5fe38640 1 --2- 172.21.15.81:0/3831230792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d40003a30 0x7f5d40003ea0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f5d50006fc0 tx=0x7f5d40005830 comp rx=0 tx=0).stop 2024-06-29T15:57:22.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.433+0000 7f5d5e3d5640 1 -- 172.21.15.81:0/3831230792 reap_dead start 2024-06-29T15:57:22.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.434+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3831230792 shutdown_connections 2024-06-29T15:57:22.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.434+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3831230792 >> 172.21.15.81:0/3831230792 conn(0x7f5d5806ccc0 msgr2=0x7f5d580b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:22.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.434+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3831230792 shutdown_connections 2024-06-29T15:57:22.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.434+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3831230792 wait complete. 2024-06-29T15:57:22.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.435+0000 7f5d5fe38640 1 Processor -- start 2024-06-29T15:57:22.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.435+0000 7f5d5fe38640 1 -- start start 2024-06-29T15:57:22.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5fe38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d40003a30 0x7f5d5814c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5fe38640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d58153580 0x7f5d58159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5fe38640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d58159a40 0x7f5d58151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5fe38640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5d580b4640 con 0x7f5d40003a30 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5fe38640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5d58070010 con 0x7f5d58153580 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5fe38640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5d580b47c0 con 0x7f5d58159a40 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5dbd4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d40003a30 0x7f5d5814c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5dbd4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d40003a30 0x7f5d5814c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59770/0 (socket says 172.21.15.81:59770) 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5dbd4640 1 -- 172.21.15.81:0/3658141923 learned_addr learned my addr 172.21.15.81:0/3658141923 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5e3d5640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d58159a40 0x7f5d58151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5d3d3640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d58153580 0x7f5d58159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5dbd4640 1 -- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d58159a40 msgr2=0x7f5d58151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5dbd4640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d58159a40 0x7f5d58151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5dbd4640 1 -- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d58153580 msgr2=0x7f5d58159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5dbd4640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d58153580 0x7f5d58159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5dbd4640 1 -- 172.21.15.81:0/3658141923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d58159e50 con 0x7f5d40003a30 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5e3d5640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d58159a40 0x7f5d58151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5d3d3640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d58153580 0x7f5d58159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5dbd4640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d40003a30 0x7f5d5814c3a0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000d430 tx=0x7f5d500065b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d567fc640 1 -- 172.21.15.81:0/3658141923 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5003a070 con 0x7f5d40003a30 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d567fc640 1 -- 172.21.15.81:0/3658141923 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d5002cc60 con 0x7f5d40003a30 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3658141923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d58153f90 con 0x7f5d40003a30 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d567fc640 1 -- 172.21.15.81:0/3658141923 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50001cb0 con 0x7f5d40003a30 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.437+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3658141923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d58154550 con 0x7f5d40003a30 2024-06-29T15:57:22.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.438+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3658141923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d580cf130 con 0x7f5d40003a30 2024-06-29T15:57:22.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.439+0000 7f5d567fc640 1 -- 172.21.15.81:0/3658141923 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f5d50045020 con 0x7f5d40003a30 2024-06-29T15:57:22.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.439+0000 7f5d567fc640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5d44076420 0x7f5d44078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.440+0000 7f5d567fc640 1 -- 172.21.15.81:0/3658141923 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5837+0+0 (secure 0 0 0) 0x7f5d500b7b30 con 0x7f5d40003a30 2024-06-29T15:57:22.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.440+0000 7f5d5d3d3640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5d44076420 0x7f5d44078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.440+0000 7f5d5d3d3640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5d44076420 0x7f5d44078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5d580cd490 tx=0x7f5d4c006c00 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:22.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.444+0000 7f5d567fc640 1 -- 172.21.15.81:0/3658141923 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d58159e50 con 0x7f5d40003a30 2024-06-29T15:57:22.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.626+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3658141923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d5815c240 con 0x7f5d40003a30 2024-06-29T15:57:22.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.627+0000 7f5d567fc640 1 -- 172.21.15.81:0/3658141923 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) ==== 74+0+16888 (secure 0 0 0) 0x7f5d50081470 con 0x7f5d40003a30 2024-06-29T15:57:22.629 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:22.629 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":212,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:21.637096+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:22.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5d44076420 msgr2=0x7f5d44078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5d44076420 0x7f5d44078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5d580cd490 tx=0x7f5d4c006c00 comp rx=0 tx=0).stop 2024-06-29T15:57:22.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3658141923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d40003a30 msgr2=0x7f5d5814c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d40003a30 0x7f5d5814c3a0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000d430 tx=0x7f5d500065b0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3658141923 shutdown_connections 2024-06-29T15:57:22.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5d44076420 0x7f5d44078930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d58159a40 0x7f5d58151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d58153580 0x7f5d58159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 --2- 172.21.15.81:0/3658141923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d40003a30 0x7f5d5814c3a0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3658141923 >> 172.21.15.81:0/3658141923 conn(0x7f5d5806ccc0 msgr2=0x7f5d5806f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:22.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3658141923 shutdown_connections 2024-06-29T15:57:22.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.629+0000 7f5d5fe38640 1 -- 172.21.15.81:0/3658141923 wait complete. 2024-06-29T15:57:22.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:22.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.754+0000 7fef51a3d640 1 -- 172.21.15.81:0/1322607846 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef4c0d0b90 msgr2=0x7fef4c0d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.754+0000 7fef51a3d640 1 --2- 172.21.15.81:0/1322607846 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef4c0d0b90 0x7fef4c0d1a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef40005fd0 tx=0x7fef4002a670 comp rx=0 tx=0).stop 2024-06-29T15:57:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.754+0000 7fef51a3d640 1 -- 172.21.15.81:0/1322607846 shutdown_connections 2024-06-29T15:57:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.754+0000 7fef51a3d640 1 --2- 172.21.15.81:0/1322607846 >> v2:172.21.15.167:3300/0 conn(0x7fef4c0b5250 0x7fef4c0d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.754+0000 7fef51a3d640 1 --2- 172.21.15.81:0/1322607846 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef4c0d0b90 0x7fef4c0d1a80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.754+0000 7fef51a3d640 1 -- 172.21.15.81:0/1322607846 >> 172.21.15.81:0/1322607846 conn(0x7fef4c09cf30 msgr2=0x7fef4c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.754+0000 7fef51a3d640 1 -- 172.21.15.81:0/1322607846 shutdown_connections 2024-06-29T15:57:22.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.754+0000 7fef51a3d640 1 -- 172.21.15.81:0/1322607846 wait complete. 2024-06-29T15:57:22.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.756+0000 7fef51a3d640 1 Processor -- start 2024-06-29T15:57:22.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.756+0000 7fef51a3d640 1 -- start start 2024-06-29T15:57:22.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef51a3d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef4c0b5250 0x7fef4c0a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef51a3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef4c0b9040 0x7fef4c15d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef51a3d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef4c09f580 0x7fef4c0b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef51a3d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c0d2680 con 0x7fef4c0b9040 2024-06-29T15:57:22.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef51a3d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c0d2500 con 0x7fef4c0b5250 2024-06-29T15:57:22.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef51a3d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fef4c0d2800 con 0x7fef4c09f580 2024-06-29T15:57:22.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef4a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef4c0b9040 0x7fef4c15d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef4a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef4c0b9040 0x7fef4c15d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59774/0 (socket says 172.21.15.81:59774) 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef4a7fc640 1 -- 172.21.15.81:0/2797849817 learned_addr learned my addr 172.21.15.81:0/2797849817 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef4affd640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef4c0b5250 0x7fef4c0a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.757+0000 7fef4b7fe640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef4c09f580 0x7fef4c0b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef4a7fc640 1 -- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef4c09f580 msgr2=0x7fef4c0b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef4a7fc640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef4c09f580 0x7fef4c0b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef4a7fc640 1 -- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef4c0b5250 msgr2=0x7fef4c0a14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef4a7fc640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef4c0b5250 0x7fef4c0a14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef4a7fc640 1 -- 172.21.15.81:0/2797849817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4c0a1a20 con 0x7fef4c0b9040 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef4a7fc640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef4c0b9040 0x7fef4c15d0b0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fef40034040 tx=0x7fef400038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef33fff640 1 -- 172.21.15.81:0/2797849817 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4003e030 con 0x7fef4c0b9040 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef51a3d640 1 -- 172.21.15.81:0/2797849817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4c09fb70 con 0x7fef4c0b9040 2024-06-29T15:57:22.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef33fff640 1 -- 172.21.15.81:0/2797849817 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef40003db0 con 0x7fef4c0b9040 2024-06-29T15:57:22.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef33fff640 1 -- 172.21.15.81:0/2797849817 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40035040 con 0x7fef4c0b9040 2024-06-29T15:57:22.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.758+0000 7fef51a3d640 1 -- 172.21.15.81:0/2797849817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef4c0a0130 con 0x7fef4c0b9040 2024-06-29T15:57:22.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.759+0000 7fef51a3d640 1 -- 172.21.15.81:0/2797849817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef4c0d2d90 con 0x7fef4c0b9040 2024-06-29T15:57:22.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.760+0000 7fef33fff640 1 -- 172.21.15.81:0/2797849817 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fef40041030 con 0x7fef4c0b9040 2024-06-29T15:57:22.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.760+0000 7fef33fff640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fef38076420 0x7fef38078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:22.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.761+0000 7fef33fff640 1 -- 172.21.15.81:0/2797849817 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5837+0+0 (secure 0 0 0) 0x7fef400b8120 con 0x7fef4c0b9040 2024-06-29T15:57:22.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.761+0000 7fef4affd640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fef38076420 0x7fef38078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:22.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.761+0000 7fef4affd640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fef38076420 0x7fef38078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fef3c001e20 tx=0x7fef3c009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:22.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.765+0000 7fef33fff640 1 -- 172.21.15.81:0/2797849817 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fef40081110 con 0x7fef4c0b9040 2024-06-29T15:57:22.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.945+0000 7fef51a3d640 1 -- 172.21.15.81:0/2797849817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef4c0d0480 con 0x7fef4c0b9040 2024-06-29T15:57:22.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.946+0000 7fef33fff640 1 -- 172.21.15.81:0/2797849817 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) ==== 74+0+16888 (secure 0 0 0) 0x7fef40085410 con 0x7fef4c0b9040 2024-06-29T15:57:22.947 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:22.948 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":212,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:21.637096+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:22.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 -- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fef38076420 msgr2=0x7fef38078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fef38076420 0x7fef38078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fef3c001e20 tx=0x7fef3c009040 comp rx=0 tx=0).stop 2024-06-29T15:57:22.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 -- 172.21.15.81:0/2797849817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef4c0b9040 msgr2=0x7fef4c15d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:22.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef4c0b9040 0x7fef4c15d0b0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fef40034040 tx=0x7fef400038a0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 -- 172.21.15.81:0/2797849817 shutdown_connections 2024-06-29T15:57:22.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fef38076420 0x7fef38078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef4c09f580 0x7fef4c0b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef4c0b9040 0x7fef4c15d0b0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 --2- 172.21.15.81:0/2797849817 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef4c0b5250 0x7fef4c0a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:22.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 -- 172.21.15.81:0/2797849817 >> 172.21.15.81:0/2797849817 conn(0x7fef4c09cf30 msgr2=0x7fef4c15e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:22.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 -- 172.21.15.81:0/2797849817 shutdown_connections 2024-06-29T15:57:22.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:22.948+0000 7fef51a3d640 1 -- 172.21.15.81:0/2797849817 wait complete. 2024-06-29T15:57:22.962 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:57:23.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.078+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2380599370 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6c00b5250 msgr2=0x7fc6c00d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:23.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.078+0000 7fc6c6da5640 1 --2- 172.21.15.81:0/2380599370 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6c00b5250 0x7fc6c00d1fc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8005fd0 tx=0x7fc6b802ab20 comp rx=0 tx=0).stop 2024-06-29T15:57:23.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.078+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2380599370 shutdown_connections 2024-06-29T15:57:23.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.078+0000 7fc6c6da5640 1 --2- 172.21.15.81:0/2380599370 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6c00b5250 0x7fc6c00d1fc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:23.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.078+0000 7fc6c6da5640 1 --2- 172.21.15.81:0/2380599370 >> v2:172.21.15.167:3300/0 conn(0x7fc6c00d0b90 0x7fc6c00d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:23.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.078+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2380599370 >> 172.21.15.81:0/2380599370 conn(0x7fc6c009cf30 msgr2=0x7fc6c00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:23.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.078+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2380599370 shutdown_connections 2024-06-29T15:57:23.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.078+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2380599370 wait complete. 2024-06-29T15:57:23.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.080+0000 7fc6c6da5640 1 Processor -- start 2024-06-29T15:57:23.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.080+0000 7fc6c6da5640 1 -- start start 2024-06-29T15:57:23.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.081+0000 7fc6c6da5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6c00b5250 0x7fc6c0154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.081+0000 7fc6c6da5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6c00b9040 0x7fc6c009e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.081+0000 7fc6c6da5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6c00cd430 0x7fc6c00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.081+0000 7fc6c6da5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c00d2500 con 0x7fc6c00b5250 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.081+0000 7fc6c6da5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c00b8590 con 0x7fc6c00cd430 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.081+0000 7fc6c6da5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc6c00d2680 con 0x7fc6c00b9040 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.081+0000 7fc6c4b41640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6c00b5250 0x7fc6c0154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.081+0000 7fc6c4b41640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6c00b5250 0x7fc6c0154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59778/0 (socket says 172.21.15.81:59778) 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.081+0000 7fc6c4b41640 1 -- 172.21.15.81:0/2751590464 learned_addr learned my addr 172.21.15.81:0/2751590464 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6b7fff640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6c00b9040 0x7fc6c009e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6c5342640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6c00cd430 0x7fc6c00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6c4b41640 1 -- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6c00b9040 msgr2=0x7fc6c009e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:23.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6c4b41640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6c00b9040 0x7fc6c009e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:23.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6c4b41640 1 -- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6c00cd430 msgr2=0x7fc6c00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:23.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6c4b41640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6c00cd430 0x7fc6c00cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:23.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6c4b41640 1 -- 172.21.15.81:0/2751590464 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6c00cdd80 con 0x7fc6c00b5250 2024-06-29T15:57:23.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6c4b41640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6c00b5250 0x7fc6c0154100 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00c3d0 tx=0x7fc6bc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:23.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6b57fa640 1 -- 172.21.15.81:0/2751590464 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc017070 con 0x7fc6c00b5250 2024-06-29T15:57:23.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6b57fa640 1 -- 172.21.15.81:0/2751590464 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6bc0053c0 con 0x7fc6c00b5250 2024-06-29T15:57:23.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6b57fa640 1 -- 172.21.15.81:0/2751590464 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc003850 con 0x7fc6c00b5250 2024-06-29T15:57:23.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2751590464 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c00a1140 con 0x7fc6c00b5250 2024-06-29T15:57:23.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.082+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2751590464 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c00a1700 con 0x7fc6c00b5250 2024-06-29T15:57:23.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.083+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2751590464 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c00d2bc0 con 0x7fc6c00b5250 2024-06-29T15:57:23.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.084+0000 7fc6b57fa640 1 -- 172.21.15.81:0/2751590464 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fc6bc01a070 con 0x7fc6c00b5250 2024-06-29T15:57:23.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.084+0000 7fc6b57fa640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6ac076420 0x7fc6ac078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:23.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.084+0000 7fc6b57fa640 1 -- 172.21.15.81:0/2751590464 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 5837+0+0 (secure 0 0 0) 0x7fc6bc096c10 con 0x7fc6c00b5250 2024-06-29T15:57:23.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.084+0000 7fc6b7fff640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6ac076420 0x7fc6ac078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:23.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.085+0000 7fc6b7fff640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6ac076420 0x7fc6ac078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc6c009dcc0 tx=0x7fc6b0005f90 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:23.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.089+0000 7fc6b57fa640 1 -- 172.21.15.81:0/2751590464 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc6c00cdd80 con 0x7fc6c00b5250 2024-06-29T15:57:23.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:23.312+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2751590464 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc6c006dbb0 con 0x7fc6c00b5250 2024-06-29T15:57:24.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.292+0000 7fc6b57fa640 1 -- 172.21.15.81:0/2751590464 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v111) ==== 118+0+0 (secure 0 0 0) 0x7fc6bc05fe20 con 0x7fc6c00b5250 2024-06-29T15:57:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6ac076420 msgr2=0x7fc6ac078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6ac076420 0x7fc6ac078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc6c009dcc0 tx=0x7fc6b0005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2751590464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6c00b5250 msgr2=0x7fc6c0154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6c00b5250 0x7fc6c0154100 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00c3d0 tx=0x7fc6bc00c890 comp rx=0 tx=0).stop 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2751590464 shutdown_connections 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6ac076420 0x7fc6ac078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6c00cd430 0x7fc6c00cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6c00b9040 0x7fc6c009e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 --2- 172.21.15.81:0/2751590464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6c00b5250 0x7fc6c0154100 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2751590464 >> 172.21.15.81:0/2751590464 conn(0x7fc6c009cf30 msgr2=0x7fc6c00b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2751590464 shutdown_connections 2024-06-29T15:57:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.295+0000 7fc6c6da5640 1 -- 172.21.15.81:0/2751590464 wait complete. 2024-06-29T15:57:24.309 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:57:24.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:57:24.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.426+0000 7f32b8826640 1 -- 172.21.15.81:0/672970834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32b415bdf0 msgr2=0x7f32b406ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:24.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.426+0000 7f32b8826640 1 --2- 172.21.15.81:0/672970834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32b415bdf0 0x7f32b406ed90 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f32b415c380 tx=0x7f32ac02a530 comp rx=0 tx=0).stop 2024-06-29T15:57:24.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.426+0000 7f32b8826640 1 -- 172.21.15.81:0/672970834 shutdown_connections 2024-06-29T15:57:24.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.426+0000 7f32b8826640 1 --2- 172.21.15.81:0/672970834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32b415bdf0 0x7f32b406ed90 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.426+0000 7f32b8826640 1 --2- 172.21.15.81:0/672970834 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32b40cd150 0x7f32b4072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.426+0000 7f32b8826640 1 -- 172.21.15.81:0/672970834 >> 172.21.15.81:0/672970834 conn(0x7f32b40b2920 msgr2=0x7f32b40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:24.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.426+0000 7f32b8826640 1 -- 172.21.15.81:0/672970834 shutdown_connections 2024-06-29T15:57:24.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.426+0000 7f32b8826640 1 -- 172.21.15.81:0/672970834 wait complete. 2024-06-29T15:57:24.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.428+0000 7f32b8826640 1 Processor -- start 2024-06-29T15:57:24.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.428+0000 7f32b8826640 1 -- start start 2024-06-29T15:57:24.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.429+0000 7f32b8826640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32b4073480 0x7f32b41ef190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:24.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.429+0000 7f32b8826640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32b40cd150 0x7f32b4158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:24.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.429+0000 7f32b8826640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32b415bdf0 0x7f32b40d0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:24.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.429+0000 7f32b8826640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f32b406f660 con 0x7f32b40cd150 2024-06-29T15:57:24.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.429+0000 7f32b8826640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f32b406f4e0 con 0x7f32b415bdf0 2024-06-29T15:57:24.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.429+0000 7f32b8826640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f32b406f7e0 con 0x7f32b4073480 2024-06-29T15:57:24.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b1d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32b40cd150 0x7f32b4158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:24.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b1d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32b40cd150 0x7f32b4158900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59794/0 (socket says 172.21.15.81:59794) 2024-06-29T15:57:24.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b1d9b640 1 -- 172.21.15.81:0/2779355771 learned_addr learned my addr 172.21.15.81:0/2779355771 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:24.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b2d9d640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32b415bdf0 0x7f32b40d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:24.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b259c640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32b4073480 0x7f32b41ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:24.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b1d9b640 1 -- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32b4073480 msgr2=0x7f32b41ef190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b1d9b640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32b4073480 0x7f32b41ef190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b1d9b640 1 -- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32b415bdf0 msgr2=0x7f32b40d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b1d9b640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32b415bdf0 0x7f32b40d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b1d9b640 1 -- 172.21.15.81:0/2779355771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b41f5530 con 0x7f32b40cd150 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b1d9b640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32b40cd150 0x7f32b4158900 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f32a800c400 tx=0x7f32a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32a2ffd640 1 -- 172.21.15.81:0/2779355771 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8017070 con 0x7f32b40cd150 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32a2ffd640 1 -- 172.21.15.81:0/2779355771 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32a80053c0 con 0x7f32b40cd150 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b8826640 1 -- 172.21.15.81:0/2779355771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b41f5810 con 0x7f32b40cd150 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32a2ffd640 1 -- 172.21.15.81:0/2779355771 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a8003830 con 0x7f32b40cd150 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.430+0000 7f32b8826640 1 -- 172.21.15.81:0/2779355771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b41f5dd0 con 0x7f32b40cd150 2024-06-29T15:57:24.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.431+0000 7f32b8826640 1 -- 172.21.15.81:0/2779355771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b4158f80 con 0x7f32b40cd150 2024-06-29T15:57:24.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.432+0000 7f32a2ffd640 1 -- 172.21.15.81:0/2779355771 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f32a801a070 con 0x7f32b40cd150 2024-06-29T15:57:24.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.433+0000 7f32a2ffd640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f32940763f0 0x7f3294078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:24.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.433+0000 7f32a2ffd640 1 -- 172.21.15.81:0/2779355771 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7f32a8096cf0 con 0x7f32b40cd150 2024-06-29T15:57:24.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.433+0000 7f32b259c640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f32940763f0 0x7f3294078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:24.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.433+0000 7f32b259c640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f32940763f0 0x7f3294078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32a40092f0 tx=0x7f32a4009260 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:24.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.437+0000 7f32a2ffd640 1 -- 172.21.15.81:0/2779355771 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f32a809b070 con 0x7f32b40cd150 2024-06-29T15:57:24.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:24.654+0000 7f32b8826640 1 -- 172.21.15.81:0/2779355771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f32b406fd70 con 0x7f32b40cd150 2024-06-29T15:57:25.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.302+0000 7f32a2ffd640 1 -- 172.21.15.81:0/2779355771 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v112) ==== 122+0+0 (secure 0 0 0) 0x7f32b4073890 con 0x7f32b40cd150 2024-06-29T15:57:25.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 -- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f32940763f0 msgr2=0x7f3294078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f32940763f0 0x7f3294078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32a40092f0 tx=0x7f32a4009260 comp rx=0 tx=0).stop 2024-06-29T15:57:25.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 -- 172.21.15.81:0/2779355771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32b40cd150 msgr2=0x7f32b4158900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32b40cd150 0x7f32b4158900 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f32a800c400 tx=0x7f32a800c8c0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 -- 172.21.15.81:0/2779355771 shutdown_connections 2024-06-29T15:57:25.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f32940763f0 0x7f3294078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f32b415bdf0 0x7f32b40d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32b40cd150 0x7f32b4158900 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 --2- 172.21.15.81:0/2779355771 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f32b4073480 0x7f32b41ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 -- 172.21.15.81:0/2779355771 >> 172.21.15.81:0/2779355771 conn(0x7f32b40b2920 msgr2=0x7f32b40b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:25.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 -- 172.21.15.81:0/2779355771 shutdown_connections 2024-06-29T15:57:25.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.305+0000 7f32b8826640 1 -- 172.21.15.81:0/2779355771 wait complete. 2024-06-29T15:57:25.318 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:57:25.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:25.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.432+0000 7f03996ac640 1 -- 172.21.15.81:0/308183093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0394073740 msgr2=0x7f039415e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.432+0000 7f03996ac640 1 --2- 172.21.15.81:0/308183093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0394073740 0x7f039415e960 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f038c005fd0 tx=0x7f038c02aa00 comp rx=0 tx=0).stop 2024-06-29T15:57:25.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.432+0000 7f03996ac640 1 -- 172.21.15.81:0/308183093 shutdown_connections 2024-06-29T15:57:25.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.432+0000 7f03996ac640 1 --2- 172.21.15.81:0/308183093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0394073740 0x7f039415e960 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.432+0000 7f03996ac640 1 --2- 172.21.15.81:0/308183093 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03941412f0 0x7f03940cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.432+0000 7f03996ac640 1 -- 172.21.15.81:0/308183093 >> 172.21.15.81:0/308183093 conn(0x7f039406ccc0 msgr2=0x7f03940b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:25.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.432+0000 7f03996ac640 1 -- 172.21.15.81:0/308183093 shutdown_connections 2024-06-29T15:57:25.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.432+0000 7f03996ac640 1 -- 172.21.15.81:0/308183093 wait complete. 2024-06-29T15:57:25.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.434+0000 7f03996ac640 1 Processor -- start 2024-06-29T15:57:25.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.434+0000 7f03996ac640 1 -- start start 2024-06-29T15:57:25.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.435+0000 7f03996ac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03940cf4d0 0x7f0394158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.435+0000 7f03996ac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03941412f0 0x7f03940a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03996ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f039413e350 0x7f0394154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03996ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f03940b3210 con 0x7f039413e350 2024-06-29T15:57:25.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03996ac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f039415f270 con 0x7f03940cf4d0 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03996ac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f03940b3390 con 0x7f03941412f0 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03937fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f039413e350 0x7f0394154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03937fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f039413e350 0x7f0394154e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59804/0 (socket says 172.21.15.81:59804) 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03937fe640 1 -- 172.21.15.81:0/3704997002 learned_addr learned my addr 172.21.15.81:0/3704997002 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f0392ffd640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03940cf4d0 0x7f0394158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f038a7fc640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03941412f0 0x7f03940a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03937fe640 1 -- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03941412f0 msgr2=0x7f03940a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03937fe640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03941412f0 0x7f03940a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03937fe640 1 -- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03940cf4d0 msgr2=0x7f0394158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03937fe640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03940cf4d0 0x7f0394158840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03937fe640 1 -- 172.21.15.81:0/3704997002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0394156540 con 0x7f039413e350 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03937fe640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f039413e350 0x7f0394154e60 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f038c009d90 tx=0x7f038c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:25.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f038bfff640 1 -- 172.21.15.81:0/3704997002 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c033050 con 0x7f039413e350 2024-06-29T15:57:25.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f038bfff640 1 -- 172.21.15.81:0/3704997002 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f038c004450 con 0x7f039413e350 2024-06-29T15:57:25.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03996ac640 1 -- 172.21.15.81:0/3704997002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0394153dc0 con 0x7f039413e350 2024-06-29T15:57:25.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f038bfff640 1 -- 172.21.15.81:0/3704997002 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c004860 con 0x7f039413e350 2024-06-29T15:57:25.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.436+0000 7f03996ac640 1 -- 172.21.15.81:0/3704997002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0394154380 con 0x7f039413e350 2024-06-29T15:57:25.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.437+0000 7f03996ac640 1 -- 172.21.15.81:0/3704997002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03940b38b0 con 0x7f039413e350 2024-06-29T15:57:25.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.438+0000 7f038bfff640 1 -- 172.21.15.81:0/3704997002 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f038c036070 con 0x7f039413e350 2024-06-29T15:57:25.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.439+0000 7f038bfff640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f03740763f0 0x7f0374078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.439+0000 7f038bfff640 1 -- 172.21.15.81:0/3704997002 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7f038c0b8dd0 con 0x7f039413e350 2024-06-29T15:57:25.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.439+0000 7f0392ffd640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f03740763f0 0x7f0374078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.440+0000 7f0392ffd640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f03740763f0 0x7f0374078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0384005fd0 tx=0x7f0384005850 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:25.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.443+0000 7f038bfff640 1 -- 172.21.15.81:0/3704997002 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f038c0824a0 con 0x7f039413e350 2024-06-29T15:57:25.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.661+0000 7f03996ac640 1 -- 172.21.15.81:0/3704997002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f039406e870 con 0x7f039413e350 2024-06-29T15:57:25.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.661+0000 7f038bfff640 1 -- 172.21.15.81:0/3704997002 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5240 (secure 0 0 0) 0x7f038c081da0 con 0x7f039413e350 2024-06-29T15:57:25.663 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:25.663 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":112,"btime":"2024-06-29T15:57:25:299142+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10114,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.637042+0000","modified":"2024-06-29T15:57:24.656503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11226":{"gid":11226,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/739691411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":739691411},{"type":"v1","addr":"172.21.15.81:6833","nonce":739691411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11226,"qdb_cluster":[11226]},"id":13}]} 2024-06-29T15:57:25.663 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 112 2024-06-29T15:57:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 -- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f03740763f0 msgr2=0x7f0374078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f03740763f0 0x7f0374078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0384005fd0 tx=0x7f0384005850 comp rx=0 tx=0).stop 2024-06-29T15:57:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 -- 172.21.15.81:0/3704997002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f039413e350 msgr2=0x7f0394154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f039413e350 0x7f0394154e60 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f038c009d90 tx=0x7f038c003f10 comp rx=0 tx=0).stop 2024-06-29T15:57:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 -- 172.21.15.81:0/3704997002 shutdown_connections 2024-06-29T15:57:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f03740763f0 0x7f0374078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f039413e350 0x7f0394154e60 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03941412f0 0x7f03940a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 --2- 172.21.15.81:0/3704997002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03940cf4d0 0x7f0394158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 -- 172.21.15.81:0/3704997002 >> 172.21.15.81:0/3704997002 conn(0x7f039406ccc0 msgr2=0x7f039415e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:25.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 -- 172.21.15.81:0/3704997002 shutdown_connections 2024-06-29T15:57:25.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.663+0000 7f03996ac640 1 -- 172.21.15.81:0/3704997002 wait complete. 2024-06-29T15:57:25.676 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:25.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.787+0000 7faf5e5d9640 1 -- 172.21.15.81:0/1335861876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf580b3200 msgr2=0x7faf580cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.787+0000 7faf5e5d9640 1 --2- 172.21.15.81:0/1335861876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf580b3200 0x7faf580cc7b0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7faf48005fd0 tx=0x7faf4802a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.787+0000 7faf5e5d9640 1 -- 172.21.15.81:0/1335861876 shutdown_connections 2024-06-29T15:57:25.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.787+0000 7faf5e5d9640 1 --2- 172.21.15.81:0/1335861876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf580d0940 0x7faf580b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.787+0000 7faf5e5d9640 1 --2- 172.21.15.81:0/1335861876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf580b3200 0x7faf580cc7b0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.787+0000 7faf5e5d9640 1 -- 172.21.15.81:0/1335861876 >> 172.21.15.81:0/1335861876 conn(0x7faf580b3680 msgr2=0x7faf580b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:25.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.787+0000 7faf5e5d9640 1 -- 172.21.15.81:0/1335861876 shutdown_connections 2024-06-29T15:57:25.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.787+0000 7faf5e5d9640 1 -- 172.21.15.81:0/1335861876 wait complete. 2024-06-29T15:57:25.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.789+0000 7faf5e5d9640 1 Processor -- start 2024-06-29T15:57:25.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.789+0000 7faf5e5d9640 1 -- start start 2024-06-29T15:57:25.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.790+0000 7faf5e5d9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf580b3200 0x7faf580c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.790+0000 7faf5e5d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf580cccf0 0x7faf58155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.790+0000 7faf5e5d9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5809f430 0x7faf580cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.790+0000 7faf5e5d9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf580b5c50 con 0x7faf580cccf0 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.790+0000 7faf5e5d9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faf580b5ad0 con 0x7faf5809f430 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.790+0000 7faf5e5d9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faf580b5dd0 con 0x7faf580b3200 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf580cccf0 0x7faf58155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf577fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf580cccf0 0x7faf58155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59816/0 (socket says 172.21.15.81:59816) 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf577fe640 1 -- 172.21.15.81:0/3627979474 learned_addr learned my addr 172.21.15.81:0/3627979474 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf57fff640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf580b3200 0x7faf580c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf5cb76640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5809f430 0x7faf580cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf577fe640 1 -- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf580b3200 msgr2=0x7faf580c84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf577fe640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf580b3200 0x7faf580c84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf577fe640 1 -- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5809f430 msgr2=0x7faf580cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf577fe640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5809f430 0x7faf580cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf577fe640 1 -- 172.21.15.81:0/3627979474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf580c9790 con 0x7faf580cccf0 2024-06-29T15:57:25.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf57fff640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf580b3200 0x7faf580c84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:25.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf5cb76640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5809f430 0x7faf580cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:25.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf577fe640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf580cccf0 0x7faf58155190 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c3c0 tx=0x7faf4c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:25.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf54ff9640 1 -- 172.21.15.81:0/3627979474 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c017070 con 0x7faf580cccf0 2024-06-29T15:57:25.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf54ff9640 1 -- 172.21.15.81:0/3627979474 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf4c0035f0 con 0x7faf580cccf0 2024-06-29T15:57:25.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf5e5d9640 1 -- 172.21.15.81:0/3627979474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf580c9a70 con 0x7faf580cccf0 2024-06-29T15:57:25.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf54ff9640 1 -- 172.21.15.81:0/3627979474 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c003a20 con 0x7faf580cccf0 2024-06-29T15:57:25.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.791+0000 7faf5e5d9640 1 -- 172.21.15.81:0/3627979474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf580ca030 con 0x7faf580cccf0 2024-06-29T15:57:25.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.792+0000 7faf5e5d9640 1 -- 172.21.15.81:0/3627979474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf5809fc50 con 0x7faf580cccf0 2024-06-29T15:57:25.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.793+0000 7faf54ff9640 1 -- 172.21.15.81:0/3627979474 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7faf4c01a070 con 0x7faf580cccf0 2024-06-29T15:57:25.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.793+0000 7faf54ff9640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7faf440763f0 0x7faf44078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:25.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.794+0000 7faf54ff9640 1 -- 172.21.15.81:0/3627979474 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7faf4c096c60 con 0x7faf580cccf0 2024-06-29T15:57:25.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.794+0000 7faf57fff640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7faf440763f0 0x7faf44078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:25.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.794+0000 7faf57fff640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7faf440763f0 0x7faf44078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faf48033000 tx=0x7faf48004730 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:25.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.798+0000 7faf54ff9640 1 -- 172.21.15.81:0/3627979474 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faf5809fc50 con 0x7faf580cccf0 2024-06-29T15:57:25.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.977+0000 7faf5e5d9640 1 -- 172.21.15.81:0/3627979474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faf58071300 con 0x7faf580cccf0 2024-06-29T15:57:25.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.978+0000 7faf54ff9640 1 -- 172.21.15.81:0/3627979474 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) ==== 74+0+16875 (secure 0 0 0) 0x7faf58071300 con 0x7faf580cccf0 2024-06-29T15:57:25.980 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:25.980 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":213,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:23.285939+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:25.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.980+0000 7faf5e5d9640 1 -- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7faf440763f0 msgr2=0x7faf44078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.980+0000 7faf5e5d9640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7faf440763f0 0x7faf44078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faf48033000 tx=0x7faf48004730 comp rx=0 tx=0).stop 2024-06-29T15:57:25.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.980+0000 7faf5e5d9640 1 -- 172.21.15.81:0/3627979474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf580cccf0 msgr2=0x7faf58155190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:25.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.980+0000 7faf5e5d9640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf580cccf0 0x7faf58155190 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00c3c0 tx=0x7faf4c00c880 comp rx=0 tx=0).stop 2024-06-29T15:57:25.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.981+0000 7faf5e5d9640 1 -- 172.21.15.81:0/3627979474 shutdown_connections 2024-06-29T15:57:25.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.981+0000 7faf5e5d9640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7faf440763f0 0x7faf44078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.981+0000 7faf5e5d9640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5809f430 0x7faf580cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.981+0000 7faf5e5d9640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf580cccf0 0x7faf58155190 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.981+0000 7faf5e5d9640 1 --2- 172.21.15.81:0/3627979474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf580b3200 0x7faf580c84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:25.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.981+0000 7faf5e5d9640 1 -- 172.21.15.81:0/3627979474 >> 172.21.15.81:0/3627979474 conn(0x7faf580b3680 msgr2=0x7faf58070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:25.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.981+0000 7faf5e5d9640 1 -- 172.21.15.81:0/3627979474 shutdown_connections 2024-06-29T15:57:25.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:25.981+0000 7faf5e5d9640 1 -- 172.21.15.81:0/3627979474 wait complete. 2024-06-29T15:57:25.993 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:57:25.994 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:26.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.105+0000 7f46b04ef640 1 -- 172.21.15.81:0/1133978261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a80d0b90 msgr2=0x7f46a80d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.105+0000 7f46b04ef640 1 --2- 172.21.15.81:0/1133978261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a80d0b90 0x7f46a80d1a80 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f46a808e330 tx=0x7f469c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.105+0000 7f46b04ef640 1 -- 172.21.15.81:0/1133978261 shutdown_connections 2024-06-29T15:57:26.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.105+0000 7f46b04ef640 1 --2- 172.21.15.81:0/1133978261 >> v2:172.21.15.167:3300/0 conn(0x7f46a80b5250 0x7f46a80d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.105+0000 7f46b04ef640 1 --2- 172.21.15.81:0/1133978261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a80d0b90 0x7f46a80d1a80 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.105+0000 7f46b04ef640 1 -- 172.21.15.81:0/1133978261 >> 172.21.15.81:0/1133978261 conn(0x7f46a809cf30 msgr2=0x7f46a80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:26.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.105+0000 7f46b04ef640 1 -- 172.21.15.81:0/1133978261 shutdown_connections 2024-06-29T15:57:26.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.105+0000 7f46b04ef640 1 -- 172.21.15.81:0/1133978261 wait complete. 2024-06-29T15:57:26.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.107+0000 7f46b04ef640 1 Processor -- start 2024-06-29T15:57:26.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.107+0000 7f46b04ef640 1 -- start start 2024-06-29T15:57:26.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.108+0000 7f46b04ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a80b5250 0x7f46a815c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.108+0000 7f46b04ef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a80b9040 0x7f46a815ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.108+0000 7f46b04ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a81f5b70 0x7f46a81f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.108+0000 7f46b04ef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46a80d2500 con 0x7f46a80b5250 2024-06-29T15:57:26.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.108+0000 7f46b04ef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f46a80b8530 con 0x7f46a80b9040 2024-06-29T15:57:26.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.108+0000 7f46b04ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f46a80d2680 con 0x7f46a81f5b70 2024-06-29T15:57:26.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.108+0000 7f46ae28b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a80b5250 0x7f46a815c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46ae28b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a80b5250 0x7f46a815c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59834/0 (socket says 172.21.15.81:59834) 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46ae28b640 1 -- 172.21.15.81:0/2862362321 learned_addr learned my addr 172.21.15.81:0/2862362321 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46aea8c640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a81f5b70 0x7f46a81f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46ada8a640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a80b9040 0x7f46a815ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46ae28b640 1 -- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a81f5b70 msgr2=0x7f46a81f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46ae28b640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a81f5b70 0x7f46a81f5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46ae28b640 1 -- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a80b9040 msgr2=0x7f46a815ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46ae28b640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a80b9040 0x7f46a815ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46ae28b640 1 -- 172.21.15.81:0/2862362321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a815b180 con 0x7f46a80b5250 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46aea8c640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a81f5b70 0x7f46a81f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46ae28b640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a80b5250 0x7f46a815c230 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f469800c3c0 tx=0x7f469800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46a6ffd640 1 -- 172.21.15.81:0/2862362321 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698017070 con 0x7f46a80b5250 2024-06-29T15:57:26.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46a6ffd640 1 -- 172.21.15.81:0/2862362321 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46980035f0 con 0x7f46a80b5250 2024-06-29T15:57:26.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46a6ffd640 1 -- 172.21.15.81:0/2862362321 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698003a80 con 0x7f46a80b5250 2024-06-29T15:57:26.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46b04ef640 1 -- 172.21.15.81:0/2862362321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a81f58e0 con 0x7f46a80b5250 2024-06-29T15:57:26.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.109+0000 7f46b04ef640 1 -- 172.21.15.81:0/2862362321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a815b750 con 0x7f46a80b5250 2024-06-29T15:57:26.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.110+0000 7f46b04ef640 1 -- 172.21.15.81:0/2862362321 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a80d2c10 con 0x7f46a80b5250 2024-06-29T15:57:26.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.111+0000 7f46a6ffd640 1 -- 172.21.15.81:0/2862362321 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f469801a070 con 0x7f46a80b5250 2024-06-29T15:57:26.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.112+0000 7f46a6ffd640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4694076420 0x7f4694078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.112+0000 7f46a6ffd640 1 -- 172.21.15.81:0/2862362321 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7f4698096df0 con 0x7f46a80b5250 2024-06-29T15:57:26.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.112+0000 7f46ada8a640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4694076420 0x7f4694078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.112+0000 7f46ada8a640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4694076420 0x7f4694078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f469c034000 tx=0x7f469c004730 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.116+0000 7f46a6ffd640 1 -- 172.21.15.81:0/2862362321 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4698060530 con 0x7f46a80b5250 2024-06-29T15:57:26.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.293+0000 7f46b04ef640 1 -- 172.21.15.81:0/2862362321 --> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f46a806deb0 con 0x7f4694076420 2024-06-29T15:57:26.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.298+0000 7f46a6ffd640 1 -- 172.21.15.81:0/2862362321 <== mgr.8666 v2:172.21.15.167:6832/2018755589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252496 (secure 0 0 0) 0x7f46a806deb0 con 0x7f4694076420 2024-06-29T15:57:26.302 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:57:26.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 -- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4694076420 msgr2=0x7f4694078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4694076420 0x7f4694078930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f469c034000 tx=0x7f469c004730 comp rx=0 tx=0).stop 2024-06-29T15:57:26.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 -- 172.21.15.81:0/2862362321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a80b5250 msgr2=0x7f46a815c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a80b5250 0x7f46a815c230 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f469800c3c0 tx=0x7f469800c880 comp rx=0 tx=0).stop 2024-06-29T15:57:26.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 -- 172.21.15.81:0/2862362321 shutdown_connections 2024-06-29T15:57:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f4694076420 0x7f4694078930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a81f5b70 0x7f46a81f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a80b9040 0x7f46a815ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 --2- 172.21.15.81:0/2862362321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a80b5250 0x7f46a815c230 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 -- 172.21.15.81:0/2862362321 >> 172.21.15.81:0/2862362321 conn(0x7f46a809cf30 msgr2=0x7f46a815f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 -- 172.21.15.81:0/2862362321 shutdown_connections 2024-06-29T15:57:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.301+0000 7f46b04ef640 1 -- 172.21.15.81:0/2862362321 wait complete. 2024-06-29T15:57:26.314 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:57:25.869761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":398,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":258,"ondisk_log_size":258,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":408988,"kb_used_data":22096,"kb_used_omap":12,"kb_used_meta":386739,"kb_avail":754565732,"statfs":{"total":773094113280,"available":772675309568,"internally_reserved":0,"allocated":22626304,"data_stored":11628580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":396021298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.607342"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301243+0000","last_change":"2024-06-29T15:57:22.301019+0000","last_active":"2024-06-29T15:57:23.301243+0000","last_peered":"2024-06-29T15:57:23.301243+0000","last_clean":"2024-06-29T15:57:23.301243+0000","last_became_active":"2024-06-29T15:57:22.300506+0000","last_became_peered":"2024-06-29T15:57:22.300506+0000","last_unstale":"2024-06-29T15:57:23.301243+0000","last_undegraded":"2024-06-29T15:57:23.301243+0000","last_fullsized":"2024-06-29T15:57:23.301243+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296998+0000","last_change":"2024-06-29T15:57:22.297971+0000","last_active":"2024-06-29T15:57:23.296998+0000","last_peered":"2024-06-29T15:57:23.296998+0000","last_clean":"2024-06-29T15:57:23.296998+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:23.296998+0000","last_undegraded":"2024-06-29T15:57:23.296998+0000","last_fullsized":"2024-06-29T15:57:23.296998+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301936+0000","last_change":"2024-06-29T15:57:22.300745+0000","last_active":"2024-06-29T15:57:23.301936+0000","last_peered":"2024-06-29T15:57:23.301936+0000","last_clean":"2024-06-29T15:57:23.301936+0000","last_became_active":"2024-06-29T15:57:22.300414+0000","last_became_peered":"2024-06-29T15:57:22.300414+0000","last_unstale":"2024-06-29T15:57:23.301936+0000","last_undegraded":"2024-06-29T15:57:23.301936+0000","last_fullsized":"2024-06-29T15:57:23.301936+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301238+0000","last_change":"2024-06-29T15:57:22.298374+0000","last_active":"2024-06-29T15:57:23.301238+0000","last_peered":"2024-06-29T15:57:23.301238+0000","last_clean":"2024-06-29T15:57:23.301238+0000","last_became_active":"2024-06-29T15:57:22.297962+0000","last_became_peered":"2024-06-29T15:57:22.297962+0000","last_unstale":"2024-06-29T15:57:23.301238+0000","last_undegraded":"2024-06-29T15:57:23.301238+0000","last_fullsized":"2024-06-29T15:57:23.301238+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:13.834244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.299314+0000","last_change":"2024-06-29T15:57:22.290909+0000","last_active":"2024-06-29T15:57:23.299314+0000","last_peered":"2024-06-29T15:57:23.299314+0000","last_clean":"2024-06-29T15:57:23.299314+0000","last_became_active":"2024-06-29T15:57:22.290643+0000","last_became_peered":"2024-06-29T15:57:22.290643+0000","last_unstale":"2024-06-29T15:57:23.299314+0000","last_undegraded":"2024-06-29T15:57:23.299314+0000","last_fullsized":"2024-06-29T15:57:23.299314+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298581+0000","last_change":"2024-06-29T15:57:22.288041+0000","last_active":"2024-06-29T15:57:23.298581+0000","last_peered":"2024-06-29T15:57:23.298581+0000","last_clean":"2024-06-29T15:57:23.298581+0000","last_became_active":"2024-06-29T15:57:22.287729+0000","last_became_peered":"2024-06-29T15:57:22.287729+0000","last_unstale":"2024-06-29T15:57:23.298581+0000","last_undegraded":"2024-06-29T15:57:23.298581+0000","last_fullsized":"2024-06-29T15:57:23.298581+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300700+0000","last_change":"2024-06-29T15:57:22.304052+0000","last_active":"2024-06-29T15:57:23.300700+0000","last_peered":"2024-06-29T15:57:23.300700+0000","last_clean":"2024-06-29T15:57:23.300700+0000","last_became_active":"2024-06-29T15:57:22.303598+0000","last_became_peered":"2024-06-29T15:57:22.303598+0000","last_unstale":"2024-06-29T15:57:23.300700+0000","last_undegraded":"2024-06-29T15:57:23.300700+0000","last_fullsized":"2024-06-29T15:57:23.300700+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298325+0000","last_change":"2024-06-29T15:57:22.297617+0000","last_active":"2024-06-29T15:57:23.298325+0000","last_peered":"2024-06-29T15:57:23.298325+0000","last_clean":"2024-06-29T15:57:23.298325+0000","last_became_active":"2024-06-29T15:57:22.297256+0000","last_became_peered":"2024-06-29T15:57:22.297256+0000","last_unstale":"2024-06-29T15:57:23.298325+0000","last_undegraded":"2024-06-29T15:57:23.298325+0000","last_fullsized":"2024-06-29T15:57:23.298325+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.297058+0000","last_change":"2024-06-29T15:57:22.291096+0000","last_active":"2024-06-29T15:57:23.297058+0000","last_peered":"2024-06-29T15:57:23.297058+0000","last_clean":"2024-06-29T15:57:23.297058+0000","last_became_active":"2024-06-29T15:57:22.290428+0000","last_became_peered":"2024-06-29T15:57:22.290428+0000","last_unstale":"2024-06-29T15:57:23.297058+0000","last_undegraded":"2024-06-29T15:57:23.297058+0000","last_fullsized":"2024-06-29T15:57:23.297058+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:20.813793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302381+0000","last_change":"2024-06-29T15:57:22.296441+0000","last_active":"2024-06-29T15:57:23.302381+0000","last_peered":"2024-06-29T15:57:23.302381+0000","last_clean":"2024-06-29T15:57:23.302381+0000","last_became_active":"2024-06-29T15:57:22.296146+0000","last_became_peered":"2024-06-29T15:57:22.296146+0000","last_unstale":"2024-06-29T15:57:23.302381+0000","last_undegraded":"2024-06-29T15:57:23.302381+0000","last_fullsized":"2024-06-29T15:57:23.302381+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:54.360521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302086+0000","last_change":"2024-06-29T15:57:22.298781+0000","last_active":"2024-06-29T15:57:23.302086+0000","last_peered":"2024-06-29T15:57:23.302086+0000","last_clean":"2024-06-29T15:57:23.302086+0000","last_became_active":"2024-06-29T15:57:22.298300+0000","last_became_peered":"2024-06-29T15:57:22.298300+0000","last_unstale":"2024-06-29T15:57:23.302086+0000","last_undegraded":"2024-06-29T15:57:23.302086+0000","last_fullsized":"2024-06-29T15:57:23.302086+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295597+0000","last_change":"2024-06-29T15:57:22.298612+0000","last_active":"2024-06-29T15:57:23.295597+0000","last_peered":"2024-06-29T15:57:23.295597+0000","last_clean":"2024-06-29T15:57:23.295597+0000","last_became_active":"2024-06-29T15:57:22.298288+0000","last_became_peered":"2024-06-29T15:57:22.298288+0000","last_unstale":"2024-06-29T15:57:23.295597+0000","last_undegraded":"2024-06-29T15:57:23.295597+0000","last_fullsized":"2024-06-29T15:57:23.295597+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295805+0000","last_change":"2024-06-29T15:57:22.295681+0000","last_active":"2024-06-29T15:57:23.295805+0000","last_peered":"2024-06-29T15:57:23.295805+0000","last_clean":"2024-06-29T15:57:23.295805+0000","last_became_active":"2024-06-29T15:57:22.295357+0000","last_became_peered":"2024-06-29T15:57:22.295357+0000","last_unstale":"2024-06-29T15:57:23.295805+0000","last_undegraded":"2024-06-29T15:57:23.295805+0000","last_fullsized":"2024-06-29T15:57:23.295805+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300606+0000","last_change":"2024-06-29T15:57:22.303092+0000","last_active":"2024-06-29T15:57:23.300606+0000","last_peered":"2024-06-29T15:57:23.300606+0000","last_clean":"2024-06-29T15:57:23.300606+0000","last_became_active":"2024-06-29T15:57:22.302807+0000","last_became_peered":"2024-06-29T15:57:22.302807+0000","last_unstale":"2024-06-29T15:57:23.300606+0000","last_undegraded":"2024-06-29T15:57:23.300606+0000","last_fullsized":"2024-06-29T15:57:23.300606+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300861+0000","last_change":"2024-06-29T15:57:22.302418+0000","last_active":"2024-06-29T15:57:23.300861+0000","last_peered":"2024-06-29T15:57:23.300861+0000","last_clean":"2024-06-29T15:57:23.300861+0000","last_became_active":"2024-06-29T15:57:22.302075+0000","last_became_peered":"2024-06-29T15:57:22.302075+0000","last_unstale":"2024-06-29T15:57:23.300861+0000","last_undegraded":"2024-06-29T15:57:23.300861+0000","last_fullsized":"2024-06-29T15:57:23.300861+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302094+0000","last_change":"2024-06-29T15:57:22.290047+0000","last_active":"2024-06-29T15:57:23.302094+0000","last_peered":"2024-06-29T15:57:23.302094+0000","last_clean":"2024-06-29T15:57:23.302094+0000","last_became_active":"2024-06-29T15:57:22.289742+0000","last_became_peered":"2024-06-29T15:57:22.289742+0000","last_unstale":"2024-06-29T15:57:23.302094+0000","last_undegraded":"2024-06-29T15:57:23.302094+0000","last_fullsized":"2024-06-29T15:57:23.302094+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:03:46.271206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301400+0000","last_change":"2024-06-29T15:57:22.300787+0000","last_active":"2024-06-29T15:57:23.301400+0000","last_peered":"2024-06-29T15:57:23.301400+0000","last_clean":"2024-06-29T15:57:23.301400+0000","last_became_active":"2024-06-29T15:57:22.300112+0000","last_became_peered":"2024-06-29T15:57:22.300112+0000","last_unstale":"2024-06-29T15:57:23.301400+0000","last_undegraded":"2024-06-29T15:57:23.301400+0000","last_fullsized":"2024-06-29T15:57:23.301400+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:30.249483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.303141+0000","last_change":"2024-06-29T15:57:22.289633+0000","last_active":"2024-06-29T15:57:23.303141+0000","last_peered":"2024-06-29T15:57:23.303141+0000","last_clean":"2024-06-29T15:57:23.303141+0000","last_became_active":"2024-06-29T15:57:22.289349+0000","last_became_peered":"2024-06-29T15:57:22.289349+0000","last_unstale":"2024-06-29T15:57:23.303141+0000","last_undegraded":"2024-06-29T15:57:23.303141+0000","last_fullsized":"2024-06-29T15:57:23.303141+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.873806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302730+0000","last_change":"2024-06-29T15:57:22.306593+0000","last_active":"2024-06-29T15:57:23.302730+0000","last_peered":"2024-06-29T15:57:23.302730+0000","last_clean":"2024-06-29T15:57:23.302730+0000","last_became_active":"2024-06-29T15:57:22.296653+0000","last_became_peered":"2024-06-29T15:57:22.296653+0000","last_unstale":"2024-06-29T15:57:23.302730+0000","last_undegraded":"2024-06-29T15:57:23.302730+0000","last_fullsized":"2024-06-29T15:57:23.302730+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:41.738155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298929+0000","last_change":"2024-06-29T15:57:22.290186+0000","last_active":"2024-06-29T15:57:23.298929+0000","last_peered":"2024-06-29T15:57:23.298929+0000","last_clean":"2024-06-29T15:57:23.298929+0000","last_became_active":"2024-06-29T15:57:22.289908+0000","last_became_peered":"2024-06-29T15:57:22.289908+0000","last_unstale":"2024-06-29T15:57:23.298929+0000","last_undegraded":"2024-06-29T15:57:23.298929+0000","last_fullsized":"2024-06-29T15:57:23.298929+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300867+0000","last_change":"2024-06-29T15:57:22.307230+0000","last_active":"2024-06-29T15:57:23.300867+0000","last_peered":"2024-06-29T15:57:23.300867+0000","last_clean":"2024-06-29T15:57:23.300867+0000","last_became_active":"2024-06-29T15:57:22.306920+0000","last_became_peered":"2024-06-29T15:57:22.306920+0000","last_unstale":"2024-06-29T15:57:23.300867+0000","last_undegraded":"2024-06-29T15:57:23.300867+0000","last_fullsized":"2024-06-29T15:57:23.300867+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:22.598647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301465+0000","last_change":"2024-06-29T15:57:22.307686+0000","last_active":"2024-06-29T15:57:23.301465+0000","last_peered":"2024-06-29T15:57:23.301465+0000","last_clean":"2024-06-29T15:57:23.301465+0000","last_became_active":"2024-06-29T15:57:22.307324+0000","last_became_peered":"2024-06-29T15:57:22.307324+0000","last_unstale":"2024-06-29T15:57:23.301465+0000","last_undegraded":"2024-06-29T15:57:23.301465+0000","last_fullsized":"2024-06-29T15:57:23.301465+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301078+0000","last_change":"2024-06-29T15:57:22.301362+0000","last_active":"2024-06-29T15:57:23.301078+0000","last_peered":"2024-06-29T15:57:23.301078+0000","last_clean":"2024-06-29T15:57:23.301078+0000","last_became_active":"2024-06-29T15:57:22.300704+0000","last_became_peered":"2024-06-29T15:57:22.300704+0000","last_unstale":"2024-06-29T15:57:23.301078+0000","last_undegraded":"2024-06-29T15:57:23.301078+0000","last_fullsized":"2024-06-29T15:57:23.301078+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301716+0000","last_change":"2024-06-29T15:57:22.301601+0000","last_active":"2024-06-29T15:57:23.301716+0000","last_peered":"2024-06-29T15:57:23.301716+0000","last_clean":"2024-06-29T15:57:23.301716+0000","last_became_active":"2024-06-29T15:57:22.301244+0000","last_became_peered":"2024-06-29T15:57:22.301244+0000","last_unstale":"2024-06-29T15:57:23.301716+0000","last_undegraded":"2024-06-29T15:57:23.301716+0000","last_fullsized":"2024-06-29T15:57:23.301716+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:24:56.938464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296780+0000","last_change":"2024-06-29T15:57:22.292584+0000","last_active":"2024-06-29T15:57:23.296780+0000","last_peered":"2024-06-29T15:57:23.296780+0000","last_clean":"2024-06-29T15:57:23.296780+0000","last_became_active":"2024-06-29T15:57:22.291806+0000","last_became_peered":"2024-06-29T15:57:22.291806+0000","last_unstale":"2024-06-29T15:57:23.296780+0000","last_undegraded":"2024-06-29T15:57:23.296780+0000","last_fullsized":"2024-06-29T15:57:23.296780+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301720+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:23.301720+0000","last_peered":"2024-06-29T15:57:23.301720+0000","last_clean":"2024-06-29T15:57:23.301720+0000","last_became_active":"2024-06-29T15:57:22.301541+0000","last_became_peered":"2024-06-29T15:57:22.301541+0000","last_unstale":"2024-06-29T15:57:23.301720+0000","last_undegraded":"2024-06-29T15:57:23.301720+0000","last_fullsized":"2024-06-29T15:57:23.301720+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:55.249516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295670+0000","last_change":"2024-06-29T15:57:22.298174+0000","last_active":"2024-06-29T15:57:23.295670+0000","last_peered":"2024-06-29T15:57:23.295670+0000","last_clean":"2024-06-29T15:57:23.295670+0000","last_became_active":"2024-06-29T15:57:22.297862+0000","last_became_peered":"2024-06-29T15:57:22.297862+0000","last_unstale":"2024-06-29T15:57:23.295670+0000","last_undegraded":"2024-06-29T15:57:23.295670+0000","last_fullsized":"2024-06-29T15:57:23.295670+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300944+0000","last_change":"2024-06-29T15:57:22.297917+0000","last_active":"2024-06-29T15:57:23.300944+0000","last_peered":"2024-06-29T15:57:23.300944+0000","last_clean":"2024-06-29T15:57:23.300944+0000","last_became_active":"2024-06-29T15:57:22.297638+0000","last_became_peered":"2024-06-29T15:57:22.297638+0000","last_unstale":"2024-06-29T15:57:23.300944+0000","last_undegraded":"2024-06-29T15:57:23.300944+0000","last_fullsized":"2024-06-29T15:57:23.300944+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295387+0000","last_change":"2024-06-29T15:57:22.297998+0000","last_active":"2024-06-29T15:57:23.295387+0000","last_peered":"2024-06-29T15:57:23.295387+0000","last_clean":"2024-06-29T15:57:23.295387+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:23.295387+0000","last_undegraded":"2024-06-29T15:57:23.295387+0000","last_fullsized":"2024-06-29T15:57:23.295387+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296584+0000","last_change":"2024-06-29T15:57:22.285362+0000","last_active":"2024-06-29T15:57:23.296584+0000","last_peered":"2024-06-29T15:57:23.296584+0000","last_clean":"2024-06-29T15:57:23.296584+0000","last_became_active":"2024-06-29T15:57:22.284982+0000","last_became_peered":"2024-06-29T15:57:22.284982+0000","last_unstale":"2024-06-29T15:57:23.296584+0000","last_undegraded":"2024-06-29T15:57:23.296584+0000","last_fullsized":"2024-06-29T15:57:23.296584+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302288+0000","last_change":"2024-06-29T15:57:22.292249+0000","last_active":"2024-06-29T15:57:23.302288+0000","last_peered":"2024-06-29T15:57:23.302288+0000","last_clean":"2024-06-29T15:57:23.302288+0000","last_became_active":"2024-06-29T15:57:22.291918+0000","last_became_peered":"2024-06-29T15:57:22.291918+0000","last_unstale":"2024-06-29T15:57:23.302288+0000","last_undegraded":"2024-06-29T15:57:23.302288+0000","last_fullsized":"2024-06-29T15:57:23.302288+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:57.129905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300605+0000","last_change":"2024-06-29T15:57:22.298975+0000","last_active":"2024-06-29T15:57:23.300605+0000","last_peered":"2024-06-29T15:57:23.300605+0000","last_clean":"2024-06-29T15:57:23.300605+0000","last_became_active":"2024-06-29T15:57:22.298689+0000","last_became_peered":"2024-06-29T15:57:22.298689+0000","last_unstale":"2024-06-29T15:57:23.300605+0000","last_undegraded":"2024-06-29T15:57:23.300605+0000","last_fullsized":"2024-06-29T15:57:23.300605+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.297006+0000","last_change":"2024-06-29T15:57:22.293417+0000","last_active":"2024-06-29T15:57:23.297006+0000","last_peered":"2024-06-29T15:57:23.297006+0000","last_clean":"2024-06-29T15:57:23.297006+0000","last_became_active":"2024-06-29T15:57:22.293083+0000","last_became_peered":"2024-06-29T15:57:22.293083+0000","last_unstale":"2024-06-29T15:57:23.297006+0000","last_undegraded":"2024-06-29T15:57:23.297006+0000","last_fullsized":"2024-06-29T15:57:23.297006+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"212'4","reported_seq":18,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295865+0000","last_change":"2024-06-29T15:57:21.276327+0000","last_active":"2024-06-29T15:57:23.295865+0000","last_peered":"2024-06-29T15:57:23.295865+0000","last_clean":"2024-06-29T15:57:23.295865+0000","last_became_active":"2024-06-29T15:57:21.276031+0000","last_became_peered":"2024-06-29T15:57:21.276031+0000","last_unstale":"2024-06-29T15:57:23.295865+0000","last_undegraded":"2024-06-29T15:57:23.295865+0000","last_fullsized":"2024-06-29T15:57:23.295865+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:23.626777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301563+0000","last_change":"2024-06-29T15:57:22.301084+0000","last_active":"2024-06-29T15:57:23.301563+0000","last_peered":"2024-06-29T15:57:23.301563+0000","last_clean":"2024-06-29T15:57:23.301563+0000","last_became_active":"2024-06-29T15:57:22.300639+0000","last_became_peered":"2024-06-29T15:57:22.300639+0000","last_unstale":"2024-06-29T15:57:23.301563+0000","last_undegraded":"2024-06-29T15:57:23.301563+0000","last_fullsized":"2024-06-29T15:57:23.301563+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.297104+0000","last_change":"2024-06-29T15:57:21.274833+0000","last_active":"2024-06-29T15:57:23.297104+0000","last_peered":"2024-06-29T15:57:23.297104+0000","last_clean":"2024-06-29T15:57:23.297104+0000","last_became_active":"2024-06-29T15:57:21.274451+0000","last_became_peered":"2024-06-29T15:57:21.274451+0000","last_unstale":"2024-06-29T15:57:23.297104+0000","last_undegraded":"2024-06-29T15:57:23.297104+0000","last_fullsized":"2024-06-29T15:57:23.297104+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:25.284538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301062+0000","last_change":"2024-06-29T15:57:22.300202+0000","last_active":"2024-06-29T15:57:23.301062+0000","last_peered":"2024-06-29T15:57:23.301062+0000","last_clean":"2024-06-29T15:57:23.301062+0000","last_became_active":"2024-06-29T15:57:22.299897+0000","last_became_peered":"2024-06-29T15:57:22.299897+0000","last_unstale":"2024-06-29T15:57:23.301062+0000","last_undegraded":"2024-06-29T15:57:23.301062+0000","last_fullsized":"2024-06-29T15:57:23.301062+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302071+0000","last_change":"2024-06-29T15:57:21.284516+0000","last_active":"2024-06-29T15:57:23.302071+0000","last_peered":"2024-06-29T15:57:23.302071+0000","last_clean":"2024-06-29T15:57:23.302071+0000","last_became_active":"2024-06-29T15:57:21.284170+0000","last_became_peered":"2024-06-29T15:57:21.284170+0000","last_unstale":"2024-06-29T15:57:23.302071+0000","last_undegraded":"2024-06-29T15:57:23.302071+0000","last_fullsized":"2024-06-29T15:57:23.302071+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302987+0000","last_change":"2024-06-29T15:57:22.306835+0000","last_active":"2024-06-29T15:57:23.302987+0000","last_peered":"2024-06-29T15:57:23.302987+0000","last_clean":"2024-06-29T15:57:23.302987+0000","last_became_active":"2024-06-29T15:57:22.306295+0000","last_became_peered":"2024-06-29T15:57:22.306295+0000","last_unstale":"2024-06-29T15:57:23.302987+0000","last_undegraded":"2024-06-29T15:57:23.302987+0000","last_fullsized":"2024-06-29T15:57:23.302987+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:52.124235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301387+0000","last_change":"2024-06-29T15:57:21.283411+0000","last_active":"2024-06-29T15:57:23.301387+0000","last_peered":"2024-06-29T15:57:23.301387+0000","last_clean":"2024-06-29T15:57:23.301387+0000","last_became_active":"2024-06-29T15:57:21.283081+0000","last_became_peered":"2024-06-29T15:57:21.283081+0000","last_unstale":"2024-06-29T15:57:23.301387+0000","last_undegraded":"2024-06-29T15:57:23.301387+0000","last_fullsized":"2024-06-29T15:57:23.301387+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:27.866266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301740+0000","last_change":"2024-06-29T15:57:22.297633+0000","last_active":"2024-06-29T15:57:23.301740+0000","last_peered":"2024-06-29T15:57:23.301740+0000","last_clean":"2024-06-29T15:57:23.301740+0000","last_became_active":"2024-06-29T15:57:22.297319+0000","last_became_peered":"2024-06-29T15:57:22.297319+0000","last_unstale":"2024-06-29T15:57:23.301740+0000","last_undegraded":"2024-06-29T15:57:23.301740+0000","last_fullsized":"2024-06-29T15:57:23.301740+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:19.625176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298911+0000","last_change":"2024-06-29T15:57:21.274306+0000","last_active":"2024-06-29T15:57:23.298911+0000","last_peered":"2024-06-29T15:57:23.298911+0000","last_clean":"2024-06-29T15:57:23.298911+0000","last_became_active":"2024-06-29T15:57:21.273632+0000","last_became_peered":"2024-06-29T15:57:21.273632+0000","last_unstale":"2024-06-29T15:57:23.298911+0000","last_undegraded":"2024-06-29T15:57:23.298911+0000","last_fullsized":"2024-06-29T15:57:23.298911+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298413+0000","last_change":"2024-06-29T15:57:22.293696+0000","last_active":"2024-06-29T15:57:23.298413+0000","last_peered":"2024-06-29T15:57:23.298413+0000","last_clean":"2024-06-29T15:57:23.298413+0000","last_became_active":"2024-06-29T15:57:22.293263+0000","last_became_peered":"2024-06-29T15:57:22.293263+0000","last_unstale":"2024-06-29T15:57:23.298413+0000","last_undegraded":"2024-06-29T15:57:23.298413+0000","last_fullsized":"2024-06-29T15:57:23.298413+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298286+0000","last_change":"2024-06-29T15:57:21.277884+0000","last_active":"2024-06-29T15:57:23.298286+0000","last_peered":"2024-06-29T15:57:23.298286+0000","last_clean":"2024-06-29T15:57:23.298286+0000","last_became_active":"2024-06-29T15:57:21.277589+0000","last_became_peered":"2024-06-29T15:57:21.277589+0000","last_unstale":"2024-06-29T15:57:23.298286+0000","last_undegraded":"2024-06-29T15:57:23.298286+0000","last_fullsized":"2024-06-29T15:57:23.298286+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:09.017396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300957+0000","last_change":"2024-06-29T15:57:22.299457+0000","last_active":"2024-06-29T15:57:23.300957+0000","last_peered":"2024-06-29T15:57:23.300957+0000","last_clean":"2024-06-29T15:57:23.300957+0000","last_became_active":"2024-06-29T15:57:22.299157+0000","last_became_peered":"2024-06-29T15:57:22.299157+0000","last_unstale":"2024-06-29T15:57:23.300957+0000","last_undegraded":"2024-06-29T15:57:23.300957+0000","last_fullsized":"2024-06-29T15:57:23.300957+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:40.967292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300567+0000","last_change":"2024-06-29T15:57:21.277245+0000","last_active":"2024-06-29T15:57:23.300567+0000","last_peered":"2024-06-29T15:57:23.300567+0000","last_clean":"2024-06-29T15:57:23.300567+0000","last_became_active":"2024-06-29T15:57:21.276931+0000","last_became_peered":"2024-06-29T15:57:21.276931+0000","last_unstale":"2024-06-29T15:57:23.300567+0000","last_undegraded":"2024-06-29T15:57:23.300567+0000","last_fullsized":"2024-06-29T15:57:23.300567+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301386+0000","last_change":"2024-06-29T15:57:22.303487+0000","last_active":"2024-06-29T15:57:23.301386+0000","last_peered":"2024-06-29T15:57:23.301386+0000","last_clean":"2024-06-29T15:57:23.301386+0000","last_became_active":"2024-06-29T15:57:22.302933+0000","last_became_peered":"2024-06-29T15:57:22.302933+0000","last_unstale":"2024-06-29T15:57:23.301386+0000","last_undegraded":"2024-06-29T15:57:23.301386+0000","last_fullsized":"2024-06-29T15:57:23.301386+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:28.493187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.299107+0000","last_change":"2024-06-29T15:57:21.272019+0000","last_active":"2024-06-29T15:57:23.299107+0000","last_peered":"2024-06-29T15:57:23.299107+0000","last_clean":"2024-06-29T15:57:23.299107+0000","last_became_active":"2024-06-29T15:57:21.271715+0000","last_became_peered":"2024-06-29T15:57:21.271715+0000","last_unstale":"2024-06-29T15:57:23.299107+0000","last_undegraded":"2024-06-29T15:57:23.299107+0000","last_fullsized":"2024-06-29T15:57:23.299107+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301851+0000","last_change":"2024-06-29T15:57:22.298454+0000","last_active":"2024-06-29T15:57:23.301851+0000","last_peered":"2024-06-29T15:57:23.301851+0000","last_clean":"2024-06-29T15:57:23.301851+0000","last_became_active":"2024-06-29T15:57:22.298162+0000","last_became_peered":"2024-06-29T15:57:22.298162+0000","last_unstale":"2024-06-29T15:57:23.301851+0000","last_undegraded":"2024-06-29T15:57:23.301851+0000","last_fullsized":"2024-06-29T15:57:23.301851+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301313+0000","last_change":"2024-06-29T15:57:21.275926+0000","last_active":"2024-06-29T15:57:23.301313+0000","last_peered":"2024-06-29T15:57:23.301313+0000","last_clean":"2024-06-29T15:57:23.301313+0000","last_became_active":"2024-06-29T15:57:21.275593+0000","last_became_peered":"2024-06-29T15:57:21.275593+0000","last_unstale":"2024-06-29T15:57:23.301313+0000","last_undegraded":"2024-06-29T15:57:23.301313+0000","last_fullsized":"2024-06-29T15:57:23.301313+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301303+0000","last_change":"2024-06-29T15:57:22.295260+0000","last_active":"2024-06-29T15:57:23.301303+0000","last_peered":"2024-06-29T15:57:23.301303+0000","last_clean":"2024-06-29T15:57:23.301303+0000","last_became_active":"2024-06-29T15:57:22.294952+0000","last_became_peered":"2024-06-29T15:57:22.294952+0000","last_unstale":"2024-06-29T15:57:23.301303+0000","last_undegraded":"2024-06-29T15:57:23.301303+0000","last_fullsized":"2024-06-29T15:57:23.301303+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:56.473182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"212'3","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302839+0000","last_change":"2024-06-29T15:57:21.283791+0000","last_active":"2024-06-29T15:57:23.302839+0000","last_peered":"2024-06-29T15:57:23.302839+0000","last_clean":"2024-06-29T15:57:23.302839+0000","last_became_active":"2024-06-29T15:57:21.283458+0000","last_became_peered":"2024-06-29T15:57:21.283458+0000","last_unstale":"2024-06-29T15:57:23.302839+0000","last_undegraded":"2024-06-29T15:57:23.302839+0000","last_fullsized":"2024-06-29T15:57:23.302839+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:54.365881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301313+0000","last_change":"2024-06-29T15:57:22.302666+0000","last_active":"2024-06-29T15:57:23.301313+0000","last_peered":"2024-06-29T15:57:23.301313+0000","last_clean":"2024-06-29T15:57:23.301313+0000","last_became_active":"2024-06-29T15:57:22.302254+0000","last_became_peered":"2024-06-29T15:57:22.302254+0000","last_unstale":"2024-06-29T15:57:23.301313+0000","last_undegraded":"2024-06-29T15:57:23.301313+0000","last_fullsized":"2024-06-29T15:57:23.301313+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"212'6","reported_seq":20,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301489+0000","last_change":"2024-06-29T15:57:21.278161+0000","last_active":"2024-06-29T15:57:23.301489+0000","last_peered":"2024-06-29T15:57:23.301489+0000","last_clean":"2024-06-29T15:57:23.301489+0000","last_became_active":"2024-06-29T15:57:21.277794+0000","last_became_peered":"2024-06-29T15:57:21.277794+0000","last_unstale":"2024-06-29T15:57:23.301489+0000","last_undegraded":"2024-06-29T15:57:23.301489+0000","last_fullsized":"2024-06-29T15:57:23.301489+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:56.330168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298754+0000","last_change":"2024-06-29T15:57:22.291164+0000","last_active":"2024-06-29T15:57:23.298754+0000","last_peered":"2024-06-29T15:57:23.298754+0000","last_clean":"2024-06-29T15:57:23.298754+0000","last_became_active":"2024-06-29T15:57:22.290897+0000","last_became_peered":"2024-06-29T15:57:22.290897+0000","last_unstale":"2024-06-29T15:57:23.298754+0000","last_undegraded":"2024-06-29T15:57:23.298754+0000","last_fullsized":"2024-06-29T15:57:23.298754+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301727+0000","last_change":"2024-06-29T15:57:21.281309+0000","last_active":"2024-06-29T15:57:23.301727+0000","last_peered":"2024-06-29T15:57:23.301727+0000","last_clean":"2024-06-29T15:57:23.301727+0000","last_became_active":"2024-06-29T15:57:21.280944+0000","last_became_peered":"2024-06-29T15:57:21.280944+0000","last_unstale":"2024-06-29T15:57:23.301727+0000","last_undegraded":"2024-06-29T15:57:23.301727+0000","last_fullsized":"2024-06-29T15:57:23.301727+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.303325+0000","last_change":"2024-06-29T15:57:22.294011+0000","last_active":"2024-06-29T15:57:23.303325+0000","last_peered":"2024-06-29T15:57:23.303325+0000","last_clean":"2024-06-29T15:57:23.303325+0000","last_became_active":"2024-06-29T15:57:22.293721+0000","last_became_peered":"2024-06-29T15:57:22.293721+0000","last_unstale":"2024-06-29T15:57:23.303325+0000","last_undegraded":"2024-06-29T15:57:23.303325+0000","last_fullsized":"2024-06-29T15:57:23.303325+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:57.337708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301020+0000","last_change":"2024-06-29T15:57:21.282872+0000","last_active":"2024-06-29T15:57:23.301020+0000","last_peered":"2024-06-29T15:57:23.301020+0000","last_clean":"2024-06-29T15:57:23.301020+0000","last_became_active":"2024-06-29T15:57:21.282549+0000","last_became_peered":"2024-06-29T15:57:21.282549+0000","last_unstale":"2024-06-29T15:57:23.301020+0000","last_undegraded":"2024-06-29T15:57:23.301020+0000","last_fullsized":"2024-06-29T15:57:23.301020+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301202+0000","last_change":"2024-06-29T15:57:22.299631+0000","last_active":"2024-06-29T15:57:23.301202+0000","last_peered":"2024-06-29T15:57:23.301202+0000","last_clean":"2024-06-29T15:57:23.301202+0000","last_became_active":"2024-06-29T15:57:22.299285+0000","last_became_peered":"2024-06-29T15:57:22.299285+0000","last_unstale":"2024-06-29T15:57:23.301202+0000","last_undegraded":"2024-06-29T15:57:23.301202+0000","last_fullsized":"2024-06-29T15:57:23.301202+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302228+0000","last_change":"2024-06-29T15:57:21.276731+0000","last_active":"2024-06-29T15:57:23.302228+0000","last_peered":"2024-06-29T15:57:23.302228+0000","last_clean":"2024-06-29T15:57:23.302228+0000","last_became_active":"2024-06-29T15:57:21.276408+0000","last_became_peered":"2024-06-29T15:57:21.276408+0000","last_unstale":"2024-06-29T15:57:23.302228+0000","last_undegraded":"2024-06-29T15:57:23.302228+0000","last_fullsized":"2024-06-29T15:57:23.302228+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:48.911239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300987+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:23.300987+0000","last_peered":"2024-06-29T15:57:23.300987+0000","last_clean":"2024-06-29T15:57:23.300987+0000","last_became_active":"2024-06-29T15:57:22.301832+0000","last_became_peered":"2024-06-29T15:57:22.301832+0000","last_unstale":"2024-06-29T15:57:23.300987+0000","last_undegraded":"2024-06-29T15:57:23.300987+0000","last_fullsized":"2024-06-29T15:57:23.300987+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"212'8","reported_seq":22,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301444+0000","last_change":"2024-06-29T15:57:21.283745+0000","last_active":"2024-06-29T15:57:23.301444+0000","last_peered":"2024-06-29T15:57:23.301444+0000","last_clean":"2024-06-29T15:57:23.301444+0000","last_became_active":"2024-06-29T15:57:21.283403+0000","last_became_peered":"2024-06-29T15:57:21.283403+0000","last_unstale":"2024-06-29T15:57:23.301444+0000","last_undegraded":"2024-06-29T15:57:23.301444+0000","last_fullsized":"2024-06-29T15:57:23.301444+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.297167+0000","last_change":"2024-06-29T15:57:22.300916+0000","last_active":"2024-06-29T15:57:23.297167+0000","last_peered":"2024-06-29T15:57:23.297167+0000","last_clean":"2024-06-29T15:57:23.297167+0000","last_became_active":"2024-06-29T15:57:22.300620+0000","last_became_peered":"2024-06-29T15:57:22.300620+0000","last_unstale":"2024-06-29T15:57:23.297167+0000","last_undegraded":"2024-06-29T15:57:23.297167+0000","last_fullsized":"2024-06-29T15:57:23.297167+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:41.172455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.299177+0000","last_change":"2024-06-29T15:57:21.273304+0000","last_active":"2024-06-29T15:57:23.299177+0000","last_peered":"2024-06-29T15:57:23.299177+0000","last_clean":"2024-06-29T15:57:23.299177+0000","last_became_active":"2024-06-29T15:57:21.272972+0000","last_became_peered":"2024-06-29T15:57:21.272972+0000","last_unstale":"2024-06-29T15:57:23.299177+0000","last_undegraded":"2024-06-29T15:57:23.299177+0000","last_fullsized":"2024-06-29T15:57:23.299177+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296846+0000","last_change":"2024-06-29T15:57:22.291360+0000","last_active":"2024-06-29T15:57:23.296846+0000","last_peered":"2024-06-29T15:57:23.296846+0000","last_clean":"2024-06-29T15:57:23.296846+0000","last_became_active":"2024-06-29T15:57:22.290801+0000","last_became_peered":"2024-06-29T15:57:22.290801+0000","last_unstale":"2024-06-29T15:57:23.296846+0000","last_undegraded":"2024-06-29T15:57:23.296846+0000","last_fullsized":"2024-06-29T15:57:23.296846+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.297197+0000","last_change":"2024-06-29T15:57:21.278476+0000","last_active":"2024-06-29T15:57:23.297197+0000","last_peered":"2024-06-29T15:57:23.297197+0000","last_clean":"2024-06-29T15:57:23.297197+0000","last_became_active":"2024-06-29T15:57:21.278150+0000","last_became_peered":"2024-06-29T15:57:21.278150+0000","last_unstale":"2024-06-29T15:57:23.297197+0000","last_undegraded":"2024-06-29T15:57:23.297197+0000","last_fullsized":"2024-06-29T15:57:23.297197+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:33.991332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301102+0000","last_change":"2024-06-29T15:57:22.297275+0000","last_active":"2024-06-29T15:57:23.301102+0000","last_peered":"2024-06-29T15:57:23.301102+0000","last_clean":"2024-06-29T15:57:23.301102+0000","last_became_active":"2024-06-29T15:57:22.296984+0000","last_became_peered":"2024-06-29T15:57:22.296984+0000","last_unstale":"2024-06-29T15:57:23.301102+0000","last_undegraded":"2024-06-29T15:57:23.301102+0000","last_fullsized":"2024-06-29T15:57:23.301102+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302945+0000","last_change":"2024-06-29T15:57:21.279139+0000","last_active":"2024-06-29T15:57:23.302945+0000","last_peered":"2024-06-29T15:57:23.302945+0000","last_clean":"2024-06-29T15:57:23.302945+0000","last_became_active":"2024-06-29T15:57:21.278488+0000","last_became_peered":"2024-06-29T15:57:21.278488+0000","last_unstale":"2024-06-29T15:57:23.302945+0000","last_undegraded":"2024-06-29T15:57:23.302945+0000","last_fullsized":"2024-06-29T15:57:23.302945+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:54.126338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302570+0000","last_change":"2024-06-29T15:57:22.305806+0000","last_active":"2024-06-29T15:57:23.302570+0000","last_peered":"2024-06-29T15:57:23.302570+0000","last_clean":"2024-06-29T15:57:23.302570+0000","last_became_active":"2024-06-29T15:57:22.296276+0000","last_became_peered":"2024-06-29T15:57:22.296276+0000","last_unstale":"2024-06-29T15:57:23.302570+0000","last_undegraded":"2024-06-29T15:57:23.302570+0000","last_fullsized":"2024-06-29T15:57:23.302570+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:43:51.754223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301194+0000","last_change":"2024-06-29T15:57:21.282111+0000","last_active":"2024-06-29T15:57:23.301194+0000","last_peered":"2024-06-29T15:57:23.301194+0000","last_clean":"2024-06-29T15:57:23.301194+0000","last_became_active":"2024-06-29T15:57:21.281761+0000","last_became_peered":"2024-06-29T15:57:21.281761+0000","last_unstale":"2024-06-29T15:57:23.301194+0000","last_undegraded":"2024-06-29T15:57:23.301194+0000","last_fullsized":"2024-06-29T15:57:23.301194+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302432+0000","last_change":"2024-06-29T15:57:22.288958+0000","last_active":"2024-06-29T15:57:23.302432+0000","last_peered":"2024-06-29T15:57:23.302432+0000","last_clean":"2024-06-29T15:57:23.302432+0000","last_became_active":"2024-06-29T15:57:22.288621+0000","last_became_peered":"2024-06-29T15:57:22.288621+0000","last_unstale":"2024-06-29T15:57:23.302432+0000","last_undegraded":"2024-06-29T15:57:23.302432+0000","last_fullsized":"2024-06-29T15:57:23.302432+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:31.720751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.303301+0000","last_change":"2024-06-29T15:57:21.284009+0000","last_active":"2024-06-29T15:57:23.303301+0000","last_peered":"2024-06-29T15:57:23.303301+0000","last_clean":"2024-06-29T15:57:23.303301+0000","last_became_active":"2024-06-29T15:57:21.283679+0000","last_became_peered":"2024-06-29T15:57:21.283679+0000","last_unstale":"2024-06-29T15:57:23.303301+0000","last_undegraded":"2024-06-29T15:57:23.303301+0000","last_fullsized":"2024-06-29T15:57:23.303301+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:20.284468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302233+0000","last_change":"2024-06-29T15:57:22.291304+0000","last_active":"2024-06-29T15:57:23.302233+0000","last_peered":"2024-06-29T15:57:23.302233+0000","last_clean":"2024-06-29T15:57:23.302233+0000","last_became_active":"2024-06-29T15:57:22.290983+0000","last_became_peered":"2024-06-29T15:57:22.290983+0000","last_unstale":"2024-06-29T15:57:23.302233+0000","last_undegraded":"2024-06-29T15:57:23.302233+0000","last_fullsized":"2024-06-29T15:57:23.302233+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301512+0000","last_change":"2024-06-29T15:57:21.284780+0000","last_active":"2024-06-29T15:57:23.301512+0000","last_peered":"2024-06-29T15:57:23.301512+0000","last_clean":"2024-06-29T15:57:23.301512+0000","last_became_active":"2024-06-29T15:57:21.284273+0000","last_became_peered":"2024-06-29T15:57:21.284273+0000","last_unstale":"2024-06-29T15:57:23.301512+0000","last_undegraded":"2024-06-29T15:57:23.301512+0000","last_fullsized":"2024-06-29T15:57:23.301512+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:16.085990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296420+0000","last_change":"2024-06-29T15:57:22.298744+0000","last_active":"2024-06-29T15:57:23.296420+0000","last_peered":"2024-06-29T15:57:23.296420+0000","last_clean":"2024-06-29T15:57:23.296420+0000","last_became_active":"2024-06-29T15:57:22.296925+0000","last_became_peered":"2024-06-29T15:57:22.296925+0000","last_unstale":"2024-06-29T15:57:23.296420+0000","last_undegraded":"2024-06-29T15:57:23.296420+0000","last_fullsized":"2024-06-29T15:57:23.296420+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:40.578182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300970+0000","last_change":"2024-06-29T15:57:21.284982+0000","last_active":"2024-06-29T15:57:23.300970+0000","last_peered":"2024-06-29T15:57:23.300970+0000","last_clean":"2024-06-29T15:57:23.300970+0000","last_became_active":"2024-06-29T15:57:21.284684+0000","last_became_peered":"2024-06-29T15:57:21.284684+0000","last_unstale":"2024-06-29T15:57:23.300970+0000","last_undegraded":"2024-06-29T15:57:23.300970+0000","last_fullsized":"2024-06-29T15:57:23.300970+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301567+0000","last_change":"2024-06-29T15:57:22.300220+0000","last_active":"2024-06-29T15:57:23.301567+0000","last_peered":"2024-06-29T15:57:23.301567+0000","last_clean":"2024-06-29T15:57:23.301567+0000","last_became_active":"2024-06-29T15:57:22.299880+0000","last_became_peered":"2024-06-29T15:57:22.299880+0000","last_unstale":"2024-06-29T15:57:23.301567+0000","last_undegraded":"2024-06-29T15:57:23.301567+0000","last_fullsized":"2024-06-29T15:57:23.301567+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298430+0000","last_change":"2024-06-29T15:57:21.278011+0000","last_active":"2024-06-29T15:57:23.298430+0000","last_peered":"2024-06-29T15:57:23.298430+0000","last_clean":"2024-06-29T15:57:23.298430+0000","last_became_active":"2024-06-29T15:57:21.276648+0000","last_became_peered":"2024-06-29T15:57:21.276648+0000","last_unstale":"2024-06-29T15:57:23.298430+0000","last_undegraded":"2024-06-29T15:57:23.298430+0000","last_fullsized":"2024-06-29T15:57:23.298430+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295724+0000","last_change":"2024-06-29T15:57:22.298914+0000","last_active":"2024-06-29T15:57:23.295724+0000","last_peered":"2024-06-29T15:57:23.295724+0000","last_clean":"2024-06-29T15:57:23.295724+0000","last_became_active":"2024-06-29T15:57:22.298614+0000","last_became_peered":"2024-06-29T15:57:22.298614+0000","last_unstale":"2024-06-29T15:57:23.295724+0000","last_undegraded":"2024-06-29T15:57:23.295724+0000","last_fullsized":"2024-06-29T15:57:23.295724+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301951+0000","last_change":"2024-06-29T15:57:21.277662+0000","last_active":"2024-06-29T15:57:23.301951+0000","last_peered":"2024-06-29T15:57:23.301951+0000","last_clean":"2024-06-29T15:57:23.301951+0000","last_became_active":"2024-06-29T15:57:21.277335+0000","last_became_peered":"2024-06-29T15:57:21.277335+0000","last_unstale":"2024-06-29T15:57:23.301951+0000","last_undegraded":"2024-06-29T15:57:23.301951+0000","last_fullsized":"2024-06-29T15:57:23.301951+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301549+0000","last_change":"2024-06-29T15:57:22.303802+0000","last_active":"2024-06-29T15:57:23.301549+0000","last_peered":"2024-06-29T15:57:23.301549+0000","last_clean":"2024-06-29T15:57:23.301549+0000","last_became_active":"2024-06-29T15:57:22.303511+0000","last_became_peered":"2024-06-29T15:57:22.303511+0000","last_unstale":"2024-06-29T15:57:23.301549+0000","last_undegraded":"2024-06-29T15:57:23.301549+0000","last_fullsized":"2024-06-29T15:57:23.301549+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:10.536799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301634+0000","last_change":"2024-06-29T15:57:21.276972+0000","last_active":"2024-06-29T15:57:23.301634+0000","last_peered":"2024-06-29T15:57:23.301634+0000","last_clean":"2024-06-29T15:57:23.301634+0000","last_became_active":"2024-06-29T15:57:21.276647+0000","last_became_peered":"2024-06-29T15:57:21.276647+0000","last_unstale":"2024-06-29T15:57:23.301634+0000","last_undegraded":"2024-06-29T15:57:23.301634+0000","last_fullsized":"2024-06-29T15:57:23.301634+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:24.871281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296727+0000","last_change":"2024-06-29T15:57:22.296472+0000","last_active":"2024-06-29T15:57:23.296727+0000","last_peered":"2024-06-29T15:57:23.296727+0000","last_clean":"2024-06-29T15:57:23.296727+0000","last_became_active":"2024-06-29T15:57:22.296130+0000","last_became_peered":"2024-06-29T15:57:22.296130+0000","last_unstale":"2024-06-29T15:57:23.296727+0000","last_undegraded":"2024-06-29T15:57:23.296727+0000","last_fullsized":"2024-06-29T15:57:23.296727+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298687+0000","last_change":"2024-06-29T15:57:21.280224+0000","last_active":"2024-06-29T15:57:23.298687+0000","last_peered":"2024-06-29T15:57:23.298687+0000","last_clean":"2024-06-29T15:57:23.298687+0000","last_became_active":"2024-06-29T15:57:21.279899+0000","last_became_peered":"2024-06-29T15:57:21.279899+0000","last_unstale":"2024-06-29T15:57:23.298687+0000","last_undegraded":"2024-06-29T15:57:23.298687+0000","last_fullsized":"2024-06-29T15:57:23.298687+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301495+0000","last_change":"2024-06-29T15:57:22.301166+0000","last_active":"2024-06-29T15:57:23.301495+0000","last_peered":"2024-06-29T15:57:23.301495+0000","last_clean":"2024-06-29T15:57:23.301495+0000","last_became_active":"2024-06-29T15:57:22.300676+0000","last_became_peered":"2024-06-29T15:57:22.300676+0000","last_unstale":"2024-06-29T15:57:23.301495+0000","last_undegraded":"2024-06-29T15:57:23.301495+0000","last_fullsized":"2024-06-29T15:57:23.301495+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301286+0000","last_change":"2024-06-29T15:57:21.277560+0000","last_active":"2024-06-29T15:57:23.301286+0000","last_peered":"2024-06-29T15:57:23.301286+0000","last_clean":"2024-06-29T15:57:23.301286+0000","last_became_active":"2024-06-29T15:57:21.277255+0000","last_became_peered":"2024-06-29T15:57:21.277255+0000","last_unstale":"2024-06-29T15:57:23.301286+0000","last_undegraded":"2024-06-29T15:57:23.301286+0000","last_fullsized":"2024-06-29T15:57:23.301286+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301342+0000","last_change":"2024-06-29T15:57:22.304057+0000","last_active":"2024-06-29T15:57:23.301342+0000","last_peered":"2024-06-29T15:57:23.301342+0000","last_clean":"2024-06-29T15:57:23.301342+0000","last_became_active":"2024-06-29T15:57:22.303762+0000","last_became_peered":"2024-06-29T15:57:22.303762+0000","last_unstale":"2024-06-29T15:57:23.301342+0000","last_undegraded":"2024-06-29T15:57:23.301342+0000","last_fullsized":"2024-06-29T15:57:23.301342+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:40.125779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301784+0000","last_change":"2024-06-29T15:57:21.281405+0000","last_active":"2024-06-29T15:57:23.301784+0000","last_peered":"2024-06-29T15:57:23.301784+0000","last_clean":"2024-06-29T15:57:23.301784+0000","last_became_active":"2024-06-29T15:57:21.281080+0000","last_became_peered":"2024-06-29T15:57:21.281080+0000","last_unstale":"2024-06-29T15:57:23.301784+0000","last_undegraded":"2024-06-29T15:57:23.301784+0000","last_fullsized":"2024-06-29T15:57:23.301784+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:51.116695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295627+0000","last_change":"2024-06-29T15:57:22.291143+0000","last_active":"2024-06-29T15:57:23.295627+0000","last_peered":"2024-06-29T15:57:23.295627+0000","last_clean":"2024-06-29T15:57:23.295627+0000","last_became_active":"2024-06-29T15:57:22.290849+0000","last_became_peered":"2024-06-29T15:57:22.290849+0000","last_unstale":"2024-06-29T15:57:23.295627+0000","last_undegraded":"2024-06-29T15:57:23.295627+0000","last_fullsized":"2024-06-29T15:57:23.295627+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.299043+0000","last_change":"2024-06-29T15:57:21.274551+0000","last_active":"2024-06-29T15:57:23.299043+0000","last_peered":"2024-06-29T15:57:23.299043+0000","last_clean":"2024-06-29T15:57:23.299043+0000","last_became_active":"2024-06-29T15:57:21.274029+0000","last_became_peered":"2024-06-29T15:57:21.274029+0000","last_unstale":"2024-06-29T15:57:23.299043+0000","last_undegraded":"2024-06-29T15:57:23.299043+0000","last_fullsized":"2024-06-29T15:57:23.299043+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301337+0000","last_change":"2024-06-29T15:57:22.295101+0000","last_active":"2024-06-29T15:57:23.301337+0000","last_peered":"2024-06-29T15:57:23.301337+0000","last_clean":"2024-06-29T15:57:23.301337+0000","last_became_active":"2024-06-29T15:57:22.294805+0000","last_became_peered":"2024-06-29T15:57:22.294805+0000","last_unstale":"2024-06-29T15:57:23.301337+0000","last_undegraded":"2024-06-29T15:57:23.301337+0000","last_fullsized":"2024-06-29T15:57:23.301337+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301205+0000","last_change":"2024-06-29T15:57:21.274359+0000","last_active":"2024-06-29T15:57:23.301205+0000","last_peered":"2024-06-29T15:57:23.301205+0000","last_clean":"2024-06-29T15:57:23.301205+0000","last_became_active":"2024-06-29T15:57:21.274036+0000","last_became_peered":"2024-06-29T15:57:21.274036+0000","last_unstale":"2024-06-29T15:57:23.301205+0000","last_undegraded":"2024-06-29T15:57:23.301205+0000","last_fullsized":"2024-06-29T15:57:23.301205+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:02.747066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296457+0000","last_change":"2024-06-29T15:57:22.292872+0000","last_active":"2024-06-29T15:57:23.296457+0000","last_peered":"2024-06-29T15:57:23.296457+0000","last_clean":"2024-06-29T15:57:23.296457+0000","last_became_active":"2024-06-29T15:57:22.292485+0000","last_became_peered":"2024-06-29T15:57:22.292485+0000","last_unstale":"2024-06-29T15:57:23.296457+0000","last_undegraded":"2024-06-29T15:57:23.296457+0000","last_fullsized":"2024-06-29T15:57:23.296457+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:19.369559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295273+0000","last_change":"2024-06-29T15:57:21.283495+0000","last_active":"2024-06-29T15:57:23.295273+0000","last_peered":"2024-06-29T15:57:23.295273+0000","last_clean":"2024-06-29T15:57:23.295273+0000","last_became_active":"2024-06-29T15:57:21.283125+0000","last_became_peered":"2024-06-29T15:57:21.283125+0000","last_unstale":"2024-06-29T15:57:23.295273+0000","last_undegraded":"2024-06-29T15:57:23.295273+0000","last_fullsized":"2024-06-29T15:57:23.295273+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301166+0000","last_change":"2024-06-29T15:57:22.302267+0000","last_active":"2024-06-29T15:57:23.301166+0000","last_peered":"2024-06-29T15:57:23.301166+0000","last_clean":"2024-06-29T15:57:23.301166+0000","last_became_active":"2024-06-29T15:57:22.301972+0000","last_became_peered":"2024-06-29T15:57:22.301972+0000","last_unstale":"2024-06-29T15:57:23.301166+0000","last_undegraded":"2024-06-29T15:57:23.301166+0000","last_fullsized":"2024-06-29T15:57:23.301166+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295755+0000","last_change":"2024-06-29T15:57:21.282026+0000","last_active":"2024-06-29T15:57:23.295755+0000","last_peered":"2024-06-29T15:57:23.295755+0000","last_clean":"2024-06-29T15:57:23.295755+0000","last_became_active":"2024-06-29T15:57:21.281709+0000","last_became_peered":"2024-06-29T15:57:21.281709+0000","last_unstale":"2024-06-29T15:57:23.295755+0000","last_undegraded":"2024-06-29T15:57:23.295755+0000","last_fullsized":"2024-06-29T15:57:23.295755+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:54:07.641300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"202'219","reported_seq":1039,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301114+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:57:23.301114+0000","last_peered":"2024-06-29T15:57:23.301114+0000","last_clean":"2024-06-29T15:57:23.301114+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:57:23.301114+0000","last_undegraded":"2024-06-29T15:57:23.301114+0000","last_fullsized":"2024-06-29T15:57:23.301114+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575985,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48280,"kb_used_data":3204,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323560,"statfs":{"total":96636764160,"available":96587325440,"internally_reserved":0,"allocated":3280896,"data_stored":1898965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42124,"kb_used_data":2616,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329716,"statfs":{"total":96636764160,"available":96593629184,"internally_reserved":0,"allocated":2678784,"data_stored":1307567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42108,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329732,"statfs":{"total":96636764160,"available":96593645568,"internally_reserved":0,"allocated":2666496,"data_stored":1301139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575985,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60180,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311660,"statfs":{"total":96636764160,"available":96575139840,"internally_reserved":0,"allocated":2695168,"data_stored":1308701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575985,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47700,"kb_used_data":2624,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324140,"statfs":{"total":96636764160,"available":96587919360,"internally_reserved":0,"allocated":2686976,"data_stored":1307167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575985,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47688,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324152,"statfs":{"total":96636764160,"available":96587931648,"internally_reserved":0,"allocated":2674688,"data_stored":1306985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575984,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60176,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311664,"statfs":{"total":96636764160,"available":96575143936,"internally_reserved":0,"allocated":2682880,"data_stored":1307041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575985,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60732,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311108,"statfs":{"total":96636764160,"available":96574574592,"internally_reserved":0,"allocated":3260416,"data_stored":1891015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:26.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:26.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.465+0000 7fd1aec64640 1 -- 172.21.15.81:0/2024125737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1a806f660 msgr2=0x7fd1a806fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.465+0000 7fd1aec64640 1 --2- 172.21.15.81:0/2024125737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1a806f660 0x7fd1a806fa70 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009830 tx=0x7fd1a002a480 comp rx=0 tx=0).stop 2024-06-29T15:57:26.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.465+0000 7fd1aec64640 1 -- 172.21.15.81:0/2024125737 shutdown_connections 2024-06-29T15:57:26.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.465+0000 7fd1aec64640 1 --2- 172.21.15.81:0/2024125737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1a806f660 0x7fd1a806fa70 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.465+0000 7fd1aec64640 1 --2- 172.21.15.81:0/2024125737 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1a80b35b0 0x7fd1a815b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.465+0000 7fd1aec64640 1 -- 172.21.15.81:0/2024125737 >> 172.21.15.81:0/2024125737 conn(0x7fd1a806ccc0 msgr2=0x7fd1a80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:26.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.465+0000 7fd1aec64640 1 -- 172.21.15.81:0/2024125737 shutdown_connections 2024-06-29T15:57:26.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.465+0000 7fd1aec64640 1 -- 172.21.15.81:0/2024125737 wait complete. 2024-06-29T15:57:26.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.467+0000 7fd1aec64640 1 Processor -- start 2024-06-29T15:57:26.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.467+0000 7fd1aec64640 1 -- start start 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd1aec64640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1a806f660 0x7fd1a80a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd1aec64640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1a80b35b0 0x7fd1a813f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd1aec64640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1a80b4430 0x7fd1a81f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd1aec64640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a806f160 con 0x7fd1a80b35b0 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd1aec64640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a806efe0 con 0x7fd1a806f660 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd1aec64640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd1a806f2e0 con 0x7fd1a80b4430 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1a80b35b0 0x7fd1a813f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1a80b35b0 0x7fd1a813f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59848/0 (socket says 172.21.15.81:59848) 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd19ffff640 1 -- 172.21.15.81:0/2892639726 learned_addr learned my addr 172.21.15.81:0/2892639726 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd1ad201640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1a80b4430 0x7fd1a81f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd1aca00640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1a806f660 0x7fd1a80a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd19ffff640 1 -- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1a80b4430 msgr2=0x7fd1a81f6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd19ffff640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1a80b4430 0x7fd1a81f6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd19ffff640 1 -- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1a806f660 msgr2=0x7fd1a80a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd19ffff640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1a806f660 0x7fd1a80a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd19ffff640 1 -- 172.21.15.81:0/2892639726 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a81574f0 con 0x7fd1a80b35b0 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd1aca00640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1a806f660 0x7fd1a80a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.469+0000 7fd19ffff640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1a80b35b0 0x7fd1a813f360 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fd19800c730 tx=0x7fd19800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.470+0000 7fd19d7fa640 1 -- 172.21.15.81:0/2892639726 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd198016020 con 0x7fd1a80b35b0 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.470+0000 7fd19d7fa640 1 -- 172.21.15.81:0/2892639726 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd19800a950 con 0x7fd1a80b35b0 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.470+0000 7fd1aec64640 1 -- 172.21.15.81:0/2892639726 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a81577d0 con 0x7fd1a80b35b0 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.470+0000 7fd19d7fa640 1 -- 172.21.15.81:0/2892639726 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19800ada0 con 0x7fd1a80b35b0 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.470+0000 7fd1aec64640 1 -- 172.21.15.81:0/2892639726 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a81589d0 con 0x7fd1a80b35b0 2024-06-29T15:57:26.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.471+0000 7fd1aec64640 1 -- 172.21.15.81:0/2892639726 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1a815d610 con 0x7fd1a80b35b0 2024-06-29T15:57:26.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.471+0000 7fd19d7fa640 1 -- 172.21.15.81:0/2892639726 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fd198019020 con 0x7fd1a80b35b0 2024-06-29T15:57:26.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.472+0000 7fd19d7fa640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd1940763f0 0x7fd194078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.472+0000 7fd19d7fa640 1 -- 172.21.15.81:0/2892639726 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7fd198097520 con 0x7fd1a80b35b0 2024-06-29T15:57:26.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.472+0000 7fd1aca00640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd1940763f0 0x7fd194078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.473+0000 7fd1aca00640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd1940763f0 0x7fd194078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4005fd0 tx=0x7fd1a4009500 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.476+0000 7fd19d7fa640 1 -- 172.21.15.81:0/2892639726 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd1a815d610 con 0x7fd1a80b35b0 2024-06-29T15:57:26.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.653+0000 7fd1aec64640 1 -- 172.21.15.81:0/2892639726 --> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd1a8068f50 con 0x7fd1940763f0 2024-06-29T15:57:26.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.658+0000 7fd19d7fa640 1 -- 172.21.15.81:0/2892639726 <== mgr.8666 v2:172.21.15.167:6832/2018755589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252496 (secure 0 0 0) 0x7fd1a8068f50 con 0x7fd1940763f0 2024-06-29T15:57:26.661 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:26.663 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:57:26.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 -- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd1940763f0 msgr2=0x7fd194078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd1940763f0 0x7fd194078900 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4005fd0 tx=0x7fd1a4009500 comp rx=0 tx=0).stop 2024-06-29T15:57:26.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 -- 172.21.15.81:0/2892639726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1a80b35b0 msgr2=0x7fd1a813f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1a80b35b0 0x7fd1a813f360 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fd19800c730 tx=0x7fd19800cbf0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 -- 172.21.15.81:0/2892639726 shutdown_connections 2024-06-29T15:57:26.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd1940763f0 0x7fd194078900 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1a80b4430 0x7fd1a81f6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1a80b35b0 0x7fd1a813f360 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 --2- 172.21.15.81:0/2892639726 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1a806f660 0x7fd1a80a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 -- 172.21.15.81:0/2892639726 >> 172.21.15.81:0/2892639726 conn(0x7fd1a806ccc0 msgr2=0x7fd1a8072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:26.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 -- 172.21.15.81:0/2892639726 shutdown_connections 2024-06-29T15:57:26.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.661+0000 7fd1aec64640 1 -- 172.21.15.81:0/2892639726 wait complete. 2024-06-29T15:57:26.673 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:57:25.869761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":398,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":258,"ondisk_log_size":258,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":408988,"kb_used_data":22096,"kb_used_omap":12,"kb_used_meta":386739,"kb_avail":754565732,"statfs":{"total":773094113280,"available":772675309568,"internally_reserved":0,"allocated":22626304,"data_stored":11628580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":396021298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.607342"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301243+0000","last_change":"2024-06-29T15:57:22.301019+0000","last_active":"2024-06-29T15:57:23.301243+0000","last_peered":"2024-06-29T15:57:23.301243+0000","last_clean":"2024-06-29T15:57:23.301243+0000","last_became_active":"2024-06-29T15:57:22.300506+0000","last_became_peered":"2024-06-29T15:57:22.300506+0000","last_unstale":"2024-06-29T15:57:23.301243+0000","last_undegraded":"2024-06-29T15:57:23.301243+0000","last_fullsized":"2024-06-29T15:57:23.301243+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296998+0000","last_change":"2024-06-29T15:57:22.297971+0000","last_active":"2024-06-29T15:57:23.296998+0000","last_peered":"2024-06-29T15:57:23.296998+0000","last_clean":"2024-06-29T15:57:23.296998+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:23.296998+0000","last_undegraded":"2024-06-29T15:57:23.296998+0000","last_fullsized":"2024-06-29T15:57:23.296998+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301936+0000","last_change":"2024-06-29T15:57:22.300745+0000","last_active":"2024-06-29T15:57:23.301936+0000","last_peered":"2024-06-29T15:57:23.301936+0000","last_clean":"2024-06-29T15:57:23.301936+0000","last_became_active":"2024-06-29T15:57:22.300414+0000","last_became_peered":"2024-06-29T15:57:22.300414+0000","last_unstale":"2024-06-29T15:57:23.301936+0000","last_undegraded":"2024-06-29T15:57:23.301936+0000","last_fullsized":"2024-06-29T15:57:23.301936+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301238+0000","last_change":"2024-06-29T15:57:22.298374+0000","last_active":"2024-06-29T15:57:23.301238+0000","last_peered":"2024-06-29T15:57:23.301238+0000","last_clean":"2024-06-29T15:57:23.301238+0000","last_became_active":"2024-06-29T15:57:22.297962+0000","last_became_peered":"2024-06-29T15:57:22.297962+0000","last_unstale":"2024-06-29T15:57:23.301238+0000","last_undegraded":"2024-06-29T15:57:23.301238+0000","last_fullsized":"2024-06-29T15:57:23.301238+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:13.834244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.299314+0000","last_change":"2024-06-29T15:57:22.290909+0000","last_active":"2024-06-29T15:57:23.299314+0000","last_peered":"2024-06-29T15:57:23.299314+0000","last_clean":"2024-06-29T15:57:23.299314+0000","last_became_active":"2024-06-29T15:57:22.290643+0000","last_became_peered":"2024-06-29T15:57:22.290643+0000","last_unstale":"2024-06-29T15:57:23.299314+0000","last_undegraded":"2024-06-29T15:57:23.299314+0000","last_fullsized":"2024-06-29T15:57:23.299314+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298581+0000","last_change":"2024-06-29T15:57:22.288041+0000","last_active":"2024-06-29T15:57:23.298581+0000","last_peered":"2024-06-29T15:57:23.298581+0000","last_clean":"2024-06-29T15:57:23.298581+0000","last_became_active":"2024-06-29T15:57:22.287729+0000","last_became_peered":"2024-06-29T15:57:22.287729+0000","last_unstale":"2024-06-29T15:57:23.298581+0000","last_undegraded":"2024-06-29T15:57:23.298581+0000","last_fullsized":"2024-06-29T15:57:23.298581+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300700+0000","last_change":"2024-06-29T15:57:22.304052+0000","last_active":"2024-06-29T15:57:23.300700+0000","last_peered":"2024-06-29T15:57:23.300700+0000","last_clean":"2024-06-29T15:57:23.300700+0000","last_became_active":"2024-06-29T15:57:22.303598+0000","last_became_peered":"2024-06-29T15:57:22.303598+0000","last_unstale":"2024-06-29T15:57:23.300700+0000","last_undegraded":"2024-06-29T15:57:23.300700+0000","last_fullsized":"2024-06-29T15:57:23.300700+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298325+0000","last_change":"2024-06-29T15:57:22.297617+0000","last_active":"2024-06-29T15:57:23.298325+0000","last_peered":"2024-06-29T15:57:23.298325+0000","last_clean":"2024-06-29T15:57:23.298325+0000","last_became_active":"2024-06-29T15:57:22.297256+0000","last_became_peered":"2024-06-29T15:57:22.297256+0000","last_unstale":"2024-06-29T15:57:23.298325+0000","last_undegraded":"2024-06-29T15:57:23.298325+0000","last_fullsized":"2024-06-29T15:57:23.298325+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.297058+0000","last_change":"2024-06-29T15:57:22.291096+0000","last_active":"2024-06-29T15:57:23.297058+0000","last_peered":"2024-06-29T15:57:23.297058+0000","last_clean":"2024-06-29T15:57:23.297058+0000","last_became_active":"2024-06-29T15:57:22.290428+0000","last_became_peered":"2024-06-29T15:57:22.290428+0000","last_unstale":"2024-06-29T15:57:23.297058+0000","last_undegraded":"2024-06-29T15:57:23.297058+0000","last_fullsized":"2024-06-29T15:57:23.297058+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:20.813793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302381+0000","last_change":"2024-06-29T15:57:22.296441+0000","last_active":"2024-06-29T15:57:23.302381+0000","last_peered":"2024-06-29T15:57:23.302381+0000","last_clean":"2024-06-29T15:57:23.302381+0000","last_became_active":"2024-06-29T15:57:22.296146+0000","last_became_peered":"2024-06-29T15:57:22.296146+0000","last_unstale":"2024-06-29T15:57:23.302381+0000","last_undegraded":"2024-06-29T15:57:23.302381+0000","last_fullsized":"2024-06-29T15:57:23.302381+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:54.360521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302086+0000","last_change":"2024-06-29T15:57:22.298781+0000","last_active":"2024-06-29T15:57:23.302086+0000","last_peered":"2024-06-29T15:57:23.302086+0000","last_clean":"2024-06-29T15:57:23.302086+0000","last_became_active":"2024-06-29T15:57:22.298300+0000","last_became_peered":"2024-06-29T15:57:22.298300+0000","last_unstale":"2024-06-29T15:57:23.302086+0000","last_undegraded":"2024-06-29T15:57:23.302086+0000","last_fullsized":"2024-06-29T15:57:23.302086+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295597+0000","last_change":"2024-06-29T15:57:22.298612+0000","last_active":"2024-06-29T15:57:23.295597+0000","last_peered":"2024-06-29T15:57:23.295597+0000","last_clean":"2024-06-29T15:57:23.295597+0000","last_became_active":"2024-06-29T15:57:22.298288+0000","last_became_peered":"2024-06-29T15:57:22.298288+0000","last_unstale":"2024-06-29T15:57:23.295597+0000","last_undegraded":"2024-06-29T15:57:23.295597+0000","last_fullsized":"2024-06-29T15:57:23.295597+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295805+0000","last_change":"2024-06-29T15:57:22.295681+0000","last_active":"2024-06-29T15:57:23.295805+0000","last_peered":"2024-06-29T15:57:23.295805+0000","last_clean":"2024-06-29T15:57:23.295805+0000","last_became_active":"2024-06-29T15:57:22.295357+0000","last_became_peered":"2024-06-29T15:57:22.295357+0000","last_unstale":"2024-06-29T15:57:23.295805+0000","last_undegraded":"2024-06-29T15:57:23.295805+0000","last_fullsized":"2024-06-29T15:57:23.295805+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300606+0000","last_change":"2024-06-29T15:57:22.303092+0000","last_active":"2024-06-29T15:57:23.300606+0000","last_peered":"2024-06-29T15:57:23.300606+0000","last_clean":"2024-06-29T15:57:23.300606+0000","last_became_active":"2024-06-29T15:57:22.302807+0000","last_became_peered":"2024-06-29T15:57:22.302807+0000","last_unstale":"2024-06-29T15:57:23.300606+0000","last_undegraded":"2024-06-29T15:57:23.300606+0000","last_fullsized":"2024-06-29T15:57:23.300606+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300861+0000","last_change":"2024-06-29T15:57:22.302418+0000","last_active":"2024-06-29T15:57:23.300861+0000","last_peered":"2024-06-29T15:57:23.300861+0000","last_clean":"2024-06-29T15:57:23.300861+0000","last_became_active":"2024-06-29T15:57:22.302075+0000","last_became_peered":"2024-06-29T15:57:22.302075+0000","last_unstale":"2024-06-29T15:57:23.300861+0000","last_undegraded":"2024-06-29T15:57:23.300861+0000","last_fullsized":"2024-06-29T15:57:23.300861+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302094+0000","last_change":"2024-06-29T15:57:22.290047+0000","last_active":"2024-06-29T15:57:23.302094+0000","last_peered":"2024-06-29T15:57:23.302094+0000","last_clean":"2024-06-29T15:57:23.302094+0000","last_became_active":"2024-06-29T15:57:22.289742+0000","last_became_peered":"2024-06-29T15:57:22.289742+0000","last_unstale":"2024-06-29T15:57:23.302094+0000","last_undegraded":"2024-06-29T15:57:23.302094+0000","last_fullsized":"2024-06-29T15:57:23.302094+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:03:46.271206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301400+0000","last_change":"2024-06-29T15:57:22.300787+0000","last_active":"2024-06-29T15:57:23.301400+0000","last_peered":"2024-06-29T15:57:23.301400+0000","last_clean":"2024-06-29T15:57:23.301400+0000","last_became_active":"2024-06-29T15:57:22.300112+0000","last_became_peered":"2024-06-29T15:57:22.300112+0000","last_unstale":"2024-06-29T15:57:23.301400+0000","last_undegraded":"2024-06-29T15:57:23.301400+0000","last_fullsized":"2024-06-29T15:57:23.301400+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:30.249483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.303141+0000","last_change":"2024-06-29T15:57:22.289633+0000","last_active":"2024-06-29T15:57:23.303141+0000","last_peered":"2024-06-29T15:57:23.303141+0000","last_clean":"2024-06-29T15:57:23.303141+0000","last_became_active":"2024-06-29T15:57:22.289349+0000","last_became_peered":"2024-06-29T15:57:22.289349+0000","last_unstale":"2024-06-29T15:57:23.303141+0000","last_undegraded":"2024-06-29T15:57:23.303141+0000","last_fullsized":"2024-06-29T15:57:23.303141+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.873806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302730+0000","last_change":"2024-06-29T15:57:22.306593+0000","last_active":"2024-06-29T15:57:23.302730+0000","last_peered":"2024-06-29T15:57:23.302730+0000","last_clean":"2024-06-29T15:57:23.302730+0000","last_became_active":"2024-06-29T15:57:22.296653+0000","last_became_peered":"2024-06-29T15:57:22.296653+0000","last_unstale":"2024-06-29T15:57:23.302730+0000","last_undegraded":"2024-06-29T15:57:23.302730+0000","last_fullsized":"2024-06-29T15:57:23.302730+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:41.738155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298929+0000","last_change":"2024-06-29T15:57:22.290186+0000","last_active":"2024-06-29T15:57:23.298929+0000","last_peered":"2024-06-29T15:57:23.298929+0000","last_clean":"2024-06-29T15:57:23.298929+0000","last_became_active":"2024-06-29T15:57:22.289908+0000","last_became_peered":"2024-06-29T15:57:22.289908+0000","last_unstale":"2024-06-29T15:57:23.298929+0000","last_undegraded":"2024-06-29T15:57:23.298929+0000","last_fullsized":"2024-06-29T15:57:23.298929+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300867+0000","last_change":"2024-06-29T15:57:22.307230+0000","last_active":"2024-06-29T15:57:23.300867+0000","last_peered":"2024-06-29T15:57:23.300867+0000","last_clean":"2024-06-29T15:57:23.300867+0000","last_became_active":"2024-06-29T15:57:22.306920+0000","last_became_peered":"2024-06-29T15:57:22.306920+0000","last_unstale":"2024-06-29T15:57:23.300867+0000","last_undegraded":"2024-06-29T15:57:23.300867+0000","last_fullsized":"2024-06-29T15:57:23.300867+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:22.598647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301465+0000","last_change":"2024-06-29T15:57:22.307686+0000","last_active":"2024-06-29T15:57:23.301465+0000","last_peered":"2024-06-29T15:57:23.301465+0000","last_clean":"2024-06-29T15:57:23.301465+0000","last_became_active":"2024-06-29T15:57:22.307324+0000","last_became_peered":"2024-06-29T15:57:22.307324+0000","last_unstale":"2024-06-29T15:57:23.301465+0000","last_undegraded":"2024-06-29T15:57:23.301465+0000","last_fullsized":"2024-06-29T15:57:23.301465+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301078+0000","last_change":"2024-06-29T15:57:22.301362+0000","last_active":"2024-06-29T15:57:23.301078+0000","last_peered":"2024-06-29T15:57:23.301078+0000","last_clean":"2024-06-29T15:57:23.301078+0000","last_became_active":"2024-06-29T15:57:22.300704+0000","last_became_peered":"2024-06-29T15:57:22.300704+0000","last_unstale":"2024-06-29T15:57:23.301078+0000","last_undegraded":"2024-06-29T15:57:23.301078+0000","last_fullsized":"2024-06-29T15:57:23.301078+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301716+0000","last_change":"2024-06-29T15:57:22.301601+0000","last_active":"2024-06-29T15:57:23.301716+0000","last_peered":"2024-06-29T15:57:23.301716+0000","last_clean":"2024-06-29T15:57:23.301716+0000","last_became_active":"2024-06-29T15:57:22.301244+0000","last_became_peered":"2024-06-29T15:57:22.301244+0000","last_unstale":"2024-06-29T15:57:23.301716+0000","last_undegraded":"2024-06-29T15:57:23.301716+0000","last_fullsized":"2024-06-29T15:57:23.301716+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:24:56.938464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296780+0000","last_change":"2024-06-29T15:57:22.292584+0000","last_active":"2024-06-29T15:57:23.296780+0000","last_peered":"2024-06-29T15:57:23.296780+0000","last_clean":"2024-06-29T15:57:23.296780+0000","last_became_active":"2024-06-29T15:57:22.291806+0000","last_became_peered":"2024-06-29T15:57:22.291806+0000","last_unstale":"2024-06-29T15:57:23.296780+0000","last_undegraded":"2024-06-29T15:57:23.296780+0000","last_fullsized":"2024-06-29T15:57:23.296780+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301720+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:23.301720+0000","last_peered":"2024-06-29T15:57:23.301720+0000","last_clean":"2024-06-29T15:57:23.301720+0000","last_became_active":"2024-06-29T15:57:22.301541+0000","last_became_peered":"2024-06-29T15:57:22.301541+0000","last_unstale":"2024-06-29T15:57:23.301720+0000","last_undegraded":"2024-06-29T15:57:23.301720+0000","last_fullsized":"2024-06-29T15:57:23.301720+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:55.249516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295670+0000","last_change":"2024-06-29T15:57:22.298174+0000","last_active":"2024-06-29T15:57:23.295670+0000","last_peered":"2024-06-29T15:57:23.295670+0000","last_clean":"2024-06-29T15:57:23.295670+0000","last_became_active":"2024-06-29T15:57:22.297862+0000","last_became_peered":"2024-06-29T15:57:22.297862+0000","last_unstale":"2024-06-29T15:57:23.295670+0000","last_undegraded":"2024-06-29T15:57:23.295670+0000","last_fullsized":"2024-06-29T15:57:23.295670+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300944+0000","last_change":"2024-06-29T15:57:22.297917+0000","last_active":"2024-06-29T15:57:23.300944+0000","last_peered":"2024-06-29T15:57:23.300944+0000","last_clean":"2024-06-29T15:57:23.300944+0000","last_became_active":"2024-06-29T15:57:22.297638+0000","last_became_peered":"2024-06-29T15:57:22.297638+0000","last_unstale":"2024-06-29T15:57:23.300944+0000","last_undegraded":"2024-06-29T15:57:23.300944+0000","last_fullsized":"2024-06-29T15:57:23.300944+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295387+0000","last_change":"2024-06-29T15:57:22.297998+0000","last_active":"2024-06-29T15:57:23.295387+0000","last_peered":"2024-06-29T15:57:23.295387+0000","last_clean":"2024-06-29T15:57:23.295387+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:23.295387+0000","last_undegraded":"2024-06-29T15:57:23.295387+0000","last_fullsized":"2024-06-29T15:57:23.295387+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296584+0000","last_change":"2024-06-29T15:57:22.285362+0000","last_active":"2024-06-29T15:57:23.296584+0000","last_peered":"2024-06-29T15:57:23.296584+0000","last_clean":"2024-06-29T15:57:23.296584+0000","last_became_active":"2024-06-29T15:57:22.284982+0000","last_became_peered":"2024-06-29T15:57:22.284982+0000","last_unstale":"2024-06-29T15:57:23.296584+0000","last_undegraded":"2024-06-29T15:57:23.296584+0000","last_fullsized":"2024-06-29T15:57:23.296584+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302288+0000","last_change":"2024-06-29T15:57:22.292249+0000","last_active":"2024-06-29T15:57:23.302288+0000","last_peered":"2024-06-29T15:57:23.302288+0000","last_clean":"2024-06-29T15:57:23.302288+0000","last_became_active":"2024-06-29T15:57:22.291918+0000","last_became_peered":"2024-06-29T15:57:22.291918+0000","last_unstale":"2024-06-29T15:57:23.302288+0000","last_undegraded":"2024-06-29T15:57:23.302288+0000","last_fullsized":"2024-06-29T15:57:23.302288+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:57.129905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300605+0000","last_change":"2024-06-29T15:57:22.298975+0000","last_active":"2024-06-29T15:57:23.300605+0000","last_peered":"2024-06-29T15:57:23.300605+0000","last_clean":"2024-06-29T15:57:23.300605+0000","last_became_active":"2024-06-29T15:57:22.298689+0000","last_became_peered":"2024-06-29T15:57:22.298689+0000","last_unstale":"2024-06-29T15:57:23.300605+0000","last_undegraded":"2024-06-29T15:57:23.300605+0000","last_fullsized":"2024-06-29T15:57:23.300605+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.297006+0000","last_change":"2024-06-29T15:57:22.293417+0000","last_active":"2024-06-29T15:57:23.297006+0000","last_peered":"2024-06-29T15:57:23.297006+0000","last_clean":"2024-06-29T15:57:23.297006+0000","last_became_active":"2024-06-29T15:57:22.293083+0000","last_became_peered":"2024-06-29T15:57:22.293083+0000","last_unstale":"2024-06-29T15:57:23.297006+0000","last_undegraded":"2024-06-29T15:57:23.297006+0000","last_fullsized":"2024-06-29T15:57:23.297006+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"212'4","reported_seq":18,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295865+0000","last_change":"2024-06-29T15:57:21.276327+0000","last_active":"2024-06-29T15:57:23.295865+0000","last_peered":"2024-06-29T15:57:23.295865+0000","last_clean":"2024-06-29T15:57:23.295865+0000","last_became_active":"2024-06-29T15:57:21.276031+0000","last_became_peered":"2024-06-29T15:57:21.276031+0000","last_unstale":"2024-06-29T15:57:23.295865+0000","last_undegraded":"2024-06-29T15:57:23.295865+0000","last_fullsized":"2024-06-29T15:57:23.295865+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:23.626777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301563+0000","last_change":"2024-06-29T15:57:22.301084+0000","last_active":"2024-06-29T15:57:23.301563+0000","last_peered":"2024-06-29T15:57:23.301563+0000","last_clean":"2024-06-29T15:57:23.301563+0000","last_became_active":"2024-06-29T15:57:22.300639+0000","last_became_peered":"2024-06-29T15:57:22.300639+0000","last_unstale":"2024-06-29T15:57:23.301563+0000","last_undegraded":"2024-06-29T15:57:23.301563+0000","last_fullsized":"2024-06-29T15:57:23.301563+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.297104+0000","last_change":"2024-06-29T15:57:21.274833+0000","last_active":"2024-06-29T15:57:23.297104+0000","last_peered":"2024-06-29T15:57:23.297104+0000","last_clean":"2024-06-29T15:57:23.297104+0000","last_became_active":"2024-06-29T15:57:21.274451+0000","last_became_peered":"2024-06-29T15:57:21.274451+0000","last_unstale":"2024-06-29T15:57:23.297104+0000","last_undegraded":"2024-06-29T15:57:23.297104+0000","last_fullsized":"2024-06-29T15:57:23.297104+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:25.284538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301062+0000","last_change":"2024-06-29T15:57:22.300202+0000","last_active":"2024-06-29T15:57:23.301062+0000","last_peered":"2024-06-29T15:57:23.301062+0000","last_clean":"2024-06-29T15:57:23.301062+0000","last_became_active":"2024-06-29T15:57:22.299897+0000","last_became_peered":"2024-06-29T15:57:22.299897+0000","last_unstale":"2024-06-29T15:57:23.301062+0000","last_undegraded":"2024-06-29T15:57:23.301062+0000","last_fullsized":"2024-06-29T15:57:23.301062+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302071+0000","last_change":"2024-06-29T15:57:21.284516+0000","last_active":"2024-06-29T15:57:23.302071+0000","last_peered":"2024-06-29T15:57:23.302071+0000","last_clean":"2024-06-29T15:57:23.302071+0000","last_became_active":"2024-06-29T15:57:21.284170+0000","last_became_peered":"2024-06-29T15:57:21.284170+0000","last_unstale":"2024-06-29T15:57:23.302071+0000","last_undegraded":"2024-06-29T15:57:23.302071+0000","last_fullsized":"2024-06-29T15:57:23.302071+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302987+0000","last_change":"2024-06-29T15:57:22.306835+0000","last_active":"2024-06-29T15:57:23.302987+0000","last_peered":"2024-06-29T15:57:23.302987+0000","last_clean":"2024-06-29T15:57:23.302987+0000","last_became_active":"2024-06-29T15:57:22.306295+0000","last_became_peered":"2024-06-29T15:57:22.306295+0000","last_unstale":"2024-06-29T15:57:23.302987+0000","last_undegraded":"2024-06-29T15:57:23.302987+0000","last_fullsized":"2024-06-29T15:57:23.302987+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:52.124235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301387+0000","last_change":"2024-06-29T15:57:21.283411+0000","last_active":"2024-06-29T15:57:23.301387+0000","last_peered":"2024-06-29T15:57:23.301387+0000","last_clean":"2024-06-29T15:57:23.301387+0000","last_became_active":"2024-06-29T15:57:21.283081+0000","last_became_peered":"2024-06-29T15:57:21.283081+0000","last_unstale":"2024-06-29T15:57:23.301387+0000","last_undegraded":"2024-06-29T15:57:23.301387+0000","last_fullsized":"2024-06-29T15:57:23.301387+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:27.866266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301740+0000","last_change":"2024-06-29T15:57:22.297633+0000","last_active":"2024-06-29T15:57:23.301740+0000","last_peered":"2024-06-29T15:57:23.301740+0000","last_clean":"2024-06-29T15:57:23.301740+0000","last_became_active":"2024-06-29T15:57:22.297319+0000","last_became_peered":"2024-06-29T15:57:22.297319+0000","last_unstale":"2024-06-29T15:57:23.301740+0000","last_undegraded":"2024-06-29T15:57:23.301740+0000","last_fullsized":"2024-06-29T15:57:23.301740+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:19.625176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298911+0000","last_change":"2024-06-29T15:57:21.274306+0000","last_active":"2024-06-29T15:57:23.298911+0000","last_peered":"2024-06-29T15:57:23.298911+0000","last_clean":"2024-06-29T15:57:23.298911+0000","last_became_active":"2024-06-29T15:57:21.273632+0000","last_became_peered":"2024-06-29T15:57:21.273632+0000","last_unstale":"2024-06-29T15:57:23.298911+0000","last_undegraded":"2024-06-29T15:57:23.298911+0000","last_fullsized":"2024-06-29T15:57:23.298911+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298413+0000","last_change":"2024-06-29T15:57:22.293696+0000","last_active":"2024-06-29T15:57:23.298413+0000","last_peered":"2024-06-29T15:57:23.298413+0000","last_clean":"2024-06-29T15:57:23.298413+0000","last_became_active":"2024-06-29T15:57:22.293263+0000","last_became_peered":"2024-06-29T15:57:22.293263+0000","last_unstale":"2024-06-29T15:57:23.298413+0000","last_undegraded":"2024-06-29T15:57:23.298413+0000","last_fullsized":"2024-06-29T15:57:23.298413+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298286+0000","last_change":"2024-06-29T15:57:21.277884+0000","last_active":"2024-06-29T15:57:23.298286+0000","last_peered":"2024-06-29T15:57:23.298286+0000","last_clean":"2024-06-29T15:57:23.298286+0000","last_became_active":"2024-06-29T15:57:21.277589+0000","last_became_peered":"2024-06-29T15:57:21.277589+0000","last_unstale":"2024-06-29T15:57:23.298286+0000","last_undegraded":"2024-06-29T15:57:23.298286+0000","last_fullsized":"2024-06-29T15:57:23.298286+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:09.017396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300957+0000","last_change":"2024-06-29T15:57:22.299457+0000","last_active":"2024-06-29T15:57:23.300957+0000","last_peered":"2024-06-29T15:57:23.300957+0000","last_clean":"2024-06-29T15:57:23.300957+0000","last_became_active":"2024-06-29T15:57:22.299157+0000","last_became_peered":"2024-06-29T15:57:22.299157+0000","last_unstale":"2024-06-29T15:57:23.300957+0000","last_undegraded":"2024-06-29T15:57:23.300957+0000","last_fullsized":"2024-06-29T15:57:23.300957+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:40.967292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300567+0000","last_change":"2024-06-29T15:57:21.277245+0000","last_active":"2024-06-29T15:57:23.300567+0000","last_peered":"2024-06-29T15:57:23.300567+0000","last_clean":"2024-06-29T15:57:23.300567+0000","last_became_active":"2024-06-29T15:57:21.276931+0000","last_became_peered":"2024-06-29T15:57:21.276931+0000","last_unstale":"2024-06-29T15:57:23.300567+0000","last_undegraded":"2024-06-29T15:57:23.300567+0000","last_fullsized":"2024-06-29T15:57:23.300567+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301386+0000","last_change":"2024-06-29T15:57:22.303487+0000","last_active":"2024-06-29T15:57:23.301386+0000","last_peered":"2024-06-29T15:57:23.301386+0000","last_clean":"2024-06-29T15:57:23.301386+0000","last_became_active":"2024-06-29T15:57:22.302933+0000","last_became_peered":"2024-06-29T15:57:22.302933+0000","last_unstale":"2024-06-29T15:57:23.301386+0000","last_undegraded":"2024-06-29T15:57:23.301386+0000","last_fullsized":"2024-06-29T15:57:23.301386+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:28.493187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.299107+0000","last_change":"2024-06-29T15:57:21.272019+0000","last_active":"2024-06-29T15:57:23.299107+0000","last_peered":"2024-06-29T15:57:23.299107+0000","last_clean":"2024-06-29T15:57:23.299107+0000","last_became_active":"2024-06-29T15:57:21.271715+0000","last_became_peered":"2024-06-29T15:57:21.271715+0000","last_unstale":"2024-06-29T15:57:23.299107+0000","last_undegraded":"2024-06-29T15:57:23.299107+0000","last_fullsized":"2024-06-29T15:57:23.299107+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301851+0000","last_change":"2024-06-29T15:57:22.298454+0000","last_active":"2024-06-29T15:57:23.301851+0000","last_peered":"2024-06-29T15:57:23.301851+0000","last_clean":"2024-06-29T15:57:23.301851+0000","last_became_active":"2024-06-29T15:57:22.298162+0000","last_became_peered":"2024-06-29T15:57:22.298162+0000","last_unstale":"2024-06-29T15:57:23.301851+0000","last_undegraded":"2024-06-29T15:57:23.301851+0000","last_fullsized":"2024-06-29T15:57:23.301851+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301313+0000","last_change":"2024-06-29T15:57:21.275926+0000","last_active":"2024-06-29T15:57:23.301313+0000","last_peered":"2024-06-29T15:57:23.301313+0000","last_clean":"2024-06-29T15:57:23.301313+0000","last_became_active":"2024-06-29T15:57:21.275593+0000","last_became_peered":"2024-06-29T15:57:21.275593+0000","last_unstale":"2024-06-29T15:57:23.301313+0000","last_undegraded":"2024-06-29T15:57:23.301313+0000","last_fullsized":"2024-06-29T15:57:23.301313+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301303+0000","last_change":"2024-06-29T15:57:22.295260+0000","last_active":"2024-06-29T15:57:23.301303+0000","last_peered":"2024-06-29T15:57:23.301303+0000","last_clean":"2024-06-29T15:57:23.301303+0000","last_became_active":"2024-06-29T15:57:22.294952+0000","last_became_peered":"2024-06-29T15:57:22.294952+0000","last_unstale":"2024-06-29T15:57:23.301303+0000","last_undegraded":"2024-06-29T15:57:23.301303+0000","last_fullsized":"2024-06-29T15:57:23.301303+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:56.473182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"212'3","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302839+0000","last_change":"2024-06-29T15:57:21.283791+0000","last_active":"2024-06-29T15:57:23.302839+0000","last_peered":"2024-06-29T15:57:23.302839+0000","last_clean":"2024-06-29T15:57:23.302839+0000","last_became_active":"2024-06-29T15:57:21.283458+0000","last_became_peered":"2024-06-29T15:57:21.283458+0000","last_unstale":"2024-06-29T15:57:23.302839+0000","last_undegraded":"2024-06-29T15:57:23.302839+0000","last_fullsized":"2024-06-29T15:57:23.302839+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:54.365881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301313+0000","last_change":"2024-06-29T15:57:22.302666+0000","last_active":"2024-06-29T15:57:23.301313+0000","last_peered":"2024-06-29T15:57:23.301313+0000","last_clean":"2024-06-29T15:57:23.301313+0000","last_became_active":"2024-06-29T15:57:22.302254+0000","last_became_peered":"2024-06-29T15:57:22.302254+0000","last_unstale":"2024-06-29T15:57:23.301313+0000","last_undegraded":"2024-06-29T15:57:23.301313+0000","last_fullsized":"2024-06-29T15:57:23.301313+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"212'6","reported_seq":20,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301489+0000","last_change":"2024-06-29T15:57:21.278161+0000","last_active":"2024-06-29T15:57:23.301489+0000","last_peered":"2024-06-29T15:57:23.301489+0000","last_clean":"2024-06-29T15:57:23.301489+0000","last_became_active":"2024-06-29T15:57:21.277794+0000","last_became_peered":"2024-06-29T15:57:21.277794+0000","last_unstale":"2024-06-29T15:57:23.301489+0000","last_undegraded":"2024-06-29T15:57:23.301489+0000","last_fullsized":"2024-06-29T15:57:23.301489+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:56.330168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298754+0000","last_change":"2024-06-29T15:57:22.291164+0000","last_active":"2024-06-29T15:57:23.298754+0000","last_peered":"2024-06-29T15:57:23.298754+0000","last_clean":"2024-06-29T15:57:23.298754+0000","last_became_active":"2024-06-29T15:57:22.290897+0000","last_became_peered":"2024-06-29T15:57:22.290897+0000","last_unstale":"2024-06-29T15:57:23.298754+0000","last_undegraded":"2024-06-29T15:57:23.298754+0000","last_fullsized":"2024-06-29T15:57:23.298754+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301727+0000","last_change":"2024-06-29T15:57:21.281309+0000","last_active":"2024-06-29T15:57:23.301727+0000","last_peered":"2024-06-29T15:57:23.301727+0000","last_clean":"2024-06-29T15:57:23.301727+0000","last_became_active":"2024-06-29T15:57:21.280944+0000","last_became_peered":"2024-06-29T15:57:21.280944+0000","last_unstale":"2024-06-29T15:57:23.301727+0000","last_undegraded":"2024-06-29T15:57:23.301727+0000","last_fullsized":"2024-06-29T15:57:23.301727+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.303325+0000","last_change":"2024-06-29T15:57:22.294011+0000","last_active":"2024-06-29T15:57:23.303325+0000","last_peered":"2024-06-29T15:57:23.303325+0000","last_clean":"2024-06-29T15:57:23.303325+0000","last_became_active":"2024-06-29T15:57:22.293721+0000","last_became_peered":"2024-06-29T15:57:22.293721+0000","last_unstale":"2024-06-29T15:57:23.303325+0000","last_undegraded":"2024-06-29T15:57:23.303325+0000","last_fullsized":"2024-06-29T15:57:23.303325+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:57.337708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301020+0000","last_change":"2024-06-29T15:57:21.282872+0000","last_active":"2024-06-29T15:57:23.301020+0000","last_peered":"2024-06-29T15:57:23.301020+0000","last_clean":"2024-06-29T15:57:23.301020+0000","last_became_active":"2024-06-29T15:57:21.282549+0000","last_became_peered":"2024-06-29T15:57:21.282549+0000","last_unstale":"2024-06-29T15:57:23.301020+0000","last_undegraded":"2024-06-29T15:57:23.301020+0000","last_fullsized":"2024-06-29T15:57:23.301020+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301202+0000","last_change":"2024-06-29T15:57:22.299631+0000","last_active":"2024-06-29T15:57:23.301202+0000","last_peered":"2024-06-29T15:57:23.301202+0000","last_clean":"2024-06-29T15:57:23.301202+0000","last_became_active":"2024-06-29T15:57:22.299285+0000","last_became_peered":"2024-06-29T15:57:22.299285+0000","last_unstale":"2024-06-29T15:57:23.301202+0000","last_undegraded":"2024-06-29T15:57:23.301202+0000","last_fullsized":"2024-06-29T15:57:23.301202+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302228+0000","last_change":"2024-06-29T15:57:21.276731+0000","last_active":"2024-06-29T15:57:23.302228+0000","last_peered":"2024-06-29T15:57:23.302228+0000","last_clean":"2024-06-29T15:57:23.302228+0000","last_became_active":"2024-06-29T15:57:21.276408+0000","last_became_peered":"2024-06-29T15:57:21.276408+0000","last_unstale":"2024-06-29T15:57:23.302228+0000","last_undegraded":"2024-06-29T15:57:23.302228+0000","last_fullsized":"2024-06-29T15:57:23.302228+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:48.911239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300987+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:23.300987+0000","last_peered":"2024-06-29T15:57:23.300987+0000","last_clean":"2024-06-29T15:57:23.300987+0000","last_became_active":"2024-06-29T15:57:22.301832+0000","last_became_peered":"2024-06-29T15:57:22.301832+0000","last_unstale":"2024-06-29T15:57:23.300987+0000","last_undegraded":"2024-06-29T15:57:23.300987+0000","last_fullsized":"2024-06-29T15:57:23.300987+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"212'8","reported_seq":22,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301444+0000","last_change":"2024-06-29T15:57:21.283745+0000","last_active":"2024-06-29T15:57:23.301444+0000","last_peered":"2024-06-29T15:57:23.301444+0000","last_clean":"2024-06-29T15:57:23.301444+0000","last_became_active":"2024-06-29T15:57:21.283403+0000","last_became_peered":"2024-06-29T15:57:21.283403+0000","last_unstale":"2024-06-29T15:57:23.301444+0000","last_undegraded":"2024-06-29T15:57:23.301444+0000","last_fullsized":"2024-06-29T15:57:23.301444+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.297167+0000","last_change":"2024-06-29T15:57:22.300916+0000","last_active":"2024-06-29T15:57:23.297167+0000","last_peered":"2024-06-29T15:57:23.297167+0000","last_clean":"2024-06-29T15:57:23.297167+0000","last_became_active":"2024-06-29T15:57:22.300620+0000","last_became_peered":"2024-06-29T15:57:22.300620+0000","last_unstale":"2024-06-29T15:57:23.297167+0000","last_undegraded":"2024-06-29T15:57:23.297167+0000","last_fullsized":"2024-06-29T15:57:23.297167+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:41.172455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.299177+0000","last_change":"2024-06-29T15:57:21.273304+0000","last_active":"2024-06-29T15:57:23.299177+0000","last_peered":"2024-06-29T15:57:23.299177+0000","last_clean":"2024-06-29T15:57:23.299177+0000","last_became_active":"2024-06-29T15:57:21.272972+0000","last_became_peered":"2024-06-29T15:57:21.272972+0000","last_unstale":"2024-06-29T15:57:23.299177+0000","last_undegraded":"2024-06-29T15:57:23.299177+0000","last_fullsized":"2024-06-29T15:57:23.299177+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296846+0000","last_change":"2024-06-29T15:57:22.291360+0000","last_active":"2024-06-29T15:57:23.296846+0000","last_peered":"2024-06-29T15:57:23.296846+0000","last_clean":"2024-06-29T15:57:23.296846+0000","last_became_active":"2024-06-29T15:57:22.290801+0000","last_became_peered":"2024-06-29T15:57:22.290801+0000","last_unstale":"2024-06-29T15:57:23.296846+0000","last_undegraded":"2024-06-29T15:57:23.296846+0000","last_fullsized":"2024-06-29T15:57:23.296846+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.297197+0000","last_change":"2024-06-29T15:57:21.278476+0000","last_active":"2024-06-29T15:57:23.297197+0000","last_peered":"2024-06-29T15:57:23.297197+0000","last_clean":"2024-06-29T15:57:23.297197+0000","last_became_active":"2024-06-29T15:57:21.278150+0000","last_became_peered":"2024-06-29T15:57:21.278150+0000","last_unstale":"2024-06-29T15:57:23.297197+0000","last_undegraded":"2024-06-29T15:57:23.297197+0000","last_fullsized":"2024-06-29T15:57:23.297197+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:33.991332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301102+0000","last_change":"2024-06-29T15:57:22.297275+0000","last_active":"2024-06-29T15:57:23.301102+0000","last_peered":"2024-06-29T15:57:23.301102+0000","last_clean":"2024-06-29T15:57:23.301102+0000","last_became_active":"2024-06-29T15:57:22.296984+0000","last_became_peered":"2024-06-29T15:57:22.296984+0000","last_unstale":"2024-06-29T15:57:23.301102+0000","last_undegraded":"2024-06-29T15:57:23.301102+0000","last_fullsized":"2024-06-29T15:57:23.301102+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302945+0000","last_change":"2024-06-29T15:57:21.279139+0000","last_active":"2024-06-29T15:57:23.302945+0000","last_peered":"2024-06-29T15:57:23.302945+0000","last_clean":"2024-06-29T15:57:23.302945+0000","last_became_active":"2024-06-29T15:57:21.278488+0000","last_became_peered":"2024-06-29T15:57:21.278488+0000","last_unstale":"2024-06-29T15:57:23.302945+0000","last_undegraded":"2024-06-29T15:57:23.302945+0000","last_fullsized":"2024-06-29T15:57:23.302945+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:54.126338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302570+0000","last_change":"2024-06-29T15:57:22.305806+0000","last_active":"2024-06-29T15:57:23.302570+0000","last_peered":"2024-06-29T15:57:23.302570+0000","last_clean":"2024-06-29T15:57:23.302570+0000","last_became_active":"2024-06-29T15:57:22.296276+0000","last_became_peered":"2024-06-29T15:57:22.296276+0000","last_unstale":"2024-06-29T15:57:23.302570+0000","last_undegraded":"2024-06-29T15:57:23.302570+0000","last_fullsized":"2024-06-29T15:57:23.302570+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:43:51.754223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301194+0000","last_change":"2024-06-29T15:57:21.282111+0000","last_active":"2024-06-29T15:57:23.301194+0000","last_peered":"2024-06-29T15:57:23.301194+0000","last_clean":"2024-06-29T15:57:23.301194+0000","last_became_active":"2024-06-29T15:57:21.281761+0000","last_became_peered":"2024-06-29T15:57:21.281761+0000","last_unstale":"2024-06-29T15:57:23.301194+0000","last_undegraded":"2024-06-29T15:57:23.301194+0000","last_fullsized":"2024-06-29T15:57:23.301194+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302432+0000","last_change":"2024-06-29T15:57:22.288958+0000","last_active":"2024-06-29T15:57:23.302432+0000","last_peered":"2024-06-29T15:57:23.302432+0000","last_clean":"2024-06-29T15:57:23.302432+0000","last_became_active":"2024-06-29T15:57:22.288621+0000","last_became_peered":"2024-06-29T15:57:22.288621+0000","last_unstale":"2024-06-29T15:57:23.302432+0000","last_undegraded":"2024-06-29T15:57:23.302432+0000","last_fullsized":"2024-06-29T15:57:23.302432+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:31.720751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.303301+0000","last_change":"2024-06-29T15:57:21.284009+0000","last_active":"2024-06-29T15:57:23.303301+0000","last_peered":"2024-06-29T15:57:23.303301+0000","last_clean":"2024-06-29T15:57:23.303301+0000","last_became_active":"2024-06-29T15:57:21.283679+0000","last_became_peered":"2024-06-29T15:57:21.283679+0000","last_unstale":"2024-06-29T15:57:23.303301+0000","last_undegraded":"2024-06-29T15:57:23.303301+0000","last_fullsized":"2024-06-29T15:57:23.303301+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:20.284468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.302233+0000","last_change":"2024-06-29T15:57:22.291304+0000","last_active":"2024-06-29T15:57:23.302233+0000","last_peered":"2024-06-29T15:57:23.302233+0000","last_clean":"2024-06-29T15:57:23.302233+0000","last_became_active":"2024-06-29T15:57:22.290983+0000","last_became_peered":"2024-06-29T15:57:22.290983+0000","last_unstale":"2024-06-29T15:57:23.302233+0000","last_undegraded":"2024-06-29T15:57:23.302233+0000","last_fullsized":"2024-06-29T15:57:23.302233+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301512+0000","last_change":"2024-06-29T15:57:21.284780+0000","last_active":"2024-06-29T15:57:23.301512+0000","last_peered":"2024-06-29T15:57:23.301512+0000","last_clean":"2024-06-29T15:57:23.301512+0000","last_became_active":"2024-06-29T15:57:21.284273+0000","last_became_peered":"2024-06-29T15:57:21.284273+0000","last_unstale":"2024-06-29T15:57:23.301512+0000","last_undegraded":"2024-06-29T15:57:23.301512+0000","last_fullsized":"2024-06-29T15:57:23.301512+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:16.085990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296420+0000","last_change":"2024-06-29T15:57:22.298744+0000","last_active":"2024-06-29T15:57:23.296420+0000","last_peered":"2024-06-29T15:57:23.296420+0000","last_clean":"2024-06-29T15:57:23.296420+0000","last_became_active":"2024-06-29T15:57:22.296925+0000","last_became_peered":"2024-06-29T15:57:22.296925+0000","last_unstale":"2024-06-29T15:57:23.296420+0000","last_undegraded":"2024-06-29T15:57:23.296420+0000","last_fullsized":"2024-06-29T15:57:23.296420+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:40.578182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.300970+0000","last_change":"2024-06-29T15:57:21.284982+0000","last_active":"2024-06-29T15:57:23.300970+0000","last_peered":"2024-06-29T15:57:23.300970+0000","last_clean":"2024-06-29T15:57:23.300970+0000","last_became_active":"2024-06-29T15:57:21.284684+0000","last_became_peered":"2024-06-29T15:57:21.284684+0000","last_unstale":"2024-06-29T15:57:23.300970+0000","last_undegraded":"2024-06-29T15:57:23.300970+0000","last_fullsized":"2024-06-29T15:57:23.300970+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301567+0000","last_change":"2024-06-29T15:57:22.300220+0000","last_active":"2024-06-29T15:57:23.301567+0000","last_peered":"2024-06-29T15:57:23.301567+0000","last_clean":"2024-06-29T15:57:23.301567+0000","last_became_active":"2024-06-29T15:57:22.299880+0000","last_became_peered":"2024-06-29T15:57:22.299880+0000","last_unstale":"2024-06-29T15:57:23.301567+0000","last_undegraded":"2024-06-29T15:57:23.301567+0000","last_fullsized":"2024-06-29T15:57:23.301567+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298430+0000","last_change":"2024-06-29T15:57:21.278011+0000","last_active":"2024-06-29T15:57:23.298430+0000","last_peered":"2024-06-29T15:57:23.298430+0000","last_clean":"2024-06-29T15:57:23.298430+0000","last_became_active":"2024-06-29T15:57:21.276648+0000","last_became_peered":"2024-06-29T15:57:21.276648+0000","last_unstale":"2024-06-29T15:57:23.298430+0000","last_undegraded":"2024-06-29T15:57:23.298430+0000","last_fullsized":"2024-06-29T15:57:23.298430+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295724+0000","last_change":"2024-06-29T15:57:22.298914+0000","last_active":"2024-06-29T15:57:23.295724+0000","last_peered":"2024-06-29T15:57:23.295724+0000","last_clean":"2024-06-29T15:57:23.295724+0000","last_became_active":"2024-06-29T15:57:22.298614+0000","last_became_peered":"2024-06-29T15:57:22.298614+0000","last_unstale":"2024-06-29T15:57:23.295724+0000","last_undegraded":"2024-06-29T15:57:23.295724+0000","last_fullsized":"2024-06-29T15:57:23.295724+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"212'2","reported_seq":16,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301951+0000","last_change":"2024-06-29T15:57:21.277662+0000","last_active":"2024-06-29T15:57:23.301951+0000","last_peered":"2024-06-29T15:57:23.301951+0000","last_clean":"2024-06-29T15:57:23.301951+0000","last_became_active":"2024-06-29T15:57:21.277335+0000","last_became_peered":"2024-06-29T15:57:21.277335+0000","last_unstale":"2024-06-29T15:57:23.301951+0000","last_undegraded":"2024-06-29T15:57:23.301951+0000","last_fullsized":"2024-06-29T15:57:23.301951+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301549+0000","last_change":"2024-06-29T15:57:22.303802+0000","last_active":"2024-06-29T15:57:23.301549+0000","last_peered":"2024-06-29T15:57:23.301549+0000","last_clean":"2024-06-29T15:57:23.301549+0000","last_became_active":"2024-06-29T15:57:22.303511+0000","last_became_peered":"2024-06-29T15:57:22.303511+0000","last_unstale":"2024-06-29T15:57:23.301549+0000","last_undegraded":"2024-06-29T15:57:23.301549+0000","last_fullsized":"2024-06-29T15:57:23.301549+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:10.536799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301634+0000","last_change":"2024-06-29T15:57:21.276972+0000","last_active":"2024-06-29T15:57:23.301634+0000","last_peered":"2024-06-29T15:57:23.301634+0000","last_clean":"2024-06-29T15:57:23.301634+0000","last_became_active":"2024-06-29T15:57:21.276647+0000","last_became_peered":"2024-06-29T15:57:21.276647+0000","last_unstale":"2024-06-29T15:57:23.301634+0000","last_undegraded":"2024-06-29T15:57:23.301634+0000","last_fullsized":"2024-06-29T15:57:23.301634+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:24.871281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296727+0000","last_change":"2024-06-29T15:57:22.296472+0000","last_active":"2024-06-29T15:57:23.296727+0000","last_peered":"2024-06-29T15:57:23.296727+0000","last_clean":"2024-06-29T15:57:23.296727+0000","last_became_active":"2024-06-29T15:57:22.296130+0000","last_became_peered":"2024-06-29T15:57:22.296130+0000","last_unstale":"2024-06-29T15:57:23.296727+0000","last_undegraded":"2024-06-29T15:57:23.296727+0000","last_fullsized":"2024-06-29T15:57:23.296727+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.298687+0000","last_change":"2024-06-29T15:57:21.280224+0000","last_active":"2024-06-29T15:57:23.298687+0000","last_peered":"2024-06-29T15:57:23.298687+0000","last_clean":"2024-06-29T15:57:23.298687+0000","last_became_active":"2024-06-29T15:57:21.279899+0000","last_became_peered":"2024-06-29T15:57:21.279899+0000","last_unstale":"2024-06-29T15:57:23.298687+0000","last_undegraded":"2024-06-29T15:57:23.298687+0000","last_fullsized":"2024-06-29T15:57:23.298687+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301495+0000","last_change":"2024-06-29T15:57:22.301166+0000","last_active":"2024-06-29T15:57:23.301495+0000","last_peered":"2024-06-29T15:57:23.301495+0000","last_clean":"2024-06-29T15:57:23.301495+0000","last_became_active":"2024-06-29T15:57:22.300676+0000","last_became_peered":"2024-06-29T15:57:22.300676+0000","last_unstale":"2024-06-29T15:57:23.301495+0000","last_undegraded":"2024-06-29T15:57:23.301495+0000","last_fullsized":"2024-06-29T15:57:23.301495+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301286+0000","last_change":"2024-06-29T15:57:21.277560+0000","last_active":"2024-06-29T15:57:23.301286+0000","last_peered":"2024-06-29T15:57:23.301286+0000","last_clean":"2024-06-29T15:57:23.301286+0000","last_became_active":"2024-06-29T15:57:21.277255+0000","last_became_peered":"2024-06-29T15:57:21.277255+0000","last_unstale":"2024-06-29T15:57:23.301286+0000","last_undegraded":"2024-06-29T15:57:23.301286+0000","last_fullsized":"2024-06-29T15:57:23.301286+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301342+0000","last_change":"2024-06-29T15:57:22.304057+0000","last_active":"2024-06-29T15:57:23.301342+0000","last_peered":"2024-06-29T15:57:23.301342+0000","last_clean":"2024-06-29T15:57:23.301342+0000","last_became_active":"2024-06-29T15:57:22.303762+0000","last_became_peered":"2024-06-29T15:57:22.303762+0000","last_unstale":"2024-06-29T15:57:23.301342+0000","last_undegraded":"2024-06-29T15:57:23.301342+0000","last_fullsized":"2024-06-29T15:57:23.301342+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:40.125779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301784+0000","last_change":"2024-06-29T15:57:21.281405+0000","last_active":"2024-06-29T15:57:23.301784+0000","last_peered":"2024-06-29T15:57:23.301784+0000","last_clean":"2024-06-29T15:57:23.301784+0000","last_became_active":"2024-06-29T15:57:21.281080+0000","last_became_peered":"2024-06-29T15:57:21.281080+0000","last_unstale":"2024-06-29T15:57:23.301784+0000","last_undegraded":"2024-06-29T15:57:23.301784+0000","last_fullsized":"2024-06-29T15:57:23.301784+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:51.116695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295627+0000","last_change":"2024-06-29T15:57:22.291143+0000","last_active":"2024-06-29T15:57:23.295627+0000","last_peered":"2024-06-29T15:57:23.295627+0000","last_clean":"2024-06-29T15:57:23.295627+0000","last_became_active":"2024-06-29T15:57:22.290849+0000","last_became_peered":"2024-06-29T15:57:22.290849+0000","last_unstale":"2024-06-29T15:57:23.295627+0000","last_undegraded":"2024-06-29T15:57:23.295627+0000","last_fullsized":"2024-06-29T15:57:23.295627+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.299043+0000","last_change":"2024-06-29T15:57:21.274551+0000","last_active":"2024-06-29T15:57:23.299043+0000","last_peered":"2024-06-29T15:57:23.299043+0000","last_clean":"2024-06-29T15:57:23.299043+0000","last_became_active":"2024-06-29T15:57:21.274029+0000","last_became_peered":"2024-06-29T15:57:21.274029+0000","last_unstale":"2024-06-29T15:57:23.299043+0000","last_undegraded":"2024-06-29T15:57:23.299043+0000","last_fullsized":"2024-06-29T15:57:23.299043+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301337+0000","last_change":"2024-06-29T15:57:22.295101+0000","last_active":"2024-06-29T15:57:23.301337+0000","last_peered":"2024-06-29T15:57:23.301337+0000","last_clean":"2024-06-29T15:57:23.301337+0000","last_became_active":"2024-06-29T15:57:22.294805+0000","last_became_peered":"2024-06-29T15:57:22.294805+0000","last_unstale":"2024-06-29T15:57:23.301337+0000","last_undegraded":"2024-06-29T15:57:23.301337+0000","last_fullsized":"2024-06-29T15:57:23.301337+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301205+0000","last_change":"2024-06-29T15:57:21.274359+0000","last_active":"2024-06-29T15:57:23.301205+0000","last_peered":"2024-06-29T15:57:23.301205+0000","last_clean":"2024-06-29T15:57:23.301205+0000","last_became_active":"2024-06-29T15:57:21.274036+0000","last_became_peered":"2024-06-29T15:57:21.274036+0000","last_unstale":"2024-06-29T15:57:23.301205+0000","last_undegraded":"2024-06-29T15:57:23.301205+0000","last_fullsized":"2024-06-29T15:57:23.301205+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:02.747066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.296457+0000","last_change":"2024-06-29T15:57:22.292872+0000","last_active":"2024-06-29T15:57:23.296457+0000","last_peered":"2024-06-29T15:57:23.296457+0000","last_clean":"2024-06-29T15:57:23.296457+0000","last_became_active":"2024-06-29T15:57:22.292485+0000","last_became_peered":"2024-06-29T15:57:22.292485+0000","last_unstale":"2024-06-29T15:57:23.296457+0000","last_undegraded":"2024-06-29T15:57:23.296457+0000","last_fullsized":"2024-06-29T15:57:23.296457+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:19.369559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":14,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295273+0000","last_change":"2024-06-29T15:57:21.283495+0000","last_active":"2024-06-29T15:57:23.295273+0000","last_peered":"2024-06-29T15:57:23.295273+0000","last_clean":"2024-06-29T15:57:23.295273+0000","last_became_active":"2024-06-29T15:57:21.283125+0000","last_became_peered":"2024-06-29T15:57:21.283125+0000","last_unstale":"2024-06-29T15:57:23.295273+0000","last_undegraded":"2024-06-29T15:57:23.295273+0000","last_fullsized":"2024-06-29T15:57:23.295273+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":12,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301166+0000","last_change":"2024-06-29T15:57:22.302267+0000","last_active":"2024-06-29T15:57:23.301166+0000","last_peered":"2024-06-29T15:57:23.301166+0000","last_clean":"2024-06-29T15:57:23.301166+0000","last_became_active":"2024-06-29T15:57:22.301972+0000","last_became_peered":"2024-06-29T15:57:22.301972+0000","last_unstale":"2024-06-29T15:57:23.301166+0000","last_undegraded":"2024-06-29T15:57:23.301166+0000","last_fullsized":"2024-06-29T15:57:23.301166+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"212'1","reported_seq":15,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.295755+0000","last_change":"2024-06-29T15:57:21.282026+0000","last_active":"2024-06-29T15:57:23.295755+0000","last_peered":"2024-06-29T15:57:23.295755+0000","last_clean":"2024-06-29T15:57:23.295755+0000","last_became_active":"2024-06-29T15:57:21.281709+0000","last_became_peered":"2024-06-29T15:57:21.281709+0000","last_unstale":"2024-06-29T15:57:23.295755+0000","last_undegraded":"2024-06-29T15:57:23.295755+0000","last_fullsized":"2024-06-29T15:57:23.295755+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:54:07.641300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"202'219","reported_seq":1039,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-06-29T15:57:23.301114+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:57:23.301114+0000","last_peered":"2024-06-29T15:57:23.301114+0000","last_clean":"2024-06-29T15:57:23.301114+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:57:23.301114+0000","last_undegraded":"2024-06-29T15:57:23.301114+0000","last_fullsized":"2024-06-29T15:57:23.301114+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575985,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48280,"kb_used_data":3204,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323560,"statfs":{"total":96636764160,"available":96587325440,"internally_reserved":0,"allocated":3280896,"data_stored":1898965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42124,"kb_used_data":2616,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329716,"statfs":{"total":96636764160,"available":96593629184,"internally_reserved":0,"allocated":2678784,"data_stored":1307567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42108,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329732,"statfs":{"total":96636764160,"available":96593645568,"internally_reserved":0,"allocated":2666496,"data_stored":1301139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575985,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60180,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311660,"statfs":{"total":96636764160,"available":96575139840,"internally_reserved":0,"allocated":2695168,"data_stored":1308701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575985,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47700,"kb_used_data":2624,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324140,"statfs":{"total":96636764160,"available":96587919360,"internally_reserved":0,"allocated":2686976,"data_stored":1307167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575985,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47688,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324152,"statfs":{"total":96636764160,"available":96587931648,"internally_reserved":0,"allocated":2674688,"data_stored":1306985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575984,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60176,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311664,"statfs":{"total":96636764160,"available":96575143936,"internally_reserved":0,"allocated":2682880,"data_stored":1307041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575985,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60732,"kb_used_data":3184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311108,"statfs":{"total":96636764160,"available":96574574592,"internally_reserved":0,"allocated":3260416,"data_stored":1891015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:57:26.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:57:26.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.833+0000 7fb5feb73640 1 -- 172.21.15.81:0/754241339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5f80b6d60 msgr2=0x7fb5f80ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.833+0000 7fb5feb73640 1 --2- 172.21.15.81:0/754241339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5f80b6d60 0x7fb5f80ce4d0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7fb5f80b2c10 tx=0x7fb5e802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.833+0000 7fb5feb73640 1 -- 172.21.15.81:0/754241339 shutdown_connections 2024-06-29T15:57:26.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.833+0000 7fb5feb73640 1 --2- 172.21.15.81:0/754241339 >> v2:172.21.15.167:3300/0 conn(0x7fb5f80cea10 0x7fb5f80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.833+0000 7fb5feb73640 1 --2- 172.21.15.81:0/754241339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5f80b6d60 0x7fb5f80ce4d0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.833+0000 7fb5feb73640 1 -- 172.21.15.81:0/754241339 >> 172.21.15.81:0/754241339 conn(0x7fb5f80d1960 msgr2=0x7fb5f80d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:26.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.833+0000 7fb5feb73640 1 -- 172.21.15.81:0/754241339 shutdown_connections 2024-06-29T15:57:26.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.833+0000 7fb5feb73640 1 -- 172.21.15.81:0/754241339 wait complete. 2024-06-29T15:57:26.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.835+0000 7fb5feb73640 1 Processor -- start 2024-06-29T15:57:26.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.835+0000 7fb5feb73640 1 -- start start 2024-06-29T15:57:26.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.836+0000 7fb5feb73640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5f80b4cf0 0x7fb5f81dfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.836+0000 7fb5feb73640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5f80b6d60 0x7fb5f81f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.836+0000 7fb5feb73640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5f80cea10 0x7fb5f80b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.836+0000 7fb5feb73640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f80cd540 con 0x7fb5f80cea10 2024-06-29T15:57:26.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.836+0000 7fb5feb73640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f80d0a10 con 0x7fb5f80b4cf0 2024-06-29T15:57:26.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.836+0000 7fb5feb73640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f80cd6c0 con 0x7fb5f80b6d60 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fd110640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5f80cea10 0x7fb5f80b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fd110640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5f80cea10 0x7fb5f80b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59868/0 (socket says 172.21.15.81:59868) 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fd110640 1 -- 172.21.15.81:0/1505584286 learned_addr learned my addr 172.21.15.81:0/1505584286 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fc90f640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5f80b4cf0 0x7fb5f81dfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5f7fff640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5f80b6d60 0x7fb5f81f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fd110640 1 -- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5f80b6d60 msgr2=0x7fb5f81f31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fd110640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5f80b6d60 0x7fb5f81f31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fd110640 1 -- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5f80b4cf0 msgr2=0x7fb5f81dfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fd110640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5f80b4cf0 0x7fb5f81dfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fd110640 1 -- 172.21.15.81:0/1505584286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f81e03e0 con 0x7fb5f80cea10 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fc90f640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5f80b4cf0 0x7fb5f81dfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:26.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5fd110640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5f80cea10 0x7fb5f80b08e0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fb5f000c390 tx=0x7fb5f000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5f57fa640 1 -- 172.21.15.81:0/1505584286 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0017070 con 0x7fb5f80cea10 2024-06-29T15:57:26.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5f57fa640 1 -- 172.21.15.81:0/1505584286 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5f00035f0 con 0x7fb5f80cea10 2024-06-29T15:57:26.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5feb73640 1 -- 172.21.15.81:0/1505584286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f81f3a70 con 0x7fb5f80cea10 2024-06-29T15:57:26.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5f57fa640 1 -- 172.21.15.81:0/1505584286 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0003a40 con 0x7fb5f80cea10 2024-06-29T15:57:26.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.837+0000 7fb5feb73640 1 -- 172.21.15.81:0/1505584286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f81f4030 con 0x7fb5f80cea10 2024-06-29T15:57:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.838+0000 7fb5feb73640 1 -- 172.21.15.81:0/1505584286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f80ce4d0 con 0x7fb5f80cea10 2024-06-29T15:57:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.839+0000 7fb5f57fa640 1 -- 172.21.15.81:0/1505584286 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fb5f001a070 con 0x7fb5f80cea10 2024-06-29T15:57:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.839+0000 7fb5f57fa640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb5e4076420 0x7fb5e4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.839+0000 7fb5f57fa640 1 -- 172.21.15.81:0/1505584286 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7fb5f0096b30 con 0x7fb5f80cea10 2024-06-29T15:57:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.840+0000 7fb5fc90f640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb5e4076420 0x7fb5e4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:26.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.840+0000 7fb5fc90f640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb5e4076420 0x7fb5e4078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8033000 tx=0x7fb5e8004730 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:26.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:26.844+0000 7fb5f57fa640 1 -- 172.21.15.81:0/1505584286 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb5f81e03e0 con 0x7fb5f80cea10 2024-06-29T15:57:27.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.094+0000 7fb5feb73640 1 -- 172.21.15.81:0/1505584286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fb5f8068b20 con 0x7fb5f80cea10 2024-06-29T15:57:27.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.097+0000 7fb5f57fa640 1 -- 172.21.15.81:0/1505584286 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v67) ==== 190+0+147 (secure 0 0 0) 0x7fb5f0060270 con 0x7fb5f80cea10 2024-06-29T15:57:27.101 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T15:57:27.101 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T15:57:27.101 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T15:57:27.102 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T15:57:27.102 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 -- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb5e4076420 msgr2=0x7fb5e4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb5e4076420 0x7fb5e4078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8033000 tx=0x7fb5e8004730 comp rx=0 tx=0).stop 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 -- 172.21.15.81:0/1505584286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5f80cea10 msgr2=0x7fb5f80b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5f80cea10 0x7fb5f80b08e0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7fb5f000c390 tx=0x7fb5f000c850 comp rx=0 tx=0).stop 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 -- 172.21.15.81:0/1505584286 shutdown_connections 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fb5e4076420 0x7fb5e4078930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5f80cea10 0x7fb5f80b08e0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5f80b6d60 0x7fb5f81f31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 --2- 172.21.15.81:0/1505584286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5f80b4cf0 0x7fb5f81dfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 -- 172.21.15.81:0/1505584286 >> 172.21.15.81:0/1505584286 conn(0x7fb5f80d1960 msgr2=0x7fb5f8070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:27.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 -- 172.21.15.81:0/1505584286 shutdown_connections 2024-06-29T15:57:27.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.100+0000 7fb5feb73640 1 -- 172.21.15.81:0/1505584286 wait complete. 2024-06-29T15:57:27.112 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:27.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.224+0000 7f1371968640 1 -- 172.21.15.81:0/1527087642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1358003a50 msgr2=0x7f1358003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.224+0000 7f1371968640 1 --2- 172.21.15.81:0/1527087642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1358003a50 0x7f1358003ec0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f135c006fa0 tx=0x7f1358005830 comp rx=0 tx=0).stop 2024-06-29T15:57:27.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.224+0000 7f1371968640 1 -- 172.21.15.81:0/1527087642 shutdown_connections 2024-06-29T15:57:27.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.224+0000 7f1371968640 1 --2- 172.21.15.81:0/1527087642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1358003a50 0x7f1358003ec0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.224+0000 7f1371968640 1 -- 172.21.15.81:0/1527087642 >> 172.21.15.81:0/1527087642 conn(0x7f136c0d1960 msgr2=0x7f136c0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:27.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.224+0000 7f1371968640 1 -- 172.21.15.81:0/1527087642 shutdown_connections 2024-06-29T15:57:27.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.224+0000 7f1371968640 1 -- 172.21.15.81:0/1527087642 wait complete. 2024-06-29T15:57:27.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.226+0000 7f1371968640 1 Processor -- start 2024-06-29T15:57:27.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.226+0000 7f1371968640 1 -- start start 2024-06-29T15:57:27.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f1371968640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1358003a50 0x7f136c0aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f1371968640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f136c0b06e0 0x7f136c0b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f1371968640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f136c0b36b0 0x7f136c0b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f1371968640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f136c0d0890 con 0x7f1358003a50 2024-06-29T15:57:27.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f1371968640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f136c0d0710 con 0x7f136c0b36b0 2024-06-29T15:57:27.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f1371968640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f136c0d0a10 con 0x7f136c0b06e0 2024-06-29T15:57:27.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1358003a50 0x7f136c0aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1358003a50 0x7f136c0aedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59878/0 (socket says 172.21.15.81:59878) 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136affd640 1 -- 172.21.15.81:0/2046585546 learned_addr learned my addr 172.21.15.81:0/2046585546 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f13627fc640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f136c0b06e0 0x7f136c0b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136b7fe640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f136c0b36b0 0x7f136c0b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136affd640 1 -- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f136c0b06e0 msgr2=0x7f136c0b0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136affd640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f136c0b06e0 0x7f136c0b0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136affd640 1 -- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f136c0b36b0 msgr2=0x7f136c0b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136affd640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f136c0b36b0 0x7f136c0b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136affd640 1 -- 172.21.15.81:0/2046585546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c0b4000 con 0x7f1358003a50 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136b7fe640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f136c0b36b0 0x7f136c0b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f136affd640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1358003a50 0x7f136c0aedd0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c320 tx=0x7f135c001d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f1368ff9640 1 -- 172.21.15.81:0/2046585546 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c032030 con 0x7f1358003a50 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f1368ff9640 1 -- 172.21.15.81:0/2046585546 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f135c003b10 con 0x7f1358003a50 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f1371968640 1 -- 172.21.15.81:0/2046585546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136c0c9dd0 con 0x7f1358003a50 2024-06-29T15:57:27.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.228+0000 7f1368ff9640 1 -- 172.21.15.81:0/2046585546 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c02e480 con 0x7f1358003a50 2024-06-29T15:57:27.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.229+0000 7f1371968640 1 -- 172.21.15.81:0/2046585546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136c0ca350 con 0x7f1358003a50 2024-06-29T15:57:27.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.229+0000 7f1371968640 1 -- 172.21.15.81:0/2046585546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f136c0ce7c0 con 0x7f1358003a50 2024-06-29T15:57:27.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.230+0000 7f1368ff9640 1 -- 172.21.15.81:0/2046585546 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f135c035020 con 0x7f1358003a50 2024-06-29T15:57:27.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.231+0000 7f1368ff9640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f134c0763f0 0x7f134c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.231+0000 7f1368ff9640 1 -- 172.21.15.81:0/2046585546 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7f135c0b7ea0 con 0x7f1358003a50 2024-06-29T15:57:27.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.231+0000 7f13627fc640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f134c0763f0 0x7f134c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.232+0000 7f13627fc640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f134c0763f0 0x7f134c078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f136c1e72a0 tx=0x7f1354006d40 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:27.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.235+0000 7f1368ff9640 1 -- 172.21.15.81:0/2046585546 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f135c047030 con 0x7f1358003a50 2024-06-29T15:57:27.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.452+0000 7f1371968640 1 -- 172.21.15.81:0/2046585546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f136c0cdec0 con 0x7f1358003a50 2024-06-29T15:57:27.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.453+0000 7f1368ff9640 1 -- 172.21.15.81:0/2046585546 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5240 (secure 0 0 0) 0x7f136c0b4000 con 0x7f1358003a50 2024-06-29T15:57:27.454 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:27.454 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":112,"btime":"2024-06-29T15:57:25:299142+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10114,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.637042+0000","modified":"2024-06-29T15:57:24.656503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11226":{"gid":11226,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/739691411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":739691411},{"type":"v1","addr":"172.21.15.81:6833","nonce":739691411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11226,"qdb_cluster":[11226]},"id":13}]} 2024-06-29T15:57:27.455 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 112 2024-06-29T15:57:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.454+0000 7f1371968640 1 -- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f134c0763f0 msgr2=0x7f134c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.454+0000 7f1371968640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f134c0763f0 0x7f134c078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f136c1e72a0 tx=0x7f1354006d40 comp rx=0 tx=0).stop 2024-06-29T15:57:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.454+0000 7f1371968640 1 -- 172.21.15.81:0/2046585546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1358003a50 msgr2=0x7f136c0aedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.454+0000 7f1371968640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1358003a50 0x7f136c0aedd0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c320 tx=0x7f135c001d60 comp rx=0 tx=0).stop 2024-06-29T15:57:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.455+0000 7f1371968640 1 -- 172.21.15.81:0/2046585546 shutdown_connections 2024-06-29T15:57:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.455+0000 7f1371968640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f134c0763f0 0x7f134c078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.455+0000 7f1371968640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f136c0b36b0 0x7f136c0b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.455+0000 7f1371968640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f136c0b06e0 0x7f136c0b0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.455+0000 7f1371968640 1 --2- 172.21.15.81:0/2046585546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1358003a50 0x7f136c0aedd0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.455+0000 7f1371968640 1 -- 172.21.15.81:0/2046585546 >> 172.21.15.81:0/2046585546 conn(0x7f136c0d1960 msgr2=0x7f136c0b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.455+0000 7f1371968640 1 -- 172.21.15.81:0/2046585546 shutdown_connections 2024-06-29T15:57:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.455+0000 7f1371968640 1 -- 172.21.15.81:0/2046585546 wait complete. 2024-06-29T15:57:27.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 112, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:57:21.637042+0000', 'modified': '2024-06-29T15:57:24.656503+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11226}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11226': {'gid': 11226, 'name': 'a', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6833/739691411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6832', 'nonce': 739691411}, {'type': 'v1', 'addr': '172.21.15.81:6833', 'nonce': 739691411}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11226, 'qdb_cluster': [11226]} 2024-06-29T15:57:27.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:57:27.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T15:57:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.985+0000 7fe807c9d640 1 -- 172.21.15.81:0/3878860640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7ec004800 msgr2=0x7fe7ec004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.985+0000 7fe807c9d640 1 --2- 172.21.15.81:0/3878860640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7ec004800 0x7fe7ec004c10 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fe8000b1d90 tx=0x7fe7ec005830 comp rx=0 tx=0).stop 2024-06-29T15:57:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.985+0000 7fe807c9d640 1 -- 172.21.15.81:0/3878860640 shutdown_connections 2024-06-29T15:57:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.985+0000 7fe807c9d640 1 --2- 172.21.15.81:0/3878860640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7ec004800 0x7fe7ec004c10 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.985+0000 7fe807c9d640 1 -- 172.21.15.81:0/3878860640 >> 172.21.15.81:0/3878860640 conn(0x7fe800072950 msgr2=0x7fe8000b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.985+0000 7fe807c9d640 1 -- 172.21.15.81:0/3878860640 shutdown_connections 2024-06-29T15:57:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.985+0000 7fe807c9d640 1 -- 172.21.15.81:0/3878860640 wait complete. 2024-06-29T15:57:27.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.987+0000 7fe807c9d640 1 Processor -- start 2024-06-29T15:57:27.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.987+0000 7fe807c9d640 1 -- start start 2024-06-29T15:57:27.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe807c9d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7ec004800 0x7fe800153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe807c9d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe800156790 0x7fe800156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe807c9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe800158790 0x7fe800158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe807c9d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe8000740a0 con 0x7fe800158790 2024-06-29T15:57:27.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe807c9d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe800073f20 con 0x7fe800156790 2024-06-29T15:57:27.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe807c9d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe800074220 con 0x7fe7ec004800 2024-06-29T15:57:27.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe80623a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe800158790 0x7fe800158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe80623a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe800158790 0x7fe800158ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59896/0 (socket says 172.21.15.81:59896) 2024-06-29T15:57:27.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe80623a640 1 -- 172.21.15.81:0/1679158491 learned_addr learned my addr 172.21.15.81:0/1679158491 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe805a39640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7ec004800 0x7fe800153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.988+0000 7fe805238640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe800156790 0x7fe800156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe80623a640 1 -- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7ec004800 msgr2=0x7fe800153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe80623a640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7ec004800 0x7fe800153d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe80623a640 1 -- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe800156790 msgr2=0x7fe800156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe80623a640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe800156790 0x7fe800156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe80623a640 1 -- 172.21.15.81:0/1679158491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe800159460 con 0x7fe800158790 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe805a39640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7ec004800 0x7fe800153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe80623a640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe800158790 0x7fe800158ba0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fe800161a90 tx=0x7fe7f8011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe7f67fc640 1 -- 172.21.15.81:0/1679158491 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8004230 con 0x7fe800158790 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe7f67fc640 1 -- 172.21.15.81:0/1679158491 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7f800f860 con 0x7fe800158790 2024-06-29T15:57:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe807c9d640 1 -- 172.21.15.81:0/1679158491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe800159740 con 0x7fe800158790 2024-06-29T15:57:27.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe7f67fc640 1 -- 172.21.15.81:0/1679158491 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80044c0 con 0x7fe800158790 2024-06-29T15:57:27.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.989+0000 7fe807c9d640 1 -- 172.21.15.81:0/1679158491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe800159d00 con 0x7fe800158790 2024-06-29T15:57:27.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.991+0000 7fe7f67fc640 1 -- 172.21.15.81:0/1679158491 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fe7f800fcf0 con 0x7fe800158790 2024-06-29T15:57:27.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.991+0000 7fe7f67fc640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe7e40763f0 0x7fe7e4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.991+0000 7fe7f67fc640 1 -- 172.21.15.81:0/1679158491 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7fe7f8098350 con 0x7fe800158790 2024-06-29T15:57:27.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.991+0000 7fe805a39640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe7e40763f0 0x7fe7e4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.992+0000 7fe805a39640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe7e40763f0 0x7fe7e4078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc004fd0 tx=0x7fe7fc006b60 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:27.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.992+0000 7fe80ec31740 1 Processor -- start 2024-06-29T15:57:27.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.992+0000 7fe80ec31740 1 -- start start 2024-06-29T15:57:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af74d46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af74b83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80ec31740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557af7450bc0 con 0x557af70c7590 2024-06-29T15:57:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80ec31740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x557af73ff630 con 0x557af70e8270 2024-06-29T15:57:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80ec31740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x557af73eebd0 con 0x557af70bf3b0 2024-06-29T15:57:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80623a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80623a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74b88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59906/0 (socket says 172.21.15.81:59906) 2024-06-29T15:57:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80623a640 1 -- 172.21.15.81:0/1759611024 learned_addr learned my addr 172.21.15.81:0/1759611024 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe805a39640 1 --2- 172.21.15.81:0/1759611024 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af74d46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe805238640 1 --2- 172.21.15.81:0/1759611024 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af74b83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80623a640 1 -- 172.21.15.81:0/1759611024 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 msgr2=0x557af74b83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80623a640 1 --2- 172.21.15.81:0/1759611024 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af74b83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80623a640 1 -- 172.21.15.81:0/1759611024 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 msgr2=0x557af74d46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80623a640 1 --2- 172.21.15.81:0/1759611024 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af74d46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80623a640 1 -- 172.21.15.81:0/1759611024 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557af73ebca0 con 0x557af70c7590 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe80623a640 1 --2- 172.21.15.81:0/1759611024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74b88e0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8096770 tx=0x7fe7f80613b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd1ef00f7fcf77a server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.994+0000 7fe7e37fe640 1 -- 172.21.15.81:0/1759611024 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80614b0 con 0x557af70c7590 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe7e37fe640 1 -- 172.21.15.81:0/1759611024 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7f8065860 con 0x557af70c7590 2024-06-29T15:57:27.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe7e37fe640 1 -- 172.21.15.81:0/1759611024 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f806a280 con 0x557af70c7590 2024-06-29T15:57:28.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe80ec31740 1 -- 172.21.15.81:0/1759611024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 msgr2=0x557af74b88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe80ec31740 1 --2- 172.21.15.81:0/1759611024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74b88e0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8096770 tx=0x7fe7f80613b0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe80ec31740 1 -- 172.21.15.81:0/1759611024 shutdown_connections 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe80ec31740 1 --2- 172.21.15.81:0/1759611024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74b88e0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe80ec31740 1 --2- 172.21.15.81:0/1759611024 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af74b83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe80ec31740 1 --2- 172.21.15.81:0/1759611024 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af74d46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe80ec31740 1 -- 172.21.15.81:0/1759611024 >> 172.21.15.81:0/1759611024 conn(0x557af70dc3c0 msgr2=0x557af748bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe80ec31740 1 -- 172.21.15.81:0/1759611024 shutdown_connections 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.995+0000 7fe80ec31740 1 -- 172.21.15.81:0/1759611024 wait complete. 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.996+0000 7fe80ec31740 1 -- start start 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.996+0000 7fe80ec31740 1 Processor -- start 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af74b8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74e2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af74e3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe80ec31740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557af738cae0 con 0x557af70c7590 2024-06-29T15:57:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe80ec31740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x557af73f2c10 con 0x557af70e8270 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe80ec31740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x557af7162620 con 0x557af70bf3b0 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe80ec31740 10 client.0 _ll_register_callbacks cb 0x557af7399480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe805238640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74e2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe805238640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74e2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59908/0 (socket says 172.21.15.81:59908) 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe805238640 1 -- 172.21.15.81:0/3276045999 learned_addr learned my addr 172.21.15.81:0/3276045999 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe805a39640 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af74b8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe80623a640 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af74e3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe805238640 1 -- 172.21.15.81:0/3276045999 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 msgr2=0x557af74b8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe805238640 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af74b8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe805238640 1 -- 172.21.15.81:0/3276045999 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 msgr2=0x557af74e3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe805238640 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af74e3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe805238640 1 -- 172.21.15.81:0/3276045999 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557af7153850 con 0x557af70c7590 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe805238640 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74e2d50 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fe7f002b0a0 tx=0x7fe7f0006c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e689e559343c9556 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe7e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.998+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/3276045999 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0032040 con 0x557af70c7590 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/3276045999 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7fe7f0006260 con 0x557af70c7590 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe80ec31740 1 -- 172.21.15.81:0/3276045999 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7f8065860 con 0x557af70c7590 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/3276045999 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fe7f004a050 con 0x557af70c7590 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe80ec31740 10 client.11370 fetch_fsmap learned FSMap version 112 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe80ec31740 1 -- 172.21.15.81:0/3276045999 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) -- 0x557af71ca6b0 con 0x557af70c7590 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/3276045999 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7f8065860 con 0x557af70c7590 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/3276045999 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 112) ==== 3502+0+0 (secure 0 0 0) 0x7fe7f004d020 con 0x557af70c7590 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe80ec31740 10 client.11370 fetch_fsmap finished waiting for FSMap version 112 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe80ec31740 10 client.11370 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe80ec31740 20 client.11370 populate_metadata read hostname 'smithi081' 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe80ec31740 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.81:6832/739691411,v1:172.21.15.81:6833/739691411] conn(0x557af70a6e40 0x557af74f04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe80ec31740 4 client.11370 mds_command: new command op to 11226 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe80ec31740 1 -- 172.21.15.81:0/3276045999 --> [v2:172.21.15.81:6832/739691411,v1:172.21.15.81:6833/739691411] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557af7401580 con 0x557af70a6e40 2024-06-29T15:57:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe805a39640 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.81:6832/739691411,v1:172.21.15.81:6833/739691411] conn(0x557af70a6e40 0x557af74f04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:27.999+0000 7fe7e0ff9640 10 client.11370 ms_handle_connect on v2:172.21.15.81:6832/739691411 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe805a39640 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.81:6832/739691411,v1:172.21.15.81:6833/739691411] conn(0x557af70a6e40 0x557af74f04d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8cf86a5dde74cde server_cookie=28352fcbd6d59fdd in_seq=0 out_seq=0 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/3276045999 <== mds.0 v2:172.21.15.81:6832/739691411 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x557af7401580 con 0x557af70a6e40 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe7e0ff9640 10 client.11370 handle_command_reply: tid=0 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe80ec31740 1 client.11370 shutdown 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe80ec31740 1 -- 172.21.15.81:0/3276045999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 msgr2=0x557af74e2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe80ec31740 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74e2d50 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7fe7f002b0a0 tx=0x7fe7f0006c70 comp rx=0 tx=0).stop 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe80ec31740 1 -- 172.21.15.81:0/3276045999 shutdown_connections 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe80ec31740 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.81:6832/739691411,v1:172.21.15.81:6833/739691411] conn(0x557af70a6e40 0x557af74f04d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe80ec31740 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af74e3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe80ec31740 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74e2d50 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe80ec31740 1 --2- 172.21.15.81:0/3276045999 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af74b8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe80ec31740 1 -- 172.21.15.81:0/3276045999 >> 172.21.15.81:0/3276045999 conn(0x557af70dc3c0 msgr2=0x557af74b4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.000+0000 7fe7e0ff9640 0 client.11370 ms_handle_reset on v2:172.21.15.81:6832/739691411 2024-06-29T15:57:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.001+0000 7fe80ec31740 1 -- 172.21.15.81:0/3276045999 shutdown_connections 2024-06-29T15:57:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.001+0000 7fe80ec31740 1 -- 172.21.15.81:0/3276045999 wait complete. 2024-06-29T15:57:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.001+0000 7fe80ec31740 20 client.11370 trim_cache size 0 max 16384 2024-06-29T15:57:28.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.020+0000 7fe80ec31740 1 Processor -- start 2024-06-29T15:57:28.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.020+0000 7fe80ec31740 1 -- start start 2024-06-29T15:57:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70bf3b0 0x557af74ca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70e8270 0x557af74b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe80ec31740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe800201f00 con 0x557af70c7590 2024-06-29T15:57:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe80ec31740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe800200f70 con 0x557af70bf3b0 2024-06-29T15:57:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe80ec31740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe80009c5d0 con 0x557af70e8270 2024-06-29T15:57:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe805238640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe805238640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59918/0 (socket says 172.21.15.81:59918) 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe805238640 1 -- 172.21.15.81:0/2767002533 learned_addr learned my addr 172.21.15.81:0/2767002533 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe805a39640 1 --2- 172.21.15.81:0/2767002533 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70bf3b0 0x557af74ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe805238640 1 -- 172.21.15.81:0/2767002533 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70e8270 msgr2=0x557af74b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe805238640 1 --2- 172.21.15.81:0/2767002533 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70e8270 0x557af74b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe805238640 1 -- 172.21.15.81:0/2767002533 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70bf3b0 msgr2=0x557af74ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe805238640 1 --2- 172.21.15.81:0/2767002533 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70bf3b0 0x557af74ca090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe805238640 1 -- 172.21.15.81:0/2767002533 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557af748f650 con 0x557af70c7590 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.021+0000 7fe805238640 1 --2- 172.21.15.81:0/2767002533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74ca5d0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0006720 tx=0x7fe7f00063f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1bf2d7db58d224a server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe7e2ffd640 1 -- 172.21.15.81:0/2767002533 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0032040 con 0x557af70c7590 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe7e2ffd640 1 -- 172.21.15.81:0/2767002533 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7f003d020 con 0x557af70c7590 2024-06-29T15:57:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe7e2ffd640 1 -- 172.21.15.81:0/2767002533 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0003850 con 0x557af70c7590 2024-06-29T15:57:28.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe80ec31740 1 -- 172.21.15.81:0/2767002533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 msgr2=0x557af74ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe80ec31740 1 --2- 172.21.15.81:0/2767002533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74ca5d0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0006720 tx=0x7fe7f00063f0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe80ec31740 1 -- 172.21.15.81:0/2767002533 shutdown_connections 2024-06-29T15:57:28.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe80ec31740 1 --2- 172.21.15.81:0/2767002533 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70e8270 0x557af74b5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe80ec31740 1 --2- 172.21.15.81:0/2767002533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74ca5d0 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe80ec31740 1 --2- 172.21.15.81:0/2767002533 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70bf3b0 0x557af74ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe80ec31740 1 -- 172.21.15.81:0/2767002533 >> 172.21.15.81:0/2767002533 conn(0x557af70dc3c0 msgr2=0x557af74b8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe80ec31740 1 -- 172.21.15.81:0/2767002533 shutdown_connections 2024-06-29T15:57:28.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.022+0000 7fe80ec31740 1 -- 172.21.15.81:0/2767002533 wait complete. 2024-06-29T15:57:28.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.023+0000 7fe80ec31740 1 -- start start 2024-06-29T15:57:28.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.023+0000 7fe80ec31740 1 Processor -- start 2024-06-29T15:57:28.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af754f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74a4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe80ec31740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af7553370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe80ec31740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557af74e37d0 con 0x557af70c7590 2024-06-29T15:57:28.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe80ec31740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe800200920 con 0x557af70e8270 2024-06-29T15:57:28.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe80ec31740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x557af74b0210 con 0x557af70bf3b0 2024-06-29T15:57:28.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe80ec31740 10 client.0 _ll_register_callbacks cb 0x7fe7ec00c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe805238640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe805238640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74a4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59930/0 (socket says 172.21.15.81:59930) 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe805238640 1 -- 172.21.15.81:0/4092856529 learned_addr learned my addr 172.21.15.81:0/4092856529 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe805a39640 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af754f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe80623a640 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af7553370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe805238640 1 -- 172.21.15.81:0/4092856529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 msgr2=0x557af754f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe805238640 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af754f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe805238640 1 -- 172.21.15.81:0/4092856529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 msgr2=0x557af7553370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe805238640 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af7553370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.025+0000 7fe805238640 1 -- 172.21.15.81:0/4092856529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557af74e34a0 con 0x557af70c7590 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe805238640 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74a4910 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0001240 tx=0x7fe7f0003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc102adfc2a024e9 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe7e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/4092856529 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0053060 con 0x557af70c7590 2024-06-29T15:57:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/4092856529 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7fe7f002e690 con 0x557af70c7590 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe80ec31740 1 -- 172.21.15.81:0/4092856529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7f003d020 con 0x557af70c7590 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/4092856529 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fe7f0059050 con 0x557af70c7590 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe80ec31740 10 client.11376 fetch_fsmap learned FSMap version 112 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe80ec31740 1 -- 172.21.15.81:0/4092856529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) -- 0x557af7553d30 con 0x557af70c7590 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/4092856529 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7f003d020 con 0x557af70c7590 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/4092856529 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 112) ==== 3502+0+0 (secure 0 0 0) 0x7fe7f004d070 con 0x557af70c7590 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe80ec31740 10 client.11376 fetch_fsmap finished waiting for FSMap version 112 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe80ec31740 10 client.11376 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe80ec31740 20 client.11376 populate_metadata read hostname 'smithi081' 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe80ec31740 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.81:6832/739691411,v1:172.21.15.81:6833/739691411] conn(0x557af748a190 0x557af7488c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe80ec31740 4 client.11376 mds_command: new command op to 11226 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe80ec31740 1 -- 172.21.15.81:0/4092856529 --> [v2:172.21.15.81:6832/739691411,v1:172.21.15.81:6833/739691411] -- command(tid 0: {"prefix": "status"}) -- 0x557af74cb260 con 0x557af748a190 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.026+0000 7fe805a39640 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.81:6832/739691411,v1:172.21.15.81:6833/739691411] conn(0x557af748a190 0x557af7488c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe7e0ff9640 10 client.11376 ms_handle_connect on v2:172.21.15.81:6832/739691411 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe805a39640 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.81:6832/739691411,v1:172.21.15.81:6833/739691411] conn(0x557af748a190 0x557af7488c70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1b8edcaa126b33c server_cookie=cac4008168edca69 in_seq=0 out_seq=0 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe7e0ff9640 1 -- 172.21.15.81:0/4092856529 <== mds.0 v2:172.21.15.81:6832/739691411 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x557af74cb260 con 0x557af748a190 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe7e0ff9640 10 client.11376 handle_command_reply: tid=0 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 client.11376 shutdown 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 -- 172.21.15.81:0/4092856529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 msgr2=0x557af74a4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74a4910 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0001240 tx=0x7fe7f0003d10 comp rx=0 tx=0).stop 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 -- 172.21.15.81:0/4092856529 shutdown_connections 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.81:6832/739691411,v1:172.21.15.81:6833/739691411] conn(0x557af748a190 0x557af7488c70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x557af70e8270 0x557af7553370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557af70c7590 0x557af74a4910 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe7e0ff9640 0 client.11376 ms_handle_reset on v2:172.21.15.81:6832/739691411 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 --2- 172.21.15.81:0/4092856529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x557af70bf3b0 0x557af754f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 -- 172.21.15.81:0/4092856529 >> 172.21.15.81:0/4092856529 conn(0x557af70dc3c0 msgr2=0x557af74ca090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 -- 172.21.15.81:0/4092856529 shutdown_connections 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 1 -- 172.21.15.81:0/4092856529 wait complete. 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.027+0000 7fe80ec31740 20 client.11376 trim_cache size 0 max 16384 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 11226, 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T15:57:28.035 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7425234999999999, 2024-06-29T15:57:28.036 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 112, 2024-06-29T15:57:28.036 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 213, 2024-06-29T15:57:28.036 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 213, 2024-06-29T15:57:28.036 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.43992527 2024-06-29T15:57:28.036 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 -- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe7e40763f0 msgr2=0x7fe7e4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe7e40763f0 0x7fe7e4078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc004fd0 tx=0x7fe7fc006b60 comp rx=0 tx=0).stop 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 -- 172.21.15.81:0/1679158491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe800158790 msgr2=0x7fe800158ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe800158790 0x7fe800158ba0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fe800161a90 tx=0x7fe7f8011b10 comp rx=0 tx=0).stop 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 -- 172.21.15.81:0/1679158491 shutdown_connections 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe7e40763f0 0x7fe7e4078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe800158790 0x7fe800158ba0 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe800156790 0x7fe800156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 --2- 172.21.15.81:0/1679158491 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe7ec004800 0x7fe800153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 -- 172.21.15.81:0/1679158491 >> 172.21.15.81:0/1679158491 conn(0x7fe800072950 msgr2=0x7fe8001604c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 -- 172.21.15.81:0/1679158491 shutdown_connections 2024-06-29T15:57:28.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.028+0000 7fe807c9d640 1 -- 172.21.15.81:0/1679158491 wait complete. 2024-06-29T15:57:28.040 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:57:28.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.151+0000 7f5e2c48f640 1 -- 172.21.15.81:0/3905580575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e241412f0 msgr2=0x7f5e240cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.151+0000 7f5e2c48f640 1 --2- 172.21.15.81:0/3905580575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e241412f0 0x7f5e240cef90 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f5e20005fd0 tx=0x7f5e2002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.152+0000 7f5e2c48f640 1 -- 172.21.15.81:0/3905580575 shutdown_connections 2024-06-29T15:57:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.152+0000 7f5e2c48f640 1 --2- 172.21.15.81:0/3905580575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e241412f0 0x7f5e240cef90 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.152+0000 7f5e2c48f640 1 -- 172.21.15.81:0/3905580575 >> 172.21.15.81:0/3905580575 conn(0x7f5e2406ccc0 msgr2=0x7f5e240b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.152+0000 7f5e2c48f640 1 -- 172.21.15.81:0/3905580575 shutdown_connections 2024-06-29T15:57:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.152+0000 7f5e2c48f640 1 -- 172.21.15.81:0/3905580575 wait complete. 2024-06-29T15:57:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.153+0000 7f5e2c48f640 1 Processor -- start 2024-06-29T15:57:28.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.153+0000 7f5e2c48f640 1 -- start start 2024-06-29T15:57:28.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2c48f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e240cf4d0 0x7f5e240ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2c48f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e241412f0 0x7f5e2413f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2c48f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2415a450 0x7f5e24071f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2c48f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5e240b3390 con 0x7f5e240cf4d0 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2c48f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5e240b3210 con 0x7f5e2415a450 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2c48f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5e240b3510 con 0x7f5e241412f0 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2a22b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e240cf4d0 0x7f5e240ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2a22b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e240cf4d0 0x7f5e240ae810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59944/0 (socket says 172.21.15.81:59944) 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2a22b640 1 -- 172.21.15.81:0/949833913 learned_addr learned my addr 172.21.15.81:0/949833913 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e29a2a640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e241412f0 0x7f5e2413f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2aa2c640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2415a450 0x7f5e24071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2a22b640 1 -- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e241412f0 msgr2=0x7f5e2413f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2a22b640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e241412f0 0x7f5e2413f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2a22b640 1 -- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2415a450 msgr2=0x7f5e24071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2a22b640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2415a450 0x7f5e24071f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2a22b640 1 -- 172.21.15.81:0/949833913 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2413f680 con 0x7f5e240cf4d0 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2a22b640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e240cf4d0 0x7f5e240ae810 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f5e20034040 tx=0x7f5e200038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e1affd640 1 -- 172.21.15.81:0/949833913 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2003e030 con 0x7f5e240cf4d0 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e1affd640 1 -- 172.21.15.81:0/949833913 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e20003ca0 con 0x7f5e240cf4d0 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e2c48f640 1 -- 172.21.15.81:0/949833913 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e240af060 con 0x7f5e240cf4d0 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.155+0000 7f5e1affd640 1 -- 172.21.15.81:0/949833913 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e20035040 con 0x7f5e240cf4d0 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.156+0000 7f5e2c48f640 1 -- 172.21.15.81:0/949833913 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e24154a00 con 0x7f5e240cf4d0 2024-06-29T15:57:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.156+0000 7f5e2c48f640 1 -- 172.21.15.81:0/949833913 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e2415b4a0 con 0x7f5e240cf4d0 2024-06-29T15:57:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.157+0000 7f5e1affd640 1 -- 172.21.15.81:0/949833913 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f5e20041030 con 0x7f5e240cf4d0 2024-06-29T15:57:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.158+0000 7f5e1affd640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5e080763f0 0x7f5e08078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.158+0000 7f5e1affd640 1 -- 172.21.15.81:0/949833913 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7f5e200b7f90 con 0x7f5e240cf4d0 2024-06-29T15:57:28.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.158+0000 7f5e29a2a640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5e080763f0 0x7f5e08078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:28.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.158+0000 7f5e29a2a640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5e080763f0 0x7f5e08078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5e14007fd0 tx=0x7f5e14009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:28.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.162+0000 7f5e1affd640 1 -- 172.21.15.81:0/949833913 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5e2415b4a0 con 0x7f5e240cf4d0 2024-06-29T15:57:28.253 INFO:tasks.ceph.mds.a.smithi081.stderr:2024-06-29T15:57:28.251+0000 7ffaf9717640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:57:28.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.379+0000 7f5e2c48f640 1 -- 172.21.15.81:0/949833913 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5e240b3a50 con 0x7f5e240cf4d0 2024-06-29T15:57:28.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.379+0000 7f5e1affd640 1 -- 172.21.15.81:0/949833913 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v112) ==== 53+0+73 (secure 0 0 0) 0x7f5e200816e0 con 0x7f5e240cf4d0 2024-06-29T15:57:28.382 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:57:28.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 -- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5e080763f0 msgr2=0x7f5e08078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5e080763f0 0x7f5e08078900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5e14007fd0 tx=0x7f5e14009040 comp rx=0 tx=0).stop 2024-06-29T15:57:28.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 -- 172.21.15.81:0/949833913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e240cf4d0 msgr2=0x7f5e240ae810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:28.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e240cf4d0 0x7f5e240ae810 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f5e20034040 tx=0x7f5e200038a0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 -- 172.21.15.81:0/949833913 shutdown_connections 2024-06-29T15:57:28.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5e080763f0 0x7f5e08078900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5e2415a450 0x7f5e24071f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5e241412f0 0x7f5e2413f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 --2- 172.21.15.81:0/949833913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e240cf4d0 0x7f5e240ae810 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 -- 172.21.15.81:0/949833913 >> 172.21.15.81:0/949833913 conn(0x7f5e2406ccc0 msgr2=0x7f5e24073030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 -- 172.21.15.81:0/949833913 shutdown_connections 2024-06-29T15:57:28.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28.381+0000 7f5e2c48f640 1 -- 172.21.15.81:0/949833913 wait complete. 2024-06-29T15:57:28.394 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:57:28.394 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:57:28.395 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T15:57:28.395 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:57:28.395 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:28.396 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:57:28.396 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:57:28.396 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:57:28.396 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:57:28.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.397 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:57:28.414 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 37071sec preferred_lft 37071sec 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:57:28.415 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:57:28.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.417 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:57:28.477 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T15:57:28.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.479 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T15:57:28.536 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T15:57:28.536 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:57:28.537 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T15:57:28.537 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T15:57:28.537 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:57:28.537 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:57:28.538 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T15:57:28.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T15:57:28.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T15:57:28.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T15:57:28.538 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T15:57:28.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:57:28.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:57:28.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.663 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T15:57:28.679 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:57:28.679 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:57:28.679 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:57:28.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T15:57:28.744 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:57:28.744 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:28.800 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:57:28.800 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:57:28.800 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:28.855 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T15:57:28.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.855 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:57:28.950 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:57:28.950 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:57:28.959 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:57:28.959 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:57:28.959 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T15:57:28.959 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T15:57:28.959 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T15:57:28.960 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:57:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:57:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:57:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:57:28.960 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:57:28.967 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:57:28.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:28.967 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:28.992 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:57:29.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:57:29.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:57:29.093 INFO:teuthology.orchestra.run.smithi081.stdout:client8936 2024-06-29T15:57:29.093 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:57:29.093 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:29.109 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-06-29T15:57:29.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.265+0000 7f804cdc7640 1 -- 172.21.15.81:0/3362718131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8030003a00 msgr2=0x7f8030003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:29.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.265+0000 7f804cdc7640 1 --2- 172.21.15.81:0/3362718131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8030003a00 0x7f8030003e70 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f8038009b50 tx=0x7f8030005830 comp rx=0 tx=0).stop 2024-06-29T15:57:29.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.265+0000 7f8046d9d640 1 -- 172.21.15.81:0/3362718131 reap_dead start 2024-06-29T15:57:29.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.265+0000 7f804cdc7640 1 -- 172.21.15.81:0/3362718131 shutdown_connections 2024-06-29T15:57:29.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.265+0000 7f804cdc7640 1 -- 172.21.15.81:0/3362718131 >> 172.21.15.81:0/3362718131 conn(0x7f804809aa60 msgr2=0x7f804809ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:29.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.265+0000 7f804cdc7640 1 -- 172.21.15.81:0/3362718131 shutdown_connections 2024-06-29T15:57:29.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.265+0000 7f804cdc7640 1 -- 172.21.15.81:0/3362718131 wait complete. 2024-06-29T15:57:29.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.267+0000 7f804cdc7640 1 Processor -- start 2024-06-29T15:57:29.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.267+0000 7f804cdc7640 1 -- start start 2024-06-29T15:57:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.268+0000 7f804cdc7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8030003a00 0x7f8048150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.268+0000 7f804cdc7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8048159ee0 0x7f804815a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.268+0000 7f804cdc7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80480b5960 0x7f80480b5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.268+0000 7f804cdc7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f804815fb80 con 0x7f8030003a00 2024-06-29T15:57:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.268+0000 7f804cdc7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f804815c400 con 0x7f80480b5960 2024-06-29T15:57:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.268+0000 7f804cdc7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f804815fd00 con 0x7f8048159ee0 2024-06-29T15:57:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.268+0000 7f804659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8030003a00 0x7f8048150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.268+0000 7f804659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8030003a00 0x7f8048150800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59950/0 (socket says 172.21.15.81:59950) 2024-06-29T15:57:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.268+0000 7f804659c640 1 -- 172.21.15.81:0/2122236433 learned_addr learned my addr 172.21.15.81:0/2122236433 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:29.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f8046d9d640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80480b5960 0x7f80480b5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:29.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f8045d9b640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8048159ee0 0x7f804815a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:29.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f804659c640 1 -- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8048159ee0 msgr2=0x7f804815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:29.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f804659c640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8048159ee0 0x7f804815a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:29.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f804659c640 1 -- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80480b5960 msgr2=0x7f80480b5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:29.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f804659c640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80480b5960 0x7f80480b5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:29.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f804659c640 1 -- 172.21.15.81:0/2122236433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80480b62b0 con 0x7f8030003a00 2024-06-29T15:57:29.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f8046d9d640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80480b5960 0x7f80480b5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:29.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f804659c640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8030003a00 0x7f8048150800 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f803802bfd0 tx=0x7f8038033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:29.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f8036ffd640 1 -- 172.21.15.81:0/2122236433 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038038050 con 0x7f8030003a00 2024-06-29T15:57:29.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f8036ffd640 1 -- 172.21.15.81:0/2122236433 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8038039040 con 0x7f8030003a00 2024-06-29T15:57:29.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f804cdc7640 1 -- 172.21.15.81:0/2122236433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8048158c50 con 0x7f8030003a00 2024-06-29T15:57:29.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f8036ffd640 1 -- 172.21.15.81:0/2122236433 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038033c50 con 0x7f8030003a00 2024-06-29T15:57:29.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.269+0000 7f804cdc7640 1 -- 172.21.15.81:0/2122236433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8048159210 con 0x7f8030003a00 2024-06-29T15:57:29.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.270+0000 7f804cdc7640 1 -- 172.21.15.81:0/2122236433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80480b0bd0 con 0x7f8030003a00 2024-06-29T15:57:29.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.271+0000 7f8036ffd640 1 -- 172.21.15.81:0/2122236433 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f8038046020 con 0x7f8030003a00 2024-06-29T15:57:29.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.271+0000 7f8036ffd640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f80280763f0 0x7f8028078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:29.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.272+0000 7f8036ffd640 1 -- 172.21.15.81:0/2122236433 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7f80380b8500 con 0x7f8030003a00 2024-06-29T15:57:29.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.272+0000 7f8045d9b640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f80280763f0 0x7f8028078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:29.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.272+0000 7f8045d9b640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f80280763f0 0x7f8028078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8048158770 tx=0x7f803c0067a0 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:29.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.276+0000 7f8036ffd640 1 -- 172.21.15.81:0/2122236433 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f80480b0bd0 con 0x7f8030003a00 2024-06-29T15:57:29.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:29.502+0000 7f804cdc7640 1 -- 172.21.15.81:0/2122236433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f80480d16b0 con 0x7f8030003a00 2024-06-29T15:57:30.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.328+0000 7f8036ffd640 1 -- 172.21.15.81:0/2122236433 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v113) ==== 136+0+0 (secure 0 0 0) 0x7f803804d020 con 0x7f8030003a00 2024-06-29T15:57:30.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 -- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f80280763f0 msgr2=0x7f8028078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f80280763f0 0x7f8028078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8048158770 tx=0x7f803c0067a0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 -- 172.21.15.81:0/2122236433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8030003a00 msgr2=0x7f8048150800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8030003a00 0x7f8048150800 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f803802bfd0 tx=0x7f8038033670 comp rx=0 tx=0).stop 2024-06-29T15:57:30.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 -- 172.21.15.81:0/2122236433 shutdown_connections 2024-06-29T15:57:30.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f80280763f0 0x7f8028078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f80480b5960 0x7f80480b5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8048159ee0 0x7f804815a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 --2- 172.21.15.81:0/2122236433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8030003a00 0x7f8048150800 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 -- 172.21.15.81:0/2122236433 >> 172.21.15.81:0/2122236433 conn(0x7f804809aa60 msgr2=0x7f8048151550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:30.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 -- 172.21.15.81:0/2122236433 shutdown_connections 2024-06-29T15:57:30.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.331+0000 7f804cdc7640 1 -- 172.21.15.81:0/2122236433 wait complete. 2024-06-29T15:57:30.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:30.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.458+0000 7fa0caeb3640 1 -- 172.21.15.81:0/1027459059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0ac004b40 msgr2=0x7fa0ac004fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.458+0000 7fa0caeb3640 1 --2- 172.21.15.81:0/1027459059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0ac004b40 0x7fa0ac004fb0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4070270 tx=0x7fa0ac006940 comp rx=0 tx=0).stop 2024-06-29T15:57:30.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.458+0000 7fa0c9450640 1 -- 172.21.15.81:0/1027459059 reap_dead start 2024-06-29T15:57:30.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.458+0000 7fa0caeb3640 1 -- 172.21.15.81:0/1027459059 shutdown_connections 2024-06-29T15:57:30.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.458+0000 7fa0caeb3640 1 -- 172.21.15.81:0/1027459059 >> 172.21.15.81:0/1027459059 conn(0x7fa0c406ccc0 msgr2=0x7fa0c40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:30.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.458+0000 7fa0caeb3640 1 -- 172.21.15.81:0/1027459059 shutdown_connections 2024-06-29T15:57:30.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.458+0000 7fa0caeb3640 1 -- 172.21.15.81:0/1027459059 wait complete. 2024-06-29T15:57:30.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.460+0000 7fa0caeb3640 1 Processor -- start 2024-06-29T15:57:30.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.460+0000 7fa0caeb3640 1 -- start start 2024-06-29T15:57:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.461+0000 7fa0caeb3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0ac004b40 0x7fa0c4140990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.461+0000 7fa0caeb3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0c4143200 0x7fa0c4159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.461+0000 7fa0caeb3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0c4143610 0x7fa0c4159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.461+0000 7fa0caeb3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c406ef50 con 0x7fa0ac004b40 2024-06-29T15:57:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.461+0000 7fa0caeb3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c406edd0 con 0x7fa0c4143610 2024-06-29T15:57:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.461+0000 7fa0caeb3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c406f0d0 con 0x7fa0c4143200 2024-06-29T15:57:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0c8c4f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0ac004b40 0x7fa0c4140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0c8c4f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0ac004b40 0x7fa0c4140990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35186/0 (socket says 172.21.15.81:35186) 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0c8c4f640 1 -- 172.21.15.81:0/3735389205 learned_addr learned my addr 172.21.15.81:0/3735389205 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0c9450640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0c4143610 0x7fa0c4159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0bbfff640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0c4143200 0x7fa0c4159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0c8c4f640 1 -- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0c4143200 msgr2=0x7fa0c4159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0c8c4f640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0c4143200 0x7fa0c4159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0c8c4f640 1 -- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0c4143610 msgr2=0x7fa0c4159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0c8c4f640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0c4143610 0x7fa0c4159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0c8c4f640 1 -- 172.21.15.81:0/3735389205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c4140ed0 con 0x7fa0ac004b40 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0c8c4f640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0ac004b40 0x7fa0c4140990 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000bac0 tx=0x7fa0c0034c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0b97fa640 1 -- 172.21.15.81:0/3735389205 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0039040 con 0x7fa0ac004b40 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0b97fa640 1 -- 172.21.15.81:0/3735389205 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0c0005110 con 0x7fa0ac004b40 2024-06-29T15:57:30.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0caeb3640 1 -- 172.21.15.81:0/3735389205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c414f570 con 0x7fa0ac004b40 2024-06-29T15:57:30.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0b97fa640 1 -- 172.21.15.81:0/3735389205 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c000b620 con 0x7fa0ac004b40 2024-06-29T15:57:30.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.462+0000 7fa0caeb3640 1 -- 172.21.15.81:0/3735389205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c41440a0 con 0x7fa0ac004b40 2024-06-29T15:57:30.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.463+0000 7fa0caeb3640 1 -- 172.21.15.81:0/3735389205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c40cf130 con 0x7fa0ac004b40 2024-06-29T15:57:30.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.464+0000 7fa0b97fa640 1 -- 172.21.15.81:0/3735389205 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fa0c0044020 con 0x7fa0ac004b40 2024-06-29T15:57:30.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.464+0000 7fa0b97fa640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa0b00763f0 0x7fa0b0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.465+0000 7fa0b97fa640 1 -- 172.21.15.81:0/3735389205 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7fa0c00b7790 con 0x7fa0ac004b40 2024-06-29T15:57:30.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.465+0000 7fa0bbfff640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa0b00763f0 0x7fa0b0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.465+0000 7fa0bbfff640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa0b00763f0 0x7fa0b0078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa0c41f8730 tx=0x7fa0b4006b70 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:30.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.469+0000 7fa0b97fa640 1 -- 172.21.15.81:0/3735389205 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa0c0080e50 con 0x7fa0ac004b40 2024-06-29T15:57:30.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.650+0000 7fa0caeb3640 1 -- 172.21.15.81:0/3735389205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0c415c620 con 0x7fa0ac004b40 2024-06-29T15:57:30.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.651+0000 7fa0b97fa640 1 -- 172.21.15.81:0/3735389205 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) ==== 74+0+16875 (secure 0 0 0) 0x7fa0c0080750 con 0x7fa0ac004b40 2024-06-29T15:57:30.652 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:30.653 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":213,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:23.285939+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":211,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 -- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa0b00763f0 msgr2=0x7fa0b0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa0b00763f0 0x7fa0b0078900 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa0c41f8730 tx=0x7fa0b4006b70 comp rx=0 tx=0).stop 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 -- 172.21.15.81:0/3735389205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0ac004b40 msgr2=0x7fa0c4140990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0ac004b40 0x7fa0c4140990 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000bac0 tx=0x7fa0c0034c90 comp rx=0 tx=0).stop 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 -- 172.21.15.81:0/3735389205 shutdown_connections 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa0b00763f0 0x7fa0b0078900 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa0c4143610 0x7fa0c4159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa0c4143200 0x7fa0c4159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 --2- 172.21.15.81:0/3735389205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0ac004b40 0x7fa0c4140990 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 -- 172.21.15.81:0/3735389205 >> 172.21.15.81:0/3735389205 conn(0x7fa0c406ccc0 msgr2=0x7fa0c40b15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 -- 172.21.15.81:0/3735389205 shutdown_connections 2024-06-29T15:57:30.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.653+0000 7fa0caeb3640 1 -- 172.21.15.81:0/3735389205 wait complete. 2024-06-29T15:57:30.666 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:57:19.937473+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:57:20.596779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:30.666 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:30.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.781+0000 7f8b3c368640 1 -- 172.21.15.81:0/3133046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b3406f660 msgr2=0x7f8b3406fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.781+0000 7f8b3c368640 1 --2- 172.21.15.81:0/3133046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b3406f660 0x7f8b3406fa70 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f8b34070220 tx=0x7f8b2c02a4d0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.781+0000 7f8b3c368640 1 -- 172.21.15.81:0/3133046 shutdown_connections 2024-06-29T15:57:30.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.781+0000 7f8b3c368640 1 --2- 172.21.15.81:0/3133046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b3406f660 0x7f8b3406fa70 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.781+0000 7f8b3c368640 1 --2- 172.21.15.81:0/3133046 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b340b4430 0x7f8b340cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.781+0000 7f8b3c368640 1 --2- 172.21.15.81:0/3133046 >> v2:172.21.15.167:3300/0 conn(0x7f8b340b35b0 0x7f8b3415b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.781+0000 7f8b3c368640 1 -- 172.21.15.81:0/3133046 >> 172.21.15.81:0/3133046 conn(0x7f8b3406ccc0 msgr2=0x7f8b340b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:30.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.781+0000 7f8b3c368640 1 -- 172.21.15.81:0/3133046 shutdown_connections 2024-06-29T15:57:30.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.781+0000 7f8b3c368640 1 -- 172.21.15.81:0/3133046 wait complete. 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.783+0000 7f8b3c368640 1 Processor -- start 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.783+0000 7f8b3c368640 1 -- start start 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3c368640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b3406f660 0x7f8b341e51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3c368640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b340b35b0 0x7f8b34147250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3c368640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b340b4430 0x7f8b340a9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3c368640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3406f160 con 0x7f8b340b4430 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3c368640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3406efe0 con 0x7f8b3406f660 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3c368640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8b3406f2e0 con 0x7f8b340b35b0 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3a905640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b340b4430 0x7f8b340a9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3a905640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b340b4430 0x7f8b340a9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35194/0 (socket says 172.21.15.81:35194) 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3a905640 1 -- 172.21.15.81:0/3647822943 learned_addr learned my addr 172.21.15.81:0/3647822943 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:30.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b39903640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b340b35b0 0x7f8b34147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3a104640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b3406f660 0x7f8b341e51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3a905640 1 -- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b340b35b0 msgr2=0x7f8b34147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3a905640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b340b35b0 0x7f8b34147250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3a905640 1 -- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b3406f660 msgr2=0x7f8b341e51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3a905640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b3406f660 0x7f8b341e51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.784+0000 7f8b3a905640 1 -- 172.21.15.81:0/3647822943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b341e5c70 con 0x7f8b340b4430 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.785+0000 7f8b3a905640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b340b4430 0x7f8b340a9cd0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0099c0 tx=0x7f8b2c003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.785+0000 7f8b32ffd640 1 -- 172.21.15.81:0/3647822943 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c03e030 con 0x7f8b340b4430 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.785+0000 7f8b32ffd640 1 -- 172.21.15.81:0/3647822943 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b2c003e40 con 0x7f8b340b4430 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.785+0000 7f8b3c368640 1 -- 172.21.15.81:0/3647822943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b341e5f50 con 0x7f8b340b4430 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.785+0000 7f8b32ffd640 1 -- 172.21.15.81:0/3647822943 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c035040 con 0x7f8b340b4430 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.785+0000 7f8b3c368640 1 -- 172.21.15.81:0/3647822943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b341e6510 con 0x7f8b340b4430 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.786+0000 7f8b3c368640 1 -- 172.21.15.81:0/3647822943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b3415b5b0 con 0x7f8b340b4430 2024-06-29T15:57:30.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.786+0000 7f8b32ffd640 1 -- 172.21.15.81:0/3647822943 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f8b2c041030 con 0x7f8b340b4430 2024-06-29T15:57:30.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.787+0000 7f8b32ffd640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8b20076420 0x7f8b20078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:30.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.787+0000 7f8b32ffd640 1 -- 172.21.15.81:0/3647822943 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7f8b2c0b8190 con 0x7f8b340b4430 2024-06-29T15:57:30.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.787+0000 7f8b3a104640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8b20076420 0x7f8b20078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:30.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.788+0000 7f8b3a104640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8b20076420 0x7f8b20078930 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b24005fd0 tx=0x7f8b2400a310 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:30.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.791+0000 7f8b32ffd640 1 -- 172.21.15.81:0/3647822943 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8b2c0817c0 con 0x7f8b340b4430 2024-06-29T15:57:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.971+0000 7f8b3c368640 1 -- 172.21.15.81:0/3647822943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8b341e5a00 con 0x7f8b340b4430 2024-06-29T15:57:30.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.972+0000 7f8b32ffd640 1 -- 172.21.15.81:0/3647822943 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v213) ==== 93+0+10 (secure 0 0 0) 0x7f8b2c0810c0 con 0x7f8b340b4430 2024-06-29T15:57:30.973 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:57:30.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 -- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8b20076420 msgr2=0x7f8b20078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8b20076420 0x7f8b20078930 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b24005fd0 tx=0x7f8b2400a310 comp rx=0 tx=0).stop 2024-06-29T15:57:30.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 -- 172.21.15.81:0/3647822943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b340b4430 msgr2=0x7f8b340a9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:30.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b340b4430 0x7f8b340a9cd0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0099c0 tx=0x7f8b2c003910 comp rx=0 tx=0).stop 2024-06-29T15:57:30.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 -- 172.21.15.81:0/3647822943 shutdown_connections 2024-06-29T15:57:30.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f8b20076420 0x7f8b20078930 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b340b4430 0x7f8b340a9cd0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8b340b35b0 0x7f8b34147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 --2- 172.21.15.81:0/3647822943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8b3406f660 0x7f8b341e51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:30.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 -- 172.21.15.81:0/3647822943 >> 172.21.15.81:0/3647822943 conn(0x7f8b3406ccc0 msgr2=0x7f8b3415cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:30.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 -- 172.21.15.81:0/3647822943 shutdown_connections 2024-06-29T15:57:30.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:30.974+0000 7f8b3c368640 1 -- 172.21.15.81:0/3647822943 wait complete. 2024-06-29T15:57:30.986 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:57:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.101+0000 7f588bd3c640 1 -- 172.21.15.81:0/1888135015 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f588408ccd0 msgr2=0x7f58840b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.101+0000 7f588bd3c640 1 --2- 172.21.15.81:0/1888135015 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f588408ccd0 0x7f58840b3240 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5874009860 tx=0x7f587402a480 comp rx=0 tx=0).stop 2024-06-29T15:57:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.101+0000 7f588bd3c640 1 -- 172.21.15.81:0/1888135015 shutdown_connections 2024-06-29T15:57:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.101+0000 7f588bd3c640 1 --2- 172.21.15.81:0/1888135015 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f588408ccd0 0x7f58840b3240 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.101+0000 7f588bd3c640 1 -- 172.21.15.81:0/1888135015 >> 172.21.15.81:0/1888135015 conn(0x7f58840b56d0 msgr2=0x7f58840b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.102+0000 7f588bd3c640 1 -- 172.21.15.81:0/1888135015 shutdown_connections 2024-06-29T15:57:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.102+0000 7f588bd3c640 1 -- 172.21.15.81:0/1888135015 wait complete. 2024-06-29T15:57:31.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.103+0000 7f588bd3c640 1 Processor -- start 2024-06-29T15:57:31.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.103+0000 7f588bd3c640 1 -- start start 2024-06-29T15:57:31.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f588bd3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f588408ccd0 0x7f58841625a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f588bd3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58840cf040 0x7f58841583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f588bd3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f588415ca30 0x7f58840b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f588bd3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f58840b44f0 con 0x7f58840cf040 2024-06-29T15:57:31.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f588bd3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f58840b4370 con 0x7f588415ca30 2024-06-29T15:57:31.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f588bd3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f58840b4670 con 0x7f588408ccd0 2024-06-29T15:57:31.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f58892d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58840cf040 0x7f58841583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f58892d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58840cf040 0x7f58841583c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35210/0 (socket says 172.21.15.81:35210) 2024-06-29T15:57:31.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f58892d7640 1 -- 172.21.15.81:0/738720144 learned_addr learned my addr 172.21.15.81:0/738720144 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:31.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f5889ad8640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f588408ccd0 0x7f58841625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f588a2d9640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f588415ca30 0x7f58840b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f58892d7640 1 -- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f588408ccd0 msgr2=0x7f58841625a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f58892d7640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f588408ccd0 0x7f58841625a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f58892d7640 1 -- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f588415ca30 msgr2=0x7f58840b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f58892d7640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f588415ca30 0x7f58840b1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f58892d7640 1 -- 172.21.15.81:0/738720144 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58840b8640 con 0x7f58840cf040 2024-06-29T15:57:31.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f58892d7640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58840cf040 0x7f58841583c0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f587402aa70 tx=0x7f5874033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f587a7fc640 1 -- 172.21.15.81:0/738720144 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874031b50 con 0x7f58840cf040 2024-06-29T15:57:31.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f588bd3c640 1 -- 172.21.15.81:0/738720144 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58840b88b0 con 0x7f58840cf040 2024-06-29T15:57:31.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f587a7fc640 1 -- 172.21.15.81:0/738720144 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5874031de0 con 0x7f58840cf040 2024-06-29T15:57:31.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.105+0000 7f587a7fc640 1 -- 172.21.15.81:0/738720144 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874033330 con 0x7f58840cf040 2024-06-29T15:57:31.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.106+0000 7f588bd3c640 1 -- 172.21.15.81:0/738720144 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5884164c30 con 0x7f58840cf040 2024-06-29T15:57:31.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.106+0000 7f588bd3c640 1 -- 172.21.15.81:0/738720144 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f588408c270 con 0x7f58840cf040 2024-06-29T15:57:31.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.107+0000 7f587a7fc640 1 -- 172.21.15.81:0/738720144 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f58740334d0 con 0x7f58840cf040 2024-06-29T15:57:31.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.108+0000 7f587a7fc640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f58680763f0 0x7f5868078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.108+0000 7f587a7fc640 1 -- 172.21.15.81:0/738720144 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7f58740b8e70 con 0x7f58840cf040 2024-06-29T15:57:31.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.108+0000 7f5889ad8640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f58680763f0 0x7f5868078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.108+0000 7f5889ad8640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f58680763f0 0x7f5868078900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5880007fd0 tx=0x7f5880009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.112+0000 7f587a7fc640 1 -- 172.21.15.81:0/738720144 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f587403f030 con 0x7f58840cf040 2024-06-29T15:57:31.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.291+0000 7f588bd3c640 1 -- 172.21.15.81:0/738720144 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f58840a1ba0 con 0x7f58840cf040 2024-06-29T15:57:31.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.291+0000 7f587a7fc640 1 -- 172.21.15.81:0/738720144 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v213) ==== 104+0+11 (secure 0 0 0) 0x7f5874082420 con 0x7f58840cf040 2024-06-29T15:57:31.293 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:57:31.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.293+0000 7f588bd3c640 1 -- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f58680763f0 msgr2=0x7f5868078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.293+0000 7f588bd3c640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f58680763f0 0x7f5868078900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5880007fd0 tx=0x7f5880009040 comp rx=0 tx=0).stop 2024-06-29T15:57:31.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.293+0000 7f588bd3c640 1 -- 172.21.15.81:0/738720144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58840cf040 msgr2=0x7f58841583c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.293+0000 7f588bd3c640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58840cf040 0x7f58841583c0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f587402aa70 tx=0x7f5874033ba0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.293+0000 7f588bd3c640 1 -- 172.21.15.81:0/738720144 shutdown_connections 2024-06-29T15:57:31.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.293+0000 7f588bd3c640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f58680763f0 0x7f5868078900 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.293+0000 7f588bd3c640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f588415ca30 0x7f58840b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.293+0000 7f588bd3c640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58840cf040 0x7f58841583c0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.293+0000 7f588bd3c640 1 --2- 172.21.15.81:0/738720144 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f588408ccd0 0x7f58841625a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.293+0000 7f588bd3c640 1 -- 172.21.15.81:0/738720144 >> 172.21.15.81:0/738720144 conn(0x7f58840b56d0 msgr2=0x7f5884068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.294+0000 7f588bd3c640 1 -- 172.21.15.81:0/738720144 shutdown_connections 2024-06-29T15:57:31.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.294+0000 7f588bd3c640 1 -- 172.21.15.81:0/738720144 wait complete. 2024-06-29T15:57:31.306 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:57:31.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.421+0000 7f12f223d640 1 -- 172.21.15.81:0/62493961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d4005300 msgr2=0x7f12d400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.421+0000 7f12f223d640 1 --2- 172.21.15.81:0/62493961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d4005300 0x7f12d400b9c0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f12e40013d0 tx=0x7f12d4005830 comp rx=0 tx=0).stop 2024-06-29T15:57:31.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.421+0000 7f12f223d640 1 -- 172.21.15.81:0/62493961 shutdown_connections 2024-06-29T15:57:31.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.421+0000 7f12f223d640 1 --2- 172.21.15.81:0/62493961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d4005300 0x7f12d400b9c0 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.421+0000 7f12f223d640 1 -- 172.21.15.81:0/62493961 >> 172.21.15.81:0/62493961 conn(0x7f12ec072950 msgr2=0x7f12ec0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.422+0000 7f12f223d640 1 -- 172.21.15.81:0/62493961 shutdown_connections 2024-06-29T15:57:31.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.422+0000 7f12f223d640 1 -- 172.21.15.81:0/62493961 wait complete. 2024-06-29T15:57:31.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.423+0000 7f12f223d640 1 Processor -- start 2024-06-29T15:57:31.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.423+0000 7f12f223d640 1 -- start start 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12f223d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12d4005300 0x7f12ec0ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12f223d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12ec156c50 0x7f12ec0c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12f223d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12ec0c83d0 0x7f12ec0c5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12f223d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec15b190 con 0x7f12ec156c50 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12f223d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec15b010 con 0x7f12d4005300 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12f223d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f12ec06f140 con 0x7f12ec0c83d0 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12ec156c50 0x7f12ec0c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12ec156c50 0x7f12ec0c7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35228/0 (socket says 172.21.15.81:35228) 2024-06-29T15:57:31.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eaffd640 1 -- 172.21.15.81:0/4253296719 learned_addr learned my addr 172.21.15.81:0/4253296719 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12ebfff640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12ec0c83d0 0x7f12ec0c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eb7fe640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12d4005300 0x7f12ec0ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eaffd640 1 -- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12ec0c83d0 msgr2=0x7f12ec0c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eaffd640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12ec0c83d0 0x7f12ec0c5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eaffd640 1 -- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12d4005300 msgr2=0x7f12ec0ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eaffd640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12d4005300 0x7f12ec0ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eaffd640 1 -- 172.21.15.81:0/4253296719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ec0c87e0 con 0x7f12ec156c50 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12ebfff640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12ec0c83d0 0x7f12ec0c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eb7fe640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12d4005300 0x7f12ec0ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12eaffd640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12ec156c50 0x7f12ec0c7e90 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f12e0004d20 tx=0x7f12e0001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12d3fff640 1 -- 172.21.15.81:0/4253296719 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e00107d0 con 0x7f12ec156c50 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12d3fff640 1 -- 172.21.15.81:0/4253296719 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12e0016070 con 0x7f12ec156c50 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12f223d640 1 -- 172.21.15.81:0/4253296719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ec0cf090 con 0x7f12ec156c50 2024-06-29T15:57:31.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12d3fff640 1 -- 172.21.15.81:0/4253296719 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0010d20 con 0x7f12ec156c50 2024-06-29T15:57:31.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.425+0000 7f12f223d640 1 -- 172.21.15.81:0/4253296719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ec0cf880 con 0x7f12ec156c50 2024-06-29T15:57:31.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.426+0000 7f12f223d640 1 -- 172.21.15.81:0/4253296719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12ec0cf310 con 0x7f12ec156c50 2024-06-29T15:57:31.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.427+0000 7f12d3fff640 1 -- 172.21.15.81:0/4253296719 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f12e001b460 con 0x7f12ec156c50 2024-06-29T15:57:31.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.428+0000 7f12d3fff640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f12d80763f0 0x7f12d8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.428+0000 7f12d3fff640 1 -- 172.21.15.81:0/4253296719 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7f12e0098200 con 0x7f12ec156c50 2024-06-29T15:57:31.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.428+0000 7f12eb7fe640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f12d80763f0 0x7f12d8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.429+0000 7f12eb7fe640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f12d80763f0 0x7f12d8078900 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00ea50 tx=0x7f12dc00e450 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.432+0000 7f12d3fff640 1 -- 172.21.15.81:0/4253296719 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f12ec0c87e0 con 0x7f12ec156c50 2024-06-29T15:57:31.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.610+0000 7f12f223d640 1 -- 172.21.15.81:0/4253296719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f12ec06c2f0 con 0x7f12ec156c50 2024-06-29T15:57:31.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.610+0000 7f12d3fff640 1 -- 172.21.15.81:0/4253296719 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v213) ==== 100+0+11 (secure 0 0 0) 0x7f12e0061830 con 0x7f12ec156c50 2024-06-29T15:57:31.611 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:57:31.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 -- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f12d80763f0 msgr2=0x7f12d8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f12d80763f0 0x7f12d8078900 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00ea50 tx=0x7f12dc00e450 comp rx=0 tx=0).stop 2024-06-29T15:57:31.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 -- 172.21.15.81:0/4253296719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12ec156c50 msgr2=0x7f12ec0c7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12ec156c50 0x7f12ec0c7e90 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f12e0004d20 tx=0x7f12e0001630 comp rx=0 tx=0).stop 2024-06-29T15:57:31.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 -- 172.21.15.81:0/4253296719 shutdown_connections 2024-06-29T15:57:31.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f12d80763f0 0x7f12d8078900 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f12ec0c83d0 0x7f12ec0c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12ec156c50 0x7f12ec0c7e90 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 --2- 172.21.15.81:0/4253296719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f12d4005300 0x7f12ec0ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 -- 172.21.15.81:0/4253296719 >> 172.21.15.81:0/4253296719 conn(0x7f12ec072950 msgr2=0x7f12ec09d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 -- 172.21.15.81:0/4253296719 shutdown_connections 2024-06-29T15:57:31.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.612+0000 7f12f223d640 1 -- 172.21.15.81:0/4253296719 wait complete. 2024-06-29T15:57:31.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:57:31.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.737+0000 7fe184ba1640 1 -- 172.21.15.81:0/1939516422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe168004d20 msgr2=0x7fe168005170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.737+0000 7fe184ba1640 1 --2- 172.21.15.81:0/1939516422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe168004d20 0x7fe168005170 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7fe170006fb0 tx=0x7fe168006940 comp rx=0 tx=0).stop 2024-06-29T15:57:31.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.737+0000 7fe17ed9d640 1 -- 172.21.15.81:0/1939516422 reap_dead start 2024-06-29T15:57:31.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.738+0000 7fe184ba1640 1 -- 172.21.15.81:0/1939516422 shutdown_connections 2024-06-29T15:57:31.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.738+0000 7fe184ba1640 1 -- 172.21.15.81:0/1939516422 >> 172.21.15.81:0/1939516422 conn(0x7fe18006ccc0 msgr2=0x7fe1800b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.738+0000 7fe184ba1640 1 -- 172.21.15.81:0/1939516422 shutdown_connections 2024-06-29T15:57:31.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.738+0000 7fe184ba1640 1 -- 172.21.15.81:0/1939516422 wait complete. 2024-06-29T15:57:31.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.739+0000 7fe184ba1640 1 Processor -- start 2024-06-29T15:57:31.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.739+0000 7fe184ba1640 1 -- start start 2024-06-29T15:57:31.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe184ba1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe168004d20 0x7fe1801eff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe184ba1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1800ad890 0x7fe1800adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe184ba1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe1800a95c0 0x7fe1800a99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe184ba1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe18006ef50 con 0x7fe1800ad890 2024-06-29T15:57:31.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe184ba1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe18006edd0 con 0x7fe168004d20 2024-06-29T15:57:31.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe184ba1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe18006f0d0 con 0x7fe1800a95c0 2024-06-29T15:57:31.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1800ad890 0x7fe1800adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1800ad890 0x7fe1800adca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35246/0 (socket says 172.21.15.81:35246) 2024-06-29T15:57:31.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17dd9b640 1 -- 172.21.15.81:0/2867114719 learned_addr learned my addr 172.21.15.81:0/2867114719 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:31.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17e59c640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe168004d20 0x7fe1801eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17ed9d640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe1800a95c0 0x7fe1800a99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17dd9b640 1 -- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe1800a95c0 msgr2=0x7fe1800a99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17dd9b640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe1800a95c0 0x7fe1800a99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17dd9b640 1 -- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe168004d20 msgr2=0x7fe1801eff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17dd9b640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe168004d20 0x7fe1801eff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17dd9b640 1 -- 172.21.15.81:0/2867114719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1800ae510 con 0x7fe1800ad890 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe17dd9b640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1800ad890 0x7fe1800adca0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fe174011b10 tx=0x7fe174009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.741+0000 7fe16effd640 1 -- 172.21.15.81:0/2867114719 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17401e070 con 0x7fe1800ad890 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.742+0000 7fe184ba1640 1 -- 172.21.15.81:0/2867114719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1800ae7f0 con 0x7fe1800ad890 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.742+0000 7fe16effd640 1 -- 172.21.15.81:0/2867114719 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe174004050 con 0x7fe1800ad890 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.742+0000 7fe16effd640 1 -- 172.21.15.81:0/2867114719 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17400fea0 con 0x7fe1800ad890 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.742+0000 7fe184ba1640 1 -- 172.21.15.81:0/2867114719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1800aedb0 con 0x7fe1800ad890 2024-06-29T15:57:31.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.743+0000 7fe184ba1640 1 -- 172.21.15.81:0/2867114719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1800b4800 con 0x7fe1800ad890 2024-06-29T15:57:31.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.743+0000 7fe16effd640 1 -- 172.21.15.81:0/2867114719 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fe174021070 con 0x7fe1800ad890 2024-06-29T15:57:31.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.744+0000 7fe16effd640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe160076420 0x7fe160078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:31.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.744+0000 7fe16effd640 1 -- 172.21.15.81:0/2867114719 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7fe174098450 con 0x7fe1800ad890 2024-06-29T15:57:31.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.744+0000 7fe17e59c640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe160076420 0x7fe160078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:31.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.744+0000 7fe17e59c640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe160076420 0x7fe160078930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe170033000 tx=0x7fe17002d000 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:31.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.748+0000 7fe16effd640 1 -- 172.21.15.81:0/2867114719 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe174061590 con 0x7fe1800ad890 2024-06-29T15:57:31.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.929+0000 7fe184ba1640 1 -- 172.21.15.81:0/2867114719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe1800ae2a0 con 0x7fe1800ad890 2024-06-29T15:57:31.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.929+0000 7fe16effd640 1 -- 172.21.15.81:0/2867114719 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v213) ==== 84+0+209 (secure 0 0 0) 0x7fe174065890 con 0x7fe1800ad890 2024-06-29T15:57:31.930 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 26, 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": "cephfs_metadata" 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 27, 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": "cephfs_data" 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:31.931 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:57:31.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 -- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe160076420 msgr2=0x7fe160078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe160076420 0x7fe160078930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe170033000 tx=0x7fe17002d000 comp rx=0 tx=0).stop 2024-06-29T15:57:31.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 -- 172.21.15.81:0/2867114719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1800ad890 msgr2=0x7fe1800adca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1800ad890 0x7fe1800adca0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fe174011b10 tx=0x7fe174009e90 comp rx=0 tx=0).stop 2024-06-29T15:57:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 -- 172.21.15.81:0/2867114719 shutdown_connections 2024-06-29T15:57:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe160076420 0x7fe160078930 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe1800a95c0 0x7fe1800a99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1800ad890 0x7fe1800adca0 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 --2- 172.21.15.81:0/2867114719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe168004d20 0x7fe1801eff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 -- 172.21.15.81:0/2867114719 >> 172.21.15.81:0/2867114719 conn(0x7fe18006ccc0 msgr2=0x7fe18015ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 -- 172.21.15.81:0/2867114719 shutdown_connections 2024-06-29T15:57:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:31.931+0000 7fe184ba1640 1 -- 172.21.15.81:0/2867114719 wait complete. 2024-06-29T15:57:31.944 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-06-29T15:57:31.944 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-06-29T15:57:32.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.060+0000 7f781f63f640 1 -- 172.21.15.81:0/105104998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7818074070 msgr2=0x7f7818074480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.060+0000 7f781f63f640 1 --2- 172.21.15.81:0/105104998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7818074070 0x7f7818074480 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f781806dec0 tx=0x7f780802a440 comp rx=0 tx=0).stop 2024-06-29T15:57:32.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.060+0000 7f781f63f640 1 -- 172.21.15.81:0/105104998 shutdown_connections 2024-06-29T15:57:32.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.060+0000 7f781f63f640 1 --2- 172.21.15.81:0/105104998 >> v2:172.21.15.167:3300/0 conn(0x7f781806c2f0 0x7f781806c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.060+0000 7f781f63f640 1 --2- 172.21.15.81:0/105104998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7818074070 0x7f7818074480 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.060+0000 7f781f63f640 1 --2- 172.21.15.81:0/105104998 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78180b04b0 0x7f7818073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.060+0000 7f781f63f640 1 -- 172.21.15.81:0/105104998 >> 172.21.15.81:0/105104998 conn(0x7f7818072970 msgr2=0x7f78180b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:32.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.060+0000 7f781f63f640 1 -- 172.21.15.81:0/105104998 shutdown_connections 2024-06-29T15:57:32.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.060+0000 7f781f63f640 1 -- 172.21.15.81:0/105104998 wait complete. 2024-06-29T15:57:32.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.062+0000 7f781f63f640 1 Processor -- start 2024-06-29T15:57:32.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.062+0000 7f781f63f640 1 -- start start 2024-06-29T15:57:32.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781f63f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f781806c2f0 0x7f7818150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781f63f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7818074070 0x7f78181544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781f63f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78180b04b0 0x7f78181e79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781f63f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f781806cdc0 con 0x7f78180b04b0 2024-06-29T15:57:32.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781f63f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f781806cc40 con 0x7f781806c2f0 2024-06-29T15:57:32.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781f63f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f781806cf40 con 0x7f7818074070 2024-06-29T15:57:32.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781dbdc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78180b04b0 0x7f78181e79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781dbdc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78180b04b0 0x7f78181e79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35274/0 (socket says 172.21.15.81:35274) 2024-06-29T15:57:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781dbdc640 1 -- 172.21.15.81:0/2935368941 learned_addr learned my addr 172.21.15.81:0/2935368941 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781cbda640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7818074070 0x7f78181544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.063+0000 7f781d3db640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f781806c2f0 0x7f7818150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f781dbdc640 1 -- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7818074070 msgr2=0x7f78181544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f781dbdc640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7818074070 0x7f78181544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f781dbdc640 1 -- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f781806c2f0 msgr2=0x7f7818150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f781dbdc640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f781806c2f0 0x7f7818150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f781dbdc640 1 -- 172.21.15.81:0/2935368941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7818154030 con 0x7f78180b04b0 2024-06-29T15:57:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f781dbdc640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78180b04b0 0x7f78181e79e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f781000cab0 tx=0x7f781000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:32.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f780dffb640 1 -- 172.21.15.81:0/2935368941 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7810007d90 con 0x7f78180b04b0 2024-06-29T15:57:32.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f780dffb640 1 -- 172.21.15.81:0/2935368941 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f781000f540 con 0x7f78180b04b0 2024-06-29T15:57:32.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f781f63f640 1 -- 172.21.15.81:0/2935368941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7818155090 con 0x7f78180b04b0 2024-06-29T15:57:32.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f780dffb640 1 -- 172.21.15.81:0/2935368941 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781000f990 con 0x7f78180b04b0 2024-06-29T15:57:32.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.064+0000 7f781f63f640 1 -- 172.21.15.81:0/2935368941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78181f5950 con 0x7f78180b04b0 2024-06-29T15:57:32.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.065+0000 7f781f63f640 1 -- 172.21.15.81:0/2935368941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7818154e00 con 0x7f78180b04b0 2024-06-29T15:57:32.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.066+0000 7f780dffb640 1 -- 172.21.15.81:0/2935368941 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f781000fb30 con 0x7f78180b04b0 2024-06-29T15:57:32.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.066+0000 7f780dffb640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77fc0763f0 0x7f77fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.066+0000 7f780dffb640 1 -- 172.21.15.81:0/2935368941 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5837+0+0 (secure 0 0 0) 0x7f78100983b0 con 0x7f78180b04b0 2024-06-29T15:57:32.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.066+0000 7f781d3db640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77fc0763f0 0x7f77fc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:32.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.067+0000 7f781d3db640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77fc0763f0 0x7f77fc078900 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f78140098e0 tx=0x7f7814009060 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:32.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.070+0000 7f780dffb640 1 -- 172.21.15.81:0/2935368941 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7818154030 con 0x7f78180b04b0 2024-06-29T15:57:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.250+0000 7f781f63f640 1 -- 172.21.15.81:0/2935368941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) -- 0x7f78181f5460 con 0x7f78180b04b0 2024-06-29T15:57:32.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.337+0000 7f780dffb640 1 -- 172.21.15.81:0/2935368941 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v214) ==== 144+0+0 (secure 0 0 0) 0x7f7810014070 con 0x7f78180b04b0 2024-06-29T15:57:32.339 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'backup_fs_metadata' created 2024-06-29T15:57:32.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.340+0000 7f781f63f640 1 -- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77fc0763f0 msgr2=0x7f77fc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.340+0000 7f781f63f640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77fc0763f0 0x7f77fc078900 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f78140098e0 tx=0x7f7814009060 comp rx=0 tx=0).stop 2024-06-29T15:57:32.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.340+0000 7f781f63f640 1 -- 172.21.15.81:0/2935368941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78180b04b0 msgr2=0x7f78181e79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.340+0000 7f781f63f640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78180b04b0 0x7f78181e79e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f781000cab0 tx=0x7f781000cf70 comp rx=0 tx=0).stop 2024-06-29T15:57:32.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.340+0000 7f781f63f640 1 -- 172.21.15.81:0/2935368941 shutdown_connections 2024-06-29T15:57:32.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.340+0000 7f781f63f640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f77fc0763f0 0x7f77fc078900 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.340+0000 7f781f63f640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78180b04b0 0x7f78181e79e0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.340+0000 7f781f63f640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7818074070 0x7f78181544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.340+0000 7f781f63f640 1 --2- 172.21.15.81:0/2935368941 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f781806c2f0 0x7f7818150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.341+0000 7f781f63f640 1 -- 172.21.15.81:0/2935368941 >> 172.21.15.81:0/2935368941 conn(0x7f7818072970 msgr2=0x7f781806e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:32.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.341+0000 7f781f63f640 1 -- 172.21.15.81:0/2935368941 shutdown_connections 2024-06-29T15:57:32.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.341+0000 7f781f63f640 1 -- 172.21.15.81:0/2935368941 wait complete. 2024-06-29T15:57:32.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:57:32.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.476+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/953245754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4c40aea10 msgr2=0x7fa4c40aee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.476+0000 7fa4cb7d2640 1 --2- 172.21.15.81:0/953245754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4c40aea10 0x7fa4c40aee20 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc005fd0 tx=0x7fa4bc02ab40 comp rx=0 tx=0).stop 2024-06-29T15:57:32.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.476+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/953245754 shutdown_connections 2024-06-29T15:57:32.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.476+0000 7fa4cb7d2640 1 --2- 172.21.15.81:0/953245754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4c40aea10 0x7fa4c40aee20 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.476+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/953245754 >> 172.21.15.81:0/953245754 conn(0x7fa4c4072950 msgr2=0x7fa4c40b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:32.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.476+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/953245754 shutdown_connections 2024-06-29T15:57:32.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.476+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/953245754 wait complete. 2024-06-29T15:57:32.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.478+0000 7fa4cb7d2640 1 Processor -- start 2024-06-29T15:57:32.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.478+0000 7fa4cb7d2640 1 -- start start 2024-06-29T15:57:32.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.479+0000 7fa4cb7d2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4c406f850 0x7fa4c41f12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.479+0000 7fa4cb7d2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4c4074520 0x7fa4c40b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.479+0000 7fa4cb7d2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4c40aea10 0x7fa4c40b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.479+0000 7fa4cb7d2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c415b190 con 0x7fa4c4074520 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.479+0000 7fa4cb7d2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c415b010 con 0x7fa4c406f850 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.479+0000 7fa4cb7d2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c40af360 con 0x7fa4c40aea10 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.479+0000 7fa4c8d6d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4c4074520 0x7fa4c40b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.479+0000 7fa4c8d6d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4c4074520 0x7fa4c40b0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35288/0 (socket says 172.21.15.81:35288) 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.479+0000 7fa4c8d6d640 1 -- 172.21.15.81:0/1300919588 learned_addr learned my addr 172.21.15.81:0/1300919588 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4c8d6d640 1 -- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4c40aea10 msgr2=0x7fa4c40b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4c8d6d640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4c40aea10 0x7fa4c40b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4c8d6d640 1 -- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4c406f850 msgr2=0x7fa4c41f12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:32.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4c8d6d640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4c406f850 0x7fa4c41f12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:32.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4c8d6d640 1 -- 172.21.15.81:0/1300919588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c414f050 con 0x7fa4c4074520 2024-06-29T15:57:32.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4c8d6d640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4c4074520 0x7fa4c40b0d80 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400c370 tx=0x7fa4b400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:32.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4b9ffb640 1 -- 172.21.15.81:0/1300919588 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4017070 con 0x7fa4c4074520 2024-06-29T15:57:32.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4b9ffb640 1 -- 172.21.15.81:0/1300919588 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4b40035f0 con 0x7fa4c4074520 2024-06-29T15:57:32.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/1300919588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c414f9a0 con 0x7fa4c4074520 2024-06-29T15:57:32.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4b9ffb640 1 -- 172.21.15.81:0/1300919588 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4003a40 con 0x7fa4c4074520 2024-06-29T15:57:32.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.480+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/1300919588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c414ff60 con 0x7fa4c4074520 2024-06-29T15:57:32.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.481+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/1300919588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c415e120 con 0x7fa4c4074520 2024-06-29T15:57:32.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.482+0000 7fa4b9ffb640 1 -- 172.21.15.81:0/1300919588 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fa4b401a070 con 0x7fa4c4074520 2024-06-29T15:57:32.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.483+0000 7fa4b9ffb640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa4a80763f0 0x7fa4a8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:32.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.483+0000 7fa4b9ffb640 1 -- 172.21.15.81:0/1300919588 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6210+0+0 (secure 0 0 0) 0x7fa4b4096df0 con 0x7fa4c4074520 2024-06-29T15:57:32.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.483+0000 7fa4c956e640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa4a80763f0 0x7fa4a8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:32.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.484+0000 7fa4c956e640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa4a80763f0 0x7fa4a8078900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0001550 tx=0x7fa4c0007970 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:32.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.487+0000 7fa4b9ffb640 1 -- 172.21.15.81:0/1300919588 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa4b4060440 con 0x7fa4c4074520 2024-06-29T15:57:32.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:32.666+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/1300919588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa4c406d5b0 con 0x7fa4c4074520 2024-06-29T15:57:33.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.344+0000 7fa4b9ffb640 1 -- 172.21.15.81:0/1300919588 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v215) ==== 176+0+0 (secure 0 0 0) 0x7fa4b4068430 con 0x7fa4c4074520 2024-06-29T15:57:33.346 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'backup_fs_data' created 2024-06-29T15:57:33.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa4a80763f0 msgr2=0x7fa4a8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:33.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa4a80763f0 0x7fa4a8078900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0001550 tx=0x7fa4c0007970 comp rx=0 tx=0).stop 2024-06-29T15:57:33.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/1300919588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4c4074520 msgr2=0x7fa4c40b0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:33.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4c4074520 0x7fa4c40b0d80 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400c370 tx=0x7fa4b400c830 comp rx=0 tx=0).stop 2024-06-29T15:57:33.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/1300919588 shutdown_connections 2024-06-29T15:57:33.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fa4a80763f0 0x7fa4a8078900 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa4c40aea10 0x7fa4c40b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4c4074520 0x7fa4c40b0d80 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 --2- 172.21.15.81:0/1300919588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa4c406f850 0x7fa4c41f12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/1300919588 >> 172.21.15.81:0/1300919588 conn(0x7fa4c4072950 msgr2=0x7fa4c4156400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:33.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/1300919588 shutdown_connections 2024-06-29T15:57:33.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.347+0000 7fa4cb7d2640 1 -- 172.21.15.81:0/1300919588 wait complete. 2024-06-29T15:57:33.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-06-29T15:57:33.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.490+0000 7f01709b9640 1 -- 172.21.15.81:0/3730622525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158005250 msgr2=0x7f015800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:33.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.490+0000 7f01709b9640 1 --2- 172.21.15.81:0/3730622525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158005250 0x7f015800b9c0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f0164001690 tx=0x7f0158005830 comp rx=0 tx=0).stop 2024-06-29T15:57:33.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.491+0000 7f01709b9640 1 -- 172.21.15.81:0/3730622525 shutdown_connections 2024-06-29T15:57:33.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.491+0000 7f01709b9640 1 --2- 172.21.15.81:0/3730622525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158005250 0x7f015800b9c0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.491+0000 7f01709b9640 1 -- 172.21.15.81:0/3730622525 >> 172.21.15.81:0/3730622525 conn(0x7f016c06ccc0 msgr2=0x7f016c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:33.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.491+0000 7f01709b9640 1 -- 172.21.15.81:0/3730622525 shutdown_connections 2024-06-29T15:57:33.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.491+0000 7f01709b9640 1 -- 172.21.15.81:0/3730622525 wait complete. 2024-06-29T15:57:33.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.493+0000 7f01709b9640 1 Processor -- start 2024-06-29T15:57:33.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.493+0000 7f01709b9640 1 -- start start 2024-06-29T15:57:33.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f01709b9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158005250 0x7f016c1f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:33.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f01709b9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f016c0aefd0 0x7f016c0abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:33.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f01709b9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f016c15d4f0 0x7f016c15d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:33.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f01709b9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f016c0737d0 con 0x7f0158005250 2024-06-29T15:57:33.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f01709b9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f016c073650 con 0x7f016c0aefd0 2024-06-29T15:57:33.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f01709b9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f016c15ed70 con 0x7f016c15d4f0 2024-06-29T15:57:33.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f016a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158005250 0x7f016c1f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:33.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f016a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158005250 0x7f016c1f17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35310/0 (socket says 172.21.15.81:35310) 2024-06-29T15:57:33.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f016a59c640 1 -- 172.21.15.81:0/3605656962 learned_addr learned my addr 172.21.15.81:0/3605656962 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:33.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f0169d9b640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f016c0aefd0 0x7f016c0abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f016ad9d640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f016c15d4f0 0x7f016c15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f016a59c640 1 -- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f016c15d4f0 msgr2=0x7f016c15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f016a59c640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f016c15d4f0 0x7f016c15d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f016a59c640 1 -- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f016c0aefd0 msgr2=0x7f016c0abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f016a59c640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f016c0aefd0 0x7f016c0abda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.494+0000 7f016a59c640 1 -- 172.21.15.81:0/3605656962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f016c15de40 con 0x7f0158005250 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.495+0000 7f016a59c640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158005250 0x7f016c1f17d0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f015c006c60 tx=0x7f015c0067a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.495+0000 7f0162ffd640 1 -- 172.21.15.81:0/3605656962 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c00fd10 con 0x7f0158005250 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.495+0000 7f0162ffd640 1 -- 172.21.15.81:0/3605656962 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f015c011430 con 0x7f0158005250 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.495+0000 7f01709b9640 1 -- 172.21.15.81:0/3605656962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f016c1f2090 con 0x7f0158005250 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.495+0000 7f0162ffd640 1 -- 172.21.15.81:0/3605656962 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c011880 con 0x7f0158005250 2024-06-29T15:57:33.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.495+0000 7f01709b9640 1 -- 172.21.15.81:0/3605656962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f016c1f2650 con 0x7f0158005250 2024-06-29T15:57:33.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.496+0000 7f01709b9640 1 -- 172.21.15.81:0/3605656962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f016c0cce40 con 0x7f0158005250 2024-06-29T15:57:33.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.497+0000 7f0162ffd640 1 -- 172.21.15.81:0/3605656962 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f015c011a20 con 0x7f0158005250 2024-06-29T15:57:33.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.497+0000 7f0162ffd640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f014c0763f0 0x7f014c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:33.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.497+0000 7f0162ffd640 1 -- 172.21.15.81:0/3605656962 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 6595+0+0 (secure 0 0 0) 0x7f015c098d60 con 0x7f0158005250 2024-06-29T15:57:33.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.497+0000 7f0169d9b640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f014c0763f0 0x7f014c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:33.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.498+0000 7f0169d9b640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f014c0763f0 0x7f014c078900 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f016c06eef0 tx=0x7f0154005f90 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:33.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.501+0000 7f0162ffd640 1 -- 172.21.15.81:0/3605656962 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f016c0cce40 con 0x7f0158005250 2024-06-29T15:57:33.663 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:57:33.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:57:33.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:33.725+0000 7f01709b9640 1 -- 172.21.15.81:0/3605656962 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) -- 0x7f016c0cef90 con 0x7f0158005250 2024-06-29T15:57:34.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.361+0000 7f0162ffd640 1 -- 172.21.15.81:0/3605656962 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 28 and data pool 29 v114) ==== 183+0+0 (secure 0 0 0) 0x7f015c028080 con 0x7f0158005250 2024-06-29T15:57:34.363 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 28 and data pool 29 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 -- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f014c0763f0 msgr2=0x7f014c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f014c0763f0 0x7f014c078900 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f016c06eef0 tx=0x7f0154005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 -- 172.21.15.81:0/3605656962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158005250 msgr2=0x7f016c1f17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158005250 0x7f016c1f17d0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f015c006c60 tx=0x7f015c0067a0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 -- 172.21.15.81:0/3605656962 shutdown_connections 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f014c0763f0 0x7f014c078900 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f016c06eef0 tx=0x7f0154005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f016c15d4f0 0x7f016c15d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f016c0aefd0 0x7f016c0abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 --2- 172.21.15.81:0/3605656962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0158005250 0x7f016c1f17d0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 -- 172.21.15.81:0/3605656962 >> 172.21.15.81:0/3605656962 conn(0x7f016c06ccc0 msgr2=0x7f016c143980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 -- 172.21.15.81:0/3605656962 shutdown_connections 2024-06-29T15:57:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.365+0000 7f01709b9640 1 -- 172.21.15.81:0/3605656962 wait complete. 2024-06-29T15:57:34.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:34.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.537+0000 7fc622647640 1 -- 172.21.15.81:0/3901667679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc61c1412f0 msgr2=0x7fc61c0cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.537+0000 7fc622647640 1 --2- 172.21.15.81:0/3901667679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc61c1412f0 0x7fc61c0cef90 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fc61c06ff00 tx=0x7fc60c02a560 comp rx=0 tx=0).stop 2024-06-29T15:57:34.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.537+0000 7fc622647640 1 -- 172.21.15.81:0/3901667679 shutdown_connections 2024-06-29T15:57:34.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.537+0000 7fc622647640 1 --2- 172.21.15.81:0/3901667679 >> v2:172.21.15.167:3300/0 conn(0x7fc61c073570 0x7fc61c15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.537+0000 7fc622647640 1 --2- 172.21.15.81:0/3901667679 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc61c0cf4d0 0x7fc61c073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.537+0000 7fc622647640 1 --2- 172.21.15.81:0/3901667679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc61c1412f0 0x7fc61c0cef90 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.537+0000 7fc622647640 1 -- 172.21.15.81:0/3901667679 >> 172.21.15.81:0/3901667679 conn(0x7fc61c06ccc0 msgr2=0x7fc61c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:34.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.537+0000 7fc622647640 1 -- 172.21.15.81:0/3901667679 shutdown_connections 2024-06-29T15:57:34.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.537+0000 7fc622647640 1 -- 172.21.15.81:0/3901667679 wait complete. 2024-06-29T15:57:34.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.538+0000 7fc622647640 1 Processor -- start 2024-06-29T15:57:34.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.539+0000 7fc622647640 1 -- start start 2024-06-29T15:57:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc622647640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc61c073570 0x7fc61c13e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc622647640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc61c0cf4d0 0x7fc61c156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc622647640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc61c1412f0 0x7fc61c0b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc622647640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c15f230 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc622647640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c15f0b0 con 0x7fc61c073570 2024-06-29T15:57:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc622647640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc61c0b3210 con 0x7fc61c1412f0 2024-06-29T15:57:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc61b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc61c0cf4d0 0x7fc61c156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc61b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc61c0cf4d0 0x7fc61c156860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35330/0 (socket says 172.21.15.81:35330) 2024-06-29T15:57:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc61b7fe640 1 -- 172.21.15.81:0/3578397512 learned_addr learned my addr 172.21.15.81:0/3578397512 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc61bfff640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc61c073570 0x7fc61c13e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc620be4640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc61c1412f0 0x7fc61c0b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc61b7fe640 1 -- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc61c1412f0 msgr2=0x7fc61c0b0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc61b7fe640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc61c1412f0 0x7fc61c0b0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc61b7fe640 1 -- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc61c073570 msgr2=0x7fc61c13e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc61b7fe640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc61c073570 0x7fc61c13e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc61b7fe640 1 -- 172.21.15.81:0/3578397512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c0a9bc0 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.540+0000 7fc61b7fe640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc61c0cf4d0 0x7fc61c156860 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c9a0 tx=0x7fc61000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.541+0000 7fc618ff9640 1 -- 172.21.15.81:0/3578397512 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610007bb0 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.541+0000 7fc618ff9640 1 -- 172.21.15.81:0/3578397512 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc610007e40 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.541+0000 7fc622647640 1 -- 172.21.15.81:0/3578397512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c0a9ea0 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.541+0000 7fc618ff9640 1 -- 172.21.15.81:0/3578397512 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61000f690 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.541+0000 7fc622647640 1 -- 172.21.15.81:0/3578397512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c0aa460 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.542+0000 7fc622647640 1 -- 172.21.15.81:0/3578397512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61c073030 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.542+0000 7fc618ff9640 1 -- 172.21.15.81:0/3578397512 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fc610018400 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.543+0000 7fc618ff9640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6080763f0 0x7fc608078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.543+0000 7fc618ff9640 1 -- 172.21.15.81:0/3578397512 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 6701+0+0 (secure 0 0 0) 0x7fc610098480 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.543+0000 7fc61bfff640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6080763f0 0x7fc608078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.544+0000 7fc61bfff640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6080763f0 0x7fc608078900 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc60c033000 tx=0x7fc60c004730 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:34.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.547+0000 7fc618ff9640 1 -- 172.21.15.81:0/3578397512 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc610014030 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.727+0000 7fc622647640 1 -- 172.21.15.81:0/3578397512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc61c15ec60 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.728+0000 7fc618ff9640 1 -- 172.21.15.81:0/3578397512 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v216) ==== 74+0+20678 (secure 0 0 0) 0x7fc610061100 con 0x7fc61c0cf4d0 2024-06-29T15:57:34.730 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":216,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:33.727860+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.259110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:32.675276+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:34.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 -- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6080763f0 msgr2=0x7fc608078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6080763f0 0x7fc608078900 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc60c033000 tx=0x7fc60c004730 comp rx=0 tx=0).stop 2024-06-29T15:57:34.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 -- 172.21.15.81:0/3578397512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc61c0cf4d0 msgr2=0x7fc61c156860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc61c0cf4d0 0x7fc61c156860 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c9a0 tx=0x7fc61000ce60 comp rx=0 tx=0).stop 2024-06-29T15:57:34.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 -- 172.21.15.81:0/3578397512 shutdown_connections 2024-06-29T15:57:34.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc6080763f0 0x7fc608078900 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc61c1412f0 0x7fc61c0b0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc61c0cf4d0 0x7fc61c156860 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 --2- 172.21.15.81:0/3578397512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc61c073570 0x7fc61c13e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 -- 172.21.15.81:0/3578397512 >> 172.21.15.81:0/3578397512 conn(0x7fc61c06ccc0 msgr2=0x7fc61c15e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:34.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 -- 172.21.15.81:0/3578397512 shutdown_connections 2024-06-29T15:57:34.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.731+0000 7fc622647640 1 -- 172.21.15.81:0/3578397512 wait complete. 2024-06-29T15:57:34.744 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:34.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.855+0000 7f86870d1640 1 -- 172.21.15.81:0/1995616738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86800ce920 msgr2=0x7f86800cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.855+0000 7f86870d1640 1 --2- 172.21.15.81:0/1995616738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86800ce920 0x7f86800cd480 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f8680073340 tx=0x7f867802a820 comp rx=0 tx=0).stop 2024-06-29T15:57:34.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.855+0000 7f86870d1640 1 -- 172.21.15.81:0/1995616738 shutdown_connections 2024-06-29T15:57:34.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.855+0000 7f86870d1640 1 --2- 172.21.15.81:0/1995616738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86800ce920 0x7f86800cd480 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.855+0000 7f86870d1640 1 --2- 172.21.15.81:0/1995616738 >> v2:172.21.15.167:3300/0 conn(0x7f86800b6eb0 0x7f86800d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.855+0000 7f86870d1640 1 -- 172.21.15.81:0/1995616738 >> 172.21.15.81:0/1995616738 conn(0x7f86800b3510 msgr2=0x7f86800b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:34.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.855+0000 7f86870d1640 1 -- 172.21.15.81:0/1995616738 shutdown_connections 2024-06-29T15:57:34.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.855+0000 7f86870d1640 1 -- 172.21.15.81:0/1995616738 wait complete. 2024-06-29T15:57:34.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.857+0000 7f86870d1640 1 Processor -- start 2024-06-29T15:57:34.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.857+0000 7f86870d1640 1 -- start start 2024-06-29T15:57:34.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.858+0000 7f86870d1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86800b6eb0 0x7f868014de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.858+0000 7f86870d1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86800ce510 0x7f86800c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.858+0000 7f86870d1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86800ce920 0x7f86800cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.858+0000 7f86870d1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f86800b4f20 con 0x7f86800ce920 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.858+0000 7f86870d1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f86800b29c0 con 0x7f86800ce510 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.858+0000 7f86870d1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f86800b50a0 con 0x7f86800b6eb0 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f868566e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86800ce920 0x7f86800cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f868566e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86800ce920 0x7f86800cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35334/0 (socket says 172.21.15.81:35334) 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f868566e640 1 -- 172.21.15.81:0/3780987012 learned_addr learned my addr 172.21.15.81:0/3780987012 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f8684e6d640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86800b6eb0 0x7f868014de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f8677fff640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86800ce510 0x7f86800c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f868566e640 1 -- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86800b6eb0 msgr2=0x7f868014de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f868566e640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86800b6eb0 0x7f868014de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f868566e640 1 -- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86800ce510 msgr2=0x7f86800c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f868566e640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86800ce510 0x7f86800c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:34.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f868566e640 1 -- 172.21.15.81:0/3780987012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868014e3b0 con 0x7f86800ce920 2024-06-29T15:57:34.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f868566e640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86800ce920 0x7f86800cc430 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f867802ae10 tx=0x7f8678032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:34.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f86757fa640 1 -- 172.21.15.81:0/3780987012 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678037040 con 0x7f86800ce920 2024-06-29T15:57:34.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f86757fa640 1 -- 172.21.15.81:0/3780987012 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8678042070 con 0x7f86800ce920 2024-06-29T15:57:34.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f86870d1640 1 -- 172.21.15.81:0/3780987012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8680164d50 con 0x7f86800ce920 2024-06-29T15:57:34.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f86757fa640 1 -- 172.21.15.81:0/3780987012 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86780042a0 con 0x7f86800ce920 2024-06-29T15:57:34.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.859+0000 7f86870d1640 1 -- 172.21.15.81:0/3780987012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8680165310 con 0x7f86800ce920 2024-06-29T15:57:34.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.860+0000 7f86870d1640 1 -- 172.21.15.81:0/3780987012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86800d02d0 con 0x7f86800ce920 2024-06-29T15:57:34.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.861+0000 7f86757fa640 1 -- 172.21.15.81:0/3780987012 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f8678004440 con 0x7f86800ce920 2024-06-29T15:57:34.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.862+0000 7f86757fa640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f866c0763f0 0x7f866c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:34.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.862+0000 7f86757fa640 1 -- 172.21.15.81:0/3780987012 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 6701+0+0 (secure 0 0 0) 0x7f86780b8cc0 con 0x7f86800ce920 2024-06-29T15:57:34.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.862+0000 7f8684e6d640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f866c0763f0 0x7f866c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:34.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.862+0000 7f8684e6d640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f866c0763f0 0x7f866c078900 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f867c005fd0 tx=0x7f867c005850 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:34.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:34.866+0000 7f86757fa640 1 -- 172.21.15.81:0/3780987012 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f868014e3b0 con 0x7f86800ce920 2024-06-29T15:57:35.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.046+0000 7f86870d1640 1 -- 172.21.15.81:0/3780987012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86800cdf00 con 0x7f86800ce920 2024-06-29T15:57:35.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.047+0000 7f86757fa640 1 -- 172.21.15.81:0/3780987012 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v216) ==== 74+0+20678 (secure 0 0 0) 0x7f8678081f90 con 0x7f86800ce920 2024-06-29T15:57:35.049 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:35.049 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":216,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:33.727860+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.259110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:32.675276+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:35.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.049+0000 7f86870d1640 1 -- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f866c0763f0 msgr2=0x7f866c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.049+0000 7f86870d1640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f866c0763f0 0x7f866c078900 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f867c005fd0 tx=0x7f867c005850 comp rx=0 tx=0).stop 2024-06-29T15:57:35.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.049+0000 7f86870d1640 1 -- 172.21.15.81:0/3780987012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86800ce920 msgr2=0x7f86800cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.049+0000 7f86870d1640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86800ce920 0x7f86800cc430 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f867802ae10 tx=0x7f8678032d10 comp rx=0 tx=0).stop 2024-06-29T15:57:35.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.049+0000 7f86870d1640 1 -- 172.21.15.81:0/3780987012 shutdown_connections 2024-06-29T15:57:35.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.049+0000 7f86870d1640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f866c0763f0 0x7f866c078900 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.049+0000 7f86870d1640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86800ce920 0x7f86800cc430 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.049+0000 7f86870d1640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86800ce510 0x7f86800c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.049+0000 7f86870d1640 1 --2- 172.21.15.81:0/3780987012 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86800b6eb0 0x7f868014de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.049+0000 7f86870d1640 1 -- 172.21.15.81:0/3780987012 >> 172.21.15.81:0/3780987012 conn(0x7f86800b3510 msgr2=0x7f868015fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:35.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.050+0000 7f86870d1640 1 -- 172.21.15.81:0/3780987012 shutdown_connections 2024-06-29T15:57:35.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.050+0000 7f86870d1640 1 -- 172.21.15.81:0/3780987012 wait complete. 2024-06-29T15:57:35.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-06-29T15:57:35.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.177+0000 7f089ce3f640 1 -- 172.21.15.81:0/1636093346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08980732a0 msgr2=0x7f08980736b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.177+0000 7f089ce3f640 1 --2- 172.21.15.81:0/1636093346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08980732a0 0x7f08980736b0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f0890005fd0 tx=0x7f089002a5c0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.177+0000 7f089ce3f640 1 -- 172.21.15.81:0/1636093346 shutdown_connections 2024-06-29T15:57:35.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.177+0000 7f089ce3f640 1 --2- 172.21.15.81:0/1636093346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08980732a0 0x7f08980736b0 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.177+0000 7f089ce3f640 1 --2- 172.21.15.81:0/1636093346 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08981451c0 0x7f0898144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.177+0000 7f089ce3f640 1 -- 172.21.15.81:0/1636093346 >> 172.21.15.81:0/1636093346 conn(0x7f08980b1ce0 msgr2=0x7f08980b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:35.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.177+0000 7f089ce3f640 1 -- 172.21.15.81:0/1636093346 shutdown_connections 2024-06-29T15:57:35.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.177+0000 7f089ce3f640 1 -- 172.21.15.81:0/1636093346 wait complete. 2024-06-29T15:57:35.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.179+0000 7f089ce3f640 1 Processor -- start 2024-06-29T15:57:35.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.179+0000 7f089ce3f640 1 -- start start 2024-06-29T15:57:35.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.180+0000 7f089ce3f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08980732a0 0x7f089815b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.180+0000 7f089ce3f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08980cf6e0 0x7f08980addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.180+0000 7f089ce3f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08981451c0 0x7f089813ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.180+0000 7f089ce3f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f089815c610 con 0x7f08980732a0 2024-06-29T15:57:35.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.180+0000 7f089ce3f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f089815c490 con 0x7f08981451c0 2024-06-29T15:57:35.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.180+0000 7f089ce3f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f089815c790 con 0x7f08980cf6e0 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.180+0000 7f089659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08980732a0 0x7f089815b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.180+0000 7f089659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08980732a0 0x7f089815b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35346/0 (socket says 172.21.15.81:35346) 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.180+0000 7f089659c640 1 -- 172.21.15.81:0/4080753439 learned_addr learned my addr 172.21.15.81:0/4080753439 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f0895d9b640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08980cf6e0 0x7f08980addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f0896d9d640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08981451c0 0x7f089813ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f089659c640 1 -- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08980cf6e0 msgr2=0x7f08980addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f089659c640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08980cf6e0 0x7f08980addf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f089659c640 1 -- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08981451c0 msgr2=0x7f089813ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f089659c640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08981451c0 0x7f089813ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f089659c640 1 -- 172.21.15.81:0/4080753439 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089813cfb0 con 0x7f08980732a0 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f0896d9d640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08981451c0 0x7f089813ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:35.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f089659c640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08980732a0 0x7f089815b680 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f088800ea20 tx=0x7f088800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:35.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f0886ffd640 1 -- 172.21.15.81:0/4080753439 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088800cde0 con 0x7f08980732a0 2024-06-29T15:57:35.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f0886ffd640 1 -- 172.21.15.81:0/4080753439 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0888004c10 con 0x7f08980732a0 2024-06-29T15:57:35.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f089ce3f640 1 -- 172.21.15.81:0/4080753439 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f089813d290 con 0x7f08980732a0 2024-06-29T15:57:35.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f0886ffd640 1 -- 172.21.15.81:0/4080753439 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0888005020 con 0x7f08980732a0 2024-06-29T15:57:35.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.181+0000 7f089ce3f640 1 -- 172.21.15.81:0/4080753439 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f089813d850 con 0x7f08980732a0 2024-06-29T15:57:35.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.182+0000 7f089ce3f640 1 -- 172.21.15.81:0/4080753439 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f089815cc40 con 0x7f08980732a0 2024-06-29T15:57:35.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.183+0000 7f0886ffd640 1 -- 172.21.15.81:0/4080753439 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f08880051c0 con 0x7f08980732a0 2024-06-29T15:57:35.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.183+0000 7f0886ffd640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f08780763f0 0x7f0878078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:35.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.183+0000 7f0886ffd640 1 -- 172.21.15.81:0/4080753439 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 6701+0+0 (secure 0 0 0) 0x7f0888098630 con 0x7f08980732a0 2024-06-29T15:57:35.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.184+0000 7f0895d9b640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f08780763f0 0x7f0878078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:35.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.184+0000 7f0895d9b640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f08780763f0 0x7f0878078900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f089813ec20 tx=0x7f088c009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:35.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.188+0000 7f0886ffd640 1 -- 172.21.15.81:0/4080753439 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0888016030 con 0x7f08980732a0 2024-06-29T15:57:35.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:35.409+0000 7f089ce3f640 1 -- 172.21.15.81:0/4080753439 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f08980ae330 con 0x7f08980732a0 2024-06-29T15:57:36.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.376+0000 7f0886ffd640 1 -- 172.21.15.81:0/4080753439 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v117) ==== 121+0+0 (secure 0 0 0) 0x7f0888061340 con 0x7f08980732a0 2024-06-29T15:57:36.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 -- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f08780763f0 msgr2=0x7f0878078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:36.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f08780763f0 0x7f0878078900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f089813ec20 tx=0x7f088c009040 comp rx=0 tx=0).stop 2024-06-29T15:57:36.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 -- 172.21.15.81:0/4080753439 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08980732a0 msgr2=0x7f089815b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:36.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08980732a0 0x7f089815b680 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f088800ea20 tx=0x7f088800eee0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 -- 172.21.15.81:0/4080753439 shutdown_connections 2024-06-29T15:57:36.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f08780763f0 0x7f0878078900 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f08981451c0 0x7f089813ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f08980cf6e0 0x7f08980addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 --2- 172.21.15.81:0/4080753439 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08980732a0 0x7f089815b680 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 -- 172.21.15.81:0/4080753439 >> 172.21.15.81:0/4080753439 conn(0x7f08980b1ce0 msgr2=0x7f0898140510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:36.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 -- 172.21.15.81:0/4080753439 shutdown_connections 2024-06-29T15:57:36.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.378+0000 7f089ce3f640 1 -- 172.21.15.81:0/4080753439 wait complete. 2024-06-29T15:57:36.392 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:57:36.392 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-06-29T15:57:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.507+0000 7f62e7681640 1 -- 172.21.15.81:0/571551790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e00b3960 msgr2=0x7f62e00b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.507+0000 7f62e7681640 1 --2- 172.21.15.81:0/571551790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e00b3960 0x7f62e00b3d70 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009830 tx=0x7f62d802a530 comp rx=0 tx=0).stop 2024-06-29T15:57:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.507+0000 7f62e7681640 1 -- 172.21.15.81:0/571551790 shutdown_connections 2024-06-29T15:57:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.507+0000 7f62e7681640 1 --2- 172.21.15.81:0/571551790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e00b3960 0x7f62e00b3d70 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.507+0000 7f62e7681640 1 --2- 172.21.15.81:0/571551790 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62e006f6a0 0x7f62e015aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.507+0000 7f62e7681640 1 -- 172.21.15.81:0/571551790 >> 172.21.15.81:0/571551790 conn(0x7f62e00d0930 msgr2=0x7f62e00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.507+0000 7f62e7681640 1 -- 172.21.15.81:0/571551790 shutdown_connections 2024-06-29T15:57:36.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.507+0000 7f62e7681640 1 -- 172.21.15.81:0/571551790 wait complete. 2024-06-29T15:57:36.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.509+0000 7f62e7681640 1 Processor -- start 2024-06-29T15:57:36.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.509+0000 7f62e7681640 1 -- start start 2024-06-29T15:57:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e7681640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62e006f6a0 0x7f62e01f5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:36.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e7681640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62e00b3960 0x7f62e0145580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e7681640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e00b57b0 0x7f62e006c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e7681640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62e00b4640 con 0x7f62e00b57b0 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e7681640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f62e00b44c0 con 0x7f62e00b3960 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e7681640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f62e00b47c0 con 0x7f62e006f6a0 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e5c1e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e00b57b0 0x7f62e006c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e5c1e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e00b57b0 0x7f62e006c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35362/0 (socket says 172.21.15.81:35362) 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e5c1e640 1 -- 172.21.15.81:0/3604535516 learned_addr learned my addr 172.21.15.81:0/3604535516 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e541d640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62e006f6a0 0x7f62e01f5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e4c1c640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62e00b3960 0x7f62e0145580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e5c1e640 1 -- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62e006f6a0 msgr2=0x7f62e01f5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e5c1e640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62e006f6a0 0x7f62e01f5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e5c1e640 1 -- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62e00b3960 msgr2=0x7f62e0145580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e5c1e640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62e00b3960 0x7f62e0145580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.510+0000 7f62e5c1e640 1 -- 172.21.15.81:0/3604535516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e00aa7d0 con 0x7f62e00b57b0 2024-06-29T15:57:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.511+0000 7f62e5c1e640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e00b57b0 0x7f62e006c2f0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f62d802ab20 tx=0x7f62d8033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.511+0000 7f62d5ffb640 1 -- 172.21.15.81:0/3604535516 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8031d60 con 0x7f62e00b57b0 2024-06-29T15:57:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.511+0000 7f62d5ffb640 1 -- 172.21.15.81:0/3604535516 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62d8005020 con 0x7f62e00b57b0 2024-06-29T15:57:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.511+0000 7f62e7681640 1 -- 172.21.15.81:0/3604535516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e01f6350 con 0x7f62e00b57b0 2024-06-29T15:57:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.511+0000 7f62d5ffb640 1 -- 172.21.15.81:0/3604535516 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8005470 con 0x7f62e00b57b0 2024-06-29T15:57:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.511+0000 7f62e7681640 1 -- 172.21.15.81:0/3604535516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62e0146540 con 0x7f62e00b57b0 2024-06-29T15:57:36.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.512+0000 7f62e7681640 1 -- 172.21.15.81:0/3604535516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62e00c9e60 con 0x7f62e00b57b0 2024-06-29T15:57:36.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.513+0000 7f62d5ffb640 1 -- 172.21.15.81:0/3604535516 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f62d80047a0 con 0x7f62e00b57b0 2024-06-29T15:57:36.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.513+0000 7f62d5ffb640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62c4076420 0x7f62c4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:36.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.513+0000 7f62d5ffb640 1 -- 172.21.15.81:0/3604535516 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7f62d80b8f60 con 0x7f62e00b57b0 2024-06-29T15:57:36.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.514+0000 7f62e541d640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62c4076420 0x7f62c4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:36.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.514+0000 7f62e541d640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62c4076420 0x7f62c4078930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f62dc002200 tx=0x7f62dc0019f0 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:36.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.518+0000 7f62d5ffb640 1 -- 172.21.15.81:0/3604535516 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f62d803f030 con 0x7f62e00b57b0 2024-06-29T15:57:36.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:36.734+0000 7f62e7681640 1 -- 172.21.15.81:0/3604535516 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f62e0145ff0 con 0x7f62e00b57b0 2024-06-29T15:57:37.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.379+0000 7f62d5ffb640 1 -- 172.21.15.81:0/3604535516 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v118) ==== 125+0+0 (secure 0 0 0) 0x7f62d808a2b0 con 0x7f62e00b57b0 2024-06-29T15:57:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 -- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62c4076420 msgr2=0x7f62c4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62c4076420 0x7f62c4078930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f62dc002200 tx=0x7f62dc0019f0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 -- 172.21.15.81:0/3604535516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e00b57b0 msgr2=0x7f62e006c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e00b57b0 0x7f62e006c2f0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f62d802ab20 tx=0x7f62d8033ce0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 -- 172.21.15.81:0/3604535516 shutdown_connections 2024-06-29T15:57:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62c4076420 0x7f62c4078930 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e00b57b0 0x7f62e006c2f0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62e00b3960 0x7f62e0145580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 --2- 172.21.15.81:0/3604535516 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62e006f6a0 0x7f62e01f5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 -- 172.21.15.81:0/3604535516 >> 172.21.15.81:0/3604535516 conn(0x7f62e00d0930 msgr2=0x7f62e0159f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 -- 172.21.15.81:0/3604535516 shutdown_connections 2024-06-29T15:57:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.382+0000 7f62e7681640 1 -- 172.21.15.81:0/3604535516 wait complete. 2024-06-29T15:57:37.395 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-06-29T15:57:37.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:37.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.508+0000 7ff0b437a640 1 -- 172.21.15.81:0/187190844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0980063b0 msgr2=0x7ff09800cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.508+0000 7ff0b437a640 1 --2- 172.21.15.81:0/187190844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0980063b0 0x7ff09800cad0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400a990 tx=0x7ff098006940 comp rx=0 tx=0).stop 2024-06-29T15:57:37.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.508+0000 7ff0b2917640 1 -- 172.21.15.81:0/187190844 reap_dead start 2024-06-29T15:57:37.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.509+0000 7ff0b437a640 1 -- 172.21.15.81:0/187190844 shutdown_connections 2024-06-29T15:57:37.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.509+0000 7ff0b437a640 1 -- 172.21.15.81:0/187190844 >> 172.21.15.81:0/187190844 conn(0x7ff0ac06ccc0 msgr2=0x7ff0ac0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:37.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.509+0000 7ff0b437a640 1 -- 172.21.15.81:0/187190844 shutdown_connections 2024-06-29T15:57:37.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.509+0000 7ff0b437a640 1 -- 172.21.15.81:0/187190844 wait complete. 2024-06-29T15:57:37.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.510+0000 7ff0b437a640 1 Processor -- start 2024-06-29T15:57:37.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.510+0000 7ff0b437a640 1 -- start start 2024-06-29T15:57:37.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b437a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0980063b0 0x7ff0ac146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b437a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0ac0b4870 0x7ff0ac14e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b437a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0ac14ec40 0x7ff0ac143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b437a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac06fef0 con 0x7ff0ac0b4870 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b437a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac06fd70 con 0x7ff0980063b0 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b437a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff0ac06edd0 con 0x7ff0ac14ec40 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b1915640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0ac0b4870 0x7ff0ac14e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b1915640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0ac0b4870 0x7ff0ac14e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35388/0 (socket says 172.21.15.81:35388) 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b1915640 1 -- 172.21.15.81:0/3561602426 learned_addr learned my addr 172.21.15.81:0/3561602426 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b2917640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0ac14ec40 0x7ff0ac143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b2116640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0980063b0 0x7ff0ac146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b1915640 1 -- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0ac14ec40 msgr2=0x7ff0ac143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b1915640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0ac14ec40 0x7ff0ac143bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b1915640 1 -- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0980063b0 msgr2=0x7ff0ac146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b1915640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0980063b0 0x7ff0ac146640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b1915640 1 -- 172.21.15.81:0/3561602426 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac14f050 con 0x7ff0ac0b4870 2024-06-29T15:57:37.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b1915640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0ac0b4870 0x7ff0ac14e700 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00e630 tx=0x7ff09c00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:37.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0a2ffd640 1 -- 172.21.15.81:0/3561602426 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c015070 con 0x7ff0ac0b4870 2024-06-29T15:57:37.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0a2ffd640 1 -- 172.21.15.81:0/3561602426 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff09c004310 con 0x7ff0ac0b4870 2024-06-29T15:57:37.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.512+0000 7ff0b437a640 1 -- 172.21.15.81:0/3561602426 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0ac1516f0 con 0x7ff0ac0b4870 2024-06-29T15:57:37.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.513+0000 7ff0a2ffd640 1 -- 172.21.15.81:0/3561602426 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00ca00 con 0x7ff0ac0b4870 2024-06-29T15:57:37.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.513+0000 7ff0b437a640 1 -- 172.21.15.81:0/3561602426 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0ac151cb0 con 0x7ff0ac0b4870 2024-06-29T15:57:37.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.513+0000 7ff0b437a640 1 -- 172.21.15.81:0/3561602426 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0ac151460 con 0x7ff0ac0b4870 2024-06-29T15:57:37.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.514+0000 7ff0a2ffd640 1 -- 172.21.15.81:0/3561602426 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7ff09c00cba0 con 0x7ff0ac0b4870 2024-06-29T15:57:37.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.515+0000 7ff0a2ffd640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff0900763f0 0x7ff090078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.515+0000 7ff0a2ffd640 1 -- 172.21.15.81:0/3561602426 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7ff09c098120 con 0x7ff0ac0b4870 2024-06-29T15:57:37.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.515+0000 7ff0b2116640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff0900763f0 0x7ff090078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.515+0000 7ff0b2116640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff0900763f0 0x7ff090078900 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8006870 tx=0x7ff0a8006ef0 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:37.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.519+0000 7ff0a2ffd640 1 -- 172.21.15.81:0/3561602426 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff0ac14f050 con 0x7ff0ac0b4870 2024-06-29T15:57:37.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.734+0000 7ff0b437a640 1 -- 172.21.15.81:0/3561602426 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0ac15ab20 con 0x7ff0ac0b4870 2024-06-29T15:57:37.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.735+0000 7ff0a2ffd640 1 -- 172.21.15.81:0/3561602426 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+6565 (secure 0 0 0) 0x7ff09c060e40 con 0x7ff0ac0b4870 2024-06-29T15:57:37.736 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:37.737 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":118,"btime":"2024-06-29T15:57:37:378022+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.637042+0000","modified":"2024-06-29T15:57:24.656503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11226":{"gid":11226,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/739691411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":739691411},{"type":"v1","addr":"172.21.15.81:6833","nonce":739691411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11226,"qdb_cluster":[11226]},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:33.727815+0000","modified":"2024-06-29T15:57:36.736776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10114":{"gid":10114,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10114,"qdb_cluster":[10114]},"id":14}]} 2024-06-29T15:57:37.737 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 118 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 -- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff0900763f0 msgr2=0x7ff090078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff0900763f0 0x7ff090078900 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8006870 tx=0x7ff0a8006ef0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 -- 172.21.15.81:0/3561602426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0ac0b4870 msgr2=0x7ff0ac14e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0ac0b4870 0x7ff0ac14e700 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00e630 tx=0x7ff09c00eaf0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 -- 172.21.15.81:0/3561602426 shutdown_connections 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff0900763f0 0x7ff090078900 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff0ac14ec40 0x7ff0ac143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0ac0b4870 0x7ff0ac14e700 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 --2- 172.21.15.81:0/3561602426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff0980063b0 0x7ff0ac146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 -- 172.21.15.81:0/3561602426 >> 172.21.15.81:0/3561602426 conn(0x7ff0ac06ccc0 msgr2=0x7ff0ac0ce150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:37.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 -- 172.21.15.81:0/3561602426 shutdown_connections 2024-06-29T15:57:37.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.737+0000 7ff0b437a640 1 -- 172.21.15.81:0/3561602426 wait complete. 2024-06-29T15:57:37.751 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:37.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.863+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/1126229964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ae4073740 msgr2=0x7f0ae415e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.863+0000 7f0aeb6cc640 1 --2- 172.21.15.81:0/1126229964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ae4073740 0x7f0ae415e960 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4070e50 tx=0x7f0adc02aac0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.863+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/1126229964 shutdown_connections 2024-06-29T15:57:37.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.863+0000 7f0aeb6cc640 1 --2- 172.21.15.81:0/1126229964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ae4073740 0x7f0ae415e960 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.863+0000 7f0aeb6cc640 1 --2- 172.21.15.81:0/1126229964 >> v2:172.21.15.167:3300/0 conn(0x7f0ae41412f0 0x7f0ae40cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.863+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/1126229964 >> 172.21.15.81:0/1126229964 conn(0x7f0ae406ccc0 msgr2=0x7f0ae40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:37.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.863+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/1126229964 shutdown_connections 2024-06-29T15:57:37.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.863+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/1126229964 wait complete. 2024-06-29T15:57:37.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.865+0000 7f0aeb6cc640 1 Processor -- start 2024-06-29T15:57:37.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.865+0000 7f0aeb6cc640 1 -- start start 2024-06-29T15:57:37.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0aeb6cc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ae40cf4d0 0x7f0ae4140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0aeb6cc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ae41412f0 0x7f0ae41f52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0aeb6cc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ae41f5830 0x7f0ae41f5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0aeb6cc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae40b3210 con 0x7f0ae41412f0 2024-06-29T15:57:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0aeb6cc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae415f270 con 0x7f0ae41f5830 2024-06-29T15:57:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0aeb6cc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae40b3390 con 0x7f0ae40cf4d0 2024-06-29T15:57:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae8c67640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ae41412f0 0x7f0ae41f52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae8c67640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ae41412f0 0x7f0ae41f52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35416/0 (socket says 172.21.15.81:35416) 2024-06-29T15:57:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae8c67640 1 -- 172.21.15.81:0/553766147 learned_addr learned my addr 172.21.15.81:0/553766147 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae9468640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ae40cf4d0 0x7f0ae4140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae9c69640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ae41f5830 0x7f0ae41f5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae8c67640 1 -- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ae40cf4d0 msgr2=0x7f0ae4140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae8c67640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ae40cf4d0 0x7f0ae4140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae8c67640 1 -- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ae41f5830 msgr2=0x7f0ae41f5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae8c67640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ae41f5830 0x7f0ae41f5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae8c67640 1 -- 172.21.15.81:0/553766147 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae40af350 con 0x7f0ae41412f0 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.866+0000 7f0ae9468640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ae40cf4d0 0x7f0ae4140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.867+0000 7f0ae8c67640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ae41412f0 0x7f0ae41f52f0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400c3a0 tx=0x7f0ad400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.867+0000 7f0ad9ffb640 1 -- 172.21.15.81:0/553766147 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4017070 con 0x7f0ae41412f0 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.867+0000 7f0ad9ffb640 1 -- 172.21.15.81:0/553766147 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ad40035f0 con 0x7f0ae41412f0 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.867+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/553766147 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae40aed90 con 0x7f0ae41412f0 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.867+0000 7f0ad9ffb640 1 -- 172.21.15.81:0/553766147 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4003a40 con 0x7f0ae41412f0 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.867+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/553766147 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae40aa620 con 0x7f0ae41412f0 2024-06-29T15:57:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.868+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/553766147 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae40b38b0 con 0x7f0ae41412f0 2024-06-29T15:57:37.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.869+0000 7f0ad9ffb640 1 -- 172.21.15.81:0/553766147 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f0ad401a070 con 0x7f0ae41412f0 2024-06-29T15:57:37.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.869+0000 7f0ad9ffb640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0ac8076420 0x7f0ac8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:37.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.869+0000 7f0ad9ffb640 1 -- 172.21.15.81:0/553766147 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7f0ad4097050 con 0x7f0ae41412f0 2024-06-29T15:57:37.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.869+0000 7f0ae9468640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0ac8076420 0x7f0ac8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:37.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.870+0000 7f0ae9468640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0ac8076420 0x7f0ac8078930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009b80 tx=0x7f0ae0005f90 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:37.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:37.874+0000 7f0ad9ffb640 1 -- 172.21.15.81:0/553766147 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0ad40604c0 con 0x7f0ae41412f0 2024-06-29T15:57:38.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.054+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/553766147 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ae40ae5c0 con 0x7f0ae41412f0 2024-06-29T15:57:38.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.055+0000 7f0ad9ffb640 1 -- 172.21.15.81:0/553766147 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) ==== 74+0+20665 (secure 0 0 0) 0x7f0ad405fdc0 con 0x7f0ae41412f0 2024-06-29T15:57:38.057 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:38.057 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":217,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:35.347681+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.259110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:32.675276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0ac8076420 msgr2=0x7f0ac8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0ac8076420 0x7f0ac8078930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009b80 tx=0x7f0ae0005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/553766147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ae41412f0 msgr2=0x7f0ae41f52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ae41412f0 0x7f0ae41f52f0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400c3a0 tx=0x7f0ad400c860 comp rx=0 tx=0).stop 2024-06-29T15:57:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/553766147 shutdown_connections 2024-06-29T15:57:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0ac8076420 0x7f0ac8078930 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ae41f5830 0x7f0ae41f5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ae41412f0 0x7f0ae41f52f0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 --2- 172.21.15.81:0/553766147 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ae40cf4d0 0x7f0ae4140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/553766147 >> 172.21.15.81:0/553766147 conn(0x7f0ae406ccc0 msgr2=0x7f0ae415e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/553766147 shutdown_connections 2024-06-29T15:57:38.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.057+0000 7f0aeb6cc640 1 -- 172.21.15.81:0/553766147 wait complete. 2024-06-29T15:57:38.070 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:57:38.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:38.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.183+0000 7fc81f2b3640 1 -- 172.21.15.81:0/695621575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc804003a50 msgr2=0x7fc804003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.183+0000 7fc81f2b3640 1 --2- 172.21.15.81:0/695621575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc804003a50 0x7fc804003ec0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7fc81400c310 tx=0x7fc804005830 comp rx=0 tx=0).stop 2024-06-29T15:57:38.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.183+0000 7fc81f2b3640 1 -- 172.21.15.81:0/695621575 shutdown_connections 2024-06-29T15:57:38.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.183+0000 7fc81f2b3640 1 --2- 172.21.15.81:0/695621575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc804003a50 0x7fc804003ec0 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.183+0000 7fc81f2b3640 1 -- 172.21.15.81:0/695621575 >> 172.21.15.81:0/695621575 conn(0x7fc81809cf30 msgr2=0x7fc8180d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:38.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.183+0000 7fc81f2b3640 1 -- 172.21.15.81:0/695621575 shutdown_connections 2024-06-29T15:57:38.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.183+0000 7fc81f2b3640 1 -- 172.21.15.81:0/695621575 wait complete. 2024-06-29T15:57:38.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.185+0000 7fc81f2b3640 1 Processor -- start 2024-06-29T15:57:38.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.185+0000 7fc81f2b3640 1 -- start start 2024-06-29T15:57:38.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81f2b3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc804003a50 0x7fc8181e7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81f2b3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc818156cb0 0x7fc818152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81f2b3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8181e8dd0 0x7fc8181e91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81f2b3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc8180d0860 con 0x7fc818156cb0 2024-06-29T15:57:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81f2b3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc8180d06e0 con 0x7fc804003a50 2024-06-29T15:57:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81f2b3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc8180d09e0 con 0x7fc8181e8dd0 2024-06-29T15:57:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81c84e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc818156cb0 0x7fc818152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81c84e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc818156cb0 0x7fc818152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35432/0 (socket says 172.21.15.81:35432) 2024-06-29T15:57:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81c84e640 1 -- 172.21.15.81:0/1080707341 learned_addr learned my addr 172.21.15.81:0/1080707341 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81d04f640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc804003a50 0x7fc8181e7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81d850640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8181e8dd0 0x7fc8181e91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81c84e640 1 -- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8181e8dd0 msgr2=0x7fc8181e91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81c84e640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8181e8dd0 0x7fc8181e91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81c84e640 1 -- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc804003a50 msgr2=0x7fc8181e7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81c84e640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc804003a50 0x7fc8181e7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81c84e640 1 -- 172.21.15.81:0/1080707341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8181e9720 con 0x7fc818156cb0 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81d04f640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc804003a50 0x7fc8181e7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.186+0000 7fc81d850640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8181e8dd0 0x7fc8181e91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.187+0000 7fc81c84e640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc818156cb0 0x7fc818152fa0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fc808006690 tx=0x7fc8080016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.187+0000 7fc80dffb640 1 -- 172.21.15.81:0/1080707341 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808015020 con 0x7fc818156cb0 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.187+0000 7fc80dffb640 1 -- 172.21.15.81:0/1080707341 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc808011900 con 0x7fc818156cb0 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.187+0000 7fc80dffb640 1 -- 172.21.15.81:0/1080707341 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808011d50 con 0x7fc818156cb0 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.187+0000 7fc81f2b3640 1 -- 172.21.15.81:0/1080707341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8181e6e30 con 0x7fc818156cb0 2024-06-29T15:57:38.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.187+0000 7fc81f2b3640 1 -- 172.21.15.81:0/1080707341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8181e73f0 con 0x7fc818156cb0 2024-06-29T15:57:38.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.188+0000 7fc81f2b3640 1 -- 172.21.15.81:0/1080707341 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8180b7290 con 0x7fc818156cb0 2024-06-29T15:57:38.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.188+0000 7fc80dffb640 1 -- 172.21.15.81:0/1080707341 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fc808018020 con 0x7fc818156cb0 2024-06-29T15:57:38.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.189+0000 7fc80dffb640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc7fc0763f0 0x7fc7fc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.189+0000 7fc80dffb640 1 -- 172.21.15.81:0/1080707341 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7fc808098850 con 0x7fc818156cb0 2024-06-29T15:57:38.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.189+0000 7fc81d04f640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc7fc0763f0 0x7fc7fc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.190+0000 7fc81d04f640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc7fc0763f0 0x7fc7fc078900 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc814033000 tx=0x7fc814001a60 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:38.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.193+0000 7fc80dffb640 1 -- 172.21.15.81:0/1080707341 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc80801f020 con 0x7fc818156cb0 2024-06-29T15:57:38.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.374+0000 7fc81f2b3640 1 -- 172.21.15.81:0/1080707341 --> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc81815cdc0 con 0x7fc7fc0763f0 2024-06-29T15:57:38.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.384+0000 7fc80dffb640 1 -- 172.21.15.81:0/1080707341 <== mgr.8666 v2:172.21.15.167:6832/2018755589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+486086 (secure 0 0 0) 0x7fc8140baae0 con 0x7fc7fc0763f0 2024-06-29T15:57:38.391 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:38.394 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 -- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc7fc0763f0 msgr2=0x7fc7fc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc7fc0763f0 0x7fc7fc078900 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc814033000 tx=0x7fc814001a60 comp rx=0 tx=0).stop 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 -- 172.21.15.81:0/1080707341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc818156cb0 msgr2=0x7fc818152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc818156cb0 0x7fc818152fa0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fc808006690 tx=0x7fc8080016f0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 -- 172.21.15.81:0/1080707341 shutdown_connections 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fc7fc0763f0 0x7fc7fc078900 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8181e8dd0 0x7fc8181e91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc818156cb0 0x7fc818152fa0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 --2- 172.21.15.81:0/1080707341 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc804003a50 0x7fc8181e7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 -- 172.21.15.81:0/1080707341 >> 172.21.15.81:0/1080707341 conn(0x7fc81809cf30 msgr2=0x7fc818068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:38.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 -- 172.21.15.81:0/1080707341 shutdown_connections 2024-06-29T15:57:38.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.388+0000 7fc81f2b3640 1 -- 172.21.15.81:0/1080707341 wait complete. 2024-06-29T15:57:38.401 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-06-29T15:57:37.873180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594021,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":400,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":217,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413288,"kb_used_data":22276,"kb_used_omap":12,"kb_used_meta":390899,"kb_avail":754561432,"statfs":{"total":773094113280,"available":772670906368,"internally_reserved":0,"allocated":22810624,"data_stored":11734666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":400281138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1243,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.532321"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357377+0000","last_change":"2024-06-29T15:57:33.357377+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357377+0000","last_undegraded":"2024-06-29T15:57:33.357377+0000","last_fullsized":"2024-06-29T15:57:33.357377+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"202'219","reported_seq":1043,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363727+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:57:33.363727+0000","last_peered":"2024-06-29T15:57:33.363727+0000","last_clean":"2024-06-29T15:57:33.363727+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:57:33.363727+0000","last_undegraded":"2024-06-29T15:57:33.363727+0000","last_fullsized":"2024-06-29T15:57:33.363727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365224+0000","last_change":"2024-06-29T15:57:21.282026+0000","last_active":"2024-06-29T15:57:33.365224+0000","last_peered":"2024-06-29T15:57:33.365224+0000","last_clean":"2024-06-29T15:57:33.365224+0000","last_became_active":"2024-06-29T15:57:21.281709+0000","last_became_peered":"2024-06-29T15:57:21.281709+0000","last_unstale":"2024-06-29T15:57:33.365224+0000","last_undegraded":"2024-06-29T15:57:33.365224+0000","last_fullsized":"2024-06-29T15:57:33.365224+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:54:07.641300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354339+0000","last_change":"2024-06-29T15:57:22.302267+0000","last_active":"2024-06-29T15:57:33.354339+0000","last_peered":"2024-06-29T15:57:33.354339+0000","last_clean":"2024-06-29T15:57:33.354339+0000","last_became_active":"2024-06-29T15:57:22.301972+0000","last_became_peered":"2024-06-29T15:57:22.301972+0000","last_unstale":"2024-06-29T15:57:33.354339+0000","last_undegraded":"2024-06-29T15:57:33.354339+0000","last_fullsized":"2024-06-29T15:57:33.354339+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371713+0000","last_change":"2024-06-29T15:57:33.370756+0000","last_active":"2024-06-29T15:57:33.371713+0000","last_peered":"2024-06-29T15:57:33.371713+0000","last_clean":"2024-06-29T15:57:33.371713+0000","last_became_active":"2024-06-29T15:57:33.370363+0000","last_became_peered":"2024-06-29T15:57:33.370363+0000","last_unstale":"2024-06-29T15:57:33.371713+0000","last_undegraded":"2024-06-29T15:57:33.371713+0000","last_fullsized":"2024-06-29T15:57:33.371713+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:27.582662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362213+0000","last_change":"2024-06-29T15:57:33.362213+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362213+0000","last_undegraded":"2024-06-29T15:57:33.362213+0000","last_fullsized":"2024-06-29T15:57:33.362213+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364223+0000","last_change":"2024-06-29T15:57:21.283495+0000","last_active":"2024-06-29T15:57:33.364223+0000","last_peered":"2024-06-29T15:57:33.364223+0000","last_clean":"2024-06-29T15:57:33.364223+0000","last_became_active":"2024-06-29T15:57:21.283125+0000","last_became_peered":"2024-06-29T15:57:21.283125+0000","last_unstale":"2024-06-29T15:57:33.364223+0000","last_undegraded":"2024-06-29T15:57:33.364223+0000","last_fullsized":"2024-06-29T15:57:33.364223+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354722+0000","last_change":"2024-06-29T15:57:22.292872+0000","last_active":"2024-06-29T15:57:33.354722+0000","last_peered":"2024-06-29T15:57:33.354722+0000","last_clean":"2024-06-29T15:57:33.354722+0000","last_became_active":"2024-06-29T15:57:22.292485+0000","last_became_peered":"2024-06-29T15:57:22.292485+0000","last_unstale":"2024-06-29T15:57:33.354722+0000","last_undegraded":"2024-06-29T15:57:33.354722+0000","last_fullsized":"2024-06-29T15:57:33.354722+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:19.369559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370805+0000","last_change":"2024-06-29T15:57:33.369465+0000","last_active":"2024-06-29T15:57:33.370805+0000","last_peered":"2024-06-29T15:57:33.370805+0000","last_clean":"2024-06-29T15:57:33.370805+0000","last_became_active":"2024-06-29T15:57:33.369174+0000","last_became_peered":"2024-06-29T15:57:33.369174+0000","last_unstale":"2024-06-29T15:57:33.370805+0000","last_undegraded":"2024-06-29T15:57:33.370805+0000","last_fullsized":"2024-06-29T15:57:33.370805+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:02.836842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351888+0000","last_change":"2024-06-29T15:57:33.351888+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351888+0000","last_undegraded":"2024-06-29T15:57:33.351888+0000","last_fullsized":"2024-06-29T15:57:33.351888+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354196+0000","last_change":"2024-06-29T15:57:21.274359+0000","last_active":"2024-06-29T15:57:33.354196+0000","last_peered":"2024-06-29T15:57:33.354196+0000","last_clean":"2024-06-29T15:57:33.354196+0000","last_became_active":"2024-06-29T15:57:21.274036+0000","last_became_peered":"2024-06-29T15:57:21.274036+0000","last_unstale":"2024-06-29T15:57:33.354196+0000","last_undegraded":"2024-06-29T15:57:33.354196+0000","last_fullsized":"2024-06-29T15:57:33.354196+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:02.747066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358900+0000","last_change":"2024-06-29T15:57:22.295101+0000","last_active":"2024-06-29T15:57:33.358900+0000","last_peered":"2024-06-29T15:57:33.358900+0000","last_clean":"2024-06-29T15:57:33.358900+0000","last_became_active":"2024-06-29T15:57:22.294805+0000","last_became_peered":"2024-06-29T15:57:22.294805+0000","last_unstale":"2024-06-29T15:57:33.358900+0000","last_undegraded":"2024-06-29T15:57:33.358900+0000","last_fullsized":"2024-06-29T15:57:33.358900+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.369924+0000","last_change":"2024-06-29T15:57:33.369487+0000","last_active":"2024-06-29T15:57:33.369924+0000","last_peered":"2024-06-29T15:57:33.369924+0000","last_clean":"2024-06-29T15:57:33.369924+0000","last_became_active":"2024-06-29T15:57:33.369138+0000","last_became_peered":"2024-06-29T15:57:33.369138+0000","last_unstale":"2024-06-29T15:57:33.369924+0000","last_undegraded":"2024-06-29T15:57:33.369924+0000","last_fullsized":"2024-06-29T15:57:33.369924+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:06.322225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354530+0000","last_change":"2024-06-29T15:57:33.354530+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354530+0000","last_undegraded":"2024-06-29T15:57:33.354530+0000","last_fullsized":"2024-06-29T15:57:33.354530+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356787+0000","last_change":"2024-06-29T15:57:21.274551+0000","last_active":"2024-06-29T15:57:33.356787+0000","last_peered":"2024-06-29T15:57:33.356787+0000","last_clean":"2024-06-29T15:57:33.356787+0000","last_became_active":"2024-06-29T15:57:21.274029+0000","last_became_peered":"2024-06-29T15:57:21.274029+0000","last_unstale":"2024-06-29T15:57:33.356787+0000","last_undegraded":"2024-06-29T15:57:33.356787+0000","last_fullsized":"2024-06-29T15:57:33.356787+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364941+0000","last_change":"2024-06-29T15:57:22.291143+0000","last_active":"2024-06-29T15:57:33.364941+0000","last_peered":"2024-06-29T15:57:33.364941+0000","last_clean":"2024-06-29T15:57:33.364941+0000","last_became_active":"2024-06-29T15:57:22.290849+0000","last_became_peered":"2024-06-29T15:57:22.290849+0000","last_unstale":"2024-06-29T15:57:33.364941+0000","last_undegraded":"2024-06-29T15:57:33.364941+0000","last_fullsized":"2024-06-29T15:57:33.364941+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367095+0000","last_change":"2024-06-29T15:57:33.366161+0000","last_active":"2024-06-29T15:57:33.367095+0000","last_peered":"2024-06-29T15:57:33.367095+0000","last_clean":"2024-06-29T15:57:33.367095+0000","last_became_active":"2024-06-29T15:57:33.365820+0000","last_became_peered":"2024-06-29T15:57:33.365820+0000","last_unstale":"2024-06-29T15:57:33.367095+0000","last_undegraded":"2024-06-29T15:57:33.367095+0000","last_fullsized":"2024-06-29T15:57:33.367095+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:05.543032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358295+0000","last_change":"2024-06-29T15:57:33.358295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358295+0000","last_undegraded":"2024-06-29T15:57:33.358295+0000","last_fullsized":"2024-06-29T15:57:33.358295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361368+0000","last_change":"2024-06-29T15:57:21.281405+0000","last_active":"2024-06-29T15:57:33.361368+0000","last_peered":"2024-06-29T15:57:33.361368+0000","last_clean":"2024-06-29T15:57:33.361368+0000","last_became_active":"2024-06-29T15:57:21.281080+0000","last_became_peered":"2024-06-29T15:57:21.281080+0000","last_unstale":"2024-06-29T15:57:33.361368+0000","last_undegraded":"2024-06-29T15:57:33.361368+0000","last_fullsized":"2024-06-29T15:57:33.361368+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:51.116695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354452+0000","last_change":"2024-06-29T15:57:22.304057+0000","last_active":"2024-06-29T15:57:33.354452+0000","last_peered":"2024-06-29T15:57:33.354452+0000","last_clean":"2024-06-29T15:57:33.354452+0000","last_became_active":"2024-06-29T15:57:22.303762+0000","last_became_peered":"2024-06-29T15:57:22.303762+0000","last_unstale":"2024-06-29T15:57:33.354452+0000","last_undegraded":"2024-06-29T15:57:33.354452+0000","last_fullsized":"2024-06-29T15:57:33.354452+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:40.125779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370833+0000","last_change":"2024-06-29T15:57:33.369260+0000","last_active":"2024-06-29T15:57:33.370833+0000","last_peered":"2024-06-29T15:57:33.370833+0000","last_clean":"2024-06-29T15:57:33.370833+0000","last_became_active":"2024-06-29T15:57:33.368945+0000","last_became_peered":"2024-06-29T15:57:33.368945+0000","last_unstale":"2024-06-29T15:57:33.370833+0000","last_undegraded":"2024-06-29T15:57:33.370833+0000","last_fullsized":"2024-06-29T15:57:33.370833+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:11.821898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360097+0000","last_change":"2024-06-29T15:57:33.360097+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360097+0000","last_undegraded":"2024-06-29T15:57:33.360097+0000","last_fullsized":"2024-06-29T15:57:33.360097+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357223+0000","last_change":"2024-06-29T15:57:21.277560+0000","last_active":"2024-06-29T15:57:33.357223+0000","last_peered":"2024-06-29T15:57:33.357223+0000","last_clean":"2024-06-29T15:57:33.357223+0000","last_became_active":"2024-06-29T15:57:21.277255+0000","last_became_peered":"2024-06-29T15:57:21.277255+0000","last_unstale":"2024-06-29T15:57:33.357223+0000","last_undegraded":"2024-06-29T15:57:33.357223+0000","last_fullsized":"2024-06-29T15:57:33.357223+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354119+0000","last_change":"2024-06-29T15:57:22.301166+0000","last_active":"2024-06-29T15:57:33.354119+0000","last_peered":"2024-06-29T15:57:33.354119+0000","last_clean":"2024-06-29T15:57:33.354119+0000","last_became_active":"2024-06-29T15:57:22.300676+0000","last_became_peered":"2024-06-29T15:57:22.300676+0000","last_unstale":"2024-06-29T15:57:33.354119+0000","last_undegraded":"2024-06-29T15:57:33.354119+0000","last_fullsized":"2024-06-29T15:57:33.354119+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363490+0000","last_change":"2024-06-29T15:57:33.362184+0000","last_active":"2024-06-29T15:57:33.363490+0000","last_peered":"2024-06-29T15:57:33.363490+0000","last_clean":"2024-06-29T15:57:33.363490+0000","last_became_active":"2024-06-29T15:57:33.361905+0000","last_became_peered":"2024-06-29T15:57:33.361905+0000","last_unstale":"2024-06-29T15:57:33.363490+0000","last_undegraded":"2024-06-29T15:57:33.363490+0000","last_fullsized":"2024-06-29T15:57:33.363490+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:04.833173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349702+0000","last_change":"2024-06-29T15:57:33.349702+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349702+0000","last_undegraded":"2024-06-29T15:57:33.349702+0000","last_fullsized":"2024-06-29T15:57:33.349702+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357105+0000","last_change":"2024-06-29T15:57:21.280224+0000","last_active":"2024-06-29T15:57:33.357105+0000","last_peered":"2024-06-29T15:57:33.357105+0000","last_clean":"2024-06-29T15:57:33.357105+0000","last_became_active":"2024-06-29T15:57:21.279899+0000","last_became_peered":"2024-06-29T15:57:21.279899+0000","last_unstale":"2024-06-29T15:57:33.357105+0000","last_undegraded":"2024-06-29T15:57:33.357105+0000","last_fullsized":"2024-06-29T15:57:33.357105+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355301+0000","last_change":"2024-06-29T15:57:22.296472+0000","last_active":"2024-06-29T15:57:33.355301+0000","last_peered":"2024-06-29T15:57:33.355301+0000","last_clean":"2024-06-29T15:57:33.355301+0000","last_became_active":"2024-06-29T15:57:22.296130+0000","last_became_peered":"2024-06-29T15:57:22.296130+0000","last_unstale":"2024-06-29T15:57:33.355301+0000","last_undegraded":"2024-06-29T15:57:33.355301+0000","last_fullsized":"2024-06-29T15:57:33.355301+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368843+0000","last_change":"2024-06-29T15:57:33.367380+0000","last_active":"2024-06-29T15:57:33.368843+0000","last_peered":"2024-06-29T15:57:33.368843+0000","last_clean":"2024-06-29T15:57:33.368843+0000","last_became_active":"2024-06-29T15:57:33.367054+0000","last_became_peered":"2024-06-29T15:57:33.367054+0000","last_unstale":"2024-06-29T15:57:33.368843+0000","last_undegraded":"2024-06-29T15:57:33.368843+0000","last_fullsized":"2024-06-29T15:57:33.368843+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:29.345255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357030+0000","last_change":"2024-06-29T15:57:33.357030+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357030+0000","last_undegraded":"2024-06-29T15:57:33.357030+0000","last_fullsized":"2024-06-29T15:57:33.357030+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362303+0000","last_change":"2024-06-29T15:57:21.276972+0000","last_active":"2024-06-29T15:57:33.362303+0000","last_peered":"2024-06-29T15:57:33.362303+0000","last_clean":"2024-06-29T15:57:33.362303+0000","last_became_active":"2024-06-29T15:57:21.276647+0000","last_became_peered":"2024-06-29T15:57:21.276647+0000","last_unstale":"2024-06-29T15:57:33.362303+0000","last_undegraded":"2024-06-29T15:57:33.362303+0000","last_fullsized":"2024-06-29T15:57:33.362303+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:24.871281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355716+0000","last_change":"2024-06-29T15:57:22.303802+0000","last_active":"2024-06-29T15:57:33.355716+0000","last_peered":"2024-06-29T15:57:33.355716+0000","last_clean":"2024-06-29T15:57:33.355716+0000","last_became_active":"2024-06-29T15:57:22.303511+0000","last_became_peered":"2024-06-29T15:57:22.303511+0000","last_unstale":"2024-06-29T15:57:33.355716+0000","last_undegraded":"2024-06-29T15:57:33.355716+0000","last_fullsized":"2024-06-29T15:57:33.355716+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:10.536799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371208+0000","last_change":"2024-06-29T15:57:33.367162+0000","last_active":"2024-06-29T15:57:33.371208+0000","last_peered":"2024-06-29T15:57:33.371208+0000","last_clean":"2024-06-29T15:57:33.371208+0000","last_became_active":"2024-06-29T15:57:33.366816+0000","last_became_peered":"2024-06-29T15:57:33.366816+0000","last_unstale":"2024-06-29T15:57:33.371208+0000","last_undegraded":"2024-06-29T15:57:33.371208+0000","last_fullsized":"2024-06-29T15:57:33.371208+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:41.683812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.363662+0000","last_change":"2024-06-29T15:57:33.363662+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.363662+0000","last_undegraded":"2024-06-29T15:57:33.363662+0000","last_fullsized":"2024-06-29T15:57:33.363662+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358022+0000","last_change":"2024-06-29T15:57:21.277662+0000","last_active":"2024-06-29T15:57:33.358022+0000","last_peered":"2024-06-29T15:57:33.358022+0000","last_clean":"2024-06-29T15:57:33.358022+0000","last_became_active":"2024-06-29T15:57:21.277335+0000","last_became_peered":"2024-06-29T15:57:21.277335+0000","last_unstale":"2024-06-29T15:57:33.358022+0000","last_undegraded":"2024-06-29T15:57:33.358022+0000","last_fullsized":"2024-06-29T15:57:33.358022+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364773+0000","last_change":"2024-06-29T15:57:22.298914+0000","last_active":"2024-06-29T15:57:33.364773+0000","last_peered":"2024-06-29T15:57:33.364773+0000","last_clean":"2024-06-29T15:57:33.364773+0000","last_became_active":"2024-06-29T15:57:22.298614+0000","last_became_peered":"2024-06-29T15:57:22.298614+0000","last_unstale":"2024-06-29T15:57:33.364773+0000","last_undegraded":"2024-06-29T15:57:33.364773+0000","last_fullsized":"2024-06-29T15:57:33.364773+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367924+0000","last_change":"2024-06-29T15:57:33.367133+0000","last_active":"2024-06-29T15:57:33.367924+0000","last_peered":"2024-06-29T15:57:33.367924+0000","last_clean":"2024-06-29T15:57:33.367924+0000","last_became_active":"2024-06-29T15:57:33.366782+0000","last_became_peered":"2024-06-29T15:57:33.366782+0000","last_unstale":"2024-06-29T15:57:33.367924+0000","last_undegraded":"2024-06-29T15:57:33.367924+0000","last_fullsized":"2024-06-29T15:57:33.367924+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351295+0000","last_change":"2024-06-29T15:57:33.351295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351295+0000","last_undegraded":"2024-06-29T15:57:33.351295+0000","last_fullsized":"2024-06-29T15:57:33.351295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355165+0000","last_change":"2024-06-29T15:57:21.278011+0000","last_active":"2024-06-29T15:57:33.355165+0000","last_peered":"2024-06-29T15:57:33.355165+0000","last_clean":"2024-06-29T15:57:33.355165+0000","last_became_active":"2024-06-29T15:57:21.276648+0000","last_became_peered":"2024-06-29T15:57:21.276648+0000","last_unstale":"2024-06-29T15:57:33.355165+0000","last_undegraded":"2024-06-29T15:57:33.355165+0000","last_fullsized":"2024-06-29T15:57:33.355165+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354181+0000","last_change":"2024-06-29T15:57:22.300220+0000","last_active":"2024-06-29T15:57:33.354181+0000","last_peered":"2024-06-29T15:57:33.354181+0000","last_clean":"2024-06-29T15:57:33.354181+0000","last_became_active":"2024-06-29T15:57:22.299880+0000","last_became_peered":"2024-06-29T15:57:22.299880+0000","last_unstale":"2024-06-29T15:57:33.354181+0000","last_undegraded":"2024-06-29T15:57:33.354181+0000","last_fullsized":"2024-06-29T15:57:33.354181+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368552+0000","last_change":"2024-06-29T15:57:33.367636+0000","last_active":"2024-06-29T15:57:33.368552+0000","last_peered":"2024-06-29T15:57:33.368552+0000","last_clean":"2024-06-29T15:57:33.368552+0000","last_became_active":"2024-06-29T15:57:33.367321+0000","last_became_peered":"2024-06-29T15:57:33.367321+0000","last_unstale":"2024-06-29T15:57:33.368552+0000","last_undegraded":"2024-06-29T15:57:33.368552+0000","last_fullsized":"2024-06-29T15:57:33.368552+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:28.846172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353695+0000","last_change":"2024-06-29T15:57:33.353695+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353695+0000","last_undegraded":"2024-06-29T15:57:33.353695+0000","last_fullsized":"2024-06-29T15:57:33.353695+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357156+0000","last_change":"2024-06-29T15:57:21.284982+0000","last_active":"2024-06-29T15:57:33.357156+0000","last_peered":"2024-06-29T15:57:33.357156+0000","last_clean":"2024-06-29T15:57:33.357156+0000","last_became_active":"2024-06-29T15:57:21.284684+0000","last_became_peered":"2024-06-29T15:57:21.284684+0000","last_unstale":"2024-06-29T15:57:33.357156+0000","last_undegraded":"2024-06-29T15:57:33.357156+0000","last_fullsized":"2024-06-29T15:57:33.357156+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354906+0000","last_change":"2024-06-29T15:57:22.298744+0000","last_active":"2024-06-29T15:57:33.354906+0000","last_peered":"2024-06-29T15:57:33.354906+0000","last_clean":"2024-06-29T15:57:33.354906+0000","last_became_active":"2024-06-29T15:57:22.296925+0000","last_became_peered":"2024-06-29T15:57:22.296925+0000","last_unstale":"2024-06-29T15:57:33.354906+0000","last_undegraded":"2024-06-29T15:57:33.354906+0000","last_fullsized":"2024-06-29T15:57:33.354906+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:40.578182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371721+0000","last_change":"2024-06-29T15:57:33.368936+0000","last_active":"2024-06-29T15:57:33.371721+0000","last_peered":"2024-06-29T15:57:33.371721+0000","last_clean":"2024-06-29T15:57:33.371721+0000","last_became_active":"2024-06-29T15:57:33.368600+0000","last_became_peered":"2024-06-29T15:57:33.368600+0000","last_unstale":"2024-06-29T15:57:33.371721+0000","last_undegraded":"2024-06-29T15:57:33.371721+0000","last_fullsized":"2024-06-29T15:57:33.371721+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:22.830532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.361651+0000","last_change":"2024-06-29T15:57:33.361651+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.361651+0000","last_undegraded":"2024-06-29T15:57:33.361651+0000","last_fullsized":"2024-06-29T15:57:33.361651+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355165+0000","last_change":"2024-06-29T15:57:21.284780+0000","last_active":"2024-06-29T15:57:33.355165+0000","last_peered":"2024-06-29T15:57:33.355165+0000","last_clean":"2024-06-29T15:57:33.355165+0000","last_became_active":"2024-06-29T15:57:21.284273+0000","last_became_peered":"2024-06-29T15:57:21.284273+0000","last_unstale":"2024-06-29T15:57:33.355165+0000","last_undegraded":"2024-06-29T15:57:33.355165+0000","last_fullsized":"2024-06-29T15:57:33.355165+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:16.085990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356696+0000","last_change":"2024-06-29T15:57:22.291304+0000","last_active":"2024-06-29T15:57:33.356696+0000","last_peered":"2024-06-29T15:57:33.356696+0000","last_clean":"2024-06-29T15:57:33.356696+0000","last_became_active":"2024-06-29T15:57:22.290983+0000","last_became_peered":"2024-06-29T15:57:22.290983+0000","last_unstale":"2024-06-29T15:57:33.356696+0000","last_undegraded":"2024-06-29T15:57:33.356696+0000","last_fullsized":"2024-06-29T15:57:33.356696+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368254+0000","last_change":"2024-06-29T15:57:33.366553+0000","last_active":"2024-06-29T15:57:33.368254+0000","last_peered":"2024-06-29T15:57:33.368254+0000","last_clean":"2024-06-29T15:57:33.368254+0000","last_became_active":"2024-06-29T15:57:33.366230+0000","last_became_peered":"2024-06-29T15:57:33.366230+0000","last_unstale":"2024-06-29T15:57:33.368254+0000","last_undegraded":"2024-06-29T15:57:33.368254+0000","last_fullsized":"2024-06-29T15:57:33.368254+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:17:11.929406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358755+0000","last_change":"2024-06-29T15:57:33.358755+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358755+0000","last_undegraded":"2024-06-29T15:57:33.358755+0000","last_fullsized":"2024-06-29T15:57:33.358755+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348756+0000","last_change":"2024-06-29T15:57:21.284009+0000","last_active":"2024-06-29T15:57:33.348756+0000","last_peered":"2024-06-29T15:57:33.348756+0000","last_clean":"2024-06-29T15:57:33.348756+0000","last_became_active":"2024-06-29T15:57:21.283679+0000","last_became_peered":"2024-06-29T15:57:21.283679+0000","last_unstale":"2024-06-29T15:57:33.348756+0000","last_undegraded":"2024-06-29T15:57:33.348756+0000","last_fullsized":"2024-06-29T15:57:33.348756+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:20.284468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348277+0000","last_change":"2024-06-29T15:57:22.288958+0000","last_active":"2024-06-29T15:57:33.348277+0000","last_peered":"2024-06-29T15:57:33.348277+0000","last_clean":"2024-06-29T15:57:33.348277+0000","last_became_active":"2024-06-29T15:57:22.288621+0000","last_became_peered":"2024-06-29T15:57:22.288621+0000","last_unstale":"2024-06-29T15:57:33.348277+0000","last_undegraded":"2024-06-29T15:57:33.348277+0000","last_fullsized":"2024-06-29T15:57:33.348277+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:31.720751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371349+0000","last_change":"2024-06-29T15:57:33.370560+0000","last_active":"2024-06-29T15:57:33.371349+0000","last_peered":"2024-06-29T15:57:33.371349+0000","last_clean":"2024-06-29T15:57:33.371349+0000","last_became_active":"2024-06-29T15:57:33.370206+0000","last_became_peered":"2024-06-29T15:57:33.370206+0000","last_unstale":"2024-06-29T15:57:33.371349+0000","last_undegraded":"2024-06-29T15:57:33.371349+0000","last_fullsized":"2024-06-29T15:57:33.371349+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:12:47.692442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.361054+0000","last_change":"2024-06-29T15:57:33.361054+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.361054+0000","last_undegraded":"2024-06-29T15:57:33.361054+0000","last_fullsized":"2024-06-29T15:57:33.361054+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357091+0000","last_change":"2024-06-29T15:57:21.282111+0000","last_active":"2024-06-29T15:57:33.357091+0000","last_peered":"2024-06-29T15:57:33.357091+0000","last_clean":"2024-06-29T15:57:33.357091+0000","last_became_active":"2024-06-29T15:57:21.281761+0000","last_became_peered":"2024-06-29T15:57:21.281761+0000","last_unstale":"2024-06-29T15:57:33.357091+0000","last_undegraded":"2024-06-29T15:57:33.357091+0000","last_fullsized":"2024-06-29T15:57:33.357091+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348792+0000","last_change":"2024-06-29T15:57:22.305806+0000","last_active":"2024-06-29T15:57:33.348792+0000","last_peered":"2024-06-29T15:57:33.348792+0000","last_clean":"2024-06-29T15:57:33.348792+0000","last_became_active":"2024-06-29T15:57:22.296276+0000","last_became_peered":"2024-06-29T15:57:22.296276+0000","last_unstale":"2024-06-29T15:57:33.348792+0000","last_undegraded":"2024-06-29T15:57:33.348792+0000","last_fullsized":"2024-06-29T15:57:33.348792+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:43:51.754223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.369952+0000","last_change":"2024-06-29T15:57:33.367821+0000","last_active":"2024-06-29T15:57:33.369952+0000","last_peered":"2024-06-29T15:57:33.369952+0000","last_clean":"2024-06-29T15:57:33.369952+0000","last_became_active":"2024-06-29T15:57:33.367494+0000","last_became_peered":"2024-06-29T15:57:33.367494+0000","last_unstale":"2024-06-29T15:57:33.369952+0000","last_undegraded":"2024-06-29T15:57:33.369952+0000","last_fullsized":"2024-06-29T15:57:33.369952+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:46.165772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352248+0000","last_change":"2024-06-29T15:57:33.352248+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352248+0000","last_undegraded":"2024-06-29T15:57:33.352248+0000","last_fullsized":"2024-06-29T15:57:33.352248+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.349023+0000","last_change":"2024-06-29T15:57:21.279139+0000","last_active":"2024-06-29T15:57:33.349023+0000","last_peered":"2024-06-29T15:57:33.349023+0000","last_clean":"2024-06-29T15:57:33.349023+0000","last_became_active":"2024-06-29T15:57:21.278488+0000","last_became_peered":"2024-06-29T15:57:21.278488+0000","last_unstale":"2024-06-29T15:57:33.349023+0000","last_undegraded":"2024-06-29T15:57:33.349023+0000","last_fullsized":"2024-06-29T15:57:33.349023+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:54.126338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356194+0000","last_change":"2024-06-29T15:57:22.297275+0000","last_active":"2024-06-29T15:57:33.356194+0000","last_peered":"2024-06-29T15:57:33.356194+0000","last_clean":"2024-06-29T15:57:33.356194+0000","last_became_active":"2024-06-29T15:57:22.296984+0000","last_became_peered":"2024-06-29T15:57:22.296984+0000","last_unstale":"2024-06-29T15:57:33.356194+0000","last_undegraded":"2024-06-29T15:57:33.356194+0000","last_fullsized":"2024-06-29T15:57:33.356194+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371810+0000","last_change":"2024-06-29T15:57:33.369358+0000","last_active":"2024-06-29T15:57:33.371810+0000","last_peered":"2024-06-29T15:57:33.371810+0000","last_clean":"2024-06-29T15:57:33.371810+0000","last_became_active":"2024-06-29T15:57:33.369052+0000","last_became_peered":"2024-06-29T15:57:33.369052+0000","last_unstale":"2024-06-29T15:57:33.371810+0000","last_undegraded":"2024-06-29T15:57:33.371810+0000","last_fullsized":"2024-06-29T15:57:33.371810+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:29.345255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352493+0000","last_change":"2024-06-29T15:57:33.352493+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352493+0000","last_undegraded":"2024-06-29T15:57:33.352493+0000","last_fullsized":"2024-06-29T15:57:33.352493+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356473+0000","last_change":"2024-06-29T15:57:21.278476+0000","last_active":"2024-06-29T15:57:33.356473+0000","last_peered":"2024-06-29T15:57:33.356473+0000","last_clean":"2024-06-29T15:57:33.356473+0000","last_became_active":"2024-06-29T15:57:21.278150+0000","last_became_peered":"2024-06-29T15:57:21.278150+0000","last_unstale":"2024-06-29T15:57:33.356473+0000","last_undegraded":"2024-06-29T15:57:33.356473+0000","last_fullsized":"2024-06-29T15:57:33.356473+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:33.991332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355923+0000","last_change":"2024-06-29T15:57:22.291360+0000","last_active":"2024-06-29T15:57:33.355923+0000","last_peered":"2024-06-29T15:57:33.355923+0000","last_clean":"2024-06-29T15:57:33.355923+0000","last_became_active":"2024-06-29T15:57:22.290801+0000","last_became_peered":"2024-06-29T15:57:22.290801+0000","last_unstale":"2024-06-29T15:57:33.355923+0000","last_undegraded":"2024-06-29T15:57:33.355923+0000","last_fullsized":"2024-06-29T15:57:33.355923+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371558+0000","last_change":"2024-06-29T15:57:33.367933+0000","last_active":"2024-06-29T15:57:33.371558+0000","last_peered":"2024-06-29T15:57:33.371558+0000","last_clean":"2024-06-29T15:57:33.371558+0000","last_became_active":"2024-06-29T15:57:33.367630+0000","last_became_peered":"2024-06-29T15:57:33.367630+0000","last_unstale":"2024-06-29T15:57:33.371558+0000","last_undegraded":"2024-06-29T15:57:33.371558+0000","last_fullsized":"2024-06-29T15:57:33.371558+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:01.393531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358484+0000","last_change":"2024-06-29T15:57:33.358484+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358484+0000","last_undegraded":"2024-06-29T15:57:33.358484+0000","last_fullsized":"2024-06-29T15:57:33.358484+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358778+0000","last_change":"2024-06-29T15:57:21.273304+0000","last_active":"2024-06-29T15:57:33.358778+0000","last_peered":"2024-06-29T15:57:33.358778+0000","last_clean":"2024-06-29T15:57:33.358778+0000","last_became_active":"2024-06-29T15:57:21.272972+0000","last_became_peered":"2024-06-29T15:57:21.272972+0000","last_unstale":"2024-06-29T15:57:33.358778+0000","last_undegraded":"2024-06-29T15:57:33.358778+0000","last_fullsized":"2024-06-29T15:57:33.358778+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356200+0000","last_change":"2024-06-29T15:57:22.300916+0000","last_active":"2024-06-29T15:57:33.356200+0000","last_peered":"2024-06-29T15:57:33.356200+0000","last_clean":"2024-06-29T15:57:33.356200+0000","last_became_active":"2024-06-29T15:57:22.300620+0000","last_became_peered":"2024-06-29T15:57:22.300620+0000","last_unstale":"2024-06-29T15:57:33.356200+0000","last_undegraded":"2024-06-29T15:57:33.356200+0000","last_fullsized":"2024-06-29T15:57:33.356200+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:41.172455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368260+0000","last_change":"2024-06-29T15:57:33.367013+0000","last_active":"2024-06-29T15:57:33.368260+0000","last_peered":"2024-06-29T15:57:33.368260+0000","last_clean":"2024-06-29T15:57:33.368260+0000","last_became_active":"2024-06-29T15:57:33.366664+0000","last_became_peered":"2024-06-29T15:57:33.366664+0000","last_unstale":"2024-06-29T15:57:33.368260+0000","last_undegraded":"2024-06-29T15:57:33.368260+0000","last_fullsized":"2024-06-29T15:57:33.368260+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:22.830532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357163+0000","last_change":"2024-06-29T15:57:33.357163+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357163+0000","last_undegraded":"2024-06-29T15:57:33.357163+0000","last_fullsized":"2024-06-29T15:57:33.357163+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"212'8","reported_seq":26,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359054+0000","last_change":"2024-06-29T15:57:21.283745+0000","last_active":"2024-06-29T15:57:33.359054+0000","last_peered":"2024-06-29T15:57:33.359054+0000","last_clean":"2024-06-29T15:57:33.359054+0000","last_became_active":"2024-06-29T15:57:21.283403+0000","last_became_peered":"2024-06-29T15:57:21.283403+0000","last_unstale":"2024-06-29T15:57:33.359054+0000","last_undegraded":"2024-06-29T15:57:33.359054+0000","last_fullsized":"2024-06-29T15:57:33.359054+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357048+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:33.357048+0000","last_peered":"2024-06-29T15:57:33.357048+0000","last_clean":"2024-06-29T15:57:33.357048+0000","last_became_active":"2024-06-29T15:57:22.301832+0000","last_became_peered":"2024-06-29T15:57:22.301832+0000","last_unstale":"2024-06-29T15:57:33.357048+0000","last_undegraded":"2024-06-29T15:57:33.357048+0000","last_fullsized":"2024-06-29T15:57:33.357048+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367778+0000","last_change":"2024-06-29T15:57:33.366032+0000","last_active":"2024-06-29T15:57:33.367778+0000","last_peered":"2024-06-29T15:57:33.367778+0000","last_clean":"2024-06-29T15:57:33.367778+0000","last_became_active":"2024-06-29T15:57:33.365731+0000","last_became_peered":"2024-06-29T15:57:33.365731+0000","last_unstale":"2024-06-29T15:57:33.367778+0000","last_undegraded":"2024-06-29T15:57:33.367778+0000","last_fullsized":"2024-06-29T15:57:33.367778+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:58.462755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349766+0000","last_change":"2024-06-29T15:57:33.349766+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349766+0000","last_undegraded":"2024-06-29T15:57:33.349766+0000","last_fullsized":"2024-06-29T15:57:33.349766+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348560+0000","last_change":"2024-06-29T15:57:21.276731+0000","last_active":"2024-06-29T15:57:33.348560+0000","last_peered":"2024-06-29T15:57:33.348560+0000","last_clean":"2024-06-29T15:57:33.348560+0000","last_became_active":"2024-06-29T15:57:21.276408+0000","last_became_peered":"2024-06-29T15:57:21.276408+0000","last_unstale":"2024-06-29T15:57:33.348560+0000","last_undegraded":"2024-06-29T15:57:33.348560+0000","last_fullsized":"2024-06-29T15:57:33.348560+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:48.911239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358795+0000","last_change":"2024-06-29T15:57:22.299631+0000","last_active":"2024-06-29T15:57:33.358795+0000","last_peered":"2024-06-29T15:57:33.358795+0000","last_clean":"2024-06-29T15:57:33.358795+0000","last_became_active":"2024-06-29T15:57:22.299285+0000","last_became_peered":"2024-06-29T15:57:22.299285+0000","last_unstale":"2024-06-29T15:57:33.358795+0000","last_undegraded":"2024-06-29T15:57:33.358795+0000","last_fullsized":"2024-06-29T15:57:33.358795+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370910+0000","last_change":"2024-06-29T15:57:33.369952+0000","last_active":"2024-06-29T15:57:33.370910+0000","last_peered":"2024-06-29T15:57:33.370910+0000","last_clean":"2024-06-29T15:57:33.370910+0000","last_became_active":"2024-06-29T15:57:33.369598+0000","last_became_peered":"2024-06-29T15:57:33.369598+0000","last_unstale":"2024-06-29T15:57:33.370910+0000","last_undegraded":"2024-06-29T15:57:33.370910+0000","last_fullsized":"2024-06-29T15:57:33.370910+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:24.974909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358887+0000","last_change":"2024-06-29T15:57:33.358887+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358887+0000","last_undegraded":"2024-06-29T15:57:33.358887+0000","last_fullsized":"2024-06-29T15:57:33.358887+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354289+0000","last_change":"2024-06-29T15:57:21.282872+0000","last_active":"2024-06-29T15:57:33.354289+0000","last_peered":"2024-06-29T15:57:33.354289+0000","last_clean":"2024-06-29T15:57:33.354289+0000","last_became_active":"2024-06-29T15:57:21.282549+0000","last_became_peered":"2024-06-29T15:57:21.282549+0000","last_unstale":"2024-06-29T15:57:33.354289+0000","last_undegraded":"2024-06-29T15:57:33.354289+0000","last_fullsized":"2024-06-29T15:57:33.354289+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348643+0000","last_change":"2024-06-29T15:57:22.294011+0000","last_active":"2024-06-29T15:57:33.348643+0000","last_peered":"2024-06-29T15:57:33.348643+0000","last_clean":"2024-06-29T15:57:33.348643+0000","last_became_active":"2024-06-29T15:57:22.293721+0000","last_became_peered":"2024-06-29T15:57:22.293721+0000","last_unstale":"2024-06-29T15:57:33.348643+0000","last_undegraded":"2024-06-29T15:57:33.348643+0000","last_fullsized":"2024-06-29T15:57:33.348643+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:57.337708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362703+0000","last_change":"2024-06-29T15:57:33.361883+0000","last_active":"2024-06-29T15:57:33.362703+0000","last_peered":"2024-06-29T15:57:33.362703+0000","last_clean":"2024-06-29T15:57:33.362703+0000","last_became_active":"2024-06-29T15:57:33.361569+0000","last_became_peered":"2024-06-29T15:57:33.361569+0000","last_unstale":"2024-06-29T15:57:33.362703+0000","last_undegraded":"2024-06-29T15:57:33.362703+0000","last_fullsized":"2024-06-29T15:57:33.362703+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:08.554869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351919+0000","last_change":"2024-06-29T15:57:33.351919+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351919+0000","last_undegraded":"2024-06-29T15:57:33.351919+0000","last_fullsized":"2024-06-29T15:57:33.351919+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353877+0000","last_change":"2024-06-29T15:57:21.281309+0000","last_active":"2024-06-29T15:57:33.353877+0000","last_peered":"2024-06-29T15:57:33.353877+0000","last_clean":"2024-06-29T15:57:33.353877+0000","last_became_active":"2024-06-29T15:57:21.280944+0000","last_became_peered":"2024-06-29T15:57:21.280944+0000","last_unstale":"2024-06-29T15:57:33.353877+0000","last_undegraded":"2024-06-29T15:57:33.353877+0000","last_fullsized":"2024-06-29T15:57:33.353877+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356544+0000","last_change":"2024-06-29T15:57:22.291164+0000","last_active":"2024-06-29T15:57:33.356544+0000","last_peered":"2024-06-29T15:57:33.356544+0000","last_clean":"2024-06-29T15:57:33.356544+0000","last_became_active":"2024-06-29T15:57:22.290897+0000","last_became_peered":"2024-06-29T15:57:22.290897+0000","last_unstale":"2024-06-29T15:57:33.356544+0000","last_undegraded":"2024-06-29T15:57:33.356544+0000","last_fullsized":"2024-06-29T15:57:33.356544+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364704+0000","last_change":"2024-06-29T15:57:33.364026+0000","last_active":"2024-06-29T15:57:33.364704+0000","last_peered":"2024-06-29T15:57:33.364704+0000","last_clean":"2024-06-29T15:57:33.364704+0000","last_became_active":"2024-06-29T15:57:33.363658+0000","last_became_peered":"2024-06-29T15:57:33.363658+0000","last_unstale":"2024-06-29T15:57:33.364704+0000","last_undegraded":"2024-06-29T15:57:33.364704+0000","last_fullsized":"2024-06-29T15:57:33.364704+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:41.322192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359617+0000","last_change":"2024-06-29T15:57:33.359617+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359617+0000","last_undegraded":"2024-06-29T15:57:33.359617+0000","last_fullsized":"2024-06-29T15:57:33.359617+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"214'8","reported_seq":26,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365217+0000","last_change":"2024-06-29T15:57:21.278161+0000","last_active":"2024-06-29T15:57:33.365217+0000","last_peered":"2024-06-29T15:57:33.365217+0000","last_clean":"2024-06-29T15:57:33.365217+0000","last_became_active":"2024-06-29T15:57:21.277794+0000","last_became_peered":"2024-06-29T15:57:21.277794+0000","last_unstale":"2024-06-29T15:57:33.365217+0000","last_undegraded":"2024-06-29T15:57:33.365217+0000","last_fullsized":"2024-06-29T15:57:33.365217+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:56.330168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.352292+0000","last_change":"2024-06-29T15:57:22.302666+0000","last_active":"2024-06-29T15:57:33.352292+0000","last_peered":"2024-06-29T15:57:33.352292+0000","last_clean":"2024-06-29T15:57:33.352292+0000","last_became_active":"2024-06-29T15:57:22.302254+0000","last_became_peered":"2024-06-29T15:57:22.302254+0000","last_unstale":"2024-06-29T15:57:33.352292+0000","last_undegraded":"2024-06-29T15:57:33.352292+0000","last_fullsized":"2024-06-29T15:57:33.352292+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363467+0000","last_change":"2024-06-29T15:57:33.361374+0000","last_active":"2024-06-29T15:57:33.363467+0000","last_peered":"2024-06-29T15:57:33.363467+0000","last_clean":"2024-06-29T15:57:33.363467+0000","last_became_active":"2024-06-29T15:57:33.361002+0000","last_became_peered":"2024-06-29T15:57:33.361002+0000","last_unstale":"2024-06-29T15:57:33.363467+0000","last_undegraded":"2024-06-29T15:57:33.363467+0000","last_fullsized":"2024-06-29T15:57:33.363467+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:47.234190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353505+0000","last_change":"2024-06-29T15:57:33.353505+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353505+0000","last_undegraded":"2024-06-29T15:57:33.353505+0000","last_fullsized":"2024-06-29T15:57:33.353505+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"212'3","reported_seq":21,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348900+0000","last_change":"2024-06-29T15:57:21.283791+0000","last_active":"2024-06-29T15:57:33.348900+0000","last_peered":"2024-06-29T15:57:33.348900+0000","last_clean":"2024-06-29T15:57:33.348900+0000","last_became_active":"2024-06-29T15:57:21.283458+0000","last_became_peered":"2024-06-29T15:57:21.283458+0000","last_unstale":"2024-06-29T15:57:33.348900+0000","last_undegraded":"2024-06-29T15:57:33.348900+0000","last_fullsized":"2024-06-29T15:57:33.348900+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:54.365881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361117+0000","last_change":"2024-06-29T15:57:22.295260+0000","last_active":"2024-06-29T15:57:33.361117+0000","last_peered":"2024-06-29T15:57:33.361117+0000","last_clean":"2024-06-29T15:57:33.361117+0000","last_became_active":"2024-06-29T15:57:22.294952+0000","last_became_peered":"2024-06-29T15:57:22.294952+0000","last_unstale":"2024-06-29T15:57:33.361117+0000","last_undegraded":"2024-06-29T15:57:33.361117+0000","last_fullsized":"2024-06-29T15:57:33.361117+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:56.473182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370910+0000","last_change":"2024-06-29T15:57:33.370460+0000","last_active":"2024-06-29T15:57:33.370910+0000","last_peered":"2024-06-29T15:57:33.370910+0000","last_clean":"2024-06-29T15:57:33.370910+0000","last_became_active":"2024-06-29T15:57:33.370183+0000","last_became_peered":"2024-06-29T15:57:33.370183+0000","last_unstale":"2024-06-29T15:57:33.370910+0000","last_undegraded":"2024-06-29T15:57:33.370910+0000","last_fullsized":"2024-06-29T15:57:33.370910+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:06.322225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352295+0000","last_change":"2024-06-29T15:57:33.352295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352295+0000","last_undegraded":"2024-06-29T15:57:33.352295+0000","last_fullsized":"2024-06-29T15:57:33.352295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354550+0000","last_change":"2024-06-29T15:57:21.275926+0000","last_active":"2024-06-29T15:57:33.354550+0000","last_peered":"2024-06-29T15:57:33.354550+0000","last_clean":"2024-06-29T15:57:33.354550+0000","last_became_active":"2024-06-29T15:57:21.275593+0000","last_became_peered":"2024-06-29T15:57:21.275593+0000","last_unstale":"2024-06-29T15:57:33.354550+0000","last_undegraded":"2024-06-29T15:57:33.354550+0000","last_fullsized":"2024-06-29T15:57:33.354550+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357897+0000","last_change":"2024-06-29T15:57:22.298454+0000","last_active":"2024-06-29T15:57:33.357897+0000","last_peered":"2024-06-29T15:57:33.357897+0000","last_clean":"2024-06-29T15:57:33.357897+0000","last_became_active":"2024-06-29T15:57:22.298162+0000","last_became_peered":"2024-06-29T15:57:22.298162+0000","last_unstale":"2024-06-29T15:57:33.357897+0000","last_undegraded":"2024-06-29T15:57:33.357897+0000","last_fullsized":"2024-06-29T15:57:33.357897+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363344+0000","last_change":"2024-06-29T15:57:33.362039+0000","last_active":"2024-06-29T15:57:33.363344+0000","last_peered":"2024-06-29T15:57:33.363344+0000","last_clean":"2024-06-29T15:57:33.363344+0000","last_became_active":"2024-06-29T15:57:33.361665+0000","last_became_peered":"2024-06-29T15:57:33.361665+0000","last_unstale":"2024-06-29T15:57:33.363344+0000","last_undegraded":"2024-06-29T15:57:33.363344+0000","last_fullsized":"2024-06-29T15:57:33.363344+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:41.322192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358599+0000","last_change":"2024-06-29T15:57:33.358599+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358599+0000","last_undegraded":"2024-06-29T15:57:33.358599+0000","last_fullsized":"2024-06-29T15:57:33.358599+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358654+0000","last_change":"2024-06-29T15:57:21.272019+0000","last_active":"2024-06-29T15:57:33.358654+0000","last_peered":"2024-06-29T15:57:33.358654+0000","last_clean":"2024-06-29T15:57:33.358654+0000","last_became_active":"2024-06-29T15:57:21.271715+0000","last_became_peered":"2024-06-29T15:57:21.271715+0000","last_unstale":"2024-06-29T15:57:33.358654+0000","last_undegraded":"2024-06-29T15:57:33.358654+0000","last_fullsized":"2024-06-29T15:57:33.358654+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362094+0000","last_change":"2024-06-29T15:57:22.303487+0000","last_active":"2024-06-29T15:57:33.362094+0000","last_peered":"2024-06-29T15:57:33.362094+0000","last_clean":"2024-06-29T15:57:33.362094+0000","last_became_active":"2024-06-29T15:57:22.302933+0000","last_became_peered":"2024-06-29T15:57:22.302933+0000","last_unstale":"2024-06-29T15:57:33.362094+0000","last_undegraded":"2024-06-29T15:57:33.362094+0000","last_fullsized":"2024-06-29T15:57:33.362094+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:28.493187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362444+0000","last_change":"2024-06-29T15:57:33.361560+0000","last_active":"2024-06-29T15:57:33.362444+0000","last_peered":"2024-06-29T15:57:33.362444+0000","last_clean":"2024-06-29T15:57:33.362444+0000","last_became_active":"2024-06-29T15:57:33.361270+0000","last_became_peered":"2024-06-29T15:57:33.361270+0000","last_unstale":"2024-06-29T15:57:33.362444+0000","last_undegraded":"2024-06-29T15:57:33.362444+0000","last_fullsized":"2024-06-29T15:57:33.362444+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:51.198488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356450+0000","last_change":"2024-06-29T15:57:33.356450+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356450+0000","last_undegraded":"2024-06-29T15:57:33.356450+0000","last_fullsized":"2024-06-29T15:57:33.356450+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364019+0000","last_change":"2024-06-29T15:57:33.362045+0000","last_active":"2024-06-29T15:57:33.364019+0000","last_peered":"2024-06-29T15:57:33.364019+0000","last_clean":"2024-06-29T15:57:33.364019+0000","last_became_active":"2024-06-29T15:57:33.361647+0000","last_became_peered":"2024-06-29T15:57:33.361647+0000","last_unstale":"2024-06-29T15:57:33.364019+0000","last_undegraded":"2024-06-29T15:57:33.364019+0000","last_fullsized":"2024-06-29T15:57:33.364019+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:51.198488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353878+0000","last_change":"2024-06-29T15:57:21.277245+0000","last_active":"2024-06-29T15:57:33.353878+0000","last_peered":"2024-06-29T15:57:33.353878+0000","last_clean":"2024-06-29T15:57:33.353878+0000","last_became_active":"2024-06-29T15:57:21.276931+0000","last_became_peered":"2024-06-29T15:57:21.276931+0000","last_unstale":"2024-06-29T15:57:33.353878+0000","last_undegraded":"2024-06-29T15:57:33.353878+0000","last_fullsized":"2024-06-29T15:57:33.353878+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361080+0000","last_change":"2024-06-29T15:57:22.299457+0000","last_active":"2024-06-29T15:57:33.361080+0000","last_peered":"2024-06-29T15:57:33.361080+0000","last_clean":"2024-06-29T15:57:33.361080+0000","last_became_active":"2024-06-29T15:57:22.299157+0000","last_became_peered":"2024-06-29T15:57:22.299157+0000","last_unstale":"2024-06-29T15:57:33.361080+0000","last_undegraded":"2024-06-29T15:57:33.361080+0000","last_fullsized":"2024-06-29T15:57:33.361080+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:40.967292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356113+0000","last_change":"2024-06-29T15:57:33.356113+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356113+0000","last_undegraded":"2024-06-29T15:57:33.356113+0000","last_fullsized":"2024-06-29T15:57:33.356113+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363418+0000","last_change":"2024-06-29T15:57:33.360662+0000","last_active":"2024-06-29T15:57:33.363418+0000","last_peered":"2024-06-29T15:57:33.363418+0000","last_clean":"2024-06-29T15:57:33.363418+0000","last_became_active":"2024-06-29T15:57:33.360087+0000","last_became_peered":"2024-06-29T15:57:33.360087+0000","last_unstale":"2024-06-29T15:57:33.363418+0000","last_undegraded":"2024-06-29T15:57:33.363418+0000","last_fullsized":"2024-06-29T15:57:33.363418+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:46:09.697181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355322+0000","last_change":"2024-06-29T15:57:21.277884+0000","last_active":"2024-06-29T15:57:33.355322+0000","last_peered":"2024-06-29T15:57:33.355322+0000","last_clean":"2024-06-29T15:57:33.355322+0000","last_became_active":"2024-06-29T15:57:21.277589+0000","last_became_peered":"2024-06-29T15:57:21.277589+0000","last_unstale":"2024-06-29T15:57:33.355322+0000","last_undegraded":"2024-06-29T15:57:33.355322+0000","last_fullsized":"2024-06-29T15:57:33.355322+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:09.017396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355087+0000","last_change":"2024-06-29T15:57:22.293696+0000","last_active":"2024-06-29T15:57:33.355087+0000","last_peered":"2024-06-29T15:57:33.355087+0000","last_clean":"2024-06-29T15:57:33.355087+0000","last_became_active":"2024-06-29T15:57:22.293263+0000","last_became_peered":"2024-06-29T15:57:22.293263+0000","last_unstale":"2024-06-29T15:57:33.355087+0000","last_undegraded":"2024-06-29T15:57:33.355087+0000","last_fullsized":"2024-06-29T15:57:33.355087+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352369+0000","last_change":"2024-06-29T15:57:33.352369+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352369+0000","last_undegraded":"2024-06-29T15:57:33.352369+0000","last_fullsized":"2024-06-29T15:57:33.352369+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356661+0000","last_change":"2024-06-29T15:57:21.274306+0000","last_active":"2024-06-29T15:57:33.356661+0000","last_peered":"2024-06-29T15:57:33.356661+0000","last_clean":"2024-06-29T15:57:33.356661+0000","last_became_active":"2024-06-29T15:57:21.273632+0000","last_became_peered":"2024-06-29T15:57:21.273632+0000","last_unstale":"2024-06-29T15:57:33.356661+0000","last_undegraded":"2024-06-29T15:57:33.356661+0000","last_fullsized":"2024-06-29T15:57:33.356661+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361270+0000","last_change":"2024-06-29T15:57:22.297633+0000","last_active":"2024-06-29T15:57:33.361270+0000","last_peered":"2024-06-29T15:57:33.361270+0000","last_clean":"2024-06-29T15:57:33.361270+0000","last_became_active":"2024-06-29T15:57:22.297319+0000","last_became_peered":"2024-06-29T15:57:22.297319+0000","last_unstale":"2024-06-29T15:57:33.361270+0000","last_undegraded":"2024-06-29T15:57:33.361270+0000","last_fullsized":"2024-06-29T15:57:33.361270+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:19.625176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356850+0000","last_change":"2024-06-29T15:57:33.355194+0000","last_active":"2024-06-29T15:57:33.356850+0000","last_peered":"2024-06-29T15:57:33.356850+0000","last_clean":"2024-06-29T15:57:33.356850+0000","last_became_active":"2024-06-29T15:57:33.354883+0000","last_became_peered":"2024-06-29T15:57:33.354883+0000","last_unstale":"2024-06-29T15:57:33.356850+0000","last_undegraded":"2024-06-29T15:57:33.356850+0000","last_fullsized":"2024-06-29T15:57:33.356850+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:06.173920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351788+0000","last_change":"2024-06-29T15:57:33.351788+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351788+0000","last_undegraded":"2024-06-29T15:57:33.351788+0000","last_fullsized":"2024-06-29T15:57:33.351788+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361211+0000","last_change":"2024-06-29T15:57:21.283411+0000","last_active":"2024-06-29T15:57:33.361211+0000","last_peered":"2024-06-29T15:57:33.361211+0000","last_clean":"2024-06-29T15:57:33.361211+0000","last_became_active":"2024-06-29T15:57:21.283081+0000","last_became_peered":"2024-06-29T15:57:21.283081+0000","last_unstale":"2024-06-29T15:57:33.361211+0000","last_undegraded":"2024-06-29T15:57:33.361211+0000","last_fullsized":"2024-06-29T15:57:33.361211+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:27.866266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348680+0000","last_change":"2024-06-29T15:57:22.306835+0000","last_active":"2024-06-29T15:57:33.348680+0000","last_peered":"2024-06-29T15:57:33.348680+0000","last_clean":"2024-06-29T15:57:33.348680+0000","last_became_active":"2024-06-29T15:57:22.306295+0000","last_became_peered":"2024-06-29T15:57:22.306295+0000","last_unstale":"2024-06-29T15:57:33.348680+0000","last_undegraded":"2024-06-29T15:57:33.348680+0000","last_fullsized":"2024-06-29T15:57:33.348680+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:52.124235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367249+0000","last_change":"2024-06-29T15:57:33.365463+0000","last_active":"2024-06-29T15:57:33.367249+0000","last_peered":"2024-06-29T15:57:33.367249+0000","last_clean":"2024-06-29T15:57:33.367249+0000","last_became_active":"2024-06-29T15:57:33.365149+0000","last_became_peered":"2024-06-29T15:57:33.365149+0000","last_unstale":"2024-06-29T15:57:33.367249+0000","last_undegraded":"2024-06-29T15:57:33.367249+0000","last_fullsized":"2024-06-29T15:57:33.367249+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:35.252184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356954+0000","last_change":"2024-06-29T15:57:33.356954+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356954+0000","last_undegraded":"2024-06-29T15:57:33.356954+0000","last_fullsized":"2024-06-29T15:57:33.356954+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356973+0000","last_change":"2024-06-29T15:57:21.284516+0000","last_active":"2024-06-29T15:57:33.356973+0000","last_peered":"2024-06-29T15:57:33.356973+0000","last_clean":"2024-06-29T15:57:33.356973+0000","last_became_active":"2024-06-29T15:57:21.284170+0000","last_became_peered":"2024-06-29T15:57:21.284170+0000","last_unstale":"2024-06-29T15:57:33.356973+0000","last_undegraded":"2024-06-29T15:57:33.356973+0000","last_fullsized":"2024-06-29T15:57:33.356973+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.352840+0000","last_change":"2024-06-29T15:57:22.300202+0000","last_active":"2024-06-29T15:57:33.352840+0000","last_peered":"2024-06-29T15:57:33.352840+0000","last_clean":"2024-06-29T15:57:33.352840+0000","last_became_active":"2024-06-29T15:57:22.299897+0000","last_became_peered":"2024-06-29T15:57:22.299897+0000","last_unstale":"2024-06-29T15:57:33.352840+0000","last_undegraded":"2024-06-29T15:57:33.352840+0000","last_fullsized":"2024-06-29T15:57:33.352840+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364931+0000","last_change":"2024-06-29T15:57:33.363355+0000","last_active":"2024-06-29T15:57:33.364931+0000","last_peered":"2024-06-29T15:57:33.364931+0000","last_clean":"2024-06-29T15:57:33.364931+0000","last_became_active":"2024-06-29T15:57:33.363067+0000","last_became_peered":"2024-06-29T15:57:33.363067+0000","last_unstale":"2024-06-29T15:57:33.364931+0000","last_undegraded":"2024-06-29T15:57:33.364931+0000","last_fullsized":"2024-06-29T15:57:33.364931+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351200+0000","last_change":"2024-06-29T15:57:33.351200+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351200+0000","last_undegraded":"2024-06-29T15:57:33.351200+0000","last_fullsized":"2024-06-29T15:57:33.351200+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356344+0000","last_change":"2024-06-29T15:57:21.274833+0000","last_active":"2024-06-29T15:57:33.356344+0000","last_peered":"2024-06-29T15:57:33.356344+0000","last_clean":"2024-06-29T15:57:33.356344+0000","last_became_active":"2024-06-29T15:57:21.274451+0000","last_became_peered":"2024-06-29T15:57:21.274451+0000","last_unstale":"2024-06-29T15:57:33.356344+0000","last_undegraded":"2024-06-29T15:57:33.356344+0000","last_fullsized":"2024-06-29T15:57:33.356344+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:25.284538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354475+0000","last_change":"2024-06-29T15:57:22.301084+0000","last_active":"2024-06-29T15:57:33.354475+0000","last_peered":"2024-06-29T15:57:33.354475+0000","last_clean":"2024-06-29T15:57:33.354475+0000","last_became_active":"2024-06-29T15:57:22.300639+0000","last_became_peered":"2024-06-29T15:57:22.300639+0000","last_unstale":"2024-06-29T15:57:33.354475+0000","last_undegraded":"2024-06-29T15:57:33.354475+0000","last_fullsized":"2024-06-29T15:57:33.354475+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361064+0000","last_change":"2024-06-29T15:57:33.358185+0000","last_active":"2024-06-29T15:57:33.361064+0000","last_peered":"2024-06-29T15:57:33.361064+0000","last_clean":"2024-06-29T15:57:33.361064+0000","last_became_active":"2024-06-29T15:57:33.357885+0000","last_became_peered":"2024-06-29T15:57:33.357885+0000","last_unstale":"2024-06-29T15:57:33.361064+0000","last_undegraded":"2024-06-29T15:57:33.361064+0000","last_fullsized":"2024-06-29T15:57:33.361064+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:50:30.619228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351153+0000","last_change":"2024-06-29T15:57:33.351153+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351153+0000","last_undegraded":"2024-06-29T15:57:33.351153+0000","last_fullsized":"2024-06-29T15:57:33.351153+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"212'4","reported_seq":22,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365147+0000","last_change":"2024-06-29T15:57:21.276327+0000","last_active":"2024-06-29T15:57:33.365147+0000","last_peered":"2024-06-29T15:57:33.365147+0000","last_clean":"2024-06-29T15:57:33.365147+0000","last_became_active":"2024-06-29T15:57:21.276031+0000","last_became_peered":"2024-06-29T15:57:21.276031+0000","last_unstale":"2024-06-29T15:57:33.365147+0000","last_undegraded":"2024-06-29T15:57:33.365147+0000","last_fullsized":"2024-06-29T15:57:33.365147+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:23.626777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356074+0000","last_change":"2024-06-29T15:57:22.293417+0000","last_active":"2024-06-29T15:57:33.356074+0000","last_peered":"2024-06-29T15:57:33.356074+0000","last_clean":"2024-06-29T15:57:33.356074+0000","last_became_active":"2024-06-29T15:57:22.293083+0000","last_became_peered":"2024-06-29T15:57:22.293083+0000","last_unstale":"2024-06-29T15:57:33.356074+0000","last_undegraded":"2024-06-29T15:57:33.356074+0000","last_fullsized":"2024-06-29T15:57:33.356074+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359057+0000","last_change":"2024-06-29T15:57:33.356187+0000","last_active":"2024-06-29T15:57:33.359057+0000","last_peered":"2024-06-29T15:57:33.359057+0000","last_clean":"2024-06-29T15:57:33.359057+0000","last_became_active":"2024-06-29T15:57:33.355400+0000","last_became_peered":"2024-06-29T15:57:33.355400+0000","last_unstale":"2024-06-29T15:57:33.359057+0000","last_undegraded":"2024-06-29T15:57:33.359057+0000","last_fullsized":"2024-06-29T15:57:33.359057+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:01.393531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354238+0000","last_change":"2024-06-29T15:57:33.354238+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354238+0000","last_undegraded":"2024-06-29T15:57:33.354238+0000","last_fullsized":"2024-06-29T15:57:33.354238+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356423+0000","last_change":"2024-06-29T15:57:22.298975+0000","last_active":"2024-06-29T15:57:33.356423+0000","last_peered":"2024-06-29T15:57:33.356423+0000","last_clean":"2024-06-29T15:57:33.356423+0000","last_became_active":"2024-06-29T15:57:22.298689+0000","last_became_peered":"2024-06-29T15:57:22.298689+0000","last_unstale":"2024-06-29T15:57:33.356423+0000","last_undegraded":"2024-06-29T15:57:33.356423+0000","last_fullsized":"2024-06-29T15:57:33.356423+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359369+0000","last_change":"2024-06-29T15:57:33.359369+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359369+0000","last_undegraded":"2024-06-29T15:57:33.359369+0000","last_fullsized":"2024-06-29T15:57:33.359369+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348068+0000","last_change":"2024-06-29T15:57:22.292249+0000","last_active":"2024-06-29T15:57:33.348068+0000","last_peered":"2024-06-29T15:57:33.348068+0000","last_clean":"2024-06-29T15:57:33.348068+0000","last_became_active":"2024-06-29T15:57:22.291918+0000","last_became_peered":"2024-06-29T15:57:22.291918+0000","last_unstale":"2024-06-29T15:57:33.348068+0000","last_undegraded":"2024-06-29T15:57:33.348068+0000","last_fullsized":"2024-06-29T15:57:33.348068+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:57.129905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351129+0000","last_change":"2024-06-29T15:57:33.351129+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351129+0000","last_undegraded":"2024-06-29T15:57:33.351129+0000","last_fullsized":"2024-06-29T15:57:33.351129+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355012+0000","last_change":"2024-06-29T15:57:22.285362+0000","last_active":"2024-06-29T15:57:33.355012+0000","last_peered":"2024-06-29T15:57:33.355012+0000","last_clean":"2024-06-29T15:57:33.355012+0000","last_became_active":"2024-06-29T15:57:22.284982+0000","last_became_peered":"2024-06-29T15:57:22.284982+0000","last_unstale":"2024-06-29T15:57:33.355012+0000","last_undegraded":"2024-06-29T15:57:33.355012+0000","last_fullsized":"2024-06-29T15:57:33.355012+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357913+0000","last_change":"2024-06-29T15:57:33.357913+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357913+0000","last_undegraded":"2024-06-29T15:57:33.357913+0000","last_fullsized":"2024-06-29T15:57:33.357913+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364482+0000","last_change":"2024-06-29T15:57:22.297998+0000","last_active":"2024-06-29T15:57:33.364482+0000","last_peered":"2024-06-29T15:57:33.364482+0000","last_clean":"2024-06-29T15:57:33.364482+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:33.364482+0000","last_undegraded":"2024-06-29T15:57:33.364482+0000","last_fullsized":"2024-06-29T15:57:33.364482+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357574+0000","last_change":"2024-06-29T15:57:33.357574+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357574+0000","last_undegraded":"2024-06-29T15:57:33.357574+0000","last_fullsized":"2024-06-29T15:57:33.357574+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356051+0000","last_change":"2024-06-29T15:57:22.297917+0000","last_active":"2024-06-29T15:57:33.356051+0000","last_peered":"2024-06-29T15:57:33.356051+0000","last_clean":"2024-06-29T15:57:33.356051+0000","last_became_active":"2024-06-29T15:57:22.297638+0000","last_became_peered":"2024-06-29T15:57:22.297638+0000","last_unstale":"2024-06-29T15:57:33.356051+0000","last_undegraded":"2024-06-29T15:57:33.356051+0000","last_fullsized":"2024-06-29T15:57:33.356051+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351060+0000","last_change":"2024-06-29T15:57:33.351060+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351060+0000","last_undegraded":"2024-06-29T15:57:33.351060+0000","last_fullsized":"2024-06-29T15:57:33.351060+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364399+0000","last_change":"2024-06-29T15:57:22.298174+0000","last_active":"2024-06-29T15:57:33.364399+0000","last_peered":"2024-06-29T15:57:33.364399+0000","last_clean":"2024-06-29T15:57:33.364399+0000","last_became_active":"2024-06-29T15:57:22.297862+0000","last_became_peered":"2024-06-29T15:57:22.297862+0000","last_unstale":"2024-06-29T15:57:33.364399+0000","last_undegraded":"2024-06-29T15:57:33.364399+0000","last_fullsized":"2024-06-29T15:57:33.364399+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354825+0000","last_change":"2024-06-29T15:57:33.354825+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354825+0000","last_undegraded":"2024-06-29T15:57:33.354825+0000","last_fullsized":"2024-06-29T15:57:33.354825+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358628+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:33.358628+0000","last_peered":"2024-06-29T15:57:33.358628+0000","last_clean":"2024-06-29T15:57:33.358628+0000","last_became_active":"2024-06-29T15:57:22.301541+0000","last_became_peered":"2024-06-29T15:57:22.301541+0000","last_unstale":"2024-06-29T15:57:33.358628+0000","last_undegraded":"2024-06-29T15:57:33.358628+0000","last_fullsized":"2024-06-29T15:57:33.358628+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:55.249516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353618+0000","last_change":"2024-06-29T15:57:33.353618+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353618+0000","last_undegraded":"2024-06-29T15:57:33.353618+0000","last_fullsized":"2024-06-29T15:57:33.353618+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355658+0000","last_change":"2024-06-29T15:57:22.292584+0000","last_active":"2024-06-29T15:57:33.355658+0000","last_peered":"2024-06-29T15:57:33.355658+0000","last_clean":"2024-06-29T15:57:33.355658+0000","last_became_active":"2024-06-29T15:57:22.291806+0000","last_became_peered":"2024-06-29T15:57:22.291806+0000","last_unstale":"2024-06-29T15:57:33.355658+0000","last_undegraded":"2024-06-29T15:57:33.355658+0000","last_fullsized":"2024-06-29T15:57:33.355658+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352348+0000","last_change":"2024-06-29T15:57:33.352348+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352348+0000","last_undegraded":"2024-06-29T15:57:33.352348+0000","last_fullsized":"2024-06-29T15:57:33.352348+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363619+0000","last_change":"2024-06-29T15:57:22.301601+0000","last_active":"2024-06-29T15:57:33.363619+0000","last_peered":"2024-06-29T15:57:33.363619+0000","last_clean":"2024-06-29T15:57:33.363619+0000","last_became_active":"2024-06-29T15:57:22.301244+0000","last_became_peered":"2024-06-29T15:57:22.301244+0000","last_unstale":"2024-06-29T15:57:33.363619+0000","last_undegraded":"2024-06-29T15:57:33.363619+0000","last_fullsized":"2024-06-29T15:57:33.363619+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:24:56.938464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352379+0000","last_change":"2024-06-29T15:57:33.352379+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352379+0000","last_undegraded":"2024-06-29T15:57:33.352379+0000","last_fullsized":"2024-06-29T15:57:33.352379+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353891+0000","last_change":"2024-06-29T15:57:22.301362+0000","last_active":"2024-06-29T15:57:33.353891+0000","last_peered":"2024-06-29T15:57:33.353891+0000","last_clean":"2024-06-29T15:57:33.353891+0000","last_became_active":"2024-06-29T15:57:22.300704+0000","last_became_peered":"2024-06-29T15:57:22.300704+0000","last_unstale":"2024-06-29T15:57:33.353891+0000","last_undegraded":"2024-06-29T15:57:33.353891+0000","last_fullsized":"2024-06-29T15:57:33.353891+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351810+0000","last_change":"2024-06-29T15:57:33.351810+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351810+0000","last_undegraded":"2024-06-29T15:57:33.351810+0000","last_fullsized":"2024-06-29T15:57:33.351810+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353925+0000","last_change":"2024-06-29T15:57:22.307686+0000","last_active":"2024-06-29T15:57:33.353925+0000","last_peered":"2024-06-29T15:57:33.353925+0000","last_clean":"2024-06-29T15:57:33.353925+0000","last_became_active":"2024-06-29T15:57:22.307324+0000","last_became_peered":"2024-06-29T15:57:22.307324+0000","last_unstale":"2024-06-29T15:57:33.353925+0000","last_undegraded":"2024-06-29T15:57:33.353925+0000","last_fullsized":"2024-06-29T15:57:33.353925+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353324+0000","last_change":"2024-06-29T15:57:33.353324+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353324+0000","last_undegraded":"2024-06-29T15:57:33.353324+0000","last_fullsized":"2024-06-29T15:57:33.353324+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361177+0000","last_change":"2024-06-29T15:57:22.307230+0000","last_active":"2024-06-29T15:57:33.361177+0000","last_peered":"2024-06-29T15:57:33.361177+0000","last_clean":"2024-06-29T15:57:33.361177+0000","last_became_active":"2024-06-29T15:57:22.306920+0000","last_became_peered":"2024-06-29T15:57:22.306920+0000","last_unstale":"2024-06-29T15:57:33.361177+0000","last_undegraded":"2024-06-29T15:57:33.361177+0000","last_fullsized":"2024-06-29T15:57:33.361177+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:22.598647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352631+0000","last_change":"2024-06-29T15:57:33.352631+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352631+0000","last_undegraded":"2024-06-29T15:57:33.352631+0000","last_fullsized":"2024-06-29T15:57:33.352631+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359396+0000","last_change":"2024-06-29T15:57:22.290186+0000","last_active":"2024-06-29T15:57:33.359396+0000","last_peered":"2024-06-29T15:57:33.359396+0000","last_clean":"2024-06-29T15:57:33.359396+0000","last_became_active":"2024-06-29T15:57:22.289908+0000","last_became_peered":"2024-06-29T15:57:22.289908+0000","last_unstale":"2024-06-29T15:57:33.359396+0000","last_undegraded":"2024-06-29T15:57:33.359396+0000","last_fullsized":"2024-06-29T15:57:33.359396+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353284+0000","last_change":"2024-06-29T15:57:33.353284+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353284+0000","last_undegraded":"2024-06-29T15:57:33.353284+0000","last_fullsized":"2024-06-29T15:57:33.353284+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348980+0000","last_change":"2024-06-29T15:57:22.306593+0000","last_active":"2024-06-29T15:57:33.348980+0000","last_peered":"2024-06-29T15:57:33.348980+0000","last_clean":"2024-06-29T15:57:33.348980+0000","last_became_active":"2024-06-29T15:57:22.296653+0000","last_became_peered":"2024-06-29T15:57:22.296653+0000","last_unstale":"2024-06-29T15:57:33.348980+0000","last_undegraded":"2024-06-29T15:57:33.348980+0000","last_fullsized":"2024-06-29T15:57:33.348980+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:41.738155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351749+0000","last_change":"2024-06-29T15:57:33.351749+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351749+0000","last_undegraded":"2024-06-29T15:57:33.351749+0000","last_fullsized":"2024-06-29T15:57:33.351749+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348828+0000","last_change":"2024-06-29T15:57:22.289633+0000","last_active":"2024-06-29T15:57:33.348828+0000","last_peered":"2024-06-29T15:57:33.348828+0000","last_clean":"2024-06-29T15:57:33.348828+0000","last_became_active":"2024-06-29T15:57:22.289349+0000","last_became_peered":"2024-06-29T15:57:22.289349+0000","last_unstale":"2024-06-29T15:57:33.348828+0000","last_undegraded":"2024-06-29T15:57:33.348828+0000","last_fullsized":"2024-06-29T15:57:33.348828+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.873806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357678+0000","last_change":"2024-06-29T15:57:33.357678+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357678+0000","last_undegraded":"2024-06-29T15:57:33.357678+0000","last_fullsized":"2024-06-29T15:57:33.357678+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355608+0000","last_change":"2024-06-29T15:57:22.300787+0000","last_active":"2024-06-29T15:57:33.355608+0000","last_peered":"2024-06-29T15:57:33.355608+0000","last_clean":"2024-06-29T15:57:33.355608+0000","last_became_active":"2024-06-29T15:57:22.300112+0000","last_became_peered":"2024-06-29T15:57:22.300112+0000","last_unstale":"2024-06-29T15:57:33.355608+0000","last_undegraded":"2024-06-29T15:57:33.355608+0000","last_fullsized":"2024-06-29T15:57:33.355608+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:30.249483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353010+0000","last_change":"2024-06-29T15:57:33.353010+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353010+0000","last_undegraded":"2024-06-29T15:57:33.353010+0000","last_fullsized":"2024-06-29T15:57:33.353010+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348033+0000","last_change":"2024-06-29T15:57:22.290047+0000","last_active":"2024-06-29T15:57:33.348033+0000","last_peered":"2024-06-29T15:57:33.348033+0000","last_clean":"2024-06-29T15:57:33.348033+0000","last_became_active":"2024-06-29T15:57:22.289742+0000","last_became_peered":"2024-06-29T15:57:22.289742+0000","last_unstale":"2024-06-29T15:57:33.348033+0000","last_undegraded":"2024-06-29T15:57:33.348033+0000","last_fullsized":"2024-06-29T15:57:33.348033+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:03:46.271206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362654+0000","last_change":"2024-06-29T15:57:33.362654+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362654+0000","last_undegraded":"2024-06-29T15:57:33.362654+0000","last_fullsized":"2024-06-29T15:57:33.362654+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356944+0000","last_change":"2024-06-29T15:57:22.302418+0000","last_active":"2024-06-29T15:57:33.356944+0000","last_peered":"2024-06-29T15:57:33.356944+0000","last_clean":"2024-06-29T15:57:33.356944+0000","last_became_active":"2024-06-29T15:57:22.302075+0000","last_became_peered":"2024-06-29T15:57:22.302075+0000","last_unstale":"2024-06-29T15:57:33.356944+0000","last_undegraded":"2024-06-29T15:57:33.356944+0000","last_fullsized":"2024-06-29T15:57:33.356944+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.350522+0000","last_change":"2024-06-29T15:57:33.350522+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.350522+0000","last_undegraded":"2024-06-29T15:57:33.350522+0000","last_fullsized":"2024-06-29T15:57:33.350522+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354196+0000","last_change":"2024-06-29T15:57:22.303092+0000","last_active":"2024-06-29T15:57:33.354196+0000","last_peered":"2024-06-29T15:57:33.354196+0000","last_clean":"2024-06-29T15:57:33.354196+0000","last_became_active":"2024-06-29T15:57:22.302807+0000","last_became_peered":"2024-06-29T15:57:22.302807+0000","last_unstale":"2024-06-29T15:57:33.354196+0000","last_undegraded":"2024-06-29T15:57:33.354196+0000","last_fullsized":"2024-06-29T15:57:33.354196+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360360+0000","last_change":"2024-06-29T15:57:33.360360+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360360+0000","last_undegraded":"2024-06-29T15:57:33.360360+0000","last_fullsized":"2024-06-29T15:57:33.360360+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365064+0000","last_change":"2024-06-29T15:57:22.295681+0000","last_active":"2024-06-29T15:57:33.365064+0000","last_peered":"2024-06-29T15:57:33.365064+0000","last_clean":"2024-06-29T15:57:33.365064+0000","last_became_active":"2024-06-29T15:57:22.295357+0000","last_became_peered":"2024-06-29T15:57:22.295357+0000","last_unstale":"2024-06-29T15:57:33.365064+0000","last_undegraded":"2024-06-29T15:57:33.365064+0000","last_fullsized":"2024-06-29T15:57:33.365064+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353724+0000","last_change":"2024-06-29T15:57:33.353724+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353724+0000","last_undegraded":"2024-06-29T15:57:33.353724+0000","last_fullsized":"2024-06-29T15:57:33.353724+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365558+0000","last_change":"2024-06-29T15:57:22.298612+0000","last_active":"2024-06-29T15:57:33.365558+0000","last_peered":"2024-06-29T15:57:33.365558+0000","last_clean":"2024-06-29T15:57:33.365558+0000","last_became_active":"2024-06-29T15:57:22.298288+0000","last_became_peered":"2024-06-29T15:57:22.298288+0000","last_unstale":"2024-06-29T15:57:33.365558+0000","last_undegraded":"2024-06-29T15:57:33.365558+0000","last_fullsized":"2024-06-29T15:57:33.365558+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351291+0000","last_change":"2024-06-29T15:57:33.351291+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351291+0000","last_undegraded":"2024-06-29T15:57:33.351291+0000","last_fullsized":"2024-06-29T15:57:33.351291+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356870+0000","last_change":"2024-06-29T15:57:22.298781+0000","last_active":"2024-06-29T15:57:33.356870+0000","last_peered":"2024-06-29T15:57:33.356870+0000","last_clean":"2024-06-29T15:57:33.356870+0000","last_became_active":"2024-06-29T15:57:22.298300+0000","last_became_peered":"2024-06-29T15:57:22.298300+0000","last_unstale":"2024-06-29T15:57:33.356870+0000","last_undegraded":"2024-06-29T15:57:33.356870+0000","last_fullsized":"2024-06-29T15:57:33.356870+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349621+0000","last_change":"2024-06-29T15:57:33.349621+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349621+0000","last_undegraded":"2024-06-29T15:57:33.349621+0000","last_fullsized":"2024-06-29T15:57:33.349621+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348689+0000","last_change":"2024-06-29T15:57:22.296441+0000","last_active":"2024-06-29T15:57:33.348689+0000","last_peered":"2024-06-29T15:57:33.348689+0000","last_clean":"2024-06-29T15:57:33.348689+0000","last_became_active":"2024-06-29T15:57:22.296146+0000","last_became_peered":"2024-06-29T15:57:22.296146+0000","last_unstale":"2024-06-29T15:57:33.348689+0000","last_undegraded":"2024-06-29T15:57:33.348689+0000","last_fullsized":"2024-06-29T15:57:33.348689+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:54.360521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356517+0000","last_change":"2024-06-29T15:57:33.356517+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356517+0000","last_undegraded":"2024-06-29T15:57:33.356517+0000","last_fullsized":"2024-06-29T15:57:33.356517+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356057+0000","last_change":"2024-06-29T15:57:22.291096+0000","last_active":"2024-06-29T15:57:33.356057+0000","last_peered":"2024-06-29T15:57:33.356057+0000","last_clean":"2024-06-29T15:57:33.356057+0000","last_became_active":"2024-06-29T15:57:22.290428+0000","last_became_peered":"2024-06-29T15:57:22.290428+0000","last_unstale":"2024-06-29T15:57:33.356057+0000","last_undegraded":"2024-06-29T15:57:33.356057+0000","last_fullsized":"2024-06-29T15:57:33.356057+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:20.813793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352280+0000","last_change":"2024-06-29T15:57:33.352280+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352280+0000","last_undegraded":"2024-06-29T15:57:33.352280+0000","last_fullsized":"2024-06-29T15:57:33.352280+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355022+0000","last_change":"2024-06-29T15:57:22.297617+0000","last_active":"2024-06-29T15:57:33.355022+0000","last_peered":"2024-06-29T15:57:33.355022+0000","last_clean":"2024-06-29T15:57:33.355022+0000","last_became_active":"2024-06-29T15:57:22.297256+0000","last_became_peered":"2024-06-29T15:57:22.297256+0000","last_unstale":"2024-06-29T15:57:33.355022+0000","last_undegraded":"2024-06-29T15:57:33.355022+0000","last_fullsized":"2024-06-29T15:57:33.355022+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359862+0000","last_change":"2024-06-29T15:57:33.359862+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359862+0000","last_undegraded":"2024-06-29T15:57:33.359862+0000","last_fullsized":"2024-06-29T15:57:33.359862+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356834+0000","last_change":"2024-06-29T15:57:22.304052+0000","last_active":"2024-06-29T15:57:33.356834+0000","last_peered":"2024-06-29T15:57:33.356834+0000","last_clean":"2024-06-29T15:57:33.356834+0000","last_became_active":"2024-06-29T15:57:22.303598+0000","last_became_peered":"2024-06-29T15:57:22.303598+0000","last_unstale":"2024-06-29T15:57:33.356834+0000","last_undegraded":"2024-06-29T15:57:33.356834+0000","last_fullsized":"2024-06-29T15:57:33.356834+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352198+0000","last_change":"2024-06-29T15:57:33.352198+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352198+0000","last_undegraded":"2024-06-29T15:57:33.352198+0000","last_fullsized":"2024-06-29T15:57:33.352198+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356415+0000","last_change":"2024-06-29T15:57:22.288041+0000","last_active":"2024-06-29T15:57:33.356415+0000","last_peered":"2024-06-29T15:57:33.356415+0000","last_clean":"2024-06-29T15:57:33.356415+0000","last_became_active":"2024-06-29T15:57:22.287729+0000","last_became_peered":"2024-06-29T15:57:22.287729+0000","last_unstale":"2024-06-29T15:57:33.356415+0000","last_undegraded":"2024-06-29T15:57:33.356415+0000","last_fullsized":"2024-06-29T15:57:33.356415+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362099+0000","last_change":"2024-06-29T15:57:33.362099+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362099+0000","last_undegraded":"2024-06-29T15:57:33.362099+0000","last_fullsized":"2024-06-29T15:57:33.362099+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356673+0000","last_change":"2024-06-29T15:57:22.290909+0000","last_active":"2024-06-29T15:57:33.356673+0000","last_peered":"2024-06-29T15:57:33.356673+0000","last_clean":"2024-06-29T15:57:33.356673+0000","last_became_active":"2024-06-29T15:57:22.290643+0000","last_became_peered":"2024-06-29T15:57:22.290643+0000","last_unstale":"2024-06-29T15:57:33.356673+0000","last_undegraded":"2024-06-29T15:57:33.356673+0000","last_fullsized":"2024-06-29T15:57:33.356673+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354913+0000","last_change":"2024-06-29T15:57:33.354913+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354913+0000","last_undegraded":"2024-06-29T15:57:33.354913+0000","last_fullsized":"2024-06-29T15:57:33.354913+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361010+0000","last_change":"2024-06-29T15:57:22.298374+0000","last_active":"2024-06-29T15:57:33.361010+0000","last_peered":"2024-06-29T15:57:33.361010+0000","last_clean":"2024-06-29T15:57:33.361010+0000","last_became_active":"2024-06-29T15:57:22.297962+0000","last_became_peered":"2024-06-29T15:57:22.297962+0000","last_unstale":"2024-06-29T15:57:33.361010+0000","last_undegraded":"2024-06-29T15:57:33.361010+0000","last_fullsized":"2024-06-29T15:57:33.361010+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:13.834244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354164+0000","last_change":"2024-06-29T15:57:33.354164+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354164+0000","last_undegraded":"2024-06-29T15:57:33.354164+0000","last_fullsized":"2024-06-29T15:57:33.354164+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356686+0000","last_change":"2024-06-29T15:57:22.300745+0000","last_active":"2024-06-29T15:57:33.356686+0000","last_peered":"2024-06-29T15:57:33.356686+0000","last_clean":"2024-06-29T15:57:33.356686+0000","last_became_active":"2024-06-29T15:57:22.300414+0000","last_became_peered":"2024-06-29T15:57:22.300414+0000","last_unstale":"2024-06-29T15:57:33.356686+0000","last_undegraded":"2024-06-29T15:57:33.356686+0000","last_fullsized":"2024-06-29T15:57:33.356686+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356852+0000","last_change":"2024-06-29T15:57:33.356852+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356852+0000","last_undegraded":"2024-06-29T15:57:33.356852+0000","last_fullsized":"2024-06-29T15:57:33.356852+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355941+0000","last_change":"2024-06-29T15:57:22.297971+0000","last_active":"2024-06-29T15:57:33.355941+0000","last_peered":"2024-06-29T15:57:33.355941+0000","last_clean":"2024-06-29T15:57:33.355941+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:33.355941+0000","last_undegraded":"2024-06-29T15:57:33.355941+0000","last_fullsized":"2024-06-29T15:57:33.355941+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360424+0000","last_change":"2024-06-29T15:57:33.360424+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360424+0000","last_undegraded":"2024-06-29T15:57:33.360424+0000","last_fullsized":"2024-06-29T15:57:33.360424+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355500+0000","last_change":"2024-06-29T15:57:22.301019+0000","last_active":"2024-06-29T15:57:33.355500+0000","last_peered":"2024-06-29T15:57:33.355500+0000","last_clean":"2024-06-29T15:57:33.355500+0000","last_became_active":"2024-06-29T15:57:22.300506+0000","last_became_peered":"2024-06-29T15:57:22.300506+0000","last_unstale":"2024-06-29T15:57:33.355500+0000","last_undegraded":"2024-06-29T15:57:33.355500+0000","last_fullsized":"2024-06-29T15:57:33.355500+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52456,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319384,"statfs":{"total":96636764160,"available":96583049216,"internally_reserved":0,"allocated":3305472,"data_stored":1914275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575988,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42140,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329700,"statfs":{"total":96636764160,"available":96593612800,"internally_reserved":0,"allocated":2703360,"data_stored":1321634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575986,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42128,"kb_used_data":2628,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329712,"statfs":{"total":96636764160,"available":96593625088,"internally_reserved":0,"allocated":2691072,"data_stored":1314370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60208,"kb_used_data":2656,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311632,"statfs":{"total":96636764160,"available":96575111168,"internally_reserved":0,"allocated":2719744,"data_stored":1324011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47716,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324124,"statfs":{"total":96636764160,"available":96587902976,"internally_reserved":0,"allocated":2711552,"data_stored":1321234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575987,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47704,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324136,"statfs":{"total":96636764160,"available":96587915264,"internally_reserved":0,"allocated":2699264,"data_stored":1321052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575986,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60180,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311660,"statfs":{"total":96636764160,"available":96575139840,"internally_reserved":0,"allocated":2695168,"data_stored":1313844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60756,"kb_used_data":3208,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311084,"statfs":{"total":96636764160,"available":96574550016,"internally_reserved":0,"allocated":3284992,"data_stored":1904246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:38.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.553+0000 7fcca28c1640 1 -- 172.21.15.81:0/1175841754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9c06fa90 msgr2=0x7fcc9c06ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.553+0000 7fcca28c1640 1 --2- 172.21.15.81:0/1175841754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9c06fa90 0x7fcc9c06ca00 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fcc94005fd0 tx=0x7fcc9402aa00 comp rx=0 tx=0).stop 2024-06-29T15:57:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.554+0000 7fcca28c1640 1 -- 172.21.15.81:0/1175841754 shutdown_connections 2024-06-29T15:57:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.554+0000 7fcca28c1640 1 --2- 172.21.15.81:0/1175841754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9c06fa90 0x7fcc9c06ca00 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.554+0000 7fcca28c1640 1 --2- 172.21.15.81:0/1175841754 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc9c074520 0x7fcc9c06f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.554+0000 7fcca28c1640 1 -- 172.21.15.81:0/1175841754 >> 172.21.15.81:0/1175841754 conn(0x7fcc9c072950 msgr2=0x7fcc9c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.554+0000 7fcca28c1640 1 -- 172.21.15.81:0/1175841754 shutdown_connections 2024-06-29T15:57:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.554+0000 7fcca28c1640 1 -- 172.21.15.81:0/1175841754 wait complete. 2024-06-29T15:57:38.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.555+0000 7fcca28c1640 1 Processor -- start 2024-06-29T15:57:38.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.555+0000 7fcca28c1640 1 -- start start 2024-06-29T15:57:38.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcca28c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9c06f680 0x7fcc9c14fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcca28c1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9c06fa90 0x7fcc9c1502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcca28c1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc9c074520 0x7fcc9c147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcca28c1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c15b010 con 0x7fcc9c06f680 2024-06-29T15:57:38.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcca28c1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c06d150 con 0x7fcc9c06fa90 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcca28c1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcc9c15b190 con 0x7fcc9c074520 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc9bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9c06f680 0x7fcc9c14fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc9bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9c06f680 0x7fcc9c14fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35446/0 (socket says 172.21.15.81:35446) 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc9bfff640 1 -- 172.21.15.81:0/101926377 learned_addr learned my addr 172.21.15.81:0/101926377 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcca0e5e640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc9c074520 0x7fcc9c147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc93fff640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9c06fa90 0x7fcc9c1502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc9bfff640 1 -- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc9c074520 msgr2=0x7fcc9c147b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc9bfff640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc9c074520 0x7fcc9c147b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc9bfff640 1 -- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9c06fa90 msgr2=0x7fcc9c1502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc9bfff640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9c06fa90 0x7fcc9c1502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc9bfff640 1 -- 172.21.15.81:0/101926377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9c1f30a0 con 0x7fcc9c06f680 2024-06-29T15:57:38.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc93fff640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9c06fa90 0x7fcc9c1502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcca0e5e640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc9c074520 0x7fcc9c147b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc9bfff640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9c06f680 0x7fcc9c14fd70 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00d6d0 tx=0x7fcc8c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.557+0000 7fcc997fa640 1 -- 172.21.15.81:0/101926377 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c017070 con 0x7fcc9c06f680 2024-06-29T15:57:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.558+0000 7fcc997fa640 1 -- 172.21.15.81:0/101926377 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc8c005330 con 0x7fcc9c06f680 2024-06-29T15:57:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.558+0000 7fcca28c1640 1 -- 172.21.15.81:0/101926377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc9c1f3380 con 0x7fcc9c06f680 2024-06-29T15:57:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.558+0000 7fcc997fa640 1 -- 172.21.15.81:0/101926377 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c009820 con 0x7fcc9c06f680 2024-06-29T15:57:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.558+0000 7fcca28c1640 1 -- 172.21.15.81:0/101926377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc9c1f3940 con 0x7fcc9c06f680 2024-06-29T15:57:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.558+0000 7fcca28c1640 1 -- 172.21.15.81:0/101926377 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc9c0ad590 con 0x7fcc9c06f680 2024-06-29T15:57:38.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.559+0000 7fcc997fa640 1 -- 172.21.15.81:0/101926377 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fcc8c0099c0 con 0x7fcc9c06f680 2024-06-29T15:57:38.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.560+0000 7fcc997fa640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcc7c076420 0x7fcc7c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.560+0000 7fcc997fa640 1 -- 172.21.15.81:0/101926377 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7fcc8c097df0 con 0x7fcc9c06f680 2024-06-29T15:57:38.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.560+0000 7fcc93fff640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcc7c076420 0x7fcc7c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.560+0000 7fcc93fff640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcc7c076420 0x7fcc7c078930 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c148d60 tx=0x7fcc84008040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:38.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.564+0000 7fcc997fa640 1 -- 172.21.15.81:0/101926377 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcc8c061150 con 0x7fcc9c06f680 2024-06-29T15:57:38.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.741+0000 7fcca28c1640 1 -- 172.21.15.81:0/101926377 --> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc9c06f140 con 0x7fcc7c076420 2024-06-29T15:57:38.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.749+0000 7fcc997fa640 1 -- 172.21.15.81:0/101926377 <== mgr.8666 v2:172.21.15.167:6832/2018755589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+486086 (secure 0 0 0) 0x7fcc9c06f140 con 0x7fcc7c076420 2024-06-29T15:57:38.756 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:38.759 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:57:38.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 -- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcc7c076420 msgr2=0x7fcc7c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcc7c076420 0x7fcc7c078930 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c148d60 tx=0x7fcc84008040 comp rx=0 tx=0).stop 2024-06-29T15:57:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 -- 172.21.15.81:0/101926377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9c06f680 msgr2=0x7fcc9c14fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9c06f680 0x7fcc9c14fd70 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00d6d0 tx=0x7fcc8c00db90 comp rx=0 tx=0).stop 2024-06-29T15:57:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 -- 172.21.15.81:0/101926377 shutdown_connections 2024-06-29T15:57:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fcc7c076420 0x7fcc7c078930 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcc9c074520 0x7fcc9c147b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcc9c06fa90 0x7fcc9c1502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 --2- 172.21.15.81:0/101926377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc9c06f680 0x7fcc9c14fd70 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 -- 172.21.15.81:0/101926377 >> 172.21.15.81:0/101926377 conn(0x7fcc9c072950 msgr2=0x7fcc9c15b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 -- 172.21.15.81:0/101926377 shutdown_connections 2024-06-29T15:57:38.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.753+0000 7fcca28c1640 1 -- 172.21.15.81:0/101926377 wait complete. 2024-06-29T15:57:38.766 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-06-29T15:57:37.873180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594021,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":400,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":217,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413288,"kb_used_data":22276,"kb_used_omap":12,"kb_used_meta":390899,"kb_avail":754561432,"statfs":{"total":773094113280,"available":772670906368,"internally_reserved":0,"allocated":22810624,"data_stored":11734666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":400281138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1243,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.532321"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357377+0000","last_change":"2024-06-29T15:57:33.357377+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357377+0000","last_undegraded":"2024-06-29T15:57:33.357377+0000","last_fullsized":"2024-06-29T15:57:33.357377+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"202'219","reported_seq":1043,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363727+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:57:33.363727+0000","last_peered":"2024-06-29T15:57:33.363727+0000","last_clean":"2024-06-29T15:57:33.363727+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:57:33.363727+0000","last_undegraded":"2024-06-29T15:57:33.363727+0000","last_fullsized":"2024-06-29T15:57:33.363727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365224+0000","last_change":"2024-06-29T15:57:21.282026+0000","last_active":"2024-06-29T15:57:33.365224+0000","last_peered":"2024-06-29T15:57:33.365224+0000","last_clean":"2024-06-29T15:57:33.365224+0000","last_became_active":"2024-06-29T15:57:21.281709+0000","last_became_peered":"2024-06-29T15:57:21.281709+0000","last_unstale":"2024-06-29T15:57:33.365224+0000","last_undegraded":"2024-06-29T15:57:33.365224+0000","last_fullsized":"2024-06-29T15:57:33.365224+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:54:07.641300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354339+0000","last_change":"2024-06-29T15:57:22.302267+0000","last_active":"2024-06-29T15:57:33.354339+0000","last_peered":"2024-06-29T15:57:33.354339+0000","last_clean":"2024-06-29T15:57:33.354339+0000","last_became_active":"2024-06-29T15:57:22.301972+0000","last_became_peered":"2024-06-29T15:57:22.301972+0000","last_unstale":"2024-06-29T15:57:33.354339+0000","last_undegraded":"2024-06-29T15:57:33.354339+0000","last_fullsized":"2024-06-29T15:57:33.354339+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371713+0000","last_change":"2024-06-29T15:57:33.370756+0000","last_active":"2024-06-29T15:57:33.371713+0000","last_peered":"2024-06-29T15:57:33.371713+0000","last_clean":"2024-06-29T15:57:33.371713+0000","last_became_active":"2024-06-29T15:57:33.370363+0000","last_became_peered":"2024-06-29T15:57:33.370363+0000","last_unstale":"2024-06-29T15:57:33.371713+0000","last_undegraded":"2024-06-29T15:57:33.371713+0000","last_fullsized":"2024-06-29T15:57:33.371713+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:27.582662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362213+0000","last_change":"2024-06-29T15:57:33.362213+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362213+0000","last_undegraded":"2024-06-29T15:57:33.362213+0000","last_fullsized":"2024-06-29T15:57:33.362213+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364223+0000","last_change":"2024-06-29T15:57:21.283495+0000","last_active":"2024-06-29T15:57:33.364223+0000","last_peered":"2024-06-29T15:57:33.364223+0000","last_clean":"2024-06-29T15:57:33.364223+0000","last_became_active":"2024-06-29T15:57:21.283125+0000","last_became_peered":"2024-06-29T15:57:21.283125+0000","last_unstale":"2024-06-29T15:57:33.364223+0000","last_undegraded":"2024-06-29T15:57:33.364223+0000","last_fullsized":"2024-06-29T15:57:33.364223+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354722+0000","last_change":"2024-06-29T15:57:22.292872+0000","last_active":"2024-06-29T15:57:33.354722+0000","last_peered":"2024-06-29T15:57:33.354722+0000","last_clean":"2024-06-29T15:57:33.354722+0000","last_became_active":"2024-06-29T15:57:22.292485+0000","last_became_peered":"2024-06-29T15:57:22.292485+0000","last_unstale":"2024-06-29T15:57:33.354722+0000","last_undegraded":"2024-06-29T15:57:33.354722+0000","last_fullsized":"2024-06-29T15:57:33.354722+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:19.369559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370805+0000","last_change":"2024-06-29T15:57:33.369465+0000","last_active":"2024-06-29T15:57:33.370805+0000","last_peered":"2024-06-29T15:57:33.370805+0000","last_clean":"2024-06-29T15:57:33.370805+0000","last_became_active":"2024-06-29T15:57:33.369174+0000","last_became_peered":"2024-06-29T15:57:33.369174+0000","last_unstale":"2024-06-29T15:57:33.370805+0000","last_undegraded":"2024-06-29T15:57:33.370805+0000","last_fullsized":"2024-06-29T15:57:33.370805+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:02.836842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351888+0000","last_change":"2024-06-29T15:57:33.351888+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351888+0000","last_undegraded":"2024-06-29T15:57:33.351888+0000","last_fullsized":"2024-06-29T15:57:33.351888+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354196+0000","last_change":"2024-06-29T15:57:21.274359+0000","last_active":"2024-06-29T15:57:33.354196+0000","last_peered":"2024-06-29T15:57:33.354196+0000","last_clean":"2024-06-29T15:57:33.354196+0000","last_became_active":"2024-06-29T15:57:21.274036+0000","last_became_peered":"2024-06-29T15:57:21.274036+0000","last_unstale":"2024-06-29T15:57:33.354196+0000","last_undegraded":"2024-06-29T15:57:33.354196+0000","last_fullsized":"2024-06-29T15:57:33.354196+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:02.747066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358900+0000","last_change":"2024-06-29T15:57:22.295101+0000","last_active":"2024-06-29T15:57:33.358900+0000","last_peered":"2024-06-29T15:57:33.358900+0000","last_clean":"2024-06-29T15:57:33.358900+0000","last_became_active":"2024-06-29T15:57:22.294805+0000","last_became_peered":"2024-06-29T15:57:22.294805+0000","last_unstale":"2024-06-29T15:57:33.358900+0000","last_undegraded":"2024-06-29T15:57:33.358900+0000","last_fullsized":"2024-06-29T15:57:33.358900+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.369924+0000","last_change":"2024-06-29T15:57:33.369487+0000","last_active":"2024-06-29T15:57:33.369924+0000","last_peered":"2024-06-29T15:57:33.369924+0000","last_clean":"2024-06-29T15:57:33.369924+0000","last_became_active":"2024-06-29T15:57:33.369138+0000","last_became_peered":"2024-06-29T15:57:33.369138+0000","last_unstale":"2024-06-29T15:57:33.369924+0000","last_undegraded":"2024-06-29T15:57:33.369924+0000","last_fullsized":"2024-06-29T15:57:33.369924+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:06.322225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354530+0000","last_change":"2024-06-29T15:57:33.354530+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354530+0000","last_undegraded":"2024-06-29T15:57:33.354530+0000","last_fullsized":"2024-06-29T15:57:33.354530+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356787+0000","last_change":"2024-06-29T15:57:21.274551+0000","last_active":"2024-06-29T15:57:33.356787+0000","last_peered":"2024-06-29T15:57:33.356787+0000","last_clean":"2024-06-29T15:57:33.356787+0000","last_became_active":"2024-06-29T15:57:21.274029+0000","last_became_peered":"2024-06-29T15:57:21.274029+0000","last_unstale":"2024-06-29T15:57:33.356787+0000","last_undegraded":"2024-06-29T15:57:33.356787+0000","last_fullsized":"2024-06-29T15:57:33.356787+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364941+0000","last_change":"2024-06-29T15:57:22.291143+0000","last_active":"2024-06-29T15:57:33.364941+0000","last_peered":"2024-06-29T15:57:33.364941+0000","last_clean":"2024-06-29T15:57:33.364941+0000","last_became_active":"2024-06-29T15:57:22.290849+0000","last_became_peered":"2024-06-29T15:57:22.290849+0000","last_unstale":"2024-06-29T15:57:33.364941+0000","last_undegraded":"2024-06-29T15:57:33.364941+0000","last_fullsized":"2024-06-29T15:57:33.364941+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367095+0000","last_change":"2024-06-29T15:57:33.366161+0000","last_active":"2024-06-29T15:57:33.367095+0000","last_peered":"2024-06-29T15:57:33.367095+0000","last_clean":"2024-06-29T15:57:33.367095+0000","last_became_active":"2024-06-29T15:57:33.365820+0000","last_became_peered":"2024-06-29T15:57:33.365820+0000","last_unstale":"2024-06-29T15:57:33.367095+0000","last_undegraded":"2024-06-29T15:57:33.367095+0000","last_fullsized":"2024-06-29T15:57:33.367095+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:05.543032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358295+0000","last_change":"2024-06-29T15:57:33.358295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358295+0000","last_undegraded":"2024-06-29T15:57:33.358295+0000","last_fullsized":"2024-06-29T15:57:33.358295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361368+0000","last_change":"2024-06-29T15:57:21.281405+0000","last_active":"2024-06-29T15:57:33.361368+0000","last_peered":"2024-06-29T15:57:33.361368+0000","last_clean":"2024-06-29T15:57:33.361368+0000","last_became_active":"2024-06-29T15:57:21.281080+0000","last_became_peered":"2024-06-29T15:57:21.281080+0000","last_unstale":"2024-06-29T15:57:33.361368+0000","last_undegraded":"2024-06-29T15:57:33.361368+0000","last_fullsized":"2024-06-29T15:57:33.361368+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:51.116695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354452+0000","last_change":"2024-06-29T15:57:22.304057+0000","last_active":"2024-06-29T15:57:33.354452+0000","last_peered":"2024-06-29T15:57:33.354452+0000","last_clean":"2024-06-29T15:57:33.354452+0000","last_became_active":"2024-06-29T15:57:22.303762+0000","last_became_peered":"2024-06-29T15:57:22.303762+0000","last_unstale":"2024-06-29T15:57:33.354452+0000","last_undegraded":"2024-06-29T15:57:33.354452+0000","last_fullsized":"2024-06-29T15:57:33.354452+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:40.125779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370833+0000","last_change":"2024-06-29T15:57:33.369260+0000","last_active":"2024-06-29T15:57:33.370833+0000","last_peered":"2024-06-29T15:57:33.370833+0000","last_clean":"2024-06-29T15:57:33.370833+0000","last_became_active":"2024-06-29T15:57:33.368945+0000","last_became_peered":"2024-06-29T15:57:33.368945+0000","last_unstale":"2024-06-29T15:57:33.370833+0000","last_undegraded":"2024-06-29T15:57:33.370833+0000","last_fullsized":"2024-06-29T15:57:33.370833+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:11.821898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360097+0000","last_change":"2024-06-29T15:57:33.360097+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360097+0000","last_undegraded":"2024-06-29T15:57:33.360097+0000","last_fullsized":"2024-06-29T15:57:33.360097+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357223+0000","last_change":"2024-06-29T15:57:21.277560+0000","last_active":"2024-06-29T15:57:33.357223+0000","last_peered":"2024-06-29T15:57:33.357223+0000","last_clean":"2024-06-29T15:57:33.357223+0000","last_became_active":"2024-06-29T15:57:21.277255+0000","last_became_peered":"2024-06-29T15:57:21.277255+0000","last_unstale":"2024-06-29T15:57:33.357223+0000","last_undegraded":"2024-06-29T15:57:33.357223+0000","last_fullsized":"2024-06-29T15:57:33.357223+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354119+0000","last_change":"2024-06-29T15:57:22.301166+0000","last_active":"2024-06-29T15:57:33.354119+0000","last_peered":"2024-06-29T15:57:33.354119+0000","last_clean":"2024-06-29T15:57:33.354119+0000","last_became_active":"2024-06-29T15:57:22.300676+0000","last_became_peered":"2024-06-29T15:57:22.300676+0000","last_unstale":"2024-06-29T15:57:33.354119+0000","last_undegraded":"2024-06-29T15:57:33.354119+0000","last_fullsized":"2024-06-29T15:57:33.354119+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363490+0000","last_change":"2024-06-29T15:57:33.362184+0000","last_active":"2024-06-29T15:57:33.363490+0000","last_peered":"2024-06-29T15:57:33.363490+0000","last_clean":"2024-06-29T15:57:33.363490+0000","last_became_active":"2024-06-29T15:57:33.361905+0000","last_became_peered":"2024-06-29T15:57:33.361905+0000","last_unstale":"2024-06-29T15:57:33.363490+0000","last_undegraded":"2024-06-29T15:57:33.363490+0000","last_fullsized":"2024-06-29T15:57:33.363490+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:04.833173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349702+0000","last_change":"2024-06-29T15:57:33.349702+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349702+0000","last_undegraded":"2024-06-29T15:57:33.349702+0000","last_fullsized":"2024-06-29T15:57:33.349702+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357105+0000","last_change":"2024-06-29T15:57:21.280224+0000","last_active":"2024-06-29T15:57:33.357105+0000","last_peered":"2024-06-29T15:57:33.357105+0000","last_clean":"2024-06-29T15:57:33.357105+0000","last_became_active":"2024-06-29T15:57:21.279899+0000","last_became_peered":"2024-06-29T15:57:21.279899+0000","last_unstale":"2024-06-29T15:57:33.357105+0000","last_undegraded":"2024-06-29T15:57:33.357105+0000","last_fullsized":"2024-06-29T15:57:33.357105+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355301+0000","last_change":"2024-06-29T15:57:22.296472+0000","last_active":"2024-06-29T15:57:33.355301+0000","last_peered":"2024-06-29T15:57:33.355301+0000","last_clean":"2024-06-29T15:57:33.355301+0000","last_became_active":"2024-06-29T15:57:22.296130+0000","last_became_peered":"2024-06-29T15:57:22.296130+0000","last_unstale":"2024-06-29T15:57:33.355301+0000","last_undegraded":"2024-06-29T15:57:33.355301+0000","last_fullsized":"2024-06-29T15:57:33.355301+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368843+0000","last_change":"2024-06-29T15:57:33.367380+0000","last_active":"2024-06-29T15:57:33.368843+0000","last_peered":"2024-06-29T15:57:33.368843+0000","last_clean":"2024-06-29T15:57:33.368843+0000","last_became_active":"2024-06-29T15:57:33.367054+0000","last_became_peered":"2024-06-29T15:57:33.367054+0000","last_unstale":"2024-06-29T15:57:33.368843+0000","last_undegraded":"2024-06-29T15:57:33.368843+0000","last_fullsized":"2024-06-29T15:57:33.368843+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:29.345255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357030+0000","last_change":"2024-06-29T15:57:33.357030+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357030+0000","last_undegraded":"2024-06-29T15:57:33.357030+0000","last_fullsized":"2024-06-29T15:57:33.357030+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362303+0000","last_change":"2024-06-29T15:57:21.276972+0000","last_active":"2024-06-29T15:57:33.362303+0000","last_peered":"2024-06-29T15:57:33.362303+0000","last_clean":"2024-06-29T15:57:33.362303+0000","last_became_active":"2024-06-29T15:57:21.276647+0000","last_became_peered":"2024-06-29T15:57:21.276647+0000","last_unstale":"2024-06-29T15:57:33.362303+0000","last_undegraded":"2024-06-29T15:57:33.362303+0000","last_fullsized":"2024-06-29T15:57:33.362303+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:24.871281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355716+0000","last_change":"2024-06-29T15:57:22.303802+0000","last_active":"2024-06-29T15:57:33.355716+0000","last_peered":"2024-06-29T15:57:33.355716+0000","last_clean":"2024-06-29T15:57:33.355716+0000","last_became_active":"2024-06-29T15:57:22.303511+0000","last_became_peered":"2024-06-29T15:57:22.303511+0000","last_unstale":"2024-06-29T15:57:33.355716+0000","last_undegraded":"2024-06-29T15:57:33.355716+0000","last_fullsized":"2024-06-29T15:57:33.355716+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:10.536799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371208+0000","last_change":"2024-06-29T15:57:33.367162+0000","last_active":"2024-06-29T15:57:33.371208+0000","last_peered":"2024-06-29T15:57:33.371208+0000","last_clean":"2024-06-29T15:57:33.371208+0000","last_became_active":"2024-06-29T15:57:33.366816+0000","last_became_peered":"2024-06-29T15:57:33.366816+0000","last_unstale":"2024-06-29T15:57:33.371208+0000","last_undegraded":"2024-06-29T15:57:33.371208+0000","last_fullsized":"2024-06-29T15:57:33.371208+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:41.683812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.363662+0000","last_change":"2024-06-29T15:57:33.363662+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.363662+0000","last_undegraded":"2024-06-29T15:57:33.363662+0000","last_fullsized":"2024-06-29T15:57:33.363662+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358022+0000","last_change":"2024-06-29T15:57:21.277662+0000","last_active":"2024-06-29T15:57:33.358022+0000","last_peered":"2024-06-29T15:57:33.358022+0000","last_clean":"2024-06-29T15:57:33.358022+0000","last_became_active":"2024-06-29T15:57:21.277335+0000","last_became_peered":"2024-06-29T15:57:21.277335+0000","last_unstale":"2024-06-29T15:57:33.358022+0000","last_undegraded":"2024-06-29T15:57:33.358022+0000","last_fullsized":"2024-06-29T15:57:33.358022+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364773+0000","last_change":"2024-06-29T15:57:22.298914+0000","last_active":"2024-06-29T15:57:33.364773+0000","last_peered":"2024-06-29T15:57:33.364773+0000","last_clean":"2024-06-29T15:57:33.364773+0000","last_became_active":"2024-06-29T15:57:22.298614+0000","last_became_peered":"2024-06-29T15:57:22.298614+0000","last_unstale":"2024-06-29T15:57:33.364773+0000","last_undegraded":"2024-06-29T15:57:33.364773+0000","last_fullsized":"2024-06-29T15:57:33.364773+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367924+0000","last_change":"2024-06-29T15:57:33.367133+0000","last_active":"2024-06-29T15:57:33.367924+0000","last_peered":"2024-06-29T15:57:33.367924+0000","last_clean":"2024-06-29T15:57:33.367924+0000","last_became_active":"2024-06-29T15:57:33.366782+0000","last_became_peered":"2024-06-29T15:57:33.366782+0000","last_unstale":"2024-06-29T15:57:33.367924+0000","last_undegraded":"2024-06-29T15:57:33.367924+0000","last_fullsized":"2024-06-29T15:57:33.367924+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351295+0000","last_change":"2024-06-29T15:57:33.351295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351295+0000","last_undegraded":"2024-06-29T15:57:33.351295+0000","last_fullsized":"2024-06-29T15:57:33.351295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355165+0000","last_change":"2024-06-29T15:57:21.278011+0000","last_active":"2024-06-29T15:57:33.355165+0000","last_peered":"2024-06-29T15:57:33.355165+0000","last_clean":"2024-06-29T15:57:33.355165+0000","last_became_active":"2024-06-29T15:57:21.276648+0000","last_became_peered":"2024-06-29T15:57:21.276648+0000","last_unstale":"2024-06-29T15:57:33.355165+0000","last_undegraded":"2024-06-29T15:57:33.355165+0000","last_fullsized":"2024-06-29T15:57:33.355165+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354181+0000","last_change":"2024-06-29T15:57:22.300220+0000","last_active":"2024-06-29T15:57:33.354181+0000","last_peered":"2024-06-29T15:57:33.354181+0000","last_clean":"2024-06-29T15:57:33.354181+0000","last_became_active":"2024-06-29T15:57:22.299880+0000","last_became_peered":"2024-06-29T15:57:22.299880+0000","last_unstale":"2024-06-29T15:57:33.354181+0000","last_undegraded":"2024-06-29T15:57:33.354181+0000","last_fullsized":"2024-06-29T15:57:33.354181+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368552+0000","last_change":"2024-06-29T15:57:33.367636+0000","last_active":"2024-06-29T15:57:33.368552+0000","last_peered":"2024-06-29T15:57:33.368552+0000","last_clean":"2024-06-29T15:57:33.368552+0000","last_became_active":"2024-06-29T15:57:33.367321+0000","last_became_peered":"2024-06-29T15:57:33.367321+0000","last_unstale":"2024-06-29T15:57:33.368552+0000","last_undegraded":"2024-06-29T15:57:33.368552+0000","last_fullsized":"2024-06-29T15:57:33.368552+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:28.846172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353695+0000","last_change":"2024-06-29T15:57:33.353695+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353695+0000","last_undegraded":"2024-06-29T15:57:33.353695+0000","last_fullsized":"2024-06-29T15:57:33.353695+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357156+0000","last_change":"2024-06-29T15:57:21.284982+0000","last_active":"2024-06-29T15:57:33.357156+0000","last_peered":"2024-06-29T15:57:33.357156+0000","last_clean":"2024-06-29T15:57:33.357156+0000","last_became_active":"2024-06-29T15:57:21.284684+0000","last_became_peered":"2024-06-29T15:57:21.284684+0000","last_unstale":"2024-06-29T15:57:33.357156+0000","last_undegraded":"2024-06-29T15:57:33.357156+0000","last_fullsized":"2024-06-29T15:57:33.357156+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354906+0000","last_change":"2024-06-29T15:57:22.298744+0000","last_active":"2024-06-29T15:57:33.354906+0000","last_peered":"2024-06-29T15:57:33.354906+0000","last_clean":"2024-06-29T15:57:33.354906+0000","last_became_active":"2024-06-29T15:57:22.296925+0000","last_became_peered":"2024-06-29T15:57:22.296925+0000","last_unstale":"2024-06-29T15:57:33.354906+0000","last_undegraded":"2024-06-29T15:57:33.354906+0000","last_fullsized":"2024-06-29T15:57:33.354906+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:40.578182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371721+0000","last_change":"2024-06-29T15:57:33.368936+0000","last_active":"2024-06-29T15:57:33.371721+0000","last_peered":"2024-06-29T15:57:33.371721+0000","last_clean":"2024-06-29T15:57:33.371721+0000","last_became_active":"2024-06-29T15:57:33.368600+0000","last_became_peered":"2024-06-29T15:57:33.368600+0000","last_unstale":"2024-06-29T15:57:33.371721+0000","last_undegraded":"2024-06-29T15:57:33.371721+0000","last_fullsized":"2024-06-29T15:57:33.371721+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:22.830532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.361651+0000","last_change":"2024-06-29T15:57:33.361651+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.361651+0000","last_undegraded":"2024-06-29T15:57:33.361651+0000","last_fullsized":"2024-06-29T15:57:33.361651+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355165+0000","last_change":"2024-06-29T15:57:21.284780+0000","last_active":"2024-06-29T15:57:33.355165+0000","last_peered":"2024-06-29T15:57:33.355165+0000","last_clean":"2024-06-29T15:57:33.355165+0000","last_became_active":"2024-06-29T15:57:21.284273+0000","last_became_peered":"2024-06-29T15:57:21.284273+0000","last_unstale":"2024-06-29T15:57:33.355165+0000","last_undegraded":"2024-06-29T15:57:33.355165+0000","last_fullsized":"2024-06-29T15:57:33.355165+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:16.085990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356696+0000","last_change":"2024-06-29T15:57:22.291304+0000","last_active":"2024-06-29T15:57:33.356696+0000","last_peered":"2024-06-29T15:57:33.356696+0000","last_clean":"2024-06-29T15:57:33.356696+0000","last_became_active":"2024-06-29T15:57:22.290983+0000","last_became_peered":"2024-06-29T15:57:22.290983+0000","last_unstale":"2024-06-29T15:57:33.356696+0000","last_undegraded":"2024-06-29T15:57:33.356696+0000","last_fullsized":"2024-06-29T15:57:33.356696+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368254+0000","last_change":"2024-06-29T15:57:33.366553+0000","last_active":"2024-06-29T15:57:33.368254+0000","last_peered":"2024-06-29T15:57:33.368254+0000","last_clean":"2024-06-29T15:57:33.368254+0000","last_became_active":"2024-06-29T15:57:33.366230+0000","last_became_peered":"2024-06-29T15:57:33.366230+0000","last_unstale":"2024-06-29T15:57:33.368254+0000","last_undegraded":"2024-06-29T15:57:33.368254+0000","last_fullsized":"2024-06-29T15:57:33.368254+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:17:11.929406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358755+0000","last_change":"2024-06-29T15:57:33.358755+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358755+0000","last_undegraded":"2024-06-29T15:57:33.358755+0000","last_fullsized":"2024-06-29T15:57:33.358755+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348756+0000","last_change":"2024-06-29T15:57:21.284009+0000","last_active":"2024-06-29T15:57:33.348756+0000","last_peered":"2024-06-29T15:57:33.348756+0000","last_clean":"2024-06-29T15:57:33.348756+0000","last_became_active":"2024-06-29T15:57:21.283679+0000","last_became_peered":"2024-06-29T15:57:21.283679+0000","last_unstale":"2024-06-29T15:57:33.348756+0000","last_undegraded":"2024-06-29T15:57:33.348756+0000","last_fullsized":"2024-06-29T15:57:33.348756+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:20.284468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348277+0000","last_change":"2024-06-29T15:57:22.288958+0000","last_active":"2024-06-29T15:57:33.348277+0000","last_peered":"2024-06-29T15:57:33.348277+0000","last_clean":"2024-06-29T15:57:33.348277+0000","last_became_active":"2024-06-29T15:57:22.288621+0000","last_became_peered":"2024-06-29T15:57:22.288621+0000","last_unstale":"2024-06-29T15:57:33.348277+0000","last_undegraded":"2024-06-29T15:57:33.348277+0000","last_fullsized":"2024-06-29T15:57:33.348277+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:31.720751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371349+0000","last_change":"2024-06-29T15:57:33.370560+0000","last_active":"2024-06-29T15:57:33.371349+0000","last_peered":"2024-06-29T15:57:33.371349+0000","last_clean":"2024-06-29T15:57:33.371349+0000","last_became_active":"2024-06-29T15:57:33.370206+0000","last_became_peered":"2024-06-29T15:57:33.370206+0000","last_unstale":"2024-06-29T15:57:33.371349+0000","last_undegraded":"2024-06-29T15:57:33.371349+0000","last_fullsized":"2024-06-29T15:57:33.371349+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:12:47.692442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.361054+0000","last_change":"2024-06-29T15:57:33.361054+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.361054+0000","last_undegraded":"2024-06-29T15:57:33.361054+0000","last_fullsized":"2024-06-29T15:57:33.361054+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357091+0000","last_change":"2024-06-29T15:57:21.282111+0000","last_active":"2024-06-29T15:57:33.357091+0000","last_peered":"2024-06-29T15:57:33.357091+0000","last_clean":"2024-06-29T15:57:33.357091+0000","last_became_active":"2024-06-29T15:57:21.281761+0000","last_became_peered":"2024-06-29T15:57:21.281761+0000","last_unstale":"2024-06-29T15:57:33.357091+0000","last_undegraded":"2024-06-29T15:57:33.357091+0000","last_fullsized":"2024-06-29T15:57:33.357091+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348792+0000","last_change":"2024-06-29T15:57:22.305806+0000","last_active":"2024-06-29T15:57:33.348792+0000","last_peered":"2024-06-29T15:57:33.348792+0000","last_clean":"2024-06-29T15:57:33.348792+0000","last_became_active":"2024-06-29T15:57:22.296276+0000","last_became_peered":"2024-06-29T15:57:22.296276+0000","last_unstale":"2024-06-29T15:57:33.348792+0000","last_undegraded":"2024-06-29T15:57:33.348792+0000","last_fullsized":"2024-06-29T15:57:33.348792+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:43:51.754223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.369952+0000","last_change":"2024-06-29T15:57:33.367821+0000","last_active":"2024-06-29T15:57:33.369952+0000","last_peered":"2024-06-29T15:57:33.369952+0000","last_clean":"2024-06-29T15:57:33.369952+0000","last_became_active":"2024-06-29T15:57:33.367494+0000","last_became_peered":"2024-06-29T15:57:33.367494+0000","last_unstale":"2024-06-29T15:57:33.369952+0000","last_undegraded":"2024-06-29T15:57:33.369952+0000","last_fullsized":"2024-06-29T15:57:33.369952+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:46.165772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352248+0000","last_change":"2024-06-29T15:57:33.352248+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352248+0000","last_undegraded":"2024-06-29T15:57:33.352248+0000","last_fullsized":"2024-06-29T15:57:33.352248+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.349023+0000","last_change":"2024-06-29T15:57:21.279139+0000","last_active":"2024-06-29T15:57:33.349023+0000","last_peered":"2024-06-29T15:57:33.349023+0000","last_clean":"2024-06-29T15:57:33.349023+0000","last_became_active":"2024-06-29T15:57:21.278488+0000","last_became_peered":"2024-06-29T15:57:21.278488+0000","last_unstale":"2024-06-29T15:57:33.349023+0000","last_undegraded":"2024-06-29T15:57:33.349023+0000","last_fullsized":"2024-06-29T15:57:33.349023+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:54.126338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356194+0000","last_change":"2024-06-29T15:57:22.297275+0000","last_active":"2024-06-29T15:57:33.356194+0000","last_peered":"2024-06-29T15:57:33.356194+0000","last_clean":"2024-06-29T15:57:33.356194+0000","last_became_active":"2024-06-29T15:57:22.296984+0000","last_became_peered":"2024-06-29T15:57:22.296984+0000","last_unstale":"2024-06-29T15:57:33.356194+0000","last_undegraded":"2024-06-29T15:57:33.356194+0000","last_fullsized":"2024-06-29T15:57:33.356194+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371810+0000","last_change":"2024-06-29T15:57:33.369358+0000","last_active":"2024-06-29T15:57:33.371810+0000","last_peered":"2024-06-29T15:57:33.371810+0000","last_clean":"2024-06-29T15:57:33.371810+0000","last_became_active":"2024-06-29T15:57:33.369052+0000","last_became_peered":"2024-06-29T15:57:33.369052+0000","last_unstale":"2024-06-29T15:57:33.371810+0000","last_undegraded":"2024-06-29T15:57:33.371810+0000","last_fullsized":"2024-06-29T15:57:33.371810+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:29.345255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352493+0000","last_change":"2024-06-29T15:57:33.352493+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352493+0000","last_undegraded":"2024-06-29T15:57:33.352493+0000","last_fullsized":"2024-06-29T15:57:33.352493+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356473+0000","last_change":"2024-06-29T15:57:21.278476+0000","last_active":"2024-06-29T15:57:33.356473+0000","last_peered":"2024-06-29T15:57:33.356473+0000","last_clean":"2024-06-29T15:57:33.356473+0000","last_became_active":"2024-06-29T15:57:21.278150+0000","last_became_peered":"2024-06-29T15:57:21.278150+0000","last_unstale":"2024-06-29T15:57:33.356473+0000","last_undegraded":"2024-06-29T15:57:33.356473+0000","last_fullsized":"2024-06-29T15:57:33.356473+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:33.991332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355923+0000","last_change":"2024-06-29T15:57:22.291360+0000","last_active":"2024-06-29T15:57:33.355923+0000","last_peered":"2024-06-29T15:57:33.355923+0000","last_clean":"2024-06-29T15:57:33.355923+0000","last_became_active":"2024-06-29T15:57:22.290801+0000","last_became_peered":"2024-06-29T15:57:22.290801+0000","last_unstale":"2024-06-29T15:57:33.355923+0000","last_undegraded":"2024-06-29T15:57:33.355923+0000","last_fullsized":"2024-06-29T15:57:33.355923+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371558+0000","last_change":"2024-06-29T15:57:33.367933+0000","last_active":"2024-06-29T15:57:33.371558+0000","last_peered":"2024-06-29T15:57:33.371558+0000","last_clean":"2024-06-29T15:57:33.371558+0000","last_became_active":"2024-06-29T15:57:33.367630+0000","last_became_peered":"2024-06-29T15:57:33.367630+0000","last_unstale":"2024-06-29T15:57:33.371558+0000","last_undegraded":"2024-06-29T15:57:33.371558+0000","last_fullsized":"2024-06-29T15:57:33.371558+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:01.393531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358484+0000","last_change":"2024-06-29T15:57:33.358484+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358484+0000","last_undegraded":"2024-06-29T15:57:33.358484+0000","last_fullsized":"2024-06-29T15:57:33.358484+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358778+0000","last_change":"2024-06-29T15:57:21.273304+0000","last_active":"2024-06-29T15:57:33.358778+0000","last_peered":"2024-06-29T15:57:33.358778+0000","last_clean":"2024-06-29T15:57:33.358778+0000","last_became_active":"2024-06-29T15:57:21.272972+0000","last_became_peered":"2024-06-29T15:57:21.272972+0000","last_unstale":"2024-06-29T15:57:33.358778+0000","last_undegraded":"2024-06-29T15:57:33.358778+0000","last_fullsized":"2024-06-29T15:57:33.358778+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356200+0000","last_change":"2024-06-29T15:57:22.300916+0000","last_active":"2024-06-29T15:57:33.356200+0000","last_peered":"2024-06-29T15:57:33.356200+0000","last_clean":"2024-06-29T15:57:33.356200+0000","last_became_active":"2024-06-29T15:57:22.300620+0000","last_became_peered":"2024-06-29T15:57:22.300620+0000","last_unstale":"2024-06-29T15:57:33.356200+0000","last_undegraded":"2024-06-29T15:57:33.356200+0000","last_fullsized":"2024-06-29T15:57:33.356200+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:41.172455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368260+0000","last_change":"2024-06-29T15:57:33.367013+0000","last_active":"2024-06-29T15:57:33.368260+0000","last_peered":"2024-06-29T15:57:33.368260+0000","last_clean":"2024-06-29T15:57:33.368260+0000","last_became_active":"2024-06-29T15:57:33.366664+0000","last_became_peered":"2024-06-29T15:57:33.366664+0000","last_unstale":"2024-06-29T15:57:33.368260+0000","last_undegraded":"2024-06-29T15:57:33.368260+0000","last_fullsized":"2024-06-29T15:57:33.368260+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:22.830532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357163+0000","last_change":"2024-06-29T15:57:33.357163+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357163+0000","last_undegraded":"2024-06-29T15:57:33.357163+0000","last_fullsized":"2024-06-29T15:57:33.357163+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"212'8","reported_seq":26,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359054+0000","last_change":"2024-06-29T15:57:21.283745+0000","last_active":"2024-06-29T15:57:33.359054+0000","last_peered":"2024-06-29T15:57:33.359054+0000","last_clean":"2024-06-29T15:57:33.359054+0000","last_became_active":"2024-06-29T15:57:21.283403+0000","last_became_peered":"2024-06-29T15:57:21.283403+0000","last_unstale":"2024-06-29T15:57:33.359054+0000","last_undegraded":"2024-06-29T15:57:33.359054+0000","last_fullsized":"2024-06-29T15:57:33.359054+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357048+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:33.357048+0000","last_peered":"2024-06-29T15:57:33.357048+0000","last_clean":"2024-06-29T15:57:33.357048+0000","last_became_active":"2024-06-29T15:57:22.301832+0000","last_became_peered":"2024-06-29T15:57:22.301832+0000","last_unstale":"2024-06-29T15:57:33.357048+0000","last_undegraded":"2024-06-29T15:57:33.357048+0000","last_fullsized":"2024-06-29T15:57:33.357048+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367778+0000","last_change":"2024-06-29T15:57:33.366032+0000","last_active":"2024-06-29T15:57:33.367778+0000","last_peered":"2024-06-29T15:57:33.367778+0000","last_clean":"2024-06-29T15:57:33.367778+0000","last_became_active":"2024-06-29T15:57:33.365731+0000","last_became_peered":"2024-06-29T15:57:33.365731+0000","last_unstale":"2024-06-29T15:57:33.367778+0000","last_undegraded":"2024-06-29T15:57:33.367778+0000","last_fullsized":"2024-06-29T15:57:33.367778+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:58.462755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349766+0000","last_change":"2024-06-29T15:57:33.349766+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349766+0000","last_undegraded":"2024-06-29T15:57:33.349766+0000","last_fullsized":"2024-06-29T15:57:33.349766+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348560+0000","last_change":"2024-06-29T15:57:21.276731+0000","last_active":"2024-06-29T15:57:33.348560+0000","last_peered":"2024-06-29T15:57:33.348560+0000","last_clean":"2024-06-29T15:57:33.348560+0000","last_became_active":"2024-06-29T15:57:21.276408+0000","last_became_peered":"2024-06-29T15:57:21.276408+0000","last_unstale":"2024-06-29T15:57:33.348560+0000","last_undegraded":"2024-06-29T15:57:33.348560+0000","last_fullsized":"2024-06-29T15:57:33.348560+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:48.911239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358795+0000","last_change":"2024-06-29T15:57:22.299631+0000","last_active":"2024-06-29T15:57:33.358795+0000","last_peered":"2024-06-29T15:57:33.358795+0000","last_clean":"2024-06-29T15:57:33.358795+0000","last_became_active":"2024-06-29T15:57:22.299285+0000","last_became_peered":"2024-06-29T15:57:22.299285+0000","last_unstale":"2024-06-29T15:57:33.358795+0000","last_undegraded":"2024-06-29T15:57:33.358795+0000","last_fullsized":"2024-06-29T15:57:33.358795+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370910+0000","last_change":"2024-06-29T15:57:33.369952+0000","last_active":"2024-06-29T15:57:33.370910+0000","last_peered":"2024-06-29T15:57:33.370910+0000","last_clean":"2024-06-29T15:57:33.370910+0000","last_became_active":"2024-06-29T15:57:33.369598+0000","last_became_peered":"2024-06-29T15:57:33.369598+0000","last_unstale":"2024-06-29T15:57:33.370910+0000","last_undegraded":"2024-06-29T15:57:33.370910+0000","last_fullsized":"2024-06-29T15:57:33.370910+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:24.974909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358887+0000","last_change":"2024-06-29T15:57:33.358887+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358887+0000","last_undegraded":"2024-06-29T15:57:33.358887+0000","last_fullsized":"2024-06-29T15:57:33.358887+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354289+0000","last_change":"2024-06-29T15:57:21.282872+0000","last_active":"2024-06-29T15:57:33.354289+0000","last_peered":"2024-06-29T15:57:33.354289+0000","last_clean":"2024-06-29T15:57:33.354289+0000","last_became_active":"2024-06-29T15:57:21.282549+0000","last_became_peered":"2024-06-29T15:57:21.282549+0000","last_unstale":"2024-06-29T15:57:33.354289+0000","last_undegraded":"2024-06-29T15:57:33.354289+0000","last_fullsized":"2024-06-29T15:57:33.354289+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348643+0000","last_change":"2024-06-29T15:57:22.294011+0000","last_active":"2024-06-29T15:57:33.348643+0000","last_peered":"2024-06-29T15:57:33.348643+0000","last_clean":"2024-06-29T15:57:33.348643+0000","last_became_active":"2024-06-29T15:57:22.293721+0000","last_became_peered":"2024-06-29T15:57:22.293721+0000","last_unstale":"2024-06-29T15:57:33.348643+0000","last_undegraded":"2024-06-29T15:57:33.348643+0000","last_fullsized":"2024-06-29T15:57:33.348643+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:57.337708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362703+0000","last_change":"2024-06-29T15:57:33.361883+0000","last_active":"2024-06-29T15:57:33.362703+0000","last_peered":"2024-06-29T15:57:33.362703+0000","last_clean":"2024-06-29T15:57:33.362703+0000","last_became_active":"2024-06-29T15:57:33.361569+0000","last_became_peered":"2024-06-29T15:57:33.361569+0000","last_unstale":"2024-06-29T15:57:33.362703+0000","last_undegraded":"2024-06-29T15:57:33.362703+0000","last_fullsized":"2024-06-29T15:57:33.362703+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:08.554869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351919+0000","last_change":"2024-06-29T15:57:33.351919+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351919+0000","last_undegraded":"2024-06-29T15:57:33.351919+0000","last_fullsized":"2024-06-29T15:57:33.351919+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353877+0000","last_change":"2024-06-29T15:57:21.281309+0000","last_active":"2024-06-29T15:57:33.353877+0000","last_peered":"2024-06-29T15:57:33.353877+0000","last_clean":"2024-06-29T15:57:33.353877+0000","last_became_active":"2024-06-29T15:57:21.280944+0000","last_became_peered":"2024-06-29T15:57:21.280944+0000","last_unstale":"2024-06-29T15:57:33.353877+0000","last_undegraded":"2024-06-29T15:57:33.353877+0000","last_fullsized":"2024-06-29T15:57:33.353877+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356544+0000","last_change":"2024-06-29T15:57:22.291164+0000","last_active":"2024-06-29T15:57:33.356544+0000","last_peered":"2024-06-29T15:57:33.356544+0000","last_clean":"2024-06-29T15:57:33.356544+0000","last_became_active":"2024-06-29T15:57:22.290897+0000","last_became_peered":"2024-06-29T15:57:22.290897+0000","last_unstale":"2024-06-29T15:57:33.356544+0000","last_undegraded":"2024-06-29T15:57:33.356544+0000","last_fullsized":"2024-06-29T15:57:33.356544+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364704+0000","last_change":"2024-06-29T15:57:33.364026+0000","last_active":"2024-06-29T15:57:33.364704+0000","last_peered":"2024-06-29T15:57:33.364704+0000","last_clean":"2024-06-29T15:57:33.364704+0000","last_became_active":"2024-06-29T15:57:33.363658+0000","last_became_peered":"2024-06-29T15:57:33.363658+0000","last_unstale":"2024-06-29T15:57:33.364704+0000","last_undegraded":"2024-06-29T15:57:33.364704+0000","last_fullsized":"2024-06-29T15:57:33.364704+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:41.322192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359617+0000","last_change":"2024-06-29T15:57:33.359617+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359617+0000","last_undegraded":"2024-06-29T15:57:33.359617+0000","last_fullsized":"2024-06-29T15:57:33.359617+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"214'8","reported_seq":26,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365217+0000","last_change":"2024-06-29T15:57:21.278161+0000","last_active":"2024-06-29T15:57:33.365217+0000","last_peered":"2024-06-29T15:57:33.365217+0000","last_clean":"2024-06-29T15:57:33.365217+0000","last_became_active":"2024-06-29T15:57:21.277794+0000","last_became_peered":"2024-06-29T15:57:21.277794+0000","last_unstale":"2024-06-29T15:57:33.365217+0000","last_undegraded":"2024-06-29T15:57:33.365217+0000","last_fullsized":"2024-06-29T15:57:33.365217+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:56.330168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.352292+0000","last_change":"2024-06-29T15:57:22.302666+0000","last_active":"2024-06-29T15:57:33.352292+0000","last_peered":"2024-06-29T15:57:33.352292+0000","last_clean":"2024-06-29T15:57:33.352292+0000","last_became_active":"2024-06-29T15:57:22.302254+0000","last_became_peered":"2024-06-29T15:57:22.302254+0000","last_unstale":"2024-06-29T15:57:33.352292+0000","last_undegraded":"2024-06-29T15:57:33.352292+0000","last_fullsized":"2024-06-29T15:57:33.352292+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363467+0000","last_change":"2024-06-29T15:57:33.361374+0000","last_active":"2024-06-29T15:57:33.363467+0000","last_peered":"2024-06-29T15:57:33.363467+0000","last_clean":"2024-06-29T15:57:33.363467+0000","last_became_active":"2024-06-29T15:57:33.361002+0000","last_became_peered":"2024-06-29T15:57:33.361002+0000","last_unstale":"2024-06-29T15:57:33.363467+0000","last_undegraded":"2024-06-29T15:57:33.363467+0000","last_fullsized":"2024-06-29T15:57:33.363467+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:47.234190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353505+0000","last_change":"2024-06-29T15:57:33.353505+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353505+0000","last_undegraded":"2024-06-29T15:57:33.353505+0000","last_fullsized":"2024-06-29T15:57:33.353505+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"212'3","reported_seq":21,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348900+0000","last_change":"2024-06-29T15:57:21.283791+0000","last_active":"2024-06-29T15:57:33.348900+0000","last_peered":"2024-06-29T15:57:33.348900+0000","last_clean":"2024-06-29T15:57:33.348900+0000","last_became_active":"2024-06-29T15:57:21.283458+0000","last_became_peered":"2024-06-29T15:57:21.283458+0000","last_unstale":"2024-06-29T15:57:33.348900+0000","last_undegraded":"2024-06-29T15:57:33.348900+0000","last_fullsized":"2024-06-29T15:57:33.348900+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:54.365881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361117+0000","last_change":"2024-06-29T15:57:22.295260+0000","last_active":"2024-06-29T15:57:33.361117+0000","last_peered":"2024-06-29T15:57:33.361117+0000","last_clean":"2024-06-29T15:57:33.361117+0000","last_became_active":"2024-06-29T15:57:22.294952+0000","last_became_peered":"2024-06-29T15:57:22.294952+0000","last_unstale":"2024-06-29T15:57:33.361117+0000","last_undegraded":"2024-06-29T15:57:33.361117+0000","last_fullsized":"2024-06-29T15:57:33.361117+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:56.473182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370910+0000","last_change":"2024-06-29T15:57:33.370460+0000","last_active":"2024-06-29T15:57:33.370910+0000","last_peered":"2024-06-29T15:57:33.370910+0000","last_clean":"2024-06-29T15:57:33.370910+0000","last_became_active":"2024-06-29T15:57:33.370183+0000","last_became_peered":"2024-06-29T15:57:33.370183+0000","last_unstale":"2024-06-29T15:57:33.370910+0000","last_undegraded":"2024-06-29T15:57:33.370910+0000","last_fullsized":"2024-06-29T15:57:33.370910+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:06.322225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352295+0000","last_change":"2024-06-29T15:57:33.352295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352295+0000","last_undegraded":"2024-06-29T15:57:33.352295+0000","last_fullsized":"2024-06-29T15:57:33.352295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354550+0000","last_change":"2024-06-29T15:57:21.275926+0000","last_active":"2024-06-29T15:57:33.354550+0000","last_peered":"2024-06-29T15:57:33.354550+0000","last_clean":"2024-06-29T15:57:33.354550+0000","last_became_active":"2024-06-29T15:57:21.275593+0000","last_became_peered":"2024-06-29T15:57:21.275593+0000","last_unstale":"2024-06-29T15:57:33.354550+0000","last_undegraded":"2024-06-29T15:57:33.354550+0000","last_fullsized":"2024-06-29T15:57:33.354550+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357897+0000","last_change":"2024-06-29T15:57:22.298454+0000","last_active":"2024-06-29T15:57:33.357897+0000","last_peered":"2024-06-29T15:57:33.357897+0000","last_clean":"2024-06-29T15:57:33.357897+0000","last_became_active":"2024-06-29T15:57:22.298162+0000","last_became_peered":"2024-06-29T15:57:22.298162+0000","last_unstale":"2024-06-29T15:57:33.357897+0000","last_undegraded":"2024-06-29T15:57:33.357897+0000","last_fullsized":"2024-06-29T15:57:33.357897+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363344+0000","last_change":"2024-06-29T15:57:33.362039+0000","last_active":"2024-06-29T15:57:33.363344+0000","last_peered":"2024-06-29T15:57:33.363344+0000","last_clean":"2024-06-29T15:57:33.363344+0000","last_became_active":"2024-06-29T15:57:33.361665+0000","last_became_peered":"2024-06-29T15:57:33.361665+0000","last_unstale":"2024-06-29T15:57:33.363344+0000","last_undegraded":"2024-06-29T15:57:33.363344+0000","last_fullsized":"2024-06-29T15:57:33.363344+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:41.322192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358599+0000","last_change":"2024-06-29T15:57:33.358599+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358599+0000","last_undegraded":"2024-06-29T15:57:33.358599+0000","last_fullsized":"2024-06-29T15:57:33.358599+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358654+0000","last_change":"2024-06-29T15:57:21.272019+0000","last_active":"2024-06-29T15:57:33.358654+0000","last_peered":"2024-06-29T15:57:33.358654+0000","last_clean":"2024-06-29T15:57:33.358654+0000","last_became_active":"2024-06-29T15:57:21.271715+0000","last_became_peered":"2024-06-29T15:57:21.271715+0000","last_unstale":"2024-06-29T15:57:33.358654+0000","last_undegraded":"2024-06-29T15:57:33.358654+0000","last_fullsized":"2024-06-29T15:57:33.358654+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362094+0000","last_change":"2024-06-29T15:57:22.303487+0000","last_active":"2024-06-29T15:57:33.362094+0000","last_peered":"2024-06-29T15:57:33.362094+0000","last_clean":"2024-06-29T15:57:33.362094+0000","last_became_active":"2024-06-29T15:57:22.302933+0000","last_became_peered":"2024-06-29T15:57:22.302933+0000","last_unstale":"2024-06-29T15:57:33.362094+0000","last_undegraded":"2024-06-29T15:57:33.362094+0000","last_fullsized":"2024-06-29T15:57:33.362094+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:28.493187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362444+0000","last_change":"2024-06-29T15:57:33.361560+0000","last_active":"2024-06-29T15:57:33.362444+0000","last_peered":"2024-06-29T15:57:33.362444+0000","last_clean":"2024-06-29T15:57:33.362444+0000","last_became_active":"2024-06-29T15:57:33.361270+0000","last_became_peered":"2024-06-29T15:57:33.361270+0000","last_unstale":"2024-06-29T15:57:33.362444+0000","last_undegraded":"2024-06-29T15:57:33.362444+0000","last_fullsized":"2024-06-29T15:57:33.362444+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:51.198488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356450+0000","last_change":"2024-06-29T15:57:33.356450+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356450+0000","last_undegraded":"2024-06-29T15:57:33.356450+0000","last_fullsized":"2024-06-29T15:57:33.356450+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364019+0000","last_change":"2024-06-29T15:57:33.362045+0000","last_active":"2024-06-29T15:57:33.364019+0000","last_peered":"2024-06-29T15:57:33.364019+0000","last_clean":"2024-06-29T15:57:33.364019+0000","last_became_active":"2024-06-29T15:57:33.361647+0000","last_became_peered":"2024-06-29T15:57:33.361647+0000","last_unstale":"2024-06-29T15:57:33.364019+0000","last_undegraded":"2024-06-29T15:57:33.364019+0000","last_fullsized":"2024-06-29T15:57:33.364019+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:51.198488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353878+0000","last_change":"2024-06-29T15:57:21.277245+0000","last_active":"2024-06-29T15:57:33.353878+0000","last_peered":"2024-06-29T15:57:33.353878+0000","last_clean":"2024-06-29T15:57:33.353878+0000","last_became_active":"2024-06-29T15:57:21.276931+0000","last_became_peered":"2024-06-29T15:57:21.276931+0000","last_unstale":"2024-06-29T15:57:33.353878+0000","last_undegraded":"2024-06-29T15:57:33.353878+0000","last_fullsized":"2024-06-29T15:57:33.353878+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361080+0000","last_change":"2024-06-29T15:57:22.299457+0000","last_active":"2024-06-29T15:57:33.361080+0000","last_peered":"2024-06-29T15:57:33.361080+0000","last_clean":"2024-06-29T15:57:33.361080+0000","last_became_active":"2024-06-29T15:57:22.299157+0000","last_became_peered":"2024-06-29T15:57:22.299157+0000","last_unstale":"2024-06-29T15:57:33.361080+0000","last_undegraded":"2024-06-29T15:57:33.361080+0000","last_fullsized":"2024-06-29T15:57:33.361080+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:40.967292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356113+0000","last_change":"2024-06-29T15:57:33.356113+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356113+0000","last_undegraded":"2024-06-29T15:57:33.356113+0000","last_fullsized":"2024-06-29T15:57:33.356113+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363418+0000","last_change":"2024-06-29T15:57:33.360662+0000","last_active":"2024-06-29T15:57:33.363418+0000","last_peered":"2024-06-29T15:57:33.363418+0000","last_clean":"2024-06-29T15:57:33.363418+0000","last_became_active":"2024-06-29T15:57:33.360087+0000","last_became_peered":"2024-06-29T15:57:33.360087+0000","last_unstale":"2024-06-29T15:57:33.363418+0000","last_undegraded":"2024-06-29T15:57:33.363418+0000","last_fullsized":"2024-06-29T15:57:33.363418+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:46:09.697181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355322+0000","last_change":"2024-06-29T15:57:21.277884+0000","last_active":"2024-06-29T15:57:33.355322+0000","last_peered":"2024-06-29T15:57:33.355322+0000","last_clean":"2024-06-29T15:57:33.355322+0000","last_became_active":"2024-06-29T15:57:21.277589+0000","last_became_peered":"2024-06-29T15:57:21.277589+0000","last_unstale":"2024-06-29T15:57:33.355322+0000","last_undegraded":"2024-06-29T15:57:33.355322+0000","last_fullsized":"2024-06-29T15:57:33.355322+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:09.017396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355087+0000","last_change":"2024-06-29T15:57:22.293696+0000","last_active":"2024-06-29T15:57:33.355087+0000","last_peered":"2024-06-29T15:57:33.355087+0000","last_clean":"2024-06-29T15:57:33.355087+0000","last_became_active":"2024-06-29T15:57:22.293263+0000","last_became_peered":"2024-06-29T15:57:22.293263+0000","last_unstale":"2024-06-29T15:57:33.355087+0000","last_undegraded":"2024-06-29T15:57:33.355087+0000","last_fullsized":"2024-06-29T15:57:33.355087+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352369+0000","last_change":"2024-06-29T15:57:33.352369+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352369+0000","last_undegraded":"2024-06-29T15:57:33.352369+0000","last_fullsized":"2024-06-29T15:57:33.352369+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356661+0000","last_change":"2024-06-29T15:57:21.274306+0000","last_active":"2024-06-29T15:57:33.356661+0000","last_peered":"2024-06-29T15:57:33.356661+0000","last_clean":"2024-06-29T15:57:33.356661+0000","last_became_active":"2024-06-29T15:57:21.273632+0000","last_became_peered":"2024-06-29T15:57:21.273632+0000","last_unstale":"2024-06-29T15:57:33.356661+0000","last_undegraded":"2024-06-29T15:57:33.356661+0000","last_fullsized":"2024-06-29T15:57:33.356661+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361270+0000","last_change":"2024-06-29T15:57:22.297633+0000","last_active":"2024-06-29T15:57:33.361270+0000","last_peered":"2024-06-29T15:57:33.361270+0000","last_clean":"2024-06-29T15:57:33.361270+0000","last_became_active":"2024-06-29T15:57:22.297319+0000","last_became_peered":"2024-06-29T15:57:22.297319+0000","last_unstale":"2024-06-29T15:57:33.361270+0000","last_undegraded":"2024-06-29T15:57:33.361270+0000","last_fullsized":"2024-06-29T15:57:33.361270+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:19.625176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356850+0000","last_change":"2024-06-29T15:57:33.355194+0000","last_active":"2024-06-29T15:57:33.356850+0000","last_peered":"2024-06-29T15:57:33.356850+0000","last_clean":"2024-06-29T15:57:33.356850+0000","last_became_active":"2024-06-29T15:57:33.354883+0000","last_became_peered":"2024-06-29T15:57:33.354883+0000","last_unstale":"2024-06-29T15:57:33.356850+0000","last_undegraded":"2024-06-29T15:57:33.356850+0000","last_fullsized":"2024-06-29T15:57:33.356850+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:06.173920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351788+0000","last_change":"2024-06-29T15:57:33.351788+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351788+0000","last_undegraded":"2024-06-29T15:57:33.351788+0000","last_fullsized":"2024-06-29T15:57:33.351788+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361211+0000","last_change":"2024-06-29T15:57:21.283411+0000","last_active":"2024-06-29T15:57:33.361211+0000","last_peered":"2024-06-29T15:57:33.361211+0000","last_clean":"2024-06-29T15:57:33.361211+0000","last_became_active":"2024-06-29T15:57:21.283081+0000","last_became_peered":"2024-06-29T15:57:21.283081+0000","last_unstale":"2024-06-29T15:57:33.361211+0000","last_undegraded":"2024-06-29T15:57:33.361211+0000","last_fullsized":"2024-06-29T15:57:33.361211+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:27.866266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348680+0000","last_change":"2024-06-29T15:57:22.306835+0000","last_active":"2024-06-29T15:57:33.348680+0000","last_peered":"2024-06-29T15:57:33.348680+0000","last_clean":"2024-06-29T15:57:33.348680+0000","last_became_active":"2024-06-29T15:57:22.306295+0000","last_became_peered":"2024-06-29T15:57:22.306295+0000","last_unstale":"2024-06-29T15:57:33.348680+0000","last_undegraded":"2024-06-29T15:57:33.348680+0000","last_fullsized":"2024-06-29T15:57:33.348680+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:52.124235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367249+0000","last_change":"2024-06-29T15:57:33.365463+0000","last_active":"2024-06-29T15:57:33.367249+0000","last_peered":"2024-06-29T15:57:33.367249+0000","last_clean":"2024-06-29T15:57:33.367249+0000","last_became_active":"2024-06-29T15:57:33.365149+0000","last_became_peered":"2024-06-29T15:57:33.365149+0000","last_unstale":"2024-06-29T15:57:33.367249+0000","last_undegraded":"2024-06-29T15:57:33.367249+0000","last_fullsized":"2024-06-29T15:57:33.367249+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:35.252184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356954+0000","last_change":"2024-06-29T15:57:33.356954+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356954+0000","last_undegraded":"2024-06-29T15:57:33.356954+0000","last_fullsized":"2024-06-29T15:57:33.356954+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356973+0000","last_change":"2024-06-29T15:57:21.284516+0000","last_active":"2024-06-29T15:57:33.356973+0000","last_peered":"2024-06-29T15:57:33.356973+0000","last_clean":"2024-06-29T15:57:33.356973+0000","last_became_active":"2024-06-29T15:57:21.284170+0000","last_became_peered":"2024-06-29T15:57:21.284170+0000","last_unstale":"2024-06-29T15:57:33.356973+0000","last_undegraded":"2024-06-29T15:57:33.356973+0000","last_fullsized":"2024-06-29T15:57:33.356973+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.352840+0000","last_change":"2024-06-29T15:57:22.300202+0000","last_active":"2024-06-29T15:57:33.352840+0000","last_peered":"2024-06-29T15:57:33.352840+0000","last_clean":"2024-06-29T15:57:33.352840+0000","last_became_active":"2024-06-29T15:57:22.299897+0000","last_became_peered":"2024-06-29T15:57:22.299897+0000","last_unstale":"2024-06-29T15:57:33.352840+0000","last_undegraded":"2024-06-29T15:57:33.352840+0000","last_fullsized":"2024-06-29T15:57:33.352840+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364931+0000","last_change":"2024-06-29T15:57:33.363355+0000","last_active":"2024-06-29T15:57:33.364931+0000","last_peered":"2024-06-29T15:57:33.364931+0000","last_clean":"2024-06-29T15:57:33.364931+0000","last_became_active":"2024-06-29T15:57:33.363067+0000","last_became_peered":"2024-06-29T15:57:33.363067+0000","last_unstale":"2024-06-29T15:57:33.364931+0000","last_undegraded":"2024-06-29T15:57:33.364931+0000","last_fullsized":"2024-06-29T15:57:33.364931+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351200+0000","last_change":"2024-06-29T15:57:33.351200+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351200+0000","last_undegraded":"2024-06-29T15:57:33.351200+0000","last_fullsized":"2024-06-29T15:57:33.351200+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356344+0000","last_change":"2024-06-29T15:57:21.274833+0000","last_active":"2024-06-29T15:57:33.356344+0000","last_peered":"2024-06-29T15:57:33.356344+0000","last_clean":"2024-06-29T15:57:33.356344+0000","last_became_active":"2024-06-29T15:57:21.274451+0000","last_became_peered":"2024-06-29T15:57:21.274451+0000","last_unstale":"2024-06-29T15:57:33.356344+0000","last_undegraded":"2024-06-29T15:57:33.356344+0000","last_fullsized":"2024-06-29T15:57:33.356344+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:25.284538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354475+0000","last_change":"2024-06-29T15:57:22.301084+0000","last_active":"2024-06-29T15:57:33.354475+0000","last_peered":"2024-06-29T15:57:33.354475+0000","last_clean":"2024-06-29T15:57:33.354475+0000","last_became_active":"2024-06-29T15:57:22.300639+0000","last_became_peered":"2024-06-29T15:57:22.300639+0000","last_unstale":"2024-06-29T15:57:33.354475+0000","last_undegraded":"2024-06-29T15:57:33.354475+0000","last_fullsized":"2024-06-29T15:57:33.354475+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361064+0000","last_change":"2024-06-29T15:57:33.358185+0000","last_active":"2024-06-29T15:57:33.361064+0000","last_peered":"2024-06-29T15:57:33.361064+0000","last_clean":"2024-06-29T15:57:33.361064+0000","last_became_active":"2024-06-29T15:57:33.357885+0000","last_became_peered":"2024-06-29T15:57:33.357885+0000","last_unstale":"2024-06-29T15:57:33.361064+0000","last_undegraded":"2024-06-29T15:57:33.361064+0000","last_fullsized":"2024-06-29T15:57:33.361064+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:50:30.619228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351153+0000","last_change":"2024-06-29T15:57:33.351153+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351153+0000","last_undegraded":"2024-06-29T15:57:33.351153+0000","last_fullsized":"2024-06-29T15:57:33.351153+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"212'4","reported_seq":22,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365147+0000","last_change":"2024-06-29T15:57:21.276327+0000","last_active":"2024-06-29T15:57:33.365147+0000","last_peered":"2024-06-29T15:57:33.365147+0000","last_clean":"2024-06-29T15:57:33.365147+0000","last_became_active":"2024-06-29T15:57:21.276031+0000","last_became_peered":"2024-06-29T15:57:21.276031+0000","last_unstale":"2024-06-29T15:57:33.365147+0000","last_undegraded":"2024-06-29T15:57:33.365147+0000","last_fullsized":"2024-06-29T15:57:33.365147+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:23.626777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356074+0000","last_change":"2024-06-29T15:57:22.293417+0000","last_active":"2024-06-29T15:57:33.356074+0000","last_peered":"2024-06-29T15:57:33.356074+0000","last_clean":"2024-06-29T15:57:33.356074+0000","last_became_active":"2024-06-29T15:57:22.293083+0000","last_became_peered":"2024-06-29T15:57:22.293083+0000","last_unstale":"2024-06-29T15:57:33.356074+0000","last_undegraded":"2024-06-29T15:57:33.356074+0000","last_fullsized":"2024-06-29T15:57:33.356074+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359057+0000","last_change":"2024-06-29T15:57:33.356187+0000","last_active":"2024-06-29T15:57:33.359057+0000","last_peered":"2024-06-29T15:57:33.359057+0000","last_clean":"2024-06-29T15:57:33.359057+0000","last_became_active":"2024-06-29T15:57:33.355400+0000","last_became_peered":"2024-06-29T15:57:33.355400+0000","last_unstale":"2024-06-29T15:57:33.359057+0000","last_undegraded":"2024-06-29T15:57:33.359057+0000","last_fullsized":"2024-06-29T15:57:33.359057+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:01.393531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354238+0000","last_change":"2024-06-29T15:57:33.354238+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354238+0000","last_undegraded":"2024-06-29T15:57:33.354238+0000","last_fullsized":"2024-06-29T15:57:33.354238+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356423+0000","last_change":"2024-06-29T15:57:22.298975+0000","last_active":"2024-06-29T15:57:33.356423+0000","last_peered":"2024-06-29T15:57:33.356423+0000","last_clean":"2024-06-29T15:57:33.356423+0000","last_became_active":"2024-06-29T15:57:22.298689+0000","last_became_peered":"2024-06-29T15:57:22.298689+0000","last_unstale":"2024-06-29T15:57:33.356423+0000","last_undegraded":"2024-06-29T15:57:33.356423+0000","last_fullsized":"2024-06-29T15:57:33.356423+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359369+0000","last_change":"2024-06-29T15:57:33.359369+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359369+0000","last_undegraded":"2024-06-29T15:57:33.359369+0000","last_fullsized":"2024-06-29T15:57:33.359369+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348068+0000","last_change":"2024-06-29T15:57:22.292249+0000","last_active":"2024-06-29T15:57:33.348068+0000","last_peered":"2024-06-29T15:57:33.348068+0000","last_clean":"2024-06-29T15:57:33.348068+0000","last_became_active":"2024-06-29T15:57:22.291918+0000","last_became_peered":"2024-06-29T15:57:22.291918+0000","last_unstale":"2024-06-29T15:57:33.348068+0000","last_undegraded":"2024-06-29T15:57:33.348068+0000","last_fullsized":"2024-06-29T15:57:33.348068+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:57.129905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351129+0000","last_change":"2024-06-29T15:57:33.351129+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351129+0000","last_undegraded":"2024-06-29T15:57:33.351129+0000","last_fullsized":"2024-06-29T15:57:33.351129+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355012+0000","last_change":"2024-06-29T15:57:22.285362+0000","last_active":"2024-06-29T15:57:33.355012+0000","last_peered":"2024-06-29T15:57:33.355012+0000","last_clean":"2024-06-29T15:57:33.355012+0000","last_became_active":"2024-06-29T15:57:22.284982+0000","last_became_peered":"2024-06-29T15:57:22.284982+0000","last_unstale":"2024-06-29T15:57:33.355012+0000","last_undegraded":"2024-06-29T15:57:33.355012+0000","last_fullsized":"2024-06-29T15:57:33.355012+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357913+0000","last_change":"2024-06-29T15:57:33.357913+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357913+0000","last_undegraded":"2024-06-29T15:57:33.357913+0000","last_fullsized":"2024-06-29T15:57:33.357913+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364482+0000","last_change":"2024-06-29T15:57:22.297998+0000","last_active":"2024-06-29T15:57:33.364482+0000","last_peered":"2024-06-29T15:57:33.364482+0000","last_clean":"2024-06-29T15:57:33.364482+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:33.364482+0000","last_undegraded":"2024-06-29T15:57:33.364482+0000","last_fullsized":"2024-06-29T15:57:33.364482+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357574+0000","last_change":"2024-06-29T15:57:33.357574+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357574+0000","last_undegraded":"2024-06-29T15:57:33.357574+0000","last_fullsized":"2024-06-29T15:57:33.357574+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356051+0000","last_change":"2024-06-29T15:57:22.297917+0000","last_active":"2024-06-29T15:57:33.356051+0000","last_peered":"2024-06-29T15:57:33.356051+0000","last_clean":"2024-06-29T15:57:33.356051+0000","last_became_active":"2024-06-29T15:57:22.297638+0000","last_became_peered":"2024-06-29T15:57:22.297638+0000","last_unstale":"2024-06-29T15:57:33.356051+0000","last_undegraded":"2024-06-29T15:57:33.356051+0000","last_fullsized":"2024-06-29T15:57:33.356051+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351060+0000","last_change":"2024-06-29T15:57:33.351060+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351060+0000","last_undegraded":"2024-06-29T15:57:33.351060+0000","last_fullsized":"2024-06-29T15:57:33.351060+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364399+0000","last_change":"2024-06-29T15:57:22.298174+0000","last_active":"2024-06-29T15:57:33.364399+0000","last_peered":"2024-06-29T15:57:33.364399+0000","last_clean":"2024-06-29T15:57:33.364399+0000","last_became_active":"2024-06-29T15:57:22.297862+0000","last_became_peered":"2024-06-29T15:57:22.297862+0000","last_unstale":"2024-06-29T15:57:33.364399+0000","last_undegraded":"2024-06-29T15:57:33.364399+0000","last_fullsized":"2024-06-29T15:57:33.364399+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354825+0000","last_change":"2024-06-29T15:57:33.354825+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354825+0000","last_undegraded":"2024-06-29T15:57:33.354825+0000","last_fullsized":"2024-06-29T15:57:33.354825+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358628+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:33.358628+0000","last_peered":"2024-06-29T15:57:33.358628+0000","last_clean":"2024-06-29T15:57:33.358628+0000","last_became_active":"2024-06-29T15:57:22.301541+0000","last_became_peered":"2024-06-29T15:57:22.301541+0000","last_unstale":"2024-06-29T15:57:33.358628+0000","last_undegraded":"2024-06-29T15:57:33.358628+0000","last_fullsized":"2024-06-29T15:57:33.358628+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:55.249516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353618+0000","last_change":"2024-06-29T15:57:33.353618+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353618+0000","last_undegraded":"2024-06-29T15:57:33.353618+0000","last_fullsized":"2024-06-29T15:57:33.353618+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355658+0000","last_change":"2024-06-29T15:57:22.292584+0000","last_active":"2024-06-29T15:57:33.355658+0000","last_peered":"2024-06-29T15:57:33.355658+0000","last_clean":"2024-06-29T15:57:33.355658+0000","last_became_active":"2024-06-29T15:57:22.291806+0000","last_became_peered":"2024-06-29T15:57:22.291806+0000","last_unstale":"2024-06-29T15:57:33.355658+0000","last_undegraded":"2024-06-29T15:57:33.355658+0000","last_fullsized":"2024-06-29T15:57:33.355658+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352348+0000","last_change":"2024-06-29T15:57:33.352348+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352348+0000","last_undegraded":"2024-06-29T15:57:33.352348+0000","last_fullsized":"2024-06-29T15:57:33.352348+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363619+0000","last_change":"2024-06-29T15:57:22.301601+0000","last_active":"2024-06-29T15:57:33.363619+0000","last_peered":"2024-06-29T15:57:33.363619+0000","last_clean":"2024-06-29T15:57:33.363619+0000","last_became_active":"2024-06-29T15:57:22.301244+0000","last_became_peered":"2024-06-29T15:57:22.301244+0000","last_unstale":"2024-06-29T15:57:33.363619+0000","last_undegraded":"2024-06-29T15:57:33.363619+0000","last_fullsized":"2024-06-29T15:57:33.363619+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:24:56.938464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352379+0000","last_change":"2024-06-29T15:57:33.352379+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352379+0000","last_undegraded":"2024-06-29T15:57:33.352379+0000","last_fullsized":"2024-06-29T15:57:33.352379+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353891+0000","last_change":"2024-06-29T15:57:22.301362+0000","last_active":"2024-06-29T15:57:33.353891+0000","last_peered":"2024-06-29T15:57:33.353891+0000","last_clean":"2024-06-29T15:57:33.353891+0000","last_became_active":"2024-06-29T15:57:22.300704+0000","last_became_peered":"2024-06-29T15:57:22.300704+0000","last_unstale":"2024-06-29T15:57:33.353891+0000","last_undegraded":"2024-06-29T15:57:33.353891+0000","last_fullsized":"2024-06-29T15:57:33.353891+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351810+0000","last_change":"2024-06-29T15:57:33.351810+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351810+0000","last_undegraded":"2024-06-29T15:57:33.351810+0000","last_fullsized":"2024-06-29T15:57:33.351810+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353925+0000","last_change":"2024-06-29T15:57:22.307686+0000","last_active":"2024-06-29T15:57:33.353925+0000","last_peered":"2024-06-29T15:57:33.353925+0000","last_clean":"2024-06-29T15:57:33.353925+0000","last_became_active":"2024-06-29T15:57:22.307324+0000","last_became_peered":"2024-06-29T15:57:22.307324+0000","last_unstale":"2024-06-29T15:57:33.353925+0000","last_undegraded":"2024-06-29T15:57:33.353925+0000","last_fullsized":"2024-06-29T15:57:33.353925+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353324+0000","last_change":"2024-06-29T15:57:33.353324+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353324+0000","last_undegraded":"2024-06-29T15:57:33.353324+0000","last_fullsized":"2024-06-29T15:57:33.353324+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361177+0000","last_change":"2024-06-29T15:57:22.307230+0000","last_active":"2024-06-29T15:57:33.361177+0000","last_peered":"2024-06-29T15:57:33.361177+0000","last_clean":"2024-06-29T15:57:33.361177+0000","last_became_active":"2024-06-29T15:57:22.306920+0000","last_became_peered":"2024-06-29T15:57:22.306920+0000","last_unstale":"2024-06-29T15:57:33.361177+0000","last_undegraded":"2024-06-29T15:57:33.361177+0000","last_fullsized":"2024-06-29T15:57:33.361177+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:22.598647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352631+0000","last_change":"2024-06-29T15:57:33.352631+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352631+0000","last_undegraded":"2024-06-29T15:57:33.352631+0000","last_fullsized":"2024-06-29T15:57:33.352631+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359396+0000","last_change":"2024-06-29T15:57:22.290186+0000","last_active":"2024-06-29T15:57:33.359396+0000","last_peered":"2024-06-29T15:57:33.359396+0000","last_clean":"2024-06-29T15:57:33.359396+0000","last_became_active":"2024-06-29T15:57:22.289908+0000","last_became_peered":"2024-06-29T15:57:22.289908+0000","last_unstale":"2024-06-29T15:57:33.359396+0000","last_undegraded":"2024-06-29T15:57:33.359396+0000","last_fullsized":"2024-06-29T15:57:33.359396+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353284+0000","last_change":"2024-06-29T15:57:33.353284+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353284+0000","last_undegraded":"2024-06-29T15:57:33.353284+0000","last_fullsized":"2024-06-29T15:57:33.353284+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348980+0000","last_change":"2024-06-29T15:57:22.306593+0000","last_active":"2024-06-29T15:57:33.348980+0000","last_peered":"2024-06-29T15:57:33.348980+0000","last_clean":"2024-06-29T15:57:33.348980+0000","last_became_active":"2024-06-29T15:57:22.296653+0000","last_became_peered":"2024-06-29T15:57:22.296653+0000","last_unstale":"2024-06-29T15:57:33.348980+0000","last_undegraded":"2024-06-29T15:57:33.348980+0000","last_fullsized":"2024-06-29T15:57:33.348980+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:41.738155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351749+0000","last_change":"2024-06-29T15:57:33.351749+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351749+0000","last_undegraded":"2024-06-29T15:57:33.351749+0000","last_fullsized":"2024-06-29T15:57:33.351749+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348828+0000","last_change":"2024-06-29T15:57:22.289633+0000","last_active":"2024-06-29T15:57:33.348828+0000","last_peered":"2024-06-29T15:57:33.348828+0000","last_clean":"2024-06-29T15:57:33.348828+0000","last_became_active":"2024-06-29T15:57:22.289349+0000","last_became_peered":"2024-06-29T15:57:22.289349+0000","last_unstale":"2024-06-29T15:57:33.348828+0000","last_undegraded":"2024-06-29T15:57:33.348828+0000","last_fullsized":"2024-06-29T15:57:33.348828+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.873806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357678+0000","last_change":"2024-06-29T15:57:33.357678+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357678+0000","last_undegraded":"2024-06-29T15:57:33.357678+0000","last_fullsized":"2024-06-29T15:57:33.357678+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355608+0000","last_change":"2024-06-29T15:57:22.300787+0000","last_active":"2024-06-29T15:57:33.355608+0000","last_peered":"2024-06-29T15:57:33.355608+0000","last_clean":"2024-06-29T15:57:33.355608+0000","last_became_active":"2024-06-29T15:57:22.300112+0000","last_became_peered":"2024-06-29T15:57:22.300112+0000","last_unstale":"2024-06-29T15:57:33.355608+0000","last_undegraded":"2024-06-29T15:57:33.355608+0000","last_fullsized":"2024-06-29T15:57:33.355608+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:30.249483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353010+0000","last_change":"2024-06-29T15:57:33.353010+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353010+0000","last_undegraded":"2024-06-29T15:57:33.353010+0000","last_fullsized":"2024-06-29T15:57:33.353010+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348033+0000","last_change":"2024-06-29T15:57:22.290047+0000","last_active":"2024-06-29T15:57:33.348033+0000","last_peered":"2024-06-29T15:57:33.348033+0000","last_clean":"2024-06-29T15:57:33.348033+0000","last_became_active":"2024-06-29T15:57:22.289742+0000","last_became_peered":"2024-06-29T15:57:22.289742+0000","last_unstale":"2024-06-29T15:57:33.348033+0000","last_undegraded":"2024-06-29T15:57:33.348033+0000","last_fullsized":"2024-06-29T15:57:33.348033+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:03:46.271206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362654+0000","last_change":"2024-06-29T15:57:33.362654+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362654+0000","last_undegraded":"2024-06-29T15:57:33.362654+0000","last_fullsized":"2024-06-29T15:57:33.362654+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356944+0000","last_change":"2024-06-29T15:57:22.302418+0000","last_active":"2024-06-29T15:57:33.356944+0000","last_peered":"2024-06-29T15:57:33.356944+0000","last_clean":"2024-06-29T15:57:33.356944+0000","last_became_active":"2024-06-29T15:57:22.302075+0000","last_became_peered":"2024-06-29T15:57:22.302075+0000","last_unstale":"2024-06-29T15:57:33.356944+0000","last_undegraded":"2024-06-29T15:57:33.356944+0000","last_fullsized":"2024-06-29T15:57:33.356944+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.350522+0000","last_change":"2024-06-29T15:57:33.350522+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.350522+0000","last_undegraded":"2024-06-29T15:57:33.350522+0000","last_fullsized":"2024-06-29T15:57:33.350522+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354196+0000","last_change":"2024-06-29T15:57:22.303092+0000","last_active":"2024-06-29T15:57:33.354196+0000","last_peered":"2024-06-29T15:57:33.354196+0000","last_clean":"2024-06-29T15:57:33.354196+0000","last_became_active":"2024-06-29T15:57:22.302807+0000","last_became_peered":"2024-06-29T15:57:22.302807+0000","last_unstale":"2024-06-29T15:57:33.354196+0000","last_undegraded":"2024-06-29T15:57:33.354196+0000","last_fullsized":"2024-06-29T15:57:33.354196+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360360+0000","last_change":"2024-06-29T15:57:33.360360+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360360+0000","last_undegraded":"2024-06-29T15:57:33.360360+0000","last_fullsized":"2024-06-29T15:57:33.360360+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365064+0000","last_change":"2024-06-29T15:57:22.295681+0000","last_active":"2024-06-29T15:57:33.365064+0000","last_peered":"2024-06-29T15:57:33.365064+0000","last_clean":"2024-06-29T15:57:33.365064+0000","last_became_active":"2024-06-29T15:57:22.295357+0000","last_became_peered":"2024-06-29T15:57:22.295357+0000","last_unstale":"2024-06-29T15:57:33.365064+0000","last_undegraded":"2024-06-29T15:57:33.365064+0000","last_fullsized":"2024-06-29T15:57:33.365064+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353724+0000","last_change":"2024-06-29T15:57:33.353724+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353724+0000","last_undegraded":"2024-06-29T15:57:33.353724+0000","last_fullsized":"2024-06-29T15:57:33.353724+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365558+0000","last_change":"2024-06-29T15:57:22.298612+0000","last_active":"2024-06-29T15:57:33.365558+0000","last_peered":"2024-06-29T15:57:33.365558+0000","last_clean":"2024-06-29T15:57:33.365558+0000","last_became_active":"2024-06-29T15:57:22.298288+0000","last_became_peered":"2024-06-29T15:57:22.298288+0000","last_unstale":"2024-06-29T15:57:33.365558+0000","last_undegraded":"2024-06-29T15:57:33.365558+0000","last_fullsized":"2024-06-29T15:57:33.365558+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351291+0000","last_change":"2024-06-29T15:57:33.351291+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351291+0000","last_undegraded":"2024-06-29T15:57:33.351291+0000","last_fullsized":"2024-06-29T15:57:33.351291+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356870+0000","last_change":"2024-06-29T15:57:22.298781+0000","last_active":"2024-06-29T15:57:33.356870+0000","last_peered":"2024-06-29T15:57:33.356870+0000","last_clean":"2024-06-29T15:57:33.356870+0000","last_became_active":"2024-06-29T15:57:22.298300+0000","last_became_peered":"2024-06-29T15:57:22.298300+0000","last_unstale":"2024-06-29T15:57:33.356870+0000","last_undegraded":"2024-06-29T15:57:33.356870+0000","last_fullsized":"2024-06-29T15:57:33.356870+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349621+0000","last_change":"2024-06-29T15:57:33.349621+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349621+0000","last_undegraded":"2024-06-29T15:57:33.349621+0000","last_fullsized":"2024-06-29T15:57:33.349621+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348689+0000","last_change":"2024-06-29T15:57:22.296441+0000","last_active":"2024-06-29T15:57:33.348689+0000","last_peered":"2024-06-29T15:57:33.348689+0000","last_clean":"2024-06-29T15:57:33.348689+0000","last_became_active":"2024-06-29T15:57:22.296146+0000","last_became_peered":"2024-06-29T15:57:22.296146+0000","last_unstale":"2024-06-29T15:57:33.348689+0000","last_undegraded":"2024-06-29T15:57:33.348689+0000","last_fullsized":"2024-06-29T15:57:33.348689+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:54.360521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356517+0000","last_change":"2024-06-29T15:57:33.356517+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356517+0000","last_undegraded":"2024-06-29T15:57:33.356517+0000","last_fullsized":"2024-06-29T15:57:33.356517+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356057+0000","last_change":"2024-06-29T15:57:22.291096+0000","last_active":"2024-06-29T15:57:33.356057+0000","last_peered":"2024-06-29T15:57:33.356057+0000","last_clean":"2024-06-29T15:57:33.356057+0000","last_became_active":"2024-06-29T15:57:22.290428+0000","last_became_peered":"2024-06-29T15:57:22.290428+0000","last_unstale":"2024-06-29T15:57:33.356057+0000","last_undegraded":"2024-06-29T15:57:33.356057+0000","last_fullsized":"2024-06-29T15:57:33.356057+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:20.813793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352280+0000","last_change":"2024-06-29T15:57:33.352280+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352280+0000","last_undegraded":"2024-06-29T15:57:33.352280+0000","last_fullsized":"2024-06-29T15:57:33.352280+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355022+0000","last_change":"2024-06-29T15:57:22.297617+0000","last_active":"2024-06-29T15:57:33.355022+0000","last_peered":"2024-06-29T15:57:33.355022+0000","last_clean":"2024-06-29T15:57:33.355022+0000","last_became_active":"2024-06-29T15:57:22.297256+0000","last_became_peered":"2024-06-29T15:57:22.297256+0000","last_unstale":"2024-06-29T15:57:33.355022+0000","last_undegraded":"2024-06-29T15:57:33.355022+0000","last_fullsized":"2024-06-29T15:57:33.355022+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359862+0000","last_change":"2024-06-29T15:57:33.359862+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359862+0000","last_undegraded":"2024-06-29T15:57:33.359862+0000","last_fullsized":"2024-06-29T15:57:33.359862+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356834+0000","last_change":"2024-06-29T15:57:22.304052+0000","last_active":"2024-06-29T15:57:33.356834+0000","last_peered":"2024-06-29T15:57:33.356834+0000","last_clean":"2024-06-29T15:57:33.356834+0000","last_became_active":"2024-06-29T15:57:22.303598+0000","last_became_peered":"2024-06-29T15:57:22.303598+0000","last_unstale":"2024-06-29T15:57:33.356834+0000","last_undegraded":"2024-06-29T15:57:33.356834+0000","last_fullsized":"2024-06-29T15:57:33.356834+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352198+0000","last_change":"2024-06-29T15:57:33.352198+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352198+0000","last_undegraded":"2024-06-29T15:57:33.352198+0000","last_fullsized":"2024-06-29T15:57:33.352198+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356415+0000","last_change":"2024-06-29T15:57:22.288041+0000","last_active":"2024-06-29T15:57:33.356415+0000","last_peered":"2024-06-29T15:57:33.356415+0000","last_clean":"2024-06-29T15:57:33.356415+0000","last_became_active":"2024-06-29T15:57:22.287729+0000","last_became_peered":"2024-06-29T15:57:22.287729+0000","last_unstale":"2024-06-29T15:57:33.356415+0000","last_undegraded":"2024-06-29T15:57:33.356415+0000","last_fullsized":"2024-06-29T15:57:33.356415+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362099+0000","last_change":"2024-06-29T15:57:33.362099+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362099+0000","last_undegraded":"2024-06-29T15:57:33.362099+0000","last_fullsized":"2024-06-29T15:57:33.362099+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356673+0000","last_change":"2024-06-29T15:57:22.290909+0000","last_active":"2024-06-29T15:57:33.356673+0000","last_peered":"2024-06-29T15:57:33.356673+0000","last_clean":"2024-06-29T15:57:33.356673+0000","last_became_active":"2024-06-29T15:57:22.290643+0000","last_became_peered":"2024-06-29T15:57:22.290643+0000","last_unstale":"2024-06-29T15:57:33.356673+0000","last_undegraded":"2024-06-29T15:57:33.356673+0000","last_fullsized":"2024-06-29T15:57:33.356673+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354913+0000","last_change":"2024-06-29T15:57:33.354913+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354913+0000","last_undegraded":"2024-06-29T15:57:33.354913+0000","last_fullsized":"2024-06-29T15:57:33.354913+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361010+0000","last_change":"2024-06-29T15:57:22.298374+0000","last_active":"2024-06-29T15:57:33.361010+0000","last_peered":"2024-06-29T15:57:33.361010+0000","last_clean":"2024-06-29T15:57:33.361010+0000","last_became_active":"2024-06-29T15:57:22.297962+0000","last_became_peered":"2024-06-29T15:57:22.297962+0000","last_unstale":"2024-06-29T15:57:33.361010+0000","last_undegraded":"2024-06-29T15:57:33.361010+0000","last_fullsized":"2024-06-29T15:57:33.361010+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:13.834244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354164+0000","last_change":"2024-06-29T15:57:33.354164+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354164+0000","last_undegraded":"2024-06-29T15:57:33.354164+0000","last_fullsized":"2024-06-29T15:57:33.354164+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356686+0000","last_change":"2024-06-29T15:57:22.300745+0000","last_active":"2024-06-29T15:57:33.356686+0000","last_peered":"2024-06-29T15:57:33.356686+0000","last_clean":"2024-06-29T15:57:33.356686+0000","last_became_active":"2024-06-29T15:57:22.300414+0000","last_became_peered":"2024-06-29T15:57:22.300414+0000","last_unstale":"2024-06-29T15:57:33.356686+0000","last_undegraded":"2024-06-29T15:57:33.356686+0000","last_fullsized":"2024-06-29T15:57:33.356686+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356852+0000","last_change":"2024-06-29T15:57:33.356852+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356852+0000","last_undegraded":"2024-06-29T15:57:33.356852+0000","last_fullsized":"2024-06-29T15:57:33.356852+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355941+0000","last_change":"2024-06-29T15:57:22.297971+0000","last_active":"2024-06-29T15:57:33.355941+0000","last_peered":"2024-06-29T15:57:33.355941+0000","last_clean":"2024-06-29T15:57:33.355941+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:33.355941+0000","last_undegraded":"2024-06-29T15:57:33.355941+0000","last_fullsized":"2024-06-29T15:57:33.355941+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360424+0000","last_change":"2024-06-29T15:57:33.360424+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360424+0000","last_undegraded":"2024-06-29T15:57:33.360424+0000","last_fullsized":"2024-06-29T15:57:33.360424+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355500+0000","last_change":"2024-06-29T15:57:22.301019+0000","last_active":"2024-06-29T15:57:33.355500+0000","last_peered":"2024-06-29T15:57:33.355500+0000","last_clean":"2024-06-29T15:57:33.355500+0000","last_became_active":"2024-06-29T15:57:22.300506+0000","last_became_peered":"2024-06-29T15:57:22.300506+0000","last_unstale":"2024-06-29T15:57:33.355500+0000","last_undegraded":"2024-06-29T15:57:33.355500+0000","last_fullsized":"2024-06-29T15:57:33.355500+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52456,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319384,"statfs":{"total":96636764160,"available":96583049216,"internally_reserved":0,"allocated":3305472,"data_stored":1914275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575988,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42140,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329700,"statfs":{"total":96636764160,"available":96593612800,"internally_reserved":0,"allocated":2703360,"data_stored":1321634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575986,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42128,"kb_used_data":2628,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329712,"statfs":{"total":96636764160,"available":96593625088,"internally_reserved":0,"allocated":2691072,"data_stored":1314370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60208,"kb_used_data":2656,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311632,"statfs":{"total":96636764160,"available":96575111168,"internally_reserved":0,"allocated":2719744,"data_stored":1324011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47716,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324124,"statfs":{"total":96636764160,"available":96587902976,"internally_reserved":0,"allocated":2711552,"data_stored":1321234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575987,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47704,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324136,"statfs":{"total":96636764160,"available":96587915264,"internally_reserved":0,"allocated":2699264,"data_stored":1321052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575986,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60180,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311660,"statfs":{"total":96636764160,"available":96575139840,"internally_reserved":0,"allocated":2695168,"data_stored":1313844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60756,"kb_used_data":3208,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311084,"statfs":{"total":96636764160,"available":96574550016,"internally_reserved":0,"allocated":3284992,"data_stored":1904246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:38.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:38.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.919+0000 7f0bafe26640 1 -- 172.21.15.81:0/2156579903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b90003a00 msgr2=0x7f0b90003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.919+0000 7f0bafe26640 1 --2- 172.21.15.81:0/2156579903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b90003a00 0x7f0b90003e70 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f0b98009ed0 tx=0x7f0b90005830 comp rx=0 tx=0).stop 2024-06-29T15:57:38.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.919+0000 7f0bae3c3640 1 -- 172.21.15.81:0/2156579903 reap_dead start 2024-06-29T15:57:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.919+0000 7f0bafe26640 1 -- 172.21.15.81:0/2156579903 shutdown_connections 2024-06-29T15:57:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.919+0000 7f0bafe26640 1 -- 172.21.15.81:0/2156579903 >> 172.21.15.81:0/2156579903 conn(0x7f0ba80b1ce0 msgr2=0x7f0ba80b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.919+0000 7f0bafe26640 1 -- 172.21.15.81:0/2156579903 shutdown_connections 2024-06-29T15:57:38.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.919+0000 7f0bafe26640 1 -- 172.21.15.81:0/2156579903 wait complete. 2024-06-29T15:57:38.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.921+0000 7f0bafe26640 1 Processor -- start 2024-06-29T15:57:38.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.921+0000 7f0bafe26640 1 -- start start 2024-06-29T15:57:38.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0bafe26640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b90003a00 0x7f0ba80a9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0bafe26640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba80ca120 0x7f0ba813d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0bafe26640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba80cb960 0x7f0ba815cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0bafe26640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba8140d10 con 0x7f0b90003a00 2024-06-29T15:57:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0bafe26640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba815bd70 con 0x7f0ba80ca120 2024-06-29T15:57:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0bafe26640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba8140e90 con 0x7f0ba80cb960 2024-06-29T15:57:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0badbc2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b90003a00 0x7f0ba80a9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0badbc2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b90003a00 0x7f0ba80a9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35474/0 (socket says 172.21.15.81:35474) 2024-06-29T15:57:38.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0badbc2640 1 -- 172.21.15.81:0/239591899 learned_addr learned my addr 172.21.15.81:0/239591899 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0bad3c1640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba80ca120 0x7f0ba813d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0bae3c3640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba80cb960 0x7f0ba815cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0badbc2640 1 -- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba80cb960 msgr2=0x7f0ba815cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0badbc2640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba80cb960 0x7f0ba815cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0badbc2640 1 -- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba80ca120 msgr2=0x7f0ba813d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0badbc2640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba80ca120 0x7f0ba813d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0badbc2640 1 -- 172.21.15.81:0/239591899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba813dca0 con 0x7f0b90003a00 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0bad3c1640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba80ca120 0x7f0ba813d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.922+0000 7f0badbc2640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b90003a00 0x7f0ba80a9b90 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f0b98036040 tx=0x7f0b98033bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.923+0000 7f0ba67fc640 1 -- 172.21.15.81:0/239591899 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98043070 con 0x7f0b90003a00 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.923+0000 7f0ba67fc640 1 -- 172.21.15.81:0/239591899 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b98033db0 con 0x7f0b90003a00 2024-06-29T15:57:38.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.923+0000 7f0bafe26640 1 -- 172.21.15.81:0/239591899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba80ca8c0 con 0x7f0b90003a00 2024-06-29T15:57:38.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.923+0000 7f0ba67fc640 1 -- 172.21.15.81:0/239591899 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98038040 con 0x7f0b90003a00 2024-06-29T15:57:38.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.923+0000 7f0bafe26640 1 -- 172.21.15.81:0/239591899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba80cae80 con 0x7f0b90003a00 2024-06-29T15:57:38.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.924+0000 7f0bafe26640 1 -- 172.21.15.81:0/239591899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ba80732a0 con 0x7f0b90003a00 2024-06-29T15:57:38.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.924+0000 7f0ba67fc640 1 -- 172.21.15.81:0/239591899 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f0b98046070 con 0x7f0b90003a00 2024-06-29T15:57:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.925+0000 7f0ba67fc640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b94076420 0x7f0b94078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.925+0000 7f0ba67fc640 1 -- 172.21.15.81:0/239591899 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7f0b980b8750 con 0x7f0b90003a00 2024-06-29T15:57:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.925+0000 7f0bad3c1640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b94076420 0x7f0b94078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.926+0000 7f0bad3c1640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b94076420 0x7f0b94078930 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8134f90 tx=0x7f0b9c005f90 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:38.929+0000 7f0ba67fc640 1 -- 172.21.15.81:0/239591899 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0b98081460 con 0x7f0b90003a00 2024-06-29T15:57:39.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.108+0000 7f0bafe26640 1 -- 172.21.15.81:0/239591899 --> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ba80aa0d0 con 0x7f0b94076420 2024-06-29T15:57:39.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.117+0000 7f0ba67fc640 1 -- 172.21.15.81:0/239591899 <== mgr.8666 v2:172.21.15.167:6832/2018755589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+486086 (secure 0 0 0) 0x7f0b9c013bd0 con 0x7f0b94076420 2024-06-29T15:57:39.124 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:57:39.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 -- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b94076420 msgr2=0x7f0b94078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b94076420 0x7f0b94078930 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8134f90 tx=0x7f0b9c005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:39.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 -- 172.21.15.81:0/239591899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b90003a00 msgr2=0x7f0ba80a9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b90003a00 0x7f0ba80a9b90 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f0b98036040 tx=0x7f0b98033bc0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 -- 172.21.15.81:0/239591899 shutdown_connections 2024-06-29T15:57:39.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f0b94076420 0x7f0b94078930 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0ba80cb960 0x7f0ba815cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0ba80ca120 0x7f0ba813d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 --2- 172.21.15.81:0/239591899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b90003a00 0x7f0ba80a9b90 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 -- 172.21.15.81:0/239591899 >> 172.21.15.81:0/239591899 conn(0x7f0ba80b1ce0 msgr2=0x7f0ba813e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 -- 172.21.15.81:0/239591899 shutdown_connections 2024-06-29T15:57:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.121+0000 7f0bafe26640 1 -- 172.21.15.81:0/239591899 wait complete. 2024-06-29T15:57:39.134 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-06-29T15:57:37.873180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594021,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":400,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":217,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413288,"kb_used_data":22276,"kb_used_omap":12,"kb_used_meta":390899,"kb_avail":754561432,"statfs":{"total":773094113280,"available":772670906368,"internally_reserved":0,"allocated":22810624,"data_stored":11734666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":400281138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1243,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.532321"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357377+0000","last_change":"2024-06-29T15:57:33.357377+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357377+0000","last_undegraded":"2024-06-29T15:57:33.357377+0000","last_fullsized":"2024-06-29T15:57:33.357377+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"202'219","reported_seq":1043,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363727+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:57:33.363727+0000","last_peered":"2024-06-29T15:57:33.363727+0000","last_clean":"2024-06-29T15:57:33.363727+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:57:33.363727+0000","last_undegraded":"2024-06-29T15:57:33.363727+0000","last_fullsized":"2024-06-29T15:57:33.363727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365224+0000","last_change":"2024-06-29T15:57:21.282026+0000","last_active":"2024-06-29T15:57:33.365224+0000","last_peered":"2024-06-29T15:57:33.365224+0000","last_clean":"2024-06-29T15:57:33.365224+0000","last_became_active":"2024-06-29T15:57:21.281709+0000","last_became_peered":"2024-06-29T15:57:21.281709+0000","last_unstale":"2024-06-29T15:57:33.365224+0000","last_undegraded":"2024-06-29T15:57:33.365224+0000","last_fullsized":"2024-06-29T15:57:33.365224+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:54:07.641300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354339+0000","last_change":"2024-06-29T15:57:22.302267+0000","last_active":"2024-06-29T15:57:33.354339+0000","last_peered":"2024-06-29T15:57:33.354339+0000","last_clean":"2024-06-29T15:57:33.354339+0000","last_became_active":"2024-06-29T15:57:22.301972+0000","last_became_peered":"2024-06-29T15:57:22.301972+0000","last_unstale":"2024-06-29T15:57:33.354339+0000","last_undegraded":"2024-06-29T15:57:33.354339+0000","last_fullsized":"2024-06-29T15:57:33.354339+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371713+0000","last_change":"2024-06-29T15:57:33.370756+0000","last_active":"2024-06-29T15:57:33.371713+0000","last_peered":"2024-06-29T15:57:33.371713+0000","last_clean":"2024-06-29T15:57:33.371713+0000","last_became_active":"2024-06-29T15:57:33.370363+0000","last_became_peered":"2024-06-29T15:57:33.370363+0000","last_unstale":"2024-06-29T15:57:33.371713+0000","last_undegraded":"2024-06-29T15:57:33.371713+0000","last_fullsized":"2024-06-29T15:57:33.371713+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:27.582662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362213+0000","last_change":"2024-06-29T15:57:33.362213+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362213+0000","last_undegraded":"2024-06-29T15:57:33.362213+0000","last_fullsized":"2024-06-29T15:57:33.362213+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364223+0000","last_change":"2024-06-29T15:57:21.283495+0000","last_active":"2024-06-29T15:57:33.364223+0000","last_peered":"2024-06-29T15:57:33.364223+0000","last_clean":"2024-06-29T15:57:33.364223+0000","last_became_active":"2024-06-29T15:57:21.283125+0000","last_became_peered":"2024-06-29T15:57:21.283125+0000","last_unstale":"2024-06-29T15:57:33.364223+0000","last_undegraded":"2024-06-29T15:57:33.364223+0000","last_fullsized":"2024-06-29T15:57:33.364223+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354722+0000","last_change":"2024-06-29T15:57:22.292872+0000","last_active":"2024-06-29T15:57:33.354722+0000","last_peered":"2024-06-29T15:57:33.354722+0000","last_clean":"2024-06-29T15:57:33.354722+0000","last_became_active":"2024-06-29T15:57:22.292485+0000","last_became_peered":"2024-06-29T15:57:22.292485+0000","last_unstale":"2024-06-29T15:57:33.354722+0000","last_undegraded":"2024-06-29T15:57:33.354722+0000","last_fullsized":"2024-06-29T15:57:33.354722+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:19.369559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370805+0000","last_change":"2024-06-29T15:57:33.369465+0000","last_active":"2024-06-29T15:57:33.370805+0000","last_peered":"2024-06-29T15:57:33.370805+0000","last_clean":"2024-06-29T15:57:33.370805+0000","last_became_active":"2024-06-29T15:57:33.369174+0000","last_became_peered":"2024-06-29T15:57:33.369174+0000","last_unstale":"2024-06-29T15:57:33.370805+0000","last_undegraded":"2024-06-29T15:57:33.370805+0000","last_fullsized":"2024-06-29T15:57:33.370805+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:02.836842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351888+0000","last_change":"2024-06-29T15:57:33.351888+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351888+0000","last_undegraded":"2024-06-29T15:57:33.351888+0000","last_fullsized":"2024-06-29T15:57:33.351888+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354196+0000","last_change":"2024-06-29T15:57:21.274359+0000","last_active":"2024-06-29T15:57:33.354196+0000","last_peered":"2024-06-29T15:57:33.354196+0000","last_clean":"2024-06-29T15:57:33.354196+0000","last_became_active":"2024-06-29T15:57:21.274036+0000","last_became_peered":"2024-06-29T15:57:21.274036+0000","last_unstale":"2024-06-29T15:57:33.354196+0000","last_undegraded":"2024-06-29T15:57:33.354196+0000","last_fullsized":"2024-06-29T15:57:33.354196+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:02.747066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358900+0000","last_change":"2024-06-29T15:57:22.295101+0000","last_active":"2024-06-29T15:57:33.358900+0000","last_peered":"2024-06-29T15:57:33.358900+0000","last_clean":"2024-06-29T15:57:33.358900+0000","last_became_active":"2024-06-29T15:57:22.294805+0000","last_became_peered":"2024-06-29T15:57:22.294805+0000","last_unstale":"2024-06-29T15:57:33.358900+0000","last_undegraded":"2024-06-29T15:57:33.358900+0000","last_fullsized":"2024-06-29T15:57:33.358900+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.369924+0000","last_change":"2024-06-29T15:57:33.369487+0000","last_active":"2024-06-29T15:57:33.369924+0000","last_peered":"2024-06-29T15:57:33.369924+0000","last_clean":"2024-06-29T15:57:33.369924+0000","last_became_active":"2024-06-29T15:57:33.369138+0000","last_became_peered":"2024-06-29T15:57:33.369138+0000","last_unstale":"2024-06-29T15:57:33.369924+0000","last_undegraded":"2024-06-29T15:57:33.369924+0000","last_fullsized":"2024-06-29T15:57:33.369924+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:06.322225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354530+0000","last_change":"2024-06-29T15:57:33.354530+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354530+0000","last_undegraded":"2024-06-29T15:57:33.354530+0000","last_fullsized":"2024-06-29T15:57:33.354530+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356787+0000","last_change":"2024-06-29T15:57:21.274551+0000","last_active":"2024-06-29T15:57:33.356787+0000","last_peered":"2024-06-29T15:57:33.356787+0000","last_clean":"2024-06-29T15:57:33.356787+0000","last_became_active":"2024-06-29T15:57:21.274029+0000","last_became_peered":"2024-06-29T15:57:21.274029+0000","last_unstale":"2024-06-29T15:57:33.356787+0000","last_undegraded":"2024-06-29T15:57:33.356787+0000","last_fullsized":"2024-06-29T15:57:33.356787+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364941+0000","last_change":"2024-06-29T15:57:22.291143+0000","last_active":"2024-06-29T15:57:33.364941+0000","last_peered":"2024-06-29T15:57:33.364941+0000","last_clean":"2024-06-29T15:57:33.364941+0000","last_became_active":"2024-06-29T15:57:22.290849+0000","last_became_peered":"2024-06-29T15:57:22.290849+0000","last_unstale":"2024-06-29T15:57:33.364941+0000","last_undegraded":"2024-06-29T15:57:33.364941+0000","last_fullsized":"2024-06-29T15:57:33.364941+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367095+0000","last_change":"2024-06-29T15:57:33.366161+0000","last_active":"2024-06-29T15:57:33.367095+0000","last_peered":"2024-06-29T15:57:33.367095+0000","last_clean":"2024-06-29T15:57:33.367095+0000","last_became_active":"2024-06-29T15:57:33.365820+0000","last_became_peered":"2024-06-29T15:57:33.365820+0000","last_unstale":"2024-06-29T15:57:33.367095+0000","last_undegraded":"2024-06-29T15:57:33.367095+0000","last_fullsized":"2024-06-29T15:57:33.367095+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:05.543032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358295+0000","last_change":"2024-06-29T15:57:33.358295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358295+0000","last_undegraded":"2024-06-29T15:57:33.358295+0000","last_fullsized":"2024-06-29T15:57:33.358295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361368+0000","last_change":"2024-06-29T15:57:21.281405+0000","last_active":"2024-06-29T15:57:33.361368+0000","last_peered":"2024-06-29T15:57:33.361368+0000","last_clean":"2024-06-29T15:57:33.361368+0000","last_became_active":"2024-06-29T15:57:21.281080+0000","last_became_peered":"2024-06-29T15:57:21.281080+0000","last_unstale":"2024-06-29T15:57:33.361368+0000","last_undegraded":"2024-06-29T15:57:33.361368+0000","last_fullsized":"2024-06-29T15:57:33.361368+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:51.116695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354452+0000","last_change":"2024-06-29T15:57:22.304057+0000","last_active":"2024-06-29T15:57:33.354452+0000","last_peered":"2024-06-29T15:57:33.354452+0000","last_clean":"2024-06-29T15:57:33.354452+0000","last_became_active":"2024-06-29T15:57:22.303762+0000","last_became_peered":"2024-06-29T15:57:22.303762+0000","last_unstale":"2024-06-29T15:57:33.354452+0000","last_undegraded":"2024-06-29T15:57:33.354452+0000","last_fullsized":"2024-06-29T15:57:33.354452+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:40.125779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370833+0000","last_change":"2024-06-29T15:57:33.369260+0000","last_active":"2024-06-29T15:57:33.370833+0000","last_peered":"2024-06-29T15:57:33.370833+0000","last_clean":"2024-06-29T15:57:33.370833+0000","last_became_active":"2024-06-29T15:57:33.368945+0000","last_became_peered":"2024-06-29T15:57:33.368945+0000","last_unstale":"2024-06-29T15:57:33.370833+0000","last_undegraded":"2024-06-29T15:57:33.370833+0000","last_fullsized":"2024-06-29T15:57:33.370833+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:11.821898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360097+0000","last_change":"2024-06-29T15:57:33.360097+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360097+0000","last_undegraded":"2024-06-29T15:57:33.360097+0000","last_fullsized":"2024-06-29T15:57:33.360097+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357223+0000","last_change":"2024-06-29T15:57:21.277560+0000","last_active":"2024-06-29T15:57:33.357223+0000","last_peered":"2024-06-29T15:57:33.357223+0000","last_clean":"2024-06-29T15:57:33.357223+0000","last_became_active":"2024-06-29T15:57:21.277255+0000","last_became_peered":"2024-06-29T15:57:21.277255+0000","last_unstale":"2024-06-29T15:57:33.357223+0000","last_undegraded":"2024-06-29T15:57:33.357223+0000","last_fullsized":"2024-06-29T15:57:33.357223+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354119+0000","last_change":"2024-06-29T15:57:22.301166+0000","last_active":"2024-06-29T15:57:33.354119+0000","last_peered":"2024-06-29T15:57:33.354119+0000","last_clean":"2024-06-29T15:57:33.354119+0000","last_became_active":"2024-06-29T15:57:22.300676+0000","last_became_peered":"2024-06-29T15:57:22.300676+0000","last_unstale":"2024-06-29T15:57:33.354119+0000","last_undegraded":"2024-06-29T15:57:33.354119+0000","last_fullsized":"2024-06-29T15:57:33.354119+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363490+0000","last_change":"2024-06-29T15:57:33.362184+0000","last_active":"2024-06-29T15:57:33.363490+0000","last_peered":"2024-06-29T15:57:33.363490+0000","last_clean":"2024-06-29T15:57:33.363490+0000","last_became_active":"2024-06-29T15:57:33.361905+0000","last_became_peered":"2024-06-29T15:57:33.361905+0000","last_unstale":"2024-06-29T15:57:33.363490+0000","last_undegraded":"2024-06-29T15:57:33.363490+0000","last_fullsized":"2024-06-29T15:57:33.363490+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:04.833173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349702+0000","last_change":"2024-06-29T15:57:33.349702+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349702+0000","last_undegraded":"2024-06-29T15:57:33.349702+0000","last_fullsized":"2024-06-29T15:57:33.349702+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357105+0000","last_change":"2024-06-29T15:57:21.280224+0000","last_active":"2024-06-29T15:57:33.357105+0000","last_peered":"2024-06-29T15:57:33.357105+0000","last_clean":"2024-06-29T15:57:33.357105+0000","last_became_active":"2024-06-29T15:57:21.279899+0000","last_became_peered":"2024-06-29T15:57:21.279899+0000","last_unstale":"2024-06-29T15:57:33.357105+0000","last_undegraded":"2024-06-29T15:57:33.357105+0000","last_fullsized":"2024-06-29T15:57:33.357105+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355301+0000","last_change":"2024-06-29T15:57:22.296472+0000","last_active":"2024-06-29T15:57:33.355301+0000","last_peered":"2024-06-29T15:57:33.355301+0000","last_clean":"2024-06-29T15:57:33.355301+0000","last_became_active":"2024-06-29T15:57:22.296130+0000","last_became_peered":"2024-06-29T15:57:22.296130+0000","last_unstale":"2024-06-29T15:57:33.355301+0000","last_undegraded":"2024-06-29T15:57:33.355301+0000","last_fullsized":"2024-06-29T15:57:33.355301+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368843+0000","last_change":"2024-06-29T15:57:33.367380+0000","last_active":"2024-06-29T15:57:33.368843+0000","last_peered":"2024-06-29T15:57:33.368843+0000","last_clean":"2024-06-29T15:57:33.368843+0000","last_became_active":"2024-06-29T15:57:33.367054+0000","last_became_peered":"2024-06-29T15:57:33.367054+0000","last_unstale":"2024-06-29T15:57:33.368843+0000","last_undegraded":"2024-06-29T15:57:33.368843+0000","last_fullsized":"2024-06-29T15:57:33.368843+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:29.345255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357030+0000","last_change":"2024-06-29T15:57:33.357030+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357030+0000","last_undegraded":"2024-06-29T15:57:33.357030+0000","last_fullsized":"2024-06-29T15:57:33.357030+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362303+0000","last_change":"2024-06-29T15:57:21.276972+0000","last_active":"2024-06-29T15:57:33.362303+0000","last_peered":"2024-06-29T15:57:33.362303+0000","last_clean":"2024-06-29T15:57:33.362303+0000","last_became_active":"2024-06-29T15:57:21.276647+0000","last_became_peered":"2024-06-29T15:57:21.276647+0000","last_unstale":"2024-06-29T15:57:33.362303+0000","last_undegraded":"2024-06-29T15:57:33.362303+0000","last_fullsized":"2024-06-29T15:57:33.362303+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:24.871281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355716+0000","last_change":"2024-06-29T15:57:22.303802+0000","last_active":"2024-06-29T15:57:33.355716+0000","last_peered":"2024-06-29T15:57:33.355716+0000","last_clean":"2024-06-29T15:57:33.355716+0000","last_became_active":"2024-06-29T15:57:22.303511+0000","last_became_peered":"2024-06-29T15:57:22.303511+0000","last_unstale":"2024-06-29T15:57:33.355716+0000","last_undegraded":"2024-06-29T15:57:33.355716+0000","last_fullsized":"2024-06-29T15:57:33.355716+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:10.536799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371208+0000","last_change":"2024-06-29T15:57:33.367162+0000","last_active":"2024-06-29T15:57:33.371208+0000","last_peered":"2024-06-29T15:57:33.371208+0000","last_clean":"2024-06-29T15:57:33.371208+0000","last_became_active":"2024-06-29T15:57:33.366816+0000","last_became_peered":"2024-06-29T15:57:33.366816+0000","last_unstale":"2024-06-29T15:57:33.371208+0000","last_undegraded":"2024-06-29T15:57:33.371208+0000","last_fullsized":"2024-06-29T15:57:33.371208+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:41.683812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.363662+0000","last_change":"2024-06-29T15:57:33.363662+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.363662+0000","last_undegraded":"2024-06-29T15:57:33.363662+0000","last_fullsized":"2024-06-29T15:57:33.363662+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358022+0000","last_change":"2024-06-29T15:57:21.277662+0000","last_active":"2024-06-29T15:57:33.358022+0000","last_peered":"2024-06-29T15:57:33.358022+0000","last_clean":"2024-06-29T15:57:33.358022+0000","last_became_active":"2024-06-29T15:57:21.277335+0000","last_became_peered":"2024-06-29T15:57:21.277335+0000","last_unstale":"2024-06-29T15:57:33.358022+0000","last_undegraded":"2024-06-29T15:57:33.358022+0000","last_fullsized":"2024-06-29T15:57:33.358022+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364773+0000","last_change":"2024-06-29T15:57:22.298914+0000","last_active":"2024-06-29T15:57:33.364773+0000","last_peered":"2024-06-29T15:57:33.364773+0000","last_clean":"2024-06-29T15:57:33.364773+0000","last_became_active":"2024-06-29T15:57:22.298614+0000","last_became_peered":"2024-06-29T15:57:22.298614+0000","last_unstale":"2024-06-29T15:57:33.364773+0000","last_undegraded":"2024-06-29T15:57:33.364773+0000","last_fullsized":"2024-06-29T15:57:33.364773+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367924+0000","last_change":"2024-06-29T15:57:33.367133+0000","last_active":"2024-06-29T15:57:33.367924+0000","last_peered":"2024-06-29T15:57:33.367924+0000","last_clean":"2024-06-29T15:57:33.367924+0000","last_became_active":"2024-06-29T15:57:33.366782+0000","last_became_peered":"2024-06-29T15:57:33.366782+0000","last_unstale":"2024-06-29T15:57:33.367924+0000","last_undegraded":"2024-06-29T15:57:33.367924+0000","last_fullsized":"2024-06-29T15:57:33.367924+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351295+0000","last_change":"2024-06-29T15:57:33.351295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351295+0000","last_undegraded":"2024-06-29T15:57:33.351295+0000","last_fullsized":"2024-06-29T15:57:33.351295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355165+0000","last_change":"2024-06-29T15:57:21.278011+0000","last_active":"2024-06-29T15:57:33.355165+0000","last_peered":"2024-06-29T15:57:33.355165+0000","last_clean":"2024-06-29T15:57:33.355165+0000","last_became_active":"2024-06-29T15:57:21.276648+0000","last_became_peered":"2024-06-29T15:57:21.276648+0000","last_unstale":"2024-06-29T15:57:33.355165+0000","last_undegraded":"2024-06-29T15:57:33.355165+0000","last_fullsized":"2024-06-29T15:57:33.355165+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354181+0000","last_change":"2024-06-29T15:57:22.300220+0000","last_active":"2024-06-29T15:57:33.354181+0000","last_peered":"2024-06-29T15:57:33.354181+0000","last_clean":"2024-06-29T15:57:33.354181+0000","last_became_active":"2024-06-29T15:57:22.299880+0000","last_became_peered":"2024-06-29T15:57:22.299880+0000","last_unstale":"2024-06-29T15:57:33.354181+0000","last_undegraded":"2024-06-29T15:57:33.354181+0000","last_fullsized":"2024-06-29T15:57:33.354181+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368552+0000","last_change":"2024-06-29T15:57:33.367636+0000","last_active":"2024-06-29T15:57:33.368552+0000","last_peered":"2024-06-29T15:57:33.368552+0000","last_clean":"2024-06-29T15:57:33.368552+0000","last_became_active":"2024-06-29T15:57:33.367321+0000","last_became_peered":"2024-06-29T15:57:33.367321+0000","last_unstale":"2024-06-29T15:57:33.368552+0000","last_undegraded":"2024-06-29T15:57:33.368552+0000","last_fullsized":"2024-06-29T15:57:33.368552+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:28.846172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353695+0000","last_change":"2024-06-29T15:57:33.353695+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353695+0000","last_undegraded":"2024-06-29T15:57:33.353695+0000","last_fullsized":"2024-06-29T15:57:33.353695+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357156+0000","last_change":"2024-06-29T15:57:21.284982+0000","last_active":"2024-06-29T15:57:33.357156+0000","last_peered":"2024-06-29T15:57:33.357156+0000","last_clean":"2024-06-29T15:57:33.357156+0000","last_became_active":"2024-06-29T15:57:21.284684+0000","last_became_peered":"2024-06-29T15:57:21.284684+0000","last_unstale":"2024-06-29T15:57:33.357156+0000","last_undegraded":"2024-06-29T15:57:33.357156+0000","last_fullsized":"2024-06-29T15:57:33.357156+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354906+0000","last_change":"2024-06-29T15:57:22.298744+0000","last_active":"2024-06-29T15:57:33.354906+0000","last_peered":"2024-06-29T15:57:33.354906+0000","last_clean":"2024-06-29T15:57:33.354906+0000","last_became_active":"2024-06-29T15:57:22.296925+0000","last_became_peered":"2024-06-29T15:57:22.296925+0000","last_unstale":"2024-06-29T15:57:33.354906+0000","last_undegraded":"2024-06-29T15:57:33.354906+0000","last_fullsized":"2024-06-29T15:57:33.354906+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:40.578182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371721+0000","last_change":"2024-06-29T15:57:33.368936+0000","last_active":"2024-06-29T15:57:33.371721+0000","last_peered":"2024-06-29T15:57:33.371721+0000","last_clean":"2024-06-29T15:57:33.371721+0000","last_became_active":"2024-06-29T15:57:33.368600+0000","last_became_peered":"2024-06-29T15:57:33.368600+0000","last_unstale":"2024-06-29T15:57:33.371721+0000","last_undegraded":"2024-06-29T15:57:33.371721+0000","last_fullsized":"2024-06-29T15:57:33.371721+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:22.830532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.361651+0000","last_change":"2024-06-29T15:57:33.361651+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.361651+0000","last_undegraded":"2024-06-29T15:57:33.361651+0000","last_fullsized":"2024-06-29T15:57:33.361651+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355165+0000","last_change":"2024-06-29T15:57:21.284780+0000","last_active":"2024-06-29T15:57:33.355165+0000","last_peered":"2024-06-29T15:57:33.355165+0000","last_clean":"2024-06-29T15:57:33.355165+0000","last_became_active":"2024-06-29T15:57:21.284273+0000","last_became_peered":"2024-06-29T15:57:21.284273+0000","last_unstale":"2024-06-29T15:57:33.355165+0000","last_undegraded":"2024-06-29T15:57:33.355165+0000","last_fullsized":"2024-06-29T15:57:33.355165+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:16.085990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356696+0000","last_change":"2024-06-29T15:57:22.291304+0000","last_active":"2024-06-29T15:57:33.356696+0000","last_peered":"2024-06-29T15:57:33.356696+0000","last_clean":"2024-06-29T15:57:33.356696+0000","last_became_active":"2024-06-29T15:57:22.290983+0000","last_became_peered":"2024-06-29T15:57:22.290983+0000","last_unstale":"2024-06-29T15:57:33.356696+0000","last_undegraded":"2024-06-29T15:57:33.356696+0000","last_fullsized":"2024-06-29T15:57:33.356696+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368254+0000","last_change":"2024-06-29T15:57:33.366553+0000","last_active":"2024-06-29T15:57:33.368254+0000","last_peered":"2024-06-29T15:57:33.368254+0000","last_clean":"2024-06-29T15:57:33.368254+0000","last_became_active":"2024-06-29T15:57:33.366230+0000","last_became_peered":"2024-06-29T15:57:33.366230+0000","last_unstale":"2024-06-29T15:57:33.368254+0000","last_undegraded":"2024-06-29T15:57:33.368254+0000","last_fullsized":"2024-06-29T15:57:33.368254+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:17:11.929406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358755+0000","last_change":"2024-06-29T15:57:33.358755+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358755+0000","last_undegraded":"2024-06-29T15:57:33.358755+0000","last_fullsized":"2024-06-29T15:57:33.358755+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348756+0000","last_change":"2024-06-29T15:57:21.284009+0000","last_active":"2024-06-29T15:57:33.348756+0000","last_peered":"2024-06-29T15:57:33.348756+0000","last_clean":"2024-06-29T15:57:33.348756+0000","last_became_active":"2024-06-29T15:57:21.283679+0000","last_became_peered":"2024-06-29T15:57:21.283679+0000","last_unstale":"2024-06-29T15:57:33.348756+0000","last_undegraded":"2024-06-29T15:57:33.348756+0000","last_fullsized":"2024-06-29T15:57:33.348756+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:20.284468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348277+0000","last_change":"2024-06-29T15:57:22.288958+0000","last_active":"2024-06-29T15:57:33.348277+0000","last_peered":"2024-06-29T15:57:33.348277+0000","last_clean":"2024-06-29T15:57:33.348277+0000","last_became_active":"2024-06-29T15:57:22.288621+0000","last_became_peered":"2024-06-29T15:57:22.288621+0000","last_unstale":"2024-06-29T15:57:33.348277+0000","last_undegraded":"2024-06-29T15:57:33.348277+0000","last_fullsized":"2024-06-29T15:57:33.348277+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:31.720751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371349+0000","last_change":"2024-06-29T15:57:33.370560+0000","last_active":"2024-06-29T15:57:33.371349+0000","last_peered":"2024-06-29T15:57:33.371349+0000","last_clean":"2024-06-29T15:57:33.371349+0000","last_became_active":"2024-06-29T15:57:33.370206+0000","last_became_peered":"2024-06-29T15:57:33.370206+0000","last_unstale":"2024-06-29T15:57:33.371349+0000","last_undegraded":"2024-06-29T15:57:33.371349+0000","last_fullsized":"2024-06-29T15:57:33.371349+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:12:47.692442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.361054+0000","last_change":"2024-06-29T15:57:33.361054+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.361054+0000","last_undegraded":"2024-06-29T15:57:33.361054+0000","last_fullsized":"2024-06-29T15:57:33.361054+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357091+0000","last_change":"2024-06-29T15:57:21.282111+0000","last_active":"2024-06-29T15:57:33.357091+0000","last_peered":"2024-06-29T15:57:33.357091+0000","last_clean":"2024-06-29T15:57:33.357091+0000","last_became_active":"2024-06-29T15:57:21.281761+0000","last_became_peered":"2024-06-29T15:57:21.281761+0000","last_unstale":"2024-06-29T15:57:33.357091+0000","last_undegraded":"2024-06-29T15:57:33.357091+0000","last_fullsized":"2024-06-29T15:57:33.357091+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348792+0000","last_change":"2024-06-29T15:57:22.305806+0000","last_active":"2024-06-29T15:57:33.348792+0000","last_peered":"2024-06-29T15:57:33.348792+0000","last_clean":"2024-06-29T15:57:33.348792+0000","last_became_active":"2024-06-29T15:57:22.296276+0000","last_became_peered":"2024-06-29T15:57:22.296276+0000","last_unstale":"2024-06-29T15:57:33.348792+0000","last_undegraded":"2024-06-29T15:57:33.348792+0000","last_fullsized":"2024-06-29T15:57:33.348792+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:43:51.754223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.369952+0000","last_change":"2024-06-29T15:57:33.367821+0000","last_active":"2024-06-29T15:57:33.369952+0000","last_peered":"2024-06-29T15:57:33.369952+0000","last_clean":"2024-06-29T15:57:33.369952+0000","last_became_active":"2024-06-29T15:57:33.367494+0000","last_became_peered":"2024-06-29T15:57:33.367494+0000","last_unstale":"2024-06-29T15:57:33.369952+0000","last_undegraded":"2024-06-29T15:57:33.369952+0000","last_fullsized":"2024-06-29T15:57:33.369952+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:46.165772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352248+0000","last_change":"2024-06-29T15:57:33.352248+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352248+0000","last_undegraded":"2024-06-29T15:57:33.352248+0000","last_fullsized":"2024-06-29T15:57:33.352248+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.349023+0000","last_change":"2024-06-29T15:57:21.279139+0000","last_active":"2024-06-29T15:57:33.349023+0000","last_peered":"2024-06-29T15:57:33.349023+0000","last_clean":"2024-06-29T15:57:33.349023+0000","last_became_active":"2024-06-29T15:57:21.278488+0000","last_became_peered":"2024-06-29T15:57:21.278488+0000","last_unstale":"2024-06-29T15:57:33.349023+0000","last_undegraded":"2024-06-29T15:57:33.349023+0000","last_fullsized":"2024-06-29T15:57:33.349023+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:54.126338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356194+0000","last_change":"2024-06-29T15:57:22.297275+0000","last_active":"2024-06-29T15:57:33.356194+0000","last_peered":"2024-06-29T15:57:33.356194+0000","last_clean":"2024-06-29T15:57:33.356194+0000","last_became_active":"2024-06-29T15:57:22.296984+0000","last_became_peered":"2024-06-29T15:57:22.296984+0000","last_unstale":"2024-06-29T15:57:33.356194+0000","last_undegraded":"2024-06-29T15:57:33.356194+0000","last_fullsized":"2024-06-29T15:57:33.356194+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371810+0000","last_change":"2024-06-29T15:57:33.369358+0000","last_active":"2024-06-29T15:57:33.371810+0000","last_peered":"2024-06-29T15:57:33.371810+0000","last_clean":"2024-06-29T15:57:33.371810+0000","last_became_active":"2024-06-29T15:57:33.369052+0000","last_became_peered":"2024-06-29T15:57:33.369052+0000","last_unstale":"2024-06-29T15:57:33.371810+0000","last_undegraded":"2024-06-29T15:57:33.371810+0000","last_fullsized":"2024-06-29T15:57:33.371810+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:29.345255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352493+0000","last_change":"2024-06-29T15:57:33.352493+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352493+0000","last_undegraded":"2024-06-29T15:57:33.352493+0000","last_fullsized":"2024-06-29T15:57:33.352493+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356473+0000","last_change":"2024-06-29T15:57:21.278476+0000","last_active":"2024-06-29T15:57:33.356473+0000","last_peered":"2024-06-29T15:57:33.356473+0000","last_clean":"2024-06-29T15:57:33.356473+0000","last_became_active":"2024-06-29T15:57:21.278150+0000","last_became_peered":"2024-06-29T15:57:21.278150+0000","last_unstale":"2024-06-29T15:57:33.356473+0000","last_undegraded":"2024-06-29T15:57:33.356473+0000","last_fullsized":"2024-06-29T15:57:33.356473+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:33.991332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355923+0000","last_change":"2024-06-29T15:57:22.291360+0000","last_active":"2024-06-29T15:57:33.355923+0000","last_peered":"2024-06-29T15:57:33.355923+0000","last_clean":"2024-06-29T15:57:33.355923+0000","last_became_active":"2024-06-29T15:57:22.290801+0000","last_became_peered":"2024-06-29T15:57:22.290801+0000","last_unstale":"2024-06-29T15:57:33.355923+0000","last_undegraded":"2024-06-29T15:57:33.355923+0000","last_fullsized":"2024-06-29T15:57:33.355923+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371558+0000","last_change":"2024-06-29T15:57:33.367933+0000","last_active":"2024-06-29T15:57:33.371558+0000","last_peered":"2024-06-29T15:57:33.371558+0000","last_clean":"2024-06-29T15:57:33.371558+0000","last_became_active":"2024-06-29T15:57:33.367630+0000","last_became_peered":"2024-06-29T15:57:33.367630+0000","last_unstale":"2024-06-29T15:57:33.371558+0000","last_undegraded":"2024-06-29T15:57:33.371558+0000","last_fullsized":"2024-06-29T15:57:33.371558+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:01.393531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358484+0000","last_change":"2024-06-29T15:57:33.358484+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358484+0000","last_undegraded":"2024-06-29T15:57:33.358484+0000","last_fullsized":"2024-06-29T15:57:33.358484+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358778+0000","last_change":"2024-06-29T15:57:21.273304+0000","last_active":"2024-06-29T15:57:33.358778+0000","last_peered":"2024-06-29T15:57:33.358778+0000","last_clean":"2024-06-29T15:57:33.358778+0000","last_became_active":"2024-06-29T15:57:21.272972+0000","last_became_peered":"2024-06-29T15:57:21.272972+0000","last_unstale":"2024-06-29T15:57:33.358778+0000","last_undegraded":"2024-06-29T15:57:33.358778+0000","last_fullsized":"2024-06-29T15:57:33.358778+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356200+0000","last_change":"2024-06-29T15:57:22.300916+0000","last_active":"2024-06-29T15:57:33.356200+0000","last_peered":"2024-06-29T15:57:33.356200+0000","last_clean":"2024-06-29T15:57:33.356200+0000","last_became_active":"2024-06-29T15:57:22.300620+0000","last_became_peered":"2024-06-29T15:57:22.300620+0000","last_unstale":"2024-06-29T15:57:33.356200+0000","last_undegraded":"2024-06-29T15:57:33.356200+0000","last_fullsized":"2024-06-29T15:57:33.356200+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:41.172455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368260+0000","last_change":"2024-06-29T15:57:33.367013+0000","last_active":"2024-06-29T15:57:33.368260+0000","last_peered":"2024-06-29T15:57:33.368260+0000","last_clean":"2024-06-29T15:57:33.368260+0000","last_became_active":"2024-06-29T15:57:33.366664+0000","last_became_peered":"2024-06-29T15:57:33.366664+0000","last_unstale":"2024-06-29T15:57:33.368260+0000","last_undegraded":"2024-06-29T15:57:33.368260+0000","last_fullsized":"2024-06-29T15:57:33.368260+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:22.830532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357163+0000","last_change":"2024-06-29T15:57:33.357163+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357163+0000","last_undegraded":"2024-06-29T15:57:33.357163+0000","last_fullsized":"2024-06-29T15:57:33.357163+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"212'8","reported_seq":26,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359054+0000","last_change":"2024-06-29T15:57:21.283745+0000","last_active":"2024-06-29T15:57:33.359054+0000","last_peered":"2024-06-29T15:57:33.359054+0000","last_clean":"2024-06-29T15:57:33.359054+0000","last_became_active":"2024-06-29T15:57:21.283403+0000","last_became_peered":"2024-06-29T15:57:21.283403+0000","last_unstale":"2024-06-29T15:57:33.359054+0000","last_undegraded":"2024-06-29T15:57:33.359054+0000","last_fullsized":"2024-06-29T15:57:33.359054+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357048+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:33.357048+0000","last_peered":"2024-06-29T15:57:33.357048+0000","last_clean":"2024-06-29T15:57:33.357048+0000","last_became_active":"2024-06-29T15:57:22.301832+0000","last_became_peered":"2024-06-29T15:57:22.301832+0000","last_unstale":"2024-06-29T15:57:33.357048+0000","last_undegraded":"2024-06-29T15:57:33.357048+0000","last_fullsized":"2024-06-29T15:57:33.357048+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367778+0000","last_change":"2024-06-29T15:57:33.366032+0000","last_active":"2024-06-29T15:57:33.367778+0000","last_peered":"2024-06-29T15:57:33.367778+0000","last_clean":"2024-06-29T15:57:33.367778+0000","last_became_active":"2024-06-29T15:57:33.365731+0000","last_became_peered":"2024-06-29T15:57:33.365731+0000","last_unstale":"2024-06-29T15:57:33.367778+0000","last_undegraded":"2024-06-29T15:57:33.367778+0000","last_fullsized":"2024-06-29T15:57:33.367778+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:58.462755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349766+0000","last_change":"2024-06-29T15:57:33.349766+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349766+0000","last_undegraded":"2024-06-29T15:57:33.349766+0000","last_fullsized":"2024-06-29T15:57:33.349766+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348560+0000","last_change":"2024-06-29T15:57:21.276731+0000","last_active":"2024-06-29T15:57:33.348560+0000","last_peered":"2024-06-29T15:57:33.348560+0000","last_clean":"2024-06-29T15:57:33.348560+0000","last_became_active":"2024-06-29T15:57:21.276408+0000","last_became_peered":"2024-06-29T15:57:21.276408+0000","last_unstale":"2024-06-29T15:57:33.348560+0000","last_undegraded":"2024-06-29T15:57:33.348560+0000","last_fullsized":"2024-06-29T15:57:33.348560+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:48.911239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358795+0000","last_change":"2024-06-29T15:57:22.299631+0000","last_active":"2024-06-29T15:57:33.358795+0000","last_peered":"2024-06-29T15:57:33.358795+0000","last_clean":"2024-06-29T15:57:33.358795+0000","last_became_active":"2024-06-29T15:57:22.299285+0000","last_became_peered":"2024-06-29T15:57:22.299285+0000","last_unstale":"2024-06-29T15:57:33.358795+0000","last_undegraded":"2024-06-29T15:57:33.358795+0000","last_fullsized":"2024-06-29T15:57:33.358795+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370910+0000","last_change":"2024-06-29T15:57:33.369952+0000","last_active":"2024-06-29T15:57:33.370910+0000","last_peered":"2024-06-29T15:57:33.370910+0000","last_clean":"2024-06-29T15:57:33.370910+0000","last_became_active":"2024-06-29T15:57:33.369598+0000","last_became_peered":"2024-06-29T15:57:33.369598+0000","last_unstale":"2024-06-29T15:57:33.370910+0000","last_undegraded":"2024-06-29T15:57:33.370910+0000","last_fullsized":"2024-06-29T15:57:33.370910+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:24.974909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358887+0000","last_change":"2024-06-29T15:57:33.358887+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358887+0000","last_undegraded":"2024-06-29T15:57:33.358887+0000","last_fullsized":"2024-06-29T15:57:33.358887+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354289+0000","last_change":"2024-06-29T15:57:21.282872+0000","last_active":"2024-06-29T15:57:33.354289+0000","last_peered":"2024-06-29T15:57:33.354289+0000","last_clean":"2024-06-29T15:57:33.354289+0000","last_became_active":"2024-06-29T15:57:21.282549+0000","last_became_peered":"2024-06-29T15:57:21.282549+0000","last_unstale":"2024-06-29T15:57:33.354289+0000","last_undegraded":"2024-06-29T15:57:33.354289+0000","last_fullsized":"2024-06-29T15:57:33.354289+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348643+0000","last_change":"2024-06-29T15:57:22.294011+0000","last_active":"2024-06-29T15:57:33.348643+0000","last_peered":"2024-06-29T15:57:33.348643+0000","last_clean":"2024-06-29T15:57:33.348643+0000","last_became_active":"2024-06-29T15:57:22.293721+0000","last_became_peered":"2024-06-29T15:57:22.293721+0000","last_unstale":"2024-06-29T15:57:33.348643+0000","last_undegraded":"2024-06-29T15:57:33.348643+0000","last_fullsized":"2024-06-29T15:57:33.348643+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:57.337708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362703+0000","last_change":"2024-06-29T15:57:33.361883+0000","last_active":"2024-06-29T15:57:33.362703+0000","last_peered":"2024-06-29T15:57:33.362703+0000","last_clean":"2024-06-29T15:57:33.362703+0000","last_became_active":"2024-06-29T15:57:33.361569+0000","last_became_peered":"2024-06-29T15:57:33.361569+0000","last_unstale":"2024-06-29T15:57:33.362703+0000","last_undegraded":"2024-06-29T15:57:33.362703+0000","last_fullsized":"2024-06-29T15:57:33.362703+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:08.554869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351919+0000","last_change":"2024-06-29T15:57:33.351919+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351919+0000","last_undegraded":"2024-06-29T15:57:33.351919+0000","last_fullsized":"2024-06-29T15:57:33.351919+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353877+0000","last_change":"2024-06-29T15:57:21.281309+0000","last_active":"2024-06-29T15:57:33.353877+0000","last_peered":"2024-06-29T15:57:33.353877+0000","last_clean":"2024-06-29T15:57:33.353877+0000","last_became_active":"2024-06-29T15:57:21.280944+0000","last_became_peered":"2024-06-29T15:57:21.280944+0000","last_unstale":"2024-06-29T15:57:33.353877+0000","last_undegraded":"2024-06-29T15:57:33.353877+0000","last_fullsized":"2024-06-29T15:57:33.353877+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356544+0000","last_change":"2024-06-29T15:57:22.291164+0000","last_active":"2024-06-29T15:57:33.356544+0000","last_peered":"2024-06-29T15:57:33.356544+0000","last_clean":"2024-06-29T15:57:33.356544+0000","last_became_active":"2024-06-29T15:57:22.290897+0000","last_became_peered":"2024-06-29T15:57:22.290897+0000","last_unstale":"2024-06-29T15:57:33.356544+0000","last_undegraded":"2024-06-29T15:57:33.356544+0000","last_fullsized":"2024-06-29T15:57:33.356544+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364704+0000","last_change":"2024-06-29T15:57:33.364026+0000","last_active":"2024-06-29T15:57:33.364704+0000","last_peered":"2024-06-29T15:57:33.364704+0000","last_clean":"2024-06-29T15:57:33.364704+0000","last_became_active":"2024-06-29T15:57:33.363658+0000","last_became_peered":"2024-06-29T15:57:33.363658+0000","last_unstale":"2024-06-29T15:57:33.364704+0000","last_undegraded":"2024-06-29T15:57:33.364704+0000","last_fullsized":"2024-06-29T15:57:33.364704+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:41.322192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359617+0000","last_change":"2024-06-29T15:57:33.359617+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359617+0000","last_undegraded":"2024-06-29T15:57:33.359617+0000","last_fullsized":"2024-06-29T15:57:33.359617+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"214'8","reported_seq":26,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365217+0000","last_change":"2024-06-29T15:57:21.278161+0000","last_active":"2024-06-29T15:57:33.365217+0000","last_peered":"2024-06-29T15:57:33.365217+0000","last_clean":"2024-06-29T15:57:33.365217+0000","last_became_active":"2024-06-29T15:57:21.277794+0000","last_became_peered":"2024-06-29T15:57:21.277794+0000","last_unstale":"2024-06-29T15:57:33.365217+0000","last_undegraded":"2024-06-29T15:57:33.365217+0000","last_fullsized":"2024-06-29T15:57:33.365217+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:56.330168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.352292+0000","last_change":"2024-06-29T15:57:22.302666+0000","last_active":"2024-06-29T15:57:33.352292+0000","last_peered":"2024-06-29T15:57:33.352292+0000","last_clean":"2024-06-29T15:57:33.352292+0000","last_became_active":"2024-06-29T15:57:22.302254+0000","last_became_peered":"2024-06-29T15:57:22.302254+0000","last_unstale":"2024-06-29T15:57:33.352292+0000","last_undegraded":"2024-06-29T15:57:33.352292+0000","last_fullsized":"2024-06-29T15:57:33.352292+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363467+0000","last_change":"2024-06-29T15:57:33.361374+0000","last_active":"2024-06-29T15:57:33.363467+0000","last_peered":"2024-06-29T15:57:33.363467+0000","last_clean":"2024-06-29T15:57:33.363467+0000","last_became_active":"2024-06-29T15:57:33.361002+0000","last_became_peered":"2024-06-29T15:57:33.361002+0000","last_unstale":"2024-06-29T15:57:33.363467+0000","last_undegraded":"2024-06-29T15:57:33.363467+0000","last_fullsized":"2024-06-29T15:57:33.363467+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:47.234190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353505+0000","last_change":"2024-06-29T15:57:33.353505+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353505+0000","last_undegraded":"2024-06-29T15:57:33.353505+0000","last_fullsized":"2024-06-29T15:57:33.353505+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"212'3","reported_seq":21,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348900+0000","last_change":"2024-06-29T15:57:21.283791+0000","last_active":"2024-06-29T15:57:33.348900+0000","last_peered":"2024-06-29T15:57:33.348900+0000","last_clean":"2024-06-29T15:57:33.348900+0000","last_became_active":"2024-06-29T15:57:21.283458+0000","last_became_peered":"2024-06-29T15:57:21.283458+0000","last_unstale":"2024-06-29T15:57:33.348900+0000","last_undegraded":"2024-06-29T15:57:33.348900+0000","last_fullsized":"2024-06-29T15:57:33.348900+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:54.365881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361117+0000","last_change":"2024-06-29T15:57:22.295260+0000","last_active":"2024-06-29T15:57:33.361117+0000","last_peered":"2024-06-29T15:57:33.361117+0000","last_clean":"2024-06-29T15:57:33.361117+0000","last_became_active":"2024-06-29T15:57:22.294952+0000","last_became_peered":"2024-06-29T15:57:22.294952+0000","last_unstale":"2024-06-29T15:57:33.361117+0000","last_undegraded":"2024-06-29T15:57:33.361117+0000","last_fullsized":"2024-06-29T15:57:33.361117+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:56.473182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370910+0000","last_change":"2024-06-29T15:57:33.370460+0000","last_active":"2024-06-29T15:57:33.370910+0000","last_peered":"2024-06-29T15:57:33.370910+0000","last_clean":"2024-06-29T15:57:33.370910+0000","last_became_active":"2024-06-29T15:57:33.370183+0000","last_became_peered":"2024-06-29T15:57:33.370183+0000","last_unstale":"2024-06-29T15:57:33.370910+0000","last_undegraded":"2024-06-29T15:57:33.370910+0000","last_fullsized":"2024-06-29T15:57:33.370910+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:06.322225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352295+0000","last_change":"2024-06-29T15:57:33.352295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352295+0000","last_undegraded":"2024-06-29T15:57:33.352295+0000","last_fullsized":"2024-06-29T15:57:33.352295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354550+0000","last_change":"2024-06-29T15:57:21.275926+0000","last_active":"2024-06-29T15:57:33.354550+0000","last_peered":"2024-06-29T15:57:33.354550+0000","last_clean":"2024-06-29T15:57:33.354550+0000","last_became_active":"2024-06-29T15:57:21.275593+0000","last_became_peered":"2024-06-29T15:57:21.275593+0000","last_unstale":"2024-06-29T15:57:33.354550+0000","last_undegraded":"2024-06-29T15:57:33.354550+0000","last_fullsized":"2024-06-29T15:57:33.354550+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357897+0000","last_change":"2024-06-29T15:57:22.298454+0000","last_active":"2024-06-29T15:57:33.357897+0000","last_peered":"2024-06-29T15:57:33.357897+0000","last_clean":"2024-06-29T15:57:33.357897+0000","last_became_active":"2024-06-29T15:57:22.298162+0000","last_became_peered":"2024-06-29T15:57:22.298162+0000","last_unstale":"2024-06-29T15:57:33.357897+0000","last_undegraded":"2024-06-29T15:57:33.357897+0000","last_fullsized":"2024-06-29T15:57:33.357897+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363344+0000","last_change":"2024-06-29T15:57:33.362039+0000","last_active":"2024-06-29T15:57:33.363344+0000","last_peered":"2024-06-29T15:57:33.363344+0000","last_clean":"2024-06-29T15:57:33.363344+0000","last_became_active":"2024-06-29T15:57:33.361665+0000","last_became_peered":"2024-06-29T15:57:33.361665+0000","last_unstale":"2024-06-29T15:57:33.363344+0000","last_undegraded":"2024-06-29T15:57:33.363344+0000","last_fullsized":"2024-06-29T15:57:33.363344+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:41.322192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358599+0000","last_change":"2024-06-29T15:57:33.358599+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358599+0000","last_undegraded":"2024-06-29T15:57:33.358599+0000","last_fullsized":"2024-06-29T15:57:33.358599+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358654+0000","last_change":"2024-06-29T15:57:21.272019+0000","last_active":"2024-06-29T15:57:33.358654+0000","last_peered":"2024-06-29T15:57:33.358654+0000","last_clean":"2024-06-29T15:57:33.358654+0000","last_became_active":"2024-06-29T15:57:21.271715+0000","last_became_peered":"2024-06-29T15:57:21.271715+0000","last_unstale":"2024-06-29T15:57:33.358654+0000","last_undegraded":"2024-06-29T15:57:33.358654+0000","last_fullsized":"2024-06-29T15:57:33.358654+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362094+0000","last_change":"2024-06-29T15:57:22.303487+0000","last_active":"2024-06-29T15:57:33.362094+0000","last_peered":"2024-06-29T15:57:33.362094+0000","last_clean":"2024-06-29T15:57:33.362094+0000","last_became_active":"2024-06-29T15:57:22.302933+0000","last_became_peered":"2024-06-29T15:57:22.302933+0000","last_unstale":"2024-06-29T15:57:33.362094+0000","last_undegraded":"2024-06-29T15:57:33.362094+0000","last_fullsized":"2024-06-29T15:57:33.362094+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:28.493187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362444+0000","last_change":"2024-06-29T15:57:33.361560+0000","last_active":"2024-06-29T15:57:33.362444+0000","last_peered":"2024-06-29T15:57:33.362444+0000","last_clean":"2024-06-29T15:57:33.362444+0000","last_became_active":"2024-06-29T15:57:33.361270+0000","last_became_peered":"2024-06-29T15:57:33.361270+0000","last_unstale":"2024-06-29T15:57:33.362444+0000","last_undegraded":"2024-06-29T15:57:33.362444+0000","last_fullsized":"2024-06-29T15:57:33.362444+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:51.198488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356450+0000","last_change":"2024-06-29T15:57:33.356450+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356450+0000","last_undegraded":"2024-06-29T15:57:33.356450+0000","last_fullsized":"2024-06-29T15:57:33.356450+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364019+0000","last_change":"2024-06-29T15:57:33.362045+0000","last_active":"2024-06-29T15:57:33.364019+0000","last_peered":"2024-06-29T15:57:33.364019+0000","last_clean":"2024-06-29T15:57:33.364019+0000","last_became_active":"2024-06-29T15:57:33.361647+0000","last_became_peered":"2024-06-29T15:57:33.361647+0000","last_unstale":"2024-06-29T15:57:33.364019+0000","last_undegraded":"2024-06-29T15:57:33.364019+0000","last_fullsized":"2024-06-29T15:57:33.364019+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:51.198488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353878+0000","last_change":"2024-06-29T15:57:21.277245+0000","last_active":"2024-06-29T15:57:33.353878+0000","last_peered":"2024-06-29T15:57:33.353878+0000","last_clean":"2024-06-29T15:57:33.353878+0000","last_became_active":"2024-06-29T15:57:21.276931+0000","last_became_peered":"2024-06-29T15:57:21.276931+0000","last_unstale":"2024-06-29T15:57:33.353878+0000","last_undegraded":"2024-06-29T15:57:33.353878+0000","last_fullsized":"2024-06-29T15:57:33.353878+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361080+0000","last_change":"2024-06-29T15:57:22.299457+0000","last_active":"2024-06-29T15:57:33.361080+0000","last_peered":"2024-06-29T15:57:33.361080+0000","last_clean":"2024-06-29T15:57:33.361080+0000","last_became_active":"2024-06-29T15:57:22.299157+0000","last_became_peered":"2024-06-29T15:57:22.299157+0000","last_unstale":"2024-06-29T15:57:33.361080+0000","last_undegraded":"2024-06-29T15:57:33.361080+0000","last_fullsized":"2024-06-29T15:57:33.361080+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:40.967292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356113+0000","last_change":"2024-06-29T15:57:33.356113+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356113+0000","last_undegraded":"2024-06-29T15:57:33.356113+0000","last_fullsized":"2024-06-29T15:57:33.356113+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363418+0000","last_change":"2024-06-29T15:57:33.360662+0000","last_active":"2024-06-29T15:57:33.363418+0000","last_peered":"2024-06-29T15:57:33.363418+0000","last_clean":"2024-06-29T15:57:33.363418+0000","last_became_active":"2024-06-29T15:57:33.360087+0000","last_became_peered":"2024-06-29T15:57:33.360087+0000","last_unstale":"2024-06-29T15:57:33.363418+0000","last_undegraded":"2024-06-29T15:57:33.363418+0000","last_fullsized":"2024-06-29T15:57:33.363418+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:46:09.697181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355322+0000","last_change":"2024-06-29T15:57:21.277884+0000","last_active":"2024-06-29T15:57:33.355322+0000","last_peered":"2024-06-29T15:57:33.355322+0000","last_clean":"2024-06-29T15:57:33.355322+0000","last_became_active":"2024-06-29T15:57:21.277589+0000","last_became_peered":"2024-06-29T15:57:21.277589+0000","last_unstale":"2024-06-29T15:57:33.355322+0000","last_undegraded":"2024-06-29T15:57:33.355322+0000","last_fullsized":"2024-06-29T15:57:33.355322+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:09.017396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355087+0000","last_change":"2024-06-29T15:57:22.293696+0000","last_active":"2024-06-29T15:57:33.355087+0000","last_peered":"2024-06-29T15:57:33.355087+0000","last_clean":"2024-06-29T15:57:33.355087+0000","last_became_active":"2024-06-29T15:57:22.293263+0000","last_became_peered":"2024-06-29T15:57:22.293263+0000","last_unstale":"2024-06-29T15:57:33.355087+0000","last_undegraded":"2024-06-29T15:57:33.355087+0000","last_fullsized":"2024-06-29T15:57:33.355087+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352369+0000","last_change":"2024-06-29T15:57:33.352369+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352369+0000","last_undegraded":"2024-06-29T15:57:33.352369+0000","last_fullsized":"2024-06-29T15:57:33.352369+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356661+0000","last_change":"2024-06-29T15:57:21.274306+0000","last_active":"2024-06-29T15:57:33.356661+0000","last_peered":"2024-06-29T15:57:33.356661+0000","last_clean":"2024-06-29T15:57:33.356661+0000","last_became_active":"2024-06-29T15:57:21.273632+0000","last_became_peered":"2024-06-29T15:57:21.273632+0000","last_unstale":"2024-06-29T15:57:33.356661+0000","last_undegraded":"2024-06-29T15:57:33.356661+0000","last_fullsized":"2024-06-29T15:57:33.356661+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361270+0000","last_change":"2024-06-29T15:57:22.297633+0000","last_active":"2024-06-29T15:57:33.361270+0000","last_peered":"2024-06-29T15:57:33.361270+0000","last_clean":"2024-06-29T15:57:33.361270+0000","last_became_active":"2024-06-29T15:57:22.297319+0000","last_became_peered":"2024-06-29T15:57:22.297319+0000","last_unstale":"2024-06-29T15:57:33.361270+0000","last_undegraded":"2024-06-29T15:57:33.361270+0000","last_fullsized":"2024-06-29T15:57:33.361270+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:19.625176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356850+0000","last_change":"2024-06-29T15:57:33.355194+0000","last_active":"2024-06-29T15:57:33.356850+0000","last_peered":"2024-06-29T15:57:33.356850+0000","last_clean":"2024-06-29T15:57:33.356850+0000","last_became_active":"2024-06-29T15:57:33.354883+0000","last_became_peered":"2024-06-29T15:57:33.354883+0000","last_unstale":"2024-06-29T15:57:33.356850+0000","last_undegraded":"2024-06-29T15:57:33.356850+0000","last_fullsized":"2024-06-29T15:57:33.356850+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:06.173920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351788+0000","last_change":"2024-06-29T15:57:33.351788+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351788+0000","last_undegraded":"2024-06-29T15:57:33.351788+0000","last_fullsized":"2024-06-29T15:57:33.351788+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361211+0000","last_change":"2024-06-29T15:57:21.283411+0000","last_active":"2024-06-29T15:57:33.361211+0000","last_peered":"2024-06-29T15:57:33.361211+0000","last_clean":"2024-06-29T15:57:33.361211+0000","last_became_active":"2024-06-29T15:57:21.283081+0000","last_became_peered":"2024-06-29T15:57:21.283081+0000","last_unstale":"2024-06-29T15:57:33.361211+0000","last_undegraded":"2024-06-29T15:57:33.361211+0000","last_fullsized":"2024-06-29T15:57:33.361211+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:27.866266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348680+0000","last_change":"2024-06-29T15:57:22.306835+0000","last_active":"2024-06-29T15:57:33.348680+0000","last_peered":"2024-06-29T15:57:33.348680+0000","last_clean":"2024-06-29T15:57:33.348680+0000","last_became_active":"2024-06-29T15:57:22.306295+0000","last_became_peered":"2024-06-29T15:57:22.306295+0000","last_unstale":"2024-06-29T15:57:33.348680+0000","last_undegraded":"2024-06-29T15:57:33.348680+0000","last_fullsized":"2024-06-29T15:57:33.348680+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:52.124235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367249+0000","last_change":"2024-06-29T15:57:33.365463+0000","last_active":"2024-06-29T15:57:33.367249+0000","last_peered":"2024-06-29T15:57:33.367249+0000","last_clean":"2024-06-29T15:57:33.367249+0000","last_became_active":"2024-06-29T15:57:33.365149+0000","last_became_peered":"2024-06-29T15:57:33.365149+0000","last_unstale":"2024-06-29T15:57:33.367249+0000","last_undegraded":"2024-06-29T15:57:33.367249+0000","last_fullsized":"2024-06-29T15:57:33.367249+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:35.252184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356954+0000","last_change":"2024-06-29T15:57:33.356954+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356954+0000","last_undegraded":"2024-06-29T15:57:33.356954+0000","last_fullsized":"2024-06-29T15:57:33.356954+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356973+0000","last_change":"2024-06-29T15:57:21.284516+0000","last_active":"2024-06-29T15:57:33.356973+0000","last_peered":"2024-06-29T15:57:33.356973+0000","last_clean":"2024-06-29T15:57:33.356973+0000","last_became_active":"2024-06-29T15:57:21.284170+0000","last_became_peered":"2024-06-29T15:57:21.284170+0000","last_unstale":"2024-06-29T15:57:33.356973+0000","last_undegraded":"2024-06-29T15:57:33.356973+0000","last_fullsized":"2024-06-29T15:57:33.356973+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.352840+0000","last_change":"2024-06-29T15:57:22.300202+0000","last_active":"2024-06-29T15:57:33.352840+0000","last_peered":"2024-06-29T15:57:33.352840+0000","last_clean":"2024-06-29T15:57:33.352840+0000","last_became_active":"2024-06-29T15:57:22.299897+0000","last_became_peered":"2024-06-29T15:57:22.299897+0000","last_unstale":"2024-06-29T15:57:33.352840+0000","last_undegraded":"2024-06-29T15:57:33.352840+0000","last_fullsized":"2024-06-29T15:57:33.352840+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364931+0000","last_change":"2024-06-29T15:57:33.363355+0000","last_active":"2024-06-29T15:57:33.364931+0000","last_peered":"2024-06-29T15:57:33.364931+0000","last_clean":"2024-06-29T15:57:33.364931+0000","last_became_active":"2024-06-29T15:57:33.363067+0000","last_became_peered":"2024-06-29T15:57:33.363067+0000","last_unstale":"2024-06-29T15:57:33.364931+0000","last_undegraded":"2024-06-29T15:57:33.364931+0000","last_fullsized":"2024-06-29T15:57:33.364931+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351200+0000","last_change":"2024-06-29T15:57:33.351200+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351200+0000","last_undegraded":"2024-06-29T15:57:33.351200+0000","last_fullsized":"2024-06-29T15:57:33.351200+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356344+0000","last_change":"2024-06-29T15:57:21.274833+0000","last_active":"2024-06-29T15:57:33.356344+0000","last_peered":"2024-06-29T15:57:33.356344+0000","last_clean":"2024-06-29T15:57:33.356344+0000","last_became_active":"2024-06-29T15:57:21.274451+0000","last_became_peered":"2024-06-29T15:57:21.274451+0000","last_unstale":"2024-06-29T15:57:33.356344+0000","last_undegraded":"2024-06-29T15:57:33.356344+0000","last_fullsized":"2024-06-29T15:57:33.356344+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:25.284538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354475+0000","last_change":"2024-06-29T15:57:22.301084+0000","last_active":"2024-06-29T15:57:33.354475+0000","last_peered":"2024-06-29T15:57:33.354475+0000","last_clean":"2024-06-29T15:57:33.354475+0000","last_became_active":"2024-06-29T15:57:22.300639+0000","last_became_peered":"2024-06-29T15:57:22.300639+0000","last_unstale":"2024-06-29T15:57:33.354475+0000","last_undegraded":"2024-06-29T15:57:33.354475+0000","last_fullsized":"2024-06-29T15:57:33.354475+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361064+0000","last_change":"2024-06-29T15:57:33.358185+0000","last_active":"2024-06-29T15:57:33.361064+0000","last_peered":"2024-06-29T15:57:33.361064+0000","last_clean":"2024-06-29T15:57:33.361064+0000","last_became_active":"2024-06-29T15:57:33.357885+0000","last_became_peered":"2024-06-29T15:57:33.357885+0000","last_unstale":"2024-06-29T15:57:33.361064+0000","last_undegraded":"2024-06-29T15:57:33.361064+0000","last_fullsized":"2024-06-29T15:57:33.361064+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:50:30.619228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351153+0000","last_change":"2024-06-29T15:57:33.351153+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351153+0000","last_undegraded":"2024-06-29T15:57:33.351153+0000","last_fullsized":"2024-06-29T15:57:33.351153+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"212'4","reported_seq":22,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365147+0000","last_change":"2024-06-29T15:57:21.276327+0000","last_active":"2024-06-29T15:57:33.365147+0000","last_peered":"2024-06-29T15:57:33.365147+0000","last_clean":"2024-06-29T15:57:33.365147+0000","last_became_active":"2024-06-29T15:57:21.276031+0000","last_became_peered":"2024-06-29T15:57:21.276031+0000","last_unstale":"2024-06-29T15:57:33.365147+0000","last_undegraded":"2024-06-29T15:57:33.365147+0000","last_fullsized":"2024-06-29T15:57:33.365147+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:23.626777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356074+0000","last_change":"2024-06-29T15:57:22.293417+0000","last_active":"2024-06-29T15:57:33.356074+0000","last_peered":"2024-06-29T15:57:33.356074+0000","last_clean":"2024-06-29T15:57:33.356074+0000","last_became_active":"2024-06-29T15:57:22.293083+0000","last_became_peered":"2024-06-29T15:57:22.293083+0000","last_unstale":"2024-06-29T15:57:33.356074+0000","last_undegraded":"2024-06-29T15:57:33.356074+0000","last_fullsized":"2024-06-29T15:57:33.356074+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359057+0000","last_change":"2024-06-29T15:57:33.356187+0000","last_active":"2024-06-29T15:57:33.359057+0000","last_peered":"2024-06-29T15:57:33.359057+0000","last_clean":"2024-06-29T15:57:33.359057+0000","last_became_active":"2024-06-29T15:57:33.355400+0000","last_became_peered":"2024-06-29T15:57:33.355400+0000","last_unstale":"2024-06-29T15:57:33.359057+0000","last_undegraded":"2024-06-29T15:57:33.359057+0000","last_fullsized":"2024-06-29T15:57:33.359057+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:01.393531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354238+0000","last_change":"2024-06-29T15:57:33.354238+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354238+0000","last_undegraded":"2024-06-29T15:57:33.354238+0000","last_fullsized":"2024-06-29T15:57:33.354238+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356423+0000","last_change":"2024-06-29T15:57:22.298975+0000","last_active":"2024-06-29T15:57:33.356423+0000","last_peered":"2024-06-29T15:57:33.356423+0000","last_clean":"2024-06-29T15:57:33.356423+0000","last_became_active":"2024-06-29T15:57:22.298689+0000","last_became_peered":"2024-06-29T15:57:22.298689+0000","last_unstale":"2024-06-29T15:57:33.356423+0000","last_undegraded":"2024-06-29T15:57:33.356423+0000","last_fullsized":"2024-06-29T15:57:33.356423+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359369+0000","last_change":"2024-06-29T15:57:33.359369+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359369+0000","last_undegraded":"2024-06-29T15:57:33.359369+0000","last_fullsized":"2024-06-29T15:57:33.359369+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348068+0000","last_change":"2024-06-29T15:57:22.292249+0000","last_active":"2024-06-29T15:57:33.348068+0000","last_peered":"2024-06-29T15:57:33.348068+0000","last_clean":"2024-06-29T15:57:33.348068+0000","last_became_active":"2024-06-29T15:57:22.291918+0000","last_became_peered":"2024-06-29T15:57:22.291918+0000","last_unstale":"2024-06-29T15:57:33.348068+0000","last_undegraded":"2024-06-29T15:57:33.348068+0000","last_fullsized":"2024-06-29T15:57:33.348068+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:57.129905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351129+0000","last_change":"2024-06-29T15:57:33.351129+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351129+0000","last_undegraded":"2024-06-29T15:57:33.351129+0000","last_fullsized":"2024-06-29T15:57:33.351129+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355012+0000","last_change":"2024-06-29T15:57:22.285362+0000","last_active":"2024-06-29T15:57:33.355012+0000","last_peered":"2024-06-29T15:57:33.355012+0000","last_clean":"2024-06-29T15:57:33.355012+0000","last_became_active":"2024-06-29T15:57:22.284982+0000","last_became_peered":"2024-06-29T15:57:22.284982+0000","last_unstale":"2024-06-29T15:57:33.355012+0000","last_undegraded":"2024-06-29T15:57:33.355012+0000","last_fullsized":"2024-06-29T15:57:33.355012+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357913+0000","last_change":"2024-06-29T15:57:33.357913+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357913+0000","last_undegraded":"2024-06-29T15:57:33.357913+0000","last_fullsized":"2024-06-29T15:57:33.357913+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364482+0000","last_change":"2024-06-29T15:57:22.297998+0000","last_active":"2024-06-29T15:57:33.364482+0000","last_peered":"2024-06-29T15:57:33.364482+0000","last_clean":"2024-06-29T15:57:33.364482+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:33.364482+0000","last_undegraded":"2024-06-29T15:57:33.364482+0000","last_fullsized":"2024-06-29T15:57:33.364482+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357574+0000","last_change":"2024-06-29T15:57:33.357574+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357574+0000","last_undegraded":"2024-06-29T15:57:33.357574+0000","last_fullsized":"2024-06-29T15:57:33.357574+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356051+0000","last_change":"2024-06-29T15:57:22.297917+0000","last_active":"2024-06-29T15:57:33.356051+0000","last_peered":"2024-06-29T15:57:33.356051+0000","last_clean":"2024-06-29T15:57:33.356051+0000","last_became_active":"2024-06-29T15:57:22.297638+0000","last_became_peered":"2024-06-29T15:57:22.297638+0000","last_unstale":"2024-06-29T15:57:33.356051+0000","last_undegraded":"2024-06-29T15:57:33.356051+0000","last_fullsized":"2024-06-29T15:57:33.356051+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351060+0000","last_change":"2024-06-29T15:57:33.351060+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351060+0000","last_undegraded":"2024-06-29T15:57:33.351060+0000","last_fullsized":"2024-06-29T15:57:33.351060+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364399+0000","last_change":"2024-06-29T15:57:22.298174+0000","last_active":"2024-06-29T15:57:33.364399+0000","last_peered":"2024-06-29T15:57:33.364399+0000","last_clean":"2024-06-29T15:57:33.364399+0000","last_became_active":"2024-06-29T15:57:22.297862+0000","last_became_peered":"2024-06-29T15:57:22.297862+0000","last_unstale":"2024-06-29T15:57:33.364399+0000","last_undegraded":"2024-06-29T15:57:33.364399+0000","last_fullsized":"2024-06-29T15:57:33.364399+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354825+0000","last_change":"2024-06-29T15:57:33.354825+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354825+0000","last_undegraded":"2024-06-29T15:57:33.354825+0000","last_fullsized":"2024-06-29T15:57:33.354825+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358628+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:33.358628+0000","last_peered":"2024-06-29T15:57:33.358628+0000","last_clean":"2024-06-29T15:57:33.358628+0000","last_became_active":"2024-06-29T15:57:22.301541+0000","last_became_peered":"2024-06-29T15:57:22.301541+0000","last_unstale":"2024-06-29T15:57:33.358628+0000","last_undegraded":"2024-06-29T15:57:33.358628+0000","last_fullsized":"2024-06-29T15:57:33.358628+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:55.249516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353618+0000","last_change":"2024-06-29T15:57:33.353618+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353618+0000","last_undegraded":"2024-06-29T15:57:33.353618+0000","last_fullsized":"2024-06-29T15:57:33.353618+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355658+0000","last_change":"2024-06-29T15:57:22.292584+0000","last_active":"2024-06-29T15:57:33.355658+0000","last_peered":"2024-06-29T15:57:33.355658+0000","last_clean":"2024-06-29T15:57:33.355658+0000","last_became_active":"2024-06-29T15:57:22.291806+0000","last_became_peered":"2024-06-29T15:57:22.291806+0000","last_unstale":"2024-06-29T15:57:33.355658+0000","last_undegraded":"2024-06-29T15:57:33.355658+0000","last_fullsized":"2024-06-29T15:57:33.355658+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352348+0000","last_change":"2024-06-29T15:57:33.352348+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352348+0000","last_undegraded":"2024-06-29T15:57:33.352348+0000","last_fullsized":"2024-06-29T15:57:33.352348+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363619+0000","last_change":"2024-06-29T15:57:22.301601+0000","last_active":"2024-06-29T15:57:33.363619+0000","last_peered":"2024-06-29T15:57:33.363619+0000","last_clean":"2024-06-29T15:57:33.363619+0000","last_became_active":"2024-06-29T15:57:22.301244+0000","last_became_peered":"2024-06-29T15:57:22.301244+0000","last_unstale":"2024-06-29T15:57:33.363619+0000","last_undegraded":"2024-06-29T15:57:33.363619+0000","last_fullsized":"2024-06-29T15:57:33.363619+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:24:56.938464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352379+0000","last_change":"2024-06-29T15:57:33.352379+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352379+0000","last_undegraded":"2024-06-29T15:57:33.352379+0000","last_fullsized":"2024-06-29T15:57:33.352379+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353891+0000","last_change":"2024-06-29T15:57:22.301362+0000","last_active":"2024-06-29T15:57:33.353891+0000","last_peered":"2024-06-29T15:57:33.353891+0000","last_clean":"2024-06-29T15:57:33.353891+0000","last_became_active":"2024-06-29T15:57:22.300704+0000","last_became_peered":"2024-06-29T15:57:22.300704+0000","last_unstale":"2024-06-29T15:57:33.353891+0000","last_undegraded":"2024-06-29T15:57:33.353891+0000","last_fullsized":"2024-06-29T15:57:33.353891+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351810+0000","last_change":"2024-06-29T15:57:33.351810+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351810+0000","last_undegraded":"2024-06-29T15:57:33.351810+0000","last_fullsized":"2024-06-29T15:57:33.351810+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353925+0000","last_change":"2024-06-29T15:57:22.307686+0000","last_active":"2024-06-29T15:57:33.353925+0000","last_peered":"2024-06-29T15:57:33.353925+0000","last_clean":"2024-06-29T15:57:33.353925+0000","last_became_active":"2024-06-29T15:57:22.307324+0000","last_became_peered":"2024-06-29T15:57:22.307324+0000","last_unstale":"2024-06-29T15:57:33.353925+0000","last_undegraded":"2024-06-29T15:57:33.353925+0000","last_fullsized":"2024-06-29T15:57:33.353925+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353324+0000","last_change":"2024-06-29T15:57:33.353324+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353324+0000","last_undegraded":"2024-06-29T15:57:33.353324+0000","last_fullsized":"2024-06-29T15:57:33.353324+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361177+0000","last_change":"2024-06-29T15:57:22.307230+0000","last_active":"2024-06-29T15:57:33.361177+0000","last_peered":"2024-06-29T15:57:33.361177+0000","last_clean":"2024-06-29T15:57:33.361177+0000","last_became_active":"2024-06-29T15:57:22.306920+0000","last_became_peered":"2024-06-29T15:57:22.306920+0000","last_unstale":"2024-06-29T15:57:33.361177+0000","last_undegraded":"2024-06-29T15:57:33.361177+0000","last_fullsized":"2024-06-29T15:57:33.361177+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:22.598647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352631+0000","last_change":"2024-06-29T15:57:33.352631+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352631+0000","last_undegraded":"2024-06-29T15:57:33.352631+0000","last_fullsized":"2024-06-29T15:57:33.352631+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359396+0000","last_change":"2024-06-29T15:57:22.290186+0000","last_active":"2024-06-29T15:57:33.359396+0000","last_peered":"2024-06-29T15:57:33.359396+0000","last_clean":"2024-06-29T15:57:33.359396+0000","last_became_active":"2024-06-29T15:57:22.289908+0000","last_became_peered":"2024-06-29T15:57:22.289908+0000","last_unstale":"2024-06-29T15:57:33.359396+0000","last_undegraded":"2024-06-29T15:57:33.359396+0000","last_fullsized":"2024-06-29T15:57:33.359396+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353284+0000","last_change":"2024-06-29T15:57:33.353284+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353284+0000","last_undegraded":"2024-06-29T15:57:33.353284+0000","last_fullsized":"2024-06-29T15:57:33.353284+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348980+0000","last_change":"2024-06-29T15:57:22.306593+0000","last_active":"2024-06-29T15:57:33.348980+0000","last_peered":"2024-06-29T15:57:33.348980+0000","last_clean":"2024-06-29T15:57:33.348980+0000","last_became_active":"2024-06-29T15:57:22.296653+0000","last_became_peered":"2024-06-29T15:57:22.296653+0000","last_unstale":"2024-06-29T15:57:33.348980+0000","last_undegraded":"2024-06-29T15:57:33.348980+0000","last_fullsized":"2024-06-29T15:57:33.348980+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:41.738155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351749+0000","last_change":"2024-06-29T15:57:33.351749+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351749+0000","last_undegraded":"2024-06-29T15:57:33.351749+0000","last_fullsized":"2024-06-29T15:57:33.351749+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348828+0000","last_change":"2024-06-29T15:57:22.289633+0000","last_active":"2024-06-29T15:57:33.348828+0000","last_peered":"2024-06-29T15:57:33.348828+0000","last_clean":"2024-06-29T15:57:33.348828+0000","last_became_active":"2024-06-29T15:57:22.289349+0000","last_became_peered":"2024-06-29T15:57:22.289349+0000","last_unstale":"2024-06-29T15:57:33.348828+0000","last_undegraded":"2024-06-29T15:57:33.348828+0000","last_fullsized":"2024-06-29T15:57:33.348828+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.873806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357678+0000","last_change":"2024-06-29T15:57:33.357678+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357678+0000","last_undegraded":"2024-06-29T15:57:33.357678+0000","last_fullsized":"2024-06-29T15:57:33.357678+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355608+0000","last_change":"2024-06-29T15:57:22.300787+0000","last_active":"2024-06-29T15:57:33.355608+0000","last_peered":"2024-06-29T15:57:33.355608+0000","last_clean":"2024-06-29T15:57:33.355608+0000","last_became_active":"2024-06-29T15:57:22.300112+0000","last_became_peered":"2024-06-29T15:57:22.300112+0000","last_unstale":"2024-06-29T15:57:33.355608+0000","last_undegraded":"2024-06-29T15:57:33.355608+0000","last_fullsized":"2024-06-29T15:57:33.355608+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:30.249483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353010+0000","last_change":"2024-06-29T15:57:33.353010+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353010+0000","last_undegraded":"2024-06-29T15:57:33.353010+0000","last_fullsized":"2024-06-29T15:57:33.353010+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348033+0000","last_change":"2024-06-29T15:57:22.290047+0000","last_active":"2024-06-29T15:57:33.348033+0000","last_peered":"2024-06-29T15:57:33.348033+0000","last_clean":"2024-06-29T15:57:33.348033+0000","last_became_active":"2024-06-29T15:57:22.289742+0000","last_became_peered":"2024-06-29T15:57:22.289742+0000","last_unstale":"2024-06-29T15:57:33.348033+0000","last_undegraded":"2024-06-29T15:57:33.348033+0000","last_fullsized":"2024-06-29T15:57:33.348033+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:03:46.271206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362654+0000","last_change":"2024-06-29T15:57:33.362654+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362654+0000","last_undegraded":"2024-06-29T15:57:33.362654+0000","last_fullsized":"2024-06-29T15:57:33.362654+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356944+0000","last_change":"2024-06-29T15:57:22.302418+0000","last_active":"2024-06-29T15:57:33.356944+0000","last_peered":"2024-06-29T15:57:33.356944+0000","last_clean":"2024-06-29T15:57:33.356944+0000","last_became_active":"2024-06-29T15:57:22.302075+0000","last_became_peered":"2024-06-29T15:57:22.302075+0000","last_unstale":"2024-06-29T15:57:33.356944+0000","last_undegraded":"2024-06-29T15:57:33.356944+0000","last_fullsized":"2024-06-29T15:57:33.356944+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.350522+0000","last_change":"2024-06-29T15:57:33.350522+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.350522+0000","last_undegraded":"2024-06-29T15:57:33.350522+0000","last_fullsized":"2024-06-29T15:57:33.350522+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354196+0000","last_change":"2024-06-29T15:57:22.303092+0000","last_active":"2024-06-29T15:57:33.354196+0000","last_peered":"2024-06-29T15:57:33.354196+0000","last_clean":"2024-06-29T15:57:33.354196+0000","last_became_active":"2024-06-29T15:57:22.302807+0000","last_became_peered":"2024-06-29T15:57:22.302807+0000","last_unstale":"2024-06-29T15:57:33.354196+0000","last_undegraded":"2024-06-29T15:57:33.354196+0000","last_fullsized":"2024-06-29T15:57:33.354196+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360360+0000","last_change":"2024-06-29T15:57:33.360360+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360360+0000","last_undegraded":"2024-06-29T15:57:33.360360+0000","last_fullsized":"2024-06-29T15:57:33.360360+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365064+0000","last_change":"2024-06-29T15:57:22.295681+0000","last_active":"2024-06-29T15:57:33.365064+0000","last_peered":"2024-06-29T15:57:33.365064+0000","last_clean":"2024-06-29T15:57:33.365064+0000","last_became_active":"2024-06-29T15:57:22.295357+0000","last_became_peered":"2024-06-29T15:57:22.295357+0000","last_unstale":"2024-06-29T15:57:33.365064+0000","last_undegraded":"2024-06-29T15:57:33.365064+0000","last_fullsized":"2024-06-29T15:57:33.365064+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353724+0000","last_change":"2024-06-29T15:57:33.353724+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353724+0000","last_undegraded":"2024-06-29T15:57:33.353724+0000","last_fullsized":"2024-06-29T15:57:33.353724+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365558+0000","last_change":"2024-06-29T15:57:22.298612+0000","last_active":"2024-06-29T15:57:33.365558+0000","last_peered":"2024-06-29T15:57:33.365558+0000","last_clean":"2024-06-29T15:57:33.365558+0000","last_became_active":"2024-06-29T15:57:22.298288+0000","last_became_peered":"2024-06-29T15:57:22.298288+0000","last_unstale":"2024-06-29T15:57:33.365558+0000","last_undegraded":"2024-06-29T15:57:33.365558+0000","last_fullsized":"2024-06-29T15:57:33.365558+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351291+0000","last_change":"2024-06-29T15:57:33.351291+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351291+0000","last_undegraded":"2024-06-29T15:57:33.351291+0000","last_fullsized":"2024-06-29T15:57:33.351291+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356870+0000","last_change":"2024-06-29T15:57:22.298781+0000","last_active":"2024-06-29T15:57:33.356870+0000","last_peered":"2024-06-29T15:57:33.356870+0000","last_clean":"2024-06-29T15:57:33.356870+0000","last_became_active":"2024-06-29T15:57:22.298300+0000","last_became_peered":"2024-06-29T15:57:22.298300+0000","last_unstale":"2024-06-29T15:57:33.356870+0000","last_undegraded":"2024-06-29T15:57:33.356870+0000","last_fullsized":"2024-06-29T15:57:33.356870+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349621+0000","last_change":"2024-06-29T15:57:33.349621+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349621+0000","last_undegraded":"2024-06-29T15:57:33.349621+0000","last_fullsized":"2024-06-29T15:57:33.349621+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348689+0000","last_change":"2024-06-29T15:57:22.296441+0000","last_active":"2024-06-29T15:57:33.348689+0000","last_peered":"2024-06-29T15:57:33.348689+0000","last_clean":"2024-06-29T15:57:33.348689+0000","last_became_active":"2024-06-29T15:57:22.296146+0000","last_became_peered":"2024-06-29T15:57:22.296146+0000","last_unstale":"2024-06-29T15:57:33.348689+0000","last_undegraded":"2024-06-29T15:57:33.348689+0000","last_fullsized":"2024-06-29T15:57:33.348689+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:54.360521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356517+0000","last_change":"2024-06-29T15:57:33.356517+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356517+0000","last_undegraded":"2024-06-29T15:57:33.356517+0000","last_fullsized":"2024-06-29T15:57:33.356517+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356057+0000","last_change":"2024-06-29T15:57:22.291096+0000","last_active":"2024-06-29T15:57:33.356057+0000","last_peered":"2024-06-29T15:57:33.356057+0000","last_clean":"2024-06-29T15:57:33.356057+0000","last_became_active":"2024-06-29T15:57:22.290428+0000","last_became_peered":"2024-06-29T15:57:22.290428+0000","last_unstale":"2024-06-29T15:57:33.356057+0000","last_undegraded":"2024-06-29T15:57:33.356057+0000","last_fullsized":"2024-06-29T15:57:33.356057+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:20.813793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352280+0000","last_change":"2024-06-29T15:57:33.352280+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352280+0000","last_undegraded":"2024-06-29T15:57:33.352280+0000","last_fullsized":"2024-06-29T15:57:33.352280+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355022+0000","last_change":"2024-06-29T15:57:22.297617+0000","last_active":"2024-06-29T15:57:33.355022+0000","last_peered":"2024-06-29T15:57:33.355022+0000","last_clean":"2024-06-29T15:57:33.355022+0000","last_became_active":"2024-06-29T15:57:22.297256+0000","last_became_peered":"2024-06-29T15:57:22.297256+0000","last_unstale":"2024-06-29T15:57:33.355022+0000","last_undegraded":"2024-06-29T15:57:33.355022+0000","last_fullsized":"2024-06-29T15:57:33.355022+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359862+0000","last_change":"2024-06-29T15:57:33.359862+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359862+0000","last_undegraded":"2024-06-29T15:57:33.359862+0000","last_fullsized":"2024-06-29T15:57:33.359862+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356834+0000","last_change":"2024-06-29T15:57:22.304052+0000","last_active":"2024-06-29T15:57:33.356834+0000","last_peered":"2024-06-29T15:57:33.356834+0000","last_clean":"2024-06-29T15:57:33.356834+0000","last_became_active":"2024-06-29T15:57:22.303598+0000","last_became_peered":"2024-06-29T15:57:22.303598+0000","last_unstale":"2024-06-29T15:57:33.356834+0000","last_undegraded":"2024-06-29T15:57:33.356834+0000","last_fullsized":"2024-06-29T15:57:33.356834+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352198+0000","last_change":"2024-06-29T15:57:33.352198+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352198+0000","last_undegraded":"2024-06-29T15:57:33.352198+0000","last_fullsized":"2024-06-29T15:57:33.352198+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356415+0000","last_change":"2024-06-29T15:57:22.288041+0000","last_active":"2024-06-29T15:57:33.356415+0000","last_peered":"2024-06-29T15:57:33.356415+0000","last_clean":"2024-06-29T15:57:33.356415+0000","last_became_active":"2024-06-29T15:57:22.287729+0000","last_became_peered":"2024-06-29T15:57:22.287729+0000","last_unstale":"2024-06-29T15:57:33.356415+0000","last_undegraded":"2024-06-29T15:57:33.356415+0000","last_fullsized":"2024-06-29T15:57:33.356415+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362099+0000","last_change":"2024-06-29T15:57:33.362099+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362099+0000","last_undegraded":"2024-06-29T15:57:33.362099+0000","last_fullsized":"2024-06-29T15:57:33.362099+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356673+0000","last_change":"2024-06-29T15:57:22.290909+0000","last_active":"2024-06-29T15:57:33.356673+0000","last_peered":"2024-06-29T15:57:33.356673+0000","last_clean":"2024-06-29T15:57:33.356673+0000","last_became_active":"2024-06-29T15:57:22.290643+0000","last_became_peered":"2024-06-29T15:57:22.290643+0000","last_unstale":"2024-06-29T15:57:33.356673+0000","last_undegraded":"2024-06-29T15:57:33.356673+0000","last_fullsized":"2024-06-29T15:57:33.356673+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354913+0000","last_change":"2024-06-29T15:57:33.354913+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354913+0000","last_undegraded":"2024-06-29T15:57:33.354913+0000","last_fullsized":"2024-06-29T15:57:33.354913+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361010+0000","last_change":"2024-06-29T15:57:22.298374+0000","last_active":"2024-06-29T15:57:33.361010+0000","last_peered":"2024-06-29T15:57:33.361010+0000","last_clean":"2024-06-29T15:57:33.361010+0000","last_became_active":"2024-06-29T15:57:22.297962+0000","last_became_peered":"2024-06-29T15:57:22.297962+0000","last_unstale":"2024-06-29T15:57:33.361010+0000","last_undegraded":"2024-06-29T15:57:33.361010+0000","last_fullsized":"2024-06-29T15:57:33.361010+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:13.834244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354164+0000","last_change":"2024-06-29T15:57:33.354164+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354164+0000","last_undegraded":"2024-06-29T15:57:33.354164+0000","last_fullsized":"2024-06-29T15:57:33.354164+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356686+0000","last_change":"2024-06-29T15:57:22.300745+0000","last_active":"2024-06-29T15:57:33.356686+0000","last_peered":"2024-06-29T15:57:33.356686+0000","last_clean":"2024-06-29T15:57:33.356686+0000","last_became_active":"2024-06-29T15:57:22.300414+0000","last_became_peered":"2024-06-29T15:57:22.300414+0000","last_unstale":"2024-06-29T15:57:33.356686+0000","last_undegraded":"2024-06-29T15:57:33.356686+0000","last_fullsized":"2024-06-29T15:57:33.356686+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356852+0000","last_change":"2024-06-29T15:57:33.356852+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356852+0000","last_undegraded":"2024-06-29T15:57:33.356852+0000","last_fullsized":"2024-06-29T15:57:33.356852+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355941+0000","last_change":"2024-06-29T15:57:22.297971+0000","last_active":"2024-06-29T15:57:33.355941+0000","last_peered":"2024-06-29T15:57:33.355941+0000","last_clean":"2024-06-29T15:57:33.355941+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:33.355941+0000","last_undegraded":"2024-06-29T15:57:33.355941+0000","last_fullsized":"2024-06-29T15:57:33.355941+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360424+0000","last_change":"2024-06-29T15:57:33.360424+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360424+0000","last_undegraded":"2024-06-29T15:57:33.360424+0000","last_fullsized":"2024-06-29T15:57:33.360424+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355500+0000","last_change":"2024-06-29T15:57:22.301019+0000","last_active":"2024-06-29T15:57:33.355500+0000","last_peered":"2024-06-29T15:57:33.355500+0000","last_clean":"2024-06-29T15:57:33.355500+0000","last_became_active":"2024-06-29T15:57:22.300506+0000","last_became_peered":"2024-06-29T15:57:22.300506+0000","last_unstale":"2024-06-29T15:57:33.355500+0000","last_undegraded":"2024-06-29T15:57:33.355500+0000","last_fullsized":"2024-06-29T15:57:33.355500+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52456,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319384,"statfs":{"total":96636764160,"available":96583049216,"internally_reserved":0,"allocated":3305472,"data_stored":1914275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575988,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42140,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329700,"statfs":{"total":96636764160,"available":96593612800,"internally_reserved":0,"allocated":2703360,"data_stored":1321634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575986,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42128,"kb_used_data":2628,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329712,"statfs":{"total":96636764160,"available":96593625088,"internally_reserved":0,"allocated":2691072,"data_stored":1314370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60208,"kb_used_data":2656,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311632,"statfs":{"total":96636764160,"available":96575111168,"internally_reserved":0,"allocated":2719744,"data_stored":1324011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47716,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324124,"statfs":{"total":96636764160,"available":96587902976,"internally_reserved":0,"allocated":2711552,"data_stored":1321234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575987,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47704,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324136,"statfs":{"total":96636764160,"available":96587915264,"internally_reserved":0,"allocated":2699264,"data_stored":1321052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575986,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60180,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311660,"statfs":{"total":96636764160,"available":96575139840,"internally_reserved":0,"allocated":2695168,"data_stored":1313844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60756,"kb_used_data":3208,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311084,"statfs":{"total":96636764160,"available":96574550016,"internally_reserved":0,"allocated":3284992,"data_stored":1904246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:39.141 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1c is not active+clean 2024-06-29T15:57:39.141 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.357377+0000', 'last_change': '2024-06-29T15:57:33.357377+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.357377+0000', 'last_undegraded': '2024-06-29T15:57:33.357377+0000', 'last_fullsized': '2024-06-29T15:57:33.357377+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.141 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.7 is not active+clean 2024-06-29T15:57:39.141 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.362213+0000', 'last_change': '2024-06-29T15:57:33.362213+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.362213+0000', 'last_undegraded': '2024-06-29T15:57:33.362213+0000', 'last_fullsized': '2024-06-29T15:57:33.362213+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.6 is not active+clean 2024-06-29T15:57:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351888+0000', 'last_change': '2024-06-29T15:57:33.351888+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351888+0000', 'last_undegraded': '2024-06-29T15:57:33.351888+0000', 'last_fullsized': '2024-06-29T15:57:33.351888+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.5 is not active+clean 2024-06-29T15:57:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.354530+0000', 'last_change': '2024-06-29T15:57:33.354530+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.354530+0000', 'last_undegraded': '2024-06-29T15:57:33.354530+0000', 'last_fullsized': '2024-06-29T15:57:33.354530+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.4 is not active+clean 2024-06-29T15:57:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.358295+0000', 'last_change': '2024-06-29T15:57:33.358295+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.358295+0000', 'last_undegraded': '2024-06-29T15:57:33.358295+0000', 'last_fullsized': '2024-06-29T15:57:33.358295+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.3 is not active+clean 2024-06-29T15:57:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.360097+0000', 'last_change': '2024-06-29T15:57:33.360097+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.360097+0000', 'last_undegraded': '2024-06-29T15:57:33.360097+0000', 'last_fullsized': '2024-06-29T15:57:33.360097+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.2 is not active+clean 2024-06-29T15:57:39.142 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.349702+0000', 'last_change': '2024-06-29T15:57:33.349702+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.349702+0000', 'last_undegraded': '2024-06-29T15:57:33.349702+0000', 'last_fullsized': '2024-06-29T15:57:33.349702+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1 is not active+clean 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.357030+0000', 'last_change': '2024-06-29T15:57:33.357030+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.357030+0000', 'last_undegraded': '2024-06-29T15:57:33.357030+0000', 'last_fullsized': '2024-06-29T15:57:33.357030+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.0 is not active+clean 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.363662+0000', 'last_change': '2024-06-29T15:57:33.363662+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.363662+0000', 'last_undegraded': '2024-06-29T15:57:33.363662+0000', 'last_fullsized': '2024-06-29T15:57:33.363662+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.f is not active+clean 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351295+0000', 'last_change': '2024-06-29T15:57:33.351295+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351295+0000', 'last_undegraded': '2024-06-29T15:57:33.351295+0000', 'last_fullsized': '2024-06-29T15:57:33.351295+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.e is not active+clean 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.353695+0000', 'last_change': '2024-06-29T15:57:33.353695+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.353695+0000', 'last_undegraded': '2024-06-29T15:57:33.353695+0000', 'last_fullsized': '2024-06-29T15:57:33.353695+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.d is not active+clean 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.361651+0000', 'last_change': '2024-06-29T15:57:33.361651+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.361651+0000', 'last_undegraded': '2024-06-29T15:57:33.361651+0000', 'last_fullsized': '2024-06-29T15:57:33.361651+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.c is not active+clean 2024-06-29T15:57:39.143 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.358755+0000', 'last_change': '2024-06-29T15:57:33.358755+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.358755+0000', 'last_undegraded': '2024-06-29T15:57:33.358755+0000', 'last_fullsized': '2024-06-29T15:57:33.358755+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.b is not active+clean 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.361054+0000', 'last_change': '2024-06-29T15:57:33.361054+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.361054+0000', 'last_undegraded': '2024-06-29T15:57:33.361054+0000', 'last_fullsized': '2024-06-29T15:57:33.361054+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.a is not active+clean 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.352248+0000', 'last_change': '2024-06-29T15:57:33.352248+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.352248+0000', 'last_undegraded': '2024-06-29T15:57:33.352248+0000', 'last_fullsized': '2024-06-29T15:57:33.352248+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.9 is not active+clean 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.352493+0000', 'last_change': '2024-06-29T15:57:33.352493+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.352493+0000', 'last_undegraded': '2024-06-29T15:57:33.352493+0000', 'last_fullsized': '2024-06-29T15:57:33.352493+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.8 is not active+clean 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.358484+0000', 'last_change': '2024-06-29T15:57:33.358484+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.358484+0000', 'last_undegraded': '2024-06-29T15:57:33.358484+0000', 'last_fullsized': '2024-06-29T15:57:33.358484+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.17 is not active+clean 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.357163+0000', 'last_change': '2024-06-29T15:57:33.357163+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.357163+0000', 'last_undegraded': '2024-06-29T15:57:33.357163+0000', 'last_fullsized': '2024-06-29T15:57:33.357163+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.144 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.16 is not active+clean 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.349766+0000', 'last_change': '2024-06-29T15:57:33.349766+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.349766+0000', 'last_undegraded': '2024-06-29T15:57:33.349766+0000', 'last_fullsized': '2024-06-29T15:57:33.349766+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.15 is not active+clean 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.358887+0000', 'last_change': '2024-06-29T15:57:33.358887+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.358887+0000', 'last_undegraded': '2024-06-29T15:57:33.358887+0000', 'last_fullsized': '2024-06-29T15:57:33.358887+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.14 is not active+clean 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351919+0000', 'last_change': '2024-06-29T15:57:33.351919+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351919+0000', 'last_undegraded': '2024-06-29T15:57:33.351919+0000', 'last_fullsized': '2024-06-29T15:57:33.351919+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.13 is not active+clean 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.359617+0000', 'last_change': '2024-06-29T15:57:33.359617+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.359617+0000', 'last_undegraded': '2024-06-29T15:57:33.359617+0000', 'last_fullsized': '2024-06-29T15:57:33.359617+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.12 is not active+clean 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.353505+0000', 'last_change': '2024-06-29T15:57:33.353505+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.353505+0000', 'last_undegraded': '2024-06-29T15:57:33.353505+0000', 'last_fullsized': '2024-06-29T15:57:33.353505+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.11 is not active+clean 2024-06-29T15:57:39.145 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.352295+0000', 'last_change': '2024-06-29T15:57:33.352295+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.352295+0000', 'last_undegraded': '2024-06-29T15:57:33.352295+0000', 'last_fullsized': '2024-06-29T15:57:33.352295+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T15:57:39.146 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.10 is not active+clean 2024-06-29T15:57:39.146 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.358599+0000', 'last_change': '2024-06-29T15:57:33.358599+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.358599+0000', 'last_undegraded': '2024-06-29T15:57:33.358599+0000', 'last_fullsized': '2024-06-29T15:57:33.358599+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.146 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1f is not active+clean 2024-06-29T15:57:39.146 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.356450+0000', 'last_change': '2024-06-29T15:57:33.356450+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.356450+0000', 'last_undegraded': '2024-06-29T15:57:33.356450+0000', 'last_fullsized': '2024-06-29T15:57:33.356450+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-29T15:57:39.146 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1e is not active+clean 2024-06-29T15:57:39.146 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.356113+0000', 'last_change': '2024-06-29T15:57:33.356113+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.356113+0000', 'last_undegraded': '2024-06-29T15:57:33.356113+0000', 'last_fullsized': '2024-06-29T15:57:33.356113+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.146 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1d is not active+clean 2024-06-29T15:57:39.146 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.352369+0000', 'last_change': '2024-06-29T15:57:33.352369+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.352369+0000', 'last_undegraded': '2024-06-29T15:57:33.352369+0000', 'last_fullsized': '2024-06-29T15:57:33.352369+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T15:57:39.146 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1b is not active+clean 2024-06-29T15:57:39.146 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351788+0000', 'last_change': '2024-06-29T15:57:33.351788+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351788+0000', 'last_undegraded': '2024-06-29T15:57:33.351788+0000', 'last_fullsized': '2024-06-29T15:57:33.351788+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.1a is not active+clean 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.356954+0000', 'last_change': '2024-06-29T15:57:33.356954+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.356954+0000', 'last_undegraded': '2024-06-29T15:57:33.356954+0000', 'last_fullsized': '2024-06-29T15:57:33.356954+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.19 is not active+clean 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351200+0000', 'last_change': '2024-06-29T15:57:33.351200+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351200+0000', 'last_undegraded': '2024-06-29T15:57:33.351200+0000', 'last_fullsized': '2024-06-29T15:57:33.351200+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.18 is not active+clean 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351153+0000', 'last_change': '2024-06-29T15:57:33.351153+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351153+0000', 'last_undegraded': '2024-06-29T15:57:33.351153+0000', 'last_fullsized': '2024-06-29T15:57:33.351153+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.26 is not active+clean 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.354238+0000', 'last_change': '2024-06-29T15:57:33.354238+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.354238+0000', 'last_undegraded': '2024-06-29T15:57:33.354238+0000', 'last_fullsized': '2024-06-29T15:57:33.354238+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.27 is not active+clean 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.359369+0000', 'last_change': '2024-06-29T15:57:33.359369+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.359369+0000', 'last_undegraded': '2024-06-29T15:57:33.359369+0000', 'last_fullsized': '2024-06-29T15:57:33.359369+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.24 is not active+clean 2024-06-29T15:57:39.147 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351129+0000', 'last_change': '2024-06-29T15:57:33.351129+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351129+0000', 'last_undegraded': '2024-06-29T15:57:33.351129+0000', 'last_fullsized': '2024-06-29T15:57:33.351129+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.25 is not active+clean 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.357913+0000', 'last_change': '2024-06-29T15:57:33.357913+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.357913+0000', 'last_undegraded': '2024-06-29T15:57:33.357913+0000', 'last_fullsized': '2024-06-29T15:57:33.357913+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.22 is not active+clean 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.357574+0000', 'last_change': '2024-06-29T15:57:33.357574+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.357574+0000', 'last_undegraded': '2024-06-29T15:57:33.357574+0000', 'last_fullsized': '2024-06-29T15:57:33.357574+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.23 is not active+clean 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351060+0000', 'last_change': '2024-06-29T15:57:33.351060+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351060+0000', 'last_undegraded': '2024-06-29T15:57:33.351060+0000', 'last_fullsized': '2024-06-29T15:57:33.351060+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.20 is not active+clean 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.354825+0000', 'last_change': '2024-06-29T15:57:33.354825+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.354825+0000', 'last_undegraded': '2024-06-29T15:57:33.354825+0000', 'last_fullsized': '2024-06-29T15:57:33.354825+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.21 is not active+clean 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.353618+0000', 'last_change': '2024-06-29T15:57:33.353618+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.353618+0000', 'last_undegraded': '2024-06-29T15:57:33.353618+0000', 'last_fullsized': '2024-06-29T15:57:33.353618+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.2e is not active+clean 2024-06-29T15:57:39.148 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.352348+0000', 'last_change': '2024-06-29T15:57:33.352348+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.352348+0000', 'last_undegraded': '2024-06-29T15:57:33.352348+0000', 'last_fullsized': '2024-06-29T15:57:33.352348+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.2f is not active+clean 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.352379+0000', 'last_change': '2024-06-29T15:57:33.352379+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.352379+0000', 'last_undegraded': '2024-06-29T15:57:33.352379+0000', 'last_fullsized': '2024-06-29T15:57:33.352379+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.2c is not active+clean 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351810+0000', 'last_change': '2024-06-29T15:57:33.351810+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351810+0000', 'last_undegraded': '2024-06-29T15:57:33.351810+0000', 'last_fullsized': '2024-06-29T15:57:33.351810+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.2d is not active+clean 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.353324+0000', 'last_change': '2024-06-29T15:57:33.353324+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.353324+0000', 'last_undegraded': '2024-06-29T15:57:33.353324+0000', 'last_fullsized': '2024-06-29T15:57:33.353324+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.2a is not active+clean 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.352631+0000', 'last_change': '2024-06-29T15:57:33.352631+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.352631+0000', 'last_undegraded': '2024-06-29T15:57:33.352631+0000', 'last_fullsized': '2024-06-29T15:57:33.352631+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.2b is not active+clean 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.353284+0000', 'last_change': '2024-06-29T15:57:33.353284+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.353284+0000', 'last_undegraded': '2024-06-29T15:57:33.353284+0000', 'last_fullsized': '2024-06-29T15:57:33.353284+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.149 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.28 is not active+clean 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351749+0000', 'last_change': '2024-06-29T15:57:33.351749+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351749+0000', 'last_undegraded': '2024-06-29T15:57:33.351749+0000', 'last_fullsized': '2024-06-29T15:57:33.351749+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.29 is not active+clean 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.357678+0000', 'last_change': '2024-06-29T15:57:33.357678+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.357678+0000', 'last_undegraded': '2024-06-29T15:57:33.357678+0000', 'last_fullsized': '2024-06-29T15:57:33.357678+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.36 is not active+clean 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.353010+0000', 'last_change': '2024-06-29T15:57:33.353010+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.353010+0000', 'last_undegraded': '2024-06-29T15:57:33.353010+0000', 'last_fullsized': '2024-06-29T15:57:33.353010+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.37 is not active+clean 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.362654+0000', 'last_change': '2024-06-29T15:57:33.362654+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.362654+0000', 'last_undegraded': '2024-06-29T15:57:33.362654+0000', 'last_fullsized': '2024-06-29T15:57:33.362654+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.34 is not active+clean 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.350522+0000', 'last_change': '2024-06-29T15:57:33.350522+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.350522+0000', 'last_undegraded': '2024-06-29T15:57:33.350522+0000', 'last_fullsized': '2024-06-29T15:57:33.350522+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.35 is not active+clean 2024-06-29T15:57:39.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.360360+0000', 'last_change': '2024-06-29T15:57:33.360360+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.360360+0000', 'last_undegraded': '2024-06-29T15:57:33.360360+0000', 'last_fullsized': '2024-06-29T15:57:33.360360+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.32 is not active+clean 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.353724+0000', 'last_change': '2024-06-29T15:57:33.353724+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.353724+0000', 'last_undegraded': '2024-06-29T15:57:33.353724+0000', 'last_fullsized': '2024-06-29T15:57:33.353724+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.33 is not active+clean 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.351291+0000', 'last_change': '2024-06-29T15:57:33.351291+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.351291+0000', 'last_undegraded': '2024-06-29T15:57:33.351291+0000', 'last_fullsized': '2024-06-29T15:57:33.351291+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.30 is not active+clean 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.349621+0000', 'last_change': '2024-06-29T15:57:33.349621+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.349621+0000', 'last_undegraded': '2024-06-29T15:57:33.349621+0000', 'last_fullsized': '2024-06-29T15:57:33.349621+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.31 is not active+clean 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.356517+0000', 'last_change': '2024-06-29T15:57:33.356517+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.356517+0000', 'last_undegraded': '2024-06-29T15:57:33.356517+0000', 'last_fullsized': '2024-06-29T15:57:33.356517+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.3e is not active+clean 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.352280+0000', 'last_change': '2024-06-29T15:57:33.352280+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.352280+0000', 'last_undegraded': '2024-06-29T15:57:33.352280+0000', 'last_fullsized': '2024-06-29T15:57:33.352280+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-29T15:57:39.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.3f is not active+clean 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.359862+0000', 'last_change': '2024-06-29T15:57:33.359862+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.359862+0000', 'last_undegraded': '2024-06-29T15:57:33.359862+0000', 'last_fullsized': '2024-06-29T15:57:33.359862+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.3c is not active+clean 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.352198+0000', 'last_change': '2024-06-29T15:57:33.352198+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.352198+0000', 'last_undegraded': '2024-06-29T15:57:33.352198+0000', 'last_fullsized': '2024-06-29T15:57:33.352198+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.3d is not active+clean 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.362099+0000', 'last_change': '2024-06-29T15:57:33.362099+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.362099+0000', 'last_undegraded': '2024-06-29T15:57:33.362099+0000', 'last_fullsized': '2024-06-29T15:57:33.362099+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.3a is not active+clean 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.354913+0000', 'last_change': '2024-06-29T15:57:33.354913+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.354913+0000', 'last_undegraded': '2024-06-29T15:57:33.354913+0000', 'last_fullsized': '2024-06-29T15:57:33.354913+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.3b is not active+clean 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.354164+0000', 'last_change': '2024-06-29T15:57:33.354164+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.354164+0000', 'last_undegraded': '2024-06-29T15:57:33.354164+0000', 'last_fullsized': '2024-06-29T15:57:33.354164+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.38 is not active+clean 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.356852+0000', 'last_change': '2024-06-29T15:57:33.356852+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.356852+0000', 'last_undegraded': '2024-06-29T15:57:33.356852+0000', 'last_fullsized': '2024-06-29T15:57:33.356852+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-29T15:57:39.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 29.39 is not active+clean 2024-06-29T15:57:39.153 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '29.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 215, 'state': 'creating+peering', 'last_fresh': '2024-06-29T15:57:33.360424+0000', 'last_change': '2024-06-29T15:57:33.360424+0000', 'last_active': '2024-06-29T15:57:33.339954+0000', 'last_peered': '2024-06-29T15:57:33.339954+0000', 'last_clean': '2024-06-29T15:57:33.339954+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-06-29T15:57:33.360424+0000', 'last_undegraded': '2024-06-29T15:57:33.360424+0000', 'last_fullsized': '2024-06-29T15:57:33.360424+0000', 'mapping_epoch': 215, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 215, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'last_clean_scrub_stamp': '2024-06-29T15:57:33.339954+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-06-29T15:57:39.153 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-29T15:57:39.260 INFO:tasks.ceph.mds.d.smithi167.stderr:2024-06-29T15:57:39.260+0000 7fed41148640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.285+0000 7f624bd9e640 1 -- 172.21.15.81:0/2005065031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f624415e960 msgr2=0x7f624415ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.285+0000 7f624bd9e640 1 --2- 172.21.15.81:0/2005065031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f624415e960 0x7f624415ed70 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f62440b2dd0 tx=0x7f623c02a530 comp rx=0 tx=0).stop 2024-06-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.285+0000 7f624bd9e640 1 -- 172.21.15.81:0/2005065031 shutdown_connections 2024-06-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.285+0000 7f624bd9e640 1 --2- 172.21.15.81:0/2005065031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f624415e960 0x7f624415ed70 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.285+0000 7f624bd9e640 1 --2- 172.21.15.81:0/2005065031 >> v2:172.21.15.167:3300/0 conn(0x7f6244073030 0x7f6244073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.285+0000 7f624bd9e640 1 -- 172.21.15.81:0/2005065031 >> 172.21.15.81:0/2005065031 conn(0x7f624406ccc0 msgr2=0x7f62440b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.285+0000 7f624bd9e640 1 -- 172.21.15.81:0/2005065031 shutdown_connections 2024-06-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.285+0000 7f624bd9e640 1 -- 172.21.15.81:0/2005065031 wait complete. 2024-06-29T15:57:39.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.287+0000 7f624bd9e640 1 Processor -- start 2024-06-29T15:57:39.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.287+0000 7f624bd9e640 1 -- start start 2024-06-29T15:57:39.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f624bd9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6244073030 0x7f62440ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f624bd9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62441412f0 0x7f62440c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f624bd9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624415e960 0x7f62440c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f624bd9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62440b3210 con 0x7f6244073030 2024-06-29T15:57:39.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f624bd9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6244141740 con 0x7f624415e960 2024-06-29T15:57:39.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f624bd9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f62440b3390 con 0x7f62441412f0 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f6249b3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6244073030 0x7f62440ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f6249b3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6244073030 0x7f62440ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35490/0 (socket says 172.21.15.81:35490) 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f6249b3a640 1 -- 172.21.15.81:0/2608539944 learned_addr learned my addr 172.21.15.81:0/2608539944 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f6249339640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62441412f0 0x7f62440c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f624a33b640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624415e960 0x7f62440c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f6249b3a640 1 -- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62441412f0 msgr2=0x7f62440c5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f6249b3a640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62441412f0 0x7f62440c5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f6249b3a640 1 -- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624415e960 msgr2=0x7f62440c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f6249b3a640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624415e960 0x7f62440c59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.288+0000 7f6249b3a640 1 -- 172.21.15.81:0/2608539944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62440aca50 con 0x7f6244073030 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.289+0000 7f6249b3a640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6244073030 0x7f62440ae040 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f623400c7d0 tx=0x7f623400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.289+0000 7f62427fc640 1 -- 172.21.15.81:0/2608539944 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6234017070 con 0x7f6244073030 2024-06-29T15:57:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.289+0000 7f62427fc640 1 -- 172.21.15.81:0/2608539944 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6234005070 con 0x7f6244073030 2024-06-29T15:57:39.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.289+0000 7f62427fc640 1 -- 172.21.15.81:0/2608539944 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62340054a0 con 0x7f6244073030 2024-06-29T15:57:39.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.289+0000 7f624bd9e640 1 -- 172.21.15.81:0/2608539944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62440acd30 con 0x7f6244073030 2024-06-29T15:57:39.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.289+0000 7f624bd9e640 1 -- 172.21.15.81:0/2608539944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62440c4c80 con 0x7f6244073030 2024-06-29T15:57:39.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.290+0000 7f624bd9e640 1 -- 172.21.15.81:0/2608539944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62440b3910 con 0x7f6244073030 2024-06-29T15:57:39.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.290+0000 7f62427fc640 1 -- 172.21.15.81:0/2608539944 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f623401a070 con 0x7f6244073030 2024-06-29T15:57:39.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.291+0000 7f62427fc640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62300763f0 0x7f6230078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.291+0000 7f62427fc640 1 -- 172.21.15.81:0/2608539944 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7f6234097410 con 0x7f6244073030 2024-06-29T15:57:39.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.291+0000 7f6249339640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62300763f0 0x7f6230078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.292+0000 7f6249339640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62300763f0 0x7f6230078900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f624415a390 tx=0x7f6238008040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:39.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.295+0000 7f62427fc640 1 -- 172.21.15.81:0/2608539944 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6244073980 con 0x7f6244073030 2024-06-29T15:57:39.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.552+0000 7f624bd9e640 1 -- 172.21.15.81:0/2608539944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f624415c820 con 0x7f6244073030 2024-06-29T15:57:39.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.552+0000 7f62427fc640 1 -- 172.21.15.81:0/2608539944 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1253 (secure 0 0 0) 0x7f62340607f0 con 0x7f6244073030 2024-06-29T15:57:39.554 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:39.554 INFO:teuthology.orchestra.run.smithi081.stdout:{"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":5721,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":217,"num_osds":8,"num_up_osds":8,"osd_up_since":1719670950,"num_in_osds":8,"osd_in_since":1719670941,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129},{"state_name":"creating+peering","count":64}],"num_pgs":193,"num_pools":5,"num_objects":24,"data_bytes":594021,"bytes_used":423206912,"bytes_avail":772670906368,"bytes_total":773094113280,"inactive_pgs_ratio":0.33160620927810669,"write_bytes_sec":370,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":118,"btime":"2024-06-29T15:57:37:378022+0000","id":13,"up":1,"in":1,"max":1,"id":14,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":14,"rank":0,"name":"d","status":"up:active","gid":10114},{"filesystem_id":13,"rank":0,"name":"a","status":"up:active","gid":11226}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-06-29T15:56:51.994185+0000","services":{}},"progress_events":{}} 2024-06-29T15:57:39.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 -- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62300763f0 msgr2=0x7f6230078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62300763f0 0x7f6230078900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f624415a390 tx=0x7f6238008040 comp rx=0 tx=0).stop 2024-06-29T15:57:39.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 -- 172.21.15.81:0/2608539944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6244073030 msgr2=0x7f62440ae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6244073030 0x7f62440ae040 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f623400c7d0 tx=0x7f623400cc90 comp rx=0 tx=0).stop 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 -- 172.21.15.81:0/2608539944 shutdown_connections 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f62300763f0 0x7f6230078900 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f624415e960 0x7f62440c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62441412f0 0x7f62440c5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 --2- 172.21.15.81:0/2608539944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6244073030 0x7f62440ae040 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 -- 172.21.15.81:0/2608539944 >> 172.21.15.81:0/2608539944 conn(0x7f624406ccc0 msgr2=0x7f624415aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 -- 172.21.15.81:0/2608539944 shutdown_connections 2024-06-29T15:57:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.555+0000 7f624bd9e640 1 -- 172.21.15.81:0/2608539944 wait complete. 2024-06-29T15:57:39.567 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-06-29T15:57:39.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:39.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.681+0000 7fd8367fc640 1 -- 172.21.15.81:0/1266276088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd818004d40 msgr2=0x7fd818005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.681+0000 7fd8367fc640 1 --2- 172.21.15.81:0/1266276088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd818004d40 0x7fd818005150 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fd82800c640 tx=0x7fd818005830 comp rx=0 tx=0).stop 2024-06-29T15:57:39.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.681+0000 7fd834d99640 1 -- 172.21.15.81:0/1266276088 reap_dead start 2024-06-29T15:57:39.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.681+0000 7fd8367fc640 1 -- 172.21.15.81:0/1266276088 shutdown_connections 2024-06-29T15:57:39.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.681+0000 7fd8367fc640 1 -- 172.21.15.81:0/1266276088 >> 172.21.15.81:0/1266276088 conn(0x7fd8300d0930 msgr2=0x7fd8300d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.681+0000 7fd8367fc640 1 -- 172.21.15.81:0/1266276088 shutdown_connections 2024-06-29T15:57:39.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.681+0000 7fd8367fc640 1 -- 172.21.15.81:0/1266276088 wait complete. 2024-06-29T15:57:39.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.683+0000 7fd8367fc640 1 Processor -- start 2024-06-29T15:57:39.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.683+0000 7fd8367fc640 1 -- start start 2024-06-29T15:57:39.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd8367fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300cca60 0x7fd83014cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd8367fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8301535c0 0x7fd830158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd8367fc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8301539d0 0x7fd830159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd8367fc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd83015aed0 con 0x7fd8300cca60 2024-06-29T15:57:39.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd8367fc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd8300b4c00 con 0x7fd8301535c0 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd8367fc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd83015b050 con 0x7fd8301539d0 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300cca60 0x7fd83014cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300cca60 0x7fd83014cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35500/0 (socket says 172.21.15.81:35500) 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82ffff640 1 -- 172.21.15.81:0/369085861 learned_addr learned my addr 172.21.15.81:0/369085861 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82f7fe640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8301535c0 0x7fd830158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd834d99640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8301539d0 0x7fd830159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82ffff640 1 -- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8301539d0 msgr2=0x7fd830159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82ffff640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8301539d0 0x7fd830159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82ffff640 1 -- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8301535c0 msgr2=0x7fd830158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82ffff640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8301535c0 0x7fd830158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82ffff640 1 -- 172.21.15.81:0/369085861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8300cb9c0 con 0x7fd8300cca60 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82f7fe640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8301535c0 0x7fd830158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82ffff640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300cca60 0x7fd83014cf40 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fd820011a50 tx=0x7fd820011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:39.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82cff9640 1 -- 172.21.15.81:0/369085861 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd82001f070 con 0x7fd8300cca60 2024-06-29T15:57:39.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82cff9640 1 -- 172.21.15.81:0/369085861 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd82000fdc0 con 0x7fd8300cca60 2024-06-29T15:57:39.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd82cff9640 1 -- 172.21.15.81:0/369085861 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8200126b0 con 0x7fd8300cca60 2024-06-29T15:57:39.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.685+0000 7fd8367fc640 1 -- 172.21.15.81:0/369085861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8300cbca0 con 0x7fd8300cca60 2024-06-29T15:57:39.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.686+0000 7fd8367fc640 1 -- 172.21.15.81:0/369085861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8300cc260 con 0x7fd8300cca60 2024-06-29T15:57:39.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.687+0000 7fd8367fc640 1 -- 172.21.15.81:0/369085861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd83006e2a0 con 0x7fd8300cca60 2024-06-29T15:57:39.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.687+0000 7fd82cff9640 1 -- 172.21.15.81:0/369085861 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fd820022070 con 0x7fd8300cca60 2024-06-29T15:57:39.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.688+0000 7fd82cff9640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd81c0763f0 0x7fd81c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:39.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.688+0000 7fd82cff9640 1 -- 172.21.15.81:0/369085861 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7fd8200987c0 con 0x7fd8300cca60 2024-06-29T15:57:39.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.688+0000 7fd82f7fe640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd81c0763f0 0x7fd81c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:39.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.689+0000 7fd82f7fe640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd81c0763f0 0x7fd81c078900 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd8300b11a0 tx=0x7fd824005f90 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:39.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.692+0000 7fd82cff9640 1 -- 172.21.15.81:0/369085861 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd83006e2a0 con 0x7fd8300cca60 2024-06-29T15:57:39.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.870+0000 7fd8367fc640 1 -- 172.21.15.81:0/369085861 --> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd83006f3a0 con 0x7fd81c0763f0 2024-06-29T15:57:39.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.878+0000 7fd82cff9640 1 -- 172.21.15.81:0/369085861 <== mgr.8666 v2:172.21.15.167:6832/2018755589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+486086 (secure 0 0 0) 0x7fd83006f3a0 con 0x7fd81c0763f0 2024-06-29T15:57:39.884 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:39.888 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:57:39.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 -- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd81c0763f0 msgr2=0x7fd81c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd81c0763f0 0x7fd81c078900 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd8300b11a0 tx=0x7fd824005f90 comp rx=0 tx=0).stop 2024-06-29T15:57:39.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 -- 172.21.15.81:0/369085861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300cca60 msgr2=0x7fd83014cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:39.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300cca60 0x7fd83014cf40 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fd820011a50 tx=0x7fd820011f10 comp rx=0 tx=0).stop 2024-06-29T15:57:39.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 -- 172.21.15.81:0/369085861 shutdown_connections 2024-06-29T15:57:39.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fd81c0763f0 0x7fd81c078900 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd8301539d0 0x7fd830159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd8301535c0 0x7fd830158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 --2- 172.21.15.81:0/369085861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300cca60 0x7fd83014cf40 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:39.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 -- 172.21.15.81:0/369085861 >> 172.21.15.81:0/369085861 conn(0x7fd8300d0930 msgr2=0x7fd830070b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:39.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 -- 172.21.15.81:0/369085861 shutdown_connections 2024-06-29T15:57:39.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:39.882+0000 7fd8367fc640 1 -- 172.21.15.81:0/369085861 wait complete. 2024-06-29T15:57:39.894 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-06-29T15:57:37.873180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594021,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":400,"num_write_kb":1644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":217,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413288,"kb_used_data":22276,"kb_used_omap":12,"kb_used_meta":390899,"kb_avail":754561432,"statfs":{"total":773094113280,"available":772670906368,"internally_reserved":0,"allocated":22810624,"data_stored":11734666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":400281138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1243,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.532321"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357377+0000","last_change":"2024-06-29T15:57:33.357377+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357377+0000","last_undegraded":"2024-06-29T15:57:33.357377+0000","last_fullsized":"2024-06-29T15:57:33.357377+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"202'219","reported_seq":1043,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363727+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:57:33.363727+0000","last_peered":"2024-06-29T15:57:33.363727+0000","last_clean":"2024-06-29T15:57:33.363727+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:57:33.363727+0000","last_undegraded":"2024-06-29T15:57:33.363727+0000","last_fullsized":"2024-06-29T15:57:33.363727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365224+0000","last_change":"2024-06-29T15:57:21.282026+0000","last_active":"2024-06-29T15:57:33.365224+0000","last_peered":"2024-06-29T15:57:33.365224+0000","last_clean":"2024-06-29T15:57:33.365224+0000","last_became_active":"2024-06-29T15:57:21.281709+0000","last_became_peered":"2024-06-29T15:57:21.281709+0000","last_unstale":"2024-06-29T15:57:33.365224+0000","last_undegraded":"2024-06-29T15:57:33.365224+0000","last_fullsized":"2024-06-29T15:57:33.365224+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:54:07.641300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354339+0000","last_change":"2024-06-29T15:57:22.302267+0000","last_active":"2024-06-29T15:57:33.354339+0000","last_peered":"2024-06-29T15:57:33.354339+0000","last_clean":"2024-06-29T15:57:33.354339+0000","last_became_active":"2024-06-29T15:57:22.301972+0000","last_became_peered":"2024-06-29T15:57:22.301972+0000","last_unstale":"2024-06-29T15:57:33.354339+0000","last_undegraded":"2024-06-29T15:57:33.354339+0000","last_fullsized":"2024-06-29T15:57:33.354339+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371713+0000","last_change":"2024-06-29T15:57:33.370756+0000","last_active":"2024-06-29T15:57:33.371713+0000","last_peered":"2024-06-29T15:57:33.371713+0000","last_clean":"2024-06-29T15:57:33.371713+0000","last_became_active":"2024-06-29T15:57:33.370363+0000","last_became_peered":"2024-06-29T15:57:33.370363+0000","last_unstale":"2024-06-29T15:57:33.371713+0000","last_undegraded":"2024-06-29T15:57:33.371713+0000","last_fullsized":"2024-06-29T15:57:33.371713+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:27.582662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362213+0000","last_change":"2024-06-29T15:57:33.362213+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362213+0000","last_undegraded":"2024-06-29T15:57:33.362213+0000","last_fullsized":"2024-06-29T15:57:33.362213+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364223+0000","last_change":"2024-06-29T15:57:21.283495+0000","last_active":"2024-06-29T15:57:33.364223+0000","last_peered":"2024-06-29T15:57:33.364223+0000","last_clean":"2024-06-29T15:57:33.364223+0000","last_became_active":"2024-06-29T15:57:21.283125+0000","last_became_peered":"2024-06-29T15:57:21.283125+0000","last_unstale":"2024-06-29T15:57:33.364223+0000","last_undegraded":"2024-06-29T15:57:33.364223+0000","last_fullsized":"2024-06-29T15:57:33.364223+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354722+0000","last_change":"2024-06-29T15:57:22.292872+0000","last_active":"2024-06-29T15:57:33.354722+0000","last_peered":"2024-06-29T15:57:33.354722+0000","last_clean":"2024-06-29T15:57:33.354722+0000","last_became_active":"2024-06-29T15:57:22.292485+0000","last_became_peered":"2024-06-29T15:57:22.292485+0000","last_unstale":"2024-06-29T15:57:33.354722+0000","last_undegraded":"2024-06-29T15:57:33.354722+0000","last_fullsized":"2024-06-29T15:57:33.354722+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:19.369559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370805+0000","last_change":"2024-06-29T15:57:33.369465+0000","last_active":"2024-06-29T15:57:33.370805+0000","last_peered":"2024-06-29T15:57:33.370805+0000","last_clean":"2024-06-29T15:57:33.370805+0000","last_became_active":"2024-06-29T15:57:33.369174+0000","last_became_peered":"2024-06-29T15:57:33.369174+0000","last_unstale":"2024-06-29T15:57:33.370805+0000","last_undegraded":"2024-06-29T15:57:33.370805+0000","last_fullsized":"2024-06-29T15:57:33.370805+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:02.836842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351888+0000","last_change":"2024-06-29T15:57:33.351888+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351888+0000","last_undegraded":"2024-06-29T15:57:33.351888+0000","last_fullsized":"2024-06-29T15:57:33.351888+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354196+0000","last_change":"2024-06-29T15:57:21.274359+0000","last_active":"2024-06-29T15:57:33.354196+0000","last_peered":"2024-06-29T15:57:33.354196+0000","last_clean":"2024-06-29T15:57:33.354196+0000","last_became_active":"2024-06-29T15:57:21.274036+0000","last_became_peered":"2024-06-29T15:57:21.274036+0000","last_unstale":"2024-06-29T15:57:33.354196+0000","last_undegraded":"2024-06-29T15:57:33.354196+0000","last_fullsized":"2024-06-29T15:57:33.354196+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:02.747066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358900+0000","last_change":"2024-06-29T15:57:22.295101+0000","last_active":"2024-06-29T15:57:33.358900+0000","last_peered":"2024-06-29T15:57:33.358900+0000","last_clean":"2024-06-29T15:57:33.358900+0000","last_became_active":"2024-06-29T15:57:22.294805+0000","last_became_peered":"2024-06-29T15:57:22.294805+0000","last_unstale":"2024-06-29T15:57:33.358900+0000","last_undegraded":"2024-06-29T15:57:33.358900+0000","last_fullsized":"2024-06-29T15:57:33.358900+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.369924+0000","last_change":"2024-06-29T15:57:33.369487+0000","last_active":"2024-06-29T15:57:33.369924+0000","last_peered":"2024-06-29T15:57:33.369924+0000","last_clean":"2024-06-29T15:57:33.369924+0000","last_became_active":"2024-06-29T15:57:33.369138+0000","last_became_peered":"2024-06-29T15:57:33.369138+0000","last_unstale":"2024-06-29T15:57:33.369924+0000","last_undegraded":"2024-06-29T15:57:33.369924+0000","last_fullsized":"2024-06-29T15:57:33.369924+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:06.322225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354530+0000","last_change":"2024-06-29T15:57:33.354530+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354530+0000","last_undegraded":"2024-06-29T15:57:33.354530+0000","last_fullsized":"2024-06-29T15:57:33.354530+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356787+0000","last_change":"2024-06-29T15:57:21.274551+0000","last_active":"2024-06-29T15:57:33.356787+0000","last_peered":"2024-06-29T15:57:33.356787+0000","last_clean":"2024-06-29T15:57:33.356787+0000","last_became_active":"2024-06-29T15:57:21.274029+0000","last_became_peered":"2024-06-29T15:57:21.274029+0000","last_unstale":"2024-06-29T15:57:33.356787+0000","last_undegraded":"2024-06-29T15:57:33.356787+0000","last_fullsized":"2024-06-29T15:57:33.356787+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364941+0000","last_change":"2024-06-29T15:57:22.291143+0000","last_active":"2024-06-29T15:57:33.364941+0000","last_peered":"2024-06-29T15:57:33.364941+0000","last_clean":"2024-06-29T15:57:33.364941+0000","last_became_active":"2024-06-29T15:57:22.290849+0000","last_became_peered":"2024-06-29T15:57:22.290849+0000","last_unstale":"2024-06-29T15:57:33.364941+0000","last_undegraded":"2024-06-29T15:57:33.364941+0000","last_fullsized":"2024-06-29T15:57:33.364941+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367095+0000","last_change":"2024-06-29T15:57:33.366161+0000","last_active":"2024-06-29T15:57:33.367095+0000","last_peered":"2024-06-29T15:57:33.367095+0000","last_clean":"2024-06-29T15:57:33.367095+0000","last_became_active":"2024-06-29T15:57:33.365820+0000","last_became_peered":"2024-06-29T15:57:33.365820+0000","last_unstale":"2024-06-29T15:57:33.367095+0000","last_undegraded":"2024-06-29T15:57:33.367095+0000","last_fullsized":"2024-06-29T15:57:33.367095+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:55:05.543032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358295+0000","last_change":"2024-06-29T15:57:33.358295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358295+0000","last_undegraded":"2024-06-29T15:57:33.358295+0000","last_fullsized":"2024-06-29T15:57:33.358295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361368+0000","last_change":"2024-06-29T15:57:21.281405+0000","last_active":"2024-06-29T15:57:33.361368+0000","last_peered":"2024-06-29T15:57:33.361368+0000","last_clean":"2024-06-29T15:57:33.361368+0000","last_became_active":"2024-06-29T15:57:21.281080+0000","last_became_peered":"2024-06-29T15:57:21.281080+0000","last_unstale":"2024-06-29T15:57:33.361368+0000","last_undegraded":"2024-06-29T15:57:33.361368+0000","last_fullsized":"2024-06-29T15:57:33.361368+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:51.116695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354452+0000","last_change":"2024-06-29T15:57:22.304057+0000","last_active":"2024-06-29T15:57:33.354452+0000","last_peered":"2024-06-29T15:57:33.354452+0000","last_clean":"2024-06-29T15:57:33.354452+0000","last_became_active":"2024-06-29T15:57:22.303762+0000","last_became_peered":"2024-06-29T15:57:22.303762+0000","last_unstale":"2024-06-29T15:57:33.354452+0000","last_undegraded":"2024-06-29T15:57:33.354452+0000","last_fullsized":"2024-06-29T15:57:33.354452+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:40.125779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370833+0000","last_change":"2024-06-29T15:57:33.369260+0000","last_active":"2024-06-29T15:57:33.370833+0000","last_peered":"2024-06-29T15:57:33.370833+0000","last_clean":"2024-06-29T15:57:33.370833+0000","last_became_active":"2024-06-29T15:57:33.368945+0000","last_became_peered":"2024-06-29T15:57:33.368945+0000","last_unstale":"2024-06-29T15:57:33.370833+0000","last_undegraded":"2024-06-29T15:57:33.370833+0000","last_fullsized":"2024-06-29T15:57:33.370833+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:11.821898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360097+0000","last_change":"2024-06-29T15:57:33.360097+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360097+0000","last_undegraded":"2024-06-29T15:57:33.360097+0000","last_fullsized":"2024-06-29T15:57:33.360097+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357223+0000","last_change":"2024-06-29T15:57:21.277560+0000","last_active":"2024-06-29T15:57:33.357223+0000","last_peered":"2024-06-29T15:57:33.357223+0000","last_clean":"2024-06-29T15:57:33.357223+0000","last_became_active":"2024-06-29T15:57:21.277255+0000","last_became_peered":"2024-06-29T15:57:21.277255+0000","last_unstale":"2024-06-29T15:57:33.357223+0000","last_undegraded":"2024-06-29T15:57:33.357223+0000","last_fullsized":"2024-06-29T15:57:33.357223+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354119+0000","last_change":"2024-06-29T15:57:22.301166+0000","last_active":"2024-06-29T15:57:33.354119+0000","last_peered":"2024-06-29T15:57:33.354119+0000","last_clean":"2024-06-29T15:57:33.354119+0000","last_became_active":"2024-06-29T15:57:22.300676+0000","last_became_peered":"2024-06-29T15:57:22.300676+0000","last_unstale":"2024-06-29T15:57:33.354119+0000","last_undegraded":"2024-06-29T15:57:33.354119+0000","last_fullsized":"2024-06-29T15:57:33.354119+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363490+0000","last_change":"2024-06-29T15:57:33.362184+0000","last_active":"2024-06-29T15:57:33.363490+0000","last_peered":"2024-06-29T15:57:33.363490+0000","last_clean":"2024-06-29T15:57:33.363490+0000","last_became_active":"2024-06-29T15:57:33.361905+0000","last_became_peered":"2024-06-29T15:57:33.361905+0000","last_unstale":"2024-06-29T15:57:33.363490+0000","last_undegraded":"2024-06-29T15:57:33.363490+0000","last_fullsized":"2024-06-29T15:57:33.363490+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:09:04.833173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349702+0000","last_change":"2024-06-29T15:57:33.349702+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349702+0000","last_undegraded":"2024-06-29T15:57:33.349702+0000","last_fullsized":"2024-06-29T15:57:33.349702+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357105+0000","last_change":"2024-06-29T15:57:21.280224+0000","last_active":"2024-06-29T15:57:33.357105+0000","last_peered":"2024-06-29T15:57:33.357105+0000","last_clean":"2024-06-29T15:57:33.357105+0000","last_became_active":"2024-06-29T15:57:21.279899+0000","last_became_peered":"2024-06-29T15:57:21.279899+0000","last_unstale":"2024-06-29T15:57:33.357105+0000","last_undegraded":"2024-06-29T15:57:33.357105+0000","last_fullsized":"2024-06-29T15:57:33.357105+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355301+0000","last_change":"2024-06-29T15:57:22.296472+0000","last_active":"2024-06-29T15:57:33.355301+0000","last_peered":"2024-06-29T15:57:33.355301+0000","last_clean":"2024-06-29T15:57:33.355301+0000","last_became_active":"2024-06-29T15:57:22.296130+0000","last_became_peered":"2024-06-29T15:57:22.296130+0000","last_unstale":"2024-06-29T15:57:33.355301+0000","last_undegraded":"2024-06-29T15:57:33.355301+0000","last_fullsized":"2024-06-29T15:57:33.355301+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368843+0000","last_change":"2024-06-29T15:57:33.367380+0000","last_active":"2024-06-29T15:57:33.368843+0000","last_peered":"2024-06-29T15:57:33.368843+0000","last_clean":"2024-06-29T15:57:33.368843+0000","last_became_active":"2024-06-29T15:57:33.367054+0000","last_became_peered":"2024-06-29T15:57:33.367054+0000","last_unstale":"2024-06-29T15:57:33.368843+0000","last_undegraded":"2024-06-29T15:57:33.368843+0000","last_fullsized":"2024-06-29T15:57:33.368843+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:29.345255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357030+0000","last_change":"2024-06-29T15:57:33.357030+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357030+0000","last_undegraded":"2024-06-29T15:57:33.357030+0000","last_fullsized":"2024-06-29T15:57:33.357030+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362303+0000","last_change":"2024-06-29T15:57:21.276972+0000","last_active":"2024-06-29T15:57:33.362303+0000","last_peered":"2024-06-29T15:57:33.362303+0000","last_clean":"2024-06-29T15:57:33.362303+0000","last_became_active":"2024-06-29T15:57:21.276647+0000","last_became_peered":"2024-06-29T15:57:21.276647+0000","last_unstale":"2024-06-29T15:57:33.362303+0000","last_undegraded":"2024-06-29T15:57:33.362303+0000","last_fullsized":"2024-06-29T15:57:33.362303+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:24.871281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355716+0000","last_change":"2024-06-29T15:57:22.303802+0000","last_active":"2024-06-29T15:57:33.355716+0000","last_peered":"2024-06-29T15:57:33.355716+0000","last_clean":"2024-06-29T15:57:33.355716+0000","last_became_active":"2024-06-29T15:57:22.303511+0000","last_became_peered":"2024-06-29T15:57:22.303511+0000","last_unstale":"2024-06-29T15:57:33.355716+0000","last_undegraded":"2024-06-29T15:57:33.355716+0000","last_fullsized":"2024-06-29T15:57:33.355716+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:10.536799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371208+0000","last_change":"2024-06-29T15:57:33.367162+0000","last_active":"2024-06-29T15:57:33.371208+0000","last_peered":"2024-06-29T15:57:33.371208+0000","last_clean":"2024-06-29T15:57:33.371208+0000","last_became_active":"2024-06-29T15:57:33.366816+0000","last_became_peered":"2024-06-29T15:57:33.366816+0000","last_unstale":"2024-06-29T15:57:33.371208+0000","last_undegraded":"2024-06-29T15:57:33.371208+0000","last_fullsized":"2024-06-29T15:57:33.371208+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:41.683812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.363662+0000","last_change":"2024-06-29T15:57:33.363662+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.363662+0000","last_undegraded":"2024-06-29T15:57:33.363662+0000","last_fullsized":"2024-06-29T15:57:33.363662+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358022+0000","last_change":"2024-06-29T15:57:21.277662+0000","last_active":"2024-06-29T15:57:33.358022+0000","last_peered":"2024-06-29T15:57:33.358022+0000","last_clean":"2024-06-29T15:57:33.358022+0000","last_became_active":"2024-06-29T15:57:21.277335+0000","last_became_peered":"2024-06-29T15:57:21.277335+0000","last_unstale":"2024-06-29T15:57:33.358022+0000","last_undegraded":"2024-06-29T15:57:33.358022+0000","last_fullsized":"2024-06-29T15:57:33.358022+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364773+0000","last_change":"2024-06-29T15:57:22.298914+0000","last_active":"2024-06-29T15:57:33.364773+0000","last_peered":"2024-06-29T15:57:33.364773+0000","last_clean":"2024-06-29T15:57:33.364773+0000","last_became_active":"2024-06-29T15:57:22.298614+0000","last_became_peered":"2024-06-29T15:57:22.298614+0000","last_unstale":"2024-06-29T15:57:33.364773+0000","last_undegraded":"2024-06-29T15:57:33.364773+0000","last_fullsized":"2024-06-29T15:57:33.364773+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367924+0000","last_change":"2024-06-29T15:57:33.367133+0000","last_active":"2024-06-29T15:57:33.367924+0000","last_peered":"2024-06-29T15:57:33.367924+0000","last_clean":"2024-06-29T15:57:33.367924+0000","last_became_active":"2024-06-29T15:57:33.366782+0000","last_became_peered":"2024-06-29T15:57:33.366782+0000","last_unstale":"2024-06-29T15:57:33.367924+0000","last_undegraded":"2024-06-29T15:57:33.367924+0000","last_fullsized":"2024-06-29T15:57:33.367924+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351295+0000","last_change":"2024-06-29T15:57:33.351295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351295+0000","last_undegraded":"2024-06-29T15:57:33.351295+0000","last_fullsized":"2024-06-29T15:57:33.351295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355165+0000","last_change":"2024-06-29T15:57:21.278011+0000","last_active":"2024-06-29T15:57:33.355165+0000","last_peered":"2024-06-29T15:57:33.355165+0000","last_clean":"2024-06-29T15:57:33.355165+0000","last_became_active":"2024-06-29T15:57:21.276648+0000","last_became_peered":"2024-06-29T15:57:21.276648+0000","last_unstale":"2024-06-29T15:57:33.355165+0000","last_undegraded":"2024-06-29T15:57:33.355165+0000","last_fullsized":"2024-06-29T15:57:33.355165+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354181+0000","last_change":"2024-06-29T15:57:22.300220+0000","last_active":"2024-06-29T15:57:33.354181+0000","last_peered":"2024-06-29T15:57:33.354181+0000","last_clean":"2024-06-29T15:57:33.354181+0000","last_became_active":"2024-06-29T15:57:22.299880+0000","last_became_peered":"2024-06-29T15:57:22.299880+0000","last_unstale":"2024-06-29T15:57:33.354181+0000","last_undegraded":"2024-06-29T15:57:33.354181+0000","last_fullsized":"2024-06-29T15:57:33.354181+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368552+0000","last_change":"2024-06-29T15:57:33.367636+0000","last_active":"2024-06-29T15:57:33.368552+0000","last_peered":"2024-06-29T15:57:33.368552+0000","last_clean":"2024-06-29T15:57:33.368552+0000","last_became_active":"2024-06-29T15:57:33.367321+0000","last_became_peered":"2024-06-29T15:57:33.367321+0000","last_unstale":"2024-06-29T15:57:33.368552+0000","last_undegraded":"2024-06-29T15:57:33.368552+0000","last_fullsized":"2024-06-29T15:57:33.368552+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:28.846172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353695+0000","last_change":"2024-06-29T15:57:33.353695+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353695+0000","last_undegraded":"2024-06-29T15:57:33.353695+0000","last_fullsized":"2024-06-29T15:57:33.353695+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357156+0000","last_change":"2024-06-29T15:57:21.284982+0000","last_active":"2024-06-29T15:57:33.357156+0000","last_peered":"2024-06-29T15:57:33.357156+0000","last_clean":"2024-06-29T15:57:33.357156+0000","last_became_active":"2024-06-29T15:57:21.284684+0000","last_became_peered":"2024-06-29T15:57:21.284684+0000","last_unstale":"2024-06-29T15:57:33.357156+0000","last_undegraded":"2024-06-29T15:57:33.357156+0000","last_fullsized":"2024-06-29T15:57:33.357156+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354906+0000","last_change":"2024-06-29T15:57:22.298744+0000","last_active":"2024-06-29T15:57:33.354906+0000","last_peered":"2024-06-29T15:57:33.354906+0000","last_clean":"2024-06-29T15:57:33.354906+0000","last_became_active":"2024-06-29T15:57:22.296925+0000","last_became_peered":"2024-06-29T15:57:22.296925+0000","last_unstale":"2024-06-29T15:57:33.354906+0000","last_undegraded":"2024-06-29T15:57:33.354906+0000","last_fullsized":"2024-06-29T15:57:33.354906+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:40.578182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371721+0000","last_change":"2024-06-29T15:57:33.368936+0000","last_active":"2024-06-29T15:57:33.371721+0000","last_peered":"2024-06-29T15:57:33.371721+0000","last_clean":"2024-06-29T15:57:33.371721+0000","last_became_active":"2024-06-29T15:57:33.368600+0000","last_became_peered":"2024-06-29T15:57:33.368600+0000","last_unstale":"2024-06-29T15:57:33.371721+0000","last_undegraded":"2024-06-29T15:57:33.371721+0000","last_fullsized":"2024-06-29T15:57:33.371721+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:22.830532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.361651+0000","last_change":"2024-06-29T15:57:33.361651+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.361651+0000","last_undegraded":"2024-06-29T15:57:33.361651+0000","last_fullsized":"2024-06-29T15:57:33.361651+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355165+0000","last_change":"2024-06-29T15:57:21.284780+0000","last_active":"2024-06-29T15:57:33.355165+0000","last_peered":"2024-06-29T15:57:33.355165+0000","last_clean":"2024-06-29T15:57:33.355165+0000","last_became_active":"2024-06-29T15:57:21.284273+0000","last_became_peered":"2024-06-29T15:57:21.284273+0000","last_unstale":"2024-06-29T15:57:33.355165+0000","last_undegraded":"2024-06-29T15:57:33.355165+0000","last_fullsized":"2024-06-29T15:57:33.355165+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:16.085990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356696+0000","last_change":"2024-06-29T15:57:22.291304+0000","last_active":"2024-06-29T15:57:33.356696+0000","last_peered":"2024-06-29T15:57:33.356696+0000","last_clean":"2024-06-29T15:57:33.356696+0000","last_became_active":"2024-06-29T15:57:22.290983+0000","last_became_peered":"2024-06-29T15:57:22.290983+0000","last_unstale":"2024-06-29T15:57:33.356696+0000","last_undegraded":"2024-06-29T15:57:33.356696+0000","last_fullsized":"2024-06-29T15:57:33.356696+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368254+0000","last_change":"2024-06-29T15:57:33.366553+0000","last_active":"2024-06-29T15:57:33.368254+0000","last_peered":"2024-06-29T15:57:33.368254+0000","last_clean":"2024-06-29T15:57:33.368254+0000","last_became_active":"2024-06-29T15:57:33.366230+0000","last_became_peered":"2024-06-29T15:57:33.366230+0000","last_unstale":"2024-06-29T15:57:33.368254+0000","last_undegraded":"2024-06-29T15:57:33.368254+0000","last_fullsized":"2024-06-29T15:57:33.368254+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:17:11.929406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358755+0000","last_change":"2024-06-29T15:57:33.358755+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358755+0000","last_undegraded":"2024-06-29T15:57:33.358755+0000","last_fullsized":"2024-06-29T15:57:33.358755+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348756+0000","last_change":"2024-06-29T15:57:21.284009+0000","last_active":"2024-06-29T15:57:33.348756+0000","last_peered":"2024-06-29T15:57:33.348756+0000","last_clean":"2024-06-29T15:57:33.348756+0000","last_became_active":"2024-06-29T15:57:21.283679+0000","last_became_peered":"2024-06-29T15:57:21.283679+0000","last_unstale":"2024-06-29T15:57:33.348756+0000","last_undegraded":"2024-06-29T15:57:33.348756+0000","last_fullsized":"2024-06-29T15:57:33.348756+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:20.284468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348277+0000","last_change":"2024-06-29T15:57:22.288958+0000","last_active":"2024-06-29T15:57:33.348277+0000","last_peered":"2024-06-29T15:57:33.348277+0000","last_clean":"2024-06-29T15:57:33.348277+0000","last_became_active":"2024-06-29T15:57:22.288621+0000","last_became_peered":"2024-06-29T15:57:22.288621+0000","last_unstale":"2024-06-29T15:57:33.348277+0000","last_undegraded":"2024-06-29T15:57:33.348277+0000","last_fullsized":"2024-06-29T15:57:33.348277+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:31.720751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371349+0000","last_change":"2024-06-29T15:57:33.370560+0000","last_active":"2024-06-29T15:57:33.371349+0000","last_peered":"2024-06-29T15:57:33.371349+0000","last_clean":"2024-06-29T15:57:33.371349+0000","last_became_active":"2024-06-29T15:57:33.370206+0000","last_became_peered":"2024-06-29T15:57:33.370206+0000","last_unstale":"2024-06-29T15:57:33.371349+0000","last_undegraded":"2024-06-29T15:57:33.371349+0000","last_fullsized":"2024-06-29T15:57:33.371349+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:12:47.692442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.361054+0000","last_change":"2024-06-29T15:57:33.361054+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.361054+0000","last_undegraded":"2024-06-29T15:57:33.361054+0000","last_fullsized":"2024-06-29T15:57:33.361054+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357091+0000","last_change":"2024-06-29T15:57:21.282111+0000","last_active":"2024-06-29T15:57:33.357091+0000","last_peered":"2024-06-29T15:57:33.357091+0000","last_clean":"2024-06-29T15:57:33.357091+0000","last_became_active":"2024-06-29T15:57:21.281761+0000","last_became_peered":"2024-06-29T15:57:21.281761+0000","last_unstale":"2024-06-29T15:57:33.357091+0000","last_undegraded":"2024-06-29T15:57:33.357091+0000","last_fullsized":"2024-06-29T15:57:33.357091+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348792+0000","last_change":"2024-06-29T15:57:22.305806+0000","last_active":"2024-06-29T15:57:33.348792+0000","last_peered":"2024-06-29T15:57:33.348792+0000","last_clean":"2024-06-29T15:57:33.348792+0000","last_became_active":"2024-06-29T15:57:22.296276+0000","last_became_peered":"2024-06-29T15:57:22.296276+0000","last_unstale":"2024-06-29T15:57:33.348792+0000","last_undegraded":"2024-06-29T15:57:33.348792+0000","last_fullsized":"2024-06-29T15:57:33.348792+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:43:51.754223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.369952+0000","last_change":"2024-06-29T15:57:33.367821+0000","last_active":"2024-06-29T15:57:33.369952+0000","last_peered":"2024-06-29T15:57:33.369952+0000","last_clean":"2024-06-29T15:57:33.369952+0000","last_became_active":"2024-06-29T15:57:33.367494+0000","last_became_peered":"2024-06-29T15:57:33.367494+0000","last_unstale":"2024-06-29T15:57:33.369952+0000","last_undegraded":"2024-06-29T15:57:33.369952+0000","last_fullsized":"2024-06-29T15:57:33.369952+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:46.165772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352248+0000","last_change":"2024-06-29T15:57:33.352248+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352248+0000","last_undegraded":"2024-06-29T15:57:33.352248+0000","last_fullsized":"2024-06-29T15:57:33.352248+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.349023+0000","last_change":"2024-06-29T15:57:21.279139+0000","last_active":"2024-06-29T15:57:33.349023+0000","last_peered":"2024-06-29T15:57:33.349023+0000","last_clean":"2024-06-29T15:57:33.349023+0000","last_became_active":"2024-06-29T15:57:21.278488+0000","last_became_peered":"2024-06-29T15:57:21.278488+0000","last_unstale":"2024-06-29T15:57:33.349023+0000","last_undegraded":"2024-06-29T15:57:33.349023+0000","last_fullsized":"2024-06-29T15:57:33.349023+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:54.126338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356194+0000","last_change":"2024-06-29T15:57:22.297275+0000","last_active":"2024-06-29T15:57:33.356194+0000","last_peered":"2024-06-29T15:57:33.356194+0000","last_clean":"2024-06-29T15:57:33.356194+0000","last_became_active":"2024-06-29T15:57:22.296984+0000","last_became_peered":"2024-06-29T15:57:22.296984+0000","last_unstale":"2024-06-29T15:57:33.356194+0000","last_undegraded":"2024-06-29T15:57:33.356194+0000","last_fullsized":"2024-06-29T15:57:33.356194+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371810+0000","last_change":"2024-06-29T15:57:33.369358+0000","last_active":"2024-06-29T15:57:33.371810+0000","last_peered":"2024-06-29T15:57:33.371810+0000","last_clean":"2024-06-29T15:57:33.371810+0000","last_became_active":"2024-06-29T15:57:33.369052+0000","last_became_peered":"2024-06-29T15:57:33.369052+0000","last_unstale":"2024-06-29T15:57:33.371810+0000","last_undegraded":"2024-06-29T15:57:33.371810+0000","last_fullsized":"2024-06-29T15:57:33.371810+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:29.345255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352493+0000","last_change":"2024-06-29T15:57:33.352493+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352493+0000","last_undegraded":"2024-06-29T15:57:33.352493+0000","last_fullsized":"2024-06-29T15:57:33.352493+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356473+0000","last_change":"2024-06-29T15:57:21.278476+0000","last_active":"2024-06-29T15:57:33.356473+0000","last_peered":"2024-06-29T15:57:33.356473+0000","last_clean":"2024-06-29T15:57:33.356473+0000","last_became_active":"2024-06-29T15:57:21.278150+0000","last_became_peered":"2024-06-29T15:57:21.278150+0000","last_unstale":"2024-06-29T15:57:33.356473+0000","last_undegraded":"2024-06-29T15:57:33.356473+0000","last_fullsized":"2024-06-29T15:57:33.356473+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:33.991332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355923+0000","last_change":"2024-06-29T15:57:22.291360+0000","last_active":"2024-06-29T15:57:33.355923+0000","last_peered":"2024-06-29T15:57:33.355923+0000","last_clean":"2024-06-29T15:57:33.355923+0000","last_became_active":"2024-06-29T15:57:22.290801+0000","last_became_peered":"2024-06-29T15:57:22.290801+0000","last_unstale":"2024-06-29T15:57:33.355923+0000","last_undegraded":"2024-06-29T15:57:33.355923+0000","last_fullsized":"2024-06-29T15:57:33.355923+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.371558+0000","last_change":"2024-06-29T15:57:33.367933+0000","last_active":"2024-06-29T15:57:33.371558+0000","last_peered":"2024-06-29T15:57:33.371558+0000","last_clean":"2024-06-29T15:57:33.371558+0000","last_became_active":"2024-06-29T15:57:33.367630+0000","last_became_peered":"2024-06-29T15:57:33.367630+0000","last_unstale":"2024-06-29T15:57:33.371558+0000","last_undegraded":"2024-06-29T15:57:33.371558+0000","last_fullsized":"2024-06-29T15:57:33.371558+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:01.393531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358484+0000","last_change":"2024-06-29T15:57:33.358484+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358484+0000","last_undegraded":"2024-06-29T15:57:33.358484+0000","last_fullsized":"2024-06-29T15:57:33.358484+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358778+0000","last_change":"2024-06-29T15:57:21.273304+0000","last_active":"2024-06-29T15:57:33.358778+0000","last_peered":"2024-06-29T15:57:33.358778+0000","last_clean":"2024-06-29T15:57:33.358778+0000","last_became_active":"2024-06-29T15:57:21.272972+0000","last_became_peered":"2024-06-29T15:57:21.272972+0000","last_unstale":"2024-06-29T15:57:33.358778+0000","last_undegraded":"2024-06-29T15:57:33.358778+0000","last_fullsized":"2024-06-29T15:57:33.358778+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356200+0000","last_change":"2024-06-29T15:57:22.300916+0000","last_active":"2024-06-29T15:57:33.356200+0000","last_peered":"2024-06-29T15:57:33.356200+0000","last_clean":"2024-06-29T15:57:33.356200+0000","last_became_active":"2024-06-29T15:57:22.300620+0000","last_became_peered":"2024-06-29T15:57:22.300620+0000","last_unstale":"2024-06-29T15:57:33.356200+0000","last_undegraded":"2024-06-29T15:57:33.356200+0000","last_fullsized":"2024-06-29T15:57:33.356200+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:41.172455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.368260+0000","last_change":"2024-06-29T15:57:33.367013+0000","last_active":"2024-06-29T15:57:33.368260+0000","last_peered":"2024-06-29T15:57:33.368260+0000","last_clean":"2024-06-29T15:57:33.368260+0000","last_became_active":"2024-06-29T15:57:33.366664+0000","last_became_peered":"2024-06-29T15:57:33.366664+0000","last_unstale":"2024-06-29T15:57:33.368260+0000","last_undegraded":"2024-06-29T15:57:33.368260+0000","last_fullsized":"2024-06-29T15:57:33.368260+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:22.830532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357163+0000","last_change":"2024-06-29T15:57:33.357163+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357163+0000","last_undegraded":"2024-06-29T15:57:33.357163+0000","last_fullsized":"2024-06-29T15:57:33.357163+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"212'8","reported_seq":26,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359054+0000","last_change":"2024-06-29T15:57:21.283745+0000","last_active":"2024-06-29T15:57:33.359054+0000","last_peered":"2024-06-29T15:57:33.359054+0000","last_clean":"2024-06-29T15:57:33.359054+0000","last_became_active":"2024-06-29T15:57:21.283403+0000","last_became_peered":"2024-06-29T15:57:21.283403+0000","last_unstale":"2024-06-29T15:57:33.359054+0000","last_undegraded":"2024-06-29T15:57:33.359054+0000","last_fullsized":"2024-06-29T15:57:33.359054+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357048+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:33.357048+0000","last_peered":"2024-06-29T15:57:33.357048+0000","last_clean":"2024-06-29T15:57:33.357048+0000","last_became_active":"2024-06-29T15:57:22.301832+0000","last_became_peered":"2024-06-29T15:57:22.301832+0000","last_unstale":"2024-06-29T15:57:33.357048+0000","last_undegraded":"2024-06-29T15:57:33.357048+0000","last_fullsized":"2024-06-29T15:57:33.357048+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367778+0000","last_change":"2024-06-29T15:57:33.366032+0000","last_active":"2024-06-29T15:57:33.367778+0000","last_peered":"2024-06-29T15:57:33.367778+0000","last_clean":"2024-06-29T15:57:33.367778+0000","last_became_active":"2024-06-29T15:57:33.365731+0000","last_became_peered":"2024-06-29T15:57:33.365731+0000","last_unstale":"2024-06-29T15:57:33.367778+0000","last_undegraded":"2024-06-29T15:57:33.367778+0000","last_fullsized":"2024-06-29T15:57:33.367778+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:53:58.462755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349766+0000","last_change":"2024-06-29T15:57:33.349766+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349766+0000","last_undegraded":"2024-06-29T15:57:33.349766+0000","last_fullsized":"2024-06-29T15:57:33.349766+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348560+0000","last_change":"2024-06-29T15:57:21.276731+0000","last_active":"2024-06-29T15:57:33.348560+0000","last_peered":"2024-06-29T15:57:33.348560+0000","last_clean":"2024-06-29T15:57:33.348560+0000","last_became_active":"2024-06-29T15:57:21.276408+0000","last_became_peered":"2024-06-29T15:57:21.276408+0000","last_unstale":"2024-06-29T15:57:33.348560+0000","last_undegraded":"2024-06-29T15:57:33.348560+0000","last_fullsized":"2024-06-29T15:57:33.348560+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:48.911239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358795+0000","last_change":"2024-06-29T15:57:22.299631+0000","last_active":"2024-06-29T15:57:33.358795+0000","last_peered":"2024-06-29T15:57:33.358795+0000","last_clean":"2024-06-29T15:57:33.358795+0000","last_became_active":"2024-06-29T15:57:22.299285+0000","last_became_peered":"2024-06-29T15:57:22.299285+0000","last_unstale":"2024-06-29T15:57:33.358795+0000","last_undegraded":"2024-06-29T15:57:33.358795+0000","last_fullsized":"2024-06-29T15:57:33.358795+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370910+0000","last_change":"2024-06-29T15:57:33.369952+0000","last_active":"2024-06-29T15:57:33.370910+0000","last_peered":"2024-06-29T15:57:33.370910+0000","last_clean":"2024-06-29T15:57:33.370910+0000","last_became_active":"2024-06-29T15:57:33.369598+0000","last_became_peered":"2024-06-29T15:57:33.369598+0000","last_unstale":"2024-06-29T15:57:33.370910+0000","last_undegraded":"2024-06-29T15:57:33.370910+0000","last_fullsized":"2024-06-29T15:57:33.370910+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:24.974909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358887+0000","last_change":"2024-06-29T15:57:33.358887+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358887+0000","last_undegraded":"2024-06-29T15:57:33.358887+0000","last_fullsized":"2024-06-29T15:57:33.358887+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354289+0000","last_change":"2024-06-29T15:57:21.282872+0000","last_active":"2024-06-29T15:57:33.354289+0000","last_peered":"2024-06-29T15:57:33.354289+0000","last_clean":"2024-06-29T15:57:33.354289+0000","last_became_active":"2024-06-29T15:57:21.282549+0000","last_became_peered":"2024-06-29T15:57:21.282549+0000","last_unstale":"2024-06-29T15:57:33.354289+0000","last_undegraded":"2024-06-29T15:57:33.354289+0000","last_fullsized":"2024-06-29T15:57:33.354289+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348643+0000","last_change":"2024-06-29T15:57:22.294011+0000","last_active":"2024-06-29T15:57:33.348643+0000","last_peered":"2024-06-29T15:57:33.348643+0000","last_clean":"2024-06-29T15:57:33.348643+0000","last_became_active":"2024-06-29T15:57:22.293721+0000","last_became_peered":"2024-06-29T15:57:22.293721+0000","last_unstale":"2024-06-29T15:57:33.348643+0000","last_undegraded":"2024-06-29T15:57:33.348643+0000","last_fullsized":"2024-06-29T15:57:33.348643+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:57.337708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362703+0000","last_change":"2024-06-29T15:57:33.361883+0000","last_active":"2024-06-29T15:57:33.362703+0000","last_peered":"2024-06-29T15:57:33.362703+0000","last_clean":"2024-06-29T15:57:33.362703+0000","last_became_active":"2024-06-29T15:57:33.361569+0000","last_became_peered":"2024-06-29T15:57:33.361569+0000","last_unstale":"2024-06-29T15:57:33.362703+0000","last_undegraded":"2024-06-29T15:57:33.362703+0000","last_fullsized":"2024-06-29T15:57:33.362703+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:08.554869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351919+0000","last_change":"2024-06-29T15:57:33.351919+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351919+0000","last_undegraded":"2024-06-29T15:57:33.351919+0000","last_fullsized":"2024-06-29T15:57:33.351919+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353877+0000","last_change":"2024-06-29T15:57:21.281309+0000","last_active":"2024-06-29T15:57:33.353877+0000","last_peered":"2024-06-29T15:57:33.353877+0000","last_clean":"2024-06-29T15:57:33.353877+0000","last_became_active":"2024-06-29T15:57:21.280944+0000","last_became_peered":"2024-06-29T15:57:21.280944+0000","last_unstale":"2024-06-29T15:57:33.353877+0000","last_undegraded":"2024-06-29T15:57:33.353877+0000","last_fullsized":"2024-06-29T15:57:33.353877+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356544+0000","last_change":"2024-06-29T15:57:22.291164+0000","last_active":"2024-06-29T15:57:33.356544+0000","last_peered":"2024-06-29T15:57:33.356544+0000","last_clean":"2024-06-29T15:57:33.356544+0000","last_became_active":"2024-06-29T15:57:22.290897+0000","last_became_peered":"2024-06-29T15:57:22.290897+0000","last_unstale":"2024-06-29T15:57:33.356544+0000","last_undegraded":"2024-06-29T15:57:33.356544+0000","last_fullsized":"2024-06-29T15:57:33.356544+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364704+0000","last_change":"2024-06-29T15:57:33.364026+0000","last_active":"2024-06-29T15:57:33.364704+0000","last_peered":"2024-06-29T15:57:33.364704+0000","last_clean":"2024-06-29T15:57:33.364704+0000","last_became_active":"2024-06-29T15:57:33.363658+0000","last_became_peered":"2024-06-29T15:57:33.363658+0000","last_unstale":"2024-06-29T15:57:33.364704+0000","last_undegraded":"2024-06-29T15:57:33.364704+0000","last_fullsized":"2024-06-29T15:57:33.364704+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:41.322192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359617+0000","last_change":"2024-06-29T15:57:33.359617+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359617+0000","last_undegraded":"2024-06-29T15:57:33.359617+0000","last_fullsized":"2024-06-29T15:57:33.359617+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"214'8","reported_seq":26,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365217+0000","last_change":"2024-06-29T15:57:21.278161+0000","last_active":"2024-06-29T15:57:33.365217+0000","last_peered":"2024-06-29T15:57:33.365217+0000","last_clean":"2024-06-29T15:57:33.365217+0000","last_became_active":"2024-06-29T15:57:21.277794+0000","last_became_peered":"2024-06-29T15:57:21.277794+0000","last_unstale":"2024-06-29T15:57:33.365217+0000","last_undegraded":"2024-06-29T15:57:33.365217+0000","last_fullsized":"2024-06-29T15:57:33.365217+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:56.330168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.352292+0000","last_change":"2024-06-29T15:57:22.302666+0000","last_active":"2024-06-29T15:57:33.352292+0000","last_peered":"2024-06-29T15:57:33.352292+0000","last_clean":"2024-06-29T15:57:33.352292+0000","last_became_active":"2024-06-29T15:57:22.302254+0000","last_became_peered":"2024-06-29T15:57:22.302254+0000","last_unstale":"2024-06-29T15:57:33.352292+0000","last_undegraded":"2024-06-29T15:57:33.352292+0000","last_fullsized":"2024-06-29T15:57:33.352292+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363467+0000","last_change":"2024-06-29T15:57:33.361374+0000","last_active":"2024-06-29T15:57:33.363467+0000","last_peered":"2024-06-29T15:57:33.363467+0000","last_clean":"2024-06-29T15:57:33.363467+0000","last_became_active":"2024-06-29T15:57:33.361002+0000","last_became_peered":"2024-06-29T15:57:33.361002+0000","last_unstale":"2024-06-29T15:57:33.363467+0000","last_undegraded":"2024-06-29T15:57:33.363467+0000","last_fullsized":"2024-06-29T15:57:33.363467+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:47.234190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353505+0000","last_change":"2024-06-29T15:57:33.353505+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353505+0000","last_undegraded":"2024-06-29T15:57:33.353505+0000","last_fullsized":"2024-06-29T15:57:33.353505+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"212'3","reported_seq":21,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348900+0000","last_change":"2024-06-29T15:57:21.283791+0000","last_active":"2024-06-29T15:57:33.348900+0000","last_peered":"2024-06-29T15:57:33.348900+0000","last_clean":"2024-06-29T15:57:33.348900+0000","last_became_active":"2024-06-29T15:57:21.283458+0000","last_became_peered":"2024-06-29T15:57:21.283458+0000","last_unstale":"2024-06-29T15:57:33.348900+0000","last_undegraded":"2024-06-29T15:57:33.348900+0000","last_fullsized":"2024-06-29T15:57:33.348900+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:54.365881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361117+0000","last_change":"2024-06-29T15:57:22.295260+0000","last_active":"2024-06-29T15:57:33.361117+0000","last_peered":"2024-06-29T15:57:33.361117+0000","last_clean":"2024-06-29T15:57:33.361117+0000","last_became_active":"2024-06-29T15:57:22.294952+0000","last_became_peered":"2024-06-29T15:57:22.294952+0000","last_unstale":"2024-06-29T15:57:33.361117+0000","last_undegraded":"2024-06-29T15:57:33.361117+0000","last_fullsized":"2024-06-29T15:57:33.361117+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:56.473182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.370910+0000","last_change":"2024-06-29T15:57:33.370460+0000","last_active":"2024-06-29T15:57:33.370910+0000","last_peered":"2024-06-29T15:57:33.370910+0000","last_clean":"2024-06-29T15:57:33.370910+0000","last_became_active":"2024-06-29T15:57:33.370183+0000","last_became_peered":"2024-06-29T15:57:33.370183+0000","last_unstale":"2024-06-29T15:57:33.370910+0000","last_undegraded":"2024-06-29T15:57:33.370910+0000","last_fullsized":"2024-06-29T15:57:33.370910+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:06.322225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352295+0000","last_change":"2024-06-29T15:57:33.352295+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352295+0000","last_undegraded":"2024-06-29T15:57:33.352295+0000","last_fullsized":"2024-06-29T15:57:33.352295+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354550+0000","last_change":"2024-06-29T15:57:21.275926+0000","last_active":"2024-06-29T15:57:33.354550+0000","last_peered":"2024-06-29T15:57:33.354550+0000","last_clean":"2024-06-29T15:57:33.354550+0000","last_became_active":"2024-06-29T15:57:21.275593+0000","last_became_peered":"2024-06-29T15:57:21.275593+0000","last_unstale":"2024-06-29T15:57:33.354550+0000","last_undegraded":"2024-06-29T15:57:33.354550+0000","last_fullsized":"2024-06-29T15:57:33.354550+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.357897+0000","last_change":"2024-06-29T15:57:22.298454+0000","last_active":"2024-06-29T15:57:33.357897+0000","last_peered":"2024-06-29T15:57:33.357897+0000","last_clean":"2024-06-29T15:57:33.357897+0000","last_became_active":"2024-06-29T15:57:22.298162+0000","last_became_peered":"2024-06-29T15:57:22.298162+0000","last_unstale":"2024-06-29T15:57:33.357897+0000","last_undegraded":"2024-06-29T15:57:33.357897+0000","last_fullsized":"2024-06-29T15:57:33.357897+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363344+0000","last_change":"2024-06-29T15:57:33.362039+0000","last_active":"2024-06-29T15:57:33.363344+0000","last_peered":"2024-06-29T15:57:33.363344+0000","last_clean":"2024-06-29T15:57:33.363344+0000","last_became_active":"2024-06-29T15:57:33.361665+0000","last_became_peered":"2024-06-29T15:57:33.361665+0000","last_unstale":"2024-06-29T15:57:33.363344+0000","last_undegraded":"2024-06-29T15:57:33.363344+0000","last_fullsized":"2024-06-29T15:57:33.363344+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:41.322192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.358599+0000","last_change":"2024-06-29T15:57:33.358599+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.358599+0000","last_undegraded":"2024-06-29T15:57:33.358599+0000","last_fullsized":"2024-06-29T15:57:33.358599+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358654+0000","last_change":"2024-06-29T15:57:21.272019+0000","last_active":"2024-06-29T15:57:33.358654+0000","last_peered":"2024-06-29T15:57:33.358654+0000","last_clean":"2024-06-29T15:57:33.358654+0000","last_became_active":"2024-06-29T15:57:21.271715+0000","last_became_peered":"2024-06-29T15:57:21.271715+0000","last_unstale":"2024-06-29T15:57:33.358654+0000","last_undegraded":"2024-06-29T15:57:33.358654+0000","last_fullsized":"2024-06-29T15:57:33.358654+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362094+0000","last_change":"2024-06-29T15:57:22.303487+0000","last_active":"2024-06-29T15:57:33.362094+0000","last_peered":"2024-06-29T15:57:33.362094+0000","last_clean":"2024-06-29T15:57:33.362094+0000","last_became_active":"2024-06-29T15:57:22.302933+0000","last_became_peered":"2024-06-29T15:57:22.302933+0000","last_unstale":"2024-06-29T15:57:33.362094+0000","last_undegraded":"2024-06-29T15:57:33.362094+0000","last_fullsized":"2024-06-29T15:57:33.362094+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:28.493187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.362444+0000","last_change":"2024-06-29T15:57:33.361560+0000","last_active":"2024-06-29T15:57:33.362444+0000","last_peered":"2024-06-29T15:57:33.362444+0000","last_clean":"2024-06-29T15:57:33.362444+0000","last_became_active":"2024-06-29T15:57:33.361270+0000","last_became_peered":"2024-06-29T15:57:33.361270+0000","last_unstale":"2024-06-29T15:57:33.362444+0000","last_undegraded":"2024-06-29T15:57:33.362444+0000","last_fullsized":"2024-06-29T15:57:33.362444+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:51.198488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356450+0000","last_change":"2024-06-29T15:57:33.356450+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356450+0000","last_undegraded":"2024-06-29T15:57:33.356450+0000","last_fullsized":"2024-06-29T15:57:33.356450+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364019+0000","last_change":"2024-06-29T15:57:33.362045+0000","last_active":"2024-06-29T15:57:33.364019+0000","last_peered":"2024-06-29T15:57:33.364019+0000","last_clean":"2024-06-29T15:57:33.364019+0000","last_became_active":"2024-06-29T15:57:33.361647+0000","last_became_peered":"2024-06-29T15:57:33.361647+0000","last_unstale":"2024-06-29T15:57:33.364019+0000","last_undegraded":"2024-06-29T15:57:33.364019+0000","last_fullsized":"2024-06-29T15:57:33.364019+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:51.198488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353878+0000","last_change":"2024-06-29T15:57:21.277245+0000","last_active":"2024-06-29T15:57:33.353878+0000","last_peered":"2024-06-29T15:57:33.353878+0000","last_clean":"2024-06-29T15:57:33.353878+0000","last_became_active":"2024-06-29T15:57:21.276931+0000","last_became_peered":"2024-06-29T15:57:21.276931+0000","last_unstale":"2024-06-29T15:57:33.353878+0000","last_undegraded":"2024-06-29T15:57:33.353878+0000","last_fullsized":"2024-06-29T15:57:33.353878+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361080+0000","last_change":"2024-06-29T15:57:22.299457+0000","last_active":"2024-06-29T15:57:33.361080+0000","last_peered":"2024-06-29T15:57:33.361080+0000","last_clean":"2024-06-29T15:57:33.361080+0000","last_became_active":"2024-06-29T15:57:22.299157+0000","last_became_peered":"2024-06-29T15:57:22.299157+0000","last_unstale":"2024-06-29T15:57:33.361080+0000","last_undegraded":"2024-06-29T15:57:33.361080+0000","last_fullsized":"2024-06-29T15:57:33.361080+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:40.967292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356113+0000","last_change":"2024-06-29T15:57:33.356113+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356113+0000","last_undegraded":"2024-06-29T15:57:33.356113+0000","last_fullsized":"2024-06-29T15:57:33.356113+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363418+0000","last_change":"2024-06-29T15:57:33.360662+0000","last_active":"2024-06-29T15:57:33.363418+0000","last_peered":"2024-06-29T15:57:33.363418+0000","last_clean":"2024-06-29T15:57:33.363418+0000","last_became_active":"2024-06-29T15:57:33.360087+0000","last_became_peered":"2024-06-29T15:57:33.360087+0000","last_unstale":"2024-06-29T15:57:33.363418+0000","last_undegraded":"2024-06-29T15:57:33.363418+0000","last_fullsized":"2024-06-29T15:57:33.363418+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:46:09.697181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":18,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355322+0000","last_change":"2024-06-29T15:57:21.277884+0000","last_active":"2024-06-29T15:57:33.355322+0000","last_peered":"2024-06-29T15:57:33.355322+0000","last_clean":"2024-06-29T15:57:33.355322+0000","last_became_active":"2024-06-29T15:57:21.277589+0000","last_became_peered":"2024-06-29T15:57:21.277589+0000","last_unstale":"2024-06-29T15:57:33.355322+0000","last_undegraded":"2024-06-29T15:57:33.355322+0000","last_fullsized":"2024-06-29T15:57:33.355322+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:09.017396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355087+0000","last_change":"2024-06-29T15:57:22.293696+0000","last_active":"2024-06-29T15:57:33.355087+0000","last_peered":"2024-06-29T15:57:33.355087+0000","last_clean":"2024-06-29T15:57:33.355087+0000","last_became_active":"2024-06-29T15:57:22.293263+0000","last_became_peered":"2024-06-29T15:57:22.293263+0000","last_unstale":"2024-06-29T15:57:33.355087+0000","last_undegraded":"2024-06-29T15:57:33.355087+0000","last_fullsized":"2024-06-29T15:57:33.355087+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352369+0000","last_change":"2024-06-29T15:57:33.352369+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352369+0000","last_undegraded":"2024-06-29T15:57:33.352369+0000","last_fullsized":"2024-06-29T15:57:33.352369+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356661+0000","last_change":"2024-06-29T15:57:21.274306+0000","last_active":"2024-06-29T15:57:33.356661+0000","last_peered":"2024-06-29T15:57:33.356661+0000","last_clean":"2024-06-29T15:57:33.356661+0000","last_became_active":"2024-06-29T15:57:21.273632+0000","last_became_peered":"2024-06-29T15:57:21.273632+0000","last_unstale":"2024-06-29T15:57:33.356661+0000","last_undegraded":"2024-06-29T15:57:33.356661+0000","last_fullsized":"2024-06-29T15:57:33.356661+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361270+0000","last_change":"2024-06-29T15:57:22.297633+0000","last_active":"2024-06-29T15:57:33.361270+0000","last_peered":"2024-06-29T15:57:33.361270+0000","last_clean":"2024-06-29T15:57:33.361270+0000","last_became_active":"2024-06-29T15:57:22.297319+0000","last_became_peered":"2024-06-29T15:57:22.297319+0000","last_unstale":"2024-06-29T15:57:33.361270+0000","last_undegraded":"2024-06-29T15:57:33.361270+0000","last_fullsized":"2024-06-29T15:57:33.361270+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:19.625176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356850+0000","last_change":"2024-06-29T15:57:33.355194+0000","last_active":"2024-06-29T15:57:33.356850+0000","last_peered":"2024-06-29T15:57:33.356850+0000","last_clean":"2024-06-29T15:57:33.356850+0000","last_became_active":"2024-06-29T15:57:33.354883+0000","last_became_peered":"2024-06-29T15:57:33.354883+0000","last_unstale":"2024-06-29T15:57:33.356850+0000","last_undegraded":"2024-06-29T15:57:33.356850+0000","last_fullsized":"2024-06-29T15:57:33.356850+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:06.173920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351788+0000","last_change":"2024-06-29T15:57:33.351788+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351788+0000","last_undegraded":"2024-06-29T15:57:33.351788+0000","last_fullsized":"2024-06-29T15:57:33.351788+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361211+0000","last_change":"2024-06-29T15:57:21.283411+0000","last_active":"2024-06-29T15:57:33.361211+0000","last_peered":"2024-06-29T15:57:33.361211+0000","last_clean":"2024-06-29T15:57:33.361211+0000","last_became_active":"2024-06-29T15:57:21.283081+0000","last_became_peered":"2024-06-29T15:57:21.283081+0000","last_unstale":"2024-06-29T15:57:33.361211+0000","last_undegraded":"2024-06-29T15:57:33.361211+0000","last_fullsized":"2024-06-29T15:57:33.361211+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:27.866266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348680+0000","last_change":"2024-06-29T15:57:22.306835+0000","last_active":"2024-06-29T15:57:33.348680+0000","last_peered":"2024-06-29T15:57:33.348680+0000","last_clean":"2024-06-29T15:57:33.348680+0000","last_became_active":"2024-06-29T15:57:22.306295+0000","last_became_peered":"2024-06-29T15:57:22.306295+0000","last_unstale":"2024-06-29T15:57:33.348680+0000","last_undegraded":"2024-06-29T15:57:33.348680+0000","last_fullsized":"2024-06-29T15:57:33.348680+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:52.124235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.367249+0000","last_change":"2024-06-29T15:57:33.365463+0000","last_active":"2024-06-29T15:57:33.367249+0000","last_peered":"2024-06-29T15:57:33.367249+0000","last_clean":"2024-06-29T15:57:33.367249+0000","last_became_active":"2024-06-29T15:57:33.365149+0000","last_became_peered":"2024-06-29T15:57:33.365149+0000","last_unstale":"2024-06-29T15:57:33.367249+0000","last_undegraded":"2024-06-29T15:57:33.367249+0000","last_fullsized":"2024-06-29T15:57:33.367249+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:35.252184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356954+0000","last_change":"2024-06-29T15:57:33.356954+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356954+0000","last_undegraded":"2024-06-29T15:57:33.356954+0000","last_fullsized":"2024-06-29T15:57:33.356954+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"212'2","reported_seq":20,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356973+0000","last_change":"2024-06-29T15:57:21.284516+0000","last_active":"2024-06-29T15:57:33.356973+0000","last_peered":"2024-06-29T15:57:33.356973+0000","last_clean":"2024-06-29T15:57:33.356973+0000","last_became_active":"2024-06-29T15:57:21.284170+0000","last_became_peered":"2024-06-29T15:57:21.284170+0000","last_unstale":"2024-06-29T15:57:33.356973+0000","last_undegraded":"2024-06-29T15:57:33.356973+0000","last_fullsized":"2024-06-29T15:57:33.356973+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.352840+0000","last_change":"2024-06-29T15:57:22.300202+0000","last_active":"2024-06-29T15:57:33.352840+0000","last_peered":"2024-06-29T15:57:33.352840+0000","last_clean":"2024-06-29T15:57:33.352840+0000","last_became_active":"2024-06-29T15:57:22.299897+0000","last_became_peered":"2024-06-29T15:57:22.299897+0000","last_unstale":"2024-06-29T15:57:33.352840+0000","last_undegraded":"2024-06-29T15:57:33.352840+0000","last_fullsized":"2024-06-29T15:57:33.352840+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364931+0000","last_change":"2024-06-29T15:57:33.363355+0000","last_active":"2024-06-29T15:57:33.364931+0000","last_peered":"2024-06-29T15:57:33.364931+0000","last_clean":"2024-06-29T15:57:33.364931+0000","last_became_active":"2024-06-29T15:57:33.363067+0000","last_became_peered":"2024-06-29T15:57:33.363067+0000","last_unstale":"2024-06-29T15:57:33.364931+0000","last_undegraded":"2024-06-29T15:57:33.364931+0000","last_fullsized":"2024-06-29T15:57:33.364931+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351200+0000","last_change":"2024-06-29T15:57:33.351200+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351200+0000","last_undegraded":"2024-06-29T15:57:33.351200+0000","last_fullsized":"2024-06-29T15:57:33.351200+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"212'1","reported_seq":19,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356344+0000","last_change":"2024-06-29T15:57:21.274833+0000","last_active":"2024-06-29T15:57:33.356344+0000","last_peered":"2024-06-29T15:57:33.356344+0000","last_clean":"2024-06-29T15:57:33.356344+0000","last_became_active":"2024-06-29T15:57:21.274451+0000","last_became_peered":"2024-06-29T15:57:21.274451+0000","last_unstale":"2024-06-29T15:57:33.356344+0000","last_undegraded":"2024-06-29T15:57:33.356344+0000","last_fullsized":"2024-06-29T15:57:33.356344+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:25.284538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354475+0000","last_change":"2024-06-29T15:57:22.301084+0000","last_active":"2024-06-29T15:57:33.354475+0000","last_peered":"2024-06-29T15:57:33.354475+0000","last_clean":"2024-06-29T15:57:33.354475+0000","last_became_active":"2024-06-29T15:57:22.300639+0000","last_became_peered":"2024-06-29T15:57:22.300639+0000","last_unstale":"2024-06-29T15:57:33.354475+0000","last_undegraded":"2024-06-29T15:57:33.354475+0000","last_fullsized":"2024-06-29T15:57:33.354475+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361064+0000","last_change":"2024-06-29T15:57:33.358185+0000","last_active":"2024-06-29T15:57:33.361064+0000","last_peered":"2024-06-29T15:57:33.361064+0000","last_clean":"2024-06-29T15:57:33.361064+0000","last_became_active":"2024-06-29T15:57:33.357885+0000","last_became_peered":"2024-06-29T15:57:33.357885+0000","last_unstale":"2024-06-29T15:57:33.361064+0000","last_undegraded":"2024-06-29T15:57:33.361064+0000","last_fullsized":"2024-06-29T15:57:33.361064+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:50:30.619228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351153+0000","last_change":"2024-06-29T15:57:33.351153+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351153+0000","last_undegraded":"2024-06-29T15:57:33.351153+0000","last_fullsized":"2024-06-29T15:57:33.351153+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"212'4","reported_seq":22,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365147+0000","last_change":"2024-06-29T15:57:21.276327+0000","last_active":"2024-06-29T15:57:33.365147+0000","last_peered":"2024-06-29T15:57:33.365147+0000","last_clean":"2024-06-29T15:57:33.365147+0000","last_became_active":"2024-06-29T15:57:21.276031+0000","last_became_peered":"2024-06-29T15:57:21.276031+0000","last_unstale":"2024-06-29T15:57:33.365147+0000","last_undegraded":"2024-06-29T15:57:33.365147+0000","last_fullsized":"2024-06-29T15:57:33.365147+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:23.626777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356074+0000","last_change":"2024-06-29T15:57:22.293417+0000","last_active":"2024-06-29T15:57:33.356074+0000","last_peered":"2024-06-29T15:57:33.356074+0000","last_clean":"2024-06-29T15:57:33.356074+0000","last_became_active":"2024-06-29T15:57:22.293083+0000","last_became_peered":"2024-06-29T15:57:22.293083+0000","last_unstale":"2024-06-29T15:57:33.356074+0000","last_undegraded":"2024-06-29T15:57:33.356074+0000","last_fullsized":"2024-06-29T15:57:33.356074+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":9,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359057+0000","last_change":"2024-06-29T15:57:33.356187+0000","last_active":"2024-06-29T15:57:33.359057+0000","last_peered":"2024-06-29T15:57:33.359057+0000","last_clean":"2024-06-29T15:57:33.359057+0000","last_became_active":"2024-06-29T15:57:33.355400+0000","last_became_peered":"2024-06-29T15:57:33.355400+0000","last_unstale":"2024-06-29T15:57:33.359057+0000","last_undegraded":"2024-06-29T15:57:33.359057+0000","last_fullsized":"2024-06-29T15:57:33.359057+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:28:01.393531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354238+0000","last_change":"2024-06-29T15:57:33.354238+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354238+0000","last_undegraded":"2024-06-29T15:57:33.354238+0000","last_fullsized":"2024-06-29T15:57:33.354238+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356423+0000","last_change":"2024-06-29T15:57:22.298975+0000","last_active":"2024-06-29T15:57:33.356423+0000","last_peered":"2024-06-29T15:57:33.356423+0000","last_clean":"2024-06-29T15:57:33.356423+0000","last_became_active":"2024-06-29T15:57:22.298689+0000","last_became_peered":"2024-06-29T15:57:22.298689+0000","last_unstale":"2024-06-29T15:57:33.356423+0000","last_undegraded":"2024-06-29T15:57:33.356423+0000","last_fullsized":"2024-06-29T15:57:33.356423+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359369+0000","last_change":"2024-06-29T15:57:33.359369+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359369+0000","last_undegraded":"2024-06-29T15:57:33.359369+0000","last_fullsized":"2024-06-29T15:57:33.359369+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348068+0000","last_change":"2024-06-29T15:57:22.292249+0000","last_active":"2024-06-29T15:57:33.348068+0000","last_peered":"2024-06-29T15:57:33.348068+0000","last_clean":"2024-06-29T15:57:33.348068+0000","last_became_active":"2024-06-29T15:57:22.291918+0000","last_became_peered":"2024-06-29T15:57:22.291918+0000","last_unstale":"2024-06-29T15:57:33.348068+0000","last_undegraded":"2024-06-29T15:57:33.348068+0000","last_fullsized":"2024-06-29T15:57:33.348068+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:57.129905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351129+0000","last_change":"2024-06-29T15:57:33.351129+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351129+0000","last_undegraded":"2024-06-29T15:57:33.351129+0000","last_fullsized":"2024-06-29T15:57:33.351129+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355012+0000","last_change":"2024-06-29T15:57:22.285362+0000","last_active":"2024-06-29T15:57:33.355012+0000","last_peered":"2024-06-29T15:57:33.355012+0000","last_clean":"2024-06-29T15:57:33.355012+0000","last_became_active":"2024-06-29T15:57:22.284982+0000","last_became_peered":"2024-06-29T15:57:22.284982+0000","last_unstale":"2024-06-29T15:57:33.355012+0000","last_undegraded":"2024-06-29T15:57:33.355012+0000","last_fullsized":"2024-06-29T15:57:33.355012+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357913+0000","last_change":"2024-06-29T15:57:33.357913+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357913+0000","last_undegraded":"2024-06-29T15:57:33.357913+0000","last_fullsized":"2024-06-29T15:57:33.357913+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364482+0000","last_change":"2024-06-29T15:57:22.297998+0000","last_active":"2024-06-29T15:57:33.364482+0000","last_peered":"2024-06-29T15:57:33.364482+0000","last_clean":"2024-06-29T15:57:33.364482+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:33.364482+0000","last_undegraded":"2024-06-29T15:57:33.364482+0000","last_fullsized":"2024-06-29T15:57:33.364482+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357574+0000","last_change":"2024-06-29T15:57:33.357574+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357574+0000","last_undegraded":"2024-06-29T15:57:33.357574+0000","last_fullsized":"2024-06-29T15:57:33.357574+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356051+0000","last_change":"2024-06-29T15:57:22.297917+0000","last_active":"2024-06-29T15:57:33.356051+0000","last_peered":"2024-06-29T15:57:33.356051+0000","last_clean":"2024-06-29T15:57:33.356051+0000","last_became_active":"2024-06-29T15:57:22.297638+0000","last_became_peered":"2024-06-29T15:57:22.297638+0000","last_unstale":"2024-06-29T15:57:33.356051+0000","last_undegraded":"2024-06-29T15:57:33.356051+0000","last_fullsized":"2024-06-29T15:57:33.356051+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351060+0000","last_change":"2024-06-29T15:57:33.351060+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351060+0000","last_undegraded":"2024-06-29T15:57:33.351060+0000","last_fullsized":"2024-06-29T15:57:33.351060+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.364399+0000","last_change":"2024-06-29T15:57:22.298174+0000","last_active":"2024-06-29T15:57:33.364399+0000","last_peered":"2024-06-29T15:57:33.364399+0000","last_clean":"2024-06-29T15:57:33.364399+0000","last_became_active":"2024-06-29T15:57:22.297862+0000","last_became_peered":"2024-06-29T15:57:22.297862+0000","last_unstale":"2024-06-29T15:57:33.364399+0000","last_undegraded":"2024-06-29T15:57:33.364399+0000","last_fullsized":"2024-06-29T15:57:33.364399+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354825+0000","last_change":"2024-06-29T15:57:33.354825+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354825+0000","last_undegraded":"2024-06-29T15:57:33.354825+0000","last_fullsized":"2024-06-29T15:57:33.354825+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.358628+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:33.358628+0000","last_peered":"2024-06-29T15:57:33.358628+0000","last_clean":"2024-06-29T15:57:33.358628+0000","last_became_active":"2024-06-29T15:57:22.301541+0000","last_became_peered":"2024-06-29T15:57:22.301541+0000","last_unstale":"2024-06-29T15:57:33.358628+0000","last_undegraded":"2024-06-29T15:57:33.358628+0000","last_fullsized":"2024-06-29T15:57:33.358628+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:55.249516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353618+0000","last_change":"2024-06-29T15:57:33.353618+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353618+0000","last_undegraded":"2024-06-29T15:57:33.353618+0000","last_fullsized":"2024-06-29T15:57:33.353618+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355658+0000","last_change":"2024-06-29T15:57:22.292584+0000","last_active":"2024-06-29T15:57:33.355658+0000","last_peered":"2024-06-29T15:57:33.355658+0000","last_clean":"2024-06-29T15:57:33.355658+0000","last_became_active":"2024-06-29T15:57:22.291806+0000","last_became_peered":"2024-06-29T15:57:22.291806+0000","last_unstale":"2024-06-29T15:57:33.355658+0000","last_undegraded":"2024-06-29T15:57:33.355658+0000","last_fullsized":"2024-06-29T15:57:33.355658+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352348+0000","last_change":"2024-06-29T15:57:33.352348+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352348+0000","last_undegraded":"2024-06-29T15:57:33.352348+0000","last_fullsized":"2024-06-29T15:57:33.352348+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.363619+0000","last_change":"2024-06-29T15:57:22.301601+0000","last_active":"2024-06-29T15:57:33.363619+0000","last_peered":"2024-06-29T15:57:33.363619+0000","last_clean":"2024-06-29T15:57:33.363619+0000","last_became_active":"2024-06-29T15:57:22.301244+0000","last_became_peered":"2024-06-29T15:57:22.301244+0000","last_unstale":"2024-06-29T15:57:33.363619+0000","last_undegraded":"2024-06-29T15:57:33.363619+0000","last_fullsized":"2024-06-29T15:57:33.363619+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:24:56.938464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352379+0000","last_change":"2024-06-29T15:57:33.352379+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352379+0000","last_undegraded":"2024-06-29T15:57:33.352379+0000","last_fullsized":"2024-06-29T15:57:33.352379+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353891+0000","last_change":"2024-06-29T15:57:22.301362+0000","last_active":"2024-06-29T15:57:33.353891+0000","last_peered":"2024-06-29T15:57:33.353891+0000","last_clean":"2024-06-29T15:57:33.353891+0000","last_became_active":"2024-06-29T15:57:22.300704+0000","last_became_peered":"2024-06-29T15:57:22.300704+0000","last_unstale":"2024-06-29T15:57:33.353891+0000","last_undegraded":"2024-06-29T15:57:33.353891+0000","last_fullsized":"2024-06-29T15:57:33.353891+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351810+0000","last_change":"2024-06-29T15:57:33.351810+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351810+0000","last_undegraded":"2024-06-29T15:57:33.351810+0000","last_fullsized":"2024-06-29T15:57:33.351810+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.353925+0000","last_change":"2024-06-29T15:57:22.307686+0000","last_active":"2024-06-29T15:57:33.353925+0000","last_peered":"2024-06-29T15:57:33.353925+0000","last_clean":"2024-06-29T15:57:33.353925+0000","last_became_active":"2024-06-29T15:57:22.307324+0000","last_became_peered":"2024-06-29T15:57:22.307324+0000","last_unstale":"2024-06-29T15:57:33.353925+0000","last_undegraded":"2024-06-29T15:57:33.353925+0000","last_fullsized":"2024-06-29T15:57:33.353925+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353324+0000","last_change":"2024-06-29T15:57:33.353324+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353324+0000","last_undegraded":"2024-06-29T15:57:33.353324+0000","last_fullsized":"2024-06-29T15:57:33.353324+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361177+0000","last_change":"2024-06-29T15:57:22.307230+0000","last_active":"2024-06-29T15:57:33.361177+0000","last_peered":"2024-06-29T15:57:33.361177+0000","last_clean":"2024-06-29T15:57:33.361177+0000","last_became_active":"2024-06-29T15:57:22.306920+0000","last_became_peered":"2024-06-29T15:57:22.306920+0000","last_unstale":"2024-06-29T15:57:33.361177+0000","last_undegraded":"2024-06-29T15:57:33.361177+0000","last_fullsized":"2024-06-29T15:57:33.361177+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:22.598647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352631+0000","last_change":"2024-06-29T15:57:33.352631+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352631+0000","last_undegraded":"2024-06-29T15:57:33.352631+0000","last_fullsized":"2024-06-29T15:57:33.352631+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.359396+0000","last_change":"2024-06-29T15:57:22.290186+0000","last_active":"2024-06-29T15:57:33.359396+0000","last_peered":"2024-06-29T15:57:33.359396+0000","last_clean":"2024-06-29T15:57:33.359396+0000","last_became_active":"2024-06-29T15:57:22.289908+0000","last_became_peered":"2024-06-29T15:57:22.289908+0000","last_unstale":"2024-06-29T15:57:33.359396+0000","last_undegraded":"2024-06-29T15:57:33.359396+0000","last_fullsized":"2024-06-29T15:57:33.359396+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353284+0000","last_change":"2024-06-29T15:57:33.353284+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353284+0000","last_undegraded":"2024-06-29T15:57:33.353284+0000","last_fullsized":"2024-06-29T15:57:33.353284+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348980+0000","last_change":"2024-06-29T15:57:22.306593+0000","last_active":"2024-06-29T15:57:33.348980+0000","last_peered":"2024-06-29T15:57:33.348980+0000","last_clean":"2024-06-29T15:57:33.348980+0000","last_became_active":"2024-06-29T15:57:22.296653+0000","last_became_peered":"2024-06-29T15:57:22.296653+0000","last_unstale":"2024-06-29T15:57:33.348980+0000","last_undegraded":"2024-06-29T15:57:33.348980+0000","last_fullsized":"2024-06-29T15:57:33.348980+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:41.738155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351749+0000","last_change":"2024-06-29T15:57:33.351749+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351749+0000","last_undegraded":"2024-06-29T15:57:33.351749+0000","last_fullsized":"2024-06-29T15:57:33.351749+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348828+0000","last_change":"2024-06-29T15:57:22.289633+0000","last_active":"2024-06-29T15:57:33.348828+0000","last_peered":"2024-06-29T15:57:33.348828+0000","last_clean":"2024-06-29T15:57:33.348828+0000","last_became_active":"2024-06-29T15:57:22.289349+0000","last_became_peered":"2024-06-29T15:57:22.289349+0000","last_unstale":"2024-06-29T15:57:33.348828+0000","last_undegraded":"2024-06-29T15:57:33.348828+0000","last_fullsized":"2024-06-29T15:57:33.348828+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.873806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.357678+0000","last_change":"2024-06-29T15:57:33.357678+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.357678+0000","last_undegraded":"2024-06-29T15:57:33.357678+0000","last_fullsized":"2024-06-29T15:57:33.357678+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355608+0000","last_change":"2024-06-29T15:57:22.300787+0000","last_active":"2024-06-29T15:57:33.355608+0000","last_peered":"2024-06-29T15:57:33.355608+0000","last_clean":"2024-06-29T15:57:33.355608+0000","last_became_active":"2024-06-29T15:57:22.300112+0000","last_became_peered":"2024-06-29T15:57:22.300112+0000","last_unstale":"2024-06-29T15:57:33.355608+0000","last_undegraded":"2024-06-29T15:57:33.355608+0000","last_fullsized":"2024-06-29T15:57:33.355608+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:30.249483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353010+0000","last_change":"2024-06-29T15:57:33.353010+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353010+0000","last_undegraded":"2024-06-29T15:57:33.353010+0000","last_fullsized":"2024-06-29T15:57:33.353010+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348033+0000","last_change":"2024-06-29T15:57:22.290047+0000","last_active":"2024-06-29T15:57:33.348033+0000","last_peered":"2024-06-29T15:57:33.348033+0000","last_clean":"2024-06-29T15:57:33.348033+0000","last_became_active":"2024-06-29T15:57:22.289742+0000","last_became_peered":"2024-06-29T15:57:22.289742+0000","last_unstale":"2024-06-29T15:57:33.348033+0000","last_undegraded":"2024-06-29T15:57:33.348033+0000","last_fullsized":"2024-06-29T15:57:33.348033+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:03:46.271206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362654+0000","last_change":"2024-06-29T15:57:33.362654+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362654+0000","last_undegraded":"2024-06-29T15:57:33.362654+0000","last_fullsized":"2024-06-29T15:57:33.362654+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356944+0000","last_change":"2024-06-29T15:57:22.302418+0000","last_active":"2024-06-29T15:57:33.356944+0000","last_peered":"2024-06-29T15:57:33.356944+0000","last_clean":"2024-06-29T15:57:33.356944+0000","last_became_active":"2024-06-29T15:57:22.302075+0000","last_became_peered":"2024-06-29T15:57:22.302075+0000","last_unstale":"2024-06-29T15:57:33.356944+0000","last_undegraded":"2024-06-29T15:57:33.356944+0000","last_fullsized":"2024-06-29T15:57:33.356944+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.350522+0000","last_change":"2024-06-29T15:57:33.350522+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.350522+0000","last_undegraded":"2024-06-29T15:57:33.350522+0000","last_fullsized":"2024-06-29T15:57:33.350522+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.354196+0000","last_change":"2024-06-29T15:57:22.303092+0000","last_active":"2024-06-29T15:57:33.354196+0000","last_peered":"2024-06-29T15:57:33.354196+0000","last_clean":"2024-06-29T15:57:33.354196+0000","last_became_active":"2024-06-29T15:57:22.302807+0000","last_became_peered":"2024-06-29T15:57:22.302807+0000","last_unstale":"2024-06-29T15:57:33.354196+0000","last_undegraded":"2024-06-29T15:57:33.354196+0000","last_fullsized":"2024-06-29T15:57:33.354196+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360360+0000","last_change":"2024-06-29T15:57:33.360360+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360360+0000","last_undegraded":"2024-06-29T15:57:33.360360+0000","last_fullsized":"2024-06-29T15:57:33.360360+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365064+0000","last_change":"2024-06-29T15:57:22.295681+0000","last_active":"2024-06-29T15:57:33.365064+0000","last_peered":"2024-06-29T15:57:33.365064+0000","last_clean":"2024-06-29T15:57:33.365064+0000","last_became_active":"2024-06-29T15:57:22.295357+0000","last_became_peered":"2024-06-29T15:57:22.295357+0000","last_unstale":"2024-06-29T15:57:33.365064+0000","last_undegraded":"2024-06-29T15:57:33.365064+0000","last_fullsized":"2024-06-29T15:57:33.365064+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.353724+0000","last_change":"2024-06-29T15:57:33.353724+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.353724+0000","last_undegraded":"2024-06-29T15:57:33.353724+0000","last_fullsized":"2024-06-29T15:57:33.353724+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.365558+0000","last_change":"2024-06-29T15:57:22.298612+0000","last_active":"2024-06-29T15:57:33.365558+0000","last_peered":"2024-06-29T15:57:33.365558+0000","last_clean":"2024-06-29T15:57:33.365558+0000","last_became_active":"2024-06-29T15:57:22.298288+0000","last_became_peered":"2024-06-29T15:57:22.298288+0000","last_unstale":"2024-06-29T15:57:33.365558+0000","last_undegraded":"2024-06-29T15:57:33.365558+0000","last_fullsized":"2024-06-29T15:57:33.365558+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.351291+0000","last_change":"2024-06-29T15:57:33.351291+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.351291+0000","last_undegraded":"2024-06-29T15:57:33.351291+0000","last_fullsized":"2024-06-29T15:57:33.351291+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356870+0000","last_change":"2024-06-29T15:57:22.298781+0000","last_active":"2024-06-29T15:57:33.356870+0000","last_peered":"2024-06-29T15:57:33.356870+0000","last_clean":"2024-06-29T15:57:33.356870+0000","last_became_active":"2024-06-29T15:57:22.298300+0000","last_became_peered":"2024-06-29T15:57:22.298300+0000","last_unstale":"2024-06-29T15:57:33.356870+0000","last_undegraded":"2024-06-29T15:57:33.356870+0000","last_fullsized":"2024-06-29T15:57:33.356870+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.349621+0000","last_change":"2024-06-29T15:57:33.349621+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.349621+0000","last_undegraded":"2024-06-29T15:57:33.349621+0000","last_fullsized":"2024-06-29T15:57:33.349621+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.348689+0000","last_change":"2024-06-29T15:57:22.296441+0000","last_active":"2024-06-29T15:57:33.348689+0000","last_peered":"2024-06-29T15:57:33.348689+0000","last_clean":"2024-06-29T15:57:33.348689+0000","last_became_active":"2024-06-29T15:57:22.296146+0000","last_became_peered":"2024-06-29T15:57:22.296146+0000","last_unstale":"2024-06-29T15:57:33.348689+0000","last_undegraded":"2024-06-29T15:57:33.348689+0000","last_fullsized":"2024-06-29T15:57:33.348689+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:54.360521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356517+0000","last_change":"2024-06-29T15:57:33.356517+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356517+0000","last_undegraded":"2024-06-29T15:57:33.356517+0000","last_fullsized":"2024-06-29T15:57:33.356517+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356057+0000","last_change":"2024-06-29T15:57:22.291096+0000","last_active":"2024-06-29T15:57:33.356057+0000","last_peered":"2024-06-29T15:57:33.356057+0000","last_clean":"2024-06-29T15:57:33.356057+0000","last_became_active":"2024-06-29T15:57:22.290428+0000","last_became_peered":"2024-06-29T15:57:22.290428+0000","last_unstale":"2024-06-29T15:57:33.356057+0000","last_undegraded":"2024-06-29T15:57:33.356057+0000","last_fullsized":"2024-06-29T15:57:33.356057+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:20.813793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352280+0000","last_change":"2024-06-29T15:57:33.352280+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352280+0000","last_undegraded":"2024-06-29T15:57:33.352280+0000","last_fullsized":"2024-06-29T15:57:33.352280+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355022+0000","last_change":"2024-06-29T15:57:22.297617+0000","last_active":"2024-06-29T15:57:33.355022+0000","last_peered":"2024-06-29T15:57:33.355022+0000","last_clean":"2024-06-29T15:57:33.355022+0000","last_became_active":"2024-06-29T15:57:22.297256+0000","last_became_peered":"2024-06-29T15:57:22.297256+0000","last_unstale":"2024-06-29T15:57:33.355022+0000","last_undegraded":"2024-06-29T15:57:33.355022+0000","last_fullsized":"2024-06-29T15:57:33.355022+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.359862+0000","last_change":"2024-06-29T15:57:33.359862+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.359862+0000","last_undegraded":"2024-06-29T15:57:33.359862+0000","last_fullsized":"2024-06-29T15:57:33.359862+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356834+0000","last_change":"2024-06-29T15:57:22.304052+0000","last_active":"2024-06-29T15:57:33.356834+0000","last_peered":"2024-06-29T15:57:33.356834+0000","last_clean":"2024-06-29T15:57:33.356834+0000","last_became_active":"2024-06-29T15:57:22.303598+0000","last_became_peered":"2024-06-29T15:57:22.303598+0000","last_unstale":"2024-06-29T15:57:33.356834+0000","last_undegraded":"2024-06-29T15:57:33.356834+0000","last_fullsized":"2024-06-29T15:57:33.356834+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.352198+0000","last_change":"2024-06-29T15:57:33.352198+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.352198+0000","last_undegraded":"2024-06-29T15:57:33.352198+0000","last_fullsized":"2024-06-29T15:57:33.352198+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356415+0000","last_change":"2024-06-29T15:57:22.288041+0000","last_active":"2024-06-29T15:57:33.356415+0000","last_peered":"2024-06-29T15:57:33.356415+0000","last_clean":"2024-06-29T15:57:33.356415+0000","last_became_active":"2024-06-29T15:57:22.287729+0000","last_became_peered":"2024-06-29T15:57:22.287729+0000","last_unstale":"2024-06-29T15:57:33.356415+0000","last_undegraded":"2024-06-29T15:57:33.356415+0000","last_fullsized":"2024-06-29T15:57:33.356415+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.362099+0000","last_change":"2024-06-29T15:57:33.362099+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.362099+0000","last_undegraded":"2024-06-29T15:57:33.362099+0000","last_fullsized":"2024-06-29T15:57:33.362099+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356673+0000","last_change":"2024-06-29T15:57:22.290909+0000","last_active":"2024-06-29T15:57:33.356673+0000","last_peered":"2024-06-29T15:57:33.356673+0000","last_clean":"2024-06-29T15:57:33.356673+0000","last_became_active":"2024-06-29T15:57:22.290643+0000","last_became_peered":"2024-06-29T15:57:22.290643+0000","last_unstale":"2024-06-29T15:57:33.356673+0000","last_undegraded":"2024-06-29T15:57:33.356673+0000","last_fullsized":"2024-06-29T15:57:33.356673+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354913+0000","last_change":"2024-06-29T15:57:33.354913+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354913+0000","last_undegraded":"2024-06-29T15:57:33.354913+0000","last_fullsized":"2024-06-29T15:57:33.354913+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.361010+0000","last_change":"2024-06-29T15:57:22.298374+0000","last_active":"2024-06-29T15:57:33.361010+0000","last_peered":"2024-06-29T15:57:33.361010+0000","last_clean":"2024-06-29T15:57:33.361010+0000","last_became_active":"2024-06-29T15:57:22.297962+0000","last_became_peered":"2024-06-29T15:57:22.297962+0000","last_unstale":"2024-06-29T15:57:33.361010+0000","last_undegraded":"2024-06-29T15:57:33.361010+0000","last_fullsized":"2024-06-29T15:57:33.361010+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:13.834244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.354164+0000","last_change":"2024-06-29T15:57:33.354164+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.354164+0000","last_undegraded":"2024-06-29T15:57:33.354164+0000","last_fullsized":"2024-06-29T15:57:33.354164+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.356686+0000","last_change":"2024-06-29T15:57:22.300745+0000","last_active":"2024-06-29T15:57:33.356686+0000","last_peered":"2024-06-29T15:57:33.356686+0000","last_clean":"2024-06-29T15:57:33.356686+0000","last_became_active":"2024-06-29T15:57:22.300414+0000","last_became_peered":"2024-06-29T15:57:22.300414+0000","last_unstale":"2024-06-29T15:57:33.356686+0000","last_undegraded":"2024-06-29T15:57:33.356686+0000","last_fullsized":"2024-06-29T15:57:33.356686+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.356852+0000","last_change":"2024-06-29T15:57:33.356852+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.356852+0000","last_undegraded":"2024-06-29T15:57:33.356852+0000","last_fullsized":"2024-06-29T15:57:33.356852+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355941+0000","last_change":"2024-06-29T15:57:22.297971+0000","last_active":"2024-06-29T15:57:33.355941+0000","last_peered":"2024-06-29T15:57:33.355941+0000","last_clean":"2024-06-29T15:57:33.355941+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:33.355941+0000","last_undegraded":"2024-06-29T15:57:33.355941+0000","last_fullsized":"2024-06-29T15:57:33.355941+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":1,"reported_epoch":215,"state":"creating+peering","last_fresh":"2024-06-29T15:57:33.360424+0000","last_change":"2024-06-29T15:57:33.360424+0000","last_active":"2024-06-29T15:57:33.339954+0000","last_peered":"2024-06-29T15:57:33.339954+0000","last_clean":"2024-06-29T15:57:33.339954+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:57:33.360424+0000","last_undegraded":"2024-06-29T15:57:33.360424+0000","last_fullsized":"2024-06-29T15:57:33.360424+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":16,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-06-29T15:57:33.355500+0000","last_change":"2024-06-29T15:57:22.301019+0000","last_active":"2024-06-29T15:57:33.355500+0000","last_peered":"2024-06-29T15:57:33.355500+0000","last_clean":"2024-06-29T15:57:33.355500+0000","last_became_active":"2024-06-29T15:57:22.300506+0000","last_became_peered":"2024-06-29T15:57:22.300506+0000","last_unstale":"2024-06-29T15:57:33.355500+0000","last_undegraded":"2024-06-29T15:57:33.355500+0000","last_fullsized":"2024-06-29T15:57:33.355500+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52456,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319384,"statfs":{"total":96636764160,"available":96583049216,"internally_reserved":0,"allocated":3305472,"data_stored":1914275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575988,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42140,"kb_used_data":2640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329700,"statfs":{"total":96636764160,"available":96593612800,"internally_reserved":0,"allocated":2703360,"data_stored":1321634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575986,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42128,"kb_used_data":2628,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329712,"statfs":{"total":96636764160,"available":96593625088,"internally_reserved":0,"allocated":2691072,"data_stored":1314370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60208,"kb_used_data":2656,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311632,"statfs":{"total":96636764160,"available":96575111168,"internally_reserved":0,"allocated":2719744,"data_stored":1324011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575987,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47716,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324124,"statfs":{"total":96636764160,"available":96587902976,"internally_reserved":0,"allocated":2711552,"data_stored":1321234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575987,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47704,"kb_used_data":2636,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324136,"statfs":{"total":96636764160,"available":96587915264,"internally_reserved":0,"allocated":2699264,"data_stored":1321052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575986,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60180,"kb_used_data":2632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311660,"statfs":{"total":96636764160,"available":96575139840,"internally_reserved":0,"allocated":2695168,"data_stored":1313844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575987,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60756,"kb_used_data":3208,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311084,"statfs":{"total":96636764160,"available":96574550016,"internally_reserved":0,"allocated":3284992,"data_stored":1904246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:42.902 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:57:43.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.029+0000 7fdd1c626640 1 -- 172.21.15.81:0/1435935322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd14073740 msgr2=0x7fdd1415e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:43.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.029+0000 7fdd1c626640 1 --2- 172.21.15.81:0/1435935322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd14073740 0x7fdd1415e960 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c009830 tx=0x7fdd0c02a510 comp rx=0 tx=0).stop 2024-06-29T15:57:43.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.029+0000 7fdd1c626640 1 -- 172.21.15.81:0/1435935322 shutdown_connections 2024-06-29T15:57:43.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.029+0000 7fdd1c626640 1 --2- 172.21.15.81:0/1435935322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd14073740 0x7fdd1415e960 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.029+0000 7fdd1c626640 1 --2- 172.21.15.81:0/1435935322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd141412f0 0x7fdd140cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.029+0000 7fdd1c626640 1 -- 172.21.15.81:0/1435935322 >> 172.21.15.81:0/1435935322 conn(0x7fdd1406ccc0 msgr2=0x7fdd140b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:43.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.029+0000 7fdd1c626640 1 -- 172.21.15.81:0/1435935322 shutdown_connections 2024-06-29T15:57:43.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.029+0000 7fdd1c626640 1 -- 172.21.15.81:0/1435935322 wait complete. 2024-06-29T15:57:43.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.031+0000 7fdd1c626640 1 Processor -- start 2024-06-29T15:57:43.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.031+0000 7fdd1c626640 1 -- start start 2024-06-29T15:57:43.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1c626640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd140cf4d0 0x7fdd141e46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:43.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1c626640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd141412f0 0x7fdd1413fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:43.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1c626640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd141e5dd0 0x7fdd14152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:43.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1c626640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdd140b3210 con 0x7fdd141412f0 2024-06-29T15:57:43.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1c626640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1415f270 con 0x7fdd141e5dd0 2024-06-29T15:57:43.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1c626640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdd140b3390 con 0x7fdd140cf4d0 2024-06-29T15:57:43.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd19bc1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd141412f0 0x7fdd1413fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:43.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd19bc1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd141412f0 0x7fdd1413fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54192/0 (socket says 172.21.15.81:54192) 2024-06-29T15:57:43.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd19bc1640 1 -- 172.21.15.81:0/161030269 learned_addr learned my addr 172.21.15.81:0/161030269 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:43.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1abc3640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd141e5dd0 0x7fdd14152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:43.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1a3c2640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd140cf4d0 0x7fdd141e46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:43.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd19bc1640 1 -- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd140cf4d0 msgr2=0x7fdd141e46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:43.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd19bc1640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd140cf4d0 0x7fdd141e46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd19bc1640 1 -- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd141e5dd0 msgr2=0x7fdd14152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:43.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd19bc1640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd141e5dd0 0x7fdd14152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd19bc1640 1 -- 172.21.15.81:0/161030269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd14151ef0 con 0x7fdd141412f0 2024-06-29T15:57:43.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1abc3640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd141e5dd0 0x7fdd14152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:43.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd19bc1640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd141412f0 0x7fdd1413fe00 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fdd0400c3c0 tx=0x7fdd0400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:43.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd0affd640 1 -- 172.21.15.81:0/161030269 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd04017070 con 0x7fdd141412f0 2024-06-29T15:57:43.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd0affd640 1 -- 172.21.15.81:0/161030269 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd040035f0 con 0x7fdd141412f0 2024-06-29T15:57:43.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1c626640 1 -- 172.21.15.81:0/161030269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd141e4d30 con 0x7fdd141412f0 2024-06-29T15:57:43.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd0affd640 1 -- 172.21.15.81:0/161030269 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd04003a40 con 0x7fdd141412f0 2024-06-29T15:57:43.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.033+0000 7fdd1c626640 1 -- 172.21.15.81:0/161030269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd141e52f0 con 0x7fdd141412f0 2024-06-29T15:57:43.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.035+0000 7fdd1c626640 1 -- 172.21.15.81:0/161030269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd140b38b0 con 0x7fdd141412f0 2024-06-29T15:57:43.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.035+0000 7fdd0affd640 1 -- 172.21.15.81:0/161030269 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fdd0401a070 con 0x7fdd141412f0 2024-06-29T15:57:43.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.036+0000 7fdd0affd640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fdcf8076420 0x7fdcf8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:43.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.036+0000 7fdd0affd640 1 -- 172.21.15.81:0/161030269 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7fdd04096ef0 con 0x7fdd141412f0 2024-06-29T15:57:43.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.036+0000 7fdd1a3c2640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fdcf8076420 0x7fdcf8078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:43.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.037+0000 7fdd1a3c2640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fdcf8076420 0x7fdcf8078930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005fd0 tx=0x7fdd100095e0 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.040+0000 7fdd0affd640 1 -- 172.21.15.81:0/161030269 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdd0405fd10 con 0x7fdd141412f0 2024-06-29T15:57:43.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.216+0000 7fdd1c626640 1 -- 172.21.15.81:0/161030269 --> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdd1415c1d0 con 0x7fdcf8076420 2024-06-29T15:57:43.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.225+0000 7fdd0affd640 1 -- 172.21.15.81:0/161030269 <== mgr.8666 v2:172.21.15.167:6832/2018755589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491480 (secure 0 0 0) 0x7fdd1415c1d0 con 0x7fdcf8076420 2024-06-29T15:57:43.230 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:43.234 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:57:43.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 -- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fdcf8076420 msgr2=0x7fdcf8078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fdcf8076420 0x7fdcf8078930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005fd0 tx=0x7fdd100095e0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 -- 172.21.15.81:0/161030269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd141412f0 msgr2=0x7fdd1413fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd141412f0 0x7fdd1413fe00 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fdd0400c3c0 tx=0x7fdd0400c880 comp rx=0 tx=0).stop 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 -- 172.21.15.81:0/161030269 shutdown_connections 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fdcf8076420 0x7fdcf8078930 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd141e5dd0 0x7fdd14152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd141412f0 0x7fdd1413fe00 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 --2- 172.21.15.81:0/161030269 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd140cf4d0 0x7fdd141e46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 -- 172.21.15.81:0/161030269 >> 172.21.15.81:0/161030269 conn(0x7fdd1406ccc0 msgr2=0x7fdd14159ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 -- 172.21.15.81:0/161030269 shutdown_connections 2024-06-29T15:57:43.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.229+0000 7fdd1c626640 1 -- 172.21.15.81:0/161030269 wait complete. 2024-06-29T15:57:43.240 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-06-29T15:57:41.879872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596431,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":445,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413620,"kb_used_data":22568,"kb_used_omap":12,"kb_used_meta":390899,"kb_avail":754561100,"statfs":{"total":773094113280,"available":772670566400,"internally_reserved":0,"allocated":23109632,"data_stored":11881822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":400281138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.007670"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375043+0000","last_change":"2024-06-29T15:57:34.384271+0000","last_active":"2024-06-29T15:57:35.375043+0000","last_peered":"2024-06-29T15:57:35.375043+0000","last_clean":"2024-06-29T15:57:35.375043+0000","last_became_active":"2024-06-29T15:57:34.383945+0000","last_became_peered":"2024-06-29T15:57:34.383945+0000","last_unstale":"2024-06-29T15:57:35.375043+0000","last_undegraded":"2024-06-29T15:57:35.375043+0000","last_fullsized":"2024-06-29T15:57:35.375043+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:30:46.441332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"202'219","reported_seq":1047,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374491+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:57:35.374491+0000","last_peered":"2024-06-29T15:57:35.374491+0000","last_clean":"2024-06-29T15:57:35.374491+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:57:35.374491+0000","last_undegraded":"2024-06-29T15:57:35.374491+0000","last_fullsized":"2024-06-29T15:57:35.374491+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"212'1","reported_seq":23,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788827+0000","last_change":"2024-06-29T15:57:21.282026+0000","last_active":"2024-06-29T15:57:35.788827+0000","last_peered":"2024-06-29T15:57:35.788827+0000","last_clean":"2024-06-29T15:57:35.788827+0000","last_became_active":"2024-06-29T15:57:21.281709+0000","last_became_peered":"2024-06-29T15:57:21.281709+0000","last_unstale":"2024-06-29T15:57:35.788827+0000","last_undegraded":"2024-06-29T15:57:35.788827+0000","last_fullsized":"2024-06-29T15:57:35.788827+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:54:07.641300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288684+0000","last_change":"2024-06-29T15:57:22.302267+0000","last_active":"2024-06-29T15:57:36.288684+0000","last_peered":"2024-06-29T15:57:36.288684+0000","last_clean":"2024-06-29T15:57:36.288684+0000","last_became_active":"2024-06-29T15:57:22.301972+0000","last_became_peered":"2024-06-29T15:57:22.301972+0000","last_unstale":"2024-06-29T15:57:36.288684+0000","last_undegraded":"2024-06-29T15:57:36.288684+0000","last_fullsized":"2024-06-29T15:57:36.288684+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"216'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353460+0000","last_change":"2024-06-29T15:57:33.370756+0000","last_active":"2024-06-29T15:57:35.353460+0000","last_peered":"2024-06-29T15:57:35.353460+0000","last_clean":"2024-06-29T15:57:35.353460+0000","last_became_active":"2024-06-29T15:57:33.370363+0000","last_became_peered":"2024-06-29T15:57:33.370363+0000","last_unstale":"2024-06-29T15:57:35.353460+0000","last_undegraded":"2024-06-29T15:57:35.353460+0000","last_fullsized":"2024-06-29T15:57:35.353460+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:06.322225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788791+0000","last_change":"2024-06-29T15:57:34.371768+0000","last_active":"2024-06-29T15:57:35.788791+0000","last_peered":"2024-06-29T15:57:35.788791+0000","last_clean":"2024-06-29T15:57:35.788791+0000","last_became_active":"2024-06-29T15:57:34.371464+0000","last_became_peered":"2024-06-29T15:57:34.371464+0000","last_unstale":"2024-06-29T15:57:35.788791+0000","last_undegraded":"2024-06-29T15:57:35.788791+0000","last_fullsized":"2024-06-29T15:57:35.788791+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:01.999391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788165+0000","last_change":"2024-06-29T15:57:21.283495+0000","last_active":"2024-06-29T15:57:35.788165+0000","last_peered":"2024-06-29T15:57:35.788165+0000","last_clean":"2024-06-29T15:57:35.788165+0000","last_became_active":"2024-06-29T15:57:21.283125+0000","last_became_peered":"2024-06-29T15:57:21.283125+0000","last_unstale":"2024-06-29T15:57:35.788165+0000","last_undegraded":"2024-06-29T15:57:35.788165+0000","last_fullsized":"2024-06-29T15:57:35.788165+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353556+0000","last_change":"2024-06-29T15:57:22.292872+0000","last_active":"2024-06-29T15:57:35.353556+0000","last_peered":"2024-06-29T15:57:35.353556+0000","last_clean":"2024-06-29T15:57:35.353556+0000","last_became_active":"2024-06-29T15:57:22.292485+0000","last_became_peered":"2024-06-29T15:57:22.292485+0000","last_unstale":"2024-06-29T15:57:35.353556+0000","last_undegraded":"2024-06-29T15:57:35.353556+0000","last_fullsized":"2024-06-29T15:57:35.353556+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:19.369559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"217'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375231+0000","last_change":"2024-06-29T15:57:33.369465+0000","last_active":"2024-06-29T15:57:35.375231+0000","last_peered":"2024-06-29T15:57:35.375231+0000","last_clean":"2024-06-29T15:57:35.375231+0000","last_became_active":"2024-06-29T15:57:33.369174+0000","last_became_peered":"2024-06-29T15:57:33.369174+0000","last_unstale":"2024-06-29T15:57:35.375231+0000","last_undegraded":"2024-06-29T15:57:35.375231+0000","last_fullsized":"2024-06-29T15:57:35.375231+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:36:35.938041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601854+0000","last_change":"2024-06-29T15:57:34.373815+0000","last_active":"2024-06-29T15:57:35.601854+0000","last_peered":"2024-06-29T15:57:35.601854+0000","last_clean":"2024-06-29T15:57:35.601854+0000","last_became_active":"2024-06-29T15:57:34.373518+0000","last_became_peered":"2024-06-29T15:57:34.373518+0000","last_unstale":"2024-06-29T15:57:35.601854+0000","last_undegraded":"2024-06-29T15:57:35.601854+0000","last_fullsized":"2024-06-29T15:57:35.601854+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:49:01.999391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355261+0000","last_change":"2024-06-29T15:57:21.274359+0000","last_active":"2024-06-29T15:57:35.355261+0000","last_peered":"2024-06-29T15:57:35.355261+0000","last_clean":"2024-06-29T15:57:35.355261+0000","last_became_active":"2024-06-29T15:57:21.274036+0000","last_became_peered":"2024-06-29T15:57:21.274036+0000","last_unstale":"2024-06-29T15:57:35.355261+0000","last_undegraded":"2024-06-29T15:57:35.355261+0000","last_fullsized":"2024-06-29T15:57:35.355261+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:02.747066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351427+0000","last_change":"2024-06-29T15:57:22.295101+0000","last_active":"2024-06-29T15:57:35.351427+0000","last_peered":"2024-06-29T15:57:35.351427+0000","last_clean":"2024-06-29T15:57:35.351427+0000","last_became_active":"2024-06-29T15:57:22.294805+0000","last_became_peered":"2024-06-29T15:57:22.294805+0000","last_unstale":"2024-06-29T15:57:35.351427+0000","last_undegraded":"2024-06-29T15:57:35.351427+0000","last_fullsized":"2024-06-29T15:57:35.351427+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"216'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601848+0000","last_change":"2024-06-29T15:57:33.369487+0000","last_active":"2024-06-29T15:57:35.601848+0000","last_peered":"2024-06-29T15:57:35.601848+0000","last_clean":"2024-06-29T15:57:35.601848+0000","last_became_active":"2024-06-29T15:57:33.369138+0000","last_became_peered":"2024-06-29T15:57:33.369138+0000","last_unstale":"2024-06-29T15:57:35.601848+0000","last_undegraded":"2024-06-29T15:57:35.601848+0000","last_fullsized":"2024-06-29T15:57:35.601848+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:13.682132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601910+0000","last_change":"2024-06-29T15:57:34.362062+0000","last_active":"2024-06-29T15:57:35.601910+0000","last_peered":"2024-06-29T15:57:35.601910+0000","last_clean":"2024-06-29T15:57:35.601910+0000","last_became_active":"2024-06-29T15:57:34.361770+0000","last_became_peered":"2024-06-29T15:57:34.361770+0000","last_unstale":"2024-06-29T15:57:35.601910+0000","last_undegraded":"2024-06-29T15:57:35.601910+0000","last_fullsized":"2024-06-29T15:57:35.601910+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:25:07.454033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601619+0000","last_change":"2024-06-29T15:57:21.274551+0000","last_active":"2024-06-29T15:57:35.601619+0000","last_peered":"2024-06-29T15:57:35.601619+0000","last_clean":"2024-06-29T15:57:35.601619+0000","last_became_active":"2024-06-29T15:57:21.274029+0000","last_became_peered":"2024-06-29T15:57:21.274029+0000","last_unstale":"2024-06-29T15:57:35.601619+0000","last_undegraded":"2024-06-29T15:57:35.601619+0000","last_fullsized":"2024-06-29T15:57:35.601619+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788492+0000","last_change":"2024-06-29T15:57:22.291143+0000","last_active":"2024-06-29T15:57:35.788492+0000","last_peered":"2024-06-29T15:57:35.788492+0000","last_clean":"2024-06-29T15:57:35.788492+0000","last_became_active":"2024-06-29T15:57:22.290849+0000","last_became_peered":"2024-06-29T15:57:22.290849+0000","last_unstale":"2024-06-29T15:57:35.788492+0000","last_undegraded":"2024-06-29T15:57:35.788492+0000","last_fullsized":"2024-06-29T15:57:35.788492+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788564+0000","last_change":"2024-06-29T15:57:33.366161+0000","last_active":"2024-06-29T15:57:35.788564+0000","last_peered":"2024-06-29T15:57:35.788564+0000","last_clean":"2024-06-29T15:57:35.788564+0000","last_became_active":"2024-06-29T15:57:33.365820+0000","last_became_peered":"2024-06-29T15:57:33.365820+0000","last_unstale":"2024-06-29T15:57:35.788564+0000","last_undegraded":"2024-06-29T15:57:35.788564+0000","last_fullsized":"2024-06-29T15:57:35.788564+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:41.322192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788527+0000","last_change":"2024-06-29T15:57:34.387176+0000","last_active":"2024-06-29T15:57:35.788527+0000","last_peered":"2024-06-29T15:57:35.788527+0000","last_clean":"2024-06-29T15:57:35.788527+0000","last_became_active":"2024-06-29T15:57:34.386826+0000","last_became_peered":"2024-06-29T15:57:34.386826+0000","last_unstale":"2024-06-29T15:57:35.788527+0000","last_undegraded":"2024-06-29T15:57:35.788527+0000","last_fullsized":"2024-06-29T15:57:35.788527+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:07:36.359415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374968+0000","last_change":"2024-06-29T15:57:21.281405+0000","last_active":"2024-06-29T15:57:35.374968+0000","last_peered":"2024-06-29T15:57:35.374968+0000","last_clean":"2024-06-29T15:57:35.374968+0000","last_became_active":"2024-06-29T15:57:21.281080+0000","last_became_peered":"2024-06-29T15:57:21.281080+0000","last_unstale":"2024-06-29T15:57:35.374968+0000","last_undegraded":"2024-06-29T15:57:35.374968+0000","last_fullsized":"2024-06-29T15:57:35.374968+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:51.116695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288791+0000","last_change":"2024-06-29T15:57:22.304057+0000","last_active":"2024-06-29T15:57:36.288791+0000","last_peered":"2024-06-29T15:57:36.288791+0000","last_clean":"2024-06-29T15:57:36.288791+0000","last_became_active":"2024-06-29T15:57:22.303762+0000","last_became_peered":"2024-06-29T15:57:22.303762+0000","last_unstale":"2024-06-29T15:57:36.288791+0000","last_undegraded":"2024-06-29T15:57:36.288791+0000","last_fullsized":"2024-06-29T15:57:36.288791+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:40.125779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375312+0000","last_change":"2024-06-29T15:57:33.369260+0000","last_active":"2024-06-29T15:57:35.375312+0000","last_peered":"2024-06-29T15:57:35.375312+0000","last_clean":"2024-06-29T15:57:35.375312+0000","last_became_active":"2024-06-29T15:57:33.368945+0000","last_became_peered":"2024-06-29T15:57:33.368945+0000","last_unstale":"2024-06-29T15:57:35.375312+0000","last_undegraded":"2024-06-29T15:57:35.375312+0000","last_fullsized":"2024-06-29T15:57:35.375312+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:59:18.187323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788384+0000","last_change":"2024-06-29T15:57:34.373565+0000","last_active":"2024-06-29T15:57:35.788384+0000","last_peered":"2024-06-29T15:57:35.788384+0000","last_clean":"2024-06-29T15:57:35.788384+0000","last_became_active":"2024-06-29T15:57:34.372342+0000","last_became_peered":"2024-06-29T15:57:34.372342+0000","last_unstale":"2024-06-29T15:57:35.788384+0000","last_undegraded":"2024-06-29T15:57:35.788384+0000","last_fullsized":"2024-06-29T15:57:35.788384+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:49:19.104188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351694+0000","last_change":"2024-06-29T15:57:21.277560+0000","last_active":"2024-06-29T15:57:35.351694+0000","last_peered":"2024-06-29T15:57:35.351694+0000","last_clean":"2024-06-29T15:57:35.351694+0000","last_became_active":"2024-06-29T15:57:21.277255+0000","last_became_peered":"2024-06-29T15:57:21.277255+0000","last_unstale":"2024-06-29T15:57:35.351694+0000","last_undegraded":"2024-06-29T15:57:35.351694+0000","last_fullsized":"2024-06-29T15:57:35.351694+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355016+0000","last_change":"2024-06-29T15:57:22.301166+0000","last_active":"2024-06-29T15:57:35.355016+0000","last_peered":"2024-06-29T15:57:35.355016+0000","last_clean":"2024-06-29T15:57:35.355016+0000","last_became_active":"2024-06-29T15:57:22.300676+0000","last_became_peered":"2024-06-29T15:57:22.300676+0000","last_unstale":"2024-06-29T15:57:35.355016+0000","last_undegraded":"2024-06-29T15:57:35.355016+0000","last_fullsized":"2024-06-29T15:57:35.355016+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288268+0000","last_change":"2024-06-29T15:57:33.362184+0000","last_active":"2024-06-29T15:57:36.288268+0000","last_peered":"2024-06-29T15:57:36.288268+0000","last_clean":"2024-06-29T15:57:36.288268+0000","last_became_active":"2024-06-29T15:57:33.361905+0000","last_became_peered":"2024-06-29T15:57:33.361905+0000","last_unstale":"2024-06-29T15:57:36.288268+0000","last_undegraded":"2024-06-29T15:57:36.288268+0000","last_fullsized":"2024-06-29T15:57:36.288268+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:33.113179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354904+0000","last_change":"2024-06-29T15:57:34.373708+0000","last_active":"2024-06-29T15:57:35.354904+0000","last_peered":"2024-06-29T15:57:35.354904+0000","last_clean":"2024-06-29T15:57:35.354904+0000","last_became_active":"2024-06-29T15:57:34.373373+0000","last_became_peered":"2024-06-29T15:57:34.373373+0000","last_unstale":"2024-06-29T15:57:35.354904+0000","last_undegraded":"2024-06-29T15:57:35.354904+0000","last_fullsized":"2024-06-29T15:57:35.354904+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:27.964272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.602056+0000","last_change":"2024-06-29T15:57:21.280224+0000","last_active":"2024-06-29T15:57:35.602056+0000","last_peered":"2024-06-29T15:57:35.602056+0000","last_clean":"2024-06-29T15:57:35.602056+0000","last_became_active":"2024-06-29T15:57:21.279899+0000","last_became_peered":"2024-06-29T15:57:21.279899+0000","last_unstale":"2024-06-29T15:57:35.602056+0000","last_undegraded":"2024-06-29T15:57:35.602056+0000","last_fullsized":"2024-06-29T15:57:35.602056+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353640+0000","last_change":"2024-06-29T15:57:22.296472+0000","last_active":"2024-06-29T15:57:35.353640+0000","last_peered":"2024-06-29T15:57:35.353640+0000","last_clean":"2024-06-29T15:57:35.353640+0000","last_became_active":"2024-06-29T15:57:22.296130+0000","last_became_peered":"2024-06-29T15:57:22.296130+0000","last_unstale":"2024-06-29T15:57:35.353640+0000","last_undegraded":"2024-06-29T15:57:35.353640+0000","last_fullsized":"2024-06-29T15:57:35.353640+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:56.213300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353378+0000","last_change":"2024-06-29T15:57:33.367380+0000","last_active":"2024-06-29T15:57:35.353378+0000","last_peered":"2024-06-29T15:57:35.353378+0000","last_clean":"2024-06-29T15:57:35.353378+0000","last_became_active":"2024-06-29T15:57:33.367054+0000","last_became_peered":"2024-06-29T15:57:33.367054+0000","last_unstale":"2024-06-29T15:57:35.353378+0000","last_undegraded":"2024-06-29T15:57:35.353378+0000","last_fullsized":"2024-06-29T15:57:35.353378+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:41.322192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374600+0000","last_change":"2024-06-29T15:57:34.376817+0000","last_active":"2024-06-29T15:57:35.374600+0000","last_peered":"2024-06-29T15:57:35.374600+0000","last_clean":"2024-06-29T15:57:35.374600+0000","last_became_active":"2024-06-29T15:57:34.376389+0000","last_became_peered":"2024-06-29T15:57:34.376389+0000","last_unstale":"2024-06-29T15:57:35.374600+0000","last_undegraded":"2024-06-29T15:57:35.374600+0000","last_fullsized":"2024-06-29T15:57:35.374600+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:13:48.075518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"212'1","reported_seq":23,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375357+0000","last_change":"2024-06-29T15:57:21.276972+0000","last_active":"2024-06-29T15:57:35.375357+0000","last_peered":"2024-06-29T15:57:35.375357+0000","last_clean":"2024-06-29T15:57:35.375357+0000","last_became_active":"2024-06-29T15:57:21.276647+0000","last_became_peered":"2024-06-29T15:57:21.276647+0000","last_unstale":"2024-06-29T15:57:35.375357+0000","last_undegraded":"2024-06-29T15:57:35.375357+0000","last_fullsized":"2024-06-29T15:57:35.375357+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:02:24.871281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289044+0000","last_change":"2024-06-29T15:57:22.303802+0000","last_active":"2024-06-29T15:57:36.289044+0000","last_peered":"2024-06-29T15:57:36.289044+0000","last_clean":"2024-06-29T15:57:36.289044+0000","last_became_active":"2024-06-29T15:57:22.303511+0000","last_became_peered":"2024-06-29T15:57:22.303511+0000","last_unstale":"2024-06-29T15:57:36.289044+0000","last_undegraded":"2024-06-29T15:57:36.289044+0000","last_fullsized":"2024-06-29T15:57:36.289044+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:10.536799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.787837+0000","last_change":"2024-06-29T15:57:33.367162+0000","last_active":"2024-06-29T15:57:35.787837+0000","last_peered":"2024-06-29T15:57:35.787837+0000","last_clean":"2024-06-29T15:57:35.787837+0000","last_became_active":"2024-06-29T15:57:33.366816+0000","last_became_peered":"2024-06-29T15:57:33.366816+0000","last_unstale":"2024-06-29T15:57:35.787837+0000","last_undegraded":"2024-06-29T15:57:35.787837+0000","last_fullsized":"2024-06-29T15:57:35.787837+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:46.165772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.787705+0000","last_change":"2024-06-29T15:57:34.375917+0000","last_active":"2024-06-29T15:57:35.787705+0000","last_peered":"2024-06-29T15:57:35.787705+0000","last_clean":"2024-06-29T15:57:35.787705+0000","last_became_active":"2024-06-29T15:57:34.372317+0000","last_became_peered":"2024-06-29T15:57:34.372317+0000","last_unstale":"2024-06-29T15:57:35.787705+0000","last_undegraded":"2024-06-29T15:57:35.787705+0000","last_fullsized":"2024-06-29T15:57:35.787705+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:11:07.094539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"212'2","reported_seq":24,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.352082+0000","last_change":"2024-06-29T15:57:21.277662+0000","last_active":"2024-06-29T15:57:35.352082+0000","last_peered":"2024-06-29T15:57:35.352082+0000","last_clean":"2024-06-29T15:57:35.352082+0000","last_became_active":"2024-06-29T15:57:21.277335+0000","last_became_peered":"2024-06-29T15:57:21.277335+0000","last_unstale":"2024-06-29T15:57:35.352082+0000","last_undegraded":"2024-06-29T15:57:35.352082+0000","last_fullsized":"2024-06-29T15:57:35.352082+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788920+0000","last_change":"2024-06-29T15:57:22.298914+0000","last_active":"2024-06-29T15:57:35.788920+0000","last_peered":"2024-06-29T15:57:35.788920+0000","last_clean":"2024-06-29T15:57:35.788920+0000","last_became_active":"2024-06-29T15:57:22.298614+0000","last_became_peered":"2024-06-29T15:57:22.298614+0000","last_unstale":"2024-06-29T15:57:35.788920+0000","last_undegraded":"2024-06-29T15:57:35.788920+0000","last_fullsized":"2024-06-29T15:57:35.788920+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601376+0000","last_change":"2024-06-29T15:57:33.367133+0000","last_active":"2024-06-29T15:57:35.601376+0000","last_peered":"2024-06-29T15:57:35.601376+0000","last_clean":"2024-06-29T15:57:35.601376+0000","last_became_active":"2024-06-29T15:57:33.366782+0000","last_became_peered":"2024-06-29T15:57:33.366782+0000","last_unstale":"2024-06-29T15:57:35.601376+0000","last_undegraded":"2024-06-29T15:57:35.601376+0000","last_fullsized":"2024-06-29T15:57:35.601376+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:46:09.697181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289627+0000","last_change":"2024-06-29T15:57:34.373307+0000","last_active":"2024-06-29T15:57:36.289627+0000","last_peered":"2024-06-29T15:57:36.289627+0000","last_clean":"2024-06-29T15:57:36.289627+0000","last_became_active":"2024-06-29T15:57:34.373007+0000","last_became_peered":"2024-06-29T15:57:34.373007+0000","last_unstale":"2024-06-29T15:57:36.289627+0000","last_undegraded":"2024-06-29T15:57:36.289627+0000","last_fullsized":"2024-06-29T15:57:36.289627+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:28:25.982813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601365+0000","last_change":"2024-06-29T15:57:21.278011+0000","last_active":"2024-06-29T15:57:35.601365+0000","last_peered":"2024-06-29T15:57:35.601365+0000","last_clean":"2024-06-29T15:57:35.601365+0000","last_became_active":"2024-06-29T15:57:21.276648+0000","last_became_peered":"2024-06-29T15:57:21.276648+0000","last_unstale":"2024-06-29T15:57:35.601365+0000","last_undegraded":"2024-06-29T15:57:35.601365+0000","last_fullsized":"2024-06-29T15:57:35.601365+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355127+0000","last_change":"2024-06-29T15:57:22.300220+0000","last_active":"2024-06-29T15:57:35.355127+0000","last_peered":"2024-06-29T15:57:35.355127+0000","last_clean":"2024-06-29T15:57:35.355127+0000","last_became_active":"2024-06-29T15:57:22.299880+0000","last_became_peered":"2024-06-29T15:57:22.299880+0000","last_unstale":"2024-06-29T15:57:35.355127+0000","last_undegraded":"2024-06-29T15:57:35.355127+0000","last_fullsized":"2024-06-29T15:57:35.355127+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"216'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288942+0000","last_change":"2024-06-29T15:57:33.367636+0000","last_active":"2024-06-29T15:57:36.288942+0000","last_peered":"2024-06-29T15:57:36.288942+0000","last_clean":"2024-06-29T15:57:36.288942+0000","last_became_active":"2024-06-29T15:57:33.367321+0000","last_became_peered":"2024-06-29T15:57:33.367321+0000","last_unstale":"2024-06-29T15:57:36.288942+0000","last_undegraded":"2024-06-29T15:57:36.288942+0000","last_fullsized":"2024-06-29T15:57:36.288942+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:41:19.709685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.352041+0000","last_change":"2024-06-29T15:57:34.365554+0000","last_active":"2024-06-29T15:57:35.352041+0000","last_peered":"2024-06-29T15:57:35.352041+0000","last_clean":"2024-06-29T15:57:35.352041+0000","last_became_active":"2024-06-29T15:57:34.364580+0000","last_became_peered":"2024-06-29T15:57:34.364580+0000","last_unstale":"2024-06-29T15:57:35.352041+0000","last_undegraded":"2024-06-29T15:57:35.352041+0000","last_fullsized":"2024-06-29T15:57:35.352041+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:10.459536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288553+0000","last_change":"2024-06-29T15:57:21.284982+0000","last_active":"2024-06-29T15:57:36.288553+0000","last_peered":"2024-06-29T15:57:36.288553+0000","last_clean":"2024-06-29T15:57:36.288553+0000","last_became_active":"2024-06-29T15:57:21.284684+0000","last_became_peered":"2024-06-29T15:57:21.284684+0000","last_unstale":"2024-06-29T15:57:36.288553+0000","last_undegraded":"2024-06-29T15:57:36.288553+0000","last_fullsized":"2024-06-29T15:57:36.288553+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:32:53.803859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.352813+0000","last_change":"2024-06-29T15:57:22.298744+0000","last_active":"2024-06-29T15:57:35.352813+0000","last_peered":"2024-06-29T15:57:35.352813+0000","last_clean":"2024-06-29T15:57:35.352813+0000","last_became_active":"2024-06-29T15:57:22.296925+0000","last_became_peered":"2024-06-29T15:57:22.296925+0000","last_unstale":"2024-06-29T15:57:35.352813+0000","last_undegraded":"2024-06-29T15:57:35.352813+0000","last_fullsized":"2024-06-29T15:57:35.352813+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:40.578182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"216'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.789027+0000","last_change":"2024-06-29T15:57:33.368936+0000","last_active":"2024-06-29T15:57:35.789027+0000","last_peered":"2024-06-29T15:57:35.789027+0000","last_clean":"2024-06-29T15:57:35.789027+0000","last_became_active":"2024-06-29T15:57:33.368600+0000","last_became_peered":"2024-06-29T15:57:33.368600+0000","last_unstale":"2024-06-29T15:57:35.789027+0000","last_undegraded":"2024-06-29T15:57:35.789027+0000","last_fullsized":"2024-06-29T15:57:35.789027+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788587+0000","last_change":"2024-06-29T15:57:34.380125+0000","last_active":"2024-06-29T15:57:35.788587+0000","last_peered":"2024-06-29T15:57:35.788587+0000","last_clean":"2024-06-29T15:57:35.788587+0000","last_became_active":"2024-06-29T15:57:34.379744+0000","last_became_peered":"2024-06-29T15:57:34.379744+0000","last_unstale":"2024-06-29T15:57:35.788587+0000","last_undegraded":"2024-06-29T15:57:35.788587+0000","last_fullsized":"2024-06-29T15:57:35.788587+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:12.203421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288906+0000","last_change":"2024-06-29T15:57:21.284780+0000","last_active":"2024-06-29T15:57:36.288906+0000","last_peered":"2024-06-29T15:57:36.288906+0000","last_clean":"2024-06-29T15:57:36.288906+0000","last_became_active":"2024-06-29T15:57:21.284273+0000","last_became_peered":"2024-06-29T15:57:21.284273+0000","last_unstale":"2024-06-29T15:57:36.288906+0000","last_undegraded":"2024-06-29T15:57:36.288906+0000","last_fullsized":"2024-06-29T15:57:36.288906+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:21:16.085990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351652+0000","last_change":"2024-06-29T15:57:22.291304+0000","last_active":"2024-06-29T15:57:35.351652+0000","last_peered":"2024-06-29T15:57:35.351652+0000","last_clean":"2024-06-29T15:57:35.351652+0000","last_became_active":"2024-06-29T15:57:22.290983+0000","last_became_peered":"2024-06-29T15:57:22.290983+0000","last_unstale":"2024-06-29T15:57:35.351652+0000","last_undegraded":"2024-06-29T15:57:35.351652+0000","last_fullsized":"2024-06-29T15:57:35.351652+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289556+0000","last_change":"2024-06-29T15:57:33.366553+0000","last_active":"2024-06-29T15:57:36.289556+0000","last_peered":"2024-06-29T15:57:36.289556+0000","last_clean":"2024-06-29T15:57:36.289556+0000","last_became_active":"2024-06-29T15:57:33.366230+0000","last_became_peered":"2024-06-29T15:57:33.366230+0000","last_unstale":"2024-06-29T15:57:36.289556+0000","last_undegraded":"2024-06-29T15:57:36.289556+0000","last_fullsized":"2024-06-29T15:57:36.289556+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:54:23.380584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788437+0000","last_change":"2024-06-29T15:57:34.377948+0000","last_active":"2024-06-29T15:57:35.788437+0000","last_peered":"2024-06-29T15:57:35.788437+0000","last_clean":"2024-06-29T15:57:35.788437+0000","last_became_active":"2024-06-29T15:57:34.377336+0000","last_became_peered":"2024-06-29T15:57:34.377336+0000","last_unstale":"2024-06-29T15:57:35.788437+0000","last_undegraded":"2024-06-29T15:57:35.788437+0000","last_fullsized":"2024-06-29T15:57:35.788437+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:09:21.704940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"212'2","reported_seq":24,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289310+0000","last_change":"2024-06-29T15:57:21.284009+0000","last_active":"2024-06-29T15:57:36.289310+0000","last_peered":"2024-06-29T15:57:36.289310+0000","last_clean":"2024-06-29T15:57:36.289310+0000","last_became_active":"2024-06-29T15:57:21.283679+0000","last_became_peered":"2024-06-29T15:57:21.283679+0000","last_unstale":"2024-06-29T15:57:36.289310+0000","last_undegraded":"2024-06-29T15:57:36.289310+0000","last_fullsized":"2024-06-29T15:57:36.289310+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:20.284468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288681+0000","last_change":"2024-06-29T15:57:22.288958+0000","last_active":"2024-06-29T15:57:36.288681+0000","last_peered":"2024-06-29T15:57:36.288681+0000","last_clean":"2024-06-29T15:57:36.288681+0000","last_became_active":"2024-06-29T15:57:22.288621+0000","last_became_peered":"2024-06-29T15:57:22.288621+0000","last_unstale":"2024-06-29T15:57:36.288681+0000","last_undegraded":"2024-06-29T15:57:36.288681+0000","last_fullsized":"2024-06-29T15:57:36.288681+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:14:31.720751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"216'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375159+0000","last_change":"2024-06-29T15:57:33.370560+0000","last_active":"2024-06-29T15:57:35.375159+0000","last_peered":"2024-06-29T15:57:35.375159+0000","last_clean":"2024-06-29T15:57:35.375159+0000","last_became_active":"2024-06-29T15:57:33.370206+0000","last_became_peered":"2024-06-29T15:57:33.370206+0000","last_unstale":"2024-06-29T15:57:35.375159+0000","last_undegraded":"2024-06-29T15:57:35.375159+0000","last_fullsized":"2024-06-29T15:57:35.375159+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:35.436319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788146+0000","last_change":"2024-06-29T15:57:34.379883+0000","last_active":"2024-06-29T15:57:35.788146+0000","last_peered":"2024-06-29T15:57:35.788146+0000","last_clean":"2024-06-29T15:57:35.788146+0000","last_became_active":"2024-06-29T15:57:34.379502+0000","last_became_peered":"2024-06-29T15:57:34.379502+0000","last_unstale":"2024-06-29T15:57:35.788146+0000","last_undegraded":"2024-06-29T15:57:35.788146+0000","last_fullsized":"2024-06-29T15:57:35.788146+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:11:48.493958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351652+0000","last_change":"2024-06-29T15:57:21.282111+0000","last_active":"2024-06-29T15:57:35.351652+0000","last_peered":"2024-06-29T15:57:35.351652+0000","last_clean":"2024-06-29T15:57:35.351652+0000","last_became_active":"2024-06-29T15:57:21.281761+0000","last_became_peered":"2024-06-29T15:57:21.281761+0000","last_unstale":"2024-06-29T15:57:35.351652+0000","last_undegraded":"2024-06-29T15:57:35.351652+0000","last_fullsized":"2024-06-29T15:57:35.351652+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289456+0000","last_change":"2024-06-29T15:57:22.305806+0000","last_active":"2024-06-29T15:57:36.289456+0000","last_peered":"2024-06-29T15:57:36.289456+0000","last_clean":"2024-06-29T15:57:36.289456+0000","last_became_active":"2024-06-29T15:57:22.296276+0000","last_became_peered":"2024-06-29T15:57:22.296276+0000","last_unstale":"2024-06-29T15:57:36.289456+0000","last_undegraded":"2024-06-29T15:57:36.289456+0000","last_fullsized":"2024-06-29T15:57:36.289456+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:43:51.754223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353944+0000","last_change":"2024-06-29T15:57:33.367821+0000","last_active":"2024-06-29T15:57:35.353944+0000","last_peered":"2024-06-29T15:57:35.353944+0000","last_clean":"2024-06-29T15:57:35.353944+0000","last_became_active":"2024-06-29T15:57:33.367494+0000","last_became_peered":"2024-06-29T15:57:33.367494+0000","last_unstale":"2024-06-29T15:57:35.353944+0000","last_undegraded":"2024-06-29T15:57:35.353944+0000","last_fullsized":"2024-06-29T15:57:35.353944+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601254+0000","last_change":"2024-06-29T15:57:34.362924+0000","last_active":"2024-06-29T15:57:35.601254+0000","last_peered":"2024-06-29T15:57:35.601254+0000","last_clean":"2024-06-29T15:57:35.601254+0000","last_became_active":"2024-06-29T15:57:34.362570+0000","last_became_peered":"2024-06-29T15:57:34.362570+0000","last_unstale":"2024-06-29T15:57:35.601254+0000","last_undegraded":"2024-06-29T15:57:35.601254+0000","last_fullsized":"2024-06-29T15:57:35.601254+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:09:21.704940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"212'1","reported_seq":23,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289633+0000","last_change":"2024-06-29T15:57:21.279139+0000","last_active":"2024-06-29T15:57:36.289633+0000","last_peered":"2024-06-29T15:57:36.289633+0000","last_clean":"2024-06-29T15:57:36.289633+0000","last_became_active":"2024-06-29T15:57:21.278488+0000","last_became_peered":"2024-06-29T15:57:21.278488+0000","last_unstale":"2024-06-29T15:57:36.289633+0000","last_undegraded":"2024-06-29T15:57:36.289633+0000","last_fullsized":"2024-06-29T15:57:36.289633+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:42:54.126338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351588+0000","last_change":"2024-06-29T15:57:22.297275+0000","last_active":"2024-06-29T15:57:35.351588+0000","last_peered":"2024-06-29T15:57:35.351588+0000","last_clean":"2024-06-29T15:57:35.351588+0000","last_became_active":"2024-06-29T15:57:22.296984+0000","last_became_peered":"2024-06-29T15:57:22.296984+0000","last_unstale":"2024-06-29T15:57:35.351588+0000","last_undegraded":"2024-06-29T15:57:35.351588+0000","last_fullsized":"2024-06-29T15:57:35.351588+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"216'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788205+0000","last_change":"2024-06-29T15:57:33.369358+0000","last_active":"2024-06-29T15:57:35.788205+0000","last_peered":"2024-06-29T15:57:35.788205+0000","last_clean":"2024-06-29T15:57:35.788205+0000","last_became_active":"2024-06-29T15:57:33.369052+0000","last_became_peered":"2024-06-29T15:57:33.369052+0000","last_unstale":"2024-06-29T15:57:35.788205+0000","last_undegraded":"2024-06-29T15:57:35.788205+0000","last_fullsized":"2024-06-29T15:57:35.788205+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:51.198488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601233+0000","last_change":"2024-06-29T15:57:34.373191+0000","last_active":"2024-06-29T15:57:35.601233+0000","last_peered":"2024-06-29T15:57:35.601233+0000","last_clean":"2024-06-29T15:57:35.601233+0000","last_became_active":"2024-06-29T15:57:34.372928+0000","last_became_peered":"2024-06-29T15:57:34.372928+0000","last_unstale":"2024-06-29T15:57:35.601233+0000","last_undegraded":"2024-06-29T15:57:35.601233+0000","last_fullsized":"2024-06-29T15:57:35.601233+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:49:19.104188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"212'2","reported_seq":24,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354425+0000","last_change":"2024-06-29T15:57:21.278476+0000","last_active":"2024-06-29T15:57:35.354425+0000","last_peered":"2024-06-29T15:57:35.354425+0000","last_clean":"2024-06-29T15:57:35.354425+0000","last_became_active":"2024-06-29T15:57:21.278150+0000","last_became_peered":"2024-06-29T15:57:21.278150+0000","last_unstale":"2024-06-29T15:57:35.354425+0000","last_undegraded":"2024-06-29T15:57:35.354425+0000","last_fullsized":"2024-06-29T15:57:35.354425+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:42:33.991332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354043+0000","last_change":"2024-06-29T15:57:22.291360+0000","last_active":"2024-06-29T15:57:35.354043+0000","last_peered":"2024-06-29T15:57:35.354043+0000","last_clean":"2024-06-29T15:57:35.354043+0000","last_became_active":"2024-06-29T15:57:22.290801+0000","last_became_peered":"2024-06-29T15:57:22.290801+0000","last_unstale":"2024-06-29T15:57:35.354043+0000","last_undegraded":"2024-06-29T15:57:35.354043+0000","last_fullsized":"2024-06-29T15:57:35.354043+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.787744+0000","last_change":"2024-06-29T15:57:33.367933+0000","last_active":"2024-06-29T15:57:35.787744+0000","last_peered":"2024-06-29T15:57:35.787744+0000","last_clean":"2024-06-29T15:57:35.787744+0000","last_became_active":"2024-06-29T15:57:33.367630+0000","last_became_peered":"2024-06-29T15:57:33.367630+0000","last_unstale":"2024-06-29T15:57:35.787744+0000","last_undegraded":"2024-06-29T15:57:35.787744+0000","last_fullsized":"2024-06-29T15:57:35.787744+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:06.173920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351187+0000","last_change":"2024-06-29T15:57:34.368629+0000","last_active":"2024-06-29T15:57:35.351187+0000","last_peered":"2024-06-29T15:57:35.351187+0000","last_clean":"2024-06-29T15:57:35.351187+0000","last_became_active":"2024-06-29T15:57:34.368328+0000","last_became_peered":"2024-06-29T15:57:34.368328+0000","last_unstale":"2024-06-29T15:57:35.351187+0000","last_undegraded":"2024-06-29T15:57:35.351187+0000","last_fullsized":"2024-06-29T15:57:35.351187+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:18.789499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"212'1","reported_seq":23,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601987+0000","last_change":"2024-06-29T15:57:21.273304+0000","last_active":"2024-06-29T15:57:35.601987+0000","last_peered":"2024-06-29T15:57:35.601987+0000","last_clean":"2024-06-29T15:57:35.601987+0000","last_became_active":"2024-06-29T15:57:21.272972+0000","last_became_peered":"2024-06-29T15:57:21.272972+0000","last_unstale":"2024-06-29T15:57:35.601987+0000","last_undegraded":"2024-06-29T15:57:35.601987+0000","last_fullsized":"2024-06-29T15:57:35.601987+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353623+0000","last_change":"2024-06-29T15:57:22.300916+0000","last_active":"2024-06-29T15:57:35.353623+0000","last_peered":"2024-06-29T15:57:35.353623+0000","last_clean":"2024-06-29T15:57:35.353623+0000","last_became_active":"2024-06-29T15:57:22.300620+0000","last_became_peered":"2024-06-29T15:57:22.300620+0000","last_unstale":"2024-06-29T15:57:35.353623+0000","last_undegraded":"2024-06-29T15:57:35.353623+0000","last_fullsized":"2024-06-29T15:57:35.353623+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:31:41.172455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353467+0000","last_change":"2024-06-29T15:57:33.367013+0000","last_active":"2024-06-29T15:57:35.353467+0000","last_peered":"2024-06-29T15:57:35.353467+0000","last_clean":"2024-06-29T15:57:35.353467+0000","last_became_active":"2024-06-29T15:57:33.366664+0000","last_became_peered":"2024-06-29T15:57:33.366664+0000","last_unstale":"2024-06-29T15:57:35.353467+0000","last_undegraded":"2024-06-29T15:57:35.353467+0000","last_fullsized":"2024-06-29T15:57:35.353467+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:10:51.198488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375560+0000","last_change":"2024-06-29T15:57:34.386970+0000","last_active":"2024-06-29T15:57:35.375560+0000","last_peered":"2024-06-29T15:57:35.375560+0000","last_clean":"2024-06-29T15:57:35.375560+0000","last_became_active":"2024-06-29T15:57:34.386646+0000","last_became_peered":"2024-06-29T15:57:34.386646+0000","last_unstale":"2024-06-29T15:57:35.375560+0000","last_undegraded":"2024-06-29T15:57:35.375560+0000","last_fullsized":"2024-06-29T15:57:35.375560+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:07.862253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"212'8","reported_seq":30,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351444+0000","last_change":"2024-06-29T15:57:21.283745+0000","last_active":"2024-06-29T15:57:35.351444+0000","last_peered":"2024-06-29T15:57:35.351444+0000","last_clean":"2024-06-29T15:57:35.351444+0000","last_became_active":"2024-06-29T15:57:21.283403+0000","last_became_peered":"2024-06-29T15:57:21.283403+0000","last_unstale":"2024-06-29T15:57:35.351444+0000","last_undegraded":"2024-06-29T15:57:35.351444+0000","last_fullsized":"2024-06-29T15:57:35.351444+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288487+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:36.288487+0000","last_peered":"2024-06-29T15:57:36.288487+0000","last_clean":"2024-06-29T15:57:36.288487+0000","last_became_active":"2024-06-29T15:57:22.301832+0000","last_became_peered":"2024-06-29T15:57:22.301832+0000","last_unstale":"2024-06-29T15:57:36.288487+0000","last_undegraded":"2024-06-29T15:57:36.288487+0000","last_fullsized":"2024-06-29T15:57:36.288487+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288810+0000","last_change":"2024-06-29T15:57:33.366032+0000","last_active":"2024-06-29T15:57:36.288810+0000","last_peered":"2024-06-29T15:57:36.288810+0000","last_clean":"2024-06-29T15:57:36.288810+0000","last_became_active":"2024-06-29T15:57:33.365731+0000","last_became_peered":"2024-06-29T15:57:33.365731+0000","last_unstale":"2024-06-29T15:57:36.288810+0000","last_undegraded":"2024-06-29T15:57:36.288810+0000","last_fullsized":"2024-06-29T15:57:36.288810+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:26.956369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355786+0000","last_change":"2024-06-29T15:57:34.380170+0000","last_active":"2024-06-29T15:57:35.355786+0000","last_peered":"2024-06-29T15:57:35.355786+0000","last_clean":"2024-06-29T15:57:35.355786+0000","last_became_active":"2024-06-29T15:57:34.379231+0000","last_became_peered":"2024-06-29T15:57:34.379231+0000","last_unstale":"2024-06-29T15:57:35.355786+0000","last_undegraded":"2024-06-29T15:57:35.355786+0000","last_fullsized":"2024-06-29T15:57:35.355786+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:11:07.094539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288843+0000","last_change":"2024-06-29T15:57:21.276731+0000","last_active":"2024-06-29T15:57:36.288843+0000","last_peered":"2024-06-29T15:57:36.288843+0000","last_clean":"2024-06-29T15:57:36.288843+0000","last_became_active":"2024-06-29T15:57:21.276408+0000","last_became_peered":"2024-06-29T15:57:21.276408+0000","last_unstale":"2024-06-29T15:57:36.288843+0000","last_undegraded":"2024-06-29T15:57:36.288843+0000","last_fullsized":"2024-06-29T15:57:36.288843+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:48:48.911239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351298+0000","last_change":"2024-06-29T15:57:22.299631+0000","last_active":"2024-06-29T15:57:35.351298+0000","last_peered":"2024-06-29T15:57:35.351298+0000","last_clean":"2024-06-29T15:57:35.351298+0000","last_became_active":"2024-06-29T15:57:22.299285+0000","last_became_peered":"2024-06-29T15:57:22.299285+0000","last_unstale":"2024-06-29T15:57:35.351298+0000","last_undegraded":"2024-06-29T15:57:35.351298+0000","last_fullsized":"2024-06-29T15:57:35.351298+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375530+0000","last_change":"2024-06-29T15:57:33.369952+0000","last_active":"2024-06-29T15:57:35.375530+0000","last_peered":"2024-06-29T15:57:35.375530+0000","last_clean":"2024-06-29T15:57:35.375530+0000","last_became_active":"2024-06-29T15:57:33.369598+0000","last_became_peered":"2024-06-29T15:57:33.369598+0000","last_unstale":"2024-06-29T15:57:35.375530+0000","last_undegraded":"2024-06-29T15:57:35.375530+0000","last_fullsized":"2024-06-29T15:57:35.375530+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:42:56.416855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374966+0000","last_change":"2024-06-29T15:57:34.386990+0000","last_active":"2024-06-29T15:57:35.374966+0000","last_peered":"2024-06-29T15:57:35.374966+0000","last_clean":"2024-06-29T15:57:35.374966+0000","last_became_active":"2024-06-29T15:57:34.386682+0000","last_became_peered":"2024-06-29T15:57:34.386682+0000","last_unstale":"2024-06-29T15:57:35.374966+0000","last_undegraded":"2024-06-29T15:57:35.374966+0000","last_fullsized":"2024-06-29T15:57:35.374966+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:01:51.047639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355142+0000","last_change":"2024-06-29T15:57:21.282872+0000","last_active":"2024-06-29T15:57:35.355142+0000","last_peered":"2024-06-29T15:57:35.355142+0000","last_clean":"2024-06-29T15:57:35.355142+0000","last_became_active":"2024-06-29T15:57:21.282549+0000","last_became_peered":"2024-06-29T15:57:21.282549+0000","last_unstale":"2024-06-29T15:57:35.355142+0000","last_undegraded":"2024-06-29T15:57:35.355142+0000","last_fullsized":"2024-06-29T15:57:35.355142+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:11:39.570643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289235+0000","last_change":"2024-06-29T15:57:22.294011+0000","last_active":"2024-06-29T15:57:36.289235+0000","last_peered":"2024-06-29T15:57:36.289235+0000","last_clean":"2024-06-29T15:57:36.289235+0000","last_became_active":"2024-06-29T15:57:22.293721+0000","last_became_peered":"2024-06-29T15:57:22.293721+0000","last_unstale":"2024-06-29T15:57:36.289235+0000","last_undegraded":"2024-06-29T15:57:36.289235+0000","last_fullsized":"2024-06-29T15:57:36.289235+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:57.337708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"216'6","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289475+0000","last_change":"2024-06-29T15:57:33.361883+0000","last_active":"2024-06-29T15:57:36.289475+0000","last_peered":"2024-06-29T15:57:36.289475+0000","last_clean":"2024-06-29T15:57:36.289475+0000","last_became_active":"2024-06-29T15:57:33.361569+0000","last_became_peered":"2024-06-29T15:57:33.361569+0000","last_unstale":"2024-06-29T15:57:36.289475+0000","last_undegraded":"2024-06-29T15:57:36.289475+0000","last_fullsized":"2024-06-29T15:57:36.289475+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:58:50.792873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353294+0000","last_change":"2024-06-29T15:57:34.381340+0000","last_active":"2024-06-29T15:57:35.353294+0000","last_peered":"2024-06-29T15:57:35.353294+0000","last_clean":"2024-06-29T15:57:35.353294+0000","last_became_active":"2024-06-29T15:57:34.380966+0000","last_became_peered":"2024-06-29T15:57:34.380966+0000","last_unstale":"2024-06-29T15:57:35.353294+0000","last_undegraded":"2024-06-29T15:57:35.353294+0000","last_fullsized":"2024-06-29T15:57:35.353294+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:18.789499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"212'2","reported_seq":24,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354945+0000","last_change":"2024-06-29T15:57:21.281309+0000","last_active":"2024-06-29T15:57:35.354945+0000","last_peered":"2024-06-29T15:57:35.354945+0000","last_clean":"2024-06-29T15:57:35.354945+0000","last_became_active":"2024-06-29T15:57:21.280944+0000","last_became_peered":"2024-06-29T15:57:21.280944+0000","last_unstale":"2024-06-29T15:57:35.354945+0000","last_undegraded":"2024-06-29T15:57:35.354945+0000","last_fullsized":"2024-06-29T15:57:35.354945+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:20:55.205760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601427+0000","last_change":"2024-06-29T15:57:22.291164+0000","last_active":"2024-06-29T15:57:35.601427+0000","last_peered":"2024-06-29T15:57:35.601427+0000","last_clean":"2024-06-29T15:57:35.601427+0000","last_became_active":"2024-06-29T15:57:22.290897+0000","last_became_peered":"2024-06-29T15:57:22.290897+0000","last_unstale":"2024-06-29T15:57:35.601427+0000","last_undegraded":"2024-06-29T15:57:35.601427+0000","last_fullsized":"2024-06-29T15:57:35.601427+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:27:11.757823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"216'3","reported_seq":17,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.602014+0000","last_change":"2024-06-29T15:57:33.364026+0000","last_active":"2024-06-29T15:57:35.602014+0000","last_peered":"2024-06-29T15:57:35.602014+0000","last_clean":"2024-06-29T15:57:35.602014+0000","last_became_active":"2024-06-29T15:57:33.363658+0000","last_became_peered":"2024-06-29T15:57:33.363658+0000","last_unstale":"2024-06-29T15:57:35.602014+0000","last_undegraded":"2024-06-29T15:57:35.602014+0000","last_fullsized":"2024-06-29T15:57:35.602014+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:47.234190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788620+0000","last_change":"2024-06-29T15:57:34.372240+0000","last_active":"2024-06-29T15:57:35.788620+0000","last_peered":"2024-06-29T15:57:35.788620+0000","last_clean":"2024-06-29T15:57:35.788620+0000","last_became_active":"2024-06-29T15:57:34.371893+0000","last_became_peered":"2024-06-29T15:57:34.371893+0000","last_unstale":"2024-06-29T15:57:35.788620+0000","last_undegraded":"2024-06-29T15:57:35.788620+0000","last_fullsized":"2024-06-29T15:57:35.788620+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:48.165689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"214'8","reported_seq":30,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375081+0000","last_change":"2024-06-29T15:57:21.278161+0000","last_active":"2024-06-29T15:57:35.375081+0000","last_peered":"2024-06-29T15:57:35.375081+0000","last_clean":"2024-06-29T15:57:35.375081+0000","last_became_active":"2024-06-29T15:57:21.277794+0000","last_became_peered":"2024-06-29T15:57:21.277794+0000","last_unstale":"2024-06-29T15:57:35.375081+0000","last_undegraded":"2024-06-29T15:57:35.375081+0000","last_fullsized":"2024-06-29T15:57:35.375081+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:56.330168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288921+0000","last_change":"2024-06-29T15:57:22.302666+0000","last_active":"2024-06-29T15:57:36.288921+0000","last_peered":"2024-06-29T15:57:36.288921+0000","last_clean":"2024-06-29T15:57:36.288921+0000","last_became_active":"2024-06-29T15:57:22.302254+0000","last_became_peered":"2024-06-29T15:57:22.302254+0000","last_unstale":"2024-06-29T15:57:36.288921+0000","last_undegraded":"2024-06-29T15:57:36.288921+0000","last_fullsized":"2024-06-29T15:57:36.288921+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:55.101212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288667+0000","last_change":"2024-06-29T15:57:33.361374+0000","last_active":"2024-06-29T15:57:36.288667+0000","last_peered":"2024-06-29T15:57:36.288667+0000","last_clean":"2024-06-29T15:57:36.288667+0000","last_became_active":"2024-06-29T15:57:33.361002+0000","last_became_peered":"2024-06-29T15:57:33.361002+0000","last_unstale":"2024-06-29T15:57:36.288667+0000","last_undegraded":"2024-06-29T15:57:36.288667+0000","last_fullsized":"2024-06-29T15:57:36.288667+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:17.781596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288454+0000","last_change":"2024-06-29T15:57:34.364598+0000","last_active":"2024-06-29T15:57:36.288454+0000","last_peered":"2024-06-29T15:57:36.288454+0000","last_clean":"2024-06-29T15:57:36.288454+0000","last_became_active":"2024-06-29T15:57:34.364292+0000","last_became_peered":"2024-06-29T15:57:34.364292+0000","last_unstale":"2024-06-29T15:57:36.288454+0000","last_undegraded":"2024-06-29T15:57:36.288454+0000","last_fullsized":"2024-06-29T15:57:36.288454+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:49:19.104188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"212'3","reported_seq":25,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289598+0000","last_change":"2024-06-29T15:57:21.283791+0000","last_active":"2024-06-29T15:57:36.289598+0000","last_peered":"2024-06-29T15:57:36.289598+0000","last_clean":"2024-06-29T15:57:36.289598+0000","last_became_active":"2024-06-29T15:57:21.283458+0000","last_became_peered":"2024-06-29T15:57:21.283458+0000","last_unstale":"2024-06-29T15:57:36.289598+0000","last_undegraded":"2024-06-29T15:57:36.289598+0000","last_fullsized":"2024-06-29T15:57:36.289598+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:24:54.365881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374770+0000","last_change":"2024-06-29T15:57:22.295260+0000","last_active":"2024-06-29T15:57:35.374770+0000","last_peered":"2024-06-29T15:57:35.374770+0000","last_clean":"2024-06-29T15:57:35.374770+0000","last_became_active":"2024-06-29T15:57:22.294952+0000","last_became_peered":"2024-06-29T15:57:22.294952+0000","last_unstale":"2024-06-29T15:57:35.374770+0000","last_undegraded":"2024-06-29T15:57:35.374770+0000","last_fullsized":"2024-06-29T15:57:35.374770+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:37:56.473182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"216'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351477+0000","last_change":"2024-06-29T15:57:33.370460+0000","last_active":"2024-06-29T15:57:35.351477+0000","last_peered":"2024-06-29T15:57:35.351477+0000","last_clean":"2024-06-29T15:57:35.351477+0000","last_became_active":"2024-06-29T15:57:33.370183+0000","last_became_peered":"2024-06-29T15:57:33.370183+0000","last_unstale":"2024-06-29T15:57:35.351477+0000","last_undegraded":"2024-06-29T15:57:35.351477+0000","last_fullsized":"2024-06-29T15:57:35.351477+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:46:09.697181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.352921+0000","last_change":"2024-06-29T15:57:34.359380+0000","last_active":"2024-06-29T15:57:35.352921+0000","last_peered":"2024-06-29T15:57:35.352921+0000","last_clean":"2024-06-29T15:57:35.352921+0000","last_became_active":"2024-06-29T15:57:34.359025+0000","last_became_peered":"2024-06-29T15:57:34.359025+0000","last_unstale":"2024-06-29T15:57:35.352921+0000","last_undegraded":"2024-06-29T15:57:35.352921+0000","last_fullsized":"2024-06-29T15:57:35.352921+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:19:29.854075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355795+0000","last_change":"2024-06-29T15:57:21.275926+0000","last_active":"2024-06-29T15:57:35.355795+0000","last_peered":"2024-06-29T15:57:35.355795+0000","last_clean":"2024-06-29T15:57:35.355795+0000","last_became_active":"2024-06-29T15:57:21.275593+0000","last_became_peered":"2024-06-29T15:57:21.275593+0000","last_unstale":"2024-06-29T15:57:35.355795+0000","last_undegraded":"2024-06-29T15:57:35.355795+0000","last_fullsized":"2024-06-29T15:57:35.355795+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.352009+0000","last_change":"2024-06-29T15:57:22.298454+0000","last_active":"2024-06-29T15:57:35.352009+0000","last_peered":"2024-06-29T15:57:35.352009+0000","last_clean":"2024-06-29T15:57:35.352009+0000","last_became_active":"2024-06-29T15:57:22.298162+0000","last_became_peered":"2024-06-29T15:57:22.298162+0000","last_unstale":"2024-06-29T15:57:35.352009+0000","last_undegraded":"2024-06-29T15:57:35.352009+0000","last_fullsized":"2024-06-29T15:57:35.352009+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"216'8","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354412+0000","last_change":"2024-06-29T15:57:33.362039+0000","last_active":"2024-06-29T15:57:35.354412+0000","last_peered":"2024-06-29T15:57:35.354412+0000","last_clean":"2024-06-29T15:57:35.354412+0000","last_became_active":"2024-06-29T15:57:33.361665+0000","last_became_peered":"2024-06-29T15:57:33.361665+0000","last_unstale":"2024-06-29T15:57:35.354412+0000","last_undegraded":"2024-06-29T15:57:35.354412+0000","last_fullsized":"2024-06-29T15:57:35.354412+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:48:21.609507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788184+0000","last_change":"2024-06-29T15:57:34.370578+0000","last_active":"2024-06-29T15:57:35.788184+0000","last_peered":"2024-06-29T15:57:35.788184+0000","last_clean":"2024-06-29T15:57:35.788184+0000","last_became_active":"2024-06-29T15:57:34.370279+0000","last_became_peered":"2024-06-29T15:57:34.370279+0000","last_unstale":"2024-06-29T15:57:35.788184+0000","last_undegraded":"2024-06-29T15:57:35.788184+0000","last_fullsized":"2024-06-29T15:57:35.788184+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:45:27.007653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601906+0000","last_change":"2024-06-29T15:57:21.272019+0000","last_active":"2024-06-29T15:57:35.601906+0000","last_peered":"2024-06-29T15:57:35.601906+0000","last_clean":"2024-06-29T15:57:35.601906+0000","last_became_active":"2024-06-29T15:57:21.271715+0000","last_became_peered":"2024-06-29T15:57:21.271715+0000","last_unstale":"2024-06-29T15:57:35.601906+0000","last_undegraded":"2024-06-29T15:57:35.601906+0000","last_fullsized":"2024-06-29T15:57:35.601906+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:48.531828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375188+0000","last_change":"2024-06-29T15:57:22.303487+0000","last_active":"2024-06-29T15:57:35.375188+0000","last_peered":"2024-06-29T15:57:35.375188+0000","last_clean":"2024-06-29T15:57:35.375188+0000","last_became_active":"2024-06-29T15:57:22.302933+0000","last_became_peered":"2024-06-29T15:57:22.302933+0000","last_unstale":"2024-06-29T15:57:35.375188+0000","last_undegraded":"2024-06-29T15:57:35.375188+0000","last_fullsized":"2024-06-29T15:57:35.375188+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:28.493187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601297+0000","last_change":"2024-06-29T15:57:33.361560+0000","last_active":"2024-06-29T15:57:35.601297+0000","last_peered":"2024-06-29T15:57:35.601297+0000","last_clean":"2024-06-29T15:57:35.601297+0000","last_became_active":"2024-06-29T15:57:33.361270+0000","last_became_peered":"2024-06-29T15:57:33.361270+0000","last_unstale":"2024-06-29T15:57:35.601297+0000","last_undegraded":"2024-06-29T15:57:35.601297+0000","last_fullsized":"2024-06-29T15:57:35.601297+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:08.999005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.352017+0000","last_change":"2024-06-29T15:57:34.387085+0000","last_active":"2024-06-29T15:57:35.352017+0000","last_peered":"2024-06-29T15:57:35.352017+0000","last_clean":"2024-06-29T15:57:35.352017+0000","last_became_active":"2024-06-29T15:57:34.386739+0000","last_became_peered":"2024-06-29T15:57:34.386739+0000","last_unstale":"2024-06-29T15:57:35.352017+0000","last_undegraded":"2024-06-29T15:57:35.352017+0000","last_fullsized":"2024-06-29T15:57:35.352017+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:53.743438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"216'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355347+0000","last_change":"2024-06-29T15:57:33.362045+0000","last_active":"2024-06-29T15:57:35.355347+0000","last_peered":"2024-06-29T15:57:35.355347+0000","last_clean":"2024-06-29T15:57:35.355347+0000","last_became_active":"2024-06-29T15:57:33.361647+0000","last_became_peered":"2024-06-29T15:57:33.361647+0000","last_unstale":"2024-06-29T15:57:35.355347+0000","last_undegraded":"2024-06-29T15:57:35.355347+0000","last_fullsized":"2024-06-29T15:57:35.355347+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:01:13.682132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355164+0000","last_change":"2024-06-29T15:57:21.277245+0000","last_active":"2024-06-29T15:57:35.355164+0000","last_peered":"2024-06-29T15:57:35.355164+0000","last_clean":"2024-06-29T15:57:35.355164+0000","last_became_active":"2024-06-29T15:57:21.276931+0000","last_became_peered":"2024-06-29T15:57:21.276931+0000","last_unstale":"2024-06-29T15:57:35.355164+0000","last_undegraded":"2024-06-29T15:57:35.355164+0000","last_fullsized":"2024-06-29T15:57:35.355164+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:18.362019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374733+0000","last_change":"2024-06-29T15:57:22.299457+0000","last_active":"2024-06-29T15:57:35.374733+0000","last_peered":"2024-06-29T15:57:35.374733+0000","last_clean":"2024-06-29T15:57:35.374733+0000","last_became_active":"2024-06-29T15:57:22.299157+0000","last_became_peered":"2024-06-29T15:57:22.299157+0000","last_unstale":"2024-06-29T15:57:35.374733+0000","last_undegraded":"2024-06-29T15:57:35.374733+0000","last_fullsized":"2024-06-29T15:57:35.374733+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:40.967292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.789021+0000","last_change":"2024-06-29T15:57:34.385314+0000","last_active":"2024-06-29T15:57:35.789021+0000","last_peered":"2024-06-29T15:57:35.789021+0000","last_clean":"2024-06-29T15:57:35.789021+0000","last_became_active":"2024-06-29T15:57:34.384883+0000","last_became_peered":"2024-06-29T15:57:34.384883+0000","last_unstale":"2024-06-29T15:57:35.789021+0000","last_undegraded":"2024-06-29T15:57:35.789021+0000","last_fullsized":"2024-06-29T15:57:35.789021+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:43:07.214490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"216'4","reported_seq":18,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288810+0000","last_change":"2024-06-29T15:57:33.360662+0000","last_active":"2024-06-29T15:57:36.288810+0000","last_peered":"2024-06-29T15:57:36.288810+0000","last_clean":"2024-06-29T15:57:36.288810+0000","last_became_active":"2024-06-29T15:57:33.360087+0000","last_became_peered":"2024-06-29T15:57:33.360087+0000","last_unstale":"2024-06-29T15:57:36.288810+0000","last_undegraded":"2024-06-29T15:57:36.288810+0000","last_fullsized":"2024-06-29T15:57:36.288810+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:52.735534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":22,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601413+0000","last_change":"2024-06-29T15:57:21.277884+0000","last_active":"2024-06-29T15:57:35.601413+0000","last_peered":"2024-06-29T15:57:35.601413+0000","last_clean":"2024-06-29T15:57:35.601413+0000","last_became_active":"2024-06-29T15:57:21.277589+0000","last_became_peered":"2024-06-29T15:57:21.277589+0000","last_unstale":"2024-06-29T15:57:35.601413+0000","last_undegraded":"2024-06-29T15:57:35.601413+0000","last_fullsized":"2024-06-29T15:57:35.601413+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:05:09.017396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601261+0000","last_change":"2024-06-29T15:57:22.293696+0000","last_active":"2024-06-29T15:57:35.601261+0000","last_peered":"2024-06-29T15:57:35.601261+0000","last_clean":"2024-06-29T15:57:35.601261+0000","last_became_active":"2024-06-29T15:57:22.293263+0000","last_became_peered":"2024-06-29T15:57:22.293263+0000","last_unstale":"2024-06-29T15:57:35.601261+0000","last_undegraded":"2024-06-29T15:57:35.601261+0000","last_fullsized":"2024-06-29T15:57:35.601261+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353393+0000","last_change":"2024-06-29T15:57:34.381186+0000","last_active":"2024-06-29T15:57:35.353393+0000","last_peered":"2024-06-29T15:57:35.353393+0000","last_clean":"2024-06-29T15:57:35.353393+0000","last_became_active":"2024-06-29T15:57:34.380847+0000","last_became_peered":"2024-06-29T15:57:34.380847+0000","last_unstale":"2024-06-29T15:57:35.353393+0000","last_undegraded":"2024-06-29T15:57:35.353393+0000","last_fullsized":"2024-06-29T15:57:35.353393+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:27.964272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"212'1","reported_seq":23,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601493+0000","last_change":"2024-06-29T15:57:21.274306+0000","last_active":"2024-06-29T15:57:35.601493+0000","last_peered":"2024-06-29T15:57:35.601493+0000","last_clean":"2024-06-29T15:57:35.601493+0000","last_became_active":"2024-06-29T15:57:21.273632+0000","last_became_peered":"2024-06-29T15:57:21.273632+0000","last_unstale":"2024-06-29T15:57:35.601493+0000","last_undegraded":"2024-06-29T15:57:35.601493+0000","last_fullsized":"2024-06-29T15:57:35.601493+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:06.848308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374802+0000","last_change":"2024-06-29T15:57:22.297633+0000","last_active":"2024-06-29T15:57:35.374802+0000","last_peered":"2024-06-29T15:57:35.374802+0000","last_clean":"2024-06-29T15:57:35.374802+0000","last_became_active":"2024-06-29T15:57:22.297319+0000","last_became_peered":"2024-06-29T15:57:22.297319+0000","last_unstale":"2024-06-29T15:57:35.374802+0000","last_undegraded":"2024-06-29T15:57:35.374802+0000","last_fullsized":"2024-06-29T15:57:35.374802+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:14:19.625176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"216'2","reported_seq":16,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355476+0000","last_change":"2024-06-29T15:57:33.355194+0000","last_active":"2024-06-29T15:57:35.355476+0000","last_peered":"2024-06-29T15:57:35.355476+0000","last_clean":"2024-06-29T15:57:35.355476+0000","last_became_active":"2024-06-29T15:57:33.354883+0000","last_became_peered":"2024-06-29T15:57:33.354883+0000","last_unstale":"2024-06-29T15:57:35.355476+0000","last_undegraded":"2024-06-29T15:57:35.355476+0000","last_fullsized":"2024-06-29T15:57:35.355476+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:14:06.322225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601689+0000","last_change":"2024-06-29T15:57:34.378134+0000","last_active":"2024-06-29T15:57:35.601689+0000","last_peered":"2024-06-29T15:57:35.601689+0000","last_clean":"2024-06-29T15:57:35.601689+0000","last_became_active":"2024-06-29T15:57:34.377816+0000","last_became_peered":"2024-06-29T15:57:34.377816+0000","last_unstale":"2024-06-29T15:57:35.601689+0000","last_undegraded":"2024-06-29T15:57:35.601689+0000","last_fullsized":"2024-06-29T15:57:35.601689+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:33.372620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"212'2","reported_seq":24,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374935+0000","last_change":"2024-06-29T15:57:21.283411+0000","last_active":"2024-06-29T15:57:35.374935+0000","last_peered":"2024-06-29T15:57:35.374935+0000","last_clean":"2024-06-29T15:57:35.374935+0000","last_became_active":"2024-06-29T15:57:21.283081+0000","last_became_peered":"2024-06-29T15:57:21.283081+0000","last_unstale":"2024-06-29T15:57:35.374935+0000","last_undegraded":"2024-06-29T15:57:35.374935+0000","last_fullsized":"2024-06-29T15:57:35.374935+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:27.866266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289392+0000","last_change":"2024-06-29T15:57:22.306835+0000","last_active":"2024-06-29T15:57:36.289392+0000","last_peered":"2024-06-29T15:57:36.289392+0000","last_clean":"2024-06-29T15:57:36.289392+0000","last_became_active":"2024-06-29T15:57:22.306295+0000","last_became_peered":"2024-06-29T15:57:22.306295+0000","last_unstale":"2024-06-29T15:57:36.289392+0000","last_undegraded":"2024-06-29T15:57:36.289392+0000","last_fullsized":"2024-06-29T15:57:36.289392+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:48:52.124235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"216'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.787969+0000","last_change":"2024-06-29T15:57:33.365463+0000","last_active":"2024-06-29T15:57:35.787969+0000","last_peered":"2024-06-29T15:57:35.787969+0000","last_clean":"2024-06-29T15:57:35.787969+0000","last_became_active":"2024-06-29T15:57:33.365149+0000","last_became_peered":"2024-06-29T15:57:33.365149+0000","last_unstale":"2024-06-29T15:57:35.787969+0000","last_undegraded":"2024-06-29T15:57:35.787969+0000","last_fullsized":"2024-06-29T15:57:35.787969+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:27.582662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374570+0000","last_change":"2024-06-29T15:57:34.378237+0000","last_active":"2024-06-29T15:57:35.374570+0000","last_peered":"2024-06-29T15:57:35.374570+0000","last_clean":"2024-06-29T15:57:35.374570+0000","last_became_active":"2024-06-29T15:57:34.377917+0000","last_became_peered":"2024-06-29T15:57:34.377917+0000","last_unstale":"2024-06-29T15:57:35.374570+0000","last_undegraded":"2024-06-29T15:57:35.374570+0000","last_fullsized":"2024-06-29T15:57:35.374570+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:15.368536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"212'2","reported_seq":24,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351935+0000","last_change":"2024-06-29T15:57:21.284516+0000","last_active":"2024-06-29T15:57:35.351935+0000","last_peered":"2024-06-29T15:57:35.351935+0000","last_clean":"2024-06-29T15:57:35.351935+0000","last_became_active":"2024-06-29T15:57:21.284170+0000","last_became_peered":"2024-06-29T15:57:21.284170+0000","last_unstale":"2024-06-29T15:57:35.351935+0000","last_undegraded":"2024-06-29T15:57:35.351935+0000","last_fullsized":"2024-06-29T15:57:35.351935+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:40:01.863764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288564+0000","last_change":"2024-06-29T15:57:22.300202+0000","last_active":"2024-06-29T15:57:36.288564+0000","last_peered":"2024-06-29T15:57:36.288564+0000","last_clean":"2024-06-29T15:57:36.288564+0000","last_became_active":"2024-06-29T15:57:22.299897+0000","last_became_peered":"2024-06-29T15:57:22.299897+0000","last_unstale":"2024-06-29T15:57:36.288564+0000","last_undegraded":"2024-06-29T15:57:36.288564+0000","last_fullsized":"2024-06-29T15:57:36.288564+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"216'1","reported_seq":15,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351288+0000","last_change":"2024-06-29T15:57:33.363355+0000","last_active":"2024-06-29T15:57:35.351288+0000","last_peered":"2024-06-29T15:57:35.351288+0000","last_clean":"2024-06-29T15:57:35.351288+0000","last_became_active":"2024-06-29T15:57:33.363067+0000","last_became_peered":"2024-06-29T15:57:33.363067+0000","last_unstale":"2024-06-29T15:57:35.351288+0000","last_undegraded":"2024-06-29T15:57:35.351288+0000","last_fullsized":"2024-06-29T15:57:35.351288+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:45:08.999005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354602+0000","last_change":"2024-06-29T15:57:34.380150+0000","last_active":"2024-06-29T15:57:35.354602+0000","last_peered":"2024-06-29T15:57:35.354602+0000","last_clean":"2024-06-29T15:57:35.354602+0000","last_became_active":"2024-06-29T15:57:34.379166+0000","last_became_peered":"2024-06-29T15:57:34.379166+0000","last_unstale":"2024-06-29T15:57:35.354602+0000","last_undegraded":"2024-06-29T15:57:35.354602+0000","last_fullsized":"2024-06-29T15:57:35.354602+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:18.789499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"212'1","reported_seq":23,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354296+0000","last_change":"2024-06-29T15:57:21.274833+0000","last_active":"2024-06-29T15:57:35.354296+0000","last_peered":"2024-06-29T15:57:35.354296+0000","last_clean":"2024-06-29T15:57:35.354296+0000","last_became_active":"2024-06-29T15:57:21.274451+0000","last_became_peered":"2024-06-29T15:57:21.274451+0000","last_unstale":"2024-06-29T15:57:35.354296+0000","last_undegraded":"2024-06-29T15:57:35.354296+0000","last_fullsized":"2024-06-29T15:57:35.354296+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:41:25.284538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355547+0000","last_change":"2024-06-29T15:57:22.301084+0000","last_active":"2024-06-29T15:57:35.355547+0000","last_peered":"2024-06-29T15:57:35.355547+0000","last_clean":"2024-06-29T15:57:35.355547+0000","last_became_active":"2024-06-29T15:57:22.300639+0000","last_became_peered":"2024-06-29T15:57:22.300639+0000","last_unstale":"2024-06-29T15:57:35.355547+0000","last_undegraded":"2024-06-29T15:57:35.355547+0000","last_fullsized":"2024-06-29T15:57:35.355547+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288229+0000","last_change":"2024-06-29T15:57:33.358185+0000","last_active":"2024-06-29T15:57:36.288229+0000","last_peered":"2024-06-29T15:57:36.288229+0000","last_clean":"2024-06-29T15:57:36.288229+0000","last_became_active":"2024-06-29T15:57:33.357885+0000","last_became_peered":"2024-06-29T15:57:33.357885+0000","last_unstale":"2024-06-29T15:57:36.288229+0000","last_undegraded":"2024-06-29T15:57:36.288229+0000","last_fullsized":"2024-06-29T15:57:36.288229+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:33.638690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288480+0000","last_change":"2024-06-29T15:57:34.362204+0000","last_active":"2024-06-29T15:57:36.288480+0000","last_peered":"2024-06-29T15:57:36.288480+0000","last_clean":"2024-06-29T15:57:36.288480+0000","last_became_active":"2024-06-29T15:57:34.361833+0000","last_became_peered":"2024-06-29T15:57:34.361833+0000","last_unstale":"2024-06-29T15:57:36.288480+0000","last_undegraded":"2024-06-29T15:57:36.288480+0000","last_fullsized":"2024-06-29T15:57:36.288480+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:31.524696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"212'4","reported_seq":26,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.789184+0000","last_change":"2024-06-29T15:57:21.276327+0000","last_active":"2024-06-29T15:57:35.789184+0000","last_peered":"2024-06-29T15:57:35.789184+0000","last_clean":"2024-06-29T15:57:35.789184+0000","last_became_active":"2024-06-29T15:57:21.276031+0000","last_became_peered":"2024-06-29T15:57:21.276031+0000","last_unstale":"2024-06-29T15:57:35.789184+0000","last_undegraded":"2024-06-29T15:57:35.789184+0000","last_fullsized":"2024-06-29T15:57:35.789184+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":210,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:20.251802+0000","last_clean_scrub_stamp":"2024-06-29T15:57:20.251802+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:29:23.626777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353472+0000","last_change":"2024-06-29T15:57:22.293417+0000","last_active":"2024-06-29T15:57:35.353472+0000","last_peered":"2024-06-29T15:57:35.353472+0000","last_clean":"2024-06-29T15:57:35.353472+0000","last_became_active":"2024-06-29T15:57:22.293083+0000","last_became_peered":"2024-06-29T15:57:22.293083+0000","last_unstale":"2024-06-29T15:57:35.353472+0000","last_undegraded":"2024-06-29T15:57:35.353472+0000","last_fullsized":"2024-06-29T15:57:35.353472+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":14,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.352813+0000","last_change":"2024-06-29T15:57:33.356187+0000","last_active":"2024-06-29T15:57:35.352813+0000","last_peered":"2024-06-29T15:57:35.352813+0000","last_clean":"2024-06-29T15:57:35.352813+0000","last_became_active":"2024-06-29T15:57:33.355400+0000","last_became_peered":"2024-06-29T15:57:33.355400+0000","last_unstale":"2024-06-29T15:57:35.352813+0000","last_undegraded":"2024-06-29T15:57:35.352813+0000","last_fullsized":"2024-06-29T15:57:35.352813+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":214,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:32.332050+0000","last_clean_scrub_stamp":"2024-06-29T15:57:32.332050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:51:06.173920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351975+0000","last_change":"2024-06-29T15:57:34.371307+0000","last_active":"2024-06-29T15:57:35.351975+0000","last_peered":"2024-06-29T15:57:35.351975+0000","last_clean":"2024-06-29T15:57:35.351975+0000","last_became_active":"2024-06-29T15:57:34.370958+0000","last_became_peered":"2024-06-29T15:57:34.370958+0000","last_unstale":"2024-06-29T15:57:35.351975+0000","last_undegraded":"2024-06-29T15:57:35.351975+0000","last_fullsized":"2024-06-29T15:57:35.351975+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:41:20.717589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288144+0000","last_change":"2024-06-29T15:57:22.298975+0000","last_active":"2024-06-29T15:57:36.288144+0000","last_peered":"2024-06-29T15:57:36.288144+0000","last_clean":"2024-06-29T15:57:36.288144+0000","last_became_active":"2024-06-29T15:57:22.298689+0000","last_became_peered":"2024-06-29T15:57:22.298689+0000","last_unstale":"2024-06-29T15:57:36.288144+0000","last_undegraded":"2024-06-29T15:57:36.288144+0000","last_fullsized":"2024-06-29T15:57:36.288144+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375428+0000","last_change":"2024-06-29T15:57:34.386053+0000","last_active":"2024-06-29T15:57:35.375428+0000","last_peered":"2024-06-29T15:57:35.375428+0000","last_clean":"2024-06-29T15:57:35.375428+0000","last_became_active":"2024-06-29T15:57:34.384907+0000","last_became_peered":"2024-06-29T15:57:34.384907+0000","last_unstale":"2024-06-29T15:57:35.375428+0000","last_undegraded":"2024-06-29T15:57:35.375428+0000","last_fullsized":"2024-06-29T15:57:35.375428+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:11:44.626093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288725+0000","last_change":"2024-06-29T15:57:22.292249+0000","last_active":"2024-06-29T15:57:36.288725+0000","last_peered":"2024-06-29T15:57:36.288725+0000","last_clean":"2024-06-29T15:57:36.288725+0000","last_became_active":"2024-06-29T15:57:22.291918+0000","last_became_peered":"2024-06-29T15:57:22.291918+0000","last_unstale":"2024-06-29T15:57:36.288725+0000","last_undegraded":"2024-06-29T15:57:36.288725+0000","last_fullsized":"2024-06-29T15:57:36.288725+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:00:57.129905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289433+0000","last_change":"2024-06-29T15:57:34.358618+0000","last_active":"2024-06-29T15:57:36.289433+0000","last_peered":"2024-06-29T15:57:36.289433+0000","last_clean":"2024-06-29T15:57:36.289433+0000","last_became_active":"2024-06-29T15:57:34.358278+0000","last_became_peered":"2024-06-29T15:57:34.358278+0000","last_unstale":"2024-06-29T15:57:36.289433+0000","last_undegraded":"2024-06-29T15:57:36.289433+0000","last_fullsized":"2024-06-29T15:57:36.289433+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:12.829802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354223+0000","last_change":"2024-06-29T15:57:22.285362+0000","last_active":"2024-06-29T15:57:35.354223+0000","last_peered":"2024-06-29T15:57:35.354223+0000","last_clean":"2024-06-29T15:57:35.354223+0000","last_became_active":"2024-06-29T15:57:22.284982+0000","last_became_peered":"2024-06-29T15:57:22.284982+0000","last_unstale":"2024-06-29T15:57:35.354223+0000","last_undegraded":"2024-06-29T15:57:35.354223+0000","last_fullsized":"2024-06-29T15:57:35.354223+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375241+0000","last_change":"2024-06-29T15:57:34.387924+0000","last_active":"2024-06-29T15:57:35.375241+0000","last_peered":"2024-06-29T15:57:35.375241+0000","last_clean":"2024-06-29T15:57:35.375241+0000","last_became_active":"2024-06-29T15:57:34.386397+0000","last_became_peered":"2024-06-29T15:57:34.386397+0000","last_unstale":"2024-06-29T15:57:35.375241+0000","last_undegraded":"2024-06-29T15:57:35.375241+0000","last_fullsized":"2024-06-29T15:57:35.375241+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:03:40.231259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788347+0000","last_change":"2024-06-29T15:57:22.297998+0000","last_active":"2024-06-29T15:57:35.788347+0000","last_peered":"2024-06-29T15:57:35.788347+0000","last_clean":"2024-06-29T15:57:35.788347+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:35.788347+0000","last_undegraded":"2024-06-29T15:57:35.788347+0000","last_fullsized":"2024-06-29T15:57:35.788347+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:07.855847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374521+0000","last_change":"2024-06-29T15:57:34.382524+0000","last_active":"2024-06-29T15:57:35.374521+0000","last_peered":"2024-06-29T15:57:35.374521+0000","last_clean":"2024-06-29T15:57:35.374521+0000","last_became_active":"2024-06-29T15:57:34.382182+0000","last_became_peered":"2024-06-29T15:57:34.382182+0000","last_unstale":"2024-06-29T15:57:35.374521+0000","last_undegraded":"2024-06-29T15:57:35.374521+0000","last_fullsized":"2024-06-29T15:57:35.374521+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:12.861697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351560+0000","last_change":"2024-06-29T15:57:22.297917+0000","last_active":"2024-06-29T15:57:35.351560+0000","last_peered":"2024-06-29T15:57:35.351560+0000","last_clean":"2024-06-29T15:57:35.351560+0000","last_became_active":"2024-06-29T15:57:22.297638+0000","last_became_peered":"2024-06-29T15:57:22.297638+0000","last_unstale":"2024-06-29T15:57:35.351560+0000","last_undegraded":"2024-06-29T15:57:35.351560+0000","last_fullsized":"2024-06-29T15:57:35.351560+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353173+0000","last_change":"2024-06-29T15:57:34.381258+0000","last_active":"2024-06-29T15:57:35.353173+0000","last_peered":"2024-06-29T15:57:35.353173+0000","last_clean":"2024-06-29T15:57:35.353173+0000","last_became_active":"2024-06-29T15:57:34.380981+0000","last_became_peered":"2024-06-29T15:57:34.380981+0000","last_unstale":"2024-06-29T15:57:35.353173+0000","last_undegraded":"2024-06-29T15:57:35.353173+0000","last_fullsized":"2024-06-29T15:57:35.353173+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:44:34.121083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.789016+0000","last_change":"2024-06-29T15:57:22.298174+0000","last_active":"2024-06-29T15:57:35.789016+0000","last_peered":"2024-06-29T15:57:35.789016+0000","last_clean":"2024-06-29T15:57:35.789016+0000","last_became_active":"2024-06-29T15:57:22.297862+0000","last_became_peered":"2024-06-29T15:57:22.297862+0000","last_unstale":"2024-06-29T15:57:35.789016+0000","last_undegraded":"2024-06-29T15:57:35.789016+0000","last_fullsized":"2024-06-29T15:57:35.789016+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:00:49.481259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288010+0000","last_change":"2024-06-29T15:57:34.377935+0000","last_active":"2024-06-29T15:57:36.288010+0000","last_peered":"2024-06-29T15:57:36.288010+0000","last_clean":"2024-06-29T15:57:36.288010+0000","last_became_active":"2024-06-29T15:57:34.377599+0000","last_became_peered":"2024-06-29T15:57:34.377599+0000","last_unstale":"2024-06-29T15:57:36.288010+0000","last_undegraded":"2024-06-29T15:57:36.288010+0000","last_fullsized":"2024-06-29T15:57:36.288010+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:09:21.704940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289069+0000","last_change":"2024-06-29T15:57:22.302136+0000","last_active":"2024-06-29T15:57:36.289069+0000","last_peered":"2024-06-29T15:57:36.289069+0000","last_clean":"2024-06-29T15:57:36.289069+0000","last_became_active":"2024-06-29T15:57:22.301541+0000","last_became_peered":"2024-06-29T15:57:22.301541+0000","last_unstale":"2024-06-29T15:57:36.289069+0000","last_undegraded":"2024-06-29T15:57:36.289069+0000","last_fullsized":"2024-06-29T15:57:36.289069+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:13:55.249516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601187+0000","last_change":"2024-06-29T15:57:34.365376+0000","last_active":"2024-06-29T15:57:35.601187+0000","last_peered":"2024-06-29T15:57:35.601187+0000","last_clean":"2024-06-29T15:57:35.601187+0000","last_became_active":"2024-06-29T15:57:34.365103+0000","last_became_peered":"2024-06-29T15:57:34.365103+0000","last_unstale":"2024-06-29T15:57:35.601187+0000","last_undegraded":"2024-06-29T15:57:35.601187+0000","last_fullsized":"2024-06-29T15:57:35.601187+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:45:27.007653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354011+0000","last_change":"2024-06-29T15:57:22.292584+0000","last_active":"2024-06-29T15:57:35.354011+0000","last_peered":"2024-06-29T15:57:35.354011+0000","last_clean":"2024-06-29T15:57:35.354011+0000","last_became_active":"2024-06-29T15:57:22.291806+0000","last_became_peered":"2024-06-29T15:57:22.291806+0000","last_unstale":"2024-06-29T15:57:35.354011+0000","last_undegraded":"2024-06-29T15:57:35.354011+0000","last_fullsized":"2024-06-29T15:57:35.354011+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289527+0000","last_change":"2024-06-29T15:57:34.374043+0000","last_active":"2024-06-29T15:57:36.289527+0000","last_peered":"2024-06-29T15:57:36.289527+0000","last_clean":"2024-06-29T15:57:36.289527+0000","last_became_active":"2024-06-29T15:57:34.373740+0000","last_became_peered":"2024-06-29T15:57:34.373740+0000","last_unstale":"2024-06-29T15:57:36.289527+0000","last_undegraded":"2024-06-29T15:57:36.289527+0000","last_fullsized":"2024-06-29T15:57:36.289527+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:03.844746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374566+0000","last_change":"2024-06-29T15:57:22.301601+0000","last_active":"2024-06-29T15:57:35.374566+0000","last_peered":"2024-06-29T15:57:35.374566+0000","last_clean":"2024-06-29T15:57:35.374566+0000","last_became_active":"2024-06-29T15:57:22.301244+0000","last_became_peered":"2024-06-29T15:57:22.301244+0000","last_unstale":"2024-06-29T15:57:35.374566+0000","last_undegraded":"2024-06-29T15:57:35.374566+0000","last_fullsized":"2024-06-29T15:57:35.374566+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:24:56.938464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353888+0000","last_change":"2024-06-29T15:57:34.370224+0000","last_active":"2024-06-29T15:57:35.353888+0000","last_peered":"2024-06-29T15:57:35.353888+0000","last_clean":"2024-06-29T15:57:35.353888+0000","last_became_active":"2024-06-29T15:57:34.369898+0000","last_became_peered":"2024-06-29T15:57:34.369898+0000","last_unstale":"2024-06-29T15:57:35.353888+0000","last_undegraded":"2024-06-29T15:57:35.353888+0000","last_fullsized":"2024-06-29T15:57:35.353888+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:53.743438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354933+0000","last_change":"2024-06-29T15:57:22.301362+0000","last_active":"2024-06-29T15:57:35.354933+0000","last_peered":"2024-06-29T15:57:35.354933+0000","last_clean":"2024-06-29T15:57:35.354933+0000","last_became_active":"2024-06-29T15:57:22.300704+0000","last_became_peered":"2024-06-29T15:57:22.300704+0000","last_unstale":"2024-06-29T15:57:35.354933+0000","last_undegraded":"2024-06-29T15:57:35.354933+0000","last_fullsized":"2024-06-29T15:57:35.354933+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:24.179475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289379+0000","last_change":"2024-06-29T15:57:34.362173+0000","last_active":"2024-06-29T15:57:36.289379+0000","last_peered":"2024-06-29T15:57:36.289379+0000","last_clean":"2024-06-29T15:57:36.289379+0000","last_became_active":"2024-06-29T15:57:34.361877+0000","last_became_peered":"2024-06-29T15:57:34.361877+0000","last_unstale":"2024-06-29T15:57:36.289379+0000","last_undegraded":"2024-06-29T15:57:36.289379+0000","last_fullsized":"2024-06-29T15:57:36.289379+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:00.970948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355008+0000","last_change":"2024-06-29T15:57:22.307686+0000","last_active":"2024-06-29T15:57:35.355008+0000","last_peered":"2024-06-29T15:57:35.355008+0000","last_clean":"2024-06-29T15:57:35.355008+0000","last_became_active":"2024-06-29T15:57:22.307324+0000","last_became_peered":"2024-06-29T15:57:22.307324+0000","last_unstale":"2024-06-29T15:57:35.355008+0000","last_undegraded":"2024-06-29T15:57:35.355008+0000","last_fullsized":"2024-06-29T15:57:35.355008+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601934+0000","last_change":"2024-06-29T15:57:34.384184+0000","last_active":"2024-06-29T15:57:35.601934+0000","last_peered":"2024-06-29T15:57:35.601934+0000","last_clean":"2024-06-29T15:57:35.601934+0000","last_became_active":"2024-06-29T15:57:34.383889+0000","last_became_peered":"2024-06-29T15:57:34.383889+0000","last_unstale":"2024-06-29T15:57:35.601934+0000","last_undegraded":"2024-06-29T15:57:35.601934+0000","last_fullsized":"2024-06-29T15:57:35.601934+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:43:07.214490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374778+0000","last_change":"2024-06-29T15:57:22.307230+0000","last_active":"2024-06-29T15:57:35.374778+0000","last_peered":"2024-06-29T15:57:35.374778+0000","last_clean":"2024-06-29T15:57:35.374778+0000","last_became_active":"2024-06-29T15:57:22.306920+0000","last_became_peered":"2024-06-29T15:57:22.306920+0000","last_unstale":"2024-06-29T15:57:35.374778+0000","last_undegraded":"2024-06-29T15:57:35.374778+0000","last_fullsized":"2024-06-29T15:57:35.374778+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:52:22.598647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288916+0000","last_change":"2024-06-29T15:57:34.375690+0000","last_active":"2024-06-29T15:57:36.288916+0000","last_peered":"2024-06-29T15:57:36.288916+0000","last_clean":"2024-06-29T15:57:36.288916+0000","last_became_active":"2024-06-29T15:57:34.375364+0000","last_became_peered":"2024-06-29T15:57:34.375364+0000","last_unstale":"2024-06-29T15:57:36.288916+0000","last_undegraded":"2024-06-29T15:57:36.288916+0000","last_fullsized":"2024-06-29T15:57:36.288916+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:07:36.359415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601845+0000","last_change":"2024-06-29T15:57:22.290186+0000","last_active":"2024-06-29T15:57:35.601845+0000","last_peered":"2024-06-29T15:57:35.601845+0000","last_clean":"2024-06-29T15:57:35.601845+0000","last_became_active":"2024-06-29T15:57:22.289908+0000","last_became_peered":"2024-06-29T15:57:22.289908+0000","last_unstale":"2024-06-29T15:57:35.601845+0000","last_undegraded":"2024-06-29T15:57:35.601845+0000","last_fullsized":"2024-06-29T15:57:35.601845+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:39:18.272546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601312+0000","last_change":"2024-06-29T15:57:34.376561+0000","last_active":"2024-06-29T15:57:35.601312+0000","last_peered":"2024-06-29T15:57:35.601312+0000","last_clean":"2024-06-29T15:57:35.601312+0000","last_became_active":"2024-06-29T15:57:34.376238+0000","last_became_peered":"2024-06-29T15:57:34.376238+0000","last_unstale":"2024-06-29T15:57:35.601312+0000","last_undegraded":"2024-06-29T15:57:35.601312+0000","last_fullsized":"2024-06-29T15:57:35.601312+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:07:36.359415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289508+0000","last_change":"2024-06-29T15:57:22.306593+0000","last_active":"2024-06-29T15:57:36.289508+0000","last_peered":"2024-06-29T15:57:36.289508+0000","last_clean":"2024-06-29T15:57:36.289508+0000","last_became_active":"2024-06-29T15:57:22.296653+0000","last_became_peered":"2024-06-29T15:57:22.296653+0000","last_unstale":"2024-06-29T15:57:36.289508+0000","last_undegraded":"2024-06-29T15:57:36.289508+0000","last_fullsized":"2024-06-29T15:57:36.289508+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:41.738155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601138+0000","last_change":"2024-06-29T15:57:34.358432+0000","last_active":"2024-06-29T15:57:35.601138+0000","last_peered":"2024-06-29T15:57:35.601138+0000","last_clean":"2024-06-29T15:57:35.601138+0000","last_became_active":"2024-06-29T15:57:34.358108+0000","last_became_peered":"2024-06-29T15:57:34.358108+0000","last_unstale":"2024-06-29T15:57:35.601138+0000","last_undegraded":"2024-06-29T15:57:35.601138+0000","last_fullsized":"2024-06-29T15:57:35.601138+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:11:48.493958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289756+0000","last_change":"2024-06-29T15:57:22.289633+0000","last_active":"2024-06-29T15:57:36.289756+0000","last_peered":"2024-06-29T15:57:36.289756+0000","last_clean":"2024-06-29T15:57:36.289756+0000","last_became_active":"2024-06-29T15:57:22.289349+0000","last_became_peered":"2024-06-29T15:57:22.289349+0000","last_unstale":"2024-06-29T15:57:36.289756+0000","last_undegraded":"2024-06-29T15:57:36.289756+0000","last_fullsized":"2024-06-29T15:57:36.289756+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:51:28.873806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374483+0000","last_change":"2024-06-29T15:57:34.377123+0000","last_active":"2024-06-29T15:57:35.374483+0000","last_peered":"2024-06-29T15:57:35.374483+0000","last_clean":"2024-06-29T15:57:35.374483+0000","last_became_active":"2024-06-29T15:57:34.375757+0000","last_became_peered":"2024-06-29T15:57:34.375757+0000","last_unstale":"2024-06-29T15:57:35.374483+0000","last_undegraded":"2024-06-29T15:57:35.374483+0000","last_fullsized":"2024-06-29T15:57:35.374483+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:56:04.247579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289014+0000","last_change":"2024-06-29T15:57:22.300787+0000","last_active":"2024-06-29T15:57:36.289014+0000","last_peered":"2024-06-29T15:57:36.289014+0000","last_clean":"2024-06-29T15:57:36.289014+0000","last_became_active":"2024-06-29T15:57:22.300112+0000","last_became_peered":"2024-06-29T15:57:22.300112+0000","last_unstale":"2024-06-29T15:57:36.289014+0000","last_undegraded":"2024-06-29T15:57:36.289014+0000","last_fullsized":"2024-06-29T15:57:36.289014+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:28:30.249483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289648+0000","last_change":"2024-06-29T15:57:34.366979+0000","last_active":"2024-06-29T15:57:36.289648+0000","last_peered":"2024-06-29T15:57:36.289648+0000","last_clean":"2024-06-29T15:57:36.289648+0000","last_became_active":"2024-06-29T15:57:34.366649+0000","last_became_peered":"2024-06-29T15:57:34.366649+0000","last_unstale":"2024-06-29T15:57:36.289648+0000","last_undegraded":"2024-06-29T15:57:36.289648+0000","last_fullsized":"2024-06-29T15:57:36.289648+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:12:48.700345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288272+0000","last_change":"2024-06-29T15:57:22.290047+0000","last_active":"2024-06-29T15:57:36.288272+0000","last_peered":"2024-06-29T15:57:36.288272+0000","last_clean":"2024-06-29T15:57:36.288272+0000","last_became_active":"2024-06-29T15:57:22.289742+0000","last_became_peered":"2024-06-29T15:57:22.289742+0000","last_unstale":"2024-06-29T15:57:36.288272+0000","last_undegraded":"2024-06-29T15:57:36.288272+0000","last_fullsized":"2024-06-29T15:57:36.288272+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:03:46.271206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788933+0000","last_change":"2024-06-29T15:57:34.388260+0000","last_active":"2024-06-29T15:57:35.788933+0000","last_peered":"2024-06-29T15:57:35.788933+0000","last_clean":"2024-06-29T15:57:35.788933+0000","last_became_active":"2024-06-29T15:57:34.387932+0000","last_became_peered":"2024-06-29T15:57:34.387932+0000","last_unstale":"2024-06-29T15:57:35.788933+0000","last_undegraded":"2024-06-29T15:57:35.788933+0000","last_fullsized":"2024-06-29T15:57:35.788933+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:25:07.454033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288378+0000","last_change":"2024-06-29T15:57:22.302418+0000","last_active":"2024-06-29T15:57:36.288378+0000","last_peered":"2024-06-29T15:57:36.288378+0000","last_clean":"2024-06-29T15:57:36.288378+0000","last_became_active":"2024-06-29T15:57:22.302075+0000","last_became_peered":"2024-06-29T15:57:22.302075+0000","last_unstale":"2024-06-29T15:57:36.288378+0000","last_undegraded":"2024-06-29T15:57:36.288378+0000","last_fullsized":"2024-06-29T15:57:36.288378+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355283+0000","last_change":"2024-06-29T15:57:34.376642+0000","last_active":"2024-06-29T15:57:35.355283+0000","last_peered":"2024-06-29T15:57:35.355283+0000","last_clean":"2024-06-29T15:57:35.355283+0000","last_became_active":"2024-06-29T15:57:34.376340+0000","last_became_peered":"2024-06-29T15:57:34.376340+0000","last_unstale":"2024-06-29T15:57:35.355283+0000","last_undegraded":"2024-06-29T15:57:35.355283+0000","last_fullsized":"2024-06-29T15:57:35.355283+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:31:53.743438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288567+0000","last_change":"2024-06-29T15:57:22.303092+0000","last_active":"2024-06-29T15:57:36.288567+0000","last_peered":"2024-06-29T15:57:36.288567+0000","last_clean":"2024-06-29T15:57:36.288567+0000","last_became_active":"2024-06-29T15:57:22.302807+0000","last_became_peered":"2024-06-29T15:57:22.302807+0000","last_unstale":"2024-06-29T15:57:36.288567+0000","last_undegraded":"2024-06-29T15:57:36.288567+0000","last_fullsized":"2024-06-29T15:57:36.288567+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374933+0000","last_change":"2024-06-29T15:57:34.385814+0000","last_active":"2024-06-29T15:57:35.374933+0000","last_peered":"2024-06-29T15:57:35.374933+0000","last_clean":"2024-06-29T15:57:35.374933+0000","last_became_active":"2024-06-29T15:57:34.383437+0000","last_became_peered":"2024-06-29T15:57:34.383437+0000","last_unstale":"2024-06-29T15:57:35.374933+0000","last_undegraded":"2024-06-29T15:57:35.374933+0000","last_fullsized":"2024-06-29T15:57:35.374933+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:49:11.481404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788621+0000","last_change":"2024-06-29T15:57:22.295681+0000","last_active":"2024-06-29T15:57:35.788621+0000","last_peered":"2024-06-29T15:57:35.788621+0000","last_clean":"2024-06-29T15:57:35.788621+0000","last_became_active":"2024-06-29T15:57:22.295357+0000","last_became_peered":"2024-06-29T15:57:22.295357+0000","last_unstale":"2024-06-29T15:57:35.788621+0000","last_undegraded":"2024-06-29T15:57:35.788621+0000","last_fullsized":"2024-06-29T15:57:35.788621+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:26.983900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353811+0000","last_change":"2024-06-29T15:57:34.365020+0000","last_active":"2024-06-29T15:57:35.353811+0000","last_peered":"2024-06-29T15:57:35.353811+0000","last_clean":"2024-06-29T15:57:35.353811+0000","last_became_active":"2024-06-29T15:57:34.364639+0000","last_became_peered":"2024-06-29T15:57:34.364639+0000","last_unstale":"2024-06-29T15:57:35.353811+0000","last_undegraded":"2024-06-29T15:57:35.353811+0000","last_fullsized":"2024-06-29T15:57:35.353811+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:41:20.717589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788721+0000","last_change":"2024-06-29T15:57:22.298612+0000","last_active":"2024-06-29T15:57:35.788721+0000","last_peered":"2024-06-29T15:57:35.788721+0000","last_clean":"2024-06-29T15:57:35.788721+0000","last_became_active":"2024-06-29T15:57:22.298288+0000","last_became_peered":"2024-06-29T15:57:22.298288+0000","last_unstale":"2024-06-29T15:57:35.788721+0000","last_undegraded":"2024-06-29T15:57:35.788721+0000","last_fullsized":"2024-06-29T15:57:35.788721+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:22:03.754605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354774+0000","last_change":"2024-06-29T15:57:34.382813+0000","last_active":"2024-06-29T15:57:35.354774+0000","last_peered":"2024-06-29T15:57:35.354774+0000","last_clean":"2024-06-29T15:57:35.354774+0000","last_became_active":"2024-06-29T15:57:34.382505+0000","last_became_peered":"2024-06-29T15:57:34.382505+0000","last_unstale":"2024-06-29T15:57:35.354774+0000","last_undegraded":"2024-06-29T15:57:35.354774+0000","last_fullsized":"2024-06-29T15:57:35.354774+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:41:20.717589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351905+0000","last_change":"2024-06-29T15:57:22.298781+0000","last_active":"2024-06-29T15:57:35.351905+0000","last_peered":"2024-06-29T15:57:35.351905+0000","last_clean":"2024-06-29T15:57:35.351905+0000","last_became_active":"2024-06-29T15:57:22.298300+0000","last_became_peered":"2024-06-29T15:57:22.298300+0000","last_unstale":"2024-06-29T15:57:35.351905+0000","last_undegraded":"2024-06-29T15:57:35.351905+0000","last_fullsized":"2024-06-29T15:57:35.351905+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355496+0000","last_change":"2024-06-29T15:57:34.367975+0000","last_active":"2024-06-29T15:57:35.355496+0000","last_peered":"2024-06-29T15:57:35.355496+0000","last_clean":"2024-06-29T15:57:35.355496+0000","last_became_active":"2024-06-29T15:57:34.367645+0000","last_became_peered":"2024-06-29T15:57:34.367645+0000","last_unstale":"2024-06-29T15:57:35.355496+0000","last_undegraded":"2024-06-29T15:57:35.355496+0000","last_fullsized":"2024-06-29T15:57:35.355496+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:10.459536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.289748+0000","last_change":"2024-06-29T15:57:22.296441+0000","last_active":"2024-06-29T15:57:36.289748+0000","last_peered":"2024-06-29T15:57:36.289748+0000","last_clean":"2024-06-29T15:57:36.289748+0000","last_became_active":"2024-06-29T15:57:22.296146+0000","last_became_peered":"2024-06-29T15:57:22.296146+0000","last_unstale":"2024-06-29T15:57:36.289748+0000","last_undegraded":"2024-06-29T15:57:36.289748+0000","last_fullsized":"2024-06-29T15:57:36.289748+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:19:54.360521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288083+0000","last_change":"2024-06-29T15:57:34.377167+0000","last_active":"2024-06-29T15:57:36.288083+0000","last_peered":"2024-06-29T15:57:36.288083+0000","last_clean":"2024-06-29T15:57:36.288083+0000","last_became_active":"2024-06-29T15:57:34.376880+0000","last_became_peered":"2024-06-29T15:57:34.376880+0000","last_unstale":"2024-06-29T15:57:36.288083+0000","last_undegraded":"2024-06-29T15:57:36.288083+0000","last_fullsized":"2024-06-29T15:57:36.288083+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:45:27.007653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.354206+0000","last_change":"2024-06-29T15:57:22.291096+0000","last_active":"2024-06-29T15:57:35.354206+0000","last_peered":"2024-06-29T15:57:35.354206+0000","last_clean":"2024-06-29T15:57:35.354206+0000","last_became_active":"2024-06-29T15:57:22.290428+0000","last_became_peered":"2024-06-29T15:57:22.290428+0000","last_unstale":"2024-06-29T15:57:35.354206+0000","last_undegraded":"2024-06-29T15:57:35.354206+0000","last_fullsized":"2024-06-29T15:57:35.354206+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:24:20.813793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.355629+0000","last_change":"2024-06-29T15:57:34.379759+0000","last_active":"2024-06-29T15:57:35.355629+0000","last_peered":"2024-06-29T15:57:35.355629+0000","last_clean":"2024-06-29T15:57:35.355629+0000","last_became_active":"2024-06-29T15:57:34.378980+0000","last_became_peered":"2024-06-29T15:57:34.378980+0000","last_unstale":"2024-06-29T15:57:35.355629+0000","last_undegraded":"2024-06-29T15:57:35.355629+0000","last_fullsized":"2024-06-29T15:57:35.355629+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:44:48.939506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601187+0000","last_change":"2024-06-29T15:57:22.297617+0000","last_active":"2024-06-29T15:57:35.601187+0000","last_peered":"2024-06-29T15:57:35.601187+0000","last_clean":"2024-06-29T15:57:35.601187+0000","last_became_active":"2024-06-29T15:57:22.297256+0000","last_became_peered":"2024-06-29T15:57:22.297256+0000","last_unstale":"2024-06-29T15:57:35.601187+0000","last_undegraded":"2024-06-29T15:57:35.601187+0000","last_fullsized":"2024-06-29T15:57:35.601187+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:54.470323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.375367+0000","last_change":"2024-06-29T15:57:34.385881+0000","last_active":"2024-06-29T15:57:35.375367+0000","last_peered":"2024-06-29T15:57:35.375367+0000","last_clean":"2024-06-29T15:57:35.375367+0000","last_became_active":"2024-06-29T15:57:34.385239+0000","last_became_peered":"2024-06-29T15:57:34.385239+0000","last_unstale":"2024-06-29T15:57:35.375367+0000","last_undegraded":"2024-06-29T15:57:35.375367+0000","last_fullsized":"2024-06-29T15:57:35.375367+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:31:51.181024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288268+0000","last_change":"2024-06-29T15:57:22.304052+0000","last_active":"2024-06-29T15:57:36.288268+0000","last_peered":"2024-06-29T15:57:36.288268+0000","last_clean":"2024-06-29T15:57:36.288268+0000","last_became_active":"2024-06-29T15:57:22.303598+0000","last_became_peered":"2024-06-29T15:57:22.303598+0000","last_unstale":"2024-06-29T15:57:36.288268+0000","last_undegraded":"2024-06-29T15:57:36.288268+0000","last_fullsized":"2024-06-29T15:57:36.288268+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:19:30.611104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601663+0000","last_change":"2024-06-29T15:57:34.372683+0000","last_active":"2024-06-29T15:57:35.601663+0000","last_peered":"2024-06-29T15:57:35.601663+0000","last_clean":"2024-06-29T15:57:35.601663+0000","last_became_active":"2024-06-29T15:57:34.372354+0000","last_became_peered":"2024-06-29T15:57:34.372354+0000","last_unstale":"2024-06-29T15:57:35.601663+0000","last_undegraded":"2024-06-29T15:57:35.601663+0000","last_fullsized":"2024-06-29T15:57:35.601663+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:39:12.203421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601324+0000","last_change":"2024-06-29T15:57:22.288041+0000","last_active":"2024-06-29T15:57:35.601324+0000","last_peered":"2024-06-29T15:57:35.601324+0000","last_clean":"2024-06-29T15:57:35.601324+0000","last_became_active":"2024-06-29T15:57:22.287729+0000","last_became_peered":"2024-06-29T15:57:22.287729+0000","last_unstale":"2024-06-29T15:57:35.601324+0000","last_undegraded":"2024-06-29T15:57:35.601324+0000","last_fullsized":"2024-06-29T15:57:35.601324+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:27:50.320822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.788732+0000","last_change":"2024-06-29T15:57:34.377129+0000","last_active":"2024-06-29T15:57:35.788732+0000","last_peered":"2024-06-29T15:57:35.788732+0000","last_clean":"2024-06-29T15:57:35.788732+0000","last_became_active":"2024-06-29T15:57:34.376788+0000","last_became_peered":"2024-06-29T15:57:34.376788+0000","last_unstale":"2024-06-29T15:57:35.788732+0000","last_undegraded":"2024-06-29T15:57:35.788732+0000","last_fullsized":"2024-06-29T15:57:35.788732+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:33.372620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601684+0000","last_change":"2024-06-29T15:57:22.290909+0000","last_active":"2024-06-29T15:57:35.601684+0000","last_peered":"2024-06-29T15:57:35.601684+0000","last_clean":"2024-06-29T15:57:35.601684+0000","last_became_active":"2024-06-29T15:57:22.290643+0000","last_became_peered":"2024-06-29T15:57:22.290643+0000","last_unstale":"2024-06-29T15:57:35.601684+0000","last_undegraded":"2024-06-29T15:57:35.601684+0000","last_fullsized":"2024-06-29T15:57:35.601684+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288253+0000","last_change":"2024-06-29T15:57:34.371019+0000","last_active":"2024-06-29T15:57:36.288253+0000","last_peered":"2024-06-29T15:57:36.288253+0000","last_clean":"2024-06-29T15:57:36.288253+0000","last_became_active":"2024-06-29T15:57:34.370696+0000","last_became_peered":"2024-06-29T15:57:34.370696+0000","last_unstale":"2024-06-29T15:57:36.288253+0000","last_undegraded":"2024-06-29T15:57:36.288253+0000","last_fullsized":"2024-06-29T15:57:36.288253+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:27:33.473360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.374763+0000","last_change":"2024-06-29T15:57:22.298374+0000","last_active":"2024-06-29T15:57:35.374763+0000","last_peered":"2024-06-29T15:57:35.374763+0000","last_clean":"2024-06-29T15:57:35.374763+0000","last_became_active":"2024-06-29T15:57:22.297962+0000","last_became_peered":"2024-06-29T15:57:22.297962+0000","last_unstale":"2024-06-29T15:57:35.374763+0000","last_undegraded":"2024-06-29T15:57:35.374763+0000","last_fullsized":"2024-06-29T15:57:35.374763+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:26:13.834244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.601550+0000","last_change":"2024-06-29T15:57:34.358389+0000","last_active":"2024-06-29T15:57:35.601550+0000","last_peered":"2024-06-29T15:57:35.601550+0000","last_clean":"2024-06-29T15:57:35.601550+0000","last_became_active":"2024-06-29T15:57:34.358053+0000","last_became_peered":"2024-06-29T15:57:34.358053+0000","last_unstale":"2024-06-29T15:57:35.601550+0000","last_undegraded":"2024-06-29T15:57:35.601550+0000","last_fullsized":"2024-06-29T15:57:35.601550+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:48.165689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351802+0000","last_change":"2024-06-29T15:57:22.300745+0000","last_active":"2024-06-29T15:57:35.351802+0000","last_peered":"2024-06-29T15:57:35.351802+0000","last_clean":"2024-06-29T15:57:35.351802+0000","last_became_active":"2024-06-29T15:57:22.300414+0000","last_became_peered":"2024-06-29T15:57:22.300414+0000","last_unstale":"2024-06-29T15:57:35.351802+0000","last_undegraded":"2024-06-29T15:57:35.351802+0000","last_fullsized":"2024-06-29T15:57:35.351802+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:26:35.093064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.351290+0000","last_change":"2024-06-29T15:57:34.368006+0000","last_active":"2024-06-29T15:57:35.351290+0000","last_peered":"2024-06-29T15:57:35.351290+0000","last_clean":"2024-06-29T15:57:35.351290+0000","last_became_active":"2024-06-29T15:57:34.367641+0000","last_became_peered":"2024-06-29T15:57:34.367641+0000","last_unstale":"2024-06-29T15:57:35.351290+0000","last_undegraded":"2024-06-29T15:57:35.351290+0000","last_fullsized":"2024-06-29T15:57:35.351290+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:40:27.964272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.353371+0000","last_change":"2024-06-29T15:57:22.297971+0000","last_active":"2024-06-29T15:57:35.353371+0000","last_peered":"2024-06-29T15:57:35.353371+0000","last_clean":"2024-06-29T15:57:35.353371+0000","last_became_active":"2024-06-29T15:57:22.297270+0000","last_became_peered":"2024-06-29T15:57:22.297270+0000","last_unstale":"2024-06-29T15:57:35.353371+0000","last_undegraded":"2024-06-29T15:57:35.353371+0000","last_fullsized":"2024-06-29T15:57:35.353371+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:34.610463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":12,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:35.787844+0000","last_change":"2024-06-29T15:57:34.364332+0000","last_active":"2024-06-29T15:57:35.787844+0000","last_peered":"2024-06-29T15:57:35.787844+0000","last_clean":"2024-06-29T15:57:35.787844+0000","last_became_active":"2024-06-29T15:57:34.363826+0000","last_became_peered":"2024-06-29T15:57:34.363826+0000","last_unstale":"2024-06-29T15:57:35.787844+0000","last_undegraded":"2024-06-29T15:57:35.787844+0000","last_fullsized":"2024-06-29T15:57:35.787844+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:33.339954+0000","last_clean_scrub_stamp":"2024-06-29T15:57:33.339954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:46:16.184075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":20,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-06-29T15:57:36.288933+0000","last_change":"2024-06-29T15:57:22.301019+0000","last_active":"2024-06-29T15:57:36.288933+0000","last_peered":"2024-06-29T15:57:36.288933+0000","last_clean":"2024-06-29T15:57:36.288933+0000","last_became_active":"2024-06-29T15:57:22.300506+0000","last_became_peered":"2024-06-29T15:57:22.300506+0000","last_unstale":"2024-06-29T15:57:36.288933+0000","last_undegraded":"2024-06-29T15:57:36.288933+0000","last_fullsized":"2024-06-29T15:57:36.288933+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":211,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:57:21.259342+0000","last_clean_scrub_stamp":"2024-06-29T15:57:21.259342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:16.509953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1326,"num_read_kb":3751,"num_write":353,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575988,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52492,"kb_used_data":3256,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319348,"statfs":{"total":96636764160,"available":96583012352,"internally_reserved":0,"allocated":3334144,"data_stored":1929413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575989,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42180,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329660,"statfs":{"total":96636764160,"available":96593571840,"internally_reserved":0,"allocated":2736128,"data_stored":1336758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575987,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42172,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329668,"statfs":{"total":96636764160,"available":96593580032,"internally_reserved":0,"allocated":2727936,"data_stored":1336702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575988,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60232,"kb_used_data":2684,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311608,"statfs":{"total":96636764160,"available":96575086592,"internally_reserved":0,"allocated":2748416,"data_stored":1339661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575988,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47756,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324084,"statfs":{"total":96636764160,"available":96587862016,"internally_reserved":0,"allocated":2752512,"data_stored":1337984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575988,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47756,"kb_used_data":2680,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324084,"statfs":{"total":96636764160,"available":96587862016,"internally_reserved":0,"allocated":2744320,"data_stored":1337788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575987,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60228,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311612,"statfs":{"total":96636764160,"available":96575090688,"internally_reserved":0,"allocated":2736128,"data_stored":1336266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575988,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60804,"kb_used_data":3252,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311036,"statfs":{"total":96636764160,"available":96574500864,"internally_reserved":0,"allocated":3330048,"data_stored":1927250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:57:43.248 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:57:43.248 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:43.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.393+0000 7ff566eef640 1 -- 172.21.15.81:0/2922405653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5600d0230 msgr2=0x7ff5600d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:43.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.393+0000 7ff566eef640 1 --2- 172.21.15.81:0/2922405653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5600d0230 0x7ff5600d0640 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7ff5600c5590 tx=0x7ff55002a4d0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.393+0000 7ff566eef640 1 -- 172.21.15.81:0/2922405653 shutdown_connections 2024-06-29T15:57:43.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.393+0000 7ff566eef640 1 --2- 172.21.15.81:0/2922405653 >> v2:172.21.15.167:3300/0 conn(0x7ff5600b4f40 0x7ff5600b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.393+0000 7ff566eef640 1 --2- 172.21.15.81:0/2922405653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5600d0230 0x7ff5600d0640 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.393+0000 7ff566eef640 1 -- 172.21.15.81:0/2922405653 >> 172.21.15.81:0/2922405653 conn(0x7ff5600b3680 msgr2=0x7ff5600b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.393+0000 7ff566eef640 1 -- 172.21.15.81:0/2922405653 shutdown_connections 2024-06-29T15:57:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.394+0000 7ff566eef640 1 -- 172.21.15.81:0/2922405653 wait complete. 2024-06-29T15:57:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.395+0000 7ff566eef640 1 Processor -- start 2024-06-29T15:57:43.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.395+0000 7ff566eef640 1 -- start start 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff566eef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5600b3200 0x7ff56014b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff566eef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5600b4f40 0x7ff5600ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff566eef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5600d0230 0x7ff5601f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff566eef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff5600b5c20 con 0x7ff5600b4f40 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff566eef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff5600b5aa0 con 0x7ff5600d0230 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff566eef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff5600b5da0 con 0x7ff5600b3200 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff557fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5600b4f40 0x7ff5600ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff557fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5600b4f40 0x7ff5600ae960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54210/0 (socket says 172.21.15.81:54210) 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff557fff640 1 -- 172.21.15.81:0/1325310436 learned_addr learned my addr 172.21.15.81:0/1325310436 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff564c8b640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5600b3200 0x7ff56014b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff56548c640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5600d0230 0x7ff5601f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff557fff640 1 -- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5600b3200 msgr2=0x7ff56014b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:43.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff557fff640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5600b3200 0x7ff56014b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff557fff640 1 -- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5600d0230 msgr2=0x7ff5601f6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:43.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff557fff640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5600d0230 0x7ff5601f6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff557fff640 1 -- 172.21.15.81:0/1325310436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5600b2000 con 0x7ff5600b4f40 2024-06-29T15:57:43.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.397+0000 7ff557fff640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5600b4f40 0x7ff5600ae960 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7ff550034040 tx=0x7ff5500039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:43.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.398+0000 7ff5557fa640 1 -- 172.21.15.81:0/1325310436 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55003e030 con 0x7ff5600b4f40 2024-06-29T15:57:43.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.398+0000 7ff5557fa640 1 -- 172.21.15.81:0/1325310436 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5500047a0 con 0x7ff5600b4f40 2024-06-29T15:57:43.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.398+0000 7ff566eef640 1 -- 172.21.15.81:0/1325310436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5600b22e0 con 0x7ff5600b4f40 2024-06-29T15:57:43.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.398+0000 7ff5557fa640 1 -- 172.21.15.81:0/1325310436 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff550035040 con 0x7ff5600b4f40 2024-06-29T15:57:43.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.398+0000 7ff566eef640 1 -- 172.21.15.81:0/1325310436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5601f66f0 con 0x7ff5600b4f40 2024-06-29T15:57:43.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.399+0000 7ff566eef640 1 -- 172.21.15.81:0/1325310436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5601f6510 con 0x7ff5600b4f40 2024-06-29T15:57:43.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.399+0000 7ff5557fa640 1 -- 172.21.15.81:0/1325310436 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7ff550041030 con 0x7ff5600b4f40 2024-06-29T15:57:43.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.400+0000 7ff5557fa640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff54c076420 0x7ff54c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:43.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.400+0000 7ff5557fa640 1 -- 172.21.15.81:0/1325310436 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7ff5500b85e0 con 0x7ff5600b4f40 2024-06-29T15:57:43.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.400+0000 7ff564c8b640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff54c076420 0x7ff54c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:43.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.401+0000 7ff564c8b640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff54c076420 0x7ff54c078930 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff55c006fd0 tx=0x7ff55c009060 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:43.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.405+0000 7ff5557fa640 1 -- 172.21.15.81:0/1325310436 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff5601f6510 con 0x7ff5600b4f40 2024-06-29T15:57:43.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.621+0000 7ff566eef640 1 -- 172.21.15.81:0/1325310436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5600b1d90 con 0x7ff5600b4f40 2024-06-29T15:57:43.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.622+0000 7ff5557fa640 1 -- 172.21.15.81:0/1325310436 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+6565 (secure 0 0 0) 0x7ff5600d0b80 con 0x7ff5600b4f40 2024-06-29T15:57:43.622 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:43.622 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":118,"btime":"2024-06-29T15:57:37:378022+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.637042+0000","modified":"2024-06-29T15:57:24.656503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11226":{"gid":11226,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/739691411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":739691411},{"type":"v1","addr":"172.21.15.81:6833","nonce":739691411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11226,"qdb_cluster":[11226]},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:33.727815+0000","modified":"2024-06-29T15:57:36.736776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10114":{"gid":10114,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10114,"qdb_cluster":[10114]},"id":14}]} 2024-06-29T15:57:43.623 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 118 2024-06-29T15:57:43.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 -- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff54c076420 msgr2=0x7ff54c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:43.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff54c076420 0x7ff54c078930 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff55c006fd0 tx=0x7ff55c009060 comp rx=0 tx=0).stop 2024-06-29T15:57:43.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 -- 172.21.15.81:0/1325310436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5600b4f40 msgr2=0x7ff5600ae960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5600b4f40 0x7ff5600ae960 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7ff550034040 tx=0x7ff5500039e0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 -- 172.21.15.81:0/1325310436 shutdown_connections 2024-06-29T15:57:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff54c076420 0x7ff54c078930 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff5600d0230 0x7ff5601f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5600b4f40 0x7ff5600ae960 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 --2- 172.21.15.81:0/1325310436 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff5600b3200 0x7ff56014b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 -- 172.21.15.81:0/1325310436 >> 172.21.15.81:0/1325310436 conn(0x7ff5600b3680 msgr2=0x7ff5600cb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 -- 172.21.15.81:0/1325310436 shutdown_connections 2024-06-29T15:57:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:43.624+0000 7ff566eef640 1 -- 172.21.15.81:0/1325310436 wait complete. 2024-06-29T15:57:44.038 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 118, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:57:33.727815+0000', 'modified': '2024-06-29T15:57:36.736776+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10114}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10114': {'gid': 10114, 'name': 'd', 'rank': 0, 'incarnation': 114, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.167:6837/1664603190', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6835', 'nonce': 1664603190}, {'type': 'v1', 'addr': '172.21.15.167:6837', 'nonce': 1664603190}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10114, 'qdb_cluster': [10114]} 2024-06-29T15:57:44.038 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:57:44.038 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-06-29T15:57:44.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.154+0000 7f5f2e438640 1 -- 172.21.15.81:0/2843998864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2815c030 msgr2=0x7f5f2806c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.154+0000 7f5f2e438640 1 --2- 172.21.15.81:0/2843998864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2815c030 0x7f5f2806c2f0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c005fd0 tx=0x7f5f1c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.154+0000 7f5f2e438640 1 -- 172.21.15.81:0/2843998864 shutdown_connections 2024-06-29T15:57:44.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.154+0000 7f5f2e438640 1 --2- 172.21.15.81:0/2843998864 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f2815c440 0x7f5f2806c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.154+0000 7f5f2e438640 1 --2- 172.21.15.81:0/2843998864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2815c030 0x7f5f2806c2f0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.154+0000 7f5f2e438640 1 -- 172.21.15.81:0/2843998864 >> 172.21.15.81:0/2843998864 conn(0x7f5f280d0930 msgr2=0x7f5f280d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.154+0000 7f5f2e438640 1 -- 172.21.15.81:0/2843998864 shutdown_connections 2024-06-29T15:57:44.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.154+0000 7f5f2e438640 1 -- 172.21.15.81:0/2843998864 wait complete. 2024-06-29T15:57:44.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.156+0000 7f5f2e438640 1 Processor -- start 2024-06-29T15:57:44.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.156+0000 7f5f2e438640 1 -- start start 2024-06-29T15:57:44.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.157+0000 7f5f2e438640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f2815c030 0x7f5f280b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.157+0000 7f5f2e438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2815c440 0x7f5f280b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.157+0000 7f5f2e438640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f2815d750 0x7f5f281566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.157+0000 7f5f2e438640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f280cfda0 con 0x7f5f2815c440 2024-06-29T15:57:44.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.157+0000 7f5f2e438640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5f280cfc20 con 0x7f5f2815d750 2024-06-29T15:57:44.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.157+0000 7f5f2e438640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5f280cff20 con 0x7f5f2815c030 2024-06-29T15:57:44.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.157+0000 7f5f277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2815c440 0x7f5f280b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.157+0000 7f5f277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2815c440 0x7f5f280b4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54234/0 (socket says 172.21.15.81:54234) 2024-06-29T15:57:44.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.157+0000 7f5f277fe640 1 -- 172.21.15.81:0/1477925139 learned_addr learned my addr 172.21.15.81:0/1477925139 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:44.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f27fff640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f2815c030 0x7f5f280b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f2c9d5640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f2815d750 0x7f5f281566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f277fe640 1 -- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f2815c030 msgr2=0x7f5f280b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f277fe640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f2815c030 0x7f5f280b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f277fe640 1 -- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f2815d750 msgr2=0x7f5f281566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f277fe640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f2815d750 0x7f5f281566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f277fe640 1 -- 172.21.15.81:0/1477925139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f280cb490 con 0x7f5f2815c440 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f277fe640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2815c440 0x7f5f280b4990 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c034040 tx=0x7f5f1c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f24ff9640 1 -- 172.21.15.81:0/1477925139 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c03e030 con 0x7f5f2815c440 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f24ff9640 1 -- 172.21.15.81:0/1477925139 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f1c003ca0 con 0x7f5f2815c440 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f24ff9640 1 -- 172.21.15.81:0/1477925139 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c035040 con 0x7f5f2815c440 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f2e438640 1 -- 172.21.15.81:0/1477925139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f280cb750 con 0x7f5f2815c440 2024-06-29T15:57:44.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.158+0000 7f5f2e438640 1 -- 172.21.15.81:0/1477925139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f280cbd10 con 0x7f5f2815c440 2024-06-29T15:57:44.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.160+0000 7f5f24ff9640 1 -- 172.21.15.81:0/1477925139 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7f5f1c041030 con 0x7f5f2815c440 2024-06-29T15:57:44.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.161+0000 7f5f24ff9640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5f140763f0 0x7f5f14078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.161+0000 7f5f24ff9640 1 -- 172.21.15.81:0/1477925139 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7f5f1c0b8410 con 0x7f5f2815c440 2024-06-29T15:57:44.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.161+0000 7f5f27fff640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5f140763f0 0x7f5f14078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.162+0000 7f5f353c7740 1 Processor -- start 2024-06-29T15:57:44.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.162+0000 7f5f353c7740 1 -- start start 2024-06-29T15:57:44.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.162+0000 7f5f27fff640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5f140763f0 0x7f5f14078900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f18007fd0 tx=0x7f5f18009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d048e270 0x5649d087a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 0x5649d085e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f353c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5649d07f6bc0 con 0x5649d046d590 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f353c7740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x5649d07a5630 con 0x5649d04653b0 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f353c7740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x5649d0794bd0 con 0x5649d048e270 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f2c9d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f27fff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d048e270 0x5649d087a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f2c9d5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54250/0 (socket says 172.21.15.81:54250) 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f2c9d5640 1 -- 172.21.15.81:0/1394347658 learned_addr learned my addr 172.21.15.81:0/1394347658 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f277fe640 1 --2- 172.21.15.81:0/1394347658 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 0x5649d085e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f2c9d5640 1 -- 172.21.15.81:0/1394347658 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d048e270 msgr2=0x5649d087a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f2c9d5640 1 --2- 172.21.15.81:0/1394347658 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d048e270 0x5649d087a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f2c9d5640 1 -- 172.21.15.81:0/1394347658 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 msgr2=0x5649d085e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:44.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f2c9d5640 1 --2- 172.21.15.81:0/1394347658 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 0x5649d085e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f2c9d5640 1 -- 172.21.15.81:0/1394347658 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649d0791ca0 con 0x5649d046d590 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.163+0000 7f5f2c9d5640 1 --2- 172.21.15.81:0/1394347658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e8e0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f5f20008db0 tx=0x7f5f2000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfcb311eeae1e1e2 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f0dffb640 1 -- 172.21.15.81:0/1394347658 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f20014070 con 0x5649d046d590 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f0dffb640 1 -- 172.21.15.81:0/1394347658 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f20019750 con 0x5649d046d590 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f0dffb640 1 -- 172.21.15.81:0/1394347658 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f20019b00 con 0x5649d046d590 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f353c7740 1 -- 172.21.15.81:0/1394347658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 msgr2=0x5649d085e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f353c7740 1 --2- 172.21.15.81:0/1394347658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e8e0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f5f20008db0 tx=0x7f5f2000e720 comp rx=0 tx=0).stop 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f353c7740 1 -- 172.21.15.81:0/1394347658 shutdown_connections 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f353c7740 1 --2- 172.21.15.81:0/1394347658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e8e0 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f353c7740 1 --2- 172.21.15.81:0/1394347658 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 0x5649d085e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f353c7740 1 --2- 172.21.15.81:0/1394347658 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d048e270 0x5649d087a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f353c7740 1 -- 172.21.15.81:0/1394347658 >> 172.21.15.81:0/1394347658 conn(0x5649d04823c0 msgr2=0x5649d0831f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f353c7740 1 -- 172.21.15.81:0/1394347658 shutdown_connections 2024-06-29T15:57:44.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.164+0000 7f5f353c7740 1 -- 172.21.15.81:0/1394347658 wait complete. 2024-06-29T15:57:44.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.165+0000 7f5f353c7740 1 -- start start 2024-06-29T15:57:44.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.166+0000 7f5f353c7740 1 Processor -- start 2024-06-29T15:57:44.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 0x5649d0870450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d084c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d048e270 0x5649d0871d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f353c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5649d0732ae0 con 0x5649d046d590 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f353c7740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x5649d0798c10 con 0x5649d04653b0 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f353c7740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x5649d0508620 con 0x5649d048e270 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f353c7740 10 client.0 _ll_register_callbacks cb 0x5649d073f480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d084c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d084c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54256/0 (socket says 172.21.15.81:54256) 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f277fe640 1 -- 172.21.15.81:0/3954264869 learned_addr learned my addr 172.21.15.81:0/3954264869 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f2c9d5640 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d048e270 0x5649d0871d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f27fff640 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 0x5649d0870450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f277fe640 1 -- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d048e270 msgr2=0x5649d0871d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:44.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f277fe640 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d048e270 0x5649d0871d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f277fe640 1 -- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 msgr2=0x5649d0870450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f277fe640 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 0x5649d0870450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f277fe640 1 -- 172.21.15.81:0/3954264869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649d04f9850 con 0x5649d046d590 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f27fff640 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 0x5649d0870450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f277fe640 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d084c110 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c07d670 tx=0x7f5f1c0809d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4913d499c587526d server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.167+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3954264869 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c048040 con 0x5649d046d590 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3954264869 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7f5f1c0899c0 con 0x5649d046d590 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f353c7740 1 -- 172.21.15.81:0/3954264869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f20019750 con 0x5649d046d590 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3954264869 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7f5f1c081780 con 0x5649d046d590 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f353c7740 10 client.11550 fetch_fsmap learned FSMap version 118 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f353c7740 1 -- 172.21.15.81:0/3954264869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) -- 0x5649d05706b0 con 0x5649d046d590 2024-06-29T15:57:44.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3954264869 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f20019750 con 0x5649d046d590 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3954264869 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 118) ==== 4149+0+0 (secure 0 0 0) 0x7f5f1c0802e0 con 0x5649d046d590 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f353c7740 10 client.11550 fetch_fsmap finished waiting for FSMap version 118 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f353c7740 10 client.11550 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f353c7740 20 client.11550 populate_metadata read hostname 'smithi081' 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:6835/1664603190,v1:172.21.15.167:6837/1664603190] conn(0x5649d044ce40 0x5649d085c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f353c7740 4 client.11550 mds_command: new command op to 10114 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f353c7740 1 -- 172.21.15.81:0/3954264869 --> [v2:172.21.15.167:6835/1664603190,v1:172.21.15.167:6837/1664603190] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649d07a7580 con 0x5649d044ce40 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.168+0000 7f5f27fff640 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:6835/1664603190,v1:172.21.15.167:6837/1664603190] conn(0x5649d044ce40 0x5649d085c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.169+0000 7f5f06ffd640 10 client.11550 ms_handle_connect on v2:172.21.15.167:6835/1664603190 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.169+0000 7f5f27fff640 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:6835/1664603190,v1:172.21.15.167:6837/1664603190] conn(0x5649d044ce40 0x5649d085c530 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2bc562b0a8badf5 server_cookie=cd0472cf8a0cc144 in_seq=0 out_seq=0 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.169+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3954264869 <== mds.0 v2:172.21.15.167:6835/1664603190 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5649d07a7580 con 0x5649d044ce40 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.169+0000 7f5f06ffd640 10 client.11550 handle_command_reply: tid=0 2024-06-29T15:57:44.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 client.11550 shutdown 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 -- 172.21.15.81:0/3954264869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 msgr2=0x5649d084c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d084c110 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c07d670 tx=0x7f5f1c0809d0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 -- 172.21.15.81:0/3954264869 shutdown_connections 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:6835/1664603190,v1:172.21.15.167:6837/1664603190] conn(0x5649d044ce40 0x5649d085c530 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d048e270 0x5649d0871d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f06ffd640 0 client.11550 ms_handle_reset on v2:172.21.15.167:6835/1664603190 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d084c110 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3954264869 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d04653b0 0x5649d0870450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 -- 172.21.15.81:0/3954264869 >> 172.21.15.81:0/3954264869 conn(0x5649d04823c0 msgr2=0x5649d085ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 -- 172.21.15.81:0/3954264869 shutdown_connections 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 1 -- 172.21.15.81:0/3954264869 wait complete. 2024-06-29T15:57:44.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.170+0000 7f5f353c7740 20 client.11550 trim_cache size 0 max 16384 2024-06-29T15:57:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.189+0000 7f5f353c7740 1 Processor -- start 2024-06-29T15:57:44.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.189+0000 7f5f353c7740 1 -- start start 2024-06-29T15:57:44.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 0x5649d085e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 0x5649d0854960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f353c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f28204730 con 0x5649d046d590 2024-06-29T15:57:44.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f353c7740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5f281575a0 con 0x5649d048e270 2024-06-29T15:57:44.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f353c7740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5f282027c0 con 0x5649d04653b0 2024-06-29T15:57:44.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54260/0 (socket says 172.21.15.81:54260) 2024-06-29T15:57:44.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f277fe640 1 -- 172.21.15.81:0/731010480 learned_addr learned my addr 172.21.15.81:0/731010480 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f2c9d5640 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 0x5649d0854960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f27fff640 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 0x5649d085e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f277fe640 1 -- 172.21.15.81:0/731010480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 msgr2=0x5649d085e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f277fe640 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 0x5649d085e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f277fe640 1 -- 172.21.15.81:0/731010480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 msgr2=0x5649d0854960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f277fe640 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 0x5649d0854960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f277fe640 1 -- 172.21.15.81:0/731010480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649d08315f0 con 0x5649d046d590 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f27fff640 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 0x5649d085e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.190+0000 7f5f2c9d5640 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 0x5649d0854960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f277fe640 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e970 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c07d100 tx=0x7f5f1c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82fc4cdcf8769bb server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f0d7fa640 1 -- 172.21.15.81:0/731010480 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c048040 con 0x5649d046d590 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f0d7fa640 1 -- 172.21.15.81:0/731010480 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f1c081080 con 0x5649d046d590 2024-06-29T15:57:44.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f0d7fa640 1 -- 172.21.15.81:0/731010480 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c085430 con 0x5649d046d590 2024-06-29T15:57:44.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f353c7740 1 -- 172.21.15.81:0/731010480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 msgr2=0x5649d085e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f353c7740 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e970 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c07d100 tx=0x7f5f1c005fd0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f353c7740 1 -- 172.21.15.81:0/731010480 shutdown_connections 2024-06-29T15:57:44.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f353c7740 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 0x5649d0854960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f353c7740 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d085e970 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f353c7740 1 --2- 172.21.15.81:0/731010480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 0x5649d085e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f353c7740 1 -- 172.21.15.81:0/731010480 >> 172.21.15.81:0/731010480 conn(0x5649d04823c0 msgr2=0x5649d0873840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f353c7740 1 -- 172.21.15.81:0/731010480 shutdown_connections 2024-06-29T15:57:44.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.191+0000 7f5f353c7740 1 -- 172.21.15.81:0/731010480 wait complete. 2024-06-29T15:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.192+0000 7f5f353c7740 1 -- start start 2024-06-29T15:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.192+0000 7f5f353c7740 1 Processor -- start 2024-06-29T15:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 0x5649d08f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d0858fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f353c7740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 0x5649d0859510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f353c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5649d08722b0 con 0x5649d046d590 2024-06-29T15:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f353c7740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x5649d0871580 con 0x5649d048e270 2024-06-29T15:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f353c7740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x5649d0872b30 con 0x5649d04653b0 2024-06-29T15:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f353c7740 10 client.0 _ll_register_callbacks cb 0x7f5f1c07d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d0858fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d0858fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54272/0 (socket says 172.21.15.81:54272) 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f277fe640 1 -- 172.21.15.81:0/3200436302 learned_addr learned my addr 172.21.15.81:0/3200436302 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f2c9d5640 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 0x5649d0859510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f27fff640 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 0x5649d08f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f277fe640 1 -- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 msgr2=0x5649d08f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f277fe640 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 0x5649d08f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f277fe640 1 -- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 msgr2=0x5649d0859510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f277fe640 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 0x5649d0859510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f277fe640 1 -- 172.21.15.81:0/3200436302 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649d0836440 con 0x5649d046d590 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f277fe640 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d0858fd0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c07d6f0 tx=0x7f5f1c0b6760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b130e6e93fc294 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:57:44.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3200436302 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c048040 con 0x5649d046d590 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3200436302 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7f5f1c07c9f0 con 0x5649d046d590 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.194+0000 7f5f353c7740 1 -- 172.21.15.81:0/3200436302 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f1c081080 con 0x5649d046d590 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3200436302 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=118) ==== 24+0+0 (secure 0 0 0) 0x7f5f1c089e50 con 0x5649d046d590 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f353c7740 10 client.11556 fetch_fsmap learned FSMap version 118 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f353c7740 1 -- 172.21.15.81:0/3200436302 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=118,monmap=2+}) -- 0x5649d0836440 con 0x5649d046d590 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3200436302 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f1c081080 con 0x5649d046d590 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3200436302 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 118) ==== 4149+0+0 (secure 0 0 0) 0x7f5f1c07cbf0 con 0x5649d046d590 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f353c7740 10 client.11556 fetch_fsmap finished waiting for FSMap version 118 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f353c7740 10 client.11556 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f353c7740 20 client.11556 populate_metadata read hostname 'smithi081' 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:6835/1664603190,v1:172.21.15.167:6837/1664603190] conn(0x5649d0878e20 0x5649d0879230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f353c7740 4 client.11556 mds_command: new command op to 10114 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f353c7740 1 -- 172.21.15.81:0/3200436302 --> [v2:172.21.15.167:6835/1664603190,v1:172.21.15.167:6837/1664603190] -- command(tid 0: {"prefix": "status"}) -- 0x5649d0874450 con 0x5649d0878e20 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.195+0000 7f5f27fff640 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:6835/1664603190,v1:172.21.15.167:6837/1664603190] conn(0x5649d0878e20 0x5649d0879230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f27fff640 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:6835/1664603190,v1:172.21.15.167:6837/1664603190] conn(0x5649d0878e20 0x5649d0879230 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5146866e152e7fc3 server_cookie=a6bd23717bd20b88 in_seq=0 out_seq=0 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f06ffd640 10 client.11556 ms_handle_connect on v2:172.21.15.167:6835/1664603190 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f06ffd640 1 -- 172.21.15.81:0/3200436302 <== mds.0 v2:172.21.15.167:6835/1664603190 1 ==== command_reply(tid 0: 0 ) ==== 8+0+338 (crc 0 0 0) 0x5649d0874450 con 0x5649d0878e20 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f06ffd640 10 client.11556 handle_command_reply: tid=0 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 client.11556 shutdown 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 -- 172.21.15.81:0/3200436302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 msgr2=0x5649d0858fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d0858fd0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c07d6f0 tx=0x7f5f1c0b6760 comp rx=0 tx=0).stop 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 -- 172.21.15.81:0/3200436302 shutdown_connections 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:6835/1664603190,v1:172.21.15.167:6837/1664603190] conn(0x5649d0878e20 0x5649d0879230 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x5649d048e270 0x5649d0859510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5649d046d590 0x5649d0858fd0 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f06ffd640 0 client.11556 ms_handle_reset on v2:172.21.15.167:6835/1664603190 2024-06-29T15:57:44.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 --2- 172.21.15.81:0/3200436302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x5649d04653b0 0x5649d08f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 -- 172.21.15.81:0/3200436302 >> 172.21.15.81:0/3200436302 conn(0x5649d04823c0 msgr2=0x5649d08885c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 -- 172.21.15.81:0/3200436302 shutdown_connections 2024-06-29T15:57:44.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 1 -- 172.21.15.81:0/3200436302 wait complete. 2024-06-29T15:57:44.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.196+0000 7f5f353c7740 20 client.11556 trim_cache size 0 max 16384 2024-06-29T15:57:44.203 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:57:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T15:57:44.203 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T15:57:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 10114, 2024-06-29T15:57:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T15:57:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T15:57:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "backup_fs", 2024-06-29T15:57:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 9.8330138280000003, 2024-06-29T15:57:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 118, 2024-06-29T15:57:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 217, 2024-06-29T15:57:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 217, 2024-06-29T15:57:44.204 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 26.593657361000002 2024-06-29T15:57:44.204 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:57:44.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 -- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5f140763f0 msgr2=0x7f5f14078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5f140763f0 0x7f5f14078900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f18007fd0 tx=0x7f5f18009040 comp rx=0 tx=0).stop 2024-06-29T15:57:44.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 -- 172.21.15.81:0/1477925139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2815c440 msgr2=0x7f5f280b4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2815c440 0x7f5f280b4990 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c034040 tx=0x7f5f1c0038a0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 -- 172.21.15.81:0/1477925139 shutdown_connections 2024-06-29T15:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f5f140763f0 0x7f5f14078900 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f2815d750 0x7f5f281566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2815c440 0x7f5f280b4990 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 --2- 172.21.15.81:0/1477925139 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f2815c030 0x7f5f280b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 -- 172.21.15.81:0/1477925139 >> 172.21.15.81:0/1477925139 conn(0x7f5f280d0930 msgr2=0x7f5f280c5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 -- 172.21.15.81:0/1477925139 shutdown_connections 2024-06-29T15:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.197+0000 7f5f2e438640 1 -- 172.21.15.81:0/1477925139 wait complete. 2024-06-29T15:57:44.208 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:57:44.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.323+0000 7fbf859be640 1 -- 172.21.15.81:0/2686396576 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf800d0b90 msgr2=0x7fbf800d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.323+0000 7fbf859be640 1 --2- 172.21.15.81:0/2686396576 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf800d0b90 0x7fbf800d1a80 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fbf8008e330 tx=0x7fbf7402a680 comp rx=0 tx=0).stop 2024-06-29T15:57:44.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.323+0000 7fbf859be640 1 -- 172.21.15.81:0/2686396576 shutdown_connections 2024-06-29T15:57:44.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.323+0000 7fbf859be640 1 --2- 172.21.15.81:0/2686396576 >> v2:172.21.15.167:3300/0 conn(0x7fbf800b5250 0x7fbf800d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.323+0000 7fbf859be640 1 --2- 172.21.15.81:0/2686396576 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf800d0b90 0x7fbf800d1a80 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.323+0000 7fbf859be640 1 -- 172.21.15.81:0/2686396576 >> 172.21.15.81:0/2686396576 conn(0x7fbf8009cf30 msgr2=0x7fbf800d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.323+0000 7fbf859be640 1 -- 172.21.15.81:0/2686396576 shutdown_connections 2024-06-29T15:57:44.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.323+0000 7fbf859be640 1 -- 172.21.15.81:0/2686396576 wait complete. 2024-06-29T15:57:44.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.324+0000 7fbf859be640 1 Processor -- start 2024-06-29T15:57:44.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.325+0000 7fbf859be640 1 -- start start 2024-06-29T15:57:44.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf859be640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf800b5250 0x7fbf8015b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf859be640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf800b9040 0x7fbf8014eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf859be640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf801f5100 0x7fbf800b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf859be640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbf800d2500 con 0x7fbf801f5100 2024-06-29T15:57:44.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf859be640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbf800b8530 con 0x7fbf800b9040 2024-06-29T15:57:44.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf859be640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbf800d2680 con 0x7fbf800b5250 2024-06-29T15:57:44.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf7f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf801f5100 0x7fbf800b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf7f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf801f5100 0x7fbf800b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54284/0 (socket says 172.21.15.81:54284) 2024-06-29T15:57:44.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2234361221 learned_addr learned my addr 172.21.15.81:0/2234361221 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:44.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf7effd640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf800b5250 0x7fbf8015b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf7e7fc640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf800b9040 0x7fbf8014eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf800b5250 msgr2=0x7fbf8015b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf7f7fe640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf800b5250 0x7fbf8015b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf800b9040 msgr2=0x7fbf8014eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf7f7fe640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf800b9040 0x7fbf8014eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.326+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2234361221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf801e7920 con 0x7fbf801f5100 2024-06-29T15:57:44.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.327+0000 7fbf7f7fe640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf801f5100 0x7fbf800b2590 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800e960 tx=0x7fbf7800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.327+0000 7fbf67fff640 1 -- 172.21.15.81:0/2234361221 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78004d20 con 0x7fbf801f5100 2024-06-29T15:57:44.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.327+0000 7fbf67fff640 1 -- 172.21.15.81:0/2234361221 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf7800cac0 con 0x7fbf801f5100 2024-06-29T15:57:44.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.327+0000 7fbf67fff640 1 -- 172.21.15.81:0/2234361221 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78005020 con 0x7fbf801f5100 2024-06-29T15:57:44.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.327+0000 7fbf859be640 1 -- 172.21.15.81:0/2234361221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf801e7c00 con 0x7fbf801f5100 2024-06-29T15:57:44.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.327+0000 7fbf859be640 1 -- 172.21.15.81:0/2234361221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf801e81c0 con 0x7fbf801f5100 2024-06-29T15:57:44.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.328+0000 7fbf859be640 1 -- 172.21.15.81:0/2234361221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf800b4700 con 0x7fbf801f5100 2024-06-29T15:57:44.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.329+0000 7fbf67fff640 1 -- 172.21.15.81:0/2234361221 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fbf780051c0 con 0x7fbf801f5100 2024-06-29T15:57:44.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.329+0000 7fbf67fff640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fbf6c076420 0x7fbf6c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.329+0000 7fbf67fff640 1 -- 172.21.15.81:0/2234361221 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7fbf78098580 con 0x7fbf801f5100 2024-06-29T15:57:44.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.329+0000 7fbf7effd640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fbf6c076420 0x7fbf6c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.330+0000 7fbf7effd640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fbf6c076420 0x7fbf6c078930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbf70000f60 tx=0x7fbf70009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.334+0000 7fbf67fff640 1 -- 172.21.15.81:0/2234361221 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbf78016030 con 0x7fbf801f5100 2024-06-29T15:57:44.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.538+0000 7fbf859be640 1 -- 172.21.15.81:0/2234361221 --> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fbf800b44b0 con 0x7fbf6c076420 2024-06-29T15:57:44.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.540+0000 7fbf67fff640 1 -- 172.21.15.81:0/2234361221 <== mgr.8666 v2:172.21.15.167:6832/2018755589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+82 (secure 0 0 0) 0x7fbf800b44b0 con 0x7fbf6c076420 2024-06-29T15:57:44.540 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:57:44.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:44.541 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T15:57:44.541 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:57:44.541 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:57:44.541 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "backup_fs" 2024-06-29T15:57:44.541 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:57:44.541 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:57:44.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 -- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fbf6c076420 msgr2=0x7fbf6c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fbf6c076420 0x7fbf6c078930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbf70000f60 tx=0x7fbf70009040 comp rx=0 tx=0).stop 2024-06-29T15:57:44.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 -- 172.21.15.81:0/2234361221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf801f5100 msgr2=0x7fbf800b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf801f5100 0x7fbf800b2590 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800e960 tx=0x7fbf7800ee20 comp rx=0 tx=0).stop 2024-06-29T15:57:44.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 -- 172.21.15.81:0/2234361221 shutdown_connections 2024-06-29T15:57:44.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fbf6c076420 0x7fbf6c078930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf801f5100 0x7fbf800b2590 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf800b9040 0x7fbf8014eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 --2- 172.21.15.81:0/2234361221 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf800b5250 0x7fbf8015b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 -- 172.21.15.81:0/2234361221 >> 172.21.15.81:0/2234361221 conn(0x7fbf8009cf30 msgr2=0x7fbf8015e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 -- 172.21.15.81:0/2234361221 shutdown_connections 2024-06-29T15:57:44.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.542+0000 7fbf859be640 1 -- 172.21.15.81:0/2234361221 wait complete. 2024-06-29T15:57:44.554 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:57:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.669+0000 7fe0b2295640 1 -- 172.21.15.81:0/2392604404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0ac073030 msgr2=0x7fe0ac073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.669+0000 7fe0b2295640 1 --2- 172.21.15.81:0/2392604404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0ac073030 0x7fe0ac073440 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0005fd0 tx=0x7fe0a002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.669+0000 7fe0b2295640 1 -- 172.21.15.81:0/2392604404 shutdown_connections 2024-06-29T15:57:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.669+0000 7fe0b2295640 1 --2- 172.21.15.81:0/2392604404 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0ac15e960 0x7fe0ac15ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.669+0000 7fe0b2295640 1 --2- 172.21.15.81:0/2392604404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0ac073030 0x7fe0ac073440 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.669+0000 7fe0b2295640 1 -- 172.21.15.81:0/2392604404 >> 172.21.15.81:0/2392604404 conn(0x7fe0ac06ccc0 msgr2=0x7fe0ac0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.669+0000 7fe0b2295640 1 -- 172.21.15.81:0/2392604404 shutdown_connections 2024-06-29T15:57:44.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.669+0000 7fe0b2295640 1 -- 172.21.15.81:0/2392604404 wait complete. 2024-06-29T15:57:44.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.671+0000 7fe0b2295640 1 Processor -- start 2024-06-29T15:57:44.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.671+0000 7fe0b2295640 1 -- start start 2024-06-29T15:57:44.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0b2295640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0ac073030 0x7fe0ac1568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0b2295640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0ac1412f0 0x7fe0ac158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0b2295640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0ac15e960 0x7fe0ac1384e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0b2295640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ac0b3390 con 0x7fe0ac15e960 2024-06-29T15:57:44.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0b2295640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ac0b3210 con 0x7fe0ac1412f0 2024-06-29T15:57:44.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0b2295640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe0ac0b3510 con 0x7fe0ac073030 2024-06-29T15:57:44.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0b0832640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0ac15e960 0x7fe0ac1384e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0b0832640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0ac15e960 0x7fe0ac1384e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54302/0 (socket says 172.21.15.81:54302) 2024-06-29T15:57:44.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0b0832640 1 -- 172.21.15.81:0/2701570729 learned_addr learned my addr 172.21.15.81:0/2701570729 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:44.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0ab7fe640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0ac1412f0 0x7fe0ac158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.672+0000 7fe0abfff640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0ac073030 0x7fe0ac1568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0b0832640 1 -- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0ac073030 msgr2=0x7fe0ac1568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0b0832640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0ac073030 0x7fe0ac1568a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0b0832640 1 -- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0ac1412f0 msgr2=0x7fe0ac158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0b0832640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0ac1412f0 0x7fe0ac158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0b0832640 1 -- 172.21.15.81:0/2701570729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ac1578f0 con 0x7fe0ac15e960 2024-06-29T15:57:44.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0b0832640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0ac15e960 0x7fe0ac1384e0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400e9c0 tx=0x7fe0a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0a8ff9640 1 -- 172.21.15.81:0/2701570729 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4004d20 con 0x7fe0ac15e960 2024-06-29T15:57:44.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0a8ff9640 1 -- 172.21.15.81:0/2701570729 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0a400cb20 con 0x7fe0ac15e960 2024-06-29T15:57:44.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0b2295640 1 -- 172.21.15.81:0/2701570729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0ac157050 con 0x7fe0ac15e960 2024-06-29T15:57:44.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0a8ff9640 1 -- 172.21.15.81:0/2701570729 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4005020 con 0x7fe0ac15e960 2024-06-29T15:57:44.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.673+0000 7fe0b2295640 1 -- 172.21.15.81:0/2701570729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0ac157610 con 0x7fe0ac15e960 2024-06-29T15:57:44.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.674+0000 7fe0b2295640 1 -- 172.21.15.81:0/2701570729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0ac0b3a10 con 0x7fe0ac15e960 2024-06-29T15:57:44.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.675+0000 7fe0a8ff9640 1 -- 172.21.15.81:0/2701570729 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 128) ==== 98303+0+0 (secure 0 0 0) 0x7fe0a40051c0 con 0x7fe0ac15e960 2024-06-29T15:57:44.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.675+0000 7fe0a8ff9640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe0980763f0 0x7fe098078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:44.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.675+0000 7fe0a8ff9640 1 -- 172.21.15.81:0/2701570729 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7fe0a40984d0 con 0x7fe0ac15e960 2024-06-29T15:57:44.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.675+0000 7fe0abfff640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe0980763f0 0x7fe098078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:44.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.676+0000 7fe0abfff640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe0980763f0 0x7fe098078900 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe09c001720 tx=0x7fe09c009040 comp rx=0 tx=0).ready entity=mgr.8666 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:44.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.680+0000 7fe0a8ff9640 1 -- 172.21.15.81:0/2701570729 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe0a4016030 con 0x7fe0ac15e960 2024-06-29T15:57:44.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.905+0000 7fe0b2295640 1 -- 172.21.15.81:0/2701570729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fe0ac156de0 con 0x7fe0ac15e960 2024-06-29T15:57:44.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.944+0000 7fe0a8ff9640 1 -- 172.21.15.81:0/2701570729 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v129) ==== 92+0+0 (secure 0 0 0) 0x7fe0a4061270 con 0x7fe0ac15e960 2024-06-29T15:57:44.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.946+0000 7fe0b2295640 1 -- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe0980763f0 msgr2=0x7fe098078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.946+0000 7fe0b2295640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe0980763f0 0x7fe098078900 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe09c001720 tx=0x7fe09c009040 comp rx=0 tx=0).stop 2024-06-29T15:57:44.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.946+0000 7fe0b2295640 1 -- 172.21.15.81:0/2701570729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0ac15e960 msgr2=0x7fe0ac1384e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:44.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.946+0000 7fe0b2295640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0ac15e960 0x7fe0ac1384e0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400e9c0 tx=0x7fe0a400ee80 comp rx=0 tx=0).stop 2024-06-29T15:57:44.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.946+0000 7fe0b2295640 1 -- 172.21.15.81:0/2701570729 shutdown_connections 2024-06-29T15:57:44.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.946+0000 7fe0b2295640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7fe0980763f0 0x7fe098078900 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.946+0000 7fe0b2295640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0ac15e960 0x7fe0ac1384e0 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.946+0000 7fe0b2295640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0ac1412f0 0x7fe0ac158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.946+0000 7fe0b2295640 1 --2- 172.21.15.81:0/2701570729 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0ac073030 0x7fe0ac1568a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:44.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.946+0000 7fe0b2295640 1 -- 172.21.15.81:0/2701570729 >> 172.21.15.81:0/2701570729 conn(0x7fe0ac06ccc0 msgr2=0x7fe0ac06f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:44.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.947+0000 7fe0b2295640 1 -- 172.21.15.81:0/2701570729 shutdown_connections 2024-06-29T15:57:44.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:44.947+0000 7fe0b2295640 1 -- 172.21.15.81:0/2701570729 wait complete. 2024-06-29T15:57:44.961 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:57:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.082+0000 7ff7a9efd640 1 -- 172.21.15.81:0/180648602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff790003a00 msgr2=0x7ff790003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.082+0000 7ff7a9efd640 1 --2- 172.21.15.81:0/180648602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff790003a00 0x7ff790003e70 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7ff794009b20 tx=0x7ff790005830 comp rx=0 tx=0).stop 2024-06-29T15:57:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.082+0000 7ff7a3fff640 1 -- 172.21.15.81:0/180648602 reap_dead start 2024-06-29T15:57:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.082+0000 7ff7a9efd640 1 -- 172.21.15.81:0/180648602 shutdown_connections 2024-06-29T15:57:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.082+0000 7ff7a9efd640 1 -- 172.21.15.81:0/180648602 >> 172.21.15.81:0/180648602 conn(0x7ff7a40d3c60 msgr2=0x7ff7a40d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.082+0000 7ff7a9efd640 1 -- 172.21.15.81:0/180648602 shutdown_connections 2024-06-29T15:57:45.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.082+0000 7ff7a9efd640 1 -- 172.21.15.81:0/180648602 wait complete. 2024-06-29T15:57:45.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.084+0000 7ff7a9efd640 1 Processor -- start 2024-06-29T15:57:45.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.084+0000 7ff7a9efd640 1 -- start start 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.085+0000 7ff7a9efd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff790003a00 0x7ff7a40ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.085+0000 7ff7a9efd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a40c7f10 0x7ff7a40c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.085+0000 7ff7a9efd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a40c8860 0x7ff7a41f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.085+0000 7ff7a9efd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a40d15a0 con 0x7ff790003a00 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.085+0000 7ff7a9efd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a40d1420 con 0x7ff7a40c7f10 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.085+0000 7ff7a9efd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a40d4990 con 0x7ff7a40c8860 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff790003a00 0x7ff7a40ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff790003a00 0x7ff7a40ca300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54316/0 (socket says 172.21.15.81:54316) 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a37fe640 1 -- 172.21.15.81:0/1207399279 learned_addr learned my addr 172.21.15.81:0/1207399279 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a3fff640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a40c8860 0x7ff7a41f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a2ffd640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a40c7f10 0x7ff7a40c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a37fe640 1 -- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a40c8860 msgr2=0x7ff7a41f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a37fe640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a40c8860 0x7ff7a41f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a37fe640 1 -- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a40c7f10 msgr2=0x7ff7a40c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a37fe640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a40c7f10 0x7ff7a40c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a37fe640 1 -- 172.21.15.81:0/1207399279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a40c7a60 con 0x7ff790003a00 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a37fe640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff790003a00 0x7ff7a40ca300 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7ff794009b20 tx=0x7ff794033650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff79bfff640 1 -- 172.21.15.81:0/1207399279 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794038040 con 0x7ff790003a00 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff79bfff640 1 -- 172.21.15.81:0/1207399279 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff79402b7e0 con 0x7ff790003a00 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff79bfff640 1 -- 172.21.15.81:0/1207399279 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff794005280 con 0x7ff790003a00 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a9efd640 1 -- 172.21.15.81:0/1207399279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a40cc270 con 0x7ff790003a00 2024-06-29T15:57:45.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.086+0000 7ff7a9efd640 1 -- 172.21.15.81:0/1207399279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a40cc830 con 0x7ff790003a00 2024-06-29T15:57:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.087+0000 7ff7a9efd640 1 -- 172.21.15.81:0/1207399279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7a40b3be0 con 0x7ff790003a00 2024-06-29T15:57:45.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.089+0000 7ff79bfff640 1 -- 172.21.15.81:0/1207399279 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 129) ==== 98320+0+0 (secure 0 0 0) 0x7ff794045020 con 0x7ff790003a00 2024-06-29T15:57:45.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.089+0000 7ff79bfff640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff784076470 0x7ff784078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.089+0000 7ff7a2ffd640 1 -- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff784076470 msgr2=0x7ff784078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2018755589 2024-06-29T15:57:45.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.090+0000 7ff79bfff640 1 -- 172.21.15.81:0/1207399279 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7ff7940b84b0 con 0x7ff790003a00 2024-06-29T15:57:45.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.090+0000 7ff7a2ffd640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff784076470 0x7ff784078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:45.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.094+0000 7ff79bfff640 1 -- 172.21.15.81:0/1207399279 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff7940819e0 con 0x7ff790003a00 2024-06-29T15:57:45.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.286+0000 7ff7a9efd640 1 -- 172.21.15.81:0/1207399279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7ff7a40d0460 con 0x7ff790003a00 2024-06-29T15:57:45.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.289+0000 7ff79bfff640 1 -- 172.21.15.81:0/1207399279 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v73)=0 v73) ==== 138+0+0 (secure 0 0 0) 0x7ff7940812e0 con 0x7ff790003a00 2024-06-29T15:57:45.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a2ffd640 1 -- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff784076470 msgr2=0x7ff784078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2018755589 2024-06-29T15:57:45.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a2ffd640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff784076470 0x7ff784078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:45.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a9efd640 1 -- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff784076470 msgr2=0x7ff784078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:45.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a9efd640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff784076470 0x7ff784078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a9efd640 1 -- 172.21.15.81:0/1207399279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff790003a00 msgr2=0x7ff7a40ca300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a9efd640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff790003a00 0x7ff7a40ca300 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7ff794009b20 tx=0x7ff794033650 comp rx=0 tx=0).stop 2024-06-29T15:57:45.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a9efd640 1 -- 172.21.15.81:0/1207399279 shutdown_connections 2024-06-29T15:57:45.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a9efd640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7ff784076470 0x7ff784078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a9efd640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7a40c8860 0x7ff7a41f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a9efd640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7a40c7f10 0x7ff7a40c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a9efd640 1 --2- 172.21.15.81:0/1207399279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff790003a00 0x7ff7a40ca300 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.291+0000 7ff7a9efd640 1 -- 172.21.15.81:0/1207399279 >> 172.21.15.81:0/1207399279 conn(0x7ff7a40d3c60 msgr2=0x7ff7a40b6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.292+0000 7ff7a9efd640 1 -- 172.21.15.81:0/1207399279 shutdown_connections 2024-06-29T15:57:45.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.292+0000 7ff7a9efd640 1 -- 172.21.15.81:0/1207399279 wait complete. 2024-06-29T15:57:45.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:57:45.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.423+0000 7febe2e3d640 1 -- 172.21.15.81:0/1363253231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febdc15bc20 msgr2=0x7febdc06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.423+0000 7febe2e3d640 1 --2- 172.21.15.81:0/1363253231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febdc15bc20 0x7febdc06ed90 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7febd4005fd0 tx=0x7febd402a820 comp rx=0 tx=0).stop 2024-06-29T15:57:45.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.423+0000 7febe2e3d640 1 -- 172.21.15.81:0/1363253231 shutdown_connections 2024-06-29T15:57:45.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.423+0000 7febe2e3d640 1 --2- 172.21.15.81:0/1363253231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febdc15bc20 0x7febdc06ed90 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.423+0000 7febe2e3d640 1 --2- 172.21.15.81:0/1363253231 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7febdc073650 0x7febdc15b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.423+0000 7febe2e3d640 1 -- 172.21.15.81:0/1363253231 >> 172.21.15.81:0/1363253231 conn(0x7febdc0b2920 msgr2=0x7febdc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.423+0000 7febe2e3d640 1 -- 172.21.15.81:0/1363253231 shutdown_connections 2024-06-29T15:57:45.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.423+0000 7febe2e3d640 1 -- 172.21.15.81:0/1363253231 wait complete. 2024-06-29T15:57:45.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.424+0000 7febe2e3d640 1 Processor -- start 2024-06-29T15:57:45.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.424+0000 7febe2e3d640 1 -- start start 2024-06-29T15:57:45.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe2e3d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7febdc0cd150 0x7febdc14aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe2e3d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febdc15bc20 0x7febdc0d0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe2e3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febdc158660 0x7febdc14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe2e3d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7febdc06f450 con 0x7febdc158660 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe2e3d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7febdc06f2d0 con 0x7febdc15bc20 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe2e3d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7febdc06f5d0 con 0x7febdc0cd150 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe13da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febdc158660 0x7febdc14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe13da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febdc158660 0x7febdc14afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54334/0 (socket says 172.21.15.81:54334) 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe13da640 1 -- 172.21.15.81:0/2287388889 learned_addr learned my addr 172.21.15.81:0/2287388889 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe0bd9640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7febdc0cd150 0x7febdc14aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febdbfff640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febdc15bc20 0x7febdc0d0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe13da640 1 -- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7febdc0cd150 msgr2=0x7febdc14aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe13da640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7febdc0cd150 0x7febdc14aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe13da640 1 -- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febdc15bc20 msgr2=0x7febdc0d0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe13da640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febdc15bc20 0x7febdc0d0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.426+0000 7febe13da640 1 -- 172.21.15.81:0/2287388889 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febdc0cfb60 con 0x7febdc158660 2024-06-29T15:57:45.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.427+0000 7febe13da640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febdc158660 0x7febdc14afd0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7febd4006010 tx=0x7febd4032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.427+0000 7febd97fa640 1 -- 172.21.15.81:0/2287388889 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4037040 con 0x7febdc158660 2024-06-29T15:57:45.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.427+0000 7febd97fa640 1 -- 172.21.15.81:0/2287388889 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febd4042070 con 0x7febdc158660 2024-06-29T15:57:45.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.427+0000 7febd97fa640 1 -- 172.21.15.81:0/2287388889 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd40042a0 con 0x7febdc158660 2024-06-29T15:57:45.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.427+0000 7febe2e3d640 1 -- 172.21.15.81:0/2287388889 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febdc0cfe40 con 0x7febdc158660 2024-06-29T15:57:45.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.427+0000 7febe2e3d640 1 -- 172.21.15.81:0/2287388889 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febdc0d0400 con 0x7febdc158660 2024-06-29T15:57:45.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.428+0000 7febe2e3d640 1 -- 172.21.15.81:0/2287388889 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febdc15b6e0 con 0x7febdc158660 2024-06-29T15:57:45.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.428+0000 7febd97fa640 1 -- 172.21.15.81:0/2287388889 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 129) ==== 98320+0+0 (secure 0 0 0) 0x7febd4004440 con 0x7febdc158660 2024-06-29T15:57:45.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.429+0000 7febd97fa640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7febc0076440 0x7febc0078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.429+0000 7febe0bd9640 1 -- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7febc0076440 msgr2=0x7febc0078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2018755589 2024-06-29T15:57:45.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.429+0000 7febd97fa640 1 -- 172.21.15.81:0/2287388889 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7febd40b8b40 con 0x7febdc158660 2024-06-29T15:57:45.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.429+0000 7febe0bd9640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7febc0076440 0x7febc0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:45.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.433+0000 7febd97fa640 1 -- 172.21.15.81:0/2287388889 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7febd40817c0 con 0x7febdc158660 2024-06-29T15:57:45.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.622+0000 7febe2e3d640 1 -- 172.21.15.81:0/2287388889 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7febdc15f260 con 0x7febdc158660 2024-06-29T15:57:45.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.626+0000 7febd97fa640 1 -- 172.21.15.81:0/2287388889 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v74)=0 v74) ==== 133+0+0 (secure 0 0 0) 0x7febd4085ac0 con 0x7febdc158660 2024-06-29T15:57:45.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 -- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7febc0076440 msgr2=0x7febc0078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7febc0076440 0x7febc0078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 -- 172.21.15.81:0/2287388889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febdc158660 msgr2=0x7febdc14afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febdc158660 0x7febdc14afd0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7febd4006010 tx=0x7febd4032d10 comp rx=0 tx=0).stop 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 -- 172.21.15.81:0/2287388889 shutdown_connections 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7febc0076440 0x7febc0078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febdc158660 0x7febdc14afd0 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7febdc15bc20 0x7febdc0d0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 --2- 172.21.15.81:0/2287388889 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7febdc0cd150 0x7febdc14aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 -- 172.21.15.81:0/2287388889 >> 172.21.15.81:0/2287388889 conn(0x7febdc0b2920 msgr2=0x7febdc15d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 -- 172.21.15.81:0/2287388889 shutdown_connections 2024-06-29T15:57:45.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.628+0000 7febe2e3d640 1 -- 172.21.15.81:0/2287388889 wait complete. 2024-06-29T15:57:45.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:45.640 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-06-29T15:57:45.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.821+0000 7f3122b70640 1 -- 172.21.15.81:0/1638971149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c0d2440 msgr2=0x7f311c0d4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.821+0000 7f3122b70640 1 --2- 172.21.15.81:0/1638971149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c0d2440 0x7f311c0d4770 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f311c0ce5f0 tx=0x7f311002a4d0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.821+0000 7f3122b70640 1 -- 172.21.15.81:0/1638971149 shutdown_connections 2024-06-29T15:57:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.821+0000 7f3122b70640 1 --2- 172.21.15.81:0/1638971149 >> v2:172.21.15.167:3300/0 conn(0x7f311c0d03b0 0x7f311c0d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.821+0000 7f3122b70640 1 --2- 172.21.15.81:0/1638971149 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f311c0b2ea0 0x7f311c0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.821+0000 7f3122b70640 1 --2- 172.21.15.81:0/1638971149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c0d2440 0x7f311c0d4770 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.821+0000 7f3122b70640 1 -- 172.21.15.81:0/1638971149 >> 172.21.15.81:0/1638971149 conn(0x7f311c0d3c50 msgr2=0x7f311c0d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.821+0000 7f3122b70640 1 -- 172.21.15.81:0/1638971149 shutdown_connections 2024-06-29T15:57:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.821+0000 7f3122b70640 1 -- 172.21.15.81:0/1638971149 wait complete. 2024-06-29T15:57:45.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.823+0000 7f3122b70640 1 Processor -- start 2024-06-29T15:57:45.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.823+0000 7f3122b70640 1 -- start start 2024-06-29T15:57:45.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.824+0000 7f3122b70640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f311c0b2ea0 0x7f311c0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.824+0000 7f3122b70640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f311c0d03b0 0x7f311c159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.824+0000 7f3122b70640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c0d2440 0x7f311c15afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.824+0000 7f3122b70640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f311c0cfbd0 con 0x7f311c0d2440 2024-06-29T15:57:45.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.824+0000 7f3122b70640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f311c0cfa50 con 0x7f311c0b2ea0 2024-06-29T15:57:45.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.824+0000 7f3122b70640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f311c06cb10 con 0x7f311c0d03b0 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.824+0000 7f312110d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c0d2440 0x7f311c15afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.824+0000 7f312110d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c0d2440 0x7f311c15afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54358/0 (socket says 172.21.15.81:54358) 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.824+0000 7f312110d640 1 -- 172.21.15.81:0/3955005281 learned_addr learned my addr 172.21.15.81:0/3955005281 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f311bfff640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f311c0d03b0 0x7f311c159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f312090c640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f311c0b2ea0 0x7f311c0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f312110d640 1 -- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f311c0d03b0 msgr2=0x7f311c159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f312110d640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f311c0d03b0 0x7f311c159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f312110d640 1 -- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f311c0b2ea0 msgr2=0x7f311c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f312110d640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f311c0b2ea0 0x7f311c0cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f312110d640 1 -- 172.21.15.81:0/3955005281 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311c15c190 con 0x7f311c0d2440 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f312110d640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c0d2440 0x7f311c15afd0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f311400e990 tx=0x7f311400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3114004d20 con 0x7f311c0d2440 2024-06-29T15:57:45.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f311400ca40 con 0x7f311c0d2440 2024-06-29T15:57:45.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f3122b70640 1 -- 172.21.15.81:0/3955005281 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311c15c470 con 0x7f311c0d2440 2024-06-29T15:57:45.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3114005020 con 0x7f311c0d2440 2024-06-29T15:57:45.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.825+0000 7f3122b70640 1 -- 172.21.15.81:0/3955005281 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f311c0cbaa0 con 0x7f311c0d2440 2024-06-29T15:57:45.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.826+0000 7f3122b70640 1 -- 172.21.15.81:0/3955005281 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f311c0d14a0 con 0x7f311c0d2440 2024-06-29T15:57:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.827+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 129) ==== 98320+0+0 (secure 0 0 0) 0x7f31140051c0 con 0x7f311c0d2440 2024-06-29T15:57:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.827+0000 7f31197fa640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 0x7f3100078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.827+0000 7f312090c640 1 -- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 msgr2=0x7f3100078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2018755589 2024-06-29T15:57:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.827+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6701+0+0 (secure 0 0 0) 0x7f31140985e0 con 0x7f311c0d2440 2024-06-29T15:57:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.827+0000 7f312090c640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 0x7f3100078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:45.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:45.832+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3114016030 con 0x7f311c0d2440 2024-06-29T15:57:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:46.028+0000 7f312090c640 1 -- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 msgr2=0x7f3100078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2018755589 2024-06-29T15:57:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:46.028+0000 7f312090c640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 0x7f3100078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:46.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:46.041+0000 7f3122b70640 1 -- 172.21.15.81:0/3955005281 --> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f311c15beb0 con 0x7f3100076440 2024-06-29T15:57:46.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:46.429+0000 7f312090c640 1 -- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 msgr2=0x7f3100078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2018755589 2024-06-29T15:57:46.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:46.429+0000 7f312090c640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 0x7f3100078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:57:47.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:47.230+0000 7f312090c640 1 -- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 msgr2=0x7f3100078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2018755589 2024-06-29T15:57:47.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:47.230+0000 7f312090c640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 0x7f3100078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:57:48.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:48.602+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 130) ==== 97929+0+0 (secure 0 0 0) 0x7f3114096720 con 0x7f311c0d2440 2024-06-29T15:57:48.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:48.603+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 msgr2=0x7f3100078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:48.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:48.603+0000 7f31197fa640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2018755589,v1:172.21.15.167:6833/2018755589] conn(0x7f3100076440 0x7f3100078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:48.835 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.835+0000 7fda2ca0e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.836 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.835+0000 7fda2ca0e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.836 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.835+0000 7fda2ca0e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.836 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.835+0000 7fda2ca0e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.836 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.835+0000 7fda2ca0e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.836 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.836+0000 7fda2fa14640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.837 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.836+0000 7fda2fa14640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.837 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.836+0000 7fda2fa14640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.837 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.836+0000 7fda2fa14640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.837 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.836+0000 7fda2fa14640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.846 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.845+0000 7fda2fa14640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.846 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.845+0000 7fda2fa14640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.846 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.845+0000 7fda2fa14640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.846 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.845+0000 7fda2fa14640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.846 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.845+0000 7fda2fa14640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.846 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.846+0000 7fda2ca0e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.847 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.846+0000 7fda2ca0e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.847 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.846+0000 7fda2ca0e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.847 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.846+0000 7fda2ca0e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:48.847 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:48.846+0000 7fda2ca0e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:49.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.605+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 131) ==== 98056+0+0 (secure 0 0 0) 0x7f311405d610 con 0x7f311c0d2440 2024-06-29T15:57:49.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.606+0000 7f31197fa640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f3100080b20 0x7f3100082f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:49.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.606+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 --> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f3114015e80 con 0x7f3100080b20 2024-06-29T15:57:49.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.611+0000 7f312090c640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f3100080b20 0x7f3100082f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:49.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.614+0000 7f312090c640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f3100080b20 0x7f3100082f40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3110033000 tx=0x7f3110004730 comp rx=0 tx=0).ready entity=mgr.10348 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:49.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.677+0000 7f31197fa640 1 -- 172.21.15.81:0/3955005281 <== mgr.10348 v2:172.21.15.167:6832/2452570162 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+53 (secure 0 0 0) 0x7f3114015e80 con 0x7f3100080b20 2024-06-29T15:57:49.678 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path mandatory_fs_argument_test_dir 2024-06-29T15:57:49.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.680+0000 7f3122b70640 1 -- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f3100080b20 msgr2=0x7f3100082f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:49.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.680+0000 7f3122b70640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f3100080b20 0x7f3100082f40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3110033000 tx=0x7f3110004730 comp rx=0 tx=0).stop 2024-06-29T15:57:49.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.680+0000 7f3122b70640 1 -- 172.21.15.81:0/3955005281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c0d2440 msgr2=0x7f311c15afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:49.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.680+0000 7f3122b70640 1 --2- 172.21.15.81:0/3955005281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c0d2440 0x7f311c15afd0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f311400e990 tx=0x7f311400ee50 comp rx=0 tx=0).stop 2024-06-29T15:57:49.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.680+0000 7f312110d640 1 -- 172.21.15.81:0/3955005281 reap_dead start 2024-06-29T15:57:49.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.680+0000 7f3122b70640 1 -- 172.21.15.81:0/3955005281 shutdown_connections 2024-06-29T15:57:49.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.680+0000 7f3122b70640 1 -- 172.21.15.81:0/3955005281 >> 172.21.15.81:0/3955005281 conn(0x7f311c0d3c50 msgr2=0x7f311c0d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:49.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.680+0000 7f3122b70640 1 -- 172.21.15.81:0/3955005281 shutdown_connections 2024-06-29T15:57:49.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.680+0000 7f3122b70640 1 -- 172.21.15.81:0/3955005281 wait complete. 2024-06-29T15:57:49.692 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2024-06-29T15:57:49.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:49.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.811+0000 7f05adbeb640 1 -- 172.21.15.81:0/3631080133 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05a8144630 msgr2=0x7f05a8072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:49.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.811+0000 7f05adbeb640 1 --2- 172.21.15.81:0/3631080133 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05a8144630 0x7f05a8072720 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f059c009830 tx=0x7f059c02a510 comp rx=0 tx=0).stop 2024-06-29T15:57:49.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.811+0000 7f05adbeb640 1 -- 172.21.15.81:0/3631080133 shutdown_connections 2024-06-29T15:57:49.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.811+0000 7f05adbeb640 1 --2- 172.21.15.81:0/3631080133 >> v2:172.21.15.167:3300/0 conn(0x7f05a806f760 0x7f05a806fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:49.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.811+0000 7f05adbeb640 1 --2- 172.21.15.81:0/3631080133 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05a8144630 0x7f05a8072720 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:49.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.811+0000 7f05adbeb640 1 -- 172.21.15.81:0/3631080133 >> 172.21.15.81:0/3631080133 conn(0x7f05a806cd60 msgr2=0x7f05a80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:49.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.811+0000 7f05adbeb640 1 -- 172.21.15.81:0/3631080133 shutdown_connections 2024-06-29T15:57:49.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.811+0000 7f05adbeb640 1 -- 172.21.15.81:0/3631080133 wait complete. 2024-06-29T15:57:49.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.813+0000 7f05adbeb640 1 Processor -- start 2024-06-29T15:57:49.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.813+0000 7f05adbeb640 1 -- start start 2024-06-29T15:57:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05adbeb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05a806f760 0x7f05a813e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05adbeb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05a80b4130 0x7f05a81f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05adbeb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05a8144630 0x7f05a80cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05adbeb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f05a80b2a30 con 0x7f05a806f760 2024-06-29T15:57:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05adbeb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f05a8072e70 con 0x7f05a8144630 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05adbeb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f05a80b2bb0 con 0x7f05a80b4130 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05a806f760 0x7f05a813e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05a806f760 0x7f05a813e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54362/0 (socket says 172.21.15.81:54362) 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a77fe640 1 -- 172.21.15.81:0/1646274983 learned_addr learned my addr 172.21.15.81:0/1646274983 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a6ffd640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05a80b4130 0x7f05a81f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a7fff640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05a8144630 0x7f05a80cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a77fe640 1 -- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05a80b4130 msgr2=0x7f05a81f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a77fe640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05a80b4130 0x7f05a81f6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a77fe640 1 -- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05a8144630 msgr2=0x7f05a80cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a77fe640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05a8144630 0x7f05a80cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a77fe640 1 -- 172.21.15.81:0/1646274983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a81f68d0 con 0x7f05a806f760 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05a77fe640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05a806f760 0x7f05a813e640 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f059800cb80 tx=0x7f059800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f058ffff640 1 -- 172.21.15.81:0/1646274983 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598016020 con 0x7f05a806f760 2024-06-29T15:57:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f058ffff640 1 -- 172.21.15.81:0/1646274983 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f059800ad20 con 0x7f05a806f760 2024-06-29T15:57:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f05adbeb640 1 -- 172.21.15.81:0/1646274983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a81f6de0 con 0x7f05a806f760 2024-06-29T15:57:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.815+0000 7f058ffff640 1 -- 172.21.15.81:0/1646274983 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598012680 con 0x7f05a806f760 2024-06-29T15:57:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.816+0000 7f05adbeb640 1 -- 172.21.15.81:0/1646274983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a80aaea0 con 0x7f05a806f760 2024-06-29T15:57:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.816+0000 7f05adbeb640 1 -- 172.21.15.81:0/1646274983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a806fb70 con 0x7f05a806f760 2024-06-29T15:57:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.817+0000 7f058ffff640 1 -- 172.21.15.81:0/1646274983 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 131) ==== 98056+0+0 (secure 0 0 0) 0x7f0598019020 con 0x7f05a806f760 2024-06-29T15:57:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.818+0000 7f058ffff640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f05940761a0 0x7f05940786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.818+0000 7f058ffff640 1 -- 172.21.15.81:0/1646274983 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f0598097ea0 con 0x7f05a806f760 2024-06-29T15:57:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.818+0000 7f05a6ffd640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f05940761a0 0x7f05940786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.818+0000 7f05a6ffd640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f05940761a0 0x7f05940786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f059c034000 tx=0x7f059c004730 comp rx=0 tx=0).ready entity=mgr.10348 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:49.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:49.822+0000 7f058ffff640 1 -- 172.21.15.81:0/1646274983 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0598020020 con 0x7f05a806f760 2024-06-29T15:57:50.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.025+0000 7f05adbeb640 1 -- 172.21.15.81:0/1646274983 --> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) -- 0x7f05a80af6c0 con 0x7f05940761a0 2024-06-29T15:57:50.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.077+0000 7f058ffff640 1 -- 172.21.15.81:0/1646274983 <== mgr.10348 v2:172.21.15.167:6832/2452570162 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f05a80af6c0 con 0x7f05940761a0 2024-06-29T15:57:50.077 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2024-06-29T15:57:50.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 -- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f05940761a0 msgr2=0x7f05940786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f05940761a0 0x7f05940786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f059c034000 tx=0x7f059c004730 comp rx=0 tx=0).stop 2024-06-29T15:57:50.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 -- 172.21.15.81:0/1646274983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05a806f760 msgr2=0x7f05a813e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05a806f760 0x7f05a813e640 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f059800cb80 tx=0x7f059800cf80 comp rx=0 tx=0).stop 2024-06-29T15:57:50.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 -- 172.21.15.81:0/1646274983 shutdown_connections 2024-06-29T15:57:50.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f05940761a0 0x7f05940786b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05a8144630 0x7f05a80cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05a80b4130 0x7f05a81f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 --2- 172.21.15.81:0/1646274983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05a806f760 0x7f05a813e640 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 -- 172.21.15.81:0/1646274983 >> 172.21.15.81:0/1646274983 conn(0x7f05a806cd60 msgr2=0x7f05a80b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:50.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 -- 172.21.15.81:0/1646274983 shutdown_connections 2024-06-29T15:57:50.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.079+0000 7f05adbeb640 1 -- 172.21.15.81:0/1646274983 wait complete. 2024-06-29T15:57:50.091 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2024-06-29T15:57:50.091 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:50.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.209+0000 7f75a1c3c640 1 -- 172.21.15.81:0/4118453256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759c074520 msgr2=0x7f759c06f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.209+0000 7f75a1c3c640 1 --2- 172.21.15.81:0/4118453256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759c074520 0x7f759c06f140 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f759c073780 tx=0x7f758c02a5f0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.209+0000 7f75a1c3c640 1 -- 172.21.15.81:0/4118453256 shutdown_connections 2024-06-29T15:57:50.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.209+0000 7f75a1c3c640 1 --2- 172.21.15.81:0/4118453256 >> v2:172.21.15.167:3300/0 conn(0x7f759c06fa90 0x7f759c06ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.209+0000 7f75a1c3c640 1 --2- 172.21.15.81:0/4118453256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759c074520 0x7f759c06f140 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.209+0000 7f75a1c3c640 1 -- 172.21.15.81:0/4118453256 >> 172.21.15.81:0/4118453256 conn(0x7f759c072950 msgr2=0x7f759c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:50.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.209+0000 7f75a1c3c640 1 -- 172.21.15.81:0/4118453256 shutdown_connections 2024-06-29T15:57:50.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.209+0000 7f75a1c3c640 1 -- 172.21.15.81:0/4118453256 wait complete. 2024-06-29T15:57:50.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.211+0000 7f75a1c3c640 1 Processor -- start 2024-06-29T15:57:50.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.211+0000 7f75a1c3c640 1 -- start start 2024-06-29T15:57:50.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.212+0000 7f75a1c3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f759c06f680 0x7f759c1f16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.212+0000 7f75a1c3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f759c06fa90 0x7f759c0b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.212+0000 7f75a1c3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759c074520 0x7f759c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.212+0000 7f75a1c3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f759c15b010 con 0x7f759c074520 2024-06-29T15:57:50.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.212+0000 7f75a1c3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f759c06d150 con 0x7f759c06f680 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.212+0000 7f75a1c3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f759c15b190 con 0x7f759c06fa90 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.212+0000 7f759bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759c074520 0x7f759c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.212+0000 7f759bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759c074520 0x7f759c0b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35144/0 (socket says 172.21.15.81:35144) 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.212+0000 7f759bfff640 1 -- 172.21.15.81:0/286485315 learned_addr learned my addr 172.21.15.81:0/286485315 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.212+0000 7f759b7fe640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f759c06f680 0x7f759c1f16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f759affd640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f759c06fa90 0x7f759c0b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f759bfff640 1 -- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f759c06fa90 msgr2=0x7f759c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f759bfff640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f759c06fa90 0x7f759c0b0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f759bfff640 1 -- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f759c06f680 msgr2=0x7f759c1f16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f759bfff640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f759c06f680 0x7f759c1f16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f759bfff640 1 -- 172.21.15.81:0/286485315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759c15c4f0 con 0x7f759c074520 2024-06-29T15:57:50.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f759bfff640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759c074520 0x7f759c0b1500 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f7594009ab0 tx=0x7f7594009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:50.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f7583fff640 1 -- 172.21.15.81:0/286485315 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759400e2e0 con 0x7f759c074520 2024-06-29T15:57:50.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f7583fff640 1 -- 172.21.15.81:0/286485315 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7594004b20 con 0x7f759c074520 2024-06-29T15:57:50.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f75a1c3c640 1 -- 172.21.15.81:0/286485315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f759c15c7d0 con 0x7f759c074520 2024-06-29T15:57:50.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f7583fff640 1 -- 172.21.15.81:0/286485315 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7594007980 con 0x7f759c074520 2024-06-29T15:57:50.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.213+0000 7f75a1c3c640 1 -- 172.21.15.81:0/286485315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f759c15cd90 con 0x7f759c074520 2024-06-29T15:57:50.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.214+0000 7f75a1c3c640 1 -- 172.21.15.81:0/286485315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f759c0af4d0 con 0x7f759c074520 2024-06-29T15:57:50.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.215+0000 7f7583fff640 1 -- 172.21.15.81:0/286485315 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 131) ==== 98056+0+0 (secure 0 0 0) 0x7f7594007b20 con 0x7f759c074520 2024-06-29T15:57:50.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.215+0000 7f7583fff640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f75880761a0 0x7f75880786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.215+0000 7f7583fff640 1 -- 172.21.15.81:0/286485315 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f7594099320 con 0x7f759c074520 2024-06-29T15:57:50.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.215+0000 7f759b7fe640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f75880761a0 0x7f75880786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.216+0000 7f759b7fe640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f75880761a0 0x7f75880786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f758c033000 tx=0x7f758c004730 comp rx=0 tx=0).ready entity=mgr.10348 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:50.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.219+0000 7f7583fff640 1 -- 172.21.15.81:0/286485315 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7594062430 con 0x7f759c074520 2024-06-29T15:57:50.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.425+0000 7f75a1c3c640 1 -- 172.21.15.81:0/286485315 --> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) -- 0x7f759c06f460 con 0x7f75880761a0 2024-06-29T15:57:50.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.483+0000 7f7583fff640 1 -- 172.21.15.81:0/286485315 <== mgr.10348 v2:172.21.15.167:6832/2452570162 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+55 (secure 0 0 0) 0x7f759c06f460 con 0x7f75880761a0 2024-06-29T15:57:50.484 INFO:teuthology.orchestra.run.smithi081.stdout:Retention added to path mandatory_fs_argument_test_dir 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 -- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f75880761a0 msgr2=0x7f75880786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f75880761a0 0x7f75880786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f758c033000 tx=0x7f758c004730 comp rx=0 tx=0).stop 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 -- 172.21.15.81:0/286485315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759c074520 msgr2=0x7f759c0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759c074520 0x7f759c0b1500 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f7594009ab0 tx=0x7f7594009f70 comp rx=0 tx=0).stop 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 -- 172.21.15.81:0/286485315 shutdown_connections 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f75880761a0 0x7f75880786b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f759c074520 0x7f759c0b1500 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f759c06fa90 0x7f759c0b0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 --2- 172.21.15.81:0/286485315 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f759c06f680 0x7f759c1f16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 -- 172.21.15.81:0/286485315 >> 172.21.15.81:0/286485315 conn(0x7f759c072950 msgr2=0x7f759c156490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:50.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 -- 172.21.15.81:0/286485315 shutdown_connections 2024-06-29T15:57:50.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.486+0000 7f75a1c3c640 1 -- 172.21.15.81:0/286485315 wait complete. 2024-06-29T15:57:50.498 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2024-06-29T15:57:50.498 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-06-29T15:57:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.617+0000 7f4ed7a36640 1 -- 172.21.15.81:0/2664145911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ed00cefe0 msgr2=0x7f4ed00cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.617+0000 7f4ed7a36640 1 --2- 172.21.15.81:0/2664145911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ed00cefe0 0x7f4ed00cf3f0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f4ed006e320 tx=0x7f4ec002a560 comp rx=0 tx=0).stop 2024-06-29T15:57:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.617+0000 7f4ed7a36640 1 -- 172.21.15.81:0/2664145911 shutdown_connections 2024-06-29T15:57:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.617+0000 7f4ed7a36640 1 --2- 172.21.15.81:0/2664145911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ed00cefe0 0x7f4ed00cf3f0 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.617+0000 7f4ed7a36640 1 --2- 172.21.15.81:0/2664145911 >> v2:172.21.15.167:3300/0 conn(0x7f4ed006c2f0 0x7f4ed006c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.617+0000 7f4ed7a36640 1 -- 172.21.15.81:0/2664145911 >> 172.21.15.81:0/2664145911 conn(0x7f4ed00b2580 msgr2=0x7f4ed00b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.617+0000 7f4ed7a36640 1 -- 172.21.15.81:0/2664145911 shutdown_connections 2024-06-29T15:57:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.617+0000 7f4ed7a36640 1 -- 172.21.15.81:0/2664145911 wait complete. 2024-06-29T15:57:50.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.619+0000 7f4ed7a36640 1 Processor -- start 2024-06-29T15:57:50.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.619+0000 7f4ed7a36640 1 -- start start 2024-06-29T15:57:50.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.620+0000 7f4ed7a36640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ed006c2f0 0x7f4ed01f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.620+0000 7f4ed7a36640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ed0072ef0 0x7f4ed00b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.620+0000 7f4ed7a36640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ed00cefe0 0x7f4ed015e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.620+0000 7f4ed7a36640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed0070f60 con 0x7f4ed00cefe0 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.620+0000 7f4ed7a36640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed00cfb40 con 0x7f4ed0072ef0 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.620+0000 7f4ed7a36640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed00710e0 con 0x7f4ed006c2f0 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed5fd3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ed00cefe0 0x7f4ed015e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed5fd3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ed00cefe0 0x7f4ed015e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35166/0 (socket says 172.21.15.81:35166) 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed5fd3640 1 -- 172.21.15.81:0/1980201938 learned_addr learned my addr 172.21.15.81:0/1980201938 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed57d2640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ed006c2f0 0x7f4ed01f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed5fd3640 1 -- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ed006c2f0 msgr2=0x7f4ed01f7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed5fd3640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ed006c2f0 0x7f4ed01f7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed5fd3640 1 -- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ed0072ef0 msgr2=0x7f4ed00b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed4fd1640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ed0072ef0 0x7f4ed00b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed5fd3640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ed0072ef0 0x7f4ed00b51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed5fd3640 1 -- 172.21.15.81:0/1980201938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed00ae2e0 con 0x7f4ed00cefe0 2024-06-29T15:57:50.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed5fd3640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ed00cefe0 0x7f4ed015e940 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800b410 tx=0x7f4ec800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:50.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ec5ffb640 1 -- 172.21.15.81:0/1980201938 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8015030 con 0x7f4ed00cefe0 2024-06-29T15:57:50.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ec5ffb640 1 -- 172.21.15.81:0/1980201938 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ec8004020 con 0x7f4ed00cefe0 2024-06-29T15:57:50.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ec5ffb640 1 -- 172.21.15.81:0/1980201938 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8004470 con 0x7f4ed00cefe0 2024-06-29T15:57:50.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed7a36640 1 -- 172.21.15.81:0/1980201938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed00ae5c0 con 0x7f4ed00cefe0 2024-06-29T15:57:50.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.621+0000 7f4ed7a36640 1 -- 172.21.15.81:0/1980201938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed00aeb80 con 0x7f4ed00cefe0 2024-06-29T15:57:50.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.623+0000 7f4ed7a36640 1 -- 172.21.15.81:0/1980201938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed0070cd0 con 0x7f4ed00cefe0 2024-06-29T15:57:50.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.623+0000 7f4ec5ffb640 1 -- 172.21.15.81:0/1980201938 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 131) ==== 98056+0+0 (secure 0 0 0) 0x7f4ec8018070 con 0x7f4ed00cefe0 2024-06-29T15:57:50.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.623+0000 7f4ec5ffb640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4eb4076170 0x7f4eb4078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.624+0000 7f4ec5ffb640 1 -- 172.21.15.81:0/1980201938 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f4ec8097530 con 0x7f4ed00cefe0 2024-06-29T15:57:50.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.624+0000 7f4ed57d2640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4eb4076170 0x7f4eb4078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.624+0000 7f4ed57d2640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4eb4076170 0x7f4eb4078680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc007dc0 tx=0x7f4ecc0072f0 comp rx=0 tx=0).ready entity=mgr.10348 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:50.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.628+0000 7f4ec5ffb640 1 -- 172.21.15.81:0/1980201938 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4ed0070cd0 con 0x7f4ed00cefe0 2024-06-29T15:57:50.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.657+0000 7f4ec5ffb640 1 -- 172.21.15.81:0/1980201938 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7f4ec805c790 con 0x7f4ed00cefe0 2024-06-29T15:57:50.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.835+0000 7f4ed7a36640 1 -- 172.21.15.81:0/1980201938 --> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4ed0073c40 con 0x7f4eb4076170 2024-06-29T15:57:50.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.841+0000 7f4ec5ffb640 1 -- 172.21.15.81:0/1980201938 <== mgr.10348 v2:172.21.15.167:6832/2452570162 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+88 (secure 0 0 0) 0x7f4ed0073c40 con 0x7f4eb4076170 2024-06-29T15:57:50.842 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:50.842 INFO:teuthology.orchestra.run.smithi081.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-06-29T15:57:50.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 -- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4eb4076170 msgr2=0x7f4eb4078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4eb4076170 0x7f4eb4078680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc007dc0 tx=0x7f4ecc0072f0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 -- 172.21.15.81:0/1980201938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ed00cefe0 msgr2=0x7f4ed015e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ed00cefe0 0x7f4ed015e940 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800b410 tx=0x7f4ec800b8d0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 -- 172.21.15.81:0/1980201938 shutdown_connections 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4eb4076170 0x7f4eb4078680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ed00cefe0 0x7f4ed015e940 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ed0072ef0 0x7f4ed00b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 --2- 172.21.15.81:0/1980201938 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ed006c2f0 0x7f4ed01f7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 -- 172.21.15.81:0/1980201938 >> 172.21.15.81:0/1980201938 conn(0x7f4ed00b2580 msgr2=0x7f4ed015dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 -- 172.21.15.81:0/1980201938 shutdown_connections 2024-06-29T15:57:50.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.844+0000 7f4ed7a36640 1 -- 172.21.15.81:0/1980201938 wait complete. 2024-06-29T15:57:50.856 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2024-06-29T15:57:50.856 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2024-06-29T15:57:50.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.972+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/1242958303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb440af730 msgr2=0x7ffb440afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.972+0000 7ffb4b4e4640 1 --2- 172.21.15.81:0/1242958303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb440af730 0x7ffb440afb40 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c005fd0 tx=0x7ffb3c02aa00 comp rx=0 tx=0).stop 2024-06-29T15:57:50.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.972+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/1242958303 shutdown_connections 2024-06-29T15:57:50.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.972+0000 7ffb4b4e4640 1 --2- 172.21.15.81:0/1242958303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb440af730 0x7ffb440afb40 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.972+0000 7ffb4b4e4640 1 --2- 172.21.15.81:0/1242958303 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb441555e0 0x7ffb4415bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.972+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/1242958303 >> 172.21.15.81:0/1242958303 conn(0x7ffb440d08f0 msgr2=0x7ffb440d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:50.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.972+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/1242958303 shutdown_connections 2024-06-29T15:57:50.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.973+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/1242958303 wait complete. 2024-06-29T15:57:50.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.974+0000 7ffb4b4e4640 1 Processor -- start 2024-06-29T15:57:50.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.974+0000 7ffb4b4e4640 1 -- start start 2024-06-29T15:57:50.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.975+0000 7ffb4b4e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb440708f0 0x7ffb4415d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb4b4e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb440af730 0x7ffb441f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb4b4e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb441555e0 0x7ffb441f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb4b4e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4406ff10 con 0x7ffb440708f0 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb4b4e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4406fd90 con 0x7ffb440af730 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb4b4e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ffb44070090 con 0x7ffb441555e0 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb49280640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb440708f0 0x7ffb4415d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb49280640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb440708f0 0x7ffb4415d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35178/0 (socket says 172.21.15.81:35178) 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb49280640 1 -- 172.21.15.81:0/3654261670 learned_addr learned my addr 172.21.15.81:0/3654261670 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb49280640 1 -- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb441555e0 msgr2=0x7ffb441f6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb49280640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb441555e0 0x7ffb441f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb49280640 1 -- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb440af730 msgr2=0x7ffb441f6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb48a7f640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb440af730 0x7ffb441f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb49280640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb440af730 0x7ffb441f6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb49280640 1 -- 172.21.15.81:0/3654261670 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb441408b0 con 0x7ffb440708f0 2024-06-29T15:57:50.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb49280640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb440708f0 0x7ffb4415d890 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7ffb4000d780 tx=0x7ffb4000dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:50.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb39ffb640 1 -- 172.21.15.81:0/3654261670 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40017070 con 0x7ffb440708f0 2024-06-29T15:57:50.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb39ffb640 1 -- 172.21.15.81:0/3654261670 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb40009b10 con 0x7ffb440708f0 2024-06-29T15:57:50.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/3654261670 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb44140b90 con 0x7ffb440708f0 2024-06-29T15:57:50.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.976+0000 7ffb39ffb640 1 -- 172.21.15.81:0/3654261670 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb40010460 con 0x7ffb440708f0 2024-06-29T15:57:50.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.977+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/3654261670 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb44141150 con 0x7ffb440708f0 2024-06-29T15:57:50.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.978+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/3654261670 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb440afb40 con 0x7ffb440708f0 2024-06-29T15:57:50.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.978+0000 7ffb39ffb640 1 -- 172.21.15.81:0/3654261670 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7ffb4001a070 con 0x7ffb440708f0 2024-06-29T15:57:50.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.979+0000 7ffb39ffb640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb28076560 0x7ffb28078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:50.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.979+0000 7ffb39ffb640 1 -- 172.21.15.81:0/3654261670 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7ffb40098330 con 0x7ffb440708f0 2024-06-29T15:57:50.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.979+0000 7ffb48a7f640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb28076560 0x7ffb28078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:50.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.980+0000 7ffb48a7f640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb28076560 0x7ffb28078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb34006fd0 tx=0x7ffb34008040 comp rx=0 tx=0).ready entity=mgr.10348 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:50.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:50.983+0000 7ffb39ffb640 1 -- 172.21.15.81:0/3654261670 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffb440afb40 con 0x7ffb440708f0 2024-06-29T15:57:51.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.182+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/3654261670 --> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffb44140660 con 0x7ffb28076560 2024-06-29T15:57:51.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.189+0000 7ffb39ffb640 1 -- 172.21.15.81:0/3654261670 <== mgr.10348 v2:172.21.15.167:6832/2452570162 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+353 (secure 0 0 0) 0x7ffb44140660 con 0x7ffb28076560 2024-06-29T15:57:51.189 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:51.189 INFO:teuthology.orchestra.run.smithi081.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:57:49", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-06-29T15:57:51.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb28076560 msgr2=0x7ffb28078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb28076560 0x7ffb28078a70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb34006fd0 tx=0x7ffb34008040 comp rx=0 tx=0).stop 2024-06-29T15:57:51.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/3654261670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb440708f0 msgr2=0x7ffb4415d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb440708f0 0x7ffb4415d890 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7ffb4000d780 tx=0x7ffb4000dc40 comp rx=0 tx=0).stop 2024-06-29T15:57:51.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/3654261670 shutdown_connections 2024-06-29T15:57:51.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb28076560 0x7ffb28078a70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb441555e0 0x7ffb441f6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb440af730 0x7ffb441f6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 --2- 172.21.15.81:0/3654261670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb440708f0 0x7ffb4415d890 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/3654261670 >> 172.21.15.81:0/3654261670 conn(0x7ffb440d08f0 msgr2=0x7ffb4415a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:51.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/3654261670 shutdown_connections 2024-06-29T15:57:51.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.191+0000 7ffb4b4e4640 1 -- 172.21.15.81:0/3654261670 wait complete. 2024-06-29T15:57:51.203 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2024-06-29T00:00:00", "created": "2024-06-29T15:57:49", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2024-06-29T15:57:51.203 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:51.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.322+0000 7f73a1481640 1 -- 172.21.15.81:0/2283665597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7388003a50 msgr2=0x7f7388003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.322+0000 7f73a1481640 1 --2- 172.21.15.81:0/2283665597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7388003a50 0x7f7388003ec0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f738c00bb20 tx=0x7f7388005830 comp rx=0 tx=0).stop 2024-06-29T15:57:51.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.322+0000 7f73a1481640 1 -- 172.21.15.81:0/2283665597 shutdown_connections 2024-06-29T15:57:51.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.322+0000 7f73a1481640 1 --2- 172.21.15.81:0/2283665597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7388003a50 0x7f7388003ec0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.322+0000 7f73a1481640 1 -- 172.21.15.81:0/2283665597 >> 172.21.15.81:0/2283665597 conn(0x7f739c09cf30 msgr2=0x7f739c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:51.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.322+0000 7f73a1481640 1 -- 172.21.15.81:0/2283665597 shutdown_connections 2024-06-29T15:57:51.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.322+0000 7f73a1481640 1 -- 172.21.15.81:0/2283665597 wait complete. 2024-06-29T15:57:51.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.324+0000 7f73a1481640 1 Processor -- start 2024-06-29T15:57:51.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.324+0000 7f73a1481640 1 -- start start 2024-06-29T15:57:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.325+0000 7f73a1481640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7388003a50 0x7f739c0a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.325+0000 7f73a1481640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f739c15d7c0 0x7f739c152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.325+0000 7f73a1481640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f739c0cd490 0x7f739c0cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.325+0000 7f73a1481640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f739c0d0810 con 0x7f739c15d7c0 2024-06-29T15:57:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.325+0000 7f73a1481640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f739c0d0690 con 0x7f7388003a50 2024-06-29T15:57:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.325+0000 7f73a1481640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f739c0d0990 con 0x7f739c0cd490 2024-06-29T15:57:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73927fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f739c15d7c0 0x7f739c152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73927fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f739c15d7c0 0x7f739c152100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35198/0 (socket says 172.21.15.81:35198) 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73927fc640 1 -- 172.21.15.81:0/4166294240 learned_addr learned my addr 172.21.15.81:0/4166294240 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f739affd640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7388003a50 0x7f739c0a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f739b7fe640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f739c0cd490 0x7f739c0cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73927fc640 1 -- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f739c0cd490 msgr2=0x7f739c0cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73927fc640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f739c0cd490 0x7f739c0cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73927fc640 1 -- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7388003a50 msgr2=0x7f739c0a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73927fc640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7388003a50 0x7f739c0a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73927fc640 1 -- 172.21.15.81:0/4166294240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c0cdde0 con 0x7f739c15d7c0 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f739affd640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7388003a50 0x7f739c0a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73927fc640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f739c15d7c0 0x7f739c152100 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f7384003fb0 tx=0x7f73840016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:51.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f7393fff640 1 -- 172.21.15.81:0/4166294240 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7384015070 con 0x7f739c15d7c0 2024-06-29T15:57:51.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f7393fff640 1 -- 172.21.15.81:0/4166294240 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7384011900 con 0x7f739c15d7c0 2024-06-29T15:57:51.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73a1481640 1 -- 172.21.15.81:0/4166294240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739c0b08c0 con 0x7f739c15d7c0 2024-06-29T15:57:51.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f7393fff640 1 -- 172.21.15.81:0/4166294240 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7384011d50 con 0x7f739c15d7c0 2024-06-29T15:57:51.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.326+0000 7f73a1481640 1 -- 172.21.15.81:0/4166294240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f739c0a12f0 con 0x7f739c15d7c0 2024-06-29T15:57:51.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.327+0000 7f73a1481640 1 -- 172.21.15.81:0/4166294240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f739c0d1a80 con 0x7f739c15d7c0 2024-06-29T15:57:51.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.328+0000 7f7393fff640 1 -- 172.21.15.81:0/4166294240 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7f7384018070 con 0x7f739c15d7c0 2024-06-29T15:57:51.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.328+0000 7f7393fff640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f737c076560 0x7f737c078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.328+0000 7f7393fff640 1 -- 172.21.15.81:0/4166294240 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f7384098ec0 con 0x7f739c15d7c0 2024-06-29T15:57:51.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.329+0000 7f739affd640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f737c076560 0x7f737c078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.329+0000 7f739affd640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f737c076560 0x7f737c078a70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f738c036000 tx=0x7f738c0046c0 comp rx=0 tx=0).ready entity=mgr.10348 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:51.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.333+0000 7f7393fff640 1 -- 172.21.15.81:0/4166294240 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f739c0cdde0 con 0x7f739c15d7c0 2024-06-29T15:57:51.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.538+0000 7f73a1481640 1 -- 172.21.15.81:0/4166294240 --> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) -- 0x7f739c0a0da0 con 0x7f737c076560 2024-06-29T15:57:51.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.597+0000 7f7393fff640 1 -- 172.21.15.81:0/4166294240 <== mgr.10348 v2:172.21.15.167:6832/2452570162 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+59 (secure 0 0 0) 0x7f739c0a0da0 con 0x7f737c076560 2024-06-29T15:57:51.598 INFO:teuthology.orchestra.run.smithi081.stdout:Retention removed from path mandatory_fs_argument_test_dir 2024-06-29T15:57:51.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 -- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f737c076560 msgr2=0x7f737c078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f737c076560 0x7f737c078a70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f738c036000 tx=0x7f738c0046c0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 -- 172.21.15.81:0/4166294240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f739c15d7c0 msgr2=0x7f739c152100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f739c15d7c0 0x7f739c152100 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f7384003fb0 tx=0x7f73840016f0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 -- 172.21.15.81:0/4166294240 shutdown_connections 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f737c076560 0x7f737c078a70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f739c0cd490 0x7f739c0cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f739c15d7c0 0x7f739c152100 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 --2- 172.21.15.81:0/4166294240 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7388003a50 0x7f739c0a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 -- 172.21.15.81:0/4166294240 >> 172.21.15.81:0/4166294240 conn(0x7f739c09cf30 msgr2=0x7f739c06d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 -- 172.21.15.81:0/4166294240 shutdown_connections 2024-06-29T15:57:51.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.600+0000 7f73a1481640 1 -- 172.21.15.81:0/4166294240 wait complete. 2024-06-29T15:57:51.612 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2024-06-29T15:57:51.612 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:51.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.730+0000 7f4bbb763640 1 -- 172.21.15.81:0/4228423350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bb4074070 msgr2=0x7f4bb4074480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.730+0000 7f4bbb763640 1 --2- 172.21.15.81:0/4228423350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bb4074070 0x7f4bb4074480 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40739b0 tx=0x7f4ba802a410 comp rx=0 tx=0).stop 2024-06-29T15:57:51.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.731+0000 7f4bbb763640 1 -- 172.21.15.81:0/4228423350 shutdown_connections 2024-06-29T15:57:51.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.731+0000 7f4bbb763640 1 --2- 172.21.15.81:0/4228423350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bb4074070 0x7f4bb4074480 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.731+0000 7f4bbb763640 1 --2- 172.21.15.81:0/4228423350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bb40b04b0 0x7f4bb4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.731+0000 7f4bbb763640 1 -- 172.21.15.81:0/4228423350 >> 172.21.15.81:0/4228423350 conn(0x7f4bb4072970 msgr2=0x7f4bb40b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:51.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.731+0000 7f4bbb763640 1 -- 172.21.15.81:0/4228423350 shutdown_connections 2024-06-29T15:57:51.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.731+0000 7f4bbb763640 1 -- 172.21.15.81:0/4228423350 wait complete. 2024-06-29T15:57:51.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.732+0000 7f4bbb763640 1 Processor -- start 2024-06-29T15:57:51.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.732+0000 7f4bbb763640 1 -- start start 2024-06-29T15:57:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bbb763640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bb406c2f0 0x7f4bb40cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bbb763640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bb4074070 0x7f4bb40cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bbb763640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bb40b04b0 0x7f4bb409bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bbb763640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb40af6f0 con 0x7f4bb4074070 2024-06-29T15:57:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bbb763640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb40af570 con 0x7f4bb406c2f0 2024-06-29T15:57:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bbb763640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb40af870 con 0x7f4bb40b04b0 2024-06-29T15:57:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb8cfe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bb4074070 0x7f4bb40cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb8cfe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bb4074070 0x7f4bb40cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35222/0 (socket says 172.21.15.81:35222) 2024-06-29T15:57:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb8cfe640 1 -- 172.21.15.81:0/997537299 learned_addr learned my addr 172.21.15.81:0/997537299 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb9d00640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bb40b04b0 0x7f4bb409bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb94ff640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bb406c2f0 0x7f4bb40cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb8cfe640 1 -- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bb40b04b0 msgr2=0x7f4bb409bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb8cfe640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bb40b04b0 0x7f4bb409bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb8cfe640 1 -- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bb406c2f0 msgr2=0x7f4bb40cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb8cfe640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bb406c2f0 0x7f4bb40cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb8cfe640 1 -- 172.21.15.81:0/997537299 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb409df80 con 0x7f4bb4074070 2024-06-29T15:57:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb9d00640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bb40b04b0 0x7f4bb409bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb94ff640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bb406c2f0 0x7f4bb40cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:51.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb8cfe640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bb4074070 0x7f4bb40cddb0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80099c0 tx=0x7f4ba80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:51.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb1ffb640 1 -- 172.21.15.81:0/997537299 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba803e030 con 0x7f4bb4074070 2024-06-29T15:57:51.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bb1ffb640 1 -- 172.21.15.81:0/997537299 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ba80047a0 con 0x7f4bb4074070 2024-06-29T15:57:51.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.734+0000 7f4bbb763640 1 -- 172.21.15.81:0/997537299 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb40cc7d0 con 0x7f4bb4074070 2024-06-29T15:57:51.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.735+0000 7f4bb1ffb640 1 -- 172.21.15.81:0/997537299 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8035040 con 0x7f4bb4074070 2024-06-29T15:57:51.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.735+0000 7f4bbb763640 1 -- 172.21.15.81:0/997537299 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb40ccd90 con 0x7f4bb4074070 2024-06-29T15:57:51.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.736+0000 7f4bbb763640 1 -- 172.21.15.81:0/997537299 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb409d750 con 0x7f4bb4074070 2024-06-29T15:57:51.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.737+0000 7f4bb1ffb640 1 -- 172.21.15.81:0/997537299 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7f4ba8041030 con 0x7f4bb4074070 2024-06-29T15:57:51.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.737+0000 7f4bb1ffb640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4ba0076590 0x7f4ba0078aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:51.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.737+0000 7f4bb1ffb640 1 -- 172.21.15.81:0/997537299 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f4ba80b88e0 con 0x7f4bb4074070 2024-06-29T15:57:51.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.737+0000 7f4bb94ff640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4ba0076590 0x7f4ba0078aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:51.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.738+0000 7f4bb94ff640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4ba0076590 0x7f4ba0078aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009890 tx=0x7f4bac008040 comp rx=0 tx=0).ready entity=mgr.10348 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:51.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.741+0000 7f4bb1ffb640 1 -- 172.21.15.81:0/997537299 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4ba8081960 con 0x7f4bb4074070 2024-06-29T15:57:51.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:51.944+0000 7f4bbb763640 1 -- 172.21.15.81:0/997537299 --> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) -- 0x7f4bb406cfd0 con 0x7f4ba0076590 2024-06-29T15:57:52.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.001+0000 7f4bb1ffb640 1 -- 172.21.15.81:0/997537299 <== mgr.10348 v2:172.21.15.167:6832/2452570162 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+61 (secure 0 0 0) 0x7f4bb406cfd0 con 0x7f4ba0076590 2024-06-29T15:57:52.001 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2024-06-29T15:57:52.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 -- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4ba0076590 msgr2=0x7f4ba0078aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4ba0076590 0x7f4ba0078aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009890 tx=0x7f4bac008040 comp rx=0 tx=0).stop 2024-06-29T15:57:52.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 -- 172.21.15.81:0/997537299 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bb4074070 msgr2=0x7f4bb40cddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bb4074070 0x7f4bb40cddb0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80099c0 tx=0x7f4ba80039e0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 -- 172.21.15.81:0/997537299 shutdown_connections 2024-06-29T15:57:52.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f4ba0076590 0x7f4ba0078aa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bb40b04b0 0x7f4bb409bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bb4074070 0x7f4bb40cddb0 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 --2- 172.21.15.81:0/997537299 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bb406c2f0 0x7f4bb40cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 -- 172.21.15.81:0/997537299 >> 172.21.15.81:0/997537299 conn(0x7f4bb4072970 msgr2=0x7f4bb406f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:52.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 -- 172.21.15.81:0/997537299 shutdown_connections 2024-06-29T15:57:52.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.003+0000 7f4bbb763640 1 -- 172.21.15.81:0/997537299 wait complete. 2024-06-29T15:57:52.014 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2024-06-29T15:57:52.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2024-06-29T15:57:52.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.134+0000 7faf79a66640 1 -- 172.21.15.81:0/440003175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf5c004800 msgr2=0x7faf5c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.134+0000 7faf79a66640 1 --2- 172.21.15.81:0/440003175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf5c004800 0x7faf5c004c10 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7faf68009b50 tx=0x7faf5c005830 comp rx=0 tx=0).stop 2024-06-29T15:57:52.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.134+0000 7faf73fff640 1 -- 172.21.15.81:0/440003175 reap_dead start 2024-06-29T15:57:52.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.134+0000 7faf79a66640 1 -- 172.21.15.81:0/440003175 shutdown_connections 2024-06-29T15:57:52.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.134+0000 7faf79a66640 1 -- 172.21.15.81:0/440003175 >> 172.21.15.81:0/440003175 conn(0x7faf7406cd60 msgr2=0x7faf740b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:52.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.134+0000 7faf79a66640 1 -- 172.21.15.81:0/440003175 shutdown_connections 2024-06-29T15:57:52.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.134+0000 7faf79a66640 1 -- 172.21.15.81:0/440003175 wait complete. 2024-06-29T15:57:52.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.135+0000 7faf79a66640 1 Processor -- start 2024-06-29T15:57:52.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.136+0000 7faf79a66640 1 -- start start 2024-06-29T15:57:52.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf79a66640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5c004800 0x7faf74138ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf79a66640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf7406e830 0x7faf7406ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf79a66640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf740ad8d0 0x7faf740adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf79a66640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf74072ab0 con 0x7faf7406e830 2024-06-29T15:57:52.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf79a66640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faf74072930 con 0x7faf5c004800 2024-06-29T15:57:52.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf79a66640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faf74072c30 con 0x7faf740ad8d0 2024-06-29T15:57:52.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf72ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf7406e830 0x7faf7406ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf72ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf7406e830 0x7faf7406ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35240/0 (socket says 172.21.15.81:35240) 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf72ffd640 1 -- 172.21.15.81:0/4273156167 learned_addr learned my addr 172.21.15.81:0/4273156167 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf73fff640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf740ad8d0 0x7faf740adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf737fe640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5c004800 0x7faf74138ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf72ffd640 1 -- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf740ad8d0 msgr2=0x7faf740adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf72ffd640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf740ad8d0 0x7faf740adce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf72ffd640 1 -- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5c004800 msgr2=0x7faf74138ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf72ffd640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5c004800 0x7faf74138ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf72ffd640 1 -- 172.21.15.81:0/4273156167 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf7415e4d0 con 0x7faf7406e830 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.137+0000 7faf72ffd640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf7406e830 0x7faf7406ec40 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7faf68009a70 tx=0x7faf68033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:52.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.138+0000 7faf5bfff640 1 -- 172.21.15.81:0/4273156167 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf68043070 con 0x7faf7406e830 2024-06-29T15:57:52.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.138+0000 7faf5bfff640 1 -- 172.21.15.81:0/4273156167 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf68004340 con 0x7faf7406e830 2024-06-29T15:57:52.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.138+0000 7faf79a66640 1 -- 172.21.15.81:0/4273156167 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf74139780 con 0x7faf7406e830 2024-06-29T15:57:52.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.138+0000 7faf5bfff640 1 -- 172.21.15.81:0/4273156167 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf68038040 con 0x7faf7406e830 2024-06-29T15:57:52.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.138+0000 7faf79a66640 1 -- 172.21.15.81:0/4273156167 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf74139d40 con 0x7faf7406e830 2024-06-29T15:57:52.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.139+0000 7faf79a66640 1 -- 172.21.15.81:0/4273156167 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf7406fb70 con 0x7faf7406e830 2024-06-29T15:57:52.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.139+0000 7faf5bfff640 1 -- 172.21.15.81:0/4273156167 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7faf68046070 con 0x7faf7406e830 2024-06-29T15:57:52.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.140+0000 7faf5bfff640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7faf60076560 0x7faf60078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.140+0000 7faf5bfff640 1 -- 172.21.15.81:0/4273156167 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7faf680b8760 con 0x7faf7406e830 2024-06-29T15:57:52.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.140+0000 7faf737fe640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7faf60076560 0x7faf60078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.141+0000 7faf737fe640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7faf60076560 0x7faf60078a70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faf64001010 tx=0x7faf64000f60 comp rx=0 tx=0).ready entity=mgr.10348 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:52.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.144+0000 7faf5bfff640 1 -- 172.21.15.81:0/4273156167 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faf68081140 con 0x7faf7406e830 2024-06-29T15:57:52.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.347+0000 7faf79a66640 1 -- 172.21.15.81:0/4273156167 --> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) -- 0x7faf74139f80 con 0x7faf60076560 2024-06-29T15:57:52.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.412+0000 7faf5bfff640 1 -- 172.21.15.81:0/4273156167 <== mgr.10348 v2:172.21.15.167:6832/2452570162 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+57 (secure 0 0 0) 0x7faf64004d90 con 0x7faf60076560 2024-06-29T15:57:52.413 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2024-06-29T15:57:52.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 -- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7faf60076560 msgr2=0x7faf60078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7faf60076560 0x7faf60078a70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faf64001010 tx=0x7faf64000f60 comp rx=0 tx=0).stop 2024-06-29T15:57:52.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 -- 172.21.15.81:0/4273156167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf7406e830 msgr2=0x7faf7406ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf7406e830 0x7faf7406ec40 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7faf68009a70 tx=0x7faf68033670 comp rx=0 tx=0).stop 2024-06-29T15:57:52.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 -- 172.21.15.81:0/4273156167 shutdown_connections 2024-06-29T15:57:52.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7faf60076560 0x7faf60078a70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faf740ad8d0 0x7faf740adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf7406e830 0x7faf7406ec40 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 --2- 172.21.15.81:0/4273156167 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faf5c004800 0x7faf74138ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 -- 172.21.15.81:0/4273156167 >> 172.21.15.81:0/4273156167 conn(0x7faf7406cd60 msgr2=0x7faf74140350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:52.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 -- 172.21.15.81:0/4273156167 shutdown_connections 2024-06-29T15:57:52.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.414+0000 7faf79a66640 1 -- 172.21.15.81:0/4273156167 wait complete. 2024-06-29T15:57:52.426 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2024-06-29T15:57:52.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:52.427 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-06-29T15:57:52.444 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:57:52.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.600+0000 7f4472226640 1 -- 172.21.15.81:0/2519306331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f446c0b5250 msgr2=0x7f446c0d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.600+0000 7f4472226640 1 --2- 172.21.15.81:0/2519306331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f446c0b5250 0x7f446c0d1fc0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f4464005fd0 tx=0x7f446402aa90 comp rx=0 tx=0).stop 2024-06-29T15:57:52.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.600+0000 7f4472226640 1 -- 172.21.15.81:0/2519306331 shutdown_connections 2024-06-29T15:57:52.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.600+0000 7f4472226640 1 --2- 172.21.15.81:0/2519306331 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f446c0d0b90 0x7f446c0d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.600+0000 7f4472226640 1 --2- 172.21.15.81:0/2519306331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f446c0b5250 0x7f446c0d1fc0 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.600+0000 7f4472226640 1 -- 172.21.15.81:0/2519306331 >> 172.21.15.81:0/2519306331 conn(0x7f446c09cf30 msgr2=0x7f446c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:52.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.600+0000 7f4472226640 1 -- 172.21.15.81:0/2519306331 shutdown_connections 2024-06-29T15:57:52.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.600+0000 7f4472226640 1 -- 172.21.15.81:0/2519306331 wait complete. 2024-06-29T15:57:52.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.602+0000 7f4472226640 1 Processor -- start 2024-06-29T15:57:52.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.602+0000 7f4472226640 1 -- start start 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f4472226640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f446c0b5250 0x7f446c15cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f4472226640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0b9040 0x7f446c0c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f4472226640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f446c0ca050 0x7f446c0a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f4472226640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f446c0d2500 con 0x7f446c0ca050 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f4472226640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f446c0b8590 con 0x7f446c0b9040 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f4472226640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f446c0d2680 con 0x7f446c0b5250 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f446bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f446c0ca050 0x7f446c0a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f446bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f446c0ca050 0x7f446c0a0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35262/0 (socket says 172.21.15.81:35262) 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f446bfff640 1 -- 172.21.15.81:0/4028632422 learned_addr learned my addr 172.21.15.81:0/4028632422 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f446b7fe640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f446c0b5250 0x7f446c15cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.603+0000 7f446affd640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0b9040 0x7f446c0c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f446bfff640 1 -- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f446c0b5250 msgr2=0x7f446c15cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f446bfff640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f446c0b5250 0x7f446c15cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f446bfff640 1 -- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0b9040 msgr2=0x7f446c0c9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f446bfff640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0b9040 0x7f446c0c9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f446bfff640 1 -- 172.21.15.81:0/4028632422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446c0ca680 con 0x7f446c0ca050 2024-06-29T15:57:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f446affd640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0b9040 0x7f446c0c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f446bfff640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f446c0ca050 0x7f446c0a0fe0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f4464009d90 tx=0x7f4464003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f4463fff640 1 -- 172.21.15.81:0/4028632422 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464044070 con 0x7f446c0ca050 2024-06-29T15:57:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f4463fff640 1 -- 172.21.15.81:0/4028632422 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4464004250 con 0x7f446c0ca050 2024-06-29T15:57:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f4472226640 1 -- 172.21.15.81:0/4028632422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f446c0c7f90 con 0x7f446c0ca050 2024-06-29T15:57:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f4463fff640 1 -- 172.21.15.81:0/4028632422 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464004660 con 0x7f446c0ca050 2024-06-29T15:57:52.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.604+0000 7f4472226640 1 -- 172.21.15.81:0/4028632422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f446c0c8510 con 0x7f446c0ca050 2024-06-29T15:57:52.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.605+0000 7f4472226640 1 -- 172.21.15.81:0/4028632422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f446c0d2bc0 con 0x7f446c0ca050 2024-06-29T15:57:52.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.606+0000 7f4463fff640 1 -- 172.21.15.81:0/4028632422 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 132) ==== 98423+0+0 (secure 0 0 0) 0x7f4464047070 con 0x7f446c0ca050 2024-06-29T15:57:52.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.606+0000 7f4463fff640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f444c076560 0x7f444c078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:52.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.606+0000 7f4463fff640 1 -- 172.21.15.81:0/4028632422 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f44640b91d0 con 0x7f446c0ca050 2024-06-29T15:57:52.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.606+0000 7f446b7fe640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f444c076560 0x7f444c078a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:52.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.607+0000 7f446b7fe640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f444c076560 0x7f444c078a70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f445c006fd0 tx=0x7f445c009060 comp rx=0 tx=0).ready entity=mgr.10348 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:52.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.611+0000 7f4463fff640 1 -- 172.21.15.81:0/4028632422 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f446c0c7f90 con 0x7f446c0ca050 2024-06-29T15:57:52.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.668+0000 7f4463fff640 1 -- 172.21.15.81:0/4028632422 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 133) ==== 98479+0+0 (secure 0 0 0) 0x7f446407dcb0 con 0x7f446c0ca050 2024-06-29T15:57:52.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:52.843+0000 7f4472226640 1 -- 172.21.15.81:0/4028632422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f446c0d0480 con 0x7f446c0ca050 2024-06-29T15:57:53.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.667+0000 7f4463fff640 1 -- 172.21.15.81:0/4028632422 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v134) ==== 93+0+0 (secure 0 0 0) 0x7f4464032030 con 0x7f446c0ca050 2024-06-29T15:57:53.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.670+0000 7f4472226640 1 -- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f444c076560 msgr2=0x7f444c078a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.670+0000 7f4472226640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f444c076560 0x7f444c078a70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f445c006fd0 tx=0x7f445c009060 comp rx=0 tx=0).stop 2024-06-29T15:57:53.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.670+0000 7f4472226640 1 -- 172.21.15.81:0/4028632422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f446c0ca050 msgr2=0x7f446c0a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:53.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.671+0000 7f4463fff640 1 -- 172.21.15.81:0/4028632422 <== mon.0 v2:172.21.15.81:3300/0 9 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f446409b520 con 0x7f446c0ca050 2024-06-29T15:57:53.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.671+0000 7f4472226640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f446c0ca050 0x7f446c0a0fe0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f4464009d90 tx=0x7f4464003ce0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.672+0000 7f4472226640 1 -- 172.21.15.81:0/4028632422 shutdown_connections 2024-06-29T15:57:53.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.672+0000 7f4472226640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f444c076560 0x7f444c078a70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.672+0000 7f4472226640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f446c0ca050 0x7f446c0a0fe0 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.672+0000 7f4472226640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f446c0b9040 0x7f446c0c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.672+0000 7f4472226640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f446c0b5250 0x7f446c15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.672+0000 7f4472226640 1 -- 172.21.15.81:0/4028632422 >> 172.21.15.81:0/4028632422 conn(0x7f446c09cf30 msgr2=0x7f446c06df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:53.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.672+0000 7f4463fff640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f444c039960 0x7f444c039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:53.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.672+0000 7f4472226640 1 -- 172.21.15.81:0/4028632422 shutdown_connections 2024-06-29T15:57:53.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.672+0000 7f4472226640 1 --2- 172.21.15.81:0/4028632422 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f444c039960 0x7f444c039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:53.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:53.672+0000 7f4472226640 1 -- 172.21.15.81:0/4028632422 wait complete. 2024-06-29T15:57:53.687 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:57:53.687 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:53.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:57:53.793 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:57:53.793 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:53.793 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:57:53.793 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:53.809 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:57:53.809 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:53.866 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:54.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.021+0000 7eff8388f640 1 -- 172.21.15.81:0/1975905555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff7c0cccf0 msgr2=0x7eff7c0d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.021+0000 7eff8388f640 1 --2- 172.21.15.81:0/1975905555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff7c0cccf0 0x7eff7c0d0230 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7eff78005fd0 tx=0x7eff7802a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.021+0000 7eff8388f640 1 -- 172.21.15.81:0/1975905555 shutdown_connections 2024-06-29T15:57:54.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.021+0000 7eff8388f640 1 --2- 172.21.15.81:0/1975905555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff7c0cccf0 0x7eff7c0d0230 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.021+0000 7eff8388f640 1 --2- 172.21.15.81:0/1975905555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff7c0b3200 0x7eff7c0cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.021+0000 7eff8388f640 1 -- 172.21.15.81:0/1975905555 >> 172.21.15.81:0/1975905555 conn(0x7eff7c0b3680 msgr2=0x7eff7c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:54.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.021+0000 7eff8388f640 1 -- 172.21.15.81:0/1975905555 shutdown_connections 2024-06-29T15:57:54.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.021+0000 7eff8388f640 1 -- 172.21.15.81:0/1975905555 wait complete. 2024-06-29T15:57:54.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.024+0000 7eff8388f640 1 Processor -- start 2024-06-29T15:57:54.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.024+0000 7eff8388f640 1 -- start start 2024-06-29T15:57:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff8388f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff7c0b3200 0x7eff7c159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff8388f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff7c0cccf0 0x7eff7c1e3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff8388f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff7c0d0770 0x7eff7c1e3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff8388f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c0cedb0 con 0x7eff7c0cccf0 2024-06-29T15:57:54.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff8388f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7eff7c0cec30 con 0x7eff7c0d0770 2024-06-29T15:57:54.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff8388f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7eff7c0cef30 con 0x7eff7c0b3200 2024-06-29T15:57:54.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff80e2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff7c0cccf0 0x7eff7c1e3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff80e2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff7c0cccf0 0x7eff7c1e3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35298/0 (socket says 172.21.15.81:35298) 2024-06-29T15:57:54.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff80e2a640 1 -- 172.21.15.81:0/3264826 learned_addr learned my addr 172.21.15.81:0/3264826 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff8162b640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff7c0b3200 0x7eff7c159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff81e2c640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff7c0d0770 0x7eff7c1e3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff80e2a640 1 -- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff7c0b3200 msgr2=0x7eff7c159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff80e2a640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff7c0b3200 0x7eff7c159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff80e2a640 1 -- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff7c0d0770 msgr2=0x7eff7c1e3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff80e2a640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff7c0d0770 0x7eff7c1e3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff80e2a640 1 -- 172.21.15.81:0/3264826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7c0b20b0 con 0x7eff7c0cccf0 2024-06-29T15:57:54.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff80e2a640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff7c0cccf0 0x7eff7c1e3760 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7eff78034040 tx=0x7eff780038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:54.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff71ffb640 1 -- 172.21.15.81:0/3264826 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7803e030 con 0x7eff7c0cccf0 2024-06-29T15:57:54.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff71ffb640 1 -- 172.21.15.81:0/3264826 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff78003ca0 con 0x7eff7c0cccf0 2024-06-29T15:57:54.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.027+0000 7eff8388f640 1 -- 172.21.15.81:0/3264826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff7c0b2390 con 0x7eff7c0cccf0 2024-06-29T15:57:54.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.028+0000 7eff71ffb640 1 -- 172.21.15.81:0/3264826 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff78035040 con 0x7eff7c0cccf0 2024-06-29T15:57:54.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.028+0000 7eff8388f640 1 -- 172.21.15.81:0/3264826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff7c09c870 con 0x7eff7c0cccf0 2024-06-29T15:57:54.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.029+0000 7eff8388f640 1 -- 172.21.15.81:0/3264826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff7c0b28b0 con 0x7eff7c0cccf0 2024-06-29T15:57:54.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.030+0000 7eff71ffb640 1 -- 172.21.15.81:0/3264826 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7eff78041030 con 0x7eff7c0cccf0 2024-06-29T15:57:54.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.030+0000 7eff71ffb640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7eff600765a0 0x7eff60078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.031+0000 7eff8162b640 1 -- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7eff600765a0 msgr2=0x7eff60078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:54.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.031+0000 7eff71ffb640 1 -- 172.21.15.81:0/3264826 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7eff780b86b0 con 0x7eff7c0cccf0 2024-06-29T15:57:54.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.031+0000 7eff8162b640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7eff600765a0 0x7eff60078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:54.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.035+0000 7eff71ffb640 1 -- 172.21.15.81:0/3264826 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7eff78081030 con 0x7eff7c0cccf0 2024-06-29T15:57:54.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.224+0000 7eff8388f640 1 -- 172.21.15.81:0/3264826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff7c0cc7b0 con 0x7eff7c0cccf0 2024-06-29T15:57:54.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.225+0000 7eff71ffb640 1 -- 172.21.15.81:0/3264826 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) ==== 74+0+21172 (secure 0 0 0) 0x7eff78085330 con 0x7eff7c0cccf0 2024-06-29T15:57:54.226 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:54.226 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":218,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:48.595402+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.259110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:32.675276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3042354413":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1619575755":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/760717235":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/3781943725":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/376674682":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/3640842563":"2024-06-30T15:57:48.595362+0000","172.21.15.167:6833/2018755589":"2024-06-30T15:57:48.595362+0000","172.21.15.167:6832/2018755589":"2024-06-30T15:57:48.595362+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:54.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 -- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7eff600765a0 msgr2=0x7eff60078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:54.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7eff600765a0 0x7eff60078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 -- 172.21.15.81:0/3264826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff7c0cccf0 msgr2=0x7eff7c1e3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff7c0cccf0 0x7eff7c1e3760 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7eff78034040 tx=0x7eff780038a0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 -- 172.21.15.81:0/3264826 shutdown_connections 2024-06-29T15:57:54.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7eff600765a0 0x7eff60078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff7c0d0770 0x7eff7c1e3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff7c0cccf0 0x7eff7c1e3760 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 --2- 172.21.15.81:0/3264826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff7c0b3200 0x7eff7c159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 -- 172.21.15.81:0/3264826 >> 172.21.15.81:0/3264826 conn(0x7eff7c0b3680 msgr2=0x7eff7c0d1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:54.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 -- 172.21.15.81:0/3264826 shutdown_connections 2024-06-29T15:57:54.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.227+0000 7eff8388f640 1 -- 172.21.15.81:0/3264826 wait complete. 2024-06-29T15:57:54.240 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:57:19.937473+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:57:20.596779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:57:32.259110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '216', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:57:32.675276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '217', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:54.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:54.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.362+0000 7fb5481f5640 1 -- 172.21.15.81:0/3162992152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b7290 msgr2=0x7fb5400b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.362+0000 7fb5481f5640 1 --2- 172.21.15.81:0/3162992152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b7290 0x7fb5400b76a0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fb54008e230 tx=0x7fb53802aac0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.362+0000 7fb5481f5640 1 -- 172.21.15.81:0/3162992152 shutdown_connections 2024-06-29T15:57:54.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.362+0000 7fb5481f5640 1 --2- 172.21.15.81:0/3162992152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b7290 0x7fb5400b76a0 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.362+0000 7fb5481f5640 1 --2- 172.21.15.81:0/3162992152 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b5250 0x7fb5400d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.362+0000 7fb5481f5640 1 --2- 172.21.15.81:0/3162992152 >> v2:172.21.15.167:3300/0 conn(0x7fb5400b9040 0x7fb5400d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.362+0000 7fb5481f5640 1 -- 172.21.15.81:0/3162992152 >> 172.21.15.81:0/3162992152 conn(0x7fb54009cf30 msgr2=0x7fb5400d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:54.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.362+0000 7fb5481f5640 1 -- 172.21.15.81:0/3162992152 shutdown_connections 2024-06-29T15:57:54.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.362+0000 7fb5481f5640 1 -- 172.21.15.81:0/3162992152 wait complete. 2024-06-29T15:57:54.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.364+0000 7fb5481f5640 1 Processor -- start 2024-06-29T15:57:54.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.364+0000 7fb5481f5640 1 -- start start 2024-06-29T15:57:54.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb5481f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b5250 0x7fb54015d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb5481f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b7290 0x7fb5400ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb5481f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b9040 0x7fb5400ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb5481f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400b8020 con 0x7fb5400b9040 2024-06-29T15:57:54.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb5481f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb5400b7ea0 con 0x7fb5400b7290 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb5481f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb5400b81a0 con 0x7fb5400b5250 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb546792640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b9040 0x7fb5400ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb546792640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b9040 0x7fb5400ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35316/0 (socket says 172.21.15.81:35316) 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb546792640 1 -- 172.21.15.81:0/1194577063 learned_addr learned my addr 172.21.15.81:0/1194577063 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb545f91640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b5250 0x7fb54015d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb545790640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b7290 0x7fb5400ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb546792640 1 -- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b5250 msgr2=0x7fb54015d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb546792640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b5250 0x7fb54015d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb546792640 1 -- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b7290 msgr2=0x7fb5400ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb546792640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b7290 0x7fb5400ca060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.365+0000 7fb546792640 1 -- 172.21.15.81:0/1194577063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5400b0820 con 0x7fb5400b9040 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.366+0000 7fb546792640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b9040 0x7fb5400ca5a0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fb53802b0b0 tx=0x7fb538003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:54.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.366+0000 7fb5367fc640 1 -- 172.21.15.81:0/1194577063 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb538004040 con 0x7fb5400b9040 2024-06-29T15:57:54.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.366+0000 7fb5367fc640 1 -- 172.21.15.81:0/1194577063 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5380041e0 con 0x7fb5400b9040 2024-06-29T15:57:54.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.366+0000 7fb5481f5640 1 -- 172.21.15.81:0/1194577063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5400c8e40 con 0x7fb5400b9040 2024-06-29T15:57:54.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.366+0000 7fb5367fc640 1 -- 172.21.15.81:0/1194577063 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb538004610 con 0x7fb5400b9040 2024-06-29T15:57:54.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.366+0000 7fb5481f5640 1 -- 172.21.15.81:0/1194577063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54009ddb0 con 0x7fb5400b9040 2024-06-29T15:57:54.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.367+0000 7fb5481f5640 1 -- 172.21.15.81:0/1194577063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5400b76a0 con 0x7fb5400b9040 2024-06-29T15:57:54.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.367+0000 7fb5367fc640 1 -- 172.21.15.81:0/1194577063 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7fb538049430 con 0x7fb5400b9040 2024-06-29T15:57:54.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.368+0000 7fb5367fc640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7fb5240765a0 0x7fb524078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.368+0000 7fb545f91640 1 -- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7fb5240765a0 msgr2=0x7fb524078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:54.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.368+0000 7fb5367fc640 1 -- 172.21.15.81:0/1194577063 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7fb5380b8d20 con 0x7fb5400b9040 2024-06-29T15:57:54.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.368+0000 7fb545f91640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7fb5240765a0 0x7fb524078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:54.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.372+0000 7fb5367fc640 1 -- 172.21.15.81:0/1194577063 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb538032030 con 0x7fb5400b9040 2024-06-29T15:57:54.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.558+0000 7fb5481f5640 1 -- 172.21.15.81:0/1194577063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb5400d2b00 con 0x7fb5400b9040 2024-06-29T15:57:54.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.559+0000 7fb5367fc640 1 -- 172.21.15.81:0/1194577063 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v218) ==== 93+0+10 (secure 0 0 0) 0x7fb538081690 con 0x7fb5400b9040 2024-06-29T15:57:54.559 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:57:54.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 -- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7fb5240765a0 msgr2=0x7fb524078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:54.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7fb5240765a0 0x7fb524078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 -- 172.21.15.81:0/1194577063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b9040 msgr2=0x7fb5400ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b9040 0x7fb5400ca5a0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fb53802b0b0 tx=0x7fb538003c10 comp rx=0 tx=0).stop 2024-06-29T15:57:54.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 -- 172.21.15.81:0/1194577063 shutdown_connections 2024-06-29T15:57:54.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7fb5240765a0 0x7fb524078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5400b9040 0x7fb5400ca5a0 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb5400b7290 0x7fb5400ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 --2- 172.21.15.81:0/1194577063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb5400b5250 0x7fb54015d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 -- 172.21.15.81:0/1194577063 >> 172.21.15.81:0/1194577063 conn(0x7fb54009cf30 msgr2=0x7fb5400d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:54.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 -- 172.21.15.81:0/1194577063 shutdown_connections 2024-06-29T15:57:54.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.561+0000 7fb5481f5640 1 -- 172.21.15.81:0/1194577063 wait complete. 2024-06-29T15:57:54.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:57:54.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.693+0000 7f9302b2a640 1 -- 172.21.15.81:0/163807857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc073650 msgr2=0x7f92fc15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.693+0000 7f9302b2a640 1 --2- 172.21.15.81:0/163807857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc073650 0x7f92fc15b6e0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f92fc072cf0 tx=0x7f92e802a560 comp rx=0 tx=0).stop 2024-06-29T15:57:54.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.693+0000 7f9302b2a640 1 -- 172.21.15.81:0/163807857 shutdown_connections 2024-06-29T15:57:54.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.693+0000 7f9302b2a640 1 --2- 172.21.15.81:0/163807857 >> v2:172.21.15.167:3300/0 conn(0x7f92fc15bc20 0x7f92fc06ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.693+0000 7f9302b2a640 1 --2- 172.21.15.81:0/163807857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc073650 0x7f92fc15b6e0 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.693+0000 7f9302b2a640 1 -- 172.21.15.81:0/163807857 >> 172.21.15.81:0/163807857 conn(0x7f92fc0b2920 msgr2=0x7f92fc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:54.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.693+0000 7f9302b2a640 1 -- 172.21.15.81:0/163807857 shutdown_connections 2024-06-29T15:57:54.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.693+0000 7f9302b2a640 1 -- 172.21.15.81:0/163807857 wait complete. 2024-06-29T15:57:54.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.695+0000 7f9302b2a640 1 Processor -- start 2024-06-29T15:57:54.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.695+0000 7f9302b2a640 1 -- start start 2024-06-29T15:57:54.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.696+0000 7f9302b2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc0cd150 0x7f92fc149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.696+0000 7f9302b2a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92fc15bc20 0x7f92fc06c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.696+0000 7f9302b2a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92fc14f030 0x7f92fc06c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.696+0000 7f9302b2a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc06f450 con 0x7f92fc0cd150 2024-06-29T15:57:54.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.696+0000 7f9302b2a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc06f2d0 con 0x7f92fc15bc20 2024-06-29T15:57:54.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.696+0000 7f9302b2a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f92fc06f5d0 con 0x7f92fc14f030 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f93008c6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc0cd150 0x7f92fc149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f93008c6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc0cd150 0x7f92fc149450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35338/0 (socket says 172.21.15.81:35338) 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f93008c6640 1 -- 172.21.15.81:0/2170053085 learned_addr learned my addr 172.21.15.81:0/2170053085 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f92fbfff640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92fc15bc20 0x7f92fc06c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f93010c7640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92fc14f030 0x7f92fc06c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f93008c6640 1 -- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92fc14f030 msgr2=0x7f92fc06c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f93008c6640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92fc14f030 0x7f92fc06c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f93008c6640 1 -- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92fc15bc20 msgr2=0x7f92fc06c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f93008c6640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92fc15bc20 0x7f92fc06c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f93008c6640 1 -- 172.21.15.81:0/2170053085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92fc150e00 con 0x7f92fc0cd150 2024-06-29T15:57:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f93008c6640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc0cd150 0x7f92fc149450 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f92f000c3c0 tx=0x7f92f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:54.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f92f97fa640 1 -- 172.21.15.81:0/2170053085 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0017070 con 0x7f92fc0cd150 2024-06-29T15:57:54.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f92f97fa640 1 -- 172.21.15.81:0/2170053085 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92f00035f0 con 0x7f92fc0cd150 2024-06-29T15:57:54.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f9302b2a640 1 -- 172.21.15.81:0/2170053085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92fc1510e0 con 0x7f92fc0cd150 2024-06-29T15:57:54.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f92f97fa640 1 -- 172.21.15.81:0/2170053085 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0003a80 con 0x7f92fc0cd150 2024-06-29T15:57:54.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.697+0000 7f9302b2a640 1 -- 172.21.15.81:0/2170053085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92fc157060 con 0x7f92fc0cd150 2024-06-29T15:57:54.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.699+0000 7f9302b2a640 1 -- 172.21.15.81:0/2170053085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92fc150b60 con 0x7f92fc0cd150 2024-06-29T15:57:54.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.699+0000 7f92f97fa640 1 -- 172.21.15.81:0/2170053085 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f92f001a070 con 0x7f92fc0cd150 2024-06-29T15:57:54.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.699+0000 7f92f97fa640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f92e00765a0 0x7f92e0078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:54.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.700+0000 7f92fbfff640 1 -- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f92e00765a0 msgr2=0x7f92e0078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:54.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.700+0000 7f92f97fa640 1 -- 172.21.15.81:0/2170053085 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f92f00973a0 con 0x7f92fc0cd150 2024-06-29T15:57:54.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.700+0000 7f92fbfff640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f92e00765a0 0x7f92e0078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:54.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.704+0000 7f92f97fa640 1 -- 172.21.15.81:0/2170053085 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f92f005fe20 con 0x7f92fc0cd150 2024-06-29T15:57:54.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.890+0000 7f9302b2a640 1 -- 172.21.15.81:0/2170053085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f92fc15b6e0 con 0x7f92fc0cd150 2024-06-29T15:57:54.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.890+0000 7f92f97fa640 1 -- 172.21.15.81:0/2170053085 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v218) ==== 104+0+11 (secure 0 0 0) 0x7f92f0064120 con 0x7f92fc0cd150 2024-06-29T15:57:54.890 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:57:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.892+0000 7f9302b2a640 1 -- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f92e00765a0 msgr2=0x7f92e0078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.892+0000 7f9302b2a640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f92e00765a0 0x7f92e0078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.892+0000 7f9302b2a640 1 -- 172.21.15.81:0/2170053085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc0cd150 msgr2=0x7f92fc149450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.892+0000 7f9302b2a640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc0cd150 0x7f92fc149450 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f92f000c3c0 tx=0x7f92f000c880 comp rx=0 tx=0).stop 2024-06-29T15:57:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.892+0000 7f9302b2a640 1 -- 172.21.15.81:0/2170053085 shutdown_connections 2024-06-29T15:57:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.892+0000 7f9302b2a640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f92e00765a0 0x7f92e0078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.892+0000 7f9302b2a640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f92fc14f030 0x7f92fc06c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.892+0000 7f9302b2a640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f92fc15bc20 0x7f92fc06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.892+0000 7f9302b2a640 1 --2- 172.21.15.81:0/2170053085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92fc0cd150 0x7f92fc149450 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:54.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.893+0000 7f9302b2a640 1 -- 172.21.15.81:0/2170053085 >> 172.21.15.81:0/2170053085 conn(0x7f92fc0b2920 msgr2=0x7f92fc14a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:54.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.893+0000 7f9302b2a640 1 -- 172.21.15.81:0/2170053085 shutdown_connections 2024-06-29T15:57:54.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:54.893+0000 7f9302b2a640 1 -- 172.21.15.81:0/2170053085 wait complete. 2024-06-29T15:57:54.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:57:55.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.024+0000 7f57a75f0640 1 -- 172.21.15.81:0/2307579675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57a00b4f40 msgr2=0x7f57a00b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.024+0000 7f57a75f0640 1 --2- 172.21.15.81:0/2307579675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57a00b4f40 0x7f57a00b5350 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f57a0074b10 tx=0x7f579802a820 comp rx=0 tx=0).stop 2024-06-29T15:57:55.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.024+0000 7f57a75f0640 1 -- 172.21.15.81:0/2307579675 shutdown_connections 2024-06-29T15:57:55.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.024+0000 7f57a75f0640 1 --2- 172.21.15.81:0/2307579675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57a00b4f40 0x7f57a00b5350 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.024+0000 7f57a75f0640 1 --2- 172.21.15.81:0/2307579675 >> v2:172.21.15.167:3300/0 conn(0x7f57a00d0230 0x7f57a00d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.024+0000 7f57a75f0640 1 -- 172.21.15.81:0/2307579675 >> 172.21.15.81:0/2307579675 conn(0x7f57a00b3680 msgr2=0x7f57a00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:55.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.024+0000 7f57a75f0640 1 -- 172.21.15.81:0/2307579675 shutdown_connections 2024-06-29T15:57:55.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.024+0000 7f57a75f0640 1 -- 172.21.15.81:0/2307579675 wait complete. 2024-06-29T15:57:55.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.026+0000 7f57a75f0640 1 Processor -- start 2024-06-29T15:57:55.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.026+0000 7f57a75f0640 1 -- start start 2024-06-29T15:57:55.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a75f0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57a00b3200 0x7f57a0159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a75f0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57a00b4f40 0x7f57a0157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a75f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57a00d0230 0x7f57a0157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a75f0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f57a00b5c20 con 0x7f57a00d0230 2024-06-29T15:57:55.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a75f0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f57a00b5aa0 con 0x7f57a00b4f40 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a75f0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f57a00b5da0 con 0x7f57a00b3200 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a5b8d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57a00d0230 0x7f57a0157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a5b8d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57a00d0230 0x7f57a0157fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35360/0 (socket says 172.21.15.81:35360) 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a5b8d640 1 -- 172.21.15.81:0/1041540811 learned_addr learned my addr 172.21.15.81:0/1041540811 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a4b8b640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57a00b4f40 0x7f57a0157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.027+0000 7f57a538c640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57a00b3200 0x7f57a0159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f57a5b8d640 1 -- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57a00b3200 msgr2=0x7f57a0159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f57a5b8d640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57a00b3200 0x7f57a0159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f57a5b8d640 1 -- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57a00b4f40 msgr2=0x7f57a0157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f57a5b8d640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57a00b4f40 0x7f57a0157a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f57a5b8d640 1 -- 172.21.15.81:0/1041540811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a01e34f0 con 0x7f57a00d0230 2024-06-29T15:57:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f57a5b8d640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57a00d0230 0x7f57a0157fb0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f579802ae10 tx=0x7f5798032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f5795ffb640 1 -- 172.21.15.81:0/1041540811 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798037040 con 0x7f57a00d0230 2024-06-29T15:57:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f5795ffb640 1 -- 172.21.15.81:0/1041540811 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5798042070 con 0x7f57a00d0230 2024-06-29T15:57:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f5795ffb640 1 -- 172.21.15.81:0/1041540811 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798003e60 con 0x7f57a00d0230 2024-06-29T15:57:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f57a75f0640 1 -- 172.21.15.81:0/1041540811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a01e37d0 con 0x7f57a00d0230 2024-06-29T15:57:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.028+0000 7f57a75f0640 1 -- 172.21.15.81:0/1041540811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a01e3d90 con 0x7f57a00d0230 2024-06-29T15:57:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.029+0000 7f57a75f0640 1 -- 172.21.15.81:0/1041540811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57a00b5350 con 0x7f57a00d0230 2024-06-29T15:57:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.030+0000 7f5795ffb640 1 -- 172.21.15.81:0/1041540811 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f57980042a0 con 0x7f57a00d0230 2024-06-29T15:57:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.031+0000 7f5795ffb640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f57840765d0 0x7f5784078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.031+0000 7f57a538c640 1 -- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f57840765d0 msgr2=0x7f5784078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.031+0000 7f5795ffb640 1 -- 172.21.15.81:0/1041540811 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f57980b8d60 con 0x7f57a00d0230 2024-06-29T15:57:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.031+0000 7f57a538c640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f57840765d0 0x7f5784078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.039+0000 7f5795ffb640 1 -- 172.21.15.81:0/1041540811 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5798081640 con 0x7f57a00d0230 2024-06-29T15:57:55.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.228+0000 7f57a75f0640 1 -- 172.21.15.81:0/1041540811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f57a0071150 con 0x7f57a00d0230 2024-06-29T15:57:55.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.228+0000 7f5795ffb640 1 -- 172.21.15.81:0/1041540811 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v218) ==== 100+0+11 (secure 0 0 0) 0x7f5798085940 con 0x7f57a00d0230 2024-06-29T15:57:55.229 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:57:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.230+0000 7f57a75f0640 1 -- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f57840765d0 msgr2=0x7f5784078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.230+0000 7f57a75f0640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f57840765d0 0x7f5784078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.230+0000 7f57a75f0640 1 -- 172.21.15.81:0/1041540811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57a00d0230 msgr2=0x7f57a0157fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.230+0000 7f57a75f0640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57a00d0230 0x7f57a0157fb0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f579802ae10 tx=0x7f5798032c60 comp rx=0 tx=0).stop 2024-06-29T15:57:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.230+0000 7f57a75f0640 1 -- 172.21.15.81:0/1041540811 shutdown_connections 2024-06-29T15:57:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.230+0000 7f57a75f0640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f57840765d0 0x7f5784078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.230+0000 7f57a75f0640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57a00d0230 0x7f57a0157fb0 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.230+0000 7f57a75f0640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f57a00b4f40 0x7f57a0157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.230+0000 7f57a75f0640 1 --2- 172.21.15.81:0/1041540811 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f57a00b3200 0x7f57a0159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.230+0000 7f57a75f0640 1 -- 172.21.15.81:0/1041540811 >> 172.21.15.81:0/1041540811 conn(0x7f57a00b3680 msgr2=0x7f57a0071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.231+0000 7f57a75f0640 1 -- 172.21.15.81:0/1041540811 shutdown_connections 2024-06-29T15:57:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.231+0000 7f57a75f0640 1 -- 172.21.15.81:0/1041540811 wait complete. 2024-06-29T15:57:55.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:57:55.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.360+0000 7f608eb3c640 1 -- 172.21.15.81:0/1734107508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088073650 msgr2=0x7f608815b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.360+0000 7f608eb3c640 1 --2- 172.21.15.81:0/1734107508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088073650 0x7f608815b6e0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f6088072cf0 tx=0x7f607402a4d0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.360+0000 7f608eb3c640 1 -- 172.21.15.81:0/1734107508 shutdown_connections 2024-06-29T15:57:55.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.360+0000 7f608eb3c640 1 --2- 172.21.15.81:0/1734107508 >> v2:172.21.15.167:3300/0 conn(0x7f608815bc20 0x7f608806ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.360+0000 7f608eb3c640 1 --2- 172.21.15.81:0/1734107508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6088073650 0x7f608815b6e0 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.360+0000 7f608eb3c640 1 -- 172.21.15.81:0/1734107508 >> 172.21.15.81:0/1734107508 conn(0x7f60880b2920 msgr2=0x7f60880b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:55.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.360+0000 7f608eb3c640 1 -- 172.21.15.81:0/1734107508 shutdown_connections 2024-06-29T15:57:55.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.360+0000 7f608eb3c640 1 -- 172.21.15.81:0/1734107508 wait complete. 2024-06-29T15:57:55.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.362+0000 7f608eb3c640 1 Processor -- start 2024-06-29T15:57:55.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.362+0000 7f608eb3c640 1 -- start start 2024-06-29T15:57:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.363+0000 7f608eb3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60880cd150 0x7f60881f44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.363+0000 7f608eb3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f608815bc20 0x7f60880aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.363+0000 7f608eb3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608806ce70 0x7f6088149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608eb3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f608806f450 con 0x7f608806ce70 2024-06-29T15:57:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608eb3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f608806f2d0 con 0x7f608815bc20 2024-06-29T15:57:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608eb3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f608806f5d0 con 0x7f60880cd150 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608d0d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608806ce70 0x7f6088149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608d0d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608806ce70 0x7f6088149850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35390/0 (socket says 172.21.15.81:35390) 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608d0d9640 1 -- 172.21.15.81:0/1079579252 learned_addr learned my addr 172.21.15.81:0/1079579252 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f6087fff640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f608815bc20 0x7f60880aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608c8d8640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60880cd150 0x7f60881f44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608d0d9640 1 -- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60880cd150 msgr2=0x7f60881f44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608d0d9640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60880cd150 0x7f60881f44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608d0d9640 1 -- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f608815bc20 msgr2=0x7f60880aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608d0d9640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f608815bc20 0x7f60880aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608d0d9640 1 -- 172.21.15.81:0/1079579252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60880af590 con 0x7f608806ce70 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608d0d9640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608806ce70 0x7f6088149850 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f608000da00 tx=0x7f608000dec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f60857fa640 1 -- 172.21.15.81:0/1079579252 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080017070 con 0x7f608806ce70 2024-06-29T15:57:55.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f60857fa640 1 -- 172.21.15.81:0/1079579252 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f608000bda0 con 0x7f608806ce70 2024-06-29T15:57:55.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f608eb3c640 1 -- 172.21.15.81:0/1079579252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f608806c390 con 0x7f608806ce70 2024-06-29T15:57:55.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.364+0000 7f60857fa640 1 -- 172.21.15.81:0/1079579252 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080003bd0 con 0x7f608806ce70 2024-06-29T15:57:55.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.365+0000 7f608eb3c640 1 -- 172.21.15.81:0/1079579252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f608806c950 con 0x7f608806ce70 2024-06-29T15:57:55.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.366+0000 7f608eb3c640 1 -- 172.21.15.81:0/1079579252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f608815b6e0 con 0x7f608806ce70 2024-06-29T15:57:55.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.367+0000 7f60857fa640 1 -- 172.21.15.81:0/1079579252 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f6080010460 con 0x7f608806ce70 2024-06-29T15:57:55.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.368+0000 7f60857fa640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f606c0765d0 0x7f606c078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.368+0000 7f608c8d8640 1 -- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f606c0765d0 msgr2=0x7f606c078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:55.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.368+0000 7f60857fa640 1 -- 172.21.15.81:0/1079579252 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f60800983d0 con 0x7f608806ce70 2024-06-29T15:57:55.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.368+0000 7f608c8d8640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f606c0765d0 0x7f606c078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:55.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.372+0000 7f60857fa640 1 -- 172.21.15.81:0/1079579252 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6080060d40 con 0x7f608806ce70 2024-06-29T15:57:55.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.558+0000 7f608eb3c640 1 -- 172.21.15.81:0/1079579252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f6088155040 con 0x7f608806ce70 2024-06-29T15:57:55.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.559+0000 7f60857fa640 1 -- 172.21.15.81:0/1079579252 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v218) ==== 107+0+11 (secure 0 0 0) 0x7f6080065040 con 0x7f608806ce70 2024-06-29T15:57:55.560 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:57:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 -- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f606c0765d0 msgr2=0x7f606c078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f606c0765d0 0x7f606c078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 -- 172.21.15.81:0/1079579252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608806ce70 msgr2=0x7f6088149850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608806ce70 0x7f6088149850 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f608000da00 tx=0x7f608000dec0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 -- 172.21.15.81:0/1079579252 shutdown_connections 2024-06-29T15:57:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f606c0765d0 0x7f606c078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608806ce70 0x7f6088149850 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f608815bc20 0x7f60880aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 --2- 172.21.15.81:0/1079579252 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f60880cd150 0x7f60881f44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 -- 172.21.15.81:0/1079579252 >> 172.21.15.81:0/1079579252 conn(0x7f60880b2920 msgr2=0x7f608815e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:55.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 -- 172.21.15.81:0/1079579252 shutdown_connections 2024-06-29T15:57:55.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.561+0000 7f608eb3c640 1 -- 172.21.15.81:0/1079579252 wait complete. 2024-06-29T15:57:55.574 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:57:55.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.693+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2155731303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f3406ca00 msgr2=0x7f5f340d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.693+0000 7f5f3ad7b640 1 --2- 172.21.15.81:0/2155731303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f3406ca00 0x7f5f340d0340 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009be0 tx=0x7f5f2c02a840 comp rx=0 tx=0).stop 2024-06-29T15:57:55.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.693+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2155731303 shutdown_connections 2024-06-29T15:57:55.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.693+0000 7f5f3ad7b640 1 --2- 172.21.15.81:0/2155731303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f3406ca00 0x7f5f340d0340 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.693+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2155731303 >> 172.21.15.81:0/2155731303 conn(0x7f5f340b2920 msgr2=0x7f5f340b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:55.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.693+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2155731303 shutdown_connections 2024-06-29T15:57:55.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.693+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2155731303 wait complete. 2024-06-29T15:57:55.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.695+0000 7f5f3ad7b640 1 Processor -- start 2024-06-29T15:57:55.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.695+0000 7f5f3ad7b640 1 -- start start 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f3ad7b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f34070d60 0x7f5f3415a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f3ad7b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f340cff30 0x7f5f3414a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f3ad7b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f340abe90 0x7f5f340ac2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f3ad7b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3415fd40 con 0x7f5f340abe90 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f3ad7b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5f34070380 con 0x7f5f34070d60 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f3ad7b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5f3415fec0 con 0x7f5f340cff30 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f39318640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f340abe90 0x7f5f340ac2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f39318640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f340abe90 0x7f5f340ac2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35406/0 (socket says 172.21.15.81:35406) 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f39318640 1 -- 172.21.15.81:0/2557136110 learned_addr learned my addr 172.21.15.81:0/2557136110 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f38b17640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f34070d60 0x7f5f3415a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.696+0000 7f5f33fff640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f340cff30 0x7f5f3414a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f39318640 1 -- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f340cff30 msgr2=0x7f5f3414a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f39318640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f340cff30 0x7f5f3414a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f39318640 1 -- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f34070d60 msgr2=0x7f5f3415a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f39318640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f34070d60 0x7f5f3415a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f39318640 1 -- 172.21.15.81:0/2557136110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f340aa6f0 con 0x7f5f340abe90 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f39318640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f340abe90 0x7f5f340ac2a0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c005fd0 tx=0x7f5f2c033f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f317fa640 1 -- 172.21.15.81:0/2557136110 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c042020 con 0x7f5f340abe90 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f317fa640 1 -- 172.21.15.81:0/2557136110 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f2c004330 con 0x7f5f340abe90 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f317fa640 1 -- 172.21.15.81:0/2557136110 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c037040 con 0x7f5f340abe90 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2557136110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f340aa9d0 con 0x7f5f340abe90 2024-06-29T15:57:55.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.697+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2557136110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f340aaf90 con 0x7f5f340abe90 2024-06-29T15:57:55.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.698+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2557136110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f340aa460 con 0x7f5f340abe90 2024-06-29T15:57:55.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.700+0000 7f5f317fa640 1 -- 172.21.15.81:0/2557136110 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f5f2c045020 con 0x7f5f340abe90 2024-06-29T15:57:55.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.700+0000 7f5f317fa640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f5f180765a0 0x7f5f18078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:55.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.700+0000 7f5f38b17640 1 -- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f5f180765a0 msgr2=0x7f5f18078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:55.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.700+0000 7f5f317fa640 1 -- 172.21.15.81:0/2557136110 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f5f2c0b9210 con 0x7f5f340abe90 2024-06-29T15:57:55.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.700+0000 7f5f38b17640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f5f180765a0 0x7f5f18078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:55.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.705+0000 7f5f317fa640 1 -- 172.21.15.81:0/2557136110 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5f2c04c030 con 0x7f5f340abe90 2024-06-29T15:57:55.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.890+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2557136110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f5f34160390 con 0x7f5f340abe90 2024-06-29T15:57:55.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.890+0000 7f5f317fa640 1 -- 172.21.15.81:0/2557136110 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v218) ==== 103+0+11 (secure 0 0 0) 0x7f5f2c02f070 con 0x7f5f340abe90 2024-06-29T15:57:55.891 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:57:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f5f180765a0 msgr2=0x7f5f18078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f5f180765a0 0x7f5f18078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2557136110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f340abe90 msgr2=0x7f5f340ac2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f340abe90 0x7f5f340ac2a0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c005fd0 tx=0x7f5f2c033f00 comp rx=0 tx=0).stop 2024-06-29T15:57:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2557136110 shutdown_connections 2024-06-29T15:57:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f5f180765a0 0x7f5f18078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f340abe90 0x7f5f340ac2a0 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f340cff30 0x7f5f3414a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 --2- 172.21.15.81:0/2557136110 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f34070d60 0x7f5f3415a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:55.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2557136110 >> 172.21.15.81:0/2557136110 conn(0x7f5f340b2920 msgr2=0x7f5f34155730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:55.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2557136110 shutdown_connections 2024-06-29T15:57:55.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:55.892+0000 7f5f3ad7b640 1 -- 172.21.15.81:0/2557136110 wait complete. 2024-06-29T15:57:55.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:57:56.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.020+0000 7f46aa18c640 1 -- 172.21.15.81:0/195477086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4690004d90 msgr2=0x7f4690005240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.020+0000 7f46aa18c640 1 --2- 172.21.15.81:0/195477086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4690004d90 0x7f4690005240 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f469c001f00 tx=0x7f4690005830 comp rx=0 tx=0).stop 2024-06-29T15:57:56.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.020+0000 7f46a3fff640 1 -- 172.21.15.81:0/195477086 reap_dead start 2024-06-29T15:57:56.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.021+0000 7f46aa18c640 1 -- 172.21.15.81:0/195477086 shutdown_connections 2024-06-29T15:57:56.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.021+0000 7f46aa18c640 1 -- 172.21.15.81:0/195477086 >> 172.21.15.81:0/195477086 conn(0x7f46a40b1ec0 msgr2=0x7f46a40b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.021+0000 7f46aa18c640 1 -- 172.21.15.81:0/195477086 shutdown_connections 2024-06-29T15:57:56.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.021+0000 7f46aa18c640 1 -- 172.21.15.81:0/195477086 wait complete. 2024-06-29T15:57:56.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.022+0000 7f46aa18c640 1 Processor -- start 2024-06-29T15:57:56.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.023+0000 7f46aa18c640 1 -- start start 2024-06-29T15:57:56.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46aa18c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4690004d90 0x7f46a409b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46aa18c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a414f920 0x7f46a40b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46aa18c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a414fd30 0x7f46a40b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46aa18c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46a40749e0 con 0x7f46a414f920 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46aa18c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f46a4074860 con 0x7f46a414fd30 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46aa18c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f46a4073610 con 0x7f4690004d90 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a414f920 0x7f46a40b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a414f920 0x7f46a40b0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35424/0 (socket says 172.21.15.81:35424) 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a2ffd640 1 -- 172.21.15.81:0/1655172181 learned_addr learned my addr 172.21.15.81:0/1655172181 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a37fe640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4690004d90 0x7f46a409b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a3fff640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a414fd30 0x7f46a40b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a2ffd640 1 -- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4690004d90 msgr2=0x7f46a409b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a2ffd640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4690004d90 0x7f46a409b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a2ffd640 1 -- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a414fd30 msgr2=0x7f46a40b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a2ffd640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a414fd30 0x7f46a40b10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a2ffd640 1 -- 172.21.15.81:0/1655172181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a409c340 con 0x7f46a414f920 2024-06-29T15:57:56.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.024+0000 7f46a37fe640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4690004d90 0x7f46a409b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:56.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.025+0000 7f46a2ffd640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a414f920 0x7f46a40b0ba0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f4694011ae0 tx=0x7f4694009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:56.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.025+0000 7f469bfff640 1 -- 172.21.15.81:0/1655172181 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694004460 con 0x7f46a414f920 2024-06-29T15:57:56.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.025+0000 7f469bfff640 1 -- 172.21.15.81:0/1655172181 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f469400f980 con 0x7f46a414f920 2024-06-29T15:57:56.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.025+0000 7f46aa18c640 1 -- 172.21.15.81:0/1655172181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a409c620 con 0x7f46a414f920 2024-06-29T15:57:56.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.025+0000 7f469bfff640 1 -- 172.21.15.81:0/1655172181 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469400fdb0 con 0x7f46a414f920 2024-06-29T15:57:56.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.025+0000 7f46aa18c640 1 -- 172.21.15.81:0/1655172181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a409be90 con 0x7f46a414f920 2024-06-29T15:57:56.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.026+0000 7f46aa18c640 1 -- 172.21.15.81:0/1655172181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a409c860 con 0x7f46a414f920 2024-06-29T15:57:56.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.027+0000 7f469bfff640 1 -- 172.21.15.81:0/1655172181 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f46940035f0 con 0x7f46a414f920 2024-06-29T15:57:56.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.027+0000 7f469bfff640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f46840765d0 0x7f4684078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.027+0000 7f46a37fe640 1 -- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f46840765d0 msgr2=0x7f4684078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:56.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.028+0000 7f469bfff640 1 -- 172.21.15.81:0/1655172181 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f4694099080 con 0x7f46a414f920 2024-06-29T15:57:56.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.028+0000 7f46a37fe640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f46840765d0 0x7f4684078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:56.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.032+0000 7f469bfff640 1 -- 172.21.15.81:0/1655172181 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f46940619f0 con 0x7f46a414f920 2024-06-29T15:57:56.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.229+0000 7f46a37fe640 1 -- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f46840765d0 msgr2=0x7f4684078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:56.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.229+0000 7f46a37fe640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f46840765d0 0x7f4684078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:56.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.231+0000 7f46aa18c640 1 -- 172.21.15.81:0/1655172181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f46a41555f0 con 0x7f46a414f920 2024-06-29T15:57:56.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.232+0000 7f469bfff640 1 -- 172.21.15.81:0/1655172181 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v218) ==== 80+0+476 (secure 0 0 0) 0x7f4694065cf0 con 0x7f46a414f920 2024-06-29T15:57:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3042354413 2024-06-30T15:57:48.595362+0000 2024-06-29T15:57:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1619575755 2024-06-30T15:57:48.595362+0000 2024-06-29T15:57:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/760717235 2024-06-30T15:57:48.595362+0000 2024-06-29T15:57:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3781943725 2024-06-30T15:57:48.595362+0000 2024-06-29T15:57:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/376674682 2024-06-30T15:57:48.595362+0000 2024-06-29T15:57:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3640842563 2024-06-30T15:57:48.595362+0000 2024-06-29T15:57:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/2018755589 2024-06-30T15:57:48.595362+0000 2024-06-29T15:57:56.232 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/2018755589 2024-06-30T15:57:48.595362+0000 2024-06-29T15:57:56.233 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T15:57:56.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 -- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f46840765d0 msgr2=0x7f4684078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:56.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f46840765d0 0x7f4684078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 -- 172.21.15.81:0/1655172181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a414f920 msgr2=0x7f46a40b0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a414f920 0x7f46a40b0ba0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f4694011ae0 tx=0x7f4694009e40 comp rx=0 tx=0).stop 2024-06-29T15:57:56.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 -- 172.21.15.81:0/1655172181 shutdown_connections 2024-06-29T15:57:56.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f46840765d0 0x7f4684078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a414fd30 0x7f46a40b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a414f920 0x7f46a40b0ba0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 --2- 172.21.15.81:0/1655172181 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4690004d90 0x7f46a409b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 -- 172.21.15.81:0/1655172181 >> 172.21.15.81:0/1655172181 conn(0x7f46a40b1ec0 msgr2=0x7f46a40cc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 -- 172.21.15.81:0/1655172181 shutdown_connections 2024-06-29T15:57:56.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.234+0000 7f46aa18c640 1 -- 172.21.15.81:0/1655172181 wait complete. 2024-06-29T15:57:56.245 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:57:56.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:56.246 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:57:56.273 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T15:57:56.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:57:56.273 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:57:56.329 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:56.329 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:57:56.329 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:56.345 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:56.345 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:56.345 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:57:56.346 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:56.385 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:57:56.385 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:57:56.385 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:57:56.385 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:57:56.401 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:57:56.401 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:56.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.540+0000 7f1da5b03640 1 -- 172.21.15.81:0/3495290960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1da00b4f40 msgr2=0x7f1da00b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.540+0000 7f1da5b03640 1 --2- 172.21.15.81:0/3495290960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1da00b4f40 0x7f1da00b5350 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f1da0074b10 tx=0x7f1d9802aa30 comp rx=0 tx=0).stop 2024-06-29T15:57:56.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.540+0000 7f1da5b03640 1 -- 172.21.15.81:0/3495290960 shutdown_connections 2024-06-29T15:57:56.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.540+0000 7f1da5b03640 1 --2- 172.21.15.81:0/3495290960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1da00b4f40 0x7f1da00b5350 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.540+0000 7f1da5b03640 1 --2- 172.21.15.81:0/3495290960 >> v2:172.21.15.167:3300/0 conn(0x7f1da00d0230 0x7f1da00d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.540+0000 7f1da5b03640 1 -- 172.21.15.81:0/3495290960 >> 172.21.15.81:0/3495290960 conn(0x7f1da00b3680 msgr2=0x7f1da00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.540+0000 7f1da5b03640 1 -- 172.21.15.81:0/3495290960 shutdown_connections 2024-06-29T15:57:56.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.540+0000 7f1da5b03640 1 -- 172.21.15.81:0/3495290960 wait complete. 2024-06-29T15:57:56.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.542+0000 7f1da5b03640 1 Processor -- start 2024-06-29T15:57:56.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.542+0000 7f1da5b03640 1 -- start start 2024-06-29T15:57:56.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.543+0000 7f1da5b03640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1da00b3200 0x7f1da01f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.543+0000 7f1da5b03640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1da00b4f40 0x7f1da01f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.543+0000 7f1da5b03640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1da00d0230 0x7f1da01f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.543+0000 7f1da5b03640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1da00b5c20 con 0x7f1da00b4f40 2024-06-29T15:57:56.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.543+0000 7f1da5b03640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1da00b5aa0 con 0x7f1da00d0230 2024-06-29T15:57:56.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.543+0000 7f1da5b03640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1da00b5da0 con 0x7f1da00b3200 2024-06-29T15:57:56.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.543+0000 7f1d9effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1da00b4f40 0x7f1da01f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.543+0000 7f1d9effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1da00b4f40 0x7f1da01f3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35444/0 (socket says 172.21.15.81:35444) 2024-06-29T15:57:56.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9ffff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1da00d0230 0x7f1da01f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9effd640 1 -- 172.21.15.81:0/4094181538 learned_addr learned my addr 172.21.15.81:0/4094181538 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9f7fe640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1da00b3200 0x7f1da01f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9effd640 1 -- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1da00b3200 msgr2=0x7f1da01f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9effd640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1da00b3200 0x7f1da01f32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9effd640 1 -- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1da00d0230 msgr2=0x7f1da01f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9effd640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1da00d0230 0x7f1da01f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9effd640 1 -- 172.21.15.81:0/4094181538 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da0156590 con 0x7f1da00b4f40 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9ffff640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1da00d0230 0x7f1da01f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9f7fe640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1da00b3200 0x7f1da01f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d9effd640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1da00b4f40 0x7f1da01f3810 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009fd0 tx=0x7f1d8800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d97fff640 1 -- 172.21.15.81:0/4094181538 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8800ccc0 con 0x7f1da00b4f40 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d97fff640 1 -- 172.21.15.81:0/4094181538 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d8800ee70 con 0x7f1da00b4f40 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1da5b03640 1 -- 172.21.15.81:0/4094181538 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da0159f30 con 0x7f1da00b4f40 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1d97fff640 1 -- 172.21.15.81:0/4094181538 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8800ce60 con 0x7f1da00b4f40 2024-06-29T15:57:56.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.544+0000 7f1da5b03640 1 -- 172.21.15.81:0/4094181538 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da0159970 con 0x7f1da00b4f40 2024-06-29T15:57:56.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.546+0000 7f1da5b03640 1 -- 172.21.15.81:0/4094181538 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da01593e0 con 0x7f1da00b4f40 2024-06-29T15:57:56.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.546+0000 7f1d97fff640 1 -- 172.21.15.81:0/4094181538 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f1d88005260 con 0x7f1da00b4f40 2024-06-29T15:57:56.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.546+0000 7f1d97fff640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f1d800765d0 0x7f1d80078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.547+0000 7f1d9f7fe640 1 -- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f1d800765d0 msgr2=0x7f1d80078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:56.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.547+0000 7f1d97fff640 1 -- 172.21.15.81:0/4094181538 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f1d88098960 con 0x7f1da00b4f40 2024-06-29T15:57:56.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.547+0000 7f1d9f7fe640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f1d800765d0 0x7f1d80078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:56.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.551+0000 7f1d97fff640 1 -- 172.21.15.81:0/4094181538 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1d880612d0 con 0x7f1da00b4f40 2024-06-29T15:57:56.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.748+0000 7f1d9f7fe640 1 -- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f1d800765d0 msgr2=0x7f1d80078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:56.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.748+0000 7f1d9f7fe640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f1d800765d0 0x7f1d80078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:57:56.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.777+0000 7f1da5b03640 1 -- 172.21.15.81:0/4094181538 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1da00b5350 con 0x7f1da00b4f40 2024-06-29T15:57:56.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.778+0000 7f1d97fff640 1 -- 172.21.15.81:0/4094181538 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+6565 (secure 0 0 0) 0x7f1d880655d0 con 0x7f1da00b4f40 2024-06-29T15:57:56.778 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:56.779 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":118,"btime":"2024-06-29T15:57:37:378022+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.637042+0000","modified":"2024-06-29T15:57:24.656503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11226":{"gid":11226,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/739691411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":739691411},{"type":"v1","addr":"172.21.15.81:6833","nonce":739691411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11226,"qdb_cluster":[11226]},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:33.727815+0000","modified":"2024-06-29T15:57:36.736776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10114":{"gid":10114,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10114,"qdb_cluster":[10114]},"id":14}]} 2024-06-29T15:57:56.779 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 118 2024-06-29T15:57:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 -- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f1d800765d0 msgr2=0x7f1d80078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f1d800765d0 0x7f1d80078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 -- 172.21.15.81:0/4094181538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1da00b4f40 msgr2=0x7f1da01f3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1da00b4f40 0x7f1da01f3810 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009fd0 tx=0x7f1d8800ec20 comp rx=0 tx=0).stop 2024-06-29T15:57:56.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 -- 172.21.15.81:0/4094181538 shutdown_connections 2024-06-29T15:57:56.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f1d800765d0 0x7f1d80078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1da00d0230 0x7f1da01f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1da00b4f40 0x7f1da01f3810 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 --2- 172.21.15.81:0/4094181538 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1da00b3200 0x7f1da01f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 -- 172.21.15.81:0/4094181538 >> 172.21.15.81:0/4094181538 conn(0x7f1da00b3680 msgr2=0x7f1da00cb810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 -- 172.21.15.81:0/4094181538 shutdown_connections 2024-06-29T15:57:56.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.780+0000 7f1da5b03640 1 -- 172.21.15.81:0/4094181538 wait complete. 2024-06-29T15:57:56.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:56.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.907+0000 7ffb872fd640 1 -- 172.21.15.81:0/1608417677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8006f660 msgr2=0x7ffb8006fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.907+0000 7ffb872fd640 1 --2- 172.21.15.81:0/1608417677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8006f660 0x7ffb8006fa70 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7ffb78005fd0 tx=0x7ffb7802aa10 comp rx=0 tx=0).stop 2024-06-29T15:57:56.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.907+0000 7ffb872fd640 1 -- 172.21.15.81:0/1608417677 shutdown_connections 2024-06-29T15:57:56.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.907+0000 7ffb872fd640 1 --2- 172.21.15.81:0/1608417677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8006f660 0x7ffb8006fa70 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.907+0000 7ffb872fd640 1 --2- 172.21.15.81:0/1608417677 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb800b35b0 0x7ffb8015b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.907+0000 7ffb872fd640 1 -- 172.21.15.81:0/1608417677 >> 172.21.15.81:0/1608417677 conn(0x7ffb8006ccc0 msgr2=0x7ffb800b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:56.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.907+0000 7ffb872fd640 1 -- 172.21.15.81:0/1608417677 shutdown_connections 2024-06-29T15:57:56.908 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.907+0000 7ffb872fd640 1 -- 172.21.15.81:0/1608417677 wait complete. 2024-06-29T15:57:56.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.909+0000 7ffb872fd640 1 Processor -- start 2024-06-29T15:57:56.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.909+0000 7ffb872fd640 1 -- start start 2024-06-29T15:57:56.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.910+0000 7ffb872fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8006f660 0x7ffb8014f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.910+0000 7ffb872fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb800b35b0 0x7ffb8014dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.910+0000 7ffb872fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb800b4430 0x7ffb801390e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb872fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8006f1e0 con 0x7ffb8006f660 2024-06-29T15:57:56.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb872fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8006f060 con 0x7ffb800b35b0 2024-06-29T15:57:56.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb872fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ffb8006f360 con 0x7ffb800b4430 2024-06-29T15:57:56.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb85099640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8006f660 0x7ffb8014f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb85099640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8006f660 0x7ffb8014f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35462/0 (socket says 172.21.15.81:35462) 2024-06-29T15:57:56.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb85099640 1 -- 172.21.15.81:0/592158479 learned_addr learned my addr 172.21.15.81:0/592158479 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:56.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb84898640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb800b35b0 0x7ffb8014dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb8589a640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb800b4430 0x7ffb801390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:56.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb85099640 1 -- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb800b4430 msgr2=0x7ffb801390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb85099640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb800b4430 0x7ffb801390e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb85099640 1 -- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb800b35b0 msgr2=0x7ffb8014dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:56.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb85099640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb800b35b0 0x7ffb8014dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:56.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb85099640 1 -- 172.21.15.81:0/592158479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8014fe00 con 0x7ffb8006f660 2024-06-29T15:57:56.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb8589a640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb800b4430 0x7ffb801390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:57:56.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb85099640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8006f660 0x7ffb8014f8c0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009a40 tx=0x7ffb70009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:56.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb75ffb640 1 -- 172.21.15.81:0/592158479 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7001c070 con 0x7ffb8006f660 2024-06-29T15:57:56.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb75ffb640 1 -- 172.21.15.81:0/592158479 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb7000bb70 con 0x7ffb8006f660 2024-06-29T15:57:56.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb75ffb640 1 -- 172.21.15.81:0/592158479 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb70017410 con 0x7ffb8006f660 2024-06-29T15:57:56.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.911+0000 7ffb872fd640 1 -- 172.21.15.81:0/592158479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb8014e820 con 0x7ffb8006f660 2024-06-29T15:57:56.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.912+0000 7ffb872fd640 1 -- 172.21.15.81:0/592158479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb8014ede0 con 0x7ffb8006f660 2024-06-29T15:57:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.913+0000 7ffb872fd640 1 -- 172.21.15.81:0/592158479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb8015c640 con 0x7ffb8006f660 2024-06-29T15:57:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.913+0000 7ffb75ffb640 1 -- 172.21.15.81:0/592158479 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7ffb7001f070 con 0x7ffb8006f660 2024-06-29T15:57:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.914+0000 7ffb75ffb640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb640765a0 0x7ffb64078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.914+0000 7ffb84898640 1 -- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb640765a0 msgr2=0x7ffb64078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.914+0000 7ffb75ffb640 1 -- 172.21.15.81:0/592158479 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7ffb70099d60 con 0x7ffb8006f660 2024-06-29T15:57:56.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.914+0000 7ffb84898640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb640765a0 0x7ffb64078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:56.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:56.918+0000 7ffb75ffb640 1 -- 172.21.15.81:0/592158479 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffb8015c640 con 0x7ffb8006f660 2024-06-29T15:57:57.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.104+0000 7ffb872fd640 1 -- 172.21.15.81:0/592158479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb8006fa70 con 0x7ffb8006f660 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.106+0000 7ffb75ffb640 1 -- 172.21.15.81:0/592158479 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) ==== 74+0+21172 (secure 0 0 0) 0x7ffb70062640 con 0x7ffb8006f660 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:57.106 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":218,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:48.595402+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.259110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:32.675276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3042354413":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1619575755":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/760717235":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/3781943725":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/376674682":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/3640842563":"2024-06-30T15:57:48.595362+0000","172.21.15.167:6833/2018755589":"2024-06-30T15:57:48.595362+0000","172.21.15.167:6832/2018755589":"2024-06-30T15:57:48.595362+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:57.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 -- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb640765a0 msgr2=0x7ffb64078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:57.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb640765a0 0x7ffb64078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 -- 172.21.15.81:0/592158479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8006f660 msgr2=0x7ffb8014f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8006f660 0x7ffb8014f8c0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009a40 tx=0x7ffb70009f00 comp rx=0 tx=0).stop 2024-06-29T15:57:57.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 -- 172.21.15.81:0/592158479 shutdown_connections 2024-06-29T15:57:57.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7ffb640765a0 0x7ffb64078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffb800b4430 0x7ffb801390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffb800b35b0 0x7ffb8014dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 --2- 172.21.15.81:0/592158479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8006f660 0x7ffb8014f8c0 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 -- 172.21.15.81:0/592158479 >> 172.21.15.81:0/592158479 conn(0x7ffb8006ccc0 msgr2=0x7ffb8014cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 -- 172.21.15.81:0/592158479 shutdown_connections 2024-06-29T15:57:57.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.108+0000 7ffb872fd640 1 -- 172.21.15.81:0/592158479 wait complete. 2024-06-29T15:57:57.120 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:57:19.937473+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:57:20.596779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '213', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:57:32.259110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '216', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:57:32.675276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '217', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:57:57.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.238+0000 7f9777bf5640 1 -- 172.21.15.81:0/371598834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f975c003a50 msgr2=0x7f975c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.238+0000 7f9777bf5640 1 --2- 172.21.15.81:0/371598834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f975c003a50 0x7f975c003ec0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f97600016c0 tx=0x7f975c005830 comp rx=0 tx=0).stop 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.238+0000 7f9777bf5640 1 -- 172.21.15.81:0/371598834 shutdown_connections 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.238+0000 7f9777bf5640 1 --2- 172.21.15.81:0/371598834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f975c003a50 0x7f975c003ec0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.238+0000 7f9777bf5640 1 -- 172.21.15.81:0/371598834 >> 172.21.15.81:0/371598834 conn(0x7f97700b2920 msgr2=0x7f97700b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.239+0000 7f9777bf5640 1 -- 172.21.15.81:0/371598834 shutdown_connections 2024-06-29T15:57:57.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.239+0000 7f9777bf5640 1 -- 172.21.15.81:0/371598834 wait complete. 2024-06-29T15:57:57.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.240+0000 7f9777bf5640 1 Processor -- start 2024-06-29T15:57:57.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.240+0000 7f9777bf5640 1 -- start start 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9777bf5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f975c003a50 0x7f97701570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9777bf5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f977014b520 0x7f977014ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9777bf5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f977014fe30 0x7f9770150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9777bf5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f977015ba70 con 0x7f975c003a50 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9777bf5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f977015b8f0 con 0x7f977014fe30 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9777bf5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f977015bbf0 con 0x7f977014b520 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9775991640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f975c003a50 0x7f97701570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9775991640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f975c003a50 0x7f97701570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35480/0 (socket says 172.21.15.81:35480) 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9775991640 1 -- 172.21.15.81:0/339810813 learned_addr learned my addr 172.21.15.81:0/339810813 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9775190640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f977014b520 0x7f977014ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9776192640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f977014fe30 0x7f9770150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9775991640 1 -- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f977014b520 msgr2=0x7f977014ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9775991640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f977014b520 0x7f977014ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9775991640 1 -- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f977014fe30 msgr2=0x7f9770150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9775991640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f977014fe30 0x7f9770150240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9775991640 1 -- 172.21.15.81:0/339810813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9770156000 con 0x7f975c003a50 2024-06-29T15:57:57.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9775991640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f975c003a50 0x7f97701570a0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f97600017e0 tx=0x7f97600063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:57.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f97667fc640 1 -- 172.21.15.81:0/339810813 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760042070 con 0x7f975c003a50 2024-06-29T15:57:57.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f97667fc640 1 -- 172.21.15.81:0/339810813 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9760039070 con 0x7f975c003a50 2024-06-29T15:57:57.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9777bf5640 1 -- 172.21.15.81:0/339810813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97701562e0 con 0x7f975c003a50 2024-06-29T15:57:57.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f97667fc640 1 -- 172.21.15.81:0/339810813 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9760003d70 con 0x7f975c003a50 2024-06-29T15:57:57.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.242+0000 7f9777bf5640 1 -- 172.21.15.81:0/339810813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97701568a0 con 0x7f975c003a50 2024-06-29T15:57:57.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.244+0000 7f9777bf5640 1 -- 172.21.15.81:0/339810813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97700732c0 con 0x7f975c003a50 2024-06-29T15:57:57.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.244+0000 7f97667fc640 1 -- 172.21.15.81:0/339810813 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 134) ==== 98462+0+0 (secure 0 0 0) 0x7f976003b070 con 0x7f975c003a50 2024-06-29T15:57:57.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.245+0000 7f97667fc640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f97540765d0 0x7f9754078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.245+0000 7f97667fc640 1 -- 172.21.15.81:0/339810813 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 7045+0+0 (secure 0 0 0) 0x7f97600b76c0 con 0x7f975c003a50 2024-06-29T15:57:57.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.245+0000 7f9775190640 1 -- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f97540765d0 msgr2=0x7f9754078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2452570162 2024-06-29T15:57:57.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.245+0000 7f9775190640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f97540765d0 0x7f9754078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:57:57.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.249+0000 7f97667fc640 1 -- 172.21.15.81:0/339810813 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f97600800c0 con 0x7f975c003a50 2024-06-29T15:57:57.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.324+0000 7f97667fc640 1 -- 172.21.15.81:0/339810813 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 135) ==== 97912+0+0 (secure 0 0 0) 0x7f9760030050 con 0x7f975c003a50 2024-06-29T15:57:57.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.325+0000 7f97667fc640 1 -- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f97540765d0 msgr2=0x7f9754078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:57:57.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.325+0000 7f97667fc640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f97540765d0 0x7f9754078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.390 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.388+0000 7f8adaab0640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.391 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.388+0000 7f8adaab0640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.391 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.388+0000 7f8adaab0640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.391 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.388+0000 7f8adaab0640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.391 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.388+0000 7f8adaab0640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.391 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.389+0000 7f8addab6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.391 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.389+0000 7f8addab6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.391 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.389+0000 7f8addab6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.391 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.389+0000 7f8addab6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.391 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.389+0000 7f8addab6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.401 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.399+0000 7f8addab6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.401 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.399+0000 7f8addab6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.401 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.399+0000 7f8addab6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.401 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.399+0000 7f8addab6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.401 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.399+0000 7f8addab6640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.401 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.399+0000 7f8adaab0640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.401 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.399+0000 7f8adaab0640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.401 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.399+0000 7f8adaab0640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.401 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.399+0000 7f8adaab0640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.401 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:57:57.399+0000 7f8adaab0640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:57:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.452+0000 7f9777bf5640 1 -- 172.21.15.81:0/339810813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f97700b1220 con 0x7f975c003a50 2024-06-29T15:57:57.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.452+0000 7f97667fc640 1 -- 172.21.15.81:0/339810813 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v219) ==== 93+0+10 (secure 0 0 0) 0x7f97600b0640 con 0x7f975c003a50 2024-06-29T15:57:57.485 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:57:57.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.454+0000 7f9777bf5640 1 -- 172.21.15.81:0/339810813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f975c003a50 msgr2=0x7f97701570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.454+0000 7f9777bf5640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f975c003a50 0x7f97701570a0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f97600017e0 tx=0x7f97600063a0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.455+0000 7f9777bf5640 1 -- 172.21.15.81:0/339810813 shutdown_connections 2024-06-29T15:57:57.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.455+0000 7f9777bf5640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:6832/2452570162,v1:172.21.15.167:6833/2452570162] conn(0x7f97540765d0 0x7f9754078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.455+0000 7f9777bf5640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f977014fe30 0x7f9770150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.455+0000 7f9777bf5640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f977014b520 0x7f977014ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.455+0000 7f9777bf5640 1 --2- 172.21.15.81:0/339810813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f975c003a50 0x7f97701570a0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.455+0000 7f9777bf5640 1 -- 172.21.15.81:0/339810813 >> 172.21.15.81:0/339810813 conn(0x7f97700b2920 msgr2=0x7f977015cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.455+0000 7f9777bf5640 1 -- 172.21.15.81:0/339810813 shutdown_connections 2024-06-29T15:57:57.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.455+0000 7f9777bf5640 1 -- 172.21.15.81:0/339810813 wait complete. 2024-06-29T15:57:57.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:57:57.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.603+0000 7ff363af6640 1 -- 172.21.15.81:0/2646059303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff35c0b5250 msgr2=0x7ff35c0d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.603+0000 7ff363af6640 1 --2- 172.21.15.81:0/2646059303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff35c0b5250 0x7ff35c0d09c0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7ff35c08e230 tx=0x7ff35802a680 comp rx=0 tx=0).stop 2024-06-29T15:57:57.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.603+0000 7ff363af6640 1 -- 172.21.15.81:0/2646059303 shutdown_connections 2024-06-29T15:57:57.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.603+0000 7ff363af6640 1 --2- 172.21.15.81:0/2646059303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff35c0b5250 0x7ff35c0d09c0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.603+0000 7ff363af6640 1 --2- 172.21.15.81:0/2646059303 >> v2:172.21.15.167:3300/0 conn(0x7ff35c0b9040 0x7ff35c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.603+0000 7ff363af6640 1 -- 172.21.15.81:0/2646059303 >> 172.21.15.81:0/2646059303 conn(0x7ff35c09cf30 msgr2=0x7ff35c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.603+0000 7ff363af6640 1 -- 172.21.15.81:0/2646059303 shutdown_connections 2024-06-29T15:57:57.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.603+0000 7ff363af6640 1 -- 172.21.15.81:0/2646059303 wait complete. 2024-06-29T15:57:57.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.605+0000 7ff363af6640 1 Processor -- start 2024-06-29T15:57:57.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.605+0000 7ff363af6640 1 -- start start 2024-06-29T15:57:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.606+0000 7ff363af6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff35c0b5250 0x7ff35c0b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.606+0000 7ff363af6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff35c0b7290 0x7ff35c156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.606+0000 7ff363af6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff35c0b9040 0x7ff35c0b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.606+0000 7ff363af6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c0d2e20 con 0x7ff35c0b9040 2024-06-29T15:57:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.606+0000 7ff363af6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c0d2ca0 con 0x7ff35c0b7290 2024-06-29T15:57:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.606+0000 7ff363af6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff35c0b7db0 con 0x7ff35c0b5250 2024-06-29T15:57:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff362093640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff35c0b9040 0x7ff35c0b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff362093640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff35c0b9040 0x7ff35c0b33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35502/0 (socket says 172.21.15.81:35502) 2024-06-29T15:57:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff362093640 1 -- 172.21.15.81:0/1935404176 learned_addr learned my addr 172.21.15.81:0/1935404176 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff361892640 1 --2- 172.21.15.81:0/1935404176 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff35c0b5250 0x7ff35c0b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff361091640 1 --2- 172.21.15.81:0/1935404176 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff35c0b7290 0x7ff35c156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff362093640 1 -- 172.21.15.81:0/1935404176 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff35c0b5250 msgr2=0x7ff35c0b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff362093640 1 --2- 172.21.15.81:0/1935404176 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff35c0b5250 0x7ff35c0b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff362093640 1 -- 172.21.15.81:0/1935404176 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff35c0b7290 msgr2=0x7ff35c156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff362093640 1 --2- 172.21.15.81:0/1935404176 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff35c0b7290 0x7ff35c156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff362093640 1 -- 172.21.15.81:0/1935404176 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35c156680 con 0x7ff35c0b9040 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff362093640 1 --2- 172.21.15.81:0/1935404176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff35c0b9040 0x7ff35c0b33f0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7ff35400c760 tx=0x7ff35400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff34e7fc640 1 -- 172.21.15.81:0/1935404176 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff354003b70 con 0x7ff35c0b9040 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff34e7fc640 1 -- 172.21.15.81:0/1935404176 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff354003d30 con 0x7ff35c0b9040 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff34e7fc640 1 -- 172.21.15.81:0/1935404176 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3540052d0 con 0x7ff35c0b9040 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff363af6640 1 -- 172.21.15.81:0/1935404176 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff35c0b20c0 con 0x7ff35c0b9040 2024-06-29T15:57:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.607+0000 7ff363af6640 1 -- 172.21.15.81:0/1935404176 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff35c0b2680 con 0x7ff35c0b9040 2024-06-29T15:57:57.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.608+0000 7ff363af6640 1 -- 172.21.15.81:0/1935404176 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff35c0b1e30 con 0x7ff35c0b9040 2024-06-29T15:57:57.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.609+0000 7ff34e7fc640 1 -- 172.21.15.81:0/1935404176 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 135) ==== 97912+0+0 (secure 0 0 0) 0x7ff354005470 con 0x7ff35c0b9040 2024-06-29T15:57:57.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.610+0000 7ff34e7fc640 1 -- 172.21.15.81:0/1935404176 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 7518+0+0 (secure 0 0 0) 0x7ff3540976e0 con 0x7ff35c0b9040 2024-06-29T15:57:57.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.614+0000 7ff34e7fc640 1 -- 172.21.15.81:0/1935404176 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff354060d80 con 0x7ff35c0b9040 2024-06-29T15:57:57.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.796+0000 7ff363af6640 1 -- 172.21.15.81:0/1935404176 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff35c0b82f0 con 0x7ff35c0b9040 2024-06-29T15:57:57.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.797+0000 7ff34e7fc640 1 -- 172.21.15.81:0/1935404176 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v219) ==== 104+0+11 (secure 0 0 0) 0x7ff354060680 con 0x7ff35c0b9040 2024-06-29T15:57:57.797 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:57:57.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.799+0000 7ff363af6640 1 -- 172.21.15.81:0/1935404176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff35c0b9040 msgr2=0x7ff35c0b33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.799+0000 7ff363af6640 1 --2- 172.21.15.81:0/1935404176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff35c0b9040 0x7ff35c0b33f0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7ff35400c760 tx=0x7ff35400cc20 comp rx=0 tx=0).stop 2024-06-29T15:57:57.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.799+0000 7ff363af6640 1 -- 172.21.15.81:0/1935404176 shutdown_connections 2024-06-29T15:57:57.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.799+0000 7ff363af6640 1 --2- 172.21.15.81:0/1935404176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff35c0b9040 0x7ff35c0b33f0 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.799+0000 7ff363af6640 1 --2- 172.21.15.81:0/1935404176 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff35c0b7290 0x7ff35c156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.799+0000 7ff363af6640 1 --2- 172.21.15.81:0/1935404176 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff35c0b5250 0x7ff35c0b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.799+0000 7ff363af6640 1 -- 172.21.15.81:0/1935404176 >> 172.21.15.81:0/1935404176 conn(0x7ff35c09cf30 msgr2=0x7ff35c15fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.799+0000 7ff363af6640 1 -- 172.21.15.81:0/1935404176 shutdown_connections 2024-06-29T15:57:57.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.799+0000 7ff363af6640 1 -- 172.21.15.81:0/1935404176 wait complete. 2024-06-29T15:57:57.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:57:57.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.925+0000 7fbd5deac640 1 -- 172.21.15.81:0/2016262966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd580b57b0 msgr2=0x7fbd5815aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.926+0000 7fbd5deac640 1 --2- 172.21.15.81:0/2016262966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd580b57b0 0x7fbd5815aed0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009830 tx=0x7fbd4802a5c0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.926+0000 7fbd5deac640 1 -- 172.21.15.81:0/2016262966 shutdown_connections 2024-06-29T15:57:57.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.926+0000 7fbd5deac640 1 --2- 172.21.15.81:0/2016262966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd580b57b0 0x7fbd5815aed0 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.926+0000 7fbd5deac640 1 -- 172.21.15.81:0/2016262966 >> 172.21.15.81:0/2016262966 conn(0x7fbd580d0930 msgr2=0x7fbd580d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:57.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.926+0000 7fbd5deac640 1 -- 172.21.15.81:0/2016262966 shutdown_connections 2024-06-29T15:57:57.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.926+0000 7fbd5deac640 1 -- 172.21.15.81:0/2016262966 wait complete. 2024-06-29T15:57:57.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.927+0000 7fbd5deac640 1 Processor -- start 2024-06-29T15:57:57.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.928+0000 7fbd5deac640 1 -- start start 2024-06-29T15:57:57.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd5deac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd5806ef90 0x7fbd58155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd5deac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd580b3960 0x7fbd58154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd5deac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd580b57b0 0x7fbd581548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:57.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd5deac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbd580b4600 con 0x7fbd580b57b0 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd5deac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbd580b4480 con 0x7fbd5806ef90 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd5deac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbd580b4780 con 0x7fbd580b3960 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd57fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd580b57b0 0x7fbd581548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd57fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd580b57b0 0x7fbd581548c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35516/0 (socket says 172.21.15.81:35516) 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd57fff640 1 -- 172.21.15.81:0/1068118302 learned_addr learned my addr 172.21.15.81:0/1068118302 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd577fe640 1 --2- 172.21.15.81:0/1068118302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd5806ef90 0x7fbd58155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd56ffd640 1 --2- 172.21.15.81:0/1068118302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd580b3960 0x7fbd58154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd57fff640 1 -- 172.21.15.81:0/1068118302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd580b3960 msgr2=0x7fbd58154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd57fff640 1 --2- 172.21.15.81:0/1068118302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd580b3960 0x7fbd58154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd57fff640 1 -- 172.21.15.81:0/1068118302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd5806ef90 msgr2=0x7fbd58155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd57fff640 1 --2- 172.21.15.81:0/1068118302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd5806ef90 0x7fbd58155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd57fff640 1 -- 172.21.15.81:0/1068118302 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd581572b0 con 0x7fbd580b57b0 2024-06-29T15:57:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.929+0000 7fbd57fff640 1 --2- 172.21.15.81:0/1068118302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd580b57b0 0x7fbd581548c0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000cb40 tx=0x7fbd5000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.930+0000 7fbd3ffff640 1 -- 172.21.15.81:0/1068118302 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd50016020 con 0x7fbd580b57b0 2024-06-29T15:57:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.930+0000 7fbd3ffff640 1 -- 172.21.15.81:0/1068118302 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd5000ad30 con 0x7fbd580b57b0 2024-06-29T15:57:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.930+0000 7fbd3ffff640 1 -- 172.21.15.81:0/1068118302 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd50012680 con 0x7fbd580b57b0 2024-06-29T15:57:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.930+0000 7fbd5deac640 1 -- 172.21.15.81:0/1068118302 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd580ab8e0 con 0x7fbd580b57b0 2024-06-29T15:57:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.930+0000 7fbd5deac640 1 -- 172.21.15.81:0/1068118302 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd580abea0 con 0x7fbd580b57b0 2024-06-29T15:57:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.931+0000 7fbd5deac640 1 -- 172.21.15.81:0/1068118302 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd5806e2a0 con 0x7fbd580b57b0 2024-06-29T15:57:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.931+0000 7fbd3ffff640 1 -- 172.21.15.81:0/1068118302 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 135) ==== 97912+0+0 (secure 0 0 0) 0x7fbd50019020 con 0x7fbd580b57b0 2024-06-29T15:57:57.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.932+0000 7fbd3ffff640 1 -- 172.21.15.81:0/1068118302 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 7518+0+0 (secure 0 0 0) 0x7fbd50097ac0 con 0x7fbd580b57b0 2024-06-29T15:57:57.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:57.936+0000 7fbd3ffff640 1 -- 172.21.15.81:0/1068118302 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbd5806e2a0 con 0x7fbd580b57b0 2024-06-29T15:57:58.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.117+0000 7fbd5deac640 1 -- 172.21.15.81:0/1068118302 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbd580b4bd0 con 0x7fbd580b57b0 2024-06-29T15:57:58.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.118+0000 7fbd3ffff640 1 -- 172.21.15.81:0/1068118302 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v219) ==== 100+0+11 (secure 0 0 0) 0x7fbd50020030 con 0x7fbd580b57b0 2024-06-29T15:57:58.118 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:57:58.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.120+0000 7fbd5deac640 1 -- 172.21.15.81:0/1068118302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd580b57b0 msgr2=0x7fbd581548c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.120+0000 7fbd5deac640 1 --2- 172.21.15.81:0/1068118302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd580b57b0 0x7fbd581548c0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000cb40 tx=0x7fbd5000a060 comp rx=0 tx=0).stop 2024-06-29T15:57:58.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.120+0000 7fbd5deac640 1 -- 172.21.15.81:0/1068118302 shutdown_connections 2024-06-29T15:57:58.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.120+0000 7fbd5deac640 1 --2- 172.21.15.81:0/1068118302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd580b57b0 0x7fbd581548c0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.120+0000 7fbd5deac640 1 --2- 172.21.15.81:0/1068118302 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbd580b3960 0x7fbd58154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.120+0000 7fbd5deac640 1 --2- 172.21.15.81:0/1068118302 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbd5806ef90 0x7fbd58155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.120+0000 7fbd5deac640 1 -- 172.21.15.81:0/1068118302 >> 172.21.15.81:0/1068118302 conn(0x7fbd580d0930 msgr2=0x7fbd58159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:58.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.120+0000 7fbd5deac640 1 -- 172.21.15.81:0/1068118302 shutdown_connections 2024-06-29T15:57:58.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.120+0000 7fbd5deac640 1 -- 172.21.15.81:0/1068118302 wait complete. 2024-06-29T15:57:58.132 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.248+0000 7f75fb088640 1 -- 172.21.15.81:0/1930576063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75f40b3960 msgr2=0x7f75f40b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.248+0000 7f75fb088640 1 --2- 172.21.15.81:0/1930576063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75f40b3960 0x7f75f40b3d70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f75ec005fd0 tx=0x7f75ec02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.248+0000 7f75fb088640 1 -- 172.21.15.81:0/1930576063 shutdown_connections 2024-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.248+0000 7f75fb088640 1 --2- 172.21.15.81:0/1930576063 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75f40b3960 0x7f75f40b3d70 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.248+0000 7f75fb088640 1 --2- 172.21.15.81:0/1930576063 >> v2:172.21.15.167:3300/0 conn(0x7f75f40b57b0 0x7f75f415aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.248+0000 7f75fb088640 1 -- 172.21.15.81:0/1930576063 >> 172.21.15.81:0/1930576063 conn(0x7f75f40d0930 msgr2=0x7f75f40d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.248+0000 7f75fb088640 1 -- 172.21.15.81:0/1930576063 shutdown_connections 2024-06-29T15:57:58.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.248+0000 7f75fb088640 1 -- 172.21.15.81:0/1930576063 wait complete. 2024-06-29T15:57:58.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.250+0000 7f75fb088640 1 Processor -- start 2024-06-29T15:57:58.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.250+0000 7f75fb088640 1 -- start start 2024-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.251+0000 7f75fb088640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f406ef90 0x7f75f41f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.251+0000 7f75fb088640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75f40b3960 0x7f75f40ad470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.251+0000 7f75fb088640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75f40b57b0 0x7f75f40ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.251+0000 7f75fb088640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f75f40b4430 con 0x7f75f406ef90 2024-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.251+0000 7f75fb088640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f75f40b42b0 con 0x7f75f40b3960 2024-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.251+0000 7f75fb088640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f75f40b45b0 con 0x7f75f40b57b0 2024-06-29T15:57:58.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f8e24640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f406ef90 0x7f75f41f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f8e24640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f406ef90 0x7f75f41f2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35526/0 (socket says 172.21.15.81:35526) 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f8e24640 1 -- 172.21.15.81:0/1130992140 learned_addr learned my addr 172.21.15.81:0/1130992140 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f9625640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75f40b57b0 0x7f75f40ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f3fff640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75f40b3960 0x7f75f40ad470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f8e24640 1 -- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75f40b57b0 msgr2=0x7f75f40ac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f8e24640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75f40b57b0 0x7f75f40ac530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f8e24640 1 -- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75f40b3960 msgr2=0x7f75f40ad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f8e24640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75f40b3960 0x7f75f40ad470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f8e24640 1 -- 172.21.15.81:0/1130992140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f40ab770 con 0x7f75f406ef90 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f8e24640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f406ef90 0x7f75f41f2ea0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f75e400ea20 tx=0x7f75e400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f17fa640 1 -- 172.21.15.81:0/1130992140 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e400ce40 con 0x7f75f406ef90 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f17fa640 1 -- 172.21.15.81:0/1130992140 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75e4004020 con 0x7f75f406ef90 2024-06-29T15:57:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75fb088640 1 -- 172.21.15.81:0/1130992140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f40aba50 con 0x7f75f406ef90 2024-06-29T15:57:58.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75f17fa640 1 -- 172.21.15.81:0/1130992140 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4004470 con 0x7f75f406ef90 2024-06-29T15:57:58.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.252+0000 7f75fb088640 1 -- 172.21.15.81:0/1130992140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f41f33e0 con 0x7f75f406ef90 2024-06-29T15:57:58.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.253+0000 7f75fb088640 1 -- 172.21.15.81:0/1130992140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f406e2a0 con 0x7f75f406ef90 2024-06-29T15:57:58.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.254+0000 7f75f17fa640 1 -- 172.21.15.81:0/1130992140 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 135) ==== 97912+0+0 (secure 0 0 0) 0x7f75e40035f0 con 0x7f75f406ef90 2024-06-29T15:57:58.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.255+0000 7f75f17fa640 1 -- 172.21.15.81:0/1130992140 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 7518+0+0 (secure 0 0 0) 0x7f75e40982c0 con 0x7f75f406ef90 2024-06-29T15:57:58.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.259+0000 7f75f17fa640 1 -- 172.21.15.81:0/1130992140 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f75f406e2a0 con 0x7f75f406ef90 2024-06-29T15:57:58.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.327+0000 7f75f17fa640 1 -- 172.21.15.81:0/1130992140 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 136) ==== 98039+0+0 (secure 0 0 0) 0x7f75e405cac0 con 0x7f75f406ef90 2024-06-29T15:57:58.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.327+0000 7f75f17fa640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f75e007e2d0 0x7f75e00806f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.330+0000 7f75f3fff640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f75e007e2d0 0x7f75e00806f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.331+0000 7f75f3fff640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f75e007e2d0 0x7f75e00806f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f75f406d3d0 tx=0x7f75e8009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:58.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.446+0000 7f75fb088640 1 -- 172.21.15.81:0/1130992140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f75f40b4a90 con 0x7f75f406ef90 2024-06-29T15:57:58.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.447+0000 7f75f17fa640 1 -- 172.21.15.81:0/1130992140 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v219) ==== 107+0+11 (secure 0 0 0) 0x7f75e4061850 con 0x7f75f406ef90 2024-06-29T15:57:58.447 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:57:58.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 -- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f75e007e2d0 msgr2=0x7f75e00806f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f75e007e2d0 0x7f75e00806f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f75f406d3d0 tx=0x7f75e8009040 comp rx=0 tx=0).stop 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 -- 172.21.15.81:0/1130992140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f406ef90 msgr2=0x7f75f41f2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f406ef90 0x7f75f41f2ea0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f75e400ea20 tx=0x7f75e400eee0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 -- 172.21.15.81:0/1130992140 shutdown_connections 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f75e007e2d0 0x7f75e00806f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f75f40b57b0 0x7f75f40ac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75f40b3960 0x7f75f40ad470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 --2- 172.21.15.81:0/1130992140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f406ef90 0x7f75f41f2ea0 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 -- 172.21.15.81:0/1130992140 >> 172.21.15.81:0/1130992140 conn(0x7f75f40d0930 msgr2=0x7f75f4159500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 -- 172.21.15.81:0/1130992140 shutdown_connections 2024-06-29T15:57:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.449+0000 7f75fb088640 1 -- 172.21.15.81:0/1130992140 wait complete. 2024-06-29T15:57:58.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:57:58.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.577+0000 7f748af73640 1 -- 172.21.15.81:0/879777217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b6eb0 msgr2=0x7f74840d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.577+0000 7f748af73640 1 --2- 172.21.15.81:0/879777217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b6eb0 0x7f74840d02d0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f7474009830 tx=0x7f747402a470 comp rx=0 tx=0).stop 2024-06-29T15:57:58.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.577+0000 7f748af73640 1 -- 172.21.15.81:0/879777217 shutdown_connections 2024-06-29T15:57:58.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.577+0000 7f748af73640 1 --2- 172.21.15.81:0/879777217 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74840ce920 0x7f74840cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.577+0000 7f748af73640 1 --2- 172.21.15.81:0/879777217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b6eb0 0x7f74840d02d0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.577+0000 7f748af73640 1 -- 172.21.15.81:0/879777217 >> 172.21.15.81:0/879777217 conn(0x7f74840b3510 msgr2=0x7f74840b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:58.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.577+0000 7f748af73640 1 -- 172.21.15.81:0/879777217 shutdown_connections 2024-06-29T15:57:58.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.577+0000 7f748af73640 1 -- 172.21.15.81:0/879777217 wait complete. 2024-06-29T15:57:58.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.579+0000 7f748af73640 1 Processor -- start 2024-06-29T15:57:58.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.579+0000 7f748af73640 1 -- start start 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f748af73640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b6eb0 0x7f74840af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f748af73640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74840ce510 0x7f748414cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f748af73640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74840ce920 0x7f748415cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f748af73640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74840b50a0 con 0x7f74840b6eb0 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f748af73640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f74840b4f20 con 0x7f74840ce920 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f748af73640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f74840b5220 con 0x7f74840ce510 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f7488d0f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b6eb0 0x7f74840af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f7488d0f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b6eb0 0x7f74840af980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35538/0 (socket says 172.21.15.81:35538) 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f7488d0f640 1 -- 172.21.15.81:0/4094154563 learned_addr learned my addr 172.21.15.81:0/4094154563 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f7489510640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74840ce920 0x7f748415cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f7483fff640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74840ce510 0x7f748414cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.580+0000 7f7488d0f640 1 -- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74840ce510 msgr2=0x7f748414cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f7488d0f640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74840ce510 0x7f748414cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f7488d0f640 1 -- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74840ce920 msgr2=0x7f748415cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f7488d0f640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74840ce920 0x7f748415cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f7488d0f640 1 -- 172.21.15.81:0/4094154563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74840ae890 con 0x7f74840b6eb0 2024-06-29T15:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f7483fff640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74840ce510 0x7f748414cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f7488d0f640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b6eb0 0x7f74840af980 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f7474000c00 tx=0x7f7474033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f74817fa640 1 -- 172.21.15.81:0/4094154563 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7474005c90 con 0x7f74840b6eb0 2024-06-29T15:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f74817fa640 1 -- 172.21.15.81:0/4094154563 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7474031e50 con 0x7f74840b6eb0 2024-06-29T15:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f74817fa640 1 -- 172.21.15.81:0/4094154563 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74740332f0 con 0x7f74840b6eb0 2024-06-29T15:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f748af73640 1 -- 172.21.15.81:0/4094154563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74840aeb70 con 0x7f74840b6eb0 2024-06-29T15:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.581+0000 7f748af73640 1 -- 172.21.15.81:0/4094154563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74840af130 con 0x7f74840b6eb0 2024-06-29T15:57:58.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.582+0000 7f748af73640 1 -- 172.21.15.81:0/4094154563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74840d02d0 con 0x7f74840b6eb0 2024-06-29T15:57:58.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.583+0000 7f74817fa640 1 -- 172.21.15.81:0/4094154563 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 136) ==== 98039+0+0 (secure 0 0 0) 0x7f7474033490 con 0x7f74840b6eb0 2024-06-29T15:57:58.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.583+0000 7f74817fa640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7470076120 0x7f7470078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.583+0000 7f74817fa640 1 -- 172.21.15.81:0/4094154563 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 7518+0+0 (secure 0 0 0) 0x7f74740b8f20 con 0x7f74840b6eb0 2024-06-29T15:57:58.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.583+0000 7f7483fff640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7470076120 0x7f7470078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.584+0000 7f7483fff640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7470076120 0x7f7470078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7478007fd0 tx=0x7f7478009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:58.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.587+0000 7f74817fa640 1 -- 172.21.15.81:0/4094154563 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f747403f030 con 0x7f74840b6eb0 2024-06-29T15:57:58.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.769+0000 7f748af73640 1 -- 172.21.15.81:0/4094154563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f74840cdf00 con 0x7f74840b6eb0 2024-06-29T15:57:58.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.769+0000 7f74817fa640 1 -- 172.21.15.81:0/4094154563 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v219) ==== 103+0+11 (secure 0 0 0) 0x7f74740824b0 con 0x7f74840b6eb0 2024-06-29T15:57:58.770 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:57:58.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 -- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7470076120 msgr2=0x7f7470078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7470076120 0x7f7470078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7478007fd0 tx=0x7f7478009040 comp rx=0 tx=0).stop 2024-06-29T15:57:58.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 -- 172.21.15.81:0/4094154563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b6eb0 msgr2=0x7f74840af980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b6eb0 0x7f74840af980 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f7474000c00 tx=0x7f7474033c90 comp rx=0 tx=0).stop 2024-06-29T15:57:58.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 -- 172.21.15.81:0/4094154563 shutdown_connections 2024-06-29T15:57:58.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7470076120 0x7f7470078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f74840ce920 0x7f748415cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f74840ce510 0x7f748414cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 --2- 172.21.15.81:0/4094154563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74840b6eb0 0x7f74840af980 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 -- 172.21.15.81:0/4094154563 >> 172.21.15.81:0/4094154563 conn(0x7f74840b3510 msgr2=0x7f74840cd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:58.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 -- 172.21.15.81:0/4094154563 shutdown_connections 2024-06-29T15:57:58.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.772+0000 7f748af73640 1 -- 172.21.15.81:0/4094154563 wait complete. 2024-06-29T15:57:58.783 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:58.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.896+0000 7f93711eb640 1 -- 172.21.15.81:0/2067213918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f936c0b9040 msgr2=0x7f936c0d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.896+0000 7f93711eb640 1 --2- 172.21.15.81:0/2067213918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f936c0b9040 0x7f936c0d0480 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f936c08e330 tx=0x7f935c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.896+0000 7f93711eb640 1 -- 172.21.15.81:0/2067213918 shutdown_connections 2024-06-29T15:57:58.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.896+0000 7f93711eb640 1 --2- 172.21.15.81:0/2067213918 >> v2:172.21.15.167:3300/0 conn(0x7f936c0d2a90 0x7f936c0d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.896+0000 7f93711eb640 1 --2- 172.21.15.81:0/2067213918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f936c0b9040 0x7f936c0d0480 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.896+0000 7f93711eb640 1 -- 172.21.15.81:0/2067213918 >> 172.21.15.81:0/2067213918 conn(0x7f936c09cf30 msgr2=0x7f936c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:58.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.896+0000 7f93711eb640 1 -- 172.21.15.81:0/2067213918 shutdown_connections 2024-06-29T15:57:58.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.896+0000 7f93711eb640 1 -- 172.21.15.81:0/2067213918 wait complete. 2024-06-29T15:57:58.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.898+0000 7f93711eb640 1 Processor -- start 2024-06-29T15:57:58.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.898+0000 7f93711eb640 1 -- start start 2024-06-29T15:57:58.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.899+0000 7f93711eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f936c0b5250 0x7f936c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.899+0000 7f93711eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f936c0b9040 0x7f936c14e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f93711eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f936c0d2a90 0x7f936c151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f93711eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f936c0b7d30 con 0x7f936c0b9040 2024-06-29T15:57:58.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f93711eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f936c0b7bb0 con 0x7f936c0b5250 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f93711eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f936c0b7eb0 con 0x7f936c0d2a90 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f936c0b9040 0x7f936c14e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f936c0b9040 0x7f936c14e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35546/0 (socket says 172.21.15.81:35546) 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936259c640 1 -- 172.21.15.81:0/48994988 learned_addr learned my addr 172.21.15.81:0/48994988 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936ad9d640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f936c0b5250 0x7f936c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936b59e640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f936c0d2a90 0x7f936c151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936259c640 1 -- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f936c0d2a90 msgr2=0x7f936c151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936259c640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f936c0d2a90 0x7f936c151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936259c640 1 -- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f936c0b5250 msgr2=0x7f936c0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936259c640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f936c0b5250 0x7f936c0b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936259c640 1 -- 172.21.15.81:0/48994988 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936c14f110 con 0x7f936c0b9040 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936ad9d640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f936c0b5250 0x7f936c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:57:58.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f936259c640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f936c0b9040 0x7f936c14e950 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f935400c3f0 tx=0x7f935400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:58.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f9363fff640 1 -- 172.21.15.81:0/48994988 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9354017070 con 0x7f936c0b9040 2024-06-29T15:57:58.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f9363fff640 1 -- 172.21.15.81:0/48994988 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93540035f0 con 0x7f936c0b9040 2024-06-29T15:57:58.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f93711eb640 1 -- 172.21.15.81:0/48994988 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f936c1520a0 con 0x7f936c0b9040 2024-06-29T15:57:58.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f9363fff640 1 -- 172.21.15.81:0/48994988 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9354003a40 con 0x7f936c0b9040 2024-06-29T15:57:58.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.900+0000 7f93711eb640 1 -- 172.21.15.81:0/48994988 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f936c14f920 con 0x7f936c0b9040 2024-06-29T15:57:58.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.902+0000 7f93711eb640 1 -- 172.21.15.81:0/48994988 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f936c0b8440 con 0x7f936c0b9040 2024-06-29T15:57:58.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.902+0000 7f9363fff640 1 -- 172.21.15.81:0/48994988 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 136) ==== 98039+0+0 (secure 0 0 0) 0x7f935401a070 con 0x7f936c0b9040 2024-06-29T15:57:58.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.903+0000 7f9363fff640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f934c076120 0x7f934c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:58.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.903+0000 7f9363fff640 1 -- 172.21.15.81:0/48994988 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 7518+0+0 (secure 0 0 0) 0x7f9354096ea0 con 0x7f936c0b9040 2024-06-29T15:57:58.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.903+0000 7f936ad9d640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f934c076120 0x7f934c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:58.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.903+0000 7f936ad9d640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f934c076120 0x7f934c078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f935c034000 tx=0x7f935c004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:58.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:58.907+0000 7f9363fff640 1 -- 172.21.15.81:0/48994988 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9354060430 con 0x7f936c0b9040 2024-06-29T15:57:59.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.122+0000 7f93711eb640 1 -- 172.21.15.81:0/48994988 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f936c068860 con 0x7f936c0b9040 2024-06-29T15:57:59.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.123+0000 7f9363fff640 1 -- 172.21.15.81:0/48994988 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+6565 (secure 0 0 0) 0x7f9354068420 con 0x7f936c0b9040 2024-06-29T15:57:59.123 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:59.123 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":118,"btime":"2024-06-29T15:57:37:378022+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.637042+0000","modified":"2024-06-29T15:57:24.656503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11226":{"gid":11226,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/739691411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":739691411},{"type":"v1","addr":"172.21.15.81:6833","nonce":739691411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11226,"qdb_cluster":[11226]},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:33.727815+0000","modified":"2024-06-29T15:57:36.736776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10114":{"gid":10114,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10114,"qdb_cluster":[10114]},"id":14}]} 2024-06-29T15:57:59.124 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 118 2024-06-29T15:57:59.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 -- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f934c076120 msgr2=0x7f934c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f934c076120 0x7f934c078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f935c034000 tx=0x7f935c004730 comp rx=0 tx=0).stop 2024-06-29T15:57:59.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 -- 172.21.15.81:0/48994988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f936c0b9040 msgr2=0x7f936c14e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f936c0b9040 0x7f936c14e950 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f935400c3f0 tx=0x7f935400c8b0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 -- 172.21.15.81:0/48994988 shutdown_connections 2024-06-29T15:57:59.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f934c076120 0x7f934c078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f936c0d2a90 0x7f936c151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f936c0b9040 0x7f936c14e950 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 --2- 172.21.15.81:0/48994988 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f936c0b5250 0x7f936c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 -- 172.21.15.81:0/48994988 >> 172.21.15.81:0/48994988 conn(0x7f936c09cf30 msgr2=0x7f936c0cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 -- 172.21.15.81:0/48994988 shutdown_connections 2024-06-29T15:57:59.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.125+0000 7f93711eb640 1 -- 172.21.15.81:0/48994988 wait complete. 2024-06-29T15:57:59.136 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:57:59.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.249+0000 7ffa24e47640 1 -- 172.21.15.81:0/771126673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa0c004800 msgr2=0x7ffa0c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.249+0000 7ffa24e47640 1 --2- 172.21.15.81:0/771126673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa0c004800 0x7ffa0c004c10 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7ffa08009b50 tx=0x7ffa0c005830 comp rx=0 tx=0).stop 2024-06-29T15:57:59.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.249+0000 7ffa24e47640 1 -- 172.21.15.81:0/771126673 shutdown_connections 2024-06-29T15:57:59.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.249+0000 7ffa24e47640 1 --2- 172.21.15.81:0/771126673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa0c004800 0x7ffa0c004c10 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.249+0000 7ffa24e47640 1 -- 172.21.15.81:0/771126673 >> 172.21.15.81:0/771126673 conn(0x7ffa2006ccc0 msgr2=0x7ffa200b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.249+0000 7ffa24e47640 1 -- 172.21.15.81:0/771126673 shutdown_connections 2024-06-29T15:57:59.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.249+0000 7ffa24e47640 1 -- 172.21.15.81:0/771126673 wait complete. 2024-06-29T15:57:59.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.251+0000 7ffa24e47640 1 Processor -- start 2024-06-29T15:57:59.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.251+0000 7ffa24e47640 1 -- start start 2024-06-29T15:57:59.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.252+0000 7ffa24e47640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa0c004800 0x7ffa2015a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.252+0000 7ffa24e47640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa2013aa90 0x7ffa20143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.252+0000 7ffa24e47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa20144190 0x7ffa201386a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.252+0000 7ffa24e47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffa200737d0 con 0x7ffa20144190 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.252+0000 7ffa24e47640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ffa20073650 con 0x7ffa0c004800 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.252+0000 7ffa24e47640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ffa2015ed70 con 0x7ffa2013aa90 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa1ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa20144190 0x7ffa201386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa1ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa20144190 0x7ffa201386a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35562/0 (socket says 172.21.15.81:35562) 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa1ed9d640 1 -- 172.21.15.81:0/3165695948 learned_addr learned my addr 172.21.15.81:0/3165695948 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa1e59c640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa0c004800 0x7ffa2015a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa15d9b640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa2013aa90 0x7ffa20143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa1ed9d640 1 -- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa2013aa90 msgr2=0x7ffa20143c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa1ed9d640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa2013aa90 0x7ffa20143c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa1ed9d640 1 -- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa0c004800 msgr2=0x7ffa2015a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa1ed9d640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa0c004800 0x7ffa2015a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa1ed9d640 1 -- 172.21.15.81:0/3165695948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa201f1f80 con 0x7ffa20144190 2024-06-29T15:57:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa15d9b640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa2013aa90 0x7ffa20143c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa1ed9d640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa20144190 0x7ffa201386a0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7ffa18002230 tx=0x7ffa18006420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa177fe640 1 -- 172.21.15.81:0/3165695948 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18010690 con 0x7ffa20144190 2024-06-29T15:57:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa177fe640 1 -- 172.21.15.81:0/3165695948 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa18010920 con 0x7ffa20144190 2024-06-29T15:57:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa24e47640 1 -- 172.21.15.81:0/3165695948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa200aeba0 con 0x7ffa20144190 2024-06-29T15:57:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa177fe640 1 -- 172.21.15.81:0/3165695948 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18010d70 con 0x7ffa20144190 2024-06-29T15:57:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.253+0000 7ffa24e47640 1 -- 172.21.15.81:0/3165695948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa2013da40 con 0x7ffa20144190 2024-06-29T15:57:59.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.254+0000 7ffa24e47640 1 -- 172.21.15.81:0/3165695948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa200cef90 con 0x7ffa20144190 2024-06-29T15:57:59.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.255+0000 7ffa177fe640 1 -- 172.21.15.81:0/3165695948 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 136) ==== 98039+0+0 (secure 0 0 0) 0x7ffa18019020 con 0x7ffa20144190 2024-06-29T15:57:59.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.255+0000 7ffa177fe640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ffa00076120 0x7ffa00078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.255+0000 7ffa177fe640 1 -- 172.21.15.81:0/3165695948 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 7518+0+0 (secure 0 0 0) 0x7ffa18097e50 con 0x7ffa20144190 2024-06-29T15:57:59.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.256+0000 7ffa1e59c640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ffa00076120 0x7ffa00078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.256+0000 7ffa1e59c640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ffa00076120 0x7ffa00078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffa10000c00 tx=0x7ffa10006b20 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:59.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.260+0000 7ffa177fe640 1 -- 172.21.15.81:0/3165695948 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ffa200aeba0 con 0x7ffa20144190 2024-06-29T15:57:59.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.444+0000 7ffa24e47640 1 -- 172.21.15.81:0/3165695948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa200afad0 con 0x7ffa20144190 2024-06-29T15:57:59.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.446+0000 7ffa177fe640 1 -- 172.21.15.81:0/3165695948 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) ==== 74+0+21870 (secure 0 0 0) 0x7ffa201f1f80 con 0x7ffa20144190 2024-06-29T15:57:59.446 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:59.446 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":219,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:57.317173+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.259110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:32.675276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2503226350":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/2924173136":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6833/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6832/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3022587441":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/575713984":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/1101183266":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3261132296":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3042354413":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1619575755":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/760717235":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/3781943725":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/376674682":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1164120698":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3640842563":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/2165320595":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3538782057":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6833/2018755589":"2024-06-30T15:57:48.595362+0000","172.21.15.167:6832/2018755589":"2024-06-30T15:57:48.595362+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:57:59.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 -- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ffa00076120 msgr2=0x7ffa00078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ffa00076120 0x7ffa00078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ffa10000c00 tx=0x7ffa10006b20 comp rx=0 tx=0).stop 2024-06-29T15:57:59.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 -- 172.21.15.81:0/3165695948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa20144190 msgr2=0x7ffa201386a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa20144190 0x7ffa201386a0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7ffa18002230 tx=0x7ffa18006420 comp rx=0 tx=0).stop 2024-06-29T15:57:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 -- 172.21.15.81:0/3165695948 shutdown_connections 2024-06-29T15:57:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ffa00076120 0x7ffa00078630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa20144190 0x7ffa201386a0 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ffa2013aa90 0x7ffa20143c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 --2- 172.21.15.81:0/3165695948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ffa0c004800 0x7ffa2015a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 -- 172.21.15.81:0/3165695948 >> 172.21.15.81:0/3165695948 conn(0x7ffa2006ccc0 msgr2=0x7ffa2015b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 -- 172.21.15.81:0/3165695948 shutdown_connections 2024-06-29T15:57:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.448+0000 7ffa24e47640 1 -- 172.21.15.81:0/3165695948 wait complete. 2024-06-29T15:57:59.460 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:57:59.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:59.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.574+0000 7f6b136e4640 1 -- 172.21.15.81:0/2272972306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0b57b0 msgr2=0x7f6b0c15aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.574+0000 7f6b136e4640 1 --2- 172.21.15.81:0/2272972306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0b57b0 0x7f6b0c15aed0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f6b08009830 tx=0x7f6b0802a510 comp rx=0 tx=0).stop 2024-06-29T15:57:59.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.574+0000 7f6b136e4640 1 -- 172.21.15.81:0/2272972306 shutdown_connections 2024-06-29T15:57:59.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.574+0000 7f6b136e4640 1 --2- 172.21.15.81:0/2272972306 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b0c06ef90 0x7f6b0c06f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.574+0000 7f6b136e4640 1 --2- 172.21.15.81:0/2272972306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0b57b0 0x7f6b0c15aed0 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.574+0000 7f6b136e4640 1 -- 172.21.15.81:0/2272972306 >> 172.21.15.81:0/2272972306 conn(0x7f6b0c0d0930 msgr2=0x7f6b0c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.574+0000 7f6b136e4640 1 -- 172.21.15.81:0/2272972306 shutdown_connections 2024-06-29T15:57:59.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.574+0000 7f6b136e4640 1 -- 172.21.15.81:0/2272972306 wait complete. 2024-06-29T15:57:59.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.576+0000 7f6b136e4640 1 Processor -- start 2024-06-29T15:57:59.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.576+0000 7f6b136e4640 1 -- start start 2024-06-29T15:57:59.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b136e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b0c06ef90 0x7f6b0c150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b136e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0b57b0 0x7f6b0c14fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b136e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b0c0ce1d0 0x7f6b0c0b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b136e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c0cecf0 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b136e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c0b4b80 con 0x7f6b0c06ef90 2024-06-29T15:57:59.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b136e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6b0c0cee70 con 0x7f6b0c0ce1d0 2024-06-29T15:57:59.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b10c7f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0b57b0 0x7f6b0c14fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b10c7f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0b57b0 0x7f6b0c14fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35572/0 (socket says 172.21.15.81:35572) 2024-06-29T15:57:59.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b10c7f640 1 -- 172.21.15.81:0/652532481 learned_addr learned my addr 172.21.15.81:0/652532481 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:59.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b11c81640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b0c0ce1d0 0x7f6b0c0b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b11480640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b0c06ef90 0x7f6b0c150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b10c7f640 1 -- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b0c0ce1d0 msgr2=0x7f6b0c0b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b10c7f640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b0c0ce1d0 0x7f6b0c0b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b10c7f640 1 -- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b0c06ef90 msgr2=0x7f6b0c150880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b10c7f640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b0c06ef90 0x7f6b0c150880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b10c7f640 1 -- 172.21.15.81:0/652532481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0c150130 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.577+0000 7f6b11c81640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b0c0ce1d0 0x7f6b0c0b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:57:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.578+0000 7f6b10c7f640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0b57b0 0x7f6b0c14fbf0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00c970 tx=0x7f6afc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.578+0000 7f6b01ffb640 1 -- 172.21.15.81:0/652532481 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc007bb0 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.578+0000 7f6b01ffb640 1 -- 172.21.15.81:0/652532481 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6afc007e40 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.578+0000 7f6b136e4640 1 -- 172.21.15.81:0/652532481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b0c0c58d0 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.578+0000 7f6b01ffb640 1 -- 172.21.15.81:0/652532481 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc00f6b0 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.578+0000 7f6b136e4640 1 -- 172.21.15.81:0/652532481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b0c0ae060 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.579+0000 7f6b136e4640 1 -- 172.21.15.81:0/652532481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b0c06e2a0 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.579+0000 7f6b01ffb640 1 -- 172.21.15.81:0/652532481 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 136) ==== 98039+0+0 (secure 0 0 0) 0x7f6afc00f850 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.580+0000 7f6b01ffb640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6af0076120 0x7f6af0078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.580+0000 7f6b01ffb640 1 -- 172.21.15.81:0/652532481 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 7518+0+0 (secure 0 0 0) 0x7f6afc098220 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.580+0000 7f6b11480640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6af0076120 0x7f6af0078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.581+0000 7f6b11480640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6af0076120 0x7f6af0078630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6b08034000 tx=0x7f6b08004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:59.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.584+0000 7f6b01ffb640 1 -- 172.21.15.81:0/652532481 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6b0c06e2a0 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.695+0000 7f6b01ffb640 1 -- 172.21.15.81:0/652532481 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 137) ==== 98349+0+0 (secure 0 0 0) 0x7f6afc05d2c0 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.804+0000 7f6b136e4640 1 -- 172.21.15.81:0/652532481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b0c06f3a0 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.805+0000 7f6b01ffb640 1 -- 172.21.15.81:0/652532481 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+6565 (secure 0 0 0) 0x7f6b0c150130 con 0x7f6b0c0b57b0 2024-06-29T15:57:59.805 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:57:59.805 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":118,"btime":"2024-06-29T15:57:37:378022+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.637042+0000","modified":"2024-06-29T15:57:24.656503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11226":{"gid":11226,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/739691411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":739691411},{"type":"v1","addr":"172.21.15.81:6833","nonce":739691411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11226,"qdb_cluster":[11226]},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:33.727815+0000","modified":"2024-06-29T15:57:36.736776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10114":{"gid":10114,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10114,"qdb_cluster":[10114]},"id":14}]} 2024-06-29T15:57:59.806 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 118 2024-06-29T15:57:59.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 -- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6af0076120 msgr2=0x7f6af0078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6af0076120 0x7f6af0078630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6b08034000 tx=0x7f6b08004730 comp rx=0 tx=0).stop 2024-06-29T15:57:59.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 -- 172.21.15.81:0/652532481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0b57b0 msgr2=0x7f6b0c14fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0b57b0 0x7f6b0c14fbf0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f6afc00c970 tx=0x7f6afc00ce30 comp rx=0 tx=0).stop 2024-06-29T15:57:59.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 -- 172.21.15.81:0/652532481 shutdown_connections 2024-06-29T15:57:59.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6af0076120 0x7f6af0078630 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b0c0ce1d0 0x7f6b0c0b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0b57b0 0x7f6b0c14fbf0 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 --2- 172.21.15.81:0/652532481 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b0c06ef90 0x7f6b0c150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 -- 172.21.15.81:0/652532481 >> 172.21.15.81:0/652532481 conn(0x7f6b0c0d0930 msgr2=0x7f6b0c0cf2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 -- 172.21.15.81:0/652532481 shutdown_connections 2024-06-29T15:57:59.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.807+0000 7f6b136e4640 1 -- 172.21.15.81:0/652532481 wait complete. 2024-06-29T15:57:59.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:57:59.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.930+0000 7f6b79081640 1 -- 172.21.15.81:0/1346847284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b740ce510 msgr2=0x7f6b740ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.930+0000 7f6b79081640 1 --2- 172.21.15.81:0/1346847284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b740ce510 0x7f6b740ce920 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f6b740b41b0 tx=0x7f6b5c02a470 comp rx=0 tx=0).stop 2024-06-29T15:57:59.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.930+0000 7f6b79081640 1 -- 172.21.15.81:0/1346847284 shutdown_connections 2024-06-29T15:57:59.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.930+0000 7f6b79081640 1 --2- 172.21.15.81:0/1346847284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b740ce510 0x7f6b740ce920 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.930+0000 7f6b79081640 1 --2- 172.21.15.81:0/1346847284 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740b6eb0 0x7f6b740d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.930+0000 7f6b79081640 1 -- 172.21.15.81:0/1346847284 >> 172.21.15.81:0/1346847284 conn(0x7f6b740b3510 msgr2=0x7f6b740b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:57:59.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.930+0000 7f6b79081640 1 -- 172.21.15.81:0/1346847284 shutdown_connections 2024-06-29T15:57:59.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.930+0000 7f6b79081640 1 -- 172.21.15.81:0/1346847284 wait complete. 2024-06-29T15:57:59.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.932+0000 7f6b79081640 1 Processor -- start 2024-06-29T15:57:59.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.932+0000 7f6b79081640 1 -- start start 2024-06-29T15:57:59.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.933+0000 7f6b79081640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740b4810 0x7f6b741f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.933+0000 7f6b79081640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b740b6eb0 0x7f6b7415b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.933+0000 7f6b79081640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740ce510 0x7f6b74156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b79081640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6b740cd480 con 0x7f6b740b6eb0 2024-06-29T15:57:59.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b79081640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6b740b29c0 con 0x7f6b740b4810 2024-06-29T15:57:59.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b79081640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6b740cd600 con 0x7f6b740ce510 2024-06-29T15:57:59.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b740b6eb0 0x7f6b7415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b740b6eb0 0x7f6b7415b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35576/0 (socket says 172.21.15.81:35576) 2024-06-29T15:57:59.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6bfff640 1 -- 172.21.15.81:0/1374916935 learned_addr learned my addr 172.21.15.81:0/1374916935 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:57:59.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b7359e640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740ce510 0x7f6b74156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b72d9d640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740b4810 0x7f6b741f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6bfff640 1 -- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740ce510 msgr2=0x7f6b74156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6bfff640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740ce510 0x7f6b74156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6bfff640 1 -- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740b4810 msgr2=0x7f6b741f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:57:59.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6bfff640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740b4810 0x7f6b741f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:57:59.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6bfff640 1 -- 172.21.15.81:0/1374916935 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b740ca540 con 0x7f6b740b6eb0 2024-06-29T15:57:59.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6bfff640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b740b6eb0 0x7f6b7415b520 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009780 tx=0x7f6b5c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:59.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6b7fe640 1 -- 172.21.15.81:0/1374916935 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c031d60 con 0x7f6b740b6eb0 2024-06-29T15:57:59.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6b7fe640 1 -- 172.21.15.81:0/1374916935 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b5c033e10 con 0x7f6b740b6eb0 2024-06-29T15:57:59.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b79081640 1 -- 172.21.15.81:0/1374916935 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b740ca820 con 0x7f6b740b6eb0 2024-06-29T15:57:59.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b6b7fe640 1 -- 172.21.15.81:0/1374916935 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c005260 con 0x7f6b740b6eb0 2024-06-29T15:57:59.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.934+0000 7f6b79081640 1 -- 172.21.15.81:0/1374916935 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b7415f920 con 0x7f6b740b6eb0 2024-06-29T15:57:59.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.935+0000 7f6b79081640 1 -- 172.21.15.81:0/1374916935 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b740d02d0 con 0x7f6b740b6eb0 2024-06-29T15:57:59.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.936+0000 7f6b6b7fe640 1 -- 172.21.15.81:0/1374916935 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 137) ==== 98349+0+0 (secure 0 0 0) 0x7f6b5c005400 con 0x7f6b740b6eb0 2024-06-29T15:57:59.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.937+0000 7f6b6b7fe640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6b540764b0 0x7f6b540789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:57:59.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.937+0000 7f6b6b7fe640 1 -- 172.21.15.81:0/1374916935 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 7518+0+0 (secure 0 0 0) 0x7f6b5c0b98a0 con 0x7f6b740b6eb0 2024-06-29T15:57:59.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.937+0000 7f6b72d9d640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6b540764b0 0x7f6b540789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:57:59.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.938+0000 7f6b72d9d640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6b540764b0 0x7f6b540789c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b64005fd0 tx=0x7f6b64005850 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:57:59.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:57:59.940+0000 7f6b6b7fe640 1 -- 172.21.15.81:0/1374916935 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6b5c03f030 con 0x7f6b740b6eb0 2024-06-29T15:58:00.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.155+0000 7f6b79081640 1 -- 172.21.15.81:0/1374916935 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b740cdf00 con 0x7f6b740b6eb0 2024-06-29T15:58:00.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.155+0000 7f6b6b7fe640 1 -- 172.21.15.81:0/1374916935 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+6565 (secure 0 0 0) 0x7f6b5c082760 con 0x7f6b740b6eb0 2024-06-29T15:58:00.157 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:00.157 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":118,"btime":"2024-06-29T15:57:37:378022+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:21.637042+0000","modified":"2024-06-29T15:57:24.656503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11226":{"gid":11226,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6833/739691411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":739691411},{"type":"v1","addr":"172.21.15.81:6833","nonce":739691411}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11226,"qdb_cluster":[11226]},"id":13},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:33.727815+0000","modified":"2024-06-29T15:57:36.736776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10114":{"gid":10114,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10114,"qdb_cluster":[10114]},"id":14}]} 2024-06-29T15:58:00.157 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 118 2024-06-29T15:58:00.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.157+0000 7f6b79081640 1 -- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6b540764b0 msgr2=0x7f6b540789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.157+0000 7f6b79081640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6b540764b0 0x7f6b540789c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b64005fd0 tx=0x7f6b64005850 comp rx=0 tx=0).stop 2024-06-29T15:58:00.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.157+0000 7f6b79081640 1 -- 172.21.15.81:0/1374916935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b740b6eb0 msgr2=0x7f6b7415b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.157+0000 7f6b79081640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b740b6eb0 0x7f6b7415b520 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009780 tx=0x7f6b5c033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:00.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.157+0000 7f6b79081640 1 -- 172.21.15.81:0/1374916935 shutdown_connections 2024-06-29T15:58:00.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.157+0000 7f6b79081640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6b540764b0 0x7f6b540789c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.157+0000 7f6b79081640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6b740ce510 0x7f6b74156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.157+0000 7f6b79081640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b740b6eb0 0x7f6b7415b520 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.158+0000 7f6b79081640 1 --2- 172.21.15.81:0/1374916935 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6b740b4810 0x7f6b741f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.158+0000 7f6b79081640 1 -- 172.21.15.81:0/1374916935 >> 172.21.15.81:0/1374916935 conn(0x7f6b740b3510 msgr2=0x7f6b740b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.158+0000 7f6b79081640 1 -- 172.21.15.81:0/1374916935 shutdown_connections 2024-06-29T15:58:00.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.158+0000 7f6b79081640 1 -- 172.21.15.81:0/1374916935 wait complete. 2024-06-29T15:58:00.169 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:00.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.281+0000 7f6513017640 1 -- 172.21.15.81:0/2778161102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f650c073740 msgr2=0x7f650c15e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.281+0000 7f6513017640 1 --2- 172.21.15.81:0/2778161102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f650c073740 0x7f650c15e960 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f650c070e50 tx=0x7f650402aac0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.281+0000 7f6513017640 1 -- 172.21.15.81:0/2778161102 shutdown_connections 2024-06-29T15:58:00.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.281+0000 7f6513017640 1 --2- 172.21.15.81:0/2778161102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f650c073740 0x7f650c15e960 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.281+0000 7f6513017640 1 --2- 172.21.15.81:0/2778161102 >> v2:172.21.15.167:3300/0 conn(0x7f650c1412f0 0x7f650c0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.281+0000 7f6513017640 1 -- 172.21.15.81:0/2778161102 >> 172.21.15.81:0/2778161102 conn(0x7f650c06ccc0 msgr2=0x7f650c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.281+0000 7f6513017640 1 -- 172.21.15.81:0/2778161102 shutdown_connections 2024-06-29T15:58:00.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.281+0000 7f6513017640 1 -- 172.21.15.81:0/2778161102 wait complete. 2024-06-29T15:58:00.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.283+0000 7f6513017640 1 Processor -- start 2024-06-29T15:58:00.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.283+0000 7f6513017640 1 -- start start 2024-06-29T15:58:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6513017640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f650c0cf4d0 0x7f650c156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6513017640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f650c1412f0 0x7f650c155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6513017640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f650c155760 0x7f650c155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6513017640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f650c0b3210 con 0x7f650c0cf4d0 2024-06-29T15:58:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6513017640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f650c15f270 con 0x7f650c1412f0 2024-06-29T15:58:00.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6513017640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f650c0b3390 con 0x7f650c155760 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6510db3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f650c0cf4d0 0x7f650c156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6510db3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f650c0cf4d0 0x7f650c156740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47272/0 (socket says 172.21.15.81:47272) 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6510db3640 1 -- 172.21.15.81:0/2721760984 learned_addr learned my addr 172.21.15.81:0/2721760984 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6503fff640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f650c1412f0 0x7f650c155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f65115b4640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f650c155760 0x7f650c155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6510db3640 1 -- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f650c155760 msgr2=0x7f650c155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6510db3640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f650c155760 0x7f650c155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6510db3640 1 -- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f650c1412f0 msgr2=0x7f650c155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6510db3640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f650c1412f0 0x7f650c155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6510db3640 1 -- 172.21.15.81:0/2721760984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650c12e000 con 0x7f650c0cf4d0 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f65115b4640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f650c155760 0x7f650c155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6503fff640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f650c1412f0 0x7f650c155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:00.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6510db3640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f650c0cf4d0 0x7f650c156740 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f650800d770 tx=0x7f650800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f65017fa640 1 -- 172.21.15.81:0/2721760984 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508017070 con 0x7f650c0cf4d0 2024-06-29T15:58:00.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f65017fa640 1 -- 172.21.15.81:0/2721760984 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6508009d20 con 0x7f650c0cf4d0 2024-06-29T15:58:00.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6513017640 1 -- 172.21.15.81:0/2721760984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f650c12e270 con 0x7f650c0cf4d0 2024-06-29T15:58:00.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f65017fa640 1 -- 172.21.15.81:0/2721760984 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080106c0 con 0x7f650c0cf4d0 2024-06-29T15:58:00.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.285+0000 7f6513017640 1 -- 172.21.15.81:0/2721760984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f650c1e46b0 con 0x7f650c0cf4d0 2024-06-29T15:58:00.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.287+0000 7f6513017640 1 -- 172.21.15.81:0/2721760984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f650c0b38b0 con 0x7f650c0cf4d0 2024-06-29T15:58:00.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.287+0000 7f65017fa640 1 -- 172.21.15.81:0/2721760984 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 137) ==== 98349+0+0 (secure 0 0 0) 0x7f650801a070 con 0x7f650c0cf4d0 2024-06-29T15:58:00.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.288+0000 7f65017fa640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64f80764b0 0x7f64f80789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.288+0000 7f65017fa640 1 -- 172.21.15.81:0/2721760984 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 7518+0+0 (secure 0 0 0) 0x7f65080983b0 con 0x7f650c0cf4d0 2024-06-29T15:58:00.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.288+0000 7f6503fff640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64f80764b0 0x7f64f80789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.288+0000 7f6503fff640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64f80764b0 0x7f64f80789c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f64fc007fd0 tx=0x7f64fc009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.292+0000 7f65017fa640 1 -- 172.21.15.81:0/2721760984 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f65080611e0 con 0x7f650c0cf4d0 2024-06-29T15:58:00.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.476+0000 7f6513017640 1 -- 172.21.15.81:0/2721760984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f650c068c10 con 0x7f650c0cf4d0 2024-06-29T15:58:00.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.477+0000 7f65017fa640 1 -- 172.21.15.81:0/2721760984 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) ==== 74+0+21870 (secure 0 0 0) 0x7f6508060ae0 con 0x7f650c0cf4d0 2024-06-29T15:58:00.478 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:00.478 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":219,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:57:57.317173+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:57:19.937473+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"212","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-06-29T15:57:20.596779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"213","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.259110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:32.675276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/2503226350":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/2924173136":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6833/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6832/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3022587441":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/575713984":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/1101183266":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3261132296":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3042354413":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1619575755":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/760717235":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/3781943725":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/376674682":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1164120698":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3640842563":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/2165320595":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3538782057":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6833/2018755589":"2024-06-30T15:57:48.595362+0000","172.21.15.167:6832/2018755589":"2024-06-30T15:57:48.595362+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:00.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 -- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64f80764b0 msgr2=0x7f64f80789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64f80764b0 0x7f64f80789c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f64fc007fd0 tx=0x7f64fc009040 comp rx=0 tx=0).stop 2024-06-29T15:58:00.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 -- 172.21.15.81:0/2721760984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f650c0cf4d0 msgr2=0x7f650c156740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f650c0cf4d0 0x7f650c156740 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f650800d770 tx=0x7f650800dc30 comp rx=0 tx=0).stop 2024-06-29T15:58:00.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 -- 172.21.15.81:0/2721760984 shutdown_connections 2024-06-29T15:58:00.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64f80764b0 0x7f64f80789c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f650c155760 0x7f650c155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f650c1412f0 0x7f650c155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 --2- 172.21.15.81:0/2721760984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f650c0cf4d0 0x7f650c156740 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 -- 172.21.15.81:0/2721760984 >> 172.21.15.81:0/2721760984 conn(0x7f650c06ccc0 msgr2=0x7f650c0ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 -- 172.21.15.81:0/2721760984 shutdown_connections 2024-06-29T15:58:00.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.479+0000 7f6513017640 1 -- 172.21.15.81:0/2721760984 wait complete. 2024-06-29T15:58:00.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:58:00.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.605+0000 7fda615ef640 1 -- 172.21.15.81:0/2117424313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda5c0d0b20 msgr2=0x7fda5c0cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.605+0000 7fda615ef640 1 --2- 172.21.15.81:0/2117424313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda5c0d0b20 0x7fda5c0cf5c0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7fda54005fd0 tx=0x7fda5402a5c0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.605+0000 7fda615ef640 1 -- 172.21.15.81:0/2117424313 shutdown_connections 2024-06-29T15:58:00.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.605+0000 7fda615ef640 1 --2- 172.21.15.81:0/2117424313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda5c0d0b20 0x7fda5c0cf5c0 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.605+0000 7fda615ef640 1 --2- 172.21.15.81:0/2117424313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda5c0b8ba0 0x7fda5c0d46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.605+0000 7fda615ef640 1 -- 172.21.15.81:0/2117424313 >> 172.21.15.81:0/2117424313 conn(0x7fda5c0d3bd0 msgr2=0x7fda5c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.605+0000 7fda615ef640 1 -- 172.21.15.81:0/2117424313 shutdown_connections 2024-06-29T15:58:00.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.605+0000 7fda615ef640 1 -- 172.21.15.81:0/2117424313 wait complete. 2024-06-29T15:58:00.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.607+0000 7fda615ef640 1 Processor -- start 2024-06-29T15:58:00.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.607+0000 7fda615ef640 1 -- start start 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.608+0000 7fda615ef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda5c0b6a20 0x7fda5c0b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.608+0000 7fda615ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda5c0b8ba0 0x7fda5c09df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.608+0000 7fda615ef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda5c09e4b0 0x7fda5c09e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.608+0000 7fda615ef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c0b9410 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.608+0000 7fda615ef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c0b9290 con 0x7fda5c09e4b0 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.608+0000 7fda615ef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fda5c0cfb00 con 0x7fda5c0b6a20 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.608+0000 7fda5a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda5c0b8ba0 0x7fda5c09df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.608+0000 7fda5a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda5c0b8ba0 0x7fda5c09df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47292/0 (socket says 172.21.15.81:47292) 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.608+0000 7fda5a7fc640 1 -- 172.21.15.81:0/2010696116 learned_addr learned my addr 172.21.15.81:0/2010696116 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda5b7fe640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda5c09e4b0 0x7fda5c09e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda5affd640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda5c0b6a20 0x7fda5c0b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda5a7fc640 1 -- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda5c0b6a20 msgr2=0x7fda5c0b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda5a7fc640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda5c0b6a20 0x7fda5c0b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda5a7fc640 1 -- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda5c09e4b0 msgr2=0x7fda5c09e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda5a7fc640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda5c09e4b0 0x7fda5c09e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda5a7fc640 1 -- 172.21.15.81:0/2010696116 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c09ee00 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda5a7fc640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda5c0b8ba0 0x7fda5c09df70 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7fda4400b3c0 tx=0x7fda4400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda537fe640 1 -- 172.21.15.81:0/2010696116 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44015020 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda537fe640 1 -- 172.21.15.81:0/2010696116 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda44011920 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda615ef640 1 -- 172.21.15.81:0/2010696116 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5c0b4a20 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda537fe640 1 -- 172.21.15.81:0/2010696116 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44011d90 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.609+0000 7fda615ef640 1 -- 172.21.15.81:0/2010696116 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5c0b4460 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.610+0000 7fda615ef640 1 -- 172.21.15.81:0/2010696116 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda5c0cff60 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.611+0000 7fda537fe640 1 -- 172.21.15.81:0/2010696116 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 137) ==== 98349+0+0 (secure 0 0 0) 0x7fda44018020 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.611+0000 7fda537fe640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fda3c076480 0x7fda3c078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.611+0000 7fda537fe640 1 -- 172.21.15.81:0/2010696116 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 7518+0+0 (secure 0 0 0) 0x7fda44098080 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.612+0000 7fda5affd640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fda3c076480 0x7fda3c078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.612+0000 7fda5affd640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fda3c076480 0x7fda3c078990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fda4c005c40 tx=0x7fda4c00a550 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.615+0000 7fda537fe640 1 -- 172.21.15.81:0/2010696116 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fda4401f020 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.834+0000 7fda615ef640 1 -- 172.21.15.81:0/2010696116 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fda5c0b9700 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.835 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T15:58:00.835+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:58:00.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.839+0000 7fda537fe640 1 -- 172.21.15.81:0/2010696116 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v119) ==== 161+0+0 (secure 0 0 0) 0x7fda44060970 con 0x7fda5c0b8ba0 2024-06-29T15:58:00.839 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:58:00.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 -- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fda3c076480 msgr2=0x7fda3c078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fda3c076480 0x7fda3c078990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fda4c005c40 tx=0x7fda4c00a550 comp rx=0 tx=0).stop 2024-06-29T15:58:00.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 -- 172.21.15.81:0/2010696116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda5c0b8ba0 msgr2=0x7fda5c09df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda5c0b8ba0 0x7fda5c09df70 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7fda4400b3c0 tx=0x7fda4400b880 comp rx=0 tx=0).stop 2024-06-29T15:58:00.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 -- 172.21.15.81:0/2010696116 shutdown_connections 2024-06-29T15:58:00.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fda3c076480 0x7fda3c078990 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fda5c09e4b0 0x7fda5c09e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda5c0b8ba0 0x7fda5c09df70 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 --2- 172.21.15.81:0/2010696116 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fda5c0b6a20 0x7fda5c0b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 -- 172.21.15.81:0/2010696116 >> 172.21.15.81:0/2010696116 conn(0x7fda5c0d3bd0 msgr2=0x7fda5c0b5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 -- 172.21.15.81:0/2010696116 shutdown_connections 2024-06-29T15:58:00.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.842+0000 7fda615ef640 1 -- 172.21.15.81:0/2010696116 wait complete. 2024-06-29T15:58:00.845 INFO:tasks.ceph.mds.a.smithi081.stderr: -4732> 2024-06-29T15:57:28.251+0000 7ffaf9717640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:00.858 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:58:00.903 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T15:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.975+0000 7fd06cc09640 1 -- 172.21.15.81:0/828198702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd054003a00 msgr2=0x7fd054003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.975+0000 7fd06cc09640 1 --2- 172.21.15.81:0/828198702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd054003a00 0x7fd054003e30 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7fd058009b40 tx=0x7fd054005830 comp rx=0 tx=0).stop 2024-06-29T15:58:00.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.975+0000 7fd06cc09640 1 -- 172.21.15.81:0/828198702 shutdown_connections 2024-06-29T15:58:00.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.975+0000 7fd06cc09640 1 --2- 172.21.15.81:0/828198702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd054003a00 0x7fd054003e30 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.975+0000 7fd06cc09640 1 -- 172.21.15.81:0/828198702 >> 172.21.15.81:0/828198702 conn(0x7fd0680d3b70 msgr2=0x7fd0680d3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:00.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.975+0000 7fd06cc09640 1 -- 172.21.15.81:0/828198702 shutdown_connections 2024-06-29T15:58:00.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.975+0000 7fd06cc09640 1 -- 172.21.15.81:0/828198702 wait complete. 2024-06-29T15:58:00.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.977+0000 7fd06cc09640 1 Processor -- start 2024-06-29T15:58:00.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.977+0000 7fd06cc09640 1 -- start start 2024-06-29T15:58:00.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.978+0000 7fd06cc09640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd054003a00 0x7fd068154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.978+0000 7fd06cc09640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd06815b1d0 0x7fd06815dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.978+0000 7fd06cc09640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd06815b5e0 0x7fd06815e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.978+0000 7fd06cc09640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0680d49a0 con 0x7fd054003a00 2024-06-29T15:58:00.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.978+0000 7fd06cc09640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd0680b8e80 con 0x7fd06815b1d0 2024-06-29T15:58:00.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.978+0000 7fd06cc09640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd0681f6470 con 0x7fd06815b5e0 2024-06-29T15:58:00.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.978+0000 7fd06659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd054003a00 0x7fd068154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.978+0000 7fd06659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd054003a00 0x7fd068154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47330/0 (socket says 172.21.15.81:47330) 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.978+0000 7fd06659c640 1 -- 172.21.15.81:0/917726950 learned_addr learned my addr 172.21.15.81:0/917726950 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd066d9d640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd06815b5e0 0x7fd06815e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd05ffff640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd06815b1d0 0x7fd06815dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd06659c640 1 -- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd06815b5e0 msgr2=0x7fd06815e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd06659c640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd06815b5e0 0x7fd06815e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd06659c640 1 -- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd06815b1d0 msgr2=0x7fd06815dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd06659c640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd06815b1d0 0x7fd06815dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd06659c640 1 -- 172.21.15.81:0/917726950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd068155080 con 0x7fd054003a00 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd06659c640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd054003a00 0x7fd068154b40 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fd058009c60 tx=0x7fd0580336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd05effd640 1 -- 172.21.15.81:0/917726950 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd058038040 con 0x7fd054003a00 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd05effd640 1 -- 172.21.15.81:0/917726950 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd058041070 con 0x7fd054003a00 2024-06-29T15:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd06cc09640 1 -- 172.21.15.81:0/917726950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd068155e80 con 0x7fd054003a00 2024-06-29T15:58:00.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd05effd640 1 -- 172.21.15.81:0/917726950 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd058033b40 con 0x7fd054003a00 2024-06-29T15:58:00.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.979+0000 7fd06cc09640 1 -- 172.21.15.81:0/917726950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd068156440 con 0x7fd054003a00 2024-06-29T15:58:00.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.980+0000 7fd06cc09640 1 -- 172.21.15.81:0/917726950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0680d26d0 con 0x7fd054003a00 2024-06-29T15:58:00.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.981+0000 7fd05effd640 1 -- 172.21.15.81:0/917726950 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 137) ==== 98349+0+0 (secure 0 0 0) 0x7fd058033ce0 con 0x7fd054003a00 2024-06-29T15:58:00.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.981+0000 7fd05effd640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd0480764b0 0x7fd0480789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:00.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.982+0000 7fd05effd640 1 -- 172.21.15.81:0/917726950 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(220..220 src has 1..220) ==== 7604+0+0 (secure 0 0 0) 0x7fd0580b8bd0 con 0x7fd054003a00 2024-06-29T15:58:00.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.982+0000 7fd05ffff640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd0480764b0 0x7fd0480789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:00.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.982+0000 7fd05ffff640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd0480764b0 0x7fd0480789c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd050006fd0 tx=0x7fd050006b10 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:00.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:00.986+0000 7fd05effd640 1 -- 172.21.15.81:0/917726950 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd058081b40 con 0x7fd054003a00 2024-06-29T15:58:01.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.202+0000 7fd06cc09640 1 -- 172.21.15.81:0/917726950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd0680d1fe0 con 0x7fd054003a00 2024-06-29T15:58:01.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.845+0000 7fd05effd640 1 -- 172.21.15.81:0/917726950 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v120) ==== 104+0+0 (secure 0 0 0) 0x7fd058081440 con 0x7fd054003a00 2024-06-29T15:58:01.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.847+0000 7fd06cc09640 1 -- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd0480764b0 msgr2=0x7fd0480789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.847+0000 7fd06cc09640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd0480764b0 0x7fd0480789c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd050006fd0 tx=0x7fd050006b10 comp rx=0 tx=0).stop 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.847+0000 7fd06cc09640 1 -- 172.21.15.81:0/917726950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd054003a00 msgr2=0x7fd068154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.847+0000 7fd06cc09640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd054003a00 0x7fd068154b40 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fd058009c60 tx=0x7fd0580336f0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.848+0000 7fd06cc09640 1 -- 172.21.15.81:0/917726950 shutdown_connections 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.848+0000 7fd06cc09640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd0480764b0 0x7fd0480789c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.848+0000 7fd06cc09640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd06815b5e0 0x7fd06815e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.848+0000 7fd06cc09640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd06815b1d0 0x7fd06815dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.848+0000 7fd06cc09640 1 --2- 172.21.15.81:0/917726950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd054003a00 0x7fd068154b40 secure :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7fd058009c60 tx=0x7fd0580336f0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.848+0000 7fd06cc09640 1 -- 172.21.15.81:0/917726950 >> 172.21.15.81:0/917726950 conn(0x7fd0680d3b70 msgr2=0x7fd0680b75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.848+0000 7fd06cc09640 1 -- 172.21.15.81:0/917726950 shutdown_connections 2024-06-29T15:58:01.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.848+0000 7fd06cc09640 1 -- 172.21.15.81:0/917726950 wait complete. 2024-06-29T15:58:01.862 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:58:01.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.979+0000 7f4ebe610640 1 -- 172.21.15.81:0/2529454726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb80cc940 msgr2=0x7f4eb815a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.979+0000 7f4ebe610640 1 --2- 172.21.15.81:0/2529454726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb80cc940 0x7f4eb815a390 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009830 tx=0x7f4eac02a510 comp rx=0 tx=0).stop 2024-06-29T15:58:01.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.979+0000 7f4ebe610640 1 -- 172.21.15.81:0/2529454726 shutdown_connections 2024-06-29T15:58:01.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.979+0000 7f4ebe610640 1 --2- 172.21.15.81:0/2529454726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb80cc940 0x7f4eb815a390 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.979+0000 7f4ebe610640 1 --2- 172.21.15.81:0/2529454726 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eb80cbd60 0x7f4eb8073b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.979+0000 7f4ebe610640 1 -- 172.21.15.81:0/2529454726 >> 172.21.15.81:0/2529454726 conn(0x7f4eb806cd60 msgr2=0x7f4eb80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:01.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.979+0000 7f4ebe610640 1 -- 172.21.15.81:0/2529454726 shutdown_connections 2024-06-29T15:58:01.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.979+0000 7f4ebe610640 1 -- 172.21.15.81:0/2529454726 wait complete. 2024-06-29T15:58:01.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.981+0000 7f4ebe610640 1 Processor -- start 2024-06-29T15:58:01.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.981+0000 7f4ebe610640 1 -- start start 2024-06-29T15:58:01.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4ebe610640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eb80cbd60 0x7f4eb80aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4ebe610640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eb80cc940 0x7f4eb8141d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4ebe610640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb80ccd50 0x7f4eb8142290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4ebe610640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb806f630 con 0x7f4eb80ccd50 2024-06-29T15:58:01.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4ebe610640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb806f4b0 con 0x7f4eb80cc940 2024-06-29T15:58:01.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4ebe610640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4eb806f7b0 con 0x7f4eb80cbd60 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4ebcbad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb80ccd50 0x7f4eb8142290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4ebcbad640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb80ccd50 0x7f4eb8142290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47344/0 (socket says 172.21.15.81:47344) 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4ebcbad640 1 -- 172.21.15.81:0/3518662651 learned_addr learned my addr 172.21.15.81:0/3518662651 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4eb7fff640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eb80cbd60 0x7f4eb80aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.982+0000 7f4eb77fe640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eb80cc940 0x7f4eb8141d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4ebcbad640 1 -- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eb80cbd60 msgr2=0x7f4eb80aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4ebcbad640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eb80cbd60 0x7f4eb80aeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4ebcbad640 1 -- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eb80cc940 msgr2=0x7f4eb8141d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4ebcbad640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eb80cc940 0x7f4eb8141d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4ebcbad640 1 -- 172.21.15.81:0/3518662651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb814aab0 con 0x7f4eb80ccd50 2024-06-29T15:58:01.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4eb7fff640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eb80cbd60 0x7f4eb80aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:01.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4eb77fe640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eb80cc940 0x7f4eb8141d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:01.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4ebcbad640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb80ccd50 0x7f4eb8142290 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000c730 tx=0x7f4eb000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:01.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4eb4ff9640 1 -- 172.21.15.81:0/3518662651 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0003b70 con 0x7f4eb80ccd50 2024-06-29T15:58:01.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4eb4ff9640 1 -- 172.21.15.81:0/3518662651 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4eb0003d30 con 0x7f4eb80ccd50 2024-06-29T15:58:01.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4ebe610640 1 -- 172.21.15.81:0/3518662651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eb8150610 con 0x7f4eb80ccd50 2024-06-29T15:58:01.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4eb4ff9640 1 -- 172.21.15.81:0/3518662651 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb00052d0 con 0x7f4eb80ccd50 2024-06-29T15:58:01.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.983+0000 7f4ebe610640 1 -- 172.21.15.81:0/3518662651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eb8141270 con 0x7f4eb80ccd50 2024-06-29T15:58:01.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.984+0000 7f4ebe610640 1 -- 172.21.15.81:0/3518662651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eb806fcd0 con 0x7f4eb80ccd50 2024-06-29T15:58:01.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.985+0000 7f4eb4ff9640 1 -- 172.21.15.81:0/3518662651 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f4eb0005470 con 0x7f4eb80ccd50 2024-06-29T15:58:01.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.985+0000 7f4eb4ff9640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4ea4076510 0x7f4ea4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:01.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.985+0000 7f4eb4ff9640 1 -- 172.21.15.81:0/3518662651 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(220..220 src has 1..220) ==== 7604+0+0 (secure 0 0 0) 0x7f4eb0098160 con 0x7f4eb80ccd50 2024-06-29T15:58:01.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.986+0000 7f4eb7fff640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4ea4076510 0x7f4ea4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:01.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.986+0000 7f4eb7fff640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4ea4076510 0x7f4ea4078a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4ea800ab40 tx=0x7f4ea800a520 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:01.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:01.990+0000 7f4eb4ff9640 1 -- 172.21.15.81:0/3518662651 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4eb814aab0 con 0x7f4eb80ccd50 2024-06-29T15:58:02.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.172+0000 7f4ebe610640 1 -- 172.21.15.81:0/3518662651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4eb81500c0 con 0x7f4eb80ccd50 2024-06-29T15:58:02.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.846+0000 7f4eb4ff9640 1 -- 172.21.15.81:0/3518662651 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v221) ==== 181+0+0 (secure 0 0 0) 0x7f4eb0060ff0 con 0x7f4eb80ccd50 2024-06-29T15:58:02.847 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 -- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4ea4076510 msgr2=0x7f4ea4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4ea4076510 0x7f4ea4078a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4ea800ab40 tx=0x7f4ea800a520 comp rx=0 tx=0).stop 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 -- 172.21.15.81:0/3518662651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb80ccd50 msgr2=0x7f4eb8142290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb80ccd50 0x7f4eb8142290 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000c730 tx=0x7f4eb000cbf0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 -- 172.21.15.81:0/3518662651 shutdown_connections 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4ea4076510 0x7f4ea4078a20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb80ccd50 0x7f4eb8142290 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4eb80cc940 0x7f4eb8141d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 --2- 172.21.15.81:0/3518662651 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4eb80cbd60 0x7f4eb80aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 -- 172.21.15.81:0/3518662651 >> 172.21.15.81:0/3518662651 conn(0x7f4eb806cd60 msgr2=0x7f4eb814c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 -- 172.21.15.81:0/3518662651 shutdown_connections 2024-06-29T15:58:02.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.850+0000 7f4ebe610640 1 -- 172.21.15.81:0/3518662651 wait complete. 2024-06-29T15:58:02.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:58:02.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.984+0000 7f9a77194640 1 -- 172.21.15.81:0/3626369492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a7008c200 msgr2=0x7f9a700d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.984+0000 7f9a77194640 1 --2- 172.21.15.81:0/3626369492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a7008c200 0x7f9a700d1840 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f9a7008d0c0 tx=0x7f9a6c02a530 comp rx=0 tx=0).stop 2024-06-29T15:58:02.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.984+0000 7f9a77194640 1 -- 172.21.15.81:0/3626369492 shutdown_connections 2024-06-29T15:58:02.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.984+0000 7f9a77194640 1 --2- 172.21.15.81:0/3626369492 >> v2:172.21.15.167:3300/0 conn(0x7f9a70163020 0x7f9a700ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.984+0000 7f9a77194640 1 --2- 172.21.15.81:0/3626369492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a7008c200 0x7f9a700d1840 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.984+0000 7f9a77194640 1 -- 172.21.15.81:0/3626369492 >> 172.21.15.81:0/3626369492 conn(0x7f9a700b5760 msgr2=0x7f9a700b5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:02.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.984+0000 7f9a77194640 1 -- 172.21.15.81:0/3626369492 shutdown_connections 2024-06-29T15:58:02.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.984+0000 7f9a77194640 1 -- 172.21.15.81:0/3626369492 wait complete. 2024-06-29T15:58:02.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.986+0000 7f9a77194640 1 Processor -- start 2024-06-29T15:58:02.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.986+0000 7f9a77194640 1 -- start start 2024-06-29T15:58:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a77194640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a7008c200 0x7f9a701582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a77194640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a700d1d80 0x7f9a700b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a77194640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a70163020 0x7f9a701ea800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a77194640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a700b55e0 con 0x7f9a70163020 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a77194640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9a700b5460 con 0x7f9a7008c200 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a77194640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9a70163430 con 0x7f9a700d1d80 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a75731640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a70163020 0x7f9a701ea800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a75731640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a70163020 0x7f9a701ea800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47362/0 (socket says 172.21.15.81:47362) 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a75731640 1 -- 172.21.15.81:0/239791713 learned_addr learned my addr 172.21.15.81:0/239791713 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a74f30640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a7008c200 0x7f9a701582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a67fff640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a700d1d80 0x7f9a700b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a75731640 1 -- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a700d1d80 msgr2=0x7f9a700b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a75731640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a700d1d80 0x7f9a700b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a75731640 1 -- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a7008c200 msgr2=0x7f9a701582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a75731640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a7008c200 0x7f9a701582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a75731640 1 -- 172.21.15.81:0/239791713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a700b0f30 con 0x7f9a70163020 2024-06-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a74f30640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a7008c200 0x7f9a701582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:02.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a75731640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a70163020 0x7f9a701ea800 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800ca80 tx=0x7f9a6800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:02.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a657fa640 1 -- 172.21.15.81:0/239791713 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68007d90 con 0x7f9a70163020 2024-06-29T15:58:02.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a657fa640 1 -- 172.21.15.81:0/239791713 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a68004c30 con 0x7f9a70163020 2024-06-29T15:58:02.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a657fa640 1 -- 172.21.15.81:0/239791713 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68005280 con 0x7f9a70163020 2024-06-29T15:58:02.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.988+0000 7f9a77194640 1 -- 172.21.15.81:0/239791713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a700b1210 con 0x7f9a70163020 2024-06-29T15:58:02.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.989+0000 7f9a77194640 1 -- 172.21.15.81:0/239791713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a700b17d0 con 0x7f9a70163020 2024-06-29T15:58:02.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.990+0000 7f9a77194640 1 -- 172.21.15.81:0/239791713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a700d1840 con 0x7f9a70163020 2024-06-29T15:58:02.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.990+0000 7f9a657fa640 1 -- 172.21.15.81:0/239791713 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f9a68005420 con 0x7f9a70163020 2024-06-29T15:58:02.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.991+0000 7f9a657fa640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9a5c076510 0x7f9a5c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:02.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.991+0000 7f9a657fa640 1 -- 172.21.15.81:0/239791713 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(221..221 src has 1..221) ==== 7166+0+0 (secure 0 0 0) 0x7f9a68098cc0 con 0x7f9a70163020 2024-06-29T15:58:02.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.991+0000 7f9a74f30640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9a5c076510 0x7f9a5c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:02.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.992+0000 7f9a74f30640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9a5c076510 0x7f9a5c078a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c034000 tx=0x7f9a6c004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:02.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:02.995+0000 7f9a657fa640 1 -- 172.21.15.81:0/239791713 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9a68014070 con 0x7f9a70163020 2024-06-29T15:58:03.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.183+0000 7f9a77194640 1 -- 172.21.15.81:0/239791713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9a700b6530 con 0x7f9a70163020 2024-06-29T15:58:03.728 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:58:03.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:58:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.852+0000 7f9a657fa640 1 -- 172.21.15.81:0/239791713 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v222) ==== 169+0+0 (secure 0 0 0) 0x7f9a68061c00 con 0x7f9a70163020 2024-06-29T15:58:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T15:58:03.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.854+0000 7f9a77194640 1 -- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9a5c076510 msgr2=0x7f9a5c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:03.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.854+0000 7f9a77194640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9a5c076510 0x7f9a5c078a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c034000 tx=0x7f9a6c004730 comp rx=0 tx=0).stop 2024-06-29T15:58:03.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.854+0000 7f9a77194640 1 -- 172.21.15.81:0/239791713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a70163020 msgr2=0x7f9a701ea800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.854+0000 7f9a77194640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a70163020 0x7f9a701ea800 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800ca80 tx=0x7f9a6800cf40 comp rx=0 tx=0).stop 2024-06-29T15:58:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.855+0000 7f9a77194640 1 -- 172.21.15.81:0/239791713 shutdown_connections 2024-06-29T15:58:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.855+0000 7f9a77194640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9a5c076510 0x7f9a5c078a20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.855+0000 7f9a77194640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a70163020 0x7f9a701ea800 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.855+0000 7f9a77194640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a700d1d80 0x7f9a700b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.855+0000 7f9a77194640 1 --2- 172.21.15.81:0/239791713 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a7008c200 0x7f9a701582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.855+0000 7f9a77194640 1 -- 172.21.15.81:0/239791713 >> 172.21.15.81:0/239791713 conn(0x7f9a700b5760 msgr2=0x7f9a700b5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.856+0000 7f9a77194640 1 -- 172.21.15.81:0/239791713 shutdown_connections 2024-06-29T15:58:03.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.856+0000 7f9a77194640 1 -- 172.21.15.81:0/239791713 wait complete. 2024-06-29T15:58:03.873 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.985+0000 7f7a33391640 1 -- 172.21.15.81:0/3353587050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06c2f0 msgr2=0x7f7a2c06c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.985+0000 7f7a33391640 1 --2- 172.21.15.81:0/3353587050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06c2f0 0x7f7a2c06c700 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009830 tx=0x7f7a1c02a510 comp rx=0 tx=0).stop 2024-06-29T15:58:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.985+0000 7f7a33391640 1 -- 172.21.15.81:0/3353587050 shutdown_connections 2024-06-29T15:58:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.985+0000 7f7a33391640 1 --2- 172.21.15.81:0/3353587050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06c2f0 0x7f7a2c06c700 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.985+0000 7f7a33391640 1 -- 172.21.15.81:0/3353587050 >> 172.21.15.81:0/3353587050 conn(0x7f7a2c0b2920 msgr2=0x7f7a2c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.985+0000 7f7a33391640 1 -- 172.21.15.81:0/3353587050 shutdown_connections 2024-06-29T15:58:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.985+0000 7f7a33391640 1 -- 172.21.15.81:0/3353587050 wait complete. 2024-06-29T15:58:03.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.987+0000 7f7a33391640 1 Processor -- start 2024-06-29T15:58:03.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.987+0000 7f7a33391640 1 -- start start 2024-06-29T15:58:03.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.988+0000 7f7a33391640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06c2f0 0x7f7a2c1f9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:03.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.988+0000 7f7a33391640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c06f9a0 0x7f7a2c0b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:03.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.988+0000 7f7a33391640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a2c159bd0 0x7f7a2c0b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:03.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.988+0000 7f7a33391640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c15fe20 con 0x7f7a2c06c2f0 2024-06-29T15:58:03.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.988+0000 7f7a33391640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c15fca0 con 0x7f7a2c06f9a0 2024-06-29T15:58:03.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.988+0000 7f7a33391640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2c15ffa0 con 0x7f7a2c159bd0 2024-06-29T15:58:03.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.988+0000 7f7a3112d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06c2f0 0x7f7a2c1f9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:03.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.988+0000 7f7a3112d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06c2f0 0x7f7a2c1f9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47392/0 (socket says 172.21.15.81:47392) 2024-06-29T15:58:03.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.988+0000 7f7a3112d640 1 -- 172.21.15.81:0/193834400 learned_addr learned my addr 172.21.15.81:0/193834400 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a3092c640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c06f9a0 0x7f7a2c0b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a3192e640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a2c159bd0 0x7f7a2c0b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a3112d640 1 -- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a2c159bd0 msgr2=0x7f7a2c0b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a3112d640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a2c159bd0 0x7f7a2c0b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a3112d640 1 -- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c06f9a0 msgr2=0x7f7a2c0b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a3112d640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c06f9a0 0x7f7a2c0b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a3112d640 1 -- 172.21.15.81:0/193834400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2c135200 con 0x7f7a2c06c2f0 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a3112d640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06c2f0 0x7f7a2c1f9150 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800c7a0 tx=0x7f7a2800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a21ffb640 1 -- 172.21.15.81:0/193834400 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28017070 con 0x7f7a2c06c2f0 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a21ffb640 1 -- 172.21.15.81:0/193834400 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a28005070 con 0x7f7a2c06c2f0 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a21ffb640 1 -- 172.21.15.81:0/193834400 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a280054a0 con 0x7f7a2c06c2f0 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a33391640 1 -- 172.21.15.81:0/193834400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a2c1354e0 con 0x7f7a2c06c2f0 2024-06-29T15:58:03.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.989+0000 7f7a33391640 1 -- 172.21.15.81:0/193834400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a2c0b4030 con 0x7f7a2c06c2f0 2024-06-29T15:58:03.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.990+0000 7f7a33391640 1 -- 172.21.15.81:0/193834400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a2c0cede0 con 0x7f7a2c06c2f0 2024-06-29T15:58:03.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.991+0000 7f7a21ffb640 1 -- 172.21.15.81:0/193834400 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f7a2801a070 con 0x7f7a2c06c2f0 2024-06-29T15:58:03.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.991+0000 7f7a21ffb640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7a10076540 0x7f7a10078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:03.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.991+0000 7f7a21ffb640 1 -- 172.21.15.81:0/193834400 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6752+0+0 (secure 0 0 0) 0x7f7a280976e0 con 0x7f7a2c06c2f0 2024-06-29T15:58:03.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.992+0000 7f7a3092c640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7a10076540 0x7f7a10078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:03.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.992+0000 7f7a3092c640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7a10076540 0x7f7a10078a50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c034000 tx=0x7f7a1c004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:03.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:03.996+0000 7f7a21ffb640 1 -- 172.21.15.81:0/193834400 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7a2c0cede0 con 0x7f7a2c06c2f0 2024-06-29T15:58:04.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.175+0000 7f7a33391640 1 -- 172.21.15.81:0/193834400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a2c0b3b40 con 0x7f7a2c06c2f0 2024-06-29T15:58:04.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.176+0000 7f7a21ffb640 1 -- 172.21.15.81:0/193834400 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) ==== 74+0+18216 (secure 0 0 0) 0x7f7a2c06cc40 con 0x7f7a2c06c2f0 2024-06-29T15:58:04.177 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:04.177 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":222,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:03.847479+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.259110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:32.675276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6833/739691411":"2024-06-30T15:58:00.835804+0000","172.21.15.167:0/2503226350":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/2924173136":"2024-06-30T15:57:57.317102+0000","172.21.15.81:6832/739691411":"2024-06-30T15:58:00.835804+0000","172.21.15.167:6833/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6832/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3022587441":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/575713984":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/1101183266":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3261132296":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3042354413":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1619575755":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/760717235":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/3781943725":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/376674682":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1164120698":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3640842563":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/2165320595":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3538782057":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6833/2018755589":"2024-06-30T15:57:48.595362+0000","172.21.15.167:6832/2018755589":"2024-06-30T15:57:48.595362+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:04.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 -- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7a10076540 msgr2=0x7f7a10078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7a10076540 0x7f7a10078a50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c034000 tx=0x7f7a1c004730 comp rx=0 tx=0).stop 2024-06-29T15:58:04.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 -- 172.21.15.81:0/193834400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06c2f0 msgr2=0x7f7a2c1f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06c2f0 0x7f7a2c1f9150 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800c7a0 tx=0x7f7a2800cc60 comp rx=0 tx=0).stop 2024-06-29T15:58:04.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 -- 172.21.15.81:0/193834400 shutdown_connections 2024-06-29T15:58:04.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7a10076540 0x7f7a10078a50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a2c159bd0 0x7f7a2c0b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c06f9a0 0x7f7a2c0b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 --2- 172.21.15.81:0/193834400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06c2f0 0x7f7a2c1f9150 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 -- 172.21.15.81:0/193834400 >> 172.21.15.81:0/193834400 conn(0x7f7a2c0b2920 msgr2=0x7f7a2c15e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:04.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 -- 172.21.15.81:0/193834400 shutdown_connections 2024-06-29T15:58:04.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.179+0000 7f7a33391640 1 -- 172.21.15.81:0/193834400 wait complete. 2024-06-29T15:58:04.192 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:57:32.259110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '216', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:57:32.675276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '217', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:04.192 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:04.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.308+0000 7ff829e09640 1 -- 172.21.15.81:0/889599606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff80c003a00 msgr2=0x7ff80c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.309+0000 7ff829e09640 1 --2- 172.21.15.81:0/889599606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff80c003a00 0x7ff80c003e70 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7ff814009b20 tx=0x7ff80c005830 comp rx=0 tx=0).stop 2024-06-29T15:58:04.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.309+0000 7ff823fff640 1 -- 172.21.15.81:0/889599606 reap_dead start 2024-06-29T15:58:04.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.309+0000 7ff829e09640 1 -- 172.21.15.81:0/889599606 shutdown_connections 2024-06-29T15:58:04.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.309+0000 7ff829e09640 1 -- 172.21.15.81:0/889599606 >> 172.21.15.81:0/889599606 conn(0x7ff8240d1960 msgr2=0x7ff8240d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:04.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.309+0000 7ff829e09640 1 -- 172.21.15.81:0/889599606 shutdown_connections 2024-06-29T15:58:04.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.309+0000 7ff829e09640 1 -- 172.21.15.81:0/889599606 wait complete. 2024-06-29T15:58:04.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.310+0000 7ff829e09640 1 Processor -- start 2024-06-29T15:58:04.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.311+0000 7ff829e09640 1 -- start start 2024-06-29T15:58:04.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff829e09640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff80c003a00 0x7ff8241deb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff829e09640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8240c7df0 0x7ff8240b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff829e09640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240b0af0 0x7ff8240c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff829e09640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff8240ce4d0 con 0x7ff8240b0af0 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff829e09640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff8240b2790 con 0x7ff8240c7df0 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff829e09640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff8240ce650 con 0x7ff80c003a00 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff823fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240b0af0 0x7ff8240c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff823fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240b0af0 0x7ff8240c9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47422/0 (socket says 172.21.15.81:47422) 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff823fff640 1 -- 172.21.15.81:0/725269876 learned_addr learned my addr 172.21.15.81:0/725269876 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff822ffd640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8240c7df0 0x7ff8240b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff8237fe640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff80c003a00 0x7ff8241deb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff823fff640 1 -- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff80c003a00 msgr2=0x7ff8241deb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff823fff640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff80c003a00 0x7ff8241deb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff823fff640 1 -- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8240c7df0 msgr2=0x7ff8240b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff823fff640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8240c7df0 0x7ff8240b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff823fff640 1 -- 172.21.15.81:0/725269876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8240c4f00 con 0x7ff8240b0af0 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff8237fe640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff80c003a00 0x7ff8241deb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.312+0000 7ff823fff640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240b0af0 0x7ff8240c9840 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7ff8240c5400 tx=0x7ff81c010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.313+0000 7ff80bfff640 1 -- 172.21.15.81:0/725269876 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c003970 con 0x7ff8240b0af0 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.313+0000 7ff80bfff640 1 -- 172.21.15.81:0/725269876 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff81c015070 con 0x7ff8240b0af0 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.313+0000 7ff829e09640 1 -- 172.21.15.81:0/725269876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8240b0f00 con 0x7ff8240b0af0 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.313+0000 7ff80bfff640 1 -- 172.21.15.81:0/725269876 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c004030 con 0x7ff8240b0af0 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.313+0000 7ff829e09640 1 -- 172.21.15.81:0/725269876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8240c6250 con 0x7ff8240b0af0 2024-06-29T15:58:04.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.314+0000 7ff829e09640 1 -- 172.21.15.81:0/725269876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff824070b10 con 0x7ff8240b0af0 2024-06-29T15:58:04.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.315+0000 7ff80bfff640 1 -- 172.21.15.81:0/725269876 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7ff81c0041d0 con 0x7ff8240b0af0 2024-06-29T15:58:04.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.315+0000 7ff80bfff640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ff810076510 0x7ff810078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.315+0000 7ff80bfff640 1 -- 172.21.15.81:0/725269876 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6752+0+0 (secure 0 0 0) 0x7ff81c098b40 con 0x7ff8240b0af0 2024-06-29T15:58:04.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.315+0000 7ff8237fe640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ff810076510 0x7ff810078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.316+0000 7ff8237fe640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ff810076510 0x7ff810078a20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff814009b20 tx=0x7ff81402d040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:04.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.319+0000 7ff80bfff640 1 -- 172.21.15.81:0/725269876 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff824070b10 con 0x7ff8240b0af0 2024-06-29T15:58:04.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.498+0000 7ff829e09640 1 -- 172.21.15.81:0/725269876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff8240d07f0 con 0x7ff8240b0af0 2024-06-29T15:58:04.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.499+0000 7ff80bfff640 1 -- 172.21.15.81:0/725269876 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v222) ==== 93+0+10 (secure 0 0 0) 0x7ff8240b0f00 con 0x7ff8240b0af0 2024-06-29T15:58:04.499 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:58:04.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 -- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ff810076510 msgr2=0x7ff810078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ff810076510 0x7ff810078a20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff814009b20 tx=0x7ff81402d040 comp rx=0 tx=0).stop 2024-06-29T15:58:04.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 -- 172.21.15.81:0/725269876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240b0af0 msgr2=0x7ff8240c9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240b0af0 0x7ff8240c9840 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7ff8240c5400 tx=0x7ff81c010f40 comp rx=0 tx=0).stop 2024-06-29T15:58:04.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 -- 172.21.15.81:0/725269876 shutdown_connections 2024-06-29T15:58:04.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7ff810076510 0x7ff810078a20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240b0af0 0x7ff8240c9840 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff8240c7df0 0x7ff8240b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 --2- 172.21.15.81:0/725269876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff80c003a00 0x7ff8241deb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 -- 172.21.15.81:0/725269876 >> 172.21.15.81:0/725269876 conn(0x7ff8240d1960 msgr2=0x7ff8240b5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:04.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 -- 172.21.15.81:0/725269876 shutdown_connections 2024-06-29T15:58:04.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.501+0000 7ff829e09640 1 -- 172.21.15.81:0/725269876 wait complete. 2024-06-29T15:58:04.512 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:58:04.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.628+0000 7f1c72895640 1 -- 172.21.15.81:0/1584269293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c06fa90 msgr2=0x7f1c6c06ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.628+0000 7f1c72895640 1 --2- 172.21.15.81:0/1584269293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c06fa90 0x7f1c6c06ca00 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f1c64005fd0 tx=0x7f1c6402aa00 comp rx=0 tx=0).stop 2024-06-29T15:58:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.628+0000 7f1c72895640 1 -- 172.21.15.81:0/1584269293 shutdown_connections 2024-06-29T15:58:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.628+0000 7f1c72895640 1 --2- 172.21.15.81:0/1584269293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c06fa90 0x7f1c6c06ca00 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.628+0000 7f1c72895640 1 --2- 172.21.15.81:0/1584269293 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c6c074520 0x7f1c6c06f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.628+0000 7f1c72895640 1 -- 172.21.15.81:0/1584269293 >> 172.21.15.81:0/1584269293 conn(0x7f1c6c072950 msgr2=0x7f1c6c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.628+0000 7f1c72895640 1 -- 172.21.15.81:0/1584269293 shutdown_connections 2024-06-29T15:58:04.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.628+0000 7f1c72895640 1 -- 172.21.15.81:0/1584269293 wait complete. 2024-06-29T15:58:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.629+0000 7f1c72895640 1 Processor -- start 2024-06-29T15:58:04.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.630+0000 7f1c72895640 1 -- start start 2024-06-29T15:58:04.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c72895640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c6c06f680 0x7f1c6c14f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c72895640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c6c06fa90 0x7f1c6c150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c72895640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c074520 0x7f1c6c150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c72895640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c15b010 con 0x7f1c6c074520 2024-06-29T15:58:04.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c72895640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c06d150 con 0x7f1c6c06f680 2024-06-29T15:58:04.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c72895640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1c6c15b190 con 0x7f1c6c06fa90 2024-06-29T15:58:04.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c70e32640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c074520 0x7f1c6c150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c70e32640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c074520 0x7f1c6c150e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47450/0 (socket says 172.21.15.81:47450) 2024-06-29T15:58:04.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c70e32640 1 -- 172.21.15.81:0/53492271 learned_addr learned my addr 172.21.15.81:0/53492271 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:04.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c6b7fe640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c6c06fa90 0x7f1c6c150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c6bfff640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c6c06f680 0x7f1c6c14f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c70e32640 1 -- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c6c06fa90 msgr2=0x7f1c6c150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c70e32640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c6c06fa90 0x7f1c6c150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c70e32640 1 -- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c6c06f680 msgr2=0x7f1c6c14f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c70e32640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c6c06f680 0x7f1c6c14f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.631+0000 7f1c70e32640 1 -- 172.21.15.81:0/53492271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6c09a300 con 0x7f1c6c074520 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.632+0000 7f1c70e32640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c074520 0x7f1c6c150e70 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009d90 tx=0x7f1c64003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.632+0000 7f1c68ff9640 1 -- 172.21.15.81:0/53492271 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64033050 con 0x7f1c6c074520 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.632+0000 7f1c68ff9640 1 -- 172.21.15.81:0/53492271 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c64004450 con 0x7f1c6c074520 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.632+0000 7f1c72895640 1 -- 172.21.15.81:0/53492271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c6c09a5e0 con 0x7f1c6c074520 2024-06-29T15:58:04.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.632+0000 7f1c68ff9640 1 -- 172.21.15.81:0/53492271 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64038450 con 0x7f1c6c074520 2024-06-29T15:58:04.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.632+0000 7f1c72895640 1 -- 172.21.15.81:0/53492271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c6c09aba0 con 0x7f1c6c074520 2024-06-29T15:58:04.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.633+0000 7f1c72895640 1 -- 172.21.15.81:0/53492271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c6c06f140 con 0x7f1c6c074520 2024-06-29T15:58:04.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.634+0000 7f1c68ff9640 1 -- 172.21.15.81:0/53492271 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f1c64036070 con 0x7f1c6c074520 2024-06-29T15:58:04.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.634+0000 7f1c68ff9640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1c58076510 0x7f1c58078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.634+0000 7f1c68ff9640 1 -- 172.21.15.81:0/53492271 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6752+0+0 (secure 0 0 0) 0x7f1c640b9460 con 0x7f1c6c074520 2024-06-29T15:58:04.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.634+0000 7f1c6bfff640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1c58076510 0x7f1c58078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.635+0000 7f1c6bfff640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1c58076510 0x7f1c58078a20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0058c0 tx=0x7f1c5c005f60 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:04.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.639+0000 7f1c68ff9640 1 -- 172.21.15.81:0/53492271 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1c64081ef0 con 0x7f1c6c074520 2024-06-29T15:58:04.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.819+0000 7f1c72895640 1 -- 172.21.15.81:0/53492271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f1c6c0b58c0 con 0x7f1c6c074520 2024-06-29T15:58:04.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.820+0000 7f1c68ff9640 1 -- 172.21.15.81:0/53492271 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v222) ==== 107+0+11 (secure 0 0 0) 0x7f1c640861f0 con 0x7f1c6c074520 2024-06-29T15:58:04.820 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:58:04.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 -- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1c58076510 msgr2=0x7f1c58078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1c58076510 0x7f1c58078a20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0058c0 tx=0x7f1c5c005f60 comp rx=0 tx=0).stop 2024-06-29T15:58:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 -- 172.21.15.81:0/53492271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c074520 msgr2=0x7f1c6c150e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c074520 0x7f1c6c150e70 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009d90 tx=0x7f1c64003f10 comp rx=0 tx=0).stop 2024-06-29T15:58:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 -- 172.21.15.81:0/53492271 shutdown_connections 2024-06-29T15:58:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1c58076510 0x7f1c58078a20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c074520 0x7f1c6c150e70 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c6c06fa90 0x7f1c6c150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 --2- 172.21.15.81:0/53492271 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c6c06f680 0x7f1c6c14f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 -- 172.21.15.81:0/53492271 >> 172.21.15.81:0/53492271 conn(0x7f1c6c072950 msgr2=0x7f1c6c0afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 -- 172.21.15.81:0/53492271 shutdown_connections 2024-06-29T15:58:04.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.822+0000 7f1c72895640 1 -- 172.21.15.81:0/53492271 wait complete. 2024-06-29T15:58:04.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:58:04.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.949+0000 7f2eb877f640 1 -- 172.21.15.81:0/1095023244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2eb01412f0 msgr2=0x7f2eb00cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.949+0000 7f2eb877f640 1 --2- 172.21.15.81:0/1095023244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2eb01412f0 0x7f2eb00cef90 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009830 tx=0x7f2eac02a510 comp rx=0 tx=0).stop 2024-06-29T15:58:04.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.949+0000 7f2eb877f640 1 -- 172.21.15.81:0/1095023244 shutdown_connections 2024-06-29T15:58:04.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.949+0000 7f2eb877f640 1 --2- 172.21.15.81:0/1095023244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2eb01412f0 0x7f2eb00cef90 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.949+0000 7f2eb877f640 1 -- 172.21.15.81:0/1095023244 >> 172.21.15.81:0/1095023244 conn(0x7f2eb006ccc0 msgr2=0x7f2eb00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:04.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.949+0000 7f2eb877f640 1 -- 172.21.15.81:0/1095023244 shutdown_connections 2024-06-29T15:58:04.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.949+0000 7f2eb877f640 1 -- 172.21.15.81:0/1095023244 wait complete. 2024-06-29T15:58:04.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.951+0000 7f2eb877f640 1 Processor -- start 2024-06-29T15:58:04.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.951+0000 7f2eb877f640 1 -- start start 2024-06-29T15:58:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb877f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2eb00cf4d0 0x7f2eb015aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb877f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2eb01412f0 0x7f2eb0071f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb877f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2eb0136340 0x7f2eb0072460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb877f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb00b3390 con 0x7f2eb0136340 2024-06-29T15:58:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb877f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb00b3210 con 0x7f2eb01412f0 2024-06-29T15:58:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb877f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb00b3510 con 0x7f2eb00cf4d0 2024-06-29T15:58:04.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb6d1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2eb0136340 0x7f2eb0072460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb6d1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2eb0136340 0x7f2eb0072460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47474/0 (socket says 172.21.15.81:47474) 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb6d1c640 1 -- 172.21.15.81:0/1727507212 learned_addr learned my addr 172.21.15.81:0/1727507212 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb651b640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2eb00cf4d0 0x7f2eb015aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb5d1a640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2eb01412f0 0x7f2eb0071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb6d1c640 1 -- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2eb00cf4d0 msgr2=0x7f2eb015aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb6d1c640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2eb00cf4d0 0x7f2eb015aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb6d1c640 1 -- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2eb01412f0 msgr2=0x7f2eb0071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb6d1c640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2eb01412f0 0x7f2eb0071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.952+0000 7f2eb6d1c640 1 -- 172.21.15.81:0/1727507212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb0154df0 con 0x7f2eb0136340 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.953+0000 7f2eb6d1c640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2eb0136340 0x7f2eb0072460 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800b9e0 tx=0x7f2ea800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:04.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.953+0000 7f2ea6ffd640 1 -- 172.21.15.81:0/1727507212 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8003b70 con 0x7f2eb0136340 2024-06-29T15:58:04.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.953+0000 7f2ea6ffd640 1 -- 172.21.15.81:0/1727507212 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ea8003e00 con 0x7f2eb0136340 2024-06-29T15:58:04.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.953+0000 7f2eb877f640 1 -- 172.21.15.81:0/1727507212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb01550d0 con 0x7f2eb0136340 2024-06-29T15:58:04.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.953+0000 7f2ea6ffd640 1 -- 172.21.15.81:0/1727507212 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea800cb20 con 0x7f2eb0136340 2024-06-29T15:58:04.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.953+0000 7f2eb877f640 1 -- 172.21.15.81:0/1727507212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb0155690 con 0x7f2eb0136340 2024-06-29T15:58:04.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.954+0000 7f2eb877f640 1 -- 172.21.15.81:0/1727507212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eb00b3a50 con 0x7f2eb0136340 2024-06-29T15:58:04.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.954+0000 7f2ea6ffd640 1 -- 172.21.15.81:0/1727507212 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f2ea800ccc0 con 0x7f2eb0136340 2024-06-29T15:58:04.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.955+0000 7f2ea6ffd640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2e94076540 0x7f2e94078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:04.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.955+0000 7f2ea6ffd640 1 -- 172.21.15.81:0/1727507212 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6752+0+0 (secure 0 0 0) 0x7f2ea8097e30 con 0x7f2eb0136340 2024-06-29T15:58:04.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.955+0000 7f2eb651b640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2e94076540 0x7f2e94078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:04.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.956+0000 7f2eb651b640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2e94076540 0x7f2e94078a50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2eac034000 tx=0x7f2eac004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:04.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:04.960+0000 7f2ea6ffd640 1 -- 172.21.15.81:0/1727507212 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2ea8060940 con 0x7f2eb0136340 2024-06-29T15:58:05.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.143+0000 7f2eb877f640 1 -- 172.21.15.81:0/1727507212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f2eb01e5120 con 0x7f2eb0136340 2024-06-29T15:58:05.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.143+0000 7f2ea6ffd640 1 -- 172.21.15.81:0/1727507212 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v222) ==== 103+0+11 (secure 0 0 0) 0x7f2ea8064c40 con 0x7f2eb0136340 2024-06-29T15:58:05.144 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:58:05.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 -- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2e94076540 msgr2=0x7f2e94078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2e94076540 0x7f2e94078a50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2eac034000 tx=0x7f2eac004730 comp rx=0 tx=0).stop 2024-06-29T15:58:05.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 -- 172.21.15.81:0/1727507212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2eb0136340 msgr2=0x7f2eb0072460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2eb0136340 0x7f2eb0072460 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800b9e0 tx=0x7f2ea800bea0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 -- 172.21.15.81:0/1727507212 shutdown_connections 2024-06-29T15:58:05.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2e94076540 0x7f2e94078a50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2eb0136340 0x7f2eb0072460 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2eb01412f0 0x7f2eb0071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 --2- 172.21.15.81:0/1727507212 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2eb00cf4d0 0x7f2eb015aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 -- 172.21.15.81:0/1727507212 >> 172.21.15.81:0/1727507212 conn(0x7f2eb006ccc0 msgr2=0x7f2eb0068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:05.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 -- 172.21.15.81:0/1727507212 shutdown_connections 2024-06-29T15:58:05.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.145+0000 7f2eb877f640 1 -- 172.21.15.81:0/1727507212 wait complete. 2024-06-29T15:58:05.157 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:05.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.270+0000 7fa190412640 1 -- 172.21.15.81:0/413795934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa174003a00 msgr2=0x7fa174003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.270+0000 7fa190412640 1 --2- 172.21.15.81:0/413795934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa174003a00 0x7fa174003e90 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7fa184009ed0 tx=0x7fa174005830 comp rx=0 tx=0).stop 2024-06-29T15:58:05.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.270+0000 7fa18e9af640 1 -- 172.21.15.81:0/413795934 reap_dead start 2024-06-29T15:58:05.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.270+0000 7fa190412640 1 -- 172.21.15.81:0/413795934 shutdown_connections 2024-06-29T15:58:05.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.270+0000 7fa190412640 1 -- 172.21.15.81:0/413795934 >> 172.21.15.81:0/413795934 conn(0x7fa1880b2920 msgr2=0x7fa1880b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:05.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.270+0000 7fa190412640 1 -- 172.21.15.81:0/413795934 shutdown_connections 2024-06-29T15:58:05.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.270+0000 7fa190412640 1 -- 172.21.15.81:0/413795934 wait complete. 2024-06-29T15:58:05.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.272+0000 7fa190412640 1 Processor -- start 2024-06-29T15:58:05.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.272+0000 7fa190412640 1 -- start start 2024-06-29T15:58:05.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa190412640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa174003a00 0x7fa188151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa190412640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa188157220 0x7fa18814a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa190412640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa188150fb0 0x7fa1881513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa190412640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa1880732d0 con 0x7fa174003a00 2024-06-29T15:58:05.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa190412640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa188073150 con 0x7fa188150fb0 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa190412640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa1880b4570 con 0x7fa188157220 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18e1ae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa174003a00 0x7fa188151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18e1ae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa174003a00 0x7fa188151b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47494/0 (socket says 172.21.15.81:47494) 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18e1ae640 1 -- 172.21.15.81:0/1072523723 learned_addr learned my addr 172.21.15.81:0/1072523723 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18e9af640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa188150fb0 0x7fa1881513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18d9ad640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa188157220 0x7fa18814a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18e1ae640 1 -- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa188157220 msgr2=0x7fa18814a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18e1ae640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa188157220 0x7fa18814a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18e1ae640 1 -- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa188150fb0 msgr2=0x7fa1881513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18e1ae640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa188150fb0 0x7fa1881513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18e1ae640 1 -- 172.21.15.81:0/1072523723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18814a630 con 0x7fa174003a00 2024-06-29T15:58:05.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18d9ad640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa188157220 0x7fa18814a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:05.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.273+0000 7fa18e1ae640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa174003a00 0x7fa188151b50 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7fa184036040 tx=0x7fa184033bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:05.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.274+0000 7fa17effd640 1 -- 172.21.15.81:0/1072523723 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184043070 con 0x7fa174003a00 2024-06-29T15:58:05.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.274+0000 7fa17effd640 1 -- 172.21.15.81:0/1072523723 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa184033db0 con 0x7fa174003a00 2024-06-29T15:58:05.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.274+0000 7fa190412640 1 -- 172.21.15.81:0/1072523723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa188151900 con 0x7fa174003a00 2024-06-29T15:58:05.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.274+0000 7fa17effd640 1 -- 172.21.15.81:0/1072523723 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184038040 con 0x7fa174003a00 2024-06-29T15:58:05.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.274+0000 7fa190412640 1 -- 172.21.15.81:0/1072523723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa188157a00 con 0x7fa174003a00 2024-06-29T15:58:05.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.275+0000 7fa190412640 1 -- 172.21.15.81:0/1072523723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa18806ed90 con 0x7fa174003a00 2024-06-29T15:58:05.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.275+0000 7fa17effd640 1 -- 172.21.15.81:0/1072523723 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fa184003bd0 con 0x7fa174003a00 2024-06-29T15:58:05.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.276+0000 7fa17effd640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa16c076510 0x7fa16c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.276+0000 7fa17effd640 1 -- 172.21.15.81:0/1072523723 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6752+0+0 (secure 0 0 0) 0x7fa1840b8930 con 0x7fa174003a00 2024-06-29T15:58:05.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.276+0000 7fa18d9ad640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa16c076510 0x7fa16c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.277+0000 7fa18d9ad640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa16c076510 0x7fa16c078a20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa178001330 tx=0x7fa178000ef0 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:05.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.280+0000 7fa17effd640 1 -- 172.21.15.81:0/1072523723 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa188151900 con 0x7fa174003a00 2024-06-29T15:58:05.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.500+0000 7fa190412640 1 -- 172.21.15.81:0/1072523723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa18815e300 con 0x7fa174003a00 2024-06-29T15:58:05.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.501+0000 7fa17effd640 1 -- 172.21.15.81:0/1072523723 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) ==== 95+0+5246 (secure 0 0 0) 0x7fa18814a630 con 0x7fa174003a00 2024-06-29T15:58:05.501 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:05.501 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":120,"btime":"2024-06-29T15:58:01:839815+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":11907,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/2295234543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2295234543},{"type":"v1","addr":"172.21.15.81:6833","nonce":2295234543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:33.727815+0000","modified":"2024-06-29T15:57:36.736776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10114":{"gid":10114,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10114,"qdb_cluster":[10114]},"id":14}]} 2024-06-29T15:58:05.502 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 120 2024-06-29T15:58:05.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 -- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa16c076510 msgr2=0x7fa16c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa16c076510 0x7fa16c078a20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa178001330 tx=0x7fa178000ef0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 -- 172.21.15.81:0/1072523723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa174003a00 msgr2=0x7fa188151b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa174003a00 0x7fa188151b50 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7fa184036040 tx=0x7fa184033bc0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 -- 172.21.15.81:0/1072523723 shutdown_connections 2024-06-29T15:58:05.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa16c076510 0x7fa16c078a20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa188150fb0 0x7fa1881513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa188157220 0x7fa18814a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 --2- 172.21.15.81:0/1072523723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa174003a00 0x7fa188151b50 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 -- 172.21.15.81:0/1072523723 >> 172.21.15.81:0/1072523723 conn(0x7fa1880b2920 msgr2=0x7fa188158f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:05.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 -- 172.21.15.81:0/1072523723 shutdown_connections 2024-06-29T15:58:05.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.503+0000 7fa190412640 1 -- 172.21.15.81:0/1072523723 wait complete. 2024-06-29T15:58:05.514 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:05.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.627+0000 7fe28508f640 1 -- 172.21.15.81:0/3059389429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe268004800 msgr2=0x7fe268004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.627+0000 7fe28508f640 1 --2- 172.21.15.81:0/3059389429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe268004800 0x7fe268004c10 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fe27400c260 tx=0x7fe268005830 comp rx=0 tx=0).stop 2024-06-29T15:58:05.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.627+0000 7fe28508f640 1 -- 172.21.15.81:0/3059389429 shutdown_connections 2024-06-29T15:58:05.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.627+0000 7fe28508f640 1 --2- 172.21.15.81:0/3059389429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe268004800 0x7fe268004c10 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.627+0000 7fe28508f640 1 -- 172.21.15.81:0/3059389429 >> 172.21.15.81:0/3059389429 conn(0x7fe2800b3510 msgr2=0x7fe2800b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:05.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.627+0000 7fe28508f640 1 -- 172.21.15.81:0/3059389429 shutdown_connections 2024-06-29T15:58:05.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.627+0000 7fe28508f640 1 -- 172.21.15.81:0/3059389429 wait complete. 2024-06-29T15:58:05.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.629+0000 7fe28508f640 1 Processor -- start 2024-06-29T15:58:05.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.629+0000 7fe28508f640 1 -- start start 2024-06-29T15:58:05.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe28508f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe268004800 0x7fe2800aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe28508f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2801db290 0x7fe2800caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe28508f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2801db6a0 0x7fe2800caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe28508f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe2800d0660 con 0x7fe2801db6a0 2024-06-29T15:58:05.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe28508f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe2800d04e0 con 0x7fe2801db290 2024-06-29T15:58:05.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe28508f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe2800d07e0 con 0x7fe268004800 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27f59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2801db6a0 0x7fe2800caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27f59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2801db6a0 0x7fe2800caf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47506/0 (socket says 172.21.15.81:47506) 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27f59e640 1 -- 172.21.15.81:0/1376857473 learned_addr learned my addr 172.21.15.81:0/1376857473 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27ed9d640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe268004800 0x7fe2800aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27e59c640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2801db290 0x7fe2800caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27f59e640 1 -- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe268004800 msgr2=0x7fe2800aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27f59e640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe268004800 0x7fe2800aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27f59e640 1 -- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2801db290 msgr2=0x7fe2800caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27f59e640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2801db290 0x7fe2800caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27f59e640 1 -- 172.21.15.81:0/1376857473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2801e8610 con 0x7fe2801db6a0 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe27f59e640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2801db6a0 0x7fe2800caf80 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fe2800cbf80 tx=0x7fe2780019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe2677fe640 1 -- 172.21.15.81:0/1376857473 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278011870 con 0x7fe2801db6a0 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe2677fe640 1 -- 172.21.15.81:0/1376857473 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe278017070 con 0x7fe2801db6a0 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe2677fe640 1 -- 172.21.15.81:0/1376857473 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278011da0 con 0x7fe2801db6a0 2024-06-29T15:58:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.631+0000 7fe28508f640 1 -- 172.21.15.81:0/1376857473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2801e88f0 con 0x7fe2801db6a0 2024-06-29T15:58:05.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.632+0000 7fe28508f640 1 -- 172.21.15.81:0/1376857473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2801e56d0 con 0x7fe2801db6a0 2024-06-29T15:58:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.632+0000 7fe28508f640 1 -- 172.21.15.81:0/1376857473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2801e8380 con 0x7fe2801db6a0 2024-06-29T15:58:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.633+0000 7fe2677fe640 1 -- 172.21.15.81:0/1376857473 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fe27801c460 con 0x7fe2801db6a0 2024-06-29T15:58:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.634+0000 7fe2677fe640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe26c076510 0x7fe26c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.634+0000 7fe2677fe640 1 -- 172.21.15.81:0/1376857473 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6752+0+0 (secure 0 0 0) 0x7fe278098680 con 0x7fe2801db6a0 2024-06-29T15:58:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.634+0000 7fe27ed9d640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe26c076510 0x7fe26c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.634+0000 7fe27ed9d640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe26c076510 0x7fe26c078a20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe270009cd0 tx=0x7fe270005f90 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:05.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.638+0000 7fe2677fe640 1 -- 172.21.15.81:0/1376857473 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe278061760 con 0x7fe2801db6a0 2024-06-29T15:58:05.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.819+0000 7fe28508f640 1 -- 172.21.15.81:0/1376857473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe2800b49d0 con 0x7fe2801db6a0 2024-06-29T15:58:05.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.821+0000 7fe2677fe640 1 -- 172.21.15.81:0/1376857473 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) ==== 74+0+18216 (secure 0 0 0) 0x7fe278061060 con 0x7fe2801db6a0 2024-06-29T15:58:05.821 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:05.821 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":222,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:03.847479+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.259110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:32.675276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6833/739691411":"2024-06-30T15:58:00.835804+0000","172.21.15.167:0/2503226350":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/2924173136":"2024-06-30T15:57:57.317102+0000","172.21.15.81:6832/739691411":"2024-06-30T15:58:00.835804+0000","172.21.15.167:6833/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6832/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3022587441":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/575713984":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/1101183266":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3261132296":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3042354413":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1619575755":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/760717235":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/3781943725":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/376674682":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1164120698":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3640842563":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/2165320595":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3538782057":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6833/2018755589":"2024-06-30T15:57:48.595362+0000","172.21.15.167:6832/2018755589":"2024-06-30T15:57:48.595362+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:05.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 -- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe26c076510 msgr2=0x7fe26c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe26c076510 0x7fe26c078a20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe270009cd0 tx=0x7fe270005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:05.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 -- 172.21.15.81:0/1376857473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2801db6a0 msgr2=0x7fe2800caf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2801db6a0 0x7fe2800caf80 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fe2800cbf80 tx=0x7fe2780019b0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 -- 172.21.15.81:0/1376857473 shutdown_connections 2024-06-29T15:58:05.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe26c076510 0x7fe26c078a20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2801db6a0 0x7fe2800caf80 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe2801db290 0x7fe2800caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 --2- 172.21.15.81:0/1376857473 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe268004800 0x7fe2800aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 -- 172.21.15.81:0/1376857473 >> 172.21.15.81:0/1376857473 conn(0x7fe2800b3510 msgr2=0x7fe2800cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:05.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 -- 172.21.15.81:0/1376857473 shutdown_connections 2024-06-29T15:58:05.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.823+0000 7fe28508f640 1 -- 172.21.15.81:0/1376857473 wait complete. 2024-06-29T15:58:05.835 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-06-29T15:58:05.835 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:05.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.948+0000 7f958eace640 1 -- 172.21.15.81:0/1586804122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95880cea10 msgr2=0x7f95880b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.948+0000 7f958eace640 1 --2- 172.21.15.81:0/1586804122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95880cea10 0x7f95880b47b0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f95880b2c10 tx=0x7f957802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.948+0000 7f958eace640 1 -- 172.21.15.81:0/1586804122 shutdown_connections 2024-06-29T15:58:05.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.948+0000 7f958eace640 1 --2- 172.21.15.81:0/1586804122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95880cea10 0x7f95880b47b0 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.948+0000 7f958eace640 1 --2- 172.21.15.81:0/1586804122 >> v2:172.21.15.167:3300/0 conn(0x7f95880b6d60 0x7f95880ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.948+0000 7f958eace640 1 -- 172.21.15.81:0/1586804122 >> 172.21.15.81:0/1586804122 conn(0x7f95880d1960 msgr2=0x7f95880d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:05.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.948+0000 7f958eace640 1 -- 172.21.15.81:0/1586804122 shutdown_connections 2024-06-29T15:58:05.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.948+0000 7f958eace640 1 -- 172.21.15.81:0/1586804122 wait complete. 2024-06-29T15:58:05.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.950+0000 7f958eace640 1 Processor -- start 2024-06-29T15:58:05.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.950+0000 7f958eace640 1 -- start start 2024-06-29T15:58:05.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958eace640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95880b4cf0 0x7f95881e3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958eace640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95880b6d60 0x7f95880cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958eace640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95880cea10 0x7f95881e7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958eace640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f95880cd540 con 0x7f95880cea10 2024-06-29T15:58:05.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958eace640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f95880d0a10 con 0x7f95880b6d60 2024-06-29T15:58:05.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958eace640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f95880cd6c0 con 0x7f95880b4cf0 2024-06-29T15:58:05.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958d06b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95880cea10 0x7f95881e7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958d06b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95880cea10 0x7f95881e7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47522/0 (socket says 172.21.15.81:47522) 2024-06-29T15:58:05.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958d06b640 1 -- 172.21.15.81:0/758074980 learned_addr learned my addr 172.21.15.81:0/758074980 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:05.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f957ffff640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95880b6d60 0x7f95880cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958c86a640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95880b4cf0 0x7f95881e3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958d06b640 1 -- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95880b4cf0 msgr2=0x7f95881e3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958d06b640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95880b4cf0 0x7f95881e3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958d06b640 1 -- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95880b6d60 msgr2=0x7f95880cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958d06b640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95880b6d60 0x7f95880cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958d06b640 1 -- 172.21.15.81:0/758074980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95880c5780 con 0x7f95880cea10 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958c86a640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95880b4cf0 0x7f95881e3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f958d06b640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95880cea10 0x7f95881e7fc0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f958000c6e0 tx=0x7f958000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.952+0000 7f957d7fa640 1 -- 172.21.15.81:0/758074980 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580003b70 con 0x7f95880cea10 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.953+0000 7f957d7fa640 1 -- 172.21.15.81:0/758074980 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9580003d30 con 0x7f95880cea10 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.953+0000 7f958eace640 1 -- 172.21.15.81:0/758074980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95881e74e0 con 0x7f95880cea10 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.953+0000 7f957d7fa640 1 -- 172.21.15.81:0/758074980 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580005330 con 0x7f95880cea10 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.953+0000 7f958eace640 1 -- 172.21.15.81:0/758074980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95881e7aa0 con 0x7f95880cea10 2024-06-29T15:58:05.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.954+0000 7f958eace640 1 -- 172.21.15.81:0/758074980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9588071180 con 0x7f95880cea10 2024-06-29T15:58:05.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.954+0000 7f957d7fa640 1 -- 172.21.15.81:0/758074980 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f95800054d0 con 0x7f95880cea10 2024-06-29T15:58:05.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.955+0000 7f957d7fa640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9574076510 0x7f9574078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:05.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.955+0000 7f957d7fa640 1 -- 172.21.15.81:0/758074980 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6752+0+0 (secure 0 0 0) 0x7f9580097d50 con 0x7f95880cea10 2024-06-29T15:58:05.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.955+0000 7f958c86a640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9574076510 0x7f9574078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:05.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.956+0000 7f958c86a640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9574076510 0x7f9574078a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9584005fd0 tx=0x7f958400f360 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:05.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:05.960+0000 7f957d7fa640 1 -- 172.21.15.81:0/758074980 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9588071180 con 0x7f95880cea10 2024-06-29T15:58:06.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.175+0000 7f958eace640 1 -- 172.21.15.81:0/758074980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95880ce4d0 con 0x7f95880cea10 2024-06-29T15:58:06.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.176+0000 7f957d7fa640 1 -- 172.21.15.81:0/758074980 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) ==== 95+0+5246 (secure 0 0 0) 0x7f958009b030 con 0x7f95880cea10 2024-06-29T15:58:06.176 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:06.177 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":120,"btime":"2024-06-29T15:58:01:839815+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":11907,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/2295234543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2295234543},{"type":"v1","addr":"172.21.15.81:6833","nonce":2295234543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:33.727815+0000","modified":"2024-06-29T15:57:36.736776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10114":{"gid":10114,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10114,"qdb_cluster":[10114]},"id":14}]} 2024-06-29T15:58:06.177 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 120 2024-06-29T15:58:06.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 -- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9574076510 msgr2=0x7f9574078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9574076510 0x7f9574078a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9584005fd0 tx=0x7f958400f360 comp rx=0 tx=0).stop 2024-06-29T15:58:06.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 -- 172.21.15.81:0/758074980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95880cea10 msgr2=0x7f95881e7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95880cea10 0x7f95881e7fc0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f958000c6e0 tx=0x7f958000cba0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 -- 172.21.15.81:0/758074980 shutdown_connections 2024-06-29T15:58:06.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9574076510 0x7f9574078a20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95880cea10 0x7f95881e7fc0 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f95880b6d60 0x7f95880cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 --2- 172.21.15.81:0/758074980 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f95880b4cf0 0x7f95881e3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 -- 172.21.15.81:0/758074980 >> 172.21.15.81:0/758074980 conn(0x7f95880d1960 msgr2=0x7f9588070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 -- 172.21.15.81:0/758074980 shutdown_connections 2024-06-29T15:58:06.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.178+0000 7f958eace640 1 -- 172.21.15.81:0/758074980 wait complete. 2024-06-29T15:58:06.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:06.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.301+0000 7fbca933c640 1 -- 172.21.15.81:0/3547785715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca40b57f0 msgr2=0x7fbca4072f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.301+0000 7fbca933c640 1 --2- 172.21.15.81:0/3547785715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca40b57f0 0x7fbca4072f10 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fbca4072d10 tx=0x7fbc9802a4d0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.301+0000 7fbca933c640 1 -- 172.21.15.81:0/3547785715 shutdown_connections 2024-06-29T15:58:06.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.301+0000 7fbca933c640 1 --2- 172.21.15.81:0/3547785715 >> v2:172.21.15.167:3300/0 conn(0x7fbca4073450 0x7fbca415e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.301+0000 7fbca933c640 1 --2- 172.21.15.81:0/3547785715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca40b57f0 0x7fbca4072f10 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.301+0000 7fbca933c640 1 --2- 172.21.15.81:0/3547785715 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbca4143860 0x7fbca40b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.301+0000 7fbca933c640 1 -- 172.21.15.81:0/3547785715 >> 172.21.15.81:0/3547785715 conn(0x7fbca40d0b70 msgr2=0x7fbca40d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.301+0000 7fbca933c640 1 -- 172.21.15.81:0/3547785715 shutdown_connections 2024-06-29T15:58:06.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.301+0000 7fbca933c640 1 -- 172.21.15.81:0/3547785715 wait complete. 2024-06-29T15:58:06.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.303+0000 7fbca933c640 1 Processor -- start 2024-06-29T15:58:06.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.303+0000 7fbca933c640 1 -- start start 2024-06-29T15:58:06.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca933c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca4073450 0x7fbca40b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca933c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbca40b57f0 0x7fbca413cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca933c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbca4143860 0x7fbca40cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca933c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbca415ed20 con 0x7fbca4073450 2024-06-29T15:58:06.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca933c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbca415eea0 con 0x7fbca40b57f0 2024-06-29T15:58:06.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca933c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbca415f020 con 0x7fbca4143860 2024-06-29T15:58:06.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca4073450 0x7fbca40b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca4073450 0x7fbca40b05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47536/0 (socket says 172.21.15.81:47536) 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca2ffd640 1 -- 172.21.15.81:0/1991643974 learned_addr learned my addr 172.21.15.81:0/1991643974 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca37fe640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbca4143860 0x7fbca40cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.304+0000 7fbca27fc640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbca40b57f0 0x7fbca413cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbca2ffd640 1 -- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbca4143860 msgr2=0x7fbca40cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbca2ffd640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbca4143860 0x7fbca40cf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbca2ffd640 1 -- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbca40b57f0 msgr2=0x7fbca413cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbca2ffd640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbca40b57f0 0x7fbca413cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbca2ffd640 1 -- 172.21.15.81:0/1991643974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca40b0af0 con 0x7fbca4073450 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbca2ffd640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca4073450 0x7fbca40b05b0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400cd10 tx=0x7fbc940075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbc8b7fe640 1 -- 172.21.15.81:0/1991643974 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94004b20 con 0x7fbca4073450 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbc8b7fe640 1 -- 172.21.15.81:0/1991643974 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc94004ce0 con 0x7fbca4073450 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbca933c640 1 -- 172.21.15.81:0/1991643974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca40b0f70 con 0x7fbca4073450 2024-06-29T15:58:06.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbc8b7fe640 1 -- 172.21.15.81:0/1991643974 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94005280 con 0x7fbca4073450 2024-06-29T15:58:06.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.305+0000 7fbca933c640 1 -- 172.21.15.81:0/1991643974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca40b1530 con 0x7fbca4073450 2024-06-29T15:58:06.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.306+0000 7fbca933c640 1 -- 172.21.15.81:0/1991643974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca415f4b0 con 0x7fbca4073450 2024-06-29T15:58:06.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.307+0000 7fbc8b7fe640 1 -- 172.21.15.81:0/1991643974 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fbc94005420 con 0x7fbca4073450 2024-06-29T15:58:06.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.307+0000 7fbc8b7fe640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbc90076510 0x7fbc90078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.307+0000 7fbc8b7fe640 1 -- 172.21.15.81:0/1991643974 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6752+0+0 (secure 0 0 0) 0x7fbc940986e0 con 0x7fbca4073450 2024-06-29T15:58:06.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.308+0000 7fbca27fc640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbc90076510 0x7fbc90078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.308+0000 7fbca27fc640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbc90076510 0x7fbc90078a20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc98033000 tx=0x7fbc98004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.312+0000 7fbc8b7fe640 1 -- 172.21.15.81:0/1991643974 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbc94014030 con 0x7fbca4073450 2024-06-29T15:58:06.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.531+0000 7fbca933c640 1 -- 172.21.15.81:0/1991643974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbca4154290 con 0x7fbca4073450 2024-06-29T15:58:06.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.532+0000 7fbc8b7fe640 1 -- 172.21.15.81:0/1991643974 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) ==== 95+0+5246 (secure 0 0 0) 0x7fbc94061200 con 0x7fbca4073450 2024-06-29T15:58:06.532 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:06.532 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":120,"btime":"2024-06-29T15:58:01:839815+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":11907,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/2295234543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2295234543},{"type":"v1","addr":"172.21.15.81:6833","nonce":2295234543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:57:33.727815+0000","modified":"2024-06-29T15:57:36.736776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10114":{"gid":10114,"name":"d","rank":0,"incarnation":114,"state":"up:active","state_seq":6,"addr":"172.21.15.167:6837/1664603190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":1664603190},{"type":"v1","addr":"172.21.15.167:6837","nonce":1664603190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10114,"qdb_cluster":[10114]},"id":14}]} 2024-06-29T15:58:06.532 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 120 2024-06-29T15:58:06.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 -- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbc90076510 msgr2=0x7fbc90078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbc90076510 0x7fbc90078a20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc98033000 tx=0x7fbc98004730 comp rx=0 tx=0).stop 2024-06-29T15:58:06.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 -- 172.21.15.81:0/1991643974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca4073450 msgr2=0x7fbca40b05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca4073450 0x7fbca40b05b0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400cd10 tx=0x7fbc940075f0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 -- 172.21.15.81:0/1991643974 shutdown_connections 2024-06-29T15:58:06.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbc90076510 0x7fbc90078a20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbca4143860 0x7fbca40cf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbca40b57f0 0x7fbca413cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 --2- 172.21.15.81:0/1991643974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca4073450 0x7fbca40b05b0 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 -- 172.21.15.81:0/1991643974 >> 172.21.15.81:0/1991643974 conn(0x7fbca40d0b70 msgr2=0x7fbca4142310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 -- 172.21.15.81:0/1991643974 shutdown_connections 2024-06-29T15:58:06.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.534+0000 7fbca933c640 1 -- 172.21.15.81:0/1991643974 wait complete. 2024-06-29T15:58:06.545 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:06.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.658+0000 7f1a1b04e640 1 -- 172.21.15.81:0/2262178404 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1406f660 msgr2=0x7f1a1406fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.658+0000 7f1a1b04e640 1 --2- 172.21.15.81:0/2262178404 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1406f660 0x7f1a1406fa70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c005fd0 tx=0x7f1a0c02a820 comp rx=0 tx=0).stop 2024-06-29T15:58:06.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.658+0000 7f1a1b04e640 1 -- 172.21.15.81:0/2262178404 shutdown_connections 2024-06-29T15:58:06.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.658+0000 7f1a1b04e640 1 --2- 172.21.15.81:0/2262178404 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a1406f660 0x7f1a1406fa70 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.658+0000 7f1a1b04e640 1 --2- 172.21.15.81:0/2262178404 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a140b35b0 0x7f1a1415b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.658+0000 7f1a1b04e640 1 -- 172.21.15.81:0/2262178404 >> 172.21.15.81:0/2262178404 conn(0x7f1a1406ccc0 msgr2=0x7f1a140b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.658+0000 7f1a1b04e640 1 -- 172.21.15.81:0/2262178404 shutdown_connections 2024-06-29T15:58:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.658+0000 7f1a1b04e640 1 -- 172.21.15.81:0/2262178404 wait complete. 2024-06-29T15:58:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.660+0000 7f1a1b04e640 1 Processor -- start 2024-06-29T15:58:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.660+0000 7f1a1b04e640 1 -- start start 2024-06-29T15:58:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.661+0000 7f1a1b04e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1406f660 0x7f1a14154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.661+0000 7f1a1b04e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a140b35b0 0x7f1a14142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.661+0000 7f1a1b04e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a140b4430 0x7f1a14152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.661+0000 7f1a1b04e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1406f160 con 0x7f1a140b4430 2024-06-29T15:58:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a1b04e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1406efe0 con 0x7f1a140b35b0 2024-06-29T15:58:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a1b04e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1406f2e0 con 0x7f1a1406f660 2024-06-29T15:58:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a195eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a140b4430 0x7f1a14152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a195eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a140b4430 0x7f1a14152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47538/0 (socket says 172.21.15.81:47538) 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a195eb640 1 -- 172.21.15.81:0/3280987338 learned_addr learned my addr 172.21.15.81:0/3280987338 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a18dea640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1406f660 0x7f1a14154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a13fff640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a140b35b0 0x7f1a14142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a195eb640 1 -- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1406f660 msgr2=0x7f1a14154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a195eb640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1406f660 0x7f1a14154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a195eb640 1 -- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a140b35b0 msgr2=0x7f1a14142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a195eb640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a140b35b0 0x7f1a14142170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a195eb640 1 -- 172.21.15.81:0/3280987338 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a14150f70 con 0x7f1a140b4430 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a195eb640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a140b4430 0x7f1a14152760 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009b30 tx=0x7f1a0c032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a117fa640 1 -- 172.21.15.81:0/3280987338 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c037040 con 0x7f1a140b4430 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a1b04e640 1 -- 172.21.15.81:0/3280987338 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a141534e0 con 0x7f1a140b4430 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a117fa640 1 -- 172.21.15.81:0/3280987338 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a0c042070 con 0x7f1a140b4430 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a117fa640 1 -- 172.21.15.81:0/3280987338 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c003e60 con 0x7f1a140b4430 2024-06-29T15:58:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.662+0000 7f1a1b04e640 1 -- 172.21.15.81:0/3280987338 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a14153aa0 con 0x7f1a140b4430 2024-06-29T15:58:06.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.663+0000 7f1a1b04e640 1 -- 172.21.15.81:0/3280987338 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a1406fa70 con 0x7f1a140b4430 2024-06-29T15:58:06.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.664+0000 7f1a117fa640 1 -- 172.21.15.81:0/3280987338 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f1a0c004280 con 0x7f1a140b4430 2024-06-29T15:58:06.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.664+0000 7f1a117fa640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1a00076510 0x7f1a00078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.665+0000 7f1a117fa640 1 -- 172.21.15.81:0/3280987338 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6752+0+0 (secure 0 0 0) 0x7f1a0c0b8cb0 con 0x7f1a140b4430 2024-06-29T15:58:06.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.665+0000 7f1a18dea640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1a00076510 0x7f1a00078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.665+0000 7f1a18dea640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1a00076510 0x7f1a00078a20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009530 tx=0x7f1a04009260 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.669+0000 7f1a117fa640 1 -- 172.21.15.81:0/3280987338 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a0c081750 con 0x7f1a140b4430 2024-06-29T15:58:06.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.850+0000 7f1a1b04e640 1 -- 172.21.15.81:0/3280987338 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a14167100 con 0x7f1a140b4430 2024-06-29T15:58:06.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.851+0000 7f1a117fa640 1 -- 172.21.15.81:0/3280987338 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) ==== 74+0+18216 (secure 0 0 0) 0x7f1a0c085a50 con 0x7f1a140b4430 2024-06-29T15:58:06.852 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:06.852 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":222,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:03.847479+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:57:32.259110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"216","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:57:32.675276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6833/739691411":"2024-06-30T15:58:00.835804+0000","172.21.15.167:0/2503226350":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/2924173136":"2024-06-30T15:57:57.317102+0000","172.21.15.81:6832/739691411":"2024-06-30T15:58:00.835804+0000","172.21.15.167:6833/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6832/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3022587441":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/575713984":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/1101183266":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3261132296":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3042354413":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1619575755":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/760717235":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/3781943725":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/376674682":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1164120698":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3640842563":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/2165320595":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3538782057":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6833/2018755589":"2024-06-30T15:57:48.595362+0000","172.21.15.167:6832/2018755589":"2024-06-30T15:57:48.595362+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 -- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1a00076510 msgr2=0x7f1a00078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1a00076510 0x7f1a00078a20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009530 tx=0x7f1a04009260 comp rx=0 tx=0).stop 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 -- 172.21.15.81:0/3280987338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a140b4430 msgr2=0x7f1a14152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a140b4430 0x7f1a14152760 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009b30 tx=0x7f1a0c032d10 comp rx=0 tx=0).stop 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 -- 172.21.15.81:0/3280987338 shutdown_connections 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1a00076510 0x7f1a00078a20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a140b4430 0x7f1a14152760 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a140b35b0 0x7f1a14142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 --2- 172.21.15.81:0/3280987338 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a1406f660 0x7f1a14154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 -- 172.21.15.81:0/3280987338 >> 172.21.15.81:0/3280987338 conn(0x7f1a1406ccc0 msgr2=0x7f1a14072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 -- 172.21.15.81:0/3280987338 shutdown_connections 2024-06-29T15:58:06.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.853+0000 7f1a1b04e640 1 -- 172.21.15.81:0/3280987338 wait complete. 2024-06-29T15:58:06.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-06-29T15:58:06.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.980+0000 7f5d34f19640 1 -- 172.21.15.81:0/1251462050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d300b3640 msgr2=0x7f5d3015ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.980+0000 7f5d34f19640 1 --2- 172.21.15.81:0/1251462050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d300b3640 0x7f5d3015ab20 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f5d20005fd0 tx=0x7f5d2002a810 comp rx=0 tx=0).stop 2024-06-29T15:58:06.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.980+0000 7f5d34f19640 1 -- 172.21.15.81:0/1251462050 shutdown_connections 2024-06-29T15:58:06.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.980+0000 7f5d34f19640 1 --2- 172.21.15.81:0/1251462050 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d3015b060 0x7f5d3006f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.980+0000 7f5d34f19640 1 --2- 172.21.15.81:0/1251462050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d300b3640 0x7f5d3015ab20 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.980+0000 7f5d34f19640 1 -- 172.21.15.81:0/1251462050 >> 172.21.15.81:0/1251462050 conn(0x7f5d3006ccc0 msgr2=0x7f5d300b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:06.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.980+0000 7f5d34f19640 1 -- 172.21.15.81:0/1251462050 shutdown_connections 2024-06-29T15:58:06.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.980+0000 7f5d34f19640 1 -- 172.21.15.81:0/1251462050 wait complete. 2024-06-29T15:58:06.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.982+0000 7f5d34f19640 1 Processor -- start 2024-06-29T15:58:06.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.982+0000 7f5d34f19640 1 -- start start 2024-06-29T15:58:06.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d34f19640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d3006fba0 0x7f5d30151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d34f19640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d300b3640 0x7f5d30147670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d34f19640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d3015b060 0x7f5d30147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d34f19640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5d300cf740 con 0x7f5d300b3640 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d34f19640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5d300cf5c0 con 0x7f5d3015b060 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d34f19640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5d30072630 con 0x7f5d3006fba0 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d300b3640 0x7f5d30147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d300b3640 0x7f5d30147670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47552/0 (socket says 172.21.15.81:47552) 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2dd9b640 1 -- 172.21.15.81:0/415426938 learned_addr learned my addr 172.21.15.81:0/415426938 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2e59c640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d3006fba0 0x7f5d30151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2ed9d640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d3015b060 0x7f5d30147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2dd9b640 1 -- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d3006fba0 msgr2=0x7f5d30151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2dd9b640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d3006fba0 0x7f5d30151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2dd9b640 1 -- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d3015b060 msgr2=0x7f5d30147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2dd9b640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d3015b060 0x7f5d30147e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2dd9b640 1 -- 172.21.15.81:0/415426938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d30150070 con 0x7f5d300b3640 2024-06-29T15:58:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.983+0000 7f5d2e59c640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d3006fba0 0x7f5d30151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.984+0000 7f5d2dd9b640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d300b3640 0x7f5d30147670 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400d6b0 tx=0x7f5d2400db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.984+0000 7f5d1effd640 1 -- 172.21.15.81:0/415426938 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d24017070 con 0x7f5d300b3640 2024-06-29T15:58:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.984+0000 7f5d1effd640 1 -- 172.21.15.81:0/415426938 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d24005310 con 0x7f5d300b3640 2024-06-29T15:58:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.984+0000 7f5d1effd640 1 -- 172.21.15.81:0/415426938 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d24009820 con 0x7f5d300b3640 2024-06-29T15:58:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.984+0000 7f5d34f19640 1 -- 172.21.15.81:0/415426938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d30147bb0 con 0x7f5d300b3640 2024-06-29T15:58:06.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.984+0000 7f5d34f19640 1 -- 172.21.15.81:0/415426938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d301507c0 con 0x7f5d300b3640 2024-06-29T15:58:06.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.985+0000 7f5d34f19640 1 -- 172.21.15.81:0/415426938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d30072be0 con 0x7f5d300b3640 2024-06-29T15:58:06.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.986+0000 7f5d1effd640 1 -- 172.21.15.81:0/415426938 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f5d240099c0 con 0x7f5d300b3640 2024-06-29T15:58:06.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.986+0000 7f5d1effd640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5d10076510 0x7f5d10078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:06.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.986+0000 7f5d1effd640 1 -- 172.21.15.81:0/415426938 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 6752+0+0 (secure 0 0 0) 0x7f5d240980d0 con 0x7f5d300b3640 2024-06-29T15:58:06.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.986+0000 7f5d2e59c640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5d10076510 0x7f5d10078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:06.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.987+0000 7f5d2e59c640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5d10076510 0x7f5d10078a20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5d20034000 tx=0x7f5d20004ab0 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:06.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:06.990+0000 7f5d1effd640 1 -- 172.21.15.81:0/415426938 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d240611b0 con 0x7f5d300b3640 2024-06-29T15:58:07.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.205+0000 7f5d34f19640 1 -- 172.21.15.81:0/415426938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) -- 0x7f5d30150240 con 0x7f5d300b3640 2024-06-29T15:58:07.206 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T15:58:07.206+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:58:07.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.208+0000 7f5d1effd640 1 -- 172.21.15.81:0/415426938 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v121) ==== 167+0+0 (secure 0 0 0) 0x7f5d240691a0 con 0x7f5d300b3640 2024-06-29T15:58:07.208 INFO:teuthology.orchestra.run.smithi081.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:58:07.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.210+0000 7f5d34f19640 1 -- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5d10076510 msgr2=0x7f5d10078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.210+0000 7f5d34f19640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5d10076510 0x7f5d10078a20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5d20034000 tx=0x7f5d20004ab0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.210+0000 7f5d34f19640 1 -- 172.21.15.81:0/415426938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d300b3640 msgr2=0x7f5d30147670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.210+0000 7f5d34f19640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d300b3640 0x7f5d30147670 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400d6b0 tx=0x7f5d2400db70 comp rx=0 tx=0).stop 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.210+0000 7f5d34f19640 1 -- 172.21.15.81:0/415426938 shutdown_connections 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.210+0000 7f5d34f19640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5d10076510 0x7f5d10078a20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.210+0000 7f5d34f19640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d3015b060 0x7f5d30147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.210+0000 7f5d34f19640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d300b3640 0x7f5d30147670 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.210+0000 7f5d34f19640 1 --2- 172.21.15.81:0/415426938 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d3006fba0 0x7f5d30151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.210+0000 7f5d34f19640 1 -- 172.21.15.81:0/415426938 >> 172.21.15.81:0/415426938 conn(0x7f5d3006ccc0 msgr2=0x7f5d3006ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.211+0000 7f5d34f19640 1 -- 172.21.15.81:0/415426938 shutdown_connections 2024-06-29T15:58:07.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.211+0000 7f5d34f19640 1 -- 172.21.15.81:0/415426938 wait complete. 2024-06-29T15:58:07.215 INFO:tasks.ceph.mds.d.smithi167.stderr: -1891> 2024-06-29T15:57:39.260+0000 7fed41148640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:07.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-06-29T15:58:07.270 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:58:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.338+0000 7fac55362640 1 -- 172.21.15.81:0/2131626163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac500b4800 msgr2=0x7fac50144590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.338+0000 7fac55362640 1 --2- 172.21.15.81:0/2131626163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac500b4800 0x7fac50144590 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fac50072ce0 tx=0x7fac4002aac0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.338+0000 7fac55362640 1 -- 172.21.15.81:0/2131626163 shutdown_connections 2024-06-29T15:58:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.338+0000 7fac55362640 1 --2- 172.21.15.81:0/2131626163 >> v2:172.21.15.167:3300/0 conn(0x7fac5015b400 0x7fac5015b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.338+0000 7fac55362640 1 --2- 172.21.15.81:0/2131626163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac500b4800 0x7fac50144590 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.338+0000 7fac55362640 1 -- 172.21.15.81:0/2131626163 >> 172.21.15.81:0/2131626163 conn(0x7fac500b2920 msgr2=0x7fac500b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.338+0000 7fac55362640 1 -- 172.21.15.81:0/2131626163 shutdown_connections 2024-06-29T15:58:07.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.338+0000 7fac55362640 1 -- 172.21.15.81:0/2131626163 wait complete. 2024-06-29T15:58:07.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.340+0000 7fac55362640 1 Processor -- start 2024-06-29T15:58:07.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.340+0000 7fac55362640 1 -- start start 2024-06-29T15:58:07.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac55362640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac50072f80 0x7fac5013e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac55362640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fac500b4800 0x7fac5006c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac55362640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac5015b400 0x7fac5015a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac55362640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac5006f1f0 con 0x7fac5015b400 2024-06-29T15:58:07.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac55362640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fac5006f070 con 0x7fac50072f80 2024-06-29T15:58:07.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac55362640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fac5006f370 con 0x7fac500b4800 2024-06-29T15:58:07.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac5015b400 0x7fac5015a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac4effd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac50072f80 0x7fac5013e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac5015b400 0x7fac5015a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47574/0 (socket says 172.21.15.81:47574) 2024-06-29T15:58:07.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac4f7fe640 1 -- 172.21.15.81:0/2918210351 learned_addr learned my addr 172.21.15.81:0/2918210351 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.341+0000 7fac4e7fc640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fac500b4800 0x7fac5006c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac4f7fe640 1 -- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fac500b4800 msgr2=0x7fac5006c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac4f7fe640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fac500b4800 0x7fac5006c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac4f7fe640 1 -- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac50072f80 msgr2=0x7fac5013e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac4f7fe640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac50072f80 0x7fac5013e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac4f7fe640 1 -- 172.21.15.81:0/2918210351 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5013f000 con 0x7fac5015b400 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac4f7fe640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac5015b400 0x7fac5015a640 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fac4800d650 tx=0x7fac4800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac377fe640 1 -- 172.21.15.81:0/2918210351 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48017070 con 0x7fac5015b400 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac377fe640 1 -- 172.21.15.81:0/2918210351 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac48005310 con 0x7fac5015b400 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac377fe640 1 -- 172.21.15.81:0/2918210351 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48009820 con 0x7fac5015b400 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac55362640 1 -- 172.21.15.81:0/2918210351 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5013f2e0 con 0x7fac5015b400 2024-06-29T15:58:07.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.342+0000 7fac55362640 1 -- 172.21.15.81:0/2918210351 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5013f8a0 con 0x7fac5015b400 2024-06-29T15:58:07.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.343+0000 7fac55362640 1 -- 172.21.15.81:0/2918210351 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac5006f820 con 0x7fac5015b400 2024-06-29T15:58:07.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.344+0000 7fac377fe640 1 -- 172.21.15.81:0/2918210351 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fac480099c0 con 0x7fac5015b400 2024-06-29T15:58:07.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.344+0000 7fac377fe640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fac3c076510 0x7fac3c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:07.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.344+0000 7fac377fe640 1 -- 172.21.15.81:0/2918210351 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 6838+0+0 (secure 0 0 0) 0x7fac480980a0 con 0x7fac5015b400 2024-06-29T15:58:07.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.344+0000 7fac4effd640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fac3c076510 0x7fac3c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:07.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.345+0000 7fac4effd640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fac3c076510 0x7fac3c078a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac40035000 tx=0x7fac40038040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:07.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.349+0000 7fac377fe640 1 -- 172.21.15.81:0/2918210351 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fac48060ad0 con 0x7fac5015b400 2024-06-29T15:58:07.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:07.566+0000 7fac55362640 1 -- 172.21.15.81:0/2918210351 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) -- 0x7fac5015bb30 con 0x7fac5015b400 2024-06-29T15:58:08.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.213+0000 7fac377fe640 1 -- 172.21.15.81:0/2918210351 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v122) ==== 107+0+0 (secure 0 0 0) 0x7fac48069110 con 0x7fac5015b400 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 -- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fac3c076510 msgr2=0x7fac3c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fac3c076510 0x7fac3c078a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac40035000 tx=0x7fac40038040 comp rx=0 tx=0).stop 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 -- 172.21.15.81:0/2918210351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac5015b400 msgr2=0x7fac5015a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac5015b400 0x7fac5015a640 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fac4800d650 tx=0x7fac4800db10 comp rx=0 tx=0).stop 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 -- 172.21.15.81:0/2918210351 shutdown_connections 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fac3c076510 0x7fac3c078a20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac5015b400 0x7fac5015a640 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fac500b4800 0x7fac5006c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 --2- 172.21.15.81:0/2918210351 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fac50072f80 0x7fac5013e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 -- 172.21.15.81:0/2918210351 >> 172.21.15.81:0/2918210351 conn(0x7fac500b2920 msgr2=0x7fac500d02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 -- 172.21.15.81:0/2918210351 shutdown_connections 2024-06-29T15:58:08.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.216+0000 7fac55362640 1 -- 172.21.15.81:0/2918210351 wait complete. 2024-06-29T15:58:08.230 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-06-29T15:58:08.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.354+0000 7fce49af6640 1 -- 172.21.15.81:0/1922638679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce440b3420 msgr2=0x7fce440b3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.354+0000 7fce49af6640 1 --2- 172.21.15.81:0/1922638679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce440b3420 0x7fce440b3830 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fce3c005fd0 tx=0x7fce3c02a820 comp rx=0 tx=0).stop 2024-06-29T15:58:08.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.354+0000 7fce49af6640 1 -- 172.21.15.81:0/1922638679 shutdown_connections 2024-06-29T15:58:08.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.354+0000 7fce49af6640 1 --2- 172.21.15.81:0/1922638679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce440b3420 0x7fce440b3830 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.354+0000 7fce49af6640 1 -- 172.21.15.81:0/1922638679 >> 172.21.15.81:0/1922638679 conn(0x7fce4406cd60 msgr2=0x7fce440b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:08.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.354+0000 7fce49af6640 1 -- 172.21.15.81:0/1922638679 shutdown_connections 2024-06-29T15:58:08.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.354+0000 7fce49af6640 1 -- 172.21.15.81:0/1922638679 wait complete. 2024-06-29T15:58:08.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.356+0000 7fce49af6640 1 Processor -- start 2024-06-29T15:58:08.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.356+0000 7fce49af6640 1 -- start start 2024-06-29T15:58:08.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce49af6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce44073b90 0x7fce44155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce49af6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce440b3420 0x7fce440cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce49af6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce440cbd60 0x7fce440ce530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce49af6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fce4406f840 con 0x7fce44073b90 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce49af6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fce4406f6c0 con 0x7fce440b3420 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce49af6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fce4406f9c0 con 0x7fce440cbd60 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce437fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce44073b90 0x7fce44155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce437fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce44073b90 0x7fce44155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47592/0 (socket says 172.21.15.81:47592) 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce437fe640 1 -- 172.21.15.81:0/1767898200 learned_addr learned my addr 172.21.15.81:0/1767898200 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce3bfff640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce440b3420 0x7fce440cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce43fff640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce440cbd60 0x7fce440ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce437fe640 1 -- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce440cbd60 msgr2=0x7fce440ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce437fe640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce440cbd60 0x7fce440ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce437fe640 1 -- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce440b3420 msgr2=0x7fce440cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce437fe640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce440b3420 0x7fce440cdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce437fe640 1 -- 172.21.15.81:0/1767898200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce441f1b70 con 0x7fce44073b90 2024-06-29T15:58:08.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.357+0000 7fce437fe640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce44073b90 0x7fce44155a20 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fce3400b420 tx=0x7fce3400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:08.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.358+0000 7fce40ff9640 1 -- 172.21.15.81:0/1767898200 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34011830 con 0x7fce44073b90 2024-06-29T15:58:08.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.358+0000 7fce40ff9640 1 -- 172.21.15.81:0/1767898200 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce34016070 con 0x7fce44073b90 2024-06-29T15:58:08.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.358+0000 7fce49af6640 1 -- 172.21.15.81:0/1767898200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce441537c0 con 0x7fce44073b90 2024-06-29T15:58:08.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.358+0000 7fce40ff9640 1 -- 172.21.15.81:0/1767898200 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34011d60 con 0x7fce44073b90 2024-06-29T15:58:08.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.358+0000 7fce49af6640 1 -- 172.21.15.81:0/1767898200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce440cd510 con 0x7fce44073b90 2024-06-29T15:58:08.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.359+0000 7fce49af6640 1 -- 172.21.15.81:0/1767898200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce440b3830 con 0x7fce44073b90 2024-06-29T15:58:08.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.359+0000 7fce40ff9640 1 -- 172.21.15.81:0/1767898200 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fce3401b460 con 0x7fce44073b90 2024-06-29T15:58:08.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.360+0000 7fce40ff9640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fce24076540 0x7fce24078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:08.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.360+0000 7fce40ff9640 1 -- 172.21.15.81:0/1767898200 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 6838+0+0 (secure 0 0 0) 0x7fce34097b90 con 0x7fce44073b90 2024-06-29T15:58:08.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.360+0000 7fce3bfff640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fce24076540 0x7fce24078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:08.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.361+0000 7fce3bfff640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fce24076540 0x7fce24078a50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fce44150410 tx=0x7fce2c009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:08.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.364+0000 7fce40ff9640 1 -- 172.21.15.81:0/1767898200 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fce34020020 con 0x7fce44073b90 2024-06-29T15:58:08.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:08.544+0000 7fce49af6640 1 -- 172.21.15.81:0/1767898200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fce44069550 con 0x7fce44073b90 2024-06-29T15:58:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.214+0000 7fce40ff9640 1 -- 172.21.15.81:0/1767898200 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v224) ==== 190+0+0 (secure 0 0 0) 0x7fce34060d20 con 0x7fce44073b90 2024-06-29T15:58:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'backup_fs_metadata' removed 2024-06-29T15:58:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.217+0000 7fce49af6640 1 -- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fce24076540 msgr2=0x7fce24078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.217+0000 7fce49af6640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fce24076540 0x7fce24078a50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fce44150410 tx=0x7fce2c009040 comp rx=0 tx=0).stop 2024-06-29T15:58:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.217+0000 7fce49af6640 1 -- 172.21.15.81:0/1767898200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce44073b90 msgr2=0x7fce44155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.217+0000 7fce49af6640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce44073b90 0x7fce44155a20 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7fce3400b420 tx=0x7fce3400b8e0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.217+0000 7fce49af6640 1 -- 172.21.15.81:0/1767898200 shutdown_connections 2024-06-29T15:58:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.217+0000 7fce49af6640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fce24076540 0x7fce24078a50 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fce44150410 tx=0x7fce2c009040 comp rx=0 tx=0).stop 2024-06-29T15:58:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.217+0000 7fce49af6640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fce440cbd60 0x7fce440ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.217+0000 7fce49af6640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fce440b3420 0x7fce440cdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.217+0000 7fce49af6640 1 --2- 172.21.15.81:0/1767898200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce44073b90 0x7fce44155a20 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.217+0000 7fce49af6640 1 -- 172.21.15.81:0/1767898200 >> 172.21.15.81:0/1767898200 conn(0x7fce4406cd60 msgr2=0x7fce44150680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.218+0000 7fce49af6640 1 -- 172.21.15.81:0/1767898200 shutdown_connections 2024-06-29T15:58:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.218+0000 7fce49af6640 1 -- 172.21.15.81:0/1767898200 wait complete. 2024-06-29T15:58:09.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-06-29T15:58:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.349+0000 7f76ce74a640 1 -- 172.21.15.81:0/1190151036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76c80d1e80 msgr2=0x7f76c80d2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.349+0000 7f76ce74a640 1 --2- 172.21.15.81:0/1190151036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76c80d1e80 0x7f76c80d2290 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f76b80098e0 tx=0x7f76b802a520 comp rx=0 tx=0).stop 2024-06-29T15:58:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.349+0000 7f76ce74a640 1 -- 172.21.15.81:0/1190151036 shutdown_connections 2024-06-29T15:58:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.349+0000 7f76ce74a640 1 --2- 172.21.15.81:0/1190151036 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76c80cea50 0x7f76c80cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.349+0000 7f76ce74a640 1 --2- 172.21.15.81:0/1190151036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76c80d1e80 0x7f76c80d2290 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.349+0000 7f76ce74a640 1 -- 172.21.15.81:0/1190151036 >> 172.21.15.81:0/1190151036 conn(0x7f76c80b5700 msgr2=0x7f76c80b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.349+0000 7f76ce74a640 1 -- 172.21.15.81:0/1190151036 shutdown_connections 2024-06-29T15:58:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.349+0000 7f76ce74a640 1 -- 172.21.15.81:0/1190151036 wait complete. 2024-06-29T15:58:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.351+0000 7f76ce74a640 1 Processor -- start 2024-06-29T15:58:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.351+0000 7f76ce74a640 1 -- start start 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76ce74a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76c80cea50 0x7f76c80a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76ce74a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76c80cf3a0 0x7f76c81de980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76ce74a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76c80d1e80 0x7f76c81deec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76ce74a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76c80d1960 con 0x7f76c80cf3a0 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76ce74a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f76c80d17e0 con 0x7f76c80cea50 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76ce74a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f76c80cf7b0 con 0x7f76c80d1e80 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76bffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76c80cf3a0 0x7f76c81de980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76bffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76c80cf3a0 0x7f76c81de980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47606/0 (socket says 172.21.15.81:47606) 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76bffff640 1 -- 172.21.15.81:0/3090353705 learned_addr learned my addr 172.21.15.81:0/3090353705 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76c7fff640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76c80cea50 0x7f76c80a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76ccce7640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76c80d1e80 0x7f76c81deec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76bffff640 1 -- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76c80d1e80 msgr2=0x7f76c81deec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76bffff640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76c80d1e80 0x7f76c81deec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76bffff640 1 -- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76c80cea50 msgr2=0x7f76c80a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76bffff640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76c80cea50 0x7f76c80a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76bffff640 1 -- 172.21.15.81:0/3090353705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c809dc00 con 0x7f76c80cf3a0 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76ccce7640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76c80d1e80 0x7f76c81deec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76bffff640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76c80cf3a0 0x7f76c81de980 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f76b000c740 tx=0x7f76b000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76c57fa640 1 -- 172.21.15.81:0/3090353705 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0007bb0 con 0x7f76c80cf3a0 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76c57fa640 1 -- 172.21.15.81:0/3090353705 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76b000ce50 con 0x7f76c80cf3a0 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76c57fa640 1 -- 172.21.15.81:0/3090353705 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0005280 con 0x7f76c80cf3a0 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76ce74a640 1 -- 172.21.15.81:0/3090353705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c809dee0 con 0x7f76c80cf3a0 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.353+0000 7f76ce74a640 1 -- 172.21.15.81:0/3090353705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c809e4a0 con 0x7f76c80cf3a0 2024-06-29T15:58:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.354+0000 7f76ce74a640 1 -- 172.21.15.81:0/3090353705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c809d970 con 0x7f76c80cf3a0 2024-06-29T15:58:09.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.355+0000 7f76c57fa640 1 -- 172.21.15.81:0/3090353705 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f76b0005420 con 0x7f76c80cf3a0 2024-06-29T15:58:09.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.356+0000 7f76c57fa640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f76a8076510 0x7f76a8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:09.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.356+0000 7f76c57fa640 1 -- 172.21.15.81:0/3090353705 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(224..224 src has 1..224) ==== 6394+0+0 (secure 0 0 0) 0x7f76b0098340 con 0x7f76c80cf3a0 2024-06-29T15:58:09.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.356+0000 7f76c7fff640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f76a8076510 0x7f76a8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:09.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.356+0000 7f76c7fff640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f76a8076510 0x7f76a8078a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f76b8034000 tx=0x7f76b8004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:09.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.360+0000 7f76c57fa640 1 -- 172.21.15.81:0/3090353705 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76b0014030 con 0x7f76c80cf3a0 2024-06-29T15:58:09.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:09.541+0000 7f76ce74a640 1 -- 172.21.15.81:0/3090353705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f76c80cee60 con 0x7f76c80cf3a0 2024-06-29T15:58:10.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.220+0000 7f76c57fa640 1 -- 172.21.15.81:0/3090353705 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v225) ==== 178+0+0 (secure 0 0 0) 0x7f76b0061580 con 0x7f76c80cf3a0 2024-06-29T15:58:10.220 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'backup_fs_data' removed 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 -- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f76a8076510 msgr2=0x7f76a8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f76a8076510 0x7f76a8078a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f76b8034000 tx=0x7f76b8004730 comp rx=0 tx=0).stop 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 -- 172.21.15.81:0/3090353705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76c80cf3a0 msgr2=0x7f76c81de980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76c80cf3a0 0x7f76c81de980 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f76b000c740 tx=0x7f76b000cc00 comp rx=0 tx=0).stop 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 -- 172.21.15.81:0/3090353705 shutdown_connections 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f76a8076510 0x7f76a8078a20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76c80d1e80 0x7f76c81deec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76c80cf3a0 0x7f76c81de980 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 --2- 172.21.15.81:0/3090353705 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76c80cea50 0x7f76c80a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 -- 172.21.15.81:0/3090353705 >> 172.21.15.81:0/3090353705 conn(0x7f76c80b5700 msgr2=0x7f76c81e6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:10.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 -- 172.21.15.81:0/3090353705 shutdown_connections 2024-06-29T15:58:10.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.222+0000 7f76ce74a640 1 -- 172.21.15.81:0/3090353705 wait complete. 2024-06-29T15:58:10.239 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:58:10.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.358+0000 7f225b21f640 1 -- 172.21.15.81:0/3942229266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f225406f660 msgr2=0x7f225406fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.358+0000 7f225b21f640 1 --2- 172.21.15.81:0/3942229266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f225406f660 0x7f225406fa70 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f2254070220 tx=0x7f224c02a980 comp rx=0 tx=0).stop 2024-06-29T15:58:10.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.358+0000 7f225b21f640 1 -- 172.21.15.81:0/3942229266 shutdown_connections 2024-06-29T15:58:10.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.358+0000 7f225b21f640 1 --2- 172.21.15.81:0/3942229266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f225406f660 0x7f225406fa70 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.358+0000 7f225b21f640 1 --2- 172.21.15.81:0/3942229266 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22540b4430 0x7f22540cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.358+0000 7f225b21f640 1 --2- 172.21.15.81:0/3942229266 >> v2:172.21.15.167:3300/0 conn(0x7f22540b35b0 0x7f225415b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.358+0000 7f225b21f640 1 -- 172.21.15.81:0/3942229266 >> 172.21.15.81:0/3942229266 conn(0x7f225406ccc0 msgr2=0x7f22540b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:10.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.358+0000 7f225b21f640 1 -- 172.21.15.81:0/3942229266 shutdown_connections 2024-06-29T15:58:10.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.358+0000 7f225b21f640 1 -- 172.21.15.81:0/3942229266 wait complete. 2024-06-29T15:58:10.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.360+0000 7f225b21f640 1 Processor -- start 2024-06-29T15:58:10.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.360+0000 7f225b21f640 1 -- start start 2024-06-29T15:58:10.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f225b21f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f225406f660 0x7f2254145650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f225b21f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22540b35b0 0x7f2254159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f225b21f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22540b4430 0x7f2254159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f225b21f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f225406f160 con 0x7f22540b35b0 2024-06-29T15:58:10.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f225b21f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f225406efe0 con 0x7f22540b4430 2024-06-29T15:58:10.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f225b21f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f225406f2e0 con 0x7f225406f660 2024-06-29T15:58:10.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f2253fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22540b35b0 0x7f2254159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f2253fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22540b35b0 0x7f2254159500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34540/0 (socket says 172.21.15.81:34540) 2024-06-29T15:58:10.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f2253fff640 1 -- 172.21.15.81:0/3035214943 learned_addr learned my addr 172.21.15.81:0/3035214943 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:10.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f2258fbb640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f225406f660 0x7f2254145650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f22597bc640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22540b4430 0x7f2254159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f2253fff640 1 -- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f225406f660 msgr2=0x7f2254145650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f2253fff640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f225406f660 0x7f2254145650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f2253fff640 1 -- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22540b4430 msgr2=0x7f2254159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f2253fff640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22540b4430 0x7f2254159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f2253fff640 1 -- 172.21.15.81:0/3035214943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2254145b90 con 0x7f22540b35b0 2024-06-29T15:58:10.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f2253fff640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22540b35b0 0x7f2254159500 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f224000d770 tx=0x7f224000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:10.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f22517fa640 1 -- 172.21.15.81:0/3035214943 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2240017070 con 0x7f22540b35b0 2024-06-29T15:58:10.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f22517fa640 1 -- 172.21.15.81:0/3035214943 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2240009d20 con 0x7f22540b35b0 2024-06-29T15:58:10.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f225b21f640 1 -- 172.21.15.81:0/3035214943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2254153860 con 0x7f22540b35b0 2024-06-29T15:58:10.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f22517fa640 1 -- 172.21.15.81:0/3035214943 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22400106c0 con 0x7f22540b35b0 2024-06-29T15:58:10.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.362+0000 7f225b21f640 1 -- 172.21.15.81:0/3035214943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2254153e20 con 0x7f22540b35b0 2024-06-29T15:58:10.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.364+0000 7f225b21f640 1 -- 172.21.15.81:0/3035214943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22540af660 con 0x7f22540b35b0 2024-06-29T15:58:10.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.364+0000 7f22517fa640 1 -- 172.21.15.81:0/3035214943 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f224001a070 con 0x7f22540b35b0 2024-06-29T15:58:10.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.365+0000 7f22517fa640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2238076510 0x7f2238078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.365+0000 7f22517fa640 1 -- 172.21.15.81:0/3035214943 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5974+0+0 (secure 0 0 0) 0x7f2240097e00 con 0x7f22540b35b0 2024-06-29T15:58:10.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.365+0000 7f2258fbb640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2238076510 0x7f2238078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.366+0000 7f2258fbb640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2238076510 0x7f2238078a20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2248005fd0 tx=0x7f224800b260 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:10.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.369+0000 7f22517fa640 1 -- 172.21.15.81:0/3035214943 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f22540af660 con 0x7f22540b35b0 2024-06-29T15:58:10.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.550+0000 7f225b21f640 1 -- 172.21.15.81:0/3035214943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f22541535f0 con 0x7f22540b35b0 2024-06-29T15:58:10.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.555+0000 7f22517fa640 1 -- 172.21.15.81:0/3035214943 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v75) ==== 71+0+0 (secure 0 0 0) 0x7f2240060b90 con 0x7f22540b35b0 2024-06-29T15:58:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 -- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2238076510 msgr2=0x7f2238078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2238076510 0x7f2238078a20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2248005fd0 tx=0x7f224800b260 comp rx=0 tx=0).stop 2024-06-29T15:58:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 -- 172.21.15.81:0/3035214943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22540b35b0 msgr2=0x7f2254159500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22540b35b0 0x7f2254159500 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f224000d770 tx=0x7f224000dc30 comp rx=0 tx=0).stop 2024-06-29T15:58:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 -- 172.21.15.81:0/3035214943 shutdown_connections 2024-06-29T15:58:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2238076510 0x7f2238078a20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22540b4430 0x7f2254159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22540b35b0 0x7f2254159500 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 --2- 172.21.15.81:0/3035214943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f225406f660 0x7f2254145650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 -- 172.21.15.81:0/3035214943 >> 172.21.15.81:0/3035214943 conn(0x7f225406ccc0 msgr2=0x7f2254071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 -- 172.21.15.81:0/3035214943 shutdown_connections 2024-06-29T15:58:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.557+0000 7f225b21f640 1 -- 172.21.15.81:0/3035214943 wait complete. 2024-06-29T15:58:10.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2024-06-29T15:58:10.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.681+0000 7f46ac865640 1 -- 172.21.15.81:0/3934461488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a806ccb0 msgr2=0x7f46a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.681+0000 7f46ac865640 1 --2- 172.21.15.81:0/3934461488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a806ccb0 0x7f46a806d0c0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f46a80726b0 tx=0x7f469c02a4d0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.681+0000 7f46ac865640 1 -- 172.21.15.81:0/3934461488 shutdown_connections 2024-06-29T15:58:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.681+0000 7f46ac865640 1 --2- 172.21.15.81:0/3934461488 >> v2:172.21.15.167:3300/0 conn(0x7f46a80cf2e0 0x7f46a80cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.681+0000 7f46ac865640 1 --2- 172.21.15.81:0/3934461488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a806ccb0 0x7f46a806d0c0 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.681+0000 7f46ac865640 1 --2- 172.21.15.81:0/3934461488 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a80ad0c0 0x7f46a815cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.681+0000 7f46ac865640 1 -- 172.21.15.81:0/3934461488 >> 172.21.15.81:0/3934461488 conn(0x7f46a8072950 msgr2=0x7f46a80b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.681+0000 7f46ac865640 1 -- 172.21.15.81:0/3934461488 shutdown_connections 2024-06-29T15:58:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.681+0000 7f46ac865640 1 -- 172.21.15.81:0/3934461488 wait complete. 2024-06-29T15:58:10.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.683+0000 7f46ac865640 1 Processor -- start 2024-06-29T15:58:10.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.683+0000 7f46ac865640 1 -- start start 2024-06-29T15:58:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46ac865640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a806ccb0 0x7f46a81501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46ac865640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a80ad0c0 0x7f46a809b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46ac865640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a80cf2e0 0x7f46a81527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46ac865640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46a80aed00 con 0x7f46a806ccb0 2024-06-29T15:58:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46ac865640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f46a80aeb80 con 0x7f46a80ad0c0 2024-06-29T15:58:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46ac865640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f46a80aee80 con 0x7f46a80cf2e0 2024-06-29T15:58:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46a659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a806ccb0 0x7f46a81501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46a659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a806ccb0 0x7f46a81501f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34564/0 (socket says 172.21.15.81:34564) 2024-06-29T15:58:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46a659c640 1 -- 172.21.15.81:0/32110474 learned_addr learned my addr 172.21.15.81:0/32110474 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46a5d9b640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a80ad0c0 0x7f46a809b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46a6d9d640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a80cf2e0 0x7f46a81527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46a659c640 1 -- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a80cf2e0 msgr2=0x7f46a81527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46a659c640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a80cf2e0 0x7f46a81527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46a659c640 1 -- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a80ad0c0 msgr2=0x7f46a809b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46a659c640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a80ad0c0 0x7f46a809b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.684+0000 7f46a659c640 1 -- 172.21.15.81:0/32110474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a809b650 con 0x7f46a806ccb0 2024-06-29T15:58:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.685+0000 7f46a659c640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a806ccb0 0x7f46a81501f0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f469800d660 tx=0x7f469800db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.685+0000 7f4696ffd640 1 -- 172.21.15.81:0/32110474 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698017070 con 0x7f46a806ccb0 2024-06-29T15:58:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.685+0000 7f46ac865640 1 -- 172.21.15.81:0/32110474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a81547c0 con 0x7f46a806ccb0 2024-06-29T15:58:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.685+0000 7f4696ffd640 1 -- 172.21.15.81:0/32110474 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4698005310 con 0x7f46a806ccb0 2024-06-29T15:58:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.685+0000 7f4696ffd640 1 -- 172.21.15.81:0/32110474 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698009820 con 0x7f46a806ccb0 2024-06-29T15:58:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.685+0000 7f46ac865640 1 -- 172.21.15.81:0/32110474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a8151fe0 con 0x7f46a806ccb0 2024-06-29T15:58:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.686+0000 7f46ac865640 1 -- 172.21.15.81:0/32110474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a80af410 con 0x7f46a806ccb0 2024-06-29T15:58:10.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.687+0000 7f4696ffd640 1 -- 172.21.15.81:0/32110474 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f46980099c0 con 0x7f46a806ccb0 2024-06-29T15:58:10.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.687+0000 7f4696ffd640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4688076540 0x7f4688078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:10.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.687+0000 7f4696ffd640 1 -- 172.21.15.81:0/32110474 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5974+0+0 (secure 0 0 0) 0x7f4698097de0 con 0x7f46a806ccb0 2024-06-29T15:58:10.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.687+0000 7f46a5d9b640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4688076540 0x7f4688078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:10.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.688+0000 7f46a5d9b640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4688076540 0x7f4688078a50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f469c033000 tx=0x7f469c004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:10.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.692+0000 7f4696ffd640 1 -- 172.21.15.81:0/32110474 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4698060a60 con 0x7f46a806ccb0 2024-06-29T15:58:10.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:10.955+0000 7f46ac865640 1 -- 172.21.15.81:0/32110474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) -- 0x7f46a80b11c0 con 0x7f46a806ccb0 2024-06-29T15:58:11.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.221+0000 7f4696ffd640 1 -- 172.21.15.81:0/32110474 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3118) ==== 186+0+0 (secure 0 0 0) 0x7f4698069130 con 0x7f46a806ccb0 2024-06-29T15:58:11.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 -- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4688076540 msgr2=0x7f4688078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4688076540 0x7f4688078a50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f469c033000 tx=0x7f469c004730 comp rx=0 tx=0).stop 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 -- 172.21.15.81:0/32110474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a806ccb0 msgr2=0x7f46a81501f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a806ccb0 0x7f46a81501f0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f469800d660 tx=0x7f469800db20 comp rx=0 tx=0).stop 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 -- 172.21.15.81:0/32110474 shutdown_connections 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4688076540 0x7f4688078a50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f46a80cf2e0 0x7f46a81527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f46a80ad0c0 0x7f46a809b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 --2- 172.21.15.81:0/32110474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a806ccb0 0x7f46a81501f0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 -- 172.21.15.81:0/32110474 >> 172.21.15.81:0/32110474 conn(0x7f46a8072950 msgr2=0x7f46a8155030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 -- 172.21.15.81:0/32110474 shutdown_connections 2024-06-29T15:58:11.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.223+0000 7f46ac865640 1 -- 172.21.15.81:0/32110474 wait complete. 2024-06-29T15:58:11.235 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2024-06-29T15:58:11.235 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2024-06-29T15:58:11.235 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2024-06-29T15:58:11.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:11.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.349+0000 7f98afccc640 1 -- 172.21.15.81:0/914529819 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a80b4cf0 msgr2=0x7f98a80d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.349+0000 7f98afccc640 1 --2- 172.21.15.81:0/914529819 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a80b4cf0 0x7f98a80d00f0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f98a0005fd0 tx=0x7f98a002a530 comp rx=0 tx=0).stop 2024-06-29T15:58:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.349+0000 7f98afccc640 1 -- 172.21.15.81:0/914529819 shutdown_connections 2024-06-29T15:58:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.349+0000 7f98afccc640 1 --2- 172.21.15.81:0/914529819 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98a80cd540 0x7f98a80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.349+0000 7f98afccc640 1 --2- 172.21.15.81:0/914529819 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a80b4cf0 0x7f98a80d00f0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.349+0000 7f98afccc640 1 -- 172.21.15.81:0/914529819 >> 172.21.15.81:0/914529819 conn(0x7f98a80d1960 msgr2=0x7f98a80d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.349+0000 7f98afccc640 1 -- 172.21.15.81:0/914529819 shutdown_connections 2024-06-29T15:58:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.349+0000 7f98afccc640 1 -- 172.21.15.81:0/914529819 wait complete. 2024-06-29T15:58:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.350+0000 7f98afccc640 1 Processor -- start 2024-06-29T15:58:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.350+0000 7f98afccc640 1 -- start start 2024-06-29T15:58:11.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98afccc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98a80b4cf0 0x7f98a80af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98afccc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a80b6d60 0x7f98a815c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98afccc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98a80cd540 0x7f98a80b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98afccc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f98a80d07b0 con 0x7f98a80b6d60 2024-06-29T15:58:11.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98afccc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f98a80d0630 con 0x7f98a80b4cf0 2024-06-29T15:58:11.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98afccc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f98a80d0930 con 0x7f98a80cd540 2024-06-29T15:58:11.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ad267640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a80b6d60 0x7f98a815c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ad267640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a80b6d60 0x7f98a815c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34584/0 (socket says 172.21.15.81:34584) 2024-06-29T15:58:11.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ad267640 1 -- 172.21.15.81:0/3740943408 learned_addr learned my addr 172.21.15.81:0/3740943408 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:11.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ada68640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98a80b4cf0 0x7f98a80af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ad267640 1 -- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98a80cd540 msgr2=0x7f98a80b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ae269640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98a80cd540 0x7f98a80b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ad267640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98a80cd540 0x7f98a80b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ad267640 1 -- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98a80b4cf0 msgr2=0x7f98a80af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ad267640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98a80b4cf0 0x7f98a80af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ad267640 1 -- 172.21.15.81:0/3740943408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a80af870 con 0x7f98a80b6d60 2024-06-29T15:58:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98ad267640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a80b6d60 0x7f98a815c5c0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f989c00b410 tx=0x7f989c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.352+0000 7f98a67fc640 1 -- 172.21.15.81:0/3740943408 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989c015020 con 0x7f98a80b6d60 2024-06-29T15:58:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.353+0000 7f98a67fc640 1 -- 172.21.15.81:0/3740943408 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f989c011940 con 0x7f98a80b6d60 2024-06-29T15:58:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.353+0000 7f98a67fc640 1 -- 172.21.15.81:0/3740943408 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989c011d90 con 0x7f98a80b6d60 2024-06-29T15:58:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.353+0000 7f98afccc640 1 -- 172.21.15.81:0/3740943408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98a815cb00 con 0x7f98a80b6d60 2024-06-29T15:58:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.353+0000 7f98afccc640 1 -- 172.21.15.81:0/3740943408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98a80b1480 con 0x7f98a80b6d60 2024-06-29T15:58:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.354+0000 7f98afccc640 1 -- 172.21.15.81:0/3740943408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98a80d0450 con 0x7f98a80b6d60 2024-06-29T15:58:11.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.354+0000 7f98a67fc640 1 -- 172.21.15.81:0/3740943408 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f989c018020 con 0x7f98a80b6d60 2024-06-29T15:58:11.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.355+0000 7f98a67fc640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9894076510 0x7f9894078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.355+0000 7f98a67fc640 1 -- 172.21.15.81:0/3740943408 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5974+0+0 (secure 0 0 0) 0x7f989c097c90 con 0x7f98a80b6d60 2024-06-29T15:58:11.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.355+0000 7f98ada68640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9894076510 0x7f9894078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.356+0000 7f98ada68640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9894076510 0x7f9894078a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f98a80c5220 tx=0x7f98980072d0 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:11.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.359+0000 7f98a67fc640 1 -- 172.21.15.81:0/3740943408 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f98a80d0450 con 0x7f98a80b6d60 2024-06-29T15:58:11.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.541+0000 7f98afccc640 1 -- 172.21.15.81:0/3740943408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98a80d00f0 con 0x7f98a80b6d60 2024-06-29T15:58:11.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.542+0000 7f98a67fc640 1 -- 172.21.15.81:0/3740943408 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) ==== 74+0+14558 (secure 0 0 0) 0x7f989c01f020 con 0x7f98a80b6d60 2024-06-29T15:58:11.542 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:11.542 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":225,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:10.215833+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6833/739691411":"2024-06-30T15:58:00.835804+0000","172.21.15.167:0/2503226350":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6835/1664603190":"2024-06-30T15:58:07.206941+0000","172.21.15.167:0/2924173136":"2024-06-30T15:57:57.317102+0000","172.21.15.81:6832/739691411":"2024-06-30T15:58:00.835804+0000","172.21.15.167:6833/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6832/2452570162":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3022587441":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6837/1664603190":"2024-06-30T15:58:07.206941+0000","172.21.15.167:0/575713984":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/1101183266":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3261132296":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3042354413":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1619575755":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/760717235":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/3781943725":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/376674682":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/1164120698":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3640842563":"2024-06-30T15:57:48.595362+0000","172.21.15.167:0/2165320595":"2024-06-30T15:57:57.317102+0000","172.21.15.167:0/3538782057":"2024-06-30T15:57:57.317102+0000","172.21.15.167:6833/2018755589":"2024-06-30T15:57:48.595362+0000","172.21.15.167:6832/2018755589":"2024-06-30T15:57:48.595362+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:11.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 -- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9894076510 msgr2=0x7f9894078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9894076510 0x7f9894078a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f98a80c5220 tx=0x7f98980072d0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 -- 172.21.15.81:0/3740943408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a80b6d60 msgr2=0x7f98a815c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a80b6d60 0x7f98a815c5c0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f989c00b410 tx=0x7f989c00b8d0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 -- 172.21.15.81:0/3740943408 shutdown_connections 2024-06-29T15:58:11.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9894076510 0x7f9894078a20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f98a80cd540 0x7f98a80b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a80b6d60 0x7f98a815c5c0 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 --2- 172.21.15.81:0/3740943408 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f98a80b4cf0 0x7f98a80af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 -- 172.21.15.81:0/3740943408 >> 172.21.15.81:0/3740943408 conn(0x7f98a80d1960 msgr2=0x7f98a80ce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:11.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 -- 172.21.15.81:0/3740943408 shutdown_connections 2024-06-29T15:58:11.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.544+0000 7f98afccc640 1 -- 172.21.15.81:0/3740943408 wait complete. 2024-06-29T15:58:11.556 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:11.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:11.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.672+0000 7f64d2b8c640 1 -- 172.21.15.81:0/3477210191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64cc0b6eb0 msgr2=0x7f64cc0d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.672+0000 7f64d2b8c640 1 --2- 172.21.15.81:0/3477210191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64cc0b6eb0 0x7f64cc0d02d0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f64cc073340 tx=0x7f64bc02a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.672+0000 7f64d2b8c640 1 -- 172.21.15.81:0/3477210191 shutdown_connections 2024-06-29T15:58:11.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.672+0000 7f64d2b8c640 1 --2- 172.21.15.81:0/3477210191 >> v2:172.21.15.167:3300/0 conn(0x7f64cc0ce920 0x7f64cc0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.672+0000 7f64d2b8c640 1 --2- 172.21.15.81:0/3477210191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64cc0b6eb0 0x7f64cc0d02d0 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.672+0000 7f64d2b8c640 1 -- 172.21.15.81:0/3477210191 >> 172.21.15.81:0/3477210191 conn(0x7f64cc0b3510 msgr2=0x7f64cc0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:11.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.672+0000 7f64d2b8c640 1 -- 172.21.15.81:0/3477210191 shutdown_connections 2024-06-29T15:58:11.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.672+0000 7f64d2b8c640 1 -- 172.21.15.81:0/3477210191 wait complete. 2024-06-29T15:58:11.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.674+0000 7f64d2b8c640 1 Processor -- start 2024-06-29T15:58:11.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.674+0000 7f64d2b8c640 1 -- start start 2024-06-29T15:58:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64d2b8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64cc0b6eb0 0x7f64cc0cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64d2b8c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64cc0ce510 0x7f64cc0c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64d2b8c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64cc0ce920 0x7f64cc1d4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64d2b8c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc0b4f20 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64d2b8c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc0b29c0 con 0x7f64cc0ce510 2024-06-29T15:58:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64d2b8c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f64cc0b50a0 con 0x7f64cc0ce920 2024-06-29T15:58:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64d0928640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64cc0b6eb0 0x7f64cc0cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64d0928640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64cc0b6eb0 0x7f64cc0cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34604/0 (socket says 172.21.15.81:34604) 2024-06-29T15:58:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64d0928640 1 -- 172.21.15.81:0/904457350 learned_addr learned my addr 172.21.15.81:0/904457350 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64d1129640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64cc0ce920 0x7f64cc1d4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.675+0000 7f64cbfff640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64cc0ce510 0x7f64cc0c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64d0928640 1 -- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64cc0ce920 msgr2=0x7f64cc1d4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64d0928640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64cc0ce920 0x7f64cc1d4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64d0928640 1 -- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64cc0ce510 msgr2=0x7f64cc0c4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64d0928640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64cc0ce510 0x7f64cc0c4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64d0928640 1 -- 172.21.15.81:0/904457350 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64cc0c5430 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64d1129640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64cc0ce920 0x7f64cc1d4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64d0928640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64cc0b6eb0 0x7f64cc0cb670 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f64bc034040 tx=0x7f64bc003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64c97fa640 1 -- 172.21.15.81:0/904457350 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc03e030 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64c97fa640 1 -- 172.21.15.81:0/904457350 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64bc003e00 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64c97fa640 1 -- 172.21.15.81:0/904457350 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc035040 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64d2b8c640 1 -- 172.21.15.81:0/904457350 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64cc1e6340 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.676+0000 7f64d2b8c640 1 -- 172.21.15.81:0/904457350 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64cc1e6900 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.677+0000 7f64d2b8c640 1 -- 172.21.15.81:0/904457350 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64cc0d02d0 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.678+0000 7f64c97fa640 1 -- 172.21.15.81:0/904457350 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f64bc041030 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.678+0000 7f64c97fa640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64b8076510 0x7f64b8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.678+0000 7f64c97fa640 1 -- 172.21.15.81:0/904457350 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5974+0+0 (secure 0 0 0) 0x7f64bc0b84e0 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.678+0000 7f64cbfff640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64b8076510 0x7f64b8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.679+0000 7f64cbfff640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64b8076510 0x7f64b8078a20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f64c0006fd0 tx=0x7f64c0008040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:11.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.682+0000 7f64c97fa640 1 -- 172.21.15.81:0/904457350 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f64bc081900 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.860+0000 7f64d2b8c640 1 -- 172.21.15.81:0/904457350 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f64cc0cdf00 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.860+0000 7f64c97fa640 1 -- 172.21.15.81:0/904457350 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v225) ==== 93+0+10 (secure 0 0 0) 0x7f64bc081200 con 0x7f64cc0b6eb0 2024-06-29T15:58:11.861 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 -- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64b8076510 msgr2=0x7f64b8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64b8076510 0x7f64b8078a20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f64c0006fd0 tx=0x7f64c0008040 comp rx=0 tx=0).stop 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 -- 172.21.15.81:0/904457350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64cc0b6eb0 msgr2=0x7f64cc0cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64cc0b6eb0 0x7f64cc0cb670 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f64bc034040 tx=0x7f64bc003910 comp rx=0 tx=0).stop 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 -- 172.21.15.81:0/904457350 shutdown_connections 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f64b8076510 0x7f64b8078a20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64cc0ce920 0x7f64cc1d4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64cc0ce510 0x7f64cc0c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 --2- 172.21.15.81:0/904457350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64cc0b6eb0 0x7f64cc0cb670 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 -- 172.21.15.81:0/904457350 >> 172.21.15.81:0/904457350 conn(0x7f64cc0b3510 msgr2=0x7f64cc0b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:11.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 -- 172.21.15.81:0/904457350 shutdown_connections 2024-06-29T15:58:11.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.862+0000 7f64d2b8c640 1 -- 172.21.15.81:0/904457350 wait complete. 2024-06-29T15:58:11.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-06-29T15:58:11.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.988+0000 7f8dbcae7640 1 -- 172.21.15.81:0/2248796063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8da0004000 msgr2=0x7f8da0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.988+0000 7f8dbcae7640 1 --2- 172.21.15.81:0/2248796063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8da0004000 0x7f8da0004800 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f8dac006fc0 tx=0x7f8da0005830 comp rx=0 tx=0).stop 2024-06-29T15:58:11.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.988+0000 7f8db6d9d640 1 -- 172.21.15.81:0/2248796063 reap_dead start 2024-06-29T15:58:11.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.988+0000 7f8dbcae7640 1 -- 172.21.15.81:0/2248796063 shutdown_connections 2024-06-29T15:58:11.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.988+0000 7f8dbcae7640 1 -- 172.21.15.81:0/2248796063 >> 172.21.15.81:0/2248796063 conn(0x7f8db806ccc0 msgr2=0x7f8db80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:11.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.989+0000 7f8dbcae7640 1 -- 172.21.15.81:0/2248796063 shutdown_connections 2024-06-29T15:58:11.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.989+0000 7f8dbcae7640 1 -- 172.21.15.81:0/2248796063 wait complete. 2024-06-29T15:58:11.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.990+0000 7f8dbcae7640 1 Processor -- start 2024-06-29T15:58:11.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.990+0000 7f8dbcae7640 1 -- start start 2024-06-29T15:58:11.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8dbcae7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8da0004000 0x7f8db8140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8dbcae7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8db815d8e0 0x7f8db815dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8dbcae7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db806f070 0x7f8db806f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8dbcae7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8db815eef0 con 0x7f8db806f070 2024-06-29T15:58:11.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8dbcae7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8db815ed70 con 0x7f8db815d8e0 2024-06-29T15:58:11.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8dbcae7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8db815f070 con 0x7f8da0004000 2024-06-29T15:58:11.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db806f070 0x7f8db806f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db806f070 0x7f8db806f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34624/0 (socket says 172.21.15.81:34624) 2024-06-29T15:58:11.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db6d9d640 1 -- 172.21.15.81:0/334068430 learned_addr learned my addr 172.21.15.81:0/334068430 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:11.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db5d9b640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8db815d8e0 0x7f8db815dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db659c640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8da0004000 0x7f8db8140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db6d9d640 1 -- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8da0004000 msgr2=0x7f8db8140170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db6d9d640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8da0004000 0x7f8db8140170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db6d9d640 1 -- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8db815d8e0 msgr2=0x7f8db815dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:11.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db6d9d640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8db815d8e0 0x7f8db815dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:11.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db6d9d640 1 -- 172.21.15.81:0/334068430 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db806f9c0 con 0x7f8db806f070 2024-06-29T15:58:11.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8db6d9d640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db806f070 0x7f8db806f480 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f8db0005fd0 tx=0x7f8db000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:11.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8da6ffd640 1 -- 172.21.15.81:0/334068430 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0004b20 con 0x7f8db806f070 2024-06-29T15:58:11.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.992+0000 7f8da6ffd640 1 -- 172.21.15.81:0/334068430 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8db0004cc0 con 0x7f8db806f070 2024-06-29T15:58:11.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.993+0000 7f8da6ffd640 1 -- 172.21.15.81:0/334068430 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0003bd0 con 0x7f8db806f070 2024-06-29T15:58:11.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.993+0000 7f8dbcae7640 1 -- 172.21.15.81:0/334068430 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db815e530 con 0x7f8db806f070 2024-06-29T15:58:11.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.993+0000 7f8dbcae7640 1 -- 172.21.15.81:0/334068430 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db8152520 con 0x7f8db806f070 2024-06-29T15:58:11.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.994+0000 7f8dbcae7640 1 -- 172.21.15.81:0/334068430 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db815f250 con 0x7f8db806f070 2024-06-29T15:58:11.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.994+0000 7f8da6ffd640 1 -- 172.21.15.81:0/334068430 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f8db0004030 con 0x7f8db806f070 2024-06-29T15:58:11.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.995+0000 7f8da6ffd640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f8d98076540 0x7f8d98078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:11.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.995+0000 7f8da6ffd640 1 -- 172.21.15.81:0/334068430 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5974+0+0 (secure 0 0 0) 0x7f8db0098240 con 0x7f8db806f070 2024-06-29T15:58:11.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.995+0000 7f8db659c640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f8d98076540 0x7f8d98078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:11.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.995+0000 7f8db659c640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f8d98076540 0x7f8d98078a50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8da8000c00 tx=0x7f8da8006b70 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:11.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:11.999+0000 7f8da6ffd640 1 -- 172.21.15.81:0/334068430 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8db806f9c0 con 0x7f8db806f070 2024-06-29T15:58:12.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:12.253+0000 7f8dbcae7640 1 -- 172.21.15.81:0/334068430 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) -- 0x7f8db815e2c0 con 0x7f8db806f070 2024-06-29T15:58:13.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.228+0000 7f8da6ffd640 1 -- 172.21.15.81:0/334068430 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3120) ==== 190+0+0 (secure 0 0 0) 0x7f8db0061620 con 0x7f8db806f070 2024-06-29T15:58:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.231+0000 7f8dbcae7640 1 -- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f8d98076540 msgr2=0x7f8d98078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.231+0000 7f8dbcae7640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f8d98076540 0x7f8d98078a50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8da8000c00 tx=0x7f8da8006b70 comp rx=0 tx=0).stop 2024-06-29T15:58:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.231+0000 7f8dbcae7640 1 -- 172.21.15.81:0/334068430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db806f070 msgr2=0x7f8db806f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.231+0000 7f8dbcae7640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db806f070 0x7f8db806f480 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f8db0005fd0 tx=0x7f8db000f3c0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.231+0000 7f8dbcae7640 1 -- 172.21.15.81:0/334068430 shutdown_connections 2024-06-29T15:58:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.231+0000 7f8dbcae7640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f8d98076540 0x7f8d98078a50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.231+0000 7f8dbcae7640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db806f070 0x7f8db806f480 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.231+0000 7f8dbcae7640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8db815d8e0 0x7f8db815dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.231+0000 7f8dbcae7640 1 --2- 172.21.15.81:0/334068430 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8da0004000 0x7f8db8140170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.231+0000 7f8dbcae7640 1 -- 172.21.15.81:0/334068430 >> 172.21.15.81:0/334068430 conn(0x7f8db806ccc0 msgr2=0x7f8db80afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:13.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.232+0000 7f8dbcae7640 1 -- 172.21.15.81:0/334068430 shutdown_connections 2024-06-29T15:58:13.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.232+0000 7f8dbcae7640 1 -- 172.21.15.81:0/334068430 wait complete. 2024-06-29T15:58:13.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:58:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.364+0000 7f01af695640 1 -- 172.21.15.81:0/3950941883 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01a80d0b90 msgr2=0x7f01a80d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.364+0000 7f01af695640 1 --2- 172.21.15.81:0/3950941883 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01a80d0b90 0x7f01a80d1a80 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f0198009830 tx=0x7f019802a5a0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.364+0000 7f01af695640 1 -- 172.21.15.81:0/3950941883 shutdown_connections 2024-06-29T15:58:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.364+0000 7f01af695640 1 --2- 172.21.15.81:0/3950941883 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01a80d0b90 0x7f01a80d1a80 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.364+0000 7f01af695640 1 -- 172.21.15.81:0/3950941883 >> 172.21.15.81:0/3950941883 conn(0x7f01a809cf30 msgr2=0x7f01a80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.364+0000 7f01af695640 1 -- 172.21.15.81:0/3950941883 shutdown_connections 2024-06-29T15:58:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.364+0000 7f01af695640 1 -- 172.21.15.81:0/3950941883 wait complete. 2024-06-29T15:58:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.366+0000 7f01af695640 1 Processor -- start 2024-06-29T15:58:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.366+0000 7f01af695640 1 -- start start 2024-06-29T15:58:13.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.367+0000 7f01af695640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01a80b5250 0x7f01a8158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.367+0000 7f01af695640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01a80b9040 0x7f01a80a03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.367+0000 7f01af695640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01a80b1db0 0x7f01a80a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.367+0000 7f01af695640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f01a80d26d0 con 0x7f01a80b9040 2024-06-29T15:58:13.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.367+0000 7f01af695640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f01a80b85c0 con 0x7f01a80b1db0 2024-06-29T15:58:13.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.367+0000 7f01af695640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f01a80d2850 con 0x7f01a80b5250 2024-06-29T15:58:13.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01acc30640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01a80b9040 0x7f01a80a03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01acc30640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01a80b9040 0x7f01a80a03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34642/0 (socket says 172.21.15.81:34642) 2024-06-29T15:58:13.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01acc30640 1 -- 172.21.15.81:0/2456692181 learned_addr learned my addr 172.21.15.81:0/2456692181 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01ad431640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01a80b5250 0x7f01a8158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01adc32640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01a80b1db0 0x7f01a80a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01acc30640 1 -- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01a80b5250 msgr2=0x7f01a8158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01acc30640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01a80b5250 0x7f01a8158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01acc30640 1 -- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01a80b1db0 msgr2=0x7f01a80a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01acc30640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01a80b1db0 0x7f01a80a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01acc30640 1 -- 172.21.15.81:0/2456692181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a80b35c0 con 0x7f01a80b9040 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01acc30640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01a80b9040 0x7f01a80a03b0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0198009950 tx=0x7f0198033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f019dffb640 1 -- 172.21.15.81:0/2456692181 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198031d60 con 0x7f01a80b9040 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f019dffb640 1 -- 172.21.15.81:0/2456692181 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0198033e10 con 0x7f01a80b9040 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f01af695640 1 -- 172.21.15.81:0/2456692181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a8159280 con 0x7f01a80b9040 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.368+0000 7f019dffb640 1 -- 172.21.15.81:0/2456692181 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198005260 con 0x7f01a80b9040 2024-06-29T15:58:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.369+0000 7f01af695640 1 -- 172.21.15.81:0/2456692181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a80b2800 con 0x7f01a80b9040 2024-06-29T15:58:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.369+0000 7f01af695640 1 -- 172.21.15.81:0/2456692181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a80d2de0 con 0x7f01a80b9040 2024-06-29T15:58:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.370+0000 7f019dffb640 1 -- 172.21.15.81:0/2456692181 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f0198005400 con 0x7f01a80b9040 2024-06-29T15:58:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.371+0000 7f019dffb640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f018c076510 0x7f018c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.371+0000 7f019dffb640 1 -- 172.21.15.81:0/2456692181 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5974+0+0 (secure 0 0 0) 0x7f01980b8f10 con 0x7f01a80b9040 2024-06-29T15:58:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.371+0000 7f01ad431640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f018c076510 0x7f018c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.371+0000 7f01ad431640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f018c076510 0x7f018c078a20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f01a400b450 tx=0x7f01a400d040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:13.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.375+0000 7f019dffb640 1 -- 172.21.15.81:0/2456692181 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f019803f030 con 0x7f01a80b9040 2024-06-29T15:58:13.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.554+0000 7f01af695640 1 -- 172.21.15.81:0/2456692181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f01a80689a0 con 0x7f01a80b9040 2024-06-29T15:58:13.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.555+0000 7f019dffb640 1 -- 172.21.15.81:0/2456692181 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v75) ==== 59+0+1998 (secure 0 0 0) 0x7f019808a3c0 con 0x7f01a80b9040 2024-06-29T15:58:13.555 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:58:13.556 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:58:13.556 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:58:13.556 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:58:13.556 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T15:58:13.556 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T15:58:13.556 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T15:58:13.556 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T15:58:13.556 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T15:58:13.556 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T15:58:13.556 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T15:58:13.557 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T15:58:13.558 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:58:13.558 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 -- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f018c076510 msgr2=0x7f018c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f018c076510 0x7f018c078a20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f01a400b450 tx=0x7f01a400d040 comp rx=0 tx=0).stop 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 -- 172.21.15.81:0/2456692181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01a80b9040 msgr2=0x7f01a80a03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01a80b9040 0x7f01a80a03b0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0198009950 tx=0x7f0198033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 -- 172.21.15.81:0/2456692181 shutdown_connections 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f018c076510 0x7f018c078a20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01a80b1db0 0x7f01a80a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01a80b9040 0x7f01a80a03b0 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 --2- 172.21.15.81:0/2456692181 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01a80b5250 0x7f01a8158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 -- 172.21.15.81:0/2456692181 >> 172.21.15.81:0/2456692181 conn(0x7f01a809cf30 msgr2=0x7f01a80b76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 -- 172.21.15.81:0/2456692181 shutdown_connections 2024-06-29T15:58:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.557+0000 7f01af695640 1 -- 172.21.15.81:0/2456692181 wait complete. 2024-06-29T15:58:13.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:58:13.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.684+0000 7fb847209640 1 -- 172.21.15.81:0/1353947362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb84006f6a0 msgr2=0x7fb84015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.684+0000 7fb847209640 1 --2- 172.21.15.81:0/1353947362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb84006f6a0 0x7fb84015aed0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fb8400743b0 tx=0x7fb83402a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.684+0000 7fb847209640 1 -- 172.21.15.81:0/1353947362 shutdown_connections 2024-06-29T15:58:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.684+0000 7fb847209640 1 --2- 172.21.15.81:0/1353947362 >> v2:172.21.15.167:3300/0 conn(0x7fb8400b3960 0x7fb8400b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.684+0000 7fb847209640 1 --2- 172.21.15.81:0/1353947362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb84006f6a0 0x7fb84015aed0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.684+0000 7fb847209640 1 -- 172.21.15.81:0/1353947362 >> 172.21.15.81:0/1353947362 conn(0x7fb8400d0930 msgr2=0x7fb8400d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.684+0000 7fb847209640 1 -- 172.21.15.81:0/1353947362 shutdown_connections 2024-06-29T15:58:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.684+0000 7fb847209640 1 -- 172.21.15.81:0/1353947362 wait complete. 2024-06-29T15:58:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.685+0000 7fb847209640 1 Processor -- start 2024-06-29T15:58:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.686+0000 7fb847209640 1 -- start start 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb847209640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb84006f6a0 0x7fb8401528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb847209640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8400b3960 0x7fb8401eb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb847209640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8400b57b0 0x7fb8401ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb847209640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8400b4430 con 0x7fb8400b3960 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb847209640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb8400b42b0 con 0x7fb84006f6a0 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb847209640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb8400b45b0 con 0x7fb8400b57b0 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb83ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8400b3960 0x7fb8401eb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb83ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8400b3960 0x7fb8401eb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34664/0 (socket says 172.21.15.81:34664) 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb83ffff640 1 -- 172.21.15.81:0/2567301453 learned_addr learned my addr 172.21.15.81:0/2567301453 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb844fa5640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb84006f6a0 0x7fb8401528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb8457a6640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8400b57b0 0x7fb8401ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb83ffff640 1 -- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8400b57b0 msgr2=0x7fb8401ebb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb83ffff640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8400b57b0 0x7fb8401ebb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb83ffff640 1 -- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb84006f6a0 msgr2=0x7fb8401528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb83ffff640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb84006f6a0 0x7fb8401528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb83ffff640 1 -- 172.21.15.81:0/2567301453 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8401f2820 con 0x7fb8400b3960 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.687+0000 7fb8457a6640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8400b57b0 0x7fb8401ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.688+0000 7fb83ffff640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8400b3960 0x7fb8401eb600 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fb834009b30 tx=0x7fb8340039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.688+0000 7fb83d7fa640 1 -- 172.21.15.81:0/2567301453 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834031990 con 0x7fb8400b3960 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.688+0000 7fb83d7fa640 1 -- 172.21.15.81:0/2567301453 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb834031c20 con 0x7fb8400b3960 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.688+0000 7fb847209640 1 -- 172.21.15.81:0/2567301453 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb840150830 con 0x7fb8400b3960 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.688+0000 7fb83d7fa640 1 -- 172.21.15.81:0/2567301453 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834033330 con 0x7fb8400b3960 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.688+0000 7fb847209640 1 -- 172.21.15.81:0/2567301453 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8400a9660 con 0x7fb8400b3960 2024-06-29T15:58:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.689+0000 7fb847209640 1 -- 172.21.15.81:0/2567301453 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb84006e2a0 con 0x7fb8400b3960 2024-06-29T15:58:13.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.689+0000 7fb83d7fa640 1 -- 172.21.15.81:0/2567301453 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fb8340334d0 con 0x7fb8400b3960 2024-06-29T15:58:13.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.690+0000 7fb83d7fa640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb82c076540 0x7fb82c078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:13.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.690+0000 7fb83d7fa640 1 -- 172.21.15.81:0/2567301453 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5974+0+0 (secure 0 0 0) 0x7fb8340b92f0 con 0x7fb8400b3960 2024-06-29T15:58:13.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.690+0000 7fb844fa5640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb82c076540 0x7fb82c078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:13.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.691+0000 7fb844fa5640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb82c076540 0x7fb82c078a50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb830007fd0 tx=0x7fb830009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.694+0000 7fb83d7fa640 1 -- 172.21.15.81:0/2567301453 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb84006e2a0 con 0x7fb8400b3960 2024-06-29T15:58:13.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.877+0000 7fb847209640 1 -- 172.21.15.81:0/2567301453 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fb8400b4a60 con 0x7fb8400b3960 2024-06-29T15:58:13.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.881+0000 7fb83d7fa640 1 -- 172.21.15.81:0/2567301453 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) ==== 122+0+0 (secure 0 0 0) 0x7fb83403f030 con 0x7fb8400b3960 2024-06-29T15:58:13.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 -- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb82c076540 msgr2=0x7fb82c078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb82c076540 0x7fb82c078a50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb830007fd0 tx=0x7fb830009040 comp rx=0 tx=0).stop 2024-06-29T15:58:13.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 -- 172.21.15.81:0/2567301453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8400b3960 msgr2=0x7fb8401eb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:13.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8400b3960 0x7fb8401eb600 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fb834009b30 tx=0x7fb8340039f0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 -- 172.21.15.81:0/2567301453 shutdown_connections 2024-06-29T15:58:13.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb82c076540 0x7fb82c078a50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb8400b57b0 0x7fb8401ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8400b3960 0x7fb8401eb600 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 --2- 172.21.15.81:0/2567301453 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb84006f6a0 0x7fb8401528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:13.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 -- 172.21.15.81:0/2567301453 >> 172.21.15.81:0/2567301453 conn(0x7fb8400d0930 msgr2=0x7fb8400ce1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:13.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 -- 172.21.15.81:0/2567301453 shutdown_connections 2024-06-29T15:58:13.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:13.883+0000 7fb847209640 1 -- 172.21.15.81:0/2567301453 wait complete. 2024-06-29T15:58:13.894 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:58:13.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:58:13.939 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:58:13.980 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:58:13.981 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:58:13.981 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:13.996 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:58:13.996 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:58:13.996 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:14.038 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:58:14.039 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:58:14.039 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:14.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:58:14.054 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:58:14.054 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:14.096 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:58:14.097 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:58:14.097 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:14.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.210+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3006590985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f580aea70 msgr2=0x7f7f580aee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.210+0000 7f7f5cd70640 1 --2- 172.21.15.81:0/3006590985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f580aea70 0x7f7f580aee80 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f7f58072e90 tx=0x7f7f4c02a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.210+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3006590985 shutdown_connections 2024-06-29T15:58:14.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.210+0000 7f7f5cd70640 1 --2- 172.21.15.81:0/3006590985 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f580af3c0 0x7f7f5806c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.210+0000 7f7f5cd70640 1 --2- 172.21.15.81:0/3006590985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f580aea70 0x7f7f580aee80 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.210+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3006590985 >> 172.21.15.81:0/3006590985 conn(0x7f7f58072950 msgr2=0x7f7f580b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:14.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.210+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3006590985 shutdown_connections 2024-06-29T15:58:14.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.210+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3006590985 wait complete. 2024-06-29T15:58:14.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.212+0000 7f7f5cd70640 1 Processor -- start 2024-06-29T15:58:14.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.212+0000 7f7f5cd70640 1 -- start start 2024-06-29T15:58:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.213+0000 7f7f5cd70640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f580ad0c0 0x7f7f5814d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.213+0000 7f7f5cd70640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f580aea70 0x7f7f5815a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.213+0000 7f7f5cd70640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f580af3c0 0x7f7f5815a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.213+0000 7f7f5cd70640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5806cf80 con 0x7f7f580ad0c0 2024-06-29T15:58:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.213+0000 7f7f5cd70640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5806ce00 con 0x7f7f580af3c0 2024-06-29T15:58:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.213+0000 7f7f5cd70640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7f5806d100 con 0x7f7f580aea70 2024-06-29T15:58:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.213+0000 7f7f5659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f580ad0c0 0x7f7f5814d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.213+0000 7f7f5659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f580ad0c0 0x7f7f5814d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34680/0 (socket says 172.21.15.81:34680) 2024-06-29T15:58:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.213+0000 7f7f5659c640 1 -- 172.21.15.81:0/3474518536 learned_addr learned my addr 172.21.15.81:0/3474518536 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f55d9b640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f580aea70 0x7f7f5815a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f56d9d640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f580af3c0 0x7f7f5815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f5659c640 1 -- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f580aea70 msgr2=0x7f7f5815a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f5659c640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f580aea70 0x7f7f5815a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f5659c640 1 -- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f580af3c0 msgr2=0x7f7f5815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f5659c640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f580af3c0 0x7f7f5815a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f5659c640 1 -- 172.21.15.81:0/3474518536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5809d8f0 con 0x7f7f580ad0c0 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f55d9b640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f580aea70 0x7f7f5815a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f56d9d640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f580af3c0 0x7f7f5815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f5659c640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f580ad0c0 0x7f7f5814d960 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800b410 tx=0x7f7f4800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3474518536 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48015020 con 0x7f7f580ad0c0 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3474518536 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f48011920 con 0x7f7f580ad0c0 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3474518536 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48011d70 con 0x7f7f580ad0c0 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3474518536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f58159950 con 0x7f7f580ad0c0 2024-06-29T15:58:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.214+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3474518536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f5814dea0 con 0x7f7f580ad0c0 2024-06-29T15:58:14.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.215+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3474518536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f5806d650 con 0x7f7f580ad0c0 2024-06-29T15:58:14.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.216+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3474518536 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f7f48018020 con 0x7f7f580ad0c0 2024-06-29T15:58:14.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.216+0000 7f7f46ffd640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7f38076510 0x7f7f38078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.216+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3474518536 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5974+0+0 (secure 0 0 0) 0x7f7f48097d90 con 0x7f7f580ad0c0 2024-06-29T15:58:14.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.216+0000 7f7f55d9b640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7f38076510 0x7f7f38078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.217+0000 7f7f55d9b640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7f38076510 0x7f7f38078a20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c033000 tx=0x7f7f4c004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:14.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.221+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3474518536 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7f4801f020 con 0x7f7f580ad0c0 2024-06-29T15:58:14.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.439+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3474518536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f5815c9e0 con 0x7f7f580ad0c0 2024-06-29T15:58:14.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.439+0000 7f7f46ffd640 1 -- 172.21.15.81:0/3474518536 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+3921 (secure 0 0 0) 0x7f7f480611f0 con 0x7f7f580ad0c0 2024-06-29T15:58:14.440 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:14.440 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":122,"btime":"2024-06-29T15:58:08:209257+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8834,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/789467725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":789467725},{"type":"v1","addr":"172.21.15.167:6836","nonce":789467725}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":9332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/3972149208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":3972149208},{"type":"v1","addr":"172.21.15.167:6837","nonce":3972149208}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122},{"gid":10105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2698501994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2698501994},{"type":"v1","addr":"172.21.15.81:6835","nonce":2698501994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":11907,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/2295234543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2295234543},{"type":"v1","addr":"172.21.15.81:6833","nonce":2295234543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[]} 2024-06-29T15:58:14.440 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 122 2024-06-29T15:58:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7f38076510 msgr2=0x7f7f38078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7f38076510 0x7f7f38078a20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c033000 tx=0x7f7f4c004730 comp rx=0 tx=0).stop 2024-06-29T15:58:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3474518536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f580ad0c0 msgr2=0x7f7f5814d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f580ad0c0 0x7f7f5814d960 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800b410 tx=0x7f7f4800b8d0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3474518536 shutdown_connections 2024-06-29T15:58:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7f38076510 0x7f7f38078a20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f580af3c0 0x7f7f5815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f580aea70 0x7f7f5815a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 --2- 172.21.15.81:0/3474518536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f580ad0c0 0x7f7f5814d960 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3474518536 >> 172.21.15.81:0/3474518536 conn(0x7f7f58072950 msgr2=0x7f7f58156780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:14.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3474518536 shutdown_connections 2024-06-29T15:58:14.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.441+0000 7f7f5cd70640 1 -- 172.21.15.81:0/3474518536 wait complete. 2024-06-29T15:58:14.453 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:58:14.453 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:58:14.453 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:58:14.453 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:58:14.454 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:58:14.454 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:58:14.454 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:58:14.454 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:58:14.454 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:58:14.454 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:58:14.455 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:58:14.455 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:58:14.455 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:58:14.455 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:58:14.455 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:58:14.455 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:58:14.465 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:58:14.465 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:58:14.467 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:58:14.467 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:58:14.469 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:58:14.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:58:14.470 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:58:14.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:58:14.509 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:58:14.510 DEBUG:teuthology.parallel:result is None 2024-06-29T15:58:14.511 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:58:14.511 DEBUG:teuthology.parallel:result is None 2024-06-29T15:58:14.512 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:58:14.512 DEBUG:teuthology.parallel:result is None 2024-06-29T15:58:14.514 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:58:14.514 DEBUG:teuthology.parallel:result is None 2024-06-29T15:58:14.514 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:58:14.596 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T15:58:14.596 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T15:58:14.608 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T15:58:14.608 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:58:14.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.635+0000 7f84cf53c640 1 -- 172.21.15.81:0/418050810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84c80b3200 msgr2=0x7f84c80cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.635+0000 7f84cf53c640 1 --2- 172.21.15.81:0/418050810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84c80b3200 0x7f84c80cc7b0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f84c4005fd0 tx=0x7f84c402a530 comp rx=0 tx=0).stop 2024-06-29T15:58:14.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.635+0000 7f84cf53c640 1 -- 172.21.15.81:0/418050810 shutdown_connections 2024-06-29T15:58:14.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.635+0000 7f84cf53c640 1 --2- 172.21.15.81:0/418050810 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84c80cccf0 0x7f84c80d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.635+0000 7f84cf53c640 1 --2- 172.21.15.81:0/418050810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84c80b3200 0x7f84c80cc7b0 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.635+0000 7f84cf53c640 1 -- 172.21.15.81:0/418050810 >> 172.21.15.81:0/418050810 conn(0x7f84c80b3680 msgr2=0x7f84c80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:14.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.635+0000 7f84cf53c640 1 -- 172.21.15.81:0/418050810 shutdown_connections 2024-06-29T15:58:14.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.635+0000 7f84cf53c640 1 -- 172.21.15.81:0/418050810 wait complete. 2024-06-29T15:58:14.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.637+0000 7f84cf53c640 1 Processor -- start 2024-06-29T15:58:14.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.637+0000 7f84cf53c640 1 -- start start 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.638+0000 7f84cf53c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84c80b3200 0x7f84c8155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.638+0000 7f84cf53c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84c80cccf0 0x7f84c81e4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.638+0000 7f84cf53c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84c80d0770 0x7f84c81e4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.638+0000 7f84cf53c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f84c80cedb0 con 0x7f84c80cccf0 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.638+0000 7f84cf53c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f84c80cec30 con 0x7f84c80b3200 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.638+0000 7f84cf53c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f84c80cef30 con 0x7f84c80d0770 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.638+0000 7f84ccad7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84c80cccf0 0x7f84c81e4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.638+0000 7f84ccad7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84c80cccf0 0x7f84c81e4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34734/0 (socket says 172.21.15.81:34734) 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84ccad7640 1 -- 172.21.15.81:0/3066373427 learned_addr learned my addr 172.21.15.81:0/3066373427 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84ccad7640 1 -- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84c80d0770 msgr2=0x7f84c81e4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84ccad7640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84c80d0770 0x7f84c81e4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84ccad7640 1 -- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84c80b3200 msgr2=0x7f84c8155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:14.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84cd2d8640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84c80b3200 0x7f84c8155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84ccad7640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84c80b3200 0x7f84c8155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:14.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84ccad7640 1 -- 172.21.15.81:0/3066373427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c81e3950 con 0x7f84c80cccf0 2024-06-29T15:58:14.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84cd2d8640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84c80b3200 0x7f84c8155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:14.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84ccad7640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84c80cccf0 0x7f84c81e4a10 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f84b800b440 tx=0x7f84b800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:14.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84bdffb640 1 -- 172.21.15.81:0/3066373427 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b8015020 con 0x7f84c80cccf0 2024-06-29T15:58:14.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84bdffb640 1 -- 172.21.15.81:0/3066373427 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84b8011980 con 0x7f84c80cccf0 2024-06-29T15:58:14.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84cf53c640 1 -- 172.21.15.81:0/3066373427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c81e3c30 con 0x7f84c80cccf0 2024-06-29T15:58:14.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84bdffb640 1 -- 172.21.15.81:0/3066373427 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b8011dd0 con 0x7f84c80cccf0 2024-06-29T15:58:14.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.639+0000 7f84cf53c640 1 -- 172.21.15.81:0/3066373427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c81e60b0 con 0x7f84c80cccf0 2024-06-29T15:58:14.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.640+0000 7f84cf53c640 1 -- 172.21.15.81:0/3066373427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c80cc7b0 con 0x7f84c80cccf0 2024-06-29T15:58:14.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.641+0000 7f84bdffb640 1 -- 172.21.15.81:0/3066373427 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f84b8018020 con 0x7f84c80cccf0 2024-06-29T15:58:14.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.641+0000 7f84bdffb640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f84ac076540 0x7f84ac078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:14.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.641+0000 7f84bdffb640 1 -- 172.21.15.81:0/3066373427 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5974+0+0 (secure 0 0 0) 0x7f84b8097f20 con 0x7f84c80cccf0 2024-06-29T15:58:14.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.642+0000 7f84cd2d8640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f84ac076540 0x7f84ac078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:14.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.642+0000 7f84cd2d8640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f84ac076540 0x7f84ac078a50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f84c4033000 tx=0x7f84c4004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:14.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.646+0000 7f84bdffb640 1 -- 172.21.15.81:0/3066373427 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f84b801f020 con 0x7f84c80cccf0 2024-06-29T15:58:14.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:14.839+0000 7f84cf53c640 1 -- 172.21.15.81:0/3066373427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f84c81e46a0 con 0x7f84c80cccf0 2024-06-29T15:58:15.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.253+0000 7f84bdffb640 1 -- 172.21.15.81:0/3066373427 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v227) ==== 97+0+0 (secure 0 0 0) 0x7f84b800d040 con 0x7f84c80cccf0 2024-06-29T15:58:15.254 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T15:58:15.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 -- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f84ac076540 msgr2=0x7f84ac078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f84ac076540 0x7f84ac078a50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f84c4033000 tx=0x7f84c4004730 comp rx=0 tx=0).stop 2024-06-29T15:58:15.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 -- 172.21.15.81:0/3066373427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84c80cccf0 msgr2=0x7f84c81e4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84c80cccf0 0x7f84c81e4a10 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f84b800b440 tx=0x7f84b800b900 comp rx=0 tx=0).stop 2024-06-29T15:58:15.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 -- 172.21.15.81:0/3066373427 shutdown_connections 2024-06-29T15:58:15.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f84ac076540 0x7f84ac078a50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84c80d0770 0x7f84c81e4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84c80cccf0 0x7f84c81e4a10 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 --2- 172.21.15.81:0/3066373427 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84c80b3200 0x7f84c8155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 -- 172.21.15.81:0/3066373427 >> 172.21.15.81:0/3066373427 conn(0x7f84c80b3680 msgr2=0x7f84c80702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:15.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 -- 172.21.15.81:0/3066373427 shutdown_connections 2024-06-29T15:58:15.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.256+0000 7f84cf53c640 1 -- 172.21.15.81:0/3066373427 wait complete. 2024-06-29T15:58:15.269 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:58:15.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.382+0000 7fab59ec1640 1 -- 172.21.15.81:0/1451514082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab5406f6a0 msgr2=0x7fab5415aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.382+0000 7fab59ec1640 1 --2- 172.21.15.81:0/1451514082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab5406f6a0 0x7fab5415aed0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fab540743b0 tx=0x7fab4802a560 comp rx=0 tx=0).stop 2024-06-29T15:58:15.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.382+0000 7fab59ec1640 1 -- 172.21.15.81:0/1451514082 shutdown_connections 2024-06-29T15:58:15.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.382+0000 7fab59ec1640 1 --2- 172.21.15.81:0/1451514082 >> v2:172.21.15.167:3300/0 conn(0x7fab540b3960 0x7fab540b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.382+0000 7fab59ec1640 1 --2- 172.21.15.81:0/1451514082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab5406f6a0 0x7fab5415aed0 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.382+0000 7fab59ec1640 1 -- 172.21.15.81:0/1451514082 >> 172.21.15.81:0/1451514082 conn(0x7fab540d0930 msgr2=0x7fab540d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:15.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.382+0000 7fab59ec1640 1 -- 172.21.15.81:0/1451514082 shutdown_connections 2024-06-29T15:58:15.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.382+0000 7fab59ec1640 1 -- 172.21.15.81:0/1451514082 wait complete. 2024-06-29T15:58:15.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.384+0000 7fab59ec1640 1 Processor -- start 2024-06-29T15:58:15.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.384+0000 7fab59ec1640 1 -- start start 2024-06-29T15:58:15.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab59ec1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab5406f6a0 0x7fab5406c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab59ec1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab540b3960 0x7fab5406c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab59ec1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab540b57b0 0x7fab541f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab59ec1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fab540b4430 con 0x7fab5406f6a0 2024-06-29T15:58:15.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab59ec1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fab540b42b0 con 0x7fab540b3960 2024-06-29T15:58:15.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab59ec1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fab540b45b0 con 0x7fab540b57b0 2024-06-29T15:58:15.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab537fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab5406f6a0 0x7fab5406c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab537fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab5406f6a0 0x7fab5406c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34740/0 (socket says 172.21.15.81:34740) 2024-06-29T15:58:15.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab537fe640 1 -- 172.21.15.81:0/143035182 learned_addr learned my addr 172.21.15.81:0/143035182 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab52ffd640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab540b3960 0x7fab5406c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab53fff640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab540b57b0 0x7fab541f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab537fe640 1 -- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab540b57b0 msgr2=0x7fab541f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.385+0000 7fab537fe640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab540b57b0 0x7fab541f5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.386+0000 7fab537fe640 1 -- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab540b3960 msgr2=0x7fab5406c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.386+0000 7fab537fe640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab540b3960 0x7fab5406c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.386+0000 7fab537fe640 1 -- 172.21.15.81:0/143035182 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab541e2120 con 0x7fab5406f6a0 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.386+0000 7fab52ffd640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab540b3960 0x7fab5406c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.386+0000 7fab537fe640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab5406f6a0 0x7fab5406c2f0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fab4400b3e0 tx=0x7fab4400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.386+0000 7fab3bfff640 1 -- 172.21.15.81:0/143035182 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab44015020 con 0x7fab5406f6a0 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.386+0000 7fab3bfff640 1 -- 172.21.15.81:0/143035182 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab44011940 con 0x7fab5406f6a0 2024-06-29T15:58:15.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.386+0000 7fab59ec1640 1 -- 172.21.15.81:0/143035182 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab541e2400 con 0x7fab5406f6a0 2024-06-29T15:58:15.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.386+0000 7fab3bfff640 1 -- 172.21.15.81:0/143035182 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab44011d90 con 0x7fab5406f6a0 2024-06-29T15:58:15.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.386+0000 7fab59ec1640 1 -- 172.21.15.81:0/143035182 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab541e29c0 con 0x7fab5406f6a0 2024-06-29T15:58:15.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.387+0000 7fab59ec1640 1 -- 172.21.15.81:0/143035182 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab5406e2a0 con 0x7fab5406f6a0 2024-06-29T15:58:15.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.388+0000 7fab3bfff640 1 -- 172.21.15.81:0/143035182 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fab44018020 con 0x7fab5406f6a0 2024-06-29T15:58:15.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.388+0000 7fab3bfff640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab40076510 0x7fab40078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.388+0000 7fab3bfff640 1 -- 172.21.15.81:0/143035182 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 4985+0+0 (secure 0 0 0) 0x7fab44097880 con 0x7fab5406f6a0 2024-06-29T15:58:15.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.388+0000 7fab52ffd640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab40076510 0x7fab40078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.389+0000 7fab52ffd640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab40076510 0x7fab40078a20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fab48033000 tx=0x7fab48004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:15.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.393+0000 7fab3bfff640 1 -- 172.21.15.81:0/143035182 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fab5406e2a0 con 0x7fab5406f6a0 2024-06-29T15:58:15.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.644+0000 7fab59ec1640 1 -- 172.21.15.81:0/143035182 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fab540b4a80 con 0x7fab5406f6a0 2024-06-29T15:58:15.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.645+0000 7fab3bfff640 1 -- 172.21.15.81:0/143035182 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v67) ==== 80+0+5304 (secure 0 0 0) 0x7fab4401f020 con 0x7fab5406f6a0 2024-06-29T15:58:15.645 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:15.645 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:58:15.645 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.646 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T15:58:15.647 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.648 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:58:15.648 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:58:15.648 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:58:15.648 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.648 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.648 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.648 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T15:58:15.648 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T15:58:15.648 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.649 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.650 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.651 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.652 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T15:58:15.653 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.654 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:15.654 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:58:15.654 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.654 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.654 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.654 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T15:58:15.654 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T15:58:15.654 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.654 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:15.657 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:58:15.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.657 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T15:58:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:58:15.659 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:58:15.660 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:58:15.660 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.660 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.660 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T15:58:15.660 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T15:58:15.660 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.660 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:58:15.660 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:58:15.661 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:58:15.662 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.662 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.662 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.662 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T15:58:15.662 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T15:58:15.662 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.662 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:58:15.662 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:15.663 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:15.664 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T15:58:15.664 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T15:58:15.664 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:58:15.664 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:58:15.664 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:58:15.664 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:58:15.664 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.664 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:15.664 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T15:58:15.664 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 -- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab40076510 msgr2=0x7fab40078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab40076510 0x7fab40078a20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fab48033000 tx=0x7fab48004730 comp rx=0 tx=0).stop 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 -- 172.21.15.81:0/143035182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab5406f6a0 msgr2=0x7fab5406c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab5406f6a0 0x7fab5406c2f0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fab4400b3e0 tx=0x7fab4400b8a0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 -- 172.21.15.81:0/143035182 shutdown_connections 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab40076510 0x7fab40078a20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab540b57b0 0x7fab541f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab540b3960 0x7fab5406c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 --2- 172.21.15.81:0/143035182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab5406f6a0 0x7fab5406c2f0 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 -- 172.21.15.81:0/143035182 >> 172.21.15.81:0/143035182 conn(0x7fab540d0930 msgr2=0x7fab541583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:15.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 -- 172.21.15.81:0/143035182 shutdown_connections 2024-06-29T15:58:15.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.647+0000 7fab59ec1640 1 -- 172.21.15.81:0/143035182 wait complete. 2024-06-29T15:58:15.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:15.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.781+0000 7f87b692b640 1 -- 172.21.15.81:0/3693150490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8798004800 msgr2=0x7f8798004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.781+0000 7f87b692b640 1 --2- 172.21.15.81:0/3693150490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8798004800 0x7f8798004c10 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f87a4001050 tx=0x7f8798005830 comp rx=0 tx=0).stop 2024-06-29T15:58:15.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.781+0000 7f87b692b640 1 -- 172.21.15.81:0/3693150490 shutdown_connections 2024-06-29T15:58:15.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.781+0000 7f87b692b640 1 --2- 172.21.15.81:0/3693150490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8798004800 0x7f8798004c10 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.781+0000 7f87b692b640 1 -- 172.21.15.81:0/3693150490 >> 172.21.15.81:0/3693150490 conn(0x7f87b009cf30 msgr2=0x7f87b00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:15.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.781+0000 7f87b692b640 1 -- 172.21.15.81:0/3693150490 shutdown_connections 2024-06-29T15:58:15.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.781+0000 7f87b692b640 1 -- 172.21.15.81:0/3693150490 wait complete. 2024-06-29T15:58:15.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.783+0000 7f87b692b640 1 Processor -- start 2024-06-29T15:58:15.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.783+0000 7f87b692b640 1 -- start start 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87b692b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8798004800 0x7f87b00b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87b692b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87b00b3540 0x7f87b00b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87b692b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87b00b3e90 0x7f87b01f4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87b692b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87b00d0810 con 0x7f87b00b3e90 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87b692b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f87b00d0690 con 0x7f8798004800 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87b692b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f87b00d0990 con 0x7f87b00b3540 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87b4ec8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87b00b3e90 0x7f87b01f4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87b4ec8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87b00b3e90 0x7f87b01f4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34768/0 (socket says 172.21.15.81:34768) 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87b4ec8640 1 -- 172.21.15.81:0/3483127390 learned_addr learned my addr 172.21.15.81:0/3483127390 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87affff640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8798004800 0x7f87b00b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.784+0000 7f87af7fe640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87b00b3540 0x7f87b00b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87b4ec8640 1 -- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87b00b3540 msgr2=0x7f87b00b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87b4ec8640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87b00b3540 0x7f87b00b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87b4ec8640 1 -- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8798004800 msgr2=0x7f87b00b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87b4ec8640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8798004800 0x7f87b00b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87b4ec8640 1 -- 172.21.15.81:0/3483127390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b009ea40 con 0x7f87b00b3e90 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87af7fe640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87b00b3540 0x7f87b00b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87b4ec8640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87b00b3e90 0x7f87b01f4820 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f87b00d43d0 tx=0x7f87a8001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87acff9640 1 -- 172.21.15.81:0/3483127390 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8011870 con 0x7f87b00b3e90 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87acff9640 1 -- 172.21.15.81:0/3483127390 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87a8017070 con 0x7f87b00b3e90 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87b692b640 1 -- 172.21.15.81:0/3483127390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b009ed20 con 0x7f87b00b3e90 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87acff9640 1 -- 172.21.15.81:0/3483127390 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8011dc0 con 0x7f87b00b3e90 2024-06-29T15:58:15.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.785+0000 7f87b692b640 1 -- 172.21.15.81:0/3483127390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b009f2e0 con 0x7f87b00b3e90 2024-06-29T15:58:15.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.786+0000 7f87b692b640 1 -- 172.21.15.81:0/3483127390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87b00d1a80 con 0x7f87b00b3e90 2024-06-29T15:58:15.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.787+0000 7f87acff9640 1 -- 172.21.15.81:0/3483127390 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f87a801c460 con 0x7f87b00b3e90 2024-06-29T15:58:15.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.787+0000 7f87acff9640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f879c076540 0x7f879c078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:15.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.787+0000 7f87acff9640 1 -- 172.21.15.81:0/3483127390 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 4985+0+0 (secure 0 0 0) 0x7f87a8098210 con 0x7f87b00b3e90 2024-06-29T15:58:15.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.788+0000 7f87affff640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f879c076540 0x7f879c078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:15.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.788+0000 7f87affff640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f879c076540 0x7f879c078a50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87b006adf0 tx=0x7f87a0005f90 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:15.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.792+0000 7f87acff9640 1 -- 172.21.15.81:0/3483127390 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f87a80618c0 con 0x7f87b00b3e90 2024-06-29T15:58:15.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.971+0000 7f87b692b640 1 -- 172.21.15.81:0/3483127390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f87b009e7d0 con 0x7f87b00b3e90 2024-06-29T15:58:15.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.972+0000 7f87acff9640 1 -- 172.21.15.81:0/3483127390 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) ==== 74+0+13093 (secure 0 0 0) 0x7f87a80611c0 con 0x7f87b00b3e90 2024-06-29T15:58:15.973 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:15.973 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":229,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:15.265628+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:15.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.973+0000 7f87b692b640 1 -- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f879c076540 msgr2=0x7f879c078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.973+0000 7f87b692b640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f879c076540 0x7f879c078a50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87b006adf0 tx=0x7f87a0005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:15.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.973+0000 7f87b692b640 1 -- 172.21.15.81:0/3483127390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87b00b3e90 msgr2=0x7f87b01f4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:15.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.973+0000 7f87b692b640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87b00b3e90 0x7f87b01f4820 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f87b00d43d0 tx=0x7f87a8001a70 comp rx=0 tx=0).stop 2024-06-29T15:58:15.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.974+0000 7f87b692b640 1 -- 172.21.15.81:0/3483127390 shutdown_connections 2024-06-29T15:58:15.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.974+0000 7f87b692b640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f879c076540 0x7f879c078a50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.974+0000 7f87b692b640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87b00b3e90 0x7f87b01f4820 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.974+0000 7f87b692b640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f87b00b3540 0x7f87b00b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.974+0000 7f87b692b640 1 --2- 172.21.15.81:0/3483127390 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8798004800 0x7f87b00b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:15.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.974+0000 7f87b692b640 1 -- 172.21.15.81:0/3483127390 >> 172.21.15.81:0/3483127390 conn(0x7f87b009cf30 msgr2=0x7f87b015e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:15.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.974+0000 7f87b692b640 1 -- 172.21.15.81:0/3483127390 shutdown_connections 2024-06-29T15:58:15.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:15.974+0000 7f87b692b640 1 -- 172.21.15.81:0/3483127390 wait complete. 2024-06-29T15:58:15.986 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:15.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.101+0000 7f54ddac0640 1 -- 172.21.15.81:0/1759404037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54d8073990 msgr2=0x7f54d815c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.101+0000 7f54ddac0640 1 --2- 172.21.15.81:0/1759404037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54d8073990 0x7f54d815c5f0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f54cc005fd0 tx=0x7f54cc02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.101+0000 7f54ddac0640 1 -- 172.21.15.81:0/1759404037 shutdown_connections 2024-06-29T15:58:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.101+0000 7f54ddac0640 1 --2- 172.21.15.81:0/1759404037 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54d815cb30 0x7f54d815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.101+0000 7f54ddac0640 1 --2- 172.21.15.81:0/1759404037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54d8073990 0x7f54d815c5f0 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.101+0000 7f54ddac0640 1 -- 172.21.15.81:0/1759404037 >> 172.21.15.81:0/1759404037 conn(0x7f54d806ccc0 msgr2=0x7f54d80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:16.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.101+0000 7f54ddac0640 1 -- 172.21.15.81:0/1759404037 shutdown_connections 2024-06-29T15:58:16.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.102+0000 7f54ddac0640 1 -- 172.21.15.81:0/1759404037 wait complete. 2024-06-29T15:58:16.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.103+0000 7f54ddac0640 1 Processor -- start 2024-06-29T15:58:16.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.104+0000 7f54ddac0640 1 -- start start 2024-06-29T15:58:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54ddac0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54d8145530 0x7f54d80b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54ddac0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54d815cb30 0x7f54d80afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54ddac0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54d80af140 0x7f54d815f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54ddac0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f54d815dde0 con 0x7f54d80af140 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54ddac0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f54d815dc60 con 0x7f54d815cb30 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54ddac0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f54d815df60 con 0x7f54d8145530 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54d7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54d80af140 0x7f54d815f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54d7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54d80af140 0x7f54d815f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34782/0 (socket says 172.21.15.81:34782) 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54d7fff640 1 -- 172.21.15.81:0/1487291220 learned_addr learned my addr 172.21.15.81:0/1487291220 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54d6ffd640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54d815cb30 0x7f54d80afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54d77fe640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54d8145530 0x7f54d80b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54d7fff640 1 -- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54d8145530 msgr2=0x7f54d80b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54d7fff640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54d8145530 0x7f54d80b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54d7fff640 1 -- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54d815cb30 msgr2=0x7f54d80afc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54d7fff640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54d815cb30 0x7f54d80afc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.105+0000 7f54d7fff640 1 -- 172.21.15.81:0/1487291220 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d80b0180 con 0x7f54d80af140 2024-06-29T15:58:16.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.106+0000 7f54d7fff640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54d80af140 0x7f54d815f6a0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f54d000d660 tx=0x7f54d000db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:16.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.106+0000 7f54bffff640 1 -- 172.21.15.81:0/1487291220 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0017070 con 0x7f54d80af140 2024-06-29T15:58:16.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.106+0000 7f54bffff640 1 -- 172.21.15.81:0/1487291220 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54d0005310 con 0x7f54d80af140 2024-06-29T15:58:16.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.106+0000 7f54bffff640 1 -- 172.21.15.81:0/1487291220 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0009820 con 0x7f54d80af140 2024-06-29T15:58:16.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.106+0000 7f54ddac0640 1 -- 172.21.15.81:0/1487291220 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d80adc00 con 0x7f54d80af140 2024-06-29T15:58:16.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.106+0000 7f54ddac0640 1 -- 172.21.15.81:0/1487291220 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d80ae1c0 con 0x7f54d80af140 2024-06-29T15:58:16.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.107+0000 7f54ddac0640 1 -- 172.21.15.81:0/1487291220 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54d815b510 con 0x7f54d80af140 2024-06-29T15:58:16.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.107+0000 7f54bffff640 1 -- 172.21.15.81:0/1487291220 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f54d00099c0 con 0x7f54d80af140 2024-06-29T15:58:16.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.108+0000 7f54bffff640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f54c4076510 0x7f54c4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.108+0000 7f54bffff640 1 -- 172.21.15.81:0/1487291220 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 4985+0+0 (secure 0 0 0) 0x7f54d0097960 con 0x7f54d80af140 2024-06-29T15:58:16.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.108+0000 7f54d77fe640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f54c4076510 0x7f54c4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.109+0000 7f54d77fe640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f54c4076510 0x7f54c4078a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f54c8001e20 tx=0x7f54c8009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:16.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.113+0000 7f54bffff640 1 -- 172.21.15.81:0/1487291220 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f54d815b510 con 0x7f54d80af140 2024-06-29T15:58:16.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.296+0000 7f54ddac0640 1 -- 172.21.15.81:0/1487291220 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f54d815e480 con 0x7f54d80af140 2024-06-29T15:58:16.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.296+0000 7f54bffff640 1 -- 172.21.15.81:0/1487291220 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v229) ==== 93+0+10 (secure 0 0 0) 0x7f54d0069110 con 0x7f54d80af140 2024-06-29T15:58:16.297 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:58:16.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 -- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f54c4076510 msgr2=0x7f54c4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f54c4076510 0x7f54c4078a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f54c8001e20 tx=0x7f54c8009040 comp rx=0 tx=0).stop 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 -- 172.21.15.81:0/1487291220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54d80af140 msgr2=0x7f54d815f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54d80af140 0x7f54d815f6a0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f54d000d660 tx=0x7f54d000db20 comp rx=0 tx=0).stop 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 -- 172.21.15.81:0/1487291220 shutdown_connections 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f54c4076510 0x7f54c4078a20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54d80af140 0x7f54d815f6a0 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54d815cb30 0x7f54d80afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 --2- 172.21.15.81:0/1487291220 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54d8145530 0x7f54d80b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 -- 172.21.15.81:0/1487291220 >> 172.21.15.81:0/1487291220 conn(0x7f54d806ccc0 msgr2=0x7f54d8141200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 -- 172.21.15.81:0/1487291220 shutdown_connections 2024-06-29T15:58:16.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.298+0000 7f54ddac0640 1 -- 172.21.15.81:0/1487291220 wait complete. 2024-06-29T15:58:16.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:58:16.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.422+0000 7fbdae612640 1 -- 172.21.15.81:0/723544133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbda80b6eb0 msgr2=0x7fbda80d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.422+0000 7fbdae612640 1 --2- 172.21.15.81:0/723544133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbda80b6eb0 0x7fbda80d02d0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fbda8073550 tx=0x7fbd9802a4d0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.422+0000 7fbdae612640 1 -- 172.21.15.81:0/723544133 shutdown_connections 2024-06-29T15:58:16.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.422+0000 7fbdae612640 1 --2- 172.21.15.81:0/723544133 >> v2:172.21.15.167:3300/0 conn(0x7fbda80ce510 0x7fbda80ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.422+0000 7fbdae612640 1 --2- 172.21.15.81:0/723544133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbda80b6eb0 0x7fbda80d02d0 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.422+0000 7fbdae612640 1 -- 172.21.15.81:0/723544133 >> 172.21.15.81:0/723544133 conn(0x7fbda80b3510 msgr2=0x7fbda80b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:16.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.423+0000 7fbdae612640 1 -- 172.21.15.81:0/723544133 shutdown_connections 2024-06-29T15:58:16.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.423+0000 7fbdae612640 1 -- 172.21.15.81:0/723544133 wait complete. 2024-06-29T15:58:16.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.424+0000 7fbdae612640 1 Processor -- start 2024-06-29T15:58:16.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.425+0000 7fbdae612640 1 -- start start 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbdae612640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbda80b4810 0x7fbda80c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbdae612640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbda80b6eb0 0x7fbda8157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbdae612640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbda80ce510 0x7fbda815e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbdae612640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbda80cd480 con 0x7fbda80b6eb0 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbdae612640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbda80b29c0 con 0x7fbda80b4810 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbdae612640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbda80cd600 con 0x7fbda80ce510 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbda77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbda80b6eb0 0x7fbda8157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbda77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbda80b6eb0 0x7fbda8157dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34800/0 (socket says 172.21.15.81:34800) 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbda77fe640 1 -- 172.21.15.81:0/2585911857 learned_addr learned my addr 172.21.15.81:0/2585911857 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbda7fff640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbda80b4810 0x7fbda80c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbdacbaf640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbda80ce510 0x7fbda815e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbda77fe640 1 -- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbda80ce510 msgr2=0x7fbda815e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbda77fe640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbda80ce510 0x7fbda815e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbda77fe640 1 -- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbda80b4810 msgr2=0x7fbda80c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbda77fe640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbda80b4810 0x7fbda80c8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbda77fe640 1 -- 172.21.15.81:0/2585911857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda8165ea0 con 0x7fbda80b6eb0 2024-06-29T15:58:16.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.426+0000 7fbda77fe640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbda80b6eb0 0x7fbda8157dd0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00b410 tx=0x7fbd9c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:16.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.427+0000 7fbda4ff9640 1 -- 172.21.15.81:0/2585911857 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c015020 con 0x7fbda80b6eb0 2024-06-29T15:58:16.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.427+0000 7fbda4ff9640 1 -- 172.21.15.81:0/2585911857 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd9c011940 con 0x7fbda80b6eb0 2024-06-29T15:58:16.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.427+0000 7fbdae612640 1 -- 172.21.15.81:0/2585911857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda81658e0 con 0x7fbda80b6eb0 2024-06-29T15:58:16.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.427+0000 7fbda4ff9640 1 -- 172.21.15.81:0/2585911857 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c011d90 con 0x7fbda80b6eb0 2024-06-29T15:58:16.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.427+0000 7fbdae612640 1 -- 172.21.15.81:0/2585911857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda815e480 con 0x7fbda80b6eb0 2024-06-29T15:58:16.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.428+0000 7fbdae612640 1 -- 172.21.15.81:0/2585911857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbda81592e0 con 0x7fbda80b6eb0 2024-06-29T15:58:16.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.429+0000 7fbda4ff9640 1 -- 172.21.15.81:0/2585911857 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fbd9c018020 con 0x7fbda80b6eb0 2024-06-29T15:58:16.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.429+0000 7fbda4ff9640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbd94076540 0x7fbd94078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.429+0000 7fbda4ff9640 1 -- 172.21.15.81:0/2585911857 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 4985+0+0 (secure 0 0 0) 0x7fbd9c0976f0 con 0x7fbda80b6eb0 2024-06-29T15:58:16.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.429+0000 7fbda7fff640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbd94076540 0x7fbd94078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.430+0000 7fbda7fff640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbd94076540 0x7fbd94078a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbd98033000 tx=0x7fbd98004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:16.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.433+0000 7fbda4ff9640 1 -- 172.21.15.81:0/2585911857 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbd9c01f020 con 0x7fbda80b6eb0 2024-06-29T15:58:16.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.615+0000 7fbdae612640 1 -- 172.21.15.81:0/2585911857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbda80d02d0 con 0x7fbda80b6eb0 2024-06-29T15:58:16.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.615+0000 7fbda4ff9640 1 -- 172.21.15.81:0/2585911857 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v229) ==== 84+0+65 (secure 0 0 0) 0x7fbd9c0608e0 con 0x7fbda80b6eb0 2024-06-29T15:58:16.617 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:16.617 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:58:16.617 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:16.617 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T15:58:16.617 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T15:58:16.617 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:16.618 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:58:16.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 -- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbd94076540 msgr2=0x7fbd94078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbd94076540 0x7fbd94078a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbd98033000 tx=0x7fbd98004730 comp rx=0 tx=0).stop 2024-06-29T15:58:16.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 -- 172.21.15.81:0/2585911857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbda80b6eb0 msgr2=0x7fbda8157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbda80b6eb0 0x7fbda8157dd0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00b410 tx=0x7fbd9c00b8d0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 -- 172.21.15.81:0/2585911857 shutdown_connections 2024-06-29T15:58:16.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fbd94076540 0x7fbd94078a50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbda80ce510 0x7fbda815e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbda80b6eb0 0x7fbda8157dd0 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 --2- 172.21.15.81:0/2585911857 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbda80b4810 0x7fbda80c8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 -- 172.21.15.81:0/2585911857 >> 172.21.15.81:0/2585911857 conn(0x7fbda80b3510 msgr2=0x7fbda8070d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:16.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.617+0000 7fbdae612640 1 -- 172.21.15.81:0/2585911857 shutdown_connections 2024-06-29T15:58:16.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.618+0000 7fbdae612640 1 -- 172.21.15.81:0/2585911857 wait complete. 2024-06-29T15:58:16.630 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:58:16.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:58:16.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.747+0000 7f9f40026640 1 -- 172.21.15.81:0/3062564134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f3806fe70 msgr2=0x7f9f380cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.747+0000 7f9f40026640 1 --2- 172.21.15.81:0/3062564134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f3806fe70 0x7f9f380cdbf0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f9f30005fd0 tx=0x7f9f3002a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.747+0000 7f9f40026640 1 -- 172.21.15.81:0/3062564134 shutdown_connections 2024-06-29T15:58:16.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.747+0000 7f9f40026640 1 --2- 172.21.15.81:0/3062564134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f3806fe70 0x7f9f380cdbf0 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.747+0000 7f9f40026640 1 --2- 172.21.15.81:0/3062564134 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f381415e0 0x7f9f380b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.747+0000 7f9f40026640 1 -- 172.21.15.81:0/3062564134 >> 172.21.15.81:0/3062564134 conn(0x7f9f3806cd60 msgr2=0x7f9f380b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:16.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.747+0000 7f9f40026640 1 -- 172.21.15.81:0/3062564134 shutdown_connections 2024-06-29T15:58:16.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.747+0000 7f9f40026640 1 -- 172.21.15.81:0/3062564134 wait complete. 2024-06-29T15:58:16.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.749+0000 7f9f40026640 1 Processor -- start 2024-06-29T15:58:16.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.749+0000 7f9f40026640 1 -- start start 2024-06-29T15:58:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f40026640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f380b2f70 0x7f9f381444d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f40026640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f381415e0 0x7f9f38143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f40026640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f38144050 0x7f9f3815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f40026640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f380ce710 con 0x7f9f381415e0 2024-06-29T15:58:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f40026640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9f380ce590 con 0x7f9f380b2f70 2024-06-29T15:58:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f40026640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9f380ce890 con 0x7f9f38144050 2024-06-29T15:58:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3d5c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f381415e0 0x7f9f38143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3d5c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f381415e0 0x7f9f38143b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34808/0 (socket says 172.21.15.81:34808) 2024-06-29T15:58:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3d5c1640 1 -- 172.21.15.81:0/2034111445 learned_addr learned my addr 172.21.15.81:0/2034111445 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3e5c3640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f38144050 0x7f9f3815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3ddc2640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f380b2f70 0x7f9f381444d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3d5c1640 1 -- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f38144050 msgr2=0x7f9f3815c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3d5c1640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f38144050 0x7f9f3815c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3d5c1640 1 -- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f380b2f70 msgr2=0x7f9f381444d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3d5c1640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f380b2f70 0x7f9f381444d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3d5c1640 1 -- 172.21.15.81:0/2034111445 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f381e5cd0 con 0x7f9f381415e0 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.750+0000 7f9f3ddc2640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f380b2f70 0x7f9f381444d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.751+0000 7f9f3d5c1640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f381415e0 0x7f9f38143b10 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00b3e0 tx=0x7f9f2c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.751+0000 7f9f367fc640 1 -- 172.21.15.81:0/2034111445 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c015030 con 0x7f9f381415e0 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.751+0000 7f9f367fc640 1 -- 172.21.15.81:0/2034111445 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f2c004110 con 0x7f9f381415e0 2024-06-29T15:58:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.751+0000 7f9f40026640 1 -- 172.21.15.81:0/2034111445 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f381e5fb0 con 0x7f9f381415e0 2024-06-29T15:58:16.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.751+0000 7f9f367fc640 1 -- 172.21.15.81:0/2034111445 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c011630 con 0x7f9f381415e0 2024-06-29T15:58:16.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.751+0000 7f9f40026640 1 -- 172.21.15.81:0/2034111445 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f3815c0b0 con 0x7f9f381415e0 2024-06-29T15:58:16.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.752+0000 7f9f40026640 1 -- 172.21.15.81:0/2034111445 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f380cedb0 con 0x7f9f381415e0 2024-06-29T15:58:16.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.752+0000 7f9f367fc640 1 -- 172.21.15.81:0/2034111445 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f9f2c018070 con 0x7f9f381415e0 2024-06-29T15:58:16.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.753+0000 7f9f367fc640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f24076540 0x7f9f24078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:16.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.753+0000 7f9f367fc640 1 -- 172.21.15.81:0/2034111445 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 4985+0+0 (secure 0 0 0) 0x7f9f2c0973d0 con 0x7f9f381415e0 2024-06-29T15:58:16.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.753+0000 7f9f3ddc2640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f24076540 0x7f9f24078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:16.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.754+0000 7f9f3ddc2640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f24076540 0x7f9f24078a50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9f28005c40 tx=0x7f9f2800a410 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:16.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.757+0000 7f9f367fc640 1 -- 172.21.15.81:0/2034111445 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9f2c060c10 con 0x7f9f381415e0 2024-06-29T15:58:16.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:16.940+0000 7f9f40026640 1 -- 172.21.15.81:0/2034111445 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f9f381e6180 con 0x7f9f381415e0 2024-06-29T15:58:17.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.274+0000 7f9f367fc640 1 -- 172.21.15.81:0/2034111445 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v230) ==== 138+0+0 (secure 0 0 0) 0x7f9f2c060510 con 0x7f9f381415e0 2024-06-29T15:58:17.276 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T15:58:17.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 -- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f24076540 msgr2=0x7f9f24078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f24076540 0x7f9f24078a50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9f28005c40 tx=0x7f9f2800a410 comp rx=0 tx=0).stop 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 -- 172.21.15.81:0/2034111445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f381415e0 msgr2=0x7f9f38143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f381415e0 0x7f9f38143b10 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00b3e0 tx=0x7f9f2c009f90 comp rx=0 tx=0).stop 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 -- 172.21.15.81:0/2034111445 shutdown_connections 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f24076540 0x7f9f24078a50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f38144050 0x7f9f3815c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f381415e0 0x7f9f38143b10 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 --2- 172.21.15.81:0/2034111445 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f380b2f70 0x7f9f381444d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 -- 172.21.15.81:0/2034111445 >> 172.21.15.81:0/2034111445 conn(0x7f9f3806cd60 msgr2=0x7f9f3815aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 -- 172.21.15.81:0/2034111445 shutdown_connections 2024-06-29T15:58:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.277+0000 7f9f40026640 1 -- 172.21.15.81:0/2034111445 wait complete. 2024-06-29T15:58:17.294 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:58:17.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.413+0000 7f536de3c640 1 -- 172.21.15.81:0/2576361155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5350003a30 msgr2=0x7f5350003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.413+0000 7f536de3c640 1 --2- 172.21.15.81:0/2576361155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5350003a30 0x7f5350003ea0 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f5358001dc0 tx=0x7f5350005830 comp rx=0 tx=0).stop 2024-06-29T15:58:17.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.413+0000 7f5367fff640 1 -- 172.21.15.81:0/2576361155 reap_dead start 2024-06-29T15:58:17.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.413+0000 7f536de3c640 1 -- 172.21.15.81:0/2576361155 shutdown_connections 2024-06-29T15:58:17.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.413+0000 7f536de3c640 1 -- 172.21.15.81:0/2576361155 >> 172.21.15.81:0/2576361155 conn(0x7f5368072950 msgr2=0x7f53680b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:17.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.413+0000 7f536de3c640 1 -- 172.21.15.81:0/2576361155 shutdown_connections 2024-06-29T15:58:17.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.414+0000 7f536de3c640 1 -- 172.21.15.81:0/2576361155 wait complete. 2024-06-29T15:58:17.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.415+0000 7f536de3c640 1 Processor -- start 2024-06-29T15:58:17.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.415+0000 7f536de3c640 1 -- start start 2024-06-29T15:58:17.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f536de3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5350003a30 0x7f53681eff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:17.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f536de3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5368153b10 0x7f53681519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:17.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f536de3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5368151ef0 0x7f5368159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:17.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f536de3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f536806cdc0 con 0x7f5368153b10 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f536de3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f53680af1d0 con 0x7f5368151ef0 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f536de3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f536806cf40 con 0x7f5350003a30 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5366ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5368153b10 0x7f53681519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5366ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5368153b10 0x7f53681519b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34820/0 (socket says 172.21.15.81:34820) 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5366ffd640 1 -- 172.21.15.81:0/287477076 learned_addr learned my addr 172.21.15.81:0/287477076 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5367fff640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5368151ef0 0x7f5368159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f53677fe640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5350003a30 0x7f53681eff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5366ffd640 1 -- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5350003a30 msgr2=0x7f53681eff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5366ffd640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5350003a30 0x7f53681eff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5366ffd640 1 -- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5368151ef0 msgr2=0x7f5368159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:17.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5366ffd640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5368151ef0 0x7f5368159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:17.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5366ffd640 1 -- 172.21.15.81:0/287477076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5368152300 con 0x7f5368153b10 2024-06-29T15:58:17.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5367fff640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5368151ef0 0x7f5368159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:17.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f5366ffd640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5368153b10 0x7f53681519b0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f535c00f750 tx=0x7f535c00fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:17.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f534ffff640 1 -- 172.21.15.81:0/287477076 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c00bc50 con 0x7f5368153b10 2024-06-29T15:58:17.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f534ffff640 1 -- 172.21.15.81:0/287477076 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f535c00fe60 con 0x7f5368153b10 2024-06-29T15:58:17.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f536de3c640 1 -- 172.21.15.81:0/287477076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5368154cc0 con 0x7f5368153b10 2024-06-29T15:58:17.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.417+0000 7f534ffff640 1 -- 172.21.15.81:0/287477076 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c0116a0 con 0x7f5368153b10 2024-06-29T15:58:17.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.418+0000 7f536de3c640 1 -- 172.21.15.81:0/287477076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f536814fd40 con 0x7f5368153b10 2024-06-29T15:58:17.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.419+0000 7f536de3c640 1 -- 172.21.15.81:0/287477076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f536806d3c0 con 0x7f5368153b10 2024-06-29T15:58:17.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.419+0000 7f534ffff640 1 -- 172.21.15.81:0/287477076 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f535c019020 con 0x7f5368153b10 2024-06-29T15:58:17.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.420+0000 7f534ffff640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5354076540 0x7f5354078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:17.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.420+0000 7f534ffff640 1 -- 172.21.15.81:0/287477076 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5355+0+0 (secure 0 0 0) 0x7f535c0984b0 con 0x7f5368153b10 2024-06-29T15:58:17.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.420+0000 7f53677fe640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5354076540 0x7f5354078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:17.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.421+0000 7f53677fe640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5354076540 0x7f5354078a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5358032000 tx=0x7f5358001510 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:17.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.424+0000 7f534ffff640 1 -- 172.21.15.81:0/287477076 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5368152300 con 0x7f5368153b10 2024-06-29T15:58:17.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:17.609+0000 7f536de3c640 1 -- 172.21.15.81:0/287477076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f536815fcf0 con 0x7f5368153b10 2024-06-29T15:58:18.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.281+0000 7f534ffff640 1 -- 172.21.15.81:0/287477076 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v231) ==== 170+0+0 (secure 0 0 0) 0x7f535c0619f0 con 0x7f5368153b10 2024-06-29T15:58:18.282 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T15:58:18.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.283+0000 7f536de3c640 1 -- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5354076540 msgr2=0x7f5354078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.283+0000 7f536de3c640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5354076540 0x7f5354078a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5358032000 tx=0x7f5358001510 comp rx=0 tx=0).stop 2024-06-29T15:58:18.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.284+0000 7f536de3c640 1 -- 172.21.15.81:0/287477076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5368153b10 msgr2=0x7f53681519b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.284+0000 7f536de3c640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5368153b10 0x7f53681519b0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f535c00f750 tx=0x7f535c00fc10 comp rx=0 tx=0).stop 2024-06-29T15:58:18.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.284+0000 7f536de3c640 1 -- 172.21.15.81:0/287477076 shutdown_connections 2024-06-29T15:58:18.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.284+0000 7f536de3c640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5354076540 0x7f5354078a50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.284+0000 7f536de3c640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5368151ef0 0x7f5368159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.284+0000 7f536de3c640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5368153b10 0x7f53681519b0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.284+0000 7f536de3c640 1 --2- 172.21.15.81:0/287477076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5350003a30 0x7f53681eff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.284+0000 7f536de3c640 1 -- 172.21.15.81:0/287477076 >> 172.21.15.81:0/287477076 conn(0x7f5368072950 msgr2=0x7f536815c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:18.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.284+0000 7f536de3c640 1 -- 172.21.15.81:0/287477076 shutdown_connections 2024-06-29T15:58:18.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.284+0000 7f536de3c640 1 -- 172.21.15.81:0/287477076 wait complete. 2024-06-29T15:58:18.307 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:58:18.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.423+0000 7f133f4e4640 1 -- 172.21.15.81:0/1775415895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f133806ef90 msgr2=0x7f133806f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.423+0000 7f133f4e4640 1 --2- 172.21.15.81:0/1775415895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f133806ef90 0x7f133806f3a0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f1328005fd0 tx=0x7f132802a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.423+0000 7f133f4e4640 1 -- 172.21.15.81:0/1775415895 shutdown_connections 2024-06-29T15:58:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.423+0000 7f133f4e4640 1 --2- 172.21.15.81:0/1775415895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f133806ef90 0x7f133806f3a0 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.423+0000 7f133f4e4640 1 --2- 172.21.15.81:0/1775415895 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13380b57b0 0x7f133815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.423+0000 7f133f4e4640 1 -- 172.21.15.81:0/1775415895 >> 172.21.15.81:0/1775415895 conn(0x7f13380d0930 msgr2=0x7f13380d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.423+0000 7f133f4e4640 1 -- 172.21.15.81:0/1775415895 shutdown_connections 2024-06-29T15:58:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.423+0000 7f133f4e4640 1 -- 172.21.15.81:0/1775415895 wait complete. 2024-06-29T15:58:18.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.425+0000 7f133f4e4640 1 Processor -- start 2024-06-29T15:58:18.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.425+0000 7f133f4e4640 1 -- start start 2024-06-29T15:58:18.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133f4e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f133806ef90 0x7f133814f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133f4e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13380b57b0 0x7f133806c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133f4e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13380ce1d0 0x7f133806c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133f4e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f13380cecf0 con 0x7f133806ef90 2024-06-29T15:58:18.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133f4e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f13380b4b80 con 0x7f13380b57b0 2024-06-29T15:58:18.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133f4e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f13380cee70 con 0x7f13380ce1d0 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133d280640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f133806ef90 0x7f133814f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133d280640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f133806ef90 0x7f133814f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34840/0 (socket says 172.21.15.81:34840) 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133d280640 1 -- 172.21.15.81:0/2228182753 learned_addr learned my addr 172.21.15.81:0/2228182753 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133ca7f640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13380b57b0 0x7f133806c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133da81640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13380ce1d0 0x7f133806c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133d280640 1 -- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13380ce1d0 msgr2=0x7f133806c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133d280640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13380ce1d0 0x7f133806c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133d280640 1 -- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13380b57b0 msgr2=0x7f133806c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133d280640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13380b57b0 0x7f133806c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.426+0000 7f133d280640 1 -- 172.21.15.81:0/2228182753 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13380aec40 con 0x7f133806ef90 2024-06-29T15:58:18.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.427+0000 7f133d280640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f133806ef90 0x7f133814f1f0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f133400e790 tx=0x7f133400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:18.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.427+0000 7f132dffb640 1 -- 172.21.15.81:0/2228182753 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133400eea0 con 0x7f133806ef90 2024-06-29T15:58:18.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.427+0000 7f132dffb640 1 -- 172.21.15.81:0/2228182753 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f133400cdb0 con 0x7f133806ef90 2024-06-29T15:58:18.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.427+0000 7f133f4e4640 1 -- 172.21.15.81:0/2228182753 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13381f3970 con 0x7f133806ef90 2024-06-29T15:58:18.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.427+0000 7f132dffb640 1 -- 172.21.15.81:0/2228182753 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1334010690 con 0x7f133806ef90 2024-06-29T15:58:18.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.427+0000 7f133f4e4640 1 -- 172.21.15.81:0/2228182753 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13380ad3c0 con 0x7f133806ef90 2024-06-29T15:58:18.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.428+0000 7f133f4e4640 1 -- 172.21.15.81:0/2228182753 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f133806e2a0 con 0x7f133806ef90 2024-06-29T15:58:18.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.429+0000 7f132dffb640 1 -- 172.21.15.81:0/2228182753 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f1334010830 con 0x7f133806ef90 2024-06-29T15:58:18.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.429+0000 7f132dffb640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f131c076510 0x7f131c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:18.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.429+0000 7f132dffb640 1 -- 172.21.15.81:0/2228182753 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 5737+0+0 (secure 0 0 0) 0x7f1334098f50 con 0x7f133806ef90 2024-06-29T15:58:18.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.429+0000 7f133ca7f640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f131c076510 0x7f131c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:18.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.430+0000 7f133ca7f640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f131c076510 0x7f131c078a20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1328033000 tx=0x7f1328004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:18.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.433+0000 7f132dffb640 1 -- 172.21.15.81:0/2228182753 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f133806e2a0 con 0x7f133806ef90 2024-06-29T15:58:18.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:18.652+0000 7f133f4e4640 1 -- 172.21.15.81:0/2228182753 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f133806f3a0 con 0x7f133806ef90 2024-06-29T15:58:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.282+0000 7f132dffb640 1 -- 172.21.15.81:0/2228182753 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 30 and data pool 31 v125) ==== 174+0+0 (secure 0 0 0) 0x7f1334062390 con 0x7f133806ef90 2024-06-29T15:58:19.284 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 30 and data pool 31 2024-06-29T15:58:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.285+0000 7f133f4e4640 1 -- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f131c076510 msgr2=0x7f131c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.285+0000 7f133f4e4640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f131c076510 0x7f131c078a20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1328033000 tx=0x7f1328004730 comp rx=0 tx=0).stop 2024-06-29T15:58:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.285+0000 7f133f4e4640 1 -- 172.21.15.81:0/2228182753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f133806ef90 msgr2=0x7f133814f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.285+0000 7f133f4e4640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f133806ef90 0x7f133814f1f0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f133400e790 tx=0x7f133400ec50 comp rx=0 tx=0).stop 2024-06-29T15:58:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.285+0000 7f133f4e4640 1 -- 172.21.15.81:0/2228182753 shutdown_connections 2024-06-29T15:58:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.285+0000 7f133f4e4640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f131c076510 0x7f131c078a20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.285+0000 7f133f4e4640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f13380ce1d0 0x7f133806c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.285+0000 7f133f4e4640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f13380b57b0 0x7f133806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.285+0000 7f133f4e4640 1 --2- 172.21.15.81:0/2228182753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f133806ef90 0x7f133814f1f0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.285+0000 7f133f4e4640 1 -- 172.21.15.81:0/2228182753 >> 172.21.15.81:0/2228182753 conn(0x7f13380d0930 msgr2=0x7f13380cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:19.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.286+0000 7f133f4e4640 1 -- 172.21.15.81:0/2228182753 shutdown_connections 2024-06-29T15:58:19.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.286+0000 7f133f4e4640 1 -- 172.21.15.81:0/2228182753 wait complete. 2024-06-29T15:58:19.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:19.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.439+0000 7f0720a47640 1 -- 172.21.15.81:0/317996636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f071c06fa90 msgr2=0x7f071c06c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.439+0000 7f0720a47640 1 --2- 172.21.15.81:0/317996636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f071c06fa90 0x7f071c06c830 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f071c073990 tx=0x7f071402a560 comp rx=0 tx=0).stop 2024-06-29T15:58:19.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.439+0000 7f0720a47640 1 -- 172.21.15.81:0/317996636 shutdown_connections 2024-06-29T15:58:19.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.439+0000 7f0720a47640 1 --2- 172.21.15.81:0/317996636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f071c06fa90 0x7f071c06c830 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.439+0000 7f0720a47640 1 --2- 172.21.15.81:0/317996636 >> v2:172.21.15.167:3300/0 conn(0x7f071c06f680 0x7f071c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.439+0000 7f0720a47640 1 --2- 172.21.15.81:0/317996636 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f071c074520 0x7f071c06f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.439+0000 7f0720a47640 1 -- 172.21.15.81:0/317996636 >> 172.21.15.81:0/317996636 conn(0x7f071c072950 msgr2=0x7f071c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:19.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.439+0000 7f0720a47640 1 -- 172.21.15.81:0/317996636 shutdown_connections 2024-06-29T15:58:19.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.439+0000 7f0720a47640 1 -- 172.21.15.81:0/317996636 wait complete. 2024-06-29T15:58:19.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.441+0000 7f0720a47640 1 Processor -- start 2024-06-29T15:58:19.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.441+0000 7f0720a47640 1 -- start start 2024-06-29T15:58:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.442+0000 7f0720a47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f071c06f680 0x7f071c1f2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.442+0000 7f0720a47640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f071c06fa90 0x7f071c0ced00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.442+0000 7f0720a47640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f071c074520 0x7f071c147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.442+0000 7f0720a47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f071c06cd70 con 0x7f071c06f680 2024-06-29T15:58:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.442+0000 7f0720a47640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f071c15b220 con 0x7f071c06fa90 2024-06-29T15:58:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.442+0000 7f0720a47640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f071c06cef0 con 0x7f071c074520 2024-06-29T15:58:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.442+0000 7f071a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f071c06f680 0x7f071c1f2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f071a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f071c06f680 0x7f071c1f2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34856/0 (socket says 172.21.15.81:34856) 2024-06-29T15:58:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f071a59c640 1 -- 172.21.15.81:0/3820045949 learned_addr learned my addr 172.21.15.81:0/3820045949 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f071ad9d640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f071c074520 0x7f071c147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f0719d9b640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f071c06fa90 0x7f071c0ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f071a59c640 1 -- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f071c074520 msgr2=0x7f071c147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f071a59c640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f071c074520 0x7f071c147550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f071a59c640 1 -- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f071c06fa90 msgr2=0x7f071c0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f071a59c640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f071c06fa90 0x7f071c0ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f071a59c640 1 -- 172.21.15.81:0/3820045949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c1f68d0 con 0x7f071c06f680 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f071ad9d640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f071c074520 0x7f071c147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f0719d9b640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f071c06fa90 0x7f071c0ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f071a59c640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f071c06f680 0x7f071c1f2930 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f070c00e9f0 tx=0x7f070c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f0712ffd640 1 -- 172.21.15.81:0/3820045949 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c00cd80 con 0x7f071c06f680 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f0712ffd640 1 -- 172.21.15.81:0/3820045949 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f070c003a60 con 0x7f071c06f680 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f0720a47640 1 -- 172.21.15.81:0/3820045949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071c1f6bb0 con 0x7f071c06f680 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f0712ffd640 1 -- 172.21.15.81:0/3820045949 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c003e30 con 0x7f071c06f680 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.443+0000 7f0720a47640 1 -- 172.21.15.81:0/3820045949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071c1f7170 con 0x7f071c06f680 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.444+0000 7f0720a47640 1 -- 172.21.15.81:0/3820045949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f071c0af7d0 con 0x7f071c06f680 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.445+0000 7f0712ffd640 1 -- 172.21.15.81:0/3820045949 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f070c004000 con 0x7f071c06f680 2024-06-29T15:58:19.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.445+0000 7f0712ffd640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f06fc076510 0x7f06fc078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.445+0000 7f0712ffd640 1 -- 172.21.15.81:0/3820045949 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(232..232 src has 1..232) ==== 5837+0+0 (secure 0 0 0) 0x7f070c098650 con 0x7f071c06f680 2024-06-29T15:58:19.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.446+0000 7f0719d9b640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f06fc076510 0x7f06fc078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.446+0000 7f0719d9b640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f06fc076510 0x7f06fc078a20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f071c157680 tx=0x7f0704005d50 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:19.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.450+0000 7f0712ffd640 1 -- 172.21.15.81:0/3820045949 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f071c0af7d0 con 0x7f071c06f680 2024-06-29T15:58:19.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.633+0000 7f0720a47640 1 -- 172.21.15.81:0/3820045949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f071c0cfce0 con 0x7f071c06f680 2024-06-29T15:58:19.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.634+0000 7f0712ffd640 1 -- 172.21.15.81:0/3820045949 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) ==== 74+0+16824 (secure 0 0 0) 0x7f070c016030 con 0x7f071c06f680 2024-06-29T15:58:19.636 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:19.636 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":232,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:18.654628+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:19.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 -- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f06fc076510 msgr2=0x7f06fc078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f06fc076510 0x7f06fc078a20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f071c157680 tx=0x7f0704005d50 comp rx=0 tx=0).stop 2024-06-29T15:58:19.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 -- 172.21.15.81:0/3820045949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f071c06f680 msgr2=0x7f071c1f2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f071c06f680 0x7f071c1f2930 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f070c00e9f0 tx=0x7f070c00eeb0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 -- 172.21.15.81:0/3820045949 shutdown_connections 2024-06-29T15:58:19.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f06fc076510 0x7f06fc078a20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f071c074520 0x7f071c147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f071c06fa90 0x7f071c0ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 --2- 172.21.15.81:0/3820045949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f071c06f680 0x7f071c1f2930 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 -- 172.21.15.81:0/3820045949 >> 172.21.15.81:0/3820045949 conn(0x7f071c072950 msgr2=0x7f071c159c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:19.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 -- 172.21.15.81:0/3820045949 shutdown_connections 2024-06-29T15:58:19.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.636+0000 7f0720a47640 1 -- 172.21.15.81:0/3820045949 wait complete. 2024-06-29T15:58:19.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:19.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.762+0000 7f4019ae4640 1 -- 172.21.15.81:0/3946196889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f401415c030 msgr2=0x7f401406c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.762+0000 7f4019ae4640 1 --2- 172.21.15.81:0/3946196889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f401415c030 0x7f401406c2f0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f4014071e60 tx=0x7f400802a560 comp rx=0 tx=0).stop 2024-06-29T15:58:19.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.762+0000 7f4019ae4640 1 -- 172.21.15.81:0/3946196889 shutdown_connections 2024-06-29T15:58:19.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.762+0000 7f4019ae4640 1 --2- 172.21.15.81:0/3946196889 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f401415c440 0x7f401406c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.762+0000 7f4019ae4640 1 --2- 172.21.15.81:0/3946196889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f401415c030 0x7f401406c2f0 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.762+0000 7f4019ae4640 1 --2- 172.21.15.81:0/3946196889 >> v2:172.21.15.167:3300/0 conn(0x7f401415d750 0x7f40140afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.762+0000 7f4019ae4640 1 -- 172.21.15.81:0/3946196889 >> 172.21.15.81:0/3946196889 conn(0x7f40140d0930 msgr2=0x7f40140d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:19.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.762+0000 7f4019ae4640 1 -- 172.21.15.81:0/3946196889 shutdown_connections 2024-06-29T15:58:19.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.762+0000 7f4019ae4640 1 -- 172.21.15.81:0/3946196889 wait complete. 2024-06-29T15:58:19.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.763+0000 7f4019ae4640 1 Processor -- start 2024-06-29T15:58:19.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.764+0000 7f4019ae4640 1 -- start start 2024-06-29T15:58:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4019ae4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f401415c030 0x7f4014153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4019ae4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f401415c440 0x7f40141462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4019ae4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f401415d750 0x7f40141f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4019ae4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f401415ff10 con 0x7f401415d750 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4019ae4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f401415fd90 con 0x7f401415c440 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4019ae4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4014160090 con 0x7f401415c030 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4013fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f401415d750 0x7f40141f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4013fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f401415d750 0x7f40141f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34876/0 (socket says 172.21.15.81:34876) 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4013fff640 1 -- 172.21.15.81:0/3202041584 learned_addr learned my addr 172.21.15.81:0/3202041584 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f40137fe640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f401415c030 0x7f4014153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4012ffd640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f401415c440 0x7f40141462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4013fff640 1 -- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f401415c030 msgr2=0x7f4014153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4013fff640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f401415c030 0x7f4014153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4013fff640 1 -- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f401415c440 msgr2=0x7f40141462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4013fff640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f401415c440 0x7f40141462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.765+0000 7f4013fff640 1 -- 172.21.15.81:0/3202041584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40141467f0 con 0x7f401415d750 2024-06-29T15:58:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.766+0000 7f4013fff640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f401415d750 0x7f40141f3090 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f400c00e960 tx=0x7f400c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.766+0000 7f3ffbfff640 1 -- 172.21.15.81:0/3202041584 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c004d20 con 0x7f401415d750 2024-06-29T15:58:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.766+0000 7f3ffbfff640 1 -- 172.21.15.81:0/3202041584 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f400c00ca60 con 0x7f401415d750 2024-06-29T15:58:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.766+0000 7f4019ae4640 1 -- 172.21.15.81:0/3202041584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4014152b90 con 0x7f401415d750 2024-06-29T15:58:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.766+0000 7f3ffbfff640 1 -- 172.21.15.81:0/3202041584 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c005020 con 0x7f401415d750 2024-06-29T15:58:19.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.766+0000 7f4019ae4640 1 -- 172.21.15.81:0/3202041584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4014153150 con 0x7f401415d750 2024-06-29T15:58:19.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.767+0000 7f4019ae4640 1 -- 172.21.15.81:0/3202041584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4014152900 con 0x7f401415d750 2024-06-29T15:58:19.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.768+0000 7f3ffbfff640 1 -- 172.21.15.81:0/3202041584 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f400c0051c0 con 0x7f401415d750 2024-06-29T15:58:19.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.768+0000 7f3ffbfff640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4000076540 0x7f4000078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:19.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.768+0000 7f3ffbfff640 1 -- 172.21.15.81:0/3202041584 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(232..232 src has 1..232) ==== 5837+0+0 (secure 0 0 0) 0x7f400c098800 con 0x7f401415d750 2024-06-29T15:58:19.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.769+0000 7f40137fe640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4000076540 0x7f4000078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:19.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.769+0000 7f40137fe640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4000076540 0x7f4000078a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f400400bb50 tx=0x7f400400b2e0 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:19.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.772+0000 7f3ffbfff640 1 -- 172.21.15.81:0/3202041584 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f400c016030 con 0x7f401415d750 2024-06-29T15:58:19.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.952+0000 7f4019ae4640 1 -- 172.21.15.81:0/3202041584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f401406e2a0 con 0x7f401415d750 2024-06-29T15:58:19.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.953+0000 7f3ffbfff640 1 -- 172.21.15.81:0/3202041584 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) ==== 74+0+16824 (secure 0 0 0) 0x7f401406e2a0 con 0x7f401415d750 2024-06-29T15:58:19.954 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:19.954 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":232,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:18.654628+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:19.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 -- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4000076540 msgr2=0x7f4000078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4000076540 0x7f4000078a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f400400bb50 tx=0x7f400400b2e0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 -- 172.21.15.81:0/3202041584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f401415d750 msgr2=0x7f40141f3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:19.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f401415d750 0x7f40141f3090 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f400c00e960 tx=0x7f400c00ee20 comp rx=0 tx=0).stop 2024-06-29T15:58:19.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 -- 172.21.15.81:0/3202041584 shutdown_connections 2024-06-29T15:58:19.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4000076540 0x7f4000078a50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f401415d750 0x7f40141f3090 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f401415c440 0x7f40141462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 --2- 172.21.15.81:0/3202041584 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f401415c030 0x7f4014153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:19.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 -- 172.21.15.81:0/3202041584 >> 172.21.15.81:0/3202041584 conn(0x7f40140d0930 msgr2=0x7f4014072f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:19.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 -- 172.21.15.81:0/3202041584 shutdown_connections 2024-06-29T15:58:19.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:19.955+0000 7f4019ae4640 1 -- 172.21.15.81:0/3202041584 wait complete. 2024-06-29T15:58:19.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:58:20.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.083+0000 7fb368fd9640 1 -- 172.21.15.81:0/2390027247 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3640ce510 msgr2=0x7fb3640ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:20.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.083+0000 7fb368fd9640 1 --2- 172.21.15.81:0/2390027247 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3640ce510 0x7fb3640ce920 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb3640b41b0 tx=0x7fb34c02a470 comp rx=0 tx=0).stop 2024-06-29T15:58:20.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.083+0000 7fb368fd9640 1 -- 172.21.15.81:0/2390027247 shutdown_connections 2024-06-29T15:58:20.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.083+0000 7fb368fd9640 1 --2- 172.21.15.81:0/2390027247 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3640ce510 0x7fb3640ce920 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.083+0000 7fb368fd9640 1 --2- 172.21.15.81:0/2390027247 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3640b6eb0 0x7fb3640d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.083+0000 7fb368fd9640 1 -- 172.21.15.81:0/2390027247 >> 172.21.15.81:0/2390027247 conn(0x7fb3640b3510 msgr2=0x7fb3640b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:20.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.083+0000 7fb368fd9640 1 -- 172.21.15.81:0/2390027247 shutdown_connections 2024-06-29T15:58:20.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.083+0000 7fb368fd9640 1 -- 172.21.15.81:0/2390027247 wait complete. 2024-06-29T15:58:20.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.085+0000 7fb368fd9640 1 Processor -- start 2024-06-29T15:58:20.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.085+0000 7fb368fd9640 1 -- start start 2024-06-29T15:58:20.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.086+0000 7fb368fd9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3640b4810 0x7fb36409c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:20.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.086+0000 7fb368fd9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3640b6eb0 0x7fb364152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:20.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.086+0000 7fb368fd9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3640ce510 0x7fb364151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:20.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.086+0000 7fb368fd9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb3640cd480 con 0x7fb3640b4810 2024-06-29T15:58:20.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.086+0000 7fb368fd9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb3640b29c0 con 0x7fb3640ce510 2024-06-29T15:58:20.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.086+0000 7fb368fd9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb3640cd600 con 0x7fb3640b6eb0 2024-06-29T15:58:20.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.086+0000 7fb36259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3640b4810 0x7fb36409c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:20.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.086+0000 7fb36259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3640b4810 0x7fb36409c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59732/0 (socket says 172.21.15.81:59732) 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb36259c640 1 -- 172.21.15.81:0/1578628140 learned_addr learned my addr 172.21.15.81:0/1578628140 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb359d9b640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3640b6eb0 0x7fb364152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb362d9d640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3640ce510 0x7fb364151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb36259c640 1 -- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3640b6eb0 msgr2=0x7fb364152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb36259c640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3640b6eb0 0x7fb364152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb36259c640 1 -- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3640ce510 msgr2=0x7fb364151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb36259c640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3640ce510 0x7fb364151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb36259c640 1 -- 172.21.15.81:0/1578628140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36409cb80 con 0x7fb3640b4810 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb359d9b640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3640b6eb0 0x7fb364152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb36259c640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3640b4810 0x7fb36409c280 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fb35400e7c0 tx=0x7fb35400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb35b7fe640 1 -- 172.21.15.81:0/1578628140 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35400ccc0 con 0x7fb3640b4810 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb368fd9640 1 -- 172.21.15.81:0/1578628140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36409ce60 con 0x7fb3640b4810 2024-06-29T15:58:20.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb35b7fe640 1 -- 172.21.15.81:0/1578628140 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb354010430 con 0x7fb3640b4810 2024-06-29T15:58:20.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb35b7fe640 1 -- 172.21.15.81:0/1578628140 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354010860 con 0x7fb3640b4810 2024-06-29T15:58:20.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.087+0000 7fb368fd9640 1 -- 172.21.15.81:0/1578628140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb36409d420 con 0x7fb3640b4810 2024-06-29T15:58:20.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.088+0000 7fb368fd9640 1 -- 172.21.15.81:0/1578628140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3640d02d0 con 0x7fb3640b4810 2024-06-29T15:58:20.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.089+0000 7fb35b7fe640 1 -- 172.21.15.81:0/1578628140 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fb354010a00 con 0x7fb3640b4810 2024-06-29T15:58:20.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.090+0000 7fb35b7fe640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb344076510 0x7fb344078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:20.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.090+0000 7fb35b7fe640 1 -- 172.21.15.81:0/1578628140 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(232..232 src has 1..232) ==== 5837+0+0 (secure 0 0 0) 0x7fb354098740 con 0x7fb3640b4810 2024-06-29T15:58:20.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.090+0000 7fb359d9b640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb344076510 0x7fb344078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:20.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.090+0000 7fb359d9b640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb344076510 0x7fb344078a20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb34c034000 tx=0x7fb34c004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:20.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.094+0000 7fb35b7fe640 1 -- 172.21.15.81:0/1578628140 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb354061b30 con 0x7fb3640b4810 2024-06-29T15:58:20.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:20.316+0000 7fb368fd9640 1 -- 172.21.15.81:0/1578628140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb3640cdf00 con 0x7fb3640b4810 2024-06-29T15:58:21.261 INFO:tasks.ceph.mds.d.smithi167.stderr:2024-06-29T15:58:21.260+0000 7f16caa62640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:21.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.309+0000 7fb35b7fe640 1 -- 172.21.15.81:0/1578628140 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v128) ==== 118+0+0 (secure 0 0 0) 0x7fb354061430 con 0x7fb3640b4810 2024-06-29T15:58:21.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 -- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb344076510 msgr2=0x7fb344078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb344076510 0x7fb344078a20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb34c034000 tx=0x7fb34c004730 comp rx=0 tx=0).stop 2024-06-29T15:58:21.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 -- 172.21.15.81:0/1578628140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3640b4810 msgr2=0x7fb36409c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3640b4810 0x7fb36409c280 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fb35400e7c0 tx=0x7fb35400ec80 comp rx=0 tx=0).stop 2024-06-29T15:58:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 -- 172.21.15.81:0/1578628140 shutdown_connections 2024-06-29T15:58:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fb344076510 0x7fb344078a20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb3640ce510 0x7fb364151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb3640b6eb0 0x7fb364152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 --2- 172.21.15.81:0/1578628140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3640b4810 0x7fb36409c280 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 -- 172.21.15.81:0/1578628140 >> 172.21.15.81:0/1578628140 conn(0x7fb3640b3510 msgr2=0x7fb3640cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 -- 172.21.15.81:0/1578628140 shutdown_connections 2024-06-29T15:58:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.314+0000 7fb368fd9640 1 -- 172.21.15.81:0/1578628140 wait complete. 2024-06-29T15:58:21.329 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:58:21.330 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:58:21.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.450+0000 7f33a208f640 1 -- 172.21.15.81:0/1461240589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f339c0d0b90 msgr2=0x7f339c0d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.450+0000 7f33a208f640 1 --2- 172.21.15.81:0/1461240589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f339c0d0b90 0x7f339c0d1a80 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f3390009830 tx=0x7f339002a5a0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.450+0000 7f33a208f640 1 -- 172.21.15.81:0/1461240589 shutdown_connections 2024-06-29T15:58:21.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.450+0000 7f33a208f640 1 --2- 172.21.15.81:0/1461240589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f339c0d0b90 0x7f339c0d1a80 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.450+0000 7f33a208f640 1 -- 172.21.15.81:0/1461240589 >> 172.21.15.81:0/1461240589 conn(0x7f339c09cf30 msgr2=0x7f339c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:21.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.450+0000 7f33a208f640 1 -- 172.21.15.81:0/1461240589 shutdown_connections 2024-06-29T15:58:21.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.450+0000 7f33a208f640 1 -- 172.21.15.81:0/1461240589 wait complete. 2024-06-29T15:58:21.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.452+0000 7f33a208f640 1 Processor -- start 2024-06-29T15:58:21.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.452+0000 7f33a208f640 1 -- start start 2024-06-29T15:58:21.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.453+0000 7f33a208f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f339c0b5250 0x7f339c09f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:21.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.453+0000 7f33a208f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f339c0b9040 0x7f339c159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:21.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f33a208f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f339c09df60 0x7f339c15a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:21.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f33a208f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f339c0d26d0 con 0x7f339c0b9040 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f33a208f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f339c0b85c0 con 0x7f339c09df60 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f33a208f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f339c0d2850 con 0x7f339c0b5250 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f339c0b9040 0x7f339c159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f339c0b9040 0x7f339c159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59756/0 (socket says 172.21.15.81:59756) 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339affd640 1 -- 172.21.15.81:0/1073209046 learned_addr learned my addr 172.21.15.81:0/1073209046 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339b7fe640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f339c0b5250 0x7f339c09f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339bfff640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f339c09df60 0x7f339c15a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339affd640 1 -- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f339c0b5250 msgr2=0x7f339c09f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339affd640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f339c0b5250 0x7f339c09f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339affd640 1 -- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f339c09df60 msgr2=0x7f339c15a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339affd640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f339c09df60 0x7f339c15a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:21.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339affd640 1 -- 172.21.15.81:0/1073209046 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339c0a11e0 con 0x7f339c0b9040 2024-06-29T15:58:21.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339bfff640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f339c09df60 0x7f339c15a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:21.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f339affd640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f339c0b9040 0x7f339c159c10 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f3390009950 tx=0x7f3390033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:21.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f3383fff640 1 -- 172.21.15.81:0/1073209046 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390031d60 con 0x7f339c0b9040 2024-06-29T15:58:21.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f3383fff640 1 -- 172.21.15.81:0/1073209046 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3390005020 con 0x7f339c0b9040 2024-06-29T15:58:21.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f3383fff640 1 -- 172.21.15.81:0/1073209046 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390005470 con 0x7f339c0b9040 2024-06-29T15:58:21.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f33a208f640 1 -- 172.21.15.81:0/1073209046 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f339c09fc30 con 0x7f339c0b9040 2024-06-29T15:58:21.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.454+0000 7f33a208f640 1 -- 172.21.15.81:0/1073209046 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f339c0a0140 con 0x7f339c0b9040 2024-06-29T15:58:21.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.456+0000 7f33a208f640 1 -- 172.21.15.81:0/1073209046 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f339c09feb0 con 0x7f339c0b9040 2024-06-29T15:58:21.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.456+0000 7f3383fff640 1 -- 172.21.15.81:0/1073209046 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f33900047a0 con 0x7f339c0b9040 2024-06-29T15:58:21.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.456+0000 7f3383fff640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f3388076510 0x7f3388078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:21.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.457+0000 7f3383fff640 1 -- 172.21.15.81:0/1073209046 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f33900b8e50 con 0x7f339c0b9040 2024-06-29T15:58:21.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.457+0000 7f339b7fe640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f3388076510 0x7f3388078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:21.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.457+0000 7f339b7fe640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f3388076510 0x7f3388078a20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f338c006fd0 tx=0x7f338c008040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:21.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.461+0000 7f3383fff640 1 -- 172.21.15.81:0/1073209046 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f339003f030 con 0x7f339c0b9040 2024-06-29T15:58:21.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:21.677+0000 7f33a208f640 1 -- 172.21.15.81:0/1073209046 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f339c0d2de0 con 0x7f339c0b9040 2024-06-29T15:58:22.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.315+0000 7f3383fff640 1 -- 172.21.15.81:0/1073209046 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v129) ==== 122+0+0 (secure 0 0 0) 0x7f3390082390 con 0x7f339c0b9040 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 -- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f3388076510 msgr2=0x7f3388078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f3388076510 0x7f3388078a20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f338c006fd0 tx=0x7f338c008040 comp rx=0 tx=0).stop 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 -- 172.21.15.81:0/1073209046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f339c0b9040 msgr2=0x7f339c159c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f339c0b9040 0x7f339c159c10 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f3390009950 tx=0x7f3390033ce0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 -- 172.21.15.81:0/1073209046 shutdown_connections 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f3388076510 0x7f3388078a20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f339c09df60 0x7f339c15a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f339c0b9040 0x7f339c159c10 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 --2- 172.21.15.81:0/1073209046 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f339c0b5250 0x7f339c09f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 -- 172.21.15.81:0/1073209046 >> 172.21.15.81:0/1073209046 conn(0x7f339c09cf30 msgr2=0x7f339c161300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:22.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 -- 172.21.15.81:0/1073209046 shutdown_connections 2024-06-29T15:58:22.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.319+0000 7f33a208f640 1 -- 172.21.15.81:0/1073209046 wait complete. 2024-06-29T15:58:22.335 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:58:22.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.452+0000 7fcf25d4b640 1 -- 172.21.15.81:0/783820688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf200cd540 msgr2=0x7fcf200b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.452+0000 7fcf25d4b640 1 --2- 172.21.15.81:0/783820688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf200cd540 0x7fcf200b47b0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7fcf200d2600 tx=0x7fcf0802a510 comp rx=0 tx=0).stop 2024-06-29T15:58:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.452+0000 7fcf25d4b640 1 -- 172.21.15.81:0/783820688 shutdown_connections 2024-06-29T15:58:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.452+0000 7fcf25d4b640 1 --2- 172.21.15.81:0/783820688 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf200b4cf0 0x7fcf200d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.452+0000 7fcf25d4b640 1 --2- 172.21.15.81:0/783820688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf200cd540 0x7fcf200b47b0 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.452+0000 7fcf25d4b640 1 -- 172.21.15.81:0/783820688 >> 172.21.15.81:0/783820688 conn(0x7fcf200d1960 msgr2=0x7fcf200d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:22.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.452+0000 7fcf25d4b640 1 -- 172.21.15.81:0/783820688 shutdown_connections 2024-06-29T15:58:22.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.452+0000 7fcf25d4b640 1 -- 172.21.15.81:0/783820688 wait complete. 2024-06-29T15:58:22.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.454+0000 7fcf25d4b640 1 Processor -- start 2024-06-29T15:58:22.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.454+0000 7fcf25d4b640 1 -- start start 2024-06-29T15:58:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.455+0000 7fcf25d4b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf200b4cf0 0x7fcf2015a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf25d4b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf200b6d60 0x7fcf200af670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf25d4b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf200cd540 0x7fcf200afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf25d4b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcf200d07b0 con 0x7fcf200b6d60 2024-06-29T15:58:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf25d4b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcf200d0630 con 0x7fcf200cd540 2024-06-29T15:58:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf25d4b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcf200d0930 con 0x7fcf200b4cf0 2024-06-29T15:58:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf200b6d60 0x7fcf200af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf200b6d60 0x7fcf200af670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59766/0 (socket says 172.21.15.81:59766) 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1effd640 1 -- 172.21.15.81:0/4269545334 learned_addr learned my addr 172.21.15.81:0/4269545334 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1f7fe640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf200b4cf0 0x7fcf2015a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1ffff640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf200cd540 0x7fcf200afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1effd640 1 -- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf200b4cf0 msgr2=0x7fcf2015a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1effd640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf200b4cf0 0x7fcf2015a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1effd640 1 -- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf200cd540 msgr2=0x7fcf200afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1effd640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf200cd540 0x7fcf200afbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1effd640 1 -- 172.21.15.81:0/4269545334 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf200af1c0 con 0x7fcf200b6d60 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf1effd640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf200b6d60 0x7fcf200af670 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fcf0802ab00 tx=0x7fcf08033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf17fff640 1 -- 172.21.15.81:0/4269545334 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08031d60 con 0x7fcf200b6d60 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf17fff640 1 -- 172.21.15.81:0/4269545334 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf08033e10 con 0x7fcf200b6d60 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf25d4b640 1 -- 172.21.15.81:0/4269545334 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf2015ab70 con 0x7fcf200b6d60 2024-06-29T15:58:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf17fff640 1 -- 172.21.15.81:0/4269545334 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08005260 con 0x7fcf200b6d60 2024-06-29T15:58:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.456+0000 7fcf25d4b640 1 -- 172.21.15.81:0/4269545334 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf200c66d0 con 0x7fcf200b6d60 2024-06-29T15:58:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.457+0000 7fcf25d4b640 1 -- 172.21.15.81:0/4269545334 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf200d00f0 con 0x7fcf200b6d60 2024-06-29T15:58:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.458+0000 7fcf17fff640 1 -- 172.21.15.81:0/4269545334 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fcf08005400 con 0x7fcf200b6d60 2024-06-29T15:58:22.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.459+0000 7fcf17fff640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fcf00076540 0x7fcf00078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.459+0000 7fcf17fff640 1 -- 172.21.15.81:0/4269545334 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7fcf080b8eb0 con 0x7fcf200b6d60 2024-06-29T15:58:22.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.459+0000 7fcf1f7fe640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fcf00076540 0x7fcf00078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.460+0000 7fcf1f7fe640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fcf00076540 0x7fcf00078a50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcf10007fd0 tx=0x7fcf10009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:22.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.463+0000 7fcf17fff640 1 -- 172.21.15.81:0/4269545334 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcf0803f030 con 0x7fcf200b6d60 2024-06-29T15:58:22.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.681+0000 7fcf25d4b640 1 -- 172.21.15.81:0/4269545334 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf200b1310 con 0x7fcf200b6d60 2024-06-29T15:58:22.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.681+0000 7fcf17fff640 1 -- 172.21.15.81:0/4269545334 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+5237 (secure 0 0 0) 0x7fcf08088030 con 0x7fcf200b6d60 2024-06-29T15:58:22.683 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:22.683 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":129,"btime":"2024-06-29T15:58:22:311756+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123},{"gid":12093,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:18.654579+0000","modified":"2024-06-29T15:58:21.679033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12105":{"gid":12105,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/30158598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":30158598},{"type":"v1","addr":"172.21.15.167:6837","nonce":30158598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12105,"qdb_cluster":[12105]},"id":15}]} 2024-06-29T15:58:22.683 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 129 2024-06-29T15:58:22.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.683+0000 7fcf25d4b640 1 -- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fcf00076540 msgr2=0x7fcf00078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.683+0000 7fcf25d4b640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fcf00076540 0x7fcf00078a50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcf10007fd0 tx=0x7fcf10009040 comp rx=0 tx=0).stop 2024-06-29T15:58:22.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.683+0000 7fcf25d4b640 1 -- 172.21.15.81:0/4269545334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf200b6d60 msgr2=0x7fcf200af670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.683+0000 7fcf25d4b640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf200b6d60 0x7fcf200af670 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fcf0802ab00 tx=0x7fcf08033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:22.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.683+0000 7fcf25d4b640 1 -- 172.21.15.81:0/4269545334 shutdown_connections 2024-06-29T15:58:22.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.683+0000 7fcf25d4b640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fcf00076540 0x7fcf00078a50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.683+0000 7fcf25d4b640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcf200cd540 0x7fcf200afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.683+0000 7fcf25d4b640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf200b6d60 0x7fcf200af670 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.683+0000 7fcf25d4b640 1 --2- 172.21.15.81:0/4269545334 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcf200b4cf0 0x7fcf2015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.683+0000 7fcf25d4b640 1 -- 172.21.15.81:0/4269545334 >> 172.21.15.81:0/4269545334 conn(0x7fcf200d1960 msgr2=0x7fcf20070800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:22.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.684+0000 7fcf25d4b640 1 -- 172.21.15.81:0/4269545334 shutdown_connections 2024-06-29T15:58:22.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.684+0000 7fcf25d4b640 1 -- 172.21.15.81:0/4269545334 wait complete. 2024-06-29T15:58:22.696 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:22.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.809+0000 7f9fcea12640 1 -- 172.21.15.81:0/980053400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc80b5250 msgr2=0x7f9fc80d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.809+0000 7f9fcea12640 1 --2- 172.21.15.81:0/980053400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc80b5250 0x7f9fc80d09c0 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc005fd0 tx=0x7f9fbc02a5c0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.810+0000 7f9fcea12640 1 -- 172.21.15.81:0/980053400 shutdown_connections 2024-06-29T15:58:22.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.810+0000 7f9fcea12640 1 --2- 172.21.15.81:0/980053400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc80b5250 0x7f9fc80d09c0 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.810+0000 7f9fcea12640 1 --2- 172.21.15.81:0/980053400 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9fc80b9040 0x7f9fc80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.810+0000 7f9fcea12640 1 -- 172.21.15.81:0/980053400 >> 172.21.15.81:0/980053400 conn(0x7f9fc809cf30 msgr2=0x7f9fc80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:22.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.810+0000 7f9fcea12640 1 -- 172.21.15.81:0/980053400 shutdown_connections 2024-06-29T15:58:22.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.810+0000 7f9fcea12640 1 -- 172.21.15.81:0/980053400 wait complete. 2024-06-29T15:58:22.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.811+0000 7f9fcea12640 1 Processor -- start 2024-06-29T15:58:22.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.811+0000 7f9fcea12640 1 -- start start 2024-06-29T15:58:22.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fcea12640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9fc80b5250 0x7f9fc80b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fcea12640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc80b7290 0x7f9fc80b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fcea12640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fc80b9040 0x7f9fc814c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fcea12640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9fc80b7db0 con 0x7f9fc80b7290 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fcea12640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9fc80d2d50 con 0x7f9fc80b9040 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fcea12640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9fc80b7f30 con 0x7f9fc80b5250 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc80b7290 0x7f9fc80b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc80b7290 0x7f9fc80b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59776/0 (socket says 172.21.15.81:59776) 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc77fe640 1 -- 172.21.15.81:0/1591839327 learned_addr learned my addr 172.21.15.81:0/1591839327 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fccfaf640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fc80b9040 0x7f9fc814c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc7fff640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9fc80b5250 0x7f9fc80b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc77fe640 1 -- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9fc80b5250 msgr2=0x7f9fc80b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc77fe640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9fc80b5250 0x7f9fc80b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc77fe640 1 -- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fc80b9040 msgr2=0x7f9fc814c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc77fe640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fc80b9040 0x7f9fc814c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc77fe640 1 -- 172.21.15.81:0/1591839327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fc814d2b0 con 0x7f9fc80b7290 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fccfaf640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fc80b9040 0x7f9fc814c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc7fff640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9fc80b5250 0x7f9fc80b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc77fe640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc80b7290 0x7f9fc80b3dc0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc034040 tx=0x7f9fbc003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.813+0000 7f9fc4ff9640 1 -- 172.21.15.81:0/1591839327 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc03e030 con 0x7f9fc80b7290 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.814+0000 7f9fc4ff9640 1 -- 172.21.15.81:0/1591839327 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fbc0047a0 con 0x7f9fc80b7290 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.814+0000 7f9fcea12640 1 -- 172.21.15.81:0/1591839327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fc814d590 con 0x7f9fc80b7290 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.814+0000 7f9fc4ff9640 1 -- 172.21.15.81:0/1591839327 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc035040 con 0x7f9fc80b7290 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.814+0000 7f9fcea12640 1 -- 172.21.15.81:0/1591839327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fc814db50 con 0x7f9fc80b7290 2024-06-29T15:58:22.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.814+0000 7f9fcea12640 1 -- 172.21.15.81:0/1591839327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fc814d020 con 0x7f9fc80b7290 2024-06-29T15:58:22.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.815+0000 7f9fc4ff9640 1 -- 172.21.15.81:0/1591839327 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f9fbc041030 con 0x7f9fc80b7290 2024-06-29T15:58:22.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.816+0000 7f9fc4ff9640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9fb4076540 0x7f9fb4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:22.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.816+0000 7f9fc4ff9640 1 -- 172.21.15.81:0/1591839327 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f9fbc0b84b0 con 0x7f9fc80b7290 2024-06-29T15:58:22.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.816+0000 7f9fc7fff640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9fb4076540 0x7f9fb4078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:22.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.817+0000 7f9fc7fff640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9fb4076540 0x7f9fb4078a50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8005fd0 tx=0x7f9fb8005e00 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:22.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.820+0000 7f9fc4ff9640 1 -- 172.21.15.81:0/1591839327 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9fbc081960 con 0x7f9fc80b7290 2024-06-29T15:58:23.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:22.999+0000 7f9fcea12640 1 -- 172.21.15.81:0/1591839327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9fc80b84c0 con 0x7f9fc80b7290 2024-06-29T15:58:23.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.000+0000 7f9fc4ff9640 1 -- 172.21.15.81:0/1591839327 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) ==== 74+0+16811 (secure 0 0 0) 0x7f9fbc081260 con 0x7f9fc80b7290 2024-06-29T15:58:23.002 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:23.002 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":233,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:20.296467+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:23.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 -- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9fb4076540 msgr2=0x7f9fb4078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9fb4076540 0x7f9fb4078a50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8005fd0 tx=0x7f9fb8005e00 comp rx=0 tx=0).stop 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 -- 172.21.15.81:0/1591839327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc80b7290 msgr2=0x7f9fc80b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc80b7290 0x7f9fc80b3dc0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc034040 tx=0x7f9fbc003970 comp rx=0 tx=0).stop 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 -- 172.21.15.81:0/1591839327 shutdown_connections 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9fb4076540 0x7f9fb4078a50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9fc80b9040 0x7f9fc814c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc80b7290 0x7f9fc80b3dc0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 --2- 172.21.15.81:0/1591839327 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9fc80b5250 0x7f9fc80b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 -- 172.21.15.81:0/1591839327 >> 172.21.15.81:0/1591839327 conn(0x7f9fc809cf30 msgr2=0x7f9fc815e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 -- 172.21.15.81:0/1591839327 shutdown_connections 2024-06-29T15:58:23.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.003+0000 7f9fcea12640 1 -- 172.21.15.81:0/1591839327 wait complete. 2024-06-29T15:58:23.016 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:58:23.016 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:58:23.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.129+0000 7f7db81eb640 1 -- 172.21.15.81:0/569598139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7db006f850 msgr2=0x7f7db00ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.129+0000 7f7db81eb640 1 --2- 172.21.15.81:0/569598139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7db006f850 0x7f7db00ae300 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009830 tx=0x7f7da002a510 comp rx=0 tx=0).stop 2024-06-29T15:58:23.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.129+0000 7f7db81eb640 1 -- 172.21.15.81:0/569598139 shutdown_connections 2024-06-29T15:58:23.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.129+0000 7f7db81eb640 1 --2- 172.21.15.81:0/569598139 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7db00ae840 0x7f7db00aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.129+0000 7f7db81eb640 1 --2- 172.21.15.81:0/569598139 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7db006f850 0x7f7db00ae300 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.129+0000 7f7db81eb640 1 -- 172.21.15.81:0/569598139 >> 172.21.15.81:0/569598139 conn(0x7f7db0072950 msgr2=0x7f7db00b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:23.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.129+0000 7f7db81eb640 1 -- 172.21.15.81:0/569598139 shutdown_connections 2024-06-29T15:58:23.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.129+0000 7f7db81eb640 1 -- 172.21.15.81:0/569598139 wait complete. 2024-06-29T15:58:23.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.131+0000 7f7db81eb640 1 Processor -- start 2024-06-29T15:58:23.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.131+0000 7f7db81eb640 1 -- start start 2024-06-29T15:58:23.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db81eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7db006f850 0x7f7db01523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db81eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7db0074520 0x7f7db01f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db81eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7db00ae840 0x7f7db00d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db81eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7db015b190 con 0x7f7db00ae840 2024-06-29T15:58:23.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db81eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7db015b010 con 0x7f7db006f850 2024-06-29T15:58:23.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db81eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7db00af190 con 0x7f7db0074520 2024-06-29T15:58:23.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db6788640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7db00ae840 0x7f7db00d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db6788640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7db00ae840 0x7f7db00d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59792/0 (socket says 172.21.15.81:59792) 2024-06-29T15:58:23.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db6788640 1 -- 172.21.15.81:0/3404150280 learned_addr learned my addr 172.21.15.81:0/3404150280 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:23.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db5786640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7db0074520 0x7f7db01f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.132+0000 7f7db5f87640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7db006f850 0x7f7db01523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7db6788640 1 -- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7db0074520 msgr2=0x7f7db01f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7db6788640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7db0074520 0x7f7db01f2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7db6788640 1 -- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7db006f850 msgr2=0x7f7db01523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7db6788640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7db006f850 0x7f7db01523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7db6788640 1 -- 172.21.15.81:0/3404150280 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db01446d0 con 0x7f7db00ae840 2024-06-29T15:58:23.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7db5786640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7db0074520 0x7f7db01f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:23.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7db5f87640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7db006f850 0x7f7db01523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:23.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7db6788640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7db00ae840 0x7f7db00d0030 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f7da800ebf0 tx=0x7f7da800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:23.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7da67fc640 1 -- 172.21.15.81:0/3404150280 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da8003b70 con 0x7f7db00ae840 2024-06-29T15:58:23.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7db81eb640 1 -- 172.21.15.81:0/3404150280 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db01449b0 con 0x7f7db00ae840 2024-06-29T15:58:23.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7da67fc640 1 -- 172.21.15.81:0/3404150280 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7da8003d10 con 0x7f7db00ae840 2024-06-29T15:58:23.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7da67fc640 1 -- 172.21.15.81:0/3404150280 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da8004280 con 0x7f7db00ae840 2024-06-29T15:58:23.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.133+0000 7f7db81eb640 1 -- 172.21.15.81:0/3404150280 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db01545f0 con 0x7f7db00ae840 2024-06-29T15:58:23.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.134+0000 7f7db81eb640 1 -- 172.21.15.81:0/3404150280 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db00ae300 con 0x7f7db00ae840 2024-06-29T15:58:23.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.135+0000 7f7da67fc640 1 -- 172.21.15.81:0/3404150280 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f7da8004420 con 0x7f7db00ae840 2024-06-29T15:58:23.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.135+0000 7f7da67fc640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7d94076510 0x7f7d94078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.135+0000 7f7da67fc640 1 -- 172.21.15.81:0/3404150280 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f7da80989a0 con 0x7f7db00ae840 2024-06-29T15:58:23.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.136+0000 7f7db5f87640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7d94076510 0x7f7d94078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.136+0000 7f7db5f87640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7d94076510 0x7f7d94078a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7dac007a50 tx=0x7f7dac009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:23.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.139+0000 7f7da67fc640 1 -- 172.21.15.81:0/3404150280 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7da8061d90 con 0x7f7db00ae840 2024-06-29T15:58:23.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.314+0000 7f7db81eb640 1 -- 172.21.15.81:0/3404150280 --> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7db0152960 con 0x7f7d94076510 2024-06-29T15:58:23.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.322+0000 7f7da67fc640 1 -- 172.21.15.81:0/3404150280 <== mgr.9155 v2:172.21.15.167:6832/3661607921 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247103 (secure 0 0 0) 0x7f7db0152960 con 0x7f7d94076510 2024-06-29T15:58:23.326 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:23.327 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 -- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7d94076510 msgr2=0x7f7d94078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7d94076510 0x7f7d94078a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7dac007a50 tx=0x7f7dac009040 comp rx=0 tx=0).stop 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 -- 172.21.15.81:0/3404150280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7db00ae840 msgr2=0x7f7db00d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7db00ae840 0x7f7db00d0030 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f7da800ebf0 tx=0x7f7da800c6a0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 -- 172.21.15.81:0/3404150280 shutdown_connections 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7d94076510 0x7f7d94078a20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7db00ae840 0x7f7db00d0030 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7db0074520 0x7f7db01f2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 --2- 172.21.15.81:0/3404150280 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7db006f850 0x7f7db01523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 -- 172.21.15.81:0/3404150280 >> 172.21.15.81:0/3404150280 conn(0x7f7db0072950 msgr2=0x7f7db00b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:23.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 -- 172.21.15.81:0/3404150280 shutdown_connections 2024-06-29T15:58:23.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.325+0000 7f7db81eb640 1 -- 172.21.15.81:0/3404150280 wait complete. 2024-06-29T15:58:23.337 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-06-29T15:58:21.344306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":419036,"kb_used_data":23816,"kb_used_omap":12,"kb_used_meta":395059,"kb_avail":754555684,"statfs":{"total":773094113280,"available":772665020416,"internally_reserved":0,"allocated":24387584,"data_stored":12574552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":404540978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.066945"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.291785+0000","last_change":"2024-06-29T15:58:18.291785+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.291785+0000","last_undegraded":"2024-06-29T15:58:18.291785+0000","last_fullsized":"2024-06-29T15:58:18.291785+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.292594+0000","last_change":"2024-06-29T15:58:18.292594+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.292594+0000","last_undegraded":"2024-06-29T15:58:18.292594+0000","last_fullsized":"2024-06-29T15:58:18.292594+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.293674+0000","last_change":"2024-06-29T15:58:18.293674+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.293674+0000","last_undegraded":"2024-06-29T15:58:18.293674+0000","last_fullsized":"2024-06-29T15:58:18.293674+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.288784+0000","last_change":"2024-06-29T15:58:18.288784+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.288784+0000","last_undegraded":"2024-06-29T15:58:18.288784+0000","last_fullsized":"2024-06-29T15:58:18.288784+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.293406+0000","last_change":"2024-06-29T15:58:18.293406+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.293406+0000","last_undegraded":"2024-06-29T15:58:18.293406+0000","last_fullsized":"2024-06-29T15:58:18.293406+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.288887+0000","last_change":"2024-06-29T15:58:18.288887+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.288887+0000","last_undegraded":"2024-06-29T15:58:18.288887+0000","last_fullsized":"2024-06-29T15:58:18.288887+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.296896+0000","last_change":"2024-06-29T15:58:18.296896+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.296896+0000","last_undegraded":"2024-06-29T15:58:18.296896+0000","last_fullsized":"2024-06-29T15:58:18.296896+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289382+0000","last_change":"2024-06-29T15:58:18.289382+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289382+0000","last_undegraded":"2024-06-29T15:58:18.289382+0000","last_fullsized":"2024-06-29T15:58:18.289382+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.286449+0000","last_change":"2024-06-29T15:58:18.286449+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.286449+0000","last_undegraded":"2024-06-29T15:58:18.286449+0000","last_fullsized":"2024-06-29T15:58:18.286449+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.299287+0000","last_change":"2024-06-29T15:58:18.299287+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.299287+0000","last_undegraded":"2024-06-29T15:58:18.299287+0000","last_fullsized":"2024-06-29T15:58:18.299287+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.297985+0000","last_change":"2024-06-29T15:58:18.297985+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.297985+0000","last_undegraded":"2024-06-29T15:58:18.297985+0000","last_fullsized":"2024-06-29T15:58:18.297985+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.287753+0000","last_change":"2024-06-29T15:58:18.287753+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.287753+0000","last_undegraded":"2024-06-29T15:58:18.287753+0000","last_fullsized":"2024-06-29T15:58:18.287753+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.296791+0000","last_change":"2024-06-29T15:58:18.296791+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.296791+0000","last_undegraded":"2024-06-29T15:58:18.296791+0000","last_fullsized":"2024-06-29T15:58:18.296791+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.290285+0000","last_change":"2024-06-29T15:58:18.290285+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.290285+0000","last_undegraded":"2024-06-29T15:58:18.290285+0000","last_fullsized":"2024-06-29T15:58:18.290285+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.286205+0000","last_change":"2024-06-29T15:58:18.286205+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.286205+0000","last_undegraded":"2024-06-29T15:58:18.286205+0000","last_fullsized":"2024-06-29T15:58:18.286205+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.293073+0000","last_change":"2024-06-29T15:58:18.293073+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.293073+0000","last_undegraded":"2024-06-29T15:58:18.293073+0000","last_fullsized":"2024-06-29T15:58:18.293073+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.288663+0000","last_change":"2024-06-29T15:58:18.288663+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.288663+0000","last_undegraded":"2024-06-29T15:58:18.288663+0000","last_fullsized":"2024-06-29T15:58:18.288663+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.292067+0000","last_change":"2024-06-29T15:58:18.292067+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.292067+0000","last_undegraded":"2024-06-29T15:58:18.292067+0000","last_fullsized":"2024-06-29T15:58:18.292067+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.290211+0000","last_change":"2024-06-29T15:58:18.290211+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.290211+0000","last_undegraded":"2024-06-29T15:58:18.290211+0000","last_fullsized":"2024-06-29T15:58:18.290211+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.292561+0000","last_change":"2024-06-29T15:58:18.292561+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.292561+0000","last_undegraded":"2024-06-29T15:58:18.292561+0000","last_fullsized":"2024-06-29T15:58:18.292561+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.288163+0000","last_change":"2024-06-29T15:58:18.288163+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.288163+0000","last_undegraded":"2024-06-29T15:58:18.288163+0000","last_fullsized":"2024-06-29T15:58:18.288163+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.297083+0000","last_change":"2024-06-29T15:58:18.297083+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.297083+0000","last_undegraded":"2024-06-29T15:58:18.297083+0000","last_fullsized":"2024-06-29T15:58:18.297083+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289903+0000","last_change":"2024-06-29T15:58:18.289903+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289903+0000","last_undegraded":"2024-06-29T15:58:18.289903+0000","last_fullsized":"2024-06-29T15:58:18.289903+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289843+0000","last_change":"2024-06-29T15:58:18.289843+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289843+0000","last_undegraded":"2024-06-29T15:58:18.289843+0000","last_fullsized":"2024-06-29T15:58:18.289843+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289331+0000","last_change":"2024-06-29T15:58:18.289331+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289331+0000","last_undegraded":"2024-06-29T15:58:18.289331+0000","last_fullsized":"2024-06-29T15:58:18.289331+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.298666+0000","last_change":"2024-06-29T15:58:18.298666+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.298666+0000","last_undegraded":"2024-06-29T15:58:18.298666+0000","last_fullsized":"2024-06-29T15:58:18.298666+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.296785+0000","last_change":"2024-06-29T15:58:18.296785+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.296785+0000","last_undegraded":"2024-06-29T15:58:18.296785+0000","last_fullsized":"2024-06-29T15:58:18.296785+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.290799+0000","last_change":"2024-06-29T15:58:18.290799+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.290799+0000","last_undegraded":"2024-06-29T15:58:18.290799+0000","last_fullsized":"2024-06-29T15:58:18.290799+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.296080+0000","last_change":"2024-06-29T15:58:18.296080+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.296080+0000","last_undegraded":"2024-06-29T15:58:18.296080+0000","last_fullsized":"2024-06-29T15:58:18.296080+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.292210+0000","last_change":"2024-06-29T15:58:18.292210+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.292210+0000","last_undegraded":"2024-06-29T15:58:18.292210+0000","last_fullsized":"2024-06-29T15:58:18.292210+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.288705+0000","last_change":"2024-06-29T15:58:18.288705+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.288705+0000","last_undegraded":"2024-06-29T15:58:18.288705+0000","last_fullsized":"2024-06-29T15:58:18.288705+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289284+0000","last_change":"2024-06-29T15:58:18.289284+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289284+0000","last_undegraded":"2024-06-29T15:58:18.289284+0000","last_fullsized":"2024-06-29T15:58:18.289284+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.292068+0000","last_change":"2024-06-29T15:58:18.292068+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.292068+0000","last_undegraded":"2024-06-29T15:58:18.292068+0000","last_fullsized":"2024-06-29T15:58:18.292068+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.304474+0000","last_change":"2024-06-29T15:58:18.301939+0000","last_active":"2024-06-29T15:58:18.304474+0000","last_peered":"2024-06-29T15:58:18.304474+0000","last_clean":"2024-06-29T15:58:18.304474+0000","last_became_active":"2024-06-29T15:58:18.301258+0000","last_became_peered":"2024-06-29T15:58:18.301258+0000","last_unstale":"2024-06-29T15:58:18.304474+0000","last_undegraded":"2024-06-29T15:58:18.304474+0000","last_fullsized":"2024-06-29T15:58:18.304474+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:24.889240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.297569+0000","last_change":"2024-06-29T15:58:18.297569+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.297569+0000","last_undegraded":"2024-06-29T15:58:18.297569+0000","last_fullsized":"2024-06-29T15:58:18.297569+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.298332+0000","last_change":"2024-06-29T15:58:18.297534+0000","last_active":"2024-06-29T15:58:18.298332+0000","last_peered":"2024-06-29T15:58:18.298332+0000","last_clean":"2024-06-29T15:58:18.298332+0000","last_became_active":"2024-06-29T15:58:18.297260+0000","last_became_peered":"2024-06-29T15:58:18.297260+0000","last_unstale":"2024-06-29T15:58:18.298332+0000","last_undegraded":"2024-06-29T15:58:18.298332+0000","last_fullsized":"2024-06-29T15:58:18.298332+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:56:48.182401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289627+0000","last_change":"2024-06-29T15:58:18.289627+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289627+0000","last_undegraded":"2024-06-29T15:58:18.289627+0000","last_fullsized":"2024-06-29T15:58:18.289627+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.305546+0000","last_change":"2024-06-29T15:58:18.302160+0000","last_active":"2024-06-29T15:58:18.305546+0000","last_peered":"2024-06-29T15:58:18.305546+0000","last_clean":"2024-06-29T15:58:18.305546+0000","last_became_active":"2024-06-29T15:58:18.301895+0000","last_became_peered":"2024-06-29T15:58:18.301895+0000","last_unstale":"2024-06-29T15:58:18.305546+0000","last_undegraded":"2024-06-29T15:58:18.305546+0000","last_fullsized":"2024-06-29T15:58:18.305546+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:49:55.416226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.297042+0000","last_change":"2024-06-29T15:58:18.297042+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.297042+0000","last_undegraded":"2024-06-29T15:58:18.297042+0000","last_fullsized":"2024-06-29T15:58:18.297042+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.301263+0000","last_change":"2024-06-29T15:58:18.299460+0000","last_active":"2024-06-29T15:58:18.301263+0000","last_peered":"2024-06-29T15:58:18.301263+0000","last_clean":"2024-06-29T15:58:18.301263+0000","last_became_active":"2024-06-29T15:58:18.299171+0000","last_became_peered":"2024-06-29T15:58:18.299171+0000","last_unstale":"2024-06-29T15:58:18.301263+0000","last_undegraded":"2024-06-29T15:58:18.301263+0000","last_fullsized":"2024-06-29T15:58:18.301263+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:04.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.286958+0000","last_change":"2024-06-29T15:58:18.286958+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.286958+0000","last_undegraded":"2024-06-29T15:58:18.286958+0000","last_fullsized":"2024-06-29T15:58:18.286958+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.300170+0000","last_change":"2024-06-29T15:58:18.297242+0000","last_active":"2024-06-29T15:58:18.300170+0000","last_peered":"2024-06-29T15:58:18.300170+0000","last_clean":"2024-06-29T15:58:18.300170+0000","last_became_active":"2024-06-29T15:58:18.296876+0000","last_became_peered":"2024-06-29T15:58:18.296876+0000","last_unstale":"2024-06-29T15:58:18.300170+0000","last_undegraded":"2024-06-29T15:58:18.300170+0000","last_fullsized":"2024-06-29T15:58:18.300170+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:24.889240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.291567+0000","last_change":"2024-06-29T15:58:18.291567+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.291567+0000","last_undegraded":"2024-06-29T15:58:18.291567+0000","last_fullsized":"2024-06-29T15:58:18.291567+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.307711+0000","last_change":"2024-06-29T15:58:18.302669+0000","last_active":"2024-06-29T15:58:18.307711+0000","last_peered":"2024-06-29T15:58:18.307711+0000","last_clean":"2024-06-29T15:58:18.307711+0000","last_became_active":"2024-06-29T15:58:18.302098+0000","last_became_peered":"2024-06-29T15:58:18.302098+0000","last_unstale":"2024-06-29T15:58:18.307711+0000","last_undegraded":"2024-06-29T15:58:18.307711+0000","last_fullsized":"2024-06-29T15:58:18.307711+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:21.894255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.292497+0000","last_change":"2024-06-29T15:58:18.292497+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.292497+0000","last_undegraded":"2024-06-29T15:58:18.292497+0000","last_fullsized":"2024-06-29T15:58:18.292497+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.305396+0000","last_change":"2024-06-29T15:58:18.300805+0000","last_active":"2024-06-29T15:58:18.305396+0000","last_peered":"2024-06-29T15:58:18.305396+0000","last_clean":"2024-06-29T15:58:18.305396+0000","last_became_active":"2024-06-29T15:58:18.300515+0000","last_became_peered":"2024-06-29T15:58:18.300515+0000","last_unstale":"2024-06-29T15:58:18.305396+0000","last_undegraded":"2024-06-29T15:58:18.305396+0000","last_fullsized":"2024-06-29T15:58:18.305396+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:14:32.010340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.290977+0000","last_change":"2024-06-29T15:58:18.290977+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.290977+0000","last_undegraded":"2024-06-29T15:58:18.290977+0000","last_fullsized":"2024-06-29T15:58:18.290977+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.307599+0000","last_change":"2024-06-29T15:58:18.303872+0000","last_active":"2024-06-29T15:58:18.307599+0000","last_peered":"2024-06-29T15:58:18.307599+0000","last_clean":"2024-06-29T15:58:18.307599+0000","last_became_active":"2024-06-29T15:58:18.303542+0000","last_became_peered":"2024-06-29T15:58:18.303542+0000","last_unstale":"2024-06-29T15:58:18.307599+0000","last_undegraded":"2024-06-29T15:58:18.307599+0000","last_fullsized":"2024-06-29T15:58:18.307599+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:43:51.149312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.297790+0000","last_change":"2024-06-29T15:58:18.297790+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.297790+0000","last_undegraded":"2024-06-29T15:58:18.297790+0000","last_fullsized":"2024-06-29T15:58:18.297790+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.298286+0000","last_change":"2024-06-29T15:58:18.294635+0000","last_active":"2024-06-29T15:58:18.298286+0000","last_peered":"2024-06-29T15:58:18.298286+0000","last_clean":"2024-06-29T15:58:18.298286+0000","last_became_active":"2024-06-29T15:58:18.294343+0000","last_became_peered":"2024-06-29T15:58:18.294343+0000","last_unstale":"2024-06-29T15:58:18.298286+0000","last_undegraded":"2024-06-29T15:58:18.298286+0000","last_fullsized":"2024-06-29T15:58:18.298286+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:06.728456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.291460+0000","last_change":"2024-06-29T15:58:18.291460+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.291460+0000","last_undegraded":"2024-06-29T15:58:18.291460+0000","last_fullsized":"2024-06-29T15:58:18.291460+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.302199+0000","last_change":"2024-06-29T15:58:18.300841+0000","last_active":"2024-06-29T15:58:18.302199+0000","last_peered":"2024-06-29T15:58:18.302199+0000","last_clean":"2024-06-29T15:58:18.302199+0000","last_became_active":"2024-06-29T15:58:18.300525+0000","last_became_peered":"2024-06-29T15:58:18.300525+0000","last_unstale":"2024-06-29T15:58:18.302199+0000","last_undegraded":"2024-06-29T15:58:18.302199+0000","last_fullsized":"2024-06-29T15:58:18.302199+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:53.353143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.294668+0000","last_change":"2024-06-29T15:58:18.294668+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.294668+0000","last_undegraded":"2024-06-29T15:58:18.294668+0000","last_fullsized":"2024-06-29T15:58:18.294668+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.299219+0000","last_change":"2024-06-29T15:58:18.296167+0000","last_active":"2024-06-29T15:58:18.299219+0000","last_peered":"2024-06-29T15:58:18.299219+0000","last_clean":"2024-06-29T15:58:18.299219+0000","last_became_active":"2024-06-29T15:58:18.294984+0000","last_became_peered":"2024-06-29T15:58:18.294984+0000","last_unstale":"2024-06-29T15:58:18.299219+0000","last_undegraded":"2024-06-29T15:58:18.299219+0000","last_fullsized":"2024-06-29T15:58:18.299219+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:49:48.139670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.296495+0000","last_change":"2024-06-29T15:58:18.296495+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.296495+0000","last_undegraded":"2024-06-29T15:58:18.296495+0000","last_fullsized":"2024-06-29T15:58:18.296495+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.303905+0000","last_change":"2024-06-29T15:58:18.303307+0000","last_active":"2024-06-29T15:58:18.303905+0000","last_peered":"2024-06-29T15:58:18.303905+0000","last_clean":"2024-06-29T15:58:18.303905+0000","last_became_active":"2024-06-29T15:58:18.303009+0000","last_became_peered":"2024-06-29T15:58:18.303009+0000","last_unstale":"2024-06-29T15:58:18.303905+0000","last_undegraded":"2024-06-29T15:58:18.303905+0000","last_fullsized":"2024-06-29T15:58:18.303905+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:21.894255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.286128+0000","last_change":"2024-06-29T15:58:18.286128+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.286128+0000","last_undegraded":"2024-06-29T15:58:18.286128+0000","last_fullsized":"2024-06-29T15:58:18.286128+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.296032+0000","last_change":"2024-06-29T15:58:18.290469+0000","last_active":"2024-06-29T15:58:18.296032+0000","last_peered":"2024-06-29T15:58:18.296032+0000","last_clean":"2024-06-29T15:58:18.296032+0000","last_became_active":"2024-06-29T15:58:18.290177+0000","last_became_peered":"2024-06-29T15:58:18.290177+0000","last_unstale":"2024-06-29T15:58:18.296032+0000","last_undegraded":"2024-06-29T15:58:18.296032+0000","last_fullsized":"2024-06-29T15:58:18.296032+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:08:28.474451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.288890+0000","last_change":"2024-06-29T15:58:18.288890+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.288890+0000","last_undegraded":"2024-06-29T15:58:18.288890+0000","last_fullsized":"2024-06-29T15:58:18.288890+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.307742+0000","last_change":"2024-06-29T15:58:18.303515+0000","last_active":"2024-06-29T15:58:18.307742+0000","last_peered":"2024-06-29T15:58:18.307742+0000","last_clean":"2024-06-29T15:58:18.307742+0000","last_became_active":"2024-06-29T15:58:18.303008+0000","last_became_peered":"2024-06-29T15:58:18.303008+0000","last_unstale":"2024-06-29T15:58:18.307742+0000","last_undegraded":"2024-06-29T15:58:18.307742+0000","last_fullsized":"2024-06-29T15:58:18.307742+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:08.507147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.296350+0000","last_change":"2024-06-29T15:58:18.296350+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.296350+0000","last_undegraded":"2024-06-29T15:58:18.296350+0000","last_fullsized":"2024-06-29T15:58:18.296350+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.303183+0000","last_change":"2024-06-29T15:58:18.302451+0000","last_active":"2024-06-29T15:58:18.303183+0000","last_peered":"2024-06-29T15:58:18.303183+0000","last_clean":"2024-06-29T15:58:18.303183+0000","last_became_active":"2024-06-29T15:58:18.302144+0000","last_became_peered":"2024-06-29T15:58:18.302144+0000","last_unstale":"2024-06-29T15:58:18.303183+0000","last_undegraded":"2024-06-29T15:58:18.303183+0000","last_fullsized":"2024-06-29T15:58:18.303183+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:39:56.631239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289572+0000","last_change":"2024-06-29T15:58:18.289572+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289572+0000","last_undegraded":"2024-06-29T15:58:18.289572+0000","last_fullsized":"2024-06-29T15:58:18.289572+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.307747+0000","last_change":"2024-06-29T15:58:18.303063+0000","last_active":"2024-06-29T15:58:18.307747+0000","last_peered":"2024-06-29T15:58:18.307747+0000","last_clean":"2024-06-29T15:58:18.307747+0000","last_became_active":"2024-06-29T15:58:18.302403+0000","last_became_peered":"2024-06-29T15:58:18.302403+0000","last_unstale":"2024-06-29T15:58:18.307747+0000","last_undegraded":"2024-06-29T15:58:18.307747+0000","last_fullsized":"2024-06-29T15:58:18.307747+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:07.733361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289577+0000","last_change":"2024-06-29T15:58:18.289577+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289577+0000","last_undegraded":"2024-06-29T15:58:18.289577+0000","last_fullsized":"2024-06-29T15:58:18.289577+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.298308+0000","last_change":"2024-06-29T15:58:18.296680+0000","last_active":"2024-06-29T15:58:18.298308+0000","last_peered":"2024-06-29T15:58:18.298308+0000","last_clean":"2024-06-29T15:58:18.298308+0000","last_became_active":"2024-06-29T15:58:18.296405+0000","last_became_peered":"2024-06-29T15:58:18.296405+0000","last_unstale":"2024-06-29T15:58:18.298308+0000","last_undegraded":"2024-06-29T15:58:18.298308+0000","last_fullsized":"2024-06-29T15:58:18.298308+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:33:56.796519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.286231+0000","last_change":"2024-06-29T15:58:18.286231+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.286231+0000","last_undegraded":"2024-06-29T15:58:18.286231+0000","last_fullsized":"2024-06-29T15:58:18.286231+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.295845+0000","last_change":"2024-06-29T15:58:18.294949+0000","last_active":"2024-06-29T15:58:18.295845+0000","last_peered":"2024-06-29T15:58:18.295845+0000","last_clean":"2024-06-29T15:58:18.295845+0000","last_became_active":"2024-06-29T15:58:18.294015+0000","last_became_peered":"2024-06-29T15:58:18.294015+0000","last_unstale":"2024-06-29T15:58:18.295845+0000","last_undegraded":"2024-06-29T15:58:18.295845+0000","last_fullsized":"2024-06-29T15:58:18.295845+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:32:26.919635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289053+0000","last_change":"2024-06-29T15:58:18.289053+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289053+0000","last_undegraded":"2024-06-29T15:58:18.289053+0000","last_fullsized":"2024-06-29T15:58:18.289053+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.306801+0000","last_change":"2024-06-29T15:58:18.306207+0000","last_active":"2024-06-29T15:58:18.306801+0000","last_peered":"2024-06-29T15:58:18.306801+0000","last_clean":"2024-06-29T15:58:18.306801+0000","last_became_active":"2024-06-29T15:58:18.305661+0000","last_became_peered":"2024-06-29T15:58:18.305661+0000","last_unstale":"2024-06-29T15:58:18.306801+0000","last_undegraded":"2024-06-29T15:58:18.306801+0000","last_fullsized":"2024-06-29T15:58:18.306801+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:04:42.286641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.287738+0000","last_change":"2024-06-29T15:58:18.287738+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.287738+0000","last_undegraded":"2024-06-29T15:58:18.287738+0000","last_fullsized":"2024-06-29T15:58:18.287738+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.305475+0000","last_change":"2024-06-29T15:58:18.302497+0000","last_active":"2024-06-29T15:58:18.305475+0000","last_peered":"2024-06-29T15:58:18.305475+0000","last_clean":"2024-06-29T15:58:18.305475+0000","last_became_active":"2024-06-29T15:58:18.302200+0000","last_became_peered":"2024-06-29T15:58:18.302200+0000","last_unstale":"2024-06-29T15:58:18.305475+0000","last_undegraded":"2024-06-29T15:58:18.305475+0000","last_fullsized":"2024-06-29T15:58:18.305475+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:02:34.982461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.292063+0000","last_change":"2024-06-29T15:58:18.292063+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.292063+0000","last_undegraded":"2024-06-29T15:58:18.292063+0000","last_fullsized":"2024-06-29T15:58:18.292063+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.307008+0000","last_change":"2024-06-29T15:58:18.306269+0000","last_active":"2024-06-29T15:58:18.307008+0000","last_peered":"2024-06-29T15:58:18.307008+0000","last_clean":"2024-06-29T15:58:18.307008+0000","last_became_active":"2024-06-29T15:58:18.305098+0000","last_became_peered":"2024-06-29T15:58:18.305098+0000","last_unstale":"2024-06-29T15:58:18.307008+0000","last_undegraded":"2024-06-29T15:58:18.307008+0000","last_fullsized":"2024-06-29T15:58:18.307008+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:53.145340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.297544+0000","last_change":"2024-06-29T15:58:18.297544+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.297544+0000","last_undegraded":"2024-06-29T15:58:18.297544+0000","last_fullsized":"2024-06-29T15:58:18.297544+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.307875+0000","last_change":"2024-06-29T15:58:18.304832+0000","last_active":"2024-06-29T15:58:18.307875+0000","last_peered":"2024-06-29T15:58:18.307875+0000","last_clean":"2024-06-29T15:58:18.307875+0000","last_became_active":"2024-06-29T15:58:18.304230+0000","last_became_peered":"2024-06-29T15:58:18.304230+0000","last_unstale":"2024-06-29T15:58:18.307875+0000","last_undegraded":"2024-06-29T15:58:18.307875+0000","last_fullsized":"2024-06-29T15:58:18.307875+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:35:53.219398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289214+0000","last_change":"2024-06-29T15:58:18.289214+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289214+0000","last_undegraded":"2024-06-29T15:58:18.289214+0000","last_fullsized":"2024-06-29T15:58:18.289214+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.301102+0000","last_change":"2024-06-29T15:58:18.297661+0000","last_active":"2024-06-29T15:58:18.301102+0000","last_peered":"2024-06-29T15:58:18.301102+0000","last_clean":"2024-06-29T15:58:18.301102+0000","last_became_active":"2024-06-29T15:58:18.297272+0000","last_became_peered":"2024-06-29T15:58:18.297272+0000","last_unstale":"2024-06-29T15:58:18.301102+0000","last_undegraded":"2024-06-29T15:58:18.301102+0000","last_fullsized":"2024-06-29T15:58:18.301102+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:49:48.139670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.298064+0000","last_change":"2024-06-29T15:58:18.298064+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.298064+0000","last_undegraded":"2024-06-29T15:58:18.298064+0000","last_fullsized":"2024-06-29T15:58:18.298064+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.304097+0000","last_change":"2024-06-29T15:58:18.302543+0000","last_active":"2024-06-29T15:58:18.304097+0000","last_peered":"2024-06-29T15:58:18.304097+0000","last_clean":"2024-06-29T15:58:18.304097+0000","last_became_active":"2024-06-29T15:58:18.302237+0000","last_became_peered":"2024-06-29T15:58:18.302237+0000","last_unstale":"2024-06-29T15:58:18.304097+0000","last_undegraded":"2024-06-29T15:58:18.304097+0000","last_fullsized":"2024-06-29T15:58:18.304097+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:00.378997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.290096+0000","last_change":"2024-06-29T15:58:18.290096+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.290096+0000","last_undegraded":"2024-06-29T15:58:18.290096+0000","last_fullsized":"2024-06-29T15:58:18.290096+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.305226+0000","last_change":"2024-06-29T15:58:18.303588+0000","last_active":"2024-06-29T15:58:18.305226+0000","last_peered":"2024-06-29T15:58:18.305226+0000","last_clean":"2024-06-29T15:58:18.305226+0000","last_became_active":"2024-06-29T15:58:18.303274+0000","last_became_peered":"2024-06-29T15:58:18.303274+0000","last_unstale":"2024-06-29T15:58:18.305226+0000","last_undegraded":"2024-06-29T15:58:18.305226+0000","last_fullsized":"2024-06-29T15:58:18.305226+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:43:51.149312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.297128+0000","last_change":"2024-06-29T15:58:18.297128+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.297128+0000","last_undegraded":"2024-06-29T15:58:18.297128+0000","last_fullsized":"2024-06-29T15:58:18.297128+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.304936+0000","last_change":"2024-06-29T15:58:18.303622+0000","last_active":"2024-06-29T15:58:18.304936+0000","last_peered":"2024-06-29T15:58:18.304936+0000","last_clean":"2024-06-29T15:58:18.304936+0000","last_became_active":"2024-06-29T15:58:18.303319+0000","last_became_peered":"2024-06-29T15:58:18.303319+0000","last_unstale":"2024-06-29T15:58:18.304936+0000","last_undegraded":"2024-06-29T15:58:18.304936+0000","last_fullsized":"2024-06-29T15:58:18.304936+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:12:00.101946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.296339+0000","last_change":"2024-06-29T15:58:18.296339+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.296339+0000","last_undegraded":"2024-06-29T15:58:18.296339+0000","last_fullsized":"2024-06-29T15:58:18.296339+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.306770+0000","last_change":"2024-06-29T15:58:18.305967+0000","last_active":"2024-06-29T15:58:18.306770+0000","last_peered":"2024-06-29T15:58:18.306770+0000","last_clean":"2024-06-29T15:58:18.306770+0000","last_became_active":"2024-06-29T15:58:18.304464+0000","last_became_peered":"2024-06-29T15:58:18.304464+0000","last_unstale":"2024-06-29T15:58:18.306770+0000","last_undegraded":"2024-06-29T15:58:18.306770+0000","last_fullsized":"2024-06-29T15:58:18.306770+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:33:49.594847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.294982+0000","last_change":"2024-06-29T15:58:18.294982+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.294982+0000","last_undegraded":"2024-06-29T15:58:18.294982+0000","last_fullsized":"2024-06-29T15:58:18.294982+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.307705+0000","last_change":"2024-06-29T15:58:18.302661+0000","last_active":"2024-06-29T15:58:18.307705+0000","last_peered":"2024-06-29T15:58:18.307705+0000","last_clean":"2024-06-29T15:58:18.307705+0000","last_became_active":"2024-06-29T15:58:18.302330+0000","last_became_peered":"2024-06-29T15:58:18.302330+0000","last_unstale":"2024-06-29T15:58:18.307705+0000","last_undegraded":"2024-06-29T15:58:18.307705+0000","last_fullsized":"2024-06-29T15:58:18.307705+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:53.353143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.287579+0000","last_change":"2024-06-29T15:58:18.287579+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.287579+0000","last_undegraded":"2024-06-29T15:58:18.287579+0000","last_fullsized":"2024-06-29T15:58:18.287579+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.299816+0000","last_change":"2024-06-29T15:58:18.294669+0000","last_active":"2024-06-29T15:58:18.299816+0000","last_peered":"2024-06-29T15:58:18.299816+0000","last_clean":"2024-06-29T15:58:18.299816+0000","last_became_active":"2024-06-29T15:58:18.294353+0000","last_became_peered":"2024-06-29T15:58:18.294353+0000","last_unstale":"2024-06-29T15:58:18.299816+0000","last_undegraded":"2024-06-29T15:58:18.299816+0000","last_fullsized":"2024-06-29T15:58:18.299816+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:43:51.149312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.288276+0000","last_change":"2024-06-29T15:58:18.288276+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.288276+0000","last_undegraded":"2024-06-29T15:58:18.288276+0000","last_fullsized":"2024-06-29T15:58:18.288276+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.303730+0000","last_change":"2024-06-29T15:58:18.298256+0000","last_active":"2024-06-29T15:58:18.303730+0000","last_peered":"2024-06-29T15:58:18.303730+0000","last_clean":"2024-06-29T15:58:18.303730+0000","last_became_active":"2024-06-29T15:58:18.297928+0000","last_became_peered":"2024-06-29T15:58:18.297928+0000","last_unstale":"2024-06-29T15:58:18.303730+0000","last_undegraded":"2024-06-29T15:58:18.303730+0000","last_fullsized":"2024-06-29T15:58:18.303730+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:34:59.303359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.289308+0000","last_change":"2024-06-29T15:58:18.289308+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.289308+0000","last_undegraded":"2024-06-29T15:58:18.289308+0000","last_fullsized":"2024-06-29T15:58:18.289308+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.305564+0000","last_change":"2024-06-29T15:58:18.302761+0000","last_active":"2024-06-29T15:58:18.305564+0000","last_peered":"2024-06-29T15:58:18.305564+0000","last_clean":"2024-06-29T15:58:18.305564+0000","last_became_active":"2024-06-29T15:58:18.302490+0000","last_became_peered":"2024-06-29T15:58:18.302490+0000","last_unstale":"2024-06-29T15:58:18.305564+0000","last_undegraded":"2024-06-29T15:58:18.305564+0000","last_fullsized":"2024-06-29T15:58:18.305564+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:31:30.376154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-06-29T15:58:18.293173+0000","last_change":"2024-06-29T15:58:18.293173+0000","last_active":"2024-06-29T15:58:18.276901+0000","last_peered":"2024-06-29T15:58:18.276901+0000","last_clean":"2024-06-29T15:58:18.276901+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:58:18.293173+0000","last_undegraded":"2024-06-29T15:58:18.293173+0000","last_fullsized":"2024-06-29T15:58:18.293173+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":9,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.305594+0000","last_change":"2024-06-29T15:58:18.303249+0000","last_active":"2024-06-29T15:58:18.305594+0000","last_peered":"2024-06-29T15:58:18.305594+0000","last_clean":"2024-06-29T15:58:18.305594+0000","last_became_active":"2024-06-29T15:58:18.302916+0000","last_became_peered":"2024-06-29T15:58:18.302916+0000","last_unstale":"2024-06-29T15:58:18.305594+0000","last_undegraded":"2024-06-29T15:58:18.305594+0000","last_fullsized":"2024-06-29T15:58:18.305594+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:04:24.166081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"219'233","reported_seq":1121,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-06-29T15:58:18.287782+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:58:18.287782+0000","last_peered":"2024-06-29T15:58:18.287782+0000","last_clean":"2024-06-29T15:58:18.287782+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:58:18.287782+0000","last_undegraded":"2024-06-29T15:58:18.287782+0000","last_fullsized":"2024-06-29T15:58:18.287782+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575995,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52640,"kb_used_data":3412,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319200,"statfs":{"total":96636764160,"available":96582860800,"internally_reserved":0,"allocated":3493888,"data_stored":2014595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42340,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329500,"statfs":{"total":96636764160,"available":96593408000,"internally_reserved":0,"allocated":2899968,"data_stored":1424227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575994,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46508,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325332,"statfs":{"total":96636764160,"available":96589139968,"internally_reserved":0,"allocated":2899968,"data_stored":1424227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60388,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311452,"statfs":{"total":96636764160,"available":96574926848,"internally_reserved":0,"allocated":2899968,"data_stored":1424227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575995,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47900,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323940,"statfs":{"total":96636764160,"available":96587714560,"internally_reserved":0,"allocated":2899968,"data_stored":1424227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575995,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47912,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323928,"statfs":{"total":96636764160,"available":96587702272,"internally_reserved":0,"allocated":2899968,"data_stored":1424227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60388,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311452,"statfs":{"total":96636764160,"available":96574926848,"internally_reserved":0,"allocated":2899968,"data_stored":1424227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575995,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60960,"kb_used_data":3412,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310880,"statfs":{"total":96636764160,"available":96574341120,"internally_reserved":0,"allocated":3493888,"data_stored":2014595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:58:23.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:58:23.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.492+0000 7fdf65f94640 1 -- 172.21.15.81:0/892888782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf48004800 msgr2=0x7fdf48004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.492+0000 7fdf65f94640 1 --2- 172.21.15.81:0/892888782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf48004800 0x7fdf48004c10 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7fdf540016c0 tx=0x7fdf48005830 comp rx=0 tx=0).stop 2024-06-29T15:58:23.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.492+0000 7fdf65f94640 1 -- 172.21.15.81:0/892888782 shutdown_connections 2024-06-29T15:58:23.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.492+0000 7fdf65f94640 1 --2- 172.21.15.81:0/892888782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf48004800 0x7fdf48004c10 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.492+0000 7fdf65f94640 1 -- 172.21.15.81:0/892888782 >> 172.21.15.81:0/892888782 conn(0x7fdf600b56d0 msgr2=0x7fdf600b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:23.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.492+0000 7fdf65f94640 1 -- 172.21.15.81:0/892888782 shutdown_connections 2024-06-29T15:58:23.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.492+0000 7fdf65f94640 1 -- 172.21.15.81:0/892888782 wait complete. 2024-06-29T15:58:23.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.494+0000 7fdf65f94640 1 Processor -- start 2024-06-29T15:58:23.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.494+0000 7fdf65f94640 1 -- start start 2024-06-29T15:58:23.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf65f94640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf48004800 0x7fdf6009e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf65f94640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf600a1530 0x7fdf601e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf65f94640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf601ea380 0x7fdf601e7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf65f94640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdf600ceab0 con 0x7fdf48004800 2024-06-29T15:58:23.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf65f94640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdf600ce930 con 0x7fdf600a1530 2024-06-29T15:58:23.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf65f94640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdf600cec30 con 0x7fdf601ea380 2024-06-29T15:58:23.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf5f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf48004800 0x7fdf6009e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf5f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf48004800 0x7fdf6009e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59806/0 (socket says 172.21.15.81:59806) 2024-06-29T15:58:23.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf5f7fe640 1 -- 172.21.15.81:0/111315814 learned_addr learned my addr 172.21.15.81:0/111315814 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:23.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf5effd640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf600a1530 0x7fdf601e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf5ffff640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf601ea380 0x7fdf601e7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf5f7fe640 1 -- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf601ea380 msgr2=0x7fdf601e7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf5f7fe640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf601ea380 0x7fdf601e7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf5f7fe640 1 -- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf600a1530 msgr2=0x7fdf601e9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf5f7fe640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf600a1530 0x7fdf601e9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.495+0000 7fdf5f7fe640 1 -- 172.21.15.81:0/111315814 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf601e3980 con 0x7fdf48004800 2024-06-29T15:58:23.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.496+0000 7fdf5f7fe640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf48004800 0x7fdf6009e5a0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7fdf50011a80 tx=0x7fdf50011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:23.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.496+0000 7fdf47fff640 1 -- 172.21.15.81:0/111315814 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5000fd50 con 0x7fdf48004800 2024-06-29T15:58:23.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.496+0000 7fdf47fff640 1 -- 172.21.15.81:0/111315814 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf50016070 con 0x7fdf48004800 2024-06-29T15:58:23.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.496+0000 7fdf65f94640 1 -- 172.21.15.81:0/111315814 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6009dac0 con 0x7fdf48004800 2024-06-29T15:58:23.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.496+0000 7fdf47fff640 1 -- 172.21.15.81:0/111315814 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50012690 con 0x7fdf48004800 2024-06-29T15:58:23.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.496+0000 7fdf65f94640 1 -- 172.21.15.81:0/111315814 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6009e080 con 0x7fdf48004800 2024-06-29T15:58:23.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.497+0000 7fdf65f94640 1 -- 172.21.15.81:0/111315814 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6009d8b0 con 0x7fdf48004800 2024-06-29T15:58:23.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.498+0000 7fdf47fff640 1 -- 172.21.15.81:0/111315814 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fdf50018030 con 0x7fdf48004800 2024-06-29T15:58:23.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.498+0000 7fdf47fff640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf4c076510 0x7fdf4c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.498+0000 7fdf47fff640 1 -- 172.21.15.81:0/111315814 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7fdf50098560 con 0x7fdf48004800 2024-06-29T15:58:23.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.498+0000 7fdf5effd640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf4c076510 0x7fdf4c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.499+0000 7fdf5effd640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf4c076510 0x7fdf4c078a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf54032000 tx=0x7fdf5402c040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:23.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.503+0000 7fdf47fff640 1 -- 172.21.15.81:0/111315814 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdf50061380 con 0x7fdf48004800 2024-06-29T15:58:23.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.678+0000 7fdf65f94640 1 -- 172.21.15.81:0/111315814 --> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf600cf190 con 0x7fdf4c076510 2024-06-29T15:58:23.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.683+0000 7fdf47fff640 1 -- 172.21.15.81:0/111315814 <== mgr.9155 v2:172.21.15.167:6832/3661607921 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252500 (secure 0 0 0) 0x7fdf600cf190 con 0x7fdf4c076510 2024-06-29T15:58:23.687 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:23.688 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 -- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf4c076510 msgr2=0x7fdf4c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf4c076510 0x7fdf4c078a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdf54032000 tx=0x7fdf5402c040 comp rx=0 tx=0).stop 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 -- 172.21.15.81:0/111315814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf48004800 msgr2=0x7fdf6009e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf48004800 0x7fdf6009e5a0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7fdf50011a80 tx=0x7fdf50011e80 comp rx=0 tx=0).stop 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 -- 172.21.15.81:0/111315814 shutdown_connections 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf4c076510 0x7fdf4c078a20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdf601ea380 0x7fdf601e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdf600a1530 0x7fdf601e9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 --2- 172.21.15.81:0/111315814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf48004800 0x7fdf6009e5a0 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 -- 172.21.15.81:0/111315814 >> 172.21.15.81:0/111315814 conn(0x7fdf600b56d0 msgr2=0x7fdf601e6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:23.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 -- 172.21.15.81:0/111315814 shutdown_connections 2024-06-29T15:58:23.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.686+0000 7fdf65f94640 1 -- 172.21.15.81:0/111315814 wait complete. 2024-06-29T15:58:23.699 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-06-29T15:58:23.344868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":422,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272,"ondisk_log_size":272,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":437400,"kb_used_data":24144,"kb_used_omap":12,"kb_used_meta":413107,"kb_avail":754537320,"statfs":{"total":773094113280,"available":772646215680,"internally_reserved":0,"allocated":24723456,"data_stored":12724388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":423022130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001251"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707675+0000","last_change":"2024-06-29T15:58:19.306321+0000","last_active":"2024-06-29T15:58:20.707675+0000","last_peered":"2024-06-29T15:58:20.707675+0000","last_clean":"2024-06-29T15:58:20.707675+0000","last_became_active":"2024-06-29T15:58:19.306018+0000","last_became_peered":"2024-06-29T15:58:19.306018+0000","last_unstale":"2024-06-29T15:58:20.707675+0000","last_undegraded":"2024-06-29T15:58:20.707675+0000","last_fullsized":"2024-06-29T15:58:20.707675+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:42.361706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707672+0000","last_change":"2024-06-29T15:58:19.312428+0000","last_active":"2024-06-29T15:58:20.707672+0000","last_peered":"2024-06-29T15:58:20.707672+0000","last_clean":"2024-06-29T15:58:20.707672+0000","last_became_active":"2024-06-29T15:58:19.312036+0000","last_became_peered":"2024-06-29T15:58:19.312036+0000","last_unstale":"2024-06-29T15:58:20.707672+0000","last_undegraded":"2024-06-29T15:58:20.707672+0000","last_fullsized":"2024-06-29T15:58:20.707672+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:04.132174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304222+0000","last_change":"2024-06-29T15:58:19.315147+0000","last_active":"2024-06-29T15:58:20.304222+0000","last_peered":"2024-06-29T15:58:20.304222+0000","last_clean":"2024-06-29T15:58:20.304222+0000","last_became_active":"2024-06-29T15:58:19.314838+0000","last_became_peered":"2024-06-29T15:58:19.314838+0000","last_unstale":"2024-06-29T15:58:20.304222+0000","last_undegraded":"2024-06-29T15:58:20.304222+0000","last_fullsized":"2024-06-29T15:58:20.304222+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:33:39.763837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764272+0000","last_change":"2024-06-29T15:58:19.297774+0000","last_active":"2024-06-29T15:58:22.764272+0000","last_peered":"2024-06-29T15:58:22.764272+0000","last_clean":"2024-06-29T15:58:22.764272+0000","last_became_active":"2024-06-29T15:58:19.297463+0000","last_became_peered":"2024-06-29T15:58:19.297463+0000","last_unstale":"2024-06-29T15:58:22.764272+0000","last_undegraded":"2024-06-29T15:58:22.764272+0000","last_fullsized":"2024-06-29T15:58:22.764272+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:38.755714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.306478+0000","last_change":"2024-06-29T15:58:19.313121+0000","last_active":"2024-06-29T15:58:20.306478+0000","last_peered":"2024-06-29T15:58:20.306478+0000","last_clean":"2024-06-29T15:58:20.306478+0000","last_became_active":"2024-06-29T15:58:19.312783+0000","last_became_peered":"2024-06-29T15:58:19.312783+0000","last_unstale":"2024-06-29T15:58:20.306478+0000","last_undegraded":"2024-06-29T15:58:20.306478+0000","last_fullsized":"2024-06-29T15:58:20.306478+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:00:42.731203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764239+0000","last_change":"2024-06-29T15:58:19.302569+0000","last_active":"2024-06-29T15:58:22.764239+0000","last_peered":"2024-06-29T15:58:22.764239+0000","last_clean":"2024-06-29T15:58:22.764239+0000","last_became_active":"2024-06-29T15:58:19.302261+0000","last_became_peered":"2024-06-29T15:58:19.302261+0000","last_unstale":"2024-06-29T15:58:22.764239+0000","last_undegraded":"2024-06-29T15:58:22.764239+0000","last_fullsized":"2024-06-29T15:58:22.764239+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:44:48.771783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307553+0000","last_change":"2024-06-29T15:58:19.314579+0000","last_active":"2024-06-29T15:58:20.307553+0000","last_peered":"2024-06-29T15:58:20.307553+0000","last_clean":"2024-06-29T15:58:20.307553+0000","last_became_active":"2024-06-29T15:58:19.314282+0000","last_became_peered":"2024-06-29T15:58:19.314282+0000","last_unstale":"2024-06-29T15:58:20.307553+0000","last_undegraded":"2024-06-29T15:58:20.307553+0000","last_fullsized":"2024-06-29T15:58:20.307553+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:10.919759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829338+0000","last_change":"2024-06-29T15:58:19.303095+0000","last_active":"2024-06-29T15:58:21.829338+0000","last_peered":"2024-06-29T15:58:21.829338+0000","last_clean":"2024-06-29T15:58:21.829338+0000","last_became_active":"2024-06-29T15:58:19.302658+0000","last_became_peered":"2024-06-29T15:58:19.302658+0000","last_unstale":"2024-06-29T15:58:21.829338+0000","last_undegraded":"2024-06-29T15:58:21.829338+0000","last_fullsized":"2024-06-29T15:58:21.829338+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:22.896379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764869+0000","last_change":"2024-06-29T15:58:19.310261+0000","last_active":"2024-06-29T15:58:22.764869+0000","last_peered":"2024-06-29T15:58:22.764869+0000","last_clean":"2024-06-29T15:58:22.764869+0000","last_became_active":"2024-06-29T15:58:19.309742+0000","last_became_peered":"2024-06-29T15:58:19.309742+0000","last_unstale":"2024-06-29T15:58:22.764869+0000","last_undegraded":"2024-06-29T15:58:22.764869+0000","last_fullsized":"2024-06-29T15:58:22.764869+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:55:23.521010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.306999+0000","last_change":"2024-06-29T15:58:19.315434+0000","last_active":"2024-06-29T15:58:20.306999+0000","last_peered":"2024-06-29T15:58:20.306999+0000","last_clean":"2024-06-29T15:58:20.306999+0000","last_became_active":"2024-06-29T15:58:19.315065+0000","last_became_peered":"2024-06-29T15:58:19.315065+0000","last_unstale":"2024-06-29T15:58:20.306999+0000","last_undegraded":"2024-06-29T15:58:20.306999+0000","last_fullsized":"2024-06-29T15:58:20.306999+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:21.882892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764259+0000","last_change":"2024-06-29T15:58:19.312563+0000","last_active":"2024-06-29T15:58:22.764259+0000","last_peered":"2024-06-29T15:58:22.764259+0000","last_clean":"2024-06-29T15:58:22.764259+0000","last_became_active":"2024-06-29T15:58:19.311751+0000","last_became_peered":"2024-06-29T15:58:19.311751+0000","last_unstale":"2024-06-29T15:58:22.764259+0000","last_undegraded":"2024-06-29T15:58:22.764259+0000","last_fullsized":"2024-06-29T15:58:22.764259+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:19.583540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829945+0000","last_change":"2024-06-29T15:58:19.303967+0000","last_active":"2024-06-29T15:58:21.829945+0000","last_peered":"2024-06-29T15:58:21.829945+0000","last_clean":"2024-06-29T15:58:21.829945+0000","last_became_active":"2024-06-29T15:58:19.303620+0000","last_became_peered":"2024-06-29T15:58:19.303620+0000","last_unstale":"2024-06-29T15:58:21.829945+0000","last_undegraded":"2024-06-29T15:58:21.829945+0000","last_fullsized":"2024-06-29T15:58:21.829945+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:17:57.874256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307497+0000","last_change":"2024-06-29T15:58:19.317397+0000","last_active":"2024-06-29T15:58:20.307497+0000","last_peered":"2024-06-29T15:58:20.307497+0000","last_clean":"2024-06-29T15:58:20.307497+0000","last_became_active":"2024-06-29T15:58:19.317110+0000","last_became_peered":"2024-06-29T15:58:19.317110+0000","last_unstale":"2024-06-29T15:58:20.307497+0000","last_undegraded":"2024-06-29T15:58:20.307497+0000","last_fullsized":"2024-06-29T15:58:20.307497+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:33.637292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707368+0000","last_change":"2024-06-29T15:58:19.308273+0000","last_active":"2024-06-29T15:58:20.707368+0000","last_peered":"2024-06-29T15:58:20.707368+0000","last_clean":"2024-06-29T15:58:20.707368+0000","last_became_active":"2024-06-29T15:58:19.307956+0000","last_became_peered":"2024-06-29T15:58:19.307956+0000","last_unstale":"2024-06-29T15:58:20.707368+0000","last_undegraded":"2024-06-29T15:58:20.707368+0000","last_fullsized":"2024-06-29T15:58:20.707368+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:48.781692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764610+0000","last_change":"2024-06-29T15:58:19.300026+0000","last_active":"2024-06-29T15:58:22.764610+0000","last_peered":"2024-06-29T15:58:22.764610+0000","last_clean":"2024-06-29T15:58:22.764610+0000","last_became_active":"2024-06-29T15:58:19.298956+0000","last_became_peered":"2024-06-29T15:58:19.298956+0000","last_unstale":"2024-06-29T15:58:22.764610+0000","last_undegraded":"2024-06-29T15:58:22.764610+0000","last_fullsized":"2024-06-29T15:58:22.764610+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:10.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.303788+0000","last_change":"2024-06-29T15:58:19.308922+0000","last_active":"2024-06-29T15:58:20.303788+0000","last_peered":"2024-06-29T15:58:20.303788+0000","last_clean":"2024-06-29T15:58:20.303788+0000","last_became_active":"2024-06-29T15:58:19.308502+0000","last_became_peered":"2024-06-29T15:58:19.308502+0000","last_unstale":"2024-06-29T15:58:20.303788+0000","last_undegraded":"2024-06-29T15:58:20.303788+0000","last_fullsized":"2024-06-29T15:58:20.303788+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:14.372622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829928+0000","last_change":"2024-06-29T15:58:19.309759+0000","last_active":"2024-06-29T15:58:21.829928+0000","last_peered":"2024-06-29T15:58:21.829928+0000","last_clean":"2024-06-29T15:58:21.829928+0000","last_became_active":"2024-06-29T15:58:19.309439+0000","last_became_peered":"2024-06-29T15:58:19.309439+0000","last_unstale":"2024-06-29T15:58:21.829928+0000","last_undegraded":"2024-06-29T15:58:21.829928+0000","last_fullsized":"2024-06-29T15:58:21.829928+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:54.499719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707693+0000","last_change":"2024-06-29T15:58:19.306249+0000","last_active":"2024-06-29T15:58:20.707693+0000","last_peered":"2024-06-29T15:58:20.707693+0000","last_clean":"2024-06-29T15:58:20.707693+0000","last_became_active":"2024-06-29T15:58:19.305935+0000","last_became_peered":"2024-06-29T15:58:19.305935+0000","last_unstale":"2024-06-29T15:58:20.707693+0000","last_undegraded":"2024-06-29T15:58:20.707693+0000","last_fullsized":"2024-06-29T15:58:20.707693+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:52.246043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829887+0000","last_change":"2024-06-29T15:58:19.313204+0000","last_active":"2024-06-29T15:58:21.829887+0000","last_peered":"2024-06-29T15:58:21.829887+0000","last_clean":"2024-06-29T15:58:21.829887+0000","last_became_active":"2024-06-29T15:58:19.312752+0000","last_became_peered":"2024-06-29T15:58:19.312752+0000","last_unstale":"2024-06-29T15:58:21.829887+0000","last_undegraded":"2024-06-29T15:58:21.829887+0000","last_fullsized":"2024-06-29T15:58:21.829887+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:16.564078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304093+0000","last_change":"2024-06-29T15:58:19.303230+0000","last_active":"2024-06-29T15:58:20.304093+0000","last_peered":"2024-06-29T15:58:20.304093+0000","last_clean":"2024-06-29T15:58:20.304093+0000","last_became_active":"2024-06-29T15:58:19.302904+0000","last_became_peered":"2024-06-29T15:58:19.302904+0000","last_unstale":"2024-06-29T15:58:20.304093+0000","last_undegraded":"2024-06-29T15:58:20.304093+0000","last_fullsized":"2024-06-29T15:58:20.304093+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:30:16.162326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829634+0000","last_change":"2024-06-29T15:58:19.302093+0000","last_active":"2024-06-29T15:58:21.829634+0000","last_peered":"2024-06-29T15:58:21.829634+0000","last_clean":"2024-06-29T15:58:21.829634+0000","last_became_active":"2024-06-29T15:58:19.301677+0000","last_became_peered":"2024-06-29T15:58:19.301677+0000","last_unstale":"2024-06-29T15:58:21.829634+0000","last_undegraded":"2024-06-29T15:58:21.829634+0000","last_fullsized":"2024-06-29T15:58:21.829634+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:08.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307654+0000","last_change":"2024-06-29T15:58:19.307655+0000","last_active":"2024-06-29T15:58:20.307654+0000","last_peered":"2024-06-29T15:58:20.307654+0000","last_clean":"2024-06-29T15:58:20.307654+0000","last_became_active":"2024-06-29T15:58:19.307369+0000","last_became_peered":"2024-06-29T15:58:19.307369+0000","last_unstale":"2024-06-29T15:58:20.307654+0000","last_undegraded":"2024-06-29T15:58:20.307654+0000","last_fullsized":"2024-06-29T15:58:20.307654+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:04.132174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829396+0000","last_change":"2024-06-29T15:58:19.308486+0000","last_active":"2024-06-29T15:58:21.829396+0000","last_peered":"2024-06-29T15:58:21.829396+0000","last_clean":"2024-06-29T15:58:21.829396+0000","last_became_active":"2024-06-29T15:58:19.308158+0000","last_became_peered":"2024-06-29T15:58:19.308158+0000","last_unstale":"2024-06-29T15:58:21.829396+0000","last_undegraded":"2024-06-29T15:58:21.829396+0000","last_fullsized":"2024-06-29T15:58:21.829396+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:38.755714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707371+0000","last_change":"2024-06-29T15:58:19.312754+0000","last_active":"2024-06-29T15:58:20.707371+0000","last_peered":"2024-06-29T15:58:20.707371+0000","last_clean":"2024-06-29T15:58:20.707371+0000","last_became_active":"2024-06-29T15:58:19.312061+0000","last_became_peered":"2024-06-29T15:58:19.312061+0000","last_unstale":"2024-06-29T15:58:20.707371+0000","last_undegraded":"2024-06-29T15:58:20.707371+0000","last_fullsized":"2024-06-29T15:58:20.707371+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:10.919759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764672+0000","last_change":"2024-06-29T15:58:19.295810+0000","last_active":"2024-06-29T15:58:22.764672+0000","last_peered":"2024-06-29T15:58:22.764672+0000","last_clean":"2024-06-29T15:58:22.764672+0000","last_became_active":"2024-06-29T15:58:19.295445+0000","last_became_peered":"2024-06-29T15:58:19.295445+0000","last_unstale":"2024-06-29T15:58:22.764672+0000","last_undegraded":"2024-06-29T15:58:22.764672+0000","last_fullsized":"2024-06-29T15:58:22.764672+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:49:49.141794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307436+0000","last_change":"2024-06-29T15:58:19.315083+0000","last_active":"2024-06-29T15:58:20.307436+0000","last_peered":"2024-06-29T15:58:20.307436+0000","last_clean":"2024-06-29T15:58:20.307436+0000","last_became_active":"2024-06-29T15:58:19.313876+0000","last_became_peered":"2024-06-29T15:58:19.313876+0000","last_unstale":"2024-06-29T15:58:20.307436+0000","last_undegraded":"2024-06-29T15:58:20.307436+0000","last_fullsized":"2024-06-29T15:58:20.307436+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:57.766748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764428+0000","last_change":"2024-06-29T15:58:19.309408+0000","last_active":"2024-06-29T15:58:22.764428+0000","last_peered":"2024-06-29T15:58:22.764428+0000","last_clean":"2024-06-29T15:58:22.764428+0000","last_became_active":"2024-06-29T15:58:19.309050+0000","last_became_peered":"2024-06-29T15:58:19.309050+0000","last_unstale":"2024-06-29T15:58:22.764428+0000","last_undegraded":"2024-06-29T15:58:22.764428+0000","last_fullsized":"2024-06-29T15:58:22.764428+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:53.490742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707629+0000","last_change":"2024-06-29T15:58:19.305539+0000","last_active":"2024-06-29T15:58:20.707629+0000","last_peered":"2024-06-29T15:58:20.707629+0000","last_clean":"2024-06-29T15:58:20.707629+0000","last_became_active":"2024-06-29T15:58:19.305224+0000","last_became_peered":"2024-06-29T15:58:19.305224+0000","last_unstale":"2024-06-29T15:58:20.707629+0000","last_undegraded":"2024-06-29T15:58:20.707629+0000","last_fullsized":"2024-06-29T15:58:20.707629+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:21.381169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.765514+0000","last_change":"2024-06-29T15:58:19.299124+0000","last_active":"2024-06-29T15:58:22.765514+0000","last_peered":"2024-06-29T15:58:22.765514+0000","last_clean":"2024-06-29T15:58:22.765514+0000","last_became_active":"2024-06-29T15:58:19.298839+0000","last_became_peered":"2024-06-29T15:58:19.298839+0000","last_unstale":"2024-06-29T15:58:22.765514+0000","last_undegraded":"2024-06-29T15:58:22.765514+0000","last_fullsized":"2024-06-29T15:58:22.765514+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:25:53.956024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307134+0000","last_change":"2024-06-29T15:58:19.298224+0000","last_active":"2024-06-29T15:58:20.307134+0000","last_peered":"2024-06-29T15:58:20.307134+0000","last_clean":"2024-06-29T15:58:20.307134+0000","last_became_active":"2024-06-29T15:58:19.297909+0000","last_became_peered":"2024-06-29T15:58:19.297909+0000","last_unstale":"2024-06-29T15:58:20.307134+0000","last_undegraded":"2024-06-29T15:58:20.307134+0000","last_fullsized":"2024-06-29T15:58:20.307134+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:17.158060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764055+0000","last_change":"2024-06-29T15:58:19.304924+0000","last_active":"2024-06-29T15:58:22.764055+0000","last_peered":"2024-06-29T15:58:22.764055+0000","last_clean":"2024-06-29T15:58:22.764055+0000","last_became_active":"2024-06-29T15:58:19.304317+0000","last_became_peered":"2024-06-29T15:58:19.304317+0000","last_unstale":"2024-06-29T15:58:22.764055+0000","last_undegraded":"2024-06-29T15:58:22.764055+0000","last_fullsized":"2024-06-29T15:58:22.764055+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:15:28.738310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707541+0000","last_change":"2024-06-29T15:58:19.309529+0000","last_active":"2024-06-29T15:58:20.707541+0000","last_peered":"2024-06-29T15:58:20.707541+0000","last_clean":"2024-06-29T15:58:20.707541+0000","last_became_active":"2024-06-29T15:58:19.309193+0000","last_became_peered":"2024-06-29T15:58:19.309193+0000","last_unstale":"2024-06-29T15:58:20.707541+0000","last_undegraded":"2024-06-29T15:58:20.707541+0000","last_fullsized":"2024-06-29T15:58:20.707541+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:21.882892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304099+0000","last_change":"2024-06-29T15:58:19.306514+0000","last_active":"2024-06-29T15:58:20.304099+0000","last_peered":"2024-06-29T15:58:20.304099+0000","last_clean":"2024-06-29T15:58:20.304099+0000","last_became_active":"2024-06-29T15:58:19.306099+0000","last_became_peered":"2024-06-29T15:58:19.306099+0000","last_unstale":"2024-06-29T15:58:20.304099+0000","last_undegraded":"2024-06-29T15:58:20.304099+0000","last_fullsized":"2024-06-29T15:58:20.304099+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:25.572276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"232'1","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307888+0000","last_change":"2024-06-29T15:58:18.301939+0000","last_active":"2024-06-29T15:58:20.307888+0000","last_peered":"2024-06-29T15:58:20.307888+0000","last_clean":"2024-06-29T15:58:20.307888+0000","last_became_active":"2024-06-29T15:58:18.301258+0000","last_became_peered":"2024-06-29T15:58:18.301258+0000","last_unstale":"2024-06-29T15:58:20.307888+0000","last_undegraded":"2024-06-29T15:58:20.307888+0000","last_fullsized":"2024-06-29T15:58:20.307888+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:53.497595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764421+0000","last_change":"2024-06-29T15:58:19.298015+0000","last_active":"2024-06-29T15:58:22.764421+0000","last_peered":"2024-06-29T15:58:22.764421+0000","last_clean":"2024-06-29T15:58:22.764421+0000","last_became_active":"2024-06-29T15:58:19.297706+0000","last_became_peered":"2024-06-29T15:58:19.297706+0000","last_unstale":"2024-06-29T15:58:22.764421+0000","last_undegraded":"2024-06-29T15:58:22.764421+0000","last_fullsized":"2024-06-29T15:58:22.764421+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:59:36.737724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"232'2","reported_seq":16,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304323+0000","last_change":"2024-06-29T15:58:18.297534+0000","last_active":"2024-06-29T15:58:20.304323+0000","last_peered":"2024-06-29T15:58:20.304323+0000","last_clean":"2024-06-29T15:58:20.304323+0000","last_became_active":"2024-06-29T15:58:18.297260+0000","last_became_peered":"2024-06-29T15:58:18.297260+0000","last_unstale":"2024-06-29T15:58:20.304323+0000","last_undegraded":"2024-06-29T15:58:20.304323+0000","last_fullsized":"2024-06-29T15:58:20.304323+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:47:47.544183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829977+0000","last_change":"2024-06-29T15:58:19.305103+0000","last_active":"2024-06-29T15:58:21.829977+0000","last_peered":"2024-06-29T15:58:21.829977+0000","last_clean":"2024-06-29T15:58:21.829977+0000","last_became_active":"2024-06-29T15:58:19.304612+0000","last_became_peered":"2024-06-29T15:58:19.304612+0000","last_unstale":"2024-06-29T15:58:21.829977+0000","last_undegraded":"2024-06-29T15:58:21.829977+0000","last_fullsized":"2024-06-29T15:58:21.829977+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:19.583540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"232'2","reported_seq":16,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304363+0000","last_change":"2024-06-29T15:58:18.302160+0000","last_active":"2024-06-29T15:58:20.304363+0000","last_peered":"2024-06-29T15:58:20.304363+0000","last_clean":"2024-06-29T15:58:20.304363+0000","last_became_active":"2024-06-29T15:58:18.301895+0000","last_became_peered":"2024-06-29T15:58:18.301895+0000","last_unstale":"2024-06-29T15:58:20.304363+0000","last_undegraded":"2024-06-29T15:58:20.304363+0000","last_fullsized":"2024-06-29T15:58:20.304363+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:26.756484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764376+0000","last_change":"2024-06-29T15:58:19.299884+0000","last_active":"2024-06-29T15:58:22.764376+0000","last_peered":"2024-06-29T15:58:22.764376+0000","last_clean":"2024-06-29T15:58:22.764376+0000","last_became_active":"2024-06-29T15:58:19.299621+0000","last_became_peered":"2024-06-29T15:58:19.299621+0000","last_unstale":"2024-06-29T15:58:22.764376+0000","last_undegraded":"2024-06-29T15:58:22.764376+0000","last_fullsized":"2024-06-29T15:58:22.764376+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:09.325434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"232'1","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764077+0000","last_change":"2024-06-29T15:58:18.299460+0000","last_active":"2024-06-29T15:58:22.764077+0000","last_peered":"2024-06-29T15:58:22.764077+0000","last_clean":"2024-06-29T15:58:22.764077+0000","last_became_active":"2024-06-29T15:58:18.299171+0000","last_became_peered":"2024-06-29T15:58:18.299171+0000","last_unstale":"2024-06-29T15:58:22.764077+0000","last_undegraded":"2024-06-29T15:58:22.764077+0000","last_fullsized":"2024-06-29T15:58:22.764077+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:34:40.426770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764607+0000","last_change":"2024-06-29T15:58:19.306300+0000","last_active":"2024-06-29T15:58:22.764607+0000","last_peered":"2024-06-29T15:58:22.764607+0000","last_clean":"2024-06-29T15:58:22.764607+0000","last_became_active":"2024-06-29T15:58:19.305976+0000","last_became_peered":"2024-06-29T15:58:19.305976+0000","last_unstale":"2024-06-29T15:58:22.764607+0000","last_undegraded":"2024-06-29T15:58:22.764607+0000","last_fullsized":"2024-06-29T15:58:22.764607+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:21.792399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"232'1","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707298+0000","last_change":"2024-06-29T15:58:18.297242+0000","last_active":"2024-06-29T15:58:20.707298+0000","last_peered":"2024-06-29T15:58:20.707298+0000","last_clean":"2024-06-29T15:58:20.707298+0000","last_became_active":"2024-06-29T15:58:18.296876+0000","last_became_peered":"2024-06-29T15:58:18.296876+0000","last_unstale":"2024-06-29T15:58:20.707298+0000","last_undegraded":"2024-06-29T15:58:20.707298+0000","last_fullsized":"2024-06-29T15:58:20.707298+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:53.497595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.303800+0000","last_change":"2024-06-29T15:58:19.300049+0000","last_active":"2024-06-29T15:58:20.303800+0000","last_peered":"2024-06-29T15:58:20.303800+0000","last_clean":"2024-06-29T15:58:20.303800+0000","last_became_active":"2024-06-29T15:58:19.299756+0000","last_became_peered":"2024-06-29T15:58:19.299756+0000","last_unstale":"2024-06-29T15:58:20.303800+0000","last_undegraded":"2024-06-29T15:58:20.303800+0000","last_fullsized":"2024-06-29T15:58:20.303800+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:25:50.310157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307740+0000","last_change":"2024-06-29T15:58:18.302669+0000","last_active":"2024-06-29T15:58:20.307740+0000","last_peered":"2024-06-29T15:58:20.307740+0000","last_clean":"2024-06-29T15:58:20.307740+0000","last_became_active":"2024-06-29T15:58:18.302098+0000","last_became_peered":"2024-06-29T15:58:18.302098+0000","last_unstale":"2024-06-29T15:58:20.307740+0000","last_undegraded":"2024-06-29T15:58:20.307740+0000","last_fullsized":"2024-06-29T15:58:20.307740+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:18.581416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707349+0000","last_change":"2024-06-29T15:58:19.308028+0000","last_active":"2024-06-29T15:58:20.707349+0000","last_peered":"2024-06-29T15:58:20.707349+0000","last_clean":"2024-06-29T15:58:20.707349+0000","last_became_active":"2024-06-29T15:58:19.306593+0000","last_became_peered":"2024-06-29T15:58:19.306593+0000","last_unstale":"2024-06-29T15:58:20.707349+0000","last_undegraded":"2024-06-29T15:58:20.707349+0000","last_fullsized":"2024-06-29T15:58:20.707349+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:45.907895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.303917+0000","last_change":"2024-06-29T15:58:18.300805+0000","last_active":"2024-06-29T15:58:20.303917+0000","last_peered":"2024-06-29T15:58:20.303917+0000","last_clean":"2024-06-29T15:58:20.303917+0000","last_became_active":"2024-06-29T15:58:18.300515+0000","last_became_peered":"2024-06-29T15:58:18.300515+0000","last_unstale":"2024-06-29T15:58:20.303917+0000","last_undegraded":"2024-06-29T15:58:20.303917+0000","last_fullsized":"2024-06-29T15:58:20.303917+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:14.605715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707810+0000","last_change":"2024-06-29T15:58:19.303106+0000","last_active":"2024-06-29T15:58:20.707810+0000","last_peered":"2024-06-29T15:58:20.707810+0000","last_clean":"2024-06-29T15:58:20.707810+0000","last_became_active":"2024-06-29T15:58:19.302806+0000","last_became_peered":"2024-06-29T15:58:19.302806+0000","last_unstale":"2024-06-29T15:58:20.707810+0000","last_undegraded":"2024-06-29T15:58:20.707810+0000","last_fullsized":"2024-06-29T15:58:20.707810+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:18.440072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829847+0000","last_change":"2024-06-29T15:58:18.303872+0000","last_active":"2024-06-29T15:58:21.829847+0000","last_peered":"2024-06-29T15:58:21.829847+0000","last_clean":"2024-06-29T15:58:21.829847+0000","last_became_active":"2024-06-29T15:58:18.303542+0000","last_became_peered":"2024-06-29T15:58:18.303542+0000","last_unstale":"2024-06-29T15:58:21.829847+0000","last_undegraded":"2024-06-29T15:58:21.829847+0000","last_fullsized":"2024-06-29T15:58:21.829847+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:04.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764517+0000","last_change":"2024-06-29T15:58:19.295640+0000","last_active":"2024-06-29T15:58:22.764517+0000","last_peered":"2024-06-29T15:58:22.764517+0000","last_clean":"2024-06-29T15:58:22.764517+0000","last_became_active":"2024-06-29T15:58:19.295239+0000","last_became_peered":"2024-06-29T15:58:19.295239+0000","last_unstale":"2024-06-29T15:58:22.764517+0000","last_undegraded":"2024-06-29T15:58:22.764517+0000","last_fullsized":"2024-06-29T15:58:22.764517+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:19.616206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304032+0000","last_change":"2024-06-29T15:58:18.294635+0000","last_active":"2024-06-29T15:58:20.304032+0000","last_peered":"2024-06-29T15:58:20.304032+0000","last_clean":"2024-06-29T15:58:20.304032+0000","last_became_active":"2024-06-29T15:58:18.294343+0000","last_became_peered":"2024-06-29T15:58:18.294343+0000","last_unstale":"2024-06-29T15:58:20.304032+0000","last_undegraded":"2024-06-29T15:58:20.304032+0000","last_fullsized":"2024-06-29T15:58:20.304032+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:44.439934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304254+0000","last_change":"2024-06-29T15:58:19.306811+0000","last_active":"2024-06-29T15:58:20.304254+0000","last_peered":"2024-06-29T15:58:20.304254+0000","last_clean":"2024-06-29T15:58:20.304254+0000","last_became_active":"2024-06-29T15:58:19.306354+0000","last_became_peered":"2024-06-29T15:58:19.306354+0000","last_unstale":"2024-06-29T15:58:20.304254+0000","last_undegraded":"2024-06-29T15:58:20.304254+0000","last_fullsized":"2024-06-29T15:58:20.304254+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:55.684049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"232'3","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707561+0000","last_change":"2024-06-29T15:58:18.300841+0000","last_active":"2024-06-29T15:58:20.707561+0000","last_peered":"2024-06-29T15:58:20.707561+0000","last_clean":"2024-06-29T15:58:20.707561+0000","last_became_active":"2024-06-29T15:58:18.300525+0000","last_became_peered":"2024-06-29T15:58:18.300525+0000","last_unstale":"2024-06-29T15:58:20.707561+0000","last_undegraded":"2024-06-29T15:58:20.707561+0000","last_fullsized":"2024-06-29T15:58:20.707561+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:15.561954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307511+0000","last_change":"2024-06-29T15:58:19.310218+0000","last_active":"2024-06-29T15:58:20.307511+0000","last_peered":"2024-06-29T15:58:20.307511+0000","last_clean":"2024-06-29T15:58:20.307511+0000","last_became_active":"2024-06-29T15:58:19.309910+0000","last_became_peered":"2024-06-29T15:58:19.309910+0000","last_unstale":"2024-06-29T15:58:20.307511+0000","last_undegraded":"2024-06-29T15:58:20.307511+0000","last_fullsized":"2024-06-29T15:58:20.307511+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:48.781692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"232'6","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707515+0000","last_change":"2024-06-29T15:58:18.296167+0000","last_active":"2024-06-29T15:58:20.707515+0000","last_peered":"2024-06-29T15:58:20.707515+0000","last_clean":"2024-06-29T15:58:20.707515+0000","last_became_active":"2024-06-29T15:58:18.294984+0000","last_became_peered":"2024-06-29T15:58:18.294984+0000","last_unstale":"2024-06-29T15:58:20.707515+0000","last_undegraded":"2024-06-29T15:58:20.707515+0000","last_fullsized":"2024-06-29T15:58:20.707515+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:07.733361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307759+0000","last_change":"2024-06-29T15:58:19.315403+0000","last_active":"2024-06-29T15:58:20.307759+0000","last_peered":"2024-06-29T15:58:20.307759+0000","last_clean":"2024-06-29T15:58:20.307759+0000","last_became_active":"2024-06-29T15:58:19.314969+0000","last_became_peered":"2024-06-29T15:58:19.314969+0000","last_unstale":"2024-06-29T15:58:20.307759+0000","last_undegraded":"2024-06-29T15:58:20.307759+0000","last_fullsized":"2024-06-29T15:58:20.307759+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:42.361706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"232'2","reported_seq":16,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707471+0000","last_change":"2024-06-29T15:58:18.303307+0000","last_active":"2024-06-29T15:58:20.707471+0000","last_peered":"2024-06-29T15:58:20.707471+0000","last_clean":"2024-06-29T15:58:20.707471+0000","last_became_active":"2024-06-29T15:58:18.303009+0000","last_became_peered":"2024-06-29T15:58:18.303009+0000","last_unstale":"2024-06-29T15:58:20.707471+0000","last_undegraded":"2024-06-29T15:58:20.707471+0000","last_fullsized":"2024-06-29T15:58:20.707471+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:08.507147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764649+0000","last_change":"2024-06-29T15:58:19.304892+0000","last_active":"2024-06-29T15:58:22.764649+0000","last_peered":"2024-06-29T15:58:22.764649+0000","last_clean":"2024-06-29T15:58:22.764649+0000","last_became_active":"2024-06-29T15:58:19.304368+0000","last_became_peered":"2024-06-29T15:58:19.304368+0000","last_unstale":"2024-06-29T15:58:22.764649+0000","last_undegraded":"2024-06-29T15:58:22.764649+0000","last_fullsized":"2024-06-29T15:58:22.764649+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:34.621330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304210+0000","last_change":"2024-06-29T15:58:18.290469+0000","last_active":"2024-06-29T15:58:20.304210+0000","last_peered":"2024-06-29T15:58:20.304210+0000","last_clean":"2024-06-29T15:58:20.304210+0000","last_became_active":"2024-06-29T15:58:18.290177+0000","last_became_peered":"2024-06-29T15:58:18.290177+0000","last_unstale":"2024-06-29T15:58:20.304210+0000","last_undegraded":"2024-06-29T15:58:20.304210+0000","last_fullsized":"2024-06-29T15:58:20.304210+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:51.797075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.763943+0000","last_change":"2024-06-29T15:58:19.304069+0000","last_active":"2024-06-29T15:58:22.763943+0000","last_peered":"2024-06-29T15:58:22.763943+0000","last_clean":"2024-06-29T15:58:22.763943+0000","last_became_active":"2024-06-29T15:58:19.303758+0000","last_became_peered":"2024-06-29T15:58:19.303758+0000","last_unstale":"2024-06-29T15:58:22.763943+0000","last_undegraded":"2024-06-29T15:58:22.763943+0000","last_fullsized":"2024-06-29T15:58:22.763943+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:51.378080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307796+0000","last_change":"2024-06-29T15:58:18.303515+0000","last_active":"2024-06-29T15:58:20.307796+0000","last_peered":"2024-06-29T15:58:20.307796+0000","last_clean":"2024-06-29T15:58:20.307796+0000","last_became_active":"2024-06-29T15:58:18.303008+0000","last_became_peered":"2024-06-29T15:58:18.303008+0000","last_unstale":"2024-06-29T15:58:20.307796+0000","last_undegraded":"2024-06-29T15:58:20.307796+0000","last_fullsized":"2024-06-29T15:58:20.307796+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:08.323310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307637+0000","last_change":"2024-06-29T15:58:19.316203+0000","last_active":"2024-06-29T15:58:20.307637+0000","last_peered":"2024-06-29T15:58:20.307637+0000","last_clean":"2024-06-29T15:58:20.307637+0000","last_became_active":"2024-06-29T15:58:19.315777+0000","last_became_peered":"2024-06-29T15:58:19.315777+0000","last_unstale":"2024-06-29T15:58:20.307637+0000","last_undegraded":"2024-06-29T15:58:20.307637+0000","last_fullsized":"2024-06-29T15:58:20.307637+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:21.381169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"232'8","reported_seq":22,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764859+0000","last_change":"2024-06-29T15:58:18.302451+0000","last_active":"2024-06-29T15:58:22.764859+0000","last_peered":"2024-06-29T15:58:22.764859+0000","last_clean":"2024-06-29T15:58:22.764859+0000","last_became_active":"2024-06-29T15:58:18.302144+0000","last_became_peered":"2024-06-29T15:58:18.302144+0000","last_unstale":"2024-06-29T15:58:22.764859+0000","last_undegraded":"2024-06-29T15:58:22.764859+0000","last_fullsized":"2024-06-29T15:58:22.764859+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:15:27.736186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.765097+0000","last_change":"2024-06-29T15:58:19.297900+0000","last_active":"2024-06-29T15:58:22.765097+0000","last_peered":"2024-06-29T15:58:22.765097+0000","last_clean":"2024-06-29T15:58:22.765097+0000","last_became_active":"2024-06-29T15:58:19.297605+0000","last_became_peered":"2024-06-29T15:58:19.297605+0000","last_unstale":"2024-06-29T15:58:22.765097+0000","last_undegraded":"2024-06-29T15:58:22.765097+0000","last_fullsized":"2024-06-29T15:58:22.765097+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:25.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"232'1","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307470+0000","last_change":"2024-06-29T15:58:18.303063+0000","last_active":"2024-06-29T15:58:20.307470+0000","last_peered":"2024-06-29T15:58:20.307470+0000","last_clean":"2024-06-29T15:58:20.307470+0000","last_became_active":"2024-06-29T15:58:18.302403+0000","last_became_peered":"2024-06-29T15:58:18.302403+0000","last_unstale":"2024-06-29T15:58:20.307470+0000","last_undegraded":"2024-06-29T15:58:20.307470+0000","last_fullsized":"2024-06-29T15:58:20.307470+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:59:35.735600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304292+0000","last_change":"2024-06-29T15:58:19.310412+0000","last_active":"2024-06-29T15:58:20.304292+0000","last_peered":"2024-06-29T15:58:20.304292+0000","last_clean":"2024-06-29T15:58:20.304292+0000","last_became_active":"2024-06-29T15:58:19.310109+0000","last_became_peered":"2024-06-29T15:58:19.310109+0000","last_unstale":"2024-06-29T15:58:20.304292+0000","last_undegraded":"2024-06-29T15:58:20.304292+0000","last_fullsized":"2024-06-29T15:58:20.304292+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:56.479901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"232'2","reported_seq":16,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304108+0000","last_change":"2024-06-29T15:58:18.296680+0000","last_active":"2024-06-29T15:58:20.304108+0000","last_peered":"2024-06-29T15:58:20.304108+0000","last_clean":"2024-06-29T15:58:20.304108+0000","last_became_active":"2024-06-29T15:58:18.296405+0000","last_became_peered":"2024-06-29T15:58:18.296405+0000","last_unstale":"2024-06-29T15:58:20.304108+0000","last_undegraded":"2024-06-29T15:58:20.304108+0000","last_fullsized":"2024-06-29T15:58:20.304108+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:02.671679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764599+0000","last_change":"2024-06-29T15:58:19.308630+0000","last_active":"2024-06-29T15:58:22.764599+0000","last_peered":"2024-06-29T15:58:22.764599+0000","last_clean":"2024-06-29T15:58:22.764599+0000","last_became_active":"2024-06-29T15:58:19.308323+0000","last_became_peered":"2024-06-29T15:58:19.308323+0000","last_unstale":"2024-06-29T15:58:22.764599+0000","last_undegraded":"2024-06-29T15:58:22.764599+0000","last_fullsized":"2024-06-29T15:58:22.764599+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:14:39.500064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"232'1","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764671+0000","last_change":"2024-06-29T15:58:18.294949+0000","last_active":"2024-06-29T15:58:22.764671+0000","last_peered":"2024-06-29T15:58:22.764671+0000","last_clean":"2024-06-29T15:58:22.764671+0000","last_became_active":"2024-06-29T15:58:18.294015+0000","last_became_peered":"2024-06-29T15:58:18.294015+0000","last_unstale":"2024-06-29T15:58:22.764671+0000","last_undegraded":"2024-06-29T15:58:22.764671+0000","last_fullsized":"2024-06-29T15:58:22.764671+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:36.977744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.830101+0000","last_change":"2024-06-29T15:58:19.305909+0000","last_active":"2024-06-29T15:58:21.830101+0000","last_peered":"2024-06-29T15:58:21.830101+0000","last_clean":"2024-06-29T15:58:21.830101+0000","last_became_active":"2024-06-29T15:58:19.305594+0000","last_became_peered":"2024-06-29T15:58:19.305594+0000","last_unstale":"2024-06-29T15:58:21.830101+0000","last_undegraded":"2024-06-29T15:58:21.830101+0000","last_fullsized":"2024-06-29T15:58:21.830101+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:09.325434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764761+0000","last_change":"2024-06-29T15:58:18.306207+0000","last_active":"2024-06-29T15:58:22.764761+0000","last_peered":"2024-06-29T15:58:22.764761+0000","last_clean":"2024-06-29T15:58:22.764761+0000","last_became_active":"2024-06-29T15:58:18.305661+0000","last_became_peered":"2024-06-29T15:58:18.305661+0000","last_unstale":"2024-06-29T15:58:22.764761+0000","last_undegraded":"2024-06-29T15:58:22.764761+0000","last_fullsized":"2024-06-29T15:58:22.764761+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:44:47.769658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829559+0000","last_change":"2024-06-29T15:58:19.307416+0000","last_active":"2024-06-29T15:58:21.829559+0000","last_peered":"2024-06-29T15:58:21.829559+0000","last_clean":"2024-06-29T15:58:21.829559+0000","last_became_active":"2024-06-29T15:58:19.307089+0000","last_became_peered":"2024-06-29T15:58:19.307089+0000","last_unstale":"2024-06-29T15:58:21.829559+0000","last_undegraded":"2024-06-29T15:58:21.829559+0000","last_fullsized":"2024-06-29T15:58:21.829559+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:54.355267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"232'2","reported_seq":16,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304055+0000","last_change":"2024-06-29T15:58:18.302497+0000","last_active":"2024-06-29T15:58:20.304055+0000","last_peered":"2024-06-29T15:58:20.304055+0000","last_clean":"2024-06-29T15:58:20.304055+0000","last_became_active":"2024-06-29T15:58:18.302200+0000","last_became_peered":"2024-06-29T15:58:18.302200+0000","last_unstale":"2024-06-29T15:58:20.304055+0000","last_undegraded":"2024-06-29T15:58:20.304055+0000","last_fullsized":"2024-06-29T15:58:20.304055+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:59.065708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707353+0000","last_change":"2024-06-29T15:58:19.313662+0000","last_active":"2024-06-29T15:58:20.707353+0000","last_peered":"2024-06-29T15:58:20.707353+0000","last_clean":"2024-06-29T15:58:20.707353+0000","last_became_active":"2024-06-29T15:58:19.313340+0000","last_became_peered":"2024-06-29T15:58:19.313340+0000","last_unstale":"2024-06-29T15:58:20.707353+0000","last_undegraded":"2024-06-29T15:58:20.707353+0000","last_fullsized":"2024-06-29T15:58:20.707353+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:57.766748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764493+0000","last_change":"2024-06-29T15:58:18.306269+0000","last_active":"2024-06-29T15:58:22.764493+0000","last_peered":"2024-06-29T15:58:22.764493+0000","last_clean":"2024-06-29T15:58:22.764493+0000","last_became_active":"2024-06-29T15:58:18.305098+0000","last_became_peered":"2024-06-29T15:58:18.305098+0000","last_unstale":"2024-06-29T15:58:22.764493+0000","last_undegraded":"2024-06-29T15:58:22.764493+0000","last_fullsized":"2024-06-29T15:58:22.764493+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:50.375956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764765+0000","last_change":"2024-06-29T15:58:19.308352+0000","last_active":"2024-06-29T15:58:22.764765+0000","last_peered":"2024-06-29T15:58:22.764765+0000","last_clean":"2024-06-29T15:58:22.764765+0000","last_became_active":"2024-06-29T15:58:19.308057+0000","last_became_peered":"2024-06-29T15:58:19.308057+0000","last_unstale":"2024-06-29T15:58:22.764765+0000","last_undegraded":"2024-06-29T15:58:22.764765+0000","last_fullsized":"2024-06-29T15:58:22.764765+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:25.510746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764649+0000","last_change":"2024-06-29T15:58:18.304832+0000","last_active":"2024-06-29T15:58:22.764649+0000","last_peered":"2024-06-29T15:58:22.764649+0000","last_clean":"2024-06-29T15:58:22.764649+0000","last_became_active":"2024-06-29T15:58:18.304230+0000","last_became_peered":"2024-06-29T15:58:18.304230+0000","last_unstale":"2024-06-29T15:58:22.764649+0000","last_undegraded":"2024-06-29T15:58:22.764649+0000","last_fullsized":"2024-06-29T15:58:22.764649+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:18.315156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829493+0000","last_change":"2024-06-29T15:58:19.306776+0000","last_active":"2024-06-29T15:58:21.829493+0000","last_peered":"2024-06-29T15:58:21.829493+0000","last_clean":"2024-06-29T15:58:21.829493+0000","last_became_active":"2024-06-29T15:58:19.306385+0000","last_became_peered":"2024-06-29T15:58:19.306385+0000","last_unstale":"2024-06-29T15:58:21.829493+0000","last_undegraded":"2024-06-29T15:58:21.829493+0000","last_fullsized":"2024-06-29T15:58:21.829493+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:25.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.306938+0000","last_change":"2024-06-29T15:58:18.297661+0000","last_active":"2024-06-29T15:58:20.306938+0000","last_peered":"2024-06-29T15:58:20.306938+0000","last_clean":"2024-06-29T15:58:20.306938+0000","last_became_active":"2024-06-29T15:58:18.297272+0000","last_became_peered":"2024-06-29T15:58:18.297272+0000","last_unstale":"2024-06-29T15:58:20.306938+0000","last_undegraded":"2024-06-29T15:58:20.306938+0000","last_fullsized":"2024-06-29T15:58:20.306938+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:17:56.872132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307823+0000","last_change":"2024-06-29T15:58:19.311825+0000","last_active":"2024-06-29T15:58:20.307823+0000","last_peered":"2024-06-29T15:58:20.307823+0000","last_clean":"2024-06-29T15:58:20.307823+0000","last_became_active":"2024-06-29T15:58:19.311471+0000","last_became_peered":"2024-06-29T15:58:19.311471+0000","last_unstale":"2024-06-29T15:58:20.307823+0000","last_undegraded":"2024-06-29T15:58:20.307823+0000","last_fullsized":"2024-06-29T15:58:20.307823+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:52.246043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"232'2","reported_seq":16,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764770+0000","last_change":"2024-06-29T15:58:18.302543+0000","last_active":"2024-06-29T15:58:22.764770+0000","last_peered":"2024-06-29T15:58:22.764770+0000","last_clean":"2024-06-29T15:58:22.764770+0000","last_became_active":"2024-06-29T15:58:18.302237+0000","last_became_peered":"2024-06-29T15:58:18.302237+0000","last_unstale":"2024-06-29T15:58:22.764770+0000","last_undegraded":"2024-06-29T15:58:22.764770+0000","last_fullsized":"2024-06-29T15:58:22.764770+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:19:38.790806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707789+0000","last_change":"2024-06-29T15:58:19.312366+0000","last_active":"2024-06-29T15:58:20.707789+0000","last_peered":"2024-06-29T15:58:20.707789+0000","last_clean":"2024-06-29T15:58:20.707789+0000","last_became_active":"2024-06-29T15:58:19.309633+0000","last_became_peered":"2024-06-29T15:58:19.309633+0000","last_unstale":"2024-06-29T15:58:20.707789+0000","last_undegraded":"2024-06-29T15:58:20.707789+0000","last_fullsized":"2024-06-29T15:58:20.707789+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:11.746638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"233'1","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.306899+0000","last_change":"2024-06-29T15:58:18.303588+0000","last_active":"2024-06-29T15:58:20.306899+0000","last_peered":"2024-06-29T15:58:20.306899+0000","last_clean":"2024-06-29T15:58:20.306899+0000","last_became_active":"2024-06-29T15:58:18.303274+0000","last_became_peered":"2024-06-29T15:58:18.303274+0000","last_unstale":"2024-06-29T15:58:20.306899+0000","last_undegraded":"2024-06-29T15:58:20.306899+0000","last_fullsized":"2024-06-29T15:58:20.306899+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:04.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764579+0000","last_change":"2024-06-29T15:58:19.307220+0000","last_active":"2024-06-29T15:58:22.764579+0000","last_peered":"2024-06-29T15:58:22.764579+0000","last_clean":"2024-06-29T15:58:22.764579+0000","last_became_active":"2024-06-29T15:58:19.306905+0000","last_became_peered":"2024-06-29T15:58:19.306905+0000","last_unstale":"2024-06-29T15:58:22.764579+0000","last_undegraded":"2024-06-29T15:58:22.764579+0000","last_fullsized":"2024-06-29T15:58:22.764579+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:10.851803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764869+0000","last_change":"2024-06-29T15:58:18.303622+0000","last_active":"2024-06-29T15:58:22.764869+0000","last_peered":"2024-06-29T15:58:22.764869+0000","last_clean":"2024-06-29T15:58:22.764869+0000","last_became_active":"2024-06-29T15:58:18.303319+0000","last_became_peered":"2024-06-29T15:58:18.303319+0000","last_unstale":"2024-06-29T15:58:22.764869+0000","last_undegraded":"2024-06-29T15:58:22.764869+0000","last_fullsized":"2024-06-29T15:58:22.764869+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:13.370498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764532+0000","last_change":"2024-06-29T15:58:19.294460+0000","last_active":"2024-06-29T15:58:22.764532+0000","last_peered":"2024-06-29T15:58:22.764532+0000","last_clean":"2024-06-29T15:58:22.764532+0000","last_became_active":"2024-06-29T15:58:19.294111+0000","last_became_peered":"2024-06-29T15:58:19.294111+0000","last_unstale":"2024-06-29T15:58:22.764532+0000","last_undegraded":"2024-06-29T15:58:22.764532+0000","last_fullsized":"2024-06-29T15:58:22.764532+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:37.984851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764720+0000","last_change":"2024-06-29T15:58:18.305967+0000","last_active":"2024-06-29T15:58:22.764720+0000","last_peered":"2024-06-29T15:58:22.764720+0000","last_clean":"2024-06-29T15:58:22.764720+0000","last_became_active":"2024-06-29T15:58:18.304464+0000","last_became_peered":"2024-06-29T15:58:18.304464+0000","last_unstale":"2024-06-29T15:58:22.764720+0000","last_undegraded":"2024-06-29T15:58:22.764720+0000","last_fullsized":"2024-06-29T15:58:22.764720+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:37.753590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307869+0000","last_change":"2024-06-29T15:58:19.311597+0000","last_active":"2024-06-29T15:58:20.307869+0000","last_peered":"2024-06-29T15:58:20.307869+0000","last_clean":"2024-06-29T15:58:20.307869+0000","last_became_active":"2024-06-29T15:58:19.311093+0000","last_became_peered":"2024-06-29T15:58:19.311093+0000","last_unstale":"2024-06-29T15:58:20.307869+0000","last_undegraded":"2024-06-29T15:58:20.307869+0000","last_fullsized":"2024-06-29T15:58:20.307869+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:11.746638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.307207+0000","last_change":"2024-06-29T15:58:18.302661+0000","last_active":"2024-06-29T15:58:20.307207+0000","last_peered":"2024-06-29T15:58:20.307207+0000","last_clean":"2024-06-29T15:58:20.307207+0000","last_became_active":"2024-06-29T15:58:18.302330+0000","last_became_peered":"2024-06-29T15:58:18.302330+0000","last_unstale":"2024-06-29T15:58:20.307207+0000","last_undegraded":"2024-06-29T15:58:20.307207+0000","last_fullsized":"2024-06-29T15:58:20.307207+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:36.982727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829704+0000","last_change":"2024-06-29T15:58:19.300548+0000","last_active":"2024-06-29T15:58:21.829704+0000","last_peered":"2024-06-29T15:58:21.829704+0000","last_clean":"2024-06-29T15:58:21.829704+0000","last_became_active":"2024-06-29T15:58:19.300152+0000","last_became_peered":"2024-06-29T15:58:19.300152+0000","last_unstale":"2024-06-29T15:58:21.829704+0000","last_undegraded":"2024-06-29T15:58:21.829704+0000","last_fullsized":"2024-06-29T15:58:21.829704+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:09.509271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764225+0000","last_change":"2024-06-29T15:58:18.294669+0000","last_active":"2024-06-29T15:58:22.764225+0000","last_peered":"2024-06-29T15:58:22.764225+0000","last_clean":"2024-06-29T15:58:22.764225+0000","last_became_active":"2024-06-29T15:58:18.294353+0000","last_became_peered":"2024-06-29T15:58:18.294353+0000","last_unstale":"2024-06-29T15:58:22.764225+0000","last_undegraded":"2024-06-29T15:58:22.764225+0000","last_fullsized":"2024-06-29T15:58:22.764225+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:00:41.729079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:21.829435+0000","last_change":"2024-06-29T15:58:19.303904+0000","last_active":"2024-06-29T15:58:21.829435+0000","last_peered":"2024-06-29T15:58:21.829435+0000","last_clean":"2024-06-29T15:58:21.829435+0000","last_became_active":"2024-06-29T15:58:19.303572+0000","last_became_peered":"2024-06-29T15:58:19.303572+0000","last_unstale":"2024-06-29T15:58:21.829435+0000","last_undegraded":"2024-06-29T15:58:21.829435+0000","last_fullsized":"2024-06-29T15:58:21.829435+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:49:49.141794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.303846+0000","last_change":"2024-06-29T15:58:18.298256+0000","last_active":"2024-06-29T15:58:20.303846+0000","last_peered":"2024-06-29T15:58:20.303846+0000","last_clean":"2024-06-29T15:58:20.303846+0000","last_became_active":"2024-06-29T15:58:18.297928+0000","last_became_peered":"2024-06-29T15:58:18.297928+0000","last_unstale":"2024-06-29T15:58:20.303846+0000","last_undegraded":"2024-06-29T15:58:20.303846+0000","last_fullsized":"2024-06-29T15:58:20.303846+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:28.560916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.707461+0000","last_change":"2024-06-29T15:58:19.305160+0000","last_active":"2024-06-29T15:58:20.707461+0000","last_peered":"2024-06-29T15:58:20.707461+0000","last_clean":"2024-06-29T15:58:20.707461+0000","last_became_active":"2024-06-29T15:58:19.304650+0000","last_became_peered":"2024-06-29T15:58:19.304650+0000","last_unstale":"2024-06-29T15:58:20.707461+0000","last_undegraded":"2024-06-29T15:58:20.707461+0000","last_fullsized":"2024-06-29T15:58:20.707461+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:33.637292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304018+0000","last_change":"2024-06-29T15:58:18.302761+0000","last_active":"2024-06-29T15:58:20.304018+0000","last_peered":"2024-06-29T15:58:20.304018+0000","last_clean":"2024-06-29T15:58:20.304018+0000","last_became_active":"2024-06-29T15:58:18.302490+0000","last_became_peered":"2024-06-29T15:58:18.302490+0000","last_unstale":"2024-06-29T15:58:20.304018+0000","last_undegraded":"2024-06-29T15:58:20.304018+0000","last_fullsized":"2024-06-29T15:58:20.304018+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:59:29.966106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.306827+0000","last_change":"2024-06-29T15:58:19.292511+0000","last_active":"2024-06-29T15:58:20.306827+0000","last_peered":"2024-06-29T15:58:20.306827+0000","last_clean":"2024-06-29T15:58:20.306827+0000","last_became_active":"2024-06-29T15:58:19.291923+0000","last_became_peered":"2024-06-29T15:58:19.291923+0000","last_unstale":"2024-06-29T15:58:20.306827+0000","last_undegraded":"2024-06-29T15:58:20.306827+0000","last_fullsized":"2024-06-29T15:58:20.306827+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:18.095828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"232'4","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:20.304313+0000","last_change":"2024-06-29T15:58:18.303249+0000","last_active":"2024-06-29T15:58:20.304313+0000","last_peered":"2024-06-29T15:58:20.304313+0000","last_clean":"2024-06-29T15:58:20.304313+0000","last_became_active":"2024-06-29T15:58:18.302916+0000","last_became_peered":"2024-06-29T15:58:18.302916+0000","last_unstale":"2024-06-29T15:58:20.304313+0000","last_undegraded":"2024-06-29T15:58:20.304313+0000","last_fullsized":"2024-06-29T15:58:20.304313+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:15.821379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"219'233","reported_seq":1125,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-06-29T15:58:22.764522+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:58:22.764522+0000","last_peered":"2024-06-29T15:58:22.764522+0000","last_clean":"2024-06-29T15:58:22.764522+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:58:22.764522+0000","last_undegraded":"2024-06-29T15:58:22.764522+0000","last_fullsized":"2024-06-29T15:58:22.764522+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575996,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52676,"kb_used_data":3440,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319164,"statfs":{"total":96636764160,"available":96582823936,"internally_reserved":0,"allocated":3522560,"data_stored":2027971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575997,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60432,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311408,"statfs":{"total":96636764160,"available":96574881792,"internally_reserved":0,"allocated":2945024,"data_stored":1444565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575995,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46536,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325304,"statfs":{"total":96636764160,"available":96589111296,"internally_reserved":0,"allocated":2936832,"data_stored":1443961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575996,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60440,"kb_used_data":2888,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311400,"statfs":{"total":96636764160,"available":96574873600,"internally_reserved":0,"allocated":2957312,"data_stored":1445665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575996,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47940,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323900,"statfs":{"total":96636764160,"available":96587673600,"internally_reserved":0,"allocated":2940928,"data_stored":1444543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575996,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47932,"kb_used_data":2860,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323908,"statfs":{"total":96636764160,"available":96587681792,"internally_reserved":0,"allocated":2928640,"data_stored":1437623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575995,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60444,"kb_used_data":2888,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311396,"statfs":{"total":96636764160,"available":96574869504,"internally_reserved":0,"allocated":2957312,"data_stored":1445697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575996,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61000,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310840,"statfs":{"total":96636764160,"available":96574300160,"internally_reserved":0,"allocated":3534848,"data_stored":2034363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:58:23.703 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:58:23.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:58:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.859+0000 7fdd5cf54640 1 -- 172.21.15.81:0/1811995176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd5806f660 msgr2=0x7fdd5806fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.859+0000 7fdd5cf54640 1 --2- 172.21.15.81:0/1811995176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd5806f660 0x7fdd5806fa70 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fdd58070220 tx=0x7fdd5002b7d0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.859+0000 7fdd5cf54640 1 -- 172.21.15.81:0/1811995176 shutdown_connections 2024-06-29T15:58:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.859+0000 7fdd5cf54640 1 --2- 172.21.15.81:0/1811995176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd5806f660 0x7fdd5806fa70 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.859+0000 7fdd5cf54640 1 --2- 172.21.15.81:0/1811995176 >> v2:172.21.15.167:3300/0 conn(0x7fdd580b35b0 0x7fdd5815b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.859+0000 7fdd5cf54640 1 -- 172.21.15.81:0/1811995176 >> 172.21.15.81:0/1811995176 conn(0x7fdd5806ccc0 msgr2=0x7fdd580b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.859+0000 7fdd5cf54640 1 -- 172.21.15.81:0/1811995176 shutdown_connections 2024-06-29T15:58:23.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.859+0000 7fdd5cf54640 1 -- 172.21.15.81:0/1811995176 wait complete. 2024-06-29T15:58:23.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.861+0000 7fdd5cf54640 1 Processor -- start 2024-06-29T15:58:23.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.861+0000 7fdd5cf54640 1 -- start start 2024-06-29T15:58:23.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.862+0000 7fdd5cf54640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd5806f660 0x7fdd581412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.862+0000 7fdd5cf54640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd580b35b0 0x7fdd58154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.862+0000 7fdd5cf54640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd580b4430 0x7fdd58159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.862+0000 7fdd5cf54640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5806f270 con 0x7fdd5806f660 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.862+0000 7fdd5cf54640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdd5806f0f0 con 0x7fdd580b4430 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.862+0000 7fdd5cf54640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdd5806f3f0 con 0x7fdd580b35b0 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd5806f660 0x7fdd581412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd5806f660 0x7fdd581412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59824/0 (socket says 172.21.15.81:59824) 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5659c640 1 -- 172.21.15.81:0/523328903 learned_addr learned my addr 172.21.15.81:0/523328903 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5659c640 1 -- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd580b35b0 msgr2=0x7fdd58154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5659c640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd580b35b0 0x7fdd58154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5659c640 1 -- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd580b4430 msgr2=0x7fdd58159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5659c640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd580b4430 0x7fdd58159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5659c640 1 -- 172.21.15.81:0/523328903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd58141c60 con 0x7fdd5806f660 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5659c640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd5806f660 0x7fdd581412f0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800ea50 tx=0x7fdd4800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd46ffd640 1 -- 172.21.15.81:0/523328903 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4800cd60 con 0x7fdd5806f660 2024-06-29T15:58:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd46ffd640 1 -- 172.21.15.81:0/523328903 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd48011470 con 0x7fdd5806f660 2024-06-29T15:58:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd46ffd640 1 -- 172.21.15.81:0/523328903 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd480118a0 con 0x7fdd5806f660 2024-06-29T15:58:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5cf54640 1 -- 172.21.15.81:0/523328903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd58141f40 con 0x7fdd5806f660 2024-06-29T15:58:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.863+0000 7fdd5cf54640 1 -- 172.21.15.81:0/523328903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd58142500 con 0x7fdd5806f660 2024-06-29T15:58:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.864+0000 7fdd5cf54640 1 -- 172.21.15.81:0/523328903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd5806fa70 con 0x7fdd5806f660 2024-06-29T15:58:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.865+0000 7fdd46ffd640 1 -- 172.21.15.81:0/523328903 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fdd48011a40 con 0x7fdd5806f660 2024-06-29T15:58:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.865+0000 7fdd46ffd640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdd38076540 0x7fdd38078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.865+0000 7fdd46ffd640 1 -- 172.21.15.81:0/523328903 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7fdd48098930 con 0x7fdd5806f660 2024-06-29T15:58:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.866+0000 7fdd55d9b640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdd38076540 0x7fdd38078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.866+0000 7fdd55d9b640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdd38076540 0x7fdd38078a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd5815c700 tx=0x7fdd4c007a30 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:23.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:23.870+0000 7fdd46ffd640 1 -- 172.21.15.81:0/523328903 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdd48016030 con 0x7fdd5806f660 2024-06-29T15:58:24.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.118+0000 7fdd5cf54640 1 -- 172.21.15.81:0/523328903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fdd58069550 con 0x7fdd5806f660 2024-06-29T15:58:24.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.120+0000 7fdd46ffd640 1 -- 172.21.15.81:0/523328903 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v68) ==== 190+0+147 (secure 0 0 0) 0x7fdd48061d10 con 0x7fdd5806f660 2024-06-29T15:58:24.122 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T15:58:24.122 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T15:58:24.122 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T15:58:24.122 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T15:58:24.122 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:58:24.122 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T15:58:24.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.122+0000 7fdd5cf54640 1 -- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdd38076540 msgr2=0x7fdd38078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.122+0000 7fdd5cf54640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdd38076540 0x7fdd38078a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd5815c700 tx=0x7fdd4c007a30 comp rx=0 tx=0).stop 2024-06-29T15:58:24.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.122+0000 7fdd5cf54640 1 -- 172.21.15.81:0/523328903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd5806f660 msgr2=0x7fdd581412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.122+0000 7fdd5cf54640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd5806f660 0x7fdd581412f0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800ea50 tx=0x7fdd4800ef10 comp rx=0 tx=0).stop 2024-06-29T15:58:24.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.123+0000 7fdd5cf54640 1 -- 172.21.15.81:0/523328903 shutdown_connections 2024-06-29T15:58:24.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.123+0000 7fdd5cf54640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdd38076540 0x7fdd38078a50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.123+0000 7fdd5cf54640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd580b4430 0x7fdd58159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.123+0000 7fdd5cf54640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdd580b35b0 0x7fdd58154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.123+0000 7fdd5cf54640 1 --2- 172.21.15.81:0/523328903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd5806f660 0x7fdd581412f0 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.123+0000 7fdd5cf54640 1 -- 172.21.15.81:0/523328903 >> 172.21.15.81:0/523328903 conn(0x7fdd5806ccc0 msgr2=0x7fdd5815bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:24.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.123+0000 7fdd5cf54640 1 -- 172.21.15.81:0/523328903 shutdown_connections 2024-06-29T15:58:24.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.123+0000 7fdd5cf54640 1 -- 172.21.15.81:0/523328903 wait complete. 2024-06-29T15:58:24.136 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:24.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.247+0000 7f2300f0a640 1 -- 172.21.15.81:0/3017999400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22fc0d0230 msgr2=0x7f22fc0d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.247+0000 7f2300f0a640 1 --2- 172.21.15.81:0/3017999400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22fc0d0230 0x7f22fc0d0640 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0c5590 tx=0x7f22f002a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.247+0000 7f2300f0a640 1 -- 172.21.15.81:0/3017999400 shutdown_connections 2024-06-29T15:58:24.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.247+0000 7f2300f0a640 1 --2- 172.21.15.81:0/3017999400 >> v2:172.21.15.167:3300/0 conn(0x7f22fc0b4f40 0x7f22fc0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.247+0000 7f2300f0a640 1 --2- 172.21.15.81:0/3017999400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22fc0d0230 0x7f22fc0d0640 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.247+0000 7f2300f0a640 1 -- 172.21.15.81:0/3017999400 >> 172.21.15.81:0/3017999400 conn(0x7f22fc0b3680 msgr2=0x7f22fc0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:24.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.247+0000 7f2300f0a640 1 -- 172.21.15.81:0/3017999400 shutdown_connections 2024-06-29T15:58:24.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.247+0000 7f2300f0a640 1 -- 172.21.15.81:0/3017999400 wait complete. 2024-06-29T15:58:24.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.249+0000 7f2300f0a640 1 Processor -- start 2024-06-29T15:58:24.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.249+0000 7f2300f0a640 1 -- start start 2024-06-29T15:58:24.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.250+0000 7f2300f0a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22fc0b3200 0x7f22fc09e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:24.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.250+0000 7f2300f0a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22fc0b4f40 0x7f22fc1f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:24.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.250+0000 7f2300f0a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22fc0d0230 0x7f22fc1e65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:24.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.250+0000 7f2300f0a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc0b5c20 con 0x7f22fc0d0230 2024-06-29T15:58:24.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.250+0000 7f2300f0a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc0b5aa0 con 0x7f22fc0b3200 2024-06-29T15:58:24.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.250+0000 7f2300f0a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f22fc0b5da0 con 0x7f22fc0b4f40 2024-06-29T15:58:24.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22fc0d0230 0x7f22fc1e65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:24.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22fc0d0230 0x7f22fc1e65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59840/0 (socket says 172.21.15.81:59840) 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fad9d640 1 -- 172.21.15.81:0/3623528904 learned_addr learned my addr 172.21.15.81:0/3623528904 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22f9d9b640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22fc0b4f40 0x7f22fc1f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fa59c640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22fc0b3200 0x7f22fc09e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fad9d640 1 -- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22fc0b4f40 msgr2=0x7f22fc1f38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fad9d640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22fc0b4f40 0x7f22fc1f38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fad9d640 1 -- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22fc0b3200 msgr2=0x7f22fc09e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fad9d640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22fc0b3200 0x7f22fc09e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fad9d640 1 -- 172.21.15.81:0/3623528904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22fc09c190 con 0x7f22fc0d0230 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fa59c640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22fc0b3200 0x7f22fc09e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22f9d9b640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22fc0b4f40 0x7f22fc1f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22fad9d640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22fc0d0230 0x7f22fc1e65c0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f22f400b440 tx=0x7f22f400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:24.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22eaffd640 1 -- 172.21.15.81:0/3623528904 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4015020 con 0x7f22fc0d0230 2024-06-29T15:58:24.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22eaffd640 1 -- 172.21.15.81:0/3623528904 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22f4011920 con 0x7f22fc0d0230 2024-06-29T15:58:24.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f2300f0a640 1 -- 172.21.15.81:0/3623528904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22fc09c470 con 0x7f22fc0d0230 2024-06-29T15:58:24.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f22eaffd640 1 -- 172.21.15.81:0/3623528904 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4011d70 con 0x7f22fc0d0230 2024-06-29T15:58:24.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.251+0000 7f2300f0a640 1 -- 172.21.15.81:0/3623528904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22fc09ca30 con 0x7f22fc0d0230 2024-06-29T15:58:24.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.252+0000 7f2300f0a640 1 -- 172.21.15.81:0/3623528904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22fc1f43b0 con 0x7f22fc0d0230 2024-06-29T15:58:24.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.253+0000 7f22eaffd640 1 -- 172.21.15.81:0/3623528904 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f22f4018020 con 0x7f22fc0d0230 2024-06-29T15:58:24.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.253+0000 7f22eaffd640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f22dc076540 0x7f22dc078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:24.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.254+0000 7f22eaffd640 1 -- 172.21.15.81:0/3623528904 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f22f4097b80 con 0x7f22fc0d0230 2024-06-29T15:58:24.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.254+0000 7f22fa59c640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f22dc076540 0x7f22dc078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:24.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.254+0000 7f22fa59c640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f22dc076540 0x7f22dc078a50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0020f0 tx=0x7f22ec009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:24.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.258+0000 7f22eaffd640 1 -- 172.21.15.81:0/3623528904 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f22f401f020 con 0x7f22fc0d0230 2024-06-29T15:58:24.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.476+0000 7f2300f0a640 1 -- 172.21.15.81:0/3623528904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22fc070050 con 0x7f22fc0d0230 2024-06-29T15:58:24.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.476+0000 7f22eaffd640 1 -- 172.21.15.81:0/3623528904 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) ==== 95+0+5237 (secure 0 0 0) 0x7f22fc070050 con 0x7f22fc0d0230 2024-06-29T15:58:24.478 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:24.478 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":129,"btime":"2024-06-29T15:58:22:311756+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123},{"gid":12093,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:18.654579+0000","modified":"2024-06-29T15:58:21.679033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12105":{"gid":12105,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/30158598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":30158598},{"type":"v1","addr":"172.21.15.167:6837","nonce":30158598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12105,"qdb_cluster":[12105]},"id":15}]} 2024-06-29T15:58:24.478 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 129 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 -- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f22dc076540 msgr2=0x7f22dc078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f22dc076540 0x7f22dc078a50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0020f0 tx=0x7f22ec009040 comp rx=0 tx=0).stop 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 -- 172.21.15.81:0/3623528904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22fc0d0230 msgr2=0x7f22fc1e65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22fc0d0230 0x7f22fc1e65c0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f22f400b440 tx=0x7f22f400b900 comp rx=0 tx=0).stop 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 -- 172.21.15.81:0/3623528904 shutdown_connections 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f22dc076540 0x7f22dc078a50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22fc0d0230 0x7f22fc1e65c0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f22fc0b4f40 0x7f22fc1f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 --2- 172.21.15.81:0/3623528904 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f22fc0b3200 0x7f22fc09e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 -- 172.21.15.81:0/3623528904 >> 172.21.15.81:0/3623528904 conn(0x7f22fc0b3680 msgr2=0x7f22fc1e37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:24.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 -- 172.21.15.81:0/3623528904 shutdown_connections 2024-06-29T15:58:24.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:24.478+0000 7f2300f0a640 1 -- 172.21.15.81:0/3623528904 wait complete. 2024-06-29T15:58:24.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 129, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:58:18.654579+0000', 'modified': '2024-06-29T15:58:21.679033+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12105}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12105': {'gid': 12105, 'name': 'd', 'rank': 0, 'incarnation': 125, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.167:6837/30158598', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6835', 'nonce': 30158598}, {'type': 'v1', 'addr': '172.21.15.167:6837', 'nonce': 30158598}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12105, 'qdb_cluster': [12105]} 2024-06-29T15:58:24.894 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:58:24.894 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-06-29T15:58:25.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.008+0000 7f20ba36f640 1 -- 172.21.15.81:0/2576319370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b415aed0 msgr2=0x7f20b40b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.008+0000 7f20ba36f640 1 --2- 172.21.15.81:0/2576319370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b415aed0 0x7f20b40b4070 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f20ac005fd0 tx=0x7f20ac02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.008+0000 7f20ba36f640 1 -- 172.21.15.81:0/2576319370 shutdown_connections 2024-06-29T15:58:25.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.008+0000 7f20ba36f640 1 --2- 172.21.15.81:0/2576319370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b415aed0 0x7f20b40b4070 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.008+0000 7f20ba36f640 1 -- 172.21.15.81:0/2576319370 >> 172.21.15.81:0/2576319370 conn(0x7f20b40d0930 msgr2=0x7f20b40d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.008+0000 7f20ba36f640 1 -- 172.21.15.81:0/2576319370 shutdown_connections 2024-06-29T15:58:25.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.008+0000 7f20ba36f640 1 -- 172.21.15.81:0/2576319370 wait complete. 2024-06-29T15:58:25.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.009+0000 7f20ba36f640 1 Processor -- start 2024-06-29T15:58:25.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.010+0000 7f20ba36f640 1 -- start start 2024-06-29T15:58:25.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20ba36f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20b406f6a0 0x7f20b41520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20ba36f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20b40b57b0 0x7f20b4140140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20ba36f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b415aed0 0x7f20b41e7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20ba36f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f20b40b4730 con 0x7f20b415aed0 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20ba36f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f20b40b45b0 con 0x7f20b406f6a0 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20ba36f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f20b40b48b0 con 0x7f20b40b57b0 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b890c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b415aed0 0x7f20b41e7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b890c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b415aed0 0x7f20b41e7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59854/0 (socket says 172.21.15.81:59854) 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b890c640 1 -- 172.21.15.81:0/976786541 learned_addr learned my addr 172.21.15.81:0/976786541 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b37fe640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20b40b57b0 0x7f20b4140140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b3fff640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20b406f6a0 0x7f20b41520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b890c640 1 -- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20b40b57b0 msgr2=0x7f20b4140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b890c640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20b40b57b0 0x7f20b4140140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b890c640 1 -- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20b406f6a0 msgr2=0x7f20b41520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b890c640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20b406f6a0 0x7f20b41520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b890c640 1 -- 172.21.15.81:0/976786541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b41e6c70 con 0x7f20b415aed0 2024-06-29T15:58:25.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.011+0000 7f20b3fff640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20b406f6a0 0x7f20b41520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:25.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.012+0000 7f20b890c640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b415aed0 0x7f20b41e7d10 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f20ac02aea0 tx=0x7f20ac003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.012+0000 7f20b0ff9640 1 -- 172.21.15.81:0/976786541 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac031da0 con 0x7f20b415aed0 2024-06-29T15:58:25.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.012+0000 7f20b0ff9640 1 -- 172.21.15.81:0/976786541 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20ac003e10 con 0x7f20b415aed0 2024-06-29T15:58:25.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.012+0000 7f20b0ff9640 1 -- 172.21.15.81:0/976786541 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac033330 con 0x7f20b415aed0 2024-06-29T15:58:25.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.012+0000 7f20ba36f640 1 -- 172.21.15.81:0/976786541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b41e6f50 con 0x7f20b415aed0 2024-06-29T15:58:25.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.012+0000 7f20ba36f640 1 -- 172.21.15.81:0/976786541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b41e7510 con 0x7f20b415aed0 2024-06-29T15:58:25.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.014+0000 7f20b0ff9640 1 -- 172.21.15.81:0/976786541 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f20ac0334d0 con 0x7f20b415aed0 2024-06-29T15:58:25.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.014+0000 7f20b0ff9640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f20a0076510 0x7f20a0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.014+0000 7f20b0ff9640 1 -- 172.21.15.81:0/976786541 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f20ac0b93f0 con 0x7f20b415aed0 2024-06-29T15:58:25.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.014+0000 7f20b3fff640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f20a0076510 0x7f20a0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.015+0000 7f20b3fff640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f20a0076510 0x7f20a0078a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20a40016a0 tx=0x7f20a4009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.015+0000 7f20c1222740 1 Processor -- start 2024-06-29T15:58:25.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.015+0000 7f20c1222740 1 -- start start 2024-06-29T15:58:25.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.016+0000 7f20c1222740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x55833423d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.016+0000 7f20c1222740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e283b0 0x5583342213a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.016+0000 7f20c1222740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 0x5583342218e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.016+0000 7f20c1222740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5583341b9bc0 con 0x558333e283b0 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.016+0000 7f20c1222740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x558334168630 con 0x558333e30590 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.016+0000 7f20c1222740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x558334157bd0 con 0x558333e51270 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.016+0000 7f20b37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e283b0 0x5583342213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.016+0000 7f20b37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e283b0 0x5583342213a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59862/0 (socket says 172.21.15.81:59862) 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.016+0000 7f20b37fe640 1 -- 172.21.15.81:0/2490259108 learned_addr learned my addr 172.21.15.81:0/2490259108 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f20b3fff640 1 --2- 172.21.15.81:0/2490259108 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x55833423d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f20b37fe640 1 -- 172.21.15.81:0/2490259108 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 msgr2=0x55833423d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f20b890c640 1 --2- 172.21.15.81:0/2490259108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 0x5583342218e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f20b37fe640 1 --2- 172.21.15.81:0/2490259108 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x55833423d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f20b37fe640 1 -- 172.21.15.81:0/2490259108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 msgr2=0x5583342218e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f20b37fe640 1 --2- 172.21.15.81:0/2490259108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 0x5583342218e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f20b37fe640 1 -- 172.21.15.81:0/2490259108 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558334154ca0 con 0x558333e283b0 2024-06-29T15:58:25.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f20b37fe640 1 --2- 172.21.15.81:0/2490259108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e283b0 0x5583342213a0 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7f20a80099c0 tx=0x7f20a8009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1078a904815e7b59 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f2099ffb640 1 -- 172.21.15.81:0/2490259108 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8017070 con 0x558333e283b0 2024-06-29T15:58:25.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f2099ffb640 1 -- 172.21.15.81:0/2490259108 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20a800eae0 con 0x558333e283b0 2024-06-29T15:58:25.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.017+0000 7f2099ffb640 1 -- 172.21.15.81:0/2490259108 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8013470 con 0x558333e283b0 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.018+0000 7f20c1222740 1 -- 172.21.15.81:0/2490259108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e283b0 msgr2=0x5583342213a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.018+0000 7f20c1222740 1 --2- 172.21.15.81:0/2490259108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e283b0 0x5583342213a0 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7f20a80099c0 tx=0x7f20a8009e80 comp rx=0 tx=0).stop 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.018+0000 7f20c1222740 1 -- 172.21.15.81:0/2490259108 shutdown_connections 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.018+0000 7f20c1222740 1 --2- 172.21.15.81:0/2490259108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 0x5583342218e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.018+0000 7f20c1222740 1 --2- 172.21.15.81:0/2490259108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e283b0 0x5583342213a0 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.018+0000 7f20c1222740 1 --2- 172.21.15.81:0/2490259108 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x55833423d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.018+0000 7f20c1222740 1 -- 172.21.15.81:0/2490259108 >> 172.21.15.81:0/2490259108 conn(0x558333e453c0 msgr2=0x5583341f4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.018+0000 7f20c1222740 1 -- 172.21.15.81:0/2490259108 shutdown_connections 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.018+0000 7f20c1222740 1 -- 172.21.15.81:0/2490259108 wait complete. 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.019+0000 7f20c1222740 1 -- start start 2024-06-29T15:58:25.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.019+0000 7f20c1222740 1 Processor -- start 2024-06-29T15:58:25.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20c1222740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e283b0 0x558334232c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20c1222740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x55833421b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20c1222740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x55833424f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20c1222740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5583340f5ae0 con 0x558333e30590 2024-06-29T15:58:25.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20c1222740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55833415bc10 con 0x558333e283b0 2024-06-29T15:58:25.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20c1222740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x558333ecb620 con 0x558333e51270 2024-06-29T15:58:25.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20c1222740 10 client.0 _ll_register_callbacks cb 0x558334102480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20b37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x55833421b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20b37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x55833421b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59870/0 (socket says 172.21.15.81:59870) 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20b37fe640 1 -- 172.21.15.81:0/821713324 learned_addr learned my addr 172.21.15.81:0/821713324 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20b3fff640 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e283b0 0x558334232c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.020+0000 7f20b890c640 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x55833424f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20b37fe640 1 -- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 msgr2=0x55833424f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20b37fe640 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x55833424f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20b37fe640 1 -- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e283b0 msgr2=0x558334232c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20b37fe640 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e283b0 0x558334232c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20b37fe640 1 -- 172.21.15.81:0/821713324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558333ebc850 con 0x558333e30590 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20b3fff640 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e283b0 0x558334232c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20b37fe640 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x55833421b450 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f20a8006fd0 tx=0x7f20a8004480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808832c5d95c0cf9 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f2092ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f2092ffd640 1 -- 172.21.15.81:0/821713324 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8017050 con 0x558333e30590 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f2092ffd640 1 -- 172.21.15.81:0/821713324 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f20a8025b30 con 0x558333e30590 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20c1222740 1 -- 172.21.15.81:0/821713324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20a800eae0 con 0x558333e30590 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f2092ffd640 1 -- 172.21.15.81:0/821713324 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f20a802c050 con 0x558333e30590 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20c1222740 10 client.12234 fetch_fsmap learned FSMap version 129 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20c1222740 1 -- 172.21.15.81:0/821713324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) -- 0x558333f336b0 con 0x558333e30590 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f2092ffd640 1 -- 172.21.15.81:0/821713324 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20a800eae0 con 0x558333e30590 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f2092ffd640 1 -- 172.21.15.81:0/821713324 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 129) ==== 3502+0+0 (secure 0 0 0) 0x7f20a802f020 con 0x558333e30590 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20c1222740 10 client.12234 fetch_fsmap finished waiting for FSMap version 129 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20c1222740 10 client.12234 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20c1222740 20 client.12234 populate_metadata read hostname 'smithi081' 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20c1222740 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:6835/30158598,v1:172.21.15.167:6837/30158598] conn(0x558333e0fe40 0x55833424bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20c1222740 4 client.12234 mds_command: new command op to 12105 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:58:25.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.021+0000 7f20c1222740 1 -- 172.21.15.81:0/821713324 --> [v2:172.21.15.167:6835/30158598,v1:172.21.15.167:6837/30158598] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55833416a580 con 0x558333e0fe40 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.022+0000 7f20b3fff640 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:6835/30158598,v1:172.21.15.167:6837/30158598] conn(0x558333e0fe40 0x55833424bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.022+0000 7f2092ffd640 10 client.12234 ms_handle_connect on v2:172.21.15.167:6835/30158598 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.022+0000 7f20b3fff640 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:6835/30158598,v1:172.21.15.167:6837/30158598] conn(0x558333e0fe40 0x55833424bb00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db48b31d96b222a server_cookie=ef567969ef241e47 in_seq=0 out_seq=0 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f2092ffd640 1 -- 172.21.15.81:0/821713324 <== mds.0 v2:172.21.15.167:6835/30158598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55833416a580 con 0x558333e0fe40 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f2092ffd640 10 client.12234 handle_command_reply: tid=0 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 client.12234 shutdown 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 -- 172.21.15.81:0/821713324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 msgr2=0x55833421b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x55833421b450 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f20a8006fd0 tx=0x7f20a8004480 comp rx=0 tx=0).stop 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 -- 172.21.15.81:0/821713324 shutdown_connections 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x55833424f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x55833421b450 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:6835/30158598,v1:172.21.15.167:6837/30158598] conn(0x558333e0fe40 0x55833424bb00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 --2- 172.21.15.81:0/821713324 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e283b0 0x558334232c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f2092ffd640 0 client.12234 ms_handle_reset on v2:172.21.15.167:6835/30158598 2024-06-29T15:58:25.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 -- 172.21.15.81:0/821713324 >> 172.21.15.81:0/821713324 conn(0x558333e453c0 msgr2=0x55833421dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 -- 172.21.15.81:0/821713324 shutdown_connections 2024-06-29T15:58:25.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 1 -- 172.21.15.81:0/821713324 wait complete. 2024-06-29T15:58:25.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.023+0000 7f20c1222740 20 client.12234 trim_cache size 0 max 16384 2024-06-29T15:58:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.042+0000 7f20c1222740 1 Processor -- start 2024-06-29T15:58:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.042+0000 7f20c1222740 1 -- start start 2024-06-29T15:58:25.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20c1222740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e283b0 0x55833424af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20c1222740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x5583341f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20c1222740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x5583341f1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20c1222740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f20b41fe7e0 con 0x558333e30590 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20c1222740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f20b41e5db0 con 0x558333e283b0 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20c1222740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f20b41fe1c0 con 0x558333e51270 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20b37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x5583341f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20b37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x5583341f0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59880/0 (socket says 172.21.15.81:59880) 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20b37fe640 1 -- 172.21.15.81:0/2956909993 learned_addr learned my addr 172.21.15.81:0/2956909993 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20b37fe640 1 -- 172.21.15.81:0/2956909993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 msgr2=0x5583341f1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20b37fe640 1 --2- 172.21.15.81:0/2956909993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x5583341f1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20b37fe640 1 -- 172.21.15.81:0/2956909993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e283b0 msgr2=0x55833424af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20b37fe640 1 --2- 172.21.15.81:0/2956909993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e283b0 0x55833424af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20b37fe640 1 -- 172.21.15.81:0/2956909993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583341edb20 con 0x558333e30590 2024-06-29T15:58:25.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20b37fe640 1 --2- 172.21.15.81:0/2956909993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x5583341f0b40 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009280 tx=0x7f20a8025dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e81843638188c9c server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20997fa640 1 -- 172.21.15.81:0/2956909993 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8017040 con 0x558333e30590 2024-06-29T15:58:25.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20997fa640 1 -- 172.21.15.81:0/2956909993 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20a8003d20 con 0x558333e30590 2024-06-29T15:58:25.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.044+0000 7f20997fa640 1 -- 172.21.15.81:0/2956909993 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8013470 con 0x558333e30590 2024-06-29T15:58:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.045+0000 7f20c1222740 1 -- 172.21.15.81:0/2956909993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 msgr2=0x5583341f0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.045+0000 7f20c1222740 1 --2- 172.21.15.81:0/2956909993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x5583341f0b40 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009280 tx=0x7f20a8025dc0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.045+0000 7f20c1222740 1 -- 172.21.15.81:0/2956909993 shutdown_connections 2024-06-29T15:58:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.045+0000 7f20c1222740 1 --2- 172.21.15.81:0/2956909993 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e51270 0x5583341f1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.045+0000 7f20c1222740 1 --2- 172.21.15.81:0/2956909993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e30590 0x5583341f0b40 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.045+0000 7f20c1222740 1 --2- 172.21.15.81:0/2956909993 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e283b0 0x55833424af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.045+0000 7f20c1222740 1 -- 172.21.15.81:0/2956909993 >> 172.21.15.81:0/2956909993 conn(0x558333e453c0 msgr2=0x5583341f8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.045+0000 7f20c1222740 1 -- 172.21.15.81:0/2956909993 shutdown_connections 2024-06-29T15:58:25.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.045+0000 7f20c1222740 1 -- 172.21.15.81:0/2956909993 wait complete. 2024-06-29T15:58:25.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.046+0000 7f20c1222740 1 -- start start 2024-06-29T15:58:25.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.046+0000 7f20c1222740 1 Processor -- start 2024-06-29T15:58:25.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.047+0000 7f20c1222740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e283b0 0x5583342bc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.047+0000 7f20c1222740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 0x5583342bcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.047+0000 7f20c1222740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e51270 0x5583342bd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.047+0000 7f20c1222740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55833424e6e0 con 0x558333e51270 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.047+0000 7f20c1222740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55833424ff20 con 0x558333e30590 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.047+0000 7f20c1222740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55833424d6b0 con 0x558333e283b0 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.047+0000 7f20c1222740 10 client.0 _ll_register_callbacks cb 0x7f20a80097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.047+0000 7f20b890c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e51270 0x5583342bd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.047+0000 7f20b890c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e51270 0x5583342bd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59886/0 (socket says 172.21.15.81:59886) 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b890c640 1 -- 172.21.15.81:0/1349475744 learned_addr learned my addr 172.21.15.81:0/1349475744 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b3fff640 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e283b0 0x5583342bc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b37fe640 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 0x5583342bcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b890c640 1 -- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e283b0 msgr2=0x5583342bc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b890c640 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e283b0 0x5583342bc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b890c640 1 -- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 msgr2=0x5583342bcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b890c640 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 0x5583342bcf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b890c640 1 -- 172.21.15.81:0/1349475744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55833425f4d0 con 0x558333e51270 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b3fff640 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e283b0 0x5583342bc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b37fe640 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 0x5583342bcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20b890c640 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e51270 0x5583342bd490 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f20ac081de0 tx=0x7f20ac0863b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d72f664f77ebc4 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f2092ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f2092ffd640 1 -- 172.21.15.81:0/1349475744 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac048040 con 0x558333e51270 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f2092ffd640 1 -- 172.21.15.81:0/1349475744 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f20ac082540 con 0x558333e51270 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20c1222740 1 -- 172.21.15.81:0/1349475744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20a8003d20 con 0x558333e51270 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f2092ffd640 1 -- 172.21.15.81:0/1349475744 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f20ac081bf0 con 0x558333e51270 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20c1222740 10 client.12240 fetch_fsmap learned FSMap version 129 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20c1222740 1 -- 172.21.15.81:0/1349475744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) -- 0x558334220fb0 con 0x558333e51270 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f2092ffd640 1 -- 172.21.15.81:0/1349475744 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20a8003d20 con 0x558333e51270 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f2092ffd640 1 -- 172.21.15.81:0/1349475744 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 129) ==== 3502+0+0 (secure 0 0 0) 0x7f20ac08aa20 con 0x558333e51270 2024-06-29T15:58:25.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20c1222740 10 client.12240 fetch_fsmap finished waiting for FSMap version 129 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20c1222740 10 client.12240 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.048+0000 7f20c1222740 20 client.12240 populate_metadata read hostname 'smithi081' 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.049+0000 7f20c1222740 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:6835/30158598,v1:172.21.15.167:6837/30158598] conn(0x5583341f1140 0x558334235780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.049+0000 7f20c1222740 4 client.12240 mds_command: new command op to 12105 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.049+0000 7f20c1222740 1 -- 172.21.15.81:0/1349475744 --> [v2:172.21.15.167:6835/30158598,v1:172.21.15.167:6837/30158598] -- command(tid 0: {"prefix": "status"}) -- 0x558334237cf0 con 0x5583341f1140 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.049+0000 7f20b3fff640 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:6835/30158598,v1:172.21.15.167:6837/30158598] conn(0x5583341f1140 0x558334235780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.049+0000 7f20b3fff640 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:6835/30158598,v1:172.21.15.167:6837/30158598] conn(0x5583341f1140 0x558334235780 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67047ed21433d09e server_cookie=d70a56b87362303 in_seq=0 out_seq=0 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.049+0000 7f2092ffd640 10 client.12240 ms_handle_connect on v2:172.21.15.167:6835/30158598 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.049+0000 7f2092ffd640 1 -- 172.21.15.81:0/1349475744 <== mds.0 v2:172.21.15.167:6835/30158598 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x558334237cf0 con 0x5583341f1140 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.049+0000 7f2092ffd640 10 client.12240 handle_command_reply: tid=0 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.049+0000 7f20c1222740 1 client.12240 shutdown 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 1 -- 172.21.15.81:0/1349475744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e51270 msgr2=0x5583342bd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e51270 0x5583342bd490 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f20ac081de0 tx=0x7f20ac0863b0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 1 -- 172.21.15.81:0/1349475744 shutdown_connections 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558333e51270 0x5583342bd490 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:6835/30158598,v1:172.21.15.167:6837/30158598] conn(0x5583341f1140 0x558334235780 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x558333e30590 0x5583342bcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 1 --2- 172.21.15.81:0/1349475744 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x558333e283b0 0x5583342bc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f2092ffd640 0 client.12240 ms_handle_reset on v2:172.21.15.167:6835/30158598 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 1 -- 172.21.15.81:0/1349475744 >> 172.21.15.81:0/1349475744 conn(0x558333e453c0 msgr2=0x55833421a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 1 -- 172.21.15.81:0/1349475744 shutdown_connections 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 1 -- 172.21.15.81:0/1349475744 wait complete. 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20c1222740 20 client.12240 trim_cache size 0 max 16384 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:58:25.056 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 12105, 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7660639199999997, 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 129, 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 233, 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 233, 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.44087356 2024-06-29T15:58:25.057 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:58:25.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 -- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f20a0076510 msgr2=0x7f20a0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f20a0076510 0x7f20a0078a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20a40016a0 tx=0x7f20a4009040 comp rx=0 tx=0).stop 2024-06-29T15:58:25.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 -- 172.21.15.81:0/976786541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b415aed0 msgr2=0x7f20b41e7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b415aed0 0x7f20b41e7d10 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f20ac02aea0 tx=0x7f20ac003c20 comp rx=0 tx=0).stop 2024-06-29T15:58:25.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 -- 172.21.15.81:0/976786541 shutdown_connections 2024-06-29T15:58:25.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f20a0076510 0x7f20a0078a20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b415aed0 0x7f20b41e7d10 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f20b40b57b0 0x7f20b4140140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 --2- 172.21.15.81:0/976786541 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f20b406f6a0 0x7f20b41520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 -- 172.21.15.81:0/976786541 >> 172.21.15.81:0/976786541 conn(0x7f20b40d0930 msgr2=0x7f20b41583d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 -- 172.21.15.81:0/976786541 shutdown_connections 2024-06-29T15:58:25.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.050+0000 7f20ba36f640 1 -- 172.21.15.81:0/976786541 wait complete. 2024-06-29T15:58:25.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:58:25.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.175+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2232654457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d00005f10 msgr2=0x7f5d000063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.175+0000 7f5d1b4fe640 1 --2- 172.21.15.81:0/2232654457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d00005f10 0x7f5d000063e0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f5d14073170 tx=0x7f5d00006940 comp rx=0 tx=0).stop 2024-06-29T15:58:25.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.175+0000 7f5d19a9b640 1 -- 172.21.15.81:0/2232654457 reap_dead start 2024-06-29T15:58:25.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.175+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2232654457 shutdown_connections 2024-06-29T15:58:25.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.175+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2232654457 >> 172.21.15.81:0/2232654457 conn(0x7f5d1406ccc0 msgr2=0x7f5d140b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.176+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2232654457 shutdown_connections 2024-06-29T15:58:25.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.176+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2232654457 wait complete. 2024-06-29T15:58:25.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.177+0000 7f5d1b4fe640 1 Processor -- start 2024-06-29T15:58:25.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.177+0000 7f5d1b4fe640 1 -- start start 2024-06-29T15:58:25.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d1b4fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d00005f10 0x7f5d140a9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d1b4fe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d141f8330 0x7f5d141594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d1b4fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d14159a30 0x7f5d141472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d1b4fe640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1406ef50 con 0x7f5d14159a30 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d1b4fe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1406edd0 con 0x7f5d141f8330 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d1b4fe640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5d1406f0d0 con 0x7f5d00005f10 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d19a9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d14159a30 0x7f5d141472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d19a9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d14159a30 0x7f5d141472b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59916/0 (socket says 172.21.15.81:59916) 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d19a9b640 1 -- 172.21.15.81:0/2142291314 learned_addr learned my addr 172.21.15.81:0/2142291314 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d1929a640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d00005f10 0x7f5d140a9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d18a99640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d141f8330 0x7f5d141594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d19a9b640 1 -- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d00005f10 msgr2=0x7f5d140a9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d19a9b640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d00005f10 0x7f5d140a9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d19a9b640 1 -- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d141f8330 msgr2=0x7f5d141594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d19a9b640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d141f8330 0x7f5d141594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d19a9b640 1 -- 172.21.15.81:0/2142291314 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d14159e40 con 0x7f5d14159a30 2024-06-29T15:58:25.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d18a99640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d141f8330 0x7f5d141594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:25.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d19a9b640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d14159a30 0x7f5d141472b0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c02b020 tx=0x7f5d0c006bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d09ffb640 1 -- 172.21.15.81:0/2142291314 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c0037f0 con 0x7f5d14159a30 2024-06-29T15:58:25.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d09ffb640 1 -- 172.21.15.81:0/2142291314 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d0c003990 con 0x7f5d14159a30 2024-06-29T15:58:25.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2142291314 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d141e4430 con 0x7f5d14159a30 2024-06-29T15:58:25.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.179+0000 7f5d09ffb640 1 -- 172.21.15.81:0/2142291314 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c003de0 con 0x7f5d14159a30 2024-06-29T15:58:25.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.180+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2142291314 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d141e49f0 con 0x7f5d14159a30 2024-06-29T15:58:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.181+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2142291314 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d141e41a0 con 0x7f5d14159a30 2024-06-29T15:58:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.181+0000 7f5d09ffb640 1 -- 172.21.15.81:0/2142291314 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f5d0c035020 con 0x7f5d14159a30 2024-06-29T15:58:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.182+0000 7f5d09ffb640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5cf8076510 0x7f5cf8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.182+0000 7f5d09ffb640 1 -- 172.21.15.81:0/2142291314 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f5d0c0b7ed0 con 0x7f5d14159a30 2024-06-29T15:58:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.182+0000 7f5d1929a640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5cf8076510 0x7f5cf8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.183+0000 7f5d1929a640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5cf8076510 0x7f5cf8078a20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d10005fd0 tx=0x7f5d10009500 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:25.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.186+0000 7f5d09ffb640 1 -- 172.21.15.81:0/2142291314 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5d0c047020 con 0x7f5d14159a30 2024-06-29T15:58:25.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.403+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2142291314 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5d141e50b0 con 0x7f5d14159a30 2024-06-29T15:58:25.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.404+0000 7f5d09ffb640 1 -- 172.21.15.81:0/2142291314 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v129) ==== 53+0+73 (secure 0 0 0) 0x7f5d141e50b0 con 0x7f5d14159a30 2024-06-29T15:58:25.405 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:58:25.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5cf8076510 msgr2=0x7f5cf8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5cf8076510 0x7f5cf8078a20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d10005fd0 tx=0x7f5d10009500 comp rx=0 tx=0).stop 2024-06-29T15:58:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2142291314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d14159a30 msgr2=0x7f5d141472b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d14159a30 0x7f5d141472b0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c02b020 tx=0x7f5d0c006bd0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2142291314 shutdown_connections 2024-06-29T15:58:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f5cf8076510 0x7f5cf8078a20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d14159a30 0x7f5d141472b0 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5d141f8330 0x7f5d141594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 --2- 172.21.15.81:0/2142291314 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5d00005f10 0x7f5d140a9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2142291314 >> 172.21.15.81:0/2142291314 conn(0x7f5d1406ccc0 msgr2=0x7f5d140af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2142291314 shutdown_connections 2024-06-29T15:58:25.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25.406+0000 7f5d1b4fe640 1 -- 172.21.15.81:0/2142291314 wait complete. 2024-06-29T15:58:25.419 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:58:25.419 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:58:25.419 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T15:58:25.419 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:58:25.419 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:58:25.419 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:58:25.419 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:58:25.419 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:58:25.419 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:58:25.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:25.420 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T15:58:25.436 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:58:25.436 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:58:25.436 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:58:25.436 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:25.436 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T15:58:25.436 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 37014sec preferred_lft 37014sec 2024-06-29T15:58:25.437 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:58:25.438 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:58:25.439 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T15:58:25.439 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:58:25.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:25.439 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:58:25.500 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T15:58:25.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:25.501 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T15:58:25.558 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T15:58:25.558 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:58:25.558 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T15:58:25.558 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T15:58:25.558 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:58:25.558 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:58:25.559 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T15:58:25.559 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T15:58:25.559 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T15:58:25.559 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T15:58:25.559 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T15:58:25.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:58:25.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:58:25.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:25.684 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T15:58:25.699 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:58:25.699 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:58:25.700 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:58:25.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:25.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T15:58:25.765 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:58:25.765 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:58:25.820 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:58:25.820 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:58:25.821 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:58:25.876 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T15:58:25.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:25.876 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:58:25.969 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:58:25.970 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:58:25.991 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:58:25.991 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:58:25.991 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T15:58:25.991 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T15:58:25.991 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T15:58:25.992 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:58:25.992 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:58:25.992 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:58:25.992 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:58:25.992 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:58:25.993 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:58:25.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:25.994 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:58:26.019 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:58:26.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:58:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:58:26.122 INFO:teuthology.orchestra.run.smithi081.stdout:client12249 2024-06-29T15:58:26.122 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:58:26.122 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:26.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-06-29T15:58:26.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.295+0000 7f271b53c640 1 -- 172.21.15.81:0/3026825962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000043a0 msgr2=0x7f27000048c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.295+0000 7f271b53c640 1 --2- 172.21.15.81:0/3026825962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000043a0 0x7f27000048c0 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f2704001730 tx=0x7f2700005830 comp rx=0 tx=0).stop 2024-06-29T15:58:26.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.295+0000 7f2719ad9640 1 -- 172.21.15.81:0/3026825962 reap_dead start 2024-06-29T15:58:26.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.295+0000 7f271b53c640 1 -- 172.21.15.81:0/3026825962 shutdown_connections 2024-06-29T15:58:26.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.295+0000 7f271b53c640 1 -- 172.21.15.81:0/3026825962 >> 172.21.15.81:0/3026825962 conn(0x7f27140d08f0 msgr2=0x7f27140d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:26.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.295+0000 7f271b53c640 1 -- 172.21.15.81:0/3026825962 shutdown_connections 2024-06-29T15:58:26.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.295+0000 7f271b53c640 1 -- 172.21.15.81:0/3026825962 wait complete. 2024-06-29T15:58:26.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.297+0000 7f271b53c640 1 Processor -- start 2024-06-29T15:58:26.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.297+0000 7f271b53c640 1 -- start start 2024-06-29T15:58:26.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f271b53c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000043a0 0x7f27141f00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f271b53c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f27141529a0 0x7f271415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f271b53c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f27140af000 0x7f271415a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f271b53c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f271415c020 con 0x7f27000043a0 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f271b53c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f271415bea0 con 0x7f27140af000 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f271b53c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f271415c1a0 con 0x7f27141529a0 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f27192d8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000043a0 0x7f27141f00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f27192d8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000043a0 0x7f27141f00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59932/0 (socket says 172.21.15.81:59932) 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f27192d8640 1 -- 172.21.15.81:0/3723931554 learned_addr learned my addr 172.21.15.81:0/3723931554 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f2718ad7640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f27141529a0 0x7f271415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f2719ad9640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f27140af000 0x7f271415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f27192d8640 1 -- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f27141529a0 msgr2=0x7f271415a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f27192d8640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f27141529a0 0x7f271415a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f27192d8640 1 -- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f27140af000 msgr2=0x7f271415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f27192d8640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f27140af000 0x7f271415a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:26.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.298+0000 7f27192d8640 1 -- 172.21.15.81:0/3723931554 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271415dfb0 con 0x7f27000043a0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.299+0000 7f27192d8640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000043a0 0x7f27141f00b0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f2710006e60 tx=0x7f27100049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.299+0000 7f2709ffb640 1 -- 172.21.15.81:0/3723931554 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2710015020 con 0x7f27000043a0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.299+0000 7f2709ffb640 1 -- 172.21.15.81:0/3723931554 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27100109a0 con 0x7f27000043a0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.299+0000 7f2709ffb640 1 -- 172.21.15.81:0/3723931554 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2710010df0 con 0x7f27000043a0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.299+0000 7f271b53c640 1 -- 172.21.15.81:0/3723931554 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271415e4b0 con 0x7f27000043a0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.299+0000 7f271b53c640 1 -- 172.21.15.81:0/3723931554 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f271415ea70 con 0x7f27000043a0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.300+0000 7f271b53c640 1 -- 172.21.15.81:0/3723931554 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2714075c90 con 0x7f27000043a0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.301+0000 7f2709ffb640 1 -- 172.21.15.81:0/3723931554 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f2710018020 con 0x7f27000043a0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.301+0000 7f2709ffb640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f26f8076540 0x7f26f8078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.302+0000 7f2709ffb640 1 -- 172.21.15.81:0/3723931554 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f27100984d0 con 0x7f27000043a0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.302+0000 7f2718ad7640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f26f8076540 0x7f26f8078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:26.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.302+0000 7f2718ad7640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f26f8076540 0x7f26f8078a50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2704001700 tx=0x7f2704001270 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:26.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.306+0000 7f2709ffb640 1 -- 172.21.15.81:0/3723931554 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2714075c90 con 0x7f27000043a0 2024-06-29T15:58:26.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:26.525+0000 7f271b53c640 1 -- 172.21.15.81:0/3723931554 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f27140af730 con 0x7f27000043a0 2024-06-29T15:58:27.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.344+0000 7f2709ffb640 1 -- 172.21.15.81:0/3723931554 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v130) ==== 136+0+0 (secure 0 0 0) 0x7f271001f030 con 0x7f27000043a0 2024-06-29T15:58:27.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 -- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f26f8076540 msgr2=0x7f26f8078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f26f8076540 0x7f26f8078a50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2704001700 tx=0x7f2704001270 comp rx=0 tx=0).stop 2024-06-29T15:58:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 -- 172.21.15.81:0/3723931554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000043a0 msgr2=0x7f27141f00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000043a0 0x7f27141f00b0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f2710006e60 tx=0x7f27100049d0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 -- 172.21.15.81:0/3723931554 shutdown_connections 2024-06-29T15:58:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f26f8076540 0x7f26f8078a50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f27140af000 0x7f271415a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f27141529a0 0x7f271415a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 --2- 172.21.15.81:0/3723931554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000043a0 0x7f27141f00b0 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 -- 172.21.15.81:0/3723931554 >> 172.21.15.81:0/3723931554 conn(0x7f27140d08f0 msgr2=0x7f27140b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 -- 172.21.15.81:0/3723931554 shutdown_connections 2024-06-29T15:58:27.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.348+0000 7f271b53c640 1 -- 172.21.15.81:0/3723931554 wait complete. 2024-06-29T15:58:27.362 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:27.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.478+0000 7f67c2272640 1 -- 172.21.15.81:0/4201869108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67bc15c030 msgr2=0x7f67bc06c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.478+0000 7f67c2272640 1 --2- 172.21.15.81:0/4201869108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67bc15c030 0x7f67bc06c2f0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f67a4009830 tx=0x7f67a402a510 comp rx=0 tx=0).stop 2024-06-29T15:58:27.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.478+0000 7f67c2272640 1 -- 172.21.15.81:0/4201869108 shutdown_connections 2024-06-29T15:58:27.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.478+0000 7f67c2272640 1 --2- 172.21.15.81:0/4201869108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67bc15c030 0x7f67bc06c2f0 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.478+0000 7f67c2272640 1 -- 172.21.15.81:0/4201869108 >> 172.21.15.81:0/4201869108 conn(0x7f67bc0d0930 msgr2=0x7f67bc0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.478+0000 7f67c2272640 1 -- 172.21.15.81:0/4201869108 shutdown_connections 2024-06-29T15:58:27.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.478+0000 7f67c2272640 1 -- 172.21.15.81:0/4201869108 wait complete. 2024-06-29T15:58:27.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.480+0000 7f67c2272640 1 Processor -- start 2024-06-29T15:58:27.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.480+0000 7f67c2272640 1 -- start start 2024-06-29T15:58:27.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.481+0000 7f67c2272640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67bc15c030 0x7f67bc0a95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.481+0000 7f67c2272640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67bc15c440 0x7f67bc14a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.481+0000 7f67c2272640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67bc15d750 0x7f67bc155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.481+0000 7f67c2272640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc0d0030 con 0x7f67bc15c030 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.481+0000 7f67c2272640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc0cfeb0 con 0x7f67bc15d750 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.481+0000 7f67c2272640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc0d01b0 con 0x7f67bc15c440 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.481+0000 7f67bbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67bc15c030 0x7f67bc0a95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.481+0000 7f67bbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67bc15c030 0x7f67bc0a95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59942/0 (socket says 172.21.15.81:59942) 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.481+0000 7f67bbfff640 1 -- 172.21.15.81:0/1895067602 learned_addr learned my addr 172.21.15.81:0/1895067602 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67c080f640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67bc15d750 0x7f67bc155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67b3fff640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67bc15c440 0x7f67bc14a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67bbfff640 1 -- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67bc15c440 msgr2=0x7f67bc14a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67bbfff640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67bc15c440 0x7f67bc14a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67bbfff640 1 -- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67bc15d750 msgr2=0x7f67bc155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67bbfff640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67bc15d750 0x7f67bc155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67bbfff640 1 -- 172.21.15.81:0/1895067602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc156dd0 con 0x7f67bc15c030 2024-06-29T15:58:27.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67c080f640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67bc15d750 0x7f67bc155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:27.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67b3fff640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67bc15c440 0x7f67bc14a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:27.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67bbfff640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67bc15c030 0x7f67bc0a95c0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00c3a0 tx=0x7f67ac00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67b97fa640 1 -- 172.21.15.81:0/1895067602 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac011790 con 0x7f67bc15c030 2024-06-29T15:58:27.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67b97fa640 1 -- 172.21.15.81:0/1895067602 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67ac017070 con 0x7f67bc15c030 2024-06-29T15:58:27.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67c2272640 1 -- 172.21.15.81:0/1895067602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc1570b0 con 0x7f67bc15c030 2024-06-29T15:58:27.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67b97fa640 1 -- 172.21.15.81:0/1895067602 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac011cc0 con 0x7f67bc15c030 2024-06-29T15:58:27.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.482+0000 7f67c2272640 1 -- 172.21.15.81:0/1895067602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc157670 con 0x7f67bc15c030 2024-06-29T15:58:27.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.483+0000 7f67c2272640 1 -- 172.21.15.81:0/1895067602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67bc06e2a0 con 0x7f67bc15c030 2024-06-29T15:58:27.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.484+0000 7f67b97fa640 1 -- 172.21.15.81:0/1895067602 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f67ac01c460 con 0x7f67bc15c030 2024-06-29T15:58:27.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.484+0000 7f67b97fa640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f679c076510 0x7f679c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.484+0000 7f67b97fa640 1 -- 172.21.15.81:0/1895067602 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f67ac097a00 con 0x7f67bc15c030 2024-06-29T15:58:27.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.485+0000 7f67b3fff640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f679c076510 0x7f679c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.485+0000 7f67b3fff640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f679c076510 0x7f679c078a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67a4034000 tx=0x7f67a4004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.489+0000 7f67b97fa640 1 -- 172.21.15.81:0/1895067602 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f67bc06e2a0 con 0x7f67bc15c030 2024-06-29T15:58:27.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.670+0000 7f67c2272640 1 -- 172.21.15.81:0/1895067602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67bc06c2f0 con 0x7f67bc15c030 2024-06-29T15:58:27.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.671+0000 7f67b97fa640 1 -- 172.21.15.81:0/1895067602 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) ==== 74+0+16811 (secure 0 0 0) 0x7f67ac060fc0 con 0x7f67bc15c030 2024-06-29T15:58:27.673 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:27.673 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":233,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:20.296467+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:27.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 -- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f679c076510 msgr2=0x7f679c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f679c076510 0x7f679c078a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67a4034000 tx=0x7f67a4004730 comp rx=0 tx=0).stop 2024-06-29T15:58:27.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 -- 172.21.15.81:0/1895067602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67bc15c030 msgr2=0x7f67bc0a95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67bc15c030 0x7f67bc0a95c0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00c3a0 tx=0x7f67ac00c860 comp rx=0 tx=0).stop 2024-06-29T15:58:27.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 -- 172.21.15.81:0/1895067602 shutdown_connections 2024-06-29T15:58:27.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f679c076510 0x7f679c078a20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f67bc15d750 0x7f67bc155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f67bc15c440 0x7f67bc14a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 --2- 172.21.15.81:0/1895067602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67bc15c030 0x7f67bc0a95c0 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 -- 172.21.15.81:0/1895067602 >> 172.21.15.81:0/1895067602 conn(0x7f67bc0d0930 msgr2=0x7f67bc15ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 -- 172.21.15.81:0/1895067602 shutdown_connections 2024-06-29T15:58:27.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.673+0000 7f67c2272640 1 -- 172.21.15.81:0/1895067602 wait complete. 2024-06-29T15:58:27.686 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:58:16.949415+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:58:17.618219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:27.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:27.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.802+0000 7f44ac481640 1 -- 172.21.15.81:0/3356997682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44a40b5f50 msgr2=0x7f44a4166870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.802+0000 7f44ac481640 1 --2- 172.21.15.81:0/3356997682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44a40b5f50 0x7f44a4166870 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f449c005fd0 tx=0x7f449c02a820 comp rx=0 tx=0).stop 2024-06-29T15:58:27.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.802+0000 7f44ac481640 1 -- 172.21.15.81:0/3356997682 shutdown_connections 2024-06-29T15:58:27.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.802+0000 7f44ac481640 1 --2- 172.21.15.81:0/3356997682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44a40b5f50 0x7f44a4166870 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.802+0000 7f44ac481640 1 -- 172.21.15.81:0/3356997682 >> 172.21.15.81:0/3356997682 conn(0x7f44a40b3680 msgr2=0x7f44a40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.802+0000 7f44ac481640 1 -- 172.21.15.81:0/3356997682 shutdown_connections 2024-06-29T15:58:27.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.802+0000 7f44ac481640 1 -- 172.21.15.81:0/3356997682 wait complete. 2024-06-29T15:58:27.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.804+0000 7f44ac481640 1 Processor -- start 2024-06-29T15:58:27.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.804+0000 7f44ac481640 1 -- start start 2024-06-29T15:58:27.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44ac481640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44a40b3200 0x7f44a4154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44ac481640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44a40b5f50 0x7f44a414b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44ac481640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a40d0230 0x7f44a414bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44ac481640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f44a4167280 con 0x7f44a40b5f50 2024-06-29T15:58:27.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44ac481640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f44a4167100 con 0x7f44a40d0230 2024-06-29T15:58:27.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44ac481640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f44a4167400 con 0x7f44a40b3200 2024-06-29T15:58:27.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44a9a1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44a40b5f50 0x7f44a414b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44a9a1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44a40b5f50 0x7f44a414b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59956/0 (socket says 172.21.15.81:59956) 2024-06-29T15:58:27.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44a9a1c640 1 -- 172.21.15.81:0/3873912654 learned_addr learned my addr 172.21.15.81:0/3873912654 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:27.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44aaa1e640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a40d0230 0x7f44a414bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44aa21d640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44a40b3200 0x7f44a4154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44a9a1c640 1 -- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44a40b3200 msgr2=0x7f44a4154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44a9a1c640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44a40b3200 0x7f44a4154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44a9a1c640 1 -- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a40d0230 msgr2=0x7f44a414bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44a9a1c640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a40d0230 0x7f44a414bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44a9a1c640 1 -- 172.21.15.81:0/3873912654 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a40cadb0 con 0x7f44a40b5f50 2024-06-29T15:58:27.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44aa21d640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44a40b3200 0x7f44a4154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:27.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44aaa1e640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a40d0230 0x7f44a414bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:27.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f44a9a1c640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44a40b5f50 0x7f44a414b980 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f449400b3f0 tx=0x7f449400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.805+0000 7f449affd640 1 -- 172.21.15.81:0/3873912654 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4494015020 con 0x7f44a40b5f50 2024-06-29T15:58:27.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.806+0000 7f449affd640 1 -- 172.21.15.81:0/3873912654 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4494011920 con 0x7f44a40b5f50 2024-06-29T15:58:27.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.806+0000 7f44ac481640 1 -- 172.21.15.81:0/3873912654 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a40cb090 con 0x7f44a40b5f50 2024-06-29T15:58:27.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.806+0000 7f449affd640 1 -- 172.21.15.81:0/3873912654 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4494011d70 con 0x7f44a40b5f50 2024-06-29T15:58:27.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.806+0000 7f44ac481640 1 -- 172.21.15.81:0/3873912654 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a414a210 con 0x7f44a40b5f50 2024-06-29T15:58:27.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.807+0000 7f44ac481640 1 -- 172.21.15.81:0/3873912654 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a40d1000 con 0x7f44a40b5f50 2024-06-29T15:58:27.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.807+0000 7f449affd640 1 -- 172.21.15.81:0/3873912654 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f4494018020 con 0x7f44a40b5f50 2024-06-29T15:58:27.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.808+0000 7f449affd640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4488076510 0x7f4488078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:27.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.808+0000 7f449affd640 1 -- 172.21.15.81:0/3873912654 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f4494097990 con 0x7f44a40b5f50 2024-06-29T15:58:27.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.808+0000 7f44aa21d640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4488076510 0x7f4488078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:27.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.809+0000 7f44aa21d640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4488076510 0x7f4488078a20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f44a0000f60 tx=0x7f44a0009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:27.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.813+0000 7f449affd640 1 -- 172.21.15.81:0/3873912654 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f44a40d1000 con 0x7f44a40b5f50 2024-06-29T15:58:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.990+0000 7f44ac481640 1 -- 172.21.15.81:0/3873912654 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f44a40b7460 con 0x7f44a40b5f50 2024-06-29T15:58:27.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.991+0000 7f449affd640 1 -- 172.21.15.81:0/3873912654 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v233) ==== 93+0+10 (secure 0 0 0) 0x7f44a40b7460 con 0x7f44a40b5f50 2024-06-29T15:58:27.992 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:58:27.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 -- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4488076510 msgr2=0x7f4488078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4488076510 0x7f4488078a20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f44a0000f60 tx=0x7f44a0009040 comp rx=0 tx=0).stop 2024-06-29T15:58:27.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 -- 172.21.15.81:0/3873912654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44a40b5f50 msgr2=0x7f44a414b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:27.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44a40b5f50 0x7f44a414b980 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f449400b3f0 tx=0x7f449400b8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 -- 172.21.15.81:0/3873912654 shutdown_connections 2024-06-29T15:58:27.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4488076510 0x7f4488078a20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44a40d0230 0x7f44a414bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44a40b5f50 0x7f44a414b980 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 --2- 172.21.15.81:0/3873912654 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f44a40b3200 0x7f44a4154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 -- 172.21.15.81:0/3873912654 >> 172.21.15.81:0/3873912654 conn(0x7f44a40b3680 msgr2=0x7f44a40cf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 -- 172.21.15.81:0/3873912654 shutdown_connections 2024-06-29T15:58:27.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:27.993+0000 7f44ac481640 1 -- 172.21.15.81:0/3873912654 wait complete. 2024-06-29T15:58:28.005 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:58:28.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.121+0000 7fc6efda1640 1 -- 172.21.15.81:0/489651249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6e80b6eb0 msgr2=0x7fc6e80d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.121+0000 7fc6efda1640 1 --2- 172.21.15.81:0/489651249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6e80b6eb0 0x7fc6e80d02d0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80b41b0 tx=0x7fc6d802a470 comp rx=0 tx=0).stop 2024-06-29T15:58:28.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.121+0000 7fc6efda1640 1 -- 172.21.15.81:0/489651249 shutdown_connections 2024-06-29T15:58:28.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.121+0000 7fc6efda1640 1 --2- 172.21.15.81:0/489651249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6e80ce510 0x7fc6e80ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.121+0000 7fc6efda1640 1 --2- 172.21.15.81:0/489651249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6e80b6eb0 0x7fc6e80d02d0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.121+0000 7fc6efda1640 1 -- 172.21.15.81:0/489651249 >> 172.21.15.81:0/489651249 conn(0x7fc6e80b3510 msgr2=0x7fc6e80b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:28.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.121+0000 7fc6efda1640 1 -- 172.21.15.81:0/489651249 shutdown_connections 2024-06-29T15:58:28.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.121+0000 7fc6efda1640 1 -- 172.21.15.81:0/489651249 wait complete. 2024-06-29T15:58:28.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.123+0000 7fc6efda1640 1 Processor -- start 2024-06-29T15:58:28.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.123+0000 7fc6efda1640 1 -- start start 2024-06-29T15:58:28.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6efda1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6e80b4810 0x7fc6e8162050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6efda1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6e80b6eb0 0x7fc6e80af630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6efda1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6e80ce510 0x7fc6e815a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6efda1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e80cd480 con 0x7fc6e80b4810 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6efda1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e80b29c0 con 0x7fc6e80ce510 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6efda1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e80cd600 con 0x7fc6e80b6eb0 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6edb3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6e80b4810 0x7fc6e8162050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6edb3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6e80b4810 0x7fc6e8162050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59974/0 (socket says 172.21.15.81:59974) 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6edb3d640 1 -- 172.21.15.81:0/1645846215 learned_addr learned my addr 172.21.15.81:0/1645846215 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6ed33c640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6e80b6eb0 0x7fc6e80af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6ee33e640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6e80ce510 0x7fc6e815a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6edb3d640 1 -- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6e80b6eb0 msgr2=0x7fc6e80af630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6edb3d640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6e80b6eb0 0x7fc6e80af630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6edb3d640 1 -- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6e80ce510 msgr2=0x7fc6e815a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6edb3d640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6e80ce510 0x7fc6e815a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6edb3d640 1 -- 172.21.15.81:0/1645846215 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e81628e0 con 0x7fc6e80b4810 2024-06-29T15:58:28.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6ed33c640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6e80b6eb0 0x7fc6e80af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:28.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.124+0000 7fc6ee33e640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6e80ce510 0x7fc6e815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:28.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.125+0000 7fc6edb3d640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6e80b4810 0x7fc6e8162050 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fc6d802aa60 tx=0x7fc6d8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:28.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.125+0000 7fc6e67fc640 1 -- 172.21.15.81:0/1645846215 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8031d60 con 0x7fc6e80b4810 2024-06-29T15:58:28.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.125+0000 7fc6efda1640 1 -- 172.21.15.81:0/1645846215 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e8162bc0 con 0x7fc6e80b4810 2024-06-29T15:58:28.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.125+0000 7fc6e67fc640 1 -- 172.21.15.81:0/1645846215 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6d8033e10 con 0x7fc6e80b4810 2024-06-29T15:58:28.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.125+0000 7fc6e67fc640 1 -- 172.21.15.81:0/1645846215 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8005260 con 0x7fc6e80b4810 2024-06-29T15:58:28.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.125+0000 7fc6efda1640 1 -- 172.21.15.81:0/1645846215 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6e81f6160 con 0x7fc6e80b4810 2024-06-29T15:58:28.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.126+0000 7fc6efda1640 1 -- 172.21.15.81:0/1645846215 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6e80b3e50 con 0x7fc6e80b4810 2024-06-29T15:58:28.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.127+0000 7fc6e67fc640 1 -- 172.21.15.81:0/1645846215 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fc6d8005400 con 0x7fc6e80b4810 2024-06-29T15:58:28.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.127+0000 7fc6e67fc640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fc6d4076510 0x7fc6d4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.127+0000 7fc6e67fc640 1 -- 172.21.15.81:0/1645846215 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7fc6d80b8e60 con 0x7fc6e80b4810 2024-06-29T15:58:28.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.127+0000 7fc6ed33c640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fc6d4076510 0x7fc6d4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.128+0000 7fc6ed33c640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fc6d4076510 0x7fc6d4078a20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80cbae0 tx=0x7fc6dc005f10 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:28.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.132+0000 7fc6e67fc640 1 -- 172.21.15.81:0/1645846215 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc6e80b3e50 con 0x7fc6e80b4810 2024-06-29T15:58:28.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.312+0000 7fc6efda1640 1 -- 172.21.15.81:0/1645846215 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc6e80aec20 con 0x7fc6e80b4810 2024-06-29T15:58:28.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.312+0000 7fc6e67fc640 1 -- 172.21.15.81:0/1645846215 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v233) ==== 104+0+11 (secure 0 0 0) 0x7fc6d803f030 con 0x7fc6e80b4810 2024-06-29T15:58:28.314 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:58:28.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 -- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fc6d4076510 msgr2=0x7fc6d4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fc6d4076510 0x7fc6d4078a20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80cbae0 tx=0x7fc6dc005f10 comp rx=0 tx=0).stop 2024-06-29T15:58:28.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 -- 172.21.15.81:0/1645846215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6e80b4810 msgr2=0x7fc6e8162050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6e80b4810 0x7fc6e8162050 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fc6d802aa60 tx=0x7fc6d8033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:28.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 -- 172.21.15.81:0/1645846215 shutdown_connections 2024-06-29T15:58:28.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fc6d4076510 0x7fc6d4078a20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc6e80ce510 0x7fc6e815a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc6e80b6eb0 0x7fc6e80af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 --2- 172.21.15.81:0/1645846215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6e80b4810 0x7fc6e8162050 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 -- 172.21.15.81:0/1645846215 >> 172.21.15.81:0/1645846215 conn(0x7fc6e80b3510 msgr2=0x7fc6e81637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:28.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 -- 172.21.15.81:0/1645846215 shutdown_connections 2024-06-29T15:58:28.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.314+0000 7fc6efda1640 1 -- 172.21.15.81:0/1645846215 wait complete. 2024-06-29T15:58:28.327 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:58:28.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.442+0000 7f463264e640 1 -- 172.21.15.81:0/1460713471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462c06fa90 msgr2=0x7f462c06c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.442+0000 7f463264e640 1 --2- 172.21.15.81:0/1460713471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462c06fa90 0x7f462c06c830 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f462c073990 tx=0x7f462402aa30 comp rx=0 tx=0).stop 2024-06-29T15:58:28.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.442+0000 7f463264e640 1 -- 172.21.15.81:0/1460713471 shutdown_connections 2024-06-29T15:58:28.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.442+0000 7f463264e640 1 --2- 172.21.15.81:0/1460713471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462c06fa90 0x7f462c06c830 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.442+0000 7f463264e640 1 --2- 172.21.15.81:0/1460713471 >> v2:172.21.15.167:3300/0 conn(0x7f462c06f680 0x7f462c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.442+0000 7f463264e640 1 --2- 172.21.15.81:0/1460713471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f462c074520 0x7f462c06f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.442+0000 7f463264e640 1 -- 172.21.15.81:0/1460713471 >> 172.21.15.81:0/1460713471 conn(0x7f462c072950 msgr2=0x7f462c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:28.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.442+0000 7f463264e640 1 -- 172.21.15.81:0/1460713471 shutdown_connections 2024-06-29T15:58:28.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.442+0000 7f463264e640 1 -- 172.21.15.81:0/1460713471 wait complete. 2024-06-29T15:58:28.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.444+0000 7f463264e640 1 Processor -- start 2024-06-29T15:58:28.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.444+0000 7f463264e640 1 -- start start 2024-06-29T15:58:28.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.445+0000 7f463264e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462c06f680 0x7f462c14ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.445+0000 7f463264e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f462c06fa90 0x7f462c0d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.445+0000 7f463264e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f462c074520 0x7f462c0d0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.445+0000 7f463264e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f462c06cd70 con 0x7f462c06f680 2024-06-29T15:58:28.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.445+0000 7f463264e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f462c15b220 con 0x7f462c074520 2024-06-29T15:58:28.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.445+0000 7f463264e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f462c06cef0 con 0x7f462c06fa90 2024-06-29T15:58:28.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462c06f680 0x7f462c14ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462c06f680 0x7f462c14ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59992/0 (socket says 172.21.15.81:59992) 2024-06-29T15:58:28.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462bfff640 1 -- 172.21.15.81:0/2911877003 learned_addr learned my addr 172.21.15.81:0/2911877003 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:28.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f4630beb640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f462c074520 0x7f462c0d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462b7fe640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f462c06fa90 0x7f462c0d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462bfff640 1 -- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f462c06fa90 msgr2=0x7f462c0d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462bfff640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f462c06fa90 0x7f462c0d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462bfff640 1 -- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f462c074520 msgr2=0x7f462c0d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462bfff640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f462c074520 0x7f462c0d0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462bfff640 1 -- 172.21.15.81:0/2911877003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462c0cfae0 con 0x7f462c06f680 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f4630beb640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f462c074520 0x7f462c0d0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462b7fe640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f462c06fa90 0x7f462c0d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f462bfff640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462c06f680 0x7f462c14ea30 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f461c00c9a0 tx=0x7f461c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f4628ff9640 1 -- 172.21.15.81:0/2911877003 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c007b50 con 0x7f462c06f680 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f4628ff9640 1 -- 172.21.15.81:0/2911877003 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f461c007e00 con 0x7f462c06f680 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f463264e640 1 -- 172.21.15.81:0/2911877003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f462c0cf800 con 0x7f462c06f680 2024-06-29T15:58:28.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f4628ff9640 1 -- 172.21.15.81:0/2911877003 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c005280 con 0x7f462c06f680 2024-06-29T15:58:28.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.446+0000 7f463264e640 1 -- 172.21.15.81:0/2911877003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f462c1f68a0 con 0x7f462c06f680 2024-06-29T15:58:28.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.447+0000 7f463264e640 1 -- 172.21.15.81:0/2911877003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f462c0cf360 con 0x7f462c06f680 2024-06-29T15:58:28.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.448+0000 7f4628ff9640 1 -- 172.21.15.81:0/2911877003 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f461c005420 con 0x7f462c06f680 2024-06-29T15:58:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.448+0000 7f4628ff9640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4618076510 0x7f4618078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.448+0000 7f4628ff9640 1 -- 172.21.15.81:0/2911877003 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f461c098050 con 0x7f462c06f680 2024-06-29T15:58:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.449+0000 7f462b7fe640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4618076510 0x7f4618078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.449+0000 7f462b7fe640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4618076510 0x7f4618078a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f462c1f88b0 tx=0x7f4620005d90 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:28.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.453+0000 7f4628ff9640 1 -- 172.21.15.81:0/2911877003 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f462c0cf360 con 0x7f462c06f680 2024-06-29T15:58:28.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.632+0000 7f463264e640 1 -- 172.21.15.81:0/2911877003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f462c06f140 con 0x7f462c06f680 2024-06-29T15:58:28.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.632+0000 7f4628ff9640 1 -- 172.21.15.81:0/2911877003 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v233) ==== 100+0+11 (secure 0 0 0) 0x7f461c014030 con 0x7f462c06f680 2024-06-29T15:58:28.634 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:58:28.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.634+0000 7f463264e640 1 -- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4618076510 msgr2=0x7f4618078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.634+0000 7f463264e640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4618076510 0x7f4618078a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f462c1f88b0 tx=0x7f4620005d90 comp rx=0 tx=0).stop 2024-06-29T15:58:28.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.634+0000 7f463264e640 1 -- 172.21.15.81:0/2911877003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462c06f680 msgr2=0x7f462c14ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.634+0000 7f463264e640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462c06f680 0x7f462c14ea30 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f461c00c9a0 tx=0x7f461c00ce60 comp rx=0 tx=0).stop 2024-06-29T15:58:28.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.634+0000 7f463264e640 1 -- 172.21.15.81:0/2911877003 shutdown_connections 2024-06-29T15:58:28.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.634+0000 7f463264e640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4618076510 0x7f4618078a20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.634+0000 7f463264e640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f462c074520 0x7f462c0d0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.634+0000 7f463264e640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f462c06fa90 0x7f462c0d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.634+0000 7f463264e640 1 --2- 172.21.15.81:0/2911877003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462c06f680 0x7f462c14ea30 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.634+0000 7f463264e640 1 -- 172.21.15.81:0/2911877003 >> 172.21.15.81:0/2911877003 conn(0x7f462c072950 msgr2=0x7f462c157d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:28.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.635+0000 7f463264e640 1 -- 172.21.15.81:0/2911877003 shutdown_connections 2024-06-29T15:58:28.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.635+0000 7f463264e640 1 -- 172.21.15.81:0/2911877003 wait complete. 2024-06-29T15:58:28.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:58:28.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.761+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1168190543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa260003a50 msgr2=0x7fa260003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.761+0000 7fa27c0a7640 1 --2- 172.21.15.81:0/1168190543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa260003a50 0x7fa260003ec0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fa2700016c0 tx=0x7fa260005830 comp rx=0 tx=0).stop 2024-06-29T15:58:28.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.761+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1168190543 shutdown_connections 2024-06-29T15:58:28.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.761+0000 7fa27c0a7640 1 --2- 172.21.15.81:0/1168190543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa260003a50 0x7fa260003ec0 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.761+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1168190543 >> 172.21.15.81:0/1168190543 conn(0x7fa2740b3510 msgr2=0x7fa2740b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:28.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.761+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1168190543 shutdown_connections 2024-06-29T15:58:28.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.761+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1168190543 wait complete. 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.762+0000 7fa27c0a7640 1 Processor -- start 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.763+0000 7fa27c0a7640 1 -- start start 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa27c0a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa260003a50 0x7fa2741e0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa27c0a7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2741f2c90 0x7fa2740cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa27c0a7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2740c8a30 0x7fa2740cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa27c0a7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa2740ce690 con 0x7fa260003a50 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa27c0a7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa2740ce510 con 0x7fa2741f2c90 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa27c0a7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa2740ce810 con 0x7fa2740c8a30 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279e43640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa260003a50 0x7fa2741e0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279e43640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa260003a50 0x7fa2741e0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60008/0 (socket says 172.21.15.81:60008) 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279e43640 1 -- 172.21.15.81:0/1946458892 learned_addr learned my addr 172.21.15.81:0/1946458892 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:28.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279642640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2741f2c90 0x7fa2740cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa27a644640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2740c8a30 0x7fa2740cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279e43640 1 -- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2740c8a30 msgr2=0x7fa2740cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279e43640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2740c8a30 0x7fa2740cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279e43640 1 -- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2741f2c90 msgr2=0x7fa2740cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279e43640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2741f2c90 0x7fa2740cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279e43640 1 -- 172.21.15.81:0/1946458892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2741e14a0 con 0x7fa260003a50 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279642640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2741f2c90 0x7fa2740cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa279e43640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa260003a50 0x7fa2741e0f60 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fa2700017e0 tx=0x7fa2700063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.764+0000 7fa26a7fc640 1 -- 172.21.15.81:0/1946458892 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa270042070 con 0x7fa260003a50 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.765+0000 7fa26a7fc640 1 -- 172.21.15.81:0/1946458892 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa270039070 con 0x7fa260003a50 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.765+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1946458892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2741f3c90 con 0x7fa260003a50 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.765+0000 7fa26a7fc640 1 -- 172.21.15.81:0/1946458892 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa270003d70 con 0x7fa260003a50 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.765+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1946458892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2741f36d0 con 0x7fa260003a50 2024-06-29T15:58:28.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.765+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1946458892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2740d0320 con 0x7fa260003a50 2024-06-29T15:58:28.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.766+0000 7fa26a7fc640 1 -- 172.21.15.81:0/1946458892 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fa27003b070 con 0x7fa260003a50 2024-06-29T15:58:28.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.767+0000 7fa26a7fc640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa258076510 0x7fa258078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:28.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.767+0000 7fa26a7fc640 1 -- 172.21.15.81:0/1946458892 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7fa2700b71f0 con 0x7fa260003a50 2024-06-29T15:58:28.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.767+0000 7fa279642640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa258076510 0x7fa258078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:28.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.767+0000 7fa279642640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa258076510 0x7fa258078a20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2740c93f0 tx=0x7fa264001cb0 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:28.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.771+0000 7fa26a7fc640 1 -- 172.21.15.81:0/1946458892 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa2700808d0 con 0x7fa260003a50 2024-06-29T15:58:28.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.956+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1946458892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa2740cdf00 con 0x7fa260003a50 2024-06-29T15:58:28.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.957+0000 7fa26a7fc640 1 -- 172.21.15.81:0/1946458892 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v233) ==== 84+0+209 (secure 0 0 0) 0x7fa2700801d0 con 0x7fa260003a50 2024-06-29T15:58:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:28.958 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:58:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T15:58:28.958 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T15:58:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 30, 2024-06-29T15:58:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": "cephfs_metadata" 2024-06-29T15:58:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 31, 2024-06-29T15:58:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": "cephfs_data" 2024-06-29T15:58:28.959 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:28.959 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:58:28.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.958+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa258076510 msgr2=0x7fa258078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.958+0000 7fa27c0a7640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa258076510 0x7fa258078a20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2740c93f0 tx=0x7fa264001cb0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.958+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1946458892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa260003a50 msgr2=0x7fa2741e0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:28.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.958+0000 7fa27c0a7640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa260003a50 0x7fa2741e0f60 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fa2700017e0 tx=0x7fa2700063a0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.959+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1946458892 shutdown_connections 2024-06-29T15:58:28.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.959+0000 7fa27c0a7640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa258076510 0x7fa258078a20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.959+0000 7fa27c0a7640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2740c8a30 0x7fa2740cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.959+0000 7fa27c0a7640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2741f2c90 0x7fa2740cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.959+0000 7fa27c0a7640 1 --2- 172.21.15.81:0/1946458892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa260003a50 0x7fa2741e0f60 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:28.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.959+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1946458892 >> 172.21.15.81:0/1946458892 conn(0x7fa2740b3510 msgr2=0x7fa2740b5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:28.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.959+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1946458892 shutdown_connections 2024-06-29T15:58:28.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:28.959+0000 7fa27c0a7640 1 -- 172.21.15.81:0/1946458892 wait complete. 2024-06-29T15:58:28.971 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2024-06-29T15:58:28.972 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2024-06-29T15:58:29.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.088+0000 7f0c32ad6640 1 -- 172.21.15.81:0/1873346935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c2c0b1a90 msgr2=0x7f0c2c0b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.088+0000 7f0c32ad6640 1 --2- 172.21.15.81:0/1873346935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c2c0b1a90 0x7f0c2c0b1ea0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c073b40 tx=0x7f0c2402aa10 comp rx=0 tx=0).stop 2024-06-29T15:58:29.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.088+0000 7f0c32ad6640 1 -- 172.21.15.81:0/1873346935 shutdown_connections 2024-06-29T15:58:29.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.088+0000 7f0c32ad6640 1 --2- 172.21.15.81:0/1873346935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c2c0b1a90 0x7f0c2c0b1ea0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.088+0000 7f0c32ad6640 1 --2- 172.21.15.81:0/1873346935 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c2c0b2cb0 0x7f0c2c0d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.088+0000 7f0c32ad6640 1 --2- 172.21.15.81:0/1873346935 >> v2:172.21.15.167:3300/0 conn(0x7f0c2c0ae8f0 0x7f0c2c15c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.088+0000 7f0c32ad6640 1 -- 172.21.15.81:0/1873346935 >> 172.21.15.81:0/1873346935 conn(0x7f0c2c09aa60 msgr2=0x7f0c2c09ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:29.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.088+0000 7f0c32ad6640 1 -- 172.21.15.81:0/1873346935 shutdown_connections 2024-06-29T15:58:29.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.088+0000 7f0c32ad6640 1 -- 172.21.15.81:0/1873346935 wait complete. 2024-06-29T15:58:29.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.089+0000 7f0c32ad6640 1 Processor -- start 2024-06-29T15:58:29.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.090+0000 7f0c32ad6640 1 -- start start 2024-06-29T15:58:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c32ad6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c2c0ae8f0 0x7f0c2c14e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c32ad6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c2c0b1a90 0x7f0c2c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c32ad6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c2c0b2cb0 0x7f0c2c14efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c32ad6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c0ceb80 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c32ad6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c0cea00 con 0x7f0c2c0b1a90 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c32ad6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0c2c0ced00 con 0x7f0c2c0ae8f0 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c31073640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c2c0b2cb0 0x7f0c2c14efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c31073640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c2c0b2cb0 0x7f0c2c14efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60028/0 (socket says 172.21.15.81:60028) 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c31073640 1 -- 172.21.15.81:0/889197668 learned_addr learned my addr 172.21.15.81:0/889197668 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c30872640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c2c0ae8f0 0x7f0c2c14e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c2bfff640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c2c0b1a90 0x7f0c2c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c31073640 1 -- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c2c0ae8f0 msgr2=0x7f0c2c14e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c31073640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c2c0ae8f0 0x7f0c2c14e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c31073640 1 -- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c2c0b1a90 msgr2=0x7f0c2c06ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c31073640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c2c0b1a90 0x7f0c2c06ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.091+0000 7f0c31073640 1 -- 172.21.15.81:0/889197668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2c06dda0 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.092+0000 7f0c31073640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c2c0b2cb0 0x7f0c2c14efc0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f0c2402b000 tx=0x7f0c24003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:29.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.092+0000 7f0c297fa640 1 -- 172.21.15.81:0/889197668 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c24033050 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.092+0000 7f0c297fa640 1 -- 172.21.15.81:0/889197668 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c24004550 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.092+0000 7f0c297fa640 1 -- 172.21.15.81:0/889197668 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c24038450 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.092+0000 7f0c32ad6640 1 -- 172.21.15.81:0/889197668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c2c06e080 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.092+0000 7f0c32ad6640 1 -- 172.21.15.81:0/889197668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c2c06e640 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.093+0000 7f0c32ad6640 1 -- 172.21.15.81:0/889197668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c2c0b21d0 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.094+0000 7f0c297fa640 1 -- 172.21.15.81:0/889197668 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f0c24036070 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.094+0000 7f0c297fa640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f0c18076540 0x7f0c18078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.094+0000 7f0c297fa640 1 -- 172.21.15.81:0/889197668 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5837+0+0 (secure 0 0 0) 0x7f0c240b9270 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.094+0000 7f0c30872640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f0c18076540 0x7f0c18078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.095+0000 7f0c30872640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f0c18076540 0x7f0c18078a50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c005fd0 tx=0x7f0c1c00c460 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:29.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.099+0000 7f0c297fa640 1 -- 172.21.15.81:0/889197668 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0c2c0b21d0 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.282+0000 7f0c32ad6640 1 -- 172.21.15.81:0/889197668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) -- 0x7f0c2c0d06d0 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.358+0000 7f0c297fa640 1 -- 172.21.15.81:0/889197668 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v234) ==== 144+0+0 (secure 0 0 0) 0x7f0c2c0d06d0 con 0x7f0c2c0b2cb0 2024-06-29T15:58:29.360 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'backup_fs_metadata' created 2024-06-29T15:58:29.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.360+0000 7f0c32ad6640 1 -- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f0c18076540 msgr2=0x7f0c18078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.360+0000 7f0c32ad6640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f0c18076540 0x7f0c18078a50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c005fd0 tx=0x7f0c1c00c460 comp rx=0 tx=0).stop 2024-06-29T15:58:29.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.360+0000 7f0c32ad6640 1 -- 172.21.15.81:0/889197668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c2c0b2cb0 msgr2=0x7f0c2c14efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.360+0000 7f0c32ad6640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c2c0b2cb0 0x7f0c2c14efc0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f0c2402b000 tx=0x7f0c24003f10 comp rx=0 tx=0).stop 2024-06-29T15:58:29.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.361+0000 7f0c32ad6640 1 -- 172.21.15.81:0/889197668 shutdown_connections 2024-06-29T15:58:29.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.361+0000 7f0c32ad6640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f0c18076540 0x7f0c18078a50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.361+0000 7f0c32ad6640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c2c0b2cb0 0x7f0c2c14efc0 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.361+0000 7f0c32ad6640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c2c0b1a90 0x7f0c2c06ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.361+0000 7f0c32ad6640 1 --2- 172.21.15.81:0/889197668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0c2c0ae8f0 0x7f0c2c14e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.361+0000 7f0c32ad6640 1 -- 172.21.15.81:0/889197668 >> 172.21.15.81:0/889197668 conn(0x7f0c2c09aa60 msgr2=0x7f0c2c0af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:29.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.361+0000 7f0c32ad6640 1 -- 172.21.15.81:0/889197668 shutdown_connections 2024-06-29T15:58:29.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.361+0000 7f0c32ad6640 1 -- 172.21.15.81:0/889197668 wait complete. 2024-06-29T15:58:29.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:58:29.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.500+0000 7f6bd5274640 1 -- 172.21.15.81:0/51744989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bd00b9040 msgr2=0x7f6bd00d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.500+0000 7f6bd5274640 1 --2- 172.21.15.81:0/51744989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bd00b9040 0x7f6bd00d0480 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0005fd0 tx=0x7f6bc002a680 comp rx=0 tx=0).stop 2024-06-29T15:58:29.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.500+0000 7f6bd5274640 1 -- 172.21.15.81:0/51744989 shutdown_connections 2024-06-29T15:58:29.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.500+0000 7f6bd5274640 1 --2- 172.21.15.81:0/51744989 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bd00d2a90 0x7f6bd00d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.500+0000 7f6bd5274640 1 --2- 172.21.15.81:0/51744989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bd00b9040 0x7f6bd00d0480 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.500+0000 7f6bd5274640 1 -- 172.21.15.81:0/51744989 >> 172.21.15.81:0/51744989 conn(0x7f6bd009cf30 msgr2=0x7f6bd00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:29.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.501+0000 7f6bd5274640 1 -- 172.21.15.81:0/51744989 shutdown_connections 2024-06-29T15:58:29.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.501+0000 7f6bd5274640 1 -- 172.21.15.81:0/51744989 wait complete. 2024-06-29T15:58:29.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.502+0000 7f6bd5274640 1 Processor -- start 2024-06-29T15:58:29.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.502+0000 7f6bd5274640 1 -- start start 2024-06-29T15:58:29.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bd5274640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bd00b5250 0x7f6bd01f7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bd5274640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bd00b9040 0x7f6bd00b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bd5274640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bd00d2a90 0x7f6bd015fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bd5274640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd00b7d30 con 0x7f6bd00b9040 2024-06-29T15:58:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bd5274640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd00b7bb0 con 0x7f6bd00b5250 2024-06-29T15:58:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bd5274640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6bd00b7eb0 con 0x7f6bd00d2a90 2024-06-29T15:58:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bd00b9040 0x7f6bd00b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bd00b9040 0x7f6bd00b2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60034/0 (socket says 172.21.15.81:60034) 2024-06-29T15:58:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc7fff640 1 -- 172.21.15.81:0/659914948 learned_addr learned my addr 172.21.15.81:0/659914948 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc7fff640 1 -- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bd00d2a90 msgr2=0x7f6bd015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bcf7fe640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bd00d2a90 0x7f6bd015fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc7fff640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bd00d2a90 0x7f6bd015fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc7fff640 1 -- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bd00b5250 msgr2=0x7f6bd01f7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc7fff640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bd00b5250 0x7f6bd01f7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc7fff640 1 -- 172.21.15.81:0/659914948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd00b3960 con 0x7f6bd00b9040 2024-06-29T15:58:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc7fff640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bd00b9040 0x7f6bd00b2c30 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800c3d0 tx=0x7f6bb800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc77fe640 1 -- 172.21.15.81:0/659914948 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8017070 con 0x7f6bd00b9040 2024-06-29T15:58:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc77fe640 1 -- 172.21.15.81:0/659914948 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bb80053c0 con 0x7f6bd00b9040 2024-06-29T15:58:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bd5274640 1 -- 172.21.15.81:0/659914948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd00b3c40 con 0x7f6bd00b9040 2024-06-29T15:58:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bc77fe640 1 -- 172.21.15.81:0/659914948 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8003850 con 0x7f6bd00b9040 2024-06-29T15:58:29.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.504+0000 7f6bd5274640 1 -- 172.21.15.81:0/659914948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd00b4200 con 0x7f6bd00b9040 2024-06-29T15:58:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.506+0000 7f6bd5274640 1 -- 172.21.15.81:0/659914948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd00d46a0 con 0x7f6bd00b9040 2024-06-29T15:58:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.506+0000 7f6bc77fe640 1 -- 172.21.15.81:0/659914948 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f6bb801a070 con 0x7f6bd00b9040 2024-06-29T15:58:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.507+0000 7f6bc77fe640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6bb0076510 0x7f6bb0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.507+0000 7f6bc77fe640 1 -- 172.21.15.81:0/659914948 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(234..234 src has 1..234) ==== 6210+0+0 (secure 0 0 0) 0x7f6bb80970c0 con 0x7f6bd00b9040 2024-06-29T15:58:29.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.507+0000 7f6bceffd640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6bb0076510 0x7f6bb0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:29.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.508+0000 7f6bceffd640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6bb0076510 0x7f6bb0078a20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0033000 tx=0x7f6bc00046c0 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:29.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.512+0000 7f6bc77fe640 1 -- 172.21.15.81:0/659914948 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6bd00d46a0 con 0x7f6bd00b9040 2024-06-29T15:58:29.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:29.693+0000 7f6bd5274640 1 -- 172.21.15.81:0/659914948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6bd00b8440 con 0x7f6bd00b9040 2024-06-29T15:58:30.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.361+0000 7f6bc77fe640 1 -- 172.21.15.81:0/659914948 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v235) ==== 176+0+0 (secure 0 0 0) 0x7f6bb8060500 con 0x7f6bd00b9040 2024-06-29T15:58:30.362 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'backup_fs_data' created 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 -- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6bb0076510 msgr2=0x7f6bb0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6bb0076510 0x7f6bb0078a20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0033000 tx=0x7f6bc00046c0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 -- 172.21.15.81:0/659914948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bd00b9040 msgr2=0x7f6bd00b2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bd00b9040 0x7f6bd00b2c30 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800c3d0 tx=0x7f6bb800c890 comp rx=0 tx=0).stop 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 -- 172.21.15.81:0/659914948 shutdown_connections 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f6bb0076510 0x7f6bb0078a20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bd00d2a90 0x7f6bd015fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bd00b9040 0x7f6bd00b2c30 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 --2- 172.21.15.81:0/659914948 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bd00b5250 0x7f6bd01f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 -- 172.21.15.81:0/659914948 >> 172.21.15.81:0/659914948 conn(0x7f6bd009cf30 msgr2=0x7f6bd00d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:30.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 -- 172.21.15.81:0/659914948 shutdown_connections 2024-06-29T15:58:30.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.363+0000 7f6bd5274640 1 -- 172.21.15.81:0/659914948 wait complete. 2024-06-29T15:58:30.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2024-06-29T15:58:30.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.505+0000 7f72b8b43640 1 -- 172.21.15.81:0/1321629156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b4073650 msgr2=0x7f72b415b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.505+0000 7f72b8b43640 1 --2- 172.21.15.81:0/1321629156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b4073650 0x7f72b415b6e0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f72b4072cf0 tx=0x7f729c02a560 comp rx=0 tx=0).stop 2024-06-29T15:58:30.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.505+0000 7f72b8b43640 1 -- 172.21.15.81:0/1321629156 shutdown_connections 2024-06-29T15:58:30.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.505+0000 7f72b8b43640 1 --2- 172.21.15.81:0/1321629156 >> v2:172.21.15.167:3300/0 conn(0x7f72b415bc20 0x7f72b406ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.505+0000 7f72b8b43640 1 --2- 172.21.15.81:0/1321629156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b4073650 0x7f72b415b6e0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.505+0000 7f72b8b43640 1 -- 172.21.15.81:0/1321629156 >> 172.21.15.81:0/1321629156 conn(0x7f72b40b2920 msgr2=0x7f72b40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:30.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.505+0000 7f72b8b43640 1 -- 172.21.15.81:0/1321629156 shutdown_connections 2024-06-29T15:58:30.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.505+0000 7f72b8b43640 1 -- 172.21.15.81:0/1321629156 wait complete. 2024-06-29T15:58:30.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.507+0000 7f72b8b43640 1 Processor -- start 2024-06-29T15:58:30.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.507+0000 7f72b8b43640 1 -- start start 2024-06-29T15:58:30.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72b8b43640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72b40cd150 0x7f72b40d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72b8b43640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b415bc20 0x7f72b41f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72b8b43640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72b4153970 0x7f72b4153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72b8b43640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72b406f450 con 0x7f72b415bc20 2024-06-29T15:58:30.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72b8b43640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f72b406f2d0 con 0x7f72b4153970 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72b8b43640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f72b406f5d0 con 0x7f72b40cd150 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72a9d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b415bc20 0x7f72b41f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72a9d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b415bc20 0x7f72b41f2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33126/0 (socket says 172.21.15.81:33126) 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72a9d9b640 1 -- 172.21.15.81:0/3046980044 learned_addr learned my addr 172.21.15.81:0/3046980044 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72b2d9d640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72b4153970 0x7f72b4153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72b259c640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72b40cd150 0x7f72b40d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72a9d9b640 1 -- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72b40cd150 msgr2=0x7f72b40d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72a9d9b640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72b40cd150 0x7f72b40d0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72a9d9b640 1 -- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72b4153970 msgr2=0x7f72b4153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72a9d9b640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72b4153970 0x7f72b4153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72a9d9b640 1 -- 172.21.15.81:0/3046980044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b41542c0 con 0x7f72b415bc20 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.509+0000 7f72a9d9b640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b415bc20 0x7f72b41f2f80 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f729c034040 tx=0x7f729c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:30.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.510+0000 7f72ab7fe640 1 -- 172.21.15.81:0/3046980044 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c03e030 con 0x7f72b415bc20 2024-06-29T15:58:30.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.510+0000 7f72ab7fe640 1 -- 172.21.15.81:0/3046980044 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f729c003db0 con 0x7f72b415bc20 2024-06-29T15:58:30.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.510+0000 7f72ab7fe640 1 -- 172.21.15.81:0/3046980044 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c035040 con 0x7f72b415bc20 2024-06-29T15:58:30.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.510+0000 7f72b8b43640 1 -- 172.21.15.81:0/3046980044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b41f3ac0 con 0x7f72b415bc20 2024-06-29T15:58:30.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.510+0000 7f72b8b43640 1 -- 172.21.15.81:0/3046980044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b4151bf0 con 0x7f72b415bc20 2024-06-29T15:58:30.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.511+0000 7f72b8b43640 1 -- 172.21.15.81:0/3046980044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b41f37e0 con 0x7f72b415bc20 2024-06-29T15:58:30.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.512+0000 7f72ab7fe640 1 -- 172.21.15.81:0/3046980044 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f729c041030 con 0x7f72b415bc20 2024-06-29T15:58:30.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.512+0000 7f72ab7fe640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7294076510 0x7f7294078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:30.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.512+0000 7f72ab7fe640 1 -- 172.21.15.81:0/3046980044 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 6595+0+0 (secure 0 0 0) 0x7f729c0b8930 con 0x7f72b415bc20 2024-06-29T15:58:30.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.512+0000 7f72b259c640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7294076510 0x7f7294078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:30.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.513+0000 7f72b259c640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7294076510 0x7f7294078a20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f72a4001720 tx=0x7f72a4009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:30.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.517+0000 7f72ab7fe640 1 -- 172.21.15.81:0/3046980044 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f72b41542c0 con 0x7f72b415bc20 2024-06-29T15:58:30.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:30.735+0000 7f72b8b43640 1 -- 172.21.15.81:0/3046980044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) -- 0x7f72b415dde0 con 0x7f72b415bc20 2024-06-29T15:58:31.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.392+0000 7f72ab7fe640 1 -- 172.21.15.81:0/3046980044 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 32 and data pool 33 v131) ==== 183+0+0 (secure 0 0 0) 0x7f729c081490 con 0x7f72b415bc20 2024-06-29T15:58:31.394 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 32 and data pool 33 2024-06-29T15:58:31.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.400+0000 7f72b8b43640 1 -- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7294076510 msgr2=0x7f7294078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.400+0000 7f72b8b43640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7294076510 0x7f7294078a20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f72a4001720 tx=0x7f72a4009040 comp rx=0 tx=0).stop 2024-06-29T15:58:31.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.400+0000 7f72b8b43640 1 -- 172.21.15.81:0/3046980044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b415bc20 msgr2=0x7f72b41f2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.400+0000 7f72b8b43640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b415bc20 0x7f72b41f2f80 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f729c034040 tx=0x7f729c0038a0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.402+0000 7f72b8b43640 1 -- 172.21.15.81:0/3046980044 shutdown_connections 2024-06-29T15:58:31.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.402+0000 7f72b8b43640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7294076510 0x7f7294078a20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.402+0000 7f72b8b43640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f72b4153970 0x7f72b4153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.402+0000 7f72b8b43640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72b415bc20 0x7f72b41f2f80 secure :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f729c034040 tx=0x7f729c0038a0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.402+0000 7f72b8b43640 1 --2- 172.21.15.81:0/3046980044 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f72b40cd150 0x7f72b40d0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.402+0000 7f72b8b43640 1 -- 172.21.15.81:0/3046980044 >> 172.21.15.81:0/3046980044 conn(0x7f72b40b2920 msgr2=0x7f72b40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:31.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.403+0000 7f72b8b43640 1 -- 172.21.15.81:0/3046980044 shutdown_connections 2024-06-29T15:58:31.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.403+0000 7f72b8b43640 1 -- 172.21.15.81:0/3046980044 wait complete. 2024-06-29T15:58:31.424 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.537+0000 7fab31d91640 1 -- 172.21.15.81:0/671791418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab2c0b5250 msgr2=0x7fab2c0d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.537+0000 7fab31d91640 1 --2- 172.21.15.81:0/671791418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab2c0b5250 0x7fab2c0d09c0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7fab2c08e230 tx=0x7fab1402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.537+0000 7fab31d91640 1 -- 172.21.15.81:0/671791418 shutdown_connections 2024-06-29T15:58:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.537+0000 7fab31d91640 1 --2- 172.21.15.81:0/671791418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab2c0b5250 0x7fab2c0d09c0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.537+0000 7fab31d91640 1 --2- 172.21.15.81:0/671791418 >> v2:172.21.15.167:3300/0 conn(0x7fab2c0b9040 0x7fab2c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.537+0000 7fab31d91640 1 -- 172.21.15.81:0/671791418 >> 172.21.15.81:0/671791418 conn(0x7fab2c09cf30 msgr2=0x7fab2c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:31.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.537+0000 7fab31d91640 1 -- 172.21.15.81:0/671791418 shutdown_connections 2024-06-29T15:58:31.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.538+0000 7fab31d91640 1 -- 172.21.15.81:0/671791418 wait complete. 2024-06-29T15:58:31.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.539+0000 7fab31d91640 1 Processor -- start 2024-06-29T15:58:31.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.539+0000 7fab31d91640 1 -- start start 2024-06-29T15:58:31.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab31d91640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab2c0b5250 0x7fab2c0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab31d91640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab2c0b7290 0x7fab2c15cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab31d91640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab2c0b9040 0x7fab2c1f8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab31d91640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c0d2e20 con 0x7fab2c0b9040 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab31d91640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c0d2ca0 con 0x7fab2c0b7290 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab31d91640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fab2c0b7db0 con 0x7fab2c0b5250 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab2c0b9040 0x7fab2c1f8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab2c0b9040 0x7fab2c1f8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33144/0 (socket says 172.21.15.81:33144) 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2bfff640 1 -- 172.21.15.81:0/622115336 learned_addr learned my addr 172.21.15.81:0/622115336 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab23fff640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab2c0b7290 0x7fab2c15cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2b7fe640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab2c0b5250 0x7fab2c0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2bfff640 1 -- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab2c0b5250 msgr2=0x7fab2c0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2bfff640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab2c0b5250 0x7fab2c0b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2bfff640 1 -- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab2c0b7290 msgr2=0x7fab2c15cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2bfff640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab2c0b7290 0x7fab2c15cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2bfff640 1 -- 172.21.15.81:0/622115336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2c1f2510 con 0x7fab2c0b9040 2024-06-29T15:58:31.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2b7fe640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab2c0b5250 0x7fab2c0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:31.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab2bfff640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab2c0b9040 0x7fab2c1f8140 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7fab2400caa0 tx=0x7fab2400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:31.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab28ff9640 1 -- 172.21.15.81:0/622115336 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24016020 con 0x7fab2c0b9040 2024-06-29T15:58:31.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab28ff9640 1 -- 172.21.15.81:0/622115336 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab2400acf0 con 0x7fab2c0b9040 2024-06-29T15:58:31.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab31d91640 1 -- 172.21.15.81:0/622115336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab2c15d1a0 con 0x7fab2c0b9040 2024-06-29T15:58:31.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab28ff9640 1 -- 172.21.15.81:0/622115336 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24012680 con 0x7fab2c0b9040 2024-06-29T15:58:31.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.541+0000 7fab31d91640 1 -- 172.21.15.81:0/622115336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab2c0b1fc0 con 0x7fab2c0b9040 2024-06-29T15:58:31.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.542+0000 7fab31d91640 1 -- 172.21.15.81:0/622115336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab2c0b82f0 con 0x7fab2c0b9040 2024-06-29T15:58:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.543+0000 7fab28ff9640 1 -- 172.21.15.81:0/622115336 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fab24019020 con 0x7fab2c0b9040 2024-06-29T15:58:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.543+0000 7fab28ff9640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab0c076540 0x7fab0c078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.544+0000 7fab28ff9640 1 -- 172.21.15.81:0/622115336 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 6701+0+0 (secure 0 0 0) 0x7fab240984c0 con 0x7fab2c0b9040 2024-06-29T15:58:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.544+0000 7fab2b7fe640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab0c076540 0x7fab0c078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.544+0000 7fab2b7fe640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab0c076540 0x7fab0c078a50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fab1c005fd0 tx=0x7fab1c00f360 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:31.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.548+0000 7fab28ff9640 1 -- 172.21.15.81:0/622115336 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fab24020020 con 0x7fab2c0b9040 2024-06-29T15:58:31.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.730+0000 7fab31d91640 1 -- 172.21.15.81:0/622115336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fab2c0d2440 con 0x7fab2c0b9040 2024-06-29T15:58:31.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.732+0000 7fab28ff9640 1 -- 172.21.15.81:0/622115336 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) ==== 74+0+20558 (secure 0 0 0) 0x7fab2c15d1a0 con 0x7fab2c0b9040 2024-06-29T15:58:31.733 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:31.733 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":236,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:30.737126+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:29.291185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:29.702735+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:31.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 -- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab0c076540 msgr2=0x7fab0c078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab0c076540 0x7fab0c078a50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fab1c005fd0 tx=0x7fab1c00f360 comp rx=0 tx=0).stop 2024-06-29T15:58:31.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 -- 172.21.15.81:0/622115336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab2c0b9040 msgr2=0x7fab2c1f8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab2c0b9040 0x7fab2c1f8140 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7fab2400caa0 tx=0x7fab2400cf60 comp rx=0 tx=0).stop 2024-06-29T15:58:31.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 -- 172.21.15.81:0/622115336 shutdown_connections 2024-06-29T15:58:31.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fab0c076540 0x7fab0c078a50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab2c0b9040 0x7fab2c1f8140 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fab2c0b7290 0x7fab2c15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 --2- 172.21.15.81:0/622115336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fab2c0b5250 0x7fab2c0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 -- 172.21.15.81:0/622115336 >> 172.21.15.81:0/622115336 conn(0x7fab2c09cf30 msgr2=0x7fab2c1613b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:31.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 -- 172.21.15.81:0/622115336 shutdown_connections 2024-06-29T15:58:31.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.734+0000 7fab31d91640 1 -- 172.21.15.81:0/622115336 wait complete. 2024-06-29T15:58:31.747 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:31.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.860+0000 7f441ecce640 1 -- 172.21.15.81:0/457993844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f441815c030 msgr2=0x7f441806c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.860+0000 7f441ecce640 1 --2- 172.21.15.81:0/457993844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f441815c030 0x7f441806c2f0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f4418071e60 tx=0x7f440c02a560 comp rx=0 tx=0).stop 2024-06-29T15:58:31.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.860+0000 7f441ecce640 1 -- 172.21.15.81:0/457993844 shutdown_connections 2024-06-29T15:58:31.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.860+0000 7f441ecce640 1 --2- 172.21.15.81:0/457993844 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f441815c440 0x7f441806c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.860+0000 7f441ecce640 1 --2- 172.21.15.81:0/457993844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f441815c030 0x7f441806c2f0 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.860+0000 7f441ecce640 1 --2- 172.21.15.81:0/457993844 >> v2:172.21.15.167:3300/0 conn(0x7f441815d750 0x7f44180afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.860+0000 7f441ecce640 1 -- 172.21.15.81:0/457993844 >> 172.21.15.81:0/457993844 conn(0x7f44180d0930 msgr2=0x7f44180d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.860+0000 7f441ecce640 1 -- 172.21.15.81:0/457993844 shutdown_connections 2024-06-29T15:58:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.860+0000 7f441ecce640 1 -- 172.21.15.81:0/457993844 wait complete. 2024-06-29T15:58:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.861+0000 7f441ecce640 1 Processor -- start 2024-06-29T15:58:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.861+0000 7f441ecce640 1 -- start start 2024-06-29T15:58:31.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ecce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f441815c030 0x7f4418153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ecce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f441815c440 0x7f44181462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ecce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f441815d750 0x7f44181f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ecce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f441815ff10 con 0x7f441815c030 2024-06-29T15:58:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ecce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f441815fd90 con 0x7f441815d750 2024-06-29T15:58:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ecce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4418160090 con 0x7f441815c440 2024-06-29T15:58:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ca6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f441815c030 0x7f4418153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ca6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f441815c030 0x7f4418153c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33160/0 (socket says 172.21.15.81:33160) 2024-06-29T15:58:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ca6a640 1 -- 172.21.15.81:0/2750787346 learned_addr learned my addr 172.21.15.81:0/2750787346 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f4417fff640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f441815c440 0x7f44181462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441d26b640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f441815d750 0x7f44181f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ca6a640 1 -- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f441815c440 msgr2=0x7f44181462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ca6a640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f441815c440 0x7f44181462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ca6a640 1 -- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f441815d750 msgr2=0x7f44181f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:31.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ca6a640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f441815d750 0x7f44181f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:31.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ca6a640 1 -- 172.21.15.81:0/2750787346 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44181467f0 con 0x7f441815c030 2024-06-29T15:58:31.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.863+0000 7f441ca6a640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f441815c030 0x7f4418153c30 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f440800d750 tx=0x7f440800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:31.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.864+0000 7f44157fa640 1 -- 172.21.15.81:0/2750787346 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408017070 con 0x7f441815c030 2024-06-29T15:58:31.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.864+0000 7f44157fa640 1 -- 172.21.15.81:0/2750787346 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4408004310 con 0x7f441815c030 2024-06-29T15:58:31.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.864+0000 7f44157fa640 1 -- 172.21.15.81:0/2750787346 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440800de60 con 0x7f441815c030 2024-06-29T15:58:31.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.864+0000 7f441ecce640 1 -- 172.21.15.81:0/2750787346 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4418152b90 con 0x7f441815c030 2024-06-29T15:58:31.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.864+0000 7f441ecce640 1 -- 172.21.15.81:0/2750787346 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4418153150 con 0x7f441815c030 2024-06-29T15:58:31.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.865+0000 7f441ecce640 1 -- 172.21.15.81:0/2750787346 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f441806e2a0 con 0x7f441815c030 2024-06-29T15:58:31.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.866+0000 7f44157fa640 1 -- 172.21.15.81:0/2750787346 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f4408009c30 con 0x7f441815c030 2024-06-29T15:58:31.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.866+0000 7f44157fa640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4404076540 0x7f4404078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:31.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.866+0000 7f44157fa640 1 -- 172.21.15.81:0/2750787346 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 6701+0+0 (secure 0 0 0) 0x7f4408098310 con 0x7f441815c030 2024-06-29T15:58:31.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.866+0000 7f4417fff640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4404076540 0x7f4404078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:31.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.867+0000 7f4417fff640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4404076540 0x7f4404078a50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f440c033000 tx=0x7f440c004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:31.870+0000 7f44157fa640 1 -- 172.21.15.81:0/2750787346 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f441806e2a0 con 0x7f441815c030 2024-06-29T15:58:32.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.050+0000 7f441ecce640 1 -- 172.21.15.81:0/2750787346 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44180afb60 con 0x7f441815c030 2024-06-29T15:58:32.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.052+0000 7f44157fa640 1 -- 172.21.15.81:0/2750787346 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v236) ==== 74+0+20558 (secure 0 0 0) 0x7f4408020020 con 0x7f441815c030 2024-06-29T15:58:32.053 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:32.054 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":236,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:30.737126+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:29.291185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:29.702735+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:32.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 -- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4404076540 msgr2=0x7f4404078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:32.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4404076540 0x7f4404078a50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f440c033000 tx=0x7f440c004730 comp rx=0 tx=0).stop 2024-06-29T15:58:32.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 -- 172.21.15.81:0/2750787346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f441815c030 msgr2=0x7f4418153c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:32.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f441815c030 0x7f4418153c30 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f440800d750 tx=0x7f440800dc10 comp rx=0 tx=0).stop 2024-06-29T15:58:32.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 -- 172.21.15.81:0/2750787346 shutdown_connections 2024-06-29T15:58:32.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4404076540 0x7f4404078a50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f441815d750 0x7f44181f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f441815c440 0x7f44181462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 --2- 172.21.15.81:0/2750787346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f441815c030 0x7f4418153c30 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 -- 172.21.15.81:0/2750787346 >> 172.21.15.81:0/2750787346 conn(0x7f44180d0930 msgr2=0x7f4418072f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:32.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 -- 172.21.15.81:0/2750787346 shutdown_connections 2024-06-29T15:58:32.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.054+0000 7f441ecce640 1 -- 172.21.15.81:0/2750787346 wait complete. 2024-06-29T15:58:32.067 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2024-06-29T15:58:32.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.182+0000 7f76395ba640 1 -- 172.21.15.81:0/3684674164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76340b4250 msgr2=0x7f76340cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:32.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.182+0000 7f76395ba640 1 --2- 172.21.15.81:0/3684674164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76340b4250 0x7f76340cee70 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f76340d4c50 tx=0x7f762c02a410 comp rx=0 tx=0).stop 2024-06-29T15:58:32.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.182+0000 7f76395ba640 1 -- 172.21.15.81:0/3684674164 shutdown_connections 2024-06-29T15:58:32.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.182+0000 7f76395ba640 1 --2- 172.21.15.81:0/3684674164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76340b4250 0x7f76340cee70 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.182+0000 7f76395ba640 1 --2- 172.21.15.81:0/3684674164 >> v2:172.21.15.167:3300/0 conn(0x7f763408c270 0x7f76340ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.182+0000 7f76395ba640 1 -- 172.21.15.81:0/3684674164 >> 172.21.15.81:0/3684674164 conn(0x7f76340b56d0 msgr2=0x7f76340b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:32.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.182+0000 7f76395ba640 1 -- 172.21.15.81:0/3684674164 shutdown_connections 2024-06-29T15:58:32.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.182+0000 7f76395ba640 1 -- 172.21.15.81:0/3684674164 wait complete. 2024-06-29T15:58:32.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.184+0000 7f76395ba640 1 Processor -- start 2024-06-29T15:58:32.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.184+0000 7f76395ba640 1 -- start start 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.185+0000 7f76395ba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f763408ccd0 0x7f76341e1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.185+0000 7f76395ba640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76340b4250 0x7f76341de4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.185+0000 7f76395ba640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76340b7a60 0x7f76340b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.185+0000 7f76395ba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76340d2250 con 0x7f763408ccd0 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.185+0000 7f76395ba640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f76340d20d0 con 0x7f76340b7a60 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.185+0000 7f76395ba640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f76340d23d0 con 0x7f76340b4250 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.185+0000 7f7632ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f763408ccd0 0x7f76341e1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.185+0000 7f7632ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f763408ccd0 0x7f76341e1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33164/0 (socket says 172.21.15.81:33164) 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.185+0000 7f7632ffd640 1 -- 172.21.15.81:0/3925797952 learned_addr learned my addr 172.21.15.81:0/3925797952 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f7632ffd640 1 -- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76340b4250 msgr2=0x7f76341de4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:32.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f7632ffd640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76340b4250 0x7f76341de4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f76337fe640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76340b7a60 0x7f76340b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f7632ffd640 1 -- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76340b7a60 msgr2=0x7f76340b7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f7632ffd640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76340b7a60 0x7f76340b7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f7632ffd640 1 -- 172.21.15.81:0/3925797952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76340b83b0 con 0x7f763408ccd0 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f7632ffd640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f763408ccd0 0x7f76341e1c70 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f762400cce0 tx=0x7f762400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f761b7fe640 1 -- 172.21.15.81:0/3925797952 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762400f450 con 0x7f763408ccd0 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f761b7fe640 1 -- 172.21.15.81:0/3925797952 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f762400f6e0 con 0x7f763408ccd0 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f76395ba640 1 -- 172.21.15.81:0/3925797952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76341df630 con 0x7f763408ccd0 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f761b7fe640 1 -- 172.21.15.81:0/3925797952 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762400fb30 con 0x7f763408ccd0 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.186+0000 7f76395ba640 1 -- 172.21.15.81:0/3925797952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76341dfb30 con 0x7f763408ccd0 2024-06-29T15:58:32.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.187+0000 7f76395ba640 1 -- 172.21.15.81:0/3925797952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f763408c270 con 0x7f763408ccd0 2024-06-29T15:58:32.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.188+0000 7f761b7fe640 1 -- 172.21.15.81:0/3925797952 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f762400fcd0 con 0x7f763408ccd0 2024-06-29T15:58:32.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.188+0000 7f761b7fe640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7620076540 0x7f7620078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:32.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.188+0000 7f761b7fe640 1 -- 172.21.15.81:0/3925797952 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 6701+0+0 (secure 0 0 0) 0x7f7624098bf0 con 0x7f763408ccd0 2024-06-29T15:58:32.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.188+0000 7f76327fc640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7620076540 0x7f7620078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:32.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.189+0000 7f76327fc640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7620076540 0x7f7620078a50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f76341df2c0 tx=0x7f7628007930 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:32.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.193+0000 7f761b7fe640 1 -- 172.21.15.81:0/3925797952 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76340b83b0 con 0x7f763408ccd0 2024-06-29T15:58:32.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:32.415+0000 7f76395ba640 1 -- 172.21.15.81:0/3925797952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f763406d0b0 con 0x7f763408ccd0 2024-06-29T15:58:33.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.406+0000 7f761b7fe640 1 -- 172.21.15.81:0/3925797952 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v134) ==== 121+0+0 (secure 0 0 0) 0x7f7624014030 con 0x7f763408ccd0 2024-06-29T15:58:33.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.408+0000 7f76395ba640 1 -- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7620076540 msgr2=0x7f7620078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:33.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.408+0000 7f76395ba640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7620076540 0x7f7620078a50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f76341df2c0 tx=0x7f7628007930 comp rx=0 tx=0).stop 2024-06-29T15:58:33.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.408+0000 7f76395ba640 1 -- 172.21.15.81:0/3925797952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f763408ccd0 msgr2=0x7f76341e1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:33.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.408+0000 7f76395ba640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f763408ccd0 0x7f76341e1c70 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f762400cce0 tx=0x7f762400a620 comp rx=0 tx=0).stop 2024-06-29T15:58:33.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.408+0000 7f76395ba640 1 -- 172.21.15.81:0/3925797952 shutdown_connections 2024-06-29T15:58:33.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.408+0000 7f76395ba640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f7620076540 0x7f7620078a50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.408+0000 7f76395ba640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f76340b7a60 0x7f76340b7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.408+0000 7f76395ba640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f76340b4250 0x7f76341de4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.408+0000 7f76395ba640 1 --2- 172.21.15.81:0/3925797952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f763408ccd0 0x7f76341e1c70 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.409+0000 7f76395ba640 1 -- 172.21.15.81:0/3925797952 >> 172.21.15.81:0/3925797952 conn(0x7f76340b56d0 msgr2=0x7f7634068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:33.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.409+0000 7f76395ba640 1 -- 172.21.15.81:0/3925797952 shutdown_connections 2024-06-29T15:58:33.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.409+0000 7f76395ba640 1 -- 172.21.15.81:0/3925797952 wait complete. 2024-06-29T15:58:33.423 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:58:33.424 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2024-06-29T15:58:33.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.545+0000 7f9f288c1640 1 -- 172.21.15.81:0/2649478168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f240cccf0 msgr2=0x7f9f240d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:33.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.545+0000 7f9f288c1640 1 --2- 172.21.15.81:0/2649478168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f240cccf0 0x7f9f240d0230 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009830 tx=0x7f9f1802a510 comp rx=0 tx=0).stop 2024-06-29T15:58:33.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.545+0000 7f9f288c1640 1 -- 172.21.15.81:0/2649478168 shutdown_connections 2024-06-29T15:58:33.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.545+0000 7f9f288c1640 1 --2- 172.21.15.81:0/2649478168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f240cccf0 0x7f9f240d0230 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.545+0000 7f9f288c1640 1 --2- 172.21.15.81:0/2649478168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f240b3200 0x7f9f240cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.545+0000 7f9f288c1640 1 -- 172.21.15.81:0/2649478168 >> 172.21.15.81:0/2649478168 conn(0x7f9f240b3680 msgr2=0x7f9f240b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:33.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.545+0000 7f9f288c1640 1 -- 172.21.15.81:0/2649478168 shutdown_connections 2024-06-29T15:58:33.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.545+0000 7f9f288c1640 1 -- 172.21.15.81:0/2649478168 wait complete. 2024-06-29T15:58:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.547+0000 7f9f288c1640 1 Processor -- start 2024-06-29T15:58:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.547+0000 7f9f288c1640 1 -- start start 2024-06-29T15:58:33.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f288c1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f240b3200 0x7f9f24149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:33.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f288c1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f240cccf0 0x7f9f240d1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:33.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f288c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f240d0770 0x7f9f240d1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:33.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f288c1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f240cedb0 con 0x7f9f240d0770 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f288c1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9f240cec30 con 0x7f9f240b3200 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f288c1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9f240cef30 con 0x7f9f240cccf0 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f22d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f240d0770 0x7f9f240d1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f22d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f240d0770 0x7f9f240d1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33168/0 (socket says 172.21.15.81:33168) 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f22d9d640 1 -- 172.21.15.81:0/958350099 learned_addr learned my addr 172.21.15.81:0/958350099 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f2259c640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f240b3200 0x7f9f24149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f21d9b640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f240cccf0 0x7f9f240d1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f22d9d640 1 -- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f240cccf0 msgr2=0x7f9f240d1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f22d9d640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f240cccf0 0x7f9f240d1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f22d9d640 1 -- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f240b3200 msgr2=0x7f9f24149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f22d9d640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f240b3200 0x7f9f24149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.548+0000 7f9f22d9d640 1 -- 172.21.15.81:0/958350099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f241e2b00 con 0x7f9f240d0770 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.549+0000 7f9f22d9d640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f240d0770 0x7f9f240d1eb0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00c390 tx=0x7f9f1c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:33.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.549+0000 7f9f12ffd640 1 -- 172.21.15.81:0/958350099 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c011790 con 0x7f9f240d0770 2024-06-29T15:58:33.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.549+0000 7f9f12ffd640 1 -- 172.21.15.81:0/958350099 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f1c017070 con 0x7f9f240d0770 2024-06-29T15:58:33.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.549+0000 7f9f288c1640 1 -- 172.21.15.81:0/958350099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f241e2de0 con 0x7f9f240d0770 2024-06-29T15:58:33.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.549+0000 7f9f12ffd640 1 -- 172.21.15.81:0/958350099 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c011c60 con 0x7f9f240d0770 2024-06-29T15:58:33.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.549+0000 7f9f288c1640 1 -- 172.21.15.81:0/958350099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f241e33a0 con 0x7f9f240d0770 2024-06-29T15:58:33.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.550+0000 7f9f288c1640 1 -- 172.21.15.81:0/958350099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f240cc7b0 con 0x7f9f240d0770 2024-06-29T15:58:33.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.551+0000 7f9f12ffd640 1 -- 172.21.15.81:0/958350099 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f9f1c01c460 con 0x7f9f240d0770 2024-06-29T15:58:33.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.551+0000 7f9f12ffd640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f04076510 0x7f9f04078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:33.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.551+0000 7f9f12ffd640 1 -- 172.21.15.81:0/958350099 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7f9f1c097a80 con 0x7f9f240d0770 2024-06-29T15:58:33.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.552+0000 7f9f2259c640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f04076510 0x7f9f04078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:33.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.552+0000 7f9f2259c640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f04076510 0x7f9f04078a20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9f140099a0 tx=0x7f9f14009500 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:33.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.556+0000 7f9f12ffd640 1 -- 172.21.15.81:0/958350099 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9f240d0b80 con 0x7f9f240d0770 2024-06-29T15:58:33.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:33.771+0000 7f9f288c1640 1 -- 172.21.15.81:0/958350099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f9f24159d40 con 0x7f9f240d0770 2024-06-29T15:58:33.861 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:58:33.864 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:58:34.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.409+0000 7f9f12ffd640 1 -- 172.21.15.81:0/958350099 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v135) ==== 125+0+0 (secure 0 0 0) 0x7f9f1c060720 con 0x7f9f240d0770 2024-06-29T15:58:34.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.412+0000 7f9f288c1640 1 -- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f04076510 msgr2=0x7f9f04078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.412+0000 7f9f288c1640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f04076510 0x7f9f04078a20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9f140099a0 tx=0x7f9f14009500 comp rx=0 tx=0).stop 2024-06-29T15:58:34.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.412+0000 7f9f288c1640 1 -- 172.21.15.81:0/958350099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f240d0770 msgr2=0x7f9f240d1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.412+0000 7f9f288c1640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f240d0770 0x7f9f240d1eb0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00c390 tx=0x7f9f1c00c850 comp rx=0 tx=0).stop 2024-06-29T15:58:34.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.412+0000 7f9f288c1640 1 -- 172.21.15.81:0/958350099 shutdown_connections 2024-06-29T15:58:34.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.412+0000 7f9f288c1640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f9f04076510 0x7f9f04078a20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.412+0000 7f9f288c1640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f240d0770 0x7f9f240d1eb0 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.412+0000 7f9f288c1640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9f240cccf0 0x7f9f240d1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.412+0000 7f9f288c1640 1 --2- 172.21.15.81:0/958350099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9f240b3200 0x7f9f24149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.412+0000 7f9f288c1640 1 -- 172.21.15.81:0/958350099 >> 172.21.15.81:0/958350099 conn(0x7f9f240b3680 msgr2=0x7f9f240718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:34.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.413+0000 7f9f288c1640 1 -- 172.21.15.81:0/958350099 shutdown_connections 2024-06-29T15:58:34.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.413+0000 7f9f288c1640 1 -- 172.21.15.81:0/958350099 wait complete. 2024-06-29T15:58:34.427 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2024-06-29T15:58:34.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:34.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.544+0000 7fe550f8c640 1 -- 172.21.15.81:0/1980460147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe538004800 msgr2=0x7fe538004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.544+0000 7fe550f8c640 1 --2- 172.21.15.81:0/1980460147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe538004800 0x7fe538004c10 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7fe534001050 tx=0x7fe538005830 comp rx=0 tx=0).stop 2024-06-29T15:58:34.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.544+0000 7fe550f8c640 1 -- 172.21.15.81:0/1980460147 shutdown_connections 2024-06-29T15:58:34.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.544+0000 7fe550f8c640 1 --2- 172.21.15.81:0/1980460147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe538004800 0x7fe538004c10 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.544+0000 7fe550f8c640 1 -- 172.21.15.81:0/1980460147 >> 172.21.15.81:0/1980460147 conn(0x7fe54c09cf30 msgr2=0x7fe54c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:34.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.544+0000 7fe550f8c640 1 -- 172.21.15.81:0/1980460147 shutdown_connections 2024-06-29T15:58:34.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.544+0000 7fe550f8c640 1 -- 172.21.15.81:0/1980460147 wait complete. 2024-06-29T15:58:34.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.546+0000 7fe550f8c640 1 Processor -- start 2024-06-29T15:58:34.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.546+0000 7fe550f8c640 1 -- start start 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe550f8c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe538004800 0x7fe54c0b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe550f8c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe54c15d140 0x7fe54c1f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe550f8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe54c1f8640 0x7fe54c0b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe550f8c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c0d0860 con 0x7fe54c1f8640 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe550f8c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c0d06e0 con 0x7fe538004800 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe550f8c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe54c0d09e0 con 0x7fe54c15d140 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe54ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe54c1f8640 0x7fe54c0b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe54ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe54c1f8640 0x7fe54c0b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33192/0 (socket says 172.21.15.81:33192) 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe54ad9d640 1 -- 172.21.15.81:0/4202941665 learned_addr learned my addr 172.21.15.81:0/4202941665 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe54a59c640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe538004800 0x7fe54c0b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe543fff640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe54c15d140 0x7fe54c1f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe54ad9d640 1 -- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe54c15d140 msgr2=0x7fe54c1f8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe54ad9d640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe54c15d140 0x7fe54c1f8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe54ad9d640 1 -- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe538004800 msgr2=0x7fe54c0b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.547+0000 7fe54ad9d640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe538004800 0x7fe54c0b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.548+0000 7fe54ad9d640 1 -- 172.21.15.81:0/4202941665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54c1f8a50 con 0x7fe54c1f8640 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.548+0000 7fe54ad9d640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe54c1f8640 0x7fe54c0b0820 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fe54c0d43d0 tx=0x7fe54400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.548+0000 7fe542ffd640 1 -- 172.21.15.81:0/4202941665 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544003d80 con 0x7fe54c1f8640 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.548+0000 7fe542ffd640 1 -- 172.21.15.81:0/4202941665 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe544004b20 con 0x7fe54c1f8640 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.548+0000 7fe550f8c640 1 -- 172.21.15.81:0/4202941665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54c0b26c0 con 0x7fe54c1f8640 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.548+0000 7fe542ffd640 1 -- 172.21.15.81:0/4202941665 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544004030 con 0x7fe54c1f8640 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.548+0000 7fe550f8c640 1 -- 172.21.15.81:0/4202941665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54c0b2e10 con 0x7fe54c1f8640 2024-06-29T15:58:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.549+0000 7fe550f8c640 1 -- 172.21.15.81:0/4202941665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe54c0b2890 con 0x7fe54c1f8640 2024-06-29T15:58:34.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.550+0000 7fe542ffd640 1 -- 172.21.15.81:0/4202941665 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fe5440041d0 con 0x7fe54c1f8640 2024-06-29T15:58:34.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.550+0000 7fe542ffd640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe52c076510 0x7fe52c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.550+0000 7fe542ffd640 1 -- 172.21.15.81:0/4202941665 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7fe544098f40 con 0x7fe54c1f8640 2024-06-29T15:58:34.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.550+0000 7fe54a59c640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe52c076510 0x7fe52c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.551+0000 7fe54a59c640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe52c076510 0x7fe52c078a20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe54c06adf0 tx=0x7fe53c006ca0 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:34.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.555+0000 7fe542ffd640 1 -- 172.21.15.81:0/4202941665 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe54c0b2890 con 0x7fe54c1f8640 2024-06-29T15:58:34.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.771+0000 7fe550f8c640 1 -- 172.21.15.81:0/4202941665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe54c0b8410 con 0x7fe54c1f8640 2024-06-29T15:58:34.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.772+0000 7fe542ffd640 1 -- 172.21.15.81:0/4202941665 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6562 (secure 0 0 0) 0x7fe54c1f8a50 con 0x7fe54c1f8640 2024-06-29T15:58:34.773 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:34.774 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":135,"btime":"2024-06-29T15:58:34:407667+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:18.654579+0000","modified":"2024-06-29T15:58:21.679033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12105":{"gid":12105,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/30158598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":30158598},{"type":"v1","addr":"172.21.15.167:6837","nonce":30158598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12105,"qdb_cluster":[12105]},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:30.737083+0000","modified":"2024-06-29T15:58:33.773132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12093":{"gid":12093,"name":"a","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12093,"qdb_cluster":[12093]},"id":16}]} 2024-06-29T15:58:34.774 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 135 2024-06-29T15:58:34.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 -- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe52c076510 msgr2=0x7fe52c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe52c076510 0x7fe52c078a20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe54c06adf0 tx=0x7fe53c006ca0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 -- 172.21.15.81:0/4202941665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe54c1f8640 msgr2=0x7fe54c0b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe54c1f8640 0x7fe54c0b0820 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7fe54c0d43d0 tx=0x7fe54400f790 comp rx=0 tx=0).stop 2024-06-29T15:58:34.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 -- 172.21.15.81:0/4202941665 shutdown_connections 2024-06-29T15:58:34.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe52c076510 0x7fe52c078a20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe54c1f8640 0x7fe54c0b0820 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe54c15d140 0x7fe54c1f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 --2- 172.21.15.81:0/4202941665 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe538004800 0x7fe54c0b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 -- 172.21.15.81:0/4202941665 >> 172.21.15.81:0/4202941665 conn(0x7fe54c09cf30 msgr2=0x7fe54c15ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:34.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 -- 172.21.15.81:0/4202941665 shutdown_connections 2024-06-29T15:58:34.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.774+0000 7fe550f8c640 1 -- 172.21.15.81:0/4202941665 wait complete. 2024-06-29T15:58:34.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:34.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.898+0000 7fd2cec71640 1 -- 172.21.15.81:0/2990615791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2c815b5b0 msgr2=0x7fd2c80b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.898+0000 7fd2cec71640 1 --2- 172.21.15.81:0/2990615791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2c815b5b0 0x7fd2c80b4430 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc005fd0 tx=0x7fd2bc02bbe0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.898+0000 7fd2cec71640 1 -- 172.21.15.81:0/2990615791 shutdown_connections 2024-06-29T15:58:34.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.898+0000 7fd2cec71640 1 --2- 172.21.15.81:0/2990615791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2c815b5b0 0x7fd2c80b4430 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.898+0000 7fd2cec71640 1 -- 172.21.15.81:0/2990615791 >> 172.21.15.81:0/2990615791 conn(0x7fd2c806ccc0 msgr2=0x7fd2c80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:34.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.898+0000 7fd2cec71640 1 -- 172.21.15.81:0/2990615791 shutdown_connections 2024-06-29T15:58:34.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.898+0000 7fd2cec71640 1 -- 172.21.15.81:0/2990615791 wait complete. 2024-06-29T15:58:34.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.900+0000 7fd2cec71640 1 Processor -- start 2024-06-29T15:58:34.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.900+0000 7fd2cec71640 1 -- start start 2024-06-29T15:58:34.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.901+0000 7fd2cec71640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2c806f660 0x7fd2c81f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.901+0000 7fd2cec71640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2c80b35b0 0x7fd2c8153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.901+0000 7fd2cec71640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2c815b5b0 0x7fd2c8159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.901+0000 7fd2cec71640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c806f460 con 0x7fd2c815b5b0 2024-06-29T15:58:34.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.901+0000 7fd2cec71640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c806f2e0 con 0x7fd2c806f660 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.901+0000 7fd2cec71640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c8071f20 con 0x7fd2c80b35b0 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.901+0000 7fd2cd20e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2c815b5b0 0x7fd2c8159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.901+0000 7fd2cd20e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2c815b5b0 0x7fd2c8159500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33216/0 (socket says 172.21.15.81:33216) 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.901+0000 7fd2cd20e640 1 -- 172.21.15.81:0/3294191672 learned_addr learned my addr 172.21.15.81:0/3294191672 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2c7fff640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2c80b35b0 0x7fd2c8153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2cca0d640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2c806f660 0x7fd2c81f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2cd20e640 1 -- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2c80b35b0 msgr2=0x7fd2c8153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2cd20e640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2c80b35b0 0x7fd2c8153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2cd20e640 1 -- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2c806f660 msgr2=0x7fd2c81f5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2cd20e640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2c806f660 0x7fd2c81f5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2cd20e640 1 -- 172.21.15.81:0/3294191672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c81f6280 con 0x7fd2c815b5b0 2024-06-29T15:58:34.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2cca0d640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2c806f660 0x7fd2c81f5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:34.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2cd20e640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2c815b5b0 0x7fd2c8159500 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000c3c0 tx=0x7fd2c000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:34.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2c57fa640 1 -- 172.21.15.81:0/3294191672 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0017070 con 0x7fd2c815b5b0 2024-06-29T15:58:34.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2c57fa640 1 -- 172.21.15.81:0/3294191672 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2c00035f0 con 0x7fd2c815b5b0 2024-06-29T15:58:34.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2c57fa640 1 -- 172.21.15.81:0/3294191672 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0003a40 con 0x7fd2c815b5b0 2024-06-29T15:58:34.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2cec71640 1 -- 172.21.15.81:0/3294191672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c81530b0 con 0x7fd2c815b5b0 2024-06-29T15:58:34.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.902+0000 7fd2cec71640 1 -- 172.21.15.81:0/3294191672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c8154670 con 0x7fd2c815b5b0 2024-06-29T15:58:34.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.903+0000 7fd2cec71640 1 -- 172.21.15.81:0/3294191672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c81540b0 con 0x7fd2c815b5b0 2024-06-29T15:58:34.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.904+0000 7fd2c57fa640 1 -- 172.21.15.81:0/3294191672 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fd2c001a070 con 0x7fd2c815b5b0 2024-06-29T15:58:34.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.905+0000 7fd2c57fa640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd2b4076510 0x7fd2b4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:34.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.905+0000 7fd2c57fa640 1 -- 172.21.15.81:0/3294191672 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7fd2c00971e0 con 0x7fd2c815b5b0 2024-06-29T15:58:34.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.905+0000 7fd2cca0d640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd2b4076510 0x7fd2b4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:34.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.905+0000 7fd2cca0d640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd2b4076510 0x7fd2b4078a20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80016a0 tx=0x7fd2b8009040 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:34.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:34.909+0000 7fd2c57fa640 1 -- 172.21.15.81:0/3294191672 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd2c0060490 con 0x7fd2c815b5b0 2024-06-29T15:58:35.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.090+0000 7fd2cec71640 1 -- 172.21.15.81:0/3294191672 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2c815cbc0 con 0x7fd2c815b5b0 2024-06-29T15:58:35.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.092+0000 7fd2c57fa640 1 -- 172.21.15.81:0/3294191672 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) ==== 74+0+20545 (secure 0 0 0) 0x7fd2c005fd90 con 0x7fd2c815b5b0 2024-06-29T15:58:35.093 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:35.093 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":237,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:32.379021+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:29.291185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:29.702735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:35.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 -- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd2b4076510 msgr2=0x7fd2b4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd2b4076510 0x7fd2b4078a20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80016a0 tx=0x7fd2b8009040 comp rx=0 tx=0).stop 2024-06-29T15:58:35.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 -- 172.21.15.81:0/3294191672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2c815b5b0 msgr2=0x7fd2c8159500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2c815b5b0 0x7fd2c8159500 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000c3c0 tx=0x7fd2c000c880 comp rx=0 tx=0).stop 2024-06-29T15:58:35.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 -- 172.21.15.81:0/3294191672 shutdown_connections 2024-06-29T15:58:35.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd2b4076510 0x7fd2b4078a20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2c815b5b0 0x7fd2c8159500 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd2c80b35b0 0x7fd2c8153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 --2- 172.21.15.81:0/3294191672 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd2c806f660 0x7fd2c81f5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 -- 172.21.15.81:0/3294191672 >> 172.21.15.81:0/3294191672 conn(0x7fd2c806ccc0 msgr2=0x7fd2c815bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:35.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 -- 172.21.15.81:0/3294191672 shutdown_connections 2024-06-29T15:58:35.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.094+0000 7fd2cec71640 1 -- 172.21.15.81:0/3294191672 wait complete. 2024-06-29T15:58:35.107 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:58:35.107 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:58:35.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.218+0000 7f4383143640 1 -- 172.21.15.81:0/1107466500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437c073030 msgr2=0x7f437c073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.218+0000 7f4383143640 1 --2- 172.21.15.81:0/1107466500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437c073030 0x7f437c073440 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f4370005fd0 tx=0x7f437002a530 comp rx=0 tx=0).stop 2024-06-29T15:58:35.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.219+0000 7f4383143640 1 -- 172.21.15.81:0/1107466500 shutdown_connections 2024-06-29T15:58:35.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.219+0000 7f4383143640 1 --2- 172.21.15.81:0/1107466500 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f437c15e960 0x7f437c15ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.219+0000 7f4383143640 1 --2- 172.21.15.81:0/1107466500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437c073030 0x7f437c073440 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.219+0000 7f4383143640 1 -- 172.21.15.81:0/1107466500 >> 172.21.15.81:0/1107466500 conn(0x7f437c06ccc0 msgr2=0x7f437c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:35.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.219+0000 7f4383143640 1 -- 172.21.15.81:0/1107466500 shutdown_connections 2024-06-29T15:58:35.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.219+0000 7f4383143640 1 -- 172.21.15.81:0/1107466500 wait complete. 2024-06-29T15:58:35.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.220+0000 7f4383143640 1 Processor -- start 2024-06-29T15:58:35.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.220+0000 7f4383143640 1 -- start start 2024-06-29T15:58:35.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f4383143640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f437c073030 0x7f437c13fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f4383143640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437c1412f0 0x7f437c1556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f4383143640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f437c15e960 0x7f437c1f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f4383143640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f437c0b3390 con 0x7f437c1412f0 2024-06-29T15:58:35.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f4383143640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f437c0b3210 con 0x7f437c15e960 2024-06-29T15:58:35.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f4383143640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f437c0b3510 con 0x7f437c073030 2024-06-29T15:58:35.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f437bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437c1412f0 0x7f437c1556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f437bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437c1412f0 0x7f437c1556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33238/0 (socket says 172.21.15.81:33238) 2024-06-29T15:58:35.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f437bfff640 1 -- 172.21.15.81:0/2616979269 learned_addr learned my addr 172.21.15.81:0/2616979269 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:35.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f43816e0640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f437c15e960 0x7f437c1f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f4380edf640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f437c073030 0x7f437c13fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f437bfff640 1 -- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f437c073030 msgr2=0x7f437c13fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f437bfff640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f437c073030 0x7f437c13fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f437bfff640 1 -- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f437c15e960 msgr2=0x7f437c1f2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f437bfff640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f437c15e960 0x7f437c1f2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f437bfff640 1 -- 172.21.15.81:0/2616979269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c156690 con 0x7f437c1412f0 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.222+0000 7f437bfff640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437c1412f0 0x7f437c1556e0 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f4370000c00 tx=0x7f4370003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.223+0000 7f43797fa640 1 -- 172.21.15.81:0/2616979269 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370035040 con 0x7f437c1412f0 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.223+0000 7f43797fa640 1 -- 172.21.15.81:0/2616979269 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43700047a0 con 0x7f437c1412f0 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.223+0000 7f4383143640 1 -- 172.21.15.81:0/2616979269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437c156970 con 0x7f437c1412f0 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.223+0000 7f43797fa640 1 -- 172.21.15.81:0/2616979269 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370004bf0 con 0x7f437c1412f0 2024-06-29T15:58:35.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.223+0000 7f4383143640 1 -- 172.21.15.81:0/2616979269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f437c156f30 con 0x7f437c1412f0 2024-06-29T15:58:35.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.224+0000 7f4383143640 1 -- 172.21.15.81:0/2616979269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f437c0b3a10 con 0x7f437c1412f0 2024-06-29T15:58:35.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.224+0000 7f43797fa640 1 -- 172.21.15.81:0/2616979269 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f4370004d90 con 0x7f437c1412f0 2024-06-29T15:58:35.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.225+0000 7f43797fa640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4368076510 0x7f4368078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.225+0000 7f43797fa640 1 -- 172.21.15.81:0/2616979269 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7f43700ba830 con 0x7f437c1412f0 2024-06-29T15:58:35.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.225+0000 7f4380edf640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4368076510 0x7f4368078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.226+0000 7f4380edf640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4368076510 0x7f4368078a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f436c006fd0 tx=0x7f436c009060 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:35.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.230+0000 7f43797fa640 1 -- 172.21.15.81:0/2616979269 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f437c073980 con 0x7f437c1412f0 2024-06-29T15:58:35.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.412+0000 7f4383143640 1 -- 172.21.15.81:0/2616979269 --> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f437c15a1d0 con 0x7f4368076510 2024-06-29T15:58:35.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.424+0000 7f43797fa640 1 -- 172.21.15.81:0/2616979269 <== mgr.9155 v2:172.21.15.167:6832/3661607921 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491471 (secure 0 0 0) 0x7f437c15a1d0 con 0x7f4368076510 2024-06-29T15:58:35.430 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:35.434 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 -- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4368076510 msgr2=0x7f4368078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4368076510 0x7f4368078a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f436c006fd0 tx=0x7f436c009060 comp rx=0 tx=0).stop 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 -- 172.21.15.81:0/2616979269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437c1412f0 msgr2=0x7f437c1556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437c1412f0 0x7f437c1556e0 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f4370000c00 tx=0x7f4370003970 comp rx=0 tx=0).stop 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 -- 172.21.15.81:0/2616979269 shutdown_connections 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4368076510 0x7f4368078a20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f437c15e960 0x7f437c1f2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f437c1412f0 0x7f437c1556e0 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 --2- 172.21.15.81:0/2616979269 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f437c073030 0x7f437c13fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 -- 172.21.15.81:0/2616979269 >> 172.21.15.81:0/2616979269 conn(0x7f437c06ccc0 msgr2=0x7f437c0b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:35.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 -- 172.21.15.81:0/2616979269 shutdown_connections 2024-06-29T15:58:35.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.428+0000 7f4383143640 1 -- 172.21.15.81:0/2616979269 wait complete. 2024-06-29T15:58:35.441 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-06-29T15:58:35.348879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596431,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":469,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":353,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":437856,"kb_used_data":24604,"kb_used_omap":12,"kb_used_meta":413107,"kb_avail":754536864,"statfs":{"total":773094113280,"available":772645748736,"internally_reserved":0,"allocated":25194496,"data_stored":12970335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":423022130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.986580"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403282+0000","last_change":"2024-06-29T15:58:31.394424+0000","last_active":"2024-06-29T15:58:32.403282+0000","last_peered":"2024-06-29T15:58:32.403282+0000","last_clean":"2024-06-29T15:58:32.403282+0000","last_became_active":"2024-06-29T15:58:31.394079+0000","last_became_peered":"2024-06-29T15:58:31.394079+0000","last_unstale":"2024-06-29T15:58:32.403282+0000","last_undegraded":"2024-06-29T15:58:32.403282+0000","last_fullsized":"2024-06-29T15:58:32.403282+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:59:48.821074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471869+0000","last_change":"2024-06-29T15:58:30.387552+0000","last_active":"2024-06-29T15:58:32.471869+0000","last_peered":"2024-06-29T15:58:32.471869+0000","last_clean":"2024-06-29T15:58:32.471869+0000","last_became_active":"2024-06-29T15:58:30.387263+0000","last_became_peered":"2024-06-29T15:58:30.387263+0000","last_unstale":"2024-06-29T15:58:32.471869+0000","last_undegraded":"2024-06-29T15:58:32.471869+0000","last_fullsized":"2024-06-29T15:58:32.471869+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:17:24.545993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408848+0000","last_change":"2024-06-29T15:58:19.306321+0000","last_active":"2024-06-29T15:58:32.408848+0000","last_peered":"2024-06-29T15:58:32.408848+0000","last_clean":"2024-06-29T15:58:32.408848+0000","last_became_active":"2024-06-29T15:58:19.306018+0000","last_became_peered":"2024-06-29T15:58:19.306018+0000","last_unstale":"2024-06-29T15:58:32.408848+0000","last_undegraded":"2024-06-29T15:58:32.408848+0000","last_fullsized":"2024-06-29T15:58:32.408848+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:42.361706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382505+0000","last_change":"2024-06-29T15:58:31.407272+0000","last_active":"2024-06-29T15:58:32.382505+0000","last_peered":"2024-06-29T15:58:32.382505+0000","last_clean":"2024-06-29T15:58:32.382505+0000","last_became_active":"2024-06-29T15:58:31.406988+0000","last_became_peered":"2024-06-29T15:58:31.406988+0000","last_unstale":"2024-06-29T15:58:32.382505+0000","last_undegraded":"2024-06-29T15:58:32.382505+0000","last_fullsized":"2024-06-29T15:58:32.382505+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:33.966242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383620+0000","last_change":"2024-06-29T15:58:30.388423+0000","last_active":"2024-06-29T15:58:32.383620+0000","last_peered":"2024-06-29T15:58:32.383620+0000","last_clean":"2024-06-29T15:58:32.383620+0000","last_became_active":"2024-06-29T15:58:30.388144+0000","last_became_peered":"2024-06-29T15:58:30.388144+0000","last_unstale":"2024-06-29T15:58:32.383620+0000","last_undegraded":"2024-06-29T15:58:32.383620+0000","last_fullsized":"2024-06-29T15:58:32.383620+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:30.660014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409034+0000","last_change":"2024-06-29T15:58:19.312428+0000","last_active":"2024-06-29T15:58:32.409034+0000","last_peered":"2024-06-29T15:58:32.409034+0000","last_clean":"2024-06-29T15:58:32.409034+0000","last_became_active":"2024-06-29T15:58:19.312036+0000","last_became_peered":"2024-06-29T15:58:19.312036+0000","last_unstale":"2024-06-29T15:58:32.409034+0000","last_undegraded":"2024-06-29T15:58:32.409034+0000","last_fullsized":"2024-06-29T15:58:32.409034+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:04.132174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382681+0000","last_change":"2024-06-29T15:58:31.402000+0000","last_active":"2024-06-29T15:58:32.382681+0000","last_peered":"2024-06-29T15:58:32.382681+0000","last_clean":"2024-06-29T15:58:32.382681+0000","last_became_active":"2024-06-29T15:58:31.401674+0000","last_became_peered":"2024-06-29T15:58:31.401674+0000","last_unstale":"2024-06-29T15:58:32.382681+0000","last_undegraded":"2024-06-29T15:58:32.382681+0000","last_fullsized":"2024-06-29T15:58:32.382681+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:16.215524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470426+0000","last_change":"2024-06-29T15:58:30.388832+0000","last_active":"2024-06-29T15:58:32.470426+0000","last_peered":"2024-06-29T15:58:32.470426+0000","last_clean":"2024-06-29T15:58:32.470426+0000","last_became_active":"2024-06-29T15:58:30.388524+0000","last_became_peered":"2024-06-29T15:58:30.388524+0000","last_unstale":"2024-06-29T15:58:32.470426+0000","last_undegraded":"2024-06-29T15:58:32.470426+0000","last_fullsized":"2024-06-29T15:58:32.470426+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:44.713766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471031+0000","last_change":"2024-06-29T15:58:19.315147+0000","last_active":"2024-06-29T15:58:32.471031+0000","last_peered":"2024-06-29T15:58:32.471031+0000","last_clean":"2024-06-29T15:58:32.471031+0000","last_became_active":"2024-06-29T15:58:19.314838+0000","last_became_peered":"2024-06-29T15:58:19.314838+0000","last_unstale":"2024-06-29T15:58:32.471031+0000","last_undegraded":"2024-06-29T15:58:32.471031+0000","last_fullsized":"2024-06-29T15:58:32.471031+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:33:39.763837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383401+0000","last_change":"2024-06-29T15:58:31.390355+0000","last_active":"2024-06-29T15:58:32.383401+0000","last_peered":"2024-06-29T15:58:32.383401+0000","last_clean":"2024-06-29T15:58:32.383401+0000","last_became_active":"2024-06-29T15:58:31.390036+0000","last_became_peered":"2024-06-29T15:58:31.390036+0000","last_unstale":"2024-06-29T15:58:32.383401+0000","last_undegraded":"2024-06-29T15:58:32.383401+0000","last_fullsized":"2024-06-29T15:58:32.383401+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:01.353128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470930+0000","last_change":"2024-06-29T15:58:30.380541+0000","last_active":"2024-06-29T15:58:32.470930+0000","last_peered":"2024-06-29T15:58:32.470930+0000","last_clean":"2024-06-29T15:58:32.470930+0000","last_became_active":"2024-06-29T15:58:30.380226+0000","last_became_peered":"2024-06-29T15:58:30.380226+0000","last_unstale":"2024-06-29T15:58:32.470930+0000","last_undegraded":"2024-06-29T15:58:32.470930+0000","last_fullsized":"2024-06-29T15:58:32.470930+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:32.457644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383220+0000","last_change":"2024-06-29T15:58:19.297774+0000","last_active":"2024-06-29T15:58:32.383220+0000","last_peered":"2024-06-29T15:58:32.383220+0000","last_clean":"2024-06-29T15:58:32.383220+0000","last_became_active":"2024-06-29T15:58:19.297463+0000","last_became_peered":"2024-06-29T15:58:19.297463+0000","last_unstale":"2024-06-29T15:58:32.383220+0000","last_undegraded":"2024-06-29T15:58:32.383220+0000","last_fullsized":"2024-06-29T15:58:32.383220+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:38.755714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403906+0000","last_change":"2024-06-29T15:58:31.400649+0000","last_active":"2024-06-29T15:58:32.403906+0000","last_peered":"2024-06-29T15:58:32.403906+0000","last_clean":"2024-06-29T15:58:32.403906+0000","last_became_active":"2024-06-29T15:58:31.400335+0000","last_became_peered":"2024-06-29T15:58:31.400335+0000","last_unstale":"2024-06-29T15:58:32.403906+0000","last_undegraded":"2024-06-29T15:58:32.403906+0000","last_fullsized":"2024-06-29T15:58:32.403906+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:15:28.726085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470916+0000","last_change":"2024-06-29T15:58:30.390704+0000","last_active":"2024-06-29T15:58:32.470916+0000","last_peered":"2024-06-29T15:58:32.470916+0000","last_clean":"2024-06-29T15:58:32.470916+0000","last_became_active":"2024-06-29T15:58:30.390373+0000","last_became_peered":"2024-06-29T15:58:30.390373+0000","last_unstale":"2024-06-29T15:58:32.470916+0000","last_undegraded":"2024-06-29T15:58:32.470916+0000","last_fullsized":"2024-06-29T15:58:32.470916+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:18.439877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403458+0000","last_change":"2024-06-29T15:58:19.313121+0000","last_active":"2024-06-29T15:58:32.403458+0000","last_peered":"2024-06-29T15:58:32.403458+0000","last_clean":"2024-06-29T15:58:32.403458+0000","last_became_active":"2024-06-29T15:58:19.312783+0000","last_became_peered":"2024-06-29T15:58:19.312783+0000","last_unstale":"2024-06-29T15:58:32.403458+0000","last_undegraded":"2024-06-29T15:58:32.403458+0000","last_fullsized":"2024-06-29T15:58:32.403458+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:00:42.731203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403597+0000","last_change":"2024-06-29T15:58:31.399640+0000","last_active":"2024-06-29T15:58:32.403597+0000","last_peered":"2024-06-29T15:58:32.403597+0000","last_clean":"2024-06-29T15:58:32.403597+0000","last_became_active":"2024-06-29T15:58:31.399297+0000","last_became_peered":"2024-06-29T15:58:31.399297+0000","last_unstale":"2024-06-29T15:58:32.403597+0000","last_undegraded":"2024-06-29T15:58:32.403597+0000","last_fullsized":"2024-06-29T15:58:32.403597+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:50.068201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385919+0000","last_change":"2024-06-29T15:58:30.388387+0000","last_active":"2024-06-29T15:58:32.385919+0000","last_peered":"2024-06-29T15:58:32.385919+0000","last_clean":"2024-06-29T15:58:32.385919+0000","last_became_active":"2024-06-29T15:58:30.388001+0000","last_became_peered":"2024-06-29T15:58:30.388001+0000","last_unstale":"2024-06-29T15:58:32.385919+0000","last_undegraded":"2024-06-29T15:58:32.385919+0000","last_fullsized":"2024-06-29T15:58:32.385919+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:07.535813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383187+0000","last_change":"2024-06-29T15:58:19.302569+0000","last_active":"2024-06-29T15:58:32.383187+0000","last_peered":"2024-06-29T15:58:32.383187+0000","last_clean":"2024-06-29T15:58:32.383187+0000","last_became_active":"2024-06-29T15:58:19.302261+0000","last_became_peered":"2024-06-29T15:58:19.302261+0000","last_unstale":"2024-06-29T15:58:32.383187+0000","last_undegraded":"2024-06-29T15:58:32.383187+0000","last_fullsized":"2024-06-29T15:58:32.383187+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:44:48.771783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471441+0000","last_change":"2024-06-29T15:58:31.403127+0000","last_active":"2024-06-29T15:58:32.471441+0000","last_peered":"2024-06-29T15:58:32.471441+0000","last_clean":"2024-06-29T15:58:32.471441+0000","last_became_active":"2024-06-29T15:58:31.402843+0000","last_became_peered":"2024-06-29T15:58:31.402843+0000","last_unstale":"2024-06-29T15:58:32.471441+0000","last_undegraded":"2024-06-29T15:58:32.471441+0000","last_fullsized":"2024-06-29T15:58:32.471441+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:07.614128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"237'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.405357+0000","last_change":"2024-06-29T15:58:30.388669+0000","last_active":"2024-06-29T15:58:32.405357+0000","last_peered":"2024-06-29T15:58:32.405357+0000","last_clean":"2024-06-29T15:58:32.405357+0000","last_became_active":"2024-06-29T15:58:30.388310+0000","last_became_peered":"2024-06-29T15:58:30.388310+0000","last_unstale":"2024-06-29T15:58:32.405357+0000","last_undegraded":"2024-06-29T15:58:32.405357+0000","last_fullsized":"2024-06-29T15:58:32.405357+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:36.967839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385652+0000","last_change":"2024-06-29T15:58:19.314579+0000","last_active":"2024-06-29T15:58:32.385652+0000","last_peered":"2024-06-29T15:58:32.385652+0000","last_clean":"2024-06-29T15:58:32.385652+0000","last_became_active":"2024-06-29T15:58:19.314282+0000","last_became_peered":"2024-06-29T15:58:19.314282+0000","last_unstale":"2024-06-29T15:58:32.385652+0000","last_undegraded":"2024-06-29T15:58:32.385652+0000","last_fullsized":"2024-06-29T15:58:32.385652+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:10.919759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382961+0000","last_change":"2024-06-29T15:58:31.402794+0000","last_active":"2024-06-29T15:58:32.382961+0000","last_peered":"2024-06-29T15:58:32.382961+0000","last_clean":"2024-06-29T15:58:32.382961+0000","last_became_active":"2024-06-29T15:58:31.402013+0000","last_became_peered":"2024-06-29T15:58:31.402013+0000","last_unstale":"2024-06-29T15:58:32.382961+0000","last_undegraded":"2024-06-29T15:58:32.382961+0000","last_fullsized":"2024-06-29T15:58:32.382961+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:30.523422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383240+0000","last_change":"2024-06-29T15:58:30.386798+0000","last_active":"2024-06-29T15:58:32.383240+0000","last_peered":"2024-06-29T15:58:32.383240+0000","last_clean":"2024-06-29T15:58:32.383240+0000","last_became_active":"2024-06-29T15:58:30.386468+0000","last_became_peered":"2024-06-29T15:58:30.386468+0000","last_unstale":"2024-06-29T15:58:32.383240+0000","last_undegraded":"2024-06-29T15:58:32.383240+0000","last_fullsized":"2024-06-29T15:58:32.383240+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:20.401909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470384+0000","last_change":"2024-06-29T15:58:19.303095+0000","last_active":"2024-06-29T15:58:32.470384+0000","last_peered":"2024-06-29T15:58:32.470384+0000","last_clean":"2024-06-29T15:58:32.470384+0000","last_became_active":"2024-06-29T15:58:19.302658+0000","last_became_peered":"2024-06-29T15:58:19.302658+0000","last_unstale":"2024-06-29T15:58:32.470384+0000","last_undegraded":"2024-06-29T15:58:32.470384+0000","last_fullsized":"2024-06-29T15:58:32.470384+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:22.896379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470283+0000","last_change":"2024-06-29T15:58:31.396045+0000","last_active":"2024-06-29T15:58:32.470283+0000","last_peered":"2024-06-29T15:58:32.470283+0000","last_clean":"2024-06-29T15:58:32.470283+0000","last_became_active":"2024-06-29T15:58:31.393369+0000","last_became_peered":"2024-06-29T15:58:31.393369+0000","last_unstale":"2024-06-29T15:58:32.470283+0000","last_undegraded":"2024-06-29T15:58:32.470283+0000","last_fullsized":"2024-06-29T15:58:32.470283+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:43:08.948872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470246+0000","last_change":"2024-06-29T15:58:30.381617+0000","last_active":"2024-06-29T15:58:32.470246+0000","last_peered":"2024-06-29T15:58:32.470246+0000","last_clean":"2024-06-29T15:58:32.470246+0000","last_became_active":"2024-06-29T15:58:30.381233+0000","last_became_peered":"2024-06-29T15:58:30.381233+0000","last_unstale":"2024-06-29T15:58:32.470246+0000","last_undegraded":"2024-06-29T15:58:32.470246+0000","last_fullsized":"2024-06-29T15:58:32.470246+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:28:29.486781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384701+0000","last_change":"2024-06-29T15:58:19.310261+0000","last_active":"2024-06-29T15:58:32.384701+0000","last_peered":"2024-06-29T15:58:32.384701+0000","last_clean":"2024-06-29T15:58:32.384701+0000","last_became_active":"2024-06-29T15:58:19.309742+0000","last_became_peered":"2024-06-29T15:58:19.309742+0000","last_unstale":"2024-06-29T15:58:32.384701+0000","last_undegraded":"2024-06-29T15:58:32.384701+0000","last_fullsized":"2024-06-29T15:58:32.384701+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:55:23.521010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471766+0000","last_change":"2024-06-29T15:58:31.403840+0000","last_active":"2024-06-29T15:58:32.471766+0000","last_peered":"2024-06-29T15:58:32.471766+0000","last_clean":"2024-06-29T15:58:32.471766+0000","last_became_active":"2024-06-29T15:58:31.403530+0000","last_became_peered":"2024-06-29T15:58:31.403530+0000","last_unstale":"2024-06-29T15:58:32.471766+0000","last_undegraded":"2024-06-29T15:58:32.471766+0000","last_fullsized":"2024-06-29T15:58:32.471766+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:52.504053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383953+0000","last_change":"2024-06-29T15:58:30.387115+0000","last_active":"2024-06-29T15:58:32.383953+0000","last_peered":"2024-06-29T15:58:32.383953+0000","last_clean":"2024-06-29T15:58:32.383953+0000","last_became_active":"2024-06-29T15:58:30.386778+0000","last_became_peered":"2024-06-29T15:58:30.386778+0000","last_unstale":"2024-06-29T15:58:32.383953+0000","last_undegraded":"2024-06-29T15:58:32.383953+0000","last_fullsized":"2024-06-29T15:58:32.383953+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:27.674515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386450+0000","last_change":"2024-06-29T15:58:19.315434+0000","last_active":"2024-06-29T15:58:32.386450+0000","last_peered":"2024-06-29T15:58:32.386450+0000","last_clean":"2024-06-29T15:58:32.386450+0000","last_became_active":"2024-06-29T15:58:19.315065+0000","last_became_peered":"2024-06-29T15:58:19.315065+0000","last_unstale":"2024-06-29T15:58:32.386450+0000","last_undegraded":"2024-06-29T15:58:32.386450+0000","last_fullsized":"2024-06-29T15:58:32.386450+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:21.882892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409228+0000","last_change":"2024-06-29T15:58:31.391129+0000","last_active":"2024-06-29T15:58:32.409228+0000","last_peered":"2024-06-29T15:58:32.409228+0000","last_clean":"2024-06-29T15:58:32.409228+0000","last_became_active":"2024-06-29T15:58:31.390831+0000","last_became_peered":"2024-06-29T15:58:31.390831+0000","last_unstale":"2024-06-29T15:58:32.409228+0000","last_undegraded":"2024-06-29T15:58:32.409228+0000","last_fullsized":"2024-06-29T15:58:32.409228+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:57:22.376570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470457+0000","last_change":"2024-06-29T15:58:30.384784+0000","last_active":"2024-06-29T15:58:32.470457+0000","last_peered":"2024-06-29T15:58:32.470457+0000","last_clean":"2024-06-29T15:58:32.470457+0000","last_became_active":"2024-06-29T15:58:30.384431+0000","last_became_peered":"2024-06-29T15:58:30.384431+0000","last_unstale":"2024-06-29T15:58:32.470457+0000","last_undegraded":"2024-06-29T15:58:32.470457+0000","last_fullsized":"2024-06-29T15:58:32.470457+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:58.103230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470619+0000","last_change":"2024-06-29T15:58:19.312563+0000","last_active":"2024-06-29T15:58:32.470619+0000","last_peered":"2024-06-29T15:58:32.470619+0000","last_clean":"2024-06-29T15:58:32.470619+0000","last_became_active":"2024-06-29T15:58:19.311751+0000","last_became_peered":"2024-06-29T15:58:19.311751+0000","last_unstale":"2024-06-29T15:58:32.470619+0000","last_undegraded":"2024-06-29T15:58:32.470619+0000","last_fullsized":"2024-06-29T15:58:32.470619+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:19.583540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409058+0000","last_change":"2024-06-29T15:58:31.400729+0000","last_active":"2024-06-29T15:58:32.409058+0000","last_peered":"2024-06-29T15:58:32.409058+0000","last_clean":"2024-06-29T15:58:32.409058+0000","last_became_active":"2024-06-29T15:58:31.400437+0000","last_became_peered":"2024-06-29T15:58:31.400437+0000","last_unstale":"2024-06-29T15:58:32.409058+0000","last_undegraded":"2024-06-29T15:58:32.409058+0000","last_fullsized":"2024-06-29T15:58:32.409058+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:39.841958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403390+0000","last_change":"2024-06-29T15:58:30.380368+0000","last_active":"2024-06-29T15:58:32.403390+0000","last_peered":"2024-06-29T15:58:32.403390+0000","last_clean":"2024-06-29T15:58:32.403390+0000","last_became_active":"2024-06-29T15:58:30.380035+0000","last_became_peered":"2024-06-29T15:58:30.380035+0000","last_unstale":"2024-06-29T15:58:32.403390+0000","last_undegraded":"2024-06-29T15:58:32.403390+0000","last_fullsized":"2024-06-29T15:58:32.403390+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:52:05.431742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471041+0000","last_change":"2024-06-29T15:58:19.303967+0000","last_active":"2024-06-29T15:58:32.471041+0000","last_peered":"2024-06-29T15:58:32.471041+0000","last_clean":"2024-06-29T15:58:32.471041+0000","last_became_active":"2024-06-29T15:58:19.303620+0000","last_became_peered":"2024-06-29T15:58:19.303620+0000","last_unstale":"2024-06-29T15:58:32.471041+0000","last_undegraded":"2024-06-29T15:58:32.471041+0000","last_fullsized":"2024-06-29T15:58:32.471041+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:17:57.874256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403756+0000","last_change":"2024-06-29T15:58:31.399661+0000","last_active":"2024-06-29T15:58:32.403756+0000","last_peered":"2024-06-29T15:58:32.403756+0000","last_clean":"2024-06-29T15:58:32.403756+0000","last_became_active":"2024-06-29T15:58:31.399309+0000","last_became_peered":"2024-06-29T15:58:31.399309+0000","last_unstale":"2024-06-29T15:58:32.403756+0000","last_undegraded":"2024-06-29T15:58:32.403756+0000","last_fullsized":"2024-06-29T15:58:32.403756+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:26:06.039374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383117+0000","last_change":"2024-06-29T15:58:30.389595+0000","last_active":"2024-06-29T15:58:32.383117+0000","last_peered":"2024-06-29T15:58:32.383117+0000","last_clean":"2024-06-29T15:58:32.383117+0000","last_became_active":"2024-06-29T15:58:30.389243+0000","last_became_peered":"2024-06-29T15:58:30.389243+0000","last_unstale":"2024-06-29T15:58:32.383117+0000","last_undegraded":"2024-06-29T15:58:32.383117+0000","last_fullsized":"2024-06-29T15:58:32.383117+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:21.928277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386211+0000","last_change":"2024-06-29T15:58:19.317397+0000","last_active":"2024-06-29T15:58:32.386211+0000","last_peered":"2024-06-29T15:58:32.386211+0000","last_clean":"2024-06-29T15:58:32.386211+0000","last_became_active":"2024-06-29T15:58:19.317110+0000","last_became_peered":"2024-06-29T15:58:19.317110+0000","last_unstale":"2024-06-29T15:58:32.386211+0000","last_undegraded":"2024-06-29T15:58:32.386211+0000","last_fullsized":"2024-06-29T15:58:32.386211+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:33.637292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383004+0000","last_change":"2024-06-29T15:58:31.398339+0000","last_active":"2024-06-29T15:58:32.383004+0000","last_peered":"2024-06-29T15:58:32.383004+0000","last_clean":"2024-06-29T15:58:32.383004+0000","last_became_active":"2024-06-29T15:58:31.398019+0000","last_became_peered":"2024-06-29T15:58:31.398019+0000","last_unstale":"2024-06-29T15:58:32.383004+0000","last_undegraded":"2024-06-29T15:58:32.383004+0000","last_fullsized":"2024-06-29T15:58:32.383004+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:14.288552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"236'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383906+0000","last_change":"2024-06-29T15:58:30.386307+0000","last_active":"2024-06-29T15:58:32.383906+0000","last_peered":"2024-06-29T15:58:32.383906+0000","last_clean":"2024-06-29T15:58:32.383906+0000","last_became_active":"2024-06-29T15:58:30.386016+0000","last_became_peered":"2024-06-29T15:58:30.386016+0000","last_unstale":"2024-06-29T15:58:32.383906+0000","last_undegraded":"2024-06-29T15:58:32.383906+0000","last_fullsized":"2024-06-29T15:58:32.383906+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:27:54.276566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408629+0000","last_change":"2024-06-29T15:58:19.308273+0000","last_active":"2024-06-29T15:58:32.408629+0000","last_peered":"2024-06-29T15:58:32.408629+0000","last_clean":"2024-06-29T15:58:32.408629+0000","last_became_active":"2024-06-29T15:58:19.307956+0000","last_became_peered":"2024-06-29T15:58:19.307956+0000","last_unstale":"2024-06-29T15:58:32.408629+0000","last_undegraded":"2024-06-29T15:58:32.408629+0000","last_fullsized":"2024-06-29T15:58:32.408629+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:48.781692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384631+0000","last_change":"2024-06-29T15:58:31.409700+0000","last_active":"2024-06-29T15:58:32.384631+0000","last_peered":"2024-06-29T15:58:32.384631+0000","last_clean":"2024-06-29T15:58:32.384631+0000","last_became_active":"2024-06-29T15:58:31.409352+0000","last_became_peered":"2024-06-29T15:58:31.409352+0000","last_unstale":"2024-06-29T15:58:32.384631+0000","last_undegraded":"2024-06-29T15:58:32.384631+0000","last_fullsized":"2024-06-29T15:58:32.384631+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:23.560000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408975+0000","last_change":"2024-06-29T15:58:30.377175+0000","last_active":"2024-06-29T15:58:32.408975+0000","last_peered":"2024-06-29T15:58:32.408975+0000","last_clean":"2024-06-29T15:58:32.408975+0000","last_became_active":"2024-06-29T15:58:30.376867+0000","last_became_peered":"2024-06-29T15:58:30.376867+0000","last_unstale":"2024-06-29T15:58:32.408975+0000","last_undegraded":"2024-06-29T15:58:32.408975+0000","last_fullsized":"2024-06-29T15:58:32.408975+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:07.494825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384041+0000","last_change":"2024-06-29T15:58:19.300026+0000","last_active":"2024-06-29T15:58:32.384041+0000","last_peered":"2024-06-29T15:58:32.384041+0000","last_clean":"2024-06-29T15:58:32.384041+0000","last_became_active":"2024-06-29T15:58:19.298956+0000","last_became_peered":"2024-06-29T15:58:19.298956+0000","last_unstale":"2024-06-29T15:58:32.384041+0000","last_undegraded":"2024-06-29T15:58:32.384041+0000","last_fullsized":"2024-06-29T15:58:32.384041+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:10.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383728+0000","last_change":"2024-06-29T15:58:31.399206+0000","last_active":"2024-06-29T15:58:32.383728+0000","last_peered":"2024-06-29T15:58:32.383728+0000","last_clean":"2024-06-29T15:58:32.383728+0000","last_became_active":"2024-06-29T15:58:31.398935+0000","last_became_peered":"2024-06-29T15:58:31.398935+0000","last_unstale":"2024-06-29T15:58:32.383728+0000","last_undegraded":"2024-06-29T15:58:32.383728+0000","last_fullsized":"2024-06-29T15:58:32.383728+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:08.542689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"236'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471114+0000","last_change":"2024-06-29T15:58:30.389011+0000","last_active":"2024-06-29T15:58:32.471114+0000","last_peered":"2024-06-29T15:58:32.471114+0000","last_clean":"2024-06-29T15:58:32.471114+0000","last_became_active":"2024-06-29T15:58:30.388677+0000","last_became_peered":"2024-06-29T15:58:30.388677+0000","last_unstale":"2024-06-29T15:58:32.471114+0000","last_undegraded":"2024-06-29T15:58:32.471114+0000","last_fullsized":"2024-06-29T15:58:32.471114+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:27.538117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471685+0000","last_change":"2024-06-29T15:58:19.308922+0000","last_active":"2024-06-29T15:58:32.471685+0000","last_peered":"2024-06-29T15:58:32.471685+0000","last_clean":"2024-06-29T15:58:32.471685+0000","last_became_active":"2024-06-29T15:58:19.308502+0000","last_became_peered":"2024-06-29T15:58:19.308502+0000","last_unstale":"2024-06-29T15:58:32.471685+0000","last_undegraded":"2024-06-29T15:58:32.471685+0000","last_fullsized":"2024-06-29T15:58:32.471685+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:14.372622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470577+0000","last_change":"2024-06-29T15:58:31.396486+0000","last_active":"2024-06-29T15:58:32.470577+0000","last_peered":"2024-06-29T15:58:32.470577+0000","last_clean":"2024-06-29T15:58:32.470577+0000","last_became_active":"2024-06-29T15:58:31.396169+0000","last_became_peered":"2024-06-29T15:58:31.396169+0000","last_unstale":"2024-06-29T15:58:32.470577+0000","last_undegraded":"2024-06-29T15:58:32.470577+0000","last_fullsized":"2024-06-29T15:58:32.470577+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:30.206980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"236'8","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408571+0000","last_change":"2024-06-29T15:58:30.378533+0000","last_active":"2024-06-29T15:58:32.408571+0000","last_peered":"2024-06-29T15:58:32.408571+0000","last_clean":"2024-06-29T15:58:32.408571+0000","last_became_active":"2024-06-29T15:58:30.378214+0000","last_became_peered":"2024-06-29T15:58:30.378214+0000","last_unstale":"2024-06-29T15:58:32.408571+0000","last_undegraded":"2024-06-29T15:58:32.408571+0000","last_fullsized":"2024-06-29T15:58:32.408571+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:18.807055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471254+0000","last_change":"2024-06-29T15:58:19.309759+0000","last_active":"2024-06-29T15:58:32.471254+0000","last_peered":"2024-06-29T15:58:32.471254+0000","last_clean":"2024-06-29T15:58:32.471254+0000","last_became_active":"2024-06-29T15:58:19.309439+0000","last_became_peered":"2024-06-29T15:58:19.309439+0000","last_unstale":"2024-06-29T15:58:32.471254+0000","last_undegraded":"2024-06-29T15:58:32.471254+0000","last_fullsized":"2024-06-29T15:58:32.471254+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:54.499719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471585+0000","last_change":"2024-06-29T15:58:31.398394+0000","last_active":"2024-06-29T15:58:32.471585+0000","last_peered":"2024-06-29T15:58:32.471585+0000","last_clean":"2024-06-29T15:58:32.471585+0000","last_became_active":"2024-06-29T15:58:31.397578+0000","last_became_peered":"2024-06-29T15:58:31.397578+0000","last_unstale":"2024-06-29T15:58:32.471585+0000","last_undegraded":"2024-06-29T15:58:32.471585+0000","last_fullsized":"2024-06-29T15:58:32.471585+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:54:59.756820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384235+0000","last_change":"2024-06-29T15:58:30.385343+0000","last_active":"2024-06-29T15:58:32.384235+0000","last_peered":"2024-06-29T15:58:32.384235+0000","last_clean":"2024-06-29T15:58:32.384235+0000","last_became_active":"2024-06-29T15:58:30.384981+0000","last_became_peered":"2024-06-29T15:58:30.384981+0000","last_unstale":"2024-06-29T15:58:32.384235+0000","last_undegraded":"2024-06-29T15:58:32.384235+0000","last_fullsized":"2024-06-29T15:58:32.384235+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:18.439877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409134+0000","last_change":"2024-06-29T15:58:19.306249+0000","last_active":"2024-06-29T15:58:32.409134+0000","last_peered":"2024-06-29T15:58:32.409134+0000","last_clean":"2024-06-29T15:58:32.409134+0000","last_became_active":"2024-06-29T15:58:19.305935+0000","last_became_peered":"2024-06-29T15:58:19.305935+0000","last_unstale":"2024-06-29T15:58:32.409134+0000","last_undegraded":"2024-06-29T15:58:32.409134+0000","last_fullsized":"2024-06-29T15:58:32.409134+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:52.246043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409243+0000","last_change":"2024-06-29T15:58:31.405635+0000","last_active":"2024-06-29T15:58:32.409243+0000","last_peered":"2024-06-29T15:58:32.409243+0000","last_clean":"2024-06-29T15:58:32.409243+0000","last_became_active":"2024-06-29T15:58:31.405306+0000","last_became_peered":"2024-06-29T15:58:31.405306+0000","last_unstale":"2024-06-29T15:58:32.409243+0000","last_undegraded":"2024-06-29T15:58:32.409243+0000","last_fullsized":"2024-06-29T15:58:32.409243+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:32:40.005109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470532+0000","last_change":"2024-06-29T15:58:30.385427+0000","last_active":"2024-06-29T15:58:32.470532+0000","last_peered":"2024-06-29T15:58:32.470532+0000","last_clean":"2024-06-29T15:58:32.470532+0000","last_became_active":"2024-06-29T15:58:30.385055+0000","last_became_peered":"2024-06-29T15:58:30.385055+0000","last_unstale":"2024-06-29T15:58:32.470532+0000","last_undegraded":"2024-06-29T15:58:32.470532+0000","last_fullsized":"2024-06-29T15:58:32.470532+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:32.959366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470963+0000","last_change":"2024-06-29T15:58:19.313204+0000","last_active":"2024-06-29T15:58:32.470963+0000","last_peered":"2024-06-29T15:58:32.470963+0000","last_clean":"2024-06-29T15:58:32.470963+0000","last_became_active":"2024-06-29T15:58:19.312752+0000","last_became_peered":"2024-06-29T15:58:19.312752+0000","last_unstale":"2024-06-29T15:58:32.470963+0000","last_undegraded":"2024-06-29T15:58:32.470963+0000","last_fullsized":"2024-06-29T15:58:32.470963+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:16.564078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471480+0000","last_change":"2024-06-29T15:58:31.406977+0000","last_active":"2024-06-29T15:58:32.471480+0000","last_peered":"2024-06-29T15:58:32.471480+0000","last_clean":"2024-06-29T15:58:32.471480+0000","last_became_active":"2024-06-29T15:58:31.406714+0000","last_became_peered":"2024-06-29T15:58:31.406714+0000","last_unstale":"2024-06-29T15:58:32.471480+0000","last_undegraded":"2024-06-29T15:58:32.471480+0000","last_fullsized":"2024-06-29T15:58:32.471480+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:07:27.548352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470573+0000","last_change":"2024-06-29T15:58:30.385679+0000","last_active":"2024-06-29T15:58:32.470573+0000","last_peered":"2024-06-29T15:58:32.470573+0000","last_clean":"2024-06-29T15:58:32.470573+0000","last_became_active":"2024-06-29T15:58:30.385296+0000","last_became_peered":"2024-06-29T15:58:30.385296+0000","last_unstale":"2024-06-29T15:58:32.470573+0000","last_undegraded":"2024-06-29T15:58:32.470573+0000","last_fullsized":"2024-06-29T15:58:32.470573+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:15:27.719209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471008+0000","last_change":"2024-06-29T15:58:19.303230+0000","last_active":"2024-06-29T15:58:32.471008+0000","last_peered":"2024-06-29T15:58:32.471008+0000","last_clean":"2024-06-29T15:58:32.471008+0000","last_became_active":"2024-06-29T15:58:19.302904+0000","last_became_peered":"2024-06-29T15:58:19.302904+0000","last_unstale":"2024-06-29T15:58:32.471008+0000","last_undegraded":"2024-06-29T15:58:32.471008+0000","last_fullsized":"2024-06-29T15:58:32.471008+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:30:16.162326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471539+0000","last_change":"2024-06-29T15:58:31.398026+0000","last_active":"2024-06-29T15:58:32.471539+0000","last_peered":"2024-06-29T15:58:32.471539+0000","last_clean":"2024-06-29T15:58:32.471539+0000","last_became_active":"2024-06-29T15:58:31.397728+0000","last_became_peered":"2024-06-29T15:58:31.397728+0000","last_unstale":"2024-06-29T15:58:32.471539+0000","last_undegraded":"2024-06-29T15:58:32.471539+0000","last_fullsized":"2024-06-29T15:58:32.471539+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:31.280890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"236'6","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470884+0000","last_change":"2024-06-29T15:58:30.384767+0000","last_active":"2024-06-29T15:58:32.470884+0000","last_peered":"2024-06-29T15:58:32.470884+0000","last_clean":"2024-06-29T15:58:32.470884+0000","last_became_active":"2024-06-29T15:58:30.384344+0000","last_became_peered":"2024-06-29T15:58:30.384344+0000","last_unstale":"2024-06-29T15:58:32.470884+0000","last_undegraded":"2024-06-29T15:58:32.470884+0000","last_fullsized":"2024-06-29T15:58:32.470884+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:27:54.276566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470659+0000","last_change":"2024-06-29T15:58:19.302093+0000","last_active":"2024-06-29T15:58:32.470659+0000","last_peered":"2024-06-29T15:58:32.470659+0000","last_clean":"2024-06-29T15:58:32.470659+0000","last_became_active":"2024-06-29T15:58:19.301677+0000","last_became_peered":"2024-06-29T15:58:19.301677+0000","last_unstale":"2024-06-29T15:58:32.470659+0000","last_undegraded":"2024-06-29T15:58:32.470659+0000","last_fullsized":"2024-06-29T15:58:32.470659+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:08.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382986+0000","last_change":"2024-06-29T15:58:31.405773+0000","last_active":"2024-06-29T15:58:32.382986+0000","last_peered":"2024-06-29T15:58:32.382986+0000","last_clean":"2024-06-29T15:58:32.382986+0000","last_became_active":"2024-06-29T15:58:31.405406+0000","last_became_peered":"2024-06-29T15:58:31.405406+0000","last_unstale":"2024-06-29T15:58:32.382986+0000","last_undegraded":"2024-06-29T15:58:32.382986+0000","last_fullsized":"2024-06-29T15:58:32.382986+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:43:08.948872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"236'3","reported_seq":17,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408949+0000","last_change":"2024-06-29T15:58:30.389211+0000","last_active":"2024-06-29T15:58:32.408949+0000","last_peered":"2024-06-29T15:58:32.408949+0000","last_clean":"2024-06-29T15:58:32.408949+0000","last_became_active":"2024-06-29T15:58:30.388877+0000","last_became_peered":"2024-06-29T15:58:30.388877+0000","last_unstale":"2024-06-29T15:58:32.408949+0000","last_undegraded":"2024-06-29T15:58:32.408949+0000","last_fullsized":"2024-06-29T15:58:32.408949+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:35:11.381957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385591+0000","last_change":"2024-06-29T15:58:19.307655+0000","last_active":"2024-06-29T15:58:32.385591+0000","last_peered":"2024-06-29T15:58:32.385591+0000","last_clean":"2024-06-29T15:58:32.385591+0000","last_became_active":"2024-06-29T15:58:19.307369+0000","last_became_peered":"2024-06-29T15:58:19.307369+0000","last_unstale":"2024-06-29T15:58:32.385591+0000","last_undegraded":"2024-06-29T15:58:32.385591+0000","last_fullsized":"2024-06-29T15:58:32.385591+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:04.132174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471271+0000","last_change":"2024-06-29T15:58:31.399030+0000","last_active":"2024-06-29T15:58:32.471271+0000","last_peered":"2024-06-29T15:58:32.471271+0000","last_clean":"2024-06-29T15:58:32.471271+0000","last_became_active":"2024-06-29T15:58:31.398440+0000","last_became_peered":"2024-06-29T15:58:31.398440+0000","last_unstale":"2024-06-29T15:58:32.471271+0000","last_undegraded":"2024-06-29T15:58:32.471271+0000","last_fullsized":"2024-06-29T15:58:32.471271+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:07:49.345352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409145+0000","last_change":"2024-06-29T15:58:30.389217+0000","last_active":"2024-06-29T15:58:32.409145+0000","last_peered":"2024-06-29T15:58:32.409145+0000","last_clean":"2024-06-29T15:58:32.409145+0000","last_became_active":"2024-06-29T15:58:30.388891+0000","last_became_peered":"2024-06-29T15:58:30.388891+0000","last_unstale":"2024-06-29T15:58:32.409145+0000","last_undegraded":"2024-06-29T15:58:32.409145+0000","last_fullsized":"2024-06-29T15:58:32.409145+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:57:00.261000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470277+0000","last_change":"2024-06-29T15:58:19.308486+0000","last_active":"2024-06-29T15:58:32.470277+0000","last_peered":"2024-06-29T15:58:32.470277+0000","last_clean":"2024-06-29T15:58:32.470277+0000","last_became_active":"2024-06-29T15:58:19.308158+0000","last_became_peered":"2024-06-29T15:58:19.308158+0000","last_unstale":"2024-06-29T15:58:32.470277+0000","last_undegraded":"2024-06-29T15:58:32.470277+0000","last_fullsized":"2024-06-29T15:58:32.470277+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:38.755714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386243+0000","last_change":"2024-06-29T15:58:31.405690+0000","last_active":"2024-06-29T15:58:32.386243+0000","last_peered":"2024-06-29T15:58:32.386243+0000","last_clean":"2024-06-29T15:58:32.386243+0000","last_became_active":"2024-06-29T15:58:31.404937+0000","last_became_peered":"2024-06-29T15:58:31.404937+0000","last_unstale":"2024-06-29T15:58:32.386243+0000","last_undegraded":"2024-06-29T15:58:32.386243+0000","last_fullsized":"2024-06-29T15:58:32.386243+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:34.598819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471775+0000","last_change":"2024-06-29T15:58:30.390358+0000","last_active":"2024-06-29T15:58:32.471775+0000","last_peered":"2024-06-29T15:58:32.471775+0000","last_clean":"2024-06-29T15:58:32.471775+0000","last_became_active":"2024-06-29T15:58:30.390049+0000","last_became_peered":"2024-06-29T15:58:30.390049+0000","last_unstale":"2024-06-29T15:58:32.471775+0000","last_undegraded":"2024-06-29T15:58:32.471775+0000","last_fullsized":"2024-06-29T15:58:32.471775+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:39:16.986459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408636+0000","last_change":"2024-06-29T15:58:19.312754+0000","last_active":"2024-06-29T15:58:32.408636+0000","last_peered":"2024-06-29T15:58:32.408636+0000","last_clean":"2024-06-29T15:58:32.408636+0000","last_became_active":"2024-06-29T15:58:19.312061+0000","last_became_peered":"2024-06-29T15:58:19.312061+0000","last_unstale":"2024-06-29T15:58:32.408636+0000","last_undegraded":"2024-06-29T15:58:32.408636+0000","last_fullsized":"2024-06-29T15:58:32.408636+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:10.919759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385564+0000","last_change":"2024-06-29T15:58:31.405442+0000","last_active":"2024-06-29T15:58:32.385564+0000","last_peered":"2024-06-29T15:58:32.385564+0000","last_clean":"2024-06-29T15:58:32.385564+0000","last_became_active":"2024-06-29T15:58:31.405046+0000","last_became_peered":"2024-06-29T15:58:31.405046+0000","last_unstale":"2024-06-29T15:58:32.385564+0000","last_undegraded":"2024-06-29T15:58:32.385564+0000","last_fullsized":"2024-06-29T15:58:32.385564+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:57.525409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385164+0000","last_change":"2024-06-29T15:58:30.391143+0000","last_active":"2024-06-29T15:58:32.385164+0000","last_peered":"2024-06-29T15:58:32.385164+0000","last_clean":"2024-06-29T15:58:32.385164+0000","last_became_active":"2024-06-29T15:58:30.390774+0000","last_became_peered":"2024-06-29T15:58:30.390774+0000","last_unstale":"2024-06-29T15:58:32.385164+0000","last_undegraded":"2024-06-29T15:58:32.385164+0000","last_fullsized":"2024-06-29T15:58:32.385164+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:00.346252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383385+0000","last_change":"2024-06-29T15:58:19.295810+0000","last_active":"2024-06-29T15:58:32.383385+0000","last_peered":"2024-06-29T15:58:32.383385+0000","last_clean":"2024-06-29T15:58:32.383385+0000","last_became_active":"2024-06-29T15:58:19.295445+0000","last_became_peered":"2024-06-29T15:58:19.295445+0000","last_unstale":"2024-06-29T15:58:32.383385+0000","last_undegraded":"2024-06-29T15:58:32.383385+0000","last_fullsized":"2024-06-29T15:58:32.383385+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:49:49.141794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408487+0000","last_change":"2024-06-29T15:58:31.402689+0000","last_active":"2024-06-29T15:58:32.408487+0000","last_peered":"2024-06-29T15:58:32.408487+0000","last_clean":"2024-06-29T15:58:32.408487+0000","last_became_active":"2024-06-29T15:58:31.402393+0000","last_became_peered":"2024-06-29T15:58:31.402393+0000","last_unstale":"2024-06-29T15:58:32.408487+0000","last_undegraded":"2024-06-29T15:58:32.408487+0000","last_fullsized":"2024-06-29T15:58:32.408487+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:15.757153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403354+0000","last_change":"2024-06-29T15:58:30.390626+0000","last_active":"2024-06-29T15:58:32.403354+0000","last_peered":"2024-06-29T15:58:32.403354+0000","last_clean":"2024-06-29T15:58:32.403354+0000","last_became_active":"2024-06-29T15:58:30.390230+0000","last_became_peered":"2024-06-29T15:58:30.390230+0000","last_unstale":"2024-06-29T15:58:32.403354+0000","last_undegraded":"2024-06-29T15:58:32.403354+0000","last_fullsized":"2024-06-29T15:58:32.403354+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:50:00.218269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386569+0000","last_change":"2024-06-29T15:58:19.315083+0000","last_active":"2024-06-29T15:58:32.386569+0000","last_peered":"2024-06-29T15:58:32.386569+0000","last_clean":"2024-06-29T15:58:32.386569+0000","last_became_active":"2024-06-29T15:58:19.313876+0000","last_became_peered":"2024-06-29T15:58:19.313876+0000","last_unstale":"2024-06-29T15:58:32.386569+0000","last_undegraded":"2024-06-29T15:58:32.386569+0000","last_fullsized":"2024-06-29T15:58:32.386569+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:57.766748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409078+0000","last_change":"2024-06-29T15:58:31.395696+0000","last_active":"2024-06-29T15:58:32.409078+0000","last_peered":"2024-06-29T15:58:32.409078+0000","last_clean":"2024-06-29T15:58:32.409078+0000","last_became_active":"2024-06-29T15:58:31.395401+0000","last_became_peered":"2024-06-29T15:58:31.395401+0000","last_unstale":"2024-06-29T15:58:32.409078+0000","last_undegraded":"2024-06-29T15:58:32.409078+0000","last_fullsized":"2024-06-29T15:58:32.409078+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:34.598819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"236'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471285+0000","last_change":"2024-06-29T15:58:30.383069+0000","last_active":"2024-06-29T15:58:32.471285+0000","last_peered":"2024-06-29T15:58:32.471285+0000","last_clean":"2024-06-29T15:58:32.471285+0000","last_became_active":"2024-06-29T15:58:30.382783+0000","last_became_peered":"2024-06-29T15:58:30.382783+0000","last_unstale":"2024-06-29T15:58:32.471285+0000","last_undegraded":"2024-06-29T15:58:32.471285+0000","last_fullsized":"2024-06-29T15:58:32.471285+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:27.674515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470719+0000","last_change":"2024-06-29T15:58:19.309408+0000","last_active":"2024-06-29T15:58:32.470719+0000","last_peered":"2024-06-29T15:58:32.470719+0000","last_clean":"2024-06-29T15:58:32.470719+0000","last_became_active":"2024-06-29T15:58:19.309050+0000","last_became_peered":"2024-06-29T15:58:19.309050+0000","last_unstale":"2024-06-29T15:58:32.470719+0000","last_undegraded":"2024-06-29T15:58:32.470719+0000","last_fullsized":"2024-06-29T15:58:32.470719+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:53.490742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470731+0000","last_change":"2024-06-29T15:58:31.413593+0000","last_active":"2024-06-29T15:58:32.470731+0000","last_peered":"2024-06-29T15:58:32.470731+0000","last_clean":"2024-06-29T15:58:32.470731+0000","last_became_active":"2024-06-29T15:58:31.413289+0000","last_became_peered":"2024-06-29T15:58:31.413289+0000","last_unstale":"2024-06-29T15:58:32.470731+0000","last_undegraded":"2024-06-29T15:58:32.470731+0000","last_fullsized":"2024-06-29T15:58:32.470731+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:21.755772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"236'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385764+0000","last_change":"2024-06-29T15:58:30.384132+0000","last_active":"2024-06-29T15:58:32.385764+0000","last_peered":"2024-06-29T15:58:32.385764+0000","last_clean":"2024-06-29T15:58:32.385764+0000","last_became_active":"2024-06-29T15:58:30.383773+0000","last_became_peered":"2024-06-29T15:58:30.383773+0000","last_unstale":"2024-06-29T15:58:32.385764+0000","last_undegraded":"2024-06-29T15:58:32.385764+0000","last_fullsized":"2024-06-29T15:58:32.385764+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:49.504495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408881+0000","last_change":"2024-06-29T15:58:19.305539+0000","last_active":"2024-06-29T15:58:32.408881+0000","last_peered":"2024-06-29T15:58:32.408881+0000","last_clean":"2024-06-29T15:58:32.408881+0000","last_became_active":"2024-06-29T15:58:19.305224+0000","last_became_peered":"2024-06-29T15:58:19.305224+0000","last_unstale":"2024-06-29T15:58:32.408881+0000","last_undegraded":"2024-06-29T15:58:32.408881+0000","last_fullsized":"2024-06-29T15:58:32.408881+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:21.381169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470959+0000","last_change":"2024-06-29T15:58:31.395948+0000","last_active":"2024-06-29T15:58:32.470959+0000","last_peered":"2024-06-29T15:58:32.470959+0000","last_clean":"2024-06-29T15:58:32.470959+0000","last_became_active":"2024-06-29T15:58:31.395228+0000","last_became_peered":"2024-06-29T15:58:31.395228+0000","last_unstale":"2024-06-29T15:58:32.470959+0000","last_undegraded":"2024-06-29T15:58:32.470959+0000","last_fullsized":"2024-06-29T15:58:32.470959+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:04.329393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408968+0000","last_change":"2024-06-29T15:58:30.382075+0000","last_active":"2024-06-29T15:58:32.408968+0000","last_peered":"2024-06-29T15:58:32.408968+0000","last_clean":"2024-06-29T15:58:32.408968+0000","last_became_active":"2024-06-29T15:58:30.381739+0000","last_became_peered":"2024-06-29T15:58:30.381739+0000","last_unstale":"2024-06-29T15:58:32.408968+0000","last_undegraded":"2024-06-29T15:58:32.408968+0000","last_fullsized":"2024-06-29T15:58:32.408968+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:08.875118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471086+0000","last_change":"2024-06-29T15:58:19.299124+0000","last_active":"2024-06-29T15:58:32.471086+0000","last_peered":"2024-06-29T15:58:32.471086+0000","last_clean":"2024-06-29T15:58:32.471086+0000","last_became_active":"2024-06-29T15:58:19.298839+0000","last_became_peered":"2024-06-29T15:58:19.298839+0000","last_unstale":"2024-06-29T15:58:32.471086+0000","last_undegraded":"2024-06-29T15:58:32.471086+0000","last_fullsized":"2024-06-29T15:58:32.471086+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:25:53.956024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385768+0000","last_change":"2024-06-29T15:58:31.404399+0000","last_active":"2024-06-29T15:58:32.385768+0000","last_peered":"2024-06-29T15:58:32.385768+0000","last_clean":"2024-06-29T15:58:32.385768+0000","last_became_active":"2024-06-29T15:58:31.402265+0000","last_became_peered":"2024-06-29T15:58:31.402265+0000","last_unstale":"2024-06-29T15:58:32.385768+0000","last_undegraded":"2024-06-29T15:58:32.385768+0000","last_fullsized":"2024-06-29T15:58:32.385768+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:15.757153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409018+0000","last_change":"2024-06-29T15:58:30.374918+0000","last_active":"2024-06-29T15:58:32.409018+0000","last_peered":"2024-06-29T15:58:32.409018+0000","last_clean":"2024-06-29T15:58:32.409018+0000","last_became_active":"2024-06-29T15:58:30.373985+0000","last_became_peered":"2024-06-29T15:58:30.373985+0000","last_unstale":"2024-06-29T15:58:32.409018+0000","last_undegraded":"2024-06-29T15:58:32.409018+0000","last_fullsized":"2024-06-29T15:58:32.409018+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:14:44.088939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403356+0000","last_change":"2024-06-29T15:58:19.298224+0000","last_active":"2024-06-29T15:58:32.403356+0000","last_peered":"2024-06-29T15:58:32.403356+0000","last_clean":"2024-06-29T15:58:32.403356+0000","last_became_active":"2024-06-29T15:58:19.297909+0000","last_became_peered":"2024-06-29T15:58:19.297909+0000","last_unstale":"2024-06-29T15:58:32.403356+0000","last_undegraded":"2024-06-29T15:58:32.403356+0000","last_fullsized":"2024-06-29T15:58:32.403356+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:17.158060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382878+0000","last_change":"2024-06-29T15:58:31.402837+0000","last_active":"2024-06-29T15:58:32.382878+0000","last_peered":"2024-06-29T15:58:32.382878+0000","last_clean":"2024-06-29T15:58:32.382878+0000","last_became_active":"2024-06-29T15:58:31.402439+0000","last_became_peered":"2024-06-29T15:58:31.402439+0000","last_unstale":"2024-06-29T15:58:32.382878+0000","last_undegraded":"2024-06-29T15:58:32.382878+0000","last_fullsized":"2024-06-29T15:58:32.382878+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:23.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"236'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383228+0000","last_change":"2024-06-29T15:58:30.390882+0000","last_active":"2024-06-29T15:58:32.383228+0000","last_peered":"2024-06-29T15:58:32.383228+0000","last_clean":"2024-06-29T15:58:32.383228+0000","last_became_active":"2024-06-29T15:58:30.390622+0000","last_became_peered":"2024-06-29T15:58:30.390622+0000","last_unstale":"2024-06-29T15:58:32.383228+0000","last_undegraded":"2024-06-29T15:58:32.383228+0000","last_fullsized":"2024-06-29T15:58:32.383228+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:49.061326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383542+0000","last_change":"2024-06-29T15:58:19.304924+0000","last_active":"2024-06-29T15:58:32.383542+0000","last_peered":"2024-06-29T15:58:32.383542+0000","last_clean":"2024-06-29T15:58:32.383542+0000","last_became_active":"2024-06-29T15:58:19.304317+0000","last_became_peered":"2024-06-29T15:58:19.304317+0000","last_unstale":"2024-06-29T15:58:32.383542+0000","last_undegraded":"2024-06-29T15:58:32.383542+0000","last_fullsized":"2024-06-29T15:58:32.383542+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:15:28.738310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386796+0000","last_change":"2024-06-29T15:58:31.403333+0000","last_active":"2024-06-29T15:58:32.386796+0000","last_peered":"2024-06-29T15:58:32.386796+0000","last_clean":"2024-06-29T15:58:32.386796+0000","last_became_active":"2024-06-29T15:58:31.402715+0000","last_became_peered":"2024-06-29T15:58:31.402715+0000","last_unstale":"2024-06-29T15:58:32.386796+0000","last_undegraded":"2024-06-29T15:58:32.386796+0000","last_fullsized":"2024-06-29T15:58:32.386796+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:32:40.005109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"236'4","reported_seq":18,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471608+0000","last_change":"2024-06-29T15:58:30.385595+0000","last_active":"2024-06-29T15:58:32.471608+0000","last_peered":"2024-06-29T15:58:32.471608+0000","last_clean":"2024-06-29T15:58:32.471608+0000","last_became_active":"2024-06-29T15:58:30.385187+0000","last_became_peered":"2024-06-29T15:58:30.385187+0000","last_unstale":"2024-06-29T15:58:32.471608+0000","last_undegraded":"2024-06-29T15:58:32.471608+0000","last_fullsized":"2024-06-29T15:58:32.471608+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:13.071438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408823+0000","last_change":"2024-06-29T15:58:19.309529+0000","last_active":"2024-06-29T15:58:32.408823+0000","last_peered":"2024-06-29T15:58:32.408823+0000","last_clean":"2024-06-29T15:58:32.408823+0000","last_became_active":"2024-06-29T15:58:19.309193+0000","last_became_peered":"2024-06-29T15:58:19.309193+0000","last_unstale":"2024-06-29T15:58:32.408823+0000","last_undegraded":"2024-06-29T15:58:32.408823+0000","last_fullsized":"2024-06-29T15:58:32.408823+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:21.882892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470299+0000","last_change":"2024-06-29T15:58:31.398191+0000","last_active":"2024-06-29T15:58:32.470299+0000","last_peered":"2024-06-29T15:58:32.470299+0000","last_clean":"2024-06-29T15:58:32.470299+0000","last_became_active":"2024-06-29T15:58:31.397845+0000","last_became_peered":"2024-06-29T15:58:31.397845+0000","last_unstale":"2024-06-29T15:58:32.470299+0000","last_undegraded":"2024-06-29T15:58:32.470299+0000","last_fullsized":"2024-06-29T15:58:32.470299+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:16.215524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471673+0000","last_change":"2024-06-29T15:58:19.306514+0000","last_active":"2024-06-29T15:58:32.471673+0000","last_peered":"2024-06-29T15:58:32.471673+0000","last_clean":"2024-06-29T15:58:32.471673+0000","last_became_active":"2024-06-29T15:58:19.306099+0000","last_became_peered":"2024-06-29T15:58:19.306099+0000","last_unstale":"2024-06-29T15:58:32.471673+0000","last_undegraded":"2024-06-29T15:58:32.471673+0000","last_fullsized":"2024-06-29T15:58:32.471673+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:25.572276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"232'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386775+0000","last_change":"2024-06-29T15:58:18.301939+0000","last_active":"2024-06-29T15:58:32.386775+0000","last_peered":"2024-06-29T15:58:32.386775+0000","last_clean":"2024-06-29T15:58:32.386775+0000","last_became_active":"2024-06-29T15:58:18.301258+0000","last_became_peered":"2024-06-29T15:58:18.301258+0000","last_unstale":"2024-06-29T15:58:32.386775+0000","last_undegraded":"2024-06-29T15:58:32.386775+0000","last_fullsized":"2024-06-29T15:58:32.386775+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:53.497595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383084+0000","last_change":"2024-06-29T15:58:31.404199+0000","last_active":"2024-06-29T15:58:32.383084+0000","last_peered":"2024-06-29T15:58:32.383084+0000","last_clean":"2024-06-29T15:58:32.383084+0000","last_became_active":"2024-06-29T15:58:31.403910+0000","last_became_peered":"2024-06-29T15:58:31.403910+0000","last_unstale":"2024-06-29T15:58:32.383084+0000","last_undegraded":"2024-06-29T15:58:32.383084+0000","last_fullsized":"2024-06-29T15:58:32.383084+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:21.755772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470398+0000","last_change":"2024-06-29T15:58:19.298015+0000","last_active":"2024-06-29T15:58:32.470398+0000","last_peered":"2024-06-29T15:58:32.470398+0000","last_clean":"2024-06-29T15:58:32.470398+0000","last_became_active":"2024-06-29T15:58:19.297706+0000","last_became_peered":"2024-06-29T15:58:19.297706+0000","last_unstale":"2024-06-29T15:58:32.470398+0000","last_undegraded":"2024-06-29T15:58:32.470398+0000","last_fullsized":"2024-06-29T15:58:32.470398+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:59:36.737724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471128+0000","last_change":"2024-06-29T15:58:18.297534+0000","last_active":"2024-06-29T15:58:32.471128+0000","last_peered":"2024-06-29T15:58:32.471128+0000","last_clean":"2024-06-29T15:58:32.471128+0000","last_became_active":"2024-06-29T15:58:18.297260+0000","last_became_peered":"2024-06-29T15:58:18.297260+0000","last_unstale":"2024-06-29T15:58:32.471128+0000","last_undegraded":"2024-06-29T15:58:32.471128+0000","last_fullsized":"2024-06-29T15:58:32.471128+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:47:47.544183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471365+0000","last_change":"2024-06-29T15:58:31.397157+0000","last_active":"2024-06-29T15:58:32.471365+0000","last_peered":"2024-06-29T15:58:32.471365+0000","last_clean":"2024-06-29T15:58:32.471365+0000","last_became_active":"2024-06-29T15:58:31.396872+0000","last_became_peered":"2024-06-29T15:58:31.396872+0000","last_unstale":"2024-06-29T15:58:32.471365+0000","last_undegraded":"2024-06-29T15:58:32.471365+0000","last_fullsized":"2024-06-29T15:58:32.471365+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:27.765411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471038+0000","last_change":"2024-06-29T15:58:19.305103+0000","last_active":"2024-06-29T15:58:32.471038+0000","last_peered":"2024-06-29T15:58:32.471038+0000","last_clean":"2024-06-29T15:58:32.471038+0000","last_became_active":"2024-06-29T15:58:19.304612+0000","last_became_peered":"2024-06-29T15:58:19.304612+0000","last_unstale":"2024-06-29T15:58:32.471038+0000","last_undegraded":"2024-06-29T15:58:32.471038+0000","last_fullsized":"2024-06-29T15:58:32.471038+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:19.583540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471702+0000","last_change":"2024-06-29T15:58:18.302160+0000","last_active":"2024-06-29T15:58:32.471702+0000","last_peered":"2024-06-29T15:58:32.471702+0000","last_clean":"2024-06-29T15:58:32.471702+0000","last_became_active":"2024-06-29T15:58:18.301895+0000","last_became_peered":"2024-06-29T15:58:18.301895+0000","last_unstale":"2024-06-29T15:58:32.471702+0000","last_undegraded":"2024-06-29T15:58:32.471702+0000","last_fullsized":"2024-06-29T15:58:32.471702+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:26.756484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383158+0000","last_change":"2024-06-29T15:58:31.394840+0000","last_active":"2024-06-29T15:58:32.383158+0000","last_peered":"2024-06-29T15:58:32.383158+0000","last_clean":"2024-06-29T15:58:32.383158+0000","last_became_active":"2024-06-29T15:58:31.393905+0000","last_became_peered":"2024-06-29T15:58:31.393905+0000","last_unstale":"2024-06-29T15:58:32.383158+0000","last_undegraded":"2024-06-29T15:58:32.383158+0000","last_fullsized":"2024-06-29T15:58:32.383158+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:07.236049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470593+0000","last_change":"2024-06-29T15:58:19.299884+0000","last_active":"2024-06-29T15:58:32.470593+0000","last_peered":"2024-06-29T15:58:32.470593+0000","last_clean":"2024-06-29T15:58:32.470593+0000","last_became_active":"2024-06-29T15:58:19.299621+0000","last_became_peered":"2024-06-29T15:58:19.299621+0000","last_unstale":"2024-06-29T15:58:32.470593+0000","last_undegraded":"2024-06-29T15:58:32.470593+0000","last_fullsized":"2024-06-29T15:58:32.470593+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:09.325434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"232'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382745+0000","last_change":"2024-06-29T15:58:18.299460+0000","last_active":"2024-06-29T15:58:32.382745+0000","last_peered":"2024-06-29T15:58:32.382745+0000","last_clean":"2024-06-29T15:58:32.382745+0000","last_became_active":"2024-06-29T15:58:18.299171+0000","last_became_peered":"2024-06-29T15:58:18.299171+0000","last_unstale":"2024-06-29T15:58:32.382745+0000","last_undegraded":"2024-06-29T15:58:32.382745+0000","last_fullsized":"2024-06-29T15:58:32.382745+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:34:40.426770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385789+0000","last_change":"2024-06-29T15:58:31.403071+0000","last_active":"2024-06-29T15:58:32.385789+0000","last_peered":"2024-06-29T15:58:32.385789+0000","last_clean":"2024-06-29T15:58:32.385789+0000","last_became_active":"2024-06-29T15:58:31.402742+0000","last_became_peered":"2024-06-29T15:58:31.402742+0000","last_unstale":"2024-06-29T15:58:32.385789+0000","last_undegraded":"2024-06-29T15:58:32.385789+0000","last_fullsized":"2024-06-29T15:58:32.385789+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:00.629657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384209+0000","last_change":"2024-06-29T15:58:19.306300+0000","last_active":"2024-06-29T15:58:32.384209+0000","last_peered":"2024-06-29T15:58:32.384209+0000","last_clean":"2024-06-29T15:58:32.384209+0000","last_became_active":"2024-06-29T15:58:19.305976+0000","last_became_peered":"2024-06-29T15:58:19.305976+0000","last_unstale":"2024-06-29T15:58:32.384209+0000","last_undegraded":"2024-06-29T15:58:32.384209+0000","last_fullsized":"2024-06-29T15:58:32.384209+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:21.792399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"232'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409092+0000","last_change":"2024-06-29T15:58:18.297242+0000","last_active":"2024-06-29T15:58:32.409092+0000","last_peered":"2024-06-29T15:58:32.409092+0000","last_clean":"2024-06-29T15:58:32.409092+0000","last_became_active":"2024-06-29T15:58:18.296876+0000","last_became_peered":"2024-06-29T15:58:18.296876+0000","last_unstale":"2024-06-29T15:58:32.409092+0000","last_undegraded":"2024-06-29T15:58:32.409092+0000","last_fullsized":"2024-06-29T15:58:32.409092+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:53.497595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403749+0000","last_change":"2024-06-29T15:58:31.408178+0000","last_active":"2024-06-29T15:58:32.403749+0000","last_peered":"2024-06-29T15:58:32.403749+0000","last_clean":"2024-06-29T15:58:32.403749+0000","last_became_active":"2024-06-29T15:58:31.407829+0000","last_became_peered":"2024-06-29T15:58:31.407829+0000","last_unstale":"2024-06-29T15:58:32.403749+0000","last_undegraded":"2024-06-29T15:58:32.403749+0000","last_fullsized":"2024-06-29T15:58:32.403749+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:37.594096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471795+0000","last_change":"2024-06-29T15:58:19.300049+0000","last_active":"2024-06-29T15:58:32.471795+0000","last_peered":"2024-06-29T15:58:32.471795+0000","last_clean":"2024-06-29T15:58:32.471795+0000","last_became_active":"2024-06-29T15:58:19.299756+0000","last_became_peered":"2024-06-29T15:58:19.299756+0000","last_unstale":"2024-06-29T15:58:32.471795+0000","last_undegraded":"2024-06-29T15:58:32.471795+0000","last_fullsized":"2024-06-29T15:58:32.471795+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:25:50.310157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386054+0000","last_change":"2024-06-29T15:58:18.302669+0000","last_active":"2024-06-29T15:58:32.386054+0000","last_peered":"2024-06-29T15:58:32.386054+0000","last_clean":"2024-06-29T15:58:32.386054+0000","last_became_active":"2024-06-29T15:58:18.302098+0000","last_became_peered":"2024-06-29T15:58:18.302098+0000","last_unstale":"2024-06-29T15:58:32.386054+0000","last_undegraded":"2024-06-29T15:58:32.386054+0000","last_fullsized":"2024-06-29T15:58:32.386054+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:18.581416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383328+0000","last_change":"2024-06-29T15:58:31.398566+0000","last_active":"2024-06-29T15:58:32.383328+0000","last_peered":"2024-06-29T15:58:32.383328+0000","last_clean":"2024-06-29T15:58:32.383328+0000","last_became_active":"2024-06-29T15:58:31.398272+0000","last_became_peered":"2024-06-29T15:58:31.398272+0000","last_unstale":"2024-06-29T15:58:32.383328+0000","last_undegraded":"2024-06-29T15:58:32.383328+0000","last_fullsized":"2024-06-29T15:58:32.383328+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:50.511371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408466+0000","last_change":"2024-06-29T15:58:19.308028+0000","last_active":"2024-06-29T15:58:32.408466+0000","last_peered":"2024-06-29T15:58:32.408466+0000","last_clean":"2024-06-29T15:58:32.408466+0000","last_became_active":"2024-06-29T15:58:19.306593+0000","last_became_peered":"2024-06-29T15:58:19.306593+0000","last_unstale":"2024-06-29T15:58:32.408466+0000","last_undegraded":"2024-06-29T15:58:32.408466+0000","last_fullsized":"2024-06-29T15:58:32.408466+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:45.907895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471895+0000","last_change":"2024-06-29T15:58:18.300805+0000","last_active":"2024-06-29T15:58:32.471895+0000","last_peered":"2024-06-29T15:58:32.471895+0000","last_clean":"2024-06-29T15:58:32.471895+0000","last_became_active":"2024-06-29T15:58:18.300515+0000","last_became_peered":"2024-06-29T15:58:18.300515+0000","last_unstale":"2024-06-29T15:58:32.471895+0000","last_undegraded":"2024-06-29T15:58:32.471895+0000","last_fullsized":"2024-06-29T15:58:32.471895+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:14.605715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471605+0000","last_change":"2024-06-29T15:58:31.398694+0000","last_active":"2024-06-29T15:58:32.471605+0000","last_peered":"2024-06-29T15:58:32.471605+0000","last_clean":"2024-06-29T15:58:32.471605+0000","last_became_active":"2024-06-29T15:58:31.398005+0000","last_became_peered":"2024-06-29T15:58:31.398005+0000","last_unstale":"2024-06-29T15:58:32.471605+0000","last_undegraded":"2024-06-29T15:58:32.471605+0000","last_fullsized":"2024-06-29T15:58:32.471605+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:43.892782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409202+0000","last_change":"2024-06-29T15:58:19.303106+0000","last_active":"2024-06-29T15:58:32.409202+0000","last_peered":"2024-06-29T15:58:32.409202+0000","last_clean":"2024-06-29T15:58:32.409202+0000","last_became_active":"2024-06-29T15:58:19.302806+0000","last_became_peered":"2024-06-29T15:58:19.302806+0000","last_unstale":"2024-06-29T15:58:32.409202+0000","last_undegraded":"2024-06-29T15:58:32.409202+0000","last_fullsized":"2024-06-29T15:58:32.409202+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:18.440072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471318+0000","last_change":"2024-06-29T15:58:18.303872+0000","last_active":"2024-06-29T15:58:32.471318+0000","last_peered":"2024-06-29T15:58:32.471318+0000","last_clean":"2024-06-29T15:58:32.471318+0000","last_became_active":"2024-06-29T15:58:18.303542+0000","last_became_peered":"2024-06-29T15:58:18.303542+0000","last_unstale":"2024-06-29T15:58:32.471318+0000","last_undegraded":"2024-06-29T15:58:32.471318+0000","last_fullsized":"2024-06-29T15:58:32.471318+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:04.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385249+0000","last_change":"2024-06-29T15:58:31.403850+0000","last_active":"2024-06-29T15:58:32.385249+0000","last_peered":"2024-06-29T15:58:32.385249+0000","last_clean":"2024-06-29T15:58:32.385249+0000","last_became_active":"2024-06-29T15:58:31.402437+0000","last_became_peered":"2024-06-29T15:58:31.402437+0000","last_unstale":"2024-06-29T15:58:32.385249+0000","last_undegraded":"2024-06-29T15:58:32.385249+0000","last_fullsized":"2024-06-29T15:58:32.385249+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:27.691189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471493+0000","last_change":"2024-06-29T15:58:19.295640+0000","last_active":"2024-06-29T15:58:32.471493+0000","last_peered":"2024-06-29T15:58:32.471493+0000","last_clean":"2024-06-29T15:58:32.471493+0000","last_became_active":"2024-06-29T15:58:19.295239+0000","last_became_peered":"2024-06-29T15:58:19.295239+0000","last_unstale":"2024-06-29T15:58:32.471493+0000","last_undegraded":"2024-06-29T15:58:32.471493+0000","last_fullsized":"2024-06-29T15:58:32.471493+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:19.616206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471033+0000","last_change":"2024-06-29T15:58:18.294635+0000","last_active":"2024-06-29T15:58:32.471033+0000","last_peered":"2024-06-29T15:58:32.471033+0000","last_clean":"2024-06-29T15:58:32.471033+0000","last_became_active":"2024-06-29T15:58:18.294343+0000","last_became_peered":"2024-06-29T15:58:18.294343+0000","last_unstale":"2024-06-29T15:58:32.471033+0000","last_undegraded":"2024-06-29T15:58:32.471033+0000","last_fullsized":"2024-06-29T15:58:32.471033+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:44.439934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470505+0000","last_change":"2024-06-29T15:58:31.408169+0000","last_active":"2024-06-29T15:58:32.470505+0000","last_peered":"2024-06-29T15:58:32.470505+0000","last_clean":"2024-06-29T15:58:32.470505+0000","last_became_active":"2024-06-29T15:58:31.407860+0000","last_became_peered":"2024-06-29T15:58:31.407860+0000","last_unstale":"2024-06-29T15:58:32.470505+0000","last_undegraded":"2024-06-29T15:58:32.470505+0000","last_fullsized":"2024-06-29T15:58:32.470505+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:14.288552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471582+0000","last_change":"2024-06-29T15:58:19.306811+0000","last_active":"2024-06-29T15:58:32.471582+0000","last_peered":"2024-06-29T15:58:32.471582+0000","last_clean":"2024-06-29T15:58:32.471582+0000","last_became_active":"2024-06-29T15:58:19.306354+0000","last_became_peered":"2024-06-29T15:58:19.306354+0000","last_unstale":"2024-06-29T15:58:32.471582+0000","last_undegraded":"2024-06-29T15:58:32.471582+0000","last_fullsized":"2024-06-29T15:58:32.471582+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:55.684049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"232'3","reported_seq":25,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409089+0000","last_change":"2024-06-29T15:58:18.300841+0000","last_active":"2024-06-29T15:58:32.409089+0000","last_peered":"2024-06-29T15:58:32.409089+0000","last_clean":"2024-06-29T15:58:32.409089+0000","last_became_active":"2024-06-29T15:58:18.300525+0000","last_became_peered":"2024-06-29T15:58:18.300525+0000","last_unstale":"2024-06-29T15:58:32.409089+0000","last_undegraded":"2024-06-29T15:58:32.409089+0000","last_fullsized":"2024-06-29T15:58:32.409089+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:15.561954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470875+0000","last_change":"2024-06-29T15:58:31.412217+0000","last_active":"2024-06-29T15:58:32.470875+0000","last_peered":"2024-06-29T15:58:32.470875+0000","last_clean":"2024-06-29T15:58:32.470875+0000","last_became_active":"2024-06-29T15:58:31.411886+0000","last_became_peered":"2024-06-29T15:58:31.411886+0000","last_unstale":"2024-06-29T15:58:32.470875+0000","last_undegraded":"2024-06-29T15:58:32.470875+0000","last_fullsized":"2024-06-29T15:58:32.470875+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:07.236049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386255+0000","last_change":"2024-06-29T15:58:19.310218+0000","last_active":"2024-06-29T15:58:32.386255+0000","last_peered":"2024-06-29T15:58:32.386255+0000","last_clean":"2024-06-29T15:58:32.386255+0000","last_became_active":"2024-06-29T15:58:19.309910+0000","last_became_peered":"2024-06-29T15:58:19.309910+0000","last_unstale":"2024-06-29T15:58:32.386255+0000","last_undegraded":"2024-06-29T15:58:32.386255+0000","last_fullsized":"2024-06-29T15:58:32.386255+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:48.781692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"234'8","reported_seq":30,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408984+0000","last_change":"2024-06-29T15:58:18.296167+0000","last_active":"2024-06-29T15:58:32.408984+0000","last_peered":"2024-06-29T15:58:32.408984+0000","last_clean":"2024-06-29T15:58:32.408984+0000","last_became_active":"2024-06-29T15:58:18.294984+0000","last_became_peered":"2024-06-29T15:58:18.294984+0000","last_unstale":"2024-06-29T15:58:32.408984+0000","last_undegraded":"2024-06-29T15:58:32.408984+0000","last_fullsized":"2024-06-29T15:58:32.408984+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:07.733361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408826+0000","last_change":"2024-06-29T15:58:31.394062+0000","last_active":"2024-06-29T15:58:32.408826+0000","last_peered":"2024-06-29T15:58:32.408826+0000","last_clean":"2024-06-29T15:58:32.408826+0000","last_became_active":"2024-06-29T15:58:31.393756+0000","last_became_peered":"2024-06-29T15:58:31.393756+0000","last_unstale":"2024-06-29T15:58:32.408826+0000","last_undegraded":"2024-06-29T15:58:32.408826+0000","last_fullsized":"2024-06-29T15:58:32.408826+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:28.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385645+0000","last_change":"2024-06-29T15:58:19.315403+0000","last_active":"2024-06-29T15:58:32.385645+0000","last_peered":"2024-06-29T15:58:32.385645+0000","last_clean":"2024-06-29T15:58:32.385645+0000","last_became_active":"2024-06-29T15:58:19.314969+0000","last_became_peered":"2024-06-29T15:58:19.314969+0000","last_unstale":"2024-06-29T15:58:32.385645+0000","last_undegraded":"2024-06-29T15:58:32.385645+0000","last_fullsized":"2024-06-29T15:58:32.385645+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:42.361706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409315+0000","last_change":"2024-06-29T15:58:18.303307+0000","last_active":"2024-06-29T15:58:32.409315+0000","last_peered":"2024-06-29T15:58:32.409315+0000","last_clean":"2024-06-29T15:58:32.409315+0000","last_became_active":"2024-06-29T15:58:18.303009+0000","last_became_peered":"2024-06-29T15:58:18.303009+0000","last_unstale":"2024-06-29T15:58:32.409315+0000","last_undegraded":"2024-06-29T15:58:32.409315+0000","last_fullsized":"2024-06-29T15:58:32.409315+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:08.507147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385551+0000","last_change":"2024-06-29T15:58:31.410977+0000","last_active":"2024-06-29T15:58:32.385551+0000","last_peered":"2024-06-29T15:58:32.385551+0000","last_clean":"2024-06-29T15:58:32.385551+0000","last_became_active":"2024-06-29T15:58:31.410506+0000","last_became_peered":"2024-06-29T15:58:31.410506+0000","last_unstale":"2024-06-29T15:58:32.385551+0000","last_undegraded":"2024-06-29T15:58:32.385551+0000","last_fullsized":"2024-06-29T15:58:32.385551+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:59:43.051580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384172+0000","last_change":"2024-06-29T15:58:19.304892+0000","last_active":"2024-06-29T15:58:32.384172+0000","last_peered":"2024-06-29T15:58:32.384172+0000","last_clean":"2024-06-29T15:58:32.384172+0000","last_became_active":"2024-06-29T15:58:19.304368+0000","last_became_peered":"2024-06-29T15:58:19.304368+0000","last_unstale":"2024-06-29T15:58:32.384172+0000","last_undegraded":"2024-06-29T15:58:32.384172+0000","last_fullsized":"2024-06-29T15:58:32.384172+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:34.621330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471411+0000","last_change":"2024-06-29T15:58:18.290469+0000","last_active":"2024-06-29T15:58:32.471411+0000","last_peered":"2024-06-29T15:58:32.471411+0000","last_clean":"2024-06-29T15:58:32.471411+0000","last_became_active":"2024-06-29T15:58:18.290177+0000","last_became_peered":"2024-06-29T15:58:18.290177+0000","last_unstale":"2024-06-29T15:58:32.471411+0000","last_undegraded":"2024-06-29T15:58:32.471411+0000","last_fullsized":"2024-06-29T15:58:32.471411+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:51.797075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386382+0000","last_change":"2024-06-29T15:58:31.404104+0000","last_active":"2024-06-29T15:58:32.386382+0000","last_peered":"2024-06-29T15:58:32.386382+0000","last_clean":"2024-06-29T15:58:32.386382+0000","last_became_active":"2024-06-29T15:58:31.403759+0000","last_became_peered":"2024-06-29T15:58:31.403759+0000","last_unstale":"2024-06-29T15:58:32.386382+0000","last_undegraded":"2024-06-29T15:58:32.386382+0000","last_fullsized":"2024-06-29T15:58:32.386382+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:57:22.376570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382482+0000","last_change":"2024-06-29T15:58:19.304069+0000","last_active":"2024-06-29T15:58:32.382482+0000","last_peered":"2024-06-29T15:58:32.382482+0000","last_clean":"2024-06-29T15:58:32.382482+0000","last_became_active":"2024-06-29T15:58:19.303758+0000","last_became_peered":"2024-06-29T15:58:19.303758+0000","last_unstale":"2024-06-29T15:58:32.382482+0000","last_undegraded":"2024-06-29T15:58:32.382482+0000","last_fullsized":"2024-06-29T15:58:32.382482+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:51.378080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386174+0000","last_change":"2024-06-29T15:58:18.303515+0000","last_active":"2024-06-29T15:58:32.386174+0000","last_peered":"2024-06-29T15:58:32.386174+0000","last_clean":"2024-06-29T15:58:32.386174+0000","last_became_active":"2024-06-29T15:58:18.303008+0000","last_became_peered":"2024-06-29T15:58:18.303008+0000","last_unstale":"2024-06-29T15:58:32.386174+0000","last_undegraded":"2024-06-29T15:58:32.386174+0000","last_fullsized":"2024-06-29T15:58:32.386174+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:08.323310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471875+0000","last_change":"2024-06-29T15:58:31.406136+0000","last_active":"2024-06-29T15:58:32.471875+0000","last_peered":"2024-06-29T15:58:32.471875+0000","last_clean":"2024-06-29T15:58:32.471875+0000","last_became_active":"2024-06-29T15:58:31.405829+0000","last_became_peered":"2024-06-29T15:58:31.405829+0000","last_unstale":"2024-06-29T15:58:32.471875+0000","last_undegraded":"2024-06-29T15:58:32.471875+0000","last_fullsized":"2024-06-29T15:58:32.471875+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:05.000910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386012+0000","last_change":"2024-06-29T15:58:19.316203+0000","last_active":"2024-06-29T15:58:32.386012+0000","last_peered":"2024-06-29T15:58:32.386012+0000","last_clean":"2024-06-29T15:58:32.386012+0000","last_became_active":"2024-06-29T15:58:19.315777+0000","last_became_peered":"2024-06-29T15:58:19.315777+0000","last_unstale":"2024-06-29T15:58:32.386012+0000","last_undegraded":"2024-06-29T15:58:32.386012+0000","last_fullsized":"2024-06-29T15:58:32.386012+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:21.381169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"232'8","reported_seq":30,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470099+0000","last_change":"2024-06-29T15:58:18.302451+0000","last_active":"2024-06-29T15:58:32.470099+0000","last_peered":"2024-06-29T15:58:32.470099+0000","last_clean":"2024-06-29T15:58:32.470099+0000","last_became_active":"2024-06-29T15:58:18.302144+0000","last_became_peered":"2024-06-29T15:58:18.302144+0000","last_unstale":"2024-06-29T15:58:32.470099+0000","last_undegraded":"2024-06-29T15:58:32.470099+0000","last_fullsized":"2024-06-29T15:58:32.470099+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:15:27.736186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408461+0000","last_change":"2024-06-29T15:58:31.390906+0000","last_active":"2024-06-29T15:58:32.408461+0000","last_peered":"2024-06-29T15:58:32.408461+0000","last_clean":"2024-06-29T15:58:32.408461+0000","last_became_active":"2024-06-29T15:58:31.390187+0000","last_became_peered":"2024-06-29T15:58:31.390187+0000","last_unstale":"2024-06-29T15:58:32.408461+0000","last_undegraded":"2024-06-29T15:58:32.408461+0000","last_fullsized":"2024-06-29T15:58:32.408461+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:59:43.051580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383655+0000","last_change":"2024-06-29T15:58:19.297900+0000","last_active":"2024-06-29T15:58:32.383655+0000","last_peered":"2024-06-29T15:58:32.383655+0000","last_clean":"2024-06-29T15:58:32.383655+0000","last_became_active":"2024-06-29T15:58:19.297605+0000","last_became_peered":"2024-06-29T15:58:19.297605+0000","last_unstale":"2024-06-29T15:58:32.383655+0000","last_undegraded":"2024-06-29T15:58:32.383655+0000","last_fullsized":"2024-06-29T15:58:32.383655+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:25.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"232'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386465+0000","last_change":"2024-06-29T15:58:18.303063+0000","last_active":"2024-06-29T15:58:32.386465+0000","last_peered":"2024-06-29T15:58:32.386465+0000","last_clean":"2024-06-29T15:58:32.386465+0000","last_became_active":"2024-06-29T15:58:18.302403+0000","last_became_peered":"2024-06-29T15:58:18.302403+0000","last_unstale":"2024-06-29T15:58:32.386465+0000","last_undegraded":"2024-06-29T15:58:32.386465+0000","last_fullsized":"2024-06-29T15:58:32.386465+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:59:35.735600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382562+0000","last_change":"2024-06-29T15:58:31.402054+0000","last_active":"2024-06-29T15:58:32.382562+0000","last_peered":"2024-06-29T15:58:32.382562+0000","last_clean":"2024-06-29T15:58:32.382562+0000","last_became_active":"2024-06-29T15:58:31.401774+0000","last_became_peered":"2024-06-29T15:58:31.401774+0000","last_unstale":"2024-06-29T15:58:32.382562+0000","last_undegraded":"2024-06-29T15:58:32.382562+0000","last_fullsized":"2024-06-29T15:58:32.382562+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:33.464520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471788+0000","last_change":"2024-06-29T15:58:19.310412+0000","last_active":"2024-06-29T15:58:32.471788+0000","last_peered":"2024-06-29T15:58:32.471788+0000","last_clean":"2024-06-29T15:58:32.471788+0000","last_became_active":"2024-06-29T15:58:19.310109+0000","last_became_peered":"2024-06-29T15:58:19.310109+0000","last_unstale":"2024-06-29T15:58:32.471788+0000","last_undegraded":"2024-06-29T15:58:32.471788+0000","last_fullsized":"2024-06-29T15:58:32.471788+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:56.479901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471159+0000","last_change":"2024-06-29T15:58:18.296680+0000","last_active":"2024-06-29T15:58:32.471159+0000","last_peered":"2024-06-29T15:58:32.471159+0000","last_clean":"2024-06-29T15:58:32.471159+0000","last_became_active":"2024-06-29T15:58:18.296405+0000","last_became_peered":"2024-06-29T15:58:18.296405+0000","last_unstale":"2024-06-29T15:58:32.471159+0000","last_undegraded":"2024-06-29T15:58:32.471159+0000","last_fullsized":"2024-06-29T15:58:32.471159+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:02.671679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382818+0000","last_change":"2024-06-29T15:58:31.404958+0000","last_active":"2024-06-29T15:58:32.382818+0000","last_peered":"2024-06-29T15:58:32.382818+0000","last_clean":"2024-06-29T15:58:32.382818+0000","last_became_active":"2024-06-29T15:58:31.404664+0000","last_became_peered":"2024-06-29T15:58:31.404664+0000","last_unstale":"2024-06-29T15:58:32.382818+0000","last_undegraded":"2024-06-29T15:58:32.382818+0000","last_fullsized":"2024-06-29T15:58:32.382818+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:04.329393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383683+0000","last_change":"2024-06-29T15:58:19.308630+0000","last_active":"2024-06-29T15:58:32.383683+0000","last_peered":"2024-06-29T15:58:32.383683+0000","last_clean":"2024-06-29T15:58:32.383683+0000","last_became_active":"2024-06-29T15:58:19.308323+0000","last_became_peered":"2024-06-29T15:58:19.308323+0000","last_unstale":"2024-06-29T15:58:32.383683+0000","last_undegraded":"2024-06-29T15:58:32.383683+0000","last_fullsized":"2024-06-29T15:58:32.383683+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:14:39.500064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"232'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383767+0000","last_change":"2024-06-29T15:58:18.294949+0000","last_active":"2024-06-29T15:58:32.383767+0000","last_peered":"2024-06-29T15:58:32.383767+0000","last_clean":"2024-06-29T15:58:32.383767+0000","last_became_active":"2024-06-29T15:58:18.294015+0000","last_became_peered":"2024-06-29T15:58:18.294015+0000","last_unstale":"2024-06-29T15:58:32.383767+0000","last_undegraded":"2024-06-29T15:58:32.383767+0000","last_fullsized":"2024-06-29T15:58:32.383767+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:36.977744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382766+0000","last_change":"2024-06-29T15:58:31.390484+0000","last_active":"2024-06-29T15:58:32.382766+0000","last_peered":"2024-06-29T15:58:32.382766+0000","last_clean":"2024-06-29T15:58:32.382766+0000","last_became_active":"2024-06-29T15:58:31.390054+0000","last_became_peered":"2024-06-29T15:58:31.390054+0000","last_unstale":"2024-06-29T15:58:32.382766+0000","last_undegraded":"2024-06-29T15:58:32.382766+0000","last_fullsized":"2024-06-29T15:58:32.382766+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:54.445056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471189+0000","last_change":"2024-06-29T15:58:19.305909+0000","last_active":"2024-06-29T15:58:32.471189+0000","last_peered":"2024-06-29T15:58:32.471189+0000","last_clean":"2024-06-29T15:58:32.471189+0000","last_became_active":"2024-06-29T15:58:19.305594+0000","last_became_peered":"2024-06-29T15:58:19.305594+0000","last_unstale":"2024-06-29T15:58:32.471189+0000","last_undegraded":"2024-06-29T15:58:32.471189+0000","last_fullsized":"2024-06-29T15:58:32.471189+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:09.325434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470535+0000","last_change":"2024-06-29T15:58:18.306207+0000","last_active":"2024-06-29T15:58:32.470535+0000","last_peered":"2024-06-29T15:58:32.470535+0000","last_clean":"2024-06-29T15:58:32.470535+0000","last_became_active":"2024-06-29T15:58:18.305661+0000","last_became_peered":"2024-06-29T15:58:18.305661+0000","last_unstale":"2024-06-29T15:58:32.470535+0000","last_undegraded":"2024-06-29T15:58:32.470535+0000","last_fullsized":"2024-06-29T15:58:32.470535+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:44:47.769658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470826+0000","last_change":"2024-06-29T15:58:31.399014+0000","last_active":"2024-06-29T15:58:32.470826+0000","last_peered":"2024-06-29T15:58:32.470826+0000","last_clean":"2024-06-29T15:58:32.470826+0000","last_became_active":"2024-06-29T15:58:31.398631+0000","last_became_peered":"2024-06-29T15:58:31.398631+0000","last_unstale":"2024-06-29T15:58:32.470826+0000","last_undegraded":"2024-06-29T15:58:32.470826+0000","last_fullsized":"2024-06-29T15:58:32.470826+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:54.445056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470672+0000","last_change":"2024-06-29T15:58:19.307416+0000","last_active":"2024-06-29T15:58:32.470672+0000","last_peered":"2024-06-29T15:58:32.470672+0000","last_clean":"2024-06-29T15:58:32.470672+0000","last_became_active":"2024-06-29T15:58:19.307089+0000","last_became_peered":"2024-06-29T15:58:19.307089+0000","last_unstale":"2024-06-29T15:58:32.470672+0000","last_undegraded":"2024-06-29T15:58:32.470672+0000","last_fullsized":"2024-06-29T15:58:32.470672+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:54.355267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471513+0000","last_change":"2024-06-29T15:58:18.302497+0000","last_active":"2024-06-29T15:58:32.471513+0000","last_peered":"2024-06-29T15:58:32.471513+0000","last_clean":"2024-06-29T15:58:32.471513+0000","last_became_active":"2024-06-29T15:58:18.302200+0000","last_became_peered":"2024-06-29T15:58:18.302200+0000","last_unstale":"2024-06-29T15:58:32.471513+0000","last_undegraded":"2024-06-29T15:58:32.471513+0000","last_fullsized":"2024-06-29T15:58:32.471513+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:59.065708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385238+0000","last_change":"2024-06-29T15:58:31.403813+0000","last_active":"2024-06-29T15:58:32.385238+0000","last_peered":"2024-06-29T15:58:32.385238+0000","last_clean":"2024-06-29T15:58:32.385238+0000","last_became_active":"2024-06-29T15:58:31.403362+0000","last_became_peered":"2024-06-29T15:58:31.403362+0000","last_unstale":"2024-06-29T15:58:32.385238+0000","last_undegraded":"2024-06-29T15:58:32.385238+0000","last_fullsized":"2024-06-29T15:58:32.385238+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:12.151182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409273+0000","last_change":"2024-06-29T15:58:19.313662+0000","last_active":"2024-06-29T15:58:32.409273+0000","last_peered":"2024-06-29T15:58:32.409273+0000","last_clean":"2024-06-29T15:58:32.409273+0000","last_became_active":"2024-06-29T15:58:19.313340+0000","last_became_peered":"2024-06-29T15:58:19.313340+0000","last_unstale":"2024-06-29T15:58:32.409273+0000","last_undegraded":"2024-06-29T15:58:32.409273+0000","last_fullsized":"2024-06-29T15:58:32.409273+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:57.766748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470712+0000","last_change":"2024-06-29T15:58:18.306269+0000","last_active":"2024-06-29T15:58:32.470712+0000","last_peered":"2024-06-29T15:58:32.470712+0000","last_clean":"2024-06-29T15:58:32.470712+0000","last_became_active":"2024-06-29T15:58:18.305098+0000","last_became_peered":"2024-06-29T15:58:18.305098+0000","last_unstale":"2024-06-29T15:58:32.470712+0000","last_undegraded":"2024-06-29T15:58:32.470712+0000","last_fullsized":"2024-06-29T15:58:32.470712+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:50.375956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386134+0000","last_change":"2024-06-29T15:58:31.408521+0000","last_active":"2024-06-29T15:58:32.386134+0000","last_peered":"2024-06-29T15:58:32.386134+0000","last_clean":"2024-06-29T15:58:32.386134+0000","last_became_active":"2024-06-29T15:58:31.408199+0000","last_became_peered":"2024-06-29T15:58:31.408199+0000","last_unstale":"2024-06-29T15:58:32.386134+0000","last_undegraded":"2024-06-29T15:58:32.386134+0000","last_fullsized":"2024-06-29T15:58:32.386134+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:39.841958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470071+0000","last_change":"2024-06-29T15:58:19.308352+0000","last_active":"2024-06-29T15:58:32.470071+0000","last_peered":"2024-06-29T15:58:32.470071+0000","last_clean":"2024-06-29T15:58:32.470071+0000","last_became_active":"2024-06-29T15:58:19.308057+0000","last_became_peered":"2024-06-29T15:58:19.308057+0000","last_unstale":"2024-06-29T15:58:32.470071+0000","last_undegraded":"2024-06-29T15:58:32.470071+0000","last_fullsized":"2024-06-29T15:58:32.470071+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:25.510746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384172+0000","last_change":"2024-06-29T15:58:18.304832+0000","last_active":"2024-06-29T15:58:32.384172+0000","last_peered":"2024-06-29T15:58:32.384172+0000","last_clean":"2024-06-29T15:58:32.384172+0000","last_became_active":"2024-06-29T15:58:18.304230+0000","last_became_peered":"2024-06-29T15:58:18.304230+0000","last_unstale":"2024-06-29T15:58:32.384172+0000","last_undegraded":"2024-06-29T15:58:32.384172+0000","last_fullsized":"2024-06-29T15:58:32.384172+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:18.315156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471636+0000","last_change":"2024-06-29T15:58:31.401124+0000","last_active":"2024-06-29T15:58:32.471636+0000","last_peered":"2024-06-29T15:58:32.471636+0000","last_clean":"2024-06-29T15:58:32.471636+0000","last_became_active":"2024-06-29T15:58:31.400816+0000","last_became_peered":"2024-06-29T15:58:31.400816+0000","last_unstale":"2024-06-29T15:58:32.471636+0000","last_undegraded":"2024-06-29T15:58:32.471636+0000","last_fullsized":"2024-06-29T15:58:32.471636+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:50.511371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470394+0000","last_change":"2024-06-29T15:58:19.306776+0000","last_active":"2024-06-29T15:58:32.470394+0000","last_peered":"2024-06-29T15:58:32.470394+0000","last_clean":"2024-06-29T15:58:32.470394+0000","last_became_active":"2024-06-29T15:58:19.306385+0000","last_became_peered":"2024-06-29T15:58:19.306385+0000","last_unstale":"2024-06-29T15:58:32.470394+0000","last_undegraded":"2024-06-29T15:58:32.470394+0000","last_fullsized":"2024-06-29T15:58:32.470394+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:25.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385230+0000","last_change":"2024-06-29T15:58:18.297661+0000","last_active":"2024-06-29T15:58:32.385230+0000","last_peered":"2024-06-29T15:58:32.385230+0000","last_clean":"2024-06-29T15:58:32.385230+0000","last_became_active":"2024-06-29T15:58:18.297272+0000","last_became_peered":"2024-06-29T15:58:18.297272+0000","last_unstale":"2024-06-29T15:58:32.385230+0000","last_undegraded":"2024-06-29T15:58:32.385230+0000","last_fullsized":"2024-06-29T15:58:32.385230+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:17:56.872132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470256+0000","last_change":"2024-06-29T15:58:31.409558+0000","last_active":"2024-06-29T15:58:32.470256+0000","last_peered":"2024-06-29T15:58:32.470256+0000","last_clean":"2024-06-29T15:58:32.470256+0000","last_became_active":"2024-06-29T15:58:31.409241+0000","last_became_peered":"2024-06-29T15:58:31.409241+0000","last_unstale":"2024-06-29T15:58:32.470256+0000","last_undegraded":"2024-06-29T15:58:32.470256+0000","last_fullsized":"2024-06-29T15:58:32.470256+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:33.464520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386572+0000","last_change":"2024-06-29T15:58:19.311825+0000","last_active":"2024-06-29T15:58:32.386572+0000","last_peered":"2024-06-29T15:58:32.386572+0000","last_clean":"2024-06-29T15:58:32.386572+0000","last_became_active":"2024-06-29T15:58:19.311471+0000","last_became_peered":"2024-06-29T15:58:19.311471+0000","last_unstale":"2024-06-29T15:58:32.386572+0000","last_undegraded":"2024-06-29T15:58:32.386572+0000","last_fullsized":"2024-06-29T15:58:32.386572+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:52.246043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383482+0000","last_change":"2024-06-29T15:58:18.302543+0000","last_active":"2024-06-29T15:58:32.383482+0000","last_peered":"2024-06-29T15:58:32.383482+0000","last_clean":"2024-06-29T15:58:32.383482+0000","last_became_active":"2024-06-29T15:58:18.302237+0000","last_became_peered":"2024-06-29T15:58:18.302237+0000","last_unstale":"2024-06-29T15:58:32.383482+0000","last_undegraded":"2024-06-29T15:58:32.383482+0000","last_fullsized":"2024-06-29T15:58:32.383482+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:19:38.790806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385987+0000","last_change":"2024-06-29T15:58:31.403820+0000","last_active":"2024-06-29T15:58:32.385987+0000","last_peered":"2024-06-29T15:58:32.385987+0000","last_clean":"2024-06-29T15:58:32.385987+0000","last_became_active":"2024-06-29T15:58:31.403478+0000","last_became_peered":"2024-06-29T15:58:31.403478+0000","last_unstale":"2024-06-29T15:58:32.385987+0000","last_undegraded":"2024-06-29T15:58:32.385987+0000","last_fullsized":"2024-06-29T15:58:32.385987+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:28.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409226+0000","last_change":"2024-06-29T15:58:19.312366+0000","last_active":"2024-06-29T15:58:32.409226+0000","last_peered":"2024-06-29T15:58:32.409226+0000","last_clean":"2024-06-29T15:58:32.409226+0000","last_became_active":"2024-06-29T15:58:19.309633+0000","last_became_peered":"2024-06-29T15:58:19.309633+0000","last_unstale":"2024-06-29T15:58:32.409226+0000","last_undegraded":"2024-06-29T15:58:32.409226+0000","last_fullsized":"2024-06-29T15:58:32.409226+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:11.746638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"233'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.404022+0000","last_change":"2024-06-29T15:58:18.303588+0000","last_active":"2024-06-29T15:58:32.404022+0000","last_peered":"2024-06-29T15:58:32.404022+0000","last_clean":"2024-06-29T15:58:32.404022+0000","last_became_active":"2024-06-29T15:58:18.303274+0000","last_became_peered":"2024-06-29T15:58:18.303274+0000","last_unstale":"2024-06-29T15:58:32.404022+0000","last_undegraded":"2024-06-29T15:58:32.404022+0000","last_fullsized":"2024-06-29T15:58:32.404022+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:04.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408660+0000","last_change":"2024-06-29T15:58:31.394954+0000","last_active":"2024-06-29T15:58:32.408660+0000","last_peered":"2024-06-29T15:58:32.408660+0000","last_clean":"2024-06-29T15:58:32.408660+0000","last_became_active":"2024-06-29T15:58:31.394623+0000","last_became_peered":"2024-06-29T15:58:31.394623+0000","last_unstale":"2024-06-29T15:58:32.408660+0000","last_undegraded":"2024-06-29T15:58:32.408660+0000","last_fullsized":"2024-06-29T15:58:32.408660+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:00.629657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470433+0000","last_change":"2024-06-29T15:58:19.307220+0000","last_active":"2024-06-29T15:58:32.470433+0000","last_peered":"2024-06-29T15:58:32.470433+0000","last_clean":"2024-06-29T15:58:32.470433+0000","last_became_active":"2024-06-29T15:58:19.306905+0000","last_became_peered":"2024-06-29T15:58:19.306905+0000","last_unstale":"2024-06-29T15:58:32.470433+0000","last_undegraded":"2024-06-29T15:58:32.470433+0000","last_fullsized":"2024-06-29T15:58:32.470433+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:10.851803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383919+0000","last_change":"2024-06-29T15:58:18.303622+0000","last_active":"2024-06-29T15:58:32.383919+0000","last_peered":"2024-06-29T15:58:32.383919+0000","last_clean":"2024-06-29T15:58:32.383919+0000","last_became_active":"2024-06-29T15:58:18.303319+0000","last_became_peered":"2024-06-29T15:58:18.303319+0000","last_unstale":"2024-06-29T15:58:32.383919+0000","last_undegraded":"2024-06-29T15:58:32.383919+0000","last_fullsized":"2024-06-29T15:58:32.383919+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:13.370498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403437+0000","last_change":"2024-06-29T15:58:31.407351+0000","last_active":"2024-06-29T15:58:32.403437+0000","last_peered":"2024-06-29T15:58:32.403437+0000","last_clean":"2024-06-29T15:58:32.403437+0000","last_became_active":"2024-06-29T15:58:31.407006+0000","last_became_peered":"2024-06-29T15:58:31.407006+0000","last_unstale":"2024-06-29T15:58:32.403437+0000","last_undegraded":"2024-06-29T15:58:32.403437+0000","last_fullsized":"2024-06-29T15:58:32.403437+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:31.699556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471045+0000","last_change":"2024-06-29T15:58:19.294460+0000","last_active":"2024-06-29T15:58:32.471045+0000","last_peered":"2024-06-29T15:58:32.471045+0000","last_clean":"2024-06-29T15:58:32.471045+0000","last_became_active":"2024-06-29T15:58:19.294111+0000","last_became_peered":"2024-06-29T15:58:19.294111+0000","last_unstale":"2024-06-29T15:58:32.471045+0000","last_undegraded":"2024-06-29T15:58:32.471045+0000","last_fullsized":"2024-06-29T15:58:32.471045+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:37.984851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470636+0000","last_change":"2024-06-29T15:58:18.305967+0000","last_active":"2024-06-29T15:58:32.470636+0000","last_peered":"2024-06-29T15:58:32.470636+0000","last_clean":"2024-06-29T15:58:32.470636+0000","last_became_active":"2024-06-29T15:58:18.304464+0000","last_became_peered":"2024-06-29T15:58:18.304464+0000","last_unstale":"2024-06-29T15:58:32.470636+0000","last_undegraded":"2024-06-29T15:58:32.470636+0000","last_fullsized":"2024-06-29T15:58:32.470636+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:37.753590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403631+0000","last_change":"2024-06-29T15:58:31.407838+0000","last_active":"2024-06-29T15:58:32.403631+0000","last_peered":"2024-06-29T15:58:32.403631+0000","last_clean":"2024-06-29T15:58:32.403631+0000","last_became_active":"2024-06-29T15:58:31.406987+0000","last_became_peered":"2024-06-29T15:58:31.406987+0000","last_unstale":"2024-06-29T15:58:32.403631+0000","last_undegraded":"2024-06-29T15:58:32.403631+0000","last_fullsized":"2024-06-29T15:58:32.403631+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:22.935153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385791+0000","last_change":"2024-06-29T15:58:19.311597+0000","last_active":"2024-06-29T15:58:32.385791+0000","last_peered":"2024-06-29T15:58:32.385791+0000","last_clean":"2024-06-29T15:58:32.385791+0000","last_became_active":"2024-06-29T15:58:19.311093+0000","last_became_peered":"2024-06-29T15:58:19.311093+0000","last_unstale":"2024-06-29T15:58:32.385791+0000","last_undegraded":"2024-06-29T15:58:32.385791+0000","last_fullsized":"2024-06-29T15:58:32.385791+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:11.746638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386303+0000","last_change":"2024-06-29T15:58:18.302661+0000","last_active":"2024-06-29T15:58:32.386303+0000","last_peered":"2024-06-29T15:58:32.386303+0000","last_clean":"2024-06-29T15:58:32.386303+0000","last_became_active":"2024-06-29T15:58:18.302330+0000","last_became_peered":"2024-06-29T15:58:18.302330+0000","last_unstale":"2024-06-29T15:58:32.386303+0000","last_undegraded":"2024-06-29T15:58:32.386303+0000","last_fullsized":"2024-06-29T15:58:32.386303+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:36.982727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403701+0000","last_change":"2024-06-29T15:58:31.408494+0000","last_active":"2024-06-29T15:58:32.403701+0000","last_peered":"2024-06-29T15:58:32.403701+0000","last_clean":"2024-06-29T15:58:32.403701+0000","last_became_active":"2024-06-29T15:58:31.407493+0000","last_became_peered":"2024-06-29T15:58:31.407493+0000","last_unstale":"2024-06-29T15:58:32.403701+0000","last_undegraded":"2024-06-29T15:58:32.403701+0000","last_fullsized":"2024-06-29T15:58:32.403701+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:39:05.574092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470734+0000","last_change":"2024-06-29T15:58:19.300548+0000","last_active":"2024-06-29T15:58:32.470734+0000","last_peered":"2024-06-29T15:58:32.470734+0000","last_clean":"2024-06-29T15:58:32.470734+0000","last_became_active":"2024-06-29T15:58:19.300152+0000","last_became_peered":"2024-06-29T15:58:19.300152+0000","last_unstale":"2024-06-29T15:58:32.470734+0000","last_undegraded":"2024-06-29T15:58:32.470734+0000","last_fullsized":"2024-06-29T15:58:32.470734+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:09.509271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383285+0000","last_change":"2024-06-29T15:58:18.294669+0000","last_active":"2024-06-29T15:58:32.383285+0000","last_peered":"2024-06-29T15:58:32.383285+0000","last_clean":"2024-06-29T15:58:32.383285+0000","last_became_active":"2024-06-29T15:58:18.294353+0000","last_became_peered":"2024-06-29T15:58:18.294353+0000","last_unstale":"2024-06-29T15:58:32.383285+0000","last_undegraded":"2024-06-29T15:58:32.383285+0000","last_fullsized":"2024-06-29T15:58:32.383285+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:00:41.729079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384531+0000","last_change":"2024-06-29T15:58:31.410700+0000","last_active":"2024-06-29T15:58:32.384531+0000","last_peered":"2024-06-29T15:58:32.384531+0000","last_clean":"2024-06-29T15:58:32.384531+0000","last_became_active":"2024-06-29T15:58:31.410393+0000","last_became_peered":"2024-06-29T15:58:31.410393+0000","last_unstale":"2024-06-29T15:58:32.384531+0000","last_undegraded":"2024-06-29T15:58:32.384531+0000","last_fullsized":"2024-06-29T15:58:32.384531+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:39:17.993335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470529+0000","last_change":"2024-06-29T15:58:19.303904+0000","last_active":"2024-06-29T15:58:32.470529+0000","last_peered":"2024-06-29T15:58:32.470529+0000","last_clean":"2024-06-29T15:58:32.470529+0000","last_became_active":"2024-06-29T15:58:19.303572+0000","last_became_peered":"2024-06-29T15:58:19.303572+0000","last_unstale":"2024-06-29T15:58:32.470529+0000","last_undegraded":"2024-06-29T15:58:32.470529+0000","last_fullsized":"2024-06-29T15:58:32.470529+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:49:49.141794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470763+0000","last_change":"2024-06-29T15:58:18.298256+0000","last_active":"2024-06-29T15:58:32.470763+0000","last_peered":"2024-06-29T15:58:32.470763+0000","last_clean":"2024-06-29T15:58:32.470763+0000","last_became_active":"2024-06-29T15:58:18.297928+0000","last_became_peered":"2024-06-29T15:58:18.297928+0000","last_unstale":"2024-06-29T15:58:32.470763+0000","last_undegraded":"2024-06-29T15:58:32.470763+0000","last_fullsized":"2024-06-29T15:58:32.470763+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:28.560916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471197+0000","last_change":"2024-06-29T15:58:31.408208+0000","last_active":"2024-06-29T15:58:32.471197+0000","last_peered":"2024-06-29T15:58:32.471197+0000","last_clean":"2024-06-29T15:58:32.471197+0000","last_became_active":"2024-06-29T15:58:31.407838+0000","last_became_peered":"2024-06-29T15:58:31.407838+0000","last_unstale":"2024-06-29T15:58:32.471197+0000","last_undegraded":"2024-06-29T15:58:32.471197+0000","last_fullsized":"2024-06-29T15:58:32.471197+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:23.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408477+0000","last_change":"2024-06-29T15:58:19.305160+0000","last_active":"2024-06-29T15:58:32.408477+0000","last_peered":"2024-06-29T15:58:32.408477+0000","last_clean":"2024-06-29T15:58:32.408477+0000","last_became_active":"2024-06-29T15:58:19.304650+0000","last_became_peered":"2024-06-29T15:58:19.304650+0000","last_unstale":"2024-06-29T15:58:32.408477+0000","last_undegraded":"2024-06-29T15:58:32.408477+0000","last_fullsized":"2024-06-29T15:58:32.408477+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:33.637292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.472004+0000","last_change":"2024-06-29T15:58:18.302761+0000","last_active":"2024-06-29T15:58:32.472004+0000","last_peered":"2024-06-29T15:58:32.472004+0000","last_clean":"2024-06-29T15:58:32.472004+0000","last_became_active":"2024-06-29T15:58:18.302490+0000","last_became_peered":"2024-06-29T15:58:18.302490+0000","last_unstale":"2024-06-29T15:58:32.472004+0000","last_undegraded":"2024-06-29T15:58:32.472004+0000","last_fullsized":"2024-06-29T15:58:32.472004+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:59:29.966106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385018+0000","last_change":"2024-06-29T15:58:31.405025+0000","last_active":"2024-06-29T15:58:32.385018+0000","last_peered":"2024-06-29T15:58:32.385018+0000","last_clean":"2024-06-29T15:58:32.385018+0000","last_became_active":"2024-06-29T15:58:31.402188+0000","last_became_peered":"2024-06-29T15:58:31.402188+0000","last_unstale":"2024-06-29T15:58:32.385018+0000","last_undegraded":"2024-06-29T15:58:32.385018+0000","last_fullsized":"2024-06-29T15:58:32.385018+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:04.882550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403913+0000","last_change":"2024-06-29T15:58:19.292511+0000","last_active":"2024-06-29T15:58:32.403913+0000","last_peered":"2024-06-29T15:58:32.403913+0000","last_clean":"2024-06-29T15:58:32.403913+0000","last_became_active":"2024-06-29T15:58:19.291923+0000","last_became_peered":"2024-06-29T15:58:19.291923+0000","last_unstale":"2024-06-29T15:58:32.403913+0000","last_undegraded":"2024-06-29T15:58:32.403913+0000","last_fullsized":"2024-06-29T15:58:32.403913+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:18.095828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"232'4","reported_seq":26,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471901+0000","last_change":"2024-06-29T15:58:18.303249+0000","last_active":"2024-06-29T15:58:32.471901+0000","last_peered":"2024-06-29T15:58:32.471901+0000","last_clean":"2024-06-29T15:58:32.471901+0000","last_became_active":"2024-06-29T15:58:18.302916+0000","last_became_peered":"2024-06-29T15:58:18.302916+0000","last_unstale":"2024-06-29T15:58:32.471901+0000","last_undegraded":"2024-06-29T15:58:32.471901+0000","last_fullsized":"2024-06-29T15:58:32.471901+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:15.821379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"219'233","reported_seq":1133,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383527+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:58:32.383527+0000","last_peered":"2024-06-29T15:58:32.383527+0000","last_clean":"2024-06-29T15:58:32.383527+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:58:32.383527+0000","last_undegraded":"2024-06-29T15:58:32.383527+0000","last_fullsized":"2024-06-29T15:58:32.383527+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575998,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52732,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319108,"statfs":{"total":96636764160,"available":96582766592,"internally_reserved":0,"allocated":3588096,"data_stored":2063604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575999,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60472,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311368,"statfs":{"total":96636764160,"available":96574840832,"internally_reserved":0,"allocated":2990080,"data_stored":1466517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575997,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46584,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325256,"statfs":{"total":96636764160,"available":96589062144,"internally_reserved":0,"allocated":2985984,"data_stored":1473096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575998,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60508,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311332,"statfs":{"total":96636764160,"available":96574803968,"internally_reserved":0,"allocated":3026944,"data_stored":1477711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575998,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48004,"kb_used_data":2936,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323836,"statfs":{"total":96636764160,"available":96587608064,"internally_reserved":0,"allocated":3006464,"data_stored":1475872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575998,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47996,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323844,"statfs":{"total":96636764160,"available":96587616256,"internally_reserved":0,"allocated":2990080,"data_stored":1473186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575997,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60508,"kb_used_data":2952,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311332,"statfs":{"total":96636764160,"available":96574803968,"internally_reserved":0,"allocated":3022848,"data_stored":1476851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575998,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61052,"kb_used_data":3500,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310788,"statfs":{"total":96636764160,"available":96574246912,"internally_reserved":0,"allocated":3584000,"data_stored":2063498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:58:35.449 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:58:35.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.594+0000 7fd1533b2640 1 -- 172.21.15.81:0/1828201646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd138003a50 msgr2=0x7fd138003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.594+0000 7fd1533b2640 1 --2- 172.21.15.81:0/1828201646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd138003a50 0x7fd138003ec0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0b2c10 tx=0x7fd138005830 comp rx=0 tx=0).stop 2024-06-29T15:58:35.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.594+0000 7fd1533b2640 1 -- 172.21.15.81:0/1828201646 shutdown_connections 2024-06-29T15:58:35.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.594+0000 7fd1533b2640 1 --2- 172.21.15.81:0/1828201646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd138003a50 0x7fd138003ec0 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.594+0000 7fd1533b2640 1 -- 172.21.15.81:0/1828201646 >> 172.21.15.81:0/1828201646 conn(0x7fd14c0d1960 msgr2=0x7fd14c0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:35.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.594+0000 7fd1533b2640 1 -- 172.21.15.81:0/1828201646 shutdown_connections 2024-06-29T15:58:35.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.594+0000 7fd1533b2640 1 -- 172.21.15.81:0/1828201646 wait complete. 2024-06-29T15:58:35.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.596+0000 7fd1533b2640 1 Processor -- start 2024-06-29T15:58:35.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.596+0000 7fd1533b2640 1 -- start start 2024-06-29T15:58:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.597+0000 7fd1533b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd138003a50 0x7fd14c1df0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd1533b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd14c1e1280 0x7fd14c0af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd1533b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c0b0e70 0x7fd14c0b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd1533b2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c0b4b40 con 0x7fd14c0b0e70 2024-06-29T15:58:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd1533b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c0b49c0 con 0x7fd138003a50 2024-06-29T15:58:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd1533b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd14c0ce4d0 con 0x7fd14c1e1280 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15194f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c0b0e70 0x7fd14c0b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15194f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c0b0e70 0x7fd14c0b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33256/0 (socket says 172.21.15.81:33256) 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15194f640 1 -- 172.21.15.81:0/2592585839 learned_addr learned my addr 172.21.15.81:0/2592585839 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15114e640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd138003a50 0x7fd14c1df0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15094d640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd14c1e1280 0x7fd14c0af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15194f640 1 -- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd14c1e1280 msgr2=0x7fd14c0af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15194f640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd14c1e1280 0x7fd14c0af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15194f640 1 -- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd138003a50 msgr2=0x7fd14c1df0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15194f640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd138003a50 0x7fd14c1df0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15194f640 1 -- 172.21.15.81:0/2592585839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14c0c4aa0 con 0x7fd14c0b0e70 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd15194f640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c0b0e70 0x7fd14c0b1280 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fd144000f80 tx=0x7fd14400eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd141ffb640 1 -- 172.21.15.81:0/2592585839 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14400ca10 con 0x7fd14c0b0e70 2024-06-29T15:58:35.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd141ffb640 1 -- 172.21.15.81:0/2592585839 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd144017070 con 0x7fd14c0b0e70 2024-06-29T15:58:35.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd141ffb640 1 -- 172.21.15.81:0/2592585839 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144011460 con 0x7fd14c0b0e70 2024-06-29T15:58:35.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.598+0000 7fd1533b2640 1 -- 172.21.15.81:0/2592585839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd14c0afdd0 con 0x7fd14c0b0e70 2024-06-29T15:58:35.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.599+0000 7fd1533b2640 1 -- 172.21.15.81:0/2592585839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd14c0b0390 con 0x7fd14c0b0e70 2024-06-29T15:58:35.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.600+0000 7fd1533b2640 1 -- 172.21.15.81:0/2592585839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd14c0cea30 con 0x7fd14c0b0e70 2024-06-29T15:58:35.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.600+0000 7fd141ffb640 1 -- 172.21.15.81:0/2592585839 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fd14401c460 con 0x7fd14c0b0e70 2024-06-29T15:58:35.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.600+0000 7fd141ffb640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd130076510 0x7fd130078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.601+0000 7fd141ffb640 1 -- 172.21.15.81:0/2592585839 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7fd144098390 con 0x7fd14c0b0e70 2024-06-29T15:58:35.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.601+0000 7fd15114e640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd130076510 0x7fd130078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.601+0000 7fd15114e640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd130076510 0x7fd130078a20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd148032000 tx=0x7fd1480011d0 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.605+0000 7fd141ffb640 1 -- 172.21.15.81:0/2592585839 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd14c0c4aa0 con 0x7fd14c0b0e70 2024-06-29T15:58:35.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.780+0000 7fd1533b2640 1 -- 172.21.15.81:0/2592585839 --> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd14c0afb60 con 0x7fd130076510 2024-06-29T15:58:35.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.789+0000 7fd141ffb640 1 -- 172.21.15.81:0/2592585839 <== mgr.9155 v2:172.21.15.167:6832/3661607921 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+491471 (secure 0 0 0) 0x7fd14c0afb60 con 0x7fd130076510 2024-06-29T15:58:35.795 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:35.799 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 -- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd130076510 msgr2=0x7fd130078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd130076510 0x7fd130078a20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd148032000 tx=0x7fd1480011d0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 -- 172.21.15.81:0/2592585839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c0b0e70 msgr2=0x7fd14c0b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c0b0e70 0x7fd14c0b1280 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fd144000f80 tx=0x7fd14400eb60 comp rx=0 tx=0).stop 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 -- 172.21.15.81:0/2592585839 shutdown_connections 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fd130076510 0x7fd130078a20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd14c0b0e70 0x7fd14c0b1280 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd14c1e1280 0x7fd14c0af1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 --2- 172.21.15.81:0/2592585839 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd138003a50 0x7fd14c1df0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 -- 172.21.15.81:0/2592585839 >> 172.21.15.81:0/2592585839 conn(0x7fd14c0d1960 msgr2=0x7fd14c1db390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 -- 172.21.15.81:0/2592585839 shutdown_connections 2024-06-29T15:58:35.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.793+0000 7fd1533b2640 1 -- 172.21.15.81:0/2592585839 wait complete. 2024-06-29T15:58:35.805 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-06-29T15:58:35.348879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596431,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":469,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":353,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":437856,"kb_used_data":24604,"kb_used_omap":12,"kb_used_meta":413107,"kb_avail":754536864,"statfs":{"total":773094113280,"available":772645748736,"internally_reserved":0,"allocated":25194496,"data_stored":12970335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":423022130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3397,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.986580"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403282+0000","last_change":"2024-06-29T15:58:31.394424+0000","last_active":"2024-06-29T15:58:32.403282+0000","last_peered":"2024-06-29T15:58:32.403282+0000","last_clean":"2024-06-29T15:58:32.403282+0000","last_became_active":"2024-06-29T15:58:31.394079+0000","last_became_peered":"2024-06-29T15:58:31.394079+0000","last_unstale":"2024-06-29T15:58:32.403282+0000","last_undegraded":"2024-06-29T15:58:32.403282+0000","last_fullsized":"2024-06-29T15:58:32.403282+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:59:48.821074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471869+0000","last_change":"2024-06-29T15:58:30.387552+0000","last_active":"2024-06-29T15:58:32.471869+0000","last_peered":"2024-06-29T15:58:32.471869+0000","last_clean":"2024-06-29T15:58:32.471869+0000","last_became_active":"2024-06-29T15:58:30.387263+0000","last_became_peered":"2024-06-29T15:58:30.387263+0000","last_unstale":"2024-06-29T15:58:32.471869+0000","last_undegraded":"2024-06-29T15:58:32.471869+0000","last_fullsized":"2024-06-29T15:58:32.471869+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:17:24.545993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408848+0000","last_change":"2024-06-29T15:58:19.306321+0000","last_active":"2024-06-29T15:58:32.408848+0000","last_peered":"2024-06-29T15:58:32.408848+0000","last_clean":"2024-06-29T15:58:32.408848+0000","last_became_active":"2024-06-29T15:58:19.306018+0000","last_became_peered":"2024-06-29T15:58:19.306018+0000","last_unstale":"2024-06-29T15:58:32.408848+0000","last_undegraded":"2024-06-29T15:58:32.408848+0000","last_fullsized":"2024-06-29T15:58:32.408848+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:42.361706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382505+0000","last_change":"2024-06-29T15:58:31.407272+0000","last_active":"2024-06-29T15:58:32.382505+0000","last_peered":"2024-06-29T15:58:32.382505+0000","last_clean":"2024-06-29T15:58:32.382505+0000","last_became_active":"2024-06-29T15:58:31.406988+0000","last_became_peered":"2024-06-29T15:58:31.406988+0000","last_unstale":"2024-06-29T15:58:32.382505+0000","last_undegraded":"2024-06-29T15:58:32.382505+0000","last_fullsized":"2024-06-29T15:58:32.382505+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:33.966242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383620+0000","last_change":"2024-06-29T15:58:30.388423+0000","last_active":"2024-06-29T15:58:32.383620+0000","last_peered":"2024-06-29T15:58:32.383620+0000","last_clean":"2024-06-29T15:58:32.383620+0000","last_became_active":"2024-06-29T15:58:30.388144+0000","last_became_peered":"2024-06-29T15:58:30.388144+0000","last_unstale":"2024-06-29T15:58:32.383620+0000","last_undegraded":"2024-06-29T15:58:32.383620+0000","last_fullsized":"2024-06-29T15:58:32.383620+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:30.660014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409034+0000","last_change":"2024-06-29T15:58:19.312428+0000","last_active":"2024-06-29T15:58:32.409034+0000","last_peered":"2024-06-29T15:58:32.409034+0000","last_clean":"2024-06-29T15:58:32.409034+0000","last_became_active":"2024-06-29T15:58:19.312036+0000","last_became_peered":"2024-06-29T15:58:19.312036+0000","last_unstale":"2024-06-29T15:58:32.409034+0000","last_undegraded":"2024-06-29T15:58:32.409034+0000","last_fullsized":"2024-06-29T15:58:32.409034+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:04.132174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382681+0000","last_change":"2024-06-29T15:58:31.402000+0000","last_active":"2024-06-29T15:58:32.382681+0000","last_peered":"2024-06-29T15:58:32.382681+0000","last_clean":"2024-06-29T15:58:32.382681+0000","last_became_active":"2024-06-29T15:58:31.401674+0000","last_became_peered":"2024-06-29T15:58:31.401674+0000","last_unstale":"2024-06-29T15:58:32.382681+0000","last_undegraded":"2024-06-29T15:58:32.382681+0000","last_fullsized":"2024-06-29T15:58:32.382681+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:16.215524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470426+0000","last_change":"2024-06-29T15:58:30.388832+0000","last_active":"2024-06-29T15:58:32.470426+0000","last_peered":"2024-06-29T15:58:32.470426+0000","last_clean":"2024-06-29T15:58:32.470426+0000","last_became_active":"2024-06-29T15:58:30.388524+0000","last_became_peered":"2024-06-29T15:58:30.388524+0000","last_unstale":"2024-06-29T15:58:32.470426+0000","last_undegraded":"2024-06-29T15:58:32.470426+0000","last_fullsized":"2024-06-29T15:58:32.470426+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:44.713766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471031+0000","last_change":"2024-06-29T15:58:19.315147+0000","last_active":"2024-06-29T15:58:32.471031+0000","last_peered":"2024-06-29T15:58:32.471031+0000","last_clean":"2024-06-29T15:58:32.471031+0000","last_became_active":"2024-06-29T15:58:19.314838+0000","last_became_peered":"2024-06-29T15:58:19.314838+0000","last_unstale":"2024-06-29T15:58:32.471031+0000","last_undegraded":"2024-06-29T15:58:32.471031+0000","last_fullsized":"2024-06-29T15:58:32.471031+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:33:39.763837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383401+0000","last_change":"2024-06-29T15:58:31.390355+0000","last_active":"2024-06-29T15:58:32.383401+0000","last_peered":"2024-06-29T15:58:32.383401+0000","last_clean":"2024-06-29T15:58:32.383401+0000","last_became_active":"2024-06-29T15:58:31.390036+0000","last_became_peered":"2024-06-29T15:58:31.390036+0000","last_unstale":"2024-06-29T15:58:32.383401+0000","last_undegraded":"2024-06-29T15:58:32.383401+0000","last_fullsized":"2024-06-29T15:58:32.383401+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:01.353128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470930+0000","last_change":"2024-06-29T15:58:30.380541+0000","last_active":"2024-06-29T15:58:32.470930+0000","last_peered":"2024-06-29T15:58:32.470930+0000","last_clean":"2024-06-29T15:58:32.470930+0000","last_became_active":"2024-06-29T15:58:30.380226+0000","last_became_peered":"2024-06-29T15:58:30.380226+0000","last_unstale":"2024-06-29T15:58:32.470930+0000","last_undegraded":"2024-06-29T15:58:32.470930+0000","last_fullsized":"2024-06-29T15:58:32.470930+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:32.457644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383220+0000","last_change":"2024-06-29T15:58:19.297774+0000","last_active":"2024-06-29T15:58:32.383220+0000","last_peered":"2024-06-29T15:58:32.383220+0000","last_clean":"2024-06-29T15:58:32.383220+0000","last_became_active":"2024-06-29T15:58:19.297463+0000","last_became_peered":"2024-06-29T15:58:19.297463+0000","last_unstale":"2024-06-29T15:58:32.383220+0000","last_undegraded":"2024-06-29T15:58:32.383220+0000","last_fullsized":"2024-06-29T15:58:32.383220+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:38.755714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403906+0000","last_change":"2024-06-29T15:58:31.400649+0000","last_active":"2024-06-29T15:58:32.403906+0000","last_peered":"2024-06-29T15:58:32.403906+0000","last_clean":"2024-06-29T15:58:32.403906+0000","last_became_active":"2024-06-29T15:58:31.400335+0000","last_became_peered":"2024-06-29T15:58:31.400335+0000","last_unstale":"2024-06-29T15:58:32.403906+0000","last_undegraded":"2024-06-29T15:58:32.403906+0000","last_fullsized":"2024-06-29T15:58:32.403906+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:15:28.726085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470916+0000","last_change":"2024-06-29T15:58:30.390704+0000","last_active":"2024-06-29T15:58:32.470916+0000","last_peered":"2024-06-29T15:58:32.470916+0000","last_clean":"2024-06-29T15:58:32.470916+0000","last_became_active":"2024-06-29T15:58:30.390373+0000","last_became_peered":"2024-06-29T15:58:30.390373+0000","last_unstale":"2024-06-29T15:58:32.470916+0000","last_undegraded":"2024-06-29T15:58:32.470916+0000","last_fullsized":"2024-06-29T15:58:32.470916+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:18.439877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403458+0000","last_change":"2024-06-29T15:58:19.313121+0000","last_active":"2024-06-29T15:58:32.403458+0000","last_peered":"2024-06-29T15:58:32.403458+0000","last_clean":"2024-06-29T15:58:32.403458+0000","last_became_active":"2024-06-29T15:58:19.312783+0000","last_became_peered":"2024-06-29T15:58:19.312783+0000","last_unstale":"2024-06-29T15:58:32.403458+0000","last_undegraded":"2024-06-29T15:58:32.403458+0000","last_fullsized":"2024-06-29T15:58:32.403458+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:00:42.731203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403597+0000","last_change":"2024-06-29T15:58:31.399640+0000","last_active":"2024-06-29T15:58:32.403597+0000","last_peered":"2024-06-29T15:58:32.403597+0000","last_clean":"2024-06-29T15:58:32.403597+0000","last_became_active":"2024-06-29T15:58:31.399297+0000","last_became_peered":"2024-06-29T15:58:31.399297+0000","last_unstale":"2024-06-29T15:58:32.403597+0000","last_undegraded":"2024-06-29T15:58:32.403597+0000","last_fullsized":"2024-06-29T15:58:32.403597+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:50.068201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385919+0000","last_change":"2024-06-29T15:58:30.388387+0000","last_active":"2024-06-29T15:58:32.385919+0000","last_peered":"2024-06-29T15:58:32.385919+0000","last_clean":"2024-06-29T15:58:32.385919+0000","last_became_active":"2024-06-29T15:58:30.388001+0000","last_became_peered":"2024-06-29T15:58:30.388001+0000","last_unstale":"2024-06-29T15:58:32.385919+0000","last_undegraded":"2024-06-29T15:58:32.385919+0000","last_fullsized":"2024-06-29T15:58:32.385919+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:07.535813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383187+0000","last_change":"2024-06-29T15:58:19.302569+0000","last_active":"2024-06-29T15:58:32.383187+0000","last_peered":"2024-06-29T15:58:32.383187+0000","last_clean":"2024-06-29T15:58:32.383187+0000","last_became_active":"2024-06-29T15:58:19.302261+0000","last_became_peered":"2024-06-29T15:58:19.302261+0000","last_unstale":"2024-06-29T15:58:32.383187+0000","last_undegraded":"2024-06-29T15:58:32.383187+0000","last_fullsized":"2024-06-29T15:58:32.383187+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:44:48.771783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471441+0000","last_change":"2024-06-29T15:58:31.403127+0000","last_active":"2024-06-29T15:58:32.471441+0000","last_peered":"2024-06-29T15:58:32.471441+0000","last_clean":"2024-06-29T15:58:32.471441+0000","last_became_active":"2024-06-29T15:58:31.402843+0000","last_became_peered":"2024-06-29T15:58:31.402843+0000","last_unstale":"2024-06-29T15:58:32.471441+0000","last_undegraded":"2024-06-29T15:58:32.471441+0000","last_fullsized":"2024-06-29T15:58:32.471441+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:07.614128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"237'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.405357+0000","last_change":"2024-06-29T15:58:30.388669+0000","last_active":"2024-06-29T15:58:32.405357+0000","last_peered":"2024-06-29T15:58:32.405357+0000","last_clean":"2024-06-29T15:58:32.405357+0000","last_became_active":"2024-06-29T15:58:30.388310+0000","last_became_peered":"2024-06-29T15:58:30.388310+0000","last_unstale":"2024-06-29T15:58:32.405357+0000","last_undegraded":"2024-06-29T15:58:32.405357+0000","last_fullsized":"2024-06-29T15:58:32.405357+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:36.967839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385652+0000","last_change":"2024-06-29T15:58:19.314579+0000","last_active":"2024-06-29T15:58:32.385652+0000","last_peered":"2024-06-29T15:58:32.385652+0000","last_clean":"2024-06-29T15:58:32.385652+0000","last_became_active":"2024-06-29T15:58:19.314282+0000","last_became_peered":"2024-06-29T15:58:19.314282+0000","last_unstale":"2024-06-29T15:58:32.385652+0000","last_undegraded":"2024-06-29T15:58:32.385652+0000","last_fullsized":"2024-06-29T15:58:32.385652+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:10.919759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382961+0000","last_change":"2024-06-29T15:58:31.402794+0000","last_active":"2024-06-29T15:58:32.382961+0000","last_peered":"2024-06-29T15:58:32.382961+0000","last_clean":"2024-06-29T15:58:32.382961+0000","last_became_active":"2024-06-29T15:58:31.402013+0000","last_became_peered":"2024-06-29T15:58:31.402013+0000","last_unstale":"2024-06-29T15:58:32.382961+0000","last_undegraded":"2024-06-29T15:58:32.382961+0000","last_fullsized":"2024-06-29T15:58:32.382961+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:30.523422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383240+0000","last_change":"2024-06-29T15:58:30.386798+0000","last_active":"2024-06-29T15:58:32.383240+0000","last_peered":"2024-06-29T15:58:32.383240+0000","last_clean":"2024-06-29T15:58:32.383240+0000","last_became_active":"2024-06-29T15:58:30.386468+0000","last_became_peered":"2024-06-29T15:58:30.386468+0000","last_unstale":"2024-06-29T15:58:32.383240+0000","last_undegraded":"2024-06-29T15:58:32.383240+0000","last_fullsized":"2024-06-29T15:58:32.383240+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:20.401909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470384+0000","last_change":"2024-06-29T15:58:19.303095+0000","last_active":"2024-06-29T15:58:32.470384+0000","last_peered":"2024-06-29T15:58:32.470384+0000","last_clean":"2024-06-29T15:58:32.470384+0000","last_became_active":"2024-06-29T15:58:19.302658+0000","last_became_peered":"2024-06-29T15:58:19.302658+0000","last_unstale":"2024-06-29T15:58:32.470384+0000","last_undegraded":"2024-06-29T15:58:32.470384+0000","last_fullsized":"2024-06-29T15:58:32.470384+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:03:22.896379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470283+0000","last_change":"2024-06-29T15:58:31.396045+0000","last_active":"2024-06-29T15:58:32.470283+0000","last_peered":"2024-06-29T15:58:32.470283+0000","last_clean":"2024-06-29T15:58:32.470283+0000","last_became_active":"2024-06-29T15:58:31.393369+0000","last_became_peered":"2024-06-29T15:58:31.393369+0000","last_unstale":"2024-06-29T15:58:32.470283+0000","last_undegraded":"2024-06-29T15:58:32.470283+0000","last_fullsized":"2024-06-29T15:58:32.470283+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:43:08.948872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470246+0000","last_change":"2024-06-29T15:58:30.381617+0000","last_active":"2024-06-29T15:58:32.470246+0000","last_peered":"2024-06-29T15:58:32.470246+0000","last_clean":"2024-06-29T15:58:32.470246+0000","last_became_active":"2024-06-29T15:58:30.381233+0000","last_became_peered":"2024-06-29T15:58:30.381233+0000","last_unstale":"2024-06-29T15:58:32.470246+0000","last_undegraded":"2024-06-29T15:58:32.470246+0000","last_fullsized":"2024-06-29T15:58:32.470246+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:28:29.486781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384701+0000","last_change":"2024-06-29T15:58:19.310261+0000","last_active":"2024-06-29T15:58:32.384701+0000","last_peered":"2024-06-29T15:58:32.384701+0000","last_clean":"2024-06-29T15:58:32.384701+0000","last_became_active":"2024-06-29T15:58:19.309742+0000","last_became_peered":"2024-06-29T15:58:19.309742+0000","last_unstale":"2024-06-29T15:58:32.384701+0000","last_undegraded":"2024-06-29T15:58:32.384701+0000","last_fullsized":"2024-06-29T15:58:32.384701+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:55:23.521010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471766+0000","last_change":"2024-06-29T15:58:31.403840+0000","last_active":"2024-06-29T15:58:32.471766+0000","last_peered":"2024-06-29T15:58:32.471766+0000","last_clean":"2024-06-29T15:58:32.471766+0000","last_became_active":"2024-06-29T15:58:31.403530+0000","last_became_peered":"2024-06-29T15:58:31.403530+0000","last_unstale":"2024-06-29T15:58:32.471766+0000","last_undegraded":"2024-06-29T15:58:32.471766+0000","last_fullsized":"2024-06-29T15:58:32.471766+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:11:52.504053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383953+0000","last_change":"2024-06-29T15:58:30.387115+0000","last_active":"2024-06-29T15:58:32.383953+0000","last_peered":"2024-06-29T15:58:32.383953+0000","last_clean":"2024-06-29T15:58:32.383953+0000","last_became_active":"2024-06-29T15:58:30.386778+0000","last_became_peered":"2024-06-29T15:58:30.386778+0000","last_unstale":"2024-06-29T15:58:32.383953+0000","last_undegraded":"2024-06-29T15:58:32.383953+0000","last_fullsized":"2024-06-29T15:58:32.383953+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:27.674515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386450+0000","last_change":"2024-06-29T15:58:19.315434+0000","last_active":"2024-06-29T15:58:32.386450+0000","last_peered":"2024-06-29T15:58:32.386450+0000","last_clean":"2024-06-29T15:58:32.386450+0000","last_became_active":"2024-06-29T15:58:19.315065+0000","last_became_peered":"2024-06-29T15:58:19.315065+0000","last_unstale":"2024-06-29T15:58:32.386450+0000","last_undegraded":"2024-06-29T15:58:32.386450+0000","last_fullsized":"2024-06-29T15:58:32.386450+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:21.882892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409228+0000","last_change":"2024-06-29T15:58:31.391129+0000","last_active":"2024-06-29T15:58:32.409228+0000","last_peered":"2024-06-29T15:58:32.409228+0000","last_clean":"2024-06-29T15:58:32.409228+0000","last_became_active":"2024-06-29T15:58:31.390831+0000","last_became_peered":"2024-06-29T15:58:31.390831+0000","last_unstale":"2024-06-29T15:58:32.409228+0000","last_undegraded":"2024-06-29T15:58:32.409228+0000","last_fullsized":"2024-06-29T15:58:32.409228+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:57:22.376570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470457+0000","last_change":"2024-06-29T15:58:30.384784+0000","last_active":"2024-06-29T15:58:32.470457+0000","last_peered":"2024-06-29T15:58:32.470457+0000","last_clean":"2024-06-29T15:58:32.470457+0000","last_became_active":"2024-06-29T15:58:30.384431+0000","last_became_peered":"2024-06-29T15:58:30.384431+0000","last_unstale":"2024-06-29T15:58:32.470457+0000","last_undegraded":"2024-06-29T15:58:32.470457+0000","last_fullsized":"2024-06-29T15:58:32.470457+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:58.103230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470619+0000","last_change":"2024-06-29T15:58:19.312563+0000","last_active":"2024-06-29T15:58:32.470619+0000","last_peered":"2024-06-29T15:58:32.470619+0000","last_clean":"2024-06-29T15:58:32.470619+0000","last_became_active":"2024-06-29T15:58:19.311751+0000","last_became_peered":"2024-06-29T15:58:19.311751+0000","last_unstale":"2024-06-29T15:58:32.470619+0000","last_undegraded":"2024-06-29T15:58:32.470619+0000","last_fullsized":"2024-06-29T15:58:32.470619+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:19.583540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409058+0000","last_change":"2024-06-29T15:58:31.400729+0000","last_active":"2024-06-29T15:58:32.409058+0000","last_peered":"2024-06-29T15:58:32.409058+0000","last_clean":"2024-06-29T15:58:32.409058+0000","last_became_active":"2024-06-29T15:58:31.400437+0000","last_became_peered":"2024-06-29T15:58:31.400437+0000","last_unstale":"2024-06-29T15:58:32.409058+0000","last_undegraded":"2024-06-29T15:58:32.409058+0000","last_fullsized":"2024-06-29T15:58:32.409058+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:39.841958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403390+0000","last_change":"2024-06-29T15:58:30.380368+0000","last_active":"2024-06-29T15:58:32.403390+0000","last_peered":"2024-06-29T15:58:32.403390+0000","last_clean":"2024-06-29T15:58:32.403390+0000","last_became_active":"2024-06-29T15:58:30.380035+0000","last_became_peered":"2024-06-29T15:58:30.380035+0000","last_unstale":"2024-06-29T15:58:32.403390+0000","last_undegraded":"2024-06-29T15:58:32.403390+0000","last_fullsized":"2024-06-29T15:58:32.403390+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:52:05.431742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471041+0000","last_change":"2024-06-29T15:58:19.303967+0000","last_active":"2024-06-29T15:58:32.471041+0000","last_peered":"2024-06-29T15:58:32.471041+0000","last_clean":"2024-06-29T15:58:32.471041+0000","last_became_active":"2024-06-29T15:58:19.303620+0000","last_became_peered":"2024-06-29T15:58:19.303620+0000","last_unstale":"2024-06-29T15:58:32.471041+0000","last_undegraded":"2024-06-29T15:58:32.471041+0000","last_fullsized":"2024-06-29T15:58:32.471041+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:17:57.874256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403756+0000","last_change":"2024-06-29T15:58:31.399661+0000","last_active":"2024-06-29T15:58:32.403756+0000","last_peered":"2024-06-29T15:58:32.403756+0000","last_clean":"2024-06-29T15:58:32.403756+0000","last_became_active":"2024-06-29T15:58:31.399309+0000","last_became_peered":"2024-06-29T15:58:31.399309+0000","last_unstale":"2024-06-29T15:58:32.403756+0000","last_undegraded":"2024-06-29T15:58:32.403756+0000","last_fullsized":"2024-06-29T15:58:32.403756+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:26:06.039374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383117+0000","last_change":"2024-06-29T15:58:30.389595+0000","last_active":"2024-06-29T15:58:32.383117+0000","last_peered":"2024-06-29T15:58:32.383117+0000","last_clean":"2024-06-29T15:58:32.383117+0000","last_became_active":"2024-06-29T15:58:30.389243+0000","last_became_peered":"2024-06-29T15:58:30.389243+0000","last_unstale":"2024-06-29T15:58:32.383117+0000","last_undegraded":"2024-06-29T15:58:32.383117+0000","last_fullsized":"2024-06-29T15:58:32.383117+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:21.928277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386211+0000","last_change":"2024-06-29T15:58:19.317397+0000","last_active":"2024-06-29T15:58:32.386211+0000","last_peered":"2024-06-29T15:58:32.386211+0000","last_clean":"2024-06-29T15:58:32.386211+0000","last_became_active":"2024-06-29T15:58:19.317110+0000","last_became_peered":"2024-06-29T15:58:19.317110+0000","last_unstale":"2024-06-29T15:58:32.386211+0000","last_undegraded":"2024-06-29T15:58:32.386211+0000","last_fullsized":"2024-06-29T15:58:32.386211+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:33.637292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383004+0000","last_change":"2024-06-29T15:58:31.398339+0000","last_active":"2024-06-29T15:58:32.383004+0000","last_peered":"2024-06-29T15:58:32.383004+0000","last_clean":"2024-06-29T15:58:32.383004+0000","last_became_active":"2024-06-29T15:58:31.398019+0000","last_became_peered":"2024-06-29T15:58:31.398019+0000","last_unstale":"2024-06-29T15:58:32.383004+0000","last_undegraded":"2024-06-29T15:58:32.383004+0000","last_fullsized":"2024-06-29T15:58:32.383004+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:14.288552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"236'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383906+0000","last_change":"2024-06-29T15:58:30.386307+0000","last_active":"2024-06-29T15:58:32.383906+0000","last_peered":"2024-06-29T15:58:32.383906+0000","last_clean":"2024-06-29T15:58:32.383906+0000","last_became_active":"2024-06-29T15:58:30.386016+0000","last_became_peered":"2024-06-29T15:58:30.386016+0000","last_unstale":"2024-06-29T15:58:32.383906+0000","last_undegraded":"2024-06-29T15:58:32.383906+0000","last_fullsized":"2024-06-29T15:58:32.383906+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:27:54.276566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408629+0000","last_change":"2024-06-29T15:58:19.308273+0000","last_active":"2024-06-29T15:58:32.408629+0000","last_peered":"2024-06-29T15:58:32.408629+0000","last_clean":"2024-06-29T15:58:32.408629+0000","last_became_active":"2024-06-29T15:58:19.307956+0000","last_became_peered":"2024-06-29T15:58:19.307956+0000","last_unstale":"2024-06-29T15:58:32.408629+0000","last_undegraded":"2024-06-29T15:58:32.408629+0000","last_fullsized":"2024-06-29T15:58:32.408629+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:48.781692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384631+0000","last_change":"2024-06-29T15:58:31.409700+0000","last_active":"2024-06-29T15:58:32.384631+0000","last_peered":"2024-06-29T15:58:32.384631+0000","last_clean":"2024-06-29T15:58:32.384631+0000","last_became_active":"2024-06-29T15:58:31.409352+0000","last_became_peered":"2024-06-29T15:58:31.409352+0000","last_unstale":"2024-06-29T15:58:32.384631+0000","last_undegraded":"2024-06-29T15:58:32.384631+0000","last_fullsized":"2024-06-29T15:58:32.384631+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:45:23.560000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408975+0000","last_change":"2024-06-29T15:58:30.377175+0000","last_active":"2024-06-29T15:58:32.408975+0000","last_peered":"2024-06-29T15:58:32.408975+0000","last_clean":"2024-06-29T15:58:32.408975+0000","last_became_active":"2024-06-29T15:58:30.376867+0000","last_became_peered":"2024-06-29T15:58:30.376867+0000","last_unstale":"2024-06-29T15:58:32.408975+0000","last_undegraded":"2024-06-29T15:58:32.408975+0000","last_fullsized":"2024-06-29T15:58:32.408975+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:07.494825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384041+0000","last_change":"2024-06-29T15:58:19.300026+0000","last_active":"2024-06-29T15:58:32.384041+0000","last_peered":"2024-06-29T15:58:32.384041+0000","last_clean":"2024-06-29T15:58:32.384041+0000","last_became_active":"2024-06-29T15:58:19.298956+0000","last_became_peered":"2024-06-29T15:58:19.298956+0000","last_unstale":"2024-06-29T15:58:32.384041+0000","last_undegraded":"2024-06-29T15:58:32.384041+0000","last_fullsized":"2024-06-29T15:58:32.384041+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:10.419613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383728+0000","last_change":"2024-06-29T15:58:31.399206+0000","last_active":"2024-06-29T15:58:32.383728+0000","last_peered":"2024-06-29T15:58:32.383728+0000","last_clean":"2024-06-29T15:58:32.383728+0000","last_became_active":"2024-06-29T15:58:31.398935+0000","last_became_peered":"2024-06-29T15:58:31.398935+0000","last_unstale":"2024-06-29T15:58:32.383728+0000","last_undegraded":"2024-06-29T15:58:32.383728+0000","last_fullsized":"2024-06-29T15:58:32.383728+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:39:08.542689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"236'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471114+0000","last_change":"2024-06-29T15:58:30.389011+0000","last_active":"2024-06-29T15:58:32.471114+0000","last_peered":"2024-06-29T15:58:32.471114+0000","last_clean":"2024-06-29T15:58:32.471114+0000","last_became_active":"2024-06-29T15:58:30.388677+0000","last_became_peered":"2024-06-29T15:58:30.388677+0000","last_unstale":"2024-06-29T15:58:32.471114+0000","last_undegraded":"2024-06-29T15:58:32.471114+0000","last_fullsized":"2024-06-29T15:58:32.471114+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:39:27.538117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471685+0000","last_change":"2024-06-29T15:58:19.308922+0000","last_active":"2024-06-29T15:58:32.471685+0000","last_peered":"2024-06-29T15:58:32.471685+0000","last_clean":"2024-06-29T15:58:32.471685+0000","last_became_active":"2024-06-29T15:58:19.308502+0000","last_became_peered":"2024-06-29T15:58:19.308502+0000","last_unstale":"2024-06-29T15:58:32.471685+0000","last_undegraded":"2024-06-29T15:58:32.471685+0000","last_fullsized":"2024-06-29T15:58:32.471685+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:14.372622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470577+0000","last_change":"2024-06-29T15:58:31.396486+0000","last_active":"2024-06-29T15:58:32.470577+0000","last_peered":"2024-06-29T15:58:32.470577+0000","last_clean":"2024-06-29T15:58:32.470577+0000","last_became_active":"2024-06-29T15:58:31.396169+0000","last_became_peered":"2024-06-29T15:58:31.396169+0000","last_unstale":"2024-06-29T15:58:32.470577+0000","last_undegraded":"2024-06-29T15:58:32.470577+0000","last_fullsized":"2024-06-29T15:58:32.470577+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:23:30.206980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"236'8","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408571+0000","last_change":"2024-06-29T15:58:30.378533+0000","last_active":"2024-06-29T15:58:32.408571+0000","last_peered":"2024-06-29T15:58:32.408571+0000","last_clean":"2024-06-29T15:58:32.408571+0000","last_became_active":"2024-06-29T15:58:30.378214+0000","last_became_peered":"2024-06-29T15:58:30.378214+0000","last_unstale":"2024-06-29T15:58:32.408571+0000","last_undegraded":"2024-06-29T15:58:32.408571+0000","last_fullsized":"2024-06-29T15:58:32.408571+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:06:18.807055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471254+0000","last_change":"2024-06-29T15:58:19.309759+0000","last_active":"2024-06-29T15:58:32.471254+0000","last_peered":"2024-06-29T15:58:32.471254+0000","last_clean":"2024-06-29T15:58:32.471254+0000","last_became_active":"2024-06-29T15:58:19.309439+0000","last_became_peered":"2024-06-29T15:58:19.309439+0000","last_unstale":"2024-06-29T15:58:32.471254+0000","last_undegraded":"2024-06-29T15:58:32.471254+0000","last_fullsized":"2024-06-29T15:58:32.471254+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:54.499719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471585+0000","last_change":"2024-06-29T15:58:31.398394+0000","last_active":"2024-06-29T15:58:32.471585+0000","last_peered":"2024-06-29T15:58:32.471585+0000","last_clean":"2024-06-29T15:58:32.471585+0000","last_became_active":"2024-06-29T15:58:31.397578+0000","last_became_peered":"2024-06-29T15:58:31.397578+0000","last_unstale":"2024-06-29T15:58:32.471585+0000","last_undegraded":"2024-06-29T15:58:32.471585+0000","last_fullsized":"2024-06-29T15:58:32.471585+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:54:59.756820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384235+0000","last_change":"2024-06-29T15:58:30.385343+0000","last_active":"2024-06-29T15:58:32.384235+0000","last_peered":"2024-06-29T15:58:32.384235+0000","last_clean":"2024-06-29T15:58:32.384235+0000","last_became_active":"2024-06-29T15:58:30.384981+0000","last_became_peered":"2024-06-29T15:58:30.384981+0000","last_unstale":"2024-06-29T15:58:32.384235+0000","last_undegraded":"2024-06-29T15:58:32.384235+0000","last_fullsized":"2024-06-29T15:58:32.384235+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:18.439877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409134+0000","last_change":"2024-06-29T15:58:19.306249+0000","last_active":"2024-06-29T15:58:32.409134+0000","last_peered":"2024-06-29T15:58:32.409134+0000","last_clean":"2024-06-29T15:58:32.409134+0000","last_became_active":"2024-06-29T15:58:19.305935+0000","last_became_peered":"2024-06-29T15:58:19.305935+0000","last_unstale":"2024-06-29T15:58:32.409134+0000","last_undegraded":"2024-06-29T15:58:32.409134+0000","last_fullsized":"2024-06-29T15:58:32.409134+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:52.246043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409243+0000","last_change":"2024-06-29T15:58:31.405635+0000","last_active":"2024-06-29T15:58:32.409243+0000","last_peered":"2024-06-29T15:58:32.409243+0000","last_clean":"2024-06-29T15:58:32.409243+0000","last_became_active":"2024-06-29T15:58:31.405306+0000","last_became_peered":"2024-06-29T15:58:31.405306+0000","last_unstale":"2024-06-29T15:58:32.409243+0000","last_undegraded":"2024-06-29T15:58:32.409243+0000","last_fullsized":"2024-06-29T15:58:32.409243+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:32:40.005109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470532+0000","last_change":"2024-06-29T15:58:30.385427+0000","last_active":"2024-06-29T15:58:32.470532+0000","last_peered":"2024-06-29T15:58:32.470532+0000","last_clean":"2024-06-29T15:58:32.470532+0000","last_became_active":"2024-06-29T15:58:30.385055+0000","last_became_peered":"2024-06-29T15:58:30.385055+0000","last_unstale":"2024-06-29T15:58:32.470532+0000","last_undegraded":"2024-06-29T15:58:32.470532+0000","last_fullsized":"2024-06-29T15:58:32.470532+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:32.959366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470963+0000","last_change":"2024-06-29T15:58:19.313204+0000","last_active":"2024-06-29T15:58:32.470963+0000","last_peered":"2024-06-29T15:58:32.470963+0000","last_clean":"2024-06-29T15:58:32.470963+0000","last_became_active":"2024-06-29T15:58:19.312752+0000","last_became_peered":"2024-06-29T15:58:19.312752+0000","last_unstale":"2024-06-29T15:58:32.470963+0000","last_undegraded":"2024-06-29T15:58:32.470963+0000","last_fullsized":"2024-06-29T15:58:32.470963+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:16.564078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471480+0000","last_change":"2024-06-29T15:58:31.406977+0000","last_active":"2024-06-29T15:58:32.471480+0000","last_peered":"2024-06-29T15:58:32.471480+0000","last_clean":"2024-06-29T15:58:32.471480+0000","last_became_active":"2024-06-29T15:58:31.406714+0000","last_became_peered":"2024-06-29T15:58:31.406714+0000","last_unstale":"2024-06-29T15:58:32.471480+0000","last_undegraded":"2024-06-29T15:58:32.471480+0000","last_fullsized":"2024-06-29T15:58:32.471480+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:07:27.548352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470573+0000","last_change":"2024-06-29T15:58:30.385679+0000","last_active":"2024-06-29T15:58:32.470573+0000","last_peered":"2024-06-29T15:58:32.470573+0000","last_clean":"2024-06-29T15:58:32.470573+0000","last_became_active":"2024-06-29T15:58:30.385296+0000","last_became_peered":"2024-06-29T15:58:30.385296+0000","last_unstale":"2024-06-29T15:58:32.470573+0000","last_undegraded":"2024-06-29T15:58:32.470573+0000","last_fullsized":"2024-06-29T15:58:32.470573+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:15:27.719209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471008+0000","last_change":"2024-06-29T15:58:19.303230+0000","last_active":"2024-06-29T15:58:32.471008+0000","last_peered":"2024-06-29T15:58:32.471008+0000","last_clean":"2024-06-29T15:58:32.471008+0000","last_became_active":"2024-06-29T15:58:19.302904+0000","last_became_peered":"2024-06-29T15:58:19.302904+0000","last_unstale":"2024-06-29T15:58:32.471008+0000","last_undegraded":"2024-06-29T15:58:32.471008+0000","last_fullsized":"2024-06-29T15:58:32.471008+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:30:16.162326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471539+0000","last_change":"2024-06-29T15:58:31.398026+0000","last_active":"2024-06-29T15:58:32.471539+0000","last_peered":"2024-06-29T15:58:32.471539+0000","last_clean":"2024-06-29T15:58:32.471539+0000","last_became_active":"2024-06-29T15:58:31.397728+0000","last_became_peered":"2024-06-29T15:58:31.397728+0000","last_unstale":"2024-06-29T15:58:32.471539+0000","last_undegraded":"2024-06-29T15:58:32.471539+0000","last_fullsized":"2024-06-29T15:58:32.471539+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:31.280890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"236'6","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470884+0000","last_change":"2024-06-29T15:58:30.384767+0000","last_active":"2024-06-29T15:58:32.470884+0000","last_peered":"2024-06-29T15:58:32.470884+0000","last_clean":"2024-06-29T15:58:32.470884+0000","last_became_active":"2024-06-29T15:58:30.384344+0000","last_became_peered":"2024-06-29T15:58:30.384344+0000","last_unstale":"2024-06-29T15:58:32.470884+0000","last_undegraded":"2024-06-29T15:58:32.470884+0000","last_fullsized":"2024-06-29T15:58:32.470884+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:27:54.276566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470659+0000","last_change":"2024-06-29T15:58:19.302093+0000","last_active":"2024-06-29T15:58:32.470659+0000","last_peered":"2024-06-29T15:58:32.470659+0000","last_clean":"2024-06-29T15:58:32.470659+0000","last_became_active":"2024-06-29T15:58:19.301677+0000","last_became_peered":"2024-06-29T15:58:19.301677+0000","last_unstale":"2024-06-29T15:58:32.470659+0000","last_undegraded":"2024-06-29T15:58:32.470659+0000","last_fullsized":"2024-06-29T15:58:32.470659+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:08.735485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382986+0000","last_change":"2024-06-29T15:58:31.405773+0000","last_active":"2024-06-29T15:58:32.382986+0000","last_peered":"2024-06-29T15:58:32.382986+0000","last_clean":"2024-06-29T15:58:32.382986+0000","last_became_active":"2024-06-29T15:58:31.405406+0000","last_became_peered":"2024-06-29T15:58:31.405406+0000","last_unstale":"2024-06-29T15:58:32.382986+0000","last_undegraded":"2024-06-29T15:58:32.382986+0000","last_fullsized":"2024-06-29T15:58:32.382986+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:43:08.948872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"236'3","reported_seq":17,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408949+0000","last_change":"2024-06-29T15:58:30.389211+0000","last_active":"2024-06-29T15:58:32.408949+0000","last_peered":"2024-06-29T15:58:32.408949+0000","last_clean":"2024-06-29T15:58:32.408949+0000","last_became_active":"2024-06-29T15:58:30.388877+0000","last_became_peered":"2024-06-29T15:58:30.388877+0000","last_unstale":"2024-06-29T15:58:32.408949+0000","last_undegraded":"2024-06-29T15:58:32.408949+0000","last_fullsized":"2024-06-29T15:58:32.408949+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:35:11.381957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385591+0000","last_change":"2024-06-29T15:58:19.307655+0000","last_active":"2024-06-29T15:58:32.385591+0000","last_peered":"2024-06-29T15:58:32.385591+0000","last_clean":"2024-06-29T15:58:32.385591+0000","last_became_active":"2024-06-29T15:58:19.307369+0000","last_became_peered":"2024-06-29T15:58:19.307369+0000","last_unstale":"2024-06-29T15:58:32.385591+0000","last_undegraded":"2024-06-29T15:58:32.385591+0000","last_fullsized":"2024-06-29T15:58:32.385591+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:04.132174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471271+0000","last_change":"2024-06-29T15:58:31.399030+0000","last_active":"2024-06-29T15:58:32.471271+0000","last_peered":"2024-06-29T15:58:32.471271+0000","last_clean":"2024-06-29T15:58:32.471271+0000","last_became_active":"2024-06-29T15:58:31.398440+0000","last_became_peered":"2024-06-29T15:58:31.398440+0000","last_unstale":"2024-06-29T15:58:32.471271+0000","last_undegraded":"2024-06-29T15:58:32.471271+0000","last_fullsized":"2024-06-29T15:58:32.471271+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:07:49.345352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409145+0000","last_change":"2024-06-29T15:58:30.389217+0000","last_active":"2024-06-29T15:58:32.409145+0000","last_peered":"2024-06-29T15:58:32.409145+0000","last_clean":"2024-06-29T15:58:32.409145+0000","last_became_active":"2024-06-29T15:58:30.388891+0000","last_became_peered":"2024-06-29T15:58:30.388891+0000","last_unstale":"2024-06-29T15:58:32.409145+0000","last_undegraded":"2024-06-29T15:58:32.409145+0000","last_fullsized":"2024-06-29T15:58:32.409145+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:57:00.261000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470277+0000","last_change":"2024-06-29T15:58:19.308486+0000","last_active":"2024-06-29T15:58:32.470277+0000","last_peered":"2024-06-29T15:58:32.470277+0000","last_clean":"2024-06-29T15:58:32.470277+0000","last_became_active":"2024-06-29T15:58:19.308158+0000","last_became_peered":"2024-06-29T15:58:19.308158+0000","last_unstale":"2024-06-29T15:58:32.470277+0000","last_undegraded":"2024-06-29T15:58:32.470277+0000","last_fullsized":"2024-06-29T15:58:32.470277+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:38.755714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386243+0000","last_change":"2024-06-29T15:58:31.405690+0000","last_active":"2024-06-29T15:58:32.386243+0000","last_peered":"2024-06-29T15:58:32.386243+0000","last_clean":"2024-06-29T15:58:32.386243+0000","last_became_active":"2024-06-29T15:58:31.404937+0000","last_became_peered":"2024-06-29T15:58:31.404937+0000","last_unstale":"2024-06-29T15:58:32.386243+0000","last_undegraded":"2024-06-29T15:58:32.386243+0000","last_fullsized":"2024-06-29T15:58:32.386243+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:34.598819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471775+0000","last_change":"2024-06-29T15:58:30.390358+0000","last_active":"2024-06-29T15:58:32.471775+0000","last_peered":"2024-06-29T15:58:32.471775+0000","last_clean":"2024-06-29T15:58:32.471775+0000","last_became_active":"2024-06-29T15:58:30.390049+0000","last_became_peered":"2024-06-29T15:58:30.390049+0000","last_unstale":"2024-06-29T15:58:32.471775+0000","last_undegraded":"2024-06-29T15:58:32.471775+0000","last_fullsized":"2024-06-29T15:58:32.471775+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:39:16.986459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408636+0000","last_change":"2024-06-29T15:58:19.312754+0000","last_active":"2024-06-29T15:58:32.408636+0000","last_peered":"2024-06-29T15:58:32.408636+0000","last_clean":"2024-06-29T15:58:32.408636+0000","last_became_active":"2024-06-29T15:58:19.312061+0000","last_became_peered":"2024-06-29T15:58:19.312061+0000","last_unstale":"2024-06-29T15:58:32.408636+0000","last_undegraded":"2024-06-29T15:58:32.408636+0000","last_fullsized":"2024-06-29T15:58:32.408636+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:29:10.919759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385564+0000","last_change":"2024-06-29T15:58:31.405442+0000","last_active":"2024-06-29T15:58:32.385564+0000","last_peered":"2024-06-29T15:58:32.385564+0000","last_clean":"2024-06-29T15:58:32.385564+0000","last_became_active":"2024-06-29T15:58:31.405046+0000","last_became_peered":"2024-06-29T15:58:31.405046+0000","last_unstale":"2024-06-29T15:58:32.385564+0000","last_undegraded":"2024-06-29T15:58:32.385564+0000","last_fullsized":"2024-06-29T15:58:32.385564+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:57.525409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385164+0000","last_change":"2024-06-29T15:58:30.391143+0000","last_active":"2024-06-29T15:58:32.385164+0000","last_peered":"2024-06-29T15:58:32.385164+0000","last_clean":"2024-06-29T15:58:32.385164+0000","last_became_active":"2024-06-29T15:58:30.390774+0000","last_became_peered":"2024-06-29T15:58:30.390774+0000","last_unstale":"2024-06-29T15:58:32.385164+0000","last_undegraded":"2024-06-29T15:58:32.385164+0000","last_fullsized":"2024-06-29T15:58:32.385164+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:05:00.346252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383385+0000","last_change":"2024-06-29T15:58:19.295810+0000","last_active":"2024-06-29T15:58:32.383385+0000","last_peered":"2024-06-29T15:58:32.383385+0000","last_clean":"2024-06-29T15:58:32.383385+0000","last_became_active":"2024-06-29T15:58:19.295445+0000","last_became_peered":"2024-06-29T15:58:19.295445+0000","last_unstale":"2024-06-29T15:58:32.383385+0000","last_undegraded":"2024-06-29T15:58:32.383385+0000","last_fullsized":"2024-06-29T15:58:32.383385+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:49:49.141794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408487+0000","last_change":"2024-06-29T15:58:31.402689+0000","last_active":"2024-06-29T15:58:32.408487+0000","last_peered":"2024-06-29T15:58:32.408487+0000","last_clean":"2024-06-29T15:58:32.408487+0000","last_became_active":"2024-06-29T15:58:31.402393+0000","last_became_peered":"2024-06-29T15:58:31.402393+0000","last_unstale":"2024-06-29T15:58:32.408487+0000","last_undegraded":"2024-06-29T15:58:32.408487+0000","last_fullsized":"2024-06-29T15:58:32.408487+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:15.757153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":14,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403354+0000","last_change":"2024-06-29T15:58:30.390626+0000","last_active":"2024-06-29T15:58:32.403354+0000","last_peered":"2024-06-29T15:58:32.403354+0000","last_clean":"2024-06-29T15:58:32.403354+0000","last_became_active":"2024-06-29T15:58:30.390230+0000","last_became_peered":"2024-06-29T15:58:30.390230+0000","last_unstale":"2024-06-29T15:58:32.403354+0000","last_undegraded":"2024-06-29T15:58:32.403354+0000","last_fullsized":"2024-06-29T15:58:32.403354+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:50:00.218269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386569+0000","last_change":"2024-06-29T15:58:19.315083+0000","last_active":"2024-06-29T15:58:32.386569+0000","last_peered":"2024-06-29T15:58:32.386569+0000","last_clean":"2024-06-29T15:58:32.386569+0000","last_became_active":"2024-06-29T15:58:19.313876+0000","last_became_peered":"2024-06-29T15:58:19.313876+0000","last_unstale":"2024-06-29T15:58:32.386569+0000","last_undegraded":"2024-06-29T15:58:32.386569+0000","last_fullsized":"2024-06-29T15:58:32.386569+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:57.766748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409078+0000","last_change":"2024-06-29T15:58:31.395696+0000","last_active":"2024-06-29T15:58:32.409078+0000","last_peered":"2024-06-29T15:58:32.409078+0000","last_clean":"2024-06-29T15:58:32.409078+0000","last_became_active":"2024-06-29T15:58:31.395401+0000","last_became_peered":"2024-06-29T15:58:31.395401+0000","last_unstale":"2024-06-29T15:58:32.409078+0000","last_undegraded":"2024-06-29T15:58:32.409078+0000","last_fullsized":"2024-06-29T15:58:32.409078+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:34.598819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"236'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471285+0000","last_change":"2024-06-29T15:58:30.383069+0000","last_active":"2024-06-29T15:58:32.471285+0000","last_peered":"2024-06-29T15:58:32.471285+0000","last_clean":"2024-06-29T15:58:32.471285+0000","last_became_active":"2024-06-29T15:58:30.382783+0000","last_became_peered":"2024-06-29T15:58:30.382783+0000","last_unstale":"2024-06-29T15:58:32.471285+0000","last_undegraded":"2024-06-29T15:58:32.471285+0000","last_fullsized":"2024-06-29T15:58:32.471285+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:40:27.674515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470719+0000","last_change":"2024-06-29T15:58:19.309408+0000","last_active":"2024-06-29T15:58:32.470719+0000","last_peered":"2024-06-29T15:58:32.470719+0000","last_clean":"2024-06-29T15:58:32.470719+0000","last_became_active":"2024-06-29T15:58:19.309050+0000","last_became_peered":"2024-06-29T15:58:19.309050+0000","last_unstale":"2024-06-29T15:58:32.470719+0000","last_undegraded":"2024-06-29T15:58:32.470719+0000","last_fullsized":"2024-06-29T15:58:32.470719+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:38:53.490742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470731+0000","last_change":"2024-06-29T15:58:31.413593+0000","last_active":"2024-06-29T15:58:32.470731+0000","last_peered":"2024-06-29T15:58:32.470731+0000","last_clean":"2024-06-29T15:58:32.470731+0000","last_became_active":"2024-06-29T15:58:31.413289+0000","last_became_peered":"2024-06-29T15:58:31.413289+0000","last_unstale":"2024-06-29T15:58:32.470731+0000","last_undegraded":"2024-06-29T15:58:32.470731+0000","last_fullsized":"2024-06-29T15:58:32.470731+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:21.755772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"236'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385764+0000","last_change":"2024-06-29T15:58:30.384132+0000","last_active":"2024-06-29T15:58:32.385764+0000","last_peered":"2024-06-29T15:58:32.385764+0000","last_clean":"2024-06-29T15:58:32.385764+0000","last_became_active":"2024-06-29T15:58:30.383773+0000","last_became_peered":"2024-06-29T15:58:30.383773+0000","last_unstale":"2024-06-29T15:58:32.385764+0000","last_undegraded":"2024-06-29T15:58:32.385764+0000","last_fullsized":"2024-06-29T15:58:32.385764+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:49.504495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408881+0000","last_change":"2024-06-29T15:58:19.305539+0000","last_active":"2024-06-29T15:58:32.408881+0000","last_peered":"2024-06-29T15:58:32.408881+0000","last_clean":"2024-06-29T15:58:32.408881+0000","last_became_active":"2024-06-29T15:58:19.305224+0000","last_became_peered":"2024-06-29T15:58:19.305224+0000","last_unstale":"2024-06-29T15:58:32.408881+0000","last_undegraded":"2024-06-29T15:58:32.408881+0000","last_fullsized":"2024-06-29T15:58:32.408881+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:21.381169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470959+0000","last_change":"2024-06-29T15:58:31.395948+0000","last_active":"2024-06-29T15:58:32.470959+0000","last_peered":"2024-06-29T15:58:32.470959+0000","last_clean":"2024-06-29T15:58:32.470959+0000","last_became_active":"2024-06-29T15:58:31.395228+0000","last_became_peered":"2024-06-29T15:58:31.395228+0000","last_unstale":"2024-06-29T15:58:32.470959+0000","last_undegraded":"2024-06-29T15:58:32.470959+0000","last_fullsized":"2024-06-29T15:58:32.470959+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:04.329393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408968+0000","last_change":"2024-06-29T15:58:30.382075+0000","last_active":"2024-06-29T15:58:32.408968+0000","last_peered":"2024-06-29T15:58:32.408968+0000","last_clean":"2024-06-29T15:58:32.408968+0000","last_became_active":"2024-06-29T15:58:30.381739+0000","last_became_peered":"2024-06-29T15:58:30.381739+0000","last_unstale":"2024-06-29T15:58:32.408968+0000","last_undegraded":"2024-06-29T15:58:32.408968+0000","last_fullsized":"2024-06-29T15:58:32.408968+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:08.875118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471086+0000","last_change":"2024-06-29T15:58:19.299124+0000","last_active":"2024-06-29T15:58:32.471086+0000","last_peered":"2024-06-29T15:58:32.471086+0000","last_clean":"2024-06-29T15:58:32.471086+0000","last_became_active":"2024-06-29T15:58:19.298839+0000","last_became_peered":"2024-06-29T15:58:19.298839+0000","last_unstale":"2024-06-29T15:58:32.471086+0000","last_undegraded":"2024-06-29T15:58:32.471086+0000","last_fullsized":"2024-06-29T15:58:32.471086+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:25:53.956024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385768+0000","last_change":"2024-06-29T15:58:31.404399+0000","last_active":"2024-06-29T15:58:32.385768+0000","last_peered":"2024-06-29T15:58:32.385768+0000","last_clean":"2024-06-29T15:58:32.385768+0000","last_became_active":"2024-06-29T15:58:31.402265+0000","last_became_peered":"2024-06-29T15:58:31.402265+0000","last_unstale":"2024-06-29T15:58:32.385768+0000","last_undegraded":"2024-06-29T15:58:32.385768+0000","last_fullsized":"2024-06-29T15:58:32.385768+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:15.757153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"236'2","reported_seq":16,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409018+0000","last_change":"2024-06-29T15:58:30.374918+0000","last_active":"2024-06-29T15:58:32.409018+0000","last_peered":"2024-06-29T15:58:32.409018+0000","last_clean":"2024-06-29T15:58:32.409018+0000","last_became_active":"2024-06-29T15:58:30.373985+0000","last_became_peered":"2024-06-29T15:58:30.373985+0000","last_unstale":"2024-06-29T15:58:32.409018+0000","last_undegraded":"2024-06-29T15:58:32.409018+0000","last_fullsized":"2024-06-29T15:58:32.409018+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:14:44.088939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403356+0000","last_change":"2024-06-29T15:58:19.298224+0000","last_active":"2024-06-29T15:58:32.403356+0000","last_peered":"2024-06-29T15:58:32.403356+0000","last_clean":"2024-06-29T15:58:32.403356+0000","last_became_active":"2024-06-29T15:58:19.297909+0000","last_became_peered":"2024-06-29T15:58:19.297909+0000","last_unstale":"2024-06-29T15:58:32.403356+0000","last_undegraded":"2024-06-29T15:58:32.403356+0000","last_fullsized":"2024-06-29T15:58:32.403356+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:08:17.158060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382878+0000","last_change":"2024-06-29T15:58:31.402837+0000","last_active":"2024-06-29T15:58:32.382878+0000","last_peered":"2024-06-29T15:58:32.382878+0000","last_clean":"2024-06-29T15:58:32.382878+0000","last_became_active":"2024-06-29T15:58:31.402439+0000","last_became_peered":"2024-06-29T15:58:31.402439+0000","last_unstale":"2024-06-29T15:58:32.382878+0000","last_undegraded":"2024-06-29T15:58:32.382878+0000","last_fullsized":"2024-06-29T15:58:32.382878+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:23.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"236'1","reported_seq":15,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383228+0000","last_change":"2024-06-29T15:58:30.390882+0000","last_active":"2024-06-29T15:58:32.383228+0000","last_peered":"2024-06-29T15:58:32.383228+0000","last_clean":"2024-06-29T15:58:32.383228+0000","last_became_active":"2024-06-29T15:58:30.390622+0000","last_became_peered":"2024-06-29T15:58:30.390622+0000","last_unstale":"2024-06-29T15:58:32.383228+0000","last_undegraded":"2024-06-29T15:58:32.383228+0000","last_fullsized":"2024-06-29T15:58:32.383228+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:49.061326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383542+0000","last_change":"2024-06-29T15:58:19.304924+0000","last_active":"2024-06-29T15:58:32.383542+0000","last_peered":"2024-06-29T15:58:32.383542+0000","last_clean":"2024-06-29T15:58:32.383542+0000","last_became_active":"2024-06-29T15:58:19.304317+0000","last_became_peered":"2024-06-29T15:58:19.304317+0000","last_unstale":"2024-06-29T15:58:32.383542+0000","last_undegraded":"2024-06-29T15:58:32.383542+0000","last_fullsized":"2024-06-29T15:58:32.383542+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:15:28.738310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386796+0000","last_change":"2024-06-29T15:58:31.403333+0000","last_active":"2024-06-29T15:58:32.386796+0000","last_peered":"2024-06-29T15:58:32.386796+0000","last_clean":"2024-06-29T15:58:32.386796+0000","last_became_active":"2024-06-29T15:58:31.402715+0000","last_became_peered":"2024-06-29T15:58:31.402715+0000","last_unstale":"2024-06-29T15:58:32.386796+0000","last_undegraded":"2024-06-29T15:58:32.386796+0000","last_fullsized":"2024-06-29T15:58:32.386796+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:32:40.005109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"236'4","reported_seq":18,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471608+0000","last_change":"2024-06-29T15:58:30.385595+0000","last_active":"2024-06-29T15:58:32.471608+0000","last_peered":"2024-06-29T15:58:32.471608+0000","last_clean":"2024-06-29T15:58:32.471608+0000","last_became_active":"2024-06-29T15:58:30.385187+0000","last_became_peered":"2024-06-29T15:58:30.385187+0000","last_unstale":"2024-06-29T15:58:32.471608+0000","last_undegraded":"2024-06-29T15:58:32.471608+0000","last_fullsized":"2024-06-29T15:58:32.471608+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:29.353375+0000","last_clean_scrub_stamp":"2024-06-29T15:58:29.353375+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:13.071438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408823+0000","last_change":"2024-06-29T15:58:19.309529+0000","last_active":"2024-06-29T15:58:32.408823+0000","last_peered":"2024-06-29T15:58:32.408823+0000","last_clean":"2024-06-29T15:58:32.408823+0000","last_became_active":"2024-06-29T15:58:19.309193+0000","last_became_peered":"2024-06-29T15:58:19.309193+0000","last_unstale":"2024-06-29T15:58:32.408823+0000","last_undegraded":"2024-06-29T15:58:32.408823+0000","last_fullsized":"2024-06-29T15:58:32.408823+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:37:21.882892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470299+0000","last_change":"2024-06-29T15:58:31.398191+0000","last_active":"2024-06-29T15:58:32.470299+0000","last_peered":"2024-06-29T15:58:32.470299+0000","last_clean":"2024-06-29T15:58:32.470299+0000","last_became_active":"2024-06-29T15:58:31.397845+0000","last_became_peered":"2024-06-29T15:58:31.397845+0000","last_unstale":"2024-06-29T15:58:32.470299+0000","last_undegraded":"2024-06-29T15:58:32.470299+0000","last_fullsized":"2024-06-29T15:58:32.470299+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:00:16.215524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471673+0000","last_change":"2024-06-29T15:58:19.306514+0000","last_active":"2024-06-29T15:58:32.471673+0000","last_peered":"2024-06-29T15:58:32.471673+0000","last_clean":"2024-06-29T15:58:32.471673+0000","last_became_active":"2024-06-29T15:58:19.306099+0000","last_became_peered":"2024-06-29T15:58:19.306099+0000","last_unstale":"2024-06-29T15:58:32.471673+0000","last_undegraded":"2024-06-29T15:58:32.471673+0000","last_fullsized":"2024-06-29T15:58:32.471673+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:59:25.572276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"232'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386775+0000","last_change":"2024-06-29T15:58:18.301939+0000","last_active":"2024-06-29T15:58:32.386775+0000","last_peered":"2024-06-29T15:58:32.386775+0000","last_clean":"2024-06-29T15:58:32.386775+0000","last_became_active":"2024-06-29T15:58:18.301258+0000","last_became_peered":"2024-06-29T15:58:18.301258+0000","last_unstale":"2024-06-29T15:58:32.386775+0000","last_undegraded":"2024-06-29T15:58:32.386775+0000","last_fullsized":"2024-06-29T15:58:32.386775+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:53.497595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383084+0000","last_change":"2024-06-29T15:58:31.404199+0000","last_active":"2024-06-29T15:58:32.383084+0000","last_peered":"2024-06-29T15:58:32.383084+0000","last_clean":"2024-06-29T15:58:32.383084+0000","last_became_active":"2024-06-29T15:58:31.403910+0000","last_became_peered":"2024-06-29T15:58:31.403910+0000","last_unstale":"2024-06-29T15:58:32.383084+0000","last_undegraded":"2024-06-29T15:58:32.383084+0000","last_fullsized":"2024-06-29T15:58:32.383084+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:21.755772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470398+0000","last_change":"2024-06-29T15:58:19.298015+0000","last_active":"2024-06-29T15:58:32.470398+0000","last_peered":"2024-06-29T15:58:32.470398+0000","last_clean":"2024-06-29T15:58:32.470398+0000","last_became_active":"2024-06-29T15:58:19.297706+0000","last_became_peered":"2024-06-29T15:58:19.297706+0000","last_unstale":"2024-06-29T15:58:32.470398+0000","last_undegraded":"2024-06-29T15:58:32.470398+0000","last_fullsized":"2024-06-29T15:58:32.470398+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:59:36.737724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471128+0000","last_change":"2024-06-29T15:58:18.297534+0000","last_active":"2024-06-29T15:58:32.471128+0000","last_peered":"2024-06-29T15:58:32.471128+0000","last_clean":"2024-06-29T15:58:32.471128+0000","last_became_active":"2024-06-29T15:58:18.297260+0000","last_became_peered":"2024-06-29T15:58:18.297260+0000","last_unstale":"2024-06-29T15:58:32.471128+0000","last_undegraded":"2024-06-29T15:58:32.471128+0000","last_fullsized":"2024-06-29T15:58:32.471128+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:47:47.544183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471365+0000","last_change":"2024-06-29T15:58:31.397157+0000","last_active":"2024-06-29T15:58:32.471365+0000","last_peered":"2024-06-29T15:58:32.471365+0000","last_clean":"2024-06-29T15:58:32.471365+0000","last_became_active":"2024-06-29T15:58:31.396872+0000","last_became_peered":"2024-06-29T15:58:31.396872+0000","last_unstale":"2024-06-29T15:58:32.471365+0000","last_undegraded":"2024-06-29T15:58:32.471365+0000","last_fullsized":"2024-06-29T15:58:32.471365+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:43:27.765411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471038+0000","last_change":"2024-06-29T15:58:19.305103+0000","last_active":"2024-06-29T15:58:32.471038+0000","last_peered":"2024-06-29T15:58:32.471038+0000","last_clean":"2024-06-29T15:58:32.471038+0000","last_became_active":"2024-06-29T15:58:19.304612+0000","last_became_peered":"2024-06-29T15:58:19.304612+0000","last_unstale":"2024-06-29T15:58:32.471038+0000","last_undegraded":"2024-06-29T15:58:32.471038+0000","last_fullsized":"2024-06-29T15:58:32.471038+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:19.583540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471702+0000","last_change":"2024-06-29T15:58:18.302160+0000","last_active":"2024-06-29T15:58:32.471702+0000","last_peered":"2024-06-29T15:58:32.471702+0000","last_clean":"2024-06-29T15:58:32.471702+0000","last_became_active":"2024-06-29T15:58:18.301895+0000","last_became_peered":"2024-06-29T15:58:18.301895+0000","last_unstale":"2024-06-29T15:58:32.471702+0000","last_undegraded":"2024-06-29T15:58:32.471702+0000","last_fullsized":"2024-06-29T15:58:32.471702+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:26.756484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383158+0000","last_change":"2024-06-29T15:58:31.394840+0000","last_active":"2024-06-29T15:58:32.383158+0000","last_peered":"2024-06-29T15:58:32.383158+0000","last_clean":"2024-06-29T15:58:32.383158+0000","last_became_active":"2024-06-29T15:58:31.393905+0000","last_became_peered":"2024-06-29T15:58:31.393905+0000","last_unstale":"2024-06-29T15:58:32.383158+0000","last_undegraded":"2024-06-29T15:58:32.383158+0000","last_fullsized":"2024-06-29T15:58:32.383158+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:07.236049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470593+0000","last_change":"2024-06-29T15:58:19.299884+0000","last_active":"2024-06-29T15:58:32.470593+0000","last_peered":"2024-06-29T15:58:32.470593+0000","last_clean":"2024-06-29T15:58:32.470593+0000","last_became_active":"2024-06-29T15:58:19.299621+0000","last_became_peered":"2024-06-29T15:58:19.299621+0000","last_unstale":"2024-06-29T15:58:32.470593+0000","last_undegraded":"2024-06-29T15:58:32.470593+0000","last_fullsized":"2024-06-29T15:58:32.470593+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:09.325434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"232'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382745+0000","last_change":"2024-06-29T15:58:18.299460+0000","last_active":"2024-06-29T15:58:32.382745+0000","last_peered":"2024-06-29T15:58:32.382745+0000","last_clean":"2024-06-29T15:58:32.382745+0000","last_became_active":"2024-06-29T15:58:18.299171+0000","last_became_peered":"2024-06-29T15:58:18.299171+0000","last_unstale":"2024-06-29T15:58:32.382745+0000","last_undegraded":"2024-06-29T15:58:32.382745+0000","last_fullsized":"2024-06-29T15:58:32.382745+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:34:40.426770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385789+0000","last_change":"2024-06-29T15:58:31.403071+0000","last_active":"2024-06-29T15:58:32.385789+0000","last_peered":"2024-06-29T15:58:32.385789+0000","last_clean":"2024-06-29T15:58:32.385789+0000","last_became_active":"2024-06-29T15:58:31.402742+0000","last_became_peered":"2024-06-29T15:58:31.402742+0000","last_unstale":"2024-06-29T15:58:32.385789+0000","last_undegraded":"2024-06-29T15:58:32.385789+0000","last_fullsized":"2024-06-29T15:58:32.385789+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:00.629657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384209+0000","last_change":"2024-06-29T15:58:19.306300+0000","last_active":"2024-06-29T15:58:32.384209+0000","last_peered":"2024-06-29T15:58:32.384209+0000","last_clean":"2024-06-29T15:58:32.384209+0000","last_became_active":"2024-06-29T15:58:19.305976+0000","last_became_peered":"2024-06-29T15:58:19.305976+0000","last_unstale":"2024-06-29T15:58:32.384209+0000","last_undegraded":"2024-06-29T15:58:32.384209+0000","last_fullsized":"2024-06-29T15:58:32.384209+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:10:21.792399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"232'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409092+0000","last_change":"2024-06-29T15:58:18.297242+0000","last_active":"2024-06-29T15:58:32.409092+0000","last_peered":"2024-06-29T15:58:32.409092+0000","last_clean":"2024-06-29T15:58:32.409092+0000","last_became_active":"2024-06-29T15:58:18.296876+0000","last_became_peered":"2024-06-29T15:58:18.296876+0000","last_unstale":"2024-06-29T15:58:32.409092+0000","last_undegraded":"2024-06-29T15:58:32.409092+0000","last_fullsized":"2024-06-29T15:58:32.409092+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:32:53.497595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403749+0000","last_change":"2024-06-29T15:58:31.408178+0000","last_active":"2024-06-29T15:58:32.403749+0000","last_peered":"2024-06-29T15:58:32.403749+0000","last_clean":"2024-06-29T15:58:32.403749+0000","last_became_active":"2024-06-29T15:58:31.407829+0000","last_became_peered":"2024-06-29T15:58:31.407829+0000","last_unstale":"2024-06-29T15:58:32.403749+0000","last_undegraded":"2024-06-29T15:58:32.403749+0000","last_fullsized":"2024-06-29T15:58:32.403749+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:37.594096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471795+0000","last_change":"2024-06-29T15:58:19.300049+0000","last_active":"2024-06-29T15:58:32.471795+0000","last_peered":"2024-06-29T15:58:32.471795+0000","last_clean":"2024-06-29T15:58:32.471795+0000","last_became_active":"2024-06-29T15:58:19.299756+0000","last_became_peered":"2024-06-29T15:58:19.299756+0000","last_unstale":"2024-06-29T15:58:32.471795+0000","last_undegraded":"2024-06-29T15:58:32.471795+0000","last_fullsized":"2024-06-29T15:58:32.471795+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:25:50.310157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386054+0000","last_change":"2024-06-29T15:58:18.302669+0000","last_active":"2024-06-29T15:58:32.386054+0000","last_peered":"2024-06-29T15:58:32.386054+0000","last_clean":"2024-06-29T15:58:32.386054+0000","last_became_active":"2024-06-29T15:58:18.302098+0000","last_became_peered":"2024-06-29T15:58:18.302098+0000","last_unstale":"2024-06-29T15:58:32.386054+0000","last_undegraded":"2024-06-29T15:58:32.386054+0000","last_fullsized":"2024-06-29T15:58:32.386054+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:01:18.581416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383328+0000","last_change":"2024-06-29T15:58:31.398566+0000","last_active":"2024-06-29T15:58:32.383328+0000","last_peered":"2024-06-29T15:58:32.383328+0000","last_clean":"2024-06-29T15:58:32.383328+0000","last_became_active":"2024-06-29T15:58:31.398272+0000","last_became_peered":"2024-06-29T15:58:31.398272+0000","last_unstale":"2024-06-29T15:58:32.383328+0000","last_undegraded":"2024-06-29T15:58:32.383328+0000","last_fullsized":"2024-06-29T15:58:32.383328+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:50.511371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408466+0000","last_change":"2024-06-29T15:58:19.308028+0000","last_active":"2024-06-29T15:58:32.408466+0000","last_peered":"2024-06-29T15:58:32.408466+0000","last_clean":"2024-06-29T15:58:32.408466+0000","last_became_active":"2024-06-29T15:58:19.306593+0000","last_became_peered":"2024-06-29T15:58:19.306593+0000","last_unstale":"2024-06-29T15:58:32.408466+0000","last_undegraded":"2024-06-29T15:58:32.408466+0000","last_fullsized":"2024-06-29T15:58:32.408466+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:22:45.907895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471895+0000","last_change":"2024-06-29T15:58:18.300805+0000","last_active":"2024-06-29T15:58:32.471895+0000","last_peered":"2024-06-29T15:58:32.471895+0000","last_clean":"2024-06-29T15:58:32.471895+0000","last_became_active":"2024-06-29T15:58:18.300515+0000","last_became_peered":"2024-06-29T15:58:18.300515+0000","last_unstale":"2024-06-29T15:58:32.471895+0000","last_undegraded":"2024-06-29T15:58:32.471895+0000","last_fullsized":"2024-06-29T15:58:32.471895+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:14.605715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471605+0000","last_change":"2024-06-29T15:58:31.398694+0000","last_active":"2024-06-29T15:58:32.471605+0000","last_peered":"2024-06-29T15:58:32.471605+0000","last_clean":"2024-06-29T15:58:32.471605+0000","last_became_active":"2024-06-29T15:58:31.398005+0000","last_became_peered":"2024-06-29T15:58:31.398005+0000","last_unstale":"2024-06-29T15:58:32.471605+0000","last_undegraded":"2024-06-29T15:58:32.471605+0000","last_fullsized":"2024-06-29T15:58:32.471605+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:43.892782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409202+0000","last_change":"2024-06-29T15:58:19.303106+0000","last_active":"2024-06-29T15:58:32.409202+0000","last_peered":"2024-06-29T15:58:32.409202+0000","last_clean":"2024-06-29T15:58:32.409202+0000","last_became_active":"2024-06-29T15:58:19.302806+0000","last_became_peered":"2024-06-29T15:58:19.302806+0000","last_unstale":"2024-06-29T15:58:32.409202+0000","last_undegraded":"2024-06-29T15:58:32.409202+0000","last_fullsized":"2024-06-29T15:58:32.409202+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:37:18.440072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471318+0000","last_change":"2024-06-29T15:58:18.303872+0000","last_active":"2024-06-29T15:58:32.471318+0000","last_peered":"2024-06-29T15:58:32.471318+0000","last_clean":"2024-06-29T15:58:32.471318+0000","last_became_active":"2024-06-29T15:58:18.303542+0000","last_became_peered":"2024-06-29T15:58:18.303542+0000","last_unstale":"2024-06-29T15:58:32.471318+0000","last_undegraded":"2024-06-29T15:58:32.471318+0000","last_fullsized":"2024-06-29T15:58:32.471318+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:04.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385249+0000","last_change":"2024-06-29T15:58:31.403850+0000","last_active":"2024-06-29T15:58:32.385249+0000","last_peered":"2024-06-29T15:58:32.385249+0000","last_clean":"2024-06-29T15:58:32.385249+0000","last_became_active":"2024-06-29T15:58:31.402437+0000","last_became_peered":"2024-06-29T15:58:31.402437+0000","last_unstale":"2024-06-29T15:58:32.385249+0000","last_undegraded":"2024-06-29T15:58:32.385249+0000","last_fullsized":"2024-06-29T15:58:32.385249+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:27.691189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471493+0000","last_change":"2024-06-29T15:58:19.295640+0000","last_active":"2024-06-29T15:58:32.471493+0000","last_peered":"2024-06-29T15:58:32.471493+0000","last_clean":"2024-06-29T15:58:32.471493+0000","last_became_active":"2024-06-29T15:58:19.295239+0000","last_became_peered":"2024-06-29T15:58:19.295239+0000","last_unstale":"2024-06-29T15:58:32.471493+0000","last_undegraded":"2024-06-29T15:58:32.471493+0000","last_fullsized":"2024-06-29T15:58:32.471493+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:19.616206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471033+0000","last_change":"2024-06-29T15:58:18.294635+0000","last_active":"2024-06-29T15:58:32.471033+0000","last_peered":"2024-06-29T15:58:32.471033+0000","last_clean":"2024-06-29T15:58:32.471033+0000","last_became_active":"2024-06-29T15:58:18.294343+0000","last_became_peered":"2024-06-29T15:58:18.294343+0000","last_unstale":"2024-06-29T15:58:32.471033+0000","last_undegraded":"2024-06-29T15:58:32.471033+0000","last_fullsized":"2024-06-29T15:58:32.471033+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:59:44.439934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470505+0000","last_change":"2024-06-29T15:58:31.408169+0000","last_active":"2024-06-29T15:58:32.470505+0000","last_peered":"2024-06-29T15:58:32.470505+0000","last_clean":"2024-06-29T15:58:32.470505+0000","last_became_active":"2024-06-29T15:58:31.407860+0000","last_became_peered":"2024-06-29T15:58:31.407860+0000","last_unstale":"2024-06-29T15:58:32.470505+0000","last_undegraded":"2024-06-29T15:58:32.470505+0000","last_fullsized":"2024-06-29T15:58:32.470505+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:14.288552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471582+0000","last_change":"2024-06-29T15:58:19.306811+0000","last_active":"2024-06-29T15:58:32.471582+0000","last_peered":"2024-06-29T15:58:32.471582+0000","last_clean":"2024-06-29T15:58:32.471582+0000","last_became_active":"2024-06-29T15:58:19.306354+0000","last_became_peered":"2024-06-29T15:58:19.306354+0000","last_unstale":"2024-06-29T15:58:32.471582+0000","last_undegraded":"2024-06-29T15:58:32.471582+0000","last_fullsized":"2024-06-29T15:58:32.471582+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:55.684049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"232'3","reported_seq":25,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409089+0000","last_change":"2024-06-29T15:58:18.300841+0000","last_active":"2024-06-29T15:58:32.409089+0000","last_peered":"2024-06-29T15:58:32.409089+0000","last_clean":"2024-06-29T15:58:32.409089+0000","last_became_active":"2024-06-29T15:58:18.300525+0000","last_became_peered":"2024-06-29T15:58:18.300525+0000","last_unstale":"2024-06-29T15:58:32.409089+0000","last_undegraded":"2024-06-29T15:58:32.409089+0000","last_fullsized":"2024-06-29T15:58:32.409089+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:51:15.561954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470875+0000","last_change":"2024-06-29T15:58:31.412217+0000","last_active":"2024-06-29T15:58:32.470875+0000","last_peered":"2024-06-29T15:58:32.470875+0000","last_clean":"2024-06-29T15:58:32.470875+0000","last_became_active":"2024-06-29T15:58:31.411886+0000","last_became_peered":"2024-06-29T15:58:31.411886+0000","last_unstale":"2024-06-29T15:58:32.470875+0000","last_undegraded":"2024-06-29T15:58:32.470875+0000","last_fullsized":"2024-06-29T15:58:32.470875+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:36:07.236049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386255+0000","last_change":"2024-06-29T15:58:19.310218+0000","last_active":"2024-06-29T15:58:32.386255+0000","last_peered":"2024-06-29T15:58:32.386255+0000","last_clean":"2024-06-29T15:58:32.386255+0000","last_became_active":"2024-06-29T15:58:19.309910+0000","last_became_peered":"2024-06-29T15:58:19.309910+0000","last_unstale":"2024-06-29T15:58:32.386255+0000","last_undegraded":"2024-06-29T15:58:32.386255+0000","last_fullsized":"2024-06-29T15:58:32.386255+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:14:48.781692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"234'8","reported_seq":30,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408984+0000","last_change":"2024-06-29T15:58:18.296167+0000","last_active":"2024-06-29T15:58:32.408984+0000","last_peered":"2024-06-29T15:58:32.408984+0000","last_clean":"2024-06-29T15:58:32.408984+0000","last_became_active":"2024-06-29T15:58:18.294984+0000","last_became_peered":"2024-06-29T15:58:18.294984+0000","last_unstale":"2024-06-29T15:58:32.408984+0000","last_undegraded":"2024-06-29T15:58:32.408984+0000","last_fullsized":"2024-06-29T15:58:32.408984+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:41:07.733361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2855,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408826+0000","last_change":"2024-06-29T15:58:31.394062+0000","last_active":"2024-06-29T15:58:32.408826+0000","last_peered":"2024-06-29T15:58:32.408826+0000","last_clean":"2024-06-29T15:58:32.408826+0000","last_became_active":"2024-06-29T15:58:31.393756+0000","last_became_peered":"2024-06-29T15:58:31.393756+0000","last_unstale":"2024-06-29T15:58:32.408826+0000","last_undegraded":"2024-06-29T15:58:32.408826+0000","last_fullsized":"2024-06-29T15:58:32.408826+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:28.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385645+0000","last_change":"2024-06-29T15:58:19.315403+0000","last_active":"2024-06-29T15:58:32.385645+0000","last_peered":"2024-06-29T15:58:32.385645+0000","last_clean":"2024-06-29T15:58:32.385645+0000","last_became_active":"2024-06-29T15:58:19.314969+0000","last_became_peered":"2024-06-29T15:58:19.314969+0000","last_unstale":"2024-06-29T15:58:32.385645+0000","last_undegraded":"2024-06-29T15:58:32.385645+0000","last_fullsized":"2024-06-29T15:58:32.385645+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:42.361706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409315+0000","last_change":"2024-06-29T15:58:18.303307+0000","last_active":"2024-06-29T15:58:32.409315+0000","last_peered":"2024-06-29T15:58:32.409315+0000","last_clean":"2024-06-29T15:58:32.409315+0000","last_became_active":"2024-06-29T15:58:18.303009+0000","last_became_peered":"2024-06-29T15:58:18.303009+0000","last_unstale":"2024-06-29T15:58:32.409315+0000","last_undegraded":"2024-06-29T15:58:32.409315+0000","last_fullsized":"2024-06-29T15:58:32.409315+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:08.507147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385551+0000","last_change":"2024-06-29T15:58:31.410977+0000","last_active":"2024-06-29T15:58:32.385551+0000","last_peered":"2024-06-29T15:58:32.385551+0000","last_clean":"2024-06-29T15:58:32.385551+0000","last_became_active":"2024-06-29T15:58:31.410506+0000","last_became_peered":"2024-06-29T15:58:31.410506+0000","last_unstale":"2024-06-29T15:58:32.385551+0000","last_undegraded":"2024-06-29T15:58:32.385551+0000","last_fullsized":"2024-06-29T15:58:32.385551+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:59:43.051580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384172+0000","last_change":"2024-06-29T15:58:19.304892+0000","last_active":"2024-06-29T15:58:32.384172+0000","last_peered":"2024-06-29T15:58:32.384172+0000","last_clean":"2024-06-29T15:58:32.384172+0000","last_became_active":"2024-06-29T15:58:19.304368+0000","last_became_peered":"2024-06-29T15:58:19.304368+0000","last_unstale":"2024-06-29T15:58:32.384172+0000","last_undegraded":"2024-06-29T15:58:32.384172+0000","last_fullsized":"2024-06-29T15:58:32.384172+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:49:34.621330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471411+0000","last_change":"2024-06-29T15:58:18.290469+0000","last_active":"2024-06-29T15:58:32.471411+0000","last_peered":"2024-06-29T15:58:32.471411+0000","last_clean":"2024-06-29T15:58:32.471411+0000","last_became_active":"2024-06-29T15:58:18.290177+0000","last_became_peered":"2024-06-29T15:58:18.290177+0000","last_unstale":"2024-06-29T15:58:32.471411+0000","last_undegraded":"2024-06-29T15:58:32.471411+0000","last_fullsized":"2024-06-29T15:58:32.471411+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:28:51.797075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386382+0000","last_change":"2024-06-29T15:58:31.404104+0000","last_active":"2024-06-29T15:58:32.386382+0000","last_peered":"2024-06-29T15:58:32.386382+0000","last_clean":"2024-06-29T15:58:32.386382+0000","last_became_active":"2024-06-29T15:58:31.403759+0000","last_became_peered":"2024-06-29T15:58:31.403759+0000","last_unstale":"2024-06-29T15:58:32.386382+0000","last_undegraded":"2024-06-29T15:58:32.386382+0000","last_fullsized":"2024-06-29T15:58:32.386382+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:57:22.376570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382482+0000","last_change":"2024-06-29T15:58:19.304069+0000","last_active":"2024-06-29T15:58:32.382482+0000","last_peered":"2024-06-29T15:58:32.382482+0000","last_clean":"2024-06-29T15:58:32.382482+0000","last_became_active":"2024-06-29T15:58:19.303758+0000","last_became_peered":"2024-06-29T15:58:19.303758+0000","last_unstale":"2024-06-29T15:58:32.382482+0000","last_undegraded":"2024-06-29T15:58:32.382482+0000","last_fullsized":"2024-06-29T15:58:32.382482+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:51.378080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386174+0000","last_change":"2024-06-29T15:58:18.303515+0000","last_active":"2024-06-29T15:58:32.386174+0000","last_peered":"2024-06-29T15:58:32.386174+0000","last_clean":"2024-06-29T15:58:32.386174+0000","last_became_active":"2024-06-29T15:58:18.303008+0000","last_became_peered":"2024-06-29T15:58:18.303008+0000","last_unstale":"2024-06-29T15:58:32.386174+0000","last_undegraded":"2024-06-29T15:58:32.386174+0000","last_fullsized":"2024-06-29T15:58:32.386174+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:08.323310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471875+0000","last_change":"2024-06-29T15:58:31.406136+0000","last_active":"2024-06-29T15:58:32.471875+0000","last_peered":"2024-06-29T15:58:32.471875+0000","last_clean":"2024-06-29T15:58:32.471875+0000","last_became_active":"2024-06-29T15:58:31.405829+0000","last_became_peered":"2024-06-29T15:58:31.405829+0000","last_unstale":"2024-06-29T15:58:32.471875+0000","last_undegraded":"2024-06-29T15:58:32.471875+0000","last_fullsized":"2024-06-29T15:58:32.471875+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:05.000910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386012+0000","last_change":"2024-06-29T15:58:19.316203+0000","last_active":"2024-06-29T15:58:32.386012+0000","last_peered":"2024-06-29T15:58:32.386012+0000","last_clean":"2024-06-29T15:58:32.386012+0000","last_became_active":"2024-06-29T15:58:19.315777+0000","last_became_peered":"2024-06-29T15:58:19.315777+0000","last_unstale":"2024-06-29T15:58:32.386012+0000","last_undegraded":"2024-06-29T15:58:32.386012+0000","last_fullsized":"2024-06-29T15:58:32.386012+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:21.381169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"232'8","reported_seq":30,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470099+0000","last_change":"2024-06-29T15:58:18.302451+0000","last_active":"2024-06-29T15:58:32.470099+0000","last_peered":"2024-06-29T15:58:32.470099+0000","last_clean":"2024-06-29T15:58:32.470099+0000","last_became_active":"2024-06-29T15:58:18.302144+0000","last_became_peered":"2024-06-29T15:58:18.302144+0000","last_unstale":"2024-06-29T15:58:32.470099+0000","last_undegraded":"2024-06-29T15:58:32.470099+0000","last_fullsized":"2024-06-29T15:58:32.470099+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:15:27.736186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408461+0000","last_change":"2024-06-29T15:58:31.390906+0000","last_active":"2024-06-29T15:58:32.408461+0000","last_peered":"2024-06-29T15:58:32.408461+0000","last_clean":"2024-06-29T15:58:32.408461+0000","last_became_active":"2024-06-29T15:58:31.390187+0000","last_became_peered":"2024-06-29T15:58:31.390187+0000","last_unstale":"2024-06-29T15:58:32.408461+0000","last_undegraded":"2024-06-29T15:58:32.408461+0000","last_fullsized":"2024-06-29T15:58:32.408461+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:59:43.051580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383655+0000","last_change":"2024-06-29T15:58:19.297900+0000","last_active":"2024-06-29T15:58:32.383655+0000","last_peered":"2024-06-29T15:58:32.383655+0000","last_clean":"2024-06-29T15:58:32.383655+0000","last_became_active":"2024-06-29T15:58:19.297605+0000","last_became_peered":"2024-06-29T15:58:19.297605+0000","last_unstale":"2024-06-29T15:58:32.383655+0000","last_undegraded":"2024-06-29T15:58:32.383655+0000","last_fullsized":"2024-06-29T15:58:32.383655+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:25.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"232'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386465+0000","last_change":"2024-06-29T15:58:18.303063+0000","last_active":"2024-06-29T15:58:32.386465+0000","last_peered":"2024-06-29T15:58:32.386465+0000","last_clean":"2024-06-29T15:58:32.386465+0000","last_became_active":"2024-06-29T15:58:18.302403+0000","last_became_peered":"2024-06-29T15:58:18.302403+0000","last_unstale":"2024-06-29T15:58:32.386465+0000","last_undegraded":"2024-06-29T15:58:32.386465+0000","last_fullsized":"2024-06-29T15:58:32.386465+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:59:35.735600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382562+0000","last_change":"2024-06-29T15:58:31.402054+0000","last_active":"2024-06-29T15:58:32.382562+0000","last_peered":"2024-06-29T15:58:32.382562+0000","last_clean":"2024-06-29T15:58:32.382562+0000","last_became_active":"2024-06-29T15:58:31.401774+0000","last_became_peered":"2024-06-29T15:58:31.401774+0000","last_unstale":"2024-06-29T15:58:32.382562+0000","last_undegraded":"2024-06-29T15:58:32.382562+0000","last_fullsized":"2024-06-29T15:58:32.382562+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:33.464520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471788+0000","last_change":"2024-06-29T15:58:19.310412+0000","last_active":"2024-06-29T15:58:32.471788+0000","last_peered":"2024-06-29T15:58:32.471788+0000","last_clean":"2024-06-29T15:58:32.471788+0000","last_became_active":"2024-06-29T15:58:19.310109+0000","last_became_peered":"2024-06-29T15:58:19.310109+0000","last_unstale":"2024-06-29T15:58:32.471788+0000","last_undegraded":"2024-06-29T15:58:32.471788+0000","last_fullsized":"2024-06-29T15:58:32.471788+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:57:56.479901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471159+0000","last_change":"2024-06-29T15:58:18.296680+0000","last_active":"2024-06-29T15:58:32.471159+0000","last_peered":"2024-06-29T15:58:32.471159+0000","last_clean":"2024-06-29T15:58:32.471159+0000","last_became_active":"2024-06-29T15:58:18.296405+0000","last_became_peered":"2024-06-29T15:58:18.296405+0000","last_unstale":"2024-06-29T15:58:32.471159+0000","last_undegraded":"2024-06-29T15:58:32.471159+0000","last_fullsized":"2024-06-29T15:58:32.471159+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:02.671679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382818+0000","last_change":"2024-06-29T15:58:31.404958+0000","last_active":"2024-06-29T15:58:32.382818+0000","last_peered":"2024-06-29T15:58:32.382818+0000","last_clean":"2024-06-29T15:58:32.382818+0000","last_became_active":"2024-06-29T15:58:31.404664+0000","last_became_peered":"2024-06-29T15:58:31.404664+0000","last_unstale":"2024-06-29T15:58:32.382818+0000","last_undegraded":"2024-06-29T15:58:32.382818+0000","last_fullsized":"2024-06-29T15:58:32.382818+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:38:04.329393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383683+0000","last_change":"2024-06-29T15:58:19.308630+0000","last_active":"2024-06-29T15:58:32.383683+0000","last_peered":"2024-06-29T15:58:32.383683+0000","last_clean":"2024-06-29T15:58:32.383683+0000","last_became_active":"2024-06-29T15:58:19.308323+0000","last_became_peered":"2024-06-29T15:58:19.308323+0000","last_unstale":"2024-06-29T15:58:32.383683+0000","last_undegraded":"2024-06-29T15:58:32.383683+0000","last_fullsized":"2024-06-29T15:58:32.383683+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:14:39.500064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"232'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383767+0000","last_change":"2024-06-29T15:58:18.294949+0000","last_active":"2024-06-29T15:58:32.383767+0000","last_peered":"2024-06-29T15:58:32.383767+0000","last_clean":"2024-06-29T15:58:32.383767+0000","last_became_active":"2024-06-29T15:58:18.294015+0000","last_became_peered":"2024-06-29T15:58:18.294015+0000","last_unstale":"2024-06-29T15:58:32.383767+0000","last_undegraded":"2024-06-29T15:58:32.383767+0000","last_fullsized":"2024-06-29T15:58:32.383767+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:49:36.977744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.382766+0000","last_change":"2024-06-29T15:58:31.390484+0000","last_active":"2024-06-29T15:58:32.382766+0000","last_peered":"2024-06-29T15:58:32.382766+0000","last_clean":"2024-06-29T15:58:32.382766+0000","last_became_active":"2024-06-29T15:58:31.390054+0000","last_became_peered":"2024-06-29T15:58:31.390054+0000","last_unstale":"2024-06-29T15:58:32.382766+0000","last_undegraded":"2024-06-29T15:58:32.382766+0000","last_fullsized":"2024-06-29T15:58:32.382766+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:54.445056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471189+0000","last_change":"2024-06-29T15:58:19.305909+0000","last_active":"2024-06-29T15:58:32.471189+0000","last_peered":"2024-06-29T15:58:32.471189+0000","last_clean":"2024-06-29T15:58:32.471189+0000","last_became_active":"2024-06-29T15:58:19.305594+0000","last_became_peered":"2024-06-29T15:58:19.305594+0000","last_unstale":"2024-06-29T15:58:32.471189+0000","last_undegraded":"2024-06-29T15:58:32.471189+0000","last_fullsized":"2024-06-29T15:58:32.471189+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:09.325434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470535+0000","last_change":"2024-06-29T15:58:18.306207+0000","last_active":"2024-06-29T15:58:32.470535+0000","last_peered":"2024-06-29T15:58:32.470535+0000","last_clean":"2024-06-29T15:58:32.470535+0000","last_became_active":"2024-06-29T15:58:18.305661+0000","last_became_peered":"2024-06-29T15:58:18.305661+0000","last_unstale":"2024-06-29T15:58:32.470535+0000","last_undegraded":"2024-06-29T15:58:32.470535+0000","last_fullsized":"2024-06-29T15:58:32.470535+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:44:47.769658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470826+0000","last_change":"2024-06-29T15:58:31.399014+0000","last_active":"2024-06-29T15:58:32.470826+0000","last_peered":"2024-06-29T15:58:32.470826+0000","last_clean":"2024-06-29T15:58:32.470826+0000","last_became_active":"2024-06-29T15:58:31.398631+0000","last_became_peered":"2024-06-29T15:58:31.398631+0000","last_unstale":"2024-06-29T15:58:32.470826+0000","last_undegraded":"2024-06-29T15:58:32.470826+0000","last_fullsized":"2024-06-29T15:58:32.470826+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:43:54.445056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470672+0000","last_change":"2024-06-29T15:58:19.307416+0000","last_active":"2024-06-29T15:58:32.470672+0000","last_peered":"2024-06-29T15:58:32.470672+0000","last_clean":"2024-06-29T15:58:32.470672+0000","last_became_active":"2024-06-29T15:58:19.307089+0000","last_became_peered":"2024-06-29T15:58:19.307089+0000","last_unstale":"2024-06-29T15:58:32.470672+0000","last_undegraded":"2024-06-29T15:58:32.470672+0000","last_fullsized":"2024-06-29T15:58:32.470672+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:54.355267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471513+0000","last_change":"2024-06-29T15:58:18.302497+0000","last_active":"2024-06-29T15:58:32.471513+0000","last_peered":"2024-06-29T15:58:32.471513+0000","last_clean":"2024-06-29T15:58:32.471513+0000","last_became_active":"2024-06-29T15:58:18.302200+0000","last_became_peered":"2024-06-29T15:58:18.302200+0000","last_unstale":"2024-06-29T15:58:32.471513+0000","last_undegraded":"2024-06-29T15:58:32.471513+0000","last_fullsized":"2024-06-29T15:58:32.471513+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:28:59.065708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385238+0000","last_change":"2024-06-29T15:58:31.403813+0000","last_active":"2024-06-29T15:58:32.385238+0000","last_peered":"2024-06-29T15:58:32.385238+0000","last_clean":"2024-06-29T15:58:32.385238+0000","last_became_active":"2024-06-29T15:58:31.403362+0000","last_became_peered":"2024-06-29T15:58:31.403362+0000","last_unstale":"2024-06-29T15:58:32.385238+0000","last_undegraded":"2024-06-29T15:58:32.385238+0000","last_fullsized":"2024-06-29T15:58:32.385238+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:12.151182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409273+0000","last_change":"2024-06-29T15:58:19.313662+0000","last_active":"2024-06-29T15:58:32.409273+0000","last_peered":"2024-06-29T15:58:32.409273+0000","last_clean":"2024-06-29T15:58:32.409273+0000","last_became_active":"2024-06-29T15:58:19.313340+0000","last_became_peered":"2024-06-29T15:58:19.313340+0000","last_unstale":"2024-06-29T15:58:32.409273+0000","last_undegraded":"2024-06-29T15:58:32.409273+0000","last_fullsized":"2024-06-29T15:58:32.409273+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:09:57.766748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470712+0000","last_change":"2024-06-29T15:58:18.306269+0000","last_active":"2024-06-29T15:58:32.470712+0000","last_peered":"2024-06-29T15:58:32.470712+0000","last_clean":"2024-06-29T15:58:32.470712+0000","last_became_active":"2024-06-29T15:58:18.305098+0000","last_became_peered":"2024-06-29T15:58:18.305098+0000","last_unstale":"2024-06-29T15:58:32.470712+0000","last_undegraded":"2024-06-29T15:58:32.470712+0000","last_fullsized":"2024-06-29T15:58:32.470712+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:20:50.375956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386134+0000","last_change":"2024-06-29T15:58:31.408521+0000","last_active":"2024-06-29T15:58:32.386134+0000","last_peered":"2024-06-29T15:58:32.386134+0000","last_clean":"2024-06-29T15:58:32.386134+0000","last_became_active":"2024-06-29T15:58:31.408199+0000","last_became_peered":"2024-06-29T15:58:31.408199+0000","last_unstale":"2024-06-29T15:58:32.386134+0000","last_undegraded":"2024-06-29T15:58:32.386134+0000","last_fullsized":"2024-06-29T15:58:32.386134+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:39.841958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470071+0000","last_change":"2024-06-29T15:58:19.308352+0000","last_active":"2024-06-29T15:58:32.470071+0000","last_peered":"2024-06-29T15:58:32.470071+0000","last_clean":"2024-06-29T15:58:32.470071+0000","last_became_active":"2024-06-29T15:58:19.308057+0000","last_became_peered":"2024-06-29T15:58:19.308057+0000","last_unstale":"2024-06-29T15:58:32.470071+0000","last_undegraded":"2024-06-29T15:58:32.470071+0000","last_fullsized":"2024-06-29T15:58:32.470071+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:51:25.510746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384172+0000","last_change":"2024-06-29T15:58:18.304832+0000","last_active":"2024-06-29T15:58:32.384172+0000","last_peered":"2024-06-29T15:58:32.384172+0000","last_clean":"2024-06-29T15:58:32.384172+0000","last_became_active":"2024-06-29T15:58:18.304230+0000","last_became_peered":"2024-06-29T15:58:18.304230+0000","last_unstale":"2024-06-29T15:58:32.384172+0000","last_undegraded":"2024-06-29T15:58:32.384172+0000","last_fullsized":"2024-06-29T15:58:32.384172+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:19:18.315156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471636+0000","last_change":"2024-06-29T15:58:31.401124+0000","last_active":"2024-06-29T15:58:32.471636+0000","last_peered":"2024-06-29T15:58:32.471636+0000","last_clean":"2024-06-29T15:58:32.471636+0000","last_became_active":"2024-06-29T15:58:31.400816+0000","last_became_peered":"2024-06-29T15:58:31.400816+0000","last_unstale":"2024-06-29T15:58:32.471636+0000","last_undegraded":"2024-06-29T15:58:32.471636+0000","last_fullsized":"2024-06-29T15:58:32.471636+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:11:50.511371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470394+0000","last_change":"2024-06-29T15:58:19.306776+0000","last_active":"2024-06-29T15:58:32.470394+0000","last_peered":"2024-06-29T15:58:32.470394+0000","last_clean":"2024-06-29T15:58:32.470394+0000","last_became_active":"2024-06-29T15:58:19.306385+0000","last_became_peered":"2024-06-29T15:58:19.306385+0000","last_unstale":"2024-06-29T15:58:32.470394+0000","last_undegraded":"2024-06-29T15:58:32.470394+0000","last_fullsized":"2024-06-29T15:58:32.470394+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:52:25.891365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385230+0000","last_change":"2024-06-29T15:58:18.297661+0000","last_active":"2024-06-29T15:58:32.385230+0000","last_peered":"2024-06-29T15:58:32.385230+0000","last_clean":"2024-06-29T15:58:32.385230+0000","last_became_active":"2024-06-29T15:58:18.297272+0000","last_became_peered":"2024-06-29T15:58:18.297272+0000","last_unstale":"2024-06-29T15:58:32.385230+0000","last_undegraded":"2024-06-29T15:58:32.385230+0000","last_fullsized":"2024-06-29T15:58:32.385230+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:17:56.872132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470256+0000","last_change":"2024-06-29T15:58:31.409558+0000","last_active":"2024-06-29T15:58:32.470256+0000","last_peered":"2024-06-29T15:58:32.470256+0000","last_clean":"2024-06-29T15:58:32.470256+0000","last_became_active":"2024-06-29T15:58:31.409241+0000","last_became_peered":"2024-06-29T15:58:31.409241+0000","last_unstale":"2024-06-29T15:58:32.470256+0000","last_undegraded":"2024-06-29T15:58:32.470256+0000","last_fullsized":"2024-06-29T15:58:32.470256+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:46:33.464520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386572+0000","last_change":"2024-06-29T15:58:19.311825+0000","last_active":"2024-06-29T15:58:32.386572+0000","last_peered":"2024-06-29T15:58:32.386572+0000","last_clean":"2024-06-29T15:58:32.386572+0000","last_became_active":"2024-06-29T15:58:19.311471+0000","last_became_peered":"2024-06-29T15:58:19.311471+0000","last_unstale":"2024-06-29T15:58:32.386572+0000","last_undegraded":"2024-06-29T15:58:32.386572+0000","last_fullsized":"2024-06-29T15:58:32.386572+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:37:52.246043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"232'2","reported_seq":24,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383482+0000","last_change":"2024-06-29T15:58:18.302543+0000","last_active":"2024-06-29T15:58:32.383482+0000","last_peered":"2024-06-29T15:58:32.383482+0000","last_clean":"2024-06-29T15:58:32.383482+0000","last_became_active":"2024-06-29T15:58:18.302237+0000","last_became_peered":"2024-06-29T15:58:18.302237+0000","last_unstale":"2024-06-29T15:58:32.383482+0000","last_undegraded":"2024-06-29T15:58:32.383482+0000","last_fullsized":"2024-06-29T15:58:32.383482+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:19:38.790806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385987+0000","last_change":"2024-06-29T15:58:31.403820+0000","last_active":"2024-06-29T15:58:32.385987+0000","last_peered":"2024-06-29T15:58:32.385987+0000","last_clean":"2024-06-29T15:58:32.385987+0000","last_became_active":"2024-06-29T15:58:31.403478+0000","last_became_peered":"2024-06-29T15:58:31.403478+0000","last_unstale":"2024-06-29T15:58:32.385987+0000","last_undegraded":"2024-06-29T15:58:32.385987+0000","last_fullsized":"2024-06-29T15:58:32.385987+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:28.906853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.409226+0000","last_change":"2024-06-29T15:58:19.312366+0000","last_active":"2024-06-29T15:58:32.409226+0000","last_peered":"2024-06-29T15:58:32.409226+0000","last_clean":"2024-06-29T15:58:32.409226+0000","last_became_active":"2024-06-29T15:58:19.309633+0000","last_became_peered":"2024-06-29T15:58:19.309633+0000","last_unstale":"2024-06-29T15:58:32.409226+0000","last_undegraded":"2024-06-29T15:58:32.409226+0000","last_fullsized":"2024-06-29T15:58:32.409226+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:11.746638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"233'1","reported_seq":23,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.404022+0000","last_change":"2024-06-29T15:58:18.303588+0000","last_active":"2024-06-29T15:58:32.404022+0000","last_peered":"2024-06-29T15:58:32.404022+0000","last_clean":"2024-06-29T15:58:32.404022+0000","last_became_active":"2024-06-29T15:58:18.303274+0000","last_became_peered":"2024-06-29T15:58:18.303274+0000","last_unstale":"2024-06-29T15:58:32.404022+0000","last_undegraded":"2024-06-29T15:58:32.404022+0000","last_fullsized":"2024-06-29T15:58:32.404022+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:03:04.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408660+0000","last_change":"2024-06-29T15:58:31.394954+0000","last_active":"2024-06-29T15:58:32.408660+0000","last_peered":"2024-06-29T15:58:32.408660+0000","last_clean":"2024-06-29T15:58:32.408660+0000","last_became_active":"2024-06-29T15:58:31.394623+0000","last_became_peered":"2024-06-29T15:58:31.394623+0000","last_unstale":"2024-06-29T15:58:32.408660+0000","last_undegraded":"2024-06-29T15:58:32.408660+0000","last_fullsized":"2024-06-29T15:58:32.408660+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:00.629657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470433+0000","last_change":"2024-06-29T15:58:19.307220+0000","last_active":"2024-06-29T15:58:32.470433+0000","last_peered":"2024-06-29T15:58:32.470433+0000","last_clean":"2024-06-29T15:58:32.470433+0000","last_became_active":"2024-06-29T15:58:19.306905+0000","last_became_peered":"2024-06-29T15:58:19.306905+0000","last_unstale":"2024-06-29T15:58:32.470433+0000","last_undegraded":"2024-06-29T15:58:32.470433+0000","last_fullsized":"2024-06-29T15:58:32.470433+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:10.851803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383919+0000","last_change":"2024-06-29T15:58:18.303622+0000","last_active":"2024-06-29T15:58:32.383919+0000","last_peered":"2024-06-29T15:58:32.383919+0000","last_clean":"2024-06-29T15:58:32.383919+0000","last_became_active":"2024-06-29T15:58:18.303319+0000","last_became_peered":"2024-06-29T15:58:18.303319+0000","last_unstale":"2024-06-29T15:58:32.383919+0000","last_undegraded":"2024-06-29T15:58:32.383919+0000","last_fullsized":"2024-06-29T15:58:32.383919+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:49:13.370498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403437+0000","last_change":"2024-06-29T15:58:31.407351+0000","last_active":"2024-06-29T15:58:32.403437+0000","last_peered":"2024-06-29T15:58:32.403437+0000","last_clean":"2024-06-29T15:58:32.403437+0000","last_became_active":"2024-06-29T15:58:31.407006+0000","last_became_peered":"2024-06-29T15:58:31.407006+0000","last_unstale":"2024-06-29T15:58:32.403437+0000","last_undegraded":"2024-06-29T15:58:32.403437+0000","last_fullsized":"2024-06-29T15:58:32.403437+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:53:31.699556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471045+0000","last_change":"2024-06-29T15:58:19.294460+0000","last_active":"2024-06-29T15:58:32.471045+0000","last_peered":"2024-06-29T15:58:32.471045+0000","last_clean":"2024-06-29T15:58:32.471045+0000","last_became_active":"2024-06-29T15:58:19.294111+0000","last_became_peered":"2024-06-29T15:58:19.294111+0000","last_unstale":"2024-06-29T15:58:32.471045+0000","last_undegraded":"2024-06-29T15:58:32.471045+0000","last_fullsized":"2024-06-29T15:58:32.471045+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:37.984851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470636+0000","last_change":"2024-06-29T15:58:18.305967+0000","last_active":"2024-06-29T15:58:32.470636+0000","last_peered":"2024-06-29T15:58:32.470636+0000","last_clean":"2024-06-29T15:58:32.470636+0000","last_became_active":"2024-06-29T15:58:18.304464+0000","last_became_peered":"2024-06-29T15:58:18.304464+0000","last_unstale":"2024-06-29T15:58:32.470636+0000","last_undegraded":"2024-06-29T15:58:32.470636+0000","last_fullsized":"2024-06-29T15:58:32.470636+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:04:37.753590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403631+0000","last_change":"2024-06-29T15:58:31.407838+0000","last_active":"2024-06-29T15:58:32.403631+0000","last_peered":"2024-06-29T15:58:32.403631+0000","last_clean":"2024-06-29T15:58:32.403631+0000","last_became_active":"2024-06-29T15:58:31.406987+0000","last_became_peered":"2024-06-29T15:58:31.406987+0000","last_unstale":"2024-06-29T15:58:32.403631+0000","last_undegraded":"2024-06-29T15:58:32.403631+0000","last_fullsized":"2024-06-29T15:58:32.403631+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:27:22.935153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385791+0000","last_change":"2024-06-29T15:58:19.311597+0000","last_active":"2024-06-29T15:58:32.385791+0000","last_peered":"2024-06-29T15:58:32.385791+0000","last_clean":"2024-06-29T15:58:32.385791+0000","last_became_active":"2024-06-29T15:58:19.311093+0000","last_became_peered":"2024-06-29T15:58:19.311093+0000","last_unstale":"2024-06-29T15:58:32.385791+0000","last_undegraded":"2024-06-29T15:58:32.385791+0000","last_fullsized":"2024-06-29T15:58:32.385791+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:11.746638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.386303+0000","last_change":"2024-06-29T15:58:18.302661+0000","last_active":"2024-06-29T15:58:32.386303+0000","last_peered":"2024-06-29T15:58:32.386303+0000","last_clean":"2024-06-29T15:58:32.386303+0000","last_became_active":"2024-06-29T15:58:18.302330+0000","last_became_peered":"2024-06-29T15:58:18.302330+0000","last_unstale":"2024-06-29T15:58:32.386303+0000","last_undegraded":"2024-06-29T15:58:32.386303+0000","last_fullsized":"2024-06-29T15:58:32.386303+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:36.982727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403701+0000","last_change":"2024-06-29T15:58:31.408494+0000","last_active":"2024-06-29T15:58:32.403701+0000","last_peered":"2024-06-29T15:58:32.403701+0000","last_clean":"2024-06-29T15:58:32.403701+0000","last_became_active":"2024-06-29T15:58:31.407493+0000","last_became_peered":"2024-06-29T15:58:31.407493+0000","last_unstale":"2024-06-29T15:58:32.403701+0000","last_undegraded":"2024-06-29T15:58:32.403701+0000","last_fullsized":"2024-06-29T15:58:32.403701+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:39:05.574092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470734+0000","last_change":"2024-06-29T15:58:19.300548+0000","last_active":"2024-06-29T15:58:32.470734+0000","last_peered":"2024-06-29T15:58:32.470734+0000","last_clean":"2024-06-29T15:58:32.470734+0000","last_became_active":"2024-06-29T15:58:19.300152+0000","last_became_peered":"2024-06-29T15:58:19.300152+0000","last_unstale":"2024-06-29T15:58:32.470734+0000","last_undegraded":"2024-06-29T15:58:32.470734+0000","last_fullsized":"2024-06-29T15:58:32.470734+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:06:09.509271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383285+0000","last_change":"2024-06-29T15:58:18.294669+0000","last_active":"2024-06-29T15:58:32.383285+0000","last_peered":"2024-06-29T15:58:32.383285+0000","last_clean":"2024-06-29T15:58:32.383285+0000","last_became_active":"2024-06-29T15:58:18.294353+0000","last_became_peered":"2024-06-29T15:58:18.294353+0000","last_unstale":"2024-06-29T15:58:32.383285+0000","last_undegraded":"2024-06-29T15:58:32.383285+0000","last_fullsized":"2024-06-29T15:58:32.383285+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:00:41.729079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.384531+0000","last_change":"2024-06-29T15:58:31.410700+0000","last_active":"2024-06-29T15:58:32.384531+0000","last_peered":"2024-06-29T15:58:32.384531+0000","last_clean":"2024-06-29T15:58:32.384531+0000","last_became_active":"2024-06-29T15:58:31.410393+0000","last_became_peered":"2024-06-29T15:58:31.410393+0000","last_unstale":"2024-06-29T15:58:32.384531+0000","last_undegraded":"2024-06-29T15:58:32.384531+0000","last_fullsized":"2024-06-29T15:58:32.384531+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:39:17.993335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470529+0000","last_change":"2024-06-29T15:58:19.303904+0000","last_active":"2024-06-29T15:58:32.470529+0000","last_peered":"2024-06-29T15:58:32.470529+0000","last_clean":"2024-06-29T15:58:32.470529+0000","last_became_active":"2024-06-29T15:58:19.303572+0000","last_became_peered":"2024-06-29T15:58:19.303572+0000","last_unstale":"2024-06-29T15:58:32.470529+0000","last_undegraded":"2024-06-29T15:58:32.470529+0000","last_fullsized":"2024-06-29T15:58:32.470529+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:49:49.141794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.470763+0000","last_change":"2024-06-29T15:58:18.298256+0000","last_active":"2024-06-29T15:58:32.470763+0000","last_peered":"2024-06-29T15:58:32.470763+0000","last_clean":"2024-06-29T15:58:32.470763+0000","last_became_active":"2024-06-29T15:58:18.297928+0000","last_became_peered":"2024-06-29T15:58:18.297928+0000","last_unstale":"2024-06-29T15:58:32.470763+0000","last_undegraded":"2024-06-29T15:58:32.470763+0000","last_fullsized":"2024-06-29T15:58:32.470763+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:12:28.560916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471197+0000","last_change":"2024-06-29T15:58:31.408208+0000","last_active":"2024-06-29T15:58:32.471197+0000","last_peered":"2024-06-29T15:58:32.471197+0000","last_clean":"2024-06-29T15:58:32.471197+0000","last_became_active":"2024-06-29T15:58:31.407838+0000","last_became_peered":"2024-06-29T15:58:31.407838+0000","last_unstale":"2024-06-29T15:58:32.471197+0000","last_undegraded":"2024-06-29T15:58:32.471197+0000","last_fullsized":"2024-06-29T15:58:32.471197+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:54:23.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.408477+0000","last_change":"2024-06-29T15:58:19.305160+0000","last_active":"2024-06-29T15:58:32.408477+0000","last_peered":"2024-06-29T15:58:32.408477+0000","last_clean":"2024-06-29T15:58:32.408477+0000","last_became_active":"2024-06-29T15:58:19.304650+0000","last_became_peered":"2024-06-29T15:58:19.304650+0000","last_unstale":"2024-06-29T15:58:32.408477+0000","last_undegraded":"2024-06-29T15:58:32.408477+0000","last_fullsized":"2024-06-29T15:58:32.408477+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:13:33.637292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":22,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.472004+0000","last_change":"2024-06-29T15:58:18.302761+0000","last_active":"2024-06-29T15:58:32.472004+0000","last_peered":"2024-06-29T15:58:32.472004+0000","last_clean":"2024-06-29T15:58:32.472004+0000","last_became_active":"2024-06-29T15:58:18.302490+0000","last_became_peered":"2024-06-29T15:58:18.302490+0000","last_unstale":"2024-06-29T15:58:32.472004+0000","last_undegraded":"2024-06-29T15:58:32.472004+0000","last_fullsized":"2024-06-29T15:58:32.472004+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:59:29.966106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":12,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.385018+0000","last_change":"2024-06-29T15:58:31.405025+0000","last_active":"2024-06-29T15:58:32.385018+0000","last_peered":"2024-06-29T15:58:32.385018+0000","last_clean":"2024-06-29T15:58:32.385018+0000","last_became_active":"2024-06-29T15:58:31.402188+0000","last_became_peered":"2024-06-29T15:58:31.402188+0000","last_unstale":"2024-06-29T15:58:32.385018+0000","last_undegraded":"2024-06-29T15:58:32.385018+0000","last_fullsized":"2024-06-29T15:58:32.385018+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:30.360251+0000","last_clean_scrub_stamp":"2024-06-29T15:58:30.360251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:04.882550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":20,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.403913+0000","last_change":"2024-06-29T15:58:19.292511+0000","last_active":"2024-06-29T15:58:32.403913+0000","last_peered":"2024-06-29T15:58:32.403913+0000","last_clean":"2024-06-29T15:58:32.403913+0000","last_became_active":"2024-06-29T15:58:19.291923+0000","last_became_peered":"2024-06-29T15:58:19.291923+0000","last_unstale":"2024-06-29T15:58:32.403913+0000","last_undegraded":"2024-06-29T15:58:32.403913+0000","last_fullsized":"2024-06-29T15:58:32.403913+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:18.276901+0000","last_clean_scrub_stamp":"2024-06-29T15:58:18.276901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:22:18.095828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"232'4","reported_seq":26,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.471901+0000","last_change":"2024-06-29T15:58:18.303249+0000","last_active":"2024-06-29T15:58:32.471901+0000","last_peered":"2024-06-29T15:58:32.471901+0000","last_clean":"2024-06-29T15:58:32.471901+0000","last_became_active":"2024-06-29T15:58:18.302916+0000","last_became_peered":"2024-06-29T15:58:18.302916+0000","last_unstale":"2024-06-29T15:58:32.471901+0000","last_undegraded":"2024-06-29T15:58:32.471901+0000","last_fullsized":"2024-06-29T15:58:32.471901+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:58:17.274776+0000","last_clean_scrub_stamp":"2024-06-29T15:58:17.274776+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:01:15.821379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"219'233","reported_seq":1133,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-06-29T15:58:32.383527+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:58:32.383527+0000","last_peered":"2024-06-29T15:58:32.383527+0000","last_clean":"2024-06-29T15:58:32.383527+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:58:32.383527+0000","last_undegraded":"2024-06-29T15:58:32.383527+0000","last_fullsized":"2024-06-29T15:58:32.383527+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3653,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1446,"num_read_kb":4103,"num_write":377,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575998,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52732,"kb_used_data":3504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319108,"statfs":{"total":96636764160,"available":96582766592,"internally_reserved":0,"allocated":3588096,"data_stored":2063604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575999,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60472,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311368,"statfs":{"total":96636764160,"available":96574840832,"internally_reserved":0,"allocated":2990080,"data_stored":1466517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575997,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46584,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94325256,"statfs":{"total":96636764160,"available":96589062144,"internally_reserved":0,"allocated":2985984,"data_stored":1473096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575998,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60508,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311332,"statfs":{"total":96636764160,"available":96574803968,"internally_reserved":0,"allocated":3026944,"data_stored":1477711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575998,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48004,"kb_used_data":2936,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323836,"statfs":{"total":96636764160,"available":96587608064,"internally_reserved":0,"allocated":3006464,"data_stored":1475872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575998,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47996,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323844,"statfs":{"total":96636764160,"available":96587616256,"internally_reserved":0,"allocated":2990080,"data_stored":1473186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575997,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60508,"kb_used_data":2952,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311332,"statfs":{"total":96636764160,"available":96574803968,"internally_reserved":0,"allocated":3022848,"data_stored":1476851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575998,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61052,"kb_used_data":3500,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310788,"statfs":{"total":96636764160,"available":96574246912,"internally_reserved":0,"allocated":3584000,"data_stored":2063498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:58:35.813 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:58:35.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.958+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/2158234436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa406f680 msgr2=0x7f4aa406c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.958+0000 7f4aaaa9e640 1 --2- 172.21.15.81:0/2158234436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa406f680 0x7f4aa406c2f0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4072d60 tx=0x7f4a9402a560 comp rx=0 tx=0).stop 2024-06-29T15:58:35.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.958+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/2158234436 shutdown_connections 2024-06-29T15:58:35.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.958+0000 7f4aaaa9e640 1 --2- 172.21.15.81:0/2158234436 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4aa406fa90 0x7f4aa406c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.958+0000 7f4aaaa9e640 1 --2- 172.21.15.81:0/2158234436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa406f680 0x7f4aa406c2f0 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.958+0000 7f4aaaa9e640 1 --2- 172.21.15.81:0/2158234436 >> v2:172.21.15.167:3300/0 conn(0x7f4aa4074520 0x7f4aa406f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.958+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/2158234436 >> 172.21.15.81:0/2158234436 conn(0x7f4aa4072950 msgr2=0x7f4aa40b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:35.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.958+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/2158234436 shutdown_connections 2024-06-29T15:58:35.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.958+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/2158234436 wait complete. 2024-06-29T15:58:35.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.960+0000 7f4aaaa9e640 1 Processor -- start 2024-06-29T15:58:35.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.960+0000 7f4aaaa9e640 1 -- start start 2024-06-29T15:58:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4aaaa9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa406f680 0x7f4aa409d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4aaaa9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4aa406fa90 0x7f4aa40d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4aaaa9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4aa4074520 0x7f4aa4155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4aaaa9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa406cd70 con 0x7f4aa406f680 2024-06-29T15:58:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4aaaa9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa415b220 con 0x7f4aa4074520 2024-06-29T15:58:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4aaaa9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa406cef0 con 0x7f4aa406fa90 2024-06-29T15:58:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4aa883a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa406f680 0x7f4aa409d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4aa883a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa406f680 0x7f4aa409d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33280/0 (socket says 172.21.15.81:33280) 2024-06-29T15:58:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4aa883a640 1 -- 172.21.15.81:0/3978350653 learned_addr learned my addr 172.21.15.81:0/3978350653 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4aa903b640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4aa4074520 0x7f4aa4155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.961+0000 7f4a9bfff640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4aa406fa90 0x7f4aa40d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4aa883a640 1 -- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4aa406fa90 msgr2=0x7f4aa40d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4aa883a640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4aa406fa90 0x7f4aa40d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4aa883a640 1 -- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4aa4074520 msgr2=0x7f4aa4155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4aa883a640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4aa4074520 0x7f4aa4155390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4aa883a640 1 -- 172.21.15.81:0/3978350653 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa41f5ad0 con 0x7f4aa406f680 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4aa903b640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4aa4074520 0x7f4aa4155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4aa883a640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa406f680 0x7f4aa409d8a0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000d740 tx=0x7f4aa000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4a997fa640 1 -- 172.21.15.81:0/3978350653 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0017070 con 0x7f4aa406f680 2024-06-29T15:58:35.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4a997fa640 1 -- 172.21.15.81:0/3978350653 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4aa0009d20 con 0x7f4aa406f680 2024-06-29T15:58:35.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/3978350653 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa41f4d50 con 0x7f4aa406f680 2024-06-29T15:58:35.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4a997fa640 1 -- 172.21.15.81:0/3978350653 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00106c0 con 0x7f4aa406f680 2024-06-29T15:58:35.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.962+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/3978350653 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa41f52d0 con 0x7f4aa406f680 2024-06-29T15:58:35.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.963+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/3978350653 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa406f140 con 0x7f4aa406f680 2024-06-29T15:58:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.964+0000 7f4a997fa640 1 -- 172.21.15.81:0/3978350653 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f4aa001a070 con 0x7f4aa406f680 2024-06-29T15:58:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.964+0000 7f4a997fa640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4a90076540 0x7f4a90078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.964+0000 7f4a997fa640 1 -- 172.21.15.81:0/3978350653 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7f4aa0098220 con 0x7f4aa406f680 2024-06-29T15:58:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.964+0000 7f4a9bfff640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4a90076540 0x7f4a90078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.965+0000 7f4a9bfff640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4a90076540 0x7f4a90078a50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4a94033000 tx=0x7f4a94004730 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:35.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:35.968+0000 7f4a997fa640 1 -- 172.21.15.81:0/3978350653 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4aa00612a0 con 0x7f4aa406f680 2024-06-29T15:58:36.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.186+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/3978350653 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4aa40aee20 con 0x7f4aa406f680 2024-06-29T15:58:36.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.187+0000 7f4a997fa640 1 -- 172.21.15.81:0/3978350653 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6562 (secure 0 0 0) 0x7f4aa0060ba0 con 0x7f4aa406f680 2024-06-29T15:58:36.188 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:36.188 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":135,"btime":"2024-06-29T15:58:34:407667+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:18.654579+0000","modified":"2024-06-29T15:58:21.679033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12105":{"gid":12105,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/30158598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":30158598},{"type":"v1","addr":"172.21.15.167:6837","nonce":30158598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12105,"qdb_cluster":[12105]},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:30.737083+0000","modified":"2024-06-29T15:58:33.773132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12093":{"gid":12093,"name":"a","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12093,"qdb_cluster":[12093]},"id":16}]} 2024-06-29T15:58:36.189 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 135 2024-06-29T15:58:36.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4a90076540 msgr2=0x7f4a90078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4a90076540 0x7f4a90078a50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4a94033000 tx=0x7f4a94004730 comp rx=0 tx=0).stop 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/3978350653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa406f680 msgr2=0x7f4aa409d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa406f680 0x7f4aa409d8a0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000d740 tx=0x7f4aa000dc00 comp rx=0 tx=0).stop 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/3978350653 shutdown_connections 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4a90076540 0x7f4a90078a50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4aa4074520 0x7f4aa4155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4aa406fa90 0x7f4aa40d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 --2- 172.21.15.81:0/3978350653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa406f680 0x7f4aa409d8a0 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/3978350653 >> 172.21.15.81:0/3978350653 conn(0x7f4aa4072950 msgr2=0x7f4aa40b0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/3978350653 shutdown_connections 2024-06-29T15:58:36.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.189+0000 7f4aaaa9e640 1 -- 172.21.15.81:0/3978350653 wait complete. 2024-06-29T15:58:36.260 INFO:tasks.ceph.mds.a.smithi081.stderr:2024-06-29T15:58:36.259+0000 7f2fbd9a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:36.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 135, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:58:30.737083+0000', 'modified': '2024-06-29T15:58:33.773132+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12093}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12093': {'gid': 12093, 'name': 'a', 'rank': 0, 'incarnation': 131, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.81:6835/3228546258', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6833', 'nonce': 3228546258}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 3228546258}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12093, 'qdb_cluster': [12093]} 2024-06-29T15:58:36.605 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:58:36.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T15:58:36.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.721+0000 7f404757a640 1 -- 172.21.15.81:0/1697611030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40401412f0 msgr2=0x7f40400cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.721+0000 7f404757a640 1 --2- 172.21.15.81:0/1697611030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40401412f0 0x7f40400cef90 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f403c005fd0 tx=0x7f403c02aa00 comp rx=0 tx=0).stop 2024-06-29T15:58:36.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.721+0000 7f404757a640 1 -- 172.21.15.81:0/1697611030 shutdown_connections 2024-06-29T15:58:36.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.721+0000 7f404757a640 1 --2- 172.21.15.81:0/1697611030 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40400cf4d0 0x7f4040073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.721+0000 7f404757a640 1 --2- 172.21.15.81:0/1697611030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40401412f0 0x7f40400cef90 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.721+0000 7f404757a640 1 -- 172.21.15.81:0/1697611030 >> 172.21.15.81:0/1697611030 conn(0x7f404006ccc0 msgr2=0x7f40400b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:36.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.721+0000 7f404757a640 1 -- 172.21.15.81:0/1697611030 shutdown_connections 2024-06-29T15:58:36.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.721+0000 7f404757a640 1 -- 172.21.15.81:0/1697611030 wait complete. 2024-06-29T15:58:36.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.723+0000 7f404757a640 1 Processor -- start 2024-06-29T15:58:36.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.723+0000 7f404757a640 1 -- start start 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f404757a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4040073570 0x7f40401f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f404757a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40400cf4d0 0x7f40401eeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f404757a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40401412f0 0x7f40400c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f404757a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f40400b3210 con 0x7f4040073570 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f404757a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f404015f280 con 0x7f40400cf4d0 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f404757a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f40400b3390 con 0x7f40401412f0 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f4045316640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4040073570 0x7f40401f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f4045316640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4040073570 0x7f40401f5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33312/0 (socket says 172.21.15.81:33312) 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f4045316640 1 -- 172.21.15.81:0/1088700365 learned_addr learned my addr 172.21.15.81:0/1088700365 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f4045b17640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40401412f0 0x7f40400c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.724+0000 7f4044b15640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40400cf4d0 0x7f40401eeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f4045316640 1 -- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40401412f0 msgr2=0x7f40400c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f4045316640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40401412f0 0x7f40400c4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f4045316640 1 -- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40400cf4d0 msgr2=0x7f40401eeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f4045316640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40400cf4d0 0x7f40401eeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f4045316640 1 -- 172.21.15.81:0/1088700365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40401f8cd0 con 0x7f4040073570 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f4044b15640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40400cf4d0 0x7f40401eeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f4045316640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4040073570 0x7f40401f5af0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f403c000f80 tx=0x7f403c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f4035ffb640 1 -- 172.21.15.81:0/1088700365 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c046070 con 0x7f4040073570 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f4035ffb640 1 -- 172.21.15.81:0/1088700365 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f403c0045c0 con 0x7f4040073570 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f404757a640 1 -- 172.21.15.81:0/1088700365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40401f8fb0 con 0x7f4040073570 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f4035ffb640 1 -- 172.21.15.81:0/1088700365 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c033070 con 0x7f4040073570 2024-06-29T15:58:36.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.725+0000 7f404757a640 1 -- 172.21.15.81:0/1088700365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40400c4ad0 con 0x7f4040073570 2024-06-29T15:58:36.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.727+0000 7f4035ffb640 1 -- 172.21.15.81:0/1088700365 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7f403c049070 con 0x7f4040073570 2024-06-29T15:58:36.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.727+0000 7f4035ffb640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4024076540 0x7f4024078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.728+0000 7f4035ffb640 1 -- 172.21.15.81:0/1088700365 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7f403c0b9180 con 0x7f4040073570 2024-06-29T15:58:36.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.728+0000 7f4044b15640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4024076540 0x7f4024078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.728+0000 7f404e43b740 1 Processor -- start 2024-06-29T15:58:36.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.728+0000 7f4044b15640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4024076540 0x7f4024078a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40400c6460 tx=0x7f4030005cb0 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:36.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.728+0000 7f404e43b740 1 -- start start 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f404e43b740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db6a270 0x55f51df566e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f404e43b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f404e43b740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db49590 0x55f51df3a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f404e43b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f51ded2bc0 con 0x55f51db413b0 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f404e43b740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55f51de81630 con 0x55f51db6a270 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f404e43b740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55f51de70bd0 con 0x55f51db49590 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4044b15640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4044b15640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33320/0 (socket says 172.21.15.81:33320) 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4044b15640 1 -- 172.21.15.81:0/3393546394 learned_addr learned my addr 172.21.15.81:0/3393546394 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4045316640 1 --2- 172.21.15.81:0/3393546394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db6a270 0x55f51df566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4045b17640 1 --2- 172.21.15.81:0/3393546394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db49590 0x55f51df3a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4044b15640 1 -- 172.21.15.81:0/3393546394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db49590 msgr2=0x55f51df3a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:36.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4044b15640 1 --2- 172.21.15.81:0/3393546394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db49590 0x55f51df3a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4044b15640 1 -- 172.21.15.81:0/3393546394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db6a270 msgr2=0x55f51df566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:36.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4044b15640 1 --2- 172.21.15.81:0/3393546394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db6a270 0x55f51df566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4044b15640 1 -- 172.21.15.81:0/3393546394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51de6dca0 con 0x55f51db413b0 2024-06-29T15:58:36.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f4044b15640 1 --2- 172.21.15.81:0/3393546394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a3a0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f403000de10 tx=0x7f4030010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17705161c01d0606 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:36.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f402affd640 1 -- 172.21.15.81:0/3393546394 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030020070 con 0x55f51db413b0 2024-06-29T15:58:36.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f402affd640 1 -- 172.21.15.81:0/3393546394 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f403001c660 con 0x55f51db413b0 2024-06-29T15:58:36.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.730+0000 7f402affd640 1 -- 172.21.15.81:0/3393546394 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403001ca50 con 0x55f51db413b0 2024-06-29T15:58:36.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.731+0000 7f404e43b740 1 -- 172.21.15.81:0/3393546394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 msgr2=0x55f51df3a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.731+0000 7f404e43b740 1 --2- 172.21.15.81:0/3393546394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a3a0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f403000de10 tx=0x7f4030010000 comp rx=0 tx=0).stop 2024-06-29T15:58:36.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.731+0000 7f404e43b740 1 -- 172.21.15.81:0/3393546394 shutdown_connections 2024-06-29T15:58:36.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.731+0000 7f404e43b740 1 --2- 172.21.15.81:0/3393546394 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db49590 0x55f51df3a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.731+0000 7f404e43b740 1 --2- 172.21.15.81:0/3393546394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a3a0 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.731+0000 7f404e43b740 1 --2- 172.21.15.81:0/3393546394 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db6a270 0x55f51df566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.731+0000 7f404e43b740 1 -- 172.21.15.81:0/3393546394 >> 172.21.15.81:0/3393546394 conn(0x55f51db5e3c0 msgr2=0x55f51df0df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:36.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.731+0000 7f404e43b740 1 -- 172.21.15.81:0/3393546394 shutdown_connections 2024-06-29T15:58:36.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.731+0000 7f404e43b740 1 -- 172.21.15.81:0/3393546394 wait complete. 2024-06-29T15:58:36.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.732+0000 7f404e43b740 1 -- start start 2024-06-29T15:58:36.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.732+0000 7f404e43b740 1 Processor -- start 2024-06-29T15:58:36.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f404e43b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f404e43b740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db49590 0x55f51df3a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f404e43b740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db6a270 0x55f51df64dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f404e43b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f51de0eae0 con 0x55f51db413b0 2024-06-29T15:58:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f404e43b740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55f51de74c10 con 0x55f51db49590 2024-06-29T15:58:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f404e43b740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55f51dbe4620 con 0x55f51db6a270 2024-06-29T15:58:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f404e43b740 10 client.0 _ll_register_callbacks cb 0x55f51de1b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:58:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045316640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045316640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33336/0 (socket says 172.21.15.81:33336) 2024-06-29T15:58:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045316640 1 -- 172.21.15.81:0/1638759887 learned_addr learned my addr 172.21.15.81:0/1638759887 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4044b15640 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db49590 0x55f51df3a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045b17640 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db6a270 0x55f51df64dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045316640 1 -- 172.21.15.81:0/1638759887 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db6a270 msgr2=0x55f51df64dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045316640 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db6a270 0x55f51df64dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045316640 1 -- 172.21.15.81:0/1638759887 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db49590 msgr2=0x55f51df3a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045316640 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db49590 0x55f51df3a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045316640 1 -- 172.21.15.81:0/1638759887 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f51dbd5850 con 0x55f51db413b0 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045b17640 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db6a270 0x55f51df64dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f4045316640 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a0d0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f403c07ee00 tx=0x7f403c082160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199581ca66a27fee server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f401ffff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f401ffff640 1 -- 172.21.15.81:0/1638759887 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c081b90 con 0x55f51db413b0 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f401ffff640 1 -- 172.21.15.81:0/1638759887 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7f403c004d50 con 0x55f51db413b0 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f404e43b740 1 -- 172.21.15.81:0/1638759887 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f403001c660 con 0x55f51db413b0 2024-06-29T15:58:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f401ffff640 1 -- 172.21.15.81:0/1638759887 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7f403c0b6050 con 0x55f51db413b0 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f404e43b740 10 client.12387 fetch_fsmap learned FSMap version 135 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.734+0000 7f404e43b740 1 -- 172.21.15.81:0/1638759887 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=2+}) -- 0x55f51dc4c6b0 con 0x55f51db413b0 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f401ffff640 1 -- 172.21.15.81:0/1638759887 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f403001c660 con 0x55f51db413b0 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f401ffff640 1 -- 172.21.15.81:0/1638759887 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 135) ==== 4149+0+0 (secure 0 0 0) 0x7f403c09c070 con 0x55f51db413b0 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f404e43b740 10 client.12387 fetch_fsmap finished waiting for FSMap version 135 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f404e43b740 10 client.12387 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f404e43b740 20 client.12387 populate_metadata read hostname 'smithi081' 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f404e43b740 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.81:6833/3228546258,v1:172.21.15.81:6835/3228546258] conn(0x55f51db28e40 0x55f51df534b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f404e43b740 4 client.12387 mds_command: new command op to 12093 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f404e43b740 1 -- 172.21.15.81:0/1638759887 --> [v2:172.21.15.81:6833/3228546258,v1:172.21.15.81:6835/3228546258] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f51de83580 con 0x55f51db28e40 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f4044b15640 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.81:6833/3228546258,v1:172.21.15.81:6835/3228546258] conn(0x55f51db28e40 0x55f51df534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f401ffff640 10 client.12387 ms_handle_connect on v2:172.21.15.81:6833/3228546258 2024-06-29T15:58:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.735+0000 7f4044b15640 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.81:6833/3228546258,v1:172.21.15.81:6835/3228546258] conn(0x55f51db28e40 0x55f51df534b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b8e035962c484c2 server_cookie=99529f5e9971d5a1 in_seq=0 out_seq=0 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f401ffff640 1 -- 172.21.15.81:0/1638759887 <== mds.0 v2:172.21.15.81:6833/3228546258 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55f51de83580 con 0x55f51db28e40 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f401ffff640 10 client.12387 handle_command_reply: tid=0 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 client.12387 shutdown 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 -- 172.21.15.81:0/1638759887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 msgr2=0x55f51df3a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a0d0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f403c07ee00 tx=0x7f403c082160 comp rx=0 tx=0).stop 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 -- 172.21.15.81:0/1638759887 shutdown_connections 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.81:6833/3228546258,v1:172.21.15.81:6835/3228546258] conn(0x55f51db28e40 0x55f51df534b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db6a270 0x55f51df64dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db49590 0x55f51df3a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f401ffff640 0 client.12387 ms_handle_reset on v2:172.21.15.81:6833/3228546258 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 --2- 172.21.15.81:0/1638759887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df3a0d0 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 -- 172.21.15.81:0/1638759887 >> 172.21.15.81:0/1638759887 conn(0x55f51db5e3c0 msgr2=0x55f51df09b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 -- 172.21.15.81:0/1638759887 shutdown_connections 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 1 -- 172.21.15.81:0/1638759887 wait complete. 2024-06-29T15:58:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.736+0000 7f404e43b740 20 client.12387 trim_cache size 0 max 16384 2024-06-29T15:58:36.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.755+0000 7f404e43b740 1 Processor -- start 2024-06-29T15:58:36.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.755+0000 7f404e43b740 1 -- start start 2024-06-29T15:58:36.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f404e43b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df64820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f404e43b740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db49590 0x55f51df09b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f404e43b740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db6a270 0x55f51df0a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f404e43b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4040204980 con 0x55f51db413b0 2024-06-29T15:58:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f404e43b740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f40400c7660 con 0x55f51db49590 2024-06-29T15:58:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f404e43b740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f40400c7cb0 con 0x55f51db6a270 2024-06-29T15:58:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f4045316640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df64820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f4045316640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df64820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33344/0 (socket says 172.21.15.81:33344) 2024-06-29T15:58:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f4045316640 1 -- 172.21.15.81:0/4092503837 learned_addr learned my addr 172.21.15.81:0/4092503837 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f4045316640 1 -- 172.21.15.81:0/4092503837 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db6a270 msgr2=0x55f51df0a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f4045316640 1 --2- 172.21.15.81:0/4092503837 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db6a270 0x55f51df0a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f4045316640 1 -- 172.21.15.81:0/4092503837 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db49590 msgr2=0x55f51df09b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f4044b15640 1 --2- 172.21.15.81:0/4092503837 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db49590 0x55f51df09b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f4045316640 1 --2- 172.21.15.81:0/4092503837 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db49590 0x55f51df09b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f4045316640 1 -- 172.21.15.81:0/4092503837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51df06b20 con 0x55f51db413b0 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f4045316640 1 --2- 172.21.15.81:0/4092503837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df64820 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f403c0819b0 tx=0x7f403c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b057b259dd18ad server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f402a7fc640 1 -- 172.21.15.81:0/4092503837 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c081b90 con 0x55f51db413b0 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f402a7fc640 1 -- 172.21.15.81:0/4092503837 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f403c08a960 con 0x55f51db413b0 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.757+0000 7f402a7fc640 1 -- 172.21.15.81:0/4092503837 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c080df0 con 0x55f51db413b0 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.758+0000 7f404e43b740 1 -- 172.21.15.81:0/4092503837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 msgr2=0x55f51df64820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.758+0000 7f404e43b740 1 --2- 172.21.15.81:0/4092503837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df64820 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f403c0819b0 tx=0x7f403c005fd0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.758+0000 7f404e43b740 1 -- 172.21.15.81:0/4092503837 shutdown_connections 2024-06-29T15:58:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.758+0000 7f404e43b740 1 --2- 172.21.15.81:0/4092503837 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db6a270 0x55f51df0a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.758+0000 7f404e43b740 1 --2- 172.21.15.81:0/4092503837 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db49590 0x55f51df09b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.758+0000 7f404e43b740 1 --2- 172.21.15.81:0/4092503837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df64820 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.758+0000 7f404e43b740 1 -- 172.21.15.81:0/4092503837 >> 172.21.15.81:0/4092503837 conn(0x55f51db5e3c0 msgr2=0x55f51df32470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:36.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.758+0000 7f404e43b740 1 -- 172.21.15.81:0/4092503837 shutdown_connections 2024-06-29T15:58:36.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.758+0000 7f404e43b740 1 -- 172.21.15.81:0/4092503837 wait complete. 2024-06-29T15:58:36.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.759+0000 7f404e43b740 1 -- start start 2024-06-29T15:58:36.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.759+0000 7f404e43b740 1 Processor -- start 2024-06-29T15:58:36.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.760+0000 7f404e43b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df53ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.760+0000 7f404e43b740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db49590 0x55f51dfd3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.760+0000 7f404e43b740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db6a270 0x55f51df09b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.760+0000 7f404e43b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f51df657d0 con 0x55f51db413b0 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.760+0000 7f404e43b740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55f51df37da0 con 0x55f51db6a270 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.760+0000 7f404e43b740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55f51df687a0 con 0x55f51db49590 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.760+0000 7f404e43b740 10 client.0 _ll_register_callbacks cb 0x7f403c07db20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.760+0000 7f4045316640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df53ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f4045316640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df53ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33348/0 (socket says 172.21.15.81:33348) 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f4045316640 1 -- 172.21.15.81:0/2161903824 learned_addr learned my addr 172.21.15.81:0/2161903824 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f4045316640 1 -- 172.21.15.81:0/2161903824 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db49590 msgr2=0x55f51dfd3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f4045316640 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db49590 0x55f51dfd3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f4045316640 1 -- 172.21.15.81:0/2161903824 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db6a270 msgr2=0x55f51df09b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:58:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f4045316640 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db6a270 0x55f51df09b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f4045316640 1 -- 172.21.15.81:0/2161903824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f51df54680 con 0x55f51db413b0 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f4045316640 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df53ca0 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f403c081b20 tx=0x7f403c004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad9837eb12f4cb server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f401ffff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f401ffff640 1 -- 172.21.15.81:0/2161903824 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c0a3060 con 0x55f51db413b0 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f401ffff640 1 -- 172.21.15.81:0/2161903824 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7f403c0a3200 con 0x55f51db413b0 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f404e43b740 1 -- 172.21.15.81:0/2161903824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f403c08a960 con 0x55f51db413b0 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f401ffff640 1 -- 172.21.15.81:0/2161903824 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7f403c08f050 con 0x55f51db413b0 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f404e43b740 10 client.12393 fetch_fsmap learned FSMap version 135 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f404e43b740 1 -- 172.21.15.81:0/2161903824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=2+}) -- 0x55f51df54680 con 0x55f51db413b0 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f401ffff640 1 -- 172.21.15.81:0/2161903824 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f403c08a960 con 0x55f51db413b0 2024-06-29T15:58:36.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.761+0000 7f401ffff640 1 -- 172.21.15.81:0/2161903824 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 135) ==== 4149+0+0 (secure 0 0 0) 0x7f403c092020 con 0x55f51db413b0 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 10 client.12393 fetch_fsmap finished waiting for FSMap version 135 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 10 client.12393 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 20 client.12393 populate_metadata read hostname 'smithi081' 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.81:6833/3228546258,v1:172.21.15.81:6835/3228546258] conn(0x55f51dfb9180 0x55f51df0a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 4 client.12393 mds_command: new command op to 12093 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 -- 172.21.15.81:0/2161903824 --> [v2:172.21.15.81:6833/3228546258,v1:172.21.15.81:6835/3228546258] -- command(tid 0: {"prefix": "status"}) -- 0x55f51de7f020 con 0x55f51dfb9180 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f4044b15640 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.81:6833/3228546258,v1:172.21.15.81:6835/3228546258] conn(0x55f51dfb9180 0x55f51df0a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f401ffff640 10 client.12393 ms_handle_connect on v2:172.21.15.81:6833/3228546258 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f4044b15640 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.81:6833/3228546258,v1:172.21.15.81:6835/3228546258] conn(0x55f51dfb9180 0x55f51df0a080 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d09078f5b7abfde3 server_cookie=c7556fdb56f2c342 in_seq=0 out_seq=0 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f401ffff640 1 -- 172.21.15.81:0/2161903824 <== mds.0 v2:172.21.15.81:6833/3228546258 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55f51de7f020 con 0x55f51dfb9180 2024-06-29T15:58:36.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f401ffff640 10 client.12393 handle_command_reply: tid=0 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 client.12393 shutdown 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 -- 172.21.15.81:0/2161903824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 msgr2=0x55f51df53ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df53ca0 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f403c081b20 tx=0x7f403c004f80 comp rx=0 tx=0).stop 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 -- 172.21.15.81:0/2161903824 shutdown_connections 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.81:6833/3228546258,v1:172.21.15.81:6835/3228546258] conn(0x55f51dfb9180 0x55f51df0a080 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55f51db6a270 0x55f51df09b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f401ffff640 0 client.12393 ms_handle_reset on v2:172.21.15.81:6833/3228546258 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55f51db49590 0x55f51dfd3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 --2- 172.21.15.81:0/2161903824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f51db413b0 0x55f51df53ca0 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.762+0000 7f404e43b740 1 -- 172.21.15.81:0/2161903824 >> 172.21.15.81:0/2161903824 conn(0x55f51db5e3c0 msgr2=0x55f51df646a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404e43b740 1 -- 172.21.15.81:0/2161903824 shutdown_connections 2024-06-29T15:58:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404e43b740 1 -- 172.21.15.81:0/2161903824 wait complete. 2024-06-29T15:58:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404e43b740 20 client.12393 trim_cache size 0 max 16384 2024-06-29T15:58:36.771 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:58:36.771 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T15:58:36.771 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T15:58:36.771 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 12093, 2024-06-29T15:58:36.771 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T15:58:36.771 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T15:58:36.772 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "backup_fs", 2024-06-29T15:58:36.772 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.370254697, 2024-06-29T15:58:36.772 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 135, 2024-06-29T15:58:36.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 237, 2024-06-29T15:58:36.772 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 237, 2024-06-29T15:58:36.772 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 22.165596228999998 2024-06-29T15:58:36.772 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:58:36.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 -- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4024076540 msgr2=0x7f4024078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4024076540 0x7f4024078a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40400c6460 tx=0x7f4030005cb0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 -- 172.21.15.81:0/1088700365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4040073570 msgr2=0x7f40401f5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4040073570 0x7f40401f5af0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f403c000f80 tx=0x7f403c003f90 comp rx=0 tx=0).stop 2024-06-29T15:58:36.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 -- 172.21.15.81:0/1088700365 shutdown_connections 2024-06-29T15:58:36.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f4024076540 0x7f4024078a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f40401412f0 0x7f40400c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f40400cf4d0 0x7f40401eeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 --2- 172.21.15.81:0/1088700365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4040073570 0x7f40401f5af0 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 -- 172.21.15.81:0/1088700365 >> 172.21.15.81:0/1088700365 conn(0x7f404006ccc0 msgr2=0x7f404015c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:36.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 -- 172.21.15.81:0/1088700365 shutdown_connections 2024-06-29T15:58:36.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.763+0000 7f404757a640 1 -- 172.21.15.81:0/1088700365 wait complete. 2024-06-29T15:58:36.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:58:36.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.889+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/2279251139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfb406f680 msgr2=0x7fdfb406c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.889+0000 7fdfbb7ba640 1 --2- 172.21.15.81:0/2279251139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfb406f680 0x7fdfb406c2f0 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4072d60 tx=0x7fdfa402a560 comp rx=0 tx=0).stop 2024-06-29T15:58:36.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.889+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/2279251139 shutdown_connections 2024-06-29T15:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.889+0000 7fdfbb7ba640 1 --2- 172.21.15.81:0/2279251139 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdfb406fa90 0x7fdfb406c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.889+0000 7fdfbb7ba640 1 --2- 172.21.15.81:0/2279251139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfb406f680 0x7fdfb406c2f0 unknown :-1 s=CLOSED pgs=2590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.889+0000 7fdfbb7ba640 1 --2- 172.21.15.81:0/2279251139 >> v2:172.21.15.167:3300/0 conn(0x7fdfb4074520 0x7fdfb406f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.889+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/2279251139 >> 172.21.15.81:0/2279251139 conn(0x7fdfb4072950 msgr2=0x7fdfb40b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.889+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/2279251139 shutdown_connections 2024-06-29T15:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.889+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/2279251139 wait complete. 2024-06-29T15:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.890+0000 7fdfbb7ba640 1 Processor -- start 2024-06-29T15:58:36.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.890+0000 7fdfbb7ba640 1 -- start start 2024-06-29T15:58:36.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfbb7ba640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdfb406f680 0x7fdfb4155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfbb7ba640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdfb406fa90 0x7fdfb40cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfbb7ba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfb4074520 0x7fdfb40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfbb7ba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb406cd70 con 0x7fdfb4074520 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfbb7ba640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb415b220 con 0x7fdfb406f680 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfbb7ba640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb406cef0 con 0x7fdfb406fa90 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb9d57640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfb4074520 0x7fdfb40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb9d57640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfb4074520 0x7fdfb40cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33364/0 (socket says 172.21.15.81:33364) 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb9d57640 1 -- 172.21.15.81:0/1581151033 learned_addr learned my addr 172.21.15.81:0/1581151033 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb8d55640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdfb406fa90 0x7fdfb40cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb9556640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdfb406f680 0x7fdfb4155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb9d57640 1 -- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdfb406fa90 msgr2=0x7fdfb40cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb9d57640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdfb406fa90 0x7fdfb40cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb9d57640 1 -- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdfb406f680 msgr2=0x7fdfb4155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb9d57640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdfb406f680 0x7fdfb4155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb9d57640 1 -- 172.21.15.81:0/1581151033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb414fea0 con 0x7fdfb4074520 2024-06-29T15:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfb9d57640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfb4074520 0x7fdfb40cef60 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7fdfac007b50 tx=0x7fdfac007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.892+0000 7fdfa9ffb640 1 -- 172.21.15.81:0/1581151033 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac019020 con 0x7fdfb4074520 2024-06-29T15:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.893+0000 7fdfa9ffb640 1 -- 172.21.15.81:0/1581151033 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdfac00c2e0 con 0x7fdfb4074520 2024-06-29T15:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.893+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/1581151033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb4150180 con 0x7fdfb4074520 2024-06-29T15:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.893+0000 7fdfa9ffb640 1 -- 172.21.15.81:0/1581151033 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac004d80 con 0x7fdfb4074520 2024-06-29T15:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.893+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/1581151033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb4150740 con 0x7fdfb4074520 2024-06-29T15:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.894+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/1581151033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb4074260 con 0x7fdfb4074520 2024-06-29T15:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.894+0000 7fdfa9ffb640 1 -- 172.21.15.81:0/1581151033 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fdfac00e750 con 0x7fdfb4074520 2024-06-29T15:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.895+0000 7fdfa9ffb640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf98076540 0x7fdf98078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.895+0000 7fdfa9ffb640 1 -- 172.21.15.81:0/1581151033 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7fdfac099e40 con 0x7fdfb4074520 2024-06-29T15:58:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.895+0000 7fdfb9556640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf98076540 0x7fdf98078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.895+0000 7fdfb9556640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf98076540 0x7fdf98078a50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0005d70 tx=0x7fdfb0005d00 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:36.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:36.899+0000 7fdfa9ffb640 1 -- 172.21.15.81:0/1581151033 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdfb4074260 con 0x7fdfb4074520 2024-06-29T15:58:37.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.104+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/1581151033 --> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fdfb40b55e0 con 0x7fdf98076540 2024-06-29T15:58:37.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.106+0000 7fdfa9ffb640 1 -- 172.21.15.81:0/1581151033 <== mgr.9155 v2:172.21.15.167:6832/3661607921 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+82 (secure 0 0 0) 0x7fdfb40b55e0 con 0x7fdf98076540 2024-06-29T15:58:37.106 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:58:37.106 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:37.106 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T15:58:37.107 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:58:37.107 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:58:37.107 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "backup_fs" 2024-06-29T15:58:37.107 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:58:37.107 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:58:37.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf98076540 msgr2=0x7fdf98078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:37.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf98076540 0x7fdf98078a50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0005d70 tx=0x7fdfb0005d00 comp rx=0 tx=0).stop 2024-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/1581151033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfb4074520 msgr2=0x7fdfb40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfb4074520 0x7fdfb40cef60 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7fdfac007b50 tx=0x7fdfac007f50 comp rx=0 tx=0).stop 2024-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/1581151033 shutdown_connections 2024-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fdf98076540 0x7fdf98078a50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfb4074520 0x7fdfb40cef60 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdfb406fa90 0x7fdfb40cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 --2- 172.21.15.81:0/1581151033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdfb406f680 0x7fdfb4155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/1581151033 >> 172.21.15.81:0/1581151033 conn(0x7fdfb4072950 msgr2=0x7fdfb4156300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/1581151033 shutdown_connections 2024-06-29T15:58:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.108+0000 7fdfbb7ba640 1 -- 172.21.15.81:0/1581151033 wait complete. 2024-06-29T15:58:37.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:58:37.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.234+0000 7fe081426640 1 -- 172.21.15.81:0/3403172581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe07c0d2a90 msgr2=0x7fe07c0d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:37.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.234+0000 7fe081426640 1 --2- 172.21.15.81:0/3403172581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe07c0d2a90 0x7fe07c0d09c0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7fe074009830 tx=0x7fe07402a5c0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.235+0000 7fe081426640 1 -- 172.21.15.81:0/3403172581 shutdown_connections 2024-06-29T15:58:37.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.235+0000 7fe081426640 1 --2- 172.21.15.81:0/3403172581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe07c0d2a90 0x7fe07c0d09c0 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.235+0000 7fe081426640 1 --2- 172.21.15.81:0/3403172581 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe07c0b9040 0x7fe07c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.235+0000 7fe081426640 1 -- 172.21.15.81:0/3403172581 >> 172.21.15.81:0/3403172581 conn(0x7fe07c09cf30 msgr2=0x7fe07c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:37.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.235+0000 7fe081426640 1 -- 172.21.15.81:0/3403172581 shutdown_connections 2024-06-29T15:58:37.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.235+0000 7fe081426640 1 -- 172.21.15.81:0/3403172581 wait complete. 2024-06-29T15:58:37.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.236+0000 7fe081426640 1 Processor -- start 2024-06-29T15:58:37.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.236+0000 7fe081426640 1 -- start start 2024-06-29T15:58:37.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe081426640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe07c0b5250 0x7fe07c1f8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:37.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe081426640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe07c0b9040 0x7fe07c0b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:37.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe081426640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe07c0d2a90 0x7fe07c0b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:37.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe081426640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c0b7d30 con 0x7fe07c0b9040 2024-06-29T15:58:37.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe081426640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c0b7bb0 con 0x7fe07c0d2a90 2024-06-29T15:58:37.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe081426640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe07c0b7eb0 con 0x7fe07c0b5250 2024-06-29T15:58:37.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe07c0b9040 0x7fe07c0b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:37.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe07c0b9040 0x7fe07c0b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33380/0 (socket says 172.21.15.81:33380) 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07a7fc640 1 -- 172.21.15.81:0/190136729 learned_addr learned my addr 172.21.15.81:0/190136729 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07b7fe640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe07c0d2a90 0x7fe07c0b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07affd640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe07c0b5250 0x7fe07c1f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07a7fc640 1 -- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe07c0b5250 msgr2=0x7fe07c1f8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07a7fc640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe07c0b5250 0x7fe07c1f8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07a7fc640 1 -- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe07c0d2a90 msgr2=0x7fe07c0b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07a7fc640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe07c0d2a90 0x7fe07c0b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07a7fc640 1 -- 172.21.15.81:0/190136729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe07c1f8540 con 0x7fe07c0b9040 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe07a7fc640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe07c0b9040 0x7fe07c0b0820 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c370 tx=0x7fe06400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.238+0000 7fe0737fe640 1 -- 172.21.15.81:0/190136729 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe064017070 con 0x7fe07c0b9040 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.239+0000 7fe0737fe640 1 -- 172.21.15.81:0/190136729 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0640035f0 con 0x7fe07c0b9040 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.239+0000 7fe081426640 1 -- 172.21.15.81:0/190136729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe07c0b4d90 con 0x7fe07c0b9040 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.239+0000 7fe0737fe640 1 -- 172.21.15.81:0/190136729 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe064003a40 con 0x7fe07c0b9040 2024-06-29T15:58:37.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.239+0000 7fe081426640 1 -- 172.21.15.81:0/190136729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe07c0a1600 con 0x7fe07c0b9040 2024-06-29T15:58:37.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.239+0000 7fe081426640 1 -- 172.21.15.81:0/190136729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe07c06e420 con 0x7fe07c0b9040 2024-06-29T15:58:37.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.240+0000 7fe0737fe640 1 -- 172.21.15.81:0/190136729 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 138) ==== 98405+0+0 (secure 0 0 0) 0x7fe06401a070 con 0x7fe07c0b9040 2024-06-29T15:58:37.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.241+0000 7fe0737fe640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe05c076510 0x7fe05c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:37.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.241+0000 7fe0737fe640 1 -- 172.21.15.81:0/190136729 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7fe064097340 con 0x7fe07c0b9040 2024-06-29T15:58:37.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.241+0000 7fe07affd640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe05c076510 0x7fe05c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:37.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.242+0000 7fe07affd640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe05c076510 0x7fe05c078a20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe06c005fd0 tx=0x7fe06c0097d0 comp rx=0 tx=0).ready entity=mgr.9155 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:37.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.245+0000 7fe0737fe640 1 -- 172.21.15.81:0/190136729 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe07c06e420 con 0x7fe07c0b9040 2024-06-29T15:58:37.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:37.476+0000 7fe081426640 1 -- 172.21.15.81:0/190136729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fe07c0b83b0 con 0x7fe07c0b9040 2024-06-29T15:58:38.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.434+0000 7fe0737fe640 1 -- 172.21.15.81:0/190136729 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v139) ==== 92+0+0 (secure 0 0 0) 0x7fe0640604d0 con 0x7fe07c0b9040 2024-06-29T15:58:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.437+0000 7fe081426640 1 -- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe05c076510 msgr2=0x7fe05c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.437+0000 7fe081426640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe05c076510 0x7fe05c078a20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe06c005fd0 tx=0x7fe06c0097d0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.437+0000 7fe081426640 1 -- 172.21.15.81:0/190136729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe07c0b9040 msgr2=0x7fe07c0b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.437+0000 7fe081426640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe07c0b9040 0x7fe07c0b0820 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c370 tx=0x7fe06400c830 comp rx=0 tx=0).stop 2024-06-29T15:58:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.437+0000 7fe081426640 1 -- 172.21.15.81:0/190136729 shutdown_connections 2024-06-29T15:58:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.437+0000 7fe081426640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fe05c076510 0x7fe05c078a20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.438+0000 7fe081426640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe07c0d2a90 0x7fe07c0b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.438+0000 7fe081426640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe07c0b9040 0x7fe07c0b0820 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.438+0000 7fe081426640 1 --2- 172.21.15.81:0/190136729 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe07c0b5250 0x7fe07c1f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.438+0000 7fe081426640 1 -- 172.21.15.81:0/190136729 >> 172.21.15.81:0/190136729 conn(0x7fe07c09cf30 msgr2=0x7fe07c0ca2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:38.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.438+0000 7fe081426640 1 -- 172.21.15.81:0/190136729 shutdown_connections 2024-06-29T15:58:38.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.438+0000 7fe081426640 1 -- 172.21.15.81:0/190136729 wait complete. 2024-06-29T15:58:38.455 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:58:38.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.585+0000 7fa92317a640 1 -- 172.21.15.81:0/148204727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91c0cd150 msgr2=0x7fa91c072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.585+0000 7fa92317a640 1 --2- 172.21.15.81:0/148204727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91c0cd150 0x7fa91c072f40 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7fa91c072cf0 tx=0x7fa91402a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.585+0000 7fa92317a640 1 -- 172.21.15.81:0/148204727 shutdown_connections 2024-06-29T15:58:38.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.585+0000 7fa92317a640 1 --2- 172.21.15.81:0/148204727 >> v2:172.21.15.167:3300/0 conn(0x7fa91c15bdf0 0x7fa91c06ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.585+0000 7fa92317a640 1 --2- 172.21.15.81:0/148204727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91c0cd150 0x7fa91c072f40 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.585+0000 7fa92317a640 1 -- 172.21.15.81:0/148204727 >> 172.21.15.81:0/148204727 conn(0x7fa91c0b2920 msgr2=0x7fa91c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:38.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.585+0000 7fa92317a640 1 -- 172.21.15.81:0/148204727 shutdown_connections 2024-06-29T15:58:38.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.586+0000 7fa92317a640 1 -- 172.21.15.81:0/148204727 wait complete. 2024-06-29T15:58:38.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.587+0000 7fa92317a640 1 Processor -- start 2024-06-29T15:58:38.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.587+0000 7fa92317a640 1 -- start start 2024-06-29T15:58:38.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa92317a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91c073480 0x7fa91c1434f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa92317a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa91c0cd150 0x7fa91c0d1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa92317a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa91c15bdf0 0x7fa91c0cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa92317a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c06f450 con 0x7fa91c073480 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa92317a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c06f2d0 con 0x7fa91c15bdf0 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa92317a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa91c06f5d0 con 0x7fa91c0cd150 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa920f16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91c073480 0x7fa91c1434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa920f16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91c073480 0x7fa91c1434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33400/0 (socket says 172.21.15.81:33400) 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa920f16640 1 -- 172.21.15.81:0/3640444434 learned_addr learned my addr 172.21.15.81:0/3640444434 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa921717640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa91c15bdf0 0x7fa91c0cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa91bfff640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa91c0cd150 0x7fa91c0d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa920f16640 1 -- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa91c0cd150 msgr2=0x7fa91c0d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa920f16640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa91c0cd150 0x7fa91c0d1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa920f16640 1 -- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa91c15bdf0 msgr2=0x7fa91c0cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa920f16640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa91c15bdf0 0x7fa91c0cf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa920f16640 1 -- 172.21.15.81:0/3640444434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91c0ce610 con 0x7fa91c073480 2024-06-29T15:58:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.589+0000 7fa920f16640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91c073480 0x7fa91c1434f0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fa9140098f0 tx=0x7fa914033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.590+0000 7fa9197fa640 1 -- 172.21.15.81:0/3640444434 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914005c90 con 0x7fa91c073480 2024-06-29T15:58:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.590+0000 7fa9197fa640 1 -- 172.21.15.81:0/3640444434 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa914031e50 con 0x7fa91c073480 2024-06-29T15:58:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.590+0000 7fa92317a640 1 -- 172.21.15.81:0/3640444434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91c0ce8f0 con 0x7fa91c073480 2024-06-29T15:58:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.590+0000 7fa9197fa640 1 -- 172.21.15.81:0/3640444434 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914005260 con 0x7fa91c073480 2024-06-29T15:58:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.590+0000 7fa92317a640 1 -- 172.21.15.81:0/3640444434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91c0ceeb0 con 0x7fa91c073480 2024-06-29T15:58:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.591+0000 7fa92317a640 1 -- 172.21.15.81:0/3640444434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa91c072f40 con 0x7fa91c073480 2024-06-29T15:58:38.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.592+0000 7fa9197fa640 1 -- 172.21.15.81:0/3640444434 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 139) ==== 98422+0+0 (secure 0 0 0) 0x7fa914005400 con 0x7fa91c073480 2024-06-29T15:58:38.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.592+0000 7fa9197fa640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa8fc076590 0x7fa8fc078aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.592+0000 7fa91bfff640 1 -- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa8fc076590 msgr2=0x7fa8fc078aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3661607921 2024-06-29T15:58:38.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.592+0000 7fa9197fa640 1 -- 172.21.15.81:0/3640444434 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7fa9140b8fe0 con 0x7fa91c073480 2024-06-29T15:58:38.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.592+0000 7fa91bfff640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa8fc076590 0x7fa8fc078aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.596+0000 7fa9197fa640 1 -- 172.21.15.81:0/3640444434 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa91403f030 con 0x7fa91c073480 2024-06-29T15:58:38.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.785+0000 7fa92317a640 1 -- 172.21.15.81:0/3640444434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fa91c0d1790 con 0x7fa91c073480 2024-06-29T15:58:38.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.789+0000 7fa9197fa640 1 -- 172.21.15.81:0/3640444434 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) ==== 138+0+0 (secure 0 0 0) 0x7fa91408a150 con 0x7fa91c073480 2024-06-29T15:58:38.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 -- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa8fc076590 msgr2=0x7fa8fc078aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:38.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa8fc076590 0x7fa8fc078aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 -- 172.21.15.81:0/3640444434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91c073480 msgr2=0x7fa91c1434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91c073480 0x7fa91c1434f0 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fa9140098f0 tx=0x7fa914033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:38.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 -- 172.21.15.81:0/3640444434 shutdown_connections 2024-06-29T15:58:38.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7fa8fc076590 0x7fa8fc078aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa91c15bdf0 0x7fa91c0cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa91c0cd150 0x7fa91c0d1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 --2- 172.21.15.81:0/3640444434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91c073480 0x7fa91c1434f0 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 -- 172.21.15.81:0/3640444434 >> 172.21.15.81:0/3640444434 conn(0x7fa91c0b2920 msgr2=0x7fa91c0709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:38.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 -- 172.21.15.81:0/3640444434 shutdown_connections 2024-06-29T15:58:38.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.791+0000 7fa92317a640 1 -- 172.21.15.81:0/3640444434 wait complete. 2024-06-29T15:58:38.803 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:58:38.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.921+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/2783961875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d280d0230 msgr2=0x7f2d280d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.921+0000 7f2d2ef7a640 1 --2- 172.21.15.81:0/2783961875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d280d0230 0x7f2d280d0640 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009830 tx=0x7f2d1402a530 comp rx=0 tx=0).stop 2024-06-29T15:58:38.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.921+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/2783961875 shutdown_connections 2024-06-29T15:58:38.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.921+0000 7f2d2ef7a640 1 --2- 172.21.15.81:0/2783961875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d280d0230 0x7f2d280d0640 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.921+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/2783961875 >> 172.21.15.81:0/2783961875 conn(0x7f2d280b3680 msgr2=0x7f2d280b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:38.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.922+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/2783961875 shutdown_connections 2024-06-29T15:58:38.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.922+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/2783961875 wait complete. 2024-06-29T15:58:38.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.924+0000 7f2d2ef7a640 1 Processor -- start 2024-06-29T15:58:38.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.925+0000 7f2d2ef7a640 1 -- start start 2024-06-29T15:58:38.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2ef7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d280b3200 0x7f2d2814e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2ef7a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d280b4f40 0x7f2d2814f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2ef7a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d280d0230 0x7f2d2814f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2ef7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2d280b5e50 con 0x7f2d280b3200 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2ef7a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2d280b5cd0 con 0x7f2d280b4f40 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2ef7a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2d280b5fd0 con 0x7f2d280d0230 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2cd16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d280b3200 0x7f2d2814e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2cd16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d280b3200 0x7f2d2814e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33418/0 (socket says 172.21.15.81:33418) 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2cd16640 1 -- 172.21.15.81:0/4208186002 learned_addr learned my addr 172.21.15.81:0/4208186002 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d27fff640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d280b4f40 0x7f2d2814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2d517640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d280d0230 0x7f2d2814f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2cd16640 1 -- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d280d0230 msgr2=0x7f2d2814f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2cd16640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d280d0230 0x7f2d2814f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2cd16640 1 -- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d280b4f40 msgr2=0x7f2d2814f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2cd16640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d280b4f40 0x7f2d2814f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.927+0000 7f2d2cd16640 1 -- 172.21.15.81:0/4208186002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d28159200 con 0x7f2d280b3200 2024-06-29T15:58:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.928+0000 7f2d2cd16640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d280b3200 0x7f2d2814e200 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00c730 tx=0x7f2d1c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.928+0000 7f2d257fa640 1 -- 172.21.15.81:0/4208186002 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c017070 con 0x7f2d280b3200 2024-06-29T15:58:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.928+0000 7f2d257fa640 1 -- 172.21.15.81:0/4208186002 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d1c006890 con 0x7f2d280b3200 2024-06-29T15:58:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.928+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/4208186002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d2809d820 con 0x7f2d280b3200 2024-06-29T15:58:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.928+0000 7f2d257fa640 1 -- 172.21.15.81:0/4208186002 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c006ce0 con 0x7f2d280b3200 2024-06-29T15:58:38.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.928+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/4208186002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d2809dde0 con 0x7f2d280b3200 2024-06-29T15:58:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.929+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/4208186002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d281593d0 con 0x7f2d280b3200 2024-06-29T15:58:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.930+0000 7f2d257fa640 1 -- 172.21.15.81:0/4208186002 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 139) ==== 98422+0+0 (secure 0 0 0) 0x7f2d1c01a070 con 0x7f2d280b3200 2024-06-29T15:58:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.930+0000 7f2d257fa640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2d0c076560 0x7f2d0c078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.930+0000 7f2d257fa640 1 -- 172.21.15.81:0/4208186002 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7f2d1c0973b0 con 0x7f2d280b3200 2024-06-29T15:58:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.930+0000 7f2d27fff640 1 -- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2d0c076560 msgr2=0x7f2d0c078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3661607921 2024-06-29T15:58:38.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.930+0000 7f2d27fff640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2d0c076560 0x7f2d0c078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:38.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:38.935+0000 7f2d257fa640 1 -- 172.21.15.81:0/4208186002 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2d280d0b80 con 0x7f2d280b3200 2024-06-29T15:58:39.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.123+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/4208186002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f2d2809e1c0 con 0x7f2d280b3200 2024-06-29T15:58:39.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.127+0000 7f2d257fa640 1 -- 172.21.15.81:0/4208186002 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) ==== 133+0+0 (secure 0 0 0) 0x7f2d1c060010 con 0x7f2d280b3200 2024-06-29T15:58:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2d0c076560 msgr2=0x7f2d0c078a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2d0c076560 0x7f2d0c078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/4208186002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d280b3200 msgr2=0x7f2d2814e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d280b3200 0x7f2d2814e200 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00c730 tx=0x7f2d1c00cbf0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/4208186002 shutdown_connections 2024-06-29T15:58:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f2d0c076560 0x7f2d0c078a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d280d0230 0x7f2d2814f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d280b4f40 0x7f2d2814f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 --2- 172.21.15.81:0/4208186002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d280b3200 0x7f2d2814e200 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/4208186002 >> 172.21.15.81:0/4208186002 conn(0x7f2d280b3680 msgr2=0x7f2d281551a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/4208186002 shutdown_connections 2024-06-29T15:58:39.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.129+0000 7f2d2ef7a640 1 -- 172.21.15.81:0/4208186002 wait complete. 2024-06-29T15:58:39.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:39.142 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2024-06-29T15:58:39.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2024-06-29T15:58:39.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.318+0000 7f1589410640 1 -- 172.21.15.81:0/2389135636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15840b4250 msgr2=0x7f15840cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.318+0000 7f1589410640 1 --2- 172.21.15.81:0/2389135636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15840b4250 0x7f15840cee70 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f15840d4c50 tx=0x7f157c02a790 comp rx=0 tx=0).stop 2024-06-29T15:58:39.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.318+0000 7f1589410640 1 -- 172.21.15.81:0/2389135636 shutdown_connections 2024-06-29T15:58:39.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.318+0000 7f1589410640 1 --2- 172.21.15.81:0/2389135636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15840b4250 0x7f15840cee70 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.318+0000 7f1589410640 1 --2- 172.21.15.81:0/2389135636 >> v2:172.21.15.167:3300/0 conn(0x7f158408c270 0x7f15840ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.318+0000 7f1589410640 1 -- 172.21.15.81:0/2389135636 >> 172.21.15.81:0/2389135636 conn(0x7f15840b56d0 msgr2=0x7f15840b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:39.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.318+0000 7f1589410640 1 -- 172.21.15.81:0/2389135636 shutdown_connections 2024-06-29T15:58:39.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.318+0000 7f1589410640 1 -- 172.21.15.81:0/2389135636 wait complete. 2024-06-29T15:58:39.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.321+0000 7f1589410640 1 Processor -- start 2024-06-29T15:58:39.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.321+0000 7f1589410640 1 -- start start 2024-06-29T15:58:39.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f1589410640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f158408ccd0 0x7f15841d3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:39.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f1589410640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15840b4250 0x7f15841f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:39.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f1589410640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15840b6f50 0x7f15840b7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:39.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f1589410640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f15840d2250 con 0x7f158408ccd0 2024-06-29T15:58:39.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f1589410640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f15840d20d0 con 0x7f15840b6f50 2024-06-29T15:58:39.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f1589410640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f15840d23d0 con 0x7f15840b4250 2024-06-29T15:58:39.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f1582ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f158408ccd0 0x7f15841d3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:39.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f1582ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f158408ccd0 0x7f15841d3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33434/0 (socket says 172.21.15.81:33434) 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f1582ffd640 1 -- 172.21.15.81:0/921144036 learned_addr learned my addr 172.21.15.81:0/921144036 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f15827fc640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15840b4250 0x7f15841f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.323+0000 7f15837fe640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15840b6f50 0x7f15840b7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f1582ffd640 1 -- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15840b4250 msgr2=0x7f15841f5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f1582ffd640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15840b4250 0x7f15841f5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f1582ffd640 1 -- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15840b6f50 msgr2=0x7f15840b7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f1582ffd640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15840b6f50 0x7f15840b7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f1582ffd640 1 -- 172.21.15.81:0/921144036 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15840b78a0 con 0x7f158408ccd0 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f15837fe640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f15840b6f50 0x7f15840b7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f15827fc640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f15840b4250 0x7f15841f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f1582ffd640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f158408ccd0 0x7f15841d3860 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f157800e790 tx=0x7f157800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:39.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157800eea0 con 0x7f158408ccd0 2024-06-29T15:58:39.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f157800cdb0 con 0x7f158408ccd0 2024-06-29T15:58:39.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f1589410640 1 -- 172.21.15.81:0/921144036 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f158409dce0 con 0x7f158408ccd0 2024-06-29T15:58:39.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578010690 con 0x7f158408ccd0 2024-06-29T15:58:39.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.324+0000 7f1589410640 1 -- 172.21.15.81:0/921144036 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f158409e2a0 con 0x7f158408ccd0 2024-06-29T15:58:39.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.325+0000 7f1589410640 1 -- 172.21.15.81:0/921144036 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f158408c270 con 0x7f158408ccd0 2024-06-29T15:58:39.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.326+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 139) ==== 98422+0+0 (secure 0 0 0) 0x7f1578010830 con 0x7f158408ccd0 2024-06-29T15:58:39.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.327+0000 7f156b7fe640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 0x7f1570078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:39.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.327+0000 7f15827fc640 1 -- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 msgr2=0x7f1570078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3661607921 2024-06-29T15:58:39.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.327+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 6701+0+0 (secure 0 0 0) 0x7f1578098b70 con 0x7f158408ccd0 2024-06-29T15:58:39.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.327+0000 7f15827fc640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 0x7f1570078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:39.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.331+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f15840b78a0 con 0x7f158408ccd0 2024-06-29T15:58:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.528+0000 7f15827fc640 1 -- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 msgr2=0x7f1570078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3661607921 2024-06-29T15:58:39.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.528+0000 7f15827fc640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 0x7f1570078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:39.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.544+0000 7f1589410640 1 -- 172.21.15.81:0/921144036 --> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f15840b8f80 con 0x7f1570076560 2024-06-29T15:58:39.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.929+0000 7f15827fc640 1 -- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 msgr2=0x7f1570078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3661607921 2024-06-29T15:58:39.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:39.929+0000 7f15827fc640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 0x7f1570078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:58:40.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:40.730+0000 7f15827fc640 1 -- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 msgr2=0x7f1570078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/3661607921 2024-06-29T15:58:40.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:40.730+0000 7f15827fc640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 0x7f1570078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:58:42.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:42.096+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 140) ==== 97929+0+0 (secure 0 0 0) 0x7f157805d760 con 0x7f158408ccd0 2024-06-29T15:58:42.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:42.096+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 msgr2=0x7f1570078a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:42.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:42.096+0000 7f156b7fe640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/3661607921,v1:172.21.15.167:6833/3661607921] conn(0x7f1570076560 0x7f1570078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:42.324 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.323+0000 7f5732bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.324 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.323+0000 7f5732bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.324 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.323+0000 7f5732bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.324 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.323+0000 7f5732bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.324 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.323+0000 7f5732bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.324 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.323+0000 7f5735bd7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.325 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.323+0000 7f5735bd7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.325 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.323+0000 7f5735bd7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.325 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.323+0000 7f5735bd7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.325 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.323+0000 7f5735bd7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.335 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.334+0000 7f5735bd7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.335 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.334+0000 7f5735bd7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.335 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.334+0000 7f5735bd7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.335 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.334+0000 7f5735bd7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.335 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.334+0000 7f5735bd7640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.336 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.334+0000 7f5732bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.336 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.334+0000 7f5732bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.336 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.334+0000 7f5732bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.336 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.334+0000 7f5732bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:42.336 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:42.334+0000 7f5732bd1640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:43.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.098+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 141) ==== 98056+0+0 (secure 0 0 0) 0x7f157805c2c0 con 0x7f158408ccd0 2024-06-29T15:58:43.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.098+0000 7f156b7fe640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f157003adf0 0x7f1570082c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:43.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.098+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 --> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) -- 0x7f15780989a0 con 0x7f157003adf0 2024-06-29T15:58:43.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.099+0000 7f15827fc640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f157003adf0 0x7f1570082c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:43.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.100+0000 7f15827fc640 1 --2- 172.21.15.81:0/921144036 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f157003adf0 0x7f1570082c50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1574007fd0 tx=0x7f1574009040 comp rx=0 tx=0).ready entity=mgr.10966 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:43.103 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:43.103+0000 7f573cbe5640 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2024-06-29T15:58:43.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.104+0000 7f156b7fe640 1 -- 172.21.15.81:0/921144036 <== mgr.10966 v2:172.21.15.167:6832/884808428 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) ==== 79+0+0 (secure 0 0 0) 0x7f15780989a0 con 0x7f157003adf0 2024-06-29T15:58:43.104 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2024-06-29T15:58:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-29T15:58:43.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:43.112 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2024-06-29T15:58:43.131 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:58:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.290+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3963189531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47d806f850 msgr2=0x7f47d80ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.290+0000 7f47dcc5e640 1 --2- 172.21.15.81:0/3963189531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47d806f850 0x7f47d80ae300 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009830 tx=0x7f47c002a510 comp rx=0 tx=0).stop 2024-06-29T15:58:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.290+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3963189531 shutdown_connections 2024-06-29T15:58:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.290+0000 7f47dcc5e640 1 --2- 172.21.15.81:0/3963189531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47d806f850 0x7f47d80ae300 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.290+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3963189531 >> 172.21.15.81:0/3963189531 conn(0x7f47d8072950 msgr2=0x7f47d80b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.290+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3963189531 shutdown_connections 2024-06-29T15:58:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.290+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3963189531 wait complete. 2024-06-29T15:58:43.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.292+0000 7f47dcc5e640 1 Processor -- start 2024-06-29T15:58:43.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.292+0000 7f47dcc5e640 1 -- start start 2024-06-29T15:58:43.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47dcc5e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47d806f850 0x7f47d81e7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:43.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47dcc5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47d8074520 0x7f47d815d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:43.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47dcc5e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47d80ae840 0x7f47d81569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:43.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47dcc5e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f47d815b190 con 0x7f47d8074520 2024-06-29T15:58:43.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47dcc5e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f47d815b010 con 0x7f47d80ae840 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47dcc5e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f47d80af360 con 0x7f47d806f850 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47cffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47d8074520 0x7f47d815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47cffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47d8074520 0x7f47d815d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53326/0 (socket says 172.21.15.81:53326) 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47cffff640 1 -- 172.21.15.81:0/3057960374 learned_addr learned my addr 172.21.15.81:0/3057960374 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47d659c640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47d806f850 0x7f47d81e7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47d6d9d640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47d80ae840 0x7f47d81569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47cffff640 1 -- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47d806f850 msgr2=0x7f47d81e7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47cffff640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47d806f850 0x7f47d81e7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47cffff640 1 -- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47d80ae840 msgr2=0x7f47d81569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47cffff640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47d80ae840 0x7f47d81569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47cffff640 1 -- 172.21.15.81:0/3057960374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d81e8450 con 0x7f47d8074520 2024-06-29T15:58:43.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.294+0000 7f47cffff640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47d8074520 0x7f47d815d750 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009950 tx=0x7f47c0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:43.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.295+0000 7f47ceffd640 1 -- 172.21.15.81:0/3057960374 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0031d60 con 0x7f47d8074520 2024-06-29T15:58:43.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.295+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3057960374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d81e8730 con 0x7f47d8074520 2024-06-29T15:58:43.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.295+0000 7f47ceffd640 1 -- 172.21.15.81:0/3057960374 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47c0033e10 con 0x7f47d8074520 2024-06-29T15:58:43.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.295+0000 7f47ceffd640 1 -- 172.21.15.81:0/3057960374 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0005260 con 0x7f47d8074520 2024-06-29T15:58:43.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.295+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3057960374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d8156480 con 0x7f47d8074520 2024-06-29T15:58:43.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.296+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3057960374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d81e81c0 con 0x7f47d8074520 2024-06-29T15:58:43.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.297+0000 7f47ceffd640 1 -- 172.21.15.81:0/3057960374 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 141) ==== 98056+0+0 (secure 0 0 0) 0x7f47c0005400 con 0x7f47d8074520 2024-06-29T15:58:43.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.297+0000 7f47ceffd640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f47b8076170 0x7f47b8078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:43.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.297+0000 7f47ceffd640 1 -- 172.21.15.81:0/3057960374 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7f47c00b8af0 con 0x7f47d8074520 2024-06-29T15:58:43.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.297+0000 7f47d659c640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f47b8076170 0x7f47b8078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:43.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.298+0000 7f47d659c640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f47b8076170 0x7f47b8078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f47c8006fd0 tx=0x7f47c8009060 comp rx=0 tx=0).ready entity=mgr.10966 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:43.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.302+0000 7f47ceffd640 1 -- 172.21.15.81:0/3057960374 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f47c003f030 con 0x7f47d8074520 2024-06-29T15:58:43.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:43.535+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3057960374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f47d80ae300 con 0x7f47d8074520 2024-06-29T15:58:44.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.145+0000 7f47ceffd640 1 -- 172.21.15.81:0/3057960374 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v142) ==== 93+0+0 (secure 0 0 0) 0x7f47c008a170 con 0x7f47d8074520 2024-06-29T15:58:44.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.149+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f47b8076170 msgr2=0x7f47b8078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:44.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.149+0000 7f47dcc5e640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f47b8076170 0x7f47b8078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f47c8006fd0 tx=0x7f47c8009060 comp rx=0 tx=0).stop 2024-06-29T15:58:44.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.149+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3057960374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47d8074520 msgr2=0x7f47d815d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:44.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.149+0000 7f47dcc5e640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47d8074520 0x7f47d815d750 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009950 tx=0x7f47c0033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:44.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.150+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3057960374 shutdown_connections 2024-06-29T15:58:44.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.150+0000 7f47dcc5e640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f47d80ae840 0x7f47d81569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.150+0000 7f47dcc5e640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f47b8076170 0x7f47b8078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.150+0000 7f47dcc5e640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47d8074520 0x7f47d815d750 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.150+0000 7f47dcc5e640 1 --2- 172.21.15.81:0/3057960374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f47d806f850 0x7f47d81e7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.150+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3057960374 >> 172.21.15.81:0/3057960374 conn(0x7f47d8072950 msgr2=0x7f47d80b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.150+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3057960374 shutdown_connections 2024-06-29T15:58:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.150+0000 7f47dcc5e640 1 -- 172.21.15.81:0/3057960374 wait complete. 2024-06-29T15:58:44.165 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:58:44.165 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:58:44.207 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:58:44.250 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:58:44.250 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:58:44.250 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:58:44.251 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:44.266 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:58:44.266 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:44.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:44.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.480+0000 7f78fe5a1640 1 -- 172.21.15.81:0/3116203416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78f80b3200 msgr2=0x7f78f80cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:44.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.480+0000 7f78fe5a1640 1 --2- 172.21.15.81:0/3116203416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78f80b3200 0x7f78f80cc7b0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f78e8005fd0 tx=0x7f78e802a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.480+0000 7f78fe5a1640 1 -- 172.21.15.81:0/3116203416 shutdown_connections 2024-06-29T15:58:44.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.480+0000 7f78fe5a1640 1 --2- 172.21.15.81:0/3116203416 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78f80d0940 0x7f78f80b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.480+0000 7f78fe5a1640 1 --2- 172.21.15.81:0/3116203416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78f80b3200 0x7f78f80cc7b0 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.480+0000 7f78fe5a1640 1 -- 172.21.15.81:0/3116203416 >> 172.21.15.81:0/3116203416 conn(0x7f78f80b3680 msgr2=0x7f78f80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:44.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.480+0000 7f78fe5a1640 1 -- 172.21.15.81:0/3116203416 shutdown_connections 2024-06-29T15:58:44.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.480+0000 7f78fe5a1640 1 -- 172.21.15.81:0/3116203416 wait complete. 2024-06-29T15:58:44.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.483+0000 7f78fe5a1640 1 Processor -- start 2024-06-29T15:58:44.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.483+0000 7f78fe5a1640 1 -- start start 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78fe5a1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78f80b3200 0x7f78f8156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78fe5a1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78f80cccf0 0x7f78f81586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78fe5a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78f8158be0 0x7f78f8158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78fe5a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f78f80b5c60 con 0x7f78f8158be0 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78fe5a1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f78f80b5ae0 con 0x7f78f80b3200 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78fe5a1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f78f80b5de0 con 0x7f78f80cccf0 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78fcb3e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78f8158be0 0x7f78f8158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78fcb3e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78f8158be0 0x7f78f8158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53362/0 (socket says 172.21.15.81:53362) 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78fcb3e640 1 -- 172.21.15.81:0/1943745851 learned_addr learned my addr 172.21.15.81:0/1943745851 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78f7fff640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78f80b3200 0x7f78f8156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:44.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.485+0000 7f78f77fe640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78f80cccf0 0x7f78f81586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78fcb3e640 1 -- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78f80cccf0 msgr2=0x7f78f81586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78fcb3e640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78f80cccf0 0x7f78f81586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78fcb3e640 1 -- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78f80b3200 msgr2=0x7f78f8156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78fcb3e640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78f80b3200 0x7f78f8156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78fcb3e640 1 -- 172.21.15.81:0/1943745851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f80c7890 con 0x7f78f8158be0 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78fcb3e640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78f8158be0 0x7f78f8158ff0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f78f000ea40 tx=0x7f78f000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78f4ff9640 1 -- 172.21.15.81:0/1943745851 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f000ce40 con 0x7f78f8158be0 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78f4ff9640 1 -- 172.21.15.81:0/1943745851 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78f0004c10 con 0x7f78f8158be0 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78fe5a1640 1 -- 172.21.15.81:0/1943745851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f80c7b70 con 0x7f78f8158be0 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78f4ff9640 1 -- 172.21.15.81:0/1943745851 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f0005020 con 0x7f78f8158be0 2024-06-29T15:58:44.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.486+0000 7f78fe5a1640 1 -- 172.21.15.81:0/1943745851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f80c8130 con 0x7f78f8158be0 2024-06-29T15:58:44.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.487+0000 7f78fe5a1640 1 -- 172.21.15.81:0/1943745851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f80c9560 con 0x7f78f8158be0 2024-06-29T15:58:44.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.488+0000 7f78f4ff9640 1 -- 172.21.15.81:0/1943745851 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7f78f00051c0 con 0x7f78f8158be0 2024-06-29T15:58:44.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.488+0000 7f78f4ff9640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f78d8076510 0x7f78d8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:44.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.489+0000 7f78f7fff640 1 -- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f78d8076510 msgr2=0x7f78d8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:44.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.489+0000 7f78f4ff9640 1 -- 172.21.15.81:0/1943745851 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7f78f0098ee0 con 0x7f78f8158be0 2024-06-29T15:58:44.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.489+0000 7f78f7fff640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f78d8076510 0x7f78d8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:44.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.493+0000 7f78f4ff9640 1 -- 172.21.15.81:0/1943745851 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f78f0061770 con 0x7f78f8158be0 2024-06-29T15:58:44.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.682+0000 7f78fe5a1640 1 -- 172.21.15.81:0/1943745851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f78f80cc7b0 con 0x7f78f8158be0 2024-06-29T15:58:44.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.684+0000 7f78f4ff9640 1 -- 172.21.15.81:0/1943745851 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) ==== 74+0+21178 (secure 0 0 0) 0x7f78f007ae20 con 0x7f78f8158be0 2024-06-29T15:58:44.684 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:44.684 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":238,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:42.086750+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:29.291185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:29.702735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/1116386548":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6832/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/3057737710":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6833/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1418041738":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2207847535":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/874327121":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/273305307":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2269612087":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1188031263":"2024-06-30T15:58:42.086693+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:44.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 -- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f78d8076510 msgr2=0x7f78d8078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:44.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f78d8076510 0x7f78d8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 -- 172.21.15.81:0/1943745851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78f8158be0 msgr2=0x7f78f8158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:44.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78f8158be0 0x7f78f8158ff0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f78f000ea40 tx=0x7f78f000ef00 comp rx=0 tx=0).stop 2024-06-29T15:58:44.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 -- 172.21.15.81:0/1943745851 shutdown_connections 2024-06-29T15:58:44.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f78d8076510 0x7f78d8078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78f8158be0 0x7f78f8158ff0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78f80cccf0 0x7f78f81586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 --2- 172.21.15.81:0/1943745851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78f80b3200 0x7f78f8156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 -- 172.21.15.81:0/1943745851 >> 172.21.15.81:0/1943745851 conn(0x7f78f80b3680 msgr2=0x7f78f80700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:44.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 -- 172.21.15.81:0/1943745851 shutdown_connections 2024-06-29T15:58:44.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.686+0000 7f78fe5a1640 1 -- 172.21.15.81:0/1943745851 wait complete. 2024-06-29T15:58:44.698 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:58:16.949415+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:58:17.618219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:58:29.291185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:58:29.702735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '237', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:44.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:44.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.818+0000 7fde28622640 1 -- 172.21.15.81:0/2031451951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2015b400 msgr2=0x7fde2015b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:44.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.818+0000 7fde28622640 1 --2- 172.21.15.81:0/2031451951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2015b400 0x7fde2015b810 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7fde20073c30 tx=0x7fde1802aac0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.818+0000 7fde28622640 1 -- 172.21.15.81:0/2031451951 shutdown_connections 2024-06-29T15:58:44.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.818+0000 7fde28622640 1 --2- 172.21.15.81:0/2031451951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde2015b400 0x7fde2015b810 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.818+0000 7fde28622640 1 --2- 172.21.15.81:0/2031451951 >> v2:172.21.15.167:3300/0 conn(0x7fde200b4800 0x7fde20144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.818+0000 7fde28622640 1 -- 172.21.15.81:0/2031451951 >> 172.21.15.81:0/2031451951 conn(0x7fde200b2920 msgr2=0x7fde200b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:44.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.818+0000 7fde28622640 1 -- 172.21.15.81:0/2031451951 shutdown_connections 2024-06-29T15:58:44.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.818+0000 7fde28622640 1 -- 172.21.15.81:0/2031451951 wait complete. 2024-06-29T15:58:44.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.820+0000 7fde28622640 1 Processor -- start 2024-06-29T15:58:44.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.820+0000 7fde28622640 1 -- start start 2024-06-29T15:58:44.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde28622640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde20072f80 0x7fde201420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:44.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde28622640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde200b4800 0x7fde2013f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:44.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde28622640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2015b400 0x7fde2013fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:44.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde28622640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fde2006f250 con 0x7fde20072f80 2024-06-29T15:58:44.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde28622640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fde2006f0d0 con 0x7fde200b4800 2024-06-29T15:58:44.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde28622640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fde2006f3d0 con 0x7fde2015b400 2024-06-29T15:58:44.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde263be640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde20072f80 0x7fde201420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:44.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde263be640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde20072f80 0x7fde201420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53372/0 (socket says 172.21.15.81:53372) 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde263be640 1 -- 172.21.15.81:0/1390335268 learned_addr learned my addr 172.21.15.81:0/1390335268 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde25bbd640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde200b4800 0x7fde2013f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.821+0000 7fde26bbf640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2015b400 0x7fde2013fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde263be640 1 -- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2015b400 msgr2=0x7fde2013fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde263be640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2015b400 0x7fde2013fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde263be640 1 -- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde200b4800 msgr2=0x7fde2013f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde263be640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde200b4800 0x7fde2013f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde263be640 1 -- 172.21.15.81:0/1390335268 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2013b140 con 0x7fde20072f80 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde263be640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde20072f80 0x7fde201420d0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009a10 tx=0x7fde1c009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde12ffd640 1 -- 172.21.15.81:0/1390335268 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c01c070 con 0x7fde20072f80 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde12ffd640 1 -- 172.21.15.81:0/1390335268 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde1c00bab0 con 0x7fde20072f80 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde28622640 1 -- 172.21.15.81:0/1390335268 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2013a0e0 con 0x7fde20072f80 2024-06-29T15:58:44.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde12ffd640 1 -- 172.21.15.81:0/1390335268 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c017410 con 0x7fde20072f80 2024-06-29T15:58:44.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.822+0000 7fde28622640 1 -- 172.21.15.81:0/1390335268 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2013a660 con 0x7fde20072f80 2024-06-29T15:58:44.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.823+0000 7fde28622640 1 -- 172.21.15.81:0/1390335268 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde2006f880 con 0x7fde20072f80 2024-06-29T15:58:44.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.824+0000 7fde12ffd640 1 -- 172.21.15.81:0/1390335268 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7fde1c01f070 con 0x7fde20072f80 2024-06-29T15:58:44.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.824+0000 7fde12ffd640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fde04076510 0x7fde04078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:44.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.825+0000 7fde25bbd640 1 -- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fde04076510 msgr2=0x7fde04078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:44.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.825+0000 7fde12ffd640 1 -- 172.21.15.81:0/1390335268 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7fde1c09a4e0 con 0x7fde20072f80 2024-06-29T15:58:44.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.825+0000 7fde25bbd640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fde04076510 0x7fde04078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:44.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:44.833+0000 7fde12ffd640 1 -- 172.21.15.81:0/1390335268 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fde1c062ed0 con 0x7fde20072f80 2024-06-29T15:58:45.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.019+0000 7fde28622640 1 -- 172.21.15.81:0/1390335268 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fde2015e380 con 0x7fde20072f80 2024-06-29T15:58:45.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.019+0000 7fde12ffd640 1 -- 172.21.15.81:0/1390335268 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v238) ==== 93+0+10 (secure 0 0 0) 0x7fde1c0671d0 con 0x7fde20072f80 2024-06-29T15:58:45.019 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:58:45.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 -- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fde04076510 msgr2=0x7fde04078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:45.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fde04076510 0x7fde04078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 -- 172.21.15.81:0/1390335268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde20072f80 msgr2=0x7fde201420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde20072f80 0x7fde201420d0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009a10 tx=0x7fde1c009ed0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 -- 172.21.15.81:0/1390335268 shutdown_connections 2024-06-29T15:58:45.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fde2015b400 0x7fde2013fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fde200b4800 0x7fde2013f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fde04076510 0x7fde04078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 --2- 172.21.15.81:0/1390335268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde20072f80 0x7fde201420d0 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 -- 172.21.15.81:0/1390335268 >> 172.21.15.81:0/1390335268 conn(0x7fde200b2920 msgr2=0x7fde200b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:45.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 -- 172.21.15.81:0/1390335268 shutdown_connections 2024-06-29T15:58:45.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.021+0000 7fde28622640 1 -- 172.21.15.81:0/1390335268 wait complete. 2024-06-29T15:58:45.033 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:58:45.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.153+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/1879376046 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb2b40cd150 msgr2=0x7fb2b4072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.153+0000 7fb2b9aa4640 1 --2- 172.21.15.81:0/1879376046 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb2b40cd150 0x7fb2b4072f40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009830 tx=0x7fb2a402a480 comp rx=0 tx=0).stop 2024-06-29T15:58:45.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.153+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/1879376046 shutdown_connections 2024-06-29T15:58:45.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.153+0000 7fb2b9aa4640 1 --2- 172.21.15.81:0/1879376046 >> v2:172.21.15.167:3300/0 conn(0x7fb2b4073480 0x7fb2b415b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.153+0000 7fb2b9aa4640 1 --2- 172.21.15.81:0/1879376046 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb2b40cd150 0x7fb2b4072f40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.153+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/1879376046 >> 172.21.15.81:0/1879376046 conn(0x7fb2b40b2920 msgr2=0x7fb2b40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:45.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.153+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/1879376046 shutdown_connections 2024-06-29T15:58:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.153+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/1879376046 wait complete. 2024-06-29T15:58:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.155+0000 7fb2b9aa4640 1 Processor -- start 2024-06-29T15:58:45.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.155+0000 7fb2b9aa4640 1 -- start start 2024-06-29T15:58:45.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.156+0000 7fb2b9aa4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b4073480 0x7fb2b4152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.156+0000 7fb2b9aa4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2b40cd150 0x7fb2b41433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.156+0000 7fb2b9aa4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb2b415bc20 0x7fb2b406c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.156+0000 7fb2b9aa4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b415c1b0 con 0x7fb2b4073480 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.156+0000 7fb2b9aa4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b415c030 con 0x7fb2b40cd150 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.156+0000 7fb2b9aa4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb2b415f060 con 0x7fb2b415bc20 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2b37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b4073480 0x7fb2b4152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2b37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b4073480 0x7fb2b4152ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53374/0 (socket says 172.21.15.81:53374) 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2b37fe640 1 -- 172.21.15.81:0/484442793 learned_addr learned my addr 172.21.15.81:0/484442793 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2abfff640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2b40cd150 0x7fb2b41433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2b3fff640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb2b415bc20 0x7fb2b406c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2abfff640 1 -- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb2b415bc20 msgr2=0x7fb2b406c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2abfff640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb2b415bc20 0x7fb2b406c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2abfff640 1 -- 172.21.15.81:0/484442793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b4073480 msgr2=0x7fb2b4152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2abfff640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b4073480 0x7fb2b4152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2abfff640 1 -- 172.21.15.81:0/484442793 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b4150020 con 0x7fb2b40cd150 2024-06-29T15:58:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2b37fe640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b4073480 0x7fb2b4152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:45.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.157+0000 7fb2abfff640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2b40cd150 0x7fb2b41433f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00e9c0 tx=0x7fb29c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:45.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.162+0000 7fb2b0ff9640 1 -- 172.21.15.81:0/484442793 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c00ccc0 con 0x7fb2b40cd150 2024-06-29T15:58:45.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.162+0000 7fb2b0ff9640 1 -- 172.21.15.81:0/484442793 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb29c010430 con 0x7fb2b40cd150 2024-06-29T15:58:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.162+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/484442793 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b40cec60 con 0x7fb2b40cd150 2024-06-29T15:58:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.162+0000 7fb2b0ff9640 1 -- 172.21.15.81:0/484442793 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c010860 con 0x7fb2b40cd150 2024-06-29T15:58:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.162+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/484442793 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b40d05d0 con 0x7fb2b40cd150 2024-06-29T15:58:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.163+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/484442793 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2b415f580 con 0x7fb2b40cd150 2024-06-29T15:58:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.164+0000 7fb2b0ff9640 1 -- 172.21.15.81:0/484442793 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7fb29c01f020 con 0x7fb2b40cd150 2024-06-29T15:58:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.164+0000 7fb2b0ff9640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fb294076540 0x7fb294078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.164+0000 7fb2b37fe640 1 -- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fb294076540 msgr2=0x7fb294078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.164+0000 7fb2b0ff9640 1 -- 172.21.15.81:0/484442793 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7fb29c0987d0 con 0x7fb2b40cd150 2024-06-29T15:58:45.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.164+0000 7fb2b37fe640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fb294076540 0x7fb294078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:45.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.169+0000 7fb2b0ff9640 1 -- 172.21.15.81:0/484442793 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb29c061040 con 0x7fb2b40cd150 2024-06-29T15:58:45.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.355+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/484442793 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb2b40ceeb0 con 0x7fb2b40cd150 2024-06-29T15:58:45.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.356+0000 7fb2b0ff9640 1 -- 172.21.15.81:0/484442793 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v238) ==== 104+0+11 (secure 0 0 0) 0x7fb29c065340 con 0x7fb2b40cd150 2024-06-29T15:58:45.356 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fb294076540 msgr2=0x7fb294078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fb294076540 0x7fb294078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2b40cd150 msgr2=0x7fb2b41433f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2b40cd150 0x7fb2b41433f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00e9c0 tx=0x7fb29c00ee80 comp rx=0 tx=0).stop 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/484442793 shutdown_connections 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb2b415bc20 0x7fb2b406c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb2b40cd150 0x7fb2b41433f0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fb294076540 0x7fb294078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 --2- 172.21.15.81:0/484442793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b4073480 0x7fb2b4152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/484442793 >> 172.21.15.81:0/484442793 conn(0x7fb2b40b2920 msgr2=0x7fb2b415d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:45.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/484442793 shutdown_connections 2024-06-29T15:58:45.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.358+0000 7fb2b9aa4640 1 -- 172.21.15.81:0/484442793 wait complete. 2024-06-29T15:58:45.370 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:58:45.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.489+0000 7fd01159e640 1 -- 172.21.15.81:0/2744750877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00c0b6a20 msgr2=0x7fd00c0d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.489+0000 7fd01159e640 1 --2- 172.21.15.81:0/2744750877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00c0b6a20 0x7fd00c0d0410 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fd00c08c7e0 tx=0x7fcff402a680 comp rx=0 tx=0).stop 2024-06-29T15:58:45.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.489+0000 7fd01159e640 1 -- 172.21.15.81:0/2744750877 shutdown_connections 2024-06-29T15:58:45.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.489+0000 7fd01159e640 1 --2- 172.21.15.81:0/2744750877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00c0b6a20 0x7fd00c0d0410 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.489+0000 7fd01159e640 1 --2- 172.21.15.81:0/2744750877 >> v2:172.21.15.167:3300/0 conn(0x7fd00c0b8ba0 0x7fd00c0d46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.489+0000 7fd01159e640 1 -- 172.21.15.81:0/2744750877 >> 172.21.15.81:0/2744750877 conn(0x7fd00c0d3bd0 msgr2=0x7fd00c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:45.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.489+0000 7fd01159e640 1 -- 172.21.15.81:0/2744750877 shutdown_connections 2024-06-29T15:58:45.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.489+0000 7fd01159e640 1 -- 172.21.15.81:0/2744750877 wait complete. 2024-06-29T15:58:45.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.491+0000 7fd01159e640 1 Processor -- start 2024-06-29T15:58:45.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.491+0000 7fd01159e640 1 -- start start 2024-06-29T15:58:45.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd01159e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd00c0b6a20 0x7fd00c0a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd01159e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd00c0b8ba0 0x7fd00c09db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd01159e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00c0d0950 0x7fd00c1ea510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd01159e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c0b9290 con 0x7fd00c0d0950 2024-06-29T15:58:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd01159e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c0b7040 con 0x7fd00c0b8ba0 2024-06-29T15:58:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd01159e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd00c0b9410 con 0x7fd00c0b6a20 2024-06-29T15:58:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd00b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00c0d0950 0x7fd00c1ea510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd00affd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd00c0b6a20 0x7fd00c0a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd00affd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd00c0b6a20 0x7fd00c0a15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:40808/0 (socket says 172.21.15.81:40808) 2024-06-29T15:58:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd003fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd00c0b8ba0 0x7fd00c09db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd00affd640 1 -- 172.21.15.81:0/1569191720 learned_addr learned my addr 172.21.15.81:0/1569191720 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd00b7fe640 1 -- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd00c0b6a20 msgr2=0x7fd00c0a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd00b7fe640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd00c0b6a20 0x7fd00c0a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd00b7fe640 1 -- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd00c0b8ba0 msgr2=0x7fd00c09db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.492+0000 7fd00b7fe640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd00c0b8ba0 0x7fd00c09db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.493+0000 7fd00b7fe640 1 -- 172.21.15.81:0/1569191720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00c14a5f0 con 0x7fd00c0d0950 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.493+0000 7fd003fff640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd00c0b8ba0 0x7fd00c09db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.493+0000 7fd00b7fe640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00c0d0950 0x7fd00c1ea510 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fd00400c730 tx=0x7fd00400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.493+0000 7fd0037fe640 1 -- 172.21.15.81:0/1569191720 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd004003b70 con 0x7fd00c0d0950 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.493+0000 7fd0037fe640 1 -- 172.21.15.81:0/1569191720 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd004003d10 con 0x7fd00c0d0950 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.493+0000 7fd01159e640 1 -- 172.21.15.81:0/1569191720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd00c14a8d0 con 0x7fd00c0d0950 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.493+0000 7fd0037fe640 1 -- 172.21.15.81:0/1569191720 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd004005330 con 0x7fd00c0d0950 2024-06-29T15:58:45.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.493+0000 7fd01159e640 1 -- 172.21.15.81:0/1569191720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00c14ae90 con 0x7fd00c0d0950 2024-06-29T15:58:45.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.494+0000 7fd01159e640 1 -- 172.21.15.81:0/1569191720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd00c06d590 con 0x7fd00c0d0950 2024-06-29T15:58:45.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.495+0000 7fd0037fe640 1 -- 172.21.15.81:0/1569191720 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7fd0040054d0 con 0x7fd00c0d0950 2024-06-29T15:58:45.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.496+0000 7fd0037fe640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fcfec076510 0x7fcfec078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.496+0000 7fd00affd640 1 -- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fcfec076510 msgr2=0x7fcfec078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:45.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.496+0000 7fd0037fe640 1 -- 172.21.15.81:0/1569191720 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7fd004097d50 con 0x7fd00c0d0950 2024-06-29T15:58:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.496+0000 7fd00affd640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fcfec076510 0x7fcfec078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:45.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.500+0000 7fd0037fe640 1 -- 172.21.15.81:0/1569191720 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd00c06d590 con 0x7fd00c0d0950 2024-06-29T15:58:45.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.690+0000 7fd01159e640 1 -- 172.21.15.81:0/1569191720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd00c0cff00 con 0x7fd00c0d0950 2024-06-29T15:58:45.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.691+0000 7fd0037fe640 1 -- 172.21.15.81:0/1569191720 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v238) ==== 100+0+11 (secure 0 0 0) 0x7fd00409b030 con 0x7fd00c0d0950 2024-06-29T15:58:45.691 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:58:45.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 -- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fcfec076510 msgr2=0x7fcfec078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:45.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fcfec076510 0x7fcfec078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 -- 172.21.15.81:0/1569191720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00c0d0950 msgr2=0x7fd00c1ea510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00c0d0950 0x7fd00c1ea510 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fd00400c730 tx=0x7fd00400cbf0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 -- 172.21.15.81:0/1569191720 shutdown_connections 2024-06-29T15:58:45.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fcfec076510 0x7fcfec078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00c0d0950 0x7fd00c1ea510 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd00c0b8ba0 0x7fd00c09db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 --2- 172.21.15.81:0/1569191720 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd00c0b6a20 0x7fd00c0a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 -- 172.21.15.81:0/1569191720 >> 172.21.15.81:0/1569191720 conn(0x7fd00c0d3bd0 msgr2=0x7fd00c06ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:45.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 -- 172.21.15.81:0/1569191720 shutdown_connections 2024-06-29T15:58:45.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.693+0000 7fd01159e640 1 -- 172.21.15.81:0/1569191720 wait complete. 2024-06-29T15:58:45.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:58:45.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.823+0000 7f641497f640 1 -- 172.21.15.81:0/3857327389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64100cd150 msgr2=0x7f6410072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.823+0000 7f641497f640 1 --2- 172.21.15.81:0/3857327389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64100cd150 0x7f6410072f40 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009830 tx=0x7f63f802a410 comp rx=0 tx=0).stop 2024-06-29T15:58:45.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.824+0000 7f641497f640 1 -- 172.21.15.81:0/3857327389 shutdown_connections 2024-06-29T15:58:45.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.824+0000 7f641497f640 1 --2- 172.21.15.81:0/3857327389 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6410073480 0x7f641015b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.824+0000 7f641497f640 1 --2- 172.21.15.81:0/3857327389 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64100cd150 0x7f6410072f40 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.824+0000 7f641497f640 1 -- 172.21.15.81:0/3857327389 >> 172.21.15.81:0/3857327389 conn(0x7f64100b2920 msgr2=0x7f64100b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:45.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.824+0000 7f641497f640 1 -- 172.21.15.81:0/3857327389 shutdown_connections 2024-06-29T15:58:45.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.824+0000 7f641497f640 1 -- 172.21.15.81:0/3857327389 wait complete. 2024-06-29T15:58:45.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.826+0000 7f641497f640 1 Processor -- start 2024-06-29T15:58:45.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.826+0000 7f641497f640 1 -- start start 2024-06-29T15:58:45.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f641497f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6410073480 0x7f641014d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f641497f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64100cd150 0x7f64100b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f641497f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f641015bc20 0x7f64100b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f641497f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f641015c1b0 con 0x7f6410073480 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f641497f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f641015c030 con 0x7f641015bc20 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f641497f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f641015f060 con 0x7f64100cd150 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f640e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6410073480 0x7f641014d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f640e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6410073480 0x7f641014d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53390/0 (socket says 172.21.15.81:53390) 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f640e59c640 1 -- 172.21.15.81:0/666268186 learned_addr learned my addr 172.21.15.81:0/666268186 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f640dd9b640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64100cd150 0x7f64100b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f640ed9d640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f641015bc20 0x7f64100b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f640e59c640 1 -- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64100cd150 msgr2=0x7f64100b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f640e59c640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64100cd150 0x7f64100b08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f640e59c640 1 -- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f641015bc20 msgr2=0x7f64100b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f640e59c640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f641015bc20 0x7f64100b0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:45.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.827+0000 7f640e59c640 1 -- 172.21.15.81:0/666268186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6410151fd0 con 0x7f6410073480 2024-06-29T15:58:45.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.828+0000 7f640e59c640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6410073480 0x7f641014d720 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009950 tx=0x7f63f8003b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:45.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.828+0000 7f6406ffd640 1 -- 172.21.15.81:0/666268186 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f803f030 con 0x7f6410073480 2024-06-29T15:58:45.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.828+0000 7f6406ffd640 1 -- 172.21.15.81:0/666268186 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63f8004510 con 0x7f6410073480 2024-06-29T15:58:45.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.828+0000 7f641497f640 1 -- 172.21.15.81:0/666268186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64100ce4a0 con 0x7f6410073480 2024-06-29T15:58:45.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.828+0000 7f6406ffd640 1 -- 172.21.15.81:0/666268186 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8037040 con 0x7f6410073480 2024-06-29T15:58:45.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.828+0000 7f641497f640 1 -- 172.21.15.81:0/666268186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64100d0000 con 0x7f6410073480 2024-06-29T15:58:45.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.829+0000 7f641497f640 1 -- 172.21.15.81:0/666268186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f641014eab0 con 0x7f6410073480 2024-06-29T15:58:45.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.830+0000 7f6406ffd640 1 -- 172.21.15.81:0/666268186 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7f63f8042070 con 0x7f6410073480 2024-06-29T15:58:45.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.830+0000 7f6406ffd640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f63f0076510 0x7f63f0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:45.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.830+0000 7f640dd9b640 1 -- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f63f0076510 msgr2=0x7f63f0078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:45.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.830+0000 7f6406ffd640 1 -- 172.21.15.81:0/666268186 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7f63f80b8480 con 0x7f6410073480 2024-06-29T15:58:45.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.830+0000 7f640dd9b640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f63f0076510 0x7f63f0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:45.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:45.838+0000 7f6406ffd640 1 -- 172.21.15.81:0/666268186 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f641014eab0 con 0x7f6410073480 2024-06-29T15:58:46.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.025+0000 7f641497f640 1 -- 172.21.15.81:0/666268186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f641015f580 con 0x7f6410073480 2024-06-29T15:58:46.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.025+0000 7f6406ffd640 1 -- 172.21.15.81:0/666268186 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v238) ==== 107+0+11 (secure 0 0 0) 0x7f63f8049070 con 0x7f6410073480 2024-06-29T15:58:46.026 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.027+0000 7f641497f640 1 -- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f63f0076510 msgr2=0x7f63f0078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.027+0000 7f641497f640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f63f0076510 0x7f63f0078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.027+0000 7f641497f640 1 -- 172.21.15.81:0/666268186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6410073480 msgr2=0x7f641014d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.027+0000 7f641497f640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6410073480 0x7f641014d720 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009950 tx=0x7f63f8003b60 comp rx=0 tx=0).stop 2024-06-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.027+0000 7f641497f640 1 -- 172.21.15.81:0/666268186 shutdown_connections 2024-06-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.028+0000 7f641497f640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f641015bc20 0x7f64100b0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.028+0000 7f641497f640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64100cd150 0x7f64100b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.028+0000 7f641497f640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f63f0076510 0x7f63f0078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.028+0000 7f641497f640 1 --2- 172.21.15.81:0/666268186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6410073480 0x7f641014d720 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.028+0000 7f641497f640 1 -- 172.21.15.81:0/666268186 >> 172.21.15.81:0/666268186 conn(0x7f64100b2920 msgr2=0x7f641006ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:46.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.028+0000 7f641497f640 1 -- 172.21.15.81:0/666268186 shutdown_connections 2024-06-29T15:58:46.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.028+0000 7f641497f640 1 -- 172.21.15.81:0/666268186 wait complete. 2024-06-29T15:58:46.039 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:58:46.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.157+0000 7ff26ad43640 1 -- 172.21.15.81:0/1296696678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2640b59c0 msgr2=0x7ff264072f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.157+0000 7ff26ad43640 1 --2- 172.21.15.81:0/1296696678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2640b59c0 0x7ff264072f10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff250009860 tx=0x7ff25002a510 comp rx=0 tx=0).stop 2024-06-29T15:58:46.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.157+0000 7ff26ad43640 1 -- 172.21.15.81:0/1296696678 shutdown_connections 2024-06-29T15:58:46.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.157+0000 7ff26ad43640 1 --2- 172.21.15.81:0/1296696678 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2640b59c0 0x7ff264072f10 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.157+0000 7ff26ad43640 1 -- 172.21.15.81:0/1296696678 >> 172.21.15.81:0/1296696678 conn(0x7ff2640d0b70 msgr2=0x7ff2640d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:46.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.157+0000 7ff26ad43640 1 -- 172.21.15.81:0/1296696678 shutdown_connections 2024-06-29T15:58:46.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.157+0000 7ff26ad43640 1 -- 172.21.15.81:0/1296696678 wait complete. 2024-06-29T15:58:46.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.159+0000 7ff26ad43640 1 Processor -- start 2024-06-29T15:58:46.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.159+0000 7ff26ad43640 1 -- start start 2024-06-29T15:58:46.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.160+0000 7ff26ad43640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff264073450 0x7ff264159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.160+0000 7ff26ad43640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2640b59c0 0x7ff26413e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.160+0000 7ff26ad43640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff264143860 0x7ff26415d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.160+0000 7ff26ad43640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff26406f850 con 0x7ff264073450 2024-06-29T15:58:46.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.160+0000 7ff26ad43640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff26406f6d0 con 0x7ff264143860 2024-06-29T15:58:46.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.160+0000 7ff26ad43640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff26406f9d0 con 0x7ff2640b59c0 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff268adf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff264073450 0x7ff264159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff268adf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff264073450 0x7ff264159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53396/0 (socket says 172.21.15.81:53396) 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff268adf640 1 -- 172.21.15.81:0/1365342876 learned_addr learned my addr 172.21.15.81:0/1365342876 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff263fff640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2640b59c0 0x7ff26413e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff2692e0640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff264143860 0x7ff26415d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff268adf640 1 -- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2640b59c0 msgr2=0x7ff26413e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff268adf640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2640b59c0 0x7ff26413e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff268adf640 1 -- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff264143860 msgr2=0x7ff26415d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff268adf640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff264143860 0x7ff26415d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff268adf640 1 -- 172.21.15.81:0/1365342876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26413eb00 con 0x7ff264073450 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff268adf640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff264073450 0x7ff264159330 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7ff25800c760 tx=0x7ff25800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.161+0000 7ff2617fa640 1 -- 172.21.15.81:0/1365342876 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff258016020 con 0x7ff264073450 2024-06-29T15:58:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.162+0000 7ff26ad43640 1 -- 172.21.15.81:0/1365342876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2640b2e30 con 0x7ff264073450 2024-06-29T15:58:46.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.162+0000 7ff2617fa640 1 -- 172.21.15.81:0/1365342876 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff25800ab00 con 0x7ff264073450 2024-06-29T15:58:46.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.162+0000 7ff2617fa640 1 -- 172.21.15.81:0/1365342876 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25800ce10 con 0x7ff264073450 2024-06-29T15:58:46.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.162+0000 7ff26ad43640 1 -- 172.21.15.81:0/1365342876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2640b33f0 con 0x7ff264073450 2024-06-29T15:58:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.163+0000 7ff26ad43640 1 -- 172.21.15.81:0/1365342876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff264072f10 con 0x7ff264073450 2024-06-29T15:58:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.163+0000 7ff2617fa640 1 -- 172.21.15.81:0/1365342876 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7ff258019020 con 0x7ff264073450 2024-06-29T15:58:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.164+0000 7ff2617fa640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7ff248076540 0x7ff248078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.164+0000 7ff263fff640 1 -- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7ff248076540 msgr2=0x7ff248078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.164+0000 7ff2617fa640 1 -- 172.21.15.81:0/1365342876 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7ff258098240 con 0x7ff264073450 2024-06-29T15:58:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.164+0000 7ff263fff640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7ff248076540 0x7ff248078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:46.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.168+0000 7ff2617fa640 1 -- 172.21.15.81:0/1365342876 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff258020020 con 0x7ff264073450 2024-06-29T15:58:46.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.355+0000 7ff26ad43640 1 -- 172.21.15.81:0/1365342876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7ff26406f0f0 con 0x7ff264073450 2024-06-29T15:58:46.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.356+0000 7ff2617fa640 1 -- 172.21.15.81:0/1365342876 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v238) ==== 103+0+11 (secure 0 0 0) 0x7ff26413eb00 con 0x7ff264073450 2024-06-29T15:58:46.356 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:58:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 -- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7ff248076540 msgr2=0x7ff248078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7ff248076540 0x7ff248078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 -- 172.21.15.81:0/1365342876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff264073450 msgr2=0x7ff264159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff264073450 0x7ff264159330 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7ff25800c760 tx=0x7ff25800cc20 comp rx=0 tx=0).stop 2024-06-29T15:58:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 -- 172.21.15.81:0/1365342876 shutdown_connections 2024-06-29T15:58:46.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff264143860 0x7ff26415d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff2640b59c0 0x7ff26413e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7ff248076540 0x7ff248078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 --2- 172.21.15.81:0/1365342876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff264073450 0x7ff264159330 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 -- 172.21.15.81:0/1365342876 >> 172.21.15.81:0/1365342876 conn(0x7ff2640d0b70 msgr2=0x7ff264068960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:46.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 -- 172.21.15.81:0/1365342876 shutdown_connections 2024-06-29T15:58:46.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.358+0000 7ff26ad43640 1 -- 172.21.15.81:0/1365342876 wait complete. 2024-06-29T15:58:46.370 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:58:46.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.487+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3135034011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf40d0770 msgr2=0x7fbcf40ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.487+0000 7fbcfa12b640 1 --2- 172.21.15.81:0/3135034011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf40d0770 0x7fbcf40ce520 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4073550 tx=0x7fbcec02aa30 comp rx=0 tx=0).stop 2024-06-29T15:58:46.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.487+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3135034011 shutdown_connections 2024-06-29T15:58:46.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.487+0000 7fbcfa12b640 1 --2- 172.21.15.81:0/3135034011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf40d0770 0x7fbcf40ce520 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.487+0000 7fbcfa12b640 1 --2- 172.21.15.81:0/3135034011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf40cccf0 0x7fbcf40d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.487+0000 7fbcfa12b640 1 --2- 172.21.15.81:0/3135034011 >> v2:172.21.15.167:3300/0 conn(0x7fbcf40b3200 0x7fbcf40cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.487+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3135034011 >> 172.21.15.81:0/3135034011 conn(0x7fbcf40b3680 msgr2=0x7fbcf40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:46.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.487+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3135034011 shutdown_connections 2024-06-29T15:58:46.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.487+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3135034011 wait complete. 2024-06-29T15:58:46.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.489+0000 7fbcfa12b640 1 Processor -- start 2024-06-29T15:58:46.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.489+0000 7fbcfa12b640 1 -- start start 2024-06-29T15:58:46.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.490+0000 7fbcfa12b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf40b3200 0x7fbcf4155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.490+0000 7fbcfa12b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf40cccf0 0x7fbcf409e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.490+0000 7fbcfa12b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf40d0770 0x7fbcf41e36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.490+0000 7fbcfa12b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf40cef90 con 0x7fbcf40cccf0 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.490+0000 7fbcfa12b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf40cee10 con 0x7fbcf40b3200 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.490+0000 7fbcfa12b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf40cf110 con 0x7fbcf40d0770 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf40cccf0 0x7fbcf409e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf40cccf0 0x7fbcf409e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53422/0 (socket says 172.21.15.81:53422) 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf2ffd640 1 -- 172.21.15.81:0/3618091377 learned_addr learned my addr 172.21.15.81:0/3618091377 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf3fff640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf40d0770 0x7fbcf41e36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf37fe640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf40b3200 0x7fbcf4155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:46.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf3fff640 1 -- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf40b3200 msgr2=0x7fbcf4155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf3fff640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf40b3200 0x7fbcf4155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf3fff640 1 -- 172.21.15.81:0/3618091377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf40cccf0 msgr2=0x7fbcf409e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf3fff640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf40cccf0 0x7fbcf409e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf3fff640 1 -- 172.21.15.81:0/3618091377 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf41e4780 con 0x7fbcf40d0770 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf2ffd640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf40cccf0 0x7fbcf409e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcf3fff640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf40d0770 0x7fbcf41e36a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40d2b40 tx=0x7fbcec003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcdbfff640 1 -- 172.21.15.81:0/3618091377 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec033050 con 0x7fbcf40d0770 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcdbfff640 1 -- 172.21.15.81:0/3618091377 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbcec004450 con 0x7fbcf40d0770 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.491+0000 7fbcdbfff640 1 -- 172.21.15.81:0/3618091377 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec004860 con 0x7fbcf40d0770 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.492+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3618091377 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf41e4a40 con 0x7fbcf40d0770 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.492+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3618091377 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf41e5000 con 0x7fbcf40d0770 2024-06-29T15:58:46.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.492+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3618091377 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcf40d0230 con 0x7fbcf40d0770 2024-06-29T15:58:46.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.493+0000 7fbcdbfff640 1 -- 172.21.15.81:0/3618091377 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7fbcec036070 con 0x7fbcf40d0770 2024-06-29T15:58:46.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.494+0000 7fbcdbfff640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fbce0076540 0x7fbce0078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:46.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.494+0000 7fbcdbfff640 1 -- 172.21.15.81:0/3618091377 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7fbcec0b9560 con 0x7fbcf40d0770 2024-06-29T15:58:46.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.494+0000 7fbcf37fe640 1 -- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fbce0076540 msgr2=0x7fbce0078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:46.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.494+0000 7fbcf37fe640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fbce0076540 0x7fbce0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:46.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.498+0000 7fbcdbfff640 1 -- 172.21.15.81:0/3618091377 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbcec081d50 con 0x7fbcf40d0770 2024-06-29T15:58:46.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.695+0000 7fbcf37fe640 1 -- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fbce0076540 msgr2=0x7fbce0078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:46.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.695+0000 7fbcf37fe640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fbce0076540 0x7fbce0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:46.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.699+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3618091377 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbcf409ee80 con 0x7fbcf40d0770 2024-06-29T15:58:46.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.700+0000 7fbcdbfff640 1 -- 172.21.15.81:0/3618091377 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v238) ==== 81+0+594 (secure 0 0 0) 0x7fbcec08a390 con 0x7fbcf40d0770 2024-06-29T15:58:46.700 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1116386548 2024-06-30T15:58:42.086693+0000 2024-06-29T15:58:46.700 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/3661607921 2024-06-30T15:58:42.086693+0000 2024-06-29T15:58:46.700 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3057737710 2024-06-30T15:58:42.086693+0000 2024-06-29T15:58:46.701 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/3661607921 2024-06-30T15:58:42.086693+0000 2024-06-29T15:58:46.701 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1418041738 2024-06-30T15:58:42.086693+0000 2024-06-29T15:58:46.701 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2207847535 2024-06-30T15:58:42.086693+0000 2024-06-29T15:58:46.701 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/874327121 2024-06-30T15:58:42.086693+0000 2024-06-29T15:58:46.701 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/273305307 2024-06-30T15:58:42.086693+0000 2024-06-29T15:58:46.701 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2269612087 2024-06-30T15:58:42.086693+0000 2024-06-29T15:58:46.701 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1188031263 2024-06-30T15:58:42.086693+0000 2024-06-29T15:58:46.701 INFO:teuthology.orchestra.run.smithi081.stderr:listed 10 entries 2024-06-29T15:58:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fbce0076540 msgr2=0x7fbce0078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fbce0076540 0x7fbce0078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf40d0770 msgr2=0x7fbcf41e36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf40d0770 0x7fbcf41e36a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbcf40d2b40 tx=0x7fbcec003f10 comp rx=0 tx=0).stop 2024-06-29T15:58:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3618091377 shutdown_connections 2024-06-29T15:58:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf40d0770 0x7fbcf41e36a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7fbce0076540 0x7fbce0078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf40cccf0 0x7fbcf409e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 --2- 172.21.15.81:0/3618091377 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf40b3200 0x7fbcf4155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:46.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3618091377 >> 172.21.15.81:0/3618091377 conn(0x7fbcf40b3680 msgr2=0x7fbcf4070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3618091377 shutdown_connections 2024-06-29T15:58:46.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:46.702+0000 7fbcfa12b640 1 -- 172.21.15.81:0/3618091377 wait complete. 2024-06-29T15:58:46.714 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:58:46.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:46.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:58:46.742 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T15:58:46.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:58:46.742 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:58:46.798 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:58:46.798 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:58:46.798 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:46.814 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:58:46.814 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:58:46.815 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:58:46.815 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:46.854 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:58:46.854 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:58:46.854 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:58:46.854 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:58:46.870 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:58:46.870 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:47.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.009+0000 7f625b12e640 1 -- 172.21.15.81:0/2822854207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62540b3200 msgr2=0x7f62540cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.009+0000 7f625b12e640 1 --2- 172.21.15.81:0/2822854207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62540b3200 0x7f62540cc7b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6248009830 tx=0x7f624802a510 comp rx=0 tx=0).stop 2024-06-29T15:58:47.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.009+0000 7f625b12e640 1 -- 172.21.15.81:0/2822854207 shutdown_connections 2024-06-29T15:58:47.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.009+0000 7f625b12e640 1 --2- 172.21.15.81:0/2822854207 >> v2:172.21.15.167:3300/0 conn(0x7f62540cccf0 0x7f62540d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.009+0000 7f625b12e640 1 --2- 172.21.15.81:0/2822854207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62540b3200 0x7f62540cc7b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.009+0000 7f625b12e640 1 -- 172.21.15.81:0/2822854207 >> 172.21.15.81:0/2822854207 conn(0x7f62540b3680 msgr2=0x7f62540b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:47.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.009+0000 7f625b12e640 1 -- 172.21.15.81:0/2822854207 shutdown_connections 2024-06-29T15:58:47.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.009+0000 7f625b12e640 1 -- 172.21.15.81:0/2822854207 wait complete. 2024-06-29T15:58:47.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.010+0000 7f625b12e640 1 Processor -- start 2024-06-29T15:58:47.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.010+0000 7f625b12e640 1 -- start start 2024-06-29T15:58:47.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f625b12e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540b3200 0x7f62541f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f625b12e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62540cccf0 0x7f62541f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f625b12e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62540d0770 0x7f62540c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f625b12e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62540cedb0 con 0x7f62540cccf0 2024-06-29T15:58:47.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f625b12e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f62540cec30 con 0x7f62540b3200 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f625b12e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f62540cef30 con 0x7f62540d0770 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f6258eca640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540b3200 0x7f62541f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f6253fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62540cccf0 0x7f62541f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f6258eca640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540b3200 0x7f62541f5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3300/0 says I am v2:172.21.15.81:51964/0 (socket says 172.21.15.81:51964) 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f6258eca640 1 -- 172.21.15.81:0/2696724480 learned_addr learned my addr 172.21.15.81:0/2696724480 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f62596cb640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62540d0770 0x7f62540c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f6258eca640 1 -- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62540d0770 msgr2=0x7f62540c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f6258eca640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62540d0770 0x7f62540c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f6258eca640 1 -- 172.21.15.81:0/2696724480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62540cccf0 msgr2=0x7f62541f26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f6258eca640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62540cccf0 0x7f62541f26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.012+0000 7f6258eca640 1 -- 172.21.15.81:0/2696724480 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62540c9f50 con 0x7f62540b3200 2024-06-29T15:58:47.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.013+0000 7f6258eca640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540b3200 0x7f62541f5170 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f624802ab00 tx=0x7f6248033c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.013+0000 7f62517fa640 1 -- 172.21.15.81:0/2696724480 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6248031d60 con 0x7f62540b3200 2024-06-29T15:58:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.013+0000 7f62517fa640 1 -- 172.21.15.81:0/2696724480 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6248033e10 con 0x7f62540b3200 2024-06-29T15:58:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.013+0000 7f625b12e640 1 -- 172.21.15.81:0/2696724480 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62540ca220 con 0x7f62540b3200 2024-06-29T15:58:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.013+0000 7f62517fa640 1 -- 172.21.15.81:0/2696724480 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6248005260 con 0x7f62540b3200 2024-06-29T15:58:47.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.013+0000 7f625b12e640 1 -- 172.21.15.81:0/2696724480 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62540ca7e0 con 0x7f62540b3200 2024-06-29T15:58:47.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.014+0000 7f625b12e640 1 -- 172.21.15.81:0/2696724480 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62540cc7b0 con 0x7f62540b3200 2024-06-29T15:58:47.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.015+0000 7f62517fa640 1 -- 172.21.15.81:0/2696724480 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7f6248005400 con 0x7f62540b3200 2024-06-29T15:58:47.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.015+0000 7f62517fa640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f6238076510 0x7f6238078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.015+0000 7f6253fff640 1 -- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f6238076510 msgr2=0x7f6238078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:47.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.015+0000 7f62517fa640 1 -- 172.21.15.81:0/2696724480 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7f62480b91b0 con 0x7f62540b3200 2024-06-29T15:58:47.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.015+0000 7f6253fff640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f6238076510 0x7f6238078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:47.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.020+0000 7f62517fa640 1 -- 172.21.15.81:0/2696724480 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f624803f030 con 0x7f62540b3200 2024-06-29T15:58:47.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.217+0000 7f6253fff640 1 -- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f6238076510 msgr2=0x7f6238078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:47.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.217+0000 7f6253fff640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f6238076510 0x7f6238078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:58:47.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.245+0000 7f625b12e640 1 -- 172.21.15.81:0/2696724480 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62541e7f50 con 0x7f62540b3200 2024-06-29T15:58:47.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.245+0000 7f62517fa640 1 -- 172.21.15.81:0/2696724480 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6562 (secure 0 0 0) 0x7f624808a1d0 con 0x7f62540b3200 2024-06-29T15:58:47.246 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:47.246 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":135,"btime":"2024-06-29T15:58:34:407667+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:18.654579+0000","modified":"2024-06-29T15:58:21.679033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12105":{"gid":12105,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/30158598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":30158598},{"type":"v1","addr":"172.21.15.167:6837","nonce":30158598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12105,"qdb_cluster":[12105]},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:30.737083+0000","modified":"2024-06-29T15:58:33.773132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12093":{"gid":12093,"name":"a","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12093,"qdb_cluster":[12093]},"id":16}]} 2024-06-29T15:58:47.246 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 135 2024-06-29T15:58:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.247+0000 7f625b12e640 1 -- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f6238076510 msgr2=0x7f6238078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.247+0000 7f625b12e640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f6238076510 0x7f6238078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.247+0000 7f625b12e640 1 -- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540b3200 msgr2=0x7f62541f5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.247+0000 7f625b12e640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540b3200 0x7f62541f5170 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f624802ab00 tx=0x7f6248033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.247+0000 7f625b12e640 1 -- 172.21.15.81:0/2696724480 shutdown_connections 2024-06-29T15:58:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.247+0000 7f625b12e640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f62540d0770 0x7f62540c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.247+0000 7f625b12e640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f6238076510 0x7f6238078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.247+0000 7f625b12e640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62540cccf0 0x7f62541f26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.247+0000 7f625b12e640 1 --2- 172.21.15.81:0/2696724480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f62540b3200 0x7f62541f5170 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.247+0000 7f625b12e640 1 -- 172.21.15.81:0/2696724480 >> 172.21.15.81:0/2696724480 conn(0x7f62540b3680 msgr2=0x7f62541e8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:47.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.248+0000 7f625b12e640 1 -- 172.21.15.81:0/2696724480 shutdown_connections 2024-06-29T15:58:47.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.248+0000 7f625b12e640 1 -- 172.21.15.81:0/2696724480 wait complete. 2024-06-29T15:58:47.259 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:47.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.376+0000 7efdbd2b2640 1 -- 172.21.15.81:0/226024761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdb815aed0 msgr2=0x7efdb80b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.376+0000 7efdbd2b2640 1 --2- 172.21.15.81:0/226024761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdb815aed0 0x7efdb80b4070 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009be0 tx=0x7efdb002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.376+0000 7efdbd2b2640 1 -- 172.21.15.81:0/226024761 shutdown_connections 2024-06-29T15:58:47.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.376+0000 7efdbd2b2640 1 --2- 172.21.15.81:0/226024761 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdb815aed0 0x7efdb80b4070 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.376+0000 7efdbd2b2640 1 -- 172.21.15.81:0/226024761 >> 172.21.15.81:0/226024761 conn(0x7efdb80d0930 msgr2=0x7efdb80d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:47.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.376+0000 7efdbd2b2640 1 -- 172.21.15.81:0/226024761 shutdown_connections 2024-06-29T15:58:47.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.376+0000 7efdbd2b2640 1 -- 172.21.15.81:0/226024761 wait complete. 2024-06-29T15:58:47.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.378+0000 7efdbd2b2640 1 Processor -- start 2024-06-29T15:58:47.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.378+0000 7efdbd2b2640 1 -- start start 2024-06-29T15:58:47.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdbd2b2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb806f6a0 0x7efdb80acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdbd2b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdb80b57b0 0x7efdb80aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdbd2b2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdb815aed0 0x7efdb81f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdbd2b2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efdb80b4730 con 0x7efdb80b57b0 2024-06-29T15:58:47.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdbd2b2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efdb80b45b0 con 0x7efdb806f6a0 2024-06-29T15:58:47.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdbd2b2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efdb80b48b0 con 0x7efdb815aed0 2024-06-29T15:58:47.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdb67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdb80b57b0 0x7efdb80aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdb6ffd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb806f6a0 0x7efdb80acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdb67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdb80b57b0 0x7efdb80aed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53450/0 (socket says 172.21.15.81:53450) 2024-06-29T15:58:47.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdb67fc640 1 -- 172.21.15.81:0/4234727317 learned_addr learned my addr 172.21.15.81:0/4234727317 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.379+0000 7efdb77fe640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdb815aed0 0x7efdb81f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efdb6ffd640 1 -- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdb815aed0 msgr2=0x7efdb81f5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efdb6ffd640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdb815aed0 0x7efdb81f5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efdb6ffd640 1 -- 172.21.15.81:0/4234727317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdb80b57b0 msgr2=0x7efdb80aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efdb6ffd640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdb80b57b0 0x7efdb80aed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efdb6ffd640 1 -- 172.21.15.81:0/4234727317 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb80ad340 con 0x7efdb806f6a0 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efdb77fe640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdb815aed0 0x7efdb81f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efdb6ffd640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb806f6a0 0x7efdb80acb10 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7efdac00c3c0 tx=0x7efdac00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efd9f7fe640 1 -- 172.21.15.81:0/4234727317 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac017070 con 0x7efdb806f6a0 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efd9f7fe640 1 -- 172.21.15.81:0/4234727317 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efdac005400 con 0x7efdb806f6a0 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efdbd2b2640 1 -- 172.21.15.81:0/4234727317 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb80ad620 con 0x7efdb806f6a0 2024-06-29T15:58:47.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efd9f7fe640 1 -- 172.21.15.81:0/4234727317 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac0118b0 con 0x7efdb806f6a0 2024-06-29T15:58:47.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.380+0000 7efdbd2b2640 1 -- 172.21.15.81:0/4234727317 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb80adbe0 con 0x7efdb806f6a0 2024-06-29T15:58:47.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.381+0000 7efdbd2b2640 1 -- 172.21.15.81:0/4234727317 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdb806e2a0 con 0x7efdb806f6a0 2024-06-29T15:58:47.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.382+0000 7efd9f7fe640 1 -- 172.21.15.81:0/4234727317 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7efdac01a070 con 0x7efdb806f6a0 2024-06-29T15:58:47.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.382+0000 7efd9f7fe640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7efda4076540 0x7efda4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.382+0000 7efdb67fc640 1 -- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7efda4076540 msgr2=0x7efda4078a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:47.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.383+0000 7efd9f7fe640 1 -- 172.21.15.81:0/4234727317 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7efdac0973c0 con 0x7efdb806f6a0 2024-06-29T15:58:47.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.383+0000 7efdb67fc640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7efda4076540 0x7efda4078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:47.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.387+0000 7efd9f7fe640 1 -- 172.21.15.81:0/4234727317 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efdb806e2a0 con 0x7efdb806f6a0 2024-06-29T15:58:47.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.578+0000 7efdbd2b2640 1 -- 172.21.15.81:0/4234727317 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efdb81584e0 con 0x7efdb806f6a0 2024-06-29T15:58:47.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.580+0000 7efd9f7fe640 1 -- 172.21.15.81:0/4234727317 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) ==== 74+0+21178 (secure 0 0 0) 0x7efdac05fe90 con 0x7efdb806f6a0 2024-06-29T15:58:47.581 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:47.581 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":238,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:42.086750+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:29.291185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:29.702735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/1116386548":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6832/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/3057737710":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6833/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1418041738":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2207847535":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/874327121":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/273305307":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2269612087":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1188031263":"2024-06-30T15:58:42.086693+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 -- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7efda4076540 msgr2=0x7efda4078a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7efda4076540 0x7efda4078a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 -- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb806f6a0 msgr2=0x7efdb80acb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb806f6a0 0x7efdb80acb10 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7efdac00c3c0 tx=0x7efdac00c880 comp rx=0 tx=0).stop 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 -- 172.21.15.81:0/4234727317 shutdown_connections 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efdb815aed0 0x7efdb81f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7efda4076540 0x7efda4078a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdb80b57b0 0x7efdb80aed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 --2- 172.21.15.81:0/4234727317 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efdb806f6a0 0x7efdb80acb10 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 -- 172.21.15.81:0/4234727317 >> 172.21.15.81:0/4234727317 conn(0x7efdb80d0930 msgr2=0x7efdb8159500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:47.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 -- 172.21.15.81:0/4234727317 shutdown_connections 2024-06-29T15:58:47.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.582+0000 7efdbd2b2640 1 -- 172.21.15.81:0/4234727317 wait complete. 2024-06-29T15:58:47.594 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:58:16.949415+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:58:17.618219+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:58:29.291185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:58:29.702735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '237', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:47.595 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:47.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.711+0000 7f5cc11d2640 1 -- 172.21.15.81:0/1482151076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ca4005150 msgr2=0x7f5ca400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.711+0000 7f5cc11d2640 1 --2- 172.21.15.81:0/1482151076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ca4005150 0x7f5ca400b9c0 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400c760 tx=0x7f5ca4005830 comp rx=0 tx=0).stop 2024-06-29T15:58:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.711+0000 7f5cbb59e640 1 -- 172.21.15.81:0/1482151076 reap_dead start 2024-06-29T15:58:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.711+0000 7f5cc11d2640 1 -- 172.21.15.81:0/1482151076 shutdown_connections 2024-06-29T15:58:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.711+0000 7f5cc11d2640 1 -- 172.21.15.81:0/1482151076 >> 172.21.15.81:0/1482151076 conn(0x7f5cbc0b2920 msgr2=0x7f5cbc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.711+0000 7f5cc11d2640 1 -- 172.21.15.81:0/1482151076 shutdown_connections 2024-06-29T15:58:47.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.711+0000 7f5cc11d2640 1 -- 172.21.15.81:0/1482151076 wait complete. 2024-06-29T15:58:47.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.713+0000 7f5cc11d2640 1 Processor -- start 2024-06-29T15:58:47.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.713+0000 7f5cc11d2640 1 -- start start 2024-06-29T15:58:47.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cc11d2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5ca4005150 0x7f5cbc139db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cc11d2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cbc15a000 0x7f5cbc0ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cc11d2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cbc1400f0 0x7f5cbc141db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cc11d2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5cbc144920 con 0x7f5cbc15a000 2024-06-29T15:58:47.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cc11d2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5cbc1447a0 con 0x7f5ca4005150 2024-06-29T15:58:47.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cc11d2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5cbc06edb0 con 0x7f5cbc1400f0 2024-06-29T15:58:47.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cba59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cbc15a000 0x7f5cbc0ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cba59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cbc15a000 0x7f5cbc0ae620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53462/0 (socket says 172.21.15.81:53462) 2024-06-29T15:58:47.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cbad9d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5ca4005150 0x7f5cbc139db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cba59c640 1 -- 172.21.15.81:0/3745685374 learned_addr learned my addr 172.21.15.81:0/3745685374 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cbb59e640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cbc1400f0 0x7f5cbc141db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cba59c640 1 -- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cbc1400f0 msgr2=0x7f5cbc141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cba59c640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cbc1400f0 0x7f5cbc141db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cba59c640 1 -- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5ca4005150 msgr2=0x7f5cbc139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cba59c640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5ca4005150 0x7f5cbc139db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cba59c640 1 -- 172.21.15.81:0/3745685374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cbc15a410 con 0x7f5cbc15a000 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cba59c640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cbc15a000 0x7f5cbc0ae620 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f5cac011ae0 tx=0x7f5cac011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5ca37fe640 1 -- 172.21.15.81:0/3745685374 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac004460 con 0x7f5cbc15a000 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5cc11d2640 1 -- 172.21.15.81:0/3745685374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cbc140d10 con 0x7f5cbc15a000 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.715+0000 7f5ca37fe640 1 -- 172.21.15.81:0/3745685374 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cac00f920 con 0x7f5cbc15a000 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.716+0000 7f5ca37fe640 1 -- 172.21.15.81:0/3745685374 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac00fd50 con 0x7f5cbc15a000 2024-06-29T15:58:47.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.716+0000 7f5cc11d2640 1 -- 172.21.15.81:0/3745685374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cbc1412d0 con 0x7f5cbc15a000 2024-06-29T15:58:47.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.717+0000 7f5cc11d2640 1 -- 172.21.15.81:0/3745685374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cbc072f80 con 0x7f5cbc15a000 2024-06-29T15:58:47.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.717+0000 7f5ca37fe640 1 -- 172.21.15.81:0/3745685374 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 142) ==== 98406+0+0 (secure 0 0 0) 0x7f5cac0035f0 con 0x7f5cbc15a000 2024-06-29T15:58:47.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.718+0000 7f5ca37fe640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f5ca8076510 0x7f5ca8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:47.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.718+0000 7f5cbad9d640 1 -- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f5ca8076510 msgr2=0x7f5ca8078a20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/884808428 2024-06-29T15:58:47.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.718+0000 7f5ca37fe640 1 -- 172.21.15.81:0/3745685374 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 7131+0+0 (secure 0 0 0) 0x7f5cac098e70 con 0x7f5cbc15a000 2024-06-29T15:58:47.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.718+0000 7f5cbad9d640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f5ca8076510 0x7f5ca8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:58:47.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.722+0000 7f5ca37fe640 1 -- 172.21.15.81:0/3745685374 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5cac061670 con 0x7f5cbc15a000 2024-06-29T15:58:47.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.827+0000 7f5ca37fe640 1 -- 172.21.15.81:0/3745685374 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 143) ==== 97912+0+0 (secure 0 0 0) 0x7f5cac05d900 con 0x7f5cbc15a000 2024-06-29T15:58:47.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.827+0000 7f5ca37fe640 1 -- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f5ca8076510 msgr2=0x7f5ca8078a20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:58:47.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.827+0000 7f5ca37fe640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f5ca8076510 0x7f5ca8078a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.884 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.883+0000 7f6962448640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.884 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.883+0000 7f6962448640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.884 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.883+0000 7f6962448640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.884 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.883+0000 7f6962448640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.884 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.883+0000 7f6962448640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.884 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.883+0000 7f695f442640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.883+0000 7f695f442640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.883+0000 7f695f442640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.883+0000 7f695f442640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.885 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.883+0000 7f695f442640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.893 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.892+0000 7f6962448640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.894 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.893+0000 7f6962448640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.894 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.893+0000 7f6962448640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.894 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.893+0000 7f6962448640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.894 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.893+0000 7f6962448640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.894 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.893+0000 7f695f442640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.894 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.893+0000 7f695f442640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.894 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.893+0000 7f695f442640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.894 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.893+0000 7f695f442640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.894 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:58:47.893+0000 7f695f442640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:58:47.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.928+0000 7f5cc11d2640 1 -- 172.21.15.81:0/3745685374 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5cbc0d1850 con 0x7f5cbc15a000 2024-06-29T15:58:47.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.928+0000 7f5ca37fe640 1 -- 172.21.15.81:0/3745685374 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v239) ==== 93+0+10 (secure 0 0 0) 0x7f5cac0960b0 con 0x7f5cbc15a000 2024-06-29T15:58:47.928 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:58:47.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.930+0000 7f5cc11d2640 1 -- 172.21.15.81:0/3745685374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cbc15a000 msgr2=0x7f5cbc0ae620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:47.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.930+0000 7f5cc11d2640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cbc15a000 0x7f5cbc0ae620 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f5cac011ae0 tx=0x7f5cac011ee0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.930+0000 7f5cc11d2640 1 -- 172.21.15.81:0/3745685374 shutdown_connections 2024-06-29T15:58:47.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.930+0000 7f5cc11d2640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5cbc1400f0 0x7f5cbc141db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.930+0000 7f5cc11d2640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:6832/884808428,v1:172.21.15.167:6833/884808428] conn(0x7f5ca8076510 0x7f5ca8078a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.930+0000 7f5cc11d2640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cbc15a000 0x7f5cbc0ae620 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.930+0000 7f5cc11d2640 1 --2- 172.21.15.81:0/3745685374 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5ca4005150 0x7f5cbc139db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:47.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.930+0000 7f5cc11d2640 1 -- 172.21.15.81:0/3745685374 >> 172.21.15.81:0/3745685374 conn(0x7f5cbc0b2920 msgr2=0x7f5cbc0cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:47.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.930+0000 7f5cc11d2640 1 -- 172.21.15.81:0/3745685374 shutdown_connections 2024-06-29T15:58:47.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:47.930+0000 7f5cc11d2640 1 -- 172.21.15.81:0/3745685374 wait complete. 2024-06-29T15:58:47.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:58:48.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.059+0000 7fe4d4e47640 1 -- 172.21.15.81:0/4040046912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4d006ef90 msgr2=0x7fe4d006f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.059+0000 7fe4d4e47640 1 --2- 172.21.15.81:0/4040046912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4d006ef90 0x7fe4d006f3a0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0071e60 tx=0x7fe4b802a530 comp rx=0 tx=0).stop 2024-06-29T15:58:48.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.059+0000 7fe4d4e47640 1 -- 172.21.15.81:0/4040046912 shutdown_connections 2024-06-29T15:58:48.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.059+0000 7fe4d4e47640 1 --2- 172.21.15.81:0/4040046912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4d006ef90 0x7fe4d006f3a0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.059+0000 7fe4d4e47640 1 --2- 172.21.15.81:0/4040046912 >> v2:172.21.15.167:3300/0 conn(0x7fe4d00b57b0 0x7fe4d015aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.059+0000 7fe4d4e47640 1 -- 172.21.15.81:0/4040046912 >> 172.21.15.81:0/4040046912 conn(0x7fe4d00d0930 msgr2=0x7fe4d00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.059+0000 7fe4d4e47640 1 -- 172.21.15.81:0/4040046912 shutdown_connections 2024-06-29T15:58:48.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.059+0000 7fe4d4e47640 1 -- 172.21.15.81:0/4040046912 wait complete. 2024-06-29T15:58:48.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.061+0000 7fe4d4e47640 1 Processor -- start 2024-06-29T15:58:48.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.061+0000 7fe4d4e47640 1 -- start start 2024-06-29T15:58:48.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4d4e47640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4d006ef90 0x7fe4d01f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4d4e47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4d00b57b0 0x7fe4d01e97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4d4e47640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe4d00ce1d0 0x7fe4d00aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4d4e47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d00b4b30 con 0x7fe4d00b57b0 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4d4e47640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d00b49b0 con 0x7fe4d00ce1d0 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4d4e47640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d0154be0 con 0x7fe4d006ef90 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4cdd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4d00b57b0 0x7fe4d01e97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4cdd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4d00b57b0 0x7fe4d01e97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53482/0 (socket says 172.21.15.81:53482) 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4cdd9b640 1 -- 172.21.15.81:0/768784305 learned_addr learned my addr 172.21.15.81:0/768784305 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4ce59c640 1 --2- 172.21.15.81:0/768784305 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4d006ef90 0x7fe4d01f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.062+0000 7fe4ced9d640 1 --2- 172.21.15.81:0/768784305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe4d00ce1d0 0x7fe4d00aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4cdd9b640 1 -- 172.21.15.81:0/768784305 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4d006ef90 msgr2=0x7fe4d01f2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4cdd9b640 1 --2- 172.21.15.81:0/768784305 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4d006ef90 0x7fe4d01f2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4cdd9b640 1 -- 172.21.15.81:0/768784305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe4d00ce1d0 msgr2=0x7fe4d00aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4cdd9b640 1 --2- 172.21.15.81:0/768784305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe4d00ce1d0 0x7fe4d00aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4cdd9b640 1 -- 172.21.15.81:0/768784305 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d01f3710 con 0x7fe4d00b57b0 2024-06-29T15:58:48.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4cdd9b640 1 --2- 172.21.15.81:0/768784305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4d00b57b0 0x7fe4d01e97f0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009780 tx=0x7fe4b8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:48.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4c6ffd640 1 -- 172.21.15.81:0/768784305 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8031d60 con 0x7fe4d00b57b0 2024-06-29T15:58:48.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4c6ffd640 1 -- 172.21.15.81:0/768784305 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4b8033e10 con 0x7fe4d00b57b0 2024-06-29T15:58:48.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4d4e47640 1 -- 172.21.15.81:0/768784305 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d01f39f0 con 0x7fe4d00b57b0 2024-06-29T15:58:48.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4c6ffd640 1 -- 172.21.15.81:0/768784305 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8005260 con 0x7fe4d00b57b0 2024-06-29T15:58:48.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.063+0000 7fe4d4e47640 1 -- 172.21.15.81:0/768784305 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d0153380 con 0x7fe4d00b57b0 2024-06-29T15:58:48.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.064+0000 7fe4d4e47640 1 -- 172.21.15.81:0/768784305 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4d006e2a0 con 0x7fe4d00b57b0 2024-06-29T15:58:48.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.065+0000 7fe4c6ffd640 1 -- 172.21.15.81:0/768784305 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 143) ==== 97912+0+0 (secure 0 0 0) 0x7fe4b8005400 con 0x7fe4d00b57b0 2024-06-29T15:58:48.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.065+0000 7fe4c6ffd640 1 -- 172.21.15.81:0/768784305 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 7561+0+0 (secure 0 0 0) 0x7fe4b80b8af0 con 0x7fe4d00b57b0 2024-06-29T15:58:48.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.069+0000 7fe4c6ffd640 1 -- 172.21.15.81:0/768784305 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe4d006e2a0 con 0x7fe4d00b57b0 2024-06-29T15:58:48.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.250+0000 7fe4d4e47640 1 -- 172.21.15.81:0/768784305 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe4d00cf020 con 0x7fe4d00b57b0 2024-06-29T15:58:48.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.250+0000 7fe4c6ffd640 1 -- 172.21.15.81:0/768784305 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v239) ==== 104+0+11 (secure 0 0 0) 0x7fe4b803f030 con 0x7fe4d00b57b0 2024-06-29T15:58:48.250 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:58:48.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.252+0000 7fe4d4e47640 1 -- 172.21.15.81:0/768784305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4d00b57b0 msgr2=0x7fe4d01e97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.252+0000 7fe4d4e47640 1 --2- 172.21.15.81:0/768784305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4d00b57b0 0x7fe4d01e97f0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009780 tx=0x7fe4b8033c20 comp rx=0 tx=0).stop 2024-06-29T15:58:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.252+0000 7fe4d4e47640 1 -- 172.21.15.81:0/768784305 shutdown_connections 2024-06-29T15:58:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.252+0000 7fe4d4e47640 1 --2- 172.21.15.81:0/768784305 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe4d00ce1d0 0x7fe4d00aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.252+0000 7fe4d4e47640 1 --2- 172.21.15.81:0/768784305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4d00b57b0 0x7fe4d01e97f0 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.252+0000 7fe4d4e47640 1 --2- 172.21.15.81:0/768784305 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe4d006ef90 0x7fe4d01f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.252+0000 7fe4d4e47640 1 -- 172.21.15.81:0/768784305 >> 172.21.15.81:0/768784305 conn(0x7fe4d00d0930 msgr2=0x7fe4d0159910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.252+0000 7fe4d4e47640 1 -- 172.21.15.81:0/768784305 shutdown_connections 2024-06-29T15:58:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.252+0000 7fe4d4e47640 1 -- 172.21.15.81:0/768784305 wait complete. 2024-06-29T15:58:48.263 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:58:48.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.379+0000 7f99b241f640 1 -- 172.21.15.81:0/3369091338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99ac06ef90 msgr2=0x7f99ac06f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.379+0000 7f99b241f640 1 --2- 172.21.15.81:0/3369091338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99ac06ef90 0x7f99ac06f3a0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f99a0005fd0 tx=0x7f99a002a530 comp rx=0 tx=0).stop 2024-06-29T15:58:48.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.379+0000 7f99b241f640 1 -- 172.21.15.81:0/3369091338 shutdown_connections 2024-06-29T15:58:48.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.379+0000 7f99b241f640 1 --2- 172.21.15.81:0/3369091338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99ac06ef90 0x7f99ac06f3a0 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.379+0000 7f99b241f640 1 --2- 172.21.15.81:0/3369091338 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99ac0b57b0 0x7f99ac15aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.379+0000 7f99b241f640 1 -- 172.21.15.81:0/3369091338 >> 172.21.15.81:0/3369091338 conn(0x7f99ac0d0930 msgr2=0x7f99ac0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.379+0000 7f99b241f640 1 -- 172.21.15.81:0/3369091338 shutdown_connections 2024-06-29T15:58:48.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.379+0000 7f99b241f640 1 -- 172.21.15.81:0/3369091338 wait complete. 2024-06-29T15:58:48.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.381+0000 7f99b241f640 1 Processor -- start 2024-06-29T15:58:48.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.381+0000 7f99b241f640 1 -- start start 2024-06-29T15:58:48.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.382+0000 7f99b241f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99ac06ef90 0x7f99ac1f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.382+0000 7f99b241f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99ac0b57b0 0x7f99ac14b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.382+0000 7f99b241f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99ac0ce1d0 0x7f99ac06c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.382+0000 7f99b241f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f99ac0cecf0 con 0x7f99ac06ef90 2024-06-29T15:58:48.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.382+0000 7f99b241f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f99ac0b4b80 con 0x7f99ac0b57b0 2024-06-29T15:58:48.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.382+0000 7f99b241f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f99ac0cee70 con 0x7f99ac0ce1d0 2024-06-29T15:58:48.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99abfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99ac06ef90 0x7f99ac1f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99abfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99ac06ef90 0x7f99ac1f38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53506/0 (socket says 172.21.15.81:53506) 2024-06-29T15:58:48.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99abfff640 1 -- 172.21.15.81:0/3098335622 learned_addr learned my addr 172.21.15.81:0/3098335622 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:48.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99b09bc640 1 --2- 172.21.15.81:0/3098335622 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99ac0ce1d0 0x7f99ac06c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99ab7fe640 1 --2- 172.21.15.81:0/3098335622 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99ac0b57b0 0x7f99ac14b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99abfff640 1 -- 172.21.15.81:0/3098335622 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99ac0ce1d0 msgr2=0x7f99ac06c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99abfff640 1 --2- 172.21.15.81:0/3098335622 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99ac0ce1d0 0x7f99ac06c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99abfff640 1 -- 172.21.15.81:0/3098335622 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99ac0b57b0 msgr2=0x7f99ac14b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99abfff640 1 --2- 172.21.15.81:0/3098335622 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99ac0b57b0 0x7f99ac14b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99abfff640 1 -- 172.21.15.81:0/3098335622 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ac1f3de0 con 0x7f99ac06ef90 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99abfff640 1 --2- 172.21.15.81:0/3098335622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99ac06ef90 0x7f99ac1f38a0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f999c00d770 tx=0x7f999c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99a8ff9640 1 -- 172.21.15.81:0/3098335622 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999c017070 con 0x7f99ac06ef90 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99a8ff9640 1 -- 172.21.15.81:0/3098335622 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f999c009d20 con 0x7f99ac06ef90 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99b241f640 1 -- 172.21.15.81:0/3098335622 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ac14bb70 con 0x7f99ac06ef90 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99a8ff9640 1 -- 172.21.15.81:0/3098335622 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f999c004280 con 0x7f99ac06ef90 2024-06-29T15:58:48.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.383+0000 7f99b241f640 1 -- 172.21.15.81:0/3098335622 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ac0abeb0 con 0x7f99ac06ef90 2024-06-29T15:58:48.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.384+0000 7f99b241f640 1 -- 172.21.15.81:0/3098335622 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99ac06e2a0 con 0x7f99ac06ef90 2024-06-29T15:58:48.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.385+0000 7f99a8ff9640 1 -- 172.21.15.81:0/3098335622 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 143) ==== 97912+0+0 (secure 0 0 0) 0x7f999c005020 con 0x7f99ac06ef90 2024-06-29T15:58:48.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.386+0000 7f99a8ff9640 1 -- 172.21.15.81:0/3098335622 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 7561+0+0 (secure 0 0 0) 0x7f999c097c10 con 0x7f99ac06ef90 2024-06-29T15:58:48.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.390+0000 7f99a8ff9640 1 -- 172.21.15.81:0/3098335622 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f99ac06e2a0 con 0x7f99ac06ef90 2024-06-29T15:58:48.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.570+0000 7f99b241f640 1 -- 172.21.15.81:0/3098335622 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f99ac06f3a0 con 0x7f99ac06ef90 2024-06-29T15:58:48.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.570+0000 7f99a8ff9640 1 -- 172.21.15.81:0/3098335622 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v239) ==== 100+0+11 (secure 0 0 0) 0x7f999c095e20 con 0x7f99ac06ef90 2024-06-29T15:58:48.571 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:58:48.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.573+0000 7f99b241f640 1 -- 172.21.15.81:0/3098335622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99ac06ef90 msgr2=0x7f99ac1f38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.573+0000 7f99b241f640 1 --2- 172.21.15.81:0/3098335622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99ac06ef90 0x7f99ac1f38a0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f999c00d770 tx=0x7f999c00dc30 comp rx=0 tx=0).stop 2024-06-29T15:58:48.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.573+0000 7f99b241f640 1 -- 172.21.15.81:0/3098335622 shutdown_connections 2024-06-29T15:58:48.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.573+0000 7f99b241f640 1 --2- 172.21.15.81:0/3098335622 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f99ac0ce1d0 0x7f99ac06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.573+0000 7f99b241f640 1 --2- 172.21.15.81:0/3098335622 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f99ac0b57b0 0x7f99ac14b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.573+0000 7f99b241f640 1 --2- 172.21.15.81:0/3098335622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99ac06ef90 0x7f99ac1f38a0 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.573+0000 7f99b241f640 1 -- 172.21.15.81:0/3098335622 >> 172.21.15.81:0/3098335622 conn(0x7f99ac0d0930 msgr2=0x7f99ac0b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.573+0000 7f99b241f640 1 -- 172.21.15.81:0/3098335622 shutdown_connections 2024-06-29T15:58:48.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.573+0000 7f99b241f640 1 -- 172.21.15.81:0/3098335622 wait complete. 2024-06-29T15:58:48.584 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:58:48.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.701+0000 7f96f8866640 1 -- 172.21.15.81:0/3177726834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96dc004d40 msgr2=0x7f96dc005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.701+0000 7f96f8866640 1 --2- 172.21.15.81:0/3177726834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96dc004d40 0x7f96dc005150 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f96f40b61c0 tx=0x7f96dc005830 comp rx=0 tx=0).stop 2024-06-29T15:58:48.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.701+0000 7f96f2d9d640 1 -- 172.21.15.81:0/3177726834 reap_dead start 2024-06-29T15:58:48.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.701+0000 7f96f8866640 1 -- 172.21.15.81:0/3177726834 shutdown_connections 2024-06-29T15:58:48.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.701+0000 7f96f8866640 1 -- 172.21.15.81:0/3177726834 >> 172.21.15.81:0/3177726834 conn(0x7f96f4072950 msgr2=0x7f96f40b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.701+0000 7f96f8866640 1 -- 172.21.15.81:0/3177726834 shutdown_connections 2024-06-29T15:58:48.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.701+0000 7f96f8866640 1 -- 172.21.15.81:0/3177726834 wait complete. 2024-06-29T15:58:48.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.703+0000 7f96f8866640 1 Processor -- start 2024-06-29T15:58:48.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.703+0000 7f96f8866640 1 -- start start 2024-06-29T15:58:48.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.704+0000 7f96f8866640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96dc004d40 0x7f96f40cf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.704+0000 7f96f8866640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96f41527e0 0x7f96f414e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.704+0000 7f96f8866640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96f41f5050 0x7f96f41f5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f8866640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f96f415b010 con 0x7f96dc004d40 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f8866640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f96f406d150 con 0x7f96f41f5050 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f8866640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f96f415b190 con 0x7f96f41527e0 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96dc004d40 0x7f96f40cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96dc004d40 0x7f96f40cf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53514/0 (socket says 172.21.15.81:53514) 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f259c640 1 -- 172.21.15.81:0/1972019277 learned_addr learned my addr 172.21.15.81:0/1972019277 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f1d9b640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96f41527e0 0x7f96f414e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f259c640 1 -- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96f41527e0 msgr2=0x7f96f414e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f259c640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96f41527e0 0x7f96f414e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f259c640 1 -- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96f41f5050 msgr2=0x7f96f41f5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f259c640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96f41f5050 0x7f96f41f5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f259c640 1 -- 172.21.15.81:0/1972019277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f41f59a0 con 0x7f96dc004d40 2024-06-29T15:58:48.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f259c640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96dc004d40 0x7f96f40cf710 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f96e4011b10 tx=0x7f96e4009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:48.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96e2ffd640 1 -- 172.21.15.81:0/1972019277 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e401e070 con 0x7f96dc004d40 2024-06-29T15:58:48.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96e2ffd640 1 -- 172.21.15.81:0/1972019277 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96e4004460 con 0x7f96dc004d40 2024-06-29T15:58:48.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96e2ffd640 1 -- 172.21.15.81:0/1972019277 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e400fc70 con 0x7f96dc004d40 2024-06-29T15:58:48.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f8866640 1 -- 172.21.15.81:0/1972019277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f4159120 con 0x7f96dc004d40 2024-06-29T15:58:48.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.705+0000 7f96f8866640 1 -- 172.21.15.81:0/1972019277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f41564f0 con 0x7f96dc004d40 2024-06-29T15:58:48.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.706+0000 7f96f8866640 1 -- 172.21.15.81:0/1972019277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96f40af590 con 0x7f96dc004d40 2024-06-29T15:58:48.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.707+0000 7f96e2ffd640 1 -- 172.21.15.81:0/1972019277 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 143) ==== 97912+0+0 (secure 0 0 0) 0x7f96e4021070 con 0x7f96dc004d40 2024-06-29T15:58:48.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.708+0000 7f96e2ffd640 1 -- 172.21.15.81:0/1972019277 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 7561+0+0 (secure 0 0 0) 0x7f96e40982c0 con 0x7f96dc004d40 2024-06-29T15:58:48.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.712+0000 7f96e2ffd640 1 -- 172.21.15.81:0/1972019277 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f96f41f59a0 con 0x7f96dc004d40 2024-06-29T15:58:48.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.826+0000 7f96e2ffd640 1 -- 172.21.15.81:0/1972019277 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 144) ==== 98039+0+0 (secure 0 0 0) 0x7f96e405cd10 con 0x7f96dc004d40 2024-06-29T15:58:48.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.826+0000 7f96e2ffd640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f96d407e3b0 0x7f96d40807d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:48.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.827+0000 7f96f1d9b640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f96d407e3b0 0x7f96d40807d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:48.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.832+0000 7f96f1d9b640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f96d407e3b0 0x7f96d40807d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f96f40b0030 tx=0x7f96e8005f90 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:48.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.895+0000 7f96f8866640 1 -- 172.21.15.81:0/1972019277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7f96f40b0060 con 0x7f96dc004d40 2024-06-29T15:58:48.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.895+0000 7f96e2ffd640 1 -- 172.21.15.81:0/1972019277 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v239) ==== 107+0+11 (secure 0 0 0) 0x7f96e4040ed0 con 0x7f96dc004d40 2024-06-29T15:58:48.896 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:58:48.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 -- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f96d407e3b0 msgr2=0x7f96d40807d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f96d407e3b0 0x7f96d40807d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f96f40b0030 tx=0x7f96e8005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:48.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 -- 172.21.15.81:0/1972019277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96dc004d40 msgr2=0x7f96f40cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:48.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96dc004d40 0x7f96f40cf710 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f96e4011b10 tx=0x7f96e4009e90 comp rx=0 tx=0).stop 2024-06-29T15:58:48.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 -- 172.21.15.81:0/1972019277 shutdown_connections 2024-06-29T15:58:48.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f96d407e3b0 0x7f96d40807d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f96f41f5050 0x7f96f41f5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f96f41527e0 0x7f96f414e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 --2- 172.21.15.81:0/1972019277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96dc004d40 0x7f96f40cf710 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:48.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 -- 172.21.15.81:0/1972019277 >> 172.21.15.81:0/1972019277 conn(0x7f96f4072950 msgr2=0x7f96f40b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:48.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.897+0000 7f96f8866640 1 -- 172.21.15.81:0/1972019277 shutdown_connections 2024-06-29T15:58:48.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:48.898+0000 7f96f8866640 1 -- 172.21.15.81:0/1972019277 wait complete. 2024-06-29T15:58:48.909 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:58:49.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.025+0000 7f2912f43640 1 -- 172.21.15.81:0/4214782827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0ce510 msgr2=0x7f290c0ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.025+0000 7f2912f43640 1 --2- 172.21.15.81:0/4214782827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0ce510 0x7f290c0ce920 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f290c0b41b0 tx=0x7f28f402a470 comp rx=0 tx=0).stop 2024-06-29T15:58:49.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.025+0000 7f2912f43640 1 -- 172.21.15.81:0/4214782827 shutdown_connections 2024-06-29T15:58:49.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.025+0000 7f2912f43640 1 --2- 172.21.15.81:0/4214782827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0ce510 0x7f290c0ce920 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.025+0000 7f2912f43640 1 --2- 172.21.15.81:0/4214782827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c0b6eb0 0x7f290c0d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.025+0000 7f2912f43640 1 -- 172.21.15.81:0/4214782827 >> 172.21.15.81:0/4214782827 conn(0x7f290c0b3510 msgr2=0x7f290c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.025+0000 7f2912f43640 1 -- 172.21.15.81:0/4214782827 shutdown_connections 2024-06-29T15:58:49.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.025+0000 7f2912f43640 1 -- 172.21.15.81:0/4214782827 wait complete. 2024-06-29T15:58:49.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.027+0000 7f2912f43640 1 Processor -- start 2024-06-29T15:58:49.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.027+0000 7f2912f43640 1 -- start start 2024-06-29T15:58:49.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.028+0000 7f2912f43640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0b4810 0x7f290c0cb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f2912f43640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c0b6eb0 0x7f290c1652e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f2912f43640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0ce510 0x7f290c165820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f2912f43640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f290c0cd480 con 0x7f290c0ce510 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f2912f43640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f290c0b29c0 con 0x7f290c0b4810 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f2912f43640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f290c0cd600 con 0x7f290c0b6eb0 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29114e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0ce510 0x7f290c165820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29114e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0ce510 0x7f290c165820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53516/0 (socket says 172.21.15.81:53516) 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29114e0640 1 -- 172.21.15.81:0/3673511363 learned_addr learned my addr 172.21.15.81:0/3673511363 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f2910cdf640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0b4810 0x7f290c0cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f290bfff640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c0b6eb0 0x7f290c1652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29114e0640 1 -- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c0b6eb0 msgr2=0x7f290c1652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29114e0640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c0b6eb0 0x7f290c1652e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29114e0640 1 -- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0b4810 msgr2=0x7f290c0cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29114e0640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0b4810 0x7f290c0cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29114e0640 1 -- 172.21.15.81:0/3673511363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290c0aec80 con 0x7f290c0ce510 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f2910cdf640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0b4810 0x7f290c0cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:49.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29114e0640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0ce510 0x7f290c165820 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f290400c390 tx=0x7f290400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:49.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29097fa640 1 -- 172.21.15.81:0/3673511363 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904011790 con 0x7f290c0ce510 2024-06-29T15:58:49.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29097fa640 1 -- 172.21.15.81:0/3673511363 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2904017070 con 0x7f290c0ce510 2024-06-29T15:58:49.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f2912f43640 1 -- 172.21.15.81:0/3673511363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290c0b0dc0 con 0x7f290c0ce510 2024-06-29T15:58:49.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f29097fa640 1 -- 172.21.15.81:0/3673511363 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904011ce0 con 0x7f290c0ce510 2024-06-29T15:58:49.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.029+0000 7f2912f43640 1 -- 172.21.15.81:0/3673511363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f290c0b1380 con 0x7f290c0ce510 2024-06-29T15:58:49.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.030+0000 7f2912f43640 1 -- 172.21.15.81:0/3673511363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f290c0ae9f0 con 0x7f290c0ce510 2024-06-29T15:58:49.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.031+0000 7f29097fa640 1 -- 172.21.15.81:0/3673511363 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 144) ==== 98039+0+0 (secure 0 0 0) 0x7f290401c460 con 0x7f290c0ce510 2024-06-29T15:58:49.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.032+0000 7f29097fa640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f28ec076120 0x7f28ec078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.032+0000 7f29097fa640 1 -- 172.21.15.81:0/3673511363 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 7561+0+0 (secure 0 0 0) 0x7f29040979e0 con 0x7f290c0ce510 2024-06-29T15:58:49.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.032+0000 7f2910cdf640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f28ec076120 0x7f28ec078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.032+0000 7f2910cdf640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f28ec076120 0x7f28ec078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f28fc005fd0 tx=0x7f28fc009500 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:49.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.036+0000 7f29097fa640 1 -- 172.21.15.81:0/3673511363 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2904060e30 con 0x7f290c0ce510 2024-06-29T15:58:49.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.219+0000 7f2912f43640 1 -- 172.21.15.81:0/3673511363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f290c0d02d0 con 0x7f290c0ce510 2024-06-29T15:58:49.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.220+0000 7f29097fa640 1 -- 172.21.15.81:0/3673511363 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v239) ==== 103+0+11 (secure 0 0 0) 0x7f2904060730 con 0x7f290c0ce510 2024-06-29T15:58:49.220 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:58:49.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 -- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f28ec076120 msgr2=0x7f28ec078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f28ec076120 0x7f28ec078630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f28fc005fd0 tx=0x7f28fc009500 comp rx=0 tx=0).stop 2024-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 -- 172.21.15.81:0/3673511363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0ce510 msgr2=0x7f290c165820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0ce510 0x7f290c165820 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f290400c390 tx=0x7f290400c850 comp rx=0 tx=0).stop 2024-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 -- 172.21.15.81:0/3673511363 shutdown_connections 2024-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f28ec076120 0x7f28ec078630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f290c0ce510 0x7f290c165820 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f290c0b6eb0 0x7f290c1652e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 --2- 172.21.15.81:0/3673511363 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f290c0b4810 0x7f290c0cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 -- 172.21.15.81:0/3673511363 >> 172.21.15.81:0/3673511363 conn(0x7f290c0b3510 msgr2=0x7f290c15f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 -- 172.21.15.81:0/3673511363 shutdown_connections 2024-06-29T15:58:49.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.222+0000 7f2912f43640 1 -- 172.21.15.81:0/3673511363 wait complete. 2024-06-29T15:58:49.234 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:49.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.348+0000 7ff5677f5640 1 -- 172.21.15.81:0/3195119868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54c003a50 msgr2=0x7ff54c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.348+0000 7ff5677f5640 1 --2- 172.21.15.81:0/3195119868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54c003a50 0x7ff54c003ec0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009b50 tx=0x7ff54c005830 comp rx=0 tx=0).stop 2024-06-29T15:58:49.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.348+0000 7ff5677f5640 1 -- 172.21.15.81:0/3195119868 shutdown_connections 2024-06-29T15:58:49.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.348+0000 7ff5677f5640 1 --2- 172.21.15.81:0/3195119868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54c003a50 0x7ff54c003ec0 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.348+0000 7ff5677f5640 1 -- 172.21.15.81:0/3195119868 >> 172.21.15.81:0/3195119868 conn(0x7ff5600d0930 msgr2=0x7ff5600d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.348+0000 7ff5677f5640 1 -- 172.21.15.81:0/3195119868 shutdown_connections 2024-06-29T15:58:49.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.348+0000 7ff5677f5640 1 -- 172.21.15.81:0/3195119868 wait complete. 2024-06-29T15:58:49.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.350+0000 7ff5677f5640 1 Processor -- start 2024-06-29T15:58:49.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.350+0000 7ff5677f5640 1 -- start start 2024-06-29T15:58:49.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.351+0000 7ff5677f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54c003a50 0x7ff560150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.351+0000 7ff5677f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff560154b80 0x7ff560153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff5677f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff560151c10 0x7ff560154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff5677f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff5600affa0 con 0x7ff54c003a50 2024-06-29T15:58:49.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff5677f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff5600afe20 con 0x7ff560151c10 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff5677f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff5600b0120 con 0x7ff560154b80 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff565591640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54c003a50 0x7ff560150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff565591640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54c003a50 0x7ff560150ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53540/0 (socket says 172.21.15.81:53540) 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff565591640 1 -- 172.21.15.81:0/4270491740 learned_addr learned my addr 172.21.15.81:0/4270491740 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff564d90640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff560154b80 0x7ff560153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff565d92640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff560151c10 0x7ff560154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff565591640 1 -- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff560154b80 msgr2=0x7ff560153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff565591640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff560154b80 0x7ff560153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff565591640 1 -- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff560151c10 msgr2=0x7ff560154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff565591640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff560151c10 0x7ff560154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff565591640 1 -- 172.21.15.81:0/4270491740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff560152ef0 con 0x7ff54c003a50 2024-06-29T15:58:49.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff565591640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54c003a50 0x7ff560150ed0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009a70 tx=0x7ff55c0336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:49.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff555ffb640 1 -- 172.21.15.81:0/4270491740 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c038040 con 0x7ff54c003a50 2024-06-29T15:58:49.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff555ffb640 1 -- 172.21.15.81:0/4270491740 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff55c0339d0 con 0x7ff54c003a50 2024-06-29T15:58:49.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff5677f5640 1 -- 172.21.15.81:0/4270491740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5601531d0 con 0x7ff54c003a50 2024-06-29T15:58:49.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff555ffb640 1 -- 172.21.15.81:0/4270491740 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c033e20 con 0x7ff54c003a50 2024-06-29T15:58:49.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.352+0000 7ff5677f5640 1 -- 172.21.15.81:0/4270491740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5601567d0 con 0x7ff54c003a50 2024-06-29T15:58:49.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.353+0000 7ff5677f5640 1 -- 172.21.15.81:0/4270491740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff560153410 con 0x7ff54c003a50 2024-06-29T15:58:49.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.354+0000 7ff555ffb640 1 -- 172.21.15.81:0/4270491740 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 144) ==== 98039+0+0 (secure 0 0 0) 0x7ff55c043020 con 0x7ff54c003a50 2024-06-29T15:58:49.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.355+0000 7ff555ffb640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff544076120 0x7ff544078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.355+0000 7ff555ffb640 1 -- 172.21.15.81:0/4270491740 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 7561+0+0 (secure 0 0 0) 0x7ff55c0b7d90 con 0x7ff54c003a50 2024-06-29T15:58:49.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.355+0000 7ff564d90640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff544076120 0x7ff544078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.355+0000 7ff564d90640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff544076120 0x7ff544078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff56006f920 tx=0x7ff550006b70 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:49.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.359+0000 7ff555ffb640 1 -- 172.21.15.81:0/4270491740 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff560153410 con 0x7ff54c003a50 2024-06-29T15:58:49.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.576+0000 7ff5677f5640 1 -- 172.21.15.81:0/4270491740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff56006e2a0 con 0x7ff54c003a50 2024-06-29T15:58:49.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.576+0000 7ff555ffb640 1 -- 172.21.15.81:0/4270491740 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6562 (secure 0 0 0) 0x7ff56006e2a0 con 0x7ff54c003a50 2024-06-29T15:58:49.577 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:49.577 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":135,"btime":"2024-06-29T15:58:34:407667+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:18.654579+0000","modified":"2024-06-29T15:58:21.679033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12105":{"gid":12105,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/30158598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":30158598},{"type":"v1","addr":"172.21.15.167:6837","nonce":30158598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12105,"qdb_cluster":[12105]},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:30.737083+0000","modified":"2024-06-29T15:58:33.773132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12093":{"gid":12093,"name":"a","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12093,"qdb_cluster":[12093]},"id":16}]} 2024-06-29T15:58:49.577 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 135 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 -- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff544076120 msgr2=0x7ff544078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff544076120 0x7ff544078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff56006f920 tx=0x7ff550006b70 comp rx=0 tx=0).stop 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 -- 172.21.15.81:0/4270491740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54c003a50 msgr2=0x7ff560150ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54c003a50 0x7ff560150ed0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009a70 tx=0x7ff55c0336f0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 -- 172.21.15.81:0/4270491740 shutdown_connections 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff544076120 0x7ff544078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff560151c10 0x7ff560154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff560154b80 0x7ff560153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 --2- 172.21.15.81:0/4270491740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54c003a50 0x7ff560150ed0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 -- 172.21.15.81:0/4270491740 >> 172.21.15.81:0/4270491740 conn(0x7ff5600d0930 msgr2=0x7ff56015db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 -- 172.21.15.81:0/4270491740 shutdown_connections 2024-06-29T15:58:49.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.578+0000 7ff5677f5640 1 -- 172.21.15.81:0/4270491740 wait complete. 2024-06-29T15:58:49.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:49.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.702+0000 7f0203009640 1 -- 172.21.15.81:0/630350848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01fc073030 msgr2=0x7f01fc073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.702+0000 7f0203009640 1 --2- 172.21.15.81:0/630350848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01fc073030 0x7f01fc073440 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f01e8005fd0 tx=0x7f01e802a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.702+0000 7f0203009640 1 -- 172.21.15.81:0/630350848 shutdown_connections 2024-06-29T15:58:49.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.702+0000 7f0203009640 1 --2- 172.21.15.81:0/630350848 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01fc15e960 0x7f01fc15ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.702+0000 7f0203009640 1 --2- 172.21.15.81:0/630350848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01fc073030 0x7f01fc073440 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.702+0000 7f0203009640 1 -- 172.21.15.81:0/630350848 >> 172.21.15.81:0/630350848 conn(0x7f01fc06ccc0 msgr2=0x7f01fc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.702+0000 7f0203009640 1 -- 172.21.15.81:0/630350848 shutdown_connections 2024-06-29T15:58:49.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.702+0000 7f0203009640 1 -- 172.21.15.81:0/630350848 wait complete. 2024-06-29T15:58:49.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.704+0000 7f0203009640 1 Processor -- start 2024-06-29T15:58:49.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.704+0000 7f0203009640 1 -- start start 2024-06-29T15:58:49.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.705+0000 7f0203009640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01fc073030 0x7f01fc1402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.705+0000 7f0203009640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01fc1412f0 0x7f01fc13ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.705+0000 7f0203009640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01fc15e960 0x7f01fc0a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.705+0000 7f0203009640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc0b3390 con 0x7f01fc1412f0 2024-06-29T15:58:49.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.705+0000 7f0203009640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc0b3210 con 0x7f01fc15e960 2024-06-29T15:58:49.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.705+0000 7f0203009640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f01fc0b3510 con 0x7f01fc073030 2024-06-29T15:58:49.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.705+0000 7f01fbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01fc1412f0 0x7f01fc13ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.705+0000 7f01fbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01fc1412f0 0x7f01fc13ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53548/0 (socket says 172.21.15.81:53548) 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.705+0000 7f01fbfff640 1 -- 172.21.15.81:0/568809555 learned_addr learned my addr 172.21.15.81:0/568809555 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f02015a6640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01fc15e960 0x7f01fc0a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f0200da5640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01fc073030 0x7f01fc1402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f01fbfff640 1 -- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01fc073030 msgr2=0x7f01fc1402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f01fbfff640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01fc073030 0x7f01fc1402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f01fbfff640 1 -- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01fc15e960 msgr2=0x7f01fc0a9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f01fbfff640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01fc15e960 0x7f01fc0a9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f01fbfff640 1 -- 172.21.15.81:0/568809555 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc157eb0 con 0x7f01fc1412f0 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f02015a6640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01fc15e960 0x7f01fc0a9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:49.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f0200da5640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01fc073030 0x7f01fc1402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:49.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f01fbfff640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01fc1412f0 0x7f01fc13ed80 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f01e8034040 tx=0x7f01e80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:49.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f01f97fa640 1 -- 172.21.15.81:0/568809555 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8035040 con 0x7f01fc1412f0 2024-06-29T15:58:49.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f01f97fa640 1 -- 172.21.15.81:0/568809555 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01e80047a0 con 0x7f01fc1412f0 2024-06-29T15:58:49.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f0203009640 1 -- 172.21.15.81:0/568809555 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01fc0aa3f0 con 0x7f01fc1412f0 2024-06-29T15:58:49.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f01f97fa640 1 -- 172.21.15.81:0/568809555 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8004b90 con 0x7f01fc1412f0 2024-06-29T15:58:49.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.706+0000 7f0203009640 1 -- 172.21.15.81:0/568809555 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01fc13f800 con 0x7f01fc1412f0 2024-06-29T15:58:49.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.707+0000 7f0203009640 1 -- 172.21.15.81:0/568809555 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01fc0aa670 con 0x7f01fc1412f0 2024-06-29T15:58:49.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.708+0000 7f01f97fa640 1 -- 172.21.15.81:0/568809555 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 144) ==== 98039+0+0 (secure 0 0 0) 0x7f01e8004d30 con 0x7f01fc1412f0 2024-06-29T15:58:49.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.708+0000 7f01f97fa640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f01e0076150 0x7f01e0078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:49.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.708+0000 7f01f97fa640 1 -- 172.21.15.81:0/568809555 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 7561+0+0 (secure 0 0 0) 0x7f01e80ba350 con 0x7f01fc1412f0 2024-06-29T15:58:49.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.708+0000 7f0200da5640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f01e0076150 0x7f01e0078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:49.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.709+0000 7f0200da5640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f01e0076150 0x7f01e0078660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f01f00016a0 tx=0x7f01f0009040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:49.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.713+0000 7f01f97fa640 1 -- 172.21.15.81:0/568809555 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f01e8083830 con 0x7f01fc1412f0 2024-06-29T15:58:49.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.899+0000 7f0203009640 1 -- 172.21.15.81:0/568809555 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01fc15e4f0 con 0x7f01fc1412f0 2024-06-29T15:58:49.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.900+0000 7f01f97fa640 1 -- 172.21.15.81:0/568809555 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v239) ==== 74+0+21809 (secure 0 0 0) 0x7f01fc15e4f0 con 0x7f01fc1412f0 2024-06-29T15:58:49.901 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:49.901 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":239,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:47.815576+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:29.291185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:29.702735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:6832/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/355438800":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4195196310":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4186712115":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/3601299355":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2623346726":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/972265206":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/1116386548":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6832/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/3057737710":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6833/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1418041738":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/561620344":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2207847535":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2067929640":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/874327121":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/273305307":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2269612087":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1188031263":"2024-06-30T15:58:42.086693+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:49.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.902+0000 7f0203009640 1 -- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f01e0076150 msgr2=0x7f01e0078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.902+0000 7f0203009640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f01e0076150 0x7f01e0078660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f01f00016a0 tx=0x7f01f0009040 comp rx=0 tx=0).stop 2024-06-29T15:58:49.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.902+0000 7f0203009640 1 -- 172.21.15.81:0/568809555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01fc1412f0 msgr2=0x7f01fc13ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:49.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.902+0000 7f0203009640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01fc1412f0 0x7f01fc13ed80 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f01e8034040 tx=0x7f01e80038a0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.902+0000 7f0203009640 1 -- 172.21.15.81:0/568809555 shutdown_connections 2024-06-29T15:58:49.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.902+0000 7f0203009640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f01e0076150 0x7f01e0078660 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.902+0000 7f0203009640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f01fc15e960 0x7f01fc0a9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.902+0000 7f0203009640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01fc1412f0 0x7f01fc13ed80 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.902+0000 7f0203009640 1 --2- 172.21.15.81:0/568809555 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f01fc073030 0x7f01fc1402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:49.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.902+0000 7f0203009640 1 -- 172.21.15.81:0/568809555 >> 172.21.15.81:0/568809555 conn(0x7f01fc06ccc0 msgr2=0x7f01fc15ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:49.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.903+0000 7f0203009640 1 -- 172.21.15.81:0/568809555 shutdown_connections 2024-06-29T15:58:49.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:49.903+0000 7f0203009640 1 -- 172.21.15.81:0/568809555 wait complete. 2024-06-29T15:58:49.914 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:58:49.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:50.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.028+0000 7f9a95bd4640 1 -- 172.21.15.81:0/2247822898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a900b6d60 msgr2=0x7f9a900ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.028+0000 7f9a95bd4640 1 --2- 172.21.15.81:0/2247822898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a900b6d60 0x7f9a900ce4d0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f9a900b2c10 tx=0x7f9a7802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.028+0000 7f9a95bd4640 1 -- 172.21.15.81:0/2247822898 shutdown_connections 2024-06-29T15:58:50.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.028+0000 7f9a95bd4640 1 --2- 172.21.15.81:0/2247822898 >> v2:172.21.15.167:3300/0 conn(0x7f9a900cd540 0x7f9a900d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.028+0000 7f9a95bd4640 1 --2- 172.21.15.81:0/2247822898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a900b6d60 0x7f9a900ce4d0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.028+0000 7f9a95bd4640 1 -- 172.21.15.81:0/2247822898 >> 172.21.15.81:0/2247822898 conn(0x7f9a900d1960 msgr2=0x7f9a900d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.028+0000 7f9a95bd4640 1 -- 172.21.15.81:0/2247822898 shutdown_connections 2024-06-29T15:58:50.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.028+0000 7f9a95bd4640 1 -- 172.21.15.81:0/2247822898 wait complete. 2024-06-29T15:58:50.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.030+0000 7f9a95bd4640 1 Processor -- start 2024-06-29T15:58:50.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.030+0000 7f9a95bd4640 1 -- start start 2024-06-29T15:58:50.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a95bd4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a900b6d60 0x7f9a9006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a95bd4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a900cd540 0x7f9a901f5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a95bd4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a900cea10 0x7f9a901f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a95bd4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a900d0630 con 0x7f9a900cd540 2024-06-29T15:58:50.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a95bd4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9a900b2790 con 0x7f9a900cea10 2024-06-29T15:58:50.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a95bd4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9a900d07b0 con 0x7f9a900b6d60 2024-06-29T15:58:50.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a900cd540 0x7f9a901f5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a900cd540 0x7f9a901f5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60986/0 (socket says 172.21.15.81:60986) 2024-06-29T15:58:50.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8effd640 1 -- 172.21.15.81:0/3847058168 learned_addr learned my addr 172.21.15.81:0/3847058168 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:50.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8f7fe640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a900b6d60 0x7f9a9006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8ffff640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a900cea10 0x7f9a901f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8effd640 1 -- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a900b6d60 msgr2=0x7f9a9006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8effd640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a900b6d60 0x7f9a9006e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8effd640 1 -- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a900cea10 msgr2=0x7f9a901f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8effd640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a900cea10 0x7f9a901f5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8effd640 1 -- 172.21.15.81:0/3847058168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a900ca5e0 con 0x7f9a900cd540 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8ffff640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a900cea10 0x7f9a901f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a8effd640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a900cd540 0x7f9a901f5320 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000c3a0 tx=0x7f9a8000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a87fff640 1 -- 172.21.15.81:0/3847058168 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80017070 con 0x7f9a900cd540 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a87fff640 1 -- 172.21.15.81:0/3847058168 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a800053c0 con 0x7f9a900cd540 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a87fff640 1 -- 172.21.15.81:0/3847058168 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80003850 con 0x7f9a900cd540 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a95bd4640 1 -- 172.21.15.81:0/3847058168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a900ca8c0 con 0x7f9a900cd540 2024-06-29T15:58:50.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.032+0000 7f9a95bd4640 1 -- 172.21.15.81:0/3847058168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a900cae80 con 0x7f9a900cd540 2024-06-29T15:58:50.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.034+0000 7f9a95bd4640 1 -- 172.21.15.81:0/3847058168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a900ce4d0 con 0x7f9a900cd540 2024-06-29T15:58:50.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.034+0000 7f9a87fff640 1 -- 172.21.15.81:0/3847058168 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 144) ==== 98039+0+0 (secure 0 0 0) 0x7f9a8001a070 con 0x7f9a900cd540 2024-06-29T15:58:50.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.035+0000 7f9a87fff640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9a70076120 0x7f9a70078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.035+0000 7f9a87fff640 1 -- 172.21.15.81:0/3847058168 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 7561+0+0 (secure 0 0 0) 0x7f9a80096e90 con 0x7f9a900cd540 2024-06-29T15:58:50.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.035+0000 7f9a8f7fe640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9a70076120 0x7f9a70078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.036+0000 7f9a8f7fe640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9a70076120 0x7f9a70078630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9a78033000 tx=0x7f9a78004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:50.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.039+0000 7f9a87fff640 1 -- 172.21.15.81:0/3847058168 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9a800603f0 con 0x7f9a900cd540 2024-06-29T15:58:50.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.178+0000 7f9a87fff640 1 -- 172.21.15.81:0/3847058168 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 145) ==== 98349+0+0 (secure 0 0 0) 0x7f9a8005bf80 con 0x7f9a900cd540 2024-06-29T15:58:50.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.259+0000 7f9a95bd4640 1 -- 172.21.15.81:0/3847058168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a900cdec0 con 0x7f9a900cd540 2024-06-29T15:58:50.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.260+0000 7f9a87fff640 1 -- 172.21.15.81:0/3847058168 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6562 (secure 0 0 0) 0x7f9a8006cc20 con 0x7f9a900cd540 2024-06-29T15:58:50.260 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:50.260 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":135,"btime":"2024-06-29T15:58:34:407667+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:18.654579+0000","modified":"2024-06-29T15:58:21.679033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12105":{"gid":12105,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/30158598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":30158598},{"type":"v1","addr":"172.21.15.167:6837","nonce":30158598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12105,"qdb_cluster":[12105]},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:30.737083+0000","modified":"2024-06-29T15:58:33.773132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12093":{"gid":12093,"name":"a","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12093,"qdb_cluster":[12093]},"id":16}]} 2024-06-29T15:58:50.261 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 135 2024-06-29T15:58:50.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 -- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9a70076120 msgr2=0x7f9a70078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9a70076120 0x7f9a70078630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9a78033000 tx=0x7f9a78004730 comp rx=0 tx=0).stop 2024-06-29T15:58:50.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 -- 172.21.15.81:0/3847058168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a900cd540 msgr2=0x7f9a901f5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a900cd540 0x7f9a901f5320 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000c3a0 tx=0x7f9a8000c860 comp rx=0 tx=0).stop 2024-06-29T15:58:50.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 -- 172.21.15.81:0/3847058168 shutdown_connections 2024-06-29T15:58:50.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9a70076120 0x7f9a70078630 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9a900cea10 0x7f9a901f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a900cd540 0x7f9a901f5320 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 --2- 172.21.15.81:0/3847058168 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9a900b6d60 0x7f9a9006e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 -- 172.21.15.81:0/3847058168 >> 172.21.15.81:0/3847058168 conn(0x7f9a900d1960 msgr2=0x7f9a900b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 -- 172.21.15.81:0/3847058168 shutdown_connections 2024-06-29T15:58:50.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.262+0000 7f9a95bd4640 1 -- 172.21.15.81:0/3847058168 wait complete. 2024-06-29T15:58:50.274 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:50.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.388+0000 7fed3229c640 1 -- 172.21.15.81:0/2521236541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2c0cd480 msgr2=0x7fed2c0ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.388+0000 7fed3229c640 1 --2- 172.21.15.81:0/2521236541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2c0cd480 0x7fed2c0ce510 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fed14005fd0 tx=0x7fed1402a540 comp rx=0 tx=0).stop 2024-06-29T15:58:50.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.388+0000 7fed3229c640 1 -- 172.21.15.81:0/2521236541 shutdown_connections 2024-06-29T15:58:50.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.388+0000 7fed3229c640 1 --2- 172.21.15.81:0/2521236541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2c0cd480 0x7fed2c0ce510 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.388+0000 7fed3229c640 1 -- 172.21.15.81:0/2521236541 >> 172.21.15.81:0/2521236541 conn(0x7fed2c0b3510 msgr2=0x7fed2c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.388+0000 7fed3229c640 1 -- 172.21.15.81:0/2521236541 shutdown_connections 2024-06-29T15:58:50.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.388+0000 7fed3229c640 1 -- 172.21.15.81:0/2521236541 wait complete. 2024-06-29T15:58:50.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.390+0000 7fed3229c640 1 Processor -- start 2024-06-29T15:58:50.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.390+0000 7fed3229c640 1 -- start start 2024-06-29T15:58:50.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed3229c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2c0b6eb0 0x7fed2c1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed3229c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed2c0cd480 0x7fed2c0ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed3229c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed2c0cea50 0x7fed2c0ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed3229c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c0b52b0 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed3229c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c0b5130 con 0x7fed2c0cea50 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed3229c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fed2c167660 con 0x7fed2c0cd480 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed2bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2c0b6eb0 0x7fed2c1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed2bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2c0b6eb0 0x7fed2c1579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60998/0 (socket says 172.21.15.81:60998) 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed2bfff640 1 -- 172.21.15.81:0/673711986 learned_addr learned my addr 172.21.15.81:0/673711986 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed30839640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed2c0cea50 0x7fed2c0ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed23fff640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed2c0cd480 0x7fed2c0ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed2bfff640 1 -- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed2c0cd480 msgr2=0x7fed2c0ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed2bfff640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed2c0cd480 0x7fed2c0ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed2bfff640 1 -- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed2c0cea50 msgr2=0x7fed2c0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed2bfff640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed2c0cea50 0x7fed2c0ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.391+0000 7fed2bfff640 1 -- 172.21.15.81:0/673711986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2c09f580 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.392+0000 7fed2bfff640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2c0b6eb0 0x7fed2c1579a0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00c3d0 tx=0x7fed1c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:50.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.392+0000 7fed297fa640 1 -- 172.21.15.81:0/673711986 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c017070 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.392+0000 7fed297fa640 1 -- 172.21.15.81:0/673711986 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed1c0053c0 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.392+0000 7fed3229c640 1 -- 172.21.15.81:0/673711986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2c164c60 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.392+0000 7fed297fa640 1 -- 172.21.15.81:0/673711986 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c003850 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.392+0000 7fed3229c640 1 -- 172.21.15.81:0/673711986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed2c164f20 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.393+0000 7fed3229c640 1 -- 172.21.15.81:0/673711986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed2c070ae0 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.393+0000 7fed297fa640 1 -- 172.21.15.81:0/673711986 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 145) ==== 98349+0+0 (secure 0 0 0) 0x7fed1c01a070 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.394+0000 7fed297fa640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fed0c076480 0x7fed0c078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.394+0000 7fed297fa640 1 -- 172.21.15.81:0/673711986 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 7561+0+0 (secure 0 0 0) 0x7fed1c0975b0 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.394+0000 7fed23fff640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fed0c076480 0x7fed0c078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.395+0000 7fed23fff640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fed0c076480 0x7fed0c078990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fed14033000 tx=0x7fed14004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:50.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.398+0000 7fed297fa640 1 -- 172.21.15.81:0/673711986 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fed2c070ae0 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.615+0000 7fed3229c640 1 -- 172.21.15.81:0/673711986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed2c0cdf00 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.615+0000 7fed297fa640 1 -- 172.21.15.81:0/673711986 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+6562 (secure 0 0 0) 0x7fed1c0605e0 con 0x7fed2c0b6eb0 2024-06-29T15:58:50.616 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:50.616 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":135,"btime":"2024-06-29T15:58:34:407667+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:18.654579+0000","modified":"2024-06-29T15:58:21.679033+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12105":{"gid":12105,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/30158598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":30158598},{"type":"v1","addr":"172.21.15.167:6837","nonce":30158598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12105,"qdb_cluster":[12105]},"id":15},{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:30.737083+0000","modified":"2024-06-29T15:58:33.773132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12093":{"gid":12093,"name":"a","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12093,"qdb_cluster":[12093]},"id":16}]} 2024-06-29T15:58:50.617 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 135 2024-06-29T15:58:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 -- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fed0c076480 msgr2=0x7fed0c078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fed0c076480 0x7fed0c078990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fed14033000 tx=0x7fed14004730 comp rx=0 tx=0).stop 2024-06-29T15:58:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 -- 172.21.15.81:0/673711986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2c0b6eb0 msgr2=0x7fed2c1579a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2c0b6eb0 0x7fed2c1579a0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00c3d0 tx=0x7fed1c00c890 comp rx=0 tx=0).stop 2024-06-29T15:58:50.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 -- 172.21.15.81:0/673711986 shutdown_connections 2024-06-29T15:58:50.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fed0c076480 0x7fed0c078990 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fed2c0cea50 0x7fed2c0ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fed2c0cd480 0x7fed2c0ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 --2- 172.21.15.81:0/673711986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2c0b6eb0 0x7fed2c1579a0 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 -- 172.21.15.81:0/673711986 >> 172.21.15.81:0/673711986 conn(0x7fed2c0b3510 msgr2=0x7fed2c1639d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 -- 172.21.15.81:0/673711986 shutdown_connections 2024-06-29T15:58:50.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.617+0000 7fed3229c640 1 -- 172.21.15.81:0/673711986 wait complete. 2024-06-29T15:58:50.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:50.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.743+0000 7f6bd2363640 1 -- 172.21.15.81:0/1973964826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bcc0b35b0 msgr2=0x7f6bcc15b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.743+0000 7f6bd2363640 1 --2- 172.21.15.81:0/1973964826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bcc0b35b0 0x7f6bcc15b5b0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc005fd0 tx=0x7f6bbc02a540 comp rx=0 tx=0).stop 2024-06-29T15:58:50.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.743+0000 7f6bd2363640 1 -- 172.21.15.81:0/1973964826 shutdown_connections 2024-06-29T15:58:50.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.743+0000 7f6bd2363640 1 --2- 172.21.15.81:0/1973964826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bcc06f660 0x7f6bcc06fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.743+0000 7f6bd2363640 1 --2- 172.21.15.81:0/1973964826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bcc0b35b0 0x7f6bcc15b5b0 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.743+0000 7f6bd2363640 1 -- 172.21.15.81:0/1973964826 >> 172.21.15.81:0/1973964826 conn(0x7f6bcc06ccc0 msgr2=0x7f6bcc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.743+0000 7f6bd2363640 1 -- 172.21.15.81:0/1973964826 shutdown_connections 2024-06-29T15:58:50.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.743+0000 7f6bd2363640 1 -- 172.21.15.81:0/1973964826 wait complete. 2024-06-29T15:58:50.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.744+0000 7f6bd2363640 1 Processor -- start 2024-06-29T15:58:50.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.745+0000 7f6bd2363640 1 -- start start 2024-06-29T15:58:50.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bd2363640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bcc06f660 0x7f6bcc152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bd2363640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bcc0b35b0 0x7f6bcc1f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bd2363640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bcc0b4430 0x7f6bcc1f4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bd2363640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc06f240 con 0x7f6bcc06f660 2024-06-29T15:58:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bd2363640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc06f0c0 con 0x7f6bcc0b4430 2024-06-29T15:58:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bd2363640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6bcc06f3c0 con 0x7f6bcc0b35b0 2024-06-29T15:58:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bcbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bcc06f660 0x7f6bcc152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bcbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bcc06f660 0x7f6bcc152a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32786/0 (socket says 172.21.15.81:32786) 2024-06-29T15:58:50.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bcbfff640 1 -- 172.21.15.81:0/1659829191 learned_addr learned my addr 172.21.15.81:0/1659829191 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:50.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bcb7fe640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bcc0b35b0 0x7f6bcc1f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bd0900640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bcc0b4430 0x7f6bcc1f4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bcbfff640 1 -- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bcc0b35b0 msgr2=0x7f6bcc1f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bcbfff640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bcc0b35b0 0x7f6bcc1f4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bcbfff640 1 -- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bcc0b4430 msgr2=0x7f6bcc1f4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bcbfff640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bcc0b4430 0x7f6bcc1f4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.746+0000 7f6bcbfff640 1 -- 172.21.15.81:0/1659829191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bcc153620 con 0x7f6bcc06f660 2024-06-29T15:58:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.747+0000 7f6bcbfff640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bcc06f660 0x7f6bcc152a40 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc000c00 tx=0x7f6bbc003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.747+0000 7f6bc8ff9640 1 -- 172.21.15.81:0/1659829191 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc035040 con 0x7f6bcc06f660 2024-06-29T15:58:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.747+0000 7f6bc8ff9640 1 -- 172.21.15.81:0/1659829191 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bbc0047a0 con 0x7f6bcc06f660 2024-06-29T15:58:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.747+0000 7f6bd2363640 1 -- 172.21.15.81:0/1659829191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bcc153900 con 0x7f6bcc06f660 2024-06-29T15:58:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.747+0000 7f6bc8ff9640 1 -- 172.21.15.81:0/1659829191 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc004bf0 con 0x7f6bcc06f660 2024-06-29T15:58:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.747+0000 7f6bd2363640 1 -- 172.21.15.81:0/1659829191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bcc153ec0 con 0x7f6bcc06f660 2024-06-29T15:58:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.748+0000 7f6bd2363640 1 -- 172.21.15.81:0/1659829191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bcc06fa70 con 0x7f6bcc06f660 2024-06-29T15:58:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.749+0000 7f6bc8ff9640 1 -- 172.21.15.81:0/1659829191 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 145) ==== 98349+0+0 (secure 0 0 0) 0x7f6bbc004d90 con 0x7f6bcc06f660 2024-06-29T15:58:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.749+0000 7f6bc8ff9640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6bb80764b0 0x7f6bb80789c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:50.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.749+0000 7f6bc8ff9640 1 -- 172.21.15.81:0/1659829191 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 7561+0+0 (secure 0 0 0) 0x7f6bbc0baf10 con 0x7f6bcc06f660 2024-06-29T15:58:50.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.749+0000 7f6bcb7fe640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6bb80764b0 0x7f6bb80789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:50.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.750+0000 7f6bcb7fe640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6bb80764b0 0x7f6bb80789c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc072e60 tx=0x7f6bc0009040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:50.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.753+0000 7f6bc8ff9640 1 -- 172.21.15.81:0/1659829191 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6bbc083d70 con 0x7f6bcc06f660 2024-06-29T15:58:50.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.934+0000 7f6bd2363640 1 -- 172.21.15.81:0/1659829191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6bcc15c8e0 con 0x7f6bcc06f660 2024-06-29T15:58:50.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.935+0000 7f6bc8ff9640 1 -- 172.21.15.81:0/1659829191 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v239) ==== 74+0+21809 (secure 0 0 0) 0x7f6bbc083670 con 0x7f6bcc06f660 2024-06-29T15:58:50.936 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:50.936 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":239,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:47.815576+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:58:16.949415+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-06-29T15:58:17.618219+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:29.291185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:29.702735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:6832/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/355438800":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4195196310":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4186712115":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/3601299355":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2623346726":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/972265206":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/1116386548":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6832/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/3057737710":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6833/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1418041738":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/561620344":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2207847535":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2067929640":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/874327121":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/273305307":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2269612087":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1188031263":"2024-06-30T15:58:42.086693+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:50.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 -- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6bb80764b0 msgr2=0x7f6bb80789c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6bb80764b0 0x7f6bb80789c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc072e60 tx=0x7f6bc0009040 comp rx=0 tx=0).stop 2024-06-29T15:58:50.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 -- 172.21.15.81:0/1659829191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bcc06f660 msgr2=0x7f6bcc152a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:50.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bcc06f660 0x7f6bcc152a40 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc000c00 tx=0x7f6bbc003910 comp rx=0 tx=0).stop 2024-06-29T15:58:50.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 -- 172.21.15.81:0/1659829191 shutdown_connections 2024-06-29T15:58:50.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6bb80764b0 0x7f6bb80789c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6bcc0b4430 0x7f6bcc1f4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6bcc0b35b0 0x7f6bcc1f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 --2- 172.21.15.81:0/1659829191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bcc06f660 0x7f6bcc152a40 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:50.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 -- 172.21.15.81:0/1659829191 >> 172.21.15.81:0/1659829191 conn(0x7f6bcc06ccc0 msgr2=0x7f6bcc14b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:50.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 -- 172.21.15.81:0/1659829191 shutdown_connections 2024-06-29T15:58:50.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:50.938+0000 7f6bd2363640 1 -- 172.21.15.81:0/1659829191 wait complete. 2024-06-29T15:58:50.950 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:58:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.064+0000 7f9c986fd640 1 -- 172.21.15.81:0/339699489 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c90074520 msgr2=0x7f9c9006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.064+0000 7f9c986fd640 1 --2- 172.21.15.81:0/339699489 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c90074520 0x7f9c9006f140 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9c80005fd0 tx=0x7f9c8002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.064+0000 7f9c986fd640 1 -- 172.21.15.81:0/339699489 shutdown_connections 2024-06-29T15:58:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.064+0000 7f9c986fd640 1 --2- 172.21.15.81:0/339699489 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9006fa90 0x7f9c9006ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.064+0000 7f9c986fd640 1 --2- 172.21.15.81:0/339699489 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c90074520 0x7f9c9006f140 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.064+0000 7f9c986fd640 1 -- 172.21.15.81:0/339699489 >> 172.21.15.81:0/339699489 conn(0x7f9c90072950 msgr2=0x7f9c900b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:51.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.064+0000 7f9c986fd640 1 -- 172.21.15.81:0/339699489 shutdown_connections 2024-06-29T15:58:51.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.064+0000 7f9c986fd640 1 -- 172.21.15.81:0/339699489 wait complete. 2024-06-29T15:58:51.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.066+0000 7f9c986fd640 1 Processor -- start 2024-06-29T15:58:51.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.066+0000 7f9c986fd640 1 -- start start 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.067+0000 7f9c986fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9006f680 0x7f9c9009db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.067+0000 7f9c986fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c9006fa90 0x7f9c900ce520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.067+0000 7f9c986fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c90074520 0x7f9c90151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.067+0000 7f9c986fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9015b010 con 0x7f9c9006fa90 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.067+0000 7f9c986fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9006d150 con 0x7f9c90074520 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.067+0000 7f9c986fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9c9015b190 con 0x7f9c9006f680 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.067+0000 7f9c95c98640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c9006fa90 0x7f9c900ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c95c98640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c9006fa90 0x7f9c900ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32802/0 (socket says 172.21.15.81:32802) 2024-06-29T15:58:51.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c95c98640 1 -- 172.21.15.81:0/171214211 learned_addr learned my addr 172.21.15.81:0/171214211 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c96499640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9006f680 0x7f9c9009db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c96c9a640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c90074520 0x7f9c90151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c95c98640 1 -- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9006f680 msgr2=0x7f9c9009db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c95c98640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9006f680 0x7f9c9009db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c95c98640 1 -- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c90074520 msgr2=0x7f9c90151b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c95c98640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c90074520 0x7f9c90151b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c95c98640 1 -- 172.21.15.81:0/171214211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c900cefe0 con 0x7f9c9006fa90 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c95c98640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c9006fa90 0x7f9c900ce520 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400c430 tx=0x7f9c8400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c8effd640 1 -- 172.21.15.81:0/171214211 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c84011790 con 0x7f9c9006fa90 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c986fd640 1 -- 172.21.15.81:0/171214211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c900cf2c0 con 0x7f9c9006fa90 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c8effd640 1 -- 172.21.15.81:0/171214211 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c84017070 con 0x7f9c9006fa90 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c8effd640 1 -- 172.21.15.81:0/171214211 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c84011ce0 con 0x7f9c9006fa90 2024-06-29T15:58:51.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.068+0000 7f9c986fd640 1 -- 172.21.15.81:0/171214211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c900cea60 con 0x7f9c9006fa90 2024-06-29T15:58:51.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.069+0000 7f9c986fd640 1 -- 172.21.15.81:0/171214211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c9014f790 con 0x7f9c9006fa90 2024-06-29T15:58:51.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.070+0000 7f9c8effd640 1 -- 172.21.15.81:0/171214211 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 145) ==== 98349+0+0 (secure 0 0 0) 0x7f9c84026430 con 0x7f9c9006fa90 2024-06-29T15:58:51.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.070+0000 7f9c8effd640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9c7c076480 0x7f9c7c078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.071+0000 7f9c8effd640 1 -- 172.21.15.81:0/171214211 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 7561+0+0 (secure 0 0 0) 0x7f9c84098b70 con 0x7f9c9006fa90 2024-06-29T15:58:51.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.071+0000 7f9c96499640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9c7c076480 0x7f9c7c078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.071+0000 7f9c96499640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9c7c076480 0x7f9c7c078990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c80034000 tx=0x7f9c80004ab0 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:51.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.074+0000 7f9c8effd640 1 -- 172.21.15.81:0/171214211 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9c84061b10 con 0x7f9c9006fa90 2024-06-29T15:58:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.290+0000 7f9c986fd640 1 -- 172.21.15.81:0/171214211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f9c900af4d0 con 0x7f9c9006fa90 2024-06-29T15:58:51.291 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T15:58:51.291+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:58:51.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.292+0000 7f9c8effd640 1 -- 172.21.15.81:0/171214211 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v136) ==== 161+0+0 (secure 0 0 0) 0x7f9c84061410 con 0x7f9c9006fa90 2024-06-29T15:58:51.293 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:58:51.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.295+0000 7f9c986fd640 1 -- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9c7c076480 msgr2=0x7f9c7c078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.295+0000 7f9c986fd640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9c7c076480 0x7f9c7c078990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c80034000 tx=0x7f9c80004ab0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.295+0000 7f9c986fd640 1 -- 172.21.15.81:0/171214211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c9006fa90 msgr2=0x7f9c900ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.295+0000 7f9c986fd640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c9006fa90 0x7f9c900ce520 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400c430 tx=0x7f9c8400c8f0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.296+0000 7f9c986fd640 1 -- 172.21.15.81:0/171214211 shutdown_connections 2024-06-29T15:58:51.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.296+0000 7f9c986fd640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f9c7c076480 0x7f9c7c078990 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.296+0000 7f9c986fd640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9c90074520 0x7f9c90151b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.296+0000 7f9c986fd640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c9006fa90 0x7f9c900ce520 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.296+0000 7f9c986fd640 1 --2- 172.21.15.81:0/171214211 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9c9006f680 0x7f9c9009db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.296+0000 7f9c986fd640 1 -- 172.21.15.81:0/171214211 >> 172.21.15.81:0/171214211 conn(0x7f9c90072950 msgr2=0x7f9c90068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:51.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.296+0000 7f9c986fd640 1 -- 172.21.15.81:0/171214211 shutdown_connections 2024-06-29T15:58:51.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.296+0000 7f9c986fd640 1 -- 172.21.15.81:0/171214211 wait complete. 2024-06-29T15:58:51.299 INFO:tasks.ceph.mds.d.smithi167.stderr: -4568> 2024-06-29T15:58:21.260+0000 7f16caa62640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:51.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:58:51.355 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:58:51.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.424+0000 7f48dcc10640 1 -- 172.21.15.81:0/2353562251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48d80ce260 msgr2=0x7f48d80ce670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.424+0000 7f48dcc10640 1 --2- 172.21.15.81:0/2353562251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48d80ce260 0x7f48d80ce670 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009830 tx=0x7f48cc02a4a0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.424+0000 7f48dcc10640 1 -- 172.21.15.81:0/2353562251 shutdown_connections 2024-06-29T15:58:51.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.424+0000 7f48dcc10640 1 --2- 172.21.15.81:0/2353562251 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48d80ce260 0x7f48d80ce670 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.424+0000 7f48dcc10640 1 --2- 172.21.15.81:0/2353562251 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f48d80b6f50 0x7f48d80d0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.424+0000 7f48dcc10640 1 -- 172.21.15.81:0/2353562251 >> 172.21.15.81:0/2353562251 conn(0x7f48d80b34a0 msgr2=0x7f48d80b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:51.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.424+0000 7f48dcc10640 1 -- 172.21.15.81:0/2353562251 shutdown_connections 2024-06-29T15:58:51.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.424+0000 7f48dcc10640 1 -- 172.21.15.81:0/2353562251 wait complete. 2024-06-29T15:58:51.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.426+0000 7f48dcc10640 1 Processor -- start 2024-06-29T15:58:51.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.426+0000 7f48dcc10640 1 -- start start 2024-06-29T15:58:51.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.427+0000 7f48dcc10640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f48d80b09b0 0x7f48d809f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.427+0000 7f48dcc10640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48d80b6f50 0x7f48d8150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.427+0000 7f48dcc10640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48d80ce260 0x7f48d80b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.427+0000 7f48dcc10640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f48d80cd900 con 0x7f48d80b6f50 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.427+0000 7f48dcc10640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f48d80cd780 con 0x7f48d80ce260 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.427+0000 7f48dcc10640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f48d80b2d40 con 0x7f48d80b09b0 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48d80b6f50 0x7f48d8150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48d80b6f50 0x7f48d8150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32808/0 (socket says 172.21.15.81:32808) 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d5d9b640 1 -- 172.21.15.81:0/1468203243 learned_addr learned my addr 172.21.15.81:0/1468203243 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d6d9d640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48d80ce260 0x7f48d80b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d659c640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f48d80b09b0 0x7f48d809f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d5d9b640 1 -- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f48d80b09b0 msgr2=0x7f48d809f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d5d9b640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f48d80b09b0 0x7f48d809f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d5d9b640 1 -- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48d80ce260 msgr2=0x7f48d80b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d5d9b640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48d80ce260 0x7f48d80b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:51.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d5d9b640 1 -- 172.21.15.81:0/1468203243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d806d530 con 0x7f48d80b6f50 2024-06-29T15:58:51.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48d5d9b640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48d80b6f50 0x7f48d8150110 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f48cc02aa90 tx=0x7f48cc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:51.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48c6ffd640 1 -- 172.21.15.81:0/1468203243 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc005c90 con 0x7f48d80b6f50 2024-06-29T15:58:51.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48c6ffd640 1 -- 172.21.15.81:0/1468203243 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48cc005020 con 0x7f48d80b6f50 2024-06-29T15:58:51.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48dcc10640 1 -- 172.21.15.81:0/1468203243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d809c9a0 con 0x7f48d80b6f50 2024-06-29T15:58:51.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48c6ffd640 1 -- 172.21.15.81:0/1468203243 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc005450 con 0x7f48d80b6f50 2024-06-29T15:58:51.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.428+0000 7f48dcc10640 1 -- 172.21.15.81:0/1468203243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d806dab0 con 0x7f48d80b6f50 2024-06-29T15:58:51.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.429+0000 7f48dcc10640 1 -- 172.21.15.81:0/1468203243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d80ce670 con 0x7f48d80b6f50 2024-06-29T15:58:51.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.430+0000 7f48c6ffd640 1 -- 172.21.15.81:0/1468203243 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 145) ==== 98349+0+0 (secure 0 0 0) 0x7f48cc0317c0 con 0x7f48d80b6f50 2024-06-29T15:58:51.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.431+0000 7f48c6ffd640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f48b8076480 0x7f48b8078990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:51.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.431+0000 7f48c6ffd640 1 -- 172.21.15.81:0/1468203243 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(240..240 src has 1..240) ==== 7647+0+0 (secure 0 0 0) 0x7f48cc0b91f0 con 0x7f48d80b6f50 2024-06-29T15:58:51.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.431+0000 7f48d659c640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f48b8076480 0x7f48b8078990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:51.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.431+0000 7f48d659c640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f48b8076480 0x7f48b8078990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f48d8074b10 tx=0x7f48c8005f90 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:51.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.435+0000 7f48c6ffd640 1 -- 172.21.15.81:0/1468203243 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f48cc03f030 con 0x7f48d80b6f50 2024-06-29T15:58:51.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:51.655+0000 7f48dcc10640 1 -- 172.21.15.81:0/1468203243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f48d80b74c0 con 0x7f48d80b6f50 2024-06-29T15:58:52.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.298+0000 7f48c6ffd640 1 -- 172.21.15.81:0/1468203243 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v137) ==== 104+0+0 (secure 0 0 0) 0x7f48cc081a00 con 0x7f48d80b6f50 2024-06-29T15:58:52.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.303+0000 7f48dcc10640 1 -- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f48b8076480 msgr2=0x7f48b8078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.303+0000 7f48dcc10640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f48b8076480 0x7f48b8078990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f48d8074b10 tx=0x7f48c8005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:52.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.303+0000 7f48dcc10640 1 -- 172.21.15.81:0/1468203243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48d80b6f50 msgr2=0x7f48d8150110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.303+0000 7f48dcc10640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48d80b6f50 0x7f48d8150110 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f48cc02aa90 tx=0x7f48cc003c10 comp rx=0 tx=0).stop 2024-06-29T15:58:52.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.303+0000 7f48dcc10640 1 -- 172.21.15.81:0/1468203243 shutdown_connections 2024-06-29T15:58:52.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.303+0000 7f48dcc10640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f48b8076480 0x7f48b8078990 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.303+0000 7f48dcc10640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f48d80ce260 0x7f48d80b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.303+0000 7f48dcc10640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48d80b6f50 0x7f48d8150110 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.303+0000 7f48dcc10640 1 --2- 172.21.15.81:0/1468203243 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f48d80b09b0 0x7f48d809f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.303+0000 7f48dcc10640 1 -- 172.21.15.81:0/1468203243 >> 172.21.15.81:0/1468203243 conn(0x7f48d80b34a0 msgr2=0x7f48d815efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:52.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.304+0000 7f48dcc10640 1 -- 172.21.15.81:0/1468203243 shutdown_connections 2024-06-29T15:58:52.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.304+0000 7f48dcc10640 1 -- 172.21.15.81:0/1468203243 wait complete. 2024-06-29T15:58:52.317 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:58:52.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.434+0000 7f639f9f0640 1 -- 172.21.15.81:0/2888129247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f639815dde0 msgr2=0x7f639815e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.434+0000 7f639f9f0640 1 --2- 172.21.15.81:0/2888129247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f639815dde0 0x7f639815e1f0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f6398071130 tx=0x7f639002aac0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.434+0000 7f639f9f0640 1 -- 172.21.15.81:0/2888129247 shutdown_connections 2024-06-29T15:58:52.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.434+0000 7f639f9f0640 1 --2- 172.21.15.81:0/2888129247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f639815dde0 0x7f639815e1f0 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.434+0000 7f639f9f0640 1 --2- 172.21.15.81:0/2888129247 >> v2:172.21.15.167:3300/0 conn(0x7f63980b2a30 0x7f6398073ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.434+0000 7f639f9f0640 1 -- 172.21.15.81:0/2888129247 >> 172.21.15.81:0/2888129247 conn(0x7f639806cd60 msgr2=0x7f63980b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:52.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.434+0000 7f639f9f0640 1 -- 172.21.15.81:0/2888129247 shutdown_connections 2024-06-29T15:58:52.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.434+0000 7f639f9f0640 1 -- 172.21.15.81:0/2888129247 wait complete. 2024-06-29T15:58:52.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.435+0000 7f639f9f0640 1 Processor -- start 2024-06-29T15:58:52.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.436+0000 7f639f9f0640 1 -- start start 2024-06-29T15:58:52.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639f9f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63980b2a30 0x7f639814e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639f9f0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63980cf100 0x7f63980cf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639f9f0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f639815dde0 0x7f63980cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639f9f0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6398074190 con 0x7f63980b2a30 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639f9f0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6398074010 con 0x7f639815dde0 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639f9f0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6398070010 con 0x7f63980cf100 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639d78c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63980b2a30 0x7f639814e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639d78c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63980b2a30 0x7f639814e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32826/0 (socket says 172.21.15.81:32826) 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639d78c640 1 -- 172.21.15.81:0/4195147401 learned_addr learned my addr 172.21.15.81:0/4195147401 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639df8d640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f639815dde0 0x7f63980cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639cf8b640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63980cf100 0x7f63980cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639d78c640 1 -- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63980cf100 msgr2=0x7f63980cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639d78c640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63980cf100 0x7f63980cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639d78c640 1 -- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f639815dde0 msgr2=0x7f63980cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639d78c640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f639815dde0 0x7f63980cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639d78c640 1 -- 172.21.15.81:0/4195147401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63980cba10 con 0x7f63980b2a30 2024-06-29T15:58:52.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639cf8b640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63980cf100 0x7f63980cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:52.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.437+0000 7f639d78c640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63980b2a30 0x7f639814e170 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f639400c3d0 tx=0x7f639400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:52.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.438+0000 7f638dffb640 1 -- 172.21.15.81:0/4195147401 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6394017070 con 0x7f63980b2a30 2024-06-29T15:58:52.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.438+0000 7f638dffb640 1 -- 172.21.15.81:0/4195147401 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f63940053c0 con 0x7f63980b2a30 2024-06-29T15:58:52.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.438+0000 7f639f9f0640 1 -- 172.21.15.81:0/4195147401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639814c010 con 0x7f63980b2a30 2024-06-29T15:58:52.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.438+0000 7f638dffb640 1 -- 172.21.15.81:0/4195147401 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6394003850 con 0x7f63980b2a30 2024-06-29T15:58:52.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.438+0000 7f639f9f0640 1 -- 172.21.15.81:0/4195147401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63980cb450 con 0x7f63980b2a30 2024-06-29T15:58:52.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.439+0000 7f639f9f0640 1 -- 172.21.15.81:0/4195147401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6398073ad0 con 0x7f63980b2a30 2024-06-29T15:58:52.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.439+0000 7f638dffb640 1 -- 172.21.15.81:0/4195147401 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f639401a070 con 0x7f63980b2a30 2024-06-29T15:58:52.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.440+0000 7f638dffb640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f637c076510 0x7f637c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:52.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.440+0000 7f638dffb640 1 -- 172.21.15.81:0/4195147401 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(240..240 src has 1..240) ==== 7647+0+0 (secure 0 0 0) 0x7f6394097510 con 0x7f63980b2a30 2024-06-29T15:58:52.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.440+0000 7f639cf8b640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f637c076510 0x7f637c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:52.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.441+0000 7f639cf8b640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f637c076510 0x7f637c078a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6388005fd0 tx=0x7f6388005e20 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:52.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.444+0000 7f638dffb640 1 -- 172.21.15.81:0/4195147401 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f639815e730 con 0x7f63980b2a30 2024-06-29T15:58:52.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:52.625+0000 7f639f9f0640 1 -- 172.21.15.81:0/4195147401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6398149690 con 0x7f63980b2a30 2024-06-29T15:58:53.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.300+0000 7f638dffb640 1 -- 172.21.15.81:0/4195147401 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v241) ==== 181+0+0 (secure 0 0 0) 0x7f639405fe90 con 0x7f63980b2a30 2024-06-29T15:58:53.300 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:58:53.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.305+0000 7f639f9f0640 1 -- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f637c076510 msgr2=0x7f637c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.305+0000 7f639f9f0640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f637c076510 0x7f637c078a20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6388005fd0 tx=0x7f6388005e20 comp rx=0 tx=0).stop 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.305+0000 7f639f9f0640 1 -- 172.21.15.81:0/4195147401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63980b2a30 msgr2=0x7f639814e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.305+0000 7f639f9f0640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63980b2a30 0x7f639814e170 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f639400c3d0 tx=0x7f639400c890 comp rx=0 tx=0).stop 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.307+0000 7f639f9f0640 1 -- 172.21.15.81:0/4195147401 shutdown_connections 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.307+0000 7f639f9f0640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f637c076510 0x7f637c078a20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.307+0000 7f639f9f0640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f639815dde0 0x7f63980cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.307+0000 7f639f9f0640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f63980cf100 0x7f63980cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.307+0000 7f639f9f0640 1 --2- 172.21.15.81:0/4195147401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63980b2a30 0x7f639814e170 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.307+0000 7f639f9f0640 1 -- 172.21.15.81:0/4195147401 >> 172.21.15.81:0/4195147401 conn(0x7f639806cd60 msgr2=0x7f63980ce5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.307+0000 7f639f9f0640 1 -- 172.21.15.81:0/4195147401 shutdown_connections 2024-06-29T15:58:53.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.307+0000 7f639f9f0640 1 -- 172.21.15.81:0/4195147401 wait complete. 2024-06-29T15:58:53.322 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:58:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.447+0000 7fe260b9f640 1 -- 172.21.15.81:0/669876076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe25c0b2ea0 msgr2=0x7fe25c0cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.447+0000 7fe260b9f640 1 --2- 172.21.15.81:0/669876076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe25c0b2ea0 0x7fe25c0cf510 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fe244005fd0 tx=0x7fe24402a540 comp rx=0 tx=0).stop 2024-06-29T15:58:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.447+0000 7fe260b9f640 1 -- 172.21.15.81:0/669876076 shutdown_connections 2024-06-29T15:58:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.447+0000 7fe260b9f640 1 --2- 172.21.15.81:0/669876076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe25c0b2ea0 0x7fe25c0cf510 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.447+0000 7fe260b9f640 1 --2- 172.21.15.81:0/669876076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe25c0d2440 0x7fe25c0d4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.447+0000 7fe260b9f640 1 -- 172.21.15.81:0/669876076 >> 172.21.15.81:0/669876076 conn(0x7fe25c0d3c50 msgr2=0x7fe25c0d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:53.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.447+0000 7fe260b9f640 1 -- 172.21.15.81:0/669876076 shutdown_connections 2024-06-29T15:58:53.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.447+0000 7fe260b9f640 1 -- 172.21.15.81:0/669876076 wait complete. 2024-06-29T15:58:53.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.449+0000 7fe260b9f640 1 Processor -- start 2024-06-29T15:58:53.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.449+0000 7fe260b9f640 1 -- start start 2024-06-29T15:58:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe260b9f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe25c0b2ea0 0x7fe25c1f6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe260b9f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe25c0d03b0 0x7fe25c15bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe260b9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe25c0d2440 0x7fe25c0cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe260b9f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe25c06cb10 con 0x7fe25c0d2440 2024-06-29T15:58:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe260b9f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe25c06cc90 con 0x7fe25c0b2ea0 2024-06-29T15:58:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe260b9f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe25c06ce10 con 0x7fe25c0d03b0 2024-06-29T15:58:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe25ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe25c0d2440 0x7fe25c0cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe25ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe25c0d2440 0x7fe25c0cd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32840/0 (socket says 172.21.15.81:32840) 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe25ad9d640 1 -- 172.21.15.81:0/1461730580 learned_addr learned my addr 172.21.15.81:0/1461730580 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe25a59c640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe25c0b2ea0 0x7fe25c1f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe253fff640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe25c0d03b0 0x7fe25c15bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe25ad9d640 1 -- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe25c0d03b0 msgr2=0x7fe25c15bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe25ad9d640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe25c0d03b0 0x7fe25c15bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe25ad9d640 1 -- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe25c0b2ea0 msgr2=0x7fe25c1f6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe25ad9d640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe25c0b2ea0 0x7fe25c1f6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe25ad9d640 1 -- 172.21.15.81:0/1461730580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe25c0cb620 con 0x7fe25c0d2440 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe25ad9d640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe25c0d2440 0x7fe25c0cd350 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fe25400c760 tx=0x7fe25400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.451+0000 7fe252ffd640 1 -- 172.21.15.81:0/1461730580 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254016020 con 0x7fe25c0d2440 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.452+0000 7fe252ffd640 1 -- 172.21.15.81:0/1461730580 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe25400ce10 con 0x7fe25c0d2440 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.452+0000 7fe260b9f640 1 -- 172.21.15.81:0/1461730580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe25c1f6580 con 0x7fe25c0d2440 2024-06-29T15:58:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.452+0000 7fe252ffd640 1 -- 172.21.15.81:0/1461730580 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254006e60 con 0x7fe25c0d2440 2024-06-29T15:58:53.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.452+0000 7fe260b9f640 1 -- 172.21.15.81:0/1461730580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe25c0cbb70 con 0x7fe25c0d2440 2024-06-29T15:58:53.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.452+0000 7fe260b9f640 1 -- 172.21.15.81:0/1461730580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe25c06d2c0 con 0x7fe25c0d2440 2024-06-29T15:58:53.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.453+0000 7fe252ffd640 1 -- 172.21.15.81:0/1461730580 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fe254003bf0 con 0x7fe25c0d2440 2024-06-29T15:58:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.454+0000 7fe252ffd640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe23c076510 0x7fe23c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.454+0000 7fe252ffd640 1 -- 172.21.15.81:0/1461730580 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(241..241 src has 1..241) ==== 7209+0+0 (secure 0 0 0) 0x7fe254097d00 con 0x7fe25c0d2440 2024-06-29T15:58:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.454+0000 7fe25a59c640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe23c076510 0x7fe23c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:53.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.455+0000 7fe25a59c640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe23c076510 0x7fe23c078a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00a490 tx=0x7fe24c00a420 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:53.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.458+0000 7fe252ffd640 1 -- 172.21.15.81:0/1461730580 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe254020020 con 0x7fe25c0d2440 2024-06-29T15:58:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:53.641+0000 7fe260b9f640 1 -- 172.21.15.81:0/1461730580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe25c0cbdb0 con 0x7fe25c0d2440 2024-06-29T15:58:54.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.307+0000 7fe252ffd640 1 -- 172.21.15.81:0/1461730580 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v242) ==== 169+0+0 (secure 0 0 0) 0x7fe25c1f6580 con 0x7fe25c0d2440 2024-06-29T15:58:54.307 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T15:58:54.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.309+0000 7fe260b9f640 1 -- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe23c076510 msgr2=0x7fe23c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.309+0000 7fe260b9f640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe23c076510 0x7fe23c078a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00a490 tx=0x7fe24c00a420 comp rx=0 tx=0).stop 2024-06-29T15:58:54.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.309+0000 7fe260b9f640 1 -- 172.21.15.81:0/1461730580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe25c0d2440 msgr2=0x7fe25c0cd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.309+0000 7fe260b9f640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe25c0d2440 0x7fe25c0cd350 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fe25400c760 tx=0x7fe25400cc20 comp rx=0 tx=0).stop 2024-06-29T15:58:54.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.309+0000 7fe260b9f640 1 -- 172.21.15.81:0/1461730580 shutdown_connections 2024-06-29T15:58:54.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.309+0000 7fe260b9f640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe23c076510 0x7fe23c078a20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.309+0000 7fe260b9f640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe25c0d2440 0x7fe25c0cd350 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.309+0000 7fe260b9f640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe25c0d03b0 0x7fe25c15bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.309+0000 7fe260b9f640 1 --2- 172.21.15.81:0/1461730580 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe25c0b2ea0 0x7fe25c1f6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.309+0000 7fe260b9f640 1 -- 172.21.15.81:0/1461730580 >> 172.21.15.81:0/1461730580 conn(0x7fe25c0d3c50 msgr2=0x7fe25c0b71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.310+0000 7fe260b9f640 1 -- 172.21.15.81:0/1461730580 shutdown_connections 2024-06-29T15:58:54.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.310+0000 7fe260b9f640 1 -- 172.21.15.81:0/1461730580 wait complete. 2024-06-29T15:58:54.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:54.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.444+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3620841904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f380cea10 msgr2=0x7f5f380b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.444+0000 7f5f3d3d2640 1 --2- 172.21.15.81:0/3620841904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f380cea10 0x7f5f380b47b0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f5f20005fd0 tx=0x7f5f2002a530 comp rx=0 tx=0).stop 2024-06-29T15:58:54.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.444+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3620841904 shutdown_connections 2024-06-29T15:58:54.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.444+0000 7f5f3d3d2640 1 --2- 172.21.15.81:0/3620841904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f380cea10 0x7f5f380b47b0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.444+0000 7f5f3d3d2640 1 --2- 172.21.15.81:0/3620841904 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f380b6d60 0x7f5f380ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.444+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3620841904 >> 172.21.15.81:0/3620841904 conn(0x7f5f380d1960 msgr2=0x7f5f380d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.444+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3620841904 shutdown_connections 2024-06-29T15:58:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.444+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3620841904 wait complete. 2024-06-29T15:58:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.445+0000 7f5f3d3d2640 1 Processor -- start 2024-06-29T15:58:54.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.446+0000 7f5f3d3d2640 1 -- start start 2024-06-29T15:58:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f3d3d2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f380b4cf0 0x7f5f3815d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f3d3d2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f380b6d60 0x7f5f38165ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f3d3d2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f380cea10 0x7f5f380b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f3d3d2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f380cd540 con 0x7f5f380cea10 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f3d3d2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5f380d0a10 con 0x7f5f380b4cf0 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f3d3d2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5f380cd6c0 con 0x7f5f380b6d60 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f380cea10 0x7f5f380b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f380cea10 0x7f5f380b06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32870/0 (socket says 172.21.15.81:32870) 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f377fe640 1 -- 172.21.15.81:0/3373542561 learned_addr learned my addr 172.21.15.81:0/3373542561 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f36ffd640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f380b4cf0 0x7f5f3815d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f2e7fc640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f380b6d60 0x7f5f38165ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f377fe640 1 -- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f380b6d60 msgr2=0x7f5f38165ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f377fe640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f380b6d60 0x7f5f38165ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f377fe640 1 -- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f380b4cf0 msgr2=0x7f5f3815d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f377fe640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f380b4cf0 0x7f5f3815d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f377fe640 1 -- 172.21.15.81:0/3373542561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f381611d0 con 0x7f5f380cea10 2024-06-29T15:58:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.447+0000 7f5f377fe640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f380cea10 0x7f5f380b06a0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f5f3000b9e0 tx=0x7f5f3000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.448+0000 7f5f2ffff640 1 -- 172.21.15.81:0/3373542561 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f30019070 con 0x7f5f380cea10 2024-06-29T15:58:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.448+0000 7f5f2ffff640 1 -- 172.21.15.81:0/3373542561 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f3000cc10 con 0x7f5f380cea10 2024-06-29T15:58:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.448+0000 7f5f2ffff640 1 -- 172.21.15.81:0/3373542561 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f30014440 con 0x7f5f380cea10 2024-06-29T15:58:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.448+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3373542561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f38161470 con 0x7f5f380cea10 2024-06-29T15:58:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.448+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3373542561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f38161a30 con 0x7f5f380cea10 2024-06-29T15:58:54.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.449+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3373542561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f380ce4d0 con 0x7f5f380cea10 2024-06-29T15:58:54.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.450+0000 7f5f2ffff640 1 -- 172.21.15.81:0/3373542561 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f5f3001c070 con 0x7f5f380cea10 2024-06-29T15:58:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.450+0000 7f5f2ffff640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f18076510 0x7f5f18078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.450+0000 7f5f2ffff640 1 -- 172.21.15.81:0/3373542561 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6795+0+0 (secure 0 0 0) 0x7f5f30097ea0 con 0x7f5f380cea10 2024-06-29T15:58:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.450+0000 7f5f36ffd640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f18076510 0x7f5f18078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.451+0000 7f5f36ffd640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f18076510 0x7f5f18078a20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f380c5220 tx=0x7f5f28008040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:54.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.454+0000 7f5f2ffff640 1 -- 172.21.15.81:0/3373542561 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5f30060f50 con 0x7f5f380cea10 2024-06-29T15:58:54.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.640+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3373542561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f3815d6c0 con 0x7f5f380cea10 2024-06-29T15:58:54.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.641+0000 7f5f2ffff640 1 -- 172.21.15.81:0/3373542561 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) ==== 74+0+18219 (secure 0 0 0) 0x7f5f30060850 con 0x7f5f380cea10 2024-06-29T15:58:54.642 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:54.642 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":242,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:54.300992+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:29.291185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:29.702735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6837/30158598":"2024-06-30T15:58:51.291829+0000","172.21.15.167:6835/30158598":"2024-06-30T15:58:51.291829+0000","172.21.15.167:6833/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:6832/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/355438800":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4195196310":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4186712115":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/3601299355":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2623346726":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/972265206":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/1116386548":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6832/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/3057737710":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6833/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1418041738":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/561620344":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2207847535":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2067929640":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/874327121":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/273305307":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2269612087":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1188031263":"2024-06-30T15:58:42.086693+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.643+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f18076510 msgr2=0x7f5f18078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.643+0000 7f5f3d3d2640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f18076510 0x7f5f18078a20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5f380c5220 tx=0x7f5f28008040 comp rx=0 tx=0).stop 2024-06-29T15:58:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.643+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3373542561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f380cea10 msgr2=0x7f5f380b06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.643+0000 7f5f3d3d2640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f380cea10 0x7f5f380b06a0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f5f3000b9e0 tx=0x7f5f3000bea0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.643+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3373542561 shutdown_connections 2024-06-29T15:58:54.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.643+0000 7f5f3d3d2640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f18076510 0x7f5f18078a20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.643+0000 7f5f3d3d2640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f380cea10 0x7f5f380b06a0 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.643+0000 7f5f3d3d2640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f380b6d60 0x7f5f38165ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.643+0000 7f5f3d3d2640 1 --2- 172.21.15.81:0/3373542561 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f380b4cf0 0x7f5f3815d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.643+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3373542561 >> 172.21.15.81:0/3373542561 conn(0x7f5f380d1960 msgr2=0x7f5f38070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.644+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3373542561 shutdown_connections 2024-06-29T15:58:54.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.644+0000 7f5f3d3d2640 1 -- 172.21.15.81:0/3373542561 wait complete. 2024-06-29T15:58:54.656 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2024-06-29T15:58:29.291185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2024-06-29T15:58:29.702735+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '237', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:58:54.656 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:58:54.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.774+0000 7ff547b31640 1 -- 172.21.15.81:0/995647281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff540074520 msgr2=0x7ff54006f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.774+0000 7ff547b31640 1 --2- 172.21.15.81:0/995647281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff540074520 0x7ff54006f140 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7ff530005fd0 tx=0x7ff53002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.774+0000 7ff547b31640 1 -- 172.21.15.81:0/995647281 shutdown_connections 2024-06-29T15:58:54.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.774+0000 7ff547b31640 1 --2- 172.21.15.81:0/995647281 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff54006f680 0x7ff54006c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.774+0000 7ff547b31640 1 --2- 172.21.15.81:0/995647281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff540074520 0x7ff54006f140 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.774+0000 7ff547b31640 1 -- 172.21.15.81:0/995647281 >> 172.21.15.81:0/995647281 conn(0x7ff540072950 msgr2=0x7ff5400b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.774+0000 7ff547b31640 1 -- 172.21.15.81:0/995647281 shutdown_connections 2024-06-29T15:58:54.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.774+0000 7ff547b31640 1 -- 172.21.15.81:0/995647281 wait complete. 2024-06-29T15:58:54.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.776+0000 7ff547b31640 1 Processor -- start 2024-06-29T15:58:54.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.776+0000 7ff547b31640 1 -- start start 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff547b31640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff54006f680 0x7ff5401552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff547b31640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54006fa90 0x7ff5401483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff547b31640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff540074520 0x7ff5401488f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff547b31640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff54015b010 con 0x7ff54006fa90 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff547b31640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff54006d150 con 0x7ff540074520 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff547b31640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff54015b190 con 0x7ff54006f680 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5450cc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54006fa90 0x7ff5401483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5450cc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54006fa90 0x7ff5401483b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32888/0 (socket says 172.21.15.81:32888) 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5450cc640 1 -- 172.21.15.81:0/4239431818 learned_addr learned my addr 172.21.15.81:0/4239431818 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5458cd640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff54006f680 0x7ff5401552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5460ce640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff540074520 0x7ff5401488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5450cc640 1 -- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff54006f680 msgr2=0x7ff5401552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5450cc640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff54006f680 0x7ff5401552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5450cc640 1 -- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff540074520 msgr2=0x7ff5401488f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5450cc640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff540074520 0x7ff5401488f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5450cc640 1 -- 172.21.15.81:0/4239431818 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff540156060 con 0x7ff54006fa90 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.777+0000 7ff5460ce640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff540074520 0x7ff5401488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.778+0000 7ff5450cc640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54006fa90 0x7ff5401483b0 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7ff53400d740 tx=0x7ff53400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.778+0000 7ff53e7fc640 1 -- 172.21.15.81:0/4239431818 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff534017070 con 0x7ff54006fa90 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.778+0000 7ff53e7fc640 1 -- 172.21.15.81:0/4239431818 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff534009cc0 con 0x7ff54006fa90 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.778+0000 7ff547b31640 1 -- 172.21.15.81:0/4239431818 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff540156340 con 0x7ff54006fa90 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.778+0000 7ff53e7fc640 1 -- 172.21.15.81:0/4239431818 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5340106c0 con 0x7ff54006fa90 2024-06-29T15:58:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.778+0000 7ff547b31640 1 -- 172.21.15.81:0/4239431818 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff540156900 con 0x7ff54006fa90 2024-06-29T15:58:54.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.779+0000 7ff547b31640 1 -- 172.21.15.81:0/4239431818 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff54009d990 con 0x7ff54006fa90 2024-06-29T15:58:54.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.780+0000 7ff53e7fc640 1 -- 172.21.15.81:0/4239431818 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7ff53401a070 con 0x7ff54006fa90 2024-06-29T15:58:54.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.780+0000 7ff53e7fc640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff52c076510 0x7ff52c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:54.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.780+0000 7ff53e7fc640 1 -- 172.21.15.81:0/4239431818 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6795+0+0 (secure 0 0 0) 0x7ff5340980a0 con 0x7ff54006fa90 2024-06-29T15:58:54.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.780+0000 7ff5458cd640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff52c076510 0x7ff52c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:54.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.781+0000 7ff5458cd640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff52c076510 0x7ff52c078a20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff530033000 tx=0x7ff530004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:54.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.784+0000 7ff53e7fc640 1 -- 172.21.15.81:0/4239431818 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff54009d990 con 0x7ff54006fa90 2024-06-29T15:58:54.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.967+0000 7ff547b31640 1 -- 172.21.15.81:0/4239431818 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff540068ef0 con 0x7ff54006fa90 2024-06-29T15:58:54.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.968+0000 7ff53e7fc640 1 -- 172.21.15.81:0/4239431818 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v242) ==== 93+0+10 (secure 0 0 0) 0x7ff534060b90 con 0x7ff54006fa90 2024-06-29T15:58:54.968 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:58:54.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 -- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff52c076510 msgr2=0x7ff52c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff52c076510 0x7ff52c078a20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff530033000 tx=0x7ff530004730 comp rx=0 tx=0).stop 2024-06-29T15:58:54.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 -- 172.21.15.81:0/4239431818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54006fa90 msgr2=0x7ff5401483b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:54.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54006fa90 0x7ff5401483b0 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7ff53400d740 tx=0x7ff53400dc00 comp rx=0 tx=0).stop 2024-06-29T15:58:54.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 -- 172.21.15.81:0/4239431818 shutdown_connections 2024-06-29T15:58:54.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff52c076510 0x7ff52c078a20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff540074520 0x7ff5401488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff54006fa90 0x7ff5401483b0 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 --2- 172.21.15.81:0/4239431818 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff54006f680 0x7ff5401552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:54.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 -- 172.21.15.81:0/4239431818 >> 172.21.15.81:0/4239431818 conn(0x7ff540072950 msgr2=0x7ff54009d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:54.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 -- 172.21.15.81:0/4239431818 shutdown_connections 2024-06-29T15:58:54.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:54.970+0000 7ff547b31640 1 -- 172.21.15.81:0/4239431818 wait complete. 2024-06-29T15:58:54.982 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2024-06-29T15:58:55.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.100+0000 7faebe30a640 1 -- 172.21.15.81:0/4070371586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeb80b4cf0 msgr2=0x7faeb80d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.100+0000 7faebe30a640 1 --2- 172.21.15.81:0/4070371586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeb80b4cf0 0x7faeb80d00f0 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7faeb80b2c10 tx=0x7faeb002aa00 comp rx=0 tx=0).stop 2024-06-29T15:58:55.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.100+0000 7faebe30a640 1 -- 172.21.15.81:0/4070371586 shutdown_connections 2024-06-29T15:58:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.100+0000 7faebe30a640 1 --2- 172.21.15.81:0/4070371586 >> v2:172.21.15.167:3300/0 conn(0x7faeb80cd540 0x7faeb80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.100+0000 7faebe30a640 1 --2- 172.21.15.81:0/4070371586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeb80b4cf0 0x7faeb80d00f0 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.100+0000 7faebe30a640 1 -- 172.21.15.81:0/4070371586 >> 172.21.15.81:0/4070371586 conn(0x7faeb80d1960 msgr2=0x7faeb80d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.100+0000 7faebe30a640 1 -- 172.21.15.81:0/4070371586 shutdown_connections 2024-06-29T15:58:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.100+0000 7faebe30a640 1 -- 172.21.15.81:0/4070371586 wait complete. 2024-06-29T15:58:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.101+0000 7faebe30a640 1 Processor -- start 2024-06-29T15:58:55.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.102+0000 7faebe30a640 1 -- start start 2024-06-29T15:58:55.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebe30a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeb80b4cf0 0x7faeb81f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebe30a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faeb80b6d60 0x7faeb81f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebe30a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeb80cd540 0x7faeb80c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebe30a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faeb80d0630 con 0x7faeb80cd540 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebe30a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7faeb80b2790 con 0x7faeb80b4cf0 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebe30a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7faeb80d07b0 con 0x7faeb80b6d60 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebc8a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeb80cd540 0x7faeb80c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebc8a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeb80cd540 0x7faeb80c7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32908/0 (socket says 172.21.15.81:32908) 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebc8a7640 1 -- 172.21.15.81:0/2966490658 learned_addr learned my addr 172.21.15.81:0/2966490658 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faeb7fff640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeb80b4cf0 0x7faeb81f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faeb77fe640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faeb80b6d60 0x7faeb81f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebc8a7640 1 -- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faeb80b6d60 msgr2=0x7faeb81f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebc8a7640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faeb80b6d60 0x7faeb81f6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebc8a7640 1 -- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeb80b4cf0 msgr2=0x7faeb81f6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebc8a7640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeb80b4cf0 0x7faeb81f6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faebc8a7640 1 -- 172.21.15.81:0/2966490658 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb80cbd00 con 0x7faeb80cd540 2024-06-29T15:58:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faeb77fe640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faeb80b6d60 0x7faeb81f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.103+0000 7faeb7fff640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeb80b4cf0 0x7faeb81f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.104+0000 7faebc8a7640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeb80cd540 0x7faeb80c7d20 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7faeb002aff0 tx=0x7faeb0003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.104+0000 7faeb4ff9640 1 -- 172.21.15.81:0/2966490658 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb0033050 con 0x7faeb80cd540 2024-06-29T15:58:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.104+0000 7faeb4ff9640 1 -- 172.21.15.81:0/2966490658 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faeb00044d0 con 0x7faeb80cd540 2024-06-29T15:58:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.104+0000 7faeb4ff9640 1 -- 172.21.15.81:0/2966490658 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb0038450 con 0x7faeb80cd540 2024-06-29T15:58:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.104+0000 7faebe30a640 1 -- 172.21.15.81:0/2966490658 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb814abf0 con 0x7faeb80cd540 2024-06-29T15:58:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.104+0000 7faebe30a640 1 -- 172.21.15.81:0/2966490658 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb814c750 con 0x7faeb80cd540 2024-06-29T15:58:55.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.105+0000 7faebe30a640 1 -- 172.21.15.81:0/2966490658 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb8068960 con 0x7faeb80cd540 2024-06-29T15:58:55.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.105+0000 7faeb4ff9640 1 -- 172.21.15.81:0/2966490658 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7faeb0036070 con 0x7faeb80cd540 2024-06-29T15:58:55.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.106+0000 7faeb4ff9640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7faea4076540 0x7faea4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.106+0000 7faeb4ff9640 1 -- 172.21.15.81:0/2966490658 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6795+0+0 (secure 0 0 0) 0x7faeb00b9540 con 0x7faeb80cd540 2024-06-29T15:58:55.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.106+0000 7faeb7fff640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7faea4076540 0x7faea4078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.107+0000 7faeb7fff640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7faea4076540 0x7faea4078a50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faea8006fd0 tx=0x7faea8008040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:55.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.110+0000 7faeb4ff9640 1 -- 172.21.15.81:0/2966490658 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7faeb8068960 con 0x7faeb80cd540 2024-06-29T15:58:55.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.291+0000 7faebe30a640 1 -- 172.21.15.81:0/2966490658 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) -- 0x7faeb80d00f0 con 0x7faeb80cd540 2024-06-29T15:58:55.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.291+0000 7faeb4ff9640 1 -- 172.21.15.81:0/2966490658 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v242) ==== 107+0+11 (secure 0 0 0) 0x7faeb00825f0 con 0x7faeb80cd540 2024-06-29T15:58:55.291 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:58:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 -- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7faea4076540 msgr2=0x7faea4078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7faea4076540 0x7faea4078a50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faea8006fd0 tx=0x7faea8008040 comp rx=0 tx=0).stop 2024-06-29T15:58:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 -- 172.21.15.81:0/2966490658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeb80cd540 msgr2=0x7faeb80c7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeb80cd540 0x7faeb80c7d20 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7faeb002aff0 tx=0x7faeb0003f90 comp rx=0 tx=0).stop 2024-06-29T15:58:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 -- 172.21.15.81:0/2966490658 shutdown_connections 2024-06-29T15:58:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7faea4076540 0x7faea4078a50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faeb80cd540 0x7faeb80c7d20 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7faeb80b6d60 0x7faeb81f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 --2- 172.21.15.81:0/2966490658 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7faeb80b4cf0 0x7faeb81f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 -- 172.21.15.81:0/2966490658 >> 172.21.15.81:0/2966490658 conn(0x7faeb80d1960 msgr2=0x7faeb80b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:55.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.293+0000 7faebe30a640 1 -- 172.21.15.81:0/2966490658 shutdown_connections 2024-06-29T15:58:55.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.294+0000 7faebe30a640 1 -- 172.21.15.81:0/2966490658 wait complete. 2024-06-29T15:58:55.305 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2024-06-29T15:58:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.424+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3540691602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0bf0004800 msgr2=0x7f0bf0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.424+0000 7f0c0d3c7640 1 --2- 172.21.15.81:0/3540691602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0bf0004800 0x7f0bf0004c10 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009b20 tx=0x7f0bf0005830 comp rx=0 tx=0).stop 2024-06-29T15:58:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.424+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3540691602 shutdown_connections 2024-06-29T15:58:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.424+0000 7f0c0d3c7640 1 --2- 172.21.15.81:0/3540691602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0bf0004800 0x7f0bf0004c10 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.424+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3540691602 >> 172.21.15.81:0/3540691602 conn(0x7f0c080b2920 msgr2=0x7f0c080b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.424+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3540691602 shutdown_connections 2024-06-29T15:58:55.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.424+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3540691602 wait complete. 2024-06-29T15:58:55.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.426+0000 7f0c0d3c7640 1 Processor -- start 2024-06-29T15:58:55.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.426+0000 7f0c0d3c7640 1 -- start start 2024-06-29T15:58:55.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.427+0000 7f0c0d3c7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bf0004800 0x7f0c0814a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.427+0000 7f0c0d3c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c0814b520 0x7f0c0814f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.427+0000 7f0c0d3c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c080d0290 0x7f0c080d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.427+0000 7f0c0d3c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0815b860 con 0x7f0c0814b520 2024-06-29T15:58:55.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.427+0000 7f0c0d3c7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0815b6e0 con 0x7f0c080d0290 2024-06-29T15:58:55.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.427+0000 7f0c0d3c7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0c0815b9e0 con 0x7f0bf0004800 2024-06-29T15:58:55.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.427+0000 7f0c067fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c0814b520 0x7f0c0814f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c067fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c0814b520 0x7f0c0814f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32936/0 (socket says 172.21.15.81:32936) 2024-06-29T15:58:55.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c067fc640 1 -- 172.21.15.81:0/3147283668 learned_addr learned my addr 172.21.15.81:0/3147283668 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:55.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c077fe640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c080d0290 0x7f0c080d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c06ffd640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bf0004800 0x7f0c0814a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c067fc640 1 -- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bf0004800 msgr2=0x7f0c0814a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c067fc640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bf0004800 0x7f0c0814a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c067fc640 1 -- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c080d0290 msgr2=0x7f0c080d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c067fc640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c080d0290 0x7f0c080d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c067fc640 1 -- 172.21.15.81:0/3147283668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c08151920 con 0x7f0c0814b520 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c067fc640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c0814b520 0x7f0c0814f790 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc036040 tx=0x7f0bfc033c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0bef7fe640 1 -- 172.21.15.81:0/3147283668 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc042020 con 0x7f0c0814b520 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0bef7fe640 1 -- 172.21.15.81:0/3147283668 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0bfc03e470 con 0x7f0c0814b520 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0bef7fe640 1 -- 172.21.15.81:0/3147283668 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc038040 con 0x7f0c0814b520 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3147283668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c080cf4d0 con 0x7f0c0814b520 2024-06-29T15:58:55.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.428+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3147283668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c080cfa90 con 0x7f0c0814b520 2024-06-29T15:58:55.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.429+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3147283668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c08151690 con 0x7f0c0814b520 2024-06-29T15:58:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.430+0000 7f0bef7fe640 1 -- 172.21.15.81:0/3147283668 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f0bfc045020 con 0x7f0c0814b520 2024-06-29T15:58:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.430+0000 7f0bef7fe640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0bf4076510 0x7f0bf4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.430+0000 7f0bef7fe640 1 -- 172.21.15.81:0/3147283668 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6795+0+0 (secure 0 0 0) 0x7f0bfc0bb330 con 0x7f0c0814b520 2024-06-29T15:58:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.431+0000 7f0c06ffd640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0bf4076510 0x7f0bf4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.431+0000 7f0c06ffd640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0bf4076510 0x7f0bf4078a20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8004fd0 tx=0x7f0bf8006de0 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.435+0000 7f0bef7fe640 1 -- 172.21.15.81:0/3147283668 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0bfc04c030 con 0x7f0c0814b520 2024-06-29T15:58:55.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.615+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3147283668 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) -- 0x7f0c080732c0 con 0x7f0c0814b520 2024-06-29T15:58:55.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.615+0000 7f0bef7fe640 1 -- 172.21.15.81:0/3147283668 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v242) ==== 103+0+11 (secure 0 0 0) 0x7f0bfc033e20 con 0x7f0c0814b520 2024-06-29T15:58:55.615 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:58:55.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0bf4076510 msgr2=0x7f0bf4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0bf4076510 0x7f0bf4078a20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8004fd0 tx=0x7f0bf8006de0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3147283668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c0814b520 msgr2=0x7f0c0814f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c0814b520 0x7f0c0814f790 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc036040 tx=0x7f0bfc033c30 comp rx=0 tx=0).stop 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3147283668 shutdown_connections 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0bf4076510 0x7f0bf4078a20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0c080d0290 0x7f0c080d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c0814b520 0x7f0c0814f790 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 --2- 172.21.15.81:0/3147283668 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0bf0004800 0x7f0c0814a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3147283668 >> 172.21.15.81:0/3147283668 conn(0x7f0c080b2920 msgr2=0x7f0c080b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3147283668 shutdown_connections 2024-06-29T15:58:55.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.617+0000 7f0c0d3c7640 1 -- 172.21.15.81:0/3147283668 wait complete. 2024-06-29T15:58:55.629 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.742+0000 7fb7d7838640 1 -- 172.21.15.81:0/1191391082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d00d2a90 msgr2=0x7fb7d00d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.742+0000 7fb7d7838640 1 --2- 172.21.15.81:0/1191391082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d00d2a90 0x7fb7d00d09c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005fd0 tx=0x7fb7c802ab20 comp rx=0 tx=0).stop 2024-06-29T15:58:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.742+0000 7fb7d7838640 1 -- 172.21.15.81:0/1191391082 shutdown_connections 2024-06-29T15:58:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.742+0000 7fb7d7838640 1 --2- 172.21.15.81:0/1191391082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d00d2a90 0x7fb7d00d09c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.742+0000 7fb7d7838640 1 --2- 172.21.15.81:0/1191391082 >> v2:172.21.15.167:3300/0 conn(0x7fb7d00b9040 0x7fb7d00d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.742+0000 7fb7d7838640 1 -- 172.21.15.81:0/1191391082 >> 172.21.15.81:0/1191391082 conn(0x7fb7d009cf30 msgr2=0x7fb7d00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:55.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.742+0000 7fb7d7838640 1 -- 172.21.15.81:0/1191391082 shutdown_connections 2024-06-29T15:58:55.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.742+0000 7fb7d7838640 1 -- 172.21.15.81:0/1191391082 wait complete. 2024-06-29T15:58:55.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.744+0000 7fb7d7838640 1 Processor -- start 2024-06-29T15:58:55.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.744+0000 7fb7d7838640 1 -- start start 2024-06-29T15:58:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.745+0000 7fb7d7838640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d00b5250 0x7fb7d00b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.745+0000 7fb7d7838640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d00b9040 0x7fb7d00a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.745+0000 7fb7d7838640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d00d2a90 0x7fb7d0150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.745+0000 7fb7d7838640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d00b7d30 con 0x7fb7d00b5250 2024-06-29T15:58:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.745+0000 7fb7d7838640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d00b7bb0 con 0x7fb7d00b9040 2024-06-29T15:58:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.745+0000 7fb7d7838640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb7d00b7eb0 con 0x7fb7d00d2a90 2024-06-29T15:58:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.745+0000 7fb7d55d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d00b5250 0x7fb7d00b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.745+0000 7fb7d55d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d00b5250 0x7fb7d00b0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32952/0 (socket says 172.21.15.81:32952) 2024-06-29T15:58:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.745+0000 7fb7d55d4640 1 -- 172.21.15.81:0/2940121361 learned_addr learned my addr 172.21.15.81:0/2940121361 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:55.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7d5dd5640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d00d2a90 0x7fb7d0150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7d4dd3640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d00b9040 0x7fb7d00a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7d55d4640 1 -- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d00d2a90 msgr2=0x7fb7d0150370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7d55d4640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d00d2a90 0x7fb7d0150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7d55d4640 1 -- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d00b9040 msgr2=0x7fb7d00a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7d55d4640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d00b9040 0x7fb7d00a0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7d55d4640 1 -- 172.21.15.81:0/2940121361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d014e6e0 con 0x7fb7d00b5250 2024-06-29T15:58:55.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7d55d4640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d00b5250 0x7fb7d00b0b60 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00ea20 tx=0x7fb7cc00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7c5ffb640 1 -- 172.21.15.81:0/2940121361 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00ce40 con 0x7fb7d00b5250 2024-06-29T15:58:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7c5ffb640 1 -- 172.21.15.81:0/2940121361 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7cc004020 con 0x7fb7d00b5250 2024-06-29T15:58:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7d7838640 1 -- 172.21.15.81:0/2940121361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d014e9c0 con 0x7fb7d00b5250 2024-06-29T15:58:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7c5ffb640 1 -- 172.21.15.81:0/2940121361 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc004470 con 0x7fb7d00b5250 2024-06-29T15:58:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.746+0000 7fb7d7838640 1 -- 172.21.15.81:0/2940121361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d014ef80 con 0x7fb7d00b5250 2024-06-29T15:58:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.747+0000 7fb7d7838640 1 -- 172.21.15.81:0/2940121361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7d00b8440 con 0x7fb7d00b5250 2024-06-29T15:58:55.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.748+0000 7fb7c5ffb640 1 -- 172.21.15.81:0/2940121361 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fb7cc0035f0 con 0x7fb7d00b5250 2024-06-29T15:58:55.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.748+0000 7fb7c5ffb640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fb7b4076510 0x7fb7b4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:55.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.749+0000 7fb7c5ffb640 1 -- 172.21.15.81:0/2940121361 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6795+0+0 (secure 0 0 0) 0x7fb7cc098ba0 con 0x7fb7d00b5250 2024-06-29T15:58:55.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.749+0000 7fb7d4dd3640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fb7b4076510 0x7fb7b4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:55.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.749+0000 7fb7d4dd3640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fb7b4076510 0x7fb7b4078a20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb7d00c7880 tx=0x7fb7c0009040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:55.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.753+0000 7fb7c5ffb640 1 -- 172.21.15.81:0/2940121361 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb7cc061bd0 con 0x7fb7d00b5250 2024-06-29T15:58:55.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.972+0000 7fb7d7838640 1 -- 172.21.15.81:0/2940121361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7d00b4b00 con 0x7fb7d00b5250 2024-06-29T15:58:55.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.973+0000 7fb7c5ffb640 1 -- 172.21.15.81:0/2940121361 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+5243 (secure 0 0 0) 0x7fb7cc0614d0 con 0x7fb7d00b5250 2024-06-29T15:58:55.973 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:55.974 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":137,"btime":"2024-06-29T15:58:52:294249+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":11218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/113384617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":113384617},{"type":"v1","addr":"172.21.15.167:6837","nonce":113384617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:30.737083+0000","modified":"2024-06-29T15:58:33.773132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12093":{"gid":12093,"name":"a","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12093,"qdb_cluster":[12093]},"id":16}]} 2024-06-29T15:58:55.974 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 137 2024-06-29T15:58:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 -- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fb7b4076510 msgr2=0x7fb7b4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fb7b4076510 0x7fb7b4078a20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb7d00c7880 tx=0x7fb7c0009040 comp rx=0 tx=0).stop 2024-06-29T15:58:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 -- 172.21.15.81:0/2940121361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d00b5250 msgr2=0x7fb7d00b0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d00b5250 0x7fb7d00b0b60 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00ea20 tx=0x7fb7cc00eee0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 -- 172.21.15.81:0/2940121361 shutdown_connections 2024-06-29T15:58:55.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fb7b4076510 0x7fb7b4078a20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb7d00d2a90 0x7fb7d0150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb7d00b9040 0x7fb7d00a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 --2- 172.21.15.81:0/2940121361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d00b5250 0x7fb7d00b0b60 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 -- 172.21.15.81:0/2940121361 >> 172.21.15.81:0/2940121361 conn(0x7fb7d009cf30 msgr2=0x7fb7d01613b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 -- 172.21.15.81:0/2940121361 shutdown_connections 2024-06-29T15:58:55.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:55.975+0000 7fb7d7838640 1 -- 172.21.15.81:0/2940121361 wait complete. 2024-06-29T15:58:55.987 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:56.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.100+0000 7efd1be3f640 1 -- 172.21.15.81:0/683710133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd140b6eb0 msgr2=0x7efd140d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.100+0000 7efd1be3f640 1 --2- 172.21.15.81:0/683710133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd140b6eb0 0x7efd140d02d0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7efd14073550 tx=0x7efd1002a4d0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.100+0000 7efd1be3f640 1 -- 172.21.15.81:0/683710133 shutdown_connections 2024-06-29T15:58:56.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.100+0000 7efd1be3f640 1 --2- 172.21.15.81:0/683710133 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd140b4810 0x7efd140b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.100+0000 7efd1be3f640 1 --2- 172.21.15.81:0/683710133 >> v2:172.21.15.167:3300/0 conn(0x7efd140ce510 0x7efd140ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.100+0000 7efd1be3f640 1 --2- 172.21.15.81:0/683710133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd140b6eb0 0x7efd140d02d0 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.100+0000 7efd1be3f640 1 -- 172.21.15.81:0/683710133 >> 172.21.15.81:0/683710133 conn(0x7efd140b3510 msgr2=0x7efd140b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:56.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.100+0000 7efd1be3f640 1 -- 172.21.15.81:0/683710133 shutdown_connections 2024-06-29T15:58:56.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.100+0000 7efd1be3f640 1 -- 172.21.15.81:0/683710133 wait complete. 2024-06-29T15:58:56.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.102+0000 7efd1be3f640 1 Processor -- start 2024-06-29T15:58:56.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.102+0000 7efd1be3f640 1 -- start start 2024-06-29T15:58:56.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd1be3f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd140b4810 0x7efd141df390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd1be3f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd140b6eb0 0x7efd140b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd1be3f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd140ce510 0x7efd140b5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd1be3f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd140b52e0 con 0x7efd140b6eb0 2024-06-29T15:58:56.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd1be3f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7efd140b5160 con 0x7efd140ce510 2024-06-29T15:58:56.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd1be3f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7efd140cc470 con 0x7efd140b4810 2024-06-29T15:58:56.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd193da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd140b6eb0 0x7efd140b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd193da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd140b6eb0 0x7efd140b1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32974/0 (socket says 172.21.15.81:32974) 2024-06-29T15:58:56.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd193da640 1 -- 172.21.15.81:0/3292148529 learned_addr learned my addr 172.21.15.81:0/3292148529 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:56.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd19bdb640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd140b4810 0x7efd141df390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd1a3dc640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd140ce510 0x7efd140b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd193da640 1 -- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd140b4810 msgr2=0x7efd141df390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd193da640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd140b4810 0x7efd141df390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd193da640 1 -- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd140ce510 msgr2=0x7efd140b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd193da640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd140ce510 0x7efd140b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd193da640 1 -- 172.21.15.81:0/3292148529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd140c6860 con 0x7efd140b6eb0 2024-06-29T15:58:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.103+0000 7efd1a3dc640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd140ce510 0x7efd140b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.104+0000 7efd193da640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd140b6eb0 0x7efd140b1190 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7efd0400d770 tx=0x7efd0400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.104+0000 7efd0a7fc640 1 -- 172.21.15.81:0/3292148529 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd04017070 con 0x7efd140b6eb0 2024-06-29T15:58:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.104+0000 7efd0a7fc640 1 -- 172.21.15.81:0/3292148529 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd04009d20 con 0x7efd140b6eb0 2024-06-29T15:58:56.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.104+0000 7efd1be3f640 1 -- 172.21.15.81:0/3292148529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd140c6b40 con 0x7efd140b6eb0 2024-06-29T15:58:56.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.104+0000 7efd0a7fc640 1 -- 172.21.15.81:0/3292148529 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd040106c0 con 0x7efd140b6eb0 2024-06-29T15:58:56.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.104+0000 7efd1be3f640 1 -- 172.21.15.81:0/3292148529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd140c7100 con 0x7efd140b6eb0 2024-06-29T15:58:56.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.105+0000 7efd1be3f640 1 -- 172.21.15.81:0/3292148529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd140d02d0 con 0x7efd140b6eb0 2024-06-29T15:58:56.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.106+0000 7efd0a7fc640 1 -- 172.21.15.81:0/3292148529 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7efd0401a070 con 0x7efd140b6eb0 2024-06-29T15:58:56.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.106+0000 7efd0a7fc640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7efcf8076510 0x7efcf8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.106+0000 7efd0a7fc640 1 -- 172.21.15.81:0/3292148529 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6795+0+0 (secure 0 0 0) 0x7efd04098240 con 0x7efd140b6eb0 2024-06-29T15:58:56.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.106+0000 7efd19bdb640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7efcf8076510 0x7efcf8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.107+0000 7efd19bdb640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7efcf8076510 0x7efcf8078a20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efd10033000 tx=0x7efd10004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.110+0000 7efd0a7fc640 1 -- 172.21.15.81:0/3292148529 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7efd040612f0 con 0x7efd140b6eb0 2024-06-29T15:58:56.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.293+0000 7efd1be3f640 1 -- 172.21.15.81:0/3292148529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd140cdf00 con 0x7efd140b6eb0 2024-06-29T15:58:56.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.294+0000 7efd0a7fc640 1 -- 172.21.15.81:0/3292148529 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) ==== 74+0+18219 (secure 0 0 0) 0x7efd04060bf0 con 0x7efd140b6eb0 2024-06-29T15:58:56.295 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:56.295 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":242,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:54.300992+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:29.291185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:29.702735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6837/30158598":"2024-06-30T15:58:51.291829+0000","172.21.15.167:6835/30158598":"2024-06-30T15:58:51.291829+0000","172.21.15.167:6833/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:6832/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/355438800":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4195196310":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4186712115":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/3601299355":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2623346726":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/972265206":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/1116386548":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6832/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/3057737710":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6833/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1418041738":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/561620344":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2207847535":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2067929640":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/874327121":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/273305307":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2269612087":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1188031263":"2024-06-30T15:58:42.086693+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 -- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7efcf8076510 msgr2=0x7efcf8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7efcf8076510 0x7efcf8078a20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efd10033000 tx=0x7efd10004730 comp rx=0 tx=0).stop 2024-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 -- 172.21.15.81:0/3292148529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd140b6eb0 msgr2=0x7efd140b1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd140b6eb0 0x7efd140b1190 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7efd0400d770 tx=0x7efd0400dc30 comp rx=0 tx=0).stop 2024-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 -- 172.21.15.81:0/3292148529 shutdown_connections 2024-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7efcf8076510 0x7efcf8078a20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7efd140ce510 0x7efd140b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd140b6eb0 0x7efd140b1190 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 --2- 172.21.15.81:0/3292148529 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7efd140b4810 0x7efd141df390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 -- 172.21.15.81:0/3292148529 >> 172.21.15.81:0/3292148529 conn(0x7efd140b3510 msgr2=0x7efd140ce920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:56.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 -- 172.21.15.81:0/3292148529 shutdown_connections 2024-06-29T15:58:56.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.296+0000 7efd1be3f640 1 -- 172.21.15.81:0/3292148529 wait complete. 2024-06-29T15:58:56.308 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2024-06-29T15:58:56.308 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:56.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.421+0000 7f8f09212640 1 -- 172.21.15.81:0/4294823455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f0415cd00 msgr2=0x7f8f0415d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.421+0000 7f8f09212640 1 --2- 172.21.15.81:0/4294823455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f0415cd00 0x7f8f0415d510 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f8f040b4f30 tx=0x7f8efc02aa30 comp rx=0 tx=0).stop 2024-06-29T15:58:56.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.421+0000 7f8f09212640 1 -- 172.21.15.81:0/4294823455 shutdown_connections 2024-06-29T15:58:56.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.421+0000 7f8f09212640 1 --2- 172.21.15.81:0/4294823455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f0415cd00 0x7f8f0415d510 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.421+0000 7f8f09212640 1 --2- 172.21.15.81:0/4294823455 >> v2:172.21.15.167:3300/0 conn(0x7f8f04145530 0x7f8f04073280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.421+0000 7f8f09212640 1 -- 172.21.15.81:0/4294823455 >> 172.21.15.81:0/4294823455 conn(0x7f8f0406ccc0 msgr2=0x7f8f040b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:56.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.421+0000 7f8f09212640 1 -- 172.21.15.81:0/4294823455 shutdown_connections 2024-06-29T15:58:56.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.421+0000 7f8f09212640 1 -- 172.21.15.81:0/4294823455 wait complete. 2024-06-29T15:58:56.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.423+0000 7f8f09212640 1 Processor -- start 2024-06-29T15:58:56.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.423+0000 7f8f09212640 1 -- start start 2024-06-29T15:58:56.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8f09212640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f040737c0 0x7f8f04139590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8f09212640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8f04145530 0x7f8f04156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8f09212640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f0415cd00 0x7f8f041f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8f09212640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0415dbd0 con 0x7f8f040737c0 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8f09212640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0415da50 con 0x7f8f04145530 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8f09212640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8f0415dd50 con 0x7f8f0415cd00 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8f02d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f040737c0 0x7f8f04139590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8f02d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f040737c0 0x7f8f04139590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32988/0 (socket says 172.21.15.81:32988) 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8f02d9d640 1 -- 172.21.15.81:0/2473860270 learned_addr learned my addr 172.21.15.81:0/2473860270 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8efa59c640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8f04145530 0x7f8f04156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.424+0000 7f8f0359e640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f0415cd00 0x7f8f041f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8f02d9d640 1 -- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f0415cd00 msgr2=0x7f8f041f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8f02d9d640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f0415cd00 0x7f8f041f5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8f02d9d640 1 -- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8f04145530 msgr2=0x7f8f04156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8f02d9d640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8f04145530 0x7f8f04156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8f02d9d640 1 -- 172.21.15.81:0/2473860270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f041f6260 con 0x7f8f040737c0 2024-06-29T15:58:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8f02d9d640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f040737c0 0x7f8f04139590 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400b440 tx=0x7f8ef400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8efbfff640 1 -- 172.21.15.81:0/2473860270 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4015020 con 0x7f8f040737c0 2024-06-29T15:58:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8efbfff640 1 -- 172.21.15.81:0/2473860270 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ef4011980 con 0x7f8f040737c0 2024-06-29T15:58:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8f09212640 1 -- 172.21.15.81:0/2473860270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f040ad970 con 0x7f8f040737c0 2024-06-29T15:58:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8efbfff640 1 -- 172.21.15.81:0/2473860270 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4011dd0 con 0x7f8f040737c0 2024-06-29T15:58:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.425+0000 7f8f09212640 1 -- 172.21.15.81:0/2473860270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f040adef0 con 0x7f8f040737c0 2024-06-29T15:58:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.426+0000 7f8f09212640 1 -- 172.21.15.81:0/2473860270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f0415e300 con 0x7f8f040737c0 2024-06-29T15:58:56.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.427+0000 7f8efbfff640 1 -- 172.21.15.81:0/2473860270 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f8ef4018020 con 0x7f8f040737c0 2024-06-29T15:58:56.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.427+0000 7f8efbfff640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f8ee4076510 0x7f8ee4078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.427+0000 7f8efbfff640 1 -- 172.21.15.81:0/2473860270 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6795+0+0 (secure 0 0 0) 0x7f8ef40981b0 con 0x7f8f040737c0 2024-06-29T15:58:56.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.427+0000 7f8efa59c640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f8ee4076510 0x7f8ee4078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.428+0000 7f8efa59c640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f8ee4076510 0x7f8ee4078a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f0415ef40 tx=0x7f8eec009040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:56.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.432+0000 7f8efbfff640 1 -- 172.21.15.81:0/2473860270 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8ef401f020 con 0x7f8f040737c0 2024-06-29T15:58:56.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.646+0000 7f8f09212640 1 -- 172.21.15.81:0/2473860270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f0415bd70 con 0x7f8f040737c0 2024-06-29T15:58:56.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.647+0000 7f8efbfff640 1 -- 172.21.15.81:0/2473860270 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+5243 (secure 0 0 0) 0x7f8ef4067020 con 0x7f8f040737c0 2024-06-29T15:58:56.648 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:56.648 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":137,"btime":"2024-06-29T15:58:52:294249+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":11218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/113384617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":113384617},{"type":"v1","addr":"172.21.15.167:6837","nonce":113384617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:30.737083+0000","modified":"2024-06-29T15:58:33.773132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12093":{"gid":12093,"name":"a","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12093,"qdb_cluster":[12093]},"id":16}]} 2024-06-29T15:58:56.648 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 137 2024-06-29T15:58:56.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 -- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f8ee4076510 msgr2=0x7f8ee4078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f8ee4076510 0x7f8ee4078a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f0415ef40 tx=0x7f8eec009040 comp rx=0 tx=0).stop 2024-06-29T15:58:56.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 -- 172.21.15.81:0/2473860270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f040737c0 msgr2=0x7f8f04139590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f040737c0 0x7f8f04139590 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400b440 tx=0x7f8ef400b900 comp rx=0 tx=0).stop 2024-06-29T15:58:56.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 -- 172.21.15.81:0/2473860270 shutdown_connections 2024-06-29T15:58:56.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f8ee4076510 0x7f8ee4078a20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8f0415cd00 0x7f8f041f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8f04145530 0x7f8f04156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 --2- 172.21.15.81:0/2473860270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f040737c0 0x7f8f04139590 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 -- 172.21.15.81:0/2473860270 >> 172.21.15.81:0/2473860270 conn(0x7f8f0406ccc0 msgr2=0x7f8f041411f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:56.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 -- 172.21.15.81:0/2473860270 shutdown_connections 2024-06-29T15:58:56.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.649+0000 7f8f09212640 1 -- 172.21.15.81:0/2473860270 wait complete. 2024-06-29T15:58:56.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:58:56.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.774+0000 7f1c09d68640 1 -- 172.21.15.81:0/3106091067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c040b7290 msgr2=0x7f1c040b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.774+0000 7f1c09d68640 1 --2- 172.21.15.81:0/3106091067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c040b7290 0x7f1c040b76a0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f1c040b9b80 tx=0x7f1bfc02aac0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.774+0000 7f1c09d68640 1 -- 172.21.15.81:0/3106091067 shutdown_connections 2024-06-29T15:58:56.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.774+0000 7f1c09d68640 1 --2- 172.21.15.81:0/3106091067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c040b7290 0x7f1c040b76a0 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.774+0000 7f1c09d68640 1 --2- 172.21.15.81:0/3106091067 >> v2:172.21.15.167:3300/0 conn(0x7f1c040b5250 0x7f1c040d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.774+0000 7f1c09d68640 1 --2- 172.21.15.81:0/3106091067 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c040b9040 0x7f1c040d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.774+0000 7f1c09d68640 1 -- 172.21.15.81:0/3106091067 >> 172.21.15.81:0/3106091067 conn(0x7f1c0409cf30 msgr2=0x7f1c040d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:56.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.774+0000 7f1c09d68640 1 -- 172.21.15.81:0/3106091067 shutdown_connections 2024-06-29T15:58:56.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.774+0000 7f1c09d68640 1 -- 172.21.15.81:0/3106091067 wait complete. 2024-06-29T15:58:56.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.776+0000 7f1c09d68640 1 Processor -- start 2024-06-29T15:58:56.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.776+0000 7f1c09d68640 1 -- start start 2024-06-29T15:58:56.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c09d68640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c040b5250 0x7f1c040a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c09d68640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c040b7290 0x7f1c0415de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c09d68640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c040b9040 0x7f1c04151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c09d68640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1c040b8020 con 0x7f1c040b9040 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c09d68640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1c040b7ea0 con 0x7f1c040b5250 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c09d68640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1c040b81a0 con 0x7f1c040b7290 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c03fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c040b9040 0x7f1c04151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c03fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c040b9040 0x7f1c04151f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33002/0 (socket says 172.21.15.81:33002) 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c03fff640 1 -- 172.21.15.81:0/3456854515 learned_addr learned my addr 172.21.15.81:0/3456854515 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c037fe640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c040b5250 0x7f1c040a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c02ffd640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c040b7290 0x7f1c0415de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c03fff640 1 -- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c040b7290 msgr2=0x7f1c0415de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c03fff640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c040b7290 0x7f1c0415de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c03fff640 1 -- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c040b5250 msgr2=0x7f1c040a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c03fff640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c040b5250 0x7f1c040a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:56.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c03fff640 1 -- 172.21.15.81:0/3456854515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c04152cd0 con 0x7f1c040b9040 2024-06-29T15:58:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c037fe640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c040b5250 0x7f1c040a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c02ffd640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c040b7290 0x7f1c0415de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c03fff640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c040b9040 0x7f1c04151f20 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc02b0b0 tx=0x7f1bfc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1bebfff640 1 -- 172.21.15.81:0/3456854515 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc004250 con 0x7f1c040b9040 2024-06-29T15:58:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1bebfff640 1 -- 172.21.15.81:0/3456854515 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1bfc0043f0 con 0x7f1c040b9040 2024-06-29T15:58:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1bebfff640 1 -- 172.21.15.81:0/3456854515 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc004820 con 0x7f1c040b9040 2024-06-29T15:58:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.778+0000 7f1c09d68640 1 -- 172.21.15.81:0/3456854515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0409fdc0 con 0x7f1c040b9040 2024-06-29T15:58:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.779+0000 7f1c09d68640 1 -- 172.21.15.81:0/3456854515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c040a0380 con 0x7f1c040b9040 2024-06-29T15:58:56.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.779+0000 7f1c09d68640 1 -- 172.21.15.81:0/3456854515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c04152a40 con 0x7f1c040b9040 2024-06-29T15:58:56.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.780+0000 7f1bebfff640 1 -- 172.21.15.81:0/3456854515 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f1bfc040420 con 0x7f1c040b9040 2024-06-29T15:58:56.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.781+0000 7f1bebfff640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1bf0076510 0x7f1bf0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:56.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.781+0000 7f1bebfff640 1 -- 172.21.15.81:0/3456854515 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6795+0+0 (secure 0 0 0) 0x7f1bfc0b8910 con 0x7f1c040b9040 2024-06-29T15:58:56.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.781+0000 7f1c037fe640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1bf0076510 0x7f1bf0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:56.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.781+0000 7f1c037fe640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1bf0076510 0x7f1bf0078a20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8005fd0 tx=0x7f1bf8005cf0 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:56.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:56.785+0000 7f1bebfff640 1 -- 172.21.15.81:0/3456854515 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1bfc032030 con 0x7f1c040b9040 2024-06-29T15:58:57.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.002+0000 7f1c09d68640 1 -- 172.21.15.81:0/3456854515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c040b76a0 con 0x7f1c040b9040 2024-06-29T15:58:57.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.003+0000 7f1bebfff640 1 -- 172.21.15.81:0/3456854515 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+5243 (secure 0 0 0) 0x7f1bfc0b6e20 con 0x7f1c040b9040 2024-06-29T15:58:57.003 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:57.003 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":137,"btime":"2024-06-29T15:58:52:294249+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":11218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/113384617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":113384617},{"type":"v1","addr":"172.21.15.167:6837","nonce":113384617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:58:30.737083+0000","modified":"2024-06-29T15:58:33.773132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12093},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12093":{"gid":12093,"name":"a","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6835/3228546258","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":3228546258},{"type":"v1","addr":"172.21.15.81:6835","nonce":3228546258}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12093,"qdb_cluster":[12093]},"id":16}]} 2024-06-29T15:58:57.004 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 137 2024-06-29T15:58:57.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 -- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1bf0076510 msgr2=0x7f1bf0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1bf0076510 0x7f1bf0078a20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8005fd0 tx=0x7f1bf8005cf0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 -- 172.21.15.81:0/3456854515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c040b9040 msgr2=0x7f1c04151f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c040b9040 0x7f1c04151f20 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc02b0b0 tx=0x7f1bfc003f10 comp rx=0 tx=0).stop 2024-06-29T15:58:57.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 -- 172.21.15.81:0/3456854515 shutdown_connections 2024-06-29T15:58:57.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1bf0076510 0x7f1bf0078a20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c040b9040 0x7f1c04151f20 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c040b7290 0x7f1c0415de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 --2- 172.21.15.81:0/3456854515 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c040b5250 0x7f1c040a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 -- 172.21.15.81:0/3456854515 >> 172.21.15.81:0/3456854515 conn(0x7f1c0409cf30 msgr2=0x7f1c0406e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:57.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 -- 172.21.15.81:0/3456854515 shutdown_connections 2024-06-29T15:58:57.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.005+0000 7f1c09d68640 1 -- 172.21.15.81:0/3456854515 wait complete. 2024-06-29T15:58:57.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:58:57.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.130+0000 7f526e9b9640 1 -- 172.21.15.81:0/3911479892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52680b4810 msgr2=0x7f52680b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.130+0000 7f526e9b9640 1 --2- 172.21.15.81:0/3911479892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52680b4810 0x7f52680b4c20 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f5260005fd0 tx=0x7f526002aa10 comp rx=0 tx=0).stop 2024-06-29T15:58:57.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.130+0000 7f526e9b9640 1 -- 172.21.15.81:0/3911479892 shutdown_connections 2024-06-29T15:58:57.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.130+0000 7f526e9b9640 1 --2- 172.21.15.81:0/3911479892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52680b4810 0x7f52680b4c20 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.130+0000 7f526e9b9640 1 -- 172.21.15.81:0/3911479892 >> 172.21.15.81:0/3911479892 conn(0x7f52680b3510 msgr2=0x7f52680b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:57.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.130+0000 7f526e9b9640 1 -- 172.21.15.81:0/3911479892 shutdown_connections 2024-06-29T15:58:57.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.130+0000 7f526e9b9640 1 -- 172.21.15.81:0/3911479892 wait complete. 2024-06-29T15:58:57.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.132+0000 7f526e9b9640 1 Processor -- start 2024-06-29T15:58:57.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.132+0000 7f526e9b9640 1 -- start start 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.133+0000 7f526e9b9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52680b4810 0x7f52681660b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.133+0000 7f526e9b9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52680b6eb0 0x7f5268155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.133+0000 7f526e9b9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52680cd480 0x7f5268164c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.133+0000 7f526e9b9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f52680b5160 con 0x7f52680b6eb0 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.133+0000 7f526e9b9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f52680b29c0 con 0x7f52680cd480 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.133+0000 7f526e9b9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f52680b52e0 con 0x7f52680b4810 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.133+0000 7f52677fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52680b6eb0 0x7f5268155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.133+0000 7f52677fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52680b6eb0 0x7f5268155a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33008/0 (socket says 172.21.15.81:33008) 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.133+0000 7f52677fe640 1 -- 172.21.15.81:0/2119115188 learned_addr learned my addr 172.21.15.81:0/2119115188 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f5267fff640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52680b4810 0x7f52681660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f52677fe640 1 -- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52680b4810 msgr2=0x7f52681660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f526cf56640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52680cd480 0x7f5268164c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f52677fe640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52680b4810 0x7f52681660b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f52677fe640 1 -- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52680cd480 msgr2=0x7f5268164c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f52677fe640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52680cd480 0x7f5268164c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f52677fe640 1 -- 172.21.15.81:0/2119115188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52680cac00 con 0x7f52680b6eb0 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f52677fe640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52680b6eb0 0x7f5268155a80 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f525c00b3f0 tx=0x7f525c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f5264ff9640 1 -- 172.21.15.81:0/2119115188 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c015020 con 0x7f52680b6eb0 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f5264ff9640 1 -- 172.21.15.81:0/2119115188 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f525c011920 con 0x7f52680b6eb0 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f526e9b9640 1 -- 172.21.15.81:0/2119115188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52680caee0 con 0x7f52680b6eb0 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f5264ff9640 1 -- 172.21.15.81:0/2119115188 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c011d70 con 0x7f52680b6eb0 2024-06-29T15:58:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.134+0000 7f526e9b9640 1 -- 172.21.15.81:0/2119115188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5268164750 con 0x7f52680b6eb0 2024-06-29T15:58:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.135+0000 7f526e9b9640 1 -- 172.21.15.81:0/2119115188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52680b4c20 con 0x7f52680b6eb0 2024-06-29T15:58:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.136+0000 7f5264ff9640 1 -- 172.21.15.81:0/2119115188 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f525c018020 con 0x7f52680b6eb0 2024-06-29T15:58:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.136+0000 7f5264ff9640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5254076510 0x7f5254078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.137+0000 7f5264ff9640 1 -- 172.21.15.81:0/2119115188 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6795+0+0 (secure 0 0 0) 0x7f525c098020 con 0x7f52680b6eb0 2024-06-29T15:58:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.137+0000 7f5267fff640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5254076510 0x7f5254078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.137+0000 7f5267fff640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5254076510 0x7f5254078a20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f52580015a0 tx=0x7f5258009040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:57.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.141+0000 7f5264ff9640 1 -- 172.21.15.81:0/2119115188 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f525c01f020 con 0x7f52680b6eb0 2024-06-29T15:58:57.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.322+0000 7f526e9b9640 1 -- 172.21.15.81:0/2119115188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52680cb0b0 con 0x7f52680b6eb0 2024-06-29T15:58:57.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.323+0000 7f5264ff9640 1 -- 172.21.15.81:0/2119115188 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v242) ==== 74+0+18219 (secure 0 0 0) 0x7f525c0611e0 con 0x7f52680b6eb0 2024-06-29T15:58:57.324 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:58:57.324 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":242,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:58:54.300992+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2024-06-29T15:58:29.291185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"236","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2024-06-29T15:58:29.702735+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"237","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6837/30158598":"2024-06-30T15:58:51.291829+0000","172.21.15.167:6835/30158598":"2024-06-30T15:58:51.291829+0000","172.21.15.167:6833/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:6832/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/355438800":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4195196310":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4186712115":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/3601299355":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2623346726":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/972265206":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/1116386548":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6832/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/3057737710":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6833/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1418041738":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/561620344":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2207847535":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2067929640":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/874327121":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/273305307":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2269612087":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1188031263":"2024-06-30T15:58:42.086693+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:58:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 -- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5254076510 msgr2=0x7f5254078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5254076510 0x7f5254078a20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f52580015a0 tx=0x7f5258009040 comp rx=0 tx=0).stop 2024-06-29T15:58:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 -- 172.21.15.81:0/2119115188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52680b6eb0 msgr2=0x7f5268155a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52680b6eb0 0x7f5268155a80 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f525c00b3f0 tx=0x7f525c00b8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 -- 172.21.15.81:0/2119115188 shutdown_connections 2024-06-29T15:58:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5254076510 0x7f5254078a20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f52680cd480 0x7f5268164c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52680b6eb0 0x7f5268155a80 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 --2- 172.21.15.81:0/2119115188 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f52680b4810 0x7f52681660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 -- 172.21.15.81:0/2119115188 >> 172.21.15.81:0/2119115188 conn(0x7f52680b3510 msgr2=0x7f5268068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:57.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 -- 172.21.15.81:0/2119115188 shutdown_connections 2024-06-29T15:58:57.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.325+0000 7f526e9b9640 1 -- 172.21.15.81:0/2119115188 wait complete. 2024-06-29T15:58:57.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2024-06-29T15:58:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.452+0000 7f596a94a640 1 -- 172.21.15.81:0/682834074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59640aea70 msgr2=0x7f59640aee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.452+0000 7f596a94a640 1 --2- 172.21.15.81:0/682834074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59640aea70 0x7f59640aee80 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f5964072e90 tx=0x7f595802a4d0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.452+0000 7f596a94a640 1 -- 172.21.15.81:0/682834074 shutdown_connections 2024-06-29T15:58:57.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.452+0000 7f596a94a640 1 --2- 172.21.15.81:0/682834074 >> v2:172.21.15.167:3300/0 conn(0x7f59640af3c0 0x7f596406c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.452+0000 7f596a94a640 1 --2- 172.21.15.81:0/682834074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59640aea70 0x7f59640aee80 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.452+0000 7f596a94a640 1 -- 172.21.15.81:0/682834074 >> 172.21.15.81:0/682834074 conn(0x7f5964072950 msgr2=0x7f59640b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:57.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.452+0000 7f596a94a640 1 -- 172.21.15.81:0/682834074 shutdown_connections 2024-06-29T15:58:57.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.452+0000 7f596a94a640 1 -- 172.21.15.81:0/682834074 wait complete. 2024-06-29T15:58:57.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.454+0000 7f596a94a640 1 Processor -- start 2024-06-29T15:58:57.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.454+0000 7f596a94a640 1 -- start start 2024-06-29T15:58:57.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f596a94a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59640ad0c0 0x7f596415a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f596a94a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59640aea70 0x7f5964158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f596a94a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59640af3c0 0x7f5964140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f596a94a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f596406cf80 con 0x7f59640ad0c0 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f596a94a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f596406ce00 con 0x7f59640af3c0 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f596a94a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f596406d100 con 0x7f59640aea70 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f5963fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59640ad0c0 0x7f596415a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f5963fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59640ad0c0 0x7f596415a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33022/0 (socket says 172.21.15.81:33022) 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f5963fff640 1 -- 172.21.15.81:0/1958596680 learned_addr learned my addr 172.21.15.81:0/1958596680 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f5968ee7640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59640af3c0 0x7f5964140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.455+0000 7f59637fe640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59640aea70 0x7f5964158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f5963fff640 1 -- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59640aea70 msgr2=0x7f5964158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f5963fff640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59640aea70 0x7f5964158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f5963fff640 1 -- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59640af3c0 msgr2=0x7f5964140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f5963fff640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59640af3c0 0x7f5964140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f5963fff640 1 -- 172.21.15.81:0/1958596680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f596415aa00 con 0x7f59640ad0c0 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f59637fe640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59640aea70 0x7f5964158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f5968ee7640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59640af3c0 0x7f5964140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f5963fff640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59640ad0c0 0x7f596415a1b0 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f595400b3e0 tx=0x7f595400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f5960ff9640 1 -- 172.21.15.81:0/1958596680 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954015020 con 0x7f59640ad0c0 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f5960ff9640 1 -- 172.21.15.81:0/1958596680 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5954011940 con 0x7f59640ad0c0 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f596a94a640 1 -- 172.21.15.81:0/1958596680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f596415ace0 con 0x7f59640ad0c0 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f5960ff9640 1 -- 172.21.15.81:0/1958596680 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954011d90 con 0x7f59640ad0c0 2024-06-29T15:58:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.456+0000 7f596a94a640 1 -- 172.21.15.81:0/1958596680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f596415b2a0 con 0x7f59640ad0c0 2024-06-29T15:58:57.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.457+0000 7f596a94a640 1 -- 172.21.15.81:0/1958596680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f596406d630 con 0x7f59640ad0c0 2024-06-29T15:58:57.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.458+0000 7f5960ff9640 1 -- 172.21.15.81:0/1958596680 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f5954018020 con 0x7f59640ad0c0 2024-06-29T15:58:57.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.458+0000 7f5960ff9640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5950076510 0x7f5950078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.458+0000 7f5960ff9640 1 -- 172.21.15.81:0/1958596680 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 6795+0+0 (secure 0 0 0) 0x7f5954098360 con 0x7f59640ad0c0 2024-06-29T15:58:57.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.459+0000 7f59637fe640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5950076510 0x7f5950078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.459+0000 7f59637fe640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5950076510 0x7f5950078a20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5958033000 tx=0x7f5958004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:57.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.462+0000 7f5960ff9640 1 -- 172.21.15.81:0/1958596680 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f595401f020 con 0x7f59640ad0c0 2024-06-29T15:58:57.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.681+0000 7f596a94a640 1 -- 172.21.15.81:0/1958596680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) -- 0x7f5964075bd0 con 0x7f59640ad0c0 2024-06-29T15:58:57.682 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T15:58:57.682+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:58:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.685+0000 7f5960ff9640 1 -- 172.21.15.81:0/1958596680 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v138) ==== 167+0+0 (secure 0 0 0) 0x7f5954061600 con 0x7f59640ad0c0 2024-06-29T15:58:57.685 INFO:teuthology.orchestra.run.smithi081.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:58:57.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.687+0000 7f596a94a640 1 -- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5950076510 msgr2=0x7f5950078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.687+0000 7f596a94a640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5950076510 0x7f5950078a20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5958033000 tx=0x7f5958004730 comp rx=0 tx=0).stop 2024-06-29T15:58:57.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.687+0000 7f596a94a640 1 -- 172.21.15.81:0/1958596680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59640ad0c0 msgr2=0x7f596415a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.687+0000 7f596a94a640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59640ad0c0 0x7f596415a1b0 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f595400b3e0 tx=0x7f595400b8a0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.687+0000 7f596a94a640 1 -- 172.21.15.81:0/1958596680 shutdown_connections 2024-06-29T15:58:57.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.687+0000 7f596a94a640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5950076510 0x7f5950078a20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.688+0000 7f596a94a640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f59640af3c0 0x7f5964140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.688+0000 7f596a94a640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f59640aea70 0x7f5964158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.688+0000 7f596a94a640 1 --2- 172.21.15.81:0/1958596680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59640ad0c0 0x7f596415a1b0 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.688+0000 7f596a94a640 1 -- 172.21.15.81:0/1958596680 >> 172.21.15.81:0/1958596680 conn(0x7f5964072950 msgr2=0x7f596415dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:57.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.688+0000 7f596a94a640 1 -- 172.21.15.81:0/1958596680 shutdown_connections 2024-06-29T15:58:57.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.688+0000 7f596a94a640 1 -- 172.21.15.81:0/1958596680 wait complete. 2024-06-29T15:58:57.689 INFO:tasks.ceph.mds.a.smithi081.stderr: -1632> 2024-06-29T15:58:36.259+0000 7f2fbd9a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:58:57.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2024-06-29T15:58:57.742 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T15:58:57.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.819+0000 7f4d81e90640 1 -- 172.21.15.81:0/129445958 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d7c071f20 msgr2=0x7f4d7c155480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.819+0000 7f4d81e90640 1 --2- 172.21.15.81:0/129445958 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d7c071f20 0x7f4d7c155480 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c005fd0 tx=0x7f4d6c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.819+0000 7f4d81e90640 1 -- 172.21.15.81:0/129445958 shutdown_connections 2024-06-29T15:58:57.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.819+0000 7f4d81e90640 1 --2- 172.21.15.81:0/129445958 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d7c15bf20 0x7f4d7c06f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.819+0000 7f4d81e90640 1 --2- 172.21.15.81:0/129445958 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d7c071f20 0x7f4d7c155480 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.819+0000 7f4d81e90640 1 -- 172.21.15.81:0/129445958 >> 172.21.15.81:0/129445958 conn(0x7f4d7c0b1730 msgr2=0x7f4d7c0b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:57.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.819+0000 7f4d81e90640 1 -- 172.21.15.81:0/129445958 shutdown_connections 2024-06-29T15:58:57.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.819+0000 7f4d81e90640 1 -- 172.21.15.81:0/129445958 wait complete. 2024-06-29T15:58:57.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.821+0000 7f4d81e90640 1 Processor -- start 2024-06-29T15:58:57.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.821+0000 7f4d81e90640 1 -- start start 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d81e90640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d7c06fca0 0x7f4d7c0aac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d81e90640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d7c071f20 0x7f4d7c0a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d81e90640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d7c15bf20 0x7f4d7c0a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d81e90640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c15f1a0 con 0x7f4d7c06fca0 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d81e90640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c0b40a0 con 0x7f4d7c071f20 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d81e90640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7c15f320 con 0x7f4d7c15bf20 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d7c06fca0 0x7f4d7c0aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d7c06fca0 0x7f4d7c0aac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33042/0 (socket says 172.21.15.81:33042) 2024-06-29T15:58:57.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7b7fe640 1 -- 172.21.15.81:0/1662142682 learned_addr learned my addr 172.21.15.81:0/1662142682 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7affd640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d7c071f20 0x7f4d7c0a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7bfff640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d7c15bf20 0x7f4d7c0a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7b7fe640 1 -- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d7c15bf20 msgr2=0x7f4d7c0a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7b7fe640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d7c15bf20 0x7f4d7c0a9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7b7fe640 1 -- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d7c071f20 msgr2=0x7f4d7c0a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7b7fe640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d7c071f20 0x7f4d7c0a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7b7fe640 1 -- 172.21.15.81:0/1662142682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c156be0 con 0x7f4d7c06fca0 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d7b7fe640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d7c06fca0 0x7f4d7c0aac00 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009cd0 tx=0x7f4d6c0316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d63fff640 1 -- 172.21.15.81:0/1662142682 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c031990 con 0x7f4d7c06fca0 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d81e90640 1 -- 172.21.15.81:0/1662142682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7c156ec0 con 0x7f4d7c06fca0 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d63fff640 1 -- 172.21.15.81:0/1662142682 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d6c031c20 con 0x7f4d7c06fca0 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.823+0000 7f4d63fff640 1 -- 172.21.15.81:0/1662142682 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c0330d0 con 0x7f4d7c06fca0 2024-06-29T15:58:57.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.824+0000 7f4d81e90640 1 -- 172.21.15.81:0/1662142682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d7c0ae200 con 0x7f4d7c06fca0 2024-06-29T15:58:57.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.825+0000 7f4d81e90640 1 -- 172.21.15.81:0/1662142682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d7c15f7f0 con 0x7f4d7c06fca0 2024-06-29T15:58:57.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.825+0000 7f4d63fff640 1 -- 172.21.15.81:0/1662142682 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f4d6c033270 con 0x7f4d7c06fca0 2024-06-29T15:58:57.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.826+0000 7f4d63fff640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4d68076540 0x7f4d68078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:57.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.826+0000 7f4d63fff640 1 -- 172.21.15.81:0/1662142682 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(243..243 src has 1..243) ==== 6881+0+0 (secure 0 0 0) 0x7f4d6c0b93a0 con 0x7f4d7c06fca0 2024-06-29T15:58:57.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.826+0000 7f4d7affd640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4d68076540 0x7f4d68078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:57.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.827+0000 7f4d7affd640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4d68076540 0x7f4d68078a50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0aa680 tx=0x7f4d70005f90 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:57.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:57.831+0000 7f4d63fff640 1 -- 172.21.15.81:0/1662142682 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4d6c03f030 con 0x7f4d7c06fca0 2024-06-29T15:58:58.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.053+0000 7f4d81e90640 1 -- 172.21.15.81:0/1662142682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) -- 0x7f4d7c0ae430 con 0x7f4d7c06fca0 2024-06-29T15:58:58.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.691+0000 7f4d63fff640 1 -- 172.21.15.81:0/1662142682 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v139) ==== 107+0+0 (secure 0 0 0) 0x7f4d6c0a1e20 con 0x7f4d7c06fca0 2024-06-29T15:58:58.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 -- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4d68076540 msgr2=0x7f4d68078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4d68076540 0x7f4d68078a50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0aa680 tx=0x7f4d70005f90 comp rx=0 tx=0).stop 2024-06-29T15:58:58.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 -- 172.21.15.81:0/1662142682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d7c06fca0 msgr2=0x7f4d7c0aac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d7c06fca0 0x7f4d7c0aac00 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009cd0 tx=0x7f4d6c0316f0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 -- 172.21.15.81:0/1662142682 shutdown_connections 2024-06-29T15:58:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4d68076540 0x7f4d68078a50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d7c15bf20 0x7f4d7c0a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d7c071f20 0x7f4d7c0a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 --2- 172.21.15.81:0/1662142682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d7c06fca0 0x7f4d7c0aac00 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 -- 172.21.15.81:0/1662142682 >> 172.21.15.81:0/1662142682 conn(0x7f4d7c0b1730 msgr2=0x7f4d7c0cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 -- 172.21.15.81:0/1662142682 shutdown_connections 2024-06-29T15:58:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.693+0000 7f4d81e90640 1 -- 172.21.15.81:0/1662142682 wait complete. 2024-06-29T15:58:58.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2024-06-29T15:58:58.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.865+0000 7f4c9c21f640 1 -- 172.21.15.81:0/3890132411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c7c004d40 msgr2=0x7f4c7c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.865+0000 7f4c9c21f640 1 --2- 172.21.15.81:0/3890132411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c7c004d40 0x7f4c7c005210 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f4c940ce1b0 tx=0x7f4c7c005830 comp rx=0 tx=0).stop 2024-06-29T15:58:58.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.865+0000 7f4c9a7bc640 1 -- 172.21.15.81:0/3890132411 reap_dead start 2024-06-29T15:58:58.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.865+0000 7f4c9c21f640 1 -- 172.21.15.81:0/3890132411 shutdown_connections 2024-06-29T15:58:58.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.865+0000 7f4c9c21f640 1 -- 172.21.15.81:0/3890132411 >> 172.21.15.81:0/3890132411 conn(0x7f4c940b3680 msgr2=0x7f4c940b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:58.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.865+0000 7f4c9c21f640 1 -- 172.21.15.81:0/3890132411 shutdown_connections 2024-06-29T15:58:58.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.865+0000 7f4c9c21f640 1 -- 172.21.15.81:0/3890132411 wait complete. 2024-06-29T15:58:58.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.867+0000 7f4c9c21f640 1 Processor -- start 2024-06-29T15:58:58.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.867+0000 7f4c9c21f640 1 -- start start 2024-06-29T15:58:58.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c9c21f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c94156b40 0x7f4c941f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c9c21f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c941578b0 0x7f4c94151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c9c21f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c94154e40 0x7f4c94155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c9c21f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4c940b6f50 con 0x7f4c94156b40 2024-06-29T15:58:58.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c9c21f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4c940b6dd0 con 0x7f4c94154e40 2024-06-29T15:58:58.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c9c21f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4c940ccd80 con 0x7f4c941578b0 2024-06-29T15:58:58.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c99fbb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c94156b40 0x7f4c941f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c99fbb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c94156b40 0x7f4c941f3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33062/0 (socket says 172.21.15.81:33062) 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c99fbb640 1 -- 172.21.15.81:0/1103534629 learned_addr learned my addr 172.21.15.81:0/1103534629 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c9a7bc640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c94154e40 0x7f4c94155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c997ba640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c941578b0 0x7f4c94151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c99fbb640 1 -- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c941578b0 msgr2=0x7f4c94151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c99fbb640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c941578b0 0x7f4c94151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c99fbb640 1 -- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c94154e40 msgr2=0x7f4c94155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c99fbb640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c94154e40 0x7f4c94155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c99fbb640 1 -- 172.21.15.81:0/1103534629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c94155790 con 0x7f4c94156b40 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c997ba640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c941578b0 0x7f4c94151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c9a7bc640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c94154e40 0x7f4c94155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.868+0000 7f4c99fbb640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c94156b40 0x7f4c941f3650 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f4c8400b5b0 tx=0x7f4c8400ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.869+0000 7f4c927fc640 1 -- 172.21.15.81:0/1103534629 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c84010a20 con 0x7f4c94156b40 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.869+0000 7f4c927fc640 1 -- 172.21.15.81:0/1103534629 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c84015070 con 0x7f4c94156b40 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.869+0000 7f4c9c21f640 1 -- 172.21.15.81:0/1103534629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c94156f50 con 0x7f4c94156b40 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.869+0000 7f4c927fc640 1 -- 172.21.15.81:0/1103534629 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c84010e40 con 0x7f4c94156b40 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.869+0000 7f4c9c21f640 1 -- 172.21.15.81:0/1103534629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c94152cf0 con 0x7f4c94156b40 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.870+0000 7f4c9c21f640 1 -- 172.21.15.81:0/1103534629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c940cdaa0 con 0x7f4c94156b40 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.870+0000 7f4c927fc640 1 -- 172.21.15.81:0/1103534629 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f4c8401a470 con 0x7f4c94156b40 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.871+0000 7f4c927fc640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4c80076510 0x7f4c80078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.871+0000 7f4c927fc640 1 -- 172.21.15.81:0/1103534629 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(243..243 src has 1..243) ==== 6881+0+0 (secure 0 0 0) 0x7f4c84098c00 con 0x7f4c94156b40 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.871+0000 7f4c997ba640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4c80076510 0x7f4c80078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.871+0000 7f4c997ba640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4c80076510 0x7f4c80078a20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4c88000fb0 tx=0x7f4c880067a0 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:58.875+0000 7f4c927fc640 1 -- 172.21.15.81:0/1103534629 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4c8409f080 con 0x7f4c94156b40 2024-06-29T15:58:59.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.056+0000 7f4c9c21f640 1 -- 172.21.15.81:0/1103534629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4c940c53a0 con 0x7f4c94156b40 2024-06-29T15:58:59.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.695+0000 7f4c927fc640 1 -- 172.21.15.81:0/1103534629 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v244) ==== 190+0+0 (secure 0 0 0) 0x7f4c94155790 con 0x7f4c94156b40 2024-06-29T15:58:59.696 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'backup_fs_metadata' removed 2024-06-29T15:58:59.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.699+0000 7f4c9c21f640 1 -- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4c80076510 msgr2=0x7f4c80078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.699+0000 7f4c9c21f640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4c80076510 0x7f4c80078a20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4c88000fb0 tx=0x7f4c880067a0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.699+0000 7f4c9c21f640 1 -- 172.21.15.81:0/1103534629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c94156b40 msgr2=0x7f4c941f3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.699+0000 7f4c9c21f640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c94156b40 0x7f4c941f3650 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f4c8400b5b0 tx=0x7f4c8400ba70 comp rx=0 tx=0).stop 2024-06-29T15:58:59.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.699+0000 7f4c9c21f640 1 -- 172.21.15.81:0/1103534629 shutdown_connections 2024-06-29T15:58:59.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.699+0000 7f4c9c21f640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4c80076510 0x7f4c80078a20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.699+0000 7f4c9c21f640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4c94154e40 0x7f4c94155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.699+0000 7f4c9c21f640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4c941578b0 0x7f4c94151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.699+0000 7f4c9c21f640 1 --2- 172.21.15.81:0/1103534629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c94156b40 0x7f4c941f3650 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.699+0000 7f4c9c21f640 1 -- 172.21.15.81:0/1103534629 >> 172.21.15.81:0/1103534629 conn(0x7f4c940b3680 msgr2=0x7f4c940b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:59.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.700+0000 7f4c9c21f640 1 -- 172.21.15.81:0/1103534629 shutdown_connections 2024-06-29T15:58:59.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.700+0000 7f4c9c21f640 1 -- 172.21.15.81:0/1103534629 wait complete. 2024-06-29T15:58:59.713 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2024-06-29T15:58:59.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.836+0000 7f71e475b640 1 -- 172.21.15.81:0/140972512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71dc0ae8f0 msgr2=0x7f71dc15c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.836+0000 7f71e475b640 1 --2- 172.21.15.81:0/140972512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71dc0ae8f0 0x7f71dc15c0e0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0d0e60 tx=0x7f71d802a560 comp rx=0 tx=0).stop 2024-06-29T15:58:59.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.836+0000 7f71e475b640 1 -- 172.21.15.81:0/140972512 shutdown_connections 2024-06-29T15:58:59.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.836+0000 7f71e475b640 1 --2- 172.21.15.81:0/140972512 >> v2:172.21.15.167:3300/0 conn(0x7f71dc0b2cb0 0x7f71dc0d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.836+0000 7f71e475b640 1 --2- 172.21.15.81:0/140972512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71dc0ae8f0 0x7f71dc15c0e0 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.836+0000 7f71e475b640 1 -- 172.21.15.81:0/140972512 >> 172.21.15.81:0/140972512 conn(0x7f71dc09aa60 msgr2=0x7f71dc09ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:58:59.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.836+0000 7f71e475b640 1 -- 172.21.15.81:0/140972512 shutdown_connections 2024-06-29T15:58:59.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.836+0000 7f71e475b640 1 -- 172.21.15.81:0/140972512 wait complete. 2024-06-29T15:58:59.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.837+0000 7f71e475b640 1 Processor -- start 2024-06-29T15:58:59.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.837+0000 7f71e475b640 1 -- start start 2024-06-29T15:58:59.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e475b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71dc0ae8f0 0x7f71dc159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e475b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71dc0b1a90 0x7f71dc152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e475b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71dc0b2cb0 0x7f71dc153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e475b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f71dc15ffd0 con 0x7f71dc0b1a90 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e475b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f71dc15fe50 con 0x7f71dc0b2cb0 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e475b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f71dc160150 con 0x7f71dc0ae8f0 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e1cf6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71dc0b1a90 0x7f71dc152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e1cf6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71dc0b1a90 0x7f71dc152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33076/0 (socket says 172.21.15.81:33076) 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e1cf6640 1 -- 172.21.15.81:0/3376652997 learned_addr learned my addr 172.21.15.81:0/3376652997 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e24f7640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71dc0ae8f0 0x7f71dc159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e2cf8640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71dc0b2cb0 0x7f71dc153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e1cf6640 1 -- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71dc0ae8f0 msgr2=0x7f71dc159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e1cf6640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71dc0ae8f0 0x7f71dc159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e1cf6640 1 -- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71dc0b2cb0 msgr2=0x7f71dc153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:58:59.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e1cf6640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71dc0b2cb0 0x7f71dc153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:58:59.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e1cf6640 1 -- 172.21.15.81:0/3376652997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71dc0b4cc0 con 0x7f71dc0b1a90 2024-06-29T15:58:59.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71e1cf6640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71dc0b1a90 0x7f71dc152df0 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f71cc00b410 tx=0x7f71cc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:59.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71d2ffd640 1 -- 172.21.15.81:0/3376652997 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71cc015020 con 0x7f71dc0b1a90 2024-06-29T15:58:59.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.839+0000 7f71d2ffd640 1 -- 172.21.15.81:0/3376652997 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71cc011940 con 0x7f71dc0b1a90 2024-06-29T15:58:59.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.840+0000 7f71d2ffd640 1 -- 172.21.15.81:0/3376652997 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71cc011d90 con 0x7f71dc0b1a90 2024-06-29T15:58:59.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.840+0000 7f71e475b640 1 -- 172.21.15.81:0/3376652997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71dc0b4fa0 con 0x7f71dc0b1a90 2024-06-29T15:58:59.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.840+0000 7f71e475b640 1 -- 172.21.15.81:0/3376652997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71dc0b5560 con 0x7f71dc0b1a90 2024-06-29T15:58:59.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.840+0000 7f71e475b640 1 -- 172.21.15.81:0/3376652997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71dc1606f0 con 0x7f71dc0b1a90 2024-06-29T15:58:59.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.841+0000 7f71d2ffd640 1 -- 172.21.15.81:0/3376652997 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f71cc018020 con 0x7f71dc0b1a90 2024-06-29T15:58:59.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.842+0000 7f71d2ffd640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f71c0076510 0x7f71c0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:58:59.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.842+0000 7f71d2ffd640 1 -- 172.21.15.81:0/3376652997 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(244..244 src has 1..244) ==== 6437+0+0 (secure 0 0 0) 0x7f71cc097dc0 con 0x7f71dc0b1a90 2024-06-29T15:58:59.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.842+0000 7f71e24f7640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f71c0076510 0x7f71c0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:58:59.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.843+0000 7f71e24f7640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f71c0076510 0x7f71c0078a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71d8033000 tx=0x7f71d8004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:58:59.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:58:59.846+0000 7f71d2ffd640 1 -- 172.21.15.81:0/3376652997 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f71cc01f020 con 0x7f71dc0b1a90 2024-06-29T15:59:00.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.027+0000 7f71e475b640 1 -- 172.21.15.81:0/3376652997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f71dc15bb50 con 0x7f71dc0b1a90 2024-06-29T15:59:00.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.703+0000 7f71d2ffd640 1 -- 172.21.15.81:0/3376652997 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v245) ==== 178+0+0 (secure 0 0 0) 0x7f71cc060a10 con 0x7f71dc0b1a90 2024-06-29T15:59:00.704 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'backup_fs_data' removed 2024-06-29T15:59:00.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 -- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f71c0076510 msgr2=0x7f71c0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f71c0076510 0x7f71c0078a20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71d8033000 tx=0x7f71d8004730 comp rx=0 tx=0).stop 2024-06-29T15:59:00.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 -- 172.21.15.81:0/3376652997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71dc0b1a90 msgr2=0x7f71dc152df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71dc0b1a90 0x7f71dc152df0 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f71cc00b410 tx=0x7f71cc00b8d0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 -- 172.21.15.81:0/3376652997 shutdown_connections 2024-06-29T15:59:00.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f71c0076510 0x7f71c0078a20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f71dc0b2cb0 0x7f71dc153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71dc0b1a90 0x7f71dc152df0 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 --2- 172.21.15.81:0/3376652997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f71dc0ae8f0 0x7f71dc159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 -- 172.21.15.81:0/3376652997 >> 172.21.15.81:0/3376652997 conn(0x7f71dc09aa60 msgr2=0x7f71dc0afc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:00.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 -- 172.21.15.81:0/3376652997 shutdown_connections 2024-06-29T15:59:00.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.706+0000 7f71e475b640 1 -- 172.21.15.81:0/3376652997 wait complete. 2024-06-29T15:59:00.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.839+0000 7fe97db8c640 1 -- 172.21.15.81:0/873742096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9780b35b0 msgr2=0x7fe97815b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.839+0000 7fe97db8c640 1 --2- 172.21.15.81:0/873742096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9780b35b0 0x7fe97815b5b0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fe97815aac0 tx=0x7fe96802b7f0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.839+0000 7fe97db8c640 1 -- 172.21.15.81:0/873742096 shutdown_connections 2024-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.839+0000 7fe97db8c640 1 --2- 172.21.15.81:0/873742096 >> v2:172.21.15.167:3300/0 conn(0x7fe97806f660 0x7fe97806fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.839+0000 7fe97db8c640 1 --2- 172.21.15.81:0/873742096 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9780b4430 0x7fe9780cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.839+0000 7fe97db8c640 1 --2- 172.21.15.81:0/873742096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9780b35b0 0x7fe97815b5b0 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.839+0000 7fe97db8c640 1 -- 172.21.15.81:0/873742096 >> 172.21.15.81:0/873742096 conn(0x7fe97806ccc0 msgr2=0x7fe9780b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.839+0000 7fe97db8c640 1 -- 172.21.15.81:0/873742096 shutdown_connections 2024-06-29T15:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.839+0000 7fe97db8c640 1 -- 172.21.15.81:0/873742096 wait complete. 2024-06-29T15:59:00.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.841+0000 7fe97db8c640 1 Processor -- start 2024-06-29T15:59:00.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.841+0000 7fe97db8c640 1 -- start start 2024-06-29T15:59:00.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe97db8c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97806f660 0x7fe9781f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe97db8c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9780b35b0 0x7fe9781f0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe97db8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9780b4430 0x7fe9780b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe97db8c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe97806f160 con 0x7fe9780b4430 2024-06-29T15:59:00.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe97db8c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe97806efe0 con 0x7fe97806f660 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe97db8c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe97806f2e0 con 0x7fe9780b35b0 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe977fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9780b4430 0x7fe9780b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe977fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9780b4430 0x7fe9780b0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57410/0 (socket says 172.21.15.81:57410) 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe977fff640 1 -- 172.21.15.81:0/2545663719 learned_addr learned my addr 172.21.15.81:0/2545663719 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe9777fe640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97806f660 0x7fe9781f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe977fff640 1 -- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9780b35b0 msgr2=0x7fe9781f0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe976ffd640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9780b35b0 0x7fe9781f0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe977fff640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9780b35b0 0x7fe9781f0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe977fff640 1 -- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97806f660 msgr2=0x7fe9781f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe977fff640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97806f660 0x7fe9781f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe977fff640 1 -- 172.21.15.81:0/2545663719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9781f3c90 con 0x7fe9780b4430 2024-06-29T15:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe9777fe640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97806f660 0x7fe9781f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe977fff640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9780b4430 0x7fe9780b0640 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7fe97000d980 tx=0x7fe97000de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe95ffff640 1 -- 172.21.15.81:0/2545663719 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe970016070 con 0x7fe9780b4430 2024-06-29T15:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe97db8c640 1 -- 172.21.15.81:0/2545663719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe978159fc0 con 0x7fe9780b4430 2024-06-29T15:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe95ffff640 1 -- 172.21.15.81:0/2545663719 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe97000bd40 con 0x7fe9780b4430 2024-06-29T15:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe95ffff640 1 -- 172.21.15.81:0/2545663719 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe970003bd0 con 0x7fe9780b4430 2024-06-29T15:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.843+0000 7fe97db8c640 1 -- 172.21.15.81:0/2545663719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe978157c70 con 0x7fe9780b4430 2024-06-29T15:59:00.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.844+0000 7fe97db8c640 1 -- 172.21.15.81:0/2545663719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe97815a260 con 0x7fe9780b4430 2024-06-29T15:59:00.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.845+0000 7fe95ffff640 1 -- 172.21.15.81:0/2545663719 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fe970010460 con 0x7fe9780b4430 2024-06-29T15:59:00.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.846+0000 7fe95ffff640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe964076540 0x7fe964078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:00.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.846+0000 7fe95ffff640 1 -- 172.21.15.81:0/2545663719 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 6017+0+0 (secure 0 0 0) 0x7fe970098490 con 0x7fe9780b4430 2024-06-29T15:59:00.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.846+0000 7fe9777fe640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe964076540 0x7fe964078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:00.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.846+0000 7fe9777fe640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe964076540 0x7fe964078a50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe968035000 tx=0x7fe968004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:00.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:00.850+0000 7fe95ffff640 1 -- 172.21.15.81:0/2545663719 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe9781f3c90 con 0x7fe9780b4430 2024-06-29T15:59:01.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.030+0000 7fe97db8c640 1 -- 172.21.15.81:0/2545663719 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fe97815b5b0 con 0x7fe9780b4430 2024-06-29T15:59:01.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.035+0000 7fe95ffff640 1 -- 172.21.15.81:0/2545663719 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v79) ==== 71+0+0 (secure 0 0 0) 0x7fe970061730 con 0x7fe9780b4430 2024-06-29T15:59:01.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.037+0000 7fe97db8c640 1 -- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe964076540 msgr2=0x7fe964078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe964076540 0x7fe964078a50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe968035000 tx=0x7fe968004730 comp rx=0 tx=0).stop 2024-06-29T15:59:01.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 -- 172.21.15.81:0/2545663719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9780b4430 msgr2=0x7fe9780b0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9780b4430 0x7fe9780b0640 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7fe97000d980 tx=0x7fe97000de40 comp rx=0 tx=0).stop 2024-06-29T15:59:01.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 -- 172.21.15.81:0/2545663719 shutdown_connections 2024-06-29T15:59:01.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe964076540 0x7fe964078a50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9780b4430 0x7fe9780b0640 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9780b35b0 0x7fe9781f0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 --2- 172.21.15.81:0/2545663719 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97806f660 0x7fe9781f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 -- 172.21.15.81:0/2545663719 >> 172.21.15.81:0/2545663719 conn(0x7fe97806ccc0 msgr2=0x7fe9780afda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:01.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 -- 172.21.15.81:0/2545663719 shutdown_connections 2024-06-29T15:59:01.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.038+0000 7fe97db8c640 1 -- 172.21.15.81:0/2545663719 wait complete. 2024-06-29T15:59:01.049 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2024-06-29T15:59:01.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.163+0000 7f06c69e0640 1 -- 172.21.15.81:0/2302878453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c01412f0 msgr2=0x7f06c00cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.163+0000 7f06c69e0640 1 --2- 172.21.15.81:0/2302878453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c01412f0 0x7f06c00cef90 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f06c006ff00 tx=0x7f06b002a560 comp rx=0 tx=0).stop 2024-06-29T15:59:01.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.163+0000 7f06c69e0640 1 -- 172.21.15.81:0/2302878453 shutdown_connections 2024-06-29T15:59:01.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.163+0000 7f06c69e0640 1 --2- 172.21.15.81:0/2302878453 >> v2:172.21.15.167:3300/0 conn(0x7f06c0073570 0x7f06c015e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.163+0000 7f06c69e0640 1 --2- 172.21.15.81:0/2302878453 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c00cf4d0 0x7f06c0073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.163+0000 7f06c69e0640 1 --2- 172.21.15.81:0/2302878453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c01412f0 0x7f06c00cef90 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.163+0000 7f06c69e0640 1 -- 172.21.15.81:0/2302878453 >> 172.21.15.81:0/2302878453 conn(0x7f06c006ccc0 msgr2=0x7f06c00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:01.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.164+0000 7f06c69e0640 1 -- 172.21.15.81:0/2302878453 shutdown_connections 2024-06-29T15:59:01.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.164+0000 7f06c69e0640 1 -- 172.21.15.81:0/2302878453 wait complete. 2024-06-29T15:59:01.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.165+0000 7f06c69e0640 1 Processor -- start 2024-06-29T15:59:01.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.165+0000 7f06c69e0640 1 -- start start 2024-06-29T15:59:01.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c69e0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06c0073570 0x7f06c01f5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c69e0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c00cf4d0 0x7f06c013b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c69e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c01412f0 0x7f06c00b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c69e0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06c015f230 con 0x7f06c01412f0 2024-06-29T15:59:01.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c69e0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f06c015f0b0 con 0x7f06c0073570 2024-06-29T15:59:01.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c69e0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f06c00b3210 con 0x7f06c00cf4d0 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c4f7d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c01412f0 0x7f06c00b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c4f7d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c01412f0 0x7f06c00b10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57434/0 (socket says 172.21.15.81:57434) 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c4f7d640 1 -- 172.21.15.81:0/2519315301 learned_addr learned my addr 172.21.15.81:0/2519315301 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06bf7fe640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c00cf4d0 0x7f06c013b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06bffff640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06c0073570 0x7f06c01f5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c4f7d640 1 -- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c00cf4d0 msgr2=0x7f06c013b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c4f7d640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c00cf4d0 0x7f06c013b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c4f7d640 1 -- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06c0073570 msgr2=0x7f06c01f5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c4f7d640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06c0073570 0x7f06c01f5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c4f7d640 1 -- 172.21.15.81:0/2519315301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c00aa700 con 0x7f06c01412f0 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c4f7d640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c01412f0 0x7f06c00b10d0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009a50 tx=0x7f06b8009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06bcff9640 1 -- 172.21.15.81:0/2519315301 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b800e2e0 con 0x7f06c01412f0 2024-06-29T15:59:01.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06bcff9640 1 -- 172.21.15.81:0/2519315301 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06b8004b20 con 0x7f06c01412f0 2024-06-29T15:59:01.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06c69e0640 1 -- 172.21.15.81:0/2519315301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c01f5e60 con 0x7f06c01412f0 2024-06-29T15:59:01.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.167+0000 7f06bcff9640 1 -- 172.21.15.81:0/2519315301 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b80078a0 con 0x7f06c01412f0 2024-06-29T15:59:01.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.168+0000 7f06c69e0640 1 -- 172.21.15.81:0/2519315301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c01f6420 con 0x7f06c01412f0 2024-06-29T15:59:01.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.168+0000 7f06c69e0640 1 -- 172.21.15.81:0/2519315301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06c01f5bd0 con 0x7f06c01412f0 2024-06-29T15:59:01.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.169+0000 7f06bcff9640 1 -- 172.21.15.81:0/2519315301 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f06b8007a40 con 0x7f06c01412f0 2024-06-29T15:59:01.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.170+0000 7f06bcff9640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f06ac076510 0x7f06ac078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.170+0000 7f06bcff9640 1 -- 172.21.15.81:0/2519315301 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 6017+0+0 (secure 0 0 0) 0x7f06b8099810 con 0x7f06c01412f0 2024-06-29T15:59:01.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.170+0000 7f06bffff640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f06ac076510 0x7f06ac078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.171+0000 7f06bffff640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f06ac076510 0x7f06ac078a20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f06b0033000 tx=0x7f06b0004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:01.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.174+0000 7f06bcff9640 1 -- 172.21.15.81:0/2519315301 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f06b80624e0 con 0x7f06c01412f0 2024-06-29T15:59:01.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.435+0000 7f06c69e0640 1 -- 172.21.15.81:0/2519315301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) -- 0x7f06c00cf2b0 con 0x7f06c01412f0 2024-06-29T15:59:01.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.705+0000 7f06bcff9640 1 -- 172.21.15.81:0/2519315301 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3166) ==== 187+0+0 (secure 0 0 0) 0x7f06b80667e0 con 0x7f06c01412f0 2024-06-29T15:59:01.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 -- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f06ac076510 msgr2=0x7f06ac078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f06ac076510 0x7f06ac078a20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f06b0033000 tx=0x7f06b0004730 comp rx=0 tx=0).stop 2024-06-29T15:59:01.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 -- 172.21.15.81:0/2519315301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c01412f0 msgr2=0x7f06c00b10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c01412f0 0x7f06c00b10d0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009a50 tx=0x7f06b8009f10 comp rx=0 tx=0).stop 2024-06-29T15:59:01.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 -- 172.21.15.81:0/2519315301 shutdown_connections 2024-06-29T15:59:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f06ac076510 0x7f06ac078a20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c01412f0 0x7f06c00b10d0 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f06c00cf4d0 0x7f06c013b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 --2- 172.21.15.81:0/2519315301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f06c0073570 0x7f06c01f5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 -- 172.21.15.81:0/2519315301 >> 172.21.15.81:0/2519315301 conn(0x7f06c006ccc0 msgr2=0x7f06c006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 -- 172.21.15.81:0/2519315301 shutdown_connections 2024-06-29T15:59:01.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.708+0000 7f06c69e0640 1 -- 172.21.15.81:0/2519315301 wait complete. 2024-06-29T15:59:01.719 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2024-06-29T15:59:01.720 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2024-06-29T15:59:01.720 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2024-06-29T15:59:01.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:01.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.833+0000 7f0e33343640 1 -- 172.21.15.81:0/1335899408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e2c0b6d60 msgr2=0x7f0e2c0ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.833+0000 7f0e33343640 1 --2- 172.21.15.81:0/1335899408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e2c0b6d60 0x7f0e2c0ce4d0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009830 tx=0x7f0e2802a510 comp rx=0 tx=0).stop 2024-06-29T15:59:01.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.833+0000 7f0e33343640 1 -- 172.21.15.81:0/1335899408 shutdown_connections 2024-06-29T15:59:01.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.833+0000 7f0e33343640 1 --2- 172.21.15.81:0/1335899408 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e2c0cea10 0x7f0e2c0b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.833+0000 7f0e33343640 1 --2- 172.21.15.81:0/1335899408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e2c0b6d60 0x7f0e2c0ce4d0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.833+0000 7f0e33343640 1 -- 172.21.15.81:0/1335899408 >> 172.21.15.81:0/1335899408 conn(0x7f0e2c0d1960 msgr2=0x7f0e2c0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:01.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.834+0000 7f0e33343640 1 -- 172.21.15.81:0/1335899408 shutdown_connections 2024-06-29T15:59:01.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.834+0000 7f0e33343640 1 -- 172.21.15.81:0/1335899408 wait complete. 2024-06-29T15:59:01.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.835+0000 7f0e33343640 1 Processor -- start 2024-06-29T15:59:01.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.835+0000 7f0e33343640 1 -- start start 2024-06-29T15:59:01.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e33343640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e2c0b4cf0 0x7f0e2c158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e33343640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e2c0b6d60 0x7f0e2c164d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e33343640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e2c0cea10 0x7f0e2c165270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e33343640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c0cd540 con 0x7f0e2c0cea10 2024-06-29T15:59:01.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e33343640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c0d0a10 con 0x7f0e2c0b6d60 2024-06-29T15:59:01.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e33343640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0e2c0cd6c0 con 0x7f0e2c0b4cf0 2024-06-29T15:59:01.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e318e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e2c0cea10 0x7f0e2c165270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e318e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e2c0cea10 0x7f0e2c165270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57464/0 (socket says 172.21.15.81:57464) 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e318e0640 1 -- 172.21.15.81:0/2770974621 learned_addr learned my addr 172.21.15.81:0/2770974621 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e318e0640 1 -- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e2c0b4cf0 msgr2=0x7f0e2c158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e318e0640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e2c0b4cf0 0x7f0e2c158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e318e0640 1 -- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e2c0b6d60 msgr2=0x7f0e2c164d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e318e0640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e2c0b6d60 0x7f0e2c164d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e318e0640 1 -- 172.21.15.81:0/2770974621 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c0ca0f0 con 0x7f0e2c0cea10 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e318e0640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e2c0cea10 0x7f0e2c165270 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f0e2400caa0 tx=0x7f0e2400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e21ffb640 1 -- 172.21.15.81:0/2770974621 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e24016020 con 0x7f0e2c0cea10 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e21ffb640 1 -- 172.21.15.81:0/2770974621 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e2400ac90 con 0x7f0e2c0cea10 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e21ffb640 1 -- 172.21.15.81:0/2770974621 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e24012680 con 0x7f0e2c0cea10 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.837+0000 7f0e33343640 1 -- 172.21.15.81:0/2770974621 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2c159b60 con 0x7f0e2c0cea10 2024-06-29T15:59:01.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.838+0000 7f0e33343640 1 -- 172.21.15.81:0/2770974621 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e2c0ca300 con 0x7f0e2c0cea10 2024-06-29T15:59:01.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.839+0000 7f0e33343640 1 -- 172.21.15.81:0/2770974621 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e2c163de0 con 0x7f0e2c0cea10 2024-06-29T15:59:01.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.839+0000 7f0e21ffb640 1 -- 172.21.15.81:0/2770974621 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f0e24019020 con 0x7f0e2c0cea10 2024-06-29T15:59:01.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.840+0000 7f0e21ffb640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0e10076510 0x7f0e10078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:01.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.840+0000 7f0e21ffb640 1 -- 172.21.15.81:0/2770974621 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 6017+0+0 (secure 0 0 0) 0x7f0e24098390 con 0x7f0e2c0cea10 2024-06-29T15:59:01.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.840+0000 7f0e310df640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0e10076510 0x7f0e10078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:01.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.840+0000 7f0e310df640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0e10076510 0x7f0e10078a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0e28034000 tx=0x7f0e280046c0 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:01.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:01.844+0000 7f0e21ffb640 1 -- 172.21.15.81:0/2770974621 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0e24020020 con 0x7f0e2c0cea10 2024-06-29T15:59:02.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.025+0000 7f0e33343640 1 -- 172.21.15.81:0/2770974621 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e2c0cdec0 con 0x7f0e2c0cea10 2024-06-29T15:59:02.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.025+0000 7f0e21ffb640 1 -- 172.21.15.81:0/2770974621 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v245) ==== 74+0+14615 (secure 0 0 0) 0x7f0e24061850 con 0x7f0e2c0cea10 2024-06-29T15:59:02.026 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:02.026 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":245,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:00.696523+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6837/30158598":"2024-06-30T15:58:51.291829+0000","172.21.15.167:6835/30158598":"2024-06-30T15:58:51.291829+0000","172.21.15.167:6833/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.81:6835/3228546258":"2024-06-30T15:58:57.682501+0000","172.21.15.81:6833/3228546258":"2024-06-30T15:58:57.682501+0000","172.21.15.167:6832/884808428":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/355438800":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4195196310":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/4186712115":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/3601299355":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2623346726":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/972265206":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/1116386548":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6832/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/3057737710":"2024-06-30T15:58:42.086693+0000","172.21.15.167:6833/3661607921":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1418041738":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/561620344":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/2207847535":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2067929640":"2024-06-30T15:58:47.815510+0000","172.21.15.167:0/874327121":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/273305307":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/2269612087":"2024-06-30T15:58:42.086693+0000","172.21.15.167:0/1188031263":"2024-06-30T15:58:42.086693+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:02.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.027+0000 7f0e33343640 1 -- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0e10076510 msgr2=0x7f0e10078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.027+0000 7f0e33343640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0e10076510 0x7f0e10078a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0e28034000 tx=0x7f0e280046c0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.028+0000 7f0e33343640 1 -- 172.21.15.81:0/2770974621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e2c0cea10 msgr2=0x7f0e2c165270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.028+0000 7f0e33343640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e2c0cea10 0x7f0e2c165270 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f0e2400caa0 tx=0x7f0e2400cf60 comp rx=0 tx=0).stop 2024-06-29T15:59:02.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.028+0000 7f0e33343640 1 -- 172.21.15.81:0/2770974621 shutdown_connections 2024-06-29T15:59:02.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.028+0000 7f0e33343640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0e10076510 0x7f0e10078a20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.028+0000 7f0e33343640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e2c0cea10 0x7f0e2c165270 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.028+0000 7f0e33343640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0e2c0b6d60 0x7f0e2c164d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.028+0000 7f0e33343640 1 --2- 172.21.15.81:0/2770974621 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0e2c0b4cf0 0x7f0e2c158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.028+0000 7f0e33343640 1 -- 172.21.15.81:0/2770974621 >> 172.21.15.81:0/2770974621 conn(0x7f0e2c0d1960 msgr2=0x7f0e2c0b5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:02.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.028+0000 7f0e33343640 1 -- 172.21.15.81:0/2770974621 shutdown_connections 2024-06-29T15:59:02.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.028+0000 7f0e33343640 1 -- 172.21.15.81:0/2770974621 wait complete. 2024-06-29T15:59:02.039 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:02.039 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:02.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.155+0000 7f0ae0963640 1 -- 172.21.15.81:0/1808892083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0adc073030 msgr2=0x7f0adc073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.155+0000 7f0ae0963640 1 --2- 172.21.15.81:0/1808892083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0adc073030 0x7f0adc073440 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0005fd0 tx=0x7f0ad002a5d0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.155+0000 7f0ae0963640 1 -- 172.21.15.81:0/1808892083 shutdown_connections 2024-06-29T15:59:02.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.155+0000 7f0ae0963640 1 --2- 172.21.15.81:0/1808892083 >> v2:172.21.15.167:3300/0 conn(0x7f0adc15e960 0x7f0adc15ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.155+0000 7f0ae0963640 1 --2- 172.21.15.81:0/1808892083 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0adc073030 0x7f0adc073440 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.155+0000 7f0ae0963640 1 -- 172.21.15.81:0/1808892083 >> 172.21.15.81:0/1808892083 conn(0x7f0adc06ccc0 msgr2=0x7f0adc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:02.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.155+0000 7f0ae0963640 1 -- 172.21.15.81:0/1808892083 shutdown_connections 2024-06-29T15:59:02.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.155+0000 7f0ae0963640 1 -- 172.21.15.81:0/1808892083 wait complete. 2024-06-29T15:59:02.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.157+0000 7f0ae0963640 1 Processor -- start 2024-06-29T15:59:02.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.157+0000 7f0ae0963640 1 -- start start 2024-06-29T15:59:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.158+0000 7f0ae0963640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0adc073030 0x7f0adc158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.158+0000 7f0ae0963640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0adc1412f0 0x7f0adc0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.158+0000 7f0ae0963640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0adc15e960 0x7f0adc0a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ae0963640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc0b3390 con 0x7f0adc073030 2024-06-29T15:59:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ae0963640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc0b3210 con 0x7f0adc15e960 2024-06-29T15:59:02.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ae0963640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0adc0b3510 con 0x7f0adc1412f0 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ada59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0adc073030 0x7f0adc158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ada59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0adc073030 0x7f0adc158920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57480/0 (socket says 172.21.15.81:57480) 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ada59c640 1 -- 172.21.15.81:0/1315946370 learned_addr learned my addr 172.21.15.81:0/1315946370 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0adad9d640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0adc15e960 0x7f0adc0a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ad9d9b640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0adc1412f0 0x7f0adc0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ada59c640 1 -- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0adc1412f0 msgr2=0x7f0adc0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ada59c640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0adc1412f0 0x7f0adc0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ada59c640 1 -- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0adc15e960 msgr2=0x7f0adc0a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ada59c640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0adc15e960 0x7f0adc0a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ada59c640 1 -- 172.21.15.81:0/1315946370 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0adc0ad9a0 con 0x7f0adc073030 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0adad9d640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0adc15e960 0x7f0adc0a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ad9d9b640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0adc1412f0 0x7f0adc0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ada59c640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0adc073030 0x7f0adc158920 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00c770 tx=0x7f0acc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:02.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0acaffd640 1 -- 172.21.15.81:0/1315946370 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc003b70 con 0x7f0adc073030 2024-06-29T15:59:02.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0acaffd640 1 -- 172.21.15.81:0/1315946370 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0acc003d30 con 0x7f0adc073030 2024-06-29T15:59:02.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ae0963640 1 -- 172.21.15.81:0/1315946370 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0adc0adc80 con 0x7f0adc073030 2024-06-29T15:59:02.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0acaffd640 1 -- 172.21.15.81:0/1315946370 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc00aa50 con 0x7f0adc073030 2024-06-29T15:59:02.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.159+0000 7f0ae0963640 1 -- 172.21.15.81:0/1315946370 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0adc0ae240 con 0x7f0adc073030 2024-06-29T15:59:02.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.161+0000 7f0ae0963640 1 -- 172.21.15.81:0/1315946370 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0adc0cce40 con 0x7f0adc073030 2024-06-29T15:59:02.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.161+0000 7f0acaffd640 1 -- 172.21.15.81:0/1315946370 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f0acc00abf0 con 0x7f0adc073030 2024-06-29T15:59:02.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.162+0000 7f0acaffd640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0abc076510 0x7f0abc078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.162+0000 7f0acaffd640 1 -- 172.21.15.81:0/1315946370 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 6017+0+0 (secure 0 0 0) 0x7f0acc097d10 con 0x7f0adc073030 2024-06-29T15:59:02.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.162+0000 7f0ad9d9b640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0abc076510 0x7f0abc078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.163+0000 7f0ad9d9b640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0abc076510 0x7f0abc078a20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0033000 tx=0x7f0ad0004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:02.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.166+0000 7f0acaffd640 1 -- 172.21.15.81:0/1315946370 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0adc0cce40 con 0x7f0adc073030 2024-06-29T15:59:02.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.343+0000 7f0ae0963640 1 -- 172.21.15.81:0/1315946370 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0adc0b3a10 con 0x7f0adc073030 2024-06-29T15:59:02.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.344+0000 7f0acaffd640 1 -- 172.21.15.81:0/1315946370 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v245) ==== 93+0+10 (secure 0 0 0) 0x7f0adc073980 con 0x7f0adc073030 2024-06-29T15:59:02.344 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:59:02.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 -- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0abc076510 msgr2=0x7f0abc078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0abc076510 0x7f0abc078a20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0033000 tx=0x7f0ad0004730 comp rx=0 tx=0).stop 2024-06-29T15:59:02.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 -- 172.21.15.81:0/1315946370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0adc073030 msgr2=0x7f0adc158920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0adc073030 0x7f0adc158920 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00c770 tx=0x7f0acc00cc30 comp rx=0 tx=0).stop 2024-06-29T15:59:02.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 -- 172.21.15.81:0/1315946370 shutdown_connections 2024-06-29T15:59:02.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f0abc076510 0x7f0abc078a20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f0adc15e960 0x7f0adc0a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f0adc1412f0 0x7f0adc0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 --2- 172.21.15.81:0/1315946370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0adc073030 0x7f0adc158920 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 -- 172.21.15.81:0/1315946370 >> 172.21.15.81:0/1315946370 conn(0x7f0adc06ccc0 msgr2=0x7f0adc15cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:02.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 -- 172.21.15.81:0/1315946370 shutdown_connections 2024-06-29T15:59:02.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.346+0000 7f0ae0963640 1 -- 172.21.15.81:0/1315946370 wait complete. 2024-06-29T15:59:02.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-06-29T15:59:02.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.471+0000 7fe0e8f82640 1 -- 172.21.15.81:0/3602654997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0e40b6eb0 msgr2=0x7fe0e40d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.471+0000 7fe0e8f82640 1 --2- 172.21.15.81:0/3602654997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0e40b6eb0 0x7fe0e40d02d0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4009830 tx=0x7fe0d402a470 comp rx=0 tx=0).stop 2024-06-29T15:59:02.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.471+0000 7fe0e8f82640 1 -- 172.21.15.81:0/3602654997 shutdown_connections 2024-06-29T15:59:02.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.471+0000 7fe0e8f82640 1 --2- 172.21.15.81:0/3602654997 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0e40ce920 0x7fe0e40cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.471+0000 7fe0e8f82640 1 --2- 172.21.15.81:0/3602654997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0e40b6eb0 0x7fe0e40d02d0 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.471+0000 7fe0e8f82640 1 -- 172.21.15.81:0/3602654997 >> 172.21.15.81:0/3602654997 conn(0x7fe0e40b3510 msgr2=0x7fe0e40b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:02.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.471+0000 7fe0e8f82640 1 -- 172.21.15.81:0/3602654997 shutdown_connections 2024-06-29T15:59:02.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.471+0000 7fe0e8f82640 1 -- 172.21.15.81:0/3602654997 wait complete. 2024-06-29T15:59:02.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.473+0000 7fe0e8f82640 1 Processor -- start 2024-06-29T15:59:02.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.473+0000 7fe0e8f82640 1 -- start start 2024-06-29T15:59:02.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.474+0000 7fe0e8f82640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0e40b6eb0 0x7fe0e41f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.474+0000 7fe0e8f82640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0e40ce510 0x7fe0e41f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.474+0000 7fe0e8f82640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0e40ce920 0x7fe0e415e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.474+0000 7fe0e8f82640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e40b50a0 con 0x7fe0e40ce920 2024-06-29T15:59:02.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.474+0000 7fe0e8f82640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e40b4f20 con 0x7fe0e40b6eb0 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.474+0000 7fe0e8f82640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e40b5220 con 0x7fe0e40ce510 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.474+0000 7fe0e2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0e40ce920 0x7fe0e415e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.474+0000 7fe0e2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0e40ce920 0x7fe0e415e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57502/0 (socket says 172.21.15.81:57502) 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.474+0000 7fe0e2d9d640 1 -- 172.21.15.81:0/975114173 learned_addr learned my addr 172.21.15.81:0/975114173 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e1d9b640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0e40ce510 0x7fe0e41f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e259c640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0e40b6eb0 0x7fe0e41f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e2d9d640 1 -- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0e40ce510 msgr2=0x7fe0e41f6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e2d9d640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0e40ce510 0x7fe0e41f6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e2d9d640 1 -- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0e40b6eb0 msgr2=0x7fe0e41f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e2d9d640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0e40b6eb0 0x7fe0e41f6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e2d9d640 1 -- 172.21.15.81:0/975114173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e40af520 con 0x7fe0e40ce920 2024-06-29T15:59:02.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e1d9b640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0e40ce510 0x7fe0e41f6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:02.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e259c640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0e40b6eb0 0x7fe0e41f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:02.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e2d9d640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0e40ce920 0x7fe0e415e490 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00e9c0 tx=0x7fe0dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:02.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0d2ffd640 1 -- 172.21.15.81:0/975114173 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc004d20 con 0x7fe0e40ce920 2024-06-29T15:59:02.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0d2ffd640 1 -- 172.21.15.81:0/975114173 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0dc00cb20 con 0x7fe0e40ce920 2024-06-29T15:59:02.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e8f82640 1 -- 172.21.15.81:0/975114173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e40c9480 con 0x7fe0e40ce920 2024-06-29T15:59:02.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0d2ffd640 1 -- 172.21.15.81:0/975114173 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc005020 con 0x7fe0e40ce920 2024-06-29T15:59:02.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.475+0000 7fe0e8f82640 1 -- 172.21.15.81:0/975114173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e40c9720 con 0x7fe0e40ce920 2024-06-29T15:59:02.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.476+0000 7fe0e8f82640 1 -- 172.21.15.81:0/975114173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e40ca460 con 0x7fe0e40ce920 2024-06-29T15:59:02.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.477+0000 7fe0d2ffd640 1 -- 172.21.15.81:0/975114173 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fe0dc0051c0 con 0x7fe0e40ce920 2024-06-29T15:59:02.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.477+0000 7fe0d2ffd640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe0c4076540 0x7fe0c4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:02.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.478+0000 7fe0d2ffd640 1 -- 172.21.15.81:0/975114173 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 6017+0+0 (secure 0 0 0) 0x7fe0dc0986f0 con 0x7fe0e40ce920 2024-06-29T15:59:02.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.478+0000 7fe0e259c640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe0c4076540 0x7fe0c4078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:02.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.478+0000 7fe0e259c640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe0c4076540 0x7fe0c4078a50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4034000 tx=0x7fe0d4004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:02.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.482+0000 7fe0d2ffd640 1 -- 172.21.15.81:0/975114173 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe0dc016030 con 0x7fe0e40ce920 2024-06-29T15:59:02.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:02.735+0000 7fe0e8f82640 1 -- 172.21.15.81:0/975114173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) -- 0x7fe0e40d02d0 con 0x7fe0e40ce920 2024-06-29T15:59:03.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.712+0000 7fe0d2ffd640 1 -- 172.21.15.81:0/975114173 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3168) ==== 158+0+0 (secure 0 0 0) 0x7fe0dc061a10 con 0x7fe0e40ce920 2024-06-29T15:59:03.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.714+0000 7fe0e8f82640 1 -- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe0c4076540 msgr2=0x7fe0c4078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:03.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.714+0000 7fe0e8f82640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe0c4076540 0x7fe0c4078a50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4034000 tx=0x7fe0d4004730 comp rx=0 tx=0).stop 2024-06-29T15:59:03.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.714+0000 7fe0e8f82640 1 -- 172.21.15.81:0/975114173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0e40ce920 msgr2=0x7fe0e415e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:03.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.714+0000 7fe0e8f82640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0e40ce920 0x7fe0e415e490 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00e9c0 tx=0x7fe0dc00ee80 comp rx=0 tx=0).stop 2024-06-29T15:59:03.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.714+0000 7fe0e8f82640 1 -- 172.21.15.81:0/975114173 shutdown_connections 2024-06-29T15:59:03.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.715+0000 7fe0e8f82640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fe0c4076540 0x7fe0c4078a50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.715+0000 7fe0e8f82640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0e40ce920 0x7fe0e415e490 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.715+0000 7fe0e8f82640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe0e40ce510 0x7fe0e41f6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.715+0000 7fe0e8f82640 1 --2- 172.21.15.81:0/975114173 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe0e40b6eb0 0x7fe0e41f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.715+0000 7fe0e8f82640 1 -- 172.21.15.81:0/975114173 >> 172.21.15.81:0/975114173 conn(0x7fe0e40b3510 msgr2=0x7fe0e4070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:03.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.715+0000 7fe0e8f82640 1 -- 172.21.15.81:0/975114173 shutdown_connections 2024-06-29T15:59:03.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.715+0000 7fe0e8f82640 1 -- 172.21.15.81:0/975114173 wait complete. 2024-06-29T15:59:03.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:59:03.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.847+0000 7f6e09cc3640 1 -- 172.21.15.81:0/1786588826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e040b6eb0 msgr2=0x7f6e040d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:03.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.847+0000 7f6e09cc3640 1 --2- 172.21.15.81:0/1786588826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e040b6eb0 0x7f6e040d02d0 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f6e04073550 tx=0x7f6df402a4d0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.847+0000 7f6e09cc3640 1 -- 172.21.15.81:0/1786588826 shutdown_connections 2024-06-29T15:59:03.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.847+0000 7f6e09cc3640 1 --2- 172.21.15.81:0/1786588826 >> v2:172.21.15.167:3300/0 conn(0x7f6e040ce510 0x7f6e040ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.847+0000 7f6e09cc3640 1 --2- 172.21.15.81:0/1786588826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e040b6eb0 0x7f6e040d02d0 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.847+0000 7f6e09cc3640 1 -- 172.21.15.81:0/1786588826 >> 172.21.15.81:0/1786588826 conn(0x7f6e040b3510 msgr2=0x7f6e040b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:03.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.847+0000 7f6e09cc3640 1 -- 172.21.15.81:0/1786588826 shutdown_connections 2024-06-29T15:59:03.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.847+0000 7f6e09cc3640 1 -- 172.21.15.81:0/1786588826 wait complete. 2024-06-29T15:59:03.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.848+0000 7f6e09cc3640 1 Processor -- start 2024-06-29T15:59:03.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.849+0000 7f6e09cc3640 1 -- start start 2024-06-29T15:59:03.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e09cc3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e040b4810 0x7f6e041f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:03.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e09cc3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e040b6eb0 0x7f6e04165720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e09cc3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e040ce510 0x7f6e0406d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e09cc3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6e040cd480 con 0x7f6e040b4810 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e09cc3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6e040b29c0 con 0x7f6e040ce510 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e09cc3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6e040cd600 con 0x7f6e040b6eb0 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e040b4810 0x7f6e041f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e040b4810 0x7f6e041f4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57510/0 (socket says 172.21.15.81:57510) 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e037fe640 1 -- 172.21.15.81:0/4062605946 learned_addr learned my addr 172.21.15.81:0/4062605946 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e03fff640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e040ce510 0x7f6e0406d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e02ffd640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e040b6eb0 0x7f6e04165720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e037fe640 1 -- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e040b6eb0 msgr2=0x7f6e04165720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e037fe640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e040b6eb0 0x7f6e04165720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e037fe640 1 -- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e040ce510 msgr2=0x7f6e0406d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e037fe640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e040ce510 0x7f6e0406d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.850+0000 7f6e037fe640 1 -- 172.21.15.81:0/4062605946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0415db70 con 0x7f6e040b4810 2024-06-29T15:59:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.851+0000 7f6e037fe640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e040b4810 0x7f6e041f4ca0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f6df4034040 tx=0x7f6df40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.851+0000 7f6debfff640 1 -- 172.21.15.81:0/4062605946 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df403e030 con 0x7f6e040b4810 2024-06-29T15:59:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.851+0000 7f6debfff640 1 -- 172.21.15.81:0/4062605946 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6df40047a0 con 0x7f6e040b4810 2024-06-29T15:59:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.851+0000 7f6debfff640 1 -- 172.21.15.81:0/4062605946 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4035040 con 0x7f6e040b4810 2024-06-29T15:59:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.851+0000 7f6e09cc3640 1 -- 172.21.15.81:0/4062605946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0415de50 con 0x7f6e040b4810 2024-06-29T15:59:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.851+0000 7f6e09cc3640 1 -- 172.21.15.81:0/4062605946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e0415a310 con 0x7f6e040b4810 2024-06-29T15:59:03.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.852+0000 7f6e09cc3640 1 -- 172.21.15.81:0/4062605946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e040d02d0 con 0x7f6e040b4810 2024-06-29T15:59:03.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.853+0000 7f6debfff640 1 -- 172.21.15.81:0/4062605946 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f6df4041030 con 0x7f6e040b4810 2024-06-29T15:59:03.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.853+0000 7f6debfff640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6df0076510 0x7f6df0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:03.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.853+0000 7f6debfff640 1 -- 172.21.15.81:0/4062605946 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 6017+0+0 (secure 0 0 0) 0x7f6df40b84a0 con 0x7f6e040b4810 2024-06-29T15:59:03.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.853+0000 7f6e02ffd640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6df0076510 0x7f6df0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:03.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.854+0000 7f6e02ffd640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6df0076510 0x7f6df0078a20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6e0415cbe0 tx=0x7f6df8008040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:03.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:03.857+0000 7f6debfff640 1 -- 172.21.15.81:0/4062605946 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6df40812d0 con 0x7f6e040b4810 2024-06-29T15:59:03.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:59:03.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:59:04.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.044+0000 7f6e09cc3640 1 -- 172.21.15.81:0/4062605946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f6e040cdf00 con 0x7f6e040b4810 2024-06-29T15:59:04.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.044+0000 7f6debfff640 1 -- 172.21.15.81:0/4062605946 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v79) ==== 59+0+1998 (secure 0 0 0) 0x7f6df40855d0 con 0x7f6e040b4810 2024-06-29T15:59:04.045 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:59:04.045 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:59:04.045 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:59:04.045 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:59:04.045 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T15:59:04.045 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T15:59:04.045 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:59:04.046 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T15:59:04.047 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:59:04.047 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:59:04.047 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:59:04.047 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T15:59:04.047 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T15:59:04.047 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T15:59:04.047 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:59:04.047 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:59:04.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 -- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6df0076510 msgr2=0x7f6df0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6df0076510 0x7f6df0078a20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6e0415cbe0 tx=0x7f6df8008040 comp rx=0 tx=0).stop 2024-06-29T15:59:04.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 -- 172.21.15.81:0/4062605946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e040b4810 msgr2=0x7f6e041f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e040b4810 0x7f6e041f4ca0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f6df4034040 tx=0x7f6df40039e0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 -- 172.21.15.81:0/4062605946 shutdown_connections 2024-06-29T15:59:04.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f6df0076510 0x7f6df0078a20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e040ce510 0x7f6e0406d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e040b6eb0 0x7f6e04165720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 --2- 172.21.15.81:0/4062605946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e040b4810 0x7f6e041f4ca0 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 -- 172.21.15.81:0/4062605946 >> 172.21.15.81:0/4062605946 conn(0x7f6e040b3510 msgr2=0x7f6e04070d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:04.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 -- 172.21.15.81:0/4062605946 shutdown_connections 2024-06-29T15:59:04.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.046+0000 7f6e09cc3640 1 -- 172.21.15.81:0/4062605946 wait complete. 2024-06-29T15:59:04.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:59:04.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.214+0000 7f94b719e640 1 -- 172.21.15.81:0/1271378230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b00b4430 msgr2=0x7f94b00cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.214+0000 7f94b719e640 1 --2- 172.21.15.81:0/1271378230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b00b4430 0x7f94b00cf130 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f94b0070220 tx=0x7f94a402a980 comp rx=0 tx=0).stop 2024-06-29T15:59:04.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.214+0000 7f94b719e640 1 -- 172.21.15.81:0/1271378230 shutdown_connections 2024-06-29T15:59:04.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.214+0000 7f94b719e640 1 --2- 172.21.15.81:0/1271378230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b00b4430 0x7f94b00cf130 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.214+0000 7f94b719e640 1 --2- 172.21.15.81:0/1271378230 >> v2:172.21.15.167:3300/0 conn(0x7f94b00b35b0 0x7f94b015b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.214+0000 7f94b719e640 1 -- 172.21.15.81:0/1271378230 >> 172.21.15.81:0/1271378230 conn(0x7f94b006ccc0 msgr2=0x7f94b00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:04.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.214+0000 7f94b719e640 1 -- 172.21.15.81:0/1271378230 shutdown_connections 2024-06-29T15:59:04.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.214+0000 7f94b719e640 1 -- 172.21.15.81:0/1271378230 wait complete. 2024-06-29T15:59:04.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.216+0000 7f94b719e640 1 Processor -- start 2024-06-29T15:59:04.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.216+0000 7f94b719e640 1 -- start start 2024-06-29T15:59:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.217+0000 7f94b719e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b006f660 0x7f94b013db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.217+0000 7f94b719e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f94b00b35b0 0x7f94b0151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.217+0000 7f94b719e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94b00b4430 0x7f94b00cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.217+0000 7f94b719e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f94b0072330 con 0x7f94b006f660 2024-06-29T15:59:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.217+0000 7f94b719e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f94b00721b0 con 0x7f94b00b4430 2024-06-29T15:59:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.217+0000 7f94b719e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f94b00724b0 con 0x7f94b00b35b0 2024-06-29T15:59:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.217+0000 7f94b4f3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b006f660 0x7f94b013db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.217+0000 7f94b4f3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b006f660 0x7f94b013db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57538/0 (socket says 172.21.15.81:57538) 2024-06-29T15:59:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.217+0000 7f94b4f3a640 1 -- 172.21.15.81:0/3561772559 learned_addr learned my addr 172.21.15.81:0/3561772559 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:04.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.217+0000 7f94b573b640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94b00b4430 0x7f94b00cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94affff640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f94b00b35b0 0x7f94b0151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94b4f3a640 1 -- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f94b00b35b0 msgr2=0x7f94b0151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94b4f3a640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f94b00b35b0 0x7f94b0151a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94b4f3a640 1 -- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94b00b4430 msgr2=0x7f94b00cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94b4f3a640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94b00b4430 0x7f94b00cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94b4f3a640 1 -- 172.21.15.81:0/3561772559 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b013e080 con 0x7f94b006f660 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94b4f3a640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b006f660 0x7f94b013db40 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f94a000b440 tx=0x7f94a000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94ad7fa640 1 -- 172.21.15.81:0/3561772559 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0015020 con 0x7f94b006f660 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94ad7fa640 1 -- 172.21.15.81:0/3561772559 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94a0011980 con 0x7f94b006f660 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94ad7fa640 1 -- 172.21.15.81:0/3561772559 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0011dd0 con 0x7f94b006f660 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94b719e640 1 -- 172.21.15.81:0/3561772559 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b0153bc0 con 0x7f94b006f660 2024-06-29T15:59:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.218+0000 7f94b719e640 1 -- 172.21.15.81:0/3561772559 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b0154180 con 0x7f94b006f660 2024-06-29T15:59:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.219+0000 7f94b719e640 1 -- 172.21.15.81:0/3561772559 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b01536e0 con 0x7f94b006f660 2024-06-29T15:59:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.220+0000 7f94ad7fa640 1 -- 172.21.15.81:0/3561772559 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f94a0018020 con 0x7f94b006f660 2024-06-29T15:59:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.220+0000 7f94ad7fa640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f949c076510 0x7f949c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.220+0000 7f94ad7fa640 1 -- 172.21.15.81:0/3561772559 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 6017+0+0 (secure 0 0 0) 0x7f94a0097df0 con 0x7f94b006f660 2024-06-29T15:59:04.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.221+0000 7f94affff640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f949c076510 0x7f949c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.221+0000 7f94affff640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f949c076510 0x7f949c078a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f94a4035000 tx=0x7f94a4004ab0 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:04.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.225+0000 7f94ad7fa640 1 -- 172.21.15.81:0/3561772559 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f94b01536e0 con 0x7f94b006f660 2024-06-29T15:59:04.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.407+0000 7f94b719e640 1 -- 172.21.15.81:0/3561772559 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f94b00731a0 con 0x7f94b006f660 2024-06-29T15:59:04.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.411+0000 7f94ad7fa640 1 -- 172.21.15.81:0/3561772559 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v80)=0 v80) ==== 122+0+0 (secure 0 0 0) 0x7f94a001f020 con 0x7f94b006f660 2024-06-29T15:59:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 -- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f949c076510 msgr2=0x7f949c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f949c076510 0x7f949c078a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f94a4035000 tx=0x7f94a4004ab0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 -- 172.21.15.81:0/3561772559 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b006f660 msgr2=0x7f94b013db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b006f660 0x7f94b013db40 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f94a000b440 tx=0x7f94a000b900 comp rx=0 tx=0).stop 2024-06-29T15:59:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 -- 172.21.15.81:0/3561772559 shutdown_connections 2024-06-29T15:59:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f949c076510 0x7f949c078a20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f94b00b4430 0x7f94b00cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f94b00b35b0 0x7f94b0151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 --2- 172.21.15.81:0/3561772559 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b006f660 0x7f94b013db40 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 -- 172.21.15.81:0/3561772559 >> 172.21.15.81:0/3561772559 conn(0x7f94b006ccc0 msgr2=0x7f94b015b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:04.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 -- 172.21.15.81:0/3561772559 shutdown_connections 2024-06-29T15:59:04.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.413+0000 7f94b719e640 1 -- 172.21.15.81:0/3561772559 wait complete. 2024-06-29T15:59:04.425 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:59:04.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:04.472 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:04.517 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:59:04.517 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:59:04.517 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:04.533 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:59:04.533 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:59:04.533 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:04.575 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:04.575 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:59:04.575 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:04.591 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:59:04.591 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:59:04.591 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:04.633 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:04.633 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:59:04.633 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:04.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.750+0000 7f10217a1640 1 -- 172.21.15.81:0/3787090511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10040039e0 msgr2=0x7f1004003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.750+0000 7f10217a1640 1 --2- 172.21.15.81:0/3787090511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10040039e0 0x7f1004003e10 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f101c0b50b0 tx=0x7f1004005830 comp rx=0 tx=0).stop 2024-06-29T15:59:04.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.750+0000 7f1020fa0640 1 -- 172.21.15.81:0/3787090511 reap_dead start 2024-06-29T15:59:04.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.750+0000 7f10217a1640 1 -- 172.21.15.81:0/3787090511 shutdown_connections 2024-06-29T15:59:04.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.750+0000 7f10217a1640 1 -- 172.21.15.81:0/3787090511 >> 172.21.15.81:0/3787090511 conn(0x7f101c0d08f0 msgr2=0x7f101c0d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:04.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.750+0000 7f10217a1640 1 -- 172.21.15.81:0/3787090511 shutdown_connections 2024-06-29T15:59:04.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.750+0000 7f10217a1640 1 -- 172.21.15.81:0/3787090511 wait complete. 2024-06-29T15:59:04.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.752+0000 7f10217a1640 1 Processor -- start 2024-06-29T15:59:04.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.752+0000 7f10217a1640 1 -- start start 2024-06-29T15:59:04.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.753+0000 7f10217a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10040039e0 0x7f101c150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f10217a1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f101c15d350 0x7f101c0708f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f10217a1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f101c070e30 0x7f101c15b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f10217a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f101c074030 con 0x7f10040039e0 2024-06-29T15:59:04.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f10217a1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f101c073eb0 con 0x7f101c15d350 2024-06-29T15:59:04.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f10217a1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f101c0741b0 con 0x7f101c070e30 2024-06-29T15:59:04.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f101bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10040039e0 0x7f101c150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f101bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10040039e0 0x7f101c150fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57556/0 (socket says 172.21.15.81:57556) 2024-06-29T15:59:04.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f101bfff640 1 -- 172.21.15.81:0/1553675943 learned_addr learned my addr 172.21.15.81:0/1553675943 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:04.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f101b7fe640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f101c15d350 0x7f101c0708f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f1020fa0640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f101c070e30 0x7f101c15b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f101bfff640 1 -- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f101c070e30 msgr2=0x7f101c15b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f101bfff640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f101c070e30 0x7f101c15b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f101bfff640 1 -- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f101c15d350 msgr2=0x7f101c0708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f101bfff640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f101c15d350 0x7f101c0708f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f101bfff640 1 -- 172.21.15.81:0/1553675943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101c071240 con 0x7f10040039e0 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f101bfff640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10040039e0 0x7f101c150fd0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f100c02b080 tx=0x7f100c001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f1018ff9640 1 -- 172.21.15.81:0/1553675943 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c0037f0 con 0x7f10040039e0 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f1018ff9640 1 -- 172.21.15.81:0/1553675943 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f100c044070 con 0x7f10040039e0 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f10217a1640 1 -- 172.21.15.81:0/1553675943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101c1578a0 con 0x7f10040039e0 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f1018ff9640 1 -- 172.21.15.81:0/1553675943 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c003d00 con 0x7f10040039e0 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.754+0000 7f10217a1640 1 -- 172.21.15.81:0/1553675943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101c157e60 con 0x7f10040039e0 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.755+0000 7f10217a1640 1 -- 172.21.15.81:0/1553675943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f101c158520 con 0x7f10040039e0 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.756+0000 7f1018ff9640 1 -- 172.21.15.81:0/1553675943 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f100c03e070 con 0x7f10040039e0 2024-06-29T15:59:04.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.757+0000 7f1018ff9640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1008076510 0x7f1008078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:04.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.757+0000 7f1018ff9640 1 -- 172.21.15.81:0/1553675943 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 6017+0+0 (secure 0 0 0) 0x7f100c0bb020 con 0x7f10040039e0 2024-06-29T15:59:04.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.757+0000 7f101b7fe640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1008076510 0x7f1008078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:04.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.757+0000 7f101b7fe640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1008076510 0x7f1008078a20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f101c0ce220 tx=0x7f1010005f90 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:04.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.760+0000 7f1018ff9640 1 -- 172.21.15.81:0/1553675943 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f101c071240 con 0x7f10040039e0 2024-06-29T15:59:04.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.978+0000 7f10217a1640 1 -- 172.21.15.81:0/1553675943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f101c0b4150 con 0x7f10040039e0 2024-06-29T15:59:04.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.979+0000 7f1018ff9640 1 -- 172.21.15.81:0/1553675943 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+3919 (secure 0 0 0) 0x7f100c083dd0 con 0x7f10040039e0 2024-06-29T15:59:04.979 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:04.979 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":139,"btime":"2024-06-29T15:58:58:686156+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9380,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2163599993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2163599993},{"type":"v1","addr":"172.21.15.167:6836","nonce":2163599993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":124},{"gid":11218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/113384617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":113384617},{"type":"v1","addr":"172.21.15.167:6837","nonce":113384617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":12090,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/233681791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":233681791},{"type":"v1","addr":"172.21.15.81:6834","nonce":233681791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":123},{"gid":12768,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2306161626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2306161626},{"type":"v1","addr":"172.21.15.81:6835","nonce":2306161626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[]} 2024-06-29T15:59:04.979 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 139 2024-06-29T15:59:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.980+0000 7f10217a1640 1 -- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1008076510 msgr2=0x7f1008078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.980+0000 7f10217a1640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1008076510 0x7f1008078a20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f101c0ce220 tx=0x7f1010005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.980+0000 7f10217a1640 1 -- 172.21.15.81:0/1553675943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10040039e0 msgr2=0x7f101c150fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.980+0000 7f10217a1640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10040039e0 0x7f101c150fd0 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f100c02b080 tx=0x7f100c001dc0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.981+0000 7f10217a1640 1 -- 172.21.15.81:0/1553675943 shutdown_connections 2024-06-29T15:59:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.981+0000 7f10217a1640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1008076510 0x7f1008078a20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.981+0000 7f10217a1640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f101c070e30 0x7f101c15b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.981+0000 7f10217a1640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f101c15d350 0x7f101c0708f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.981+0000 7f10217a1640 1 --2- 172.21.15.81:0/1553675943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10040039e0 0x7f101c150fd0 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.981+0000 7f10217a1640 1 -- 172.21.15.81:0/1553675943 >> 172.21.15.81:0/1553675943 conn(0x7f101c0d08f0 msgr2=0x7f101c156280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.981+0000 7f10217a1640 1 -- 172.21.15.81:0/1553675943 shutdown_connections 2024-06-29T15:59:04.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:04.981+0000 7f10217a1640 1 -- 172.21.15.81:0/1553675943 wait complete. 2024-06-29T15:59:04.992 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:59:04.992 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:59:04.992 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:59:04.993 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:04.993 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:59:04.993 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:59:04.993 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:59:04.993 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:04.993 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:59:04.994 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:59:04.994 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:59:04.994 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:04.994 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:59:04.994 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:59:04.994 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:59:04.994 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:05.002 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:59:05.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:59:05.005 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:59:05.005 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:59:05.006 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:59:05.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:59:05.009 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:59:05.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:59:05.047 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:59:05.047 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:05.048 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:59:05.049 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:05.051 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:59:05.051 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:05.053 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:59:05.053 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:05.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:59:05.134 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T15:59:05.135 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T15:59:05.135 INFO:tasks.ceph.mds.a.smithi081.stderr:2024-06-29T15:59:05.134+0000 7efd308fd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-06-29T15:59:05.142 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:59:05.143 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T15:59:05.143 INFO:tasks.ceph.mds.d.smithi167.stderr:2024-06-29T15:59:05.143+0000 7ff751916640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-06-29T15:59:05.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.173+0000 7fbba85c7640 1 -- 172.21.15.81:0/1026452170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbba00b7290 msgr2=0x7fbba00b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.173+0000 7fbba85c7640 1 --2- 172.21.15.81:0/1026452170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbba00b7290 0x7fbba00b76a0 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7fbba00b9b80 tx=0x7fbb9802aac0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.173+0000 7fbba85c7640 1 -- 172.21.15.81:0/1026452170 shutdown_connections 2024-06-29T15:59:05.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.173+0000 7fbba85c7640 1 --2- 172.21.15.81:0/1026452170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbba00b7290 0x7fbba00b76a0 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.173+0000 7fbba85c7640 1 --2- 172.21.15.81:0/1026452170 >> v2:172.21.15.167:3300/0 conn(0x7fbba00b5250 0x7fbba00d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.173+0000 7fbba85c7640 1 --2- 172.21.15.81:0/1026452170 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbba00b9040 0x7fbba00d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.173+0000 7fbba85c7640 1 -- 172.21.15.81:0/1026452170 >> 172.21.15.81:0/1026452170 conn(0x7fbba009cf30 msgr2=0x7fbba00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:05.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.173+0000 7fbba85c7640 1 -- 172.21.15.81:0/1026452170 shutdown_connections 2024-06-29T15:59:05.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.173+0000 7fbba85c7640 1 -- 172.21.15.81:0/1026452170 wait complete. 2024-06-29T15:59:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.175+0000 7fbba85c7640 1 Processor -- start 2024-06-29T15:59:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.175+0000 7fbba85c7640 1 -- start start 2024-06-29T15:59:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba85c7640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbba00b5250 0x7fbba014fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba85c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbba00b7290 0x7fbba014e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba85c7640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbba00b9040 0x7fbba0150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba85c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbba00b7f70 con 0x7fbba00b7290 2024-06-29T15:59:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba85c7640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbba00b7df0 con 0x7fbba00b9040 2024-06-29T15:59:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba85c7640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbba00b80f0 con 0x7fbba00b5250 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba5b62640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbba00b7290 0x7fbba014e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba5b62640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbba00b7290 0x7fbba014e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57606/0 (socket says 172.21.15.81:57606) 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba5b62640 1 -- 172.21.15.81:0/733159002 learned_addr learned my addr 172.21.15.81:0/733159002 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba6363640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbba00b5250 0x7fbba014fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba6b64640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbba00b9040 0x7fbba0150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba5b62640 1 -- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbba00b5250 msgr2=0x7fbba014fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba5b62640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbba00b5250 0x7fbba014fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba5b62640 1 -- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbba00b9040 msgr2=0x7fbba0150a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba5b62640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbba00b9040 0x7fbba0150a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba5b62640 1 -- 172.21.15.81:0/733159002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba014ea70 con 0x7fbba00b7290 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.176+0000 7fbba5b62640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbba00b7290 0x7fbba014e100 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000c970 tx=0x7fbb9000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.177+0000 7fbb96ffd640 1 -- 172.21.15.81:0/733159002 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb90004220 con 0x7fbba00b7290 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.177+0000 7fbb96ffd640 1 -- 172.21.15.81:0/733159002 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb900079b0 con 0x7fbba00b7290 2024-06-29T15:59:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.177+0000 7fbba85c7640 1 -- 172.21.15.81:0/733159002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba014ed50 con 0x7fbba00b7290 2024-06-29T15:59:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.177+0000 7fbb96ffd640 1 -- 172.21.15.81:0/733159002 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb90007de0 con 0x7fbba00b7290 2024-06-29T15:59:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.177+0000 7fbba85c7640 1 -- 172.21.15.81:0/733159002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba014f310 con 0x7fbba00b7290 2024-06-29T15:59:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.178+0000 7fbba85c7640 1 -- 172.21.15.81:0/733159002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba00b76a0 con 0x7fbba00b7290 2024-06-29T15:59:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.178+0000 7fbb96ffd640 1 -- 172.21.15.81:0/733159002 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fbb90005020 con 0x7fbba00b7290 2024-06-29T15:59:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.179+0000 7fbb96ffd640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbb84076510 0x7fbb84078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.179+0000 7fbb96ffd640 1 -- 172.21.15.81:0/733159002 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 6017+0+0 (secure 0 0 0) 0x7fbb900980c0 con 0x7fbba00b7290 2024-06-29T15:59:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.179+0000 7fbba6363640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbb84076510 0x7fbb84078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.180+0000 7fbba6363640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbb84076510 0x7fbb84078a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0058c0 tx=0x7fbb9c005f60 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:05.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.183+0000 7fbb96ffd640 1 -- 172.21.15.81:0/733159002 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbb90014030 con 0x7fbba00b7290 2024-06-29T15:59:05.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.375+0000 7fbba85c7640 1 -- 172.21.15.81:0/733159002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fbba014e790 con 0x7fbba00b7290 2024-06-29T15:59:05.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.732+0000 7fbb96ffd640 1 -- 172.21.15.81:0/733159002 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v247) ==== 97+0+0 (secure 0 0 0) 0x7fbb90060e20 con 0x7fbba00b7290 2024-06-29T15:59:05.733 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T15:59:05.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 -- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbb84076510 msgr2=0x7fbb84078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbb84076510 0x7fbb84078a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0058c0 tx=0x7fbb9c005f60 comp rx=0 tx=0).stop 2024-06-29T15:59:05.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 -- 172.21.15.81:0/733159002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbba00b7290 msgr2=0x7fbba014e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbba00b7290 0x7fbba014e100 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fbb9000c970 tx=0x7fbb9000ce30 comp rx=0 tx=0).stop 2024-06-29T15:59:05.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 -- 172.21.15.81:0/733159002 shutdown_connections 2024-06-29T15:59:05.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbb84076510 0x7fbb84078a20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbba00b9040 0x7fbba0150a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbba00b7290 0x7fbba014e100 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 --2- 172.21.15.81:0/733159002 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbba00b5250 0x7fbba014fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 -- 172.21.15.81:0/733159002 >> 172.21.15.81:0/733159002 conn(0x7fbba009cf30 msgr2=0x7fbba01613e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:05.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 -- 172.21.15.81:0/733159002 shutdown_connections 2024-06-29T15:59:05.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.736+0000 7fbba85c7640 1 -- 172.21.15.81:0/733159002 wait complete. 2024-06-29T15:59:05.749 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:59:05.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.862+0000 7f185d9be640 1 -- 172.21.15.81:0/1121780178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18580cccf0 msgr2=0x7f18580d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.862+0000 7f185d9be640 1 --2- 172.21.15.81:0/1121780178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18580cccf0 0x7f18580d0230 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f1858073550 tx=0x7f184c02a4d0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.862+0000 7f185d9be640 1 -- 172.21.15.81:0/1121780178 shutdown_connections 2024-06-29T15:59:05.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.862+0000 7f185d9be640 1 --2- 172.21.15.81:0/1121780178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18580cccf0 0x7f18580d0230 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.862+0000 7f185d9be640 1 --2- 172.21.15.81:0/1121780178 >> v2:172.21.15.167:3300/0 conn(0x7f18580b3200 0x7f18580cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.862+0000 7f185d9be640 1 -- 172.21.15.81:0/1121780178 >> 172.21.15.81:0/1121780178 conn(0x7f18580b3680 msgr2=0x7f18580b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:05.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.862+0000 7f185d9be640 1 -- 172.21.15.81:0/1121780178 shutdown_connections 2024-06-29T15:59:05.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.862+0000 7f185d9be640 1 -- 172.21.15.81:0/1121780178 wait complete. 2024-06-29T15:59:05.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.864+0000 7f185d9be640 1 Processor -- start 2024-06-29T15:59:05.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.864+0000 7f185d9be640 1 -- start start 2024-06-29T15:59:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.865+0000 7f185d9be640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18580b3200 0x7f18581e03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.865+0000 7f185d9be640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18580cccf0 0x7f18580af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.865+0000 7f185d9be640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18580d0770 0x7f18580afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.865+0000 7f185d9be640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f18580cedb0 con 0x7f18580b3200 2024-06-29T15:59:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.865+0000 7f185d9be640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f18580cec30 con 0x7f18580d0770 2024-06-29T15:59:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.865+0000 7f185d9be640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f18580cef30 con 0x7f18580cccf0 2024-06-29T15:59:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.865+0000 7f1856ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18580b3200 0x7f18581e03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.865+0000 7f1856ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18580b3200 0x7f18581e03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57620/0 (socket says 172.21.15.81:57620) 2024-06-29T15:59:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.865+0000 7f1856ffd640 1 -- 172.21.15.81:0/14651286 learned_addr learned my addr 172.21.15.81:0/14651286 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:05.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.865+0000 7f18567fc640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18580cccf0 0x7f18580af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f18577fe640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18580d0770 0x7f18580afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f1856ffd640 1 -- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18580cccf0 msgr2=0x7f18580af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f1856ffd640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18580cccf0 0x7f18580af610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f1856ffd640 1 -- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18580d0770 msgr2=0x7f18580afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f1856ffd640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18580d0770 0x7f18580afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f1856ffd640 1 -- 172.21.15.81:0/14651286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18580b19a0 con 0x7f18580b3200 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f18567fc640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18580cccf0 0x7f18580af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f18577fe640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18580d0770 0x7f18580afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f1856ffd640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18580b3200 0x7f18581e03c0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f184800ea20 tx=0x7f184800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f183ffff640 1 -- 172.21.15.81:0/14651286 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184800cde0 con 0x7f18580b3200 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f183ffff640 1 -- 172.21.15.81:0/14651286 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1848004c10 con 0x7f18580b3200 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f185d9be640 1 -- 172.21.15.81:0/14651286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18580b1c80 con 0x7f18580b3200 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f183ffff640 1 -- 172.21.15.81:0/14651286 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848005020 con 0x7f18580b3200 2024-06-29T15:59:05.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.866+0000 7f185d9be640 1 -- 172.21.15.81:0/14651286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18580b2240 con 0x7f18580b3200 2024-06-29T15:59:05.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.867+0000 7f185d9be640 1 -- 172.21.15.81:0/14651286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18580b14c0 con 0x7f18580b3200 2024-06-29T15:59:05.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.868+0000 7f183ffff640 1 -- 172.21.15.81:0/14651286 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f18480051c0 con 0x7f18580b3200 2024-06-29T15:59:05.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.868+0000 7f183ffff640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1844076510 0x7f1844078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:05.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.868+0000 7f183ffff640 1 -- 172.21.15.81:0/14651286 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(249..249 src has 1..249) ==== 4985+0+0 (secure 0 0 0) 0x7f18480982e0 con 0x7f18580b3200 2024-06-29T15:59:05.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.868+0000 7f18567fc640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1844076510 0x7f1844078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:05.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.869+0000 7f18567fc640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1844076510 0x7f1844078a20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f184c033000 tx=0x7f184c004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:05.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:05.873+0000 7f183ffff640 1 -- 172.21.15.81:0/14651286 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f18580b14c0 con 0x7f18580b3200 2024-06-29T15:59:06.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.120+0000 7f185d9be640 1 -- 172.21.15.81:0/14651286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f18580cc7b0 con 0x7f18580b3200 2024-06-29T15:59:06.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.121+0000 7f183ffff640 1 -- 172.21.15.81:0/14651286 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v69) ==== 80+0+5304 (secure 0 0 0) 0x7f1848016030 con 0x7f18580b3200 2024-06-29T15:59:06.121 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.122 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T15:59:06.123 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:59:06.124 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T15:59:06.125 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.126 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.127 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T15:59:06.128 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.129 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.130 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:59:06.131 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.132 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:59:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:59:06.134 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.134 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.134 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:06.134 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:06.134 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T15:59:06.134 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T15:59:06.134 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:06.134 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:59:06.134 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:59:06.135 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:06.135 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.135 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:06.135 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T15:59:06.135 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:59:06.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 -- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1844076510 msgr2=0x7f1844078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1844076510 0x7f1844078a20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f184c033000 tx=0x7f184c004730 comp rx=0 tx=0).stop 2024-06-29T15:59:06.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 -- 172.21.15.81:0/14651286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18580b3200 msgr2=0x7f18581e03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18580b3200 0x7f18581e03c0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f184800ea20 tx=0x7f184800eee0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 -- 172.21.15.81:0/14651286 shutdown_connections 2024-06-29T15:59:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f1844076510 0x7f1844078a20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f18580d0770 0x7f18580afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f18580cccf0 0x7f18580af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 --2- 172.21.15.81:0/14651286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18580b3200 0x7f18581e03c0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 -- 172.21.15.81:0/14651286 >> 172.21.15.81:0/14651286 conn(0x7f18580b3680 msgr2=0x7f18581e8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 -- 172.21.15.81:0/14651286 shutdown_connections 2024-06-29T15:59:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.123+0000 7f185d9be640 1 -- 172.21.15.81:0/14651286 wait complete. 2024-06-29T15:59:06.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:06.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.250+0000 7f70cb69e640 1 -- 172.21.15.81:0/3079147937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c40b4f40 msgr2=0x7f70c40b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.250+0000 7f70cb69e640 1 --2- 172.21.15.81:0/3079147937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c40b4f40 0x7f70c40b5350 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009830 tx=0x7f70b402a510 comp rx=0 tx=0).stop 2024-06-29T15:59:06.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.250+0000 7f70cb69e640 1 -- 172.21.15.81:0/3079147937 shutdown_connections 2024-06-29T15:59:06.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.250+0000 7f70cb69e640 1 --2- 172.21.15.81:0/3079147937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c40b4f40 0x7f70c40b5350 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.250+0000 7f70cb69e640 1 --2- 172.21.15.81:0/3079147937 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c40d0230 0x7f70c40d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.250+0000 7f70cb69e640 1 -- 172.21.15.81:0/3079147937 >> 172.21.15.81:0/3079147937 conn(0x7f70c40b3680 msgr2=0x7f70c40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:06.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.250+0000 7f70cb69e640 1 -- 172.21.15.81:0/3079147937 shutdown_connections 2024-06-29T15:59:06.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.250+0000 7f70cb69e640 1 -- 172.21.15.81:0/3079147937 wait complete. 2024-06-29T15:59:06.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.252+0000 7f70cb69e640 1 Processor -- start 2024-06-29T15:59:06.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.252+0000 7f70cb69e640 1 -- start start 2024-06-29T15:59:06.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.253+0000 7f70cb69e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70c40b3200 0x7f70c409bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.253+0000 7f70cb69e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c40b4f40 0x7f70c41e08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.253+0000 7f70cb69e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c40d0230 0x7f70c409e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.253+0000 7f70cb69e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f70c40b5c20 con 0x7f70c40d0230 2024-06-29T15:59:06.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.253+0000 7f70cb69e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f70c40b5aa0 con 0x7f70c40b3200 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.253+0000 7f70cb69e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f70c40b5da0 con 0x7f70c40b4f40 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c9c3b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c40d0230 0x7f70c409e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c9c3b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c40d0230 0x7f70c409e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57632/0 (socket says 172.21.15.81:57632) 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c9c3b640 1 -- 172.21.15.81:0/3723349284 learned_addr learned my addr 172.21.15.81:0/3723349284 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c8c39640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c40b4f40 0x7f70c41e08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c943a640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70c40b3200 0x7f70c409bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c9c3b640 1 -- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c40b4f40 msgr2=0x7f70c41e08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c9c3b640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c40b4f40 0x7f70c41e08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c9c3b640 1 -- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70c40b3200 msgr2=0x7f70c409bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c9c3b640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70c40b3200 0x7f70c409bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c9c3b640 1 -- 172.21.15.81:0/3723349284 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c40ca180 con 0x7f70c40d0230 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c8c39640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c40b4f40 0x7f70c41e08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c9c3b640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c40d0230 0x7f70c409e4a0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009920 tx=0x7f70b4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c1ffb640 1 -- 172.21.15.81:0/3723349284 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4031d60 con 0x7f70c40d0230 2024-06-29T15:59:06.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c1ffb640 1 -- 172.21.15.81:0/3723349284 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70b4033e40 con 0x7f70c40d0230 2024-06-29T15:59:06.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70cb69e640 1 -- 172.21.15.81:0/3723349284 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c40ca420 con 0x7f70c40d0230 2024-06-29T15:59:06.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70c1ffb640 1 -- 172.21.15.81:0/3723349284 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4005240 con 0x7f70c40d0230 2024-06-29T15:59:06.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.254+0000 7f70cb69e640 1 -- 172.21.15.81:0/3723349284 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c40ca9e0 con 0x7f70c40d0230 2024-06-29T15:59:06.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.256+0000 7f70cb69e640 1 -- 172.21.15.81:0/3723349284 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c40b5350 con 0x7f70c40d0230 2024-06-29T15:59:06.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.256+0000 7f70c1ffb640 1 -- 172.21.15.81:0/3723349284 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f70b40053e0 con 0x7f70c40d0230 2024-06-29T15:59:06.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.257+0000 7f70c1ffb640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f70b0076510 0x7f70b0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.257+0000 7f70c1ffb640 1 -- 172.21.15.81:0/3723349284 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(249..249 src has 1..249) ==== 4985+0+0 (secure 0 0 0) 0x7f70b40b8b60 con 0x7f70c40d0230 2024-06-29T15:59:06.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.257+0000 7f70c943a640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f70b0076510 0x7f70b0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.258+0000 7f70c943a640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f70b0076510 0x7f70b0078a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f70bc007fd0 tx=0x7f70bc009040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:06.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.261+0000 7f70c1ffb640 1 -- 172.21.15.81:0/3723349284 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f70b403f030 con 0x7f70c40d0230 2024-06-29T15:59:06.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.440+0000 7f70cb69e640 1 -- 172.21.15.81:0/3723349284 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70c40cf080 con 0x7f70c40d0230 2024-06-29T15:59:06.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.440+0000 7f70c1ffb640 1 -- 172.21.15.81:0/3723349284 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) ==== 74+0+13093 (secure 0 0 0) 0x7f70b4082360 con 0x7f70c40d0230 2024-06-29T15:59:06.441 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:06.441 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":249,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:05.744489+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":235,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:06.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 -- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f70b0076510 msgr2=0x7f70b0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f70b0076510 0x7f70b0078a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f70bc007fd0 tx=0x7f70bc009040 comp rx=0 tx=0).stop 2024-06-29T15:59:06.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 -- 172.21.15.81:0/3723349284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c40d0230 msgr2=0x7f70c409e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c40d0230 0x7f70c409e4a0 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009920 tx=0x7f70b4033c20 comp rx=0 tx=0).stop 2024-06-29T15:59:06.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 -- 172.21.15.81:0/3723349284 shutdown_connections 2024-06-29T15:59:06.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f70b0076510 0x7f70b0078a20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c40d0230 0x7f70c409e4a0 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70c40b4f40 0x7f70c41e08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 --2- 172.21.15.81:0/3723349284 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70c40b3200 0x7f70c409bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 -- 172.21.15.81:0/3723349284 >> 172.21.15.81:0/3723349284 conn(0x7f70c40b3680 msgr2=0x7f70c40701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:06.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 -- 172.21.15.81:0/3723349284 shutdown_connections 2024-06-29T15:59:06.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.443+0000 7f70cb69e640 1 -- 172.21.15.81:0/3723349284 wait complete. 2024-06-29T15:59:06.455 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:06.456 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:06.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.573+0000 7f444d37b640 1 -- 172.21.15.81:0/3172067889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4434004050 msgr2=0x7f4434004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.573+0000 7f444d37b640 1 --2- 172.21.15.81:0/3172067889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4434004050 0x7f4434004800 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7f44480cac90 tx=0x7f4434005830 comp rx=0 tx=0).stop 2024-06-29T15:59:06.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.573+0000 7f44477fe640 1 -- 172.21.15.81:0/3172067889 reap_dead start 2024-06-29T15:59:06.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.573+0000 7f444d37b640 1 -- 172.21.15.81:0/3172067889 shutdown_connections 2024-06-29T15:59:06.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.573+0000 7f444d37b640 1 -- 172.21.15.81:0/3172067889 >> 172.21.15.81:0/3172067889 conn(0x7f444809cf30 msgr2=0x7f44480d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:06.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.573+0000 7f444d37b640 1 -- 172.21.15.81:0/3172067889 shutdown_connections 2024-06-29T15:59:06.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.573+0000 7f444d37b640 1 -- 172.21.15.81:0/3172067889 wait complete. 2024-06-29T15:59:06.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.575+0000 7f444d37b640 1 Processor -- start 2024-06-29T15:59:06.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.575+0000 7f444d37b640 1 -- start start 2024-06-29T15:59:06.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f444d37b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4434004050 0x7f44480b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f444d37b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44480b0d70 0x7f44480b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f444d37b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44480b16c0 0x7f44481f4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f444d37b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f44480b8490 con 0x7f44480b16c0 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f444d37b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f44480b8310 con 0x7f44480b0d70 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f444d37b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f44480d0480 con 0x7f4434004050 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f44477fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44480b16c0 0x7f44481f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f44477fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44480b16c0 0x7f44481f4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57638/0 (socket says 172.21.15.81:57638) 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f44477fe640 1 -- 172.21.15.81:0/6714409 learned_addr learned my addr 172.21.15.81:0/6714409 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f443e7fc640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44480b0d70 0x7f44480b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f4446ffd640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4434004050 0x7f44480b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f44477fe640 1 -- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4434004050 msgr2=0x7f44480b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f44477fe640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4434004050 0x7f44480b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f44477fe640 1 -- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44480b0d70 msgr2=0x7f44480b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f44477fe640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44480b0d70 0x7f44480b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f44477fe640 1 -- 172.21.15.81:0/6714409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44480ce020 con 0x7f44480b16c0 2024-06-29T15:59:06.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f44477fe640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44480b16c0 0x7f44481f4660 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f4440002230 tx=0x7f444000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f443ffff640 1 -- 172.21.15.81:0/6714409 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440001ab0 con 0x7f44480b16c0 2024-06-29T15:59:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f443ffff640 1 -- 172.21.15.81:0/6714409 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4440018070 con 0x7f44480b16c0 2024-06-29T15:59:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f444d37b640 1 -- 172.21.15.81:0/6714409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44480a09a0 con 0x7f44480b16c0 2024-06-29T15:59:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f443ffff640 1 -- 172.21.15.81:0/6714409 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440011460 con 0x7f44480b16c0 2024-06-29T15:59:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.577+0000 7f444d37b640 1 -- 172.21.15.81:0/6714409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44480a0f60 con 0x7f44480b16c0 2024-06-29T15:59:06.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.578+0000 7f444d37b640 1 -- 172.21.15.81:0/6714409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44480d09e0 con 0x7f44480b16c0 2024-06-29T15:59:06.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.579+0000 7f443ffff640 1 -- 172.21.15.81:0/6714409 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f4440011600 con 0x7f44480b16c0 2024-06-29T15:59:06.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.580+0000 7f443ffff640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4428076510 0x7f4428078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.580+0000 7f443ffff640 1 -- 172.21.15.81:0/6714409 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(249..249 src has 1..249) ==== 4985+0+0 (secure 0 0 0) 0x7f4440097e30 con 0x7f44480b16c0 2024-06-29T15:59:06.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.580+0000 7f4446ffd640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4428076510 0x7f4428078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.580+0000 7f4446ffd640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4428076510 0x7f4428078a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f44380099e0 tx=0x7f4438005f90 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:06.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.584+0000 7f443ffff640 1 -- 172.21.15.81:0/6714409 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f44400615f0 con 0x7f44480b16c0 2024-06-29T15:59:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.765+0000 7f444d37b640 1 -- 172.21.15.81:0/6714409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f44480a0730 con 0x7f44480b16c0 2024-06-29T15:59:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.765+0000 7f443ffff640 1 -- 172.21.15.81:0/6714409 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v249) ==== 93+0+10 (secure 0 0 0) 0x7f4440001ef0 con 0x7f44480b16c0 2024-06-29T15:59:06.766 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:59:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 -- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4428076510 msgr2=0x7f4428078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4428076510 0x7f4428078a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f44380099e0 tx=0x7f4438005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 -- 172.21.15.81:0/6714409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44480b16c0 msgr2=0x7f44481f4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44480b16c0 0x7f44481f4660 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f4440002230 tx=0x7f444000ed90 comp rx=0 tx=0).stop 2024-06-29T15:59:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 -- 172.21.15.81:0/6714409 shutdown_connections 2024-06-29T15:59:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f4428076510 0x7f4428078a20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44480b16c0 0x7f44481f4660 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f44480b0d70 0x7f44480b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 --2- 172.21.15.81:0/6714409 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4434004050 0x7f44480b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 -- 172.21.15.81:0/6714409 >> 172.21.15.81:0/6714409 conn(0x7f444809cf30 msgr2=0x7f4448068440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:06.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 -- 172.21.15.81:0/6714409 shutdown_connections 2024-06-29T15:59:06.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.767+0000 7f444d37b640 1 -- 172.21.15.81:0/6714409 wait complete. 2024-06-29T15:59:06.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:59:06.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.893+0000 7f770ac9e640 1 -- 172.21.15.81:0/4272006621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f770406f850 msgr2=0x7f77040ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.893+0000 7f770ac9e640 1 --2- 172.21.15.81:0/4272006621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f770406f850 0x7f77040ae300 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009830 tx=0x7f76f402a510 comp rx=0 tx=0).stop 2024-06-29T15:59:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.893+0000 7f770ac9e640 1 -- 172.21.15.81:0/4272006621 shutdown_connections 2024-06-29T15:59:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.893+0000 7f770ac9e640 1 --2- 172.21.15.81:0/4272006621 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f77040ae840 0x7f77040aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.893+0000 7f770ac9e640 1 --2- 172.21.15.81:0/4272006621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f770406f850 0x7f77040ae300 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.893+0000 7f770ac9e640 1 -- 172.21.15.81:0/4272006621 >> 172.21.15.81:0/4272006621 conn(0x7f7704072950 msgr2=0x7f77040b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.893+0000 7f770ac9e640 1 -- 172.21.15.81:0/4272006621 shutdown_connections 2024-06-29T15:59:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.893+0000 7f770ac9e640 1 -- 172.21.15.81:0/4272006621 wait complete. 2024-06-29T15:59:06.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.895+0000 7f770ac9e640 1 Processor -- start 2024-06-29T15:59:06.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.895+0000 7f770ac9e640 1 -- start start 2024-06-29T15:59:06.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.896+0000 7f770ac9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f770406f850 0x7f77041483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.896+0000 7f770ac9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7704074520 0x7f770415c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.896+0000 7f770ac9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77040ae840 0x7f770415ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.896+0000 7f770ac9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f770415b190 con 0x7f7704074520 2024-06-29T15:59:06.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.896+0000 7f770ac9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f770415b010 con 0x7f77040ae840 2024-06-29T15:59:06.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.896+0000 7f770ac9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f77040af190 con 0x7f770406f850 2024-06-29T15:59:06.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.896+0000 7f76fbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7704074520 0x7f770415c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.896+0000 7f76fbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7704074520 0x7f770415c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57654/0 (socket says 172.21.15.81:57654) 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.896+0000 7f76fbfff640 1 -- 172.21.15.81:0/2700673107 learned_addr learned my addr 172.21.15.81:0/2700673107 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.896+0000 7f7708a3a640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f770406f850 0x7f77041483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f770923b640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77040ae840 0x7f770415ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f76fbfff640 1 -- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f770406f850 msgr2=0x7f77041483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f76fbfff640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f770406f850 0x7f77041483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f76fbfff640 1 -- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77040ae840 msgr2=0x7f770415ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f76fbfff640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77040ae840 0x7f770415ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f76fbfff640 1 -- 172.21.15.81:0/2700673107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770415e200 con 0x7f7704074520 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f7708a3a640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f770406f850 0x7f77041483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f76fbfff640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7704074520 0x7f770415c990 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f76f402ab00 tx=0x7f76f4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f76f97fa640 1 -- 172.21.15.81:0/2700673107 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f4005c90 con 0x7f7704074520 2024-06-29T15:59:06.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f76f97fa640 1 -- 172.21.15.81:0/2700673107 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f76f4031e50 con 0x7f7704074520 2024-06-29T15:59:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f770ac9e640 1 -- 172.21.15.81:0/2700673107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770409b7c0 con 0x7f7704074520 2024-06-29T15:59:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f76f97fa640 1 -- 172.21.15.81:0/2700673107 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f4005260 con 0x7f7704074520 2024-06-29T15:59:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.897+0000 7f770ac9e640 1 -- 172.21.15.81:0/2700673107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770409bd80 con 0x7f7704074520 2024-06-29T15:59:06.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.898+0000 7f770ac9e640 1 -- 172.21.15.81:0/2700673107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f770409b530 con 0x7f7704074520 2024-06-29T15:59:06.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.899+0000 7f76f97fa640 1 -- 172.21.15.81:0/2700673107 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f76f4005400 con 0x7f7704074520 2024-06-29T15:59:06.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.899+0000 7f76f97fa640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f76f0076540 0x7f76f0078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:06.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.900+0000 7f76f97fa640 1 -- 172.21.15.81:0/2700673107 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(249..249 src has 1..249) ==== 4985+0+0 (secure 0 0 0) 0x7f76f40b8930 con 0x7f7704074520 2024-06-29T15:59:06.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.900+0000 7f7708a3a640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f76f0076540 0x7f76f0078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:06.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.900+0000 7f7708a3a640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f76f0076540 0x7f76f0078a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7700007a50 tx=0x7f7700009040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:06.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:06.904+0000 7f76f97fa640 1 -- 172.21.15.81:0/2700673107 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f76f403f030 con 0x7f7704074520 2024-06-29T15:59:07.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.088+0000 7f770ac9e640 1 -- 172.21.15.81:0/2700673107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f77040ae300 con 0x7f7704074520 2024-06-29T15:59:07.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.088+0000 7f76f97fa640 1 -- 172.21.15.81:0/2700673107 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v249) ==== 84+0+65 (secure 0 0 0) 0x7f76f408a1b0 con 0x7f7704074520 2024-06-29T15:59:07.089 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:07.089 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:59:07.089 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:07.089 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T15:59:07.089 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T15:59:07.089 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:07.090 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:59:07.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.090+0000 7f770ac9e640 1 -- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f76f0076540 msgr2=0x7f76f0078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.090+0000 7f770ac9e640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f76f0076540 0x7f76f0078a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7700007a50 tx=0x7f7700009040 comp rx=0 tx=0).stop 2024-06-29T15:59:07.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.091+0000 7f770ac9e640 1 -- 172.21.15.81:0/2700673107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7704074520 msgr2=0x7f770415c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.091+0000 7f770ac9e640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7704074520 0x7f770415c990 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f76f402ab00 tx=0x7f76f4033c20 comp rx=0 tx=0).stop 2024-06-29T15:59:07.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.091+0000 7f770ac9e640 1 -- 172.21.15.81:0/2700673107 shutdown_connections 2024-06-29T15:59:07.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.091+0000 7f770ac9e640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f76f0076540 0x7f76f0078a50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.091+0000 7f770ac9e640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f77040ae840 0x7f770415ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.091+0000 7f770ac9e640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7704074520 0x7f770415c990 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.091+0000 7f770ac9e640 1 --2- 172.21.15.81:0/2700673107 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f770406f850 0x7f77041483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.091+0000 7f770ac9e640 1 -- 172.21.15.81:0/2700673107 >> 172.21.15.81:0/2700673107 conn(0x7f7704072950 msgr2=0x7f77040b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:07.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.091+0000 7f770ac9e640 1 -- 172.21.15.81:0/2700673107 shutdown_connections 2024-06-29T15:59:07.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.091+0000 7f770ac9e640 1 -- 172.21.15.81:0/2700673107 wait complete. 2024-06-29T15:59:07.102 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:59:07.102 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:59:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.219+0000 7f752dbe0640 1 -- 172.21.15.81:0/4123480470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75280affa0 msgr2=0x7f7528072ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.219+0000 7f752dbe0640 1 --2- 172.21.15.81:0/4123480470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75280affa0 0x7f7528072ef0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7f75280ccc00 tx=0x7f751c02a560 comp rx=0 tx=0).stop 2024-06-29T15:59:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.219+0000 7f752dbe0640 1 -- 172.21.15.81:0/4123480470 shutdown_connections 2024-06-29T15:59:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.219+0000 7f752dbe0640 1 --2- 172.21.15.81:0/4123480470 >> v2:172.21.15.167:3300/0 conn(0x7f7528073430 0x7f752806f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.219+0000 7f752dbe0640 1 --2- 172.21.15.81:0/4123480470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75280affa0 0x7f7528072ef0 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.219+0000 7f752dbe0640 1 -- 172.21.15.81:0/4123480470 >> 172.21.15.81:0/4123480470 conn(0x7f75280b2580 msgr2=0x7f75280b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.219+0000 7f752dbe0640 1 -- 172.21.15.81:0/4123480470 shutdown_connections 2024-06-29T15:59:07.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.219+0000 7f752dbe0640 1 -- 172.21.15.81:0/4123480470 wait complete. 2024-06-29T15:59:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.221+0000 7f752dbe0640 1 Processor -- start 2024-06-29T15:59:07.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.221+0000 7f752dbe0640 1 -- start start 2024-06-29T15:59:07.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f752dbe0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7528073430 0x7f75281432e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f752dbe0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75280affa0 0x7f75280d0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f752dbe0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75280b5a60 0x7f75281f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f752dbe0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f752806f990 con 0x7f75280b5a60 2024-06-29T15:59:07.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f752dbe0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f752806f810 con 0x7f75280affa0 2024-06-29T15:59:07.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f752dbe0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f752806fb10 con 0x7f7528073430 2024-06-29T15:59:07.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f7527fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75280b5a60 0x7f75281f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f7527fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75280b5a60 0x7f75281f4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57662/0 (socket says 172.21.15.81:57662) 2024-06-29T15:59:07.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f7527fff640 1 -- 172.21.15.81:0/3293004178 learned_addr learned my addr 172.21.15.81:0/3293004178 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f75277fe640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7528073430 0x7f75281432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f7526ffd640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75280affa0 0x7f75280d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f7527fff640 1 -- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7528073430 msgr2=0x7f75281432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f7527fff640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7528073430 0x7f75281432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f7527fff640 1 -- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75280affa0 msgr2=0x7f75280d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f7527fff640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75280affa0 0x7f75280d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f7527fff640 1 -- 172.21.15.81:0/3293004178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7528143820 con 0x7f75280b5a60 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f75277fe640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7528073430 0x7f75281432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f7527fff640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75280b5a60 0x7f75281f4dd0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f752000e990 tx=0x7f752000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f750ffff640 1 -- 172.21.15.81:0/3293004178 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520004d20 con 0x7f75280b5a60 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f750ffff640 1 -- 172.21.15.81:0/3293004178 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f752000cb00 con 0x7f75280b5a60 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f752dbe0640 1 -- 172.21.15.81:0/3293004178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75280d0560 con 0x7f75280b5a60 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f750ffff640 1 -- 172.21.15.81:0/3293004178 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520011570 con 0x7f75280b5a60 2024-06-29T15:59:07.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.223+0000 7f752dbe0640 1 -- 172.21.15.81:0/3293004178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f752815c580 con 0x7f75280b5a60 2024-06-29T15:59:07.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.224+0000 7f752dbe0640 1 -- 172.21.15.81:0/3293004178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7528070080 con 0x7f75280b5a60 2024-06-29T15:59:07.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.225+0000 7f750ffff640 1 -- 172.21.15.81:0/3293004178 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f7520011710 con 0x7f75280b5a60 2024-06-29T15:59:07.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.226+0000 7f750ffff640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7514076510 0x7f7514078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.226+0000 7f750ffff640 1 -- 172.21.15.81:0/3293004178 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(249..249 src has 1..249) ==== 4985+0+0 (secure 0 0 0) 0x7f752009a7e0 con 0x7f75280b5a60 2024-06-29T15:59:07.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.226+0000 7f75277fe640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7514076510 0x7f7514078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.226+0000 7f75277fe640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7514076510 0x7f7514078a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7518006fd0 tx=0x7f7518008040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:07.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.229+0000 7f750ffff640 1 -- 172.21.15.81:0/3293004178 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7520064100 con 0x7f75280b5a60 2024-06-29T15:59:07.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.409+0000 7f752dbe0640 1 -- 172.21.15.81:0/3293004178 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f75280cd180 con 0x7f75280b5a60 2024-06-29T15:59:07.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.750+0000 7f750ffff640 1 -- 172.21.15.81:0/3293004178 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v250) ==== 138+0+0 (secure 0 0 0) 0x7f752006c180 con 0x7f75280b5a60 2024-06-29T15:59:07.750 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 -- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7514076510 msgr2=0x7f7514078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7514076510 0x7f7514078a20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7518006fd0 tx=0x7f7518008040 comp rx=0 tx=0).stop 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 -- 172.21.15.81:0/3293004178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75280b5a60 msgr2=0x7f75281f4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75280b5a60 0x7f75281f4dd0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f752000e990 tx=0x7f752000ee50 comp rx=0 tx=0).stop 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 -- 172.21.15.81:0/3293004178 shutdown_connections 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7514076510 0x7f7514078a20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75280b5a60 0x7f75281f4dd0 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f75280affa0 0x7f75280d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 --2- 172.21.15.81:0/3293004178 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7528073430 0x7f75281432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 -- 172.21.15.81:0/3293004178 >> 172.21.15.81:0/3293004178 conn(0x7f75280b2580 msgr2=0x7f752815ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:07.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 -- 172.21.15.81:0/3293004178 shutdown_connections 2024-06-29T15:59:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.752+0000 7f752dbe0640 1 -- 172.21.15.81:0/3293004178 wait complete. 2024-06-29T15:59:07.767 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:59:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.889+0000 7ff175290640 1 -- 172.21.15.81:0/2526705427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1700d0b90 msgr2=0x7ff1700d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.889+0000 7ff175290640 1 --2- 172.21.15.81:0/2526705427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1700d0b90 0x7ff1700d1a80 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7ff158009830 tx=0x7ff15802a5a0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.889+0000 7ff175290640 1 -- 172.21.15.81:0/2526705427 shutdown_connections 2024-06-29T15:59:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.889+0000 7ff175290640 1 --2- 172.21.15.81:0/2526705427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1700d0b90 0x7ff1700d1a80 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.889+0000 7ff175290640 1 -- 172.21.15.81:0/2526705427 >> 172.21.15.81:0/2526705427 conn(0x7ff17009cf30 msgr2=0x7ff1700d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.889+0000 7ff175290640 1 -- 172.21.15.81:0/2526705427 shutdown_connections 2024-06-29T15:59:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.889+0000 7ff175290640 1 -- 172.21.15.81:0/2526705427 wait complete. 2024-06-29T15:59:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.891+0000 7ff175290640 1 Processor -- start 2024-06-29T15:59:07.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.891+0000 7ff175290640 1 -- start start 2024-06-29T15:59:07.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff175290640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1700b5250 0x7ff170158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff175290640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1700b9040 0x7ff17015be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff175290640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff170155f40 0x7ff170156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff175290640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff1700d26d0 con 0x7ff170155f40 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff175290640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff1700b85c0 con 0x7ff1700b5250 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff175290640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff1700d2850 con 0x7ff1700b9040 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff16f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff170155f40 0x7ff170156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff16f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff170155f40 0x7ff170156350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57676/0 (socket says 172.21.15.81:57676) 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff16f7fe640 1 -- 172.21.15.81:0/1988986851 learned_addr learned my addr 172.21.15.81:0/1988986851 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff1667fc640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1700b9040 0x7ff17015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff16effd640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1700b5250 0x7ff170158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff16f7fe640 1 -- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1700b9040 msgr2=0x7ff17015be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff16f7fe640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1700b9040 0x7ff17015be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff16f7fe640 1 -- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1700b5250 msgr2=0x7ff170158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:07.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.892+0000 7ff16f7fe640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1700b5250 0x7ff170158ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:07.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.893+0000 7ff16f7fe640 1 -- 172.21.15.81:0/1988986851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17015ad60 con 0x7ff170155f40 2024-06-29T15:59:07.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.893+0000 7ff16f7fe640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff170155f40 0x7ff170156350 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7ff16800be90 tx=0x7ff16800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:07.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.893+0000 7ff167fff640 1 -- 172.21.15.81:0/1988986851 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff168019070 con 0x7ff170155f40 2024-06-29T15:59:07.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.893+0000 7ff167fff640 1 -- 172.21.15.81:0/1988986851 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff168014440 con 0x7ff170155f40 2024-06-29T15:59:07.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.893+0000 7ff167fff640 1 -- 172.21.15.81:0/1988986851 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1680148d0 con 0x7ff170155f40 2024-06-29T15:59:07.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.893+0000 7ff175290640 1 -- 172.21.15.81:0/1988986851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff17015b040 con 0x7ff170155f40 2024-06-29T15:59:07.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.893+0000 7ff175290640 1 -- 172.21.15.81:0/1988986851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff17015b600 con 0x7ff170155f40 2024-06-29T15:59:07.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.894+0000 7ff175290640 1 -- 172.21.15.81:0/1988986851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1700d2de0 con 0x7ff170155f40 2024-06-29T15:59:07.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.895+0000 7ff167fff640 1 -- 172.21.15.81:0/1988986851 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7ff16801c070 con 0x7ff170155f40 2024-06-29T15:59:07.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.895+0000 7ff167fff640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff150076540 0x7ff150078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:07.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.896+0000 7ff167fff640 1 -- 172.21.15.81:0/1988986851 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5355+0+0 (secure 0 0 0) 0x7ff168097f60 con 0x7ff170155f40 2024-06-29T15:59:07.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.896+0000 7ff16effd640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff150076540 0x7ff150078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:07.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.896+0000 7ff16effd640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff150076540 0x7ff150078a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff160007fd0 tx=0x7ff160009040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:07.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:07.900+0000 7ff167fff640 1 -- 172.21.15.81:0/1988986851 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff168061630 con 0x7ff170155f40 2024-06-29T15:59:08.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.081+0000 7ff175290640 1 -- 172.21.15.81:0/1988986851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff17015d130 con 0x7ff170155f40 2024-06-29T15:59:08.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.752+0000 7ff167fff640 1 -- 172.21.15.81:0/1988986851 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v251) ==== 170+0+0 (secure 0 0 0) 0x7ff168060f30 con 0x7ff170155f40 2024-06-29T15:59:08.753 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T15:59:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 -- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff150076540 msgr2=0x7ff150078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff150076540 0x7ff150078a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff160007fd0 tx=0x7ff160009040 comp rx=0 tx=0).stop 2024-06-29T15:59:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 -- 172.21.15.81:0/1988986851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff170155f40 msgr2=0x7ff170156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff170155f40 0x7ff170156350 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7ff16800be90 tx=0x7ff16800bf80 comp rx=0 tx=0).stop 2024-06-29T15:59:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 -- 172.21.15.81:0/1988986851 shutdown_connections 2024-06-29T15:59:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff150076540 0x7ff150078a50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff170155f40 0x7ff170156350 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff1700b9040 0x7ff17015be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 --2- 172.21.15.81:0/1988986851 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff1700b5250 0x7ff170158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 -- 172.21.15.81:0/1988986851 >> 172.21.15.81:0/1988986851 conn(0x7ff17009cf30 msgr2=0x7ff170161060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:08.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 -- 172.21.15.81:0/1988986851 shutdown_connections 2024-06-29T15:59:08.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.755+0000 7ff175290640 1 -- 172.21.15.81:0/1988986851 wait complete. 2024-06-29T15:59:08.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:59:08.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.893+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/1097262080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc0ae840 msgr2=0x7f5edc0aec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.893+0000 7f5ee1f9e640 1 --2- 172.21.15.81:0/1097262080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc0ae840 0x7f5edc0aec50 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4005fd0 tx=0x7f5ed402a530 comp rx=0 tx=0).stop 2024-06-29T15:59:08.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.893+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/1097262080 shutdown_connections 2024-06-29T15:59:08.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.893+0000 7f5ee1f9e640 1 --2- 172.21.15.81:0/1097262080 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc06f850 0x7f5edc0ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.893+0000 7f5ee1f9e640 1 --2- 172.21.15.81:0/1097262080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc0ae840 0x7f5edc0aec50 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.893+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/1097262080 >> 172.21.15.81:0/1097262080 conn(0x7f5edc072950 msgr2=0x7f5edc0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:08.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.893+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/1097262080 shutdown_connections 2024-06-29T15:59:08.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.893+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/1097262080 wait complete. 2024-06-29T15:59:08.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.895+0000 7f5ee1f9e640 1 Processor -- start 2024-06-29T15:59:08.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.895+0000 7f5ee1f9e640 1 -- start start 2024-06-29T15:59:08.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5ee1f9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc06f850 0x7f5edc15b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5ee1f9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc074520 0x7f5edc1f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5ee1f9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc0ae840 0x7f5edc1f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5ee1f9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc15b190 con 0x7f5edc06f850 2024-06-29T15:59:08.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5ee1f9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc15b010 con 0x7f5edc074520 2024-06-29T15:59:08.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5ee1f9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5edc0af190 con 0x7f5edc0ae840 2024-06-29T15:59:08.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc06f850 0x7f5edc15b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc06f850 0x7f5edc15b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57688/0 (socket says 172.21.15.81:57688) 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edb7fe640 1 -- 172.21.15.81:0/4106671249 learned_addr learned my addr 172.21.15.81:0/4106671249 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edbfff640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc0ae840 0x7f5edc1f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edaffd640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc074520 0x7f5edc1f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edb7fe640 1 -- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc0ae840 msgr2=0x7f5edc1f7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edb7fe640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc0ae840 0x7f5edc1f7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edb7fe640 1 -- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc074520 msgr2=0x7f5edc1f6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edb7fe640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc074520 0x7f5edc1f6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edb7fe640 1 -- 172.21.15.81:0/4106671249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5edc153a90 con 0x7f5edc06f850 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5edb7fe640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc06f850 0x7f5edc15b590 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00ba50 tx=0x7f5ecc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.897+0000 7f5ec3fff640 1 -- 172.21.15.81:0/4106671249 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc019070 con 0x7f5edc06f850 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.898+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/4106671249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5edc1537b0 con 0x7f5edc06f850 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.898+0000 7f5ec3fff640 1 -- 172.21.15.81:0/4106671249 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ecc00cd10 con 0x7f5edc06f850 2024-06-29T15:59:08.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.898+0000 7f5ec3fff640 1 -- 172.21.15.81:0/4106671249 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc0146a0 con 0x7f5edc06f850 2024-06-29T15:59:08.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.898+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/4106671249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5edc14aa10 con 0x7f5edc06f850 2024-06-29T15:59:08.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.899+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/4106671249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5edc0ae300 con 0x7f5edc06f850 2024-06-29T15:59:08.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.900+0000 7f5ec3fff640 1 -- 172.21.15.81:0/4106671249 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f5ecc01c070 con 0x7f5edc06f850 2024-06-29T15:59:08.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.900+0000 7f5ec3fff640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5ec8076510 0x7f5ec8078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:08.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.900+0000 7f5ec3fff640 1 -- 172.21.15.81:0/4106671249 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 5737+0+0 (secure 0 0 0) 0x7f5ecc097a20 con 0x7f5edc06f850 2024-06-29T15:59:08.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.901+0000 7f5edaffd640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5ec8076510 0x7f5ec8078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:08.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.901+0000 7f5edaffd640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5ec8076510 0x7f5ec8078a20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5edc0d1040 tx=0x7f5ed000b2e0 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:08.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:08.905+0000 7f5ec3fff640 1 -- 172.21.15.81:0/4106671249 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5ecc0608a0 con 0x7f5edc06f850 2024-06-29T15:59:09.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.125+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/4106671249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5edc06f1e0 con 0x7f5edc06f850 2024-06-29T15:59:09.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.762+0000 7f5ec3fff640 1 -- 172.21.15.81:0/4106671249 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 34 and data pool 35 v142) ==== 174+0+0 (secure 0 0 0) 0x7f5ecc064ba0 con 0x7f5edc06f850 2024-06-29T15:59:09.764 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 34 and data pool 35 2024-06-29T15:59:09.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5ec8076510 msgr2=0x7f5ec8078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5ec8076510 0x7f5ec8078a20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5edc0d1040 tx=0x7f5ed000b2e0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/4106671249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc06f850 msgr2=0x7f5edc15b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc06f850 0x7f5edc15b590 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00ba50 tx=0x7f5ecc00bf10 comp rx=0 tx=0).stop 2024-06-29T15:59:09.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/4106671249 shutdown_connections 2024-06-29T15:59:09.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5ec8076510 0x7f5ec8078a20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5edc0ae840 0x7f5edc1f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5edc074520 0x7f5edc1f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 --2- 172.21.15.81:0/4106671249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5edc06f850 0x7f5edc15b590 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/4106671249 >> 172.21.15.81:0/4106671249 conn(0x7f5edc072950 msgr2=0x7f5edc14f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:09.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.765+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/4106671249 shutdown_connections 2024-06-29T15:59:09.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.766+0000 7f5ee1f9e640 1 -- 172.21.15.81:0/4106671249 wait complete. 2024-06-29T15:59:09.792 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:09.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.907+0000 7fddce47f640 1 -- 172.21.15.81:0/3431322166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb0003a00 msgr2=0x7fddb0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.907+0000 7fddce47f640 1 --2- 172.21.15.81:0/3431322166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb0003a00 0x7fddb0003e70 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009b50 tx=0x7fddb0005830 comp rx=0 tx=0).stop 2024-06-29T15:59:09.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.907+0000 7fddcca1c640 1 -- 172.21.15.81:0/3431322166 reap_dead start 2024-06-29T15:59:09.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.907+0000 7fddce47f640 1 -- 172.21.15.81:0/3431322166 shutdown_connections 2024-06-29T15:59:09.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.907+0000 7fddce47f640 1 -- 172.21.15.81:0/3431322166 >> 172.21.15.81:0/3431322166 conn(0x7fddc8072950 msgr2=0x7fddc80b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:09.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.907+0000 7fddce47f640 1 -- 172.21.15.81:0/3431322166 shutdown_connections 2024-06-29T15:59:09.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.907+0000 7fddce47f640 1 -- 172.21.15.81:0/3431322166 wait complete. 2024-06-29T15:59:09.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.909+0000 7fddce47f640 1 Processor -- start 2024-06-29T15:59:09.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.909+0000 7fddce47f640 1 -- start start 2024-06-29T15:59:09.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddce47f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb0003a00 0x7fddc8099560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddce47f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddc815c5d0 0x7fddc815c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddce47f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fddc815cf20 0x7fddc814f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddce47f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fddc815b010 con 0x7fddc815c5d0 2024-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddce47f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fddc806d150 con 0x7fddc815cf20 2024-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddce47f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fddc815b190 con 0x7fddb0003a00 2024-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddc77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddc815c5d0 0x7fddc815c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddc77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddc815c5d0 0x7fddc815c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57706/0 (socket says 172.21.15.81:57706) 2024-06-29T15:59:09.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddc77fe640 1 -- 172.21.15.81:0/1086953214 learned_addr learned my addr 172.21.15.81:0/1086953214 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:09.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddc7fff640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb0003a00 0x7fddc8099560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddcca1c640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fddc815cf20 0x7fddc814f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddc77fe640 1 -- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb0003a00 msgr2=0x7fddc8099560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddc77fe640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb0003a00 0x7fddc8099560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddc77fe640 1 -- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fddc815cf20 msgr2=0x7fddc814f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddc77fe640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fddc815cf20 0x7fddc814f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddc77fe640 1 -- 172.21.15.81:0/1086953214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc814da90 con 0x7fddc815c5d0 2024-06-29T15:59:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.911+0000 7fddc77fe640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddc815c5d0 0x7fddc815c9e0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fddbc006630 tx=0x7fddbc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.912+0000 7fddc4ff9640 1 -- 172.21.15.81:0/1086953214 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc015070 con 0x7fddc815c5d0 2024-06-29T15:59:09.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.912+0000 7fddc4ff9640 1 -- 172.21.15.81:0/1086953214 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fddbc011840 con 0x7fddc815c5d0 2024-06-29T15:59:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.912+0000 7fddce47f640 1 -- 172.21.15.81:0/1086953214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc815d680 con 0x7fddc815c5d0 2024-06-29T15:59:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.912+0000 7fddc4ff9640 1 -- 172.21.15.81:0/1086953214 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc011c90 con 0x7fddc815c5d0 2024-06-29T15:59:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.912+0000 7fddce47f640 1 -- 172.21.15.81:0/1086953214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc815dc00 con 0x7fddc815c5d0 2024-06-29T15:59:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.913+0000 7fddce47f640 1 -- 172.21.15.81:0/1086953214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc8154f30 con 0x7fddc815c5d0 2024-06-29T15:59:09.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.913+0000 7fddc4ff9640 1 -- 172.21.15.81:0/1086953214 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fddbc018070 con 0x7fddc815c5d0 2024-06-29T15:59:09.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.914+0000 7fddc4ff9640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fddb4076540 0x7fddb4078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:09.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.914+0000 7fddc4ff9640 1 -- 172.21.15.81:0/1086953214 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5837+0+0 (secure 0 0 0) 0x7fddbc098850 con 0x7fddc815c5d0 2024-06-29T15:59:09.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.914+0000 7fddc7fff640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fddb4076540 0x7fddb4078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:09.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.915+0000 7fddc7fff640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fddb4076540 0x7fddb4078a50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009b50 tx=0x7fddb802d040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:09.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:09.918+0000 7fddc4ff9640 1 -- 172.21.15.81:0/1086953214 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fddbc061750 con 0x7fddc815c5d0 2024-06-29T15:59:10.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.100+0000 7fddce47f640 1 -- 172.21.15.81:0/1086953214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fddc815a6b0 con 0x7fddc815c5d0 2024-06-29T15:59:10.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.101+0000 7fddc4ff9640 1 -- 172.21.15.81:0/1086953214 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) ==== 74+0+16832 (secure 0 0 0) 0x7fddbc065a50 con 0x7fddc815c5d0 2024-06-29T15:59:10.103 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:10.103 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":252,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:09.126256+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:07.416791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:08.088694+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:10.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 -- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fddb4076540 msgr2=0x7fddb4078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fddb4076540 0x7fddb4078a50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009b50 tx=0x7fddb802d040 comp rx=0 tx=0).stop 2024-06-29T15:59:10.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 -- 172.21.15.81:0/1086953214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddc815c5d0 msgr2=0x7fddc815c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddc815c5d0 0x7fddc815c9e0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fddbc006630 tx=0x7fddbc0039c0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 -- 172.21.15.81:0/1086953214 shutdown_connections 2024-06-29T15:59:10.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fddb4076540 0x7fddb4078a50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fddc815cf20 0x7fddc814f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddc815c5d0 0x7fddc815c9e0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 --2- 172.21.15.81:0/1086953214 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fddb0003a00 0x7fddc8099560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 -- 172.21.15.81:0/1086953214 >> 172.21.15.81:0/1086953214 conn(0x7fddc8072950 msgr2=0x7fddc8158d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:10.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.103+0000 7fddce47f640 1 -- 172.21.15.81:0/1086953214 shutdown_connections 2024-06-29T15:59:10.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.104+0000 7fddce47f640 1 -- 172.21.15.81:0/1086953214 wait complete. 2024-06-29T15:59:10.116 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:10.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.230+0000 7f7ada622640 1 -- 172.21.15.81:0/303383452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ad415e960 msgr2=0x7f7ad415ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.230+0000 7f7ada622640 1 --2- 172.21.15.81:0/303383452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ad415e960 0x7f7ad415ed70 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f7acc005fd0 tx=0x7f7acc02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.230+0000 7f7ada622640 1 -- 172.21.15.81:0/303383452 shutdown_connections 2024-06-29T15:59:10.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.230+0000 7f7ada622640 1 --2- 172.21.15.81:0/303383452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ad415e960 0x7f7ad415ed70 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.230+0000 7f7ada622640 1 --2- 172.21.15.81:0/303383452 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ad4073030 0x7f7ad4073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.230+0000 7f7ada622640 1 -- 172.21.15.81:0/303383452 >> 172.21.15.81:0/303383452 conn(0x7f7ad406ccc0 msgr2=0x7f7ad40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:10.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.230+0000 7f7ada622640 1 -- 172.21.15.81:0/303383452 shutdown_connections 2024-06-29T15:59:10.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.230+0000 7f7ada622640 1 -- 172.21.15.81:0/303383452 wait complete. 2024-06-29T15:59:10.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.232+0000 7f7ada622640 1 Processor -- start 2024-06-29T15:59:10.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.232+0000 7f7ada622640 1 -- start start 2024-06-29T15:59:10.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.233+0000 7f7ada622640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ad4073030 0x7f7ad40c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.233+0000 7f7ada622640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ad41412f0 0x7f7ad40ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.233+0000 7f7ada622640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ad415e960 0x7f7ad40adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.233+0000 7f7ada622640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad40b3390 con 0x7f7ad4073030 2024-06-29T15:59:10.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.233+0000 7f7ada622640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad40b3210 con 0x7f7ad41412f0 2024-06-29T15:59:10.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.233+0000 7f7ada622640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7ad40b3510 con 0x7f7ad415e960 2024-06-29T15:59:10.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.233+0000 7f7ad3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ad4073030 0x7f7ad40c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.233+0000 7f7ad3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ad4073030 0x7f7ad40c5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60222/0 (socket says 172.21.15.81:60222) 2024-06-29T15:59:10.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.233+0000 7f7ad3fff640 1 -- 172.21.15.81:0/300360221 learned_addr learned my addr 172.21.15.81:0/300360221 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:10.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad8bbf640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ad415e960 0x7f7ad40adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad3fff640 1 -- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ad415e960 msgr2=0x7f7ad40adbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad3fff640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ad415e960 0x7f7ad40adbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad3fff640 1 -- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ad41412f0 msgr2=0x7f7ad40ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad37fe640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ad41412f0 0x7f7ad40ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad3fff640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ad41412f0 0x7f7ad40ad6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad3fff640 1 -- 172.21.15.81:0/300360221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad40c59c0 con 0x7f7ad4073030 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad3fff640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ad4073030 0x7f7ad40c5170 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400c730 tx=0x7f7ac400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad0ff9640 1 -- 172.21.15.81:0/300360221 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4016020 con 0x7f7ad4073030 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad0ff9640 1 -- 172.21.15.81:0/300360221 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ac400a950 con 0x7f7ad4073030 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ada622640 1 -- 172.21.15.81:0/300360221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad40c5ca0 con 0x7f7ad4073030 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ad0ff9640 1 -- 172.21.15.81:0/300360221 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac400ada0 con 0x7f7ad4073030 2024-06-29T15:59:10.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.234+0000 7f7ada622640 1 -- 172.21.15.81:0/300360221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad40c6260 con 0x7f7ad4073030 2024-06-29T15:59:10.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.235+0000 7f7ada622640 1 -- 172.21.15.81:0/300360221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ad40b3a10 con 0x7f7ad4073030 2024-06-29T15:59:10.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.236+0000 7f7ad0ff9640 1 -- 172.21.15.81:0/300360221 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f7ac4019020 con 0x7f7ad4073030 2024-06-29T15:59:10.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.236+0000 7f7ad0ff9640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7ac0076510 0x7f7ac0078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.236+0000 7f7ad0ff9640 1 -- 172.21.15.81:0/300360221 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5837+0+0 (secure 0 0 0) 0x7f7ac4097860 con 0x7f7ad4073030 2024-06-29T15:59:10.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.237+0000 7f7ad37fe640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7ac0076510 0x7f7ac0078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.237+0000 7f7ad37fe640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7ac0076510 0x7f7ac0078a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7ac80058c0 tx=0x7f7ac8005850 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:10.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.241+0000 7f7ad0ff9640 1 -- 172.21.15.81:0/300360221 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7ac4020030 con 0x7f7ad4073030 2024-06-29T15:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.420+0000 7f7ada622640 1 -- 172.21.15.81:0/300360221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ad415d630 con 0x7f7ad4073030 2024-06-29T15:59:10.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.421+0000 7f7ad0ff9640 1 -- 172.21.15.81:0/300360221 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) ==== 74+0+16832 (secure 0 0 0) 0x7f7ad4073980 con 0x7f7ad4073030 2024-06-29T15:59:10.422 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:10.422 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":252,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:09.126256+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:07.416791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:08.088694+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 -- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7ac0076510 msgr2=0x7f7ac0078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7ac0076510 0x7f7ac0078a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7ac80058c0 tx=0x7f7ac8005850 comp rx=0 tx=0).stop 2024-06-29T15:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 -- 172.21.15.81:0/300360221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ad4073030 msgr2=0x7f7ad40c5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ad4073030 0x7f7ad40c5170 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400c730 tx=0x7f7ac400cbf0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 -- 172.21.15.81:0/300360221 shutdown_connections 2024-06-29T15:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f7ac0076510 0x7f7ac0078a20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7ad415e960 0x7f7ad40adbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7ad41412f0 0x7f7ad40ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 --2- 172.21.15.81:0/300360221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ad4073030 0x7f7ad40c5170 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 -- 172.21.15.81:0/300360221 >> 172.21.15.81:0/300360221 conn(0x7f7ad406ccc0 msgr2=0x7f7ad406dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 -- 172.21.15.81:0/300360221 shutdown_connections 2024-06-29T15:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.423+0000 7f7ada622640 1 -- 172.21.15.81:0/300360221 wait complete. 2024-06-29T15:59:10.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:59:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.552+0000 7fa7cd447640 1 -- 172.21.15.81:0/4238564280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7c80b9040 msgr2=0x7fa7c80d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.552+0000 7fa7cd447640 1 --2- 172.21.15.81:0/4238564280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7c80b9040 0x7fa7c80d0480 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7fa7c808e330 tx=0x7fa7b802a610 comp rx=0 tx=0).stop 2024-06-29T15:59:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.552+0000 7fa7cd447640 1 -- 172.21.15.81:0/4238564280 shutdown_connections 2024-06-29T15:59:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.552+0000 7fa7cd447640 1 --2- 172.21.15.81:0/4238564280 >> v2:172.21.15.167:3300/0 conn(0x7fa7c80d2a90 0x7fa7c80d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.552+0000 7fa7cd447640 1 --2- 172.21.15.81:0/4238564280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7c80b9040 0x7fa7c80d0480 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.552+0000 7fa7cd447640 1 -- 172.21.15.81:0/4238564280 >> 172.21.15.81:0/4238564280 conn(0x7fa7c809cf30 msgr2=0x7fa7c80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:10.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.552+0000 7fa7cd447640 1 -- 172.21.15.81:0/4238564280 shutdown_connections 2024-06-29T15:59:10.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.552+0000 7fa7cd447640 1 -- 172.21.15.81:0/4238564280 wait complete. 2024-06-29T15:59:10.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.553+0000 7fa7cd447640 1 Processor -- start 2024-06-29T15:59:10.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.554+0000 7fa7cd447640 1 -- start start 2024-06-29T15:59:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7cd447640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7c80b5250 0x7fa7c809ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7cd447640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa7c80b9040 0x7fa7c80b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7cd447640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7c80d2a90 0x7fa7c80a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7cd447640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c80b7d30 con 0x7fa7c80d2a90 2024-06-29T15:59:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7cd447640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c80b7bb0 con 0x7fa7c80b5250 2024-06-29T15:59:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7cd447640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa7c80b7eb0 con 0x7fa7c80b9040 2024-06-29T15:59:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7c77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7c80d2a90 0x7fa7c80a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7c77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7c80d2a90 0x7fa7c80a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60238/0 (socket says 172.21.15.81:60238) 2024-06-29T15:59:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7c77fe640 1 -- 172.21.15.81:0/214059460 learned_addr learned my addr 172.21.15.81:0/214059460 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7be7fc640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa7c80b9040 0x7fa7c80b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7c6ffd640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7c80b5250 0x7fa7c809ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7c77fe640 1 -- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa7c80b9040 msgr2=0x7fa7c80b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7c77fe640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa7c80b9040 0x7fa7c80b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7c77fe640 1 -- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7c80b5250 msgr2=0x7fa7c809ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7c77fe640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7c80b5250 0x7fa7c809ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.555+0000 7fa7c77fe640 1 -- 172.21.15.81:0/214059460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c80b2960 con 0x7fa7c80d2a90 2024-06-29T15:59:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.556+0000 7fa7c77fe640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7c80d2a90 0x7fa7c80a0960 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000ba10 tx=0x7fa7c000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.556+0000 7fa7bffff640 1 -- 172.21.15.81:0/214059460 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0019070 con 0x7fa7c80d2a90 2024-06-29T15:59:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.556+0000 7fa7bffff640 1 -- 172.21.15.81:0/214059460 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7c000cc10 con 0x7fa7c80d2a90 2024-06-29T15:59:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.556+0000 7fa7cd447640 1 -- 172.21.15.81:0/214059460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c809f760 con 0x7fa7c80d2a90 2024-06-29T15:59:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.556+0000 7fa7bffff640 1 -- 172.21.15.81:0/214059460 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0014440 con 0x7fa7c80d2a90 2024-06-29T15:59:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.556+0000 7fa7cd447640 1 -- 172.21.15.81:0/214059460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c809e9a0 con 0x7fa7c80d2a90 2024-06-29T15:59:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.557+0000 7fa7cd447640 1 -- 172.21.15.81:0/214059460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7c80b8440 con 0x7fa7c80d2a90 2024-06-29T15:59:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.558+0000 7fa7bffff640 1 -- 172.21.15.81:0/214059460 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fa7c001c070 con 0x7fa7c80d2a90 2024-06-29T15:59:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.558+0000 7fa7bffff640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa7a8076540 0x7fa7a8078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.558+0000 7fa7bffff640 1 -- 172.21.15.81:0/214059460 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5837+0+0 (secure 0 0 0) 0x7fa7c0097960 con 0x7fa7c80d2a90 2024-06-29T15:59:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.558+0000 7fa7c6ffd640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa7a8076540 0x7fa7a8078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.559+0000 7fa7c6ffd640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa7a8076540 0x7fa7a8078a50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8033000 tx=0x7fa7b8004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.563+0000 7fa7bffff640 1 -- 172.21.15.81:0/214059460 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa7c0060800 con 0x7fa7c80d2a90 2024-06-29T15:59:10.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:10.779+0000 7fa7cd447640 1 -- 172.21.15.81:0/214059460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa7c80b8780 con 0x7fa7c80d2a90 2024-06-29T15:59:11.737 INFO:tasks.ceph.mds.d.smithi167.stderr:2024-06-29T15:59:11.736+0000 7ff74b90a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:59:11.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.794+0000 7fa7bffff640 1 -- 172.21.15.81:0/214059460 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v144) ==== 118+0+0 (secure 0 0 0) 0x7fa7c0064b00 con 0x7fa7c80d2a90 2024-06-29T15:59:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.797+0000 7fa7cd447640 1 -- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa7a8076540 msgr2=0x7fa7a8078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.797+0000 7fa7cd447640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa7a8076540 0x7fa7a8078a50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8033000 tx=0x7fa7b8004730 comp rx=0 tx=0).stop 2024-06-29T15:59:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.797+0000 7fa7cd447640 1 -- 172.21.15.81:0/214059460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7c80d2a90 msgr2=0x7fa7c80a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.797+0000 7fa7cd447640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7c80d2a90 0x7fa7c80a0960 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000ba10 tx=0x7fa7c000bed0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.797+0000 7fa7cd447640 1 -- 172.21.15.81:0/214059460 shutdown_connections 2024-06-29T15:59:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.797+0000 7fa7cd447640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa7a8076540 0x7fa7a8078a50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.797+0000 7fa7cd447640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7c80d2a90 0x7fa7c80a0960 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.797+0000 7fa7cd447640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa7c80b9040 0x7fa7c80b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.797+0000 7fa7cd447640 1 --2- 172.21.15.81:0/214059460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa7c80b5250 0x7fa7c809ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.797+0000 7fa7cd447640 1 -- 172.21.15.81:0/214059460 >> 172.21.15.81:0/214059460 conn(0x7fa7c809cf30 msgr2=0x7fa7c80d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:11.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.798+0000 7fa7cd447640 1 -- 172.21.15.81:0/214059460 shutdown_connections 2024-06-29T15:59:11.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.798+0000 7fa7cd447640 1 -- 172.21.15.81:0/214059460 wait complete. 2024-06-29T15:59:11.813 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:59:11.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:59:11.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.936+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/484125097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f5c004d40 msgr2=0x7f5f5c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:11.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.936+0000 7f5f7b1f5640 1 --2- 172.21.15.81:0/484125097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f5c004d40 0x7f5f5c005150 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0013d0 tx=0x7f5f5c005830 comp rx=0 tx=0).stop 2024-06-29T15:59:11.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.936+0000 7f5f79792640 1 -- 172.21.15.81:0/484125097 reap_dead start 2024-06-29T15:59:11.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.936+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/484125097 shutdown_connections 2024-06-29T15:59:11.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.936+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/484125097 >> 172.21.15.81:0/484125097 conn(0x7f5f7409cf30 msgr2=0x7f5f740d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:11.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.936+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/484125097 shutdown_connections 2024-06-29T15:59:11.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.936+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/484125097 wait complete. 2024-06-29T15:59:11.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.938+0000 7f5f7b1f5640 1 Processor -- start 2024-06-29T15:59:11.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.938+0000 7f5f7b1f5640 1 -- start start 2024-06-29T15:59:11.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f7b1f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f5c004d40 0x7f5f741f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:11.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f7b1f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f741f5760 0x7f5f740b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:11.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f7b1f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f7409d9a0 0x7f5f740a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:11.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f7b1f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f740d09c0 con 0x7f5f5c004d40 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f7b1f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5f740d2d80 con 0x7f5f741f5760 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f7b1f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5f740d0b40 con 0x7f5f7409d9a0 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f78f91640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f5c004d40 0x7f5f741f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f78f91640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f5c004d40 0x7f5f741f20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60262/0 (socket says 172.21.15.81:60262) 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f78f91640 1 -- 172.21.15.81:0/741598033 learned_addr learned my addr 172.21.15.81:0/741598033 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f6bfff640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f741f5760 0x7f5f740b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.939+0000 7f5f79792640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f7409d9a0 0x7f5f740a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f78f91640 1 -- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f7409d9a0 msgr2=0x7f5f740a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f78f91640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f7409d9a0 0x7f5f740a0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f78f91640 1 -- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f741f5760 msgr2=0x7f5f740b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f78f91640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f741f5760 0x7f5f740b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f78f91640 1 -- 172.21.15.81:0/741598033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f740b4b00 con 0x7f5f5c004d40 2024-06-29T15:59:11.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f79792640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f7409d9a0 0x7f5f740a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f6bfff640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f741f5760 0x7f5f740b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f78f91640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f5c004d40 0x7f5f741f20d0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f5f70011a80 tx=0x7f5f70011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f697fa640 1 -- 172.21.15.81:0/741598033 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000fd50 con 0x7f5f5c004d40 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f697fa640 1 -- 172.21.15.81:0/741598033 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f70016070 con 0x7f5f5c004d40 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/741598033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f740b3800 con 0x7f5f5c004d40 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f697fa640 1 -- 172.21.15.81:0/741598033 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f700126b0 con 0x7f5f5c004d40 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.940+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/741598033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f740b3dc0 con 0x7f5f5c004d40 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.941+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/741598033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f740d0480 con 0x7f5f5c004d40 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.942+0000 7f5f697fa640 1 -- 172.21.15.81:0/741598033 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f5f70018030 con 0x7f5f5c004d40 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.942+0000 7f5f697fa640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f60076510 0x7f5f60078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.942+0000 7f5f697fa640 1 -- 172.21.15.81:0/741598033 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7f5f70098470 con 0x7f5f5c004d40 2024-06-29T15:59:11.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.942+0000 7f5f6bfff640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f60076510 0x7f5f60078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:11.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.943+0000 7f5f6bfff640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f60076510 0x7f5f60078a20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5f740ca960 tx=0x7f5f64005f90 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:11.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:11.946+0000 7f5f697fa640 1 -- 172.21.15.81:0/741598033 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5f70061a80 con 0x7f5f5c004d40 2024-06-29T15:59:12.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.164+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/741598033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f5f74160290 con 0x7f5f5c004d40 2024-06-29T15:59:12.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.800+0000 7f5f697fa640 1 -- 172.21.15.81:0/741598033 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v145) ==== 122+0+0 (secure 0 0 0) 0x7f5f70061380 con 0x7f5f5c004d40 2024-06-29T15:59:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f60076510 msgr2=0x7f5f60078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f60076510 0x7f5f60078a20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5f740ca960 tx=0x7f5f64005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/741598033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f5c004d40 msgr2=0x7f5f741f20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f5c004d40 0x7f5f741f20d0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f5f70011a80 tx=0x7f5f70011f40 comp rx=0 tx=0).stop 2024-06-29T15:59:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/741598033 shutdown_connections 2024-06-29T15:59:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5f60076510 0x7f5f60078a20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5f7409d9a0 0x7f5f740a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5f741f5760 0x7f5f740b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 --2- 172.21.15.81:0/741598033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f5c004d40 0x7f5f741f20d0 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/741598033 >> 172.21.15.81:0/741598033 conn(0x7f5f7409cf30 msgr2=0x7f5f7415f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:12.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/741598033 shutdown_connections 2024-06-29T15:59:12.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.802+0000 7f5f7b1f5640 1 -- 172.21.15.81:0/741598033 wait complete. 2024-06-29T15:59:12.816 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:59:12.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:12.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.929+0000 7f29e1970640 1 -- 172.21.15.81:0/3429220758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29dc0cd540 msgr2=0x7f29dc0b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:12.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.929+0000 7f29e1970640 1 --2- 172.21.15.81:0/3429220758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29dc0cd540 0x7f29dc0b47b0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f29d0005fd0 tx=0x7f29d002a670 comp rx=0 tx=0).stop 2024-06-29T15:59:12.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.929+0000 7f29e1970640 1 -- 172.21.15.81:0/3429220758 shutdown_connections 2024-06-29T15:59:12.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.929+0000 7f29e1970640 1 --2- 172.21.15.81:0/3429220758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29dc0cd540 0x7f29dc0b47b0 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.929+0000 7f29e1970640 1 -- 172.21.15.81:0/3429220758 >> 172.21.15.81:0/3429220758 conn(0x7f29dc0d1960 msgr2=0x7f29dc0d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:12.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.929+0000 7f29e1970640 1 -- 172.21.15.81:0/3429220758 shutdown_connections 2024-06-29T15:59:12.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.929+0000 7f29e1970640 1 -- 172.21.15.81:0/3429220758 wait complete. 2024-06-29T15:59:12.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.931+0000 7f29e1970640 1 Processor -- start 2024-06-29T15:59:12.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.931+0000 7f29e1970640 1 -- start start 2024-06-29T15:59:12.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29e1970640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29dc0b4cf0 0x7f29dc1580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29e1970640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc0b6d60 0x7f29dc158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29e1970640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29dc0cd540 0x7f29dc158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29e1970640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc0d0980 con 0x7f29dc0b4cf0 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29e1970640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f29dc0d0800 con 0x7f29dc0cd540 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29e1970640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f29dc167640 con 0x7f29dc0b6d60 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29daffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29dc0b4cf0 0x7f29dc1580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29daffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29dc0b4cf0 0x7f29dc1580f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60278/0 (socket says 172.21.15.81:60278) 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29daffd640 1 -- 172.21.15.81:0/3086468336 learned_addr learned my addr 172.21.15.81:0/3086468336 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29da7fc640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc0b6d60 0x7f29dc158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.932+0000 7f29db7fe640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29dc0cd540 0x7f29dc158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29daffd640 1 -- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc0b6d60 msgr2=0x7f29dc158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:12.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29daffd640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc0b6d60 0x7f29dc158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29daffd640 1 -- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29dc0cd540 msgr2=0x7f29dc158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29daffd640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29dc0cd540 0x7f29dc158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29daffd640 1 -- 172.21.15.81:0/3086468336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29dc15a550 con 0x7f29dc0b4cf0 2024-06-29T15:59:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29db7fe640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29dc0cd540 0x7f29dc158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29daffd640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29dc0b4cf0 0x7f29dc1580f0 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00c340 tx=0x7f29cc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29c3fff640 1 -- 172.21.15.81:0/3086468336 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc017070 con 0x7f29dc0b4cf0 2024-06-29T15:59:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29c3fff640 1 -- 172.21.15.81:0/3086468336 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29cc0035f0 con 0x7f29dc0b4cf0 2024-06-29T15:59:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29e1970640 1 -- 172.21.15.81:0/3086468336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29dc158630 con 0x7f29dc0b4cf0 2024-06-29T15:59:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29c3fff640 1 -- 172.21.15.81:0/3086468336 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc003a80 con 0x7f29dc0b4cf0 2024-06-29T15:59:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.933+0000 7f29e1970640 1 -- 172.21.15.81:0/3086468336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29dc0af440 con 0x7f29dc0b4cf0 2024-06-29T15:59:12.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.934+0000 7f29e1970640 1 -- 172.21.15.81:0/3086468336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29dc0af230 con 0x7f29dc0b4cf0 2024-06-29T15:59:12.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.935+0000 7f29c3fff640 1 -- 172.21.15.81:0/3086468336 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f29cc01a070 con 0x7f29dc0b4cf0 2024-06-29T15:59:12.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.935+0000 7f29c3fff640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f29c8076540 0x7f29c8078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:12.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.935+0000 7f29c3fff640 1 -- 172.21.15.81:0/3086468336 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7f29cc096ec0 con 0x7f29dc0b4cf0 2024-06-29T15:59:12.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.935+0000 7f29da7fc640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f29c8076540 0x7f29c8078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:12.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.936+0000 7f29da7fc640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f29c8076540 0x7f29c8078a50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29d0033000 tx=0x7f29d0004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:12.940+0000 7f29c3fff640 1 -- 172.21.15.81:0/3086468336 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f29cc05fd60 con 0x7f29dc0b4cf0 2024-06-29T15:59:13.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.157+0000 7f29e1970640 1 -- 172.21.15.81:0/3086468336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29dc0b47b0 con 0x7f29dc0b4cf0 2024-06-29T15:59:13.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.158+0000 7f29c3fff640 1 -- 172.21.15.81:0/3086468336 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+5236 (secure 0 0 0) 0x7f29cc068430 con 0x7f29dc0b4cf0 2024-06-29T15:59:13.159 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:13.159 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":145,"btime":"2024-06-29T15:59:12:796690+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9842,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2717891820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2717891820},{"type":"v1","addr":"172.21.15.81:6835","nonce":2717891820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":9848,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2745105064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2745105064},{"type":"v1","addr":"172.21.15.167:6836","nonce":2745105064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":11329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/745441473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":745441473},{"type":"v1","addr":"172.21.15.81:6834","nonce":745441473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:09.126215+0000","modified":"2024-06-29T15:59:12.167052+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11347":{"gid":11347,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/54677014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":54677014},{"type":"v1","addr":"172.21.15.167:6837","nonce":54677014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11347,"qdb_cluster":[11347]},"id":17}]} 2024-06-29T15:59:13.160 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 145 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 -- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f29c8076540 msgr2=0x7f29c8078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f29c8076540 0x7f29c8078a50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29d0033000 tx=0x7f29d0004730 comp rx=0 tx=0).stop 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 -- 172.21.15.81:0/3086468336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29dc0b4cf0 msgr2=0x7f29dc1580f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29dc0b4cf0 0x7f29dc1580f0 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00c340 tx=0x7f29cc00c800 comp rx=0 tx=0).stop 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 -- 172.21.15.81:0/3086468336 shutdown_connections 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f29c8076540 0x7f29c8078a50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f29dc0cd540 0x7f29dc158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f29dc0b6d60 0x7f29dc158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 --2- 172.21.15.81:0/3086468336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29dc0b4cf0 0x7f29dc1580f0 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 -- 172.21.15.81:0/3086468336 >> 172.21.15.81:0/3086468336 conn(0x7f29dc0d1960 msgr2=0x7f29dc068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:13.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 -- 172.21.15.81:0/3086468336 shutdown_connections 2024-06-29T15:59:13.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.160+0000 7f29e1970640 1 -- 172.21.15.81:0/3086468336 wait complete. 2024-06-29T15:59:13.173 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.285+0000 7f88a63f5640 1 -- 172.21.15.81:0/1522987984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f88a00b6eb0 msgr2=0x7f88a00d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.285+0000 7f88a63f5640 1 --2- 172.21.15.81:0/1522987984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f88a00b6eb0 0x7f88a00d02d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f88a00b41b0 tx=0x7f889002a530 comp rx=0 tx=0).stop 2024-06-29T15:59:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.285+0000 7f88a63f5640 1 -- 172.21.15.81:0/1522987984 shutdown_connections 2024-06-29T15:59:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.285+0000 7f88a63f5640 1 --2- 172.21.15.81:0/1522987984 >> v2:172.21.15.167:3300/0 conn(0x7f88a00ce510 0x7f88a00ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.285+0000 7f88a63f5640 1 --2- 172.21.15.81:0/1522987984 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f88a00b6eb0 0x7f88a00d02d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.285+0000 7f88a63f5640 1 -- 172.21.15.81:0/1522987984 >> 172.21.15.81:0/1522987984 conn(0x7f88a00b3510 msgr2=0x7f88a00b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.285+0000 7f88a63f5640 1 -- 172.21.15.81:0/1522987984 shutdown_connections 2024-06-29T15:59:13.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.285+0000 7f88a63f5640 1 -- 172.21.15.81:0/1522987984 wait complete. 2024-06-29T15:59:13.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.287+0000 7f88a63f5640 1 Processor -- start 2024-06-29T15:59:13.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.287+0000 7f88a63f5640 1 -- start start 2024-06-29T15:59:13.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f88a63f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88a00b4810 0x7f88a009d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f88a63f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88a00b6eb0 0x7f88a009def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f88a63f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f88a00ce510 0x7f88a01601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f88a63f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f88a00cd480 con 0x7f88a00b4810 2024-06-29T15:59:13.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f88a63f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f88a00b29c0 con 0x7f88a00b6eb0 2024-06-29T15:59:13.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f88a63f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f88a00cd600 con 0x7f88a00ce510 2024-06-29T15:59:13.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88a00b4810 0x7f88a009d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88a00b4810 0x7f88a009d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60290/0 (socket says 172.21.15.81:60290) 2024-06-29T15:59:13.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889ffff640 1 -- 172.21.15.81:0/2276862795 learned_addr learned my addr 172.21.15.81:0/2276862795 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f88a4992640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f88a00ce510 0x7f88a01601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889f7fe640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88a00b6eb0 0x7f88a009def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889ffff640 1 -- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f88a00ce510 msgr2=0x7f88a01601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889ffff640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f88a00ce510 0x7f88a01601a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889ffff640 1 -- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88a00b6eb0 msgr2=0x7f88a009def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889ffff640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88a00b6eb0 0x7f88a009def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889ffff640 1 -- 172.21.15.81:0/2276862795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a009e750 con 0x7f88a00b4810 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889f7fe640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88a00b6eb0 0x7f88a009def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f88a4992640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f88a00ce510 0x7f88a01601a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889ffff640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88a00b4810 0x7f88a009d9b0 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f8890000c00 tx=0x7f8890004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:13.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.289+0000 7f889cff9640 1 -- 172.21.15.81:0/2276862795 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889003e030 con 0x7f88a00b4810 2024-06-29T15:59:13.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.290+0000 7f889cff9640 1 -- 172.21.15.81:0/2276862795 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88900047a0 con 0x7f88a00b4810 2024-06-29T15:59:13.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.290+0000 7f88a63f5640 1 -- 172.21.15.81:0/2276862795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a009ea30 con 0x7f88a00b4810 2024-06-29T15:59:13.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.290+0000 7f889cff9640 1 -- 172.21.15.81:0/2276862795 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8890035040 con 0x7f88a00b4810 2024-06-29T15:59:13.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.290+0000 7f88a63f5640 1 -- 172.21.15.81:0/2276862795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a009eff0 con 0x7f88a00b4810 2024-06-29T15:59:13.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.291+0000 7f88a63f5640 1 -- 172.21.15.81:0/2276862795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a00d02d0 con 0x7f88a00b4810 2024-06-29T15:59:13.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.291+0000 7f889cff9640 1 -- 172.21.15.81:0/2276862795 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f8890041030 con 0x7f88a00b4810 2024-06-29T15:59:13.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.292+0000 7f889cff9640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f888c076540 0x7f888c078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.292+0000 7f889cff9640 1 -- 172.21.15.81:0/2276862795 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7f88900b8670 con 0x7f88a00b4810 2024-06-29T15:59:13.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.292+0000 7f889f7fe640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f888c076540 0x7f888c078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.292+0000 7f889f7fe640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f888c076540 0x7f888c078a50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8894009b80 tx=0x7f8894005f90 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:13.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.296+0000 7f889cff9640 1 -- 172.21.15.81:0/2276862795 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8890081b30 con 0x7f88a00b4810 2024-06-29T15:59:13.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.477+0000 7f88a63f5640 1 -- 172.21.15.81:0/2276862795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88a00c8670 con 0x7f88a00b4810 2024-06-29T15:59:13.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.478+0000 7f889cff9640 1 -- 172.21.15.81:0/2276862795 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) ==== 74+0+16819 (secure 0 0 0) 0x7f8890081430 con 0x7f88a00b4810 2024-06-29T15:59:13.480 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:13.480 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":253,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:10.764096+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:07.416791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:08.088694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:13.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 -- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f888c076540 msgr2=0x7f888c078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f888c076540 0x7f888c078a50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8894009b80 tx=0x7f8894005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 -- 172.21.15.81:0/2276862795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88a00b4810 msgr2=0x7f88a009d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88a00b4810 0x7f88a009d9b0 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f8890000c00 tx=0x7f8890004c80 comp rx=0 tx=0).stop 2024-06-29T15:59:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 -- 172.21.15.81:0/2276862795 shutdown_connections 2024-06-29T15:59:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f888c076540 0x7f888c078a50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f88a00ce510 0x7f88a01601a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f88a00b6eb0 0x7f88a009def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 --2- 172.21.15.81:0/2276862795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88a00b4810 0x7f88a009d9b0 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 -- 172.21.15.81:0/2276862795 >> 172.21.15.81:0/2276862795 conn(0x7f88a00b3510 msgr2=0x7f88a00b4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 -- 172.21.15.81:0/2276862795 shutdown_connections 2024-06-29T15:59:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.481+0000 7f88a63f5640 1 -- 172.21.15.81:0/2276862795 wait complete. 2024-06-29T15:59:13.494 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:59:13.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:59:13.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.606+0000 7fa278012640 1 -- 172.21.15.81:0/1200285069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2701412f0 msgr2=0x7fa2700cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.606+0000 7fa278012640 1 --2- 172.21.15.81:0/1200285069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2701412f0 0x7fa2700cef90 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009830 tx=0x7fa26c02a5c0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.606+0000 7fa278012640 1 -- 172.21.15.81:0/1200285069 shutdown_connections 2024-06-29T15:59:13.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.606+0000 7fa278012640 1 --2- 172.21.15.81:0/1200285069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2701412f0 0x7fa2700cef90 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.606+0000 7fa278012640 1 -- 172.21.15.81:0/1200285069 >> 172.21.15.81:0/1200285069 conn(0x7fa27006ccc0 msgr2=0x7fa2700b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:13.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.606+0000 7fa278012640 1 -- 172.21.15.81:0/1200285069 shutdown_connections 2024-06-29T15:59:13.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.606+0000 7fa278012640 1 -- 172.21.15.81:0/1200285069 wait complete. 2024-06-29T15:59:13.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.608+0000 7fa278012640 1 Processor -- start 2024-06-29T15:59:13.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.608+0000 7fa278012640 1 -- start start 2024-06-29T15:59:13.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.609+0000 7fa278012640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2700cf4d0 0x7fa2701f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.609+0000 7fa278012640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2701412f0 0x7fa27013fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.609+0000 7fa278012640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2701e5b00 0x7fa2701e5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.609+0000 7fa278012640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa2700b35a0 con 0x7fa2700cf4d0 2024-06-29T15:59:13.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.609+0000 7fa278012640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa2700b3420 con 0x7fa2701e5b00 2024-06-29T15:59:13.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.609+0000 7fa278012640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa2700b3720 con 0x7fa2701412f0 2024-06-29T15:59:13.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa275dae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2700cf4d0 0x7fa2701f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa275dae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2700cf4d0 0x7fa2701f32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60310/0 (socket says 172.21.15.81:60310) 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa275dae640 1 -- 172.21.15.81:0/1619204306 learned_addr learned my addr 172.21.15.81:0/1619204306 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa2755ad640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2701412f0 0x7fa27013fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa2765af640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2701e5b00 0x7fa2701e5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa275dae640 1 -- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2701412f0 msgr2=0x7fa27013fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa275dae640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2701412f0 0x7fa27013fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa275dae640 1 -- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2701e5b00 msgr2=0x7fa2701e5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa275dae640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2701e5b00 0x7fa2701e5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa275dae640 1 -- 172.21.15.81:0/1619204306 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2701e6450 con 0x7fa2700cf4d0 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa2765af640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2701e5b00 0x7fa2701e5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa275dae640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2700cf4d0 0x7fa2701f32d0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009950 tx=0x7fa26c033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa2667fc640 1 -- 172.21.15.81:0/1619204306 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c031d60 con 0x7fa2700cf4d0 2024-06-29T15:59:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa2667fc640 1 -- 172.21.15.81:0/1619204306 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa26c0330d0 con 0x7fa2700cf4d0 2024-06-29T15:59:13.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa278012640 1 -- 172.21.15.81:0/1619204306 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2701f6900 con 0x7fa2700cf4d0 2024-06-29T15:59:13.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa2667fc640 1 -- 172.21.15.81:0/1619204306 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c033520 con 0x7fa2700cf4d0 2024-06-29T15:59:13.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.610+0000 7fa278012640 1 -- 172.21.15.81:0/1619204306 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2701f70e0 con 0x7fa2700cf4d0 2024-06-29T15:59:13.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.611+0000 7fa278012640 1 -- 172.21.15.81:0/1619204306 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa270140f00 con 0x7fa2700cf4d0 2024-06-29T15:59:13.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.612+0000 7fa2667fc640 1 -- 172.21.15.81:0/1619204306 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fa26c0035f0 con 0x7fa2700cf4d0 2024-06-29T15:59:13.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.613+0000 7fa2667fc640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa254076510 0x7fa254078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.613+0000 7fa2667fc640 1 -- 172.21.15.81:0/1619204306 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7fa26c0b9180 con 0x7fa2700cf4d0 2024-06-29T15:59:13.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.613+0000 7fa2755ad640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa254076510 0x7fa254078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.613+0000 7fa2755ad640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa254076510 0x7fa254078a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa2700b1b00 tx=0x7fa260009040 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:13.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.617+0000 7fa2667fc640 1 -- 172.21.15.81:0/1619204306 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa270140f00 con 0x7fa2700cf4d0 2024-06-29T15:59:13.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.793+0000 7fa278012640 1 -- 172.21.15.81:0/1619204306 --> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa270140360 con 0x7fa254076510 2024-06-29T15:59:13.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.798+0000 7fa2667fc640 1 -- 172.21.15.81:0/1619204306 <== mgr.11053 v2:172.21.15.167:6832/1778875525 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247101 (secure 0 0 0) 0x7fa260001cd0 con 0x7fa254076510 2024-06-29T15:59:13.802 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:13.803 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:59:13.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 -- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa254076510 msgr2=0x7fa254078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa254076510 0x7fa254078a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa2700b1b00 tx=0x7fa260009040 comp rx=0 tx=0).stop 2024-06-29T15:59:13.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 -- 172.21.15.81:0/1619204306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2700cf4d0 msgr2=0x7fa2701f32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2700cf4d0 0x7fa2701f32d0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009950 tx=0x7fa26c033d60 comp rx=0 tx=0).stop 2024-06-29T15:59:13.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 -- 172.21.15.81:0/1619204306 shutdown_connections 2024-06-29T15:59:13.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fa254076510 0x7fa254078a20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2701e5b00 0x7fa2701e5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2701412f0 0x7fa27013fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 --2- 172.21.15.81:0/1619204306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2700cf4d0 0x7fa2701f32d0 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 -- 172.21.15.81:0/1619204306 >> 172.21.15.81:0/1619204306 conn(0x7fa27006ccc0 msgr2=0x7fa27015deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:13.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 -- 172.21.15.81:0/1619204306 shutdown_connections 2024-06-29T15:59:13.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.801+0000 7fa278012640 1 -- 172.21.15.81:0/1619204306 wait complete. 2024-06-29T15:59:13.814 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-06-29T15:59:11.840299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247,"ondisk_log_size":247,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":444624,"kb_used_data":25876,"kb_used_omap":12,"kb_used_meta":418611,"kb_avail":754530096,"statfs":{"total":773094113280,"available":772638818304,"internally_reserved":0,"allocated":26497024,"data_stored":13680827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":428658226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.088545"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.767838+0000","last_change":"2024-06-29T15:59:08.767838+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.767838+0000","last_undegraded":"2024-06-29T15:59:08.767838+0000","last_fullsized":"2024-06-29T15:59:08.767838+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.765541+0000","last_change":"2024-06-29T15:59:08.765541+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.765541+0000","last_undegraded":"2024-06-29T15:59:08.765541+0000","last_fullsized":"2024-06-29T15:59:08.765541+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.764258+0000","last_change":"2024-06-29T15:59:08.764258+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.764258+0000","last_undegraded":"2024-06-29T15:59:08.764258+0000","last_fullsized":"2024-06-29T15:59:08.764258+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.759707+0000","last_change":"2024-06-29T15:59:08.759707+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.759707+0000","last_undegraded":"2024-06-29T15:59:08.759707+0000","last_fullsized":"2024-06-29T15:59:08.759707+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.764179+0000","last_change":"2024-06-29T15:59:08.764179+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.764179+0000","last_undegraded":"2024-06-29T15:59:08.764179+0000","last_fullsized":"2024-06-29T15:59:08.764179+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.763741+0000","last_change":"2024-06-29T15:59:08.763741+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.763741+0000","last_undegraded":"2024-06-29T15:59:08.763741+0000","last_fullsized":"2024-06-29T15:59:08.763741+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.761922+0000","last_change":"2024-06-29T15:59:08.761922+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.761922+0000","last_undegraded":"2024-06-29T15:59:08.761922+0000","last_fullsized":"2024-06-29T15:59:08.761922+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.762845+0000","last_change":"2024-06-29T15:59:08.762845+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.762845+0000","last_undegraded":"2024-06-29T15:59:08.762845+0000","last_fullsized":"2024-06-29T15:59:08.762845+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.763431+0000","last_change":"2024-06-29T15:59:08.763431+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.763431+0000","last_undegraded":"2024-06-29T15:59:08.763431+0000","last_fullsized":"2024-06-29T15:59:08.763431+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.758603+0000","last_change":"2024-06-29T15:59:08.758603+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.758603+0000","last_undegraded":"2024-06-29T15:59:08.758603+0000","last_fullsized":"2024-06-29T15:59:08.758603+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.765658+0000","last_change":"2024-06-29T15:59:08.765658+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.765658+0000","last_undegraded":"2024-06-29T15:59:08.765658+0000","last_fullsized":"2024-06-29T15:59:08.765658+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.766707+0000","last_change":"2024-06-29T15:59:08.766707+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.766707+0000","last_undegraded":"2024-06-29T15:59:08.766707+0000","last_fullsized":"2024-06-29T15:59:08.766707+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.768029+0000","last_change":"2024-06-29T15:59:08.768029+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.768029+0000","last_undegraded":"2024-06-29T15:59:08.768029+0000","last_fullsized":"2024-06-29T15:59:08.768029+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.757258+0000","last_change":"2024-06-29T15:59:08.757258+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.757258+0000","last_undegraded":"2024-06-29T15:59:08.757258+0000","last_fullsized":"2024-06-29T15:59:08.757258+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.761452+0000","last_change":"2024-06-29T15:59:08.761452+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.761452+0000","last_undegraded":"2024-06-29T15:59:08.761452+0000","last_fullsized":"2024-06-29T15:59:08.761452+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760334+0000","last_change":"2024-06-29T15:59:08.760334+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760334+0000","last_undegraded":"2024-06-29T15:59:08.760334+0000","last_fullsized":"2024-06-29T15:59:08.760334+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.762989+0000","last_change":"2024-06-29T15:59:08.762989+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.762989+0000","last_undegraded":"2024-06-29T15:59:08.762989+0000","last_fullsized":"2024-06-29T15:59:08.762989+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.762612+0000","last_change":"2024-06-29T15:59:08.762612+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.762612+0000","last_undegraded":"2024-06-29T15:59:08.762612+0000","last_fullsized":"2024-06-29T15:59:08.762612+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.761568+0000","last_change":"2024-06-29T15:59:08.761568+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.761568+0000","last_undegraded":"2024-06-29T15:59:08.761568+0000","last_fullsized":"2024-06-29T15:59:08.761568+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.761147+0000","last_change":"2024-06-29T15:59:08.761147+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.761147+0000","last_undegraded":"2024-06-29T15:59:08.761147+0000","last_fullsized":"2024-06-29T15:59:08.761147+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.767467+0000","last_change":"2024-06-29T15:59:08.767467+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.767467+0000","last_undegraded":"2024-06-29T15:59:08.767467+0000","last_fullsized":"2024-06-29T15:59:08.767467+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.763255+0000","last_change":"2024-06-29T15:59:08.763255+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.763255+0000","last_undegraded":"2024-06-29T15:59:08.763255+0000","last_fullsized":"2024-06-29T15:59:08.763255+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.759392+0000","last_change":"2024-06-29T15:59:08.759392+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.759392+0000","last_undegraded":"2024-06-29T15:59:08.759392+0000","last_fullsized":"2024-06-29T15:59:08.759392+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.762720+0000","last_change":"2024-06-29T15:59:08.762720+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.762720+0000","last_undegraded":"2024-06-29T15:59:08.762720+0000","last_fullsized":"2024-06-29T15:59:08.762720+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.759684+0000","last_change":"2024-06-29T15:59:08.759684+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.759684+0000","last_undegraded":"2024-06-29T15:59:08.759684+0000","last_fullsized":"2024-06-29T15:59:08.759684+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.766389+0000","last_change":"2024-06-29T15:59:08.766389+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.766389+0000","last_undegraded":"2024-06-29T15:59:08.766389+0000","last_fullsized":"2024-06-29T15:59:08.766389+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760507+0000","last_change":"2024-06-29T15:59:08.760507+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760507+0000","last_undegraded":"2024-06-29T15:59:08.760507+0000","last_fullsized":"2024-06-29T15:59:08.760507+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.758742+0000","last_change":"2024-06-29T15:59:08.758742+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.758742+0000","last_undegraded":"2024-06-29T15:59:08.758742+0000","last_fullsized":"2024-06-29T15:59:08.758742+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.762217+0000","last_change":"2024-06-29T15:59:08.762217+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.762217+0000","last_undegraded":"2024-06-29T15:59:08.762217+0000","last_fullsized":"2024-06-29T15:59:08.762217+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760943+0000","last_change":"2024-06-29T15:59:08.760943+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760943+0000","last_undegraded":"2024-06-29T15:59:08.760943+0000","last_fullsized":"2024-06-29T15:59:08.760943+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.761804+0000","last_change":"2024-06-29T15:59:08.761804+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.761804+0000","last_undegraded":"2024-06-29T15:59:08.761804+0000","last_fullsized":"2024-06-29T15:59:08.761804+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.765927+0000","last_change":"2024-06-29T15:59:08.765927+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.765927+0000","last_undegraded":"2024-06-29T15:59:08.765927+0000","last_fullsized":"2024-06-29T15:59:08.765927+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.776174+0000","last_change":"2024-06-29T15:59:08.774828+0000","last_active":"2024-06-29T15:59:08.776174+0000","last_peered":"2024-06-29T15:59:08.776174+0000","last_clean":"2024-06-29T15:59:08.776174+0000","last_became_active":"2024-06-29T15:59:08.773676+0000","last_became_peered":"2024-06-29T15:59:08.773676+0000","last_unstale":"2024-06-29T15:59:08.776174+0000","last_undegraded":"2024-06-29T15:59:08.776174+0000","last_fullsized":"2024-06-29T15:59:08.776174+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:49.539717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.766227+0000","last_change":"2024-06-29T15:59:08.766227+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.766227+0000","last_undegraded":"2024-06-29T15:59:08.766227+0000","last_fullsized":"2024-06-29T15:59:08.766227+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.777083+0000","last_change":"2024-06-29T15:59:08.776465+0000","last_active":"2024-06-29T15:59:08.777083+0000","last_peered":"2024-06-29T15:59:08.777083+0000","last_clean":"2024-06-29T15:59:08.777083+0000","last_became_active":"2024-06-29T15:59:08.772049+0000","last_became_peered":"2024-06-29T15:59:08.772049+0000","last_unstale":"2024-06-29T15:59:08.777083+0000","last_undegraded":"2024-06-29T15:59:08.777083+0000","last_fullsized":"2024-06-29T15:59:08.777083+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:42.271085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.759828+0000","last_change":"2024-06-29T15:59:08.759828+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.759828+0000","last_undegraded":"2024-06-29T15:59:08.759828+0000","last_fullsized":"2024-06-29T15:59:08.759828+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.776308+0000","last_change":"2024-06-29T15:59:08.775719+0000","last_active":"2024-06-29T15:59:08.776308+0000","last_peered":"2024-06-29T15:59:08.776308+0000","last_clean":"2024-06-29T15:59:08.776308+0000","last_became_active":"2024-06-29T15:59:08.775391+0000","last_became_peered":"2024-06-29T15:59:08.775391+0000","last_unstale":"2024-06-29T15:59:08.776308+0000","last_undegraded":"2024-06-29T15:59:08.776308+0000","last_fullsized":"2024-06-29T15:59:08.776308+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:19.034925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.761158+0000","last_change":"2024-06-29T15:59:08.761158+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.761158+0000","last_undegraded":"2024-06-29T15:59:08.761158+0000","last_fullsized":"2024-06-29T15:59:08.761158+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.775978+0000","last_change":"2024-06-29T15:59:08.775491+0000","last_active":"2024-06-29T15:59:08.775978+0000","last_peered":"2024-06-29T15:59:08.775978+0000","last_clean":"2024-06-29T15:59:08.775978+0000","last_became_active":"2024-06-29T15:59:08.775156+0000","last_became_peered":"2024-06-29T15:59:08.775156+0000","last_unstale":"2024-06-29T15:59:08.775978+0000","last_undegraded":"2024-06-29T15:59:08.775978+0000","last_fullsized":"2024-06-29T15:59:08.775978+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:59.891498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.759829+0000","last_change":"2024-06-29T15:59:08.759829+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.759829+0000","last_undegraded":"2024-06-29T15:59:08.759829+0000","last_fullsized":"2024-06-29T15:59:08.759829+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.770468+0000","last_change":"2024-06-29T15:59:08.765586+0000","last_active":"2024-06-29T15:59:08.770468+0000","last_peered":"2024-06-29T15:59:08.770468+0000","last_clean":"2024-06-29T15:59:08.770468+0000","last_became_active":"2024-06-29T15:59:08.765325+0000","last_became_peered":"2024-06-29T15:59:08.765325+0000","last_unstale":"2024-06-29T15:59:08.770468+0000","last_undegraded":"2024-06-29T15:59:08.770468+0000","last_fullsized":"2024-06-29T15:59:08.770468+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:01.720943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.761703+0000","last_change":"2024-06-29T15:59:08.761703+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.761703+0000","last_undegraded":"2024-06-29T15:59:08.761703+0000","last_fullsized":"2024-06-29T15:59:08.761703+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.777977+0000","last_change":"2024-06-29T15:59:08.776661+0000","last_active":"2024-06-29T15:59:08.777977+0000","last_peered":"2024-06-29T15:59:08.777977+0000","last_clean":"2024-06-29T15:59:08.777977+0000","last_became_active":"2024-06-29T15:59:08.776340+0000","last_became_peered":"2024-06-29T15:59:08.776340+0000","last_unstale":"2024-06-29T15:59:08.777977+0000","last_undegraded":"2024-06-29T15:59:08.777977+0000","last_fullsized":"2024-06-29T15:59:08.777977+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:07.911957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760931+0000","last_change":"2024-06-29T15:59:08.760931+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760931+0000","last_undegraded":"2024-06-29T15:59:08.760931+0000","last_fullsized":"2024-06-29T15:59:08.760931+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.774580+0000","last_change":"2024-06-29T15:59:08.773014+0000","last_active":"2024-06-29T15:59:08.774580+0000","last_peered":"2024-06-29T15:59:08.774580+0000","last_clean":"2024-06-29T15:59:08.774580+0000","last_became_active":"2024-06-29T15:59:08.772737+0000","last_became_peered":"2024-06-29T15:59:08.772737+0000","last_unstale":"2024-06-29T15:59:08.774580+0000","last_undegraded":"2024-06-29T15:59:08.774580+0000","last_fullsized":"2024-06-29T15:59:08.774580+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:19.034925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.761800+0000","last_change":"2024-06-29T15:59:08.761800+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.761800+0000","last_undegraded":"2024-06-29T15:59:08.761800+0000","last_fullsized":"2024-06-29T15:59:08.761800+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.778703+0000","last_change":"2024-06-29T15:59:08.778237+0000","last_active":"2024-06-29T15:59:08.778703+0000","last_peered":"2024-06-29T15:59:08.778703+0000","last_clean":"2024-06-29T15:59:08.778703+0000","last_became_active":"2024-06-29T15:59:08.777914+0000","last_became_peered":"2024-06-29T15:59:08.777914+0000","last_unstale":"2024-06-29T15:59:08.778703+0000","last_undegraded":"2024-06-29T15:59:08.778703+0000","last_fullsized":"2024-06-29T15:59:08.778703+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:00:34.913944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.762046+0000","last_change":"2024-06-29T15:59:08.762046+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.762046+0000","last_undegraded":"2024-06-29T15:59:08.762046+0000","last_fullsized":"2024-06-29T15:59:08.762046+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.770702+0000","last_change":"2024-06-29T15:59:08.764880+0000","last_active":"2024-06-29T15:59:08.770702+0000","last_peered":"2024-06-29T15:59:08.770702+0000","last_clean":"2024-06-29T15:59:08.770702+0000","last_became_active":"2024-06-29T15:59:08.764548+0000","last_became_peered":"2024-06-29T15:59:08.764548+0000","last_unstale":"2024-06-29T15:59:08.770702+0000","last_undegraded":"2024-06-29T15:59:08.770702+0000","last_fullsized":"2024-06-29T15:59:08.770702+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:34:47.270529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.758295+0000","last_change":"2024-06-29T15:59:08.758295+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.758295+0000","last_undegraded":"2024-06-29T15:59:08.758295+0000","last_fullsized":"2024-06-29T15:59:08.758295+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.778671+0000","last_change":"2024-06-29T15:59:08.777696+0000","last_active":"2024-06-29T15:59:08.778671+0000","last_peered":"2024-06-29T15:59:08.778671+0000","last_clean":"2024-06-29T15:59:08.778671+0000","last_became_active":"2024-06-29T15:59:08.777360+0000","last_became_peered":"2024-06-29T15:59:08.777360+0000","last_unstale":"2024-06-29T15:59:08.778671+0000","last_undegraded":"2024-06-29T15:59:08.778671+0000","last_fullsized":"2024-06-29T15:59:08.778671+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:21.281317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.767316+0000","last_change":"2024-06-29T15:59:08.767316+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.767316+0000","last_undegraded":"2024-06-29T15:59:08.767316+0000","last_fullsized":"2024-06-29T15:59:08.767316+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.771558+0000","last_change":"2024-06-29T15:59:08.769792+0000","last_active":"2024-06-29T15:59:08.771558+0000","last_peered":"2024-06-29T15:59:08.771558+0000","last_clean":"2024-06-29T15:59:08.771558+0000","last_became_active":"2024-06-29T15:59:08.769493+0000","last_became_peered":"2024-06-29T15:59:08.769493+0000","last_unstale":"2024-06-29T15:59:08.771558+0000","last_undegraded":"2024-06-29T15:59:08.771558+0000","last_fullsized":"2024-06-29T15:59:08.771558+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:25.589856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.758565+0000","last_change":"2024-06-29T15:59:08.758565+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.758565+0000","last_undegraded":"2024-06-29T15:59:08.758565+0000","last_fullsized":"2024-06-29T15:59:08.758565+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.773898+0000","last_change":"2024-06-29T15:59:08.772761+0000","last_active":"2024-06-29T15:59:08.773898+0000","last_peered":"2024-06-29T15:59:08.773898+0000","last_clean":"2024-06-29T15:59:08.773898+0000","last_became_active":"2024-06-29T15:59:08.772359+0000","last_became_peered":"2024-06-29T15:59:08.772359+0000","last_unstale":"2024-06-29T15:59:08.773898+0000","last_undegraded":"2024-06-29T15:59:08.773898+0000","last_fullsized":"2024-06-29T15:59:08.773898+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:37.003253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760157+0000","last_change":"2024-06-29T15:59:08.760157+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760157+0000","last_undegraded":"2024-06-29T15:59:08.760157+0000","last_fullsized":"2024-06-29T15:59:08.760157+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.775607+0000","last_change":"2024-06-29T15:59:08.774345+0000","last_active":"2024-06-29T15:59:08.775607+0000","last_peered":"2024-06-29T15:59:08.775607+0000","last_clean":"2024-06-29T15:59:08.775607+0000","last_became_active":"2024-06-29T15:59:08.774032+0000","last_became_peered":"2024-06-29T15:59:08.774032+0000","last_unstale":"2024-06-29T15:59:08.775607+0000","last_undegraded":"2024-06-29T15:59:08.775607+0000","last_fullsized":"2024-06-29T15:59:08.775607+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:22:59.891498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.763760+0000","last_change":"2024-06-29T15:59:08.763760+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.763760+0000","last_undegraded":"2024-06-29T15:59:08.763760+0000","last_fullsized":"2024-06-29T15:59:08.763760+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.769522+0000","last_change":"2024-06-29T15:59:08.768134+0000","last_active":"2024-06-29T15:59:08.769522+0000","last_peered":"2024-06-29T15:59:08.769522+0000","last_clean":"2024-06-29T15:59:08.769522+0000","last_became_active":"2024-06-29T15:59:08.767829+0000","last_became_peered":"2024-06-29T15:59:08.767829+0000","last_unstale":"2024-06-29T15:59:08.769522+0000","last_undegraded":"2024-06-29T15:59:08.769522+0000","last_fullsized":"2024-06-29T15:59:08.769522+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:45.951786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.770533+0000","last_change":"2024-06-29T15:59:08.770533+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.770533+0000","last_undegraded":"2024-06-29T15:59:08.770533+0000","last_fullsized":"2024-06-29T15:59:08.770533+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.775152+0000","last_change":"2024-06-29T15:59:08.774042+0000","last_active":"2024-06-29T15:59:08.775152+0000","last_peered":"2024-06-29T15:59:08.775152+0000","last_clean":"2024-06-29T15:59:08.775152+0000","last_became_active":"2024-06-29T15:59:08.773566+0000","last_became_peered":"2024-06-29T15:59:08.773566+0000","last_unstale":"2024-06-29T15:59:08.775152+0000","last_undegraded":"2024-06-29T15:59:08.775152+0000","last_fullsized":"2024-06-29T15:59:08.775152+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:15:22.484350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760146+0000","last_change":"2024-06-29T15:59:08.760146+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760146+0000","last_undegraded":"2024-06-29T15:59:08.760146+0000","last_fullsized":"2024-06-29T15:59:08.760146+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.771423+0000","last_change":"2024-06-29T15:59:08.770486+0000","last_active":"2024-06-29T15:59:08.771423+0000","last_peered":"2024-06-29T15:59:08.771423+0000","last_clean":"2024-06-29T15:59:08.771423+0000","last_became_active":"2024-06-29T15:59:08.770195+0000","last_became_peered":"2024-06-29T15:59:08.770195+0000","last_unstale":"2024-06-29T15:59:08.771423+0000","last_undegraded":"2024-06-29T15:59:08.771423+0000","last_fullsized":"2024-06-29T15:59:08.771423+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:11:11.264284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760196+0000","last_change":"2024-06-29T15:59:08.760196+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760196+0000","last_undegraded":"2024-06-29T15:59:08.760196+0000","last_fullsized":"2024-06-29T15:59:08.760196+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.775951+0000","last_change":"2024-06-29T15:59:08.774427+0000","last_active":"2024-06-29T15:59:08.775951+0000","last_peered":"2024-06-29T15:59:08.775951+0000","last_clean":"2024-06-29T15:59:08.775951+0000","last_became_active":"2024-06-29T15:59:08.774014+0000","last_became_peered":"2024-06-29T15:59:08.774014+0000","last_unstale":"2024-06-29T15:59:08.775951+0000","last_undegraded":"2024-06-29T15:59:08.775951+0000","last_fullsized":"2024-06-29T15:59:08.775951+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:45.890236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.759251+0000","last_change":"2024-06-29T15:59:08.759251+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.759251+0000","last_undegraded":"2024-06-29T15:59:08.759251+0000","last_fullsized":"2024-06-29T15:59:08.759251+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.768384+0000","last_change":"2024-06-29T15:59:08.761992+0000","last_active":"2024-06-29T15:59:08.768384+0000","last_peered":"2024-06-29T15:59:08.768384+0000","last_clean":"2024-06-29T15:59:08.768384+0000","last_became_active":"2024-06-29T15:59:08.761708+0000","last_became_peered":"2024-06-29T15:59:08.761708+0000","last_unstale":"2024-06-29T15:59:08.768384+0000","last_undegraded":"2024-06-29T15:59:08.768384+0000","last_fullsized":"2024-06-29T15:59:08.768384+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:24:34.158626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.762740+0000","last_change":"2024-06-29T15:59:08.762740+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.762740+0000","last_undegraded":"2024-06-29T15:59:08.762740+0000","last_fullsized":"2024-06-29T15:59:08.762740+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.773969+0000","last_change":"2024-06-29T15:59:08.771423+0000","last_active":"2024-06-29T15:59:08.773969+0000","last_peered":"2024-06-29T15:59:08.773969+0000","last_clean":"2024-06-29T15:59:08.773969+0000","last_became_active":"2024-06-29T15:59:08.771099+0000","last_became_peered":"2024-06-29T15:59:08.771099+0000","last_unstale":"2024-06-29T15:59:08.773969+0000","last_undegraded":"2024-06-29T15:59:08.773969+0000","last_fullsized":"2024-06-29T15:59:08.773969+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:35:49.777368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760635+0000","last_change":"2024-06-29T15:59:08.760635+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760635+0000","last_undegraded":"2024-06-29T15:59:08.760635+0000","last_fullsized":"2024-06-29T15:59:08.760635+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.776465+0000","last_change":"2024-06-29T15:59:08.775516+0000","last_active":"2024-06-29T15:59:08.776465+0000","last_peered":"2024-06-29T15:59:08.776465+0000","last_clean":"2024-06-29T15:59:08.776465+0000","last_became_active":"2024-06-29T15:59:08.775199+0000","last_became_peered":"2024-06-29T15:59:08.775199+0000","last_unstale":"2024-06-29T15:59:08.776465+0000","last_undegraded":"2024-06-29T15:59:08.776465+0000","last_fullsized":"2024-06-29T15:59:08.776465+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:25.456471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.761630+0000","last_change":"2024-06-29T15:59:08.761630+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.761630+0000","last_undegraded":"2024-06-29T15:59:08.761630+0000","last_fullsized":"2024-06-29T15:59:08.761630+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.774550+0000","last_change":"2024-06-29T15:59:08.772988+0000","last_active":"2024-06-29T15:59:08.774550+0000","last_peered":"2024-06-29T15:59:08.774550+0000","last_clean":"2024-06-29T15:59:08.774550+0000","last_became_active":"2024-06-29T15:59:08.772622+0000","last_became_peered":"2024-06-29T15:59:08.772622+0000","last_unstale":"2024-06-29T15:59:08.774550+0000","last_undegraded":"2024-06-29T15:59:08.774550+0000","last_fullsized":"2024-06-29T15:59:08.774550+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:25.589856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.762161+0000","last_change":"2024-06-29T15:59:08.762161+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.762161+0000","last_undegraded":"2024-06-29T15:59:08.762161+0000","last_fullsized":"2024-06-29T15:59:08.762161+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.777001+0000","last_change":"2024-06-29T15:59:08.774824+0000","last_active":"2024-06-29T15:59:08.777001+0000","last_peered":"2024-06-29T15:59:08.777001+0000","last_clean":"2024-06-29T15:59:08.777001+0000","last_became_active":"2024-06-29T15:59:08.772415+0000","last_became_peered":"2024-06-29T15:59:08.772415+0000","last_unstale":"2024-06-29T15:59:08.777001+0000","last_undegraded":"2024-06-29T15:59:08.777001+0000","last_fullsized":"2024-06-29T15:59:08.777001+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:02:44.080415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.764748+0000","last_change":"2024-06-29T15:59:08.764748+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.764748+0000","last_undegraded":"2024-06-29T15:59:08.764748+0000","last_fullsized":"2024-06-29T15:59:08.764748+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.772705+0000","last_change":"2024-06-29T15:59:08.769738+0000","last_active":"2024-06-29T15:59:08.772705+0000","last_peered":"2024-06-29T15:59:08.772705+0000","last_clean":"2024-06-29T15:59:08.772705+0000","last_became_active":"2024-06-29T15:59:08.769354+0000","last_became_peered":"2024-06-29T15:59:08.769354+0000","last_unstale":"2024-06-29T15:59:08.772705+0000","last_undegraded":"2024-06-29T15:59:08.772705+0000","last_fullsized":"2024-06-29T15:59:08.772705+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:57:38.656411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.765368+0000","last_change":"2024-06-29T15:59:08.765368+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.765368+0000","last_undegraded":"2024-06-29T15:59:08.765368+0000","last_fullsized":"2024-06-29T15:59:08.765368+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.775020+0000","last_change":"2024-06-29T15:59:08.774410+0000","last_active":"2024-06-29T15:59:08.775020+0000","last_peered":"2024-06-29T15:59:08.775020+0000","last_clean":"2024-06-29T15:59:08.775020+0000","last_became_active":"2024-06-29T15:59:08.774125+0000","last_became_peered":"2024-06-29T15:59:08.774125+0000","last_unstale":"2024-06-29T15:59:08.775020+0000","last_undegraded":"2024-06-29T15:59:08.775020+0000","last_fullsized":"2024-06-29T15:59:08.775020+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:05.079724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760838+0000","last_change":"2024-06-29T15:59:08.760838+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760838+0000","last_undegraded":"2024-06-29T15:59:08.760838+0000","last_fullsized":"2024-06-29T15:59:08.760838+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.776815+0000","last_change":"2024-06-29T15:59:08.775359+0000","last_active":"2024-06-29T15:59:08.776815+0000","last_peered":"2024-06-29T15:59:08.776815+0000","last_clean":"2024-06-29T15:59:08.776815+0000","last_became_active":"2024-06-29T15:59:08.775033+0000","last_became_peered":"2024-06-29T15:59:08.775033+0000","last_unstale":"2024-06-29T15:59:08.776815+0000","last_undegraded":"2024-06-29T15:59:08.776815+0000","last_fullsized":"2024-06-29T15:59:08.776815+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:04.936887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.763313+0000","last_change":"2024-06-29T15:59:08.763313+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.763313+0000","last_undegraded":"2024-06-29T15:59:08.763313+0000","last_fullsized":"2024-06-29T15:59:08.763313+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.774539+0000","last_change":"2024-06-29T15:59:08.772572+0000","last_active":"2024-06-29T15:59:08.774539+0000","last_peered":"2024-06-29T15:59:08.774539+0000","last_clean":"2024-06-29T15:59:08.774539+0000","last_became_active":"2024-06-29T15:59:08.772186+0000","last_became_peered":"2024-06-29T15:59:08.772186+0000","last_unstale":"2024-06-29T15:59:08.774539+0000","last_undegraded":"2024-06-29T15:59:08.774539+0000","last_fullsized":"2024-06-29T15:59:08.774539+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:55:01.218523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760617+0000","last_change":"2024-06-29T15:59:08.760617+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760617+0000","last_undegraded":"2024-06-29T15:59:08.760617+0000","last_fullsized":"2024-06-29T15:59:08.760617+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.778267+0000","last_change":"2024-06-29T15:59:08.777539+0000","last_active":"2024-06-29T15:59:08.778267+0000","last_peered":"2024-06-29T15:59:08.778267+0000","last_clean":"2024-06-29T15:59:08.778267+0000","last_became_active":"2024-06-29T15:59:08.777188+0000","last_became_peered":"2024-06-29T15:59:08.777188+0000","last_unstale":"2024-06-29T15:59:08.778267+0000","last_undegraded":"2024-06-29T15:59:08.778267+0000","last_fullsized":"2024-06-29T15:59:08.778267+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:37.145355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760145+0000","last_change":"2024-06-29T15:59:08.760145+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760145+0000","last_undegraded":"2024-06-29T15:59:08.760145+0000","last_fullsized":"2024-06-29T15:59:08.760145+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.778222+0000","last_change":"2024-06-29T15:59:08.776838+0000","last_active":"2024-06-29T15:59:08.778222+0000","last_peered":"2024-06-29T15:59:08.778222+0000","last_clean":"2024-06-29T15:59:08.778222+0000","last_became_active":"2024-06-29T15:59:08.775875+0000","last_became_peered":"2024-06-29T15:59:08.775875+0000","last_unstale":"2024-06-29T15:59:08.778222+0000","last_undegraded":"2024-06-29T15:59:08.778222+0000","last_fullsized":"2024-06-29T15:59:08.778222+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:08.669425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.760950+0000","last_change":"2024-06-29T15:59:08.760950+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.760950+0000","last_undegraded":"2024-06-29T15:59:08.760950+0000","last_fullsized":"2024-06-29T15:59:08.760950+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.770377+0000","last_change":"2024-06-29T15:59:08.769362+0000","last_active":"2024-06-29T15:59:08.770377+0000","last_peered":"2024-06-29T15:59:08.770377+0000","last_clean":"2024-06-29T15:59:08.770377+0000","last_became_active":"2024-06-29T15:59:08.769060+0000","last_became_peered":"2024-06-29T15:59:08.769060+0000","last_unstale":"2024-06-29T15:59:08.770377+0000","last_undegraded":"2024-06-29T15:59:08.770377+0000","last_fullsized":"2024-06-29T15:59:08.770377+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:11:11.264284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.759117+0000","last_change":"2024-06-29T15:59:08.759117+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.759117+0000","last_undegraded":"2024-06-29T15:59:08.759117+0000","last_fullsized":"2024-06-29T15:59:08.759117+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.775043+0000","last_change":"2024-06-29T15:59:08.774326+0000","last_active":"2024-06-29T15:59:08.775043+0000","last_peered":"2024-06-29T15:59:08.775043+0000","last_clean":"2024-06-29T15:59:08.775043+0000","last_became_active":"2024-06-29T15:59:08.773514+0000","last_became_peered":"2024-06-29T15:59:08.773514+0000","last_unstale":"2024-06-29T15:59:08.775043+0000","last_undegraded":"2024-06-29T15:59:08.775043+0000","last_fullsized":"2024-06-29T15:59:08.775043+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:42.271085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.761532+0000","last_change":"2024-06-29T15:59:08.761532+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.761532+0000","last_undegraded":"2024-06-29T15:59:08.761532+0000","last_fullsized":"2024-06-29T15:59:08.761532+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.774559+0000","last_change":"2024-06-29T15:59:08.771871+0000","last_active":"2024-06-29T15:59:08.774559+0000","last_peered":"2024-06-29T15:59:08.774559+0000","last_clean":"2024-06-29T15:59:08.774559+0000","last_became_active":"2024-06-29T15:59:08.771544+0000","last_became_peered":"2024-06-29T15:59:08.771544+0000","last_unstale":"2024-06-29T15:59:08.774559+0000","last_undegraded":"2024-06-29T15:59:08.774559+0000","last_fullsized":"2024-06-29T15:59:08.774559+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:14.640090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.758343+0000","last_change":"2024-06-29T15:59:08.758343+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.758343+0000","last_undegraded":"2024-06-29T15:59:08.758343+0000","last_fullsized":"2024-06-29T15:59:08.758343+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.770187+0000","last_change":"2024-06-29T15:59:08.764007+0000","last_active":"2024-06-29T15:59:08.770187+0000","last_peered":"2024-06-29T15:59:08.770187+0000","last_clean":"2024-06-29T15:59:08.770187+0000","last_became_active":"2024-06-29T15:59:08.763651+0000","last_became_peered":"2024-06-29T15:59:08.763651+0000","last_unstale":"2024-06-29T15:59:08.770187+0000","last_undegraded":"2024-06-29T15:59:08.770187+0000","last_fullsized":"2024-06-29T15:59:08.770187+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:01.822059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":1,"reported_epoch":251,"state":"creating+peering","last_fresh":"2024-06-29T15:59:08.764871+0000","last_change":"2024-06-29T15:59:08.764871+0000","last_active":"2024-06-29T15:59:08.750765+0000","last_peered":"2024-06-29T15:59:08.750765+0000","last_clean":"2024-06-29T15:59:08.750765+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-29T15:59:08.764871+0000","last_undegraded":"2024-06-29T15:59:08.764871+0000","last_fullsized":"2024-06-29T15:59:08.764871+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"239'247","reported_seq":1211,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-06-29T15:59:08.771489+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:59:08.771489+0000","last_peered":"2024-06-29T15:59:08.771489+0000","last_clean":"2024-06-29T15:59:08.771489+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:59:08.771489+0000","last_undegraded":"2024-06-29T15:59:08.771489+0000","last_fullsized":"2024-06-29T15:59:08.771489+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247,"ondisk_log_size":247,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576005,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52896,"kb_used_data":3668,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318944,"statfs":{"total":96636764160,"available":96582598656,"internally_reserved":0,"allocated":3756032,"data_stored":2152079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834576006,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60636,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311204,"statfs":{"total":96636764160,"available":96574672896,"internally_reserved":0,"allocated":3162112,"data_stored":1561711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834576004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30108,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341732,"statfs":{"total":96636764160,"available":96605933568,"internally_reserved":0,"allocated":3162112,"data_stored":1561711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834576005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78636,"kb_used_data":3100,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94293204,"statfs":{"total":96636764160,"available":96556240896,"internally_reserved":0,"allocated":3174400,"data_stored":1568114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834576005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48168,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323672,"statfs":{"total":96636764160,"available":96587440128,"internally_reserved":0,"allocated":3162112,"data_stored":1561711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48164,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323676,"statfs":{"total":96636764160,"available":96587444224,"internally_reserved":0,"allocated":3162112,"data_stored":1561711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834576004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64796,"kb_used_data":3088,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307044,"statfs":{"total":96636764160,"available":96570413056,"internally_reserved":0,"allocated":3162112,"data_stored":1561711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834576005,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61220,"kb_used_data":3668,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310620,"statfs":{"total":96636764160,"available":96574074880,"internally_reserved":0,"allocated":3756032,"data_stored":2152079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:59:13.818 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:59:13.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.966+0000 7fef8e14a640 1 -- 172.21.15.81:0/105975408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef88074520 msgr2=0x7fef8806f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.966+0000 7fef8e14a640 1 --2- 172.21.15.81:0/105975408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef88074520 0x7fef8806f140 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7fef78009830 tx=0x7fef7802a510 comp rx=0 tx=0).stop 2024-06-29T15:59:13.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.966+0000 7fef8e14a640 1 -- 172.21.15.81:0/105975408 shutdown_connections 2024-06-29T15:59:13.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.966+0000 7fef8e14a640 1 --2- 172.21.15.81:0/105975408 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef8806f680 0x7fef8806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.966+0000 7fef8e14a640 1 --2- 172.21.15.81:0/105975408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef88074520 0x7fef8806f140 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.966+0000 7fef8e14a640 1 -- 172.21.15.81:0/105975408 >> 172.21.15.81:0/105975408 conn(0x7fef88072950 msgr2=0x7fef880b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:13.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.966+0000 7fef8e14a640 1 -- 172.21.15.81:0/105975408 shutdown_connections 2024-06-29T15:59:13.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.966+0000 7fef8e14a640 1 -- 172.21.15.81:0/105975408 wait complete. 2024-06-29T15:59:13.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.968+0000 7fef8e14a640 1 Processor -- start 2024-06-29T15:59:13.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.968+0000 7fef8e14a640 1 -- start start 2024-06-29T15:59:13.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.969+0000 7fef8e14a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef8806f680 0x7fef8809daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.969+0000 7fef8e14a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8806fa90 0x7fef881f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.969+0000 7fef8e14a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef88074520 0x7fef88147d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.969+0000 7fef8e14a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fef8815b010 con 0x7fef8806fa90 2024-06-29T15:59:13.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.969+0000 7fef8e14a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fef8806d150 con 0x7fef8806f680 2024-06-29T15:59:13.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.969+0000 7fef8e14a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fef8815b190 con 0x7fef88074520 2024-06-29T15:59:13.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.969+0000 7fef86ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8806fa90 0x7fef881f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.969+0000 7fef86ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8806fa90 0x7fef881f2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60332/0 (socket says 172.21.15.81:60332) 2024-06-29T15:59:13.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.969+0000 7fef86ffd640 1 -- 172.21.15.81:0/1563616339 learned_addr learned my addr 172.21.15.81:0/1563616339 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.969+0000 7fef87fff640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef88074520 0x7fef88147d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef877fe640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef8806f680 0x7fef8809daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef86ffd640 1 -- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef88074520 msgr2=0x7fef88147d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef86ffd640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef88074520 0x7fef88147d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef86ffd640 1 -- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef8806f680 msgr2=0x7fef8809daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef86ffd640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef8806f680 0x7fef8809daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef86ffd640 1 -- 172.21.15.81:0/1563616339 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8809c480 con 0x7fef8806fa90 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef86ffd640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8806fa90 0x7fef881f2e90 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00b440 tx=0x7fef7c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef6ffff640 1 -- 172.21.15.81:0/1563616339 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c015020 con 0x7fef8806fa90 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef6ffff640 1 -- 172.21.15.81:0/1563616339 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef7c011920 con 0x7fef8806fa90 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef8e14a640 1 -- 172.21.15.81:0/1563616339 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef881f33d0 con 0x7fef8806fa90 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef6ffff640 1 -- 172.21.15.81:0/1563616339 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c011d70 con 0x7fef8806fa90 2024-06-29T15:59:13.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.970+0000 7fef8e14a640 1 -- 172.21.15.81:0/1563616339 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8809dfe0 con 0x7fef8806fa90 2024-06-29T15:59:13.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.971+0000 7fef8e14a640 1 -- 172.21.15.81:0/1563616339 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef880cf7c0 con 0x7fef8806fa90 2024-06-29T15:59:13.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.972+0000 7fef6ffff640 1 -- 172.21.15.81:0/1563616339 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fef7c018020 con 0x7fef8806fa90 2024-06-29T15:59:13.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.972+0000 7fef6ffff640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fef74076510 0x7fef74078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:13.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.972+0000 7fef6ffff640 1 -- 172.21.15.81:0/1563616339 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7fef7c097c50 con 0x7fef8806fa90 2024-06-29T15:59:13.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.973+0000 7fef877fe640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fef74076510 0x7fef74078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:13.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.973+0000 7fef877fe640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fef74076510 0x7fef74078a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fef78034000 tx=0x7fef78004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:13.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:13.977+0000 7fef6ffff640 1 -- 172.21.15.81:0/1563616339 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fef7c09d020 con 0x7fef8806fa90 2024-06-29T15:59:14.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.156+0000 7fef8e14a640 1 -- 172.21.15.81:0/1563616339 --> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef8806cc80 con 0x7fef74076510 2024-06-29T15:59:14.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.162+0000 7fef6ffff640 1 -- 172.21.15.81:0/1563616339 <== mgr.11053 v2:172.21.15.167:6832/1778875525 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252500 (secure 0 0 0) 0x7fef8806cc80 con 0x7fef74076510 2024-06-29T15:59:14.166 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:14.167 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:59:14.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.164+0000 7fef8e14a640 1 -- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fef74076510 msgr2=0x7fef74078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.164+0000 7fef8e14a640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fef74076510 0x7fef74078a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fef78034000 tx=0x7fef78004730 comp rx=0 tx=0).stop 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.164+0000 7fef8e14a640 1 -- 172.21.15.81:0/1563616339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8806fa90 msgr2=0x7fef881f2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.164+0000 7fef8e14a640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8806fa90 0x7fef881f2e90 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00b440 tx=0x7fef7c00b900 comp rx=0 tx=0).stop 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.164+0000 7fef8e14a640 1 -- 172.21.15.81:0/1563616339 shutdown_connections 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.164+0000 7fef8e14a640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fef74076510 0x7fef74078a20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.164+0000 7fef8e14a640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef88074520 0x7fef88147d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.164+0000 7fef8e14a640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8806fa90 0x7fef881f2e90 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.164+0000 7fef8e14a640 1 --2- 172.21.15.81:0/1563616339 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef8806f680 0x7fef8809daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.164+0000 7fef8e14a640 1 -- 172.21.15.81:0/1563616339 >> 172.21.15.81:0/1563616339 conn(0x7fef88072950 msgr2=0x7fef881597b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.165+0000 7fef8e14a640 1 -- 172.21.15.81:0/1563616339 shutdown_connections 2024-06-29T15:59:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.165+0000 7fef8e14a640 1 -- 172.21.15.81:0/1563616339 wait complete. 2024-06-29T15:59:14.177 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-06-29T15:59:13.840821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":446,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":286,"ondisk_log_size":286,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":449108,"kb_used_data":26216,"kb_used_omap":12,"kb_used_meta":422771,"kb_avail":754525612,"statfs":{"total":773094113280,"available":772634226688,"internally_reserved":0,"allocated":26845184,"data_stored":13837116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":432918066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001290"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104146+0000","last_change":"2024-06-29T15:59:09.785489+0000","last_active":"2024-06-29T15:59:12.104146+0000","last_peered":"2024-06-29T15:59:12.104146+0000","last_clean":"2024-06-29T15:59:12.104146+0000","last_became_active":"2024-06-29T15:59:09.785187+0000","last_became_peered":"2024-06-29T15:59:09.785187+0000","last_unstale":"2024-06-29T15:59:12.104146+0000","last_undegraded":"2024-06-29T15:59:12.104146+0000","last_fullsized":"2024-06-29T15:59:12.104146+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:06:46.157914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104829+0000","last_change":"2024-06-29T15:59:09.781525+0000","last_active":"2024-06-29T15:59:12.104829+0000","last_peered":"2024-06-29T15:59:12.104829+0000","last_clean":"2024-06-29T15:59:12.104829+0000","last_became_active":"2024-06-29T15:59:09.781194+0000","last_became_peered":"2024-06-29T15:59:09.781194+0000","last_unstale":"2024-06-29T15:59:12.104829+0000","last_undegraded":"2024-06-29T15:59:12.104829+0000","last_fullsized":"2024-06-29T15:59:12.104829+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:39:08.464376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104062+0000","last_change":"2024-06-29T15:59:09.778134+0000","last_active":"2024-06-29T15:59:12.104062+0000","last_peered":"2024-06-29T15:59:12.104062+0000","last_clean":"2024-06-29T15:59:12.104062+0000","last_became_active":"2024-06-29T15:59:09.777824+0000","last_became_peered":"2024-06-29T15:59:09.777824+0000","last_unstale":"2024-06-29T15:59:12.104062+0000","last_undegraded":"2024-06-29T15:59:12.104062+0000","last_fullsized":"2024-06-29T15:59:12.104062+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:11.006888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.771774+0000","last_change":"2024-06-29T15:59:09.778100+0000","last_active":"2024-06-29T15:59:10.771774+0000","last_peered":"2024-06-29T15:59:10.771774+0000","last_clean":"2024-06-29T15:59:10.771774+0000","last_became_active":"2024-06-29T15:59:09.777745+0000","last_became_peered":"2024-06-29T15:59:09.777745+0000","last_unstale":"2024-06-29T15:59:10.771774+0000","last_undegraded":"2024-06-29T15:59:10.771774+0000","last_fullsized":"2024-06-29T15:59:10.771774+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:58:00.767085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104716+0000","last_change":"2024-06-29T15:59:09.777134+0000","last_active":"2024-06-29T15:59:12.104716+0000","last_peered":"2024-06-29T15:59:12.104716+0000","last_clean":"2024-06-29T15:59:12.104716+0000","last_became_active":"2024-06-29T15:59:09.776834+0000","last_became_peered":"2024-06-29T15:59:09.776834+0000","last_unstale":"2024-06-29T15:59:12.104716+0000","last_undegraded":"2024-06-29T15:59:12.104716+0000","last_fullsized":"2024-06-29T15:59:12.104716+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:34:58.101691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104266+0000","last_change":"2024-06-29T15:59:09.779729+0000","last_active":"2024-06-29T15:59:12.104266+0000","last_peered":"2024-06-29T15:59:12.104266+0000","last_clean":"2024-06-29T15:59:12.104266+0000","last_became_active":"2024-06-29T15:59:09.779429+0000","last_became_peered":"2024-06-29T15:59:09.779429+0000","last_unstale":"2024-06-29T15:59:12.104266+0000","last_undegraded":"2024-06-29T15:59:12.104266+0000","last_fullsized":"2024-06-29T15:59:12.104266+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:10.456357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603275+0000","last_change":"2024-06-29T15:59:09.786174+0000","last_active":"2024-06-29T15:59:11.603275+0000","last_peered":"2024-06-29T15:59:11.603275+0000","last_clean":"2024-06-29T15:59:11.603275+0000","last_became_active":"2024-06-29T15:59:09.785185+0000","last_became_peered":"2024-06-29T15:59:09.785185+0000","last_unstale":"2024-06-29T15:59:11.603275+0000","last_undegraded":"2024-06-29T15:59:11.603275+0000","last_fullsized":"2024-06-29T15:59:11.603275+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:20.440471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603155+0000","last_change":"2024-06-29T15:59:09.794217+0000","last_active":"2024-06-29T15:59:11.603155+0000","last_peered":"2024-06-29T15:59:11.603155+0000","last_clean":"2024-06-29T15:59:11.603155+0000","last_became_active":"2024-06-29T15:59:09.789268+0000","last_became_peered":"2024-06-29T15:59:09.789268+0000","last_unstale":"2024-06-29T15:59:11.603155+0000","last_undegraded":"2024-06-29T15:59:11.603155+0000","last_fullsized":"2024-06-29T15:59:11.603155+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:24.344770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.913351+0000","last_change":"2024-06-29T15:59:09.783199+0000","last_active":"2024-06-29T15:59:11.913351+0000","last_peered":"2024-06-29T15:59:11.913351+0000","last_clean":"2024-06-29T15:59:11.913351+0000","last_became_active":"2024-06-29T15:59:09.782923+0000","last_became_peered":"2024-06-29T15:59:09.782923+0000","last_unstale":"2024-06-29T15:59:11.913351+0000","last_undegraded":"2024-06-29T15:59:11.913351+0000","last_fullsized":"2024-06-29T15:59:11.913351+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:36:44.695387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772571+0000","last_change":"2024-06-29T15:59:09.793792+0000","last_active":"2024-06-29T15:59:10.772571+0000","last_peered":"2024-06-29T15:59:10.772571+0000","last_clean":"2024-06-29T15:59:10.772571+0000","last_became_active":"2024-06-29T15:59:09.793424+0000","last_became_peered":"2024-06-29T15:59:09.793424+0000","last_unstale":"2024-06-29T15:59:10.772571+0000","last_undegraded":"2024-06-29T15:59:10.772571+0000","last_fullsized":"2024-06-29T15:59:10.772571+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:20:09.791145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103910+0000","last_change":"2024-06-29T15:59:09.778350+0000","last_active":"2024-06-29T15:59:12.103910+0000","last_peered":"2024-06-29T15:59:12.103910+0000","last_clean":"2024-06-29T15:59:12.103910+0000","last_became_active":"2024-06-29T15:59:09.778030+0000","last_became_peered":"2024-06-29T15:59:09.778030+0000","last_unstale":"2024-06-29T15:59:12.103910+0000","last_undegraded":"2024-06-29T15:59:12.103910+0000","last_fullsized":"2024-06-29T15:59:12.103910+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:24:35.160605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104340+0000","last_change":"2024-06-29T15:59:09.777523+0000","last_active":"2024-06-29T15:59:12.104340+0000","last_peered":"2024-06-29T15:59:12.104340+0000","last_clean":"2024-06-29T15:59:12.104340+0000","last_became_active":"2024-06-29T15:59:09.777230+0000","last_became_peered":"2024-06-29T15:59:09.777230+0000","last_unstale":"2024-06-29T15:59:12.104340+0000","last_undegraded":"2024-06-29T15:59:12.104340+0000","last_fullsized":"2024-06-29T15:59:12.104340+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:58:46.953765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103895+0000","last_change":"2024-06-29T15:59:09.788681+0000","last_active":"2024-06-29T15:59:12.103895+0000","last_peered":"2024-06-29T15:59:12.103895+0000","last_clean":"2024-06-29T15:59:12.103895+0000","last_became_active":"2024-06-29T15:59:09.788333+0000","last_became_peered":"2024-06-29T15:59:09.788333+0000","last_unstale":"2024-06-29T15:59:12.103895+0000","last_undegraded":"2024-06-29T15:59:12.103895+0000","last_fullsized":"2024-06-29T15:59:12.103895+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:00:16.046141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.771516+0000","last_change":"2024-06-29T15:59:09.790401+0000","last_active":"2024-06-29T15:59:10.771516+0000","last_peered":"2024-06-29T15:59:10.771516+0000","last_clean":"2024-06-29T15:59:10.771516+0000","last_became_active":"2024-06-29T15:59:09.787504+0000","last_became_peered":"2024-06-29T15:59:09.787504+0000","last_unstale":"2024-06-29T15:59:10.771516+0000","last_undegraded":"2024-06-29T15:59:10.771516+0000","last_fullsized":"2024-06-29T15:59:10.771516+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:12.989334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603360+0000","last_change":"2024-06-29T15:59:09.785906+0000","last_active":"2024-06-29T15:59:11.603360+0000","last_peered":"2024-06-29T15:59:11.603360+0000","last_clean":"2024-06-29T15:59:11.603360+0000","last_became_active":"2024-06-29T15:59:09.784811+0000","last_became_peered":"2024-06-29T15:59:09.784811+0000","last_unstale":"2024-06-29T15:59:11.603360+0000","last_undegraded":"2024-06-29T15:59:11.603360+0000","last_fullsized":"2024-06-29T15:59:11.603360+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:24:08.597494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603212+0000","last_change":"2024-06-29T15:59:09.790145+0000","last_active":"2024-06-29T15:59:11.603212+0000","last_peered":"2024-06-29T15:59:11.603212+0000","last_clean":"2024-06-29T15:59:11.603212+0000","last_became_active":"2024-06-29T15:59:09.788909+0000","last_became_peered":"2024-06-29T15:59:09.788909+0000","last_unstale":"2024-06-29T15:59:11.603212+0000","last_undegraded":"2024-06-29T15:59:11.603212+0000","last_fullsized":"2024-06-29T15:59:11.603212+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:41:12.635773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.913510+0000","last_change":"2024-06-29T15:59:09.792776+0000","last_active":"2024-06-29T15:59:11.913510+0000","last_peered":"2024-06-29T15:59:11.913510+0000","last_clean":"2024-06-29T15:59:11.913510+0000","last_became_active":"2024-06-29T15:59:09.792451+0000","last_became_peered":"2024-06-29T15:59:09.792451+0000","last_unstale":"2024-06-29T15:59:11.913510+0000","last_undegraded":"2024-06-29T15:59:11.913510+0000","last_fullsized":"2024-06-29T15:59:11.913510+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:50:04.846487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772909+0000","last_change":"2024-06-29T15:59:09.800899+0000","last_active":"2024-06-29T15:59:10.772909+0000","last_peered":"2024-06-29T15:59:10.772909+0000","last_clean":"2024-06-29T15:59:10.772909+0000","last_became_active":"2024-06-29T15:59:09.800572+0000","last_became_peered":"2024-06-29T15:59:09.800572+0000","last_unstale":"2024-06-29T15:59:10.772909+0000","last_undegraded":"2024-06-29T15:59:10.772909+0000","last_fullsized":"2024-06-29T15:59:10.772909+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:50.541697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603661+0000","last_change":"2024-06-29T15:59:09.786627+0000","last_active":"2024-06-29T15:59:11.603661+0000","last_peered":"2024-06-29T15:59:11.603661+0000","last_clean":"2024-06-29T15:59:11.603661+0000","last_became_active":"2024-06-29T15:59:09.786152+0000","last_became_peered":"2024-06-29T15:59:09.786152+0000","last_unstale":"2024-06-29T15:59:11.603661+0000","last_undegraded":"2024-06-29T15:59:11.603661+0000","last_fullsized":"2024-06-29T15:59:11.603661+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:55:38.147334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603465+0000","last_change":"2024-06-29T15:59:09.789118+0000","last_active":"2024-06-29T15:59:11.603465+0000","last_peered":"2024-06-29T15:59:11.603465+0000","last_clean":"2024-06-29T15:59:11.603465+0000","last_became_active":"2024-06-29T15:59:09.788607+0000","last_became_peered":"2024-06-29T15:59:09.788607+0000","last_unstale":"2024-06-29T15:59:11.603465+0000","last_undegraded":"2024-06-29T15:59:11.603465+0000","last_fullsized":"2024-06-29T15:59:11.603465+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:06.155926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104112+0000","last_change":"2024-06-29T15:59:09.779269+0000","last_active":"2024-06-29T15:59:12.104112+0000","last_peered":"2024-06-29T15:59:12.104112+0000","last_clean":"2024-06-29T15:59:12.104112+0000","last_became_active":"2024-06-29T15:59:09.778926+0000","last_became_peered":"2024-06-29T15:59:09.778926+0000","last_unstale":"2024-06-29T15:59:12.104112+0000","last_undegraded":"2024-06-29T15:59:12.104112+0000","last_fullsized":"2024-06-29T15:59:12.104112+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:34:30.237702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104431+0000","last_change":"2024-06-29T15:59:09.799888+0000","last_active":"2024-06-29T15:59:12.104431+0000","last_peered":"2024-06-29T15:59:12.104431+0000","last_clean":"2024-06-29T15:59:12.104431+0000","last_became_active":"2024-06-29T15:59:09.799568+0000","last_became_peered":"2024-06-29T15:59:09.799568+0000","last_unstale":"2024-06-29T15:59:12.104431+0000","last_undegraded":"2024-06-29T15:59:12.104431+0000","last_fullsized":"2024-06-29T15:59:12.104431+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:13:42.054517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.767566+0000","last_change":"2024-06-29T15:59:09.789431+0000","last_active":"2024-06-29T15:59:10.767566+0000","last_peered":"2024-06-29T15:59:10.767566+0000","last_clean":"2024-06-29T15:59:10.767566+0000","last_became_active":"2024-06-29T15:59:09.789107+0000","last_became_peered":"2024-06-29T15:59:09.789107+0000","last_unstale":"2024-06-29T15:59:10.767566+0000","last_undegraded":"2024-06-29T15:59:10.767566+0000","last_fullsized":"2024-06-29T15:59:10.767566+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:28:33.673957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103733+0000","last_change":"2024-06-29T15:59:09.793973+0000","last_active":"2024-06-29T15:59:12.103733+0000","last_peered":"2024-06-29T15:59:12.103733+0000","last_clean":"2024-06-29T15:59:12.103733+0000","last_became_active":"2024-06-29T15:59:09.793641+0000","last_became_peered":"2024-06-29T15:59:09.793641+0000","last_unstale":"2024-06-29T15:59:12.103733+0000","last_undegraded":"2024-06-29T15:59:12.103733+0000","last_fullsized":"2024-06-29T15:59:12.103733+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:36:39.486061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.767767+0000","last_change":"2024-06-29T15:59:09.776591+0000","last_active":"2024-06-29T15:59:10.767767+0000","last_peered":"2024-06-29T15:59:10.767767+0000","last_clean":"2024-06-29T15:59:10.767767+0000","last_became_active":"2024-06-29T15:59:09.776235+0000","last_became_peered":"2024-06-29T15:59:09.776235+0000","last_unstale":"2024-06-29T15:59:10.767767+0000","last_undegraded":"2024-06-29T15:59:10.767767+0000","last_fullsized":"2024-06-29T15:59:10.767767+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:50:52.468828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104628+0000","last_change":"2024-06-29T15:59:09.777214+0000","last_active":"2024-06-29T15:59:12.104628+0000","last_peered":"2024-06-29T15:59:12.104628+0000","last_clean":"2024-06-29T15:59:12.104628+0000","last_became_active":"2024-06-29T15:59:09.776867+0000","last_became_peered":"2024-06-29T15:59:09.776867+0000","last_unstale":"2024-06-29T15:59:12.104628+0000","last_undegraded":"2024-06-29T15:59:12.104628+0000","last_fullsized":"2024-06-29T15:59:12.104628+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:00.893477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603608+0000","last_change":"2024-06-29T15:59:09.799878+0000","last_active":"2024-06-29T15:59:11.603608+0000","last_peered":"2024-06-29T15:59:11.603608+0000","last_clean":"2024-06-29T15:59:11.603608+0000","last_became_active":"2024-06-29T15:59:09.799575+0000","last_became_peered":"2024-06-29T15:59:09.799575+0000","last_unstale":"2024-06-29T15:59:11.603608+0000","last_undegraded":"2024-06-29T15:59:11.603608+0000","last_fullsized":"2024-06-29T15:59:11.603608+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:09.671404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.771991+0000","last_change":"2024-06-29T15:59:09.796652+0000","last_active":"2024-06-29T15:59:10.771991+0000","last_peered":"2024-06-29T15:59:10.771991+0000","last_clean":"2024-06-29T15:59:10.771991+0000","last_became_active":"2024-06-29T15:59:09.796290+0000","last_became_peered":"2024-06-29T15:59:09.796290+0000","last_unstale":"2024-06-29T15:59:10.771991+0000","last_undegraded":"2024-06-29T15:59:10.771991+0000","last_fullsized":"2024-06-29T15:59:10.771991+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:33:18.395624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603352+0000","last_change":"2024-06-29T15:59:09.785718+0000","last_active":"2024-06-29T15:59:11.603352+0000","last_peered":"2024-06-29T15:59:11.603352+0000","last_clean":"2024-06-29T15:59:11.603352+0000","last_became_active":"2024-06-29T15:59:09.785397+0000","last_became_peered":"2024-06-29T15:59:09.785397+0000","last_unstale":"2024-06-29T15:59:11.603352+0000","last_undegraded":"2024-06-29T15:59:11.603352+0000","last_fullsized":"2024-06-29T15:59:11.603352+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:08:44.669009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603763+0000","last_change":"2024-06-29T15:59:09.785611+0000","last_active":"2024-06-29T15:59:11.603763+0000","last_peered":"2024-06-29T15:59:11.603763+0000","last_clean":"2024-06-29T15:59:11.603763+0000","last_became_active":"2024-06-29T15:59:09.784462+0000","last_became_peered":"2024-06-29T15:59:09.784462+0000","last_unstale":"2024-06-29T15:59:11.603763+0000","last_undegraded":"2024-06-29T15:59:11.603763+0000","last_fullsized":"2024-06-29T15:59:11.603763+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:30.894568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.913227+0000","last_change":"2024-06-29T15:59:09.782393+0000","last_active":"2024-06-29T15:59:11.913227+0000","last_peered":"2024-06-29T15:59:11.913227+0000","last_clean":"2024-06-29T15:59:11.913227+0000","last_became_active":"2024-06-29T15:59:09.782101+0000","last_became_peered":"2024-06-29T15:59:09.782101+0000","last_unstale":"2024-06-29T15:59:11.913227+0000","last_undegraded":"2024-06-29T15:59:11.913227+0000","last_fullsized":"2024-06-29T15:59:11.913227+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:33:18.395624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104500+0000","last_change":"2024-06-29T15:59:09.801861+0000","last_active":"2024-06-29T15:59:12.104500+0000","last_peered":"2024-06-29T15:59:12.104500+0000","last_clean":"2024-06-29T15:59:12.104500+0000","last_became_active":"2024-06-29T15:59:09.801519+0000","last_became_peered":"2024-06-29T15:59:09.801519+0000","last_unstale":"2024-06-29T15:59:12.104500+0000","last_undegraded":"2024-06-29T15:59:12.104500+0000","last_fullsized":"2024-06-29T15:59:12.104500+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:11:12.266264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"252'4","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104342+0000","last_change":"2024-06-29T15:59:08.774828+0000","last_active":"2024-06-29T15:59:12.104342+0000","last_peered":"2024-06-29T15:59:12.104342+0000","last_clean":"2024-06-29T15:59:12.104342+0000","last_became_active":"2024-06-29T15:59:08.773676+0000","last_became_peered":"2024-06-29T15:59:08.773676+0000","last_unstale":"2024-06-29T15:59:12.104342+0000","last_undegraded":"2024-06-29T15:59:12.104342+0000","last_fullsized":"2024-06-29T15:59:12.104342+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:57:59.765105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104025+0000","last_change":"2024-06-29T15:59:09.779848+0000","last_active":"2024-06-29T15:59:12.104025+0000","last_peered":"2024-06-29T15:59:12.104025+0000","last_clean":"2024-06-29T15:59:12.104025+0000","last_became_active":"2024-06-29T15:59:09.778823+0000","last_became_peered":"2024-06-29T15:59:09.778823+0000","last_unstale":"2024-06-29T15:59:12.104025+0000","last_undegraded":"2024-06-29T15:59:12.104025+0000","last_fullsized":"2024-06-29T15:59:12.104025+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:02.824039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"252'1","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.913549+0000","last_change":"2024-06-29T15:59:08.776465+0000","last_active":"2024-06-29T15:59:11.913549+0000","last_peered":"2024-06-29T15:59:11.913549+0000","last_clean":"2024-06-29T15:59:11.913549+0000","last_became_active":"2024-06-29T15:59:08.772049+0000","last_became_peered":"2024-06-29T15:59:08.772049+0000","last_unstale":"2024-06-29T15:59:11.913549+0000","last_undegraded":"2024-06-29T15:59:11.913549+0000","last_fullsized":"2024-06-29T15:59:11.913549+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:29:49.539717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104245+0000","last_change":"2024-06-29T15:59:09.778061+0000","last_active":"2024-06-29T15:59:12.104245+0000","last_peered":"2024-06-29T15:59:12.104245+0000","last_clean":"2024-06-29T15:59:12.104245+0000","last_became_active":"2024-06-29T15:59:09.777536+0000","last_became_peered":"2024-06-29T15:59:09.777536+0000","last_unstale":"2024-06-29T15:59:12.104245+0000","last_undegraded":"2024-06-29T15:59:12.104245+0000","last_fullsized":"2024-06-29T15:59:12.104245+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:08.844426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"252'2","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.913322+0000","last_change":"2024-06-29T15:59:08.775719+0000","last_active":"2024-06-29T15:59:11.913322+0000","last_peered":"2024-06-29T15:59:11.913322+0000","last_clean":"2024-06-29T15:59:11.913322+0000","last_became_active":"2024-06-29T15:59:08.775391+0000","last_became_peered":"2024-06-29T15:59:08.775391+0000","last_unstale":"2024-06-29T15:59:11.913322+0000","last_undegraded":"2024-06-29T15:59:11.913322+0000","last_fullsized":"2024-06-29T15:59:11.913322+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:05.079724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104222+0000","last_change":"2024-06-29T15:59:09.778314+0000","last_active":"2024-06-29T15:59:12.104222+0000","last_peered":"2024-06-29T15:59:12.104222+0000","last_clean":"2024-06-29T15:59:12.104222+0000","last_became_active":"2024-06-29T15:59:09.777177+0000","last_became_peered":"2024-06-29T15:59:09.777177+0000","last_unstale":"2024-06-29T15:59:12.104222+0000","last_undegraded":"2024-06-29T15:59:12.104222+0000","last_fullsized":"2024-06-29T15:59:12.104222+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:17:17.089248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"252'2","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603578+0000","last_change":"2024-06-29T15:59:08.775491+0000","last_active":"2024-06-29T15:59:11.603578+0000","last_peered":"2024-06-29T15:59:11.603578+0000","last_clean":"2024-06-29T15:59:11.603578+0000","last_became_active":"2024-06-29T15:59:08.775156+0000","last_became_peered":"2024-06-29T15:59:08.775156+0000","last_unstale":"2024-06-29T15:59:11.603578+0000","last_undegraded":"2024-06-29T15:59:11.603578+0000","last_fullsized":"2024-06-29T15:59:11.603578+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:56:12.992896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.767789+0000","last_change":"2024-06-29T15:59:09.778270+0000","last_active":"2024-06-29T15:59:10.767789+0000","last_peered":"2024-06-29T15:59:10.767789+0000","last_clean":"2024-06-29T15:59:10.767789+0000","last_became_active":"2024-06-29T15:59:09.777929+0000","last_became_peered":"2024-06-29T15:59:09.777929+0000","last_unstale":"2024-06-29T15:59:10.767789+0000","last_undegraded":"2024-06-29T15:59:10.767789+0000","last_fullsized":"2024-06-29T15:59:10.767789+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:03.943383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"252'1","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104731+0000","last_change":"2024-06-29T15:59:08.765586+0000","last_active":"2024-06-29T15:59:12.104731+0000","last_peered":"2024-06-29T15:59:12.104731+0000","last_clean":"2024-06-29T15:59:12.104731+0000","last_became_active":"2024-06-29T15:59:08.765325+0000","last_became_peered":"2024-06-29T15:59:08.765325+0000","last_unstale":"2024-06-29T15:59:12.104731+0000","last_undegraded":"2024-06-29T15:59:12.104731+0000","last_fullsized":"2024-06-29T15:59:12.104731+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:03:29.004633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603306+0000","last_change":"2024-06-29T15:59:09.790464+0000","last_active":"2024-06-29T15:59:11.603306+0000","last_peered":"2024-06-29T15:59:11.603306+0000","last_clean":"2024-06-29T15:59:11.603306+0000","last_became_active":"2024-06-29T15:59:09.790128+0000","last_became_peered":"2024-06-29T15:59:09.790128+0000","last_unstale":"2024-06-29T15:59:11.603306+0000","last_undegraded":"2024-06-29T15:59:11.603306+0000","last_fullsized":"2024-06-29T15:59:11.603306+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:31:46.004642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"252'1","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772001+0000","last_change":"2024-06-29T15:59:08.776661+0000","last_active":"2024-06-29T15:59:10.772001+0000","last_peered":"2024-06-29T15:59:10.772001+0000","last_clean":"2024-06-29T15:59:10.772001+0000","last_became_active":"2024-06-29T15:59:08.776340+0000","last_became_peered":"2024-06-29T15:59:08.776340+0000","last_unstale":"2024-06-29T15:59:10.772001+0000","last_undegraded":"2024-06-29T15:59:10.772001+0000","last_fullsized":"2024-06-29T15:59:10.772001+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:43:46.337407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.913325+0000","last_change":"2024-06-29T15:59:09.780261+0000","last_active":"2024-06-29T15:59:11.913325+0000","last_peered":"2024-06-29T15:59:11.913325+0000","last_clean":"2024-06-29T15:59:11.913325+0000","last_became_active":"2024-06-29T15:59:09.779939+0000","last_became_peered":"2024-06-29T15:59:09.779939+0000","last_unstale":"2024-06-29T15:59:11.913325+0000","last_undegraded":"2024-06-29T15:59:11.913325+0000","last_fullsized":"2024-06-29T15:59:11.913325+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:20:09.791145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103602+0000","last_change":"2024-06-29T15:59:08.773014+0000","last_active":"2024-06-29T15:59:12.103602+0000","last_peered":"2024-06-29T15:59:12.103602+0000","last_clean":"2024-06-29T15:59:12.103602+0000","last_became_active":"2024-06-29T15:59:08.772737+0000","last_became_peered":"2024-06-29T15:59:08.772737+0000","last_unstale":"2024-06-29T15:59:12.103602+0000","last_undegraded":"2024-06-29T15:59:12.103602+0000","last_fullsized":"2024-06-29T15:59:12.103602+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:48:38.018192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.771267+0000","last_change":"2024-06-29T15:59:09.798707+0000","last_active":"2024-06-29T15:59:10.771267+0000","last_peered":"2024-06-29T15:59:10.771267+0000","last_clean":"2024-06-29T15:59:10.771267+0000","last_became_active":"2024-06-29T15:59:09.798406+0000","last_became_peered":"2024-06-29T15:59:09.798406+0000","last_unstale":"2024-06-29T15:59:10.771267+0000","last_undegraded":"2024-06-29T15:59:10.771267+0000","last_fullsized":"2024-06-29T15:59:10.771267+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:15:23.486329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103555+0000","last_change":"2024-06-29T15:59:08.778237+0000","last_active":"2024-06-29T15:59:12.103555+0000","last_peered":"2024-06-29T15:59:12.103555+0000","last_clean":"2024-06-29T15:59:12.103555+0000","last_became_active":"2024-06-29T15:59:08.777914+0000","last_became_peered":"2024-06-29T15:59:08.777914+0000","last_unstale":"2024-06-29T15:59:12.103555+0000","last_undegraded":"2024-06-29T15:59:12.103555+0000","last_fullsized":"2024-06-29T15:59:12.103555+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:00:20.440115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772644+0000","last_change":"2024-06-29T15:59:09.799654+0000","last_active":"2024-06-29T15:59:10.772644+0000","last_peered":"2024-06-29T15:59:10.772644+0000","last_clean":"2024-06-29T15:59:10.772644+0000","last_became_active":"2024-06-29T15:59:09.799358+0000","last_became_peered":"2024-06-29T15:59:09.799358+0000","last_unstale":"2024-06-29T15:59:10.772644+0000","last_undegraded":"2024-06-29T15:59:10.772644+0000","last_fullsized":"2024-06-29T15:59:10.772644+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:06.081704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772186+0000","last_change":"2024-06-29T15:59:08.764880+0000","last_active":"2024-06-29T15:59:10.772186+0000","last_peered":"2024-06-29T15:59:10.772186+0000","last_clean":"2024-06-29T15:59:10.772186+0000","last_became_active":"2024-06-29T15:59:08.764548+0000","last_became_peered":"2024-06-29T15:59:08.764548+0000","last_unstale":"2024-06-29T15:59:10.772186+0000","last_undegraded":"2024-06-29T15:59:10.772186+0000","last_fullsized":"2024-06-29T15:59:10.772186+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:42.271085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772051+0000","last_change":"2024-06-29T15:59:09.792805+0000","last_active":"2024-06-29T15:59:10.772051+0000","last_peered":"2024-06-29T15:59:10.772051+0000","last_clean":"2024-06-29T15:59:10.772051+0000","last_became_active":"2024-06-29T15:59:09.792494+0000","last_became_peered":"2024-06-29T15:59:09.792494+0000","last_unstale":"2024-06-29T15:59:10.772051+0000","last_undegraded":"2024-06-29T15:59:10.772051+0000","last_fullsized":"2024-06-29T15:59:10.772051+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:12:51.577935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103923+0000","last_change":"2024-06-29T15:59:08.777696+0000","last_active":"2024-06-29T15:59:12.103923+0000","last_peered":"2024-06-29T15:59:12.103923+0000","last_clean":"2024-06-29T15:59:12.103923+0000","last_became_active":"2024-06-29T15:59:08.777360+0000","last_became_peered":"2024-06-29T15:59:08.777360+0000","last_unstale":"2024-06-29T15:59:12.103923+0000","last_undegraded":"2024-06-29T15:59:12.103923+0000","last_fullsized":"2024-06-29T15:59:12.103923+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:52:42.389445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103693+0000","last_change":"2024-06-29T15:59:09.784361+0000","last_active":"2024-06-29T15:59:12.103693+0000","last_peered":"2024-06-29T15:59:12.103693+0000","last_clean":"2024-06-29T15:59:12.103693+0000","last_became_active":"2024-06-29T15:59:09.784043+0000","last_became_peered":"2024-06-29T15:59:09.784043+0000","last_unstale":"2024-06-29T15:59:12.103693+0000","last_undegraded":"2024-06-29T15:59:12.103693+0000","last_fullsized":"2024-06-29T15:59:12.103693+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:23.073539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"252'3","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.913394+0000","last_change":"2024-06-29T15:59:08.769792+0000","last_active":"2024-06-29T15:59:11.913394+0000","last_peered":"2024-06-29T15:59:11.913394+0000","last_clean":"2024-06-29T15:59:11.913394+0000","last_became_active":"2024-06-29T15:59:08.769493+0000","last_became_peered":"2024-06-29T15:59:08.769493+0000","last_unstale":"2024-06-29T15:59:11.913394+0000","last_undegraded":"2024-06-29T15:59:11.913394+0000","last_fullsized":"2024-06-29T15:59:11.913394+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:00:34.913944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772747+0000","last_change":"2024-06-29T15:59:09.787306+0000","last_active":"2024-06-29T15:59:10.772747+0000","last_peered":"2024-06-29T15:59:10.772747+0000","last_clean":"2024-06-29T15:59:10.772747+0000","last_became_active":"2024-06-29T15:59:09.787010+0000","last_became_peered":"2024-06-29T15:59:09.787010+0000","last_unstale":"2024-06-29T15:59:10.772747+0000","last_undegraded":"2024-06-29T15:59:10.772747+0000","last_fullsized":"2024-06-29T15:59:10.772747+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:28.453733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"252'6","reported_seq":20,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104343+0000","last_change":"2024-06-29T15:59:08.772761+0000","last_active":"2024-06-29T15:59:12.104343+0000","last_peered":"2024-06-29T15:59:12.104343+0000","last_clean":"2024-06-29T15:59:12.104343+0000","last_became_active":"2024-06-29T15:59:08.772359+0000","last_became_peered":"2024-06-29T15:59:08.772359+0000","last_unstale":"2024-06-29T15:59:12.104343+0000","last_undegraded":"2024-06-29T15:59:12.104343+0000","last_fullsized":"2024-06-29T15:59:12.104343+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:40.888241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603220+0000","last_change":"2024-06-29T15:59:09.783104+0000","last_active":"2024-06-29T15:59:11.603220+0000","last_peered":"2024-06-29T15:59:11.603220+0000","last_clean":"2024-06-29T15:59:11.603220+0000","last_became_active":"2024-06-29T15:59:09.782764+0000","last_became_peered":"2024-06-29T15:59:09.782764+0000","last_unstale":"2024-06-29T15:59:11.603220+0000","last_undegraded":"2024-06-29T15:59:11.603220+0000","last_fullsized":"2024-06-29T15:59:11.603220+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:10.712089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"252'2","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.767706+0000","last_change":"2024-06-29T15:59:08.774345+0000","last_active":"2024-06-29T15:59:10.767706+0000","last_peered":"2024-06-29T15:59:10.767706+0000","last_clean":"2024-06-29T15:59:10.767706+0000","last_became_active":"2024-06-29T15:59:08.774032+0000","last_became_peered":"2024-06-29T15:59:08.774032+0000","last_unstale":"2024-06-29T15:59:10.767706+0000","last_undegraded":"2024-06-29T15:59:10.767706+0000","last_fullsized":"2024-06-29T15:59:10.767706+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:56:30.984520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.771581+0000","last_change":"2024-06-29T15:59:09.793803+0000","last_active":"2024-06-29T15:59:10.771581+0000","last_peered":"2024-06-29T15:59:10.771581+0000","last_clean":"2024-06-29T15:59:10.771581+0000","last_became_active":"2024-06-29T15:59:09.793522+0000","last_became_peered":"2024-06-29T15:59:09.793522+0000","last_unstale":"2024-06-29T15:59:10.771581+0000","last_undegraded":"2024-06-29T15:59:10.771581+0000","last_fullsized":"2024-06-29T15:59:10.771581+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:26.458450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.767522+0000","last_change":"2024-06-29T15:59:08.768134+0000","last_active":"2024-06-29T15:59:10.767522+0000","last_peered":"2024-06-29T15:59:10.767522+0000","last_clean":"2024-06-29T15:59:10.767522+0000","last_became_active":"2024-06-29T15:59:08.767829+0000","last_became_peered":"2024-06-29T15:59:08.767829+0000","last_unstale":"2024-06-29T15:59:10.767522+0000","last_undegraded":"2024-06-29T15:59:10.767522+0000","last_fullsized":"2024-06-29T15:59:10.767522+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:56:12.992896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103469+0000","last_change":"2024-06-29T15:59:09.797487+0000","last_active":"2024-06-29T15:59:12.103469+0000","last_peered":"2024-06-29T15:59:12.103469+0000","last_clean":"2024-06-29T15:59:12.103469+0000","last_became_active":"2024-06-29T15:59:09.797172+0000","last_became_peered":"2024-06-29T15:59:09.797172+0000","last_unstale":"2024-06-29T15:59:12.103469+0000","last_undegraded":"2024-06-29T15:59:12.103469+0000","last_fullsized":"2024-06-29T15:59:12.103469+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:37:41.960856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772488+0000","last_change":"2024-06-29T15:59:08.774042+0000","last_active":"2024-06-29T15:59:10.772488+0000","last_peered":"2024-06-29T15:59:10.772488+0000","last_clean":"2024-06-29T15:59:10.772488+0000","last_became_active":"2024-06-29T15:59:08.773566+0000","last_became_peered":"2024-06-29T15:59:08.773566+0000","last_unstale":"2024-06-29T15:59:10.772488+0000","last_undegraded":"2024-06-29T15:59:10.772488+0000","last_fullsized":"2024-06-29T15:59:10.772488+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:20.850163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603747+0000","last_change":"2024-06-29T15:59:09.784173+0000","last_active":"2024-06-29T15:59:11.603747+0000","last_peered":"2024-06-29T15:59:11.603747+0000","last_clean":"2024-06-29T15:59:11.603747+0000","last_became_active":"2024-06-29T15:59:09.783788+0000","last_became_peered":"2024-06-29T15:59:09.783788+0000","last_unstale":"2024-06-29T15:59:11.603747+0000","last_undegraded":"2024-06-29T15:59:11.603747+0000","last_fullsized":"2024-06-29T15:59:11.603747+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:22.283296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"252'8","reported_seq":22,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.767522+0000","last_change":"2024-06-29T15:59:08.770486+0000","last_active":"2024-06-29T15:59:10.767522+0000","last_peered":"2024-06-29T15:59:10.767522+0000","last_clean":"2024-06-29T15:59:10.767522+0000","last_became_active":"2024-06-29T15:59:08.770195+0000","last_became_peered":"2024-06-29T15:59:08.770195+0000","last_unstale":"2024-06-29T15:59:10.767522+0000","last_undegraded":"2024-06-29T15:59:10.767522+0000","last_fullsized":"2024-06-29T15:59:10.767522+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:15:28.971949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.767654+0000","last_change":"2024-06-29T15:59:09.772905+0000","last_active":"2024-06-29T15:59:10.767654+0000","last_peered":"2024-06-29T15:59:10.767654+0000","last_clean":"2024-06-29T15:59:10.767654+0000","last_became_active":"2024-06-29T15:59:09.772581+0000","last_became_peered":"2024-06-29T15:59:09.772581+0000","last_unstale":"2024-06-29T15:59:10.767654+0000","last_undegraded":"2024-06-29T15:59:10.767654+0000","last_fullsized":"2024-06-29T15:59:10.767654+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:36:57.837267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"252'1","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772383+0000","last_change":"2024-06-29T15:59:08.774427+0000","last_active":"2024-06-29T15:59:10.772383+0000","last_peered":"2024-06-29T15:59:10.772383+0000","last_clean":"2024-06-29T15:59:10.772383+0000","last_became_active":"2024-06-29T15:59:08.774014+0000","last_became_peered":"2024-06-29T15:59:08.774014+0000","last_unstale":"2024-06-29T15:59:10.772383+0000","last_undegraded":"2024-06-29T15:59:10.772383+0000","last_fullsized":"2024-06-29T15:59:10.772383+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:45:05.079724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104007+0000","last_change":"2024-06-29T15:59:09.799834+0000","last_active":"2024-06-29T15:59:12.104007+0000","last_peered":"2024-06-29T15:59:12.104007+0000","last_clean":"2024-06-29T15:59:12.104007+0000","last_became_active":"2024-06-29T15:59:09.799500+0000","last_became_peered":"2024-06-29T15:59:09.799500+0000","last_unstale":"2024-06-29T15:59:12.104007+0000","last_undegraded":"2024-06-29T15:59:12.104007+0000","last_fullsized":"2024-06-29T15:59:12.104007+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:48:19.352398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"252'2","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104820+0000","last_change":"2024-06-29T15:59:08.761992+0000","last_active":"2024-06-29T15:59:12.104820+0000","last_peered":"2024-06-29T15:59:12.104820+0000","last_clean":"2024-06-29T15:59:12.104820+0000","last_became_active":"2024-06-29T15:59:08.761708+0000","last_became_peered":"2024-06-29T15:59:08.761708+0000","last_unstale":"2024-06-29T15:59:12.104820+0000","last_undegraded":"2024-06-29T15:59:12.104820+0000","last_fullsized":"2024-06-29T15:59:12.104820+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:37:16.806464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.771720+0000","last_change":"2024-06-29T15:59:09.799080+0000","last_active":"2024-06-29T15:59:10.771720+0000","last_peered":"2024-06-29T15:59:10.771720+0000","last_clean":"2024-06-29T15:59:10.771720+0000","last_became_active":"2024-06-29T15:59:09.798539+0000","last_became_peered":"2024-06-29T15:59:09.798539+0000","last_unstale":"2024-06-29T15:59:10.771720+0000","last_undegraded":"2024-06-29T15:59:10.771720+0000","last_fullsized":"2024-06-29T15:59:10.771720+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:32:21.852143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"252'1","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772878+0000","last_change":"2024-06-29T15:59:08.771423+0000","last_active":"2024-06-29T15:59:10.772878+0000","last_peered":"2024-06-29T15:59:10.772878+0000","last_clean":"2024-06-29T15:59:10.772878+0000","last_became_active":"2024-06-29T15:59:08.771099+0000","last_became_peered":"2024-06-29T15:59:08.771099+0000","last_unstale":"2024-06-29T15:59:10.772878+0000","last_undegraded":"2024-06-29T15:59:10.772878+0000","last_fullsized":"2024-06-29T15:59:10.772878+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:25.589856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104408+0000","last_change":"2024-06-29T15:59:09.782702+0000","last_active":"2024-06-29T15:59:12.104408+0000","last_peered":"2024-06-29T15:59:12.104408+0000","last_clean":"2024-06-29T15:59:12.104408+0000","last_became_active":"2024-06-29T15:59:09.782370+0000","last_became_peered":"2024-06-29T15:59:09.782370+0000","last_unstale":"2024-06-29T15:59:12.104408+0000","last_undegraded":"2024-06-29T15:59:12.104408+0000","last_fullsized":"2024-06-29T15:59:12.104408+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:35:43.778938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772467+0000","last_change":"2024-06-29T15:59:08.775516+0000","last_active":"2024-06-29T15:59:10.772467+0000","last_peered":"2024-06-29T15:59:10.772467+0000","last_clean":"2024-06-29T15:59:10.772467+0000","last_became_active":"2024-06-29T15:59:08.775199+0000","last_became_peered":"2024-06-29T15:59:08.775199+0000","last_unstale":"2024-06-29T15:59:10.772467+0000","last_undegraded":"2024-06-29T15:59:10.772467+0000","last_fullsized":"2024-06-29T15:59:10.772467+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:19.034925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.913244+0000","last_change":"2024-06-29T15:59:09.784390+0000","last_active":"2024-06-29T15:59:11.913244+0000","last_peered":"2024-06-29T15:59:11.913244+0000","last_clean":"2024-06-29T15:59:11.913244+0000","last_became_active":"2024-06-29T15:59:09.783245+0000","last_became_peered":"2024-06-29T15:59:09.783245+0000","last_unstale":"2024-06-29T15:59:11.913244+0000","last_undegraded":"2024-06-29T15:59:11.913244+0000","last_fullsized":"2024-06-29T15:59:11.913244+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:12:51.577935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"252'2","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104240+0000","last_change":"2024-06-29T15:59:08.772988+0000","last_active":"2024-06-29T15:59:12.104240+0000","last_peered":"2024-06-29T15:59:12.104240+0000","last_clean":"2024-06-29T15:59:12.104240+0000","last_became_active":"2024-06-29T15:59:08.772622+0000","last_became_peered":"2024-06-29T15:59:08.772622+0000","last_unstale":"2024-06-29T15:59:12.104240+0000","last_undegraded":"2024-06-29T15:59:12.104240+0000","last_fullsized":"2024-06-29T15:59:12.104240+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:28:11.987354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.771770+0000","last_change":"2024-06-29T15:59:09.796939+0000","last_active":"2024-06-29T15:59:10.771770+0000","last_peered":"2024-06-29T15:59:10.771770+0000","last_clean":"2024-06-29T15:59:10.771770+0000","last_became_active":"2024-06-29T15:59:09.796434+0000","last_became_peered":"2024-06-29T15:59:09.796434+0000","last_unstale":"2024-06-29T15:59:10.771770+0000","last_undegraded":"2024-06-29T15:59:10.771770+0000","last_fullsized":"2024-06-29T15:59:10.771770+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:15.642070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104520+0000","last_change":"2024-06-29T15:59:08.774824+0000","last_active":"2024-06-29T15:59:12.104520+0000","last_peered":"2024-06-29T15:59:12.104520+0000","last_clean":"2024-06-29T15:59:12.104520+0000","last_became_active":"2024-06-29T15:59:08.772415+0000","last_became_peered":"2024-06-29T15:59:08.772415+0000","last_unstale":"2024-06-29T15:59:12.104520+0000","last_undegraded":"2024-06-29T15:59:12.104520+0000","last_fullsized":"2024-06-29T15:59:12.104520+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:55:46.041718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.771164+0000","last_change":"2024-06-29T15:59:09.791533+0000","last_active":"2024-06-29T15:59:10.771164+0000","last_peered":"2024-06-29T15:59:10.771164+0000","last_clean":"2024-06-29T15:59:10.771164+0000","last_became_active":"2024-06-29T15:59:09.790391+0000","last_became_peered":"2024-06-29T15:59:09.790391+0000","last_unstale":"2024-06-29T15:59:10.771164+0000","last_undegraded":"2024-06-29T15:59:10.771164+0000","last_fullsized":"2024-06-29T15:59:10.771164+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:33:26.591836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772547+0000","last_change":"2024-06-29T15:59:08.769738+0000","last_active":"2024-06-29T15:59:10.772547+0000","last_peered":"2024-06-29T15:59:10.772547+0000","last_clean":"2024-06-29T15:59:10.772547+0000","last_became_active":"2024-06-29T15:59:08.769354+0000","last_became_peered":"2024-06-29T15:59:08.769354+0000","last_unstale":"2024-06-29T15:59:10.772547+0000","last_undegraded":"2024-06-29T15:59:10.772547+0000","last_fullsized":"2024-06-29T15:59:10.772547+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:05:14.640090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103605+0000","last_change":"2024-06-29T15:59:09.793363+0000","last_active":"2024-06-29T15:59:12.103605+0000","last_peered":"2024-06-29T15:59:12.103605+0000","last_clean":"2024-06-29T15:59:12.103605+0000","last_became_active":"2024-06-29T15:59:09.793063+0000","last_became_peered":"2024-06-29T15:59:09.793063+0000","last_unstale":"2024-06-29T15:59:12.103605+0000","last_undegraded":"2024-06-29T15:59:12.103605+0000","last_fullsized":"2024-06-29T15:59:12.103605+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:31:06.636191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103675+0000","last_change":"2024-06-29T15:59:08.774410+0000","last_active":"2024-06-29T15:59:12.103675+0000","last_peered":"2024-06-29T15:59:12.103675+0000","last_clean":"2024-06-29T15:59:12.103675+0000","last_became_active":"2024-06-29T15:59:08.774125+0000","last_became_peered":"2024-06-29T15:59:08.774125+0000","last_unstale":"2024-06-29T15:59:12.103675+0000","last_undegraded":"2024-06-29T15:59:12.103675+0000","last_fullsized":"2024-06-29T15:59:12.103675+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:08:53.145688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603585+0000","last_change":"2024-06-29T15:59:09.791248+0000","last_active":"2024-06-29T15:59:11.603585+0000","last_peered":"2024-06-29T15:59:11.603585+0000","last_clean":"2024-06-29T15:59:11.603585+0000","last_became_active":"2024-06-29T15:59:09.789153+0000","last_became_peered":"2024-06-29T15:59:09.789153+0000","last_unstale":"2024-06-29T15:59:11.603585+0000","last_undegraded":"2024-06-29T15:59:11.603585+0000","last_fullsized":"2024-06-29T15:59:11.603585+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:05.938866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"252'2","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103989+0000","last_change":"2024-06-29T15:59:08.775359+0000","last_active":"2024-06-29T15:59:12.103989+0000","last_peered":"2024-06-29T15:59:12.103989+0000","last_clean":"2024-06-29T15:59:12.103989+0000","last_became_active":"2024-06-29T15:59:08.775033+0000","last_became_peered":"2024-06-29T15:59:08.775033+0000","last_unstale":"2024-06-29T15:59:12.103989+0000","last_undegraded":"2024-06-29T15:59:12.103989+0000","last_fullsized":"2024-06-29T15:59:12.103989+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:56:44.517051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772340+0000","last_change":"2024-06-29T15:59:09.792857+0000","last_active":"2024-06-29T15:59:10.772340+0000","last_peered":"2024-06-29T15:59:10.772340+0000","last_clean":"2024-06-29T15:59:10.772340+0000","last_became_active":"2024-06-29T15:59:09.792515+0000","last_became_peered":"2024-06-29T15:59:09.792515+0000","last_unstale":"2024-06-29T15:59:10.772340+0000","last_undegraded":"2024-06-29T15:59:10.772340+0000","last_fullsized":"2024-06-29T15:59:10.772340+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:29:43.273064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"253'1","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772827+0000","last_change":"2024-06-29T15:59:08.772572+0000","last_active":"2024-06-29T15:59:10.772827+0000","last_peered":"2024-06-29T15:59:10.772827+0000","last_clean":"2024-06-29T15:59:10.772827+0000","last_became_active":"2024-06-29T15:59:08.772186+0000","last_became_peered":"2024-06-29T15:59:08.772186+0000","last_unstale":"2024-06-29T15:59:10.772827+0000","last_undegraded":"2024-06-29T15:59:10.772827+0000","last_fullsized":"2024-06-29T15:59:10.772827+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:37:51.677087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603431+0000","last_change":"2024-06-29T15:59:09.790000+0000","last_active":"2024-06-29T15:59:11.603431+0000","last_peered":"2024-06-29T15:59:11.603431+0000","last_clean":"2024-06-29T15:59:11.603431+0000","last_became_active":"2024-06-29T15:59:09.788137+0000","last_became_peered":"2024-06-29T15:59:09.788137+0000","last_unstale":"2024-06-29T15:59:11.603431+0000","last_undegraded":"2024-06-29T15:59:11.603431+0000","last_fullsized":"2024-06-29T15:59:11.603431+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:08:27.735867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103824+0000","last_change":"2024-06-29T15:59:08.777539+0000","last_active":"2024-06-29T15:59:12.103824+0000","last_peered":"2024-06-29T15:59:12.103824+0000","last_clean":"2024-06-29T15:59:12.103824+0000","last_became_active":"2024-06-29T15:59:08.777188+0000","last_became_peered":"2024-06-29T15:59:08.777188+0000","last_unstale":"2024-06-29T15:59:12.103824+0000","last_undegraded":"2024-06-29T15:59:12.103824+0000","last_fullsized":"2024-06-29T15:59:12.103824+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:12:29.892588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.104564+0000","last_change":"2024-06-29T15:59:09.779160+0000","last_active":"2024-06-29T15:59:12.104564+0000","last_peered":"2024-06-29T15:59:12.104564+0000","last_clean":"2024-06-29T15:59:12.104564+0000","last_became_active":"2024-06-29T15:59:09.778780+0000","last_became_peered":"2024-06-29T15:59:09.778780+0000","last_unstale":"2024-06-29T15:59:12.104564+0000","last_undegraded":"2024-06-29T15:59:12.104564+0000","last_fullsized":"2024-06-29T15:59:12.104564+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:54:05.098464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103766+0000","last_change":"2024-06-29T15:59:08.776838+0000","last_active":"2024-06-29T15:59:12.103766+0000","last_peered":"2024-06-29T15:59:12.103766+0000","last_clean":"2024-06-29T15:59:12.103766+0000","last_became_active":"2024-06-29T15:59:08.775875+0000","last_became_peered":"2024-06-29T15:59:08.775875+0000","last_unstale":"2024-06-29T15:59:12.103766+0000","last_undegraded":"2024-06-29T15:59:12.103766+0000","last_fullsized":"2024-06-29T15:59:12.103766+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:08.956822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.913445+0000","last_change":"2024-06-29T15:59:09.783525+0000","last_active":"2024-06-29T15:59:11.913445+0000","last_peered":"2024-06-29T15:59:11.913445+0000","last_clean":"2024-06-29T15:59:11.913445+0000","last_became_active":"2024-06-29T15:59:09.782865+0000","last_became_peered":"2024-06-29T15:59:09.782865+0000","last_unstale":"2024-06-29T15:59:11.913445+0000","last_undegraded":"2024-06-29T15:59:11.913445+0000","last_fullsized":"2024-06-29T15:59:11.913445+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:50:28.453733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:11.603654+0000","last_change":"2024-06-29T15:59:08.769362+0000","last_active":"2024-06-29T15:59:11.603654+0000","last_peered":"2024-06-29T15:59:11.603654+0000","last_clean":"2024-06-29T15:59:11.603654+0000","last_became_active":"2024-06-29T15:59:08.769060+0000","last_became_peered":"2024-06-29T15:59:08.769060+0000","last_unstale":"2024-06-29T15:59:11.603654+0000","last_undegraded":"2024-06-29T15:59:11.603654+0000","last_fullsized":"2024-06-29T15:59:11.603654+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:15:28.971949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.771909+0000","last_change":"2024-06-29T15:59:09.792467+0000","last_active":"2024-06-29T15:59:10.771909+0000","last_peered":"2024-06-29T15:59:10.771909+0000","last_clean":"2024-06-29T15:59:10.771909+0000","last_became_active":"2024-06-29T15:59:09.788819+0000","last_became_peered":"2024-06-29T15:59:09.788819+0000","last_unstale":"2024-06-29T15:59:10.771909+0000","last_undegraded":"2024-06-29T15:59:10.771909+0000","last_fullsized":"2024-06-29T15:59:10.771909+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:13:20.036905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103834+0000","last_change":"2024-06-29T15:59:08.774326+0000","last_active":"2024-06-29T15:59:12.103834+0000","last_peered":"2024-06-29T15:59:12.103834+0000","last_clean":"2024-06-29T15:59:12.103834+0000","last_became_active":"2024-06-29T15:59:08.773514+0000","last_became_peered":"2024-06-29T15:59:08.773514+0000","last_unstale":"2024-06-29T15:59:12.103834+0000","last_undegraded":"2024-06-29T15:59:12.103834+0000","last_fullsized":"2024-06-29T15:59:12.103834+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:02:06.295388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.771363+0000","last_change":"2024-06-29T15:59:09.791259+0000","last_active":"2024-06-29T15:59:10.771363+0000","last_peered":"2024-06-29T15:59:10.771363+0000","last_clean":"2024-06-29T15:59:10.771363+0000","last_became_active":"2024-06-29T15:59:09.790050+0000","last_became_peered":"2024-06-29T15:59:09.790050+0000","last_unstale":"2024-06-29T15:59:10.771363+0000","last_undegraded":"2024-06-29T15:59:10.771363+0000","last_fullsized":"2024-06-29T15:59:10.771363+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:50:46.892215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103736+0000","last_change":"2024-06-29T15:59:08.771871+0000","last_active":"2024-06-29T15:59:12.103736+0000","last_peered":"2024-06-29T15:59:12.103736+0000","last_clean":"2024-06-29T15:59:12.103736+0000","last_became_active":"2024-06-29T15:59:08.771544+0000","last_became_peered":"2024-06-29T15:59:08.771544+0000","last_unstale":"2024-06-29T15:59:12.103736+0000","last_undegraded":"2024-06-29T15:59:12.103736+0000","last_fullsized":"2024-06-29T15:59:12.103736+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:54:17.230493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:10.772407+0000","last_change":"2024-06-29T15:59:09.792522+0000","last_active":"2024-06-29T15:59:10.772407+0000","last_peered":"2024-06-29T15:59:10.772407+0000","last_clean":"2024-06-29T15:59:10.772407+0000","last_became_active":"2024-06-29T15:59:09.792092+0000","last_became_peered":"2024-06-29T15:59:09.792092+0000","last_unstale":"2024-06-29T15:59:10.772407+0000","last_undegraded":"2024-06-29T15:59:10.772407+0000","last_fullsized":"2024-06-29T15:59:10.772407+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:36:44.695387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":14,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103841+0000","last_change":"2024-06-29T15:59:08.764007+0000","last_active":"2024-06-29T15:59:12.103841+0000","last_peered":"2024-06-29T15:59:12.103841+0000","last_clean":"2024-06-29T15:59:12.103841+0000","last_became_active":"2024-06-29T15:59:08.763651+0000","last_became_peered":"2024-06-29T15:59:08.763651+0000","last_unstale":"2024-06-29T15:59:12.103841+0000","last_undegraded":"2024-06-29T15:59:12.103841+0000","last_fullsized":"2024-06-29T15:59:12.103841+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":250,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:07.748786+0000","last_clean_scrub_stamp":"2024-06-29T15:59:07.748786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:55:17.314927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":12,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103585+0000","last_change":"2024-06-29T15:59:09.797774+0000","last_active":"2024-06-29T15:59:12.103585+0000","last_peered":"2024-06-29T15:59:12.103585+0000","last_clean":"2024-06-29T15:59:12.103585+0000","last_became_active":"2024-06-29T15:59:09.797422+0000","last_became_peered":"2024-06-29T15:59:09.797422+0000","last_unstale":"2024-06-29T15:59:12.103585+0000","last_undegraded":"2024-06-29T15:59:12.103585+0000","last_fullsized":"2024-06-29T15:59:12.103585+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:08.750765+0000","last_clean_scrub_stamp":"2024-06-29T15:59:08.750765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:03.392872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"239'247","reported_seq":1215,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-06-29T15:59:12.103464+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:59:12.103464+0000","last_peered":"2024-06-29T15:59:12.103464+0000","last_clean":"2024-06-29T15:59:12.103464+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:59:12.103464+0000","last_undegraded":"2024-06-29T15:59:12.103464+0000","last_fullsized":"2024-06-29T15:59:12.103464+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1566,"num_read_kb":4455,"num_write":401,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247,"ondisk_log_size":247,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576006,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52932,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318908,"statfs":{"total":96636764160,"available":96582561792,"internally_reserved":0,"allocated":3792896,"data_stored":2171813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834576007,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60680,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311160,"statfs":{"total":96636764160,"available":96574627840,"internally_reserved":0,"allocated":3203072,"data_stored":1581535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834576005,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30168,"kb_used_data":3144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341672,"statfs":{"total":96636764160,"available":96605872128,"internally_reserved":0,"allocated":3219456,"data_stored":1583709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834576006,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78664,"kb_used_data":3132,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94293176,"statfs":{"total":96636764160,"available":96556212224,"internally_reserved":0,"allocated":3207168,"data_stored":1581537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834576006,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48204,"kb_used_data":3136,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323636,"statfs":{"total":96636764160,"available":96587403264,"internally_reserved":0,"allocated":3211264,"data_stored":1583057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576006,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52356,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319484,"statfs":{"total":96636764160,"available":96583151616,"internally_reserved":0,"allocated":3203072,"data_stored":1581535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834576005,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64840,"kb_used_data":3128,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307000,"statfs":{"total":96636764160,"available":96570368000,"internally_reserved":0,"allocated":3203072,"data_stored":1581479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834576006,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61264,"kb_used_data":3716,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310576,"statfs":{"total":96636764160,"available":96574029824,"internally_reserved":0,"allocated":3805184,"data_stored":2172451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:59:14.181 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:59:14.181 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:59:14.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.338+0000 7ff715265640 1 -- 172.21.15.81:0/299542721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7100b4810 msgr2=0x7ff7100b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.338+0000 7ff715265640 1 --2- 172.21.15.81:0/299542721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7100b4810 0x7ff7100b4c20 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7ff708009830 tx=0x7ff70802a480 comp rx=0 tx=0).stop 2024-06-29T15:59:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.338+0000 7ff715265640 1 -- 172.21.15.81:0/299542721 shutdown_connections 2024-06-29T15:59:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.338+0000 7ff715265640 1 --2- 172.21.15.81:0/299542721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7100b4810 0x7ff7100b4c20 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.338+0000 7ff715265640 1 -- 172.21.15.81:0/299542721 >> 172.21.15.81:0/299542721 conn(0x7ff7100b3510 msgr2=0x7ff7100b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.338+0000 7ff715265640 1 -- 172.21.15.81:0/299542721 shutdown_connections 2024-06-29T15:59:14.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.338+0000 7ff715265640 1 -- 172.21.15.81:0/299542721 wait complete. 2024-06-29T15:59:14.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.340+0000 7ff715265640 1 Processor -- start 2024-06-29T15:59:14.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.340+0000 7ff715265640 1 -- start start 2024-06-29T15:59:14.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff715265640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7100b4810 0x7ff7100ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff715265640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7100b6eb0 0x7ff7100b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff715265640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7100cd480 0x7ff7100c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff715265640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff7100b5160 con 0x7ff7100b6eb0 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff715265640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7ff7100b29c0 con 0x7ff7100cd480 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff715265640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7ff7100b52e0 con 0x7ff7100b4810 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff70e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7100b6eb0 0x7ff7100b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff70e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7100b6eb0 0x7ff7100b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60354/0 (socket says 172.21.15.81:60354) 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff70e7fc640 1 -- 172.21.15.81:0/3508789508 learned_addr learned my addr 172.21.15.81:0/3508789508 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff70f7fe640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7100cd480 0x7ff7100c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff70effd640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7100b4810 0x7ff7100ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff70e7fc640 1 -- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7100b4810 msgr2=0x7ff7100ca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff70e7fc640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7100b4810 0x7ff7100ca250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff70e7fc640 1 -- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7100cd480 msgr2=0x7ff7100c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff70e7fc640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7100cd480 0x7ff7100c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.341+0000 7ff70e7fc640 1 -- 172.21.15.81:0/3508789508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7100af330 con 0x7ff7100b6eb0 2024-06-29T15:59:14.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.342+0000 7ff70e7fc640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7100b6eb0 0x7ff7100b00f0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7ff70400c3c0 tx=0x7ff70400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:14.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.342+0000 7ff6f77fe640 1 -- 172.21.15.81:0/3508789508 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704017070 con 0x7ff7100b6eb0 2024-06-29T15:59:14.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.342+0000 7ff6f77fe640 1 -- 172.21.15.81:0/3508789508 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7040035f0 con 0x7ff7100b6eb0 2024-06-29T15:59:14.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.342+0000 7ff715265640 1 -- 172.21.15.81:0/3508789508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7100af610 con 0x7ff7100b6eb0 2024-06-29T15:59:14.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.342+0000 7ff6f77fe640 1 -- 172.21.15.81:0/3508789508 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704003a80 con 0x7ff7100b6eb0 2024-06-29T15:59:14.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.342+0000 7ff715265640 1 -- 172.21.15.81:0/3508789508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7100afbd0 con 0x7ff7100b6eb0 2024-06-29T15:59:14.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.343+0000 7ff715265640 1 -- 172.21.15.81:0/3508789508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7100b4c20 con 0x7ff7100b6eb0 2024-06-29T15:59:14.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.344+0000 7ff6f77fe640 1 -- 172.21.15.81:0/3508789508 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7ff70401a070 con 0x7ff7100b6eb0 2024-06-29T15:59:14.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.344+0000 7ff6f77fe640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff6fc076540 0x7ff6fc078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.344+0000 7ff6f77fe640 1 -- 172.21.15.81:0/3508789508 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7ff704096fd0 con 0x7ff7100b6eb0 2024-06-29T15:59:14.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.344+0000 7ff70effd640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff6fc076540 0x7ff6fc078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.345+0000 7ff70effd640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff6fc076540 0x7ff6fc078a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff700009f20 tx=0x7ff7000094d0 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:14.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.349+0000 7ff6f77fe640 1 -- 172.21.15.81:0/3508789508 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7ff7040604c0 con 0x7ff7100b6eb0 2024-06-29T15:59:14.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.602+0000 7ff715265640 1 -- 172.21.15.81:0/3508789508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ff7100cdf00 con 0x7ff7100b6eb0 2024-06-29T15:59:14.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.605+0000 7ff6f77fe640 1 -- 172.21.15.81:0/3508789508 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v70) ==== 190+0+147 (secure 0 0 0) 0x7ff70405fdc0 con 0x7ff7100b6eb0 2024-06-29T15:59:14.606 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T15:59:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T15:59:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T15:59:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T15:59:14.607 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:59:14.607 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T15:59:14.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 -- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff6fc076540 msgr2=0x7ff6fc078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff6fc076540 0x7ff6fc078a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff700009f20 tx=0x7ff7000094d0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 -- 172.21.15.81:0/3508789508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7100b6eb0 msgr2=0x7ff7100b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7100b6eb0 0x7ff7100b00f0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7ff70400c3c0 tx=0x7ff70400c880 comp rx=0 tx=0).stop 2024-06-29T15:59:14.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 -- 172.21.15.81:0/3508789508 shutdown_connections 2024-06-29T15:59:14.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7ff6fc076540 0x7ff6fc078a50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7ff7100cd480 0x7ff7100c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7100b6eb0 0x7ff7100b00f0 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 --2- 172.21.15.81:0/3508789508 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7ff7100b4810 0x7ff7100ca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 -- 172.21.15.81:0/3508789508 >> 172.21.15.81:0/3508789508 conn(0x7ff7100b3510 msgr2=0x7ff710071510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:14.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 -- 172.21.15.81:0/3508789508 shutdown_connections 2024-06-29T15:59:14.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.607+0000 7ff715265640 1 -- 172.21.15.81:0/3508789508 wait complete. 2024-06-29T15:59:14.619 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:14.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.732+0000 7f3f4de12640 1 -- 172.21.15.81:0/3184463650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f480ce510 msgr2=0x7f3f480ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.732+0000 7f3f4de12640 1 --2- 172.21.15.81:0/3184463650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f480ce510 0x7f3f480ce920 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f3f480b41b0 tx=0x7f3f3c02a470 comp rx=0 tx=0).stop 2024-06-29T15:59:14.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.732+0000 7f3f4de12640 1 -- 172.21.15.81:0/3184463650 shutdown_connections 2024-06-29T15:59:14.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.732+0000 7f3f4de12640 1 --2- 172.21.15.81:0/3184463650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f480ce510 0x7f3f480ce920 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.732+0000 7f3f4de12640 1 --2- 172.21.15.81:0/3184463650 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f480b6eb0 0x7f3f480d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.732+0000 7f3f4de12640 1 -- 172.21.15.81:0/3184463650 >> 172.21.15.81:0/3184463650 conn(0x7f3f480b3510 msgr2=0x7f3f480b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:14.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.732+0000 7f3f4de12640 1 -- 172.21.15.81:0/3184463650 shutdown_connections 2024-06-29T15:59:14.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.732+0000 7f3f4de12640 1 -- 172.21.15.81:0/3184463650 wait complete. 2024-06-29T15:59:14.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.734+0000 7f3f4de12640 1 Processor -- start 2024-06-29T15:59:14.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.734+0000 7f3f4de12640 1 -- start start 2024-06-29T15:59:14.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f4de12640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f480b4810 0x7f3f48158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f4de12640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f480b6eb0 0x7f3f480af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f4de12640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f480ce510 0x7f3f480c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f4de12640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f480cd480 con 0x7f3f480b4810 2024-06-29T15:59:14.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f4de12640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3f480b29c0 con 0x7f3f480b6eb0 2024-06-29T15:59:14.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f4de12640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3f480cd600 con 0x7f3f480ce510 2024-06-29T15:59:14.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f477fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f480b4810 0x7f3f48158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f477fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f480b4810 0x7f3f48158ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60372/0 (socket says 172.21.15.81:60372) 2024-06-29T15:59:14.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f477fe640 1 -- 172.21.15.81:0/545890854 learned_addr learned my addr 172.21.15.81:0/545890854 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f47fff640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f480ce510 0x7f3f480c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f46ffd640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f480b6eb0 0x7f3f480af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f477fe640 1 -- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f480ce510 msgr2=0x7f3f480c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f477fe640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f480ce510 0x7f3f480c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f477fe640 1 -- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f480b6eb0 msgr2=0x7f3f480af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f477fe640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f480b6eb0 0x7f3f480af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f477fe640 1 -- 172.21.15.81:0/545890854 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f481f2e40 con 0x7f3f480b4810 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.735+0000 7f3f46ffd640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f480b6eb0 0x7f3f480af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.736+0000 7f3f477fe640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f480b4810 0x7f3f48158ad0 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800d6d0 tx=0x7f3f3800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.736+0000 7f3f2ffff640 1 -- 172.21.15.81:0/545890854 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38017070 con 0x7f3f480b4810 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.736+0000 7f3f2ffff640 1 -- 172.21.15.81:0/545890854 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f38005330 con 0x7f3f480b4810 2024-06-29T15:59:14.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.736+0000 7f3f2ffff640 1 -- 172.21.15.81:0/545890854 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f380097c0 con 0x7f3f480b4810 2024-06-29T15:59:14.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.736+0000 7f3f4de12640 1 -- 172.21.15.81:0/545890854 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f481f3120 con 0x7f3f480b4810 2024-06-29T15:59:14.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.736+0000 7f3f4de12640 1 -- 172.21.15.81:0/545890854 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f481f2b30 con 0x7f3f480b4810 2024-06-29T15:59:14.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.737+0000 7f3f4de12640 1 -- 172.21.15.81:0/545890854 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f480d02d0 con 0x7f3f480b4810 2024-06-29T15:59:14.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.738+0000 7f3f2ffff640 1 -- 172.21.15.81:0/545890854 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f3f38009960 con 0x7f3f480b4810 2024-06-29T15:59:14.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.738+0000 7f3f2ffff640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3f34076540 0x7f3f34078a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:14.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.738+0000 7f3f2ffff640 1 -- 172.21.15.81:0/545890854 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7f3f38097c10 con 0x7f3f480b4810 2024-06-29T15:59:14.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.738+0000 7f3f46ffd640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3f34076540 0x7f3f34078a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:14.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.739+0000 7f3f46ffd640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3f34076540 0x7f3f34078a50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c034000 tx=0x7f3f3c004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:14.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.742+0000 7f3f2ffff640 1 -- 172.21.15.81:0/545890854 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3f38060a30 con 0x7f3f480b4810 2024-06-29T15:59:14.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.960+0000 7f3f4de12640 1 -- 172.21.15.81:0/545890854 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f480cdf00 con 0x7f3f480b4810 2024-06-29T15:59:14.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.960+0000 7f3f2ffff640 1 -- 172.21.15.81:0/545890854 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+5236 (secure 0 0 0) 0x7f3f38069070 con 0x7f3f480b4810 2024-06-29T15:59:14.962 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":145,"btime":"2024-06-29T15:59:12:796690+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9842,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2717891820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2717891820},{"type":"v1","addr":"172.21.15.81:6835","nonce":2717891820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":9848,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2745105064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2745105064},{"type":"v1","addr":"172.21.15.167:6836","nonce":2745105064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":11329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/745441473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":745441473},{"type":"v1","addr":"172.21.15.81:6834","nonce":745441473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:09.126215+0000","modified":"2024-06-29T15:59:12.167052+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11347":{"gid":11347,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/54677014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":54677014},{"type":"v1","addr":"172.21.15.167:6837","nonce":54677014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11347,"qdb_cluster":[11347]},"id":17}]} 2024-06-29T15:59:14.962 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 145 2024-06-29T15:59:14.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 -- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3f34076540 msgr2=0x7f3f34078a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3f34076540 0x7f3f34078a50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c034000 tx=0x7f3f3c004730 comp rx=0 tx=0).stop 2024-06-29T15:59:14.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 -- 172.21.15.81:0/545890854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f480b4810 msgr2=0x7f3f48158ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:14.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f480b4810 0x7f3f48158ad0 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800d6d0 tx=0x7f3f3800db90 comp rx=0 tx=0).stop 2024-06-29T15:59:14.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 -- 172.21.15.81:0/545890854 shutdown_connections 2024-06-29T15:59:14.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3f34076540 0x7f3f34078a50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3f480ce510 0x7f3f480c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3f480b6eb0 0x7f3f480af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 --2- 172.21.15.81:0/545890854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f480b4810 0x7f3f48158ad0 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:14.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 -- 172.21.15.81:0/545890854 >> 172.21.15.81:0/545890854 conn(0x7f3f480b3510 msgr2=0x7f3f48070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:14.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 -- 172.21.15.81:0/545890854 shutdown_connections 2024-06-29T15:59:14.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:14.962+0000 7f3f4de12640 1 -- 172.21.15.81:0/545890854 wait complete. 2024-06-29T15:59:15.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 145, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:59:09.126215+0000', 'modified': '2024-06-29T15:59:12.167052+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11347}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11347': {'gid': 11347, 'name': 'd', 'rank': 0, 'incarnation': 142, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.167:6837/54677014', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.167:6835', 'nonce': 54677014}, {'type': 'v1', 'addr': '172.21.15.167:6837', 'nonce': 54677014}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11347, 'qdb_cluster': [11347]} 2024-06-29T15:59:15.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:59:15.380 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-06-29T15:59:15.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.498+0000 7f3c90cc1640 1 -- 172.21.15.81:0/694711135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c8c0b3960 msgr2=0x7f3c8c0b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.498+0000 7f3c90cc1640 1 --2- 172.21.15.81:0/694711135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c8c0b3960 0x7f3c8c0b3d70 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f3c84009830 tx=0x7f3c8402a510 comp rx=0 tx=0).stop 2024-06-29T15:59:15.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.498+0000 7f3c90cc1640 1 -- 172.21.15.81:0/694711135 shutdown_connections 2024-06-29T15:59:15.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.498+0000 7f3c90cc1640 1 --2- 172.21.15.81:0/694711135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c8c0b3960 0x7f3c8c0b3d70 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.498+0000 7f3c90cc1640 1 --2- 172.21.15.81:0/694711135 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c8c0b57b0 0x7f3c8c15aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.498+0000 7f3c90cc1640 1 -- 172.21.15.81:0/694711135 >> 172.21.15.81:0/694711135 conn(0x7f3c8c0d0930 msgr2=0x7f3c8c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:15.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.498+0000 7f3c90cc1640 1 -- 172.21.15.81:0/694711135 shutdown_connections 2024-06-29T15:59:15.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.498+0000 7f3c90cc1640 1 -- 172.21.15.81:0/694711135 wait complete. 2024-06-29T15:59:15.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.500+0000 7f3c90cc1640 1 Processor -- start 2024-06-29T15:59:15.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.500+0000 7f3c90cc1640 1 -- start start 2024-06-29T15:59:15.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.501+0000 7f3c90cc1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c8c06ef90 0x7f3c8c0d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.501+0000 7f3c90cc1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c8c0b3960 0x7f3c8c14ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.501+0000 7f3c90cc1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c8c0b57b0 0x7f3c8c0aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.501+0000 7f3c90cc1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c0b4640 con 0x7f3c8c06ef90 2024-06-29T15:59:15.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.501+0000 7f3c90cc1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c0b44c0 con 0x7f3c8c0b3960 2024-06-29T15:59:15.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.501+0000 7f3c90cc1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8c0b47c0 con 0x7f3c8c0b57b0 2024-06-29T15:59:15.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c8a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c8c06ef90 0x7f3c8c0d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c8a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c8c06ef90 0x7f3c8c0d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60390/0 (socket says 172.21.15.81:60390) 2024-06-29T15:59:15.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c8a59c640 1 -- 172.21.15.81:0/415976884 learned_addr learned my addr 172.21.15.81:0/415976884 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:15.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c8a59c640 1 -- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c8c0b57b0 msgr2=0x7f3c8c0aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c8c0b57b0 0x7f3c8c0aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c8a59c640 1 -- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c8c0b3960 msgr2=0x7f3c8c14ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c8c0b3960 0x7f3c8c14ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c8a59c640 1 -- 172.21.15.81:0/415976884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8c0cea50 con 0x7f3c8c06ef90 2024-06-29T15:59:15.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c8c06ef90 0x7f3c8c0d02f0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00ea50 tx=0x7f3c7c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:15.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c7affd640 1 -- 172.21.15.81:0/415976884 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c00cdc0 con 0x7f3c8c06ef90 2024-06-29T15:59:15.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c7affd640 1 -- 172.21.15.81:0/415976884 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c7c011560 con 0x7f3c8c06ef90 2024-06-29T15:59:15.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c90cc1640 1 -- 172.21.15.81:0/415976884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c8c153bc0 con 0x7f3c8c06ef90 2024-06-29T15:59:15.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c7affd640 1 -- 172.21.15.81:0/415976884 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c011990 con 0x7f3c8c06ef90 2024-06-29T15:59:15.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.502+0000 7f3c90cc1640 1 -- 172.21.15.81:0/415976884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c8c14df50 con 0x7f3c8c06ef90 2024-06-29T15:59:15.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.504+0000 7f3c7affd640 1 -- 172.21.15.81:0/415976884 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f3c7c011b30 con 0x7f3c8c06ef90 2024-06-29T15:59:15.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.505+0000 7f3c7affd640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3c6c076510 0x7f3c6c078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.505+0000 7f3c7affd640 1 -- 172.21.15.81:0/415976884 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7f3c7c098890 con 0x7f3c8c06ef90 2024-06-29T15:59:15.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.505+0000 7f3c89d9b640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3c6c076510 0x7f3c6c078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.505+0000 7f3c97b7f740 1 Processor -- start 2024-06-29T15:59:15.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.506+0000 7f3c97b7f740 1 -- start start 2024-06-29T15:59:15.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.506+0000 7f3c89d9b640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3c6c076510 0x7f3c6c078a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3c80000f60 tx=0x7f3c800079b0 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:15.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad3d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a9283b0 0x55c31ad213a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a930590 0x55c31ad218e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c31acb9bc0 con 0x55c31a951270 2024-06-29T15:59:15.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55c31ac68630 con 0x55c31a9283b0 2024-06-29T15:59:15.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55c31ac57bd0 con 0x55c31a930590 2024-06-29T15:59:15.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c8a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad3d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c8a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad3d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60402/0 (socket says 172.21.15.81:60402) 2024-06-29T15:59:15.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c8a59c640 1 -- 172.21.15.81:0/839259127 learned_addr learned my addr 172.21.15.81:0/839259127 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:15.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c8ad9d640 1 --2- 172.21.15.81:0/839259127 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a930590 0x55c31ad218e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c89d9b640 1 --2- 172.21.15.81:0/839259127 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a9283b0 0x55c31ad213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c8a59c640 1 -- 172.21.15.81:0/839259127 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a930590 msgr2=0x55c31ad218e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/839259127 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a930590 0x55c31ad218e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c8a59c640 1 -- 172.21.15.81:0/839259127 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a9283b0 msgr2=0x55c31ad213a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/839259127 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a9283b0 0x55c31ad213a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c8a59c640 1 -- 172.21.15.81:0/839259127 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c31ac54ca0 con 0x55c31a951270 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/839259127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad3d6e0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c061630 tx=0x7f3c7c0615a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f117730ab15cec server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.507+0000 7f3c6bfff640 1 -- 172.21.15.81:0/839259127 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c01f070 con 0x55c31a951270 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c6bfff640 1 -- 172.21.15.81:0/839259127 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c7c0616a0 con 0x55c31a951270 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c6bfff640 1 -- 172.21.15.81:0/839259127 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c061b00 con 0x55c31a951270 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c97b7f740 1 -- 172.21.15.81:0/839259127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 msgr2=0x55c31ad3d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/839259127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad3d6e0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c061630 tx=0x7f3c7c0615a0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c97b7f740 1 -- 172.21.15.81:0/839259127 shutdown_connections 2024-06-29T15:59:15.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/839259127 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a930590 0x55c31ad218e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/839259127 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a9283b0 0x55c31ad213a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/839259127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad3d6e0 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c97b7f740 1 -- 172.21.15.81:0/839259127 >> 172.21.15.81:0/839259127 conn(0x55c31a9453c0 msgr2=0x55c31acf4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:15.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c97b7f740 1 -- 172.21.15.81:0/839259127 shutdown_connections 2024-06-29T15:59:15.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.508+0000 7f3c97b7f740 1 -- 172.21.15.81:0/839259127 wait complete. 2024-06-29T15:59:15.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.509+0000 7f3c97b7f740 1 -- start start 2024-06-29T15:59:15.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.509+0000 7f3c97b7f740 1 Processor -- start 2024-06-29T15:59:15.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a9283b0 0x55c31ad4c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a930590 0x55c31ad1fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad357f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c31abf5ae0 con 0x55c31a951270 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55c31ac5bc10 con 0x55c31a930590 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55c31a9cb620 con 0x55c31a9283b0 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c97b7f740 10 client.0 _ll_register_callbacks cb 0x55c31ac02480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c8ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c8ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad357f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60408/0 (socket says 172.21.15.81:60408) 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/3233904410 learned_addr learned my addr 172.21.15.81:0/3233904410 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a9283b0 0x55c31ad4c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c89d9b640 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a930590 0x55c31ad1fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a9283b0 msgr2=0x55c31ad4c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c8ad9d640 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a9283b0 0x55c31ad4c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a930590 msgr2=0x55c31ad1fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c8ad9d640 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a930590 0x55c31ad1fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.511+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/3233904410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c31a9bc850 con 0x55c31a951270 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c8ad9d640 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad357f0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f3c84009920 tx=0x7f3c840035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a14fd1ba7978f09 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c697fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c697fa640 1 -- 172.21.15.81:0/3233904410 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c84003ba0 con 0x55c31a951270 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c697fa640 1 -- 172.21.15.81:0/3233904410 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7f3c84003d40 con 0x55c31a951270 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3233904410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c7c0616a0 con 0x55c31a951270 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c697fa640 1 -- 172.21.15.81:0/3233904410 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=145) ==== 24+0+0 (secure 0 0 0) 0x7f3c8404c050 con 0x55c31a951270 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c97b7f740 10 client.13008 fetch_fsmap learned FSMap version 145 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3233904410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=145,monmap=2+}) -- 0x55c31aa336b0 con 0x55c31a951270 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c697fa640 1 -- 172.21.15.81:0/3233904410 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c7c0616a0 con 0x55c31a951270 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c697fa640 1 -- 172.21.15.81:0/3233904410 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 145) ==== 3502+0+0 (secure 0 0 0) 0x7f3c8404f020 con 0x55c31a951270 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c97b7f740 10 client.13008 fetch_fsmap finished waiting for FSMap version 145 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c97b7f740 10 client.13008 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:59:15.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c97b7f740 20 client.13008 populate_metadata read hostname 'smithi081' 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:6835/54677014,v1:172.21.15.167:6837/54677014] conn(0x55c31a90fe40 0x55c31ad59420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c97b7f740 4 client.13008 mds_command: new command op to 11347 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3233904410 --> [v2:172.21.15.167:6835/54677014,v1:172.21.15.167:6837/54677014] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c31ac6a580 con 0x55c31a90fe40 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.512+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:6835/54677014,v1:172.21.15.167:6837/54677014] conn(0x55c31a90fe40 0x55c31ad59420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.513+0000 7f3c697fa640 10 client.13008 ms_handle_connect on v2:172.21.15.167:6835/54677014 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.513+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:6835/54677014,v1:172.21.15.167:6837/54677014] conn(0x55c31a90fe40 0x55c31ad59420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb4ad6a60f7c5f00 server_cookie=607d8e2e240e4e84 in_seq=0 out_seq=0 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c697fa640 1 -- 172.21.15.81:0/3233904410 <== mds.0 v2:172.21.15.167:6835/54677014 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55c31ac6a580 con 0x55c31a90fe40 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c697fa640 10 client.13008 handle_command_reply: tid=0 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 client.13008 shutdown 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3233904410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 msgr2=0x55c31ad357f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad357f0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f3c84009920 tx=0x7f3c840035f0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3233904410 shutdown_connections 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad357f0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:6835/54677014,v1:172.21.15.167:6837/54677014] conn(0x55c31a90fe40 0x55c31ad59420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a930590 0x55c31ad1fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3233904410 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a9283b0 0x55c31ad4c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c697fa640 0 client.13008 ms_handle_reset on v2:172.21.15.167:6835/54677014 2024-06-29T15:59:15.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3233904410 >> 172.21.15.81:0/3233904410 conn(0x55c31a9453c0 msgr2=0x55c31ad1dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:15.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3233904410 shutdown_connections 2024-06-29T15:59:15.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3233904410 wait complete. 2024-06-29T15:59:15.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.514+0000 7f3c97b7f740 20 client.13008 trim_cache size 0 max 16384 2024-06-29T15:59:15.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.533+0000 7f3c97b7f740 1 Processor -- start 2024-06-29T15:59:15.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.533+0000 7f3c97b7f740 1 -- start start 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a9283b0 0x55c31ad4b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a930590 0x55c31acf4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31acf5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c200830 con 0x55c31a951270 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c1f5380 con 0x55c31a9283b0 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8c2003d0 con 0x55c31a930590 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c8ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31acf5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c8ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31acf5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60424/0 (socket says 172.21.15.81:60424) 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/2956551510 learned_addr learned my addr 172.21.15.81:0/2956551510 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/2956551510 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a930590 msgr2=0x55c31acf4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c8ad9d640 1 --2- 172.21.15.81:0/2956551510 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a930590 0x55c31acf4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/2956551510 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a9283b0 msgr2=0x55c31ad4b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:15.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c8ad9d640 1 --2- 172.21.15.81:0/2956551510 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a9283b0 0x55c31ad4b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/2956551510 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c31acf8280 con 0x55c31a951270 2024-06-29T15:59:15.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c8ad9d640 1 --2- 172.21.15.81:0/2956551510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31acf5290 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f3c840043e0 tx=0x7f3c84004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467562a26f6ac4f server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:15.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c6b7fe640 1 -- 172.21.15.81:0/2956551510 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c84003ba0 con 0x55c31a951270 2024-06-29T15:59:15.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c6b7fe640 1 -- 172.21.15.81:0/2956551510 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c84003d40 con 0x55c31a951270 2024-06-29T15:59:15.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.535+0000 7f3c6b7fe640 1 -- 172.21.15.81:0/2956551510 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c840049e0 con 0x55c31a951270 2024-06-29T15:59:15.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.536+0000 7f3c97b7f740 1 -- 172.21.15.81:0/2956551510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 msgr2=0x55c31acf5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.536+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/2956551510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31acf5290 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f3c840043e0 tx=0x7f3c84004410 comp rx=0 tx=0).stop 2024-06-29T15:59:15.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.536+0000 7f3c97b7f740 1 -- 172.21.15.81:0/2956551510 shutdown_connections 2024-06-29T15:59:15.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.536+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/2956551510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31acf5290 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.536+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/2956551510 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a930590 0x55c31acf4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.536+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/2956551510 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a9283b0 0x55c31ad4b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.536+0000 7f3c97b7f740 1 -- 172.21.15.81:0/2956551510 >> 172.21.15.81:0/2956551510 conn(0x55c31a9453c0 msgr2=0x55c31aced340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:15.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.536+0000 7f3c97b7f740 1 -- 172.21.15.81:0/2956551510 shutdown_connections 2024-06-29T15:59:15.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.536+0000 7f3c97b7f740 1 -- 172.21.15.81:0/2956551510 wait complete. 2024-06-29T15:59:15.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.537+0000 7f3c97b7f740 1 -- start start 2024-06-29T15:59:15.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.537+0000 7f3c97b7f740 1 Processor -- start 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.538+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a9283b0 0x55c31adb92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.538+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a930590 0x55c31adbd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.538+0000 7f3c97b7f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad1d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.538+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c31ad20160 con 0x55c31a951270 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.538+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x55c31ad21180 con 0x55c31a930590 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.538+0000 7f3c97b7f740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x55c31ad4f200 con 0x55c31a9283b0 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.538+0000 7f3c97b7f740 10 client.0 _ll_register_callbacks cb 0x7f3c84009450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c8ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad1d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c8ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad1d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60434/0 (socket says 172.21.15.81:60434) 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/3744586098 learned_addr learned my addr 172.21.15.81:0/3744586098 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c89d9b640 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a930590 0x55c31adbd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a9283b0 0x55c31adb92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a9283b0 msgr2=0x55c31adb92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c8ad9d640 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a9283b0 0x55c31adb92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a930590 msgr2=0x55c31adbd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c8ad9d640 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a930590 0x55c31adbd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c8ad9d640 1 -- 172.21.15.81:0/3744586098 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c31adb9830 con 0x55c31a951270 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c8ad9d640 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad1d2e0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f3c84005770 tx=0x7f3c84009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226120b823d243c0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c697fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c697fa640 1 -- 172.21.15.81:0/3744586098 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8405f060 con 0x55c31a951270 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c697fa640 1 -- 172.21.15.81:0/3744586098 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7f3c8405ab80 con 0x55c31a951270 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3744586098 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c84003d40 con 0x55c31a951270 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c697fa640 1 -- 172.21.15.81:0/3744586098 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=145) ==== 24+0+0 (secure 0 0 0) 0x7f3c8405ad80 con 0x55c31a951270 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c97b7f740 10 client.13014 fetch_fsmap learned FSMap version 145 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.539+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3744586098 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=145,monmap=2+}) -- 0x55c31adbd120 con 0x55c31a951270 2024-06-29T15:59:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c697fa640 1 -- 172.21.15.81:0/3744586098 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c84003d40 con 0x55c31a951270 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c697fa640 1 -- 172.21.15.81:0/3744586098 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 145) ==== 3502+0+0 (secure 0 0 0) 0x7f3c8404f040 con 0x55c31a951270 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c97b7f740 10 client.13014 fetch_fsmap finished waiting for FSMap version 145 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c97b7f740 10 client.13014 resolve_mds: resolved name 'd' to daemon mds.d 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c97b7f740 20 client.13014 populate_metadata read hostname 'smithi081' 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:6835/54677014,v1:172.21.15.167:6837/54677014] conn(0x55c31acf5080 0x55c31ad343c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c97b7f740 4 client.13014 mds_command: new command op to 11347 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3744586098 --> [v2:172.21.15.167:6835/54677014,v1:172.21.15.167:6837/54677014] -- command(tid 0: {"prefix": "status"}) -- 0x55c31ac66020 con 0x55c31acf5080 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:6835/54677014,v1:172.21.15.167:6837/54677014] conn(0x55c31acf5080 0x55c31ad343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c697fa640 10 client.13014 ms_handle_connect on v2:172.21.15.167:6835/54677014 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.540+0000 7f3c8a59c640 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:6835/54677014,v1:172.21.15.167:6837/54677014] conn(0x55c31acf5080 0x55c31ad343c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5ca958137144a31 server_cookie=cd178e0a5681d166 in_seq=0 out_seq=0 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c697fa640 1 -- 172.21.15.81:0/3744586098 <== mds.0 v2:172.21.15.167:6835/54677014 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55c31ac66020 con 0x55c31acf5080 2024-06-29T15:59:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c697fa640 10 client.13014 handle_command_reply: tid=0 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 client.13014 shutdown 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3744586098 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 msgr2=0x55c31ad1d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad1d2e0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f3c84005770 tx=0x7f3c84009260 comp rx=0 tx=0).stop 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3744586098 shutdown_connections 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c31a951270 0x55c31ad1d2e0 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:6835/54677014,v1:172.21.15.167:6837/54677014] conn(0x55c31acf5080 0x55c31ad343c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x55c31a930590 0x55c31adbd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 --2- 172.21.15.81:0/3744586098 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x55c31a9283b0 0x55c31adb92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c697fa640 0 client.13014 ms_handle_reset on v2:172.21.15.167:6835/54677014 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3744586098 >> 172.21.15.81:0/3744586098 conn(0x55c31a9453c0 msgr2=0x55c31ad4b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3744586098 shutdown_connections 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 1 -- 172.21.15.81:0/3744586098 wait complete. 2024-06-29T15:59:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c97b7f740 20 client.13014 trim_cache size 0 max 16384 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 11347, 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.777062291, 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 145, 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 253, 2024-06-29T15:59:15.548 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 253, 2024-06-29T15:59:15.549 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.397440955 2024-06-29T15:59:15.549 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:59:15.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c90cc1640 1 -- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3c6c076510 msgr2=0x7f3c6c078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c90cc1640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3c6c076510 0x7f3c6c078a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3c80000f60 tx=0x7f3c800079b0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c90cc1640 1 -- 172.21.15.81:0/415976884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c8c06ef90 msgr2=0x7f3c8c0d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c90cc1640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c8c06ef90 0x7f3c8c0d02f0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00ea50 tx=0x7f3c7c00ef10 comp rx=0 tx=0).stop 2024-06-29T15:59:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c90cc1640 1 -- 172.21.15.81:0/415976884 shutdown_connections 2024-06-29T15:59:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c90cc1640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f3c6c076510 0x7f3c6c078a20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c90cc1640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3c8c0b57b0 0x7f3c8c0aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c90cc1640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3c8c0b3960 0x7f3c8c14ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c90cc1640 1 --2- 172.21.15.81:0/415976884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c8c06ef90 0x7f3c8c0d02f0 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.541+0000 7f3c90cc1640 1 -- 172.21.15.81:0/415976884 >> 172.21.15.81:0/415976884 conn(0x7f3c8c0d0930 msgr2=0x7f3c8c0cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.542+0000 7f3c90cc1640 1 -- 172.21.15.81:0/415976884 shutdown_connections 2024-06-29T15:59:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.542+0000 7f3c90cc1640 1 -- 172.21.15.81:0/415976884 wait complete. 2024-06-29T15:59:15.554 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:59:15.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.666+0000 7f25002f6640 1 -- 172.21.15.81:0/3641850700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24e4003a50 msgr2=0x7f24e4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.666+0000 7f25002f6640 1 --2- 172.21.15.81:0/3641850700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24e4003a50 0x7f24e4003ec0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f24f4009b50 tx=0x7f24e4005830 comp rx=0 tx=0).stop 2024-06-29T15:59:15.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.666+0000 7f25002f6640 1 -- 172.21.15.81:0/3641850700 shutdown_connections 2024-06-29T15:59:15.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.666+0000 7f25002f6640 1 --2- 172.21.15.81:0/3641850700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24e4003a50 0x7f24e4003ec0 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.666+0000 7f25002f6640 1 -- 172.21.15.81:0/3641850700 >> 172.21.15.81:0/3641850700 conn(0x7f24f80d0930 msgr2=0x7f24f80d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:15.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.666+0000 7f25002f6640 1 -- 172.21.15.81:0/3641850700 shutdown_connections 2024-06-29T15:59:15.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.666+0000 7f25002f6640 1 -- 172.21.15.81:0/3641850700 wait complete. 2024-06-29T15:59:15.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.668+0000 7f25002f6640 1 Processor -- start 2024-06-29T15:59:15.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.668+0000 7f25002f6640 1 -- start start 2024-06-29T15:59:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.669+0000 7f25002f6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24e4003a50 0x7f24f81f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.669+0000 7f25002f6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f815e490 0x7f24f8140a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.669+0000 7f25002f6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80cfaa0 0x7f24f80cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.669+0000 7f25002f6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f24f806f3a0 con 0x7f24f80cfaa0 2024-06-29T15:59:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.669+0000 7f25002f6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f24f815b2f0 con 0x7f24e4003a50 2024-06-29T15:59:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.669+0000 7f25002f6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f24f806f520 con 0x7f24f815e490 2024-06-29T15:59:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.669+0000 7f24fe893640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80cfaa0 0x7f24f80cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24fe893640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80cfaa0 0x7f24f80cfeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60452/0 (socket says 172.21.15.81:60452) 2024-06-29T15:59:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24fe893640 1 -- 172.21.15.81:0/2680164286 learned_addr learned my addr 172.21.15.81:0/2680164286 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24fd891640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f815e490 0x7f24f8140a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24fe092640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24e4003a50 0x7f24f81f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24fe893640 1 -- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f815e490 msgr2=0x7f24f8140a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24fe893640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f815e490 0x7f24f8140a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24fe893640 1 -- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24e4003a50 msgr2=0x7f24f81f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24fe893640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24e4003a50 0x7f24f81f6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24fe893640 1 -- 172.21.15.81:0/2680164286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f80d03f0 con 0x7f24f80cfaa0 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24fe893640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80cfaa0 0x7f24f80cfeb0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f24f80cda30 tx=0x7f24f00019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24eeffd640 1 -- 172.21.15.81:0/2680164286 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f0017070 con 0x7f24f80cfaa0 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24eeffd640 1 -- 172.21.15.81:0/2680164286 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f24f00108e0 con 0x7f24f80cfaa0 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f25002f6640 1 -- 172.21.15.81:0/2680164286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f81435d0 con 0x7f24f80cfaa0 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f24eeffd640 1 -- 172.21.15.81:0/2680164286 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f0010d30 con 0x7f24f80cfaa0 2024-06-29T15:59:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.670+0000 7f25002f6640 1 -- 172.21.15.81:0/2680164286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f8143b90 con 0x7f24f80cfaa0 2024-06-29T15:59:15.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.671+0000 7f25002f6640 1 -- 172.21.15.81:0/2680164286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f806e2a0 con 0x7f24f80cfaa0 2024-06-29T15:59:15.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.672+0000 7f24eeffd640 1 -- 172.21.15.81:0/2680164286 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f24f001a070 con 0x7f24f80cfaa0 2024-06-29T15:59:15.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.672+0000 7f24eeffd640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f24dc076510 0x7f24dc078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:15.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.673+0000 7f24eeffd640 1 -- 172.21.15.81:0/2680164286 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7f24f0097a10 con 0x7f24f80cfaa0 2024-06-29T15:59:15.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.673+0000 7f24fe092640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f24dc076510 0x7f24dc078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:15.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.673+0000 7f24fe092640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f24dc076510 0x7f24dc078a20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f24f4035000 tx=0x7f24f40046c0 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:15.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.677+0000 7f24eeffd640 1 -- 172.21.15.81:0/2680164286 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f24f806e2a0 con 0x7f24f80cfaa0 2024-06-29T15:59:15.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.897+0000 7f25002f6640 1 -- 172.21.15.81:0/2680164286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f24f80b3d70 con 0x7f24f80cfaa0 2024-06-29T15:59:15.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.898+0000 7f24eeffd640 1 -- 172.21.15.81:0/2680164286 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v145) ==== 53+0+73 (secure 0 0 0) 0x7f24f0067e20 con 0x7f24f80cfaa0 2024-06-29T15:59:15.899 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:59:15.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 -- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f24dc076510 msgr2=0x7f24dc078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f24dc076510 0x7f24dc078a20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f24f4035000 tx=0x7f24f40046c0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 -- 172.21.15.81:0/2680164286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80cfaa0 msgr2=0x7f24f80cfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:15.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80cfaa0 0x7f24f80cfeb0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f24f80cda30 tx=0x7f24f00019b0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 -- 172.21.15.81:0/2680164286 shutdown_connections 2024-06-29T15:59:15.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f24dc076510 0x7f24dc078a20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f80cfaa0 0x7f24f80cfeb0 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f24f815e490 0x7f24f8140a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 --2- 172.21.15.81:0/2680164286 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f24e4003a50 0x7f24f81f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:15.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 -- 172.21.15.81:0/2680164286 >> 172.21.15.81:0/2680164286 conn(0x7f24f80d0930 msgr2=0x7f24f8156c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:15.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 -- 172.21.15.81:0/2680164286 shutdown_connections 2024-06-29T15:59:15.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:15.900+0000 7f25002f6640 1 -- 172.21.15.81:0/2680164286 wait complete. 2024-06-29T15:59:15.912 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:59:15.913 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:59:15.913 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T15:59:15.913 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:59:15.913 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:15.913 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:59:15.913 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:59:15.913 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:59:15.913 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:15.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:15.913 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T15:59:15.929 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:59:15.929 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:59:15.929 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:59:15.929 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:15.929 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T15:59:15.929 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:15.929 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 36964sec preferred_lft 36964sec 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T15:59:15.930 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T15:59:15.931 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:15.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:15.932 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:59:15.993 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T15:59:15.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:15.994 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T15:59:16.050 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T15:59:16.050 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:59:16.050 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T15:59:16.050 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T15:59:16.050 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:59:16.051 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:59:16.051 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T15:59:16.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T15:59:16.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T15:59:16.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T15:59:16.051 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T15:59:16.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:59:16.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:59:16.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:16.177 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T15:59:16.192 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:59:16.192 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:59:16.192 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:16.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:16.192 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T15:59:16.257 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:59:16.257 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:16.312 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:16.313 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:59:16.313 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:16.368 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T15:59:16.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:16.369 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:59:16.465 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:59:16.465 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:59:16.473 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:59:16.473 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:59:16.473 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T15:59:16.473 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T15:59:16.473 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T15:59:16.473 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:59:16.474 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:59:16.474 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:59:16.474 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:59:16.474 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:59:16.475 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:59:16.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:16.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:16.501 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:59:16.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:59:16.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:59:16.604 INFO:teuthology.orchestra.run.smithi081.stdout:client9935 2024-06-29T15:59:16.604 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:59:16.605 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:16.620 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:59:16.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.773+0000 7fbf10903640 1 -- 172.21.15.81:0/1898672810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0c0d0230 msgr2=0x7fbf0c0d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.773+0000 7fbf10903640 1 --2- 172.21.15.81:0/1898672810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0c0d0230 0x7fbf0c0d0640 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0c5590 tx=0x7fbef402a4a0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.773+0000 7fbf10903640 1 -- 172.21.15.81:0/1898672810 shutdown_connections 2024-06-29T15:59:16.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.773+0000 7fbf10903640 1 --2- 172.21.15.81:0/1898672810 >> v2:172.21.15.167:3300/0 conn(0x7fbf0c0b4f40 0x7fbf0c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.773+0000 7fbf10903640 1 --2- 172.21.15.81:0/1898672810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0c0d0230 0x7fbf0c0d0640 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.773+0000 7fbf10903640 1 -- 172.21.15.81:0/1898672810 >> 172.21.15.81:0/1898672810 conn(0x7fbf0c0b3680 msgr2=0x7fbf0c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:16.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.773+0000 7fbf10903640 1 -- 172.21.15.81:0/1898672810 shutdown_connections 2024-06-29T15:59:16.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.773+0000 7fbf10903640 1 -- 172.21.15.81:0/1898672810 wait complete. 2024-06-29T15:59:16.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.775+0000 7fbf10903640 1 Processor -- start 2024-06-29T15:59:16.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.775+0000 7fbf10903640 1 -- start start 2024-06-29T15:59:16.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf10903640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0c0b3200 0x7fbf0c159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf10903640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf0c0b4f40 0x7fbf0c09c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf10903640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf0c0d0230 0x7fbf0c0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf10903640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c0b5c20 con 0x7fbf0c0b3200 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf10903640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c0b5aa0 con 0x7fbf0c0d0230 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf10903640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0c0b5da0 con 0x7fbf0c0b4f40 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0c0b3200 0x7fbf0c159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0c0b3200 0x7fbf0c159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60464/0 (socket says 172.21.15.81:60464) 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0a59c640 1 -- 172.21.15.81:0/2214787000 learned_addr learned my addr 172.21.15.81:0/2214787000 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0ad9d640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf0c0d0230 0x7fbf0c0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf03fff640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf0c0b4f40 0x7fbf0c09c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0a59c640 1 -- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf0c0b4f40 msgr2=0x7fbf0c09c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0a59c640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf0c0b4f40 0x7fbf0c09c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0a59c640 1 -- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf0c0d0230 msgr2=0x7fbf0c0afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0a59c640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf0c0d0230 0x7fbf0c0afea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0a59c640 1 -- 172.21.15.81:0/2214787000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c09d400 con 0x7fbf0c0b3200 2024-06-29T15:59:16.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0ad9d640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf0c0d0230 0x7fbf0c0afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:16.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf03fff640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf0c0b4f40 0x7fbf0c09c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:16.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf0a59c640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0c0b3200 0x7fbf0c159920 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00c3c0 tx=0x7fbefc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:16.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf02ffd640 1 -- 172.21.15.81:0/2214787000 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc017070 con 0x7fbf0c0b3200 2024-06-29T15:59:16.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf02ffd640 1 -- 172.21.15.81:0/2214787000 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbefc0035f0 con 0x7fbf0c0b3200 2024-06-29T15:59:16.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf10903640 1 -- 172.21.15.81:0/2214787000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0c09d6e0 con 0x7fbf0c0b3200 2024-06-29T15:59:16.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.777+0000 7fbf02ffd640 1 -- 172.21.15.81:0/2214787000 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc003a40 con 0x7fbf0c0b3200 2024-06-29T15:59:16.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.778+0000 7fbf10903640 1 -- 172.21.15.81:0/2214787000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0c09dca0 con 0x7fbf0c0b3200 2024-06-29T15:59:16.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.778+0000 7fbf10903640 1 -- 172.21.15.81:0/2214787000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf0c0b5350 con 0x7fbf0c0b3200 2024-06-29T15:59:16.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.779+0000 7fbf02ffd640 1 -- 172.21.15.81:0/2214787000 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7fbefc01a070 con 0x7fbf0c0b3200 2024-06-29T15:59:16.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.780+0000 7fbf02ffd640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbeec076510 0x7fbeec078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:16.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.780+0000 7fbf02ffd640 1 -- 172.21.15.81:0/2214787000 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7fbefc097060 con 0x7fbf0c0b3200 2024-06-29T15:59:16.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.780+0000 7fbf03fff640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbeec076510 0x7fbeec078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:16.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.780+0000 7fbf03fff640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbeec076510 0x7fbeec078a20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbef4034000 tx=0x7fbef4004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:16.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.784+0000 7fbf02ffd640 1 -- 172.21.15.81:0/2214787000 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbf0c0d0b80 con 0x7fbf0c0b3200 2024-06-29T15:59:16.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.989+0000 7fbf10903640 1 -- 172.21.15.81:0/2214787000 --> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7fbf0c0cf0e0 con 0x7fbeec076510 2024-06-29T15:59:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.991+0000 7fbf02ffd640 1 -- 172.21.15.81:0/2214787000 <== mgr.11053 v2:172.21.15.167:6832/1778875525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7fbf0c0cf0e0 con 0x7fbeec076510 2024-06-29T15:59:16.993 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:59:16.993 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:16.993 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T15:59:16.993 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:16.993 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:59:16.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 -- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbeec076510 msgr2=0x7fbeec078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbeec076510 0x7fbeec078a20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbef4034000 tx=0x7fbef4004730 comp rx=0 tx=0).stop 2024-06-29T15:59:16.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 -- 172.21.15.81:0/2214787000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0c0b3200 msgr2=0x7fbf0c159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:16.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0c0b3200 0x7fbf0c159920 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7fbefc00c3c0 tx=0x7fbefc00c880 comp rx=0 tx=0).stop 2024-06-29T15:59:16.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 -- 172.21.15.81:0/2214787000 shutdown_connections 2024-06-29T15:59:16.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fbeec076510 0x7fbeec078a20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbf0c0d0230 0x7fbf0c0afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbf0c0b4f40 0x7fbf0c09c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 --2- 172.21.15.81:0/2214787000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0c0b3200 0x7fbf0c159920 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:16.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 -- 172.21.15.81:0/2214787000 >> 172.21.15.81:0/2214787000 conn(0x7fbf0c0b3680 msgr2=0x7fbf0c0cdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:16.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 -- 172.21.15.81:0/2214787000 shutdown_connections 2024-06-29T15:59:16.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:16.994+0000 7fbf10903640 1 -- 172.21.15.81:0/2214787000 wait complete. 2024-06-29T15:59:17.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:59:17.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.120+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1935362629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c000b57b0 msgr2=0x7f5c0015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.120+0000 7f5c05b9e640 1 --2- 172.21.15.81:0/1935362629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c000b57b0 0x7f5c0015aed0 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0009830 tx=0x7f5bf002a510 comp rx=0 tx=0).stop 2024-06-29T15:59:17.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.120+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1935362629 shutdown_connections 2024-06-29T15:59:17.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.120+0000 7f5c05b9e640 1 --2- 172.21.15.81:0/1935362629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c000b57b0 0x7f5c0015aed0 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.120+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1935362629 >> 172.21.15.81:0/1935362629 conn(0x7f5c000d0930 msgr2=0x7f5c000d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:17.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.120+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1935362629 shutdown_connections 2024-06-29T15:59:17.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.120+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1935362629 wait complete. 2024-06-29T15:59:17.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.122+0000 7f5c05b9e640 1 Processor -- start 2024-06-29T15:59:17.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.122+0000 7f5c05b9e640 1 -- start start 2024-06-29T15:59:17.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5c05b9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c0006ef90 0x7f5c0014e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5c05b9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c000b3960 0x7f5c0006c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5c05b9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c000b57b0 0x7f5c0006c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5c05b9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5c000b4600 con 0x7f5c000b57b0 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5c05b9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5c000b4480 con 0x7f5c0006ef90 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5c05b9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5c000b4780 con 0x7f5c000b3960 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bfffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c000b57b0 0x7f5c0006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bfffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c000b57b0 0x7f5c0006c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60472/0 (socket says 172.21.15.81:60472) 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bfffff640 1 -- 172.21.15.81:0/1203738249 learned_addr learned my addr 172.21.15.81:0/1203738249 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bff7fe640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c0006ef90 0x7f5c0014e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bfeffd640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c000b3960 0x7f5c0006c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bfffff640 1 -- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c000b3960 msgr2=0x7f5c0006c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bfffff640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c000b3960 0x7f5c0006c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bfffff640 1 -- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c0006ef90 msgr2=0x7f5c0014e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bfffff640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c0006ef90 0x7f5c0014e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bfffff640 1 -- 172.21.15.81:0/1203738249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0014ea70 con 0x7f5c000b57b0 2024-06-29T15:59:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5bfffff640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c000b57b0 0x7f5c0006c860 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800b9e0 tx=0x7f5bf800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:17.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.124+0000 7f5be7fff640 1 -- 172.21.15.81:0/1203738249 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8003b70 con 0x7f5c000b57b0 2024-06-29T15:59:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.125+0000 7f5be7fff640 1 -- 172.21.15.81:0/1203738249 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5bf8003e00 con 0x7f5c000b57b0 2024-06-29T15:59:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.125+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1203738249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c000af560 con 0x7f5c000b57b0 2024-06-29T15:59:17.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.125+0000 7f5be7fff640 1 -- 172.21.15.81:0/1203738249 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf800cb80 con 0x7f5c000b57b0 2024-06-29T15:59:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.125+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1203738249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c0014fb00 con 0x7f5c000b57b0 2024-06-29T15:59:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.125+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1203738249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c000af7d0 con 0x7f5c000b57b0 2024-06-29T15:59:17.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.126+0000 7f5be7fff640 1 -- 172.21.15.81:0/1203738249 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 146) ==== 98405+0+0 (secure 0 0 0) 0x7f5bf8004000 con 0x7f5c000b57b0 2024-06-29T15:59:17.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.127+0000 7f5be7fff640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5bec076510 0x7f5bec078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.127+0000 7f5be7fff640 1 -- 172.21.15.81:0/1203738249 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7f5bf80979c0 con 0x7f5c000b57b0 2024-06-29T15:59:17.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.127+0000 7f5bff7fe640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5bec076510 0x7f5bec078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.128+0000 7f5bff7fe640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5bec076510 0x7f5bec078a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0034000 tx=0x7f5bf0004730 comp rx=0 tx=0).ready entity=mgr.11053 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:17.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.131+0000 7f5be7fff640 1 -- 172.21.15.81:0/1203738249 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5bf8099e20 con 0x7f5c000b57b0 2024-06-29T15:59:17.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.362+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1203738249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f5c0006e2a0 con 0x7f5c000b57b0 2024-06-29T15:59:17.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.816+0000 7f5be7fff640 1 -- 172.21.15.81:0/1203738249 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v147) ==== 92+0+0 (secure 0 0 0) 0x7f5c0006e2a0 con 0x7f5c000b57b0 2024-06-29T15:59:17.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.819+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5bec076510 msgr2=0x7f5bec078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.819+0000 7f5c05b9e640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5bec076510 0x7f5bec078a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0034000 tx=0x7f5bf0004730 comp rx=0 tx=0).stop 2024-06-29T15:59:17.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.819+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1203738249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c000b57b0 msgr2=0x7f5c0006c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.819+0000 7f5c05b9e640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c000b57b0 0x7f5c0006c860 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800b9e0 tx=0x7f5bf800bea0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.820+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1203738249 shutdown_connections 2024-06-29T15:59:17.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.820+0000 7f5c05b9e640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f5bec076510 0x7f5bec078a20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.820+0000 7f5c05b9e640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c000b57b0 0x7f5c0006c860 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.820+0000 7f5c05b9e640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c000b3960 0x7f5c0006c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.820+0000 7f5c05b9e640 1 --2- 172.21.15.81:0/1203738249 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c0006ef90 0x7f5c0014e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.820+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1203738249 >> 172.21.15.81:0/1203738249 conn(0x7f5c000d0930 msgr2=0x7f5c001561a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.820+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1203738249 shutdown_connections 2024-06-29T15:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.820+0000 7f5c05b9e640 1 -- 172.21.15.81:0/1203738249 wait complete. 2024-06-29T15:59:17.835 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:59:17.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.956+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1836663822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc108073650 msgr2=0x7fc10815b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.956+0000 7fc10e8fd640 1 --2- 172.21.15.81:0/1836663822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc108073650 0x7fc10815b6e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009860 tx=0x7fc0fc02a510 comp rx=0 tx=0).stop 2024-06-29T15:59:17.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.957+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1836663822 shutdown_connections 2024-06-29T15:59:17.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.957+0000 7fc10e8fd640 1 --2- 172.21.15.81:0/1836663822 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc108073650 0x7fc10815b6e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.957+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1836663822 >> 172.21.15.81:0/1836663822 conn(0x7fc1080b2920 msgr2=0x7fc1080b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:17.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.957+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1836663822 shutdown_connections 2024-06-29T15:59:17.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.957+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1836663822 wait complete. 2024-06-29T15:59:17.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.958+0000 7fc10e8fd640 1 Processor -- start 2024-06-29T15:59:17.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.959+0000 7fc10e8fd640 1 -- start start 2024-06-29T15:59:17.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc10e8fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1080cd150 0x7fc108157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc10e8fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc10815bc20 0x7fc1081484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc10e8fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1081472d0 0x7fc108148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc10e8fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc10806f830 con 0x7fc1081472d0 2024-06-29T15:59:17.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc10e8fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc10806f6b0 con 0x7fc1080cd150 2024-06-29T15:59:17.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc10e8fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc10806f9b0 con 0x7fc10815bc20 2024-06-29T15:59:17.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc10ce9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1081472d0 0x7fc108148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc1077fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc10815bc20 0x7fc1081484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc10ce9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1081472d0 0x7fc108148a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60484/0 (socket says 172.21.15.81:60484) 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc107fff640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1080cd150 0x7fc108157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc10ce9a640 1 -- 172.21.15.81:0/1400570215 learned_addr learned my addr 172.21.15.81:0/1400570215 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc107fff640 1 -- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc10815bc20 msgr2=0x7fc1081484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc107fff640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc10815bc20 0x7fc1081484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc107fff640 1 -- 172.21.15.81:0/1400570215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1081472d0 msgr2=0x7fc108148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc107fff640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1081472d0 0x7fc108148a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc107fff640 1 -- 172.21.15.81:0/1400570215 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc108158d00 con 0x7fc1080cd150 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc10ce9a640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1081472d0 0x7fc108148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.960+0000 7fc1077fe640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc10815bc20 0x7fc1081484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.961+0000 7fc107fff640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1080cd150 0x7fc108157e50 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800c390 tx=0x7fc0f800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.961+0000 7fc104ff9640 1 -- 172.21.15.81:0/1400570215 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8011790 con 0x7fc1080cd150 2024-06-29T15:59:17.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.961+0000 7fc104ff9640 1 -- 172.21.15.81:0/1400570215 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0f8017070 con 0x7fc1080cd150 2024-06-29T15:59:17.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.961+0000 7fc104ff9640 1 -- 172.21.15.81:0/1400570215 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8011ce0 con 0x7fc1080cd150 2024-06-29T15:59:17.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.961+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1400570215 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1081f6490 con 0x7fc1080cd150 2024-06-29T15:59:17.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.961+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1400570215 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1081f6a50 con 0x7fc1080cd150 2024-06-29T15:59:17.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.962+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1400570215 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc10815b6e0 con 0x7fc1080cd150 2024-06-29T15:59:17.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.963+0000 7fc104ff9640 1 -- 172.21.15.81:0/1400570215 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 147) ==== 98422+0+0 (secure 0 0 0) 0x7fc0f801c460 con 0x7fc1080cd150 2024-06-29T15:59:17.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.963+0000 7fc104ff9640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fc0f4076560 0x7fc0f4078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:17.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.963+0000 7fc1077fe640 1 -- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fc0f4076560 msgr2=0x7fc0f4078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1778875525 2024-06-29T15:59:17.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.963+0000 7fc104ff9640 1 -- 172.21.15.81:0/1400570215 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7fc0f80977a0 con 0x7fc1080cd150 2024-06-29T15:59:17.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.963+0000 7fc1077fe640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fc0f4076560 0x7fc0f4078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:17.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:17.968+0000 7fc104ff9640 1 -- 172.21.15.81:0/1400570215 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc0f8007e20 con 0x7fc1080cd150 2024-06-29T15:59:18.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.153+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1400570215 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fc1080b0cf0 con 0x7fc1080cd150 2024-06-29T15:59:18.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.161+0000 7fc104ff9640 1 -- 172.21.15.81:0/1400570215 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v81)=0 v81) ==== 138+0+0 (secure 0 0 0) 0x7fc0f8067e20 con 0x7fc1080cd150 2024-06-29T15:59:18.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fc0f4076560 msgr2=0x7fc0f4078a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:18.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fc0f4076560 0x7fc0f4078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1080cd150 msgr2=0x7fc108157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1080cd150 0x7fc108157e50 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800c390 tx=0x7fc0f800c850 comp rx=0 tx=0).stop 2024-06-29T15:59:18.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1400570215 shutdown_connections 2024-06-29T15:59:18.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fc0f4076560 0x7fc0f4078a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1081472d0 0x7fc108148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc10815bc20 0x7fc1081484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 --2- 172.21.15.81:0/1400570215 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc1080cd150 0x7fc108157e50 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1400570215 >> 172.21.15.81:0/1400570215 conn(0x7fc1080b2920 msgr2=0x7fc10815ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:18.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.163+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1400570215 shutdown_connections 2024-06-29T15:59:18.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.164+0000 7fc10e8fd640 1 -- 172.21.15.81:0/1400570215 wait complete. 2024-06-29T15:59:18.177 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:59:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.293+0000 7fdc44168640 1 -- 172.21.15.81:0/2972481812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc3c0b5250 msgr2=0x7fdc3c0d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.293+0000 7fdc44168640 1 --2- 172.21.15.81:0/2972481812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc3c0b5250 0x7fdc3c0d09c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009830 tx=0x7fdc3802a5a0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.293+0000 7fdc44168640 1 -- 172.21.15.81:0/2972481812 shutdown_connections 2024-06-29T15:59:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.293+0000 7fdc44168640 1 --2- 172.21.15.81:0/2972481812 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc3c0b5250 0x7fdc3c0d09c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.293+0000 7fdc44168640 1 --2- 172.21.15.81:0/2972481812 >> v2:172.21.15.167:3300/0 conn(0x7fdc3c0b9040 0x7fdc3c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.293+0000 7fdc44168640 1 -- 172.21.15.81:0/2972481812 >> 172.21.15.81:0/2972481812 conn(0x7fdc3c09cf30 msgr2=0x7fdc3c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.293+0000 7fdc44168640 1 -- 172.21.15.81:0/2972481812 shutdown_connections 2024-06-29T15:59:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.293+0000 7fdc44168640 1 -- 172.21.15.81:0/2972481812 wait complete. 2024-06-29T15:59:18.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.295+0000 7fdc44168640 1 Processor -- start 2024-06-29T15:59:18.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.295+0000 7fdc44168640 1 -- start start 2024-06-29T15:59:18.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc44168640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc3c0b5250 0x7fdc3c0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc44168640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc3c0b7290 0x7fdc3c1561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc44168640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c0b9040 0x7fdc3c0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc44168640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c0b7db0 con 0x7fdc3c0b9040 2024-06-29T15:59:18.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc44168640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c0d2d50 con 0x7fdc3c0b7290 2024-06-29T15:59:18.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc44168640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdc3c0b7f30 con 0x7fdc3c0b5250 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc41f04640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc3c0b5250 0x7fdc3c0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc41f04640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc3c0b5250 0x7fdc3c0b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:55844/0 (socket says 172.21.15.81:55844) 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc41f04640 1 -- 172.21.15.81:0/3739851916 learned_addr learned my addr 172.21.15.81:0/3739851916 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc42705640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c0b9040 0x7fdc3c0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.296+0000 7fdc41703640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc3c0b7290 0x7fdc3c1561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc41f04640 1 -- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc3c0b7290 msgr2=0x7fdc3c1561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc41f04640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc3c0b7290 0x7fdc3c1561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc41f04640 1 -- 172.21.15.81:0/3739851916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c0b9040 msgr2=0x7fdc3c0c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc41f04640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c0b9040 0x7fdc3c0c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc41f04640 1 -- 172.21.15.81:0/3739851916 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3c0b3c80 con 0x7fdc3c0b5250 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc41f04640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc3c0b5250 0x7fdc3c0b3740 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00e9f0 tx=0x7fdc2c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:18.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc327fc640 1 -- 172.21.15.81:0/3739851916 <== mon.2 v2:172.21.15.167:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c00cd20 con 0x7fdc3c0b5250 2024-06-29T15:59:18.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc327fc640 1 -- 172.21.15.81:0/3739851916 <== mon.2 v2:172.21.15.167:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc2c004020 con 0x7fdc3c0b5250 2024-06-29T15:59:18.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc44168640 1 -- 172.21.15.81:0/3739851916 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3c156710 con 0x7fdc3c0b5250 2024-06-29T15:59:18.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc327fc640 1 -- 172.21.15.81:0/3739851916 <== mon.2 v2:172.21.15.167:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c004470 con 0x7fdc3c0b5250 2024-06-29T15:59:18.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.297+0000 7fdc44168640 1 -- 172.21.15.81:0/3739851916 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3c0b4180 con 0x7fdc3c0b5250 2024-06-29T15:59:18.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.298+0000 7fdc44168640 1 -- 172.21.15.81:0/3739851916 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc3c0b84c0 con 0x7fdc3c0b5250 2024-06-29T15:59:18.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.299+0000 7fdc327fc640 1 -- 172.21.15.81:0/3739851916 <== mon.2 v2:172.21.15.167:3301/0 4 ==== mgrmap(e 147) ==== 98422+0+0 (secure 0 0 0) 0x7fdc2c0035f0 con 0x7fdc3c0b5250 2024-06-29T15:59:18.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.299+0000 7fdc327fc640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fdc20076560 0x7fdc20078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.299+0000 7fdc41703640 1 -- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fdc20076560 msgr2=0x7fdc20078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1778875525 2024-06-29T15:59:18.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.299+0000 7fdc327fc640 1 -- 172.21.15.81:0/3739851916 <== mon.2 v2:172.21.15.167:3301/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7fdc2c098740 con 0x7fdc3c0b5250 2024-06-29T15:59:18.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.299+0000 7fdc41703640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fdc20076560 0x7fdc20078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:18.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.304+0000 7fdc327fc640 1 -- 172.21.15.81:0/3739851916 <== mon.2 v2:172.21.15.167:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdc2c0614d0 con 0x7fdc3c0b5250 2024-06-29T15:59:18.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.490+0000 7fdc44168640 1 -- 172.21.15.81:0/3739851916 --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fdc3c1613e0 con 0x7fdc3c0b5250 2024-06-29T15:59:18.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.497+0000 7fdc327fc640 1 -- 172.21.15.81:0/3739851916 <== mon.2 v2:172.21.15.167:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v82)=0 v82) ==== 133+0+0 (secure 0 0 0) 0x7fdc2c0657d0 con 0x7fdc3c0b5250 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 -- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fdc20076560 msgr2=0x7fdc20078a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fdc20076560 0x7fdc20078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 -- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc3c0b5250 msgr2=0x7fdc3c0b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc3c0b5250 0x7fdc3c0b3740 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00e9f0 tx=0x7fdc2c00eeb0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 -- 172.21.15.81:0/3739851916 shutdown_connections 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7fdc20076560 0x7fdc20078a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c0b9040 0x7fdc3c0c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc3c0b7290 0x7fdc3c1561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 --2- 172.21.15.81:0/3739851916 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc3c0b5250 0x7fdc3c0b3740 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 -- 172.21.15.81:0/3739851916 >> 172.21.15.81:0/3739851916 conn(0x7fdc3c09cf30 msgr2=0x7fdc3c15f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 -- 172.21.15.81:0/3739851916 shutdown_connections 2024-06-29T15:59:18.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.499+0000 7fdc44168640 1 -- 172.21.15.81:0/3739851916 wait complete. 2024-06-29T15:59:18.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:18.512 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2024-06-29T15:59:18.533 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2024-06-29T15:59:18.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.688+0000 7f7870647640 1 -- 172.21.15.81:0/2386716596 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78680d0940 msgr2=0x7f78680b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.688+0000 7f7870647640 1 --2- 172.21.15.81:0/2386716596 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78680d0940 0x7f78680b4f40 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f7860005fd0 tx=0x7f786002a8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.689+0000 7f7870647640 1 -- 172.21.15.81:0/2386716596 shutdown_connections 2024-06-29T15:59:18.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.689+0000 7f7870647640 1 --2- 172.21.15.81:0/2386716596 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78680d0940 0x7f78680b4f40 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.689+0000 7f7870647640 1 --2- 172.21.15.81:0/2386716596 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78680b3200 0x7f78680cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.689+0000 7f7870647640 1 -- 172.21.15.81:0/2386716596 >> 172.21.15.81:0/2386716596 conn(0x7f78680b3680 msgr2=0x7f78680b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:18.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.689+0000 7f7870647640 1 -- 172.21.15.81:0/2386716596 shutdown_connections 2024-06-29T15:59:18.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.689+0000 7f7870647640 1 -- 172.21.15.81:0/2386716596 wait complete. 2024-06-29T15:59:18.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.690+0000 7f7870647640 1 Processor -- start 2024-06-29T15:59:18.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.690+0000 7f7870647640 1 -- start start 2024-06-29T15:59:18.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f7870647640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78680b3200 0x7f7868154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f7870647640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78680cccf0 0x7f78680c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f7870647640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78680b1340 0x7f78680b1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f7870647640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f78680b5c50 con 0x7f78680b3200 2024-06-29T15:59:18.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f7870647640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f78680b5ad0 con 0x7f78680cccf0 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f7870647640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f78680b5dd0 con 0x7f78680b1340 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786e3e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78680b3200 0x7f7868154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786e3e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78680b3200 0x7f7868154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60490/0 (socket says 172.21.15.81:60490) 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786e3e3640 1 -- 172.21.15.81:0/300847313 learned_addr learned my addr 172.21.15.81:0/300847313 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786dbe2640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78680cccf0 0x7f78680c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786ebe4640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78680b1340 0x7f78680b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786dbe2640 1 -- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78680b1340 msgr2=0x7f78680b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786dbe2640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78680b1340 0x7f78680b1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786dbe2640 1 -- 172.21.15.81:0/300847313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78680b3200 msgr2=0x7f7868154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786dbe2640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78680b3200 0x7f7868154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786dbe2640 1 -- 172.21.15.81:0/300847313 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78680b1c90 con 0x7f78680cccf0 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.692+0000 7f786e3e3640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78680b3200 0x7f7868154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:18.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.693+0000 7f786dbe2640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78680cccf0 0x7f78680c5490 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f786400c390 tx=0x7f786400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:18.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.693+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864011790 con 0x7f78680cccf0 2024-06-29T15:59:18.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.693+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7864017070 con 0x7f78680cccf0 2024-06-29T15:59:18.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.693+0000 7f7870647640 1 -- 172.21.15.81:0/300847313 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78680c5f50 con 0x7f78680cccf0 2024-06-29T15:59:18.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.693+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7864011ce0 con 0x7f78680cccf0 2024-06-29T15:59:18.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.693+0000 7f7870647640 1 -- 172.21.15.81:0/300847313 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78680c6510 con 0x7f78680cccf0 2024-06-29T15:59:18.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.694+0000 7f7870647640 1 -- 172.21.15.81:0/300847313 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78680cc7b0 con 0x7f78680cccf0 2024-06-29T15:59:18.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.694+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 147) ==== 98422+0+0 (secure 0 0 0) 0x7f786401c460 con 0x7f78680cccf0 2024-06-29T15:59:18.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.695+0000 7f785effd640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 0x7f784c078a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:18.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.695+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(253..253 src has 1..253) ==== 5837+0+0 (secure 0 0 0) 0x7f7864097880 con 0x7f78680cccf0 2024-06-29T15:59:18.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.695+0000 7f786e3e3640 1 -- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 msgr2=0x7f784c078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1778875525 2024-06-29T15:59:18.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.695+0000 7f786e3e3640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 0x7f784c078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:18.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.700+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f78680b1c90 con 0x7f78680cccf0 2024-06-29T15:59:18.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.897+0000 7f786e3e3640 1 -- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 msgr2=0x7f784c078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1778875525 2024-06-29T15:59:18.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.897+0000 7f786e3e3640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 0x7f784c078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:18.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:18.906+0000 7f7870647640 1 -- 172.21.15.81:0/300847313 --> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f78680c5c70 con 0x7f784c076560 2024-06-29T15:59:19.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:19.298+0000 7f786e3e3640 1 -- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 msgr2=0x7f784c078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1778875525 2024-06-29T15:59:19.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:19.298+0000 7f786e3e3640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 0x7f784c078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:59:20.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:20.099+0000 7f786e3e3640 1 -- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 msgr2=0x7f784c078a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1778875525 2024-06-29T15:59:20.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:20.099+0000 7f786e3e3640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 0x7f784c078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:59:21.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:21.479+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mgrmap(e 148) ==== 97929+0+0 (secure 0 0 0) 0x7f786405ca60 con 0x7f78680cccf0 2024-06-29T15:59:21.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:21.479+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 msgr2=0x7f784c078a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:21.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:21.479+0000 7f785effd640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1778875525,v1:172.21.15.167:6833/1778875525] conn(0x7f784c076560 0x7f784c078a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:21.620 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:21.619+0000 7f733f357640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:21.620 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:21.619+0000 7f733f357640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:21.621 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:21.619+0000 7f733f357640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:21.621 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:21.619+0000 7f733f357640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:21.621 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:21.619+0000 7f733f357640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:21.621 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:21.619+0000 7f733c351640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:21.621 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:21.619+0000 7f733c351640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:21.621 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:21.619+0000 7f733c351640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:21.621 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:21.619+0000 7f733c351640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:21.621 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:21.619+0000 7f733c351640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:22.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.509+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 <== mon.1 v2:172.21.15.167:3300/0 8 ==== mgrmap(e 149) ==== 98056+0+0 (secure 0 0 0) 0x7f7864011c80 con 0x7f78680cccf0 2024-06-29T15:59:22.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.510+0000 7f785effd640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f784c03ae50 0x7f784c080610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:22.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.510+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 --> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f7864068ed0 con 0x7f784c03ae50 2024-06-29T15:59:22.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.512+0000 7f786e3e3640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f784c03ae50 0x7f784c080610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:22.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.514+0000 7f786e3e3640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f784c03ae50 0x7f784c080610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7858009490 tx=0x7f7858005f90 comp rx=0 tx=0).ready entity=mgr.11452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:22.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.576+0000 7f785effd640 1 -- 172.21.15.81:0/300847313 <== mgr.11452 v2:172.21.15.167:6832/1493691973 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7864068ed0 con 0x7f784c03ae50 2024-06-29T15:59:22.578 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path snap_test_dir1 2024-06-29T15:59:22.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.578+0000 7f7870647640 1 -- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f784c03ae50 msgr2=0x7f784c080610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.578+0000 7f7870647640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f784c03ae50 0x7f784c080610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7858009490 tx=0x7f7858005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:22.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.578+0000 7f7870647640 1 -- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78680cccf0 msgr2=0x7f78680c5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.578+0000 7f7870647640 1 --2- 172.21.15.81:0/300847313 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78680cccf0 0x7f78680c5490 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f786400c390 tx=0x7f786400c850 comp rx=0 tx=0).stop 2024-06-29T15:59:22.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.578+0000 7f786ebe4640 1 -- 172.21.15.81:0/300847313 reap_dead start 2024-06-29T15:59:22.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.578+0000 7f7870647640 1 -- 172.21.15.81:0/300847313 shutdown_connections 2024-06-29T15:59:22.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.578+0000 7f7870647640 1 -- 172.21.15.81:0/300847313 >> 172.21.15.81:0/300847313 conn(0x7f78680b3680 msgr2=0x7f786815b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:22.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.579+0000 7f7870647640 1 -- 172.21.15.81:0/300847313 shutdown_connections 2024-06-29T15:59:22.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.579+0000 7f7870647640 1 -- 172.21.15.81:0/300847313 wait complete. 2024-06-29T15:59:22.593 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2024-06-29T15:59:22.593 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2024-06-29T15:59:22.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.711+0000 7f48a2212640 1 -- 172.21.15.81:0/1717185698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f489c145830 msgr2=0x7f489c142460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.711+0000 7f48a2212640 1 --2- 172.21.15.81:0/1717185698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f489c145830 0x7f489c142460 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f488c009830 tx=0x7f488c02a480 comp rx=0 tx=0).stop 2024-06-29T15:59:22.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.712+0000 7f48a2212640 1 -- 172.21.15.81:0/1717185698 shutdown_connections 2024-06-29T15:59:22.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.712+0000 7f48a2212640 1 --2- 172.21.15.81:0/1717185698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f489c145830 0x7f489c142460 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.712+0000 7f48a2212640 1 -- 172.21.15.81:0/1717185698 >> 172.21.15.81:0/1717185698 conn(0x7f489c06cd60 msgr2=0x7f489c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:22.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.712+0000 7f48a2212640 1 -- 172.21.15.81:0/1717185698 shutdown_connections 2024-06-29T15:59:22.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.712+0000 7f48a2212640 1 -- 172.21.15.81:0/1717185698 wait complete. 2024-06-29T15:59:22.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.713+0000 7f48a2212640 1 Processor -- start 2024-06-29T15:59:22.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.713+0000 7f48a2212640 1 -- start start 2024-06-29T15:59:22.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f48a2212640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f489c0b2a30 0x7f489c1380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:22.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f48a2212640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f489c145830 0x7f489c0b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:22.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f48a2212640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f489c15fad0 0x7f489c0b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:22.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f48a2212640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f489c06f560 con 0x7f489c145830 2024-06-29T15:59:22.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f48a2212640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f489c06f3e0 con 0x7f489c15fad0 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f48a2212640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f489c06f6e0 con 0x7f489c0b2a30 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f489c145830 0x7f489c0b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f489c145830 0x7f489c0b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50290/0 (socket says 172.21.15.81:50290) 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489affd640 1 -- 172.21.15.81:0/2685522990 learned_addr learned my addr 172.21.15.81:0/2685522990 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489bfff640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f489c15fad0 0x7f489c0b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489b7fe640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f489c0b2a30 0x7f489c1380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489affd640 1 -- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f489c0b2a30 msgr2=0x7f489c1380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489affd640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f489c0b2a30 0x7f489c1380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489affd640 1 -- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f489c15fad0 msgr2=0x7f489c0b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489affd640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f489c15fad0 0x7f489c0b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489affd640 1 -- 172.21.15.81:0/2685522990 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f489c0aed90 con 0x7f489c145830 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.715+0000 7f489affd640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f489c145830 0x7f489c0b06d0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f489000c3a0 tx=0x7f489000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.716+0000 7f4883fff640 1 -- 172.21.15.81:0/2685522990 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4890017070 con 0x7f489c145830 2024-06-29T15:59:22.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.716+0000 7f4883fff640 1 -- 172.21.15.81:0/2685522990 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48900035f0 con 0x7f489c145830 2024-06-29T15:59:22.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.716+0000 7f48a2212640 1 -- 172.21.15.81:0/2685522990 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f489c15b710 con 0x7f489c145830 2024-06-29T15:59:22.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.716+0000 7f4883fff640 1 -- 172.21.15.81:0/2685522990 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4890003a40 con 0x7f489c145830 2024-06-29T15:59:22.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.716+0000 7f48a2212640 1 -- 172.21.15.81:0/2685522990 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f489c072a60 con 0x7f489c145830 2024-06-29T15:59:22.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.717+0000 7f48a2212640 1 -- 172.21.15.81:0/2685522990 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f489c06fc90 con 0x7f489c145830 2024-06-29T15:59:22.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.717+0000 7f4883fff640 1 -- 172.21.15.81:0/2685522990 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 149) ==== 98056+0+0 (secure 0 0 0) 0x7f489001a070 con 0x7f489c145830 2024-06-29T15:59:22.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.718+0000 7f4883fff640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4888076170 0x7f4888078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:22.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.718+0000 7f4883fff640 1 -- 172.21.15.81:0/2685522990 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f4890096880 con 0x7f489c145830 2024-06-29T15:59:22.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.718+0000 7f489b7fe640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4888076170 0x7f4888078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:22.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.718+0000 7f489b7fe640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4888076170 0x7f4888078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f488c034000 tx=0x7f488c004730 comp rx=0 tx=0).ready entity=mgr.11452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:22.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.722+0000 7f4883fff640 1 -- 172.21.15.81:0/2685522990 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f489005fc20 con 0x7f489c145830 2024-06-29T15:59:22.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.921+0000 7f48a2212640 1 -- 172.21.15.81:0/2685522990 --> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) -- 0x7f489c142010 con 0x7f4888076170 2024-06-29T15:59:22.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.978+0000 7f4883fff640 1 -- 172.21.15.81:0/2685522990 <== mgr.11452 v2:172.21.15.167:6832/1493691973 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+39 (secure 0 0 0) 0x7f489c142010 con 0x7f4888076170 2024-06-29T15:59:22.980 INFO:teuthology.orchestra.run.smithi081.stdout:Retention added to path snap_test_dir1 2024-06-29T15:59:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.980+0000 7f48a2212640 1 -- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4888076170 msgr2=0x7f4888078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.980+0000 7f48a2212640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4888076170 0x7f4888078680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f488c034000 tx=0x7f488c004730 comp rx=0 tx=0).stop 2024-06-29T15:59:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.980+0000 7f48a2212640 1 -- 172.21.15.81:0/2685522990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f489c145830 msgr2=0x7f489c0b06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.980+0000 7f48a2212640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f489c145830 0x7f489c0b06d0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f489000c3a0 tx=0x7f489000c860 comp rx=0 tx=0).stop 2024-06-29T15:59:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.980+0000 7f48a2212640 1 -- 172.21.15.81:0/2685522990 shutdown_connections 2024-06-29T15:59:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.980+0000 7f48a2212640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4888076170 0x7f4888078680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.980+0000 7f48a2212640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f489c15fad0 0x7f489c0b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.980+0000 7f48a2212640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f489c145830 0x7f489c0b06d0 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.980+0000 7f48a2212640 1 --2- 172.21.15.81:0/2685522990 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f489c0b2a30 0x7f489c1380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:22.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.980+0000 7f48a2212640 1 -- 172.21.15.81:0/2685522990 >> 172.21.15.81:0/2685522990 conn(0x7f489c06cd60 msgr2=0x7f489c0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:22.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.981+0000 7f48a2212640 1 -- 172.21.15.81:0/2685522990 shutdown_connections 2024-06-29T15:59:22.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:22.981+0000 7f48a2212640 1 -- 172.21.15.81:0/2685522990 wait complete. 2024-06-29T15:59:22.993 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2024-06-29T15:59:22.998 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2024-06-29-16_00_00 in ~38s... 2024-06-29T15:59:22.998 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2024-06-29T15:59:22.999 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2024-06-29T15:59:23.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.113+0000 7f35d34df640 1 -- 172.21.15.81:0/3461967898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35cc0b3640 msgr2=0x7f35cc15ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.113+0000 7f35d34df640 1 --2- 172.21.15.81:0/3461967898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35cc0b3640 0x7f35cc15ab20 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0b2790 tx=0x7f35c802a4a0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.113+0000 7f35d34df640 1 -- 172.21.15.81:0/3461967898 shutdown_connections 2024-06-29T15:59:23.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.113+0000 7f35d34df640 1 --2- 172.21.15.81:0/3461967898 >> v2:172.21.15.167:3300/0 conn(0x7f35cc15b060 0x7f35cc06f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.113+0000 7f35d34df640 1 --2- 172.21.15.81:0/3461967898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35cc0b3640 0x7f35cc15ab20 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.113+0000 7f35d34df640 1 -- 172.21.15.81:0/3461967898 >> 172.21.15.81:0/3461967898 conn(0x7f35cc06ccc0 msgr2=0x7f35cc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:23.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.113+0000 7f35d34df640 1 -- 172.21.15.81:0/3461967898 shutdown_connections 2024-06-29T15:59:23.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.113+0000 7f35d34df640 1 -- 172.21.15.81:0/3461967898 wait complete. 2024-06-29T15:59:23.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.115+0000 7f35d34df640 1 Processor -- start 2024-06-29T15:59:23.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.115+0000 7f35d34df640 1 -- start start 2024-06-29T15:59:23.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d34df640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35cc06fba0 0x7f35cc1e5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:23.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d34df640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35cc0b3640 0x7f35cc14fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d34df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35cc15b060 0x7f35cc156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d34df640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0cf740 con 0x7f35cc15b060 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d34df640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0cf5c0 con 0x7f35cc0b3640 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d34df640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f35cc072630 con 0x7f35cc06fba0 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d1a7c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35cc15b060 0x7f35cc156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d1a7c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35cc15b060 0x7f35cc156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50294/0 (socket says 172.21.15.81:50294) 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d1a7c640 1 -- 172.21.15.81:0/1948770060 learned_addr learned my addr 172.21.15.81:0/1948770060 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d0a7a640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35cc0b3640 0x7f35cc14fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d127b640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35cc06fba0 0x7f35cc1e5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d1a7c640 1 -- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35cc06fba0 msgr2=0x7f35cc1e5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d1a7c640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35cc06fba0 0x7f35cc1e5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d1a7c640 1 -- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35cc0b3640 msgr2=0x7f35cc14fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d1a7c640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35cc0b3640 0x7f35cc14fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.116+0000 7f35d1a7c640 1 -- 172.21.15.81:0/1948770060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc1509d0 con 0x7f35cc15b060 2024-06-29T15:59:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.117+0000 7f35d1a7c640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35cc15b060 0x7f35cc156090 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f35c400cae0 tx=0x7f35c400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.117+0000 7f35c1ffb640 1 -- 172.21.15.81:0/1948770060 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c4016020 con 0x7f35cc15b060 2024-06-29T15:59:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.117+0000 7f35c1ffb640 1 -- 172.21.15.81:0/1948770060 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f35c400ad30 con 0x7f35cc15b060 2024-06-29T15:59:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.117+0000 7f35d34df640 1 -- 172.21.15.81:0/1948770060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35cc1e6160 con 0x7f35cc15b060 2024-06-29T15:59:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.117+0000 7f35c1ffb640 1 -- 172.21.15.81:0/1948770060 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c4012680 con 0x7f35cc15b060 2024-06-29T15:59:23.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.117+0000 7f35d34df640 1 -- 172.21.15.81:0/1948770060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35cc1555b0 con 0x7f35cc15b060 2024-06-29T15:59:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.118+0000 7f35d34df640 1 -- 172.21.15.81:0/1948770060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35cc072b90 con 0x7f35cc15b060 2024-06-29T15:59:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.119+0000 7f35c1ffb640 1 -- 172.21.15.81:0/1948770060 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 149) ==== 98056+0+0 (secure 0 0 0) 0x7f35c4019020 con 0x7f35cc15b060 2024-06-29T15:59:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.119+0000 7f35c1ffb640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f35b00761a0 0x7f35b00786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.119+0000 7f35c1ffb640 1 -- 172.21.15.81:0/1948770060 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f35c4097820 con 0x7f35cc15b060 2024-06-29T15:59:23.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.119+0000 7f35d127b640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f35b00761a0 0x7f35b00786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:23.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.120+0000 7f35d127b640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f35b00761a0 0x7f35b00786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f35c8034000 tx=0x7f35c8004730 comp rx=0 tx=0).ready entity=mgr.11452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:23.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.123+0000 7f35c1ffb640 1 -- 172.21.15.81:0/1948770060 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f35c4020030 con 0x7f35cc15b060 2024-06-29T15:59:23.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.324+0000 7f35d34df640 1 -- 172.21.15.81:0/1948770060 --> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f35cc15a620 con 0x7f35b00761a0 2024-06-29T15:59:23.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.331+0000 7f35c1ffb640 1 -- 172.21.15.81:0/1948770060 <== mgr.11452 v2:172.21.15.167:6832/1493691973 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+72 (secure 0 0 0) 0x7f35cc15a620 con 0x7f35b00761a0 2024-06-29T15:59:23.332 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:23.332 INFO:teuthology.orchestra.run.smithi081.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-06-29T15:59:23.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 -- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f35b00761a0 msgr2=0x7f35b00786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f35b00761a0 0x7f35b00786b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f35c8034000 tx=0x7f35c8004730 comp rx=0 tx=0).stop 2024-06-29T15:59:23.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 -- 172.21.15.81:0/1948770060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35cc15b060 msgr2=0x7f35cc156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35cc15b060 0x7f35cc156090 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f35c400cae0 tx=0x7f35c400a060 comp rx=0 tx=0).stop 2024-06-29T15:59:23.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 -- 172.21.15.81:0/1948770060 shutdown_connections 2024-06-29T15:59:23.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f35b00761a0 0x7f35b00786b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35cc15b060 0x7f35cc156090 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f35cc0b3640 0x7f35cc14fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 --2- 172.21.15.81:0/1948770060 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f35cc06fba0 0x7f35cc1e5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 -- 172.21.15.81:0/1948770060 >> 172.21.15.81:0/1948770060 conn(0x7f35cc06ccc0 msgr2=0x7f35cc06ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:23.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 -- 172.21.15.81:0/1948770060 shutdown_connections 2024-06-29T15:59:23.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.333+0000 7f35d34df640 1 -- 172.21.15.81:0/1948770060 wait complete. 2024-06-29T15:59:23.346 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2024-06-29T15:59:23.346 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2024-06-29T15:59:23.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2024-06-29T15:59:23.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.460+0000 7f3a978ce640 1 -- 172.21.15.81:0/1341431713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a900d0b90 msgr2=0x7f3a900d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.460+0000 7f3a978ce640 1 --2- 172.21.15.81:0/1341431713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a900d0b90 0x7f3a900d1a80 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009830 tx=0x7f3a8002a5a0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.460+0000 7f3a978ce640 1 -- 172.21.15.81:0/1341431713 shutdown_connections 2024-06-29T15:59:23.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.460+0000 7f3a978ce640 1 --2- 172.21.15.81:0/1341431713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a900d0b90 0x7f3a900d1a80 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.460+0000 7f3a978ce640 1 -- 172.21.15.81:0/1341431713 >> 172.21.15.81:0/1341431713 conn(0x7f3a9009cf30 msgr2=0x7f3a900d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:23.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.460+0000 7f3a978ce640 1 -- 172.21.15.81:0/1341431713 shutdown_connections 2024-06-29T15:59:23.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.460+0000 7f3a978ce640 1 -- 172.21.15.81:0/1341431713 wait complete. 2024-06-29T15:59:23.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.462+0000 7f3a978ce640 1 Processor -- start 2024-06-29T15:59:23.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.462+0000 7f3a978ce640 1 -- start start 2024-06-29T15:59:23.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.463+0000 7f3a978ce640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a900b5250 0x7f3a9015cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:23.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.463+0000 7f3a978ce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a900b9040 0x7f3a900a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:23.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.463+0000 7f3a978ce640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a900a0760 0x7f3a90150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:23.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.463+0000 7f3a978ce640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3a900d26d0 con 0x7f3a900b9040 2024-06-29T15:59:23.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.463+0000 7f3a978ce640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3a900b85c0 con 0x7f3a900b5250 2024-06-29T15:59:23.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.463+0000 7f3a978ce640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3a900d2850 con 0x7f3a900a0760 2024-06-29T15:59:23.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.463+0000 7f3a94e69640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a900b9040 0x7f3a900a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:23.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a94e69640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a900b9040 0x7f3a900a0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50324/0 (socket says 172.21.15.81:50324) 2024-06-29T15:59:23.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a94e69640 1 -- 172.21.15.81:0/533107795 learned_addr learned my addr 172.21.15.81:0/533107795 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a95e6b640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a900a0760 0x7f3a90150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a9566a640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a900b5250 0x7f3a9015cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a94e69640 1 -- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a900a0760 msgr2=0x7f3a90150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a94e69640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a900a0760 0x7f3a90150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a94e69640 1 -- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a900b5250 msgr2=0x7f3a9015cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a94e69640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a900b5250 0x7f3a9015cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a94e69640 1 -- 172.21.15.81:0/533107795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a900a0b70 con 0x7f3a900b9040 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a9566a640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a900b5250 0x7f3a9015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a94e69640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a900b9040 0x7f3a900a0220 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f3a8002ab90 tx=0x7f3a80033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a85ffb640 1 -- 172.21.15.81:0/533107795 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a80033e10 con 0x7f3a900b9040 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a85ffb640 1 -- 172.21.15.81:0/533107795 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a80031c50 con 0x7f3a900b9040 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a85ffb640 1 -- 172.21.15.81:0/533107795 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a80005280 con 0x7f3a900b9040 2024-06-29T15:59:23.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a978ce640 1 -- 172.21.15.81:0/533107795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a900b1230 con 0x7f3a900b9040 2024-06-29T15:59:23.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.464+0000 7f3a978ce640 1 -- 172.21.15.81:0/533107795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9014ec40 con 0x7f3a900b9040 2024-06-29T15:59:23.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.466+0000 7f3a978ce640 1 -- 172.21.15.81:0/533107795 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a900b1750 con 0x7f3a900b9040 2024-06-29T15:59:23.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.466+0000 7f3a85ffb640 1 -- 172.21.15.81:0/533107795 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 149) ==== 98056+0+0 (secure 0 0 0) 0x7f3a80045020 con 0x7f3a900b9040 2024-06-29T15:59:23.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.466+0000 7f3a85ffb640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f3a74076170 0x7f3a74078680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:23.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.467+0000 7f3a85ffb640 1 -- 172.21.15.81:0/533107795 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f3a800b7fc0 con 0x7f3a900b9040 2024-06-29T15:59:23.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.467+0000 7f3a9566a640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f3a74076170 0x7f3a74078680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:23.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.467+0000 7f3a9566a640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f3a74076170 0x7f3a74078680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c000f60 tx=0x7f3a8c009040 comp rx=0 tx=0).ready entity=mgr.11452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:23.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.471+0000 7f3a85ffb640 1 -- 172.21.15.81:0/533107795 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3a900b1750 con 0x7f3a900b9040 2024-06-29T15:59:23.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.512+0000 7f3a85ffb640 1 -- 172.21.15.81:0/533107795 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 150) ==== 98321+0+0 (secure 0 0 0) 0x7f3a8007d530 con 0x7f3a900b9040 2024-06-29T15:59:23.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.681+0000 7f3a978ce640 1 -- 172.21.15.81:0/533107795 --> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) -- 0x7f3a90154e60 con 0x7f3a74076170 2024-06-29T15:59:23.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.746+0000 7f3a85ffb640 1 -- 172.21.15.81:0/533107795 <== mgr.11452 v2:172.21.15.167:6832/1493691973 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f3a8c001cb0 con 0x7f3a74076170 2024-06-29T15:59:23.747 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path snap_test_dir1 2024-06-29T15:59:23.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 -- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f3a74076170 msgr2=0x7f3a74078680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f3a74076170 0x7f3a74078680 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c000f60 tx=0x7f3a8c009040 comp rx=0 tx=0).stop 2024-06-29T15:59:23.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 -- 172.21.15.81:0/533107795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a900b9040 msgr2=0x7f3a900a0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a900b9040 0x7f3a900a0220 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f3a8002ab90 tx=0x7f3a80033c20 comp rx=0 tx=0).stop 2024-06-29T15:59:23.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 -- 172.21.15.81:0/533107795 shutdown_connections 2024-06-29T15:59:23.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f3a74076170 0x7f3a74078680 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3a900a0760 0x7f3a90150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a900b9040 0x7f3a900a0220 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 --2- 172.21.15.81:0/533107795 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3a900b5250 0x7f3a9015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 -- 172.21.15.81:0/533107795 >> 172.21.15.81:0/533107795 conn(0x7f3a9009cf30 msgr2=0x7f3a90068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:23.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 -- 172.21.15.81:0/533107795 shutdown_connections 2024-06-29T15:59:23.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.748+0000 7f3a978ce640 1 -- 172.21.15.81:0/533107795 wait complete. 2024-06-29T15:59:23.761 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2024-06-29T15:59:23.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:23.761 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2024-06-29T15:59:23.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:23.778 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2024-06-29T15:59:23.836 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T15:59:23.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.993+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/2174550069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d840b5f50 msgr2=0x7f4d84166870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:23.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.993+0000 7f4d8c4a5640 1 --2- 172.21.15.81:0/2174550069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d840b5f50 0x7f4d84166870 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c005fd0 tx=0x7f4d7c02a8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.993+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/2174550069 shutdown_connections 2024-06-29T15:59:23.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.993+0000 7f4d8c4a5640 1 --2- 172.21.15.81:0/2174550069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d840b5f50 0x7f4d84166870 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:23.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.993+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/2174550069 >> 172.21.15.81:0/2174550069 conn(0x7f4d840b3680 msgr2=0x7f4d840b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:23.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.993+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/2174550069 shutdown_connections 2024-06-29T15:59:23.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.993+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/2174550069 wait complete. 2024-06-29T15:59:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.995+0000 7f4d8c4a5640 1 Processor -- start 2024-06-29T15:59:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.995+0000 7f4d8c4a5640 1 -- start start 2024-06-29T15:59:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.996+0000 7f4d8c4a5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d840b3200 0x7f4d8414e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:24.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.996+0000 7f4d8c4a5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d840b5f50 0x7f4d84150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:24.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.996+0000 7f4d8c4a5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d840d0230 0x7f4d841509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:24.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8c4a5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4d841672d0 con 0x7f4d840d0230 2024-06-29T15:59:24.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8c4a5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4d84167150 con 0x7f4d840b3200 2024-06-29T15:59:24.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8c4a5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4d84167450 con 0x7f4d840b5f50 2024-06-29T15:59:24.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8aa42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d840d0230 0x7f4d841509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:24.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8aa42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d840d0230 0x7f4d841509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50332/0 (socket says 172.21.15.81:50332) 2024-06-29T15:59:24.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8aa42640 1 -- 172.21.15.81:0/390620633 learned_addr learned my addr 172.21.15.81:0/390620633 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:24.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8a241640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d840b3200 0x7f4d8414e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:24.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d89a40640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d840b5f50 0x7f4d84150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:24.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8aa42640 1 -- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d840b5f50 msgr2=0x7f4d84150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:24.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8aa42640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d840b5f50 0x7f4d84150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:24.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8aa42640 1 -- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d840b3200 msgr2=0x7f4d8414e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:24.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8aa42640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d840b3200 0x7f4d8414e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:24.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8aa42640 1 -- 172.21.15.81:0/390620633 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d840c89c0 con 0x7f4d840d0230 2024-06-29T15:59:24.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8aa42640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d840d0230 0x7f4d841509a0 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009cd0 tx=0x7f4d7c003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:24.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d7affd640 1 -- 172.21.15.81:0/390620633 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c031b90 con 0x7f4d840d0230 2024-06-29T15:59:24.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d7affd640 1 -- 172.21.15.81:0/390620633 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d7c031e20 con 0x7f4d840d0230 2024-06-29T15:59:24.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/390620633 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d840c8400 con 0x7f4d840d0230 2024-06-29T15:59:24.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d7affd640 1 -- 172.21.15.81:0/390620633 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c033290 con 0x7f4d840d0230 2024-06-29T15:59:24.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.997+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/390620633 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d8414ba20 con 0x7f4d840d0230 2024-06-29T15:59:24.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.998+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/390620633 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d840cc7b0 con 0x7f4d840d0230 2024-06-29T15:59:24.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.999+0000 7f4d7affd640 1 -- 172.21.15.81:0/390620633 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 150) ==== 98321+0+0 (secure 0 0 0) 0x7f4d7c033430 con 0x7f4d840d0230 2024-06-29T15:59:24.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:23.999+0000 7f4d7affd640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4d68076440 0x7f4d68078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:24.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.000+0000 7f4d7affd640 1 -- 172.21.15.81:0/390620633 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f4d7c0b9150 con 0x7f4d840d0230 2024-06-29T15:59:24.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.000+0000 7f4d8a241640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4d68076440 0x7f4d68078950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:24.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.000+0000 7f4d8a241640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4d68076440 0x7f4d68078950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4d80000f60 tx=0x7f4d80009040 comp rx=0 tx=0).ready entity=mgr.11452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:24.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.004+0000 7f4d7affd640 1 -- 172.21.15.81:0/390620633 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4d7c03f030 con 0x7f4d840d0230 2024-06-29T15:59:24.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.233+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/390620633 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f4d8414e620 con 0x7f4d840d0230 2024-06-29T15:59:24.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.516+0000 7f4d7affd640 1 -- 172.21.15.81:0/390620633 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v151) ==== 93+0+0 (secure 0 0 0) 0x7f4d7c081f80 con 0x7f4d840d0230 2024-06-29T15:59:24.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.519+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4d68076440 msgr2=0x7f4d68078950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:24.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.519+0000 7f4d8c4a5640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4d68076440 0x7f4d68078950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4d80000f60 tx=0x7f4d80009040 comp rx=0 tx=0).stop 2024-06-29T15:59:24.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.519+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/390620633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d840d0230 msgr2=0x7f4d841509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:24.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.520+0000 7f4d8c4a5640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d840d0230 0x7f4d841509a0 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009cd0 tx=0x7f4d7c003a10 comp rx=0 tx=0).stop 2024-06-29T15:59:24.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.520+0000 7f4d7affd640 1 -- 172.21.15.81:0/390620633 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f4d7c0b7440 con 0x7f4d840d0230 2024-06-29T15:59:24.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.520+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/390620633 shutdown_connections 2024-06-29T15:59:24.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.520+0000 7f4d8c4a5640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f4d68076440 0x7f4d68078950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:24.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.520+0000 7f4d8c4a5640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d840d0230 0x7f4d841509a0 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:24.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.520+0000 7f4d8c4a5640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4d840b5f50 0x7f4d84150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:24.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.520+0000 7f4d8c4a5640 1 --2- 172.21.15.81:0/390620633 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4d840b3200 0x7f4d8414e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:24.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.520+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/390620633 >> 172.21.15.81:0/390620633 conn(0x7f4d840b3680 msgr2=0x7f4d840cf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:24.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.520+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/390620633 shutdown_connections 2024-06-29T15:59:24.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.520+0000 7f4d8c4a5640 1 -- 172.21.15.81:0/390620633 wait complete. 2024-06-29T15:59:24.534 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:59:24.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:24.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:24.628 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:59:24.629 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:59:24.629 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:59:24.629 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:24.646 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:59:24.646 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:24.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:24.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.864+0000 7f663a95b640 1 -- 172.21.15.81:0/1072909376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66340cd150 msgr2=0x7f6634072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.864+0000 7f663a95b640 1 --2- 172.21.15.81:0/1072909376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66340cd150 0x7f6634072f40 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f6624005fd0 tx=0x7f662402a540 comp rx=0 tx=0).stop 2024-06-29T15:59:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.864+0000 7f663a95b640 1 -- 172.21.15.81:0/1072909376 shutdown_connections 2024-06-29T15:59:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.864+0000 7f663a95b640 1 --2- 172.21.15.81:0/1072909376 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f663415bdf0 0x7f663406ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.864+0000 7f663a95b640 1 --2- 172.21.15.81:0/1072909376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66340cd150 0x7f6634072f40 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.864+0000 7f663a95b640 1 -- 172.21.15.81:0/1072909376 >> 172.21.15.81:0/1072909376 conn(0x7f66340b2920 msgr2=0x7f66340b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.864+0000 7f663a95b640 1 -- 172.21.15.81:0/1072909376 shutdown_connections 2024-06-29T15:59:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.864+0000 7f663a95b640 1 -- 172.21.15.81:0/1072909376 wait complete. 2024-06-29T15:59:24.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.866+0000 7f663a95b640 1 Processor -- start 2024-06-29T15:59:24.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.866+0000 7f663a95b640 1 -- start start 2024-06-29T15:59:24.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f663a95b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6634073480 0x7f6634152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f663a95b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66340cd150 0x7f6634146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f663a95b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f663415bdf0 0x7f66341f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f663a95b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f663406f660 con 0x7f66340cd150 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f663a95b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f663406f4e0 con 0x7f6634073480 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f663a95b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f663406f7e0 con 0x7f663415bdf0 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f66337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66340cd150 0x7f6634146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f66337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66340cd150 0x7f6634146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50370/0 (socket says 172.21.15.81:50370) 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f66337fe640 1 -- 172.21.15.81:0/664330547 learned_addr learned my addr 172.21.15.81:0/664330547 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f6633fff640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6634073480 0x7f6634152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.867+0000 7f6638ef8640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f663415bdf0 0x7f66341f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f66337fe640 1 -- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f663415bdf0 msgr2=0x7f66341f30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:24.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f66337fe640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f663415bdf0 0x7f66341f30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f66337fe640 1 -- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6634073480 msgr2=0x7f6634152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f66337fe640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6634073480 0x7f6634152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f66337fe640 1 -- 172.21.15.81:0/664330547 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66340ce5a0 con 0x7f66340cd150 2024-06-29T15:59:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f6633fff640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6634073480 0x7f6634152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f66337fe640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66340cd150 0x7f6634146fe0 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f662800c3a0 tx=0x7f662800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f6630ff9640 1 -- 172.21.15.81:0/664330547 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628017070 con 0x7f66340cd150 2024-06-29T15:59:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f6630ff9640 1 -- 172.21.15.81:0/664330547 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66280053c0 con 0x7f66340cd150 2024-06-29T15:59:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f663a95b640 1 -- 172.21.15.81:0/664330547 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66340ce880 con 0x7f66340cd150 2024-06-29T15:59:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f6630ff9640 1 -- 172.21.15.81:0/664330547 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628003850 con 0x7f66340cd150 2024-06-29T15:59:24.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.868+0000 7f663a95b640 1 -- 172.21.15.81:0/664330547 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66340cee40 con 0x7f66340cd150 2024-06-29T15:59:24.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.869+0000 7f663a95b640 1 -- 172.21.15.81:0/664330547 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66340ce300 con 0x7f66340cd150 2024-06-29T15:59:24.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.870+0000 7f6630ff9640 1 -- 172.21.15.81:0/664330547 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f662801a070 con 0x7f66340cd150 2024-06-29T15:59:24.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.870+0000 7f6630ff9640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f66200763f0 0x7f6620078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:24.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.871+0000 7f6633fff640 1 -- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f66200763f0 msgr2=0x7f6620078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:24.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.871+0000 7f6630ff9640 1 -- 172.21.15.81:0/664330547 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f6628096f30 con 0x7f66340cd150 2024-06-29T15:59:24.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.871+0000 7f6633fff640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f66200763f0 0x7f6620078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:24.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:24.875+0000 7f6630ff9640 1 -- 172.21.15.81:0/664330547 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f66340ce300 con 0x7f66340cd150 2024-06-29T15:59:25.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.063+0000 7f663a95b640 1 -- 172.21.15.81:0/664330547 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f663406fd70 con 0x7f66340cd150 2024-06-29T15:59:25.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.064+0000 7f6630ff9640 1 -- 172.21.15.81:0/664330547 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) ==== 74+0+17454 (secure 0 0 0) 0x7f6634073890 con 0x7f66340cd150 2024-06-29T15:59:25.066 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:25.066 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":254,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:21.470300+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:07.416791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:08.088694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3251388883":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2508797860":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/3181687202":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1949624986":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2055038349":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1927715461":"2024-06-30T15:59:21.470262+0000","172.21.15.167:6832/1778875525":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2270885245":"2024-06-30T15:59:21.470262+0000","172.21.15.167:6833/1778875525":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1180941913":"2024-06-30T15:59:21.470262+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 -- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f66200763f0 msgr2=0x7f6620078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f66200763f0 0x7f6620078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 -- 172.21.15.81:0/664330547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66340cd150 msgr2=0x7f6634146fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66340cd150 0x7f6634146fe0 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f662800c3a0 tx=0x7f662800c860 comp rx=0 tx=0).stop 2024-06-29T15:59:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 -- 172.21.15.81:0/664330547 shutdown_connections 2024-06-29T15:59:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f66200763f0 0x7f6620078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f663415bdf0 0x7f66341f30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66340cd150 0x7f6634146fe0 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 --2- 172.21.15.81:0/664330547 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6634073480 0x7f6634152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 -- 172.21.15.81:0/664330547 >> 172.21.15.81:0/664330547 conn(0x7f66340b2920 msgr2=0x7f663415e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:25.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 -- 172.21.15.81:0/664330547 shutdown_connections 2024-06-29T15:59:25.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.067+0000 7f663a95b640 1 -- 172.21.15.81:0/664330547 wait complete. 2024-06-29T15:59:25.080 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:59:07.416791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:59:08.088694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:25.081 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:25.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.198+0000 7fdbbc143640 1 -- 172.21.15.81:0/1453026151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb4073650 msgr2=0x7fdbb415b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.198+0000 7fdbbc143640 1 --2- 172.21.15.81:0/1453026151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb4073650 0x7fdbb415b6e0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7fdba4009830 tx=0x7fdba402a530 comp rx=0 tx=0).stop 2024-06-29T15:59:25.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.198+0000 7fdbbc143640 1 -- 172.21.15.81:0/1453026151 shutdown_connections 2024-06-29T15:59:25.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.198+0000 7fdbbc143640 1 --2- 172.21.15.81:0/1453026151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb4073650 0x7fdbb415b6e0 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.198+0000 7fdbbc143640 1 -- 172.21.15.81:0/1453026151 >> 172.21.15.81:0/1453026151 conn(0x7fdbb40b2920 msgr2=0x7fdbb40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:25.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.198+0000 7fdbbc143640 1 -- 172.21.15.81:0/1453026151 shutdown_connections 2024-06-29T15:59:25.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.198+0000 7fdbbc143640 1 -- 172.21.15.81:0/1453026151 wait complete. 2024-06-29T15:59:25.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.200+0000 7fdbbc143640 1 Processor -- start 2024-06-29T15:59:25.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.200+0000 7fdbbc143640 1 -- start start 2024-06-29T15:59:25.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.201+0000 7fdbbc143640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbb40cd150 0x7fdbb41f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.201+0000 7fdbbc143640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb415bc20 0x7fdbb4159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.201+0000 7fdbbc143640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb4159e90 0x7fdbb406c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbbc143640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb406f830 con 0x7fdbb4159e90 2024-06-29T15:59:25.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbbc143640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb406f6b0 con 0x7fdbb415bc20 2024-06-29T15:59:25.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbbc143640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdbb406f9b0 con 0x7fdbb40cd150 2024-06-29T15:59:25.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbba6e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb4159e90 0x7fdbb406c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:25.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbba6e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb4159e90 0x7fdbb406c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50378/0 (socket says 172.21.15.81:50378) 2024-06-29T15:59:25.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbba6e0640 1 -- 172.21.15.81:0/3591678133 learned_addr learned my addr 172.21.15.81:0/3591678133 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbb9edf640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbb40cd150 0x7fdbb41f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbb96de640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb415bc20 0x7fdbb4159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbba6e0640 1 -- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbb40cd150 msgr2=0x7fdbb41f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbba6e0640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbb40cd150 0x7fdbb41f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbba6e0640 1 -- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb415bc20 msgr2=0x7fdbb4159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbba6e0640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb415bc20 0x7fdbb4159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbba6e0640 1 -- 172.21.15.81:0/3591678133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb415a610 con 0x7fdbb4159e90 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbba6e0640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb4159e90 0x7fdbb406c2f0 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00c700 tx=0x7fdbac00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbaa7fc640 1 -- 172.21.15.81:0/3591678133 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac003b70 con 0x7fdbb4159e90 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbaa7fc640 1 -- 172.21.15.81:0/3591678133 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdbac003d10 con 0x7fdbb4159e90 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbbc143640 1 -- 172.21.15.81:0/3591678133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb415a8f0 con 0x7fdbb4159e90 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbaa7fc640 1 -- 172.21.15.81:0/3591678133 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00aab0 con 0x7fdbb4159e90 2024-06-29T15:59:25.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.202+0000 7fdbbc143640 1 -- 172.21.15.81:0/3591678133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb415aeb0 con 0x7fdbb4159e90 2024-06-29T15:59:25.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.203+0000 7fdbbc143640 1 -- 172.21.15.81:0/3591678133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbb415b6e0 con 0x7fdbb4159e90 2024-06-29T15:59:25.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.204+0000 7fdbaa7fc640 1 -- 172.21.15.81:0/3591678133 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7fdbac00ac50 con 0x7fdbb4159e90 2024-06-29T15:59:25.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.205+0000 7fdbaa7fc640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fdb980763f0 0x7fdb98078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.205+0000 7fdbb9edf640 1 -- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fdb980763f0 msgr2=0x7fdb98078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:25.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.205+0000 7fdbaa7fc640 1 -- 172.21.15.81:0/3591678133 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7fdbac097810 con 0x7fdbb4159e90 2024-06-29T15:59:25.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.205+0000 7fdbb9edf640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fdb980763f0 0x7fdb98078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:25.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.209+0000 7fdbaa7fc640 1 -- 172.21.15.81:0/3591678133 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdbac0607e0 con 0x7fdbb4159e90 2024-06-29T15:59:25.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.392+0000 7fdbbc143640 1 -- 172.21.15.81:0/3591678133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdbb415a330 con 0x7fdbb4159e90 2024-06-29T15:59:25.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.393+0000 7fdbaa7fc640 1 -- 172.21.15.81:0/3591678133 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v254) ==== 93+0+10 (secure 0 0 0) 0x7fdbac064ae0 con 0x7fdbb4159e90 2024-06-29T15:59:25.395 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:59:25.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 -- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fdb980763f0 msgr2=0x7fdb98078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:25.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fdb980763f0 0x7fdb98078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 -- 172.21.15.81:0/3591678133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb4159e90 msgr2=0x7fdbb406c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb4159e90 0x7fdbb406c2f0 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00c700 tx=0x7fdbac00cbc0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 -- 172.21.15.81:0/3591678133 shutdown_connections 2024-06-29T15:59:25.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fdb980763f0 0x7fdb98078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb4159e90 0x7fdbb406c2f0 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdbb415bc20 0x7fdbb4159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 --2- 172.21.15.81:0/3591678133 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdbb40cd150 0x7fdbb41f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 -- 172.21.15.81:0/3591678133 >> 172.21.15.81:0/3591678133 conn(0x7fdbb40b2920 msgr2=0x7fdbb40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:25.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 -- 172.21.15.81:0/3591678133 shutdown_connections 2024-06-29T15:59:25.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.395+0000 7fdbbc143640 1 -- 172.21.15.81:0/3591678133 wait complete. 2024-06-29T15:59:25.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:59:25.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.527+0000 7fcbc1488640 1 -- 172.21.15.81:0/1833149358 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbbc15bb10 msgr2=0x7fcbbc144590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.527+0000 7fcbc1488640 1 --2- 172.21.15.81:0/1833149358 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbbc15bb10 0x7fcbbc144590 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009be0 tx=0x7fcbb402a920 comp rx=0 tx=0).stop 2024-06-29T15:59:25.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.527+0000 7fcbc1488640 1 -- 172.21.15.81:0/1833149358 shutdown_connections 2024-06-29T15:59:25.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.527+0000 7fcbc1488640 1 --2- 172.21.15.81:0/1833149358 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbbc15bb10 0x7fcbbc144590 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.527+0000 7fcbc1488640 1 -- 172.21.15.81:0/1833149358 >> 172.21.15.81:0/1833149358 conn(0x7fcbbc0b2920 msgr2=0x7fcbbc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:25.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.527+0000 7fcbc1488640 1 -- 172.21.15.81:0/1833149358 shutdown_connections 2024-06-29T15:59:25.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.527+0000 7fcbc1488640 1 -- 172.21.15.81:0/1833149358 wait complete. 2024-06-29T15:59:25.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.529+0000 7fcbc1488640 1 Processor -- start 2024-06-29T15:59:25.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.529+0000 7fcbc1488640 1 -- start start 2024-06-29T15:59:25.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.530+0000 7fcbc1488640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbbc0b4800 0x7fcbbc15e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.530+0000 7fcbc1488640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbbc15bb10 0x7fcbbc158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.530+0000 7fcbc1488640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbbc1f5fe0 0x7fcbbc13bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.530+0000 7fcbc1488640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc06f480 con 0x7fcbbc0b4800 2024-06-29T15:59:25.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.530+0000 7fcbc1488640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc06f300 con 0x7fcbbc15bb10 2024-06-29T15:59:25.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.530+0000 7fcbc1488640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fcbbc06f600 con 0x7fcbbc1f5fe0 2024-06-29T15:59:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbbaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbbc0b4800 0x7fcbbc15e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbbaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbbc0b4800 0x7fcbbc15e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50390/0 (socket says 172.21.15.81:50390) 2024-06-29T15:59:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbbaffd640 1 -- 172.21.15.81:0/4185721536 learned_addr learned my addr 172.21.15.81:0/4185721536 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbba7fc640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbbc15bb10 0x7fcbbc158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbbb7fe640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbbc1f5fe0 0x7fcbbc13bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbba7fc640 1 -- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbbc1f5fe0 msgr2=0x7fcbbc13bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbba7fc640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbbc1f5fe0 0x7fcbbc13bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbba7fc640 1 -- 172.21.15.81:0/4185721536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbbc0b4800 msgr2=0x7fcbbc15e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbba7fc640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbbc0b4800 0x7fcbbc15e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbba7fc640 1 -- 172.21.15.81:0/4185721536 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbbc1594e0 con 0x7fcbbc15bb10 2024-06-29T15:59:25.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbbaffd640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbbc0b4800 0x7fcbbc15e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:25.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbba7fc640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbbc15bb10 0x7fcbbc158fa0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000c3f0 tx=0x7fcbb000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:25.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcba37fe640 1 -- 172.21.15.81:0/4185721536 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0017070 con 0x7fcbbc15bb10 2024-06-29T15:59:25.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcba37fe640 1 -- 172.21.15.81:0/4185721536 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbb0005400 con 0x7fcbbc15bb10 2024-06-29T15:59:25.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbc1488640 1 -- 172.21.15.81:0/4185721536 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbbc1f6750 con 0x7fcbbc15bb10 2024-06-29T15:59:25.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcba37fe640 1 -- 172.21.15.81:0/4185721536 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0003850 con 0x7fcbbc15bb10 2024-06-29T15:59:25.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.531+0000 7fcbc1488640 1 -- 172.21.15.81:0/4185721536 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbbc1f6d10 con 0x7fcbbc15bb10 2024-06-29T15:59:25.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.532+0000 7fcbc1488640 1 -- 172.21.15.81:0/4185721536 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbbc06fab0 con 0x7fcbbc15bb10 2024-06-29T15:59:25.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.533+0000 7fcba37fe640 1 -- 172.21.15.81:0/4185721536 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7fcbb001a070 con 0x7fcbbc15bb10 2024-06-29T15:59:25.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.533+0000 7fcba37fe640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fcba80763f0 0x7fcba8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.534+0000 7fcbbaffd640 1 -- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fcba80763f0 msgr2=0x7fcba8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:25.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.534+0000 7fcba37fe640 1 -- 172.21.15.81:0/4185721536 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7fcbb0096d30 con 0x7fcbbc15bb10 2024-06-29T15:59:25.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.534+0000 7fcbbaffd640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fcba80763f0 0x7fcba8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:25.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.538+0000 7fcba37fe640 1 -- 172.21.15.81:0/4185721536 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fcbb005fd10 con 0x7fcbbc15bb10 2024-06-29T15:59:25.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.727+0000 7fcbc1488640 1 -- 172.21.15.81:0/4185721536 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcbbc1f6470 con 0x7fcbbc15bb10 2024-06-29T15:59:25.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.728+0000 7fcba37fe640 1 -- 172.21.15.81:0/4185721536 <== mon.1 v2:172.21.15.167:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v254) ==== 104+0+11 (secure 0 0 0) 0x7fcbb0064010 con 0x7fcbbc15bb10 2024-06-29T15:59:25.729 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:59:25.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 -- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fcba80763f0 msgr2=0x7fcba8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:25.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fcba80763f0 0x7fcba8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 -- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbbc15bb10 msgr2=0x7fcbbc158fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbbc15bb10 0x7fcbbc158fa0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000c3f0 tx=0x7fcbb000c8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 -- 172.21.15.81:0/4185721536 shutdown_connections 2024-06-29T15:59:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fcba80763f0 0x7fcba8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fcbbc1f5fe0 0x7fcbbc13bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fcbbc15bb10 0x7fcbbc158fa0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 --2- 172.21.15.81:0/4185721536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbbc0b4800 0x7fcbbc15e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 -- 172.21.15.81:0/4185721536 >> 172.21.15.81:0/4185721536 conn(0x7fcbbc0b2920 msgr2=0x7fcbbc070630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:25.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 -- 172.21.15.81:0/4185721536 shutdown_connections 2024-06-29T15:59:25.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.730+0000 7fcbc1488640 1 -- 172.21.15.81:0/4185721536 wait complete. 2024-06-29T15:59:25.744 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:59:25.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.864+0000 7f44610c0640 1 -- 172.21.15.81:0/4195354451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0b3200 msgr2=0x7f445c0cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.864+0000 7f44610c0640 1 --2- 172.21.15.81:0/4195354451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0b3200 0x7f445c0cc7b0 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f4444009830 tx=0x7f444402a510 comp rx=0 tx=0).stop 2024-06-29T15:59:25.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.864+0000 7f44610c0640 1 -- 172.21.15.81:0/4195354451 shutdown_connections 2024-06-29T15:59:25.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.864+0000 7f44610c0640 1 --2- 172.21.15.81:0/4195354451 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f445c0cccf0 0x7f445c0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.864+0000 7f44610c0640 1 --2- 172.21.15.81:0/4195354451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0b3200 0x7f445c0cc7b0 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.864+0000 7f44610c0640 1 -- 172.21.15.81:0/4195354451 >> 172.21.15.81:0/4195354451 conn(0x7f445c0b3680 msgr2=0x7f445c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:25.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.864+0000 7f44610c0640 1 -- 172.21.15.81:0/4195354451 shutdown_connections 2024-06-29T15:59:25.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.864+0000 7f44610c0640 1 -- 172.21.15.81:0/4195354451 wait complete. 2024-06-29T15:59:25.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.866+0000 7f44610c0640 1 Processor -- start 2024-06-29T15:59:25.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.866+0000 7f44610c0640 1 -- start start 2024-06-29T15:59:25.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.867+0000 7f44610c0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f445c0b3200 0x7f445c09f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.867+0000 7f44610c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0cccf0 0x7f445c1564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.867+0000 7f44610c0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f445c0d0770 0x7f445c159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.867+0000 7f44610c0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f445c0cedb0 con 0x7f445c0cccf0 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.867+0000 7f44610c0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f445c0cec30 con 0x7f445c0d0770 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.867+0000 7f44610c0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f445c0cef30 con 0x7f445c0b3200 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.867+0000 7f445a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0cccf0 0x7f445c1564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.867+0000 7f445a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0cccf0 0x7f445c1564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50400/0 (socket says 172.21.15.81:50400) 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.867+0000 7f445a59c640 1 -- 172.21.15.81:0/770008613 learned_addr learned my addr 172.21.15.81:0/770008613 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.867+0000 7f445ad9d640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f445c0b3200 0x7f445c09f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f445b59e640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f445c0d0770 0x7f445c159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f445a59c640 1 -- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f445c0b3200 msgr2=0x7f445c09f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f445a59c640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f445c0b3200 0x7f445c09f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f445a59c640 1 -- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f445c0d0770 msgr2=0x7f445c159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:25.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f445a59c640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f445c0d0770 0x7f445c159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:25.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f445a59c640 1 -- 172.21.15.81:0/770008613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445c157690 con 0x7f445c0cccf0 2024-06-29T15:59:25.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f445a59c640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0cccf0 0x7f445c1564d0 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f444c00d740 tx=0x7f444c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:25.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f44537fe640 1 -- 172.21.15.81:0/770008613 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c017070 con 0x7f445c0cccf0 2024-06-29T15:59:25.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f44537fe640 1 -- 172.21.15.81:0/770008613 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f444c009d20 con 0x7f445c0cccf0 2024-06-29T15:59:25.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f44610c0640 1 -- 172.21.15.81:0/770008613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f445c157950 con 0x7f445c0cccf0 2024-06-29T15:59:25.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f44537fe640 1 -- 172.21.15.81:0/770008613 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c0106c0 con 0x7f445c0cccf0 2024-06-29T15:59:25.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.868+0000 7f44610c0640 1 -- 172.21.15.81:0/770008613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f445c1e42f0 con 0x7f445c0cccf0 2024-06-29T15:59:25.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.869+0000 7f44610c0640 1 -- 172.21.15.81:0/770008613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f445c0cc7b0 con 0x7f445c0cccf0 2024-06-29T15:59:25.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.870+0000 7f44537fe640 1 -- 172.21.15.81:0/770008613 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f444c01a070 con 0x7f445c0cccf0 2024-06-29T15:59:25.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.870+0000 7f44537fe640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f443c076420 0x7f443c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:25.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.871+0000 7f445ad9d640 1 -- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f443c076420 msgr2=0x7f443c078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:25.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.871+0000 7f44537fe640 1 -- 172.21.15.81:0/770008613 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f444c097ca0 con 0x7f445c0cccf0 2024-06-29T15:59:25.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.871+0000 7f445ad9d640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f443c076420 0x7f443c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:25.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:25.875+0000 7f44537fe640 1 -- 172.21.15.81:0/770008613 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f444c060ad0 con 0x7f445c0cccf0 2024-06-29T15:59:26.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.062+0000 7f44610c0640 1 -- 172.21.15.81:0/770008613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f445c0b5c10 con 0x7f445c0cccf0 2024-06-29T15:59:26.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.063+0000 7f44537fe640 1 -- 172.21.15.81:0/770008613 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v254) ==== 100+0+11 (secure 0 0 0) 0x7f444c064dd0 con 0x7f445c0cccf0 2024-06-29T15:59:26.064 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 -- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f443c076420 msgr2=0x7f443c078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f443c076420 0x7f443c078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 -- 172.21.15.81:0/770008613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0cccf0 msgr2=0x7f445c1564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0cccf0 0x7f445c1564d0 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f444c00d740 tx=0x7f444c00dc00 comp rx=0 tx=0).stop 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 -- 172.21.15.81:0/770008613 shutdown_connections 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f443c076420 0x7f443c078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f445c0d0770 0x7f445c159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0cccf0 0x7f445c1564d0 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 --2- 172.21.15.81:0/770008613 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f445c0b3200 0x7f445c09f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 -- 172.21.15.81:0/770008613 >> 172.21.15.81:0/770008613 conn(0x7f445c0b3680 msgr2=0x7f445c070d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 -- 172.21.15.81:0/770008613 shutdown_connections 2024-06-29T15:59:26.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.065+0000 7f44610c0640 1 -- 172.21.15.81:0/770008613 wait complete. 2024-06-29T15:59:26.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T15:59:26.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.193+0000 7fd367647640 1 -- 172.21.15.81:0/2549426614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3600d0550 msgr2=0x7fd36006c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.193+0000 7fd367647640 1 --2- 172.21.15.81:0/2549426614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3600d0550 0x7fd36006c2f0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7fd360072980 tx=0x7fd35c02a560 comp rx=0 tx=0).stop 2024-06-29T15:59:26.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.193+0000 7fd367647640 1 -- 172.21.15.81:0/2549426614 shutdown_connections 2024-06-29T15:59:26.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.193+0000 7fd367647640 1 --2- 172.21.15.81:0/2549426614 >> v2:172.21.15.167:3300/0 conn(0x7fd36006c830 0x7fd36006f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.193+0000 7fd367647640 1 --2- 172.21.15.81:0/2549426614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3600d0550 0x7fd36006c2f0 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.193+0000 7fd367647640 1 --2- 172.21.15.81:0/2549426614 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd360159bd0 0x7fd3600d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.193+0000 7fd367647640 1 -- 172.21.15.81:0/2549426614 >> 172.21.15.81:0/2549426614 conn(0x7fd3600b2920 msgr2=0x7fd3600b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:26.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.193+0000 7fd367647640 1 -- 172.21.15.81:0/2549426614 shutdown_connections 2024-06-29T15:59:26.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.193+0000 7fd367647640 1 -- 172.21.15.81:0/2549426614 wait complete. 2024-06-29T15:59:26.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.195+0000 7fd367647640 1 Processor -- start 2024-06-29T15:59:26.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.195+0000 7fd367647640 1 -- start start 2024-06-29T15:59:26.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.196+0000 7fd367647640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd36006c830 0x7fd360135a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.196+0000 7fd367647640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3600d0550 0x7fd3600a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.196+0000 7fd367647640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd360159bd0 0x7fd3600a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.196+0000 7fd367647640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd36015fa90 con 0x7fd3600d0550 2024-06-29T15:59:26.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.196+0000 7fd367647640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd3600702e0 con 0x7fd36006c830 2024-06-29T15:59:26.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.196+0000 7fd367647640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd36015fc10 con 0x7fd360159bd0 2024-06-29T15:59:26.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.196+0000 7fd364be2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3600d0550 0x7fd3600a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.196+0000 7fd364be2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3600d0550 0x7fd3600a9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50424/0 (socket says 172.21.15.81:50424) 2024-06-29T15:59:26.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.196+0000 7fd364be2640 1 -- 172.21.15.81:0/1432585003 learned_addr learned my addr 172.21.15.81:0/1432585003 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd3653e3640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd36006c830 0x7fd360135a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd365be4640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd360159bd0 0x7fd3600a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd364be2640 1 -- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd360159bd0 msgr2=0x7fd3600a9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd364be2640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd360159bd0 0x7fd3600a9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd364be2640 1 -- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd36006c830 msgr2=0x7fd360135a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd364be2640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd36006c830 0x7fd360135a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd364be2640 1 -- 172.21.15.81:0/1432585003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36015bf10 con 0x7fd3600d0550 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd3653e3640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd36006c830 0x7fd360135a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd364be2640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3600d0550 0x7fd3600a9550 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7fd35c034040 tx=0x7fd35c0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd351ffb640 1 -- 172.21.15.81:0/1432585003 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c03e030 con 0x7fd3600d0550 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd351ffb640 1 -- 172.21.15.81:0/1432585003 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd35c003ca0 con 0x7fd3600d0550 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd367647640 1 -- 172.21.15.81:0/1432585003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3600b3b70 con 0x7fd3600d0550 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.197+0000 7fd351ffb640 1 -- 172.21.15.81:0/1432585003 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c035040 con 0x7fd3600d0550 2024-06-29T15:59:26.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.198+0000 7fd367647640 1 -- 172.21.15.81:0/1432585003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd360132f60 con 0x7fd3600d0550 2024-06-29T15:59:26.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.198+0000 7fd367647640 1 -- 172.21.15.81:0/1432585003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd36015d150 con 0x7fd3600d0550 2024-06-29T15:59:26.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.199+0000 7fd351ffb640 1 -- 172.21.15.81:0/1432585003 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7fd35c041030 con 0x7fd3600d0550 2024-06-29T15:59:26.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.200+0000 7fd351ffb640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fd3440763f0 0x7fd344078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.200+0000 7fd3653e3640 1 -- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fd3440763f0 msgr2=0x7fd344078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:26.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.200+0000 7fd351ffb640 1 -- 172.21.15.81:0/1432585003 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7fd35c0b83f0 con 0x7fd3600d0550 2024-06-29T15:59:26.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.200+0000 7fd3653e3640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fd3440763f0 0x7fd344078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:26.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.204+0000 7fd351ffb640 1 -- 172.21.15.81:0/1432585003 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd36015d150 con 0x7fd3600d0550 2024-06-29T15:59:26.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.401+0000 7fd3653e3640 1 -- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fd3440763f0 msgr2=0x7fd344078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:26.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.401+0000 7fd3653e3640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fd3440763f0 0x7fd344078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:26.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.405+0000 7fd367647640 1 -- 172.21.15.81:0/1432585003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd36006c2f0 con 0x7fd3600d0550 2024-06-29T15:59:26.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.405+0000 7fd351ffb640 1 -- 172.21.15.81:0/1432585003 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v254) ==== 81+0+596 (secure 0 0 0) 0x7fd35c0812b0 con 0x7fd3600d0550 2024-06-29T15:59:26.407 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3251388883 2024-06-30T15:59:21.470262+0000 2024-06-29T15:59:26.407 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2508797860 2024-06-30T15:59:21.470262+0000 2024-06-29T15:59:26.407 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3181687202 2024-06-30T15:59:21.470262+0000 2024-06-29T15:59:26.408 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1949624986 2024-06-30T15:59:21.470262+0000 2024-06-29T15:59:26.408 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2055038349 2024-06-30T15:59:21.470262+0000 2024-06-29T15:59:26.408 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1927715461 2024-06-30T15:59:21.470262+0000 2024-06-29T15:59:26.408 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/1778875525 2024-06-30T15:59:21.470262+0000 2024-06-29T15:59:26.408 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2270885245 2024-06-30T15:59:21.470262+0000 2024-06-29T15:59:26.408 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/1778875525 2024-06-30T15:59:21.470262+0000 2024-06-29T15:59:26.408 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1180941913 2024-06-30T15:59:21.470262+0000 2024-06-29T15:59:26.410 INFO:teuthology.orchestra.run.smithi081.stderr:listed 10 entries 2024-06-29T15:59:26.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 -- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fd3440763f0 msgr2=0x7fd344078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:26.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fd3440763f0 0x7fd344078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 -- 172.21.15.81:0/1432585003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3600d0550 msgr2=0x7fd3600a9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3600d0550 0x7fd3600a9550 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7fd35c034040 tx=0x7fd35c0038a0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 -- 172.21.15.81:0/1432585003 shutdown_connections 2024-06-29T15:59:26.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7fd3440763f0 0x7fd344078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd360159bd0 0x7fd3600a9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3600d0550 0x7fd3600a9550 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 --2- 172.21.15.81:0/1432585003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd36006c830 0x7fd360135a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 -- 172.21.15.81:0/1432585003 >> 172.21.15.81:0/1432585003 conn(0x7fd3600b2920 msgr2=0x7fd3600b1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:26.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 -- 172.21.15.81:0/1432585003 shutdown_connections 2024-06-29T15:59:26.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.408+0000 7fd367647640 1 -- 172.21.15.81:0/1432585003 wait complete. 2024-06-29T15:59:26.420 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T15:59:26.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:26.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:26.450 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T15:59:26.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:26.450 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T15:59:26.506 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:59:26.506 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:59:26.506 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:26.522 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:26.522 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:59:26.522 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:59:26.522 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:26.566 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:59:26.566 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T15:59:26.567 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:59:26.567 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:26.582 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:26.582 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:26.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.721+0000 7f02b9036640 1 -- 172.21.15.81:0/1390257334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02b40cd150 msgr2=0x7f02b4072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.721+0000 7f02b9036640 1 --2- 172.21.15.81:0/1390257334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02b40cd150 0x7f02b4072f40 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f02b4074310 tx=0x7f02a402a4a0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.721+0000 7f02b9036640 1 -- 172.21.15.81:0/1390257334 shutdown_connections 2024-06-29T15:59:26.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.721+0000 7f02b9036640 1 --2- 172.21.15.81:0/1390257334 >> v2:172.21.15.167:3300/0 conn(0x7f02b4073480 0x7f02b415b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.721+0000 7f02b9036640 1 --2- 172.21.15.81:0/1390257334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02b40cd150 0x7f02b4072f40 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.721+0000 7f02b9036640 1 -- 172.21.15.81:0/1390257334 >> 172.21.15.81:0/1390257334 conn(0x7f02b40b2920 msgr2=0x7f02b40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:26.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.721+0000 7f02b9036640 1 -- 172.21.15.81:0/1390257334 shutdown_connections 2024-06-29T15:59:26.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.721+0000 7f02b9036640 1 -- 172.21.15.81:0/1390257334 wait complete. 2024-06-29T15:59:26.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.723+0000 7f02b9036640 1 Processor -- start 2024-06-29T15:59:26.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.723+0000 7f02b9036640 1 -- start start 2024-06-29T15:59:26.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.724+0000 7f02b9036640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02b4073480 0x7f02b40cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.724+0000 7f02b9036640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02b40cd150 0x7f02b415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.724+0000 7f02b9036640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02b415bc20 0x7f02b4151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.724+0000 7f02b9036640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f02b415c1b0 con 0x7f02b415bc20 2024-06-29T15:59:26.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.724+0000 7f02b9036640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f02b415c030 con 0x7f02b40cd150 2024-06-29T15:59:26.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.724+0000 7f02b9036640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f02b415f060 con 0x7f02b4073480 2024-06-29T15:59:26.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.724+0000 7f02b359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02b415bc20 0x7f02b4151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.724+0000 7f02b359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02b415bc20 0x7f02b4151310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50450/0 (socket says 172.21.15.81:50450) 2024-06-29T15:59:26.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b359e640 1 -- 172.21.15.81:0/3273327627 learned_addr learned my addr 172.21.15.81:0/3273327627 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:26.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b259c640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02b40cd150 0x7f02b415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b2d9d640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02b4073480 0x7f02b40cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b359e640 1 -- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02b4073480 msgr2=0x7f02b40cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b359e640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02b4073480 0x7f02b40cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b359e640 1 -- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02b40cd150 msgr2=0x7f02b415a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b359e640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02b40cd150 0x7f02b415a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b359e640 1 -- 172.21.15.81:0/3273327627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b415ad30 con 0x7f02b415bc20 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b359e640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02b415bc20 0x7f02b4151310 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00cb40 tx=0x7f02ac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02ab7fe640 1 -- 172.21.15.81:0/3273327627 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac016020 con 0x7f02b415bc20 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02ab7fe640 1 -- 172.21.15.81:0/3273327627 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02ac00ad30 con 0x7f02b415bc20 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02ab7fe640 1 -- 172.21.15.81:0/3273327627 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac012680 con 0x7f02b415bc20 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b9036640 1 -- 172.21.15.81:0/3273327627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b40c9dc0 con 0x7f02b415bc20 2024-06-29T15:59:26.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.725+0000 7f02b9036640 1 -- 172.21.15.81:0/3273327627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b40ca380 con 0x7f02b415bc20 2024-06-29T15:59:26.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.727+0000 7f02b9036640 1 -- 172.21.15.81:0/3273327627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b40714d0 con 0x7f02b415bc20 2024-06-29T15:59:26.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.727+0000 7f02ab7fe640 1 -- 172.21.15.81:0/3273327627 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f02ac019020 con 0x7f02b415bc20 2024-06-29T15:59:26.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.727+0000 7f02ab7fe640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f0294076420 0x7f0294078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:26.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.727+0000 7f02b2d9d640 1 -- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f0294076420 msgr2=0x7f0294078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:26.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.727+0000 7f02ab7fe640 1 -- 172.21.15.81:0/3273327627 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f02ac098040 con 0x7f02b415bc20 2024-06-29T15:59:26.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.727+0000 7f02b2d9d640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f0294076420 0x7f0294078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:26.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.732+0000 7f02ab7fe640 1 -- 172.21.15.81:0/3273327627 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f02b40714d0 con 0x7f02b415bc20 2024-06-29T15:59:26.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.928+0000 7f02b2d9d640 1 -- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f0294076420 msgr2=0x7f0294078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:26.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.928+0000 7f02b2d9d640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f0294076420 0x7f0294078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:26.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.993+0000 7f02b9036640 1 -- 172.21.15.81:0/3273327627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02b415f4c0 con 0x7f02b415bc20 2024-06-29T15:59:26.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.993+0000 7f02ab7fe640 1 -- 172.21.15.81:0/3273327627 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+5236 (secure 0 0 0) 0x7f02ac020020 con 0x7f02b415bc20 2024-06-29T15:59:26.995 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:26.995 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":145,"btime":"2024-06-29T15:59:12:796690+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9842,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2717891820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2717891820},{"type":"v1","addr":"172.21.15.81:6835","nonce":2717891820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":9848,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2745105064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2745105064},{"type":"v1","addr":"172.21.15.167:6836","nonce":2745105064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":11329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/745441473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":745441473},{"type":"v1","addr":"172.21.15.81:6834","nonce":745441473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:09.126215+0000","modified":"2024-06-29T15:59:12.167052+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11347":{"gid":11347,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/54677014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":54677014},{"type":"v1","addr":"172.21.15.167:6837","nonce":54677014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11347,"qdb_cluster":[11347]},"id":17}]} 2024-06-29T15:59:26.995 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 145 2024-06-29T15:59:26.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 -- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f0294076420 msgr2=0x7f0294078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:26.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f0294076420 0x7f0294078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 -- 172.21.15.81:0/3273327627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02b415bc20 msgr2=0x7f02b4151310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:26.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02b415bc20 0x7f02b4151310 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00cb40 tx=0x7f02ac00a060 comp rx=0 tx=0).stop 2024-06-29T15:59:26.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 -- 172.21.15.81:0/3273327627 shutdown_connections 2024-06-29T15:59:26.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f0294076420 0x7f0294078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02b415bc20 0x7f02b4151310 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f02b40cd150 0x7f02b415a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 --2- 172.21.15.81:0/3273327627 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f02b4073480 0x7f02b40cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:26.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 -- 172.21.15.81:0/3273327627 >> 172.21.15.81:0/3273327627 conn(0x7f02b40b2920 msgr2=0x7f02b415d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:26.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.995+0000 7f02b9036640 1 -- 172.21.15.81:0/3273327627 shutdown_connections 2024-06-29T15:59:26.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:26.996+0000 7f02b9036640 1 -- 172.21.15.81:0/3273327627 wait complete. 2024-06-29T15:59:27.008 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:27.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.123+0000 7f41cc466640 1 -- 172.21.15.81:0/2744381303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c40d2a90 msgr2=0x7f41c40d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.123+0000 7f41cc466640 1 --2- 172.21.15.81:0/2744381303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c40d2a90 0x7f41c40d09c0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7f41bc005fd0 tx=0x7f41bc02a940 comp rx=0 tx=0).stop 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.124+0000 7f41cc466640 1 -- 172.21.15.81:0/2744381303 shutdown_connections 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.124+0000 7f41cc466640 1 --2- 172.21.15.81:0/2744381303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c40d2a90 0x7f41c40d09c0 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.124+0000 7f41cc466640 1 --2- 172.21.15.81:0/2744381303 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41c40b9040 0x7f41c40d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.124+0000 7f41cc466640 1 -- 172.21.15.81:0/2744381303 >> 172.21.15.81:0/2744381303 conn(0x7f41c409cf30 msgr2=0x7f41c40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.124+0000 7f41cc466640 1 -- 172.21.15.81:0/2744381303 shutdown_connections 2024-06-29T15:59:27.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.124+0000 7f41cc466640 1 -- 172.21.15.81:0/2744381303 wait complete. 2024-06-29T15:59:27.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.126+0000 7f41cc466640 1 Processor -- start 2024-06-29T15:59:27.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.126+0000 7f41cc466640 1 -- start start 2024-06-29T15:59:27.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41cc466640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c40b5250 0x7f41c4159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41cc466640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41c40b9040 0x7f41c40c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41cc466640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41c40d2a90 0x7f41c415cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41cc466640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f41c40b7d30 con 0x7f41c40b5250 2024-06-29T15:59:27.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41cc466640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f41c40b7bb0 con 0x7f41c40d2a90 2024-06-29T15:59:27.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41cc466640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f41c40b7eb0 con 0x7f41c40b9040 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41ca202640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c40b5250 0x7f41c4159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41ca202640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c40b5250 0x7f41c4159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50470/0 (socket says 172.21.15.81:50470) 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41ca202640 1 -- 172.21.15.81:0/741440800 learned_addr learned my addr 172.21.15.81:0/741440800 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41c9a01640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41c40b9040 0x7f41c40c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41caa03640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41c40d2a90 0x7f41c415cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41ca202640 1 -- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41c40b9040 msgr2=0x7f41c40c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41ca202640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41c40b9040 0x7f41c40c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41ca202640 1 -- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41c40d2a90 msgr2=0x7f41c415cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41ca202640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41c40d2a90 0x7f41c415cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.127+0000 7f41ca202640 1 -- 172.21.15.81:0/741440800 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c415a300 con 0x7f41c40b5250 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.128+0000 7f41ca202640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c40b5250 0x7f41c4159dc0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f41b800e7c0 tx=0x7f41b800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:27.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.128+0000 7f41b6ffd640 1 -- 172.21.15.81:0/741440800 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b800ccc0 con 0x7f41c40b5250 2024-06-29T15:59:27.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.128+0000 7f41b6ffd640 1 -- 172.21.15.81:0/741440800 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41b800ce60 con 0x7f41c40b5250 2024-06-29T15:59:27.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.128+0000 7f41cc466640 1 -- 172.21.15.81:0/741440800 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c415d7e0 con 0x7f41c40b5250 2024-06-29T15:59:27.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.128+0000 7f41b6ffd640 1 -- 172.21.15.81:0/741440800 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8005280 con 0x7f41c40b5250 2024-06-29T15:59:27.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.128+0000 7f41cc466640 1 -- 172.21.15.81:0/741440800 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c40c90e0 con 0x7f41c40b5250 2024-06-29T15:59:27.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.129+0000 7f41cc466640 1 -- 172.21.15.81:0/741440800 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c40b83b0 con 0x7f41c40b5250 2024-06-29T15:59:27.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.129+0000 7f41b6ffd640 1 -- 172.21.15.81:0/741440800 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f41b8005420 con 0x7f41c40b5250 2024-06-29T15:59:27.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.130+0000 7f41b6ffd640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f41a80763f0 0x7f41a8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.130+0000 7f41c9a01640 1 -- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f41a80763f0 msgr2=0x7f41a8078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:27.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.130+0000 7f41b6ffd640 1 -- 172.21.15.81:0/741440800 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f41b8098390 con 0x7f41c40b5250 2024-06-29T15:59:27.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.130+0000 7f41c9a01640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f41a80763f0 0x7f41a8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:27.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.134+0000 7f41b6ffd640 1 -- 172.21.15.81:0/741440800 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f41b8061250 con 0x7f41c40b5250 2024-06-29T15:59:27.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.319+0000 7f41cc466640 1 -- 172.21.15.81:0/741440800 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41c40ca7c0 con 0x7f41c40b5250 2024-06-29T15:59:27.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.320+0000 7f41b6ffd640 1 -- 172.21.15.81:0/741440800 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) ==== 74+0+17454 (secure 0 0 0) 0x7f41b8065550 con 0x7f41c40b5250 2024-06-29T15:59:27.322 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:27.322 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":254,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:21.470300+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:07.416791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:08.088694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3251388883":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2508797860":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/3181687202":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1949624986":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2055038349":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1927715461":"2024-06-30T15:59:21.470262+0000","172.21.15.167:6832/1778875525":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2270885245":"2024-06-30T15:59:21.470262+0000","172.21.15.167:6833/1778875525":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1180941913":"2024-06-30T15:59:21.470262+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 -- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f41a80763f0 msgr2=0x7f41a8078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f41a80763f0 0x7f41a8078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 -- 172.21.15.81:0/741440800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c40b5250 msgr2=0x7f41c4159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c40b5250 0x7f41c4159dc0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f41b800e7c0 tx=0x7f41b800ec80 comp rx=0 tx=0).stop 2024-06-29T15:59:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 -- 172.21.15.81:0/741440800 shutdown_connections 2024-06-29T15:59:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f41a80763f0 0x7f41a8078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f41c40d2a90 0x7f41c415cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f41c40b9040 0x7f41c40c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 --2- 172.21.15.81:0/741440800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c40b5250 0x7f41c4159dc0 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 -- 172.21.15.81:0/741440800 >> 172.21.15.81:0/741440800 conn(0x7f41c409cf30 msgr2=0x7f41c406d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:27.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 -- 172.21.15.81:0/741440800 shutdown_connections 2024-06-29T15:59:27.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.322+0000 7f41cc466640 1 -- 172.21.15.81:0/741440800 wait complete. 2024-06-29T15:59:27.336 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:59:07.416791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '252', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:59:08.088694+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:27.336 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.454+0000 7f38dab5b640 1 -- 172.21.15.81:0/1684803877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38c4003a00 msgr2=0x7f38c4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.454+0000 7f38dab5b640 1 --2- 172.21.15.81:0/1684803877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38c4003a00 0x7f38c4003e70 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f38c800c4a0 tx=0x7f38c4005830 comp rx=0 tx=0).stop 2024-06-29T15:59:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.454+0000 7f38d90f8640 1 -- 172.21.15.81:0/1684803877 reap_dead start 2024-06-29T15:59:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.454+0000 7f38dab5b640 1 -- 172.21.15.81:0/1684803877 shutdown_connections 2024-06-29T15:59:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.454+0000 7f38dab5b640 1 -- 172.21.15.81:0/1684803877 >> 172.21.15.81:0/1684803877 conn(0x7f38d40b3680 msgr2=0x7f38d40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.454+0000 7f38dab5b640 1 -- 172.21.15.81:0/1684803877 shutdown_connections 2024-06-29T15:59:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.454+0000 7f38dab5b640 1 -- 172.21.15.81:0/1684803877 wait complete. 2024-06-29T15:59:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.456+0000 7f38dab5b640 1 Processor -- start 2024-06-29T15:59:27.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.456+0000 7f38dab5b640 1 -- start start 2024-06-29T15:59:27.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38dab5b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38c4003a00 0x7f38d4158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38dab5b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38d41598d0 0x7f38d41f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38dab5b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38d41f6ae0 0x7f38d41e64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38dab5b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f38d40b56e0 con 0x7f38d41598d0 2024-06-29T15:59:27.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38dab5b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f38d40b5560 con 0x7f38d41f6ae0 2024-06-29T15:59:27.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38dab5b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f38d40b5860 con 0x7f38c4003a00 2024-06-29T15:59:27.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38d41598d0 0x7f38d41f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38d41598d0 0x7f38d41f65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50502/0 (socket says 172.21.15.81:50502) 2024-06-29T15:59:27.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d3fff640 1 -- 172.21.15.81:0/1532077451 learned_addr learned my addr 172.21.15.81:0/1532077451 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:27.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d88f7640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38c4003a00 0x7f38d4158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d90f8640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38d41f6ae0 0x7f38d41e64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d3fff640 1 -- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38c4003a00 msgr2=0x7f38d4158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d3fff640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38c4003a00 0x7f38d4158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d3fff640 1 -- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38d41f6ae0 msgr2=0x7f38d41e64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d3fff640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38d41f6ae0 0x7f38d41e64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d3fff640 1 -- 172.21.15.81:0/1532077451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d41f6ef0 con 0x7f38d41598d0 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d3fff640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38d41598d0 0x7f38d41f65a0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f38c0011ae0 tx=0x7f38c0009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d17fa640 1 -- 172.21.15.81:0/1532077451 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c001e070 con 0x7f38d41598d0 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38d17fa640 1 -- 172.21.15.81:0/1532077451 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38c000fc10 con 0x7f38d41598d0 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.458+0000 7f38dab5b640 1 -- 172.21.15.81:0/1532077451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d41e5dd0 con 0x7f38d41598d0 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.459+0000 7f38d17fa640 1 -- 172.21.15.81:0/1532077451 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c0019410 con 0x7f38d41598d0 2024-06-29T15:59:27.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.459+0000 7f38dab5b640 1 -- 172.21.15.81:0/1532077451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d41f7170 con 0x7f38d41598d0 2024-06-29T15:59:27.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.459+0000 7f38dab5b640 1 -- 172.21.15.81:0/1532077451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d40d0640 con 0x7f38d41598d0 2024-06-29T15:59:27.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.460+0000 7f38d17fa640 1 -- 172.21.15.81:0/1532077451 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f38c0021070 con 0x7f38d41598d0 2024-06-29T15:59:27.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.461+0000 7f38d17fa640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f38b8076420 0x7f38b8078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.461+0000 7f38d88f7640 1 -- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f38b8076420 msgr2=0x7f38b8078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:27.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.461+0000 7f38d17fa640 1 -- 172.21.15.81:0/1532077451 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f38c0098b30 con 0x7f38d41598d0 2024-06-29T15:59:27.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.461+0000 7f38d88f7640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f38b8076420 0x7f38b8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:27.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.464+0000 7f38d17fa640 1 -- 172.21.15.81:0/1532077451 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f38d41f6ef0 con 0x7f38d41598d0 2024-06-29T15:59:27.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.652+0000 7f38dab5b640 1 -- 172.21.15.81:0/1532077451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f38d409f820 con 0x7f38d41598d0 2024-06-29T15:59:27.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.653+0000 7f38d17fa640 1 -- 172.21.15.81:0/1532077451 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v254) ==== 93+0+10 (secure 0 0 0) 0x7f38c0061ad0 con 0x7f38d41598d0 2024-06-29T15:59:27.654 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:59:27.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 -- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f38b8076420 msgr2=0x7f38b8078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:27.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f38b8076420 0x7f38b8078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 -- 172.21.15.81:0/1532077451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38d41598d0 msgr2=0x7f38d41f65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38d41598d0 0x7f38d41f65a0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f38c0011ae0 tx=0x7f38c0009e60 comp rx=0 tx=0).stop 2024-06-29T15:59:27.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 -- 172.21.15.81:0/1532077451 shutdown_connections 2024-06-29T15:59:27.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f38b8076420 0x7f38b8078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f38d41f6ae0 0x7f38d41e64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38d41598d0 0x7f38d41f65a0 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 --2- 172.21.15.81:0/1532077451 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f38c4003a00 0x7f38d4158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 -- 172.21.15.81:0/1532077451 >> 172.21.15.81:0/1532077451 conn(0x7f38d40b3680 msgr2=0x7f38d4071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:27.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 -- 172.21.15.81:0/1532077451 shutdown_connections 2024-06-29T15:59:27.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.655+0000 7f38dab5b640 1 -- 172.21.15.81:0/1532077451 wait complete. 2024-06-29T15:59:27.668 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T15:59:27.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.785+0000 7f7f36aef640 1 -- 172.21.15.81:0/3811746272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f3006c2f0 msgr2=0x7f7f3006c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.785+0000 7f7f36aef640 1 --2- 172.21.15.81:0/3811746272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f3006c2f0 0x7f7f3006c700 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c005fd0 tx=0x7f7f1c02a5d0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.785+0000 7f7f36aef640 1 -- 172.21.15.81:0/3811746272 shutdown_connections 2024-06-29T15:59:27.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.785+0000 7f7f36aef640 1 --2- 172.21.15.81:0/3811746272 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f3006f9a0 0x7f7f3006fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.785+0000 7f7f36aef640 1 --2- 172.21.15.81:0/3811746272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f3006c2f0 0x7f7f3006c700 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.785+0000 7f7f36aef640 1 -- 172.21.15.81:0/3811746272 >> 172.21.15.81:0/3811746272 conn(0x7f7f300b2920 msgr2=0x7f7f300b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:27.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.785+0000 7f7f36aef640 1 -- 172.21.15.81:0/3811746272 shutdown_connections 2024-06-29T15:59:27.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.785+0000 7f7f36aef640 1 -- 172.21.15.81:0/3811746272 wait complete. 2024-06-29T15:59:27.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.787+0000 7f7f36aef640 1 Processor -- start 2024-06-29T15:59:27.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.787+0000 7f7f36aef640 1 -- start start 2024-06-29T15:59:27.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f36aef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f3006c2f0 0x7f7f30158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f36aef640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f3006f9a0 0x7f7f301f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f36aef640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f30159bd0 0x7f7f301f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f36aef640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3015fc10 con 0x7f7f3006c2f0 2024-06-29T15:59:27.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f36aef640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3015fa90 con 0x7f7f30159bd0 2024-06-29T15:59:27.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f36aef640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7f3015fd90 con 0x7f7f3006f9a0 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f3488b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f3006c2f0 0x7f7f30158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f3488b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f3006c2f0 0x7f7f30158720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50526/0 (socket says 172.21.15.81:50526) 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f3488b640 1 -- 172.21.15.81:0/1483996050 learned_addr learned my addr 172.21.15.81:0/1483996050 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f2ffff640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f3006f9a0 0x7f7f301f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f3508c640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f30159bd0 0x7f7f301f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f3488b640 1 -- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f3006f9a0 msgr2=0x7f7f301f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f3488b640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f3006f9a0 0x7f7f301f3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f3488b640 1 -- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f30159bd0 msgr2=0x7f7f301f3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f3488b640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f30159bd0 0x7f7f301f3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f3488b640 1 -- 172.21.15.81:0/1483996050 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f301f4500 con 0x7f7f3006c2f0 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f3488b640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f3006c2f0 0x7f7f30158720 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400c3a0 tx=0x7f7f2400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f2d7fa640 1 -- 172.21.15.81:0/1483996050 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24017070 con 0x7f7f3006c2f0 2024-06-29T15:59:27.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f2d7fa640 1 -- 172.21.15.81:0/1483996050 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f240053c0 con 0x7f7f3006c2f0 2024-06-29T15:59:27.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f2d7fa640 1 -- 172.21.15.81:0/1483996050 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24003850 con 0x7f7f3006c2f0 2024-06-29T15:59:27.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.789+0000 7f7f36aef640 1 -- 172.21.15.81:0/1483996050 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f300a96d0 con 0x7f7f3006c2f0 2024-06-29T15:59:27.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.790+0000 7f7f36aef640 1 -- 172.21.15.81:0/1483996050 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f300a9c90 con 0x7f7f3006c2f0 2024-06-29T15:59:27.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.791+0000 7f7f36aef640 1 -- 172.21.15.81:0/1483996050 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f300a94c0 con 0x7f7f3006c2f0 2024-06-29T15:59:27.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.791+0000 7f7f2d7fa640 1 -- 172.21.15.81:0/1483996050 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f7f2401a070 con 0x7f7f3006c2f0 2024-06-29T15:59:27.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.792+0000 7f7f2d7fa640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f7f14076420 0x7f7f14078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:27.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.792+0000 7f7f2ffff640 1 -- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f7f14076420 msgr2=0x7f7f14078930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:27.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.792+0000 7f7f2d7fa640 1 -- 172.21.15.81:0/1483996050 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f7f24096e50 con 0x7f7f3006c2f0 2024-06-29T15:59:27.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.792+0000 7f7f2ffff640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f7f14076420 0x7f7f14078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:27.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.796+0000 7f7f2d7fa640 1 -- 172.21.15.81:0/1483996050 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7f3006cc40 con 0x7f7f3006c2f0 2024-06-29T15:59:27.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.983+0000 7f7f36aef640 1 -- 172.21.15.81:0/1483996050 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7f301602e0 con 0x7f7f3006c2f0 2024-06-29T15:59:27.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.983+0000 7f7f2d7fa640 1 -- 172.21.15.81:0/1483996050 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v254) ==== 104+0+11 (secure 0 0 0) 0x7f7f2405fe20 con 0x7f7f3006c2f0 2024-06-29T15:59:27.985 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T15:59:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 -- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f7f14076420 msgr2=0x7f7f14078930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f7f14076420 0x7f7f14078930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 -- 172.21.15.81:0/1483996050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f3006c2f0 msgr2=0x7f7f30158720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f3006c2f0 0x7f7f30158720 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400c3a0 tx=0x7f7f2400c860 comp rx=0 tx=0).stop 2024-06-29T15:59:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 -- 172.21.15.81:0/1483996050 shutdown_connections 2024-06-29T15:59:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f7f14076420 0x7f7f14078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7f30159bd0 0x7f7f301f3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7f3006f9a0 0x7f7f301f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 --2- 172.21.15.81:0/1483996050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f3006c2f0 0x7f7f30158720 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:27.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 -- 172.21.15.81:0/1483996050 >> 172.21.15.81:0/1483996050 conn(0x7f7f300b2920 msgr2=0x7f7f300b06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:27.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 -- 172.21.15.81:0/1483996050 shutdown_connections 2024-06-29T15:59:27.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:27.985+0000 7f7f36aef640 1 -- 172.21.15.81:0/1483996050 wait complete. 2024-06-29T15:59:27.998 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T15:59:28.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.113+0000 7f0539711640 1 -- 172.21.15.81:0/3409382811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05340cd480 msgr2=0x7f05340ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.113+0000 7f0539711640 1 --2- 172.21.15.81:0/3409382811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05340cd480 0x7f05340ce510 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7f0528005fd0 tx=0x7f052802a5f0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.113+0000 7f0539711640 1 -- 172.21.15.81:0/3409382811 shutdown_connections 2024-06-29T15:59:28.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.113+0000 7f0539711640 1 --2- 172.21.15.81:0/3409382811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05340cd480 0x7f05340ce510 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.113+0000 7f0539711640 1 -- 172.21.15.81:0/3409382811 >> 172.21.15.81:0/3409382811 conn(0x7f05340b3510 msgr2=0x7f05340b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:28.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.113+0000 7f0539711640 1 -- 172.21.15.81:0/3409382811 shutdown_connections 2024-06-29T15:59:28.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.113+0000 7f0539711640 1 -- 172.21.15.81:0/3409382811 wait complete. 2024-06-29T15:59:28.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.115+0000 7f0539711640 1 Processor -- start 2024-06-29T15:59:28.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.115+0000 7f0539711640 1 -- start start 2024-06-29T15:59:28.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f0539711640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05340b6eb0 0x7f0534166de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f0539711640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05340cd480 0x7f0534153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f0539711640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05340cea50 0x7f05340c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f0539711640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f05340b52b0 con 0x7f05340cea50 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f0539711640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f05340b5130 con 0x7f05340cd480 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f0539711640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f0534167660 con 0x7f05340b6eb0 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f05337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05340cea50 0x7f05340c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f05337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05340cea50 0x7f05340c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50532/0 (socket says 172.21.15.81:50532) 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f05337fe640 1 -- 172.21.15.81:0/2933327192 learned_addr learned my addr 172.21.15.81:0/2933327192 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f0532ffd640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05340b6eb0 0x7f0534166de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f05327fc640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05340cd480 0x7f0534153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f05337fe640 1 -- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05340b6eb0 msgr2=0x7f0534166de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f05337fe640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05340b6eb0 0x7f0534166de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f05337fe640 1 -- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05340cd480 msgr2=0x7f0534153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f05337fe640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05340cd480 0x7f0534153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f05337fe640 1 -- 172.21.15.81:0/2933327192 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0534166240 con 0x7f05340cea50 2024-06-29T15:59:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f05337fe640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05340cea50 0x7f05340c9bb0 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7f052c00c6b0 tx=0x7f052c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:28.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f051b7fe640 1 -- 172.21.15.81:0/2933327192 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c003b70 con 0x7f05340cea50 2024-06-29T15:59:28.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f0539711640 1 -- 172.21.15.81:0/2933327192 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05340b1470 con 0x7f05340cea50 2024-06-29T15:59:28.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f051b7fe640 1 -- 172.21.15.81:0/2933327192 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f052c003d30 con 0x7f05340cea50 2024-06-29T15:59:28.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f051b7fe640 1 -- 172.21.15.81:0/2933327192 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c005330 con 0x7f05340cea50 2024-06-29T15:59:28.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.117+0000 7f0539711640 1 -- 172.21.15.81:0/2933327192 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0534164420 con 0x7f05340cea50 2024-06-29T15:59:28.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.119+0000 7f0539711640 1 -- 172.21.15.81:0/2933327192 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05340ce510 con 0x7f05340cea50 2024-06-29T15:59:28.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.120+0000 7f051b7fe640 1 -- 172.21.15.81:0/2933327192 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 151) ==== 98304+0+0 (secure 0 0 0) 0x7f052c0054d0 con 0x7f05340cea50 2024-06-29T15:59:28.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.120+0000 7f051b7fe640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f05200763f0 0x7f0520078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.120+0000 7f0532ffd640 1 -- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f05200763f0 msgr2=0x7f0520078900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1493691973 2024-06-29T15:59:28.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.120+0000 7f051b7fe640 1 -- 172.21.15.81:0/2933327192 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(254..254 src has 1..254) ==== 6267+0+0 (secure 0 0 0) 0x7f052c097890 con 0x7f05340cea50 2024-06-29T15:59:28.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.120+0000 7f0532ffd640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f05200763f0 0x7f0520078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:28.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.125+0000 7f051b7fe640 1 -- 172.21.15.81:0/2933327192 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f052c060750 con 0x7f05340cea50 2024-06-29T15:59:28.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.188+0000 7f051b7fe640 1 -- 172.21.15.81:0/2933327192 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 152) ==== 97912+0+0 (secure 0 0 0) 0x7f052c095b80 con 0x7f05340cea50 2024-06-29T15:59:28.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.189+0000 7f051b7fe640 1 -- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f05200763f0 msgr2=0x7f0520078900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:28.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.189+0000 7f051b7fe640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f05200763f0 0x7f0520078900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.245 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:28.244+0000 7f38a5b71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:28.246 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:28.244+0000 7f38a5b71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:28.246 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:28.244+0000 7f38a5b71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:28.246 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:28.244+0000 7f38a5b71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:28.246 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:28.244+0000 7f38a5b71640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:28.246 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:28.245+0000 7f38a2b6b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:28.246 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:28.245+0000 7f38a2b6b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:28.246 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:28.245+0000 7f38a2b6b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:28.246 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:28.245+0000 7f38a2b6b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:28.246 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:28.245+0000 7f38a2b6b640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:28.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.319+0000 7f0539711640 1 -- 172.21.15.81:0/2933327192 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0534163290 con 0x7f05340cea50 2024-06-29T15:59:28.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.320+0000 7f051b7fe640 1 -- 172.21.15.81:0/2933327192 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v255) ==== 100+0+11 (secure 0 0 0) 0x7f052c06c9c0 con 0x7f05340cea50 2024-06-29T15:59:28.321 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T15:59:28.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.322+0000 7f0539711640 1 -- 172.21.15.81:0/2933327192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05340cea50 msgr2=0x7f05340c9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.322+0000 7f0539711640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05340cea50 0x7f05340c9bb0 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7f052c00c6b0 tx=0x7f052c00cb70 comp rx=0 tx=0).stop 2024-06-29T15:59:28.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.322+0000 7f0539711640 1 -- 172.21.15.81:0/2933327192 shutdown_connections 2024-06-29T15:59:28.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.322+0000 7f0539711640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:6832/1493691973,v1:172.21.15.167:6833/1493691973] conn(0x7f05200763f0 0x7f0520078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.322+0000 7f0539711640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05340cea50 0x7f05340c9bb0 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.322+0000 7f0539711640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05340cd480 0x7f0534153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.322+0000 7f0539711640 1 --2- 172.21.15.81:0/2933327192 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05340b6eb0 0x7f0534166de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.322+0000 7f0539711640 1 -- 172.21.15.81:0/2933327192 >> 172.21.15.81:0/2933327192 conn(0x7f05340b3510 msgr2=0x7f05341637b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:28.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.322+0000 7f0539711640 1 -- 172.21.15.81:0/2933327192 shutdown_connections 2024-06-29T15:59:28.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.322+0000 7f0539711640 1 -- 172.21.15.81:0/2933327192 wait complete. 2024-06-29T15:59:28.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:28.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.449+0000 7f9cc1e65640 1 -- 172.21.15.81:0/1475297801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc073650 msgr2=0x7f9cbc15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.449+0000 7f9cc1e65640 1 --2- 172.21.15.81:0/1475297801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc073650 0x7f9cbc15b6e0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc072cf0 tx=0x7f9ca402a4a0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.449+0000 7f9cc1e65640 1 -- 172.21.15.81:0/1475297801 shutdown_connections 2024-06-29T15:59:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.449+0000 7f9cc1e65640 1 --2- 172.21.15.81:0/1475297801 >> v2:172.21.15.167:3300/0 conn(0x7f9cbc15bc20 0x7f9cbc06ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.449+0000 7f9cc1e65640 1 --2- 172.21.15.81:0/1475297801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc073650 0x7f9cbc15b6e0 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.449+0000 7f9cc1e65640 1 -- 172.21.15.81:0/1475297801 >> 172.21.15.81:0/1475297801 conn(0x7f9cbc0b2920 msgr2=0x7f9cbc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.449+0000 7f9cc1e65640 1 -- 172.21.15.81:0/1475297801 shutdown_connections 2024-06-29T15:59:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.449+0000 7f9cc1e65640 1 -- 172.21.15.81:0/1475297801 wait complete. 2024-06-29T15:59:28.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.450+0000 7f9cc1e65640 1 Processor -- start 2024-06-29T15:59:28.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.451+0000 7f9cc1e65640 1 -- start start 2024-06-29T15:59:28.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cc1e65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc0cd150 0x7f9cbc14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cc1e65640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9cbc15bc20 0x7f9cbc1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cc1e65640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cbc14a810 0x7f9cbc06c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cc1e65640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc06f450 con 0x7f9cbc0cd150 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cc1e65640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc06f2d0 con 0x7f9cbc14a810 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cc1e65640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9cbc06f5d0 con 0x7f9cbc15bc20 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cbb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc0cd150 0x7f9cbc14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cbb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc0cd150 0x7f9cbc14afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50556/0 (socket says 172.21.15.81:50556) 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/4180228261 learned_addr learned my addr 172.21.15.81:0/4180228261 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cbbfff640 1 --2- 172.21.15.81:0/4180228261 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cbc14a810 0x7f9cbc06c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cbaffd640 1 --2- 172.21.15.81:0/4180228261 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9cbc15bc20 0x7f9cbc1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/4180228261 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9cbc15bc20 msgr2=0x7f9cbc1587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cbb7fe640 1 --2- 172.21.15.81:0/4180228261 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9cbc15bc20 0x7f9cbc1587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/4180228261 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cbc14a810 msgr2=0x7f9cbc06c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cbb7fe640 1 --2- 172.21.15.81:0/4180228261 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cbc14a810 0x7f9cbc06c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.452+0000 7f9cbb7fe640 1 -- 172.21.15.81:0/4180228261 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cbc158d10 con 0x7f9cbc0cd150 2024-06-29T15:59:28.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.453+0000 7f9cbb7fe640 1 --2- 172.21.15.81:0/4180228261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc0cd150 0x7f9cbc14afd0 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00c3c0 tx=0x7f9cac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:28.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.453+0000 7f9cb3fff640 1 -- 172.21.15.81:0/4180228261 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac017070 con 0x7f9cbc0cd150 2024-06-29T15:59:28.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.453+0000 7f9cb3fff640 1 -- 172.21.15.81:0/4180228261 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9cac0035f0 con 0x7f9cbc0cd150 2024-06-29T15:59:28.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.453+0000 7f9cc1e65640 1 -- 172.21.15.81:0/4180228261 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cbc156360 con 0x7f9cbc0cd150 2024-06-29T15:59:28.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.453+0000 7f9cb3fff640 1 -- 172.21.15.81:0/4180228261 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cac003a80 con 0x7f9cbc0cd150 2024-06-29T15:59:28.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.453+0000 7f9cc1e65640 1 -- 172.21.15.81:0/4180228261 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cbc0ae740 con 0x7f9cbc0cd150 2024-06-29T15:59:28.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.454+0000 7f9cc1e65640 1 -- 172.21.15.81:0/4180228261 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cbc15b6e0 con 0x7f9cbc0cd150 2024-06-29T15:59:28.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.454+0000 7f9cb3fff640 1 -- 172.21.15.81:0/4180228261 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 152) ==== 97912+0+0 (secure 0 0 0) 0x7f9cac01a070 con 0x7f9cbc0cd150 2024-06-29T15:59:28.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.455+0000 7f9cb3fff640 1 -- 172.21.15.81:0/4180228261 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 6611+0+0 (secure 0 0 0) 0x7f9cac096910 con 0x7f9cbc0cd150 2024-06-29T15:59:28.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.460+0000 7f9cb3fff640 1 -- 172.21.15.81:0/4180228261 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9cac05fbf0 con 0x7f9cbc0cd150 2024-06-29T15:59:28.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.680+0000 7f9cc1e65640 1 -- 172.21.15.81:0/4180228261 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cbc14b510 con 0x7f9cbc0cd150 2024-06-29T15:59:28.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.680+0000 7f9cb3fff640 1 -- 172.21.15.81:0/4180228261 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+5236 (secure 0 0 0) 0x7f9cac063ef0 con 0x7f9cbc0cd150 2024-06-29T15:59:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:28.682 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":145,"btime":"2024-06-29T15:59:12:796690+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9842,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2717891820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2717891820},{"type":"v1","addr":"172.21.15.81:6835","nonce":2717891820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":9848,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2745105064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2745105064},{"type":"v1","addr":"172.21.15.167:6836","nonce":2745105064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":11329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/745441473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":745441473},{"type":"v1","addr":"172.21.15.81:6834","nonce":745441473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:09.126215+0000","modified":"2024-06-29T15:59:12.167052+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11347":{"gid":11347,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/54677014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":54677014},{"type":"v1","addr":"172.21.15.167:6837","nonce":54677014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11347,"qdb_cluster":[11347]},"id":17}]} 2024-06-29T15:59:28.682 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 145 2024-06-29T15:59:28.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.682+0000 7f9cc1e65640 1 -- 172.21.15.81:0/4180228261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc0cd150 msgr2=0x7f9cbc14afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.682+0000 7f9cc1e65640 1 --2- 172.21.15.81:0/4180228261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc0cd150 0x7f9cbc14afd0 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00c3c0 tx=0x7f9cac00c880 comp rx=0 tx=0).stop 2024-06-29T15:59:28.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.682+0000 7f9cc1e65640 1 -- 172.21.15.81:0/4180228261 shutdown_connections 2024-06-29T15:59:28.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.682+0000 7f9cc1e65640 1 --2- 172.21.15.81:0/4180228261 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9cbc14a810 0x7f9cbc06c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.682+0000 7f9cc1e65640 1 --2- 172.21.15.81:0/4180228261 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9cbc15bc20 0x7f9cbc1587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.682+0000 7f9cc1e65640 1 --2- 172.21.15.81:0/4180228261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cbc0cd150 0x7f9cbc14afd0 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.682+0000 7f9cc1e65640 1 -- 172.21.15.81:0/4180228261 >> 172.21.15.81:0/4180228261 conn(0x7f9cbc0b2920 msgr2=0x7f9cbc0b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:28.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.682+0000 7f9cc1e65640 1 -- 172.21.15.81:0/4180228261 shutdown_connections 2024-06-29T15:59:28.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.682+0000 7f9cc1e65640 1 -- 172.21.15.81:0/4180228261 wait complete. 2024-06-29T15:59:28.696 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:28.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.809+0000 7f6e6084e640 1 -- 172.21.15.81:0/3388823317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e48003a30 msgr2=0x7f6e48003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.809+0000 7f6e6084e640 1 --2- 172.21.15.81:0/3388823317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e48003a30 0x7f6e48003ea0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400c610 tx=0x7f6e48005830 comp rx=0 tx=0).stop 2024-06-29T15:59:28.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.809+0000 7f6e5ad9d640 1 -- 172.21.15.81:0/3388823317 reap_dead start 2024-06-29T15:59:28.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.809+0000 7f6e6084e640 1 -- 172.21.15.81:0/3388823317 shutdown_connections 2024-06-29T15:59:28.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.809+0000 7f6e6084e640 1 -- 172.21.15.81:0/3388823317 >> 172.21.15.81:0/3388823317 conn(0x7f6e5c06ccc0 msgr2=0x7f6e5c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:28.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.809+0000 7f6e6084e640 1 -- 172.21.15.81:0/3388823317 shutdown_connections 2024-06-29T15:59:28.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.809+0000 7f6e6084e640 1 -- 172.21.15.81:0/3388823317 wait complete. 2024-06-29T15:59:28.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.811+0000 7f6e6084e640 1 Processor -- start 2024-06-29T15:59:28.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.811+0000 7f6e6084e640 1 -- start start 2024-06-29T15:59:28.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e6084e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e48003a30 0x7f6e5c13fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e6084e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e5c152f50 0x7f6e5c156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e6084e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e5c140320 0x7f6e5c1f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e6084e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c15eef0 con 0x7f6e48003a30 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e6084e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c15ed70 con 0x7f6e5c140320 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e6084e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6e5c15f070 con 0x7f6e5c152f50 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e5a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e48003a30 0x7f6e5c13fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e5a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e48003a30 0x7f6e5c13fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50574/0 (socket says 172.21.15.81:50574) 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e5a59c640 1 -- 172.21.15.81:0/31329572 learned_addr learned my addr 172.21.15.81:0/31329572 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e59d9b640 1 --2- 172.21.15.81:0/31329572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e5c152f50 0x7f6e5c156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e5ad9d640 1 --2- 172.21.15.81:0/31329572 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e5c140320 0x7f6e5c1f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e5a59c640 1 -- 172.21.15.81:0/31329572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e5c152f50 msgr2=0x7f6e5c156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e5a59c640 1 --2- 172.21.15.81:0/31329572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e5c152f50 0x7f6e5c156090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e5a59c640 1 -- 172.21.15.81:0/31329572 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e5c140320 msgr2=0x7f6e5c1f1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:28.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e5a59c640 1 --2- 172.21.15.81:0/31329572 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e5c140320 0x7f6e5c1f1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:28.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e5a59c640 1 -- 172.21.15.81:0/31329572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5c153730 con 0x7f6e48003a30 2024-06-29T15:59:28.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.812+0000 7f6e5a59c640 1 --2- 172.21.15.81:0/31329572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e48003a30 0x7f6e5c13fb60 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400c700 tx=0x7f6e54004c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:28.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.813+0000 7f6e52ffd640 1 -- 172.21.15.81:0/31329572 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e54039070 con 0x7f6e48003a30 2024-06-29T15:59:28.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.813+0000 7f6e52ffd640 1 -- 172.21.15.81:0/31329572 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e54004e50 con 0x7f6e48003a30 2024-06-29T15:59:28.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.813+0000 7f6e6084e640 1 -- 172.21.15.81:0/31329572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e5c155310 con 0x7f6e48003a30 2024-06-29T15:59:28.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.813+0000 7f6e52ffd640 1 -- 172.21.15.81:0/31329572 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e54006070 con 0x7f6e48003a30 2024-06-29T15:59:28.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.813+0000 7f6e6084e640 1 -- 172.21.15.81:0/31329572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e5c155890 con 0x7f6e48003a30 2024-06-29T15:59:28.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.814+0000 7f6e6084e640 1 -- 172.21.15.81:0/31329572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e5c15f250 con 0x7f6e48003a30 2024-06-29T15:59:28.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.814+0000 7f6e52ffd640 1 -- 172.21.15.81:0/31329572 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 152) ==== 97912+0+0 (secure 0 0 0) 0x7f6e54045020 con 0x7f6e48003a30 2024-06-29T15:59:28.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.815+0000 7f6e52ffd640 1 -- 172.21.15.81:0/31329572 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 6611+0+0 (secure 0 0 0) 0x7f6e540b7aa0 con 0x7f6e48003a30 2024-06-29T15:59:28.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:28.819+0000 7f6e52ffd640 1 -- 172.21.15.81:0/31329572 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f6e540814c0 con 0x7f6e48003a30 2024-06-29T15:59:29.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.002+0000 7f6e6084e640 1 -- 172.21.15.81:0/31329572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e5c0af390 con 0x7f6e48003a30 2024-06-29T15:59:29.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.003+0000 7f6e52ffd640 1 -- 172.21.15.81:0/31329572 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) ==== 74+0+17962 (secure 0 0 0) 0x7f6e54080dc0 con 0x7f6e48003a30 2024-06-29T15:59:29.005 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:29.005 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":255,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:28.179092+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:07.416791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:08.088694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/335340229":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/3651252088":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/2009566943":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/604621260":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/2128674418":"2024-06-30T15:59:28.179050+0000","172.21.15.167:6833/1493691973":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/3251388883":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2508797860":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/3445975737":"2024-06-30T15:59:28.179050+0000","172.21.15.167:6832/1493691973":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/3181687202":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1949624986":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2055038349":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1927715461":"2024-06-30T15:59:21.470262+0000","172.21.15.167:6832/1778875525":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2270885245":"2024-06-30T15:59:21.470262+0000","172.21.15.167:6833/1778875525":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1180941913":"2024-06-30T15:59:21.470262+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:29.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.005+0000 7f6e6084e640 1 -- 172.21.15.81:0/31329572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e48003a30 msgr2=0x7f6e5c13fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.005+0000 7f6e6084e640 1 --2- 172.21.15.81:0/31329572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e48003a30 0x7f6e5c13fb60 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400c700 tx=0x7f6e54004c30 comp rx=0 tx=0).stop 2024-06-29T15:59:29.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.005+0000 7f6e6084e640 1 -- 172.21.15.81:0/31329572 shutdown_connections 2024-06-29T15:59:29.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.005+0000 7f6e6084e640 1 --2- 172.21.15.81:0/31329572 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6e5c140320 0x7f6e5c1f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.005+0000 7f6e6084e640 1 --2- 172.21.15.81:0/31329572 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6e5c152f50 0x7f6e5c156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.005+0000 7f6e6084e640 1 --2- 172.21.15.81:0/31329572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e48003a30 0x7f6e5c13fb60 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.005+0000 7f6e6084e640 1 -- 172.21.15.81:0/31329572 >> 172.21.15.81:0/31329572 conn(0x7f6e5c06ccc0 msgr2=0x7f6e5c15b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.005+0000 7f6e6084e640 1 -- 172.21.15.81:0/31329572 shutdown_connections 2024-06-29T15:59:29.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.005+0000 7f6e6084e640 1 -- 172.21.15.81:0/31329572 wait complete. 2024-06-29T15:59:29.018 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T15:59:29.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:29.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.129+0000 7fa2f229e640 1 -- 172.21.15.81:0/1678459035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2ec0b9040 msgr2=0x7fa2ec0d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.129+0000 7fa2f229e640 1 --2- 172.21.15.81:0/1678459035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2ec0b9040 0x7fa2ec0d0480 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0a1b70 tx=0x7fa2dc02a5f0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.129+0000 7fa2f229e640 1 -- 172.21.15.81:0/1678459035 shutdown_connections 2024-06-29T15:59:29.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.129+0000 7fa2f229e640 1 --2- 172.21.15.81:0/1678459035 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2ec0b7290 0x7fa2ec0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.129+0000 7fa2f229e640 1 --2- 172.21.15.81:0/1678459035 >> v2:172.21.15.167:3300/0 conn(0x7fa2ec0b5250 0x7fa2ec0d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.129+0000 7fa2f229e640 1 --2- 172.21.15.81:0/1678459035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2ec0b9040 0x7fa2ec0d0480 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.129+0000 7fa2f229e640 1 -- 172.21.15.81:0/1678459035 >> 172.21.15.81:0/1678459035 conn(0x7fa2ec09cf30 msgr2=0x7fa2ec0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.129+0000 7fa2f229e640 1 -- 172.21.15.81:0/1678459035 shutdown_connections 2024-06-29T15:59:29.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.129+0000 7fa2f229e640 1 -- 172.21.15.81:0/1678459035 wait complete. 2024-06-29T15:59:29.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.131+0000 7fa2f229e640 1 Processor -- start 2024-06-29T15:59:29.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.131+0000 7fa2f229e640 1 -- start start 2024-06-29T15:59:29.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2f229e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2ec0b5250 0x7fa2ec1f7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2f229e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2ec0b7290 0x7fa2ec0b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2f229e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2ec0b9040 0x7fa2ec15ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2f229e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec0b7f70 con 0x7fa2ec0b5250 2024-06-29T15:59:29.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2f229e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec0b7df0 con 0x7fa2ec0b9040 2024-06-29T15:59:29.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2f229e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa2ec0b80f0 con 0x7fa2ec0b7290 2024-06-29T15:59:29.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2ebfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2ec0b5250 0x7fa2ec1f7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2ebfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2ec0b5250 0x7fa2ec1f7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50590/0 (socket says 172.21.15.81:50590) 2024-06-29T15:59:29.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2ebfff640 1 -- 172.21.15.81:0/56960679 learned_addr learned my addr 172.21.15.81:0/56960679 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:29.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2ebfff640 1 -- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2ec0b7290 msgr2=0x7fa2ec0b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2ebfff640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2ec0b7290 0x7fa2ec0b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2ebfff640 1 -- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2ec0b9040 msgr2=0x7fa2ec15ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2ebfff640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2ec0b9040 0x7fa2ec15ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2ebfff640 1 -- 172.21.15.81:0/56960679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ec0c82d0 con 0x7fa2ec0b5250 2024-06-29T15:59:29.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2ebfff640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2ec0b5250 0x7fa2ec1f7e10 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc034040 tx=0x7fa2dc0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:29.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2e8ff9640 1 -- 172.21.15.81:0/56960679 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc03e030 con 0x7fa2ec0b5250 2024-06-29T15:59:29.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2e8ff9640 1 -- 172.21.15.81:0/56960679 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2dc0047a0 con 0x7fa2ec0b5250 2024-06-29T15:59:29.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2e8ff9640 1 -- 172.21.15.81:0/56960679 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc035040 con 0x7fa2ec0b5250 2024-06-29T15:59:29.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2f229e640 1 -- 172.21.15.81:0/56960679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ec0c85b0 con 0x7fa2ec0b5250 2024-06-29T15:59:29.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.133+0000 7fa2f229e640 1 -- 172.21.15.81:0/56960679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ec0c8b70 con 0x7fa2ec0b5250 2024-06-29T15:59:29.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.134+0000 7fa2f229e640 1 -- 172.21.15.81:0/56960679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2ec0b76a0 con 0x7fa2ec0b5250 2024-06-29T15:59:29.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.135+0000 7fa2e8ff9640 1 -- 172.21.15.81:0/56960679 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 152) ==== 97912+0+0 (secure 0 0 0) 0x7fa2dc041030 con 0x7fa2ec0b5250 2024-06-29T15:59:29.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.136+0000 7fa2e8ff9640 1 -- 172.21.15.81:0/56960679 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 6611+0+0 (secure 0 0 0) 0x7fa2dc0b7d80 con 0x7fa2ec0b5250 2024-06-29T15:59:29.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.140+0000 7fa2e8ff9640 1 -- 172.21.15.81:0/56960679 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa2dc081710 con 0x7fa2ec0b5250 2024-06-29T15:59:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.189+0000 7fa2e8ff9640 1 -- 172.21.15.81:0/56960679 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 153) ==== 98039+0+0 (secure 0 0 0) 0x7fa2dc07c980 con 0x7fa2ec0b5250 2024-06-29T15:59:29.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.190+0000 7fa2e8ff9640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa2d807bc60 0x7fa2d807e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.194+0000 7fa2eb7fe640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa2d807bc60 0x7fa2d807e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.197+0000 7fa2eb7fe640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa2d807bc60 0x7fa2d807e080 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec15e680 tx=0x7fa2e0005bd0 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:29.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.362+0000 7fa2f229e640 1 -- 172.21.15.81:0/56960679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2ec0757a0 con 0x7fa2ec0b5250 2024-06-29T15:59:29.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.362+0000 7fa2e8ff9640 1 -- 172.21.15.81:0/56960679 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+5236 (secure 0 0 0) 0x7fa2dc0b23c0 con 0x7fa2ec0b5250 2024-06-29T15:59:29.364 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:29.364 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":145,"btime":"2024-06-29T15:59:12:796690+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9842,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2717891820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2717891820},{"type":"v1","addr":"172.21.15.81:6835","nonce":2717891820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":9848,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2745105064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2745105064},{"type":"v1","addr":"172.21.15.167:6836","nonce":2745105064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":11329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/745441473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":745441473},{"type":"v1","addr":"172.21.15.81:6834","nonce":745441473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:09.126215+0000","modified":"2024-06-29T15:59:12.167052+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11347":{"gid":11347,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/54677014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":54677014},{"type":"v1","addr":"172.21.15.167:6837","nonce":54677014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11347,"qdb_cluster":[11347]},"id":17}]} 2024-06-29T15:59:29.364 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 145 2024-06-29T15:59:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 -- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa2d807bc60 msgr2=0x7fa2d807e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa2d807bc60 0x7fa2d807e080 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec15e680 tx=0x7fa2e0005bd0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 -- 172.21.15.81:0/56960679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2ec0b5250 msgr2=0x7fa2ec1f7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2ec0b5250 0x7fa2ec1f7e10 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc034040 tx=0x7fa2dc0039e0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 -- 172.21.15.81:0/56960679 shutdown_connections 2024-06-29T15:59:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa2d807bc60 0x7fa2d807e080 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa2ec0b9040 0x7fa2ec15ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa2ec0b7290 0x7fa2ec0b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 --2- 172.21.15.81:0/56960679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2ec0b5250 0x7fa2ec1f7e10 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 -- 172.21.15.81:0/56960679 >> 172.21.15.81:0/56960679 conn(0x7fa2ec09cf30 msgr2=0x7fa2ec06d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 -- 172.21.15.81:0/56960679 shutdown_connections 2024-06-29T15:59:29.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.365+0000 7fa2f229e640 1 -- 172.21.15.81:0/56960679 wait complete. 2024-06-29T15:59:29.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:29.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.489+0000 7f1a97b3c640 1 -- 172.21.15.81:0/3635536115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a900b5250 msgr2=0x7f1a900d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.489+0000 7f1a97b3c640 1 --2- 172.21.15.81:0/3635536115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a900b5250 0x7f1a900d1fc0 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f1a900a1b70 tx=0x7f1a8802ab40 comp rx=0 tx=0).stop 2024-06-29T15:59:29.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.489+0000 7f1a97b3c640 1 -- 172.21.15.81:0/3635536115 shutdown_connections 2024-06-29T15:59:29.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.489+0000 7f1a97b3c640 1 --2- 172.21.15.81:0/3635536115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a900b5250 0x7f1a900d1fc0 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.489+0000 7f1a97b3c640 1 --2- 172.21.15.81:0/3635536115 >> v2:172.21.15.167:3300/0 conn(0x7f1a900d0b90 0x7f1a900d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.489+0000 7f1a97b3c640 1 -- 172.21.15.81:0/3635536115 >> 172.21.15.81:0/3635536115 conn(0x7f1a9009cf30 msgr2=0x7f1a900d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.489+0000 7f1a97b3c640 1 -- 172.21.15.81:0/3635536115 shutdown_connections 2024-06-29T15:59:29.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.489+0000 7f1a97b3c640 1 -- 172.21.15.81:0/3635536115 wait complete. 2024-06-29T15:59:29.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.491+0000 7f1a97b3c640 1 Processor -- start 2024-06-29T15:59:29.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.491+0000 7f1a97b3c640 1 -- start start 2024-06-29T15:59:29.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.492+0000 7f1a97b3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a900b5250 0x7f1a901f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.492+0000 7f1a97b3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a900b9040 0x7f1a901f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.492+0000 7f1a97b3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a900b0820 0x7f1a900a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.492+0000 7f1a97b3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a900d2500 con 0x7f1a900b9040 2024-06-29T15:59:29.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.492+0000 7f1a97b3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1a900b84b0 con 0x7f1a900b5250 2024-06-29T15:59:29.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.492+0000 7f1a97b3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1a900d2680 con 0x7f1a900b0820 2024-06-29T15:59:29.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.492+0000 7f1a950d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a900b9040 0x7f1a901f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.492+0000 7f1a950d7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a900b9040 0x7f1a901f4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50612/0 (socket says 172.21.15.81:50612) 2024-06-29T15:59:29.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.492+0000 7f1a950d7640 1 -- 172.21.15.81:0/2704623385 learned_addr learned my addr 172.21.15.81:0/2704623385 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:29.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a958d8640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a900b5250 0x7f1a901f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a960d9640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a900b0820 0x7f1a900a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a950d7640 1 -- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a900b0820 msgr2=0x7f1a900a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a950d7640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a900b0820 0x7f1a900a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a950d7640 1 -- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a900b5250 msgr2=0x7f1a901f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a950d7640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a900b5250 0x7f1a901f5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a950d7640 1 -- 172.21.15.81:0/2704623385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a900b0f70 con 0x7f1a900b9040 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a960d9640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a900b0820 0x7f1a900a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a958d8640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a900b5250 0x7f1a901f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a950d7640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a900b9040 0x7f1a901f4040 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400b410 tx=0x7f1a8400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a8e7fc640 1 -- 172.21.15.81:0/2704623385 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84015020 con 0x7f1a900b9040 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a8e7fc640 1 -- 172.21.15.81:0/2704623385 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a84011940 con 0x7f1a900b9040 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a97b3c640 1 -- 172.21.15.81:0/2704623385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a900b1250 con 0x7f1a900b9040 2024-06-29T15:59:29.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a8e7fc640 1 -- 172.21.15.81:0/2704623385 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84011d90 con 0x7f1a900b9040 2024-06-29T15:59:29.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.493+0000 7f1a97b3c640 1 -- 172.21.15.81:0/2704623385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a900b1810 con 0x7f1a900b9040 2024-06-29T15:59:29.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.494+0000 7f1a97b3c640 1 -- 172.21.15.81:0/2704623385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a900d2c10 con 0x7f1a900b9040 2024-06-29T15:59:29.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.495+0000 7f1a8e7fc640 1 -- 172.21.15.81:0/2704623385 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 153) ==== 98039+0+0 (secure 0 0 0) 0x7f1a84018020 con 0x7f1a900b9040 2024-06-29T15:59:29.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.496+0000 7f1a8e7fc640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1a7c076150 0x7f1a7c078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.496+0000 7f1a8e7fc640 1 -- 172.21.15.81:0/2704623385 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 6611+0+0 (secure 0 0 0) 0x7f1a84097700 con 0x7f1a900b9040 2024-06-29T15:59:29.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.496+0000 7f1a958d8640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1a7c076150 0x7f1a7c078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.497+0000 7f1a958d8640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1a7c076150 0x7f1a7c078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1a800016a0 tx=0x7f1a80009040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:29.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.501+0000 7f1a8e7fc640 1 -- 172.21.15.81:0/2704623385 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1a8401f020 con 0x7f1a900b9040 2024-06-29T15:59:29.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.718+0000 7f1a97b3c640 1 -- 172.21.15.81:0/2704623385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a900b0d00 con 0x7f1a900b9040 2024-06-29T15:59:29.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.719+0000 7f1a8e7fc640 1 -- 172.21.15.81:0/2704623385 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+5236 (secure 0 0 0) 0x7f1a84060f90 con 0x7f1a900b9040 2024-06-29T15:59:29.719 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:29.719 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":145,"btime":"2024-06-29T15:59:12:796690+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9842,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2717891820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2717891820},{"type":"v1","addr":"172.21.15.81:6835","nonce":2717891820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":9848,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2745105064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2745105064},{"type":"v1","addr":"172.21.15.167:6836","nonce":2745105064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":11329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/745441473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":745441473},{"type":"v1","addr":"172.21.15.81:6834","nonce":745441473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:09.126215+0000","modified":"2024-06-29T15:59:12.167052+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11347},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11347":{"gid":11347,"name":"d","rank":0,"incarnation":142,"state":"up:active","state_seq":3,"addr":"172.21.15.167:6837/54677014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":54677014},{"type":"v1","addr":"172.21.15.167:6837","nonce":54677014}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11347,"qdb_cluster":[11347]},"id":17}]} 2024-06-29T15:59:29.720 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 145 2024-06-29T15:59:29.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 -- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1a7c076150 msgr2=0x7f1a7c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1a7c076150 0x7f1a7c078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1a800016a0 tx=0x7f1a80009040 comp rx=0 tx=0).stop 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 -- 172.21.15.81:0/2704623385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a900b9040 msgr2=0x7f1a901f4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a900b9040 0x7f1a901f4040 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400b410 tx=0x7f1a8400b8d0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 -- 172.21.15.81:0/2704623385 shutdown_connections 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1a7c076150 0x7f1a7c078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1a900b0820 0x7f1a900a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a900b9040 0x7f1a901f4040 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 --2- 172.21.15.81:0/2704623385 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1a900b5250 0x7f1a901f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 -- 172.21.15.81:0/2704623385 >> 172.21.15.81:0/2704623385 conn(0x7f1a9009cf30 msgr2=0x7f1a900b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 -- 172.21.15.81:0/2704623385 shutdown_connections 2024-06-29T15:59:29.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.721+0000 7f1a97b3c640 1 -- 172.21.15.81:0/2704623385 wait complete. 2024-06-29T15:59:29.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:29.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.844+0000 7f56707f0640 1 -- 172.21.15.81:0/3259342471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5668074520 msgr2=0x7f566806f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.844+0000 7f56707f0640 1 --2- 172.21.15.81:0/3259342471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5668074520 0x7f566806f140 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f5664005fd0 tx=0x7f566402a670 comp rx=0 tx=0).stop 2024-06-29T15:59:29.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.844+0000 7f56707f0640 1 -- 172.21.15.81:0/3259342471 shutdown_connections 2024-06-29T15:59:29.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.844+0000 7f56707f0640 1 --2- 172.21.15.81:0/3259342471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5668074520 0x7f566806f140 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.844+0000 7f56707f0640 1 -- 172.21.15.81:0/3259342471 >> 172.21.15.81:0/3259342471 conn(0x7f5668072950 msgr2=0x7f56680b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.844+0000 7f56707f0640 1 -- 172.21.15.81:0/3259342471 shutdown_connections 2024-06-29T15:59:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.844+0000 7f56707f0640 1 -- 172.21.15.81:0/3259342471 wait complete. 2024-06-29T15:59:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.846+0000 7f56707f0640 1 Processor -- start 2024-06-29T15:59:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.846+0000 7f56707f0640 1 -- start start 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.847+0000 7f56707f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566806f680 0x7f56681f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.847+0000 7f56707f0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f566806fa90 0x7f5668146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.847+0000 7f56707f0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5668074520 0x7f56681f6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.847+0000 7f56707f0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f566815b190 con 0x7f566806f680 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.847+0000 7f56707f0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f566815b010 con 0x7f5668074520 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f56707f0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f56680af310 con 0x7f566806fa90 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566e58c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566806f680 0x7f56681f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566e58c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566806f680 0x7f56681f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50638/0 (socket says 172.21.15.81:50638) 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566e58c640 1 -- 172.21.15.81:0/2932251990 learned_addr learned my addr 172.21.15.81:0/2932251990 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566ed8d640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5668074520 0x7f56681f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566dd8b640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f566806fa90 0x7f5668146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566e58c640 1 -- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f566806fa90 msgr2=0x7f5668146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566e58c640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f566806fa90 0x7f5668146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566e58c640 1 -- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5668074520 msgr2=0x7f56681f6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566e58c640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5668074520 0x7f56681f6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566e58c640 1 -- 172.21.15.81:0/2932251990 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5668150e00 con 0x7f566806f680 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566dd8b640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f566806fa90 0x7f5668146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f566e58c640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566806f680 0x7f56681f3990 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f5664000c00 tx=0x7f5664003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f565effd640 1 -- 172.21.15.81:0/2932251990 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566403e030 con 0x7f566806f680 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f565effd640 1 -- 172.21.15.81:0/2932251990 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5664003e60 con 0x7f566806f680 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f56707f0640 1 -- 172.21.15.81:0/2932251990 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56681510e0 con 0x7f566806f680 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f565effd640 1 -- 172.21.15.81:0/2932251990 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5664035040 con 0x7f566806f680 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.848+0000 7f56707f0640 1 -- 172.21.15.81:0/2932251990 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56681f3470 con 0x7f566806f680 2024-06-29T15:59:29.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.849+0000 7f56707f0640 1 -- 172.21.15.81:0/2932251990 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f566806f140 con 0x7f566806f680 2024-06-29T15:59:29.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.850+0000 7f565effd640 1 -- 172.21.15.81:0/2932251990 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 153) ==== 98039+0+0 (secure 0 0 0) 0x7f5664041030 con 0x7f566806f680 2024-06-29T15:59:29.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.851+0000 7f565effd640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f564c076150 0x7f564c078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:29.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.851+0000 7f565effd640 1 -- 172.21.15.81:0/2932251990 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 6611+0+0 (secure 0 0 0) 0x7f56640b7fb0 con 0x7f566806f680 2024-06-29T15:59:29.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.851+0000 7f566dd8b640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f564c076150 0x7f564c078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:29.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.851+0000 7f566dd8b640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f564c076150 0x7f564c078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5668155740 tx=0x7f5658009040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:29.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:29.855+0000 7f565effd640 1 -- 172.21.15.81:0/2932251990 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5664081830 con 0x7f566806f680 2024-06-29T15:59:30.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.037+0000 7f56707f0640 1 -- 172.21.15.81:0/2932251990 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5668155490 con 0x7f566806f680 2024-06-29T15:59:30.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.038+0000 7f565effd640 1 -- 172.21.15.81:0/2932251990 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) ==== 74+0+17962 (secure 0 0 0) 0x7f5664081130 con 0x7f566806f680 2024-06-29T15:59:30.039 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:30.039 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":255,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:28.179092+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:07.416791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"252","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:08.088694+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/335340229":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/3651252088":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/2009566943":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/604621260":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/2128674418":"2024-06-30T15:59:28.179050+0000","172.21.15.167:6833/1493691973":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/3251388883":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2508797860":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/3445975737":"2024-06-30T15:59:28.179050+0000","172.21.15.167:6832/1493691973":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/3181687202":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1949624986":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2055038349":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1927715461":"2024-06-30T15:59:21.470262+0000","172.21.15.167:6832/1778875525":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2270885245":"2024-06-30T15:59:21.470262+0000","172.21.15.167:6833/1778875525":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1180941913":"2024-06-30T15:59:21.470262+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 -- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f564c076150 msgr2=0x7f564c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f564c076150 0x7f564c078660 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5668155740 tx=0x7f5658009040 comp rx=0 tx=0).stop 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 -- 172.21.15.81:0/2932251990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566806f680 msgr2=0x7f56681f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566806f680 0x7f56681f3990 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f5664000c00 tx=0x7f5664003970 comp rx=0 tx=0).stop 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 -- 172.21.15.81:0/2932251990 shutdown_connections 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f564c076150 0x7f564c078660 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5668074520 0x7f56681f6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f566806fa90 0x7f5668146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 --2- 172.21.15.81:0/2932251990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566806f680 0x7f56681f3990 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 -- 172.21.15.81:0/2932251990 >> 172.21.15.81:0/2932251990 conn(0x7f5668072950 msgr2=0x7f56680b0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 -- 172.21.15.81:0/2932251990 shutdown_connections 2024-06-29T15:59:30.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.040+0000 7f56707f0640 1 -- 172.21.15.81:0/2932251990 wait complete. 2024-06-29T15:59:30.052 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T15:59:30.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.168+0000 7f4f4fa47640 1 -- 172.21.15.81:0/803099531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f34004000 msgr2=0x7f4f34004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.168+0000 7f4f4fa47640 1 --2- 172.21.15.81:0/803099531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f34004000 0x7f4f34004800 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009b50 tx=0x7f4f34005830 comp rx=0 tx=0).stop 2024-06-29T15:59:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.168+0000 7f4f4dfe4640 1 -- 172.21.15.81:0/803099531 reap_dead start 2024-06-29T15:59:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.168+0000 7f4f4fa47640 1 -- 172.21.15.81:0/803099531 shutdown_connections 2024-06-29T15:59:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.168+0000 7f4f4fa47640 1 -- 172.21.15.81:0/803099531 >> 172.21.15.81:0/803099531 conn(0x7f4f480b3680 msgr2=0x7f4f480b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.168+0000 7f4f4fa47640 1 -- 172.21.15.81:0/803099531 shutdown_connections 2024-06-29T15:59:30.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.168+0000 7f4f4fa47640 1 -- 172.21.15.81:0/803099531 wait complete. 2024-06-29T15:59:30.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.170+0000 7f4f4fa47640 1 Processor -- start 2024-06-29T15:59:30.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.170+0000 7f4f4fa47640 1 -- start start 2024-06-29T15:59:30.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4fa47640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f34004000 0x7f4f48156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4fa47640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f48156400 0x7f4f481f1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4fa47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f48159e90 0x7f4f48154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4fa47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4f480d09d0 con 0x7f4f48159e90 2024-06-29T15:59:30.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4fa47640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4f480d0850 con 0x7f4f48156400 2024-06-29T15:59:30.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4fa47640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4f480b5a60 con 0x7f4f34004000 2024-06-29T15:59:30.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4dfe4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f48159e90 0x7f4f48154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4dfe4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f48159e90 0x7f4f48154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46420/0 (socket says 172.21.15.81:46420) 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4dfe4640 1 -- 172.21.15.81:0/162107480 learned_addr learned my addr 172.21.15.81:0/162107480 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4d7e3640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f34004000 0x7f4f48156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.171+0000 7f4f4cfe2640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f48156400 0x7f4f481f1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f4dfe4640 1 -- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f34004000 msgr2=0x7f4f48156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f4dfe4640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f34004000 0x7f4f48156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f4dfe4640 1 -- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f48156400 msgr2=0x7f4f481f1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f4dfe4640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f48156400 0x7f4f481f1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f4dfe4640 1 -- 172.21.15.81:0/162107480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f481f0b10 con 0x7f4f48159e90 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f4dfe4640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f48159e90 0x7f4f48154bf0 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f4f480d1010 tx=0x7f4f400064f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f3dffb640 1 -- 172.21.15.81:0/162107480 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f40011900 con 0x7f4f48159e90 2024-06-29T15:59:30.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f3dffb640 1 -- 172.21.15.81:0/162107480 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f40019070 con 0x7f4f48159e90 2024-06-29T15:59:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f3dffb640 1 -- 172.21.15.81:0/162107480 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f40011d60 con 0x7f4f48159e90 2024-06-29T15:59:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f4fa47640 1 -- 172.21.15.81:0/162107480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f481576e0 con 0x7f4f48159e90 2024-06-29T15:59:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.172+0000 7f4f4fa47640 1 -- 172.21.15.81:0/162107480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f48155650 con 0x7f4f48159e90 2024-06-29T15:59:30.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.173+0000 7f4f4fa47640 1 -- 172.21.15.81:0/162107480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f481558b0 con 0x7f4f48159e90 2024-06-29T15:59:30.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.174+0000 7f4f3dffb640 1 -- 172.21.15.81:0/162107480 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 153) ==== 98039+0+0 (secure 0 0 0) 0x7f4f4001e460 con 0x7f4f48159e90 2024-06-29T15:59:30.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.174+0000 7f4f3dffb640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4f2c076150 0x7f4f2c078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.174+0000 7f4f3dffb640 1 -- 172.21.15.81:0/162107480 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(255..255 src has 1..255) ==== 6611+0+0 (secure 0 0 0) 0x7f4f40097cf0 con 0x7f4f48159e90 2024-06-29T15:59:30.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.174+0000 7f4f4d7e3640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4f2c076150 0x7f4f2c078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.175+0000 7f4f4d7e3640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4f2c076150 0x7f4f2c078660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4f44000c00 tx=0x7f4f44006b70 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:30.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.179+0000 7f4f3dffb640 1 -- 172.21.15.81:0/162107480 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4f480d0b80 con 0x7f4f48159e90 2024-06-29T15:59:30.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.399+0000 7f4f4fa47640 1 -- 172.21.15.81:0/162107480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4f480b5f70 con 0x7f4f48159e90 2024-06-29T15:59:30.400 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T15:59:30.400+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T15:59:30.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.401+0000 7f4f3dffb640 1 -- 172.21.15.81:0/162107480 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v146) ==== 161+0+0 (secure 0 0 0) 0x7f4f40060f30 con 0x7f4f48159e90 2024-06-29T15:59:30.401 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T15:59:30.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 -- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4f2c076150 msgr2=0x7f4f2c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4f2c076150 0x7f4f2c078660 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4f44000c00 tx=0x7f4f44006b70 comp rx=0 tx=0).stop 2024-06-29T15:59:30.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 -- 172.21.15.81:0/162107480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f48159e90 msgr2=0x7f4f48154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f48159e90 0x7f4f48154bf0 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f4f480d1010 tx=0x7f4f400064f0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 -- 172.21.15.81:0/162107480 shutdown_connections 2024-06-29T15:59:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4f2c076150 0x7f4f2c078660 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f48159e90 0x7f4f48154bf0 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4f48156400 0x7f4f481f1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 --2- 172.21.15.81:0/162107480 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4f34004000 0x7f4f48156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 -- 172.21.15.81:0/162107480 >> 172.21.15.81:0/162107480 conn(0x7f4f480b3680 msgr2=0x7f4f480701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 -- 172.21.15.81:0/162107480 shutdown_connections 2024-06-29T15:59:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.404+0000 7f4f4fa47640 1 -- 172.21.15.81:0/162107480 wait complete. 2024-06-29T15:59:30.407 INFO:tasks.ceph.mds.d.smithi167.stderr: -5542> 2024-06-29T15:59:05.143+0000 7ff751916640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-06-29T15:59:30.408 INFO:tasks.ceph.mds.d.smithi167.stderr: -4396> 2024-06-29T15:59:11.736+0000 7ff74b90a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:59:30.415 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T15:59:30.465 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:59:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.533+0000 7fa13e066640 1 -- 172.21.15.81:0/1122719646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1380d0480 msgr2=0x7fa1380b8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.533+0000 7fa13e066640 1 --2- 172.21.15.81:0/1122719646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1380d0480 0x7fa1380b8e80 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7fa138161ba0 tx=0x7fa12802a560 comp rx=0 tx=0).stop 2024-06-29T15:59:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.533+0000 7fa13e066640 1 -- 172.21.15.81:0/1122719646 shutdown_connections 2024-06-29T15:59:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.533+0000 7fa13e066640 1 --2- 172.21.15.81:0/1122719646 >> v2:172.21.15.167:3300/0 conn(0x7fa1380cf1b0 0x7fa1380cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.533+0000 7fa13e066640 1 --2- 172.21.15.81:0/1122719646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1380d0480 0x7fa1380b8e80 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.533+0000 7fa13e066640 1 -- 172.21.15.81:0/1122719646 >> 172.21.15.81:0/1122719646 conn(0x7fa1380d3b70 msgr2=0x7fa1380d3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:30.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.533+0000 7fa13e066640 1 -- 172.21.15.81:0/1122719646 shutdown_connections 2024-06-29T15:59:30.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.533+0000 7fa13e066640 1 -- 172.21.15.81:0/1122719646 wait complete. 2024-06-29T15:59:30.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.535+0000 7fa13e066640 1 Processor -- start 2024-06-29T15:59:30.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.535+0000 7fa13e066640 1 -- start start 2024-06-29T15:59:30.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa13e066640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1380cf1b0 0x7fa1380c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa13e066640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1380d0480 0x7fa13815b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa13e066640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1380d2270 0x7fa1380c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa13e066640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa1380cfb00 con 0x7fa1380cf1b0 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa13e066640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa1380cfc80 con 0x7fa1380d0480 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa13e066640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa1380cfe00 con 0x7fa1380d2270 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa1377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1380cf1b0 0x7fa1380c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa1377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1380cf1b0 0x7fa1380c83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46450/0 (socket says 172.21.15.81:46450) 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa1377fe640 1 -- 172.21.15.81:0/2484347564 learned_addr learned my addr 172.21.15.81:0/2484347564 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa136ffd640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1380d0480 0x7fa13815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa137fff640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1380d2270 0x7fa1380c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa1377fe640 1 -- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1380d2270 msgr2=0x7fa1380c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa1377fe640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1380d2270 0x7fa1380c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa1377fe640 1 -- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1380d0480 msgr2=0x7fa13815b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:30.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa1377fe640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1380d0480 0x7fa13815b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:30.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa1377fe640 1 -- 172.21.15.81:0/2484347564 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1380c91c0 con 0x7fa1380cf1b0 2024-06-29T15:59:30.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa1377fe640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1380cf1b0 0x7fa1380c83b0 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7fa1280099d0 tx=0x7fa1280052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:30.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.537+0000 7fa11ffff640 1 -- 172.21.15.81:0/2484347564 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa128031b50 con 0x7fa1380cf1b0 2024-06-29T15:59:30.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.538+0000 7fa13e066640 1 -- 172.21.15.81:0/2484347564 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1380c8c00 con 0x7fa1380cf1b0 2024-06-29T15:59:30.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.538+0000 7fa11ffff640 1 -- 172.21.15.81:0/2484347564 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa128031de0 con 0x7fa1380cf1b0 2024-06-29T15:59:30.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.538+0000 7fa11ffff640 1 -- 172.21.15.81:0/2484347564 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa128033330 con 0x7fa1380cf1b0 2024-06-29T15:59:30.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.538+0000 7fa13e066640 1 -- 172.21.15.81:0/2484347564 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1380cb2e0 con 0x7fa1380cf1b0 2024-06-29T15:59:30.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.539+0000 7fa13e066640 1 -- 172.21.15.81:0/2484347564 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1380cf5c0 con 0x7fa1380cf1b0 2024-06-29T15:59:30.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.540+0000 7fa11ffff640 1 -- 172.21.15.81:0/2484347564 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 154) ==== 98247+0+0 (secure 0 0 0) 0x7fa1280334d0 con 0x7fa1380cf1b0 2024-06-29T15:59:30.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.540+0000 7fa11ffff640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa124076390 0x7fa1240788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:30.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.540+0000 7fa11ffff640 1 -- 172.21.15.81:0/2484347564 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 6697+0+0 (secure 0 0 0) 0x7fa1280b8ff0 con 0x7fa1380cf1b0 2024-06-29T15:59:30.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.540+0000 7fa136ffd640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa124076390 0x7fa1240788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:30.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.541+0000 7fa136ffd640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa124076390 0x7fa1240788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa13806e080 tx=0x7fa12c009040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:30.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.545+0000 7fa11ffff640 1 -- 172.21.15.81:0/2484347564 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa12803f030 con 0x7fa1380cf1b0 2024-06-29T15:59:30.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:30.768+0000 7fa13e066640 1 -- 172.21.15.81:0/2484347564 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa1380b5370 con 0x7fa1380cf1b0 2024-06-29T15:59:31.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.403+0000 7fa11ffff640 1 -- 172.21.15.81:0/2484347564 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v147) ==== 104+0+0 (secure 0 0 0) 0x7fa128031770 con 0x7fa1380cf1b0 2024-06-29T15:59:31.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.405+0000 7fa13e066640 1 -- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa124076390 msgr2=0x7fa1240788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.405+0000 7fa13e066640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa124076390 0x7fa1240788a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa13806e080 tx=0x7fa12c009040 comp rx=0 tx=0).stop 2024-06-29T15:59:31.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.405+0000 7fa13e066640 1 -- 172.21.15.81:0/2484347564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1380cf1b0 msgr2=0x7fa1380c83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.405+0000 7fa13e066640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1380cf1b0 0x7fa1380c83b0 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7fa1280099d0 tx=0x7fa1280052f0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.405+0000 7fa13e066640 1 -- 172.21.15.81:0/2484347564 shutdown_connections 2024-06-29T15:59:31.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.405+0000 7fa13e066640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa124076390 0x7fa1240788a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.405+0000 7fa13e066640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa1380d2270 0x7fa1380c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.405+0000 7fa13e066640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa1380d0480 0x7fa13815b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.405+0000 7fa13e066640 1 --2- 172.21.15.81:0/2484347564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1380cf1b0 0x7fa1380c83b0 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.405+0000 7fa13e066640 1 -- 172.21.15.81:0/2484347564 >> 172.21.15.81:0/2484347564 conn(0x7fa1380d3b70 msgr2=0x7fa1380d1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:31.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.406+0000 7fa13e066640 1 -- 172.21.15.81:0/2484347564 shutdown_connections 2024-06-29T15:59:31.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.406+0000 7fa13e066640 1 -- 172.21.15.81:0/2484347564 wait complete. 2024-06-29T15:59:31.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T15:59:31.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.541+0000 7fa82cec3640 1 -- 172.21.15.81:0/2356969902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8280d2440 msgr2=0x7fa8280d4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.541+0000 7fa82cec3640 1 --2- 172.21.15.81:0/2356969902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8280d2440 0x7fa8280d4770 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa818009860 tx=0x7fa81802a510 comp rx=0 tx=0).stop 2024-06-29T15:59:31.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.541+0000 7fa82cec3640 1 -- 172.21.15.81:0/2356969902 shutdown_connections 2024-06-29T15:59:31.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.541+0000 7fa82cec3640 1 --2- 172.21.15.81:0/2356969902 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8280d2440 0x7fa8280d4770 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.541+0000 7fa82cec3640 1 -- 172.21.15.81:0/2356969902 >> 172.21.15.81:0/2356969902 conn(0x7fa8280d3c50 msgr2=0x7fa8280d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:31.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.541+0000 7fa82cec3640 1 -- 172.21.15.81:0/2356969902 shutdown_connections 2024-06-29T15:59:31.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.541+0000 7fa82cec3640 1 -- 172.21.15.81:0/2356969902 wait complete. 2024-06-29T15:59:31.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.543+0000 7fa82cec3640 1 Processor -- start 2024-06-29T15:59:31.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.543+0000 7fa82cec3640 1 -- start start 2024-06-29T15:59:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82cec3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8280b2ea0 0x7fa8280c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82cec3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8280d03b0 0x7fa8281582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82cec3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8280d2440 0x7fa8280cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82cec3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa82806ccd0 con 0x7fa8280b2ea0 2024-06-29T15:59:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82cec3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa82806cb50 con 0x7fa8280d2440 2024-06-29T15:59:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82cec3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa82806ce50 con 0x7fa8280d03b0 2024-06-29T15:59:31.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8280b2ea0 0x7fa8280c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8280b2ea0 0x7fa8280c8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46466/0 (socket says 172.21.15.81:46466) 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82659c640 1 -- 172.21.15.81:0/4161586925 learned_addr learned my addr 172.21.15.81:0/4161586925 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa826d9d640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8280d2440 0x7fa8280cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa81ffff640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8280d03b0 0x7fa8281582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82659c640 1 -- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8280d03b0 msgr2=0x7fa8281582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82659c640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8280d03b0 0x7fa8281582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82659c640 1 -- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8280d2440 msgr2=0x7fa8280cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82659c640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8280d2440 0x7fa8280cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82659c640 1 -- 172.21.15.81:0/4161586925 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8280ca8e0 con 0x7fa8280b2ea0 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa82659c640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8280b2ea0 0x7fa8280c8e60 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7fa818009980 tx=0x7fa818033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa81effd640 1 -- 172.21.15.81:0/4161586925 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818031d60 con 0x7fa8280b2ea0 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.545+0000 7fa81effd640 1 -- 172.21.15.81:0/4161586925 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa818005020 con 0x7fa8280b2ea0 2024-06-29T15:59:31.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.546+0000 7fa82cec3640 1 -- 172.21.15.81:0/4161586925 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa828158800 con 0x7fa8280b2ea0 2024-06-29T15:59:31.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.546+0000 7fa81effd640 1 -- 172.21.15.81:0/4161586925 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818005470 con 0x7fa8280b2ea0 2024-06-29T15:59:31.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.546+0000 7fa82cec3640 1 -- 172.21.15.81:0/4161586925 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8280c9b20 con 0x7fa8280b2ea0 2024-06-29T15:59:31.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.546+0000 7fa82cec3640 1 -- 172.21.15.81:0/4161586925 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa82806d300 con 0x7fa8280b2ea0 2024-06-29T15:59:31.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.547+0000 7fa81effd640 1 -- 172.21.15.81:0/4161586925 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 154) ==== 98247+0+0 (secure 0 0 0) 0x7fa8180047a0 con 0x7fa8280b2ea0 2024-06-29T15:59:31.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.548+0000 7fa81effd640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa808076360 0x7fa808078870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:31.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.548+0000 7fa81effd640 1 -- 172.21.15.81:0/4161586925 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 6697+0+0 (secure 0 0 0) 0x7fa8180b8e40 con 0x7fa8280b2ea0 2024-06-29T15:59:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.548+0000 7fa81ffff640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa808076360 0x7fa808078870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.549+0000 7fa81ffff640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa808076360 0x7fa808078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa8280c8b30 tx=0x7fa810007eb0 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:31.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.552+0000 7fa81effd640 1 -- 172.21.15.81:0/4161586925 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fa8280ca8e0 con 0x7fa8280b2ea0 2024-06-29T15:59:31.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:31.733+0000 7fa82cec3640 1 -- 172.21.15.81:0/4161586925 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa8280d14a0 con 0x7fa8280b2ea0 2024-06-29T15:59:32.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.543+0000 7fa81effd640 1 -- 172.21.15.81:0/4161586925 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v257) ==== 181+0+0 (secure 0 0 0) 0x7fa81803f030 con 0x7fa8280b2ea0 2024-06-29T15:59:32.544 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T15:59:32.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.547+0000 7fa82cec3640 1 -- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa808076360 msgr2=0x7fa808078870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.547+0000 7fa82cec3640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa808076360 0x7fa808078870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa8280c8b30 tx=0x7fa810007eb0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.547+0000 7fa82cec3640 1 -- 172.21.15.81:0/4161586925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8280b2ea0 msgr2=0x7fa8280c8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.547+0000 7fa82cec3640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8280b2ea0 0x7fa8280c8e60 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7fa818009980 tx=0x7fa818033c90 comp rx=0 tx=0).stop 2024-06-29T15:59:32.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.548+0000 7fa82cec3640 1 -- 172.21.15.81:0/4161586925 shutdown_connections 2024-06-29T15:59:32.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.548+0000 7fa82cec3640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fa808076360 0x7fa808078870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.548+0000 7fa82cec3640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa8280d2440 0x7fa8280cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.548+0000 7fa82cec3640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa8280d03b0 0x7fa8281582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.548+0000 7fa82cec3640 1 --2- 172.21.15.81:0/4161586925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8280b2ea0 0x7fa8280c8e60 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.548+0000 7fa82cec3640 1 -- 172.21.15.81:0/4161586925 >> 172.21.15.81:0/4161586925 conn(0x7fa8280d3c50 msgr2=0x7fa8280b8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:32.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.549+0000 7fa82cec3640 1 -- 172.21.15.81:0/4161586925 shutdown_connections 2024-06-29T15:59:32.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.550+0000 7fa82cec3640 1 -- 172.21.15.81:0/4161586925 wait complete. 2024-06-29T15:59:32.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T15:59:32.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.687+0000 7f266a9bd640 1 -- 172.21.15.81:0/2420069043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f264c003ff0 msgr2=0x7f264c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.687+0000 7f266a9bd640 1 --2- 172.21.15.81:0/2420069043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f264c003ff0 0x7f264c004800 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f266415ea10 tx=0x7f264c005830 comp rx=0 tx=0).stop 2024-06-29T15:59:32.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.687+0000 7f2668f5a640 1 -- 172.21.15.81:0/2420069043 reap_dead start 2024-06-29T15:59:32.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.687+0000 7f266a9bd640 1 -- 172.21.15.81:0/2420069043 shutdown_connections 2024-06-29T15:59:32.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.687+0000 7f266a9bd640 1 -- 172.21.15.81:0/2420069043 >> 172.21.15.81:0/2420069043 conn(0x7f26640b1da0 msgr2=0x7f26640b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:32.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.688+0000 7f266a9bd640 1 -- 172.21.15.81:0/2420069043 shutdown_connections 2024-06-29T15:59:32.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.688+0000 7f266a9bd640 1 -- 172.21.15.81:0/2420069043 wait complete. 2024-06-29T15:59:32.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.689+0000 7f266a9bd640 1 Processor -- start 2024-06-29T15:59:32.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.690+0000 7f266a9bd640 1 -- start start 2024-06-29T15:59:32.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f266a9bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f264c003ff0 0x7f266415b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f266a9bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f266414c8f0 0x7f266414c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f266a9bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26640d0140 0x7f26640d0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f266a9bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f266415ec30 con 0x7f26640d0140 2024-06-29T15:59:32.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f266a9bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f266415eab0 con 0x7f264c003ff0 2024-06-29T15:59:32.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f266a9bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f266415edb0 con 0x7f266414c8f0 2024-06-29T15:59:32.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2668f5a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26640d0140 0x7f26640d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2668f5a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26640d0140 0x7f26640d0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46492/0 (socket says 172.21.15.81:46492) 2024-06-29T15:59:32.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2668f5a640 1 -- 172.21.15.81:0/3085093883 learned_addr learned my addr 172.21.15.81:0/3085093883 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f26637fe640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f266414c8f0 0x7f266414c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2663fff640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f264c003ff0 0x7f266415b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2668f5a640 1 -- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f266414c8f0 msgr2=0x7f266414c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2668f5a640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f266414c8f0 0x7f266414c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2668f5a640 1 -- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f264c003ff0 msgr2=0x7f266415b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2668f5a640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f264c003ff0 0x7f266415b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2668f5a640 1 -- 172.21.15.81:0/3085093883 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26640d0a90 con 0x7f26640d0140 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f26637fe640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f266414c8f0 0x7f266414c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2663fff640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f264c003ff0 0x7f266415b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2668f5a640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26640d0140 0x7f26640d0550 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7f2664074750 tx=0x7f265c010e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.691+0000 7f2660ff9640 1 -- 172.21.15.81:0/3085093883 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265c003e40 con 0x7f26640d0140 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.692+0000 7f2660ff9640 1 -- 172.21.15.81:0/3085093883 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f265c015070 con 0x7f26640d0140 2024-06-29T15:59:32.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.692+0000 7f266a9bd640 1 -- 172.21.15.81:0/3085093883 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26641f8ca0 con 0x7f26640d0140 2024-06-29T15:59:32.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.692+0000 7f2660ff9640 1 -- 172.21.15.81:0/3085093883 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265c00e8c0 con 0x7f26640d0140 2024-06-29T15:59:32.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.692+0000 7f266a9bd640 1 -- 172.21.15.81:0/3085093883 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f266415d360 con 0x7f26640d0140 2024-06-29T15:59:32.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.693+0000 7f266a9bd640 1 -- 172.21.15.81:0/3085093883 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f266415efc0 con 0x7f26640d0140 2024-06-29T15:59:32.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.693+0000 7f2660ff9640 1 -- 172.21.15.81:0/3085093883 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f265c00ea60 con 0x7f26640d0140 2024-06-29T15:59:32.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.694+0000 7f2660ff9640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f26500763f0 0x7f2650078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:32.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.694+0000 7f2660ff9640 1 -- 172.21.15.81:0/3085093883 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(257..257 src has 1..257) ==== 6259+0+0 (secure 0 0 0) 0x7f265c098390 con 0x7f26640d0140 2024-06-29T15:59:32.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.694+0000 7f2663fff640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f26500763f0 0x7f2650078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:32.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.695+0000 7f2663fff640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f26500763f0 0x7f2650078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2654001860 tx=0x7f2654000ed0 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:32.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.698+0000 7f2660ff9640 1 -- 172.21.15.81:0/3085093883 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f265c020020 con 0x7f26640d0140 2024-06-29T15:59:32.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:32.878+0000 7f266a9bd640 1 -- 172.21.15.81:0/3085093883 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f26640b2460 con 0x7f26640d0140 2024-06-29T15:59:33.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.544+0000 7f2660ff9640 1 -- 172.21.15.81:0/3085093883 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v258) ==== 169+0+0 (secure 0 0 0) 0x7f26640d0a90 con 0x7f26640d0140 2024-06-29T15:59:33.545 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T15:59:33.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 -- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f26500763f0 msgr2=0x7f2650078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f26500763f0 0x7f2650078900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2654001860 tx=0x7f2654000ed0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 -- 172.21.15.81:0/3085093883 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26640d0140 msgr2=0x7f26640d0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26640d0140 0x7f26640d0550 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7f2664074750 tx=0x7f265c010e50 comp rx=0 tx=0).stop 2024-06-29T15:59:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 -- 172.21.15.81:0/3085093883 shutdown_connections 2024-06-29T15:59:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f26500763f0 0x7f2650078900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26640d0140 0x7f26640d0550 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f266414c8f0 0x7f266414c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 --2- 172.21.15.81:0/3085093883 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f264c003ff0 0x7f266415b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 -- 172.21.15.81:0/3085093883 >> 172.21.15.81:0/3085093883 conn(0x7f26640b1da0 msgr2=0x7f26640afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 -- 172.21.15.81:0/3085093883 shutdown_connections 2024-06-29T15:59:33.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.547+0000 7f266a9bd640 1 -- 172.21.15.81:0/3085093883 wait complete. 2024-06-29T15:59:33.561 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T15:59:33.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.682+0000 7fd1e844d640 1 -- 172.21.15.81:0/2599453546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e006f6a0 msgr2=0x7fd1e015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.682+0000 7fd1e844d640 1 --2- 172.21.15.81:0/2599453546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e006f6a0 0x7fd1e015aed0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4005fd0 tx=0x7fd1d402a5d0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.682+0000 7fd1e844d640 1 -- 172.21.15.81:0/2599453546 shutdown_connections 2024-06-29T15:59:33.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.682+0000 7fd1e844d640 1 --2- 172.21.15.81:0/2599453546 >> v2:172.21.15.167:3300/0 conn(0x7fd1e00b3960 0x7fd1e00b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.682+0000 7fd1e844d640 1 --2- 172.21.15.81:0/2599453546 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e006f6a0 0x7fd1e015aed0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.682+0000 7fd1e844d640 1 -- 172.21.15.81:0/2599453546 >> 172.21.15.81:0/2599453546 conn(0x7fd1e00d0930 msgr2=0x7fd1e00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:33.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.682+0000 7fd1e844d640 1 -- 172.21.15.81:0/2599453546 shutdown_connections 2024-06-29T15:59:33.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.682+0000 7fd1e844d640 1 -- 172.21.15.81:0/2599453546 wait complete. 2024-06-29T15:59:33.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.684+0000 7fd1e844d640 1 Processor -- start 2024-06-29T15:59:33.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.684+0000 7fd1e844d640 1 -- start start 2024-06-29T15:59:33.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.685+0000 7fd1e844d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e006f6a0 0x7fd1e00abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.685+0000 7fd1e844d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3960 0x7fd1e0073110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.685+0000 7fd1e844d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e00b57b0 0x7fd1e00ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.685+0000 7fd1e844d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00b4640 con 0x7fd1e00b3960 2024-06-29T15:59:33.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.685+0000 7fd1e844d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00b44c0 con 0x7fd1e00b57b0 2024-06-29T15:59:33.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.685+0000 7fd1e844d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e00b47c0 con 0x7fd1e006f6a0 2024-06-29T15:59:33.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.685+0000 7fd1e59e8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3960 0x7fd1e0073110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e59e8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3960 0x7fd1e0073110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46496/0 (socket says 172.21.15.81:46496) 2024-06-29T15:59:33.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e59e8640 1 -- 172.21.15.81:0/3913298886 learned_addr learned my addr 172.21.15.81:0/3913298886 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:33.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e69ea640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e00b57b0 0x7fd1e00ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e61e9640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e006f6a0 0x7fd1e00abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e59e8640 1 -- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e006f6a0 msgr2=0x7fd1e00abf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e59e8640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e006f6a0 0x7fd1e00abf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e59e8640 1 -- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e00b57b0 msgr2=0x7fd1e00ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e59e8640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e00b57b0 0x7fd1e00ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e59e8640 1 -- 172.21.15.81:0/3913298886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e00ac880 con 0x7fd1e00b3960 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e59e8640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3960 0x7fd1e0073110 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4000c00 tx=0x7fd1d40035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1deffd640 1 -- 172.21.15.81:0/3913298886 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4035040 con 0x7fd1e00b3960 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1deffd640 1 -- 172.21.15.81:0/3913298886 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1d40039e0 con 0x7fd1e00b3960 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e844d640 1 -- 172.21.15.81:0/3913298886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e00acb60 con 0x7fd1e00b3960 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1deffd640 1 -- 172.21.15.81:0/3913298886 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4003e30 con 0x7fd1e00b3960 2024-06-29T15:59:33.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.686+0000 7fd1e844d640 1 -- 172.21.15.81:0/3913298886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e00ad120 con 0x7fd1e00b3960 2024-06-29T15:59:33.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.687+0000 7fd1e844d640 1 -- 172.21.15.81:0/3913298886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e00ac5f0 con 0x7fd1e00b3960 2024-06-29T15:59:33.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.688+0000 7fd1deffd640 1 -- 172.21.15.81:0/3913298886 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fd1d40047a0 con 0x7fd1e00b3960 2024-06-29T15:59:33.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.688+0000 7fd1deffd640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd1cc0763f0 0x7fd1cc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:33.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.689+0000 7fd1deffd640 1 -- 172.21.15.81:0/3913298886 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5845+0+0 (secure 0 0 0) 0x7fd1d40ba140 con 0x7fd1e00b3960 2024-06-29T15:59:33.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.689+0000 7fd1e61e9640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd1cc0763f0 0x7fd1cc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:33.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.689+0000 7fd1e61e9640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd1cc0763f0 0x7fd1cc078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0001720 tx=0x7fd1d0009040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:33.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.693+0000 7fd1deffd640 1 -- 172.21.15.81:0/3913298886 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd1d4083220 con 0x7fd1e00b3960 2024-06-29T15:59:33.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.874+0000 7fd1e844d640 1 -- 172.21.15.81:0/3913298886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7fd1e006e2a0 con 0x7fd1e00b3960 2024-06-29T15:59:33.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.879+0000 7fd1deffd640 1 -- 172.21.15.81:0/3913298886 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v83) ==== 71+0+0 (secure 0 0 0) 0x7fd1e006e2a0 con 0x7fd1e00b3960 2024-06-29T15:59:33.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 -- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd1cc0763f0 msgr2=0x7fd1cc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd1cc0763f0 0x7fd1cc078900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0001720 tx=0x7fd1d0009040 comp rx=0 tx=0).stop 2024-06-29T15:59:33.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 -- 172.21.15.81:0/3913298886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3960 msgr2=0x7fd1e0073110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:33.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3960 0x7fd1e0073110 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4000c00 tx=0x7fd1d40035f0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 -- 172.21.15.81:0/3913298886 shutdown_connections 2024-06-29T15:59:33.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd1cc0763f0 0x7fd1cc078900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e00b57b0 0x7fd1e00ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1e00b3960 0x7fd1e0073110 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 --2- 172.21.15.81:0/3913298886 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e006f6a0 0x7fd1e00abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:33.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 -- 172.21.15.81:0/3913298886 >> 172.21.15.81:0/3913298886 conn(0x7fd1e00d0930 msgr2=0x7fd1e00cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:33.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 -- 172.21.15.81:0/3913298886 shutdown_connections 2024-06-29T15:59:33.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:33.881+0000 7fd1e844d640 1 -- 172.21.15.81:0/3913298886 wait complete. 2024-06-29T15:59:33.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2024-06-29T15:59:34.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.007+0000 7fb202731640 1 -- 172.21.15.81:0/1202933417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e4003a50 msgr2=0x7fb1e4003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.007+0000 7fb202731640 1 --2- 172.21.15.81:0/1202933417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e4003a50 0x7fb1e4003ee0 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec009b50 tx=0x7fb1e4005830 comp rx=0 tx=0).stop 2024-06-29T15:59:34.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.007+0000 7fb200cce640 1 -- 172.21.15.81:0/1202933417 reap_dead start 2024-06-29T15:59:34.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.007+0000 7fb202731640 1 -- 172.21.15.81:0/1202933417 shutdown_connections 2024-06-29T15:59:34.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.007+0000 7fb202731640 1 -- 172.21.15.81:0/1202933417 >> 172.21.15.81:0/1202933417 conn(0x7fb1fc0b3510 msgr2=0x7fb1fc0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:34.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.007+0000 7fb202731640 1 -- 172.21.15.81:0/1202933417 shutdown_connections 2024-06-29T15:59:34.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.007+0000 7fb202731640 1 -- 172.21.15.81:0/1202933417 wait complete. 2024-06-29T15:59:34.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.009+0000 7fb202731640 1 Processor -- start 2024-06-29T15:59:34.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.009+0000 7fb202731640 1 -- start start 2024-06-29T15:59:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb202731640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1e4003a50 0x7fb1fc1e7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb202731640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1fc1e5d20 0x7fb1fc1d6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb202731640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc0b5550 0x7fb1fc0b5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb202731640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc0ce920 con 0x7fb1fc0b5550 2024-06-29T15:59:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb202731640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc0b29c0 con 0x7fb1fc1e5d20 2024-06-29T15:59:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb202731640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb1fc0ceaa0 con 0x7fb1e4003a50 2024-06-29T15:59:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb200cce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc0b5550 0x7fb1fc0b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb200cce640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc0b5550 0x7fb1fc0b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46506/0 (socket says 172.21.15.81:46506) 2024-06-29T15:59:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb200cce640 1 -- 172.21.15.81:0/3956137350 learned_addr learned my addr 172.21.15.81:0/3956137350 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb1fbfff640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1e4003a50 0x7fb1fc1e7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb1fb7fe640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1fc1e5d20 0x7fb1fc1d6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb200cce640 1 -- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1e4003a50 msgr2=0x7fb1fc1e7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb200cce640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1e4003a50 0x7fb1fc1e7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb200cce640 1 -- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1fc1e5d20 msgr2=0x7fb1fc1d6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb200cce640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1fc1e5d20 0x7fb1fc1d6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb200cce640 1 -- 172.21.15.81:0/3956137350 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc0b5ea0 con 0x7fb1fc0b5550 2024-06-29T15:59:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb200cce640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc0b5550 0x7fb1fc0b5960 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4001350 tx=0x7fb1f400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb1f8ff9640 1 -- 172.21.15.81:0/3956137350 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f400cba0 con 0x7fb1fc0b5550 2024-06-29T15:59:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb1f8ff9640 1 -- 172.21.15.81:0/3956137350 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1f4018070 con 0x7fb1fc0b5550 2024-06-29T15:59:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb202731640 1 -- 172.21.15.81:0/3956137350 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1fc1f8bc0 con 0x7fb1fc0b5550 2024-06-29T15:59:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb1f8ff9640 1 -- 172.21.15.81:0/3956137350 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f40116c0 con 0x7fb1fc0b5550 2024-06-29T15:59:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.011+0000 7fb202731640 1 -- 172.21.15.81:0/3956137350 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1fc1f9180 con 0x7fb1fc0b5550 2024-06-29T15:59:34.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.012+0000 7fb202731640 1 -- 172.21.15.81:0/3956137350 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1fc0cfca0 con 0x7fb1fc0b5550 2024-06-29T15:59:34.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.013+0000 7fb1f8ff9640 1 -- 172.21.15.81:0/3956137350 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fb1f4011860 con 0x7fb1fc0b5550 2024-06-29T15:59:34.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.013+0000 7fb1f8ff9640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb1e80763f0 0x7fb1e8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.014+0000 7fb1f8ff9640 1 -- 172.21.15.81:0/3956137350 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5845+0+0 (secure 0 0 0) 0x7fb1f4097e30 con 0x7fb1fc0b5550 2024-06-29T15:59:34.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.014+0000 7fb1fbfff640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb1e80763f0 0x7fb1e8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.014+0000 7fb1fbfff640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb1e80763f0 0x7fb1e8078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec035000 tx=0x7fb1ec02d040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:34.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.018+0000 7fb1f8ff9640 1 -- 172.21.15.81:0/3956137350 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb1fc0cfca0 con 0x7fb1fc0b5550 2024-06-29T15:59:34.055 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:59:34.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T15:59:34.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.272+0000 7fb202731640 1 -- 172.21.15.81:0/3956137350 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) -- 0x7fb1fc0cd5f0 con 0x7fb1fc0b5550 2024-06-29T15:59:34.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.557+0000 7fb1f8ff9640 1 -- 172.21.15.81:0/3956137350 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3197) ==== 155+0+0 (secure 0 0 0) 0x7fb1fc0b5ea0 con 0x7fb1fc0b5550 2024-06-29T15:59:34.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 -- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb1e80763f0 msgr2=0x7fb1e8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb1e80763f0 0x7fb1e8078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec035000 tx=0x7fb1ec02d040 comp rx=0 tx=0).stop 2024-06-29T15:59:34.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 -- 172.21.15.81:0/3956137350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc0b5550 msgr2=0x7fb1fc0b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc0b5550 0x7fb1fc0b5960 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4001350 tx=0x7fb1f400ed30 comp rx=0 tx=0).stop 2024-06-29T15:59:34.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 -- 172.21.15.81:0/3956137350 shutdown_connections 2024-06-29T15:59:34.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb1e80763f0 0x7fb1e8078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc0b5550 0x7fb1fc0b5960 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1fc1e5d20 0x7fb1fc1d6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 --2- 172.21.15.81:0/3956137350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1e4003a50 0x7fb1fc1e7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 -- 172.21.15.81:0/3956137350 >> 172.21.15.81:0/3956137350 conn(0x7fb1fc0b3510 msgr2=0x7fb1fc1e3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:34.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 -- 172.21.15.81:0/3956137350 shutdown_connections 2024-06-29T15:59:34.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.560+0000 7fb202731640 1 -- 172.21.15.81:0/3956137350 wait complete. 2024-06-29T15:59:34.572 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2024-06-29T15:59:34.572 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) Test subvol schedule creation succeeds for default subvolgroup. 2024-06-29T15:59:34.572 INFO:tasks.cephfs_test_runner:test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-06-29T15:59:34.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:34.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.686+0000 7fb31a38c640 1 -- 172.21.15.81:0/3305120131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb31406f680 msgr2=0x7fb31406c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.686+0000 7fb31a38c640 1 --2- 172.21.15.81:0/3305120131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb31406f680 0x7fb31406c2f0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7fb314072d60 tx=0x7fb30802a560 comp rx=0 tx=0).stop 2024-06-29T15:59:34.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.686+0000 7fb31a38c640 1 -- 172.21.15.81:0/3305120131 shutdown_connections 2024-06-29T15:59:34.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.686+0000 7fb31a38c640 1 --2- 172.21.15.81:0/3305120131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb31406f680 0x7fb31406c2f0 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.686+0000 7fb31a38c640 1 --2- 172.21.15.81:0/3305120131 >> v2:172.21.15.167:3300/0 conn(0x7fb314074520 0x7fb31406f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.686+0000 7fb31a38c640 1 -- 172.21.15.81:0/3305120131 >> 172.21.15.81:0/3305120131 conn(0x7fb314072950 msgr2=0x7fb3140b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:34.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.686+0000 7fb31a38c640 1 -- 172.21.15.81:0/3305120131 shutdown_connections 2024-06-29T15:59:34.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.686+0000 7fb31a38c640 1 -- 172.21.15.81:0/3305120131 wait complete. 2024-06-29T15:59:34.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.687+0000 7fb31a38c640 1 Processor -- start 2024-06-29T15:59:34.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.688+0000 7fb31a38c640 1 -- start start 2024-06-29T15:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb31a38c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb31406f680 0x7fb3141516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb31a38c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb31406fa90 0x7fb31409a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb31a38c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb314074520 0x7fb314159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb31a38c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb31415b010 con 0x7fb31406fa90 2024-06-29T15:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb31a38c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb31406d150 con 0x7fb31406f680 2024-06-29T15:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb31a38c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb31415b190 con 0x7fb314074520 2024-06-29T15:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb3137fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb31406fa90 0x7fb31409a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb3137fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb31406fa90 0x7fb31409a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46528/0 (socket says 172.21.15.81:46528) 2024-06-29T15:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb3137fe640 1 -- 172.21.15.81:0/452538078 learned_addr learned my addr 172.21.15.81:0/452538078 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:34.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb313fff640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb31406f680 0x7fb3141516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb318929640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb314074520 0x7fb314159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb3137fe640 1 -- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb314074520 msgr2=0x7fb314159b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb3137fe640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb314074520 0x7fb314159b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb3137fe640 1 -- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb31406f680 msgr2=0x7fb3141516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb3137fe640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb31406f680 0x7fb3141516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.689+0000 7fb3137fe640 1 -- 172.21.15.81:0/452538078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31409d3e0 con 0x7fb31406fa90 2024-06-29T15:59:34.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.690+0000 7fb3137fe640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb31406fa90 0x7fb31409a470 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7fb308034040 tx=0x7fb3080039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:34.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.690+0000 7fb310ff9640 1 -- 172.21.15.81:0/452538078 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30803e030 con 0x7fb31406fa90 2024-06-29T15:59:34.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.690+0000 7fb310ff9640 1 -- 172.21.15.81:0/452538078 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3080047a0 con 0x7fb31406fa90 2024-06-29T15:59:34.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.690+0000 7fb31a38c640 1 -- 172.21.15.81:0/452538078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb31409d6c0 con 0x7fb31406fa90 2024-06-29T15:59:34.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.690+0000 7fb310ff9640 1 -- 172.21.15.81:0/452538078 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb308035040 con 0x7fb31406fa90 2024-06-29T15:59:34.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.690+0000 7fb31a38c640 1 -- 172.21.15.81:0/452538078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb31409dc80 con 0x7fb31406fa90 2024-06-29T15:59:34.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.691+0000 7fb31a38c640 1 -- 172.21.15.81:0/452538078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb31409d150 con 0x7fb31406fa90 2024-06-29T15:59:34.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.691+0000 7fb310ff9640 1 -- 172.21.15.81:0/452538078 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fb308041030 con 0x7fb31406fa90 2024-06-29T15:59:34.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.692+0000 7fb310ff9640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb300076420 0x7fb300078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:34.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.692+0000 7fb310ff9640 1 -- 172.21.15.81:0/452538078 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5845+0+0 (secure 0 0 0) 0x7fb3080b8260 con 0x7fb31406fa90 2024-06-29T15:59:34.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.692+0000 7fb313fff640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb300076420 0x7fb300078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:34.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.693+0000 7fb313fff640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb300076420 0x7fb300078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb304005d70 tx=0x7fb304005d00 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:34.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.696+0000 7fb310ff9640 1 -- 172.21.15.81:0/452538078 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb308081910 con 0x7fb31406fa90 2024-06-29T15:59:34.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.881+0000 7fb31a38c640 1 -- 172.21.15.81:0/452538078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3140af4f0 con 0x7fb31406fa90 2024-06-29T15:59:34.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.882+0000 7fb310ff9640 1 -- 172.21.15.81:0/452538078 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v258) ==== 74+0+14364 (secure 0 0 0) 0x7fb308081210 con 0x7fb31406fa90 2024-06-29T15:59:34.883 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:34.883 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":258,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:33.543920+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6837/54677014":"2024-06-30T15:59:30.400529+0000","172.21.15.167:6835/54677014":"2024-06-30T15:59:30.400529+0000","172.21.15.167:0/335340229":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/3651252088":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/2009566943":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/604621260":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/2128674418":"2024-06-30T15:59:28.179050+0000","172.21.15.167:6833/1493691973":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/3251388883":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2508797860":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/3445975737":"2024-06-30T15:59:28.179050+0000","172.21.15.167:6832/1493691973":"2024-06-30T15:59:28.179050+0000","172.21.15.167:0/3181687202":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1949624986":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2055038349":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1927715461":"2024-06-30T15:59:21.470262+0000","172.21.15.167:6832/1778875525":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/2270885245":"2024-06-30T15:59:21.470262+0000","172.21.15.167:6833/1778875525":"2024-06-30T15:59:21.470262+0000","172.21.15.167:0/1180941913":"2024-06-30T15:59:21.470262+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:34.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 -- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb300076420 msgr2=0x7fb300078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb300076420 0x7fb300078930 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb304005d70 tx=0x7fb304005d00 comp rx=0 tx=0).stop 2024-06-29T15:59:34.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 -- 172.21.15.81:0/452538078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb31406fa90 msgr2=0x7fb31409a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:34.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb31406fa90 0x7fb31409a470 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7fb308034040 tx=0x7fb3080039e0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 -- 172.21.15.81:0/452538078 shutdown_connections 2024-06-29T15:59:34.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb300076420 0x7fb300078930 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb314074520 0x7fb314159b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb31406fa90 0x7fb31409a470 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 --2- 172.21.15.81:0/452538078 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb31406f680 0x7fb3141516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:34.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 -- 172.21.15.81:0/452538078 >> 172.21.15.81:0/452538078 conn(0x7fb314072950 msgr2=0x7fb3140afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:34.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 -- 172.21.15.81:0/452538078 shutdown_connections 2024-06-29T15:59:34.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:34.885+0000 7fb31a38c640 1 -- 172.21.15.81:0/452538078 wait complete. 2024-06-29T15:59:34.897 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:34.897 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:35.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.012+0000 7f7a32cb3640 1 -- 172.21.15.81:0/1574986890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06fd70 msgr2=0x7f7a2c071f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.012+0000 7f7a32cb3640 1 --2- 172.21.15.81:0/1574986890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06fd70 0x7f7a2c071f20 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c070220 tx=0x7f7a2402a980 comp rx=0 tx=0).stop 2024-06-29T15:59:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.012+0000 7f7a32cb3640 1 -- 172.21.15.81:0/1574986890 shutdown_connections 2024-06-29T15:59:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.012+0000 7f7a32cb3640 1 --2- 172.21.15.81:0/1574986890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c06fd70 0x7f7a2c071f20 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.012+0000 7f7a32cb3640 1 --2- 172.21.15.81:0/1574986890 >> v2:172.21.15.167:3300/0 conn(0x7f7a2c0b3640 0x7f7a2c15ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.012+0000 7f7a32cb3640 1 -- 172.21.15.81:0/1574986890 >> 172.21.15.81:0/1574986890 conn(0x7f7a2c06ccc0 msgr2=0x7f7a2c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.012+0000 7f7a32cb3640 1 -- 172.21.15.81:0/1574986890 shutdown_connections 2024-06-29T15:59:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.012+0000 7f7a32cb3640 1 -- 172.21.15.81:0/1574986890 wait complete. 2024-06-29T15:59:35.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.014+0000 7f7a32cb3640 1 Processor -- start 2024-06-29T15:59:35.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.014+0000 7f7a32cb3640 1 -- start start 2024-06-29T15:59:35.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a32cb3640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0b3640 0x7f7a2c146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a32cb3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c15b060 0x7f7a2c166220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a32cb3640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a2c149db0 0x7f7a2c166760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a32cb3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c06edd0 con 0x7f7a2c15b060 2024-06-29T15:59:35.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a32cb3640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c0cf7d0 con 0x7f7a2c0b3640 2024-06-29T15:59:35.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a32cb3640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2c06ef50 con 0x7f7a2c149db0 2024-06-29T15:59:35.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a23fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c15b060 0x7f7a2c166220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a23fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c15b060 0x7f7a2c166220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46540/0 (socket says 172.21.15.81:46540) 2024-06-29T15:59:35.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a23fff640 1 -- 172.21.15.81:0/2538682384 learned_addr learned my addr 172.21.15.81:0/2538682384 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a30a4f640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0b3640 0x7f7a2c146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.015+0000 7f7a31250640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a2c149db0 0x7f7a2c166760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a23fff640 1 -- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a2c149db0 msgr2=0x7f7a2c166760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a23fff640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a2c149db0 0x7f7a2c166760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a23fff640 1 -- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0b3640 msgr2=0x7f7a2c146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a23fff640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0b3640 0x7f7a2c146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a23fff640 1 -- 172.21.15.81:0/2538682384 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2c14a2d0 con 0x7f7a2c15b060 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a23fff640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c15b060 0x7f7a2c166220 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00b3e0 tx=0x7f7a1c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a217fa640 1 -- 172.21.15.81:0/2538682384 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c015020 con 0x7f7a2c15b060 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a217fa640 1 -- 172.21.15.81:0/2538682384 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a1c011940 con 0x7f7a2c15b060 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a32cb3640 1 -- 172.21.15.81:0/2538682384 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a2c146c20 con 0x7f7a2c15b060 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a217fa640 1 -- 172.21.15.81:0/2538682384 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c011d90 con 0x7f7a2c15b060 2024-06-29T15:59:35.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.016+0000 7f7a32cb3640 1 -- 172.21.15.81:0/2538682384 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a2c1471a0 con 0x7f7a2c15b060 2024-06-29T15:59:35.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.017+0000 7f7a32cb3640 1 -- 172.21.15.81:0/2538682384 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a2c15ab20 con 0x7f7a2c15b060 2024-06-29T15:59:35.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.018+0000 7f7a217fa640 1 -- 172.21.15.81:0/2538682384 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f7a1c018020 con 0x7f7a2c15b060 2024-06-29T15:59:35.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.018+0000 7f7a217fa640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7a180763f0 0x7f7a18078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.019+0000 7f7a217fa640 1 -- 172.21.15.81:0/2538682384 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5845+0+0 (secure 0 0 0) 0x7f7a1c097930 con 0x7f7a2c15b060 2024-06-29T15:59:35.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.019+0000 7f7a30a4f640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7a180763f0 0x7f7a18078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.019+0000 7f7a30a4f640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7a180763f0 0x7f7a18078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7a28005fd0 tx=0x7f7a2800a250 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:35.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.023+0000 7f7a217fa640 1 -- 172.21.15.81:0/2538682384 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7a1c01f020 con 0x7f7a2c15b060 2024-06-29T15:59:35.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.204+0000 7f7a32cb3640 1 -- 172.21.15.81:0/2538682384 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7a2c15a450 con 0x7f7a2c15b060 2024-06-29T15:59:35.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.204+0000 7f7a217fa640 1 -- 172.21.15.81:0/2538682384 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v258) ==== 93+0+10 (secure 0 0 0) 0x7f7a1c060990 con 0x7f7a2c15b060 2024-06-29T15:59:35.205 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:59:35.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 -- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7a180763f0 msgr2=0x7f7a18078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7a180763f0 0x7f7a18078900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7a28005fd0 tx=0x7f7a2800a250 comp rx=0 tx=0).stop 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 -- 172.21.15.81:0/2538682384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c15b060 msgr2=0x7f7a2c166220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c15b060 0x7f7a2c166220 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00b3e0 tx=0x7f7a1c00b8a0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 -- 172.21.15.81:0/2538682384 shutdown_connections 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7a180763f0 0x7f7a18078900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7a2c149db0 0x7f7a2c166760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a2c15b060 0x7f7a2c166220 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 --2- 172.21.15.81:0/2538682384 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7a2c0b3640 0x7f7a2c146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 -- 172.21.15.81:0/2538682384 >> 172.21.15.81:0/2538682384 conn(0x7f7a2c06ccc0 msgr2=0x7f7a2c0b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 -- 172.21.15.81:0/2538682384 shutdown_connections 2024-06-29T15:59:35.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.206+0000 7f7a32cb3640 1 -- 172.21.15.81:0/2538682384 wait complete. 2024-06-29T15:59:35.218 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-06-29T15:59:35.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.331+0000 7f0586bd9640 1 -- 172.21.15.81:0/134331280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800cd480 msgr2=0x7f05800ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.331+0000 7f0586bd9640 1 --2- 172.21.15.81:0/134331280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800cd480 0x7f05800ce510 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7f0568005fd0 tx=0x7f056802a540 comp rx=0 tx=0).stop 2024-06-29T15:59:35.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.331+0000 7f0586bd9640 1 -- 172.21.15.81:0/134331280 shutdown_connections 2024-06-29T15:59:35.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.331+0000 7f0586bd9640 1 --2- 172.21.15.81:0/134331280 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05800cea50 0x7f05800b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.331+0000 7f0586bd9640 1 --2- 172.21.15.81:0/134331280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800cd480 0x7f05800ce510 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.331+0000 7f0586bd9640 1 -- 172.21.15.81:0/134331280 >> 172.21.15.81:0/134331280 conn(0x7f05800b3510 msgr2=0x7f05800b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:35.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.331+0000 7f0586bd9640 1 -- 172.21.15.81:0/134331280 shutdown_connections 2024-06-29T15:59:35.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.331+0000 7f0586bd9640 1 -- 172.21.15.81:0/134331280 wait complete. 2024-06-29T15:59:35.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.333+0000 7f0586bd9640 1 Processor -- start 2024-06-29T15:59:35.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.333+0000 7f0586bd9640 1 -- start start 2024-06-29T15:59:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f0586bd9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05800b6eb0 0x7f058015f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f0586bd9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800cd480 0x7f05800c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f0586bd9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05800cea50 0x7f0580156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f0586bd9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f05800b50e0 con 0x7f05800cd480 2024-06-29T15:59:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f0586bd9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f05800b4f60 con 0x7f05800b6eb0 2024-06-29T15:59:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f0586bd9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f05800b5260 con 0x7f05800cea50 2024-06-29T15:59:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f057ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800cd480 0x7f05800c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f057ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800cd480 0x7f05800c8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46558/0 (socket says 172.21.15.81:46558) 2024-06-29T15:59:35.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f057ffff640 1 -- 172.21.15.81:0/3751733734 learned_addr learned my addr 172.21.15.81:0/3751733734 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f0585176640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05800cea50 0x7f0580156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.334+0000 7f0584975640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05800b6eb0 0x7f058015f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f057ffff640 1 -- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05800cea50 msgr2=0x7f0580156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f057ffff640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05800cea50 0x7f0580156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f057ffff640 1 -- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05800b6eb0 msgr2=0x7f058015f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f057ffff640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05800b6eb0 0x7f058015f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f057ffff640 1 -- 172.21.15.81:0/3751733734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05800c9140 con 0x7f05800cd480 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f057ffff640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800cd480 0x7f05800c8a30 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f0568034040 tx=0x7f05680039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f057d7fa640 1 -- 172.21.15.81:0/3751733734 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056803e030 con 0x7f05800cd480 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f057d7fa640 1 -- 172.21.15.81:0/3751733734 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05680047a0 con 0x7f05800cd480 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f0586bd9640 1 -- 172.21.15.81:0/3751733734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05800c9420 con 0x7f05800cd480 2024-06-29T15:59:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f057d7fa640 1 -- 172.21.15.81:0/3751733734 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0568035040 con 0x7f05800cd480 2024-06-29T15:59:35.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.335+0000 7f0586bd9640 1 -- 172.21.15.81:0/3751733734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05800c99e0 con 0x7f05800cd480 2024-06-29T15:59:35.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.336+0000 7f0586bd9640 1 -- 172.21.15.81:0/3751733734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05800ce510 con 0x7f05800cd480 2024-06-29T15:59:35.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.337+0000 7f057d7fa640 1 -- 172.21.15.81:0/3751733734 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f0568041030 con 0x7f05800cd480 2024-06-29T15:59:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.337+0000 7f057d7fa640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f05600763f0 0x7f0560078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.337+0000 7f057d7fa640 1 -- 172.21.15.81:0/3751733734 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5845+0+0 (secure 0 0 0) 0x7f05680b8290 con 0x7f05800cd480 2024-06-29T15:59:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.338+0000 7f0584975640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f05600763f0 0x7f0560078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:35.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.338+0000 7f0584975640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f05600763f0 0x7f0560078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f05800cdb60 tx=0x7f0570009040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:35.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.342+0000 7f057d7fa640 1 -- 172.21.15.81:0/3751733734 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f0568081260 con 0x7f05800cd480 2024-06-29T15:59:35.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:35.595+0000 7f0586bd9640 1 -- 172.21.15.81:0/3751733734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) -- 0x7f05800cdf00 con 0x7f05800cd480 2024-06-29T15:59:36.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.567+0000 7f057d7fa640 1 -- 172.21.15.81:0/3751733734 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3199) ==== 204+0+0 (secure 0 0 0) 0x7f0568085560 con 0x7f05800cd480 2024-06-29T15:59:36.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 -- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f05600763f0 msgr2=0x7f0560078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f05600763f0 0x7f0560078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f05800cdb60 tx=0x7f0570009040 comp rx=0 tx=0).stop 2024-06-29T15:59:36.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 -- 172.21.15.81:0/3751733734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800cd480 msgr2=0x7f05800c8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800cd480 0x7f05800c8a30 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f0568034040 tx=0x7f05680039e0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 -- 172.21.15.81:0/3751733734 shutdown_connections 2024-06-29T15:59:36.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f05600763f0 0x7f0560078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f05800cea50 0x7f0580156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800cd480 0x7f05800c8a30 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 --2- 172.21.15.81:0/3751733734 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f05800b6eb0 0x7f058015f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 -- 172.21.15.81:0/3751733734 >> 172.21.15.81:0/3751733734 conn(0x7f05800b3510 msgr2=0x7f0580070100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:36.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 -- 172.21.15.81:0/3751733734 shutdown_connections 2024-06-29T15:59:36.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.570+0000 7f0586bd9640 1 -- 172.21.15.81:0/3751733734 wait complete. 2024-06-29T15:59:36.584 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-06-29T15:59:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.704+0000 7f4b8f47f640 1 -- 172.21.15.81:0/978727015 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8806f850 msgr2=0x7f4b880ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.704+0000 7f4b8f47f640 1 --2- 172.21.15.81:0/978727015 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8806f850 0x7f4b880ae300 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009830 tx=0x7f4b7802a510 comp rx=0 tx=0).stop 2024-06-29T15:59:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.704+0000 7f4b8f47f640 1 -- 172.21.15.81:0/978727015 shutdown_connections 2024-06-29T15:59:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.704+0000 7f4b8f47f640 1 --2- 172.21.15.81:0/978727015 >> v2:172.21.15.167:3300/0 conn(0x7f4b880ae840 0x7f4b880aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.704+0000 7f4b8f47f640 1 --2- 172.21.15.81:0/978727015 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8806f850 0x7f4b880ae300 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.704+0000 7f4b8f47f640 1 -- 172.21.15.81:0/978727015 >> 172.21.15.81:0/978727015 conn(0x7f4b88072950 msgr2=0x7f4b880b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.704+0000 7f4b8f47f640 1 -- 172.21.15.81:0/978727015 shutdown_connections 2024-06-29T15:59:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.704+0000 7f4b8f47f640 1 -- 172.21.15.81:0/978727015 wait complete. 2024-06-29T15:59:36.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.706+0000 7f4b8f47f640 1 Processor -- start 2024-06-29T15:59:36.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.706+0000 7f4b8f47f640 1 -- start start 2024-06-29T15:59:36.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8f47f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8806f850 0x7f4b8814fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8f47f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b88074520 0x7f4b881f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8f47f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b880ae840 0x7f4b880cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8f47f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8815b190 con 0x7f4b880ae840 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8f47f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8815b010 con 0x7f4b88074520 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8f47f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4b880af190 con 0x7f4b8806f850 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8da1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b880ae840 0x7f4b880cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8da1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b880ae840 0x7f4b880cfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46572/0 (socket says 172.21.15.81:46572) 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8da1c640 1 -- 172.21.15.81:0/2573543498 learned_addr learned my addr 172.21.15.81:0/2573543498 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8ca1a640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b88074520 0x7f4b881f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8d21b640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8806f850 0x7f4b8814fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8da1c640 1 -- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8806f850 msgr2=0x7f4b8814fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8da1c640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8806f850 0x7f4b8814fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8da1c640 1 -- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b88074520 msgr2=0x7f4b881f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.707+0000 7f4b8da1c640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b88074520 0x7f4b881f4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.708+0000 7f4b8da1c640 1 -- 172.21.15.81:0/2573543498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b881f5840 con 0x7f4b880ae840 2024-06-29T15:59:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.708+0000 7f4b8da1c640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b880ae840 0x7f4b880cfb80 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000efc0 tx=0x7f4b8000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.708+0000 7f4b7dffb640 1 -- 172.21.15.81:0/2573543498 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80003970 con 0x7f4b880ae840 2024-06-29T15:59:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.708+0000 7f4b7dffb640 1 -- 172.21.15.81:0/2573543498 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b80003c20 con 0x7f4b880ae840 2024-06-29T15:59:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.708+0000 7f4b8f47f640 1 -- 172.21.15.81:0/2573543498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b881f82e0 con 0x7f4b880ae840 2024-06-29T15:59:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.708+0000 7f4b7dffb640 1 -- 172.21.15.81:0/2573543498 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80004280 con 0x7f4b880ae840 2024-06-29T15:59:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.708+0000 7f4b8f47f640 1 -- 172.21.15.81:0/2573543498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b88150300 con 0x7f4b880ae840 2024-06-29T15:59:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.709+0000 7f4b8f47f640 1 -- 172.21.15.81:0/2573543498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b881f80d0 con 0x7f4b880ae840 2024-06-29T15:59:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.710+0000 7f4b7dffb640 1 -- 172.21.15.81:0/2573543498 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f4b80004420 con 0x7f4b880ae840 2024-06-29T15:59:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.710+0000 7f4b7dffb640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4b6c0763f0 0x7f4b6c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.710+0000 7f4b7dffb640 1 -- 172.21.15.81:0/2573543498 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5845+0+0 (secure 0 0 0) 0x7f4b80098380 con 0x7f4b880ae840 2024-06-29T15:59:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.710+0000 7f4b8d21b640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4b6c0763f0 0x7f4b6c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.711+0000 7f4b8d21b640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4b6c0763f0 0x7f4b6c078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4b84001720 tx=0x7f4b84009040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:36.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.714+0000 7f4b7dffb640 1 -- 172.21.15.81:0/2573543498 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4b800612d0 con 0x7f4b880ae840 2024-06-29T15:59:36.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.894+0000 7f4b8f47f640 1 -- 172.21.15.81:0/2573543498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f4b88075bd0 con 0x7f4b880ae840 2024-06-29T15:59:36.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.894+0000 7f4b7dffb640 1 -- 172.21.15.81:0/2573543498 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v83) ==== 59+0+1998 (secure 0 0 0) 0x7f4b800655d0 con 0x7f4b880ae840 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:mon advanced mon_op_complaint_time 120 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:mon dev mon_warn_on_pool_no_app false 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:osd advanced osd_op_complaint_time 180.000000 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 4995.347675 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 5014.351894 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 5070.901505 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4757.922743 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 4682.905294 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 5167.296317 2024-06-29T15:59:36.895 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 5150.204265 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 5367.822178 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds 20/20 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_mds_balancer 20/20 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced debug_ms 1/1 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_frag true 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_debug_scatterstat true 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced mds_op_complaint_time 180.000000 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:mds dev mds_verify_scatter true 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced osd_op_complaint_time 180.000000 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_mon_op_timeout 900 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:mds advanced rados_osd_op_timeout 900 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced client_mount_timeout 600 2024-06-29T15:59:36.896 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_client 20/20 2024-06-29T15:59:36.897 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced debug_ms 1/1 2024-06-29T15:59:36.897 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_mon_op_timeout 900 2024-06-29T15:59:36.897 INFO:teuthology.orchestra.run.smithi081.stdout:client advanced rados_osd_op_timeout 900 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 -- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4b6c0763f0 msgr2=0x7f4b6c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4b6c0763f0 0x7f4b6c078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4b84001720 tx=0x7f4b84009040 comp rx=0 tx=0).stop 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 -- 172.21.15.81:0/2573543498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b880ae840 msgr2=0x7f4b880cfb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b880ae840 0x7f4b880cfb80 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000efc0 tx=0x7f4b8000c800 comp rx=0 tx=0).stop 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 -- 172.21.15.81:0/2573543498 shutdown_connections 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4b6c0763f0 0x7f4b6c078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b880ae840 0x7f4b880cfb80 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4b88074520 0x7f4b881f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 --2- 172.21.15.81:0/2573543498 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4b8806f850 0x7f4b8814fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 -- 172.21.15.81:0/2573543498 >> 172.21.15.81:0/2573543498 conn(0x7f4b88072950 msgr2=0x7f4b8806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 -- 172.21.15.81:0/2573543498 shutdown_connections 2024-06-29T15:59:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:36.896+0000 7f4b8f47f640 1 -- 172.21.15.81:0/2573543498 wait complete. 2024-06-29T15:59:36.908 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-06-29T15:59:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.024+0000 7f95117f5640 1 -- 172.21.15.81:0/4151553786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f950c0b5250 msgr2=0x7f950c0d2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.024+0000 7f95117f5640 1 --2- 172.21.15.81:0/4151553786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f950c0b5250 0x7f950c0d2190 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f9504005fd0 tx=0x7f950402a670 comp rx=0 tx=0).stop 2024-06-29T15:59:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.024+0000 7f95117f5640 1 -- 172.21.15.81:0/4151553786 shutdown_connections 2024-06-29T15:59:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.024+0000 7f95117f5640 1 --2- 172.21.15.81:0/4151553786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f950c0b5250 0x7f950c0d2190 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.024+0000 7f95117f5640 1 -- 172.21.15.81:0/4151553786 >> 172.21.15.81:0/4151553786 conn(0x7f950c09cf30 msgr2=0x7f950c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.024+0000 7f95117f5640 1 -- 172.21.15.81:0/4151553786 shutdown_connections 2024-06-29T15:59:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.024+0000 7f95117f5640 1 -- 172.21.15.81:0/4151553786 wait complete. 2024-06-29T15:59:37.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.026+0000 7f95117f5640 1 Processor -- start 2024-06-29T15:59:37.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.026+0000 7f95117f5640 1 -- start start 2024-06-29T15:59:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f95117f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f950c0b5250 0x7f950c0a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f95117f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f950c0b9040 0x7f950c159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f95117f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c1f4d20 0x7f950c15a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f95117f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f950c0d26d0 con 0x7f950c0b9040 2024-06-29T15:59:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f95117f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f950c0b85c0 con 0x7f950c1f4d20 2024-06-29T15:59:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f95117f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f950c0d2850 con 0x7f950c0b5250 2024-06-29T15:59:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f950c0b9040 0x7f950c159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f950c0b9040 0x7f950c159940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46600/0 (socket says 172.21.15.81:46600) 2024-06-29T15:59:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950a7fc640 1 -- 172.21.15.81:0/76659698 learned_addr learned my addr 172.21.15.81:0/76659698 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950a7fc640 1 -- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f950c0b5250 msgr2=0x7f950c0a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950b7fe640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c1f4d20 0x7f950c15a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950affd640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f950c0b5250 0x7f950c0a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950a7fc640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f950c0b5250 0x7f950c0a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950a7fc640 1 -- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c1f4d20 msgr2=0x7f950c15a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950a7fc640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c1f4d20 0x7f950c15a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950a7fc640 1 -- 172.21.15.81:0/76659698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950c0a03e0 con 0x7f950c0b9040 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950affd640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f950c0b5250 0x7f950c0a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950b7fe640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c1f4d20 0x7f950c15a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.027+0000 7f950a7fc640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f950c0b9040 0x7f950c159940 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f950000c3d0 tx=0x7f950000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.028+0000 7f94f37fe640 1 -- 172.21.15.81:0/76659698 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500017070 con 0x7f950c0b9040 2024-06-29T15:59:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.028+0000 7f94f37fe640 1 -- 172.21.15.81:0/76659698 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95000053c0 con 0x7f950c0b9040 2024-06-29T15:59:37.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.028+0000 7f94f37fe640 1 -- 172.21.15.81:0/76659698 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500003850 con 0x7f950c0b9040 2024-06-29T15:59:37.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.028+0000 7f95117f5640 1 -- 172.21.15.81:0/76659698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950c0a06c0 con 0x7f950c0b9040 2024-06-29T15:59:37.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.028+0000 7f95117f5640 1 -- 172.21.15.81:0/76659698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f950c0a0c80 con 0x7f950c0b9040 2024-06-29T15:59:37.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.029+0000 7f95117f5640 1 -- 172.21.15.81:0/76659698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f950c0a0150 con 0x7f950c0b9040 2024-06-29T15:59:37.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.029+0000 7f94f37fe640 1 -- 172.21.15.81:0/76659698 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f950001a070 con 0x7f950c0b9040 2024-06-29T15:59:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.030+0000 7f94f37fe640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f94f80763f0 0x7f94f8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.030+0000 7f94f37fe640 1 -- 172.21.15.81:0/76659698 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5845+0+0 (secure 0 0 0) 0x7f9500096e20 con 0x7f950c0b9040 2024-06-29T15:59:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.030+0000 7f950affd640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f94f80763f0 0x7f94f8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.031+0000 7f950affd640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f94f80763f0 0x7f94f8078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f94fc007eb0 tx=0x7f94fc0073c0 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:37.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.034+0000 7f94f37fe640 1 -- 172.21.15.81:0/76659698 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9500060560 con 0x7f950c0b9040 2024-06-29T15:59:37.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.213+0000 7f95117f5640 1 -- 172.21.15.81:0/76659698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f950c0d46a0 con 0x7f950c0b9040 2024-06-29T15:59:37.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.217+0000 7f94f37fe640 1 -- 172.21.15.81:0/76659698 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v84)=0 v84) ==== 122+0+0 (secure 0 0 0) 0x7f950c0d46a0 con 0x7f950c0b9040 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 -- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f94f80763f0 msgr2=0x7f94f8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f94f80763f0 0x7f94f8078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f94fc007eb0 tx=0x7f94fc0073c0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 -- 172.21.15.81:0/76659698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f950c0b9040 msgr2=0x7f950c159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f950c0b9040 0x7f950c159940 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f950000c3d0 tx=0x7f950000c890 comp rx=0 tx=0).stop 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 -- 172.21.15.81:0/76659698 shutdown_connections 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f94f80763f0 0x7f94f8078900 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f950c1f4d20 0x7f950c15a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f950c0b9040 0x7f950c159940 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 --2- 172.21.15.81:0/76659698 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f950c0b5250 0x7f950c0a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 -- 172.21.15.81:0/76659698 >> 172.21.15.81:0/76659698 conn(0x7f950c09cf30 msgr2=0x7f950c06de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:37.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 -- 172.21.15.81:0/76659698 shutdown_connections 2024-06-29T15:59:37.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.219+0000 7f95117f5640 1 -- 172.21.15.81:0/76659698 wait complete. 2024-06-29T15:59:37.231 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T15:59:37.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:37.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T15:59:37.316 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T15:59:37.316 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:59:37.317 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:37.332 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:59:37.332 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:59:37.332 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:37.374 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:37.375 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:59:37.375 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:37.390 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T15:59:37.390 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T15:59:37.390 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:37.432 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T15:59:37.433 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-06-29T15:59:37.433 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:37.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.546+0000 7f2ea5531640 1 -- 172.21.15.81:0/3202283460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea00b5250 msgr2=0x7f2ea00d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.546+0000 7f2ea5531640 1 --2- 172.21.15.81:0/3202283460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea00b5250 0x7f2ea00d09c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f2e88009830 tx=0x7f2e8802a5a0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.546+0000 7f2ea5531640 1 -- 172.21.15.81:0/3202283460 shutdown_connections 2024-06-29T15:59:37.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.546+0000 7f2ea5531640 1 --2- 172.21.15.81:0/3202283460 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea00b5250 0x7f2ea00d09c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.546+0000 7f2ea5531640 1 --2- 172.21.15.81:0/3202283460 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ea00b9040 0x7f2ea00d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.546+0000 7f2ea5531640 1 -- 172.21.15.81:0/3202283460 >> 172.21.15.81:0/3202283460 conn(0x7f2ea009cf30 msgr2=0x7f2ea00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:37.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.546+0000 7f2ea5531640 1 -- 172.21.15.81:0/3202283460 shutdown_connections 2024-06-29T15:59:37.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.546+0000 7f2ea5531640 1 -- 172.21.15.81:0/3202283460 wait complete. 2024-06-29T15:59:37.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.548+0000 7f2ea5531640 1 Processor -- start 2024-06-29T15:59:37.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.548+0000 7f2ea5531640 1 -- start start 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.549+0000 7f2ea5531640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ea00b5250 0x7f2ea00b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.549+0000 7f2ea5531640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea00b7290 0x7f2ea00a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.549+0000 7f2ea5531640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea00b9040 0x7f2ea00a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.549+0000 7f2ea5531640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea00b7db0 con 0x7f2ea00b9040 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.549+0000 7f2ea5531640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea00d2d50 con 0x7f2ea00b7290 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.549+0000 7f2ea5531640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea00b7f30 con 0x7f2ea00b5250 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.549+0000 7f2e9f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea00b9040 0x7f2ea00a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.549+0000 7f2e9f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea00b9040 0x7f2ea00a0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46608/0 (socket says 172.21.15.81:46608) 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.549+0000 7f2e9f7fe640 1 -- 172.21.15.81:0/3287162912 learned_addr learned my addr 172.21.15.81:0/3287162912 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e967fc640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea00b7290 0x7f2ea00a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e9effd640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ea00b5250 0x7f2ea00b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e9f7fe640 1 -- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ea00b5250 msgr2=0x7f2ea00b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e9f7fe640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ea00b5250 0x7f2ea00b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e9f7fe640 1 -- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea00b7290 msgr2=0x7f2ea00a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e9f7fe640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea00b7290 0x7f2ea00a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e9f7fe640 1 -- 172.21.15.81:0/3287162912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea015e680 con 0x7f2ea00b9040 2024-06-29T15:59:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e9f7fe640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea00b9040 0x7f2ea00a0ac0 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800c6e0 tx=0x7f2e9800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e97fff640 1 -- 172.21.15.81:0/3287162912 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e98003b70 con 0x7f2ea00b9040 2024-06-29T15:59:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e97fff640 1 -- 172.21.15.81:0/3287162912 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e98003d10 con 0x7f2ea00b9040 2024-06-29T15:59:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2e97fff640 1 -- 172.21.15.81:0/3287162912 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e98005330 con 0x7f2ea00b9040 2024-06-29T15:59:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2ea5531640 1 -- 172.21.15.81:0/3287162912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea0150f90 con 0x7f2ea00b9040 2024-06-29T15:59:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.550+0000 7f2ea5531640 1 -- 172.21.15.81:0/3287162912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea0151550 con 0x7f2ea00b9040 2024-06-29T15:59:37.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.551+0000 7f2ea5531640 1 -- 172.21.15.81:0/3287162912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea0150ce0 con 0x7f2ea00b9040 2024-06-29T15:59:37.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.552+0000 7f2e97fff640 1 -- 172.21.15.81:0/3287162912 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f2e980054d0 con 0x7f2ea00b9040 2024-06-29T15:59:37.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.552+0000 7f2e97fff640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e800763f0 0x7f2e80078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.552+0000 7f2e97fff640 1 -- 172.21.15.81:0/3287162912 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5845+0+0 (secure 0 0 0) 0x7f2e98097640 con 0x7f2ea00b9040 2024-06-29T15:59:37.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.553+0000 7f2e9effd640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e800763f0 0x7f2e80078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.553+0000 7f2e9effd640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e800763f0 0x7f2e80078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009530 tx=0x7f2e90009260 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:37.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.557+0000 7f2e97fff640 1 -- 172.21.15.81:0/3287162912 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2e98060730 con 0x7f2ea00b9040 2024-06-29T15:59:37.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.772+0000 7f2ea5531640 1 -- 172.21.15.81:0/3287162912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ea00b84c0 con 0x7f2ea00b9040 2024-06-29T15:59:37.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.773+0000 7f2e97fff640 1 -- 172.21.15.81:0/3287162912 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+3918 (secure 0 0 0) 0x7f2e98064a30 con 0x7f2ea00b9040 2024-06-29T15:59:37.773 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:37.773 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":147,"btime":"2024-06-29T15:59:31:402315+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9842,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2717891820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":2717891820},{"type":"v1","addr":"172.21.15.81:6835","nonce":2717891820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":9848,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/2745105064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":2745105064},{"type":"v1","addr":"172.21.15.167:6836","nonce":2745105064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141},{"gid":10133,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/778920475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":778920475},{"type":"v1","addr":"172.21.15.167:6837","nonce":778920475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147},{"gid":11329,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/745441473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":745441473},{"type":"v1","addr":"172.21.15.81:6834","nonce":745441473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":140}],"filesystems":[]} 2024-06-29T15:59:37.773 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 147 2024-06-29T15:59:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 -- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e800763f0 msgr2=0x7f2e80078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e800763f0 0x7f2e80078900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009530 tx=0x7f2e90009260 comp rx=0 tx=0).stop 2024-06-29T15:59:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 -- 172.21.15.81:0/3287162912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea00b9040 msgr2=0x7f2ea00a0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea00b9040 0x7f2ea00a0ac0 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f2e9800c6e0 tx=0x7f2e9800cba0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 -- 172.21.15.81:0/3287162912 shutdown_connections 2024-06-29T15:59:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e800763f0 0x7f2e80078900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea00b9040 0x7f2ea00a0ac0 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea00b7290 0x7f2ea00a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 --2- 172.21.15.81:0/3287162912 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ea00b5250 0x7f2ea00b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 -- 172.21.15.81:0/3287162912 >> 172.21.15.81:0/3287162912 conn(0x7f2ea009cf30 msgr2=0x7f2ea00d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:37.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 -- 172.21.15.81:0/3287162912 shutdown_connections 2024-06-29T15:59:37.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.775+0000 7f2ea5531640 1 -- 172.21.15.81:0/3287162912 wait complete. 2024-06-29T15:59:37.787 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-29T15:59:37.787 INFO:tasks.ceph.mds.a:Stopping old one... 2024-06-29T15:59:37.787 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T15:59:37.787 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:37.787 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-29T15:59:37.788 INFO:tasks.ceph.mds.c:Stopping old one... 2024-06-29T15:59:37.788 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T15:59:37.788 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:37.788 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-29T15:59:37.788 INFO:tasks.ceph.mds.b:Stopping old one... 2024-06-29T15:59:37.788 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T15:59:37.788 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:37.789 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-29T15:59:37.789 INFO:tasks.ceph.mds.d:Stopping old one... 2024-06-29T15:59:37.789 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T15:59:37.789 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T15:59:37.797 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T15:59:37.797 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-06-29T15:59:37.798 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T15:59:37.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-06-29T15:59:37.802 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T15:59:37.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-06-29T15:59:37.804 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T15:59:37.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-06-29T15:59:37.840 INFO:tasks.ceph.mds.a:Started 2024-06-29T15:59:37.840 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:37.840 INFO:tasks.ceph.mds.c:Started 2024-06-29T15:59:37.841 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:37.846 INFO:tasks.ceph.mds.b:Started 2024-06-29T15:59:37.848 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:37.849 INFO:tasks.ceph.mds.d:Started 2024-06-29T15:59:37.849 DEBUG:teuthology.parallel:result is None 2024-06-29T15:59:37.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-06-29T15:59:37.925 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T15:59:37.926 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-06-29T15:59:37.943 INFO:tasks.ceph.mds.d.smithi167.stdout:starting mds.d at 2024-06-29T15:59:37.943 INFO:tasks.ceph.mds.b.smithi167.stdout:starting mds.b at 2024-06-29T15:59:37.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.970+0000 7f86a98e4640 1 -- 172.21.15.81:0/3586400417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a40ce770 msgr2=0x7f86a40b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.970+0000 7f86a98e4640 1 --2- 172.21.15.81:0/3586400417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a40ce770 0x7f86a40b35c0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7f8694009830 tx=0x7f869402a510 comp rx=0 tx=0).stop 2024-06-29T15:59:37.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.970+0000 7f86a98e4640 1 -- 172.21.15.81:0/3586400417 shutdown_connections 2024-06-29T15:59:37.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.970+0000 7f86a98e4640 1 --2- 172.21.15.81:0/3586400417 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86a40d2950 0x7f86a40b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.970+0000 7f86a98e4640 1 --2- 172.21.15.81:0/3586400417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a40ce770 0x7f86a40b35c0 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.970+0000 7f86a98e4640 1 -- 172.21.15.81:0/3586400417 >> 172.21.15.81:0/3586400417 conn(0x7f86a40d3c60 msgr2=0x7f86a40d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:37.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.970+0000 7f86a98e4640 1 -- 172.21.15.81:0/3586400417 shutdown_connections 2024-06-29T15:59:37.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.970+0000 7f86a98e4640 1 -- 172.21.15.81:0/3586400417 wait complete. 2024-06-29T15:59:37.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.971+0000 7f86a98e4640 1 Processor -- start 2024-06-29T15:59:37.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.972+0000 7f86a98e4640 1 -- start start 2024-06-29T15:59:37.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a98e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a40ce770 0x7f86a415b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a98e4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86a40ceb80 0x7f86a40b85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a98e4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86a40d2950 0x7f86a415c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a98e4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f86a40d15a0 con 0x7f86a40ce770 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a98e4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f86a40d1420 con 0x7f86a40ceb80 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a98e4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f86a40d4990 con 0x7f86a40d2950 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a40ce770 0x7f86a415b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a40ce770 0x7f86a415b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46672/0 (socket says 172.21.15.81:46672) 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a2ffd640 1 -- 172.21.15.81:0/2484217242 learned_addr learned my addr 172.21.15.81:0/2484217242 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a37fe640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86a40d2950 0x7f86a415c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f869a7fc640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86a40ceb80 0x7f86a40b85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a2ffd640 1 -- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86a40d2950 msgr2=0x7f86a415c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a2ffd640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86a40d2950 0x7f86a415c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a2ffd640 1 -- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86a40ceb80 msgr2=0x7f86a40b85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:37.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a2ffd640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86a40ceb80 0x7f86a40b85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:37.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a2ffd640 1 -- 172.21.15.81:0/2484217242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a4167d80 con 0x7f86a40ce770 2024-06-29T15:59:37.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f869a7fc640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86a40ceb80 0x7f86a40b85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:37.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.973+0000 7f86a2ffd640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a40ce770 0x7f86a415b170 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f869402ab00 tx=0x7f8694033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:37.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.974+0000 7f86a0ff9640 1 -- 172.21.15.81:0/2484217242 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694005c90 con 0x7f86a40ce770 2024-06-29T15:59:37.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.974+0000 7f86a0ff9640 1 -- 172.21.15.81:0/2484217242 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8694031e50 con 0x7f86a40ce770 2024-06-29T15:59:37.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.974+0000 7f86a0ff9640 1 -- 172.21.15.81:0/2484217242 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694005260 con 0x7f86a40ce770 2024-06-29T15:59:37.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.974+0000 7f86a98e4640 1 -- 172.21.15.81:0/2484217242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a4168060 con 0x7f86a40ce770 2024-06-29T15:59:37.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.974+0000 7f86a98e4640 1 -- 172.21.15.81:0/2484217242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a4168620 con 0x7f86a40ce770 2024-06-29T15:59:37.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.975+0000 7f86a98e4640 1 -- 172.21.15.81:0/2484217242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a40b64c0 con 0x7f86a40ce770 2024-06-29T15:59:37.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.975+0000 7f86a0ff9640 1 -- 172.21.15.81:0/2484217242 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f8694005400 con 0x7f86a40ce770 2024-06-29T15:59:37.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.976+0000 7f86a0ff9640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8684076420 0x7f8684078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:37.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.976+0000 7f86a0ff9640 1 -- 172.21.15.81:0/2484217242 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5845+0+0 (secure 0 0 0) 0x7f86940b89c0 con 0x7f86a40ce770 2024-06-29T15:59:37.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.976+0000 7f869a7fc640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8684076420 0x7f8684078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:37.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.977+0000 7f869a7fc640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8684076420 0x7f8684078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f868c000f60 tx=0x7f868c009040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:37.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:37.980+0000 7f86a0ff9640 1 -- 172.21.15.81:0/2484217242 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f86940bae20 con 0x7f86a40ce770 2024-06-29T15:59:38.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.173+0000 7f86a98e4640 1 -- 172.21.15.81:0/2484217242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f86a40b61e0 con 0x7f86a40ce770 2024-06-29T15:59:38.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.580+0000 7f86a0ff9640 1 -- 172.21.15.81:0/2484217242 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v260) ==== 97+0+0 (secure 0 0 0) 0x7f869403f030 con 0x7f86a40ce770 2024-06-29T15:59:38.580 INFO:teuthology.orchestra.run.smithi081.stderr: removed all blocklist entries 2024-06-29T15:59:38.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 -- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8684076420 msgr2=0x7f8684078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8684076420 0x7f8684078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f868c000f60 tx=0x7f868c009040 comp rx=0 tx=0).stop 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 -- 172.21.15.81:0/2484217242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a40ce770 msgr2=0x7f86a415b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a40ce770 0x7f86a415b170 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f869402ab00 tx=0x7f8694033c20 comp rx=0 tx=0).stop 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 -- 172.21.15.81:0/2484217242 shutdown_connections 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8684076420 0x7f8684078930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f86a40d2950 0x7f86a415c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f86a40ceb80 0x7f86a40b85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 --2- 172.21.15.81:0/2484217242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a40ce770 0x7f86a415b170 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 -- 172.21.15.81:0/2484217242 >> 172.21.15.81:0/2484217242 conn(0x7f86a40d3c60 msgr2=0x7f86a415f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 -- 172.21.15.81:0/2484217242 shutdown_connections 2024-06-29T15:59:38.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.583+0000 7f86a98e4640 1 -- 172.21.15.81:0/2484217242 wait complete. 2024-06-29T15:59:38.596 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-06-29T15:59:38.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.710+0000 7f789c7f5640 1 -- 172.21.15.81:0/869044579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7880004800 msgr2=0x7f7880004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.710+0000 7f789c7f5640 1 --2- 172.21.15.81:0/869044579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7880004800 0x7f7880004c10 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f788400bb20 tx=0x7f7880005830 comp rx=0 tx=0).stop 2024-06-29T15:59:38.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.710+0000 7f789c7f5640 1 -- 172.21.15.81:0/869044579 shutdown_connections 2024-06-29T15:59:38.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.710+0000 7f789c7f5640 1 --2- 172.21.15.81:0/869044579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7880004800 0x7f7880004c10 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.710+0000 7f789c7f5640 1 -- 172.21.15.81:0/869044579 >> 172.21.15.81:0/869044579 conn(0x7f78940b3510 msgr2=0x7f78940b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:38.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.710+0000 7f789c7f5640 1 -- 172.21.15.81:0/869044579 shutdown_connections 2024-06-29T15:59:38.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.710+0000 7f789c7f5640 1 -- 172.21.15.81:0/869044579 wait complete. 2024-06-29T15:59:38.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.712+0000 7f789c7f5640 1 Processor -- start 2024-06-29T15:59:38.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.712+0000 7f789c7f5640 1 -- start start 2024-06-29T15:59:38.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.713+0000 7f789c7f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7880004800 0x7f789409b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.713+0000 7f789c7f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78941f5fd0 0x7f78941f63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.713+0000 7f789c7f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78941d01e0 0x7f78941d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.713+0000 7f789c7f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f78940ce8a0 con 0x7f7880004800 2024-06-29T15:59:38.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.713+0000 7f789c7f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f78940ce720 con 0x7f78941f5fd0 2024-06-29T15:59:38.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.713+0000 7f789c7f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f78940d02d0 con 0x7f78941d01e0 2024-06-29T15:59:38.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789a591640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7880004800 0x7f789409b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789a591640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7880004800 0x7f789409b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46688/0 (socket says 172.21.15.81:46688) 2024-06-29T15:59:38.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789a591640 1 -- 172.21.15.81:0/3304263296 learned_addr learned my addr 172.21.15.81:0/3304263296 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:38.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f7899d90640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78941f5fd0 0x7f78941f63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789ad92640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78941d01e0 0x7f78941d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789a591640 1 -- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78941d01e0 msgr2=0x7f78941d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789a591640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78941d01e0 0x7f78941d05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789a591640 1 -- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78941f5fd0 msgr2=0x7f78941f63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789a591640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78941f5fd0 0x7f78941f63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789a591640 1 -- 172.21.15.81:0/3304263296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78940c9790 con 0x7f7880004800 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789a591640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7880004800 0x7f789409b900 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f789000e700 tx=0x7f789000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f788affd640 1 -- 172.21.15.81:0/3304263296 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7890004440 con 0x7f7880004800 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f788affd640 1 -- 172.21.15.81:0/3304263296 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7890016070 con 0x7f7880004800 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789c7f5640 1 -- 172.21.15.81:0/3304263296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78940c9a70 con 0x7f7880004800 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f788affd640 1 -- 172.21.15.81:0/3304263296 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789000caa0 con 0x7f7880004800 2024-06-29T15:59:38.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.714+0000 7f789c7f5640 1 -- 172.21.15.81:0/3304263296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78940ca030 con 0x7f7880004800 2024-06-29T15:59:38.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.715+0000 7f789c7f5640 1 -- 172.21.15.81:0/3304263296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78940d0830 con 0x7f7880004800 2024-06-29T15:59:38.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.716+0000 7f788affd640 1 -- 172.21.15.81:0/3304263296 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f789000cc40 con 0x7f7880004800 2024-06-29T15:59:38.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.716+0000 7f788affd640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f78780763f0 0x7f7878078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:38.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.717+0000 7f788affd640 1 -- 172.21.15.81:0/3304263296 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 4985+0+0 (secure 0 0 0) 0x7f7890098680 con 0x7f7880004800 2024-06-29T15:59:38.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.717+0000 7f7899d90640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f78780763f0 0x7f7878078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:38.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.717+0000 7f7899d90640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f78780763f0 0x7f7878078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7884036000 tx=0x7f78840046c0 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:38.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.720+0000 7f788affd640 1 -- 172.21.15.81:0/3304263296 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f78900620e0 con 0x7f7880004800 2024-06-29T15:59:38.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.972+0000 7f789c7f5640 1 -- 172.21.15.81:0/3304263296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f78940cdf00 con 0x7f7880004800 2024-06-29T15:59:38.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.973+0000 7f788affd640 1 -- 172.21.15.81:0/3304263296 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v70) ==== 80+0+5304 (secure 0 0 0) 0x7f789006a160 con 0x7f7880004800 2024-06-29T15:59:38.973 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:38.973 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:59:38.973 INFO:teuthology.orchestra.run.smithi081.stdout: "auth_dump": [ 2024-06-29T15:59:38.973 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.973 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.a", 2024-06-29T15:59:38.973 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm4DW9DBAAjjPrta8/NiI0d+Ad1zxsmw==", 2024-06-29T15:59:38.973 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.b", 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmFCSiGRAAg+7RkMQJE/hvVox8JU4w4g==", 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:59:38.974 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.c", 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmrR1cFBAAPpTXqoJYG5h0brXZU3Hefw==", 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.975 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mds.d", 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmPIJLIRAAs2Jc5YvkdLjqim0frlPzGA==", 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.0", 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB3GIBmkN10BRAA2z2cEUfUm/QRps047Ey/ZQ==", 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:38.976 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.1", 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB6GIBmf8/6BBAACZxH0r/Bwn+Qmse1ALcCZQ==", 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.977 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.2", 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQB9GIBmACu6AxAAa9hiTDisBWb7cu0mcuDMiw==", 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.3", 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCAGIBml8MVAxAAmayUfVvSIh/hIaWwiXAn/g==", 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.978 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.4", 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCGGIBmiWeKDBAAB5hHpoeiQ52KEJeOe5bHoA==", 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.979 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.5", 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCJGIBmpvkeCxAAgUcqzxjPukwVCfKbs/VotA==", 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.6", 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCMGIBmGnErChAAs1siDJ7f10bj3hDHTMSdLQ==", 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:38.980 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "osd.7", 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQCPGIBm0KO7CBAADrqYiygP1xu2+9Ufo/Vr3g==", 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow profile osd", 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile osd", 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.0", 2024-06-29T15:59:38.981 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w==", 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow r", 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.1", 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBm8EN8LRAAicQLRuHjv9gxQN3//idxTA==", 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:59:38.982 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.2", 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBms6m0KRAAKG+RWEdijDGvKykubXSHlQ==", 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.983 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.3", 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmF7tOMRAA9BIc2sQQFG2pmgzr0sDqWw==", 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow", 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow r", 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow rw", 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow rwx" 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "client.admin", 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQByGIBmR2RgKxAA+cupjhP1i8cqmTBgoEKYDA==", 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.984 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: "mgr": "allow *", 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow *", 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.x", 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmvvL/CBAAd9i3E+wXwBvwwEKezVmnSw==", 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-06-29T15:59:38.985 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:38.986 INFO:teuthology.orchestra.run.smithi081.stdout: "entity": "mgr.y", 2024-06-29T15:59:38.986 INFO:teuthology.orchestra.run.smithi081.stdout: "key": "AQBzGIBmEE/vBBAAfveoc4ZKZVmZANGhvxiipQ==", 2024-06-29T15:59:38.986 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": { 2024-06-29T15:59:38.986 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": "allow *", 2024-06-29T15:59:38.986 INFO:teuthology.orchestra.run.smithi081.stdout: "mon": "allow profile mgr", 2024-06-29T15:59:38.986 INFO:teuthology.orchestra.run.smithi081.stdout: "osd": "allow *" 2024-06-29T15:59:38.986 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.986 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:38.986 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-06-29T15:59:38.986 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 -- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f78780763f0 msgr2=0x7f7878078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f78780763f0 0x7f7878078900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7884036000 tx=0x7f78840046c0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 -- 172.21.15.81:0/3304263296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7880004800 msgr2=0x7f789409b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7880004800 0x7f789409b900 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f789000e700 tx=0x7f789000ebc0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 -- 172.21.15.81:0/3304263296 shutdown_connections 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f78780763f0 0x7f7878078900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78941d01e0 0x7f78941d05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78941f5fd0 0x7f78941f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 --2- 172.21.15.81:0/3304263296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7880004800 0x7f789409b900 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 -- 172.21.15.81:0/3304263296 >> 172.21.15.81:0/3304263296 conn(0x7f78940b3510 msgr2=0x7f78941e4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:38.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 -- 172.21.15.81:0/3304263296 shutdown_connections 2024-06-29T15:59:38.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:38.975+0000 7f789c7f5640 1 -- 172.21.15.81:0/3304263296 wait complete. 2024-06-29T15:59:38.989 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:39.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.102+0000 7f39ab878640 1 -- 172.21.15.81:0/2141908901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a415cd00 msgr2=0x7f39a415d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.102+0000 7f39ab878640 1 --2- 172.21.15.81:0/2141908901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a415cd00 0x7f39a415d510 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f39a40b4f10 tx=0x7f399c02a4a0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.102+0000 7f39ab878640 1 -- 172.21.15.81:0/2141908901 shutdown_connections 2024-06-29T15:59:39.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.102+0000 7f39ab878640 1 --2- 172.21.15.81:0/2141908901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a415cd00 0x7f39a415d510 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.102+0000 7f39ab878640 1 --2- 172.21.15.81:0/2141908901 >> v2:172.21.15.167:3300/0 conn(0x7f39a4145530 0x7f39a4073280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.102+0000 7f39ab878640 1 -- 172.21.15.81:0/2141908901 >> 172.21.15.81:0/2141908901 conn(0x7f39a406ccc0 msgr2=0x7f39a40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:39.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.102+0000 7f39ab878640 1 -- 172.21.15.81:0/2141908901 shutdown_connections 2024-06-29T15:59:39.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.102+0000 7f39ab878640 1 -- 172.21.15.81:0/2141908901 wait complete. 2024-06-29T15:59:39.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.104+0000 7f39ab878640 1 Processor -- start 2024-06-29T15:59:39.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.104+0000 7f39ab878640 1 -- start start 2024-06-29T15:59:39.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39ab878640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40737c0 0x7f39a413bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39ab878640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39a4145530 0x7f39a413e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39ab878640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39a415cd00 0x7f39a4158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39ab878640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39a415dbd0 con 0x7f39a40737c0 2024-06-29T15:59:39.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39ab878640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f39a415da50 con 0x7f39a415cd00 2024-06-29T15:59:39.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39ab878640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f39a415dd50 con 0x7f39a4145530 2024-06-29T15:59:39.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9614640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40737c0 0x7f39a413bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9614640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40737c0 0x7f39a413bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46710/0 (socket says 172.21.15.81:46710) 2024-06-29T15:59:39.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9614640 1 -- 172.21.15.81:0/1972323497 learned_addr learned my addr 172.21.15.81:0/1972323497 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:39.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9e15640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39a415cd00 0x7f39a4158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a8e13640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39a4145530 0x7f39a413e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9614640 1 -- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39a4145530 msgr2=0x7f39a413e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9614640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39a4145530 0x7f39a413e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9614640 1 -- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39a415cd00 msgr2=0x7f39a4158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9614640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39a415cd00 0x7f39a4158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9614640 1 -- 172.21.15.81:0/1972323497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a41574c0 con 0x7f39a40737c0 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9e15640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39a415cd00 0x7f39a4158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a8e13640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39a4145530 0x7f39a413e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.105+0000 7f39a9614640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40737c0 0x7f39a413bfc0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7f399400cce0 tx=0x7f3994007610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.106+0000 7f39a1ffb640 1 -- 172.21.15.81:0/1972323497 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994004b20 con 0x7f39a40737c0 2024-06-29T15:59:39.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.106+0000 7f39a1ffb640 1 -- 172.21.15.81:0/1972323497 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3994004db0 con 0x7f39a40737c0 2024-06-29T15:59:39.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.106+0000 7f39ab878640 1 -- 172.21.15.81:0/1972323497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a4157760 con 0x7f39a40737c0 2024-06-29T15:59:39.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.106+0000 7f39a1ffb640 1 -- 172.21.15.81:0/1972323497 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994005280 con 0x7f39a40737c0 2024-06-29T15:59:39.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.106+0000 7f39ab878640 1 -- 172.21.15.81:0/1972323497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a4157d20 con 0x7f39a40737c0 2024-06-29T15:59:39.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.107+0000 7f39ab878640 1 -- 172.21.15.81:0/1972323497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a415e2b0 con 0x7f39a40737c0 2024-06-29T15:59:39.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.107+0000 7f39a1ffb640 1 -- 172.21.15.81:0/1972323497 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f3994005420 con 0x7f39a40737c0 2024-06-29T15:59:39.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.108+0000 7f39a1ffb640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3990076420 0x7f3990078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.108+0000 7f39a1ffb640 1 -- 172.21.15.81:0/1972323497 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 4985+0+0 (secure 0 0 0) 0x7f3994097ef0 con 0x7f39a40737c0 2024-06-29T15:59:39.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.108+0000 7f39a8e13640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3990076420 0x7f3990078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.109+0000 7f39a8e13640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3990076420 0x7f3990078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f39a415bf10 tx=0x7f3998009040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:39.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.112+0000 7f39a1ffb640 1 -- 172.21.15.81:0/1972323497 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3994014030 con 0x7f39a40737c0 2024-06-29T15:59:39.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.292+0000 7f39ab878640 1 -- 172.21.15.81:0/1972323497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39a41563d0 con 0x7f39a40737c0 2024-06-29T15:59:39.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.293+0000 7f39a1ffb640 1 -- 172.21.15.81:0/1972323497 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) ==== 74+0+13093 (secure 0 0 0) 0x7f3994061870 con 0x7f39a40737c0 2024-06-29T15:59:39.293 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:39.294 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":262,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:38.589353+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":251,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:39.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 -- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3990076420 msgr2=0x7f3990078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3990076420 0x7f3990078930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f39a415bf10 tx=0x7f3998009040 comp rx=0 tx=0).stop 2024-06-29T15:59:39.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 -- 172.21.15.81:0/1972323497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40737c0 msgr2=0x7f39a413bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40737c0 0x7f39a413bfc0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7f399400cce0 tx=0x7f3994007610 comp rx=0 tx=0).stop 2024-06-29T15:59:39.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 -- 172.21.15.81:0/1972323497 shutdown_connections 2024-06-29T15:59:39.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3990076420 0x7f3990078930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f39a415cd00 0x7f39a4158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f39a4145530 0x7f39a413e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 --2- 172.21.15.81:0/1972323497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40737c0 0x7f39a413bfc0 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 -- 172.21.15.81:0/1972323497 >> 172.21.15.81:0/1972323497 conn(0x7f39a406ccc0 msgr2=0x7f39a41448c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:39.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 -- 172.21.15.81:0/1972323497 shutdown_connections 2024-06-29T15:59:39.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.295+0000 7f39ab878640 1 -- 172.21.15.81:0/1972323497 wait complete. 2024-06-29T15:59:39.307 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T15:59:39.308 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T15:59:39.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.423+0000 7fc82961f640 1 -- 172.21.15.81:0/3608590234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8240d0b90 msgr2=0x7fc8240d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.423+0000 7fc82961f640 1 --2- 172.21.15.81:0/3608590234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8240d0b90 0x7fc8240d1a80 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7fc80c009830 tx=0x7fc80c02a5a0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.423+0000 7fc82961f640 1 -- 172.21.15.81:0/3608590234 shutdown_connections 2024-06-29T15:59:39.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.423+0000 7fc82961f640 1 --2- 172.21.15.81:0/3608590234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8240d0b90 0x7fc8240d1a80 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.423+0000 7fc82961f640 1 -- 172.21.15.81:0/3608590234 >> 172.21.15.81:0/3608590234 conn(0x7fc82409cf30 msgr2=0x7fc8240d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:39.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.423+0000 7fc82961f640 1 -- 172.21.15.81:0/3608590234 shutdown_connections 2024-06-29T15:59:39.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.423+0000 7fc82961f640 1 -- 172.21.15.81:0/3608590234 wait complete. 2024-06-29T15:59:39.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.424+0000 7fc82961f640 1 Processor -- start 2024-06-29T15:59:39.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.425+0000 7fc82961f640 1 -- start start 2024-06-29T15:59:39.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc82961f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8240b5250 0x7fc8240a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc82961f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8240b9040 0x7fc82409dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc82961f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8241599a0 0x7fc824159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc82961f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc8240d26d0 con 0x7fc8240b9040 2024-06-29T15:59:39.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc82961f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc8240b85c0 con 0x7fc8241599a0 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc82961f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc8240d2850 con 0x7fc8240b5250 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc81a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8240b9040 0x7fc82409dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc81a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8240b9040 0x7fc82409dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46728/0 (socket says 172.21.15.81:46728) 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc81a7fc640 1 -- 172.21.15.81:0/1879836588 learned_addr learned my addr 172.21.15.81:0/1879836588 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc8237fe640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8241599a0 0x7fc824159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc822ffd640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8240b5250 0x7fc8240a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc81a7fc640 1 -- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8240b5250 msgr2=0x7fc8240a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc81a7fc640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8240b5250 0x7fc8240a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc81a7fc640 1 -- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8241599a0 msgr2=0x7fc824159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc81a7fc640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8241599a0 0x7fc824159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc81a7fc640 1 -- 172.21.15.81:0/1879836588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82409fd50 con 0x7fc8240b9040 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.426+0000 7fc81a7fc640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8240b9040 0x7fc82409dbd0 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7fc80c02ab90 tx=0x7fc80c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:39.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.427+0000 7fc81bfff640 1 -- 172.21.15.81:0/1879836588 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c031d60 con 0x7fc8240b9040 2024-06-29T15:59:39.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.427+0000 7fc81bfff640 1 -- 172.21.15.81:0/1879836588 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc80c033e10 con 0x7fc8240b9040 2024-06-29T15:59:39.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.427+0000 7fc82961f640 1 -- 172.21.15.81:0/1879836588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8240a0030 con 0x7fc8240b9040 2024-06-29T15:59:39.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.427+0000 7fc81bfff640 1 -- 172.21.15.81:0/1879836588 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c005260 con 0x7fc8240b9040 2024-06-29T15:59:39.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.427+0000 7fc82961f640 1 -- 172.21.15.81:0/1879836588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8240a05f0 con 0x7fc8240b9040 2024-06-29T15:59:39.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.428+0000 7fc82961f640 1 -- 172.21.15.81:0/1879836588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc82415e8c0 con 0x7fc8240b9040 2024-06-29T15:59:39.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.428+0000 7fc81bfff640 1 -- 172.21.15.81:0/1879836588 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fc80c005400 con 0x7fc8240b9040 2024-06-29T15:59:39.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.429+0000 7fc81bfff640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc804076420 0x7fc804078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.429+0000 7fc81bfff640 1 -- 172.21.15.81:0/1879836588 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 4985+0+0 (secure 0 0 0) 0x7fc80c0b8980 con 0x7fc8240b9040 2024-06-29T15:59:39.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.429+0000 7fc822ffd640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc804076420 0x7fc804078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.430+0000 7fc822ffd640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc804076420 0x7fc804078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc814006fd0 tx=0x7fc814008040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:39.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.433+0000 7fc81bfff640 1 -- 172.21.15.81:0/1879836588 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc82415e8c0 con 0x7fc8240b9040 2024-06-29T15:59:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.614+0000 7fc82961f640 1 -- 172.21.15.81:0/1879836588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc8240d2de0 con 0x7fc8240b9040 2024-06-29T15:59:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.615+0000 7fc81bfff640 1 -- 172.21.15.81:0/1879836588 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v262) ==== 93+0+10 (secure 0 0 0) 0x7fc80c03f030 con 0x7fc8240b9040 2024-06-29T15:59:39.615 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T15:59:39.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 -- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc804076420 msgr2=0x7fc804078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc804076420 0x7fc804078930 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc814006fd0 tx=0x7fc814008040 comp rx=0 tx=0).stop 2024-06-29T15:59:39.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 -- 172.21.15.81:0/1879836588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8240b9040 msgr2=0x7fc82409dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8240b9040 0x7fc82409dbd0 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7fc80c02ab90 tx=0x7fc80c033c20 comp rx=0 tx=0).stop 2024-06-29T15:59:39.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 -- 172.21.15.81:0/1879836588 shutdown_connections 2024-06-29T15:59:39.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc804076420 0x7fc804078930 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc8241599a0 0x7fc824159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8240b9040 0x7fc82409dbd0 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 --2- 172.21.15.81:0/1879836588 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8240b5250 0x7fc8240a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 -- 172.21.15.81:0/1879836588 >> 172.21.15.81:0/1879836588 conn(0x7fc82409cf30 msgr2=0x7fc824068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:39.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 -- 172.21.15.81:0/1879836588 shutdown_connections 2024-06-29T15:59:39.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.617+0000 7fc82961f640 1 -- 172.21.15.81:0/1879836588 wait complete. 2024-06-29T15:59:39.628 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-29T15:59:39.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.741+0000 7fc048f3c640 1 -- 172.21.15.81:0/3237835310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc02c004d40 msgr2=0x7fc02c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.741+0000 7fc048f3c640 1 --2- 172.21.15.81:0/3237835310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc02c004d40 0x7fc02c005150 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7fc0440cc280 tx=0x7fc02c005830 comp rx=0 tx=0).stop 2024-06-29T15:59:39.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.741+0000 7fc042d9d640 1 -- 172.21.15.81:0/3237835310 reap_dead start 2024-06-29T15:59:39.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.741+0000 7fc048f3c640 1 -- 172.21.15.81:0/3237835310 shutdown_connections 2024-06-29T15:59:39.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.741+0000 7fc048f3c640 1 -- 172.21.15.81:0/3237835310 >> 172.21.15.81:0/3237835310 conn(0x7fc04406ccc0 msgr2=0x7fc0440b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:39.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.741+0000 7fc048f3c640 1 -- 172.21.15.81:0/3237835310 shutdown_connections 2024-06-29T15:59:39.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.741+0000 7fc048f3c640 1 -- 172.21.15.81:0/3237835310 wait complete. 2024-06-29T15:59:39.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.743+0000 7fc048f3c640 1 Processor -- start 2024-06-29T15:59:39.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.743+0000 7fc048f3c640 1 -- start start 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.744+0000 7fc048f3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc02c004d40 0x7fc04415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc048f3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc04414f890 0x7fc044154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc048f3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc044072210 0x7fc044072620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc048f3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc0440cf790 con 0x7fc02c004d40 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc048f3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc0440cf610 con 0x7fc044072210 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc048f3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc04406edd0 con 0x7fc04414f890 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc04259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc02c004d40 0x7fc04415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc04259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc02c004d40 0x7fc04415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46732/0 (socket says 172.21.15.81:46732) 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc04259c640 1 -- 172.21.15.81:0/773500694 learned_addr learned my addr 172.21.15.81:0/773500694 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc04259c640 1 -- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc04414f890 msgr2=0x7fc044154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc04259c640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc04414f890 0x7fc044154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc04259c640 1 -- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc044072210 msgr2=0x7fc044072620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc04259c640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc044072210 0x7fc044072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc04259c640 1 -- 172.21.15.81:0/773500694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc044072b60 con 0x7fc02c004d40 2024-06-29T15:59:39.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc04259c640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc02c004d40 0x7fc04415a380 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7fc03400ee60 tx=0x7fc03400d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:39.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc032ffd640 1 -- 172.21.15.81:0/773500694 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03400de40 con 0x7fc02c004d40 2024-06-29T15:59:39.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc032ffd640 1 -- 172.21.15.81:0/773500694 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc034004020 con 0x7fc02c004d40 2024-06-29T15:59:39.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc032ffd640 1 -- 172.21.15.81:0/773500694 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc034004470 con 0x7fc02c004d40 2024-06-29T15:59:39.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc048f3c640 1 -- 172.21.15.81:0/773500694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc044155aa0 con 0x7fc02c004d40 2024-06-29T15:59:39.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.745+0000 7fc048f3c640 1 -- 172.21.15.81:0/773500694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc044157710 con 0x7fc02c004d40 2024-06-29T15:59:39.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.746+0000 7fc048f3c640 1 -- 172.21.15.81:0/773500694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc04406f2f0 con 0x7fc02c004d40 2024-06-29T15:59:39.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.747+0000 7fc032ffd640 1 -- 172.21.15.81:0/773500694 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fc03401f030 con 0x7fc02c004d40 2024-06-29T15:59:39.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.748+0000 7fc032ffd640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc0240763f0 0x7fc024078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:39.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.748+0000 7fc032ffd640 1 -- 172.21.15.81:0/773500694 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 4985+0+0 (secure 0 0 0) 0x7fc034099230 con 0x7fc02c004d40 2024-06-29T15:59:39.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.748+0000 7fc041d9b640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc0240763f0 0x7fc024078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:39.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.748+0000 7fc041d9b640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc0240763f0 0x7fc024078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc0440b1700 tx=0x7fc038009550 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:39.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.752+0000 7fc032ffd640 1 -- 172.21.15.81:0/773500694 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc034026020 con 0x7fc02c004d40 2024-06-29T15:59:39.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.933+0000 7fc048f3c640 1 -- 172.21.15.81:0/773500694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc0440b1a30 con 0x7fc02c004d40 2024-06-29T15:59:39.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.933+0000 7fc032ffd640 1 -- 172.21.15.81:0/773500694 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v262) ==== 84+0+65 (secure 0 0 0) 0x7fc044072b60 con 0x7fc02c004d40 2024-06-29T15:59:39.933 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:39.933 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:59:39.934 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:39.934 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-06-29T15:59:39.934 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-06-29T15:59:39.934 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:39.934 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:59:39.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 -- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc0240763f0 msgr2=0x7fc024078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc0240763f0 0x7fc024078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc0440b1700 tx=0x7fc038009550 comp rx=0 tx=0).stop 2024-06-29T15:59:39.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 -- 172.21.15.81:0/773500694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc02c004d40 msgr2=0x7fc04415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:39.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc02c004d40 0x7fc04415a380 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7fc03400ee60 tx=0x7fc03400d7b0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 -- 172.21.15.81:0/773500694 shutdown_connections 2024-06-29T15:59:39.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fc0240763f0 0x7fc024078900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc044072210 0x7fc044072620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc04414f890 0x7fc044154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 --2- 172.21.15.81:0/773500694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc02c004d40 0x7fc04415a380 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:39.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 -- 172.21.15.81:0/773500694 >> 172.21.15.81:0/773500694 conn(0x7fc04406ccc0 msgr2=0x7fc04415be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:39.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 -- 172.21.15.81:0/773500694 shutdown_connections 2024-06-29T15:59:39.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:39.935+0000 7fc048f3c640 1 -- 172.21.15.81:0/773500694 wait complete. 2024-06-29T15:59:39.946 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-29T15:59:39.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-29T15:59:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.063+0000 7f516e1eb640 1 -- 172.21.15.81:0/4036586522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5150004800 msgr2=0x7f5150004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.063+0000 7f516e1eb640 1 --2- 172.21.15.81:0/4036586522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5150004800 0x7f5150004c10 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f5168144dc0 tx=0x7f5150005830 comp rx=0 tx=0).stop 2024-06-29T15:59:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.063+0000 7f5167fff640 1 -- 172.21.15.81:0/4036586522 reap_dead start 2024-06-29T15:59:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.063+0000 7f516e1eb640 1 -- 172.21.15.81:0/4036586522 shutdown_connections 2024-06-29T15:59:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.063+0000 7f516e1eb640 1 -- 172.21.15.81:0/4036586522 >> 172.21.15.81:0/4036586522 conn(0x7f516806cd60 msgr2=0x7f51680b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.063+0000 7f516e1eb640 1 -- 172.21.15.81:0/4036586522 shutdown_connections 2024-06-29T15:59:40.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.063+0000 7f516e1eb640 1 -- 172.21.15.81:0/4036586522 wait complete. 2024-06-29T15:59:40.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.065+0000 7f516e1eb640 1 Processor -- start 2024-06-29T15:59:40.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.065+0000 7f516e1eb640 1 -- start start 2024-06-29T15:59:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f516e1eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5150004800 0x7f51680ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f516e1eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f516813d1e0 0x7f51681f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f516e1eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5168156d70 0x7f51681f87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f516e1eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f51680ce480 con 0x7f5150004800 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f516e1eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f51680ce300 con 0x7f5168156d70 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f516e1eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f51680ce600 con 0x7f516813d1e0 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f51677fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5150004800 0x7f51680ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f51677fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5150004800 0x7f51680ae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48156/0 (socket says 172.21.15.81:48156) 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f51677fe640 1 -- 172.21.15.81:0/552166618 learned_addr learned my addr 172.21.15.81:0/552166618 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f5167fff640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5168156d70 0x7f51681f87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f5166ffd640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f516813d1e0 0x7f51681f82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f51677fe640 1 -- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f516813d1e0 msgr2=0x7f51681f82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f51677fe640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f516813d1e0 0x7f51681f82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f51677fe640 1 -- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5168156d70 msgr2=0x7f51681f87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f51677fe640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5168156d70 0x7f51681f87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f51677fe640 1 -- 172.21.15.81:0/552166618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5168158a00 con 0x7f5150004800 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f5166ffd640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f516813d1e0 0x7f51681f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f51677fe640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5150004800 0x7f51680ae2b0 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f5158011a30 tx=0x7f5158011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f514ffff640 1 -- 172.21.15.81:0/552166618 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515800fdb0 con 0x7f5150004800 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f514ffff640 1 -- 172.21.15.81:0/552166618 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5158005020 con 0x7f5150004800 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f516e1eb640 1 -- 172.21.15.81:0/552166618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5168158ce0 con 0x7f5150004800 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.067+0000 7f514ffff640 1 -- 172.21.15.81:0/552166618 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5158005470 con 0x7f5150004800 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.068+0000 7f516e1eb640 1 -- 172.21.15.81:0/552166618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51681592a0 con 0x7f5150004800 2024-06-29T15:59:40.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.069+0000 7f516e1eb640 1 -- 172.21.15.81:0/552166618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51680ceb90 con 0x7f5150004800 2024-06-29T15:59:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.069+0000 7f514ffff640 1 -- 172.21.15.81:0/552166618 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f51580047a0 con 0x7f5150004800 2024-06-29T15:59:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.070+0000 7f514ffff640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f51540763f0 0x7f5154078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.070+0000 7f514ffff640 1 -- 172.21.15.81:0/552166618 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 4985+0+0 (secure 0 0 0) 0x7f5158098520 con 0x7f5150004800 2024-06-29T15:59:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.070+0000 7f5166ffd640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f51540763f0 0x7f5154078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.071+0000 7f5166ffd640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f51540763f0 0x7f5154078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f515c032000 tx=0x7f515c02c070 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:40.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.074+0000 7f514ffff640 1 -- 172.21.15.81:0/552166618 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5158061fb0 con 0x7f5150004800 2024-06-29T15:59:40.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.254+0000 7f516e1eb640 1 -- 172.21.15.81:0/552166618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f516806f300 con 0x7f5150004800 2024-06-29T15:59:40.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.587+0000 7f514ffff640 1 -- 172.21.15.81:0/552166618 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v263) ==== 138+0+0 (secure 0 0 0) 0x7f51580618b0 con 0x7f5150004800 2024-06-29T15:59:40.587 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-06-29T15:59:40.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.589+0000 7f516e1eb640 1 -- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f51540763f0 msgr2=0x7f5154078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.589+0000 7f516e1eb640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f51540763f0 0x7f5154078900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f515c032000 tx=0x7f515c02c070 comp rx=0 tx=0).stop 2024-06-29T15:59:40.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.590+0000 7f516e1eb640 1 -- 172.21.15.81:0/552166618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5150004800 msgr2=0x7f51680ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.590+0000 7f516e1eb640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5150004800 0x7f51680ae2b0 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f5158011a30 tx=0x7f5158011ef0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.590+0000 7f516e1eb640 1 -- 172.21.15.81:0/552166618 shutdown_connections 2024-06-29T15:59:40.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.590+0000 7f516e1eb640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f51540763f0 0x7f5154078900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.590+0000 7f516e1eb640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5168156d70 0x7f51681f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.590+0000 7f516e1eb640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f516813d1e0 0x7f51681f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.590+0000 7f516e1eb640 1 --2- 172.21.15.81:0/552166618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5150004800 0x7f51680ae2b0 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.590+0000 7f516e1eb640 1 -- 172.21.15.81:0/552166618 >> 172.21.15.81:0/552166618 conn(0x7f516806cd60 msgr2=0x7f516815dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:40.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.590+0000 7f516e1eb640 1 -- 172.21.15.81:0/552166618 shutdown_connections 2024-06-29T15:59:40.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.590+0000 7f516e1eb640 1 -- 172.21.15.81:0/552166618 wait complete. 2024-06-29T15:59:40.605 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-06-29T15:59:40.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.726+0000 7fb13d39e640 1 -- 172.21.15.81:0/3160109185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1380b4f40 msgr2=0x7fb1380b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.726+0000 7fb13d39e640 1 --2- 172.21.15.81:0/3160109185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1380b4f40 0x7fb1380b5350 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7fb138074b10 tx=0x7fb13002a560 comp rx=0 tx=0).stop 2024-06-29T15:59:40.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.726+0000 7fb13d39e640 1 -- 172.21.15.81:0/3160109185 shutdown_connections 2024-06-29T15:59:40.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.726+0000 7fb13d39e640 1 --2- 172.21.15.81:0/3160109185 >> v2:172.21.15.167:3300/0 conn(0x7fb1380d0230 0x7fb1380d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.726+0000 7fb13d39e640 1 --2- 172.21.15.81:0/3160109185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1380b4f40 0x7fb1380b5350 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.726+0000 7fb13d39e640 1 -- 172.21.15.81:0/3160109185 >> 172.21.15.81:0/3160109185 conn(0x7fb1380b3680 msgr2=0x7fb1380b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:40.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.726+0000 7fb13d39e640 1 -- 172.21.15.81:0/3160109185 shutdown_connections 2024-06-29T15:59:40.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.726+0000 7fb13d39e640 1 -- 172.21.15.81:0/3160109185 wait complete. 2024-06-29T15:59:40.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.728+0000 7fb13d39e640 1 Processor -- start 2024-06-29T15:59:40.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.728+0000 7fb13d39e640 1 -- start start 2024-06-29T15:59:40.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.729+0000 7fb13d39e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1380b3200 0x7fb138156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.729+0000 7fb13d39e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1380b4f40 0x7fb138157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.729+0000 7fb13d39e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1380d0230 0x7fb138159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.729+0000 7fb13d39e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb1380b5c20 con 0x7fb1380b3200 2024-06-29T15:59:40.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.729+0000 7fb13d39e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fb1380b5aa0 con 0x7fb1380b4f40 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.729+0000 7fb13d39e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fb1380b5da0 con 0x7fb1380d0230 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.729+0000 7fb136ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1380b3200 0x7fb138156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.729+0000 7fb136ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1380b3200 0x7fb138156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48178/0 (socket says 172.21.15.81:48178) 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.729+0000 7fb136ffd640 1 -- 172.21.15.81:0/3514556496 learned_addr learned my addr 172.21.15.81:0/3514556496 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb1367fc640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1380b4f40 0x7fb138157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb1377fe640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1380d0230 0x7fb138159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb136ffd640 1 -- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1380d0230 msgr2=0x7fb138159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb136ffd640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1380d0230 0x7fb138159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb136ffd640 1 -- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1380b4f40 msgr2=0x7fb138157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb136ffd640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1380b4f40 0x7fb138157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb136ffd640 1 -- 172.21.15.81:0/3514556496 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1381e3010 con 0x7fb1380b3200 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb136ffd640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1380b3200 0x7fb138156010 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fb12800c760 tx=0x7fb12800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb12f7fe640 1 -- 172.21.15.81:0/3514556496 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128003b70 con 0x7fb1380b3200 2024-06-29T15:59:40.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb12f7fe640 1 -- 172.21.15.81:0/3514556496 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb128003d30 con 0x7fb1380b3200 2024-06-29T15:59:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb12f7fe640 1 -- 172.21.15.81:0/3514556496 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12800aab0 con 0x7fb1380b3200 2024-06-29T15:59:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb13d39e640 1 -- 172.21.15.81:0/3514556496 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1381e3280 con 0x7fb1380b3200 2024-06-29T15:59:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.730+0000 7fb13d39e640 1 -- 172.21.15.81:0/3514556496 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1381e3840 con 0x7fb1380b3200 2024-06-29T15:59:40.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.731+0000 7fb13d39e640 1 -- 172.21.15.81:0/3514556496 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1380b5350 con 0x7fb1380b3200 2024-06-29T15:59:40.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.732+0000 7fb12f7fe640 1 -- 172.21.15.81:0/3514556496 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fb12800ac50 con 0x7fb1380b3200 2024-06-29T15:59:40.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.732+0000 7fb12f7fe640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb118076420 0x7fb118078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:40.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.733+0000 7fb12f7fe640 1 -- 172.21.15.81:0/3514556496 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5355+0+0 (secure 0 0 0) 0x7fb1280974a0 con 0x7fb1380b3200 2024-06-29T15:59:40.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.733+0000 7fb1367fc640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb118076420 0x7fb118078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:40.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.733+0000 7fb1367fc640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb118076420 0x7fb118078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb13815aa50 tx=0x7fb12000b2e0 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:40.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.737+0000 7fb12f7fe640 1 -- 172.21.15.81:0/3514556496 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fb1380d0b80 con 0x7fb1380b3200 2024-06-29T15:59:40.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:40.915+0000 7fb13d39e640 1 -- 172.21.15.81:0/3514556496 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb13815aa80 con 0x7fb1380b3200 2024-06-29T15:59:41.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.593+0000 7fb12f7fe640 1 -- 172.21.15.81:0/3514556496 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v264) ==== 170+0+0 (secure 0 0 0) 0x7fb128060800 con 0x7fb1380b3200 2024-06-29T15:59:41.594 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-06-29T15:59:41.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.596+0000 7fb13d39e640 1 -- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb118076420 msgr2=0x7fb118078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.596+0000 7fb13d39e640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb118076420 0x7fb118078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb13815aa50 tx=0x7fb12000b2e0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.596+0000 7fb13d39e640 1 -- 172.21.15.81:0/3514556496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1380b3200 msgr2=0x7fb138156010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.596+0000 7fb13d39e640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1380b3200 0x7fb138156010 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fb12800c760 tx=0x7fb12800cc20 comp rx=0 tx=0).stop 2024-06-29T15:59:41.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.597+0000 7fb13d39e640 1 -- 172.21.15.81:0/3514556496 shutdown_connections 2024-06-29T15:59:41.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.597+0000 7fb13d39e640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fb118076420 0x7fb118078930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.597+0000 7fb13d39e640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fb1380d0230 0x7fb138159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.597+0000 7fb13d39e640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fb1380b4f40 0x7fb138157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.597+0000 7fb13d39e640 1 --2- 172.21.15.81:0/3514556496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1380b3200 0x7fb138156010 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.597+0000 7fb13d39e640 1 -- 172.21.15.81:0/3514556496 >> 172.21.15.81:0/3514556496 conn(0x7fb1380b3680 msgr2=0x7fb1380c8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:41.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.597+0000 7fb13d39e640 1 -- 172.21.15.81:0/3514556496 shutdown_connections 2024-06-29T15:59:41.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.597+0000 7fb13d39e640 1 -- 172.21.15.81:0/3514556496 wait complete. 2024-06-29T15:59:41.618 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.737+0000 7fd260b54640 1 -- 172.21.15.81:0/607011253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd248004d40 msgr2=0x7fd2480051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.737+0000 7fd260b54640 1 --2- 172.21.15.81:0/607011253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd248004d40 0x7fd2480051f0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7fd254001d60 tx=0x7fd248005830 comp rx=0 tx=0).stop 2024-06-29T15:59:41.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.737+0000 7fd25ad9d640 1 -- 172.21.15.81:0/607011253 reap_dead start 2024-06-29T15:59:41.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.737+0000 7fd260b54640 1 -- 172.21.15.81:0/607011253 shutdown_connections 2024-06-29T15:59:41.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.737+0000 7fd260b54640 1 -- 172.21.15.81:0/607011253 >> 172.21.15.81:0/607011253 conn(0x7fd25c0b3680 msgr2=0x7fd25c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:41.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.737+0000 7fd260b54640 1 -- 172.21.15.81:0/607011253 shutdown_connections 2024-06-29T15:59:41.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.737+0000 7fd260b54640 1 -- 172.21.15.81:0/607011253 wait complete. 2024-06-29T15:59:41.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.739+0000 7fd260b54640 1 Processor -- start 2024-06-29T15:59:41.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.739+0000 7fd260b54640 1 -- start start 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd260b54640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd25c0cb510 0x7fd25c1dae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd260b54640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd25c1e0a60 0x7fd25c1e3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd260b54640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd25c1e3bc0 0x7fd25c0c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd260b54640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c0d09d0 con 0x7fd25c0cb510 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd260b54640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c0d0850 con 0x7fd25c1e0a60 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd260b54640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd25c0b5a60 con 0x7fd25c1e3bc0 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd25a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd25c0cb510 0x7fd25c1dae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd25a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd25c0cb510 0x7fd25c1dae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48196/0 (socket says 172.21.15.81:48196) 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd25a59c640 1 -- 172.21.15.81:0/1858526618 learned_addr learned my addr 172.21.15.81:0/1858526618 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd25ad9d640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd25c1e3bc0 0x7fd25c0c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.740+0000 7fd251d9b640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd25c1e0a60 0x7fd25c1e3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd25a59c640 1 -- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd25c1e3bc0 msgr2=0x7fd25c0c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd25a59c640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd25c1e3bc0 0x7fd25c0c94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd25a59c640 1 -- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd25c1e0a60 msgr2=0x7fd25c1e3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd25a59c640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd25c1e0a60 0x7fd25c1e3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd25a59c640 1 -- 172.21.15.81:0/1858526618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c1e3fd0 con 0x7fd25c0cb510 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd25a59c640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd25c0cb510 0x7fd25c1dae90 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fd24c001970 tx=0x7fd24c003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd2537fe640 1 -- 172.21.15.81:0/1858526618 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c011700 con 0x7fd25c0cb510 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd2537fe640 1 -- 172.21.15.81:0/1858526618 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd24c016020 con 0x7fd25c0cb510 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd260b54640 1 -- 172.21.15.81:0/1858526618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25c1e12b0 con 0x7fd25c0cb510 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd2537fe640 1 -- 172.21.15.81:0/1858526618 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c011b00 con 0x7fd25c0cb510 2024-06-29T15:59:41.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.741+0000 7fd260b54640 1 -- 172.21.15.81:0/1858526618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25c1dbc80 con 0x7fd25c0cb510 2024-06-29T15:59:41.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.742+0000 7fd260b54640 1 -- 172.21.15.81:0/1858526618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd25c0b5f50 con 0x7fd25c0cb510 2024-06-29T15:59:41.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.743+0000 7fd2537fe640 1 -- 172.21.15.81:0/1858526618 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fd24c011ca0 con 0x7fd25c0cb510 2024-06-29T15:59:41.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.743+0000 7fd2537fe640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd23c0763f0 0x7fd23c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:41.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.744+0000 7fd2537fe640 1 -- 172.21.15.81:0/1858526618 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(264..264 src has 1..264) ==== 5737+0+0 (secure 0 0 0) 0x7fd24c098870 con 0x7fd25c0cb510 2024-06-29T15:59:41.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.744+0000 7fd251d9b640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd23c0763f0 0x7fd23c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:41.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.744+0000 7fd251d9b640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd23c0763f0 0x7fd23c078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0707f0 tx=0x7fd244005f90 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:41.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.747+0000 7fd2537fe640 1 -- 172.21.15.81:0/1858526618 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd24c020020 con 0x7fd25c0cb510 2024-06-29T15:59:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:41.964+0000 7fd260b54640 1 -- 172.21.15.81:0/1858526618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd25c0710c0 con 0x7fd25c0cb510 2024-06-29T15:59:42.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.608+0000 7fd2537fe640 1 -- 172.21.15.81:0/1858526618 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 36 and data pool 37 v150) ==== 174+0+0 (secure 0 0 0) 0x7fd25c0d0b80 con 0x7fd25c0cb510 2024-06-29T15:59:42.609 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 36 and data pool 37 2024-06-29T15:59:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.614+0000 7fd260b54640 1 -- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd23c0763f0 msgr2=0x7fd23c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.614+0000 7fd260b54640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd23c0763f0 0x7fd23c078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0707f0 tx=0x7fd244005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.614+0000 7fd260b54640 1 -- 172.21.15.81:0/1858526618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd25c0cb510 msgr2=0x7fd25c1dae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.614+0000 7fd260b54640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd25c0cb510 0x7fd25c1dae90 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fd24c001970 tx=0x7fd24c003aa0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.615+0000 7fd260b54640 1 -- 172.21.15.81:0/1858526618 shutdown_connections 2024-06-29T15:59:42.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.615+0000 7fd260b54640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd23c0763f0 0x7fd23c078900 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0707f0 tx=0x7fd244005f90 comp rx=0 tx=0).stop 2024-06-29T15:59:42.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.615+0000 7fd260b54640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd25c1e3bc0 0x7fd25c0c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.615+0000 7fd260b54640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd25c1e0a60 0x7fd25c1e3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.615+0000 7fd260b54640 1 --2- 172.21.15.81:0/1858526618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd25c0cb510 0x7fd25c1dae90 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.615+0000 7fd260b54640 1 -- 172.21.15.81:0/1858526618 >> 172.21.15.81:0/1858526618 conn(0x7fd25c0b3680 msgr2=0x7fd25c0d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:42.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.624+0000 7fd260b54640 1 -- 172.21.15.81:0/1858526618 shutdown_connections 2024-06-29T15:59:42.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.625+0000 7fd260b54640 1 -- 172.21.15.81:0/1858526618 wait complete. 2024-06-29T15:59:42.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:42.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.759+0000 7f7d8e2df640 1 -- 172.21.15.81:0/4282926553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d8815b5b0 msgr2=0x7f7d880b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.759+0000 7f7d8e2df640 1 --2- 172.21.15.81:0/4282926553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d8815b5b0 0x7f7d880b4430 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f7d70005fd0 tx=0x7f7d7002a540 comp rx=0 tx=0).stop 2024-06-29T15:59:42.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.759+0000 7f7d8e2df640 1 -- 172.21.15.81:0/4282926553 shutdown_connections 2024-06-29T15:59:42.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.759+0000 7f7d8e2df640 1 --2- 172.21.15.81:0/4282926553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d8815b5b0 0x7f7d880b4430 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.759+0000 7f7d8e2df640 1 -- 172.21.15.81:0/4282926553 >> 172.21.15.81:0/4282926553 conn(0x7f7d8806ccc0 msgr2=0x7f7d880b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:42.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.759+0000 7f7d8e2df640 1 -- 172.21.15.81:0/4282926553 shutdown_connections 2024-06-29T15:59:42.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.759+0000 7f7d8e2df640 1 -- 172.21.15.81:0/4282926553 wait complete. 2024-06-29T15:59:42.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.761+0000 7f7d8e2df640 1 Processor -- start 2024-06-29T15:59:42.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.761+0000 7f7d8e2df640 1 -- start start 2024-06-29T15:59:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.762+0000 7f7d8e2df640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d8806f660 0x7f7d8814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.762+0000 7f7d8e2df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880b35b0 0x7f7d88155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.762+0000 7f7d8e2df640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d8815b5b0 0x7f7d88158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d8e2df640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7d880725b0 con 0x7f7d880b35b0 2024-06-29T15:59:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d8e2df640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7d88072430 con 0x7f7d8806f660 2024-06-29T15:59:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d8e2df640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7d88072730 con 0x7f7d8815b5b0 2024-06-29T15:59:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d877fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880b35b0 0x7f7d88155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d877fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880b35b0 0x7f7d88155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48220/0 (socket says 172.21.15.81:48220) 2024-06-29T15:59:42.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d877fe640 1 -- 172.21.15.81:0/336636891 learned_addr learned my addr 172.21.15.81:0/336636891 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:42.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d8c87c640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d8815b5b0 0x7f7d88158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d87fff640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d8806f660 0x7f7d8814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d877fe640 1 -- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d8815b5b0 msgr2=0x7f7d88158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d877fe640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d8815b5b0 0x7f7d88158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d877fe640 1 -- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d8806f660 msgr2=0x7f7d8814a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d877fe640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d8806f660 0x7f7d8814a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d877fe640 1 -- 172.21.15.81:0/336636891 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d88156490 con 0x7f7d880b35b0 2024-06-29T15:59:42.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d87fff640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d8806f660 0x7f7d8814a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:42.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d877fe640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880b35b0 0x7f7d88155f50 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f7d70034040 tx=0x7f7d70003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:42.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d84ff9640 1 -- 172.21.15.81:0/336636891 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7003e030 con 0x7f7d880b35b0 2024-06-29T15:59:42.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d84ff9640 1 -- 172.21.15.81:0/336636891 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d700047a0 con 0x7f7d880b35b0 2024-06-29T15:59:42.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d8e2df640 1 -- 172.21.15.81:0/336636891 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d88155190 con 0x7f7d880b35b0 2024-06-29T15:59:42.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d84ff9640 1 -- 172.21.15.81:0/336636891 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d70035040 con 0x7f7d880b35b0 2024-06-29T15:59:42.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.763+0000 7f7d8e2df640 1 -- 172.21.15.81:0/336636891 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d88155750 con 0x7f7d880b35b0 2024-06-29T15:59:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.765+0000 7f7d8e2df640 1 -- 172.21.15.81:0/336636891 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d88154940 con 0x7f7d880b35b0 2024-06-29T15:59:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.765+0000 7f7d84ff9640 1 -- 172.21.15.81:0/336636891 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f7d70041030 con 0x7f7d880b35b0 2024-06-29T15:59:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.765+0000 7f7d84ff9640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7d68076420 0x7f7d68078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.766+0000 7f7d84ff9640 1 -- 172.21.15.81:0/336636891 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(265..265 src has 1..265) ==== 5837+0+0 (secure 0 0 0) 0x7f7d700b81f0 con 0x7f7d880b35b0 2024-06-29T15:59:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.766+0000 7f7d87fff640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7d68076420 0x7f7d68078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:42.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.766+0000 7f7d87fff640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7d68076420 0x7f7d68078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7d78006fd0 tx=0x7f7d78008040 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:42.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.770+0000 7f7d84ff9640 1 -- 172.21.15.81:0/336636891 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7d700811e0 con 0x7f7d880b35b0 2024-06-29T15:59:42.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.953+0000 7f7d8e2df640 1 -- 172.21.15.81:0/336636891 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d88068bd0 con 0x7f7d880b35b0 2024-06-29T15:59:42.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.954+0000 7f7d84ff9640 1 -- 172.21.15.81:0/336636891 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) ==== 74+0+16832 (secure 0 0 0) 0x7f7d88068bd0 con 0x7f7d880b35b0 2024-06-29T15:59:42.954 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:42.955 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":265,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:41.965774+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:40.261824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:40.923216+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 -- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7d68076420 msgr2=0x7f7d68078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7d68076420 0x7f7d68078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7d78006fd0 tx=0x7f7d78008040 comp rx=0 tx=0).stop 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 -- 172.21.15.81:0/336636891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880b35b0 msgr2=0x7f7d88155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880b35b0 0x7f7d88155f50 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f7d70034040 tx=0x7f7d70003970 comp rx=0 tx=0).stop 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 -- 172.21.15.81:0/336636891 shutdown_connections 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f7d68076420 0x7f7d68078930 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7d8815b5b0 0x7f7d88158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880b35b0 0x7f7d88155f50 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 --2- 172.21.15.81:0/336636891 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7d8806f660 0x7f7d8814a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 -- 172.21.15.81:0/336636891 >> 172.21.15.81:0/336636891 conn(0x7f7d8806ccc0 msgr2=0x7f7d8814c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 -- 172.21.15.81:0/336636891 shutdown_connections 2024-06-29T15:59:42.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:42.956+0000 7f7d8e2df640 1 -- 172.21.15.81:0/336636891 wait complete. 2024-06-29T15:59:42.968 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:43.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.082+0000 7f5630111640 1 -- 172.21.15.81:0/185534934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562806ef90 msgr2=0x7f562806f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.082+0000 7f5630111640 1 --2- 172.21.15.81:0/185534934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562806ef90 0x7f562806f3a0 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7f56280b53d0 tx=0x7f561802a540 comp rx=0 tx=0).stop 2024-06-29T15:59:43.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.082+0000 7f5630111640 1 -- 172.21.15.81:0/185534934 shutdown_connections 2024-06-29T15:59:43.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.082+0000 7f5630111640 1 --2- 172.21.15.81:0/185534934 >> v2:172.21.15.167:3300/0 conn(0x7f56280ce1d0 0x7f56280ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.082+0000 7f5630111640 1 --2- 172.21.15.81:0/185534934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562806ef90 0x7f562806f3a0 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.082+0000 7f5630111640 1 --2- 172.21.15.81:0/185534934 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56280b57b0 0x7f562815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.082+0000 7f5630111640 1 -- 172.21.15.81:0/185534934 >> 172.21.15.81:0/185534934 conn(0x7f56280d0930 msgr2=0x7f56280d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:43.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.082+0000 7f5630111640 1 -- 172.21.15.81:0/185534934 shutdown_connections 2024-06-29T15:59:43.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.082+0000 7f5630111640 1 -- 172.21.15.81:0/185534934 wait complete. 2024-06-29T15:59:43.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.084+0000 7f5630111640 1 Processor -- start 2024-06-29T15:59:43.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.084+0000 7f5630111640 1 -- start start 2024-06-29T15:59:43.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f5630111640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f562806ef90 0x7f56280adc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f5630111640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56280b57b0 0x7f56280b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f5630111640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56280ce1d0 0x7f562806c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f5630111640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f56280ceeb0 con 0x7f56280b57b0 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f5630111640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f56280ced30 con 0x7f56280ce1d0 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f5630111640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f56280cf030 con 0x7f562806ef90 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f562d6ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56280b57b0 0x7f56280b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f562d6ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56280b57b0 0x7f56280b1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48238/0 (socket says 172.21.15.81:48238) 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f562d6ac640 1 -- 172.21.15.81:0/3829084304 learned_addr learned my addr 172.21.15.81:0/3829084304 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f562dead640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f562806ef90 0x7f56280adc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f562e6ae640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56280ce1d0 0x7f562806c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f562d6ac640 1 -- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f562806ef90 msgr2=0x7f56280adc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f562d6ac640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f562806ef90 0x7f56280adc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f562d6ac640 1 -- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56280ce1d0 msgr2=0x7f562806c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f562d6ac640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56280ce1d0 0x7f562806c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.085+0000 7f562d6ac640 1 -- 172.21.15.81:0/3829084304 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56280ae1d0 con 0x7f56280b57b0 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.086+0000 7f562d6ac640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56280b57b0 0x7f56280b1db0 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f5618005fd0 tx=0x7f56180039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.086+0000 7f561e7fc640 1 -- 172.21.15.81:0/3829084304 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561803e030 con 0x7f56280b57b0 2024-06-29T15:59:43.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.086+0000 7f561e7fc640 1 -- 172.21.15.81:0/3829084304 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56180047a0 con 0x7f56280b57b0 2024-06-29T15:59:43.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.086+0000 7f5630111640 1 -- 172.21.15.81:0/3829084304 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f562814c4e0 con 0x7f56280b57b0 2024-06-29T15:59:43.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.086+0000 7f561e7fc640 1 -- 172.21.15.81:0/3829084304 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618035040 con 0x7f56280b57b0 2024-06-29T15:59:43.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.086+0000 7f5630111640 1 -- 172.21.15.81:0/3829084304 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f562814caa0 con 0x7f56280b57b0 2024-06-29T15:59:43.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.087+0000 7f5630111640 1 -- 172.21.15.81:0/3829084304 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f562806e2a0 con 0x7f56280b57b0 2024-06-29T15:59:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.088+0000 7f561e7fc640 1 -- 172.21.15.81:0/3829084304 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f5618041030 con 0x7f56280b57b0 2024-06-29T15:59:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.088+0000 7f561e7fc640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f560c076420 0x7f560c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.088+0000 7f561e7fc640 1 -- 172.21.15.81:0/3829084304 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(265..265 src has 1..265) ==== 5837+0+0 (secure 0 0 0) 0x7f56180b8110 con 0x7f56280b57b0 2024-06-29T15:59:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.089+0000 7f562dead640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f560c076420 0x7f560c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.089+0000 7f562dead640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f560c076420 0x7f560c078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5624005fd0 tx=0x7f5624009560 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:43.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.092+0000 7f561e7fc640 1 -- 172.21.15.81:0/3829084304 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f562806e2a0 con 0x7f56280b57b0 2024-06-29T15:59:43.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.272+0000 7f5630111640 1 -- 172.21.15.81:0/3829084304 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f562814c270 con 0x7f56280b57b0 2024-06-29T15:59:43.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.273+0000 7f561e7fc640 1 -- 172.21.15.81:0/3829084304 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v265) ==== 74+0+16832 (secure 0 0 0) 0x7f56180810f0 con 0x7f56280b57b0 2024-06-29T15:59:43.273 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:43.274 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":265,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:41.965774+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:40.261824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:40.923216+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:43.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 -- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f560c076420 msgr2=0x7f560c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f560c076420 0x7f560c078930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5624005fd0 tx=0x7f5624009560 comp rx=0 tx=0).stop 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 -- 172.21.15.81:0/3829084304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56280b57b0 msgr2=0x7f56280b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56280b57b0 0x7f56280b1db0 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f5618005fd0 tx=0x7f56180039e0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 -- 172.21.15.81:0/3829084304 shutdown_connections 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f560c076420 0x7f560c078930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56280ce1d0 0x7f562806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56280b57b0 0x7f56280b1db0 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 --2- 172.21.15.81:0/3829084304 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f562806ef90 0x7f56280adc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 -- 172.21.15.81:0/3829084304 >> 172.21.15.81:0/3829084304 conn(0x7f56280d0930 msgr2=0x7f56280cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 -- 172.21.15.81:0/3829084304 shutdown_connections 2024-06-29T15:59:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.275+0000 7f5630111640 1 -- 172.21.15.81:0/3829084304 wait complete. 2024-06-29T15:59:43.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-06-29T15:59:43.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.404+0000 7f856dd9f640 1 -- 172.21.15.81:0/3532471385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f856815c030 msgr2=0x7f856806c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.404+0000 7f856dd9f640 1 --2- 172.21.15.81:0/3532471385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f856815c030 0x7f856806c2f0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f8568071e60 tx=0x7f855c02a560 comp rx=0 tx=0).stop 2024-06-29T15:59:43.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.404+0000 7f856dd9f640 1 -- 172.21.15.81:0/3532471385 shutdown_connections 2024-06-29T15:59:43.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.404+0000 7f856dd9f640 1 --2- 172.21.15.81:0/3532471385 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f856815c440 0x7f856806c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.404+0000 7f856dd9f640 1 --2- 172.21.15.81:0/3532471385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f856815c030 0x7f856806c2f0 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.404+0000 7f856dd9f640 1 --2- 172.21.15.81:0/3532471385 >> v2:172.21.15.167:3300/0 conn(0x7f856815d750 0x7f85680afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.404+0000 7f856dd9f640 1 -- 172.21.15.81:0/3532471385 >> 172.21.15.81:0/3532471385 conn(0x7f85680d0930 msgr2=0x7f85680d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:43.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.404+0000 7f856dd9f640 1 -- 172.21.15.81:0/3532471385 shutdown_connections 2024-06-29T15:59:43.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.404+0000 7f856dd9f640 1 -- 172.21.15.81:0/3532471385 wait complete. 2024-06-29T15:59:43.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.405+0000 7f856dd9f640 1 Processor -- start 2024-06-29T15:59:43.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.406+0000 7f856dd9f640 1 -- start start 2024-06-29T15:59:43.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f856dd9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f856815c030 0x7f8568158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f856dd9f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f856815c440 0x7f8568150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f856dd9f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f856815d750 0x7f856806ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f856dd9f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f856815ff10 con 0x7f856815c030 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f856dd9f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f856815fd90 con 0x7f856815c440 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f856dd9f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8568160090 con 0x7f856815d750 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f85677fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f856815c030 0x7f8568158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f85677fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f856815c030 0x7f8568158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48256/0 (socket says 172.21.15.81:48256) 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f85677fe640 1 -- 172.21.15.81:0/3593986615 learned_addr learned my addr 172.21.15.81:0/3593986615 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f8567fff640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f856815d750 0x7f856806ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f8566ffd640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f856815c440 0x7f8568150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f85677fe640 1 -- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f856815d750 msgr2=0x7f856806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f85677fe640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f856815d750 0x7f856806ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f85677fe640 1 -- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f856815c440 msgr2=0x7f8568150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f85677fe640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f856815c440 0x7f8568150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:43.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f85677fe640 1 -- 172.21.15.81:0/3593986615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85681e8de0 con 0x7f856815c030 2024-06-29T15:59:43.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.407+0000 7f8566ffd640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f856815c440 0x7f8568150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:43.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.408+0000 7f85677fe640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f856815c030 0x7f8568158b10 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f855800ea20 tx=0x7f855800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:43.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.408+0000 7f854ffff640 1 -- 172.21.15.81:0/3593986615 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855800cde0 con 0x7f856815c030 2024-06-29T15:59:43.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.408+0000 7f854ffff640 1 -- 172.21.15.81:0/3593986615 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8558011560 con 0x7f856815c030 2024-06-29T15:59:43.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.408+0000 7f856dd9f640 1 -- 172.21.15.81:0/3593986615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85681e90c0 con 0x7f856815c030 2024-06-29T15:59:43.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.408+0000 7f854ffff640 1 -- 172.21.15.81:0/3593986615 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8558011990 con 0x7f856815c030 2024-06-29T15:59:43.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.408+0000 7f856dd9f640 1 -- 172.21.15.81:0/3593986615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85681564b0 con 0x7f856815c030 2024-06-29T15:59:43.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.409+0000 7f856dd9f640 1 -- 172.21.15.81:0/3593986615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f856806e2a0 con 0x7f856815c030 2024-06-29T15:59:43.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.410+0000 7f854ffff640 1 -- 172.21.15.81:0/3593986615 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f855801a400 con 0x7f856815c030 2024-06-29T15:59:43.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.410+0000 7f854ffff640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f85540763f0 0x7f8554078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:43.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.410+0000 7f854ffff640 1 -- 172.21.15.81:0/3593986615 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(265..265 src has 1..265) ==== 5837+0+0 (secure 0 0 0) 0x7f85580986b0 con 0x7f856815c030 2024-06-29T15:59:43.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.410+0000 7f8566ffd640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f85540763f0 0x7f8554078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:43.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.411+0000 7f8566ffd640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f85540763f0 0x7f8554078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f855c033000 tx=0x7f855c004730 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:43.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.414+0000 7f854ffff640 1 -- 172.21.15.81:0/3593986615 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f856806e2a0 con 0x7f856815c030 2024-06-29T15:59:43.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:43.641+0000 7f856dd9f640 1 -- 172.21.15.81:0/3593986615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f85680afb60 con 0x7f856815c030 2024-06-29T15:59:44.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.614+0000 7f854ffff640 1 -- 172.21.15.81:0/3593986615 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v153) ==== 118+0+0 (secure 0 0 0) 0x7f8558016030 con 0x7f856815c030 2024-06-29T15:59:44.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.617+0000 7f856dd9f640 1 -- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f85540763f0 msgr2=0x7f8554078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:44.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.617+0000 7f856dd9f640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f85540763f0 0x7f8554078900 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f855c033000 tx=0x7f855c004730 comp rx=0 tx=0).stop 2024-06-29T15:59:44.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.617+0000 7f856dd9f640 1 -- 172.21.15.81:0/3593986615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f856815c030 msgr2=0x7f8568158b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:44.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.617+0000 7f856dd9f640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f856815c030 0x7f8568158b10 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f855800ea20 tx=0x7f855800eee0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.617+0000 7f856dd9f640 1 -- 172.21.15.81:0/3593986615 shutdown_connections 2024-06-29T15:59:44.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.617+0000 7f856dd9f640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f85540763f0 0x7f8554078900 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.617+0000 7f856dd9f640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f856815d750 0x7f856806ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.617+0000 7f856dd9f640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f856815c440 0x7f8568150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.617+0000 7f856dd9f640 1 --2- 172.21.15.81:0/3593986615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f856815c030 0x7f8568158b10 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.617+0000 7f856dd9f640 1 -- 172.21.15.81:0/3593986615 >> 172.21.15.81:0/3593986615 conn(0x7f85680d0930 msgr2=0x7f85680d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:44.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.618+0000 7f856dd9f640 1 -- 172.21.15.81:0/3593986615 shutdown_connections 2024-06-29T15:59:44.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.618+0000 7f856dd9f640 1 -- 172.21.15.81:0/3593986615 wait complete. 2024-06-29T15:59:44.632 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-06-29T15:59:44.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-06-29T15:59:44.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.754+0000 7fddb07f5640 1 -- 172.21.15.81:0/2623502939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd94004800 msgr2=0x7fdd94004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:44.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.754+0000 7fddb07f5640 1 --2- 172.21.15.81:0/2623502939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd94004800 0x7fdd94004c10 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7fdda80b2ae0 tx=0x7fdd94005830 comp rx=0 tx=0).stop 2024-06-29T15:59:44.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.754+0000 7fddb07f5640 1 -- 172.21.15.81:0/2623502939 shutdown_connections 2024-06-29T15:59:44.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.754+0000 7fddb07f5640 1 --2- 172.21.15.81:0/2623502939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd94004800 0x7fdd94004c10 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.754+0000 7fddb07f5640 1 -- 172.21.15.81:0/2623502939 >> 172.21.15.81:0/2623502939 conn(0x7fdda806ccc0 msgr2=0x7fdda80b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:44.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.754+0000 7fddb07f5640 1 -- 172.21.15.81:0/2623502939 shutdown_connections 2024-06-29T15:59:44.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.754+0000 7fddb07f5640 1 -- 172.21.15.81:0/2623502939 wait complete. 2024-06-29T15:59:44.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.756+0000 7fddb07f5640 1 Processor -- start 2024-06-29T15:59:44.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.756+0000 7fddb07f5640 1 -- start start 2024-06-29T15:59:44.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddb07f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd94004800 0x7fdda80b4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:44.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddb07f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdda8158d20 0x7fdda80b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:44.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddb07f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdda80b3070 0x7fdda80b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:44.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddb07f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdda815c980 con 0x7fdda80b3070 2024-06-29T15:59:44.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddb07f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdda815c800 con 0x7fdd94004800 2024-06-29T15:59:44.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddb07f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdda815cb00 con 0x7fdda8158d20 2024-06-29T15:59:44.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddaed92640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdda80b3070 0x7fdda80b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddaed92640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdda80b3070 0x7fdda80b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48282/0 (socket says 172.21.15.81:48282) 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddaed92640 1 -- 172.21.15.81:0/2605916328 learned_addr learned my addr 172.21.15.81:0/2605916328 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddaed92640 1 -- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdda8158d20 msgr2=0x7fdda80b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddaed92640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdda8158d20 0x7fdda80b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddae591640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd94004800 0x7fdda80b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddaed92640 1 -- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd94004800 msgr2=0x7fdda80b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddaed92640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd94004800 0x7fdda80b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddaed92640 1 -- 172.21.15.81:0/2605916328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda80b45c0 con 0x7fdda80b3070 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddaed92640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdda80b3070 0x7fdda80b3480 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7fdda806ffe0 tx=0x7fdda0010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fdd9effd640 1 -- 172.21.15.81:0/2605916328 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0003e40 con 0x7fdda80b3070 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fdd9effd640 1 -- 172.21.15.81:0/2605916328 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdda0015070 con 0x7fdda80b3070 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddb07f5640 1 -- 172.21.15.81:0/2605916328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda8159400 con 0x7fdda80b3070 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fdd9effd640 1 -- 172.21.15.81:0/2605916328 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda000ed10 con 0x7fdda80b3070 2024-06-29T15:59:44.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.758+0000 7fddb07f5640 1 -- 172.21.15.81:0/2605916328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda80ada50 con 0x7fdda80b3070 2024-06-29T15:59:44.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.759+0000 7fddb07f5640 1 -- 172.21.15.81:0/2605916328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdda8155e30 con 0x7fdda80b3070 2024-06-29T15:59:44.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.760+0000 7fdd9effd640 1 -- 172.21.15.81:0/2605916328 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fdda0004000 con 0x7fdda80b3070 2024-06-29T15:59:44.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.761+0000 7fdd9effd640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fdd8c076420 0x7fdd8c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:44.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.761+0000 7fdd9effd640 1 -- 172.21.15.81:0/2605916328 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7fdda0098860 con 0x7fdda80b3070 2024-06-29T15:59:44.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.761+0000 7fddae591640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fdd8c076420 0x7fdd8c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:44.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.762+0000 7fddae591640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fdd8c076420 0x7fdd8c078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdda4001050 tx=0x7fdda40067a0 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:44.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.764+0000 7fdd9effd640 1 -- 172.21.15.81:0/2605916328 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdda0062080 con 0x7fdda80b3070 2024-06-29T15:59:44.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:44.982+0000 7fddb07f5640 1 -- 172.21.15.81:0/2605916328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fdda8159920 con 0x7fdda80b3070 2024-06-29T15:59:45.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.622+0000 7fdd9effd640 1 -- 172.21.15.81:0/2605916328 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v154) ==== 122+0+0 (secure 0 0 0) 0x7fdda0061980 con 0x7fdda80b3070 2024-06-29T15:59:45.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 -- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fdd8c076420 msgr2=0x7fdd8c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fdd8c076420 0x7fdd8c078930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdda4001050 tx=0x7fdda40067a0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 -- 172.21.15.81:0/2605916328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdda80b3070 msgr2=0x7fdda80b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdda80b3070 0x7fdda80b3480 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7fdda806ffe0 tx=0x7fdda0010f40 comp rx=0 tx=0).stop 2024-06-29T15:59:45.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 -- 172.21.15.81:0/2605916328 shutdown_connections 2024-06-29T15:59:45.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fdd8c076420 0x7fdd8c078930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdda80b3070 0x7fdda80b3480 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdda8158d20 0x7fdda80b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 --2- 172.21.15.81:0/2605916328 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdd94004800 0x7fdda80b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 -- 172.21.15.81:0/2605916328 >> 172.21.15.81:0/2605916328 conn(0x7fdda806ccc0 msgr2=0x7fdda815b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:45.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 -- 172.21.15.81:0/2605916328 shutdown_connections 2024-06-29T15:59:45.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.624+0000 7fddb07f5640 1 -- 172.21.15.81:0/2605916328 wait complete. 2024-06-29T15:59:45.636 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-06-29T15:59:45.636 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:45.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.751+0000 7f3ec71e0640 1 -- 172.21.15.81:0/777988457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ec015cb30 msgr2=0x7f3ec015d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.751+0000 7f3ec71e0640 1 --2- 172.21.15.81:0/777988457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ec015cb30 0x7f3ec015d510 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0070010 tx=0x7f3eb802a4d0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.751+0000 7f3ec71e0640 1 -- 172.21.15.81:0/777988457 shutdown_connections 2024-06-29T15:59:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.751+0000 7f3ec71e0640 1 --2- 172.21.15.81:0/777988457 >> v2:172.21.15.167:3300/0 conn(0x7f3ec0073990 0x7f3ec015c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.751+0000 7f3ec71e0640 1 --2- 172.21.15.81:0/777988457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ec015cb30 0x7f3ec015d510 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.751+0000 7f3ec71e0640 1 -- 172.21.15.81:0/777988457 >> 172.21.15.81:0/777988457 conn(0x7f3ec006ccc0 msgr2=0x7f3ec00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.751+0000 7f3ec71e0640 1 -- 172.21.15.81:0/777988457 shutdown_connections 2024-06-29T15:59:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.751+0000 7f3ec71e0640 1 -- 172.21.15.81:0/777988457 wait complete. 2024-06-29T15:59:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.753+0000 7f3ec71e0640 1 Processor -- start 2024-06-29T15:59:45.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.753+0000 7f3ec71e0640 1 -- start start 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ec71e0640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ec0145530 0x7f3ec00ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ec71e0640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ec015cb30 0x7f3ec00aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ec71e0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ec01592f0 0x7f3ec00aaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ec71e0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec015dbd0 con 0x7f3ec01592f0 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ec71e0640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec015da50 con 0x7f3ec015cb30 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ec71e0640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec015dd50 con 0x7f3ec0145530 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ec577d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ec01592f0 0x7f3ec00aaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ec577d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ec01592f0 0x7f3ec00aaa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48304/0 (socket says 172.21.15.81:48304) 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ec577d640 1 -- 172.21.15.81:0/438711826 learned_addr learned my addr 172.21.15.81:0/438711826 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ebffff640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ec015cb30 0x7f3ec00aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.754+0000 7f3ec4f7c640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ec0145530 0x7f3ec00ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ec577d640 1 -- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ec0145530 msgr2=0x7f3ec00ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ec577d640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ec0145530 0x7f3ec00ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ec577d640 1 -- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ec015cb30 msgr2=0x7f3ec00aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ec577d640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ec015cb30 0x7f3ec00aeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ec577d640 1 -- 172.21.15.81:0/438711826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec00acd20 con 0x7f3ec01592f0 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ec4f7c640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ec0145530 0x7f3ec00ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ec577d640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ec01592f0 0x7f3ec00aaa40 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8034040 tx=0x7f3eb80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ebd7fa640 1 -- 172.21.15.81:0/438711826 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb803e030 con 0x7f3ec01592f0 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ebd7fa640 1 -- 172.21.15.81:0/438711826 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3eb80047a0 con 0x7f3ec01592f0 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ec71e0640 1 -- 172.21.15.81:0/438711826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec00af0a0 con 0x7f3ec01592f0 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ebd7fa640 1 -- 172.21.15.81:0/438711826 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8035040 con 0x7f3ec01592f0 2024-06-29T15:59:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.755+0000 7f3ec71e0640 1 -- 172.21.15.81:0/438711826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec00a9c80 con 0x7f3ec01592f0 2024-06-29T15:59:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.756+0000 7f3ec71e0640 1 -- 172.21.15.81:0/438711826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec00b17b0 con 0x7f3ec01592f0 2024-06-29T15:59:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.757+0000 7f3ebd7fa640 1 -- 172.21.15.81:0/438711826 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f3eb8041030 con 0x7f3ec01592f0 2024-06-29T15:59:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.757+0000 7f3ebd7fa640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3eac0763f0 0x7f3eac078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.758+0000 7f3ebd7fa640 1 -- 172.21.15.81:0/438711826 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f3eb80b8210 con 0x7f3ec01592f0 2024-06-29T15:59:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.758+0000 7f3ec4f7c640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3eac0763f0 0x7f3eac078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.758+0000 7f3ec4f7c640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3eac0763f0 0x7f3eac078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000c420 tx=0x7f3eb0009570 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:45.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.762+0000 7f3ebd7fa640 1 -- 172.21.15.81:0/438711826 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f3eb8081280 con 0x7f3ec01592f0 2024-06-29T15:59:45.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.979+0000 7f3ec71e0640 1 -- 172.21.15.81:0/438711826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ec00b07e0 con 0x7f3ec01592f0 2024-06-29T15:59:45.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.979+0000 7f3ebd7fa640 1 -- 172.21.15.81:0/438711826 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5238 (secure 0 0 0) 0x7f3eb8085580 con 0x7f3ec01592f0 2024-06-29T15:59:45.980 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:45.980 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":154,"btime":"2024-06-29T15:59:45:615681+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10196,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2932449820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2932449820},{"type":"v1","addr":"172.21.15.167:6837","nonce":2932449820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/317823138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":317823138},{"type":"v1","addr":"172.21.15.81:6835","nonce":317823138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3582280449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3582280449},{"type":"v1","addr":"172.21.15.167:6836","nonce":3582280449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:41.965720+0000","modified":"2024-06-29T15:59:44.983893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13371},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13371":{"gid":13371,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/28043779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":28043779},{"type":"v1","addr":"172.21.15.81:6834","nonce":28043779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13371,"qdb_cluster":[13371]},"id":18}]} 2024-06-29T15:59:45.980 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 154 2024-06-29T15:59:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 -- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3eac0763f0 msgr2=0x7f3eac078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3eac0763f0 0x7f3eac078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000c420 tx=0x7f3eb0009570 comp rx=0 tx=0).stop 2024-06-29T15:59:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 -- 172.21.15.81:0/438711826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ec01592f0 msgr2=0x7f3ec00aaa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ec01592f0 0x7f3ec00aaa40 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8034040 tx=0x7f3eb80039e0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 -- 172.21.15.81:0/438711826 shutdown_connections 2024-06-29T15:59:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f3eac0763f0 0x7f3eac078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ec01592f0 0x7f3ec00aaa40 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3ec015cb30 0x7f3ec00aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 --2- 172.21.15.81:0/438711826 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3ec0145530 0x7f3ec00ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:45.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 -- 172.21.15.81:0/438711826 >> 172.21.15.81:0/438711826 conn(0x7f3ec006ccc0 msgr2=0x7f3ec015b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:45.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 -- 172.21.15.81:0/438711826 shutdown_connections 2024-06-29T15:59:45.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:45.981+0000 7f3ec71e0640 1 -- 172.21.15.81:0/438711826 wait complete. 2024-06-29T15:59:45.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T15:59:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.105+0000 7f857643f640 1 -- 172.21.15.81:0/4226396347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85700d0940 msgr2=0x7f85700b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.105+0000 7f857643f640 1 --2- 172.21.15.81:0/4226396347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85700d0940 0x7f85700b4f40 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7f8570073550 tx=0x7f856802aa30 comp rx=0 tx=0).stop 2024-06-29T15:59:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.105+0000 7f857643f640 1 -- 172.21.15.81:0/4226396347 shutdown_connections 2024-06-29T15:59:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.105+0000 7f857643f640 1 --2- 172.21.15.81:0/4226396347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85700d0940 0x7f85700b4f40 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.105+0000 7f857643f640 1 --2- 172.21.15.81:0/4226396347 >> v2:172.21.15.167:3300/0 conn(0x7f85700b3200 0x7f85700cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.105+0000 7f857643f640 1 -- 172.21.15.81:0/4226396347 >> 172.21.15.81:0/4226396347 conn(0x7f85700b3680 msgr2=0x7f85700b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.105+0000 7f857643f640 1 -- 172.21.15.81:0/4226396347 shutdown_connections 2024-06-29T15:59:46.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.105+0000 7f857643f640 1 -- 172.21.15.81:0/4226396347 wait complete. 2024-06-29T15:59:46.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.107+0000 7f857643f640 1 Processor -- start 2024-06-29T15:59:46.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.107+0000 7f857643f640 1 -- start start 2024-06-29T15:59:46.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f857643f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f85700b3200 0x7f85701e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f857643f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85700cccf0 0x7f857009bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f857643f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f857009c120 0x7f85700c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f857643f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f85700b5bb0 con 0x7f85700cccf0 2024-06-29T15:59:46.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f857643f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f85700b5a30 con 0x7f857009c120 2024-06-29T15:59:46.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f857643f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f85700b5d30 con 0x7f85700b3200 2024-06-29T15:59:46.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85700cccf0 0x7f857009bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85700cccf0 0x7f857009bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48316/0 (socket says 172.21.15.81:48316) 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856f7fe640 1 -- 172.21.15.81:0/3450720348 learned_addr learned my addr 172.21.15.81:0/3450720348 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f85749dc640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f857009c120 0x7f85700c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856ffff640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f85700b3200 0x7f85701e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856f7fe640 1 -- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f85700b3200 msgr2=0x7f85701e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856f7fe640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f85700b3200 0x7f85701e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856f7fe640 1 -- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f857009c120 msgr2=0x7f85700c81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856f7fe640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f857009c120 0x7f85700c81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856f7fe640 1 -- 172.21.15.81:0/3450720348 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857009dd50 con 0x7f85700cccf0 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856ffff640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f85700b3200 0x7f85701e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856f7fe640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85700cccf0 0x7f857009bbe0 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f856400c390 tx=0x7f856400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856cff9640 1 -- 172.21.15.81:0/3450720348 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564017070 con 0x7f85700cccf0 2024-06-29T15:59:46.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f857643f640 1 -- 172.21.15.81:0/3450720348 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f857009e030 con 0x7f85700cccf0 2024-06-29T15:59:46.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856cff9640 1 -- 172.21.15.81:0/3450720348 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85640035f0 con 0x7f85700cccf0 2024-06-29T15:59:46.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f856cff9640 1 -- 172.21.15.81:0/3450720348 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564003a20 con 0x7f85700cccf0 2024-06-29T15:59:46.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.109+0000 7f857643f640 1 -- 172.21.15.81:0/3450720348 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f857009e5f0 con 0x7f85700cccf0 2024-06-29T15:59:46.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.110+0000 7f857643f640 1 -- 172.21.15.81:0/3450720348 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f857009daa0 con 0x7f85700cccf0 2024-06-29T15:59:46.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.111+0000 7f856cff9640 1 -- 172.21.15.81:0/3450720348 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f856401a070 con 0x7f85700cccf0 2024-06-29T15:59:46.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.112+0000 7f856cff9640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f855c0763f0 0x7f855c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.112+0000 7f856cff9640 1 -- 172.21.15.81:0/3450720348 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f8564096cb0 con 0x7f85700cccf0 2024-06-29T15:59:46.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.112+0000 7f856ffff640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f855c0763f0 0x7f855c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.112+0000 7f856ffff640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f855c0763f0 0x7f855c078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f856000bbc0 tx=0x7f8560005ef0 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:46.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.116+0000 7f856cff9640 1 -- 172.21.15.81:0/3450720348 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f856405fd20 con 0x7f85700cccf0 2024-06-29T15:59:46.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.298+0000 7f857643f640 1 -- 172.21.15.81:0/3450720348 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85700cc7b0 con 0x7f85700cccf0 2024-06-29T15:59:46.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.299+0000 7f856cff9640 1 -- 172.21.15.81:0/3450720348 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v266) ==== 74+0+16819 (secure 0 0 0) 0x7f856407fe20 con 0x7f85700cccf0 2024-06-29T15:59:46.299 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:46.300 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":266,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T15:59:43.604232+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:40.261824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:40.923216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"266","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 -- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f855c0763f0 msgr2=0x7f855c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f855c0763f0 0x7f855c078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f856000bbc0 tx=0x7f8560005ef0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 -- 172.21.15.81:0/3450720348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85700cccf0 msgr2=0x7f857009bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85700cccf0 0x7f857009bbe0 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f856400c390 tx=0x7f856400c850 comp rx=0 tx=0).stop 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 -- 172.21.15.81:0/3450720348 shutdown_connections 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f855c0763f0 0x7f855c078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f857009c120 0x7f85700c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85700cccf0 0x7f857009bbe0 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 --2- 172.21.15.81:0/3450720348 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f85700b3200 0x7f85701e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 -- 172.21.15.81:0/3450720348 >> 172.21.15.81:0/3450720348 conn(0x7f85700b3680 msgr2=0x7f85700d00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 -- 172.21.15.81:0/3450720348 shutdown_connections 2024-06-29T15:59:46.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.301+0000 7f857643f640 1 -- 172.21.15.81:0/3450720348 wait complete. 2024-06-29T15:59:46.313 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-06-29T15:59:46.313 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:59:46.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.426+0000 7f045a064640 1 -- 172.21.15.81:0/3169273207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04540ce1d0 msgr2=0x7f04540ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.426+0000 7f045a064640 1 --2- 172.21.15.81:0/3169273207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04540ce1d0 0x7f04540ce5e0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f04540b53d0 tx=0x7f044c02aa00 comp rx=0 tx=0).stop 2024-06-29T15:59:46.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.427+0000 7f045a064640 1 -- 172.21.15.81:0/3169273207 shutdown_connections 2024-06-29T15:59:46.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.427+0000 7f045a064640 1 --2- 172.21.15.81:0/3169273207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04540ce1d0 0x7f04540ce5e0 unknown :-1 s=CLOSED pgs=2906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.427+0000 7f045a064640 1 --2- 172.21.15.81:0/3169273207 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f045406ef90 0x7f045406f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.427+0000 7f045a064640 1 --2- 172.21.15.81:0/3169273207 >> v2:172.21.15.167:3300/0 conn(0x7f04540b57b0 0x7f045415aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.427+0000 7f045a064640 1 -- 172.21.15.81:0/3169273207 >> 172.21.15.81:0/3169273207 conn(0x7f04540d0930 msgr2=0x7f04540d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:46.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.427+0000 7f045a064640 1 -- 172.21.15.81:0/3169273207 shutdown_connections 2024-06-29T15:59:46.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.427+0000 7f045a064640 1 -- 172.21.15.81:0/3169273207 wait complete. 2024-06-29T15:59:46.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.428+0000 7f045a064640 1 Processor -- start 2024-06-29T15:59:46.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.429+0000 7f045a064640 1 -- start start 2024-06-29T15:59:46.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f045a064640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f045406ef90 0x7f04540c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f045a064640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04540b57b0 0x7f04540af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f045a064640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f04540ce1d0 0x7f04540af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f045a064640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f04540ceeb0 con 0x7f045406ef90 2024-06-29T15:59:46.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f045a064640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f04540ced30 con 0x7f04540b57b0 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f045a064640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f04540cf030 con 0x7f04540ce1d0 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f04537fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f045406ef90 0x7f04540c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f04537fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f045406ef90 0x7f04540c6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48334/0 (socket says 172.21.15.81:48334) 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f04537fe640 1 -- 172.21.15.81:0/1357332865 learned_addr learned my addr 172.21.15.81:0/1357332865 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f0453fff640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f04540ce1d0 0x7f04540af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f0452ffd640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04540b57b0 0x7f04540af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f04537fe640 1 -- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f04540ce1d0 msgr2=0x7f04540af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f04537fe640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f04540ce1d0 0x7f04540af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f04537fe640 1 -- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04540b57b0 msgr2=0x7f04540af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f04537fe640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04540b57b0 0x7f04540af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f04537fe640 1 -- 172.21.15.81:0/1357332865 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0454150880 con 0x7f045406ef90 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.430+0000 7f04537fe640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f045406ef90 0x7f04540c6320 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7f044400c9a0 tx=0x7f044400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:46.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.431+0000 7f043bfff640 1 -- 172.21.15.81:0/1357332865 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444007bb0 con 0x7f045406ef90 2024-06-29T15:59:46.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.431+0000 7f043bfff640 1 -- 172.21.15.81:0/1357332865 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0444007e40 con 0x7f045406ef90 2024-06-29T15:59:46.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.431+0000 7f043bfff640 1 -- 172.21.15.81:0/1357332865 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044400f690 con 0x7f045406ef90 2024-06-29T15:59:46.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.431+0000 7f045a064640 1 -- 172.21.15.81:0/1357332865 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04540c6860 con 0x7f045406ef90 2024-06-29T15:59:46.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.431+0000 7f045a064640 1 -- 172.21.15.81:0/1357332865 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04540ae2a0 con 0x7f045406ef90 2024-06-29T15:59:46.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.432+0000 7f045a064640 1 -- 172.21.15.81:0/1357332865 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04540b4310 con 0x7f045406ef90 2024-06-29T15:59:46.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.432+0000 7f043bfff640 1 -- 172.21.15.81:0/1357332865 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f044400f830 con 0x7f045406ef90 2024-06-29T15:59:46.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.433+0000 7f043bfff640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f04400763f0 0x7f0440078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.433+0000 7f043bfff640 1 -- 172.21.15.81:0/1357332865 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f0444098120 con 0x7f045406ef90 2024-06-29T15:59:46.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.433+0000 7f0452ffd640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f04400763f0 0x7f0440078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.433+0000 7f0452ffd640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f04400763f0 0x7f0440078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f04541553d0 tx=0x7f0448005f60 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:46.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.437+0000 7f043bfff640 1 -- 172.21.15.81:0/1357332865 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f04540b4310 con 0x7f045406ef90 2024-06-29T15:59:46.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.618+0000 7f045a064640 1 -- 172.21.15.81:0/1357332865 --> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f045406e2a0 con 0x7f04400763f0 2024-06-29T15:59:46.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.623+0000 7f043bfff640 1 -- 172.21.15.81:0/1357332865 <== mgr.10028 v2:172.21.15.167:6832/1824030697 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252499 (secure 0 0 0) 0x7f045406e2a0 con 0x7f04400763f0 2024-06-29T15:59:46.626 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:46.628 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 -- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f04400763f0 msgr2=0x7f0440078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f04400763f0 0x7f0440078900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f04541553d0 tx=0x7f0448005f60 comp rx=0 tx=0).stop 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 -- 172.21.15.81:0/1357332865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f045406ef90 msgr2=0x7f04540c6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f045406ef90 0x7f04540c6320 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7f044400c9a0 tx=0x7f044400ce60 comp rx=0 tx=0).stop 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 -- 172.21.15.81:0/1357332865 shutdown_connections 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f04400763f0 0x7f0440078900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f04540ce1d0 0x7f04540af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f04540b57b0 0x7f04540af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 --2- 172.21.15.81:0/1357332865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f045406ef90 0x7f04540c6320 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 -- 172.21.15.81:0/1357332865 >> 172.21.15.81:0/1357332865 conn(0x7f04540d0930 msgr2=0x7f04540b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:46.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 -- 172.21.15.81:0/1357332865 shutdown_connections 2024-06-29T15:59:46.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.627+0000 7f045a064640 1 -- 172.21.15.81:0/1357332865 wait complete. 2024-06-29T15:59:46.639 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:59:46.206063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1686,"num_read_kb":4807,"num_write":470,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":460132,"kb_used_data":27504,"kb_used_omap":12,"kb_used_meta":432499,"kb_avail":754514588,"statfs":{"total":773094113280,"available":772622938112,"internally_reserved":0,"allocated":28164096,"data_stored":14483220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":442879538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.617516"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.983371+0000","last_change":"2024-06-29T15:59:42.612533+0000","last_active":"2024-06-29T15:59:43.983371+0000","last_peered":"2024-06-29T15:59:43.983371+0000","last_clean":"2024-06-29T15:59:43.983371+0000","last_became_active":"2024-06-29T15:59:42.612205+0000","last_became_peered":"2024-06-29T15:59:42.612205+0000","last_unstale":"2024-06-29T15:59:43.983371+0000","last_undegraded":"2024-06-29T15:59:43.983371+0000","last_fullsized":"2024-06-29T15:59:43.983371+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:04.823899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981487+0000","last_change":"2024-06-29T15:59:42.617082+0000","last_active":"2024-06-29T15:59:43.981487+0000","last_peered":"2024-06-29T15:59:43.981487+0000","last_clean":"2024-06-29T15:59:43.981487+0000","last_became_active":"2024-06-29T15:59:42.616754+0000","last_became_peered":"2024-06-29T15:59:42.616754+0000","last_unstale":"2024-06-29T15:59:43.981487+0000","last_undegraded":"2024-06-29T15:59:43.981487+0000","last_fullsized":"2024-06-29T15:59:43.981487+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:15.200629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612347+0000","last_change":"2024-06-29T15:59:42.619309+0000","last_active":"2024-06-29T15:59:43.612347+0000","last_peered":"2024-06-29T15:59:43.612347+0000","last_clean":"2024-06-29T15:59:43.612347+0000","last_became_active":"2024-06-29T15:59:42.619021+0000","last_became_peered":"2024-06-29T15:59:42.619021+0000","last_unstale":"2024-06-29T15:59:43.612347+0000","last_undegraded":"2024-06-29T15:59:43.612347+0000","last_fullsized":"2024-06-29T15:59:43.612347+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:13:03.731967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613138+0000","last_change":"2024-06-29T15:59:42.623148+0000","last_active":"2024-06-29T15:59:43.613138+0000","last_peered":"2024-06-29T15:59:43.613138+0000","last_clean":"2024-06-29T15:59:43.613138+0000","last_became_active":"2024-06-29T15:59:42.622338+0000","last_became_peered":"2024-06-29T15:59:42.622338+0000","last_unstale":"2024-06-29T15:59:43.613138+0000","last_undegraded":"2024-06-29T15:59:43.613138+0000","last_fullsized":"2024-06-29T15:59:43.613138+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:42.796201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612763+0000","last_change":"2024-06-29T15:59:42.620958+0000","last_active":"2024-06-29T15:59:43.612763+0000","last_peered":"2024-06-29T15:59:43.612763+0000","last_clean":"2024-06-29T15:59:43.612763+0000","last_became_active":"2024-06-29T15:59:42.620660+0000","last_became_peered":"2024-06-29T15:59:42.620660+0000","last_unstale":"2024-06-29T15:59:43.612763+0000","last_undegraded":"2024-06-29T15:59:43.612763+0000","last_fullsized":"2024-06-29T15:59:43.612763+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:18.356430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612682+0000","last_change":"2024-06-29T15:59:42.622147+0000","last_active":"2024-06-29T15:59:43.612682+0000","last_peered":"2024-06-29T15:59:43.612682+0000","last_clean":"2024-06-29T15:59:43.612682+0000","last_became_active":"2024-06-29T15:59:42.621853+0000","last_became_peered":"2024-06-29T15:59:42.621853+0000","last_unstale":"2024-06-29T15:59:43.612682+0000","last_undegraded":"2024-06-29T15:59:43.612682+0000","last_fullsized":"2024-06-29T15:59:43.612682+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:13:03.731967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982601+0000","last_change":"2024-06-29T15:59:42.613206+0000","last_active":"2024-06-29T15:59:43.982601+0000","last_peered":"2024-06-29T15:59:43.982601+0000","last_clean":"2024-06-29T15:59:43.982601+0000","last_became_active":"2024-06-29T15:59:42.612868+0000","last_became_peered":"2024-06-29T15:59:42.612868+0000","last_unstale":"2024-06-29T15:59:43.982601+0000","last_undegraded":"2024-06-29T15:59:43.982601+0000","last_fullsized":"2024-06-29T15:59:43.982601+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:16:02.811328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615533+0000","last_change":"2024-06-29T15:59:42.615693+0000","last_active":"2024-06-29T15:59:43.615533+0000","last_peered":"2024-06-29T15:59:43.615533+0000","last_clean":"2024-06-29T15:59:43.615533+0000","last_became_active":"2024-06-29T15:59:42.615368+0000","last_became_peered":"2024-06-29T15:59:42.615368+0000","last_unstale":"2024-06-29T15:59:43.615533+0000","last_undegraded":"2024-06-29T15:59:43.615533+0000","last_fullsized":"2024-06-29T15:59:43.615533+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:14.798255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981405+0000","last_change":"2024-06-29T15:59:42.620130+0000","last_active":"2024-06-29T15:59:43.981405+0000","last_peered":"2024-06-29T15:59:43.981405+0000","last_clean":"2024-06-29T15:59:43.981405+0000","last_became_active":"2024-06-29T15:59:42.616841+0000","last_became_peered":"2024-06-29T15:59:42.616841+0000","last_unstale":"2024-06-29T15:59:43.981405+0000","last_undegraded":"2024-06-29T15:59:43.981405+0000","last_fullsized":"2024-06-29T15:59:43.981405+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:54.989299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.983503+0000","last_change":"2024-06-29T15:59:42.619822+0000","last_active":"2024-06-29T15:59:43.983503+0000","last_peered":"2024-06-29T15:59:43.983503+0000","last_clean":"2024-06-29T15:59:43.983503+0000","last_became_active":"2024-06-29T15:59:42.619236+0000","last_became_peered":"2024-06-29T15:59:42.619236+0000","last_unstale":"2024-06-29T15:59:43.983503+0000","last_undegraded":"2024-06-29T15:59:43.983503+0000","last_fullsized":"2024-06-29T15:59:43.983503+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:20.652399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615893+0000","last_change":"2024-06-29T15:59:42.627259+0000","last_active":"2024-06-29T15:59:43.615893+0000","last_peered":"2024-06-29T15:59:43.615893+0000","last_clean":"2024-06-29T15:59:43.615893+0000","last_became_active":"2024-06-29T15:59:42.626511+0000","last_became_peered":"2024-06-29T15:59:42.626511+0000","last_unstale":"2024-06-29T15:59:43.615893+0000","last_undegraded":"2024-06-29T15:59:43.615893+0000","last_fullsized":"2024-06-29T15:59:43.615893+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:35.560322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612724+0000","last_change":"2024-06-29T15:59:42.615270+0000","last_active":"2024-06-29T15:59:43.612724+0000","last_peered":"2024-06-29T15:59:43.612724+0000","last_clean":"2024-06-29T15:59:43.612724+0000","last_became_active":"2024-06-29T15:59:42.614950+0000","last_became_peered":"2024-06-29T15:59:42.614950+0000","last_unstale":"2024-06-29T15:59:43.612724+0000","last_undegraded":"2024-06-29T15:59:43.612724+0000","last_fullsized":"2024-06-29T15:59:43.612724+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:54:37.935864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.983177+0000","last_change":"2024-06-29T15:59:42.615216+0000","last_active":"2024-06-29T15:59:43.983177+0000","last_peered":"2024-06-29T15:59:43.983177+0000","last_clean":"2024-06-29T15:59:43.983177+0000","last_became_active":"2024-06-29T15:59:42.613448+0000","last_became_peered":"2024-06-29T15:59:42.613448+0000","last_unstale":"2024-06-29T15:59:43.983177+0000","last_undegraded":"2024-06-29T15:59:43.983177+0000","last_fullsized":"2024-06-29T15:59:43.983177+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:01.018930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982331+0000","last_change":"2024-06-29T15:59:42.621522+0000","last_active":"2024-06-29T15:59:43.982331+0000","last_peered":"2024-06-29T15:59:43.982331+0000","last_clean":"2024-06-29T15:59:43.982331+0000","last_became_active":"2024-06-29T15:59:42.621211+0000","last_became_peered":"2024-06-29T15:59:42.621211+0000","last_unstale":"2024-06-29T15:59:43.982331+0000","last_undegraded":"2024-06-29T15:59:43.982331+0000","last_fullsized":"2024-06-29T15:59:43.982331+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:17.506409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.611907+0000","last_change":"2024-06-29T15:59:42.629405+0000","last_active":"2024-06-29T15:59:43.611907+0000","last_peered":"2024-06-29T15:59:43.611907+0000","last_clean":"2024-06-29T15:59:43.611907+0000","last_became_active":"2024-06-29T15:59:42.629036+0000","last_became_peered":"2024-06-29T15:59:42.629036+0000","last_unstale":"2024-06-29T15:59:43.611907+0000","last_undegraded":"2024-06-29T15:59:43.611907+0000","last_fullsized":"2024-06-29T15:59:43.611907+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:38.776266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612463+0000","last_change":"2024-06-29T15:59:42.620609+0000","last_active":"2024-06-29T15:59:43.612463+0000","last_peered":"2024-06-29T15:59:43.612463+0000","last_clean":"2024-06-29T15:59:43.612463+0000","last_became_active":"2024-06-29T15:59:42.620296+0000","last_became_peered":"2024-06-29T15:59:42.620296+0000","last_unstale":"2024-06-29T15:59:43.612463+0000","last_undegraded":"2024-06-29T15:59:43.612463+0000","last_fullsized":"2024-06-29T15:59:43.612463+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:43.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982582+0000","last_change":"2024-06-29T15:59:42.618052+0000","last_active":"2024-06-29T15:59:43.982582+0000","last_peered":"2024-06-29T15:59:43.982582+0000","last_clean":"2024-06-29T15:59:43.982582+0000","last_became_active":"2024-06-29T15:59:42.617707+0000","last_became_peered":"2024-06-29T15:59:42.617707+0000","last_unstale":"2024-06-29T15:59:43.982582+0000","last_undegraded":"2024-06-29T15:59:43.982582+0000","last_fullsized":"2024-06-29T15:59:43.982582+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:50:26.378251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982569+0000","last_change":"2024-06-29T15:59:42.622834+0000","last_active":"2024-06-29T15:59:43.982569+0000","last_peered":"2024-06-29T15:59:43.982569+0000","last_clean":"2024-06-29T15:59:43.982569+0000","last_became_active":"2024-06-29T15:59:42.622535+0000","last_became_peered":"2024-06-29T15:59:42.622535+0000","last_unstale":"2024-06-29T15:59:43.982569+0000","last_undegraded":"2024-06-29T15:59:43.982569+0000","last_fullsized":"2024-06-29T15:59:43.982569+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:48.971622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982450+0000","last_change":"2024-06-29T15:59:42.616261+0000","last_active":"2024-06-29T15:59:43.982450+0000","last_peered":"2024-06-29T15:59:43.982450+0000","last_clean":"2024-06-29T15:59:43.982450+0000","last_became_active":"2024-06-29T15:59:42.615914+0000","last_became_peered":"2024-06-29T15:59:42.615914+0000","last_unstale":"2024-06-29T15:59:43.982450+0000","last_undegraded":"2024-06-29T15:59:43.982450+0000","last_fullsized":"2024-06-29T15:59:43.982450+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:24:41.434894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981336+0000","last_change":"2024-06-29T15:59:42.631226+0000","last_active":"2024-06-29T15:59:43.981336+0000","last_peered":"2024-06-29T15:59:43.981336+0000","last_clean":"2024-06-29T15:59:43.981336+0000","last_became_active":"2024-06-29T15:59:42.630894+0000","last_became_peered":"2024-06-29T15:59:42.630894+0000","last_unstale":"2024-06-29T15:59:43.981336+0000","last_undegraded":"2024-06-29T15:59:43.981336+0000","last_fullsized":"2024-06-29T15:59:43.981336+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:34:32.094496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.616120+0000","last_change":"2024-06-29T15:59:42.626472+0000","last_active":"2024-06-29T15:59:43.616120+0000","last_peered":"2024-06-29T15:59:43.616120+0000","last_clean":"2024-06-29T15:59:43.616120+0000","last_became_active":"2024-06-29T15:59:42.626168+0000","last_became_peered":"2024-06-29T15:59:42.626168+0000","last_unstale":"2024-06-29T15:59:43.616120+0000","last_undegraded":"2024-06-29T15:59:43.616120+0000","last_fullsized":"2024-06-29T15:59:43.616120+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:03:17.919794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612090+0000","last_change":"2024-06-29T15:59:42.611394+0000","last_active":"2024-06-29T15:59:43.612090+0000","last_peered":"2024-06-29T15:59:43.612090+0000","last_clean":"2024-06-29T15:59:43.612090+0000","last_became_active":"2024-06-29T15:59:42.611087+0000","last_became_peered":"2024-06-29T15:59:42.611087+0000","last_unstale":"2024-06-29T15:59:43.612090+0000","last_undegraded":"2024-06-29T15:59:43.612090+0000","last_fullsized":"2024-06-29T15:59:43.612090+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:10.984734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982570+0000","last_change":"2024-06-29T15:59:42.622727+0000","last_active":"2024-06-29T15:59:43.982570+0000","last_peered":"2024-06-29T15:59:43.982570+0000","last_clean":"2024-06-29T15:59:43.982570+0000","last_became_active":"2024-06-29T15:59:42.622426+0000","last_became_peered":"2024-06-29T15:59:42.622426+0000","last_unstale":"2024-06-29T15:59:43.982570+0000","last_undegraded":"2024-06-29T15:59:43.982570+0000","last_fullsized":"2024-06-29T15:59:43.982570+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:19.017174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981999+0000","last_change":"2024-06-29T15:59:42.618541+0000","last_active":"2024-06-29T15:59:43.981999+0000","last_peered":"2024-06-29T15:59:43.981999+0000","last_clean":"2024-06-29T15:59:43.981999+0000","last_became_active":"2024-06-29T15:59:42.618192+0000","last_became_peered":"2024-06-29T15:59:42.618192+0000","last_unstale":"2024-06-29T15:59:43.981999+0000","last_undegraded":"2024-06-29T15:59:43.981999+0000","last_fullsized":"2024-06-29T15:59:43.981999+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:41:45.473172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982775+0000","last_change":"2024-06-29T15:59:42.616435+0000","last_active":"2024-06-29T15:59:43.982775+0000","last_peered":"2024-06-29T15:59:43.982775+0000","last_clean":"2024-06-29T15:59:43.982775+0000","last_became_active":"2024-06-29T15:59:42.616130+0000","last_became_peered":"2024-06-29T15:59:42.616130+0000","last_unstale":"2024-06-29T15:59:43.982775+0000","last_undegraded":"2024-06-29T15:59:43.982775+0000","last_fullsized":"2024-06-29T15:59:43.982775+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:32.549725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.983630+0000","last_change":"2024-06-29T15:59:42.604154+0000","last_active":"2024-06-29T15:59:43.983630+0000","last_peered":"2024-06-29T15:59:43.983630+0000","last_clean":"2024-06-29T15:59:43.983630+0000","last_became_active":"2024-06-29T15:59:42.603824+0000","last_became_peered":"2024-06-29T15:59:42.603824+0000","last_unstale":"2024-06-29T15:59:43.983630+0000","last_undegraded":"2024-06-29T15:59:43.983630+0000","last_fullsized":"2024-06-29T15:59:43.983630+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:58.118394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613040+0000","last_change":"2024-06-29T15:59:42.619753+0000","last_active":"2024-06-29T15:59:43.613040+0000","last_peered":"2024-06-29T15:59:43.613040+0000","last_clean":"2024-06-29T15:59:43.613040+0000","last_became_active":"2024-06-29T15:59:42.619467+0000","last_became_peered":"2024-06-29T15:59:42.619467+0000","last_unstale":"2024-06-29T15:59:43.613040+0000","last_undegraded":"2024-06-29T15:59:43.613040+0000","last_fullsized":"2024-06-29T15:59:43.613040+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:39:41.301776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613268+0000","last_change":"2024-06-29T15:59:42.626257+0000","last_active":"2024-06-29T15:59:43.613268+0000","last_peered":"2024-06-29T15:59:43.613268+0000","last_clean":"2024-06-29T15:59:43.613268+0000","last_became_active":"2024-06-29T15:59:42.625873+0000","last_became_peered":"2024-06-29T15:59:42.625873+0000","last_unstale":"2024-06-29T15:59:43.613268+0000","last_undegraded":"2024-06-29T15:59:43.613268+0000","last_fullsized":"2024-06-29T15:59:43.613268+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:32.759053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982156+0000","last_change":"2024-06-29T15:59:42.619637+0000","last_active":"2024-06-29T15:59:43.982156+0000","last_peered":"2024-06-29T15:59:43.982156+0000","last_clean":"2024-06-29T15:59:43.982156+0000","last_became_active":"2024-06-29T15:59:42.619317+0000","last_became_peered":"2024-06-29T15:59:42.619317+0000","last_unstale":"2024-06-29T15:59:43.982156+0000","last_undegraded":"2024-06-29T15:59:43.982156+0000","last_fullsized":"2024-06-29T15:59:43.982156+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:53:56.981682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981718+0000","last_change":"2024-06-29T15:59:42.611174+0000","last_active":"2024-06-29T15:59:43.981718+0000","last_peered":"2024-06-29T15:59:43.981718+0000","last_clean":"2024-06-29T15:59:43.981718+0000","last_became_active":"2024-06-29T15:59:42.610838+0000","last_became_peered":"2024-06-29T15:59:42.610838+0000","last_unstale":"2024-06-29T15:59:43.981718+0000","last_undegraded":"2024-06-29T15:59:43.981718+0000","last_fullsized":"2024-06-29T15:59:43.981718+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:57.182169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.611934+0000","last_change":"2024-06-29T15:59:42.621232+0000","last_active":"2024-06-29T15:59:43.611934+0000","last_peered":"2024-06-29T15:59:43.611934+0000","last_clean":"2024-06-29T15:59:43.611934+0000","last_became_active":"2024-06-29T15:59:42.620924+0000","last_became_peered":"2024-06-29T15:59:42.620924+0000","last_unstale":"2024-06-29T15:59:43.611934+0000","last_undegraded":"2024-06-29T15:59:43.611934+0000","last_fullsized":"2024-06-29T15:59:43.611934+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:42.508804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612989+0000","last_change":"2024-06-29T15:59:42.618187+0000","last_active":"2024-06-29T15:59:43.612989+0000","last_peered":"2024-06-29T15:59:43.612989+0000","last_clean":"2024-06-29T15:59:43.612989+0000","last_became_active":"2024-06-29T15:59:42.617766+0000","last_became_peered":"2024-06-29T15:59:42.617766+0000","last_unstale":"2024-06-29T15:59:43.612989+0000","last_undegraded":"2024-06-29T15:59:43.612989+0000","last_fullsized":"2024-06-29T15:59:43.612989+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:30.939091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"265'4","reported_seq":18,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982801+0000","last_change":"2024-06-29T15:59:41.614369+0000","last_active":"2024-06-29T15:59:43.982801+0000","last_peered":"2024-06-29T15:59:43.982801+0000","last_clean":"2024-06-29T15:59:43.982801+0000","last_became_active":"2024-06-29T15:59:41.613958+0000","last_became_peered":"2024-06-29T15:59:41.613958+0000","last_unstale":"2024-06-29T15:59:43.982801+0000","last_undegraded":"2024-06-29T15:59:43.982801+0000","last_fullsized":"2024-06-29T15:59:43.982801+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:41:38.903895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982465+0000","last_change":"2024-06-29T15:59:42.608922+0000","last_active":"2024-06-29T15:59:43.982465+0000","last_peered":"2024-06-29T15:59:43.982465+0000","last_clean":"2024-06-29T15:59:43.982465+0000","last_became_active":"2024-06-29T15:59:42.608484+0000","last_became_peered":"2024-06-29T15:59:42.608484+0000","last_unstale":"2024-06-29T15:59:43.982465+0000","last_undegraded":"2024-06-29T15:59:43.982465+0000","last_fullsized":"2024-06-29T15:59:43.982465+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:00.573266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"265'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613459+0000","last_change":"2024-06-29T15:59:41.616918+0000","last_active":"2024-06-29T15:59:43.613459+0000","last_peered":"2024-06-29T15:59:43.613459+0000","last_clean":"2024-06-29T15:59:43.613459+0000","last_became_active":"2024-06-29T15:59:41.616589+0000","last_became_peered":"2024-06-29T15:59:41.616589+0000","last_unstale":"2024-06-29T15:59:43.613459+0000","last_undegraded":"2024-06-29T15:59:43.613459+0000","last_fullsized":"2024-06-29T15:59:43.613459+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:52.272460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612910+0000","last_change":"2024-06-29T15:59:42.621818+0000","last_active":"2024-06-29T15:59:43.612910+0000","last_peered":"2024-06-29T15:59:43.612910+0000","last_clean":"2024-06-29T15:59:43.612910+0000","last_became_active":"2024-06-29T15:59:42.621493+0000","last_became_peered":"2024-06-29T15:59:42.621493+0000","last_unstale":"2024-06-29T15:59:43.612910+0000","last_undegraded":"2024-06-29T15:59:43.612910+0000","last_fullsized":"2024-06-29T15:59:43.612910+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:18.356430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613547+0000","last_change":"2024-06-29T15:59:41.614605+0000","last_active":"2024-06-29T15:59:43.613547+0000","last_peered":"2024-06-29T15:59:43.613547+0000","last_clean":"2024-06-29T15:59:43.613547+0000","last_became_active":"2024-06-29T15:59:41.613882+0000","last_became_peered":"2024-06-29T15:59:41.613882+0000","last_unstale":"2024-06-29T15:59:43.613547+0000","last_undegraded":"2024-06-29T15:59:43.613547+0000","last_fullsized":"2024-06-29T15:59:43.613547+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:56.176759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615987+0000","last_change":"2024-06-29T15:59:42.627498+0000","last_active":"2024-06-29T15:59:43.615987+0000","last_peered":"2024-06-29T15:59:43.615987+0000","last_clean":"2024-06-29T15:59:43.615987+0000","last_became_active":"2024-06-29T15:59:42.626935+0000","last_became_peered":"2024-06-29T15:59:42.626935+0000","last_unstale":"2024-06-29T15:59:43.615987+0000","last_undegraded":"2024-06-29T15:59:43.615987+0000","last_fullsized":"2024-06-29T15:59:43.615987+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:46:10.842632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613686+0000","last_change":"2024-06-29T15:59:41.613331+0000","last_active":"2024-06-29T15:59:43.613686+0000","last_peered":"2024-06-29T15:59:43.613686+0000","last_clean":"2024-06-29T15:59:43.613686+0000","last_became_active":"2024-06-29T15:59:41.613080+0000","last_became_peered":"2024-06-29T15:59:41.613080+0000","last_unstale":"2024-06-29T15:59:43.613686+0000","last_undegraded":"2024-06-29T15:59:43.613686+0000","last_fullsized":"2024-06-29T15:59:43.613686+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:40:28.215838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615728+0000","last_change":"2024-06-29T15:59:42.616863+0000","last_active":"2024-06-29T15:59:43.615728+0000","last_peered":"2024-06-29T15:59:43.615728+0000","last_clean":"2024-06-29T15:59:43.615728+0000","last_became_active":"2024-06-29T15:59:42.616516+0000","last_became_peered":"2024-06-29T15:59:42.616516+0000","last_unstale":"2024-06-29T15:59:43.615728+0000","last_undegraded":"2024-06-29T15:59:43.615728+0000","last_fullsized":"2024-06-29T15:59:43.615728+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:35.661438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"265'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612663+0000","last_change":"2024-06-29T15:59:41.613491+0000","last_active":"2024-06-29T15:59:43.612663+0000","last_peered":"2024-06-29T15:59:43.612663+0000","last_clean":"2024-06-29T15:59:43.612663+0000","last_became_active":"2024-06-29T15:59:41.613120+0000","last_became_peered":"2024-06-29T15:59:41.613120+0000","last_unstale":"2024-06-29T15:59:43.612663+0000","last_undegraded":"2024-06-29T15:59:43.612663+0000","last_fullsized":"2024-06-29T15:59:43.612663+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:31.544314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981287+0000","last_change":"2024-06-29T15:59:42.624269+0000","last_active":"2024-06-29T15:59:43.981287+0000","last_peered":"2024-06-29T15:59:43.981287+0000","last_clean":"2024-06-29T15:59:43.981287+0000","last_became_active":"2024-06-29T15:59:42.623962+0000","last_became_peered":"2024-06-29T15:59:42.623962+0000","last_unstale":"2024-06-29T15:59:43.981287+0000","last_undegraded":"2024-06-29T15:59:43.981287+0000","last_fullsized":"2024-06-29T15:59:43.981287+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:44.387666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"265'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982309+0000","last_change":"2024-06-29T15:59:41.618251+0000","last_active":"2024-06-29T15:59:43.982309+0000","last_peered":"2024-06-29T15:59:43.982309+0000","last_clean":"2024-06-29T15:59:43.982309+0000","last_became_active":"2024-06-29T15:59:41.617810+0000","last_became_peered":"2024-06-29T15:59:41.617810+0000","last_unstale":"2024-06-29T15:59:43.982309+0000","last_undegraded":"2024-06-29T15:59:43.982309+0000","last_fullsized":"2024-06-29T15:59:43.982309+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:28:13.722209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981538+0000","last_change":"2024-06-29T15:59:42.618704+0000","last_active":"2024-06-29T15:59:43.981538+0000","last_peered":"2024-06-29T15:59:43.981538+0000","last_clean":"2024-06-29T15:59:43.981538+0000","last_became_active":"2024-06-29T15:59:42.618396+0000","last_became_peered":"2024-06-29T15:59:42.618396+0000","last_unstale":"2024-06-29T15:59:43.981538+0000","last_undegraded":"2024-06-29T15:59:43.981538+0000","last_fullsized":"2024-06-29T15:59:43.981538+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:22:48.050983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612429+0000","last_change":"2024-06-29T15:59:41.619230+0000","last_active":"2024-06-29T15:59:43.612429+0000","last_peered":"2024-06-29T15:59:43.612429+0000","last_clean":"2024-06-29T15:59:43.612429+0000","last_became_active":"2024-06-29T15:59:41.618924+0000","last_became_peered":"2024-06-29T15:59:41.618924+0000","last_unstale":"2024-06-29T15:59:43.612429+0000","last_undegraded":"2024-06-29T15:59:43.612429+0000","last_fullsized":"2024-06-29T15:59:43.612429+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:47.966211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612799+0000","last_change":"2024-06-29T15:59:42.618412+0000","last_active":"2024-06-29T15:59:43.612799+0000","last_peered":"2024-06-29T15:59:43.612799+0000","last_clean":"2024-06-29T15:59:43.612799+0000","last_became_active":"2024-06-29T15:59:42.618095+0000","last_became_peered":"2024-06-29T15:59:42.618095+0000","last_unstale":"2024-06-29T15:59:43.612799+0000","last_undegraded":"2024-06-29T15:59:43.612799+0000","last_fullsized":"2024-06-29T15:59:43.612799+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:53:16.228824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612331+0000","last_change":"2024-06-29T15:59:41.620760+0000","last_active":"2024-06-29T15:59:43.612331+0000","last_peered":"2024-06-29T15:59:43.612331+0000","last_clean":"2024-06-29T15:59:43.612331+0000","last_became_active":"2024-06-29T15:59:41.620396+0000","last_became_peered":"2024-06-29T15:59:41.620396+0000","last_unstale":"2024-06-29T15:59:43.612331+0000","last_undegraded":"2024-06-29T15:59:43.612331+0000","last_fullsized":"2024-06-29T15:59:43.612331+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:20:43.427382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613341+0000","last_change":"2024-06-29T15:59:42.610163+0000","last_active":"2024-06-29T15:59:43.613341+0000","last_peered":"2024-06-29T15:59:43.613341+0000","last_clean":"2024-06-29T15:59:43.613341+0000","last_became_active":"2024-06-29T15:59:42.609815+0000","last_became_peered":"2024-06-29T15:59:42.609815+0000","last_unstale":"2024-06-29T15:59:43.613341+0000","last_undegraded":"2024-06-29T15:59:43.613341+0000","last_fullsized":"2024-06-29T15:59:43.613341+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:16.114463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982931+0000","last_change":"2024-06-29T15:59:41.617815+0000","last_active":"2024-06-29T15:59:43.982931+0000","last_peered":"2024-06-29T15:59:43.982931+0000","last_clean":"2024-06-29T15:59:43.982931+0000","last_became_active":"2024-06-29T15:59:41.617457+0000","last_became_peered":"2024-06-29T15:59:41.617457+0000","last_unstale":"2024-06-29T15:59:43.982931+0000","last_undegraded":"2024-06-29T15:59:43.982931+0000","last_fullsized":"2024-06-29T15:59:43.982931+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:29:05.505946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982668+0000","last_change":"2024-06-29T15:59:42.614736+0000","last_active":"2024-06-29T15:59:43.982668+0000","last_peered":"2024-06-29T15:59:43.982668+0000","last_clean":"2024-06-29T15:59:43.982668+0000","last_became_active":"2024-06-29T15:59:42.614418+0000","last_became_peered":"2024-06-29T15:59:42.614418+0000","last_unstale":"2024-06-29T15:59:43.982668+0000","last_undegraded":"2024-06-29T15:59:43.982668+0000","last_fullsized":"2024-06-29T15:59:43.982668+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:18.842042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982165+0000","last_change":"2024-06-29T15:59:41.616859+0000","last_active":"2024-06-29T15:59:43.982165+0000","last_peered":"2024-06-29T15:59:43.982165+0000","last_clean":"2024-06-29T15:59:43.982165+0000","last_became_active":"2024-06-29T15:59:41.616288+0000","last_became_peered":"2024-06-29T15:59:41.616288+0000","last_unstale":"2024-06-29T15:59:43.982165+0000","last_undegraded":"2024-06-29T15:59:43.982165+0000","last_fullsized":"2024-06-29T15:59:43.982165+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:36:16.977926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613478+0000","last_change":"2024-06-29T15:59:42.612221+0000","last_active":"2024-06-29T15:59:43.613478+0000","last_peered":"2024-06-29T15:59:43.613478+0000","last_clean":"2024-06-29T15:59:43.613478+0000","last_became_active":"2024-06-29T15:59:42.611931+0000","last_became_peered":"2024-06-29T15:59:42.611931+0000","last_unstale":"2024-06-29T15:59:43.613478+0000","last_undegraded":"2024-06-29T15:59:43.613478+0000","last_fullsized":"2024-06-29T15:59:43.613478+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:16.114463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"265'3","reported_seq":17,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612457+0000","last_change":"2024-06-29T15:59:41.611119+0000","last_active":"2024-06-29T15:59:43.612457+0000","last_peered":"2024-06-29T15:59:43.612457+0000","last_clean":"2024-06-29T15:59:43.612457+0000","last_became_active":"2024-06-29T15:59:41.610247+0000","last_became_peered":"2024-06-29T15:59:41.610247+0000","last_unstale":"2024-06-29T15:59:43.612457+0000","last_undegraded":"2024-06-29T15:59:43.612457+0000","last_fullsized":"2024-06-29T15:59:43.612457+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:35.775372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613431+0000","last_change":"2024-06-29T15:59:42.622109+0000","last_active":"2024-06-29T15:59:43.613431+0000","last_peered":"2024-06-29T15:59:43.613431+0000","last_clean":"2024-06-29T15:59:43.613431+0000","last_became_active":"2024-06-29T15:59:42.621768+0000","last_became_peered":"2024-06-29T15:59:42.621768+0000","last_unstale":"2024-06-29T15:59:43.613431+0000","last_undegraded":"2024-06-29T15:59:43.613431+0000","last_fullsized":"2024-06-29T15:59:43.613431+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:19.191373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"265'6","reported_seq":20,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982740+0000","last_change":"2024-06-29T15:59:41.615448+0000","last_active":"2024-06-29T15:59:43.982740+0000","last_peered":"2024-06-29T15:59:43.982740+0000","last_clean":"2024-06-29T15:59:43.982740+0000","last_became_active":"2024-06-29T15:59:41.615130+0000","last_became_peered":"2024-06-29T15:59:41.615130+0000","last_unstale":"2024-06-29T15:59:43.982740+0000","last_undegraded":"2024-06-29T15:59:43.982740+0000","last_fullsized":"2024-06-29T15:59:43.982740+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:34.656028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982252+0000","last_change":"2024-06-29T15:59:42.615550+0000","last_active":"2024-06-29T15:59:43.982252+0000","last_peered":"2024-06-29T15:59:43.982252+0000","last_clean":"2024-06-29T15:59:43.982252+0000","last_became_active":"2024-06-29T15:59:42.615247+0000","last_became_peered":"2024-06-29T15:59:42.615247+0000","last_unstale":"2024-06-29T15:59:43.982252+0000","last_undegraded":"2024-06-29T15:59:43.982252+0000","last_fullsized":"2024-06-29T15:59:43.982252+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:53.277871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615861+0000","last_change":"2024-06-29T15:59:41.604732+0000","last_active":"2024-06-29T15:59:43.615861+0000","last_peered":"2024-06-29T15:59:43.615861+0000","last_clean":"2024-06-29T15:59:43.615861+0000","last_became_active":"2024-06-29T15:59:41.604327+0000","last_became_peered":"2024-06-29T15:59:41.604327+0000","last_unstale":"2024-06-29T15:59:43.615861+0000","last_undegraded":"2024-06-29T15:59:43.615861+0000","last_fullsized":"2024-06-29T15:59:43.615861+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:41.503393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981714+0000","last_change":"2024-06-29T15:59:42.620145+0000","last_active":"2024-06-29T15:59:43.981714+0000","last_peered":"2024-06-29T15:59:43.981714+0000","last_clean":"2024-06-29T15:59:43.981714+0000","last_became_active":"2024-06-29T15:59:42.619763+0000","last_became_peered":"2024-06-29T15:59:42.619763+0000","last_unstale":"2024-06-29T15:59:43.981714+0000","last_undegraded":"2024-06-29T15:59:43.981714+0000","last_fullsized":"2024-06-29T15:59:43.981714+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:11:45.103664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612206+0000","last_change":"2024-06-29T15:59:41.608219+0000","last_active":"2024-06-29T15:59:43.612206+0000","last_peered":"2024-06-29T15:59:43.612206+0000","last_clean":"2024-06-29T15:59:43.612206+0000","last_became_active":"2024-06-29T15:59:41.607873+0000","last_became_peered":"2024-06-29T15:59:41.607873+0000","last_unstale":"2024-06-29T15:59:43.612206+0000","last_undegraded":"2024-06-29T15:59:43.612206+0000","last_fullsized":"2024-06-29T15:59:43.612206+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:37:29.669256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612468+0000","last_change":"2024-06-29T15:59:42.616037+0000","last_active":"2024-06-29T15:59:43.612468+0000","last_peered":"2024-06-29T15:59:43.612468+0000","last_clean":"2024-06-29T15:59:43.612468+0000","last_became_active":"2024-06-29T15:59:42.615704+0000","last_became_peered":"2024-06-29T15:59:42.615704+0000","last_unstale":"2024-06-29T15:59:43.612468+0000","last_undegraded":"2024-06-29T15:59:43.612468+0000","last_fullsized":"2024-06-29T15:59:43.612468+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:16.616337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982996+0000","last_change":"2024-06-29T15:59:41.619225+0000","last_active":"2024-06-29T15:59:43.982996+0000","last_peered":"2024-06-29T15:59:43.982996+0000","last_clean":"2024-06-29T15:59:43.982996+0000","last_became_active":"2024-06-29T15:59:41.618792+0000","last_became_peered":"2024-06-29T15:59:41.618792+0000","last_unstale":"2024-06-29T15:59:43.982996+0000","last_undegraded":"2024-06-29T15:59:43.982996+0000","last_fullsized":"2024-06-29T15:59:43.982996+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:34.554911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982371+0000","last_change":"2024-06-29T15:59:42.610628+0000","last_active":"2024-06-29T15:59:43.982371+0000","last_peered":"2024-06-29T15:59:43.982371+0000","last_clean":"2024-06-29T15:59:43.982371+0000","last_became_active":"2024-06-29T15:59:42.609965+0000","last_became_peered":"2024-06-29T15:59:42.609965+0000","last_unstale":"2024-06-29T15:59:43.982371+0000","last_undegraded":"2024-06-29T15:59:43.982371+0000","last_fullsized":"2024-06-29T15:59:43.982371+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:33.730877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"265'8","reported_seq":22,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612069+0000","last_change":"2024-06-29T15:59:41.619781+0000","last_active":"2024-06-29T15:59:43.612069+0000","last_peered":"2024-06-29T15:59:43.612069+0000","last_clean":"2024-06-29T15:59:43.612069+0000","last_became_active":"2024-06-29T15:59:41.619353+0000","last_became_peered":"2024-06-29T15:59:41.619353+0000","last_unstale":"2024-06-29T15:59:43.612069+0000","last_undegraded":"2024-06-29T15:59:43.612069+0000","last_fullsized":"2024-06-29T15:59:43.612069+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:29:05.505946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612338+0000","last_change":"2024-06-29T15:59:42.627910+0000","last_active":"2024-06-29T15:59:43.612338+0000","last_peered":"2024-06-29T15:59:43.612338+0000","last_clean":"2024-06-29T15:59:43.612338+0000","last_became_active":"2024-06-29T15:59:42.627587+0000","last_became_peered":"2024-06-29T15:59:42.627587+0000","last_unstale":"2024-06-29T15:59:43.612338+0000","last_undegraded":"2024-06-29T15:59:43.612338+0000","last_fullsized":"2024-06-29T15:59:43.612338+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:55.120696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"265'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615822+0000","last_change":"2024-06-29T15:59:41.613245+0000","last_active":"2024-06-29T15:59:43.615822+0000","last_peered":"2024-06-29T15:59:43.615822+0000","last_clean":"2024-06-29T15:59:43.615822+0000","last_became_active":"2024-06-29T15:59:41.612506+0000","last_became_peered":"2024-06-29T15:59:41.612506+0000","last_unstale":"2024-06-29T15:59:43.615822+0000","last_undegraded":"2024-06-29T15:59:43.615822+0000","last_fullsized":"2024-06-29T15:59:43.615822+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:37.770855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613267+0000","last_change":"2024-06-29T15:59:42.611957+0000","last_active":"2024-06-29T15:59:43.613267+0000","last_peered":"2024-06-29T15:59:43.613267+0000","last_clean":"2024-06-29T15:59:43.613267+0000","last_became_active":"2024-06-29T15:59:42.611626+0000","last_became_peered":"2024-06-29T15:59:42.611626+0000","last_unstale":"2024-06-29T15:59:43.613267+0000","last_undegraded":"2024-06-29T15:59:43.613267+0000","last_fullsized":"2024-06-29T15:59:43.613267+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:56:32.009117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982429+0000","last_change":"2024-06-29T15:59:41.616596+0000","last_active":"2024-06-29T15:59:43.982429+0000","last_peered":"2024-06-29T15:59:43.982429+0000","last_clean":"2024-06-29T15:59:43.982429+0000","last_became_active":"2024-06-29T15:59:41.615847+0000","last_became_peered":"2024-06-29T15:59:41.615847+0000","last_unstale":"2024-06-29T15:59:43.982429+0000","last_undegraded":"2024-06-29T15:59:43.982429+0000","last_fullsized":"2024-06-29T15:59:43.982429+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:38.340808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613076+0000","last_change":"2024-06-29T15:59:42.619083+0000","last_active":"2024-06-29T15:59:43.613076+0000","last_peered":"2024-06-29T15:59:43.613076+0000","last_clean":"2024-06-29T15:59:43.613076+0000","last_became_active":"2024-06-29T15:59:42.618781+0000","last_became_peered":"2024-06-29T15:59:42.618781+0000","last_unstale":"2024-06-29T15:59:43.613076+0000","last_undegraded":"2024-06-29T15:59:43.613076+0000","last_fullsized":"2024-06-29T15:59:43.613076+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:42.796201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"265'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613668+0000","last_change":"2024-06-29T15:59:41.614722+0000","last_active":"2024-06-29T15:59:43.613668+0000","last_peered":"2024-06-29T15:59:43.613668+0000","last_clean":"2024-06-29T15:59:43.613668+0000","last_became_active":"2024-06-29T15:59:41.614311+0000","last_became_peered":"2024-06-29T15:59:41.614311+0000","last_unstale":"2024-06-29T15:59:43.613668+0000","last_undegraded":"2024-06-29T15:59:43.613668+0000","last_fullsized":"2024-06-29T15:59:43.613668+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:42.544078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982629+0000","last_change":"2024-06-29T15:59:42.614651+0000","last_active":"2024-06-29T15:59:43.982629+0000","last_peered":"2024-06-29T15:59:43.982629+0000","last_clean":"2024-06-29T15:59:43.982629+0000","last_became_active":"2024-06-29T15:59:42.614321+0000","last_became_peered":"2024-06-29T15:59:42.614321+0000","last_unstale":"2024-06-29T15:59:43.982629+0000","last_undegraded":"2024-06-29T15:59:43.982629+0000","last_fullsized":"2024-06-29T15:59:43.982629+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:38.993325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613268+0000","last_change":"2024-06-29T15:59:41.620179+0000","last_active":"2024-06-29T15:59:43.613268+0000","last_peered":"2024-06-29T15:59:43.613268+0000","last_clean":"2024-06-29T15:59:43.613268+0000","last_became_active":"2024-06-29T15:59:41.619869+0000","last_became_peered":"2024-06-29T15:59:41.619869+0000","last_unstale":"2024-06-29T15:59:43.613268+0000","last_undegraded":"2024-06-29T15:59:43.613268+0000","last_fullsized":"2024-06-29T15:59:43.613268+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:22.841090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612665+0000","last_change":"2024-06-29T15:59:42.620802+0000","last_active":"2024-06-29T15:59:43.612665+0000","last_peered":"2024-06-29T15:59:43.612665+0000","last_clean":"2024-06-29T15:59:43.612665+0000","last_became_active":"2024-06-29T15:59:42.620457+0000","last_became_peered":"2024-06-29T15:59:42.620457+0000","last_unstale":"2024-06-29T15:59:43.612665+0000","last_undegraded":"2024-06-29T15:59:43.612665+0000","last_fullsized":"2024-06-29T15:59:43.612665+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:14.891916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612714+0000","last_change":"2024-06-29T15:59:41.606766+0000","last_active":"2024-06-29T15:59:43.612714+0000","last_peered":"2024-06-29T15:59:43.612714+0000","last_clean":"2024-06-29T15:59:43.612714+0000","last_became_active":"2024-06-29T15:59:41.606117+0000","last_became_peered":"2024-06-29T15:59:41.606117+0000","last_unstale":"2024-06-29T15:59:43.612714+0000","last_undegraded":"2024-06-29T15:59:43.612714+0000","last_fullsized":"2024-06-29T15:59:43.612714+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:47:40.013941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981336+0000","last_change":"2024-06-29T15:59:42.618484+0000","last_active":"2024-06-29T15:59:43.981336+0000","last_peered":"2024-06-29T15:59:43.981336+0000","last_clean":"2024-06-29T15:59:43.981336+0000","last_became_active":"2024-06-29T15:59:42.618179+0000","last_became_peered":"2024-06-29T15:59:42.618179+0000","last_unstale":"2024-06-29T15:59:43.981336+0000","last_undegraded":"2024-06-29T15:59:43.981336+0000","last_fullsized":"2024-06-29T15:59:43.981336+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:06.862146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615725+0000","last_change":"2024-06-29T15:59:41.612784+0000","last_active":"2024-06-29T15:59:43.615725+0000","last_peered":"2024-06-29T15:59:43.615725+0000","last_clean":"2024-06-29T15:59:43.615725+0000","last_became_active":"2024-06-29T15:59:41.612447+0000","last_became_peered":"2024-06-29T15:59:41.612447+0000","last_unstale":"2024-06-29T15:59:43.615725+0000","last_undegraded":"2024-06-29T15:59:43.615725+0000","last_fullsized":"2024-06-29T15:59:43.615725+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:09.979323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615640+0000","last_change":"2024-06-29T15:59:42.614715+0000","last_active":"2024-06-29T15:59:43.615640+0000","last_peered":"2024-06-29T15:59:43.615640+0000","last_clean":"2024-06-29T15:59:43.615640+0000","last_became_active":"2024-06-29T15:59:42.614391+0000","last_became_peered":"2024-06-29T15:59:42.614391+0000","last_unstale":"2024-06-29T15:59:43.615640+0000","last_undegraded":"2024-06-29T15:59:43.615640+0000","last_fullsized":"2024-06-29T15:59:43.615640+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:55.910938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612922+0000","last_change":"2024-06-29T15:59:41.620207+0000","last_active":"2024-06-29T15:59:43.612922+0000","last_peered":"2024-06-29T15:59:43.612922+0000","last_clean":"2024-06-29T15:59:43.612922+0000","last_became_active":"2024-06-29T15:59:41.619899+0000","last_became_peered":"2024-06-29T15:59:41.619899+0000","last_unstale":"2024-06-29T15:59:43.612922+0000","last_undegraded":"2024-06-29T15:59:43.612922+0000","last_fullsized":"2024-06-29T15:59:43.612922+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:41:44.467762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615606+0000","last_change":"2024-06-29T15:59:42.628488+0000","last_active":"2024-06-29T15:59:43.615606+0000","last_peered":"2024-06-29T15:59:43.615606+0000","last_clean":"2024-06-29T15:59:43.615606+0000","last_became_active":"2024-06-29T15:59:42.628198+0000","last_became_peered":"2024-06-29T15:59:42.628198+0000","last_unstale":"2024-06-29T15:59:43.615606+0000","last_undegraded":"2024-06-29T15:59:43.615606+0000","last_fullsized":"2024-06-29T15:59:43.615606+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:36.230272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615751+0000","last_change":"2024-06-29T15:59:41.604728+0000","last_active":"2024-06-29T15:59:43.615751+0000","last_peered":"2024-06-29T15:59:43.615751+0000","last_clean":"2024-06-29T15:59:43.615751+0000","last_became_active":"2024-06-29T15:59:41.604180+0000","last_became_peered":"2024-06-29T15:59:41.604180+0000","last_unstale":"2024-06-29T15:59:43.615751+0000","last_undegraded":"2024-06-29T15:59:43.615751+0000","last_fullsized":"2024-06-29T15:59:43.615751+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:54.115285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981447+0000","last_change":"2024-06-29T15:59:42.617394+0000","last_active":"2024-06-29T15:59:43.981447+0000","last_peered":"2024-06-29T15:59:43.981447+0000","last_clean":"2024-06-29T15:59:43.981447+0000","last_became_active":"2024-06-29T15:59:42.616330+0000","last_became_peered":"2024-06-29T15:59:42.616330+0000","last_unstale":"2024-06-29T15:59:43.981447+0000","last_undegraded":"2024-06-29T15:59:43.981447+0000","last_fullsized":"2024-06-29T15:59:43.981447+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:34.350599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612656+0000","last_change":"2024-06-29T15:59:41.614034+0000","last_active":"2024-06-29T15:59:43.612656+0000","last_peered":"2024-06-29T15:59:43.612656+0000","last_clean":"2024-06-29T15:59:43.612656+0000","last_became_active":"2024-06-29T15:59:41.613732+0000","last_became_peered":"2024-06-29T15:59:41.613732+0000","last_unstale":"2024-06-29T15:59:43.612656+0000","last_undegraded":"2024-06-29T15:59:43.612656+0000","last_fullsized":"2024-06-29T15:59:43.612656+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:39:20.232138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982965+0000","last_change":"2024-06-29T15:59:42.615766+0000","last_active":"2024-06-29T15:59:43.982965+0000","last_peered":"2024-06-29T15:59:43.982965+0000","last_clean":"2024-06-29T15:59:43.982965+0000","last_became_active":"2024-06-29T15:59:42.613917+0000","last_became_peered":"2024-06-29T15:59:42.613917+0000","last_unstale":"2024-06-29T15:59:43.982965+0000","last_undegraded":"2024-06-29T15:59:43.982965+0000","last_fullsized":"2024-06-29T15:59:43.982965+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:33:11.115404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"266'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.617472+0000","last_change":"2024-06-29T15:59:41.606064+0000","last_active":"2024-06-29T15:59:43.617472+0000","last_peered":"2024-06-29T15:59:43.617472+0000","last_clean":"2024-06-29T15:59:43.617472+0000","last_became_active":"2024-06-29T15:59:41.605736+0000","last_became_peered":"2024-06-29T15:59:41.605736+0000","last_unstale":"2024-06-29T15:59:43.617472+0000","last_undegraded":"2024-06-29T15:59:43.617472+0000","last_fullsized":"2024-06-29T15:59:43.617472+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:17.351019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982122+0000","last_change":"2024-06-29T15:59:42.604207+0000","last_active":"2024-06-29T15:59:43.982122+0000","last_peered":"2024-06-29T15:59:43.982122+0000","last_clean":"2024-06-29T15:59:43.982122+0000","last_became_active":"2024-06-29T15:59:42.603868+0000","last_became_peered":"2024-06-29T15:59:42.603868+0000","last_unstale":"2024-06-29T15:59:43.982122+0000","last_undegraded":"2024-06-29T15:59:43.982122+0000","last_fullsized":"2024-06-29T15:59:43.982122+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:43.549488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612557+0000","last_change":"2024-06-29T15:59:41.610960+0000","last_active":"2024-06-29T15:59:43.612557+0000","last_peered":"2024-06-29T15:59:43.612557+0000","last_clean":"2024-06-29T15:59:43.612557+0000","last_became_active":"2024-06-29T15:59:41.610665+0000","last_became_peered":"2024-06-29T15:59:41.610665+0000","last_unstale":"2024-06-29T15:59:43.612557+0000","last_undegraded":"2024-06-29T15:59:43.612557+0000","last_fullsized":"2024-06-29T15:59:43.612557+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:33.782504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.983265+0000","last_change":"2024-06-29T15:59:42.609228+0000","last_active":"2024-06-29T15:59:43.983265+0000","last_peered":"2024-06-29T15:59:43.983265+0000","last_clean":"2024-06-29T15:59:43.983265+0000","last_became_active":"2024-06-29T15:59:42.608875+0000","last_became_peered":"2024-06-29T15:59:42.608875+0000","last_unstale":"2024-06-29T15:59:43.983265+0000","last_undegraded":"2024-06-29T15:59:43.983265+0000","last_fullsized":"2024-06-29T15:59:43.983265+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:56:46.832275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612879+0000","last_change":"2024-06-29T15:59:41.617687+0000","last_active":"2024-06-29T15:59:43.612879+0000","last_peered":"2024-06-29T15:59:43.612879+0000","last_clean":"2024-06-29T15:59:43.612879+0000","last_became_active":"2024-06-29T15:59:41.616690+0000","last_became_peered":"2024-06-29T15:59:41.616690+0000","last_unstale":"2024-06-29T15:59:43.612879+0000","last_undegraded":"2024-06-29T15:59:43.612879+0000","last_fullsized":"2024-06-29T15:59:43.612879+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:50:25.372841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613597+0000","last_change":"2024-06-29T15:59:42.620784+0000","last_active":"2024-06-29T15:59:43.613597+0000","last_peered":"2024-06-29T15:59:43.613597+0000","last_clean":"2024-06-29T15:59:43.613597+0000","last_became_active":"2024-06-29T15:59:42.620497+0000","last_became_peered":"2024-06-29T15:59:42.620497+0000","last_unstale":"2024-06-29T15:59:43.613597+0000","last_undegraded":"2024-06-29T15:59:43.613597+0000","last_fullsized":"2024-06-29T15:59:43.613597+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:46:31.080213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612951+0000","last_change":"2024-06-29T15:59:41.615402+0000","last_active":"2024-06-29T15:59:43.612951+0000","last_peered":"2024-06-29T15:59:43.612951+0000","last_clean":"2024-06-29T15:59:43.612951+0000","last_became_active":"2024-06-29T15:59:41.615099+0000","last_became_peered":"2024-06-29T15:59:41.615099+0000","last_unstale":"2024-06-29T15:59:43.612951+0000","last_undegraded":"2024-06-29T15:59:43.612951+0000","last_fullsized":"2024-06-29T15:59:43.612951+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:33:10.109993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612508+0000","last_change":"2024-06-29T15:59:42.614919+0000","last_active":"2024-06-29T15:59:43.612508+0000","last_peered":"2024-06-29T15:59:43.612508+0000","last_clean":"2024-06-29T15:59:43.612508+0000","last_became_active":"2024-06-29T15:59:42.614567+0000","last_became_peered":"2024-06-29T15:59:42.614567+0000","last_unstale":"2024-06-29T15:59:43.612508+0000","last_undegraded":"2024-06-29T15:59:43.612508+0000","last_fullsized":"2024-06-29T15:59:43.612508+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:53:16.228824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612315+0000","last_change":"2024-06-29T15:59:41.619869+0000","last_active":"2024-06-29T15:59:43.612315+0000","last_peered":"2024-06-29T15:59:43.612315+0000","last_clean":"2024-06-29T15:59:43.612315+0000","last_became_active":"2024-06-29T15:59:41.619376+0000","last_became_peered":"2024-06-29T15:59:41.619376+0000","last_unstale":"2024-06-29T15:59:43.612315+0000","last_undegraded":"2024-06-29T15:59:43.612315+0000","last_fullsized":"2024-06-29T15:59:43.612315+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:24.300817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615718+0000","last_change":"2024-06-29T15:59:42.624606+0000","last_active":"2024-06-29T15:59:43.615718+0000","last_peered":"2024-06-29T15:59:43.615718+0000","last_clean":"2024-06-29T15:59:43.615718+0000","last_became_active":"2024-06-29T15:59:42.624300+0000","last_became_peered":"2024-06-29T15:59:42.624300+0000","last_unstale":"2024-06-29T15:59:43.615718+0000","last_undegraded":"2024-06-29T15:59:43.615718+0000","last_fullsized":"2024-06-29T15:59:43.615718+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:25:07.998005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982240+0000","last_change":"2024-06-29T15:59:41.613044+0000","last_active":"2024-06-29T15:59:43.982240+0000","last_peered":"2024-06-29T15:59:43.982240+0000","last_clean":"2024-06-29T15:59:43.982240+0000","last_became_active":"2024-06-29T15:59:41.612373+0000","last_became_peered":"2024-06-29T15:59:41.612373+0000","last_unstale":"2024-06-29T15:59:43.982240+0000","last_undegraded":"2024-06-29T15:59:43.982240+0000","last_fullsized":"2024-06-29T15:59:43.982240+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:57.112983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982552+0000","last_change":"2024-06-29T15:59:42.610269+0000","last_active":"2024-06-29T15:59:43.982552+0000","last_peered":"2024-06-29T15:59:43.982552+0000","last_clean":"2024-06-29T15:59:43.982552+0000","last_became_active":"2024-06-29T15:59:42.609558+0000","last_became_peered":"2024-06-29T15:59:42.609558+0000","last_unstale":"2024-06-29T15:59:43.982552+0000","last_undegraded":"2024-06-29T15:59:43.982552+0000","last_fullsized":"2024-06-29T15:59:43.982552+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:50:57.932595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612041+0000","last_change":"2024-06-29T15:59:41.619398+0000","last_active":"2024-06-29T15:59:43.612041+0000","last_peered":"2024-06-29T15:59:43.612041+0000","last_clean":"2024-06-29T15:59:43.612041+0000","last_became_active":"2024-06-29T15:59:41.619090+0000","last_became_peered":"2024-06-29T15:59:41.619090+0000","last_unstale":"2024-06-29T15:59:43.612041+0000","last_undegraded":"2024-06-29T15:59:43.612041+0000","last_fullsized":"2024-06-29T15:59:43.612041+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:20:43.427382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613305+0000","last_change":"2024-06-29T15:59:42.613952+0000","last_active":"2024-06-29T15:59:43.613305+0000","last_peered":"2024-06-29T15:59:43.613305+0000","last_clean":"2024-06-29T15:59:43.613305+0000","last_became_active":"2024-06-29T15:59:42.612934+0000","last_became_peered":"2024-06-29T15:59:42.612934+0000","last_unstale":"2024-06-29T15:59:43.613305+0000","last_undegraded":"2024-06-29T15:59:43.613305+0000","last_fullsized":"2024-06-29T15:59:43.613305+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:32.759053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"255'261","reported_seq":1291,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612548+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:59:43.612548+0000","last_peered":"2024-06-29T15:59:43.612548+0000","last_clean":"2024-06-29T15:59:43.612548+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:59:43.612548+0000","last_undegraded":"2024-06-29T15:59:43.612548+0000","last_fullsized":"2024-06-29T15:59:43.612548+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1686,"num_read_kb":4807,"num_write":425,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1686,"num_read_kb":4807,"num_write":425,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261,"ondisk_log_size":261,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576011,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53096,"kb_used_data":3868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318744,"statfs":{"total":96636764160,"available":96582393856,"internally_reserved":0,"allocated":3960832,"data_stored":2254183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834576012,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64996,"kb_used_data":3288,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306844,"statfs":{"total":96636764160,"available":96570208256,"internally_reserved":0,"allocated":3366912,"data_stored":1658999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834576010,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48376,"kb_used_data":3304,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323464,"statfs":{"total":96636764160,"available":96587227136,"internally_reserved":0,"allocated":3383296,"data_stored":1665517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834576011,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82992,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94288848,"statfs":{"total":96636764160,"available":96551780352,"internally_reserved":0,"allocated":3375104,"data_stored":1664467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834576011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48352,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323488,"statfs":{"total":96636764160,"available":96587251712,"internally_reserved":0,"allocated":3358720,"data_stored":1657969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52528,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319312,"statfs":{"total":96636764160,"available":96582975488,"internally_reserved":0,"allocated":3375104,"data_stored":1663939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834576010,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48368,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":3375104,"data_stored":1663871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834576011,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61424,"kb_used_data":3876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310416,"statfs":{"total":96636764160,"available":96573865984,"internally_reserved":0,"allocated":3969024,"data_stored":2254275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:59:46.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T15:59:46.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.792+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3241759880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea40b5250 msgr2=0x7f2ea40d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.792+0000 7f2eabd8c640 1 --2- 172.21.15.81:0/3241759880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea40b5250 0x7f2ea40d1fc0 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40a1b70 tx=0x7f2e9c02aac0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.792+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3241759880 shutdown_connections 2024-06-29T15:59:46.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.792+0000 7f2eabd8c640 1 --2- 172.21.15.81:0/3241759880 >> v2:172.21.15.167:3300/0 conn(0x7f2ea40d0b90 0x7f2ea40d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.792+0000 7f2eabd8c640 1 --2- 172.21.15.81:0/3241759880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea40b5250 0x7f2ea40d1fc0 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.792+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3241759880 >> 172.21.15.81:0/3241759880 conn(0x7f2ea409cf30 msgr2=0x7f2ea40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:46.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.792+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3241759880 shutdown_connections 2024-06-29T15:59:46.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.792+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3241759880 wait complete. 2024-06-29T15:59:46.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.794+0000 7f2eabd8c640 1 Processor -- start 2024-06-29T15:59:46.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.794+0000 7f2eabd8c640 1 -- start start 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.795+0000 7f2eabd8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea40b5250 0x7f2ea40b4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.795+0000 7f2eabd8c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea40b9040 0x7f2ea41f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.795+0000 7f2eabd8c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ea40a0a00 0x7f2ea40a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.795+0000 7f2eabd8c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea40d2500 con 0x7f2ea40b5250 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.795+0000 7f2eabd8c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea40b84b0 con 0x7f2ea40b9040 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.795+0000 7f2eabd8c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea40d2680 con 0x7f2ea40a0a00 2024-06-29T15:59:46.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2ea9b28640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea40b5250 0x7f2ea40b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2ea9b28640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea40b5250 0x7f2ea40b4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48360/0 (socket says 172.21.15.81:48360) 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2ea9b28640 1 -- 172.21.15.81:0/3353273542 learned_addr learned my addr 172.21.15.81:0/3353273542 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2ea9b28640 1 -- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ea40a0a00 msgr2=0x7f2ea40a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2ea9b28640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ea40a0a00 0x7f2ea40a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2ea9b28640 1 -- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea40b9040 msgr2=0x7f2ea41f7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2ea9b28640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea40b9040 0x7f2ea41f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2ea9b28640 1 -- 172.21.15.81:0/3353273542 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea40a1350 con 0x7f2ea40b5250 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2ea9b28640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea40b5250 0x7f2ea40b4520 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000b420 tx=0x7f2ea000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2e9a7fc640 1 -- 172.21.15.81:0/3353273542 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0011830 con 0x7f2ea40b5250 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2e9a7fc640 1 -- 172.21.15.81:0/3353273542 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ea0016070 con 0x7f2ea40b5250 2024-06-29T15:59:46.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3353273542 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea40b2e10 con 0x7f2ea40b5250 2024-06-29T15:59:46.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2e9a7fc640 1 -- 172.21.15.81:0/3353273542 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0011d60 con 0x7f2ea40b5250 2024-06-29T15:59:46.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.796+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3353273542 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea40b33d0 con 0x7f2ea40b5250 2024-06-29T15:59:46.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.797+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3353273542 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea40d2c10 con 0x7f2ea40b5250 2024-06-29T15:59:46.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.798+0000 7f2e9a7fc640 1 -- 172.21.15.81:0/3353273542 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f2ea001b460 con 0x7f2ea40b5250 2024-06-29T15:59:46.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.799+0000 7f2e9a7fc640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e88076420 0x7f2e88078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:46.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.799+0000 7f2e9a7fc640 1 -- 172.21.15.81:0/3353273542 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f2ea0097650 con 0x7f2ea40b5250 2024-06-29T15:59:46.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.799+0000 7f2ea9327640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e88076420 0x7f2e88078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:46.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.800+0000 7f2ea9327640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e88076420 0x7f2e88078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e940058c0 tx=0x7f2e94005850 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:46.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.803+0000 7f2e9a7fc640 1 -- 172.21.15.81:0/3353273542 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2ea40a1350 con 0x7f2ea40b5250 2024-06-29T15:59:46.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.979+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3353273542 --> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2ea40b4a60 con 0x7f2e88076420 2024-06-29T15:59:46.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.984+0000 7f2e9a7fc640 1 -- 172.21.15.81:0/3353273542 <== mgr.10028 v2:172.21.15.167:6832/1824030697 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252499 (secure 0 0 0) 0x7f2ea40b4a60 con 0x7f2e88076420 2024-06-29T15:59:46.987 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:46.989 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T15:59:46.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e88076420 msgr2=0x7f2e88078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e88076420 0x7f2e88078930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e940058c0 tx=0x7f2e94005850 comp rx=0 tx=0).stop 2024-06-29T15:59:46.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3353273542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea40b5250 msgr2=0x7f2ea40b4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:46.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea40b5250 0x7f2ea40b4520 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000b420 tx=0x7f2ea000b8e0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3353273542 shutdown_connections 2024-06-29T15:59:46.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f2e88076420 0x7f2e88078930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2ea40a0a00 0x7f2ea40a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2ea40b9040 0x7f2ea41f7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 --2- 172.21.15.81:0/3353273542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ea40b5250 0x7f2ea40b4520 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:46.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3353273542 >> 172.21.15.81:0/3353273542 conn(0x7f2ea409cf30 msgr2=0x7f2ea406e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:46.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3353273542 shutdown_connections 2024-06-29T15:59:46.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:46.988+0000 7f2eabd8c640 1 -- 172.21.15.81:0/3353273542 wait complete. 2024-06-29T15:59:46.999 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T15:59:46.206063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":1686,"num_read_kb":4807,"num_write":470,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":460132,"kb_used_data":27504,"kb_used_omap":12,"kb_used_meta":432499,"kb_avail":754514588,"statfs":{"total":773094113280,"available":772622938112,"internally_reserved":0,"allocated":28164096,"data_stored":14483220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":442879538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.617516"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.983371+0000","last_change":"2024-06-29T15:59:42.612533+0000","last_active":"2024-06-29T15:59:43.983371+0000","last_peered":"2024-06-29T15:59:43.983371+0000","last_clean":"2024-06-29T15:59:43.983371+0000","last_became_active":"2024-06-29T15:59:42.612205+0000","last_became_peered":"2024-06-29T15:59:42.612205+0000","last_unstale":"2024-06-29T15:59:43.983371+0000","last_undegraded":"2024-06-29T15:59:43.983371+0000","last_fullsized":"2024-06-29T15:59:43.983371+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:57:04.823899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981487+0000","last_change":"2024-06-29T15:59:42.617082+0000","last_active":"2024-06-29T15:59:43.981487+0000","last_peered":"2024-06-29T15:59:43.981487+0000","last_clean":"2024-06-29T15:59:43.981487+0000","last_became_active":"2024-06-29T15:59:42.616754+0000","last_became_peered":"2024-06-29T15:59:42.616754+0000","last_unstale":"2024-06-29T15:59:43.981487+0000","last_undegraded":"2024-06-29T15:59:43.981487+0000","last_fullsized":"2024-06-29T15:59:43.981487+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:45:15.200629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612347+0000","last_change":"2024-06-29T15:59:42.619309+0000","last_active":"2024-06-29T15:59:43.612347+0000","last_peered":"2024-06-29T15:59:43.612347+0000","last_clean":"2024-06-29T15:59:43.612347+0000","last_became_active":"2024-06-29T15:59:42.619021+0000","last_became_peered":"2024-06-29T15:59:42.619021+0000","last_unstale":"2024-06-29T15:59:43.612347+0000","last_undegraded":"2024-06-29T15:59:43.612347+0000","last_fullsized":"2024-06-29T15:59:43.612347+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:13:03.731967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613138+0000","last_change":"2024-06-29T15:59:42.623148+0000","last_active":"2024-06-29T15:59:43.613138+0000","last_peered":"2024-06-29T15:59:43.613138+0000","last_clean":"2024-06-29T15:59:43.613138+0000","last_became_active":"2024-06-29T15:59:42.622338+0000","last_became_peered":"2024-06-29T15:59:42.622338+0000","last_unstale":"2024-06-29T15:59:43.613138+0000","last_undegraded":"2024-06-29T15:59:43.613138+0000","last_fullsized":"2024-06-29T15:59:43.613138+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:42.796201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612763+0000","last_change":"2024-06-29T15:59:42.620958+0000","last_active":"2024-06-29T15:59:43.612763+0000","last_peered":"2024-06-29T15:59:43.612763+0000","last_clean":"2024-06-29T15:59:43.612763+0000","last_became_active":"2024-06-29T15:59:42.620660+0000","last_became_peered":"2024-06-29T15:59:42.620660+0000","last_unstale":"2024-06-29T15:59:43.612763+0000","last_undegraded":"2024-06-29T15:59:43.612763+0000","last_fullsized":"2024-06-29T15:59:43.612763+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:18.356430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612682+0000","last_change":"2024-06-29T15:59:42.622147+0000","last_active":"2024-06-29T15:59:43.612682+0000","last_peered":"2024-06-29T15:59:43.612682+0000","last_clean":"2024-06-29T15:59:43.612682+0000","last_became_active":"2024-06-29T15:59:42.621853+0000","last_became_peered":"2024-06-29T15:59:42.621853+0000","last_unstale":"2024-06-29T15:59:43.612682+0000","last_undegraded":"2024-06-29T15:59:43.612682+0000","last_fullsized":"2024-06-29T15:59:43.612682+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:13:03.731967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982601+0000","last_change":"2024-06-29T15:59:42.613206+0000","last_active":"2024-06-29T15:59:43.982601+0000","last_peered":"2024-06-29T15:59:43.982601+0000","last_clean":"2024-06-29T15:59:43.982601+0000","last_became_active":"2024-06-29T15:59:42.612868+0000","last_became_peered":"2024-06-29T15:59:42.612868+0000","last_unstale":"2024-06-29T15:59:43.982601+0000","last_undegraded":"2024-06-29T15:59:43.982601+0000","last_fullsized":"2024-06-29T15:59:43.982601+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:16:02.811328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615533+0000","last_change":"2024-06-29T15:59:42.615693+0000","last_active":"2024-06-29T15:59:43.615533+0000","last_peered":"2024-06-29T15:59:43.615533+0000","last_clean":"2024-06-29T15:59:43.615533+0000","last_became_active":"2024-06-29T15:59:42.615368+0000","last_became_peered":"2024-06-29T15:59:42.615368+0000","last_unstale":"2024-06-29T15:59:43.615533+0000","last_undegraded":"2024-06-29T15:59:43.615533+0000","last_fullsized":"2024-06-29T15:59:43.615533+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:38:14.798255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981405+0000","last_change":"2024-06-29T15:59:42.620130+0000","last_active":"2024-06-29T15:59:43.981405+0000","last_peered":"2024-06-29T15:59:43.981405+0000","last_clean":"2024-06-29T15:59:43.981405+0000","last_became_active":"2024-06-29T15:59:42.616841+0000","last_became_peered":"2024-06-29T15:59:42.616841+0000","last_unstale":"2024-06-29T15:59:43.981405+0000","last_undegraded":"2024-06-29T15:59:43.981405+0000","last_fullsized":"2024-06-29T15:59:43.981405+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:36:54.989299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.983503+0000","last_change":"2024-06-29T15:59:42.619822+0000","last_active":"2024-06-29T15:59:43.983503+0000","last_peered":"2024-06-29T15:59:43.983503+0000","last_clean":"2024-06-29T15:59:43.983503+0000","last_became_active":"2024-06-29T15:59:42.619236+0000","last_became_peered":"2024-06-29T15:59:42.619236+0000","last_unstale":"2024-06-29T15:59:43.983503+0000","last_undegraded":"2024-06-29T15:59:43.983503+0000","last_fullsized":"2024-06-29T15:59:43.983503+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:50:20.652399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615893+0000","last_change":"2024-06-29T15:59:42.627259+0000","last_active":"2024-06-29T15:59:43.615893+0000","last_peered":"2024-06-29T15:59:43.615893+0000","last_clean":"2024-06-29T15:59:43.615893+0000","last_became_active":"2024-06-29T15:59:42.626511+0000","last_became_peered":"2024-06-29T15:59:42.626511+0000","last_unstale":"2024-06-29T15:59:43.615893+0000","last_undegraded":"2024-06-29T15:59:43.615893+0000","last_fullsized":"2024-06-29T15:59:43.615893+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:35.560322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612724+0000","last_change":"2024-06-29T15:59:42.615270+0000","last_active":"2024-06-29T15:59:43.612724+0000","last_peered":"2024-06-29T15:59:43.612724+0000","last_clean":"2024-06-29T15:59:43.612724+0000","last_became_active":"2024-06-29T15:59:42.614950+0000","last_became_peered":"2024-06-29T15:59:42.614950+0000","last_unstale":"2024-06-29T15:59:43.612724+0000","last_undegraded":"2024-06-29T15:59:43.612724+0000","last_fullsized":"2024-06-29T15:59:43.612724+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:54:37.935864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.983177+0000","last_change":"2024-06-29T15:59:42.615216+0000","last_active":"2024-06-29T15:59:43.983177+0000","last_peered":"2024-06-29T15:59:43.983177+0000","last_clean":"2024-06-29T15:59:43.983177+0000","last_became_active":"2024-06-29T15:59:42.613448+0000","last_became_peered":"2024-06-29T15:59:42.613448+0000","last_unstale":"2024-06-29T15:59:43.983177+0000","last_undegraded":"2024-06-29T15:59:43.983177+0000","last_fullsized":"2024-06-29T15:59:43.983177+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:44:01.018930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982331+0000","last_change":"2024-06-29T15:59:42.621522+0000","last_active":"2024-06-29T15:59:43.982331+0000","last_peered":"2024-06-29T15:59:43.982331+0000","last_clean":"2024-06-29T15:59:43.982331+0000","last_became_active":"2024-06-29T15:59:42.621211+0000","last_became_peered":"2024-06-29T15:59:42.621211+0000","last_unstale":"2024-06-29T15:59:43.982331+0000","last_undegraded":"2024-06-29T15:59:43.982331+0000","last_fullsized":"2024-06-29T15:59:43.982331+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:09:17.506409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.611907+0000","last_change":"2024-06-29T15:59:42.629405+0000","last_active":"2024-06-29T15:59:43.611907+0000","last_peered":"2024-06-29T15:59:43.611907+0000","last_clean":"2024-06-29T15:59:43.611907+0000","last_became_active":"2024-06-29T15:59:42.629036+0000","last_became_peered":"2024-06-29T15:59:42.629036+0000","last_unstale":"2024-06-29T15:59:43.611907+0000","last_undegraded":"2024-06-29T15:59:43.611907+0000","last_fullsized":"2024-06-29T15:59:43.611907+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:38.776266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612463+0000","last_change":"2024-06-29T15:59:42.620609+0000","last_active":"2024-06-29T15:59:43.612463+0000","last_peered":"2024-06-29T15:59:43.612463+0000","last_clean":"2024-06-29T15:59:43.612463+0000","last_became_active":"2024-06-29T15:59:42.620296+0000","last_became_peered":"2024-06-29T15:59:42.620296+0000","last_unstale":"2024-06-29T15:59:43.612463+0000","last_undegraded":"2024-06-29T15:59:43.612463+0000","last_fullsized":"2024-06-29T15:59:43.612463+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:43.293757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982582+0000","last_change":"2024-06-29T15:59:42.618052+0000","last_active":"2024-06-29T15:59:43.982582+0000","last_peered":"2024-06-29T15:59:43.982582+0000","last_clean":"2024-06-29T15:59:43.982582+0000","last_became_active":"2024-06-29T15:59:42.617707+0000","last_became_peered":"2024-06-29T15:59:42.617707+0000","last_unstale":"2024-06-29T15:59:43.982582+0000","last_undegraded":"2024-06-29T15:59:43.982582+0000","last_fullsized":"2024-06-29T15:59:43.982582+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:50:26.378251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982569+0000","last_change":"2024-06-29T15:59:42.622834+0000","last_active":"2024-06-29T15:59:43.982569+0000","last_peered":"2024-06-29T15:59:43.982569+0000","last_clean":"2024-06-29T15:59:43.982569+0000","last_became_active":"2024-06-29T15:59:42.622535+0000","last_became_peered":"2024-06-29T15:59:42.622535+0000","last_unstale":"2024-06-29T15:59:43.982569+0000","last_undegraded":"2024-06-29T15:59:43.982569+0000","last_fullsized":"2024-06-29T15:59:43.982569+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:48.971622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982450+0000","last_change":"2024-06-29T15:59:42.616261+0000","last_active":"2024-06-29T15:59:43.982450+0000","last_peered":"2024-06-29T15:59:43.982450+0000","last_clean":"2024-06-29T15:59:43.982450+0000","last_became_active":"2024-06-29T15:59:42.615914+0000","last_became_peered":"2024-06-29T15:59:42.615914+0000","last_unstale":"2024-06-29T15:59:43.982450+0000","last_undegraded":"2024-06-29T15:59:43.982450+0000","last_fullsized":"2024-06-29T15:59:43.982450+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:24:41.434894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981336+0000","last_change":"2024-06-29T15:59:42.631226+0000","last_active":"2024-06-29T15:59:43.981336+0000","last_peered":"2024-06-29T15:59:43.981336+0000","last_clean":"2024-06-29T15:59:43.981336+0000","last_became_active":"2024-06-29T15:59:42.630894+0000","last_became_peered":"2024-06-29T15:59:42.630894+0000","last_unstale":"2024-06-29T15:59:43.981336+0000","last_undegraded":"2024-06-29T15:59:43.981336+0000","last_fullsized":"2024-06-29T15:59:43.981336+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:34:32.094496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.616120+0000","last_change":"2024-06-29T15:59:42.626472+0000","last_active":"2024-06-29T15:59:43.616120+0000","last_peered":"2024-06-29T15:59:43.616120+0000","last_clean":"2024-06-29T15:59:43.616120+0000","last_became_active":"2024-06-29T15:59:42.626168+0000","last_became_peered":"2024-06-29T15:59:42.626168+0000","last_unstale":"2024-06-29T15:59:43.616120+0000","last_undegraded":"2024-06-29T15:59:43.616120+0000","last_fullsized":"2024-06-29T15:59:43.616120+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:03:17.919794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612090+0000","last_change":"2024-06-29T15:59:42.611394+0000","last_active":"2024-06-29T15:59:43.612090+0000","last_peered":"2024-06-29T15:59:43.612090+0000","last_clean":"2024-06-29T15:59:43.612090+0000","last_became_active":"2024-06-29T15:59:42.611087+0000","last_became_peered":"2024-06-29T15:59:42.611087+0000","last_unstale":"2024-06-29T15:59:43.612090+0000","last_undegraded":"2024-06-29T15:59:43.612090+0000","last_fullsized":"2024-06-29T15:59:43.612090+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:10.984734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982570+0000","last_change":"2024-06-29T15:59:42.622727+0000","last_active":"2024-06-29T15:59:43.982570+0000","last_peered":"2024-06-29T15:59:43.982570+0000","last_clean":"2024-06-29T15:59:43.982570+0000","last_became_active":"2024-06-29T15:59:42.622426+0000","last_became_peered":"2024-06-29T15:59:42.622426+0000","last_unstale":"2024-06-29T15:59:43.982570+0000","last_undegraded":"2024-06-29T15:59:43.982570+0000","last_fullsized":"2024-06-29T15:59:43.982570+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:22:19.017174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981999+0000","last_change":"2024-06-29T15:59:42.618541+0000","last_active":"2024-06-29T15:59:43.981999+0000","last_peered":"2024-06-29T15:59:43.981999+0000","last_clean":"2024-06-29T15:59:43.981999+0000","last_became_active":"2024-06-29T15:59:42.618192+0000","last_became_peered":"2024-06-29T15:59:42.618192+0000","last_unstale":"2024-06-29T15:59:43.981999+0000","last_undegraded":"2024-06-29T15:59:43.981999+0000","last_fullsized":"2024-06-29T15:59:43.981999+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:41:45.473172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982775+0000","last_change":"2024-06-29T15:59:42.616435+0000","last_active":"2024-06-29T15:59:43.982775+0000","last_peered":"2024-06-29T15:59:43.982775+0000","last_clean":"2024-06-29T15:59:43.982775+0000","last_became_active":"2024-06-29T15:59:42.616130+0000","last_became_peered":"2024-06-29T15:59:42.616130+0000","last_unstale":"2024-06-29T15:59:43.982775+0000","last_undegraded":"2024-06-29T15:59:43.982775+0000","last_fullsized":"2024-06-29T15:59:43.982775+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:32.549725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.983630+0000","last_change":"2024-06-29T15:59:42.604154+0000","last_active":"2024-06-29T15:59:43.983630+0000","last_peered":"2024-06-29T15:59:43.983630+0000","last_clean":"2024-06-29T15:59:43.983630+0000","last_became_active":"2024-06-29T15:59:42.603824+0000","last_became_peered":"2024-06-29T15:59:42.603824+0000","last_unstale":"2024-06-29T15:59:43.983630+0000","last_undegraded":"2024-06-29T15:59:43.983630+0000","last_fullsized":"2024-06-29T15:59:43.983630+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:58.118394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613040+0000","last_change":"2024-06-29T15:59:42.619753+0000","last_active":"2024-06-29T15:59:43.613040+0000","last_peered":"2024-06-29T15:59:43.613040+0000","last_clean":"2024-06-29T15:59:43.613040+0000","last_became_active":"2024-06-29T15:59:42.619467+0000","last_became_peered":"2024-06-29T15:59:42.619467+0000","last_unstale":"2024-06-29T15:59:43.613040+0000","last_undegraded":"2024-06-29T15:59:43.613040+0000","last_fullsized":"2024-06-29T15:59:43.613040+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:39:41.301776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613268+0000","last_change":"2024-06-29T15:59:42.626257+0000","last_active":"2024-06-29T15:59:43.613268+0000","last_peered":"2024-06-29T15:59:43.613268+0000","last_clean":"2024-06-29T15:59:43.613268+0000","last_became_active":"2024-06-29T15:59:42.625873+0000","last_became_peered":"2024-06-29T15:59:42.625873+0000","last_unstale":"2024-06-29T15:59:43.613268+0000","last_undegraded":"2024-06-29T15:59:43.613268+0000","last_fullsized":"2024-06-29T15:59:43.613268+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:32.759053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982156+0000","last_change":"2024-06-29T15:59:42.619637+0000","last_active":"2024-06-29T15:59:43.982156+0000","last_peered":"2024-06-29T15:59:43.982156+0000","last_clean":"2024-06-29T15:59:43.982156+0000","last_became_active":"2024-06-29T15:59:42.619317+0000","last_became_peered":"2024-06-29T15:59:42.619317+0000","last_unstale":"2024-06-29T15:59:43.982156+0000","last_undegraded":"2024-06-29T15:59:43.982156+0000","last_fullsized":"2024-06-29T15:59:43.982156+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:53:56.981682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981718+0000","last_change":"2024-06-29T15:59:42.611174+0000","last_active":"2024-06-29T15:59:43.981718+0000","last_peered":"2024-06-29T15:59:43.981718+0000","last_clean":"2024-06-29T15:59:43.981718+0000","last_became_active":"2024-06-29T15:59:42.610838+0000","last_became_peered":"2024-06-29T15:59:42.610838+0000","last_unstale":"2024-06-29T15:59:43.981718+0000","last_undegraded":"2024-06-29T15:59:43.981718+0000","last_fullsized":"2024-06-29T15:59:43.981718+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:57.182169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.611934+0000","last_change":"2024-06-29T15:59:42.621232+0000","last_active":"2024-06-29T15:59:43.611934+0000","last_peered":"2024-06-29T15:59:43.611934+0000","last_clean":"2024-06-29T15:59:43.611934+0000","last_became_active":"2024-06-29T15:59:42.620924+0000","last_became_peered":"2024-06-29T15:59:42.620924+0000","last_unstale":"2024-06-29T15:59:43.611934+0000","last_undegraded":"2024-06-29T15:59:43.611934+0000","last_fullsized":"2024-06-29T15:59:43.611934+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:42.508804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612989+0000","last_change":"2024-06-29T15:59:42.618187+0000","last_active":"2024-06-29T15:59:43.612989+0000","last_peered":"2024-06-29T15:59:43.612989+0000","last_clean":"2024-06-29T15:59:43.612989+0000","last_became_active":"2024-06-29T15:59:42.617766+0000","last_became_peered":"2024-06-29T15:59:42.617766+0000","last_unstale":"2024-06-29T15:59:43.612989+0000","last_undegraded":"2024-06-29T15:59:43.612989+0000","last_fullsized":"2024-06-29T15:59:43.612989+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:35:30.939091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"265'4","reported_seq":18,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982801+0000","last_change":"2024-06-29T15:59:41.614369+0000","last_active":"2024-06-29T15:59:43.982801+0000","last_peered":"2024-06-29T15:59:43.982801+0000","last_clean":"2024-06-29T15:59:43.982801+0000","last_became_active":"2024-06-29T15:59:41.613958+0000","last_became_peered":"2024-06-29T15:59:41.613958+0000","last_unstale":"2024-06-29T15:59:43.982801+0000","last_undegraded":"2024-06-29T15:59:43.982801+0000","last_fullsized":"2024-06-29T15:59:43.982801+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:41:38.903895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982465+0000","last_change":"2024-06-29T15:59:42.608922+0000","last_active":"2024-06-29T15:59:43.982465+0000","last_peered":"2024-06-29T15:59:43.982465+0000","last_clean":"2024-06-29T15:59:43.982465+0000","last_became_active":"2024-06-29T15:59:42.608484+0000","last_became_peered":"2024-06-29T15:59:42.608484+0000","last_unstale":"2024-06-29T15:59:43.982465+0000","last_undegraded":"2024-06-29T15:59:43.982465+0000","last_fullsized":"2024-06-29T15:59:43.982465+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:09:00.573266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"265'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613459+0000","last_change":"2024-06-29T15:59:41.616918+0000","last_active":"2024-06-29T15:59:43.613459+0000","last_peered":"2024-06-29T15:59:43.613459+0000","last_clean":"2024-06-29T15:59:43.613459+0000","last_became_active":"2024-06-29T15:59:41.616589+0000","last_became_peered":"2024-06-29T15:59:41.616589+0000","last_unstale":"2024-06-29T15:59:43.613459+0000","last_undegraded":"2024-06-29T15:59:43.613459+0000","last_fullsized":"2024-06-29T15:59:43.613459+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:52.272460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612910+0000","last_change":"2024-06-29T15:59:42.621818+0000","last_active":"2024-06-29T15:59:43.612910+0000","last_peered":"2024-06-29T15:59:43.612910+0000","last_clean":"2024-06-29T15:59:43.612910+0000","last_became_active":"2024-06-29T15:59:42.621493+0000","last_became_peered":"2024-06-29T15:59:42.621493+0000","last_unstale":"2024-06-29T15:59:43.612910+0000","last_undegraded":"2024-06-29T15:59:43.612910+0000","last_fullsized":"2024-06-29T15:59:43.612910+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:18.356430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613547+0000","last_change":"2024-06-29T15:59:41.614605+0000","last_active":"2024-06-29T15:59:43.613547+0000","last_peered":"2024-06-29T15:59:43.613547+0000","last_clean":"2024-06-29T15:59:43.613547+0000","last_became_active":"2024-06-29T15:59:41.613882+0000","last_became_peered":"2024-06-29T15:59:41.613882+0000","last_unstale":"2024-06-29T15:59:43.613547+0000","last_undegraded":"2024-06-29T15:59:43.613547+0000","last_fullsized":"2024-06-29T15:59:43.613547+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:16:56.176759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615987+0000","last_change":"2024-06-29T15:59:42.627498+0000","last_active":"2024-06-29T15:59:43.615987+0000","last_peered":"2024-06-29T15:59:43.615987+0000","last_clean":"2024-06-29T15:59:43.615987+0000","last_became_active":"2024-06-29T15:59:42.626935+0000","last_became_peered":"2024-06-29T15:59:42.626935+0000","last_unstale":"2024-06-29T15:59:43.615987+0000","last_undegraded":"2024-06-29T15:59:43.615987+0000","last_fullsized":"2024-06-29T15:59:43.615987+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:46:10.842632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613686+0000","last_change":"2024-06-29T15:59:41.613331+0000","last_active":"2024-06-29T15:59:43.613686+0000","last_peered":"2024-06-29T15:59:43.613686+0000","last_clean":"2024-06-29T15:59:43.613686+0000","last_became_active":"2024-06-29T15:59:41.613080+0000","last_became_peered":"2024-06-29T15:59:41.613080+0000","last_unstale":"2024-06-29T15:59:43.613686+0000","last_undegraded":"2024-06-29T15:59:43.613686+0000","last_fullsized":"2024-06-29T15:59:43.613686+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:40:28.215838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615728+0000","last_change":"2024-06-29T15:59:42.616863+0000","last_active":"2024-06-29T15:59:43.615728+0000","last_peered":"2024-06-29T15:59:43.615728+0000","last_clean":"2024-06-29T15:59:43.615728+0000","last_became_active":"2024-06-29T15:59:42.616516+0000","last_became_peered":"2024-06-29T15:59:42.616516+0000","last_unstale":"2024-06-29T15:59:43.615728+0000","last_undegraded":"2024-06-29T15:59:43.615728+0000","last_fullsized":"2024-06-29T15:59:43.615728+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:35.661438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"265'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612663+0000","last_change":"2024-06-29T15:59:41.613491+0000","last_active":"2024-06-29T15:59:43.612663+0000","last_peered":"2024-06-29T15:59:43.612663+0000","last_clean":"2024-06-29T15:59:43.612663+0000","last_became_active":"2024-06-29T15:59:41.613120+0000","last_became_peered":"2024-06-29T15:59:41.613120+0000","last_unstale":"2024-06-29T15:59:43.612663+0000","last_undegraded":"2024-06-29T15:59:43.612663+0000","last_fullsized":"2024-06-29T15:59:43.612663+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:22:31.544314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981287+0000","last_change":"2024-06-29T15:59:42.624269+0000","last_active":"2024-06-29T15:59:43.981287+0000","last_peered":"2024-06-29T15:59:43.981287+0000","last_clean":"2024-06-29T15:59:43.981287+0000","last_became_active":"2024-06-29T15:59:42.623962+0000","last_became_peered":"2024-06-29T15:59:42.623962+0000","last_unstale":"2024-06-29T15:59:43.981287+0000","last_undegraded":"2024-06-29T15:59:43.981287+0000","last_fullsized":"2024-06-29T15:59:43.981287+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:47:44.387666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"265'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982309+0000","last_change":"2024-06-29T15:59:41.618251+0000","last_active":"2024-06-29T15:59:43.982309+0000","last_peered":"2024-06-29T15:59:43.982309+0000","last_clean":"2024-06-29T15:59:43.982309+0000","last_became_active":"2024-06-29T15:59:41.617810+0000","last_became_peered":"2024-06-29T15:59:41.617810+0000","last_unstale":"2024-06-29T15:59:43.982309+0000","last_undegraded":"2024-06-29T15:59:43.982309+0000","last_fullsized":"2024-06-29T15:59:43.982309+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:28:13.722209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981538+0000","last_change":"2024-06-29T15:59:42.618704+0000","last_active":"2024-06-29T15:59:43.981538+0000","last_peered":"2024-06-29T15:59:43.981538+0000","last_clean":"2024-06-29T15:59:43.981538+0000","last_became_active":"2024-06-29T15:59:42.618396+0000","last_became_peered":"2024-06-29T15:59:42.618396+0000","last_unstale":"2024-06-29T15:59:43.981538+0000","last_undegraded":"2024-06-29T15:59:43.981538+0000","last_fullsized":"2024-06-29T15:59:43.981538+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:22:48.050983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612429+0000","last_change":"2024-06-29T15:59:41.619230+0000","last_active":"2024-06-29T15:59:43.612429+0000","last_peered":"2024-06-29T15:59:43.612429+0000","last_clean":"2024-06-29T15:59:43.612429+0000","last_became_active":"2024-06-29T15:59:41.618924+0000","last_became_peered":"2024-06-29T15:59:41.618924+0000","last_unstale":"2024-06-29T15:59:43.612429+0000","last_undegraded":"2024-06-29T15:59:43.612429+0000","last_fullsized":"2024-06-29T15:59:43.612429+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:19:47.966211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612799+0000","last_change":"2024-06-29T15:59:42.618412+0000","last_active":"2024-06-29T15:59:43.612799+0000","last_peered":"2024-06-29T15:59:43.612799+0000","last_clean":"2024-06-29T15:59:43.612799+0000","last_became_active":"2024-06-29T15:59:42.618095+0000","last_became_peered":"2024-06-29T15:59:42.618095+0000","last_unstale":"2024-06-29T15:59:43.612799+0000","last_undegraded":"2024-06-29T15:59:43.612799+0000","last_fullsized":"2024-06-29T15:59:43.612799+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:53:16.228824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612331+0000","last_change":"2024-06-29T15:59:41.620760+0000","last_active":"2024-06-29T15:59:43.612331+0000","last_peered":"2024-06-29T15:59:43.612331+0000","last_clean":"2024-06-29T15:59:43.612331+0000","last_became_active":"2024-06-29T15:59:41.620396+0000","last_became_peered":"2024-06-29T15:59:41.620396+0000","last_unstale":"2024-06-29T15:59:43.612331+0000","last_undegraded":"2024-06-29T15:59:43.612331+0000","last_fullsized":"2024-06-29T15:59:43.612331+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:20:43.427382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613341+0000","last_change":"2024-06-29T15:59:42.610163+0000","last_active":"2024-06-29T15:59:43.613341+0000","last_peered":"2024-06-29T15:59:43.613341+0000","last_clean":"2024-06-29T15:59:43.613341+0000","last_became_active":"2024-06-29T15:59:42.609815+0000","last_became_peered":"2024-06-29T15:59:42.609815+0000","last_unstale":"2024-06-29T15:59:43.613341+0000","last_undegraded":"2024-06-29T15:59:43.613341+0000","last_fullsized":"2024-06-29T15:59:43.613341+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:16.114463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982931+0000","last_change":"2024-06-29T15:59:41.617815+0000","last_active":"2024-06-29T15:59:43.982931+0000","last_peered":"2024-06-29T15:59:43.982931+0000","last_clean":"2024-06-29T15:59:43.982931+0000","last_became_active":"2024-06-29T15:59:41.617457+0000","last_became_peered":"2024-06-29T15:59:41.617457+0000","last_unstale":"2024-06-29T15:59:43.982931+0000","last_undegraded":"2024-06-29T15:59:43.982931+0000","last_fullsized":"2024-06-29T15:59:43.982931+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:29:05.505946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982668+0000","last_change":"2024-06-29T15:59:42.614736+0000","last_active":"2024-06-29T15:59:43.982668+0000","last_peered":"2024-06-29T15:59:43.982668+0000","last_clean":"2024-06-29T15:59:43.982668+0000","last_became_active":"2024-06-29T15:59:42.614418+0000","last_became_peered":"2024-06-29T15:59:42.614418+0000","last_unstale":"2024-06-29T15:59:43.982668+0000","last_undegraded":"2024-06-29T15:59:43.982668+0000","last_fullsized":"2024-06-29T15:59:43.982668+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:32:18.842042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982165+0000","last_change":"2024-06-29T15:59:41.616859+0000","last_active":"2024-06-29T15:59:43.982165+0000","last_peered":"2024-06-29T15:59:43.982165+0000","last_clean":"2024-06-29T15:59:43.982165+0000","last_became_active":"2024-06-29T15:59:41.616288+0000","last_became_peered":"2024-06-29T15:59:41.616288+0000","last_unstale":"2024-06-29T15:59:43.982165+0000","last_undegraded":"2024-06-29T15:59:43.982165+0000","last_fullsized":"2024-06-29T15:59:43.982165+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:36:16.977926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613478+0000","last_change":"2024-06-29T15:59:42.612221+0000","last_active":"2024-06-29T15:59:43.613478+0000","last_peered":"2024-06-29T15:59:43.613478+0000","last_clean":"2024-06-29T15:59:43.613478+0000","last_became_active":"2024-06-29T15:59:42.611931+0000","last_became_peered":"2024-06-29T15:59:42.611931+0000","last_unstale":"2024-06-29T15:59:43.613478+0000","last_undegraded":"2024-06-29T15:59:43.613478+0000","last_fullsized":"2024-06-29T15:59:43.613478+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:25:16.114463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"265'3","reported_seq":17,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612457+0000","last_change":"2024-06-29T15:59:41.611119+0000","last_active":"2024-06-29T15:59:43.612457+0000","last_peered":"2024-06-29T15:59:43.612457+0000","last_clean":"2024-06-29T15:59:43.612457+0000","last_became_active":"2024-06-29T15:59:41.610247+0000","last_became_peered":"2024-06-29T15:59:41.610247+0000","last_unstale":"2024-06-29T15:59:43.612457+0000","last_undegraded":"2024-06-29T15:59:43.612457+0000","last_fullsized":"2024-06-29T15:59:43.612457+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:18:35.775372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613431+0000","last_change":"2024-06-29T15:59:42.622109+0000","last_active":"2024-06-29T15:59:43.613431+0000","last_peered":"2024-06-29T15:59:43.613431+0000","last_clean":"2024-06-29T15:59:43.613431+0000","last_became_active":"2024-06-29T15:59:42.621768+0000","last_became_peered":"2024-06-29T15:59:42.621768+0000","last_unstale":"2024-06-29T15:59:43.613431+0000","last_undegraded":"2024-06-29T15:59:43.613431+0000","last_fullsized":"2024-06-29T15:59:43.613431+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:40:19.191373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"265'6","reported_seq":20,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982740+0000","last_change":"2024-06-29T15:59:41.615448+0000","last_active":"2024-06-29T15:59:43.982740+0000","last_peered":"2024-06-29T15:59:43.982740+0000","last_clean":"2024-06-29T15:59:43.982740+0000","last_became_active":"2024-06-29T15:59:41.615130+0000","last_became_peered":"2024-06-29T15:59:41.615130+0000","last_unstale":"2024-06-29T15:59:43.982740+0000","last_undegraded":"2024-06-29T15:59:43.982740+0000","last_fullsized":"2024-06-29T15:59:43.982740+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:01:34.656028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982252+0000","last_change":"2024-06-29T15:59:42.615550+0000","last_active":"2024-06-29T15:59:43.982252+0000","last_peered":"2024-06-29T15:59:43.982252+0000","last_clean":"2024-06-29T15:59:43.982252+0000","last_became_active":"2024-06-29T15:59:42.615247+0000","last_became_peered":"2024-06-29T15:59:42.615247+0000","last_unstale":"2024-06-29T15:59:43.982252+0000","last_undegraded":"2024-06-29T15:59:43.982252+0000","last_fullsized":"2024-06-29T15:59:43.982252+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:07:53.277871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615861+0000","last_change":"2024-06-29T15:59:41.604732+0000","last_active":"2024-06-29T15:59:43.615861+0000","last_peered":"2024-06-29T15:59:43.615861+0000","last_clean":"2024-06-29T15:59:43.615861+0000","last_became_active":"2024-06-29T15:59:41.604327+0000","last_became_peered":"2024-06-29T15:59:41.604327+0000","last_unstale":"2024-06-29T15:59:43.615861+0000","last_undegraded":"2024-06-29T15:59:43.615861+0000","last_fullsized":"2024-06-29T15:59:43.615861+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:41.503393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981714+0000","last_change":"2024-06-29T15:59:42.620145+0000","last_active":"2024-06-29T15:59:43.981714+0000","last_peered":"2024-06-29T15:59:43.981714+0000","last_clean":"2024-06-29T15:59:43.981714+0000","last_became_active":"2024-06-29T15:59:42.619763+0000","last_became_peered":"2024-06-29T15:59:42.619763+0000","last_unstale":"2024-06-29T15:59:43.981714+0000","last_undegraded":"2024-06-29T15:59:43.981714+0000","last_fullsized":"2024-06-29T15:59:43.981714+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:11:45.103664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612206+0000","last_change":"2024-06-29T15:59:41.608219+0000","last_active":"2024-06-29T15:59:43.612206+0000","last_peered":"2024-06-29T15:59:43.612206+0000","last_clean":"2024-06-29T15:59:43.612206+0000","last_became_active":"2024-06-29T15:59:41.607873+0000","last_became_peered":"2024-06-29T15:59:41.607873+0000","last_unstale":"2024-06-29T15:59:43.612206+0000","last_undegraded":"2024-06-29T15:59:43.612206+0000","last_fullsized":"2024-06-29T15:59:43.612206+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:37:29.669256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612468+0000","last_change":"2024-06-29T15:59:42.616037+0000","last_active":"2024-06-29T15:59:43.612468+0000","last_peered":"2024-06-29T15:59:43.612468+0000","last_clean":"2024-06-29T15:59:43.612468+0000","last_became_active":"2024-06-29T15:59:42.615704+0000","last_became_peered":"2024-06-29T15:59:42.615704+0000","last_unstale":"2024-06-29T15:59:43.612468+0000","last_undegraded":"2024-06-29T15:59:43.612468+0000","last_fullsized":"2024-06-29T15:59:43.612468+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:36:16.616337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982996+0000","last_change":"2024-06-29T15:59:41.619225+0000","last_active":"2024-06-29T15:59:43.982996+0000","last_peered":"2024-06-29T15:59:43.982996+0000","last_clean":"2024-06-29T15:59:43.982996+0000","last_became_active":"2024-06-29T15:59:41.618792+0000","last_became_peered":"2024-06-29T15:59:41.618792+0000","last_unstale":"2024-06-29T15:59:43.982996+0000","last_undegraded":"2024-06-29T15:59:43.982996+0000","last_fullsized":"2024-06-29T15:59:43.982996+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:38:34.554911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982371+0000","last_change":"2024-06-29T15:59:42.610628+0000","last_active":"2024-06-29T15:59:43.982371+0000","last_peered":"2024-06-29T15:59:43.982371+0000","last_clean":"2024-06-29T15:59:43.982371+0000","last_became_active":"2024-06-29T15:59:42.609965+0000","last_became_peered":"2024-06-29T15:59:42.609965+0000","last_unstale":"2024-06-29T15:59:43.982371+0000","last_undegraded":"2024-06-29T15:59:43.982371+0000","last_fullsized":"2024-06-29T15:59:43.982371+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:23:33.730877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"265'8","reported_seq":22,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612069+0000","last_change":"2024-06-29T15:59:41.619781+0000","last_active":"2024-06-29T15:59:43.612069+0000","last_peered":"2024-06-29T15:59:43.612069+0000","last_clean":"2024-06-29T15:59:43.612069+0000","last_became_active":"2024-06-29T15:59:41.619353+0000","last_became_peered":"2024-06-29T15:59:41.619353+0000","last_unstale":"2024-06-29T15:59:43.612069+0000","last_undegraded":"2024-06-29T15:59:43.612069+0000","last_fullsized":"2024-06-29T15:59:43.612069+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:29:05.505946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612338+0000","last_change":"2024-06-29T15:59:42.627910+0000","last_active":"2024-06-29T15:59:43.612338+0000","last_peered":"2024-06-29T15:59:43.612338+0000","last_clean":"2024-06-29T15:59:43.612338+0000","last_became_active":"2024-06-29T15:59:42.627587+0000","last_became_peered":"2024-06-29T15:59:42.627587+0000","last_unstale":"2024-06-29T15:59:43.612338+0000","last_undegraded":"2024-06-29T15:59:43.612338+0000","last_fullsized":"2024-06-29T15:59:43.612338+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:55.120696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"265'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615822+0000","last_change":"2024-06-29T15:59:41.613245+0000","last_active":"2024-06-29T15:59:43.615822+0000","last_peered":"2024-06-29T15:59:43.615822+0000","last_clean":"2024-06-29T15:59:43.615822+0000","last_became_active":"2024-06-29T15:59:41.612506+0000","last_became_peered":"2024-06-29T15:59:41.612506+0000","last_unstale":"2024-06-29T15:59:43.615822+0000","last_undegraded":"2024-06-29T15:59:43.615822+0000","last_fullsized":"2024-06-29T15:59:43.615822+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:08:37.770855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613267+0000","last_change":"2024-06-29T15:59:42.611957+0000","last_active":"2024-06-29T15:59:43.613267+0000","last_peered":"2024-06-29T15:59:43.613267+0000","last_clean":"2024-06-29T15:59:43.613267+0000","last_became_active":"2024-06-29T15:59:42.611626+0000","last_became_peered":"2024-06-29T15:59:42.611626+0000","last_unstale":"2024-06-29T15:59:43.613267+0000","last_undegraded":"2024-06-29T15:59:43.613267+0000","last_fullsized":"2024-06-29T15:59:43.613267+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:56:32.009117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982429+0000","last_change":"2024-06-29T15:59:41.616596+0000","last_active":"2024-06-29T15:59:43.982429+0000","last_peered":"2024-06-29T15:59:43.982429+0000","last_clean":"2024-06-29T15:59:43.982429+0000","last_became_active":"2024-06-29T15:59:41.615847+0000","last_became_peered":"2024-06-29T15:59:41.615847+0000","last_unstale":"2024-06-29T15:59:43.982429+0000","last_undegraded":"2024-06-29T15:59:43.982429+0000","last_fullsized":"2024-06-29T15:59:43.982429+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:38.340808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613076+0000","last_change":"2024-06-29T15:59:42.619083+0000","last_active":"2024-06-29T15:59:43.613076+0000","last_peered":"2024-06-29T15:59:43.613076+0000","last_clean":"2024-06-29T15:59:43.613076+0000","last_became_active":"2024-06-29T15:59:42.618781+0000","last_became_peered":"2024-06-29T15:59:42.618781+0000","last_unstale":"2024-06-29T15:59:43.613076+0000","last_undegraded":"2024-06-29T15:59:43.613076+0000","last_fullsized":"2024-06-29T15:59:43.613076+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:03:42.796201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"265'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613668+0000","last_change":"2024-06-29T15:59:41.614722+0000","last_active":"2024-06-29T15:59:43.613668+0000","last_peered":"2024-06-29T15:59:43.613668+0000","last_clean":"2024-06-29T15:59:43.613668+0000","last_became_active":"2024-06-29T15:59:41.614311+0000","last_became_peered":"2024-06-29T15:59:41.614311+0000","last_unstale":"2024-06-29T15:59:43.613668+0000","last_undegraded":"2024-06-29T15:59:43.613668+0000","last_fullsized":"2024-06-29T15:59:43.613668+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:42.544078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982629+0000","last_change":"2024-06-29T15:59:42.614651+0000","last_active":"2024-06-29T15:59:43.982629+0000","last_peered":"2024-06-29T15:59:43.982629+0000","last_clean":"2024-06-29T15:59:43.982629+0000","last_became_active":"2024-06-29T15:59:42.614321+0000","last_became_peered":"2024-06-29T15:59:42.614321+0000","last_unstale":"2024-06-29T15:59:43.982629+0000","last_undegraded":"2024-06-29T15:59:43.982629+0000","last_fullsized":"2024-06-29T15:59:43.982629+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T01:44:38.993325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613268+0000","last_change":"2024-06-29T15:59:41.620179+0000","last_active":"2024-06-29T15:59:43.613268+0000","last_peered":"2024-06-29T15:59:43.613268+0000","last_clean":"2024-06-29T15:59:43.613268+0000","last_became_active":"2024-06-29T15:59:41.619869+0000","last_became_peered":"2024-06-29T15:59:41.619869+0000","last_unstale":"2024-06-29T15:59:43.613268+0000","last_undegraded":"2024-06-29T15:59:43.613268+0000","last_fullsized":"2024-06-29T15:59:43.613268+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:01:22.841090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612665+0000","last_change":"2024-06-29T15:59:42.620802+0000","last_active":"2024-06-29T15:59:43.612665+0000","last_peered":"2024-06-29T15:59:43.612665+0000","last_clean":"2024-06-29T15:59:43.612665+0000","last_became_active":"2024-06-29T15:59:42.620457+0000","last_became_peered":"2024-06-29T15:59:42.620457+0000","last_unstale":"2024-06-29T15:59:43.612665+0000","last_undegraded":"2024-06-29T15:59:43.612665+0000","last_fullsized":"2024-06-29T15:59:43.612665+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:14:14.891916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612714+0000","last_change":"2024-06-29T15:59:41.606766+0000","last_active":"2024-06-29T15:59:43.612714+0000","last_peered":"2024-06-29T15:59:43.612714+0000","last_clean":"2024-06-29T15:59:43.612714+0000","last_became_active":"2024-06-29T15:59:41.606117+0000","last_became_peered":"2024-06-29T15:59:41.606117+0000","last_unstale":"2024-06-29T15:59:43.612714+0000","last_undegraded":"2024-06-29T15:59:43.612714+0000","last_fullsized":"2024-06-29T15:59:43.612714+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:47:40.013941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981336+0000","last_change":"2024-06-29T15:59:42.618484+0000","last_active":"2024-06-29T15:59:43.981336+0000","last_peered":"2024-06-29T15:59:43.981336+0000","last_clean":"2024-06-29T15:59:43.981336+0000","last_became_active":"2024-06-29T15:59:42.618179+0000","last_became_peered":"2024-06-29T15:59:42.618179+0000","last_unstale":"2024-06-29T15:59:43.981336+0000","last_undegraded":"2024-06-29T15:59:43.981336+0000","last_fullsized":"2024-06-29T15:59:43.981336+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:27:06.862146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615725+0000","last_change":"2024-06-29T15:59:41.612784+0000","last_active":"2024-06-29T15:59:43.615725+0000","last_peered":"2024-06-29T15:59:43.615725+0000","last_clean":"2024-06-29T15:59:43.615725+0000","last_became_active":"2024-06-29T15:59:41.612447+0000","last_became_peered":"2024-06-29T15:59:41.612447+0000","last_unstale":"2024-06-29T15:59:43.615725+0000","last_undegraded":"2024-06-29T15:59:43.615725+0000","last_fullsized":"2024-06-29T15:59:43.615725+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:56:09.979323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615640+0000","last_change":"2024-06-29T15:59:42.614715+0000","last_active":"2024-06-29T15:59:43.615640+0000","last_peered":"2024-06-29T15:59:43.615640+0000","last_clean":"2024-06-29T15:59:43.615640+0000","last_became_active":"2024-06-29T15:59:42.614391+0000","last_became_peered":"2024-06-29T15:59:42.614391+0000","last_unstale":"2024-06-29T15:59:43.615640+0000","last_undegraded":"2024-06-29T15:59:43.615640+0000","last_fullsized":"2024-06-29T15:59:43.615640+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:58:55.910938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612922+0000","last_change":"2024-06-29T15:59:41.620207+0000","last_active":"2024-06-29T15:59:43.612922+0000","last_peered":"2024-06-29T15:59:43.612922+0000","last_clean":"2024-06-29T15:59:43.612922+0000","last_became_active":"2024-06-29T15:59:41.619899+0000","last_became_peered":"2024-06-29T15:59:41.619899+0000","last_unstale":"2024-06-29T15:59:43.612922+0000","last_undegraded":"2024-06-29T15:59:43.612922+0000","last_fullsized":"2024-06-29T15:59:43.612922+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:41:44.467762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615606+0000","last_change":"2024-06-29T15:59:42.628488+0000","last_active":"2024-06-29T15:59:43.615606+0000","last_peered":"2024-06-29T15:59:43.615606+0000","last_clean":"2024-06-29T15:59:43.615606+0000","last_became_active":"2024-06-29T15:59:42.628198+0000","last_became_peered":"2024-06-29T15:59:42.628198+0000","last_unstale":"2024-06-29T15:59:43.615606+0000","last_undegraded":"2024-06-29T15:59:43.615606+0000","last_fullsized":"2024-06-29T15:59:43.615606+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:13:36.230272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615751+0000","last_change":"2024-06-29T15:59:41.604728+0000","last_active":"2024-06-29T15:59:43.615751+0000","last_peered":"2024-06-29T15:59:43.615751+0000","last_clean":"2024-06-29T15:59:43.615751+0000","last_became_active":"2024-06-29T15:59:41.604180+0000","last_became_peered":"2024-06-29T15:59:41.604180+0000","last_unstale":"2024-06-29T15:59:43.615751+0000","last_undegraded":"2024-06-29T15:59:43.615751+0000","last_fullsized":"2024-06-29T15:59:43.615751+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:59:54.115285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.981447+0000","last_change":"2024-06-29T15:59:42.617394+0000","last_active":"2024-06-29T15:59:43.981447+0000","last_peered":"2024-06-29T15:59:43.981447+0000","last_clean":"2024-06-29T15:59:43.981447+0000","last_became_active":"2024-06-29T15:59:42.616330+0000","last_became_peered":"2024-06-29T15:59:42.616330+0000","last_unstale":"2024-06-29T15:59:43.981447+0000","last_undegraded":"2024-06-29T15:59:43.981447+0000","last_fullsized":"2024-06-29T15:59:43.981447+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:20:34.350599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"265'2","reported_seq":16,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612656+0000","last_change":"2024-06-29T15:59:41.614034+0000","last_active":"2024-06-29T15:59:43.612656+0000","last_peered":"2024-06-29T15:59:43.612656+0000","last_clean":"2024-06-29T15:59:43.612656+0000","last_became_active":"2024-06-29T15:59:41.613732+0000","last_became_peered":"2024-06-29T15:59:41.613732+0000","last_unstale":"2024-06-29T15:59:43.612656+0000","last_undegraded":"2024-06-29T15:59:43.612656+0000","last_fullsized":"2024-06-29T15:59:43.612656+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:39:20.232138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982965+0000","last_change":"2024-06-29T15:59:42.615766+0000","last_active":"2024-06-29T15:59:43.982965+0000","last_peered":"2024-06-29T15:59:43.982965+0000","last_clean":"2024-06-29T15:59:43.982965+0000","last_became_active":"2024-06-29T15:59:42.613917+0000","last_became_peered":"2024-06-29T15:59:42.613917+0000","last_unstale":"2024-06-29T15:59:43.982965+0000","last_undegraded":"2024-06-29T15:59:43.982965+0000","last_fullsized":"2024-06-29T15:59:43.982965+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:33:11.115404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"266'1","reported_seq":15,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.617472+0000","last_change":"2024-06-29T15:59:41.606064+0000","last_active":"2024-06-29T15:59:43.617472+0000","last_peered":"2024-06-29T15:59:43.617472+0000","last_clean":"2024-06-29T15:59:43.617472+0000","last_became_active":"2024-06-29T15:59:41.605736+0000","last_became_peered":"2024-06-29T15:59:41.605736+0000","last_unstale":"2024-06-29T15:59:43.617472+0000","last_undegraded":"2024-06-29T15:59:43.617472+0000","last_fullsized":"2024-06-29T15:59:43.617472+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T17:57:17.351019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982122+0000","last_change":"2024-06-29T15:59:42.604207+0000","last_active":"2024-06-29T15:59:43.982122+0000","last_peered":"2024-06-29T15:59:43.982122+0000","last_clean":"2024-06-29T15:59:43.982122+0000","last_became_active":"2024-06-29T15:59:42.603868+0000","last_became_peered":"2024-06-29T15:59:42.603868+0000","last_unstale":"2024-06-29T15:59:43.982122+0000","last_undegraded":"2024-06-29T15:59:43.982122+0000","last_fullsized":"2024-06-29T15:59:43.982122+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T19:52:43.549488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612557+0000","last_change":"2024-06-29T15:59:41.610960+0000","last_active":"2024-06-29T15:59:43.612557+0000","last_peered":"2024-06-29T15:59:43.612557+0000","last_clean":"2024-06-29T15:59:43.612557+0000","last_became_active":"2024-06-29T15:59:41.610665+0000","last_became_peered":"2024-06-29T15:59:41.610665+0000","last_unstale":"2024-06-29T15:59:43.612557+0000","last_undegraded":"2024-06-29T15:59:43.612557+0000","last_fullsized":"2024-06-29T15:59:43.612557+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:46:33.782504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.983265+0000","last_change":"2024-06-29T15:59:42.609228+0000","last_active":"2024-06-29T15:59:43.983265+0000","last_peered":"2024-06-29T15:59:43.983265+0000","last_clean":"2024-06-29T15:59:43.983265+0000","last_became_active":"2024-06-29T15:59:42.608875+0000","last_became_peered":"2024-06-29T15:59:42.608875+0000","last_unstale":"2024-06-29T15:59:43.983265+0000","last_undegraded":"2024-06-29T15:59:43.983265+0000","last_fullsized":"2024-06-29T15:59:43.983265+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:56:46.832275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612879+0000","last_change":"2024-06-29T15:59:41.617687+0000","last_active":"2024-06-29T15:59:43.612879+0000","last_peered":"2024-06-29T15:59:43.612879+0000","last_clean":"2024-06-29T15:59:43.612879+0000","last_became_active":"2024-06-29T15:59:41.616690+0000","last_became_peered":"2024-06-29T15:59:41.616690+0000","last_unstale":"2024-06-29T15:59:43.612879+0000","last_undegraded":"2024-06-29T15:59:43.612879+0000","last_fullsized":"2024-06-29T15:59:43.612879+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:50:25.372841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613597+0000","last_change":"2024-06-29T15:59:42.620784+0000","last_active":"2024-06-29T15:59:43.613597+0000","last_peered":"2024-06-29T15:59:43.613597+0000","last_clean":"2024-06-29T15:59:43.613597+0000","last_became_active":"2024-06-29T15:59:42.620497+0000","last_became_peered":"2024-06-29T15:59:42.620497+0000","last_unstale":"2024-06-29T15:59:43.613597+0000","last_undegraded":"2024-06-29T15:59:43.613597+0000","last_fullsized":"2024-06-29T15:59:43.613597+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T02:46:31.080213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612951+0000","last_change":"2024-06-29T15:59:41.615402+0000","last_active":"2024-06-29T15:59:43.612951+0000","last_peered":"2024-06-29T15:59:43.612951+0000","last_clean":"2024-06-29T15:59:43.612951+0000","last_became_active":"2024-06-29T15:59:41.615099+0000","last_became_peered":"2024-06-29T15:59:41.615099+0000","last_unstale":"2024-06-29T15:59:43.612951+0000","last_undegraded":"2024-06-29T15:59:43.612951+0000","last_fullsized":"2024-06-29T15:59:43.612951+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:33:10.109993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612508+0000","last_change":"2024-06-29T15:59:42.614919+0000","last_active":"2024-06-29T15:59:43.612508+0000","last_peered":"2024-06-29T15:59:43.612508+0000","last_clean":"2024-06-29T15:59:43.612508+0000","last_became_active":"2024-06-29T15:59:42.614567+0000","last_became_peered":"2024-06-29T15:59:42.614567+0000","last_unstale":"2024-06-29T15:59:43.612508+0000","last_undegraded":"2024-06-29T15:59:43.612508+0000","last_fullsized":"2024-06-29T15:59:43.612508+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T18:53:16.228824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612315+0000","last_change":"2024-06-29T15:59:41.619869+0000","last_active":"2024-06-29T15:59:43.612315+0000","last_peered":"2024-06-29T15:59:43.612315+0000","last_clean":"2024-06-29T15:59:43.612315+0000","last_became_active":"2024-06-29T15:59:41.619376+0000","last_became_peered":"2024-06-29T15:59:41.619376+0000","last_unstale":"2024-06-29T15:59:43.612315+0000","last_undegraded":"2024-06-29T15:59:43.612315+0000","last_fullsized":"2024-06-29T15:59:43.612315+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:51:24.300817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.615718+0000","last_change":"2024-06-29T15:59:42.624606+0000","last_active":"2024-06-29T15:59:43.615718+0000","last_peered":"2024-06-29T15:59:43.615718+0000","last_clean":"2024-06-29T15:59:43.615718+0000","last_became_active":"2024-06-29T15:59:42.624300+0000","last_became_peered":"2024-06-29T15:59:42.624300+0000","last_unstale":"2024-06-29T15:59:43.615718+0000","last_undegraded":"2024-06-29T15:59:43.615718+0000","last_fullsized":"2024-06-29T15:59:43.615718+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T20:25:07.998005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982240+0000","last_change":"2024-06-29T15:59:41.613044+0000","last_active":"2024-06-29T15:59:43.982240+0000","last_peered":"2024-06-29T15:59:43.982240+0000","last_clean":"2024-06-29T15:59:43.982240+0000","last_became_active":"2024-06-29T15:59:41.612373+0000","last_became_peered":"2024-06-29T15:59:41.612373+0000","last_unstale":"2024-06-29T15:59:43.982240+0000","last_undegraded":"2024-06-29T15:59:43.982240+0000","last_fullsized":"2024-06-29T15:59:43.982240+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T21:10:57.112983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.982552+0000","last_change":"2024-06-29T15:59:42.610269+0000","last_active":"2024-06-29T15:59:43.982552+0000","last_peered":"2024-06-29T15:59:43.982552+0000","last_clean":"2024-06-29T15:59:43.982552+0000","last_became_active":"2024-06-29T15:59:42.609558+0000","last_became_peered":"2024-06-29T15:59:42.609558+0000","last_unstale":"2024-06-29T15:59:43.982552+0000","last_undegraded":"2024-06-29T15:59:43.982552+0000","last_fullsized":"2024-06-29T15:59:43.982552+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T16:50:57.932595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":14,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612041+0000","last_change":"2024-06-29T15:59:41.619398+0000","last_active":"2024-06-29T15:59:43.612041+0000","last_peered":"2024-06-29T15:59:43.612041+0000","last_clean":"2024-06-29T15:59:43.612041+0000","last_became_active":"2024-06-29T15:59:41.619090+0000","last_became_peered":"2024-06-29T15:59:41.619090+0000","last_unstale":"2024-06-29T15:59:43.612041+0000","last_undegraded":"2024-06-29T15:59:43.612041+0000","last_fullsized":"2024-06-29T15:59:43.612041+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":263,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:40.582754+0000","last_clean_scrub_stamp":"2024-06-29T15:59:40.582754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T23:20:43.427382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":12,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.613305+0000","last_change":"2024-06-29T15:59:42.613952+0000","last_active":"2024-06-29T15:59:43.613305+0000","last_peered":"2024-06-29T15:59:43.613305+0000","last_clean":"2024-06-29T15:59:43.613305+0000","last_became_active":"2024-06-29T15:59:42.612934+0000","last_became_peered":"2024-06-29T15:59:42.612934+0000","last_unstale":"2024-06-29T15:59:43.613305+0000","last_undegraded":"2024-06-29T15:59:43.613305+0000","last_fullsized":"2024-06-29T15:59:43.613305+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":264,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T15:59:41.588165+0000","last_clean_scrub_stamp":"2024-06-29T15:59:41.588165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T00:04:32.759053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"255'261","reported_seq":1291,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-06-29T15:59:43.612548+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T15:59:43.612548+0000","last_peered":"2024-06-29T15:59:43.612548+0000","last_clean":"2024-06-29T15:59:43.612548+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T15:59:43.612548+0000","last_undegraded":"2024-06-29T15:59:43.612548+0000","last_fullsized":"2024-06-29T15:59:43.612548+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1686,"num_read_kb":4807,"num_write":425,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1686,"num_read_kb":4807,"num_write":425,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261,"ondisk_log_size":261,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576011,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53096,"kb_used_data":3868,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318744,"statfs":{"total":96636764160,"available":96582393856,"internally_reserved":0,"allocated":3960832,"data_stored":2254183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834576012,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64996,"kb_used_data":3288,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306844,"statfs":{"total":96636764160,"available":96570208256,"internally_reserved":0,"allocated":3366912,"data_stored":1658999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834576010,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48376,"kb_used_data":3304,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323464,"statfs":{"total":96636764160,"available":96587227136,"internally_reserved":0,"allocated":3383296,"data_stored":1665517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834576011,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82992,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94288848,"statfs":{"total":96636764160,"available":96551780352,"internally_reserved":0,"allocated":3375104,"data_stored":1664467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834576011,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48352,"kb_used_data":3280,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323488,"statfs":{"total":96636764160,"available":96587251712,"internally_reserved":0,"allocated":3358720,"data_stored":1657969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52528,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319312,"statfs":{"total":96636764160,"available":96582975488,"internally_reserved":0,"allocated":3375104,"data_stored":1663939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834576010,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48368,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323472,"statfs":{"total":96636764160,"available":96587235328,"internally_reserved":0,"allocated":3375104,"data_stored":1663871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834576011,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61424,"kb_used_data":3876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310416,"statfs":{"total":96636764160,"available":96573865984,"internally_reserved":0,"allocated":3969024,"data_stored":2254275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T15:59:47.003 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-06-29T15:59:47.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-06-29T15:59:47.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.159+0000 7f82b5711640 1 -- 172.21.15.81:0/1164408414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f829c003a50 msgr2=0x7f829c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.159+0000 7f82b5711640 1 --2- 172.21.15.81:0/1164408414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f829c003a50 0x7f829c003ec0 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009b50 tx=0x7f829c005830 comp rx=0 tx=0).stop 2024-06-29T15:59:47.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.159+0000 7f82b5711640 1 -- 172.21.15.81:0/1164408414 shutdown_connections 2024-06-29T15:59:47.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.159+0000 7f82b5711640 1 --2- 172.21.15.81:0/1164408414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f829c003a50 0x7f829c003ec0 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.159+0000 7f82b5711640 1 -- 172.21.15.81:0/1164408414 >> 172.21.15.81:0/1164408414 conn(0x7f82b00b56d0 msgr2=0x7f82b00b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:47.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.160+0000 7f82b5711640 1 -- 172.21.15.81:0/1164408414 shutdown_connections 2024-06-29T15:59:47.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.160+0000 7f82b5711640 1 -- 172.21.15.81:0/1164408414 wait complete. 2024-06-29T15:59:47.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.161+0000 7f82b5711640 1 Processor -- start 2024-06-29T15:59:47.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.161+0000 7f82b5711640 1 -- start start 2024-06-29T15:59:47.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82b5711640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f829c003a50 0x7f82b00cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82b5711640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82b01e93b0 0x7f82b009ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82b5711640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82b00a0880 0x7f82b00a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82b5711640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82b00ceab0 con 0x7f829c003a50 2024-06-29T15:59:47.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82b5711640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f82b00ce930 con 0x7f82b00a0880 2024-06-29T15:59:47.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82b5711640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f82b00cec30 con 0x7f82b01e93b0 2024-06-29T15:59:47.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82aeffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f829c003a50 0x7f82b00cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:47.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82aeffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f829c003a50 0x7f82b00cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48372/0 (socket says 172.21.15.81:48372) 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82aeffd640 1 -- 172.21.15.81:0/2681354809 learned_addr learned my addr 172.21.15.81:0/2681354809 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82aeffd640 1 -- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82b01e93b0 msgr2=0x7f82b009ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82aeffd640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82b01e93b0 0x7f82b009ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82aeffd640 1 -- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82b00a0880 msgr2=0x7f82b00a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82aeffd640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82b00a0880 0x7f82b00a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82aeffd640 1 -- 172.21.15.81:0/2681354809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b01ea0e0 con 0x7f829c003a50 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82aeffd640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f829c003a50 0x7f82b00cd100 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009a70 tx=0x7f82a00336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82a7fff640 1 -- 172.21.15.81:0/2681354809 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a0038040 con 0x7f829c003a50 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82a7fff640 1 -- 172.21.15.81:0/2681354809 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82a00339d0 con 0x7f829c003a50 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82b5711640 1 -- 172.21.15.81:0/2681354809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b01e9b20 con 0x7f829c003a50 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.163+0000 7f82a7fff640 1 -- 172.21.15.81:0/2681354809 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a0033e20 con 0x7f829c003a50 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.164+0000 7f82b5711640 1 -- 172.21.15.81:0/2681354809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b01e6650 con 0x7f829c003a50 2024-06-29T15:59:47.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.165+0000 7f82b5711640 1 -- 172.21.15.81:0/2681354809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82b00cf190 con 0x7f829c003a50 2024-06-29T15:59:47.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.165+0000 7f82a7fff640 1 -- 172.21.15.81:0/2681354809 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f82a0043020 con 0x7f829c003a50 2024-06-29T15:59:47.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.166+0000 7f82a7fff640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8290076420 0x7f8290078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.166+0000 7f82a7fff640 1 -- 172.21.15.81:0/2681354809 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f82a00b7c20 con 0x7f829c003a50 2024-06-29T15:59:47.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.166+0000 7f82a67fc640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8290076420 0x7f8290078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:47.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.167+0000 7f82a67fc640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8290076420 0x7f8290078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f82b006d410 tx=0x7f8298006aa0 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:47.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.170+0000 7f82a7fff640 1 -- 172.21.15.81:0/2681354809 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f82a004a020 con 0x7f829c003a50 2024-06-29T15:59:47.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.423+0000 7f82b5711640 1 -- 172.21.15.81:0/2681354809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f82b01ea8b0 con 0x7f829c003a50 2024-06-29T15:59:47.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.426+0000 7f82a7fff640 1 -- 172.21.15.81:0/2681354809 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v71) ==== 190+0+147 (secure 0 0 0) 0x7f82a0087030 con 0x7f829c003a50 2024-06-29T15:59:47.426 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-06-29T15:59:47.426 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBzGIBmZEP4JRAAx+kqCrlZK0Uz8HQFi69S8w== 2024-06-29T15:59:47.426 INFO:teuthology.orchestra.run.smithi081.stdout: caps mds = "allow" 2024-06-29T15:59:47.426 INFO:teuthology.orchestra.run.smithi081.stdout: caps mon = "allow r" 2024-06-29T15:59:47.426 INFO:teuthology.orchestra.run.smithi081.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-06-29T15:59:47.427 INFO:teuthology.orchestra.run.smithi081.stderr:updated caps for client.0 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 -- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8290076420 msgr2=0x7f8290078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8290076420 0x7f8290078930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f82b006d410 tx=0x7f8298006aa0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 -- 172.21.15.81:0/2681354809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f829c003a50 msgr2=0x7f82b00cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f829c003a50 0x7f82b00cd100 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009a70 tx=0x7f82a00336f0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 -- 172.21.15.81:0/2681354809 shutdown_connections 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8290076420 0x7f8290078930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82b00a0880 0x7f82b00a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82b01e93b0 0x7f82b009ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 --2- 172.21.15.81:0/2681354809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f829c003a50 0x7f82b00cd100 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 -- 172.21.15.81:0/2681354809 >> 172.21.15.81:0/2681354809 conn(0x7f82b00b56d0 msgr2=0x7f82b00d1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 -- 172.21.15.81:0/2681354809 shutdown_connections 2024-06-29T15:59:47.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.428+0000 7f82b5711640 1 -- 172.21.15.81:0/2681354809 wait complete. 2024-06-29T15:59:47.440 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T15:59:47.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.553+0000 7f501d44d640 1 -- 172.21.15.81:0/2232508437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50180b41a0 msgr2=0x7f50180ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.553+0000 7f501d44d640 1 --2- 172.21.15.81:0/2232508437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50180b41a0 0x7f50180ce990 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f50180c5400 tx=0x7f500002a4d0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.553+0000 7f501d44d640 1 -- 172.21.15.81:0/2232508437 shutdown_connections 2024-06-29T15:59:47.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.553+0000 7f501d44d640 1 --2- 172.21.15.81:0/2232508437 >> v2:172.21.15.167:3300/0 conn(0x7f50180ceed0 0x7f50180b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.553+0000 7f501d44d640 1 --2- 172.21.15.81:0/2232508437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50180b41a0 0x7f50180ce990 unknown :-1 s=CLOSED pgs=2913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.553+0000 7f501d44d640 1 --2- 172.21.15.81:0/2232508437 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50180b30c0 0x7f50180ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.553+0000 7f501d44d640 1 -- 172.21.15.81:0/2232508437 >> 172.21.15.81:0/2232508437 conn(0x7f50180b3680 msgr2=0x7f50180b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.553+0000 7f501d44d640 1 -- 172.21.15.81:0/2232508437 shutdown_connections 2024-06-29T15:59:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.553+0000 7f501d44d640 1 -- 172.21.15.81:0/2232508437 wait complete. 2024-06-29T15:59:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.555+0000 7f501d44d640 1 Processor -- start 2024-06-29T15:59:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.555+0000 7f501d44d640 1 -- start start 2024-06-29T15:59:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.556+0000 7f501d44d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50180b30c0 0x7f5018147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.556+0000 7f501d44d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50180b41a0 0x7f5018148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f501d44d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50180ceed0 0x7f50180cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f501d44d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f50180b5550 con 0x7f50180ceed0 2024-06-29T15:59:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f501d44d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f50180b53d0 con 0x7f50180b41a0 2024-06-29T15:59:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f501d44d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f50180b56d0 con 0x7f50180b30c0 2024-06-29T15:59:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f50177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50180ceed0 0x7f50180cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f50177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50180ceed0 0x7f50180cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48394/0 (socket says 172.21.15.81:48394) 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f50177fe640 1 -- 172.21.15.81:0/2192813836 learned_addr learned my addr 172.21.15.81:0/2192813836 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f500ffff640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50180b41a0 0x7f5018148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f50177fe640 1 -- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50180b30c0 msgr2=0x7f5018147a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f5016ffd640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50180b30c0 0x7f5018147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f50177fe640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50180b30c0 0x7f5018147a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f50177fe640 1 -- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50180b41a0 msgr2=0x7f5018148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f50177fe640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50180b41a0 0x7f5018148170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f50177fe640 1 -- 172.21.15.81:0/2192813836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501815c560 con 0x7f50180ceed0 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f5016ffd640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50180b30c0 0x7f5018147a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f500ffff640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50180b41a0 0x7f5018148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f50177fe640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50180ceed0 0x7f50180cb670 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7f501000cb00 tx=0x7f501000a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f500f7fe640 1 -- 172.21.15.81:0/2192813836 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501000f450 con 0x7f50180ceed0 2024-06-29T15:59:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f500f7fe640 1 -- 172.21.15.81:0/2192813836 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f501000f700 con 0x7f50180ceed0 2024-06-29T15:59:47.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f500f7fe640 1 -- 172.21.15.81:0/2192813836 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501000fb50 con 0x7f50180ceed0 2024-06-29T15:59:47.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f501d44d640 1 -- 172.21.15.81:0/2192813836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f501815c840 con 0x7f50180ceed0 2024-06-29T15:59:47.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.557+0000 7f501d44d640 1 -- 172.21.15.81:0/2192813836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50180c8f40 con 0x7f50180ceed0 2024-06-29T15:59:47.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.558+0000 7f501d44d640 1 -- 172.21.15.81:0/2192813836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50180b4980 con 0x7f50180ceed0 2024-06-29T15:59:47.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.559+0000 7f500f7fe640 1 -- 172.21.15.81:0/2192813836 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f501000fcf0 con 0x7f50180ceed0 2024-06-29T15:59:47.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.560+0000 7f500f7fe640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4ff80763f0 0x7f4ff8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:47.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.560+0000 7f500f7fe640 1 -- 172.21.15.81:0/2192813836 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f5010098510 con 0x7f50180ceed0 2024-06-29T15:59:47.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.560+0000 7f5016ffd640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4ff80763f0 0x7f4ff8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:47.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.560+0000 7f5016ffd640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4ff80763f0 0x7f4ff8078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f500800ba00 tx=0x7f5008005c80 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:47.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.564+0000 7f500f7fe640 1 -- 172.21.15.81:0/2192813836 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5010014030 con 0x7f50180ceed0 2024-06-29T15:59:47.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.784+0000 7f501d44d640 1 -- 172.21.15.81:0/2192813836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50180cbde0 con 0x7f50180ceed0 2024-06-29T15:59:47.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.785+0000 7f500f7fe640 1 -- 172.21.15.81:0/2192813836 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5238 (secure 0 0 0) 0x7f5010061bd0 con 0x7f50180ceed0 2024-06-29T15:59:47.785 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T15:59:47.786 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":154,"btime":"2024-06-29T15:59:45:615681+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10196,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2932449820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2932449820},{"type":"v1","addr":"172.21.15.167:6837","nonce":2932449820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/317823138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":317823138},{"type":"v1","addr":"172.21.15.81:6835","nonce":317823138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3582280449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3582280449},{"type":"v1","addr":"172.21.15.167:6836","nonce":3582280449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:41.965720+0000","modified":"2024-06-29T15:59:44.983893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13371},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13371":{"gid":13371,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/28043779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":28043779},{"type":"v1","addr":"172.21.15.81:6834","nonce":28043779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13371,"qdb_cluster":[13371]},"id":18}]} 2024-06-29T15:59:47.786 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 154 2024-06-29T15:59:47.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 -- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4ff80763f0 msgr2=0x7f4ff8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4ff80763f0 0x7f4ff8078900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f500800ba00 tx=0x7f5008005c80 comp rx=0 tx=0).stop 2024-06-29T15:59:47.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 -- 172.21.15.81:0/2192813836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50180ceed0 msgr2=0x7f50180cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:47.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50180ceed0 0x7f50180cb670 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7f501000cb00 tx=0x7f501000a250 comp rx=0 tx=0).stop 2024-06-29T15:59:47.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 -- 172.21.15.81:0/2192813836 shutdown_connections 2024-06-29T15:59:47.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f4ff80763f0 0x7f4ff8078900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50180ceed0 0x7f50180cb670 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f50180b41a0 0x7f5018148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 --2- 172.21.15.81:0/2192813836 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f50180b30c0 0x7f5018147a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:47.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 -- 172.21.15.81:0/2192813836 >> 172.21.15.81:0/2192813836 conn(0x7f50180b3680 msgr2=0x7f50180b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:47.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 -- 172.21.15.81:0/2192813836 shutdown_connections 2024-06-29T15:59:47.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:47.787+0000 7f501d44d640 1 -- 172.21.15.81:0/2192813836 wait complete. 2024-06-29T15:59:48.201 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-29T15:59:41.965720+0000', 'modified': '2024-06-29T15:59:44.983893+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 13371}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_13371': {'gid': 13371, 'name': 'a', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6834/28043779', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6832', 'nonce': 28043779}, {'type': 'v1', 'addr': '172.21.15.81:6834', 'nonce': 28043779}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 13371, 'qdb_cluster': [13371]} 2024-06-29T15:59:48.201 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-29T15:59:48.201 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-06-29T15:59:48.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.318+0000 7f6d53ee4640 1 -- 172.21.15.81:0/3586869124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d38004e60 msgr2=0x7f6d380052d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.318+0000 7f6d53ee4640 1 --2- 172.21.15.81:0/3586869124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d38004e60 0x7f6d380052d0 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f6d44001dc0 tx=0x7f6d38005830 comp rx=0 tx=0).stop 2024-06-29T15:59:48.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.318+0000 7f6d52481640 1 -- 172.21.15.81:0/3586869124 reap_dead start 2024-06-29T15:59:48.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.318+0000 7f6d53ee4640 1 -- 172.21.15.81:0/3586869124 shutdown_connections 2024-06-29T15:59:48.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.318+0000 7f6d53ee4640 1 -- 172.21.15.81:0/3586869124 >> 172.21.15.81:0/3586869124 conn(0x7f6d4c0b3510 msgr2=0x7f6d4c0b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:48.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.318+0000 7f6d53ee4640 1 -- 172.21.15.81:0/3586869124 shutdown_connections 2024-06-29T15:59:48.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.318+0000 7f6d53ee4640 1 -- 172.21.15.81:0/3586869124 wait complete. 2024-06-29T15:59:48.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.320+0000 7f6d53ee4640 1 Processor -- start 2024-06-29T15:59:48.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.320+0000 7f6d53ee4640 1 -- start start 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.321+0000 7f6d53ee4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d38004e60 0x7f6d4c1cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.321+0000 7f6d53ee4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d4c1ce2c0 0x7f6d4c0b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d53ee4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d4c0b0ca0 0x7f6d4c09b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d53ee4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c0ce8a0 con 0x7f6d38004e60 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d53ee4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c0ce720 con 0x7f6d4c1ce2c0 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d53ee4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6d4c0b4f20 con 0x7f6d4c0b0ca0 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d51c80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d38004e60 0x7f6d4c1cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d51c80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d38004e60 0x7f6d4c1cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48416/0 (socket says 172.21.15.81:48416) 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d51c80640 1 -- 172.21.15.81:0/2338642559 learned_addr learned my addr 172.21.15.81:0/2338642559 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d51c80640 1 -- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d4c0b0ca0 msgr2=0x7f6d4c09b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d51c80640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d4c0b0ca0 0x7f6d4c09b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d51c80640 1 -- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d4c1ce2c0 msgr2=0x7f6d4c0b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d51c80640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d4c1ce2c0 0x7f6d4c0b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d51c80640 1 -- 172.21.15.81:0/2338642559 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4c09db10 con 0x7f6d38004e60 2024-06-29T15:59:48.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d51c80640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d38004e60 0x7f6d4c1cdba0 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f6d48006630 tx=0x7f6d480039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:48.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d427fc640 1 -- 172.21.15.81:0/2338642559 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48015070 con 0x7f6d38004e60 2024-06-29T15:59:48.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d427fc640 1 -- 172.21.15.81:0/2338642559 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d4800f980 con 0x7f6d38004e60 2024-06-29T15:59:48.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d53ee4640 1 -- 172.21.15.81:0/2338642559 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4c09ddf0 con 0x7f6d38004e60 2024-06-29T15:59:48.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d427fc640 1 -- 172.21.15.81:0/2338642559 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4800fdd0 con 0x7f6d38004e60 2024-06-29T15:59:48.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.322+0000 7f6d53ee4640 1 -- 172.21.15.81:0/2338642559 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d4c09e3b0 con 0x7f6d38004e60 2024-06-29T15:59:48.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.325+0000 7f6d427fc640 1 -- 172.21.15.81:0/2338642559 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f6d48011430 con 0x7f6d38004e60 2024-06-29T15:59:48.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.325+0000 7f6d427fc640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f6d300763f0 0x7f6d30078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.325+0000 7f6d427fc640 1 -- 172.21.15.81:0/2338642559 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f6d48098750 con 0x7f6d38004e60 2024-06-29T15:59:48.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.326+0000 7f6d5147f640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f6d300763f0 0x7f6d30078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.326+0000 7f6d5ada5740 1 Processor -- start 2024-06-29T15:59:48.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.326+0000 7f6d5ada5740 1 -- start start 2024-06-29T15:59:48.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.326+0000 7f6d5147f640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f6d300763f0 0x7f6d30078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c005d80 tx=0x7f6d3c005c50 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:48.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.327+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e1b8270 0x559f8e5a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.327+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e5883a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.327+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e197590 0x559f8e5888e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.327+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559f8e520bc0 con 0x559f8e18f3b0 2024-06-29T15:59:48.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.327+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x559f8e4cf630 con 0x559f8e1b8270 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.327+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x559f8e4bebd0 con 0x559f8e197590 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.327+0000 7f6d5147f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e5883a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.327+0000 7f6d5147f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e5883a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48432/0 (socket says 172.21.15.81:48432) 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.327+0000 7f6d5147f640 1 -- 172.21.15.81:0/2817785065 learned_addr learned my addr 172.21.15.81:0/2817785065 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d51c80640 1 --2- 172.21.15.81:0/2817785065 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e1b8270 0x559f8e5a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d52481640 1 --2- 172.21.15.81:0/2817785065 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e197590 0x559f8e5888e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d5147f640 1 -- 172.21.15.81:0/2817785065 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e197590 msgr2=0x559f8e5888e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d5147f640 1 --2- 172.21.15.81:0/2817785065 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e197590 0x559f8e5888e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d5147f640 1 -- 172.21.15.81:0/2817785065 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e1b8270 msgr2=0x559f8e5a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d5147f640 1 --2- 172.21.15.81:0/2817785065 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e1b8270 0x559f8e5a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d5147f640 1 -- 172.21.15.81:0/2817785065 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f8e4bbca0 con 0x559f8e18f3b0 2024-06-29T15:59:48.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d51c80640 1 --2- 172.21.15.81:0/2817785065 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e1b8270 0x559f8e5a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T15:59:48.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d5147f640 1 --2- 172.21.15.81:0/2817785065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e5883a0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00ba70 tx=0x7f6d3c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d582a67a12f7f8c3 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:48.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d2f7fe640 1 -- 172.21.15.81:0/2817785065 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c011bc0 con 0x559f8e18f3b0 2024-06-29T15:59:48.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d2f7fe640 1 -- 172.21.15.81:0/2817785065 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d3c011d60 con 0x559f8e18f3b0 2024-06-29T15:59:48.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.328+0000 7f6d2f7fe640 1 -- 172.21.15.81:0/2817785065 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c005220 con 0x559f8e18f3b0 2024-06-29T15:59:48.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.329+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2817785065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 msgr2=0x559f8e5883a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.329+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/2817785065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e5883a0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00ba70 tx=0x7f6d3c013000 comp rx=0 tx=0).stop 2024-06-29T15:59:48.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.329+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2817785065 shutdown_connections 2024-06-29T15:59:48.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.329+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/2817785065 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e197590 0x559f8e5888e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.329+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/2817785065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e5883a0 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.329+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/2817785065 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e1b8270 0x559f8e5a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.329+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2817785065 >> 172.21.15.81:0/2817785065 conn(0x559f8e1ac3c0 msgr2=0x559f8e55bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:48.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.329+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2817785065 shutdown_connections 2024-06-29T15:59:48.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.329+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2817785065 wait complete. 2024-06-29T15:59:48.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.330+0000 7f6d5ada5740 1 -- start start 2024-06-29T15:59:48.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.330+0000 7f6d5ada5740 1 Processor -- start 2024-06-29T15:59:48.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.331+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e586520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.331+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e197590 0x559f8e588420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.331+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e1b8270 0x559f8e5a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.331+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559f8e45cae0 con 0x559f8e18f3b0 2024-06-29T15:59:48.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.331+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x559f8e4c2c10 con 0x559f8e197590 2024-06-29T15:59:48.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.331+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x559f8e232620 con 0x559f8e1b8270 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.331+0000 7f6d5ada5740 10 client.0 _ll_register_callbacks cb 0x559f8e469480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.331+0000 7f6d51c80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e586520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.331+0000 7f6d51c80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e586520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48440/0 (socket says 172.21.15.81:48440) 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.331+0000 7f6d51c80640 1 -- 172.21.15.81:0/1773805366 learned_addr learned my addr 172.21.15.81:0/1773805366 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d51c80640 1 -- 172.21.15.81:0/1773805366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e1b8270 msgr2=0x559f8e5a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d51c80640 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e1b8270 0x559f8e5a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d51c80640 1 -- 172.21.15.81:0/1773805366 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e197590 msgr2=0x559f8e588420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d51c80640 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e197590 0x559f8e588420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d51c80640 1 -- 172.21.15.81:0/1773805366 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f8e223850 con 0x559f8e18f3b0 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d51c80640 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e586520 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f6d48096970 tx=0x7f6d48061070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8f37cc3ef64536 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d2cff9640 1 -- 172.21.15.81:0/1773805366 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48020070 con 0x559f8e18f3b0 2024-06-29T15:59:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d2cff9640 1 -- 172.21.15.81:0/1773805366 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f6d48076ae0 con 0x559f8e18f3b0 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1773805366 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d3c011d60 con 0x559f8e18f3b0 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d2cff9640 1 -- 172.21.15.81:0/1773805366 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7f6d48061e20 con 0x559f8e18f3b0 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d5ada5740 10 client.13521 fetch_fsmap learned FSMap version 154 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1773805366 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x559f8e29a6b0 con 0x559f8e18f3b0 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d2cff9640 1 -- 172.21.15.81:0/1773805366 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d3c011d60 con 0x559f8e18f3b0 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d2cff9640 1 -- 172.21.15.81:0/1773805366 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 154) ==== 3502+0+0 (secure 0 0 0) 0x7f6d48076ce0 con 0x559f8e18f3b0 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d5ada5740 10 client.13521 fetch_fsmap finished waiting for FSMap version 154 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d5ada5740 10 client.13521 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d5ada5740 20 client.13521 populate_metadata read hostname 'smithi081' 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.81:6832/28043779,v1:172.21.15.81:6834/28043779] conn(0x559f8e176e40 0x559f8e5b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d5ada5740 4 client.13521 mds_command: new command op to 13371 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.332+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1773805366 --> [v2:172.21.15.81:6832/28043779,v1:172.21.15.81:6834/28043779] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f8e4d1580 con 0x559f8e176e40 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.333+0000 7f6d5147f640 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.81:6832/28043779,v1:172.21.15.81:6834/28043779] conn(0x559f8e176e40 0x559f8e5b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.333+0000 7f6d2cff9640 10 client.13521 ms_handle_connect on v2:172.21.15.81:6832/28043779 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.333+0000 7f6d5147f640 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.81:6832/28043779,v1:172.21.15.81:6834/28043779] conn(0x559f8e176e40 0x559f8e5b6d60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49b9d3a06703a01e server_cookie=b78da62c8afd97d5 in_seq=0 out_seq=0 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d2cff9640 1 -- 172.21.15.81:0/1773805366 <== mds.0 v2:172.21.15.81:6832/28043779 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559f8e4d1580 con 0x559f8e176e40 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d2cff9640 10 client.13521 handle_command_reply: tid=0 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 client.13521 shutdown 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1773805366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 msgr2=0x559f8e586520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e586520 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f6d48096970 tx=0x7f6d48061070 comp rx=0 tx=0).stop 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1773805366 shutdown_connections 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.81:6832/28043779,v1:172.21.15.81:6834/28043779] conn(0x559f8e176e40 0x559f8e5b6d60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e1b8270 0x559f8e5a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e197590 0x559f8e588420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d2cff9640 0 client.13521 ms_handle_reset on v2:172.21.15.81:6832/28043779 2024-06-29T15:59:48.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/1773805366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e18f3b0 0x559f8e586520 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1773805366 >> 172.21.15.81:0/1773805366 conn(0x559f8e1ac3c0 msgr2=0x559f8e557b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:48.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1773805366 shutdown_connections 2024-06-29T15:59:48.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1773805366 wait complete. 2024-06-29T15:59:48.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.334+0000 7f6d5ada5740 20 client.13521 trim_cache size 0 max 16384 2024-06-29T15:59:48.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.353+0000 7f6d5ada5740 1 Processor -- start 2024-06-29T15:59:48.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.353+0000 7f6d5ada5740 1 -- start start 2024-06-29T15:59:48.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e18f3b0 0x559f8e5861e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e197590 0x559f8e5c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e1b8270 0x559f8e5c7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c1feae0 con 0x559f8e1b8270 2024-06-29T15:59:48.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c1fde70 con 0x559f8e18f3b0 2024-06-29T15:59:48.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6d4c1fe490 con 0x559f8e197590 2024-06-29T15:59:48.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d52481640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e1b8270 0x559f8e5c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d52481640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e1b8270 0x559f8e5c7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48456/0 (socket says 172.21.15.81:48456) 2024-06-29T15:59:48.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d52481640 1 -- 172.21.15.81:0/1238020450 learned_addr learned my addr 172.21.15.81:0/1238020450 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d5147f640 1 --2- 172.21.15.81:0/1238020450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e197590 0x559f8e5c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d51c80640 1 --2- 172.21.15.81:0/1238020450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e18f3b0 0x559f8e5861e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d52481640 1 -- 172.21.15.81:0/1238020450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e197590 msgr2=0x559f8e5c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d52481640 1 --2- 172.21.15.81:0/1238020450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e197590 0x559f8e5c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d52481640 1 -- 172.21.15.81:0/1238020450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e18f3b0 msgr2=0x559f8e5861e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d52481640 1 --2- 172.21.15.81:0/1238020450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e18f3b0 0x559f8e5861e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d52481640 1 -- 172.21.15.81:0/1238020450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f8e5a1670 con 0x559f8e1b8270 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d52481640 1 --2- 172.21.15.81:0/1238020450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e1b8270 0x559f8e5c7220 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f6d44001eb0 tx=0x7f6d44001330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e906e27fe54ed4 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d2effd640 1 -- 172.21.15.81:0/1238020450 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44003b70 con 0x559f8e1b8270 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d2effd640 1 -- 172.21.15.81:0/1238020450 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d44003e00 con 0x559f8e1b8270 2024-06-29T15:59:48.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.355+0000 7f6d2effd640 1 -- 172.21.15.81:0/1238020450 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d440062d0 con 0x559f8e1b8270 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.356+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1238020450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e1b8270 msgr2=0x559f8e5c7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.356+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/1238020450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e1b8270 0x559f8e5c7220 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f6d44001eb0 tx=0x7f6d44001330 comp rx=0 tx=0).stop 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.356+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1238020450 shutdown_connections 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.356+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/1238020450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e1b8270 0x559f8e5c7220 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.356+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/1238020450 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e197590 0x559f8e5c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.356+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/1238020450 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e18f3b0 0x559f8e5861e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.356+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1238020450 >> 172.21.15.81:0/1238020450 conn(0x559f8e1ac3c0 msgr2=0x559f8e584c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.356+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1238020450 shutdown_connections 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.356+0000 7f6d5ada5740 1 -- 172.21.15.81:0/1238020450 wait complete. 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.357+0000 7f6d5ada5740 1 -- start start 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.357+0000 7f6d5ada5740 1 Processor -- start 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e18f3b0 0x559f8e5807a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e197590 0x559f8e606480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5ada5740 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e1b8270 0x559f8e55f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559f8e5a2da0 con 0x559f8e197590 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x559f8e588960 con 0x559f8e18f3b0 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5ada5740 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x559f8e5a20d0 con 0x559f8e1b8270 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5ada5740 10 client.0 _ll_register_callbacks cb 0x7f6d48006830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5147f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e197590 0x559f8e606480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5147f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e197590 0x559f8e606480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48466/0 (socket says 172.21.15.81:48466) 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5147f640 1 -- 172.21.15.81:0/2589526245 learned_addr learned my addr 172.21.15.81:0/2589526245 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d52481640 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e1b8270 0x559f8e55f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d51c80640 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e18f3b0 0x559f8e5807a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5147f640 1 -- 172.21.15.81:0/2589526245 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e1b8270 msgr2=0x559f8e55f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5147f640 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e1b8270 0x559f8e55f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5147f640 1 -- 172.21.15.81:0/2589526245 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e18f3b0 msgr2=0x559f8e5807a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-29T15:59:48.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5147f640 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e18f3b0 0x559f8e5807a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5147f640 1 -- 172.21.15.81:0/2589526245 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f8e621770 con 0x559f8e197590 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5147f640 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e197590 0x559f8e606480 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c022000 tx=0x7f6d3c011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35950f46c40f746 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d2cff9640 1 -- 172.21.15.81:0/2589526245 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c031810 con 0x559f8e197590 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d2cff9640 1 -- 172.21.15.81:0/2589526245 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f6d3c0319b0 con 0x559f8e197590 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.359+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2589526245 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d44003e00 con 0x559f8e197590 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d2cff9640 1 -- 172.21.15.81:0/2589526245 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7f6d3c03e050 con 0x559f8e197590 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d5ada5740 10 client.13527 fetch_fsmap learned FSMap version 154 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2589526245 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x559f8e5b3880 con 0x559f8e197590 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d2cff9640 1 -- 172.21.15.81:0/2589526245 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d44003e00 con 0x559f8e197590 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d2cff9640 1 -- 172.21.15.81:0/2589526245 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 154) ==== 3502+0+0 (secure 0 0 0) 0x7f6d3c041020 con 0x559f8e197590 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d5ada5740 10 client.13527 fetch_fsmap finished waiting for FSMap version 154 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d5ada5740 10 client.13527 resolve_mds: resolved name 'a' to daemon mds.a 2024-06-29T15:59:48.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d5ada5740 20 client.13527 populate_metadata read hostname 'smithi081' 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.81:6832/28043779,v1:172.21.15.81:6834/28043779] conn(0x559f8e55fd10 0x559f8e620530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d5ada5740 4 client.13527 mds_command: new command op to 13371 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2589526245 --> [v2:172.21.15.81:6832/28043779,v1:172.21.15.81:6834/28043779] -- command(tid 0: {"prefix": "status"}) -- 0x559f8e4cd020 con 0x559f8e55fd10 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.360+0000 7f6d51c80640 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.81:6832/28043779,v1:172.21.15.81:6834/28043779] conn(0x559f8e55fd10 0x559f8e620530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d2cff9640 10 client.13527 ms_handle_connect on v2:172.21.15.81:6832/28043779 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d51c80640 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.81:6832/28043779,v1:172.21.15.81:6834/28043779] conn(0x559f8e55fd10 0x559f8e620530 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a7bf5a616899273 server_cookie=ca17961c7ca55986 in_seq=0 out_seq=0 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d2cff9640 1 -- 172.21.15.81:0/2589526245 <== mds.0 v2:172.21.15.81:6832/28043779 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x559f8e4cd020 con 0x559f8e55fd10 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d2cff9640 10 client.13527 handle_command_reply: tid=0 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 client.13527 shutdown 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2589526245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e197590 msgr2=0x559f8e606480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e197590 0x559f8e606480 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c022000 tx=0x7f6d3c011cf0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2589526245 shutdown_connections 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.81:6832/28043779,v1:172.21.15.81:6834/28043779] conn(0x559f8e55fd10 0x559f8e620530 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x559f8e1b8270 0x559f8e55f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d2cff9640 0 client.13527 ms_handle_reset on v2:172.21.15.81:6832/28043779 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559f8e197590 0x559f8e606480 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 --2- 172.21.15.81:0/2589526245 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x559f8e18f3b0 0x559f8e5807a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2589526245 >> 172.21.15.81:0/2589526245 conn(0x559f8e1ac3c0 msgr2=0x559f8e59ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2589526245 shutdown_connections 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.361+0000 7f6d5ada5740 1 -- 172.21.15.81:0/2589526245 wait complete. 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d5ada5740 20 client.13527 trim_cache size 0 max 16384 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T15:59:48.367 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "e6805bd5-4748-48b4-ab28-fbf8a8d96ae4", 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 13371, 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 5.7559103059999996, 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 154, 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 266, 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 266, 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 10.434310623 2024-06-29T15:59:48.368 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T15:59:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 -- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f6d300763f0 msgr2=0x7f6d30078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f6d300763f0 0x7f6d30078900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c005d80 tx=0x7f6d3c005c50 comp rx=0 tx=0).stop 2024-06-29T15:59:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 -- 172.21.15.81:0/2338642559 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d38004e60 msgr2=0x7f6d4c1cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d38004e60 0x7f6d4c1cdba0 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f6d48006630 tx=0x7f6d480039c0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 -- 172.21.15.81:0/2338642559 shutdown_connections 2024-06-29T15:59:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f6d300763f0 0x7f6d30078900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6d4c0b0ca0 0x7f6d4c09b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6d4c1ce2c0 0x7f6d4c0b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 --2- 172.21.15.81:0/2338642559 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d38004e60 0x7f6d4c1cdba0 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 -- 172.21.15.81:0/2338642559 >> 172.21.15.81:0/2338642559 conn(0x7f6d4c0b3510 msgr2=0x7f6d4c070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:48.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 -- 172.21.15.81:0/2338642559 shutdown_connections 2024-06-29T15:59:48.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.362+0000 7f6d53ee4640 1 -- 172.21.15.81:0/2338642559 wait complete. 2024-06-29T15:59:48.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-06-29T15:59:48.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.486+0000 7fd0b59f5640 1 -- 172.21.15.81:0/1179625133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0b00b9040 msgr2=0x7fd0b00d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.486+0000 7fd0b59f5640 1 --2- 172.21.15.81:0/1179625133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0b00b9040 0x7fd0b00d0480 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0009830 tx=0x7fd0a002a5a0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.486+0000 7fd0b59f5640 1 -- 172.21.15.81:0/1179625133 shutdown_connections 2024-06-29T15:59:48.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.486+0000 7fd0b59f5640 1 --2- 172.21.15.81:0/1179625133 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0b00b5250 0x7fd0b00d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.486+0000 7fd0b59f5640 1 --2- 172.21.15.81:0/1179625133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0b00b9040 0x7fd0b00d0480 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.486+0000 7fd0b59f5640 1 -- 172.21.15.81:0/1179625133 >> 172.21.15.81:0/1179625133 conn(0x7fd0b009cf30 msgr2=0x7fd0b00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:48.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.486+0000 7fd0b59f5640 1 -- 172.21.15.81:0/1179625133 shutdown_connections 2024-06-29T15:59:48.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.486+0000 7fd0b59f5640 1 -- 172.21.15.81:0/1179625133 wait complete. 2024-06-29T15:59:48.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.488+0000 7fd0b59f5640 1 Processor -- start 2024-06-29T15:59:48.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.488+0000 7fd0b59f5640 1 -- start start 2024-06-29T15:59:48.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.489+0000 7fd0b59f5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0b00b5250 0x7fd0b01515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.489+0000 7fd0b59f5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0b00b7290 0x7fd0b014fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.489+0000 7fd0b59f5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0b00b9040 0x7fd0b01530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0b59f5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b00b7db0 con 0x7fd0b00b9040 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0b59f5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b00d2d50 con 0x7fd0b00b5250 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0b59f5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b00b7f30 con 0x7fd0b00b7290 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0af7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0b00b9040 0x7fd0b01530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0af7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0b00b9040 0x7fd0b01530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48494/0 (socket says 172.21.15.81:48494) 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0af7fe640 1 -- 172.21.15.81:0/383569108 learned_addr learned my addr 172.21.15.81:0/383569108 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0aeffd640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0b00b5250 0x7fd0b01515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0a7fff640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0b00b7290 0x7fd0b014fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0af7fe640 1 -- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0b00b7290 msgr2=0x7fd0b014fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0af7fe640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0b00b7290 0x7fd0b014fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0af7fe640 1 -- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0b00b5250 msgr2=0x7fd0b01515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0af7fe640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0b00b5250 0x7fd0b01515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0af7fe640 1 -- 172.21.15.81:0/383569108 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b0150540 con 0x7fd0b00b9040 2024-06-29T15:59:48.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0af7fe640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0b00b9040 0x7fd0b01530e0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800c760 tx=0x7fd0a800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:48.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0acff9640 1 -- 172.21.15.81:0/383569108 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8003b70 con 0x7fd0b00b9040 2024-06-29T15:59:48.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0acff9640 1 -- 172.21.15.81:0/383569108 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0a8003d30 con 0x7fd0b00b9040 2024-06-29T15:59:48.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0b59f5640 1 -- 172.21.15.81:0/383569108 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b0150820 con 0x7fd0b00b9040 2024-06-29T15:59:48.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0acff9640 1 -- 172.21.15.81:0/383569108 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a80052d0 con 0x7fd0b00b9040 2024-06-29T15:59:48.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.490+0000 7fd0b59f5640 1 -- 172.21.15.81:0/383569108 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b0150de0 con 0x7fd0b00b9040 2024-06-29T15:59:48.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.492+0000 7fd0b59f5640 1 -- 172.21.15.81:0/383569108 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b00b84c0 con 0x7fd0b00b9040 2024-06-29T15:59:48.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.492+0000 7fd0acff9640 1 -- 172.21.15.81:0/383569108 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fd0a8005470 con 0x7fd0b00b9040 2024-06-29T15:59:48.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.492+0000 7fd0acff9640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd090076420 0x7fd090078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:48.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.493+0000 7fd0acff9640 1 -- 172.21.15.81:0/383569108 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7fd0a8097550 con 0x7fd0b00b9040 2024-06-29T15:59:48.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.493+0000 7fd0aeffd640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd090076420 0x7fd090078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:48.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.493+0000 7fd0aeffd640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd090076420 0x7fd090078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0034000 tx=0x7fd0a0004730 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:48.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.497+0000 7fd0acff9640 1 -- 172.21.15.81:0/383569108 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd0a8060640 con 0x7fd0b00b9040 2024-06-29T15:59:48.584 INFO:tasks.ceph.mds.a.smithi081.stderr:2024-06-29T15:59:48.584+0000 7fdd806fe640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-29T15:59:48.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.719+0000 7fd0b59f5640 1 -- 172.21.15.81:0/383569108 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd0b0161550 con 0x7fd0b00b9040 2024-06-29T15:59:48.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.719+0000 7fd0acff9640 1 -- 172.21.15.81:0/383569108 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v154) ==== 53+0+73 (secure 0 0 0) 0x7fd0a8095e20 con 0x7fd0b00b9040 2024-06-29T15:59:48.720 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-06-29T15:59:48.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 -- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd090076420 msgr2=0x7fd090078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd090076420 0x7fd090078930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0034000 tx=0x7fd0a0004730 comp rx=0 tx=0).stop 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 -- 172.21.15.81:0/383569108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0b00b9040 msgr2=0x7fd0b01530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0b00b9040 0x7fd0b01530e0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800c760 tx=0x7fd0a800cc20 comp rx=0 tx=0).stop 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 -- 172.21.15.81:0/383569108 shutdown_connections 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fd090076420 0x7fd090078930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0b00b9040 0x7fd0b01530e0 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd0b00b7290 0x7fd0b014fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 --2- 172.21.15.81:0/383569108 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd0b00b5250 0x7fd0b01515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 -- 172.21.15.81:0/383569108 >> 172.21.15.81:0/383569108 conn(0x7fd0b009cf30 msgr2=0x7fd0b00d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 -- 172.21.15.81:0/383569108 shutdown_connections 2024-06-29T15:59:48.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48.721+0000 7fd0b59f5640 1 -- 172.21.15.81:0/383569108 wait complete. 2024-06-29T15:59:48.732 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-29T15:59:48.733 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-29T15:59:48.733 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-06-29T15:59:48.733 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-29T15:59:48.733 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:48.733 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-29T15:59:48.733 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-06-29T15:59:48.733 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-29T15:59:48.733 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:48.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:48.734 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-06-29T15:59:48.749 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:48.750 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 36931sec preferred_lft 36931sec 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::6cdb:6cff:fee6:5545/64 scope link 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:48.751 INFO:teuthology.orchestra.run.smithi081.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:59:48.752 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 66:d3:06:1d:8f:40 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T15:59:48.752 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::64d3:6ff:fe1d:8f40/64 scope link 2024-06-29T15:59:48.752 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:48.752 INFO:teuthology.orchestra.run.smithi081.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-06-29T15:59:48.752 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0e:06:ac:87:6e:91 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T15:59:48.752 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::c06:acff:fe87:6e91/64 scope link 2024-06-29T15:59:48.752 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-06-29T15:59:48.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:48.752 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-29T15:59:48.813 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-06-29T15:59:48.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:48.815 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-06-29T15:59:48.871 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-06-29T15:59:48.871 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-29T15:59:48.871 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-29T15:59:48.872 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-29T15:59:48.872 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-29T15:59:48.872 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-29T15:59:48.872 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-06-29T15:59:48.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-29T15:59:48.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-29T15:59:48.872 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-29T15:59:48.872 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-06-29T15:59:48.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-29T15:59:48.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:59:48.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:48.997 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T15:59:49.013 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T15:59:49.013 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T15:59:49.013 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:49.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:49.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-06-29T15:59:49.078 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:59:49.078 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:49.134 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-29T15:59:49.134 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-29T15:59:49.135 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:49.190 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-06-29T15:59:49.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:49.190 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:59:49.284 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:59:49.284 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:59:49.292 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-06-29T15:59:49.292 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-06-29T15:59:49.292 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.167:3300,172.21.15.167:3301" 2024-06-29T15:59:49.292 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@e6805bd5-4748-48b4-ab28-fbf8a8d96ae4.cephfs=/ 2024-06-29T15:59:49.292 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.81:3300/172.21.15.167:3300/172.21.15.167:3301" will pass to kernel 2024-06-29T15:59:49.293 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-29T15:59:49.293 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-06-29T15:59:49.293 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-29T15:59:49.293 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-06-29T15:59:49.293 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-29T15:59:49.294 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-29T15:59:49.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T15:59:49.294 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-29T15:59:49.320 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-29T15:59:49.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-29T15:59:49.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-29T15:59:49.422 INFO:teuthology.orchestra.run.smithi081.stdout:client10268 2024-06-29T15:59:49.422 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T15:59:49.422 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T15:59:49.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-06-29T15:59:49.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.592+0000 7f8fb7690640 1 -- 172.21.15.81:0/617069761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f98004d40 msgr2=0x7f8f98005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.592+0000 7f8fb7690640 1 --2- 172.21.15.81:0/617069761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f98004d40 0x7f8f98005150 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800c290 tx=0x7f8f98005830 comp rx=0 tx=0).stop 2024-06-29T15:59:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.592+0000 7f8fb5c2d640 1 -- 172.21.15.81:0/617069761 reap_dead start 2024-06-29T15:59:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.592+0000 7f8fb7690640 1 -- 172.21.15.81:0/617069761 shutdown_connections 2024-06-29T15:59:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.592+0000 7f8fb7690640 1 -- 172.21.15.81:0/617069761 >> 172.21.15.81:0/617069761 conn(0x7f8fb00d0930 msgr2=0x7f8fb00d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.593+0000 7f8fb7690640 1 -- 172.21.15.81:0/617069761 shutdown_connections 2024-06-29T15:59:49.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.593+0000 7f8fb7690640 1 -- 172.21.15.81:0/617069761 wait complete. 2024-06-29T15:59:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.594+0000 7f8fb7690640 1 Processor -- start 2024-06-29T15:59:49.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.594+0000 7f8fb7690640 1 -- start start 2024-06-29T15:59:49.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb7690640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fb00cf160 0x7f8fb0152980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb7690640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8fb0153040 0x7f8fb00cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb7690640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fb00ae2f0 0x7f8fb00ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb7690640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb015aed0 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb7690640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb00b4bd0 con 0x7f8fb0153040 2024-06-29T15:59:49.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb7690640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f8fb015b050 con 0x7f8fb00cf160 2024-06-29T15:59:49.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb5c2d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fb00ae2f0 0x7f8fb00ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb5c2d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fb00ae2f0 0x7f8fb00ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48506/0 (socket says 172.21.15.81:48506) 2024-06-29T15:59:49.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb5c2d640 1 -- 172.21.15.81:0/4292267137 learned_addr learned my addr 172.21.15.81:0/4292267137 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:49.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb4c2b640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8fb0153040 0x7f8fb00cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb5c2d640 1 -- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fb00cf160 msgr2=0x7f8fb0152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb5c2d640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fb00cf160 0x7f8fb0152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb5c2d640 1 -- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8fb0153040 msgr2=0x7f8fb00cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb5c2d640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8fb0153040 0x7f8fb00cfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb5c2d640 1 -- 172.21.15.81:0/4292267137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb00aec40 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb5c2d640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fb00ae2f0 0x7f8fb00ae700 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800c3b0 tx=0x7f8fa8004eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:49.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fadffb640 1 -- 172.21.15.81:0/4292267137 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8031070 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fadffb640 1 -- 172.21.15.81:0/4292267137 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8fa80060c0 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fb7690640 1 -- 172.21.15.81:0/4292267137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb00d02a0 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.596+0000 7f8fadffb640 1 -- 172.21.15.81:0/4292267137 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8006c30 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.597+0000 7f8fb7690640 1 -- 172.21.15.81:0/4292267137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb00ce680 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.598+0000 7f8fb7690640 1 -- 172.21.15.81:0/4292267137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fb006e2a0 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.598+0000 7f8fadffb640 1 -- 172.21.15.81:0/4292267137 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7f8fa8034070 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.599+0000 7f8fadffb640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8f9c0763f0 0x7f8f9c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.599+0000 7f8fadffb640 1 -- 172.21.15.81:0/4292267137 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f8fa80b7380 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.599+0000 7f8fb542c640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8f9c0763f0 0x7f8f9c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.600+0000 7f8fb542c640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8f9c0763f0 0x7f8f9c078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8fb006cbd0 tx=0x7f8fa0009960 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:49.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.603+0000 7f8fadffb640 1 -- 172.21.15.81:0/4292267137 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8fb006e2a0 con 0x7f8fb00ae2f0 2024-06-29T15:59:49.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.811+0000 7f8fb7690640 1 -- 172.21.15.81:0/4292267137 --> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) -- 0x7f8fb00cffb0 con 0x7f8f9c0763f0 2024-06-29T15:59:49.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.813+0000 7f8fadffb640 1 -- 172.21.15.81:0/4292267137 <== mgr.10028 v2:172.21.15.167:6832/1824030697 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+41 (secure 0 0 0) 0x7f8fb00cffb0 con 0x7f8f9c0763f0 2024-06-29T15:59:49.813 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-06-29T15:59:49.813 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-06-29T15:59:49.813 INFO:teuthology.orchestra.run.smithi081.stdout: "name": "cephfs" 2024-06-29T15:59:49.814 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-06-29T15:59:49.814 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 -- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8f9c0763f0 msgr2=0x7f8f9c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8f9c0763f0 0x7f8f9c078900 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8fb006cbd0 tx=0x7f8fa0009960 comp rx=0 tx=0).stop 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 -- 172.21.15.81:0/4292267137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fb00ae2f0 msgr2=0x7f8fb00ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fb00ae2f0 0x7f8fb00ae700 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800c3b0 tx=0x7f8fa8004eb0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 -- 172.21.15.81:0/4292267137 shutdown_connections 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f8f9c0763f0 0x7f8f9c078900 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fb00ae2f0 0x7f8fb00ae700 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8fb0153040 0x7f8fb00cfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 --2- 172.21.15.81:0/4292267137 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f8fb00cf160 0x7f8fb0152980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 -- 172.21.15.81:0/4292267137 >> 172.21.15.81:0/4292267137 conn(0x7f8fb00d0930 msgr2=0x7f8fb00b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 -- 172.21.15.81:0/4292267137 shutdown_connections 2024-06-29T15:59:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.815+0000 7f8fb7690640 1 -- 172.21.15.81:0/4292267137 wait complete. 2024-06-29T15:59:49.826 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2024-06-29T15:59:49.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.941+0000 7fe3b4c22640 1 -- 172.21.15.81:0/247982612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3b00ad0c0 msgr2=0x7fe3b015bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.941+0000 7fe3b4c22640 1 --2- 172.21.15.81:0/247982612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3b00ad0c0 0x7fe3b015bdf0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0073520 tx=0x7fe3a002a4d0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.941+0000 7fe3b4c22640 1 -- 172.21.15.81:0/247982612 shutdown_connections 2024-06-29T15:59:49.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.941+0000 7fe3b4c22640 1 --2- 172.21.15.81:0/247982612 >> v2:172.21.15.167:3300/0 conn(0x7fe3b0074630 0x7fe3b006c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.941+0000 7fe3b4c22640 1 --2- 172.21.15.81:0/247982612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3b00ad0c0 0x7fe3b015bdf0 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.941+0000 7fe3b4c22640 1 -- 172.21.15.81:0/247982612 >> 172.21.15.81:0/247982612 conn(0x7fe3b0072950 msgr2=0x7fe3b00b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:49.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.941+0000 7fe3b4c22640 1 -- 172.21.15.81:0/247982612 shutdown_connections 2024-06-29T15:59:49.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.941+0000 7fe3b4c22640 1 -- 172.21.15.81:0/247982612 wait complete. 2024-06-29T15:59:49.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.943+0000 7fe3b4c22640 1 Processor -- start 2024-06-29T15:59:49.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.943+0000 7fe3b4c22640 1 -- start start 2024-06-29T15:59:49.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3b4c22640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3b0073b10 0x7fe3b0153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3b4c22640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3b0074630 0x7fe3b014e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3b4c22640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe3b00ad0c0 0x7fe3b014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3b4c22640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b006d000 con 0x7fe3b0074630 2024-06-29T15:59:49.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3b4c22640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b006ce80 con 0x7fe3b0073b10 2024-06-29T15:59:49.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3b4c22640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe3b006d180 con 0x7fe3b00ad0c0 2024-06-29T15:59:49.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3a5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3b0074630 0x7fe3b014e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3a5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3b0074630 0x7fe3b014e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48530/0 (socket says 172.21.15.81:48530) 2024-06-29T15:59:49.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3a5d9b640 1 -- 172.21.15.81:0/1992468119 learned_addr learned my addr 172.21.15.81:0/1992468119 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:49.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3ae59c640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3b0073b10 0x7fe3b0153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3aed9d640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe3b00ad0c0 0x7fe3b014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3a5d9b640 1 -- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe3b00ad0c0 msgr2=0x7fe3b014eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3a5d9b640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe3b00ad0c0 0x7fe3b014eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3a5d9b640 1 -- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3b0073b10 msgr2=0x7fe3b0153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:49.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3a5d9b640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3b0073b10 0x7fe3b0153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:49.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.944+0000 7fe3a5d9b640 1 -- 172.21.15.81:0/1992468119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b0154a10 con 0x7fe3b0074630 2024-06-29T15:59:49.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.945+0000 7fe3a5d9b640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3b0074630 0x7fe3b014e5f0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7fe39800b410 tx=0x7fe39800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:49.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.945+0000 7fe3a77fe640 1 -- 172.21.15.81:0/1992468119 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398015020 con 0x7fe3b0074630 2024-06-29T15:59:49.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.945+0000 7fe3a77fe640 1 -- 172.21.15.81:0/1992468119 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe398011920 con 0x7fe3b0074630 2024-06-29T15:59:49.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.945+0000 7fe3b4c22640 1 -- 172.21.15.81:0/1992468119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3b0154cf0 con 0x7fe3b0074630 2024-06-29T15:59:49.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.945+0000 7fe3a77fe640 1 -- 172.21.15.81:0/1992468119 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398011d70 con 0x7fe3b0074630 2024-06-29T15:59:49.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.945+0000 7fe3b4c22640 1 -- 172.21.15.81:0/1992468119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3b01532b0 con 0x7fe3b0074630 2024-06-29T15:59:49.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.946+0000 7fe3b4c22640 1 -- 172.21.15.81:0/1992468119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3b006d6c0 con 0x7fe3b0074630 2024-06-29T15:59:49.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.947+0000 7fe3a77fe640 1 -- 172.21.15.81:0/1992468119 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 155) ==== 98303+0+0 (secure 0 0 0) 0x7fe398018020 con 0x7fe3b0074630 2024-06-29T15:59:49.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.947+0000 7fe3a77fe640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fe390076420 0x7fe390078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:49.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.948+0000 7fe3a77fe640 1 -- 172.21.15.81:0/1992468119 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7fe398097970 con 0x7fe3b0074630 2024-06-29T15:59:49.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.948+0000 7fe3ae59c640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fe390076420 0x7fe390078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:49.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.948+0000 7fe3ae59c640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fe390076420 0x7fe390078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0033000 tx=0x7fe3a0004730 comp rx=0 tx=0).ready entity=mgr.10028 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:49.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:49.952+0000 7fe3a77fe640 1 -- 172.21.15.81:0/1992468119 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe39801f020 con 0x7fe3b0074630 2024-06-29T15:59:50.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.180+0000 7fe3b4c22640 1 -- 172.21.15.81:0/1992468119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fe3b0154ec0 con 0x7fe3b0074630 2024-06-29T15:59:50.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.640+0000 7fe3a77fe640 1 -- 172.21.15.81:0/1992468119 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v156) ==== 92+0+0 (secure 0 0 0) 0x7fe398061140 con 0x7fe3b0074630 2024-06-29T15:59:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 -- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fe390076420 msgr2=0x7fe390078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fe390076420 0x7fe390078930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0033000 tx=0x7fe3a0004730 comp rx=0 tx=0).stop 2024-06-29T15:59:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 -- 172.21.15.81:0/1992468119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3b0074630 msgr2=0x7fe3b014e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3b0074630 0x7fe3b014e5f0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7fe39800b410 tx=0x7fe39800b8d0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 -- 172.21.15.81:0/1992468119 shutdown_connections 2024-06-29T15:59:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7fe390076420 0x7fe390078930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe3b00ad0c0 0x7fe3b014eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3b0074630 0x7fe3b014e5f0 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 --2- 172.21.15.81:0/1992468119 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe3b0073b10 0x7fe3b0153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 -- 172.21.15.81:0/1992468119 >> 172.21.15.81:0/1992468119 conn(0x7fe3b0072950 msgr2=0x7fe3b0156610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 -- 172.21.15.81:0/1992468119 shutdown_connections 2024-06-29T15:59:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.643+0000 7fe3b4c22640 1 -- 172.21.15.81:0/1992468119 wait complete. 2024-06-29T15:59:50.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-06-29T15:59:50.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.778+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3455825415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c080ceb80 msgr2=0x7f1c080b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.778+0000 7f1c0cccc640 1 --2- 172.21.15.81:0/3455825415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c080ceb80 0x7f1c080b3b00 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f1c0808c2f0 tx=0x7f1bfc02a4d0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.778+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3455825415 shutdown_connections 2024-06-29T15:59:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.778+0000 7f1c0cccc640 1 --2- 172.21.15.81:0/3455825415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c080ceb80 0x7f1c080b3b00 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.778+0000 7f1c0cccc640 1 --2- 172.21.15.81:0/3455825415 >> v2:172.21.15.167:3300/0 conn(0x7f1c080ce770 0x7f1c080b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.778+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3455825415 >> 172.21.15.81:0/3455825415 conn(0x7f1c080d3c60 msgr2=0x7f1c080d4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:50.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.778+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3455825415 shutdown_connections 2024-06-29T15:59:50.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.778+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3455825415 wait complete. 2024-06-29T15:59:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.780+0000 7f1c0cccc640 1 Processor -- start 2024-06-29T15:59:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.780+0000 7f1c0cccc640 1 -- start start 2024-06-29T15:59:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0cccc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c080ce770 0x7f1c081f6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0cccc640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c080ceb80 0x7f1c081e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0cccc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c080d2240 0x7f1c0809e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0cccc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1c080d15a0 con 0x7f1c080ce770 2024-06-29T15:59:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0cccc640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f1c080d1420 con 0x7f1c080d2240 2024-06-29T15:59:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0cccc640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f1c080d4990 con 0x7f1c080ceb80 2024-06-29T15:59:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c080ce770 0x7f1c081f6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c080ce770 0x7f1c081f6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60140/0 (socket says 172.21.15.81:60140) 2024-06-29T15:59:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0659c640 1 -- 172.21.15.81:0/3901504442 learned_addr learned my addr 172.21.15.81:0/3901504442 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c05d9b640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c080ceb80 0x7f1c081e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c06d9d640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c080d2240 0x7f1c0809e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:50.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0659c640 1 -- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c080ceb80 msgr2=0x7f1c081e5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0659c640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c080ceb80 0x7f1c081e5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0659c640 1 -- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c080d2240 msgr2=0x7f1c0809e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0659c640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c080d2240 0x7f1c0809e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0659c640 1 -- 172.21.15.81:0/3901504442 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0809edc0 con 0x7f1c080ce770 2024-06-29T15:59:50.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1c0659c640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c080ce770 0x7f1c081f6130 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000d700 tx=0x7f1c0000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:50.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.782+0000 7f1bf6ffd640 1 -- 172.21.15.81:0/3901504442 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00017070 con 0x7f1c080ce770 2024-06-29T15:59:50.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.783+0000 7f1bf6ffd640 1 -- 172.21.15.81:0/3901504442 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c00004310 con 0x7f1c080ce770 2024-06-29T15:59:50.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.783+0000 7f1bf6ffd640 1 -- 172.21.15.81:0/3901504442 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00009a80 con 0x7f1c080ce770 2024-06-29T15:59:50.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.783+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3901504442 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0809ead0 con 0x7f1c080ce770 2024-06-29T15:59:50.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.783+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3901504442 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c080a14b0 con 0x7f1c080ce770 2024-06-29T15:59:50.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.784+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3901504442 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c080d2650 con 0x7f1c080ce770 2024-06-29T15:59:50.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.784+0000 7f1bf6ffd640 1 -- 172.21.15.81:0/3901504442 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 156) ==== 98320+0+0 (secure 0 0 0) 0x7f1c00009c20 con 0x7f1c080ce770 2024-06-29T15:59:50.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.785+0000 7f1bf6ffd640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1be8076440 0x7f1be8078950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:50.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.785+0000 7f1c05d9b640 1 -- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1be8076440 msgr2=0x7f1be8078950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1824030697 2024-06-29T15:59:50.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.785+0000 7f1bf6ffd640 1 -- 172.21.15.81:0/3901504442 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f1c00097af0 con 0x7f1c080ce770 2024-06-29T15:59:50.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.785+0000 7f1c05d9b640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1be8076440 0x7f1be8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:50.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.789+0000 7f1bf6ffd640 1 -- 172.21.15.81:0/3901504442 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f1c0809ead0 con 0x7f1c080ce770 2024-06-29T15:59:50.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.979+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3901504442 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f1c080b3e20 con 0x7f1c080ce770 2024-06-29T15:59:50.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.982+0000 7f1bf6ffd640 1 -- 172.21.15.81:0/3901504442 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v85)=0 v85) ==== 138+0+0 (secure 0 0 0) 0x7f1c00060ad0 con 0x7f1c080ce770 2024-06-29T15:59:50.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1be8076440 msgr2=0x7f1be8078950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:50.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1be8076440 0x7f1be8078950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3901504442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c080ce770 msgr2=0x7f1c081f6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:50.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c080ce770 0x7f1c081f6130 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000d700 tx=0x7f1c0000dbc0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3901504442 shutdown_connections 2024-06-29T15:59:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f1be8076440 0x7f1be8078950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f1c080d2240 0x7f1c0809e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f1c080ceb80 0x7f1c081e5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 --2- 172.21.15.81:0/3901504442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c080ce770 0x7f1c081f6130 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3901504442 >> 172.21.15.81:0/3901504442 conn(0x7f1c080d3c60 msgr2=0x7f1c080b1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3901504442 shutdown_connections 2024-06-29T15:59:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:50.985+0000 7f1c0cccc640 1 -- 172.21.15.81:0/3901504442 wait complete. 2024-06-29T15:59:50.997 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-06-29T15:59:51.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.115+0000 7f9afc878640 1 -- 172.21.15.81:0/2617377145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9af80d0230 msgr2=0x7f9af80d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.115+0000 7f9afc878640 1 --2- 172.21.15.81:0/2617377145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9af80d0230 0x7f9af80d0640 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0009830 tx=0x7f9ae002a530 comp rx=0 tx=0).stop 2024-06-29T15:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.115+0000 7f9afc878640 1 -- 172.21.15.81:0/2617377145 shutdown_connections 2024-06-29T15:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.115+0000 7f9afc878640 1 --2- 172.21.15.81:0/2617377145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9af80d0230 0x7f9af80d0640 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.115+0000 7f9afc878640 1 -- 172.21.15.81:0/2617377145 >> 172.21.15.81:0/2617377145 conn(0x7f9af80b3680 msgr2=0x7f9af80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.115+0000 7f9afc878640 1 -- 172.21.15.81:0/2617377145 shutdown_connections 2024-06-29T15:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.115+0000 7f9afc878640 1 -- 172.21.15.81:0/2617377145 wait complete. 2024-06-29T15:59:51.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.117+0000 7f9afc878640 1 Processor -- start 2024-06-29T15:59:51.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.117+0000 7f9afc878640 1 -- start start 2024-06-29T15:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9afc878640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9af80b3200 0x7f9af814ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9afc878640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9af80b4f40 0x7f9af814f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9afc878640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9af80d0230 0x7f9af8153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9afc878640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9af80b5e50 con 0x7f9af80b3200 2024-06-29T15:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9afc878640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9af80b5cd0 con 0x7f9af80b4f40 2024-06-29T15:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9afc878640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9af80b5fd0 con 0x7f9af80d0230 2024-06-29T15:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9af659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9af80b3200 0x7f9af814ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9af659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9af80b3200 0x7f9af814ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60162/0 (socket says 172.21.15.81:60162) 2024-06-29T15:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9af659c640 1 -- 172.21.15.81:0/853813686 learned_addr learned my addr 172.21.15.81:0/853813686 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9aeffff640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9af80b4f40 0x7f9af814f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.118+0000 7f9af6d9d640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9af80d0230 0x7f9af8153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9af659c640 1 -- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9af80d0230 msgr2=0x7f9af8153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9af659c640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9af80d0230 0x7f9af8153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9af659c640 1 -- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9af80b4f40 msgr2=0x7f9af814f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9af659c640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9af80b4f40 0x7f9af814f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9af659c640 1 -- 172.21.15.81:0/853813686 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af8152ce0 con 0x7f9af80b3200 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9aeffff640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9af80b4f40 0x7f9af814f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9af659c640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9af80b3200 0x7f9af814ba90 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f9ae800c430 tx=0x7f9ae800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9aeeffd640 1 -- 172.21.15.81:0/853813686 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae8017070 con 0x7f9af80b3200 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9aeeffd640 1 -- 172.21.15.81:0/853813686 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ae80053c0 con 0x7f9af80b3200 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9aeeffd640 1 -- 172.21.15.81:0/853813686 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae8003850 con 0x7f9af80b3200 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9afc878640 1 -- 172.21.15.81:0/853813686 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af8152fc0 con 0x7f9af80b3200 2024-06-29T15:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.119+0000 7f9afc878640 1 -- 172.21.15.81:0/853813686 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af8153580 con 0x7f9af80b3200 2024-06-29T15:59:51.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.120+0000 7f9afc878640 1 -- 172.21.15.81:0/853813686 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9af80d0640 con 0x7f9af80b3200 2024-06-29T15:59:51.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.121+0000 7f9aeeffd640 1 -- 172.21.15.81:0/853813686 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 156) ==== 98320+0+0 (secure 0 0 0) 0x7f9ae801a070 con 0x7f9af80b3200 2024-06-29T15:59:51.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.122+0000 7f9aeeffd640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f9ad8076470 0x7f9ad8078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.122+0000 7f9aeffff640 1 -- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f9ad8076470 msgr2=0x7f9ad8078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1824030697 2024-06-29T15:59:51.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.122+0000 7f9aeeffd640 1 -- 172.21.15.81:0/853813686 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f9ae8097100 con 0x7f9af80b3200 2024-06-29T15:59:51.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.122+0000 7f9aeffff640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f9ad8076470 0x7f9ad8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.126+0000 7f9aeeffd640 1 -- 172.21.15.81:0/853813686 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9ae8021020 con 0x7f9af80b3200 2024-06-29T15:59:51.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.315+0000 7f9afc878640 1 -- 172.21.15.81:0/853813686 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f9af80b4110 con 0x7f9af80b3200 2024-06-29T15:59:51.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.319+0000 7f9aeeffd640 1 -- 172.21.15.81:0/853813686 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v86)=0 v86) ==== 133+0+0 (secure 0 0 0) 0x7f9ae80602d0 con 0x7f9af80b3200 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 -- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f9ad8076470 msgr2=0x7f9ad8078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f9ad8076470 0x7f9ad8078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 -- 172.21.15.81:0/853813686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9af80b3200 msgr2=0x7f9af814ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9af80b3200 0x7f9af814ba90 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f9ae800c430 tx=0x7f9ae800c8f0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 -- 172.21.15.81:0/853813686 shutdown_connections 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f9ad8076470 0x7f9ad8078980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9af80d0230 0x7f9af8153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9af80b4f40 0x7f9af814f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 --2- 172.21.15.81:0/853813686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9af80b3200 0x7f9af814ba90 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 -- 172.21.15.81:0/853813686 >> 172.21.15.81:0/853813686 conn(0x7f9af80b3680 msgr2=0x7f9af809c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:51.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 -- 172.21.15.81:0/853813686 shutdown_connections 2024-06-29T15:59:51.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.321+0000 7f9afc878640 1 -- 172.21.15.81:0/853813686 wait complete. 2024-06-29T15:59:51.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv01 2024-06-29T15:59:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.451+0000 7f70d74c1640 1 -- 172.21.15.81:0/2078186188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d00b5f50 msgr2=0x7f70d0166870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.451+0000 7f70d74c1640 1 --2- 172.21.15.81:0/2078186188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d00b5f50 0x7f70d0166870 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f70c8005fd0 tx=0x7f70c802a8b0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.452+0000 7f70d74c1640 1 -- 172.21.15.81:0/2078186188 shutdown_connections 2024-06-29T15:59:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.452+0000 7f70d74c1640 1 --2- 172.21.15.81:0/2078186188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d00b5f50 0x7f70d0166870 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.452+0000 7f70d74c1640 1 -- 172.21.15.81:0/2078186188 >> 172.21.15.81:0/2078186188 conn(0x7f70d00b3680 msgr2=0x7f70d00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.452+0000 7f70d74c1640 1 -- 172.21.15.81:0/2078186188 shutdown_connections 2024-06-29T15:59:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.452+0000 7f70d74c1640 1 -- 172.21.15.81:0/2078186188 wait complete. 2024-06-29T15:59:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.453+0000 7f70d74c1640 1 Processor -- start 2024-06-29T15:59:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.453+0000 7f70d74c1640 1 -- start start 2024-06-29T15:59:51.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d74c1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70d00b3200 0x7f70d00c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d74c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d00b5f50 0x7f70d014d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d74c1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d00d0230 0x7f70d01518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d74c1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01672d0 con 0x7f70d00b5f50 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d74c1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f70d0167150 con 0x7f70d00d0230 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d74c1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f70d0167450 con 0x7f70d00b3200 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d4a5c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d00b5f50 0x7f70d014d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d4a5c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d00b5f50 0x7f70d014d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60176/0 (socket says 172.21.15.81:60176) 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d4a5c640 1 -- 172.21.15.81:0/3407054505 learned_addr learned my addr 172.21.15.81:0/3407054505 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d5a5e640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d00d0230 0x7f70d01518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d525d640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70d00b3200 0x7f70d00c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d4a5c640 1 -- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70d00b3200 msgr2=0x7f70d00c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d4a5c640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f70d00b3200 0x7f70d00c8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d4a5c640 1 -- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d00d0230 msgr2=0x7f70d01518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:51.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d4a5c640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d00d0230 0x7f70d01518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:51.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d4a5c640 1 -- 172.21.15.81:0/3407054505 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d014fcd0 con 0x7f70d00b5f50 2024-06-29T15:59:51.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d5a5e640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f70d00d0230 0x7f70d01518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T15:59:51.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.455+0000 7f70d4a5c640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d00b5f50 0x7f70d014d570 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f70c0001eb0 tx=0x7f70c000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:51.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.456+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0017070 con 0x7f70d00b5f50 2024-06-29T15:59:51.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.456+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70c00053c0 con 0x7f70d00b5f50 2024-06-29T15:59:51.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.456+0000 7f70d74c1640 1 -- 172.21.15.81:0/3407054505 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d014ffb0 con 0x7f70d00b5f50 2024-06-29T15:59:51.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.456+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0003830 con 0x7f70d00b5f50 2024-06-29T15:59:51.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.456+0000 7f70d74c1640 1 -- 172.21.15.81:0/3407054505 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d0150570 con 0x7f70d00b5f50 2024-06-29T15:59:51.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.457+0000 7f70d74c1640 1 -- 172.21.15.81:0/3407054505 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d00cc2b0 con 0x7f70d00b5f50 2024-06-29T15:59:51.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.458+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 156) ==== 98320+0+0 (secure 0 0 0) 0x7f70c001a070 con 0x7f70d00b5f50 2024-06-29T15:59:51.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.459+0000 7f70c5ffb640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 0x7f70b4078980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:51.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.459+0000 7f70d525d640 1 -- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 msgr2=0x7f70b4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1824030697 2024-06-29T15:59:51.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.459+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5837+0+0 (secure 0 0 0) 0x7f70c0096d20 con 0x7f70d00b5f50 2024-06-29T15:59:51.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.459+0000 7f70d525d640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 0x7f70b4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T15:59:51.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.463+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f70d00cc2b0 con 0x7f70d00b5f50 2024-06-29T15:59:51.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.660+0000 7f70d525d640 1 -- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 msgr2=0x7f70b4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1824030697 2024-06-29T15:59:51.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.660+0000 7f70d525d640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 0x7f70b4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T15:59:51.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:51.677+0000 7f70d74c1640 1 -- 172.21.15.81:0/3407054505 --> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7f70d00cc7b0 con 0x7f70b4076470 2024-06-29T15:59:52.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:52.061+0000 7f70d525d640 1 -- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 msgr2=0x7f70b4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1824030697 2024-06-29T15:59:52.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:52.061+0000 7f70d525d640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 0x7f70b4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-29T15:59:52.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:52.862+0000 7f70d525d640 1 -- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 msgr2=0x7f70b4078980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/1824030697 2024-06-29T15:59:52.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:52.862+0000 7f70d525d640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 0x7f70b4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-29T15:59:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:54.325+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 157) ==== 97929+0+0 (secure 0 0 0) 0x7f70c00951c0 con 0x7f70d00b5f50 2024-06-29T15:59:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:54.326+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 msgr2=0x7f70b4078980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T15:59:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:54.326+0000 7f70c5ffb640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/1824030697,v1:172.21.15.167:6833/1824030697] conn(0x7f70b4076470 0x7f70b4078980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:54.470 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:54.468+0000 7fe239b84640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:54.471 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:54.468+0000 7fe239b84640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:54.471 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:54.468+0000 7fe239b84640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:54.471 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:54.468+0000 7fe239b84640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:54.471 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:54.468+0000 7fe239b84640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:54.471 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:54.469+0000 7fe23cb8a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:54.471 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:54.469+0000 7fe23cb8a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:54.471 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:54.469+0000 7fe23cb8a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:54.471 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:54.469+0000 7fe23cb8a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:54.471 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:54.469+0000 7fe23cb8a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:55.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.344+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 158) ==== 98056+0+0 (secure 0 0 0) 0x7f70c005c130 con 0x7f70d00b5f50 2024-06-29T15:59:55.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.344+0000 7f70c5ffb640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f70b403ad60 0x7f70b4080520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:55.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.344+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 --> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7f70c0096b70 con 0x7f70b403ad60 2024-06-29T15:59:55.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.345+0000 7f70d525d640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f70b403ad60 0x7f70b4080520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:55.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.349+0000 7f70d525d640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f70b403ad60 0x7f70b4080520 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f70cc001150 tx=0x7f70cc0010e0 comp rx=0 tx=0).ready entity=mgr.10283 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:55.358 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:55.357+0000 7fe236b7e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:55.358 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:55.357+0000 7fe236b7e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:55.359 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:55.357+0000 7fe236b7e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:55.359 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:55.357+0000 7fe236b7e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:55.359 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T15:59:55.357+0000 7fe236b7e640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T15:59:55.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.391+0000 7f70c5ffb640 1 -- 172.21.15.81:0/3407054505 <== mgr.10283 v2:172.21.15.167:6832/2899520759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f70c0096b70 con 0x7f70b403ad60 2024-06-29T15:59:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.393+0000 7f70d74c1640 1 -- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f70b403ad60 msgr2=0x7f70b4080520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.393+0000 7f70d74c1640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f70b403ad60 0x7f70b4080520 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f70cc001150 tx=0x7f70cc0010e0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.393+0000 7f70d74c1640 1 -- 172.21.15.81:0/3407054505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d00b5f50 msgr2=0x7f70d014d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.393+0000 7f70d74c1640 1 --2- 172.21.15.81:0/3407054505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70d00b5f50 0x7f70d014d570 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f70c0001eb0 tx=0x7f70c000c890 comp rx=0 tx=0).stop 2024-06-29T15:59:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.393+0000 7f70d5a5e640 1 -- 172.21.15.81:0/3407054505 reap_dead start 2024-06-29T15:59:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.393+0000 7f70d74c1640 1 -- 172.21.15.81:0/3407054505 shutdown_connections 2024-06-29T15:59:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.393+0000 7f70d74c1640 1 -- 172.21.15.81:0/3407054505 >> 172.21.15.81:0/3407054505 conn(0x7f70d00b3680 msgr2=0x7f70d00b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.393+0000 7f70d74c1640 1 -- 172.21.15.81:0/3407054505 shutdown_connections 2024-06-29T15:59:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.393+0000 7f70d74c1640 1 -- 172.21.15.81:0/3407054505 wait complete. 2024-06-29T15:59:55.406 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add --subvol sv01 . 1m 2024-06-29T15:59:55.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.526+0000 7f78e16a4640 1 -- 172.21.15.81:0/2091225634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78c8003a50 msgr2=0x7f78c8003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.526+0000 7f78e16a4640 1 --2- 172.21.15.81:0/2091225634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78c8003a50 0x7f78c8003ec0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00c630 tx=0x7f78c8005830 comp rx=0 tx=0).stop 2024-06-29T15:59:55.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.526+0000 7f78e16a4640 1 -- 172.21.15.81:0/2091225634 shutdown_connections 2024-06-29T15:59:55.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.526+0000 7f78e16a4640 1 --2- 172.21.15.81:0/2091225634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78c8003a50 0x7f78c8003ec0 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.527+0000 7f78e16a4640 1 -- 172.21.15.81:0/2091225634 >> 172.21.15.81:0/2091225634 conn(0x7f78dc0b3680 msgr2=0x7f78dc0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:55.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.527+0000 7f78e16a4640 1 -- 172.21.15.81:0/2091225634 shutdown_connections 2024-06-29T15:59:55.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.527+0000 7f78e16a4640 1 -- 172.21.15.81:0/2091225634 wait complete. 2024-06-29T15:59:55.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.528+0000 7f78e16a4640 1 Processor -- start 2024-06-29T15:59:55.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.529+0000 7f78e16a4640 1 -- start start 2024-06-29T15:59:55.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78e16a4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78c8003a50 0x7f78dc06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:55.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78e16a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78dc153ba0 0x7f78dc1550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:55.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78e16a4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78dc153fb0 0x7f78dc1f8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:55.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78e16a4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc0b4d10 con 0x7f78dc153ba0 2024-06-29T15:59:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78e16a4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc0b4b90 con 0x7f78c8003a50 2024-06-29T15:59:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78e16a4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f78dc15fe70 con 0x7f78dc153fb0 2024-06-29T15:59:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78da7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78dc153ba0 0x7f78dc1550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78da7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78dc153ba0 0x7f78dc1550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60192/0 (socket says 172.21.15.81:60192) 2024-06-29T15:59:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78da7fc640 1 -- 172.21.15.81:0/1346478265 learned_addr learned my addr 172.21.15.81:0/1346478265 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T15:59:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78db7fe640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78dc153fb0 0x7f78dc1f8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78daffd640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78c8003a50 0x7f78dc06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78da7fc640 1 -- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78dc153fb0 msgr2=0x7f78dc1f8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78da7fc640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78dc153fb0 0x7f78dc1f8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78da7fc640 1 -- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78c8003a50 msgr2=0x7f78dc06d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78da7fc640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78c8003a50 0x7f78dc06d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78da7fc640 1 -- 172.21.15.81:0/1346478265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78dc152b00 con 0x7f78dc153ba0 2024-06-29T15:59:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.530+0000 7f78da7fc640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78dc153ba0 0x7f78dc1550d0 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f78c4011ba0 tx=0x7f78c400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.531+0000 7f78d37fe640 1 -- 172.21.15.81:0/1346478265 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c401e070 con 0x7f78dc153ba0 2024-06-29T15:59:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.531+0000 7f78d37fe640 1 -- 172.21.15.81:0/1346478265 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78c400e7c0 con 0x7f78dc153ba0 2024-06-29T15:59:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.531+0000 7f78e16a4640 1 -- 172.21.15.81:0/1346478265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78dc155bb0 con 0x7f78dc153ba0 2024-06-29T15:59:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.531+0000 7f78d37fe640 1 -- 172.21.15.81:0/1346478265 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c400ebd0 con 0x7f78dc153ba0 2024-06-29T15:59:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.531+0000 7f78e16a4640 1 -- 172.21.15.81:0/1346478265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78dc1530c0 con 0x7f78dc153ba0 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.532+0000 7f78e16a4640 1 -- 172.21.15.81:0/1346478265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78dc074900 con 0x7f78dc153ba0 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.532+0000 7f78d37fe640 1 -- 172.21.15.81:0/1346478265 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 158) ==== 98056+0+0 (secure 0 0 0) 0x7f78c4021070 con 0x7f78dc153ba0 2024-06-29T15:59:55.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.533+0000 7f78d37fe640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f78bc0761a0 0x7f78bc0786b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T15:59:55.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.533+0000 7f78d37fe640 1 -- 172.21.15.81:0/1346478265 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(267..267 src has 1..267) ==== 6181+0+0 (secure 0 0 0) 0x7f78c4098560 con 0x7f78dc153ba0 2024-06-29T15:59:55.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.533+0000 7f78daffd640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f78bc0761a0 0x7f78bc0786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T15:59:55.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.534+0000 7f78daffd640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f78bc0761a0 0x7f78bc0786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f78cc033000 tx=0x7f78cc036040 comp rx=0 tx=0).ready entity=mgr.10283 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T15:59:55.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.537+0000 7f78d37fe640 1 -- 172.21.15.81:0/1346478265 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f78dc074900 con 0x7f78dc153ba0 2024-06-29T15:59:55.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.740+0000 7f78e16a4640 1 -- 172.21.15.81:0/1346478265 --> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "subvol": "sv01", "path": ".", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f78dc0707b0 con 0x7f78bc0761a0 2024-06-29T15:59:55.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.826+0000 7f78d37fe640 1 -- 172.21.15.81:0/1346478265 <== mgr.10283 v2:172.21.15.167:6832/2899520759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+85 (secure 0 0 0) 0x7f78dc0707b0 con 0x7f78bc0761a0 2024-06-29T15:59:55.827 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path /volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23/.. 2024-06-29T15:59:55.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.828+0000 7f78e16a4640 1 -- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f78bc0761a0 msgr2=0x7f78bc0786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.828+0000 7f78e16a4640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f78bc0761a0 0x7f78bc0786b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f78cc033000 tx=0x7f78cc036040 comp rx=0 tx=0).stop 2024-06-29T15:59:55.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.828+0000 7f78e16a4640 1 -- 172.21.15.81:0/1346478265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78dc153ba0 msgr2=0x7f78dc1550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T15:59:55.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.828+0000 7f78e16a4640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78dc153ba0 0x7f78dc1550d0 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f78c4011ba0 tx=0x7f78c400f790 comp rx=0 tx=0).stop 2024-06-29T15:59:55.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.828+0000 7f78e16a4640 1 -- 172.21.15.81:0/1346478265 shutdown_connections 2024-06-29T15:59:55.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.828+0000 7f78e16a4640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f78bc0761a0 0x7f78bc0786b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.828+0000 7f78e16a4640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f78dc153fb0 0x7f78dc1f8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.828+0000 7f78e16a4640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78dc153ba0 0x7f78dc1550d0 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.828+0000 7f78e16a4640 1 --2- 172.21.15.81:0/1346478265 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f78c8003a50 0x7f78dc06d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T15:59:55.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.828+0000 7f78e16a4640 1 -- 172.21.15.81:0/1346478265 >> 172.21.15.81:0/1346478265 conn(0x7f78dc0b3680 msgr2=0x7f78dc15bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T15:59:55.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.829+0000 7f78e16a4640 1 -- 172.21.15.81:0/1346478265 shutdown_connections 2024-06-29T15:59:55.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T15:59:55.829+0000 7f78e16a4640 1 -- 172.21.15.81:0/1346478265 wait complete. 2024-06-29T15:59:55.840 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23/.. 2024-06-29T16:00:00.838 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:00:00.838+0000 7fe227b20640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:00:00.839 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:00:00.838+0000 7fe227b20640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:00:00.839 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:00:00.838+0000 7fe227b20640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:00:00.839 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:00:00.838+0000 7fe227b20640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:00:00.839 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:00:00.838+0000 7fe227b20640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:00:04.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:00:04.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:00:34.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:00:34.449 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:04.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:04.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:10.842 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv01 2024-06-29T16:01:10.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.971+0000 7f9ca83bd640 1 -- 172.21.15.81:0/2015183727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ca00d0b90 msgr2=0x7f9ca00d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:10.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.971+0000 7f9ca83bd640 1 --2- 172.21.15.81:0/2015183727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ca00d0b90 0x7f9ca00d1a80 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f9ca008e330 tx=0x7f9c9002a5d0 comp rx=0 tx=0).stop 2024-06-29T16:01:10.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.971+0000 7f9ca83bd640 1 -- 172.21.15.81:0/2015183727 shutdown_connections 2024-06-29T16:01:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.971+0000 7f9ca83bd640 1 --2- 172.21.15.81:0/2015183727 >> v2:172.21.15.167:3300/0 conn(0x7f9ca00b5250 0x7f9ca00d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.971+0000 7f9ca83bd640 1 --2- 172.21.15.81:0/2015183727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ca00d0b90 0x7f9ca00d1a80 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.971+0000 7f9ca83bd640 1 -- 172.21.15.81:0/2015183727 >> 172.21.15.81:0/2015183727 conn(0x7f9ca009cf30 msgr2=0x7f9ca00d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.972+0000 7f9ca83bd640 1 -- 172.21.15.81:0/2015183727 shutdown_connections 2024-06-29T16:01:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.972+0000 7f9ca83bd640 1 -- 172.21.15.81:0/2015183727 wait complete. 2024-06-29T16:01:10.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.973+0000 7f9ca83bd640 1 Processor -- start 2024-06-29T16:01:10.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.973+0000 7f9ca83bd640 1 -- start start 2024-06-29T16:01:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca83bd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ca00b5250 0x7f9ca01f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca83bd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca00b9040 0x7f9ca01f81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca83bd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca01f69b0 0x7f9ca01f6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca83bd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca00d2500 con 0x7f9ca00b5250 2024-06-29T16:01:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca83bd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca00b8530 con 0x7f9ca01f69b0 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca83bd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca00d2680 con 0x7f9ca00b9040 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca6159640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ca00b5250 0x7f9ca01f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca6159640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ca00b5250 0x7f9ca01f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35994/0 (socket says 172.21.15.81:35994) 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca6159640 1 -- 172.21.15.81:0/70406593 learned_addr learned my addr 172.21.15.81:0/70406593 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca5958640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca00b9040 0x7f9ca01f81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca695a640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca01f69b0 0x7f9ca01f6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca6159640 1 -- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca00b9040 msgr2=0x7f9ca01f81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca6159640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca00b9040 0x7f9ca01f81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca6159640 1 -- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca01f69b0 msgr2=0x7f9ca01f6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca6159640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca01f69b0 0x7f9ca01f6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca6159640 1 -- 172.21.15.81:0/70406593 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca01f7a30 con 0x7f9ca00b5250 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca5958640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca00b9040 0x7f9ca01f81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca695a640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca01f69b0 0x7f9ca01f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca6159640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ca00b5250 0x7f9ca01f7ca0 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c00c3f0 tx=0x7f9c9c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9c96ffd640 1 -- 172.21.15.81:0/70406593 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c017070 con 0x7f9ca00b5250 2024-06-29T16:01:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9c96ffd640 1 -- 172.21.15.81:0/70406593 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c9c0035f0 con 0x7f9ca00b5250 2024-06-29T16:01:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca83bd640 1 -- 172.21.15.81:0/70406593 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca01f5910 con 0x7f9ca00b5250 2024-06-29T16:01:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9c96ffd640 1 -- 172.21.15.81:0/70406593 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c003a40 con 0x7f9ca00b5250 2024-06-29T16:01:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.975+0000 7f9ca83bd640 1 -- 172.21.15.81:0/70406593 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca01f5ed0 con 0x7f9ca00b5250 2024-06-29T16:01:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.977+0000 7f9ca83bd640 1 -- 172.21.15.81:0/70406593 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca00d2c10 con 0x7f9ca00b5250 2024-06-29T16:01:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.977+0000 7f9c96ffd640 1 -- 172.21.15.81:0/70406593 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 161) ==== 98485+0+0 (secure 0 0 0) 0x7f9c9c01a070 con 0x7f9ca00b5250 2024-06-29T16:01:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.978+0000 7f9c96ffd640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f9c840765f0 0x7f9c84078b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.978+0000 7f9c96ffd640 1 -- 172.21.15.81:0/70406593 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(269..269 src has 1..269) ==== 6210+0+0 (secure 0 0 0) 0x7f9c9c0972a0 con 0x7f9ca00b5250 2024-06-29T16:01:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.978+0000 7f9ca5958640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f9c840765f0 0x7f9c84078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.979+0000 7f9ca5958640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f9c840765f0 0x7f9c84078b00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9c90034000 tx=0x7f9c90004730 comp rx=0 tx=0).ready entity=mgr.10283 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:10.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:10.983+0000 7f9c96ffd640 1 -- 172.21.15.81:0/70406593 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f9c9c0605d0 con 0x7f9ca00b5250 2024-06-29T16:01:11.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.191+0000 7f9ca83bd640 1 -- 172.21.15.81:0/70406593 --> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7f9ca00d0960 con 0x7f9c840765f0 2024-06-29T16:01:11.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.199+0000 7f9c96ffd640 1 -- 172.21.15.81:0/70406593 <== mgr.10283 v2:172.21.15.167:6832/2899520759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+60 (secure 0 0 0) 0x7f9ca00d0960 con 0x7f9c840765f0 2024-06-29T16:01:11.200 INFO:teuthology.orchestra.run.smithi081.stdout:/volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23 2024-06-29T16:01:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.200+0000 7f9ca83bd640 1 -- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f9c840765f0 msgr2=0x7f9c84078b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.200+0000 7f9ca83bd640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f9c840765f0 0x7f9c84078b00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9c90034000 tx=0x7f9c90004730 comp rx=0 tx=0).stop 2024-06-29T16:01:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.200+0000 7f9ca83bd640 1 -- 172.21.15.81:0/70406593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ca00b5250 msgr2=0x7f9ca01f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.200+0000 7f9ca83bd640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ca00b5250 0x7f9ca01f7ca0 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c00c3f0 tx=0x7f9c9c00c8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.201+0000 7f9ca83bd640 1 -- 172.21.15.81:0/70406593 shutdown_connections 2024-06-29T16:01:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.201+0000 7f9ca83bd640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f9c840765f0 0x7f9c84078b00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.201+0000 7f9ca83bd640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9ca01f69b0 0x7f9ca01f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.201+0000 7f9ca83bd640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9ca00b9040 0x7f9ca01f81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.201+0000 7f9ca83bd640 1 --2- 172.21.15.81:0/70406593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ca00b5250 0x7f9ca01f7ca0 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.201+0000 7f9ca83bd640 1 -- 172.21.15.81:0/70406593 >> 172.21.15.81:0/70406593 conn(0x7f9ca009cf30 msgr2=0x7f9ca006e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.201+0000 7f9ca83bd640 1 -- 172.21.15.81:0/70406593 shutdown_connections 2024-06-29T16:01:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.201+0000 7f9ca83bd640 1 -- 172.21.15.81:0/70406593 wait complete. 2024-06-29T16:01:11.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:11.213 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23/../.snap) 2024-06-29T16:01:11.231 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-16_00_00_UTC 2024-06-29T16:01:11.231 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-16_01_00_UTC 2024-06-29T16:01:11.231 DEBUG:tasks.cephfs.test_snap_schedules:snaps:['scheduled-2024-06-29-16_00_00_UTC', 'scheduled-2024-06-29-16_01_00_UTC'] 2024-06-29T16:01:11.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove --subvol sv01 . 1m 2024-06-29T16:01:11.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.388+0000 7f824f5a1640 1 -- 172.21.15.81:0/3630833582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82480d1410 msgr2=0x7f82480b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:11.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.388+0000 7f824f5a1640 1 --2- 172.21.15.81:0/3630833582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82480d1410 0x7f82480b3960 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f8248070ec0 tx=0x7f823802aa30 comp rx=0 tx=0).stop 2024-06-29T16:01:11.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.388+0000 7f824f5a1640 1 -- 172.21.15.81:0/3630833582 shutdown_connections 2024-06-29T16:01:11.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.388+0000 7f824f5a1640 1 --2- 172.21.15.81:0/3630833582 >> v2:172.21.15.167:3300/0 conn(0x7f824815e200 0x7f82480b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.388+0000 7f824f5a1640 1 --2- 172.21.15.81:0/3630833582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82480d1410 0x7f82480b3960 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.388+0000 7f824f5a1640 1 -- 172.21.15.81:0/3630833582 >> 172.21.15.81:0/3630833582 conn(0x7f82480d08f0 msgr2=0x7f82480d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:11.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.388+0000 7f824f5a1640 1 -- 172.21.15.81:0/3630833582 shutdown_connections 2024-06-29T16:01:11.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.388+0000 7f824f5a1640 1 -- 172.21.15.81:0/3630833582 wait complete. 2024-06-29T16:01:11.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.389+0000 7f824f5a1640 1 Processor -- start 2024-06-29T16:01:11.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.389+0000 7f824f5a1640 1 -- start start 2024-06-29T16:01:11.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824f5a1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82480b5770 0x7f82480b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:11.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824f5a1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82480d1410 0x7f824814b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:11.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824f5a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824815e200 0x7f82481f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:11.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824f5a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82480ada50 con 0x7f824815e200 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824f5a1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f82480ad8d0 con 0x7f82480b5770 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824f5a1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f82480adbd0 con 0x7f82480d1410 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824db3e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824815e200 0x7f82481f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824db3e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824815e200 0x7f82481f5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36012/0 (socket says 172.21.15.81:36012) 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824db3e640 1 -- 172.21.15.81:0/1162293975 learned_addr learned my addr 172.21.15.81:0/1162293975 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824d33d640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82480b5770 0x7f82480b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824cb3c640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82480d1410 0x7f824814b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824db3e640 1 -- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82480d1410 msgr2=0x7f824814b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824db3e640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82480d1410 0x7f824814b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824db3e640 1 -- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82480b5770 msgr2=0x7f82480b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824db3e640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82480b5770 0x7f82480b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824db3e640 1 -- 172.21.15.81:0/1162293975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82480af520 con 0x7f824815e200 2024-06-29T16:01:11.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f824db3e640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824815e200 0x7f82481f5180 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f824000c3c0 tx=0x7f824000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.391+0000 7f8245ffb640 1 -- 172.21.15.81:0/1162293975 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240017070 con 0x7f824815e200 2024-06-29T16:01:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.392+0000 7f8245ffb640 1 -- 172.21.15.81:0/1162293975 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82400035f0 con 0x7f824815e200 2024-06-29T16:01:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.392+0000 7f824f5a1640 1 -- 172.21.15.81:0/1162293975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82480af800 con 0x7f824815e200 2024-06-29T16:01:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.392+0000 7f8245ffb640 1 -- 172.21.15.81:0/1162293975 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240003a40 con 0x7f824815e200 2024-06-29T16:01:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.392+0000 7f824f5a1640 1 -- 172.21.15.81:0/1162293975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82480afdc0 con 0x7f824815e200 2024-06-29T16:01:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.393+0000 7f824f5a1640 1 -- 172.21.15.81:0/1162293975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82480af290 con 0x7f824815e200 2024-06-29T16:01:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.393+0000 7f8245ffb640 1 -- 172.21.15.81:0/1162293975 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 161) ==== 98485+0+0 (secure 0 0 0) 0x7f824001a070 con 0x7f824815e200 2024-06-29T16:01:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.394+0000 7f8245ffb640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f82340765f0 0x7f8234078b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.394+0000 7f8245ffb640 1 -- 172.21.15.81:0/1162293975 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(269..269 src has 1..269) ==== 6210+0+0 (secure 0 0 0) 0x7f8240097080 con 0x7f824815e200 2024-06-29T16:01:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.394+0000 7f824d33d640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f82340765f0 0x7f8234078b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.395+0000 7f824d33d640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f82340765f0 0x7f8234078b00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8238035000 tx=0x7f8238004ab0 comp rx=0 tx=0).ready entity=mgr.10283 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:11.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.398+0000 7f8245ffb640 1 -- 172.21.15.81:0/1162293975 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8240060490 con 0x7f824815e200 2024-06-29T16:01:11.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.601+0000 7f824f5a1640 1 -- 172.21.15.81:0/1162293975 --> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "subvol": "sv01", "path": ".", "repeat": "1m", "target": ["mon-mgr", ""]}) -- 0x7f8248075c90 con 0x7f82340765f0 2024-06-29T16:01:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.684+0000 7f8245ffb640 1 -- 172.21.15.81:0/1162293975 <== mgr.10283 v2:172.21.15.167:6832/2899520759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+89 (secure 0 0 0) 0x7f8248075c90 con 0x7f82340765f0 2024-06-29T16:01:11.686 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule removed for path /volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23/.. 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 -- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f82340765f0 msgr2=0x7f8234078b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f82340765f0 0x7f8234078b00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8238035000 tx=0x7f8238004ab0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 -- 172.21.15.81:0/1162293975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824815e200 msgr2=0x7f82481f5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824815e200 0x7f82481f5180 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f824000c3c0 tx=0x7f824000c880 comp rx=0 tx=0).stop 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 -- 172.21.15.81:0/1162293975 shutdown_connections 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f82340765f0 0x7f8234078b00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824815e200 0x7f82481f5180 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82480d1410 0x7f824814b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 --2- 172.21.15.81:0/1162293975 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82480b5770 0x7f82480b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 -- 172.21.15.81:0/1162293975 >> 172.21.15.81:0/1162293975 conn(0x7f82480d08f0 msgr2=0x7f824806c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:11.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 -- 172.21.15.81:0/1162293975 shutdown_connections 2024-06-29T16:01:11.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.686+0000 7f824f5a1640 1 -- 172.21.15.81:0/1162293975 wait complete. 2024-06-29T16:01:11.698 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23/.. 2024-06-29T16:01:11.698 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv01 2024-06-29T16:01:11.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.813+0000 7f8295064640 1 -- 172.21.15.81:0/3342717260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8290145530 msgr2=0x7f8290073280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:11.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.813+0000 7f8295064640 1 --2- 172.21.15.81:0/3342717260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8290145530 0x7f8290073280 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f8280005fd0 tx=0x7f828002a5f0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.813+0000 7f8295064640 1 -- 172.21.15.81:0/3342717260 shutdown_connections 2024-06-29T16:01:11.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.813+0000 7f8295064640 1 --2- 172.21.15.81:0/3342717260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8290145530 0x7f8290073280 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.813+0000 7f8295064640 1 -- 172.21.15.81:0/3342717260 >> 172.21.15.81:0/3342717260 conn(0x7f829006ccc0 msgr2=0x7f82900b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:11.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.813+0000 7f8295064640 1 -- 172.21.15.81:0/3342717260 shutdown_connections 2024-06-29T16:01:11.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.813+0000 7f8295064640 1 -- 172.21.15.81:0/3342717260 wait complete. 2024-06-29T16:01:11.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.815+0000 7f8295064640 1 Processor -- start 2024-06-29T16:01:11.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.815+0000 7f8295064640 1 -- start start 2024-06-29T16:01:11.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.816+0000 7f8295064640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82900737c0 0x7f82901f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:11.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.816+0000 7f8295064640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8290145530 0x7f8290157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:11.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.816+0000 7f8295064640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f829015cd00 0x7f8290135cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:11.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.816+0000 7f8295064640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f829015dfb0 con 0x7f82900737c0 2024-06-29T16:01:11.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.816+0000 7f8295064640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f829015de30 con 0x7f8290145530 2024-06-29T16:01:11.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.816+0000 7f8295064640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f829015e130 con 0x7f829015cd00 2024-06-29T16:01:11.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.816+0000 7f828ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82900737c0 0x7f82901f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:11.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.816+0000 7f828ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82900737c0 0x7f82901f5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36026/0 (socket says 172.21.15.81:36026) 2024-06-29T16:01:11.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.816+0000 7f828ed9d640 1 -- 172.21.15.81:0/2634484347 learned_addr learned my addr 172.21.15.81:0/2634484347 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:11.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.816+0000 7f828e59c640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8290145530 0x7f8290157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f828f59e640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f829015cd00 0x7f8290135cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f828ed9d640 1 -- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f829015cd00 msgr2=0x7f8290135cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f828ed9d640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f829015cd00 0x7f8290135cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f828ed9d640 1 -- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8290145530 msgr2=0x7f8290157f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f828ed9d640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8290145530 0x7f8290157f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f828ed9d640 1 -- 172.21.15.81:0/2634484347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82900b3dc0 con 0x7f82900737c0 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f828ed9d640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82900737c0 0x7f82901f5b80 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f8280006010 tx=0x7f82800039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f82777fe640 1 -- 172.21.15.81:0/2634484347 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828003e030 con 0x7f82900737c0 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f82777fe640 1 -- 172.21.15.81:0/2634484347 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82800047a0 con 0x7f82900737c0 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f82777fe640 1 -- 172.21.15.81:0/2634484347 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280035040 con 0x7f82900737c0 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f8295064640 1 -- 172.21.15.81:0/2634484347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82900b40a0 con 0x7f82900737c0 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.817+0000 7f8295064640 1 -- 172.21.15.81:0/2634484347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82900b4660 con 0x7f82900737c0 2024-06-29T16:01:11.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.818+0000 7f8295064640 1 -- 172.21.15.81:0/2634484347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8290073280 con 0x7f82900737c0 2024-06-29T16:01:11.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.819+0000 7f82777fe640 1 -- 172.21.15.81:0/2634484347 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 161) ==== 98485+0+0 (secure 0 0 0) 0x7f8280041030 con 0x7f82900737c0 2024-06-29T16:01:11.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.819+0000 7f82777fe640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f827c076620 0x7f827c078b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:11.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.819+0000 7f82777fe640 1 -- 172.21.15.81:0/2634484347 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(269..269 src has 1..269) ==== 6210+0+0 (secure 0 0 0) 0x7f82800b8820 con 0x7f82900737c0 2024-06-29T16:01:11.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.819+0000 7f828e59c640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f827c076620 0x7f827c078b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:11.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.820+0000 7f828e59c640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f827c076620 0x7f827c078b30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f829015d5f0 tx=0x7f8284009040 comp rx=0 tx=0).ready entity=mgr.10283 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:11.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:11.824+0000 7f82777fe640 1 -- 172.21.15.81:0/2634484347 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f8280081a10 con 0x7f82900737c0 2024-06-29T16:01:12.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.034+0000 7f8295064640 1 -- 172.21.15.81:0/2634484347 --> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7f829015d620 con 0x7f827c076620 2024-06-29T16:01:12.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.039+0000 7f82777fe640 1 -- 172.21.15.81:0/2634484347 <== mgr.10283 v2:172.21.15.167:6832/2899520759 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+60 (secure 0 0 0) 0x7f829015d620 con 0x7f827c076620 2024-06-29T16:01:12.041 INFO:teuthology.orchestra.run.smithi081.stdout:/volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23 2024-06-29T16:01:12.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 -- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f827c076620 msgr2=0x7f827c078b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:12.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f827c076620 0x7f827c078b30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f829015d5f0 tx=0x7f8284009040 comp rx=0 tx=0).stop 2024-06-29T16:01:12.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 -- 172.21.15.81:0/2634484347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82900737c0 msgr2=0x7f82901f5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:12.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82900737c0 0x7f82901f5b80 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f8280006010 tx=0x7f82800039e0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 -- 172.21.15.81:0/2634484347 shutdown_connections 2024-06-29T16:01:12.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f827c076620 0x7f827c078b30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f829015cd00 0x7f8290135cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f8290145530 0x7f8290157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 --2- 172.21.15.81:0/2634484347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82900737c0 0x7f82901f5b80 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 -- 172.21.15.81:0/2634484347 >> 172.21.15.81:0/2634484347 conn(0x7f829006ccc0 msgr2=0x7f8290156200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:12.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 -- 172.21.15.81:0/2634484347 shutdown_connections 2024-06-29T16:01:12.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.041+0000 7f8295064640 1 -- 172.21.15.81:0/2634484347 wait complete. 2024-06-29T16:01:12.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:12.054 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23/../.snap) 2024-06-29T16:01:12.069 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-16_00_00_UTC 2024-06-29T16:01:12.069 INFO:teuthology.orchestra.run.smithi081.stdout:scheduled-2024-06-29-16_01_00_UTC 2024-06-29T16:01:12.070 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23/../.snap/scheduled-2024-06-29-16_00_00_UTC 2024-06-29T16:01:12.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:12.070 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rmdir volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23/../.snap/scheduled-2024-06-29-16_00_00_UTC) 2024-06-29T16:01:12.141 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23/../.snap/scheduled-2024-06-29-16_01_00_UTC 2024-06-29T16:01:12.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:12.141 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo rmdir volumes/_nogroup/sv01/0106c66b-ad02-40ea-b8c7-ba9be221dd23/../.snap/scheduled-2024-06-29-16_01_00_UTC) 2024-06-29T16:01:12.212 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv01 2024-06-29T16:01:12.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.367+0000 7f8365612640 1 -- 172.21.15.81:0/132824667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f834c004d40 msgr2=0x7f834c0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:12.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.367+0000 7f8365612640 1 --2- 172.21.15.81:0/132824667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f834c004d40 0x7f834c0051f0 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f835800c7a0 tx=0x7f834c005830 comp rx=0 tx=0).stop 2024-06-29T16:01:12.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.368+0000 7f835f7fe640 1 -- 172.21.15.81:0/132824667 reap_dead start 2024-06-29T16:01:12.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.368+0000 7f8365612640 1 -- 172.21.15.81:0/132824667 shutdown_connections 2024-06-29T16:01:12.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.368+0000 7f8365612640 1 -- 172.21.15.81:0/132824667 >> 172.21.15.81:0/132824667 conn(0x7f83600b3680 msgr2=0x7f83600b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:12.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.368+0000 7f8365612640 1 -- 172.21.15.81:0/132824667 shutdown_connections 2024-06-29T16:01:12.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.368+0000 7f8365612640 1 -- 172.21.15.81:0/132824667 wait complete. 2024-06-29T16:01:12.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.369+0000 7f8365612640 1 Processor -- start 2024-06-29T16:01:12.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.370+0000 7f8365612640 1 -- start start 2024-06-29T16:01:12.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f8365612640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f834c004d40 0x7f836006d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:12.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f8365612640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83601e7890 0x7f83601e1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:12.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f8365612640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83600c76c0 0x7f83600c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f8365612640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f83600d09d0 con 0x7f834c004d40 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f8365612640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f83600d0850 con 0x7f83601e7890 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f8365612640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f83600b5a60 con 0x7f83600c76c0 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f835effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f834c004d40 0x7f836006d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f835effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f834c004d40 0x7f836006d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36056/0 (socket says 172.21.15.81:36056) 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f835effd640 1 -- 172.21.15.81:0/220924923 learned_addr learned my addr 172.21.15.81:0/220924923 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f835effd640 1 -- 172.21.15.81:0/220924923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83600c76c0 msgr2=0x7f83600c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f835effd640 1 --2- 172.21.15.81:0/220924923 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f83600c76c0 0x7f83600c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f83567fc640 1 --2- 172.21.15.81:0/220924923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83601e7890 0x7f83601e1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f835effd640 1 -- 172.21.15.81:0/220924923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83601e7890 msgr2=0x7f83601e1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f835effd640 1 --2- 172.21.15.81:0/220924923 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f83601e7890 0x7f83601e1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f835effd640 1 -- 172.21.15.81:0/220924923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83600c8010 con 0x7f834c004d40 2024-06-29T16:01:12.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.371+0000 7f835effd640 1 --2- 172.21.15.81:0/220924923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f834c004d40 0x7f836006d400 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7f8350004e60 tx=0x7f83500015d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:12.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.372+0000 7f8357fff640 1 -- 172.21.15.81:0/220924923 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350015020 con 0x7f834c004d40 2024-06-29T16:01:12.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.372+0000 7f8357fff640 1 -- 172.21.15.81:0/220924923 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8350011960 con 0x7f834c004d40 2024-06-29T16:01:12.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.372+0000 7f8365612640 1 -- 172.21.15.81:0/220924923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83600c8bc0 con 0x7f834c004d40 2024-06-29T16:01:12.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.372+0000 7f8357fff640 1 -- 172.21.15.81:0/220924923 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350001d40 con 0x7f834c004d40 2024-06-29T16:01:12.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.372+0000 7f8365612640 1 -- 172.21.15.81:0/220924923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f836006dd00 con 0x7f834c004d40 2024-06-29T16:01:12.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.373+0000 7f8365612640 1 -- 172.21.15.81:0/220924923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83600b5f50 con 0x7f834c004d40 2024-06-29T16:01:12.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.373+0000 7f8357fff640 1 -- 172.21.15.81:0/220924923 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 161) ==== 98485+0+0 (secure 0 0 0) 0x7f8350018020 con 0x7f834c004d40 2024-06-29T16:01:12.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.374+0000 7f8357fff640 1 --2- 172.21.15.81:0/220924923 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f8340076620 0x7f8340078b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:12.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.374+0000 7f8357fff640 1 -- 172.21.15.81:0/220924923 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(269..269 src has 1..269) ==== 6210+0+0 (secure 0 0 0) 0x7f8350098940 con 0x7f834c004d40 2024-06-29T16:01:12.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.374+0000 7f83567fc640 1 --2- 172.21.15.81:0/220924923 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f8340076620 0x7f8340078b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:12.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.374+0000 7f83567fc640 1 --2- 172.21.15.81:0/220924923 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f8340076620 0x7f8340078b30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f83601e8110 tx=0x7f8348005f90 comp rx=0 tx=0).ready entity=mgr.10283 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:12.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.378+0000 7f8357fff640 1 -- 172.21.15.81:0/220924923 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f835001f020 con 0x7f834c004d40 2024-06-29T16:01:12.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.587+0000 7f8365612640 1 -- 172.21.15.81:0/220924923 --> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv01", "target": ["mon-mgr", ""]}) -- 0x7f83601e8700 con 0x7f8340076620 2024-06-29T16:01:12.600 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:12.600+0000 7fe236b7e640 -1 mgr.server reply reply (22) Invalid argument error in rename /volumes/_nogroup/sv01 to /volumes/_nogroup/sv01/.trash/90980159-fa92-4b92-9561-57452a6331d9 2024-06-29T16:01:12.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.599+0000 7f8357fff640 1 -- 172.21.15.81:0/220924923 <== mgr.10283 v2:172.21.15.167:6832/2899520759 1 ==== mgr_command_reply(tid 0: -22 error in rename /volumes/_nogroup/sv01 to /volumes/_nogroup/sv01/.trash/90980159-fa92-4b92-9561-57452a6331d9) ==== 116+0+0 (secure 0 0 0) 0x7f83601e8700 con 0x7f8340076620 2024-06-29T16:01:12.601 INFO:teuthology.orchestra.run.smithi081.stderr:Error EINVAL: error in rename /volumes/_nogroup/sv01 to /volumes/_nogroup/sv01/.trash/90980159-fa92-4b92-9561-57452a6331d9 2024-06-29T16:01:12.605 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-29T16:01:12.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2024-06-29T16:01:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.720+0000 7f1842f4a640 1 -- 172.21.15.81:0/3601784267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f183c1412f0 msgr2=0x7f183c0cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.720+0000 7f1842f4a640 1 --2- 172.21.15.81:0/3601784267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f183c1412f0 0x7f183c0cef90 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7f183c071d20 tx=0x7f182c02a8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.720+0000 7f1842f4a640 1 -- 172.21.15.81:0/3601784267 shutdown_connections 2024-06-29T16:01:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.720+0000 7f1842f4a640 1 --2- 172.21.15.81:0/3601784267 >> v2:172.21.15.167:3300/0 conn(0x7f183c073740 0x7f183c15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.720+0000 7f1842f4a640 1 --2- 172.21.15.81:0/3601784267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f183c1412f0 0x7f183c0cef90 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.720+0000 7f1842f4a640 1 -- 172.21.15.81:0/3601784267 >> 172.21.15.81:0/3601784267 conn(0x7f183c06ccc0 msgr2=0x7f183c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.720+0000 7f1842f4a640 1 -- 172.21.15.81:0/3601784267 shutdown_connections 2024-06-29T16:01:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.720+0000 7f1842f4a640 1 -- 172.21.15.81:0/3601784267 wait complete. 2024-06-29T16:01:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.722+0000 7f1842f4a640 1 Processor -- start 2024-06-29T16:01:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.722+0000 7f1842f4a640 1 -- start start 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.723+0000 7f1842f4a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f183c0cf4d0 0x7f183c13f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.723+0000 7f1842f4a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f183c1412f0 0x7f183c155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.723+0000 7f1842f4a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f183c1f5b90 0x7f183c158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.723+0000 7f1842f4a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f183c0b3210 con 0x7f183c1f5b90 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.723+0000 7f1842f4a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f183c15f1f0 con 0x7f183c1412f0 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.723+0000 7f1842f4a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f183c0b3390 con 0x7f183c0cf4d0 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18414e7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f183c1f5b90 0x7f183c158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18414e7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f183c1f5b90 0x7f183c158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36076/0 (socket says 172.21.15.81:36076) 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18414e7640 1 -- 172.21.15.81:0/2693587309 learned_addr learned my addr 172.21.15.81:0/2693587309 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f1840ce6640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f183c0cf4d0 0x7f183c13f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18414e7640 1 -- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f183c0cf4d0 msgr2=0x7f183c13f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:12.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f183bfff640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f183c1412f0 0x7f183c155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18414e7640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f183c0cf4d0 0x7f183c13f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18414e7640 1 -- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f183c1412f0 msgr2=0x7f183c155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18414e7640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f183c1412f0 0x7f183c155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18414e7640 1 -- 172.21.15.81:0/2693587309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183c156c80 con 0x7f183c1f5b90 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f1840ce6640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f183c0cf4d0 0x7f183c13f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f183bfff640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f183c1412f0 0x7f183c155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18414e7640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f183c1f5b90 0x7f183c158b80 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7f183400d650 tx=0x7f183400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18397fa640 1 -- 172.21.15.81:0/2693587309 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834017070 con 0x7f183c1f5b90 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18397fa640 1 -- 172.21.15.81:0/2693587309 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1834005310 con 0x7f183c1f5b90 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f1842f4a640 1 -- 172.21.15.81:0/2693587309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f183c1f50b0 con 0x7f183c1f5b90 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f18397fa640 1 -- 172.21.15.81:0/2693587309 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18340097c0 con 0x7f183c1f5b90 2024-06-29T16:01:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.724+0000 7f1842f4a640 1 -- 172.21.15.81:0/2693587309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f183c1f9140 con 0x7f183c1f5b90 2024-06-29T16:01:12.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.725+0000 7f1842f4a640 1 -- 172.21.15.81:0/2693587309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f183c1f93a0 con 0x7f183c1f5b90 2024-06-29T16:01:12.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.726+0000 7f18397fa640 1 -- 172.21.15.81:0/2693587309 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 161) ==== 98485+0+0 (secure 0 0 0) 0x7f1834009960 con 0x7f183c1f5b90 2024-06-29T16:01:12.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.726+0000 7f18397fa640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f1828076620 0x7f1828078b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:12.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.727+0000 7f18397fa640 1 -- 172.21.15.81:0/2693587309 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(269..269 src has 1..269) ==== 6210+0+0 (secure 0 0 0) 0x7f18340980a0 con 0x7f183c1f5b90 2024-06-29T16:01:12.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.727+0000 7f1840ce6640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f1828076620 0x7f1828078b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:12.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.727+0000 7f1840ce6640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f1828076620 0x7f1828078b30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f182c034000 tx=0x7f182c004ab0 comp rx=0 tx=0).ready entity=mgr.10283 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.732+0000 7f18397fa640 1 -- 172.21.15.81:0/2693587309 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f183c1f93a0 con 0x7f183c1f5b90 2024-06-29T16:01:12.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:12.960+0000 7f1842f4a640 1 -- 172.21.15.81:0/2693587309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) -- 0x7f183c1f4ec0 con 0x7f183c1f5b90 2024-06-29T16:01:13.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.556+0000 7f18397fa640 1 -- 172.21.15.81:0/2693587309 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v162) ==== 93+0+0 (secure 0 0 0) 0x7f18340611b0 con 0x7f183c1f5b90 2024-06-29T16:01:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.559+0000 7f1842f4a640 1 -- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f1828076620 msgr2=0x7f1828078b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.559+0000 7f1842f4a640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f1828076620 0x7f1828078b30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f182c034000 tx=0x7f182c004ab0 comp rx=0 tx=0).stop 2024-06-29T16:01:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.559+0000 7f1842f4a640 1 -- 172.21.15.81:0/2693587309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f183c1f5b90 msgr2=0x7f183c158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.559+0000 7f1842f4a640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f183c1f5b90 0x7f183c158b80 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7f183400d650 tx=0x7f183400db10 comp rx=0 tx=0).stop 2024-06-29T16:01:13.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.560+0000 7f1842f4a640 1 -- 172.21.15.81:0/2693587309 shutdown_connections 2024-06-29T16:01:13.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.560+0000 7f1842f4a640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f1828076620 0x7f1828078b30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:13.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.560+0000 7f1842f4a640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f183c1f5b90 0x7f183c158b80 unknown :-1 s=CLOSED pgs=2959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:13.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.560+0000 7f1842f4a640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f183c1412f0 0x7f183c155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:13.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.560+0000 7f1842f4a640 1 --2- 172.21.15.81:0/2693587309 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f183c0cf4d0 0x7f183c13f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:13.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.560+0000 7f1842f4a640 1 -- 172.21.15.81:0/2693587309 >> 172.21.15.81:0/2693587309 conn(0x7f183c06ccc0 msgr2=0x7f183c157260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:13.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.560+0000 7f1842f4a640 1 -- 172.21.15.81:0/2693587309 shutdown_connections 2024-06-29T16:01:13.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.560+0000 7f1842f4a640 1 -- 172.21.15.81:0/2693587309 wait complete. 2024-06-29T16:01:13.575 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-06-29T16:01:13.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T16:01:13.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-06-29T16:01:13.666 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-06-29T16:01:13.666 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T16:01:13.666 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T16:01:13.667 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:13.683 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T16:01:13.683 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:13.740 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T16:01:13.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.899+0000 7f64d54f6640 1 -- 172.21.15.81:0/4068383997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0051a0 msgr2=0x7f64bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:13.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.899+0000 7f64d54f6640 1 --2- 172.21.15.81:0/4068383997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0051a0 0x7f64bc00b9c0 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f64c800c640 tx=0x7f64bc005830 comp rx=0 tx=0).stop 2024-06-29T16:01:13.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.899+0000 7f64cf7fe640 1 -- 172.21.15.81:0/4068383997 reap_dead start 2024-06-29T16:01:13.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.899+0000 7f64d54f6640 1 -- 172.21.15.81:0/4068383997 shutdown_connections 2024-06-29T16:01:13.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.899+0000 7f64d54f6640 1 -- 172.21.15.81:0/4068383997 >> 172.21.15.81:0/4068383997 conn(0x7f64d00b2920 msgr2=0x7f64d00b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:13.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.899+0000 7f64d54f6640 1 -- 172.21.15.81:0/4068383997 shutdown_connections 2024-06-29T16:01:13.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.899+0000 7f64d54f6640 1 -- 172.21.15.81:0/4068383997 wait complete. 2024-06-29T16:01:13.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.901+0000 7f64d54f6640 1 Processor -- start 2024-06-29T16:01:13.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.901+0000 7f64d54f6640 1 -- start start 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64d54f6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0051a0 0x7f64d01f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64d54f6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64d013e6c0 0x7f64d0156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64d54f6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64d0157030 0x7f64d0153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64d54f6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64d00737d0 con 0x7f64bc0051a0 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64d54f6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f64d0073650 con 0x7f64d013e6c0 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64d54f6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f64d015b6e0 con 0x7f64d0157030 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64ceffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0051a0 0x7f64d01f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64ceffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0051a0 0x7f64d01f3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36112/0 (socket says 172.21.15.81:36112) 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64ceffd640 1 -- 172.21.15.81:0/3815255827 learned_addr learned my addr 172.21.15.81:0/3815255827 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64c7fff640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64d013e6c0 0x7f64d0156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.902+0000 7f64cf7fe640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64d0157030 0x7f64d0153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64ceffd640 1 -- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64d0157030 msgr2=0x7f64d0153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64ceffd640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64d0157030 0x7f64d0153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64ceffd640 1 -- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64d013e6c0 msgr2=0x7f64d0156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64ceffd640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64d013e6c0 0x7f64d0156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64ceffd640 1 -- 172.21.15.81:0/3815255827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d00cfb10 con 0x7f64bc0051a0 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64ceffd640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0051a0 0x7f64d01f3b00 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f64c0003ec0 tx=0x7f64c00017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64c77fe640 1 -- 172.21.15.81:0/3815255827 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c0015070 con 0x7f64bc0051a0 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64c77fe640 1 -- 172.21.15.81:0/3815255827 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64c0011920 con 0x7f64bc0051a0 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64d54f6640 1 -- 172.21.15.81:0/3815255827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d00cfdf0 con 0x7f64bc0051a0 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64c77fe640 1 -- 172.21.15.81:0/3815255827 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c0011d70 con 0x7f64bc0051a0 2024-06-29T16:01:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.903+0000 7f64d54f6640 1 -- 172.21.15.81:0/3815255827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d00d03b0 con 0x7f64bc0051a0 2024-06-29T16:01:13.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.904+0000 7f64d54f6640 1 -- 172.21.15.81:0/3815255827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d00cf870 con 0x7f64bc0051a0 2024-06-29T16:01:13.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.905+0000 7f64c77fe640 1 -- 172.21.15.81:0/3815255827 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 162) ==== 98468+0+0 (secure 0 0 0) 0x7f64c0018070 con 0x7f64bc0051a0 2024-06-29T16:01:13.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.905+0000 7f64c77fe640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f64b00765a0 0x7f64b0078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:13.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.905+0000 7f64c7fff640 1 -- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f64b00765a0 msgr2=0x7f64b0078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:13.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.905+0000 7f64c77fe640 1 -- 172.21.15.81:0/3815255827 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(270..270 src has 1..270) ==== 6298+0+0 (secure 0 0 0) 0x7f64c0098cd0 con 0x7f64bc0051a0 2024-06-29T16:01:13.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.906+0000 7f64c7fff640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f64b00765a0 0x7f64b0078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:13.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:13.910+0000 7f64c77fe640 1 -- 172.21.15.81:0/3815255827 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f64c0061820 con 0x7f64bc0051a0 2024-06-29T16:01:14.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.095+0000 7f64d54f6640 1 -- 172.21.15.81:0/3815255827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64d0072f40 con 0x7f64bc0051a0 2024-06-29T16:01:14.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.096+0000 7f64c77fe640 1 -- 172.21.15.81:0/3815255827 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) ==== 74+0+17590 (secure 0 0 0) 0x7f64c0065b20 con 0x7f64bc0051a0 2024-06-29T16:01:14.097 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:14.097 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":270,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T16:01:13.555631+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:40.261824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:40.923216+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":4,"snap_epoch":270,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.e","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3006664184":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6833/1824030697":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6832/1824030697":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/4135998660":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/432950336":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/1067289398":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/1087585815":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/2453006101":"2024-06-30T15:59:54.317815+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]},{"pool":37,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_removed_snaps":[{"pool":37,"snaps":[{"begin":2,"length":1},{"begin":4,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T16:01:14.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 -- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f64b00765a0 msgr2=0x7f64b0078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:14.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f64b00765a0 0x7f64b0078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 -- 172.21.15.81:0/3815255827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0051a0 msgr2=0x7f64d01f3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0051a0 0x7f64d01f3b00 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f64c0003ec0 tx=0x7f64c00017a0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 -- 172.21.15.81:0/3815255827 shutdown_connections 2024-06-29T16:01:14.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f64b00765a0 0x7f64b0078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f64d0157030 0x7f64d0153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f64d013e6c0 0x7f64d0156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 --2- 172.21.15.81:0/3815255827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64bc0051a0 0x7f64d01f3b00 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 -- 172.21.15.81:0/3815255827 >> 172.21.15.81:0/3815255827 conn(0x7f64d00b2920 msgr2=0x7f64d00b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:14.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 -- 172.21.15.81:0/3815255827 shutdown_connections 2024-06-29T16:01:14.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.098+0000 7f64d54f6640 1 -- 172.21.15.81:0/3815255827 wait complete. 2024-06-29T16:01:14.111 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:59:40.261824+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '265', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:59:40.923216+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '270', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 270, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T16:01:14.112 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T16:01:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.231+0000 7f65caea5640 1 -- 172.21.15.81:0/3626108601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40d0770 msgr2=0x7f65c40ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.231+0000 7f65caea5640 1 --2- 172.21.15.81:0/3626108601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40d0770 0x7f65c40ce520 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7f65c4073550 tx=0x7f65bc02aa30 comp rx=0 tx=0).stop 2024-06-29T16:01:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.231+0000 7f65caea5640 1 -- 172.21.15.81:0/3626108601 shutdown_connections 2024-06-29T16:01:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.231+0000 7f65caea5640 1 --2- 172.21.15.81:0/3626108601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40d0770 0x7f65c40ce520 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.231+0000 7f65caea5640 1 --2- 172.21.15.81:0/3626108601 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65c40cccf0 0x7f65c40d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.231+0000 7f65caea5640 1 --2- 172.21.15.81:0/3626108601 >> v2:172.21.15.167:3300/0 conn(0x7f65c40b3200 0x7f65c40cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.231+0000 7f65caea5640 1 -- 172.21.15.81:0/3626108601 >> 172.21.15.81:0/3626108601 conn(0x7f65c40b3680 msgr2=0x7f65c40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:14.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.231+0000 7f65caea5640 1 -- 172.21.15.81:0/3626108601 shutdown_connections 2024-06-29T16:01:14.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.231+0000 7f65caea5640 1 -- 172.21.15.81:0/3626108601 wait complete. 2024-06-29T16:01:14.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.233+0000 7f65caea5640 1 Processor -- start 2024-06-29T16:01:14.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.233+0000 7f65caea5640 1 -- start start 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.234+0000 7f65caea5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65c40b3200 0x7f65c415bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.234+0000 7f65caea5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65c40cccf0 0x7f65c406e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.234+0000 7f65caea5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40d0770 0x7f65c41e5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.234+0000 7f65caea5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f65c40cef90 con 0x7f65c40d0770 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.234+0000 7f65caea5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f65c40cee10 con 0x7f65c40b3200 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.234+0000 7f65caea5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f65c40cf110 con 0x7f65c40cccf0 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c9442640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40d0770 0x7f65c41e5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c9442640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40d0770 0x7f65c41e5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36130/0 (socket says 172.21.15.81:36130) 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c9442640 1 -- 172.21.15.81:0/1086562784 learned_addr learned my addr 172.21.15.81:0/1086562784 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c3fff640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65c40cccf0 0x7f65c406e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c8c41640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65c40b3200 0x7f65c415bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c9442640 1 -- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65c40cccf0 msgr2=0x7f65c406e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c9442640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65c40cccf0 0x7f65c406e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c9442640 1 -- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65c40b3200 msgr2=0x7f65c415bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c9442640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65c40b3200 0x7f65c415bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c9442640 1 -- 172.21.15.81:0/1086562784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c409f0a0 con 0x7f65c40d0770 2024-06-29T16:01:14.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c9442640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40d0770 0x7f65c41e5cd0 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7f65bc02b020 tx=0x7f65bc003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:14.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c17fa640 1 -- 172.21.15.81:0/1086562784 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc033050 con 0x7f65c40d0770 2024-06-29T16:01:14.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c17fa640 1 -- 172.21.15.81:0/1086562784 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65bc004450 con 0x7f65c40d0770 2024-06-29T16:01:14.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65caea5640 1 -- 172.21.15.81:0/1086562784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c41f8120 con 0x7f65c40d0770 2024-06-29T16:01:14.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65c17fa640 1 -- 172.21.15.81:0/1086562784 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc038450 con 0x7f65c40d0770 2024-06-29T16:01:14.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.235+0000 7f65caea5640 1 -- 172.21.15.81:0/1086562784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c41f86e0 con 0x7f65c40d0770 2024-06-29T16:01:14.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.236+0000 7f65caea5640 1 -- 172.21.15.81:0/1086562784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c409ee10 con 0x7f65c40d0770 2024-06-29T16:01:14.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.237+0000 7f65c17fa640 1 -- 172.21.15.81:0/1086562784 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 162) ==== 98468+0+0 (secure 0 0 0) 0x7f65bc036070 con 0x7f65c40d0770 2024-06-29T16:01:14.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.238+0000 7f65c17fa640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f65a80765d0 0x7f65a8078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.238+0000 7f65c8c41640 1 -- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f65a80765d0 msgr2=0x7f65a8078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:14.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.238+0000 7f65c17fa640 1 -- 172.21.15.81:0/1086562784 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(270..270 src has 1..270) ==== 6298+0+0 (secure 0 0 0) 0x7f65bc0b92e0 con 0x7f65c40d0770 2024-06-29T16:01:14.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.238+0000 7f65c8c41640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f65a80765d0 0x7f65a8078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:14.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.246+0000 7f65c17fa640 1 -- 172.21.15.81:0/1086562784 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f65bc081f40 con 0x7f65c40d0770 2024-06-29T16:01:14.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.431+0000 7f65caea5640 1 -- 172.21.15.81:0/1086562784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f65c40d0230 con 0x7f65c40d0770 2024-06-29T16:01:14.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.432+0000 7f65c17fa640 1 -- 172.21.15.81:0/1086562784 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v270) ==== 93+0+10 (secure 0 0 0) 0x7f65bc086240 con 0x7f65c40d0770 2024-06-29T16:01:14.432 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T16:01:14.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 -- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f65a80765d0 msgr2=0x7f65a8078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:14.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f65a80765d0 0x7f65a8078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 -- 172.21.15.81:0/1086562784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40d0770 msgr2=0x7f65c41e5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40d0770 0x7f65c41e5cd0 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7f65bc02b020 tx=0x7f65bc003f90 comp rx=0 tx=0).stop 2024-06-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 -- 172.21.15.81:0/1086562784 shutdown_connections 2024-06-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f65a80765d0 0x7f65a8078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40d0770 0x7f65c41e5cd0 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65c40cccf0 0x7f65c406e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 --2- 172.21.15.81:0/1086562784 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65c40b3200 0x7f65c415bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 -- 172.21.15.81:0/1086562784 >> 172.21.15.81:0/1086562784 conn(0x7f65c40b3680 msgr2=0x7f65c40b50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 -- 172.21.15.81:0/1086562784 shutdown_connections 2024-06-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.434+0000 7f65caea5640 1 -- 172.21.15.81:0/1086562784 wait complete. 2024-06-29T16:01:14.446 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T16:01:14.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.565+0000 7feaeec3c640 1 -- 172.21.15.81:0/2354741856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae8073650 msgr2=0x7feae815b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.565+0000 7feaeec3c640 1 --2- 172.21.15.81:0/2354741856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae8073650 0x7feae815b6e0 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7fead4005fd0 tx=0x7fead402a540 comp rx=0 tx=0).stop 2024-06-29T16:01:14.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.565+0000 7feaeec3c640 1 -- 172.21.15.81:0/2354741856 shutdown_connections 2024-06-29T16:01:14.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.565+0000 7feaeec3c640 1 --2- 172.21.15.81:0/2354741856 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feae815bc20 0x7feae806ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.565+0000 7feaeec3c640 1 --2- 172.21.15.81:0/2354741856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae8073650 0x7feae815b6e0 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.565+0000 7feaeec3c640 1 -- 172.21.15.81:0/2354741856 >> 172.21.15.81:0/2354741856 conn(0x7feae80b2920 msgr2=0x7feae80b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:14.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.565+0000 7feaeec3c640 1 -- 172.21.15.81:0/2354741856 shutdown_connections 2024-06-29T16:01:14.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.565+0000 7feaeec3c640 1 -- 172.21.15.81:0/2354741856 wait complete. 2024-06-29T16:01:14.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.567+0000 7feaeec3c640 1 Processor -- start 2024-06-29T16:01:14.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.567+0000 7feaeec3c640 1 -- start start 2024-06-29T16:01:14.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaeec3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feae80cd150 0x7feae815a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaeec3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feae815bc20 0x7feae806cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaeec3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae81485f0 0x7feae8147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaeec3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feae806f660 con 0x7feae81485f0 2024-06-29T16:01:14.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaeec3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7feae806f4e0 con 0x7feae815bc20 2024-06-29T16:01:14.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaeec3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7feae806f7e0 con 0x7feae80cd150 2024-06-29T16:01:14.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaed1d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae81485f0 0x7feae8147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaed1d9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae81485f0 0x7feae8147da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36156/0 (socket says 172.21.15.81:36156) 2024-06-29T16:01:14.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaed1d9640 1 -- 172.21.15.81:0/1549939171 learned_addr learned my addr 172.21.15.81:0/1549939171 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:14.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaec9d8640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feae80cd150 0x7feae815a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feae7fff640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feae815bc20 0x7feae806cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaed1d9640 1 -- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feae80cd150 msgr2=0x7feae815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaed1d9640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feae80cd150 0x7feae815a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaed1d9640 1 -- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feae815bc20 msgr2=0x7feae806cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaed1d9640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feae815bc20 0x7feae806cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.569+0000 7feaed1d9640 1 -- 172.21.15.81:0/1549939171 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae815abb0 con 0x7feae81485f0 2024-06-29T16:01:14.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.570+0000 7feaed1d9640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae81485f0 0x7feae8147da0 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7feae000ea70 tx=0x7feae000ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:14.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.572+0000 7feae57fa640 1 -- 172.21.15.81:0/1549939171 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae000ce40 con 0x7feae81485f0 2024-06-29T16:01:14.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.572+0000 7feae57fa640 1 -- 172.21.15.81:0/1549939171 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feae0004c10 con 0x7feae81485f0 2024-06-29T16:01:14.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.572+0000 7feae57fa640 1 -- 172.21.15.81:0/1549939171 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0005020 con 0x7feae81485f0 2024-06-29T16:01:14.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.572+0000 7feaeec3c640 1 -- 172.21.15.81:0/1549939171 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae815ae90 con 0x7feae81485f0 2024-06-29T16:01:14.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.572+0000 7feaeec3c640 1 -- 172.21.15.81:0/1549939171 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae806c770 con 0x7feae81485f0 2024-06-29T16:01:14.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.573+0000 7feaeec3c640 1 -- 172.21.15.81:0/1549939171 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feae815b0d0 con 0x7feae81485f0 2024-06-29T16:01:14.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.574+0000 7feae57fa640 1 -- 172.21.15.81:0/1549939171 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 162) ==== 98468+0+0 (secure 0 0 0) 0x7feae00051c0 con 0x7feae81485f0 2024-06-29T16:01:14.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.574+0000 7feae57fa640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7feacc0765d0 0x7feacc078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.575+0000 7feae57fa640 1 -- 172.21.15.81:0/1549939171 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(270..270 src has 1..270) ==== 6298+0+0 (secure 0 0 0) 0x7feae0098c00 con 0x7feae81485f0 2024-06-29T16:01:14.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.575+0000 7feaec9d8640 1 -- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7feacc0765d0 msgr2=0x7feacc078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:14.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.575+0000 7feaec9d8640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7feacc0765d0 0x7feacc078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:14.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.587+0000 7feae57fa640 1 -- 172.21.15.81:0/1549939171 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7feae815b0d0 con 0x7feae81485f0 2024-06-29T16:01:14.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.776+0000 7feaec9d8640 1 -- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7feacc0765d0 msgr2=0x7feacc078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:14.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.776+0000 7feaec9d8640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7feacc0765d0 0x7feacc078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T16:01:14.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.780+0000 7feaeec3c640 1 -- 172.21.15.81:0/1549939171 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7feae806fd00 con 0x7feae81485f0 2024-06-29T16:01:14.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.780+0000 7feae57fa640 1 -- 172.21.15.81:0/1549939171 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v271) ==== 104+0+11 (secure 0 0 0) 0x7feae00617d0 con 0x7feae81485f0 2024-06-29T16:01:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T16:01:14.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.782+0000 7feaeec3c640 1 -- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7feacc0765d0 msgr2=0x7feacc078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:14.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.782+0000 7feaeec3c640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7feacc0765d0 0x7feacc078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.782+0000 7feaeec3c640 1 -- 172.21.15.81:0/1549939171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae81485f0 msgr2=0x7feae8147da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.782+0000 7feaeec3c640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae81485f0 0x7feae8147da0 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7feae000ea70 tx=0x7feae000ef30 comp rx=0 tx=0).stop 2024-06-29T16:01:14.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.782+0000 7feaeec3c640 1 -- 172.21.15.81:0/1549939171 shutdown_connections 2024-06-29T16:01:14.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.783+0000 7feaeec3c640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7feacc0765d0 0x7feacc078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.783+0000 7feaeec3c640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae81485f0 0x7feae8147da0 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.783+0000 7feaeec3c640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7feae815bc20 0x7feae806cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.783+0000 7feaeec3c640 1 --2- 172.21.15.81:0/1549939171 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7feae80cd150 0x7feae815a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.783+0000 7feaeec3c640 1 -- 172.21.15.81:0/1549939171 >> 172.21.15.81:0/1549939171 conn(0x7feae80b2920 msgr2=0x7feae80b17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:14.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.783+0000 7feaeec3c640 1 -- 172.21.15.81:0/1549939171 shutdown_connections 2024-06-29T16:01:14.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.783+0000 7feaeec3c640 1 -- 172.21.15.81:0/1549939171 wait complete. 2024-06-29T16:01:14.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T16:01:14.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.912+0000 7f4ef0368640 1 -- 172.21.15.81:0/3131679208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ee80b3200 msgr2=0x7f4ee80cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.912+0000 7f4ef0368640 1 --2- 172.21.15.81:0/3131679208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ee80b3200 0x7f4ee80cc7b0 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8074b10 tx=0x7f4ed802a4a0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.912+0000 7f4ef0368640 1 -- 172.21.15.81:0/3131679208 shutdown_connections 2024-06-29T16:01:14.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.912+0000 7f4ef0368640 1 --2- 172.21.15.81:0/3131679208 >> v2:172.21.15.167:3300/0 conn(0x7f4ee80cccf0 0x7f4ee80d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.912+0000 7f4ef0368640 1 --2- 172.21.15.81:0/3131679208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ee80b3200 0x7f4ee80cc7b0 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.912+0000 7f4ef0368640 1 -- 172.21.15.81:0/3131679208 >> 172.21.15.81:0/3131679208 conn(0x7f4ee80b3680 msgr2=0x7f4ee80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:14.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.912+0000 7f4ef0368640 1 -- 172.21.15.81:0/3131679208 shutdown_connections 2024-06-29T16:01:14.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.912+0000 7f4ef0368640 1 -- 172.21.15.81:0/3131679208 wait complete. 2024-06-29T16:01:14.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.914+0000 7f4ef0368640 1 Processor -- start 2024-06-29T16:01:14.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.914+0000 7f4ef0368640 1 -- start start 2024-06-29T16:01:14.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4ef0368640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ee80b3200 0x7f4ee806e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4ef0368640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ee80cccf0 0x7f4ee806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4ef0368640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ee80d0770 0x7f4ee81d2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4ef0368640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee80cedb0 con 0x7f4ee80b3200 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4ef0368640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4ee80cec30 con 0x7f4ee80cccf0 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4ef0368640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4ee80cef30 con 0x7f4ee80d0770 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4eee104640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ee80b3200 0x7f4ee806e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4eee104640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ee80b3200 0x7f4ee806e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36184/0 (socket says 172.21.15.81:36184) 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4eee104640 1 -- 172.21.15.81:0/1996633867 learned_addr learned my addr 172.21.15.81:0/1996633867 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4eed903640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ee80cccf0 0x7f4ee806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4eee905640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ee80d0770 0x7f4ee81d2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4eee104640 1 -- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ee80d0770 msgr2=0x7f4ee81d2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4eee104640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ee80d0770 0x7f4ee81d2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4eee104640 1 -- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ee80cccf0 msgr2=0x7f4ee806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:14.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4eee104640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ee80cccf0 0x7f4ee806e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:14.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.915+0000 7f4eee104640 1 -- 172.21.15.81:0/1996633867 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ee81e11d0 con 0x7f4ee80b3200 2024-06-29T16:01:14.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.916+0000 7f4eee104640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ee80b3200 0x7f4ee806e260 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80098f0 tx=0x7f4ed8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:14.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.916+0000 7f4edeffd640 1 -- 172.21.15.81:0/1996633867 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8005c90 con 0x7f4ee80b3200 2024-06-29T16:01:14.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.916+0000 7f4edeffd640 1 -- 172.21.15.81:0/1996633867 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ed8031e50 con 0x7f4ee80b3200 2024-06-29T16:01:14.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.916+0000 7f4ef0368640 1 -- 172.21.15.81:0/1996633867 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ee80c9150 con 0x7f4ee80b3200 2024-06-29T16:01:14.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.916+0000 7f4edeffd640 1 -- 172.21.15.81:0/1996633867 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8005260 con 0x7f4ee80b3200 2024-06-29T16:01:14.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.916+0000 7f4ef0368640 1 -- 172.21.15.81:0/1996633867 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ee80cbe20 con 0x7f4ee80b3200 2024-06-29T16:01:14.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.917+0000 7f4ef0368640 1 -- 172.21.15.81:0/1996633867 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ee81e5430 con 0x7f4ee80b3200 2024-06-29T16:01:14.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.917+0000 7f4edeffd640 1 -- 172.21.15.81:0/1996633867 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 162) ==== 98468+0+0 (secure 0 0 0) 0x7f4ed8005400 con 0x7f4ee80b3200 2024-06-29T16:01:14.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.918+0000 7f4edeffd640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f4ecc0765d0 0x7f4ecc078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:14.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.918+0000 7f4eed903640 1 -- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f4ecc0765d0 msgr2=0x7f4ecc078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:14.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.918+0000 7f4edeffd640 1 -- 172.21.15.81:0/1996633867 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 6282+0+0 (secure 0 0 0) 0x7f4ed80b8f30 con 0x7f4ee80b3200 2024-06-29T16:01:14.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.918+0000 7f4eed903640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f4ecc0765d0 0x7f4ecc078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:14.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:14.922+0000 7f4edeffd640 1 -- 172.21.15.81:0/1996633867 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4ed803f030 con 0x7f4ee80b3200 2024-06-29T16:01:15.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.107+0000 7f4ef0368640 1 -- 172.21.15.81:0/1996633867 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4ee81e5150 con 0x7f4ee80b3200 2024-06-29T16:01:15.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.107+0000 7f4edeffd640 1 -- 172.21.15.81:0/1996633867 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v271) ==== 100+0+11 (secure 0 0 0) 0x7f4ed808a1e0 con 0x7f4ee80b3200 2024-06-29T16:01:15.108 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 -- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f4ecc0765d0 msgr2=0x7f4ecc078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f4ecc0765d0 0x7f4ecc078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 -- 172.21.15.81:0/1996633867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ee80b3200 msgr2=0x7f4ee806e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ee80b3200 0x7f4ee806e260 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80098f0 tx=0x7f4ed8033c20 comp rx=0 tx=0).stop 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 -- 172.21.15.81:0/1996633867 shutdown_connections 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f4ecc0765d0 0x7f4ecc078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ee80d0770 0x7f4ee81d2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ee80cccf0 0x7f4ee806e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 --2- 172.21.15.81:0/1996633867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ee80b3200 0x7f4ee806e260 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 -- 172.21.15.81:0/1996633867 >> 172.21.15.81:0/1996633867 conn(0x7f4ee80b3680 msgr2=0x7f4ee80701c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:15.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 -- 172.21.15.81:0/1996633867 shutdown_connections 2024-06-29T16:01:15.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.109+0000 7f4ef0368640 1 -- 172.21.15.81:0/1996633867 wait complete. 2024-06-29T16:01:15.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-06-29T16:01:15.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.238+0000 7fd45549e640 1 -- 172.21.15.81:0/944397233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4500b8ba0 msgr2=0x7fd4500d46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.238+0000 7fd45549e640 1 --2- 172.21.15.81:0/944397233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4500b8ba0 0x7fd4500d46f0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7fd4500d0f10 tx=0x7fd44002a5d0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.238+0000 7fd45549e640 1 -- 172.21.15.81:0/944397233 shutdown_connections 2024-06-29T16:01:15.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.238+0000 7fd45549e640 1 --2- 172.21.15.81:0/944397233 >> v2:172.21.15.167:3300/0 conn(0x7fd4500d0950 0x7fd4500cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.238+0000 7fd45549e640 1 --2- 172.21.15.81:0/944397233 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4500b6a20 0x7fd4500d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.238+0000 7fd45549e640 1 --2- 172.21.15.81:0/944397233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4500b8ba0 0x7fd4500d46f0 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.238+0000 7fd45549e640 1 -- 172.21.15.81:0/944397233 >> 172.21.15.81:0/944397233 conn(0x7fd4500d3bd0 msgr2=0x7fd4500d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:15.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.238+0000 7fd45549e640 1 -- 172.21.15.81:0/944397233 shutdown_connections 2024-06-29T16:01:15.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.238+0000 7fd45549e640 1 -- 172.21.15.81:0/944397233 wait complete. 2024-06-29T16:01:15.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.240+0000 7fd45549e640 1 Processor -- start 2024-06-29T16:01:15.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.240+0000 7fd45549e640 1 -- start start 2024-06-29T16:01:15.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd45549e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4500b6a20 0x7fd450154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd45549e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4500b8ba0 0x7fd450156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd45549e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4500d0950 0x7fd450158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd45549e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd4500b9290 con 0x7fd4500d0950 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd45549e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd4500b7040 con 0x7fd4500b8ba0 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd45549e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd4500b9410 con 0x7fd4500b6a20 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd44f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4500d0950 0x7fd450158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd44effd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4500b6a20 0x7fd450154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd44f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4500d0950 0x7fd450158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36204/0 (socket says 172.21.15.81:36204) 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd44f7fe640 1 -- 172.21.15.81:0/2549026663 learned_addr learned my addr 172.21.15.81:0/2549026663 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd447fff640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4500b8ba0 0x7fd450156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd44f7fe640 1 -- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4500b6a20 msgr2=0x7fd450154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd44f7fe640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4500b6a20 0x7fd450154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd44f7fe640 1 -- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4500b8ba0 msgr2=0x7fd450156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd44f7fe640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4500b8ba0 0x7fd450156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.241+0000 7fd44f7fe640 1 -- 172.21.15.81:0/2549026663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45015a6c0 con 0x7fd4500d0950 2024-06-29T16:01:15.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.242+0000 7fd44f7fe640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4500d0950 0x7fd450158330 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7fd44800e9c0 tx=0x7fd44800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:15.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.242+0000 7fd4477fe640 1 -- 172.21.15.81:0/2549026663 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448004d20 con 0x7fd4500d0950 2024-06-29T16:01:15.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.242+0000 7fd4477fe640 1 -- 172.21.15.81:0/2549026663 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd44800cb20 con 0x7fd4500d0950 2024-06-29T16:01:15.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.242+0000 7fd45549e640 1 -- 172.21.15.81:0/2549026663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd45015a9a0 con 0x7fd4500d0950 2024-06-29T16:01:15.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.242+0000 7fd4477fe640 1 -- 172.21.15.81:0/2549026663 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448005020 con 0x7fd4500d0950 2024-06-29T16:01:15.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.242+0000 7fd45549e640 1 -- 172.21.15.81:0/2549026663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd45015af60 con 0x7fd4500d0950 2024-06-29T16:01:15.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.243+0000 7fd45549e640 1 -- 172.21.15.81:0/2549026663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd45015a430 con 0x7fd4500d0950 2024-06-29T16:01:15.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.244+0000 7fd4477fe640 1 -- 172.21.15.81:0/2549026663 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 162) ==== 98468+0+0 (secure 0 0 0) 0x7fd4480051c0 con 0x7fd4500d0950 2024-06-29T16:01:15.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.244+0000 7fd4477fe640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fd4300765a0 0x7fd430078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.244+0000 7fd4477fe640 1 -- 172.21.15.81:0/2549026663 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 6282+0+0 (secure 0 0 0) 0x7fd4480987d0 con 0x7fd4500d0950 2024-06-29T16:01:15.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.244+0000 7fd44effd640 1 -- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fd4300765a0 msgr2=0x7fd430078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:15.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.244+0000 7fd44effd640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fd4300765a0 0x7fd430078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:15.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.253+0000 7fd4477fe640 1 -- 172.21.15.81:0/2549026663 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd448016030 con 0x7fd4500d0950 2024-06-29T16:01:15.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.446+0000 7fd44effd640 1 -- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fd4300765a0 msgr2=0x7fd430078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:15.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.446+0000 7fd44effd640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fd4300765a0 0x7fd430078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T16:01:15.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.453+0000 7fd45549e640 1 -- 172.21.15.81:0/2549026663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd4500d46f0 con 0x7fd4500d0950 2024-06-29T16:01:15.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.453+0000 7fd4477fe640 1 -- 172.21.15.81:0/2549026663 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v271) ==== 80+0+477 (secure 0 0 0) 0x7fd448061330 con 0x7fd4500d0950 2024-06-29T16:01:15.454 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/3006664184 2024-06-30T15:59:54.317815+0000 2024-06-29T16:01:15.454 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6833/1824030697 2024-06-30T15:59:54.317815+0000 2024-06-29T16:01:15.454 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:6832/1824030697 2024-06-30T15:59:54.317815+0000 2024-06-29T16:01:15.454 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/4135998660 2024-06-30T15:59:54.317815+0000 2024-06-29T16:01:15.454 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/432950336 2024-06-30T15:59:54.317815+0000 2024-06-29T16:01:15.454 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1067289398 2024-06-30T15:59:54.317815+0000 2024-06-29T16:01:15.454 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/1087585815 2024-06-30T15:59:54.317815+0000 2024-06-29T16:01:15.454 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.167:0/2453006101 2024-06-30T15:59:54.317815+0000 2024-06-29T16:01:15.455 INFO:teuthology.orchestra.run.smithi081.stderr:listed 8 entries 2024-06-29T16:01:15.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.455+0000 7fd45549e640 1 -- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fd4300765a0 msgr2=0x7fd430078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:15.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fd4300765a0 0x7fd430078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 -- 172.21.15.81:0/2549026663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4500d0950 msgr2=0x7fd450158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4500d0950 0x7fd450158330 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7fd44800e9c0 tx=0x7fd44800ee80 comp rx=0 tx=0).stop 2024-06-29T16:01:15.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 -- 172.21.15.81:0/2549026663 shutdown_connections 2024-06-29T16:01:15.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fd4300765a0 0x7fd430078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4500d0950 0x7fd450158330 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd4500b8ba0 0x7fd450156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 --2- 172.21.15.81:0/2549026663 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd4500b6a20 0x7fd450154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 -- 172.21.15.81:0/2549026663 >> 172.21.15.81:0/2549026663 conn(0x7fd4500d3bd0 msgr2=0x7fd4500b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:15.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 -- 172.21.15.81:0/2549026663 shutdown_connections 2024-06-29T16:01:15.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.456+0000 7fd45549e640 1 -- 172.21.15.81:0/2549026663 wait complete. 2024-06-29T16:01:15.468 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-29T16:01:15.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:15.468 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-29T16:01:15.496 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T16:01:15.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:15.497 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-29T16:01:15.553 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T16:01:15.553 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T16:01:15.553 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:15.569 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T16:01:15.569 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T16:01:15.569 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T16:01:15.570 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:15.609 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T16:01:15.609 INFO:tasks.cephfs.mount:Terminating background process 2024-06-29T16:01:15.609 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T16:01:15.609 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:15.626 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T16:01:15.626 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T16:01:15.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.764+0000 7fdffc6a5640 1 -- 172.21.15.81:0/3745855955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff40b3200 msgr2=0x7fdff40cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.764+0000 7fdffc6a5640 1 --2- 172.21.15.81:0/3745855955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff40b3200 0x7fdff40cc7b0 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7fdff415c790 tx=0x7fdfe802a560 comp rx=0 tx=0).stop 2024-06-29T16:01:15.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.764+0000 7fdffc6a5640 1 -- 172.21.15.81:0/3745855955 shutdown_connections 2024-06-29T16:01:15.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.764+0000 7fdffc6a5640 1 --2- 172.21.15.81:0/3745855955 >> v2:172.21.15.167:3300/0 conn(0x7fdff40d0770 0x7fdff40ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.764+0000 7fdffc6a5640 1 --2- 172.21.15.81:0/3745855955 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff40cccf0 0x7fdff40d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.764+0000 7fdffc6a5640 1 --2- 172.21.15.81:0/3745855955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff40b3200 0x7fdff40cc7b0 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.764+0000 7fdffc6a5640 1 -- 172.21.15.81:0/3745855955 >> 172.21.15.81:0/3745855955 conn(0x7fdff40b3680 msgr2=0x7fdff40b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:15.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.764+0000 7fdffc6a5640 1 -- 172.21.15.81:0/3745855955 shutdown_connections 2024-06-29T16:01:15.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.764+0000 7fdffc6a5640 1 -- 172.21.15.81:0/3745855955 wait complete. 2024-06-29T16:01:15.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.766+0000 7fdffc6a5640 1 Processor -- start 2024-06-29T16:01:15.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.766+0000 7fdffc6a5640 1 -- start start 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.767+0000 7fdffc6a5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff40b3200 0x7fdff415be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.767+0000 7fdffc6a5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdff40cccf0 0x7fdff4152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.767+0000 7fdffc6a5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff40d0770 0x7fdff41e7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.767+0000 7fdffc6a5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdff40cef90 con 0x7fdff40d0770 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.767+0000 7fdffc6a5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdff40cee10 con 0x7fdff40cccf0 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.767+0000 7fdffc6a5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdff40cf110 con 0x7fdff40b3200 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffac42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff40d0770 0x7fdff41e7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffa441640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff40b3200 0x7fdff415be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffac42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff40d0770 0x7fdff41e7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36224/0 (socket says 172.21.15.81:36224) 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffac42640 1 -- 172.21.15.81:0/1065293661 learned_addr learned my addr 172.21.15.81:0/1065293661 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:15.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdff9c40640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdff40cccf0 0x7fdff4152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffac42640 1 -- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff40b3200 msgr2=0x7fdff415be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffac42640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff40b3200 0x7fdff415be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffac42640 1 -- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdff40cccf0 msgr2=0x7fdff4152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffac42640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdff40cccf0 0x7fdff4152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffac42640 1 -- 172.21.15.81:0/1065293661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff41e16b0 con 0x7fdff40d0770 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffac42640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff40d0770 0x7fdff41e7630 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00e9c0 tx=0x7fdfec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdfe6ffd640 1 -- 172.21.15.81:0/1065293661 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec004d20 con 0x7fdff40d0770 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdfe6ffd640 1 -- 172.21.15.81:0/1065293661 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdfec00cac0 con 0x7fdff40d0770 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffc6a5640 1 -- 172.21.15.81:0/1065293661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff41e1950 con 0x7fdff40d0770 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdfe6ffd640 1 -- 172.21.15.81:0/1065293661 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec005020 con 0x7fdff40d0770 2024-06-29T16:01:15.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.768+0000 7fdffc6a5640 1 -- 172.21.15.81:0/1065293661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff41e1f10 con 0x7fdff40d0770 2024-06-29T16:01:15.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.769+0000 7fdffc6a5640 1 -- 172.21.15.81:0/1065293661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdff40d0230 con 0x7fdff40d0770 2024-06-29T16:01:15.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.771+0000 7fdfe6ffd640 1 -- 172.21.15.81:0/1065293661 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 162) ==== 98468+0+0 (secure 0 0 0) 0x7fdfec0051c0 con 0x7fdff40d0770 2024-06-29T16:01:15.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.771+0000 7fdfe6ffd640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fdfd80765d0 0x7fdfd8078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:15.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.771+0000 7fdffa441640 1 -- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fdfd80765d0 msgr2=0x7fdfd8078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:15.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.771+0000 7fdfe6ffd640 1 -- 172.21.15.81:0/1065293661 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 6282+0+0 (secure 0 0 0) 0x7fdfec098b50 con 0x7fdff40d0770 2024-06-29T16:01:15.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.771+0000 7fdffa441640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fdfd80765d0 0x7fdfd8078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:15.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.775+0000 7fdfe6ffd640 1 -- 172.21.15.81:0/1065293661 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdfec016030 con 0x7fdff40d0770 2024-06-29T16:01:15.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.972+0000 7fdffa441640 1 -- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fdfd80765d0 msgr2=0x7fdfd8078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:15.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:15.972+0000 7fdffa441640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fdfd80765d0 0x7fdfd8078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-29T16:01:16.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.000+0000 7fdffc6a5640 1 -- 172.21.15.81:0/1065293661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdff4070fa0 con 0x7fdff40d0770 2024-06-29T16:01:16.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.001+0000 7fdfe6ffd640 1 -- 172.21.15.81:0/1065293661 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5238 (secure 0 0 0) 0x7fdfec061730 con 0x7fdff40d0770 2024-06-29T16:01:16.001 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:16.001 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":154,"btime":"2024-06-29T15:59:45:615681+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10196,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2932449820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2932449820},{"type":"v1","addr":"172.21.15.167:6837","nonce":2932449820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/317823138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":317823138},{"type":"v1","addr":"172.21.15.81:6835","nonce":317823138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3582280449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3582280449},{"type":"v1","addr":"172.21.15.167:6836","nonce":3582280449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:41.965720+0000","modified":"2024-06-29T15:59:44.983893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13371},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13371":{"gid":13371,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/28043779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":28043779},{"type":"v1","addr":"172.21.15.81:6834","nonce":28043779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13371,"qdb_cluster":[13371]},"id":18}]} 2024-06-29T16:01:16.001 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 154 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 -- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fdfd80765d0 msgr2=0x7fdfd8078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fdfd80765d0 0x7fdfd8078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 -- 172.21.15.81:0/1065293661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff40d0770 msgr2=0x7fdff41e7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff40d0770 0x7fdff41e7630 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00e9c0 tx=0x7fdfec00ee80 comp rx=0 tx=0).stop 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 -- 172.21.15.81:0/1065293661 shutdown_connections 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fdfd80765d0 0x7fdfd8078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdff40d0770 0x7fdff41e7630 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdff40cccf0 0x7fdff4152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 --2- 172.21.15.81:0/1065293661 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdff40b3200 0x7fdff415be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 -- 172.21.15.81:0/1065293661 >> 172.21.15.81:0/1065293661 conn(0x7fdff40b3680 msgr2=0x7fdff4155f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:16.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 -- 172.21.15.81:0/1065293661 shutdown_connections 2024-06-29T16:01:16.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.003+0000 7fdffc6a5640 1 -- 172.21.15.81:0/1065293661 wait complete. 2024-06-29T16:01:16.014 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T16:01:16.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.130+0000 7f7e5badf640 1 -- 172.21.15.81:0/2811026755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e540b4f40 msgr2=0x7f7e540b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.130+0000 7f7e5badf640 1 --2- 172.21.15.81:0/2811026755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e540b4f40 0x7f7e540b5350 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c005fd0 tx=0x7f7e4c02aa90 comp rx=0 tx=0).stop 2024-06-29T16:01:16.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.130+0000 7f7e5badf640 1 -- 172.21.15.81:0/2811026755 shutdown_connections 2024-06-29T16:01:16.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.130+0000 7f7e5badf640 1 --2- 172.21.15.81:0/2811026755 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e540d0230 0x7f7e540d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.130+0000 7f7e5badf640 1 --2- 172.21.15.81:0/2811026755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e540b4f40 0x7f7e540b5350 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.130+0000 7f7e5badf640 1 -- 172.21.15.81:0/2811026755 >> 172.21.15.81:0/2811026755 conn(0x7f7e540b3680 msgr2=0x7f7e540b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:16.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.130+0000 7f7e5badf640 1 -- 172.21.15.81:0/2811026755 shutdown_connections 2024-06-29T16:01:16.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.130+0000 7f7e5badf640 1 -- 172.21.15.81:0/2811026755 wait complete. 2024-06-29T16:01:16.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.132+0000 7f7e5badf640 1 Processor -- start 2024-06-29T16:01:16.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.132+0000 7f7e5badf640 1 -- start start 2024-06-29T16:01:16.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.133+0000 7f7e5badf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e540b3200 0x7f7e541f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.133+0000 7f7e5badf640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e540b4f40 0x7f7e541e11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.133+0000 7f7e5badf640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e540d0230 0x7f7e540ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.133+0000 7f7e5badf640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7e540b5c20 con 0x7f7e540b3200 2024-06-29T16:01:16.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.133+0000 7f7e5badf640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f7e540b5aa0 con 0x7f7e540d0230 2024-06-29T16:01:16.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.133+0000 7f7e5badf640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f7e540b5da0 con 0x7f7e540b4f40 2024-06-29T16:01:16.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5987b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e540b3200 0x7f7e541f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5987b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e540b3200 0x7f7e541f2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36244/0 (socket says 172.21.15.81:36244) 2024-06-29T16:01:16.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5987b640 1 -- 172.21.15.81:0/3530071768 learned_addr learned my addr 172.21.15.81:0/3530071768 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5907a640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e540b4f40 0x7f7e541e11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5a07c640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e540d0230 0x7f7e540ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5987b640 1 -- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e540b4f40 msgr2=0x7f7e541e11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5987b640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e540b4f40 0x7f7e541e11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5987b640 1 -- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e540d0230 msgr2=0x7f7e540ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5987b640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e540d0230 0x7f7e540ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5987b640 1 -- 172.21.15.81:0/3530071768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e541f34e0 con 0x7f7e540b3200 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5987b640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e540b3200 0x7f7e541f2ba0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400c3a0 tx=0x7f7e4400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e4a7fc640 1 -- 172.21.15.81:0/3530071768 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44017070 con 0x7f7e540b3200 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e4a7fc640 1 -- 172.21.15.81:0/3530071768 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e440053c0 con 0x7f7e540b3200 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5badf640 1 -- 172.21.15.81:0/3530071768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e541f37c0 con 0x7f7e540b3200 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e4a7fc640 1 -- 172.21.15.81:0/3530071768 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44003850 con 0x7f7e540b3200 2024-06-29T16:01:16.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.134+0000 7f7e5badf640 1 -- 172.21.15.81:0/3530071768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e541e4870 con 0x7f7e540b3200 2024-06-29T16:01:16.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.135+0000 7f7e5badf640 1 -- 172.21.15.81:0/3530071768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e540b5350 con 0x7f7e540b3200 2024-06-29T16:01:16.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.136+0000 7f7e4a7fc640 1 -- 172.21.15.81:0/3530071768 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 162) ==== 98468+0+0 (secure 0 0 0) 0x7f7e4401a070 con 0x7f7e540b3200 2024-06-29T16:01:16.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.137+0000 7f7e4a7fc640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f7e380765a0 0x7f7e38078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.137+0000 7f7e5907a640 1 -- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f7e380765a0 msgr2=0x7f7e38078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:16.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.137+0000 7f7e4a7fc640 1 -- 172.21.15.81:0/3530071768 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 6282+0+0 (secure 0 0 0) 0x7f7e440971d0 con 0x7f7e540b3200 2024-06-29T16:01:16.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.137+0000 7f7e5907a640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f7e380765a0 0x7f7e38078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:16.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.141+0000 7f7e4a7fc640 1 -- 172.21.15.81:0/3530071768 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7e540d0b80 con 0x7f7e540b3200 2024-06-29T16:01:16.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.331+0000 7f7e5badf640 1 -- 172.21.15.81:0/3530071768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e54070870 con 0x7f7e540b3200 2024-06-29T16:01:16.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.332+0000 7f7e4a7fc640 1 -- 172.21.15.81:0/3530071768 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) ==== 74+0+17567 (secure 0 0 0) 0x7f7e4405fec0 con 0x7f7e540b3200 2024-06-29T16:01:16.333 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:16.333 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":271,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T16:01:14.568306+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:40.261824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:40.923216+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":271,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.e","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:0/3006664184":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6833/1824030697":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6832/1824030697":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/4135998660":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/432950336":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/1067289398":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/1087585815":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/2453006101":"2024-06-30T15:59:54.317815+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]},{"pool":37,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":37,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 -- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f7e380765a0 msgr2=0x7f7e38078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f7e380765a0 0x7f7e38078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 -- 172.21.15.81:0/3530071768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e540b3200 msgr2=0x7f7e541f2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e540b3200 0x7f7e541f2ba0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400c3a0 tx=0x7f7e4400c860 comp rx=0 tx=0).stop 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 -- 172.21.15.81:0/3530071768 shutdown_connections 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7f7e380765a0 0x7f7e38078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f7e540d0230 0x7f7e540ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f7e540b4f40 0x7f7e541e11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 --2- 172.21.15.81:0/3530071768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e540b3200 0x7f7e541f2ba0 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 -- 172.21.15.81:0/3530071768 >> 172.21.15.81:0/3530071768 conn(0x7f7e540b3680 msgr2=0x7f7e54071070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 -- 172.21.15.81:0/3530071768 shutdown_connections 2024-06-29T16:01:16.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.334+0000 7f7e5badf640 1 -- 172.21.15.81:0/3530071768 wait complete. 2024-06-29T16:01:16.346 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-29T14:22:30.719746+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-29T15:59:40.261824+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '265', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2024-06-29T15:59:40.923216+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 271, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-29T16:01:16.347 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-29T16:01:16.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.466+0000 7fc73ea9d640 1 -- 172.21.15.81:0/1633219922 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc7380d0940 msgr2=0x7fc7380b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.466+0000 7fc73ea9d640 1 --2- 172.21.15.81:0/1633219922 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc7380d0940 0x7fc7380b4f40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc730005fd0 tx=0x7fc73002a840 comp rx=0 tx=0).stop 2024-06-29T16:01:16.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.467+0000 7fc73ea9d640 1 -- 172.21.15.81:0/1633219922 shutdown_connections 2024-06-29T16:01:16.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.467+0000 7fc73ea9d640 1 --2- 172.21.15.81:0/1633219922 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc7380d0940 0x7fc7380b4f40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.467+0000 7fc73ea9d640 1 --2- 172.21.15.81:0/1633219922 >> v2:172.21.15.167:3300/0 conn(0x7fc7380b3200 0x7fc7380cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.467+0000 7fc73ea9d640 1 -- 172.21.15.81:0/1633219922 >> 172.21.15.81:0/1633219922 conn(0x7fc7380b3680 msgr2=0x7fc7380b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:16.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.467+0000 7fc73ea9d640 1 -- 172.21.15.81:0/1633219922 shutdown_connections 2024-06-29T16:01:16.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.467+0000 7fc73ea9d640 1 -- 172.21.15.81:0/1633219922 wait complete. 2024-06-29T16:01:16.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.468+0000 7fc73ea9d640 1 Processor -- start 2024-06-29T16:01:16.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.468+0000 7fc73ea9d640 1 -- start start 2024-06-29T16:01:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc73ea9d640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7380b3200 0x7fc7381f1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc73ea9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7380cccf0 0x7fc7380c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc73ea9d640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc73809eeb0 0x7fc73809f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc73ea9d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc7380b5c50 con 0x7fc7380cccf0 2024-06-29T16:01:16.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc73ea9d640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc7380b5ad0 con 0x7fc7380b3200 2024-06-29T16:01:16.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc73ea9d640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc7380b5dd0 con 0x7fc73809eeb0 2024-06-29T16:01:16.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc737fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7380cccf0 0x7fc7380c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc737fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7380cccf0 0x7fc7380c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36256/0 (socket says 172.21.15.81:36256) 2024-06-29T16:01:16.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc737fff640 1 -- 172.21.15.81:0/2242138730 learned_addr learned my addr 172.21.15.81:0/2242138730 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:16.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc73c839640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7380b3200 0x7fc7381f1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc73d03a640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc73809eeb0 0x7fc73809f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc737fff640 1 -- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc73809eeb0 msgr2=0x7fc73809f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc737fff640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc73809eeb0 0x7fc73809f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc737fff640 1 -- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7380b3200 msgr2=0x7fc7381f1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc737fff640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7380b3200 0x7fc7381f1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc737fff640 1 -- 172.21.15.81:0/2242138730 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73809f800 con 0x7fc7380cccf0 2024-06-29T16:01:16.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc73d03a640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc73809eeb0 0x7fc73809f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:16.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.470+0000 7fc73c839640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7380b3200 0x7fc7381f1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:16.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.471+0000 7fc737fff640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7380cccf0 0x7fc7380c7960 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fc72400c530 tx=0x7fc72400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:16.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.471+0000 7fc7357fa640 1 -- 172.21.15.81:0/2242138730 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724016020 con 0x7fc7380cccf0 2024-06-29T16:01:16.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.471+0000 7fc7357fa640 1 -- 172.21.15.81:0/2242138730 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc724012920 con 0x7fc7380cccf0 2024-06-29T16:01:16.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.471+0000 7fc73ea9d640 1 -- 172.21.15.81:0/2242138730 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7380c67d0 con 0x7fc7380cccf0 2024-06-29T16:01:16.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.471+0000 7fc7357fa640 1 -- 172.21.15.81:0/2242138730 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724012d70 con 0x7fc7380cccf0 2024-06-29T16:01:16.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.471+0000 7fc73ea9d640 1 -- 172.21.15.81:0/2242138730 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7380c6d90 con 0x7fc7380cccf0 2024-06-29T16:01:16.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.472+0000 7fc73ea9d640 1 -- 172.21.15.81:0/2242138730 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7380c6540 con 0x7fc7380cccf0 2024-06-29T16:01:16.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.473+0000 7fc7357fa640 1 -- 172.21.15.81:0/2242138730 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 162) ==== 98468+0+0 (secure 0 0 0) 0x7fc724019020 con 0x7fc7380cccf0 2024-06-29T16:01:16.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.473+0000 7fc7357fa640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fc71c0765d0 0x7fc71c078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.473+0000 7fc7357fa640 1 -- 172.21.15.81:0/2242138730 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 6282+0+0 (secure 0 0 0) 0x7fc724097780 con 0x7fc7380cccf0 2024-06-29T16:01:16.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.473+0000 7fc73c839640 1 -- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fc71c0765d0 msgr2=0x7fc71c078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:16.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.473+0000 7fc73c839640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fc71c0765d0 0x7fc71c078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:16.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.482+0000 7fc7357fa640 1 -- 172.21.15.81:0/2242138730 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fc724020070 con 0x7fc7380cccf0 2024-06-29T16:01:16.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.669+0000 7fc73ea9d640 1 -- 172.21.15.81:0/2242138730 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc7380cc7b0 con 0x7fc7380cccf0 2024-06-29T16:01:16.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.670+0000 7fc7357fa640 1 -- 172.21.15.81:0/2242138730 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v271) ==== 93+0+10 (secure 0 0 0) 0x7fc724060500 con 0x7fc7380cccf0 2024-06-29T16:01:16.670 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-06-29T16:01:16.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 -- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fc71c0765d0 msgr2=0x7fc71c078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:16.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fc71c0765d0 0x7fc71c078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 -- 172.21.15.81:0/2242138730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7380cccf0 msgr2=0x7fc7380c7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7380cccf0 0x7fc7380c7960 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fc72400c530 tx=0x7fc72400c9f0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 -- 172.21.15.81:0/2242138730 shutdown_connections 2024-06-29T16:01:16.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fc71c0765d0 0x7fc71c078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc73809eeb0 0x7fc73809f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7380cccf0 0x7fc7380c7960 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 --2- 172.21.15.81:0/2242138730 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc7380b3200 0x7fc7381f1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 -- 172.21.15.81:0/2242138730 >> 172.21.15.81:0/2242138730 conn(0x7fc7380b3680 msgr2=0x7fc7380ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:16.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 -- 172.21.15.81:0/2242138730 shutdown_connections 2024-06-29T16:01:16.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.672+0000 7fc73ea9d640 1 -- 172.21.15.81:0/2242138730 wait complete. 2024-06-29T16:01:16.683 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-29T16:01:16.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.802+0000 7fbddeee4640 1 -- 172.21.15.81:0/284948172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdd813ec50 msgr2=0x7fbdd806c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.802+0000 7fbddeee4640 1 --2- 172.21.15.81:0/284948172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdd813ec50 0x7fbdd806c2f0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009830 tx=0x7fbdcc02a510 comp rx=0 tx=0).stop 2024-06-29T16:01:16.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.802+0000 7fbddeee4640 1 -- 172.21.15.81:0/284948172 shutdown_connections 2024-06-29T16:01:16.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.802+0000 7fbddeee4640 1 --2- 172.21.15.81:0/284948172 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbdd806c830 0x7fbdd815fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.802+0000 7fbddeee4640 1 --2- 172.21.15.81:0/284948172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdd813ec50 0x7fbdd806c2f0 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.802+0000 7fbddeee4640 1 -- 172.21.15.81:0/284948172 >> 172.21.15.81:0/284948172 conn(0x7fbdd80d0b70 msgr2=0x7fbdd80d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:16.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.802+0000 7fbddeee4640 1 -- 172.21.15.81:0/284948172 shutdown_connections 2024-06-29T16:01:16.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.802+0000 7fbddeee4640 1 -- 172.21.15.81:0/284948172 wait complete. 2024-06-29T16:01:16.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.804+0000 7fbddeee4640 1 Processor -- start 2024-06-29T16:01:16.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.804+0000 7fbddeee4640 1 -- start start 2024-06-29T16:01:16.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.805+0000 7fbddeee4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdd806c830 0x7fbdd80afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.805+0000 7fbddeee4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbdd813ec50 0x7fbdd8153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.805+0000 7fbddeee4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbdd81601f0 0x7fbdd815b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.805+0000 7fbddeee4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd8073620 con 0x7fbdd806c830 2024-06-29T16:01:16.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.805+0000 7fbddeee4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd80d0410 con 0x7fbdd81601f0 2024-06-29T16:01:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.805+0000 7fbddeee4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbdd80737a0 con 0x7fbdd813ec50 2024-06-29T16:01:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddcc80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdd806c830 0x7fbdd80afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddcc80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdd806c830 0x7fbdd80afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36268/0 (socket says 172.21.15.81:36268) 2024-06-29T16:01:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddcc80640 1 -- 172.21.15.81:0/1148730099 learned_addr learned my addr 172.21.15.81:0/1148730099 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbdd7fff640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbdd813ec50 0x7fbdd8153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddd481640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbdd81601f0 0x7fbdd815b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddcc80640 1 -- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbdd813ec50 msgr2=0x7fbdd8153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddcc80640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbdd813ec50 0x7fbdd8153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddcc80640 1 -- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbdd81601f0 msgr2=0x7fbdd815b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddcc80640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbdd81601f0 0x7fbdd815b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddcc80640 1 -- 172.21.15.81:0/1148730099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd8154310 con 0x7fbdd806c830 2024-06-29T16:01:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddcc80640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdd806c830 0x7fbdd80afd80 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009950 tx=0x7fbdcc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbdd57fa640 1 -- 172.21.15.81:0/1148730099 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc031d60 con 0x7fbdd806c830 2024-06-29T16:01:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbdd57fa640 1 -- 172.21.15.81:0/1148730099 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbdcc033e10 con 0x7fbdd806c830 2024-06-29T16:01:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddeee4640 1 -- 172.21.15.81:0/1148730099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd81545f0 con 0x7fbdd806c830 2024-06-29T16:01:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbdd57fa640 1 -- 172.21.15.81:0/1148730099 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc0332b0 con 0x7fbdd806c830 2024-06-29T16:01:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.806+0000 7fbddeee4640 1 -- 172.21.15.81:0/1148730099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd8154bb0 con 0x7fbdd806c830 2024-06-29T16:01:16.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.807+0000 7fbddeee4640 1 -- 172.21.15.81:0/1148730099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdd8073c70 con 0x7fbdd806c830 2024-06-29T16:01:16.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.808+0000 7fbdd57fa640 1 -- 172.21.15.81:0/1148730099 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 162) ==== 98468+0+0 (secure 0 0 0) 0x7fbdcc033450 con 0x7fbdd806c830 2024-06-29T16:01:16.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.808+0000 7fbdd57fa640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fbdbc0765a0 0x7fbdbc078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:16.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.808+0000 7fbdd7fff640 1 -- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fbdbc0765a0 msgr2=0x7fbdbc078ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:16.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.808+0000 7fbdd57fa640 1 -- 172.21.15.81:0/1148730099 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 6282+0+0 (secure 0 0 0) 0x7fbdcc0b8fe0 con 0x7fbdd806c830 2024-06-29T16:01:16.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.809+0000 7fbdd7fff640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fbdbc0765a0 0x7fbdbc078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:16.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:16.813+0000 7fbdd57fa640 1 -- 172.21.15.81:0/1148730099 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbdcc04a030 con 0x7fbdd806c830 2024-06-29T16:01:17.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.000+0000 7fbddeee4640 1 -- 172.21.15.81:0/1148730099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbdd8153080 con 0x7fbdd806c830 2024-06-29T16:01:17.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.001+0000 7fbdd57fa640 1 -- 172.21.15.81:0/1148730099 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v271) ==== 104+0+11 (secure 0 0 0) 0x7fbdcc03f030 con 0x7fbdd806c830 2024-06-29T16:01:17.001 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-06-29T16:01:17.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 -- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fbdbc0765a0 msgr2=0x7fbdbc078ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:17.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fbdbc0765a0 0x7fbdbc078ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 -- 172.21.15.81:0/1148730099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdd806c830 msgr2=0x7fbdd80afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdd806c830 0x7fbdd80afd80 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009950 tx=0x7fbdcc033c20 comp rx=0 tx=0).stop 2024-06-29T16:01:17.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 -- 172.21.15.81:0/1148730099 shutdown_connections 2024-06-29T16:01:17.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fbdbc0765a0 0x7fbdbc078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbdd81601f0 0x7fbdd815b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbdd813ec50 0x7fbdd8153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 --2- 172.21.15.81:0/1148730099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdd806c830 0x7fbdd80afd80 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 -- 172.21.15.81:0/1148730099 >> 172.21.15.81:0/1148730099 conn(0x7fbdd80d0b70 msgr2=0x7fbdd80b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 -- 172.21.15.81:0/1148730099 shutdown_connections 2024-06-29T16:01:17.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.003+0000 7fbddeee4640 1 -- 172.21.15.81:0/1148730099 wait complete. 2024-06-29T16:01:17.014 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-29T16:01:17.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.134+0000 7fef82d9e640 1 -- 172.21.15.81:0/1722277347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6c004800 msgr2=0x7fef6c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.134+0000 7fef82d9e640 1 --2- 172.21.15.81:0/1722277347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6c004800 0x7fef6c004c50 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7fef68009b30 tx=0x7fef6c005830 comp rx=0 tx=0).stop 2024-06-29T16:01:17.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.134+0000 7fef8133b640 1 -- 172.21.15.81:0/1722277347 reap_dead start 2024-06-29T16:01:17.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.134+0000 7fef82d9e640 1 -- 172.21.15.81:0/1722277347 shutdown_connections 2024-06-29T16:01:17.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.134+0000 7fef82d9e640 1 -- 172.21.15.81:0/1722277347 >> 172.21.15.81:0/1722277347 conn(0x7fef7c0b35a0 msgr2=0x7fef7c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.134+0000 7fef82d9e640 1 -- 172.21.15.81:0/1722277347 shutdown_connections 2024-06-29T16:01:17.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.134+0000 7fef82d9e640 1 -- 172.21.15.81:0/1722277347 wait complete. 2024-06-29T16:01:17.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.136+0000 7fef82d9e640 1 Processor -- start 2024-06-29T16:01:17.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.136+0000 7fef82d9e640 1 -- start start 2024-06-29T16:01:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef82d9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6c004800 0x7fef7c1dc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef82d9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef7c1d9ec0 0x7fef7c1f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef82d9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef7c0b58f0 0x7fef7c0b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef82d9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fef7c0b6df0 con 0x7fef6c004800 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef82d9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fef7c0b28c0 con 0x7fef7c0b58f0 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef82d9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fef7c0b6f70 con 0x7fef7c1d9ec0 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef80b3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6c004800 0x7fef7c1dc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef80b3a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6c004800 0x7fef7c1dc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36284/0 (socket says 172.21.15.81:36284) 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef80b3a640 1 -- 172.21.15.81:0/47559313 learned_addr learned my addr 172.21.15.81:0/47559313 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef7bfff640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef7c1d9ec0 0x7fef7c1f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.137+0000 7fef8133b640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef7c0b58f0 0x7fef7c0b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef80b3a640 1 -- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef7c1d9ec0 msgr2=0x7fef7c1f6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef80b3a640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef7c1d9ec0 0x7fef7c1f6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef80b3a640 1 -- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef7c0b58f0 msgr2=0x7fef7c0b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef80b3a640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef7c0b58f0 0x7fef7c0b5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef80b3a640 1 -- 172.21.15.81:0/47559313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7c0b6240 con 0x7fef6c004800 2024-06-29T16:01:17.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef80b3a640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6c004800 0x7fef7c1dc4d0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fef70003e70 tx=0x7fef70001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:17.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef797fa640 1 -- 172.21.15.81:0/47559313 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70011700 con 0x7fef6c004800 2024-06-29T16:01:17.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef797fa640 1 -- 172.21.15.81:0/47559313 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef70016020 con 0x7fef6c004800 2024-06-29T16:01:17.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef82d9e640 1 -- 172.21.15.81:0/47559313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef7c1d9550 con 0x7fef6c004800 2024-06-29T16:01:17.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef797fa640 1 -- 172.21.15.81:0/47559313 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70011b60 con 0x7fef6c004800 2024-06-29T16:01:17.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.138+0000 7fef82d9e640 1 -- 172.21.15.81:0/47559313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef7c1d9b10 con 0x7fef6c004800 2024-06-29T16:01:17.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.139+0000 7fef82d9e640 1 -- 172.21.15.81:0/47559313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef7c0d1480 con 0x7fef6c004800 2024-06-29T16:01:17.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.140+0000 7fef797fa640 1 -- 172.21.15.81:0/47559313 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 162) ==== 98468+0+0 (secure 0 0 0) 0x7fef70011d00 con 0x7fef6c004800 2024-06-29T16:01:17.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.140+0000 7fef797fa640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fef600765d0 0x7fef60078ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.141+0000 7fef7bfff640 1 -- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fef600765d0 msgr2=0x7fef60078ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.167:6832/2899520759 2024-06-29T16:01:17.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.141+0000 7fef797fa640 1 -- 172.21.15.81:0/47559313 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 6282+0+0 (secure 0 0 0) 0x7fef70098a60 con 0x7fef6c004800 2024-06-29T16:01:17.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.141+0000 7fef7bfff640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fef600765d0 0x7fef60078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-29T16:01:17.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.145+0000 7fef797fa640 1 -- 172.21.15.81:0/47559313 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fef7c0d1480 con 0x7fef6c004800 2024-06-29T16:01:17.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.213+0000 7fef797fa640 1 -- 172.21.15.81:0/47559313 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 163) ==== 97912+0+0 (secure 0 0 0) 0x7fef7005d8e0 con 0x7fef6c004800 2024-06-29T16:01:17.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.214+0000 7fef797fa640 1 -- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fef600765d0 msgr2=0x7fef60078ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-29T16:01:17.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.214+0000 7fef797fa640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fef600765d0 0x7fef60078ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.306 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:17.305+0000 7f9a6fa34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:17.307 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:17.305+0000 7f9a6fa34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:17.307 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:17.305+0000 7f9a6fa34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:17.307 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:17.305+0000 7f9a6fa34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:17.307 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:17.305+0000 7f9a6fa34640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:17.307 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:17.306+0000 7f9a72a3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:17.307 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:17.306+0000 7f9a72a3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:17.307 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:17.306+0000 7f9a72a3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:17.307 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:17.306+0000 7f9a72a3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:17.307 INFO:tasks.ceph.mgr.x.smithi167.stderr:2024-06-29T16:01:17.306+0000 7f9a72a3a640 -1 client.0 error registering admin socket command: (17) File exists 2024-06-29T16:01:17.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.375+0000 7fef82d9e640 1 -- 172.21.15.81:0/47559313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fef7c0cea30 con 0x7fef6c004800 2024-06-29T16:01:17.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.376+0000 7fef797fa640 1 -- 172.21.15.81:0/47559313 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v272) ==== 100+0+11 (secure 0 0 0) 0x7fef70020020 con 0x7fef6c004800 2024-06-29T16:01:17.376 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-06-29T16:01:17.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.378+0000 7fef82d9e640 1 -- 172.21.15.81:0/47559313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6c004800 msgr2=0x7fef7c1dc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.378+0000 7fef82d9e640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6c004800 0x7fef7c1dc4d0 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fef70003e70 tx=0x7fef70001800 comp rx=0 tx=0).stop 2024-06-29T16:01:17.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.378+0000 7fef82d9e640 1 -- 172.21.15.81:0/47559313 shutdown_connections 2024-06-29T16:01:17.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.378+0000 7fef82d9e640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:6832/2899520759,v1:172.21.15.167:6833/2899520759] conn(0x7fef600765d0 0x7fef60078ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.378+0000 7fef82d9e640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fef7c0b58f0 0x7fef7c0b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.378+0000 7fef82d9e640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fef7c1d9ec0 0x7fef7c1f6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.378+0000 7fef82d9e640 1 --2- 172.21.15.81:0/47559313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6c004800 0x7fef7c1dc4d0 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.378+0000 7fef82d9e640 1 -- 172.21.15.81:0/47559313 >> 172.21.15.81:0/47559313 conn(0x7fef7c0b35a0 msgr2=0x7fef7c1e3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.378+0000 7fef82d9e640 1 -- 172.21.15.81:0/47559313 shutdown_connections 2024-06-29T16:01:17.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.378+0000 7fef82d9e640 1 -- 172.21.15.81:0/47559313 wait complete. 2024-06-29T16:01:17.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T16:01:17.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.505+0000 7f47a1c3c640 1 -- 172.21.15.81:0/1390151830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4784005150 msgr2=0x7f478400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.505+0000 7f47a1c3c640 1 --2- 172.21.15.81:0/1390151830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4784005150 0x7f478400b9c0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f4794009a90 tx=0x7f4784005830 comp rx=0 tx=0).stop 2024-06-29T16:01:17.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.505+0000 7f479bfff640 1 -- 172.21.15.81:0/1390151830 reap_dead start 2024-06-29T16:01:17.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.505+0000 7f47a1c3c640 1 -- 172.21.15.81:0/1390151830 shutdown_connections 2024-06-29T16:01:17.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.505+0000 7f47a1c3c640 1 -- 172.21.15.81:0/1390151830 >> 172.21.15.81:0/1390151830 conn(0x7f479c06ccc0 msgr2=0x7f479c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.505+0000 7f47a1c3c640 1 -- 172.21.15.81:0/1390151830 shutdown_connections 2024-06-29T16:01:17.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.505+0000 7f47a1c3c640 1 -- 172.21.15.81:0/1390151830 wait complete. 2024-06-29T16:01:17.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.507+0000 7f47a1c3c640 1 Processor -- start 2024-06-29T16:01:17.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.507+0000 7f47a1c3c640 1 -- start start 2024-06-29T16:01:17.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f47a1c3c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4784005150 0x7f479c0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f47a1c3c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f479c14e710 0x7f479c0aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f47a1c3c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c0a9960 0x7f479c0a9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f47a1c3c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f479c0b47c0 con 0x7f479c0a9960 2024-06-29T16:01:17.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f47a1c3c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f479c0b4640 con 0x7f479c14e710 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f47a1c3c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f479c06ffe0 con 0x7f4784005150 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f479bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c0a9960 0x7f479c0a9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f479bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c0a9960 0x7f479c0a9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36294/0 (socket says 172.21.15.81:36294) 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f479b7fe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4784005150 0x7f479c0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f479affd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f479c14e710 0x7f479c0aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f479bfff640 1 -- 172.21.15.81:0/2448881035 learned_addr learned my addr 172.21.15.81:0/2448881035 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f479bfff640 1 -- 172.21.15.81:0/2448881035 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4784005150 msgr2=0x7f479c0ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f479bfff640 1 --2- 172.21.15.81:0/2448881035 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4784005150 0x7f479c0ae5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f479bfff640 1 -- 172.21.15.81:0/2448881035 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f479c14e710 msgr2=0x7f479c0aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f479bfff640 1 --2- 172.21.15.81:0/2448881035 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f479c14e710 0x7f479c0aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.508+0000 7f479bfff640 1 -- 172.21.15.81:0/2448881035 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479c0ad820 con 0x7f479c0a9960 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.509+0000 7f479bfff640 1 --2- 172.21.15.81:0/2448881035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c0a9960 0x7f479c0a9d70 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7f479402d0e0 tx=0x7f4794009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:17.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.509+0000 7f4783fff640 1 -- 172.21.15.81:0/2448881035 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4794047070 con 0x7f479c0a9960 2024-06-29T16:01:17.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.509+0000 7f4783fff640 1 -- 172.21.15.81:0/2448881035 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47940362e0 con 0x7f479c0a9960 2024-06-29T16:01:17.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.509+0000 7f4783fff640 1 -- 172.21.15.81:0/2448881035 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4794003970 con 0x7f479c0a9960 2024-06-29T16:01:17.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.509+0000 7f47a1c3c640 1 -- 172.21.15.81:0/2448881035 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f479c14b850 con 0x7f479c0a9960 2024-06-29T16:01:17.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.509+0000 7f47a1c3c640 1 -- 172.21.15.81:0/2448881035 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f479c0adac0 con 0x7f479c0a9960 2024-06-29T16:01:17.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.510+0000 7f47a1c3c640 1 -- 172.21.15.81:0/2448881035 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f479c073900 con 0x7f479c0a9960 2024-06-29T16:01:17.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.510+0000 7f4783fff640 1 -- 172.21.15.81:0/2448881035 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 163) ==== 97912+0+0 (secure 0 0 0) 0x7f479404a070 con 0x7f479c0a9960 2024-06-29T16:01:17.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.511+0000 7f4783fff640 1 -- 172.21.15.81:0/2448881035 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 6711+0+0 (secure 0 0 0) 0x7f47940ba500 con 0x7f479c0a9960 2024-06-29T16:01:17.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.515+0000 7f4783fff640 1 -- 172.21.15.81:0/2448881035 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f479c073900 con 0x7f479c0a9960 2024-06-29T16:01:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.738+0000 7f47a1c3c640 1 -- 172.21.15.81:0/2448881035 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f479c068f50 con 0x7f479c0a9960 2024-06-29T16:01:17.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.738+0000 7f4783fff640 1 -- 172.21.15.81:0/2448881035 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5238 (secure 0 0 0) 0x7f479c068f50 con 0x7f479c0a9960 2024-06-29T16:01:17.739 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:17.739 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":154,"btime":"2024-06-29T15:59:45:615681+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10196,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2932449820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2932449820},{"type":"v1","addr":"172.21.15.167:6837","nonce":2932449820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/317823138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":317823138},{"type":"v1","addr":"172.21.15.81:6835","nonce":317823138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3582280449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3582280449},{"type":"v1","addr":"172.21.15.167:6836","nonce":3582280449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:41.965720+0000","modified":"2024-06-29T15:59:44.983893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13371},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13371":{"gid":13371,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/28043779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":28043779},{"type":"v1","addr":"172.21.15.81:6834","nonce":28043779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13371,"qdb_cluster":[13371]},"id":18}]} 2024-06-29T16:01:17.739 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 154 2024-06-29T16:01:17.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.740+0000 7f47a1c3c640 1 -- 172.21.15.81:0/2448881035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c0a9960 msgr2=0x7f479c0a9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.740+0000 7f47a1c3c640 1 --2- 172.21.15.81:0/2448881035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c0a9960 0x7f479c0a9d70 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7f479402d0e0 tx=0x7f4794009480 comp rx=0 tx=0).stop 2024-06-29T16:01:17.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.741+0000 7f47a1c3c640 1 -- 172.21.15.81:0/2448881035 shutdown_connections 2024-06-29T16:01:17.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.741+0000 7f47a1c3c640 1 --2- 172.21.15.81:0/2448881035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c0a9960 0x7f479c0a9d70 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.741+0000 7f47a1c3c640 1 --2- 172.21.15.81:0/2448881035 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f479c14e710 0x7f479c0aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.741+0000 7f47a1c3c640 1 --2- 172.21.15.81:0/2448881035 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4784005150 0x7f479c0ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.741+0000 7f47a1c3c640 1 -- 172.21.15.81:0/2448881035 >> 172.21.15.81:0/2448881035 conn(0x7f479c06ccc0 msgr2=0x7f479c15c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.741+0000 7f47a1c3c640 1 -- 172.21.15.81:0/2448881035 shutdown_connections 2024-06-29T16:01:17.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.741+0000 7f47a1c3c640 1 -- 172.21.15.81:0/2448881035 wait complete. 2024-06-29T16:01:17.752 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T16:01:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.865+0000 7f929eed6640 1 -- 172.21.15.81:0/2465291382 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9288003a00 msgr2=0x7f9288003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.865+0000 7f929eed6640 1 --2- 172.21.15.81:0/2465291382 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9288003a00 0x7f9288003e70 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f928c00c4a0 tx=0x7f9288005830 comp rx=0 tx=0).stop 2024-06-29T16:01:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.865+0000 7f929d473640 1 -- 172.21.15.81:0/2465291382 reap_dead start 2024-06-29T16:01:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.865+0000 7f929eed6640 1 -- 172.21.15.81:0/2465291382 shutdown_connections 2024-06-29T16:01:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.865+0000 7f929eed6640 1 -- 172.21.15.81:0/2465291382 >> 172.21.15.81:0/2465291382 conn(0x7f929806ccc0 msgr2=0x7f92980b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.865+0000 7f929eed6640 1 -- 172.21.15.81:0/2465291382 shutdown_connections 2024-06-29T16:01:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.865+0000 7f929eed6640 1 -- 172.21.15.81:0/2465291382 wait complete. 2024-06-29T16:01:17.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.867+0000 7f929eed6640 1 Processor -- start 2024-06-29T16:01:17.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.867+0000 7f929eed6640 1 -- start start 2024-06-29T16:01:17.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.868+0000 7f929eed6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9288003a00 0x7f9298145b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.868+0000 7f929eed6640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9298135380 0x7f929815be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.868+0000 7f929eed6640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9298135790 0x7f929815c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:17.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.868+0000 7f929eed6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f92980b47c0 con 0x7f9288003a00 2024-06-29T16:01:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.868+0000 7f929eed6640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f92980b4640 con 0x7f9298135790 2024-06-29T16:01:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.868+0000 7f929eed6640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f929806edd0 con 0x7f9298135380 2024-06-29T16:01:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.868+0000 7f929cc72640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9288003a00 0x7f9298145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.868+0000 7f929cc72640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9288003a00 0x7f9298145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36312/0 (socket says 172.21.15.81:36312) 2024-06-29T16:01:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.868+0000 7f929cc72640 1 -- 172.21.15.81:0/906827011 learned_addr learned my addr 172.21.15.81:0/906827011 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f9297fff640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9298135380 0x7f929815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f929d473640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9298135790 0x7f929815c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f929cc72640 1 -- 172.21.15.81:0/906827011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9298135380 msgr2=0x7f929815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f929cc72640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9298135380 0x7f929815be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f929cc72640 1 -- 172.21.15.81:0/906827011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9298135790 msgr2=0x7f929815c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f929cc72640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9298135790 0x7f929815c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f929cc72640 1 -- 172.21.15.81:0/906827011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929814e170 con 0x7f9288003a00 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f9297fff640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9298135380 0x7f929815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f929d473640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9298135790 0x7f929815c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f929cc72640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9288003a00 0x7f9298145b40 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f928c02b310 tx=0x7f928c006f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f92957fa640 1 -- 172.21.15.81:0/906827011 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c0065f0 con 0x7f9288003a00 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f92957fa640 1 -- 172.21.15.81:0/906827011 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f928c02b660 con 0x7f9288003a00 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f92957fa640 1 -- 172.21.15.81:0/906827011 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c02ba70 con 0x7f9288003a00 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f929eed6640 1 -- 172.21.15.81:0/906827011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929814e450 con 0x7f9288003a00 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.869+0000 7f929eed6640 1 -- 172.21.15.81:0/906827011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929814ea10 con 0x7f9288003a00 2024-06-29T16:01:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.870+0000 7f929eed6640 1 -- 172.21.15.81:0/906827011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f929806f330 con 0x7f9288003a00 2024-06-29T16:01:17.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.871+0000 7f92957fa640 1 -- 172.21.15.81:0/906827011 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 163) ==== 97912+0+0 (secure 0 0 0) 0x7f928c02bc10 con 0x7f9288003a00 2024-06-29T16:01:17.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.872+0000 7f92957fa640 1 -- 172.21.15.81:0/906827011 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 6711+0+0 (secure 0 0 0) 0x7f928c0b7800 con 0x7f9288003a00 2024-06-29T16:01:17.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:17.876+0000 7f92957fa640 1 -- 172.21.15.81:0/906827011 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f929806ffb0 con 0x7f9288003a00 2024-06-29T16:01:18.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.064+0000 7f929eed6640 1 -- 172.21.15.81:0/906827011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f929814caa0 con 0x7f9288003a00 2024-06-29T16:01:18.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.065+0000 7f92957fa640 1 -- 172.21.15.81:0/906827011 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v272) ==== 74+0+18197 (secure 0 0 0) 0x7f928c0811c0 con 0x7f9288003a00 2024-06-29T16:01:18.065 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:18.066 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":272,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T16:01:17.202936+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:40.261824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:40.923216+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":271,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.e","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/2899520759":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1667571922":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/853755847":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1133468043":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/707331717":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1199836798":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/4142686836":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1553695616":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/4002034537":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/3409979994":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/3006664184":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6833/1824030697":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6832/1824030697":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/4135998660":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6832/2899520759":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/432950336":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/1067289398":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/1087585815":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/2453006101":"2024-06-30T15:59:54.317815+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]},{"pool":37,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T16:01:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.067+0000 7f929eed6640 1 -- 172.21.15.81:0/906827011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9288003a00 msgr2=0x7f9298145b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.067+0000 7f929eed6640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9288003a00 0x7f9298145b40 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f928c02b310 tx=0x7f928c006f10 comp rx=0 tx=0).stop 2024-06-29T16:01:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.067+0000 7f929eed6640 1 -- 172.21.15.81:0/906827011 shutdown_connections 2024-06-29T16:01:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.067+0000 7f929eed6640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f9298135790 0x7f929815c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.067+0000 7f929eed6640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f9298135380 0x7f929815be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.067+0000 7f929eed6640 1 --2- 172.21.15.81:0/906827011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9288003a00 0x7f9298145b40 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.067+0000 7f929eed6640 1 -- 172.21.15.81:0/906827011 >> 172.21.15.81:0/906827011 conn(0x7f929806ccc0 msgr2=0x7f929815d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.067+0000 7f929eed6640 1 -- 172.21.15.81:0/906827011 shutdown_connections 2024-06-29T16:01:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.067+0000 7f929eed6640 1 -- 172.21.15.81:0/906827011 wait complete. 2024-06-29T16:01:18.079 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T16:01:18.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T16:01:18.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.192+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/403178426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb806ef90 msgr2=0x7fbfb806f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.192+0000 7fbfbe8d1640 1 --2- 172.21.15.81:0/403178426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb806ef90 0x7fbfb806f3a0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009830 tx=0x7fbfac02a510 comp rx=0 tx=0).stop 2024-06-29T16:01:18.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.192+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/403178426 shutdown_connections 2024-06-29T16:01:18.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.192+0000 7fbfbe8d1640 1 --2- 172.21.15.81:0/403178426 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb806ef90 0x7fbfb806f3a0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.192+0000 7fbfbe8d1640 1 --2- 172.21.15.81:0/403178426 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb80b57b0 0x7fbfb815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.192+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/403178426 >> 172.21.15.81:0/403178426 conn(0x7fbfb80d0930 msgr2=0x7fbfb80d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.192+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/403178426 shutdown_connections 2024-06-29T16:01:18.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.192+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/403178426 wait complete. 2024-06-29T16:01:18.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.194+0000 7fbfbe8d1640 1 Processor -- start 2024-06-29T16:01:18.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.194+0000 7fbfbe8d1640 1 -- start start 2024-06-29T16:01:18.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfbe8d1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb806ef90 0x7fbfb814f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfbe8d1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb80b57b0 0x7fbfb8152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfbe8d1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb80ce1d0 0x7fbfb8153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfbe8d1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb80cecf0 con 0x7fbfb806ef90 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfbe8d1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb80b4b80 con 0x7fbfb80b57b0 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfbe8d1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb80cee70 con 0x7fbfb80ce1d0 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb806ef90 0x7fbfb814f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb806ef90 0x7fbfb814f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36316/0 (socket says 172.21.15.81:36316) 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfb7fff640 1 -- 172.21.15.81:0/361116173 learned_addr learned my addr 172.21.15.81:0/361116173 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfb77fe640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb80b57b0 0x7fbfb8152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.195+0000 7fbfbce6e640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb80ce1d0 0x7fbfb8153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfb7fff640 1 -- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb80ce1d0 msgr2=0x7fbfb8153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfb7fff640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb80ce1d0 0x7fbfb8153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfb7fff640 1 -- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb80b57b0 msgr2=0x7fbfb8152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfb7fff640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb80b57b0 0x7fbfb8152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfb7fff640 1 -- 172.21.15.81:0/361116173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb8156f30 con 0x7fbfb806ef90 2024-06-29T16:01:18.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfb7fff640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb806ef90 0x7fbfb814f340 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800e7c0 tx=0x7fbfa800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:18.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/361116173 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa800cc60 con 0x7fbfb806ef90 2024-06-29T16:01:18.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/361116173 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfa800ce00 con 0x7fbfb806ef90 2024-06-29T16:01:18.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/361116173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb814fe90 con 0x7fbfb806ef90 2024-06-29T16:01:18.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/361116173 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8005280 con 0x7fbfb806ef90 2024-06-29T16:01:18.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.196+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/361116173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb8150450 con 0x7fbfb806ef90 2024-06-29T16:01:18.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.197+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/361116173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfb806e2a0 con 0x7fbfb806ef90 2024-06-29T16:01:18.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.198+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/361116173 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 163) ==== 97912+0+0 (secure 0 0 0) 0x7fbfa8005420 con 0x7fbfb806ef90 2024-06-29T16:01:18.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.198+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/361116173 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 6711+0+0 (secure 0 0 0) 0x7fbfa8097cf0 con 0x7fbfb806ef90 2024-06-29T16:01:18.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.202+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/361116173 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbfb806e2a0 con 0x7fbfb806ef90 2024-06-29T16:01:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.213+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/361116173 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 164) ==== 98039+0+0 (secure 0 0 0) 0x7fbfa805c940 con 0x7fbfb806ef90 2024-06-29T16:01:18.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.214+0000 7fbfb4ff9640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbfa407bd90 0x7fbfa407e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.217+0000 7fbfb77fe640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbfa407bd90 0x7fbfa407e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.218+0000 7fbfb77fe640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbfa407bd90 0x7fbfa407e1b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbfac034000 tx=0x7fbfac004730 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:18.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.422+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/361116173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfb806f3a0 con 0x7fbfb806ef90 2024-06-29T16:01:18.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.422+0000 7fbfb4ff9640 1 -- 172.21.15.81:0/361116173 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5238 (secure 0 0 0) 0x7fbfa806de80 con 0x7fbfb806ef90 2024-06-29T16:01:18.423 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:18.423 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":154,"btime":"2024-06-29T15:59:45:615681+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10196,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2932449820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2932449820},{"type":"v1","addr":"172.21.15.167:6837","nonce":2932449820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/317823138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":317823138},{"type":"v1","addr":"172.21.15.81:6835","nonce":317823138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3582280449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3582280449},{"type":"v1","addr":"172.21.15.167:6836","nonce":3582280449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:41.965720+0000","modified":"2024-06-29T15:59:44.983893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13371},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13371":{"gid":13371,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/28043779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":28043779},{"type":"v1","addr":"172.21.15.81:6834","nonce":28043779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13371,"qdb_cluster":[13371]},"id":18}]} 2024-06-29T16:01:18.423 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 154 2024-06-29T16:01:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.424+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbfa407bd90 msgr2=0x7fbfa407e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.424+0000 7fbfbe8d1640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbfa407bd90 0x7fbfa407e1b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbfac034000 tx=0x7fbfac004730 comp rx=0 tx=0).stop 2024-06-29T16:01:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.424+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/361116173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb806ef90 msgr2=0x7fbfb814f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.424+0000 7fbfbe8d1640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb806ef90 0x7fbfb814f340 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800e7c0 tx=0x7fbfa800ec80 comp rx=0 tx=0).stop 2024-06-29T16:01:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.424+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/361116173 shutdown_connections 2024-06-29T16:01:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.424+0000 7fbfbe8d1640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbfb80ce1d0 0x7fbfb8153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.424+0000 7fbfbe8d1640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbfa407bd90 0x7fbfa407e1b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.424+0000 7fbfbe8d1640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbfb80b57b0 0x7fbfb8152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.424+0000 7fbfbe8d1640 1 --2- 172.21.15.81:0/361116173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfb806ef90 0x7fbfb814f340 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.424+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/361116173 >> 172.21.15.81:0/361116173 conn(0x7fbfb80d0930 msgr2=0x7fbfb8159500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.425+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/361116173 shutdown_connections 2024-06-29T16:01:18.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.425+0000 7fbfbe8d1640 1 -- 172.21.15.81:0/361116173 wait complete. 2024-06-29T16:01:18.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T16:01:18.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.549+0000 7f65a89eb640 1 -- 172.21.15.81:0/3977625388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a40b7290 msgr2=0x7f65a40b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.549+0000 7f65a89eb640 1 --2- 172.21.15.81:0/3977625388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a40b7290 0x7f65a40b76a0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 crypto rx=0x7f65a40b9b80 tx=0x7f659c02aaa0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.549+0000 7f65a89eb640 1 -- 172.21.15.81:0/3977625388 shutdown_connections 2024-06-29T16:01:18.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.549+0000 7f65a89eb640 1 --2- 172.21.15.81:0/3977625388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a40b7290 0x7f65a40b76a0 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.549+0000 7f65a89eb640 1 --2- 172.21.15.81:0/3977625388 >> v2:172.21.15.167:3300/0 conn(0x7f65a40b5250 0x7f65a40d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.549+0000 7f65a89eb640 1 --2- 172.21.15.81:0/3977625388 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a40b9040 0x7f65a40d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.549+0000 7f65a89eb640 1 -- 172.21.15.81:0/3977625388 >> 172.21.15.81:0/3977625388 conn(0x7f65a409cf30 msgr2=0x7f65a40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.549+0000 7f65a89eb640 1 -- 172.21.15.81:0/3977625388 shutdown_connections 2024-06-29T16:01:18.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.549+0000 7f65a89eb640 1 -- 172.21.15.81:0/3977625388 wait complete. 2024-06-29T16:01:18.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.551+0000 7f65a89eb640 1 Processor -- start 2024-06-29T16:01:18.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.551+0000 7f65a89eb640 1 -- start start 2024-06-29T16:01:18.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a89eb640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a40b5250 0x7f65a409ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a89eb640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a40b7290 0x7f65a414d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a89eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a40b9040 0x7f65a414df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a89eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f65a40b7f70 con 0x7f65a40b9040 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a89eb640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f65a40b7df0 con 0x7f65a40b7290 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a89eb640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f65a40b80f0 con 0x7f65a40b5250 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a40b9040 0x7f65a414df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a40b9040 0x7f65a414df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36328/0 (socket says 172.21.15.81:36328) 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a2d9d640 1 -- 172.21.15.81:0/2871727064 learned_addr learned my addr 172.21.15.81:0/2871727064 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a259c640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a40b5250 0x7f65a409ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a1d9b640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a40b7290 0x7f65a414d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a2d9d640 1 -- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a40b5250 msgr2=0x7f65a409ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a2d9d640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a40b5250 0x7f65a409ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a2d9d640 1 -- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a40b7290 msgr2=0x7f65a414d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a2d9d640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a40b7290 0x7f65a414d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a2d9d640 1 -- 172.21.15.81:0/2871727064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a40b09c0 con 0x7f65a40b9040 2024-06-29T16:01:18.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a259c640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a40b5250 0x7f65a409ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:18.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a2d9d640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a40b9040 0x7f65a414df70 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7f659c02b090 tx=0x7f659c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:18.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f6592ffd640 1 -- 172.21.15.81:0/2871727064 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c004250 con 0x7f65a40b9040 2024-06-29T16:01:18.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f6592ffd640 1 -- 172.21.15.81:0/2871727064 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f659c0043f0 con 0x7f65a40b9040 2024-06-29T16:01:18.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a89eb640 1 -- 172.21.15.81:0/2871727064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a40b0ca0 con 0x7f65a40b9040 2024-06-29T16:01:18.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f6592ffd640 1 -- 172.21.15.81:0/2871727064 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c004840 con 0x7f65a40b9040 2024-06-29T16:01:18.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.553+0000 7f65a89eb640 1 -- 172.21.15.81:0/2871727064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a409db40 con 0x7f65a40b9040 2024-06-29T16:01:18.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.554+0000 7f65a89eb640 1 -- 172.21.15.81:0/2871727064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a40b76a0 con 0x7f65a40b9040 2024-06-29T16:01:18.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.555+0000 7f6592ffd640 1 -- 172.21.15.81:0/2871727064 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 164) ==== 98039+0+0 (secure 0 0 0) 0x7f659c02c040 con 0x7f65a40b9040 2024-06-29T16:01:18.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.555+0000 7f6592ffd640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6584076150 0x7f6584078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.555+0000 7f6592ffd640 1 -- 172.21.15.81:0/2871727064 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 6711+0+0 (secure 0 0 0) 0x7f659c0b89e0 con 0x7f65a40b9040 2024-06-29T16:01:18.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.556+0000 7f65a259c640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6584076150 0x7f6584078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.556+0000 7f65a259c640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6584076150 0x7f6584078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6594005d70 tx=0x7f6594005d00 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:18.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.560+0000 7f6592ffd640 1 -- 172.21.15.81:0/2871727064 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f659c032030 con 0x7f65a40b9040 2024-06-29T16:01:18.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.780+0000 7f65a89eb640 1 -- 172.21.15.81:0/2871727064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65a40689c0 con 0x7f65a40b9040 2024-06-29T16:01:18.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.781+0000 7f6592ffd640 1 -- 172.21.15.81:0/2871727064 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5238 (secure 0 0 0) 0x7f659c08a030 con 0x7f65a40b9040 2024-06-29T16:01:18.781 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:18.781 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":154,"btime":"2024-06-29T15:59:45:615681+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10196,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2932449820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2932449820},{"type":"v1","addr":"172.21.15.167:6837","nonce":2932449820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/317823138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":317823138},{"type":"v1","addr":"172.21.15.81:6835","nonce":317823138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3582280449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3582280449},{"type":"v1","addr":"172.21.15.167:6836","nonce":3582280449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-29T15:59:41.965720+0000","modified":"2024-06-29T15:59:44.983893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":13371},"failed":[],"damaged":[],"stopped":[],"info":{"gid_13371":{"gid":13371,"name":"a","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6834/28043779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":28043779},{"type":"v1","addr":"172.21.15.81:6834","nonce":28043779}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":13371,"qdb_cluster":[13371]},"id":18}]} 2024-06-29T16:01:18.781 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 154 2024-06-29T16:01:18.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 -- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6584076150 msgr2=0x7f6584078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6584076150 0x7f6584078660 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6594005d70 tx=0x7f6594005d00 comp rx=0 tx=0).stop 2024-06-29T16:01:18.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 -- 172.21.15.81:0/2871727064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a40b9040 msgr2=0x7f65a414df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a40b9040 0x7f65a414df70 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7f659c02b090 tx=0x7f659c003f10 comp rx=0 tx=0).stop 2024-06-29T16:01:18.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 -- 172.21.15.81:0/2871727064 shutdown_connections 2024-06-29T16:01:18.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a40b9040 0x7f65a414df70 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6584076150 0x7f6584078660 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65a40b7290 0x7f65a414d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 --2- 172.21.15.81:0/2871727064 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65a40b5250 0x7f65a409ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 -- 172.21.15.81:0/2871727064 >> 172.21.15.81:0/2871727064 conn(0x7f65a409cf30 msgr2=0x7f65a40d2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 -- 172.21.15.81:0/2871727064 shutdown_connections 2024-06-29T16:01:18.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.783+0000 7f65a89eb640 1 -- 172.21.15.81:0/2871727064 wait complete. 2024-06-29T16:01:18.794 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T16:01:18.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.909+0000 7fec43ff5640 1 -- 172.21.15.81:0/1019207745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec3c0ce1d0 msgr2=0x7fec3c0ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.909+0000 7fec43ff5640 1 --2- 172.21.15.81:0/1019207745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec3c0ce1d0 0x7fec3c0ce5e0 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7fec3c0b53d0 tx=0x7fec3402aa00 comp rx=0 tx=0).stop 2024-06-29T16:01:18.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.909+0000 7fec43ff5640 1 -- 172.21.15.81:0/1019207745 shutdown_connections 2024-06-29T16:01:18.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.909+0000 7fec43ff5640 1 --2- 172.21.15.81:0/1019207745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec3c0ce1d0 0x7fec3c0ce5e0 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.909+0000 7fec43ff5640 1 --2- 172.21.15.81:0/1019207745 >> v2:172.21.15.167:3300/0 conn(0x7fec3c06ef90 0x7fec3c06f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.909+0000 7fec43ff5640 1 --2- 172.21.15.81:0/1019207745 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec3c0b57b0 0x7fec3c15aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.909+0000 7fec43ff5640 1 -- 172.21.15.81:0/1019207745 >> 172.21.15.81:0/1019207745 conn(0x7fec3c0d0930 msgr2=0x7fec3c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:18.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.909+0000 7fec43ff5640 1 -- 172.21.15.81:0/1019207745 shutdown_connections 2024-06-29T16:01:18.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.909+0000 7fec43ff5640 1 -- 172.21.15.81:0/1019207745 wait complete. 2024-06-29T16:01:18.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.911+0000 7fec43ff5640 1 Processor -- start 2024-06-29T16:01:18.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.911+0000 7fec43ff5640 1 -- start start 2024-06-29T16:01:18.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec43ff5640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec3c06ef90 0x7fec3c1e4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec43ff5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec3c0b57b0 0x7fec3c1e12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec43ff5640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec3c0ce1d0 0x7fec3c06c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec43ff5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c0ceeb0 con 0x7fec3c0b57b0 2024-06-29T16:01:18.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec43ff5640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c0ced30 con 0x7fec3c06ef90 2024-06-29T16:01:18.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec43ff5640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fec3c0cf030 con 0x7fec3c0ce1d0 2024-06-29T16:01:18.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec41590640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec3c0b57b0 0x7fec3c1e12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec41590640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec3c0b57b0 0x7fec3c1e12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36342/0 (socket says 172.21.15.81:36342) 2024-06-29T16:01:18.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec41590640 1 -- 172.21.15.81:0/451655397 learned_addr learned my addr 172.21.15.81:0/451655397 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:18.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec41d91640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec3c06ef90 0x7fec3c1e4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec42592640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec3c0ce1d0 0x7fec3c06c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec41590640 1 -- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec3c0ce1d0 msgr2=0x7fec3c06c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec41590640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec3c0ce1d0 0x7fec3c06c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec41590640 1 -- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec3c06ef90 msgr2=0x7fec3c1e4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:18.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec41590640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec3c06ef90 0x7fec3c1e4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:18.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.912+0000 7fec41590640 1 -- 172.21.15.81:0/451655397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3c1e04e0 con 0x7fec3c0b57b0 2024-06-29T16:01:18.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.913+0000 7fec41590640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec3c0b57b0 0x7fec3c1e12a0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00d6d0 tx=0x7fec2c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:18.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.913+0000 7fec327fc640 1 -- 172.21.15.81:0/451655397 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2c017070 con 0x7fec3c0b57b0 2024-06-29T16:01:18.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.913+0000 7fec327fc640 1 -- 172.21.15.81:0/451655397 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec2c009b10 con 0x7fec3c0b57b0 2024-06-29T16:01:18.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.913+0000 7fec43ff5640 1 -- 172.21.15.81:0/451655397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec3c1e07c0 con 0x7fec3c0b57b0 2024-06-29T16:01:18.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.913+0000 7fec327fc640 1 -- 172.21.15.81:0/451655397 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2c010460 con 0x7fec3c0b57b0 2024-06-29T16:01:18.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.913+0000 7fec43ff5640 1 -- 172.21.15.81:0/451655397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec3c1e5060 con 0x7fec3c0b57b0 2024-06-29T16:01:18.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.914+0000 7fec43ff5640 1 -- 172.21.15.81:0/451655397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec3c0d17b0 con 0x7fec3c0b57b0 2024-06-29T16:01:18.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.914+0000 7fec327fc640 1 -- 172.21.15.81:0/451655397 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 164) ==== 98039+0+0 (secure 0 0 0) 0x7fec2c01a070 con 0x7fec3c0b57b0 2024-06-29T16:01:18.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.915+0000 7fec327fc640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fec20076120 0x7fec20078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:18.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.915+0000 7fec327fc640 1 -- 172.21.15.81:0/451655397 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 6711+0+0 (secure 0 0 0) 0x7fec2c097770 con 0x7fec3c0b57b0 2024-06-29T16:01:18.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.915+0000 7fec41d91640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fec20076120 0x7fec20078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:18.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.916+0000 7fec41d91640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fec20076120 0x7fec20078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fec380045b0 tx=0x7fec3800b2e0 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:18.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:18.919+0000 7fec327fc640 1 -- 172.21.15.81:0/451655397 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fec3c0d17b0 con 0x7fec3c0b57b0 2024-06-29T16:01:19.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.099+0000 7fec43ff5640 1 -- 172.21.15.81:0/451655397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec3c157e80 con 0x7fec3c0b57b0 2024-06-29T16:01:19.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.101+0000 7fec327fc640 1 -- 172.21.15.81:0/451655397 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v272) ==== 74+0+18197 (secure 0 0 0) 0x7fec2c0609d0 con 0x7fec3c0b57b0 2024-06-29T16:01:19.101 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:19.101 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":272,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T16:01:17.202936+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-06-29T15:59:40.261824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"265","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-06-29T15:59:40.923216+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":271,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"37.e","mappings":[{"from":2,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/2899520759":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1667571922":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/853755847":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1133468043":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/707331717":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1199836798":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/4142686836":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1553695616":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/4002034537":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/3409979994":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/3006664184":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6833/1824030697":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6832/1824030697":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/4135998660":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6832/2899520759":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/432950336":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/1067289398":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/1087585815":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/2453006101":"2024-06-30T15:59:54.317815+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]},{"pool":37,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T16:01:19.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 -- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fec20076120 msgr2=0x7fec20078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fec20076120 0x7fec20078630 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fec380045b0 tx=0x7fec3800b2e0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 -- 172.21.15.81:0/451655397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec3c0b57b0 msgr2=0x7fec3c1e12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec3c0b57b0 0x7fec3c1e12a0 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7fec2c00d6d0 tx=0x7fec2c00db90 comp rx=0 tx=0).stop 2024-06-29T16:01:19.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 -- 172.21.15.81:0/451655397 shutdown_connections 2024-06-29T16:01:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fec3c0ce1d0 0x7fec3c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec3c0b57b0 0x7fec3c1e12a0 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fec20076120 0x7fec20078630 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 --2- 172.21.15.81:0/451655397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fec3c06ef90 0x7fec3c1e4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 -- 172.21.15.81:0/451655397 >> 172.21.15.81:0/451655397 conn(0x7fec3c0d0930 msgr2=0x7fec3c1556e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 -- 172.21.15.81:0/451655397 shutdown_connections 2024-06-29T16:01:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.103+0000 7fec43ff5640 1 -- 172.21.15.81:0/451655397 wait complete. 2024-06-29T16:01:19.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-06-29T16:01:19.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.232+0000 7f54c012b640 1 -- 172.21.15.81:0/2669225951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b80d0940 msgr2=0x7f54b80b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.232+0000 7f54c012b640 1 --2- 172.21.15.81:0/2669225951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b80d0940 0x7f54b80b4f40 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 crypto rx=0x7f54b0005fd0 tx=0x7f54b002aa00 comp rx=0 tx=0).stop 2024-06-29T16:01:19.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.232+0000 7f54c012b640 1 -- 172.21.15.81:0/2669225951 shutdown_connections 2024-06-29T16:01:19.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.232+0000 7f54c012b640 1 --2- 172.21.15.81:0/2669225951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b80d0940 0x7f54b80b4f40 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.232+0000 7f54c012b640 1 --2- 172.21.15.81:0/2669225951 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54b80b3200 0x7f54b80cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.232+0000 7f54c012b640 1 -- 172.21.15.81:0/2669225951 >> 172.21.15.81:0/2669225951 conn(0x7f54b80b3680 msgr2=0x7f54b80b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:19.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.232+0000 7f54c012b640 1 -- 172.21.15.81:0/2669225951 shutdown_connections 2024-06-29T16:01:19.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.232+0000 7f54c012b640 1 -- 172.21.15.81:0/2669225951 wait complete. 2024-06-29T16:01:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.234+0000 7f54c012b640 1 Processor -- start 2024-06-29T16:01:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.234+0000 7f54c012b640 1 -- start start 2024-06-29T16:01:19.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.235+0000 7f54c012b640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54b80b3200 0x7f54b8155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.235+0000 7f54c012b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b80cccf0 0x7f54b80b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.235+0000 7f54c012b640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54b8158ca0 0x7f54b80b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.235+0000 7f54c012b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f54b80b5c50 con 0x7f54b80cccf0 2024-06-29T16:01:19.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.235+0000 7f54c012b640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f54b80b5ad0 con 0x7f54b8158ca0 2024-06-29T16:01:19.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.235+0000 7f54c012b640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f54b80b5dd0 con 0x7f54b80b3200 2024-06-29T16:01:19.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.235+0000 7f54bd6c6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b80cccf0 0x7f54b80b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.235+0000 7f54bd6c6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b80cccf0 0x7f54b80b0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36358/0 (socket says 172.21.15.81:36358) 2024-06-29T16:01:19.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.235+0000 7f54bd6c6640 1 -- 172.21.15.81:0/441282828 learned_addr learned my addr 172.21.15.81:0/441282828 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54bdec7640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54b80b3200 0x7f54b8155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54be6c8640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54b8158ca0 0x7f54b80b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54bd6c6640 1 -- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54b80b3200 msgr2=0x7f54b8155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54bd6c6640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54b80b3200 0x7f54b8155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54bd6c6640 1 -- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54b8158ca0 msgr2=0x7f54b80b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54bd6c6640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54b8158ca0 0x7f54b80b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54bd6c6640 1 -- 172.21.15.81:0/441282828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b809e560 con 0x7f54b80cccf0 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54bd6c6640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b80cccf0 0x7f54b80b0a70 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7f54b400b3c0 tx=0x7f54b400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54aa7fc640 1 -- 172.21.15.81:0/441282828 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4011830 con 0x7f54b80cccf0 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54aa7fc640 1 -- 172.21.15.81:0/441282828 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54b4016070 con 0x7f54b80cccf0 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54c012b640 1 -- 172.21.15.81:0/441282828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b809e840 con 0x7f54b80cccf0 2024-06-29T16:01:19.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54aa7fc640 1 -- 172.21.15.81:0/441282828 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4011da0 con 0x7f54b80cccf0 2024-06-29T16:01:19.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.236+0000 7f54c012b640 1 -- 172.21.15.81:0/441282828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b8158780 con 0x7f54b80cccf0 2024-06-29T16:01:19.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.237+0000 7f54c012b640 1 -- 172.21.15.81:0/441282828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54b80cc7b0 con 0x7f54b80cccf0 2024-06-29T16:01:19.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.238+0000 7f54aa7fc640 1 -- 172.21.15.81:0/441282828 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 164) ==== 98039+0+0 (secure 0 0 0) 0x7f54b401b460 con 0x7f54b80cccf0 2024-06-29T16:01:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.239+0000 7f54aa7fc640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f549c076120 0x7f549c078630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.239+0000 7f54aa7fc640 1 -- 172.21.15.81:0/441282828 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 6711+0+0 (secure 0 0 0) 0x7f54b4097360 con 0x7f54b80cccf0 2024-06-29T16:01:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.239+0000 7f54bdec7640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f549c076120 0x7f549c078630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.240+0000 7f54bdec7640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f549c076120 0x7f549c078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f54ac005fd0 tx=0x7f54ac005850 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.243+0000 7f54aa7fc640 1 -- 172.21.15.81:0/441282828 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f54b4020020 con 0x7f54b80cccf0 2024-06-29T16:01:19.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.465+0000 7f54c012b640 1 -- 172.21.15.81:0/441282828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f54b809e2b0 con 0x7f54b80cccf0 2024-06-29T16:01:19.466 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-06-29T16:01:19.466+0000 7f8b4ddcb640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-29T16:01:19.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.468+0000 7f54aa7fc640 1 -- 172.21.15.81:0/441282828 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v155) ==== 161+0+0 (secure 0 0 0) 0x7f54b4060b90 con 0x7f54b80cccf0 2024-06-29T16:01:19.468 INFO:teuthology.orchestra.run.smithi081.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-29T16:01:19.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 -- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f549c076120 msgr2=0x7f549c078630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f549c076120 0x7f549c078630 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f54ac005fd0 tx=0x7f54ac005850 comp rx=0 tx=0).stop 2024-06-29T16:01:19.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 -- 172.21.15.81:0/441282828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b80cccf0 msgr2=0x7f54b80b0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b80cccf0 0x7f54b80b0a70 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7f54b400b3c0 tx=0x7f54b400b880 comp rx=0 tx=0).stop 2024-06-29T16:01:19.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 -- 172.21.15.81:0/441282828 shutdown_connections 2024-06-29T16:01:19.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f549c076120 0x7f549c078630 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f54b8158ca0 0x7f54b80b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b80cccf0 0x7f54b80b0a70 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 --2- 172.21.15.81:0/441282828 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f54b80b3200 0x7f54b8155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 -- 172.21.15.81:0/441282828 >> 172.21.15.81:0/441282828 conn(0x7f54b80b3680 msgr2=0x7f54b80cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:19.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 -- 172.21.15.81:0/441282828 shutdown_connections 2024-06-29T16:01:19.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.471+0000 7f54c012b640 1 -- 172.21.15.81:0/441282828 wait complete. 2024-06-29T16:01:19.486 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-06-29T16:01:19.537 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-06-29T16:01:19.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.606+0000 7fdc04d43640 1 -- 172.21.15.81:0/2873799379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbec003a50 msgr2=0x7fdbec003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.606+0000 7fdc04d43640 1 --2- 172.21.15.81:0/2873799379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbec003a50 0x7fdbec003ec0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009b50 tx=0x7fdbec005830 comp rx=0 tx=0).stop 2024-06-29T16:01:19.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.606+0000 7fdc04d43640 1 -- 172.21.15.81:0/2873799379 shutdown_connections 2024-06-29T16:01:19.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.606+0000 7fdc04d43640 1 --2- 172.21.15.81:0/2873799379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbec003a50 0x7fdbec003ec0 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.606+0000 7fdc04d43640 1 -- 172.21.15.81:0/2873799379 >> 172.21.15.81:0/2873799379 conn(0x7fdc000b1730 msgr2=0x7fdc000b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:19.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.606+0000 7fdc04d43640 1 -- 172.21.15.81:0/2873799379 shutdown_connections 2024-06-29T16:01:19.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.606+0000 7fdc04d43640 1 -- 172.21.15.81:0/2873799379 wait complete. 2024-06-29T16:01:19.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.608+0000 7fdc04d43640 1 Processor -- start 2024-06-29T16:01:19.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.608+0000 7fdc04d43640 1 -- start start 2024-06-29T16:01:19.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.609+0000 7fdc04d43640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbec003a50 0x7fdc001f2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.609+0000 7fdc04d43640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc0015ac20 0x7fdc000cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.609+0000 7fdc04d43640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc001f27b0 0x7fdc000cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.609+0000 7fdc04d43640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00155920 con 0x7fdbec003a50 2024-06-29T16:01:19.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.609+0000 7fdc04d43640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001557a0 con 0x7fdc0015ac20 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.609+0000 7fdc04d43640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fdc0006f760 con 0x7fdc001f27b0 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbfe59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbec003a50 0x7fdc001f2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbfe59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbec003a50 0x7fdc001f2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36396/0 (socket says 172.21.15.81:36396) 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbfe59c640 1 -- 172.21.15.81:0/342629888 learned_addr learned my addr 172.21.15.81:0/342629888 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbfed9d640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc001f27b0 0x7fdc000cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbf7fff640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc0015ac20 0x7fdc000cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbfe59c640 1 -- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc001f27b0 msgr2=0x7fdc000cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbfe59c640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc001f27b0 0x7fdc000cdef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbfe59c640 1 -- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc0015ac20 msgr2=0x7fdc000cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbfe59c640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc0015ac20 0x7fdc000cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbfe59c640 1 -- 172.21.15.81:0/342629888 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc00158df0 con 0x7fdbec003a50 2024-06-29T16:01:19.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbfe59c640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbec003a50 0x7fdc001f2270 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009a70 tx=0x7fdbf00336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:19.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbf6ffd640 1 -- 172.21.15.81:0/342629888 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0038040 con 0x7fdbec003a50 2024-06-29T16:01:19.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbf6ffd640 1 -- 172.21.15.81:0/342629888 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdbf0004030 con 0x7fdbec003a50 2024-06-29T16:01:19.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdc04d43640 1 -- 172.21.15.81:0/342629888 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc00157f00 con 0x7fdbec003a50 2024-06-29T16:01:19.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdbf6ffd640 1 -- 172.21.15.81:0/342629888 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0004480 con 0x7fdbec003a50 2024-06-29T16:01:19.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.610+0000 7fdc04d43640 1 -- 172.21.15.81:0/342629888 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc001584c0 con 0x7fdbec003a50 2024-06-29T16:01:19.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.611+0000 7fdc04d43640 1 -- 172.21.15.81:0/342629888 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc0006fcc0 con 0x7fdbec003a50 2024-06-29T16:01:19.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.612+0000 7fdbf6ffd640 1 -- 172.21.15.81:0/342629888 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 164) ==== 98039+0+0 (secure 0 0 0) 0x7fdbf00338e0 con 0x7fdbec003a50 2024-06-29T16:01:19.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.613+0000 7fdbf6ffd640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fdbe0076150 0x7fdbe0078660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:19.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.613+0000 7fdbf6ffd640 1 -- 172.21.15.81:0/342629888 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6797+0+0 (secure 0 0 0) 0x7fdbf00b7e90 con 0x7fdbec003a50 2024-06-29T16:01:19.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.613+0000 7fdbf7fff640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fdbe0076150 0x7fdbe0078660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:19.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.614+0000 7fdbf7fff640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fdbe0076150 0x7fdbe0078660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdc00158be0 tx=0x7fdbe8006040 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:19.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.617+0000 7fdbf6ffd640 1 -- 172.21.15.81:0/342629888 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fdbf00810a0 con 0x7fdbec003a50 2024-06-29T16:01:19.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:19.837+0000 7fdc04d43640 1 -- 172.21.15.81:0/342629888 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fdc00073730 con 0x7fdbec003a50 2024-06-29T16:01:20.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.205+0000 7fdbf6ffd640 1 -- 172.21.15.81:0/342629888 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 165) ==== 98247+0+0 (secure 0 0 0) 0x7fdbf007d330 con 0x7fdbec003a50 2024-06-29T16:01:20.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.469+0000 7fdbf6ffd640 1 -- 172.21.15.81:0/342629888 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v156) ==== 104+0+0 (secure 0 0 0) 0x7fdbf00b2940 con 0x7fdbec003a50 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.472+0000 7fdc04d43640 1 -- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fdbe0076150 msgr2=0x7fdbe0078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.472+0000 7fdc04d43640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fdbe0076150 0x7fdbe0078660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdc00158be0 tx=0x7fdbe8006040 comp rx=0 tx=0).stop 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.472+0000 7fdc04d43640 1 -- 172.21.15.81:0/342629888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbec003a50 msgr2=0x7fdc001f2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.472+0000 7fdc04d43640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbec003a50 0x7fdc001f2270 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009a70 tx=0x7fdbf00336f0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.472+0000 7fdc04d43640 1 -- 172.21.15.81:0/342629888 shutdown_connections 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.472+0000 7fdc04d43640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fdc001f27b0 0x7fdc000cdef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.472+0000 7fdc04d43640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fdbe0076150 0x7fdbe0078660 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.472+0000 7fdc04d43640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fdc0015ac20 0x7fdc000cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.472+0000 7fdc04d43640 1 --2- 172.21.15.81:0/342629888 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbec003a50 0x7fdc001f2270 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.472+0000 7fdc04d43640 1 -- 172.21.15.81:0/342629888 >> 172.21.15.81:0/342629888 conn(0x7fdc000b1730 msgr2=0x7fdc000b0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.473+0000 7fdc04d43640 1 -- 172.21.15.81:0/342629888 shutdown_connections 2024-06-29T16:01:20.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.473+0000 7fdc04d43640 1 -- 172.21.15.81:0/342629888 wait complete. 2024-06-29T16:01:20.486 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-29T16:01:20.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.606+0000 7f82e110a640 1 -- 172.21.15.81:0/1530907256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82dc1555e0 msgr2=0x7f82dc15bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.606+0000 7f82e110a640 1 --2- 172.21.15.81:0/1530907256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82dc1555e0 0x7f82dc15bea0 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7f82dc071f10 tx=0x7f82d402aa30 comp rx=0 tx=0).stop 2024-06-29T16:01:20.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.606+0000 7f82e110a640 1 -- 172.21.15.81:0/1530907256 shutdown_connections 2024-06-29T16:01:20.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.606+0000 7f82e110a640 1 --2- 172.21.15.81:0/1530907256 >> v2:172.21.15.167:3300/0 conn(0x7f82dc0708f0 0x7f82dc0b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.606+0000 7f82e110a640 1 --2- 172.21.15.81:0/1530907256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82dc1555e0 0x7f82dc15bea0 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.606+0000 7f82e110a640 1 -- 172.21.15.81:0/1530907256 >> 172.21.15.81:0/1530907256 conn(0x7f82dc0d08f0 msgr2=0x7f82dc0d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:20.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.607+0000 7f82e110a640 1 -- 172.21.15.81:0/1530907256 shutdown_connections 2024-06-29T16:01:20.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.607+0000 7f82e110a640 1 -- 172.21.15.81:0/1530907256 wait complete. 2024-06-29T16:01:20.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.609+0000 7f82e110a640 1 Processor -- start 2024-06-29T16:01:20.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.609+0000 7f82e110a640 1 -- start start 2024-06-29T16:01:20.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82e110a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82dc0708f0 0x7f82dc14b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82e110a640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82dc0af730 0x7f82dc157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82e110a640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82dc1555e0 0x7f82dc1591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82e110a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc06ff10 con 0x7f82dc0708f0 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82e110a640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc06fd90 con 0x7f82dc1555e0 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82e110a640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f82dc070090 con 0x7f82dc0af730 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82dad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82dc0708f0 0x7f82dc14b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82dad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82dc0708f0 0x7f82dc14b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42686/0 (socket says 172.21.15.81:42686) 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82dad9d640 1 -- 172.21.15.81:0/1331754076 learned_addr learned my addr 172.21.15.81:0/1331754076 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82da59c640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82dc0af730 0x7f82dc157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82db59e640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82dc1555e0 0x7f82dc1591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82dad9d640 1 -- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82dc0af730 msgr2=0x7f82dc157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82dad9d640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82dc0af730 0x7f82dc157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82dad9d640 1 -- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82dc1555e0 msgr2=0x7f82dc1591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82dad9d640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82dc1555e0 0x7f82dc1591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82dad9d640 1 -- 172.21.15.81:0/1331754076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82dc1f78c0 con 0x7f82dc0708f0 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82da59c640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82dc0af730 0x7f82dc157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.610+0000 7f82db59e640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82dc1555e0 0x7f82dc1591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.611+0000 7f82dad9d640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82dc0708f0 0x7f82dc14b4d0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f82d402b020 tx=0x7f82d4003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.611+0000 7f82d37fe640 1 -- 172.21.15.81:0/1331754076 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4033050 con 0x7f82dc0708f0 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.611+0000 7f82d37fe640 1 -- 172.21.15.81:0/1331754076 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82d40045f0 con 0x7f82dc0708f0 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.611+0000 7f82e110a640 1 -- 172.21.15.81:0/1331754076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82dc152e10 con 0x7f82dc0708f0 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.611+0000 7f82d37fe640 1 -- 172.21.15.81:0/1331754076 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4038450 con 0x7f82dc0708f0 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.611+0000 7f82e110a640 1 -- 172.21.15.81:0/1331754076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82dc14f450 con 0x7f82dc0708f0 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.612+0000 7f82e110a640 1 -- 172.21.15.81:0/1331754076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82dc075c90 con 0x7f82dc0708f0 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.612+0000 7f82d37fe640 1 -- 172.21.15.81:0/1331754076 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 165) ==== 98247+0+0 (secure 0 0 0) 0x7f82d4036070 con 0x7f82dc0708f0 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.613+0000 7f82d37fe640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f82bc076390 0x7f82bc0788a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.613+0000 7f82d37fe640 1 -- 172.21.15.81:0/1331754076 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 6797+0+0 (secure 0 0 0) 0x7f82d40b9310 con 0x7f82dc0708f0 2024-06-29T16:01:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.613+0000 7f82da59c640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f82bc076390 0x7f82bc0788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:20.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.614+0000 7f82da59c640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f82bc076390 0x7f82bc0788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82dc15aa00 tx=0x7f82c4008040 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:20.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.617+0000 7f82d37fe640 1 -- 172.21.15.81:0/1331754076 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f82dc075c90 con 0x7f82dc0708f0 2024-06-29T16:01:20.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:20.795+0000 7f82e110a640 1 -- 172.21.15.81:0/1331754076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f82dc0d0560 con 0x7f82dc0708f0 2024-06-29T16:01:21.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.588+0000 7f82d37fe640 1 -- 172.21.15.81:0/1331754076 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v274) ==== 181+0+0 (secure 0 0 0) 0x7f82dc0b03f0 con 0x7f82dc0708f0 2024-06-29T16:01:21.589 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' removed 2024-06-29T16:01:21.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 -- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f82bc076390 msgr2=0x7f82bc0788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f82bc076390 0x7f82bc0788a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82dc15aa00 tx=0x7f82c4008040 comp rx=0 tx=0).stop 2024-06-29T16:01:21.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 -- 172.21.15.81:0/1331754076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82dc0708f0 msgr2=0x7f82dc14b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82dc0708f0 0x7f82dc14b4d0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f82d402b020 tx=0x7f82d4003f90 comp rx=0 tx=0).stop 2024-06-29T16:01:21.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 -- 172.21.15.81:0/1331754076 shutdown_connections 2024-06-29T16:01:21.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f82bc076390 0x7f82bc0788a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f82dc1555e0 0x7f82dc1591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f82dc0af730 0x7f82dc157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 --2- 172.21.15.81:0/1331754076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82dc0708f0 0x7f82dc14b4d0 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 -- 172.21.15.81:0/1331754076 >> 172.21.15.81:0/1331754076 conn(0x7f82dc0d08f0 msgr2=0x7f82dc0b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:21.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 -- 172.21.15.81:0/1331754076 shutdown_connections 2024-06-29T16:01:21.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.591+0000 7f82e110a640 1 -- 172.21.15.81:0/1331754076 wait complete. 2024-06-29T16:01:21.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-29T16:01:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.731+0000 7fabda495640 1 -- 172.21.15.81:0/3821342860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabd406ca00 msgr2=0x7fabd40cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.731+0000 7fabda495640 1 --2- 172.21.15.81:0/3821342860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabd406ca00 0x7fabd40cfa60 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7fabcc005fd0 tx=0x7fabcc02aa00 comp rx=0 tx=0).stop 2024-06-29T16:01:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.731+0000 7fabda495640 1 -- 172.21.15.81:0/3821342860 shutdown_connections 2024-06-29T16:01:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.731+0000 7fabda495640 1 --2- 172.21.15.81:0/3821342860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabd406ca00 0x7fabd40cfa60 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.731+0000 7fabda495640 1 --2- 172.21.15.81:0/3821342860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabd40d1410 0x7fabd406f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.731+0000 7fabda495640 1 -- 172.21.15.81:0/3821342860 >> 172.21.15.81:0/3821342860 conn(0x7fabd40d08f0 msgr2=0x7fabd40d0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.731+0000 7fabda495640 1 -- 172.21.15.81:0/3821342860 shutdown_connections 2024-06-29T16:01:21.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.731+0000 7fabda495640 1 -- 172.21.15.81:0/3821342860 wait complete. 2024-06-29T16:01:21.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.733+0000 7fabda495640 1 Processor -- start 2024-06-29T16:01:21.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.733+0000 7fabda495640 1 -- start start 2024-06-29T16:01:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.734+0000 7fabda495640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabd40d1410 0x7fabd4151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.734+0000 7fabda495640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fabd415c460 0x7fabd4151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.734+0000 7fabda495640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabd41581f0 0x7fabd41561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.734+0000 7fabda495640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fabd40cffa0 con 0x7fabd40d1410 2024-06-29T16:01:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.734+0000 7fabda495640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fabd415ca80 con 0x7fabd415c460 2024-06-29T16:01:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.734+0000 7fabda495640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fabd40d0120 con 0x7fabd41581f0 2024-06-29T16:01:21.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabd40d1410 0x7fabd4151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabd40d1410 0x7fabd4151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42700/0 (socket says 172.21.15.81:42700) 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd3fff640 1 -- 172.21.15.81:0/1366510866 learned_addr learned my addr 172.21.15.81:0/1366510866 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd8a32640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabd41581f0 0x7fabd41561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd37fe640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fabd415c460 0x7fabd4151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd3fff640 1 -- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabd41581f0 msgr2=0x7fabd41561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd3fff640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabd41581f0 0x7fabd41561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd3fff640 1 -- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fabd415c460 msgr2=0x7fabd4151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd3fff640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fabd415c460 0x7fabd4151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd3fff640 1 -- 172.21.15.81:0/1366510866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabd41526c0 con 0x7fabd40d1410 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd3fff640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabd40d1410 0x7fabd4151e70 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7fabc400d6d0 tx=0x7fabc400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd0ff9640 1 -- 172.21.15.81:0/1366510866 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4017070 con 0x7fabd40d1410 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd0ff9640 1 -- 172.21.15.81:0/1366510866 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabc4005330 con 0x7fabd40d1410 2024-06-29T16:01:21.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabda495640 1 -- 172.21.15.81:0/1366510866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabd41529a0 con 0x7fabd40d1410 2024-06-29T16:01:21.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabd0ff9640 1 -- 172.21.15.81:0/1366510866 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc40097c0 con 0x7fabd40d1410 2024-06-29T16:01:21.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.735+0000 7fabda495640 1 -- 172.21.15.81:0/1366510866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabd4152f60 con 0x7fabd40d1410 2024-06-29T16:01:21.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.736+0000 7fabda495640 1 -- 172.21.15.81:0/1366510866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabd4075c90 con 0x7fabd40d1410 2024-06-29T16:01:21.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.737+0000 7fabd0ff9640 1 -- 172.21.15.81:0/1366510866 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fabc4009960 con 0x7fabd40d1410 2024-06-29T16:01:21.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.737+0000 7fabd0ff9640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fabc0076420 0x7fabc0078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:21.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.737+0000 7fabd0ff9640 1 -- 172.21.15.81:0/1366510866 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 6359+0+0 (secure 0 0 0) 0x7fabc4097d50 con 0x7fabd40d1410 2024-06-29T16:01:21.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.738+0000 7fabd37fe640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fabc0076420 0x7fabc0078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:21.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.738+0000 7fabd37fe640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fabc0076420 0x7fabc0078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fabc8006fd0 tx=0x7fabc8008040 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:21.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.742+0000 7fabd0ff9640 1 -- 172.21.15.81:0/1366510866 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fabd4075c90 con 0x7fabd40d1410 2024-06-29T16:01:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:21.920+0000 7fabda495640 1 -- 172.21.15.81:0/1366510866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fabd40d0620 con 0x7fabd40d1410 2024-06-29T16:01:22.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.590+0000 7fabd0ff9640 1 -- 172.21.15.81:0/1366510866 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v275) ==== 169+0+0 (secure 0 0 0) 0x7fabc4061170 con 0x7fabd40d1410 2024-06-29T16:01:22.591 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' removed 2024-06-29T16:01:22.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.593+0000 7fabda495640 1 -- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fabc0076420 msgr2=0x7fabc0078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.593+0000 7fabda495640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fabc0076420 0x7fabc0078930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fabc8006fd0 tx=0x7fabc8008040 comp rx=0 tx=0).stop 2024-06-29T16:01:22.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.593+0000 7fabda495640 1 -- 172.21.15.81:0/1366510866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabd40d1410 msgr2=0x7fabd4151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.593+0000 7fabda495640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabd40d1410 0x7fabd4151e70 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7fabc400d6d0 tx=0x7fabc400db90 comp rx=0 tx=0).stop 2024-06-29T16:01:22.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.594+0000 7fabda495640 1 -- 172.21.15.81:0/1366510866 shutdown_connections 2024-06-29T16:01:22.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.594+0000 7fabda495640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fabd41581f0 0x7fabd41561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.594+0000 7fabda495640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fabc0076420 0x7fabc0078930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.594+0000 7fabda495640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fabd415c460 0x7fabd4151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.594+0000 7fabda495640 1 --2- 172.21.15.81:0/1366510866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fabd40d1410 0x7fabd4151e70 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.594+0000 7fabda495640 1 -- 172.21.15.81:0/1366510866 >> 172.21.15.81:0/1366510866 conn(0x7fabd40d08f0 msgr2=0x7fabd415da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:22.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.594+0000 7fabda495640 1 -- 172.21.15.81:0/1366510866 shutdown_connections 2024-06-29T16:01:22.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.594+0000 7fabda495640 1 -- 172.21.15.81:0/1366510866 wait complete. 2024-06-29T16:01:22.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 27 2024-06-29T16:01:22.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.728+0000 7f84b9209640 1 -- 172.21.15.81:0/1405359296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84b415ab80 msgr2=0x7f84b41650e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.728+0000 7f84b9209640 1 --2- 172.21.15.81:0/1405359296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84b415ab80 0x7f84b41650e0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f84ac005fd0 tx=0x7f84ac02a8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.728+0000 7f84b9209640 1 -- 172.21.15.81:0/1405359296 shutdown_connections 2024-06-29T16:01:22.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.728+0000 7f84b9209640 1 --2- 172.21.15.81:0/1405359296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84b415ab80 0x7f84b41650e0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.728+0000 7f84b9209640 1 -- 172.21.15.81:0/1405359296 >> 172.21.15.81:0/1405359296 conn(0x7f84b4072950 msgr2=0x7f84b40b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:22.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.728+0000 7f84b9209640 1 -- 172.21.15.81:0/1405359296 shutdown_connections 2024-06-29T16:01:22.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.728+0000 7f84b9209640 1 -- 172.21.15.81:0/1405359296 wait complete. 2024-06-29T16:01:22.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.730+0000 7f84b9209640 1 Processor -- start 2024-06-29T16:01:22.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.730+0000 7f84b9209640 1 -- start start 2024-06-29T16:01:22.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84b9209640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84b415ab80 0x7f84b4163480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84b9209640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84b415bad0 0x7f84b4159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84b9209640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84b4164b80 0x7f84b41f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84b9209640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f84b41659b0 con 0x7f84b4164b80 2024-06-29T16:01:22.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84b9209640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f84b4165830 con 0x7f84b415ab80 2024-06-29T16:01:22.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84b9209640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f84b4165b30 con 0x7f84b415bad0 2024-06-29T16:01:22.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84b359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84b4164b80 0x7f84b41f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84b359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84b4164b80 0x7f84b41f7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42728/0 (socket says 172.21.15.81:42728) 2024-06-29T16:01:22.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84b359e640 1 -- 172.21.15.81:0/1784550082 learned_addr learned my addr 172.21.15.81:0/1784550082 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:22.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84b2d9d640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84b415ab80 0x7f84b4163480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.731+0000 7f84aa59c640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84b415bad0 0x7f84b4159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84b359e640 1 -- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84b415bad0 msgr2=0x7f84b4159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84b359e640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84b415bad0 0x7f84b4159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84b359e640 1 -- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84b415ab80 msgr2=0x7f84b4163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84b359e640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84b415ab80 0x7f84b4163480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84b359e640 1 -- 172.21.15.81:0/1784550082 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b4155520 con 0x7f84b4164b80 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84aa59c640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84b415bad0 0x7f84b4159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84b2d9d640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84b415ab80 0x7f84b4163480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84b359e640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84b4164b80 0x7f84b41f7c20 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f84ac02aea0 tx=0x7f84ac0039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84abfff640 1 -- 172.21.15.81:0/1784550082 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac031990 con 0x7f84b4164b80 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84abfff640 1 -- 172.21.15.81:0/1784550082 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84ac031c20 con 0x7f84b4164b80 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84b9209640 1 -- 172.21.15.81:0/1784550082 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84b4154b10 con 0x7f84b4164b80 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84abfff640 1 -- 172.21.15.81:0/1784550082 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac033070 con 0x7f84b4164b80 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.732+0000 7f84b9209640 1 -- 172.21.15.81:0/1784550082 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84b41550d0 con 0x7f84b4164b80 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.733+0000 7f84b9209640 1 -- 172.21.15.81:0/1784550082 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84b4155740 con 0x7f84b4164b80 2024-06-29T16:01:22.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.734+0000 7f84abfff640 1 -- 172.21.15.81:0/1784550082 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f84ac033230 con 0x7f84b4164b80 2024-06-29T16:01:22.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.734+0000 7f84abfff640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f8494076420 0x7f8494078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:22.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.734+0000 7f84abfff640 1 -- 172.21.15.81:0/1784550082 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f84ac0b90e0 con 0x7f84b4164b80 2024-06-29T16:01:22.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.734+0000 7f84b2d9d640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f8494076420 0x7f8494078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:22.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.735+0000 7f84b2d9d640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f8494076420 0x7f8494078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f84a4006fd0 tx=0x7f84a4008040 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:22.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.738+0000 7f84abfff640 1 -- 172.21.15.81:0/1784550082 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f84ac03f030 con 0x7f84b4164b80 2024-06-29T16:01:22.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.919+0000 7f84b9209640 1 -- 172.21.15.81:0/1784550082 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config reset", "num": 27} v 0) -- 0x7f84b40753e0 con 0x7f84b4164b80 2024-06-29T16:01:22.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.923+0000 7f84abfff640 1 -- 172.21.15.81:0/1784550082 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 27}]=0 v87) ==== 71+0+0 (secure 0 0 0) 0x7f84ac082750 con 0x7f84b4164b80 2024-06-29T16:01:22.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.925+0000 7f84b9209640 1 -- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f8494076420 msgr2=0x7f8494078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.925+0000 7f84b9209640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f8494076420 0x7f8494078930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f84a4006fd0 tx=0x7f84a4008040 comp rx=0 tx=0).stop 2024-06-29T16:01:22.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.925+0000 7f84b9209640 1 -- 172.21.15.81:0/1784550082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84b4164b80 msgr2=0x7f84b41f7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:22.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.925+0000 7f84b9209640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84b4164b80 0x7f84b41f7c20 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f84ac02aea0 tx=0x7f84ac0039f0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.925+0000 7f84b9209640 1 -- 172.21.15.81:0/1784550082 shutdown_connections 2024-06-29T16:01:22.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.925+0000 7f84b9209640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84b4164b80 0x7f84b41f7c20 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.926+0000 7f84b9209640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f84b415bad0 0x7f84b4159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.926+0000 7f84b9209640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f8494076420 0x7f8494078930 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.926+0000 7f84b9209640 1 --2- 172.21.15.81:0/1784550082 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f84b415ab80 0x7f84b4163480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:22.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.926+0000 7f84b9209640 1 -- 172.21.15.81:0/1784550082 >> 172.21.15.81:0/1784550082 conn(0x7f84b4072950 msgr2=0x7f84b40b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:22.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.926+0000 7f84b9209640 1 -- 172.21.15.81:0/1784550082 shutdown_connections 2024-06-29T16:01:22.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:22.926+0000 7f84b9209640 1 -- 172.21.15.81:0/1784550082 wait complete. 2024-06-29T16:01:22.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01' 2024-06-29T16:01:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.049+0000 7fe98121f640 1 -- 172.21.15.81:0/550309922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c15d750 msgr2=0x7fe97c0afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.049+0000 7fe98121f640 1 --2- 172.21.15.81:0/550309922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c15d750 0x7fe97c0afb60 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0cf140 tx=0x7fe96c02a980 comp rx=0 tx=0).stop 2024-06-29T16:01:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.049+0000 7fe98121f640 1 -- 172.21.15.81:0/550309922 shutdown_connections 2024-06-29T16:01:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.049+0000 7fe98121f640 1 --2- 172.21.15.81:0/550309922 >> v2:172.21.15.167:3300/0 conn(0x7fe97c15c030 0x7fe97c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.049+0000 7fe98121f640 1 --2- 172.21.15.81:0/550309922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c15d750 0x7fe97c0afb60 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.050+0000 7fe98121f640 1 -- 172.21.15.81:0/550309922 >> 172.21.15.81:0/550309922 conn(0x7fe97c0d0930 msgr2=0x7fe97c0d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:23.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.050+0000 7fe98121f640 1 -- 172.21.15.81:0/550309922 shutdown_connections 2024-06-29T16:01:23.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.050+0000 7fe98121f640 1 -- 172.21.15.81:0/550309922 wait complete. 2024-06-29T16:01:23.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.052+0000 7fe98121f640 1 Processor -- start 2024-06-29T16:01:23.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.052+0000 7fe98121f640 1 -- start start 2024-06-29T16:01:23.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe98121f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c15c030 0x7fe97c1586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe98121f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe97c15c440 0x7fe97c0b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe98121f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97c15d750 0x7fe97c151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe98121f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c06cbc0 con 0x7fe97c15c030 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe98121f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c06ca40 con 0x7fe97c15d750 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe98121f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe97c160290 con 0x7fe97c15c440 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c15c030 0x7fe97c1586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c15c030 0x7fe97c1586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42744/0 (socket says 172.21.15.81:42744) 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97ad9d640 1 -- 172.21.15.81:0/2928910831 learned_addr learned my addr 172.21.15.81:0/2928910831 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97b59e640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97c15d750 0x7fe97c151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97a59c640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe97c15c440 0x7fe97c0b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97ad9d640 1 -- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe97c15c440 msgr2=0x7fe97c0b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:23.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97ad9d640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe97c15c440 0x7fe97c0b49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97ad9d640 1 -- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97c15d750 msgr2=0x7fe97c151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97ad9d640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97c15d750 0x7fe97c151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97ad9d640 1 -- 172.21.15.81:0/2928910831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97c0b5630 con 0x7fe97c15c030 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97b59e640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97c15d750 0x7fe97c151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.053+0000 7fe97ad9d640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c15c030 0x7fe97c1586b0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7fe96c000f80 tx=0x7fe96c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.054+0000 7fe9637fe640 1 -- 172.21.15.81:0/2928910831 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c034040 con 0x7fe97c15c030 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.054+0000 7fe9637fe640 1 -- 172.21.15.81:0/2928910831 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe96c004640 con 0x7fe97c15c030 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.054+0000 7fe9637fe640 1 -- 172.21.15.81:0/2928910831 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c045480 con 0x7fe97c15c030 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.054+0000 7fe98121f640 1 -- 172.21.15.81:0/2928910831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97c0ada50 con 0x7fe97c15c030 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.054+0000 7fe98121f640 1 -- 172.21.15.81:0/2928910831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe97c0ae010 con 0x7fe97c15c030 2024-06-29T16:01:23.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.055+0000 7fe98121f640 1 -- 172.21.15.81:0/2928910831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe97c0ccc00 con 0x7fe97c15c030 2024-06-29T16:01:23.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.055+0000 7fe9637fe640 1 -- 172.21.15.81:0/2928910831 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fe96c037020 con 0x7fe97c15c030 2024-06-29T16:01:23.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.056+0000 7fe9637fe640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe9680763f0 0x7fe968078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:23.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.056+0000 7fe9637fe640 1 -- 172.21.15.81:0/2928910831 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7fe96c0b9460 con 0x7fe97c15c030 2024-06-29T16:01:23.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.056+0000 7fe97a59c640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe9680763f0 0x7fe968078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:23.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.057+0000 7fe97a59c640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe9680763f0 0x7fe968078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe970005d70 tx=0x7fe970005d00 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:23.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.060+0000 7fe9637fe640 1 -- 172.21.15.81:0/2928910831 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe97c0ccc00 con 0x7fe97c15c030 2024-06-29T16:01:23.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.314+0000 7fe98121f640 1 -- 172.21.15.81:0/2928910831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]} v 0) -- 0x7fe97c06e2a0 con 0x7fe97c15c030 2024-06-29T16:01:23.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.603+0000 7fe9637fe640 1 -- 172.21.15.81:0/2928910831 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments.test_snap_schedule_subvol_and_group_arguments_01"]}]=0 v3271) ==== 201+0+0 (secure 0 0 0) 0x7fe97c06e2a0 con 0x7fe97c15c030 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 -- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe9680763f0 msgr2=0x7fe968078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe9680763f0 0x7fe968078900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe970005d70 tx=0x7fe970005d00 comp rx=0 tx=0).stop 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 -- 172.21.15.81:0/2928910831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c15c030 msgr2=0x7fe97c1586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c15c030 0x7fe97c1586b0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7fe96c000f80 tx=0x7fe96c003f90 comp rx=0 tx=0).stop 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 -- 172.21.15.81:0/2928910831 shutdown_connections 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe9680763f0 0x7fe968078900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe97c15d750 0x7fe97c151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe97c15c440 0x7fe97c0b49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 --2- 172.21.15.81:0/2928910831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c15c030 0x7fe97c1586b0 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 -- 172.21.15.81:0/2928910831 >> 172.21.15.81:0/2928910831 conn(0x7fe97c0d0930 msgr2=0x7fe97c15f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:23.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 -- 172.21.15.81:0/2928910831 shutdown_connections 2024-06-29T16:01:23.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:23.605+0000 7fe98121f640 1 -- 172.21.15.81:0/2928910831 wait complete. 2024-06-29T16:01:23.619 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. ... ERROR 2024-06-29T16:01:23.620 INFO:tasks.cephfs_test_runner: 2024-06-29T16:01:23.620 INFO:tasks.cephfs_test_runner:====================================================================== 2024-06-29T16:01:23.620 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-06-29T16:01:23.620 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-06-29T16:01:23.620 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-06-29T16:01:23.620 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-06-29T16:01:23.620 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs/test_snap_schedules.py", line 684, in test_snap_schedule_subvol_and_group_arguments_01 2024-06-29T16:01:23.620 INFO:tasks.cephfs_test_runner: self._fs_cmd('subvolume', 'rm', 'cephfs', 'sv01') 2024-06-29T16:01:23.620 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs/test_snap_schedules.py", line 63, in _fs_cmd 2024-06-29T16:01:23.620 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_manager.py", line 1652, in run_cluster_cmd 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/remote.py", line 523, in run 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 455, in run 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: r.wait() 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 161, in wait 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv01' 2024-06-29T16:01:23.621 INFO:tasks.cephfs_test_runner: 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner:Ran 15 tests in 5891.238s 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner: 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner: 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner:====================================================================== 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner:ERROR: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner:Test subvol schedule creation succeeds for default subvolgroup. 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs/test_snap_schedules.py", line 684, in test_snap_schedule_subvol_and_group_arguments_01 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner: self._fs_cmd('subvolume', 'rm', 'cephfs', 'sv01') 2024-06-29T16:01:23.622 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs/test_snap_schedules.py", line 63, in _fs_cmd 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: return self.get_ceph_cmd_stdout("fs", *args) 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/ceph_manager.py", line 1652, in run_cluster_cmd 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/remote.py", line 523, in run 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 455, in run 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: r.wait() 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 161, in wait 2024-06-29T16:01:23.623 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-06-29T16:01:23.624 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-06-29T16:01:23.624 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-06-29T16:01:23.624 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv01' 2024-06-29T16:01:23.624 INFO:tasks.cephfs_test_runner: 2024-06-29T16:01:23.624 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-06-29T16:01:23.802 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=92dce96014d640689d2fe9400fdde10f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_mchangir_ceph_019bb938d4977149d100ab4c3d2c706147295470/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-06-29T16:01:23.804 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-06-29T16:01:23.812 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-06-29T16:01:23.831 INFO:tasks.kclient:Unmounting kernel clients... 2024-06-29T16:01:23.831 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T16:01:23.832 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:23.852 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T16:01:23.853 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-06-29T16:01:23.853 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:23.909 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17627 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497612k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-06-29T16:01:23.909 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T16:01:23.909 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:23.929 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T16:01:23.929 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-06-29T16:01:23.929 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-29T16:01:23.988 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056332,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248804k,nr_inodes=812201,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497616k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-06-29T16:01:23.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:23.989 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T16:01:24.005 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-06-29T16:01:24.005 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-29T16:01:24.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-06-29T16:01:24.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.071 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-29T16:01:24.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link delete ceph-brx 2024-06-29T16:01:24.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.229 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-06-29T16:01:24.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link delete ceph-brx 2024-06-29T16:01:24.313 INFO:teuthology.orchestra.run.smithi081.stderr:Cannot find device "ceph-brx" 2024-06-29T16:01:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:01:24.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.314 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list 2024-06-29T16:01:24.334 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-06-29T16:01:24.335 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-06-29T16:01:24.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-06-29T16:01:24.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-06-29T16:01:24.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.472 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link delete ceph-brx 2024-06-29T16:01:24.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.573 DEBUG:teuthology.orchestra.run.smithi167:> ip netns list 2024-06-29T16:01:24.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-29T16:01:24.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo ip link delete ceph-brx 2024-06-29T16:01:24.657 INFO:teuthology.orchestra.run.smithi167.stderr:Cannot find device "ceph-brx" 2024-06-29T16:01:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:01:24.659 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-06-29T16:01:24.668 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-29T16:01:24.668 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T16:01:24.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.786+0000 7fd1e7eac640 1 -- 172.21.15.81:0/723756877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1cc005910 msgr2=0x7fd1cc005d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:24.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.786+0000 7fd1e7eac640 1 --2- 172.21.15.81:0/723756877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1cc005910 0x7fd1cc005d20 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0009b50 tx=0x7fd1cc006940 comp rx=0 tx=0).stop 2024-06-29T16:01:24.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.786+0000 7fd1e6449640 1 -- 172.21.15.81:0/723756877 reap_dead start 2024-06-29T16:01:24.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.787+0000 7fd1e7eac640 1 -- 172.21.15.81:0/723756877 shutdown_connections 2024-06-29T16:01:24.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.787+0000 7fd1e7eac640 1 -- 172.21.15.81:0/723756877 >> 172.21.15.81:0/723756877 conn(0x7fd1e006ccc0 msgr2=0x7fd1e00b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:24.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.787+0000 7fd1e7eac640 1 -- 172.21.15.81:0/723756877 shutdown_connections 2024-06-29T16:01:24.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.787+0000 7fd1e7eac640 1 -- 172.21.15.81:0/723756877 wait complete. 2024-06-29T16:01:24.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.788+0000 7fd1e7eac640 1 Processor -- start 2024-06-29T16:01:24.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.789+0000 7fd1e7eac640 1 -- start start 2024-06-29T16:01:24.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e7eac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1cc005910 0x7fd1e00ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:24.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e7eac640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e01f02e0 0x7fd1e01f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:24.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e7eac640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e01f06f0 0x7fd1e01f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e7eac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00b47c0 con 0x7fd1cc005910 2024-06-29T16:01:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e7eac640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e00b4640 con 0x7fd1e01f02e0 2024-06-29T16:01:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e7eac640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e006edd0 con 0x7fd1e01f06f0 2024-06-29T16:01:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e5c48640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1cc005910 0x7fd1e00ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e5c48640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1cc005910 0x7fd1e00ac0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42758/0 (socket says 172.21.15.81:42758) 2024-06-29T16:01:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e5c48640 1 -- 172.21.15.81:0/2551164397 learned_addr learned my addr 172.21.15.81:0/2551164397 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e5447640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e01f02e0 0x7fd1e01f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e6449640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e01f06f0 0x7fd1e01f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e5c48640 1 -- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e01f06f0 msgr2=0x7fd1e01f6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e5c48640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e01f06f0 0x7fd1e01f6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e5c48640 1 -- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e01f02e0 msgr2=0x7fd1e01f64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e5c48640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e01f02e0 0x7fd1e01f64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e5c48640 1 -- 172.21.15.81:0/2551164397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e00ac8b0 con 0x7fd1cc005910 2024-06-29T16:01:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1e5c48640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1cc005910 0x7fd1e00ac0f0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc006600 tx=0x7fd1dc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.790+0000 7fd1d67fc640 1 -- 172.21.15.81:0/2551164397 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc015070 con 0x7fd1cc005910 2024-06-29T16:01:24.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.791+0000 7fd1d67fc640 1 -- 172.21.15.81:0/2551164397 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1dc011900 con 0x7fd1cc005910 2024-06-29T16:01:24.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.791+0000 7fd1d67fc640 1 -- 172.21.15.81:0/2551164397 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc011cd0 con 0x7fd1cc005910 2024-06-29T16:01:24.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.791+0000 7fd1e7eac640 1 -- 172.21.15.81:0/2551164397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e00acb90 con 0x7fd1cc005910 2024-06-29T16:01:24.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.791+0000 7fd1e7eac640 1 -- 172.21.15.81:0/2551164397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e01f0c70 con 0x7fd1cc005910 2024-06-29T16:01:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.792+0000 7fd1e7eac640 1 -- 172.21.15.81:0/2551164397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e006f330 con 0x7fd1cc005910 2024-06-29T16:01:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.792+0000 7fd1d67fc640 1 -- 172.21.15.81:0/2551164397 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fd1dc018070 con 0x7fd1cc005910 2024-06-29T16:01:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.793+0000 7fd1d67fc640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd1c40763f0 0x7fd1c4078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.793+0000 7fd1d67fc640 1 -- 172.21.15.81:0/2551164397 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7fd1dc098730 con 0x7fd1cc005910 2024-06-29T16:01:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.793+0000 7fd1e5447640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd1c40763f0 0x7fd1c4078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.794+0000 7fd1e5447640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd1c40763f0 0x7fd1c4078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0035000 tx=0x7fd1d002d040 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:24.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.798+0000 7fd1d67fc640 1 -- 172.21.15.81:0/2551164397 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd1e006ffb0 con 0x7fd1cc005910 2024-06-29T16:01:24.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.977+0000 7fd1e7eac640 1 -- 172.21.15.81:0/2551164397 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd1e0151880 con 0x7fd1c40763f0 2024-06-29T16:01:24.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.977+0000 7fd1d67fc640 1 -- 172.21.15.81:0/2551164397 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13479 (secure 0 0 0) 0x7fd1e0151880 con 0x7fd1c40763f0 2024-06-29T16:01:24.978 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:24.978 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T16:01:24.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 -- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd1c40763f0 msgr2=0x7fd1c4078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:24.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd1c40763f0 0x7fd1c4078900 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0035000 tx=0x7fd1d002d040 comp rx=0 tx=0).stop 2024-06-29T16:01:24.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 -- 172.21.15.81:0/2551164397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1cc005910 msgr2=0x7fd1e00ac0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:24.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1cc005910 0x7fd1e00ac0f0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc006600 tx=0x7fd1dc0039c0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 -- 172.21.15.81:0/2551164397 shutdown_connections 2024-06-29T16:01:24.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd1e01f06f0 0x7fd1e01f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd1c40763f0 0x7fd1c4078900 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd1e01f02e0 0x7fd1e01f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 --2- 172.21.15.81:0/2551164397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1cc005910 0x7fd1e00ac0f0 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:24.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 -- 172.21.15.81:0/2551164397 >> 172.21.15.81:0/2551164397 conn(0x7fd1e006ccc0 msgr2=0x7fd1e015cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:24.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 -- 172.21.15.81:0/2551164397 shutdown_connections 2024-06-29T16:01:24.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:24.980+0000 7fd1e7eac640 1 -- 172.21.15.81:0/2551164397 wait complete. 2024-06-29T16:01:24.991 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-06-29T16:01:23.232496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":450832,"kb_used_data":29288,"kb_used_omap":12,"kb_used_meta":421427,"kb_avail":754523888,"statfs":{"total":773094113280,"available":772632461312,"internally_reserved":0,"allocated":29990912,"data_stored":15966707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":431541810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-806093,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-466,"num_read_kb":-293,"num_write":-435,"num_write_kb":-3002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.008377"},"pg_stats":[{"pgid":"1.0","version":"272'275","reported_seq":1356,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-29T16:01:20.194816+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T16:01:20.194816+0000","last_peered":"2024-06-29T16:01:20.194816+0000","last_clean":"2024-06-29T16:01:20.194816+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T16:01:20.194816+0000","last_undegraded":"2024-06-29T16:01:20.194816+0000","last_fullsized":"2024-06-29T16:01:20.194816+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576030,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53568,"kb_used_data":4340,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318272,"statfs":{"total":96636764160,"available":96581910528,"internally_reserved":0,"allocated":4444160,"data_stored":2695421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834576031,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48460,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323380,"statfs":{"total":96636764160,"available":96587141120,"internally_reserved":0,"allocated":3469312,"data_stored":1718719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834576029,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834576030,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84804,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287036,"statfs":{"total":96636764160,"available":96549924864,"internally_reserved":0,"allocated":3792896,"data_stored":2040598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834576030,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52604,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319236,"statfs":{"total":96636764160,"available":96582897664,"internally_reserved":0,"allocated":3457024,"data_stored":1712296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576030,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52616,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319224,"statfs":{"total":96636764160,"available":96582885376,"internally_reserved":0,"allocated":3469312,"data_stored":1718719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834576029,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48472,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834576030,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61832,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310008,"statfs":{"total":96636764160,"available":96573448192,"internally_reserved":0,"allocated":4386816,"data_stored":2630406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T16:01:24.991 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T16:01:25.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.103+0000 7f3273b6f640 1 -- 172.21.15.81:0/896901425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f326c0b35b0 msgr2=0x7f326c15b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.103+0000 7f3273b6f640 1 --2- 172.21.15.81:0/896901425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f326c0b35b0 0x7f326c15b5b0 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7f326c0cd500 tx=0x7f326802a4a0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.103+0000 7f3273b6f640 1 -- 172.21.15.81:0/896901425 shutdown_connections 2024-06-29T16:01:25.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.103+0000 7f3273b6f640 1 --2- 172.21.15.81:0/896901425 >> v2:172.21.15.167:3300/0 conn(0x7f326c06f660 0x7f326c06fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.103+0000 7f3273b6f640 1 --2- 172.21.15.81:0/896901425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f326c0b35b0 0x7f326c15b5b0 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.103+0000 7f3273b6f640 1 -- 172.21.15.81:0/896901425 >> 172.21.15.81:0/896901425 conn(0x7f326c06ccc0 msgr2=0x7f326c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:25.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.104+0000 7f3273b6f640 1 -- 172.21.15.81:0/896901425 shutdown_connections 2024-06-29T16:01:25.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.104+0000 7f3273b6f640 1 -- 172.21.15.81:0/896901425 wait complete. 2024-06-29T16:01:25.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.105+0000 7f3273b6f640 1 Processor -- start 2024-06-29T16:01:25.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.105+0000 7f3273b6f640 1 -- start start 2024-06-29T16:01:25.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f3273b6f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f326c06f660 0x7f326c158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f3273b6f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f326c0b35b0 0x7f326c1452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f3273b6f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f326c0b4430 0x7f326c145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f3273b6f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f326c06f160 con 0x7f326c0b35b0 2024-06-29T16:01:25.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f3273b6f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f326c06efe0 con 0x7f326c06f660 2024-06-29T16:01:25.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f3273b6f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f326c06f2e0 con 0x7f326c0b4430 2024-06-29T16:01:25.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327110a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f326c0b35b0 0x7f326c1452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327110a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f326c0b35b0 0x7f326c1452f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42790/0 (socket says 172.21.15.81:42790) 2024-06-29T16:01:25.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327110a640 1 -- 172.21.15.81:0/4134061020 learned_addr learned my addr 172.21.15.81:0/4134061020 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:25.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327190b640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f326c06f660 0x7f326c158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327210c640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f326c0b4430 0x7f326c145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327110a640 1 -- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f326c0b4430 msgr2=0x7f326c145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327110a640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f326c0b4430 0x7f326c145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327110a640 1 -- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f326c06f660 msgr2=0x7f326c158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327110a640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f326c06f660 0x7f326c158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327110a640 1 -- 172.21.15.81:0/4134061020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326c146000 con 0x7f326c0b35b0 2024-06-29T16:01:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f327110a640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f326c0b35b0 0x7f326c1452f0 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f325c00c390 tx=0x7f325c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f32627fc640 1 -- 172.21.15.81:0/4134061020 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c017070 con 0x7f326c0b35b0 2024-06-29T16:01:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f32627fc640 1 -- 172.21.15.81:0/4134061020 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f325c0035f0 con 0x7f326c0b35b0 2024-06-29T16:01:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f3273b6f640 1 -- 172.21.15.81:0/4134061020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326c1462e0 con 0x7f326c0b35b0 2024-06-29T16:01:25.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f32627fc640 1 -- 172.21.15.81:0/4134061020 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c003a20 con 0x7f326c0b35b0 2024-06-29T16:01:25.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.107+0000 7f3273b6f640 1 -- 172.21.15.81:0/4134061020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f326c156bc0 con 0x7f326c0b35b0 2024-06-29T16:01:25.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.108+0000 7f3273b6f640 1 -- 172.21.15.81:0/4134061020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f326c06fa70 con 0x7f326c0b35b0 2024-06-29T16:01:25.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.109+0000 7f32627fc640 1 -- 172.21.15.81:0/4134061020 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f325c01a070 con 0x7f326c0b35b0 2024-06-29T16:01:25.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.109+0000 7f32627fc640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3250076420 0x7f3250078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.110+0000 7f32627fc640 1 -- 172.21.15.81:0/4134061020 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f325c096d10 con 0x7f326c0b35b0 2024-06-29T16:01:25.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.110+0000 7f327190b640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3250076420 0x7f3250078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.110+0000 7f327190b640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3250076420 0x7f3250078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3268034000 tx=0x7f3268004730 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:25.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.114+0000 7f32627fc640 1 -- 172.21.15.81:0/4134061020 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f326c06ffb0 con 0x7f326c0b35b0 2024-06-29T16:01:25.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.294+0000 7f3273b6f640 1 -- 172.21.15.81:0/4134061020 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f326c145d90 con 0x7f3250076420 2024-06-29T16:01:25.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.295+0000 7f32627fc640 1 -- 172.21.15.81:0/4134061020 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13480 (secure 0 0 0) 0x7f326c145d90 con 0x7f3250076420 2024-06-29T16:01:25.295 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:25.295 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T16:01:25.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 -- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3250076420 msgr2=0x7f3250078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3250076420 0x7f3250078930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3268034000 tx=0x7f3268004730 comp rx=0 tx=0).stop 2024-06-29T16:01:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 -- 172.21.15.81:0/4134061020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f326c0b35b0 msgr2=0x7f326c1452f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f326c0b35b0 0x7f326c1452f0 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f325c00c390 tx=0x7f325c00c850 comp rx=0 tx=0).stop 2024-06-29T16:01:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 -- 172.21.15.81:0/4134061020 shutdown_connections 2024-06-29T16:01:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f326c0b4430 0x7f326c145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f326c0b35b0 0x7f326c1452f0 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3250076420 0x7f3250078930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 --2- 172.21.15.81:0/4134061020 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f326c06f660 0x7f326c158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 -- 172.21.15.81:0/4134061020 >> 172.21.15.81:0/4134061020 conn(0x7f326c06ccc0 msgr2=0x7f326c15cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:25.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 -- 172.21.15.81:0/4134061020 shutdown_connections 2024-06-29T16:01:25.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.297+0000 7f3273b6f640 1 -- 172.21.15.81:0/4134061020 wait complete. 2024-06-29T16:01:25.308 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-06-29T16:01:25.232970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":450832,"kb_used_data":29288,"kb_used_omap":12,"kb_used_meta":421427,"kb_avail":754523888,"statfs":{"total":773094113280,"available":772632461312,"internally_reserved":0,"allocated":29990912,"data_stored":15966707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":431541810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-806093,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-466,"num_read_kb":-293,"num_write":-435,"num_write_kb":-3002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001358"},"pg_stats":[{"pgid":"1.0","version":"272'275","reported_seq":1356,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-29T16:01:20.194816+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T16:01:20.194816+0000","last_peered":"2024-06-29T16:01:20.194816+0000","last_clean":"2024-06-29T16:01:20.194816+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T16:01:20.194816+0000","last_undegraded":"2024-06-29T16:01:20.194816+0000","last_fullsized":"2024-06-29T16:01:20.194816+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576030,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53568,"kb_used_data":4340,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318272,"statfs":{"total":96636764160,"available":96581910528,"internally_reserved":0,"allocated":4444160,"data_stored":2695421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834576031,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48460,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323380,"statfs":{"total":96636764160,"available":96587141120,"internally_reserved":0,"allocated":3469312,"data_stored":1718719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834576029,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834576030,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84804,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287036,"statfs":{"total":96636764160,"available":96549924864,"internally_reserved":0,"allocated":3792896,"data_stored":2040598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834576030,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52604,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319236,"statfs":{"total":96636764160,"available":96582897664,"internally_reserved":0,"allocated":3457024,"data_stored":1712296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576030,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52616,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319224,"statfs":{"total":96636764160,"available":96582885376,"internally_reserved":0,"allocated":3469312,"data_stored":1718719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834576029,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48472,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834576030,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61832,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310008,"statfs":{"total":96636764160,"available":96573448192,"internally_reserved":0,"allocated":4386816,"data_stored":2630406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T16:01:25.309 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-29T16:01:25.309 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T16:01:25.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.421+0000 7fe870cfe640 1 -- 172.21.15.81:0/2621199686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe86c073650 msgr2=0x7fe86c15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.421+0000 7fe870cfe640 1 --2- 172.21.15.81:0/2621199686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe86c073650 0x7fe86c15b6e0 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7fe854009830 tx=0x7fe85402a480 comp rx=0 tx=0).stop 2024-06-29T16:01:25.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.421+0000 7fe870cfe640 1 -- 172.21.15.81:0/2621199686 shutdown_connections 2024-06-29T16:01:25.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.421+0000 7fe870cfe640 1 --2- 172.21.15.81:0/2621199686 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe86c15bc20 0x7fe86c06ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.421+0000 7fe870cfe640 1 --2- 172.21.15.81:0/2621199686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe86c073650 0x7fe86c15b6e0 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.421+0000 7fe870cfe640 1 -- 172.21.15.81:0/2621199686 >> 172.21.15.81:0/2621199686 conn(0x7fe86c0b2920 msgr2=0x7fe86c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:25.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.421+0000 7fe870cfe640 1 -- 172.21.15.81:0/2621199686 shutdown_connections 2024-06-29T16:01:25.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.421+0000 7fe870cfe640 1 -- 172.21.15.81:0/2621199686 wait complete. 2024-06-29T16:01:25.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.423+0000 7fe870cfe640 1 Processor -- start 2024-06-29T16:01:25.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.423+0000 7fe870cfe640 1 -- start start 2024-06-29T16:01:25.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe870cfe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe86c0cd150 0x7fe86c149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe870cfe640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe86c15bc20 0x7fe86c0a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe870cfe640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe86c1571f0 0x7fe86c157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe870cfe640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c06f660 con 0x7fe86c0cd150 2024-06-29T16:01:25.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe870cfe640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c06f4e0 con 0x7fe86c1571f0 2024-06-29T16:01:25.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe870cfe640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe86c06f7e0 con 0x7fe86c15bc20 2024-06-29T16:01:25.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe86c0cd150 0x7fe86c149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe86c0cd150 0x7fe86c149450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42814/0 (socket says 172.21.15.81:42814) 2024-06-29T16:01:25.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86a59c640 1 -- 172.21.15.81:0/3745035386 learned_addr learned my addr 172.21.15.81:0/3745035386 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:25.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe869d9b640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe86c15bc20 0x7fe86c0a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86a59c640 1 -- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe86c15bc20 msgr2=0x7fe86c0a9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86ad9d640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe86c1571f0 0x7fe86c157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86a59c640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe86c15bc20 0x7fe86c0a9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86a59c640 1 -- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe86c1571f0 msgr2=0x7fe86c157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86a59c640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe86c1571f0 0x7fe86c157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86a59c640 1 -- 172.21.15.81:0/3745035386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c157b40 con 0x7fe86c0cd150 2024-06-29T16:01:25.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86ad9d640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe86c1571f0 0x7fe86c157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:25.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe86a59c640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe86c0cd150 0x7fe86c149450 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00c7d0 tx=0x7fe85c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:25.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.425+0000 7fe862ffd640 1 -- 172.21.15.81:0/3745035386 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c017070 con 0x7fe86c0cd150 2024-06-29T16:01:25.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.426+0000 7fe862ffd640 1 -- 172.21.15.81:0/3745035386 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe85c005070 con 0x7fe86c0cd150 2024-06-29T16:01:25.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.426+0000 7fe870cfe640 1 -- 172.21.15.81:0/3745035386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86c1f3d40 con 0x7fe86c0cd150 2024-06-29T16:01:25.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.426+0000 7fe862ffd640 1 -- 172.21.15.81:0/3745035386 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c0054a0 con 0x7fe86c0cd150 2024-06-29T16:01:25.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.426+0000 7fe870cfe640 1 -- 172.21.15.81:0/3745035386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86c1f4300 con 0x7fe86c0cd150 2024-06-29T16:01:25.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.427+0000 7fe870cfe640 1 -- 172.21.15.81:0/3745035386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86c06fd00 con 0x7fe86c0cd150 2024-06-29T16:01:25.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.427+0000 7fe862ffd640 1 -- 172.21.15.81:0/3745035386 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fe85c01a070 con 0x7fe86c0cd150 2024-06-29T16:01:25.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.428+0000 7fe862ffd640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe84c0763f0 0x7fe84c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.428+0000 7fe862ffd640 1 -- 172.21.15.81:0/3745035386 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7fe85c097100 con 0x7fe86c0cd150 2024-06-29T16:01:25.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.428+0000 7fe869d9b640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe84c0763f0 0x7fe84c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.429+0000 7fe869d9b640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe84c0763f0 0x7fe84c078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe854034000 tx=0x7fe854004730 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:25.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.432+0000 7fe862ffd640 1 -- 172.21.15.81:0/3745035386 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe86c157b40 con 0x7fe86c0cd150 2024-06-29T16:01:25.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.608+0000 7fe870cfe640 1 -- 172.21.15.81:0/3745035386 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe86c0aa1f0 con 0x7fe84c0763f0 2024-06-29T16:01:25.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.609+0000 7fe862ffd640 1 -- 172.21.15.81:0/3745035386 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13480 (secure 0 0 0) 0x7fe86c0aa1f0 con 0x7fe84c0763f0 2024-06-29T16:01:25.610 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:25.610 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T16:01:25.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 -- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe84c0763f0 msgr2=0x7fe84c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe84c0763f0 0x7fe84c078900 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe854034000 tx=0x7fe854004730 comp rx=0 tx=0).stop 2024-06-29T16:01:25.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 -- 172.21.15.81:0/3745035386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe86c0cd150 msgr2=0x7fe86c149450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe86c0cd150 0x7fe86c149450 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00c7d0 tx=0x7fe85c00cc90 comp rx=0 tx=0).stop 2024-06-29T16:01:25.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 -- 172.21.15.81:0/3745035386 shutdown_connections 2024-06-29T16:01:25.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe84c0763f0 0x7fe84c078900 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe86c1571f0 0x7fe86c157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe86c15bc20 0x7fe86c0a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 --2- 172.21.15.81:0/3745035386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe86c0cd150 0x7fe86c149450 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 -- 172.21.15.81:0/3745035386 >> 172.21.15.81:0/3745035386 conn(0x7fe86c0b2920 msgr2=0x7fe86c152260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:25.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 -- 172.21.15.81:0/3745035386 shutdown_connections 2024-06-29T16:01:25.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.611+0000 7fe870cfe640 1 -- 172.21.15.81:0/3745035386 wait complete. 2024-06-29T16:01:25.623 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-06-29T16:01:25.232970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":450832,"kb_used_data":29288,"kb_used_omap":12,"kb_used_meta":421427,"kb_avail":754523888,"statfs":{"total":773094113280,"available":772632461312,"internally_reserved":0,"allocated":29990912,"data_stored":15966707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":431541810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-806093,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-466,"num_read_kb":-293,"num_write":-435,"num_write_kb":-3002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001358"},"pg_stats":[{"pgid":"1.0","version":"272'275","reported_seq":1356,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-06-29T16:01:20.194816+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T16:01:20.194816+0000","last_peered":"2024-06-29T16:01:20.194816+0000","last_clean":"2024-06-29T16:01:20.194816+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T16:01:20.194816+0000","last_undegraded":"2024-06-29T16:01:20.194816+0000","last_fullsized":"2024-06-29T16:01:20.194816+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576030,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53568,"kb_used_data":4340,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318272,"statfs":{"total":96636764160,"available":96581910528,"internally_reserved":0,"allocated":4444160,"data_stored":2695421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834576031,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48460,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323380,"statfs":{"total":96636764160,"available":96587141120,"internally_reserved":0,"allocated":3469312,"data_stored":1718719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834576029,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834576030,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84804,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287036,"statfs":{"total":96636764160,"available":96549924864,"internally_reserved":0,"allocated":3792896,"data_stored":2040598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834576030,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52604,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319236,"statfs":{"total":96636764160,"available":96582897664,"internally_reserved":0,"allocated":3457024,"data_stored":1712296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576030,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52616,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319224,"statfs":{"total":96636764160,"available":96582885376,"internally_reserved":0,"allocated":3469312,"data_stored":1718719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834576029,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48472,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834576030,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61832,"kb_used_data":4284,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310008,"statfs":{"total":96636764160,"available":96573448192,"internally_reserved":0,"allocated":4386816,"data_stored":2630406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T16:01:25.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-29T16:01:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.735+0000 7f03ff762640 1 -- 172.21.15.81:0/1449423232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03f80b5250 msgr2=0x7f03f80d09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.735+0000 7f03ff762640 1 --2- 172.21.15.81:0/1449423232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03f80b5250 0x7f03f80d09c0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f03f808e230 tx=0x7f03e802a5f0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.735+0000 7f03ff762640 1 -- 172.21.15.81:0/1449423232 shutdown_connections 2024-06-29T16:01:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.735+0000 7f03ff762640 1 --2- 172.21.15.81:0/1449423232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03f80b5250 0x7f03f80d09c0 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.735+0000 7f03ff762640 1 --2- 172.21.15.81:0/1449423232 >> v2:172.21.15.167:3300/0 conn(0x7f03f80b9040 0x7f03f80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.735+0000 7f03ff762640 1 -- 172.21.15.81:0/1449423232 >> 172.21.15.81:0/1449423232 conn(0x7f03f809cf30 msgr2=0x7f03f80d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:25.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.735+0000 7f03ff762640 1 -- 172.21.15.81:0/1449423232 shutdown_connections 2024-06-29T16:01:25.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.735+0000 7f03ff762640 1 -- 172.21.15.81:0/1449423232 wait complete. 2024-06-29T16:01:25.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.737+0000 7f03ff762640 1 Processor -- start 2024-06-29T16:01:25.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.737+0000 7f03ff762640 1 -- start start 2024-06-29T16:01:25.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.738+0000 7f03ff762640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03f80b5250 0x7f03f80b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.738+0000 7f03ff762640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03f80b7290 0x7f03f80b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.738+0000 7f03ff762640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03f80b9040 0x7f03f80c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.738+0000 7f03ff762640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f03f80d2e20 con 0x7f03f80b7290 2024-06-29T16:01:25.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.738+0000 7f03ff762640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f03f80d2ca0 con 0x7f03f80b5250 2024-06-29T16:01:25.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.738+0000 7f03ff762640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f03f80b7db0 con 0x7f03f80b9040 2024-06-29T16:01:25.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fccfd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03f80b7290 0x7f03f80b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fccfd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03f80b7290 0x7f03f80b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42830/0 (socket says 172.21.15.81:42830) 2024-06-29T16:01:25.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fccfd640 1 -- 172.21.15.81:0/3556819560 learned_addr learned my addr 172.21.15.81:0/3556819560 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:25.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fd4fe640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03f80b5250 0x7f03f80b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fdcff640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03f80b9040 0x7f03f80c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fccfd640 1 -- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03f80b9040 msgr2=0x7f03f80c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fccfd640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03f80b9040 0x7f03f80c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fccfd640 1 -- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03f80b5250 msgr2=0x7f03f80b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fccfd640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03f80b5250 0x7f03f80b0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fccfd640 1 -- 172.21.15.81:0/3556819560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f80b0d60 con 0x7f03f80b7290 2024-06-29T16:01:25.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fd4fe640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03f80b5250 0x7f03f80b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:25.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fdcff640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03f80b9040 0x7f03f80c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:25.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03fccfd640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03f80b7290 0x7f03f80b4190 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f03e8034040 tx=0x7f03e8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:25.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03edffb640 1 -- 172.21.15.81:0/3556819560 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e803e030 con 0x7f03f80b7290 2024-06-29T16:01:25.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03edffb640 1 -- 172.21.15.81:0/3556819560 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03e8003e00 con 0x7f03f80b7290 2024-06-29T16:01:25.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03edffb640 1 -- 172.21.15.81:0/3556819560 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e8035040 con 0x7f03f80b7290 2024-06-29T16:01:25.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.739+0000 7f03ff762640 1 -- 172.21.15.81:0/3556819560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f80b1270 con 0x7f03f80b7290 2024-06-29T16:01:25.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.740+0000 7f03ff762640 1 -- 172.21.15.81:0/3556819560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f80b1830 con 0x7f03f80b7290 2024-06-29T16:01:25.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.741+0000 7f03ff762640 1 -- 172.21.15.81:0/3556819560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f80b82f0 con 0x7f03f80b7290 2024-06-29T16:01:25.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.741+0000 7f03edffb640 1 -- 172.21.15.81:0/3556819560 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f03e8041030 con 0x7f03f80b7290 2024-06-29T16:01:25.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.742+0000 7f03edffb640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f03dc0763f0 0x7f03dc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:25.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.742+0000 7f03edffb640 1 -- 172.21.15.81:0/3556819560 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f03e80b8330 con 0x7f03f80b7290 2024-06-29T16:01:25.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.742+0000 7f03fd4fe640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f03dc0763f0 0x7f03dc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:25.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.743+0000 7f03fd4fe640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f03dc0763f0 0x7f03dc078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009aa0 tx=0x7f03f4005db0 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:25.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.746+0000 7f03edffb640 1 -- 172.21.15.81:0/3556819560 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f03e80819a0 con 0x7f03f80b7290 2024-06-29T16:01:25.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.926+0000 7f03ff762640 1 -- 172.21.15.81:0/3556819560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03f80b0f90 con 0x7f03f80b7290 2024-06-29T16:01:25.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.926+0000 7f03edffb640 1 -- 172.21.15.81:0/3556819560 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v275) ==== 74+0+14469 (secure 0 0 0) 0x7f03e80812a0 con 0x7f03f80b7290 2024-06-29T16:01:25.927 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:25.927 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":275,"fsid":"e6805bd5-4748-48b4-ab28-fbf8a8d96ae4","created":"2024-06-29T14:22:18.240431+0000","modified":"2024-06-29T16:01:22.589170+0000","last_up_change":"2024-06-29T14:22:30.294847+0000","last_in_change":"2024-06-29T14:22:21.745630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-29T14:22:30.719746+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"41a27439-5583-43cf-8abe-2131cda2eb06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6801","nonce":1533488904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6803","nonce":1533488904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6807","nonce":1533488904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":1533488904},{"type":"v1","addr":"172.21.15.81:6805","nonce":1533488904}]},"public_addr":"172.21.15.81:6801/1533488904","cluster_addr":"172.21.15.81:6803/1533488904","heartbeat_back_addr":"172.21.15.81:6807/1533488904","heartbeat_front_addr":"172.21.15.81:6805/1533488904","state":["exists","up"]},{"osd":1,"uuid":"a4372bb3-c9cd-484e-9651-43ad2cce450a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6810","nonce":2523783345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6814","nonce":2523783345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6822","nonce":2523783345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2523783345},{"type":"v1","addr":"172.21.15.81:6818","nonce":2523783345}]},"public_addr":"172.21.15.81:6810/2523783345","cluster_addr":"172.21.15.81:6814/2523783345","heartbeat_back_addr":"172.21.15.81:6822/2523783345","heartbeat_front_addr":"172.21.15.81:6818/2523783345","state":["exists","up"]},{"osd":2,"uuid":"4e572bd2-c163-436a-b90b-9c2e48641954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6825","nonce":4255314520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6827","nonce":4255314520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6831","nonce":4255314520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":4255314520},{"type":"v1","addr":"172.21.15.81:6829","nonce":4255314520}]},"public_addr":"172.21.15.81:6825/4255314520","cluster_addr":"172.21.15.81:6827/4255314520","heartbeat_back_addr":"172.21.15.81:6831/4255314520","heartbeat_front_addr":"172.21.15.81:6829/4255314520","state":["exists","up"]},{"osd":3,"uuid":"cc3ff76f-fd5a-4d48-bce5-930fc8c6595c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":268,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6809","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6811","nonce":3633523901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6815","nonce":3633523901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6821","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6823","nonce":3633523901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":3633523901},{"type":"v1","addr":"172.21.15.81:6819","nonce":3633523901}]},"public_addr":"172.21.15.81:6811/3633523901","cluster_addr":"172.21.15.81:6815/3633523901","heartbeat_back_addr":"172.21.15.81:6823/3633523901","heartbeat_front_addr":"172.21.15.81:6819/3633523901","state":["exists","up"]},{"osd":4,"uuid":"8e48be9a-0cf8-4cc8-adf8-0fdd3f04d364","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6825","nonce":220767630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6826","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6827","nonce":220767630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6830","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6831","nonce":220767630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6828","nonce":220767630},{"type":"v1","addr":"172.21.15.167:6829","nonce":220767630}]},"public_addr":"172.21.15.167:6825/220767630","cluster_addr":"172.21.15.167:6827/220767630","heartbeat_back_addr":"172.21.15.167:6831/220767630","heartbeat_front_addr":"172.21.15.167:6829/220767630","state":["exists","up"]},{"osd":5,"uuid":"788dfe4a-cb5c-478b-9a7c-0861025a6f45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6801","nonce":4186590320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6803","nonce":4186590320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6807","nonce":4186590320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":4186590320},{"type":"v1","addr":"172.21.15.167:6805","nonce":4186590320}]},"public_addr":"172.21.15.167:6801/4186590320","cluster_addr":"172.21.15.167:6803/4186590320","heartbeat_back_addr":"172.21.15.167:6807/4186590320","heartbeat_front_addr":"172.21.15.167:6805/4186590320","state":["exists","up"]},{"osd":6,"uuid":"2c771d70-f7cf-49be-b6c6-066a411a0374","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6817","nonce":2006194601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6819","nonce":2006194601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6823","nonce":2006194601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":2006194601},{"type":"v1","addr":"172.21.15.167:6821","nonce":2006194601}]},"public_addr":"172.21.15.167:6817/2006194601","cluster_addr":"172.21.15.167:6819/2006194601","heartbeat_back_addr":"172.21.15.167:6823/2006194601","heartbeat_front_addr":"172.21.15.167:6821/2006194601","state":["exists","up"]},{"osd":7,"uuid":"e293ba03-0801-463c-bbe7-ed71eb33b0f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":264,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6809","nonce":607598008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6811","nonce":607598008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6815","nonce":607598008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":607598008},{"type":"v1","addr":"172.21.15.167:6813","nonce":607598008}]},"public_addr":"172.21.15.167:6809/607598008","cluster_addr":"172.21.15.167:6811/607598008","heartbeat_back_addr":"172.21.15.167:6815/607598008","heartbeat_front_addr":"172.21.15.167:6813/607598008","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.899881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.876845+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.901411+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.853515+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.963632+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.849372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.843143+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-29T14:22:26.838206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.167:6833/2899520759":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1667571922":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/853755847":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1133468043":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/707331717":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1199836798":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/4142686836":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/1553695616":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/4002034537":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/3409979994":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/3006664184":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6833/1824030697":"2024-06-30T15:59:54.317815+0000","172.21.15.81:6834/28043779":"2024-06-30T16:01:19.466814+0000","172.21.15.167:6832/1824030697":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/4135998660":"2024-06-30T15:59:54.317815+0000","172.21.15.167:6832/2899520759":"2024-06-30T16:01:17.202864+0000","172.21.15.167:0/432950336":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/1067289398":"2024-06-30T15:59:54.317815+0000","172.21.15.81:6832/28043779":"2024-06-30T16:01:19.466814+0000","172.21.15.167:0/1087585815":"2024-06-30T15:59:54.317815+0000","172.21.15.167:0/2453006101":"2024-06-30T15:59:54.317815+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":111,"length":1},{"begin":221,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":25,"snaps":[{"begin":2,"length":4}]},{"pool":37,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-29T16:01:25.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 -- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f03dc0763f0 msgr2=0x7f03dc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f03dc0763f0 0x7f03dc078900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009aa0 tx=0x7f03f4005db0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 -- 172.21.15.81:0/3556819560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03f80b7290 msgr2=0x7f03f80b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:25.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03f80b7290 0x7f03f80b4190 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f03e8034040 tx=0x7f03e8003910 comp rx=0 tx=0).stop 2024-06-29T16:01:25.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 -- 172.21.15.81:0/3556819560 shutdown_connections 2024-06-29T16:01:25.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f03f80b9040 0x7f03f80c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03f80b7290 0x7f03f80b4190 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f03dc0763f0 0x7f03dc078900 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 --2- 172.21.15.81:0/3556819560 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f03f80b5250 0x7f03f80b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:25.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 -- 172.21.15.81:0/3556819560 >> 172.21.15.81:0/3556819560 conn(0x7f03f809cf30 msgr2=0x7f03f80d17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:25.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 -- 172.21.15.81:0/3556819560 shutdown_connections 2024-06-29T16:01:25.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:25.929+0000 7f03ff762640 1 -- 172.21.15.81:0/3556819560 wait complete. 2024-06-29T16:01:26.941 INFO:tasks.ceph:Scrubbing osd.0 2024-06-29T16:01:26.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.063+0000 7f511c2b9640 1 -- 172.21.15.81:0/1561694586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51140b5a60 msgr2=0x7f51140affa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.063+0000 7f511c2b9640 1 --2- 172.21.15.81:0/1561694586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51140b5a60 0x7f51140affa0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5104009830 tx=0x7f510402a510 comp rx=0 tx=0).stop 2024-06-29T16:01:27.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.064+0000 7f511c2b9640 1 -- 172.21.15.81:0/1561694586 shutdown_connections 2024-06-29T16:01:27.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.064+0000 7f511c2b9640 1 --2- 172.21.15.81:0/1561694586 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5114073600 0x7f511406f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.064+0000 7f511c2b9640 1 --2- 172.21.15.81:0/1561694586 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f51140b5a60 0x7f51140affa0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.064+0000 7f511c2b9640 1 -- 172.21.15.81:0/1561694586 >> 172.21.15.81:0/1561694586 conn(0x7f51140b2580 msgr2=0x7f51140b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:27.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.064+0000 7f511c2b9640 1 -- 172.21.15.81:0/1561694586 shutdown_connections 2024-06-29T16:01:27.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.064+0000 7f511c2b9640 1 -- 172.21.15.81:0/1561694586 wait complete. 2024-06-29T16:01:27.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.065+0000 7f511c2b9640 1 Processor -- start 2024-06-29T16:01:27.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.065+0000 7f511c2b9640 1 -- start start 2024-06-29T16:01:27.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511c2b9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51140b5a60 0x7f5114154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511c2b9640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f511415ace0 0x7f511414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511c2b9640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51140cc070 0x7f51140cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511c2b9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f511406fa40 con 0x7f51140b5a60 2024-06-29T16:01:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511c2b9640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f511406f8c0 con 0x7f511415ace0 2024-06-29T16:01:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511c2b9640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f511406fbc0 con 0x7f51140cc070 2024-06-29T16:01:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511a055640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51140b5a60 0x7f5114154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511a055640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51140b5a60 0x7f5114154990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42842/0 (socket says 172.21.15.81:42842) 2024-06-29T16:01:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511a055640 1 -- 172.21.15.81:0/2247862223 learned_addr learned my addr 172.21.15.81:0/2247862223 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511a856640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51140cc070 0x7f51140cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f5119854640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f511415ace0 0x7f511414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511a055640 1 -- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51140cc070 msgr2=0x7f51140cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511a055640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f51140cc070 0x7f51140cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511a055640 1 -- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f511415ace0 msgr2=0x7f511414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511a055640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f511415ace0 0x7f511414c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511a055640 1 -- 172.21.15.81:0/2247862223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f511414c800 con 0x7f51140b5a60 2024-06-29T16:01:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f5119854640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f511415ace0 0x7f511414c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:27.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.067+0000 7f511a055640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51140b5a60 0x7f5114154990 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f510402ab00 tx=0x7f51040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.068+0000 7f510affd640 1 -- 172.21.15.81:0/2247862223 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51040044e0 con 0x7f51140b5a60 2024-06-29T16:01:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.068+0000 7f510affd640 1 -- 172.21.15.81:0/2247862223 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5104005310 con 0x7f51140b5a60 2024-06-29T16:01:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.068+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51140c7850 con 0x7f51140b5a60 2024-06-29T16:01:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.068+0000 7f510affd640 1 -- 172.21.15.81:0/2247862223 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104003800 con 0x7f51140b5a60 2024-06-29T16:01:27.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.068+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51140c7e10 con 0x7f51140b5a60 2024-06-29T16:01:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.068+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f511406f0c0 con 0x7f51140b5a60 2024-06-29T16:01:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.070+0000 7f510affd640 1 -- 172.21.15.81:0/2247862223 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f51040039a0 con 0x7f51140b5a60 2024-06-29T16:01:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.071+0000 7f510affd640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f50f80763f0 0x7f50f8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.071+0000 7f510affd640 1 -- 172.21.15.81:0/2247862223 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f51040bb770 con 0x7f51140b5a60 2024-06-29T16:01:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.071+0000 7f510affd640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] conn(0x7f50f807cad0 0x7f50f807efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.071+0000 7f510affd640 1 -- 172.21.15.81:0/2247862223 --> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f510402dec0 con 0x7f50f807cad0 2024-06-29T16:01:27.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.071+0000 7f510affd640 1 -- 172.21.15.81:0/2247862223 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f51040bbc00 con 0x7f51140b5a60 2024-06-29T16:01:27.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.071+0000 7f5119854640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f50f80763f0 0x7f50f8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.071+0000 7f511a856640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] conn(0x7f50f807cad0 0x7f50f807efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.071+0000 7f511a856640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] conn(0x7f50f807cad0 0x7f50f807efa0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.071+0000 7f5119854640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f50f80763f0 0x7f50f8078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5110007fd0 tx=0x7f5110009040 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.072+0000 7f510affd640 1 -- 172.21.15.81:0/2247862223 <== osd.0 v2:172.21.15.81:6800/1533488904 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f510402dec0 con 0x7f50f807cad0 2024-06-29T16:01:27.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.085+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 --> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f511406a570 con 0x7f50f807cad0 2024-06-29T16:01:27.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f510affd640 1 -- 172.21.15.81:0/2247862223 <== osd.0 v2:172.21.15.81:6800/1533488904 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f511406a570 con 0x7f50f807cad0 2024-06-29T16:01:27.086 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T16:01:27.087 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:27.087 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T16:01:27.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 >> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] conn(0x7f50f807cad0 msgr2=0x7f50f807efa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511c2b9640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.81:6800/1533488904,v1:172.21.15.81:6801/1533488904] conn(0x7f50f807cad0 0x7f50f807efa0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f50f80763f0 msgr2=0x7f50f8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511c2b9640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f50f80763f0 0x7f50f8078900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5110007fd0 tx=0x7f5110009040 comp rx=0 tx=0).stop 2024-06-29T16:01:27.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51140b5a60 msgr2=0x7f5114154990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511c2b9640 1 --2- 172.21.15.81:0/2247862223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51140b5a60 0x7f5114154990 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f510402ab00 tx=0x7f51040042f0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511a856640 1 -- 172.21.15.81:0/2247862223 reap_dead start 2024-06-29T16:01:27.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 shutdown_connections 2024-06-29T16:01:27.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 >> 172.21.15.81:0/2247862223 conn(0x7f51140b2580 msgr2=0x7f5114159b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:27.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 shutdown_connections 2024-06-29T16:01:27.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.086+0000 7f511c2b9640 1 -- 172.21.15.81:0/2247862223 wait complete. 2024-06-29T16:01:27.097 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-06-29T16:01:27.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.213+0000 7fe9b0895640 1 -- 172.21.15.81:0/1430246080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac0b3200 msgr2=0x7fe9ac0cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.213+0000 7fe9b0895640 1 --2- 172.21.15.81:0/1430246080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac0b3200 0x7fe9ac0cc7b0 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7fe99c005fd0 tx=0x7fe99c02a5d0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.213+0000 7fe9b0895640 1 -- 172.21.15.81:0/1430246080 shutdown_connections 2024-06-29T16:01:27.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.213+0000 7fe9b0895640 1 --2- 172.21.15.81:0/1430246080 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9ac0d0940 0x7fe9ac0b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.213+0000 7fe9b0895640 1 --2- 172.21.15.81:0/1430246080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac0b3200 0x7fe9ac0cc7b0 unknown :-1 s=CLOSED pgs=3029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.213+0000 7fe9b0895640 1 -- 172.21.15.81:0/1430246080 >> 172.21.15.81:0/1430246080 conn(0x7fe9ac0b3680 msgr2=0x7fe9ac0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:27.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.213+0000 7fe9b0895640 1 -- 172.21.15.81:0/1430246080 shutdown_connections 2024-06-29T16:01:27.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.213+0000 7fe9b0895640 1 -- 172.21.15.81:0/1430246080 wait complete. 2024-06-29T16:01:27.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.215+0000 7fe9b0895640 1 Processor -- start 2024-06-29T16:01:27.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.215+0000 7fe9b0895640 1 -- start start 2024-06-29T16:01:27.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.216+0000 7fe9b0895640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9ac0b3200 0x7fe9ac153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.216+0000 7fe9b0895640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac0cccf0 0x7fe9ac154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.216+0000 7fe9b0895640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9ac1553e0 0x7fe9ac1581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.216+0000 7fe9b0895640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac0b5c50 con 0x7fe9ac0cccf0 2024-06-29T16:01:27.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.216+0000 7fe9b0895640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac0b5ad0 con 0x7fe9ac1553e0 2024-06-29T16:01:27.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.216+0000 7fe9b0895640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fe9ac0b5dd0 con 0x7fe9ac0b3200 2024-06-29T16:01:27.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9a3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac0cccf0 0x7fe9ac154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9a3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac0cccf0 0x7fe9ac154ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42860/0 (socket says 172.21.15.81:42860) 2024-06-29T16:01:27.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9a3fff640 1 -- 172.21.15.81:0/587796512 learned_addr learned my addr 172.21.15.81:0/587796512 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:27.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9aa59c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9ac0b3200 0x7fe9ac153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9aad9d640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9ac1553e0 0x7fe9ac1581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9aad9d640 1 -- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9ac0b3200 msgr2=0x7fe9ac153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9aad9d640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9ac0b3200 0x7fe9ac153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9aad9d640 1 -- 172.21.15.81:0/587796512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac0cccf0 msgr2=0x7fe9ac154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9aad9d640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac0cccf0 0x7fe9ac154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9aad9d640 1 -- 172.21.15.81:0/587796512 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ac0c7a10 con 0x7fe9ac1553e0 2024-06-29T16:01:27.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9a3fff640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac0cccf0 0x7fe9ac154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-29T16:01:27.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9aad9d640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9ac1553e0 0x7fe9ac1581e0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4005fd0 tx=0x7fe9a400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9a2ffd640 1 -- 172.21.15.81:0/587796512 <== mon.1 v2:172.21.15.167:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a400cd80 con 0x7fe9ac1553e0 2024-06-29T16:01:27.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9a2ffd640 1 -- 172.21.15.81:0/587796512 <== mon.1 v2:172.21.15.167:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9a4004c10 con 0x7fe9ac1553e0 2024-06-29T16:01:27.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9a2ffd640 1 -- 172.21.15.81:0/587796512 <== mon.1 v2:172.21.15.167:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4005020 con 0x7fe9ac1553e0 2024-06-29T16:01:27.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.217+0000 7fe9b0895640 1 -- 172.21.15.81:0/587796512 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ac0c7cf0 con 0x7fe9ac1553e0 2024-06-29T16:01:27.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.218+0000 7fe9b0895640 1 -- 172.21.15.81:0/587796512 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ac0c82b0 con 0x7fe9ac1553e0 2024-06-29T16:01:27.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.218+0000 7fe9b0895640 1 -- 172.21.15.81:0/587796512 --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9ac0c9080 con 0x7fe9ac1553e0 2024-06-29T16:01:27.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.219+0000 7fe9a2ffd640 1 -- 172.21.15.81:0/587796512 <== mon.1 v2:172.21.15.167:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fe9a40051c0 con 0x7fe9ac1553e0 2024-06-29T16:01:27.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.220+0000 7fe9a2ffd640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe98c076420 0x7fe98c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.220+0000 7fe9a2ffd640 1 -- 172.21.15.81:0/587796512 <== mon.1 v2:172.21.15.167:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7fe9a40987a0 con 0x7fe9ac1553e0 2024-06-29T16:01:27.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.220+0000 7fe9aa59c640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe98c076420 0x7fe98c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.221+0000 7fe9aa59c640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe98c076420 0x7fe98c078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe99c033000 tx=0x7fe99c004730 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.224+0000 7fe9a2ffd640 1 -- 172.21.15.81:0/587796512 <== mon.1 v2:172.21.15.167:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fe9a4061610 con 0x7fe9ac1553e0 2024-06-29T16:01:27.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.411+0000 7fe9b0895640 1 -- 172.21.15.81:0/587796512 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fe9ac071d40 con 0x7fe98c076420 2024-06-29T16:01:27.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.412+0000 7fe9a2ffd640 1 -- 172.21.15.81:0/587796512 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe9ac071d40 con 0x7fe98c076420 2024-06-29T16:01:27.412 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 0 to deep-scrub 2024-06-29T16:01:27.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 -- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe98c076420 msgr2=0x7fe98c078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe98c076420 0x7fe98c078930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe99c033000 tx=0x7fe99c004730 comp rx=0 tx=0).stop 2024-06-29T16:01:27.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 -- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9ac1553e0 msgr2=0x7fe9ac1581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9ac1553e0 0x7fe9ac1581e0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4005fd0 tx=0x7fe9a400ee40 comp rx=0 tx=0).stop 2024-06-29T16:01:27.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 -- 172.21.15.81:0/587796512 shutdown_connections 2024-06-29T16:01:27.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fe98c076420 0x7fe98c078930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fe9ac1553e0 0x7fe9ac1581e0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac0cccf0 0x7fe9ac154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 --2- 172.21.15.81:0/587796512 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fe9ac0b3200 0x7fe9ac153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 -- 172.21.15.81:0/587796512 >> 172.21.15.81:0/587796512 conn(0x7fe9ac0b3680 msgr2=0x7fe9ac0d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:27.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 -- 172.21.15.81:0/587796512 shutdown_connections 2024-06-29T16:01:27.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.414+0000 7fe9b0895640 1 -- 172.21.15.81:0/587796512 wait complete. 2024-06-29T16:01:27.426 INFO:tasks.ceph:Scrubbing osd.1 2024-06-29T16:01:27.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:27.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.547+0000 7fbe63c9e640 1 -- 172.21.15.81:0/873368571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe5c073600 msgr2=0x7fbe5c06f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.547+0000 7fbe63c9e640 1 --2- 172.21.15.81:0/873368571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe5c073600 0x7fbe5c06f0c0 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0d19b0 tx=0x7fbe5402a8b0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.547+0000 7fbe63c9e640 1 -- 172.21.15.81:0/873368571 shutdown_connections 2024-06-29T16:01:27.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.547+0000 7fbe63c9e640 1 --2- 172.21.15.81:0/873368571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe5c073600 0x7fbe5c06f0c0 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.547+0000 7fbe63c9e640 1 --2- 172.21.15.81:0/873368571 >> v2:172.21.15.167:3300/0 conn(0x7fbe5c0b5a60 0x7fbe5c0affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.547+0000 7fbe63c9e640 1 -- 172.21.15.81:0/873368571 >> 172.21.15.81:0/873368571 conn(0x7fbe5c0b2580 msgr2=0x7fbe5c0b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:27.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.547+0000 7fbe63c9e640 1 -- 172.21.15.81:0/873368571 shutdown_connections 2024-06-29T16:01:27.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.547+0000 7fbe63c9e640 1 -- 172.21.15.81:0/873368571 wait complete. 2024-06-29T16:01:27.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.549+0000 7fbe63c9e640 1 Processor -- start 2024-06-29T16:01:27.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.549+0000 7fbe63c9e640 1 -- start start 2024-06-29T16:01:27.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe63c9e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe5c0b5a60 0x7fbe5c1f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe63c9e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe5c15ace0 0x7fbe5c152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe63c9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe5c1501c0 0x7fbe5c14c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe63c9e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c06f990 con 0x7fbe5c1501c0 2024-06-29T16:01:27.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe63c9e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c06f810 con 0x7fbe5c15ace0 2024-06-29T16:01:27.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe63c9e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbe5c06fb10 con 0x7fbe5c0b5a60 2024-06-29T16:01:27.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe6223b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe5c1501c0 0x7fbe5c14c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe6223b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe5c1501c0 0x7fbe5c14c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42874/0 (socket says 172.21.15.81:42874) 2024-06-29T16:01:27.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe6223b640 1 -- 172.21.15.81:0/2470567813 learned_addr learned my addr 172.21.15.81:0/2470567813 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:27.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe61a3a640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe5c0b5a60 0x7fbe5c1f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe61239640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe5c15ace0 0x7fbe5c152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe6223b640 1 -- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe5c0b5a60 msgr2=0x7fbe5c1f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe6223b640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbe5c0b5a60 0x7fbe5c1f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe6223b640 1 -- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe5c15ace0 msgr2=0x7fbe5c152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe6223b640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbe5c15ace0 0x7fbe5c152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.550+0000 7fbe6223b640 1 -- 172.21.15.81:0/2470567813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5c153550 con 0x7fbe5c1501c0 2024-06-29T16:01:27.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.551+0000 7fbe6223b640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe5c1501c0 0x7fbe5c14c590 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009ca0 tx=0x7fbe54033f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.551+0000 7fbe527fc640 1 -- 172.21.15.81:0/2470567813 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54031990 con 0x7fbe5c1501c0 2024-06-29T16:01:27.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.551+0000 7fbe527fc640 1 -- 172.21.15.81:0/2470567813 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe54031c20 con 0x7fbe5c1501c0 2024-06-29T16:01:27.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.551+0000 7fbe527fc640 1 -- 172.21.15.81:0/2470567813 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54033070 con 0x7fbe5c1501c0 2024-06-29T16:01:27.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.551+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe5c153270 con 0x7fbe5c1501c0 2024-06-29T16:01:27.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.551+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe5c1f2aa0 con 0x7fbe5c1501c0 2024-06-29T16:01:27.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.551+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbe5c0b2330 con 0x7fbe5c1501c0 2024-06-29T16:01:27.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.552+0000 7fbe527fc640 1 -- 172.21.15.81:0/2470567813 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fbe54033210 con 0x7fbe5c1501c0 2024-06-29T16:01:27.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.553+0000 7fbe527fc640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbe40076420 0x7fbe40078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.553+0000 7fbe527fc640 1 -- 172.21.15.81:0/2470567813 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7fbe540b95a0 con 0x7fbe5c1501c0 2024-06-29T16:01:27.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.553+0000 7fbe61a3a640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbe40076420 0x7fbe40078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.553+0000 7fbe527fc640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] conn(0x7fbe4007cb60 0x7fbe4007f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.553+0000 7fbe527fc640 1 -- 172.21.15.81:0/2470567813 --> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbe540317e0 con 0x7fbe4007cb60 2024-06-29T16:01:27.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.553+0000 7fbe527fc640 1 -- 172.21.15.81:0/2470567813 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7fbe540b9a30 con 0x7fbe5c1501c0 2024-06-29T16:01:27.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.553+0000 7fbe61239640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] conn(0x7fbe4007cb60 0x7fbe4007f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.553+0000 7fbe61a3a640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbe40076420 0x7fbe40078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c009b30 tx=0x7fbe4c005f90 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.553+0000 7fbe61239640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] conn(0x7fbe4007cb60 0x7fbe4007f030 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.554+0000 7fbe527fc640 1 -- 172.21.15.81:0/2470567813 <== osd.1 v2:172.21.15.81:6808/2523783345 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbe540317e0 con 0x7fbe4007cb60 2024-06-29T16:01:27.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.567+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 --> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbe5c06f0c0 con 0x7fbe4007cb60 2024-06-29T16:01:27.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.567+0000 7fbe527fc640 1 -- 172.21.15.81:0/2470567813 <== osd.1 v2:172.21.15.81:6808/2523783345 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbe5c06f0c0 con 0x7fbe4007cb60 2024-06-29T16:01:27.568 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T16:01:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:27.568 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T16:01:27.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 >> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] conn(0x7fbe4007cb60 msgr2=0x7fbe4007f030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe63c9e640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.81:6808/2523783345,v1:172.21.15.81:6810/2523783345] conn(0x7fbe4007cb60 0x7fbe4007f030 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbe40076420 msgr2=0x7fbe40078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe63c9e640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbe40076420 0x7fbe40078930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c009b30 tx=0x7fbe4c005f90 comp rx=0 tx=0).stop 2024-06-29T16:01:27.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe5c1501c0 msgr2=0x7fbe5c14c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe63c9e640 1 --2- 172.21.15.81:0/2470567813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe5c1501c0 0x7fbe5c14c590 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009ca0 tx=0x7fbe54033f70 comp rx=0 tx=0).stop 2024-06-29T16:01:27.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe6223b640 1 -- 172.21.15.81:0/2470567813 reap_dead start 2024-06-29T16:01:27.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 shutdown_connections 2024-06-29T16:01:27.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 >> 172.21.15.81:0/2470567813 conn(0x7fbe5c0b2580 msgr2=0x7fbe5c0b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:27.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 shutdown_connections 2024-06-29T16:01:27.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.568+0000 7fbe63c9e640 1 -- 172.21.15.81:0/2470567813 wait complete. 2024-06-29T16:01:27.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-06-29T16:01:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.695+0000 7f5644d8c640 1 -- 172.21.15.81:0/2880727421 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56400d0230 msgr2=0x7f56400d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.695+0000 7f5644d8c640 1 --2- 172.21.15.81:0/2880727421 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56400d0230 0x7f56400d0640 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5628009860 tx=0x7f562802a5a0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.695+0000 7f5644d8c640 1 -- 172.21.15.81:0/2880727421 shutdown_connections 2024-06-29T16:01:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.695+0000 7f5644d8c640 1 --2- 172.21.15.81:0/2880727421 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56400d0230 0x7f56400d0640 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.695+0000 7f5644d8c640 1 -- 172.21.15.81:0/2880727421 >> 172.21.15.81:0/2880727421 conn(0x7f56400b3680 msgr2=0x7f56400b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:27.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.695+0000 7f5644d8c640 1 -- 172.21.15.81:0/2880727421 shutdown_connections 2024-06-29T16:01:27.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.695+0000 7f5644d8c640 1 -- 172.21.15.81:0/2880727421 wait complete. 2024-06-29T16:01:27.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.697+0000 7f5644d8c640 1 Processor -- start 2024-06-29T16:01:27.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.697+0000 7f5644d8c640 1 -- start start 2024-06-29T16:01:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.698+0000 7f5644d8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56400b3200 0x7f56401f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.698+0000 7f5644d8c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56400b4f40 0x7f56400c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.698+0000 7f5644d8c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56400d0230 0x7f5640154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.698+0000 7f5644d8c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f56400b5e20 con 0x7f56400b3200 2024-06-29T16:01:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.698+0000 7f5644d8c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f56400b5ca0 con 0x7f56400d0230 2024-06-29T16:01:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.698+0000 7f5644d8c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f56400b5fa0 con 0x7f56400b4f40 2024-06-29T16:01:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f563e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56400b3200 0x7f56401f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f563e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56400b3200 0x7f56401f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42890/0 (socket says 172.21.15.81:42890) 2024-06-29T16:01:27.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f563e59c640 1 -- 172.21.15.81:0/1735194008 learned_addr learned my addr 172.21.15.81:0/1735194008 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:27.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f5637fff640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56400b4f40 0x7f56400c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f563ed9d640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56400d0230 0x7f5640154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f563e59c640 1 -- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56400b4f40 msgr2=0x7f56400c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f563e59c640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56400b4f40 0x7f56400c7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f563e59c640 1 -- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56400d0230 msgr2=0x7f5640154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f563e59c640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56400d0230 0x7f5640154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f563e59c640 1 -- 172.21.15.81:0/1735194008 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56400c8150 con 0x7f56400b3200 2024-06-29T16:01:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f563e59c640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56400b3200 0x7f56401f6bd0 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7f563000c390 tx=0x7f563000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f5636ffd640 1 -- 172.21.15.81:0/1735194008 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5630017070 con 0x7f56400b3200 2024-06-29T16:01:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f5636ffd640 1 -- 172.21.15.81:0/1735194008 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56300053c0 con 0x7f56400b3200 2024-06-29T16:01:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f5644d8c640 1 -- 172.21.15.81:0/1735194008 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5640159930 con 0x7f56400b3200 2024-06-29T16:01:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f5636ffd640 1 -- 172.21.15.81:0/1735194008 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5630003850 con 0x7f56400b3200 2024-06-29T16:01:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.699+0000 7f5644d8c640 1 -- 172.21.15.81:0/1735194008 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5640159ef0 con 0x7f56400b3200 2024-06-29T16:01:27.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.700+0000 7f5644d8c640 1 -- 172.21.15.81:0/1735194008 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56400d0640 con 0x7f56400b3200 2024-06-29T16:01:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.701+0000 7f5636ffd640 1 -- 172.21.15.81:0/1735194008 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f563001a070 con 0x7f56400b3200 2024-06-29T16:01:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.702+0000 7f5636ffd640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56200763f0 0x7f5620078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.702+0000 7f5636ffd640 1 -- 172.21.15.81:0/1735194008 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f5630097260 con 0x7f56400b3200 2024-06-29T16:01:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.702+0000 7f5637fff640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56200763f0 0x7f5620078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.702+0000 7f5637fff640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56200763f0 0x7f5620078900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5628034000 tx=0x7f5628004730 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.706+0000 7f5636ffd640 1 -- 172.21.15.81:0/1735194008 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5630021020 con 0x7f56400b3200 2024-06-29T16:01:27.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.890+0000 7f5644d8c640 1 -- 172.21.15.81:0/1735194008 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f56400cbf30 con 0x7f56200763f0 2024-06-29T16:01:27.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.891+0000 7f5636ffd640 1 -- 172.21.15.81:0/1735194008 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f56400cbf30 con 0x7f56200763f0 2024-06-29T16:01:27.891 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 1 to deep-scrub 2024-06-29T16:01:27.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 -- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56200763f0 msgr2=0x7f5620078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56200763f0 0x7f5620078900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5628034000 tx=0x7f5628004730 comp rx=0 tx=0).stop 2024-06-29T16:01:27.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 -- 172.21.15.81:0/1735194008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56400b3200 msgr2=0x7f56401f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:27.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56400b3200 0x7f56401f6bd0 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7f563000c390 tx=0x7f563000c850 comp rx=0 tx=0).stop 2024-06-29T16:01:27.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 -- 172.21.15.81:0/1735194008 shutdown_connections 2024-06-29T16:01:27.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56200763f0 0x7f5620078900 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56400d0230 0x7f5640154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56400b4f40 0x7f56400c7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 --2- 172.21.15.81:0/1735194008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56400b3200 0x7f56401f6bd0 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:27.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 -- 172.21.15.81:0/1735194008 >> 172.21.15.81:0/1735194008 conn(0x7f56400b3680 msgr2=0x7f5640157c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:27.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 -- 172.21.15.81:0/1735194008 shutdown_connections 2024-06-29T16:01:27.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:27.893+0000 7f5644d8c640 1 -- 172.21.15.81:0/1735194008 wait complete. 2024-06-29T16:01:27.904 INFO:tasks.ceph:Scrubbing osd.2 2024-06-29T16:01:27.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.022+0000 7fc8471a1640 1 -- 172.21.15.81:0/1764674499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8400b4d10 msgr2=0x7fc8400b5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.022+0000 7fc8471a1640 1 --2- 172.21.15.81:0/1764674499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8400b4d10 0x7fc8400b5120 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7fc840072f30 tx=0x7fc82c02a4d0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.022+0000 7fc8471a1640 1 -- 172.21.15.81:0/1764674499 shutdown_connections 2024-06-29T16:01:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.022+0000 7fc8471a1640 1 --2- 172.21.15.81:0/1764674499 >> v2:172.21.15.167:3300/0 conn(0x7fc84013e320 0x7fc84013e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.022+0000 7fc8471a1640 1 --2- 172.21.15.81:0/1764674499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8400b4d10 0x7fc8400b5120 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.022+0000 7fc8471a1640 1 --2- 172.21.15.81:0/1764674499 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc84013dc20 0x7fc84015bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.022+0000 7fc8471a1640 1 -- 172.21.15.81:0/1764674499 >> 172.21.15.81:0/1764674499 conn(0x7fc8400d0920 msgr2=0x7fc8400d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.022+0000 7fc8471a1640 1 -- 172.21.15.81:0/1764674499 shutdown_connections 2024-06-29T16:01:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.022+0000 7fc8471a1640 1 -- 172.21.15.81:0/1764674499 wait complete. 2024-06-29T16:01:28.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.024+0000 7fc8471a1640 1 Processor -- start 2024-06-29T16:01:28.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.024+0000 7fc8471a1640 1 -- start start 2024-06-29T16:01:28.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.025+0000 7fc8471a1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8400b4d10 0x7fc8400cf630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.025+0000 7fc8471a1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84013dc20 0x7fc8401f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.025+0000 7fc8471a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc84013e320 0x7fc8401f5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.025+0000 7fc8471a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc84015b550 con 0x7fc84013e320 2024-06-29T16:01:28.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.025+0000 7fc8471a1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fc84015b3d0 con 0x7fc84013dc20 2024-06-29T16:01:28.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.025+0000 7fc8471a1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fc840072fa0 con 0x7fc8400b4d10 2024-06-29T16:01:28.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc84573e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc84013e320 0x7fc8401f5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc84573e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc84013e320 0x7fc8401f5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42906/0 (socket says 172.21.15.81:42906) 2024-06-29T16:01:28.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc84573e640 1 -- 172.21.15.81:0/1879140961 learned_addr learned my addr 172.21.15.81:0/1879140961 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:28.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc83ffff640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84013dc20 0x7fc8401f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc844f3d640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8400b4d10 0x7fc8400cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc84573e640 1 -- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8400b4d10 msgr2=0x7fc8400cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc84573e640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fc8400b4d10 0x7fc8400cf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc84573e640 1 -- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84013dc20 msgr2=0x7fc8401f58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc84573e640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fc84013dc20 0x7fc8401f58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc84573e640 1 -- 172.21.15.81:0/1879140961 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc840154490 con 0x7fc84013e320 2024-06-29T16:01:28.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc84573e640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc84013e320 0x7fc8401f5e30 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7fc83800dac0 tx=0x7fc83800df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc83d7fa640 1 -- 172.21.15.81:0/1879140961 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc838017070 con 0x7fc84013e320 2024-06-29T16:01:28.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc83d7fa640 1 -- 172.21.15.81:0/1879140961 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc838003970 con 0x7fc84013e320 2024-06-29T16:01:28.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc840154770 con 0x7fc84013e320 2024-06-29T16:01:28.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc83d7fa640 1 -- 172.21.15.81:0/1879140961 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83800be60 con 0x7fc84013e320 2024-06-29T16:01:28.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.026+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc840154d30 con 0x7fc84013e320 2024-06-29T16:01:28.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.027+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc84015bb20 con 0x7fc84013e320 2024-06-29T16:01:28.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.029+0000 7fc83d7fa640 1 -- 172.21.15.81:0/1879140961 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fc838010460 con 0x7fc84013e320 2024-06-29T16:01:28.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.029+0000 7fc83d7fa640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fc824076420 0x7fc824078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.029+0000 7fc83d7fa640 1 -- 172.21.15.81:0/1879140961 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7fc838098320 con 0x7fc84013e320 2024-06-29T16:01:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.029+0000 7fc83d7fa640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] conn(0x7fc82407cb00 0x7fc82407efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.029+0000 7fc83d7fa640 1 -- 172.21.15.81:0/1879140961 --> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc838024e80 con 0x7fc82407cb00 2024-06-29T16:01:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.029+0000 7fc83d7fa640 1 -- 172.21.15.81:0/1879140961 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7fc8380987b0 con 0x7fc84013e320 2024-06-29T16:01:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.029+0000 7fc844f3d640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fc824076420 0x7fc824078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.030+0000 7fc83ffff640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] conn(0x7fc82407cb00 0x7fc82407efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.030+0000 7fc83ffff640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] conn(0x7fc82407cb00 0x7fc82407efd0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.030+0000 7fc844f3d640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fc824076420 0x7fc824078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc83400bb50 tx=0x7fc83400b2e0 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.031+0000 7fc83d7fa640 1 -- 172.21.15.81:0/1879140961 <== osd.2 v2:172.21.15.81:6824/4255314520 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc838024e80 con 0x7fc82407cb00 2024-06-29T16:01:28.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 --> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc84013ab30 con 0x7fc82407cb00 2024-06-29T16:01:28.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc83d7fa640 1 -- 172.21.15.81:0/1879140961 <== osd.2 v2:172.21.15.81:6824/4255314520 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc82c005520 con 0x7fc82407cb00 2024-06-29T16:01:28.044 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T16:01:28.044 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:28.045 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T16:01:28.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 >> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] conn(0x7fc82407cb00 msgr2=0x7fc82407efd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.81:6824/4255314520,v1:172.21.15.81:6825/4255314520] conn(0x7fc82407cb00 0x7fc82407efd0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fc824076420 msgr2=0x7fc824078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fc824076420 0x7fc824078930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc83400bb50 tx=0x7fc83400b2e0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc84013e320 msgr2=0x7fc8401f5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 --2- 172.21.15.81:0/1879140961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc84013e320 0x7fc8401f5e30 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7fc83800dac0 tx=0x7fc83800df80 comp rx=0 tx=0).stop 2024-06-29T16:01:28.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc84573e640 1 -- 172.21.15.81:0/1879140961 reap_dead start 2024-06-29T16:01:28.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 shutdown_connections 2024-06-29T16:01:28.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 >> 172.21.15.81:0/1879140961 conn(0x7fc8400d0920 msgr2=0x7fc840158d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 shutdown_connections 2024-06-29T16:01:28.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.044+0000 7fc8471a1640 1 -- 172.21.15.81:0/1879140961 wait complete. 2024-06-29T16:01:28.055 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-06-29T16:01:28.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.169+0000 7eff7dad2640 1 -- 172.21.15.81:0/4100431514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff780cd150 msgr2=0x7eff78072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.169+0000 7eff7dad2640 1 --2- 172.21.15.81:0/4100431514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff780cd150 0x7eff78072f40 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7eff68009830 tx=0x7eff6802a480 comp rx=0 tx=0).stop 2024-06-29T16:01:28.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.169+0000 7eff7dad2640 1 -- 172.21.15.81:0/4100431514 shutdown_connections 2024-06-29T16:01:28.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.169+0000 7eff7dad2640 1 --2- 172.21.15.81:0/4100431514 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff7815bdf0 0x7eff7806ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.169+0000 7eff7dad2640 1 --2- 172.21.15.81:0/4100431514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff780cd150 0x7eff78072f40 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.169+0000 7eff7dad2640 1 -- 172.21.15.81:0/4100431514 >> 172.21.15.81:0/4100431514 conn(0x7eff780b2920 msgr2=0x7eff780b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.169+0000 7eff7dad2640 1 -- 172.21.15.81:0/4100431514 shutdown_connections 2024-06-29T16:01:28.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.169+0000 7eff7dad2640 1 -- 172.21.15.81:0/4100431514 wait complete. 2024-06-29T16:01:28.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.171+0000 7eff7dad2640 1 Processor -- start 2024-06-29T16:01:28.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.171+0000 7eff7dad2640 1 -- start start 2024-06-29T16:01:28.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff7dad2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff78073480 0x7eff780ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff7dad2640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff780cd150 0x7eff780afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff7dad2640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff7815bdf0 0x7eff780cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff7dad2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7eff7806f660 con 0x7eff78073480 2024-06-29T16:01:28.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff7dad2640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7eff7806f4e0 con 0x7eff7815bdf0 2024-06-29T16:01:28.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff7dad2640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7eff7806f7e0 con 0x7eff780cd150 2024-06-29T16:01:28.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff78073480 0x7eff780ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff78073480 0x7eff780ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42920/0 (socket says 172.21.15.81:42920) 2024-06-29T16:01:28.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff777fe640 1 -- 172.21.15.81:0/2689114696 learned_addr learned my addr 172.21.15.81:0/2689114696 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:28.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff76ffd640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff780cd150 0x7eff780afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff77fff640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff7815bdf0 0x7eff780cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff777fe640 1 -- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff780cd150 msgr2=0x7eff780afd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff777fe640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff780cd150 0x7eff780afd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff777fe640 1 -- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff7815bdf0 msgr2=0x7eff780cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff777fe640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff7815bdf0 0x7eff780cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff777fe640 1 -- 172.21.15.81:0/2689114696 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff780d0bf0 con 0x7eff78073480 2024-06-29T16:01:28.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff777fe640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff78073480 0x7eff780ce250 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7eff6802aa70 tx=0x7eff68033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.173+0000 7eff5ffff640 1 -- 172.21.15.81:0/2689114696 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff68005c90 con 0x7eff78073480 2024-06-29T16:01:28.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.174+0000 7eff5ffff640 1 -- 172.21.15.81:0/2689114696 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff68031e50 con 0x7eff78073480 2024-06-29T16:01:28.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.174+0000 7eff5ffff640 1 -- 172.21.15.81:0/2689114696 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff68005260 con 0x7eff78073480 2024-06-29T16:01:28.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.174+0000 7eff7dad2640 1 -- 172.21.15.81:0/2689114696 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff780d0ed0 con 0x7eff78073480 2024-06-29T16:01:28.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.174+0000 7eff7dad2640 1 -- 172.21.15.81:0/2689114696 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff780d1490 con 0x7eff78073480 2024-06-29T16:01:28.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.175+0000 7eff7dad2640 1 -- 172.21.15.81:0/2689114696 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff7806fd90 con 0x7eff78073480 2024-06-29T16:01:28.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.175+0000 7eff5ffff640 1 -- 172.21.15.81:0/2689114696 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7eff68005400 con 0x7eff78073480 2024-06-29T16:01:28.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.176+0000 7eff5ffff640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7eff640763f0 0x7eff64078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.176+0000 7eff5ffff640 1 -- 172.21.15.81:0/2689114696 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7eff680b8b00 con 0x7eff78073480 2024-06-29T16:01:28.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.176+0000 7eff76ffd640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7eff640763f0 0x7eff64078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.177+0000 7eff76ffd640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7eff640763f0 0x7eff64078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff780ccf00 tx=0x7eff6c009040 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.181+0000 7eff5ffff640 1 -- 172.21.15.81:0/2689114696 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7eff6803f030 con 0x7eff78073480 2024-06-29T16:01:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.373+0000 7eff7dad2640 1 -- 172.21.15.81:0/2689114696 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7eff78070f10 con 0x7eff640763f0 2024-06-29T16:01:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.373+0000 7eff5ffff640 1 -- 172.21.15.81:0/2689114696 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7eff78070f10 con 0x7eff640763f0 2024-06-29T16:01:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 2 to deep-scrub 2024-06-29T16:01:28.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.375+0000 7eff7dad2640 1 -- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7eff640763f0 msgr2=0x7eff64078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.375+0000 7eff7dad2640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7eff640763f0 0x7eff64078900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff780ccf00 tx=0x7eff6c009040 comp rx=0 tx=0).stop 2024-06-29T16:01:28.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.375+0000 7eff7dad2640 1 -- 172.21.15.81:0/2689114696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff78073480 msgr2=0x7eff780ce250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.375+0000 7eff7dad2640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff78073480 0x7eff780ce250 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7eff6802aa70 tx=0x7eff68033c20 comp rx=0 tx=0).stop 2024-06-29T16:01:28.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.375+0000 7eff7dad2640 1 -- 172.21.15.81:0/2689114696 shutdown_connections 2024-06-29T16:01:28.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.375+0000 7eff7dad2640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7eff640763f0 0x7eff64078900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.375+0000 7eff7dad2640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7eff7815bdf0 0x7eff780cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.375+0000 7eff7dad2640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7eff780cd150 0x7eff780afd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.375+0000 7eff7dad2640 1 --2- 172.21.15.81:0/2689114696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff78073480 0x7eff780ce250 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.375+0000 7eff7dad2640 1 -- 172.21.15.81:0/2689114696 >> 172.21.15.81:0/2689114696 conn(0x7eff780b2920 msgr2=0x7eff7806c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.376+0000 7eff7dad2640 1 -- 172.21.15.81:0/2689114696 shutdown_connections 2024-06-29T16:01:28.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.376+0000 7eff7dad2640 1 -- 172.21.15.81:0/2689114696 wait complete. 2024-06-29T16:01:28.387 INFO:tasks.ceph:Scrubbing osd.3 2024-06-29T16:01:28.387 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:28.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.504+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/866717545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f00b21a0 msgr2=0x7fa5f015c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.504+0000 7fa5f5d8c640 1 --2- 172.21.15.81:0/866717545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f00b21a0 0x7fa5f015c0e0 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009830 tx=0x7fa5e402a510 comp rx=0 tx=0).stop 2024-06-29T16:01:28.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.504+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/866717545 shutdown_connections 2024-06-29T16:01:28.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.504+0000 7fa5f5d8c640 1 --2- 172.21.15.81:0/866717545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f00b21a0 0x7fa5f015c0e0 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.504+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/866717545 >> 172.21.15.81:0/866717545 conn(0x7fa5f009aa60 msgr2=0x7fa5f009ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.504+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/866717545 shutdown_connections 2024-06-29T16:01:28.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.504+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/866717545 wait complete. 2024-06-29T16:01:28.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.506+0000 7fa5f5d8c640 1 Processor -- start 2024-06-29T16:01:28.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.506+0000 7fa5f5d8c640 1 -- start start 2024-06-29T16:01:28.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5f5d8c640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa5f00ae8f0 0x7fa5f01453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5f5d8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f00b2cb0 0x7fa5f015c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5f5d8c640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa5f0150ad0 0x7fa5f0154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5f5d8c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f0160290 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5f5d8c640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f00d1b70 con 0x7fa5f0150ad0 2024-06-29T16:01:28.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5f5d8c640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f0160410 con 0x7fa5f00ae8f0 2024-06-29T16:01:28.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5eeffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f00b2cb0 0x7fa5f015c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5eeffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f00b2cb0 0x7fa5f015c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42936/0 (socket says 172.21.15.81:42936) 2024-06-29T16:01:28.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5eeffd640 1 -- 172.21.15.81:0/661814253 learned_addr learned my addr 172.21.15.81:0/661814253 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:28.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5ef7fe640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa5f00ae8f0 0x7fa5f01453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5effff640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa5f0150ad0 0x7fa5f0154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.507+0000 7fa5eeffd640 1 -- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa5f00ae8f0 msgr2=0x7fa5f01453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.508+0000 7fa5eeffd640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fa5f00ae8f0 0x7fa5f01453a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.508+0000 7fa5eeffd640 1 -- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa5f0150ad0 msgr2=0x7fa5f0154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.508+0000 7fa5eeffd640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fa5f0150ad0 0x7fa5f0154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.508+0000 7fa5eeffd640 1 -- 172.21.15.81:0/661814253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f0153270 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.508+0000 7fa5eeffd640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f00b2cb0 0x7fa5f015c6d0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009950 tx=0x7fa5e4033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.508+0000 7fa5d7fff640 1 -- 172.21.15.81:0/661814253 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4031d60 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.508+0000 7fa5d7fff640 1 -- 172.21.15.81:0/661814253 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5e40330d0 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.508+0000 7fa5d7fff640 1 -- 172.21.15.81:0/661814253 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4033520 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.508+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f0153550 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.508+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f0153b10 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.509+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa5f0160900 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.510+0000 7fa5d7fff640 1 -- 172.21.15.81:0/661814253 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fa5e40035f0 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.510+0000 7fa5d7fff640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fa5dc0763f0 0x7fa5dc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.510+0000 7fa5d7fff640 1 -- 172.21.15.81:0/661814253 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7fa5e40b8ed0 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.510+0000 7fa5d7fff640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] conn(0x7fa5dc07cb30 0x7fa5dc07f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.510+0000 7fa5d7fff640 1 -- 172.21.15.81:0/661814253 --> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5dc07f6d0 con 0x7fa5dc07cb30 2024-06-29T16:01:28.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.510+0000 7fa5d7fff640 1 -- 172.21.15.81:0/661814253 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7fa5e40bb5d0 con 0x7fa5f00b2cb0 2024-06-29T16:01:28.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.510+0000 7fa5effff640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] conn(0x7fa5dc07cb30 0x7fa5dc07f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.511+0000 7fa5ef7fe640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fa5dc0763f0 0x7fa5dc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.511+0000 7fa5effff640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] conn(0x7fa5dc07cb30 0x7fa5dc07f000 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.511+0000 7fa5ef7fe640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fa5dc0763f0 0x7fa5dc078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0002200 tx=0x7fa5e00019f0 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.511+0000 7fa5d7fff640 1 -- 172.21.15.81:0/661814253 <== osd.3 v2:172.21.15.81:6809/3633523901 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5dc07f6d0 con 0x7fa5dc07cb30 2024-06-29T16:01:28.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.524+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 --> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa5f015b9f0 con 0x7fa5dc07cb30 2024-06-29T16:01:28.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5d7fff640 1 -- 172.21.15.81:0/661814253 <== osd.3 v2:172.21.15.81:6809/3633523901 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa5f015b9f0 con 0x7fa5dc07cb30 2024-06-29T16:01:28.525 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T16:01:28.525 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:28.525 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T16:01:28.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 >> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] conn(0x7fa5dc07cb30 msgr2=0x7fa5dc07f000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5f5d8c640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.81:6809/3633523901,v1:172.21.15.81:6811/3633523901] conn(0x7fa5dc07cb30 0x7fa5dc07f000 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fa5dc0763f0 msgr2=0x7fa5dc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5f5d8c640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fa5dc0763f0 0x7fa5dc078900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0002200 tx=0x7fa5e00019f0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f00b2cb0 msgr2=0x7fa5f015c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5f5d8c640 1 --2- 172.21.15.81:0/661814253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f00b2cb0 0x7fa5f015c6d0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009950 tx=0x7fa5e4033d60 comp rx=0 tx=0).stop 2024-06-29T16:01:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5effff640 1 -- 172.21.15.81:0/661814253 reap_dead start 2024-06-29T16:01:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 shutdown_connections 2024-06-29T16:01:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 >> 172.21.15.81:0/661814253 conn(0x7fa5f009aa60 msgr2=0x7fa5f00b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 shutdown_connections 2024-06-29T16:01:28.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.525+0000 7fa5f5d8c640 1 -- 172.21.15.81:0/661814253 wait complete. 2024-06-29T16:01:28.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-06-29T16:01:28.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.651+0000 7f4aed735640 1 -- 172.21.15.81:0/3608556958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ad4004800 msgr2=0x7f4ad4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.651+0000 7f4aed735640 1 --2- 172.21.15.81:0/3608556958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ad4004800 0x7f4ad4004c10 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0009b20 tx=0x7f4ad4005830 comp rx=0 tx=0).stop 2024-06-29T16:01:28.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.651+0000 7f4aed735640 1 -- 172.21.15.81:0/3608556958 shutdown_connections 2024-06-29T16:01:28.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.651+0000 7f4aed735640 1 --2- 172.21.15.81:0/3608556958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ad4004800 0x7f4ad4004c10 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.651+0000 7f4aed735640 1 -- 172.21.15.81:0/3608556958 >> 172.21.15.81:0/3608556958 conn(0x7f4ae80b2920 msgr2=0x7f4ae80b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.651+0000 7f4aed735640 1 -- 172.21.15.81:0/3608556958 shutdown_connections 2024-06-29T16:01:28.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.651+0000 7f4aed735640 1 -- 172.21.15.81:0/3608556958 wait complete. 2024-06-29T16:01:28.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.652+0000 7f4aed735640 1 Processor -- start 2024-06-29T16:01:28.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.653+0000 7f4aed735640 1 -- start start 2024-06-29T16:01:28.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4aed735640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ad4004800 0x7f4ae81515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4aed735640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ae80ce350 0x7f4ae81586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4aed735640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ae81e57e0 0x7f4ae81e5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4aed735640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae815b860 con 0x7f4ae81e57e0 2024-06-29T16:01:28.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4aed735640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae815b6e0 con 0x7f4ae80ce350 2024-06-29T16:01:28.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4aed735640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae815b9e0 con 0x7f4ad4004800 2024-06-29T16:01:28.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ae77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ae81e57e0 0x7f4ae81e5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ae6ffd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ad4004800 0x7f4ae81515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ade7fc640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ae80ce350 0x7f4ae81586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ae6ffd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ad4004800 0x7f4ae81515d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.167:3301/0 says I am v2:172.21.15.81:51866/0 (socket says 172.21.15.81:51866) 2024-06-29T16:01:28.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ae6ffd640 1 -- 172.21.15.81:0/3927211793 learned_addr learned my addr 172.21.15.81:0/3927211793 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:28.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ae77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ae81e57e0 0x7f4ae81e5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42940/0 (socket says 172.21.15.81:42940) 2024-06-29T16:01:28.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ae77fe640 1 -- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ad4004800 msgr2=0x7f4ae81515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ae77fe640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ad4004800 0x7f4ae81515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ae77fe640 1 -- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ae80ce350 msgr2=0x7f4ae81586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ae77fe640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ae80ce350 0x7f4ae81586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.654+0000 7f4ae77fe640 1 -- 172.21.15.81:0/3927211793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae81e6130 con 0x7f4ae81e57e0 2024-06-29T16:01:28.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.655+0000 7f4ae77fe640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ae81e57e0 0x7f4ae81e5bf0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80701e0 tx=0x7f4ae000eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.655+0000 7f4adffff640 1 -- 172.21.15.81:0/3927211793 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0011870 con 0x7f4ae81e57e0 2024-06-29T16:01:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.655+0000 7f4adffff640 1 -- 172.21.15.81:0/3927211793 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ae0017070 con 0x7f4ae81e57e0 2024-06-29T16:01:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.655+0000 7f4aed735640 1 -- 172.21.15.81:0/3927211793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae80cec60 con 0x7f4ae81e57e0 2024-06-29T16:01:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.655+0000 7f4adffff640 1 -- 172.21.15.81:0/3927211793 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0011dc0 con 0x7f4ae81e57e0 2024-06-29T16:01:28.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.655+0000 7f4aed735640 1 -- 172.21.15.81:0/3927211793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae80cf220 con 0x7f4ae81e57e0 2024-06-29T16:01:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.656+0000 7f4aed735640 1 -- 172.21.15.81:0/3927211793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ae815d430 con 0x7f4ae81e57e0 2024-06-29T16:01:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.657+0000 7f4adffff640 1 -- 172.21.15.81:0/3927211793 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f4ae001c460 con 0x7f4ae81e57e0 2024-06-29T16:01:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.657+0000 7f4adffff640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4ac80763f0 0x7f4ac8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.657+0000 7f4adffff640 1 -- 172.21.15.81:0/3927211793 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f4ae0097c50 con 0x7f4ae81e57e0 2024-06-29T16:01:28.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.657+0000 7f4ae6ffd640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4ac80763f0 0x7f4ac8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.658+0000 7f4ae6ffd640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4ac80763f0 0x7f4ac8078900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80066b0 tx=0x7f4ad8006b70 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.662+0000 7f4adffff640 1 -- 172.21.15.81:0/3927211793 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f4ae815d430 con 0x7f4ae81e57e0 2024-06-29T16:01:28.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.841+0000 7f4aed735640 1 -- 172.21.15.81:0/3927211793 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f4ae80732c0 con 0x7f4ac80763f0 2024-06-29T16:01:28.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.841+0000 7f4adffff640 1 -- 172.21.15.81:0/3927211793 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4ae80732c0 con 0x7f4ac80763f0 2024-06-29T16:01:28.842 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 3 to deep-scrub 2024-06-29T16:01:28.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.843+0000 7f4aed735640 1 -- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4ac80763f0 msgr2=0x7f4ac8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.843+0000 7f4aed735640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4ac80763f0 0x7f4ac8078900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80066b0 tx=0x7f4ad8006b70 comp rx=0 tx=0).stop 2024-06-29T16:01:28.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.843+0000 7f4aed735640 1 -- 172.21.15.81:0/3927211793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ae81e57e0 msgr2=0x7f4ae81e5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.843+0000 7f4aed735640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ae81e57e0 0x7f4ae81e5bf0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80701e0 tx=0x7f4ae000eb00 comp rx=0 tx=0).stop 2024-06-29T16:01:28.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.843+0000 7f4aed735640 1 -- 172.21.15.81:0/3927211793 shutdown_connections 2024-06-29T16:01:28.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.843+0000 7f4aed735640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ae81e57e0 0x7f4ae81e5bf0 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.843+0000 7f4aed735640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4ac80763f0 0x7f4ac8078900 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.843+0000 7f4aed735640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4ae80ce350 0x7f4ae81586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.843+0000 7f4aed735640 1 --2- 172.21.15.81:0/3927211793 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4ad4004800 0x7f4ae81515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.843+0000 7f4aed735640 1 -- 172.21.15.81:0/3927211793 >> 172.21.15.81:0/3927211793 conn(0x7f4ae80b2920 msgr2=0x7f4ae815e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.844+0000 7f4aed735640 1 -- 172.21.15.81:0/3927211793 shutdown_connections 2024-06-29T16:01:28.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.844+0000 7f4aed735640 1 -- 172.21.15.81:0/3927211793 wait complete. 2024-06-29T16:01:28.855 INFO:tasks.ceph:Scrubbing osd.4 2024-06-29T16:01:28.855 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:28.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.971+0000 7f6df4aa4640 1 -- 172.21.15.81:0/3197140354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00d03b0 msgr2=0x7f6df00d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.971+0000 7f6df4aa4640 1 --2- 172.21.15.81:0/3197140354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00d03b0 0x7f6df00d07c0 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7f6df006ed40 tx=0x7f6de802a560 comp rx=0 tx=0).stop 2024-06-29T16:01:28.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.971+0000 7f6df4aa4640 1 -- 172.21.15.81:0/3197140354 shutdown_connections 2024-06-29T16:01:28.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.971+0000 7f6df4aa4640 1 --2- 172.21.15.81:0/3197140354 >> v2:172.21.15.167:3300/0 conn(0x7f6df00d4770 0x7f6df00b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.971+0000 7f6df4aa4640 1 --2- 172.21.15.81:0/3197140354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00d03b0 0x7f6df00d07c0 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.971+0000 7f6df4aa4640 1 -- 172.21.15.81:0/3197140354 >> 172.21.15.81:0/3197140354 conn(0x7f6df00d3c50 msgr2=0x7f6df00d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.971+0000 7f6df4aa4640 1 -- 172.21.15.81:0/3197140354 shutdown_connections 2024-06-29T16:01:28.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.971+0000 7f6df4aa4640 1 -- 172.21.15.81:0/3197140354 wait complete. 2024-06-29T16:01:28.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.973+0000 7f6df4aa4640 1 Processor -- start 2024-06-29T16:01:28.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.973+0000 7f6df4aa4640 1 -- start start 2024-06-29T16:01:28.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6df4aa4640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6df00d03b0 0x7f6df00c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6df4aa4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00d2440 0x7f6df00cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6df4aa4640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6df00d4770 0x7f6df00cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6df4aa4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6df006cb10 con 0x7f6df00d2440 2024-06-29T16:01:28.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6df4aa4640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f6df00d2d20 con 0x7f6df00d03b0 2024-06-29T16:01:28.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6df4aa4640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f6df006cc90 con 0x7f6df00d4770 2024-06-29T16:01:28.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6dedd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00d2440 0x7f6df00cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6dedd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00d2440 0x7f6df00cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42964/0 (socket says 172.21.15.81:42964) 2024-06-29T16:01:28.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6dedd9b640 1 -- 172.21.15.81:0/2046192768 learned_addr learned my addr 172.21.15.81:0/2046192768 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:28.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6dee59c640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6df00d03b0 0x7f6df00c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6deed9d640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6df00d4770 0x7f6df00cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6dedd9b640 1 -- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6df00d4770 msgr2=0x7f6df00cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6dedd9b640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f6df00d4770 0x7f6df00cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6dedd9b640 1 -- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6df00d03b0 msgr2=0x7f6df00c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6dedd9b640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f6df00d03b0 0x7f6df00c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6dedd9b640 1 -- 172.21.15.81:0/2046192768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df00c8680 con 0x7f6df00d2440 2024-06-29T16:01:28.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6dedd9b640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00d2440 0x7f6df00cceb0 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f6de400e7c0 tx=0x7f6de400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6ddeffd640 1 -- 172.21.15.81:0/2046192768 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de400ccc0 con 0x7f6df00d2440 2024-06-29T16:01:28.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6ddeffd640 1 -- 172.21.15.81:0/2046192768 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6de400ee70 con 0x7f6df00d2440 2024-06-29T16:01:28.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df00a14f0 con 0x7f6df00d2440 2024-06-29T16:01:28.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6ddeffd640 1 -- 172.21.15.81:0/2046192768 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de400ce60 con 0x7f6df00d2440 2024-06-29T16:01:28.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.975+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df015f050 con 0x7f6df00d2440 2024-06-29T16:01:28.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.976+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6df006d220 con 0x7f6df00d2440 2024-06-29T16:01:28.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.977+0000 7f6ddeffd640 1 -- 172.21.15.81:0/2046192768 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f6de4004260 con 0x7f6df00d2440 2024-06-29T16:01:28.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.978+0000 7f6ddeffd640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6dd00763f0 0x7f6dd0078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.978+0000 7f6ddeffd640 1 -- 172.21.15.81:0/2046192768 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f6de4098630 con 0x7f6df00d2440 2024-06-29T16:01:28.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.978+0000 7f6ddeffd640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] conn(0x7f6dd007cb30 0x7f6dd007f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:28.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.978+0000 7f6dee59c640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6dd00763f0 0x7f6dd0078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.978+0000 7f6ddeffd640 1 -- 172.21.15.81:0/2046192768 --> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6dd007f6d0 con 0x7f6dd007cb30 2024-06-29T16:01:28.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.978+0000 7f6ddeffd640 1 -- 172.21.15.81:0/2046192768 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f6de4098ac0 con 0x7f6df00d2440 2024-06-29T16:01:28.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.978+0000 7f6deed9d640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] conn(0x7f6dd007cb30 0x7f6dd007f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:28.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.978+0000 7f6dee59c640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6dd00763f0 0x7f6dd0078900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6de0001e20 tx=0x7f6de0009040 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.979+0000 7f6deed9d640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] conn(0x7f6dd007cb30 0x7f6dd007f000 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:28.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.979+0000 7f6ddeffd640 1 -- 172.21.15.81:0/2046192768 <== osd.4 v2:172.21.15.167:6824/220767630 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6dd007f6d0 con 0x7f6dd007cb30 2024-06-29T16:01:28.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.992+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 --> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6df00a12a0 con 0x7f6dd007cb30 2024-06-29T16:01:28.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6ddeffd640 1 -- 172.21.15.81:0/2046192768 <== osd.4 v2:172.21.15.167:6824/220767630 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6df00a12a0 con 0x7f6dd007cb30 2024-06-29T16:01:28.993 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T16:01:28.993 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:28.993 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T16:01:28.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] conn(0x7f6dd007cb30 msgr2=0x7f6dd007f000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6df4aa4640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:6824/220767630,v1:172.21.15.167:6825/220767630] conn(0x7f6dd007cb30 0x7f6dd007f000 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:28.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6dd00763f0 msgr2=0x7f6dd0078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6df4aa4640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f6dd00763f0 0x7f6dd0078900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6de0001e20 tx=0x7f6de0009040 comp rx=0 tx=0).stop 2024-06-29T16:01:28.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00d2440 msgr2=0x7f6df00cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:28.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6df4aa4640 1 --2- 172.21.15.81:0/2046192768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00d2440 0x7f6df00cceb0 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f6de400e7c0 tx=0x7f6de400ec80 comp rx=0 tx=0).stop 2024-06-29T16:01:28.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6deed9d640 1 -- 172.21.15.81:0/2046192768 reap_dead start 2024-06-29T16:01:28.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 shutdown_connections 2024-06-29T16:01:28.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 >> 172.21.15.81:0/2046192768 conn(0x7f6df00d3c50 msgr2=0x7f6df00b5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:28.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 shutdown_connections 2024-06-29T16:01:28.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:28.993+0000 7f6df4aa4640 1 -- 172.21.15.81:0/2046192768 wait complete. 2024-06-29T16:01:29.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-06-29T16:01:29.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.117+0000 7f56cb647640 1 -- 172.21.15.81:0/2457953089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56c40b35b0 msgr2=0x7f56c415b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.117+0000 7f56cb647640 1 --2- 172.21.15.81:0/2457953089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56c40b35b0 0x7f56c415b5b0 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f56c0009830 tx=0x7f56c002a480 comp rx=0 tx=0).stop 2024-06-29T16:01:29.118 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.117+0000 7f56cb647640 1 -- 172.21.15.81:0/2457953089 shutdown_connections 2024-06-29T16:01:29.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.117+0000 7f56cb647640 1 --2- 172.21.15.81:0/2457953089 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56c40b4430 0x7f56c40cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.117+0000 7f56cb647640 1 --2- 172.21.15.81:0/2457953089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56c40b35b0 0x7f56c415b5b0 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.117+0000 7f56cb647640 1 -- 172.21.15.81:0/2457953089 >> 172.21.15.81:0/2457953089 conn(0x7f56c406ccc0 msgr2=0x7f56c40b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:29.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.117+0000 7f56cb647640 1 -- 172.21.15.81:0/2457953089 shutdown_connections 2024-06-29T16:01:29.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.117+0000 7f56cb647640 1 -- 172.21.15.81:0/2457953089 wait complete. 2024-06-29T16:01:29.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.119+0000 7f56cb647640 1 Processor -- start 2024-06-29T16:01:29.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.119+0000 7f56cb647640 1 -- start start 2024-06-29T16:01:29.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56cb647640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56c406f660 0x7f56c4154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56cb647640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56c40b35b0 0x7f56c4153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56cb647640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56c40b4430 0x7f56c41478c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56cb647640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f56c406f1e0 con 0x7f56c406f660 2024-06-29T16:01:29.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56cb647640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f56c406f060 con 0x7f56c40b35b0 2024-06-29T16:01:29.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56cb647640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f56c406f360 con 0x7f56c40b4430 2024-06-29T16:01:29.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56c93e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56c406f660 0x7f56c4154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56c93e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56c406f660 0x7f56c4154a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42990/0 (socket says 172.21.15.81:42990) 2024-06-29T16:01:29.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56c93e3640 1 -- 172.21.15.81:0/2620982860 learned_addr learned my addr 172.21.15.81:0/2620982860 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:29.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56c8be2640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56c40b35b0 0x7f56c4153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56c9be4640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56c40b4430 0x7f56c41478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56c93e3640 1 -- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56c40b4430 msgr2=0x7f56c41478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.120+0000 7f56c93e3640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56c40b4430 0x7f56c41478c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.121+0000 7f56c93e3640 1 -- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56c40b35b0 msgr2=0x7f56c4153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.121+0000 7f56c93e3640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56c40b35b0 0x7f56c4153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.121+0000 7f56c93e3640 1 -- 172.21.15.81:0/2620982860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c41f5060 con 0x7f56c406f660 2024-06-29T16:01:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.121+0000 7f56c93e3640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56c406f660 0x7f56c4154a80 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f56c002aa70 tx=0x7f56c0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.121+0000 7f56b9ffb640 1 -- 172.21.15.81:0/2620982860 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c0031d60 con 0x7f56c406f660 2024-06-29T16:01:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.121+0000 7f56b9ffb640 1 -- 172.21.15.81:0/2620982860 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56c0033e10 con 0x7f56c406f660 2024-06-29T16:01:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.121+0000 7f56b9ffb640 1 -- 172.21.15.81:0/2620982860 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c0005260 con 0x7f56c406f660 2024-06-29T16:01:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.121+0000 7f56cb647640 1 -- 172.21.15.81:0/2620982860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c4153670 con 0x7f56c406f660 2024-06-29T16:01:29.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.121+0000 7f56cb647640 1 -- 172.21.15.81:0/2620982860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c41f4d80 con 0x7f56c406f660 2024-06-29T16:01:29.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.122+0000 7f56cb647640 1 -- 172.21.15.81:0/2620982860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c415b5b0 con 0x7f56c406f660 2024-06-29T16:01:29.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.123+0000 7f56b9ffb640 1 -- 172.21.15.81:0/2620982860 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f56c0005400 con 0x7f56c406f660 2024-06-29T16:01:29.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.123+0000 7f56b9ffb640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56a80763f0 0x7f56a8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.124+0000 7f56b9ffb640 1 -- 172.21.15.81:0/2620982860 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f56c00b8c70 con 0x7f56c406f660 2024-06-29T16:01:29.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.124+0000 7f56c8be2640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56a80763f0 0x7f56a8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.124+0000 7f56c8be2640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56a80763f0 0x7f56a8078900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56c4072ec0 tx=0x7f56b400b2e0 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:29.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.128+0000 7f56b9ffb640 1 -- 172.21.15.81:0/2620982860 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f56c003f030 con 0x7f56c406f660 2024-06-29T16:01:29.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.310+0000 7f56cb647640 1 -- 172.21.15.81:0/2620982860 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f56c415ca80 con 0x7f56a80763f0 2024-06-29T16:01:29.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.310+0000 7f56b9ffb640 1 -- 172.21.15.81:0/2620982860 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f56c415ca80 con 0x7f56a80763f0 2024-06-29T16:01:29.311 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 4 to deep-scrub 2024-06-29T16:01:29.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 -- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56a80763f0 msgr2=0x7f56a8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56a80763f0 0x7f56a8078900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56c4072ec0 tx=0x7f56b400b2e0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 -- 172.21.15.81:0/2620982860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56c406f660 msgr2=0x7f56c4154a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56c406f660 0x7f56c4154a80 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f56c002aa70 tx=0x7f56c0033c20 comp rx=0 tx=0).stop 2024-06-29T16:01:29.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 -- 172.21.15.81:0/2620982860 shutdown_connections 2024-06-29T16:01:29.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f56c40b4430 0x7f56c41478c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f56a80763f0 0x7f56a8078900 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f56c40b35b0 0x7f56c4153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 --2- 172.21.15.81:0/2620982860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56c406f660 0x7f56c4154a80 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 -- 172.21.15.81:0/2620982860 >> 172.21.15.81:0/2620982860 conn(0x7f56c406ccc0 msgr2=0x7f56c415cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:29.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 -- 172.21.15.81:0/2620982860 shutdown_connections 2024-06-29T16:01:29.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.312+0000 7f56cb647640 1 -- 172.21.15.81:0/2620982860 wait complete. 2024-06-29T16:01:29.324 INFO:tasks.ceph:Scrubbing osd.5 2024-06-29T16:01:29.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:29.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.442+0000 7f3e4cf94640 1 -- 172.21.15.81:0/2775898658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e30003a00 msgr2=0x7f3e30003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.442+0000 7f3e4cf94640 1 --2- 172.21.15.81:0/2775898658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e30003a00 0x7f3e30003e70 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f3e3800c670 tx=0x7f3e30005830 comp rx=0 tx=0).stop 2024-06-29T16:01:29.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.442+0000 7f3e46d9d640 1 -- 172.21.15.81:0/2775898658 reap_dead start 2024-06-29T16:01:29.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.442+0000 7f3e4cf94640 1 -- 172.21.15.81:0/2775898658 shutdown_connections 2024-06-29T16:01:29.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.442+0000 7f3e4cf94640 1 -- 172.21.15.81:0/2775898658 >> 172.21.15.81:0/2775898658 conn(0x7f3e4809aa60 msgr2=0x7f3e4809ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:29.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.443+0000 7f3e4cf94640 1 -- 172.21.15.81:0/2775898658 shutdown_connections 2024-06-29T16:01:29.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.443+0000 7f3e4cf94640 1 -- 172.21.15.81:0/2775898658 wait complete. 2024-06-29T16:01:29.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.444+0000 7f3e4cf94640 1 Processor -- start 2024-06-29T16:01:29.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.444+0000 7f3e4cf94640 1 -- start start 2024-06-29T16:01:29.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e4cf94640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e30003a00 0x7f3e4815dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e4cf94640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e480b66a0 0x7f3e48070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e4cf94640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e48071450 0x7f3e48156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e4cf94640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4815c260 con 0x7f3e480b66a0 2024-06-29T16:01:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e4cf94640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4815c0e0 con 0x7f3e30003a00 2024-06-29T16:01:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e4cf94640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f3e4815c3e0 con 0x7f3e48071450 2024-06-29T16:01:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e45d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e480b66a0 0x7f3e48070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e45d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e480b66a0 0x7f3e48070f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43010/0 (socket says 172.21.15.81:43010) 2024-06-29T16:01:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e45d9b640 1 -- 172.21.15.81:0/3895552040 learned_addr learned my addr 172.21.15.81:0/3895552040 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e45d9b640 1 -- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e48071450 msgr2=0x7f3e48156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e46d9d640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e48071450 0x7f3e48156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e45d9b640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f3e48071450 0x7f3e48156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e45d9b640 1 -- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e30003a00 msgr2=0x7f3e4815dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e4659c640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e30003a00 0x7f3e4815dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e45d9b640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f3e30003a00 0x7f3e4815dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e45d9b640 1 -- 172.21.15.81:0/3895552040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e48152cd0 con 0x7f3e480b66a0 2024-06-29T16:01:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e45d9b640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e480b66a0 0x7f3e48070f10 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c011ab0 tx=0x7f3e3c011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e36ffd640 1 -- 172.21.15.81:0/3895552040 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c01e070 con 0x7f3e480b66a0 2024-06-29T16:01:29.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e36ffd640 1 -- 172.21.15.81:0/3895552040 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e3c00fe70 con 0x7f3e480b66a0 2024-06-29T16:01:29.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4815cc60 con 0x7f3e480b66a0 2024-06-29T16:01:29.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.446+0000 7f3e36ffd640 1 -- 172.21.15.81:0/3895552040 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0042b0 con 0x7f3e480b66a0 2024-06-29T16:01:29.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.447+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4815d220 con 0x7f3e480b66a0 2024-06-29T16:01:29.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.447+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3e4815d8e0 con 0x7f3e480b66a0 2024-06-29T16:01:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.448+0000 7f3e36ffd640 1 -- 172.21.15.81:0/3895552040 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f3e3c021070 con 0x7f3e480b66a0 2024-06-29T16:01:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.449+0000 7f3e36ffd640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3e280763f0 0x7f3e28078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.449+0000 7f3e36ffd640 1 -- 172.21.15.81:0/3895552040 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f3e3c098f60 con 0x7f3e480b66a0 2024-06-29T16:01:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.449+0000 7f3e36ffd640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] conn(0x7f3e2807cad0 0x7f3e2807efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.449+0000 7f3e36ffd640 1 -- 172.21.15.81:0/3895552040 --> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e2807f670 con 0x7f3e2807cad0 2024-06-29T16:01:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.449+0000 7f3e36ffd640 1 -- 172.21.15.81:0/3895552040 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f3e3c0993f0 con 0x7f3e480b66a0 2024-06-29T16:01:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.449+0000 7f3e4659c640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3e280763f0 0x7f3e28078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.449+0000 7f3e46d9d640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] conn(0x7f3e2807cad0 0x7f3e2807efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.450+0000 7f3e4659c640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3e280763f0 0x7f3e28078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3e38033000 tx=0x7f3e480d2010 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:29.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.450+0000 7f3e46d9d640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] conn(0x7f3e2807cad0 0x7f3e2807efa0 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:29.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.451+0000 7f3e36ffd640 1 -- 172.21.15.81:0/3895552040 <== osd.5 v2:172.21.15.167:6800/4186590320 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3e2807f670 con 0x7f3e2807cad0 2024-06-29T16:01:29.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.463+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 --> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3e4815ca10 con 0x7f3e2807cad0 2024-06-29T16:01:29.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.464+0000 7f3e36ffd640 1 -- 172.21.15.81:0/3895552040 <== osd.5 v2:172.21.15.167:6800/4186590320 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3e4815ca10 con 0x7f3e2807cad0 2024-06-29T16:01:29.464 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T16:01:29.465 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:29.465 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T16:01:29.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.464+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] conn(0x7f3e2807cad0 msgr2=0x7f3e2807efa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.464+0000 7f3e4cf94640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:6800/4186590320,v1:172.21.15.167:6801/4186590320] conn(0x7f3e2807cad0 0x7f3e2807efa0 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.464+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3e280763f0 msgr2=0x7f3e28078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.465+0000 7f3e4cf94640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f3e280763f0 0x7f3e28078900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3e38033000 tx=0x7f3e480d2010 comp rx=0 tx=0).stop 2024-06-29T16:01:29.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.465+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e480b66a0 msgr2=0x7f3e48070f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.465+0000 7f3e4cf94640 1 --2- 172.21.15.81:0/3895552040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e480b66a0 0x7f3e48070f10 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c011ab0 tx=0x7f3e3c011f70 comp rx=0 tx=0).stop 2024-06-29T16:01:29.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.465+0000 7f3e46d9d640 1 -- 172.21.15.81:0/3895552040 reap_dead start 2024-06-29T16:01:29.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.465+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 shutdown_connections 2024-06-29T16:01:29.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.465+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 >> 172.21.15.81:0/3895552040 conn(0x7f3e4809aa60 msgr2=0x7f3e480afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:29.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.465+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 shutdown_connections 2024-06-29T16:01:29.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.465+0000 7f3e4cf94640 1 -- 172.21.15.81:0/3895552040 wait complete. 2024-06-29T16:01:29.475 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-06-29T16:01:29.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.588+0000 7f5c21ac1640 1 -- 172.21.15.81:0/256954322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c1c074520 msgr2=0x7f5c1c06f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.588+0000 7f5c21ac1640 1 --2- 172.21.15.81:0/256954322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c1c074520 0x7f5c1c06f140 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009830 tx=0x7f5c0c02a510 comp rx=0 tx=0).stop 2024-06-29T16:01:29.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.588+0000 7f5c21ac1640 1 -- 172.21.15.81:0/256954322 shutdown_connections 2024-06-29T16:01:29.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.588+0000 7f5c21ac1640 1 --2- 172.21.15.81:0/256954322 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c1c06fa90 0x7f5c1c06ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.588+0000 7f5c21ac1640 1 --2- 172.21.15.81:0/256954322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c1c074520 0x7f5c1c06f140 unknown :-1 s=CLOSED pgs=3055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.588+0000 7f5c21ac1640 1 -- 172.21.15.81:0/256954322 >> 172.21.15.81:0/256954322 conn(0x7f5c1c072950 msgr2=0x7f5c1c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:29.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.588+0000 7f5c21ac1640 1 -- 172.21.15.81:0/256954322 shutdown_connections 2024-06-29T16:01:29.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.589+0000 7f5c21ac1640 1 -- 172.21.15.81:0/256954322 wait complete. 2024-06-29T16:01:29.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.590+0000 7f5c21ac1640 1 Processor -- start 2024-06-29T16:01:29.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.590+0000 7f5c21ac1640 1 -- start start 2024-06-29T16:01:29.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c21ac1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c1c06f680 0x7f5c1c14d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c21ac1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c1c06fa90 0x7f5c1c068210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c21ac1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c1c074520 0x7f5c1c068750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c21ac1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1c15b010 con 0x7f5c1c06fa90 2024-06-29T16:01:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c21ac1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1c06d150 con 0x7f5c1c074520 2024-06-29T16:01:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c21ac1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f5c1c15b190 con 0x7f5c1c06f680 2024-06-29T16:01:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c1c06fa90 0x7f5c1c068210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c1c06fa90 0x7f5c1c068210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43030/0 (socket says 172.21.15.81:43030) 2024-06-29T16:01:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1affd640 1 -- 172.21.15.81:0/1568918027 learned_addr learned my addr 172.21.15.81:0/1568918027 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1b7fe640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c1c06f680 0x7f5c1c14d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1bfff640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c1c074520 0x7f5c1c068750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1affd640 1 -- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c1c06f680 msgr2=0x7f5c1c14d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1affd640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c1c06f680 0x7f5c1c14d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1affd640 1 -- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c1c074520 msgr2=0x7f5c1c068750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1affd640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c1c074520 0x7f5c1c068750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1affd640 1 -- 172.21.15.81:0/1568918027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1c1560f0 con 0x7f5c1c06fa90 2024-06-29T16:01:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1bfff640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c1c074520 0x7f5c1c068750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c1affd640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c1c06fa90 0x7f5c1c068210 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f5c1000c430 tx=0x7f5c1000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c03fff640 1 -- 172.21.15.81:0/1568918027 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c10011790 con 0x7f5c1c06fa90 2024-06-29T16:01:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.592+0000 7f5c03fff640 1 -- 172.21.15.81:0/1568918027 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c10017070 con 0x7f5c1c06fa90 2024-06-29T16:01:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.593+0000 7f5c03fff640 1 -- 172.21.15.81:0/1568918027 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c10011ca0 con 0x7f5c1c06fa90 2024-06-29T16:01:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.593+0000 7f5c21ac1640 1 -- 172.21.15.81:0/1568918027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c1c155e10 con 0x7f5c1c06fa90 2024-06-29T16:01:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.593+0000 7f5c21ac1640 1 -- 172.21.15.81:0/1568918027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c1c0cf010 con 0x7f5c1c06fa90 2024-06-29T16:01:29.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.593+0000 7f5c21ac1640 1 -- 172.21.15.81:0/1568918027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c1c0af4d0 con 0x7f5c1c06fa90 2024-06-29T16:01:29.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.594+0000 7f5c03fff640 1 -- 172.21.15.81:0/1568918027 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f5c1001c460 con 0x7f5c1c06fa90 2024-06-29T16:01:29.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.595+0000 7f5c03fff640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f5c08076420 0x7f5c08078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.595+0000 7f5c03fff640 1 -- 172.21.15.81:0/1568918027 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f5c100977a0 con 0x7f5c1c06fa90 2024-06-29T16:01:29.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.595+0000 7f5c1b7fe640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f5c08076420 0x7f5c08078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.596+0000 7f5c1b7fe640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f5c08076420 0x7f5c08078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c034000 tx=0x7f5c0c004730 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:29.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.599+0000 7f5c03fff640 1 -- 172.21.15.81:0/1568918027 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f5c10060e90 con 0x7f5c1c06fa90 2024-06-29T16:01:29.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.780+0000 7f5c21ac1640 1 -- 172.21.15.81:0/1568918027 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f5c1c155ba0 con 0x7f5c08076420 2024-06-29T16:01:29.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.781+0000 7f5c03fff640 1 -- 172.21.15.81:0/1568918027 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5c1c155ba0 con 0x7f5c08076420 2024-06-29T16:01:29.782 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 5 to deep-scrub 2024-06-29T16:01:29.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 -- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f5c08076420 msgr2=0x7f5c08078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f5c08076420 0x7f5c08078930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c034000 tx=0x7f5c0c004730 comp rx=0 tx=0).stop 2024-06-29T16:01:29.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 -- 172.21.15.81:0/1568918027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c1c06fa90 msgr2=0x7f5c1c068210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c1c06fa90 0x7f5c1c068210 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f5c1000c430 tx=0x7f5c1000c8f0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 -- 172.21.15.81:0/1568918027 shutdown_connections 2024-06-29T16:01:29.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f5c08076420 0x7f5c08078930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f5c1c074520 0x7f5c1c068750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c1c06fa90 0x7f5c1c068210 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 --2- 172.21.15.81:0/1568918027 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f5c1c06f680 0x7f5c1c14d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 -- 172.21.15.81:0/1568918027 >> 172.21.15.81:0/1568918027 conn(0x7f5c1c072950 msgr2=0x7f5c1c06c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:29.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 -- 172.21.15.81:0/1568918027 shutdown_connections 2024-06-29T16:01:29.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.783+0000 7f5c21ac1640 1 -- 172.21.15.81:0/1568918027 wait complete. 2024-06-29T16:01:29.794 INFO:tasks.ceph:Scrubbing osd.6 2024-06-29T16:01:29.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:29.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.912+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1287445072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be0003a50 msgr2=0x7f4be0003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.912+0000 7f4bf92fd640 1 --2- 172.21.15.81:0/1287445072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be0003a50 0x7f4be0003ec0 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f4be40013d0 tx=0x7f4be0005830 comp rx=0 tx=0).stop 2024-06-29T16:01:29.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.912+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1287445072 shutdown_connections 2024-06-29T16:01:29.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.912+0000 7f4bf92fd640 1 --2- 172.21.15.81:0/1287445072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be0003a50 0x7f4be0003ec0 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.912+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1287445072 >> 172.21.15.81:0/1287445072 conn(0x7f4bf40b35a0 msgr2=0x7f4bf40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:29.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.912+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1287445072 shutdown_connections 2024-06-29T16:01:29.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.912+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1287445072 wait complete. 2024-06-29T16:01:29.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.914+0000 7f4bf92fd640 1 Processor -- start 2024-06-29T16:01:29.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.914+0000 7f4bf92fd640 1 -- start start 2024-06-29T16:01:29.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf92fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be0003a50 0x7f4bf41e3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf92fd640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bf409d970 0x7f4bf409dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf92fd640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bf40b5b90 0x7f4bf40b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf92fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf40d02f0 con 0x7f4be0003a50 2024-06-29T16:01:29.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf92fd640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf40d0170 con 0x7f4bf40b5b90 2024-06-29T16:01:29.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf92fd640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf40d0470 con 0x7f4bf409d970 2024-06-29T16:01:29.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be0003a50 0x7f4bf41e3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be0003a50 0x7f4bf41e3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43054/0 (socket says 172.21.15.81:43054) 2024-06-29T16:01:29.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf2ffd640 1 -- 172.21.15.81:0/1939898899 learned_addr learned my addr 172.21.15.81:0/1939898899 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:29.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf27fc640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bf409d970 0x7f4bf409dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf37fe640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bf40b5b90 0x7f4bf40b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf2ffd640 1 -- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bf409d970 msgr2=0x7f4bf409dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf2ffd640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f4bf409d970 0x7f4bf409dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf2ffd640 1 -- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bf40b5b90 msgr2=0x7f4bf40b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf2ffd640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f4bf40b5b90 0x7f4bf40b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.915+0000 7f4bf2ffd640 1 -- 172.21.15.81:0/1939898899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf40b64e0 con 0x7f4be0003a50 2024-06-29T16:01:29.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.916+0000 7f4bf2ffd640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be0003a50 0x7f4bf41e3bb0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f4be402b5f0 tx=0x7f4be4001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:29.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.916+0000 7f4beb7fe640 1 -- 172.21.15.81:0/1939898899 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4004d50 con 0x7f4be0003a50 2024-06-29T16:01:29.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.916+0000 7f4beb7fe640 1 -- 172.21.15.81:0/1939898899 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4be402b940 con 0x7f4be0003a50 2024-06-29T16:01:29.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.916+0000 7f4beb7fe640 1 -- 172.21.15.81:0/1939898899 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be402bd50 con 0x7f4be0003a50 2024-06-29T16:01:29.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.916+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf406e7f0 con 0x7f4be0003a50 2024-06-29T16:01:29.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.916+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf406edb0 con 0x7f4be0003a50 2024-06-29T16:01:29.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.917+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4bf40cc930 con 0x7f4be0003a50 2024-06-29T16:01:29.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.917+0000 7f4beb7fe640 1 -- 172.21.15.81:0/1939898899 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f4be404a440 con 0x7f4be0003a50 2024-06-29T16:01:29.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.918+0000 7f4beb7fe640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4bd4076420 0x7f4bd4078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.918+0000 7f4beb7fe640 1 -- 172.21.15.81:0/1939898899 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f4be40b8180 con 0x7f4be0003a50 2024-06-29T16:01:29.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.918+0000 7f4beb7fe640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] conn(0x7f4bd407cb00 0x7f4bd407efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:29.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.918+0000 7f4beb7fe640 1 -- 172.21.15.81:0/1939898899 --> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4bd407f6a0 con 0x7f4bd407cb00 2024-06-29T16:01:29.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.918+0000 7f4beb7fe640 1 -- 172.21.15.81:0/1939898899 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f4be40b8610 con 0x7f4be0003a50 2024-06-29T16:01:29.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.918+0000 7f4bf27fc640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4bd4076420 0x7f4bd4078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.918+0000 7f4bf37fe640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] conn(0x7f4bd407cb00 0x7f4bd407efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:29.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.919+0000 7f4bf27fc640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4bd4076420 0x7f4bd4078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4bf409cc70 tx=0x7f4bdc006b70 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:29.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.919+0000 7f4bf37fe640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] conn(0x7f4bd407cb00 0x7f4bd407efd0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:29.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.920+0000 7f4beb7fe640 1 -- 172.21.15.81:0/1939898899 <== osd.6 v2:172.21.15.167:6816/2006194601 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bd407f6a0 con 0x7f4bd407cb00 2024-06-29T16:01:29.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.933+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 --> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4bf41f6840 con 0x7f4bd407cb00 2024-06-29T16:01:29.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.933+0000 7f4beb7fe640 1 -- 172.21.15.81:0/1939898899 <== osd.6 v2:172.21.15.167:6816/2006194601 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4bf41f6840 con 0x7f4bd407cb00 2024-06-29T16:01:29.934 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T16:01:29.934 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:29.934 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T16:01:29.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.933+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] conn(0x7f4bd407cb00 msgr2=0x7f4bd407efd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.933+0000 7f4bf92fd640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:6816/2006194601,v1:172.21.15.167:6817/2006194601] conn(0x7f4bd407cb00 0x7f4bd407efd0 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:29.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.933+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4bd4076420 msgr2=0x7f4bd4078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.933+0000 7f4bf92fd640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f4bd4076420 0x7f4bd4078930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4bf409cc70 tx=0x7f4bdc006b70 comp rx=0 tx=0).stop 2024-06-29T16:01:29.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.933+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be0003a50 msgr2=0x7f4bf41e3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:29.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.934+0000 7f4bf92fd640 1 --2- 172.21.15.81:0/1939898899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be0003a50 0x7f4bf41e3bb0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f4be402b5f0 tx=0x7f4be4001b80 comp rx=0 tx=0).stop 2024-06-29T16:01:29.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.934+0000 7f4bf37fe640 1 -- 172.21.15.81:0/1939898899 reap_dead start 2024-06-29T16:01:29.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.934+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 shutdown_connections 2024-06-29T16:01:29.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.934+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 >> 172.21.15.81:0/1939898899 conn(0x7f4bf40b35a0 msgr2=0x7f4bf4070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:29.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.934+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 shutdown_connections 2024-06-29T16:01:29.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:29.934+0000 7f4bf92fd640 1 -- 172.21.15.81:0/1939898899 wait complete. 2024-06-29T16:01:29.945 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-06-29T16:01:30.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.058+0000 7fd6c18c1640 1 -- 172.21.15.81:0/4262503367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc071f20 msgr2=0x7fd6bc144a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.058+0000 7fd6c18c1640 1 --2- 172.21.15.81:0/4262503367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc071f20 0x7fd6bc144a20 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009830 tx=0x7fd6ac02a560 comp rx=0 tx=0).stop 2024-06-29T16:01:30.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.058+0000 7fd6c18c1640 1 -- 172.21.15.81:0/4262503367 shutdown_connections 2024-06-29T16:01:30.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.058+0000 7fd6c18c1640 1 --2- 172.21.15.81:0/4262503367 >> v2:172.21.15.167:3300/0 conn(0x7fd6bc06f4e0 0x7fd6bc15c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.058+0000 7fd6c18c1640 1 --2- 172.21.15.81:0/4262503367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc071f20 0x7fd6bc144a20 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.058+0000 7fd6c18c1640 1 -- 172.21.15.81:0/4262503367 >> 172.21.15.81:0/4262503367 conn(0x7fd6bc06ccc0 msgr2=0x7fd6bc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.058+0000 7fd6c18c1640 1 -- 172.21.15.81:0/4262503367 shutdown_connections 2024-06-29T16:01:30.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.058+0000 7fd6c18c1640 1 -- 172.21.15.81:0/4262503367 wait complete. 2024-06-29T16:01:30.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.060+0000 7fd6c18c1640 1 Processor -- start 2024-06-29T16:01:30.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.060+0000 7fd6c18c1640 1 -- start start 2024-06-29T16:01:30.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.061+0000 7fd6c18c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc071f20 0x7fd6bc0a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.061+0000 7fd6c18c1640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6bc145180 0x7fd6bc0af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6c18c1640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6bc1f2b20 0x7fd6bc0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6c18c1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc15d220 con 0x7fd6bc071f20 2024-06-29T16:01:30.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6c18c1640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc15d0a0 con 0x7fd6bc145180 2024-06-29T16:01:30.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6c18c1640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd6bc15d3a0 con 0x7fd6bc1f2b20 2024-06-29T16:01:30.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6baffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc071f20 0x7fd6bc0a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6baffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc071f20 0x7fd6bc0a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51736/0 (socket says 172.21.15.81:51736) 2024-06-29T16:01:30.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6baffd640 1 -- 172.21.15.81:0/3533403301 learned_addr learned my addr 172.21.15.81:0/3533403301 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:30.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6bb7fe640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6bc1f2b20 0x7fd6bc0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6b3fff640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6bc145180 0x7fd6bc0af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6baffd640 1 -- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6bc1f2b20 msgr2=0x7fd6bc0ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6baffd640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6bc1f2b20 0x7fd6bc0ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6baffd640 1 -- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6bc145180 msgr2=0x7fd6bc0af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6baffd640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6bc145180 0x7fd6bc0af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6baffd640 1 -- 172.21.15.81:0/3533403301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6bc0a9dc0 con 0x7fd6bc071f20 2024-06-29T16:01:30.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6bb7fe640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6bc1f2b20 0x7fd6bc0ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:30.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6b3fff640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6bc145180 0x7fd6bc0af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-29T16:01:30.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6baffd640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc071f20 0x7fd6bc0a94c0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009950 tx=0x7fd6ac033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6b8ff9640 1 -- 172.21.15.81:0/3533403301 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac031d60 con 0x7fd6bc071f20 2024-06-29T16:01:30.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6b8ff9640 1 -- 172.21.15.81:0/3533403301 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6ac033e10 con 0x7fd6bc071f20 2024-06-29T16:01:30.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6b8ff9640 1 -- 172.21.15.81:0/3533403301 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac005260 con 0x7fd6bc071f20 2024-06-29T16:01:30.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6c18c1640 1 -- 172.21.15.81:0/3533403301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6bc0aa0a0 con 0x7fd6bc071f20 2024-06-29T16:01:30.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.062+0000 7fd6c18c1640 1 -- 172.21.15.81:0/3533403301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6bc0aa660 con 0x7fd6bc071f20 2024-06-29T16:01:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.063+0000 7fd6c18c1640 1 -- 172.21.15.81:0/3533403301 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6bc15ea40 con 0x7fd6bc071f20 2024-06-29T16:01:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.064+0000 7fd6b8ff9640 1 -- 172.21.15.81:0/3533403301 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fd6ac005400 con 0x7fd6bc071f20 2024-06-29T16:01:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.064+0000 7fd6b8ff9640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd69c0763f0 0x7fd69c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.065+0000 7fd6b8ff9640 1 -- 172.21.15.81:0/3533403301 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7fd6ac0b8b00 con 0x7fd6bc071f20 2024-06-29T16:01:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.065+0000 7fd6b3fff640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd69c0763f0 0x7fd69c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.065+0000 7fd6b3fff640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd69c0763f0 0x7fd69c078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc144d30 tx=0x7fd6a4009040 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.069+0000 7fd6b8ff9640 1 -- 172.21.15.81:0/3533403301 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd6ac03f030 con 0x7fd6bc071f20 2024-06-29T16:01:30.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.250+0000 7fd6c18c1640 1 -- 172.21.15.81:0/3533403301 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fd6bc0afb00 con 0x7fd69c0763f0 2024-06-29T16:01:30.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.251+0000 7fd6b8ff9640 1 -- 172.21.15.81:0/3533403301 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd6bc0afb00 con 0x7fd69c0763f0 2024-06-29T16:01:30.251 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 6 to deep-scrub 2024-06-29T16:01:30.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 -- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd69c0763f0 msgr2=0x7fd69c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd69c0763f0 0x7fd69c078900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc144d30 tx=0x7fd6a4009040 comp rx=0 tx=0).stop 2024-06-29T16:01:30.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 -- 172.21.15.81:0/3533403301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc071f20 msgr2=0x7fd6bc0a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc071f20 0x7fd6bc0a94c0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009950 tx=0x7fd6ac033c20 comp rx=0 tx=0).stop 2024-06-29T16:01:30.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 -- 172.21.15.81:0/3533403301 shutdown_connections 2024-06-29T16:01:30.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd6bc1f2b20 0x7fd6bc0ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd69c0763f0 0x7fd69c078900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd6bc145180 0x7fd6bc0af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 --2- 172.21.15.81:0/3533403301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc071f20 0x7fd6bc0a94c0 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 -- 172.21.15.81:0/3533403301 >> 172.21.15.81:0/3533403301 conn(0x7fd6bc06ccc0 msgr2=0x7fd6bc0b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 -- 172.21.15.81:0/3533403301 shutdown_connections 2024-06-29T16:01:30.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.253+0000 7fd6c18c1640 1 -- 172.21.15.81:0/3533403301 wait complete. 2024-06-29T16:01:30.264 INFO:tasks.ceph:Scrubbing osd.7 2024-06-29T16:01:30.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-06-29T16:01:30.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.381+0000 7f098921f640 1 -- 172.21.15.81:0/138763422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09840b1a90 msgr2=0x7f09840b1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.381+0000 7f098921f640 1 --2- 172.21.15.81:0/138763422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09840b1a90 0x7f09840b1ea0 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7f098409b6c0 tx=0x7f097c02aa30 comp rx=0 tx=0).stop 2024-06-29T16:01:30.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.381+0000 7f098921f640 1 -- 172.21.15.81:0/138763422 shutdown_connections 2024-06-29T16:01:30.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.381+0000 7f098921f640 1 --2- 172.21.15.81:0/138763422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09840b1a90 0x7f09840b1ea0 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.381+0000 7f098921f640 1 --2- 172.21.15.81:0/138763422 >> v2:172.21.15.167:3300/0 conn(0x7f09840b2cb0 0x7f09840d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.381+0000 7f098921f640 1 --2- 172.21.15.81:0/138763422 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f09840ae8f0 0x7f098415c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.381+0000 7f098921f640 1 -- 172.21.15.81:0/138763422 >> 172.21.15.81:0/138763422 conn(0x7f098409aa60 msgr2=0x7f098409ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.381+0000 7f098921f640 1 -- 172.21.15.81:0/138763422 shutdown_connections 2024-06-29T16:01:30.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.381+0000 7f098921f640 1 -- 172.21.15.81:0/138763422 wait complete. 2024-06-29T16:01:30.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.383+0000 7f098921f640 1 Processor -- start 2024-06-29T16:01:30.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.383+0000 7f098921f640 1 -- start start 2024-06-29T16:01:30.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f098921f640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f09840ae8f0 0x7f09841f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f098921f640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f09840b1a90 0x7f09841f5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f098921f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09840b2cb0 0x7f09840712e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f098921f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f09840ceb80 con 0x7f09840b2cb0 2024-06-29T16:01:30.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f098921f640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f09840cea00 con 0x7f09840b1a90 2024-06-29T16:01:30.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f098921f640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f09840ced00 con 0x7f09840ae8f0 2024-06-29T16:01:30.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f098359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09840b2cb0 0x7f09840712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f098359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09840b2cb0 0x7f09840712e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51748/0 (socket says 172.21.15.81:51748) 2024-06-29T16:01:30.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f098359e640 1 -- 172.21.15.81:0/2982312471 learned_addr learned my addr 172.21.15.81:0/2982312471 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:30.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f0982d9d640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f09840ae8f0 0x7f09841f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.384+0000 7f098259c640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f09840b1a90 0x7f09841f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f098359e640 1 -- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f09840ae8f0 msgr2=0x7f09841f35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f098359e640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f09840ae8f0 0x7f09841f35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f098359e640 1 -- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f09840b1a90 msgr2=0x7f09841f5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f098359e640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f09840b1a90 0x7f09841f5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f098359e640 1 -- 172.21.15.81:0/2982312471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0984156570 con 0x7f09840b2cb0 2024-06-29T16:01:30.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f098359e640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09840b2cb0 0x7f09840712e0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f097c009d60 tx=0x7f097c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f096b7fe640 1 -- 172.21.15.81:0/2982312471 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c033050 con 0x7f09840b2cb0 2024-06-29T16:01:30.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f096b7fe640 1 -- 172.21.15.81:0/2982312471 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f097c004450 con 0x7f09840b2cb0 2024-06-29T16:01:30.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f096b7fe640 1 -- 172.21.15.81:0/2982312471 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c038450 con 0x7f09840b2cb0 2024-06-29T16:01:30.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0984156850 con 0x7f09840b2cb0 2024-06-29T16:01:30.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.385+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09841f5d60 con 0x7f09840b2cb0 2024-06-29T16:01:30.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.386+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f09840b1ea0 con 0x7f09840b2cb0 2024-06-29T16:01:30.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.387+0000 7f096b7fe640 1 -- 172.21.15.81:0/2982312471 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f097c036070 con 0x7f09840b2cb0 2024-06-29T16:01:30.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.387+0000 7f096b7fe640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f0970076420 0x7f0970078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.387+0000 7f096b7fe640 1 -- 172.21.15.81:0/2982312471 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f097c0b8e70 con 0x7f09840b2cb0 2024-06-29T16:01:30.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.387+0000 7f096b7fe640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] conn(0x7f097007cb00 0x7f097007efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.387+0000 7f096b7fe640 1 -- 172.21.15.81:0/2982312471 --> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f097c03ae50 con 0x7f097007cb00 2024-06-29T16:01:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.387+0000 7f096b7fe640 1 -- 172.21.15.81:0/2982312471 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f097c0bb570 con 0x7f09840b2cb0 2024-06-29T16:01:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.387+0000 7f0982d9d640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f0970076420 0x7f0970078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.388+0000 7f098259c640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] conn(0x7f097007cb00 0x7f097007efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.388+0000 7f0982d9d640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f0970076420 0x7f0970078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0974005fd0 tx=0x7f097400b260 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.388+0000 7f098259c640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] conn(0x7f097007cb00 0x7f097007efd0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.389+0000 7f096b7fe640 1 -- 172.21.15.81:0/2982312471 <== osd.7 v2:172.21.15.167:6808/607598008 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f097c03ae50 con 0x7f097007cb00 2024-06-29T16:01:30.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.402+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 --> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0984159040 con 0x7f097007cb00 2024-06-29T16:01:30.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f096b7fe640 1 -- 172.21.15.81:0/2982312471 <== osd.7 v2:172.21.15.167:6808/607598008 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0984159040 con 0x7f097007cb00 2024-06-29T16:01:30.403 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-06-29T16:01:30.403 INFO:teuthology.orchestra.run.smithi081.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-29T16:01:30.404 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-06-29T16:01:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] conn(0x7f097007cb00 msgr2=0x7f097007efd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098921f640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:6808/607598008,v1:172.21.15.167:6809/607598008] conn(0x7f097007cb00 0x7f097007efd0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f0970076420 msgr2=0x7f0970078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098921f640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f0970076420 0x7f0970078930 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0974005fd0 tx=0x7f097400b260 comp rx=0 tx=0).stop 2024-06-29T16:01:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09840b2cb0 msgr2=0x7f09840712e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098921f640 1 --2- 172.21.15.81:0/2982312471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09840b2cb0 0x7f09840712e0 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f097c009d60 tx=0x7f097c003f10 comp rx=0 tx=0).stop 2024-06-29T16:01:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098359e640 1 -- 172.21.15.81:0/2982312471 reap_dead start 2024-06-29T16:01:30.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 shutdown_connections 2024-06-29T16:01:30.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 >> 172.21.15.81:0/2982312471 conn(0x7f098409aa60 msgr2=0x7f098415b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 shutdown_connections 2024-06-29T16:01:30.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.403+0000 7f098921f640 1 -- 172.21.15.81:0/2982312471 wait complete. 2024-06-29T16:01:30.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-06-29T16:01:30.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.527+0000 7fd171465640 1 -- 172.21.15.81:0/2556516901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c0b7290 msgr2=0x7fd16c0b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.527+0000 7fd171465640 1 --2- 172.21.15.81:0/2556516901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c0b7290 0x7fd16c0b76a0 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fd16c08e230 tx=0x7fd15c02a5f0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.527+0000 7fd171465640 1 -- 172.21.15.81:0/2556516901 shutdown_connections 2024-06-29T16:01:30.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.527+0000 7fd171465640 1 --2- 172.21.15.81:0/2556516901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c0b7290 0x7fd16c0b76a0 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.527+0000 7fd171465640 1 --2- 172.21.15.81:0/2556516901 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16c0b5250 0x7fd16c0d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.527+0000 7fd171465640 1 --2- 172.21.15.81:0/2556516901 >> v2:172.21.15.167:3300/0 conn(0x7fd16c0b9040 0x7fd16c0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.527+0000 7fd171465640 1 -- 172.21.15.81:0/2556516901 >> 172.21.15.81:0/2556516901 conn(0x7fd16c09cf30 msgr2=0x7fd16c0d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.527+0000 7fd171465640 1 -- 172.21.15.81:0/2556516901 shutdown_connections 2024-06-29T16:01:30.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.527+0000 7fd171465640 1 -- 172.21.15.81:0/2556516901 wait complete. 2024-06-29T16:01:30.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.529+0000 7fd171465640 1 Processor -- start 2024-06-29T16:01:30.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.529+0000 7fd171465640 1 -- start start 2024-06-29T16:01:30.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd171465640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd16c0b5250 0x7fd16c1f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd171465640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c0b7290 0x7fd16c1f4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd171465640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16c0b9040 0x7fd16c0cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd171465640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c0b7ff0 con 0x7fd16c0b7290 2024-06-29T16:01:30.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd171465640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c0b7e70 con 0x7fd16c0b5250 2024-06-29T16:01:30.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd171465640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fd16c0b8170 con 0x7fd16c0b9040 2024-06-29T16:01:30.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c0b7290 0x7fd16c1f4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c0b7290 0x7fd16c1f4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51768/0 (socket says 172.21.15.81:51768) 2024-06-29T16:01:30.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16a7fc640 1 -- 172.21.15.81:0/833042428 learned_addr learned my addr 172.21.15.81:0/833042428 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:30.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16b7fe640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16c0b9040 0x7fd16c0cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16affd640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd16c0b5250 0x7fd16c1f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16a7fc640 1 -- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16c0b9040 msgr2=0x7fd16c0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16a7fc640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16c0b9040 0x7fd16c0cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16a7fc640 1 -- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd16c0b5250 msgr2=0x7fd16c1f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16a7fc640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd16c0b5250 0x7fd16c1f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16a7fc640 1 -- 172.21.15.81:0/833042428 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c1f2680 con 0x7fd16c0b7290 2024-06-29T16:01:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd16a7fc640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c0b7290 0x7fd16c1f4c30 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7fd15400d680 tx=0x7fd15400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd1637fe640 1 -- 172.21.15.81:0/833042428 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd154017070 con 0x7fd16c0b7290 2024-06-29T16:01:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd1637fe640 1 -- 172.21.15.81:0/833042428 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd154005310 con 0x7fd16c0b7290 2024-06-29T16:01:30.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd171465640 1 -- 172.21.15.81:0/833042428 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c1f5170 con 0x7fd16c0b7290 2024-06-29T16:01:30.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.531+0000 7fd1637fe640 1 -- 172.21.15.81:0/833042428 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd154009820 con 0x7fd16c0b7290 2024-06-29T16:01:30.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.532+0000 7fd171465640 1 -- 172.21.15.81:0/833042428 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c1f55f0 con 0x7fd16c0b7290 2024-06-29T16:01:30.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.532+0000 7fd171465640 1 -- 172.21.15.81:0/833042428 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16c1f5340 con 0x7fd16c0b7290 2024-06-29T16:01:30.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.533+0000 7fd1637fe640 1 -- 172.21.15.81:0/833042428 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fd1540099c0 con 0x7fd16c0b7290 2024-06-29T16:01:30.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.534+0000 7fd1637fe640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd14c0763f0 0x7fd14c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.534+0000 7fd1637fe640 1 -- 172.21.15.81:0/833042428 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7fd154097bb0 con 0x7fd16c0b7290 2024-06-29T16:01:30.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.534+0000 7fd16affd640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd14c0763f0 0x7fd14c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.534+0000 7fd16affd640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd14c0763f0 0x7fd14c078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd164005c40 tx=0x7fd164005bd0 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.538+0000 7fd1637fe640 1 -- 172.21.15.81:0/833042428 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fd154061190 con 0x7fd16c0b7290 2024-06-29T16:01:30.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.720+0000 7fd171465640 1 -- 172.21.15.81:0/833042428 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fd16c160150 con 0x7fd14c0763f0 2024-06-29T16:01:30.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.721+0000 7fd1637fe640 1 -- 172.21.15.81:0/833042428 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd16c160150 con 0x7fd14c0763f0 2024-06-29T16:01:30.722 INFO:teuthology.orchestra.run.smithi081.stderr:instructed osd(s) 7 to deep-scrub 2024-06-29T16:01:30.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 -- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd14c0763f0 msgr2=0x7fd14c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd14c0763f0 0x7fd14c078900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd164005c40 tx=0x7fd164005bd0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 -- 172.21.15.81:0/833042428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c0b7290 msgr2=0x7fd16c1f4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c0b7290 0x7fd16c1f4c30 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7fd15400d680 tx=0x7fd15400db40 comp rx=0 tx=0).stop 2024-06-29T16:01:30.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 -- 172.21.15.81:0/833042428 shutdown_connections 2024-06-29T16:01:30.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fd16c0b9040 0x7fd16c0cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c0b7290 0x7fd16c1f4c30 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fd14c0763f0 0x7fd14c078900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 --2- 172.21.15.81:0/833042428 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fd16c0b5250 0x7fd16c1f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 -- 172.21.15.81:0/833042428 >> 172.21.15.81:0/833042428 conn(0x7fd16c09cf30 msgr2=0x7fd16c0d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 -- 172.21.15.81:0/833042428 shutdown_connections 2024-06-29T16:01:30.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.723+0000 7fd171465640 1 -- 172.21.15.81:0/833042428 wait complete. 2024-06-29T16:01:30.734 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T16:01:30.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.847+0000 7f2d5889e640 1 -- 172.21.15.81:0/2464471407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540d0b90 msgr2=0x7f2d540d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.847+0000 7f2d5889e640 1 --2- 172.21.15.81:0/2464471407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540d0b90 0x7f2d540d1a80 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f2d5408e330 tx=0x7f2d4802a610 comp rx=0 tx=0).stop 2024-06-29T16:01:30.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.848+0000 7f2d5889e640 1 -- 172.21.15.81:0/2464471407 shutdown_connections 2024-06-29T16:01:30.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.848+0000 7f2d5889e640 1 --2- 172.21.15.81:0/2464471407 >> v2:172.21.15.167:3300/0 conn(0x7f2d540b5250 0x7f2d540d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.848+0000 7f2d5889e640 1 --2- 172.21.15.81:0/2464471407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540d0b90 0x7f2d540d1a80 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.848+0000 7f2d5889e640 1 -- 172.21.15.81:0/2464471407 >> 172.21.15.81:0/2464471407 conn(0x7f2d5409cf30 msgr2=0x7f2d540d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:30.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.848+0000 7f2d5889e640 1 -- 172.21.15.81:0/2464471407 shutdown_connections 2024-06-29T16:01:30.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.848+0000 7f2d5889e640 1 -- 172.21.15.81:0/2464471407 wait complete. 2024-06-29T16:01:30.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.849+0000 7f2d5889e640 1 Processor -- start 2024-06-29T16:01:30.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.850+0000 7f2d5889e640 1 -- start start 2024-06-29T16:01:30.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5889e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540b5250 0x7f2d541f7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5889e640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b9040 0x7f2d540b2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5889e640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b41a0 0x7f2d540a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5889e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2d540d2500 con 0x7f2d540b5250 2024-06-29T16:01:30.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5889e640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f2d540b8530 con 0x7f2d540b41a0 2024-06-29T16:01:30.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5889e640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f2d540d2680 con 0x7f2d540b9040 2024-06-29T16:01:30.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540b5250 0x7f2d541f7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540b5250 0x7f2d541f7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51800/0 (socket says 172.21.15.81:51800) 2024-06-29T16:01:30.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5259c640 1 -- 172.21.15.81:0/509908419 learned_addr learned my addr 172.21.15.81:0/509908419 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:30.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d51d9b640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b9040 0x7f2d540b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d52d9d640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b41a0 0x7f2d540a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5259c640 1 -- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b9040 msgr2=0x7f2d540b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5259c640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b9040 0x7f2d540b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5259c640 1 -- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b41a0 msgr2=0x7f2d540a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:30.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5259c640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b41a0 0x7f2d540a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:30.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5259c640 1 -- 172.21.15.81:0/509908419 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d540b2d30 con 0x7f2d540b5250 2024-06-29T16:01:30.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d52d9d640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b41a0 0x7f2d540a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-29T16:01:30.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.851+0000 7f2d5259c640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540b5250 0x7f2d541f7b90 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400b440 tx=0x7f2d4400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.852+0000 7f2d42ffd640 1 -- 172.21.15.81:0/509908419 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44015020 con 0x7f2d540b5250 2024-06-29T16:01:30.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.852+0000 7f2d42ffd640 1 -- 172.21.15.81:0/509908419 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d440119a0 con 0x7f2d540b5250 2024-06-29T16:01:30.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.852+0000 7f2d5889e640 1 -- 172.21.15.81:0/509908419 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d540b3010 con 0x7f2d540b5250 2024-06-29T16:01:30.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.852+0000 7f2d42ffd640 1 -- 172.21.15.81:0/509908419 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44011df0 con 0x7f2d540b5250 2024-06-29T16:01:30.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.852+0000 7f2d5889e640 1 -- 172.21.15.81:0/509908419 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d540b35d0 con 0x7f2d540b5250 2024-06-29T16:01:30.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.853+0000 7f2d5889e640 1 -- 172.21.15.81:0/509908419 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d540b39b0 con 0x7f2d540b5250 2024-06-29T16:01:30.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.854+0000 7f2d42ffd640 1 -- 172.21.15.81:0/509908419 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f2d44018020 con 0x7f2d540b5250 2024-06-29T16:01:30.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.854+0000 7f2d42ffd640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f2d34076420 0x7f2d34078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:30.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.854+0000 7f2d42ffd640 1 -- 172.21.15.81:0/509908419 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f2d44097a30 con 0x7f2d540b5250 2024-06-29T16:01:30.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.854+0000 7f2d51d9b640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f2d34076420 0x7f2d34078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:30.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.855+0000 7f2d51d9b640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f2d34076420 0x7f2d34078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2d48033000 tx=0x7f2d48004730 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:30.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:30.859+0000 7f2d42ffd640 1 -- 172.21.15.81:0/509908419 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f2d540b39b0 con 0x7f2d540b5250 2024-06-29T16:01:31.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.034+0000 7f2d5889e640 1 -- 172.21.15.81:0/509908419 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2d541f4b90 con 0x7f2d34076420 2024-06-29T16:01:31.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.035+0000 7f2d42ffd640 1 -- 172.21.15.81:0/509908419 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13472 (secure 0 0 0) 0x7f2d541f4b90 con 0x7f2d34076420 2024-06-29T16:01:31.036 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:31.036 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T16:01:31.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 -- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f2d34076420 msgr2=0x7f2d34078930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f2d34076420 0x7f2d34078930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2d48033000 tx=0x7f2d48004730 comp rx=0 tx=0).stop 2024-06-29T16:01:31.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 -- 172.21.15.81:0/509908419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540b5250 msgr2=0x7f2d541f7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:31.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540b5250 0x7f2d541f7b90 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400b440 tx=0x7f2d4400b900 comp rx=0 tx=0).stop 2024-06-29T16:01:31.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 -- 172.21.15.81:0/509908419 shutdown_connections 2024-06-29T16:01:31.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f2d34076420 0x7f2d34078930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f2d540b41a0 0x7f2d540a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f2d540b9040 0x7f2d540b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 --2- 172.21.15.81:0/509908419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d540b5250 0x7f2d541f7b90 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:31.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 -- 172.21.15.81:0/509908419 >> 172.21.15.81:0/509908419 conn(0x7f2d5409cf30 msgr2=0x7f2d54160dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:31.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 -- 172.21.15.81:0/509908419 shutdown_connections 2024-06-29T16:01:31.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:31.037+0000 7f2d5889e640 1 -- 172.21.15.81:0/509908419 wait complete. 2024-06-29T16:01:31.048 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-06-29T16:01:29.233606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":449936,"kb_used_data":28392,"kb_used_omap":12,"kb_used_meta":421427,"kb_avail":754524784,"statfs":{"total":773094113280,"available":772633378816,"internally_reserved":0,"allocated":29073408,"data_stored":14982928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":431541810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-806093,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-466,"num_read_kb":-293,"num_write":-435,"num_write_kb":-3002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001572"},"pg_stats":[{"pgid":"1.0","version":"272'275","reported_seq":1359,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T16:01:23.336614+0000","last_change":"2024-06-29T14:22:33.004875+0000","last_active":"2024-06-29T16:01:23.336614+0000","last_peered":"2024-06-29T16:01:23.336614+0000","last_clean":"2024-06-29T16:01:23.336614+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T16:01:23.336614+0000","last_undegraded":"2024-06-29T16:01:23.336614+0000","last_fullsized":"2024-06-29T16:01:23.336614+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-29T14:22:31.546664+0000","last_clean_scrub_stamp":"2024-06-29T14:22:31.546664+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-30T22:16:41.555811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576031,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53212,"kb_used_data":3984,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318628,"statfs":{"total":96636764160,"available":96582275072,"internally_reserved":0,"allocated":4079616,"data_stored":2315642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834576032,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48472,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834576030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84504,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287336,"statfs":{"total":96636764160,"available":96550232064,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834576031,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52632,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319208,"statfs":{"total":96636764160,"available":96582868992,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576031,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52636,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319204,"statfs":{"total":96636764160,"available":96582864896,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834576030,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48472,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834576031,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61532,"kb_used_data":3984,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310308,"statfs":{"total":96636764160,"available":96573755392,"internally_reserved":0,"allocated":4079616,"data_stored":2315642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T16:01:31.051 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-06-29T14:22:31.546664+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=29, tm_hour=14, tm_min=22, tm_sec=31, tm_wday=5, tm_yday=181, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=29, tm_hour=16, tm_min=1, tm_sec=25, tm_wday=5, tm_yday=181, tm_isdst=0) 2024-06-29T16:01:31.051 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-06-29T16:01:34.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:34.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:51.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-29T16:01:51.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.179+0000 7fbcf7c81640 1 -- 172.21.15.81:0/1229381350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf00cccf0 msgr2=0x7fbcf00d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.179+0000 7fbcf7c81640 1 --2- 172.21.15.81:0/1229381350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf00cccf0 0x7fbcf00d0230 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009830 tx=0x7fbce402a510 comp rx=0 tx=0).stop 2024-06-29T16:01:51.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.179+0000 7fbcf7c81640 1 -- 172.21.15.81:0/1229381350 shutdown_connections 2024-06-29T16:01:51.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.179+0000 7fbcf7c81640 1 --2- 172.21.15.81:0/1229381350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf00cccf0 0x7fbcf00d0230 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.179+0000 7fbcf7c81640 1 --2- 172.21.15.81:0/1229381350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf00b3200 0x7fbcf00cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.180+0000 7fbcf7c81640 1 -- 172.21.15.81:0/1229381350 >> 172.21.15.81:0/1229381350 conn(0x7fbcf00b3680 msgr2=0x7fbcf00b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:51.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.180+0000 7fbcf7c81640 1 -- 172.21.15.81:0/1229381350 shutdown_connections 2024-06-29T16:01:51.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.180+0000 7fbcf7c81640 1 -- 172.21.15.81:0/1229381350 wait complete. 2024-06-29T16:01:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.181+0000 7fbcf7c81640 1 Processor -- start 2024-06-29T16:01:51.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.181+0000 7fbcf7c81640 1 -- start start 2024-06-29T16:01:51.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf7c81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf00b3200 0x7fbcf006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf7c81640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf00cccf0 0x7fbcf01f5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf7c81640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf00d0770 0x7fbcf009ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf7c81640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf00cedb0 con 0x7fbcf00b3200 2024-06-29T16:01:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf7c81640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf00cec30 con 0x7fbcf00cccf0 2024-06-29T16:01:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf7c81640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf00cef30 con 0x7fbcf00d0770 2024-06-29T16:01:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf5a1d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf00b3200 0x7fbcf006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf5a1d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf00b3200 0x7fbcf006e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42302/0 (socket says 172.21.15.81:42302) 2024-06-29T16:01:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf5a1d640 1 -- 172.21.15.81:0/3722094003 learned_addr learned my addr 172.21.15.81:0/3722094003 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf621e640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf00d0770 0x7fbcf009ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf521c640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf00cccf0 0x7fbcf01f5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf5a1d640 1 -- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf00d0770 msgr2=0x7fbcf009ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf5a1d640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf00d0770 0x7fbcf009ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf5a1d640 1 -- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf00cccf0 msgr2=0x7fbcf01f5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf5a1d640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf00cccf0 0x7fbcf01f5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf5a1d640 1 -- 172.21.15.81:0/3722094003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf006ec00 con 0x7fbcf00b3200 2024-06-29T16:01:51.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf5a1d640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf00b3200 0x7fbcf006e090 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7fbce000d770 tx=0x7fbce000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:51.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcee7fc640 1 -- 172.21.15.81:0/3722094003 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce0017070 con 0x7fbcf00b3200 2024-06-29T16:01:51.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcee7fc640 1 -- 172.21.15.81:0/3722094003 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbce0009d20 con 0x7fbcf00b3200 2024-06-29T16:01:51.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcf7c81640 1 -- 172.21.15.81:0/3722094003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf00c9ae0 con 0x7fbcf00b3200 2024-06-29T16:01:51.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.183+0000 7fbcee7fc640 1 -- 172.21.15.81:0/3722094003 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce00106c0 con 0x7fbcf00b3200 2024-06-29T16:01:51.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.184+0000 7fbcf7c81640 1 -- 172.21.15.81:0/3722094003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf00ca0a0 con 0x7fbcf00b3200 2024-06-29T16:01:51.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.185+0000 7fbcf7c81640 1 -- 172.21.15.81:0/3722094003 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcf00cc7b0 con 0x7fbcf00b3200 2024-06-29T16:01:51.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.185+0000 7fbcee7fc640 1 -- 172.21.15.81:0/3722094003 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7fbce001a070 con 0x7fbcf00b3200 2024-06-29T16:01:51.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.186+0000 7fbcee7fc640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbcdc0763f0 0x7fbcdc078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.186+0000 7fbcee7fc640 1 -- 172.21.15.81:0/3722094003 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7fbce0097c40 con 0x7fbcf00b3200 2024-06-29T16:01:51.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.186+0000 7fbcf521c640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbcdc0763f0 0x7fbcdc078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.187+0000 7fbcf521c640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbcdc0763f0 0x7fbcdc078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbce4034000 tx=0x7fbce4004730 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:51.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.190+0000 7fbcee7fc640 1 -- 172.21.15.81:0/3722094003 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7fbce0061220 con 0x7fbcf00b3200 2024-06-29T16:01:51.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.369+0000 7fbcf7c81640 1 -- 172.21.15.81:0/3722094003 --> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbcf015b150 con 0x7fbcdc0763f0 2024-06-29T16:01:51.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.370+0000 7fbcee7fc640 1 -- 172.21.15.81:0/3722094003 <== mgr.11941 v2:172.21.15.167:6832/3637377664 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13392 (secure 0 0 0) 0x7fbcf015b150 con 0x7fbcdc0763f0 2024-06-29T16:01:51.372 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:51.372 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-06-29T16:01:51.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 -- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbcdc0763f0 msgr2=0x7fbcdc078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbcdc0763f0 0x7fbcdc078900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbce4034000 tx=0x7fbce4004730 comp rx=0 tx=0).stop 2024-06-29T16:01:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 -- 172.21.15.81:0/3722094003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf00b3200 msgr2=0x7fbcf006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf00b3200 0x7fbcf006e090 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7fbce000d770 tx=0x7fbce000dc30 comp rx=0 tx=0).stop 2024-06-29T16:01:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 -- 172.21.15.81:0/3722094003 shutdown_connections 2024-06-29T16:01:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7fbcf00d0770 0x7fbcf009ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7fbcdc0763f0 0x7fbcdc078900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7fbcf00cccf0 0x7fbcf01f5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 --2- 172.21.15.81:0/3722094003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf00b3200 0x7fbcf006e090 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 -- 172.21.15.81:0/3722094003 >> 172.21.15.81:0/3722094003 conn(0x7fbcf00b3680 msgr2=0x7fbcf015b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:51.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 -- 172.21.15.81:0/3722094003 shutdown_connections 2024-06-29T16:01:51.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.373+0000 7fbcf7c81640 1 -- 172.21.15.81:0/3722094003 wait complete. 2024-06-29T16:01:51.385 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-06-29T16:01:51.238296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":449928,"kb_used_data":28392,"kb_used_omap":12,"kb_used_meta":421427,"kb_avail":754524792,"statfs":{"total":773094113280,"available":772633387008,"internally_reserved":0,"allocated":29073408,"data_stored":14982928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":431541810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002652"},"pg_stats":[{"pgid":"1.0","version":"275'276","reported_seq":1369,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-29T16:01:30.959171+0000","last_change":"2024-06-29T16:01:30.959211+0000","last_active":"2024-06-29T16:01:30.959171+0000","last_peered":"2024-06-29T16:01:30.959171+0000","last_clean":"2024-06-29T16:01:30.959171+0000","last_became_active":"2024-06-29T14:22:33.003666+0000","last_became_peered":"2024-06-29T14:22:33.003666+0000","last_unstale":"2024-06-29T16:01:30.959171+0000","last_undegraded":"2024-06-29T16:01:30.959171+0000","last_fullsized":"2024-06-29T16:01:30.959171+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"275'276","last_scrub_stamp":"2024-06-29T16:01:30.959072+0000","last_deep_scrub":"275'276","last_deep_scrub_stamp":"2024-06-29T16:01:30.959072+0000","last_clean_scrub_stamp":"2024-06-29T16:01:30.959072+0000","objects_scrubbed":2,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-07-01T03:06:36.890054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1806,"num_read_kb":5159,"num_write":449,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53208,"kb_used_data":3984,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318632,"statfs":{"total":96636764160,"available":96582279168,"internally_reserved":0,"allocated":4079616,"data_stored":2315642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48472,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84504,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287336,"statfs":{"total":96636764160,"available":96550232064,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52632,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319208,"statfs":{"total":96636764160,"available":96582868992,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52636,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319204,"statfs":{"total":96636764160,"available":96582864896,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48472,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":3485696,"data_stored":1725274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834576035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61528,"kb_used_data":3984,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94310312,"statfs":{"total":96636764160,"available":96573759488,"internally_reserved":0,"allocated":4079616,"data_stored":2315642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-29T16:01:51.386 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-06-29T16:01:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.497+0000 7f7360512640 1 -- 172.21.15.81:0/1859776350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73580b4cf0 msgr2=0x7f73580d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.497+0000 7f7360512640 1 --2- 172.21.15.81:0/1859776350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73580b4cf0 0x7f73580d00f0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f7350005fd0 tx=0x7f735002aa90 comp rx=0 tx=0).stop 2024-06-29T16:01:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.497+0000 7f7360512640 1 -- 172.21.15.81:0/1859776350 shutdown_connections 2024-06-29T16:01:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.497+0000 7f7360512640 1 --2- 172.21.15.81:0/1859776350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73580b4cf0 0x7f73580d00f0 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.497+0000 7f7360512640 1 --2- 172.21.15.81:0/1859776350 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73580cd540 0x7f73580b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.497+0000 7f7360512640 1 -- 172.21.15.81:0/1859776350 >> 172.21.15.81:0/1859776350 conn(0x7f73580d1960 msgr2=0x7f73580d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:51.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.497+0000 7f7360512640 1 -- 172.21.15.81:0/1859776350 shutdown_connections 2024-06-29T16:01:51.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.497+0000 7f7360512640 1 -- 172.21.15.81:0/1859776350 wait complete. 2024-06-29T16:01:51.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.499+0000 7f7360512640 1 Processor -- start 2024-06-29T16:01:51.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.499+0000 7f7360512640 1 -- start start 2024-06-29T16:01:51.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.500+0000 7f7360512640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73580b4cf0 0x7f735806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.500+0000 7f7360512640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73580b6d60 0x7f7358158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.500+0000 7f7360512640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73580cd540 0x7f7358164e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.500+0000 7f7360512640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f73580d07b0 con 0x7f73580b4cf0 2024-06-29T16:01:51.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.500+0000 7f7360512640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f73580d0630 con 0x7f73580cd540 2024-06-29T16:01:51.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.500+0000 7f7360512640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f73580d0930 con 0x7f73580b6d60 2024-06-29T16:01:51.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.500+0000 7f735e2ae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73580b4cf0 0x7f735806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.500+0000 7f735e2ae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73580b4cf0 0x7f735806efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42310/0 (socket says 172.21.15.81:42310) 2024-06-29T16:01:51.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.500+0000 7f735e2ae640 1 -- 172.21.15.81:0/1232609175 learned_addr learned my addr 172.21.15.81:0/1232609175 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:51.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f735daad640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73580b6d60 0x7f7358158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f735eaaf640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73580cd540 0x7f7358164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f735e2ae640 1 -- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73580b6d60 msgr2=0x7f7358158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f735e2ae640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73580b6d60 0x7f7358158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f735e2ae640 1 -- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73580cd540 msgr2=0x7f7358164e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f735e2ae640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73580cd540 0x7f7358164e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f735e2ae640 1 -- 172.21.15.81:0/1232609175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735815c680 con 0x7f73580b4cf0 2024-06-29T16:01:51.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f735e2ae640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73580b4cf0 0x7f735806efe0 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f735400c390 tx=0x7f735400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:51.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f734effd640 1 -- 172.21.15.81:0/1232609175 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354017070 con 0x7f73580b4cf0 2024-06-29T16:01:51.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f734effd640 1 -- 172.21.15.81:0/1232609175 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f73540035f0 con 0x7f73580b4cf0 2024-06-29T16:01:51.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f7360512640 1 -- 172.21.15.81:0/1232609175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7358166370 con 0x7f73580b4cf0 2024-06-29T16:01:51.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f734effd640 1 -- 172.21.15.81:0/1232609175 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354003a20 con 0x7f73580b4cf0 2024-06-29T16:01:51.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.501+0000 7f7360512640 1 -- 172.21.15.81:0/1232609175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7358165db0 con 0x7f73580b4cf0 2024-06-29T16:01:51.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.502+0000 7f7360512640 1 -- 172.21.15.81:0/1232609175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73580d00f0 con 0x7f73580b4cf0 2024-06-29T16:01:51.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.503+0000 7f734effd640 1 -- 172.21.15.81:0/1232609175 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f735401a070 con 0x7f73580b4cf0 2024-06-29T16:01:51.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.503+0000 7f734effd640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f733c0763f0 0x7f733c078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.504+0000 7f734effd640 1 -- 172.21.15.81:0/1232609175 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f7354096de0 con 0x7f73580b4cf0 2024-06-29T16:01:51.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.504+0000 7f735daad640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f733c0763f0 0x7f733c078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.504+0000 7f735daad640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f733c0763f0 0x7f733c078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7348005fd0 tx=0x7f7348005850 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:51.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.508+0000 7f734effd640 1 -- 172.21.15.81:0/1232609175 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f7354060450 con 0x7f73580b4cf0 2024-06-29T16:01:51.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.688+0000 7f7360512640 1 -- 172.21.15.81:0/1232609175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f73580cdec0 con 0x7f73580b4cf0 2024-06-29T16:01:51.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.691+0000 7f734effd640 1 -- 172.21.15.81:0/1232609175 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v88)=0 v88) ==== 123+0+0 (secure 0 0 0) 0x7f735405fd50 con 0x7f73580b4cf0 2024-06-29T16:01:51.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.692+0000 7f734effd640 1 -- 172.21.15.81:0/1232609175 <== mon.0 v2:172.21.15.81:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f735405f860 con 0x7f73580b4cf0 2024-06-29T16:01:51.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 -- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f733c0763f0 msgr2=0x7f733c078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f733c0763f0 0x7f733c078900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7348005fd0 tx=0x7f7348005850 comp rx=0 tx=0).stop 2024-06-29T16:01:51.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 -- 172.21.15.81:0/1232609175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73580b4cf0 msgr2=0x7f735806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73580b4cf0 0x7f735806efe0 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f735400c390 tx=0x7f735400c850 comp rx=0 tx=0).stop 2024-06-29T16:01:51.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 -- 172.21.15.81:0/1232609175 shutdown_connections 2024-06-29T16:01:51.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f733c0763f0 0x7f733c078900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f73580cd540 0x7f7358164e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f73580b6d60 0x7f7358158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 --2- 172.21.15.81:0/1232609175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73580b4cf0 0x7f735806efe0 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 -- 172.21.15.81:0/1232609175 >> 172.21.15.81:0/1232609175 conn(0x7f73580d1960 msgr2=0x7f735815cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:51.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 -- 172.21.15.81:0/1232609175 shutdown_connections 2024-06-29T16:01:51.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.694+0000 7f7360512640 1 -- 172.21.15.81:0/1232609175 wait complete. 2024-06-29T16:01:51.707 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-29T16:01:51.707 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-29T16:01:51.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.819+0000 7f65fbf54640 1 -- 172.21.15.81:0/3213578735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65f40d0b90 msgr2=0x7f65f40d1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.819+0000 7f65fbf54640 1 --2- 172.21.15.81:0/3213578735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65f40d0b90 0x7f65f40d1a80 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009830 tx=0x7f65e402a5a0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.819+0000 7f65fbf54640 1 -- 172.21.15.81:0/3213578735 shutdown_connections 2024-06-29T16:01:51.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.819+0000 7f65fbf54640 1 --2- 172.21.15.81:0/3213578735 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65f40b5250 0x7f65f40d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.819+0000 7f65fbf54640 1 --2- 172.21.15.81:0/3213578735 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65f40d0b90 0x7f65f40d1a80 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.819+0000 7f65fbf54640 1 -- 172.21.15.81:0/3213578735 >> 172.21.15.81:0/3213578735 conn(0x7f65f409cf30 msgr2=0x7f65f40d3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:51.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.819+0000 7f65fbf54640 1 -- 172.21.15.81:0/3213578735 shutdown_connections 2024-06-29T16:01:51.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.819+0000 7f65fbf54640 1 -- 172.21.15.81:0/3213578735 wait complete. 2024-06-29T16:01:51.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.821+0000 7f65fbf54640 1 Processor -- start 2024-06-29T16:01:51.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.821+0000 7f65fbf54640 1 -- start start 2024-06-29T16:01:51.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65fbf54640 1 --2- >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65f40b5250 0x7f65f40a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65fbf54640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65f40b9040 0x7f65f40a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65fbf54640 1 --2- >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65f40b1230 0x7f65f4154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65fbf54640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f65f40d2680 con 0x7f65f40b9040 2024-06-29T16:01:51.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65fbf54640 1 -- --> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] -- mon_getmap magic: 0 -- 0x7f65f40d2500 con 0x7f65f40b1230 2024-06-29T16:01:51.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65fbf54640 1 -- --> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] -- mon_getmap magic: 0 -- 0x7f65f40d2800 con 0x7f65f40b5250 2024-06-29T16:01:51.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65f94ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65f40b9040 0x7f65f40a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65f94ef640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65f40b9040 0x7f65f40a03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42322/0 (socket says 172.21.15.81:42322) 2024-06-29T16:01:51.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65f94ef640 1 -- 172.21.15.81:0/840779446 learned_addr learned my addr 172.21.15.81:0/840779446 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-06-29T16:01:51.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65fa4f1640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65f40b1230 0x7f65f4154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.822+0000 7f65f9cf0640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65f40b5250 0x7f65f40a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65f94ef640 1 -- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65f40b5250 msgr2=0x7f65f40a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65f94ef640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65f40b5250 0x7f65f40a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65f94ef640 1 -- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65f40b1230 msgr2=0x7f65f4154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:51.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65f94ef640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65f40b1230 0x7f65f4154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:51.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65f94ef640 1 -- 172.21.15.81:0/840779446 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f40b1960 con 0x7f65f40b9040 2024-06-29T16:01:51.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65f94ef640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65f40b9040 0x7f65f40a03a0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009950 tx=0x7f65e40052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:51.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65ea7fc640 1 -- 172.21.15.81:0/840779446 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4031c60 con 0x7f65f40b9040 2024-06-29T16:01:51.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65ea7fc640 1 -- 172.21.15.81:0/840779446 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f65e4004730 con 0x7f65f40b9040 2024-06-29T16:01:51.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65fbf54640 1 -- 172.21.15.81:0/840779446 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f40b1c40 con 0x7f65f40b9040 2024-06-29T16:01:51.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65ea7fc640 1 -- 172.21.15.81:0/840779446 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4004b80 con 0x7f65f40b9040 2024-06-29T16:01:51.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.823+0000 7f65fbf54640 1 -- 172.21.15.81:0/840779446 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f409dad0 con 0x7f65f40b9040 2024-06-29T16:01:51.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.824+0000 7f65fbf54640 1 -- 172.21.15.81:0/840779446 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f40d1a80 con 0x7f65f40b9040 2024-06-29T16:01:51.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.825+0000 7f65ea7fc640 1 -- 172.21.15.81:0/840779446 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 166) ==== 98303+0+0 (secure 0 0 0) 0x7f65e4031770 con 0x7f65f40b9040 2024-06-29T16:01:51.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.825+0000 7f65ea7fc640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f65d80763f0 0x7f65d8078900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-29T16:01:51.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.825+0000 7f65ea7fc640 1 -- 172.21.15.81:0/840779446 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5916+0+0 (secure 0 0 0) 0x7f65e40b8a50 con 0x7f65f40b9040 2024-06-29T16:01:51.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.825+0000 7f65f9cf0640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f65d80763f0 0x7f65d8078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-29T16:01:51.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.826+0000 7f65f9cf0640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f65d80763f0 0x7f65d8078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f65f0001720 tx=0x7f65f0009040 comp rx=0 tx=0).ready entity=mgr.11941 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-29T16:01:51.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:51.830+0000 7f65ea7fc640 1 -- 172.21.15.81:0/840779446 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191195 (secure 0 0 0) 0x7f65e404a030 con 0x7f65f40b9040 2024-06-29T16:01:52.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.044+0000 7f65fbf54640 1 -- 172.21.15.81:0/840779446 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65f40b16f0 con 0x7f65f40b9040 2024-06-29T16:01:52.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.045+0000 7f65ea7fc640 1 -- 172.21.15.81:0/840779446 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+3923 (secure 0 0 0) 0x7f65e403f030 con 0x7f65f40b9040 2024-06-29T16:01:52.047 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-06-29T16:01:52.047 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":156,"btime":"2024-06-29T16:01:20:468642+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10196,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6837/2932449820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6835","nonce":2932449820},{"type":"v1","addr":"172.21.15.167:6837","nonce":2932449820}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/317823138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6833","nonce":317823138},{"type":"v1","addr":"172.21.15.81:6835","nonce":317823138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":11728,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.167:6836/3582280449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6834","nonce":3582280449},{"type":"v1","addr":"172.21.15.167:6836","nonce":3582280449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":13827,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6834/2698260500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":2698260500},{"type":"v1","addr":"172.21.15.81:6834","nonce":2698260500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[]} 2024-06-29T16:01:52.047 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 156 2024-06-29T16:01:52.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 -- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f65d80763f0 msgr2=0x7f65d8078900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:52.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f65d80763f0 0x7f65d8078900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f65f0001720 tx=0x7f65f0009040 comp rx=0 tx=0).stop 2024-06-29T16:01:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 -- 172.21.15.81:0/840779446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65f40b9040 msgr2=0x7f65f40a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-29T16:01:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65f40b9040 0x7f65f40a03a0 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009950 tx=0x7f65e40052f0 comp rx=0 tx=0).stop 2024-06-29T16:01:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 -- 172.21.15.81:0/840779446 shutdown_connections 2024-06-29T16:01:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:6832/3637377664,v1:172.21.15.167:6833/3637377664] conn(0x7f65d80763f0 0x7f65d8078900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:3300/0,v1:172.21.15.167:6789/0] conn(0x7f65f40b1230 0x7f65f4154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65f40b9040 0x7f65f40a03a0 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 --2- 172.21.15.81:0/840779446 >> [v2:172.21.15.167:3301/0,v1:172.21.15.167:6790/0] conn(0x7f65f40b5250 0x7f65f40a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-29T16:01:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 -- 172.21.15.81:0/840779446 >> 172.21.15.81:0/840779446 conn(0x7f65f409cf30 msgr2=0x7f65f406d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-29T16:01:52.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 -- 172.21.15.81:0/840779446 shutdown_connections 2024-06-29T16:01:52.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-06-29T16:01:52.047+0000 7f65fbf54640 1 -- 172.21.15.81:0/840779446 wait complete. 2024-06-29T16:01:52.059 DEBUG:tasks.cephfs.filesystem:already dead... 2024-06-29T16:01:52.059 INFO:teuthology.misc:Shutting down mds daemons... 2024-06-29T16:01:52.060 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-29T16:01:52.060 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.071 INFO:tasks.ceph.mds.a:Stopped 2024-06-29T16:01:52.072 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-29T16:01:52.072 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.121 INFO:tasks.ceph.mds.c:Stopped 2024-06-29T16:01:52.122 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-29T16:01:52.122 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.134 INFO:tasks.ceph.mds.b:Stopped 2024-06-29T16:01:52.134 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-29T16:01:52.134 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.187 INFO:tasks.ceph.mds.d:Stopped 2024-06-29T16:01:52.187 INFO:teuthology.misc:Shutting down osd daemons... 2024-06-29T16:01:52.187 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-06-29T16:01:52.187 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.216 INFO:tasks.ceph.osd.0:Stopped 2024-06-29T16:01:52.217 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-06-29T16:01:52.217 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.288 INFO:tasks.ceph.osd.1:Stopped 2024-06-29T16:01:52.288 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-06-29T16:01:52.288 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.359 INFO:tasks.ceph.osd.2:Stopped 2024-06-29T16:01:52.359 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-06-29T16:01:52.359 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.428 INFO:tasks.ceph.osd.3:Stopped 2024-06-29T16:01:52.428 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-06-29T16:01:52.428 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.474 INFO:tasks.ceph.osd.4:Stopped 2024-06-29T16:01:52.474 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-06-29T16:01:52.474 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.514 INFO:tasks.ceph.osd.5:Stopped 2024-06-29T16:01:52.514 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-06-29T16:01:52.514 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.595 INFO:tasks.ceph.osd.6:Stopped 2024-06-29T16:01:52.595 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-06-29T16:01:52.595 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.669 INFO:tasks.ceph.osd.7:Stopped 2024-06-29T16:01:52.670 INFO:teuthology.misc:Shutting down mgr daemons... 2024-06-29T16:01:52.670 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-06-29T16:01:52.671 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.704 INFO:tasks.ceph.mgr.y:Stopped 2024-06-29T16:01:52.705 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-06-29T16:01:52.705 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.748 INFO:tasks.ceph.mgr.x:Stopped 2024-06-29T16:01:52.749 INFO:teuthology.misc:Shutting down mon daemons... 2024-06-29T16:01:52.750 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-06-29T16:01:52.750 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.766 INFO:tasks.ceph.mon.a:Stopped 2024-06-29T16:01:52.766 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-29T16:01:52.766 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.805 INFO:tasks.ceph.mon.b:Stopped 2024-06-29T16:01:52.805 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-29T16:01:52.806 INFO:teuthology.orchestra.run:waiting for 300 2024-06-29T16:01:52.861 INFO:tasks.ceph.mon.c:Stopped 2024-06-29T16:01:52.861 INFO:tasks.ceph:Checking cluster log for badness... 2024-06-29T16:01:52.861 DEBUG:teuthology.orchestra.run.smithi081:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-06-29T16:01:52.901 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T16:01:52.902 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-06-29T16:01:53.124 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T16:01:53.124 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-06-29T16:01:53.221 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T16:01:53.222 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-06-29T16:01:53.278 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T16:01:53.278 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-06-29T16:01:53.336 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T16:01:53.337 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-06-29T16:01:53.506 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T16:01:53.507 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-06-29T16:01:53.585 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T16:01:53.585 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-06-29T16:01:53.634 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T16:01:53.635 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-06-29T16:01:53.684 INFO:tasks.ceph:Archiving mon data... 2024-06-29T16:01:53.685 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926/data/mon.a.tgz 2024-06-29T16:01:53.685 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-06-29T16:01:53.701 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.cXrtWjsiCJ 2024-06-29T16:01:53.701 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.cXrtWjsiCJ 2024-06-29T16:01:54.429 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.cXrtWjsiCJ 2024-06-29T16:01:54.512 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.cXrtWjsiCJ is 4MB 2024-06-29T16:01:54.685 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.cXrtWjsiCJ 2024-06-29T16:01:54.701 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926/data/mon.b.tgz 2024-06-29T16:01:54.702 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-06-29T16:01:54.720 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.Cfr7FZLiVj 2024-06-29T16:01:54.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Cfr7FZLiVj 2024-06-29T16:01:55.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.Cfr7FZLiVj 2024-06-29T16:01:55.579 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.Cfr7FZLiVj is 4MB 2024-06-29T16:01:55.706 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.Cfr7FZLiVj 2024-06-29T16:01:55.722 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926/data/mon.c.tgz 2024-06-29T16:01:55.722 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-06-29T16:01:55.777 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.fz1xlUzDXE 2024-06-29T16:01:55.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.fz1xlUzDXE 2024-06-29T16:01:56.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.fz1xlUzDXE 2024-06-29T16:01:56.545 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.fz1xlUzDXE is 4MB 2024-06-29T16:01:56.709 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.fz1xlUzDXE 2024-06-29T16:01:56.724 INFO:tasks.ceph:Cleaning ceph cluster... 2024-06-29T16:01:56.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-29T16:01:56.727 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-29T16:01:56.812 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-06-29T16:01:56.845 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-06-29T16:01:56.845 INFO:tasks.ceph:Archiving crash dumps... 2024-06-29T16:01:56.847 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926/remote/smithi081/crash 2024-06-29T16:01:56.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-29T16:01:56.883 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926/remote/smithi167/crash 2024-06-29T16:01:56.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-29T16:01:56.909 INFO:tasks.ceph:Shutting down logrotate 2024-06-29T16:01:56.909 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:56.929 DEBUG:teuthology.orchestra.run.smithi167:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:57.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:57.033 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-06-29T16:01:57.058 INFO:tasks.ceph:Compressing logs... 2024-06-29T16:01:57.059 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-29T16:01:57.077 DEBUG:teuthology.orchestra.run.smithi167:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-29T16:01:57.101 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44983.log 2024-06-29T16:01:57.101 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101604.log 2024-06-29T16:01:57.132 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38033.log 2024-06-29T16:01:57.132 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-06-29T16:01:57.132 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38073.log 2024-06-29T16:01:57.132 INFO:teuthology.orchestra.run.smithi167.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38033.log.gz 2024-06-29T16:01:57.132 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-06-29T16:01:57.133 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.38073.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-06-29T16:01:57.133 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-06-29T16:01:57.133 INFO:teuthology.orchestra.run.smithi167.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38073.log.gz 2024-06-29T16:01:57.133 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-06-29T16:01:57.133 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43964.log 2024-06-29T16:01:57.133 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37793.log 2024-06-29T16:01:57.133 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.1.43964.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.1.43964.log.gz 2024-06-29T16:01:57.133 INFO:teuthology.orchestra.run.smithi167.stderr: --verbose -- /var/log/ceph/ceph-osd.4.log 2024-06-29T16:01:57.133 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.37793.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-06-29T16:01:57.134 INFO:teuthology.orchestra.run.smithi167.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37793.log.gz 2024-06-29T16:01:57.134 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-06-29T16:01:57.134 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-06-29T16:01:57.136 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mon.c.log: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-06-29T16:01:57.153 INFO:teuthology.orchestra.run.smithi167.stderr: 94.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-06-29T16:01:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37913.log 2024-06-29T16:01:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37953.log 2024-06-29T16:01:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.37913.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37913.log.gz 2024-06-29T16:01:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37873.log 2024-06-29T16:01:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.37953.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37953.log.gz 2024-06-29T16:01:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44362.log 2024-06-29T16:01:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.37873.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37873.log.gz 2024-06-29T16:01:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37993.log 2024-06-29T16:01:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.3.44362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.44362.log.gz 2024-06-29T16:01:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-06-29T16:01:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-client.admin.37993.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.37993.log.gz 2024-06-29T16:01:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-06-29T16:01:57.164 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37833.log 2024-06-29T16:01:57.181 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.37833.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37833.log.gz 2024-06-29T16:01:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85124.log 2024-06-29T16:01:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111183.log 2024-06-29T16:01:57.326 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44983.log.gz 2024-06-29T16:01:57.326 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.101604.log.gz 2024-06-29T16:01:57.326 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.93734.log 2024-06-29T16:01:57.327 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85124.log.gz 2024-06-29T16:01:57.327 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107881.log 2024-06-29T16:01:57.327 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.93734.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.111183.log.gz 2024-06-29T16:01:57.327 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48987.log 2024-06-29T16:01:57.327 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.93734.log.gz 2024-06-29T16:01:57.327 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.107881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73020.log 2024-06-29T16:01:57.327 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.107881.log.gz 2024-06-29T16:01:57.327 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78913.log 2024-06-29T16:01:57.328 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48987.log.gz 2024-06-29T16:01:57.328 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.73020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78209.log 2024-06-29T16:01:57.328 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73020.log.gz 2024-06-29T16:01:57.328 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98119.log 2024-06-29T16:01:57.328 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78913.log.gz 2024-06-29T16:01:57.328 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90865.log 2024-06-29T16:01:57.328 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78209.log.gz 2024-06-29T16:01:57.329 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71862.log 2024-06-29T16:01:57.329 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98119.log.gz 2024-06-29T16:01:57.329 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.90865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51075.log 2024-06-29T16:01:57.329 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.90865.log.gz 2024-06-29T16:01:57.329 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112633.log 2024-06-29T16:01:57.329 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71862.log.gz 2024-06-29T16:01:57.329 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106797.log 2024-06-29T16:01:57.330 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51075.log.gz 2024-06-29T16:01:57.330 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58784.log 2024-06-29T16:01:57.330 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112633.log.gz 2024-06-29T16:01:57.330 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70659.log 2024-06-29T16:01:57.330 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106797.log.gz 2024-06-29T16:01:57.330 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64775.log 2024-06-29T16:01:57.331 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58784.log.gz 2024-06-29T16:01:57.331 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63892.log 2024-06-29T16:01:57.331 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70659.log.gz 2024-06-29T16:01:57.331 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109481.log 2024-06-29T16:01:57.331 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64775.log.gz 2024-06-29T16:01:57.331 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69661.log 2024-06-29T16:01:57.331 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63892.log.gz 2024-06-29T16:01:57.331 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58953.log 2024-06-29T16:01:57.332 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.109481.log.gz 2024-06-29T16:01:57.332 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57480.log 2024-06-29T16:01:57.332 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69661.log.gz 2024-06-29T16:01:57.332 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102285.log 2024-06-29T16:01:57.332 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.58953.log.gz 2024-06-29T16:01:57.332 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96293.log 2024-06-29T16:01:57.333 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57480.log.gz 2024-06-29T16:01:57.333 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89113.log 2024-06-29T16:01:57.333 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102285.log.gz 2024-06-29T16:01:57.333 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66643.log 2024-06-29T16:01:57.333 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.96293.log.gz 2024-06-29T16:01:57.333 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96503.log 2024-06-29T16:01:57.333 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89113.log.gz 2024-06-29T16:01:57.334 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97659.log 2024-06-29T16:01:57.334 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66643.log.gz 2024-06-29T16:01:57.334 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85436.log 2024-06-29T16:01:57.334 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.96503.log.gz 2024-06-29T16:01:57.334 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88221.log 2024-06-29T16:01:57.334 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.97659.log.gz 2024-06-29T16:01:57.334 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91983.log 2024-06-29T16:01:57.335 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85436.log.gz 2024-06-29T16:01:57.335 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45843.log 2024-06-29T16:01:57.335 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88221.log.gz 2024-06-29T16:01:57.335 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102159.log 2024-06-29T16:01:57.335 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91983.log.gz 2024-06-29T16:01:57.335 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70738.log 2024-06-29T16:01:57.335 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.45843.log.gz 2024-06-29T16:01:57.336 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80936.log 2024-06-29T16:01:57.336 INFO:teuthology.orchestra.run.smithi081.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.102159.log.gz 2024-06-29T16:01:57.336 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105365.log 2024-06-29T16:01:57.336 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.70738.log.gz 2024-06-29T16:01:57.336 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98897.log 2024-06-29T16:01:57.336 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80936.log.gz 2024-06-29T16:01:57.336 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45025.log 2024-06-29T16:01:57.337 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105365.log.gz 2024-06-29T16:01:57.337 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91899.log 2024-06-29T16:01:57.337 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.98897.log.gz 2024-06-29T16:01:57.337 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.102833.log 2024-06-29T16:01:57.337 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45025.log.gz 2024-06-29T16:01:57.337 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98077.log 2024-06-29T16:01:57.337 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.91899.log.gz 2024-06-29T16:01:57.337 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.102833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106545.log 2024-06-29T16:01:57.338 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.102833.log.gz 2024-06-29T16:01:57.338 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45109.log 2024-06-29T16:01:57.338 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98077.log.gz 2024-06-29T16:01:57.338 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80411.log 2024-06-29T16:01:57.338 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106545.log.gz 2024-06-29T16:01:57.338 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52403.log 2024-06-29T16:01:57.339 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45109.log.gz 2024-06-29T16:01:57.339 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93856.log 2024-06-29T16:01:57.339 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80411.log.gz 2024-06-29T16:01:57.339 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46537.log 2024-06-29T16:01:57.339 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.52403.log.gz 2024-06-29T16:01:57.339 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50451.log 2024-06-29T16:01:57.339 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.93856.log.gz 2024-06-29T16:01:57.339 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66219.log 2024-06-29T16:01:57.340 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46537.log.gz 2024-06-29T16:01:57.340 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98729.log 2024-06-29T16:01:57.340 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50451.log.gz 2024-06-29T16:01:57.340 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60172.log 2024-06-29T16:01:57.340 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.66219.log.gz 2024-06-29T16:01:57.340 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86841.log 2024-06-29T16:01:57.340 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.98729.log.gz 2024-06-29T16:01:57.341 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.60172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68644.log 2024-06-29T16:01:57.341 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.60172.log.gz 2024-06-29T16:01:57.341 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110375.log 2024-06-29T16:01:57.341 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.86841.log.gz 2024-06-29T16:01:57.341 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103434.log 2024-06-29T16:01:57.341 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68644.log.gz 2024-06-29T16:01:57.342 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46411.log 2024-06-29T16:01:57.342 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110375.log.gz 2024-06-29T16:01:57.342 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100475.log 2024-06-29T16:01:57.342 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103434.log.gz 2024-06-29T16:01:57.342 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84997.log 2024-06-29T16:01:57.342 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46411.log.gz 2024-06-29T16:01:57.342 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81048.log 2024-06-29T16:01:57.343 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100475.log.gz 2024-06-29T16:01:57.343 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68290.log 2024-06-29T16:01:57.343 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84997.log.gz 2024-06-29T16:01:57.343 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.81048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37985.log 2024-06-29T16:01:57.343 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81048.log.gz 2024-06-29T16:01:57.343 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51181.log 2024-06-29T16:01:57.343 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68290.log.gz 2024-06-29T16:01:57.344 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49947.log 2024-06-29T16:01:57.344 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37985.log.gz 2024-06-29T16:01:57.344 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98161.log 2024-06-29T16:01:57.344 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51181.log.gz 2024-06-29T16:01:57.344 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75323.log 2024-06-29T16:01:57.344 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49947.log.gz 2024-06-29T16:01:57.345 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92680.log 2024-06-29T16:01:57.345 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98161.log.gz 2024-06-29T16:01:57.345 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90578.log 2024-06-29T16:01:57.345 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.75323.log.gz 2024-06-29T16:01:57.345 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46747.log 2024-06-29T16:01:57.345 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92680.log.gz 2024-06-29T16:01:57.346 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.90578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91857.log 2024-06-29T16:01:57.346 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90578.log.gz 2024-06-29T16:01:57.346 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38576.log 2024-06-29T16:01:57.346 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46747.log.gz 2024-06-29T16:01:57.346 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84388.log 2024-06-29T16:01:57.346 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.91857.log.gz 2024-06-29T16:01:57.346 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70780.log 2024-06-29T16:01:57.346 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38576.log.gz 2024-06-29T16:01:57.347 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70941.log 2024-06-29T16:01:57.347 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.84388.log.gz 2024-06-29T16:01:57.347 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44689.log 2024-06-29T16:01:57.347 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93964.log 2024-06-29T16:01:57.347 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70780.log.gz 2024-06-29T16:01:57.347 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70941.log: /var/log/ceph/ceph-client.admin.44689.log: gzip 87.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.70941.log.gz -- 2024-06-29T16:01:57.347 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.44605.log 2024-06-29T16:01:57.348 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44689.log.gz 2024-06-29T16:01:57.348 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49401.log 2024-06-29T16:01:57.348 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.93964.log.gz 2024-06-29T16:01:57.348 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49989.log 2024-06-29T16:01:57.348 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44605.log.gz 2024-06-29T16:01:57.348 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107762.log 2024-06-29T16:01:57.348 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49401.log.gz 2024-06-29T16:01:57.348 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80828.log 2024-06-29T16:01:57.349 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49989.log.gz 2024-06-29T16:01:57.349 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.107762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49821.log 2024-06-29T16:01:57.349 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.107762.log.gz 2024-06-29T16:01:57.349 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98771.log 2024-06-29T16:01:57.349 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80828.log.gz 2024-06-29T16:01:57.349 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51656.log 2024-06-29T16:01:57.349 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49821.log.gz 2024-06-29T16:01:57.349 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111571.log 2024-06-29T16:01:57.349 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.98771.log.gz 2024-06-29T16:01:57.350 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44306.log 2024-06-29T16:01:57.350 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57753.log 2024-06-29T16:01:57.350 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.111571.log.gz 2024-06-29T16:01:57.350 INFO:teuthology.orchestra.run.smithi081.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51656.log.gz 2024-06-29T16:01:57.350 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67194.log 2024-06-29T16:01:57.350 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44306.log.gz 2024-06-29T16:01:57.350 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73437.log 2024-06-29T16:01:57.351 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67194.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57753.log.gz 2024-06-29T16:01:57.351 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106372.log 2024-06-29T16:01:57.351 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67194.log.gz 2024-06-29T16:01:57.351 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.73437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49071.log 2024-06-29T16:01:57.351 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73437.log.gz 2024-06-29T16:01:57.351 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58997.log 2024-06-29T16:01:57.351 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2%/var/log/ceph/ceph-client.admin.49071.log: -- replaced with /var/log/ceph/ceph-client.admin.106372.log.gz 2024-06-29T16:01:57.351 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45291.log 2024-06-29T16:01:57.352 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49071.log.gz 2024-06-29T16:01:57.352 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105449.log 2024-06-29T16:01:57.352 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58997.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.58997.log.gz 2024-06-29T16:01:57.352 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80243.log 2024-06-29T16:01:57.352 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.45291.log.gz/var/log/ceph/ceph-client.admin.105449.log: 2024-06-29T16:01:57.352 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103578.log 2024-06-29T16:01:57.352 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105449.log.gz 2024-06-29T16:01:57.352 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62758.log 2024-06-29T16:01:57.353 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80243.log.gz 2024-06-29T16:01:57.353 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79081.log 2024-06-29T16:01:57.353 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.103578.log.gz 2024-06-29T16:01:57.353 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103704.log 2024-06-29T16:01:57.353 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62758.log.gz 2024-06-29T16:01:57.353 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103979.log 2024-06-29T16:01:57.353 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79081.log.gz 2024-06-29T16:01:57.353 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66478.log 2024-06-29T16:01:57.354 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112462.log 2024-06-29T16:01:57.354 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.103979.log.gz 2024-06-29T16:01:57.354 INFO:teuthology.orchestra.run.smithi081.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.103704.log.gz/var/log/ceph/ceph-client.admin.66478.log: 2024-06-29T16:01:57.354 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107549.log 2024-06-29T16:01:57.354 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66478.log.gz 2024-06-29T16:01:57.354 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86991.log 2024-06-29T16:01:57.354 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.112462.log.gz 2024-06-29T16:01:57.355 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.107549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44183.log 2024-06-29T16:01:57.355 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107549.log.gz 2024-06-29T16:01:57.355 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108658.log 2024-06-29T16:01:57.355 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.86991.log.gz 2024-06-29T16:01:57.355 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87203.log 2024-06-29T16:01:57.355 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44183.log.gz 2024-06-29T16:01:57.355 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111225.log 2024-06-29T16:01:57.355 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87203.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108658.log.gz 2024-06-29T16:01:57.356 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53135.log 2024-06-29T16:01:57.356 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87203.log.gz 2024-06-29T16:01:57.356 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68854.log 2024-06-29T16:01:57.356 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111225.log.gz 2024-06-29T16:01:57.356 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72328.log 2024-06-29T16:01:57.356 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53135.log.gz 2024-06-29T16:01:57.356 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64358.log 2024-06-29T16:01:57.357 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68854.log.gz 2024-06-29T16:01:57.357 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50493.log 2024-06-29T16:01:57.357 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72328.log.gz 2024-06-29T16:01:57.357 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113072.log 2024-06-29T16:01:57.357 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64358.log.gz 2024-06-29T16:01:57.357 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63113.log 2024-06-29T16:01:57.357 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50493.log.gz 2024-06-29T16:01:57.357 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.113072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50865.log 2024-06-29T16:01:57.358 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.113072.log.gz 2024-06-29T16:01:57.358 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5/var/log/ceph/ceph-client.admin.63113.log: --verbose -- /var/log/ceph/ceph-client.admin.65697.log 2024-06-29T16:01:57.358 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63113.log.gz 2024-06-29T16:01:57.358 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105617.log 2024-06-29T16:01:57.358 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50865.log.gz 2024-06-29T16:01:57.358 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98477.log 2024-06-29T16:01:57.358 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.65697.log.gz 2024-06-29T16:01:57.358 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71736.log/var/log/ceph/ceph-client.admin.105617.log: 2024-06-29T16:01:57.359 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105617.log.gz 2024-06-29T16:01:57.359 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84567.log 2024-06-29T16:01:57.359 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.98477.log.gz 2024-06-29T16:01:57.359 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98939.log 2024-06-29T16:01:57.359 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71736.log.gz 2024-06-29T16:01:57.359 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78871.log 2024-06-29T16:01:57.359 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84567.log.gz 2024-06-29T16:01:57.360 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105407.log 2024-06-29T16:01:57.360 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98939.log.gz 2024-06-29T16:01:57.360 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97096.log 2024-06-29T16:01:57.360 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.78871.log.gz 2024-06-29T16:01:57.360 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70498.log 2024-06-29T16:01:57.360 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105407.log.gz 2024-06-29T16:01:57.360 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53829.log 2024-06-29T16:01:57.360 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97096.log.gz 2024-06-29T16:01:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45957.log 2024-06-29T16:01:57.361 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.70498.log.gz 2024-06-29T16:01:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110747.log 2024-06-29T16:01:57.361 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.53829.log.gz 2024-06-29T16:01:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112328.log 2024-06-29T16:01:57.361 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45957.log.gz 2024-06-29T16:01:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71358.log 2024-06-29T16:01:57.361 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.110747.log.gz 2024-06-29T16:01:57.361 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80285.log 2024-06-29T16:01:57.362 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112328.log.gz 2024-06-29T16:01:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54056.log 2024-06-29T16:01:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 86.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.71358.log.gz 2024-06-29T16:01:57.362 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.96888.log 2024-06-29T16:01:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80285.log: /var/log/ceph/ceph-client.admin.54056.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80285.log.gz 2024-06-29T16:01:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69575.log 2024-06-29T16:01:57.362 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.54056.log.gz 2024-06-29T16:01:57.362 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79653.log 2024-06-29T16:01:57.363 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96888.log.gz 2024-06-29T16:01:57.363 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75906.log 2024-06-29T16:01:57.363 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69575.log.gz 2024-06-29T16:01:57.363 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87103.log 2024-06-29T16:01:57.363 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79653.log.gz 2024-06-29T16:01:57.363 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75822.log 2024-06-29T16:01:57.363 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75906.log.gz 2024-06-29T16:01:57.363 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101944.log 2024-06-29T16:01:57.364 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.87103.log.gz 2024-06-29T16:01:57.364 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77844.log 2024-06-29T16:01:57.364 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75822.log.gz 2024-06-29T16:01:57.364 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5/var/log/ceph/ceph-client.admin.101944.log: --verbose -- /var/log/ceph/ceph-client.admin.48582.log 2024-06-29T16:01:57.364 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72636.log 2024-06-29T16:01:57.364 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101944.log.gz 2024-06-29T16:01:57.364 INFO:teuthology.orchestra.run.smithi081.stderr: 84.2% -- replaced with /var/log/ceph/ceph-client.admin.77844.log.gz 2024-06-29T16:01:57.364 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106713.log 2024-06-29T16:01:57.365 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48582.log.gz 2024-06-29T16:01:57.365 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97491.log 2024-06-29T16:01:57.365 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72636.log.gz 2024-06-29T16:01:57.365 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51989.log 2024-06-29T16:01:57.365 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106713.log.gz 2024-06-29T16:01:57.365 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45626.log 2024-06-29T16:01:57.365 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.97491.log.gz 2024-06-29T16:01:57.365 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.51989.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52521.log 2024-06-29T16:01:57.365 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45626.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51989.log.gz 2024-06-29T16:01:57.366 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66858.log 2024-06-29T16:01:57.366 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45626.log.gz 2024-06-29T16:01:57.366 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68686.log 2024-06-29T16:01:57.366 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52521.log.gz 2024-06-29T16:01:57.366 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93480.log 2024-06-29T16:01:57.366 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66858.log.gz 2024-06-29T16:01:57.366 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68686.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-06-29T16:01:57.366 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68686.log.gz 2024-06-29T16:01:57.367 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89824.log 2024-06-29T16:01:57.367 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98981.log 2024-06-29T16:01:57.367 INFO:teuthology.orchestra.run.smithi081.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.93480.log.gz 2024-06-29T16:01:57.367 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46327.log 2024-06-29T16:01:57.367 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89824.log.gz 2024-06-29T16:01:57.367 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64442.log 2024-06-29T16:01:57.367 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44899.log 2024-06-29T16:01:57.367 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46327.log.gz 2024-06-29T16:01:57.368 INFO:teuthology.orchestra.run.smithi081.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.98981.log.gz 2024-06-29T16:01:57.368 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101902.log 2024-06-29T16:01:57.368 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75365.log 2024-06-29T16:01:57.368 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44899.log: /var/log/ceph/ceph-client.admin.101902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67068.log 2024-06-29T16:01:57.368 INFO:teuthology.orchestra.run.smithi081.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.64442.log.gz 2024-06-29T16:01:57.368 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44899.log.gz 2024-06-29T16:01:57.368 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101902.log.gz 2024-06-29T16:01:57.368 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90061.log 2024-06-29T16:01:57.369 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75365.log.gz 2024-06-29T16:01:57.369 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100848.log 2024-06-29T16:01:57.369 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.67068.log.gz 2024-06-29T16:01:57.369 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.90061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98035.log 2024-06-29T16:01:57.369 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.90061.log.gz 2024-06-29T16:01:57.369 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50823.log 2024-06-29T16:01:57.369 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98035.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.100848.log.gz 2024-06-29T16:01:57.369 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106501.log 2024-06-29T16:01:57.370 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.98035.log.gz 2024-06-29T16:01:57.370 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45275.log 2024-06-29T16:01:57.370 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50823.log.gz 2024-06-29T16:01:57.370 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51530.log 2024-06-29T16:01:57.370 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45275.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.106501.log.gz 2024-06-29T16:01:57.370 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92067.log 2024-06-29T16:01:57.370 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45275.log.gz 2024-06-29T16:01:57.370 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66559.log 2024-06-29T16:01:57.371 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51530.log.gz 2024-06-29T16:01:57.371 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76684.log 2024-06-29T16:01:57.371 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.92067.log.gz 2024-06-29T16:01:57.371 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85039.log 2024-06-29T16:01:57.371 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66559.log.gz 2024-06-29T16:01:57.371 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91374.log 2024-06-29T16:01:57.372 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76684.log.gz 2024-06-29T16:01:57.372 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102029.log 2024-06-29T16:01:57.372 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85039.log.gz 2024-06-29T16:01:57.372 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46495.log 2024-06-29T16:01:57.372 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91374.log.gz 2024-06-29T16:01:57.372 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102117.log 2024-06-29T16:01:57.372 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102029.log.gz 2024-06-29T16:01:57.372 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69997.log 2024-06-29T16:01:57.373 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102117.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46495.log.gz 2024-06-29T16:01:57.373 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.97955.log 2024-06-29T16:01:57.373 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102117.log.gz 2024-06-29T16:01:57.373 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64984.log 2024-06-29T16:01:57.373 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.69997.log.gz 2024-06-29T16:01:57.373 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.97955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76554.log 2024-06-29T16:01:57.374 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.97955.log.gz 2024-06-29T16:01:57.374 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64925.log 2024-06-29T16:01:57.374 INFO:teuthology.orchestra.run.smithi081.stderr: 89.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-06-29T16:01:57.374 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64984.log.gz 2024-06-29T16:01:57.374 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76204.log 2024-06-29T16:01:57.374 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.76554.log.gz 2024-06-29T16:01:57.374 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105281.log 2024-06-29T16:01:57.375 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.64925.log.gz 2024-06-29T16:01:57.375 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76204.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-06-29T16:01:57.375 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.76204.log.gz 2024-06-29T16:01:57.375 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71820.log 2024-06-29T16:01:57.375 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105281.log.gz 2024-06-29T16:01:57.375 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86925.log 2024-06-29T16:01:57.375 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48077.log 2024-06-29T16:01:57.376 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86925.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71820.log.gz 2024-06-29T16:01:57.376 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45623.log 2024-06-29T16:01:57.376 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86925.log.gz 2024-06-29T16:01:57.376 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48077.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.48077.log.gz 2024-06-29T16:01:57.376 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59176.log 2024-06-29T16:01:57.376 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64316.log 2024-06-29T16:01:57.377 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45623.log.gz 2024-06-29T16:01:57.377 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.59176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105029.log 2024-06-29T16:01:57.377 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59176.log.gz 2024-06-29T16:01:57.377 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108742.log 2024-06-29T16:01:57.377 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64316.log.gz 2024-06-29T16:01:57.377 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89071.log 2024-06-29T16:01:57.377 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.105029.log.gz 2024-06-29T16:01:57.378 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52841.log 2024-06-29T16:01:57.378 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108742.log.gz 2024-06-29T16:01:57.378 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86424.log 2024-06-29T16:01:57.378 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89071.log.gz 2024-06-29T16:01:57.378 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110688.log 2024-06-29T16:01:57.378 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52841.log.gz 2024-06-29T16:01:57.378 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104252.log 2024-06-29T16:01:57.379 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110688.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86424.log.gz 2024-06-29T16:01:57.379 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64106.log 2024-06-29T16:01:57.379 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.110688.log.gz 2024-06-29T16:01:57.379 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48120.log 2024-06-29T16:01:57.379 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.104252.log.gz 2024-06-29T16:01:57.379 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86340.log 2024-06-29T16:01:57.379 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64106.log.gz 2024-06-29T16:01:57.380 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108844.log 2024-06-29T16:01:57.380 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48120.log.gz 2024-06-29T16:01:57.380 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87680.log 2024-06-29T16:01:57.380 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.86340.log.gz 2024-06-29T16:01:57.380 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111361.log 2024-06-29T16:01:57.380 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.108844.log.gz 2024-06-29T16:01:57.381 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.47704.log 2024-06-29T16:01:57.381 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87680.log.gz 2024-06-29T16:01:57.381 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93018.log 2024-06-29T16:01:57.381 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.111361.log.gz 2024-06-29T16:01:57.381 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.2.47704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109310.log 2024-06-29T16:01:57.381 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.2.47704.log.gz 2024-06-29T16:01:57.381 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78370.log 2024-06-29T16:01:57.382 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93018.log.gz 2024-06-29T16:01:57.382 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31576.log 2024-06-29T16:01:57.382 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109310.log: /var/log/ceph/ceph-client.admin.78370.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109310.log.gz 2024-06-29T16:01:57.382 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52698.log 2024-06-29T16:01:57.382 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78370.log.gz 2024-06-29T16:01:57.382 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.tmp-client.admin.31576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106127.log 2024-06-29T16:01:57.382 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31576.log.gz 2024-06-29T16:01:57.383 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100517.log 2024-06-29T16:01:57.383 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.52698.log.gz 2024-06-29T16:01:57.383 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75449.log 2024-06-29T16:01:57.383 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100517.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.106127.log.gz 2024-06-29T16:01:57.383 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88943.log 2024-06-29T16:01:57.383 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100517.log.gz 2024-06-29T16:01:57.383 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73604.log 2024-06-29T16:01:57.383 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75449.log.gz 2024-06-29T16:01:57.384 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59041.log 2024-06-29T16:01:57.384 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.88943.log.gz 2024-06-29T16:01:57.384 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.73604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84269.log 2024-06-29T16:01:57.384 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.59041.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73604.log.gz 2024-06-29T16:01:57.384 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111968.log 2024-06-29T16:01:57.384 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.59041.log.gz 2024-06-29T16:01:57.384 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80519.log 2024-06-29T16:01:57.384 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84269.log.gz 2024-06-29T16:01:57.385 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73479.log 2024-06-29T16:01:57.385 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111968.log.gz/var/log/ceph/ceph-client.admin.80519.log: 2024-06-29T16:01:57.385 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108364.log 2024-06-29T16:01:57.385 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80519.log.gz 2024-06-29T16:01:57.385 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.73479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111487.log 2024-06-29T16:01:57.385 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.73479.log.gz 2024-06-29T16:01:57.385 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93102.log 2024-06-29T16:01:57.385 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.108364.log.gz 2024-06-29T16:01:57.386 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85945.log 2024-06-29T16:01:57.386 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.111487.log.gz 2024-06-29T16:01:57.386 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96808.log 2024-06-29T16:01:57.386 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.93102.log.gz 2024-06-29T16:01:57.386 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108196.log 2024-06-29T16:01:57.386 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85945.log.gz 2024-06-29T16:01:57.386 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57921.log 2024-06-29T16:01:57.387 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96808.log.gz 2024-06-29T16:01:57.387 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79734.log 2024-06-29T16:01:57.387 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108196.log.gz 2024-06-29T16:01:57.389 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58703.log 2024-06-29T16:01:57.389 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.57921.log.gz 2024-06-29T16:01:57.389 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108970.log 2024-06-29T16:01:57.389 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79734.log.gz 2024-06-29T16:01:57.389 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76461.log 2024-06-29T16:01:57.390 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58703.log.gz 2024-06-29T16:01:57.390 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87323.log 2024-06-29T16:01:57.390 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.108970.log.gz 2024-06-29T16:01:57.390 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97361.log 2024-06-29T16:01:57.390 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76461.log.gz 2024-06-29T16:01:57.390 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88179.log/var/log/ceph/ceph-client.admin.87323.log: 2024-06-29T16:01:57.390 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.87323.log.gz 2024-06-29T16:01:57.391 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38375.log 2024-06-29T16:01:57.391 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97361.log.gz 2024-06-29T16:01:57.391 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107007.log 2024-06-29T16:01:57.391 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38375.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88179.log.gz 2024-06-29T16:01:57.391 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51320.log 2024-06-29T16:01:57.391 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38375.log.gz 2024-06-29T16:01:57.391 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.107007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85184.log 2024-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72541.log 2024-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51320.log.gz 2024-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi081.stderr: 90.9%/var/log/ceph/ceph-client.admin.85184.log: -- replaced with /var/log/ceph/ceph-client.admin.107007.log.gz 2024-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84703.log 2024-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.85184.log.gz 2024-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85615.log 2024-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72541.log: gzip/var/log/ceph/ceph-client.admin.84703.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.97407.log 2024-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72541.log.gz 2024-06-29T16:01:57.392 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.84703.log.gz 2024-06-29T16:01:57.393 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104693.log 2024-06-29T16:01:57.393 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85615.log.gz 2024-06-29T16:01:57.393 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108406.log 2024-06-29T16:01:57.393 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97407.log.gz 2024-06-29T16:01:57.393 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71526.log 2024-06-29T16:01:57.393 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104693.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.104693.log.gz 2024-06-29T16:01:57.393 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99620.log 2024-06-29T16:01:57.393 INFO:teuthology.orchestra.run.smithi081.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.108406.log.gz 2024-06-29T16:01:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52344.log 2024-06-29T16:01:57.394 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71526.log.gz 2024-06-29T16:01:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100746.log 2024-06-29T16:01:57.394 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99620.log.gz 2024-06-29T16:01:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83258.log 2024-06-29T16:01:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100746.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52344.log.gz 2024-06-29T16:01:57.394 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102453.log 2024-06-29T16:01:57.394 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.100746.log.gz 2024-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.83258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77885.log 2024-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.83258.log.gz 2024-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87145.log 2024-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77885.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.102453.log.gz 2024-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.89703.log 2024-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi081.stderr: 84.0% -- replaced with /var/log/ceph/ceph-client.admin.77885.log.gz 2024-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77549.log 2024-06-29T16:01:57.395 INFO:teuthology.orchestra.run.smithi081.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.87145.log.gz 2024-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.89703.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.89703.log.gz 2024-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68938.log 2024-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.77549.log.gz 2024-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100097.log 2024-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87908.log 2024-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68938.log.gz 2024-06-29T16:01:57.396 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71442.log 2024-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.100097.log.gz 2024-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72247.log 2024-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87908.log: /var/log/ceph/ceph-client.admin.71442.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87908.log.gz 2024-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71904.log 2024-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.71442.log.gz 2024-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54892.log 2024-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72247.log: /var/log/ceph/ceph-client.admin.71904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79226.log 2024-06-29T16:01:57.397 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72247.log.gz 2024-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71904.log.gz 2024-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72370.log 2024-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54892.log.gz 2024-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109268.log 2024-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79226.log.gz 2024-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65351.log 2024-06-29T16:01:57.398 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72370.log: /var/log/ceph/ceph-client.admin.109268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68112.log 2024-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72370.log.gz 2024-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109268.log.gz 2024-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111613.log 2024-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65351.log.gz 2024-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69489.log 2024-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68112.log: /var/log/ceph/ceph-client.admin.111613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102369.log 2024-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68112.log.gz 2024-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111613.log.gz 2024-06-29T16:01:57.399 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102537.log 2024-06-29T16:01:57.400 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69489.log.gz 2024-06-29T16:01:57.400 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104987.log 2024-06-29T16:01:57.400 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102369.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.102369.log.gz 2024-06-29T16:01:57.400 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72585.log 2024-06-29T16:01:57.400 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102537.log.gz 2024-06-29T16:01:57.400 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73128.log 2024-06-29T16:01:57.400 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104987.log: /var/log/ceph/ceph-client.admin.72585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85989.log 2024-06-29T16:01:57.400 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104987.log.gz 2024-06-29T16:01:57.401 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72585.log.gz 2024-06-29T16:01:57.401 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.73128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92930.log 2024-06-29T16:01:57.401 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73128.log.gz 2024-06-29T16:01:57.401 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99131.log 2024-06-29T16:01:57.401 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85989.log: 90.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-06-29T16:01:57.401 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70456.log 2024-06-29T16:01:57.401 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.85989.log.gz 2024-06-29T16:01:57.402 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92930.log.gz 2024-06-29T16:01:57.402 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72978.log 2024-06-29T16:01:57.402 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70456.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99131.log.gz 2024-06-29T16:01:57.402 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106288.log 2024-06-29T16:01:57.402 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70456.log.gz 2024-06-29T16:01:57.402 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48708.log 2024-06-29T16:01:57.402 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106288.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72978.log.gz 2024-06-29T16:01:57.402 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37733.log 2024-06-29T16:01:57.403 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106288.log.gz 2024-06-29T16:01:57.403 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92487.log 2024-06-29T16:01:57.403 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48708.log.gz 2024-06-29T16:01:57.403 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103304.log 2024-06-29T16:01:57.403 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37733.log.gz 2024-06-29T16:01:57.403 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103494.log 2024-06-29T16:01:57.403 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92487.log.gz 2024-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77420.log 2024-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103304.log.gz 2024-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88347.log 2024-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77420.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103494.log.gz 2024-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65309.log 2024-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.77420.log.gz 2024-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66009.log 2024-06-29T16:01:57.404 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88347.log.gz 2024-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5/var/log/ceph/ceph-client.admin.65309.log: --verbose -- /var/log/ceph/ceph-client.admin.50115.log 2024-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66009.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65309.log.gz 2024-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96125.log 2024-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.66009.log.gz 2024-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99740.log 2024-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50115.log.gz 2024-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98813.log 2024-06-29T16:01:57.405 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.96125.log.gz 2024-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89197.log 2024-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99740.log.gz 2024-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99089.log 2024-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.98813.log.gz 2024-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81090.log 2024-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.89197.log.gz 2024-06-29T16:01:57.406 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49659.log 2024-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99089.log.gz 2024-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.81090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45616.log 2024-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81090.log.gz 2024-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62926.log 2024-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49659.log.gz 2024-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52215.log 2024-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45616.log.gz 2024-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92600.log 2024-06-29T16:01:57.407 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62926.log.gz 2024-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103662.log 2024-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92600.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52215.log.gz 2024-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112885.log 2024-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92600.log.gz 2024-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105888.log 2024-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103662.log.gz 2024-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96083.log 2024-06-29T16:01:57.408 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105888.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112885.log.gz 2024-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58005.log 2024-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105888.log.gz 2024-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101520.log 2024-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.96083.log.gz 2024-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99256.log 2024-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101520.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58005.log.gz 2024-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71484.log 2024-06-29T16:01:57.409 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.101520.log.gz 2024-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71610.log 2024-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.99256.log.gz 2024-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66051.log 2024-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71610.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.71484.log.gz 2024-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100559.log 2024-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71610.log.gz 2024-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76417.log 2024-06-29T16:01:57.410 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.66051.log.gz 2024-06-29T16:01:57.411 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88814.log 2024-06-29T16:01:57.411 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.100559.log.gz 2024-06-29T16:01:57.411 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102645.log 2024-06-29T16:01:57.411 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76417.log: /var/log/ceph/ceph-client.admin.88814.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.76417.log.gz 2024-06-29T16:01:57.411 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109832.log 2024-06-29T16:01:57.411 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88814.log.gz 2024-06-29T16:01:57.411 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69106.log 2024-06-29T16:01:57.412 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109832.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102645.log.gz 2024-06-29T16:01:57.412 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100349.log 2024-06-29T16:01:57.412 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.109832.log.gz 2024-06-29T16:01:57.412 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98561.log 2024-06-29T16:01:57.412 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69106.log.gz 2024-06-29T16:01:57.412 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98435.log 2024-06-29T16:01:57.412 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100349.log.gz 2024-06-29T16:01:57.412 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98561.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-06-29T16:01:57.413 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98561.log.gz 2024-06-29T16:01:57.413 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88305.log 2024-06-29T16:01:57.413 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98435.log.gz 2024-06-29T16:01:57.413 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37929.log 2024-06-29T16:01:57.413 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79573.log 2024-06-29T16:01:57.413 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88305.log.gz 2024-06-29T16:01:57.413 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37929.log.gz 2024-06-29T16:01:57.413 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69201.log 2024-06-29T16:01:57.414 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86033.log 2024-06-29T16:01:57.414 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79573.log.gz 2024-06-29T16:01:57.414 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47265.log 2024-06-29T16:01:57.414 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69201.log.gz 2024-06-29T16:01:57.414 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98245.log 2024-06-29T16:01:57.414 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.86033.log.gz 2024-06-29T16:01:57.414 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.47265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.47265.log.gz 2024-06-29T16:01:57.414 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64148.log 2024-06-29T16:01:57.415 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46159.log 2024-06-29T16:01:57.415 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.98245.log.gz 2024-06-29T16:01:57.415 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108574.log 2024-06-29T16:01:57.415 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64148.log.gz 2024-06-29T16:01:57.415 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111884.log 2024-06-29T16:01:57.415 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.46159.log.gz 2024-06-29T16:01:57.416 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58276.log 2024-06-29T16:01:57.416 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108574.log: /var/log/ceph/ceph-client.admin.111884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105575.log 2024-06-29T16:01:57.416 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.108574.log.gz 2024-06-29T16:01:57.416 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111884.log.gz 2024-06-29T16:01:57.416 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94023.log 2024-06-29T16:01:57.416 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58276.log.gz 2024-06-29T16:01:57.416 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96587.log 2024-06-29T16:01:57.417 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105575.log: /var/log/ceph/ceph-client.admin.94023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66601.log 2024-06-29T16:01:57.417 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105575.log.gz 2024-06-29T16:01:57.417 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.94023.log.gz 2024-06-29T16:01:57.417 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103920.log 2024-06-29T16:01:57.417 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 87.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.96587.log.gz /var/log/ceph/ceph-client.admin.45067.log 2024-06-29T16:01:57.417 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.417 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66601.log: /var/log/ceph/ceph-client.admin.103920.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66601.log.gz 2024-06-29T16:01:57.417 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110483.log 2024-06-29T16:01:57.418 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.103920.log.gz 2024-06-29T16:01:57.418 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88772.log 2024-06-29T16:01:57.418 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45067.log: /var/log/ceph/ceph-client.admin.110483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58047.log 2024-06-29T16:01:57.418 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.45067.log.gz 2024-06-29T16:01:57.418 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.110483.log.gz 2024-06-29T16:01:57.418 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66772.log 2024-06-29T16:01:57.419 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88772.log.gz 2024-06-29T16:01:57.419 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57669.log 2024-06-29T16:01:57.419 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58047.log.gz 2024-06-29T16:01:57.419 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75636.log 2024-06-29T16:01:57.419 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66772.log: /var/log/ceph/ceph-client.admin.57669.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.66772.log.gz 2024-06-29T16:01:57.419 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49029.log 2024-06-29T16:01:57.419 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57669.log.gz 2024-06-29T16:01:57.420 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93396.log 2024-06-29T16:01:57.420 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68560.log 2024-06-29T16:01:57.420 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.75636.log.gz 2024-06-29T16:01:57.420 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49029.log: /var/log/ceph/ceph-client.admin.93396.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49029.log.gz 2024-06-29T16:01:57.420 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58402.log 2024-06-29T16:01:57.420 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.93396.log.gz 2024-06-29T16:01:57.420 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99877.log 2024-06-29T16:01:57.421 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68560.log: /var/log/ceph/ceph-client.admin.58402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50367.log 2024-06-29T16:01:57.421 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68560.log.gz 2024-06-29T16:01:57.421 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99877.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58402.log.gz 2024-06-29T16:01:57.421 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65487.log 2024-06-29T16:01:57.421 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99877.log.gz 2024-06-29T16:01:57.421 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54233.log 2024-06-29T16:01:57.421 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50367.log.gz 2024-06-29T16:01:57.422 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62401.log 2024-06-29T16:01:57.422 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.65487.log.gz 2024-06-29T16:01:57.422 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91731.log 2024-06-29T16:01:57.422 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.54233.log.gz 2024-06-29T16:01:57.422 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101184.log 2024-06-29T16:01:57.422 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62401.log: /var/log/ceph/ceph-client.admin.91731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51362.log 2024-06-29T16:01:57.422 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62401.log.gz 2024-06-29T16:01:57.423 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91731.log.gz 2024-06-29T16:01:57.423 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75948.log 2024-06-29T16:01:57.423 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58318.log 2024-06-29T16:01:57.423 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.101184.log.gz 2024-06-29T16:01:57.423 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51362.log: /var/log/ceph/ceph-client.admin.75948.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51362.log.gz 2024-06-29T16:01:57.423 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100932.log 2024-06-29T16:01:57.424 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75948.log.gz 2024-06-29T16:01:57.424 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.58318.log: /var/log/ceph/ceph-client.admin.64274.log 2024-06-29T16:01:57.424 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107073.log 2024-06-29T16:01:57.424 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58318.log.gz 2024-06-29T16:01:57.424 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64274.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100932.log.gz 2024-06-29T16:01:57.424 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78008.log 2024-06-29T16:01:57.424 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.64274.log.gz 2024-06-29T16:01:57.425 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.107073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78507.log 2024-06-29T16:01:57.425 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107073.log.gz 2024-06-29T16:01:57.425 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110525.log 2024-06-29T16:01:57.425 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78008.log: /var/log/ceph/ceph-client.admin.78507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91689.log 2024-06-29T16:01:57.425 INFO:teuthology.orchestra.run.smithi081.stderr: 84.7% -- replaced with /var/log/ceph/ceph-client.admin.78008.log.gz 2024-06-29T16:01:57.425 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78507.log.gz 2024-06-29T16:01:57.426 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68980.log 2024-06-29T16:01:57.426 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110525.log.gz 2024-06-29T16:01:57.426 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48750.log 2024-06-29T16:01:57.426 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91689.log: /var/log/ceph/ceph-client.admin.68980.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91689.log.gz 2024-06-29T16:01:57.426 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70147.log 2024-06-29T16:01:57.426 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68980.log.gz 2024-06-29T16:01:57.426 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62317.log 2024-06-29T16:01:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48750.log: /var/log/ceph/ceph-client.admin.70147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96167.log 2024-06-29T16:01:57.427 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48750.log.gz 2024-06-29T16:01:57.427 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70147.log.gz 2024-06-29T16:01:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65739.log 2024-06-29T16:01:57.427 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62317.log.gz 2024-06-29T16:01:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96167.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-06-29T16:01:57.427 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.96167.log.gz 2024-06-29T16:01:57.427 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93312.log 2024-06-29T16:01:57.428 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65739.log: /var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46285.log 2024-06-29T16:01:57.428 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65739.log.gz 2024-06-29T16:01:57.428 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102073.log 2024-06-29T16:01:57.428 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.93312.log.gz/var/log/ceph/ceph-client.admin.46285.log: 2024-06-29T16:01:57.428 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47941.log 2024-06-29T16:01:57.428 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46285.log.gz 2024-06-29T16:01:57.428 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50325.log 2024-06-29T16:01:57.428 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47941.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.102073.log.gz 2024-06-29T16:01:57.429 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53745.log 2024-06-29T16:01:57.429 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47941.log.gz 2024-06-29T16:01:57.429 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109437.log 2024-06-29T16:01:57.429 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50325.log.gz 2024-06-29T16:01:57.429 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108154.log 2024-06-29T16:01:57.429 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53745.log.gz 2024-06-29T16:01:57.429 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95957.log 2024-06-29T16:01:57.429 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109437.log: /var/log/ceph/ceph-client.admin.108154.log: gzip -5 86.9% -- replaced with /var/log/ceph/ceph-client.admin.109437.log.gz 2024-06-29T16:01:57.430 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57879.log 2024-06-29T16:01:57.430 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108154.log.gz 2024-06-29T16:01:57.430 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.95957.log: gzip -5 --verbose -- 87.4% /var/log/ceph/ceph-client.admin.105155.log 2024-06-29T16:01:57.430 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.95957.log.gz 2024-06-29T16:01:57.430 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87050.log 2024-06-29T16:01:57.430 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57879.log: /var/log/ceph/ceph-client.admin.105155.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.57879.log.gz 2024-06-29T16:01:57.430 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92761.log 2024-06-29T16:01:57.430 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105155.log.gz 2024-06-29T16:01:57.431 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106881.log 2024-06-29T16:01:57.431 INFO:teuthology.orchestra.run.smithi081.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87050.log.gz 2024-06-29T16:01:57.431 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111046.log 2024-06-29T16:01:57.431 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106881.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92761.log.gz 2024-06-29T16:01:57.431 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76123.log 2024-06-29T16:01:57.431 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.106881.log.gz 2024-06-29T16:01:57.431 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100890.log 2024-06-29T16:01:57.431 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76123.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.111046.log.gzgzip 2024-06-29T16:01:57.432 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.111130.log 2024-06-29T16:01:57.432 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.76123.log.gz 2024-06-29T16:01:57.432 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109874.log 2024-06-29T16:01:57.432 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111130.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100890.log.gz 2024-06-29T16:01:57.432 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72167.log 2024-06-29T16:01:57.432 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111130.log.gz 2024-06-29T16:01:57.432 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102201.log 2024-06-29T16:01:57.432 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72167.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.109874.log.gz 2024-06-29T16:01:57.433 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68059.log 2024-06-29T16:01:57.433 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72167.log.gz 2024-06-29T16:01:57.433 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53661.log 2024-06-29T16:01:57.433 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96419.log 2024-06-29T16:01:57.433 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102201.log.gz 2024-06-29T16:01:57.433 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68059.log: /var/log/ceph/ceph-client.admin.53661.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68059.log.gz 2024-06-29T16:01:57.433 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92361.log 2024-06-29T16:01:57.433 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.53661.log.gz 2024-06-29T16:01:57.434 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86256.log 2024-06-29T16:01:57.434 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92361.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.96419.log.gz 2024-06-29T16:01:57.434 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60130.log 2024-06-29T16:01:57.434 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92361.log.gz 2024-06-29T16:01:57.434 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66900.log 2024-06-29T16:01:57.434 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.74935.log 2024-06-29T16:01:57.434 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86256.log.gz 2024-06-29T16:01:57.435 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.60130.log: /var/log/ceph/ceph-client.admin.66900.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.60130.log.gz 2024-06-29T16:01:57.435 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111403.log 2024-06-29T16:01:57.435 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66900.log.gz 2024-06-29T16:01:57.435 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.74935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65214.log 2024-06-29T16:01:57.435 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74935.log.gz 2024-06-29T16:01:57.435 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105762.log 2024-06-29T16:01:57.435 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111403.log.gz 2024-06-29T16:01:57.435 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100704.log/var/log/ceph/ceph-client.admin.65214.log: 2024-06-29T16:01:57.436 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65214.log.gz 2024-06-29T16:01:57.436 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100265.log 2024-06-29T16:01:57.436 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105762.log: /var/log/ceph/ceph-client.admin.100704.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105762.log.gz 2024-06-29T16:01:57.436 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45956.log 2024-06-29T16:01:57.436 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.100704.log.gz 2024-06-29T16:01:57.436 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86466.log 2024-06-29T16:01:57.436 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.100265.log.gz 2024-06-29T16:01:57.437 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104389.log 2024-06-29T16:01:57.437 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45956.log.gz 2024-06-29T16:01:57.437 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104861.log 2024-06-29T16:01:57.437 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86466.log.gz 2024-06-29T16:01:57.437 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104389.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-06-29T16:01:57.437 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.104389.log.gz 2024-06-29T16:01:57.437 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104861.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-06-29T16:01:57.437 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104861.log.gz 2024-06-29T16:01:57.438 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79947.log 2024-06-29T16:01:57.438 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65613.log 2024-06-29T16:01:57.438 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101058.log 2024-06-29T16:01:57.438 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79947.log.gz/var/log/ceph/ceph-client.admin.65613.log: 2024-06-29T16:01:57.438 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96251.log 2024-06-29T16:01:57.438 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65613.log.gz 2024-06-29T16:01:57.439 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60059.log 2024-06-29T16:01:57.439 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49779.log 2024-06-29T16:01:57.439 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.101058.log.gz 2024-06-29T16:01:57.439 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96251.log: /var/log/ceph/ceph-client.admin.60059.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.96251.log.gz 2024-06-29T16:01:57.439 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60059.log.gz 2024-06-29T16:01:57.439 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79142.log 2024-06-29T16:01:57.439 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53787.log 2024-06-29T16:01:57.439 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49779.log: gzip/var/log/ceph/ceph-client.admin.79142.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.95642.log 2024-06-29T16:01:57.440 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2%/var/log/ceph/ceph-client.admin.53787.log: -- replaced with /var/log/ceph/ceph-client.admin.79142.log.gz 2024-06-29T16:01:57.440 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.49779.log.gz 2024-06-29T16:01:57.440 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53787.log.gz 2024-06-29T16:01:57.440 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112201.log 2024-06-29T16:01:57.440 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.95642.log: gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/ceph-client.admin.95642.log.gz 2024-06-29T16:01:57.440 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.65172.log 2024-06-29T16:01:57.440 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52799.log 87.0% -- replaced with /var/log/ceph/ceph-client.admin.112201.log.gz 2024-06-29T16:01:57.440 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.440 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99519.log 87.1% 2024-06-29T16:01:57.441 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65172.log.gz 2024-06-29T16:01:57.441 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85352.log 2024-06-29T16:01:57.441 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52799.log.gz 2024-06-29T16:01:57.441 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80870.log 2024-06-29T16:01:57.441 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99519.log.gz 2024-06-29T16:01:57.441 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112591.log 2024-06-29T16:01:57.441 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85352.log: /var/log/ceph/ceph-client.admin.80870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109916.log 2024-06-29T16:01:57.442 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112591.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85352.log.gz 2024-06-29T16:01:57.442 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.80870.log.gz 2024-06-29T16:01:57.442 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101142.log 2024-06-29T16:01:57.442 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112591.log.gz 2024-06-29T16:01:57.442 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.109916.log: -- /var/log/ceph/ceph-client.admin.59264.log 2024-06-29T16:01:57.442 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101142.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.109916.log.gz 2024-06-29T16:01:57.442 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78465.log 2024-06-29T16:01:57.442 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101142.log.gz 2024-06-29T16:01:57.443 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.59264.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59264.log.gz 2024-06-29T16:01:57.443 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101394.log 2024-06-29T16:01:57.443 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68017.log 2024-06-29T16:01:57.443 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78465.log: gzip/var/log/ceph/ceph-client.admin.101394.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.71694.log 2024-06-29T16:01:57.443 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78465.log.gz 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101394.log.gz 2024-06-29T16:01:57.443 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.443 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68017.log: gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.68017.log.gz 2024-06-29T16:01:57.443 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50907.log 2024-06-29T16:01:57.444 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72894.log 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71694.log.gz 2024-06-29T16:01:57.444 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.444 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90008.log 2024-06-29T16:01:57.444 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50907.log.gz 2024-06-29T16:01:57.444 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53871.log 2024-06-29T16:01:57.444 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.72894.log.gz 2024-06-29T16:01:57.444 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.90008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63980.log 2024-06-29T16:01:57.444 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90008.log.gz 2024-06-29T16:01:57.444 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96335.log 2024-06-29T16:01:57.445 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53871.log.gz 2024-06-29T16:01:57.445 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-06-29T16:01:57.445 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63980.log: gzip/var/log/ceph/ceph-client.admin.96335.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67856.log 2024-06-29T16:01:57.445 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4%/var/log/ceph/ceph-osd.0.log: -- replaced with /var/log/ceph/ceph-client.admin.96335.log.gz 2024-06-29T16:01:57.445 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63980.log.gz 2024-06-29T16:01:57.445 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50991.log 2024-06-29T16:01:57.445 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62800.log 2024-06-29T16:01:57.445 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67856.log.gz 2024-06-29T16:01:57.446 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49113.log 2024-06-29T16:01:57.446 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50991.log.gz 2024-06-29T16:01:57.446 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62800.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.62800.log.gz 2024-06-29T16:01:57.446 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89407.log 2024-06-29T16:01:57.446 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46663.log 2024-06-29T16:01:57.446 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49113.log.gz 2024-06-29T16:01:57.446 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101436.log 2024-06-29T16:01:57.446 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46663.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89407.log.gz 2024-06-29T16:01:57.447 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46663.log.gz 2024-06-29T16:01:57.447 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87596.log 2024-06-29T16:01:57.447 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101016.log 2024-06-29T16:01:57.447 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101436.log: /var/log/ceph/ceph-client.admin.87596.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101436.log.gz 2024-06-29T16:01:57.447 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87596.log.gz 2024-06-29T16:01:57.447 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55681.log 2024-06-29T16:01:57.447 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94076.log 2024-06-29T16:01:57.448 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55681.log: /var/log/ceph/ceph-client.admin.101016.log: 87.0% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101016.log.gz 2024-06-29T16:01:57.448 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55681.log.gz 2024-06-29T16:01:57.448 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59432.log 2024-06-29T16:01:57.448 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80075.log 2024-06-29T16:01:57.448 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.94076.log: /var/log/ceph/ceph-client.admin.59432.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.94076.log.gz 2024-06-29T16:01:57.448 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59432.log.gz 2024-06-29T16:01:57.448 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.73316.log 2024-06-29T16:01:57.448 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80075.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80075.log.gz 2024-06-29T16:01:57.448 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66398.log 2024-06-29T16:01:57.449 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77667.log 2024-06-29T16:01:57.449 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.73316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.73316.log.gz 2024-06-29T16:01:57.449 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101352.log 2024-06-29T16:01:57.449 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66398.log.gz 2024-06-29T16:01:57.449 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77667.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77667.log.gz 2024-06-29T16:01:57.449 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48288.log 2024-06-29T16:01:57.449 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72497.log 2024-06-29T16:01:57.449 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101352.log: /var/log/ceph/ceph-client.admin.48288.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101352.log.gz 2024-06-29T16:01:57.450 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48288.log.gz 2024-06-29T16:01:57.450 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87764.log 2024-06-29T16:01:57.450 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72497.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72497.log.gz 2024-06-29T16:01:57.450 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96377.log 2024-06-29T16:01:57.450 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106330.log 2024-06-29T16:01:57.450 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87764.log: /var/log/ceph/ceph-client.admin.96377.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.87764.log.gz 2024-06-29T16:01:57.450 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.96377.log.gz 2024-06-29T16:01:57.450 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97319.log 2024-06-29T16:01:57.451 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106330.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.106330.log.gz 2024-06-29T16:01:57.451 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77270.log 2024-06-29T16:01:57.451 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64232.log 2024-06-29T16:01:57.451 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97319.log: /var/log/ceph/ceph-client.admin.77270.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97319.log.gz 2024-06-29T16:01:57.451 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.77270.log.gz 2024-06-29T16:01:57.451 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101699.log 2024-06-29T16:01:57.451 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64232.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64232.log.gz 2024-06-29T16:01:57.451 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71400.log 2024-06-29T16:01:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89782.log 2024-06-29T16:01:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101699.log: /var/log/ceph/ceph-client.admin.71400.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101699.log.gz 2024-06-29T16:01:57.452 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.71400.log.gz 2024-06-29T16:01:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99393.log 2024-06-29T16:01:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89782.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.89782.log.gz 2024-06-29T16:01:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53619.log 2024-06-29T16:01:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58234.log 2024-06-29T16:01:57.452 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99393.log: /var/log/ceph/ceph-client.admin.53619.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.99393.log.gz 2024-06-29T16:01:57.453 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53619.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68248.log 2024-06-29T16:01:57.453 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.453 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58234.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58234.log.gz 2024-06-29T16:01:57.453 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78829.log 2024-06-29T16:01:57.453 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106587.log 2024-06-29T16:01:57.453 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68248.log: /var/log/ceph/ceph-client.admin.78829.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68248.log.gz 2024-06-29T16:01:57.453 INFO:teuthology.orchestra.run.smithi081.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.78829.log.gz 2024-06-29T16:01:57.454 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45617.log 2024-06-29T16:01:57.454 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101987.log 2024-06-29T16:01:57.454 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106587.log: /var/log/ceph/ceph-client.admin.45617.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106587.log.gz 2024-06-29T16:01:57.454 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45617.log.gz 2024-06-29T16:01:57.454 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46927.log 2024-06-29T16:01:57.454 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101987.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.101987.log.gz 2024-06-29T16:01:57.455 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111529.log 2024-06-29T16:01:57.455 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91773.log 2024-06-29T16:01:57.455 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46927.log: /var/log/ceph/ceph-client.admin.111529.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46927.log.gz 2024-06-29T16:01:57.455 INFO:teuthology.orchestra.run.smithi081.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.111529.log.gz 2024-06-29T16:01:57.455 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108886.log 2024-06-29T16:01:57.455 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91773.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91773.log.gz 2024-06-29T16:01:57.456 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112801.log 2024-06-29T16:01:57.456 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113112.log 2024-06-29T16:01:57.456 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108886.log: /var/log/ceph/ceph-client.admin.112801.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108886.log.gz 2024-06-29T16:01:57.456 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112801.log.gzgzip 2024-06-29T16:01:57.456 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49359.log 2024-06-29T16:01:57.456 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.113112.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113112.log.gz 2024-06-29T16:01:57.456 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72852.log 2024-06-29T16:01:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49359.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49359.log.gz 2024-06-29T16:01:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44264.log 2024-06-29T16:01:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88473.log 2024-06-29T16:01:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72852.log: /var/log/ceph/ceph-client.admin.44264.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.72852.log.gz 2024-06-29T16:01:57.457 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44264.log.gz 2024-06-29T16:01:57.457 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63807.log 2024-06-29T16:01:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71120.log 2024-06-29T16:01:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88473.log: /var/log/ceph/ceph-client.admin.63807.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88473.log.gz 2024-06-29T16:01:57.458 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45842.log 2024-06-29T16:01:57.458 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63807.log.gz 2024-06-29T16:01:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71120.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71120.log.gz 2024-06-29T16:01:57.458 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79478.log 2024-06-29T16:01:57.459 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56996.log 2024-06-29T16:01:57.459 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45842.log: /var/log/ceph/ceph-client.admin.79478.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45842.log.gz 2024-06-29T16:01:57.459 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.79478.log.gz 2024-06-29T16:01:57.459 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89365.log 2024-06-29T16:01:57.459 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56996.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.56996.log.gz 2024-06-29T16:01:57.459 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80201.log 2024-06-29T16:01:57.459 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72412.log 2024-06-29T16:01:57.460 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89365.log: /var/log/ceph/ceph-client.admin.80201.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89365.log.gz 2024-06-29T16:01:57.460 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80201.log.gz 2024-06-29T16:01:57.460 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107644.log 2024-06-29T16:01:57.460 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72412.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72412.log.gz 2024-06-29T16:01:57.460 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75491.log 2024-06-29T16:01:57.460 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63425.log 2024-06-29T16:01:57.460 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.107644.log: /var/log/ceph/ceph-client.admin.75491.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.107644.log.gz 2024-06-29T16:01:57.461 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75491.log.gz 2024-06-29T16:01:57.461 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50738.log 2024-06-29T16:01:57.461 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63425.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63425.log.gz 2024-06-29T16:01:57.461 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96969.log 2024-06-29T16:01:57.461 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.77191.log 2024-06-29T16:01:57.461 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50738.log: /var/log/ceph/ceph-client.admin.96969.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50738.log.gz 2024-06-29T16:01:57.462 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.96969.log.gz 2024-06-29T16:01:57.462 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54115.log 2024-06-29T16:01:57.462 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.77191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.77191.log.gz 2024-06-29T16:01:57.462 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37957.log 2024-06-29T16:01:57.462 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63052.log 2024-06-29T16:01:57.462 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54115.log: /var/log/ceph/ceph-client.admin.37957.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54115.log.gz 2024-06-29T16:01:57.463 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37957.log.gz 2024-06-29T16:01:57.463 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101860.log 2024-06-29T16:01:57.463 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104431.log 2024-06-29T16:01:57.463 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63052.log: /var/log/ceph/ceph-client.admin.101860.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63052.log.gz 2024-06-29T16:01:57.463 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.101860.log.gz 2024-06-29T16:01:57.463 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65781.log 2024-06-29T16:01:57.463 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104431.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.104431.log.gz 2024-06-29T16:01:57.464 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52639.log 2024-06-29T16:01:57.464 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96545.log 2024-06-29T16:01:57.464 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65781.log: /var/log/ceph/ceph-client.admin.52639.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65781.log.gz 2024-06-29T16:01:57.464 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52639.log.gz 2024-06-29T16:01:57.464 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103165.log 2024-06-29T16:01:57.464 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96545.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96545.log.gz 2024-06-29T16:01:57.464 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50409.log 2024-06-29T16:01:57.465 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63383.log 2024-06-29T16:01:57.465 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103165.log: /var/log/ceph/ceph-client.admin.50409.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103165.log.gz 2024-06-29T16:01:57.465 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50409.log.gz 2024-06-29T16:01:57.465 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97226.log 2024-06-29T16:01:57.465 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63383.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.63383.log.gz 2024-06-29T16:01:57.465 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37908.log 2024-06-29T16:01:57.465 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51572.log 2024-06-29T16:01:57.466 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97226.log: /var/log/ceph/ceph-client.admin.37908.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97226.log.gz 2024-06-29T16:01:57.466 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37908.log.gz 2024-06-29T16:01:57.466 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59306.log 2024-06-29T16:01:57.466 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58192.log 2024-06-29T16:01:57.466 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51572.log: /var/log/ceph/ceph-client.admin.59306.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51572.log.gz 2024-06-29T16:01:57.466 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59306.log.gz 2024-06-29T16:01:57.466 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45288.log 2024-06-29T16:01:57.466 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58192.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58192.log.gz 2024-06-29T16:01:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37797.log 2024-06-29T16:01:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76596.log 2024-06-29T16:01:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45288.log: /var/log/ceph/ceph-client.admin.37797.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.45288.log.gz 2024-06-29T16:01:57.467 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37797.log.gz 2024-06-29T16:01:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48162.log 2024-06-29T16:01:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37637.log 2024-06-29T16:01:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76596.log: /var/log/ceph/ceph-client.admin.48162.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76596.log.gz 2024-06-29T16:01:57.467 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48162.log.gz 2024-06-29T16:01:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105678.log 2024-06-29T16:01:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37637.log.gz 2024-06-29T16:01:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59516.log 2024-06-29T16:01:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106755.log 2024-06-29T16:01:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105678.log: /var/log/ceph/ceph-client.admin.59516.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105678.log.gz 2024-06-29T16:01:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74884.log 2024-06-29T16:01:57.468 INFO:teuthology.orchestra.run.smithi081.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.59516.log.gz 2024-06-29T16:01:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106755.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106755.log.gz 2024-06-29T16:01:57.469 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59390.log 2024-06-29T16:01:57.469 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57711.log 2024-06-29T16:01:57.469 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.74884.log: /var/log/ceph/ceph-client.admin.59390.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74884.log.gz 2024-06-29T16:01:57.469 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.59390.log.gz 2024-06-29T16:01:57.469 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91416.log 2024-06-29T16:01:57.469 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57711.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57711.log.gz 2024-06-29T16:01:57.469 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80159.log 2024-06-29T16:01:57.469 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66816.log 2024-06-29T16:01:57.469 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91416.log: /var/log/ceph/ceph-client.admin.80159.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91416.log.gz 2024-06-29T16:01:57.470 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80159.log.gz 2024-06-29T16:01:57.470 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58911.log 2024-06-29T16:01:57.470 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100601.log 2024-06-29T16:01:57.470 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66816.log: /var/log/ceph/ceph-client.admin.58911.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66816.log.gz 2024-06-29T16:01:57.470 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.58911.log.gz 2024-06-29T16:01:57.470 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111842.log 2024-06-29T16:01:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100601.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100601.log.gz 2024-06-29T16:01:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78412.log 2024-06-29T16:01:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52139.log 2024-06-29T16:01:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111842.log: /var/log/ceph/ceph-client.admin.78412.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111842.log.gz 2024-06-29T16:01:57.471 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78412.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85818.log 2024-06-29T16:01:57.471 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52139.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52139.log.gz 2024-06-29T16:01:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48624.log 2024-06-29T16:01:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89866.log 2024-06-29T16:01:57.472 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85818.log: /var/log/ceph/ceph-client.admin.48624.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85818.log.gz 2024-06-29T16:01:57.472 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48624.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92403.log 2024-06-29T16:01:57.472 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.472 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89866.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89866.log.gz 2024-06-29T16:01:57.472 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69787.log 2024-06-29T16:01:57.472 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103536.log 2024-06-29T16:01:57.472 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92403.log: /var/log/ceph/ceph-client.admin.69787.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92403.log.gz 2024-06-29T16:01:57.472 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69787.log.gz 2024-06-29T16:01:57.473 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87848.log 2024-06-29T16:01:57.473 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103536.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103536.log.gz 2024-06-29T16:01:57.473 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107965.log 2024-06-29T16:01:57.473 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57837.log 2024-06-29T16:01:57.473 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87848.log: /var/log/ceph/ceph-client.admin.107965.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.87848.log.gz 2024-06-29T16:01:57.473 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104819.log 2024-06-29T16:01:57.473 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107965.log.gz 2024-06-29T16:01:57.474 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92888.log 2024-06-29T16:01:57.474 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104819.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.57837.log.gz 2024-06-29T16:01:57.474 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104819.log.gz 2024-06-29T16:01:57.474 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78661.log 2024-06-29T16:01:57.474 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78703.log 2024-06-29T16:01:57.474 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78661.log: /var/log/ceph/ceph-client.admin.92888.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92888.log.gz 2024-06-29T16:01:57.474 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.78661.log.gz 2024-06-29T16:01:57.474 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88569.log 2024-06-29T16:01:57.475 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63562.log 2024-06-29T16:01:57.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78703.log: /var/log/ceph/ceph-client.admin.88569.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78703.log.gz 2024-06-29T16:01:57.475 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88569.log.gz 2024-06-29T16:01:57.475 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64190.log 2024-06-29T16:01:57.475 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78787.log 2024-06-29T16:01:57.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63562.log: /var/log/ceph/ceph-client.admin.64190.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63562.log.gz 2024-06-29T16:01:57.475 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.64190.log.gz 2024-06-29T16:01:57.475 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65967.log 2024-06-29T16:01:57.476 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49275.log 2024-06-29T16:01:57.476 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65967.log: /var/log/ceph/ceph-client.admin.78787.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.78787.log.gz 2024-06-29T16:01:57.476 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.65967.log.gz 2024-06-29T16:01:57.476 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66984.log 2024-06-29T16:01:57.476 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.49275.log: /var/log/ceph/ceph-client.admin.68812.log 2024-06-29T16:01:57.476 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49275.log.gz 2024-06-29T16:01:57.476 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45611.log 2024-06-29T16:01:57.476 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68812.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66984.log.gz 2024-06-29T16:01:57.477 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68812.log.gz 2024-06-29T16:01:57.477 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65907.log 2024-06-29T16:01:57.477 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58826.log 2024-06-29T16:01:57.477 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45611.log.gz 2024-06-29T16:01:57.477 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103770.log 2024-06-29T16:01:57.477 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58826.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65907.log.gz 2024-06-29T16:01:57.477 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58826.log.gz 2024-06-29T16:01:57.477 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98519.log 2024-06-29T16:01:57.478 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64022.log 2024-06-29T16:01:57.478 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103770.log: /var/log/ceph/ceph-client.admin.98519.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103770.log.gz 2024-06-29T16:01:57.478 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98519.log.gz 2024-06-29T16:01:57.478 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92445.log 2024-06-29T16:01:57.478 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64022.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64022.log.gz 2024-06-29T16:01:57.478 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38015.log 2024-06-29T16:01:57.478 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-06-29T16:01:57.479 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92445.log: /var/log/ceph/ceph-client.admin.38015.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92445.log.gz 2024-06-29T16:01:57.479 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38015.log.gz 2024-06-29T16:01:57.479 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80453.log 2024-06-29T16:01:57.479 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70039.log 2024-06-29T16:01:57.479 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.80453.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.80453.log.gz 2024-06-29T16:01:57.479 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95370.log 2024-06-29T16:01:57.481 INFO:teuthology.orchestra.run.smithi167.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-06-29T16:01:57.872 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70039.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70039.log.gz 2024-06-29T16:01:57.872 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72072.log 2024-06-29T16:01:57.873 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.95370.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.95370.log.gz 2024-06-29T16:01:57.873 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54629.log 2024-06-29T16:01:57.873 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72072.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72072.log.gz 2024-06-29T16:01:57.873 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78049.log 2024-06-29T16:01:57.873 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54629.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54629.log.gz 2024-06-29T16:01:57.873 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92974.log 2024-06-29T16:01:57.873 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78049.log: 84.2% -- replaced with /var/log/ceph/ceph-client.admin.78049.log.gz 2024-06-29T16:01:57.873 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111088.log 2024-06-29T16:01:57.874 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92974.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.92974.log.gz 2024-06-29T16:01:57.874 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68728.log 2024-06-29T16:01:57.874 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111088.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.111088.log.gz 2024-06-29T16:01:57.874 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97449.log 2024-06-29T16:01:57.874 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68728.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68728.log.gz 2024-06-29T16:01:57.874 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77354.log 2024-06-29T16:01:57.875 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97449.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97449.log.gz 2024-06-29T16:01:57.875 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52967.log 2024-06-29T16:01:57.875 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77354.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.77354.log.gz 2024-06-29T16:01:57.875 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102411.log 2024-06-29T16:01:57.875 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52967.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.52967.log.gz 2024-06-29T16:01:57.875 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93270.log 2024-06-29T16:01:57.875 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102411.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.102411.log.gz 2024-06-29T16:01:57.876 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45281.log 2024-06-29T16:01:57.876 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93270.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.93270.log.gz 2024-06-29T16:01:57.876 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79903.log 2024-06-29T16:01:57.876 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45281.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45281.log.gz 2024-06-29T16:01:57.876 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76768.log 2024-06-29T16:01:57.876 INFO:teuthology.orchestra.run.smithi081.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-06-29T16:01:57.876 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79903.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79903.log.gz 2024-06-29T16:01:57.877 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71568.log 2024-06-29T16:01:57.877 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48035.log 2024-06-29T16:01:57.877 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76768.log.gz 2024-06-29T16:01:57.877 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79394.log 2024-06-29T16:01:57.877 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5%/var/log/ceph/ceph-client.admin.48035.log: -- replaced with /var/log/ceph/ceph-client.admin.71568.log.gz 2024-06-29T16:01:57.877 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48035.log.gz 2024-06-29T16:01:57.877 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62632.log 2024-06-29T16:01:57.877 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96671.log 2024-06-29T16:01:57.878 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62632.log: /var/log/ceph/ceph-client.admin.79394.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79394.log.gz 2024-06-29T16:01:57.878 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62632.log.gz 2024-06-29T16:01:57.878 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78328.log 2024-06-29T16:01:57.878 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107686.log 2024-06-29T16:01:57.878 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96671.log.gz 2024-06-29T16:01:57.878 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72030.log 2024-06-29T16:01:57.878 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.107686.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78328.log.gz 2024-06-29T16:01:57.879 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49575.log 2024-06-29T16:01:57.879 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.107686.log.gz 2024-06-29T16:01:57.879 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65571.log 2024-06-29T16:01:57.879 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72030.log: /var/log/ceph/ceph-client.admin.49575.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72030.log.gz 2024-06-29T16:01:57.879 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49575.log.gz 2024-06-29T16:01:57.879 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102495.log 2024-06-29T16:01:57.879 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48540.log 2024-06-29T16:01:57.880 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65571.log.gz 2024-06-29T16:01:57.880 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74840.log 2024-06-29T16:01:57.880 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0%/var/log/ceph/ceph-client.admin.48540.log: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-06-29T16:01:57.880 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.102495.log.gz 2024-06-29T16:01:57.880 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.48540.log.gz 2024-06-29T16:01:57.880 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73657.log 2024-06-29T16:01:57.880 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.74840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54587.log 2024-06-29T16:01:57.880 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74840.log.gz 2024-06-29T16:01:57.880 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51614.log 2024-06-29T16:01:57.881 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.73657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75281.log 2024-06-29T16:01:57.881 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.73657.log.gz 2024-06-29T16:01:57.881 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54587.log: /var/log/ceph/ceph-client.admin.51614.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.54587.log.gz 2024-06-29T16:01:57.881 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51614.log.gz 2024-06-29T16:01:57.881 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94118.log 2024-06-29T16:01:57.881 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38069.log 2024-06-29T16:01:57.881 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.75281.log.gz/var/log/ceph/ceph-client.admin.94118.log: 2024-06-29T16:01:57.881 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72455.log 2024-06-29T16:01:57.882 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.94118.log.gz 2024-06-29T16:01:57.882 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38069.log.gz 2024-06-29T16:01:57.882 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89239.log 2024-06-29T16:01:57.882 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72455.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72455.log.gz 2024-06-29T16:01:57.882 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.103209.log 2024-06-29T16:01:57.882 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51232.log 2024-06-29T16:01:57.882 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.89239.log.gz 2024-06-29T16:01:57.883 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46705.log 2024-06-29T16:01:57.883 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.103209.log.gz 2024-06-29T16:01:57.883 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95779.log 2024-06-29T16:01:57.883 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100307.log 2024-06-29T16:01:57.883 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51232.log.gz 2024-06-29T16:01:57.883 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46705.log: /var/log/ceph/ceph-client.admin.95779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87407.log 2024-06-29T16:01:57.883 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46705.log.gz 2024-06-29T16:01:57.884 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100307.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95779.log.gz 2024-06-29T16:01:57.884 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100307.log.gz 2024-06-29T16:01:57.884 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92845.log 2024-06-29T16:01:57.884 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44941.log 2024-06-29T16:01:57.884 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87407.log: /var/log/ceph/ceph-client.admin.92845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84745.log 2024-06-29T16:01:57.884 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87407.log.gz 2024-06-29T16:01:57.884 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92845.log.gz 2024-06-29T16:01:57.885 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69022.log 2024-06-29T16:01:57.885 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44941.log.gz 2024-06-29T16:01:57.885 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66177.log 2024-06-29T16:01:57.885 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.84745.log.gz 2024-06-29T16:01:57.885 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101310.log 2024-06-29T16:01:57.885 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69022.log.gz 2024-06-29T16:01:57.886 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97575.log 2024-06-29T16:01:57.886 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66177.log.gz 2024-06-29T16:01:57.886 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52580.log/var/log/ceph/ceph-client.admin.101310.log: 2024-06-29T16:01:57.886 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97575.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92109.log 2024-06-29T16:01:57.886 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.101310.log.gz 2024-06-29T16:01:57.886 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97575.log.gz 2024-06-29T16:01:57.886 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58868.log 2024-06-29T16:01:57.886 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52580.log: /var/log/ceph/ceph-client.admin.92109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78619.log 2024-06-29T16:01:57.887 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52580.log.gz 2024-06-29T16:01:57.887 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.92109.log.gz 2024-06-29T16:01:57.887 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104609.log 2024-06-29T16:01:57.887 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105533.log 2024-06-29T16:01:57.887 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58868.log.gz 2024-06-29T16:01:57.887 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104609.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78619.log.gz 2024-06-29T16:01:57.887 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48792.log 2024-06-29T16:01:57.887 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.104609.log.gz 2024-06-29T16:01:57.888 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77926.log 2024-06-29T16:01:57.888 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105533.log: /var/log/ceph/ceph-client.admin.48792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98331.log 2024-06-29T16:01:57.888 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105533.log.gz 2024-06-29T16:01:57.888 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.48792.log.gz 2024-06-29T16:01:57.888 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77926.log: 84.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83216.log 2024-06-29T16:01:57.888 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77926.log.gz 2024-06-29T16:01:57.888 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113154.log 2024-06-29T16:01:57.889 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.98331.log.gz 2024-06-29T16:01:57.889 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.83216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97533.log 2024-06-29T16:01:57.889 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.83216.log.gz 2024-06-29T16:01:57.889 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.113154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75594.log 2024-06-29T16:01:57.889 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113154.log.gz 2024-06-29T16:01:57.889 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65445.log 2024-06-29T16:01:57.889 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97533.log: /var/log/ceph/ceph-client.admin.75594.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.97533.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69871.log 2024-06-29T16:01:57.889 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.890 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75594.log.gz 2024-06-29T16:01:57.890 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5/var/log/ceph/ceph-client.admin.65445.log: --verbose -- /var/log/ceph/ceph-client.admin.54351.log 2024-06-29T16:01:57.890 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69871.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65445.log.gz 2024-06-29T16:01:57.890 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68896.log 2024-06-29T16:01:57.890 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69871.log.gz 2024-06-29T16:01:57.890 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54351.log: /var/log/ceph/ceph-client.admin.71778.log 2024-06-29T16:01:57.890 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68896.log: gzip -5 --verbose -- 87.0% /var/log/ceph/ceph-client.admin.76512.log 2024-06-29T16:01:57.891 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54351.log.gz 2024-06-29T16:01:57.891 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68896.log.gz 2024-06-29T16:01:57.891 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45960.log 2024-06-29T16:01:57.891 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71778.log.gz 2024-06-29T16:01:57.891 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75071.log 2024-06-29T16:01:57.891 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76512.log.gz 2024-06-29T16:01:57.891 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93228.log 2024-06-29T16:01:57.891 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45960.log: /var/log/ceph/ceph-client.admin.75071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108238.log 2024-06-29T16:01:57.892 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45960.log.gz 2024-06-29T16:01:57.892 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93228.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75071.log.gz 2024-06-29T16:01:57.892 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93228.log.gz 2024-06-29T16:01:57.892 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97182.log 2024-06-29T16:01:57.892 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93354.log 2024-06-29T16:01:57.892 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108238.log: /var/log/ceph/ceph-client.admin.97182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70564.log 2024-06-29T16:01:57.892 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.108238.log.gz 2024-06-29T16:01:57.893 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.97182.log.gz 2024-06-29T16:01:57.893 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45962.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.93354.log.gz 2024-06-29T16:01:57.893 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.893 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110000.log 2024-06-29T16:01:57.893 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70564.log.gz 2024-06-29T16:01:57.893 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86508.log 2024-06-29T16:01:57.893 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.45962.log.gz 2024-06-29T16:01:57.893 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112370.log 2024-06-29T16:01:57.894 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62454.log 2024-06-29T16:01:57.894 INFO:teuthology.orchestra.run.smithi081.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.110000.log.gz 2024-06-29T16:01:57.894 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112370.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.86508.log.gz 2024-06-29T16:01:57.894 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112370.log.gz 2024-06-29T16:01:57.894 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106032.log 2024-06-29T16:01:57.894 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101779.log 2024-06-29T16:01:57.894 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62454.log.gz 2024-06-29T16:01:57.894 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106032.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106032.log.gz 2024-06-29T16:01:57.895 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104525.log 2024-06-29T16:01:57.895 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51910.log 2024-06-29T16:01:57.895 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101779.log.gz 2024-06-29T16:01:57.895 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111926.log 2024-06-29T16:01:57.895 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104525.log.gz 2024-06-29T16:01:57.895 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.51910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51910.log.gz 2024-06-29T16:01:57.895 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47400.log 2024-06-29T16:01:57.895 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111926.log: gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.111926.log.gz 2024-06-29T16:01:57.896 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49863.log 2024-06-29T16:01:57.896 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44773.log 2024-06-29T16:01:57.896 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47400.log.gz 2024-06-29T16:01:57.896 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49863.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49863.log.gz 2024-06-29T16:01:57.896 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67152.log 2024-06-29T16:01:57.896 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109660.log 2024-06-29T16:01:57.896 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44773.log.gz 2024-06-29T16:01:57.897 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109525.log 2024-06-29T16:01:57.897 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67152.log.gz 2024-06-29T16:01:57.897 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58486.log 2024-06-29T16:01:57.897 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109660.log.gz 2024-06-29T16:01:57.897 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109525.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.109525.log.gz 2024-06-29T16:01:57.897 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105804.log 2024-06-29T16:01:57.897 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100433.log 2024-06-29T16:01:57.897 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6%/var/log/ceph/ceph-client.admin.105804.log: -- replaced with /var/log/ceph/ceph-client.admin.58486.log.gz 2024-06-29T16:01:57.898 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62716.log 2024-06-29T16:01:57.898 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.105804.log.gz 2024-06-29T16:01:57.898 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63341.log 2024-06-29T16:01:57.898 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100433.log: /var/log/ceph/ceph-client.admin.62716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80786.log 2024-06-29T16:01:57.898 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63341.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62716.log.gz 2024-06-29T16:01:57.898 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100433.log.gz 2024-06-29T16:01:57.898 INFO:teuthology.orchestra.run.smithi081.stderr:gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63341.log.gz 2024-06-29T16:01:57.899 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.84955.log 2024-06-29T16:01:57.899 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80786.log: gzip -5 --verbose -- 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80786.log.gz 2024-06-29T16:01:57.899 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.37634.log 2024-06-29T16:01:57.899 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84955.log: gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84955.log.gz 2024-06-29T16:01:57.899 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.63299.log 2024-06-29T16:01:57.899 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37634.log.gz 2024-06-29T16:01:57.899 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88263.log 2024-06-29T16:01:57.900 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105720.log 2024-06-29T16:01:57.900 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63299.log.gz 2024-06-29T16:01:57.900 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100643.log 2024-06-29T16:01:57.900 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88263.log.gz 2024-06-29T16:01:57.900 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63765.log 2024-06-29T16:01:57.900 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103123.log 2024-06-29T16:01:57.900 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100643.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.105720.log.gz 2024-06-29T16:01:57.901 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100643.log.gz 2024-06-29T16:01:57.901 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63765.log: gzip -5 --verbose -- 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63765.log.gz 2024-06-29T16:01:57.901 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.91553.log 2024-06-29T16:01:57.901 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108700.log 2024-06-29T16:01:57.901 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103123.log.gz 2024-06-29T16:01:57.901 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54292.log 2024-06-29T16:01:57.901 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91553.log.gz 2024-06-29T16:01:57.901 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55155.log 2024-06-29T16:01:57.902 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108700.log.gz 2024-06-29T16:01:57.902 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44349.log 2024-06-29T16:01:57.902 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54292.log.gz 2024-06-29T16:01:57.902 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104168.log 2024-06-29T16:01:57.902 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.55155.log.gz 2024-06-29T16:01:57.902 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70983.log 2024-06-29T16:01:57.902 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.44349.log.gz 2024-06-29T16:01:57.903 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49233.log 2024-06-29T16:01:57.903 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.104168.log.gz 2024-06-29T16:01:57.903 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67026.log 2024-06-29T16:01:57.903 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70983.log.gz 2024-06-29T16:01:57.903 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69955.log 2024-06-29T16:01:57.903 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49233.log.gz 2024-06-29T16:01:57.903 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70414.log 2024-06-29T16:01:57.903 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67026.log.gz 2024-06-29T16:01:57.904 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110333.log 2024-06-29T16:01:57.904 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69955.log: /var/log/ceph/ceph-client.admin.70414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52462.log 2024-06-29T16:01:57.904 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69955.log.gz 2024-06-29T16:01:57.904 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.70414.log.gz 2024-06-29T16:01:57.904 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67654.log 2024-06-29T16:01:57.904 INFO:teuthology.orchestra.run.smithi081.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.110333.log.gz 2024-06-29T16:01:57.904 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62842.log 2024-06-29T16:01:57.904 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52462.log.gz 2024-06-29T16:01:57.905 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92025.log 2024-06-29T16:01:57.905 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67654.log.gz 2024-06-29T16:01:57.905 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107602.log 2024-06-29T16:01:57.905 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.62842.log.gz 2024-06-29T16:01:57.905 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76894.log 2024-06-29T16:01:57.905 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92025.log: /var/log/ceph/ceph-client.admin.107602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75780.log 2024-06-29T16:01:57.905 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92025.log.gz 2024-06-29T16:01:57.905 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.107602.log.gz 2024-06-29T16:01:57.906 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55418.log 2024-06-29T16:01:57.906 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2%/var/log/ceph/ceph-client.admin.75780.log: -- replaced with /var/log/ceph/ceph-client.admin.76894.log.gz 2024-06-29T16:01:57.906 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51722.log 2024-06-29T16:01:57.906 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75780.log.gz 2024-06-29T16:01:57.906 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96713.log 2024-06-29T16:01:57.906 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55418.log: /var/log/ceph/ceph-client.admin.51722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110066.log 2024-06-29T16:01:57.906 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.55418.log.gz/var/log/ceph/ceph-client.admin.96713.log: 2024-06-29T16:01:57.906 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51722.log.gz 2024-06-29T16:01:57.907 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96713.log.gz 2024-06-29T16:01:57.907 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89932.log 2024-06-29T16:01:57.907 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112843.log 2024-06-29T16:01:57.907 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.110066.log.gz 2024-06-29T16:01:57.907 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106923.log 87.0% -- replaced with /var/log/ceph/ceph-client.admin.89932.log.gz 2024-06-29T16:01:57.907 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.907 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48204.log 2024-06-29T16:01:57.908 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112843.log.gz 2024-06-29T16:01:57.908 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51033.log 2024-06-29T16:01:57.908 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.106923.log.gz 2024-06-29T16:01:57.908 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45959.log 2024-06-29T16:01:57.908 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48204.log.gz 2024-06-29T16:01:57.908 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50573.log 2024-06-29T16:01:57.908 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108616.log 2024-06-29T16:01:57.908 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45959.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51033.log.gz 2024-06-29T16:01:57.908 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45959.log.gz 2024-06-29T16:01:57.909 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88987.log 2024-06-29T16:01:57.909 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50573.log.gz 2024-06-29T16:01:57.909 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103253.log 2024-06-29T16:01:57.909 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108616.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89029.log 2024-06-29T16:01:57.909 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108616.log.gz 2024-06-29T16:01:57.909 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88987.log: /var/log/ceph/ceph-client.admin.103253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50073.log 2024-06-29T16:01:57.909 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2%/var/log/ceph/ceph-client.admin.89029.log: -- replaced with /var/log/ceph/ceph-client.admin.103253.log.gz 2024-06-29T16:01:57.909 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88987.log.gz 2024-06-29T16:01:57.910 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51488.log 2024-06-29T16:01:57.910 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89029.log.gz 2024-06-29T16:01:57.910 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112717.log 2024-06-29T16:01:57.910 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50073.log.gz 2024-06-29T16:01:57.910 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71274.log 2024-06-29T16:01:57.910 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51488.log: /var/log/ceph/ceph-client.admin.112717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68374.log 2024-06-29T16:01:57.910 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112717.log.gz 2024-06-29T16:01:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71274.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.51488.log.gz 2024-06-29T16:01:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109012.log 2024-06-29T16:01:57.911 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71274.log.gz 2024-06-29T16:01:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.68374.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37704.log 2024-06-29T16:01:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109012.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68374.log.gz 2024-06-29T16:01:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106965.log 2024-06-29T16:01:57.911 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109012.log.gz 2024-06-29T16:01:57.911 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44857.log 2024-06-29T16:01:57.912 INFO:teuthology.orchestra.run.smithi081.stderr: 76.7% -- replaced with /var/log/ceph/ceph-client.admin.37704.log.gz 2024-06-29T16:01:57.912 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45625.log 2024-06-29T16:01:57.912 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106965.log: /var/log/ceph/ceph-client.admin.44857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48372.log 2024-06-29T16:01:57.912 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106965.log.gz 2024-06-29T16:01:57.912 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44857.log.gz 2024-06-29T16:01:57.912 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78745.log 2024-06-29T16:01:57.912 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87502.log 2024-06-29T16:01:57.912 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45625.log.gz 2024-06-29T16:01:57.913 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2%/var/log/ceph/ceph-client.admin.78745.log: -- replaced with /var/log/ceph/ceph-client.admin.48372.log.gz 2024-06-29T16:01:57.913 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78745.log.gz 2024-06-29T16:01:57.913 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91511.log 2024-06-29T16:01:57.913 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73395.log 2024-06-29T16:01:57.913 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87502.log.gz 2024-06-29T16:01:57.913 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70081.log 2024-06-29T16:01:57.913 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91511.log.gz 2024-06-29T16:01:57.913 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64508.log 2024-06-29T16:01:57.914 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.73395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85478.log 2024-06-29T16:01:57.914 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70081.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73395.log.gz 2024-06-29T16:01:57.914 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64508.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64508.log.gz 2024-06-29T16:01:57.914 INFO:teuthology.orchestra.run.smithi081.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.70081.log.gz 2024-06-29T16:01:57.914 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72724.log 2024-06-29T16:01:57.914 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56733.log 2024-06-29T16:01:57.914 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.85478.log.gz 2024-06-29T16:01:57.915 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75864.log 2024-06-29T16:01:57.915 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72724.log.gz 2024-06-29T16:01:57.915 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80369.log 2024-06-29T16:01:57.915 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.56733.log.gz 2024-06-29T16:01:57.915 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101268.log 2024-06-29T16:01:57.915 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75864.log: /var/log/ceph/ceph-client.admin.80369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108532.log 2024-06-29T16:01:57.915 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101268.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80369.log.gz 2024-06-29T16:01:57.915 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.75864.log.gz 2024-06-29T16:01:57.916 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101268.log.gz 2024-06-29T16:01:57.916 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106415.log 2024-06-29T16:01:57.916 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105113.log 2024-06-29T16:01:57.916 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106415.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50696.log 2024-06-29T16:01:57.916 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108532.log.gz 2024-06-29T16:01:57.916 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.106415.log.gz 2024-06-29T16:01:57.916 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104567.log 2024-06-29T16:01:57.916 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105113.log: /var/log/ceph/ceph-client.admin.50696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57575.log 2024-06-29T16:01:57.916 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.105113.log.gz 2024-06-29T16:01:57.917 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104567.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50696.log.gz 2024-06-29T16:01:57.917 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose 87.1% -- replaced with /var/log/ceph/ceph-client.admin.104567.log.gz 2024-06-29T16:01:57.917 INFO:teuthology.orchestra.run.smithi081.stderr: -- /var/log/ceph/ceph-client.admin.99351.log 2024-06-29T16:01:57.917 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57575.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57575.log.gz 2024-06-29T16:01:57.917 INFO:teuthology.orchestra.run.smithi081.stderr: -- /var/log/ceph/ceph-client.admin.71232.log 2024-06-29T16:01:57.917 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91815.log 2024-06-29T16:01:57.917 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.99351.log.gz 2024-06-29T16:01:57.917 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104735.log 2024-06-29T16:01:57.918 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71232.log.gz 2024-06-29T16:01:57.918 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95999.log 2024-06-29T16:01:57.918 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.91815.log.gz 2024-06-29T16:01:57.918 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106629.log 2024-06-29T16:01:57.918 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.104735.log.gz 2024-06-29T16:01:57.918 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.95999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108280.log 2024-06-29T16:01:57.918 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95999.log.gz 2024-06-29T16:01:57.919 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77726.log 2024-06-29T16:01:57.919 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106629.log: /var/log/ceph/ceph-client.admin.108280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48414.log 2024-06-29T16:01:57.919 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106629.log.gz 2024-06-29T16:01:57.919 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108280.log.gz 2024-06-29T16:01:57.919 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59474.log 2024-06-29T16:01:57.919 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63723.log 2024-06-29T16:01:57.919 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.77726.log.gz 2024-06-29T16:01:57.919 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.59474.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48414.log.gz 2024-06-29T16:01:57.919 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66135.log 2024-06-29T16:01:57.920 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59474.log.gz 2024-06-29T16:01:57.920 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38041.log 2024-06-29T16:01:57.920 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.63723.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.49737.log 2024-06-29T16:01:57.920 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.920 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66135.log: /var/log/ceph/ceph-client.admin.38041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106207.log 2024-06-29T16:01:57.920 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38041.log.gz 87.9% 2024-06-29T16:01:57.920 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66135.log.gz 2024-06-29T16:01:57.920 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49491.log 2024-06-29T16:01:57.921 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106207.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108448.log 2024-06-29T16:01:57.921 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49737.log.gz 2024-06-29T16:01:57.921 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.106207.log.gz 2024-06-29T16:01:57.921 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67762.log 2024-06-29T16:01:57.921 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48876.log 2024-06-29T16:01:57.921 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49491.log.gz 2024-06-29T16:01:57.921 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108448.log.gz 2024-06-29T16:01:57.922 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38679.log 2024-06-29T16:01:57.922 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67762.log.gz 2024-06-29T16:01:57.922 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109395.log 2024-06-29T16:01:57.922 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48876.log.gz 2024-06-29T16:01:57.922 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53955.log 2024-06-29T16:01:57.922 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38679.log: /var/log/ceph/ceph-client.admin.109395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109958.log 2024-06-29T16:01:57.922 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4%/var/log/ceph/ceph-client.admin.53955.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.109395.log.gz 2024-06-29T16:01:57.922 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38679.log.gz 2024-06-29T16:01:57.923 INFO:teuthology.orchestra.run.smithi081.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.53955.log.gz 2024-06-29T16:01:57.923 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99435.log 2024-06-29T16:01:57.923 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109958.log: gzip -5 --verbose 87.9% -- /var/log/ceph/ceph-client.admin.46789.log 2024-06-29T16:01:57.923 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.109958.log.gz 2024-06-29T16:01:57.923 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68770.log 2024-06-29T16:01:57.923 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.99435.log.gz 2024-06-29T16:01:57.923 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111319.log 2024-06-29T16:01:57.923 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46789.log: /var/log/ceph/ceph-client.admin.68770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80327.log 2024-06-29T16:01:57.924 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111319.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68770.log.gz 2024-06-29T16:01:57.924 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66686.log 2024-06-29T16:01:57.924 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111319.log.gz 2024-06-29T16:01:57.924 INFO:teuthology.orchestra.run.smithi081.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.46789.log.gz 2024-06-29T16:01:57.924 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80327.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80327.log.gz 2024-06-29T16:01:57.924 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85268.log 2024-06-29T16:01:57.924 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50199.log 2024-06-29T16:01:57.924 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66686.log: /var/log/ceph/ceph-client.admin.85268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92235.log 2024-06-29T16:01:57.924 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85268.log.gz 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66686.log.gz 2024-06-29T16:01:57.925 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.925 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50199.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50199.log.gz 2024-06-29T16:01:57.925 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87365.log 2024-06-29T16:01:57.925 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63936.log 2024-06-29T16:01:57.925 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92235.log.gz 2024-06-29T16:01:57.925 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112675.log 2024-06-29T16:01:57.925 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.87365.log.gz 2024-06-29T16:01:57.925 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51119.log 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63936.log.gz 2024-06-29T16:01:57.926 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.926 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110578.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112675.log.gz 2024-06-29T16:01:57.926 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.926 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51119.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51119.log.gz 2024-06-29T16:01:57.926 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.92151.log 2024-06-29T16:01:57.926 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112159.log 2024-06-29T16:01:57.926 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.110578.log.gz 2024-06-29T16:01:57.926 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74977.log 2024-06-29T16:01:57.927 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.92151.log.gz/var/log/ceph/ceph-client.admin.112159.log: 2024-06-29T16:01:57.927 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58360.log 2024-06-29T16:01:57.927 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.112159.log.gz 2024-06-29T16:01:57.927 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58131.log/var/log/ceph/ceph-client.admin.74977.log: 2024-06-29T16:01:57.927 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58360.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74977.log.gz 2024-06-29T16:01:57.927 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102243.log 2024-06-29T16:01:57.927 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58360.log.gz 2024-06-29T16:01:57.928 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.58131.log: /var/log/ceph/ceph-client.admin.54528.log 2024-06-29T16:01:57.928 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102243.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58131.log.gz 2024-06-29T16:01:57.928 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102955.log 2024-06-29T16:01:57.928 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102243.log.gz 2024-06-29T16:01:57.928 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109706.log 2024-06-29T16:01:57.928 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54528.log: /var/log/ceph/ceph-client.admin.102955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97767.log 2024-06-29T16:01:57.928 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54528.log.gz 2024-06-29T16:01:57.928 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.102955.log.gz 2024-06-29T16:01:57.929 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99197.log 2024-06-29T16:01:57.929 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0%/var/log/ceph/ceph-client.admin.97767.log: -- replaced with /var/log/ceph/ceph-client.admin.109706.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79776.log 2024-06-29T16:01:57.929 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.929 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97767.log.gz 2024-06-29T16:01:57.929 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99197.log: gzip -5 --verbose 87.2% -- /var/log/ceph/ceph-client.admin.52285.log 2024-06-29T16:01:57.929 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.99197.log.gz 2024-06-29T16:01:57.929 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84871.log 2024-06-29T16:01:57.929 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79776.log.gz 2024-06-29T16:01:57.930 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75533.log 2024-06-29T16:01:57.930 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52285.log: /var/log/ceph/ceph-client.admin.84871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93814.log 2024-06-29T16:01:57.930 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52285.log.gz 2024-06-29T16:01:57.930 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.84871.log.gz 2024-06-29T16:01:57.930 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.75533.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.69913.log 2024-06-29T16:01:57.930 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1%/var/log/ceph/ceph-client.admin.93814.log: -- replaced with /var/log/ceph/ceph-client.admin.75533.log.gz 2024-06-29T16:01:57.930 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47838.log 2024-06-29T16:01:57.931 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.93814.log.gz 2024-06-29T16:01:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88649.log 2024-06-29T16:01:57.931 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.69913.log.gz 2024-06-29T16:01:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37852.log 2024-06-29T16:01:57.931 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47838.log.gz 2024-06-29T16:01:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108784.log 2024-06-29T16:01:57.931 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88649.log.gz 2024-06-29T16:01:57.931 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37852.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112548.log 2024-06-29T16:01:57.932 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37852.log.gz 2024-06-29T16:01:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86883.log 2024-06-29T16:01:57.932 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.108784.log.gz 2024-06-29T16:01:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105197.log 2024-06-29T16:01:57.932 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.112548.log.gz 2024-06-29T16:01:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64817.log 2024-06-29T16:01:57.932 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86883.log.gz 2024-06-29T16:01:57.932 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69446.log 2024-06-29T16:01:57.933 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105197.log.gz 2024-06-29T16:01:57.933 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85310.log 2024-06-29T16:01:57.933 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64817.log.gz 2024-06-29T16:01:57.933 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67110.log 2024-06-29T16:01:57.933 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69446.log.gz 2024-06-29T16:01:57.933 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66261.log 2024-06-29T16:01:57.933 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85310.log: /var/log/ceph/ceph-client.admin.67110.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78955.log 2024-06-29T16:01:57.933 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.85310.log.gz 2024-06-29T16:01:57.934 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67110.log.gz 2024-06-29T16:01:57.934 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84913.log 2024-06-29T16:01:57.934 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66261.log: /var/log/ceph/ceph-client.admin.78955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69619.log 2024-06-29T16:01:57.934 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66261.log.gz 2024-06-29T16:01:57.934 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84913.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78955.log.gz 2024-06-29T16:01:57.934 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87722.log 86.6% -- replaced with /var/log/ceph/ceph-client.admin.84913.log.gz 2024-06-29T16:01:57.934 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.934 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69619.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69619.log.gz 2024-06-29T16:01:57.935 INFO:teuthology.orchestra.run.smithi081.stderr: -- /var/log/ceph/ceph-client.admin.86084.log 2024-06-29T16:01:57.935 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75155.log 2024-06-29T16:01:57.935 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.87722.log.gz 2024-06-29T16:01:57.935 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53009.log 2024-06-29T16:01:57.935 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86084.log.gz 2024-06-29T16:01:57.935 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88389.log 2024-06-29T16:01:57.935 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75155.log: /var/log/ceph/ceph-client.admin.53009.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.75155.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111697.log 2024-06-29T16:01:57.936 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.936 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53009.log.gz 2024-06-29T16:01:57.936 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57795.log 2024-06-29T16:01:57.936 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88389.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88389.log.gz 2024-06-29T16:01:57.936 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose --/var/log/ceph/ceph-client.admin.111697.log: /var/log/ceph/ceph-client.admin.44561.log 2024-06-29T16:01:57.936 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111697.log.gz 2024-06-29T16:01:57.936 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111739.log 2024-06-29T16:01:57.936 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98375.log 2024-06-29T16:01:57.937 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44561.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57795.log.gz 2024-06-29T16:01:57.937 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44561.log.gz 2024-06-29T16:01:57.937 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98855.log 2024-06-29T16:01:57.937 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.111739.log.gz 2024-06-29T16:01:57.937 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50781.log 2024-06-29T16:01:57.937 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98375.log.gz 2024-06-29T16:01:57.937 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79436.log 2024-06-29T16:01:57.937 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.98855.log.gz 2024-06-29T16:01:57.938 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112969.log 2024-06-29T16:01:57.938 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50781.log.gz 2024-06-29T16:01:57.938 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62674.log/var/log/ceph/ceph-client.admin.79436.log: 2024-06-29T16:01:57.938 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112969.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79436.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65865.log 2024-06-29T16:01:57.938 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.938 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112969.log.gz 2024-06-29T16:01:57.938 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59934.log 2024-06-29T16:01:57.938 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62674.log: /var/log/ceph/ceph-client.admin.65865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45620.log 2024-06-29T16:01:57.939 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62674.log.gz 2024-06-29T16:01:57.939 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3%/var/log/ceph/ceph-client.admin.59934.log: -- replaced with /var/log/ceph/ceph-client.admin.65865.log.gz 2024-06-29T16:01:57.939 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- 86.8% -- replaced with /var/log/ceph/ceph-client.admin.59934.log.gz 2024-06-29T16:01:57.939 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.54174.log 2024-06-29T16:01:57.939 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45620.log: gzip -5 --verbose 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45620.log.gz 2024-06-29T16:01:57.939 INFO:teuthology.orchestra.run.smithi081.stderr: -- /var/log/ceph/ceph-client.admin.106839.log 2024-06-29T16:01:57.939 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51446.log 2024-06-29T16:01:57.939 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54174.log.gz 2024-06-29T16:01:57.940 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66093.log 2024-06-29T16:01:57.940 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106839.log.gz 2024-06-29T16:01:57.940 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63850.log 2024-06-29T16:01:57.940 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46201.log 2024-06-29T16:01:57.940 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51446.log.gz 2024-06-29T16:01:57.940 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66093.log: /var/log/ceph/ceph-client.admin.63850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80117.log 2024-06-29T16:01:57.940 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9%/var/log/ceph/ceph-client.admin.46201.log: -- replaced with /var/log/ceph/ceph-client.admin.66093.log.gz 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63850.log.gz 2024-06-29T16:01:57.940 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.941 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46201.log.gz 2024-06-29T16:01:57.941 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99919.log 2024-06-29T16:01:57.941 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62968.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80117.log.gz 2024-06-29T16:01:57.941 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.941 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59849.log 2024-06-29T16:01:57.941 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.99919.log.gz 2024-06-29T16:01:57.941 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44815.log 2024-06-29T16:01:57.941 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62968.log.gz 2024-06-29T16:01:57.942 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106671.log 2024-06-29T16:01:57.942 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.59849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109187.log 2024-06-29T16:01:57.942 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44815.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59849.log.gz 2024-06-29T16:01:57.942 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44815.log.gz 2024-06-29T16:01:57.942 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89515.log 2024-06-29T16:01:57.942 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.106671.log.gz 2024-06-29T16:01:57.942 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.109187.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.64064.log 2024-06-29T16:01:57.942 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109187.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49317.log 2024-06-29T16:01:57.943 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.943 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103620.log 2024-06-29T16:01:57.943 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89515.log.gz 2024-06-29T16:01:57.943 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64064.log: /var/log/ceph/ceph-client.admin.49317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48330.log 2024-06-29T16:01:57.943 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64064.log.gz 2024-06-29T16:01:57.943 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49317.log.gz 2024-06-29T16:01:57.943 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103620.log: gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/ceph-client.admin.103620.log.gz 2024-06-29T16:01:57.943 INFO:teuthology.orchestra.run.smithi081.stderr: -- /var/log/ceph/ceph-client.admin.57277.log 2024-06-29T16:01:57.944 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48330.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48330.log.gz 2024-06-29T16:01:57.944 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62088.log 2024-06-29T16:01:57.944 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105846.log 2024-06-29T16:01:57.944 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57277.log.gz 2024-06-29T16:01:57.944 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71078.log 2024-06-29T16:01:57.944 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62088.log.gz 2024-06-29T16:01:57.944 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.105846.log: -- /var/log/ceph/ceph-client.admin.62590.log 2024-06-29T16:01:57.944 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71078.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.105846.log.gz 2024-06-29T16:01:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67612.log 2024-06-29T16:01:57.945 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.71078.log.gz 2024-06-29T16:01:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101226.log 2024-06-29T16:01:57.945 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62590.log: /var/log/ceph/ceph-client.admin.67612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100391.log 2024-06-29T16:01:57.945 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6%/var/log/ceph/ceph-client.admin.101226.log: -- replaced with /var/log/ceph/ceph-client.admin.62590.log.gz 2024-06-29T16:01:57.945 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.67612.log.gz 2024-06-29T16:01:57.945 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102913.log 2024-06-29T16:01:57.945 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.101226.log.gz 2024-06-29T16:01:57.946 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58528.log 2024-06-29T16:01:57.946 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100391.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67975.log 2024-06-29T16:01:57.946 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.946 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102913.log: /var/log/ceph/ceph-client.admin.58528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95915.log 2024-06-29T16:01:57.946 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.102913.log.gz 2024-06-29T16:01:57.946 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67975.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58528.log.gz 2024-06-29T16:01:57.946 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92803.log 2024-06-29T16:01:57.946 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67975.log.gz 2024-06-29T16:01:57.947 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.95915.log: /var/log/ceph/ceph-client.admin.78997.log 2024-06-29T16:01:57.947 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92803.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.95915.log.gz 2024-06-29T16:01:57.947 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56207.log 2024-06-29T16:01:57.947 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92803.log.gz 2024-06-29T16:01:57.947 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52031.log 2024-06-29T16:01:57.947 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.78997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87460.log 2024-06-29T16:01:57.947 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56207.log: 87.5%/var/log/ceph/ceph-client.admin.52031.log: -- replaced with /var/log/ceph/ceph-client.admin.78997.log.gz 2024-06-29T16:01:57.947 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.56207.log.gz 2024-06-29T16:01:57.948 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52031.log.gz 2024-06-29T16:01:57.948 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84787.log 2024-06-29T16:01:57.948 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87460.log: gzip -5 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87460.log.gz 2024-06-29T16:01:57.948 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84829.log 2024-06-29T16:01:57.948 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72809.log 2024-06-29T16:01:57.948 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.84787.log.gz 2024-06-29T16:01:57.948 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76288.log 2024-06-29T16:01:57.948 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.84829.log.gz 2024-06-29T16:01:57.949 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98287.log 2024-06-29T16:01:57.949 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72809.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.72809.log.gz -5 2024-06-29T16:01:57.949 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44433.log 2024-06-29T16:01:57.949 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76288.log: gzip/var/log/ceph/ceph-client.admin.98287.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.59348.log 2024-06-29T16:01:57.949 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44433.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98287.log.gz 2024-06-29T16:01:57.949 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76288.log.gz 2024-06-29T16:01:57.949 INFO:teuthology.orchestra.run.smithi081.stderr:gzip 87.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.112759.log 2024-06-29T16:01:57.949 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44433.log.gz 2024-06-29T16:01:57.949 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.59348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74586.log 2024-06-29T16:01:57.950 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59348.log.gz 2024-06-29T16:01:57.950 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59134.log 2024-06-29T16:01:57.950 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112759.log: /var/log/ceph/ceph-client.admin.74586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62496.log 2024-06-29T16:01:57.950 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.59134.log: 87.1% 87.0% -- replaced with /var/log/ceph/ceph-client.admin.112759.log.gz 2024-06-29T16:01:57.950 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74586.log.gz 2024-06-29T16:01:57.950 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59134.log.gz 2024-06-29T16:01:57.950 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79861.log 2024-06-29T16:01:57.950 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45290.log 2024-06-29T16:01:57.951 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62496.log: /var/log/ceph/ceph-client.admin.79861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69281.log 2024-06-29T16:01:57.951 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62496.log.gz 2024-06-29T16:01:57.951 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79861.log.gz 2024-06-29T16:01:57.951 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96209.log 2024-06-29T16:01:57.951 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.45290.log.gz 2024-06-29T16:01:57.951 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79352.log 2024-06-29T16:01:57.951 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69281.log: /var/log/ceph/ceph-client.admin.96209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75696.log 2024-06-29T16:01:57.952 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69281.log.gz 2024-06-29T16:01:57.952 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5%/var/log/ceph/ceph-client.admin.79352.log: -- replaced with /var/log/ceph/ceph-client.admin.96209.log.gz 2024-06-29T16:01:57.952 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79352.log.gz 2024-06-29T16:01:57.952 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65256.log 2024-06-29T16:01:57.952 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98687.log 2024-06-29T16:01:57.952 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75696.log: /var/log/ceph/ceph-client.admin.65256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109790.log 2024-06-29T16:01:57.952 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75696.log.gz 2024-06-29T16:01:57.952 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65256.log.gz 2024-06-29T16:01:57.952 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112285.log 2024-06-29T16:01:57.953 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4%/var/log/ceph/ceph-client.admin.109790.log: -- replaced with /var/log/ceph/ceph-client.admin.98687.log.gz 2024-06-29T16:01:57.953 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69362.log 2024-06-29T16:01:57.953 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109790.log.gz 2024-06-29T16:01:57.953 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76043.log 2024-06-29T16:01:57.953 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.112285.log.gz 2024-06-29T16:01:57.953 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.110254.log 2024-06-29T16:01:57.953 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69362.log.gz 2024-06-29T16:01:57.953 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93898.log 2024-06-29T16:01:57.954 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76043.log.gz 2024-06-29T16:01:57.954 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.110254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.110254.log.gz 2024-06-29T16:01:57.954 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59770.log 2024-06-29T16:01:57.954 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.93898.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.69829.log 2024-06-29T16:01:57.954 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.59770.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.93898.log.gz 2024-06-29T16:01:57.954 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99047.log 2024-06-29T16:01:57.954 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.0.59770.log.gz 2024-06-29T16:01:57.955 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86126.log 2024-06-29T16:01:57.955 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95737.log 2024-06-29T16:01:57.955 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69829.log.gz 2024-06-29T16:01:57.955 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99047.log: /var/log/ceph/ceph-client.admin.86126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106457.log 2024-06-29T16:01:57.955 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% 87.2%/var/log/ceph/ceph-client.admin.95737.log: -- replaced with /var/log/ceph/ceph-client.admin.99047.log.gz 2024-06-29T16:01:57.955 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86126.log.gz 2024-06-29T16:01:57.955 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.95737.log.gz 2024-06-29T16:01:57.955 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76936.log 2024-06-29T16:01:57.956 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.106457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85394.log 2024-06-29T16:01:57.956 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76936.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.106457.log.gz 2024-06-29T16:01:57.956 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90907.log 2024-06-29T16:01:57.956 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48498.log 2024-06-29T16:01:57.956 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85394.log: 90.9% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85394.log.gz 2024-06-29T16:01:57.956 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.90907.log: -- replaced with /var/log/ceph/ceph-client.admin.76936.log.gz 2024-06-29T16:01:57.956 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.90907.log.gz 2024-06-29T16:01:57.956 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59582.log 2024-06-29T16:01:57.957 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63215.log 2024-06-29T16:01:57.957 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48498.log.gz 2024-06-29T16:01:57.957 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.59582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107340.log 2024-06-29T16:01:57.957 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59582.log.gz 2024-06-29T16:01:57.957 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95873.log 2024-06-29T16:01:57.957 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63215.log.gz 2024-06-29T16:01:57.957 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73062.log/var/log/ceph/ceph-client.admin.107340.log: 2024-06-29T16:01:57.957 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.95873.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100974.log 2024-06-29T16:01:57.958 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107340.log.gz 2024-06-29T16:01:57.958 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.95873.log.gz 2024-06-29T16:01:57.958 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.73062.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.68500.log 2024-06-29T16:01:57.958 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.100974.log: /var/log/ceph/ceph-client.admin.58623.log 2024-06-29T16:01:57.958 INFO:teuthology.orchestra.run.smithi081.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.73062.log.gz 2024-06-29T16:01:57.958 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.100974.log.gz 2024-06-29T16:01:57.958 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111655.log 2024-06-29T16:01:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72682.log 2024-06-29T16:01:57.959 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.68500.log.gz 2024-06-29T16:01:57.959 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5%/var/log/ceph/ceph-client.admin.111655.log: -- replaced with /var/log/ceph/ceph-client.admin.58623.log.gz 2024-06-29T16:01:57.959 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111655.log.gz 2024-06-29T16:01:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89155.log 2024-06-29T16:01:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72682.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72682.log.gz 2024-06-29T16:01:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99477.log 2024-06-29T16:01:57.959 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98645.log 2024-06-29T16:01:57.960 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105239.log 2024-06-29T16:01:57.960 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89155.log.gz 2024-06-29T16:01:57.960 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54469.log 2024-06-29T16:01:57.960 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.99477.log.gz 2024-06-29T16:01:57.960 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98645.log: /var/log/ceph/ceph-client.admin.105239.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104210.log 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105239.log.gz 2024-06-29T16:01:57.960 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.960 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54469.log: -- replaced with /var/log/ceph/ceph-client.admin.98645.log.gz 2024-06-29T16:01:57.960 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54469.log.gz 2024-06-29T16:01:57.961 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.52757.log 2024-06-29T16:01:57.961 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104210.log: gzip -5 --verbose -- 87.2% -- replaced with /var/log/ceph/ceph-client.admin.104210.log.gz 2024-06-29T16:01:57.961 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.73545.log 2024-06-29T16:01:57.961 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52757.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52757.log.gz 2024-06-29T16:01:57.961 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.101478.log 2024-06-29T16:01:57.961 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.73545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68154.log 87.0% -- replaced with /var/log/ceph/ceph-client.admin.73545.log.gz 2024-06-29T16:01:57.961 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.962 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53913.log 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101478.log.gz 2024-06-29T16:01:57.962 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.962 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101562.log 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68154.log.gz 2024-06-29T16:01:57.962 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.962 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48666.log 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53913.log.gz 2024-06-29T16:01:57.962 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.962 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101562.log: gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101562.log.gz 2024-06-29T16:01:57.963 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63257.log 2024-06-29T16:01:57.963 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48666.log: gzip -5 --verbose 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48666.log.gz 2024-06-29T16:01:57.963 INFO:teuthology.orchestra.run.smithi081.stderr: -- /var/log/ceph/ceph-client.admin.62884.log 2024-06-29T16:01:57.963 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112927.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63257.log.gz 2024-06-29T16:01:57.963 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.963 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88034.log 2024-06-29T16:01:57.963 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62884.log.gz 2024-06-29T16:01:57.963 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80995.log 2024-06-29T16:01:57.963 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112927.log.gz 2024-06-29T16:01:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58444.log 2024-06-29T16:01:57.964 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.88034.log.gz 2024-06-29T16:01:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100181.log 2024-06-29T16:01:57.964 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80995.log.gz 2024-06-29T16:01:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67236.log 2024-06-29T16:01:57.964 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.58444.log.gz 2024-06-29T16:01:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85860.log 2024-06-29T16:01:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67804.log 2024-06-29T16:01:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67236.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.100181.log.gz 2024-06-29T16:01:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85860.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67236.log.gz 2024-06-29T16:01:57.965 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85860.log.gz 2024-06-29T16:01:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56470.log 2024-06-29T16:01:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44647.log 2024-06-29T16:01:57.965 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67804.log.gz 2024-06-29T16:01:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.56470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63173.log 86.6% 2024-06-29T16:01:57.965 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56470.log.gz 2024-06-29T16:01:57.965 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109107.log 2024-06-29T16:01:57.966 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44647.log.gz 2024-06-29T16:01:57.966 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.63173.log: /var/log/ceph/ceph-client.admin.62198.log 2024-06-29T16:01:57.966 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109107.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63173.log.gz 2024-06-29T16:01:57.966 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109352.log 2024-06-29T16:01:57.966 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109107.log.gz 2024-06-29T16:01:57.966 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71025.log 2024-06-29T16:01:57.966 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62198.log: /var/log/ceph/ceph-client.admin.109352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104305.log 2024-06-29T16:01:57.966 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62198.log.gz 2024-06-29T16:01:57.967 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3%/var/log/ceph/ceph-client.admin.71025.log: -- replaced with /var/log/ceph/ceph-client.admin.109352.log.gz 2024-06-29T16:01:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75407.log 2024-06-29T16:01:57.967 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71025.log.gz 2024-06-29T16:01:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49533.log 2024-06-29T16:01:57.967 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.104305.log.gz 2024-06-29T16:01:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103346.log 2024-06-29T16:01:57.967 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75407.log: /var/log/ceph/ceph-client.admin.49533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97138.log 2024-06-29T16:01:57.967 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3%/var/log/ceph/ceph-client.admin.103346.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49533.log.gz 2024-06-29T16:01:57.968 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75407.log.gz 2024-06-29T16:01:57.968 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103346.log.gz 2024-06-29T16:01:57.968 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50654.log 2024-06-29T16:01:57.968 INFO:teuthology.orchestra.run.smithi081.stderr:gzip/var/log/ceph/ceph-client.admin.97138.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47985.log 2024-06-29T16:01:57.968 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88431.log 2024-06-29T16:01:57.968 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97138.log.gz 2024-06-29T16:01:57.968 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45286.log 2024-06-29T16:01:57.968 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50654.log.gz 2024-06-29T16:01:57.968 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47985.log: /var/log/ceph/ceph-client.admin.88431.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47985.log.gzgzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88431.log.gz 2024-06-29T16:01:57.969 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.48834.log 2024-06-29T16:01:57.969 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.969 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45286.log: gzip -5 86.8% -- replaced with /var/log/ceph/ceph-client.admin.45286.log.gz 2024-06-29T16:01:57.969 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.100139.log 2024-06-29T16:01:57.969 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77312.log 2024-06-29T16:01:57.969 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48834.log.gz 2024-06-29T16:01:57.969 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100139.log: gzip -5 87.2% -- replaced with /var/log/ceph/ceph-client.admin.100139.log.gz 2024-06-29T16:01:57.969 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.104347.log 2024-06-29T16:01:57.970 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102327.log 2024-06-29T16:01:57.970 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77312.log.gz 2024-06-29T16:01:57.970 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77785.log 2024-06-29T16:01:57.970 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.104347.log.gz 2024-06-29T16:01:57.970 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63010.log 2024-06-29T16:01:57.970 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102327.log: /var/log/ceph/ceph-client.admin.77785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53093.log 2024-06-29T16:01:57.970 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102327.log.gz 2024-06-29T16:01:57.970 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.77785.log.gz 2024-06-29T16:01:57.971 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90536.log 2024-06-29T16:01:57.971 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63010.log.gz 2024-06-29T16:01:57.971 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.53093.log: /var/log/ceph/ceph-client.admin.97277.log 2024-06-29T16:01:57.971 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.90536.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.53093.log.gz 2024-06-29T16:01:57.971 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52073.log 2024-06-29T16:01:57.971 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.90536.log.gz 2024-06-29T16:01:57.971 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99309.log 2024-06-29T16:01:57.971 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97277.log: /var/log/ceph/ceph-client.admin.52073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103388.log 2024-06-29T16:01:57.972 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52073.log.gz 2024-06-29T16:01:57.972 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99309.log: -- replaced with /var/log/ceph/ceph-client.admin.97277.log.gz 2024-06-29T16:01:57.972 INFO:teuthology.orchestra.run.smithi081.stderr:gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.99309.log.gz 2024-06-29T16:01:57.972 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.93060.log 2024-06-29T16:01:57.972 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103388.log: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108322.log 2024-06-29T16:01:57.972 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.103388.log.gz 2024-06-29T16:01:57.972 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97701.log 2024-06-29T16:01:57.972 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59222.log 2024-06-29T16:01:57.973 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.93060.log.gz 2024-06-29T16:01:57.973 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92319.log 2024-06-29T16:01:57.973 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108322.log.gz 2024-06-29T16:01:57.973 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97701.log: /var/log/ceph/ceph-client.admin.59222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100003.log 2024-06-29T16:01:57.973 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59222.log.gz 2024-06-29T16:01:57.973 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92319.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.97701.log.gz 2024-06-29T16:01:57.973 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92319.log.gz 2024-06-29T16:01:57.973 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68602.log 2024-06-29T16:01:57.973 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46369.log 2024-06-29T16:01:57.974 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100003.log.gz 2024-06-29T16:01:57.974 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75197.log 2024-06-29T16:01:57.974 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68602.log.gz 2024-06-29T16:01:57.974 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46369.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46369.log.gz 2024-06-29T16:01:57.974 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.97011.log 2024-06-29T16:01:57.974 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49617.log 2024-06-29T16:01:57.974 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.75197.log.gz 2024-06-29T16:01:57.974 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53703.log 2024-06-29T16:01:57.975 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.97011.log.gz 2024-06-29T16:01:57.975 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45627.log 2024-06-29T16:01:57.975 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49617.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.49617.log.gz 2024-06-29T16:01:57.975 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53703.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51278.log 2024-06-29T16:01:57.975 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.53703.log.gz 2024-06-29T16:01:57.975 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104048.log 2024-06-29T16:01:57.975 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45627.log.gz 2024-06-29T16:01:57.975 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97617.log 2024-06-29T16:01:57.975 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51278.log.gz 2024-06-29T16:01:57.976 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107424.log 2024-06-29T16:01:57.976 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104048.log: /var/log/ceph/ceph-client.admin.97617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46621.log 2024-06-29T16:01:57.976 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97617.log.gz 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104048.log.gz 2024-06-29T16:01:57.976 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.976 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.107424.log: gzip -5 87.1% -- replaced with /var/log/ceph/ceph-client.admin.107424.log.gz 2024-06-29T16:01:57.976 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.76810.log 2024-06-29T16:01:57.976 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89323.log 2024-06-29T16:01:57.977 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46621.log.gz 2024-06-29T16:01:57.977 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70606.log 2024-06-29T16:01:57.977 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.76810.log.gz 2024-06-29T16:01:57.977 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88076.log 2024-06-29T16:01:57.977 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.89323.log.gz 2024-06-29T16:01:57.977 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.70606.log: /var/log/ceph/ceph-client.admin.85226.log 2024-06-29T16:01:57.977 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88076.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70606.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109748.log 2024-06-29T16:01:57.977 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.978 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88076.log.gz 2024-06-29T16:01:57.978 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5/var/log/ceph/ceph-client.admin.85226.log: --verbose -- /var/log/ceph/ceph-client.admin.68332.log 2024-06-29T16:01:57.978 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109748.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.85226.log.gz 2024-06-29T16:01:57.978 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- 87.3% -- replaced with /var/log/ceph/ceph-client.admin.109748.log.gz 2024-06-29T16:01:57.978 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.57438.log 2024-06-29T16:01:57.978 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68332.log: gzip -5 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68332.log.gz 2024-06-29T16:01:57.978 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.85082.log 2024-06-29T16:01:57.978 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84472.log 2024-06-29T16:01:57.978 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57438.log.gz 2024-06-29T16:01:57.979 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53997.log 2024-06-29T16:01:57.979 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.85082.log.gz 2024-06-29T16:01:57.979 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100806.log 2024-06-29T16:01:57.979 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84472.log: /var/log/ceph/ceph-client.admin.53997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93438.log 2024-06-29T16:01:57.979 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.84472.log.gz 2024-06-29T16:01:57.979 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.53997.log.gz 2024-06-29T16:01:57.979 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48246.log 2024-06-29T16:01:57.980 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100806.log.gz 2024-06-29T16:01:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44731.log 2024-06-29T16:01:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93438.log: /var/log/ceph/ceph-client.admin.48246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44087.log 2024-06-29T16:01:57.980 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93438.log.gz 2024-06-29T16:01:57.980 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48246.log.gz 2024-06-29T16:01:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67278.log 2024-06-29T16:01:57.980 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44731.log.gz/var/log/ceph/ceph-client.admin.44087.log: 2024-06-29T16:01:57.980 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103039.log 2024-06-29T16:01:57.981 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44087.log.gz 2024-06-29T16:01:57.981 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57533.log 2024-06-29T16:01:57.981 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67278.log: /var/log/ceph/ceph-client.admin.103039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45963.log 2024-06-29T16:01:57.981 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57533.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103039.log.gz 2024-06-29T16:01:57.981 INFO:teuthology.orchestra.run.smithi081.stderr: 90.8% 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.57533.log.gz -5 2024-06-29T16:01:57.981 INFO:teuthology.orchestra.run.smithi081.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50157.log 2024-06-29T16:01:57.981 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67278.log.gz 2024-06-29T16:01:57.982 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.45963.log: -- /var/log/ceph/ceph-client.admin.105323.log 2024-06-29T16:01:57.982 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50157.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45963.log.gz 2024-06-29T16:01:57.982 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49155.log 2024-06-29T16:01:57.982 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50157.log.gz 2024-06-29T16:01:57.982 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86298.log 2024-06-29T16:01:57.982 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105323.log: /var/log/ceph/ceph-client.admin.49155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46453.log 2024-06-29T16:01:57.982 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4%/var/log/ceph/ceph-client.admin.86298.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49155.log.gz 2024-06-29T16:01:57.983 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.105323.log.gz 2024-06-29T16:01:57.983 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86298.log.gz 2024-06-29T16:01:57.983 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-06-29T16:01:57.983 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46453.log: gzip -5 --verbose -- 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46453.log.gz 2024-06-29T16:01:57.983 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.104651.log 2024-06-29T16:01:57.983 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92193.log 2024-06-29T16:01:57.983 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76852.log 2024-06-29T16:01:57.983 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.104651.log.gz 2024-06-29T16:01:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107923.log 2024-06-29T16:01:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76852.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92193.log.gz 2024-06-29T16:01:57.984 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76852.log.gz 2024-06-29T16:01:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47880.log 2024-06-29T16:01:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.107923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93144.log 2024-06-29T16:01:57.984 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.107923.log.gz 2024-06-29T16:01:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.47880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110417.log 2024-06-29T16:01:57.984 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93144.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47880.log.gz 2024-06-29T16:01:57.985 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93144.log.gz 2024-06-29T16:01:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37825.log 2024-06-29T16:01:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79184.log 2024-06-29T16:01:57.985 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.110417.log.gz 2024-06-29T16:01:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37825.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49905.log 2024-06-29T16:01:57.985 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:57.985 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79184.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79184.log.gz 2024-06-29T16:01:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53051.log 2024-06-29T16:01:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.49905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80033.log 2024-06-29T16:01:57.986 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49905.log.gz 2024-06-29T16:01:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96629.log 2024-06-29T16:01:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.80033.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53051.log.gz 2024-06-29T16:01:57.986 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80033.log.gz 2024-06-29T16:01:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113011.log 2024-06-29T16:01:57.986 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112506.log 2024-06-29T16:01:57.987 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.96629.log.gz 2024-06-29T16:01:57.987 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.113011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57396.log 2024-06-29T16:01:57.987 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112506.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.113011.log.gz 2024-06-29T16:01:57.987 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112506.log.gz 2024-06-29T16:01:57.987 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84661.log 2024-06-29T16:01:57.987 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87638.log 2024-06-29T16:01:57.987 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57396.log.gz 2024-06-29T16:01:57.987 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37769.log 2024-06-29T16:01:57.988 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87638.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84661.log.gz 2024-06-29T16:01:57.988 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.87638.log.gz 2024-06-29T16:01:57.988 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104903.log 2024-06-29T16:01:57.988 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69404.log 2024-06-29T16:01:57.988 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37769.log.gz 2024-06-29T16:01:57.988 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90806.log 2024-06-29T16:01:57.988 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69404.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104903.log.gz 2024-06-29T16:01:57.988 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69404.log.gz 2024-06-29T16:01:57.989 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87992.log 2024-06-29T16:01:57.989 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.90806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57963.log 2024-06-29T16:01:57.989 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90806.log.gz 2024-06-29T16:01:57.989 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52925.log 2024-06-29T16:01:57.989 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.57963.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87992.log.gz 2024-06-29T16:01:57.989 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57963.log.gz 2024-06-29T16:01:57.989 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91255.log 2024-06-29T16:01:57.989 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74798.log 2024-06-29T16:01:57.990 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.52925.log.gz 2024-06-29T16:01:57.990 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103854.log 2024-06-29T16:01:57.990 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.74798.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91255.log.gz 2024-06-29T16:01:57.990 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74798.log.gz 2024-06-29T16:01:57.990 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110886.log 2024-06-29T16:01:57.990 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87950.log 2024-06-29T16:01:57.990 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.103854.log.gz 2024-06-29T16:01:57.990 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59092.log 2024-06-29T16:01:57.991 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.87950.log: 83.5% -- replaced with /var/log/ceph/ceph-client.admin.110886.log.gz 2024-06-29T16:01:57.991 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87950.log.gz 2024-06-29T16:01:57.991 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63467.log 2024-06-29T16:01:57.991 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.59092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64696.log 2024-06-29T16:01:57.991 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59092.log.gz 2024-06-29T16:01:57.991 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.63467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65529.log 2024-06-29T16:01:57.991 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.64696.log: 87.4% 0.0% -- replaced with /var/log/ceph/ceph-client.0.64696.log.gz 2024-06-29T16:01:57.991 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63467.log.gz 2024-06-29T16:01:57.992 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71316.log 2024-06-29T16:01:57.992 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107382.log 2024-06-29T16:01:57.992 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65529.log.gz 2024-06-29T16:01:57.992 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66728.log 2024-06-29T16:01:57.992 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.71316.log.gz/var/log/ceph/ceph-client.admin.107382.log: 2024-06-29T16:01:57.992 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107382.log.gz 2024-06-29T16:01:57.992 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85903.log 2024-06-29T16:01:57.993 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111445.log 2024-06-29T16:01:57.993 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66728.log.gz 2024-06-29T16:01:57.993 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59891.log 2024-06-29T16:01:57.993 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111445.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.85903.log.gz 2024-06-29T16:01:57.993 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.111445.log.gz 2024-06-29T16:01:57.993 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68458.log 2024-06-29T16:01:57.993 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.59891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112420.log 2024-06-29T16:01:57.993 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59891.log.gz 2024-06-29T16:01:57.994 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110927.log 2024-06-29T16:01:57.994 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112420.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.68458.log.gz 2024-06-29T16:01:57.994 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112420.log.gz 2024-06-29T16:01:57.994 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71652.log 2024-06-29T16:01:57.994 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88730.log 2024-06-29T16:01:57.994 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.110927.log.gz 2024-06-29T16:01:57.994 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104945.log 2024-06-29T16:01:57.994 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88730.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.71652.log.gz 2024-06-29T16:01:57.995 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88730.log.gz 2024-06-29T16:01:57.995 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55944.log 2024-06-29T16:01:57.995 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99782.log 2024-06-29T16:01:57.995 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104945.log.gz 2024-06-29T16:01:57.995 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.55944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88137.log 2024-06-29T16:01:57.995 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99782.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.55944.log.gz 2024-06-29T16:01:57.995 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.99782.log.gz 2024-06-29T16:01:57.995 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48456.log 2024-06-29T16:01:57.996 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52883.log 2024-06-29T16:01:57.996 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88137.log.gz 2024-06-29T16:01:57.996 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.48456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104777.log 2024-06-29T16:01:57.996 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.52883.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48456.log.gz 2024-06-29T16:01:57.996 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.52883.log.gz 2024-06-29T16:01:57.996 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108018.log 2024-06-29T16:01:57.996 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.104777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95684.log 2024-06-29T16:01:57.996 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104777.log.gz 2024-06-29T16:01:57.997 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108060.log 2024-06-29T16:01:57.997 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.108018.log.gz 2024-06-29T16:01:57.997 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.95684.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.95684.log.gz 2024-06-29T16:01:57.997 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107490.log 2024-06-29T16:01:57.997 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79818.log 2024-06-29T16:01:57.997 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108060.log.gz 2024-06-29T16:01:57.997 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.107490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50283.log 2024-06-29T16:01:57.997 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.107490.log.gz 2024-06-29T16:01:57.998 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79818.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79818.log.gz 2024-06-29T16:01:57.998 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76331.log 2024-06-29T16:01:57.998 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96041.log 2024-06-29T16:01:57.998 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50283.log.gz 2024-06-29T16:01:57.998 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103812.log 2024-06-29T16:01:57.998 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96041.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76331.log.gz 2024-06-29T16:01:57.998 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.96041.log.gz 2024-06-29T16:01:57.998 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68416.log 2024-06-29T16:01:57.998 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69745.log 2024-06-29T16:01:57.999 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103812.log.gz 2024-06-29T16:01:57.999 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.68416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112243.log 2024-06-29T16:01:57.999 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69745.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.68416.log.gz 2024-06-29T16:01:57.999 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69745.log.gz 2024-06-29T16:01:57.999 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96461.log 2024-06-29T16:01:57.999 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.112243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89449.log 2024-06-29T16:01:57.999 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112243.log.gz 2024-06-29T16:01:57.999 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.96461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65655.log 2024-06-29T16:01:58.000 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89449.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96461.log.gz 2024-06-29T16:01:58.000 INFO:teuthology.orchestra.run.smithi081.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.89449.log.gz 2024-06-29T16:01:58.000 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43988.log 2024-06-29T16:01:58.000 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108928.log 2024-06-29T16:01:58.000 INFO:teuthology.orchestra.run.smithi081.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.65655.log.gz 2024-06-29T16:01:58.000 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.43988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45277.log 2024-06-29T16:01:58.000 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108928.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43988.log.gz 2024-06-29T16:01:58.000 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108928.log.gz 2024-06-29T16:01:58.001 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50949.log 2024-06-29T16:01:58.001 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76246.log 2024-06-29T16:01:58.001 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.45277.log.gz 2024-06-29T16:01:58.001 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67696.log 2024-06-29T16:01:58.001 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76246.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.50949.log.gz 2024-06-29T16:01:58.001 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76246.log.gz 2024-06-29T16:01:58.001 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97053.log 2024-06-29T16:01:58.001 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50031.log 2024-06-29T16:01:58.002 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.67696.log.gz 2024-06-29T16:01:58.002 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.97053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91941.log 2024-06-29T16:01:58.002 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50031.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97053.log.gz 2024-06-29T16:01:58.002 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50031.log.gz 2024-06-29T16:01:58.002 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51404.log 2024-06-29T16:01:58.002 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85520.log 2024-06-29T16:01:58.002 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91941.log.gz 2024-06-29T16:01:58.002 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.51404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91458.log 2024-06-29T16:01:58.003 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85520.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51404.log.gz 2024-06-29T16:01:58.003 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85520.log.gz 2024-06-29T16:01:58.003 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72936.log 2024-06-29T16:01:58.003 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89281.log 2024-06-29T16:01:58.003 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91458.log.gz 2024-06-29T16:01:58.003 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77608.log 2024-06-29T16:01:58.005 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.89281.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.72936.log.gz 2024-06-29T16:01:58.006 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.89281.log.gz 2024-06-29T16:01:58.006 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60000.log 2024-06-29T16:01:58.006 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75113.log 2024-06-29T16:01:58.006 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.77608.log.gz 2024-06-29T16:01:58.006 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.60000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80707.log 2024-06-29T16:01:58.006 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75113.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.60000.log.gz 2024-06-29T16:01:58.006 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75113.log.gz 2024-06-29T16:01:58.006 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86214.log 2024-06-29T16:01:58.007 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.80707.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70335.log 2024-06-29T16:01:58.007 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.0.80707.log.gz 2024-06-29T16:01:58.007 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37875.log 2024-06-29T16:01:58.007 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86214.log: /var/log/ceph/ceph-client.0.70335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.70335.log.gz 2024-06-29T16:01:58.007 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86214.log.gz 2024-06-29T16:01:58.007 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86382.log 2024-06-29T16:01:58.007 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37875.log.gz 2024-06-29T16:01:58.008 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72766.log 2024-06-29T16:01:58.008 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79310.log 2024-06-29T16:01:58.008 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86382.log.gz 2024-06-29T16:01:58.008 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.72766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50241.log 2024-06-29T16:01:58.008 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79310.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72766.log.gz 2024-06-29T16:01:58.008 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79310.log.gz 2024-06-29T16:01:58.008 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102997.log 2024-06-29T16:01:58.008 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.50241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71988.log 2024-06-29T16:01:58.008 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50241.log.gz 2024-06-29T16:01:58.009 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69064.log 2024-06-29T16:01:58.009 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71988.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.102997.log.gz 2024-06-29T16:01:58.009 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71988.log.gz 2024-06-29T16:01:58.009 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88857.log 2024-06-29T16:01:58.009 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71946.log 2024-06-29T16:01:58.009 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69064.log.gz 2024-06-29T16:01:58.009 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95600.log 2024-06-29T16:01:58.009 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.71946.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88857.log.gz 2024-06-29T16:01:58.010 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71946.log.gz 2024-06-29T16:01:58.010 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69703.log 2024-06-29T16:01:58.010 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.95600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65053.log 2024-06-29T16:01:58.010 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.95600.log.gz 2024-06-29T16:01:58.010 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105071.log 2024-06-29T16:01:58.010 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.65053.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69703.log.gz 2024-06-29T16:01:58.010 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65053.log.gz 2024-06-29T16:01:58.010 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62359.log 2024-06-29T16:01:58.011 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77479.log 2024-06-29T16:01:58.011 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105071.log.gz 2024-06-29T16:01:58.011 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.62359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38097.log 2024-06-29T16:01:58.011 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77479.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62359.log.gz 2024-06-29T16:01:58.011 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77479.log.gz 2024-06-29T16:01:58.011 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45289.log 2024-06-29T16:01:58.011 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84525.log 2024-06-29T16:01:58.011 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.45289.log: /var/log/ceph/ceph-client.admin.38097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38097.log.gz 2024-06-29T16:01:58.012 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93186.log 2024-06-29T16:01:58.012 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45289.log.gz 2024-06-29T16:01:58.012 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77002.log 2024-06-29T16:01:58.012 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84525.log: /var/log/ceph/ceph-client.admin.93186.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.93186.log.gz 2024-06-29T16:01:58.012 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.84525.log.gz 2024-06-29T16:01:58.012 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109576.log 2024-06-29T16:01:58.012 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86574.log 2024-06-29T16:01:58.012 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77002.log.gz 2024-06-29T16:01:58.012 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98603.log 2024-06-29T16:01:58.013 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86574.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109576.log.gz 2024-06-29T16:01:58.013 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86574.log.gz 2024-06-29T16:01:58.013 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87806.log 2024-06-29T16:01:58.013 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54410.log 2024-06-29T16:01:58.013 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98603.log: /var/log/ceph/ceph-client.admin.87806.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.98603.log.gz 2024-06-29T16:01:58.013 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.87806.log.gz 2024-06-29T16:01:58.013 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63642.log 2024-06-29T16:01:58.013 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95463.log 2024-06-29T16:01:58.014 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.54410.log: /var/log/ceph/ceph-client.admin.63642.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54410.log.gz 2024-06-29T16:01:58.014 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.63642.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2024-06-29T16:01:58.014 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:58.014 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93546.log 2024-06-29T16:01:58.014 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44475.log: /var/log/ceph/ceph-client.admin.95463.log: 87.5% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95463.log.gz 2024-06-29T16:01:58.014 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-06-29T16:01:58.014 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64859.log 2024-06-29T16:01:58.014 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76726.log 2024-06-29T16:01:58.015 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.93546.log: /var/log/ceph/ceph-client.admin.64859.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93546.log.gz 2024-06-29T16:01:58.015 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.64859.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46579.log 2024-06-29T16:01:58.015 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:58.015 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67532.log 2024-06-29T16:01:58.015 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76726.log: /var/log/ceph/ceph-client.admin.46579.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76726.log.gz 2024-06-29T16:01:58.015 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99825.log 2024-06-29T16:01:58.015 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46579.log.gz 2024-06-29T16:01:58.015 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.67532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105972.log 2024-06-29T16:01:58.015 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.67532.log.gz 2024-06-29T16:01:58.016 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79268.log 2024-06-29T16:01:58.016 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.99825.log.gz 2024-06-29T16:01:58.016 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64400.log 2024-06-29T16:01:58.016 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105972.log: /var/log/ceph/ceph-client.admin.79268.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105972.log.gz 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79268.log.gz 2024-06-29T16:01:58.016 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:01:58.016 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108490.log 2024-06-29T16:01:58.016 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.64400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86168.log 2024-06-29T16:01:58.016 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64400.log.gz 2024-06-29T16:01:58.017 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.108490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46243.log 2024-06-29T16:01:58.017 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.86168.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108490.log.gz 2024-06-29T16:01:58.017 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86168.log.gz 2024-06-29T16:01:58.017 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74679.log 2024-06-29T16:01:58.017 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66303.log 2024-06-29T16:01:58.017 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.46243.log: /var/log/ceph/ceph-client.admin.74679.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46243.log.gz 2024-06-29T16:01:58.017 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.74679.log.gz 2024-06-29T16:01:58.017 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105491.log 2024-06-29T16:01:58.017 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66303.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66303.log.gz 2024-06-29T16:01:58.018 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109618.log 2024-06-29T16:01:58.018 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69531.log 2024-06-29T16:01:58.018 INFO:teuthology.orchestra.run.smithi081.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105491.log.gz 2024-06-29T16:01:58.018 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.109618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85776.log 2024-06-29T16:01:58.018 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.69531.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109618.log.gz 2024-06-29T16:01:58.018 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69531.log.gz 2024-06-29T16:01:58.018 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84430.log 2024-06-29T16:01:58.018 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.85776.log: -- /var/log/ceph/ceph-client.admin.98203.log 2024-06-29T16:01:58.019 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85776.log.gz 2024-06-29T16:01:58.019 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.84430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92277.log 2024-06-29T16:01:58.019 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.98203.log: 87.6% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98203.log.gz 2024-06-29T16:01:58.019 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84430.log.gz 2024-06-29T16:01:58.019 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79039.log 2024-06-29T16:01:58.019 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.92277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88899.log 2024-06-29T16:01:58.019 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92277.log.gz 2024-06-29T16:01:58.019 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86762.log 2024-06-29T16:01:58.020 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.88899.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79039.log.gz 2024-06-29T16:01:58.020 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.88899.log.gz 2024-06-29T16:01:58.020 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65823.log 2024-06-29T16:01:58.020 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.86762.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.0.86762.log.gz 2024-06-29T16:01:58.020 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67344.log 2024-06-29T16:01:58.020 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99961.log 2024-06-29T16:01:58.020 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.67344.log: /var/log/ceph/ceph-client.admin.65823.log: 87.7% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67344.log.gz 2024-06-29T16:01:58.020 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65823.log.gz 2024-06-29T16:01:58.020 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102579.log 2024-06-29T16:01:58.021 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100223.log 2024-06-29T16:01:58.021 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.99961.log.gz 2024-06-29T16:01:58.021 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.102579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103081.log 2024-06-29T16:01:58.021 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.100223.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.100223.log.gz 2024-06-29T16:01:58.021 INFO:teuthology.orchestra.run.smithi081.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.102579.log.gz 2024-06-29T16:01:58.021 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.107261.log 2024-06-29T16:01:58.021 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66942.log 2024-06-29T16:01:58.021 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.103081.log: /var/log/ceph/ceph-client.0.107261.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.103081.log.gz 2024-06-29T16:01:58.022 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.107261.log.gz 2024-06-29T16:01:58.022 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91647.log 2024-06-29T16:01:58.022 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.66942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53177.log 2024-06-29T16:01:58.022 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66942.log.gz 2024-06-29T16:01:58.022 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.91647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79991.log 2024-06-29T16:01:58.022 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.53177.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91647.log.gz 2024-06-29T16:01:58.022 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53177.log.gz 2024-06-29T16:01:58.022 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101100.log 2024-06-29T16:01:58.022 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.79991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75738.log 2024-06-29T16:01:58.023 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79991.log.gz 2024-06-29T16:01:58.023 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.101100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111781.log 2024-06-29T16:01:58.023 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75738.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101100.log.gz 2024-06-29T16:01:58.023 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75738.log.gz 2024-06-29T16:01:58.023 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105930.log 2024-06-29T16:01:58.023 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.111781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76642.log 2024-06-29T16:01:58.023 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.111781.log.gz 2024-06-29T16:01:58.023 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.105930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85695.log 2024-06-29T16:01:58.024 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76642.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105930.log.gz 2024-06-29T16:01:58.024 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76642.log.gz 2024-06-29T16:01:58.024 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99561.log 2024-06-29T16:01:58.024 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.85695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70822.log 2024-06-29T16:01:58.024 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85695.log.gz 2024-06-29T16:01:58.024 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.99561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75239.log 2024-06-29T16:01:58.024 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.70822.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.99561.log.gz 2024-06-29T16:01:58.024 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70822.log.gz 2024-06-29T16:01:58.024 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44517.log 2024-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.75239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37667.log 2024-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.75239.log.gz 2024-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76373.log 2024-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37667.log: 74.0% -- replaced with /var/log/ceph/ceph-client.admin.37667.log.gz 2024-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.44517.log.gz 2024-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58089.log 2024-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.76373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44391.log 2024-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76373.log.gz 2024-06-29T16:01:58.025 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.58089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77967.log 2024-06-29T16:01:58.026 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.44391.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58089.log.gz 2024-06-29T16:01:58.026 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44391.log.gz 2024-06-29T16:01:58.026 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110789.log 2024-06-29T16:01:58.026 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.77967.log: 84.1% -- replaced with /var/log/ceph/ceph-client.admin.77967.log.gz 2024-06-29T16:01:58.026 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.110789.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.110789.log.gz 2024-06-29T16:01:58.228 INFO:teuthology.orchestra.run.smithi081.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-06-29T16:01:58.343 INFO:teuthology.orchestra.run.smithi167.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-06-29T16:01:59.015 INFO:teuthology.orchestra.run.smithi167.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-06-29T16:01:59.057 INFO:teuthology.orchestra.run.smithi167.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-06-29T16:01:59.305 INFO:teuthology.orchestra.run.smithi167.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-06-29T16:02:00.120 INFO:teuthology.orchestra.run.smithi081.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-06-29T16:02:00.415 INFO:teuthology.orchestra.run.smithi167.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-06-29T16:02:00.449 INFO:teuthology.orchestra.run.smithi081.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-06-29T16:02:00.534 INFO:teuthology.orchestra.run.smithi167.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-06-29T16:02:01.049 INFO:teuthology.orchestra.run.smithi081.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-06-29T16:02:01.141 INFO:teuthology.orchestra.run.smithi081.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-06-29T16:02:01.148 INFO:teuthology.orchestra.run.smithi167.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-06-29T16:02:01.155 INFO:teuthology.orchestra.run.smithi167.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-06-29T16:02:01.157 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-06-29T16:02:01.157 INFO:teuthology.orchestra.run.smithi167.stderr:real 0m4.044s 2024-06-29T16:02:01.157 INFO:teuthology.orchestra.run.smithi167.stderr:user 0m21.159s 2024-06-29T16:02:01.157 INFO:teuthology.orchestra.run.smithi167.stderr:sys 0m0.790s 2024-06-29T16:02:01.174 INFO:teuthology.orchestra.run.smithi081.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-06-29T16:02:01.176 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-06-29T16:02:01.176 INFO:teuthology.orchestra.run.smithi081.stderr:real 0m4.086s 2024-06-29T16:02:01.176 INFO:teuthology.orchestra.run.smithi081.stderr:user 0m18.568s 2024-06-29T16:02:01.176 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m1.326s 2024-06-29T16:02:01.177 INFO:tasks.ceph:Archiving logs... 2024-06-29T16:02:01.177 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926/remote/smithi081/log 2024-06-29T16:02:01.178 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-29T16:02:04.575 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926/remote/smithi167/log 2024-06-29T16:02:04.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-29T16:02:07.221 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-29T16:02:07.231 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-29T16:02:07.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-29T16:02:07.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-29T16:02:07.324 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-29T16:02:07.334 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-29T16:02:07.334 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-29T16:02:07.337 DEBUG:teuthology.orchestra.run.smithi167:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-29T16:02:07.350 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-06-29T16:02:07.353 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-06-29T16:02:07.355 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-29T16:02:07.356 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-06-29T16:02:07.356 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv01.front.sepia.ceph.com 3 7 377 121 +1541us[+1577us] +/- 50ms 2024-06-29T16:02:07.356 INFO:teuthology.orchestra.run.smithi081.stdout:^* hv02.front.sepia.ceph.com 3 7 377 55 -1010us[ -971us] +/- 61ms 2024-06-29T16:02:07.356 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 243 -818us[ -632us] +/- 71ms 2024-06-29T16:02:07.356 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T16:02:07.358 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-29T16:02:07.358 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-06-29T16:02:07.358 INFO:teuthology.orchestra.run.smithi167.stdout:^* hv01.front.sepia.ceph.com 3 8 377 56 +1566us[+1553us] +/- 51ms 2024-06-29T16:02:07.358 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 53 -1029us[-1029us] +/- 61ms 2024-06-29T16:02:07.359 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 122 -797us[ -808us] +/- 73ms 2024-06-29T16:02:07.359 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-29T16:02:07.360 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-29T16:02:07.368 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-29T16:02:07.369 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-29T16:02:07.378 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-29T16:02:07.504 DEBUG:teuthology.orchestra.run.smithi167:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-29T16:02:07.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-29T16:02:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:02:07.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-29T16:02:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:02:07.641 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-29T16:02:07.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-29T16:02:07.695 INFO:teuthology.task.internal:Duration was 6401.495255 seconds 2024-06-29T16:02:07.695 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-29T16:02:07.705 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-29T16:02:07.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-29T16:02:07.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-29T16:02:07.744 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-29T16:02:07.749 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-29T16:02:07.944 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-29T16:02:07.945 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-06-29T16:02:07.945 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-29T16:02:07.968 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2024-06-29T16:02:07.968 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-29T16:02:08.008 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-29T16:02:08.008 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-29T16:02:08.011 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-29T16:02:08.075 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-29T16:02:08.075 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-29T16:02:08.078 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-29T16:02:08.266 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-29T16:02:08.277 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-29T16:02:08.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-29T16:02:08.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-29T16:02:08.327 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-29T16:02:08.338 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-29T16:02:08.344 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-29T16:02:08.365 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-06-29T16:02:08.390 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2024-06-29T16:02:08.404 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-29T16:02:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:02:08.433 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-29T16:02:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-29T16:02:08.459 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-29T16:02:08.472 INFO:teuthology.task.internal:Transferring archived files... 2024-06-29T16:02:08.473 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926/remote/smithi081 2024-06-29T16:02:08.473 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-29T16:02:08.510 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-06-29_14:01:10-fs-wip-mchangir-qa-handle-correct-CephManager-instantiation-debug-distro-default-smithi/7778926/remote/smithi167 2024-06-29T16:02:08.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-29T16:02:08.545 INFO:teuthology.task.internal:Removing archive directory... 2024-06-29T16:02:08.546 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-29T16:02:08.549 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-29T16:02:08.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-29T16:02:08.607 INFO:teuthology.task.internal:Not uploading archives. 2024-06-29T16:02:08.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-29T16:02:08.618 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-29T16:02:08.619 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-29T16:02:08.621 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-29T16:02:08.633 INFO:teuthology.orchestra.run.smithi081.stdout: 265811 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 29 16:02 /home/ubuntu/cephtest 2024-06-29T16:02:08.650 INFO:teuthology.orchestra.run.smithi167.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 29 16:02 /home/ubuntu/cephtest 2024-06-29T16:02:08.651 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-29T16:02:08.661 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-29T16:02:08.688 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} duration: 6401.495254993439 failure_reason: 'Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments)' flavor: default owner: scheduled_mchangir@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=92dce96014d640689d2fe9400fdde10f status: fail success: false 2024-06-29T16:02:08.688 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-29T16:02:08.775 INFO:teuthology.run:FAIL